About the Softmax in the CAM_Module

[lartpang]

DANet/encoding/nn/attention.py

Line 75 in 799424e

energy_new = torch.max(energy, -1, keepdim=True)[0].expand_as(energy)-energy

Is your code trying to improve numerical stability? Maybe it should be in this form.

       energy_new = torch.max(energy, -1, keepdim=True)
       energy_new = energy_new[0].expand_as(energy)
       energy_new = energy - energy_new
       attention = self.softmax(energy_new)

[junfu1115]

@lartpang Thanks, we have tried the form you provide, but the performance is not good

[lartpang]

Well, can I take it as a practical experience?

[junfu1115]

@lartpang This might be an attempt, in other cases,

[lartpang]

OK, thanks for your answer and your code...

This idea is so good, except for some extra memory... It's more suitable for a position where the number of channels is small.

[junfu1115]

@lartpang Thanks