Question: Position of BN in BAM.

[Ujjawal-K-Panchal]

I think there is possibly an inconsistency in the location of the application of the Batch Normalization between the paper and the code.

1st instance of possible inconsistency:

The paper mentions on Page 4, Section 3, subsection Channel attention branch that:

So it is understod that the batch norm is applied at the end of the MLP, i.e. after the final layer. However, from the implementation, shows final layer in forward() function as:

attention-module/MODELS/bam.py

Line 25 in 459efad

return self.gate_c( avg_pool ).unsqueeze(2).unsqueeze(3).expand_as(in_tensor)

Where self.gate_c's final layer is defined as:

attention-module/MODELS/bam.py

Line 22 in 459efad

self.gate_c.add_module( 'gate_c_fc_final', nn.Linear(gate_channels[-2], gate_channels[-1]) )

[Ujjawal-K-Panchal]

2nd instance of possible inconsistency:

The paper mentions on Page 5, Section 3, subsection Spatial attention branch that:

So it is understod that the batch norm is applied at the end of the spatial attention branch, i.e. after the final Linear layer. However, from the implementation, shows final layer in forward() function as:

attention-module/MODELS/bam.py

Line 41 in 459efad

return self.gate_s( in_tensor ).expand_as(in_tensor)

Where self.gate_s's final layer is defined as:

attention-module/MODELS/bam.py

Line 39 in 459efad

self.gate_s.add_module( 'gate_s_conv_final', nn.Conv2d(gate_channel//reduction_ratio, 1, kernel_size=1) )

[Ujjawal-K-Panchal]

One might suspect that the final BN layers can be added in the BAM() class.
However, looking at the BAM() class:

attention-module/MODELS/bam.py

Line 48 in 459efad

att = 1 + F.sigmoid( self.channel_att(in_tensor) * self.spatial_att(in_tensor) )

it becomes clear that the BN layer is not applied before the combination of $M_c \text{ and } M_s$.
Is this an inconsistency?