How to deal with the conv in SE module？

[RainFrost1]

No description provided.

[WailordHe]

same question

[yxchng]

any updates?

[RainFrost1]

Looking forward your reply, any suggestions? @keyu-tian

[keyu-tian]

@RainFrost1 @will1973 @yxchng SE module contains two linear layers (channel-wise) and one global average pooling. No changes required to the linear layers. As for the global pooling on a masked feature map, one should calculate the mean value at unmasked positions only. To implement this you can use our function _get_active_ex_or_ii in https://github.com/keyu-tian/SparK/blob/main/pretrain/encoder.py#L30 like:

class SparseGlobalAveragePooling(nn.Module):
    def forward(self, x):   # shape: BCHW
        B, C, H, W = x.shape
        unmasked_positions = _get_active_ex_or_ii(H=H, W=W, returning_active_ex=True)  # shape: B1HW
        mean = (x * unmasked_positions).sum(dim=(2,3), keepdims=True) / unmasked_positions.sum(dim=(2,3), keepdims=True)
        return mean         # shape: BC11