Do you realized key_padding_mask in you long-term attention and short-term Attention?

[zhouweii234]

Do you realized key_padding_mask (a parameter in torch.nn.multiheadattention) in the long-term attention and short-term Attention? Or you just do not need this in your network?

[z-x-yang]

No. The key_padding_mask is useless for long-term attention and short-term.

[zhouweii234]

Thank you for your reply! By the way , do you use attn_mask?

[z-x-yang]

Long-term attention doesn't need any mask. The implementation of short-term attention is complicated. Since short-term attention has some processes related to the relative positions between pixels. Masking out the correlations out of feature boundary is necessary.