Question about Encoder_attn in decoder layer

[gaopengcuhk]

It seems that in the decoder, both self attention and TaLK has been used. Have you ever try to replace Self Attention(Co attention between encoder input) with TaLK?

[lioutasb]

TaLK convolutions, similar to Dynamic convolutions [Wu et al., 2019], is a method for replacing self-attention. The current form of TaLK convolutions doesn't support replacing co-attention between two different sequences (i.e. source and target sentences).