ALiBi in self-Attention

[Ldoun]

What is your question?

first of all nice work! and thank you for sharing the code.
I noticed that the code use ALiBi in encoder-decoder attention but not in the transformer's self-Attention. Have you tried ALiBi in transformer self-attention? And Is there a reason you didn't use it for the self-attention layer?

[ofirpress]

Thanks!
We've actually currently implemented ALiBi for self-attention (casually masked) only. We have not implemented it for encoder-decoder attention or non-masked encoder attention yet.

[Ldoun]

oh, I missed that. thank you!