Rotary Embeddings in Encoder

[conceptofmind]

Hi all,

I was wondering how ColossalAI handled issues relating to length extrapolation for Rotary Embeddings and relative positional encoding for an encoder? From my understanding, Rotary Embeddings only work properly for the decoder since they do not length-generalize that well.

Thank you,

Enrico