attention scaling

[brianchmiel]

Hi,
I see in the code that you use by default "su" scaling policy - which is defined as sqrt[(1+log(L'/L)) /log(L)] .
Why do you use this scaling? I didn't find in the paper any comment about it

[J-shang]

Hi, we empirically set this scaling because the attention becomes dispersed as it’s spread thinly across numerous token positions