Tortoise: Scaling in QKVAttentionLegacy

[farisalasmary]

Hello community,

I came across this line when I was reading the code and I was wondering why the math.sqrt is computed twice on the ch which means that the computed number is the fourth root of ch.
Is this a typo or is it by design? Can anyone explain, please?

coqui-ai-TTS/TTS/tts/layers/tortoise/arch_utils.py

Line 67 in 36c6ff8

scale = 1 / math.sqrt(math.sqrt(ch))

[eginhard]

It's in the original code as well, so likely by design: https://github.com/neonbjb/tortoise-tts/blob/8a2563ecabe93c4fb626f876dd0c52c966edef2f/tortoise/models/arch_util.py#L64

I don't see any particular reason mentioned in the paper, but it probably helps to make training more stable.