Is Llama4TextL2Norm meant to be RMS norm?

[0x6b64]

https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama4/modeling_llama4.py#L118

x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

This is just the rms norm?

[Rocketknight1]

That does look like the RMSNorm computation, yes. However, RMSNorm was only added to PyTorch in 2.3 or 2.4 I think, so we need to do it manually until our minimum supported torch version catches up!