RMSNorm with bias kwarg option

[PeaBrane]

🚀 The feature, motivation and pitch

Currently, the RMSNorm module supports the elementwise_affine kwarg but not the bias kwarg. Would it be easy to implement a mode where elementwise_affine is set True, but bias can be set False, like LayerNorm?

This sort of configuration is fairly prevalent and is used by many modern LLM models including Llama(s), and Mamba(s).

Alternatives

No response

Additional context

No response

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki

[PeaBrane]

I actually realized just now that the current behavior of nn.RMSNorm is to not apply a bias, which actually fits the common usage. This is probably good to close, unless there is a popular usage where bias=True for RMSNorm (which I'm currently not aware)