The scale of a weight that feeds an RMSNorm is a gauge the loss cannot see, the gradient cannot move, and only weight decay controls, which is what sets the effective learning rate under RMSNorm.
In a pre-normalization block the weight W producing z = x W is followed by an
RMSNorm that divides z by its own root-mean-square. Scaling W by any positive
constant scales z by the same constant, which the norm removes exactly:
RMSNorm(x (cW)) = RMSNorm(c (x W)) = RMSNorm(x W) , c > 0 .
Measured over c from 0.01 to 100, the output deviation is about 3e-15 in double
precision. The scale of W is a gauge freedom.
Because the loss is invariant to the scale of W, it is homogeneous of degree zero
in W, and Euler's theorem gives, exactly,
<grad_W L, W> = 0 .
The gradient has no radial component. With a genuinely nonzero gradient (norm about
3.6 on a real downstream loss), the radial part <grad, W> measures about 1e-15.
Learning can rotate W on the sphere of constant norm but cannot change the norm.
This is the forward-side companion to the RMSNorm backward Jacobian annihilating the
scale direction: the scale is a complete gauge, unseen in the forward pass and
untouched by the gradient.
Under a decoupled update W <- W - lr g - wd W, the squared norm evolves as
||W_next||^2 = (1 - 2 wd) ||W||^2 - 2 lr <g, W> + lr^2 ||g||^2
= (1 - 2 wd) ||W||^2 + lr^2 ||g||^2 ,
because the middle term is exactly zero. Decay shrinks the norm geometrically while
the tangential step re-inflates it at second order, and they balance at a fixed
point 2 wd ||W*||^2 = lr^2 ||g||^2: the weight settles onto a sphere and rotates
on it, a rotational equilibrium. The measured squared-norm trajectory tracks this
zero-radial prediction to a relative 4e-4.
The rotation pace is what the network experiences as learning. The gradient is
tangent, so the angle it turns W through per step is about ||lr g|| / ||W||. As
weight decay shrinks the gauge, this effective angular learning rate grows like
1 / ||W||: halving the norm quadruples the per-step rotation (measured 0.0025,
0.010, 0.040 at norm scales 1, 1/2, 1/4). The scale is invisible to the
loss, yet weight decay's grip on it is exactly what sets how fast the model learns.
gauge.py: the exact forward scale-invariance and the Euler zero-radial-gradient identity.dynamics.py: the norm trajectory matching(1 - 2 wd) ||W||^2 + lr^2 ||g||^2, the rotational equilibrium, and the1 / ||W||effective angular learning rate.test_rmsgauge.py: the invariance, the zero radial gradient, the norm dynamics, the equilibrium, and the effective-rate growth as tests.
python gauge.py
python dynamics.py
python test_rmsgauge.py
The rotational-equilibrium behaviour of a scale-invariant weight under weight decay is the spherical-motion / effective-learning-rate phenomenon known from prior work. What this repository adds is the exact treatment for RMSNorm: the machine-precision forward gauge, the Euler identity making the radial gradient exactly zero, and the pairing with the RMSNorm backward null space, so the pre-norm weight scale is a complete gauge whose only handle is weight decay.