ngeodesic

nGPT describes its normalized residual as a retraction toward a block output on the unit sphere. With a scalar step size it is not the geodesic step: it lags the true geodesic by an exact angle law that vanishes only at three points, one of which halves the angle exactly. With its real per-dimension step size it is not even a spherical interpolation: it leaves the geodesic plane entirely.

nGPT (Loshchilov et al. 2024) replaces the residual with h <- Norm(h + alpha (.) (h_A - h)), h and h_A unit vectors, alpha a learnable per-dimension eigen learning rate. This is normalized linear interpolation: the chord is projected back to the sphere.

Scalar step: an exact retraction-versus-geodesic gap

A true geodesic (SLERP) step of size alpha moves the angle exactly alpha * theta, theta = angle(h, h_A). For a scalar alpha, nGPT's update stays in the plane span{h, h_A} but moves

phi(alpha, theta) = arccos( ((1 - alpha) + alpha cos theta)
                            / sqrt((1 - alpha)^2 + alpha^2 + 2 alpha (1 - alpha) cos theta) ),

and the gap G = alpha theta - phi has a rigid exact structure:

• Triple zero: G = 0 exactly at alpha in {0, 1/2, 1}, and G is odd about 1/2.
• The alpha = 1/2 identity: the chord midpoint normalizes onto the geodesic bisector, so phi(1/2, theta) = theta/2 exactly for every theta (measured to 4e-14 degrees). Iterating toward a fixed target halves the angle exactly every block.
• Small-angle law: max_alpha |G| ~ theta^3 (measured ratio 8.15, 8.04, 8.01 as theta halves), so the retraction agrees with the geodesic to second order.

The gap is a real several degrees where models operate: max_alpha |G| is 0.46, 4.07, 17.0, 49.6 degrees at theta = 45, 90, 135, 170.

Per-dimension step: not even planar

The paper's alpha is per-dimension, a diagonal variable metric. Then alpha (.) (h_A - h) is not a scalar multiple of h_A - h, so the updated point has a component orthogonal to span{h, h_A}. Measured for a typical eigen learning rate (alpha_i ~ N(0.3, 0.1), d = 1024): the off-plane norm is 0.18, 0.18, 0.20 across seeds, about a fifth of a unit vector, while the scalar-alpha control is machine zero. So nGPT's residual is a diagonal-metric retraction, not a SLERP or any geodesic interpolation between h and h_A.

For a scalar step, iterating toward a fixed target contracts the angle at the asymptotic rate 1 - alpha per block, exact at every step for alpha = 1/2.

Layout

• geodesic.py: the scalar-alpha retraction angle, the SLERP reference, the triple-zero gap, the alpha = 1/2 exact halving, and the theta^3 law.
• offplane.py: the per-dimension off-plane departure, the scalar in-plane control, and the 1 - alpha contraction rate.
• test_ngeodesic.py: the triple zero, the odd symmetry, the exact halving, a nonzero gap off the triple points, the theta^3 scaling, the off-plane departure, and the contraction rate.

Reproduce

python geodesic.py
python offplane.py
python test_ngeodesic.py

Note

That normalized linear interpolation differs from spherical linear interpolation is classical in graphics and quaternion interpolation; this repository does not claim it. What is here is the exact structure for nGPT's update: the triple-zero gap with the alpha = 1/2 exact-halving identity, the theta^3 deviation law, and, for the actual per-dimension eigen learning rate, the departure from the geodesic plane entirely, with a measured fp64 magnitude of about a fifth of a unit vector. This reframes nGPT's residual as a diagonal-metric retraction rather than a spherical interpolation. It is tangential geometry, distinct from the shipped radial-gauge results (rmsgauge, normnull). The named falsifier: if a trained nGPT's per-dimension alpha were near-constant across dimensions, the off-plane mass would vanish and the update would collapse to the scalar NLERP case; measuring std(alpha)/mean(alpha) on a checkpoint tests it.