softcapgrad

Tanh logit soft-capping is adopted over hard clipping because hard clipping has an exact-zero gradient at the rail. Its exact backward law is confidence-proportional gradient attenuation, and in bf16 it reintroduces the exact-zero-gradient dead-zone it was meant to avoid, at a precisely computable pre-cap horizon.

Soft-capping replaces a raw logit a with z = c * tanh(a / c), deployed in Gemma2 (attention cap c = 50, final logit cap c = 30) and Grok.

The exact gradient identity

The local derivative is exactly

d z / d a = 1 - (z / c)^2 = sech^2(a / c),

verified against autograd to 3.3e-16 over two million points. This factor is smallest for the largest-magnitude, most confident logit: 0.42 at a = c, 0.07 at a = 2c, 1.3e-3 at a = 4c, decaying asymptotically as 4 * exp(-2 a / c). So soft-capping is confidence-proportional gradient attenuation, and it is exact.

The output is bounded but the input is not. To make the output express confidence z = f * c the pre-activation must reach a = c * atanh(f), which diverges as f -> 1, while the gradient that would grow it is 1 - f^2 -> 0. Expressing z = 0.9c needs a = 44, z = 0.99c needs a = 79, z = 0.999c needs a = 114: high confidence forces large pre-activations against an exponentially vanishing gradient, a one-way ratchet.

The bf16 dead-zone

tanh(x) in a finite float format rounds to exactly 1.0 once 1 - tanh(x) drops below half a unit in the last place of 1.0. Past that point the stored output is the constant 1.0 and the gradient 1 - (z/c)^2 = 0 is exactly zero, in the format the model trains in. The knee is analytic: with 1 - tanh(x) ~ 2 exp(-2x) and half-ulp 2^-(p+1) for a p-bit mantissa, saturation is at x_knee = (ln2/2)(p + O(1)). For fp32 (p = 23) this is (ln2/2)(p+2) = 8.664, matched by direct search to 8.665. In bf16 the knee is x = 3.47, so the pre-cap horizon is a = 104 for c = 30 and a = 173 for c = 50. A bf16 gradient through the cap at a = 120 or a = 200 is bit-exactly 0.0.

So hard clipping's zero-gradient dead-zone is not removed by soft-capping, only relocated: out of the common regime, into bf16, and precisely to the confident end where a = c * atanh(0.999) = 114 > 104 already lands in it. That is the folklore inversion: the smooth alternative has its own exact-zero-gradient region.

What the attenuation does not do

It is tempting to read the pointwise 1/(1 - (z/c)^2) factor (over 4x at pre-cap 40) as a several-fold end-to-end training slowdown for a confidently-wrong logit. Measured under plain gradient descent, the total steps to drive a wrong class below probability 1e-3 grow only about 1.1x with the cap. The reason is exact: gradient descent spends almost all of its steps near the decision boundary, where z is small and the attenuation is close to 1; the large attenuation applies only in the brief confident phase. The pointwise law is exact and real, but it does not amplify into a large convergence penalty for a single logit, and this repository says so rather than reporting the pointwise factor as an end-to-end one.

Layout

• softcap.py: the exact gradient identity, the confidence-proportional attenuation, the asymptotic form, and the unbounded-input ratchet.
• deadzone.py: the bf16/fp16/fp32 tanh saturation knee, its analytic (ln2/2)(p+2) form, the pre-cap horizon, and the bit-exact-zero bf16 gradient past it.
• dynamics.py: the honest scoping, the end-to-end correction slowdown is ~1.1x, not the pointwise several-x, and why.
• test_softcapgrad.py: the identity, the attenuation ordering, the ratchet, the knee versus analytic, the exact-zero bf16 gradient, and the modest end-to-end slowdown.

Reproduce

python softcap.py
python deadzone.py
python dynamics.py
python test_softcapgrad.py

Note

That d/dx tanh = 1 - tanh^2 is elementary and appears as an implementation note in soft-cap writeups. What is here is the exact backward law read against the deployed claim: soft-capping is confidence-proportional attenuation with an unbounded input ratcheted against a vanishing gradient, and in bf16 it resurrects the hard-clip exact-zero-gradient dead-zone at the exact horizon a = 3.47 c (104 and 173 at the Gemma2 caps), which is a latent hazard reached only by the most confident activations. The pointwise attenuation is exact; whether it changes real pretraining is left to the honest end-to-end measurement, which finds the single-logit slowdown modest.