clipcouple

Global-norm gradient clipping couples every parameter to every other: one layer's gradient spike throttles the whole model's step by an exact factor.

The standard gradient clip rescales the entire gradient by a single factor built from its global norm:

g_i  <-  g_i * min(1, tau / ||g||_global) ,   ||g||_global = sqrt(sum_i ||g_i||^2) .

The factor is shared across all parameters. So a spike in any one tensor inflates the global norm and shrinks the update of every other tensor by the same factor, including the perfectly healthy ones. A single layer's instability becomes a model-wide learning-rate cut.

The exact throttle

Take L tensors of roughly equal gradient norm, let one spike to S times its healthy size, and set the clip threshold tau at the healthy global norm. The shared clip factor, and hence the multiplier on every healthy layer's step, is exactly

f = 1 / sqrt(1 + (S^2 - 1) / L) .

For a large spike this approaches sqrt(L) / S, so one layer spiking by S cuts the whole model's effective learning rate by about S / sqrt(L). Measured with L = 12 tensors, a healthy layer's step is multiplied by 0.89 at S = 2, 0.33 at S = 10, and 0.069 at S = 50: a single fifty-fold spike drops every other layer to seven percent of its intended step, matching the law.

Per-tensor clipping does not couple

Clipping each tensor against its own norm leaves the healthy tensors untouched: under the same fifty-fold spike, a healthy layer's step multiplier is 0.99. The coupling is a property of the shared global factor, not of clipping itself. Global clipping trades per-parameter isolation for a single well-behaved global norm, and the cost is that any one spike is broadcast to the entire model as a step-size cut.

Layout

• coupling.py: the shared-factor clip, the exact f = 1/sqrt(1 + (S^2-1)/L) law and its sqrt(L)/S asymptote, and the per-tensor contrast.
• test_clipcouple.py: the exact factor, the throttle, the asymptote, the per-tensor isolation, and the clip-to-tau property as tests.

Reproduce

python coupling.py
python test_clipcouple.py

The rule

Under global-norm clipping, a gradient spike anywhere is a learning-rate cut everywhere, by 1/sqrt(1 + (S^2-1)/L). If one layer or one modality is prone to spikes and you do not want it to throttle the rest of the model, clip per tensor or per group rather than globally, or detect and drop the spike before the shared norm is formed.