shiftgauge

Softmax gauges away the absolute logit level, so an attention row must always sum to one: forced allocation, the mechanism behind attention sinks. Softpick breaks that gauge. Its row sum is an exact ratio that can be zero, it reads the absolute logit level softmax throws away, and because it is not shift-invariant its stabilization needs the paper's otherwise-mysterious correction term rather than the usual subtract-the-max.

Softpick (Zuhri et al. 2025) replaces the softmax attention normalizer with softpick(x)_i = ReLU(e^{x_i} - 1) / sum_j |e^{x_j} - 1|.

The row sum is P / (P + N)

Splitting the denominator by the sign of e^{x_j} - 1, which is the sign of x_j, with

P = sum_{x_i > 0} (e^{x_i} - 1) ,   N = sum_{x_i < 0} (1 - e^{x_i}) ,

the numerator mass is exactly P and the denominator is P + N, so the attention row sum is exactly r(x) = P / (P + N) in [0, 1] (matched to the direct softpick sum to 3e-16). The boundaries are exact: r = 0 iff max_i x_i <= 0 (attend to nothing) and r = 1 iff min_i x_i >= 0 (attend to everything). Softmax always sums to exactly 1, so it must allocate a full unit of attention no matter how negative the logits; softpick's row can be empty.

The broken shift-gauge

Softmax is exactly shift-invariant, softmax(x + c 1) = softmax(x) (measured to 6e-15 out to c = 1000): the absolute logit level is a gauge it discards, and that is precisely why the row is pinned to sum one. Softpick's rectification threshold e^{x} = 1 sits at the absolute value x = 0, so softpick(x + c 1) is not softpick(x): as c sweeps from -inf to +inf the row sum sweeps monotonically from 0 to 1 (measured 0, 0, 0.71, 0.998, 1.0). Softpick reads the absolute logit level, exactly the degree of freedom softmax gauges away, and that is what lets it reach the empty row r = 0 softmax never can. This is the attention counterpart of the softmax cross-entropy logit gauge, broken here rather than supplied.

Why the stabilization needs the -e^{-m} term

Large logits overflow e^{x}, so softpick needs stabilizing. Softmax's fix is free, softmax(x) = softmax(x - m), but softpick is not shift-invariant, so it is not. The exact reason is a multiplicative identity,

e^{x_i} - 1 = e^{m} ( e^{x_i - m} - e^{-m} ) .

The common e^{m} cancels in the numerator-over-denominator ratio, leaving softpick invariant to that rescale, but only if the reference is carried as -e^{-m}, not replaced by -1. The naive softmax-style subtract-max, which drops back to -1, computes a different function: on +800 logits it collapses the whole row to 0, while the -e^{-m} form stays finite and correct. That is why the deployed stabilized softpick carries the -e^{-m} term.

Layout

• softpick.py: the softpick normalizer, the P / (P + N) row-sum law, and the exact zero/one boundaries.
• gauge.py: softmax's exact shift-invariance versus softpick's monotone 0 -> 1 row-sum sweep, and the empty row softmax cannot reach.
• stabilize.py: the -e^{-m} stabilization identity and the naive subtract-max collapse.
• test_shiftgauge.py: the row-sum law, the boundaries, the softmax gauge, the softpick sweep, the unreachable empty row, and the stabilization.

Reproduce

python softpick.py
python gauge.py
python stabilize.py
python test_shiftgauge.py

Note

The Softpick paper gives the stabilized formula and shows empirically that softpick removes attention sinks and massive activations. What is here is the exact structure behind it: the r = P / (P + N) row-sum law with the exact max <= 0 and min >= 0 boundaries, the reading of softpick as the map that breaks softmax's shift-gauge and therefore can output the empty row that forced allocation forbids, and the proof that the -e^{-m} term is the unique correction that survives stabilization because the gauge is multiplicative in e^{m} rather than additive in m. It is the attention shift-gauge, distinct from the shipped zlossgauge (the cross-entropy unembedding gauge, supplied by z-loss rather than broken) and from the ships that show sinks exist and are load-bearing; this is the exact property of the fix. The low-precision behavior of the stabilized versus naive path around the e^{x} = 1 threshold is the named systems extension.