maskoverflow

The way a fully masked attention row turns into a NaN, and the counterintuitive fact that the safe choice of mask value inverts between float32 and float16. Ranking mask values by how much attention they leak is vacuous; the value that actually matters is whether the masked logit overflows to negative infinity, and the most common idiom -1e9 is the wrong choice in float16.

The leak is a red herring

A masked key is suppressed by adding a large negative M to its logit. The softmax weight that still lands on it, relative to the largest allowed logit, is exactly

leaked(M) = exp(M) / ( (n-1) + exp(M) )

for n keys with the allowed ones held at the top. To keep the leak below a tolerance tol it suffices that M < ln( tol * (n-1) ): more keys make each masked share smaller, so the threshold is less negative as n grows. At n = 1024 and tol = 1e-6 that is M < -6.885, and the leak at M = -6.885 is 1e-6 to three figures.

But below the dtype's exp-underflow floor, exp(M) flushes to zero and the leak is bit-identical zero. The floors are M = -104 for float32, -93 for bfloat16, -18 for float16. Every common idiom (-1e4, -1e9, finfo.min) sits far below the float32 and bfloat16 floors, so in those dtypes they all leak the same zero. Ranking mask values by leakage is therefore meaningless in float32 and bfloat16.

The NaN, and where it is born

When a query row has every key masked, its softmax is undefined. The NaN is manufactured in the max-subtraction step, before any division:

row = [-inf, ..., -inf]  ->  row - row.max() = -inf - (-inf) = NaN

A finite row, even one at finfo.min, subtracts safely and gives a uniform 1/n (attention over the padding, silently wrong but not NaN). So the mask value is safe on a fully masked row only if logit + M stays finite.

The inversion

Whether logit + M stays finite depends on the dtype's range, and the answer inverts:

dtype	max finite	-1e9	finfo.min
float16	65504	NaN (casts to -inf)	safe at score 0, overflows once score <~ -1000
bfloat16	3.4e38	safe	safe (saturates)
float32	3.4e38	safe	safe (saturates)

-1e9, the single most common mask idiom, casts to -inf in float16 and turns a fully masked row into a NaN, while finfo.min does not. That is the opposite of the float32 mental model, where -1e9 is finite and fine. No float16 additive mask is unconditionally safe, because even finfo.min plus a modest score overflows. The only dtype-independent fix is to mask on the query side, zeroing the dead output rows after the softmax rather than choosing a magic value.

The blast radius is exactly two layers

One fully masked query row i contaminates only its own output row after one attention layer, and a position-wise feed-forward keeps it there. The next self-attention layer spreads it to the whole sequence: row i of the residual is NaN, so Q and K have a NaN row i, so Q K^T has a NaN row and a NaN column i, so every output row's softmax denominator sums over the NaN column and every row becomes NaN. The contamination depth from one masked row to a fully NaN sequence is exactly two attention layers. The position-wise feed-forward is the control: it does not mix positions, so it never spreads the NaN, which isolates attention's token mixing as the vector.

Layout

• leak.py: the exact leaked-mass law, the ln(tol*(n-1)) tolerance threshold, the per-dtype exp-underflow floors, and the proof that the leakage ranking is vacuous below them.
• overflow.py: the additive-overflow discriminator, the float16-vs-float32 inversion of -1e9, and the max-subtraction NaN.
• blast.py: the depth-two contamination law on a real two-layer attention stack, with the position-wise feed-forward as the non-spreading control.
• test_maskoverflow.py: the closed-form leak, the tolerance, the vacuous ranking, the floors, the dtype inversion, the max-trick NaN, the depth-two blast radius, and the partial-row control.

Reproduce

python leak.py
python overflow.py
python blast.py
python test_maskoverflow.py

Note

That a fully masked attention row produces a NaN, and that the fix is to mask keys rather than rows or to mask on the query side, is known and documented. The -10000 idiom that "rounds to zero in low precision" is folklore from early transformer code. What is here is the exact characterization: the closed-form leaked mass and the ln(tol*(n-1)) threshold, the per-dtype exp-underflow floors -104 / -93 / -18 that make the folk ranking of mask values by leakage vacuous in float32 and bfloat16, the counterintuitive inversion where -1e9 is the unsafe mask in float16 while finfo.min is safe, and the depth-two blast radius from a single masked row to a fully NaN sequence, each with a machine certificate. The named extension is the interaction with key-value caching, where a masked row that survives into the cache poisons every later query that attends to its position.