ropealias

Compute rotary embeddings in bf16 and a long-context model stops being able to tell nearby positions apart, by a clean resolution law of p/128.

Rotary position embedding (RoPE) rotates each query and key coordinate pair by an angle theta(p, i) = p * base^(-2i/d) that grows linearly with the token position p. For the highest-frequency pair the angle is essentially p itself. The trouble is representing that angle, or the position feeding it, in low precision.

The resolution law

A value near magnitude p is stored in bfloat16 with a spacing of

ULP_bf16(p) = 2^(floor(log2 p) - 7)  ~  p / 128 ,

because bf16 keeps 7 mantissa bits. Two positions closer together than that ULP round to the same bf16 value and get the same rotary angle. So bf16 resolves position only to about p/128, and that absolute resolution degrades linearly as the context grows: the spacing is 8 at position 1024, 128 at 16384, and 1024 at 131072. Measured, the fraction of adjacent positions that share a bf16 rotary angle is 87% at 1024, 99% at 16384, and 100% at 131072. fp16, with 10 mantissa bits, holds out longer but also collapses at long context (94% at 16384).

It reaches the attention scores

The collapse is not confined to the angle. When two positions share a bf16 angle, their rotary-embedded vectors are bit-identical and the attention score of a fixed query against a key at either position is identical. Measured at positions 1024, 16384, and 131072, the distance between the rotary vectors at p and p+1 is exactly 0 in bf16 (versus about 2.0 in fp32), and the attention-score gap between a key at p and at p+1 is exactly 0 in bf16 (versus about 0.77 in fp32). A model whose rotary angle is formed in bf16 cannot separate tokens that sit within one position-ULP of each other, and at 128k context that band is a thousand tokens wide.

The cos/sin is a precision firewall

The rule is sharper than "compute RoPE in fp32." The aliasing above is a property of the phase p * inv_freq, the domain side of the cosine and sine. The range side, the cos/sin values and the rotation they drive, lives in [-1, 1] with constant bf16 spacing, so a bf16 cos/sin cache built from an fp32 phase carries a bounded, non-growing error. Measured by the translation invariance of the attention logits (which should depend only on the relative offset), a bf16 phase gives a position std of 1.93 (the signal is destroyed), a bf16 cos/sin cache from an fp32 phase gives 0.013, and full fp32 gives 0.000. So the precise rule is two-sided: the phase must be fp32; the cos/sin cache and the rotation apply tolerate bf16.

Two exact companions. The aliasing onset is the universal position 256, independent of the base and the head dimension, because the per-position relative phase increment is exactly 1/p and bf16's relative spacing is 2^-8, so adjacent positions collapse once 1/p < 2^-8. And fp16 fails harder on the phase side than bf16: the top-frequency phase p * 1 overflows fp16's finite range at p > 65504, where cosine returns nan, a hard failure rather than a soft aliasing.

Layout

• alias_law.py: the p/128 position-resolution law and the measured aliased fraction by format and context length.
• firewall.py: the universal onset 256, the domain-versus-range cos/sin firewall, and the fp16 phase overflow.
• rotation.py: the aliasing carried through to the rotated vectors and the attention scores, bf16 versus fp32.
• test_ropealias.py: the ULP law, the aliasing, and the downstream collapse as tests.

Reproduce

python alias_law.py
python rotation.py
python test_ropealias.py

The rule

Form the rotary phase p * inv_freq in fp32; the cos/sin cache and the rotation apply can stay in bf16. The position multiply is where long context meets the mantissa, and bf16 does not have the bits to keep positions apart past a few hundred tokens. Once the phase has passed through the cos/sin nonlinearity, the values are bounded and bf16 is safe.