The first token in Qwen2.5 is an attention sink with a massive-magnitude residual channel, and the KV cache is low-rank. A tempting fusion of the two says the sink token's KEY is a privileged, basis-spanning position of the cache, so a few anchor positions span the whole span of key vectors and anchor-token KV compression is geometrically justified. Measured with statistical leverage and column-pivoted subset selection, that is not what happens. The rank of the key/value cache is spread across positions no more concentrated than a random cache of the same rank, and the sink token is not a privileged basis position (with one size-dependent exception). Attention mass and basis leverage are different things.
For a captured key or value matrix A = [positions x head_dim], the object of interest is the span of the
position vectors (the rows): can a few positions' key vectors reconstruct all of them. The statistical
leverage of a position is its squared norm in the top-r left singular subspace (r = 99% energy rank), which
sums to r, so uniform leverage is r/T. Two code-disjoint selectors: SVD leverage, and scipy
QR-with-column-pivoting on the transpose (position pivot order). The reconstruction residual of the full
cache from a position subset is recomputed in fp64. Baselines: random equal-size position subsets, and the
leverage of a random matrix of the SAME rank and shape (what a delocalized cache looks like; this is the
correct null, not a full-rank Gaussian). Qwen2.5-0.5B and 1.5B, post-RoPE keys and un-rotated values, four
prompts, early/mid/late layers, both KV heads, fp64.
| model / cache | leverage max/mean | matched-rank null | sink in top-5 | QR vs random residual | selector agreement |
|---|---|---|---|---|---|
| 0.5B K | 1.46 | 1.40 | 0.62 | 0.128 vs 0.175 | 0.86 |
| 0.5B V | 1.12 | 1.28 | 0.08 | 0.098 vs 0.231 | 0.98 |
| 1.5B K | 1.20 | 1.32 | 0.00 | 0.120 vs 0.186 | 0.95 |
| 1.5B V | 1.13 | 1.27 | 0.00 | 0.095 vs 0.249 | 0.98 |
Rank is delocalized. The leverage concentration of the real cache is at (0.5B keys, 1.46 vs 1.40) or below (all other cells) a random cache of the same rank and shape. There is no excess concentration of the basis in a few positions; if anything the values are slightly more spread out than random. A leverage metric that can detect a planted privileged position (a synthetic dominant row registers max/mean above 20) reports near-random spread here, so this is a property of the cache, not a limitation of the metric.
The sink is not privileged, with one exception. For the 1.5B keys and for the values of both models the sink token's leverage is near zero (leverage/uniform 0.04 to 0.41) and it is essentially never among the top-5 leverage positions (0 to 8%). The one exception is the 0.5B keys, where the sink lands in the top-5 in 62% of cases and sits near the uniform level: for that specific cache the sink carries moderate leverage. So the claim is that the sink is not a reliably privileged basis position, and it is clearly unprivileged in three of the four cache-and-size cells; it is size- and cache-dependent, not universal.
Weak positional structure exists, but no anchor set spans the cache. The two selectors agree on 86 to 98% of their top-r positions, so there is a stable, detectable set of higher-leverage positions, and a QR-pivoted subset reconstructs better than a random subset of the same size (residual 0.10 to 0.13 versus 0.18 to 0.25, CI-separated in every cell). But at the intrinsic 99% rank the best subset still leaves about 10% residual, which is largely the 99%-energy threshold restated (1% discarded energy is roughly 10% Frobenius), not evidence of a spanning anchor set. There is real but weak positional structure, and it is not the sink.
| prediction | outcome | |
|---|---|---|
| P1 | leverage highly non-uniform (>>2), sink a top-few column | falsified. leverage is at or below a matched-rank random cache; sink unprivileged except 0.5B keys |
| P2 | a small subset reconstructs to < 5% residual | falsified. best subset leaves ~10% at the intrinsic rank (largely the energy-threshold definition) |
| P3 | V more delocalized than K | holds for 0.5B (V 1.12 vs K 1.46); for 1.5B the two are close (1.13 vs 1.20), so only partial |
| P4 | interior positions near the null | holds. concentration is at or below the matched-rank null everywhere |
| falsifier 1 | leverage near random, sink not privileged -> delocalized | fires. this is the outcome |
| falsifier 2 | the two selectors disagree | does not fire. they agree 86 to 98% |
| falsifier 3 | leverage confers no reconstruction advantage | does not fire. QR beats random, CI-separated |
Do not justify anchor-token or sink-token KV compression on the geometry of the cache. KV rank is spread across positions about as much as random, and the sink, dominant as it is in attention mass and residual magnitude (massiveact) and actively torn down at the output (residwrite), does not span the key basis in general. Magnitude and basis are different questions. A functionally-weighted objective (attention-mass or output-fidelity, as in attnoracle) can still rank positions usefully; this is only about the linear span.
Qwen2.5-Instruct 0.5B/1.5B, post-RoPE keys and un-rotated values, one forward pass per prompt, ~512-token prompts, fp32 capture, fp64 analysis, CPU. n = 24 correlated samples per cell (4 prompts x 3 layers x 2 heads), so per-cell confidence intervals are wide and the point estimates are read as directional. The matched-rank random null is the reference; no attention weights and no output are used.
python run.py cpu # leverage, QR-subset selection, reconstruction residual, matched-rank null -> results/main.jsonMIT.