causalrank

The causal mask lifts linear attention to full rank, so the "rank <= d bottleneck" several papers cite is an artifact of dropping the mask. The real effective-rank story is energy concentration, not the feature dimension.

Linear attention replaces the softmax with a feature map: the token-mixing matrix is phi(Q) phi(K)^T for a feature map phi of inner dimension d. Bidirectionally, that product has rank at most d, and a line of work cites this rank <= d << N bottleneck as a limitation of linear attention. Language models run linear attention causally, and the causal mask changes the conclusion.

Full rank, for every d

Causally the mixing matrix is A = tril(phi(Q) phi(K)^T), lower triangular, so its determinant is the product of its diagonal:

det A = prod_i phi(q_i) . phi(k_i) .

For a strictly positive feature map (elu+1, exp) each diagonal entry is a dot product of two positive vectors and is strictly positive, so det A != 0 and

rank A = N ,  for every inner dimension d >= 1 .

Measured in double precision: the bidirectional map has rank exactly d (64, 8, 1), while the causal map has rank N = 512 in every case, including d = 1. A single feature gives a full-rank N x N causal mixing matrix. The log-determinant is finite with a strictly positive diagonal. A non-strictly-positive map (relu) can make a diagonal entry vanish and drop the rank below N (measured 509 with 66 zero diagonals), which is exactly why strict positivity is the condition.

Why it still looks low-rank, and why that is not the rank-d claim

The matrix is full rank yet linear attention is empirically effectively low-rank. The reason is energy concentration, not algebra. The causal mask makes each row a running prefix sum, so the later rows accumulate large partial sums and dominate the spectrum. The stable rank ||A||_F^2 / ||A||_2^2 sits near 1.2 and does not grow with the sequence length: the energy lives in about one direction despite full algebraic rank. Meanwhile the condition number climbs fast, from 2e3 at N = 256 to 3e10 at N = 4096, so any finite-precision computation sees small singular values near its noise floor.

N	exact rank	stable rank	condition number
256	256	1.24	2.3e3
512	512	1.24	1.7e4
1024	1024	1.24	2.2e5
2048	2048	1.24	1.6e7
4096	4096	1.24	2.6e10

So the effective low-rank behaviour of causal linear attention is an energy-concentration and conditioning effect, not the rank <= d factorization bottleneck of the unmasked map. The mixing matrix has all N directions; almost all of them just carry very little energy.

Layout

• rank_lift.py: the exact theorem, causal rank N for every d (including d = 1) via the triangular positive determinant, and the relu counterpoint.
• effective.py: the stable rank near 1 and the condition-number growth, the real mechanism behind the effective low rank.
• test_causalrank.py: the bidirectional-d versus causal-N rank, the determinant, the relu break, and the energy-concentration and conditioning as tests.

Reproduce

python rank_lift.py
python effective.py
python test_causalrank.py

Note

Several 2025 and 2026 papers motivate their methods with rank(phi(Q) phi(K)^T) <= d for linear-attention token mixing. That bound is for the bidirectional map. Causally, with a strictly positive feature map, the mixing matrix is full rank N. The effective-rank behaviour these methods target is real, but its mechanism is energy concentration and conditioning, not the feature-dimension bottleneck.