ropeabsorb

MLA's weight absorption makes the KV cache exactly rank d_c, and RoPE cannot join it: the position-operator family spans exactly d_R dimensions, so no single matrix folds it, partial absorption is impossible, and the decoupled RoPE dimension must be exactly d_R.

Multi-head Latent Attention (DeepSeek-V2, 2405.04434) compresses the key-value pair into a latent c_j = W_DKV h_j of dimension d_c and up-projects K_j = W_UK c_j.

Absorption is exact, and the cache is exactly rank d_c

Without position encoding the score is s_ij = (W_Q h_i) . (W_UK c_j) = h_i^T (W_Q^T W_UK) c_j, so folding A = W_Q^T W_UK lets a query attend directly against the cached latent, never materializing K. The identity is exact in exact arithmetic (agreeing to 2e-13 in fp64), though the materialized and absorbed evaluation orders are different reductions and so not bit-for-bit identical, a real prefill-versus-decode reproducibility caveat. Because K = C W_UK^T factors through the latent, the materialized key matrix is exactly rank d_c (measured, with a 4e-16 cliff): the absorbed cache loses nothing.

RoPE spans exactly d_R dimensions and cannot be folded

Routing the RoPE key through the latent gives h_i^T W_QR^T R_delta W_UKR c_j, and absorbing it into a single M requires the family F_delta = W_QR^T R_delta W_UKR to be constant in delta. Each 2x2 RoPE block contributes a cos(theta_l delta) and a sin(theta_l delta) component, so

F_delta = sum_{l=1..L} cos(theta_l delta) M_l^c + sin(theta_l delta) M_l^s ,   L = d_R/2 ,

and for L distinct nonzero frequencies the span of {F_delta} over all shifts has dimension exactly 2L = d_R (measured rank = 16 = d_R, with a 6e-16 spectral cliff). A single absorbing matrix exists if and only if that span is one-dimensional, which happens only when every frequency is zero, that is only without RoPE (measured no-RoPE control: rank 1, absorbable). The obstruction is algebraic, not numerical.

Partial absorption is impossible

The best single matrix is the DC component of F_delta, and its residual is the non-DC oscillatory energy, strictly positive whenever any frequency is nonzero. So you cannot fold even one nonzero-frequency block: a single-frequency family already spans two dimensions (measured). The residual grows toward 1 as the context window grows (0.69, 0.84, 0.94 over shifts up to 128, 1024, 4096): the RoPE part is asymptotically all oscillation, so the decoupled dimension DeepSeek carries outside the latent must be exactly d_R and cannot be compressed below without breaking exactness.

Layout

• absorb.py: the absorption identity, its non-bit-exactness, and the exact-rank-d_c cache.
• obstruction.py: the RoPE position-operator family, its exact d_R-dimensional span, the no-RoPE control, and the best-single-matrix residual growing with context.
• test_ropeabsorb.py: the absorption identity, the cache rank, the d_R span, the no-RoPE absorbability, the impossibility of partial absorption, and the single-frequency case.

Reproduce

python absorb.py
python obstruction.py
python test_ropeabsorb.py

Note

The DeepSeek-V2 paper justifies its decoupled RoPE only informally, that W_UK cannot be absorbed because matrix multiplication does not commute. What is here turns that into an exact statement: the position-operator family spans exactly 2L = d_R dimensions with a measured spectral cliff, a single absorbing matrix exists only without RoPE, partial absorption is impossible because the best-matrix residual is the strictly positive non-DC energy, and the decoupled dimension is therefore exactly d_R. The underlying algebra, the independence of the cosine and sine components across distinct frequencies, is classical harmonic analysis; the contribution is the exact dimension count and impossibility for MLA's absorption. It is distinct from the shipped ropelowrank (the rank of the keys, not of the position-operator family) and ropealias (a bf16 precision effect, not an fp64-exact count). The per-head-independent count and a closed-form for the absorbable fraction versus window length are the named extensions.