frozenrank

DeltaProduct's per-token transition freezes a state subspace of dimension at least d - n_h and pins its spectral radius at exactly 1 for any deltas, so its forgetting is rank-limited to n_h. At the reflection setting its reachable orthogonal transitions are exactly those with rank(g - I) <= n_h, so the minimal number of Householder factors to realize a rotation is its rank, and DeltaNet's single factor cannot rotate at all.

DeltaProduct (Siems et al. 2025) applies a product of n_h generalized Householder factors to the recurrent state each token, A = prod_i (I - beta_i k_i k_i^T), with ||k_i|| = 1 and beta_i in [0, 2]; DeltaNet is n_h = 1. Each factor H_i has eigenvalue 1 - beta_i along k_i and 1 on the orthogonal hyperplane, so det H_i = 1 - beta_i, singular values {|1 - beta_i|, 1^(d-1)}, non-expansive on beta_i in [0, 2].

Rank-limited forgetting: rho = 1 pinned, a frozen subspace

Each factor fixes the hyperplane orthogonal to its key. The n_h fixed hyperplanes intersect in a subspace of dimension at least d - n_h on which every factor, and so the product, acts as the identity. Two exact consequences, both independent of the beta_i:

• For n_h < d, A has eigenvalue exactly 1 with multiplicity at least d - n_h, so rho(A) >= 1; with ||A||_2 <= 1 this pins rho(A) = 1 exactly (measured |rho - 1| <= 3e-15 over hundreds of random-beta draws at n_h = 1, 2, 4, 7).
• rank(A - I) <= n_h, so the state is frozen along d - n_h directions (measured frozen dimension exactly d - n_h), and DeltaProduct's forgetting lives in an at-most-n_h dimensional active subspace.

Contraction to an actually forgetting memory, rho(A) < 1, is possible only at full rank n_h >= d (measured: n_h = d orthonormal keys with beta = 1 give rho = 0, active rank d). So n_h is exactly the per-token rank budget of what DeltaProduct can forget.

The rank expressivity ledger

At beta_i = 2 each factor is a reflection, so A is orthogonal (||A^T A - I|| < 1e-15) with det A = prod(1 - beta_i) = (-1)^{n_h}: an odd number of reflections is orientation-reversing, an even number can be a rotation. By Cartan-Dieudonne and Scherk the reachable orthogonal set is exactly {g in O(d) : rank(g - I) <= n_h}, and the minimal n_h to realize g is exactly rank(g - I).

A 2-plane rotation R_theta has rank(R_theta - I) = 2, so it needs n_h = 2. DeltaNet (n_h = 1) cannot represent any rotation: its Frobenius distance to a 2-plane rotation is exactly 2, independent of the angle (measured 2.0000 at theta = 0.3, 1.0, 2.5, pi). Two reflections realize any 2-plane rotation exactly (residual at machine zero), and a general SO(d) element with m active planes needs n_h = 2m, up to d.

Layout

• householder.py: the generalized Householder factor and the product transition.
• forgetting.py: the rho = 1 pin for any beta, the d - n_h frozen subspace, the rank(A - I) <= n_h active rank, and contraction only at full rank.
• expressivity.py: the beta = 2 orthogonality and (-1)^{n_h} det parity, the two- reflection exact rotation, and the exact-2 single-reflection floor.
• test_frozenrank.py: the factor eigenstructure, the pin, the frozen dimension, the full-rank contraction, the det parity, the exact rotation, and the reflection floor.

Reproduce

python householder.py
python forgetting.py
python expressivity.py
python test_frozenrank.py

Note

The underlying facts are classical linear algebra: Householder eigenstructure, the intersection of fixed hyperplanes, Cartan-Dieudonne, and Scherk's theorem that the minimal number of reflections realizing g is rank(g - I). This repository does not claim them. What is here is applying them to DeltaProduct as an exact hyperparameter ledger and stability law: n_h is the per-token forgetting rank, rho is pinned at 1 for any beta when n_h < d with a d - n_h frozen subspace, the reachable rotations obey minimal n_h = rank(g - I), and DeltaNet cannot rotate at all. The DeltaProduct paper argues state-tracking improves with n_h and analyzes hidden-state effective rank, but states neither the beta-independent rho = 1 pin nor the exact rank ledger. The single-key per-token transition is exact here; whether a trained model's learned beta, k sit at this pin, and how the sequence product accumulates active-subspace mixing, are the named extensions.