secularmode

RWKV-7's generalized delta rule has a diagonal-plus-rank-1 state transition whose spectrum is exactly a secular equation. Retention is safe for any removal strength because the eigenvalues interlace below the decay values; the only way to lose stability is through the bottom eigenvalue, at the exact boundary sum a_i kappa_i^2 / (w_i + 1) = 1; and adding decay makes that instability easier, not harder.

RWKV-7 (Peng et al. 2025) updates its recurrent state by a generalized delta rule with transition A = diag(w) - kappa (a (.) kappa)^T, a per-channel decay w in (0,1)^d minus a rank-1 removal along a unit key kappa scaled by a channel-wise in-context learning rate a.

An exact secular spectrum

The matrix is not symmetric, but only the products r_i = a_i kappa_i^2 enter the characteristic polynomial:

det(A - lambda I) = prod_i (w_i - lambda) * ( 1 - sum_i r_i / (w_i - lambda) ),

so the eigenvalues are the roots of the secular equation f(lambda) = 1 - sum_i r_i / (w_i - lambda) = 0. With a >= 0 the weights are nonnegative, so despite the asymmetry (||A - A^T|| = 0.52 in the example) the spectrum is real and Cauchy-interlaces the decay poles, lambda_1 <= w_1 <= ... <= lambda_d <= w_d (measured: real to 0, interlacing with no violations over hundreds of trials). The top eigenvalue is pinned lambda_d <= max(w) < 1, so the retentive modes can never expand: the removal is a downdate and only lowers eigenvalues. At w = 1 the transition is DeltaNet's symmetric I - beta kappa kappa^T and the moved eigenvalue is 1 - beta; at beta = 2 the minimum eigenvalue is -1, recovering that model's non-expansiveness edge.

Retention is safe; instability is only from below

Since the top is pinned by the decay, the only eigenvalue that can leave the unit disk is the single one below the smallest pole, and it leaves at lambda_1 = -1. Because f(-1) = 1 - sum_i a_i kappa_i^2 / (w_i + 1) is an identity, -1 is an eigenvalue exactly when

sum_i a_i kappa_i^2 / (w_i + 1) = 1 .

Below it the spectral radius is max(w) < 1 and the memory is stable; above it a sign-flipping anti-retention mode appears, an oscillatory (-1)^t channel driven by the removal, not decay. Measured: at the boundary the minimum eigenvalue is -1.000000, just below it -0.982, just above -1.018; the retention top holds <= max(w) even at removal strength 100x.

Decay makes the instability easier

Smaller w_i shrinks w_i + 1, so each budget term a_i kappa_i^2 / (w_i + 1) grows: adding diagonal decay lowers the removal budget needed to destabilize. Measured, at fixed removal the budget rises 0.244, 0.288, 0.326, 0.376 as w falls 1.0, 0.7, 0.5, 0.3, and the minimum eigenvalue drops toward -1. Retention on the positive modes and stability against the negative mode pull in opposite directions.

Retention-stable can still be step-expansive

The asymptotic bound rho(A) <= 1 is not the single-step non-expansiveness sigma_max(A) <= 1. For a non-symmetric A the two differ, and the gap is opened exactly by a non-constant a: for constant a the transition is symmetric and rho = sigma_max, but with a non-constant a there are transitions that are asymptotically retentive yet expand the state in a single step (witnessed rho = 0.883 <= 1 < sigma_max = 1.003).

Layout

• secular.py: the transition, the secular equation, the real interlacing spectrum, and the DeltaNet w = 1 limit.
• stability.py: the retention-safe top, the sum a kappa^2 / (w + 1) = 1 over-removal boundary, the decay-worsens-instability direction, and the rho versus sigma_max gap.
• test_secularmode.py: the real spectrum, the interlacing, the retention safety, the boundary, the DeltaNet limit, the decay direction, and the step-expansive gap.

Reproduce

python secular.py
python stability.py
python test_secularmode.py

Note

The secular equation for a rank-1 update of a diagonal matrix, its real interlacing spectrum, and the f(-1) identity are classical linear algebra (Golub 1973, Bunch-Nielsen-Sorensen); this repository does not claim them. What is here is the reading of RWKV-7's transition through it: the exact over-removal boundary sum a_i kappa_i^2 / (w_i + 1) = 1, the asymmetric stability where retention is safe by the decay and instability enters only through the bottom eigenvalue, the counterintuitive decay-worsens-instability direction, and the rho-versus-sigma_max step-expansiveness gap opened by a non-constant a. The RWKV-7 paper only notes qualitatively that eigenvalues can be negative and that training can be unstable; the exact boundary and its structure are the contribution. It reduces to the shipped deltafade non-expansiveness beta in [0, 2] at w = 1. The single-key transition is exact here; the multi-key composition over a sequence is the named extension.