Skip to content

v-code01/dcgain

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dcgain

Mamba's steady-state gain is a discretization artifact of its input branch, unbounded in the reset regime, and no fixed blend removes it.

A selective SSM (Mamba) discretizes a continuous diagonal channel dh/dt = A h + B x, A < 0, with an input-dependent step Delta. The state transition uses exact zero-order hold, A_bar = exp(Delta A), but the input matrix uses the cheap Euler rule, B_bar = Delta B (Mamba-2), or an exponential-trapezoidal blend (Mamba-3). Because the two branches are discretized inconsistently and Delta is selective, the layer's steady-state (DC) gain becomes a data-controlled quantity rather than the fixed continuous value -B/A.

The exact DC-gain distortion

With z = Delta A < 0, the discrete DC gain relative to the exact gain -B/A is, in closed form:

rule DC ratio limit as z -> -inf
Euler (Mamba-2) z / (e^z - 1) |z|
exp-trapezoidal (Mamba-3, lambda=1/2) (z/2) coth(z/2) |z|/2
exact ZOH 1 1

Verified against the run-to-convergence recurrence to machine precision (Euler ratio: 1.05 at z=-0.1, 5.03 at z=-5, 20.0 at z=-20). Only the exact ZOH input rule holds the DC gain at -B/A for every step. The two deployed rules inflate it without bound: in the reset regime, the DC gain grows like |z|. Mamba-3's trapezoidal rule halves the asymptotic inflation but does not remove it.

It is reached, and no fixed blend fixes it

The artifact is a live effect, not a curiosity. Mamba's step is Delta = softplus(pre) and its eigenvalues initialize across A in [-1, -16], so z = Delta A reaches tens in magnitude when Delta and |A| are both large, exactly the content-based-reset regime selectivity is built to use. Measured, pre = 0, A = -16 gives z = -11 (DC gain inflated 11x), and pre = 4, A = -16 gives z = -64 (inflated 64x). So the DC gain is inflated by an order of magnitude precisely where the model forgets hardest.

Mamba-3's fix is a convex combination with a fixed blend lambda, whose DC ratio is -z[(1-lambda)e^z + lambda]/(1-e^z). Matching the exact ZOH gain requires lambda = lambda*(z), a function of z, not a constant: measured, lambda* runs from 0.49 at z=-0.1 to 0.05 at z=-20, each giving a ratio of exactly 1. So no learned scalar lambda holds the DC gain at -B/A across selective steps; a Delta-controlled DC distortion is intrinsic to the fixed-blend design. The DC "selectivity" these models exhibit is in part an artifact of discretizing the input branch inconsistently, not a property of the selection.

Layout

  • discretization.py: the exact Euler and trapezoidal DC-gain ratios, verified against the recurrence, and the exact-ZOH invariance.
  • reachable.py: the reset-regime reachability from softplus(pre) * A, and the lambda*(z) non-constancy that defeats any fixed blend.
  • test_dcgain.py: the closed forms, the unbounded inflation, the reachability, and the no-constant-lambda impossibility as tests.

Reproduce

python discretization.py
python reachable.py
python test_dcgain.py

Note

That Mamba approximates the input branch with Delta B, justified for small Delta A, is stated in the Mamba papers, and Mamba-3's trapezoidal rule is its own accuracy fix. What is here is the exact DC-gain law for each rule, the proof that the inflation is unbounded in the large-z reset regime the papers do not analyze, the measurement that this regime is reached in normal operation, and the impossibility of a constant blend matching exact ZOH.

About

Mamba's DC gain is a discretization artifact: state branch uses ZOH but input branch uses Euler (Mamba-2)/trapezoidal (Mamba-3), so with z=Delta*A the DC gain inflates by exactly z/(e^z-1) or (z/2)coth(z/2) vs exact ZOH, unbounded as |z| grows. z reaches -11 to -64 in normal use; no fixed Mamba-3 blend matches ZOH (lambda*(z) runs 0.49 to 0.05).

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages