foxwindow

The Forgetting Transformer's effective attention window is the first-passage time of a random walk driven by its forget gate. This makes the window a data-dependent distribution with an exact mean and an exact variance sigma^2 L / kappa^3, and it places ALiBi as the zero-variance limit.

The Forgetting Transformer (FoX, Lin et al. 2025) biases the causal attention logit by the log of the cumulative forget gate, score_ij = q_i . k_j + sum_{t=j+1..i} log g_t, with a data-dependent gate g_t in (0, 1). The attention weight is the content term times a decay D_ij = prod_{t=j+1..i} g_t.

The window is a first-passage time

Every log g_t < 0, so D_ij is strictly monotone decreasing in the distance d = i - j, for FoX exactly as for ALiBi. The gate does not by itself make attention non-monotone in distance; only the content q . k can, which ALiBi has too. What the gate sets is the distance at which the decay drops attention below a threshold. With the positive increment X_t = -log g_t, the epsilon-effective window (the largest distance with D >= epsilon) is exactly the first-passage index of the walk over L = ln(1/epsilon):

N(epsilon) = inf{ n : sum_{t=1..n} X_t > L } .

An exact two-moment law

With forget rate kappa = E[X] and increment variance sigma^2 = Var(X), renewal theory gives the exact moments (the mean carrying the overshoot correction):

E[N] = L / kappa + E[X^2] / (2 kappa^2) ,   E[X^2] = sigma^2 + kappa^2 ,
Var[N] = sigma^2 L / kappa^3 .

Measured against 200k-trial first-passage simulations across three gate distributions, the mean law is exact to 2e-3 (49.82 versus 49.85, 17.79 versus 17.81, 136.88 versus 136.91) and the variance law is within about one percent (11.37 versus 11.41, 11.10 versus 11.08, 12.09 versus 12.18). The leading L / kappa is the expected window; the sigma^2 L / kappa^3 variance is what data-dependence buys over a fixed decay, a spread of width sqrt(sigma^2 L / kappa^3) that fluctuates with the input-conditioned gate.

ALiBi is the zero-variance limit

ALiBi is the constant gate g_t = e^{-m}, so X_t = m is deterministic, sigma^2 = 0, and the law collapses: Var[N] = 0 and E[N] = L / m (138.2 at m = 0.05, 69.1 at m = 0.1), a fixed window with no spread. ALiBi is exactly the zero-variance limit of the forget-gate window law, and the data-dependence of FoX is precisely the nonzero variance term.

Layout

• forget.py: the monotone gate decay, the increment X = -log g, and the effective window as a first-passage simulation.
• renewal.py: the exact E[N] and Var[N] = sigma^2 L / kappa^3 renewal laws and the ALiBi zero-variance limit.
• test_foxwindow.py: the monotone decay, the mean and variance laws versus simulation, the linear-in-L variance, and the ALiBi limit.

Reproduce

python forget.py
python renewal.py
python test_foxwindow.py

Note

First-passage and renewal theory for a positive-increment random walk are classical (Feller); this repository does not claim them. What is here is the reading of FoX's effective window through them: the effective context length is a first-passage time, its variance is exactly sigma^2 L / kappa^3, and ALiBi is the sigma^2 = 0 limit, so the data-dependence of the forget gate shows up as window variance rather than as any non-monotonicity in distance. The FoX paper derives the mechanism and shows length generalization benefits but states no effective-window distribution or variance scaling. The moments assume increments that are independent across steps; real gates are input-correlated, so the autocorrelation-adjusted variance sigma^2 L / kappa^3 (1 + 2 sum rho_k) is the named extension, and the falsifier is a trained FoX head whose per-row window variance does not scale as sigma^2 L / kappa^3.