blindband

Streaming attention has an exact blind band in the middle of the context that no depth can close, and closing it takes exactly one all-to-all layer, which is the capacity reason hybrid stacks interleave recurrence with local attention.

The StreamingLLM mask keeps S attention-sink tokens at the start of the sequence plus a causal sliding window of width w, with the full key-value cache retained (no eviction). Stacking L such layers, the input-to-output influence d out_i / d x_j has support exactly

reachable(i) = [0, S-1]  union  [i - (w-1)L, i] ,

two disjoint intervals, and it is identically zero on the blind band [S, i - (w-1)L - 1] between them.

The band is exact, and depth cannot close it

The argument is disjoint causal support. A sink at position p < S has its own causal window looking backward, so its value depends only on inputs [0, p], a subset of [0, S-1], at every layer: the sink relay is receptive-field-frozen at the sequence start and cannot forward recent information no matter how deep the stack gets. The local chain reaches back exactly w-1 positions per layer, to i - (w-1)L. Nothing bridges the middle.

Measured in double precision, the influence of the blind band on the last token is 0.00e+00 at depth 2, 4, 8, and 12; the sink region is the gradient peak; and the local cone edge marches back linearly (241, 227, 199, 171 for those depths). The band width is i - (w-1)L - S, growing linearly in context length and shrinking only linearly in depth, so for any fixed depth a long enough context has an arbitrarily wide exactly-unreachable band. This refutes the natural intuition that keeping the full cache lets information route around the window through the always-attended sinks: it provably cannot.

One all-to-all layer closes it, which is why hybrids interleave

Two kinds of layer close the band. A single global (full causal) attention layer lets every position attend to the whole prefix and makes the blind-band influence nonzero (8.1e-4 at depth 8, up from exact zero). A single recurrent (linear-SSM) layer also closes it: its influence is d h_i / d x_j = b a^{i-j} for every j <= i, strictly positive for a stable a (minimum over the band 3e-12 > 0), so the recurrence connects every past position to the present.

Griffin, Jamba, and Samba interleave a recurrence (or a global-attention layer) with local attention and justify it empirically by recall and throughput. This is the exact reason: local-plus-sink attention has a machine-zero middle, and it takes exactly one all-to-all layer, attention or recurrence, to connect it.

Layout

• reachable.py: the exact two-interval reachable support, the machine-zero blind band, and its depth-invariance.
• closure.py: one global-attention layer and one recurrent layer each closing the band, the exact hybrid-design rationale.
• test_blindband.py: the exact-zero band, the reachable support, the marching cone, and the two closures as tests.

Reproduce

python reachable.py
python closure.py
python test_blindband.py

Note

That streaming attention cannot see the middle of a long context is folklore, and StreamingLLM explains it by eviction (the middle key-value cache is discarded). What is here is the exact structural version under masking with the full cache retained: the machine-zero blind band, its exact support [0,S-1] union [i-(w-1)L,i], the depth-invariance and linear-in-length growth, and the proof that one global-attention or recurrent layer closes it, which is the exact capacity argument behind hybrid interleaving.