Sliding window doesn't bound KV memory on Gemma-3/4 — the global-attention layers grow unbounded (note + growth model)

[mlucas67]

Gemma-3/4 are described as sliding-window models, and it's natural to assume that bounds KV-cache memory. It doesn't. These are hybrid models: most layers are local (sliding), but 1 in every sliding_window_pattern layers is a full-attention "global" layer that must keep the KV of every token. In mlx-lm those global layers get a plain KVCache that grows without bound, so per-sequence KV memory grows linearly with context, forever — dominated by the global layers past ~5k tokens. This is correct behavior (global attention needs all the KV), but it surprises people who expect the window to cap memory, and it's the root of the "why does my Gemma server slowly balloon and slow down" reports. Sharing the mechanism, a growth model, and production data, and curious how others manage it.

The mechanism (it's right in make_cache)

# mlx_lm/models/gemma4_text.py:662
for i in range(...):
    if self.args.layer_types[i] == "full_attention":
        caches.append(KVCache())                                  # global → UNBOUNDED
    else:
        caches.append(RotatingKVCache(max_size=sliding_window, keep=0))  # local → bounded ring

# mlx_lm/models/gemma3_text.py:247 — same split, via modulo
if i % sliding_window_pattern == sliding_window_pattern - 1:
    caches.append(KVCache())                                      # global → UNBOUNDED
else:
    caches.append(RotatingKVCache(max_size=sliding_window))

RotatingKVCache(max_size=w) is a bounded ring. KVCache (cache.py:325) grows the buffer in step=256 blocks via mx.concatenate, advances offset, and never caps or evicts — because a global layer that dropped KV would compute wrong logits. Nothing here is a bug; the point is that "sliding-window model" hides a term that is linear in context length.

What that costs (Gemma-4-26B)

Config: 30 layers → 5 global / 25 local, sliding_window=1024, GQA num_kv_heads=8, head_dim=256, max_position_embeddings=262144.

Per-token global KV = 5 layers × 2(K,V) × 8 × 256 = 20,480 elements/token = 40 KiB/tok (fp16), 20 KiB/tok (8-bit). The 25 local layers are fixed at 200 MiB (fp16). So:

• Crossover ≈ 5,120 tokens — beyond that, the unbounded global term exceeds the entire bounded local footprint.
• ≈ 10 GiB (fp16) / 5 GiB (8-bit) of global KV per sequence at the 262k max context. 8-bit quant halves the slope, not the shape — still a line, not a plateau.

[FIGURE — drag gemma_kv_growth.png here]

Seen in production

A Gemma-4-26B deployment (served via a third-party OpenAI-compat server) exhibited exactly this:

• Resident footprint climbed to 34 GB (peak 36.6 GB per the scheduler; weights are only ~15 GB at 4-bit — the rest is KV + prefix cache).
• The paged cache then exhausted — Out of cache blocks / Cannot allocate block — which is the slowdown: requests stall once blocks can't be allocated. "Large" and "slow" are one phenomenon.
• With a larger prefix-cache budget it crept to ~65 GB over days before needing a restart; a smaller budget just traded the creep for per-turn re-prefill latency — two symptoms of the same unbounded cache.

Why it's easy to miss

1. The label — "sliding-window" reads as "bounded"; the hybrid schedule is a layer_types detail.
2. Short-context testing — below the ~5k crossover the bounded layers dominate, so it's invisible in demos and only bites in long real sessions.
3. No crash — the growth is correct, so there's no assertion to trip, just a number that climbs.

The part I'd love input on

Because the growth is correct-by-design, the fix isn't patching KVCache — it's explicit KV lifecycle management by whoever owns the generation loop: per-conversation LRU under a hard budget, context bounding, snapshot/reuse of the stable prefix, 8-bit KV. We're building that into our own mlx-lm-based server for a long-running companion app.

Questions for the community/maintainers:

• Would a short docs note (that sliding-window Gemma KV is unbounded via the global layers, with this growth model) be worth adding? It'd save people a debugging session.
• Is there interest in optional eviction/budget hooks on the cache utilities (e.g. an LRU-able cache list, or a documented pattern) so servers don't each reinvent this?
• For those running Gemma-3/4 long-context in production — how are you bounding it today? Restart cadence, context caps, custom cache management?

---

Method, full derivation, and line-level citations: happy to share the long-form write-up. Numbers verified against mlx-community/gemma-4-26b-a4b-it-4bit and mlx-lm gemma4_text.py / gemma3_text.py / cache.py.