Skip to content

v0.12.7 — one reply reserve: context budgets now measure the window your server actually serves

Choose a tag to compare

@ahwurm ahwurm released this 31 Aug 17:48
· 231 commits to main since this release

Fix release for the context-budget spine, driven by field report #145 (thanks @mjdufresne).

What was wrong

  • init wrote served_window − 4,096 into new configs while the runtime subtracted another 4,096 internally — every init-written config ran two reserves short of the window it named.
  • Compaction's 0.80/0.95 triggers measured the raw window while the emergency floor measured window − 4,096. On every window under ~82K (that is every llama.cpp/Ollama/LM Studio default) the last-resort hard truncation could become eligible before the 0.95 compaction stage — the designed rescue never got its turn.
  • Fixing the double reserve exposed a latent gap the over-reservation had been masking: requested output tokens were never fitted to the window, so small-window servers could reject requests mid-session (vLLM validates prompt + max_tokens ≤ max_model_len and 400s).

What changed

  • max_context_tokens now means the full served window. The reply reserve is subtracted internally in one shared function (response_reserve), and every stage — eviction, 0.80, 0.95, the emergency floor — measures the same effective limit. The ordering inversion is structurally impossible now, at every window size.
  • Output tokens are clamped to that same reserve wherever the window is known (session start, /model swap, bench num_ctx pins): input + output fit the window by construction.
  • A served window too small to hold any reply (≤ 1,024 tokens) is refused loudly at startup instead of starting and failing on the first turn; init declines to adopt one and says why. Practical minimum: 1,025 tokens.
  • Repeated emergency-floor fires within one turn escalate to a distinct log line with the fire count, the window, the overshoot, and what to change — instead of N identical ERROR lines.
  • Context percentages (the REPL ctx reading, heartbeats, compaction events) now read against the usable budget — a few points higher for the same history.

Honest scope

This fixes the sizing/ordering half of #145. The deeper re-fire mechanism — per-turn compaction output is not persisted, so a long tool-heavy turn can grow back toward the floor — is an architecture item and stays open on #145.

Known wart, documented rather than hidden: the reserve curve steps at ~12K, so windows in 12,288–14,847 get slightly less usable budget than 12,287 does. Safe on both sides (more conservative, never an error); smoothing it changes real 16K-window behavior, so it is a deliberate follow-up, not part of this fix.

Migration

Nothing breaks. A config an older init wrote (served − 4,096) still works — it is just conservative now. To reclaim those tokens, set context.max_context_tokens to your full served window, or re-run localharness init.