Skip to content

v0.8.26 — window-aware expert offload, Terminal-Bench harness repair

Choose a tag to compare

@cnygaard cnygaard released this 11 Oct 07:43
· 41 commits to main since this release
5eea5e2

Fixes

  • Expert offload is now sized for the served window, not just the card (#149). plan_expert_offload_gib stopped at WEIGHT_FRACTION, so a context window the card could not actually hold was planned anyway and the serve failed on KV at startup. Window and pool are now priced by one window_kv_bytes function and gated by a single kv_headroom_bytes affordability check, with the extra margin capped at WINDOW_OFFLOAD_MAX_EXTRA_GIB. The KV anchor was validated at a 262,144-token window and is ~4.8% conservative against the measured per-token cost.
  • --cpu-offload-gb on glq-code / glq-chat (#146) to override the computed expert budget.
  • Terminal-Bench harness repaired (#148). Four defects, each of which produced a clean-looking score from a run that had not happened: the dataset is addressed name@version and is now pinned (terminal-bench/terminal-bench@4.0.0); a custom agent goes on -a as module:Class, since --agent-import-path was removed and is rejected before any container starts; --max-num-seqs is pinned to 8, because a hybrid-GDN architecture reserves a Mamba cache block per decode slot and vLLM's default of 1024 refuses startup; and sampling is pinned server-side, as pi cannot send top_k at all.
  • Benchmark artifacts are mirrored to S3 mid-run (#148), so a spot reclaim no longer takes the only copy of a job's output with it. Durability was verified with objects present 20 s into a run. Inert without a bucket configured, and it never passes --delete.
  • glq-code's pi -- passthrough is discoverable, and loud when it is wrong (#147).

Validation

CI green on 3.12 / 3.13 / 3.14. The offload sizing has unit coverage for the window/pool coupling, and the decode cost of offload was measured on two cards — the per-GiB figure in %-terms is not portable across PCIe generations, while ~0.41–0.49 ms/token/GiB is; see benchmarks/flashnext_offload_slope/.

Compatibility

No quantize-path changes, so existing checkpoints are unaffected and no re-quantization is needed. Checkpoints produced by this release are byte-identical to 0.8.25's.