Repository navigation
v0.8.26 — window-aware expert offload, Terminal-Bench harness repair
Fixes
- Expert offload is now sized for the served window, not just the card (#149).
plan_expert_offload_gibstopped atWEIGHT_FRACTION, so a context window the card could not actually hold was planned anyway and the serve failed on KV at startup. Window and pool are now priced by onewindow_kv_bytesfunction and gated by a singlekv_headroom_bytesaffordability check, with the extra margin capped atWINDOW_OFFLOAD_MAX_EXTRA_GIB. The KV anchor was validated at a 262,144-token window and is ~4.8% conservative against the measured per-token cost. --cpu-offload-gbonglq-code/glq-chat(#146) to override the computed expert budget.- Terminal-Bench harness repaired (#148). Four defects, each of which produced a clean-looking score from a run that had not happened: the dataset is addressed
name@versionand is now pinned (terminal-bench/terminal-bench@4.0.0); a custom agent goes on-aasmodule:Class, since--agent-import-pathwas removed and is rejected before any container starts;--max-num-seqsis pinned to 8, because a hybrid-GDN architecture reserves a Mamba cache block per decode slot and vLLM's default of 1024 refuses startup; and sampling is pinned server-side, as pi cannot sendtop_kat all. - Benchmark artifacts are mirrored to S3 mid-run (#148), so a spot reclaim no longer takes the only copy of a job's output with it. Durability was verified with objects present 20 s into a run. Inert without a bucket configured, and it never passes
--delete. glq-code's pi--passthrough is discoverable, and loud when it is wrong (#147).
Validation
CI green on 3.12 / 3.13 / 3.14. The offload sizing has unit coverage for the window/pool coupling, and the decode cost of offload was measured on two cards — the per-GiB figure in %-terms is not portable across PCIe generations, while ~0.41–0.49 ms/token/GiB is; see benchmarks/flashnext_offload_slope/.
Compatibility
No quantize-path changes, so existing checkpoints are unaffected and no re-quantization is needed. Checkpoints produced by this release are byte-identical to 0.8.25's.