Repository navigation
v0.8.27 — routing-aware expert cache, planner sizing, opt-in MTP
Routing-aware expert cache, on by default where it pays
Weight offload makes a large MoE fit on a small card, and then every routed expert is re-read
over PCIe. glq-chat and glq-code now keep the hottest experts in a per-layer VRAM arena
with LRU residency, so a repeat routing is a local read instead of a bus transfer.
| card | arena | decode | vs no arena |
|---|---|---|---|
| L4, 22.49 GiB, sm_89 | 8 GiB | 17.211 tok/s | 2.674× (6.436) |
| RTX PRO 4500 Blackwell, 32 GiB, sm_120 | 8.95 GiB | 28.795 tok/s | 2.20× (13.067) |
Both B=1, Qwen3.8-Flash-Next 3 bpw, same checkpoint in both arms of each row.
On by default only when experts are offloaded — with the model resident there is no PCIe
traffic to remove, so a card that fits the model is untouched. GLQ_MOE_EXPERT_CACHE=0 opts
out.
Output is byte-identical with the cache on or off: md5 matched across the baseline, an
iso-VRAM control and two arena sizes. On the L4 an iso-VRAM control at a matched 14.79 GiB
footprint reached 7.052 tok/s, so 2.441× of the gain is routing-awareness rather than
simply more resident weight. It depends on CUDA graphs — the same configuration measured
2.674× graphed and 1.247× eager.
Also in this release
- The planner sizes the cache, after the context window is paid for, so an arena is never
bought with context the caller asked for. Adds window ceilings by card size, a cap on the
share of host RAM that may be pinned, and a re-plan after deepening offload frees VRAM the
first pass could not see. --mtp Nonglq-chat/glq-codefor MTP speculation — opt-in, default off. It
changes output (three speculative arms produced three distinct digests where every
non-speculative arm agreed), and it deepens the offload budget. Measured with the cache:
19.692 tok/s atnum_spec=1on an L4; on a 32 GiB Blackwell 3 beat 1 (39.659 vs 37.957),
so which N wins is hardware-dependent — measure yours.--model /path/to/checkpointnow plans like a repo id. The three sizing lookups were
HTTP-only, so a local path resolved to nothing and the window, PLE offload, expert offload
and the cache all collapsed together — 16384 tokens instead of 131072, with no message.- MTP draft-head quantization (
--mtp-bpw) and resume-store rebuild from a published
checkpoint. - README: the expert cache is documented (it had no mention despite shipping on by
default); the PLE host-memory section now names--engram-config '{"cpu_offload": true}',
since vLLM 0.31.0 no longer readsVLLM_PLE_CPU_OFFLOAD; the CUDA-graph capture-size
guidance no longer states a derivation that 0.31.0 contradicts. - CI installs
safetensorsandtransformersand applies the-m "not slow"filter that
pyproject.toml's own marker definition asks for. Three suites had been skipping entirely
rather than running, including the mixed-bpw trellis guard.
Existing checkpoints
Unaffected — no re-quantization needed. The cache is a serving-side structure and is
byte-identical. --mtp requires a checkpoint with a quantized draft head and is opt-in.
Known issues
- Host memory with expert offload is over-committed. Serving
Qwen3.8-Flash-Nextwith
offload holds 102.8 GiB of host memory where the two offload components together report
only 65.3 GiB (41.48 vLLM + 23.84 PLE) — a 37.5 GiB gap, reproduced three times, which can
exhaust a 120 GiB host and leave the kernel to reclaim the engine process. Not introduced
here, though the cache deepens offload from 32 to 42 GiB and so reaches that edge about
1 GiB sooner. Under investigation; measured not to be torch's pinned allocator, not one
upfront reservation, and not reclaimable via swap. - No container image for this release.
ghcr.io/cnygaard/glq-envhas not built since
v0.8.6 — the Dockerfile asserted the CUDA extension whiledocker build, having no GPU,
had installed the CPU stack. Fixed for 0.8.28. - Resume run keys include
__version__, so a store built under 0.8.26 is not reused by
0.8.27.
Validation
Tests green on 3.12 / 3.13 / 3.14. The published curl | bash one-liner was run on a fresh
Ubuntu 24.04 GPU box: glq-setup --verify reports 7/7. Every number above is scoped to the
card, model, bpw and batch size it was measured on.