Skip to content

v0.8.27 — routing-aware expert cache, planner sizing, opt-in MTP

Choose a tag to compare

@cnygaard cnygaard released this 14 Oct 22:07
· 41 commits to main since this release
9741aef

Routing-aware expert cache, on by default where it pays

Weight offload makes a large MoE fit on a small card, and then every routed expert is re-read
over PCIe. glq-chat and glq-code now keep the hottest experts in a per-layer VRAM arena
with LRU residency, so a repeat routing is a local read instead of a bus transfer.

card arena decode vs no arena
L4, 22.49 GiB, sm_89 8 GiB 17.211 tok/s 2.674× (6.436)
RTX PRO 4500 Blackwell, 32 GiB, sm_120 8.95 GiB 28.795 tok/s 2.20× (13.067)

Both B=1, Qwen3.8-Flash-Next 3 bpw, same checkpoint in both arms of each row.

On by default only when experts are offloaded — with the model resident there is no PCIe
traffic to remove, so a card that fits the model is untouched. GLQ_MOE_EXPERT_CACHE=0 opts
out.

Output is byte-identical with the cache on or off: md5 matched across the baseline, an
iso-VRAM control and two arena sizes. On the L4 an iso-VRAM control at a matched 14.79 GiB
footprint reached 7.052 tok/s, so 2.441× of the gain is routing-awareness rather than
simply more resident weight. It depends on CUDA graphs — the same configuration measured
2.674× graphed and 1.247× eager.

Also in this release

  • The planner sizes the cache, after the context window is paid for, so an arena is never
    bought with context the caller asked for. Adds window ceilings by card size, a cap on the
    share of host RAM that may be pinned, and a re-plan after deepening offload frees VRAM the
    first pass could not see.
  • --mtp N on glq-chat / glq-code for MTP speculation — opt-in, default off. It
    changes output (three speculative arms produced three distinct digests where every
    non-speculative arm agreed), and it deepens the offload budget. Measured with the cache:
    19.692 tok/s at num_spec=1 on an L4; on a 32 GiB Blackwell 3 beat 1 (39.659 vs 37.957),
    so which N wins is hardware-dependent — measure yours.
  • --model /path/to/checkpoint now plans like a repo id. The three sizing lookups were
    HTTP-only, so a local path resolved to nothing and the window, PLE offload, expert offload
    and the cache all collapsed together — 16384 tokens instead of 131072, with no message.
  • MTP draft-head quantization (--mtp-bpw) and resume-store rebuild from a published
    checkpoint.
  • README: the expert cache is documented (it had no mention despite shipping on by
    default); the PLE host-memory section now names --engram-config '{"cpu_offload": true}',
    since vLLM 0.31.0 no longer reads VLLM_PLE_CPU_OFFLOAD; the CUDA-graph capture-size
    guidance no longer states a derivation that 0.31.0 contradicts.
  • CI installs safetensors and transformers and applies the -m "not slow" filter that
    pyproject.toml's own marker definition asks for. Three suites had been skipping entirely
    rather than running, including the mixed-bpw trellis guard.

Existing checkpoints

Unaffected — no re-quantization needed. The cache is a serving-side structure and is
byte-identical. --mtp requires a checkpoint with a quantized draft head and is opt-in.

Known issues

  • Host memory with expert offload is over-committed. Serving Qwen3.8-Flash-Next with
    offload holds 102.8 GiB of host memory where the two offload components together report
    only 65.3 GiB (41.48 vLLM + 23.84 PLE) — a 37.5 GiB gap, reproduced three times, which can
    exhaust a 120 GiB host and leave the kernel to reclaim the engine process. Not introduced
    here, though the cache deepens offload from 32 to 42 GiB and so reaches that edge about
    1 GiB sooner. Under investigation; measured not to be torch's pinned allocator, not one
    upfront reservation, and not reclaimable via swap.
  • No container image for this release. ghcr.io/cnygaard/glq-env has not built since
    v0.8.6 — the Dockerfile asserted the CUDA extension while docker build, having no GPU,
    had installed the CPU stack. Fixed for 0.8.28.
  • Resume run keys include __version__, so a store built under 0.8.26 is not reused by
    0.8.27.

Validation

Tests green on 3.12 / 3.13 / 3.14. The published curl | bash one-liner was run on a fresh
Ubuntu 24.04 GPU box: glq-setup --verify reports 7/7. Every number above is scoped to the
card, model, bpw and batch size it was measured on.