Repository navigation
v0.8.30 — Kolibri 1 in glq-code, and an offload accounting fix
What changed
Offload accounting: a forced --cpu-offload-gb is a budget, not an amount (#162). A
correctness fix for a bug shipped in 0.8.29, and the main reason to take this release.
--cpu-offload-gb is a ceiling — vLLM saturates at the experts a checkpoint declares. The
planners cap for that reason, but a forced value returned early before reaching them, so
both derived figures were computed against a number that cannot happen. Measured on
Flash-Next's figures (23.84 GiB PLE, 42.63 GiB experts) against a 94.27 GiB host:
forced --cpu-offload-gb |
pinned host | resident VRAM |
|---|---|---|
| auto / 30 / 40 | exact | exact |
| 60 | +17.37 GiB over | −7.19 GiB under |
| 80 | +37.37 GiB over | −7.19 GiB under |
Wrong in both directions at once. The over-count made the 0.8.29 pre-flight check refuse
configurations that run. The under-count returned a resident of 0.00 GiB where
7.19 GiB is really on the card, so kv_headroom_bytes claimed +25.31 GiB against a true
+17.91 — over-promising VRAM is how a fit check says yes to something that cannot load.
If you pass --cpu-offload-gb above your checkpoint's declared expert bytes, 0.8.29 was
refusing you for a setup that works. This fixes that.
Kolibri 1 tool calling, and a guard for the plugin it needs (#161).
glq-code --model <a Kolibri-1 GLQ build> no longer exits with "no known tool-calling
setup" — the family maps to --tool-call-parser kolibri1 --reasoning-parser kolibri1 plus
the model card's sampling. Both parser names and the model implementation come from
aleph-alpha-inference, so when that is missing glq-code now refuses up front with the
install commands in the order that works, rather than letting vLLM fail minutes into a
28.68 GiB weight load with an unknown-architecture error. The interactive picker's filter
gained the same predicate, so the menu cannot offer what the guard would refuse.
Also
README staleness that CI cannot see: the picker table linked a checkpoint that no longer
exists, the public-checkpoint count is now 35, the claim that Qwen3.8-Flash-Next is the
build declaring offloadable experts is no longer true of any published checkpoint, and the
PLE host-memory section now records that the checkpoint its numbers were measured on has
been withdrawn. Every xv0y5ncu link in the README was re-checked and resolves.
Validation
- All three PRs merged with
test (3.12),test (3.13)andtest (3.14)green. - The accounting fix adds cases pinning both directions of the bug, that a planned offload
is bit-identical to before (so no working configuration moves), and that an undeclared
expert_offload_bytesstill takes the request at face value rather than collapsing to
zero. Two failing controls guard the tempting one-line version of the fix: clamping the
attribute itself would silently kill the over-large-budget warning and break the
documented "--cpu-offload-gbis honored verbatim" contract. - The Kolibri work patches
find_specin every test, so no result depends on whether the
plugin is installed on the machine running the suite.
Numbers are from one measured configuration and scoped to it in the code comments.
Existing checkpoints
Unaffected. Neither change touches the encode or quantize path, so published checkpoints
stay byte-identical and load as before.
Known limits
- Kolibri 1 end-to-end serving is not covered by CI: it needs
aleph-alpha-inference
(which pins vLLM 0.29.x) and a 28.68 GiB download. - The accounting fix deliberately leaves the window re-plan and arena sizing on the raw
request. Moving those changes the chosen window for any forced over-large budget, which
wants its own measured change. The consequence is intentional — a plan built from an
optimistic footprint is now caught by the pre-flight check rather than silently agreed
with.
For several minutes after this release the curl … | bash one-liner may still install
0.8.29, while the PyPI simple index catches up.