Skip to content

v0.8.30 — Kolibri 1 in glq-code, and an offload accounting fix

Choose a tag to compare

@cnygaard cnygaard released this 24 Oct 14:57
· 29 commits to main since this release
51abf11

What changed

Offload accounting: a forced --cpu-offload-gb is a budget, not an amount (#162). A
correctness fix for a bug shipped in 0.8.29, and the main reason to take this release.

--cpu-offload-gb is a ceiling — vLLM saturates at the experts a checkpoint declares. The
planners cap for that reason, but a forced value returned early before reaching them, so
both derived figures were computed against a number that cannot happen. Measured on
Flash-Next's figures (23.84 GiB PLE, 42.63 GiB experts) against a 94.27 GiB host:

forced --cpu-offload-gb pinned host resident VRAM
auto / 30 / 40 exact exact
60 +17.37 GiB over −7.19 GiB under
80 +37.37 GiB over −7.19 GiB under

Wrong in both directions at once. The over-count made the 0.8.29 pre-flight check refuse
configurations that run
. The under-count returned a resident of 0.00 GiB where
7.19 GiB is really on the card, so kv_headroom_bytes claimed +25.31 GiB against a true
+17.91 — over-promising VRAM is how a fit check says yes to something that cannot load.

If you pass --cpu-offload-gb above your checkpoint's declared expert bytes, 0.8.29 was
refusing you for a setup that works. This fixes that.

Kolibri 1 tool calling, and a guard for the plugin it needs (#161).
glq-code --model <a Kolibri-1 GLQ build> no longer exits with "no known tool-calling
setup" — the family maps to --tool-call-parser kolibri1 --reasoning-parser kolibri1 plus
the model card's sampling. Both parser names and the model implementation come from
aleph-alpha-inference, so when that is missing glq-code now refuses up front with the
install commands in the order that works, rather than letting vLLM fail minutes into a
28.68 GiB weight load with an unknown-architecture error. The interactive picker's filter
gained the same predicate, so the menu cannot offer what the guard would refuse.

Also

README staleness that CI cannot see: the picker table linked a checkpoint that no longer
exists, the public-checkpoint count is now 35, the claim that Qwen3.8-Flash-Next is the
build declaring offloadable experts is no longer true of any published checkpoint, and the
PLE host-memory section now records that the checkpoint its numbers were measured on has
been withdrawn. Every xv0y5ncu link in the README was re-checked and resolves.

Validation

  • All three PRs merged with test (3.12), test (3.13) and test (3.14) green.
  • The accounting fix adds cases pinning both directions of the bug, that a planned offload
    is bit-identical to before (so no working configuration moves), and that an undeclared
    expert_offload_bytes still takes the request at face value rather than collapsing to
    zero. Two failing controls guard the tempting one-line version of the fix: clamping the
    attribute itself would silently kill the over-large-budget warning and break the
    documented "--cpu-offload-gb is honored verbatim" contract.
  • The Kolibri work patches find_spec in every test, so no result depends on whether the
    plugin is installed on the machine running the suite.

Numbers are from one measured configuration and scoped to it in the code comments.

Existing checkpoints

Unaffected. Neither change touches the encode or quantize path, so published checkpoints
stay byte-identical and load as before.

Known limits

  • Kolibri 1 end-to-end serving is not covered by CI: it needs aleph-alpha-inference
    (which pins vLLM 0.29.x) and a 28.68 GiB download.
  • The accounting fix deliberately leaves the window re-plan and arena sizing on the raw
    request. Moving those changes the chosen window for any forced over-large budget, which
    wants its own measured change. The consequence is intentional — a plan built from an
    optimistic footprint is now caught by the pre-flight check rather than silently agreed
    with.

For several minutes after this release the curl … | bash one-liner may still install
0.8.29, while the PyPI simple index catches up.