Skip to content

v0.24.0 - Ternary-Bonsai-2-27B runs natively

Choose a tag to compare

@srvsngh99 srvsngh99 released this 19 Sep 21:34
· 75 commits to main since this release
13811e9

Ternary-Bonsai-2-27B runs natively now.

Prism ML's ternary requant of Qwen3.8-27B needed no new architecture. It is the same GatedDeltaNet hybrid decoder Krill already runs for Ornith, Qwythos and Qwen3.8-27B, with 402 weight matrices carrying a blockwise Hadamard rotation folded into the 2-bit quantization. Krill undoes that rotation on activations at inference, so the checkpoint loads correctly instead of silently wrong.

krill pull bonsai-2-27b
krill run bonsai-2-27b

27B-class reasoning at 2-bit, 8.6 GB on disk. Loads in about 1.2s and decodes at roughly 12.8 tok/s on an M4 Pro.

Text-only for now: the pack ships a vision tower, but nothing in Krill drives it yet.

Verification

The port was bisected against a reference implementation rather than checked by eye. The Hadamard transform and the inverse-transformed embedding are bit-exact, the GatedDeltaNet layers agree to about 1e-6, and final logits match to 3.34e-6 with the same argmax. Cached decode is identical to cacheless. tools/verify_prism_hadamard_parity.py regenerates the fixture if you want to check it yourself.

Note if you are loading this pack elsewhere

PACK-RUNTIME.md in the pack points at runtime/artifact.py, which refuses it: that loader requires schema_version == 1 while the pack ships schema_version: 2 with the tensor namespace moved to mlx-vlm's language_model. prefix. The pack's other bundled loader, runtime/vision_artifact.py, does open it.

Also worth knowing: the chat template defaults to thinking at xhigh reasoning effort, so a small explicit --max-tokens gets spent entirely inside the hidden reasoning block and returns an empty reply. Leave the budget on auto.

Full notes in CHANGELOG.md.