Skip to content

Ternary Bonsai Q2_0 support (v0.3.2 base)

Choose a tag to compare

View all tags
@Andgihat Andgihat released this 18 Jul 12:35
v0.3.2-ternary-bonsai
e502eed

BeeLlama with support for PrismML Ternary-Bonsai-27B (Q2_0, 2.125 bits/weight), so the model can use BeeLlama's TurboQuant / KVarN KV-cache compression — which the upstream PrismML fork does not provide.

Why

On a 16 GB card, Bonsai at 192K context with q8_0 KV leaves ~0.6 GB of headroom. With turbo3 KV it leaves ~5.4 GB, and 384K context fits more comfortably than 192K used to.

Verified

Tested on RTX 5060 Ti 16 GB (sm_120), CUDA 12.8, driver 610.62, Windows.

Numerically identical to the reference implementation — perplexity on the same corpus, model and settings:

build PPL
upstream PrismML (reference) 3.1125 ± 0.04339
this build 3.1125 ± 0.04339

A wrong bit layout produces plausible garbage with no error at all, so matching perplexity is the only way to rule that out.

metric upstream PrismML this build
pp512 (prefill) 956.9 ± 28.8 1079.5 ± 35.8 +13%
tg128 (generation) 44.3 ± 0.2 51.6 ± 0.1 +16%

Context limits with turbo3 KV on 16 GB:

context VRAM headroom
192000 10939 MiB 5371 MiB
262144 11867 MiB 4443 MiB
393216 13595 MiB 2715 MiB

⚠️ Known limitation — BeeLlama's own Q2_0 (type 53)

This build replaces BeeLlama's native Q2_0 rather than coexisting with it. Both formats are called "Q2_0" but are not compatible (32 vs 128 block size, planar vs sequential bit layout).

If you have a model in BeeLlama's native Q2_0 format, do not use this build yet — it will produce garbage silently, with no error. Every other quantization type is untouched.

Splitting these into two coexisting types (Q2_0 = 42 ternary, Q2_0_B = 53 native) is in progress and will land in a follow-up release.

Usage

Model file: Ternary-Bonsai-27B-Q2_0.gguf (6.66 GiB — the g128 file, not Q2_0_g64).

Bonsai is a thinking model — use the sampling values from its model card. Greedy decoding (--temp 0) can make thinking models loop and never close </think>.

Long context, 384K:

llama-server -m Ternary-Bonsai-27B-Q2_0.gguf ^
  --ctx-size 393216 --n-gpu-layers 99 --flash-attn on ^
  --cache-type-k turbo3 --cache-type-v turbo3 ^
  --temp 0.7 --top-p 0.95 --top-k 20 --jinja

Maximum speed, short context: turbo3 trades ~7% generation speed for memory (at 32K real context: q8_0 51.3 t/s vs turbo3 47.9 t/s, prefill identical). If the context fits uncompressed, don't compress:

llama-server -m Ternary-Bonsai-27B-Q2_0.gguf ^
  --ctx-size 32768 --n-gpu-layers 99 --flash-attn on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --temp 0.7 --top-p 0.95 --top-k 20 --jinja

Other KV types: turbo2/3/4, turbo2/3/4_tcq, kvarn2/3/4/5.

Not supported

DFlash / speculative decoding for Bonsai. Drafters are target-specific — each only accelerates the exact model it was trained against. The only drafter that exists for Ternary Bonsai is PrismML's own DSpark head, and Bonsai has no built-in MTP tensors (n_layer == n_layer_all == 64). Heads trained for stock Qwen3.6 will not work.

Binaries

beellama-ternary-bonsai-win-cuda12.8-sm120.zip — Windows x64, CUDA 12.8, built natively for sm_120 (RTX 50-series). Includes llama-server, llama-cli, llama-bench, llama-perplexity and the required DLLs. Other architectures: build from source (see README).

Credits

MIT, same as upstream. Builds on Anbeeld/beellama.cpp (TurboQuant / KVarN / DFlash), PrismML-Eng/llama.cpp (the ternary Q2_0 implementation) and ggml-org/llama.cpp.

See also jarkevithwlad/turboquant-prismml-cuda, which solves the same problem on a different base and handles the type conflict by remapping only for qwen35 models.

Assets 3
Loading