xyz: Ternary Bonsai 2 27B at 160k context on one 16 GB GPU, about 140 tokens/s, lossless speculative decoding

[xyznq1]

We forked llama.cpp to get PrismML's Ternary Bonsai 2 27B (PTQ1_0) running fast at long context on one 16 GB card.

On an RTX 4070 Ti SUPER (16 GB, stock power limit) at around 161k context, temperature 1.0, top-k 20, top-p 0.95, over 10 generations (11,964 tokens) we measured 139.5 tokens/s and 2.82 tokens per round, and 144.2 tokens/s with our optional CUDA runtime (XYZ_ENGINE=1), same text.

What we added:

• CUDA kernels for Bonsai's PTQ1_0 / PQ2_0 weights
• xyzkv2, a 2-bit rotated KV cache with MMA flash attention, that's what gets 160k into 16 GB
• our drafter, xyz v1.2, a one-layer draft head on the model's hidden states
• coupled sampling and block verification, every token is still an exact sample from the model, so the text is the same as without drafting
• an SSD tier under the prompt cache, and xyz-engine, an optional CUDA runtime for the speculative rounds (Windows)

On Windows it's one zip, unzip it and run scripts\xyz-serve.cmd, the model downloads on the first start. On Linux you build it with CMake and run scripts/xyz-serve.sh, we've only tested Windows so far.

Repo: https://github.com/xyznq1/xyz-llama
Drafter: https://github.com/xyznq1/xyz-drafter

It's built on llama.cpp, thanks to everyone here. The KV cache follows TurboQuant, the draft head follows EAGLE-3 and the verification follows the block verification paper, the credits are in the README.

[159753a52]

Nice work on the exact-sample verification. One note on the memory side, since the post says the 2-bit cache is what gets 160k into 16 GB: Bonsai 2's base is Qwen3.8 27B, where only 16 of the 64 layers keep a KV cache, 65,536 bytes per token at f16. At about 161K that is 9.8 GiB at f16, 5.2 GiB at q8_0 and 2.8 GiB at q4_0, so with the 5.54 GiB PTQ1_0 file even q4_0 lands around 9–10 GiB. An RTX 4070 12GB was measured at 9,918 MiB with 131K filled at q4_0 on PrismML's fork.

So on a 16 GB card the fit already works at q4_0; I'd guess xyzkv2's gain is mostly speed, since decode at that depth reads the whole cache every token. A tok/s number for q4_0 vs xyzkv2 at 161K on the same card would show it. The file sizes and that measurement are on this page: [Link removed by moderator]/new-gguf-quants/ (I built it.)

[xyznq1]

Nice work on the exact-sample verification. One note on the memory side, since the post says the 2-bit cache is what gets 160k into 16 GB: Bonsai 2's base is Qwen3.8 27B, where only 16 of the 64 layers keep a KV cache, 65,536 bytes per token at f16. At about 161K that is 9.8 GiB at f16, 5.2 GiB at q8_0 and 2.8 GiB at q4_0, so with the 5.54 GiB PTQ1_0 file even q4_0 lands around 9–10 GiB. An RTX 4070 12GB was measured at 9,918 MiB with 131K filled at q4_0 on PrismML's fork.

So on a 16 GB card the fit already works at q4_0; I'd guess xyzkv2's gain is mostly speed, since decode at that depth reads the whole cache every token. A tok/s number for q4_0 vs xyzkv2 at 161K on the same card would show it. The file sizes and that measurement are on this page: [Link removed by moderator]/new-gguf-quants/ (I built it.)

Thanks, you're right about the fit, "into 16 GB" oversells it, with q4_0 the whole thing lands around 9–10 GB at 161k and fits a 16 GB card fine. What xyzkv2 buys is the rest: 544 bytes per token per cache layer, about 1.3 GiB at 161k instead of 2.8, so model, drafter and 160k cache load in about 7.7 GB on ours and fit 10 and 12 GB cards. It's faster at depth too, moving 9 of the 16 cache layers to q4_0 cost us 6% decode at 160k. The trade is accuracy, q4_0 stays closer to f16 at that depth, that's what our 3-bit cache (xyzkv3) is for, it cuts xyzkv2's error by 77% at 160k with no speed cost we could measure.

[xyznq1]

Measured on the same card at 161k, same build and prompts: xyzkv2 156.3 tokens/s and 7.5 GiB, q4_0 141.3 tokens/s and 9.0 GiB. So about 11% faster and 1.5 GiB smaller, with the same tokens per round (2.94), so the gap is all cache reads. That's our main goal, less input for more output, lowering the bar while keeping speeds high, so a 27B with long context runs fast on cards people already have.