BeeLlama v0.4.0 + Q2_0 ternary CUDA (Blackwell / sm_120 build)

BeeLlama v0.4.0 + Q2_0 (ternary, group-64) CUDA — Blackwell build

This is an unofficial build of Anbeeld/beellama.cpp v0.4.0 with the CUDA weight
path for GGML_TYPE_Q2_0 (group-64 ternary, id 42), so ternary GGUFs such as
PrismML's Ternary-Bonsai-27B group-64 run on the GPU. Kernels are ported from
upstream ggml-org#25707 (@khosravipasha) — see PR Anbeeld#96 for provenance.

What's in the zip

Windows x64 runtime: all .dll plus the llama-*.exe tools (server, cli,
perplexity, bench, quantize, …). Test executables are omitted.

Requirements

• NVIDIA Blackwell GPU, compute capability 12.0 (RTX 50-series). The DLL is
  built only for sm_120; it will not run on older architectures.
• CUDA 12.8 runtime on PATH (cudart64_12.dll, cublas64_12.dll,
  cublasLt64_12.dll). Install the CUDA 12.8 toolkit/runtime, or place those
  DLLs next to the executables. They are not bundled here.
• Windows 10/11 x64.

Verification

GPU perplexity matches the CPU reference within FP tolerance on a group-64 GGUF:
6.7145 (GPU, -ngl 99) vs 6.7143 (CPU, -ngl 0) over 6x512-token chunks — both
decode (MMVQ) and prefill (MMQ) paths are correct.

Quick start (Ternary-Bonsai-27B group-64)

llama-server.exe -m Ternary-Bonsai-27B-Q2_g64.gguf ^
  -ngl 99 -fa on -c 65536 -ctk f16 -ctv f16 --port 8109

KV cache: use f16 up to ~128K (measured)

On Blackwell, dequant-free f16 KV is both fastest and highest quality until it
no longer fits. Decode t/s at 32K depth (llama-bench, Ternary-Bonsai-27B):

f16      39.1     <- fastest
q8_0     35.3
kvarn3   34.3
kvarn4   33.4

Bonsai f16 KV is ~64 KiB/token: 4 GiB @ 64K, 8 GiB @ 128K, 16 GiB @ 262K.
Use KVarN only near the ceiling (192-262K) where f16 no longer fits, and prefer
symmetric kvarn3/kvarn3 there. Within KVarN a higher number is not faster —
all widths pay the same normalization cost; more bits just means more bytes.

DFlash on v0.4.0

v0.4.0 uses upstream DFlash. It is a real speedup on predictable content
(code, structured text, tool calls) and roughly neutral on free prose:

Ternary-Bonsai-27B, greedy, --spec-draft-n-max 3, f16 KV:
  code   0.69 acceptance   81 t/s   (~1.85x over 44 t/s base)
  prose  0.21 acceptance   44 t/s   (= base)

--spec-draft-n-max 3 is the sweet spot. Higher depth or the adaptive
controller (uncapped n_max=15) draft too eagerly at low acceptance and end up
slower.

llama-server.exe -m Ternary-Bonsai-27B-Q2_g64.gguf ^
  -ngl 99 -fa on -c 8192 -ctk f16 -ctv f16 ^
  --spec-type draft-dflash -md DRAFTER.gguf -ngld 99 --spec-draft-n-max 3

Adapting an old DFlash drafter to v0.4.0

v0.4.0 only accepts upstream-schema drafters (general.architecture = dflash).
Older fork-era heads declare dflash-draft and will fail to load with
unknown model architecture: 'dflash-draft'. remap_dflash.py (attached)
rewrites the metadata and tensor names in place, no re-training or
re-quantization — tensor data is copied byte-for-byte:

python remap_dflash.py OLD-DFlash.gguf NEW-dflash-upstream.gguf

It maps: arch dflash-draft -> dflash; key prefixes dflash-draft. ->
dflash.; dflash.block_size, tokenizer.ggml.mask_token_id,
dflash.target_layers; tensors dflash_fc -> fc, dflash_hidden_norm ->
enc.output_norm, post_attention_norm -> ffn_norm. Requires the gguf
Python package. Validate by draft acceptance, not just that it loads.

License

MIT, same as upstream. This build adds nothing to the license terms; see the
LICENSE in the repository.

v0.3.2-ternary-bonsai.2 — native Q2_0 restored, DFlash works with Bonsai

Makes the fork a drop-in replacement for upstream BeeLlama again, and documents that speculative decoding does work with Ternary Bonsai.

Based on BeeLlama preview-v0.3.2 (commit a620cbd). The version here names that base; the .2 suffix counts revisions of this fork. Upstream version numbers belong to Anbeeld.

The two 2-bit types now coexist

The first release repurposed Q2_0 for PrismML's ternary format, which silently broke BeeLlama's own 2-bit type — same name, incompatible layout. They are now split by on-disk id:

type	id	name shown	weights	KV cache
BeeLlama native	53	q2_0	yes	yes
PrismML ternary (Bonsai)	42	q2_0_t	yes	no

The native type keeps the name q2_0, so existing models and existing command lines (-ctk q2_0, -ctk kvarn2, …) behave exactly as upstream.

The ternary format has no runtime quantizer — PrismML encodes offline — so it cannot back a KV cache. Requesting a 2-bit KV cache gives you the native planar type, as before.

Fixes

Four independent breakages in the previous release, all on the KV path:

• the planar 2-bit encoder wrote into 128-element ternary blocks
• the flash-attention K/V helpers read the wrong block geometry
• -ctk kvarn2 fell back to the ternary type on SWA layers
• case GGML_TYPE_Q2_0_B labels were missing in convert.cu / getrows.cu, leaving unreachable code after a return

If you are on the previous release, upgrade — with any 2-bit KV option it produced garbage without an error.

Speculative decoding: use the stock Qwen3.6-27B DFlash head

A DFlash drafter trained for plain Qwen3.6-27B accelerates Ternary Bonsai too. The head is tied to the architecture, not to the target's quantization — it ships no token_embd/output tensors (shared from the target at runtime) and cross-attends to the target's hidden states.

Pass --spec-draft-n-max 3 explicitly. DFlash's adaptive controller settles on 15, which suits ordinary quants accepting ~30% of drafted tokens. Bonsai accepts far less, so long draft blocks are mostly wasted:

--spec-draft-n-max	decode	acceptance
2	62.5 t/s	69.4%
3	64.4 t/s	63.1%
4	60.3 t/s	52.6%
8	55.7 t/s	34.4%
15 (adaptive default)	51.3 t/s	15.1%

Baseline without a drafter: 47.4 t/s. So n_max=3 is +36%, versus +8% if you let the controller decide. 3 happens to be llama.cpp's own default for the flag.

Measured at 32K context, short prompt, greedy, one card, one drafter. Acceptance normally falls as context grows, which would push the optimum shorter still — not measured.

Verification

The ternary path is unchanged, and this is checked rather than assumed. Perplexity on the same corpus and settings (-c 4096 -fa on --chunks 12):

PrismML reference:  3.1125 +/- 0.04339
this build:         3.1125 +/- 0.04339

A wrong bit layout fails silently, so a digit-for-digit match is the check that actually rules it out.

Type 53 is restored from the pre-port sources rather than tested — no model in that format was available here. Its code is the upstream implementation retargeted to the split type; reports welcome.

Binary

Windows x64, CUDA 12.8, built natively for sm_120 (Blackwell). Other architectures: build from source.

Known cosmetic issue inherited from upstream: the loader prints unknown type q2_0_t. BeeLlama never wired its own 2-bit types into the ftype table either; behaviour is unaffected.

MIT, same as upstream. Built on Anbeeld/beellama.cpp and PrismML-Eng/llama.cpp.

Ternary Bonsai Q2_0 support (v0.3.2 base)

BeeLlama with support for PrismML Ternary-Bonsai-27B (Q2_0, 2.125 bits/weight), so the model can use BeeLlama's TurboQuant / KVarN KV-cache compression — which the upstream PrismML fork does not provide.

Why

On a 16 GB card, Bonsai at 192K context with q8_0 KV leaves ~0.6 GB of headroom. With turbo3 KV it leaves ~5.4 GB, and 384K context fits more comfortably than 192K used to.

Verified

Tested on RTX 5060 Ti 16 GB (sm_120), CUDA 12.8, driver 610.62, Windows.

Numerically identical to the reference implementation — perplexity on the same corpus, model and settings:

build	PPL
upstream PrismML (reference)	3.1125 ± 0.04339
this build	3.1125 ± 0.04339

A wrong bit layout produces plausible garbage with no error at all, so matching perplexity is the only way to rule that out.

metric	upstream PrismML	this build
pp512 (prefill)	956.9 ± 28.8	1079.5 ± 35.8	+13%
tg128 (generation)	44.3 ± 0.2	51.6 ± 0.1	+16%

Context limits with turbo3 KV on 16 GB:

context	VRAM	headroom
192000	10939 MiB	5371 MiB
262144	11867 MiB	4443 MiB
393216	13595 MiB	2715 MiB

⚠️ Known limitation — BeeLlama's own Q2_0 (type 53)

This build replaces BeeLlama's native Q2_0 rather than coexisting with it. Both formats are called "Q2_0" but are not compatible (32 vs 128 block size, planar vs sequential bit layout).

If you have a model in BeeLlama's native Q2_0 format, do not use this build yet — it will produce garbage silently, with no error. Every other quantization type is untouched.

Splitting these into two coexisting types (Q2_0 = 42 ternary, Q2_0_B = 53 native) is in progress and will land in a follow-up release.

Usage

Model file: Ternary-Bonsai-27B-Q2_0.gguf (6.66 GiB — the g128 file, not Q2_0_g64).

Bonsai is a thinking model — use the sampling values from its model card. Greedy decoding (--temp 0) can make thinking models loop and never close </think>.

Long context, 384K:

llama-server -m Ternary-Bonsai-27B-Q2_0.gguf ^
  --ctx-size 393216 --n-gpu-layers 99 --flash-attn on ^
  --cache-type-k turbo3 --cache-type-v turbo3 ^
  --temp 0.7 --top-p 0.95 --top-k 20 --jinja

Maximum speed, short context: turbo3 trades ~7% generation speed for memory (at 32K real context: q8_0 51.3 t/s vs turbo3 47.9 t/s, prefill identical). If the context fits uncompressed, don't compress:

llama-server -m Ternary-Bonsai-27B-Q2_0.gguf ^
  --ctx-size 32768 --n-gpu-layers 99 --flash-attn on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --temp 0.7 --top-p 0.95 --top-k 20 --jinja

Other KV types: turbo2/3/4, turbo2/3/4_tcq, kvarn2/3/4/5.

Not supported

DFlash / speculative decoding for Bonsai. Drafters are target-specific — each only accelerates the exact model it was trained against. The only drafter that exists for Ternary Bonsai is PrismML's own DSpark head, and Bonsai has no built-in MTP tensors (n_layer == n_layer_all == 64). Heads trained for stock Qwen3.6 will not work.

Binaries

beellama-ternary-bonsai-win-cuda12.8-sm120.zip — Windows x64, CUDA 12.8, built natively for sm_120 (RTX 50-series). Includes llama-server, llama-cli, llama-bench, llama-perplexity and the required DLLs. Other architectures: build from source (see README).

Credits

MIT, same as upstream. Builds on Anbeeld/beellama.cpp (TurboQuant / KVarN / DFlash), PrismML-Eng/llama.cpp (the ternary Q2_0 implementation) and ggml-org/llama.cpp.

See also jarkevithwlad/turboquant-prismml-cuda, which solves the same problem on a different base and handles the type conflict by remapping only for qwen35 models.