DFlash2 speculative decoding on a ternary 27B (PrismML fork): 2.15-2.22x on math/code on one L4, with controls

[NakliTechie]

I benchmarked the DFlash2 speculative decoding from #27816 on PrismML's Ternary Bonsai 2 27B. Mainline can't load that GGUF, so I used PrismML's fork with #27816 cherry-picked on top (upstream there as PrismML-Eng#261). The DFlash2 code is unchanged from mainline, so these numbers may be useful here.

Setup: llama-server, NVIDIA L4 (sm_89, CUDA 12.8), greedy, -np 1, target Ternary-Bonsai-2-27B-PQ2_0.gguf, drafter Q4_K_M. Speedup is decode tok/s against the fastest plain config on the same GPU (PTQ1_0, no speculation).

thinking off	n	plain tok/s	DFlash2 tok/s	speedup	accuracy (same weights, plain -> spec)
GSM8K	100	31.4	67.6	2.15x	0.94 -> 0.93
MBPP (sanitized)	100	31.6	68.4	2.16x	0.80 -> 0.80
MATH-500	100	30.6	67.8	2.22x	0.76 -> 0.75
MT-Bench turn 1	80	31.3	42.9	1.37x	-

Findings that may be useful beyond this model:

• --spec-type ngram-mod on the same sets: 0.92-0.95x. It only helps on HumanEval (1.68x), where 64% of each answer copies the prompt. DFlash2 gets 2.66x there. That makes HumanEval a misleading headline for any speculative method.
• Draft length matters by content. --spec-draft-n-max 7 is best for code/math/extraction (about 2-2.3x). 3 is best for open-ended text: MT-Bench writing goes from 0.98x to 1.26x, and code/math lose about 15%. A per-request speculative.n_max exists and could switch between them.
• Batched verify is not bit-identical to single-row decode. Greedy outputs split at near-tied tokens (top-2 gap <= 0.024 nats in every split we checked). Accuracy moved by at most one problem per set.
• Thinking mode: 1.25-1.63x. Prompts where greedy reasoning falls into a repetition loop inflate speculative tok/s, because loops are trivial to draft, so we dropped them from all configs.
• Reproducibility: a second L4 machine accepted identical drafts (acceptance and tau equal to three decimals), with speed within 3%.

Build (verified on a clean L4):

git clone -b dflash2-on-prism https://github.com/NakliTechie/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --target llama-server -j $(nproc)
./build/bin/llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 999 -fa on --jinja \
  -md Qwen3.8-27B-DFlash2-r3-Q4_K_M.gguf --spec-type draft-dflash --spec-draft-n-max 7 -ngld 999

While testing I found that cmake --build ... -j with no number ran out of memory on a 30 GB machine, and a build with no CMAKE_CUDA_ARCHITECTURES compiled every architecture on CMake 3.22.

Links:

• Drafter + numbers: https://huggingface.co/naklitechie/Qwen3.8-27B-DFlash2-ternary-bonsai2
• Per-sample outputs, scoring scripts: https://huggingface.co/datasets/naklitechie/bonsai2-dflash2-bench
• Same drafter on Apple Silicon (MLX): https://github.com/NakliTechie/dflash-mlx-bonsai2
• Same drafter in the browser (WebGPU): https://localmind.naklitechie.com