MTP speculative decoding on 8GB GPUs: head quantization sweep + the KV-budget recipe (40 tok/s, Gemma 4 12B)

[vinceomas]

Everyone with big GPUs is enjoying MTP speculative decoding on Gemma 4; on 8GB
cards the usual report is "it's slower, the draft steals VRAM" — and that was
my first measurement too (-5%). Turns out that's a configuration problem, not
physics. I spent a day sweeping the whole trade-off space on an RTX 5060
Laptop (8GB) with a 12B dense model and the official MTP drafter, on top of
the mainline --spec-type draft-mtp support.

TL;DR numbers (ctx 4096, mean of 3 runs, spread ≤0.4 t/s):

config @ 48 layers, --parallel 1	tok/s
no MTP, KV f16	27.4
MTP Q6_K head, KV f16 (fully lossless)	40.0
MTP Q6_K head, KV q8 (for 16k ctx)	40.4

Real chat: 42+ tok/s fresh context, 33-37 with 16k filled, acceptance
0.73-0.91. Prefill 700-900 t/s. (Community benchmarks put this model at
~21 t/s on an RTX 4060 with stock Q4 and no MTP.)

The three findings that matter:

1. Never trade layers for the draft. MTP wins at 42/48 and 44/48 layers,
   but the moment the model itself doesn't fully fit, the full-offload
   baseline beats every MTP config. Also: past your VRAM budget the Windows
   driver doesn't OOM — it silently spills to shared memory and you lose 30%
   with zero error messages.
2. Fund the draft from the KV budget. --parallel 1 alone freed enough
   KV allocation to fit the head next to all 48 layers with an f16 cache.
   KV q8 buys you 16k context on top (~+1.3% PPL measured with
   llama-perplexity, saturating thanks to Gemma's SWA — 3 of 4 layers only
   cache a 1024 window).
3. Draft head acceptance is bit-insensitive. Q8→Q2 acceptance only drops
   55%→49% (n_max=2). Q6_K is the sweet spot: smaller than Q8, same
   acceptance, measurably faster (40.4 vs 39.6). And n_max=2 beats 3 and 4
   everywhere at these acceptance rates — though this looks like a property
   of the head, not of speculative decoding: the Gemma 4 head's per-token
   acceptance collapses with draft depth (0.80 → 0.54 → 0.43 at depth 1→2→3
   on free text), while the Qwen3.6 numbers posted in llama + spec: MTP Support  #22673 decay much more
   gently (0.83 @ n2, 0.72 @ n3, with n3 winning end-to-end there).

The recipe — step zero is sizing the weights to your VRAM: my main model
is an imatrix-guided, per-tensor fit-to-VRAM quant (6.18 GB vs 7.1 GB stock
Q4_K_M; planner in the repo). That ~0.9 GB of headroom is the currency every
other step spends — with a stock Q4_K_M you won't fit 48 layers + draft + KV:

llama-server -m <your-12B-quant>.gguf \
  --model-draft mtp-gemma-4-12b-it-Q6_K.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -ngl -1 -c 16384 --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --parallel 1 --context-shift --jinja

The Q6_K head is one llama-quantize --allow-requantize from the published
Q8_0 drafter. Drop the cache-type flags for the fully-lossless variant (fits
at 16k too, ~8% slower at long ctx because it reads twice the cache bytes).

Honest caveats: the exact numbers depend on step zero — the recipe's
shape transfers as-is, your numbers come out of the sweep tool.

Bonus — it works in-process too. llama-cpp-python doesn't expose MTP in
its high-level API, so I ported the MTP draft provider to its low-level ctypes
API (driving the same llama_set_embeddings_nextn / ctx_other machinery
from a Python token loop): 1.27-1.75× at every context frontier 2k-16k
(15.8 → 27.6 tok/s at 16k full), acceptance up to 0.94 in real chat, verified
lossless — with on-disk KV-cache sessions (instant conversation resume, no
re-prefill) and native tool calling intact. Same laws held: n_max=2 optimal,
draft paid from the KV budget. Remaining ~15% vs llama-server is Python
per-step overhead, profiled in the repo.

Full write-up, sweep tool (bd-mtpquant, reproduces everything on your
hardware), planner, in-process port, and all the CSVs: https://github.com/vinceomas/BrownDwarf

Quality validation: capability quiz unchanged (7/8 plain, 8/8 with thinking
enabled — identical to the no-MTP model), PPL parity within error bars on
llama-perplexity for MTP itself (it's verified-lossless by construction);
KV q8 costs ~+1.3% PPL at ctx 4096.

References & related work.

• Runtime: this all runs on the mainline --spec-type draft-mtp support
  (b9193+). The drafter itself is the official Gemma 4 MTP head shipped with
  the model, requantized to Q6_K.
• llama + spec: MTP Support  #22673 (MTP support, Qwen3.6) and llama: allow partial seq_rm for GDN models for speculative decoding #22400 (partial seq_rm rollback):
  related in-progress work. The design points discussed there — per-ubatch
  hidden-state hook, same-GGUF head loading, and the context tensor-sharing
  refactor mentioned in the thread — are directly relevant to driving MTP
  from outside llama-server: the in-process path above relies on the same
  ctx_other-style sharing.

If depth/acceptance data from another head is useful to #22673, I can re-run
the 8 GB sweeps (head-quant × n_max × layers) on that branch.