Qwen3.8 27B with MTP at a real 256K on one 24 GB GPU: numbers, and a drafter result I did not expect

[pich]

I spent the last weeks getting Qwen3.8 27B (dense, 3x Gated DeltaNet + 1 full-attention pattern, native 262144 ctx, 1-layer MTP head) into production on a single RTX PRO 4000 Blackwell SFF 24 GB, serving real agent work. Sharing the measured results because two of them seem useful to this project.

What runs: custom 5.01 BPW iMatrix/NVFP4 hybrid GGUF, embedded MTP with --spec-draft-n-max 8, Q4_0 KV for the target, F16 KV for the drafter, full 262144 context allocated, F16 mmproj on a second small card. Production mean 50.44 tok/s over ten runs. A genuinely filled cache (261.5K real input tokens, then generation) still decodes at 12.61 tok/s with about 515 MiB physical margin. Not a load-only claim; output hashes were checked cold vs hot.

Result 1: the runtime matters about as much as people hope. In a strict same-workload A/B with n-gram speculation disabled, clean master did 45.42 tok/s and master plus a small pinned set of PRs (#26001, #26048, #26705, the #27173 MTP chain, #24891, #25635) did 55.40 tok/s, +21.97%. The Flash Attention swizzling change alone was +7.4% prefill at 32K. Thanks to everyone behind those PRs.

Result 2, the one I did not expect: making the MTP head more precise made everything worse. I rebuilt the model with only the 8 MTP tensors changed (all 851 other tensors byte-identical): NVFP4 head 50.44 tok/s at 48.3% acceptance, iMatrix Q5_K head -3.4%, Q5_K+Q6_K head (+69 MiB) 37.02 tok/s at 33.1% acceptance. The higher-bit head is presumably closer to the BF16 parent, but the job is predicting this quantized target, and the NVFP4 errors in the head apparently line up with the NVFP4-heavy target. Is drafter-target quantization alignment something anyone else has measured? I did not find prior numbers.

Caveat I want to flag: with n_max=8 on a quantized target the outputs are not bitwise identical to target-only decoding, matching #25618. Both modes are internally deterministic.

Full write-up with the failed branches (DSpark sidecar on a second GPU lost to embedded MTP, ready-made NVFP4 quant failed PPL, n=9/10 hit CUDA graph allocation cliffs): https://piszczek.pl/blog/qwen38-27b-256k-50-tps-24gb-gpu
Model with checksums, tensor map and the pinned patch list: https://huggingface.co/cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF

[PatrickWalther]

It is possible that it only brings improvements for CUDA; that's why I gated it behind flags (so the default is off).

But I also noticed that it is really hard to benchmark. Most benchmarks have a bad mix of realistic workloads and context usage.
Ultimately, what I did is build a small UI that graphs performance, and then I just ran my day-to-day workloads to see how much avg. t/s i got. And every 2-3h i'd just change to different settings. What i noticed is that higher draft depth does sometimes result in error code 500 from llama.cpp at >200k context but when open-code retries it just works. It's a bigger issue with qwen cli as it doesn't retry on error code 500.

[pich]

Some CUDA numbers that might help narrow this down.

1. The +21.97% was CUDA only (sm120a Blackwell, though the build targets sm89 too), so a Vulkan regression and a CUDA gain aren't contradictory. On my box almost all of that gain came from the speculative : draft performance improvement (+10% t/s with deeper draft depth possible with this) + token rollback bugfix #27173 chain: the three small PRs together were +0.98%, and adding the chain took the same workload to +21.97%.

2. @sswtodo, those tg_3s dips (31-48 t/s in the middle of 60+ runs) look to me like acceptance collapse windows, not steady overhead. I'd log acceptance rate over time right next to tg. My acceptance-vs-depth curve was jagged, not monotonic: n_max=3 gave 78.7% acceptance, 4 through 7 got slower as rejected work piled up, and 8 jumped back to fast because it hit a favorable batch shape. Your n-max 3 with p-min 0.2, and n-max 5, both sit in what was the weak part of the curve for me. If Vulkan kernel shapes differ, the good depths will sit somewhere else, and a fixed n-max comparison across backends ends up comparing two different regimes.

3. The 500s at >200k with higher draft depth: I think I hit the same thing from the other side. With the full 262144 allocation, n=8 to n=9 died on one extra 162 MiB CUDA graph buffer, n=10 on an 81 MiB one, and at one point during tuning a 31 MiB scheduler allocation killed a server that supposedly had 76 MiB free. Near a full cache there's no contiguous room for the extra per-candidate buffers, and whether that shows up as an OOM or a retryable 500 seems to just be timing. That's also why the open-code retry works: the fragmentation is transient. Keeping roughly 500 MiB of real physical margin made the whole thing go away for me.

4. On noise: the worst offender was n-gram speculation. It learns the prompt across repeats, so any repeated-prompt benchmark blows up into nonsense (I watched 46.77 apparent turn into about 180). Every A/B number I'm quoting is with n-gram off, a fixed 10x512 workload, acceptance logged, and cold vs hot output hashes compared.

5. Happy to rerun any specific config on Blackwell if a same-settings CUDA vs Vulkan run helps pin down where those 3 t/s go