Qwen3.8-Flash-Next on Strix Halo, Vulkan only: 33 tok/s decode, 500 tok/s prefill

[drluoto]

I run Qwen3.8-Flash-Next as the model server for my agents on a Bosgame M5 (Ryzen AI Max+ 395, Radeon 8060S, 128 GB), on Vulkan with the RADV driver. Over the last week I went through the stack piece by piece and measured everything against ten real conversations replayed on the server. Sharing what worked and what did not, in case it is useful for someone else.

Before and after, single stream, greedy, nothing cached. "Before" is what I ran on September 4: apepojken's build, stock UD-IQ4_XS, full Q8_0 draft head, draft-mtp plus ngram-mod with n-max 6. Decode in tokens per second:

workload	before	no speculation	now	now, n-max 6
short code, no context	49.0	32.8	58.1
new code @8k	30.7	29.6	41.6
prose @8k	23.6	29.3	30.1
file rewrite @8k	30.9	29.0	55.4	63.2
new code @32k	34.5	25.1	37.8
file rewrite @32k	37.4	25.1	48.5	55.2

Prefill went from 340 to 510 tok/s at 8k and 280 to 390 at 32k. On my real agent conversations (ten replayed, prose and tool calls mixed) the median went from 25 to 33 tok/s, and time to first token on a fresh 18k prompt from 67 s to 44 s. Greedy output is identical between runs.

Without speculation the trunk does about 27 tok/s, which is where the memory bandwidth of this box puts it.

What made the difference, biggest first:

1. Row-id hoisting was silently off for this model. Upstream sizes the expert-count shader for 256 experts, and this model has 512, so every expert matmul workgroup rescanned the routing tensor. Lifting the limit gave +19 % prefill. PR: vulkan: raise the hoisted row-id limit for mul_mat_id from 256 to 512 experts #28501.

2. Always draft the full three tokens, --spec-draft-p-min 0.0. Profiling the speculation step showed the extra draft tokens cost about 4 ms each here, so stopping early loses more than it saves. +13 % decode. I had swept p-min upwards earlier and missed this.

3. A draft head with a trimmed vocabulary (FR-Spec). The draft LM head was 81 % of the bytes read per drafted token; cutting it to the 65k most frequent tokens gave +8 % decode with the text unchanged. Draft head on HF: https://huggingface.co/drluoto/Qwen3.8-Flash-Next-MTP-GGUF

4. Deterministic output under -np 3 with MTP needed two fixes: zeroing KV cells when they are freed (from nathanw1014) and disabling the ported GDN state-cache fusion.

What did not help on this hardware: fusing the small glue kernels (dispatch count is not the bottleneck), a grouped expert kernel that shares weight reads between the speculative tokens (the Infinity Cache already does that for you), ubatch 4096, and ngram-mod on Vulkan, whose 64-token drafts make every verify step cost 200–500 ms. Draft length is a trade-off: n-max 6 is the setting for file rewrites, where the drafter is right every time, but it halves prose, so the default is 3.

Everything is on one branch with a readme, the exact server command line and the requant recipe for the trunk: https://github.com/drluoto/llama.cpp/tree/strix-halo-vulkan (readme in docs/strix-halo-flash-next-vulkan.md)

Earlier thread with the ROCm side and the MTP head export: #27950

Done together with Claude Fable 5.1 in Claude Code; I reviewed and measured everything on the box myself. A Vitronia project.