Qwen3.8-Flash-Next at 1M context on AMD Strix Halo (Vulkan): a tuned fork with numbers and patches

[vahpetr]

Hi all. Over the last six weeks I've been getting Qwen3.8-Flash-Next (the 177B MoE, qwen4exp) to run well on one AMD Strix Halo box — a Ryzen AI Max+ 395 with 128 GB of unified memory, on Vulkan/RADV (no ROCm). I'm sharing the result in case it's useful to anyone with the same hardware, or in case some of the pieces are worth bringing upstream.

What I get on that box, one slot, one conversation:

• a 1,048,576-token context in 102.7 GiB of GPU memory, flat during the whole fill
• decode: 61 t/s on an empty context, 45 t/s at 524k, 35.5 t/s at 1M
• prefill: 550 t/s at 8k, ~420 t/s at 213k depth; cold fill to 1M in about 1.1 h
• for reference, stock llama.cpp on the same box: ~24–27 t/s with a dense 262k window and no working speculative decoding

What's in the fork (on top of upstream, drluoto's strix-halo branch and PR #27742):

• the model's own sparse attention (the QSA indexer) actually running at 524k–1M: a pooled-key cache, an incremental block census, a fused top-k select, indexed flash attention for prefill; dense below 64k, so short prompts are unchanged
• MTP speculative decoding that pays off on Vulkan (~1.5x decode), with the draft itself running sparse at depth
• a graph/allocation-plan cache, a KQ mask built from cell metadata instead of a context x batch plane, and a memory admission check so a prefill chunk waits instead of running the box out of memory
• a disk prompt cache with recurrent-state checkpoints: a 72k-token branch resumes in 0.6 s instead of a 209 s re-prefill
• webui: context-reserve auto-compaction for long agent sessions, per-chat settings, always-visible timing chips (prefill/decode speed, cache share, MTP acceptance, tool-call durations)
• Vulkan kernel work for RDNA3/3.5: a radix top-k select, indexed and metadata flash attention, GDN and hyper-connection fusions, mul_mat_id changes for 512 experts

Repo: https://github.com/vahpetr/llama.cpp-s1-fork (branch s1-public). The README has the measured numbers with dates; docs/s1/PATCHES.md lists all ~190 changes with one-line descriptions; deploy/ has compose files and setup scripts for Strix Halo and for a Radeon 780M box; images are on Docker Hub (vahpetr/llama-cpp-s1-fork). Everything was measured on my two machines only — ROCm and NVIDIA are untested.

One change is already open as PR #28118. Several others (the QSA runtime, the disk prompt cache, the Vulkan kernels) could become PRs if the maintainers think they fit. I'd appreciate a pointer on which parts you'd want to see first and in what shape.

Thanks to drluoto and to the authors of #27742 — this builds on their work.

[vahpetr]

Quick update after another week on the same box (v60 of the fork, Oct 16).

Prefill at depth got the biggest win. A 64k-token step on top of a long context, same machine, same quant, measured with the fork's own exactness probe on both versions:

	before	now
step → 262k	368 t/s	474 t/s
step → 524k	284 t/s	413 t/s
step → 1M	195 t/s	329 t/s
cold fill 0 → 1M	64 min	44 min
cold 32k prompt	423 t/s	459 t/s

Decode is unchanged within noise. GPU memory is up by ~0.8 GiB (an f16 mirror of the pooled indexer keys).

All of it is bit-identical to the previous version: I compare full logits, saved states and the ordered top-k ids at 1,502 positions and 224 state snapshots, batch widths 1–9, 8k to 1M, including save/restore and replay. The levers were the indexer's score epilogue fused into its GEMM, the f16 key mirror for that GEMM, a dense tile for m ≤ 16 rows, a precast of the B operand for mul_mat_id, and a GDN concat+conv fusion — none of them change a summation order.

One real bug on the way: restoring a conversation with images from the prompt cache lost the indexer's M-RoPE x/y, so images after a restore attended wrongly. Fixed; saved states now carry a provenance mark, old image entries are refused instead of silently restored, text ones still load.

I also ran the fork against stock b10689 on a different family (Qwen3.6-35B-A3B, 780M laptop, MTP off): +3/+6/+17 % prefill at 8k/32k/72k, +6–13 % decode. Same greedy tokens, but logprobs differ there — I make no exactness claim outside Qwen3.8.

Repo as before (s1-public, tag s1-public-b11165-88479e0); docs/s1/DEPLOY.md now has the step-by-step for Strix Halo and the 780M box, docs/s1/RELEASE-NOTES.md the per-version list, images v60 / v60k on Docker Hub. Still happy to cut any piece into a PR if there's interest.

[vahpetr]

Update since v60 (Oct 16): four releases, v61 → v64 (Oct 17–21). Same box (Ryzen AI Max+ 395, 128 GB, Vulkan/RADV), same model at 1M context; everything below is in the repo's docs/s1 (BENCHMARKS.md, KNOBS.md, PATCHES.md, RELEASE-NOTES.md) and on Docker Hub as v61…v64 / v64k.

v61 (Oct 17) — the sparse-attention (QSA) path: the score GEMM is dispatched key-tile-major and the selection runs without the key rows. Measured on the server with the fixed 1M corpus: cold 128k prompt 427 t/s, fill steps to 262k / 524k / 1M at 483 / 425 / 351 t/s, full 0→1M fill 0.70 h (was 1.07), decode with the MTP draft at 1M 19.9 ms/tok (was 24.4), TTFT on a 1M history + 256 new tokens 1.9 s (was 2.5). Bit-identical output to v60.

v62 (Oct 19) — correctness only, no speed change: flash-attention Q8_0 K-cache out-of-bounds read, TOP_K with n-ary ops, in-place mul_mat strides, hardening of slot-file restores (truncated / mismatched files are refused instead of asserting). Ports of the corresponding upstream fixes where they existed.

v63 (Oct 21, morning) — server-side work: skip the padded expert rows in mul_mat_id (the prepass tile list tells the kernel which rows are real), semantic checkpoints in the disk prompt cache plus a cache_pin request field, context checkpoints and the MTP draft state saved into /slots files (ports of #26004 / #30574) so a restored 1M slot decodes with a warm draft instead of a cold one. Numbers vs v61: cold 128k 440 t/s (+2.9 %), fill steps +2 %, 0→1M 41.4 min, TTFT after a slot restore 1.73 s (−8 %); decode unchanged; the slot file grows +18 % at 1M because it now carries the draft state. Gated by the full 16.7-hour exactness run (1 502 logit positions + 224 state points vs the reference).

v64 (Oct 21, evening) — three host-side tile / shared-memory choices for AMD coopmat1 on RADV, shader math unchanged: (1) flash-attention LDS layouts for the dense prefill kernel (31 → 21 KiB per workgroup, 4 → 6 workgroups per WGP); (2) an 8-warp mul_mat_id tile for the expert GEMMs (BM 128 × BN 64, LDS row pad 2 = a 72-byte stride, which is RADV-only on purpose — the conformant pad 4 keeps almost none of the gain); (3) a small tile table for the dense q4_K / q5_0 / q6_K GEMMs. A/B on the live server, same image, knobs off vs on, two reps: prefill 32k 467.5 → 500.9 t/s (+7.2 %), 72k 385.6 → 417.3 t/s (+8.2 %); 32 greedy tokens + top-8 logprobs identical across arms. Per-node (perf logger at 32k): flash-attention −16.5 %, expert GEMM −4.1 %, dense GEMM −4.1 %. Decode unchanged, GPU memory +81 MiB. On the 780M laptop (IQ3_S quant of the same model) the expert-GEMM tile alone is −12…−21 % on that node in isolation, bit-identical to stock by a byte classifier over 159 cases.

Also in v64: the probabilistic draft sampling from #27694 (ported in v63 behind --spec-draft-sampling) is now on by default in the composes. Greedy output is unaffected; I have not yet measured the decode effect at T > 0 against greedy drafting on this box (a same-image off/on run is next), so no number for it.

Process note, since I keep calling things "exact": v64 went out on the short exactness gate (the candidate's greedy passes vs the reference) rather than the full 16-hour one, with the full run scheduled as an automatic overnight check that rolls back to v63 on any mismatch. Everything else above was gated the slow way.

Nothing new to ask here; the pieces are still cut as standalone patches (docs/s1/PATCHES.md) and I'm still glad to turn any of them into a PR if there is interest.