gfx1151 / Ryzen AI Max+ 395 throughput datapoints: Qwen3.8-27B with MTP + ngram-mod speculative decoding (b10435-era, ROCm)

[simongonzalezdc]

Datapoint for the gfx1151 perf picture — 27B dense model, full GPU offload,
single box, measured 2026-08-15. Happy to run more configurations on
request; methodology and raw scripts are public (repo link in profile) and
I'll keep this post to numbers.

Stack

Component	Value
APU	AMD Ryzen AI Max+ 395 (gfx1151 iGPU), unified LPDDR5X, GTT 64GB
Build	llama.cpp b10435-era (9d57ce4), ROCm/HIP backend
Model	Qwen3.8-27B, UD-Q4_K_XL, -ngl 99 -c 98304
Spec	--spec-type draft-mtp,ngram-mod --spec-draft-n-max 12 --spec-ngram-mod-n-min 24
Env	HSA_ENABLE_SDMA=0 HSA_XNACK=1
KV	f16, FA on (--flash-attn on)

Throughput (count-to-30 bench; cold = first exposure, warm = back-to-back
repeats — warm with ngram is a repetition artifact, labeled as such)

Config	Cold tok/s	Warm tok/s	Notes
MTP n=2	22	25	day-0 default, 16k ctx
MTP n=6	32.7	—	32k ctx
MTP n=9	55.7	—	32k ctx
MTP n=9 + ngram-mod	54.6	89.6-92.3	96k ctx
MTP n=12 + ngram-mod	59.4	94.7-98.5	96k ctx
MTP n=12 + ngram, n-min 24	59.7	148-163 (artifact)	champion; mean-len 37.7, acceptance ~0.96 warm

Real novel-prose traffic on the same stack: 11-24 tok/s (bandwidth-bound);
code ~29-40. Cold 16k-prompt TTFT ~41s (PP ~390 tok/s — prefill is the
bottleneck on this build); warm-prefix TTFT 0.2-0.6s.

n_max sweep (code lane): n12 peak; n16 regresses code to 21.3 tok/s —
deep-draft waste returns.

Backend A/B (same source, stacked n12/96k)

Metric	Vulkan (RADV)	ROCm
Cold 16k-prompt TTFT	55.5-56.1s	40.6-41.4s
TG c30 (warm)	48.8 / 50.2	94.7-98.5
Code lane	15.9	30-40
Story lane	5.3	~12

i.e. for spec-decode workloads on this box, Vulkan is ~half throughput and
-36% prefill vs ROCm — datapoint against the "Vulkan wins on gfx1151"
folklore for this workload class.

KV quant: q8_0 + FA fails to load on ROCm builds of this era; K-only
loads, saves ~1GB at -1.6% — not worth it (GQA KV is ~2.1GB/32k here, so
context is cheap).

Neural drafters on unified memory (for the spec-decode picture):
DSpark 1.36B BF16 drafter = 32.6 cold vs 59.6 champion cold (drafter pass
~2.7GB reads on the same bus as the verify pass); DFlash CPU-drafted =
79.1 cold but 0.063 acceptance on creative. ngram (prompt-derived, zero
bandwidth) is the winner on iGPUs.

---

Update (2026-08-17): the 262k unlock + what changed since the table above

The q8_0-KV + FA load failure above was observed on ROCm 7.2.1 — fixed by the ROCm 7.2.4 rebuild (same dflash tree, 9d57ce4). That fix is what unlocks the full native context:

-c 262144 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on
--spec-type draft-mtp,ngram-mod --spec-draft-n-max 12 --spec-ngram-mod-n-min 24

…still Q4_K_XL, same warm decode as the table (148–163 warm; cold at this config measures 52–57), now at 262,144 ctx instead of 98,304 with f16 KV. Two corrections/clarifications to the setup block: the machine here is the 96GB model (the "GTT 64GB" line is the carve, not total), and it's a native ROCm 7.2.4 install (Ubuntu 24.04.4, kernel 6.17.0-1032-oem, no Docker) with HSA_ENABLE_SDMA=0 HSA_XNACK=1.

Practical context guidance for anyone copying the config (measured, see curves below):

• Prefill halves per position-doubling: ~317 tok/s at 6k prompt → ~168 at 35k, and it falls off from there. 262k is for maintaining long sessions (the KV cache holds it cheaply), not loading huge fresh documents — a full-ctx fresh prefill is an hours-scale operation. Practical fresh-prompt ceiling before TTFT hurts: ~60–100k.
• Retrieval is not flat across depth: needle testing across the window shows retrieval degrading past roughly the first quarter of context. Put the load-bearing content early; in our controlled needle test even 90%-depth placement missed — everything past the first quarter degrades on this stack.

(These are model-level characteristics observed on Qwen3.8-27B, not gfx1151-specific — but they matter for anyone planning to actually use the full window. Reproduction details in the thread below; the HIP-over-Vulkan lane choice also got independent corroboration from another Strix Halo user hitting Vulkan DeviceLost at ~50% of 262k ctx in the comments.)

---

Update (2026-08-17, later): official gsm8k on this config — 0.980, and the raw-lane trap behind low served-model scores

Quality number for the same rig, and it corrects a figure we had briefly recorded internally: gsm8k exact-match 0.980 on the champion config above (Qwen3.8-27B Q4_K_XL, served through the same llama-server). Conditions: /v1/chat/completions lane, greedy (temp 0), zero-shot CoT prompt ending in ####, official grading (last #### N of the answer, exact numeric match against gold; the reasoning channel is never graded), seeded stratified subset (seed 20260817, 20 problems per answer-magnitude decile, 150 per cell) across 6 thinking-budget cells — 900/900 rows complete, zero API errors.

• Unbudgeted: 0.980 (xhigh and medium both 147/150).
• Hard reasoning_budget_tokens 2048 / 4096 / 8192, plus one 8192 + budget-message cell: 0.973–0.980 — flat at ceiling. Median thinking on this class is ~130 tokens, so the caps never engage (enforcement itself is real: a 256-token cap injection binds hard).
• Effort labels: medium matches xhigh accuracy at 74% of the wall time (12.7 s vs 17.3 s mean per problem).

Why we briefly had a 0.300 here (relevant if you eval served models): that run went through the raw /v1/completions lane, which hits a premature end-of-turn artifact — generations come back empty or truncated and exact-match counts them wrong; correct-when-actually-answered was ~75% even then. The 0.300 measured the lane, not the model. On this server, chat-lane generation grading is the clean path; loglikelihood-style evals through the completions endpoint stay blocked by the prompt-logprobs gap tracked in #27174.

[AlphasCodes]

Thank you for you post. I will try it this week on my Strix Halo system because the current llama.cpp Vulkan version runs into an issue (Device lost) around 50% context usage of configured 262144 context.

• AMD AI MAX+ 395 (Strix Halo), 128 GB
• Arch Linux (Kernel 7.1.8, Mesa 26.1.7, AMDGPU firmware 20260810-2, no local ROCm installation)
• latest llama.cpp ROCm version via official llama.ccp Docker image (ghcr.io/ggml-org/llama.cpp:server-rocm) from 8/16/2026
• Qwen 3.8 27B (Unsloth Q8_K_XL)
• Pi Coding agent without any extensions and a local small coding project (casual games [e.g. Monopoly] that should run in a browser without external libraries so HTML + Java Script code)

The first test shows great initial prompt processing / prefill numbers that are about 2x higher than the Vulkan version BUT the token generation is about 30% slower than just using "--spec-type draft-mtp,ngram-mod --spec-draft-n-max 3" on the llama.cpp Vulkan build. I don't activate/use "HSA_ENABLE_SDMA=0 HSA_XNACK=1" currently and I also don't know how much of a difference both environment variables could make.

What OS (Kernel version, Mesa version, amdgpu firmware version) and ROCm version do you use ?
Do you use Docker or something like this ?

Thank you

[simongonzalezdc]

Happy to share the full environment:

OS/runtime: Ubuntu 24.04.4 LTS, kernel 6.17.0-1032-oem, stock linux-firmware 20240318 — no custom Mesa/firmware pins. No Docker: native ROCm 7.2.4 (rocm-core 7.2.4.70204-93~24.04), engine runs from a systemd unit with exactly two env vars: HSA_ENABLE_SDMA=0 HSA_XNACK=1. Model is Q4_K_XL @ -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on on the dflash tree, post-ROCm 7.2.4 rebuild. Note vs the table above: that q8_0-KV + FA load failure was a b10435-era ROCm issue — the dflash tree fixes it, which is what unlocks the full 262k ctx config here.

On your ~30% slower generation — first suspect is the draft depth, not the runtime. With --spec-type draft-mtp,ngram-mod we measured draft acceptance ≈0.96 on code-ish/low-reasoning traffic with --spec-draft-n-max 12 + ngram n-min 24; n-max 3 leaves most of that on the table (each accepted draft step amortizes the fixed per-token dispatch cost). Try 12 and re-measure — that alone was the difference between ~40 and ~60 tok/s class on this chip for us. Caveat: acceptance drops toward ~0.5 under heavy thinking/reasoning workloads, so tune per workload class.

On the Vulkan device-lost at ~half of 262144 ctx: that crash class is why we live on the ROCm lane for high-context runs. With your 128GB you can hold the full 262k q8_0 KV comfortably (a step above the 96GB machine here — 64GB GTT carve — which also fits it) — if you want to stay on Vulkan, try bisecting with --flash-attn off/on and capping ctx, but native ROCm + the two env vars above has been crash-free through long agent sessions at full ctx.

Your ROCm prefill advantage over Vulkan matches what we measured here (about 1.4x / mid-30s% faster on our prefill probes). If you post your numbers at n-max 12 I can compare against our table directly — same silicon and about a third more memory, so the comparison should be clean.

[AlphasCodes]

Thank you for the information.

On your ~30% slower generation — first suspect is the draft depth, not the runtime. With --spec-type draft-mtp,ngram-mod we measured draft acceptance ≈0.96 on code-ish/low-reasoning traffic with --spec-draft-n-max 12 + ngram n-min 24; n-max 3 leaves most of that on the table (each accepted draft step amortizes the fixed per-token dispatch cost). Try 12 and re-measure — that alone was the difference between ~40 and ~60 tok/s class on this chip for us. Caveat: acceptance drops toward ~0.5 under heavy thinking/reasoning workloads, so tune per workload class.

That's the explanation why I get less token generation. My workflow is different. The model uses heavy reasoning (xhigh setting) for the coding and I get draft acceptance of (way) below 50% mostly. The real coding work is only about 20-30% of the reasoning work.

I did a new test with "--spec-draft-n-max 12 + ngram n-min 24" and the highest token generation value was about 26 tg/s. Then some in 2x tg/s range, more in the 1x tg/s range and about 30-50% in the x tg/s range (lowest value was 5.xx tg/s). With "--spec-draft-n-max 3 + ngram n-min 24" I had mostly 1x tg/s values (high around 18 tg/s) with some dips into the x tg/s (7.xx tg/s) range. So the higher draft setting is not good for my workflow. I will also test "--spec-draft-n-max 2 + ngram n-min 24".

The important win for me is that the llama.cpp ROCm version solved my device lost issue with half full context and delivers >= 2x prompt processing / prefill values. Initial 380 - 7xx t/s with ROCm vs 240 t/s with Vulkan. 128k context ~80-100 t/s with ROCm vs ~40 t/s with Vulkan.

I will not go back to the Vulkan version for now as long as the ROCm version has the same or better performance and runs stable.

I'm a casual noob vibe coder at home and I try to improve my setup. Thank you very much for your information.