MMQ nwarps is under-tuned for gfx906

[JackBinary]

Background

While investigating the potential to port the kernels from the gfx906 fork back to mainline llama.cpp and observing that that fork's nwarps value of 2 (half that of mainline) reduced the prompt processing performance of both models I tested, I tried increasing it instead and saw major gains.

Testing (pp512)

Both models were ran at q8 on a single mi60, and validated on a second:

nwarps	Qwen3 27B (dense)	Gemma 26B-A4B (MoE)
2	84 (-53%)	412 (-51%)
4 (current)	180 (base)	837 (base)
8	237 (+32%)	1091 (+30%)
16	277 (+54%)	1168 (+40%)

Correctness

test-backend-ops -o MUL_MAT passes at nwarps=16, though there was some non-deterministic flicker of pass-fail, specifically at q5_1, m=16, n=1, k=32 measured at nwarps=8. This flicker was not observed at nwarps=16, but I can't prove that it will never happen.

Suggestion

I think it may be worth re-evaluating the 256/warp_size heuristic for non-MFMA AMD cards.

Caveats

This may be gfx906 specific behavior, and may even be specific to the mi60 (which has more cores than the mi50). I can't verify otherwise at this time. I also only tested this at q8, as I was specifically targeting q8 performance.

[Orcazephyr]

I got +76% pp512 on a mi50 32gb with Qwen 3.5 9B q8_0, getting around 860 t/sec.

So I don't think it's only the mi60 that sees gains

[rumas77]

gfx906: the nwarps change helps Q8_0 (+19%) but regresses Q4_K (-7%) — same GPU, same build

I tested this on 5x Radeon VII (gfx906, Vega20), ROCm 6.2.4, llama.cpp b10451. Short version:
I reproduce your effect on Q8_0, but it inverts on Q4_K, which suggests gfx906 MMQ tuning
needs quant/type-specific treatment rather than a blanket architecture default.

What I changed

On b10451 the MMQ config is dispatched per architecture, and gfx906 is neither CDNA nor RDNA, so
it falls through to the RDNA2 table (mmq.cuh, ggml_cuda_mmq_get_config):

if (GGML_CUDA_CC_IS_AMD(cc)) {
    if (CDNA) / if (RDNA4) / if (RDNA3_5) / if (RDNA3) ...
    return ggml_cuda_mmq_get_config_rdna2(type, J, fallback);   // gfx906 lands here
}

RDNA2 is wave32 and every one of its 242 CASE entries uses nthreads=256, which gives it the
intended 8 warps. gfx906 is wave64, and since nwarps = nthreads / warp_size, the same table
gives 4 warps. For comparison, CDNA (also wave64) uses nthreads=512 throughout.

So I added a GCN config that is the RDNA2 table with only nthreads changed 256 → 512
(nwarps 4 → 8), dispatched via the already-present GGML_CUDA_CC_IS_GCN(cc) on the host side and
the existing device-side GCN macro from vendors/hip.h. Occupancy, tile dims, SRAM layout,
K_vram and stream_k are untouched.

Note mmq.cuh has static_assert(nthreads <= 512), so on wave64 this caps at nwarps 8 —
nwarps 16 isn't reachable on this base without further changes.

Results — one GPU, one variable

llama-bench -p 128,512,2048 -n 0 -ngl 99 -r 3 on a single Radeon VII with full GPU offload, so
there is no CPU layer split or inter-GPU transport in the comparison. Same two binaries, same
model, only the quant differs:

Qwen2.5-VL-7B-Instruct

quant	pp	nwarps 4	nwarps 8	delta	stddev 4/8
Q4_K_M	128	492.9	545.6	+10.7%	11.6 / 14.4
Q4_K_M	512	637.9	594.2	−6.9%	2.4 / 2.4
Q4_K_M	2048	621.5	579.8	−6.7%	0.0 / 0.0
Q8_0	128	555.9	665.6	+19.7%	14.8 / 20.7
Q8_0	512	611.8	728.5	+19.1%	2.3 / 3.1
Q8_0	2048	596.7	706.7	+18.4%	0.1 / 0.3

The pp128 rows have large standard deviations and I would not lean on them. pp512 and pp2048 are
tight for both quants.

So on the same card, same binaries and same model, nwarps 8 is worth about +19% at Q8_0 and
about −7% at Q4_K. Your Q8 direction reproduces here. The magnitude differs from your +32%, but the runs also differ
in GPU, model, ROCm version and llama.cpp base, so I would not attribute that difference to any
one factor.

I mention the one-variable framing explicitly because my first attempt at this compared my Q4
result against your Q8 result and concluded "quantization dependence" — that wasn't justified,
since quant, model, GPU, build and harness all differed at once. The table above changes only the
quantization.

Correctness

test-backend-ops -o MUL_MAT, 5 runs on each build: all rc=0, 1188 OK, 0 FAIL, both at
nwarps 4 and 8. I did not reproduce the intermittent pass/fail you mentioned at nwarps=8 — though
five nwarps=8 runs on one machine obviously can't rule it out, and it may be MI60- or
ROCm-version-specific.

Suggestion

A blanket gfx906 nwarps=8 default would regress Q4_K_M in this test, so any change here appears
to need quant/type-specific tuning rather than one architecture-wide value. The more general point is that gfx906 has no
MMQ config of its own at all and inherits a wave32 table, so there may be more here than one
constant: nthreads, occupancy, tile dims and the quant-specific layouts are all currently
inherited rather than tuned.

Caveats on my end: one GPU model, one 7B model, two quantizations, and I have not tested Q5/Q6 or
the MoE path. Happy to run more combinations if it would help — I have the two builds set up for
A/B and can sweep whatever is useful.

[JackBinary]

My understanding is that the GFX906 Architecture is severely compute bound for LLMs, given the 1T/s memory bandwidth but measly 29 TFLOPS of fp16 compute. This bucks traditional wisdom that makes Q4 faster for most GPUs than Q8, where Q8's simplicity wins out in speed over Q4's compression. This could extend to the nwarps conversation, where at q4 we're already severely compute constrained and more nwarps doesn't help or even adds overhead, while Q8 has a bit of room to spare.

[rumas77]

Your explanation fits our measurements surprisingly well. One refinement: on gfx906 the llama.cpp quantized MMQ/MMVQ paths go through DP4A integer dot-product kernels (ggml_cuda_dp4a, and common.cuh marks VEGA20 as the dp4a floor), so for these kernels the relevant compute path is DP4A plus quant unpacking rather than the headline FP16 TFLOPS figure. The broader compute-vs-bandwidth point still looks plausible.

Same 5x Radeon VII rig as my earlier comment (gfx906, ROCm 6.2.4, llama.cpp b10451), Qwen3.6-35B-A3B, -b 16384 -ub 256, speculation off, ignore_eos with fixed 400-token completions, two interleaved pairs:

quant	pair 1	pair 2
Q4_K_XL	44.13	44.14
Q8_0	47.15	47.12

That is +6.8% for Q8_0, with 0.02% drift between the two Q4 control legs, despite the Q8 model occupying ~40.1 GB VRAM against ~26.2 GB for Q4_K_XL.

It also lines up with the nwarps result I posted earlier: Q8_0 gained +19.1% pp512 while Q4_K_M lost 6.9% on the same card.

Q4_K carries substantially more superblock unpack/scale/min handling than Q8_0, so a working hypothesis is that gfx906 is already sufficiently compute/dequant limited on Q4_K that its smaller memory footprint does not buy the usual speed advantage, while Q8_0 is simple enough that the extra bandwidth cost is outweighed by cheaper execution.

Two caveats. I have not measured DP4A utilisation directly, so that remains a hypothesis rather than a demonstrated bottleneck. And our Q4-vs-Q8 prefill comparison on the 5-GPU configuration came out inconclusive — the effect was smaller than run-to-run drift — so the +6.8% above is a decode result only.