Everything since v0.6.2: the dense prefill work that landed this week, the whole v0.6.3
beta line for DeepSeek-V4, and an upstream merge to 17 August. Driver, libdrm and shader
compiler are unchanged from v0.6.2, so the two payloads compare like for like.
Headline, measured payload against payload: prefill is 5 to 19% faster depending on the
model, decode is unchanged, and output is bit-identical.
Dense prefill, on by default
Two changes to the quantised coopmat1 matmul path, both gated on measurement rather than a
flag, both leaving output bit-identical.
wave32 for the quantised dense pipelines. RDNA3.x WMMA is wave32-native, so a wave64
subgroup issues each coopmat op as two halves. GGML_VK_MMID_WAVE32 has exploited that for
mul_mat_id since the mmid stack landed; the dense pipelines were still on the driver default
and now get the same treatment. Only the
quantised tiles are retiled, because the float paths are bandwidth-bound on the weight
stream rather than issue-bound, and the win tracks inline dequant instruction count (q6_K
goes 3907 to 3433 instructions at identical VGPR count and occupancy).
A per-path LDS pad. buf_a/buf_b are 4-byte, so the shared-memory stride in elements is
the stride in banks, and RDNA has 32. The pad was one constant for every non-Intel device
and 4 is not the optimum on gfx1151. Pad 2 spreads across 16 banks instead of 8 and is
worth about 13% on the quantised path in a standalone MUL_MAT sweep, largest where the
dequant is cheapest (q4_0 +32%, q8_0 +22%, q4_K +10%, q6_K +1%), which is the complement
of what wave32 helps. The float path stays at pad 4, on measurement rather than theory.
The stride must stay even or the 8/16-byte LDS loads lose alignment, which measures -53%.
Measured on the release payloads, gfx1151/RADV
v0.6.2's payload against this one, both extracted from their own tarballs and run against the
same pinned driver, so the only thing that differs is the llama.cpp payload. Medians of 3
launches per arm in counterbalanced order (A B B A A B), -r 3 inside each launch, f16 KV.
Launch-to-launch spread is at or under 1.4% on every prefill cell except one at 2.1%.
| model | cell | v0.6.2 | v0.6.4 | |
|---|---|---|---|---|
| Qwen3.8-27B UD-Q6_K_XL | pp2048 ub256 | 313.7 | 360.1 | +14.8% |
| pp2048 ub2048 | 286.6 | 340.8 | +18.9% | |
| tg128 | 8.5 | 8.5 | unchanged | |
| Qwen3-32B UD-Q6_K_XL | pp2048 ub256 | 260.2 | 288.8 | +11.0% |
| pp2048 ub2048 | 251.8 | 271.2 | +7.7% | |
| tg128 | 7.8 | 7.8 | unchanged | |
| Qwen3-Coder-30B-A3B UD-Q4_K_XL | pp2048 ub256 | 1269.0 | 1332.5 | +5.0% |
| pp2048 ub2048 | 1814.3 | 1940.8 | +7.0% | |
| tg128 | 97.9 | 98.0 | unchanged |
Qwen3.8-27B gains more than the other two because it is the one model here with a delta-net
path, so it also collects the transposed-concat change below. Qwen3-32B is the clean read on
the matmul work alone. Decode is untouched by all of this and measures that way: every tg128
cell lands inside its own launch spread.
Perplexity is unchanged, all 20 per-chunk values identical: the retile changes which warp
owns an output sub-tile, and the pad moves addresses, neither touches the K-reduction order.
Opt-outs: GGML_VK_DENSE_WAVE32=0, GGML_VK_SHMEM_PAD=<n> (overrides the pad on both paths, for probing).
Two more default-on changes
Transposed concat. The delta-net conv-state path transposes straight into a dim-0
concat, so the generic kernel walked src1 with a 40960-byte stride on qwen35: 160 * 256 B
with 160 divisible by 16, so every read landed on the same one of the 16 memory channels,
13.7 GB/s against 138.9 for the tiled path. The tiled route has been behind
GGML_VK_CONCAT_TRANSPOSE=1 since it landed and is now the default. Qwen3.8-27B pp2048:
+0.4% at ub 256, +4.7% at ub 1024, +7.2% at ub 2048. GGML_VK_CONCAT_TRANSPOSE=0 opts out.
Bulk reads from write-combined mappings. ggml_vk_buffer_read_2d took the direct-CPU
path whenever the buffer was host-visible on a UMA device, which holds only for host-cached
mappings. amdgpu hands out write-combined mappings for GTT, and those read back at around
200 MB/s. Measured on hybrid-attention context checkpoints, that cost about 600 ms per
prompt. Small reads still go direct so they do not pay the fence round trip.
Opt-in: f16 B operand for dense matmul
GGML_VK_DENSE_F16B=auto routes the B operand of quantised dense matmuls through the f16
pipelines that already existed for coopmat2. Numerically identical, since mul_mm stages B
into shared FLOAT_TYPE either way, and wikitext PPL matches to 4 dp. Worth +5.3 to +6.0% on
Qwen3.8-27B pp2048 across ub 256 to 2048. auto restricts it to reduction width 5120, the
only width with positive data; =1 is the old all-shapes behaviour, and costs about 1% on
narrow dense models and 0.5% on MoE. Left off by default until a per-shape predicate
replaces the width equality.
DeepSeek-V4
Everything from the v0.6.3 alpha and betas, which were never promoted to a stable tag. All
of it is kernel-level measurement on gfx1151: the UD-IQ3_XXS model is about 97 GiB and this
box has 62 GB, so end-to-end throughput has still not been reproduced here.
- Sparse prefill attention, about 3.0x on the PP2048 shape at every depth from 32k to
512k (278.0 ms to 91.9 ms at kv=11008, 306.1 to 103.0 at kv=133888). - Lightning Indexer prefill parallelized, 25 to 51% off the indexer depending on shape.
- Small-batch decode no longer falls off a cliff. Sparse prefill gates on batch >= 64
and the gather gated on batch == 1, so batches 2 to 63, exactly where a speculative draft
lands, fell through to dense attention over the whole compressed KV. Gathered against
dense: 3.23x at batch 2, 7.93x at batch 4, 6.32x at batch 8. - Selections deduplicated across drafted tokens, on by default, with the batch ceiling
priced from the measured union read back from the device rather than the worst case. - Quantised KV works, and is now the fast path. Both sparse paths hard-required an f16 K
cache and silently returned early otherwise, so-ctk q8_0disabled the sparse prefill,
the gather and the union with no warning. Fixed, and q8_0/q4_0 rows are decoded inside the
gather, which converts per-use work into per-row work. At batch 8 that is 2.45x for f16,
4.43x for q8_0 and 4.68x for q4_0 against the dense fallback each type took before. A q4_0
cache is a quarter of f16's memory and now the fastest of the three, which inverts the
usual advice for this model.
Escape hatches: GGML_VK_FA_TOPK_UNION=0 disables the dedup, GGML_VK_FA_TOPK_GATHER=0
the small-batch gather, GGML_VK_FA_TOPK_SPLIT=0 falls back to the single coopmat kernel,
GGML_VK_FA_TOPK_CM=0 to the old scalar sparse kernel, GGML_VK_FA_TOPK=0 to ordinary
dense FA. GGML_VK_FA_UNION_STATS=N reports what the gate decided and on what measurement.
Model support from the upstream merge
Motif-3 (314B.A13B, carried on the fork), BailingMoE3, Kimi-K3, MiniMax-Text-01 and
MiniMax-M1, Granite-Switch, Nemotron-3.5 with DFlash and MTP, TQ2_0 on Vulkan, plus the
Muse Glimmer mtmd and tool-call fixes. Auto-detection of the draft model type from GGUF
metadata means --spec-type is usually no longer needed.
Validation
test-backend-opson this payload's build, shipping default configuration:
33055/33055 pass on Vulkan0, both backends pass.- Tarball smoke tested from a clean extraction with every ICD variable unset, so the bundled
driver is what runs. - Perplexity across the two default-on matmul changes is unchanged: 6.9496 +/- 0.24246 in
both arms with all 20 per-chunk values identical, since the retile changes which warp owns
an output sub-tile and the pad moves addresses, and neither touches arithmetic or its
order.GGML_VK_DENSE_F16Bmatches to 4 dp on wikitext. - The DeepSeek-V4 paths pass 13318
FLASH_ATTN_EXTcases including the q8_0 and q4_0 top-k
shapes a V4 decode hits. - The A/B above is the payload measurement; the per-commit numbers quoted in the sections
below come from development builds, which use the distro shader compiler rather than the
pinned one, so treat them as the reason a change is in rather than as payload figures.
Credit
The DeepSeek-V4 sparse prefill and Lightning Indexer work is @Mushoz's, from PR #2 and PR #3
on the llama.cpp fork, carried here with authorship intact. The small-batch decode gap was
their diagnosis too. Model support is upstream work; this release brings it to the validated
Strix Halo stack.
Notes
Driver unchanged from v0.6: Mesa 26.3.0-devel RADV (git-d18d598e), libdrm 2.4.134. Payload
built with the same pinned shaderc (v2026.3-dev 49a8724d) and gcc 13 as v0.6 through v0.6.2,
which is what makes the A/B above a like-for-like comparison of the llama.cpp payload alone.
The bundled web UI assets came from the latest bucket because the pinned build-number
bucket 404'd at build time; that affects the served UI only, not the backend.
