AMD PR Set 4 of 6 Matvec and Fusion Decode Optimizations

[Geramy]

AMD PR Set 4 of 6 — Review and Summarization

Johannes and llama.cpp team, below is a group of related patches from AMD intended to optimize a set of high-priority models on the ROCm backend. I've grouped these together — from a larger set of 25 patches — to hopefully make your job easier in reviewing them, and I've provided a short summary for each. These PR numbers (#59, #67, etc.) refer to the AMD team's downstream fork at AMD-Ecosystem/llama.cpp, where this work landed first; the goal is to bring it upstream in reviewable batches.

This fourth group is the decode path: cutting launch count and lifting wave occupancy by fusing work into the GEMV dispatch, plus one fusion-related correctness fix. The gains are modest individually but compound on small models, where launch overhead dominates.

4. Matvec / Fusion — Decode Optimizations

1. #59 — gfx1151 (Strix Halo): fuse attn_k+v into single MMVQ dispatch — jeffli-xilinx
   • Concatenates wk/wv at model load so the graph emits one MUL_MAT instead of two, doubling the MMVQ row count and lifting wave occupancy from 12.8 to 16.0 waves/SIMD. +1–2% on small models (0.5B–4B), flat on large models where FFN dominates.
2. #67 — feat(cuda): fuse activations and residual add into mmv f/q epilogues — roberteg16
   • Folds several decode-path (ncols_dst==1) tails into the GEMV epilogue: non-gated silu/sigmoid, silu*mul, and residual add through a view. Launches per token drop 1263 -> 1103, ~+1.4% tg128.
3. #71 — ggml-cuda: keep the concurrent-region join node out of op-fusion — roberteg16
   • Fix for a cudaStreamEndCapture abort on decode with graph-opt on (MoE + shared expert): op-fusion was absorbing the shared-expert join add, so the aux stream was never rejoined. Caps the fusion horizon at the join node while a concurrent region is active. Depends on Port to Visual C++. #36.

Note: Happy to do the front-line review on these conventions, obvious issues, and benchmark coverage.

As mentioned in discussion 3 of 6 I have spoken to @am17an please feel free to start pushing PRs into llama.cpp and reference this discussion, also as you tag this discussion I'll update the PR links to the llama.cpp PRs as they come in for accuracy.

Discussions:
AMD PR Set 1 of 6 — Tooling and Profiling modifications and additions
AMD PR Set 2 of 6 — MMQ and MoE - MatMul Optimizations
AMD PR Set 3 of 6 — Flash Attention Optimization RDNA3.5
AMD PR Set 4 of 6 Matvec and Fusion Decode Optimizations
AMD PR Set 5 of 6 — GEMM Prefill Weight Optimizations
AMD PR Set 6 of 6 - Layout and Concurrency Optimizations

[Geramy]

A small update in general for all discussions, a Tailscale device has been provided to am17an for testing with RDNA3.5 and 4.
AMD also has offered to provide devices for CI pipeline here,

[jimw567](https://github.com/jimw567)
[1 hour ago](https://github.com/ggml-org/llama.cpp/discussions/26377#discussioncomment-17856007)
@Geramy and @am17an Regarding hardware, I have just secured two boards (strix halo and strix point) to be used as GHA runners in the public domain. Can you please let me know who I can work with to add them up to the list of runners in upstream llama.cpp?

I myself can provide another machine through Tailscale with RDNA3.5 and 4 but the 4 is over TB5 in an external razor enclosure and would include a .pem and ssh access with root.

[netrunnereve]

AMD also has offered to provide devices for CI pipeline here

If you guys are hosting the device yourself check out #20154, once you get it working locally we can send you a token to link it to our CI. If you want to provide a physical device then you should probably contact @ggerganov.

Also see the other thread #20446 where others have offered to provide CI runners.

[taronaeo]

AMD also has offered to provide devices for CI pipeline here

FYI I am assisting @jimw567 with the setup of the two boards. Will provide an update once they are online :)

[taronaeo]

PR for the CI runner: #26461

and the latest update: #26461 (comment)

[netrunnereve]

Once that's working and merged we should probably do vulkan as well.

[jimw567]

@taronaeo Thank you so much for staying so late for helping us get the strix halo runner up and running. I just raised a draft PR #26544 that should clear all CI issues with gpu-rocm task. Once that passes, I will raise a PR to merge it into your PR #26461

[am17an]

59. convert : add --fuse-qkv flag to fuse Q/K/V into QKV during HF-to-GGUF conversion #22780 should be preferred, no need to raise a PR for this