AMD PR Set 6 of 6 - Layout and Concurrency Optimizations

[Geramy]

AMD PR Set 6 of 6 — Review and Summarization

Johannes, below is a group of related patches from AMD intended to optimize a set of high-priority models on the ROCm backend.
I've grouped these together — from a larger set of 25 patches — to hopefully make your job easier in reviewing them, and I've provided a short summary for each. These PR numbers (#52, #36, etc.) refer to the AMD team's downstream fork at AMD-Ecosystem/llama.cpp, where this work landed first; the goal is to bring it upstream in reviewable batches.

This last group has the following changes, tensor layout, multi-stream concurrency, a fused delta-net kernel, and one default flip.

6. Layout and Concurrency Optimizations

1. #52 — ssm_conv: channels-major input mode to drop the delta-net transpose — roberteg16
   • Adds an opt-in channels-major layout to ggml_ssm_conv so the DeltaNet projection avoids a physical transpose of the conv input every layer. CPU + CUDA; non-implementing backends fall back to CPU, and existing time-major callers stay bit-identical. -29% total prefill at ub4096.
2. #36 — ggml-cuda: overlap the MoE shared expert on a separate stream — roberteg16
   • Decode optimization for MoE models with a gated shared expert, building on ggml-org#16991. Routed experts stay on the main stream; the shared expert forks onto one aux stream. Also carries two correctness fixes to the existing stream concurrency (honor the current stream in ggml_cuda_op_mul_mat, per-(device, stream) cuBLAS handle) and isolates branch scratch into a dedicated buffer. +7.4% tg128.
3. #54 — CUDA: fused chunked gated_delta_net kernel (RDNA3.5) — roberteg16
   • Replaces the token-by-token GDN scan with a chunked delta rule (WY / UT transform) as a single fused kernel: state in registers, chunk intermediates in LDS, no scratch. GDN op 220.9 -> 116.7 ms (1.89x); +4.3% end-to-end at default ubatch, up to +20% at -ub 4096. Scalar gate / K==1 / S_v==128 only, RDNA3.5 only, old kernel untouched.
4. #56 — ggml-cuda: enable GGML_CUDA_GRAPH_OPT by default on RDNA3.5 — roberteg16
   • Flips the graph-opt concurrency default on for RDNA3.5. This is a behavior change and leans on both Longer and infinite output #71 (Set 4 of 6) and Port to Visual C++. #36 above, so it should land last.

The dependency chain here is #36 -> #71 -> #56, and #56 is the only default flip in the whole set of 25, so we'd suggest holding it until the two below it have settled. We'd also welcome your view on #52, since the layout mode touches shared ggml code rather than the HIP backend alone.

Also in flight — already upstreamed

• #49 — CUDA: route single-column f32 mul_mat through mmvf (transpose-free operand swap) — roberteg16
  • A general CUDA optimization, not AMD-specific. Already in the upstream pipeline as ggml-org#26171, so no action needed here — noted only so the set of 25 accounts for it.

Note: Happy to do the front-line review on these conventions, obvious issues, and benchmark coverage.

As mentioned in discussion 3 of 6 I have spoken to @am17an please feel free to start pushing PRs into llama.cpp and reference this discussion, also as you tag this discussion I'll update the PR links to the llama.cpp PRs as they come in for accuracy.

Discussions:
AMD PR Set 1 of 6 — Tooling and Profiling modifications and additions
AMD PR Set 2 of 6 — MMQ and MoE - MatMul Optimizations
AMD PR Set 3 of 6 — Flash Attention Optimization RDNA3.5
AMD PR Set 4 of 6 Matvec and Fusion Decode Optimizations
AMD PR Set 5 of 6 — GEMM Prefill Weight Optimizations
AMD PR Set 6 of 6 - Layout and Concurrency Optimizations

[Geramy]

A small update in general for all discussions, a Tailscale device has been provided to am17an for testing with RDNA3.5 and 4.
AMD also has offered to provide devices for CI pipeline here,

jimw567
1 hour ago

@Geramy and @am17an Regarding hardware, I have just secured two boards (strix halo and strix point) to be used as GHA runners in the public domain. Can you please let me know who I can work with to add them up to the list of runners in upstream llama.cpp?

I myself can provide another machine through Tailscale with RDNA3.5 and 4 but the 4 is over TB5 in an external razor enclosure and would include a .pem and ssh access with root.