AMD PR Set 5 of 6 — GEMM Prefill Weight Optimizations

[Geramy]

AMD PR Set 5 of 6 — Review and Summarization

Johannes and llama.cpp team, below is a group of related patches from AMD intended to optimize a set of high-priority models on the ROCm backend.
I've grouped these together — from a larger set of 25 patches — to hopefully make your job easier in reviewing them, and I've provided a short summary for each. These PR numbers (#57, #34, etc.) refer to the AMD team's downstream fork at
AMD-Ecosystem/llama.cpp, where this work landed first; the goal is to bring it upstream in reviewable batches.

This fifth group is really one patch: two prefill GEMM weight optimizations that were originally developed separately and have since been folded together onto the refactored gfx11 matmul path.

5. GEMM Prefill Weight Optimization

1. #57 — ggml-cuda: GEMM weight row padding + one-time K-padded f16 dequant for prefill — roberteg16
   • Two prefill GEMM optimizations ported onto the refactored gfx11 matmul path:
     • (a) Pads the float-weight row stride by one cache line when the packed row size is a multiple of 2048B, to break LLC cache-set aliasing. Default on, disable with GGML_CUDA_NO_PAD_WEIGHTS.
     • (b) For large-batch quantized dense GEMMs, dequantizes the weight to a K-padded f16 shadow once at load time and runs hipBLAS instead of MMQ. Opt-in via GGML_PREFILL_DEQUANT.
   • Supersedes benchmarks? #34 and can't compile main #37.

For context

2. #34 — ggml-cuda: pad GEMM weight rows to avoid cache-set aliasing — roberteg16
   • The original row-padding piece. Superseded by Stop keywords #57; closed, unmerged.
3. #37 — ggml-cuda: one-time K-padded f16 dequantization for prefill GEMMs — roberteg16
   • The original runtime dequant-cache piece. Superseded by Stop keywords #57; closed, unmerged.

To be explicit: only #57 should be sent upstream. #34 and #37 have been closed and are only referenced for historical purposes only.

Note: Happy to do the front-line review on these conventions, obvious issues, and benchmark coverage.

As mentioned in discussion 3 of 6 I have spoken to @am17an please feel free to start pushing PRs into llama.cpp and reference this discussion, also as you tag this discussion I'll update the PR links to the llama.cpp PRs as they come in for accuracy.

Discussions:
AMD PR Set 1 of 6 — Tooling and Profiling modifications and additions
AMD PR Set 2 of 6 — MMQ and MoE - MatMul Optimizations
AMD PR Set 3 of 6 — Flash Attention Optimization RDNA3.5
AMD PR Set 4 of 6 Matvec and Fusion Decode Optimizations
AMD PR Set 5 of 6 — GEMM Prefill Weight Optimizations
AMD PR Set 6 of 6 - Layout and Concurrency Optimizations

[Geramy]

A small update in general for all discussions, a Tailscale device has been provided to am17an for testing with RDNA3.5 and 4.
AMD also has offered to provide devices for CI pipeline here,

[jimw567](https://github.com/jimw567)
[1 hour ago](https://github.com/ggml-org/llama.cpp/discussions/26377#discussioncomment-17856007)
@Geramy and @am17an Regarding hardware, I have just secured two boards (strix halo and strix point) to be used as GHA runners in the public domain. Can you please let me know who I can work with to add them up to the list of runners in upstream llama.cpp?

I myself can provide another machine through Tailscale with RDNA3.5 and 4 but the 4 is over TB5 in an external razor enclosure and would include a .pem and ssh access with root.

[roberteg16]

PR created for "AMD-Ecosystem#34" here: #26621