AMD PR Set 1 of 6 — Review and Summarization of Tooling and Profiling modifications and additions

[Geramy]

AMD PR Set 1 of 6 — Review and Summarization

Johannes, below is a group of related patches from AMD intended to optimize a set of high-priority models on the ROCm backend.
I've grouped these together — from a larger set of 25 patches — to hopefully make your job easier in reviewing them, and I've provided a short summary for each.

These PR numbers (#33, #48, etc.) refer to the AMD team's downstream fork at
AMD-Ecosystem/llama.cpp, where this work landed first; the goal is to bring it upstream in reviewable batches.
This first group is tooling: it improves per-op profiling and roofline (bandwidth vs. compute) accounting on the HIP/ROCm backend.
It's all off-by-default and HIP-specific, so it has no effect on other backends or on a default build.

1. Instrumentation and Profiling PRs (Roofline) — HIP-Specific

1. #33 — ggml-cuda: add per-op GPU roofline profiling for the HIP backend — roberteg16
   • Off by default; activated at build time with GGML_HIP_ROOFLINE and at runtime with GGML_ROOFLINE_OUT=<path.json>. This is the base the rest of the group builds on.
2. #48 — roofline: emit raw per-tensor data for DDR byte accounting (single + fused) — roberteg16
   • Stops computing byte totals in-tree; emits raw per-tensor byte-level traces in the format (byte count, storage id, source id) so the consumer owns the exclusion policy. Fixes double-counting of KV-cache writes.
3. #50 — roofline: capture per-launch MoE active-expert count (experts_used) — roberteg16
   • Emits accurate experts_used per launch so the consumer stops over-counting MoE HBM traffic — it was producing impossible readings prior.
4. #66 — roofline: emit per-launch topology edges for graph reconstruction — roberteg16
   • Records per-invocation storage-id identity so the consumer can build the execution graph.
5. #53 — roofline: emit per-expert routing histogram (tokens_per_expert) for MoE launches — roberteg16
   • Elaborates on Reproducability information #50 by adding more MoE-related profiling data. Reuses the same ids read-back, no new device work.
6. #42 — mtmd-cli: reset roofline profiler after warmup prefill — roberteg16
   • Wires the #33 profiler into llama-mtmd-cli so the VLM report is prefill-only (vision encode + text prompt), warmup excluded.

We have noticed that in the past some PRs for profiling hooks have been rejected, so we'd like your guidance on the best way to proceed with this subject of PR.

Note: Happy to do the front-line review on these conventions, obvious issues, and benchmark coverage.

Discussions:
AMD PR Set 1 of 6 — Tooling and Profiling modifications and additions
AMD PR Set 2 of 6 — MMQ and MoE - MatMul Optimizations
AMD PR Set 3 of 6 — Flash Attention Optimization RDNA3.5
AMD PR Set 4 of 6 Matvec and Fusion Decode Optimizations
AMD PR Set 5 of 6 — GEMM Prefill Weight Optimizations
AMD PR Set 6 of 6 - Layout and Concurrency Optimizations

[jeremyfowers]

Thanks for writing this up @Geramy, I hope it will set us up for success and good communications in getting these patches merged.

[am17an]

Shouldn't 1-5 be one PR? i.e. it introduces a HIP profiler?

[jimw567]

@am17an Thanks for your reply. We have an integration branch https://github.com/AMD-Ecosystem/llama.cpp/tree/gfx11 so we can test and benchmark our combined optimizations on about 50 strategical models. The PRs in @Geramy 's original posts refer to PRs to our integration branch. Some of PRs are indeed different phases of a larger feature as you point out. We can certainly combine some of them into a single upstream PR if that makes more sense upstream. Let me see what we can do.

[Geramy]

@jimw567 If you like to tag me or link me I can also pre-review the PR if you want just to give you and your team a hand as well as llama.cpp.

[jimw567]

@Geramy Thanks a lot for your help! These profiling updates have provided tremendous insights on guiding our model optimization effort. Hopefully fellow community developers will also find them useful.

[Geramy]

An continuation discussion 2 of 6 - @jimw567 @jeremyfowers @am17an
#26349