Increase llama.cpp performance - strategies for AMD Ryzen AI Max 395+

[geerlingguy]

I've gotten most things working on the Ryzen iGPU, and it's fast enough... but there are a few things that can make it go faster / harder (many mentioned in this Reddit thread on the Ryzen AI Max 395+ for LLM inference):

• NPU Support: Utilizing the 50 TOPS NPU at all
  • It seems like support could come through lemonade-sdk/lemonade, but right now it's Windows only (lol)
  • There's an open request for Ryzen AI NPU support in llama.cpp but probably has to wait on AMD to move the needle.
  • Status of AMD NPU Support on Framework Community Forum
• ROCm:
  • Getting ROCm working with llama.cpp (might need rocmwmma, and a recent patch that allows llama.cpp to be built with -DGGML_HIP_ROCWMMA_FATTN=1 (for flash attention)
  • All that's needed is installing the latest version of ROCm in Fedora Rawhide, and compiling llama.cpp with -DGGML_HIP=ON DAMDGPU_TARGETS="gfx1151" in addition to RPC if needed.
  • See Test ROCm with llama.cpp on Fedora Rawhide #7
• VRAM Allocation:
  • Allocating almost all memory to the iGPU split (using GTT) instead of the ~48 GB or so it seems to get by default. BIOS goes up to 96 GB, but amdttm.pages_limit and amdttm.page_pool_size go further. (See Increasing the VRAM allocation on AMD AI APUs under Linux.
• llama.cpp tweaks:
  • Using --no-mmap and --no-warmup on RPC runs since that can add to the model loading time substantially (and mmap in particular can cause errors when the model filesize is too large to fit in one system's RAM.

Vulkan is already giving good results, but I believe efficiency on this APU can be increased substantially with some tweaks. Especially in a clustered configuration (which I'm investigating with llama.cpp RPC in #2).

[LokiMetaSmith]

This video goes over running large LLM's and various performance options used, on a AMD Strix Halo.
https://www.youtube.com/watch?v=wCBLMXgk3No

It's quite comprehensive, the TL;DR is this repo https://github.com/kyuz0/amd-strix-halo-toolboxes (more in the video description)

[geerlingguy]

There's also a pretty comprehensive guide for optimization over on llm-tracker: llm-tracker: AMD GPUs

[lhl]

Just a couple perf improvements settings I didn't see in your list:

• using the AMDVLK vs Mesa RADV Vulkan driver can increase pp by up to 2X in some cases
• if you're not doing passthrough, then amd_iommu=off should increase memory reads by ~6% (about +2% in llama-bench tg results)
• use tuned and set the active profile to accelerator-performance for about 3% additional raw mbw and a 5-8% increase in pp512 performance
• For ROCm ROCBLAS_USE_HIPBLASLT=1 (telling llama.cpp to try hipBLASlt) can often-times be 2-3X faster than the default rocBLAS kernels

[geerlingguy]

Thanks! I will add those to my list :)

[phush0]

Just a couple perf improvements settings I didn't see in your list:

• using the AMDVLK vs Mesa RADV Vulkan driver can increase pp by up to 2X in some cases
• if you're not doing passthrough, then amd_iommu=off should increase memory reads by ~6% (about +2% in llama-bench tg results)
• use tuned and set the active profile to accelerator-performance for about 3% additional raw mbw and a 5-8% increase in pp512 performance
• For ROCm ROCBLAS_USE_HIPBLASLT=1 (telling llama.cpp to try hipBLASlt) can often-times be 2-3X faster than the default rocBLAS kernels

this is not the case anymore, as for rescent merge there is no such big difference and almost in all cases TG is faster with RADV, also RADV is faster with f16 if this matter at all

[lhl]

Well, point taken that more than usual it's probably best to emphasize that any advice are just guidelines/conditional since there's so much flux and such huge differences between versions/configs.

It probably makes sense for everyone to test for themselves on their own systems, with their exact version numbers, and for their own use cases (the different drivers behave differently with context as well).

Currently on my CachyOS system w/ the latest packages (I don't run mesa-git or however you might get recent merges), I'm still seeing consistent big low-context pp advantages for AMDVLK, but beyond model architecture, they seem to behave significantly differently for both extended context and for using FA or not (remember, no matter how speed is affected, FA will give you the same memory usage savings). One sweep where I have RADV and AMDVLK measured: https://github.com/lhl/strix-halo-testing/tree/main/llm-bench/llama-2-7b.Q4_0

@kyuz0 is doing automated pp512/tg128 runs and since the sweeps I do take hours/days to do and I don't have CI Strix Halo boxes lying around, it's probably best to reference his numbers moving forward if he's committed to keeping them up to date (he's running on an HP Z2 Mini, I believe): https://kyuz0.github.io/amd-strix-halo-toolboxes/

Those numbers can be pretty wild,bt. Even w/ just a pp512 datapoint, you can see at the top on those small Gemmas AMDVLK pulling far ahead w/ fa=0 but way behind with fa=1. For glm air, gpt-oss, qwen, llama you end up w/ more stable advantages for AMDVLK.

[kyuz0]

I have just finished updating the benchmarks with last night's run: https://kyuz0.github.io/amd-strix-halo-toolboxes/.

TL;DR:

• ROCm 7RC (therock-dist-linux-gfx1151-7.0.0rc20250714.tar.gz) + hipBLASLt + ROCWMMA + Flash Attention
  • actually stable and performs better at pp in most cases, even better than AMDVLK.
• Vulkan RADV
  • Best token generation throughput (13/15 wins or ties).
  • Most stable and broadly compatible backend overall.
• Vulkan AMDVLK
  • Competitive in both PP and TG; benefits from margin-aware tie handling.
  • Limited by ≤ 2 GiB single buffer allocation, which can block some model architectures.
• Other ROCm variants (6.4.2 w/o ROCWMMA, 7beta / no hipBLASLt / no Flash Attention)
  • Inconsistent performance and/or higher error rates; best suited for experimental use.

I should be able to keep running these benchmarks, happy to tweak my benchmark script and models I include in the mix so we can get the answers we need.

[geerlingguy]

@kyuz0 - Thanks for the details, your work along with @lhl is helpful for validating different approaches!

[github-actions]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

Please read this blog post to see the reasons why I mark issues as stale.

[iceboundrock]

I created a Github Actions workflow to build llama.cpp with the latest ROCm (7.1.1 for now)
https://github.com/Lychee-Technology/llama-cpp-for-strix-halo

[github-actions]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

Please read this blog post to see the reasons why I mark issues as stale.

[hogeheer499-commits]

One extra data source that may help this issue now that the Strix Halo stack has moved a lot since the early Reddit/llm-tracker snapshots:

I maintain a separate Strix Halo local LLM guide with raw CSV/log evidence for Ryzen AI MAX+ 395 / Radeon 8060S / gfx1151. The parts that seem most relevant to this performance-strategy thread:

• Vulkan/RADV remains the most reliable first path for generation-heavy GGUF inference on the tested Qwen rows.
• ROCm/HIP can still be worth testing for prompt-processing-heavy and high-concurrency server workloads, but it should not be treated as a single universal upgrade over RADV.
• Driver choice matters: AMDVLK can hit allocation/capability cliffs on larger llama.cpp workloads, while RADV has been the stable default in the measured guide runs.
• GTT/TTM sizing, UMA 512 MB, amd_iommu=off, tuned accelerator-performance, exact llama.cpp build, model quant, and command flags are large enough variables that benchmark rows need to carry the full setup.
• I would keep direct llama-bench, Ollama API, llama-server concurrency, RPC, and MTP/speculative decoding results separate; otherwise Strix Halo numbers become hard to compare.

Evidence index:
https://github.com/hogeheer499-commits/strix-halo-guide/blob/main/data/headline_claims.csv

Guide:
https://github.com/hogeheer499-commits/strix-halo-guide

Some current reference points from that guide: Qwen3-Coder 30B-A3B direct llama.cpp Vulkan/RADV at 96.76 t/s balanced UD-Q4_K_XL and 98.51 t/s speed-first Q4_K_S; Qwen3.6 35B-A3B direct Vulkan/RADV at 62.56 t/s balanced and 81.30 t/s speed-first Q4_0; gpt-oss-120b MXFP4 loaded locally at 55.57 t/s; and a separate Qwen3.6 MTP llama-server route around 101.1 t/s, explicitly scoped as speculative/server evidence rather than direct llama-bench.