RFC: Adding a native AMD ggml-hrx backend

[stellaraccident]

(see draft PR: #27218)

At AMD, we are actively optimizing AMD GPU performance in llama.cpp under the existing Vulkan and HIP (CUDA-derived) backends. However, as ROCm matures on client systems, we have repeatedly run into the costs of not having AMD native backends and optimized client libraries that can be independently optimized for the best possible performance of the GPUs and NPUs on our silicon. In addition, as ROCm was originally developed primarily to service the datacenter market, it often faces headwinds when trying to integrate and deploy to client operating systems in a way that best meets the needs there. As a result, we have been working on a lighter, more focused subset of ROCm that is optimized for these use cases. We call this HRX, and we have been co-developing it with the needs of a dedicated llama.cpp backend for a couple of months, for eventual inclusion into ROCm proper as a focused, redistributable set of libraries.

While the work is still early, we wanted to discuss with the community the possibility of finishing this work and landing it as a new ggml-hrx backend in llama.cpp. While we have done a lot of experimental work to optimize this backend for a wide swath of models and use cases, we decided that for an initial debut and discussion, we would focus the contribution on a minimal kernel library sufficient to run the unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF Q4_K_M model performantly on RDNA3 (W7900) and RDNA3.5 (Strix Halo). This RFC PR carries that minimum set, and we will continue to develop the branch for wide model, operating system, and system coverage.

HRX Technology

The HRX technology that underpins the ggml-hrx backend is optimized for client deployment use cases:

• Small libhrx runtime library (~2MiB) that depends on a minimal set of ROCm system components for interfacing with the device drivers and profiler. Leverages features in our Radeon GPUs to lower per-dispatch overhead to similar levels as is seen in the graphics pipelines.
• Native low-level, explicit, command-buffer based API as the primary API for libhrx for new integrations.
• Optional HIP API compatibility on top of libhrx (not used in ggml-hrx but available for systems looking to migrate incrementally and presently at 88% HIP API compliance).
• Real-time Embedded Loom JIT compiler capable of specializing and optimizing kernels for the exact hardware on a machine to HSACO in ~1-15 milliseconds, putting it in the range of being able to JIT all kernels for all architectures instead of needing to ship pre-compiled kernel libraries.
• No dependency on pre-compiled ROCm kernel libraries, large downloads, etc.
• Builds on our Linux/Windows driver unification work to have the same execution path on both Linux and Windows.

Performance

We are not going to cite absolute performance comparisons on this early work. Our existing published metrics and deployment paths remain our recommended way to get the best performance from llama.cpp on our hardware as this backend continues to evolve.

With that said, the performance benefits of this new technology are real. Compared to the faster between the existing Vulkan and HIP ggml backends, we are routinely achieving on the order of:

• 30-50% tok/s uplift on prefill
• Parity to 15% tok/s uplift on non-MTP decode (heavily dependent on the exact card and memory system behavior with non-MTP being the most overhead sensitive path which we benchmark)

Further, the simplicity and vertical integration of the HRX stack makes exploiting performance advantages of specific cards much easier, compared with attempting to isolate performance improvements in the existing ROCm stack, which often has different historic optimization goals and many more software layers.

As we land more of the work for coverage and features, we will publish absolute benchmark claims. For now, this boilerplate PR was formulated to have the bare minimum to meet functionality and demonstrate single model/use-case performance on Strix Halo and W7900 to show viability without trying to land a full-coverage backend in one patch.

Build and Size

All in, the ggml-hrx backend presently adds ~32MiB on-disk/stripped above a CPU-only llama.cpp backend. It is standalone, requiring no additional software installation beyond AMD GPU drivers. We expect this number to decrease as some of the ROCm driver interface components are optimized for this use case.

On a 16-core/32-thread Ryzen AI MAX+ 395, a clean, uncached Release build with the HRX backend enabled completes in approximately one minute. This includes libhrx, the Loom compiler, ggml-hrx, llama.cpp itself, and kernel bundling for the JIT compiler. We expect that these numbers will grow upward somewhat with full coverage of models and systems, but the delta should be incremental.

At this footprint and build time, ordinary contributors can work across all of our software, directly as part of the final project, without high end build systems or the overheads and distribution complexity traditionally assumed to come with depending on software like ROCm.

Next Steps

We have biased this initial submission to be structurally minimal. As such, once landed, much of the work involves landing pre-work that has already been done for other models and GPUs:

• Integrate with upstream CI and release process
• Land Qwen 3.8 coverage from the side branch (already showing performance uplift)
• Incrementally extend model and GPU coverage to include everything that the current backends support
• Productionize our static Windows build recipe (currently can generate a completely static/self-contained 26.1MiB llama.exe)

Disclosure on AI Use

Per project policies, we are disclosing that significant portions of this project have been built with the assistance of AI. Specific models vary by contributor, but Codex with Sol was most commonly used for the ggml-hrx backend itself. The code was meticulously reviewed, revised and tested by hand in order to produce an implementation that should fit in with the rest of the project and that the contributors can explain and defend in detail. The kernels were heavily tuned and optimized through many rounds of our automated kernel tuning technology, which we will share more about at a later date.

Co-authored-by: Rob Suderman Robert.Suderman@amd.com
Co-authored-by: Ben Vanik Ben.Vanik@amd.com
Co-authored-by: Aryaman Vasishta Aaryaman.Vasishta@amd.com
Co-authored-by: Stella Laurenzo slaurenz@amd.com

[inforithmics]

Is there a document that describes the architecture of the backend? because looking at the code files it seems that the backend optimizes by model architecture. Would be interested to know how it is done without looking through the code.

[stellaraccident]

Is there a document that describes the architecture of the backend? because looking at the code files it seems that the backend optimizes by model architecture. Would be interested to know how it is done without looking through the code.

We have a design doc that we used amongst ourselves for this. Let me get it cleaned up/current and post it today. Most of what is in this initial branch are fusions that should generalize pretty well to adjacent model families (although some of them still carry the "qwen" prefix, which is what they were optimized against). What we left out of this branch was the long-tail, single kernel coverage for all claimed ops. We included a couple of those in this submission to illustrate the point, but these are fairly mechanical/bulky to implement and we opted to save all of them for a followup.

[stellaraccident]

I'm attaching an AI generated document that summarizes our original design doc and provides some of the design thinking and history/rejected alternatives.

hrx-backend-architecture.md

(the summary is basically what I said above)

[inforithmics]

Thanks a Lot, this is very sophisticated/advanced/clever technology.

[Geramy]

On a 16-core/32-thread Ryzen AI MAX+ 395, a clean, uncached Release build with the HRX backend enabled completes in approximately one minute.

That's crazy amazing!

[neuhaus]

Sounds great! Will it also support clustering more than one Strix Halo system?

With Nvidia Spark and Apple being competitors offering RDMA tensor parallelism, this is an important aspect.

[rsuderman]

For those interested in building and running:

git clone --depth 1 --branch users/stella/hrx-rfc-v1 https://github.com/AMD-Ecosystem/llama.cpp

git clone -b main https://github.com/ROCm/hrx-system
cd hrx-system
git checkout 8ef82dbfa0385f7953ceadd884317be7e512ea38
cd ..

mkdir build
cmake -G Ninja -B build -S llama.cpp -DGGML_HRX=ON -DHRX_SOURCE_DIR=hrx-system
cmake --build build

To execute you will need access to a W7900 or AI Max+ 395. You can observe the available device via the llama-cli when built with the HRX backend enabled.

build/bin/llama-cli --list-devices

[inforithmics]

I had to adapt the build so that it took the clang from rocm

git clone --depth 1 --branch users/stella/hrx-rfc-v1 https://github.com/AMD-Ecosystem/llama.cpp

git clone -b main https://github.com/ROCm/hrx-system
cd hrx-system
git checkout 8ef82dbfa0385f7953ceadd884317be7e512ea38
cd ..

mkdir build

export ROCM_PATH=/opt/rocm
export CC="$ROCM_PATH/llvm/bin/clang"
export CXX="$ROCM_PATH/llvm/bin/clang++"

cmake -G Ninja -B build -S llama.cpp \
  -DGGML_HRX=ON \
  -DHRX_SOURCE_DIR=hrx-system \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER="$CC" \
  -DCMAKE_CXX_COMPILER="$CXX" \
  -DIREE_ROCM_PATH="$ROCM_PATH"

cmake --build build

The build took me 1m and 6s. Because I have only a GFX1150 iGPU I cannot test if it would work.

[stellaraccident]

Thanks. We're working on eliminating that requirement: It is currently required to use ROCm's clang to compile device code for certain parts of libhrx (i.e. blit/transfer kernels and such).

[benvanik]

(latest hrx-system main now uses prebuilt binaries so no rocm required)

[ckuethe]

So close, yet so far on 7900XT.

ckuethe@ryzen:~/githubs/llama-hrx$ ./build/bin/llama-cli --list-devices
Available devices:
  HRX0: AMD Radeon RX 7900 XT (Node 1) (gfx1100) (20464 MiB, 20464 MiB free)

ckuethe@ryzen:~/githubs/llama-hrx$ ./build/bin/llama-completion --list-devices
Available devices:
  HRX0: AMD Radeon RX 7900 XT (Node 1) (gfx1100) (20464 MiB, 20464 MiB free)

ckuethe@ryzen:~/githubs/llama-hrx$ ./build/bin/llama-completion -p "Who are you?" -m ~/.cache/huggingface/hub/models--ggml-org--gemma-3-4b-it-GGUF/snapshots/d0976223747697cb51e056d85c532013931fe52e/
gemma-3-4b-it-Q4_K_M.gguf  mmproj-model-f16.gguf      
ckuethe@ryzen:~/githubs/llama-hrx$ ./build/bin/llama-completion -p "Who are you?" -m ~/.cache/huggingface/hub/models--ggml-org--gemma-3-4b-it-GGUF/snapshots/d0976223747697cb51e056d85c532013931fe52e/gemma-3-4b-it-Q4_K_M.gguf 
0.00.790.454 I llama_completion: llama backend init
0.00.790.460 I llama_completion: load the model and apply lora adapter, if any
0.01.109.984 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.03.541.078 E graph_compute: unsupported HRX node 0: GET_ROWS output=2:f32[2560,2,1,1]<-GET_ROWS inputs=[0:q6_K[2560,262144,1,1], 1:i32[2,1,1,1]] consumers=[]
0.03.541.083 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
0.03.541.083 E process_ubatch: failed to compute graph, compute status: -1
0.03.541.084 W decode: removing memory module entries for seq_id = 0, pos = [0, +inf)
0.03.541.162 E llama_decode: failed to decode, ret = -3
0.03.547.433 I llama_completion: llama threadpool init, n_threads = 16
0.03.547.446 I llama_completion: chat template is available, enabling conversation mode (disable it with -no-cnv)
0.03.547.447 W *** User-specified prompt will pre-start conversation, did you mean to set --system-prompt (-sys) instead?
0.03.547.647 I llama_completion: chat template example:
<start_of_turn>user
You are a helpful assistant

Hello<end_of_turn>
<start_of_turn>model
Hi there<end_of_turn>
<start_of_turn>user
How are you?<end_of_turn>
<start_of_turn>model

0.03.547.649 I 
0.03.547.690 I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.03.547.691 I 
0.03.547.854 I llama_completion: interactive mode on.
0.03.547.857 I sampler seed: 1669429659
0.03.547.863 I sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = -1
	top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
0.03.547.868 I sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
0.03.547.868 I generate: n_ctx = 131072, n_batch = 2048, n_predict = -1, n_keep = 1
0.03.547.868 I 
0.03.547.869 I == Running in interactive mode. ==
0.03.547.869 I  - Press Ctrl+C to interject at any time.
0.03.547.869 I  - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.
0.03.547.869 I  - Not using system message. To change it, set a different value via -sys PROMPT
0.03.547.869 I 
user
Who are you?
model
0.03.548.442 E graph_compute: unsupported HRX node 0: GET_ROWS output=2:f32[2560,13,1,1]<-GET_ROWS inputs=[0:q6_K[2560,262144,1,1], 1:i32[13,1,1,1]] consumers=[]
0.03.548.444 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
0.03.548.445 E process_ubatch: failed to compute graph, compute status: -1
0.03.548.445 W decode: removing memory module entries for seq_id = 0, pos = [0, +inf)
0.03.548.515 E llama_decode: failed to decode, ret = -3
0.03.548.517 E cmn  common_promp: failed to eval

[rsuderman]

We are still working on expanding model coverage. The initial version was mostly a proof of concept running on qwen 30b whereas it looks like you are using gemma 3. We should have more support landing as we widen op coverage. Will update once I see it landing.

[rsuderman]

We have a newer version with support for llama 3 across multiple quantization formats and f16 here.
https://github.com/AMD-Ecosystem/llama.cpp/tree/hrx-graph-develop-v2

This is more generalized so we should have a good cadence for increasing model coverage while moving away from the qwen focused implementation.

[ckuethe]

I tried the binary build on my RX7900 and it's working well with gemma3 (>130t/s) and gpt-oss-20b (>140t/s)

[kheiman61]

Tried it out on my Strix Halo 395+ / 128GB. Based on some quick testing with my local coding harness, can confirm that this is definitely on the right track. The results also matched what I observed as a test just summarizing paragraphs with each directly in llama.cpp.

Great stuff!

HRX vs HIP — Qwen3-30B-A3B-Instruct-2507 Q4_K_M, gfx1151 (Radeon 8060S / Strix Halo)

Both backends built from llama.cpp 33a1f2b23 (b10224), running ROCm 10.1 + HSA 1.21.
Mean of 5 interleaved runs on an otherwise idle machine.

Test	HRX (t/s)	HIP (t/s)	HRX vs HIP
pp16	218.6 ±0.6	304.2 ±7.6	−28.1%
pp32	351.7 ±2.5	416.7 ±13.0	−15.6%
pp64	568.2 ±1.9	412.7 ±33.6	+37.7%
pp128	912.8 ±6.4	408.7 ±11.2	+123.3%
pp256	1172.5 ±66.4	636.6 ±6.5	+84.2%
pp512	1261.8 ±160.1	937.5 ±9.8	+34.6%
pp1024	1592.1 ±19.8	1319.3 ±21.8	+20.7%
tg128	85.1 ±0.7	68.0 ±0.3	+25.1%
tg512	80.6 ±0.2	66.3 ±0.2	+21.5%
tg128 @ depth 1024	78.2 ±0.3	64.6 ±0.5	+21.0%

Notes

• HIP is ahead below ~32 tokens (per-dispatch overhead dominates); crossover sits between 32 and 64.
• HRX's pp512 spread (±160) is unusually wide — treat +34.6% as approximate.
• Short-prompt rows are warm-up sensitive: measured standalone with per-size warm-up, HIP reached 273 t/s at pp16 vs HRX's stable 218. hipBLASt autotuning needs repetition; HRX is steady from the first run.
• pp1024 uses matched -b/-ub — HRX cannot chunk prefill against a non-empty KV cache. Throughput regresses above 1024, so measurements stop there.

[John-Gee]

I really appreciate the smaller size, I always disliked the waste from the huge ROCm install, and I look forward to seeing how it compares on performance to https://github.com/warpfront/hipfire in the future.