Dual AMD Radeon AI PRO R9700 llama.cpp ROCm benchmark review

I built a local LLM box with two AMD Radeon AI PRO R9700 GPUs and could not find much practical benchmark information online, so I ran a small set of llama.cpp ROCm tests and wrote up the results. This is not a perfect lab benchmark; it is a field note for people trying to decide what these cards are useful for in local inference.

The main finding: dual R9700s are very useful for long-context prompt processing, but they do not magically improve single-stream token generation. For interactive chat, one GPU can be just as fast. For long prompts, RAG, document ingestion, or throughput-oriented serving, the second GPU starts to matter.

TL;DR

I tested local GGUF inference on a desktop with 2× AMD Radeon AI PRO R9700 GPUs using ROCm + llama.cpp on Ubuntu. The short version:

• ROCm works on this setup, but the software stack matters.
• For dense models, Qwen3.6 27B Q5_K_M was faster than Gemma 4 31B it Q5_K_M.
• Dual GPUs helped a lot with prompt processing / prefill, especially at longer contexts.
• Dual GPUs did not improve single-stream token generation; generation speed was often the same or slightly slower than one GPU.
• Sparse-active / MoE-like models such as Gemma 4 26B A4B and Qwen3.6 35B A3B UD-Q5_K_M showed much higher throughput, but they should not be compared directly against dense models as if the architectures were the same.
• Practical takeaway: use one GPU when decode latency is the bottleneck and the model fits; use two GPUs when long-context prefill or throughput matters.

Why I ran these benchmarks

I bought/assembled a dual-R9700 local LLM system and found very little practical benchmark information online for these cards. Most local inference discussion is still centered around NVIDIA cards, older AMD cards, or cloud GPUs.

So this is not meant to be a definitive leaderboard. It is a practical report from one real system:

• What works?
• What kind of throughput should I expect?
• Does a second R9700 help?
• Which settings seem sensible for llama.cpp serving?

Test system

Component	Details
CPU	AMD Ryzen 9 7900X, 12 cores / 24 threads
Motherboard	ASRock X870E Taichi Lite
RAM	64 GB system memory
GPUs	2× AMD Radeon AI PRO R9700, gfx1201, ~32 GB VRAM each
iGPU	AMD Radeon Graphics, gfx1036; intentionally avoided for compute
OS	Ubuntu 26.04 LTS
Kernel	Linux 7.0.0-15-generic
ROCm	ROCm 7.2.2 userspace with Ubuntu in-kernel amdgpu
llama.cpp commit	d77599234ea6e498775aeadbce665eece5bd98cd

Important runtime detail: I used HIP_VISIBLE_DEVICES=0 or HIP_VISIBLE_DEVICES=0,1 to avoid accidentally involving the integrated GPU.

Models tested

Dense models:

• Qwen3.6-27B-Q5_K_M.gguf
• gemma-4-31B-it-Q5_K_M.gguf

Sparse-active / MoE-like models:

• Qwen3.6-35B-A3B-UD-Q5_K_M.gguf
• gemma-4-26B-A4B-it-UD-Q5_K_M.gguf

I keep dense and sparse-active models separate below because active-parameter count changes the meaning of throughput numbers.

Methodology

Main benchmark command pattern:

./build-rocm/bin/llama-bench \
  -m <model.gguf> \
  -ngl 99 \
  -sm <none|layer> \
  -p 512,1024,2048,4096,8192,16384,32768 \
  -n 128 \
  -b 2048 \
  -ub 512 \
  -ctk q8_0 \
  -ctv q8_0 \
  -fa 1 \
  -r 3 \
  -o md

Single GPU:

HIP_VISIBLE_DEVICES=0
-sm none

Dual GPU:

HIP_VISIBLE_DEVICES=0,1
-sm layer

Parameter notes:

• -ngl 99: offload all possible layers to GPU.
• -fa 1: flash attention enabled.
• -ctk q8_0 -ctv q8_0: q8_0 KV cache. I chose this for long-context VRAM practicality and comparable runs.
• -b 2048: logical batch size.
• -ub 512: initial microbatch size. I later compared 512 vs 1024.
• -r 3: three repetitions.
• pp*: prompt processing / prefill throughput.
• tg128: generation throughput for 128 generated tokens.

Dense model results

Qwen3.6 27B Q5_K_M

Test	Single R9700 tok/s	Dual R9700 tok/s
pp512	956.41	860.92
pp1024	913.89	1227.66
pp2048	877.56	1432.51
pp4096	846.78	1534.47
pp8192	798.67	1529.63
pp16384	722.98	1425.44
pp32768	611.02	1215.87
tg128	24.85	24.31

Qwen3.6 27B is a good example of the main pattern: dual GPUs roughly doubled long-context prefill performance, but did not help generation.

Gemma 4 31B it Q5_K_M

Test	Single R9700 tok/s	Dual R9700 tok/s
pp512	751.77	695.98
pp1024	703.40	931.71
pp2048	651.97	1049.70
pp4096	616.22	1105.74
pp8192	573.55	1088.55
pp16384	511.90	1000.07
pp32768	425.48	840.01
tg128	21.74	21.18

Gemma 4 31B showed the same broad behavior, but was slower than Qwen3.6 27B in this dense-model comparison.

Dense-model takeaway

For dense GGUF serving on this system, Qwen3.6 27B Q5_K_M looks like the better speed candidate than Gemma 4 31B Q5_K_M.

The second GPU is useful for prompt ingestion, especially long-context workloads. It is not useful for improving single-stream decode speed in these tests.

Sparse-active / MoE-like results

Qwen3.6 35B A3B UD-Q5_K_M

Source: unsloth/Qwen3.6-35B-A3B-GGUF, file Qwen3.6-35B-A3B-UD-Q5_K_M.gguf (~25 GB).

Test	Single R9700 tok/s	Dual R9700 tok/s
pp512	2653.96	2522.14
pp1024	2562.32	3224.92
pp2048	2477.70	3708.46
pp4096	2369.20	3913.99
pp8192	2210.13	3880.54
pp16384	1977.61	3590.30
pp32768	1636.86	3038.47
tg128	77.25	71.88

Gemma 4 26B A4B it UD Q5_K_M

Test	Single R9700 tok/s	Dual R9700 tok/s
pp512	3079.41	2360.63
pp1024	3003.94	3215.09
pp2048	2795.09	4341.08
pp4096	2667.01	4595.05
pp8192	2463.88	4562.45
pp16384	2173.67	4250.05
pp32768	1781.85	3513.11
tg128	74.26	66.56

These numbers are much higher than the dense models, but that is expected for sparse-active architectures. The fair conclusion is not “Gemma 26B beats dense 27B/31B in every way”; it is that sparse-active models deserve their own category and are very interesting for local serving.

Microbatch comparison: -ub 512 vs -ub 1024

I also tested whether a larger microbatch helped.

Qwen3.6 27B dense

GPU mode	ubatch	pp8192	pp32768	tg128
Single R9700	512	805.68	611.26	24.86
Single R9700	1024	831.85	632.91	24.86
Dual R9700	512	1531.27	1215.35	24.15
Dual R9700	1024	1462.58	1207.05	23.97

For Qwen3.6 27B, -ub 1024 helped slightly on one GPU but not on two GPUs. For dual GPU, -ub 512 looked better or equal.

Gemma 4 26B A4B

GPU mode	ubatch	pp8192	pp32768	tg128
Single R9700	512	2467.67	1776.59	74.26
Single R9700	1024	2861.23	1986.70	74.25
Dual R9700	512	4567.68	3507.79	66.68
Dual R9700	1024	4913.87	3807.60	66.56

For Gemma 4 26B A4B, -ub 1024 clearly improved prompt processing and did not materially change generation.

What dual GPUs helped with

The clearest win for two R9700s was long-context prompt processing.

For Qwen3.6 27B:

• pp32768 single GPU: 611.02 tok/s
• pp32768 dual GPU: 1215.87 tok/s

For Gemma 4 31B:

• pp32768 single GPU: 425.48 tok/s
• pp32768 dual GPU: 840.01 tok/s

For Qwen3.6 35B A3B UD-Q5_K_M:

• pp32768 single GPU: 1636.86 tok/s
• pp32768 dual GPU: 3038.47 tok/s

For Gemma 4 26B A4B:

• pp32768 single GPU: 1781.85 tok/s
• pp32768 dual GPU: 3513.11 tok/s

That is the happy story.

The less exciting story is generation:

• Qwen3.6 27B: 24.85 tok/s single vs 24.31 tok/s dual
• Gemma 4 31B: 21.74 tok/s single vs 21.18 tok/s dual
• Qwen3.6 35B A3B: 77.25 tok/s single vs 71.88 tok/s dual
• Gemma 4 26B A4B: 74.26 tok/s single vs 66.56 tok/s dual

For one interactive user, single-GPU decode can be just as good or better. For workloads with long prompts, large documents, RAG, or multi-user serving, dual GPUs become much more interesting.

Practical serving notes

For my current OpenAI-compatible llama.cpp server test, I am using Qwen3.6 27B with roughly 200k context:

HIP_VISIBLE_DEVICES=0,1 ./build-rocm/bin/llama-server \
  -m ~/models/Qwen3.6-27B-Q5_K_M.gguf \
  -a qwen3.6-27b-q5km-rocm-200k \
  -ngl 99 \
  -c 200000 \
  -fa 1 \
  --reasoning off \
  -ctk q8_0 \
  -ctv q8_0 \
  -b 2048 \
  -ub 512 \
  -np 1 \
  -sm layer \
  --host <host> \
  --port 8080 \
  --no-webui

llama.cpp rounded the context to 200192. Flash attention is enabled. The server keeps the model loaded in VRAM until stopped; it does not automatically unload and reload the model in my current setup.

I also disabled reasoning by default for interactive use because the hidden reasoning overhead made answers noticeably slower. It can still be useful for harder questions, but for a daily-driver local model I prefer fast direct answers by default.

Caveats

• These are llama-bench results, not a full application benchmark.
• I still need to run more llama-server API latency tests: time-to-first-token, full response latency, and real prompt workloads.
• q8_0 KV is practical for long context, but f16 KV should be compared for quality and speed before making final serving claims.
• Model quality is not measured here. This post is mostly about throughput and practical setup.
• ROCm support is improving quickly, so results may change with driver, ROCm, and llama.cpp updates.
• Sparse-active and dense model numbers should not be mixed into one simplistic ranking.

My current recommendations

If you have a similar dual-R9700 setup:

1. Use HIP_VISIBLE_DEVICES explicitly so the iGPU does not get involved.
2. Use one GPU for latency-sensitive single-user generation when the model fits.
3. Use two GPUs when long-context prefill is the bottleneck.
4. Keep dense and sparse-active model comparisons separate.
5. Try Qwen3.6 27B Q5_K_M as a dense baseline.
6. Try Gemma 4 26B A4B or Qwen3.6 35B A3B UD-Q5_K_M if sparse-active models fit your use case.
7. Do not assume a second GPU improves decode speed. Test it.

What I still want to test

• llama-server API latency with short, medium, and long prompts.
• Time-to-first-token under OpenAI-compatible API calls.
• f16 KV vs q8_0 KV at 8k and 32k.
• Single-user vs multi-user server behavior.
• More model quality notes, not just speed.