perf: Ryzen AI 350 (Strix Point) LLM inference appears CPU-bound / not GPU-accelerated by default — slower than a discrete RX 6600 desktop

[itomek]

Summary

A user reports that once past first boot, the default GGUF model (Gemma-4-E4B-it-GGUF) runs noticeably faster on an Intel i5-9400F + discrete Radeon RX 6600 desktop than on an AMD Ryzen 7 AI 350 (Strix Point) laptop. On the desktop the first query was ~8.5 s and warm queries dropped to ~1.2 s; on the Ryzen AI laptop even subsequent queries stayed sluggish. The user's read: the desktop "is utilizing GPU acceleration much better out of the box." This is consistent with the Ryzen AI laptop falling back to CPU inference for the GGUF model while the dGPU desktop gets GPU offload — but it needs measurement on hardware to confirm before asserting a root cause.

A Strix Point part (Radeon 890M iGPU + XDNA2 NPU) should not be dramatically slower than a mid-range dGPU for a 4B model if it is actually using its accelerator. If it is on CPU, that explains the gap.

What to verify (hypotheses)

1. Which backend/device each machine actually uses for the GGUF model — read the Lemonade server.log for the chosen recipe and the llama-server launch flags (notably -ngl / n-gpu-layers and whether the build is Vulkan/ROCm-enabled).

2. Whether GGUF gets any GPU offload on Strix Point. GAIA's device→recipe map only covers OGA recipes:

   # src/gaia/llm/lemonade_manager.py:31
   _RECIPE_BY_DEVICE = {
       "amd_npu":  "oga-hybrid",
       "amd_igpu": "oga-hybrid",
       "amd_dgpu": "oga-hybrid",
       "cpu":      "oga-cpu",
   }
   _DEVICE_PRIORITY = ["amd_npu", "amd_igpu", "amd_dgpu", "cpu"]   # line 39

   The default model is GGUF (llama.cpp), not OGA — so this map does not govern its device, and there is a standing TODO (lines 26-30) noting the recipe vocabulary is unconfirmed. GGUF iGPU offload depends on Lemonade's llama.cpp build + -ngl, which GAIA does not currently set per-device. That mismatch is the prime suspect for the Strix Point laptop running CPU-only.

3. Steady-state throughput on a cache miss. The desktop's ~1.2 s warm number is attributed by the user to an agent cache hit (see feat(ui): preload agent context so first query is as fast as subsequent ones #1223), so it is not a clean inference measurement. Compare tokens/sec on a cache-miss query across machines.

Acceptance criteria

• Documented measurement of tokens/sec (prefill + decode) for Gemma-4-E4B-it-GGUF on (a) a Strix-class Ryzen AI integrated-GPU machine and (b) a discrete Radeon GPU machine, on a cache-miss query.
• Documented determination of the actual backend/device used on each (from Lemonade server.log: recipe + llama-server flags).
• If the Strix Point machine is confirmed CPU-bound: the default load path selects the best available accelerator (Radeon iGPU via Vulkan, or NPU where supported) out of the box, or GAIA passes the correct recipe/llamacpp_args to Lemonade's /load to enable GPU offload for GGUF.
• A user-visible signal of which device is in use (e.g. surfaced in the UI/system info) so CPU fallback is not silent.
• If the gap is (partly) legitimate hardware difference, that is documented with the measured numbers rather than left as a guess.

Test plan (TDD)

Unit (tests/unit/, extend test_lemonade_manager_preload.py / device-selection coverage):

• Device→recipe/args selection: given mocked get_system_info reporting an available amd_igpu (and no dGPU), the resolved load parameters request GPU offload rather than CPU for the GGUF default model.
• Given a CPU-only host, the path resolves to CPU without error (no regression).

Integration (@pytest.mark.integration, require_lemonade):

• Load the default model against a real Lemonade Server and assert the load response / health reports a GPU device when one is available; measure decode tokens/sec over a fixed prompt and assert it clears a sane floor for the device class.

Real-world (manual, AMD Linux hardware; tear down afterward):

• On a Strix-class Ryzen AI integrated-GPU machine: run a fixed benchmark prompt, capture tokens/sec and the Lemonade server.log device/recipe/-ngl, before and after any fix.
• On a discrete Radeon GPU machine (RX-class): same benchmark for comparison.
• Confirm the chosen default uses the accelerator and report the before/after tokens/sec delta in the PR.

Environment

• GAIA Agent UI v0.19.0; model Gemma-4-E4B-it-GGUF
• Slow: AMD Ryzen 7 AI 350 (Strix Point) laptop
• Fast: Intel i5-9400F + discrete Radeon RX 6600 desktop

Related

• Feature Request: Native FLM format and Ryzen AI NPU support via dedicated Agent/Profile #1220 — Native FLM format & Ryzen AI NPU support (NPU acceleration feature track)
• feat(ui): preload agent context so first query is as fast as subsequent ones #1223 — preload agent context so the first query is as fast as subsequent ones (explains the warm-cache 1.2 s number)
• Agent UI first-boot fails to load model: interactive "Delete and re-download?" prompt dead-ends the non-interactive backend #1293 / Lemonade: _is_corrupt_download_error misclassifies generic "llama-server failed to start" as corruption → wrong recovery path + wasteful re-downloads #1294 — the separate first-boot model-load defects from the same report

Hardware/Lemonade device-selection specifics may need maintainer input (per the repo escalation guide, hardware questions route to @kovtcharov-amd).

[itomek]

Pulled benchmark numbers on real AMD hardware to settle the GPU-vs-CPU question. Short version: GPU acceleration is already the default on these machines, and it's measurably faster than CPU on identical hardware.

Methodology

• Model: Gemma-4-E4B-it-GGUF (Q4_K_M), context size 4096, fixed max_tokens=150, same ~50-token prompt every run.
• Measured via the Lemonade API directly — GAIA does not expose per-query token throughput.
• 2 warmup runs discarded, 10 runs measured. Numbers below are decode throughput only (steady-state token generation).

Results

Configuration	Backend	mean ± stdev (tok/s)	min	max
Strix Halo — Radeon 8060S iGPU (~40 CUs)	Vulkan, -ngl 99 (all layers on GPU)	55.2 ± 0.3	54.8	55.6
Strix Halo — same machine, CPU forced	Vulkan binary, -ngl 0 (0 layers offloaded)	23.3 ± 0.1	23.1	23.5
Desktop — Radeon RX 7900 XTX	Vulkan (auto)	136.5 ± 0.1	136.3	136.8

Takeaways

• GPU is the default, not opt-in. On these AMD machines Lemonade only ships the Vulkan build (there is no CPU-only build present), and the live llama-server runs with -ngl 99 — every layer on the GPU.
• GPU vs CPU on identical hardware: 55.2 vs 23.3 tok/s → the 8060S iGPU is ~2.4× faster than CPU compute on the same chip. If you are seeing CPU-like throughput, something has forced a fallback.
• Cold load is separate. The first request after model load pays a significant one-time cost (~60s on the Radeon box in earlier single-shot testing). The 10-run numbers above are all warm — they do not include load time.
• These are decode-only numbers. Time-to-first-token (TTFT) varies independently with prompt length and prefill speed.

A note on scope

The hardware in the original report (Ryzen 7 AI 350 / Radeon 890M, and Intel i5-9400F / RX 6600) was not tested here:

• The Radeon 890M has roughly 1/3 the compute units of the 8060S (~12 CUs vs ~40), so expect proportionally lower throughput — but it should still be GPU-accelerated.
• The RX 6600 is an RDNA2 discrete GPU and should perform well under Vulkan.

Diagnosing your setup

To confirm whether you are actually on the GPU:

pgrep -af llama-server

Look for a vulkan/ backend path and -ngl 99 in the command line. If you see -ngl 0, a CPU backend, or CPU-level throughput, something is forcing fallback. In that case please share your ~/.cache/lemonade/server.log and vulkaninfo output.