Qwen3.6-35B-A3B decode ~3x slower than published benchmarks on Ryzen AI 7 350 / Ubuntu 24.04 — NPU validates at full 51 TOPS

[finerik]

Qwen3.6-35B-A3B decode ~3x slower than published benchmarks on Ryzen AI 7 350 / Ubuntu 24.04 — NPU validates at full 51 TOPS

Summary

On a Ryzen AI 7 350 (Krackan Point) running Ubuntu 24.04 with FLM v0.9.45, flm bench qwen3.6-moe:35b-a3b produces decode speeds roughly 1/3 of the numbers published on the Qwen3.6 benchmark page, which lists the same CPU, the same FLM version (v0.9.45), and the same default Performance power mode.

The same gap appears with gpt-oss:20b (~2x). Prefill is much closer to spec (~1.4x off), while decode is 2-3x off.

xrt-smi validate reports the NPU running at full spec (gemm TOPS: 51.0, PASSED), so the hardware itself appears healthy.

My main question: were the published benchmarks measured on Windows or Linux? The benchmark pages don't state the OS. If they're Windows numbers, this is a platform gap and my results may be expected on Linux. If they're Linux numbers, something else is going on.

Environment

Machine	Morefine M900 (mini PC)
CPU	AMD Ryzen AI 7 350 w/ Radeon 860M
NPU	[1022:17f0] (rev 20) → NPU Krackan 1 / npu6, 8 columns
RAM	64 GB, 2× DDR5, dual channel, Configured Memory Speed 5600 MT/s
OS	Ubuntu 24.04.4 LTS
Kernel	6.17.0-40-generic
BIOS	AMI 2.22.1294, KRK.3xx.MS800.V218.P8C1M0C15.14.Morefine
FLM	v0.9.45
XRT	2.21.75
amdxdna	AMD official DKMS 2.21.260102.53.release (from RAI_1.7.1_Linux_NPU_XRT.zip)
NPU Firmware	1.1.2.64
Firmware loaded	amdnpu/17f0_10/npu_7.sbin
Memlock	infinity

Questions

1. Were the published benchmark numbers measured on Windows or Linux? The benchmark pages list CPU, RAM, and FLM version but not the OS. Given FLM's Linux support landed 2026-03-11, a Windows/Linux gap would explain this.
2. If they were measured on Linux, is there any additional configuration required beyond what I have above?
3. Is the corrupted output at 16k/32k a known issue?

Happy to run any additional diagnostics. Thanks for the great work on FLM.

[huveewomg]

Adding an independent datapoint from Windows + different memory config - same chip, same deficit. This answers your question 1 from both this issue and #629: the deficit reproduces on Windows, so it's not a Linux-specific tax.

Environment

Machine	Lenovo ThinkPad T14s Gen 7 (21TBCTO1WW)
SoC	AMD Ryzen AI 7 PRO 350 (Krackan Point)
RAM	64 GB LPDDR5X-7500, quad-channel
OS	Windows 11 Pro (Build 26200)
NPU driver	32.0.20102.3930 (May 2026)
FLM	v0.9.45 (via Lemonade 11.0)
Power	Plugged in, FLM default Performance pmode

Measurements (warm, idle, single client, engine-reported timings)

Model	decode tok/s	TTFT	vs published
qwen3.6-moe:35b-a3b	4.5–4.9	9.7–10.6s	36–39% of 12.41 @1k
same, solo (only model loaded)	4.9	9.66s	co-residency ruled out
qwen3.5:4b dense	10.1	1.29s	67% of 15.0 @1k

Dense 4B only mildly affected — MoE-specific, consistent with your findings. The ~10s TTFT on a 16–30 token prompt is a fixed per-request cost. For reference #614 measured 3.27s / 12.99 tok/s on the same CPU (Linux, 128 GB).

iGPU control (same model, same machine)

Same weights as GGUF (unsloth UD-Q4_K_XL) via llama.cpp Vulkan on the Radeon 860M:

Backend	decode	TTFT	prefill @720 tok
NPU (FLM 0.9.45)	4.9	~10s	—
iGPU (llama.cpp Vulkan)	8.9	~1s	60.2

iGPU outruns the NPU pipeline 1.8× decode, ~10× TTFT on this box.

Cross-rig pattern

Source	OS	RAM	Result
this comment	Windows	64 GB LPDDR5X ×4	36–39%
#625	Linux	64 GB DDR5-5600 ×2	~33%
#629	Linux	64 GB DDR5-5600 ×2	~50%
#614	Linux	128 GB	at spec
published page	?	96 GB	12.41 @1k

Not OS, not memory type (my LPDDR5X quad-channel = same deficit as your DDR5-5600 dual-channel). Total RAM separates the groups: every 64 GB box is 2–3× slow, the 96/128 GB boxes hit spec. NPU shared pool is typically ~half system RAM, so a ~20 GB MoE in a ~32 GB pool vs 48–64 GB could be hitting a headroom cliff.

Re your #629 hypothesis B (NPU-side effective bandwidth ~halved): my iGPU control shows ~18 GB/s effective weight-read on Vulkan, which is also well under bus limits — so both the NPU and iGPU paths are underperforming on bandwidth, just for different reasons (your per-token host-side cost theory fits the NPU side better given prefill is near-par).

Happy to run debug builds or traces — Windows + LPDDR5X complements your Linux + DDR5 setup.