FastFlowLM / Q4NX build for AMD XDNA 2 NPU (63.6 tok/s @ 2–4W on Strix Halo & Point)

[julianmb]

Hi OpenBMB team & community,

We wanted to share an unofficial native port of openbmb/MiniCPM5-2B running entirely on the AMD XDNA 2 NPU (/dev/accel/accel0) via AMD FastFlowLM.

• Weights & Precompiled AIE Kernels: huggingface.co/julianmb/MiniCPM5-2B-NPU2
• Porting Scripts & Quality Eval: github.com/julianmb/npuhalo/tree/main/ports/minicpm5-2b
• Hugging Face Discussion: huggingface.co/openbmb/MiniCPM5-2B/discussions/5

---

Performance on AMD Hardware (Ryzen AI Max+ 395 / Strix Halo & Strix Point)

• Sustained Decode: 63.1 – 63.6 tok/s on XDNA 2 (no GPU/CPU decode required)
• Prefill Speed (TTFT): 81.5 – 128.1 tok/s (~420 ms TTFT)
• Active Power: ~2–4 W (leaves the 45–65W iGPU and 16 CPU cores completely idle/free)
• Footprint: ~1.88 GB in NPU memory (Q4_1 / Q4NX format)
• Accuracy & Reasoning: Tested across multi-step math reasoning, Python bug fixing, and structured JSON output with zero degradation compared to baseline.

---

How We Solved the GQA Firmware Incompatibility

MiniCPM5-2B uses 16 Query heads and 2 Key/Value heads ($16:2 = 8:1$ GQA ratio). AMD FastFlowLM's AIE firmware currently lacks a native $8:1$ kernel for $d_{head}=128$:

1. $4\times$ KV Head Replication: We replicated the 2 KV heads $4\times$ along dimension 0 into 8 KV heads ($16:8 = 2:1$ GQA ratio). Under Grouped Query Attention, this maintains exact bit-for-bit mathematical equivalence while matching the native _gen_mha_seq_d128_q2 AIE kernel.
2. Qwen3 Runtime Engine Routing: Routed execution through FastFlowLM's Qwen3 engine (libqwen3_npu.so) to dynamically dispatch $d_{head}=128$ when intermediate_size == 6144.
3. Identity QK-Norm Injection: Injected synthetic unit RMSNorm tensors across all 42 layers in model.q4nx, making RMSNorm a transparent identity op.

---

How to Run Locally

# 1. Clone the NPU weights
mkdir -p ~/.config/flm/models
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2 ~/.config/flm/models/MiniCPM5-2B-NPU2

# 2. Register under ~/.config/flm/model_list.json
# Add under "models":
# "minicpm5:2b": {
#   "path": "~/.config/flm/models/MiniCPM5-2B-NPU2",
#   "model_type": "qwen3",
#   "tokenizer": "~/.config/flm/models/MiniCPM5-2B-NPU2"
# }

# 3. Serve via FastFlowLM
flm serve minicpm5:2b --host 127.0.0.1 --port 8001

Thank you for releasing MiniCPM5-2B!