Performance

Performance

Throughput of the ork-driver NPU stack, measured by examples/bench.c (make bench) — an end-to-end GQA transformer, matmul on NPU, ops on CPU. RK3588, governors at performance (CPU + DDR 2112 MHz). The DDR governor matters enormously: a parked DDR clock roughly halves every number here, so all figures below are at DDR 2112 MHz, confirmed via /sys/class/devfreq/dmc/cur_freq during the run.

Measurement caveat (read first). Numbers in earlier revisions of this page were taken with the DDR governor parked and were ~½ the true values; they have been replaced. Always confirm DDR=2112 MHz, and sample NPU + RAM-bandwidth utilization (/sys/kernel/debug/rknpu/load, /sys/class/devfreq/dmc/load) during the run — tok/s alone hid that the bottleneck is idle time, not bandwidth or compute. See tools/telemetry_sample.sh.

Head-to-head vs librkllmrt — same model, same quant, same board

Same-state run (DDR 2112 MHz, both runtimes back-to-back). librkllmrt ran the real Qwen3-1.7B-Base-rk3588-w8a8-v1.2.3.rkllm via a dlopen harness (tools/rknpu_bench, reading RKLLMResult.perf); ork-driver ran make bench at Qwen3-1.7B's config (H=2048, 28 layers, GQA 16/8, HD=128, FFN=6144, vocab=151936). Utilization sampled during each run (tools/telemetry_sample.sh):

runtime	decode	NPU cores (avg)	RAM BW (avg/max)	prefill (M=64)
librkllmrt (closed)	11.5 tok/s	C0/C1/C2 ≈ 48–50% each	48% / 83%	187 tok/s
ork-driver int8, 3-core + full-K (ORK_NPU_MC=3 ORK_FULLK_DEC=1)	11.0 tok/s (~96%)	—	—	67 tok/s
ork-driver int8, 3-core K-split (ORK_NPU_MC=3)	8.6 tok/s (~75%)	C0/C1/C2 ≈ 12% (peak ~52%)	16% / 88%	68 tok/s
ork-driver int8, 1-core	5.0 tok/s (~44%)	C0 44%, C1/C2 0	18% / 42%	64 tok/s

The decode gap is essentially closed: 5.0 → 8.6 → 11.0 tok/s (1-core → 3-core K-split → 3-core + per-core full-K single-submit) = 2.2×, now ~96% of librkllmrt's 11.5. Two stacked levers: (1) multi-core (1.69×, the parallelism the telemetry pointed to), then (2) per-core full-K single submit — once multi-core made decode 97% ork_mm_run with cores ~12% utilized (per-submit overhead bound), cutting each core's ~6 K-slices/matmul to one submit reclaimed the rest. Note the single-submit gave nothing single-core (5.0→5.3); it only pays in combination with multi-core, where submit overhead is the binding constraint. (Dual weight layout, int8, K≤10752; fits IOVA at ~1.7B.) Prefill is unchanged (64 → 67, 1.08×): prefill is gated by the serial CPU ops (attention scores/softmax/RMSNorm), not the matmuls, so speeding up the NPU 3× barely helps until those overlap/move off-CPU (Amdahl). Decode was the idle-bound one and benefited most. Below: why the gap was parallelism in the first place.

2 cores vs 3? Neither saturates — it's a sync-overhead wall, not bandwidth. Per-core telemetry: 2-core = 17% avg util / 3.74 tok/s per core; 3-core = 12% avg / 2.79 tok/s per core. Average RAM BW is identical (~16%) at 2 and 3 cores, so the 3rd core only adds +12% throughput (7.5 → 8.4) at much worse efficiency. Implications: (a) for raw tok/s use all cores; (b) for perf/watt on an SBC, fewer cores are far more efficient per core, and the real win is cutting the per-matmul thread create/join + sync (a persistent thread pool) so cores can be saturated — then 2 saturated cores could rival 3 partial ones at lower power; (c) the optimal core count is matmul-size-dependent (don't split a small-N matmul 3 ways — the split overhead outweighs it), arguing for an adaptive per-matmul core count rather than a fixed ORK_NPU_MC.

The gap is parallelism, and decode is idle-bound — not memory- or compute-bound on either stack. The telemetry is decisive:

• ork-driver uses core 0 only (C1/C2 idle); librkllmrt uses all 3 NPU cores. That ~3× of engines is most of the 2.3× decode / 2.8× prefill gap.
• ork-driver leaves RAM bandwidth ~82% idle (18% avg, 42% peak) and its one NPU core averages 44%. Neither resource is the limit — the core sits idle between submits (CPU ops, the fp32 quant↔int8 round-trip, per-submit DMA bsync cache flushes, dependency stalls). Decode is latency/idle-bound on a single core.
• Even librkllmrt doesn't saturate memory — RAM BW 48% avg, ~80% peak. So there is headroom above 11.5 tok/s: more concurrent engines pulling from DRAM (multi-core NPU, or a heterogeneous CPU+GPU+NPU tensor split) should push aggregate bandwidth — and decode — higher still.

⚠️ Do not call ork-driver decode "memory-bound": at 18% RAM BW it isn't. The hardware reaches 11.5 tok/s; ork-driver reaches 5.0 because a single core can't keep the memory system busy. The cause of any per-core gap that remains after going multi-core is still open — don't claim a cause until ork-driver actually matches librkllmrt. Method note: ork-driver int8 uses a dummy per-tensor quant scale (timing representative, output not validated).

Current defaults (auto-tuner) & validation

The levers below are now automatic — a caller just packs and runs, no env flags:

• Multi-core + full-K int8 decode are the default per-matmul choice. The library picks cores per matmul (≤ a budget, capped so each gets ≥2 N-tiles) and auto-builds the full-K int8 decode layout when it fits the IOMMU (bcreate-success guard → falls back to K-split, never crashes). Result with zero env: decode ~11 tok/s, prefill ~94. Policy via ork_npu_set_core_budget(ctx,n); ORK_NPU_MC / ORK_FULLK_DEC are now debug overrides.
• Persistent NPU worker pool replaced per-matmul pthread_create/join. Decode is unchanged (bandwidth-bound near librkllmrt, not spawn-bound); modest prefill gain + lower overhead.
• Per-channel int8 quant is validated (examples/quant.c): fp32 → per-output-channel weight + per-row activation int8 → matmul → dequant, ~0.55% RMS error vs fp32. The pattern a real engine follows (the bench keeps a dummy scale for pure throughput).

What the numbers mean (and which levers matter)

The decode bottleneck is idle time on a single NPU core, proven by the utilization above (RAM 18%, NPU C0 44%, C1/C2 = 0). So the levers fill idle resources rather than shave bytes:

1. Multi-core NPU — the single biggest lever (librkllmrt's ~2.3× decode comes from using all 3 cores; it raises RAM BW 18%→48%). Earlier notes wrongly said "3-core doesn't help decode (shared memory controller)" — the telemetry refutes that: 3 cores reach 2.3× higher tok/s and 2.6× higher RAM BW, because one core can't issue enough concurrent memory traffic. Risky RE: a naive core_mask flip crashes the kernel — the multi-core path needs subcore_task[] populated per core (see "Open levers"). Helps decode and prefill.
2. Heterogeneous split (CPU+GPU+NPU) — RAM BW peaks at only ~80% even on librkllmrt, so adding concurrent engines (e.g. the Mali GPU via the existing Vulkan path) pulling from DRAM could push aggregate bandwidth past what 3 NPU cores reach alone.
3. Cut single-core idle — the fp32 quant↔int8 round-trip, per-submit DMA bsync flushes, and CPU op time stall the one core between submits. Keeping activations int8/on-device and overlapping CPU ops with the NPU matmul reduce the bubbles.
4. Decode single-submit fast path — NOT a lever. Cutting the per-token submit count 2.5× gave essentially no speedup at DDR 2112 (4.85→5.34 tok/s); the earlier "1.23×" was a parked-DDR artifact. Submit count isn't the bottleneck — idle is. (Experiment + why it's also blocked on memory layout: below.)

Prefill (compute-bound, each weight reused M times) is likewise mostly the multi-core gap (2.8×): librkllmrt's 3 cores vs ork-driver's one.

Decode single-submit experiment — the K ceiling and the IOVA blocker

Idea: decode (M=1) K-splits each matmul into 2–6 submits that accumulate in software, even though the NPU does any in-range K in one submit for M=1. Doing each matmul as one full-K submit per N-slice cuts the token's submit count ~2.5× (540→215).

Result: correct (validated vs CPU), but no real speedup. At DDR 2112 the single-submit path ran 4.85 → 5.34 tok/s vs the K-split baseline — within noise. (An earlier 1.23× "win", 2.34→2.87, was measured with DDR parked, where per-submit overhead loomed larger; at full DDR the telemetry shows decode is idle-bound on one core, not submit-bound, so cutting submits buys ~nothing.) And two findings made it unshippable anyway:

1. The single-submit K ceiling is a raw contraction-length cap: K=10752 exactly, not a tile count. Confirmed by tools/ksubmit_probe (sweeps K, validates each single submit vs CPU). int8 sweep on RK3588, pinned to 32-element steps: K=10752 OK (correct), K=10784 wedges (submit timeout). 10752 = 21×512 = 336 fp16 K-tiles = 168 int8 K-tiles — i.e. the limit is on raw K, dtype-independent (fp16 dies in the same raw band: 8192 OK, 11008 wedge) and N-independent (K=10752 verified OK at N=512/2048/8192, the full N-slice range) — purely a contraction-length cap, not a 256-K-tile count and not denser for int8. (Near but under cbuf_elems/3 = 32768/3 ≈ 10923; 3×10752 = 32256. Not an exact 3K≤cbuf rule — 3×10784 = 32352 also fits cbuf yet wedges. Exact mechanism unconfirmed; the decisive test is whether the ceiling moves on RK3576 with its different cbuf_elems.) This refuted the earlier "256 tiles → int8 reaches K=16384" hypothesis — int8 K=11008 (172 tiles, well under 256) wedges, so it was the raw length, not tiles. Running the probe instead of trusting the hypothesis is what caught it.

Answer to "does rkllm do K>8192 in one submit?": yes, up to ~10752 (both dtypes) — a single op handles K in (8192, ~10752]; beyond ~10880 even the closed runtime must K-split or PC-chain. For decode, K≤8192 is the conservatively-safe validated range and covers every Qwen3-1.7B K-dim (max 6144); models with K>~10752 (very large FFNs) fall back to K-split in ork-driver (correct, more submits).

2. Keeping both layouts resident doubles NPU memory and exhausts the IOMMU IOVA range. The experiment kept the K-split weights (for prefill, M>1) and a full-K layout (for decode), ~2× the NPU-mapped weight bytes. The RK3588 rknpu IOMMU IOVA address space is far smaller than host RAM: int8 28-layer barely fit, but fp16 28-layer overflowed mid-pack — RKNPU: failed to allocate IOVA: -12 … rknpu_gem_get_pages: dma map 25165824 fail (24 MB = Wd's full-K fp16 buffer). So decode-single-submit is only viable with a single weight layout. Two candidate single-layout routes were considered: (a) a full-K layout that prefill slices in place via a weight stride register, or (b) drop K-split entirely (full-K layout, prefill M-tiled into M≤R chunks, each a full-K single submit). The dual-layout version was reverted; the K-split-only library scales to any model size.

3. The stride-register route (a) is closed — there is no per-N-tile weight stride register. Probed with tools/slice_probe (packs B[Kfull,N] full-K, runs a Kp-slice, classifies each N-tile output as PARTIAL = correct slice / FULL / WRONG). Baseline: N-tile 0 PARTIAL, N-tiles 1+ WRONG — the engine reads N-tile 0 contiguously from the base but the next N-tile lands KT_op tiles on, inside N-tile 0's full-K data. Register hunt to fix the stride:

• 0x1030, 0x1034 (weight fetch-size regs) set to their full-K values wedge the submit (the engine tries to fetch Kfull·N and hangs/over-reads).
• 0x1088, 0x107c, 0x1044 to full-K: no effect on the stride (and 0x1044=full didn't make the op sum more K either — it's not a clean pass-count knob).
• 0x1024 ((K-1)<<16 | K) is live but its two K-fields are coupled to both the data length and the addressing — every combination either left N-tiles 1+ wrong or broke N-tile 0; no setting gave all-PARTIAL.

Conclusion: the NPU consumes weights as a contiguous KT_op × N-tiles stream — the buffer must be laid out exactly as the op reads it. (Consistent with the K-split path working: its slice tiles are contiguous.) Decode (N-tile-major full-K order) and prefill K-split (slice-major order) are incompatible orderings for one buffer, so in-place slicing is impossible. The remaining single-layout route is (b): full-K-only with M≤R-chunk prefill (no K-split for K≤10752). Cost: prefill re-reads each matmul's weights ⌈M/R⌉× — a bandwidth hit on the compute-bound prefill that must be measured against the K-split baseline. Probes: tools/slice_probe, tools/ksubmit_probe.

35B MoE (Qwen3.6-35B-A3B) W4A4 — experts belong on the CPU, not the NPU

Method: llama-bench -m qwen3.6-35b-a3b-mixed.gguf -t 4 -p 512 -n 0 -r 2, RK3588, DDR governor performance. W4A4 experts via ORK_QUANT=4 ORK_HADAMARD=1. Fast load from the pre-packed ORK_PERSIST=…w4a4.orkpack (the 22 GB gguf otherwise runtime-packs to int4 every run — the multi-minute cost; the orkpack skips it). Iterate with examples/test_moe_smoke.c (~0.7 s, no model load — packs the real expert shapes, verifies bit-exact vs CPU ref + times the gate/up/down triple) — not the 15-min bench.

Decisive result (pp512):

experts on	tok/s
CPU / NEON (ORK_MOE_NPU unset; NPU still runs attention/dense)	20.99
NPU W4A4 (ORK_MOE_NPU=1, all optimizations)	6.13

CPU experts are 3.4× faster and ≈ the 22 tok/s target. Root cause: a MoE prefill is ~800k tiny per-expert int4 matmul programs (256 experts × top-8 × 40 layers × tiles), and each pays a fixed ~18 µs/program dispatch floor on the fixed-function NPU (regcmd fetch/decode + CNA/CBUF/MAC reconfigure + pipeline fill/drain — not compute (<1 µs), not weight-DMA, not activation feed). NEON has near-zero per-op setup, so it wins on tiny matmuls. This matches the edge-MoE literature (sparse-compute advantage doesn't survive on bandwidth/dispatch-bound edge HW). Neither path saturates DRAM (CPU is NEON-bound, NPU is dispatch-bound) — so DRAM is not the ceiling; there is headroom above 21.

Verdict: for this MoE on RK3588, run experts on CPU/NEON + attention/dense on NPU (the default when ORK_MOE_NPU is unset). NPU-W4A4-experts is a documented not-yet-viable path — worthwhile only for a W4A4 perplexity study or future wider-IOVA / DDR5 silicon where the dispatch floor is a smaller fraction.

Levers tried on the NPU-expert path (for the record):

• HW WEIGHT_REUSE (CNA_CBUF_CON0 0x1040 bit13) on M-group tiles sharing a weight N-tile — WORKS on int4, bit-exact, now default-on (ORK_I4_WREUSE=0 opts out). ~8 % on the expert-triple at M=32 (gate/up ~9 %, down ~1 % — down is output-bound), scaling to ~15 % at M=128 (reuse fraction (NG−1)/NG grows with tokens/expert ≈ ubatch/32). Never used on int4 before (rkllm leaves REUSE=0). Applies within an expert only (can't reuse across N-tiles or experts). 2026-09-06 — the same bit works for int8 matmul under an EXACT, derived predicate: reuse correct ⟺ K × segw ≤ WEIGHT_BANK banks × 32 KB, with correct_fraction = min(1, capacity/(K×segw)) verified at 1.0× (0.0% wrong), 2× (49.8%) and 10.9× (89.3%). The loader streams the whole weight but the banks keep only the last capacity bytes, so the reuse tile reads stale bytes beyond that — deterministic, confined to the reuse tiles, correct columns a contiguous run. Same rule the vendor's conv relies on. Off by default (ORK_MTILE_WR); realising it needs N-tiling to fit the banks plus widening WEIGHT_BANK via 0x1040. See Exp-2026-09-05 Session Findings and the Experiment Log 2026-09-06. See Exp-2026-09-05 Session Findings.
• Nonblock doorbell is the fastest correct completion path — blocking submit skips the int4 writeback drain-verify and miscomputes M>1 (the DPU writeback lags job-done); nonblock's poll waits for the actual landing, overlapped with HW (blocking+verify 768 µs > nonblock 712 µs). Now default.
• Expert-parallel (whole experts one-per-core, the coalesce path) = 1.6× on the matmuls vs per-expert colsplit (which is floor-bound ≈ 1×) — DRAM-contention-capped below the ideal 3×. Still far below CPU.
• DEAD/flat (all measured): submit-coalescing (ioctls were never the wall), SRAM-as-transfer-bus (CPU↔SRAM is uncached ~1.2 GB/s → true cacheable+flush ~3 GB/s, and 956 KiB is too small; the feed isn't the bottleneck — note it IS a confirmed separate port, 104% retention under DRAM saturation, so the kill is size and feed rate, NOT contention; see NPU-on-chip-SRAM), M-pad reduction, and a blocking+single-verify "poll fix". The floor is fixed-function per-program setup — no memory/feed trick touches it (fixed-function NVDLA has no persistent-kernel escape hatch, unlike GPU grouped-GEMM).

Open (rung 3) — the full concurrent hybrid, all engines below the DRAM wall (which has headroom):

• Little cores (A55) — TESTED, no benefit. CPU-experts pp512: -t 4 17.3, -t 6 16.9, -t 8 17.6 (all ±noise, board cool at 32 °C so not thermal). Adding the A55s doesn't help even for the independent MoE expert GEMMs — the per-mul_mat_id threadpool barrier + A55 contention with the NPU-attention driver threads still cap it, as on the dense path. (Note: CPU-experts is run-variable ~17–21 tok/s; the rung-1 20.99±0.00 fresh-reboot figure was optimistically clean. Still ~3× the NPU path either way.)
• NPU async ∥ CPU (ORK_MOE_PATHB_FRAC): run ~23 % of experts on the NPU async stream concurrently with the CPU GEMV. Balance point (NPU 6.13 ∥ CPU 21) projects to ~27 tok/s if the overlap is real.
• Combined ceiling: NPU-attention + A76-experts + A55-experts + NPU-experts, all concurrent, is bounded by DRAM (not yet saturated). Under investigation.

Open levers (priority order)

The decode bottleneck is single-core idle (RAM 18%, NPU C0 44%, C1/C2 = 0), so the levers add parallelism / fill bubbles. Both decode and prefill gaps are dominated by ork-driver using one of three NPU cores.

1. Multi-core NPU — DONE, the lever that closed decode (ORK_NPU_MC=n). Splits a matmul's output columns (N) across cores 0/1/2 (worker thread + per-core buffers per core), giving decode 1.69× and, combined with per-core full-K single-submit, 11.0 tok/s ≈ librkllmrt. Two kernel-crash lessons from getting here (both reproduced; the board has a serial console):
   • Populate all subcore_task[0..2]. core_mask=0x2/0x7 with only the single-core subcore_task[0]={0,1} → NULL deref in rknpu_job_subcore_commit → board Oops. librkllmrt round-robins single-core submits (core_mask 0x1/0x2/0x4) with all three subcore entries set; doing the same fixed it.
   • The per-call RKNPU_ACT_RESET is load-bearing. It recovers the rknpu job-queue state after a wedge, not just cold-start priming. A "fast" probe that RESET once then stormed many short-timeout (wedging) submits concurrently across cores corrupted the queue → hang in rknpu_job_next with IRQs disabled + RCU stall (hard hang, needs power-cycle). Rule: RESET after any wedge before the next submit; never storm concurrent submits over wedging values. (Corrects an earlier note that multi-core "doesn't help decode, shared memory controller" — the telemetry refutes it: 3 cores reach 2.3× decode tok/s and 2.6× RAM BW; one core can't issue enough concurrent memory traffic to saturate the bus.)
2. Heterogeneous CPU+GPU+NPU split — RAM BW peaks at only ~80% even on librkllmrt, so adding the Mali GPU (existing Vulkan path) or CPU as concurrent matmul engines could push aggregate bandwidth past 3 NPU cores alone. See Heterogeneous Serving & Scheduling.
3. Prefill's CPU ops — DONE (pool + flash attention), ~1.6× at fixed clock. Prefill barely moved with NPU multi-core (1.08×) because it's CPU-op bound, not matmul-bound (M=128: NPU matmul ~25%, attention ~47%, other-CPU norm/rope/silu/quant ~28%). Two pieces:
   • Flash-style attention — online softmax, single pass over keys, O(HD) state (no sc[L2]), parallel over heads on a persistent thread pool pinned to the perf cores.
   • Pool the other CPU ops (quant/dequant/silu) on the same pool. Gated to prefill (M≥8); decode (M=1) stays inline so the per-token pool barriers don't regress it. Same-clock isolation (single-thread vs pooled, identical clocks so it's purely the optimization): prefill 59→95 tok/s @M=128 (1.62×), 44→75 @M=256 (1.71×); attention 677→264 ms (M=128). Lessons: (a) a persistent pool is mandatory — per-op pthread_create/join regressed prefill (the spawn cost); (b) pin to the perf cores — on big.LITTLE (RK3588 4×A76 + 4×A55) unpinned threads scatter onto the A55s and the join waits for stragglers (auto-detect: cpufreq max within 90% of top, oversubscribe 2× to hide KV-read latency); (c) CPU governor must be performance (same law as DDR-for-decode; DietPi default throttles; perf_governor pins CPU+DDR on load). (Caveat: absolute tok/s also moved with a CPU 2.3→2.4 GHz overclock (~4%); the 1.6× above is the clock-controlled optimization gain. A RAM OC to 3500 MHz was attempted but did not apply — DDR available_frequencies still tops at 2112.) Remaining: long-prefill O(M²) still grows (M=256 attention 1252 ms even pooled).

Attention-on-NPU — tried, doesn't pay off yet (kept as a documented not-yet-viable result). Score (Q·Kᵀ) and context (softmax·V) are matmuls, so they can run on the NPU. tools/attn_cost de-risked the raw matmul floor and it looked great — 492 ms vs 1252 ms CPU at M=256 (~2.5×). But the integration loses: wiring it into the bench regressed prefill (M=256: 36→24 tok/s with multi-core, 27→20 single-core; NPU-matmul time +4000 ms). The de-risk floor excluded everything that actually costs: (a) 896 tiny per-head matmuls pay per-submit / per-matmul multi-core thread-spawn overhead; (b) attention is fp16 but the model matmuls are int8 → a dtype-mode RESET thrash every layer; (c) the inherent CPU gather/transpose/softmax/scatter to feed the NPU (~1100 ms, ~as much as the matmuls save). Even idealized it's ~1.4×, not 2.5×, at high complexity. Verdict: the CPU flash attention is better; don't move attention to the NPU. (ork_mm_repack, the dynamic-weight re-fill primitive built for it, was reverted with the path.) The remaining long-prefill lever is chunked prefill (engine-level scheduling), not an ork-driver change. 4. Decode single-submit: no gain single-core (5.0→5.3 at full DDR — a parked-DDR artifact made it look like 1.23×), but decisive combined with multi-core (per-core full-K → 8.6→11.0) because there per-submit overhead is the binding constraint. Parked for prefill (M>1): the recipe 0x1040=0xb1 is M=1-only, the M>1 large-K schedule is unsolved, hunting it crashed the board (see lesson above), and prefill is CPU-bound so it wouldn't help anyway.

Method: make bench (sudo ./bench [layers] [decode_tok] [prefill_tok] [f16|i8]). Always confirm the DDR governor is at performance first — a parked DDR clock ~halves decode.