Optimization Roadmap

Performance Optimization Roadmap

STATUS (2026-06-13): performance target met. On Qwen3-1.7B-w8a8 at -t 4, the open stack (ork-driver → ggml-ork → llama.cpp) reaches prefill 178 tok/s = 97% of librkllmrt (184) and decode 12.8 = 109% (beats it). Got there via big-core thread count + full-K prefill + affinity pinning. Tier 2 (decode submit) closed — no headroom. Tier 3 (architectural) not viable on this kernel. 1c-i zero-copy is implemented+measured at ~0 gain (closed). Tier 4 — int4/W4A4 is not yet viable (real-model 2026-06-24: no config is both fast and coherent — see below); w8a8 is the production path today. int4 stays an open research track pending a stronger rotation/quant approach. Tier 8 — mixed int8/int4-NF4 + imatrix quant is DONE & PPL-validated (fp16-source mixed quant = 11.33 PPL @ ~58% int8 storage), and Tier 9 — the streamed-prefill dma_buf fill chain (cacheable + direct-inflate DONE, clean-only-bsync NEXT) feeds the phase-aware saturation architecture.

Next-steps for closing the open-vs-closed gap, derived from the per-matmul apples-to-apples vs RKNN. The gap splits cleanly into two ork-driver kernel problems (not integration / framework):

Phase	Gap vs closed	Root cause	Tier
Prefill (M≫1)	~3.2×	large-M multi-core barely parallelizes (6% from 3 cores)	Tier 1
Decode (M=1)	~1.24×	per-matmul submit kernel efficiency (~100 µs/matmul)	Tier 2

Every step is validated with the standard (Benchmark Standards): rknn_vs_ork for matmul-level, llama-bench vs rknpu_bench for end-to-end. Status: ⬜ not started · 🔬 in progress · ✅ done · ❌ ruled out.

Tier 1 — Prefill: fix large-M multi-core (biggest, clearest)

The anomaly: 3-core scaling is backwards — decode 2048² (M=1) scales 1.76× across 3 cores, but prefill 2048² (M=512) scales only 1.06×. Large M should parallelize better. So ork isn't actually splitting the work across cores at large M.

✅ 1a. Diagnose. Done — root cause was CPU affinity: the multi-core workers ran on the A55 little cores (~2× slower on every phase), not a tiling/barrier issue. Fixed by pinning NPU-driver threads to big cores (ork-driver 34b539e). mc_prof 3-core prefill scaling 1.11× → 1.92×; int4 prefill +57%.
✅ 1b. Why does int8 prefill stay flat end-to-end? RESOLVED — not the NPU matmul. (Thread count & big-core scheduling.) The Bf/budget-3 hypothesis was wrong (ruled out via ORK_FULLK_DEC=0); the earlier "budget-3 scales 1.1×" was a stale /tmp/rknn_vs_ork binary (pre-affinity). The real cause: with -t 8, llama.cpp's CPU threadpool spins on all cores and starves the ork NPU-driver threads (integration matmul 7596 vs 4365 µs standalone), and the 4 little A55 cores drag the threadpool barrier. -t 4 (big cluster) → prefill 136→155, decode 9.2→12.8 (beats librkllmrt 11.71).
🔬 1c. ork's single-core large-M matmul kernel is 3.3× slower than RKNN's per core (8427 vs 2573 µs at M=512) — the remaining prefill gap (155 vs 184). Diagnosed (mc_prof single-core breakdown, ork-driver 609b46f): of 8620 µs / 4 submits — NPU submit 64% (5491 µs), host accumulate 21% (1793 µs, doubled by the Sk=2 K-split), activation copy 10%. Two sub-levers:
- ⬜ 1c-i — zero-copy buffers only (DEFERRED). Native-A RULED OUT; native-B already used. Prior-art dig + direct measurement closed both layout levers: ork already uses RKNN's native B (N/32,K/32,32,32); and native A is 14% slower at the prefill shape — measured risk-free on RKNN itself (rknn_vs_ork <n> a: normal 2801µs vs native 3192µs; the clehaxze.tw +20% was fp16 and doesn't transfer to int8 large-M). So the only residual lever is host copy/writeout → zero-copy. BUILT (ork_dma_alloc, 39d5369): NPU-coherent CPU-mapped buffers + a registry; the matmul auto-detects DMA-resident A/C and reads/writes them in place. Results (full-K 1-core M=512):
- Input zero-copy (A in DMA): VALIDATED + default on, −17% (4754→3967 µs → 1.54× of RKNN, from 1.9×). Correct vs the fp32 ref (quant ORK_TEST_DMA).
- Output zero-copy (C in DMA): FIXED (3fad74a) — it was DMA cache coherency, not an address register (my "2nd register" guess was wrong): the output buffer's dirty CPU lines evicted over the NPU's writes. Fix = bsync(cbuf, TO_DEVICE) before + bsync(cbuf, FROM_DEVICE) after. Validated ALL OK. Opt-in ORK_ZC_OUT. Full input+output ≈ −33% (ceiling, 1.26× of RKNN).
- End-to-end (Stage 2): ⛔ IMPLEMENTED + measured → ~0 gain, CLOSED. Built the DMA is_host buffer type on feature/zerocopy-buft (llama.cpp-rockchip, not merged) and measured (ORK_BUFPROBE): it does place the matmul output in ork DMA memory (dst=ORK-DMA, coherent), but the input (src1, a CPU-op output) stays CPU. Output zero-copy end-to-end (single-core + ORK_ZC_OUT) fires + coherent but pp512 unchanged (143.11→142.85) — the writeout it saves is negligible vs end-to-end prefill, and input zero-copy can't fire without a GPU device type (which cross-backend-copies inputs anyway + risks decode). Not "deferred" — implemented and measured at no gain. The matmul-level capability ships in main; the integration buffer type is the documented no-gain result on the feature branch (parked, revisitable with a future GPU-input device type).
- ✅ 1c-ii DONE — full-K prefill (writeup, ork-driver c1bb9cc, default on). One submit/M-tile over full K (M-scheduler), no K-split accumulate. Validated correct vs CPU ref (power-of-2 K; non-pow2 falls back). Single-core matmul 8567→4763 µs (1.8×); end-to-end pp512 -t 4 156→178 (+14%) = 97% of librkllmrt. Bigger than the ~15% estimate (the result cache-sync also halved).
⚖️ 1d. Auto big-core scheduling. Tried forcing process affinity to the big cluster at init → ruled out (ork-driver b9a5c4e): doesn't help -t 4, craters -t 8 decode (9.3→2.3, oversubscription). For now the recommendation is run -t <big-core-count> (a serving/CLI choice). The real automatic fix — make the CPU threadpool sleep (not spin) during NPU matmuls + size it to the big cluster — belongs in the llama.cpp threadpool / serving layer, not a driver-side affinity hammer. Deferred.
Validate: rknn_vs_ork M=512 (close the 3.3× single-core kernel gap), then llama-bench pp512 -t 4 vs rknpu_bench (155 → toward 184).

Tier 2 — Decode: trim the per-matmul submit ✅ CLOSED (no headroom)

✅ 2a. Teardown (done, ork-driver 0a25292). Split the M=1 decode submit (3-core Bf fast path, 2048²): regcmd-prep 3.3 µs (2%) / ioctl + IRQ-wait + result-sync 154 µs (98%) / writeout 0.9 µs (1%). The per-matmul decode submit is ~all irreducible NPU ioctl/IRQ round-trip floor.
❌ 2b–2d ruled out by 2a: resident regcmd saves only ~3 µs; result cache-sync is already tiny (~KB for M=1); poll-vs-IRQ could shave only the IRQ wakeup (tens of µs of 150) at the cost of a busy-spun core. Not worth it.
Bottom line: ork's per-matmul decode is already at RKNN parity (1.02× at 2048², matmul apples-to-apples) and decode already beats librkllmrt end-to-end (12.8 vs 11.71 at -t 4). Nothing to trim — Tier 2 is closed.

Tier 3 — Architectural ⚖️ NOT VIABLE / NOT NEEDED (feasibility 2026-06-13)

Tier 3's stated goal was "the only path to beat rkllm decode." That's already achieved by Tiers 1–2 (decode 12.8 vs 11.71 at -t 4), so the motivation is gone — and both items are blocked or net-negative:

❌ 3a. Persistent per-layer program — BLOCKED. Needs either (a) multiple matmul programs per RKNPU_SUBMIT, or (b) intermediates kept on-NPU across the layer. (a) times out — the kernel rejects task_number>1 (re-confirmed 2026-06-13, ntask 2 & 4 → rc=-1; see Task-batching). (b) is impossible: a layer interleaves matmuls with non-matmul ops (attention/softmax, RoPE, norm, SwiGLU) the NPU can't do, so intermediates must round-trip to the CPU. The only adjacent same-input matmuls (QKV, gate+up) are exactly what fusion tried → neutral/slower.
❌ 3b. Async submit + CPU/NPU overlap — net-negative. All-NPU decode = 213 ms/token (NPU ~72 ms + CPU/scheduler ~141 ms); perfect overlap → max(72,141) = 141 ms → ~7 tok/s, still worse than the shipped hybrid CPU decode (12.8). And it needs ggml's per-node-synchronous scheduler reworked (llama.cpp layer). Best case loses to what we already ship.

Verdict: no Tier 3 work pursued — the architecture can't amortize the submit floor on this kernel, and the hybrid (NPU prefill + big-core CPU decode) already beats the closed runtime on decode and hits 97% on prefill. The one remaining theoretical lever is 1c-i (zero-copy DMA buffers) for the last ~3% of prefill — moderate, deferred (both layout levers are closed: ork already uses native B, and native A measured 14% slower at the prefill shape).

Tier 4 — int4 / W4A4 (open; native Multi-M scheduler discovered & validated ✅)

int4 W4A4 is now fully optimized for native multi-row M execution, resolving the slow prefill bottleneck.

🟨 4a. Hadamard rotation (accuracy — prerequisite). W4A4 output is fluent-but-incoherent (~9.5% matmul error floor from outliers). A Hadamard/random-rotation pre-transform on the activations + weights flattens outliers so 4-bit quant is accurate (QuaRot/SpinQuant-style). Mechanism confirmed by a CPU probe (Hadamard int4 accuracy, tools/hadamard_int4.c + hadamard_real.c): cuts W4A4 matmul RMS 40–60% on both synthetic and real GGUF weights, the benefit scales with outlier severity (the QuaRot signature), and the error is dominated by the activations (W4A8 would be far better but the NPU can't do it). But matmul RMS is a diagnostic — int4 error is unbiased and averages out (why QuaRot int4 LLMs work), so the go/no-go is end-to-end perplexity from a wired-in path, not yet built. Sequenced behind 4b (below).
- ✅ Now validated END-TO-END ON THE NPU (2026-06-24, tools/hadamard_i4_npu.c, ork-driver 1654064). A per-group (block) Hadamard of size G is exactly rotation-invariant within each group, so it composes with the public grouped-int4 path (ork_mm_pack_i4_grouped / ork_mm_run_i4_grouped) without changing the matmul result — the caller rotates each length-G slice of A and B with the exported ork_fwht_norm before int4-quantizing, and the NPU still computes A·B. On RK3588 (M=8 K=2048 N=256 G=128, activation outliers ×20): NPU mechanism exact (err 1e-4), quant RMS vs fp32 27.18% → 14.75% (46% reduction). This de-risks the runtime mechanics fully (rotation is caller-side; no new NPU code needed).
- ✅ END-TO-END PERPLEXITY signal (tiny model, 2026-06-24, examples/perplexity_i4.c on stories15M.bin). Full transformer forward as W4A4 on the NPU (per-group quant; full-K_pad Hadamard via ork_fwht_norm; NO_HADAMARD=1 toggles), NLL vs the fp32-argmax on a greedy sequence (divergence proxy). Hadamard lowered it 32 tok 1.693→1.463, 64 tok 2.119→1.800 (−15%) — promising, BUT this is a tiny clean model and the synthetic on-NPU RMS probe used ×20 injected outliers. Both overstated the real-model benefit (see below).
- ⏳ REAL-MODEL: NOT YET VIABLE (2026-06-24, Qwen3-1.7B-Q8 + wikitext-2 through ggml-ork, -c 128). Wired the per-group Hadamard into ggml-ork's grouped int4 handler + added a per-channel/grouped × plain/Hadamard 2×2 (also fixed a bug: ORK_QUANT=4 was inverted so int4 never actually engaged via the documented flag → the old "incoherent" label was an artifact). True wikitext PPL (CPU-Q8 baseline 19.66):

	plain	+Hadamard	submits/matmul
grouped G=128	23.92 ✓	33.74	16 (slow)
per-channel	24758 💀	103.79	1 (fast)

**Hadamard's value is granularity-dependent and the rotation works as designed** — it rescues the coarse per-channel path ~240× (24758→104, the QuaRot effect) but *hurts* the fine grouped path (24→34, it just mixes already-well-conditioned groups). **As of today no config is both fast and coherent:** per-channel (1 submit) tops out at PPL 104 ≈ 5× baseline even with this block-Hadamard (4-bit struggles to span a 2048-wide row); grouped (PPL 24 ≈ baseline) needs ~16 submits/matmul = **~16× int8's submit count**, so it loses to int8 which is already coherent AND fast. So **w8a8 is the production path today and int4-on-NPU stays an open research track** — not yet viable, but actively open (a breakthrough rotation could flip it). **What would flip it:** a stronger rotation (SpinQuant / learned / paired K-and-hidden Hadamard) that makes *per-channel* int4 coherent → 1 submit/matmul + accuracy, beating int8; or a HW/layout trick that folds per-group scales into one submit. The ggml-ork int4 path is now correct + measurable (flags `ORK_QUANT=4`, `ORK_HADAMARD=1`, `ORK_PERCHAN=1`) so any future rotation experiment can be A/B'd end-to-end immediately. The Tier 4b "within ~2× of int8" bar is the target to clear.

✅ 4b. Submit-count reduction (RESOLVED & PROVEN ✅ — Discovery of INT4 Multi-M, tools/i4_multim_fuzz.c + examples/test_layouts.c). Native multi-M execution is fully supported on the RK3588 INT4 physical path. By leaving the PPU stride 0x405c register at 0 (disabled) and setting the M-count register setting to mc = 2 * H, the NPU natively computes $H$ rows in a single hardware submit, writing them with a stride-2 mapping (Ref Row m to Row 2 * m). This completely eliminates the PC-chaining fallback, unlocking optimal prefill speeds for INT4.
- Per-channel int4 (one scale/output channel, like int8) instead of per-group → full-K single submit — done in the driver (ork_mm_run_i4); confirmed to kill the K/G explosion but capped at ~2× int8 by the M=1 regcmd + intrinsic per-submit cost (above). Coarser quant — needs 4a (Hadamard) for coherence.
- Graph fusion (QKV, gate+up) — applies to int4 even though it was neutral for int8: int8 is work-bound (fusion broke even), int4 is submit-bound, so fusing 3→1 (QKV) / 2→1 (gate+up) cuts the submit count where it actually dominates. Reuses the fusion code (ORK_FUSE), already correct.
- int4-stored / int8-computed (zero-extend/dequant int4 weights to int8, run the fast int8 path) — keeps the 4-bit download/storage benefit at int8 runtime speed; drops the runtime-memory saving (irrelevant on a submit-bound NPU). The pragmatic fallback if per-channel int4 accuracy is insufficient.
- NOT viable: packing two int4 matmuls into one int8 MAC — a8·b8 of nibble-packed operands produces cross-terms (a_hi·b_lo + a_lo·b_hi) that contaminate the result; and it would target MAC throughput, which isn't the bottleneck here.
Validate: examples/quant (correctness, RMS) + rknn_vs_ork/mc_prof (submit count, perf) + end-to-end coherence. Goal: int4 coherent AND within ~2× of int8 (not 30×).
Sequencing (2026-06-13): 4b gates 4a. With 4a's mechanism now de-risked, the open question is speed: int4 is only worth the Hadamard-pipeline integration if 4b gets it within ~2× of int8. Do 4b first (per-channel int4 → full-K single submit, the cheapest big win), measure, and only then invest in wiring Hadamard + a perplexity eval to settle 4a's go/no-go.

Tier 5 — MoE expert offload (MUL_MAT_ID) ⚖️ not yet viable, but closing (now ~7× under CPU, was ~80×)

⚠️ SUPERSEDED (2026-08-19) — the "closing" framing did not resolve the way this section expects. MoE experts belong on the CPU, not the NPU: NF4-quantized experts computed with a batched 4×4 NEON GEMM (weights from the .orkpack) now beat native ggml 1.59× at prefill (36.9 vs 23.3 t/s, Qwen3.6-35B-A3B), with attention/dense int8 on the NPU. Experts-on-NPU is closed by a hard fork, not a gap that narrows: int8-inflated experts are ~35 GB ≫ the 4 GiB IOVA window, while int4-native fits but its kernel measures ~54× the W8A8-inflate path. See Tier 14 and the fork roadmap §4c.

Target: Qwen3-35B-A3B (Q4_K, 128 experts top-8). Built the full path end-to-end — MUL_MAT_ID handler (ggml_backend_ork_mul_mat_id_i8, group-tokens-by-expert), a fixed resident expert pool reused via ork_mm_repack_i8 (no IOMMU churn), and the repack-buffer exclusion to route experts to ork. It works (offload is correct, 0 fails) but is a NET LOSS — see MoE Expert Offload. All gated OFF by default behind ORK_MOE_NPU=1.

phase	CPU (default)	NPU experts (`ORK_MOE_NPU=1`)
prefill `pp128` — group-by-expert (2026-06-23)	22.59 t/s	~0.28 t/s	~80× slower
prefill `pp128` — + chaining (2026-06-24)	22.59 t/s	0.97 t/s	~23× slower
prefill `pp128` — + NEON repack (2026-06-24)	22.87 t/s	2.90 t/s	~7.9× slower
prefill `pp128` — + fused Q4_K→int8 dequant (2026-06-24, current)	22.58 t/s	3.13 t/s	~7.2× slower
decode `tg` (group-by-expert)	~6.3 t/s	~0.11 t/s	~60× slower

(All Qwen3.6-35B-A3B Q4_K, RK3588, -t 4. Benchmark gotcha: ORK_MOE_NPU=0 still enables NPU MoE — the gate is a getenv-presence check, not a value check; unset the var for a CPU baseline, or use ORK_OFF=1 to force all-CPU.)

❌ 5a. Per-token / group-by-expert offload — RULED OUT. The submit count is bounded by active experts, not tokens: a forward touches ≈ all 128 experts × 3 projections × 48 layers ≈ 18k expert-matmul submits per batch, roughly batch-size-independent once every expert is hit (~pp64). At the ~365 µs submit floor that's ~6.7 s of pure submit overhead per batch, before compute — plus Q4_K→int8 repack churn. Grouping cut submits ~8× vs per-token (147k→18k for pp128) but 18k is still far too many. Decode is bandwidth-bound on top of that.
⚖️ 5b. Chain experts within a layer — BUILT & MEASURED (2026-06-24): 3.5× but still a net loss. Implemented end-to-end: ork-driver run_chain_i8 now chains the K=2048 experts via their Bf full-K buffer (no Sk==1 requirement) and auto-M-tiles each task (commits 1bfeed3, 3f5ae01; validated bit-exact in test_matmul, mism=0); ggml-ork's mul_mat_id handler chains all active experts of a projection into ONE submit (ORK_MOE_NPU=1). Result: pp128 0.28 → 0.97 t/s (3.5×), 0 failures — but still ~23× below the 22.59 CPU baseline. Chaining killed the submit floor as intended (18k → 144 ioctls/forward), but the bottleneck just moved. Profiled (ORK_PROFILE phase timers): pack/repack 94% (our f32→int8 quant+tile loop 69%, ggml Q4_K→f32 dequant only 12%, pack_i8 tiling ~19%), NPU chain-submit just 6%. So the repack is CPU-fixable (it's our cache-hostile strided transpose store, not the dequant) — but even a free repack still loses: the single-core NPU expert-chains alone take ~9.5 s/forward while the CPU does the entire prefill in 5.7 s (128/22.59). The 6% NPU floor is itself above CPU; eliminating the repack lands MoE-on-NPU at best ~10–15 t/s, cross-core (~3×) reaches parity not a win. int4 doesn't change this: it makes the repack ~2× cheaper (4-bit output) and helps residency, but still requires a Q4_K→NPU-int4 dequant+requant (not the same format), can't keep all 18k experts resident (IOMMU ~2067 int8 / ~4000 int4 ≪ 18k), and can't touch the NPU-floor-above-CPU problem. Verdict at the time (since revised — see 5c): repack-bound; even a free repack looked unable to win. The chaining primitive (Steps A/B) is a general win on main regardless.
⚖️ 5c. Faster repack — NEON quant+tile, then fused dequant (2026-06-24): closed the gap from ~23× to ~7.2×. The "even a free repack still loses" claim in 5b was based on the OLD cache-hostile strided-transpose repack and is now obsolete. Two levers landed: (i) NEON-fused ork_mm_pack_i8_f32/repack_i8_f32 (n-major quant+tile in one pass, no strided store) cut the repack ~4.4× → pp128 0.97 → 2.90 t/s (~7.9× under CPU); (ii) fused Q4_K→int8 dequant (ork_mm_pack_i8_dequant/repack_i8_dequant, ork-driver b520ee7; per-channel ork_dequant_row_fn callback dequants one output channel into a reused K-scratch + NEON quant+tile in place, so the full f32[N][K] staging buffer is never materialized — kills its DRAM round-trip; wired into ggml-ork b5bc5ce8b) → pp128 2.90 → 3.13 t/s (+8%, ~7.2× under CPU). Bit-identical to the f32-buffer path (pack_f32_probe, mism=0). Key profile finding: after fusion the Q4_K→f32 dequant profiles at 0 ms (folded in) and pack/repack is still 77% of MoE-chain time but now entirely the NEON f32→int8 quant+tile (~~84 s) — so the next wall is the int8 quant+tile itself, NOT the dequant and NOT the submit. The earlier "repack→~~0 ⇒ near CPU parity" extrapolation was too optimistic (the dequant was cheap; the quant+tile is the real cost). Closing further needs a cheaper quant+tile or keeping weights int8-resident, not more dequant work. (run_stream_i8 in the MoE path adds a further +5% → 3.28, kept in scratchpad, not committed.) VERDICT: MoE-on-NPU is not yet viable on RK3588 (still ~7× under CPU, experts stay CPU-default) — but it is closing fast — no longer the clean net-loss the early numbers implied. Original design notes:
~~Collapse the per-expert submits with~~ the validated PC-chaining primitive (run_chain_i8: S independent matmuls, different weights, one submit). Per (layer, projection): decode chains the 8 selected experts (same input x, different W_e — exactly the chain case); prefill chains the ~128 active experts (different W_e, different gathered A_e). Needed pieces:
1. Resident pre-packed experts — DONE (ork_mm_repack_i8 pool kills the repack churn).
2. Single-K-slice (Sk==1) expert packing — the blocker. run_chain requires Sk==1, but pack_i8 tiles K=2048 at KS=1024 → Sk=2. The split is a tiling choice, not a HW limit (the int8 single-submit K ceiling is ~10752), so a chain-friendly pack mode (one K-slice, K≤ceiling) makes each expert one chainable program.
3. Grouped chain API — e.g. ork_mm_run_moe_i8(experts[], A_e[], C_e[], S) emitting one PC-chain per projection (decode: 8; prefill: the active set).
4. Cross-core chain fan-out — distribute the S experts across the 3 NPU cores (each PC-chains its own subcore_task range concurrently; a chain can't span cores). The unimplemented ~3× lever.
Projected effect (prefill): 18k → ~144 submits/batch (one chain per projection per layer), ÷3 with cross-core → ~48. That puts MoE prefill in the same submit regime as dense (which gets ~8.8× on NPU — decode-profiling A/B). Prefill only — MoE decode stays bandwidth-bound (chaining cuts submits, not the bytes streamed from DDR per token).
Note: rkllm has no MoE path (RKLLM-Toolkit through v1.3.0 lists only dense/RWKV/VL — no MoE), so there is no closed baseline; the open stack (experts on CPU + dense attn on NPU) is the only working NPU-assisted MoE runtime on RK3588. 5b would be open-stack-only headroom, not a gap-closer.
Validate: examples/quant (correctness) + mc_prof/llama-bench pp512 on the 35B-A3B vs the CPU baseline (22.59). Go/no-go: prefill must beat CPU after chaining + cross-core.

MoE-on-NPU saturation milestone thread (Branch A — NPU-centric) — M0 ✅ · M1a ✅ · M3-skew ✅

⚠️ SUPERSEDED (2026-08-19) — read M0 and M3-skew with this caveat. This thread's premise ("keep the experts int4-resident/streamed, run the GEMM on the NPU, free the CPU") was overtaken: the shipped answer is experts on the CPU (see Tier 5 banner). M0 is an ISOLATED-PRIMITIVE result and inverts when integrated. "NPU beats CPU at every expert shape and every M∈{8..128}, 3–10× at M=8, no crossover" is true per-GEMM, but the integrated path measured 5.33 t/s vs 27–28 t/s for CPU experts — ~4M tiny expert programs each pay the ~18 µs dispatch floor, which no per-GEMM win can recover. (The floor does amortize under chaining — run_chain_i4 3.40×, run_i4_experts 3.40–3.67× — but not enough, and decode-on-NPU additionally needs a per-token-varying resident set.) M3-skew's "hot-cache VIABLE / ~50 % of decode lookups NPU-resident" is contradicted: NPU decode loses to CPU decode at any hit-rate (measured 3.86 vs native ~5.7–7.5 t/s).

A re-scoped successor to Tier 5. Tier 5 measured the integrated ggml-ork mul_mat_id path at ~7× under CPU (walled by the CPU-side Q4_K→int8 repack, not the NPU). This thread restarts from the matmul-level primitive and the phase-aware saturation architecture: keep the experts int4-resident/streamed, run the GEMM on the NPU, and free the CPU for prefetch/expand/non-matmul ops.

✅ M0 — per-expert GEMM viability (DONE, 2026-06-26). Isolated the per-expert FFN GEMM with the new primitives (direct int4→int8 inflate ORK_DIRECT_I4 570cb68 + cacheable 0x403 fill). NPU beats CPU at every Qwen3-A3B expert shape (gate/up K2048×N768, down K768×N2048, +4096/1408) and every per-expert M∈{8..128} — 3–10× at M=8, no crossover, bit-exact. The direct inflate cut per-expert weight materialization 16–30× vs the old f32 repack → cost collapses to the GEMM. Dispatch: run_i8 (single submit) is already multi-core (N-tiles over 3 cores); run_chain_i8 is single-core (slower/expert) → run_stream_i8 for chainable gate/up (K%512==0,K≤4096), run_i8 for down. Caveat: baseline was 1-thread NEON; a 4-thread CPU GEMM narrows the M=8 margin to ~1.5–2.5× but the NPU still wins and frees all 4 CPU cores (the saturation point). Tool tools/moe_expert_probe.c (1840abb); fixed chan_scales_f32 NULL-guard (980cea9). → MoE Expert FFN GEMM — M0 viability. This flips the go/no-go Tier 5c's integrated ~7× couldn't.
🔬 M1 — pinned backbone + expert partition. Backbone (attn/router/dense shared-expert) resident on the NPU; routed experts int4-resident/streamed across the engine partition (the fit-by-partition layout).
- ✅ M1a — MUL_MAT_ID expert persist/load + IOVA wall (DONE, 2026-06-26). The mixed-.orkpack convert now captures the routed _exps/MUL_MAT_ID experts (before: backbone + router + dense shared-expert only, zero routed). ggml-ork c89cf8654 (synthetic per-expert key <name>#<e>; persist ALL n_expert slices via ork_orkpack_tier; load via ork_mm_load_i8/ork_mm_load_i4a8, no dequant+repack) + ork-driver 6b0d568 (ork_mm_load_i8 rebuilds full-K Bf, gated K%512==0 && K<=4096, so disk-loaded int8 experts are first-class chain inputs). LFM2.5-8B-A1B: .orkpack now has 2112 _exps entries (264×8) vs 0; reload-from-disk NPU output bit-identical to a CPU int8 ref (889 experts loaded; live NPU MoE path is itself broken on this model → gate is reload-vs-CPU-ref). ★ The same run confirmed the 4 GiB IOVA wall for real — an 8B MoE's int8 experts exceed 4 GiB → compute status -1 once ~1909 distinct experts resident (NOT a persist bug; mandates M2 stream + M3 cache). → MoE Expert Persist + IOVA Wall.
- 🔬 M1b — routing/dispatch. Which experts go NPU vs CPU/GPU per token (the partition policy).
⬜ M2 — tile-streamed prefill + IRQ affinity. Stream the expert tiles through the resident window on the prefill critical path (the Tier 9 fill chain); pin HW-IRQ handlers (rknpu/galcore) to the A55 little cores so they don't preempt the A76 cores running the NEON inflate/expand feed (submit threads stay on big cores — see CPU Big-Core Contention → IRQ affinity). IOVA anti-fragmentation is already handled (IOMMU scatters→contiguous IOVA; uniform fixed-size tile buffers = de-facto slab). Now MANDATORY (M1a's IOVA wall — resident-all-experts is impossible beyond a tiny model).
✅/⬜ M3 — hot-expert decode. Keep the most-frequently-routed experts NPU-resident for the bandwidth-bound decode phase (the long tail streams), pairing with the decode = additive-bandwidth partition.
- ✅ M3-skew — routing skew measured, hot-cache VIABLE (DONE, 2026-06-26). Profiled routing via the contiguous ffn_moe_argsort tensor (⚠️ the ffn_moe_topk view is non-contiguous → FALSE "uniform") on the UD-Q4 GGUF (skew is precision-invariant; router gate stays high-precision → q4 routing ≈ f16). Both targets SKEWED (Gini 0=uniform): LFM2-8B-A1B (32 exp, top-4) 0.44; Qwen3.6-35B-A3B (256 exp, top-8) 0.65 — skew RISES with depth (good for caching); per-expert int8 only 3.15 MB. Hit-rate vs cache (35B, pin top-K/layer): 5%(13)=33%@1.6GB · 10%(26)=48%@3.3GB · 20%(51)=67%@6.4GB · 50%(128)=92%@16GB. VERDICT: VIABLE — operating point ~3–3.5 GB (top ~26–28/256, ~10–11%/layer) → ~50% of decode lookups NPU-resident (within the 4 GiB window alongside the pinned backbone); cold tail → CPU/GPU; >50% (≥6 GB) exceeds the window → stream. Policy = per-LAYER top-K (global pin underperforms). → MoE Routing Skew.
- ⬜ M3-impl — wire the per-layer top-K LRU cache into the M1a persist path (cached experts reload from .orkpack).

Tier 6 — Cross-core: async round-robin stream ✅ ~3× (`run_stream_i8`)

A general primitive (not MoE-specific): spread an ork_mm_run_chain_i8 chain's independent matmul programs across the 3 NPU cores for ~nc× on the chain compute. Target beneficiaries: repack-free batched matmuls — EAGLE-3 target-model verification (multi-token forwards, the NPU's real sweet spot) and dense prefill batches. (Not MoE — there the chain is only 6%, repack 94%; see Tier 5b.)

❌ Option A — single submit, core_mask=0x7 + per-core subcore_task ranges: HARD-WEDGES the NPU. Hung the board so badly it needed a smart-plug power-cycle. The kernel does not drive a single multi-core PC-chain submit this way — which is why run_multicore never used it. Do not retry.
✅ Option B — concurrent per-core single-core submits: LANDED & validated (ork-driver dc4038b, refined 8044f6f/edf75c4; gated ORK_CHAIN_MC, default single-core). Per-core regcmd BO c->mrc[i], per-core task object c->mtk[i], dispatched through the persistent pool with an ioctl barrier so all cores submit together. This is exactly the closed runtime's building block — see the librkllmrt multi-core capture: librkllmrt issues ONLY single-core submits (core_mask 0x1/0x2/0x4, never 0x7), 3–6 PC-chained tasks each, round-robined across cores. So Option A (0x7) is confirmed dead; Option B is the right mechanism.
📊 MEASURED — 1.1–1.5× (not the theoretical 3×). tools/chain_bench, RK3588: floor-bound ~1.15×, compute/big-N ~1.5×. Ceiling is DDR bandwidth (N independent matmuls read N× the weight bytes through shared DDR) + concurrent-submit overhead — NOT an implementation bug (warmup-every-call fixed, per-core BOs, ioctl barrier, persistent pool all tried; each helped little). run_multicore's 2.27× is specific to N-splitting ONE job (same bytes, compute split) — which librkllmrt's matmul doesn't even do.
✅ DONE — async round-robin stream gets ~3× (ork_mm_run_stream_i8, ork-driver 888909f). Dispatch a STREAM of independent matmuls to per-core pool workers that PULL the next task dynamically (atomic counter, NO barrier) and run it single-core on their own core; a core finishing early grabs the next immediately → cores pipeline, never idle on a sync. Measured (chain_bench vs single-core): floor 3.1×, mid 3.7×, big 3.4× — consistently ~3×, vs the barrier fan-out's 1.2–1.3×. So the 1.5× ceiling was the barrier + giant-per-core-chain structure, NOT DDR bandwidth (earlier conclusion was wrong). Correct cold+warm (cold double-passes to prime fresh NPU output buffers). Envelope: K%512==0 && K≤4096 && Bf (full-K schedule-valid range; -3 otherwise). The barrier fan-out (ORK_CHAIN_MC) is now SUPERSEDED for independent matmuls.
✅ Committed (ork-driver bcb8266): run_stream_i8 (~3×), run_chain_i8 deprecated to single-core (one submit; fan-out removed), the full-K envelope correctness fix (K%512==0 && K≤4096, else -3 → caller K-splits via run_i8; was silently wrong for M>1 at e.g. down_proj K=768), caller-configurable cores (budget/ORK_NPU_MC, 1/2/3-core scaling), tools/chain_bench, and regcmd_capture ORK_SUBMIT_ONLY.
✅ MoE handler → run_stream_i8 (validated; MoE not yet viable but closing): integrated end-to-end; output coherent ("…red, blue, and yellow…"); the stream cut the NPU slice (16→11.8 s, ~5%) but the Q4_K→int8 repack still dominated. Experiment on the board/scratchpad, gated ORK_MOE_NPU (off); the stream variant of the MoE path not committed (run_chain is the committed MoE path).
✅ NEON SIMD repack: landed, validated, and it DID move the needle (ork_mm_pack_i8_f32/repack_i8_f32, ork-driver 2b0ee20). Fused f32→int8 quant+tile in one cache-friendly pass (no transpose scratch). An earlier attempt looked like a run_i8 -22 abort, but that was an inherited dirty-NPU state (power-cycle first — see CPU-Big-Core-Contention sibling lesson in board-ops), NOT a pack bug: validated bit-identical to pack_i8 (tools/pack_f32_probe.c, mism=0). Measured: pp128 0.97 → 2.90 t/s (3.26×), repack 95%→84%; deficit ~23× → ~7.9× under CPU — so the old "even free, can't flip MoE" caveat was wrong; a faster repack genuinely closes the gap.
✅ Fused Q4_K→int8 dequant: landed & committed (ork_mm_pack_i8_dequant/repack_i8_dequant, ork-driver b520ee7; ggml-ork b5bc5ce8b). Per-channel ork_dequant_row_fn callback dequants one output channel into a reused K-scratch + NEON quant+tile in place — the full f32[N][K] staging buffer is never materialized (kills its DRAM round-trip). Bit-identical (pack_f32_probe, mism=0). Measured: pp128 2.90 → 3.13 t/s (+8%); deficit ~7.2× under CPU 22.58. Profile: Q4_K→f32 dequant now 0 ms (folded in); pack/repack still 77% but entirely the NEON f32→int8 quant+tile (~84 s) — that quant+tile is the next wall, not the dequant and not the submit. (run_stream_i8 in the MoE path adds a further +5% → 3.28; scratchpad, uncommitted.)
🔬 The real open direction: EAGLE-3 verification — multi-token target forwards (repack-free, the NPU's sweet spot) are where the stream's ~3× lands end-to-end. Not yet built.
⚖️ DMA-in-chain coherency (multi-core ZC race) — now moot for run_chain_i8 (single-core) and the stream copies via per-core staging (no per-task DMA); the underlying ZC-output cold-coherency issue is still Tier 7.

Tier 8 — Mixed int8/int4-NF4 + imatrix quant ✅ DONE (PPL-validated)

The accuracy story for fitting bigger models: quantize from the fp16 source to a mixed scheme (int8 for q6_K-tier / attn, int4-NF4 for q4_K-tier / FFN), using the mixed GGUF only as an allocation oracle. See NF4, imatrix, and Mixed Precision.

✅ 8a. NF4 codebook (ork_mm_pack_i4a8, quant_kind=ORK_QK_CODEBOOK_NF4, ORK_NF4, dc782b2): per-output-channel absmax → nearest of the 16 bitsandbytes NF4 levels in the Bi4 nibble store; NEON vqtbl1q_s8 inflate via int8 LUT =round(level·127), bscale=absmax/127; reuses tile_f32_i8. RMS 0.873%; reconstruction err on Gaussian weights 0.1102 < uniform-int4 0.1581 (NF4 wins, the distribution match).
✅ 8b. Stochastic rounding (ORK_SR, folded into dc782b2): dithers between the two bracketing NF4 levels. A dot-product/PPL effect, NOT a per-weight RMS win — SR raises per-weight RMS while removing the deterministic-nearest accumulation bias across the length-K dot. Judge by PPL.
✅ 8c. imatrix-weighted scale (ork_mm_pack_i4a8_im, 93364b7): imatrix is per-INPUT-channel (K), scale per-OUTPUT-channel (N) — orthogonal, so importance can't re-weight a level pick; instead a clip-ratio grid search picks the per-channel scale minimizing Σ_k imatrix[k]·(w−dequant)² → 6.4× weighted-error reduction (4.001e5→6.273e4). imatrix=NULL byte-identical to absmax.
✅ 8d. compact 'O4N1' persist (ork_w_dump_i4a8/ork_mm_load_i4a8, d656955; NULL-guard 378045d): hdr+bscale[N]+Bi4, ~½ the tiled-int8 dump; streamed output bit-identical to resident, byte-identical re-dump (UNIFORM+NF4).
✅ 8e. gguf_tier_map (2c38a27+f78bc04 --emit-map): per-tensor ggml_type → {int8|int4} by an effective-bits threshold (≥5-bit→int8). Qwen3-1.7B-Q4_K_M → 140 q4_K→int4 + 28 q6_K→int8 (matches the convert split). The allocation oracle (and an engine-assignment oracle — see Heterogeneous Serving).
✅ PPL gate (wikitext-2, 20 chunks, Qwen3-1.7B, board): fp16-values + Q4_K_M-allocation + imatrix + NF4 = 11.33 @ ~58% int8 storage, vs Q4_K_M-native(CPU) 10.92 / all-int8 NPU 11.19 / double-quant-from-Q4_K_M 12.48 / UD-Q4_K_XL 17.91. Beating double-quant by −1.15 validates the rule: fp16 is the source; the mixed GGUF is an allocation oracle only, never re-quant its bytes. Residual gap to Q4_K_M = the int8-ACTIVATION cost (bounded by the all-int8 anchor), not the weights.
⬜ Next: feed gguf_tier_map --emit-map into ggml-ork's per-tensor pack-tier selection (end-to-end mixed-precision load).

Tier 9 — Streamed-prefill dma_buf fill chain ✅ cacheable + direct-inflate DONE; clean-only-bsync NEXT

The fill of the resident DMA buffer is the streamed-prefill bottleneck (prep is 6–26× the matmul at M≤128, TILE-bound; the crossover is large-M, not decode). See dma_buf Fill Bandwidth.

✅ 9a. Cacheable fill (the wall was an artifact). The flat ~1 GB/s fill was benching ork_dma_alloc's write-combine zero-copy flag (0x401) (b53c59b), not slow RAM — the production tiles (pack/tile_f32_i8/ork_mm_load_i8) already use cacheable 0x403 (RKNPU_MEM_CACHEABLE). Measured cacheable + clean-before bsync = ~10 GB/s (8.3–8.8×), NPU read correct (cached memcpy ~12 GB/s; the clean flush is ~15–18% of cost; NEON stnp doesn't help WC — the uncached write is the wall). Candidate: revert the input ZC path 0x401→0x403; leave the output ORK_ZC_OUT case (invalidate-after, unsafe-under-concurrent-multicore, Tier 7).
✅ 9b. Direct int4→int8 tiled inflate (ORK_DIRECT_I4, 570cb68): expand_chan_i4_i8(UNIFORM)/inflate_chan_nf4_i8(NF4)→linear int8, tile_i8_to_tiles rearranges by 32-byte k-tile memcpy — skips the f32 round-trip. Bit-identical (tiled bytes memcmp==0 + NPU exact vs f32-path + CPU ref, UNIFORM+NF4). 2.2–2.7× UNIFORM / 3.6–4.9× NF4 on inflate+tile. Inflate now near-free; residual = tile copy + a DOUBLE full bsync.
🔬 9c. Clean-only bsync(TO_DEVICE) steady-state re-fill (NEXT, identified). For re-filling an already-initialized buffer, only the clean is needed (not the double full TO|FROM-then-TO) → should drop the fill floor toward the ~7 ms/14B-slice cacheable number.
⬜ 9d. SRAM/NBUF tier (identified, not probed). RKNPU_MEM_TRY_ALLOC_SRAM + RKNPU_GET_FREE_SRAM_SIZE expose on-chip NPU SRAM (~MB) + an NBUF pool — a fourth memory tier below DRAM for the hottest small tensors/scratch.
Disk note: mmap-warm == RAM (page cache hides it, 0.5% delta); cold disk ~1.4× (mmap readahead)/~2.2× (pread), NVMe cold ceiling ~0.5 GB/s — only matters when the model exceeds RAM AND the page is cold.
The chain compounds: cacheable (8×) → direct inflate (2–5×) → clean-only bsync (next). Foundation for the phase-aware saturation architecture in Heterogeneous Serving and Scheduling.

Tier 10 — >4 GiB multi-domain zero-copy import: THREE independent limits (root-caused 2026-07-06) ✅ working

The >4 GiB sliding-window path keeps weights resident across multiple 32-bit IOMMU domains (n_domains, dom_activate per submit — see NPU IOVA 4 GiB cap). Weights beyond domain 0 are imported (dma-heap ALLOC → PRIME_FD_TO_HANDLE → MEM_CREATE, zero-copy), not native bcreate (which memcpys). When the new fused primitives (SiLU/RMSNorm) shifted the 7B's footprint onto this path it broke — and the debug uncovered THREE independent constraints, all now understood and handled. tools/mc_import_probe.c (import-first, multi-core, checks C[0] and C[last]) is the reproducer; checking only C[0] is a false-positive trap — corruption often spares C[0] (the first output col) while wrecking C[last].

Constraint 1 — PC-chained imported chain-walk fault. A K-split submit (e.g. ffn_down K=18944 → mcworker_pref_ksplit, task_number=19) PC-chains N programs; the CDMA chain-walker traverses each task's weight mapping. >~5–6 separately-imported dma-buf mappings in one chained submit → submit rejected (errno=110, rc=-1 C=0). Native bcreate chains of the same count walk fine; the 1.7B (6 imported tiles) works. → mitigated by chunking (fewer, larger mappings per chain).

Constraint 2 — large single import fails at IOMMU-map. A single imported dma-buf ≳16–64 MB fails at PRIME_FD_TO_HANDLE: Cannot allocate memory (measured: 16 MB OK, 64 MB ENOMEM) — the domain's 32-bit IOVA allocator can't find a contiguous range that large (fragmentation). The full-weight (68 MB) consolidation returns WEIGHT NULL; an earlier variant hung the load (R-state 571 s). → caps chunk size from above.

Constraint 3 — import into a not-yet-established non-0 domain corrupts (the real regression). The kernel rknpu driver lazily sets up a domain's IOVA allocator / page table on its FIRST buffer; if that first buffer is an imported dma-buf (SG-list), the import's pages land on wrong/aliased IOVAs → the NPU reads garbage for some weight tiles → non-deterministic dropped-K corruption (C[last]=14336/16896 vs 18944, varies run-to-run). Critically this is NOT a multi-core bug — single-core (core 0 alone) corrupts too; the per-core C[0]-ok/C[last]-wrong pattern was non-deterministic noise. Native weights are immune (they exercise the domain-setup path correctly). A native alloc before the first import establishes the domain; every import after is bit-exact.

Fixes shipped (both default-on):

Size-bounded chunked import (Constraints 1 & 2) — ork_mm_load_i8_import, own_bufs[]/n_own_bufs, ORK_IMPORT_CHUNK_MB (default 16), ORK_NO_CONSOLIDATE_IMPORT to disable. Packs a weight's tiles into ~~16 MB imported chunks (tiles are chunk.dma+off views): down-proj 68 MB → ~~5 chunks — in the only window that satisfies both C1 (≤~~5–6 mappings/chain) and C2 (≤~~16 MB/alloc). Cannot consolidate further (tested: per-tile faults C1; 64 MB faults C2; full 68 MB WEIGHT NULL). 16 MB is a genuine sweet spot.
Per-domain native anchor (Constraint 3) — ork_dom_prime(c,dom) does one tiny (64 KB) native bcreate per non-0 domain before any import maps into it; dom_anchor[ORK_MAXDOM], freed at teardown. Once per domain (guarded), independent of chunking — 4 tiny allocs for the whole 7B. No submit needed (alloc-only establishes the domain).

Validated (2026-07-06): mc_import_probe import-first + multi-core, K=18944 N=3584, 5/5 bit-exact (C[0]==C[last]==18944); full 7B forced-import + multi-core pp128 = 58.73 t/s, no wedge, no timeout. The zero-copy import path is now the correct, full-speed (multi-core) >4 GiB path.

⬜ 10a. Trace the exact chain-walk mapping ceiling (Constraint 1). Bracket is 6 OK → 19 faults. How: extend tools/import_count_probe.c to the CHAIN path — one ork_mm_run_chain_i8 submit, task_number sweeping 2→24 per-tile bimports (force ORK_NO_CONSOLIDATE_IMPORT), binary-search the N where errno=110 first appears; sweep domain 0 vs non-0; hold count fixed while scaling per-tile bytes to separate count-of-mappings from total-pages. Instrumentation in place (enriched rknpu_submit_ioctl fail log + g_last_* + ORK_DUMP_FAIL). Deliverable: raise ORK_IMPORT_CHUNK_MB toward the true ceiling (fewer chunks = less load overhead) instead of the conservative 16.
⬜ 10b. Trace the exact import-size ceiling (Constraint 2). 16 MB OK → 64 MB PRIME_FD_TO_HANDLE ENOMEM. How: time bimport(size) for size ∈ {16,20,24,32,48,64} MB (alloc→free, print µs + rc each); watch /proc/meminfo CmaFree and /sys/kernel/debug/dma_buf/bufinfo. It's an IOVA-contiguity limit (32-bit domain fragmentation), so it likely shrinks as a domain fills — test large imports into a nearly-full domain. Determines whether the ceiling is fixed or fill-dependent (→ pack big weights first).
⬜ 10c. Kernel-side: why does dma-buf-import-into-fresh-domain misbehave (Constraint 3)? The anchor is a clean userspace workaround, but the underlying rknpu first-buffer domain-setup ordering bug (native-first works, import-first corrupts) is worth reporting upstream / confirming in the driver source (rknpu_iommu_dma_map_sg / domain lazy-init). If fixed in-kernel the anchor becomes unnecessary; until then it stays.
Why it matters. The two limits define the usable chunk-size window [few-enough-mappings, small-enough-alloc]. Measuring them lets us (1) pick the largest safe chunk (min load overhead + min mappings), and (2) decide if a kernel-side fix is worth it — e.g. a larger CDMA chain-walker budget, or mapping a weight's chunks into one contiguous IOVA span so the walker sees a single mapping regardless of count. Feeds the streaming/saturation architecture (NPU IOVA 4 GiB cap, Heterogeneous Serving and Scheduling).

Tier 7 — Multi-core DMA cache coherency ⚖️ DEFERRED (low ROI; tracked)

Problem. Concurrent per-core execution with a zero-copy ork_dma_alloc output buffer corrupts the output (the single-core ZC-output cold-coherency bug, racing across cores). Now moot in practice: run_chain_i8 is single-core (the barrier fan-out was removed), and run_stream_i8 copies through per-core staging buffers (no per-task DMA), so neither exposes it. It would only resurface if a future path fed DMA-resident A/C into the stream. The root single-core ZC-output coherency bug remains the underlying issue (off by default; see the ORK_ZC_OUT note in AGENTS.md).

Root cause. Same class as the single-core output-ZC bug that 3fad74a fixed (CPU dirty/stale cache lines racing the NPU's writes to the shared coherent buffer) — but the single-core fix's bsync clean-before / invalidate-after is not serialized across concurrent cores, and the per-core warmup rep also writes the real ZC buffer. So the coherency window races.

Fix approach (when pursued). Bracket the entire multi-core window, not per task: one bsync(TO_DEVICE) over all DMA buffers before launching any core, one bsync(FROM_DEVICE) after joining all cores, with no CPU touch of the buffers in between; and remove the warmup rep's writes to the real ZC output (warm into scratch, or skip warmup for ZC). Validate: the test_matmul DMA-task case passes multi-core (mism=0) with the any_dma fallback removed; full make test green; no wedge.

ROI = low → deferred. The Tier-6 beneficiaries (MoE / EAGLE-3 / dense) feed the chain from malloc'd gather buffers (no per-task DMA → fallback never fires); output-ZC measured ~0 end-to-end gain; input-ZC saves a <1% memcpy. Revisit only if a future EAGLE-3/dense path keeps activations in DMA buffers end-to-end (the Stage-2 ggml-ork DMA buffer type, Tier 1c-i) and wants cross-core fan-out — then this is the enabling fix. See the ORK_ZC_OUT note in AGENTS.md.

Efficiency scan — O(n)→O(n/x) consolidation candidates 🔬 (scan 2026-07-06)

Prompted by the #tier-10 (per-tile bimport O(tiles) → chunked O(tiles/x)): a scan of the hot paths for the same class of inefficiency — a fixed per-item cost paid N times that can be amortised over a batch. Line numbers are src/npu.c at scan time (they drift — the function/pattern is the durable reference). Each needs a profiled before/after — some may be dwarfed by the weight-DMA re-stream (the known dominant cost, see AGENTS "Weight-DMA amortization") or the ~150 µs submit floor; measure before optimising.

⬜ E1. Per-tile weight bsync → one bsync per weight/chunk (load & pack). The default (non-consolidated) ork_mm_pack_i8 / ork_mm_load_i8 fill loop issues a double MEM_SYNC ioctl per tile — bsync(TO|FROM); bsync(TO) (npu.c ~1101/1112 pack, ~1193/1212 load) → 2·Sk·Sn ioctls per weight. The native ORK_CONSOLIDATE_I8 path already collapses this to one bsync_off over the whole packed buffer (npu.c ~1082–1083), and the new chunked import does one dmabuf_sync per chunk. Generalise: one bsync over the packed region (or per chunk) for the default paths too → O(2·tiles)→O(chunks) sync ioctls per load. Compounds with #tier-9 (drop the TO|FROM half; a fresh fill only needs the clean TO). Highest-confidence win at model-load / streaming-refill time.
⬜ E2. Per-matmul descriptor malloc/free in mcworker → preallocated scratch. mcworker (signalled per matmul, persistent pool) does malloc((size_t)maxpd*sizeof *pd) (npu.c ~2612) and free(pd) (~2691/2696) on every matmul, every core. At decode (M=1, thousands of matmuls × up to 3 cores) that is thousands of heap alloc/free round-trips on the latency-critical path. maxpd (=Pmax) is bounded by the weight's tile count — preallocate a per-worker (or per-context) pd[]/RC/tk scratch sized to the worst case once and reuse it → O(matmuls)→O(1) allocations. Same idea as the persistent worker pool already applied to pthread_create/join.
⬜ E3. getenv("ORK_TRACE") per submit → cached flag. trace_submit (npu.c 484, called from rknpu_submit_ioctl on every RKNPU_SUBMIT) does a fresh getenv — an O(environ) string scan per submit (thousands per prefill). Cache it in a static int at init like the other knobs (ORK_RCAP, ORK_R1040, ORK_NO_AFFINITY, …) → O(submits)→O(1). Tiny per-call but on the hottest path; trivially safe. Audit the other uncached getenvs while there (most are already static …=-1 cached; grep getenv and confirm none new sit in a run/submit loop).
Reference exemplar (DONE): [[#tier-10|per-tile bimport → size-bounded chunks]] — the fix that motivated this scan.
Validation for all: ORK_PROFILE=1 load/decode section timing + mc_prof before/after; confirm bit-exact vs CPU ref (make quant). Do not land any of these without a measured delta — E1 at load-time is the one most likely to be material.

Mode-transition layer — centralize the scattered per-op reset ✅ (Phases 1+2 landed on `main`, 2026-07-14, v0.7.0)

★ LANDED on main (v0.7.0). ork_npu_enter(c, to, profile, chain) + the XSPEC[] policy table in src/npu.c now own the transition policy; 16 clean guards → 12 profiles, validated make test byte-identical (all dtypes, both keep-warm knobs). The chaining strategy is carried as a chain state arg (OCK_NONE/SW/HW/FUSED) so the few chain-sensitive handoffs pick their own path. Phase 2 evaluated the drifted cells 1-by-1: the →I8_CHAIN profiles converged (fixed the ~105 ms fp16→chain reset drift), the two stream-int8 profiles unified, the SDP element-wise per-op reset was removed (ORK_SDP_NORESET, default on — the fp16→SDP→fp16 wedge was the already-fixed c->task descriptor axis, not a dtype reset), and the int4 clear-gating was confirmed mechanism-legit. Full write-up + per-item resolutions + the golden/build/CI notes: Exp 2026-07-14 Mode-Transition Layer. The design below is the original proposal.

Open follow-ups (from Phase 2):

⬜ Realize the fp16→chain keep-warm win via a real consumer. Phase 2's →I8_CHAIN convergence makes an int8 chain that follows an fp16 op skip the ~105 ms ACT_RESET+re-warm (previously it always paid it — the drift). The fix is byte-exact and validated by chain_xition_probe, but it is performance-neutral in the shipped path today because no production pipeline currently issues an fp16-op → int8-chain handoff back-to-back (chains are dormant; see the "chains dormant" finding). The saving only materializes once a real consumer interleaves them — e.g. attention-on-NPU (fp16 QK/AV matmul + activation → int8 FFN chain) or an SSM/GDN hybrid decode step. Item: stand up such a consumer and measure the per-layer reset saving end-to-end (expect ~105 ms × #handoffs/token).
⬜ Consecutive chain_i4 miscompute — unconfirmed (watch item). The int4-transition sweep (i4_xition_probe) is 36/36 coherent; the earlier "0/72 coherent" alarm was a probe artifact (back-to-back identical chain_i4 submits in the old probe structure), not a real fault. A residual, unconfirmed hypothesis remains that two chain_i4 ops issued consecutively without an intervening transition could re-warm-gap and miscompute in some regime. Not reproduced; no evidence it is real. Item: a dedicated back-to-back-chain_i4 probe (distinct operands, no reset between) to confirm or retire it.

Problem. The NPU precision-mode transition policy (if(dt!=c->last_dt){ … ACT_RESET / warmed=0 / mccsz=0 / last_dt=dt }) is copy-pasted inline into ~10 run entrypoints in src/npu.c, each subtly different and drifted: multi-core run (~3573), single-core run (~4188), run_stream_i8 (~7807), run_stream_f16 (~7868), run_i4 (~3977/4138), and the specialized ewmul/out8/silu/fused-chain int8 entries (~4366/4478/4753/4795), plus the fp16 single-core entry (~4588). There is no single owner of "what does moving from mode X to mode Y require." This is why the int8↔fp16 keep-warm fix had to be hand-patched into 5 separate sites and why finding them all is error-prone.

Design. A single ork_npu_enter(c, target_dt) (the transition layer) that every run path calls first, driven by an exhaustive (from_dt → to_dt) policy table — the one place that decides, per permutation, whether to ACT_RESET, re-warm (mwarm/warmed=0), realloc per-core buffers (mccsz/ccsz=0), or keep-warm (no-op). Each run path's inline reset collapses to one call; a new op/dtype becomes one table row instead of N copies. Keep-warm (int8↔int8/int4/fp16/int16) is then one cell, not a scatter of ork_f16warm()/ork_nothrash() conditionals.

Two nuances the layer MUST respect (else it wedges):

Precision-mode ≠ descriptor state. The LUT-activation c->task poisoning (the exp/silu → single-core matmul wedge, Exp-2026-07-12) is a separate axis from dtype — ACT_RESET does not fix it (proven). The transition layer owns the precision-mode reset; the c->task save/restore stays an op responsibility (or becomes a distinct second concern in the layer). Do not conflate them.
Per-core state (mwarm[i], per-core buffer sizes, dom_active) is part of the transition, not just the global last_dt.

De-risk (mandatory). This touches every matmul/activation path, so a bug miscomputes/wedges everything, not one gated experiment. Land as a behavior-preserving refactor first: ork_npu_enter reproduces each path's exact current reset so a full board make test (all dtypes, all examples) passes byte-identical. Only then is a policy change (e.g. promoting keep-warm to default) a one-line table edit validated by re-running make test + the relevant bench.

Current interim state (scattered, works). The int8↔fp16 keep-warm is applied at 5 sites (ORK_KW_DT = ORK_I8_LIVE || DT_F16): multi-core run, run_stream_i8, run_stream_f16, single-core run, fp16 single-core. Validated: SSM prefill 20.7→103.5 tok/s (5×), soft-resets 28→1, coherent. ★ Now DEFAULT ON (ork-driver main @ 6ecb168; ORK_SSM_KEEPWARM=0 to disable) — a general win for any fp16 op interleaved with int8 matmuls. The refactor replaces this scatter with the table; the ORK_MIXED_NOTHRASH (int↔int) keep-warm folds in as another set of cells. Pick this up when a second fp16/int16-interleave consumer lands (on-NPU softmax/attention, the chain-assembler), which is when the scatter cost bites again. See Exp 2026-07-13 Linear-Attention Scans on NPU (SSM scan wins ~2× at Q8 2.7B → size-gated default-on; GDN built+coherent but not-a-win).

Tier 11 — NPU Doorbell / Persistent-Kernel "Zero-Overhead Graph Cut" 🎨 (FOR THE CRAFT — explicitly NOT an efficiency lever)

Why it's here: we're going to build this because it's cool, not because it's fast. The profiling below shows it is not an efficiency win; it's on the roadmap as a deliberate research/craft moonshot — a persistent-kernel execution engine on top of a static DMA accelerator. Track it as such; do not gate any perf milestone on it. UPDATE (2026-07-15): the pure userspace-doorbell form is TESTED-DEAD on TWO independent walls. (a) The "infinite treadmill" needs an NPU free-running spin — a self-referential 0x0010 pointer was measured to COMPLETE, not spin (Wall 3). (b) The refined "bounded treadmill + exit early" fixes Wall 3 (a large-task_number ring of replicated descriptors runs a clean bounded treadmill, ~1µs/task, up to ~13ms) — BUT the escape needs a live mid-submit patch, and that was measured NOT honored: patching descriptor[1000] of a 2000-ring 950µs ahead left the marker unwritten (Wall 2). ~~The job image is snapshotted at submit; DRAM patches are not re-read~~[^dyn16], so userspace can neither hold the NPU nor redirect it. The craft build therefore targets the only feasible variant: a kernel between-task host callback — the accel/rocket driver already re-arms each task on the completion IRQ (where the descriptors ARE re-read per task, in kernel space), so hook that to run the NEON requant before re-arming. Same "one continuous chain, CPU mid-chain" goal, no spin, no free-run, no live-patch race. That's the cool-but-real build.

The idea (GPU persistent-kernel / doorbell, ported to the NPU). To do the CPU-only int32→narrow requant mid-chain (the NPU can't recombine the int16-lane split — see ISA Reference) without a kernel round-trip per hop, keep the NPU in ONE continuous hardware chain that never returns to the driver:

Infinite treadmill — after the matmul dumps its int32 accumulator to DRAM, the chain enters a spin of dummy 1×1 conv-of-zeroes ops that loops until told otherwise.
Doorbell — just before the spin, the NPU does a 1-byte DMA write to a fixed uncached on-die SRAM address (the ~956 KB pool we enabled).
CPU spinlock — a pinned user-space thread spins on that SRAM byte; when it flips, it reads the int32 buffer from DRAM and runs the NEON requant (no kernel entry).
Pointer-patch escape — CPU flushes (DC CVAC) then overwrites the dummy loop's 0x0010 next-regcmd pointer to aim at the next real layer.
Resume — the NPU follows the patched pointer out of the loop and continues on the clean scaled data.

Feasibility walls (why this is a moonshot on this silicon+driver):

Wall 1 — fenced/timed submit (soft). The chain runs inside one RKNPU_SUBMIT the kernel fences with a timeout; a job that never raises completion is indistinguishable from a hang → errno=110 → self-healing reset. A fast doorbell→patch can fit a generous timeout window, but you live inside the watchdog.
Wall 2 — live mid-submit patch is NOT honored (hard, FATAL — now TESTED 2026-07-15). The escape needs the NPU to re-read the descriptor/regcmd from DRAM as it churns the ring, so a CPU hot-patch far ahead redirects it. Probed directly (ORK_RESHAPE_PATCH/_PATCHRC): built a 2000-entry treadmill ring, and a background thread hot-patched descriptor[1000] (950µs / ~950 tasks ahead of the NPU) to a GEMM marker that writes 0x3000, with a cache flush. Result: 0x3000 stayed ZERO — patch NOT honored. Same for patching the shared dummy regcmd's output mid-run — not honored. So the job image (task[] descriptor array AND regcmd) is snapshotted at submit; ~~mid-submit DRAM patches are never re-read~~[^dyn16]. A bigger ring can't help — it's not a prefetch-horizon issue, ~~the descriptors simply aren't re-fetched at all~~[^dyn16]. This kills the userspace redirect escape.
Wall 3 — no free-running mode (hard, FATAL — now TESTED 2026-07-15). ★ The "infinite treadmill" needs the HW to spin indefinitely. Directly probed it (ORK_RESHAPE_SELFLOOP): pointed the last chained task's 0x0010 next-regcmd descriptor at its own regcmd (a self-loop). Result: it COMPLETED (rc=0, 132µs), same as the null-terminated baseline (72µs) — it did NOT spin. So the pointer chain is bounded by task_number; a self-referential 0x0010 just re-runs the tail a bounded number of times, then raises DONE. There is no loop flag in the DRM uABI (only RKNPU_JOB_PC=0x1 / RKNPU_JOB_PINGPONG=0x4), and no 0x00xx PC-block repeat/loop control. An infinite spin would need an undocumented task_number-override/free-running mode that, per this test, does not exist — consistent with a fixed-function DMA pipeline. (Contrast: an unterminated 0x0010 pointing at a NEXT (garbage) regcmd DOES get walked → faults; so the HW follows the pointer but the task count still bounds the run.) This is the fatal wall: you cannot hold the NPU in a userspace-controlled spin.

[^dyn16]: Partially corrected 2026-07-16 — see Exp-2026-07-16 Dynamic-Submit-API-and-Doorbell. "Never re-read / not re-fetched at all" is too strong. A 0x0014=0 halt-patch written into a not-yet-executed program of a forward chain IS honored: the chain stops early (validated by ork_dyn_halt/steer_probe — halt(8) on a 16-op chain → 9/16 outputs). So a distinct forward program's regcmd is re-fetched at exec-time. What stays dead is redirect/extend, but the true cause is the immutable task_number bound (Wall 3), not "patches aren't read": a 0x0010 aimed forward-beyond-count aborts (-22), a backward self-loop completes instead of spinning. Net for the doorbell program: per-op progress + early-exit (halt) are viable and shipped (ork_dyn_*, multi-core NONBLOCK ork_dyn_begin_mc); hold/redirect remain not viable (the Tier-11 conclusion is unchanged — only its mechanism is corrected).

The feasible variant — a kernel between-task callback (recon'd 2026-07-15). Do the same thing in the driver: the completion IRQ (rknpu_isr → rknpu_job_done) already re-arms the next chained task in kernel space — that IS a doorbell, and there the descriptors ARE re-read per task (unlike the snapshot userspace hits). Hook it to run the requant between tasks before re-arming. Recon findings that shape the build:

It's a FULL kernel rebuild + reflash, not a module hook. On the board (6.1.115-vendor-rk35xx-sram) CONFIG_ROCKCHIP_RKNPU=y — the driver is built into the kernel, no .ko. So changing rknpu_job.c means rebuilding the whole kernel and reflashing Image+dtb. Highest board-risk action we have (bad boot → recover via /boot/RESTORE_STOCK_KERNEL.sh + Image.orig + stock vmlinuz/dtb, which ARE present).
The kernel source tree is not on the board or .239 — must be located (wherever the -sram kernel was built) before any patch. Hard blocker to starting.
Design + honest catch: the completion handler is IRQ context, so the requant must run in a threaded IRQ (kernel_neon_begin, map the int32 DMA buffer, per-channel scale→narrow, write back) then re-arm. That saves only the userspace round-trip (the kernel/user transition) — it does NOT save the requant itself, which is on the NPU's critical path either way (the next matmul needs the scaled input, so the NPU stalls during the requant regardless of where it runs). So the achievable win is a fraction of the profiled 5–22% (the transition slice, not the compute). Reinforces: cool engine, not a perf lever.
Probes/state: all the userspace doorbell probes live on branch origin/feat/npu-doorbell (in the reshape replay: ORK_RESHAPE_SELFLOOP = Wall 3, ORK_RESHAPE_TN = bounded-treadmill ring, ORK_RESHAPE_PATCH/_PATCHRC = Wall 2 live-patch). Kept off main (they're env-gated RE). Next step for the craft build = locate the kernel source + a reflash go-ahead, then patch rknpu_job.c's re-arm path (write the patch first, stage RESTORE, reboot).

Profiling verdict (2026-07-15, mc_prof): the round-trip does NOT dominate the layers that matter. Submit floor ≈ 132 µs/submit (measured where compute≈0). Round-trip share by shape: 1×512×512 ~91% (but ~132 µs absolute — cheap), 1×4096×11008 ~22%, 128×2048×2048 ~18%, 256×4096×4096 ~5% (weight-DMA/compute dominate the heavy layers). Pipeline ceiling ~1.08× (host work ~7 µs vs ~132 µs ioctl) — no big hidden slice. And decode is latency/serialization-bound with the NPU ~90% idle, so eliminating submit latency doesn't touch the real decode wall. HW-chaining (task_number>1) already amortizes the floor for homogeneous graphs, and the fp16 SDP-path sidesteps int32 entirely — so the round-trip is only paid, and only ~5–22%, on the exact heavy int8 layers this targets. Net (heavy-layer regime): a great stunt, a poor optimization. Status: 🎨 parked as a craft/curiosity build.

★ WHEN THE KERNEL HOOK ACTUALLY PAYS (regime analysis, 2026-07-15). The profiling above is the monolithic-tensor regime (math dominates → round-trip is a rounding error). In the micro-op regime (tiny ops where math ~20µs < the ~132µs submit floor), the NPU is genuinely ioctl-bound (~70% idle waiting on the scheduler) and a kernel IRQ-chain flips Amdahl — potential ~2× end-to-end. BUT two measured facts pin down the exact trigger:

Pure-NPU micro-op chains already collapse without the kernel — task_number>1 HW-chaining walks the 0x0010 chain in ONE submit (measured ~1µs/task treadmill; ~16µs/task irreducible), and run_chain_i8/the reshape replay already chain matmul→SDP→matmul in a single submit. So a fragmented graph of NPU-executable ops (activations, depthwise convs, norms) needs no driver change.
⇒ The hook's unique value is narrow: when a CPU step must INTERLEAVE between tiny NPU ops (int32 requant, dynamic MoE routing) — the one thing that forces the HW chain to break and bounce to userspace per op. There the threaded-IRQ callback keeps it one submit; HW-chaining alone can't (it has no CPU-in-the-loop step).
It does NOT help LLM decode — decode's ~90% NPU idle is data-dependent SERIALIZATION (layer N needs N-1), not dispatch latency, so there's nothing to pipeline away. The hook helps THROUGHPUT-oriented fragmented workloads: continuous streaming (arm frame N+1 while N drains) and MoE (experts independent within a layer).
Concrete go-candidate: Tier 5 (MoE). We closed MoE partly because "submit count is per-expert (~18k/batch)" — the exact micro-op-dispatch strangulation. HW-chaining experts-within-a-layer recovered most of it (wall became the CPU repack), but a kernel hook re-arming the next expert on IRQ with CPU routing/requant in the bottom-half is the clean form. If the project pivots to MoE-on-NPU or continuous streaming, the -sram source hunt + reflash is justified. Until then: parked.

★★ BEST VARIANT — async data-plane spinlock, NO REFLASH (spin-not-sleep; TESTED 2026-07-15). The 50µs isn't the syscall (~1-2µs) — it's the scheduler-wake when the completion IRQ wakes a thread sleeping in wait_event. Kill it by BUSY-POLLING instead of sleeping, keeping ALL logic in userspace. This works on the STOCK driver — no kernel change. (Corrects an earlier draft here that wrongly said the stock driver had no async path.)

★ RKNPU_JOB_NONBLOCK EXISTS and WORKS. Flag = 1<<1 = 0x2 (PC=1<<0, PINGPONG=1<<2=0x4, FENCE_IN/OUT=1<<3/1<<4; confirmed vs the mesa/rockchip UAPI — our repo header only had the PC/PINGPONG subset). TESTED (ORK_JOB_NONBLOCK, rknpu_submit_ioctl sets flags|=0x2): a 64×2048×2048 submit returns in ~5µs (hw_elapse=0, driver did NOT wait) vs ~400-750µs blocking, and test_matmul goldens still pass with the async path + a drain wait. So SUBMIT can return before completion on the stock, built-in driver.
Completion without the fence = a DRAM doorbell. CONFIG_ROCKCHIP_RKNPU_FENCE is off, so no fence to poll — but you don't need it: end the chain with a 1-byte DMA write to a cache-coherent DRAM address and busy-poll it (invalidate-per-iteration for NPU→CPU coherence). No reflash, no fence, no IRQ-hook.
io_uring (arch #2) still N/A — rknpu submit is a custom DRM_IOCTL (no ->uring_cmd), not pollable. The NONBLOCK+doorbell path makes it unnecessary anyway.
Verdict: the async spinlock is a PURE-USERSPACE build with ZERO kernel change — the preferred form of the whole doorbell idea, and it obsoletes both the IRQ-hook code patch AND the fence-flip reflash (neither needed). This is the one doorbell variant actually worth building. Status (all userspace, no reflash): (1) DRAM-doorbell completion — BUILT + MEASURED (2026-07-15). ork_npu_doorbell_prof + tools/doorbell_prof.c on feat/npu-doorbell (f9a687e): N serial int8 matmuls, blocking (flags=0x5) vs NONBLOCK (flags=0x7) + a DRAM-doorbell DC-CIVAC busy-poll on the last output element. Bit-correct on both paths, every shape (blk-ok nb-ok). NOTE: bsync-FROM alone leaves CPU cache lines stale after a multi-submit loop — a fresh read needs an explicit seed(sentinel)+dc cvac before / dc civac-spin after (the same DC-CIVAC the doorbell poll itself needs). (2) back-to-back NONBLOCK queue — VALIDATED (the ~5µs-return-while-NPU-busy fact = the driver queues; no serialize stall, no wedge). (3) wire the int32 requant into the spin — REMAINING. Probes/flag on feat/npu-doorbell. Still gated only on a genuinely dispatch-bound workload (per the regime analysis) — but no longer on any reflash.

Efficiency math — MEASURED with NONBLOCK (2026-07-15). Two profiled facts: (1) a blocking submit's per-op OVERHEAD is ~130µs (mc_prof: 132µs floor at compute≈0; and per-submit cost = ~130µs + compute across shapes); (2) a NONBLOCK submit returns in ~5µs and the driver QUEUES it (a 2nd submit fired 100µs into a ~750µs job still returns ~5µs, no wedge). So the per-op transition drops ~130µs → ~5µs (bigger than the earlier 50µs estimate). Per-op speedup = (130+C)/(5+C), C = compute+CPU-work:

C (compute + requant)	blocking/op	nonblock/op	speedup
2µs (tiny)	~132	~7	~19×
20µs (micro-op)	~150	~25	~6×
130µs	~260	~135	1.9×
500µs (decode FFN-ish)	~630	~505	1.25×
2750µs (heavy prefill)	~2880	~2755	1.05×

MEASURED N-op pipeline wall (2026-07-15, doorbell_prof 20, RK3588, both paths bit-correct). The estimate above holds against real hardware — a real serial pipeline of N int8 matmuls, blocking submit vs NONBLOCK+doorbell-poll:

M	K	N	blocking µs/op	nonblock µs/op	speedup
1	512	512	78.4	36.2	2.17×
1	2048	2048	429.9	408.9	1.05×
64	512	512	166.5	128.8	1.29×
64	2048	2048	463.1	445.8	1.04×
64	4096	4096	2753.1	2749.4	1.00×

Real per-op overhead removed ≈ 40µs (M=1/512) rising to the full submit floor as the op shrinks; the micro-op end (M=1, K=N=512) hits 2.17× and the heavy prefill end (M=64, K=N=4096) flattens to 1.00×, exactly the (130+C)/(5+C) shape — the dispatch saving is real but only visible once C is small. (A serial pipeline can't overlap dispatch with compute the way a fanned-out one can, so these are the conservative in-order numbers; a fragmented micro-op fan-out — MoE routing, DFlash drafting — is where the tiny-C rows dominate the mix.)

★ CPU-work-in-NPU-shadow overlap is FREE (MEASURED 2026-07-15, overlap_prof). The "zero-time router" — run the (CPU-side) MoE routing math in the shadow of an async NPU op — measured at the primitive level: a real fp32 GEMV between a NONBLOCK submit and the doorbell poll. overlap_wall = max(NPU, CPU), never the sum (hidden ≈ 100% at every level), and the LPDDR4X bandwidth-contention risk did NOT materialize — even 14ms of CPU GEMV left the NPU op's length untouched (matches bench_monitored: RAM-BW headroom):

cpu_reps	NPU solo	CPU solo	overlap wall	sum	hidden
1	2720µs	221µs	2720µs	2940	99.9%
4	2744µs	881µs	2744µs	3626	100%
16	2750µs	3526µs	3530µs	6276	99.8%
64	2743µs	14108µs	14112µs	16851	99.9%

Consequence — where the router today runs (confirmed in ggml-ork supports_op): the MoE router is ALREADY on CPU (selection = softmax-over-experts + top-k/argsort has no supports_op case → always CPU; the gate MUL_MAT is declined at M=1 decode / N=n_expert rarely %64==0). So "move routing to CPU" is a no-op — the lever is OVERLAP, and it's now proven free. In serial M=1 decode the router is on the critical path (its input is the attention output → no independent NPU work to hide under). It becomes genuinely $0 only in an M>1 / speculative graph (DFlash multi-candidate, batched prefill, EAGLE-3 draft+verify) where the NPU always has another stream's work in flight — there the CPU computes stream B's routing while the NPU crunches stream A, with zero NPU gap. NOTE: ggml's scheduler is synchronous per backend split, so realizing this needs an async cross-stream submit path (the NONBLOCK+doorbell primitive), not just the CPU-side router. Probes on feat/npu-doorbell (overlap_prof).

So it AMPLIFIES the regime split (micro-ops ~6-19×; heavy ~5%) — and it's a bigger win than estimated. Bounds: doesn't touch C (compute+requant on the critical path); burns a core spinning; loses to HW-chaining where that applies (that already gets ~1µs/task in one submit). ★ Lead worth chasing: spin-not-sleep removes the ~130µs per-op overhead even in a SERIAL dependency chain (each op's wait becomes a busy-poll, no sleep/wake) — so it may help NPU DECODE (which pays that floor on ~200 matmuls/token and was shelved as latency-bound → CPU-path), NOT via pipelining but via cheaper per-op waits. Unmeasured end-to-end; a real lead given decode was the parity gap. Go-criterion: worth building (NONBLOCK is stock, no reflash) for any dispatch-bound path — fragmented micro-ops (MoE/streaming) AND possibly serial decode. Remaining to realize: DRAM-doorbell completion (coherent poll) + wire the requant into the spin; queue behavior + async correctness already validated.

Tier 12 — Heterogeneous CPU+NPU doorbell dispatcher (route to the most-efficient available resource) ⬜

The vision: one doorbell-driven scheduler that spins both the CPU worker pool and the NPU cores, and routes each unit of work (per-head / per-op / per-layer) to whichever resource is most efficient AND available right now — overlapping CPU and NPU rather than the current either/or backend split. The per-resource cost model is seeded by measured crossovers; occupancy/queue-depth breaks ties so no resource idles while the other is saturated. Generalizes Tier 6's async round-robin stream (NPU-only, ~3×) and rides Tier 11's NONBLOCK doorbell primitive; it's the concrete engine behind Heterogeneous-Serving-and-Scheduling.

✅ 12a. Concurrent multi-core chain dispatch (ork_mm_run_chains_rr). N homogeneous fused chains dispatched round-robin across all NPU cores at once, each on its own per-core scratch (work-stealing over the worker pool). Coherent, make test-gated (chainrr_conc_probe). Prefill attention chains 2.5× over single-core NPU (chainrr_bench_probe, flat vs queue depth — DRAM-read bound, not overhead).
✅ 12b. Context-length crossover measured (the routing signal). attn_decode_bench_probe: decode attention (Nq=1/head) on the NPU rr chain vs CPU scalar — NPU is submit/weight-read bound at short context (tie @ L≈512) and pulls ahead as context grows (2.6× @L=1024, 4.2× @L=2048, flat NPU line vs linear CPU). ⇒ route long-context work to the NPU, short to CPU.
✅ 12c. Static gate landed as the placeholder (examples/decode.c, ORK_ATTN_NPU_MIN_CTX, default keeps short contexts on CPU). This is the seam the dispatcher replaces — a per-call, either/or, context-length-only branch.
⬜ 12d. The dispatcher proper. Replace the static gate with a scheduler that (i) owns both pools, (ii) routes by the cost model (context length + op shape) and live occupancy so CPU and NPU run concurrently/overlapped, (iii) uses the doorbell NONBLOCK submit so the NPU side is async and the CPU side isn't blocked waiting. Needs an async cross-stream submit path (Tier 11) + the CPU threadpool made dispatchable alongside the NPU cores.
✅ 12e. Numeric enablement — DONE via HOST-SPLIT (examples/decode.c:attn_decode_npu, tools/attn_decode_bench_probe). The fused chain can't do per-head max-subtraction (no host step between on-chip QK^T→exp; set_i8_out8 has no bias; the only lever is the exp SDP idx_off at npu.c:7430, which needs on-device fixed-point calibration — deferred to the prefill/fused track). Instead: NPU does the two matmuls (QK^T, e.V) per head, host does the [1,L] softmax in fp with a real per-head max-subtraction. Correct for ARBITRARY (signed) scores — validated coherent (rel-err 0.0053 @ L=512, NH=32), no post-max precondition. Underflow fix: quantize softmax weights with ws=127/max(weight) (a fixed ws=127 rounds the ~1/L weights to 0). int8 caveat handled: the 0x1040 sched zeroes output for K<256, so the e.V leg guards L≥256 → CPU below that.
✅ 12f. Resident K/V — payoff proven + append API BUILT & VALIDATED (ork_kv_resident_alloc/ork_kv_append/ ork_kv_resident_free in npu.c; kv_append_probe). Per-key append writes only the new key's tile bytes into the packed weights (no repack): validated BIT-IDENTICAL to full-repack (max|diff|=0) at L=512 (single-tile V) and L=2048 (multi-tile V), coherent vs CPU (rel-err 0.003-0.004). Bug found: the M=1 matmul reads the full-K blob Bf, not the K-sliced Bb — must write Bf (authoritative). Remaining: wire ork_kv_* into a real decode KV-cache lifecycle (per layer/head resident weights, append per token) + lift the gate — folds into 12d. Below is the earlier payoff evidence that motivated it:
📊 12f payoff (resident-K/V perf model). Per-step repacking is the whole problem: with K/V resident (pack-once, per-step = matmuls only) the host-split decode attention is coherent (rel-err 0.005, arbitrary scores) and beats CPU 1.88× @L=512, 3.40× @L=1024, 7.32× @L=2048 (NH=32, HD=128 — attn_decode_bench_probe), vs perf-NEGATIVE (~15× slower) when repacking every step. So resident K/V is decisively worth it. Remaining = a library-level resident-KV-append API (a probe can't do it — writing the new key into the packed tile needs an internal bsync). Design (tiling bb[nt·KT·1024 + kt·1024 + nl·32 + kk], nt=n/32 kt=k/32 nl=n%32 kk=k%32): K^T appends cleanly — K=Kp=512 fixed so KT=16, a new key is a new N-column (nt=key/32) that appends at the end; the QK^T matmul runs N=roundup(L,32) (only current keys). V needs fixed-stride — its K=L (contraction) grows, changing the nt-stride, so size V for a chunk/Lmax (KT fixed), zero-pad, append the new row at kt=key/32, and run e.V with K=chunk (zeros beyond L contribute 0); re-tile only on chunk growth. Scales: per-key ks for K (host dequants each score column), uniform vs for V (a per-key V scale would mix in the e.V sum). Then wire the resident buffers into the decode KV-cache lifecycle + lift the gate. Folds into the 12d dispatcher (resident KV + concurrent heads).

Tier 13 — Weight-stationary matmul dataflow (reuse via `WEIGHT_REUSE`) ⬜ REOPENED 2026-09-05 — a narrow envelope is PROVEN and worth up to 1.52×; needs characterising before it can ship

⬜ OPEN 2026-09-06 — cross-M-tile `WEIGHT_REUSE` is SOLVED in principle; realising it is engineering

reuse correct ⟺ K × segw ≤ WEIGHT_BANK banks × 32 KB — exact, verified at three ratios, and the same rule the vendor's conv usage relies on. The earlier "withdrawn / corrupts always" note was written before the corruption was characterised and is wrong.

MEASURED 2026-09-06: the fix works; the gain does NOT clear the noise floor. With 2 runs per cell across M=144…256, within-config spread is 6.7–7.0% against a 2–5% between-config difference — the effect is real in direction but unresolvable at this sample size, and an earlier single-run "5.0%" was a noise excursion (2.0% on repeat). Correctness is solved and verified; the payoff is not demonstrable.

Superseded detail — Widening WEIGHT_BANK 4 → 11 makes a 49.8%-wrong shape 0.0% wrong. Timed against the default config at K=2048 N=384: 5.0% / 3.8% / 3.8% at M=256/512/1024 — consistent in direction, flat in relative terms, each cell inside the ~10% noise band. Not worth the N-tiling scheme plus an 8× mcap collapse plus full revalidation, for zero end-to-end value today. Parked, not closed — revisit if per-program cost drops (ork_pc_*: host 8.5 → 0.8 µs) or a model appears with shapes in the envelope.

START FROM THE 4/8 SPLIT, NOT THE BANK WIDENING. The 2026-09-06 experiment widened WEIGHT_BANK 4 → 11 to buy capacity, then spent it on wider segments — which forces DATA_BANK to 1 and collapses mcap 128 → 16, i.e. 8× the programs. Keeping the default 4/8 split and instead N-tiling to segw ≤ capacity/K (64 columns at K=2048) gives the same total weight saving with ~4 programs instead of 16, and no mcap change at all:

config (K=2048, N=384, M=256)	capacity	mcap	programs/core	weight traffic
baseline, no reuse	—	128	2	512 KB
11 banks + reuse (tested)	352 KB	16	16	256 KB
4/8 + N-tile to 64 cols + reuse	128 KB	128	4	256 KB

Correctness at 4 banks is already proven (N=192, segw=64 = exactly 128 KB → 0.0% wrong). The blocker is that colsplit gives each core ONE segment and only subdivides at N > NMAX, so this needs real N-tiling — it was not measurable here for that reason, not because it failed.

Work, if resumed, in order: 0. Implement N-tiling within a core segment (segw ≤ WEIGHT_BANK·32 KB / K), keeping the 4/8 split.

Replace the placeholder gate (K==4096 && Sn==1, which is not the predicate) with the capacity rule.
Widen WEIGHT_BANK via 0x1040 (4 → 11 banks = 352 KB). This is the load-bearing step: at 4 banks the extra per-program cost of N-tiling (~200 µs) exceeds the traffic saved (~133 µs); at 11 banks (~60 µs) it wins. Check what the reduced DATA_BANK does to mcap.
N-tile the dispatch so each tile's weight fits, then make test + ork_ppl.
Measure end-to-end with ork_bench — today the value is zero, since no local model produces a shape inside the current placeholder gate.

⬜ BACKLOG — productionising cross-M-tile `WEIGHT_REUSE` (superseded by the above)

What is proven (tools/wreuse_probe.c, ORK_MTILE_WR=1, gated to K==4096 && Sn==1): on the colsplit M-tile path the bit is bit-exact against an exact int32 CPU reference and worth 26.4% at N=2048 rising to 33.3% (1.5×) at N=8192, the saving equal to the skipped weight bytes at 10.5 GB/s. make test passes with the gate on. This is wide-N prefill — the shape colsplit actually runs in production.

What blocks shipping it: applied outside the gate it computes WRONG — make test fails three test_matmul goldens ({256,3584,3584}, {256,2048,2048}, {96,3584,18944}). K=3584 has the same mcap=64 as K=4096 and still fails, so the boundary is NOT M-tile geometry and is not understood.

⛔ MEASURED END-TO-END: the gain is currently ZERO, because the envelope matches no local model

ork_bench on qwen3.5-0.8b, 512-token prefill, ORK_MTILE_WR=1 with the application counter on:

reuse applications: 0
[ORK BENCH] prefill: 512 tok in 2949.8 ms = 173.57 tok/s (ubatch=512) | decode: 3.64 tok/s

The gate is K==4096, and no model on the board produces K=4096. Every K a real matmul sees here:

source	K values present
attention projections (`n_embd`)	1024 · 1536 · 2048
FFN down (`n_ff`)	512 · 3072 · 3584 · 6144 · 8960

(qwen3-1.7b 2048/6144 · qwen3.5-0.8b 1024/3584 · qwen3.6-35b-a3b 2048/512 · qwen25-1.5b 1536/8960 · qwen3-0.6b 1024/3072.) 4096 appears nowhere.

So item 1 below is not a refinement — it is the entire value of the finding, and item 4 is blocked on it. And the gap is worse than it looks: the nearest K a real model uses is 2048, which is one of the shapes that FAILS ({256,2048,2048}, 1830 wrong outputs). The road from "proven" to "useful" runs directly through a known-broken configuration.

Work items, in order:

Characterise the envelope — THE BLOCKER. Sweep K ∈ {512…4096} × N × Sn with the probe's CPU-reference check and map exactly where reuse is correct. Until the boundary has a mechanism, any gate is a superstition that happens to pass the suite. Start from what distinguishes K=4096 from K=3584 at equal mcap — candidates: the synthesised 0x1040 value itself, Sk (K-slice count), the nt_sz weight-tile alignment, or whether K is a power of two.
Replace the hardcoded gate (K==4096 && Sn==1) with the derived predicate from (1).
Default it on inside the envelope — make test + ork_ppl for quality, not just the probe.
Measure end-to-end, not per-matmul: 1.5× on the wide-N projections is worth far less than 1.5× overall, and the prefill share is what decides whether this matters. Use ork_bench, per Benchmark Standards.
Then reconsider the int4 path — reuse is already default-on there (ORK_I4_WREUSE); if the int8 envelope has a mechanism, it may widen int4's "within an expert only" restriction too.

Do NOT widen the gate without re-running make test on the SBC. The failure mode is silent wrong output, which the probe caught only because it checks against a CPU reference.

Full record: Exp-2026-09-05 Session Findings · Experiment Log 2026-09-05.

Original 2026-07-30 investigation (superseded in part by the above)

Follow-on to the M-fold (output-stationary) prefill matmul: the fold is weight-DMA-bound — every M-tile submit re-streams the whole K×N weight. The question: can a weight-stationary dataflow (keep the weight CBUF-resident, reuse it across M-tiles via the 0x1040 WEIGHT_REUSE bit, host/on-chip K-accumulate) read each weight byte from DRAM once and win? Investigated end-to-end board-free first, then measured (tools/re/ws_model.c + three board probes). Verdict: it sits right AT break-even — not the clear loss an early mis-measurement implied, not a clear win. Not pursued yet; documented so the next attempt starts from the real numbers.

WEIGHT_REUSE is real and the vendor uses it — for conv, not matmul. A generated big tiled conv (256×256×64, rknn-toolkit2 in a Colima aarch64 VM) captured under the rknpu_dump.so shim shows 0x1040 = 0x2a (WR=0) loader tiles + 0x202a (WR=1) reuse tiles — first observed vendor use of the bit. It works because the conv weight (~36 KB) fits WEIGHT_BANK and stays resident across spatial tiles. An LLM matmul weight is K×N ≈ 4.36 MB/slice, ~76× the ~57 KB CBUF — it can never be resident, so rkllm/librknnrt/ork all re-stream it (WR=0). Reuse for matmul therefore requires tiling the weight into CBUF-resident Kt×Nt blocks + K-accumulate.
Model (ws_model.c, pure math). Weight-stationary tiling needs ~100× more tiles than the fold (686–20 706 vs 7–29). Two accumulate variants bracket it: WS-dram (safe, DRAM K-accumulate) is a wash even at zero per-tile cost (×1.04 — weight-read-once saving is cancelled by activation re-stream/N-tile + int32 partial-sum traffic); WS-chip (on-chip K-accumulate, needs cross-task accumulator retention) is the only variant with upside.
Cross-core sharing is impossible. CBUF is per-core — the big-conv capture reused within each core's own task_number=15 stream, never shared. A multi-core M-split would 3× the weight loads.
Measured HW-chain per-tile cost (the decisive number). ORK_TRACE proved both run_chain_i8 paths (M=1 and M=128) issue one submit with task_number=S — genuine single-ioctl HW chains (an early "doorbell rings per task → 7 µs/task" reading was the wrong axis, with weight re-stream folded in). chain_marginal_probe isolates the true cost by sweeping weight size (S=32, M=1, N=32..1024): per_task = 4.38 µs floor + weight_DMA at 10.8 GB/s. The slope being exactly DRAM bandwidth proves the bulk is weight re-stream (removable by reuse); the irreducible floor is ~4.4 µs (chain transition + host per-tile handling).
Verdict at the measured floor. ws_model @ 4.4 µs (chained/overlapped): WS-chip ×0.98–1.03 = dead heat with the fold; @ 4.0 µs → ×1.07–1.13 (modest win, best on wide ops / large M like ffn-gate). WS-dram still loses (×0.78). So weight-stationary is break-even, tipping to a real win only if (a) the per-tile floor is partly reducible host overhead a lean chain submit (bypassing the ork_dyn wrapper) can shave below ~4 µs, and it grows with M.
~~Still gated on two unproven mechanisms~~ — both settled 2026-09-05 (Exp-2026-09-05 Session Findings): (a) cross-task WEIGHT_REUSE for an int8 matmul weight works in a NARROW envelope on the colsplit M-tile path (ORK_MTILE_WR=1, gated to K==4096 && Sn==1) — there bit-exact vs a CPU reference, 15% at N=512 to 34% (1.52×) at N=8192, and it does not require the weight to fit WEIGHT_BANK (works at 32 MB). Outside that envelope it computes WRONG and fails make test (K=3584, K=2048, Sn=3). So #39's verdict is refined, not refuted: the failure is real but not universal, and the boundary is unknown — K=3584 fails at the same mcap as K=4096. (b) on-chip accumulator retention is moot — K-slicing is monotonically WORSE (0.96/0.65/0.39× at S=2/4/8) in NPU time alone, because the output stage scales with S, so free accumulation would recover at most 0.39×. The weight-stationary framing was the wrong one: reuse across M-tiles at whole K needs no K-tiling and no accumulation at all.
Incidental finding — run_chain_i8 HW-chain envelope (chain_shape_probe): chains iff K%512==0 && K≤4096 (board-confirmed exactly); else rc=-3 → correct fall-back to per-task run_i8 (the synth_i8(sched=1) K-schedule is only valid there). Not a bug.
If pursued: (1) lean single-task_number=N chain submit to measure the floor without the ork_dyn host wrapper; (2) WR=1 correctness on a resident matmul tile; (3) cross-task on-chip accumulator retention. Probes: tools/re/{ws_model,chain_marginal_probe,chain_shape_probe}.c (+ superseded ws_xition_probe.c, gen_bigconv.py); ork-driver commits 0924a5d,7a7628b,c68dcc3,767b323,babb3be,c6a938a.

Tier 14 — Punch-list carried into the ork-driver refactor (2026-08-19) ⬜

Items left open at the refactor boundary, with the evidence that produced them. Line numbers will have moved — grep symbols. Landed-and-pushed context for this batch: the MoE prefill win is shipped (auto-profile, NF4 experts on CPU + int8 attn on NPU: pp512 36.9 vs native 23.3 = 1.59×, PPL 10.77 vs 10.21, zero env knobs; ork-driver f968b56, fork diverged-history 58f52c75f).

A. Driver correctness (highest value)

✅ ork_f16_mm_run_stream{,_chain} out-of-envelope M — DONE, and the envelope itself was wrong (ork-driver main 6584e2a, 2026-08-20; writeup: [[Exp 2026-08-20 fp16 M Envelope and the CBUF Bank Split|Exp-2026-08-20-fp16-M-Envelope-and-the-CBUF-Bank-Split]]). Measuring before guarding showed 0x1040 is a CBUF bank split (DBNK+WBNK==12, 32 KB banks), so the fp16 rule is banks × 32 KB / row-bytes: sched=0 → M ≤ 16384/K, sched=1 → M ≤ 180224/K (352@K512 / 176@K1024, vs the inherited 320/128). Found two live miscomputes: the stream's doorbell guard was the int8 byte budget (2× too loose at 2 B/elem → wrong output at non-pow2 K), and ork_f16_mm_run miscomputed at K=128 for any M ∈ [257,1472]. Now one orki_f16_mcap(K,sched) replaces five drifted copies and drives both TILING and refusal; only inexpressible cases (differing M across a batch) refuse. make test byte-identical. int8 needs no change (measured-correct at mg_max*64) — do not unify them; why fp16 attains the fractional bound and int8 does not is OPEN.
⚖️ ork_npu_exp_i16 row-count envelope — RE-SCOPED (2026-08-20, ork-driver b351c14). The claim does not reproduce against this op: ork_i16_npu_exp is bit-exact at all 37 probed points (M=8..8176 at N=64; N=8..2048 at M=512) — including M=4096 and M=6144 exactly, the values this item flagged at NRMSE 0.55–0.69. The knobs it cites to isolate the op (ORK_ATTN_TNORM_OFF, ORK_ATTN_MAX_CPU, ORK_ATTN_SM_SUMCPU) do not exist in ork-driver — they are fork-side, so that NRMSE was measured through the fork's attention composition, not this op. Its "good" 2e-4 figure is LUT error vs a CPU reference, not an op-vs-op comparison. Re-file against llama.cpp-rockchip. Probe: tools/re/i16_mcap_probe.c. ✅ A real bug WAS underneath it: the guard admitted M<=8192 but the measured ceiling is 8176 — M=8184 returns errno=110 (submit TIMEOUT) + a self-healing reset, so the guard was letting a FAULT-GENERATING shape through. 8192 was never measured; it is RK_DPU_DATA_CUBE_WIDTH's 13-bit field (mask 0x1fff) read as a capability — the same provenance as the fp16 bound. It is a row limit, not a size limit (8176 bit-exact at N=64 and N=128). Fixed via one ORK_SDP_MAXM constant applied to the 8 int16 sites. ✅ Follow-up DONE (3b1eb96): all 21 SDP guards now measured, per dtype — int8 SDP 8176 (exact; 8177 fails) and fp16 SDP 8188 (contiguous max; it frays 8189 BAD / 8190 OK / 8191 BAD). Two constants, since fp16 reaches 12 rows further. Zero M>8192 remain in src/. ✅ And int4 (6033334+bfefc64): H was 16384/K capped at an invented 16 → now CEIL(16384/K) capped 64, measured at six K. 2× at K=512, 4× at K=256, +1 row at every non-dividing K. ⬜ Next in this class: Wb=(131072/K)&~63 (int4, one line below the H fix) discards 25% of the N-tile at K=768/1536 and is UNPROBED — same shape, same production K. See Exp 2026-08-20 SDP and Int4 Envelopes for the ranked list.
⬜ ork_gptq_i4 is a DECLARED-BUT-UNIMPLEMENTED public symbol. include/ork_npu.h has shipped the declaration on main since the int4 pack-time API went in, but no implementation was ever committed — any caller got a link error, and the gap was invisible because nothing in-tree calls it. Found 2026-08-19 by diffing declared-vs-defined ork_* symbols across the headers (the only real hit of three candidates; ork__spine_unit_loop and ork_slice_caps are header-local). Now stubbed to -ENOSYS in src/ork_gptq.c (compiled via COBJ, deliberately not CORE/ATTEST — a pure-CPU pack-time quantizer determines no NPU output), with the header doc corrected to say so, so the symbol links and misuse fails loudly instead of silently. The real quantizer is written and parked on origin/wip/moe-saga-2026-08-18 (src/ork_gptq.c + examples/test_gptq.c, a mechanism check that GPTQ's H-weighted error beats RTN): error-compensated column-sequential int4 rounding, hand-rolled dense Cholesky, no external deps, native-W4A4 output so it composes with the Hadamard rotation (QuaRot). Blocked on task #56 — a byte/PPL cross-check against AutoGPTQ on a fixed (W,H) golden. Swap the stub for it when that passes; until then nothing should route through ORK_GPTQ. ✅ Lint landed (807b69a): check_registry.sh check 5 now fails the build on any ork_*/orkd_* prototype in include/*.h or src/*.h with no implementation in src/. Getting to zero false positives needed four discrimination rules, each earned from a real miss: a line is a prototype iff (comment-stripped) it has no { and ends in ; (an ends-with-; rule alone false-positived 13 one-liner bodies such as ork_npu_sram_free); the name comes from before the first (, not the last (else a one-line body calling another ork_* reports the callee — ork_dma_alloc_sram → ork_dma_alloc_flags); static in a header is a header-local definition that satisfies includers (ork__spine_unit_loop); and typedefs / preprocessor lines are skipped (ork_slice_caps is a struct tag, not a function). Verified on both BSD and GNU sed/grep: clean at 286 declared / 476 defined, and rc=1 when the stub is removed.
✅ dom_dirty is int4-only (fixed 2026-08-30, 5751e06). Set at the two int4 doorbell drop sites (one gated mc_dt == DT_I4), so an int8/fp16 doorbell drop never arms ork_dom_flush_if_dirty — a stuck job can still stall the next switch iommu domain for the full 60 s. Latent only because the per-domain attention pool removed the switching that exposed it. Landed and board-validated on 6.1.115-vendor-rk35xx-fence (make test ALL TESTS PASSED, attest refreshed in the same commit). The finding that makes this more than a one-word change: simply arming the flag for int8/fp16 would be worse than the bug. rknpu_job_timeout_clean reaps a job only once it has aged past its own submit timeout, and the doorbell dtypes differ by 40× — int4 submits bounded (orki_i4_submit_tmo_ms, ~1500 ms) while int8/fp16 submit with orki_mm_timeout_ms (60000 ms default). The flush's wait was hard-coded to the int4 number, so an int8 drop would wait 1.7 s, find nothing reapable, clear dom_dirty, and let the switch wedge anyway — with the defect now invisible. The wait therefore has to be keyed to the dropped job's recorded timeout (new dom_dirty_ms, capped by ORK_DOM_REAP_MAX_MS); int4's proven path is unchanged. Also corrects the internal.h comment claiming ACT_RESET "aborts+clears stuck jobs" — domain.c states the opposite ("ACT_RESET can't") and is the version that actually worked.

B. Discoverability

✅ The GDN verdict is invisible from main (fixed 2026-08-30, c425f00) — grep -i gdn OPS_REGISTRY.md AGENTS.md = 0 hits; it lives only on branch feat/gdn (GDN_ON_NPU_WIP.md). A fresh investigation on 2026-08-19 nearly rebuilt ork_gdn_scan_f32 from scratch before finding it. Add a registry row pointing at the branch. (Recap: the on-NPU GDN scan is coherent and byte-identical end-to-end but 1.9× slower (0.8B Q8) → 4.8× (9B Q4) than ggml's CPU scan, because the chunked form needs a CPU forward-substitution UT-transform (I+A)⁻¹ = 38% of the scan that ggml's solve-free per-token recurrence never pays; plus small-matmul submit overhead; plus no scale crossover — GDN state is a cache-resident 128×128/head, unlike Mamba-2's memory-bound recurrence that gave SSM its 2×.) Done — and the gap was wider than this item recorded. The registry had no sequence-scan section at all, so ork_ssm_scan_f32 was missing too, despite being on main, default-on above the fork's nh>=64 gate (ggml-ork.cpp:9844), and covered by test_ssd_chunk_npu in make test. Both now have rows: Mamba-2/SSD = PROVEN (byte-coherent, ~2x at Q8_0 2.7B) and GDN = WIP (coherent, loses ~1.9x@0.8B / ~4.8x@9B). They are deliberately two rows: the SSD result does not transfer — GDN's chunked form adds a CPU UT-solve (38% of scan) the CPU recurrence skips, and its state fits cache, so there is no crossover to cross. check-registry rejected the first draft for citing test_gdn_chunk_npu, which lives on feat/gdn and cannot be run from main; the row was reworded to say it has no probe in this tree rather than exempted.
⬜ Fold the two scratch WIP docs in the fork (NF4_ROUTE_WIP.md, FP16_BMM_DOMAIN_WIP.md) into wiki pages, then delete them (AGENTS: don't commit stale scratch). Also unpublished: experiment-log entries for big+little (PARITY), int4 expert-coalesce (1.6× at the matmul, unwired), GPTQ (unvalidated, task #56 — now stubbed on main, see Tier 14 A), and the flat ORK_MOE_MPAD/ORK_I4_M1_NPU A/B knobs — recoverable from ork-driver branch wip/moe-saga-2026-08-18.

C. Owned by the consuming repos — tracked on their roadmaps, not here

The remaining items from this batch are backend/model-prep concerns, not driver internals. They are recorded where they belong so this page stays driver-scoped:

fork llama.cpp-rockchip → SBC Performance Optimization Roadmap: the shipped knob-free MoE auto-profile, the res=-3 pp512 blocker, the ORK_ALLOW_Q4_SRC speed probe for the Q4_K ssm_* projections (supports_op lives in ggml-ork.cpp), the ORK_OPPROF per-op CPU profiler (ggml/src/ggml-cpu/ggml-cpu.c), and folding the fork's two scratch WIP docs.
oRKLLM → Large-Model Streaming & .orkpack Conversion: the GGUF quantize recipe (--tensor-type attn=q8_0 --tensor-type shexp=q8_0, candidate ssm=q8_0), the attention-precision ladder, the sbits < 5.0 rule that makes sub-5-bit tensors NPU-ineligible, and the .orkpack path now being derived from the model (no ORK_PERSIST) with one pack scheme per model type.

D. Driver loose ends

ℹ️ Board-ops: after repeated 18 GB loads (and after make test) weight import can fail with PRIME_FD_TO_HANDLE: Cannot allocate memory with GB free — page-cache pressure, not a code bug; sync; echo 3 > /proc/sys/vm/drop_caches, and a reboot clears the stubborn cases.
⬜ Board disk hygiene: qwen3.6-35b-a3b-o4n1.orkpack (~16 GB, contaminated/superseded) is deletable; the other 18 GB packs + f16/mixed/q4km GGUFs are retained deliberately.

Tier 15 — Load the model from the .orkpack, retire the source GGUF ⬜ stage 3 (eager residence + ready event) IN PROGRESS

Problem. A packed run currently opens BOTH the source GGUF and the .orkpack, so every matmul weight is resident twice: once as source bytes, once as packed int4 in IOVA. At Qwen3.6-27B that is a 28.6 GiB GGUF plus a 12 GiB pack = 40.8 GiB working set on a 31 GiB board.

It does not fail as an OOM, which is what makes it expensive to diagnose. The GGUF is mmap'd, so pages churn instead: submits time out, the driver self-heals, and dmesg fills with RKNPU: soft reset at ~100 ms intervals. It reads as a driver fault. Measured 2026-08-23 while bringing up 27B — the same command succeeded once and failed once, and the difference was page-cache state, not anything in the request.

Shipped now (stage 1): the stub GGUF. ork_write_stub_gguf emits <pack>.gguf beside the pack at build time. Header + KV + tensor-info are copied from the source verbatim, so every tensor keeps its exact data offset and llama.cpp loads it with no loader change; only the tensors the pack does not own are copied into the data section, and the rest are left as file holes. Apparent size is unchanged, blocks used are a fraction. Guard: ork_stub_verify aborts if a tensor reads as all zeros and the pack cannot serve it — the signature of a stub run without (or with a mismatched) pack. ORK_NO_STUB=1 skips emission.

This removes the duplication but keeps a GGUF in the picture, and the stub still carries the embeddings / norms / output head (~2.6 GiB at 27B) because llama.cpp requires every declared tensor to exist.

Stage 2 (this tier): an orkpack-native loader. Make the pack the whole artifact.

Extend the pack with a metadata section: the GGUF KV block (arch, hyperparams, tokenizer) plus the raw non-packed tensors. The pack index already carries name/K/N/dtype for everything it owns, which is enough to declare those tensors' shapes.
Teach llama_model_loader (fork-side) to accept an .orkpack as a model source: build gguf_context- equivalent metadata from the embedded KV, declare packed tensors from the pack index and bind them to the ORK_Weights buffer type, and serve the rest from the raw section.
Drop the stub path once this lands — keep ork_write_stub_gguf only as the escape hatch for running a pack against stock llama.cpp.

Why it is worth a tier. It is the difference between "27B fits" and "27B fits with 12 GiB of headroom wasted on a copy of the weights we already packed", and it removes an entire class of misdiagnosis: with one artifact there is no page-cache-dependent second copy to churn. It also makes a pack self-describing, which is what the persist/streaming work has wanted since multi-domain landed.

Stage 3 (added 2026-08-24): EAGER residence + a readiness event. Stage 2 makes the pack the whole artifact; it does not change when weights materialize. They are still loaded lazily on first matmul, which means the load lands inside whatever the consumer is timing.

This is not a theoretical tidiness point — it has corrupted measurements repeatedly, and on 2026-08-24 it produced a headline number that was simply wrong. Profiling the 27B G=512 run (gdb poor-man's sampler; the board has no perf) put 30 of 30 stack samples inside ork_i4a8_mm_load_tiled, hot frame orki_i4_untile_blob — i.e. the entire "scored" window was weight load, un-tile and int4->int8 inflate, not compute. The run was reported as "220 s to score 63 tokens, 12x slower than CPU"; the CPU control mmaps a GGUF already in its native format and pays no such cost, so the comparison measured loading, not throughput.

Requirements:

Load every weight the model will use at model-load time, not on first matmul — walk the pack index and materialize into domains up front.
Emit a readiness signal the consumer can wait on: weights resident, count, bytes, domain layout, and the load duration. A log line at minimum; an API/callback so ork_bench / ork_ppl can start their clock after residence.
Report load and steady-state separately everywhere. A single-window perplexity run must not fold a one-time 400-weight inflate into its scored time.
Fail loudly if a weight is still loaded lazily after the eager pass (it means the pass missed a tensor the graph actually uses) — the same "loud on pack-miss" discipline as ork_pack_miss_check.

DESIGN CORRECTION (2026-08-24, user): residence must be an OP-LESS operation, and PLACEMENT belongs in the pack. The first implementation attempt used a warmup decode to trigger residency as a side effect of computation. That is backwards, and the attempt failed three times in instructive ways — a graph walk found 1 of ~400 weights (ggml hands the backend one SPLIT at a time, 641 of them), a set_tensor hook fired ZERO times (ORK_Weights holds only the ~11 MiB compute buffer; the weights live in CPU buffers, so ork never receives a per-tensor load event), and a 1-token warmup materialized nothing at all (M==1 is deliberately declined to the CPU). Each failure is a symptom of the same thing: residency is not a first-class operation.

The right shape:

ork_pack_open(path)    -> index: name, K, N, dtype, AND placement
ork_pack_reside(pack)  -> import/map every entry into its assigned domain.
                          No graph, no ops, no submits.

The pack carries placement — per-entry domain id, tiling, group size — decided ONCE at pack time, where the whole model is visible, instead of re-derived at runtime by a byte-balanced fill whose result depends on visit order.
An op-less reside entry point maps everything and returns. Its return IS the readiness event: no callback plumbing, no warmup proxy, nothing to keep in sync.

This subsumes several open problems rather than adding a feature:

Non-deterministic domain layout (the 46 vs 4 t/s coin-flip on 9B W4A8) — layout becomes a property of the artifact, reproducible run to run.
The n_domains > kernel-max bug (2026-08-24: a 22.66 GiB footprint asked for 24 domains against the driver's 16, filled 0..15, then EINVAL'd off the end) — a pack-time planner validates against RKNPU_MAX_IOMMU_DOMAIN_NUM once and fails at BUILD time, not mid-inference at layer 48.
The Bf auto-shed (keep 22.66 GiB Bb-only vs 40 GiB with-Bf) — also a placement decision currently made at runtime from a RAM-budget guess.
Lazy loading, which disappears by construction.

GGUF compatibility (user, 2026-08-24): two roads, not one loader serving both. A plain .gguf keeps working through stock llama_model_loader, simply without ork's optimizations — lazy, unaccelerated, or whatever today's path does. The orkpack-native loader is the optimized road and owes the GGUF path no byte-compatibility. This removes the hardest constraint from stage 2: the pack does not have to reconstruct a gguf_context faithfully enough for the stock loader, it only has to describe itself well enough for ork.

LANDED (2026-08-24): deterministic placement. The planner is in (ggml-ork 7d002965f). Placement is computed once at pack open over the complete index, in canonical (layer, name) order, instead of incrementally in graph-visit order. Measured on Qwen3.6-27B — PPL bit-identical in all three tiers and the layout identical across repeated runs:

tier	domains	worst domain	PPL
W4A4 RTN	11	1.06 GiB	10.6780
W4A8 ra8	13	1.18 GiB	10.4421
W4A8 G=512	13	1.77 GiB	9.7485 (was filling 16)

LANDED: residence is queryable and excluded from timings. ggml_backend_ork_residence answers what is resident (from the wcache — an early version with hand-placed counters reported "0 weights" on a run with 400, because it instrumented only the int8 resolve), and ork_ppl warms full-width before starting its clock. A 1-token warmup is useless here: M==1 is declined to the CPU and materializes nothing.

STILL OPEN: the op-less ork_pack_reside() entry point, and stage 2's self-describing pack. Residence today is still triggered by running a real forward pass; the warmup makes that measurable, not absent.

Why it is worth doing before more optimization work. Every throughput number on a >4 GiB model is currently suspect, and the bigger the model the worse the distortion — which is exactly the regime the whole multi-domain effort exists to serve. It also removes the page-cache dependence that makes the same command succeed once and fail once.

Prereq/interaction. Nothing blocks it. It should land after the Sn>1 int4 M-ceiling work (Tier 14), since both are on the 27B bring-up path and that one gates whether large-M prefill runs at all. Stage 3 is independent of stage 2 and can land first — it needs no pack-format or llama_model_loader change, which is why it is the sensible starting point.

Tier 16 — W4A4 quality: grouped scales + int8 activations ⬜ TOP PRIORITY (quality)

The measurement that sets the priority. First absolute comparison against an external baseline — same 0.8B, same machine, same 256-token screen, CPU-only (ORK_OFFLINE=rk3588 ORK_OFF=1):

	PPL	excess over bf16
CPU bf16	11.7783	—
CPU Q4_K_M	12.0266	+0.25
CPU Q4_0	12.9964	+1.22
ork rot-int8 (best tier)	14.8577	+3.08
ork pure W4A4	15.9011	+4.12

Read as excess error over fp, which is what isolates quantisation cost: our W4A4 introduces ~3.4x the error of ggml's plainest 4-bit format, and our best tier is not even pure 4-bit (it promotes 28 weights to int8) yet still loses to Q4_0 by 2.5x. Q4_K_M (+0.25) is not the fair comparison — super-blocks plus higher-precision outlier tensors — but Q4_0 is, and we lose to it decisively.

Cause: scale granularity. Q4_0 carries one scale per 32 weights; W4A4 carries one per channel (K = 1024..3584). A 32-112x coarser grid accounts for a 3.4x error multiple on its own.

Why this outranks everything else on the quality list. Activation clipping (-20.9%), weight clipping (-3.3%), GPTQ (-34% vs RTN), Hadamard rotation and tier promotion all optimise within the grid. They reshape the distribution handed to the quantiser; they do not add resolution to it. No combination of them reaches Q4_0, let alone passes it. This is a ceiling, not a gradient.

The obstacle is performance, and it is a different class of problem. The shipped grouped path measures 13.2x (M=1) and 283x (M=128) over per-channel. (The earlier "1.18x" is RETRACTED — it timed a hand-built proxy chain, not the shipped path.) That is bad, but it is a dispatch/layout problem with known mitigations, whereas the granularity gap cannot be clipped, rotated or error-fed away.

MEASURED (2026-08-23): item 1 done — granularity is worth 38%, and it is HALF the gap

config	PPL	excess over bf16
CPU Q4_0 (target)	12.9964	+1.22
G=32 GPTQ	14.3428	+2.56
per-channel GPTQ (prior control)	15.9011	+4.12
G=32 RTN	16.6911	+4.91
per-channel RTN	26.2782	+14.50

G=32 cuts quantisation error 38% (+4.12 → +2.56, PPL -9.8%) for +24% disk (191 vs 154 MiB), 2 min to build. Confirmed as the largest single quality lever measured.

But it is roughly HALF the gap, not the whole ceiling — Q4_0 remains 2.1x better on excess error, so the original framing of this tier was over-attributed and is corrected here.

GPTQ and granularity are complements that overlap heavily. Each alone takes 26.28 to ~16 (-37 to -40%); together to 14.34 (-45%). They attack the same error source, so they do not add linearly. Note G=32 RTN (16.69) is WORSE than per-channel GPTQ (15.90) — a fine grid does NOT make GPTQ redundant.

The other half is probably ACTIVATION WIDTH — and testing it is BLOCKED

Q4_0 is effectively W4A8: ggml quantises activations to Q8_0 (8-bit, per-block scales) for the dot product. Ours is W4A4. Section 2 of the PPL-journey log established that the weight and activation error halves are at parity and independent, which predicts almost exactly the 2x that remains. That is prior evidence, NOT a new measurement — the direct test could not be run:

grouped scales + int8 activations is structurally unsupported today. The grouped branch hardcodes 4-bit activation quant (s = mx/7.0f, clamp [-8,7], or ork_i4_scale_mse which is a 4-bit search) and dispatches to ork_i4_mm_run_grouped, which is int4-weights-only. A G=32 + rotated-i4a8 pack builds fine and then refuses at run: grouped W4A4 run REFUSED (K=1024 N=6144 G=32 M=64). The refusal is correct behaviour — it declines rather than emitting garbage.

Prerequisite for settling this: a grouped INT8 run path — per-group accumulation with int8-container weights and int8 activations. Until it exists, the second half of the gap stays a hypothesis.

The quantiser is SOLVED to within 7% — measured like-for-like (2026-08-23)

tools/re/ork_qerr.cpp, same tensors and activations, fp64 reference, matmul level:

scheme	error	vs Q4_0
ggml Q4_0 (W4 x A8)	0.09144	—
ours per-channel W4A4	0.58207	6.4x
ours G=32 W4A4	0.20032	2.2x
ours G=32 rotated W4A4	0.14252	1.56x
ours G=32 rotated W4A8	0.09809	1.07x

Grouping, rotation and int8 activations together close 6.4x down to 1.07x. The tier's original framing ("per-group scales are the ceiling") was wrong in two ways: granularity is ~half the weight-side story, and activation width is the single largest remaining lever (0.14252 -> 0.09809, -31%).

So the target config is G=32 + rotated + int8 activations, and shipping it is now a RUN-PATH problem, not a quality problem.

Work items, revised again:

~~Price the quality win.~~ DONE — 38% of PPL excess, 41% of weight error.
~~Grouped int8 run path (offline).~~ DONE — ork_i8_mm_run_grouped + orki_cpu_gemm_grouped (shared with the int4 twin so they cannot drift). Offline exact; on-device refuses.
ON-DEVICE grouped int8 — now the shipping blocker. Per-group accumulation needs K-sliced int8 submits with an fp32 combine, which the int8 doorbell does not do. Until this lands, the quality-competitive config runs offline only.
Group size vs cost curve (G=64, G=128). G=128 is 4x fewer scales and may hold most of the quality; the scale array is per-(channel, K-group), so disk and layout cost scale with K/G.
Find where the 283x goes. ork_i4_mm_run_grouped is ~13x off its own achievable dispatch cost and ~2.9 ms per int4 submit at M=128 is ~17x the 167 us floor. Possibly one bug. Sequence with item 3 so both grouped paths are optimised together.

Split out: why 1.07x numerics gives 1.84x excess PPL

Our best config is 1.07x Q4_0 at the matmul level but 1.84x on excess perplexity. Depth compounding is RULED OUT (matmul error flat at 0.008-0.041 across 24 layers, near-zero bias). Untested confounds: Q4_0 quantises token_embd while our run leaves it bf16 (an advantage we hold and still lose with); llama-quantize keeps output.weight higher; and perplexity is driven by worst-case tokens, so the MEAN matmul error may simply be the wrong statistic. Next: measure a tail statistic, not a mean. This is a separate investigation from the tier above and should not gate it.

Interaction. Orthogonal to Tier 15 (orkpack-native loader) and to the Sn>1 int4 M-ceiling (Tier 14), but it lands on the same ork_i4_mm_run_grouped path as item 2, so sequence 2 before 4.

Tier 17 — Multi-domain teardown leaks IOVA until reboot ✅ FIXED 2026-08-24 (kernel patch 03)

ROOT CAUSE: the vendor driver never records an IOMMU domain for dma-buf imports. rknpu_gem_prime_import_sg_table() leaves iommu_domain_id at its kzalloc'd 0, and rknpu_gem_create_ioctl() returns early on an existing handle — so MEM_CREATE's iommu_domain_id is silently ignored for every import. The sg was mapped during PRIME_FD_TO_HANDLE against whatever domain was attached then; destroy switches to the recorded 0 and unmaps from the wrong page table. Full write-up + diff: Kernel Modifications, Change 3.

Single-domain is unaffected only because attached == recorded == 0. Nothing in userspace can fix it: the uABI offers no way to make the domain stick.

	before	after patch 03
`WARN_ON` per multi-domain run	30	0
consecutive multi-domain runs before failure	2	5+, no degradation
PPL (0.8B int4 / int8)	36.3973 / 17.5502	identical

Everything below this line predates the driver read and is kept for the record; two of its conclusions are WRONG. The "switch timeout causes the bad unmap" mechanism is refuted — our own runs show switchTO=0 with 30 WARNs. The "fixed set of per-domain SCRATCH buffers" identification is also wrong — the failing frames are drm_prime_gem_destroy, i.e. IMPORTS, and int8 (250 non-zero-domain imports) leaks identically to int4 (26). What survived: the bisect exonerating our kernels, and the 15-second reproducer.

Superseded investigation (2026-08-23/24)

Symptom. A >4 GiB model can be measured ONCE per boot. Fresh board: 27B loads 400/400 and scores. After a few runs the same command dies at ~40 weights with RKNPU: failed to allocate IOVA: -12, on a board with 12 GiB free. Only a reboot restores it. Single-domain models never show it.

Mechanism (kernel evidence, Exp-2026-08-23 §5R): rknpu_gem_object_destroy calls rknpu_iommu_domain_get_and_switch — the driver picks the object's domain on destroy. Under multi-domain teardown churn that switch TIMES OUT (switch iommu domain time out, id: 1); the unmap then runs against the wrong domain, trips WARN_ON(unmapped != size) in __iommu_dma_unmap, and that IOVA is never reclaimed — it outlives the process and accumulates until reboot.

The unmap-after-failed-switch is KERNEL-side. Userspace can reduce switch pressure; it cannot make that unmap correct. Confirmed the hard way: activating the domain caller-side before destroy (on the wrong theory that MEM_DESTROY does not) DEADLOCKS — two switches race, the process enters uninterruptible D state, and timeout cannot kill it. Reverted; do not re-apply.

Already fixed (the half that was ours). The domain sizer under-counted mixed-precision packs — one global qbits instead of per-entry dtype, missing the 2x on int4-on-disk weights that inflate to int8 containers at load (~3 GiB on 27B), plus a Bf threshold of 4096 against the loaders' 10752. That is why a domain overflowed in the first place. Now a single driver-side authority, ork_w_resident_bytes (ork-driver 587efff, consumer 95255a418); mixed 27B plans 16 domains, was 12.

RESOLVED TO A CAUSE (2026-08-24): stock vendor-driver behaviour, ~30 mappings per multi-domain RUN. Measured with a 15-second reproducer (0.8B pack + ORK_DOMAINS_FORCE, §5U): 1 domain leaks 0; 2/3/6 domains leak exactly 30; a 102-weight 155 MiB pack leaks the same 30 as a 400-weight 11.3 GiB one. It scales with neither domains nor weights — it is a fixed set of objects, almost certainly the per-domain SCRATCH (regcmd/task/Af/Cc + per-core mrc/mtk/maf/mcc, ~16 per domain). A three-way kernel bisect (§5V) shows stock, SRAM-only and SRAM+patch all leak exactly 30 — our kernel changes are EXONERATED.

Consequences: the leak is per-PROCESS, not per-workload, so it scales with how many multi-domain runs you start rather than how much work each does — an argument for a long-lived daemon owning the NPU. And it is worth reporting against drivers/accel/rocket now that the reproducer is 15 seconds.

~~PRIME SUSPECT (2026-08-24): our own SRAM kernel.~~ REFUTED by the bisect — kept for the record. The board runs 6.1.115-vendor-rk35xx-sram — CONFIG_ROCKCHIP_RKNPU_SRAM=y plus a source patch, both ours. That config makes the driver map the 956 KB on-chip SRAM into the IOMMU domain, which with MULTIPLE domains must be moved per switch — landing in rknpu_iommu_domain_get_and_switch, exactly the function that times out. It also explains what nothing else did: single-domain models never leak, and multi-domain fails on the SWITCH rather than the allocation. So the "kernel-side, not ours" framing above may be wrong. See Exp-2026-08-23 §5S.

Work items: 0. ~~BISECT against the stock kernel~~ DONE — all three kernels leak identically; ours are exonerated. 0z. BISECT against the stock kernel — /boot/RESTORE_STOCK_KERNEL.sh, two symlink flips, idempotent, microSD rescue documented. Leak gone on stock => it is ours. Leak persists => genuine vendor behaviour. This is now item ZERO: it decides whether anything below is even the right investigation. 0b. Read drivers/rknpu. Every conclusion so far was inferred from stack traces and dmesg; the vendor source is public and the .239 VM has the toolchain. Turns "the switch times out" into why. 0c. Recover what our kernel patch changed. .prepatch-bak exists with no record of the diff — a source modification to the driver under test, undocumented, is a hidden variable in every board measurement.

Measure the per-run leak rate properly. Three attempts failed — a wrapping dmesg ring used as a cumulative counter, a single-domain control for a multi-domain bug, and a self-inflicted hang. Protocol: dmesg -C before each run, count after; arms = successful vs failed, single- vs multi-domain.
Reduce switch pressure at teardown. If the timeout is contention-driven, freeing weights grouped by domain (rather than in wcache order) collapses ~N_weights switches to ~N_domains. Cheap to try, and it does not fight the kernel — it just asks for fewer switches.
Quiesce before teardown. The switch times out because the NPU is busy; an explicit drain/reap before the free sweep may let every switch succeed. ork_dom_flush_if_dirty already exists for the submit path.
Fewer domains. The sizer fix INCREASED the count (12 -> 16) for correctness, which raises switch pressure. Larger group sizes and fewer promoted weights cut residency and therefore domains — interacts directly with Tier 16, where G=256 W4A8 gives ~8x fewer buffers at equal quality.
Non-reboot reclaim (recovery, not fix). /sys/bus/platform/drivers/RKNPU/{bind,unbind} with device fdab0000.npu should tear down and rebuild the device's IOMMU domains in seconds. Untested — it needs an operator decision, since a failed rebind costs the reboot we were avoiding.
Upstream it. drivers/accel/rocket is the mainline open driver for this silicon; an unmap that proceeds after its own domain switch failed is worth reporting either way.

Progress log

(entries added as steps complete, newest first; each links to its experiment writeup)

2026-08-19 — [Tier 14] MoE prefill win shipped (1.59× native, knob-free) + three attention-path defects fixed; GDN offload closed twice over. Shipped: a routed-MoE model now auto-selects the measured-optimal scheme with no env knobs (experts int4/NF4 on CPU via a batched 4×4 NEON GEMM, attn/dense int8 on NPU at prefill, all-CPU decode; orkpack path derived from the model): pp512 36.9 vs native ggml 23.3 = 1.59×, PPL 10.7718 vs 10.2058 (+5.5 %), decode 6.3–6.6 vs 7.5 — an explicit trade, prefill-heavy workloads win. Biggest single decode lever was a file property, not code: GGUF attention precision (f16 38.4/5.26 · q8_0 36.9/6.59 · Q4_K 26.8/7.42 — below 5 bits the NPU declines the source entirely and the prefill path is lost). Driver fixes: ork_mm_repack_f16 left the fp16 multi-core colsplit's cached contiguous weight (Bbc) stale behind a never-cleared *_valid latch, so every multi-core submit after the first used the first weight packed into that slot; new ork_npu_active_domain() lets a transient scratch weight be placed in the already-active domain (a shared fp16 attention pool spanning dom0/dom1 forced two switch iommu domain per attention layer → 3×60 s stalls → pp64 30.3→1.01). ORK_ATTN itself: transposed B fill in bmm_fp16 (the 10.83→67.86 PPL blowup; proven by an emulated transposed-B product matching the NPU to NRMSE 1.9e-4), the M-envelope overrun above, and on-NPU softmax made opt-in ⇒ PPL back to 10.7607; still off by default (perf-neutral: 0.96×, NPU matmul is 11.6 % of the wall, marshalling costs ~2.4× the matmul time saved). New differential oracles tools/re/{fa_diff_probe,bmm_diff_probe}.cpp (build one node, compute on the ggml CPU backend as oracle vs the ORK backend, diff — ~20 s answers; bmm_diff_probe also carries a mechanism oracle that recomputes with a deliberately mis-indexed B). check_registry.sh's artifact scan was non-recursive and .c/.sh-only, so it could not see any tools/re/ harness — now recursive + .cpp-aware. GDN-on-NPU closed on two independent lines (this session's Amdahl/marshalling measurement + feat/gdn's 1.9–4.8×-slower measurement and its structural UT-solve cause); also established that llama.cpp emits GDN as a single fused GGML_OP_GATED_DELTA_NET, not chunk matmuls, so there was never anything to "claim and chain". ork-driver f968b56, fork 58f52c75f.
2026-07-30 — [Tier 13] Weight-stationary matmul dataflow — modeled + measured → BREAK-EVEN. Can keeping the weight CBUF-resident + reusing it (WEIGHT_REUSE) beat the weight-DMA-bound M-fold? ws_model.c (pure math) + three board probes. WEIGHT_REUSE is real but vendor-used only for conv (weight fits CBUF); an LLM matmul weight is ~76× the 57 KB CBUF, so reuse needs weight-tiling + K-accumulate → ~100× more tiles than the fold. ORK_TRACE proved run_chain_i8 is a single-ioctl task_number=S HW chain; chain_marginal_probe measured per_task = 4.38 µs floor + weight_DMA@10.8 GB/s (slope == DRAM BW ⇒ the rest is re-stream, removable by reuse). Feeding 4.4 µs into the model: WS-chip ×0.98–1.03 (dead heat), @4.0 µs ×1.07–1.13; WS-dram loses (×0.78). Break-even, not dead — a plausible modest win at scale iff a lean chain submit shaves the floor and two unproven HW mechanisms hold (cross-task WEIGHT_REUSE for a matmul weight + on-chip int32 accumulator retention). Also confirmed the run_chain_i8 envelope K%512==0 && K≤4096 (else rc=-3 → run_i8, a correct guard). ork-driver c6a938a (+ 0924a5d,7a7628b,c68dcc3,767b323,babb3be).
2026-06-26 — MoE saturation thread M3-skew — routing skew measured, hot-cache VIABLE. (MoE Routing Skew.) Profiled routing via the contiguous ffn_moe_argsort tensor (⚠️ the ffn_moe_topk view is non-contiguous → FALSE "uniform") on the UD-Q4 GGUF (skew is precision-invariant; router gate high-precision → q4 ≈ f16). Both SKEWED (Gini 0=uniform): LFM2-8B-A1B (32 exp, top-4) 0.44; Qwen3.6-35B-A3B (256 exp, top-8) 0.65 — skew rises with depth; per-expert int8 only 3.15 MB. Hit-rate vs cache (35B, top-K/layer): 5%=33%@1.6GB · 10%=48%@3.3GB · 20%=67%@6.4GB · 50%=92%@16GB. VERDICT: VIABLE — ~3–3.5 GB cache (top ~26–28/256) → ~50% of decode lookups NPU-resident within the 4 GiB window; cold tail → CPU/GPU; policy = per-LAYER top-K. Tool llama-moe-skew (~/llama.cpp/examples/moe-skew), data /tmp/skew_{lfm2,qwen36}.csv (box, CPU-only). Confirms the Branch-A decode partition.
2026-06-26 — MoE saturation thread M1a — routed-expert persist/load DONE + 4 GiB IOVA wall confirmed. (MoE Expert Persist + IOVA Wall.) The mixed-.orkpack now captures the routed _exps/MUL_MAT_ID experts (before: backbone+router+dense-shared only, zero routed). ggml-ork c89cf8654 (synthetic key <name>#<e>; persist all n_expert slices; load via ork_mm_load_i8/ork_mm_load_i4a8) + ork-driver 6b0d568 (ork_mm_load_i8 rebuilds full-K Bf, gated K%512==0 && K<=4096). LFM2.5-8B-A1B: 2112 _exps entries (264×8) vs 0; reload-from-disk NPU output bit-identical to a CPU int8 ref (889 experts; gate = reload-vs-CPU-ref since the live MoE path is itself broken on this model). ★ Confirmed the 4 GiB IOVA wall for real — int8 experts exceed 4 GiB → compute status -1 once ~1909 resident (NOT a persist bug; mandates M2 stream + M3 cache — resident-all-experts impossible beyond a tiny model).
2026-06-26 — MoE-on-NPU saturation thread M0 — per-expert GEMM VIABLE. (MoE Expert FFN GEMM — M0 viability.) Isolating the per-expert FFN GEMM with the new primitives (direct int4→int8 inflate ORK_DIRECT_I4 570cb68 + cacheable 0x403), the NPU beats CPU at every Qwen3-A3B expert shape and every per-expert M∈{8..128} — 3–10× at M=8, no crossover, bit-exact; the direct inflate cut per-expert materialization 16–30× so cost collapses to the GEMM. Dispatch: run_stream_i8 for chainable gate/up, run_i8 (already multi-core) for down. Caveat: 4-thread CPU narrows M=8 to ~1.5–2.5× but NPU wins + frees all 4 CPU cores. Flips the go/no-go Tier 5c's integrated ~7× couldn't (that was a CPU-side Q4_K-repack wall, not the NPU). Tool tools/moe_expert_probe.c (1840abb); chan_scales_f32 NULL-guard (980cea9). NEXT (M1a, in progress): persist routed _exps/MUL_MAT_ID experts in the mixed .orkpack (backbone already persists). See the MoE-on-NPU saturation milestone thread.
2026-06-26 — Tier 9 streamed-prefill fill chain — cacheable + direct-inflate DONE. (dma_buf Fill Bandwidth.) The flat ~1 GB/s fill was an artifact of benching ork_dma_alloc's write-combine ZC flag 0x401 (b53c59b) — production tiles use cacheable 0x403; benched right it's ~10 GB/s (8.3–8.8×), NPU correct. Direct int4→int8 tiled inflate (ORK_DIRECT_I4, 570cb68) skips the f32 round-trip, bit-identical, 2.2–2.7× UNIFORM / 3.6–4.9× NF4. Streamed prep is TILE-bound and the crossover is large-M (prefill), not decode. NEXT: clean-only bsync steady-state re-fill (9c); SRAM/NBUF tier (9d) identified.
2026-06-26 — Tier 8 mixed int8/int4-NF4 + imatrix quant — DONE, PPL-validated. (NF4, imatrix, Mixed Precision.) NF4 codebook (dc782b2, reconstruction 0.110 < uniform 0.158), stochastic rounding (a PPL effect not an RMS win), imatrix-weighted clip-ratio scale (93364b7, 6.4× weighted-error cut), compact 'O4N1' persist (d656955), and gguf_tier_map allocation oracle (2c38a27/f78bc04). PPL gate: fp16-source + Q4_K_M-allocation + imatrix + NF4 = 11.33 @ ~58% int8 storage vs Q4_K_M 10.92 / all-int8 11.19 / double-quant 12.48 → fp16 is the source, the mixed GGUF is allocation-oracle-only. Residual gap = the int8-activation cost.
2026-06-24 — int4 push: NEON repack fixed, run_stream_i4 built, Hadamard validated on-NPU. Three landed (ork-driver 2b0ee20, 64c2ec4, 1654064): (1) NEON-fused f32→int8 pack/repack (ork_mm_pack_i8_f32/repack_i8_f32) — quantize+tile directly from f32[N][K] in one cache-friendly pass (no transpose scratch, was ~69% of the MoE repack); validated bit-identical to pack_i8 (tools/pack_f32_probe.c, mism=0) — the earlier "wedge" was an inherited dirty-NPU state, not a pack bug (power-cycle to a clean NPU before diagnosing any wedge). (2) ork_mm_run_stream_i4 — int4 analog of the async round-robin stream: a task's M rows become M single-row regcmds PC-chained on its core, round-robined across cores; test_chain_i4 S=4 varying M verified vs CPU, full make test green. (3) Tier 4a Hadamard validated end-to-end on NPU (tools/hadamard_i4_npu.c): per-group block-Hadamard composes exactly with grouped int4 (mechanism err 1e-4), quant RMS 27.18%→14.75% (−46%) with outliers. Remaining for int4: fold the per-group rotate into ggml-ork's int4 quant + run a perplexity eval (4a's true go/no-go).
2026-06-24 — Cross-core stream lands (~3×); multi-core mechanism settled. Captured how librkllmrt uses the 3 cores (RKLLM Multicore Capture): round-robin single-core submits, 3–6 chained tasks each, never core_mask=0x7. Confirmed Option A (single multi-core submit) hard-wedges and is unused; built the async round-robin run_stream_i8 = ~3× (vs the barrier fan-out's 1.3×, now deprecated). Fixed a run_chain_i8 full-K-envelope correctness bug (K%512==0 && K≤4096; -3 + fallback otherwise). Cores caller-configurable (1/2/3 scaling). All committed (ork-driver bcb8266). MoE handler→stream wired + coherent but repack-bound (pp128 0.98 vs CPU 22.3 at the time — since improved by the NEON repack + fused dequant to 3.13, see below). Open high-ROI direction: EAGLE-3 verification (where the stream's ~3× pays off). int4 Hadamard + run_stream_i4 gated on Tier 4a coherence.
2026-06-24 — Tier 6 cross-core chain fan-out LANDED (correct, perf TBD). Option A (single core_mask=0x7 submit) hard-wedged the board (power-cycle); Option B (concurrent per-core single-core submits, mirroring run_multicore) works — ork-driver dc4038b, gated ORK_CHAIN_MC, mism=0 across 3 cores, no wedge. ~nc× speedup still unmeasured (needs a real batched workload, e.g. EAGLE-3 verification). DMA-buffer tasks fall back to single-core (coherency race, deferred). Also corrected the stale ORK_ZC_OUT docs (6d7bc2c): output zero-copy was fixed in 3fad74a (cache coherency), off for ~0 gain not bugs.
2026-06-24 — Tier 5c repack optimization — closed MoE from ~23× to ~7.2× under CPU (still not yet viable, but closing fast). After 5b chaining (pp128 0.28→0.97, 3.5×) the path was repack-bound (94%). Two repack levers then landed: NEON-fused quant+tile (ork_mm_pack_i8_f32, 2b0ee20) → 0.97→2.90 t/s (~7.9×); fused Q4_K→int8 dequant (ork_mm_pack_i8_dequant, b520ee7; ggml-ork b5bc5ce8b) eliminating the f32[N][K] staging round-trip → 2.90→3.13 t/s (+8%, ~7.2×). The "even a free repack can't beat CPU" claim from 5b was based on the old strided repack and is now obsolete. New wall: the NEON f32→int8 quant+tile itself (77% of MoE-chain time; the Q4_K→f32 dequant is now 0 ms, folded in) — closing further needs a cheaper quant+tile or int8-resident weights, not more dequant work. Experts still CPU-default. Chain + repack primitives are general wins on main.
2026-06-24 — Tier 5b chaining BUILT & MEASURED — 3.5× (superseded by 5c above). ork-driver run_chain_i8 now chains Sk=2 experts via Bf + auto-M-tiles (1bfeed3,3f5ae01, bit-exact); ggml-ork chains a projection's experts in one submit (ORK_MOE_NPU=1). pp128 0.28→0.97 t/s (3.5×, 0 fails). The chain primitive is a general win on main.
2026-06-23 — Tier 5 (MoE) — offload built, measured a NET LOSS, chaining scoped. Group-by-expert MUL_MAT_ID handler + resident pool (ork_mm_repack_i8) work correctly but lose ~60× (decode) / ~80× (prefill) — submit count is per-expert (~18k/batch), not per-token. Gated OFF behind ORK_MOE_NPU. Only viable path = chain experts within a layer (needs Sk==1 expert packing + grouped chain API + cross-core fan-out); prefill-only. rkllm has no MoE path, so no closed baseline. Also: dense prefill A/B confirmed NPU = 8.8× CPU (pp512 9.48→83.42, Qwen3-1.7B Q8).
2026-06-19 — Tier 4b RESOLVED & PROVEN: Native Multi-M execution is fully supported in INT4 mode. Verified bit-exactly up to $M=32$ (mc=64) native submits with 0 mismatches using a stride-2 layout gather de-tiling (Row m to output Row 2 * m) and leaving PPU stride 0x405c at 0.
2026-06-13 — Tier 3 NOT VIABLE / NOT NEEDED: 3a (persistent per-layer) blocked by the kernel's task_number>1 timeout (re-confirmed) + non-matmul ops forcing CPU round-trips; 3b (async overlap) best case ~7 tok/s < shipped hybrid 12.8. Goal already met by Tiers 1–2.
2026-06-13 — Tier 2 CLOSED: decode submit teardown — 98% irreducible NPU ioctl/IRQ floor; nothing to trim; decode already beats librkllmrt.
2026-06-13 — Tier 1c-ii DONE: full-K prefill (c1bb9cc). Single-core matmul 1.8×; end-to-end pp512 -t 4 156→178 (97% of librkllmrt). 1c-i (native layout) deferred.
2026-06-13 — Tier 1d ruled out: auto process big-core affinity craters -t 8 (oversubscription); recommendation = run -t <big-core-count>.
2026-06-13 — Tier 1b RESOLVED: thread count & big-core scheduling. Integration prefill was CPU-thread-contention + little-core drag, NOT the NPU matmul (the budget-3/Bf lead was a stale-binary artifact). -t 4 → pp512 155, tg128 12.84 — decode beats librkllmrt (11.71). Remaining prefill gap = ork single-core large-M kernel (1c).
2026-06-13 — Tier 1a done: CPU-affinity pinning (ork-driver 34b539e). Root cause = workers on little cores; mc_prof 3-core prefill scaling 1.11×→1.92×, int4 prefill 3.66→5.73.
2026-06-13 — roadmap created from the RKNN apples-to-apples findings.

Tier 18 — Core-partition the precision modes (2/1 split) to eliminate transition resets ⬜ UNPROVEN — and lower priority than the 12x throughput gap below

The idea (user, 2026-08-24). A mixed-tier model ping-pongs the NPU between the int4 and int8 chain pipelines and pays a device-wide ACT_RESET at every handoff. Instead of alternating in time on the same cores, partition in space: run int4 on two cores and int8 on the third. Each core then stays in one mode for the whole forward, the transition never happens, and all three cores stay busy. When only one precision is live, fall back to the 3-core variant — the split is only needed while both are in flight.

Why it is plausible. RK3588 is three independent NVDLA-derived cores, each with its own CNA/DPU/PPU and CBUF, each executing its own regcmd — and precision (0x4010 PROC_PRECISION) is programmed by the regcmd. The software, however, models precision as a single global scalar:

int warmed, last_dt;                          /* GLOBAL */
int mwarm[ORK_MAXCORE], mccsz[ORK_MAXCORE];   /* per-core */

Only warmth is per-core. If the hardware hazard is per-core too, last_dt is a conservative simplification and the reset is unnecessary for core-separated work.

The mechanism to do it already exists: the stream path runs one worker thread per core, each submitting on its own core with no barrier (~3x). Cross-core concurrency is not the blocker. (Note core_mask is a property of the SUBMIT, not of a task, so the two precisions need two submits — which is what the stream workers already do.)

What is established


`ACT_RESET` cost	105 ms measured (`ORK_DEBUG_RESET`), not the "~105us" in `chain_xition_probe`'s header
27B mixed-tier pack (`q27_ra8`)	160 resets/forward: 80x `XP_CHAIN_NT` (I4_CHAIN->I8_CHAIN) + 80x `XP_I4CHAIN` (I8_CHAIN->I4_CHAIN) = one round trip per promoted weight
Which half is load-bearing	the RESET. `ORK_XSPEC_NOCLEAR=1` -> PPL 10.4421 (unchanged); `ORK_XSPEC_NORESET=1` -> PPL 3442988.92 (garbage)
Is it per-core?	UNKNOWN — see below

What is NOT established — read this before costing the work

The prize is REAL and ~9.4%, measured n=3 interleaved (an earlier single-sample reading suggested only 4.9 s and is retracted — it was variance):

arm	scored (s), 3 runs	mean	vs baseline
baseline	169.4, 154.2, 159.9	161.2	—
`ORK_XSPEC_NOCLEAR=1`	159.1, 164.6, 165.8	163.2	+2.0 s — indistinguishable
`ORK_XSPEC_NORESET=1`	143.1, 146.6, 148.7	146.1	-15.1 s

All three no-reset runs fall below all three baselines, and 15.1 s closely matches the 16.8 s measured inside the reset ioctls — so the reset time is recoverable, not drain that gets paid elsewhere. Same-config spread is ~15 s, which is why n=1 comparisons on this workload are worthless; use >=3 interleaved.

Domain swapping is NOT a cost here and must not be blamed for it. Telemetry (ORK_DOM_PROFILE=1) on the 27B G=512 run: 12 real switches for the whole run, all first-touch (one-time bcreate), 26.7 ms total, ZERO steady-state swaps — 0.012% of a 220.9 s run. The layer-aligned domain layout means each domain is entered once and never revisited. The "multi-domain is swap-bound" finding from the 9B W4A8 era does not apply to this workload; do not reach for it as an explanation without re-measuring.

The real throughput mystery is unexplained. The NPU path scores 63 tokens in 220.9 s (G=512) or ~161 s (mixed-tier) where the CPU does it in 18.0 s — a ~12x slowdown — and it is NOT resets (G=512 pays only 2) and NOT domain swaps (12, one-time). A plausible contributor is that the grouped path multiplies submits: ork_i8_mm_run_grouped builds Sn x SG views with SG = K/G, so K=17408 at G=512 is 34 group-submits per matmul x 64 ffn_down layers — but even ~2200 extra submits at the 167 us floor is only ~0.4 s, so that does not close the gap either. This is the open question, and it dwarfs Tier 18: a 9.4% reset saving is irrelevant next to a 12x gap. Profile it (ORK_PROFILE=1, per-section) before optimizing anything here.

percore_mode_probe does not yet answer the question. It alternates an int8 chain and an int4 chain across two cores with the reset suppressed, and validates A=B=1 => C==K. Its positive control — same core, everything suppressed, expect corruption — comes back COHERENT at K=512/1024, M=1, plain packs, single domain. So the probe is not exercising the hazard and its split-core phase proves nothing; it reports INCONCLUSIVE by design rather than letting a green run be misread as a per-core result.

Closing that gap means matching the 27B configuration more closely: large K (5120/17408), M=64, DT_I4_ROT_A8 rotated/grouped packs, multi-domain with dom_activate between weights. It is possible the hazard is shape- or rotation-dependent rather than a property of the mode pair at all — which would be worth knowing on its own, since it would make the current reset over-broad rather than fundamental.

Cost side, if it does prove per-core

A static 2/1 split costs -33% int4 throughput to save transitions on a minority of weights. At 27B the int4 share dominates, so a static split could lose even at zero transitions; it needs to be dynamic (all 3 cores whenever only one precision is live).

And the alternative is already better on both axes: the uniform G=512 pack runs the same model with 2 resets instead of 160 and scores better (PPL 9.7485 vs 10.4421). Core partitioning only pays off where mixed tiers are specifically wanted — e.g. promoting a sensitive subset to a higher precision that a uniform pack cannot express.

Work items:

Extend percore_mode_probe until its positive control reproduces the corruption (large K, M=64, rotated/grouped pack, multi-domain). Nothing downstream is meaningful until it does.
Decompose the 105 ms: how much is drain that gets paid anyway? Time the submit-to-submit gap with and without the reset, not just the ioctl.
Only if 1 says per-core AND 2 says the time is recoverable: make last_dt per-core and route int4/int8 chains to disjoint cores, dynamically.

Tier 19 — The doorbell spine is dtype-agnostic substrate living in `src/npu/i8/` ✅ DONE 2026-08-25 (moved to `npu/core/`, behaviour-preserving)

The observation (user, 2026-08-25). The tree is modularized by PRECISION (src/npu/{i8,i4,f16,i16}/), with the stated rule that only dtype DISPATCH stays in the scaffold and that a contract with no dtype in it is substrate belonging in npu/core/. The NONBLOCK doorbell spine violates both. ork_dyn_begin_mc lives in src/npu/i8/dyn.c, takes a dt argument, and dispatches out of the int8 folder:

src/npu/i8/dyn.c:268:  if (dt == DT_I4) return ork_i4_dyn_begin_mc(c, S, tasks, nc);

A function that takes a dtype selector is a dispatcher, and i8/ should not know DT_I4 exists.

Extent (measured 2026-08-25). 26 foreign-dtype references inside the int8 folder:

file	`DT_F16`	`DT_I4`
`src/npu/i8/dyn.c`	9	1
`src/npu/i8/colsplit.c`	12	0
`src/npu/i8/dyn_ctl.c`	1	3

The reverse direction is nearly clean — i4/pack.c's 9 DT_I8 refs are CONVERSION, which the naming convention already exempts. So this is one-directional leakage, not mutual coupling: dyn.c, dyn_ctl.c and colsplit.c are the doorbell spine and column-split scheduler, dtype-agnostic substrate wearing an i8/ label.

Why it is not cosmetic — it caused a wasted board run. While root-causing the permanent int4 doorbell round stall, the pp=1 (no-HW-chaining) diagnostic was added to ORK_MC_ROUND in i8/dyn.c — which the int4 workload never reaches, because of the early return above. The A/B came back "no effect" from instrumentation that never executed, and the only reason it was caught is that removing HW chaining failed to produce the expected ~3.4x slowdown. A layout where the int4 submit lives in the int4 folder makes that class of mistake structurally impossible. A second symptom the same day: the diagnostic pushed i4/chain.c over its 900/935-line size budget, because precision-specific submit paths keep accreting into files that are already doing two jobs.

Proposed shape (follows the existing sdp.c / norm.c precedent for op families that span precisions):

Move the spine to npu/core/ — ork_dyn_begin*, ork_dyn_end, progress/poll, recover/resubmit. Its contract (submit NONBLOCK, poll an output sentinel, recover a dropped round) has no dtype in it.
Each precision folder contributes only program CONSTRUCTION and its completion predicate (int4's int16 full-surface poll, int8's per-row last-col, fp16's in-submit drain) behind one small vtable-ish struct.
colsplit.c likewise: the scheduler is agnostic; the fp16 accumulate path is not.
Dtype dispatch moves to npu.c, where the convention says it belongs.

Verification: behavior-preserving, so board make test must pass byte-identical, plus mode_probe and a refreshed tests/sbc_attest.txt in the same commit (all three files are CORE → ATTEST_SRCS). Restore src/npu/i4/chain.c to its 935 budget as part of the move.

OUTCOME (2026-08-25) — seam measured at ZERO crossing symbols

git mv src/npu/i8/{dyn.c,dyn_ctl.c,colsplit.c} src/npu/core/, spine API relocated from the private npu/i8/i8.h into npu/core.h (the substrate interface), the two int8-namespaced ork_i8_dyn_begin_seq* declarations kept in i8.h, Makefile CORE paths updated. Board make test: ALL TESTS PASSED, attest refreshed. make check-registry clean — including check 6, the cross-folder private-header gate that is exactly what this violated.

The seam was cheaper than the file sizes suggest, and cheaper than estimated. Of the ~130 symbols in i8.h, the three files used 7–14 each — but almost all were the spine's own API (ork_dyn_chain, _begin, _end, _progress, _halt, _append, _dump), which travels WITH the code, plus ork_npu / ork_w / ork_mm_task_i8 which were already tree-wide in internal.h. Every precision helper the spine calls (orki_f16_synth/_mcap/_sched/_slice_us, orki_i4_ktmo_ms) was already in internal.h, and the one apparent straggler, orki_i8_synth, turned out to be already declared in core.h. Net crossing symbols: 0 — the same class as the earlier three-way i8/dyn.c split, not the 28-symbol orkd.c case. Measuring first (as this repo's size-budget note instructs) turned a scary-looking 1,586-line move into a mechanical one.

Dispatch moved to the scaffold too (2026-08-25, second pass)

ork_dyn_begin_mc is now a thin dtype router in src/npu.c — the scaffold's designated role — and the int8+fp16 body in core/dyn.c was renamed orki_dyn_begin_mc_impl (agnostic; one body legitimately serves both precisions, which share a 4-byte C — the convention's MULTI case). int4 is routed to ork_i4_dyn_begin_mc before the implementation is entered.

Behaviour-preserving by construction: the router replicates ork_install_term() and both nc clamps on the int4 path, and the colsplit block int4 used to fall through is a guaranteed no-op for it (the ci8 gate requires dtype == DT_I8). Verified: board make test ALL TESTS PASSED (covers i4, test_chain_i4, perplexity_i4), plus the int4 doorbell route exercised directly — i4_widek_stall_probe 500 17408 5120 400 2 64 → 2.00% stalls (the expected residual) with 0 mismatched/rc-err runs.

This required raising the src/npu.c budget 2950 → 2975. That is a deliberate, stated exception rather than ratchet drift: the ratchet exists to stop implementation accreting in the scaffold, not to keep dispatch out of it, and this move takes routing OUT of substrate and puts it where the convention says it belongs. Keep ratcheting down.

Leakage after both passes, and why what remains is correct:

file	`DT_I4`	`DT_F16`	verdict
`core/dyn.c`	0 code (1 in a comment)	10	shared int8+fp16 body — MULTI
`core/dyn_ctl.c`	3	1	`ork_dyn_end` is the SHARED drain; it must know the output element type to de-tile int4's int16 output and arm recovery — MULTI
`core/colsplit.c`	0	12	shared int8+fp16 column split — MULTI

No file now dispatches on a dtype it doesn't own, and the i8/ folder no longer knows DT_I4 exists.

Sequencing: do it AFTER the int4 doorbell stall is root-caused — the investigation is touching these exact files, and a large move mid-investigation would invalidate the in-flight A/Bs.

NEXT: replicate the S=6 (~20 GB/s) weight-DMA rate on the CRITICAL PATH

The headline gap, measured 2026-09-04. decode_bw_probe shows the NPU reaching ~20 GB/s batching S=6 independent M=1 matmuls — against a ~22 GB/s LPDDR4X ceiling and the CPU's ~9.4 GB/s effective. The FFN decode handler, on the same shapes, gets 7.2 GB/s (gate+up) and 3.5 GB/s (down). So there is 1.75–2.4× sitting on the floor, and the task is to make the critical path achieve what the probe does.

Why this is the top item: dispatch is not the wall (48 µs of 1819 µs/layer, 2.6% — the doorbell floor of ~5 µs dispatch / ~16 µs per task is already negligible), and the hardware is not the wall (see above). It is implementation. Closing it is the difference between decode-on-NPU being a 2.8× loss and being plausibly competitive, since 3 concurrent streams already beat the CPU's effective rate.

Measured reference points (K=1024 N=3072 and K=3072 N=1024, 3 MB each, M=1):

S	GB/s (gate/up shape)	GB/s (down shape)
1	8.62	8.81
2	12.74	12.65
3	15.05	15.08
6	19.87	20.17

EIGHT disproven causes — do NOT re-try any of these

Investigated 2026-09-04. The probe and the handler make the identical call (ork_i8_mm_run) at the identical shape on a 3 MB weight, and differ 357 µs vs 1009 µs. Everything below was tested and does not explain it:

#	hypothesis	test	result
1	wide-K (K=3072 past the ks=2048 slice threshold)	probe at down's exact shape	8.81 GB/s — no
2	missing `Bf` full-K buffer	`ORK_NO_BF=1` in the probe	8.28 GB/s — no
3	submit mechanism differs	probe's S=1 short-circuits to the same `ork_i8_mm_run`	identical call — no
4	working set / IOMMU-TLB thrash (84 distinct 3 MB weights per token vs the probe's 6)	`nw` sweep 6 → 24 → 48	9.75 → 9.07 → 8.83 GB/s, only 9% — no
5	zero-copy import of mmap'd pack pages vs fresh DMA	`ORK_NO_IMPORT=1` (bcreate copy load)	6.2/3.0 vs 6.3/3.1 — no
6	CPU/DRAM contention from ggml's 4 threads	probe with 4 streaming-memcpy contenders	7.48 vs 7.43 GB/s — no
7	fresh per-call buffers defeating a per-buffer cache	`ORK_BW_FRESH=1` (malloc/free A and C every rep)	7.43 vs 7.47 GB/s — no
8	mode thrash from alternating chain → plain-int8 every layer	`ORK_MIXED_NOTHRASH=1`; `ORK_DEBUG_RESET` counts 56 resets over 224 layer-calls (~1 per 4, not per call)	6.3/3.3 vs 6.3/3.1 — no

Pipelining (was the leading hypothesis) is also effectively ruled out by #6 and #7 plus the timer layout: the handler has only ~45 µs of host work between 877–1008 µs submits, so there is no room for a 2× effect, and the PROF timers bracket only the calls themselves (verified by reading them — the byte accounting 2*K*Nff and Nff*Kd is correct).

PARTIALLY ROOT-CAUSED 2026-09-04 — and the framing was wrong

Doorbell phase timers (begin = regcmd synth + bsync + NONBLOCK ioctl, end = sentinel poll + writeback) were added at the run_multicore call site, because the existing orki_mc_* counters cover only the M>1 mcworker and this regime had no breakdown at all. The first reading reframed everything:

PROBE, down shape, S=1:  total 414 us  =  begin 255.7 (62%)  +  end 158.1 (38%)

The "achieved bandwidth" figures were never pure DMA. 62% of an M=1 call is host-side. The real hardware rate is 3 MB / 158 µs ≈ 19 GB/s at S=1 — already near the ~22 GB/s ceiling. So the S=1→S=6 "scaling to 20 GB/s" was amortising per-call host cost across more tasks, not the NPU speeding up, and the question was never "why is the DMA slow" but "why is the host half slow".

Cause found for the host half: core placement. The M=1 path is NONBLOCK, so begin AND end's spin-poll both run on the CALLING thread — but orki_pin_big_core() sits after the multi-core branch in orki_run, so it is only reached on the single-core fall-through. And ggml-ork sets ORK_NO_AFFINITY=1 in direct mode for a measured prefill reason (worker pinning oversubscribes the -t 4 threadpool, 136 → 213 t/s at M=228), applied process-wide. Right for prefill, wrong for M=1 decode.

Fixed by splitting the question: ork_big_core_set() keeps honouring the flag (its callers pin for life), and a new orki_big_core_mask() ignores it for scoped save/widen/restore around one call — which cannot oversubscribe a threadpool, so the prefill rationale does not apply. Measured:

	before	after
doorbell `begin`	654.7 µs	350.3 µs (1.87×)
Down	1032 µs / 3.0 GB/s	820 µs / 3.8 GB/s
per layer	2079 µs	1884 µs
decode (0.6b route 2)	4.93	5.58 tok/s
decode (0.8b route 1, the recipe winner)	3.13	3.40 tok/s → margin 1.28× → 1.40×

NOT applied to the run_chain_i8 doorbell — the same change there measured worse (gate+up 1005–1009 → 1091 µs, decode 5.58 → 5.45 across 3 trials each, decode stable to 0.00 tok/s so 2.3% is signal) and was reverted. Only the fix that measures is kept.

CLOSED 2026-09-04 — there is no bandwidth gap. It is host CPU work, and the NPU never waits.

Isolated to the down shape (K=3072, N=1024, Sk=3, 3 jobs/call) in both contexts, with kernel-side per-job hardware time (#patch74, summed in rknpu_job_done) and ork_dyn_end split into its wait and post-processing halves:

	probe	FFN handler
kernel hardware / job	150.9 µs	146.2 µs	identical (3%)
wait for hardware	0.0 µs	0.0 µs	never waits
`begin` (synth + bsync + ioctl)	255.8 µs	346.6 µs	1.36×
`end` (drain + K-slice accum + writeback + free)	158.5 µs	289.9 µs	1.83×

1. The wait is ZERO. By the time ork_dyn_end starts polling, the hardware has already finished — begin overlaps it. The "sentinel poll" was never waiting on the NPU, so reading end as hardware time was simply wrong.

2. The hardware is not the variable. Kernel per-job time agrees to 3% across the two contexts, and in an earlier run the consumer was faster (113.8 vs 143.9 µs/job). Every "achieved GB/s" number in this investigation was measuring CPU time to set up and tear down a submit while the NPU sat idle. 3 MB / 158 µs = "8.8 GB/s" is a host-cost artifact, not a DMA rate — and the S=1→S=6 "scaling to ~20 GB/s" was that host cost amortised over more tasks.

3. So this roadmap item's premise was wrong. There is no ~20 GB/s to replicate on the critical path. The real target is the driver's per-call host path: regcmd synth, bsync, ioctl, drain, the K-slice accumulate, writeback. The residual 1.4–1.8× is that host work running slower under the consumer's CPU load — both halves scale together, which is what contention looks like and what a DMA problem does not.

Landed from this investigation: scoped big-core placement of the doorbell caller (worth 1.13× decode, 1.28× → 1.40× on the one model whose recipe routes decode to the NPU). See the entry above.

DONE 2026-09-04 — precompiled regcmd on the decode path, 1.47× on the FFN layer

ork_pc_* compiles the regcmd once and per call only refreshes A and submits, so the ~314 µs/call of synth + validate + memcpy + bsync + descriptors + K-slice accumulate becomes ~5 µs. It pays with single-core hardware time (core_mask=1) and still wins, because host dominated hardware 3.4:1.

Per-call, probe, same weight: down 0.460 → 0.321 ms (1.43×), gate/up 0.450 → 0.322 (1.40×), q/k/v/o 0.377 → 0.108 (3.49×).

Wired into the FFN decode handler (ORK_FFN_PC=1), qwen3-0.6b route 2, P=512 G=48:

	PC off	PC on
Gate+Up	858 µs	647	1.33×
Down	743 µs	410	1.81×
host-silu	44 µs	67	the mirror copies (below)
per layer	1673 µs	1139	1.47×
decode	6.30	7.23–7.57 tok/s	~1.15–1.20×

Two contract details that dictate the integration. ork_pc_compile CAPTURES the A pointer and BAKES C's address into the program — so A must be a persistent buffer and C must come from ork_dma_alloc, hence a per-layer cache rather than per-call mallocs. And DMA memory is uncached: reading the outputs with a scalar loop cost 6144 uncached accesses and blew host-silu 45 → 311 µs, eating most of the win. One burst memcpy into cached mirrors costs ~20 µs and buys back 544.

Scope, plainly: this improves route 2 only, and route 2 is not the recipe for any model measured. The 0.6b's recipe is CPU decode (16.56 vs route 2's 7.5), and the 0.8b — the one model that does route decode to the NPU — is on route 1, which never enters this handler (measured identical, 3.75 both ways). The larger prize is the generic per-node M==1 matmul path, where the same probe measured 3.49× on the q/k/v/o shape and which is what route 1 actually uses.

⚠ SRAM PLACEMENT IS NOT UNIFORM — and getting it wrong HARD-WEDGES the board

Cost a plug power-cycle on 2026-09-04. On-chip SRAM is fine for buffers the NPU reads and fatal for buffers the KERNEL reads:

buffer	flags	SRAM?
regcmd (`c->regcmd`, `c->mrc[i]`, `ork_pc` pool)	`0x403` — NPU-read	safe
task descriptors (`c->task`, `c->mtk[i]`, `ork_pc` tk)	`0x40b` — carries `RKNPU_MEM_KERNEL_MAPPING`	HARD WEDGE (no ping)

ork_pc_compile had this right all along — it SRAM-requests its regcmd pool and nothing else. Never add TRY_ALLOC_SRAM to a 0x40b buffer.

Also, honestly: c->mrc now carries TRY_ALLOC_SRAM but the debugfs bitmap shows it still failing over to DRAM (1 × 4 KB in use during a run against a 978 KB free region), so regcmd-in-SRAM is currently a no-op. The flag is right; why it does not take is unresolved and is NOT counted as part of the win above.

And one design trap: ork_pc_run used to build its descriptors in c->task, the buffer the doorbell also uses. Fine for regime A (a fixed chain owning the NPU), instant reset-storm the moment a precompiled chain interleaves with ordinary submits. The chain now owns its descriptor array.

Hard constraint: ORK_NO_BF=1 declines the batched chain outright (S>=2 returns nonzero), so any plan that depends on batching depends on Bf being present on the weights — relevant if compact orkpacks are built without it.

Structural limit to design around: down has no independent peer in its dependency step (it consumes silu(gate)*up), so it cannot be batched with anything — reaching S>1 for it requires N-tiling, which means re-packing an orkpack-resident weight. [q,k,v] gives S=3 and [gate,up] gives S=2 for free.

Measurement discipline for this work: decode tok/s on this board has a ~9% run-to-run spread (5.12–5.58 observed for the same config), so per-change deltas under ~10% are NOT signal. Use the per-hop GB/s from ORK_FFN_PROF and decode_bw_probe, which are stable, rather than end-to-end tok/s. PPL, by contrast, is bit-deterministic here and safe for quality deltas.

Forward-looking: Edge-Asymmetric SLM (DRAFT, not started)

A custom ~2.1 B GDN+MoE model whose prefill and decode run on different silicon — W8A8 on the RKNPU2 inside the 32-bit IOVA window, INT4 on the A76 cores in 64-bit host memory. It generalises the MoE profile we already measured as optimal (experts on CPU, attention/dense on NPU) into the model architecture itself, and is sized so active-params × 0.5 B ≈ 300 MB/token.

Full specification, with every figure tagged measured / derived / proposed / unverified: Roadmap — Edge-Asymmetric SLM

Two findings there correct the incoming brief and are worth reading even if the project never starts: M=1 decode is latency-bound before it is bandwidth-bound (~8 GB/s of 21.9 on one A76), and Phase 3 should target ork-driver rather than librknnrt — the closed API cannot reach the precompiled regcmd cache, the NONBLOCK doorbell, or hardware task chaining.

Optimization Roadmap

Performance Optimization Roadmap

Tier 1 — Prefill: fix large-M multi-core (biggest, clearest)

Tier 2 — Decode: trim the per-matmul submit ✅ CLOSED (no headroom)

Tier 3 — Architectural ⚖️ NOT VIABLE / NOT NEEDED (feasibility 2026-06-13)

Tier 4 — int4 / W4A4 (open; native Multi-M scheduler discovered & validated ✅)

Tier 5 — MoE expert offload (MUL_MAT_ID) ⚖️ not yet viable, but closing (now ~7× under CPU, was ~80×)

MoE-on-NPU saturation milestone thread (Branch A — NPU-centric) — M0 ✅ · M1a ✅ · M3-skew ✅

Tier 6 — Cross-core: async round-robin stream ✅ ~3× (run_stream_i8)

Tier 8 — Mixed int8/int4-NF4 + imatrix quant ✅ DONE (PPL-validated)

Tier 9 — Streamed-prefill dma_buf fill chain ✅ cacheable + direct-inflate DONE; clean-only-bsync NEXT

Tier 10 — >4 GiB multi-domain zero-copy import: THREE independent limits (root-caused 2026-07-06) ✅ working

Tier 7 — Multi-core DMA cache coherency ⚖️ DEFERRED (low ROI; tracked)

Efficiency scan — O(n)→O(n/x) consolidation candidates 🔬 (scan 2026-07-06)

Mode-transition layer — centralize the scattered per-op reset ✅ (Phases 1+2 landed on main, 2026-07-14, v0.7.0)

Tier 11 — NPU Doorbell / Persistent-Kernel "Zero-Overhead Graph Cut" 🎨 (FOR THE CRAFT — explicitly NOT an efficiency lever)

Tier 12 — Heterogeneous CPU+NPU doorbell dispatcher (route to the most-efficient available resource) ⬜

Tier 13 — Weight-stationary matmul dataflow (reuse via WEIGHT_REUSE) ⬜ REOPENED 2026-09-05 — a narrow envelope is PROVEN and worth up to 1.52×; needs characterising before it can ship

⬜ OPEN 2026-09-06 — cross-M-tile WEIGHT_REUSE is SOLVED in principle; realising it is engineering

⬜ BACKLOG — productionising cross-M-tile WEIGHT_REUSE (superseded by the above)

⛔ MEASURED END-TO-END: the gain is currently ZERO, because the envelope matches no local model

Original 2026-07-30 investigation (superseded in part by the above)

Tier 14 — Punch-list carried into the ork-driver refactor (2026-08-19) ⬜

Tier 15 — Load the model from the .orkpack, retire the source GGUF ⬜ stage 3 (eager residence + ready event) IN PROGRESS

Tier 16 — W4A4 quality: grouped scales + int8 activations ⬜ TOP PRIORITY (quality)

MEASURED (2026-08-23): item 1 done — granularity is worth 38%, and it is HALF the gap

The other half is probably ACTIVATION WIDTH — and testing it is BLOCKED

The quantiser is SOLVED to within 7% — measured like-for-like (2026-08-23)

Split out: why 1.07x numerics gives 1.84x excess PPL

Tier 17 — Multi-domain teardown leaks IOVA until reboot ✅ FIXED 2026-08-24 (kernel patch 03)

Progress log

Tier 18 — Core-partition the precision modes (2/1 split) to eliminate transition resets ⬜ UNPROVEN — and lower priority than the 12x throughput gap below

What is established

What is NOT established — read this before costing the work

Cost side, if it does prove per-core

Tier 19 — The doorbell spine is dtype-agnostic substrate living in src/npu/i8/ ✅ DONE 2026-08-25 (moved to npu/core/, behaviour-preserving)

OUTCOME (2026-08-25) — seam measured at ZERO crossing symbols

Dispatch moved to the scaffold too (2026-08-25, second pass)

NEXT: replicate the S=6 (~20 GB/s) weight-DMA rate on the CRITICAL PATH

EIGHT disproven causes — do NOT re-try any of these

PARTIALLY ROOT-CAUSED 2026-09-04 — and the framing was wrong

CLOSED 2026-09-04 — there is no bandwidth gap. It is host CPU work, and the NPU never waits.

DONE 2026-09-04 — precompiled regcmd on the decode path, 1.47× on the FFN layer

⚠ SRAM PLACEMENT IS NOT UNIFORM — and getting it wrong HARD-WEDGES the board

Forward-looking: Edge-Asymmetric SLM (DRAFT, not started)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tier 6 — Cross-core: async round-robin stream ✅ ~3× (`run_stream_i8`)

Mode-transition layer — centralize the scattered per-op reset ✅ (Phases 1+2 landed on `main`, 2026-07-14, v0.7.0)

Tier 13 — Weight-stationary matmul dataflow (reuse via `WEIGHT_REUSE`) ⬜ REOPENED 2026-09-05 — a narrow envelope is PROVEN and worth up to 1.52×; needs characterising before it can ship

⬜ OPEN 2026-09-06 — cross-M-tile `WEIGHT_REUSE` is SOLVED in principle; realising it is engineering

⬜ BACKLOG — productionising cross-M-tile `WEIGHT_REUSE` (superseded by the above)

Tier 19 — The doorbell spine is dtype-agnostic substrate living in `src/npu/i8/` ✅ DONE 2026-08-25 (moved to `npu/core/`, behaviour-preserving)