# RKNPU regcmd Runtime — Roadmap

**Goal:** an *open* full-model NPU runtime for RK3576/RK3588 that drives the NPU
directly via raw **register-command (regcmd)** submission to `/dev/dri/renderD129`
(the RKNPU DRM driver), with **all model weights staged resident** in NPU memory.
This is the only path that closes the prefill gap (librkllmrt ~134 tok/s vs CPU
~95) in an open stack — and it unblocks multi-layer hidden states (EAGLE-3, [#462]).

## Why regcmd, and why not just patch the `rknn_matmul_api` limit
`rknn_matmul_api` (closed `librknnrt.so`) allocates per-matmul **contexts**, each
with its own NPU buffers + IOMMU mappings, and dies at ~150 contexts — far fewer
than a model's ~200 weight matrices. That ceiling lives in the closed library /
vendor kernel; patching it is fragile **and doesn't help** (you still re-upload
per context, never staging weights once). regcmd **structurally bypasses** it:
one large resident weight arena (`RKNPU_MEM_CREATE`, bounded only by the IOMMU
domain — multi-GB), with the command stream referencing weights *by offset*.
That's exactly how librkllmrt avoids the wall.

## Hardware context (recap from [ggml-backend-rknpu] / [RKLLM-Runtime-Internals])
The RK3588 NPU is an NVDLA-derived fixed-function INT8/FP16 accelerator. A regcmd
is a list of `(register-offset, value)` writes programming its blocks (CNA conv,
core/MAC array, DPU, PPU) plus DMA descriptors for input/weight/output buffers.
We drive it via `RKNPU_SUBMIT` (`struct rknpu_submit` → `rknpu_task[]`, each task
carrying `regcmd_addr`, `regcfg_amount`). Decode = batch-1 GEMV (bandwidth-bound);
prefill = batched GEMM (the NPU's strength — the target).

---

## Phase 0 — Foundations ✅ (done)
| | status |
|---|---|
| RKNPU DRM uapi transcribed + verified (`rknpu_ioctl.h`, kernel 6.1.115, drv v0.9.8) | ✅ |
| `rknpu_probe`: `MEM_CREATE → MEM_MAP → mmap → write → SYNC → DESTROY` round-trip | ✅ |
| `rknn_matmul_api` reference matmul validated (`[4×32]×[32×16]` all-ones → 32) — our **oracle** | ✅ |
| ggml-rknpu backend (rknn_matmul path), numerically validated vs CPU; perf characterized | ✅ |

These give us a known-good NPU result to diff against and the memory layer we need.

---

## Phase 1 — Capture & decode the regcmd ISA  ✅ (complete)
The keystone RE. We have a *validated* matmul (`mmtest`); capture the regcmd it
makes, then learn the format by perturbation.

- **M1.1 — ioctl trace shim.** ✅ `LD_PRELOAD` an `ioctl()` interposer that logs every
  RKNPU ioctl during `mmtest`: `MEM_CREATE` (size/handle/`dma_addr`/flags),
  `MEM_MAP`, `SUBMIT` (`regcfg_amount`, task count, `core_mask`, `task_obj_addr`).
  *Verify:* full call sequence + buffer inventory printed for the known matmul.
- **M1.2 — regcmd dump.** ✅ Track `dma_addr ↔ mmap'd CPU pointer` (hook `MEM_MAP`/`mmap`);
  on `SUBMIT`, dump the `regcfg_amount` words at each task's `regcmd_addr`, plus the
  input/weight/output buffer bytes. *Verify:* a complete hex dump of one real regcmd.
- **M1.3 — register dictionary.** ✅ Sweep matmul dims (M,K,N), quant, addresses; diff
  the regcmds to localize which words encode dims, strides, DMA addresses, quant
  scales, and the opcode/block-enable bits. *Verify:* annotated regcmd template with
  every field's meaning; documented in [RKLLM-Runtime-Internals].
- **Decision gate:** if the ISA proves too opaque to generalize, fall back to
  *replaying captured regcmds for fixed shapes* (limited but real), or stop here.

## Phase 2 — Replay & synthesize a single matmul  ✅ COMPLETE (M2.1, M2.2, M2.3)
- **M2.1 — verbatim replay.** ✅ Re-issue a captured regcmd via our own `RKNPU_SUBMIT`
  (our `MEM_CREATE` buffers, patched addresses) → reproduce the result **without
  librknnrt**. *Verify:* matches the oracle bit-for-bit.
- **M2.2 — parameterized fp16 GEMM.** ✅ Generate the regcmd from scratch for arbitrary
  (M,K,N). *Verify:* matches CPU across a shape sweep. **← make-or-break milestone.**
- **M2.3 — int8 / w8a8 GEMM.** ✅ The quant format librkllmrt uses (per-channel scales).
  *Verify:* matches a quantized CPU reference within tolerance.


### M2.3 — int8/w8a8 🔬 (deltas mapped) (2026-06-12)
Captured a librknnrt `RKNN_INT8_MM_INT8_TO_INT32` matmul (works; constraint **N%32** vs fp16's N%16). Same 108-entry regcmd structure; diffing int8 vs fp16 (4×32×32) isolates the precision/element-size deltas:
- **Element-size-scaled strides** (int8 = 1 byte vs fp16 = 2): `0x1030 = K*N` (was `K*N*2`), `0x1034 = K` (was `K*2`), `0x107c = K/16` (was `K/8`).
- **Precision-mode registers** (fixed int8 values): `801:3010 = 0x1` (fp16 `0x201`), `1001:4010 = 0x80000000` (fp16 `0xa8000002`), `201:100c`, `1001:4050`, `1001:40c0` differ — to confirm shape-(in)dependence.
- Data: int8 operands, int32 output; weights layout expected to follow the same `[Ntile][Ktile][·][·]` nesting (N-tile likely 32 for int8). Remaining: derive the int8 tile dims via the encoded probe, apply the precision registers, validate vs CPU. This is w8a8 — the LLM-relevant path.


### M2.3 — int8/w8a8 ✅ COMPLETE (2026-06-12)
From-scratch int8 (`RKNN_INT8_MM_INT8_TO_INT32`) matmul synthesis validated vs CPU for arbitrary M/K/N (4×32×32 … 32×512×256, 0 mismatches). `rknpu_synth_i8.c`. Deltas from fp16:
- **precision-mode registers** baked into the int8 template (`801:3010=0x1`, `1001:4010=0x80000000`, `100c`, `4050`, `40c0`);
- **element-size-1 strides**: `0x1030=K*N`, `0x1034=K`, `0x107c=K/16`;
- **`0x1044=ceil(K/64)`** — int8 reads 64 K-channels per hardware pass;
- **weights layout `[Ntile(32)][Ktile(32)][32][32]`** — N tiles by **32** (vs fp16's 16), K tiles by 32; feature stays flat `[M][K]`;
- constraints **K%32, N%32**.

**Phase 2 is complete: from-scratch GEMM synthesis works for both fp16 and int8 (w8a8 — the LLM precision) at arbitrary shapes.** Next: Phase 3 (resident weights — reuse one uploaded weight arena across calls).

## Phase 3 — Resident weights & a GEMM library  ✅ COMPLETE (M3.1, M3.2, M3.3)
- **M3.1 — single resident arena.** ✅ Upload all weights once into one `MEM_CREATE`
  region; matmuls reference by offset. *Verify:* hundreds of matmuls, zero re-upload,
  **no 150-context wall.**
- **M3.2 — prefill GEMM.** ✅ (arbitrary M via software M-tiling) NPU-expected weight tiling + batched M. *Verify:* a single
  layer's matmul hits librkllmrt-class throughput.
- **M3.3 — decode GEMV (M=1).** ✅ *Verify:* confirm bandwidth-bound (≈ CPU); decide
  NPU-vs-CPU routing for decode (likely keep decode on CPU).


### M3.1 — resident weights ✅ (2026-06-12)
`rknpu_resident.c`: uploads `NW` weight matrices into **one** NPU arena (`MEM_SYNC` once), then runs a matmul against each by patching the regcmd's weight address to `arena + w*wbytes`. **512/512 matmuls correct** against a single uploaded arena — i.e. **the ~150-context wall is structurally gone**: no per-weight `rknn_matmul` context, just offsets into resident memory. This is exactly the property the per-op ggml backend (Milestones 1–16) could never achieve, and the reason librkllmrt wins prefill. (A one-shot warm-up submit handles a cold-cache first-iteration artifact.) Next: M3.2 prefill-throughput benchmark, M3.3 decode GEMV.

### M3.2 — RESOLVED ✅ (2026-06-12): arbitrary-M GEMM via software M-tiling
The "multi-M-tile NPU-state wall" was two ordinary bugs (found via the CBUF/fresh-feature research hint):
1. **single-M-tile row cap = `16384/K`** (not 32768/K) — my chunk was 2× too big, so M>cap silently overran the tile (e.g. K=512 cap is **32**, not 64; verified M=32 ✓, M=48 ✗ rows 32-47).
2. **unique LIVE buffer per tile** — reusing one feature buffer (or destroy+recreate, which reuses freed IOVAs) leaves the NPU reading a **stale CBUF**; giving each tile a fresh, still-allocated feature+output buffer fixes it.
Software M-tiling (chunk = `16384/K` rows, fresh A/C per tile over resident weights, warm-up submit) → **arbitrary M/K/N correct**, validated vs CPU incl. **256×4096×512** (K=4096, 64 tiles), 512×512×512, 256×1024×1024 — 0 mismatches. **Phase 3 complete.** (Perf note: per-tile buffer alloc + separate submits is slow; a real runtime should use a buffer pool or multi-task-per-submit — correctness first.)


### Caveats addressed (2026-06-12)
- **int8 arbitrary-M ✅** — software M-tiling ported to int8; cap = **32768/K** (2× fp16; int8 packs 2× rows in CBUF — verified K=512→64, K=256→128). Validated vs CPU incl 256×4096×512, 512×512×512. Both precisions now do arbitrary M/K/N.
- **M-tiling perf 🔬** — attempted **multi-task-per-submit** (N tiles as N `rknpu_task` in one `SUBMIT`, `rknpu_mt.c`) to collapse N submits→1. **Times out, task counter 0** — even task 0 doesn't run. Tried `task_counter` 0/N and completion-interrupt on the last task only; no change. The NPU does **not** auto-sequence a regcmd task-list; **task-chaining must be encoded in the regcmd PC-control entries** (blocks `0x0101/0x0041/0x0081`, "continue vs stop") — the part mtx512 also left unimplemented. Bounded RE for a future session. Until then the correct per-tile-submit GEMM stands (slower but right); a **buffer pool** + keeping the NPU warm are partial speedups not requiring the chaining RE.


### M-tiling perf — single-submit scheduler ✅ (validated regime) (2026-06-12)
The fix is to replicate librknnrt's **internal single-task M-scheduler** (one `SUBMIT`, the NPU iterates M-tiles), not N submits. Per-submit overhead is **~32ms steady-state** (measured 8× warm), so collapsing N→1 is the real win. `rknpu_sched.c`, two scheduler regs as closed-form:
- `0x1010 = 16*min(M+1, 32768/K)` (rows per tile)
- `0x1040 = max(0x1b, base − slope*(M/64−1))`, `base=0xb1−15*(2^lg−1)`, `slope=15*2^lg`, `lg=log2(K/256)` — **valid only for K in the prefill band; it OVERSHOOTS `0xb1` below K=96 fp16 / K=256 int8 → zero/garbage output** (use `sched=0`/template `0xb1` there). See [Exp-2026-07-12-Matmul-Shape-Boundaries](Exp-2026-07-12-Matmul-Shape-Boundaries).

**Validated single-submit, 0 mismatches, K≤1024 moderate-M**: 64×256 … 256×512×128, 128×1024×64. This is the M-tiling perf fix for the common regime.

**Remaining (precisely traced):** K≥2048 and **many-tile (>~5)** have irregular `0x1040` and need the **PC-sequencer data buffer** — `0x101:0x10` (`PC_DATA_ADDR`) + `0x101:0x14` (`PC_DATA_AMOUNT`), with scaled M-fields (seen in librknnrt's 512×512 = 8 tiles). That's the deepest scheduler layer (`RKNPU_OFFSET_PC_DATA_*`), bounded RE. Until then the correct **software M-tiling** (per-tile submit, cap=16384/K fp16 / 32768/K int8) is the universal fallback.



### M-tiling perf — hybrid, FINISHED for K≤2048 ✅ (2026-06-12)
`rknpu_hybrid.c`: software-chunk M into **4-tile** pieces (`chunk = 4·32768/K` rows), each ONE single-submit scheduler call (the NPU iterates 4 internal M-tiles), fresh feature/output buffer per chunk. **~8× fewer submits** than per-tile software tiling (per-submit ≈32ms, so this is the real speedup). Validated vs CPU, **any M, K≤2048, 0 mismatches**: 1024×512 (4 submits vs 32), 2048×1024×16, 512×256×256 (**1 submit**), 256×2048, 1024×2048.
**K≥4096 — hard boundary (confirmed):** librknnrt sets the **PC_DATA buffer** (`0x101:0x10` PC_DATA_ADDR) even for small M at K=4096, with R=24 (`0x1010=0x180`) and `0x1040` floored at `0x48`. So the scheduler **cannot tile K≥4096 without the PC_DATA program** — the scheduler-reg-only fast path is fundamentally limited to **K≤2048**. K≥4096 fast tiling needs decoding the PC_DATA iteration program (deepest scheduler layer, bounded RE); the correct per-tile software path covers K≥4096 (slower). **Net: M-tiling perf is fast (≈8× fewer submits) and validated for K≤2048 any M — the common LLM contraction-dim range — and correct for all K.**


### M-tiling perf — COMPLETE for arbitrary M/K/N ✅ (2026-06-12)
Final solution (`rknpu_hybrid.c`), **two-level tiling**, no PC_DATA RE needed:
1. **K-split**: split the contraction dim K into ≤2048 slices (the scheduler-fast range), accumulate partial C's. Sidesteps the K≥4096 PC_DATA program entirely.
2. **M-tile**: within each slice, chunk M into **4-tile** pieces, each ONE single-submit scheduler call (NPU iterates 4 internal M-tiles), fresh feature/output buffer per piece.

**Validated vs CPU for arbitrary M/K/N (fp16), 0 mismatches**, incl. LLM-scale: 256×4096×512, **256×8192×16, 512×8192×128**. ~8× fewer submits than naive per-tile tiling (per-submit ≈32ms). **The M-tiling perf is finished — fast and correct for any shape.** (int8 uses the same structure; K-split bound is wider since int8 packs 2× rows/CBUF.) The PC_DATA single-pass path remains a documented optimization (fewer K-slices) but is no longer needed for correctness or the perf win.

### int8 (w8a8) fast path — COMPLETE for arbitrary M/K/N ✅ (2026-06-12)
`rknpu_hybrid_i8.c`. Same two-level hybrid as fp16, with the int8 scheduler registers reverse-engineered from librknnrt:
- **rows/tile** `0x1010 = 16·min(mc+1, 65536/Kp)` — int8 packs **2×** rows per CBUF pass vs fp16 (`65536/K` vs `32768/K`).
- **`0x1040`** = the fp16 closed form evaluated at **effective K = Kp/2** (int8 contracts 2× per CBUF pass). Verified exactly against captures: Kp=512 → base `0xb1`/slope 15 (= fp16 @256); Kp=1024 → base `0xa2`/slope 30 (= fp16 @512).

K is split into ≤1024 **clean** slices (where `R=65536/K` is exact); each slice is M-tiled into 4-tile single-submit scheduler chunks; int32 partials accumulated. **Odd remainder slices** (non-power-of-2 `Kp`, e.g. `11008 % 1024 = 768`, where `R`/`0x1040` go anomalous — same effect seen at fp16 K=1536/2048) fall back to the **proven per-M-tile submit path**, which is validated for any `Kp`. Net: fast scheduler on the bulk, always-correct fallback on the tail.

**Validated vs CPU, arbitrary M/K/N, 0 mismatches**, incl. LLM-scale: 512×8192×128, 512×4096×512, **256×11008, 256×14336**, 128×1280×64. **Both precisions (fp16 + int8/w8a8) now have a fast, correct, arbitrary-shape GEMM.** Phase 2–3 — the complete open NPU matmul engine — is done.

### Buffer pool ✅ + the "PC_DATA single-pass", investigated (2026-06-12)

**Buffer pool — done.** The hybrid now allocates ONE feature+output buffer pair and reuses it (no per-tile `MEM_CREATE`/`mmap`/`DESTROY`). Counterintuitive finding: a single STABLE reused buffer is correct AND fastest; rotating through >1 pooled buffer **corrupts** results (corruption grows with depth) — the NPU caches feature state by address across sequential submits, so a changing feature IOVA desyncs it (opposite of the multi-task-in-one-submit case). Surfaced a latent bug while testing: **non-power-of-2 K** (11008, 5120, 768, 1536) hit the scheduler anomaly and were silently wrong in the old hybrid (only power-of-2 K was ever tested); both fp16 and int8 now gate the fast scheduler on clean Kp and fall back to the proven per-M-tile path for odd remainder slices, so "arbitrary K" is genuinely true.

**"PC_DATA single-pass" — there is NO PC_DATA accumulation buffer.** The original note assumed K≥4096 single-pass needed a PC_DATA partial-sum buffer (`0x101:0x10`/`0x14`). A complete regcmd diff of librknnrt's two chained ops at K=4096 disproves it: the ops differ only in `0x1070` (feature slice) and `0x4020` (output slice) with **identical weights** — they are **M-tiles each doing the FULL K in one op**, not K-slices that accumulate. The `0x2000` bit in `0x1040` is a "subsequent M-tile" scheduler flag, not a DRAM accumulate (verified: separate-submit accumulate gives last-slice-only; chained op1 *overwrites*).

What is actually true:
- **PC-chaining works.** One submit runs S chained regcmd blocks via the word-216 trailer: `(reg 0x10, blk 0x0101)`=next regcmd addr, `(reg 0x14)`=`0x37`/`0`. This is exactly what the old `rknpu_mt.c` was missing (it listed tasks but never set the chain pointer → sequencer never advanced → timeout). Pointing `0x101:0x10` anywhere else → timeout (confirms it's the chain pointer).
- **Single-op full-K for M≤R.** The NPU iterates K internally in one op. **Decode (M=1) does ANY K in ONE submit** (`0x1010=0x20`, `0x1040=0xb1`) — validated K=2048/4096/8192 × N up to 512, all correct, baked into `rknpu_sched.c`. This is the practically important single-pass: token generation issues one submit per matmul regardless of K depth.
- **General-M large-K single-op** is gated by a **nonlinear CBUF scheduler heuristic** (rows/tile R per K: 1024→48, 2048→32, 4096→24, 8192→9) — not cleanly formularizable from a few captures. The K-split hybrid sidesteps it entirely and is already correct + fast (sub-ms submits), so this remains the production path for prefill; the single-op path is a future submit-count optimization.

### Operational polish — COMPLETE ✅ (2026-06-12)

The open NPU matmul **primitive set is complete and production-shaped** — both precisions, both regimes:

| | fp16 | int8 / w8a8 |
|---|---|---|
| **Prefill / arbitrary M,K,N** | `rknpu_hybrid.c` | `rknpu_hybrid_i8.c` |
| **Decode (M=1) / small M (M<R)** | `rknpu_sched.c` | `rknpu_sched_i8.c` |

- **int8 decode single-submit** — int8 M=1 uses the *same* scheduler regs as fp16 (`0x1010=0x20`, `0x1040=0xb1`); any power-of-2 K in ONE submit. Validated M=1 K=2048/4096/8192 × N≤512 + small-batch M<R. int8 is ~2× faster than fp16 (half the weight bandwidth: 8192×512 = 444ms vs 814ms). Non-power-of-2 K (11008/14336) route to the K-split hybrid.
- **Path selection rule** (for the M5 backend): `M==1 && K is power-of-2` → single-submit full-K; otherwise → K-split hybrid (pooled, arbitrary K, both precisions).

**General-M large-K single submit — DONE (M-tile PC-chaining).** Built for completeness and as a portable foundation (`rknpu_pcchain.c`). It is the model librknnrt actually uses: `S=ceil(M/R)` M-tiles, each computing the FULL K in one op, all chained in ONE submit via the word-216 PC trailer (`0x101:0x10`=next regcmd, `0x14`=`0x37`/`0`). M-tiles write **disjoint** C rows, so **no `0x2000` accumulate bit** is needed (confirmed). Per-tile `(R, base 0x1040)` come from a per-K CBUF calibration (nonlinear, captured from librknnrt): 512:(64,0x84), 1024:(48,0x48), 2048:(32,0x2a), 4096:(24,0x48), 8192:(8,0x84). M is padded to `S*R` (extra rows ignored). Validated vs CPU, one submit each, incl. 512×8192×512 (**64 tiles chained**) — all 0 mismatches; in the regression suite.

**RK3588 performance caveat (why it is NOT the default prefill path).** Each full-K M-tile re-reads the *entire weight matrix* (~22× for M=512,K=4096), vs the K-split hybrid's ~per-K-slice (2×). Matmul is weight-bandwidth-bound, so on RK3588 the hybrid is faster for prefill despite issuing more submits. The PC-chain primitive exists as the single-submit reference and for **future Rockchip NPUs** where more cores / higher bandwidth may flip that tradeoff. (It is calibrated for power-of-2 K; other K use the hybrid.)

**Bottom line:** Phases 1–3 + perf + polish are done. The matmul layer is correct, fast, pooled, arbitrary-shape, both precisions, with optimal decode. Everything remaining is the transformer runtime (M4–M6).

## Phase 4 — Transformer ops beyond matmul
- **M4.1 — reusable matmul library (`rknpu_mm`)** ✅ — keystone: one device handle,
  weights packed once and kept resident, many matmuls streamed against them.
- **M4.2 — non-matmul CPU ops** ✅ — RMSNorm, RoPE, attention/softmax, SiLU/SwiGLU.
- **M4.5 — non-matmul ops ON the NPU** 🟢 (2026-07-04, [[Exp 2026-07-04 PPU Element-wise / Activation Ops|Exp-2026-07-04-PPU-Elementwise-Activation-Ops]]) — supersedes the "PPU activations don't run standalone" finding. Standalone SDP ops (NVDLA-derived, enable_mask=0x18) now run **bit-exact on silicon**, in `make test`: EW-mul (int8/fp16/int16), SiLU/GELU (int8 bit-exact, int16 RKNN-class), residual Add (int8/fp16), exp/rsqrt LUT blocks. The activation is a LUT op (contents define the fn). RMSNorm reduction = matmul (no dedicated reduce op); RMSNorm/Softmax/RoPE compose from these blocks. **Outstanding (tracked):** (a) **fp16 activations** — NVDLA LE-table *exponential* mode + multi-task submit; (b) **int16 residual Add** — SRDMA/X1 operand halves negatives (X1/X2 sub-module scale decode); shared unlock = **multi-task submit** (also enables the fully-fused exNorm/exSoftmax). RE toolkit: ork-driver `tools/re/`.
- **M4.6 — llama.cpp integration of the on-NPU ops: standalone offload is a DEAD END; the needed op is FUSION** 🔴 (2026-07-04) — Wiring the M4.5 standalone ops into the `ggml-ork` backend as per-op offload paths (`ORK_PPU_OPS`/`_ADD`/`_SILU`/`_GELU`, branch `feature/ggml-ork-ppu-ops`) surfaced three independent reasons standalone element-wise/activation offload **cannot win** on this hardware, so the router correctly keeps them on CPU/NEON:
    1. **Op-shape mismatch.** Modern llama.cpp does NOT emit separate SiLU + MUL nodes for the FFN — it fuses them into one `GGML_OP_GLU` op (`ggml_swiglu_split` = `silu(gate)⊙up`; GELU FFN → `ggml_geglu_split`). So standalone SiLU/MUL handlers never even fire on Qwen/Llama.
    2. **No arithmetic intensity.** The NPU wins matmul because each weight load is reused across M·K MACs; an element-wise op is 1 op/element, so a `fp32→(quant)→DMA→submit→DMA→(dequant)` round-trip can't beat inline NEON (matches the long-standing "activations are memory-bound / NEON-competitive" finding). **Measured** (residual ADD, Qwen3-1.7B, pp128): **157 → 19 t/s (~8× slower)**. Root cause (empirically isolated): **NOT buffer churn and NOT the op itself** — a persistent-scratch build that eliminated the per-op MEM_CREATE/MEM_DESTROY churn moved it only 19.45 → 18.83 t/s (null result, `feature/ppu-scratch-reuse`), and the SDP ops run in ~18–118 µs *standalone*. The 8× comes from **INTERLEAVING** the SDP ops into the int8-matmul stream: the NPU regcmd mode is **stateful**, so a matmul→ADD→matmul sequence forces a **mode switch + re-warm** around every ADD (the "switching modes triggers a one-time NPU re-warm" behaviour), **and** it **breaks matmul PC-chaining** (consecutive matmuls can no longer be chained into one submit when an ADD splits them). Both scale with the number of interleaved ops (~56/forward) and neither is fixable by optimizing the standalone op. **Only FUSION avoids it** — applying the activation/add in the matmul's own output stage keeps a single mode, adds no submit, and preserves chaining (data already resident in NPU SRAM). This is a stronger reason for fusion than arithmetic-intensity alone.
    3. **IOVA instability.** Each standalone op allocates a fresh NPU DMA buffer; doing that per-residual-add/-activation per-layer churns the **4 GiB IOVA window already consumed by the resident matmul weights** ([[NPU IOVA 4 GiB cap|npu-iova-4gb-cap]]) → measured a hard **kernel fault in `rknpu_iommu_dma_map_sg`** (`ORK_PPU_ADD=1`, Qwen3-1.7B) that wedged the NPU. Verdict: routing standalone element-wise ops to the NPU while weights are resident is not just slow, it's unstable.
  - **★ FUSED GATE-MATMUL + SiLU: DONE, BIT-EXACT ON SILICON (2026-07-04).** `ork_mm_run_i8_silu` (ork-driver
    `feature/ffn-fusion`) runs a resident full-K int8 matmul with SiLU fused in the SDP output stage (LUT
    streamed to PPU SRAM once, then matmul+silu per N/M-tile) → int8 `silu(gate)`. Validated == the silicon
    probe (`tools/fused_silu_test.c`: mism 0/512, max|e|=0). **KEY BUG resolved:** a hand-rolled submit to a
    fresh output buffer produced bias-only output — the conv accumulator never reached the SDP. The fix is to
    submit via `submit1()` into the WARMED `c->Cc` (+ `dom_activate` at entry, `act(RESET)` right before the
    submits). This is the gate half of on-NPU SwiGLU and the first activation that rides a matmul's own submit.
  - **✗ FUSED UP-MATMUL + EW-mul (×silu(gate)): still WEDGES.** `ork_mm_run_i8_ewmul` uses the `set_i8_ewmul`
    DPU_RDMA-graft-onto-synth_i8 output stage; it wedges the NPU (job timeout, task counter 0x0) even on a
    clean NPU as the first op — the same 2nd-operand-fetch wall the standalone EW-mul RE hit (only the
    standalone `ork_npu_ewmul_i8` cracked it, via the full NVDLA cube layout, not the matmul-fused graft).
    So a fully-fused SwiGLU needs either that graft fixed OR compose: fused-SiLU gate + up matmul + the
    multiply via standalone `ork_npu_ewmul_i8` / CPU. Gated behind ORK_TEST_EWMUL (won't wedge make test).
  - **★ FUSED-EWMUL GRAFT — web-lead search exhausted, CLOSED as a hard wall (2026-07-04).** Chased the NVDLA
    docs (programming_guide + unit_description) for a config that unwedges the online-mode (conv×memory)
    EW-mul: (a) confirmed the **EW/X1 sub-module is the only per-element-multiply path** — the BS sub-module's
    per-element mode is bias-ADDITION only ("Multiplier will be bypassed during bias operations"), so BS can't
    do the SwiGLU multiply; (b) the online-mode 2nd-operand fetch (BRDMA/ERDMA + flying `D_FEATURE_MODE_CFG`)
    is exactly what the prior ~40-cycle campaign tried (all wedge, task counter 0x0 = program rejected); (c)
    the docs name no specific working RDMA/ordering/size config that wasn't already tried, and submit1 (which
    fixed the SiLU acc-not-reaching-SDP) does NOT change the ewmul's program-validity wedge. The two ork
    sub-paths both fail: the graft (RKNN EW output stage on ork's dense conv geometry) WEDGES; RKNN's verbatim
    op executes but needs RKNN's own A/B tile layout (zero output on ork data). **No new viable path from web
    search → the fused-ewmul-into-matmul graft is CLOSED.** Refs: nvdla.org programming_guide.html,
    unit_description.html (SDP X1/X2/Y + BS).
  - **★★ CHAINED on-NPU FFN inner BUILT + a 97× ACT_RESET fix + PIPELINE overlap 1.25× (2026-07-04).**
    `tools/fused_ffn_probe.c` runs the composition int8-throughout (gate+SiLU → up-out8 → ewmul → down,
    `ork_mm_run_i8_out8` added). **KEY GENERAL FIX:** the fused SDP ops (silu/out8/ewmul) were doing a
    per-call `act(RKNPU_ACT_RESET,0)` — an NPU soft-reset costs **tens of ms/op** — leftover debug
    scaffolding (`submit1` was the real correctness fix, not the reset). Removing it: chained FFN inner
    **320 ms → 3.2 ms/ffn (97×)**, and standalone `ewmul_i8` **~107 ms → 73 µs**, BOTH re-validated
    bit-exact (test_ewmul_i8 mism 0/8192; fused-silu mism 0/512). This is why standalone SDP offload
    measured so badly before (the 8× ADD regression was largely this reset, not only arithmetic intensity —
    worth re-measuring the other SDP ops after dropping their resets too). **PIPELINE (probe `pipe` mode):**
    running the NPU FFN inner concurrently with representative CPU work overlaps them — SERIAL 1621 ms →
    OVERLAPPED 1297 ms = **1.25× (ideal 1.68×)**; monitor harness shows NPU + CPU both active at once. Gap
    to ideal = thread contention (no pinning / 1 NPU stream + 1 CPU thread). Confirms the architecture:
    keep the FFN on the NPU (frees the CPU, 66%→13%), overlap the freed CPU with the NPU → throughput win.
    NEXT: wire into ggml-ork so REAL per-token CPU work (attention/quant/norm) overlaps the NPU FFN + pin
    threads + multi-core the FFN ops → push 1.25× toward the ideal and into an end-to-end model number.
  - **★ PIPELINE pinning 1.25×→1.57× + GGML_OP_GLU wired end-to-end (2026-07-04).** (a) Pinning the NPU
    thread to core 4 and the CPU work to cores 5–7 cut the overlap contention: 1.25× → **1.57×** (overlapped
    1066 ms vs 1002 ms NPU-alone; ideal 1.67×) — nearly all CPU work now hidden behind the NPU FFN. (b) Wired
    **GGML_OP_GLU** (SwiGLU) into the ggml-ork backend (supports_op + graph_compute + `ggml_backend_ork_glu`,
    fork `feature/ggml-ork-ppu-ops`): quantize gate/up per-tensor int8 → `ork_npu_silu_i8` → `ork_npu_ewmul_i8`
    (gain 1/128) → dequant, ORK_PPU_GLU-gated, fp32 CPU fallback. **Runs a real model end-to-end (Qwen3-1.7B):
    engages on-NPU (not the fallback), no wedge, COHERENT output** — but int8-degraded ("capital of the world"),
    the expected loss of the standalone fp32→int8 path. This is the wiring foothold; the perf win needs the
    round-trip-free int8-CHAIN (gate/up matmuls emit int8-out feeding the GLU + down reads int8, no fp32
    boundary) which requires FFN-pattern detection in graph_compute + per-tensor act-quant scale plumbing.
  - **★★ ROUND-TRIP-FREE int8 FFN CHAIN — BUILT END-TO-END, measured NET-LOSS (2026-07-04, decisive).**
    Implemented the full chain in ggml-ork (`ORK_FFN_CHAIN`, off by default; fork `feature/ggml-ork-ppu-ops`
    commit 3457ca744): FFN-pattern detection in `graph_compute` (validated: `[gate,up,GLU,down]` arrives as ONE
    4-node ork subgraph at prefill **M≥32**; at M<32 GLU→CPU so it fragments and no-match) → `ggml_backend_ork_ffn_swiglu_chain`
    runs gate matmul+fused-SiLU (per-layer LUT via `ork_mm_silu_build_lut`) → up int8-out → `ork_npu_ewmul_i8`
    → down, **int8 intermediates never touching fp32**; per-tensor scales (the fused SiLU stage's scalar R
    can't carry the model's per-row/per-channel — the accuracy concession), CPU fp32 fallback. **In-model
    (Qwen3-1.7B-Q8_0, RK3588): CHAIN pp128 = 140.14 t/s vs per-node baseline 147.43 → ~5% SLOWER**, plus
    intermittent NPU soft-reset. Root cause (structural, matches the weight-DMA/submit-floor findings): the
    fused chain needs MORE submits (LUT-load + silu + out8 + ewmul + down, and the fused SiLU path is
    **M-capped at 64/submit**) than the baseline's 3 larger matmuls (M-scheduler to `mg_max*64`) + a cheap
    CPU GLU; **the fp32 round-trip the chain removes is NOT the prefill bottleneck.** NOT YET VIABLE — to reach
    even parity needs a breakthrough: **(1)** large-M-tiles for the fused output-stage path (`mg_max*64`, like
    the int8 matmul cap fix) so silu/out8 aren't capped at 64/submit, **(2)** load the silu LUT ONCE not
    per-tile, **(3)** fix the fused-op intermittent wedge at `Nff=6144`/down `K=6144`. Decode can't use it
    (M=1 → CPU GLU → no match, and the serialization wall loses at M=1 anyway). Code kept, gated OFF.
  - **★★★ FFN CHAIN reaches SPEED PARITY+ (2026-07-04, flips the net-loss above).** Four levers turned the
    ~5%-slower incoherent chain into a faster, coherent one (ork-driver commit 2ef2f0a, fork 3533ac922):
    **(1)** fused output-stage M-tile cap 64→`mg_max*64` (bit-exact, tools/fused_mtile_check); **(2)** SiLU LUT
    loaded once/call not per-tile (bit-exact vs probe); **(3)** ewmul moved to **CPU int8** — the elementwise
    multiply has no arithmetic-intensity so it loses on the NPU (per-call IOMMU churn + cube-marshaling ~6ms/ffn
    + it wedged); SiLU still rides the gate matmul output stage for FREE, so this is the arithmetic-intensity
    split (matmuls NPU, elementwise CPU); **(4)** **SmoothQuant** per-layer (migrate x's per-channel outliers into
    gate/up weights x'=x/s W'=s*W, and glu's outliers into the down weight; static per-tensor scales calibrated
    on the first batch) → coherence: **PPL 23041 → ~22**. Gated OFF (ORK_FFN_CHAIN).
  - **★★★★ CORRECTION + HYBRID chain (2026-07-04).** (1) **The earlier "pp512 6×" was a benchmarking error** —
    that run's baseline pp512=32 was anomalous (cold / parked DDR governor). Re-measured with governors verified
    at performance: **baseline pp128 150 / pp512 191**; the all-int8 chain is pp128 174 (+16%) / pp512 193
    (**~parity, NOT 6×**). (2) **Per-channel SDP RE finding:** RKNN does int8 matmul per-channel requant on the
    HOST, not in the SDP regcmd (per-layer vs per-channel captures byte-identical bar an int_status task-desc
    bit; tools/re/percapture.c) — so no SDP recipe to copy. (3) **Pivot → HYBRID chain:** only the GATE stays
    per-tensor (its fused SiLU has a scalar SDP output scale); up + down use the STANDARD per-channel host-dequant
    path and the ewmul is fp32, so silu_i8 is the ONLY int8 intermediate. **PPL ~22 → ~16** (vs baseline ~9);
    speed pp128 162 (+8%) / pp512 160 (−16%, fp32 intermediates cost at large M). Quality floor ~16 = the gate's
    per-tensor + int8-silu-output (fused scalar-R). **True parity (PPL 9) needs per-row GATE activation — provably
    impossible in the fused stage** (silu(a·x)≠a·silu(x); SDP scale is per-channel/per-N not per-row/per-M). The
    only exact-output route left is **drafter-verify** (degraded chain drafts, exact baseline verifies). Also an
    intermittent NPU soft-reset persists on the fast path (ACT_RESET removed for the 97× win). Gated OFF.
  - **RESOLUTION for on-NPU SwiGLU (no graft needed): COMPOSITION of validated pieces —** gate matmul + fused
    SiLU (`ork_mm_run_i8_silu`, bit-exact) → `silu(gate)` int8; up matmul + int8-out (`set_i8_out8`) → `up`
    int8; then `silu(gate) ⊙ up` via the STANDALONE `ork_npu_ewmul_i8` (mem×mem SDP, validated bit-exact) or
    CPU. 3 submits, all validated-working, entirely on the NPU in int8/SDP mode (no per-op interleaving with
    CPU ops, so it avoids the standalone-offload 8× penalty). This is the productionizable on-NPU SwiGLU; the
    fully-fused 2-submit form is blocked on the closed graft.
  - **NEEDED OP (theoretically possible, high-value): fused matmul→activation output-stage.** The only way the NPU adds activation value is to apply the activation IN the matmul's output stage, riding its existing submit + DMA buffers (no new IOVA, no new submit, no round-trip). This maps exactly to `GGML_OP_GLU`: `swiglu(gate,up)` = a **gate matmul with SiLU fused in its output stage** (`set_i8_silu`, already RE'd + validated on silicon) followed by an **up matmul with the per-element EW-mul fused in its output stage** reading `silu(gate)` as the 2nd operand (the per-element SDP EW-mul, also RE'd + bit-exact). Both building blocks exist; the missing pieces are **(a)** a public ork-driver fused-matmul API (`ork_mm_run_i8_silu` / `_ewmul`, output int8 or dequantized) and **(b)** a `ggml-ork` graph-pattern matcher that detects `GLU(gate=Wg·x, up=Wu·x)` and emits the fused NPU sequence instead of 2 plain matmuls + a CPU GLU. This is the productionizable on-NPU-activation path.
  - **Other needed-but-unsupported ops (theoretically possible):** **fp16 SiLU/GELU** (NVDLA LE-exp LUT mode, path known — see M4.5(a)); **softmax max-reduce** (the one non-matmul reduction not expressible as a matmul — needed for a fully-on-NPU softmax; exp LUT block already exists); **RoPE** (cos/sin LUT + EW-mul + Add — composable from existing blocks, not yet built standalone). RMSNorm needs no new op (reduction = matmul + rsqrt LUT), just fusion if profiling ever justifies it (memory-bound → NEON-competitive today).
- **M4.3 — full decoder-layer hybrid forward** ✅ — matmul on NPU, ops on CPU; validated
  vs a pure-CPU reference.
- **M4.4 — KV-cache incremental decode** ✅ — token-by-token, M=1 projections, RoPE at the
  running position, attention over cached K/V; validated vs CPU and consistent with prefill.
- **Multi-layer / real-model wiring** ⏳ → Phase 5 (M5).

**Phase 4 is complete.** The open NPU stack now runs a full transformer decoder layer end
to end — prefill and KV-cache decode — with matmul on the NPU and the non-matmul ops on the
CPU, all validated to fp16 precision and in the regression suite (43 cases).

### M4.1–M4.3 — first full transformer layer on the open NPU stack ✅ (2026-06-12)
The matmul kernels are now a **library** (`rknpu_mm.{h,c}`): `rknpu_mm_init` → `rknpu_mm_pack(K,N,B)` (pack + upload weights once, resident) → `rknpu_mm_run(w,M,A,C)` (stream activations; K-split hybrid internally, arbitrary M/K/N). On top of it, `rknpu_layer.c` runs a complete **Llama/Qwen-style decoder layer** — `h = x + Wo·Attn(RoPE(Q,K),V)`, `y = h + Wdown·(SiLU(Wgate·hn) ⊙ Wup·hn)` — with the seven big projections on the NPU and the non-matmul ops (RMSNorm / RoPE-NeoX / causal softmax / SwiGLU) on the CPU (the pragmatic hybrid graph).

**Validated** against a pure-CPU reference (identical ops, CPU fp16 matmul, so the matmul engine is the only variable): H=512, NH=8, HD=64, FFN=2048, SEQ=16 → **maxabs 0.044 vs |ref|max 276 (~0.016 %, pure fp16 rounding)**. Both in the regression suite.

Two bugs found while turning the standalone kernels into a reusable library:
1. **`setr` must patch ALL matching regcmd entries, not the first.** The template carries some regs (`0x1040`) more than once and the NPU uses a *later* copy; first-match-only left a stale schedule value — wrong for multi-M-tile shapes only (single-tile masked it).
2. **A freshly (re)allocated output buffer is cold** — its first submit returns stale data. The one-time warmup must re-fire whenever the output buffer is resized (here when N grows 512→2048 between projections), not just once per context.

Both are NPU-state subtleties (not algorithm errors) — the kind of thing the standalone one-shot kernels never exercised.

### M4.4 — KV-cache incremental decode ✅ (2026-06-12)
`rknpu_decode.c` runs the same layer one token at a time: per step, M=1 NPU projections, RoPE at the running position, append K/V to a per-layer cache, attend over the cached keys/values. Validated vs a pure-CPU reference over a 16-token sequence — **maxabs 0.044 vs |ref|max 276**, *identical* to the prefill layer's error, confirming prefill ↔ incremental-decode causal consistency. This is the token-generation path (M=1 matmuls, the decode regime). **Phase 4 done.** Next: stack N layers + load a real model's weights / tokenizer / embeddings / LM-head (M5).


### M3.2 — prefill GEMM (large M) 🔬 (2026-06-12, refined)
Works across a **wide range** — pure large-M (512×32×16 ✓), large-M + moderate K/N (128×128×64 ✓), large-K/N at modest M (32×512×128 ✓), GEMV (M=1 ✓). The **raw NPU output C buffer byte-matches librknnrt** (verified at 128×32×16), and feature/weights/regcmd all match — so the matmul is correct over this range. It diverges only when **M>32 AND K≥256 together** (e.g. 128×256×256, 128×512×512) — a feature M×K co-blocking interaction (the feature, flat `[M][K]` at smaller sizes, needs M×K tiling past that threshold). This is the last GEMM edge. Feature confirmed flat `[M][K]` even at 128×256 (row0 = k0..255 contiguous), so the divergence is on the weights-at-large-(KT×NT) or output side — same byte-compare→decode method applies. **M-tile = 64** (M>64 ⇒ multiple M-tiles): two registers then change — `0x1010 = min(16*(M+1),0x800)` (saturates at M≥127, FIXED) and `0x1040` (packed M-tile-count field: 0xb1/0xa2/0x84 for M/64=1/2/4, partially decoded) — plus likely an M-tile data layout. GEMM is now correct for **any M at single/moderate-N (any K)** + GEMV: M-tiling solved via `0x1010=min(16(M+1),0x800)` and `0x1040=0xb1-15*(ceil(M/64)-1)` (both occurrences set). Remaining edge: **large-M AND large-K together** (e.g. 128×512×16, K≥512 at M>64). Root cause precisely characterized: the NPU **auto-tiles M based on an SRAM budget that depends on K** — `0x1010` and `0x1040` are K-dependent (128×256: 0x1010=0x800,0x1040=0xa2; 128×512: 0x1010=0x400,0x1040=0x84). The budget is ≈ **32768 elements**: M-tile-rows ≈ min(M, 32768/K) (K=256→128 rows, K=512→64 rows), so `0x1010 = 16·min(M, 32768/K)`. Verified correct now for **any M with K≤256** (128×256×256, 256×256×256 ✓), large-N at small K (128×64×512 ✓), and GEMV. Completing the K-driven M-tiling scheduler (0x1010 + 0x1040 + M-tile count) is the last GEMM piece — a bounded scheduler model, not a new layout.

[superseded] Earlier note (large-M>8 diverges) was wrong: it conflated pure large-M (which works) with the large-M+large-K shapes. Verified for M=128: the **regcmd byte-matches** librknnrt (no M-tile registers), and with uniform per-row probes the **feature is flat `[M][K]`** (rows in order) and the **output C is flat `[M][N]`** (row m = K·m). Yet random-data M=128 gives C[0] correct, rest wrong — the same "uniform-matches, random-diverges" signature that the nested K×N case had, i.e. a subtle M-blocking layout the constant-row probe can't expose. Resolution is the same proven method (full byte-compare synth-vs-librknnrt buffers with identical random data → localize → decode), not yet done. Decode GEMV (M=1, M3.3) and small-M prefill already work, so this is the large-batch prefill edge.

- **M4.7 — batched-GEMM (attention/GDN) + normalization ON the NPU** ⚖️ (2026-07-10, [[Exp 2026-07-10 On-NPU Attention Batched-GEMM and Norm|Exp-2026-07-10-On-NPU-Attention-Batched-GEMM-and-Norm]], ork-driver `main` `32db14c`) — the two remaining non-matmul op classes, built + bit-exact-validated + gated, all in `make test`. **`ork_bmm_i8/i4/fp16`**: a **batched dynamic GEMM** (both operands dynamic) — the attention QKᵀ/AV and Gated-Delta-Net chunked-matmul primitive the static-weight path can't serve (i8/i4 exact, fp16 4e-4). **`ork_npu_rmsnorm_f16`/`l2norm_f16`** (gated `ORK_NORM_NPU`): the full-channel `Σx²` reduction runs on the NPU as a **matmul contraction** (any n via K-split — the M4.5 "reduction = matmul" now realized), and the rsqrt via `ork_mm_build_f16_rsqrt_lut` = the **fp16 fused-into-the-reduce twin** of the M4.5 standalone `ork_npu_rsqrt` (fused K=n≤2048 one submit 6e-4 / decoupled K=512 any-n 5e-4; K=32 is degenerate for the fp16 fused-silu tiling). Plus `ork_l2norm_f32` (NEON CPU). **Corroborates M4.6 (standalone offload loses; fusion is the need):** `ORK_ATTN` on Ornith-9B routes the GDN chunked matmuls but NPU stays 1.6% / prefill unchanged (the CPU wall is the *sequential* GDN recurrence — `gated_delta_net`/`solve_tri`/`cumsum`/`exp` — not matmuls); pure-attention Qwen3-1.7B doesn't offload at all (real QKᵀ/AV are non-contiguous KV-cache/permuted-Q views → declined). Standalone norm is submit-floor-bound vs the ~µs fused NEON pass; the win needs fusing into an adjacent matmul's output stage. **RE landed:** the DPU output stage is NAMED against the mainline `drivers/accel/rocket/rocket_registers.h` (`0x4040 DPU_BS_CFG`, `0x4060 DPU_BN_CFG`, `0x4070 DPU_EW_CFG+LUT`, `0x4030/34/3c DATA_CUBE_WIDTH/HEIGHT/CHANNEL` geometry), and NVDLA's **CDP LRN reduction window is HW-capped at n∈{3,5,7,9}** — proving a full-channel (`d_model`) norm reduction cannot use the fixed-function LRN op and must ride a matmul contraction (which is what shipped). RE box: rknn-toolkit2 2.3.2 on the `.239` Colima VM.

- **M4.8 — FUSED matmul + output-stage activation: the no-crossing chain** ✅ (2026-07-10, [[Exp 2026-07-10 Fused Output Activation No-Crossing Chain|Exp-2026-07-10-Fused-Output-Activation-No-Crossing-Chain]]) — the payoff M4.6/M4.7 kept pointing at. **`ork_mm_run_f16_act(K,N,B,M,A,C,fn,…,in_lo,in_hi)` = `C=fn(A·B)` in ONE submit**, the activation riding the matmul's DPU output stage — no separate op submit, no CPU↔NPU crossing, no activation→matmul mode-switch. Validated bit-accurate in `make test`: fused **exp** (softmax) 4e-3, **silu** (SwiGLU) 2e-3, **gelu** 6e-3, **rsqrt** (RMSNorm, shipped earlier) 6e-4. Mechanism: build the fn PWL LUT over the output range, pack the weight `±S·B` so `acc` hits the fp16 index-spread band, run the fn-agnostic fused-LUT matmul (`ork_mm_run_f16_silu`), recover `fn=C·out_scale`. **One real constraint:** the fp16 SDP index spreads for a SINGLE sign of `acc`, so the matmul output must be single-signed per tile — handled by construction (positive-input `rsqrt`→pack `−S·B`; negative-input `exp(scores−max)≤0`→build `fn(−u)`, pack `+S·B`); mixed-sign returns −2 (needs two LUTs — the real meaning of the old "fp16 silu WIP"). Contrast the kernel-sequenced standalone softmax replay ([[NPU-Quirks]]): bit-exact but 9× the submit floor (~2 ms); fusion makes the transcendental free. Follow-up = wire into ggml-ork (exp on the score matmul, silu/gelu on FFN up-proj, rsqrt on the norm reduce).

## Phase 5 — Integrate into the ggml / llama.cpp fork
- **M5.1 — resident-regcmd GEMM in the ggml-rknpu backend**, replacing the
  rknn_matmul path. *Verify:* prefill re-benchmark vs librkllmrt (target: close 134).
- **M5.2 — end-to-end forward** for one architecture (Qwen3) via the
  [`oRKLLM/llama.cpp-rockchip`](https://github.com/oRKLLM/llama.cpp-rockchip) fork.
  *Verify:* `llama-bench` prefill ≈ librkllmrt; correct output.
- **M5.3 — productionize:** 3-core scheduling, error handling, per-arch graphs.

### M5.1–M5.3 — a REAL model runs end-to-end on the open NPU stack ✅ (2026-06-12)
Took the standalone-harness route first (prove the whole stack with a real model before wiring the ggml fork). Three increments, all in the regression suite:

- **M5.1 multi-layer body** (`rknpu_model.c`): N stacked decoder layers, each with its own weights; validated NPU-hybrid vs CPU to 24 layers — error grows gracefully (1L 0.0015 → 24L 0.046), no divergence.
- **M5.2 real weights** + **M5.3 end-to-end forward** (`rknpu_llama2.c`): loads Karpathy's **llama2.c `stories15M`** (real trained Llama: dim 288, 6 layers, 32000 vocab), mmaps the fp32 `.bin`, packs projections + LM head into resident fp16 NPU weights (transposed [out][in]→[in][out]), and runs the full decoder — embed → 6 layers w/ KV cache → final RMSNorm → 32000-vocab logits — matmul on NPU, ops (RMSNorm, **interleaved GPT-J RoPE**, causal softmax, SwiGLU) on CPU. Greedy-generates; **NPU-hybrid logits match a pure-CPU-fp16 reference to worst |Δ| = 0.01 over 12 steps.** A real LLM now runs on the regcmd NPU stack.

**Bugs found + regression-locked this milestone:**
1. **NPU output-width cap = 8192** (N=16384 silently wrong, N≤8192 exact). The 32000-vocab LM head exposed it. Fixed with **N-tiling** in `rknpu_mm` (weights now pack into `Sk×Sn` = K-split × N-split≤8192 buffers; run accumulates K-slices into each N-slice's columns). `test_mm` now covers N=16384 and the 1×288×32000 LM-head shape.
2. **Unsigned-arithmetic** in synthetic norm-weight init (`(i+seed)%5-2` underflowed to ~4.3e9) → RMSNorm explosion → fp16 overflow → NaN; the check then **silently passed** (NaN compares false). Hardened to fail on NaN/inf/dead output.
3. **`maxout` over-allocation** (ignored the actual chunk → a 58 MB LM-head output buffer DMA-synced every submit). Sized to the real chunk.

The reusable library (`rknpu_mm`) + ops are the substance the ggml/llama.cpp-fork integration (M5.1-fork above) will call; that wiring + multi-arch + 3-core scheduling remain.

### M3.2 — M-tiling scheduler data (the large-M+large-K edge)
The remaining GEMM edge is the NPU's **internal M-tiling scheduler** (`0x1010`, `0x1040` depend on BOTH M and K). Captured table (value = high16 of the reg word):

| M \ K | 0x1010 (256/512/1024) | 0x1040 (256/512/1024) |
|---|---|---|
| 64  | 0410 / 0400 / 0300 | b1 / a2 / 84 |
| 128 | 0800 / 0400 / 0300 | a2 / 84 / 48 |
| 192 | 0800 / 0400 / 0300 | 93 / 66 / 1b |
| 256 | 0800 / 0400 / 0300 | 84 / 48 / 1b |

`0x1010` = 16·(rows-per-M-tile): K=256→128, K=512→64, K=1024→**48** rows (not a clean 1/K — there's a floor/rounding in the SRAM heuristic). `0x1040` is a packed bitfield encoding the M-tile schedule. Modeling this scheduler is the last GEMM piece — bounded RE, but genuinely the NPU's tiling logic (what `rknn_matmul` computes internally), not a single formula. **The GEMM is correct without it for the common range: any M at K≤256, large-N at small K, GEMV, and all int8/fp16 K%32/N%(16|32) shapes that fit one M-tile.**


### M3.2 — multi-M-tile attempt (the honest wall)
Tried to finish the large-M+large-K edge two ways; both hit a genuine NPU multi-M-tile **state** behavior I could not crack this session:
- **NPU internal M-scheduler** — `0x1010`/`0x1040` are K-dependent (data table above); even with the right values for moderate cases, the **2nd M-tile's output lands wrong** for large-M+large-K.
- **Software M-tiling** (split M into ≤cap row-chunks, loop submits over resident weights) — fails on the 2nd chunk in *both* sub-variants: **offset feature addressing corrupts** the result, and **same-address re-upload reads a stale NPU feature cache** (M3.1's loop worked only because its feature was static and only the *weight* address changed). Non-cacheable buffers + warm-up submit didn't resolve it.

So consecutive submits with **changing feature data** is the unsolved bit — the NPU appears to reuse feature state across jobs in a way that needs an invalidation/handshake I haven't found, or a fresh feature buffer per chunk. **The single-M-tile GEMM is solid** (any M at K≤256, large-N at small K, GEMV, all int8/fp16); large-M+large-K prefill batching is the remaining edge and is genuinely harder than a register formula.

## Phase 6 — Payoff
- **M6.1 — multi-layer hidden states** → resolves [#462], enables **open EAGLE-3**.
- **M6.2 — open NPU prefill engine** selectable in oRKLLM (the two-engine model:
  open-NPU/llama.cpp for everything, no closed `librkllmrt` dependency).

## Open RE — int4/dtype batch & streaming program (2026-07-07)
Spun out of [Exp-2026-07-07-INT4-Multi-M-Schedule-RE]. We landed int4 native multi-M **batch** mode
(`0x107c=K/16` → CBUF activation budget 16384 int4 elems; `rows = min(M/2, 16384/K)` = 8/submit @ K=2048,
32 @ K=512; stride-2 in/out, `physrow = 4j + 4H·b`; default-on, bit-exact). Batch is **CBUF-residency-limited**
(the 8-row ceiling at K=2048). int8/fp16 instead **stream** (int8 M-tile = `mg_max*64` = 128 @ K=2048, *not*
CBUF-limited) via the `0x1040` K-reduction schedule — which **poisons** int4 with its int8 value. The
`0x107c = D_ENTRY_PER_SLICE` result (elements/entry: fp16 8, int8 16, **int4 16** — int4 rides int8's
16-element/64-bit entry, *not* a dense 32/128-bit one) ties these together and generates the leads below,
ordered by value. All share the CNA/CSC batch recipe (`0x405c=0` + `mc_phys` stride-2 + `0x107c` + de-tile),
so they compose.

**1. ★★ int4 STREAMING (biggest win: ~16×, → int8-class W4A4 prefill).** Status: OPEN — not found, not ruled
out. A streaming schedule (activations from DRAM, non-resident) would blow past the CBUF cap to ~128 rows
@ K=2048. Prime lead: int4 poisons on `0x1040` likely due to an **entry-layout mismatch** — it's in the
64-bit/16-elem batch layout the streaming schedule can't drive. So set the **dense 128-bit entry first**
(`0x107c=K/32`, 32 int4/entry, matching int8's entry *width*) and THEN sweep `0x1040`/schedule. Win criterion:
`rows×K > 16384` at K=2048. Tool: `tools/i4_multim_fuzz.c` combo mode. (The earlier standalone `0x1040` sweep
found nothing precisely because it lacked this dense-entry precondition.)

**2. ★ int8 in BOTH modes = the stream-vs-batch SELECTOR (best lead-generator for #1).** int8 is the only
dtype we can run both ways: it streams today, and the shared batch recipe (`0x107c=K/16`) should also make it
batch. **Diff streaming-int8 vs batch-int8 regcmd → the differing register(s) ARE the mode selector**; apply to
int4 → candidate streaming trigger (turns #1 from blind fuzz into "read off the diff"). Also re-confirms
`D_ENTRY_PER_SLICE` in a 2nd mode and reveals whether the CBUF budget is bytes or elements (int8 batch cap =
`16384/K` or `32768/K`? — predicts int4's dense ceiling). Worth doing even though int8-batch has no direct perf
use. Method: force `synth_i8` into the batch recipe at small M, validate bit-exact, `diff` the two regcmds.

**3. int4 DENSE entry packing (`0x107c=K/32`, standalone 2× batch).** If the HW supports a dense int4 entry
(32 elems/128-bit) *without* streaming, the batch budget doubles (16384→32768) → 16 rows @ K=2048. Smaller
(2×) but lower-risk than #1, and a stepping stone to it (same `K/32` precondition). Needs the companion
width/atom reg for 32/entry (candidates: `0x100c` precision-mode, `0x1044` K-passes) co-varied with `0x107c`.

**4. fp16 BATCH mode (a correctness fix, not perf).** fp16 has a documented latent large-tile *streaming*
M-scheduler bug (AGENTS.md). Applying the batch recipe to fp16 (`0x107c=K/8`, stride-2, resident) gives an
independent **correct** large-M fp16 path (CBUF-capped but bit-exact) that sidesteps the buggy streaming
scheduler. Test: batch recipe on `synth` (fp16) at the M where streaming breaks; check bit-exactness.

**5. int4 wide-N N-subslice (perf completion of the landed batch path).** The shipped batch path falls back
to per-row for wide N at large K (per-core `Ncore×K > 131072` weight budget). An N-subslice loop that keeps
`Ncore_sub×K ≤ 131072` per submit extends batch mode to production wide-N shapes — see the Exp-2026-07-07 log.

---

## Open RE — audit ork-driver chaining vs the kernel's task-sequencing model (2026-07-10)
Spun out of the forward-softmax replay ([[fwd-softmax|FWD_SOFTMAX_RE_WIP.md]]). The mainline open-source
**`drivers/accel/rocket`** driver (see AGENTS.md §4 "Open-source reference") shows the NPU runs **one task per
PC program** (`TASK_CON`/`OPERATION_ENABLE`, `TASK_NUMBER(1)`), and the **kernel IRQ handler re-arms the next
task** from the job's task array — sequencing is **kernel-driven via completion interrupts**
(`PC_INTERRUPT_*_DPU_0/DPU_1` = `0x300`), NOT the hardware walking a concatenated regcmd buffer via an
in-regcmd "next-PC" (`0101:0x0010`). DMA faults surface as `PC_INTERRUPT_RAW_STATUS_DMA_READ_ERROR` (the
softmax-replay hang's `raw_status 0xc0000000` with `require 0x300` = a DMA read error on a chained task, not a
completion-on-a-different-bit).

**RESOLVED (2026-07-10) via the forward-softmax replay experiments:** the vendor **`rknpu` driver uses HARDWARE
PC-chaining** (`0101:0x0010` next-regcmd), **NOT** rocket's kernel-re-arm-on-IRQ. Evidence: a `task_number=9`
job with only the descriptor array + zeroed chain **fails `errno=110`** on rknpu, whereas rocket kernel-iterates
that fine. ⇒ **ork-driver's existing `run_chain_i8/i4` + `run_multicore` PC-chaining (which sets the `0101`
words) is CORRECT for its target driver** — no change needed; the rocket kernel-re-arm model is rocket-specific
and does not port. Confirmed working paths stay as-is.

**Hardware-chain MECHANISM cracked (2026-07-10, from the vendor `rknpu` kernel `rknpu_job_commit_pc`):** the
kernel programs the HW from the **first task only** — `PC_DATA_ADDR=first->regcmd_addr`, `PC_DATA_AMOUNT` from
`first->regcfg_amount`, `INT_MASK` from the **last** task, `PC_TASK_CONTROL=task_number` — then the **hardware
walks the chain** via the in-regcmd descriptor **`0101:0x0010` = next regcmd addr**, **`0101:0x0014` = next
register-amount `=(n+3)/2`** (== kernel `(n+EXTRA+scale-1)/scale-1`, EXTRA=4, RK3588 `pc_data_amount_scale=2`).
This is the general rule behind `run_chain_i8`'s hardcoded `0x0014=0x0037` (=(108+3)/2, its uniform tasks).

**Softmax hardware-chain status: ✅ SOLVED (2026-07-10).** The full 9-task forward-softmax graph now runs as ONE
hardware-chained submit, **bit-exact 9/9, ~471µs** (vs ~2.5ms per-task = 9× floor → ~5×) — `ork_npu_replay_softmax_f16`
default. The full recipe (a COMBINATION, all required):
1. **`0101:0x0014` next-amount `=(n+3)/2`** (was zeroing it → hung at transition 1).
2. **Contiguous single-buffer image + single-delta blanket rebase** — replay the vendor's whole 5-buffer image in
   ONE allocation so task4's LUT-load writes INTO the regcmd region (splitting buffers breaks the LUT chain).
3. **★ PING-PONG OFF (`flags=0x1` not `0x5`)** — the final factor. `RKNPU_JOB_PINGPONG` (`1<<2`) double-buffers
   the register banks and swaps to the next task the instant the current task's *register config* is done — but a
   LUT-load has a long-latency **SRAM-commit side effect not in the register bank**, so ping-pong advances to the
   exp task before the LUT commits → flaky task4↔task5 stall (non-det counter 4↔5). Off → HW fully completes each
   task incl. the commit. **Rule: ping-pong OFF for any chain with a LUT-load; ON is safe only for register-config-
   only tasks (uniform matmuls, `run_chain_i8`).** See [[NPU-Quirks]].
Task alignment was a red herring (content-driven stride). "SRAM commit timing" was the right mechanism but the fix
was disabling ping-pong, not adding a barrier. `ORK_SM_PERTASK`/`ORK_SM_1SUBMIT`/`ORK_SM_NTASK` preserve the
alternate paths (gated). Still submit-floor-bound as a standalone op — the **fused output-stage** path
(M4.8 `ork_mm_run_f16_act`) is the production win; this is the definitive chaining RE + a working single-submit softmax.

---

## Risks & honest framing
- **Multi-month, mostly reverse-engineering.** M1.3 + M2.2 are the make-or-break;
  if regcmd can't be generalized across shapes, the project caps at fixed-shape replay.
- **Per-architecture graphs** (like librkllmrt) and exact-numerics matching are required
  for quality.
- **Maintenance** is tied to the NPU driver/hardware.
- **It is a research / sovereignty play**, not near-term ROI: for plain serving,
  librkllmrt already delivers prefill and the prefix cache hides most of it. The win
  is *NPU prefill + full openness + features the closed runtime lacks* (multi-layer
  EAGLE-3), together — which nothing else gives.

## Tooling we build along the way
- `rknpu_trace.so` — the `LD_PRELOAD` ioctl/regcmd capture shim (M1.1–M1.2).
- `regcmd_diff` — perturb-and-diff register localizer (M1.3).
- CPU oracle + `mmtest` — the correctness reference at every step.

[#462]: https://github.com/airockchip/rknn-llm/issues/462
[ggml-backend-rknpu]: https://github.com/oRKLLM/oRKLLM/wiki/ggml-backend-rknpu
[RKLLM-Runtime-Internals]: https://github.com/oRKLLM/oRKLLM/wiki/RKLLM-Runtime-Internals

---

## Progress log

### M1.1 — ioctl trace shim ✅ (2026-06-12)
`rknpu_trace.c` (`LD_PRELOAD` `ioctl` interposer, in `experimental/ggml-rknpu/`) run against the validated `[4×32]·[32×16]` fp16 matmul (`mmtest`, result CORRECT). Full submission anatomy captured:

**Buffer inventory** (6× `MEM_CREATE`, each one 4 KB page, IOMMU domain 0, `core_mask=0x1`):

| handle | synced bytes | dir | role |
|---|---|---|---|
| 1 | **896** | →dev | **regcmd stream** (the register-command ISA) |
| 2 | 40 | — | task descriptor — one `rknpu_task` (40 B); `task_obj_addr` points here |
| 3 | 1024 | →dev | B weight `[32×16]` fp16 |
| 4 | 256 | →dev | A input `[4×32]` fp16 |
| 5 | 1024 | — | second 1024 B (reordered-B / scratch — TBD in M1.2) |
| 6 | 256 | ←dev | C output `[4×16]` |

**Sequence:** `ACTION` probes (`GET_HW_VERSION`→`0x46495245`="FIRE"; `GET_DRV_VERSION`→`0x38c`=**v0.9.8**; `POWER_ON`→ -1 already-on; `SET_PROC_NICE`=-19; `GET_IOMMU_EN`=1 before each alloc) → 6× `MEM_CREATE`+`MEM_MAP`+`MEM_SYNC(0x3)` → `MEM_SYNC(→dev)` on regcmd/B/A → **one `SUBMIT`** (`flags=0x5`, `timeout=6000`, `task_number=3`, `task_counter=3`, `core_mask=0x1`, `hw_elapse_time=11667 µs`) → `MEM_SYNC(←dev)` on C → `MEM_DESTROY`.

**Findings:**
- The whole matmul is **896 bytes of regcmd** (handle 1) + a 40-byte task descriptor pointing at it. That's small and tractable to decode.
- `MEM_CREATE` flags: `0x403` = NON_CONTIGUOUS|CACHEABLE|IOMMU_LIMIT_IOVA_ALIGNMENT; the task buffer adds `KERNEL_MAPPING` (`0x40b`).
- IOVA `dma_addr`s are handed out top-down (`0xffffe000`, `0xffffd000`, …) — the regcmd almost certainly references A/B/C by these device addresses, so M1.3's address-localization should be straightforward by diffing.
- Single core only (`core_mask=0x1`) for this tiny op — multi-core comes later (M5.3).

**Next — M1.2:** hook `mmap` to map `handle→CPU pointer`, then dump the 896-byte regcmd + the 40-byte task descriptor + A/B/C bytes for this known shape.

### M1.2 — regcmd dump + format decoded ✅ (2026-06-12)
`rknpu_dump.c` (adds `mmap`/`mmap64` hooks → `handle→CPU ptr`, parses the task descriptor, dumps the regcmd as words). For the `[4×32]·[32×16]` matmul the task descriptor decodes as one `rknpu_task`: `enable_mask=0x0d`, `int_mask=0x300`, `int_clear=0x1ffff`, `int_status=0x100`, **`regcfg_amount=108`**, `regcmd_addr→0xffffe000` (handle 1). C output is **fp32** (`0x42000000`=32.0) from fp16 inputs.

**The regcmd is 108 register-writes as `(word0, word1)` u32 pairs, and the encoding is decoded:**

```
reg_offset = word0 & 0xFFFF
block_id   = word1 >> 16        # 0x0201, 0x0801, 0x1001 = three NPU blocks
value      = ((word1 & 0xFFFF) << 16) | (word0 >> 16)
```

Verified by the **embedded DMA addresses** found in the stream (the patch points for synthesis & resident weights):

| reg (block) | value | = buffer |
|---|---|---|
| `0x1070` (0x0201) | `0xffffb000` | A input |
| `0x1110` (0x0201) | `0xffffc000` | B weight |
| `0x4020` (0x1001) | `0xffff9000` | C output |

Dimension fields are visible too — `reg 0x1024 → 0x001f0020` (K=32 / K−1=31), `reg 0x1078 → 0x000f000f` (N=16 / N−1=15). The three blocks map to register ranges: `0x1xxx`↔0x0201, `0x3xxx`↔0x0801, `0x4xxx`↔0x1001 (CNA feature / DPU / PPU-core, NVDLA-style). Full dump saved at `/tmp/rknpu/dump_4x32x16.txt`.

**Implication:** the format isn't opaque — it's flat register writes with addresses embedded as split 32-bit fields. The M1.3 *decision gate* (could-the-ISA-generalize) is essentially passed; M1.3 now just needs to map which regs encode M/K/N/strides/quant by sweeping shapes.

**Next — M1.3:** parametric matmul harness (argv M K N), capture regcmds for several shapes, diff to finish the register dictionary; then M2.1 verbatim replay.

### M1.3 — register dictionary ✅ (2026-06-12)
Parametric harness (`rknpu_mm_param.c`, argv M K N) swept four shapes — base `(4,32,16)`, `M=8`, `K=64`, `N=32` — all CORRECT; regcmds decoded (`diffrc.mjs`) and diffed. All 108 entries have fixed structure; only values change, and every dimension-dependent field resolves to a closed form:

| reg (blk:off) | base value | tracks | formula |
|---|---|---|---|
| `201:1024` | `0x1f0020` | K | `(K-1)<<16 \| K` |
| `201:1030` | `0x400` | K | `K*32` |
| `201:1034` | `0x40` | K | `K*2` |
| `201:1044` | `0x1` | K | `K/32` (K-tiles) |
| `201:1088` | `0x20` | K | `K` |
| `201:107c` | `0x4` | K | `K/8` |
| `201:1020`,`1084` | `0x10004` | M | `M` (low16) |
| `201:102c` | `0x4` | M | `M` |
| `201:1010` | `0x50` | M | `16*(M+1)` |
| `1001:4034` | `0x3` | M | `M-1` |
| `1001:405c`,`801:3014` | `0x30000` | M | `(M-1)<<16` |
| `1001:403c` | `0xf000f` | N | `(N-1)` (both halves) |
| `1001:4058` | `0xf` | N | `N-1` |
| `1001:4038` | `0x30003` | N | `(N/4-1)` (both halves) |
| `201:1038` | `0x1010010` | N | `N` (low byte) |
| `801:3018` | `0xf` | N | `N-1` |
| `201:1070` | A dma | addr | **patch** (A input) |
| `201:1110` | B dma | addr | **patch** (B weight) |
| `1001:4020` | C dma | addr | **patch** (C output) |

The remaining ~90 entries are shape-invariant constants (block config, opcodes, int masks). **Three NPU blocks** are programmed: `0x0201` (CNA / feature+weight setup, `0x1xxx` regs), `0x0801` (`0x3xxx`), `0x1001` (PPU/core output, `0x4xxx`). Tooling: `rknpu_mm_param.c`, `diffrc.mjs` (decode+diff) in `experimental/ggml-rknpu/`.

**Verdict:** the regcmd ISA is decoded well enough to *synthesize* a matmul for arbitrary (M,K,N): emit the ~90 constants verbatim + compute the ~18 dimension fields + patch 3 addresses. Phase 1 complete.

**Next — M2.1:** verbatim replay — allocate our own buffers, write the captured regcmd with the 3 addresses patched, `RKNPU_SUBMIT` directly (no librknnrt), read C, expect `==K`.

### M2.1 — verbatim replay ✅ (2026-06-12) — **the NPU runs on our own submit, no librknnrt**
`rknpu_replay.c` drives a raw `RKNPU_SUBMIT` with the captured regcmd and produces the correct result (`[4×32]·[32×16]` all-ones → **C = 32.0**, fp32). librknnrt is not involved.

Three things had to be right (each cost a debugging round; documented so nobody repeats them):

1. **Device node — the decisive fix.** The job only executes via the **primary (card) node `/dev/dri/card1`**. On the **render node `/dev/dri/renderD129`** the kernel accepts every buffer and the submit, then the job runs **0 tasks and times out** (`task counter: 0x0`, soft reset). librknnrt uses the render node successfully *with its own full setup*, but for a from-scratch submit the card node is what works.
2. **regcmd is 112 entries (224 words), not 108.** Beyond the `regcfg_amount=108` register-writes there are **4 trailing PC-control entries** (blocks `0x0101/0x0041/0x0081`) — the `RKNPU_PC_DATA_EXTRA_AMOUNT=4` that drives the PC sequencer. Replaying only the 108 counted entries → PC never starts → 0 tasks. Capturing the full 224 words fixed it.
3. **`task_number=1` with a single `subcore_task[0]`** for a one-task matmul (not librknnrt's `task_number=3` + three subcores, which expects 3 completions that a single regcmd never signals).

Not required (verified revertible): non-cacheable buffers, `RKNPU_ACT_RESET`, PRIME/dmabuf export. **Cacheable buffers + explicit `MEM_SYNC` (librknnrt-style) work fine** on the card node.

**Cross-check:** independently confirmed by building & running [mtx512/rk3588-npu](https://github.com/mtx512/rk3588-npu) on this board — its *synthesized*-regcmd matmul also returns `RKNPU_SUBMIT 0` with correct output. That project already does M2.2 (parameterized regcmd synthesis) and even runs llama2.c, so Phase 2–3 can build on it (with attribution) rather than re-deriving from scratch.

**Next — M2.2:** parameterize the regcmd (emit the ~90 shape-invariant constants + compute the M/K/N fields from the M1.3 dictionary + patch addresses) and validate against CPU across a shape sweep — or adopt mtx512's `npu_matmul.c` synthesis as the foundation.

### M2.2 — synthesize a matmul from scratch ✅ (arbitrary M/K/N) (2026-06-12)
`rknpu_synth.c` builds the regcmd from the template + M1.3 closed-form dim fields and validates the NPU result against a CPU reference with **real (distinct) data** (small ints, exact in fp16). **Correct (0 mismatches)** for: `4×32×16, 8×32×16, 4×32×32, 8×32×48, 4×64×16, 4×96×16`.

Two register/template corrections (all-ones had masked them):
- **reg `0x1030` = `K*N*2`** (weights byte-size), not `K*32` — it depends on K *and* N (the K-only M1.3 sweep couldn't tell).
- The rest of the M1.3 closed forms held.

**Operand memory layout** (the real find — derived by encoding indices in the data and dumping the buffers librknnrt's regcmd actually reads):
- **Feature A: flat row-major `[M][K]`** — K contiguous, *not* tiled.
- **Weights B: `[Ktile][N][32]`, tile-major** — K split into 32-channel tiles; within a tile the weights are transposed to output-channel-major (`buf[t*N*32 + n*32 + kk] = B[t*32+kk][n]`). This reduces to the simple transpose at K=32.

**Remaining edge:** when **K>32 AND N>16 simultaneously**, results diverge (C[0] is right, the rest mis-placed) → the weights need *nested* K×N tiling (N also tiles by 16) and the output needs de-tiling. Single-tile-in-either-dimension is solved; the nested case is the next step. Then M2.3 (int8/w8a8) and Phase 3 (resident weights).


**Nested-tiling (K>32 AND N>16) — RESOLVED ✅:** it was a weights **layout** bug, not the node/IOMMU issue first hypothesized (that wrong guess came from comparing only the first 16 buffer words — they matched; a *full* 4 KB byte-compare showed the tile-1 region diverging). The correct weights layout is **`[Ntile][Ktile][16][32]`**: N tiles by 16, K tiles by 32, transposed within a tile (`buf[nt*KT*16*32 + kt*16*32 + nl*32 + kk] = B[kt*32+kk][nt*16+nl]`). This reduces to the earlier layouts when NT=1 or KT=1 — which is exactly why single-K-tile and single-N-tile both worked and only the doubly-tiled case broke. Feature stays flat `[M][K]`. **M2.2 is now complete: from-scratch fp16 GEMM synthesis correct for arbitrary M/K/N (validated vs CPU through 32×512×128).**

**Tooling:** `rknpu_synth.c` (synth+validate), `layout_probe*.c` (operand-layout derivation via librknnrt + dump shim).

## Future Investigations

### Fuzzing for the Undocumented int4 Multi-M Scheduler — RESOLVED & PROVEN ✅
**The Goal:** Discover the undocumented register(s) that enable the NPU to natively process multiple rows ($M>1$) in a single hardware int4 pass, bypassing the need for PC-chaining or separate ioctls.

**The Solution & Proof:**
Through exhaustive search and layout brute-forcing (using `tools/i4_multim_fuzz.c` and `examples/test_layouts.c` on the RK3588 silicon), we have definitively proven that the NPU **is fully capable of native multi-M execution in INT4 mode**.

The breakthrough relies on two key findings:
1. **The PPU Stride (`0x405c`) Constraint:** Leaving the output PPU stride register `0x405c` at its default value `0` (instead of setting it to `(mc-1)<<16` like in INT8) is the critical trigger. Setting it to non-zero corrupts the output and prevents rows $>0$ from being written.
2. **The Stride-2 Element Layout Mapping:** In INT4 mode, each output element is an `int16_t` (2 bytes), but the output DMA default logic steps by `int32_t` (4 bytes) width. This causes the NPU to write computed rows with a stride of exactly **2 rows of `int16_t` per computed row** (256 bytes per row instead of 128 bytes for $N=64$).
   - Therefore, computed logical row `m` maps to output buffer row `2 * m`.
   - The register formula to compute $H$ rows is: `mc = 2 * H`. For example, setting `mc = 32` computes exactly 16 rows, placing them in output rows 0, 2, 4, ..., 30. Setting `mc = 64` computes exactly 32 rows, placing them in output rows 0, 2, 4, ..., 62.

**Verification:**
The automated test `examples/test_layouts.c` was expanded to test large shapes (e.g., $M=32$ which corresponds to `mc=64`). All 32 rows computed natively in a single hardware submit, and after a simple de-tiling gather copy (reading from even rows `2 * m * N` in the output buffer), the NPU output matched the CPU reference **bit-exactly across all rows with 0 mismatches**.

This completely eliminates the need for the slow, row-by-row PC-chaining fallback (`run_chain_i4`), unlocking massive prefill performance for INT4.



## Correction + open lever — the two CBUF budgets (2026-07-08)
Two distinct budgets were characterized this session, and only the first was hunted for a lift:
- **Activation budget** `rows·K ≤ 16384` int4 — fuzzed to *enlarge* (`0x1104`/`0x1014`/`0x104c` bankprobe);
  **no per-task register lifts it** → treated as init-time-fixed.
- **Weight budget** `Ncore·K ≤ 131072` int4 — found later (wide-N K=2048 de-tile: 1 block computes, 2 drop).
  **Measured, worked within (guard + N-subslice); NEVER fuzzed to enlarge.**
Honesty on "independent": the coupling check (`M=32 K=1024 N=128`, both budgets at limit → bit-exact) proves
they're **simultaneously satisfiable** (enough for the N-subslice), but it's ONE data point — it does NOT
distinguish "two separate banks" from "one shared bank ≥147456 (=16384+131072)". Distinguishing needs a test
that *exceeds* one budget while the other is at its limit.
- **★ Open lever: enlarge the WEIGHT budget.** Never hunted. If a weight-bank register lifts `131072`, wide-N
  batches in a SINGLE submit (no N-subslice needed). Natural extension of the (activation-only) bank hunt —
  sweep for a reg that raises the `Ncore·K` ceiling at fixed K (e.g. K=2048: does any reg let 2+ blocks compute?).

## NVDLA resolves the bank question + a new lever (2026-07-08)
> ⚠ **SUPERSEDED — the "bank SPLIT (`D_BANK`)" lever below was REFUTED later the same day by the conv `.rknn` capture: RK3588 has no D_BANK register (weight swept to 4.7 MB, no bank-count reg found). The int4 8-row@K2048 cap is a hardware limit gated by `0x107c`, not a reallocatable bank partition. See the SUPERSEDED banner on the "NVDLA banks & batch … D_BANK is the lever" section and [regcmd-ISA-Reference](regcmd-ISA-Reference.md).**

NVDLA CBUF spec (nvdla.org/hw/v1/hwarch.html): CBUF = **2–32 banks × 4–32 KB**, storing BOTH weight and
feature data; register **`D_BANK` (`0x50bc`, CDMA block) = "Number of data banks and weight banks in CBUF."**
- **Resolves "independent vs coupled":** the two budgets are **separate bank pools** (weight banks vs data
  banks) — NOT one shared bank. Drops the earlier one-data-point hedge; the coupling check passed *because*
  they're distinct pools. Our numbers fit: activation 16384 int4 = **8 KB ≈ 1 data bank**; weight 131072 int4
  = **64 KB ≈ 2 banks**.
- **Why no register lifted either budget:** NVDLA says max batch is *"a hardware limitation in the design
  specification"* — the CBUF total is FIXED; a split register only **re-partitions** it, can't grow it.
- **★ New untried lever — the bank SPLIT (`D_BANK`).** We only fuzzed the CNA/DPU/PPU present-registers
  (`0x201`/`0x801`/`0x1001`); the split control lives in NVDLA's **CDMA (`0x50xx`) block** — un-swept, or set
  at init (NVDLA notes per-op programmability "isn't fully detailed"). Finding the RK `D_BANK` equivalent would
  let us **trade weight banks ↔ data banks**: more data banks → more rows at K=2048 (past 8) at the cost of
  narrower N/submit (fewer weight banks); or the reverse for wide-N. A workload-tunable partition, not free
  capacity. Fuzz target: the `0x50xx` region (if present in the int4 regcmd) + the ACTION/init path.

## int4-streaming: preliminary NEGATIVE (2026-07-08) — not yet viable
Direct test (int8-stream config applied to int4 via the static diff): at K=2048 M=16, best = **1 contiguous
row** (vs 8 batch) — int4 collapses to single-row under the streaming config, i.e. it REQUIRES the batch
trigger (0x405c=0/stride-2) for multi-row. Strong preliminary evidence int4 is **batch-only** (8-row cap
@ K=2048 is real). NOT fully conclusive: the config was partial (K-passes/CNA-hint/weight-layout left at
int4-batch). The rigorous close is the int8-batch A/B (get the complete verified selector set incl weight
layout) — item ② remains the path. Until then, int4 streaming = not-yet-viable, batch is production.

## int8-batch / int4-stream: CLOSED (2026-07-08) — modes are dtype-locked (not viable via config)
The int8-batch controlled A/B (ork_npu_probe_i8_mm): int8 STREAM baseline correct (8 rows contiguous), but the
int4 batch recipe (0x405c=0 + mc_phys=2M) on int8 → GARBAGE (all rows unfound), schedule on OR off. With
RESULT 1 (int4+stream config → 1 row), the batch/stream modes are **dtype-locked**: int4 batch-only, int8
stream-only, neither crosses via these registers. So there's no int8 batch mode to A/B-diff → no transferable
selector → **int4 streaming is not-yet-viable via register config** (int4's 8-row cap @ K=2048 is the real
ceiling). Not fully impossible: int8-batch garbage may be a weight-layout mismatch (untested); single shape.
Infra kept for a future attempt: ork_i8_fuzz_add/clear, ork_npu_probe_i8_mm, ORK_I8_PROBE_SCHED, fuzzer i8batch mode.

## CORRECTION (2026-07-08): "dtype-locked" overstated — int8 DOES batch; the real axis is activation residency
Prior entry overclaimed. NVDLA batch mode is not dtype-specific, and int8 ALREADY batches — its mg_max*64
M-scheduler (128 rows @ K=2048, one weight load) IS multi-row/shared-weight batching. The i8batch experiment
only showed that int4's SPECIFIC recipe (0x405c=0 + mc_phys stride-2) garbles int8 — because that recipe is an
int4 OUTPUT quirk (int16 clipped into int32 slots → stride-2 packing), not "int8's batch register". int8's
native-int32 output batches with CONTIGUOUS output via the M-scheduler; it has no 0x405c=0 quirk.
Real axis: **activation RESIDENCY**. int8 STREAMS activations (128 rows, schedule-limited); int4 holds them
RESIDENT (8 rows, CBUF-limited). Open question (unchanged): can int4 stream activations like int8?
**The conclusive test not yet run:** apply int8's FULL streaming config to int4 INCLUDING int8's resident
WEIGHT LAYOUT (my `stream` test used int8's registers but left int4's batch weight tiling — incomplete).
int4-streaming = OPEN (not closed), pending the weight-layout test.

## int4-streaming: CLOSED with a comprehensive negative (2026-07-08)
Exhausted the board-testable config space: int8-stream registers on int4 (contiguous 0x405c, M-count,
0x1040 schedule swept incl the int8 value 0x84) x BOTH A-layouts (per-row & interleaved) → best 1 contiguous
row (cap 8). int4's weight layout is datapath-fixed (can't substitute int8's [NT][KT][32][32]). And the vendor
(librknnrt) tiled int4 M across 12 BATCH tasks rather than streaming — a HW-informed choice. Conclusion:
**int4 is batch-only; streaming is not reachable via regcmd config.** 8-row cap @ K=2048 is the real ceiling.
(Not metaphysically impossible — a stream weight layout the vendor never used could exist — but out of
regcmd reach and argued-against by vendor behavior.)

## ★ ACTIONABLE lever (replaces streaming): round-robin single-core batch across matmuls
Since int4 large-wide-N can't stream and the batch only fits/helps SMALL matmuls (Ncore*K<=131072, ~1.4-1.6x),
the way to compound the win across a model is round-robin single-core dispatch (librkllmrt-style: round-robin,
never core_mask=0x7) of the MANY small matmuls (decode GEMVs, QKV/gate-up, MoE experts): each core batches a
whole small matmul independently → (a) fits the per-core weight budget, (b) NO multi-core sync barrier (the
prefill-parity wall), (c) cross-matmul overlap (one core's submit hides another's CPU tile/de-tile). Hybrid
dispatcher: big matmuls → multi-core N-split batch (current); many small → round-robin single-core batch.
This is a scheduling change in run_i4/dispatch, orthogonal to the batch mechanism. Highest-value int4 lever now.

## Terminology fix (2026-07-08): batch vs streaming = activation RESIDENCY (not loose "int8 batches")
Prior entries loosely called int8's M-scheduler "batching." Corrected, rigorously:
- TRUE BATCH (NVDLA D_BATCH_NUMBER) = weight resident AND all rows' activations RESIDENT in CBUF; capped by
  BOTH budgets. int4's multi-M (0x405c=0/stride-2, rows*K<=16384 & Ncore*K<=131072) IS true batch → 8 rows@K2048.
- WEIGHT-STATIONARY STREAMING (the M-scheduler) = weight resident, activations STREAMED (not CBUF-resident;
  AGENTS.md "activations need not be CBUF-resident, mg_max*64 is the ceiling"). int8/fp16 use this → 128 rows@K2048.
  This is NOT batch mode — I was wrong to call it that.
int4 uses true batch because it has no working streaming schedule (int8-stream config on int4 → 1 row).
int8 uses streaming (16x more rows, no activation-residency cap). Whether int8 has a SEPARATE true-batch mode
(real D_BATCH_NUMBER config, not int4's stride-2 recipe) is UNTESTED — my i8 experiment used int4's recipe,
which is int4-output-specific and garbled int8's native-int32 output, so it says nothing about int8 true-batch.

## NVDLA confirms: true batch mode IS supported in INT8 (2026-07-08)
NVDLA hwarch spec: batching is precision-AGNOSTIC — "Feature supporting: batch (Yes/No), affects Convolution";
"MAX Batch number 1~32, limited by convolution buffer size" (not precision); and the small NVDLA config is
INT8 and ships D_BATCH_NUMBER/D_BATCH_STRIDE. So int8+batch is documented/shipping.
=> RK3588's int8 datapath SHOULD have a real D_BATCH_NUMBER true-batch mode; ork just uses STREAMING instead
(weight-stationary M-scheduler, 128 rows > batch's ~8 — no perf reason to use int8 batch). int8-true-batch on
RK is untested (would be D_BATCH_NUMBER in the CDMA/0x50xx block, never swept); pure architectural completeness.
Sharper open puzzle: NVDLA gives BOTH modes generally, RK int8 streams, but RK int4 CANNOT stream (tested) —
so the anomaly is RK's int4 datapath lacking a streaming schedule (likely a Rockchip impl choice), not batch.

## int8 true-batch fuzz — DESIGN (2026-07-08), but low-priority (do NOT burn board time)
How to fuzz for int8 D_BATCH_NUMBER, bounded + fast-fail:
- WHERE: block 0x2001 / offsets 0x5000-0x50c0 (NVDLA D_BATCH_NUMBER @ 0x5058, D_BATCH_STRIDE) — the one block
  the int8 mm path never touches; + a few unset 0x201 CNA regs.
- SIGNAL (no "more rows" tell — int8 batch < streaming): reuse the i8batch ROWMAP. baseline stream = physrow=m
  (contiguous). Flag any (reg,val) where rows are still FOUND but at NON-contiguous physrows (batch stride) =
  candidate D_BATCH_STRIDE/NUMBER; garbage=wrong; unchanged=inert.
- FAST-FAIL: ORK_I8_PROBE_TO_MS=1500 (kernel job-timeout in-process) + state-resume + blacklist + external
  timeout backstop + autonomous HA power-cycle. Palette: batch#{2,4,8,16} x stride{N,2N,4N,M*N}.
WHY LOW-PRIORITY (recommend NOT pursuing on-board): (1) the 0x2001/0x50xx CDMA block is almost certainly ABSENT
from the int8 mm regcmd — can't inject a whole foreign block the sequencer doesn't parse (ignored or wedge);
(2) zero perf value (int8 streaming beats batch 16x); (3) injection is hard-wedge-prone. The DEFINITIVE test is
not a fuzz — it's a librknnrt CAPTURE (LD_PRELOAD regcmd_capture): does the vendor ever emit an int8 regcmd with
D_BATCH_NUMBER set? If the vendor's int8 is always M-scheduler/streaming, int8 true-batch isn't used in practice.

## RKNPU per-precision MODE-SUPPORT MATRIX (2026-07-08) — the mapping goal
NVDLA describes both modes for all precisions; RKNPU is a Rockchip impl that may expose them selectively. Map:
| dtype | weight-stationary STREAMING (M-scheduler) | true BATCH (resident, D_BATCH_NUMBER) |
|-------|-------------------------------------------|----------------------------------------|
| int4  | NO  (tested: int8-stream cfg -> 1 row)    | YES (0x107c=K/16, stride-2, 8 rows@K2048) |
| int8  | YES (mg_max*64, 128 rows@K2048)           | ? (i8fuzz: sweep present regs for a layout-change/batch toggle) |
| fp16  | YES (M-scheduler)                         | ? (untested — fp16 batch probe TODO)   |
Reasoning so far: RKNPU STREAMING is NOT precision-agnostic — int8/fp16 have it, int4 does NOT (likely int4
was a later/narrower add for LLM-weight compression: Rockchip wired only the resident-batch path for int4, not
the streaming M-scheduler the NVDLA base gives int8/fp16). The BATCH axis: int4 YES; i8fuzz + a fp16 probe pin
down whether batch is precision-agnostic on RKNPU or also selectively exposed. Note: the 0x2001/0x50xx CDMA
block (NVDLA D_BATCH_NUMBER) is ABSENT from REGCMD_I8/I4 — so if int8 batch isn't reachable via a present reg,
its D_BATCH_NUMBER lives in a block the mm regcmd doesn't carry (definitive answer = a librknnrt capture).

## RKNPU batch is NOT precision-agnostic in the matmul regcmd (2026-07-08, i8fuzz result)
i8fuzz swept ALL present int8 regs (1504 candidates) for a batch toggle via the layout-change signal. Only
0x1068 relocated cleanly — and it's a uniform output ROW-OFFSET (physrow=m+v), NOT a batch stride (mapped via
i8map: v=1→rows shift +1, v=4→+4). 0x1014/0x1028 = partial stride scrambles. NO coherent D_BATCH_NUMBER among
present int8 regs. Combined with the 0x2001/0x50xx CDMA block being ABSENT from REGCMD_I8:
=> RKNPU exposes conv modes PRECISION-SELECTIVELY in the matmul regcmd: int4 = batch-only (0x405c/0x107c
present; int4 can't stream), int8/fp16 = streaming-only (M-scheduler; the batch/CDMA block isn't in their mm
regcmd). NVDLA supports both for all precisions in the abstract; Rockchip wired each precision for one mode.
Matrix: int4 {stream:NO, batch:YES}; int8 {stream:YES, batch:NOT-in-mm-regcmd}; fp16 {stream:YES, batch:untested}.
Definitive int8-batch-exists check = librknnrt capture (is D_BATCH_NUMBER/0x50xx ever emitted?), not a board fuzz.

## Prior-art check (2026-07-08): our int4-mode RE is beyond the public state; CBUF banks corroborated
Web survey of RK3588 NPU RE prior art:
- mtx512/rk3588-npu (deepest public regcmd RE): fp16 + int8 matmul ONLY — no int4 matmul, no batch/stream modes.
- Tomeu Vizoso Mesa/Teflon: conv-focused (MobileNet), TRM+NVDLA docs; not matmul-modes/int4.
- jas-hacks (Matharu), P-H Husson: early RKNN matmul-API RE. clehaxze: perf benchmarks only.
- Public consensus: "RK3588 supports W8A8 for LLM, not W4A16" — int4-for-LLM treated as unsupported.
=> No public prior art on int4 streaming/batch or int8 batch-vs-streaming. Our int4 batch scheduler (0x107c,
stride-2) + the streaming-negative + the mode-matrix are NEW / state-of-the-art.
CORROBORATION: mtx512 documents CBUF = 16 banks x 64KB (NPU_CBUF_BANKS=16, NPU_CBUF_BANK_SIZE=64KB, 1MB total).
This independently confirms our budgets: int4 WEIGHT budget 131072 int4 = 64KB = EXACTLY one CBUF bank;
activation budget 16384 int4 = 8KB. The batch-CBUF model we RE'd falls out of the documented bank structure.

## NVDLA banks & batch (2026-07-08): batch uses the DATA-bank partition, not all banks — D_BANK is the lever
> ⚠ **SUPERSEDED — REFUTED later the same day by the conv .rknn capture (see [regcmd-ISA-Reference](regcmd-ISA-Reference.md) "D_BANK: no explicit bank-count register exists", and `REGCMD_CAPTURE_WIP.md`).** RK3588 has **NO D_BANK register**. A single-conv `.rknn` was built (`build_conv.py`), captured on board, and its weight swept up to **4.7 MB (≫ the ~1 MB CBUF)** — **no discrete bank-count register appears**; conv_big still runs 1 submit / 3-core split. CBUF residency is purely dim-derived via `0x1030`/`0x107c`(=K/16)/`0x1010`/`0x1040`, not a bank knob. ⇒ the int4 `rows×K ≤ 16384` (8-row @ K2048) cap is gated by **`0x107c`** and **no register combination escapes it** — a **hardware limit**, exhaustively fuzzed (the `stream2` sweep tops out at exactly 16384/K). The "~15 idle banks, reallocate via D_BANK for 8–15×" claim below is **wrong; do NOT chase D_BANK bank-realloc.** The real levers within the hard cap are (1) **dense int4 K/32 encoding** — halve `0x107c` (K/16→K/32) to double rows/submit (a packing change), and (2) **multi-task STRATEGY C chaining** (`ork_mm_run_i4_incr` / `ORK_I4_INCR`) — one weight load amortized over arbitrary M by chaining ≤8-row tasks.

NVDLA spec: CBUF is split into WEIGHT banks + DATA banks via a single register D_BANK (0x50bc CDMA / 0x605c CSC,
"Number of data banks and weight banks in CBUF"). Batched activations live in the DATA-bank partition ONLY
(subset), weight in the weight banks; total banks fixed (RK3588: 16 x 64KB = 1MB, per mtx512). So batch is NOT
all-banks — it's the data-bank allocation, and more data banks <=> fewer weight banks (a trade).
REFRAME of our int4 numbers: weight budget 64KB = 1 weight bank; activation 8KB = a fraction of one data bank
=> int4 batch uses ~1 of 16 banks; ~15 idle. The 8-row@K2048 cap is a D_BANK ALLOCATION artifact, NOT a
fundamental CBUF limit — reallocating data banks via D_BANK could grow int4 rows/submit ~8-15x toward full CBUF.
BUT D_BANK (0x50bc/0x605c) is in the CDMA/CSC region, NOT in the blocks our matmul regcmd carries (0x10xx/30xx/
40xx); our bankprobe of present CNA regs found no lever. So D_BANK is init-time / out-of-matmul-regcmd — the
same absent-block wall as D_BATCH_NUMBER. Lever reachable only via a librknnrt capture or the driver/DT init
path (does the vendor set D_BANK differently per dtype/op?), not our per-task fuzz. HIGH-VALUE follow-up.

## Mode-support matrix COMPLETE (2026-07-08): batch is precision-selective in the present matmul regcmd
f16fuzz done (1504 combos): fp16 mirrors int8 EXACTLY. Only clean layout-mover is 0x1068
(=0x1/0x2/0x20000001/0x20000002, MOVED>=6) = the dtype-agnostic OUTPUT ROW-OFFSET (physrow=m+v),
NOT a batch toggle. No D_BATCH_NUMBER-equivalent among present fp16 regs (same as int8).
FINAL MATRIX:
  int4 -> TRUE BATCH (0x405c=0 stride-2), validated 1.4-1.6x, act CBUF-resident, cap = data-bank alloc
  int8 -> STREAMING (M-scheduler), no batch toggle in present regs, only 0x1068 row-offset, cap mg_max*64
  fp16 -> STREAMING (M-scheduler), no batch toggle in present regs, only 0x1068 row-offset (== int8)
CONCLUSION: in the present matmul regcmd, batch is PRECISION-SELECTIVE (only int4 carries a batch config;
int8/fp16 are streaming-only). The int8/fp16 batch levers (D_BATCH_NUMBER, D_BANK) are init-time /
out-of-matmul-regcmd (absent-block wall) — reachable only via librknnrt capture or the driver/DT path,
NOT our per-task fuzz. This closes the fuzz-reachable mode-mapping; further batch RE needs a capture path.

## int4 incremental-task batch (STRATEGY C) — WORKING, matmul-level win (2026-07-08)
Replicated the vendor's task_number=N int4 pattern (decoded from a librknnrt w4a4 capture): ONE resident
int4 weight, M rows in ONE submit as task[0..1]=full priming + task[2..M-1]=12-CONFIG incremental tasks
that advance ONLY the A input (0x1070 += K/2) and C output (0x4020 += N*2); weight IOVA (0x1110) CONSTANT
=> weight loaded ONCE and reused across all M rows. Implemented as ork_mm_run_i4_incr (src/npu.c, probe
tools/i4_incr_probe.c). Key regcmd details: compact task = 16 pairs / 32 words, regcfg_amount=12; PC-chain
amount (0x0014) encodes the TARGET size — 0x0037 into a full task, 0x0007 into a compact. **2 full priming
tasks are REQUIRED** — chaining a full task straight into a compact returns errno 110 (chain-walk fail).
Cold output buffer needs a warmup rep.

Result (bit-exact vs CPU int4, single-core, K=512..4096 N=64): **7–28× vs per-row separate submits,
1.3–2.7× vs STRATEGY-A (stride-2 in-task batch)**. The advantage over STRATEGY A widens past STRATEGY-A's
Hcap cap (16384/K rows): at M=64/K=512, incr = 2.71× (per-row cost 9.6→4.1→2.6 µs as M=8→32→64, since the
weight is loaded once and amortized — STRATEGY A re-streams per capped submit). NOT Hcap-capped: batches
arbitrary M with one weight load. Still matmul-level (dummy data, correctness-checked); end-to-end model
validation + production wiring (ork_mm_run_i4 M>1, multi-core) are the follow-ups.

### STRATEGY C wired into ork_mm_run_i4 (ORK_I4_INCR) + production-path perf (2026-07-08)
Wired STRATEGY C into the public int4 path (gated ORK_I4_INCR, default off): N-block loop (one incremental
chain per 64-wide N-block; A packed once, block-independent), int4 mode-reset on entry, Sk/Sn==1 guard
(falls back to STRATEGY A otherwise). BIT-EXACT through the production path — i4 example passes all shapes
(M up to 64, N up to 256) both default and ORK_I4_INCR=1.
Production-path perf (ork_mm_run_i4, ms/matmul):
- **N=64 (single N-block): C WINS 1.6-2.2x** — M64/K2048 0.900->0.550, M64/K512 0.457->0.205, M32/K512 0.236->0.128.
- **N>64: C LOSES** — M64/K2048/N256 A 1.338 vs C 2.143; M128/K2048/N512 A 3.918 vs C 8.046. Reason: STRATEGY
  A (run_i4_mc) splits N-blocks across 3 CORES; the wired STRATEGY C is SINGLE-CORE (serial N-blocks), so the
  per-block weight-amortization win is outweighed by the lost 3-core N-parallelism.
⇒ **Single-core STRATEGY C only wins at N=64** (or the batched-verify M>1/single-tile regime). To beat the
multi-core STRATEGY A at realistic wide N, STRATEGY C must go MULTI-CORE (split the N-blocks across the 3-core
pool, mirroring run_i4_mc's threading) — the clear next step. Committed gated/default-off as a validated primitive.

## NVDLA documented optimization techniques -> ork-driver primitives (2026-07-08)
NVDLA (which the RK3588 NPU derives from) documents a technique list (hwarch.html / unit_description) worth
mining to optimize our primitives. RK3588 is NVDLA-DERIVED not identical, so each is a PROBE candidate, not
assumed present. Mapping:
| NVDLA technique | NVDLA description | ork-driver status |
|---|---|---|
| Direct Convolution (DC) | wide MAC pipeline, base mode | HAVE — our matmul path |
| **Batching / multi-batch DC** | fetch multiple input cubes for ONE weight set; weights reused, weight-load hidden | **HAVE — int8 M-scheduler + int4 STRATEGY C** (exactly this: weight loaded once, amortized over M rows). The int4 96-task pattern IS NVDLA batching. |
| Winograd | reduces multiplications for certain filters | conv-only; unexploited (only if we do on-NPU conv) |
| **Sparse weight/feature compression** | 60% sparse ~= half the memory bandwidth | **UNEXPLOITED + high-value**: decode is weight-DMA-BANDWIDTH-bound ([[npu-chaining]] 07-02d), so sparse-weight streaming directly attacks the decode wall for pruned/2:4 models. PROBE whether RK3588 wires it. |
| Second memory interface | on-chip buffering, weight prefetch, more BW / less latency | possibly the SRAM path; unexplored — weight-prefetch lever |
| CBUF banking (D_BANK) | configurable #data vs #weight banks, 2-32 banks x 4-32KB | RK3588 has NO explicit D_BANK register (dim-derived; see D_BANK finding above) — diverges from NVDLA here |
| Atomic-C / Atomic-K | MAC parallelism, C=16-128, K=4-16 | HW-fixed on RK3588 |

Kernel-source note (rknpu_job.c, rockchip develop-6.1): the submit path is DTYPE-UNIFORM — no int4/DT_I4
branch, no stub/TODO. So the int4-vs-int8 asymmetry (int4 multi-task vs int8 single-submit M-scheduler) is
NOT a kernel gap; it's either a librknnrt userspace choice (bypassable by crafting our own regcmd) or a HW
datapath property. Decisive open test: fuzz int4 with int8-style in-task-batch regs (0x4034=M-1, 0x405c=
(M-1)<<16) for >1 row in one task.
TOP UNEXPLOITED LEVERS from this list: (1) sparse weight compression (decode-bandwidth), (2) second memory
interface / weight prefetch. Both PROBE candidates on RK3588.

## int4 single-submit streaming: DEFINITIVELY closed (2026-07-08, broadened sweep)
Q (from the NVDLA unified-datapath reading): NVDLA lists INT4 as a supported precision and implies ONE
datapath across precisions (Atomic-C/K parallelism scales inversely with width; no per-precision pipeline)
— so int4 "should" stream like int8. Does RK3588 hide it in the driver?
Test: `i4_multim_fuzz stream2` — beyond the earlier `stream` (which fixed int4's 0x1044/0x107c + swept
0x1040 -> 1 row), this sweeps the two precision-SIZING regs it held fixed: 0x107c (entries-per-slice =
activation-residency/Hcap source) x 0x1044 (K-passes) x 0x1040, with int8's contiguous streaming row-encoding,
both A-layouts. Crafts the regcmd DIRECTLY (bypasses librknnrt), so a driver gap can't hide the HW capability.
Result: best contiguous rows == the cap EXACTLY (K2048 M32: 8=16384/2048; K4096 M16: 4=16384/4096); NO combo
exceeds it. (Mild positive: with the right 0x107c/0x1044, int4 emits CONTIGUOUS output up to the cap, not just
stride-2 — but still capped.)
⇒ **RK3588 int4 DIVERGES from NVDLA's unified-streaming model.** int4 activations are HARD-capped at 16384
resident elements (batch-mode CBUF budget); no register combination escapes it. int4 does NOT single-submit
stream like int8 — a genuine HW datapath property, NOT a driver/librknnrt gap (kernel is dtype-uniform; fuzz
bypasses librknnrt). So the vendor's 96-task pattern is int4's NATIVE streaming (weight-stationary via
multi-task), and STRATEGY C (multi-core incremental) is the correct & only many-row int4 path. Question CLOSED.

## TODO: RK3576 int4 datapath comparison (blocked on hardware access)
User has an RK3576 board at a remote site (offline until a technician restores internet). When reachable,
run on it to settle "generation-wide int4 resident-cap vs RK3576-completed-it":
1. `i4_multim_fuzz stream2` (K2048 M32, K4096 M16) — does RK3576 exceed the 16384/K int4 activation cap
   (= single-submit int4 streaming that RK3588 lacks)?
2. `mm_cap` create-probe of the mixed types (7/8/12 w4a16, 11 w4a8) — does RK3576 ACCEPT native w4a16 where
   RK3588 rejects (-5)? (Web hint: "larger models benefit from W4A16 memory efficiency on RK3576" — untested.)
3. Validate src/soc/rk3576.c params (currently validated=0, inherited from RK3588): cbuf_elems, nmax, ks.
Reachable evidence so far argues these chips share the SAME int4 generation (RKNN header groups RK3588/3576
identically for int4 alignment; "fundamentally same NPU architecture") — so expect the same cap; the w4a16
create-probe is the most likely place to find a real per-SoC difference.

## UPDATE: the two "unexploited NVDLA levers" PROBED on RK3588 — both NOT viable (2026-07-08)
Investigated the two levers flagged above (sparse-weight compression, second memory interface). Both turn
out unavailable on RK3588 — RK3588 is an NVDLA DERIVATIVE that doesn't populate these NVDLA-documented features:
- **Second memory interface (on-chip SRAM weight prefetch): NOT viable.** The rknpu driver HAS the hook
  (RKNPU_MEM_TRY_ALLOC_SRAM=1<<8, RKNPU_MEM_TRY_ALLOC_NBUF=1<<9, RKNPU_GET_TOTAL/FREE_SRAM_SIZE actions), but
  `tools/sram_probe` queries **NPU SRAM total = 0 bytes** on the board; no NPU sram node / reservation in the
  DT; nothing in probe-time dmesg. So there's no on-chip SRAM to allocate weights into — the lever has no
  hardware to stand on here. (Not a DT-reservable win at userspace level; would need the SoC to populate it.)
- **Sparse weight compression: NOT accessible.** No sparse/compress/prune hook anywhere in rknn_api.h /
  rknn_matmul_api.h (no API surface), and no compression-enable register in our captured matmul/conv regcmd.
  Driving NVDLA-style CSC weight-decompression would need deep regcmd RE + a way to feed compressed weights
  that RKNN never exposes — a speculative, uncertain track, NOT a near-term lever.
⇒ Neither NVDLA "unexploited lever" pays off on RK3588. The genuinely-shipped decode/int4 lever remains
STRATEGY C (multi-core incremental batch). Reinforces: RK3588 implements a SUBSET of the NVDLA technique menu.

## CORRECTION: NPU SRAM is PRESENT (~1MB) but DISABLED on this RK3588 board — enablement path found (2026-07-08)
Earlier entry ("NPU SRAM=0 -> no HW, not viable") was WRONG (user: the SRAM is ~1MB, just not turned on).
Corrected from the rknpu v0.9.8 source + board DT:
- RK3588 config has nbuf_size=0 (no NBUF — the 1MB NBUF is the RK3576 config). On RK3588 the on-chip fast
  memory is the SYSTEM SRAM, and the driver wires it to the NPU via a DT `rockchip,sram` phandle
  (rknpu_find_sram_resource -> of_parse_phandle(dev->of_node,"rockchip,sram",0) -> ioremap -> sram_mm allocator;
  then RKNPU_MEM_TRY_ALLOC_SRAM=1<<8 allocates from it; GET_TOTAL_SRAM_SIZE reports it).
- The board HAS system SRAM (DT nodes sram@ff001000, sram@10f000), but `npu@fdab0000` has NO `rockchip,sram`
  phandle (only `iommus`) -> driver never finds an SRAM region -> GET_TOTAL_SRAM_SIZE=0 (what sram_probe saw).
ENABLEMENT (kernel/DT change + reboot, not userspace): add to the npu node a `rockchip,sram = <&npu_sram>;`
phandle pointing to an ~1MB region carved from a system SRAM block, then reboot. Driver then registers it and
RKNPU_MEM_TRY_ALLOC_SRAM works. (NPU uses IOMMU here, so the driver must IOMMU-map the SRAM region too.)
EXPERIMENT once enabled: alloc a hot weight tile / activation in the 1MB SRAM, run a matmul reading it, measure
bandwidth vs DRAM (~11GB/s) — directly targets the weight-DMA-bound decode wall. CAVEAT: 1MB holds only a hot
TILE (int8 K2048 -> N<=512; int4 K2048 -> N<=1024), not the full per-token weight stream, so the benefit is a
hot-subset / activation-residency win, not a whole-model one. Still the one on-chip-bandwidth lever RK3588 has.

## Sparse/weight-compression: CLOSED on RK3588 (toolkit-authoritative, 2026-07-08)
Checked whether the sparse-weight NVDLA lever is real on RK3588 via the vendor toolkit (rknn-toolkit2 2.3.2,
which exposes sparse_infer / compress_weight / model_pruning in rknn.config):
- **sparse_infer=True → EXPLICITLY REJECTED**: `E config: rk3588 does not support sparse inference` (ValueError).
- **compress_weight=True / model_pruning=True → NO-OP**: built fine but produced BYTE-IDENTICAL .rknn size
  (317638) even on a 60%-zeroed conv weight (would shrink hugely if compression engaged). So no compression.
⇒ RK3588 has no usable weight sparsity/compression. Vendor toolkit (which enables sparse_infer for OTHER
family SoCs) disables it for rk3588 specifically — authoritative that the datapath isn't wired.
Considered a fuzz approach (user idea): fuzz for a register that makes a DENSE op produce garbage (=candidate
sparse-enable), then validate with compressed weights. SOUND in general, but not completable on RK3588: (1)
"produces garbage" is too noisy — nearly any mis-set mode reg garbles output, can't isolate sparse; (2) phase-2
validation needs weights in the real compressed format, which the RK3588 toolkit refuses to produce (no oracle);
(3) vendor says unsupported. So the garbage-fuzz + validate-with-compressed strategy is the RIGHT method to try
on a chip where sparse EXISTS — add it to the RK3576 task (does rknn-toolkit2 ACCEPT sparse_infer for rk3576?
if so, capture its compressed regcmd+format there).

## NPU SRAM enablement — FULL recipe (2026-07-08, tested on the Rock 5B / DietPi board)
Enabling the ~956KB system-SRAM for the NPU (RKNPU_MEM_TRY_ALLOC_SRAM / GET_TOTAL_SRAM_SIZE) needs TWO parts;
DietPi ships NEITHER, Armbian ships BOTH:
1. **Device tree**: add `rockchip,sram = <&rknpu_sram>` to npu@fdab0000 + carve a region from syssram@ff001000.
   On this board the whole 0xef000 (956KB) was claimed by rkvdec0_sram(0x78000)+rkvdec1_sram(0x77000); headless
   we re-carved: expanded rkvdec0_sram to 0xef000, dropped rkvdec1_sram, removed both rkvdec device `rockchip,sram`
   refs, pointed the npu at it. Verified live (running DT shows npu.rockchip,sram + only rkvdec-sram@0; dmesg:
   `mpp_rkvdec2 ...rkvdec-core: could not find phandle sram` = rkvdec correctly lost it). DTB edit is surgical
   (10-line diff), backed up (.orig-*.bak on board + local).
2. **Kernel config**: `CONFIG_ROCKCHIP_RKNPU_SRAM=y` (gate at rknpu_drv.c:1512 — needs CONFIG_NO_GKI=y [set] AND
   CONFIG_ROCKCHIP_RKNPU_SRAM [DietPi: `# not set`] AND iommu_en). Without it, rknpu_find_sram_resource() +
   rknpu_mm_create() are compiled out -> GET_TOTAL_SRAM_SIZE=0 even with the DT wired. rknpu is BUILT-IN, so
   this requires a KERNEL REBUILD (or booting an Armbian kernel that has it).
Status: DT done+verified; blocked on kernel rebuild with CONFIG_ROCKCHIP_RKNPU_SRAM=y. Payoff still bounded
(956KB = hot tile). Revertible via the .bak DTB.

## NPU SRAM: NO stock kernel enables it (Armbian == DietPi) — custom rebuild is the ONLY path (2026-07-08)
Tested the "swap to Armbian's vendor kernel" idea by pulling Armbian's linux-headers + linux-dtb
vendor-rk35xx .debs (v26.5.1, SAME 6.1.115-vendor-rk35xx base as the board):
- Armbian .config: `# CONFIG_ROCKCHIP_RKNPU_SRAM is not set` — IDENTICAL to DietPi (off).
- Armbian rk3588-rock-5b.dtb: npu node has NO rockchip,sram; rkvdec0/1 claim the whole syssram — IDENTICAL
  to DietPi/vendor-default.
⇒ CONFIG_ROCKCHIP_RKNPU_SRAM is a `bool` with no `default y` (Kconfig), absent from the rockchip base
defconfig AND both distros' configs. So NO stock kernel (DietPi or Armbian vendor) ships it. The
"Armbian has it" recollection = a PATCHED image (ezrknpu / manual RK3588_NPU_SRAM_usage.md), not stock.
A kernel SWAP won't enable SRAM. ONLY path: custom kernel rebuild with CONFIG_ROCKCHIP_RKNPU_SRAM=y (rknpu
is built-in, tristate but =y, so an out-of-tree module can't override it — needs a real image rebuild) + the
DT (already done on the board, revertible via .bak). Weigh against the bounded 956KB (hot-tile) payoff.

---

## Porting the fold / int8 matmul to OTHER Rockchip NPUs — the weight layout is DOCUMENTED (2026-07-29)

Task #39 spent weeks reverse-engineering rkllm's fast int8 "M-fold" matmul on RK3588. The decisive lesson for
future SoC work: **the on-NPU weight (B) pack layout does NOT need board RE — it is published in the RKNN
toolkit header.** Read it, don't grind the board.

**Source:** `airockchip/rknn-toolkit2` → `rknpu2/runtime/Linux/librknn_api/include/rknn_matmul_api.h`
(conversion `rknn_B_normal_layout_to_native_layout`). The B matrix native layout is `(N/subN, K/subK, subN, subK)`,
K innermost, with per-chip `(subN, subK)`:

| chip | dtype | native (N/subN, K/subK, subN, subK) | subN | subK | align |
|---|---|---|---|---|---|
| RK3566 / RK3568 / RK3562 | int8 | (N/16, K/32, 16, 32) | 16 | 32 | 32 B |
| **RK3588 / RK3576** | **int8** | **(N/32, K/32, 32, 32)** | **32** | **32** | **32 B** |
| RK3588 / RK3576 | int4 | (N/64, K/32, 64, 32) | 64 | 32 | 64 B |
| (all) | fp16 | — | — | — | 16 B |

**General weight byte offset** (int8):
```
woff(k,n) = ((n/subN)*(K/subK) + (k/subK)) * (subN*subK) + (n%subN)*subK + (k%subK)
```
The activation and output are NC1HWC2: input A = `nc16` (C2=16 int8 / 8 fp16, M folded into WIDTH) =
`(k/16)*(M*16)+m*16+(k%16)`; output C = `c4` (C2=4, int32) = `(n/4)*(M*4)+m*4+(n%4)`. K,N pad up to the next
multiple of `subK`/`subN`.

**Validated:** RK3588 int8 `woff` reproduces rkllm's matmul bit-exact — `tools/re/validate_layout.c` packs ork's
own A(`nc16`)/W(`woff`), replays a captured rkllm regcmd, de-tiles `c4`, compares CPU ref → **0/9728 mismatch**.

**Porting recipe for a new Rockchip SoC:** (1) read its `(subN,subK)` from the header table above; (2) pack the
weight with `woff`; (3) activation `nc16` / output `c4` are shared; (4) confirm bit-exact with the
`validate_layout` pattern (capture one real regcmd via the `rknpu_dump.so` shim, replay with ork-packed
operands). This turns a multi-week board-RE grind into a header lookup + one confirmation run. See also
[NPU-Quirks] "M-fold weight layout & capture-replay gotchas".