Skip to content

NPU Quirks

Jump to bottom
Michael Fischer edited this page Aug 27, 2026 · 21 revisions

NPU Quirks & Gotchas

Vendor-driver behaviour lives on its own page: Vendor-Kernel-Behaviour. This page is silicon/regcmd behaviour; that one is the shipped rknpu driver's own contracts and defects (IOMMU domains, imports, reset, refcounts). Several "hardware stalls" turned out to be entries there — in particular, job committed / task counter 0x0 / no interrupt has never once been silicon.

Hardware and rknpu-driver behaviours that are silent, order-dependent, or counter-intuitive — the kind that corrupt output or wedge the NPU without an obvious error, and that a fresh reader would otherwise re-derive the hard way. Each entry: the symptom, the root cause, and the guard ork-driver applies.

Scope note: perf-tuning levers live in the Optimization Roadmap; register/encoding detail lives in the regcmd ISA Reference; userspace safety layers in Robustness & Defensive Guards. This page is for "you must do X first or it breaks, and nothing tells you" traps.

Multi-domain (>4 GiB) dma-buf import — three independent traps

The >4 GiB sliding window keeps weights resident across several 32-bit IOMMU domains (n_domains, dom_activate per submit — see NPU IOVA 4 GiB cap). Weights beyond domain 0 are imported zero-copy (dma-heap ALLOC → PRIME_FD_TO_HANDLE → MEM_CREATE) rather than native bcreate. Three separate constraints bite on this path — all root-caused 2026-07-06. Reproducer: tools/mc_import_probe.c.

⚠️ Validation trap first: verify both C[0] and C[last] (and ideally a full CPU-ref compare). The corruption here frequently leaves C[0] (first output column) correct while wrecking C[last]. A probe that checks only C[0] gives a false pass — this misled the original diagnosis (tools/chain_import_probe.c checked only C[0] and "passed" while output was actually corrupt).

Quirk 1 — imported weight into a fresh non-0 IOMMU domain corrupts (the subtle one)

  • Symptom: an imported (dma-buf) weight in a non-0 domain produces non-deterministic wrong results — dropped K-partials, e.g. C[last]=14336 then 16896 then 13312 across runs, expected 18944. rc=0, no error, no wedge. Bad enough occasionally to wild-address → a 60 s NPU job timeout (job abort, ret: -22).
  • NOT a multi-core bug (the tempting conclusion): forcing single-core still corrupts (core 0 alone: C[0]=18432 wrong). The per-core C[0]-ok / C[last]-wrong pattern is just non-deterministic noise. Native weights on the same non-0 domain are always correct.
  • Root cause: the rknpu driver lazily initialises a domain's IOVA allocator / page table on its first buffer. If that first buffer is an imported dma-buf (SG-list), the import's pages land on wrong/aliased IOVAs → the NPU reads garbage for some weight tiles. A native allocation exercises the domain-setup path correctly; imports after that are bit-exact.
  • Guard: ork_dom_prime(c, dom) — one tiny (64 KB) native bcreate per non-0 domain, before any import maps into it (no submit needed; alloc alone establishes the domain). dom_anchor[ORK_MAXDOM], once per domain (guarded), freed at teardown. Cost: 4 tiny allocs for a 7B, independent of chunking. Called at the top of ork_mm_load_i8_import.
  • Upstream: looks like a genuine kernel first-buffer-domain-setup ordering bug (native-first works, import-first corrupts) — worth reporting / fixing in rknpu_iommu_dma_map_sg / the domain lazy-init; until then the anchor stays. Tracked as Optimization Roadmap Tier 10c.

Quirk 2 — too many imported mappings in one PC-chained submit → errno=110

  • Symptom: a K-split submit that PC-chains many programs (e.g. ffn_down K=18944 → task_number=19), each referencing a separately-imported dma-buf mapping, is rejected: RKNPU_SUBMIT errno=110 (rc=-1, C=0). The CDMA chain-walker can't traverse that many foreign SG-list mappings. Native bcreate chains of the same length are fine; ~6 imported mappings works, ~19 faults.
  • Guard: size-bounded chunked consolidation — pack a weight's tiles into a handful of imported chunks (tiles are chunk.dma+off views), so a chain walks few mappings. ork_mm_load_i8_import own_bufs[], ORK_IMPORT_CHUNK_MB (default 16 → down-proj ≈ 5 chunks).

Quirk 3 — a single large import fails at IOMMU-map

  • Symptom: importing a single dma-buf ≳ 16–64 MB fails at PRIME_FD_TO_HANDLE: Cannot allocate memory (16 MB OK, 64 MB ENOMEM; full 68 MB → WEIGHT NULL, an earlier variant hung the load R-state 571 s). The 32-bit domain's IOVA allocator can't find a contiguous range that large (fragmentation) — so it likely shrinks as a domain fills.
  • Guard: the same chunking (Quirk 2) caps chunk size from above. 16 MB is a genuine sweet spot bounded on both sides — per-tile (many mappings) trips Quirk 2; 64 MB trips Quirk 3. Cannot consolidate a weight into one buffer.

Net: with the anchor (Q1) + 16 MB chunks (Q2+Q3), the zero-copy multi-domain import path is correct at full multi-core speed — validated mc_import_probe 5/5 bit-exact and 7B forced-import multi-core pp128 = 58.73 t/s, no wedge.

Activation→matmul submit ordering: the pipeline runs one way (SDP is downstream of the MAC)

  • Symptom: a submit that runs a standalone SDP activation op (act-LUT: exp/rsqrt/silu — ork_npu_exp_i16 etc.) immediately followed by a separate matmul submit on the same context reliably RKNPU_SUBMIT errno=110 (ETIMEDOUT) and trips the self-healing reset — per row-batch, which then discards the (correct) NPU activation output and falls back to CPU. The reverse order — matmul then activation — is fine (the on-NPU norm path does reduce-matmul → rsqrt-LUT with no stall; the fused-output SiLU rides the matmul's own SDP stage). Found building the composed softmax (ork_npu_softmax_f16: exp on NPU, then a Σ reduce-matmul).
  • Root cause (open-source cross-reference): the RK3588 NPU is NVDLA-derived, and NVDLA's datapath is a fixed unidirectional pipeline — Convolution/MAC → SDP → PDP → CDP — with no backward path (NVDLA hwarch). Activation (SDP) sits downstream of the MAC. So "matmul→activation" is with the dataflow (and is exactly NVDLA fused mode, where the conv core streams into SDP via FIFOs); "activation→matmul" runs against it — SDP output would have to re-enter the conv core, which the hardware has no path for. Chaining a standalone-SDP submit into a fresh matmul submit leaves the pipeline mis-staged and the matmul never drains → timeout.
  • Did the open stack note it / work around it? No — because it never issues this sequence. NVDLA/Rocket run activations as the fused output stage of a convolution (fused mode), or as a standalone memory-to-memory op in "independent mode" which "begins and ends with the assigned block performing memory-to-memory operations" (NVDLA hwarch) — i.e. a standalone SDP op lands its result in DRAM and is fully decoupled; it is never PC-chained backward into a matmul. The mainline accel/rocket driver + its Mesa userspace is convolution-centric and only resets the core at init (rocket_core_reset in rocket_core.c, called from rocket_core_init — not per-job), so it has no per-submit state-scrub either; it simply doesn't produce the offending order. There is thus no documented workaround to borrow — the architecture just doesn't do activation→matmul, and neither should we.
  • Guard (ork-driver), interim: don't chain a standalone SDP activation into a matmul. Two safe shapes: (1) keep the activation on the NPU and do the following reduction/matmul on CPU — what ork_npu_softmax_f16 does (exp on NPU, Σ+normalize on CPU); (2) when a reduction must precede the activation, order it matmul→activation (the norm/rsqrt path).
  • The REAL fix — PC-chain in ONE submit (what the vendor does), CONFIRMED by capture (2026-07-09): a full softmax capture (nn.Softmax, rknn-toolkit2) decodes to a 9-task PC-chained pipeline in a single submit — row-max → broadcast → x−max → load-exp-LUT → exp (SDP) → Σ = exp·ones (MATMUL) → 1/Σ → exp·(1/Σ) → output. Crucially the Σ is a matmul that runs right after the exp SDP op — the exact activation→matmul order that ETIMEDOUTs as two separate submits. It works because every task's 0101:0x0010 points to the next task's regcmd, so all 9 run within one submit — the pipeline never crosses a submit boundary / mode-switch re-warm. So the asymmetry is a per-submit-boundary effect, not a hardware "no backward path" law: SDP→MAC is fine when PC-chained in a single submit. ork_npu_softmax_f16 stalled only because it issued exp and the reduce as two submits. The forward reduction is NOT window-capped — validated on-NPU for reduction length N up to 1024 (refutes the CDP LRN n≤9 concern), and the geometry maps exactly onto attention scores [M,N] softmaxed over N (0x4058=N−1, 0x4030=min(M,128)−1). Full register map + dataflow: FWD_SOFTMAX_RE_WIP.md / regcmd ISA Reference.
  • CONFIRMED on silicon (2026-07-10): ork_npu_replay_softmax_f16 runs the full vendor 9-task forward softmax on-NPU, bit-exact (uniform input → all outputs = 1/N). The correct mechanism is KERNEL-SEQUENCED per-task submits (one task_number=1 program per task, in dataflow order), NOT hardware PC-chaining — see the accel/rocket driver (one task per PC program, kernel re-arms on IRQ). Hardware-chaining attempts (concatenated regcmd + 0101:0x0010 + task_number=9) stalled at the chained LUT load; per-task submits (intermediates persisting in resident buffers, no reset between so the exp LUT survives) sail through. This is the empirical proof that the activation→matmul asymmetry is a per-submit-boundary/reset effect, resolved by keeping the tasks in one un-reset sequence. Interim ork_npu_softmax_f16 (exp on NPU + Σ on CPU) still applies until the geometry-parameterized replay is wired as a primitive.
  • CORRECTION — the standalone-op cross-submit case has a concrete software root cause (2026-07-12, tools/mode_probe.c): the "standalone SDP activation → separate matmul submit ETIMEDOUTs" case is not the NVDLA "no backward path" dataflow law — it is a stale shared task descriptor left by the LUT-activation op. Proof from a 2-op transition sweep: ewmul_i16/ewmul_f16/add_f16 → matmul is SAFE, and exp_i16/silu_i16 → multi-core matmul is SAFE — both impossible if SDP genuinely could not feed the MAC. Only LUT activation (exp/silu) → single-core matmul wedged. See the dedicated quirk LUT-activation op poisons the shared task descriptor below and Exp-2026-07-12. (The single-submit PC-chain / kernel-sequenced replay above remains the right structure for real graphs like softmax for other reasons — LUT-load ping-pong, resident intermediates — but per-op interleave is now safe too.)

LUT-activation op poisons the shared c->task descriptor → wedges the next single-core matmul

  • Symptom: a standalone on-NPU LUT activation (ork_npu_exp_i16 / ork_npu_silu_i16 / gelu / rsqrt — all via act_lut_i16 → ork_npu_probe_silu_std_i16) followed — not necessarily immediately — by a single-core matmul (small N, e.g. the SSD CumBA bmm at N=16) RKNPU_SUBMIT errno=110 (task counter: 0x0 → soft reset). dmesg shows a real job timeout. The activation itself succeeds; the matmul wedges. ACT_RESET, ORK_NO_PINGPONG, and a mode-invalidate do NOT clear it (measured — tools/mode_probe.c E2/E3/E4) because it is not a hardware-pipeline state.
  • This is why test_ssd_chunk_npu mode 2 wedged on CumBA "mid-sequence" while mode 1 (same exp→bmm) survived: mode 1's matmuls (N≥64) all run multi-core; mode 2 adds the CumBA (N=16), the only single-core matmul, which reads the poisoned descriptor.
  • Root cause: the LUT op memsets the SHARED context task descriptor c->task to its own SDP program (regcfg_amount 1097 for the LUT-load, then 69 for the activation; enable_mask=0x18) and left it that way — it did not restore it. The single-core matmul path (run() → submit1) does not rebuild c->task; it relies on the init value (regcfg_amount=108, enable_mask=0xd, regcmd_addr=c->regcmd.dma, set once in ork_npu_init) persisting. So a later single-core matmul submits its 108-word matmul regcmd under the stale 69-reg / 0x18 SDP descriptor → the NPU reads a truncated/mis-typed program, dispatches no task (task counter 0x0) → timeout. It is a poisoned software descriptor, not HW state, so a hardware ACT_RESET cannot fix it.
  • Why the trap is delayed & intermittent: multi-core matmuls use per-core c->mtk[] (rebuilt every call) and are immune; the plain 2-input SDP ops (ewmul/add) save+restore regcfg_amount/enable_mask around their submit. So c->task can stay poisoned across several ops (a multi-core matmul, then an ewmul that faithfully preserves the poisoned 69) until the next single-core matmul finally trips on it — which reads as "contextual/mid-sequence", not a clean A→B pair.

Transition matrix (RK3588, two ops as SEPARATE submits, warm context — tools/mode_probe.c):

first op (A) second op (B) before fix
matmul (any dtype) any SDP (exp/silu/ewmul/add) SAFE (matmul never poisons c->task)
ewmul_i16 / ewmul_f16 / add_f16 (non-LUT SDP) matmul (single- or multi-core) SAFE (these save/restore c->task)
exp_i16 / silu_i16 (LUT activation) multi-core matmul (N≳64) SAFE (rebuilds per-core c->mtk)
exp_i16 / silu_i16 (LUT activation) single-core matmul (small N, e.g. N=16) WEDGE errno=110
  • The int8 twin was missed for 6 weeks — fixed 2026-08-22. The save/restore below landed on the i16 LUT path only (npu/i16/act.c). ork_i8_npu_probe_silu_std (npu/i8/probe_sdp.c, behind every ork_i8_npu_silu/gelu/rsqrt/exp) kept poisoning c->task and returning. Symptom on the int8 side was a 2500 ms stall rather than an obvious wedge — mode_probe scored it ok(0,2500) = "safe", so the matrix never flagged it — but the stall reset-stormed and eventually Oops'd the kernel in rknpu_gem_object_destroy (Exp-2026-08-22). Fixed by the same save/restore (whole-struct, since the activation setup memsets the descriptor), on every exit including the LUT-load wedge path. GELU_I8 -> MM_F16 went 2500 ms -> 0.1 ms, and the full mode_probe sweep went from hanging-and-killing-the-board to 56 pairs in 48 s, 0 WEDGE. Measured proof it is not a mode-transition problem (fix ladder on the same pair): none = 2500 ms, invalidate = 5106 ms, reset = 5107 ms — a HW ACT_RESET makes it WORSE, because the cold re-warm is a 2-pass and each pass stalls. Do not reach for XSPEC/setdt here.
  • Guard / fix (ork_npu_probe_silu_std_i16, and since 2026-08-22 ork_i8_npu_probe_silu_std): the LUT op saves c->task's regcfg_amount/enable_mask/regcmd_addr on entry and restores them on every exit — exactly the save/restore the ewmul/add ops already do. After the fix every transition above is SAFE with no interposed reset; test_ssd_chunk_npu mode 2 goes from a hard wedge to PASS (full-G1 rel-L2 5.8e-3, full-G2 5.9e-3). The minimal safe-sequencing rule for composing heterogeneous per-op NPU sequences is therefore just "every op must leave the shared c->task as it found it"; a fused single-submit chain is not required for correctness (it remains preferable for the submit-floor + LUT-ping-pong reasons above).
  • Diagnostic knobs added alongside: ORK_MM_TIMEOUT (matmul-path submit timeout, mirrors ORK_EW_TIMEOUT; default 60000 — set low so a wedge-search self-terminates in ~2 s); ork_npu_mode_invalidate() / ork_npu_mode_reset() (the interposed-fix hooks the probe used to prove ACT_RESET does not help). See Exp-2026-07-12.

Ping-pong (RKNPU_JOB_PINGPONG) races a chained task's un-banked side effect (e.g. a LUT-load)

  • Symptom: a multi-task hardware-chained submit (task_number>1, the HW walks 0101:0x0010→next) that contains a LUT-load task stalls non-deterministically at the LUT-load→consumer transition — the kernel job times out (errno=110, raw_status 0x80000000/0xc0000000 = a busy/hang state, NOT a maskable interrupt bit) with the task counter flaky (e.g. 4↔5 across identical runs). Fewer chained tasks can stall earlier than more — the giveaway that it's a race, not a deterministic wall.
  • Root cause: submit flags bit RKNPU_JOB_PINGPONG (1<<2, i.e. the 0x4 in the common 0x5) enables double-buffered register banks (NVDLA PRODUCER/CONSUMER groups): the HW loads task N+1's register config into the alternate bank while task N runs, then swaps banks the instant task N's register config is done. That's a throughput win for register-config-only tasks (uniform matmuls) — but a LUT-load streams ~1000 entries into DPU LUT SRAM, a long-latency side effect NOT represented in the register bank. Ping-pong's fast swap advances to the consumer (e.g. the exp op) before the LUT SRAM has committed → the consumer reads a half-written LUT → hang/race.
  • Guard: for any chain containing a LUT-load (or any task with an un-banked side effect), submit with ping-pong OFF (flags = RKNPU_JOB_PC = 0x1, not 0x5). The HW then fully completes each task — including the SRAM commit — before advancing. This is exactly what the vendor does: the captured softmax chain uses flags=0x1, while pure int8-matmul chains (run_chain_i8, flags=0x5) safely keep ping-pong on because their tasks are register-config-only. Validated: the full 9-task forward-softmax graph runs as ONE submit, bit-exact 9/9, ~471µs (vs ~2.5ms per-task) once ping-pong is off (ork_npu_replay_softmax_f16). Full recipe (all required): contiguous single-buffer image + single-delta rebase (so the LUT-load writes into the regcmd region as the vendor lays it out) + 0101:0x0014 next-amount =(n+3)/2 + ping-pong off. See regcmd ISA Reference / FWD_SOFTMAX_RE_WIP.md.

★★ A commit within ~1 ms of an IOMMU domain switch may never dispatch (2026-08-26)

This is the root cause of what the section below calls the "int4 doorbell miss". That name is wrong — it named the first place the symptom was seen. The failure has nothing to do with int4, with the doorbell, or with NONBLOCK submission.

The rule

A job committed within ~1 ms of an iommu_attach_device() domain switch may fail to dispatch: accepted by the driver, PC completed-task counter stuck at 0x0, no interrupt, no IOMMU fault, MMU idle. Beyond ~1 ms after a switch, it does not happen.

Evidence (replicated across two unrelated code paths)

workload commits stalls <1 ms after switch >=1 ms after switch
1.5B-Q8 via blocking ork_dyn_colsplit 1155 86 86 / 159 = 54.1% 0 / 996
int4 probe via the async doorbell, ndom=2 12070 5 5 / 2997 = 0.2% 0 / 9073

Zero stalls outside the window across ~10,000 commits, on two paths that share nothing but the kernel submit code.

It is TIME, not commit count

Tempting to read this as "the first N commits after a switch are poisoned". The int4 probe separates the two: 9004 commits sat at position 0-2 after a switch and only 5 stalled — and all 5 were in the sub-1 ms subset, while 8993 early-but->=1 ms commits were clean. Position is a proxy; elapsed time is the operative variable.

Necessary, NOT sufficient

Inside the window the rate varies enormously by workload and shape — tn=2 is 75.7% in the 1.5B run and 0.2% in the int4 probe; within one run tn=4 is 90.9% while the larger tn=9 is 25.0%, so it is not monotonic in size either. Something else modulates it. The window is a necessary condition, and removing it removes every stall observed so far, but do not present it as the whole mechanism.

Three earlier conclusions this WITHDRAWS

Each looked solid and each was a confound with switch-proximity:

  1. "Domains >0 are stall-prone" (dom 2 at 42.9% vs dom 0 at 3.0%). Domain 2 is merely visited briefly, so most of its commits land just after a switch. Controlling for position: dom 0 = 0/180, dom 1 = 0/171, dom 2 = 0/20. The iova_reserve_iommu_regions asymmetry between domain 0 and the others is real but is NOT this bug.
  2. "Stalls follow back-to-back commits (short inter-commit gap)". The first jobs into a new domain arrive in a burst, so short gaps co-occur with post-switch position. Enforcing minimum spacing between all commits gave no dose-response and inverted (500 us -> 12 stalls, 1500 us -> 46).
  3. "ndom=2 triples the stall rate, therefore domain switching is expensive". Right observation, wrong reading: more switches means more exposure to the window.

What to do about it

dom_cooldown_us (rknpu, default 0) sleeps after a successful switch before the domain is released for use. It can usleep_range() rather than busy-wait because rknpu_iommu_switch_domain() is only reachable from process context, and it is charged once per SWITCH (~56 in a 1.5B run), not per commit.

Userspace already had the intuition and never quantified it: src/npu/core/domain.c:223 comments on a "first-submit-into-a-freshly-switched-domain hazard" and ork_dom_prime() exists for it.

How this was found (method, not luck)

Per-commit kernel profiling of BOTH populations. The stalled population alone was already known from userspace logs and was useless without the healthy base rate — every wrong turn above came from comparing a numerator against the wrong denominator. See "Measurement discipline" below.

int4 doorbell "miss" = the job is committed and the hardware NEVER STARTS (2026-08-26)

SUPERSEDED — see the section above. The name is a misnomer: the same failure occurs on plain BLOCKING submits with no doorbell and no int4 involved. The cause is committing within ~1 ms of an IOMMU domain switch. The observations below about the failure's signature still hold; the framing and the suspected causes do not.

Status: software path fully exonerated with controls; cause still open (hung sub-block or silicon). The practical target is CHEAP RECOVERY, not prevention — see the protocol at the end.

What the failure is

On a multi-domain int4 (W4A4) prefill, some doorbell rounds never complete. Cost is time = 23.01 s + 3.61 s x misses (R2 0.967) — ~23% of 27B prefill. Numerics are never wrong (PPL bit-identical 10.6780 on every run at every setting): recovery always produces the correct answer, so this is purely a throughput defect.

It is a DISPATCH failure, not a stall. A kernel watchdog sampling the PC completed-task counter at 1 kHz reports, on every miss, all three cores at once:

core 0 NO TASK PROGRESS for 64 samples (64000 us), tasks=0/832 done (counter 0x0), cm=0x1 dom=1, job age 114993 us

The submit returns rc=0, the job is committed, and the hardware never executes a single task. Control (essential — a dead register reads identically): at an 8 ms threshold a HEALTHY ~30 ms job does NOT report — 3 reports across ~900 core-jobs, not hundreds. The counter genuinely advances on good work.

The output pattern misleads. The surface shows a contiguous written PREFIX ending on a 64-element boundary. That is NOT a partial write: cores get CONTIGUOUS op ranges (lo = i*S/nc, hi = (i+1)*S/nc), so a core that never runs leaves its contiguous slice at the sentinel. Nothing was written and then stopped.

Everything that has been ELIMINATED (each with a control)

stage verdict evidence
queue / lost interrupts OK commit-done == stall count exactly; irq == done; nojob = 0
commit register writes OK full PC/INT block byte-identical healthy vs stalled
IOMMU mapping OK zero page faults / BUS_ERROR / not-attached across 14 stalls
task descriptor freshness OK monotonic stamp in rknpu_task.op_idx reads 737..6397 — always current
regcmd bytes in DRAM OK checksum cache vs post-invalidate DRAM: 0 mismatches; positive control DOES fire
timeout_clean killing live jobs REFUTED counts match, but the reaped job is 157 ms old = reaped during recovery
missed start trigger REFUTED re-pulsing PC_OP_EN on a stall: 4 kicks, 4 stalls, none recovered
ping-pong REFUTED A-B-A n=4000: 146 / 111 / 116 — OFF sits inside the ON bracket
uneven K-split geometry REFUTED even split fails at least as often as uneven, on BOTH paths
miss-window / detection tuning REFUTED rate flat across 300 ms / 614 ms / 2 s / 30 s windows
read visibility / cache REFUTED kernel FROM_DEVICE re-count recovers nothing

Register map for the block (vendor rknpu_ioctl.h): 0x00 VERSION, 0x08 PC_OP_EN, 0x10 PC_DATA_ADDR, 0x14 PC_DATA_AMOUNT, 0x20 INT_MASK, 0x24 INT_CLEAR, 0x28 INT_STATUS, 0x2c INT_RAW_STATUS, 0x30 PC_TASK_CONTROL, 0x34 PC_DMA_BASE_ADDR, 0x3c PC_TASK_STATUS, 0xf008 ENABLE_MASK. ENABLE_MASK = 0 and INT_STATUS = 0 are NORMAL — they read the same on jobs that run. Do not chase either. INT_RAW_STATUS = 0 too, so the hardware is not signalling internally.

What DOES move the rate (all condition-based, cause-agnostic)

lever effect
retire the prior job before switching domains (orki_dom_drain) 11.47% -> 1.99% (B-A-B, n=8000/12000)
programs per submit Sk=1 0.15%, Sk=2 0.38-0.62%, BCHAIN ~850 progs 2-3%
number of IOMMU domains ndom=2 roughly triples the rate vs ndom=1
K geometry (even vs uneven split) no effect — do not constrain alignment

The failing shapes are not special, they are simply the BIGGEST: the 27B ffn_down is the largest, longest int4 op in the graph, which is why 12/12 model misses landed there.

The drain barrier does NOT transfer end-to-end: 5.8x on the probe, a WASH on the real 27B (8 vs 7 misses). The probe switches domains every rep so its rate is race-dominated; the model is residual-dominated. Weight any probe result against a model run before believing it transfers.

The recovery is the cost, not the stall

A miss costs ~3.6 s: a ~614 ms guessed detection window, then a device-wide ACT_RESET, then a resubmit. rknpu_soft_reset() asserts ALL SIX DT reset lines and msleep(100)s, and sets a global soft_reseting flag that makes rknpu_job_next() bail for every core — so recovering one core resets the whole NPU and destroys the other two cores' warm state.

The six lines are PER-CORE: reset-names = "srst_a0","srst_a1","srst_a2","srst_h0","srst_h1","srst_h2" (AXI + AHB per core; index N and N+3). So a targeted per-core reset is possible and is the cheap-recovery lever. RKNPU: soft reset, num: 6 prints num_srsts, the COUNT of DT reset phandles — it is a constant, not a reset counter. Count log lines instead.

Safe protocol (what to do while the cause is unknown)

  1. Retire before switching domains (ORK_DOM_DRAIN, default on).
  2. Bound programs per submit; prefer fewer, smaller submits over one huge one.
  3. Batch work per domain to minimise switches.
  4. Recover per-core, not device-wide, and detect from the task counter rather than a guessed timeout.

Instrumentation and method notes for this board (learned the hard way, 2026-08-26)

Log capture — only two things work

  • netconsole WORKS and is the only thing that caught the hard-wedge panic. Persisted via /etc/systemd/system/netconsole.service (must be After=network-online.target; modules-load.d runs too early to bind eth0) + /usr/local/sbin/netconsole-up.sh. netpoll sends RAW frames so it needs a destination MAC — the gateway's MAC is the one that reaches the capture host here.
  • ⚠ console_loglevel defaults to 4, so only messages BELOW level 4 reach the console and hence netconsole. <4>/KERN_WARNING test messages are silently dropped; this cost an hour of wrongly concluding netconsole was broken until a <3>/KERN_ERR message arrived instantly. kernel.printk = 8 4 1 7 is now persisted in /etc/sysctl.d/. Absence of low-priority lines in an old capture is NOT evidence the event did not happen — RKNPU: soft reset is LOG_INFO (level 6) and was invisible at loglevel 4.
  • pstore/ramoops DOES NOT WORK on RK3588. Regions are reserved and re-registered every boot, but /sys/fs/pstore comes back empty even after a warm panic-reboot — the Rockchip restart re-initialises the DDR controller and wipes the region. Do not rely on it.
  • /var/log is a 50 MB tmpfs (DietPi RAMlog, AUTO_SETUP_LOGGING_INDEX=-1) and journald retains no prior boots (journalctl -b -1 -> "No journal boot entry found") despite Storage=persistent.
  • A durable fallback: a reader on /dev/kmsg writing /home/michael/ork-outbox/kern.log with fsync per message, on the NVMe, outside RAMlog.

Measurement discipline

Kernel log capture that actually survives: /var/lib/ork-logs/kmsg.log (systemd unit ork-kmsglog, dd if=/dev/kmsg ... oflag=dsync,append, on the NVMe rootfs). Two failure modes it avoids, both of which cost a dataset here:

  • dmesg alone is not enough — the kernel ring OVERFLOWS on a bad run and silently truncates, which makes a coverage denominator read as zero.
  • /var/log is tmpfs on this board (DietPi RAMlog, 50 MB), so anything written there is WIPED ON EVERY REBOOT and a verbose profiling run can fill it. An entire per-commit profile was lost to a reboot before it could be analysed.

The unit rotates the previous boot's capture to kmsg.log.prev on start, so the log from before a crash or reboot is still readable afterwards — which is the case that matters for wedges.

  • A count that matches is not a cause. THREE false root causes this session came from exactly that: lost interrupts, marginal K geometry, and timeout_clean killing live jobs. The last one had treap == stalls == commit-done — a perfect match — and was only caught because the KILL AGE was logged alongside (157 ms, not the 1.5 ms the theory needed). Always log a second independent quantity — an age, a sequence, a checksum — next to any count that "matches".
  • Always run the positive control in the same pass. "No mismatches" is worthless until the detector is shown to be able to fire; the regcmd checksum only became evidence once a deliberately corrupted word produced a mismatch.
  • Match the control to the subject. Comparing a 1-task ping-pong-off job against a 2-task ping-pong-on stall produced two convincing false signals (task_ctrl readback, status(pre) 0x7000 vs 0x5000). Both vanished with matched jobs.
  • Never sample a periodic workload on a modulo. The workload repeats a 4-job cycle (one real op plus three drain dummies); sampling every 500th completion aliased (500 %% 4 == 0) and captured ZERO matching jobs. Log the first N matches instead.
  • Sample size. At a ~2% rate, n=400 gives ~10 events (Poisson +-32%) — direction only. Two wrong conclusions came from n=60 runs. Use >=4000 for any rate comparison, and bracket A-B-A: the absolute rate DRIFTS between sessions (the same config measured 5.25% and ~11% on different runs).
  • Instrumentation perturbs. ORK_KMSG_SUBMIT (one extra syscall per submit) moved the stall rate 3.25% -> 6.00%. Use it to localise an event, never to quote a rate.
  • Verify the knob fired. A pp=1 A/B returned a clean "no effect" from code that never executed (dyn.c early-returns for int4). It was caught only because removing HW chaining failed to produce the expected slowdown. Print a one-time confirmation from inside the instrumented path.
  • Rebuild the right target. make i4_widek_stall_probe is a separate rule; a plain make leaves it stale, and a stale probe silently measures the old code.

⚠ Do not make NPU scratch buffers uncached

ORK_RC_UNCACHED=1 (dropping RKNPU_MEM_CACHEABLE) makes the probe die with SIGBUS: the kernel then maps the buffer as Device memory, and ARM64 faults on UNALIGNED ACCESS TO DEVICE MEMORY, which the regcmd memcpy/NEON writes rely on. Not a viable way to test coherency.

Other established quirks (cross-references)

  • int8 mode entry needs an NPU reset. The first int8 submit wedges unless the NPU is reset when switching into int8 (fp16 cold-starts stale and the warmup handles it; int8 does not). run() issues RKNPU_ACT_RESET on the fp16→int8 transition, then re-warms. Mixing dtypes on one context works but each mode switch triggers a one-time re-warm. See src/npu.c run() and AGENTS.md §5.
  • Never kill -9 an in-flight submit. SIGKILL mid-RKNPU_SUBMIT wedges the NPU/IOMMU (and can corrupt SPI on a hard power-cut), forcing a reset/power-cycle. Use SIGTERM + drain, and timeout on board NPU commands. See AGENTS.md §3.
  • Weight-DMA amortization / M-tile ceiling. Single-core int8 is weight-DMA-bound; the M-tile cap is the 0x1040 schedule limit mg_max*64 (bit-exact), not an R-1/CBUF-resident-rows bound. See AGENTS.md "Weight-DMA amortization" and regcmd ISA Reference.
  • Output zero-copy (ORK_ZC_OUT) is unsafe under concurrent multi-core. The per-core coherency bsyncs aren't serialized across cores. Off by default. See AGENTS.md and Optimization Roadmap Tier 7.
  • 32-bit IOMMU = ~4 GiB per domain. rk_iommu v2 "40-bit" is physical PTE reach, not IOVA width. See NPU IOVA 4 GiB cap.
  • Some int4-regcmd registers HARD-wedge the bus (kernel job-timeout can't recover; needs a power-cycle). Found while fuzzing the int4 multi-M scheduler (Exp-2026-07-07). Distinct from a soft/compute hang — a bad value mis-programs a DMA/partition so the bus locks and even sub.timeout won't return; only a power cycle recovers. Confirmed hazards: 0x1001/0x4010 and 0x1001/0x4100 (PPU/DMA output config — arbitrary overrides slip the DMA), and the 0x201/0x11xx CNA scratch region (e.g. 0x1104, though that one proved state-dependent — inert in isolation, wedged only cumulatively). These need follow-up RE (why do they bus-lock? are they the IOMMU/output-stride/CBUF-partition controls?). The fuzzer (tools/i4_multim_fuzz.c) blacklists them (i4_fuzz_blacklist.txt) and skips the 0x11xx range by default. See Exp-2026-07-07-INT4-Multi-M-Schedule-RE.
  • Two submit-stress hard-wedge circumstances (2026-07-15), for follow-up RE. (1) Concurrent multi-context submits — two ork contexts' graph_computes on two threads with no serialization → IOMMU level 0 translation fault + soft reset (the single queue can't service two IOMMU contexts at once). (2) int4 large-M submit storm — the int4 1-submit-per-row × per-group path at M=512 (~8192 submits) → hard wedge (no ping, power-cycle only); isolated safe at M≤16 (≤256 submits). Distinct axes (concurrency vs count), both single-queue+IOMMU fragility. Guards + root-cause questions in Exp-2026-07-15-NPU-Hard-Wedge-Circumstances.
  • fp16 M-scheduler has a valid Kp WINDOW [128, 2048), not just an upper bound. sched=1 programs the 0x1040 K-reduction schedule from scale=Kp/256; for small Kp it extrapolates too HIGH (K=64→0x1040=188, K=32→190) and miscomputes — constant-garbage output (rc=0, no wedge). K≥128 is fine (K=128→184), and the non-pow2 K=96 already took sched=0 (no 0x1040 override) so it worked, which masked the bug. This blocked fp16 attention at head_dim=32/64 (the common QKᵀ/AV dim) — surfaced by the strided ork_bmm_fp16 (i8 was exact at the same dims). Fix: gate fp16 sched=1 to Kp>=128 && Kp<2048; outside the window, sched=0 (general path) is correct. Validated test_bmm K-sweep 32…256. (The upper bound is the pre-existing ">8 rows garbage at Kp≥2048" gate.) See src/npu.c run()/synth().
    • Confirmed + extended 2026-07-12 (Exp-2026-07-12-Matmul-Shape-Boundaries): the single load-bearing register is 0x201:0x1040 — a silicon register-bisect (ork_f16_fuzz_add(0x201,0x1040,·)) shows overriding only 0x1040 0xBC→0xB1 (template default) at M=64,K=64 flips zero-output (rel-L2 1.0) → bit-exact (3.8e-8). The bug re-appeared in ork_bmm_fp16_fused (the on-NPU SSD-scan chain) and the raw ork_*_probe_*mm helpers, which all still hardcoded sched=1 — same gate now applied there (sched=(K&(K-1))==0&&K>=128&&K<2048), validated bit-exact across the scan shapes (K∈{32,64,96,128}, N∈{64,128}). int8 twin: synth_i8's formula scales by K/512 (vs fp16 K/256), so its invalid band is wider — K<256 (K∈{32,64,96,128} all zero; K≥256 correct); production int8 is safe because the run path only sets sched=1 at K∈{512,1024}, but any raw-synth_i8 chain must use sched=0 below K=256. Boundary is N- and M-independent.
  • int4 native multi-M is CBUF-limited batch mode. Reg 0x201/0x107c = K/16 enables a resident-activation batch: rows/submit = min(M/2, 16384/K) (activation budget) AND the per-core N-slice weight must fit Ncore*K <= 131072 int4 (weight budget) — exceed either and far rows/blocks silently don't compute. int8 avoids this by streaming activations (its 0x1040 schedule, which corrupts int4). int4 streaming is unconfirmed (open RE). See the RE Roadmap "int4 STREAMING mode".
  • Concurrent per-core NONBLOCK doorbell submits intermittently DROP (~1/2000–4000; accepted-but-never-dispatched). (2026-07-19) When an int8 matmul rides the multi-core NONBLOCK doorbell — ork_dyn_begin_mc (S>1 streams: QKV, gate/up) or ork_dyn_begin_colsplit (M=1 decode, N-columns split across cores) — a round occasionally is accepted (every per-core RKNPU_SUBMIT returns rc=0) yet the HW never executes it: no output sentinel ever lands, no task-completion IRQ fires. Timing-, gap-, ping-pong- and interleave-independent (all measured — not a retirement/EBUSY/mid-round-reset race; those were disproven dead-ends). Rate rises with submit density. Strongly consistent with a lost doorbell / IRQ-rearm race in the stock rknpu driver's concurrent per-core dispatch (kernel-side; unprovable from userspace because RKNPU_GET_TOTAL_RW_AMOUNT and task int_status read 0-always on this kernel — the output sentinel is the ONLY reliable completion signal; do not trust those counters).
    • Deceptive colsplit signature: a dropped core's N-column slice retains the previous iteration's output (fixed test inputs → still correct) except its re-seeded sentinel column, so a whole-slice drop looks like a single wrong interior column. It's the per-core sentinel, not an interior-column compute bug — a validator that checks only C[0]/C[last] can miss it entirely.
    • Why it hid so long: introduced by the blocking→NONBLOCK-doorbell spine migration (prior blocking submits waited in-kernel — no dispatch race to lose). Below single-run detection: make test runs the one guard (test_stream_interleave) ~420 rounds → passes ~80 % of the time → looked reliable; only surfaced under dozens of back-to-back runs. CI can't run the NPU tests, so no automated repetition accumulated.
    • Fix (ork_dyn_end + mc_recover_resubmit, src/npu.c): detect via a bounded sentinel timeout (300 ms for mc int8 — real decode/stream lands ≪ that), then RKNPU_ACT_RESET + ~1 ms settle (a resubmit into a not-yet-quiesced NPU re-drops → sticky/correlated miss) + re-clean + re-seed + resubmit, up to 6 retries. Both ORK_MC_ROUND and colsplit stash their round context in the chain handle so ork_dyn_end can replay it. Fires only on a real drop (≈0 common-case cost). test_stream_interleave went 25/40 → 40/40; tools/mc_miss_repro (stop-on-first-miss stress loop) survives 12000 iters bit-exact with recoveries logged (ORK_MC_DIAG=1). submit1_db (single-core nc==1) is not yet covered — no op was observed dropping on it (M=1 int8 decode routes through colsplit, not submit1_db); add the same pattern if one ever does.
    • Lessons: (1) a self-validating test run once per gate cannot catch a sub-0.1 %/round drop — use a stop-on-first-miss stress loop; (2) verify a "counter" actually advances on this kernel before trusting it — dma_rw=0-always was a trap that produced a false "never dispatched" reading and, when used to gate recovery, a reset storm that hard-wedged the board; (3) moving a path blocking→NONBLOCK trades latency for exposure to this dispatch race, so the recover is mandatory on every doorbell path.

NEVER free/reuse a DMA buffer a dropped (never-completed) NPU job still references → delayed kernel crash (2026-08-05)

  • Symptom (delayed, deadly): after an accepted-but-never-dispatched doorbell drop (the quirk directly above), if recovery abandons the job's output buffer — frees it, or de-escalates to a different code path that reuses a fresh buffer instead of completing the dropped job — the box looks fine for ~1–2 minutes, then the kernel dies: BUG: Bad page cache in process kworker … pfn:… (a run of contiguous PFNs) → page dumped because: still mapped when deleted on the shmem eviction path (filemap_unaccount_folio ← truncate_inode_folio ← shmem_undo_range ← shmem_evict_inode ← evict ← iput ← __dentry_kill ← dput) → Unable to handle kernel NULL pointer dereference → Oops in rknpu_gem_sync_ioctl (i.e. our own bsync) → soft-reset cascade → hard wedge (power-cycle only). The delay makes it easy to mis-blame the next thing that runs.
  • Root cause: an NPU job that was accepted but never dispatched (nonblock doorbell drop; hw_elapse=0, int_status=0, iommu=1 — the NPU is healthy, it just never ran) still holds references to its output buffer's shmem/DMA-buf pages. Those refs are released only when the job COMPLETES — either it finally dispatches and lands, or the kernel's ~60 s watchdog reaps it. RKNPU_ACT_RESET alone does NOT release them (verified: a reset-then-abandon still crashed at teardown). So destroying that buffer (context free / bdestroy, or the process exiting) while the phantom job still references it = pages "still mapped when deleted" = the crash.
  • Rule: never free or hand back a DMA buffer while any in-flight OR dropped NPU job may still reference it. On a doorbell-drop recovery you MUST get the job accounted first: (a) complete it on the original buffers — resubmit → poll to landing, as int8 mc_recover_resubmit does; the colsplit/mc per-core buffers are context-level and reused, so the completing job reaps the phantom refs — or (b) let the kernel watchdog reap it before freeing. Do NOT de-escalate to a path that leaves the colsplit buffers behind with the drop still outstanding.
  • Why int8 nonblock is safe and the fp16-colsplit sentinel path was not: int8's recover always lands/completes on the reused buffers → refs released → clean teardown. The fp16 sentinel recovery detected the drop fast (800 ms nonblock poll) but then de-escalated to nc=1, abandoning the colsplit buffers with the dropped slice-1 job never completed → the delayed crash. Fix direction: complete-the-job (resubmit to landing), never abandon.
  • Diagnostic that nailed it: ork_npu_dump_state / ork_dyn_dump fired at detection, before recovery, with fflush + live-streamed over ssh (not captured in a shell $(…), which buffers until exit and is lost when the box hangs). Shows the drop is always the 2nd K-slice (Kslice=1), hw_elapse=0. Fire your post-mortem before you touch the NPU, and stream it off-box — a hard wedge loses anything still in a local buffer. See COLSPLIT_MULTIPREC_WIP.md "ROOT CAUSE FOUND via auto-dump (2026-08-05)".

Second signature of the same rule: GEM destroy (not sync) — sg_free_table Oops (2026-08-22)

The 2026-08-05 crash surfaced in rknpu_gem_sync_ioctl. The identical root cause also lands in the destroy path, which is worth knowing because the trace looks unrelated:

Unable to handle kernel paging request at virtual address 0000002710000008
Internal error: Oops [#1] SMP   Comm: mode_probe   pc: virt_to_folio  lr: kfree
  virt_to_folio <- sg_kfree <- __sg_free_table <- sg_free_table
  <- rknpu_gem_object_destroy <- rknpu_gem_free_object <- drm_gem_object_free
  <- drm_gem_handle_delete <- rknpu_gem_destroy_ioctl <- drm_ioctl

Tell-tale: x20 = 000009c400000007 — 0x9c4 = 2500 = the ORK_MM_TIMEOUT that run was started with, packed against a 7. Scalar data where the sgl pointer belongs: the table was freed and its memory reused, then walked again at destroy.

Trigger (mode_probe, needs a SEQUENCE — neither pair alone does it): GELU_I8 -> MM_F16 completes but burns the full timeout and emits ~14 back-to-back RKNPU: soft reset; the next transition's buffer realloc then issues the GEM destroy into that state.

A post-reset SETTLE DELAY DOES NOT HELP — measured, don't retry it. A settle window inserted before MEM_DESTROY was tried at 2 ms and at 100 ms: identical Oops both times, same trace, and the 100 ms run took the board down. That is exactly what the rule above predicts — the refs are released when the job completes (or the ~60 s watchdog reaps it), not after any amount of wall time. The mitigation was reverted rather than shipped, because a delay that looks protective and isn't is worse than nothing.

So the rule stands unchanged and applies to destroy as well as sync: complete the job on the original buffers, or let the watchdog reap it — never just wait, and never abandon.

Note this raises the value of the ork_dyn_progress frontier fix (2026-08-22, dyn_ctl.c): before it, a hole at a non-final task read as success, so the recover loop never fired and the dropped job was silently abandoned — precisely the pattern this rule forbids. With recover reachable again, those jobs now complete on their original buffers and release their refs.

Aftermath if you hit it: Oops [#1] only taints the kernel. The box can survive minutes and then die on the next activity — once on a purely read-only command — so the death gets mis-blamed on whatever ran next. Full write-up + kernel-log capture recipe: Exp-2026-08-22 mode_probe Kernel Oops.

fp16 3-core colsplit drop = a concurrent CROSS-BUFFER weight-fetch WILD → prevent with CONTIG, do NOT try to recover (2026-08-05)

This resolves the fp16-colsplit drop above. The fix is PREVENTION, not recovery — and the two failure classes must not be conflated:

  • The fp16 drop is NOT the benign int8 dispatch-miss. It is a wild DMA: when the 3 cores loop their K-slices, the NPU prefetches the next slice's weight base (Bb[ks+1]) while Bb[ks]'s fp16 weight-DMA is still draining → two distinct dma-buf fds on the CDMA at once → mis-addressed fetch. fp16's 2-byte weights double the fetch bytes; int8 tolerates the same pattern, fp16 wilds. Signature: the drop is ALWAYS the boundary slice (Kslice=1 at Sk=2, never Kslice=0) — textbook cross-boundary prefetch. Same-buffer concurrency (3 cores reading regions of ONE buffer) is benign; only cross-buffer (distinct fds) wilds. A CPU lockstep barrier cannot fix it — it can't gate a hardware prefetch.
  • A real fp16 drop is UNRECOVERABLE from userspace (exhaustively tried 2026-08-05: in-place resubmit → sticky re-drop + uninterruptible D-state hang; close(fd) → refcount underflow; timeout_clean reap + fast-detect + reset-first → kernel panic). The wild write corrupts kernel RAM at the drop instant, microseconds before any 800 ms / 24 ms detect can react. And any BLOCKING submit that can't dispatch into the post-drop NPU (e.g. an nc=1 fallback) enters the kernel's 60→120→180 s continue_wait → rknpu_job_abort, which does rknpu_iommu_domain_put() → msleep(100) → rknpu_soft_reset() — tears down the IOMMU 100 ms BEFORE halting DMA. The device-global iommu_domain_refcount gates domain switching (switch only if refcount==0); a dropped job's double-cleanup over-decrements it → a later switch iommu_detach_device under a live matmul → wild write → slab corruption → Oops/panic in unrelated processes (percpu_ref_put, rb_insert_color, shmem eviction), delayed 1–2 min. rknpu_job_timeout_clean is safe only because it resets BEFORE teardown; abort is not.
  • THE FIX = CONTIG (w->Bbc, default-on for Sn==1): concatenate all K-slices into ONE weight buffer → there is no dma-buf boundary, so the "next base" is just the next address in the same fd/mapping → the prefetch stays in one mapping → no wild → no drop → recovery never fires. 3 cores read regions of the one buffer concurrently (benign). Validated: 1000-iter @256×3584×3584 = 0 drops / 0 wrong / 0 panic (the per-slice path dropped on iter 0), full shape suite BIT-EXACT (Sk=1 and Sk=2), make test all-PASS, 2.6× preserved (nc3/nc1=0.38×).
  • The one-line blind spot that hid CONTIG: is_valid_dma_addr (src/npu.c) validated the weight base against w->Bb[]/w->Bf[]/scratch but not w->Bbc → a valid Bbc.dma+offset base was FALSE-flagged "wild/unallocated (outside all valid buffers)" → validate_regcmd refused → CONTIG fell back to the wedging per-slice path. CONTIG's address math was correct the whole time. Lesson: when a "wild address" is inside an expected buffer's range, suspect the validator's known-buffer list before the address arithmetic.
  • Taxonomy to keep straight: benign accepted-but-never-dispatched miss (no DMA fired — int8/int4 doorbell drops) is recoverable (mc_recover_resubmit, reset+reap+resubmit-to-landing, proven 12000 iters); wild-DMA drop (fp16 cross-buffer) is not — it must be prevented. The recovery protocol (adaptive sentinel detect, reap_stuck clean timeout_clean reap, ork_dummy_probe int8 health-gate, reset-first) is retained as the safety net for the benign class and any future Sn>1 fp16 shape where CONTIG's single-buffer assumption doesn't hold. Gate/knobs: ORK_F16_COLSPLIT (path), ORK_F16_NO_CONTIG (A/B opt-out), ORK_F16_TRACE=<file> (tailable step log), ORK_CAMPAIGN (per-iter bit-exact).

M-fold weight layout & capture-replay gotchas (int8 matmul, task #39, 2026-07-29)

RE'ing rkllm's fast int8 "M-fold" prefill matmul surfaced several traps. The full fold data layout (bit-exact validated on RK3588 via tools/re/validate_layout.c):

  • A (input) nc16 = (k/16)*(M*16)+m*16+(k%16) — NC1HWC2 C2=16, M folded into WIDTH.
  • W (weight) woff = ((n/32)*(K/32)+(k/32))*1024+(n%32)*32+(k%32) — the documented RK3588 int8 rknn_matmul NATIVE layout (N/32,K/32,32,32) (from rknn_matmul_api.h; per-chip (subN,subK) table in [Reverse-Engineering-Roadmap] "Porting the fold to other Rockchip NPUs"). Do not board-RE the weight pack — read the header.
  • C (output) c4 = (n/4)*(M*4)+m*4+(n%4) — NC1HWC2 C2=4, int32.

Gotchas:

  • ORK_REPLAY_RESET is a WEDGE TRIGGER, not a stabilizer. It issues RKNPU_ACT_RESET before the submit; on the mfold replay path that reset hard-wedges the board (needs a power-cycle). WITHOUT it the replay runs clean and repeatable (validated bit-exact twice). Do NOT set it for mfold replay.

  • The captured output mm_C.bin is STALE — don't trust it as a bit-exact reference. rkllm reuses a tiny pool of ~5 scratch C buffers (e.g. 0xb17a000), and the rknpu_dump.so shim dumps C after the whole run, so it holds a LATER submit's result, not the captured tile's. The captured (A,W,C) is NOT a consistent triple. Validate with a SELF-CONSISTENT test instead (ork's own A/W → replay → compare CPU ref), not against the dumped C. (The int32 DST output IS raw signed A·W: no requant / zero-point — confirmed by one-hot injection.)

  • Injection-ramp aliasing when decoding a layout probe. A recoverable ramp A[i]=(i%251)-125 aliases a value to i ∈ {v+125, v+125+251, …}; e.g. val −120 → i ∈ {5, 256, …}. Pick the i that is a valid nc16 index (256 = nc16(0,32) → k=32), not the smallest. A naïve inverse mis-reads the k it maps to.

  • rkllm runs the fold as a task_number=21 weight-resident chain, 3-core round-robin — the per-tile speed win is weight residency across the chain, NOT a single fold tile (a lone width-8 tile does not amortize the weight DMA and loses ~4× to ork's normal M-scheduler at M=228: ~17 ms vs 4.6 ms).

  • 0x40c0 (DPU_SURFACE_ADD) is an elem-size CONFIG, NOT an output IOVA — preserve it, do not patch it. 0x20 int8 / 0x80 int32 / 0x400 M-fold (=128*M@M=8). Patching it as an address points the WDMA at an unmapped low IOVA and hard-wedges. (An earlier note in Exp-2026-07-28 called it an IOVA — corrected there.)

  • M>8 mfold runs as a chain of M<=8 tasks, and it is bit-exact (2026-07-30). ork_npu_mfold_chain_cap chains P copies of the captured bit-exact width-8 tile (mm_regcmd_m8.txt) in one task_number=P submit, shared resident weight, PC-chained at 216-219: 0-mismatch at M=8/16/32/64, no wedge. Required regcmd prep: preserve 0x40c0 (above) and neutralize the captured chain descriptor keeping the reg-ids (zero only the value halves of the 0x0010/0x0014 block-0x101 entries, like ork_npu_replay_i8) — zeroing whole words 216-219 drops the 0x0014 terminator and even one task hangs. Perf caveat: each tile re-streams the full weight (0x1040=0xb1) -> linear, ~7x slower than ork-normal at M=228; the 1.6x needs weight residency (load-once/reuse) across tiles.

Clone this wiki locally