fold int8 matmul: weight layout is documented in RKNN header + per-chip porting table + capture-replay gotchas
RE: matmul small-K shape boundaries — the 0x1040 zero-output Root-cause + silicon register-bisect of the fp16/int8 bare-synth() zero-output below the prefill K-band: the 0x201:0x1040 K-reduction schedule formula overshoots the template default 0xB1 as K drops (0xBC@K64), selecting an invalid reduction atom. fp16 breaks K<96, int8 K<256; N-/M-independent. Fixed ork_bmm_fp16_fused (the on-NPU SSD-scan chain) to use the run-path sched predicate. - new Exp-2026-07-12-Matmul-Shape-Boundaries (full matrix + fix + reproduce) - regcmd-ISA-Reference: small-K validity subsection under the M-tile scheduler - NPU-Quirks: extend the fp16 M-scheduler-window quirk (register bisect, int8 twin, fused-chain) - Experiment-Log + Reverse-Engineering-Roadmap: index + formula caveat
Forward softmax on-NPU (single-submit 9-task chain, ~471us) + RKNPU chaining rules incl. ping-pong/SRAM-commit timing: new Exp-2026-07-10 synthesis page + NPU-Quirks ping-pong quirk + RE-roadmap SOLVED
RE-roadmap: hardware-chain mechanism cracked (rknpu_job_commit_pc: 0101:0x0010 next-addr + 0x0014 next-amount=(n+3)/2); softmax chain 0x0014 bug fixed; LUT-in-chain + reorder = characterized hard wall
Exp-2026-07-10 Fused Output Activation (no-crossing chain): C=fn(A.B) in one submit — exp/silu/gelu/rsqrt bit-accurate; RE-roadmap M4.8
RE-roadmap: chaining audit RESOLVED — rknpu uses hardware PC-chaining (not rocket's kernel-rearm); ork-driver run_chain is correct. Softmax cost: per-task=9x floor, single-submit needs LUT-in-chain fix
NPU-Quirks: activation->matmul asymmetry (PC-chain resolution) + fp16 small-K sched window; RE-roadmap: chaining-vs-kernel-sequencing audit item (accel/rocket insight)
Exp 2026-07-10: on-NPU batched-GEMM (attention/GDN) + norm primitives New Experiment Log entry + RE-Roadmap M4.7. ork_bmm batched dynamic GEMM + on-NPU rmsnorm/l2norm (reduction=matmul contraction, rsqrt=fp16 fused-LUT twin of the M4.5 standalone rsqrt) + l2norm_f32, all gated + validated in make test. Perf-neutral end-to-end (corroborates M4.6: fusion is the need, not standalone offload). RE: DPU output-stage registers named vs mainline accel/rocket header; NVDLA CDP LRN window HW-capped n in {3,5,7,9} -> full-channel norm must ride a matmul, not the fixed-function LRN.
roadmap: rectify D_BANK — no bank-count register on RK3588 (conv capture); int4 8-row cap is a HW limit gated by 0x107c, not a reallocatable-bank artifact The two earlier NVDLA-derived sections framed D_BANK as an untapped ~8-15x lever (reallocate ~15 idle banks). The later conv .rknn capture (same day) refuted this: weight swept to 4.7MB found no discrete bank-count register; CBUF residency is dim-derived (0x1030/0x107c/0x1010/0x1040). rows*K<=16384 (8 rows @ K2048) is gated by 0x107c and no register combination escapes it. Add SUPERSEDED banners pointing to regcmd-ISA-Reference + the real levers (dense int4 K/32 encoding; multi-task STRATEGY C chaining).
roadmap: NPU SRAM — Armbian stock vendor kernel==DietPi (RKNPU_SRAM off + DTB unwired); no stock kernel enables it; custom kernel rebuild (CONFIG_ROCKCHIP_RKNPU_SRAM=y) is the only path; 'Armbian has it' = patched image not stock
roadmap: full NPU-SRAM enablement recipe (Rock5B/DietPi test) — needs DT (rockchip,sram + carve, DONE+verified) AND CONFIG_ROCKCHIP_RKNPU_SRAM=y kernel rebuild (DietPi lacks it; the real 'Armbian has it' gate); rknpu built-in so kernel rebuild required
roadmap: sparse/compression CLOSED on RK3588 (toolkit rejects sparse_infer, compress_weight/model_pruning no-op = identical size on 60%-zeroed weights); vendor-authoritative. Garbage-fuzz+validate-with-compressed strategy noted as sound-but-uncompletable here -> RK3576 task (does it accept sparse_infer?)
roadmap: CORRECT SRAM entry — RK3588 NPU SRAM is PRESENT (~1MB system SRAM) but DISABLED (npu node lacks rockchip,sram phandle); enablement = DT phandle + reboot; then RKNPU_MEM_TRY_ALLOC_SRAM testable for weight-tile-in-SRAM bandwidth (decode wall). Earlier 'no HW' was wrong (user corrected)
roadmap: both NVDLA 'unexploited levers' probed on RK3588, both NOT viable — NPU SRAM=0 bytes (second-memory-interface has no HW), sparse has no RKNN API/regcmd exposure; RK3588 populates a subset of NVDLA techniques
roadmap: TODO RK3576 int4 datapath comparison (blocked on hw access) — stream2 cap test + w4a16 create-probe + rk3576.c param validation, to settle generation-wide-cap vs RK3576-completed-it
roadmap: int4 single-submit streaming DEFINITIVELY closed — broadened stream2 sweep (0x107c x 0x1044 x 0x1040, bypasses librknnrt) tops out at EXACTLY the 16384/K activation cap, no combo exceeds it; RK3588 int4 diverges from NVDLA's unified-streaming model (hard HW cap, not a driver gap); 96-task multi-task IS int4's native streaming
roadmap: NVDLA documented optimization techniques -> ork-driver primitives mapping (int4-96-task = NVDLA batching; UNEXPLOITED high-value levers = sparse weight compression for decode-bandwidth + second memory interface/weight prefetch); kernel is dtype-uniform (no int4 stub) so int4/int8 asymmetry is librknnrt/HW not kernel
roadmap: STRATEGY C wired into ork_mm_run_i4 (ORK_I4_INCR) — bit-exact; wins 1.6-2.2x at N=64 but single-core C loses to multi-core A at N>64; multi-core STRATEGY C is the follow-up
roadmap: int4 incremental-task batch (STRATEGY C) WORKING — vendor task_number=N pattern, bit-exact, 7-28x vs separate / 1.3-2.7x vs STRATEGY-A (uncapped, weight loaded once); 2 full priming tasks required (else errno 110)
RE roadmap: mode-support matrix COMPLETE — fp16 mirrors int8 (0x1068 row-offset only, no batch toggle); batch is precision-selective in the present matmul regcmd (int4 batch, int8/fp16 streaming); int8/fp16 batch levers are init-time/out-of-regcmd
RE roadmap: NVDLA batch uses DATA-bank partition (not all banks); int4 uses ~1 of 16 banks -> 8-row cap is a D_BANK allocation artifact, ~15 banks idle; D_BANK (init-time/CDMA) is the big untapped lever
RE roadmap: prior-art survey — int4 modes unexplored publicly (mtx512 does fp16/int8 only), our RE is SotA there; CBUF 16x64KB banks corroborate our weight-budget=64KB=1-bank finding
RE roadmap: i8fuzz answers it — RKNPU batch is precision-SELECTIVE in the mm regcmd (int4 batch-only, int8 stream-only, no D_BATCH_NUMBER in int8 regcmd); 0x1068=row-offset not batch
RE roadmap: RKNPU per-precision mode-support matrix (streaming NOT precision-agnostic: int4 lacks it; batch axis being mapped via i8fuzz + fp16 probe)
RE roadmap: int8 true-batch fuzz DESIGN + why it's low-priority (0x50xx block absent from int8 regcmd, no perf value; the definitive test is a librknnrt capture, not a board fuzz)
RE roadmap: NVDLA confirms int8 true-batch is supported (precision-agnostic, nv_small=int8+batch); RK uses streaming for perf; the anomaly is RK int4 lacking a streaming schedule
RE roadmap: rigorous batch-vs-streaming terminology (activation residency) — int4=true batch, int8=weight-stationary streaming (not batch); int8-true-batch untested
RE roadmap: close int4-streaming (comprehensive negative, both A-layouts + vendor evidence); round-robin single-core batch is the actionable lever
RE roadmap: CORRECTION — int8 does batch (its M-scheduler); int4's 0x405c recipe is an int4 output quirk, not the batch register. Real axis = activation residency; int4-streaming still OPEN (weight-layout test pending)
RE roadmap: close int8-batch/int4-stream — modes are dtype-locked (int8 won't batch, int4 won't stream); not viable via config