Correct the registers the int4 stride narrative keeps pointing at, and record the precision enum
0x405c is DPU_WDMA_SIZE_1 (height/width), not a stride in any precision, yet
four pages call it "the PPU stride" and the roadmap sent a probe at it. The
activation-side lever is 0x107c CNA_DMA_CON1.LINE_STRIDE, which the ISA
Reference already named while flagging it open; burst length belongs to
0x1078. Both entries corrected so the index stops disagreeing with itself.
Adds the precision enum derived from our own three captures -- 0=int8,
1=int16, 2=fp16, 4=int32 out, 6=int4 -- on CNA 0x100c and DPU 0x4010. No
external source carries it: mainline is autogenerated and enum-free, Mesa
writes zero and never touches a precision field, NVDLA has no INT4 row at all.
It establishes that a dedicated int4 processing mode exists on this silicon,
and separates that from the still-open question of whether the MAC array is
correspondingly wide.
Retracts Exp-2026-07-08's "no further register lever exists" -- Wb reaches 320
at K=2048 for 2.09x bit-exact, H reaches 16-64, and the 8-rows-by-1-bank lock
is not a lock. Flags two claims on Exp-2026-06-19: it records the input as
stride-1 while being cited as the origin of the stride-2 finding, and its
parity-with-INT8 prediction predates any throughput measurement.
wiki: resolve the int8-vs-fp16 divergence — resident vs streamed weight
The split walks data-ward as M grows, so WEIGHT_BANK shrinks. int8 keeps its weight
CBUF-resident and dies once WBNK no longer fits it; fp16 streams its weight and is
unaffected, running to the full 11 data banks. Closed form for int8 at N=64:
M_max = 64*floor(12/w - 1), w = K/512 — exact at K=512/1024/1536/2048.
K=1536 was the discriminating measurement (3-bank weight): bit-exact at M=192 with
WBNK=3, every element wrong at M=193 where the clamp drops WBNK to 1. The dtypes
coincide only when the weight is exactly one bank, which is why int8 K=512 and fp16
K=256 both reach 704 and looked like agreement.
Also records the refuted hypothesis (host A-staging layout does NOT depend on
DATA_BANK — int8 K=1024 M=256 is bit-exact at both DBNK=8 and a forced DBNK=11), a
falsifiable N-dependence prediction, and the one residual gap at K=4096.
wiki: 0x1040 is a CBUF bank split — fp16 M envelope measured, two miscomputes fixed
New entry Exp-2026-08-20-fp16-M-Envelope-and-the-CBUF-Bank-Split, plus the Experiment
Log, the Tier 14 A roadmap row (now ✅), and a correction to the regcmd ISA Reference.
The "M-tile schedule" expression at 0x1040 is a LINEAR FIT to a bank split:
v = (WEIGHT_BANK<<4)|DATA_BANK with DBNK+WBNK == 12, one bank moved weights->data per
64-row group, 32 KB banks, and the 0x1b floor is exactly WBNK >= 1 (forcing v below it
hangs the submit). So the fp16 M ceiling is banks x 32 KB / row-bytes: sched=0 -> 16384/K,
sched=1 -> 180224/K. Measured 704@K256, 352@K512, 176@K1024 — the truncated mg_max*64
under-reported by 10-37%. K=128 is broken by the encoding (half-bank steps are
unrepresentable) and its envelope is non-monotonic, so only the 256-row prefix is usable.
Two live miscomputes documented and fixed in ork-driver 6584e2a: the fp16 stream's
doorbell guard was the int8 BYTE budget (2x too loose at 2 B/elem), and ork_f16_mm_run
miscomputed at K=128 for any M in [257,1472].
int8 was checked and is unchanged — measured-correct at the truncated mg_max*64. Both
dtypes carry the same 11-bank bound symbolically, yet fp16 attains the fraction and int8
floors to whole groups; that is recorded as the OPEN question with the evidence that
rules out the clamp, capacity, and register-programming explanations.
Also records two methodology rules (never bisect a non-monotonic ceiling; never derive an
fp16 tile from an int8 constant) and the board-ops cause of the hard wedge this session.
M-fold schedule RE + CBUF-bank-capacity mechanism; rocket register-name corrections
New Exp-2026-07-28 page: rkllm's M-folded int8 schedule (M into CNA WIDTH, NC1HWC2 features),
synth_i8_mfold reproduces the capture bit-for-bit, and the root cause of the partial single-submit
reduction = CBUF DATA_BANK capacity (0x1040=CNA_CBUF_CON0, 0x1044=CNA_CBUF_CON1). Decodes the
0x40c0=DPU_SURFACE_ADD absolute-output-IOVA hard-wedge. ISA reference: rocket_registers.h
cross-reference + name corrections (0x1040/0x1044/0x107c/0x1088/0x40c0). Verdict: mfold is
chain-required; banked. Experiment-Log index updated.
regcmd ISA: add parallel-doorbell completion-poll gotcha (submit fast, ork_dyn_end miss-timeout was the stall)
ISA ref: N%32 decode gap fixed for SDP-path (ork_npu_mm_perchan_f16 pads N; decode N=1 works ~59us)
ISA ref: SDP-path beats diagonal at every N (10-30x) — correct the small-N claim
ISA ref: int32 lane-recombination RESOLVED (exact low/high split, x65536 recombine not viable in pure NPU)
ISA ref: over-fetch makes int32-input SDP self-complete (terminal-counter balance); DPU still int16-lane-splits int32
ISA ref: int32->int16 downcast dead by both feeders (matmul 0-write structural, SDP 50% under-fetch); all 4 repair theories refuted
ISA Reference: narrow-output matmul (CNA->DPU requant-WDMA) does not self-complete — int32-only; int16 shortcut terminal
ISA Reference: fp16-out stage (FP32TOFP16_EN), CNA LINE_STRIDE strided-read, EW/ERDMA data-mode + notch fields, RC_SDPKW
ISA Reference: SDP EW/ERDMA + DPU output-stage registers (EW_ALU_ALGO MAX/MIN/SUM, ERDMA per-channel/element, BS/BN bypass, precision enum, chained-SDP dtype rule)
RE: matmul small-K shape boundaries — the 0x1040 zero-output
Root-cause + silicon register-bisect of the fp16/int8 bare-synth() zero-output
below the prefill K-band: the 0x201:0x1040 K-reduction schedule formula overshoots
the template default 0xB1 as K drops (0xBC@K64), selecting an invalid reduction
atom. fp16 breaks K<96, int8 K<256; N-/M-independent. Fixed ork_bmm_fp16_fused
(the on-NPU SSD-scan chain) to use the run-path sched predicate.
- new Exp-2026-07-12-Matmul-Shape-Boundaries (full matrix + fix + reproduce)
- regcmd-ISA-Reference: small-K validity subsection under the M-tile scheduler
- NPU-Quirks: extend the fp16 M-scheduler-window quirk (register bisect, int8 twin, fused-chain)
- Experiment-Log + Reverse-Engineering-Roadmap: index + formula caveat
ISA ref: close capture campaign — RUBIK ops (pixelshuffle/deconv) route through conv engine, block 0x0401 never emitted on RK3588; confirmed 7-engine set
ISA ref: expanded op battery (24 ops) — activation LUT-vs-clamp split, reductions on CNA+PDP, data-movement on PPU+CDMA, block 0x0401 unobserved
ISA ref: op->engine-block map (single-op .rknn capture) — new PDP pooling blocks 0x4001(0x60xx)/0x8001(0x70xx); activations=SDP LUT (1001); elementwise=PPU; layernorm=multi-engine chain; block IDs = (2^k<<8)|1
ISA ref: CORRECT decode_reg-filter errors — matmul is normal CNA+DPU+PPU (108 regs, K in 0201), NOT bankless/1001-only; D_BANK has no explicit RK3588 register but CBUF residency = 0201 size regs (0x1030 weight, 0x107c entries-per-slice, 0x1010/0x1040 M-tile sched); int4 batch gated by 0x107c not a bank knob
ISA ref: conv path (domain-2001 0x50xx CDMA block) mapped; D_BANK is IMPLICIT on RK3588 — no programmable bank register (conv_big 4.7MB weight forces no bank reg, only dim-linear regs + DMA); CBUF partition is driver/HW-derived, unlike NVDLA's explicit D_BANK
ISA ref: correct w4a16/mixed types — RK3588 rejects ALL mixed-precision matmul (-5 unsupported); symmetric MAC only (fp16/int8/int4), enum is cross-platform boilerplate
ISA ref: document domain-1001 0x40xx matmul engine (vendor RKNN matmul-API capture) — full register map, 0x4010 mode reg, in-task-batch (int8/fp16) vs multi-task-submit (int4, task_number=96), refutes batch_probe task_number>1 rejection, runnable type matrix, D_BANK not in matmul path
ISA ref: 0x1068 = int8 output row-offset (physrow=m+v), found by i8fuzz — a new reg mapping, NOT a batch stride
ISA ref: identify 0x107c = D_ENTRY_PER_SLICE (fp16 K/8, int8/int4 K/16) + packing note
Update wiki: fused Q4_K->int8 dequant result + reframe negative verbiage
Status updates (current as of 2026-06-24):
- Tier 5 MoE: prefill walked 0.28 -> 0.97 (chaining) -> 2.90 (NEON repack)
-> 3.13 t/s (fused Q4_K->int8 dequant, ork-driver b520ee7 / ggml-ork
b5bc5ce8b). Deficit ~80x -> ~7.2x under CPU. New wall = the NEON
f32->int8 quant+tile (77%); the Q4_K->f32 dequant is now folded in (0ms).
- Roadmap Tier 5 table + 5b/5c verdicts + progress log; MoE Expert Offload
page status/verdict + a 2026-06-24 follow-up section; Experiment-Log index.
- Recorded the ORK_MOE_NPU=0-still-enables-NPU-MoE benchmark gotcha.
Reframed discouraging "dead-end" verbiage across the wiki to "not yet
viable" / "ruled out" (per the don't-discourage-future-efforts policy):
MoE, int4/Hadamard, zero-copy, attention-on-NPU, per-op backend, and the
ISA-Reference "Ruled out" section. Hardware-impossible facts (same-precision
MAC, KV caches can't cross backends) kept as factual constraints.
docs: document CORE_CTRL 0x000f wait bit and zero-overhead hardware chaining
wiki: document the rknpu_job_next crash + that multi-core is done (1.69x/11.0)
Performance levers: multi-core is implemented (not a future risky TODO) with both crash lessons
(all-subcore-populate; per-call RESET is load-bearing job-queue recovery — RESET-once + concurrent
short-timeout submit storm hangs rknpu_job_next with IRQs disabled, power-cycle required). Prefill
reframed as the CPU-ops lever; decode single-submit marked done-in-multi-core / parked-for-prefill.
ISA reference state-gotchas updated to match.
Correct single-submit K ceiling: K=10752 exact (raw length, not 256 tiles)
The 256-K-tile / int8-reaches-16384 hypothesis was REFUTED by tools/ksubmit_probe (a K-sweep
that validates each single submit vs CPU). Findings on RK3588:
- ceiling = K=10752 exactly (10752 OK, 10784 wedges) = 21x512 = 336 fp16 / 168 int8 K-tiles
- dtype-independent (fp16: 8192 OK, 11008 wedge — same raw band)
- N-independent (K=10752 OK at N=512/2048/8192) — purely a contraction-length cap
- near but under cbuf_elems/3 (32768/3); exact mechanism unconfirmed (RK3576 cross-check would
tell whether it scales with cbuf)
So rkllm can single-submit K up to 10752 (both dtypes); beyond, even the closed runtime must
K-split. Updated Performance + regcmd-ISA-Reference; the wrong values are gone.
Performance/ISA: decode single-submit experiment, 256-K-tile ceiling, IOVA blocker
- Decode single-submit (M=1 full-K per N-slice) measured at 1.23x (2.34->2.87 tok/s int8), but
a 2.5x submit cut bought only 1.23x -> per-submit overhead is small; decode overhead is mostly
the fp32 quant round-trip + per-submit DMA syncs, not submit count. Re-ranked decode levers.
- The single-op K ceiling is a HARDWARE 256-K-tile cap (0x1044): fp16 K<=8192, int8 ~K<=16384
(denser packing). Answers 'does rkllm do K>8192 single-submit?': fp16 no (closed runtime must
K-split too), int8/w8a8 yes to ~16384 (covers all LLM K-dims). Corrected the ISA reference,
which wrongly said 'any K in one submit'.
- The dual-layout that enabled the fast path doubles NPU memory and exhausts the rknpu IOMMU
IOVA range (fp16 28-layer: 'failed to allocate IOVA: -12'); reverted. Single-layout
approaches (stride-register slicing) documented as the path forward.
wiki: int8 cold-start wedge fixed with RKNPU_ACT_RESET on int8 entry (no kernel timeout spam)
wiki: populate ork-driver wiki — RE roadmap, feasibility log, regcmd ISA reference, recording-notes convention, Home index
Migrated the regcmd reverse-engineering record from the oRKLLM wiki and consolidated the
scattered RE scratch (register meanings, tile layouts, limits, state gotchas, methodology,
dead-ends) into a single regcmd-ISA-Reference page. Home indexes them; Recording-Investigation-
Notes sets the convention for future findings.