roadmap Tier 16: granularity measured at 38% — half the gap, not the ceiling
G=32 GPTQ cuts excess quantisation error 38% (+4.12 -> +2.56, PPL 15.9011 -> 14.3428) for
+24% disk. Largest single quality lever measured, and confirms item 1.
But Q4_0 is still 2.1x better on excess, so the tier's original 'this is the ceiling'
framing was over-attributed and is corrected. GPTQ and granularity overlap heavily (each
alone -37/-40%, together -45%), and G=32 RTN is WORSE than per-channel GPTQ -- a fine grid
does not make GPTQ redundant.
Remaining half is probably activation width (Q4_0 is effectively W4A8), supported by the
earlier W/A parity finding, NOT by a new measurement: grouped+int8-activations is
structurally unsupported -- the grouped branch hardcodes 4-bit activation quant and
ork_i4_mm_run_grouped is int4-only, so the arm builds and then refuses at run. New item 2
is the grouped int8 run path, now the critical path.
roadmap: Tier 16 — per-group scales as the W4A4 quality ceiling (top quality priority)
External CPU baseline shows our W4A4 introduces ~3.4x the quantisation error of plain
Q4_0 (excess over bf16: +4.12 vs +1.22), and our best tier is not even pure 4-bit. Cause
is scale granularity: one scale per channel (K=1024..3584) vs Q4_0's one per 32 weights.
Every quality lever landed this session optimises within that grid and cannot reach past
it, so this outranks them. Obstacle is the grouped path's 13.2x/283x overhead -- a
dispatch problem with mitigations, not a ceiling. Work items sequenced cheapest-first,
starting with pricing the quality win before optimising its cost.
Exp-2026-08-23: windowed GPTQ, the stub GGUF, and Tier 15
Windowed GPTQ: why the Hessian is a bounded reduction rather than a cache, the three
implementation attempts that each silently produced a one-shot pack, and the measurement
(windowed 17.4465 vs RTN 26.2782 vs an unattainable single-pass 15.9011). Records that
ORK_GPTQ_SEQ=0 is a no-op, so the sequential-vs-one-shot question is open and the
'sequential is strictly better' claim is currently contradicted here.
Stub GGUF: a packed run was loading the model twice (40.8 GiB against 31 GiB at 27B) and
failing as an RKNPU reset storm rather than an OOM, which is what made it expensive to
find. 27B sidecar is 2.56 GiB on disk vs 28.6 GB, PPL identical.
Tier 15 on the roadmap: load the model from the .orkpack and retire the source GGUF.
wiki: SDP + int4 envelopes — 21 guards measured, and the probe list they imply
New entry Exp-2026-08-20-SDP-and-Int4-Envelopes, plus Experiment Log and the Tier 14
roadmap follow-up (now closed).
int8 SDP 8176 (exact), fp16 SDP 8188 (contiguous max — it frays: 8189 BAD, 8190 OK,
8191 BAD), int4 H = CEIL(16384/K) capped 64. The SDP 8192 was RK_DPU_DATA_CUBE_WIDTH's
13-bit field read as a capability and was admitting shapes that FAULT the NPU.
The recurring error is ladder resolution: the int4 sweep used only powers of two, every
one of which divides 16384, so floor and ceil coincided. Same error hid the fp16 352
ceiling behind a 320/384 probe.
Records the derived probe list (Wb=(131072/K)&~63 is the same class and unprobed,
discarding 25% of the N-tile at K=768), what was checked and found FINE so nobody
re-probes it, the disproved shared-budget model, and the live fp16-floors/int4-ceils
inconsistency on the same nominal 16384 budget.
wiki: correct the SDP guard count 26 -> 21
The 26 came from a grep whose globs overlapped (src/npu/*/*.c already covered
src/npu/i16/act.c), double-counting 5 lines. The true total is 21: 8 int16, 8 int8
SDP, 5 fp16 SDP. Substance unchanged.
wiki: int16 SDP ceiling measured at 8176; Tier 14 A #2 re-scoped to the fork
The int16 activation guard accepted M<=8192 while the hardware stops at 8176 —
M=8184 returns errno=110 (submit timeout) + a self-healing reset, so the guard was
admitting a fault-generating shape. 8192 was never measured: it is
RK_DPU_DATA_CUBE_WIDTH's 13-bit field read as a capability, the same provenance as
the fp16 bound fixed earlier today. Row limit, not size limit (8176 bit-exact at
N=64 and N=128).
Tier 14 A #2 itself does not reproduce: ork_i16_npu_exp is bit-exact at all 37
probed points including M=4096 and M=6144, the values the item flagged. The
ORK_ATTN_* knobs it cites do not exist in ork-driver — they are fork-side, so that
NRMSE came from the fork's attention composition. Re-filed against the fork, with
the real bug recorded as fixed and the 18 unmeasured f16/i8 SDP sites left as an
explicit follow-up.
wiki: 0x1040 is a CBUF bank split — fp16 M envelope measured, two miscomputes fixed
New entry Exp-2026-08-20-fp16-M-Envelope-and-the-CBUF-Bank-Split, plus the Experiment
Log, the Tier 14 A roadmap row (now ✅), and a correction to the regcmd ISA Reference.
The "M-tile schedule" expression at 0x1040 is a LINEAR FIT to a bank split:
v = (WEIGHT_BANK<<4)|DATA_BANK with DBNK+WBNK == 12, one bank moved weights->data per
64-row group, 32 KB banks, and the 0x1b floor is exactly WBNK >= 1 (forcing v below it
hangs the submit). So the fp16 M ceiling is banks x 32 KB / row-bytes: sched=0 -> 16384/K,
sched=1 -> 180224/K. Measured 704@K256, 352@K512, 176@K1024 — the truncated mg_max*64
under-reported by 10-37%. K=128 is broken by the encoding (half-bank steps are
unrepresentable) and its envelope is non-monotonic, so only the 256-row prefix is usable.
Two live miscomputes documented and fixed in ork-driver 6584e2a: the fp16 stream's
doorbell guard was the int8 BYTE budget (2x too loose at 2 B/elem), and ork_f16_mm_run
miscomputed at K=128 for any M in [257,1472].
int8 was checked and is unchanged — measured-correct at the truncated mg_max*64. Both
dtypes carry the same 11-bank bound symbolically, yet fp16 attains the fraction and int8
floors to whole groups; that is recorded as the OPEN question with the evidence that
rules out the clamp, capacity, and register-programming explanations.
Also records two methodology rules (never bisect a non-monotonic ceiling; never derive an
fp16 tile from an int8 constant) and the board-ops cause of the hard wedge this session.
roadmap: SUPERSEDED banners on Tier 5 + the MoE-on-NPU saturation thread
Both predate the 2026-08-19 measurements and would mislead a reader today. Tier 5's 'not yet viable, but
closing' did not resolve by the NPU catching up — experts belong on the CPU (NF4 + batched NEON, 1.59x native
prefill), and experts-on-NPU is closed by a hard fork (int8-inflate ~35GB >> 4GiB IOVA vs int4-native ~54x
slower kernel). The saturation thread's M0 is an ISOLATED-primitive win (3-10x per GEMM) that INVERTS when
integrated (5.33 vs 27-28 t/s, ~4M tiny programs x ~18us dispatch floor), and M3-skew's hot-cache verdict is
contradicted by NPU decode losing at any hit-rate. Banners rather than rewrites: the original reasoning is
instructive precisely because the integrated result inverted the isolated one.
roadmap: Tier 14 A — the declared-vs-defined lint landed as check-registry check 5
Records the four discrimination rules the lint needed to reach zero false positives
(prototype = no brace AND ends in ';'; name from before the FIRST paren; header
'static' is a definition; skip typedefs/preprocessor) and that it is verified on both
BSD and GNU sed/grep.
roadmap: Tier 14 A — ork_gptq_i4 declared without an implementation, now stubbed (-ENOSYS)
Public symbol declared in include/ork_npu.h with no definition anywhere in main;
found by a declared-vs-defined sweep of the headers. Records the stub, why it is
COBJ-not-CORE, where the real quantizer is parked (origin/wip/moe-saga-2026-08-18),
and the task #56 AutoGPTQ cross-check that unblocks it. Notes the sweep is worth
wiring into make check-registry so a future dangling declaration fails the build.
roadmap: scope Tier 14 to driver items; cross-link the fork + oRKLLM roadmaps
The backend (ggml-ork supports_op / ggml-cpu profiler / res=-3) and model-prep (quantize recipe, orkpack
derivation) items belong to llama.cpp-rockchip and oRKLLM respectively. Keeps this page driver-scoped and
points at the owning roadmaps so nothing is orphaned.
roadmap: Tier 14 — punch-list carried into the ork-driver refactor (2026-08-19)
Records the shipped MoE prefill win (1.59x native, knob-free) and the items left OPEN at the refactor
boundary, with the evidence behind each: the fp16 stream/chain M-envelope that silently miscomputed for two
callers months apart, the ork_npu_exp_i16 row-count envelope, dom_dirty being int4-only, the GDN verdict
being invisible from main (0 grep hits — a fresh investigation nearly rebuilt ork_gdn_scan_f32), the
res=-3 pp512 blocker, and the ORK_ALLOW_Q4_SRC speed probe for the Q4_K ssm_* projections (31% of
CPU-backend prefill).
Roadmap: Tier 13 — weight-stationary matmul dataflow (modeled + measured → BREAK-EVEN)
Can keeping the matmul weight CBUF-resident + reusing it (WEIGHT_REUSE) beat the weight-DMA-bound
M-fold? Investigated board-free (ws_model.c) then measured on RK3588. WEIGHT_REUSE is real but
vendor-used only for conv (weight fits CBUF); an LLM matmul weight is ~76x the 57KB CBUF, so reuse
needs weight-tiling + K-accumulate → ~100x more tiles than the fold. ORK_TRACE proved run_chain_i8 is
a single-ioctl task_number=S HW chain; chain_marginal_probe measured per_task = 4.38us floor +
weight_DMA@10.8GB/s (slope==DRAM BW ⇒ the rest is re-stream, removable by reuse). At the 4.4us floor:
WS-chip x0.98-1.03 (dead heat), @4.0us x1.07-1.13; WS-dram loses x0.78. Break-even — plausible modest
win at scale iff a lean chain submit shaves the floor AND two unproven HW mechanisms hold (cross-task
WEIGHT_REUSE for a matmul weight + on-chip int32 accumulator retention). Includes progress-log entry.
Optimization-Roadmap: add Tier 12 — heterogeneous CPU+NPU doorbell dispatcher
12a concurrent chain dispatch (run_chains_rr) done; 12b context-length crossover
measured; 12c static gate placeholder; 12d the dispatcher proper (route to the
most-efficient available resource, CPU+NPU overlapped); 12e NPU decode attention
enabled via host-split (correct for arbitrary scores); 12f resident-K/V payoff
proven (1.9-7.3x) + per-key append API built & validated bit-identical to repack.
doc: dynamic-submit API + doorbell (Exp-2026-07-16); correct Tier-11 'not re-read' claim
New page documents the NONBLOCK submission layer (ork_dyn_*/queue/ork_pc_*): the task_number
ceiling (halt=fewer OK, loop/extend=more not viable), the doorbell, multi-core NONBLOCK stream
(2.27x/1.38-2.3x), and the precompiled-program cache (6.03x). Experiment-Log entry added.
Corrects Tier-11's over-broad 'DRAM patches are never re-read / snapshotted' via strikethrough +
footnote [^dyn16]: a 0x0014=0 halt-patch on a forward chain IS honored (validated); the true bound
on redirect/extend is the immutable task_number (Wall 3), not 'patches unread'. Conclusion
(hold/redirect not viable) unchanged; mechanism corrected. Progress+halt doorbell shipped.
Tier 11: CPU-work-in-NPU-shadow overlap MEASURED FREE (hidden~100%, no BW contention) + router-is-already-CPU (ggml-ork supports_op); zero-time-router is an M>1/speculative lever
Tier 11: DRAM-doorbell completion BUILT + MEASURED N-op pipeline wall (2.17x micro-op -> 1.00x heavy, bit-correct)
roadmap: Tier 11 — MEASURED efficiency math (blocking ~130us/op overhead -> nonblock ~5us, driver QUEUES; micro-op ~6-19x, heavy ~5%) + decode spin-not-sleep lead
roadmap: Tier 11 CORRECTION — RKNPU_JOB_NONBLOCK (0x2) exists+WORKS (TESTED: ~5us async return, goldens pass); async spinlock is a PURE-USERSPACE build, NO reflash (obsoletes IRQ-hook + fence-flip). + efficiency math
roadmap: Tier 11 — async data-plane spinlock (spin-not-sleep) is the BEST doorbell variant; stock rknpu exposes no async (SUBMIT blocks, no WAIT action, FENCE compiled out, io_uring N/A) -> minimal enabler = CONFIG_ROCKCHIP_RKNPU_FENCE=y + userspace busy-poll, but rknpu built-in => still a (clean config) reflash
roadmap: Tier 11 — regime analysis (when the kernel IRQ-chain pays off: micro-op + CPU-interleaved, NOT heavy-layer or serialization-bound decode; go-candidate = MoE/streaming)
roadmap: Tier 11 — add kernel-callback recon (rknpu built-in => full reflash; source off-board; threaded-IRQ design saves only the transition not the requant; probes on feat/npu-doorbell)
roadmap: Tier 11 Wall 2 TESTED — live mid-submit patch not honored (job snapshotted at submit); userspace doorbell dead on 2 walls, craft build = kernel between-task callback
roadmap: Tier 11 Wall 3 TESTED — self-loop 0x0010 completes (no free-run); craft build targets the kernel between-task callback variant
roadmap: add Tier 11 NPU Doorbell / persistent-kernel graph-cut (for-the-craft, not an efficiency lever; profiled)
Roadmap: mode-transition layer ✅ Phases 1+2 landed on main (v0.7.0) + two Phase-2 follow-ups (fp16→chain keep-warm needs a real consumer; unconfirmed consecutive chain_i4 watch item)
Mode-transition layer (ork_npu_enter): Phase 1 landed
Add Exp-2026-07-14 page documenting the ork_npu_enter + XSPEC consolidation
(motivation, design, the 12 profiles / 16 sites, byte-identical validation, and
the Phase-2 catalogue of drifted sites). Flip the roadmap section to Phase-1
landed and link the write-up.
Exp 2026-07-13: linear-attention scans on NPU (Mamba-2 wins @scale, GDN not-a-win)
New writeup + Experiment-Log entry + Roadmap update. Mamba-2/SSD scan crosses over to
~2x at Q8 2.7B (130M "no crossover" was a small-scale + Q5_K-dilution artifact); Gated
DeltaNet built + coherent end-to-end but loses (chunked-form UT-solve overhead + no
crossover since GDN state fits cache). Keep-warm now default-on (general int8<->fp16 win).
NPU-Quirks page + Tier 10 (multi-domain import root-cause) + efficiency scan
- NPU-Quirks: new page documenting the silent/order-dependent traps — the three
multi-domain dma-buf import constraints (domain establishment, chain-walk count,
import size), the C[0]-only validation false-positive trap, + cross-refs.
- Optimization-Roadmap Tier 10: rewritten with the root-caused three-constraint
picture + shipped fixes (anchor, chunked import); Efficiency scan (E1-E3:
per-tile bsync, per-matmul malloc, per-submit getenv).
- Home: link the NPU-Quirks page.
MoE saturation: M1a expert persist/load + IOVA wall, M3-skew hot-cache viability
Two 2026-06-26 experiments (hardware-validated RK3588 10.3.0.236 / box 10.3.0.241):
- M1a: routed _exps/MUL_MAT_ID persist+load into mixed .orkpack (ggml-ork c89cf8654,
ork-driver 6b0d568); LFM2.5-8B-A1B 2112 entries, reload bit-identical to CPU ref;
confirmed the 4 GiB IOVA wall for real (compute status -1 at ~1909 resident experts).
- M3-skew: routing skew via ffn_moe_argsort; LFM2 Gini 0.44, Qwen3.6-35B-A3B 0.65;
hot-expert cache VIABLE, ~3.5 GB cache -> ~50% decode lookups NPU-resident.
Indexed in Experiment-Log; roadmap MoE thread updated (M0 -> M1a -> M3-skew).