Board provisioning for the ROCK 5B+ (.235): DDR DVFS and the A76 2400 gate
Two things were missing out of the box, neither of them a kernel problem.
DDR DVFS: there was no dmc devfreq device at all, so the DDR governor could not
be pinned and step 3 of the benchmark standard was unperformable. Armbian's
rock-5b-plus U-Boot ships upstream TF-A as BL31, which has the generic Rockchip
SIP dispatcher but none of the DDR services behind it -- confirmed by symbol
presence in the SPI image, not inferred. Rebuilding v2026.04 from Armbian's own
.config with rkbin's BL31 gives ATF version 0x100 and LPDDR5 trained to 2400 MHz.
A76 clock: the cap is Rockchip's PVTM speed bin, not a misconfiguration. The
second cell of opp-supported-hw is a pvtm-volt-sel mask and this die reports
volt-sel=2, so only the 2304 OPP matches. Lifted with a DT overlay, which is
reversible and survives kernel updates where a patched DTB would not.
Notes the part that was not previously written down anywhere: .236 carries that
same gate overridden in its custom kernel's DTB, so every A76 @ 2.4 GHz figure
in this wiki was taken on a bin-overridden board.
Roadmap: Edge-Asymmetric SLM — heterogeneous NPU-prefill / CPU-decode specification
Full architecture, training and deployment spec for a custom ~2.1B GDN+MoE SLM
whose prefill and decode run on different silicon: W8A8 inside the 32-bit IOVA
window on the RKNPU2, INT4 on the A76 cores in 64-bit host memory.
Every figure is tagged measured / derived / proposed / unverified, because the
incoming brief mixes all four and two of its premises are contradicted by
measurements we already hold:
- M=1 decode is LATENCY-bound before it is bandwidth-bound (one A76 reaches
~8 GB/s of the 21.9 it can stream, which is why PRFM is worth 9-11%). The
'2x from INT4' projection assumes bandwidth is binding; it is not, at one core.
- The DRAM ceiling is ~30 GB/s, not 20-25, and heterogeneous engines are
additive only across DIFFERENT resources -- so pipelined prefill||decode is
zero-sum, and the schedule stays sequential until DMC load says otherwise.
Also corrects the deployment target: Phase 3 should build on ork-driver, not
librknnrt. The closed API cannot reach the precompiled regcmd cache (6.03x lower
per-token build), the NONBLOCK doorbell (130us -> 5us dispatch), hardware task
chaining, or the 0x1040 bank splits -- and the open stack already measures 97%
of librkllmrt prefill and 109% of its decode.
Prices two things the brief omits: weight duplication (~3.2 GB for a 2.1B model,
since the same tensors exist as W8A8 in IOVA and INT4 in host RAM), and the
prefill->decode seam, where two quantizations of one model can diverge in a way
no per-op test detects.
Keeps GDN on the CPU: the on-NPU GDN scan is measured to LOSE to ggml's
solve-free recurrence (1.9x at 0.8B, 4.8x at 9B, gap widening, no crossover
because the state fits cache). Mamba-2/SSD is noted as the variant that does
cross over (~2x at 2.7B) should sequence-mixing offload ever be wanted.
wiki: wire the session record in, and correct the stale WEIGHT_REUSE verdicts
The session page had zero inbound links -- the exact failure that caused the SRAM
ground to be re-covered this morning. Linked from Home and the Experiment Log.
Corrections:
- Performance.md: the int4 WEIGHT_REUSE bullet now also carries the int8 result
(colsplit M-tile path, bit-exact, 15% at N=512 to 34%/1.52x at N=8192).
- Optimization-Roadmap Tier 13: both "unproven mechanisms" it was gated on are
settled. Cross-task WEIGHT_REUSE for an int8 matmul weight is PROVEN and does
not require WEIGHT_BANK residency (works at 32 MB). On-chip accumulator
retention is moot -- K-slicing is monotonically worse in NPU time alone, so
free accumulation would recover at most 0.39x. The weight-stationary framing
was wrong: reuse across M-tiles at whole K needs no K-tiling and no
accumulation.
- Experiment Log: records that WEIGHT_REUSE was ALREADY default-on for int4,
which makes the int8 result an extension of known-good behaviour rather than a
surprise, and reframes #39 as a fold-path property not a hardware limit.
wiki: Precision Capability Matrix — what each dtype implements, and which blanks are real
Generated table (tools/precision_matrix.sh, mirrored in README) plus the analysis the
table cannot carry: sorting every blank into by-design vs genuinely-open.
The headline is that the 167-vs-18 function gap between int8 and int16 is mostly not a
hole. int4 has no SDP/activation rows because RK3588 is W8A8 or W4A4 symmetric and the SDP
LUT op consumes int8/int16 — there is no int4 activation path to write. int16 is the
ACTIVATION precision, not a weight-storage tier, so none of the pack/load/import machinery
applies to it. fp16 weights are transient, so nothing to persist.
Genuinely open, and small: fp16 HW chain and slice-and-dice, int4 resident-KV and M-fold,
int16 async stream and bmm — each with a note on whether it is motivated.
Also documents the matrix's one blind spot: it classifies by dtype token in the symbol
name, so a shared implementation serving several precisions is invisible to it. Those are
the OVERRIDES rows in the generator and must be added by hand — fp16's multicore path IS
i8/colsplit.c, and counting it as int8 is part of why int8 looks so large.
architecture: hybrid CPU/NPU precision decode (int5 storage, NEON free inflate, sensitivity-routed, swap-hidden pipeline)
Add: Building the Regcmd Graph — segment/boundary methodology
Replicable technique for turning any model + NPU into the fewest-submit on-NPU regcmd graph:
segments (maximal contiguous NPU runs = one submit), the boundary taxonomy (hard/movable/free),
the chaining mechanics (HW-chain within a segment, keepwarm one-mode, DMA-resident bsync-dedup,
SW-chain precompiled across boundaries), and the 'collapse to #hard+1 mega-segments' principle.
Qwen3-1.7B worked example: 6 segments -> 2, pivoting on softmax (the one load-bearing boundary =
the 2b PPU RE). Portability notes (what varies per model/SoC vs the constant methodology).
NPU-Quirks page + Tier 10 (multi-domain import root-cause) + efficiency scan
- NPU-Quirks: new page documenting the silent/order-dependent traps — the three
multi-domain dma-buf import constraints (domain establishment, chain-walk count,
import size), the C[0]-only validation false-positive trap, + cross-refs.
- Optimization-Roadmap Tier 10: rewritten with the root-caused three-constraint
picture + shipped fixes (anchor, chunked import); Efficiency scan (E1-E3:
per-tile bsync, per-matmul malloc, per-submit getenv).
- Home: link the NPU-Quirks page.
Update wiki: fused Q4_K->int8 dequant result + reframe negative verbiage
Status updates (current as of 2026-06-24):
- Tier 5 MoE: prefill walked 0.28 -> 0.97 (chaining) -> 2.90 (NEON repack)
-> 3.13 t/s (fused Q4_K->int8 dequant, ork-driver b520ee7 / ggml-ork
b5bc5ce8b). Deficit ~80x -> ~7.2x under CPU. New wall = the NEON
f32->int8 quant+tile (77%); the Q4_K->f32 dequant is now folded in (0ms).
- Roadmap Tier 5 table + 5b/5c verdicts + progress log; MoE Expert Offload
page status/verdict + a 2026-06-24 follow-up section; Experiment-Log index.
- Recorded the ORK_MOE_NPU=0-still-enables-NPU-MoE benchmark gotcha.
Reframed discouraging "dead-end" verbiage across the wiki to "not yet
viable" / "ruled out" (per the don't-discourage-future-efforts policy):
MoE, int4/Hadamard, zero-copy, attention-on-NPU, per-op backend, and the
ISA-Reference "Ruled out" section. Hardware-impossible facts (same-precision
MAC, KV caches can't cross backends) kept as factual constraints.
roadmap cleanup: reframe int4 as 'not yet viable' (not dead-end) + what would flip it; flip NEON-repack flag to landed; add CPU-Big-Core-Contention + Quantization-Paths-and-Baselines pages + Home links
Document prior art on model and register fuzzing
revert: undo recent residency bug and summary changes
doc: add project summary and link weight-residency bug to home page
doc: implement Proposal III self-healing auto-recovery
Add Optimization Roadmap (next steps for the open-vs-closed gap)
Prioritized next-steps derived from the RKNN apples-to-apples: Tier 1 (prefill
large-M multi-core, the 3.2x gap from 6%-vs-3x scaling), Tier 2 (decode per-matmul
submit, the 1.24x/~100us gap), Tier 3 (persistent per-layer program, async overlap).
Status checkboxes + progress log; linked from Home.
Add Experiment Log blog + benchmark standards + 12 experiment writeups
A running blog of NPU experiments across the open stack (ork-driver -> ggml-ork
-> llama.cpp), most recent on top, linked from Home. Each full writeup carries
the implementing code (commit + symbol + env-knob), the standardized benchmark
(Qwen3-1.7B-w8a8, llama-bench warm, governors verified, vs librkllmrt on the same
.rkllm), and a kept/reverted/ruled-out verdict. Plus the Benchmark Standards &
Methodology page codifying the procedure and env knobs.
wiki: add Heterogeneous Serving & Scheduling design page
Captures the NPU/GPU/CPU split + multiplexing architecture: mechanism (per-device backends incl.
ork-driver) vs policy (engine scheduler); the bottleneck-complementarity principle (compute-bound
prefill + memory-bound decode share the SoC, two memory-bound tasks contend on the one LPDDR
controller); chunked prefill + core-budget primitives; 5-stage roadmap. Links from Home; fixes the
stale 'bandwidth-bound decode' line in the Home Performance blurb.
wiki: Performance page — tok/s baselines + bandwidth analysis + lever priorities
Decode/prefill tok/s for a ~1.5B GQA model on RK3588 (fp16 + int8). Key finding: decode is
memory-bandwidth-bound (reads all weights/token), so int8 (half the bytes) gives 1.58x
decode (4.52->7.12 tok/s) and is the real lever; 3-core does NOT help decode (shared memory
controller) and a naive core_mask flip crashes the kernel (rknpu_job_subcore_commit Oops —
needs per-core subcore_task[] setup); decode-single-submit is negligible (bytes, not submits).
Prefill is compute-bound. Levers documented in priority order.
wiki: populate ork-driver wiki — RE roadmap, feasibility log, regcmd ISA reference, recording-notes convention, Home index
Migrated the regcmd reverse-engineering record from the oRKLLM wiki and consolidated the
scattered RE scratch (register meanings, tile layouts, limits, state gotchas, methodology,
dead-ends) into a single regcmd-ISA-Reference page. Home indexes them; Recording-Investigation-
Notes sets the convention for future findings.