Roadmap: Edge-Asymmetric SLM — heterogeneous NPU-prefill / CPU-decode specification
Full architecture, training and deployment spec for a custom ~2.1B GDN+MoE SLM
whose prefill and decode run on different silicon: W8A8 inside the 32-bit IOVA
window on the RKNPU2, INT4 on the A76 cores in 64-bit host memory.
Every figure is tagged measured / derived / proposed / unverified, because the
incoming brief mixes all four and two of its premises are contradicted by
measurements we already hold:
- M=1 decode is LATENCY-bound before it is bandwidth-bound (one A76 reaches
~8 GB/s of the 21.9 it can stream, which is why PRFM is worth 9-11%). The
'2x from INT4' projection assumes bandwidth is binding; it is not, at one core.
- The DRAM ceiling is ~30 GB/s, not 20-25, and heterogeneous engines are
additive only across DIFFERENT resources -- so pipelined prefill||decode is
zero-sum, and the schedule stays sequential until DMC load says otherwise.
Also corrects the deployment target: Phase 3 should build on ork-driver, not
librknnrt. The closed API cannot reach the precompiled regcmd cache (6.03x lower
per-token build), the NONBLOCK doorbell (130us -> 5us dispatch), hardware task
chaining, or the 0x1040 bank splits -- and the open stack already measures 97%
of librkllmrt prefill and 109% of its decode.
Prices two things the brief omits: weight duplication (~3.2 GB for a 2.1B model,
since the same tensors exist as W8A8 in IOVA and INT4 in host RAM), and the
prefill->decode seam, where two quantizations of one model can diverge in a way
no per-op test detects.
Keeps GDN on the CPU: the on-NPU GDN scan is measured to LOSE to ggml's
solve-free recurrence (1.9x at 0.8B, 4.8x at 9B, gap widening, no crossover
because the state fits cache). Mamba-2/SSD is noted as the variant that does
cross over (~2x at 2.7B) should sequence-mixing offload ever be wanted.
Tier 13: record the 4/8 + N-tiling design as the starting point, not the bank widening
The 2026-09-06 experiment widened WEIGHT_BANK 4->11 for capacity and spent it on
wider segments, which forces DATA_BANK to 1 and collapses mcap 128->16 -- 8x the
programs. Keeping the DEFAULT 4/8 split and N-tiling to segw <= capacity/K (64
columns at K=2048) gives the SAME total weight saving with ~4 programs instead of
16, and no mcap change:
baseline no reuse cap - mcap 128 2 programs 512 KB
11 banks + reuse cap 352 KB mcap 16 16 programs 256 KB
4/8 + N-tile + reuse cap 128 KB mcap 128 4 programs 256 KB
Correctness at 4 banks is already proven (N=192, segw=64 = exactly 128 KB ->
0.0% wrong). It could not be measured here because colsplit gives each core ONE
segment and only subdivides at N > NMAX, so proper N-tiling does not exist yet --
a missing capability, not a negative result.
A 3-trial proxy at N=192 (single chunk, so half the saving) gave +0.1/+9.7/-2.5%
-- mean +2.6% but sign-flipping, consistent with a ~2-3% effect under ~7% spread.
WEIGHT_REUSE: the gain does not clear the noise floor — correctness solved, payoff not demonstrable
Swept M=144..256 with 2 runs per cell (the default does 2 M-tiles throughout, so
the redundant load is a full weight for only M-128 rows -- predicted maximum
waste just above mcap):
M=144 +3.4% M=160 +2.5% M=192 +5.1% M=224 +2.2% M=256 +2.0%
Within-config spread reaches 6.7% (M=192) and 7.0% (M=224), LARGER than the
2-5% between-config difference. The apparent peak at M=192 is one high default
reading. The predicted monotonic decline with M does not appear. Resolving ~3%
against ~7% spread needs ~20+ trials per cell.
Corrects an earlier figure in the same entry: the single-run 5.0% at M=256 is
2.0% on repeat -- a noise excursion passed on without repetition, the same
failure mode as the two SRAM false-positives recorded elsewhere in this log.
Also corrected: the extra programs do NOT cost ~10us each. Chaining already
amortises the submit, so the marginal cost is ~1.3us -- the measured per-task
hardware floor. The cap on this lever is that weight streaming is only ~8% of
runtime at these shapes (weight share falls as 1/M), not program overhead.
WEIGHT_REUSE: the bank-widening fix works, and is worth ~4% — parked, not closed
Widening WEIGHT_BANK 4 -> 11 via 0x1040 (-> 0xb1, DATA=1) turns a 49.8%-wrong
shape into 0.0% wrong, verified directly with the knob confirmed firing. 0x1040
couples the bank split to the K-reduction schedule (mc*K = DATA_BANK * 32 KB), so
DATA=1 forces mcap 128 -> 16 at K=2048; both must move together or the
activations overflow. Loader tile stayed clean, confirming the pairing.
Timed against the default config (4 banks, mcap 128, no reuse), K=2048 N=384:
M=256 836.7 -> 794.5 us 5.0%
M=512 1356.5 -> 1305.6 3.8%
M=1024 2459.0 -> 2366.4 3.8%
Absolute saving grows (42/51/93 us) but relative gain is flat at ~4%, short of
the ~7% predicted at M=1024. Each cell is inside the board's ~10% spread; three
consistent directions is suggestive, not decisive.
Two of my predictions were wrong and are corrected in place: the extra programs
do NOT cost ~10us each (no net loss), and the saving does NOT scale linearly
with M.
Recommendation recorded: do not ship. ~4% on one matmul path costs an N-tiling
scheme, an 8x mcap collapse with unaudited consequences for other shapes, and
full make test + ork_ppl revalidation -- for zero end-to-end value today, since
no local model produces a shape in the envelope. Revisit if per-program cost
drops (ork_pc_*) or shapes change.
straighten the WEIGHT_REUSE record: consolidate six contradictory entries into one
The Experiment Log had accumulated six entries written as the investigation
moved, several stating conclusions later measurement overturned ("narrow
envelope K=4096", "no parameter predicate", "corrupts every time it fires").
Replaced with a single entry stating the solved result, its mechanism, and the
method errors that produced the wrong turns.
Result: reuse correct IFF K*segw <= WEIGHT_BANK banks x 32 KB, with
correct_fraction = min(1, capacity/(K*segw)) verified at 1.0x/2x/10.9x. The
loader streams the whole weight; the banks retain only the last capacity bytes.
Deterministic, confined to reuse tiles, correct columns contiguous. Vendor conv
rule, transfers unchanged. #39 observed an out-of-envelope case.
Also corrected: ork_regs.h (which had received NEITHER prior correction and still
described the superseded K=4096 envelope), Performance.md, and Tier 13 -- whose
"WITHDRAWN / corrupts always" note was written before the corruption was
characterised.
Recorded five method errors with repeatable causes: an unvalidated detector whose
timing loop masked the corruption; stale binaries three separate ways; closing on
a black-box "corrupted"; a shell `env VAR=` that inverted an A/B; and attributing
a pre-existing CI failure to this work.
FINAL: WEIGHT_REUSE corrupts every time it fires — #39 stands, the speedup was corruption
A validated sweep closes the thread. tools/wr_sweep.c mirrors test_matmul (same
PRNG, one context, pack once, run ONCE, full checksum) and carries its own
positive control: two shapes test_matmul proves broken. Both moved, so the
harness is a real detector.
13 shapes, perfect correlation:
>1 M-tile (reuse fires): 9/9 CORRUPTED
1 M-tile (cannot fire): 4/4 unaffected
There is no envelope. The ~33% gain was the cost of skipping a REQUIRED weight
fetch -- fast because wrong. Every timing figure in the preceding entries (15% at
N=512 to 34% at N=8192, the 10.5 GB/s agreement, the order-independence) was
measured on corrupted output, and the 10.5 GB/s "corroboration" never
discriminated: skipping a necessary fetch saves the same bytes as skipping a
redundant one.
Root cause of the error: wreuse_probe was never validated as a detector. It
compared base-vs-reuse over all M*N elements AND sampled a CPU reference, and
still missed the corruption, because its timing loop re-ran the identical submit
20 times so the checked output came from a warm iteration with the weight already
resident. test_matmul runs each shape once.
Rule recorded: a probe never shown to detect a known failure carries no
information when it reports success. Four rounds of findings rested on that
probe -- the narrow envelope, the K-sweep, the N-scaling, the vendor-capacity
test -- and every "correct" cell in them is void.
Tier 13's reopening withdrawn; ork_regs.h corrected; ORK_MTILE_WR stays as
off-by-default RE instrumentation documented as producing wrong results.
WEIGHT_REUSE end-to-end: ZERO — the proven envelope matches no local model
ork_bench on qwen3.5-0.8b, 512-token prefill (173.57 tok/s), ORK_MTILE_WR=1 with
the application counter on: the reuse bit fired 0 times.
The gate is K==4096 and no model on the board produces K=4096. Attention
projections give K = n_embd in {1024, 1536, 2048}; FFN down gives K = n_ff in
{512, 3072, 3584, 6144, 8960}.
So the 1.52x is real, replicated and CPU-reference-verified, but lands on a shape
nothing uses. Characterising the envelope is the whole value of the finding, not
a follow-up item -- and the nearest K a real model uses (2048) is one of the
shapes that FAILS make test with 1830 wrong outputs.
Tier 13 reordered accordingly: item 1 marked THE BLOCKER, item 4 marked blocked
on it.
Recorded against myself: checking whether any model has n_embd=4096 is a
ten-second header read and should have preceded three rounds of probe work, a
make test cycle and a roadmap entry. Cheapest decisive test first.
roadmap: reopen Tier 13 as a backlog item — WEIGHT_REUSE has a proven narrow envelope
Tier 13 was marked BREAK-EVEN from the 2026-07-30 weight-stationary modelling.
That framing is superseded: cross-M-tile reuse at whole K needs no K-tiling and
no accumulation, and in a narrow envelope (K=4096, Sn=1) it is bit-exact and
worth 26.4% at N=2048 to 33.3% at N=8192, with make test passing under the gate.
Recorded as five ordered work items rather than a claim, because the blocker is
specific: outside the gate it computes WRONG (three test_matmul goldens), and
K=3584 fails at the same mcap=64 as K=4096 -- so the boundary is not M-tile
geometry and has no mechanism yet. Item 1 is characterising it; the current
hardcoded gate is explicitly called out as a measured envelope, not a derived
one, i.e. a superstition that happens to pass the suite.
Also flags that the end-to-end number is the one that decides whether this
matters, not the per-matmul 1.5x.
walk back the WEIGHT_REUSE claim: make test fails outside a narrow envelope
Run after the wiki was already updated -- the order should have been reversed.
With ORK_MTILE_WR=1 applied unconditionally, make test fails three test_matmul
goldens, ref-checked WRONG: {256,3584,3584}, {256,2048,2048}, {96,3584,18944}.
Baseline without the knob: ALL TESTS PASSED, 0 mismatches, so attribution is
clean.
The probe swept only K=4096, Sn=1. The honest claim is therefore narrow: there
EXISTS a shape family where cross-M-tile reuse is bit-exact and up to 1.52x.
#39's "computes wrong" is broadly correct and is refined, not refuted -- what is
new is that it is not universal, which the fold path (locked to a baked K=3584
N=1216 regcmd) could never have shown.
Most informative fact for the next attempt: K=3584 has the SAME mcap=64 as
K=4096 and still fails, so the boundary is NOT the M-tile geometry.
The knob is gated in-source to the measured envelope so it cannot be switched
into a wrong-answer configuration. Performance.md, Optimization-Roadmap Tier 13
and ork_regs.h all corrected to match.
wiki: wire the session record in, and correct the stale WEIGHT_REUSE verdicts
The session page had zero inbound links -- the exact failure that caused the SRAM
ground to be re-covered this morning. Linked from Home and the Experiment Log.
Corrections:
- Performance.md: the int4 WEIGHT_REUSE bullet now also carries the int8 result
(colsplit M-tile path, bit-exact, 15% at N=512 to 34%/1.52x at N=8192).
- Optimization-Roadmap Tier 13: both "unproven mechanisms" it was gated on are
settled. Cross-task WEIGHT_REUSE for an int8 matmul weight is PROVEN and does
not require WEIGHT_BANK residency (works at 32 MB). On-chip accumulator
retention is moot -- K-slicing is monotonically worse in NPU time alone, so
free accumulation would recover at most 0.39x. The weight-stationary framing
was wrong: reuse across M-tiles at whole K needs no K-tiling and no
accumulation.
- Experiment Log: records that WEIGHT_REUSE was ALREADY default-on for int4,
which makes the int8 result an extension of known-good behaviour rather than a
surprise, and reframes #39 as a fold-path property not a hardware limit.
Roadmap: precompiled regcmd on decode (1.47x FFN layer) + the SRAM rule (descriptors are kernel-read; SRAM there hard-wedges)
Roadmap: CLOSED — the M=1 gap is host CPU work, not bandwidth; kernel says the NPU never waits
Roadmap/Recipes: M=1 doorbell partially root-caused — 62% of a call is host, and it ran off the big cores
Roadmap: eight disproven causes for the FFN handler's 2.4x DMA gap; the next probe must decompose inside ork_i8_mm_run
Roadmap: NEXT item — replicate the probe's ~20 GB/s on the critical path (handler is 1.75-2.4x off)
roadmap: dom_dirty fix landed and board-validated (5751e06)
roadmap: close the GDN discoverability item; record the dom_dirty timeout-mismatch finding
GDN row landed (with the scan section it needed). dom_dirty fix is written but the board wedged
before make test returned, so it stays open and marked not-hardware-validated, with the 40x
per-dtype timeout mismatch that makes arming the flag alone worse than the bug.
roadmap: Tier 19 second pass — dtype router moved to npu.c, spine body renamed agnostic
roadmap: Tier 19 done — doorbell spine moved to npu/core/, seam measured at zero crossing symbols
roadmap: Tier 19 — doorbell spine is dtype-agnostic substrate living in src/npu/i8/
Tier 15: deterministic placement + queryable residence landed; GGUF stays on the stock loader
Tier 15: residence must be op-less and placement belongs in the pack — three failed hook attempts show why
Tier 15 stage 3: eager residence + readiness event — lazy load lands inside timed regions and produced a wrong 27B throughput number
Tier 18: correct the prize (9.4% real, n=3), exonerate domain swaps (12 switches, one-time), and flag the unexplained 12x CPU gap as the bigger question
Tier 18: core-partition the precision modes (2/1 split) — the idea, what is measured, and the two things that must be established first
Tier 17 RESOLVED: rknpu files dma-buf imports under domain 0 (kernel patch 03)
Root-caused the multi-domain IOVA leak to the vendor driver's import path and
fixed it: 30 WARN_ON(unmapped != size) per multi-domain run -> 0, and the board
no longer dies on the third consecutive run. Documents the patch, the measured
before/after, and marks the two superseded hypotheses (switch-timeout mechanism,
per-domain scratch identification) as wrong.
roadmap Tier 17: cause identified, our kernel patches exonerated
15-second reproducer (ORK_DOMAINS_FORCE on a 155 MiB pack) shows the leak is fixed-size
(~30 mappings per multi-domain run), scaling with neither domains nor weights, pointing at
per-domain scratch teardown. Three-way kernel bisect shows stock/SRAM-only/SRAM+patch all
leak identically, refuting the SRAM-kernel suspicion.
Notes the practical consequence: per-PROCESS not per-workload, so a long-lived daemon
owning the NPU avoids most of it.
roadmap Tier 17: prime suspect is our own SRAM kernel — bisect is now item zero
The board runs a custom kernel (CONFIG_ROCKCHIP_RKNPU_SRAM=y + an undocumented source
patch). That config maps on-chip SRAM into the IOMMU domain, which multi-domain must move
per switch -- exactly the function that times out, and it explains why single-domain never
leaks. Adds items 0/0b/0c: bisect against stock, read the driver source, recover the patch
diff. The stock bisect decides whether the rest of the tier is even the right
investigation.
roadmap: Tier 17 — multi-domain teardown leaks IOVA until reboot
Blocker for >4 GiB models: they can be measured once per boot. Mechanism is kernel-side
(destroy-path domain switch times out, unmap runs against the wrong domain, IOVA never
reclaimed). Our half -- the sizer under-counting mixed packs -- is fixed.
Work items ordered with measurement FIRST, because three attempts to quantify the leak all
failed to measurement error and nothing below can be prioritised without a real rate.
Includes the caller-side domain activation as an explicit do-not-re-apply: it deadlocks.
roadmap Tier 16: retitle and re-scope — quantiser is within 7% of Q4_0
Like-for-like matmul measurement: per-channel W4A4 is 6.4x Q4_0; grouping -> 2.2x,
rotation -> 1.56x, int8 activations -> 1.07x. The tier's original 'per-group scales are
the ceiling' framing was wrong twice over -- granularity is half the weight-side story,
and activation width is the largest single remaining lever (-31%).
Target config is G=32 + rotated + int8 activations, and shipping it is now a RUN-PATH
problem: the offline grouped int8 path is done, the on-device one is the blocker.
Split the 1.07x-numerics vs 1.84x-excess-PPL disproportion into its own item -- depth
compounding is ruled out, and the next measurement is a tail statistic rather than a mean.