straighten the WEIGHT_REUSE record: consolidate six contradictory entries into one
The Experiment Log had accumulated six entries written as the investigation
moved, several stating conclusions later measurement overturned ("narrow
envelope K=4096", "no parameter predicate", "corrupts every time it fires").
Replaced with a single entry stating the solved result, its mechanism, and the
method errors that produced the wrong turns.
Result: reuse correct IFF K*segw <= WEIGHT_BANK banks x 32 KB, with
correct_fraction = min(1, capacity/(K*segw)) verified at 1.0x/2x/10.9x. The
loader streams the whole weight; the banks retain only the last capacity bytes.
Deterministic, confined to reuse tiles, correct columns contiguous. Vendor conv
rule, transfers unchanged. #39 observed an out-of-envelope case.
Also corrected: ork_regs.h (which had received NEITHER prior correction and still
described the superseded K=4096 envelope), Performance.md, and Tier 13 -- whose
"WITHDRAWN / corrupts always" note was written before the corruption was
characterised.
Recorded five method errors with repeatable causes: an unvalidated detector whose
timing loop masked the corruption; stale binaries three separate ways; closing on
a black-box "corrupted"; a shell `env VAR=` that inverted an A/B; and attributing
a pre-existing CI failure to this work.
walk back the WEIGHT_REUSE claim: make test fails outside a narrow envelope
Run after the wiki was already updated -- the order should have been reversed.
With ORK_MTILE_WR=1 applied unconditionally, make test fails three test_matmul
goldens, ref-checked WRONG: {256,3584,3584}, {256,2048,2048}, {96,3584,18944}.
Baseline without the knob: ALL TESTS PASSED, 0 mismatches, so attribution is
clean.
The probe swept only K=4096, Sn=1. The honest claim is therefore narrow: there
EXISTS a shape family where cross-M-tile reuse is bit-exact and up to 1.52x.
#39's "computes wrong" is broadly correct and is refined, not refuted -- what is
new is that it is not universal, which the fold path (locked to a baked K=3584
N=1216 regcmd) could never have shown.
Most informative fact for the next attempt: K=3584 has the SAME mcap=64 as
K=4096 and still fails, so the boundary is NOT the M-tile geometry.
The knob is gated in-source to the measured envelope so it cannot be switched
into a wrong-answer configuration. Performance.md, Optimization-Roadmap Tier 13
and ork_regs.h all corrected to match.
wiki: wire the session record in, and correct the stale WEIGHT_REUSE verdicts
The session page had zero inbound links -- the exact failure that caused the SRAM
ground to be re-covered this morning. Linked from Home and the Experiment Log.
Corrections:
- Performance.md: the int4 WEIGHT_REUSE bullet now also carries the int8 result
(colsplit M-tile path, bit-exact, 15% at N=512 to 34%/1.52x at N=8192).
- Optimization-Roadmap Tier 13: both "unproven mechanisms" it was gated on are
settled. Cross-task WEIGHT_REUSE for an int8 matmul weight is PROVEN and does
not require WEIGHT_BANK residency (works at 32 MB). On-chip accumulator
retention is moot -- K-slicing is monotonically worse in NPU time alone, so
free accumulation would recover at most 0.39x. The weight-stationary framing
was wrong: reuse across M-tiles at whole K needs no K-tiling and no
accumulation.
- Experiment Log: records that WEIGHT_REUSE was ALREADY default-on for int4,
which makes the int8 result an extension of known-good behaviour rather than a
surprise, and reframes #39 as a fold-path property not a hardware limit.
SRAM: retract "not a separate port" — it IS one, measured 2026-08-11 and missed
The 2026-09-05 entry explained a flat NPU-weight-in-SRAM result by asserting
syssram sits behind the same LPDDR controller, "a storage class, not a second
port". That mechanism was invented to fit the data and is falsified by a direct
measurement a month earlier that this session failed to find:
CPU->SRAM cacheable+flush held 104% (2.89 -> 2.99 GB/s) while three big cores
saturated DRAM -- completely unaffected. (test_sram_pipe, A76 core 4)
Additivity is real. The flat 2026-09-05 row is the narrower claim it actually
measured: placing the NPU's *weight* in SRAM does not help -- most likely because
that NPU side is submit-overhead-bound (~95% overhead) rather than
weight-bandwidth-bound, so placement cannot move it.
Also corrected: "on-chip is slower than DRAM, 3 vs 10.5 GB/s" is about the
MAPPING, not the memory. Through the default pgprot_noncached mapping SRAM and
DRAM are identical (1.19 GB/s). ~3 GB/s is the true feed ceiling once data lands
(WC 3.17, cacheable+dc cvac 2.77) and is affinity-insensitive; the ~21 GB/s
cacheable write-only figure is a cache illusion.
Both directions still have an unclosed caveat: the 2026-08-11 saturation was CPU
threads, not the NPU, so NPU-read || CPU-write has never been measured airtight.
Probes: test_sram_bw / test_sram_npubw / test_sram_pipe (2026-08-11), preserved
on origin/wip/moe-saga-2026-08-18 in the llama.cpp fork's vendored copy.
Performance: little cores (A55) tested for MoE experts — no benefit (~17 t/s -t4/6/8)
Performance: 35B MoE W4A4 — experts belong on CPU (21 t/s) not NPU (6.13)
Records the decisive rung-1 A/B: CPU/NEON experts 20.99 t/s vs NPU-W4A4 experts
6.13 (pp512, RK3588) — CPU 3.4x faster, ~meets the 22 t/s target. Root cause:
~18us/program fixed dispatch floor x ~800k tiny expert programs; NEON has no
per-op setup. DRAM not saturated by either => headroom above 21.
Also records: HW WEIGHT_REUSE default-on (bit-exact int4, ~8-15%), nonblock is
the fastest correct completion (blocking miscomputes M>1), expert-parallel 1.6x,
the dead levers (coalesce/SRAM/M-pad/poll-fix), test_moe_smoke validator, and
rung-3 (concurrent hybrid: A76+A55 experts + NPU async ∥, ~27 t/s projected).
Update wiki: fused Q4_K->int8 dequant result + reframe negative verbiage
Status updates (current as of 2026-06-24):
- Tier 5 MoE: prefill walked 0.28 -> 0.97 (chaining) -> 2.90 (NEON repack)
-> 3.13 t/s (fused Q4_K->int8 dequant, ork-driver b520ee7 / ggml-ork
b5bc5ce8b). Deficit ~80x -> ~7.2x under CPU. New wall = the NEON
f32->int8 quant+tile (77%); the Q4_K->f32 dequant is now folded in (0ms).
- Roadmap Tier 5 table + 5b/5c verdicts + progress log; MoE Expert Offload
page status/verdict + a 2026-06-24 follow-up section; Experiment-Log index.
- Recorded the ORK_MOE_NPU=0-still-enables-NPU-MoE benchmark gotcha.
Reframed discouraging "dead-end" verbiage across the wiki to "not yet
viable" / "ruled out" (per the don't-discourage-future-efforts policy):
MoE, int4/Hadamard, zero-copy, attention-on-NPU, per-op backend, and the
ISA-Reference "Ruled out" section. Hardware-impossible facts (same-precision
MAC, KV caches can't cross backends) kept as factual constraints.
wiki: rename rkllm_bench -> rknpu_bench (tool renamed in ork-driver 3df41a5)
Performance: document auto-tuner defaults, per-channel quant validation, NPU pool
Performance: attention-on-NPU tried, doesn't pay off (documented dead-end)
attn_cost de-risk floor looked great (492 vs 1252ms @M=256, ~2.5x), but integration regressed
prefill (36->24 tok/s MC, 27->20 single-core). The floor excluded: 896 tiny per-head matmuls'
per-submit/MC-spawn overhead, fp16<->int8 dtype RESET thrash per layer, and the inherent CPU
gather/transpose/softmax/scatter (~1100ms). Even idealized ~1.4x at high complexity. Verdict:
CPU flash attention is better. Long-prefill lever is chunked prefill (engine-level), not the NPU.
Performance: prefill flash+pool result, honest same-clock numbers (1.6x)
Pool + flash-style attention: same-clock isolation (OC cancels) 59->95 tok/s @M=128 (1.62x),
44->75 @M=256. Lessons: persistent pool mandatory (per-op spawn regresses), pin to perf cores
(big.LITTLE stragglers), CPU governor=performance. Note RAM OC to 3500 didn't apply (DDR still
2112). Long-prefill O(M^2) remains -> attention-on-NPU.
Performance: prefill CPU optimization — threaded attention on perf cores (+11%)
Prefill is CPU-op bound (M=128: NPU 25% / attention 47% / other-CPU 28%). Threaded the O(M^2)
attention across heads pinned to the big.LITTLE A76 cores (auto-detect, oversubscribed 2x):
1320->515 ms, prefill 57.7->64.2 tok/s @M=128. Lessons: pin to perf cores (unpinned regresses —
A55 stragglers) and CPU governor must be performance (DietPi default throttles; perf_governor
handles it). Remaining: other-CPU malloc/ops + long-prefill O(M^2) flash-attention tiling.
wiki: document the rknpu_job_next crash + that multi-core is done (1.69x/11.0)
Performance levers: multi-core is implemented (not a future risky TODO) with both crash lessons
(all-subcore-populate; per-call RESET is load-bearing job-queue recovery — RESET-once + concurrent
short-timeout submit storm hangs rknpu_job_next with IRQs disabled, power-cycle required). Prefill
reframed as the CPU-ops lever; decode single-submit marked done-in-multi-core / parked-for-prefill.
ISA reference state-gotchas updated to match.
Performance: decode gap closed — 3-core + full-K single-submit = 11.0 tok/s (~96% of librkllmrt)
Two stacked levers: multi-core (5.0->8.6, 1.69x) then per-core full-K single-submit (8.6->11.0).
The single-submit gives nothing single-core (5.0->5.3) — it only pays combined with multi-core,
where per-submit overhead is the binding constraint (decode was 97% ork_mm_run, cores ~12% util).
int8, K<=10752, dual layout fits IOVA at ~1.7B. Prefill unchanged (CPU-op bound, Amdahl).
Performance: multi-core results (1.69x decode) + 2-vs-3-core efficiency analysis
3-core decode 8.6 tok/s (1.69x, ~75% of librkllmrt's 11.5), RAM BW peak 42->88%, validated
correct. Prefill only 1.08x — gated by serial CPU ops (Amdahl), ties to the CPU/NPU-overlap
lever. 2-vs-3: neither saturates (17% vs 12% per-core util), RAM BW avg identical ~16%, so the
3rd core adds only +12% at worse per-core efficiency (3.74->2.79 tok/s/core) — it's a sync-overhead
wall, not bandwidth. Levers: persistent thread pool (saturate fewer cores), adaptive per-matmul
core count, CPU/NPU overlap for prefill.
Performance: same-state numbers + NPU/RAM-BW telemetry overturn the decode analysis
Re-measured at DDR 2112 (prior numbers were DDR-parked, ~2x low) with utilization sampling:
librkllmrt 11.5 tok/s decode (3 cores ~48% each, RAM BW 48%/83%) vs ork-driver 5.0 (core0 only
44%, C1/C2=0, RAM BW 18%/42%). Conclusions:
- Decode is NOT memory- or compute-bound on either stack — it's idle/latency-bound; ork-driver
leaves RAM ~82% idle on a single core. Don't call it memory-bound.
- The 2.3x decode / 2.8x prefill gap is PARALLELISM (multi-core). Corrected the old claim that
multi-core 'doesn't help decode' — telemetry shows 3 cores reach 2.3x tok/s and 2.6x RAM BW.
- Even librkllmrt peaks ~80% RAM BW: headroom for heterogeneous CPU+GPU+NPU.
- Decode single-submit fast path: NO gain at full DDR (4.85->5.34); the 1.23x was a parked-DDR
artifact. Reranked levers: multi-core NPU > heterogeneous split > cut single-core idle.
Stride-register route is closed: NPU streams weights contiguously, no per-N-tile stride
Probed with tools/slice_probe (full-K buffer, run a Kp-slice, classify each N-tile). Baseline:
N-tile 0 correct, N-tiles 1+ wrong (next N-tile lands inside N-tile 0's full-K data). Hunt:
0x1030/0x1034 (fetch-size) wedge when enlarged; 0x1088/0x107c/0x1044 no effect; 0x1024's two
K-fields are coupled to both data length and addressing — no setting gives all-correct. So the
engine consumes weights as a contiguous KT_op x N-tiles stream; the buffer must match the op's K.
In-place slicing of a full-K buffer is impossible. The remaining single-layout route is (b):
full-K-only with M<=R-chunk prefill (no K-split for K<=10752), at the cost of prefill weight
re-reads — to be measured.
Correct single-submit K ceiling: K=10752 exact (raw length, not 256 tiles)
The 256-K-tile / int8-reaches-16384 hypothesis was REFUTED by tools/ksubmit_probe (a K-sweep
that validates each single submit vs CPU). Findings on RK3588:
- ceiling = K=10752 exactly (10752 OK, 10784 wedges) = 21x512 = 336 fp16 / 168 int8 K-tiles
- dtype-independent (fp16: 8192 OK, 11008 wedge — same raw band)
- N-independent (K=10752 OK at N=512/2048/8192) — purely a contraction-length cap
- near but under cbuf_elems/3 (32768/3); exact mechanism unconfirmed (RK3576 cross-check would
tell whether it scales with cbuf)
So rkllm can single-submit K up to 10752 (both dtypes); beyond, even the closed runtime must
K-split. Updated Performance + regcmd-ISA-Reference; the wrong values are gone.
Performance/ISA: decode single-submit experiment, 256-K-tile ceiling, IOVA blocker
- Decode single-submit (M=1 full-K per N-slice) measured at 1.23x (2.34->2.87 tok/s int8), but
a 2.5x submit cut bought only 1.23x -> per-submit overhead is small; decode overhead is mostly
the fp32 quant round-trip + per-submit DMA syncs, not submit count. Re-ranked decode levers.
- The single-op K ceiling is a HARDWARE 256-K-tile cap (0x1044): fp16 K<=8192, int8 ~K<=16384
(denser packing). Answers 'does rkllm do K>8192 single-submit?': fp16 no (closed runtime must
K-split too), int8/w8a8 yes to ~16384 (covers all LLM K-dims). Corrected the ISA reference,
which wrongly said 'any K in one submit'.
- The dual-layout that enabled the fast path doubles NPU memory and exhausts the rknpu IOMMU
IOVA range (fp16 28-layer: 'failed to allocate IOVA: -12'); reverted. Single-layout
approaches (stride-register slicing) documented as the path forward.
wiki: correct decode analysis — ork-driver is software-overhead-bound, NOT bandwidth-bound
The apples-to-apples disproves the earlier 'decode is bandwidth-bound' claim for the open
stack: librkllmrt hits ~9.7 GB/s (near the LPDDR ceiling, IS bandwidth-bound) but ork-driver
int8 is only ~4.0 GB/s (~40% of ceiling) on the IDENTICAL w8a8 bytes. Same bytes + 2.4x gap =
software overhead (fp32 quant/dequant round-trips, CPU ops, per-submit/N-tile cost, no overlap),
not weight-read time. Re-prioritized: overhead reduction (real int8 quant, CPU/NPU overlap,
fewer submits, on-NPU ops) is the primary DECODE lever — correcting the earlier wrong calls that
decode-single-submit and CPU-overlap were low-value (they were premised on bandwidth-bound).
wiki: real apples-to-apples — ork-driver vs librkllmrt on Qwen3-1.7B w8a8 (same model/quant/board)
librkllmrt: decode 5.70 / prefill 136.6 tok/s. ork-driver int8: 2.34 (~41%) / 34.3 (~25%).
Open stack within ~2.4x decode / ~4x prefill of the closed runtime. Prefill gap is mostly
multi-core (librkllmrt uses 3 NPU cores, ork-driver core 0); decode gap is hybrid overhead
(CPU ops + quant/dequant + N-tiling). Method documented.
wiki: flag the bench is ork-driver-internal (relative fp16/int8 + bandwidth), NOT a librkllmrt comparison
Synthetic weights, dummy int8 quant, CPU ops, no tokenizer => 'matmul-engine throughput at
model scale', not full-runtime tok/s. A meaningful vs-rkllm number needs the same model+quant
on both (run a real .rkllm w8a8 through librkllmrt; ork-driver int8 at that config; ideally the
same real weights post-llama.cpp-integration). Added as a TODO.
wiki: Performance page — tok/s baselines + bandwidth analysis + lever priorities
Decode/prefill tok/s for a ~1.5B GQA model on RK3588 (fp16 + int8). Key finding: decode is
memory-bandwidth-bound (reads all weights/token), so int8 (half the bytes) gives 1.58x
decode (4.52->7.12 tok/s) and is the real lever; 3-core does NOT help decode (shared memory
controller) and a naive core_mask flip crashes the kernel (rknpu_job_subcore_commit Oops —
needs per-core subcore_task[] setup); decode-single-submit is negligible (bytes, not submits).
Prefill is compute-bound. Levers documented in priority order.