Repository navigation
Benchmark Configs Registry
The reproducibility ledger. Every performance number in the wiki is only as good as the config that produced it. This page records, per benchmark run, the full provenance needed to reproduce it: git commit (both repos), branch, env vars, exact tool + args, model/orkpack artifact, board config, and the result — plus the gotchas that silently change the number if you miss them.
This complements Benchmark Standards & Methodology (the rules) and the Experiment Log (the narrative). This page is the config lookup table: "what exactly was set when we measured X."
Golden rule, learned the hard way: a 7B ork number is meaningless without stating (a) the git commit of both repos, (b) whether it was a single clean run on a fresh boot, and (c) the residency layout it rolled (multi-domain layout is non-deterministic — see gotchas). The same binary on the same board gives 48 or 52 t/s depending on the boot.
| Item | Value | How to set / verify |
|---|---|---|
| Board | RK3588, 10.3.0.236 (ssh alias board) |
tools/util/board -c 'echo up' |
| CPU governor |
performance @ 2.4 GHz (A76 cpu4–7) |
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor |
| DDR governor |
performance @ 2112 MHz |
echo performance | sudo tee /sys/class/devfreq/dmc/governor; verify cat /sys/class/devfreq/dmc/cur_freq → 2112000000
|
| CMA | kernel default (256 MB), no cma= bootarg |
cat /proc/cmdline → no cma=. Irrelevant to residency — ORK_PERSIST import maps IOVA, not CMA (see gotchas). |
| NPU stream | single-stream | Never run ork + rkllm (or two NPU procs) concurrently — wedges the IOMMU. |
| Run discipline | one clean NPU run per boot | Cross-run IOVA is not freed; a 2nd 7B run this boot may fail to allocate. Reboot between arms. |
| Wedge recovery |
sudo reboot (soft) / HA "Rock 5B Plug" off→35 s→on (hard) |
Never kill -9 an in-flight submit. Use SIGTERM + board-side timeout -s TERM. |
| Thread count |
-t 4 (big cores only) |
A55 little cores drag the threadpool barrier. |
The headline 7B comparison. ✅ Rewritten 2026-08-03: ork 7B direct prefill is ~73 t/s ≈ parity with rkllm — reproduced on a clean boot. Earlier revisions of this entry reported a bogus 45–59 t/s "regression"; that was measurement error (wrong P, sudo, stale driver, no-op reboots), not a real number. The correct config is in the table.
| ork (direct) | rkllm (closed) | |
|---|---|---|
| Runtime | ggml-ork → ork-driver, ORK_DIRECT=1
|
librkllmrt 1.2.3 |
| llama.cpp fork |
feat/ork-async-stream (frontend ggml-ork.cpp == bea5e49ea); build from ~/llama.cpp-orkd
|
— |
| ork-driver version (CRITICAL) |
0.7.x — build from ~/llama.cpp-orkd, NOT ~/llama.cpp/diverged-history (stale v0.3.3, 6f52ee3 → −14 t/s) |
— |
| Model artifact |
qwen2.5-7b-instruct-q8_0-*-of-00003.gguf (Q8_0, 3-way split, ~8.1 GB) |
Qwen2.5-7B-Instruct-rk3588-w8a8.rkllm (W8A8) |
| Weight residence |
ORK_PERSIST=~/qwen2.5-7b-q8.orkpack (ORKPK01 v4) |
rkllm-internal (7.23 GB) |
| Invocation |
ORK_DIRECT=1 ORK_PERSIST=…orkpack build/bin/ork_bench <gguf-shard1> ~/bench_prompt_long.txt 256 8 256 — NO sudo
|
sudo env LD_LIBRARY_PATH=~/rkbench rkllm_bench_new <rkllm> 8 ~/bench_prompt_long.txt |
P (prefill tokens) — CRITICAL |
256 (== ubatch → one full ubatch, avgM=256) | 485 |
| Governors | CPU cpu4-7 + DDR = performance; NPU = rknpu_ondemand (do NOT set NPU performance — breaks wide-K submit, errno 110) |
same CPU/DDR |
| Reps | single pass (warm-dominated); reproduced 71.89 (2026-07-29) → 73.03 (2026-08-03) | 1 |
| Prefill | 73.03 tok/s (256 tok, 3505 ms) | 75.34 tok/s (485 tok) — ork ≈ 97 % |
| Decode | 1.58 tok/s (forced-NPU; prod runs decode on CPU ~13) | 3.96 tok/s |
| Residency | 10.07 GiB / 8 domains, churn = 0 (the lean no-sudo layout — the FAST one) | 7.23 GB |
-
Benchmark at
P == ubatch(256), not 485. P=256 is one full ubatch (avgM=256, single clean submit sequence) → 73. P=485 splits into 256 + a partial 200 (avgM=228); the inefficient 2nd partial batch drops it to ~59. Prefill t/s is very sensitive to P-vs-ubatch alignment. -
Run WITHOUT
sudo→ the lean 10.07 GiB residency. That is the fast layout. Running undersudoforces the 12.26 GiB layout and is slower (~59). (Earlier revisions had this exactly backwards.)
Other confounds that prolonged the saga (all measurement error, NOT code — ggml-ork.cpp is byte-identical across trees):
-
Stale ork-driver v0.3.3 in
~/llama.cpp/diverged-history(predates the prefill lever stack;n_mm=498vs664, −14 t/s). Build from~/llama.cpp-orkd(0.7.x). Alwayslddthe bench + check the ork-driver version. -
sudo rebootsilently no-ops (systemd-bus error, rc=0) → 27 h of stranded IOVA depressing every run. Fix: SysRqs,u,b, verifyuptime -schanged. See memoryboard-reboot-and-single-stream. -
NPU governor =
performancewedges the wide-K ffn_down submit (errno 110). Keep itrknpu_ondemand.
Submit path — blocking mcworker, NOT the doorbell (measured 2026-08-03). M>1 int8 prefill runs on the blocking per-core DRM_IOCTL_RKNPU_SUBMIT (mcworker), and a clean same-boot A/B at P=256 shows this is faster, not just safer: native/blocking = 73.83 t/s vs doorbell (ORK_COLSPLIT_MGT1+ORK_COLSPLIT_PARALLEL) = 45.94 t/s (both EXIT=0, no errno-110, churn=0). The nonblock doorbell's only edge is freeing the CPU to overlap the NPU, but prefill has nothing to overlap (act-quant is 3%, weight load is one-time — see the [ork ACTQ] note below), so the doorbell only adds host-poll + parallel-pool + colsplit N-split overhead. The doorbell spine is for decode/M=1/int4, not prefill; the parity-doc's "the parity number is a doorbell number" is wrong.
The "36% quant" is not a lever (measured 2026-08-03). [ork ACTQ] decomposes t_quant = arith 3% + resolve 25% + misc 0%, and resolve is entirely load(.orkpack) (2709 ms) — the one-time weight load, in-window only because ork_bench loads-then-prefills-once (churn=0 → no per-prefill reload; a persistent server amortizes it). Real per-token CPU quant is 3% → the NPU t_run is the floor. So the 73 t/s single-shot is suppressed by the in-window load; warm/steady-state prefill is >73 (a warmup pass / eager pre-load moves the load out of the timed window).
On DRAM bandwidth (earlier "the gap" claim, now corrected): at the wrong P=485 config ork looked bursty (~26 % DMC vs rkllm 44 %) and I wrongly called bandwidth "the gap." At the correct P=256 config ork is ~96 % of rkllm, so bandwidth is at most a small residual — not the headline.
Warm token-matched A/B (2026-08-03) — both warm, and the gap is LENGTH-DEPENDENT. Both sides are warm/load-excluded: the in-tree ork_bench runs an untimed warmup prefill + memory_clear before the timed one (so the .orkpack load is out of the timed window — the [ork ACTQ] "25% resolve/load" is the warmup's one-time load, counted only because ORK_PROFILE is cumulative over the whole process); rkllm's RKLLMPerfStat.prefill_time excludes its init-time model load.
| Prefill length | ork (warm) | rkllm (warm) | ork/rkllm |
|---|---|---|---|
| ~200–256 tok | 73.0 | 76.4 | ~96 % |
| ~456–485 tok | 65.2 | 75.3 | ~87 % |
rkllm is length-flat (76.4@202 → 75.3@485, −1.4 %); ork declines with length (73.0@256 → 65.2@456, −11 %). So it's not a fixed "% of rkllm" — ork is ~96 % (near parity) at short prefills, ~87 % at long. Root cause is structural: ork runs attention on the CPU (flash-attn, O(M·context) → grows with prompt length) while rkllm runs it on the NPU (flat); the partial-M-tile penalty compounds at non-256 lengths. The remaining prefill lever is CPU-attention scaling — NOT quant (3 %) and NOT the doorbell (46 < 74, blocking wins).
Sampler: ( while :; do cat /sys/class/devfreq/dmc/load; sleep 0.1; done ) in background; parse N@freqHz → avg/peak of the N (load %). See tools/bench_monitored.sh for the full multi-resource version.
The canonical apples-to-apples that the Benchmark Standards reference cites. Open stack beats the closed runtime on decode at this size.
| open stack | librkllmrt | |
|---|---|---|
| Model | Qwen3-1.7B, Q8_0 (ork) / matching .rkllm w8a8 |
v1.2.3 |
| Tool | llama-bench |
rknpu_bench |
| Threads | -t 4 |
— |
| Prefill | 178 tok/s (97 %) | 184 tok/s |
| Decode | 12.8 tok/s (109 %, beats) | 11.71 tok/s |
| State | warm, ≥2 reps | warm |
Note the size inversion vs BR-1: at 1.7B the open stack is at parity/ahead; at 7B it falls to 66–70 %. The delta is the bandwidth-utilization wall (BR-1) scaling with weight size.
Update · 2026-08-06 · ork-driver v1.0.7 — open-side re-measure (new record). Re-ran the open side at the same config after the #45 mcworker removal + the int4-doorbell (BCHAIN) work, to validate no regression. Provenance: ork-driver v1.0.7 (267101b, code 7140726; llama.cpp-rockchip master f5a28e8 carries this submodule); llama-bench built from ~/llama-orkasync (feat/ork-async-stream, direct-default frontend — ork_npu_init()); -m qwen3-1.7b-q8_0.gguf -p 512 -n 128 -r 2 -t 4; DIRECT transport (log line [ork] routing: direct in-process lib); governors=performance; single-stream.
| open stack | Prefill (pp512) | Decode (tg128) |
|---|---|---|
| v1.0.7 · 2026-08-06 (current record) | 226.10 ± 0.07 | 13.25 ± 0.00 |
| 2026-06-13 (original BR-2 reference, kept) | 178 | 12.8 |
Prefill 178 → 226 is the cumulative prefill-lever stack landing since June (#37 pipelined accumulate, #38/#39 rkllm-schedule port, #43 doorbell-fold); decode 12.8 → 13.25, still ahead of the librkllmrt reference (11.71). The #45 mcworker removal is perf-neutral for int8 (colsplit was already the default path; mcworker was dead fall-through) — confirmed end-to-end, no regression. rkllm 1.7B side NOT re-measured this pass: the only board .rkllm is Qwen2.5-7B, and its rkllm-toolkit 1.2.3 format needs runtime ≥1.2.3, but runtime 1.2.3+ can't malloc NPU memory on the board's rknpu 0.9.8 kernel (errno 22, flags 0xa) — a version-triangle wall, not a code issue; the 184/11.71 reference above stands. The original 2026-06-13 row is retained for the progression.
For matmul/kernel questions (not full-model). Closed reference is the RKNN matmul API, single-core, identical (M,K,N)+dtype.
| Item | Value |
|---|---|
| Tool | make rknn_vs_ork RKNN_DIR=/tmp/rknn && sudo env LD_LIBRARY_PATH=/tmp/rknn ./rknn_vs_ork [iters] [a] |
| Core count | pin ork to 1 core (RKNN API rejects the 3-core mask); show multi-core separately |
| Correctness | dummy data — does NOT validate output; pair with make quant (RMS check) |
| Alt tool |
make mc_prof && sudo ./mc_prof [M] [K] [N] [iters] (per-core copy/submit/accumulate breakdown) |
Why it exists: prefill is weight-DMA-bound; the single per-matmul K×N weight stream is amortized over the rows in one M-tile, whose cap is mg_max*64 (704 @ Kp=512, 320 @ Kp=1024 — npu.c). So t/s rises with M up to the tile cap, then plateaus (past the cap the matmul splits into multiple full tiles, each re-streaming the weight). This is the curve behind "why P=256": ubatch sets the tile ceiling, and P must fill it. Path = mcworker 3-core chain-prefill with resident Bf (see BR-1). Points below are P == ubatch == M (one full batch), direct path, ORK_DIRECT=1, no sudo, governors=performance, NPU gov=rknpu_ondemand.
Qwen3-1.7B-Q8_0 — controlled single-config sweep (qwen3-1.7b-q8_0.orkpack, 0.7.x, one boot), 2026-08-03:
| M (=ubatch=P) | prefill t/s | avgM |
|---|---|---|
| 64 | 168.2 | 64.0 |
| 128 | 205.5 | 128.0 |
| 192 | 220.2 | 192.0 |
| 224 | 226.9 | 224.0 |
| 256 | 232.6 ← peak | 256.0 |
| 288 | 191.0 ← dip | 288.0 |
| 320 | 190.8 | 320.0 |
| 512 | 190.7 | 456.0 |
Same signature as 7B: steep rise to a sharp peak at M=256, a cliff at 288 (−18 %), then a flat plateau (288/320/512 ≈ 191).
⭐ Cross-model conclusion (the important one): the falloff is an M-tile-scheduling sawtooth that tracks each model's own tile size — not attention, and NOT "model-independent" (the two models are tiled differently). The M-tile is chunk = mg_max*64, keyed on the matmul's K (npu.c:4423 = the mc-cap table): 704@Kp512 · 320@Kp1024 · 128@Kp2048 · 64@Kp3584. So the attn/gate/up matmuls tile at chunk 64 on 7B (K=3584) but chunk 128 on 1.7B (K=2048) — different granularity.
Both peaking at 256 / dipping at 288 is partly a divisibility coincidence: 256 is a clean multiple of both 64 and 128 (4× / 2×), and 288 is a partial of both (4.5× / 2.25×). The decisive discriminator is M=320:
- 320 = 5×64 → clean for 7B's 64-tile → 7B recovers (288→64.9 → 320→66.5).
- 320 = 2.5×128 → partial for 1.7B's 128-tile → 1.7B stays dipped (288→191.0 → 320→190.8, flat).
The dip recovers exactly when M is a clean multiple of that model's tile — attention can't produce that (it doesn't know 64 vs 128). So: a partial-M-tile scheduling penalty (a partial final sub-tile pays a whole tile's overhead), period = the model's chunk. Practical rule: ubatch=256 — it's a clean multiple of every chunk in play (64/128/320) and the amortization/attention sweet-spot, so it's the peak across the family. The partial-tile penalty is the same one every non-multiple prompt-tail pays; the standing driver lever is to schedule a partial M-tile without wasting a full tile. (Confirmable by tile-variation — force a different chunk via ORK_KTILE on one model and watch the dip move; and by the step-16 fine sweep mapping the sawtooth period.)
Qwen2.5-7B-Q8_0 — controlled single-config sweep (qwen2.5-7b-q8.orkpack, 0.7.x ork-driver, clean boot per point), 2026-08-03:
| M (=ubatch=P) | prefill t/s | avgM | resident |
|---|---|---|---|
| 64 | 37.4 | 64.0 | 10.07 GiB |
| 128 | 65.9 | 128.0 | 10.07 GiB |
| 192 | 71.0 | 192.0 | 10.07 GiB |
| 224 | 71.4 | 224.0 | 10.07 GiB |
| 256 | 72.8 ← peak | 256.0 | 10.07 GiB |
| 288 | 64.9 ← dip | 288.0 | 10.07 GiB |
| 320 | 66.5 | 320.0 | 10.07 GiB |
| 512 | 65.2 | 456.0 | 10.07 GiB |
Shape: steep rise 64→256 (37→73), sharp PEAK at M=256, then a NON-MONOTONIC dip. Not a smooth decline — 288 (64.9) is below both 256 (72.8) and 320 (66.5). Regimes:
- M < 256 — weight under-amortized. The one K×N weight stream is spread over too few rows (64 → 37 t/s). Rises steeply, flattening by ~224–256.
- M = 256 — genuine local optimum, with penalties on both sides (not a plateau).
- M > 256 — alignment/tile-scheduling penalty. 256 (4×64) and 320 (5×64) are clean multiples of the 64-row sub-tile; 288 (4.5×64) lands mid-tile and craters to 64.9 — a partial-sub-tile penalty, not attention. 320 recovers to a clean multiple but is still below 256 (the extra 64 rows add more CPU flash-attention, O(M·context), than M-tile benefit). 512 (avgM=456) splits past the 320 tile → 65.2.
Fine sweep 240→320, step 16 (7B, 2026-08-03) — the sawtooth, resolved:
| M | 240 | 256 | 272 | 288 | 304 | 320 |
|---|---|---|---|---|---|---|
| M mod 64 | 48 | 0 | 16 | 32 | 48 | 0 |
| t/s | 62.6 | 71.4 | 62.3 ←trough | 65.0 | 64.1 | 66.6 |
Mechanism (7B tiles at chunk 64, K=3584): every M-tile is ≤64 rows and re-streams the weight (weight-DMA-bound). So:
- A partial (non-×64) M is penalized to ~62–65 regardless of tile count — 240 (4 tiles, 48-full partial) = 62.6 ≈ 272 (5 tiles, 16-full partial) = 62.3. It's the partial tile, not the count.
- 272 is the trough — its 5th tile is emptiest (16/64 rows, 48 wasted); as the partial fills (288→304→320) t/s climbs back.
- Only clean ×64 avoids the penalty: 256=71.4, 320=66.6. Among clean M, 256 (4 tiles) beats 320 (5 tiles) — each extra tile costs ~5 t/s (one more weight re-stream). Broad clean-peak at 192–256.
Chunking / long-prompt (512-token prompt, 2026-08-03): a true clean M=512 (8-tile) = 67.5 t/s — beats the misaligned M=456 (65.2) despite more tokens (512 = 64 rows/tile clean vs 456 = 57 rows/tile under-filled), confirming the tiling model. Chunking as 2×M=256 = 69.2 t/s beats one M=512 (67.5) — so ubatch=256 (tile-aligned chunking into clean 4-tile peak batches) is the right serving policy. Both 512-token configs (~68) sit below the 256-token 73, and that residual is attention (O(M·context)), not tiling — even perfectly-tiled 2×256 can't dodge batch-2's longer KV.
Takeaway: serve/benchmark 7B prefill at ubatch=256 — the clean-×64 sweet spot (4 full tiles), and it auto-chunks any prompt into clean peak batches. Partial M-tiles cost ~10–13 % (71→62), so every non-64-aligned prompt tail pays it; the frontend uses M = ubatch-tokens as-is (no auto ×64 alignment), so a driver-side pad of the final partial tile is the minor open refinement. Quantified driver lever: pad/reschedule the partial M-tile so it doesn't eat a full tile's weight-restream → recovers ~10 % on arbitrary prompt lengths (the same penalty behind the P=485→59 two-batch result). (ORK_KTILE can't test this on a resident pack — it's pack-time, a no-op under ORK_PERSIST: churn=0, unchanged. A true within-model tile-variation needs a repack.)
Method:
ORK_DIRECT=1 ORK_PERSIST=…qwen2.5-7b-q8.orkpack build/bin/ork_bench <gguf-shard1> ~/bench_prompt_long.txt <M> 8 <M>(from~/llama.cpp-orkd, no sudo), one clean SysRq-s,u,bboot per point, 2026-08-03. The 456-token prompt caps actual tokens at 456, so M=512 runs ~456 (past the 320 tile → internal split, avgM shown).
Question: the default prefill is a hybrid — attention (r_base: Sn==1, K≤4096, single Bf) on the nonblock doorbell colsplit, FFN (wide-N gate/up, wide-K down) on the blocking mcworker. Can the FFN also move to the doorbell (the "COMPLETE MIGRATION" goal) without losing speed?
A/B (P=256, warm, one clean boot):
| Config | Prefill t/s |
|---|---|
| native (attn=doorbell, FFN=mcworker) | 73.8 |
all-doorbell (ORK_COLSPLIT_MGT1=1 ORK_COLSPLIT_PARALLEL=1 → FFN also on colsplit) |
41–46 |
Moving the FFN onto the doorbell colsplit is ~1.6× slower. It runs cleanly (no wedge, EXIT=0, churn=0) — it's a speed regression, not a correctness one.
Profile — the gap is in t_run (NPU-execution phase), not poll, not submit-count:
| Metric (P=256, 332 matmuls, avgM=256) | native | all-doorbell |
|---|---|---|
t_run (NPU phase) |
5.3 s | 8.9 s (+66%) |
| µs/row | 62 | 105 (1.66×) |
| total submit-ioctls | 1663 | 990 (fewer) |
| total programs | 9319 | 7251 (fewer) |
| wall per ioctl | 3989 µs | 8082 µs (2.0×) |
NPU busy peak (rknpu/load) |
34% | 22% |
| NPU %idle (<1%) | 56% | 48% |
Conclusion: the doorbell issues fewer ioctls with more programs each (chaining is already implemented in ork_dyn_begin_colsplit, npu.c:11478 — identical structure to mcworker_pref_ksplit: gather → PC-chain Sk×M programs → 1 submit/core → host-accumulate). Yet each submit is 2× slower and the NPU peaks lower (22% vs 34%). So it's not poll overhead (that'd be idle-wait, not t_run), not more submits, and not more work (peak is lower, not higher). The nonblock doorbell submit under-engages the NPU — it dribbles the chained tasks at low intensity over more wall-time, vs the blocking ioctl where the kernel drives the chain start-to-finish (per-task IRQ re-arm, tight pipelining). Matches the 2026-07-22 "async wide-K job barely engages the NPU" note. Caveat: rknpu/load is coarse (native peaks only 34%, doesn't resolve per-submit NPU-active windows) — the peak + t_run support under-engagement but don't prove the exact sequencing fault; a per-submit ORK_DYN_TRACE is needed for that.
Root-cause drill-down (built-in counters, ORK_DYN_TRACE+ORK_MC_DIAG, 2026-08-03):
| Run | config | prefill | MC-RECOVER | INCOMPLETE | all-done |
dyn_end elapsed |
|---|---|---|---|---|---|---|
| A | parallel doorbell | 45.6 | 0 | 0 | 332 | ~1 µs (prepolled) |
| B | parallel, ping-pong OFF (ORK_NO_PINGPONG=1) |
46.4 | 0 | 0 | — | — |
| C |
serial doorbell (no COLSPLIT_PARALLEL) |
timed out | 6 | 1 | 4 | avg 604 ms, max 3.0 s |
This refutes every completion-side suspect for the parallel path: A has 0 miss / 0 recover / all 332 all-done / 1 µs ork_dyn_end — so it's not poll, not miss-timeout, not recover-loop, and not ping-pong (B≈A). (The serial path C is genuinely broken — 6 recoveries, 3 s stalls, times out — the old pathological pre-parallel poll; never use it.)
Mechanism — NARROWED to per-core job construction; NOT the submit path (all submit-side suspects ruled out). ork_csub_worker (npu.c:11439) clears the nonblock flag — the parallel colsplit does a BLOCKING submit on 3 pool threads, identical to the mcworker (same concurrency, same kernel-wait). And mtile_cap (npu.c:11052) is the same mg_max*64 formula as the mcworker's chunk (same M-tile sizes). So the 1.6× is NOT: poll (1 µs), miss/recover (0), ping-pong (B≈A), nonblock, 3-core concurrency, or M-tile size — all measured/read out. ROOT CAUSE (regcmd byte-diff via the synth_i8 call sites, 2026-08-03): the wide-N (gate/up) output surface NOTCH. synth_i8's stride arg sets the DPU output-stage regs RK_DPU_DST_N_DIMS = ((s-1)<<16)|(N-1) and RK_DPU_DATA_CUBE_NOTCH = ((s/4-1)<<16)|(N/4-1) where s = stride>0 ? stride : N (npu.c:855). For the wide-N matmuls (gate/up, N=18944 — the dominant FFN op):
-
mcworker (
_pref_chain, npu.c:4498) passes stride0→s=N=Nsub → notch = 0, contiguous DPU writeback (dense[mco][Nsub]; column placement done host-side after). -
colsplit (
ork_dyn_begin_colsplit, npu.c:11540) passes strideNcore→s=Ncore, but writes onlyNcol<Ncorecols/row → notch = Ncore−Ncol, strided DPU writeback (a gap every row, placing columns directly into the core's[M][Ncore]layout ON the NPU).
A notched/strided DPU output (partial-burst writes + per-row skip) is ~2× slower than contiguous — that's the 2× wall/ioctl, and gate/up being the biggest FFN matmul drives the ~1.6× overall. ffn_down (wide-K) is identical on both paths (stride=0 both) — so the delta is wide-N only. Fix: make the colsplit write contiguous (notch=0) into a compact per-program scratch + scatter host-side (like mcworker), or split each core's N-range on N-slice boundaries so Ncol==Ncore. (Confirm empirically with ORK_MM_DUMPRC — it dumps the 0x40xx output stage — and/or a contiguous-vs-notched output micro-bench.)
Takeaway: keep prefill FFN on the blocking mcworker (it ships and is fastest). The doorbell's gap is a job-construction difference in the colsplit, not the submit mechanism — and closing it is a robustness/unification play only (even matched, prefill is NPU-bound so no speed gain). (Earlier revisions of this note blamed poll overhead, then nonblock under-engagement, then 3-core concurrency — all three were wrong; the counters/code retired them in turn. Recorded honestly so the next agent starts from the regcmd-diff, not those dead ends.)
Full reproduce:
# board 10.3.0.236; one clean boot (sudo reboot no-ops in ssh — use SysRq, verify uptime -s changes):
sudo sh -c 'echo 1 > /proc/sys/kernel/sysrq; echo s > /proc/sysrq-trigger; echo u > /proc/sysrq-trigger; echo b > /proc/sysrq-trigger'
# after reboot:
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
echo performance | sudo tee /sys/class/devfreq/dmc/governor # NPU gov stays rknpu_ondemand (performance wedges wide-K, errno 110)
sudo mount -t debugfs none /sys/kernel/debug # for /sys/kernel/debug/rknpu/load
cd ~/llama.cpp-orkd # 0.7.x ork-driver; llama fork feat/ork-async-stream (ggml-ork.cpp == bea5e49ea)
# NATIVE (baseline):
ORK_DIRECT=1 ORK_PROFILE=1 ORK_PERSIST=/home/michael/qwen2.5-7b-q8.orkpack \
build/bin/ork_bench /home/michael/qwen2.5-7b-instruct-q8_0-00001-of-00003.gguf \
/home/michael/bench_prompt_long.txt 256 8 256 # NO sudo (sudo→12.26 GiB slower layout)
# ALL-DOORBELL (add the two knobs, same command):
ORK_COLSPLIT_MGT1=1 ORK_COLSPLIT_PARALLEL=1 ORK_DIRECT=1 ORK_PROFILE=1 ORK_PERSIST=… build/bin/ork_bench … 256 8 256
# NPU-load sampler (run as root, concurrently, 20ms): parse "Core0: X%, ..." -> per-sample core-mean
sudo sh -c 'while :; do cat /sys/kernel/debug/rknpu/load; sleep 0.02; done' > /tmp/nl.log # kill when the bench exits
# profile lines to read: "[ork DUMP] ... t_run=..." "ork profile: prefill (M>1) ... us/row" "[ork PROFILE] submits: N ioctls, M programs" "[ork FD-SPLIT] ... per-ioctl wall"Notes: ORK_PROFILE prints the [ork DUMP]/ork profile:/[ork PROFILE]/[ork FD-SPLIT] lines on backend free (cumulative over warmup+prefill+decode). hw_elapse in [ork FD-SPLIT] is a dead field (reads garbage %) — ignore its poll/idle split; use t_run and the rknpu/load peak instead.
| Tool | Where | Args | Measures |
|---|---|---|---|
ork_bench_feat |
board ~/rkbench/
|
<gguf> <prompt_FILE> <P> <G> <ubatch> |
ork full-model prefill+decode. Arg 2 is a FILE path, not a string. |
rkllm_bench_new |
board ~/rkbench/
|
<rkllm> <G> <prompt_FILE> (needs LD_LIBRARY_PATH + sudo) |
rkllm full-model prefill+decode |
llama-bench |
board ~/llama.cpp/build
|
-p <pp> -n <tg> -t 4 |
standard warm+repeated prefill/decode (preferred for methodology-clean numbers) |
rknn_vs_ork |
make on board |
[iters] [a] |
per-matmul ork vs closed RKNN API |
mc_prof |
make on board |
[M] [K] [N] [iters] |
per-core copy/submit/accumulate |
bench_monitored.sh |
tools/ |
--label L -- <cmd> |
wraps any workload, samples RAM/DMC-BW/NPU/GPU/CPU avg+peak |
| DMC one-liner | inline |
cat /sys/class/devfreq/dmc/load (N@freqHz) |
DRAM load % (the BR-1 A/B sampler) |
-
Multi-domain layout is non-deterministic. The same 7B binary rolls 10.07 GiB/5-dom (52 t/s) or 12.26 GiB/8-dom (48 t/s) across boots. Always report the layout (
[ork RESIDENT] final: … GiB across N domains) with the number. Extreme historical swings: 10 GiB→46 t/s vs 19 GiB→4.3 t/s. -
One clean NPU run per boot. Cross-run IOVA is not freed. A 2nd 7B run this boot (ork or rkllm) can fail
init/allocation. Symptom: rkllmrkllm_init failedright after an ork run. rkllm-first on a shared boot if you must, or reboot between arms. -
CMA is a red herring for residency.
ORK_PERSISTimports weights via dma-bufSCM_RIGHTSand maps them into IOVA (8 IOMMU domains × ~4 GiB), not CMA. The 256 MB CMA default does not gate a 10–18 GB resident model. The real fragility is stranded IOVA (gotcha 2). -
ork_benchmmap eviction message is cosmetic underORK_PERSIST. "cannot be used with preferred buffer type ORK_Weights … using CPU instead (N others)" fires because mmap evicts the host-buft ggml tensors — butORK_PERSISTloads the NPU-resident copy independently. Trustchurn = 0+ "loaded 139 from disk", not that line. -
Transport default flipped back to DIRECT (2026-08-06, ork-driver 1.0.4 / llama.cpp-rockchip master
6a8dfe7a9). Direct (in-process lib) is now the DEFAULT — no env needed — andORK_DIRECThas been REMOVED. orkd is opt-in viaORK_USE_ORKD=1. Transport is chosen by which init the backend calls (ork_npu_init= direct,ork_npu_init_orkd= daemon). Confirm with the run-log line[ork] routing: direct in-process lib (single-stream). TheORK_DIRECT=1in older invocations below is now a harmless no-op (the command still runs direct) — you can drop it. (History: the default had briefly flipped to orkd viabea5e49, which is what the pre-2026-08-06 notes describe.) -
ubatch matters for prefill. BR-1 uses
ubatch=256; smaller ubatch = smaller M-tile = worse weight amortization = lower prefill (weight-DMA-bound). -
Quant paths differ across runtimes. ork = Q8_0 (GGUF); rkllm = W8A8 (
.rkllm). Same family/size but not bit-identical weights — a fair throughput comparison, not an accuracy one. - Single-pass vs warm+reps. BR-1 is a single pass (no discarded warmup, no averaging) — a floor for that boot, not a methodology-clean canonical number. Within a 456-tok prefill the NPU is warm after the first submit (amortized over thousands), so it's warm-dominated, but re-run warm + ≥2 reps for a canonical row.
At capture time the Mac-side ork-driver submodule was at 9628a45 (feat/attn-biased-rr) while the board's ~/ork-driver that actually built the running lib was f446052 (v0.7.6), and the llama fork was bea5e49ea (feat/ork-async-stream). The running binary reflects the board commit. Always record the board's ~/ork-driver HEAD (what built the lib) — not just the Mac submodule pointer — since sync drift is common during active work. Reconcile the submodule pointer before citing a number as the tree's official result.
The perf validation for the fp16 CONTIG colsplit fix (drop prevented, default-on). Matmul-level (not end-to-end — see the LLM-bench blocker below).
| Item | Value |
|---|---|
| ork-driver |
main commit 44ffa40 (CONTIG default-on, is_valid_dma_addr +Bbc, adaptive detect + recovery net) |
| Tool |
examples/test_f16colsplit.c — ./test_f16colsplit M K N iters (PROBE mode; nc=1 ref + nc=3 A/B, best-of-iters µs) |
| A arm (colsplit) | default (ORK_F16_COLSPLIT now default-on → CONTIG single-buffer) |
| B arm (mcworker) |
ORK_F16_NO_COLSPLIT=1 (legacy blocking per-core mcworker) |
| Governors | CPU cpu4-7 + DDR = performance (verified dmc/governor=performance) |
| Board | RK3588 10.3.0.236, single-stream, fresh boot |
| Correctness | every arm BIT-EXACT vs the nc=1 reference (maxerr=0, diff=0) |
Result (nc=3 µs/run, best of 30; lower = faster):
| shape (M K N) | CONTIG colsplit | mcworker | colsplit speedup |
|---|---|---|---|
| 256 3584 3584 | 27910 | 28969 | 1.04× |
| 256 3584 512 | 5397 | 6262 | 1.16× |
| 256 2048 512 | 3169 | 3897 | 1.23× |
| 128 3584 3584 | 13853 | 14460 | 1.04× |
CONTIG colsplit beats mcworker on every shape, bit-exact — the gate for un-gating fp16 colsplit (bit-exact AND ≥ mcworker t/s) passes. Reliability: CONTIG 1000-iter @256×3584×3584 = 0 drops / 0 wrong / 0 panic (the per-slice path dropped on iter 0). make test all-PASS with colsplit default-on.
End-to-end LLM llama-bench (2026-08-05). Built llama-bench+llama-cli from feat/ork-async-stream (~/llama-orkasync, -DGGML_ORK=ON, links current ork-driver main). REPLICATION NOTE (updated 2026-08-06): as measured on 2026-08-05 this build defaulted to the orkd daemon and ggml_backend_ork_init failed hard without a running daemon, so the run below set ORK_DIRECT=1 to force the in-process lib. That default has since been reversed (ork-driver 1.0.4 / master 6a8dfe7a9): direct is now the default and ORK_DIRECT is gone — a current build just works with no env, and ORK_USE_ORKD=1 is the opt-in for the daemon path. On a current build the ORK_DIRECT=1 below is a harmless no-op; drop it.
Model qwen3-1.7b-f16.gguf (F16, 3.21 GiB), -t 4 -p 512 -n 128 -r 2, backend=ORK, governors=performance, ORK_DIRECT=1, single-stream fresh boot:
| arm | pp512 (prefill) t/s | tg128 (decode) t/s |
|---|---|---|
| CONTIG colsplit (default) | 225.5 ± 0.9 | 7.48 ± 0.00 |
mcworker (ORK_F16_NO_COLSPLIT=1) |
224.2 ± 1.0 | 7.48 ± 0.00 |
End-to-end prefill is ~parity (225.5 vs 224.2, within noise): the per-shape matmul wins (BR-6 table, 1.04–1.23×) net to a wash across this model's matmul mix, and decode is identical (M=1 → single-core, colsplit doesn't apply). So the CONTIG colsplit's end-to-end value is safety, not raw speed — the full model run drove hundreds of fp16 matmuls with 0 wedge / 0 panic, real-world confirmation the drop is prevented, making the doorbell colsplit a safe default at prefill parity. A raw-speed win would need shapes skewed toward the wide-N/small-N cases where colsplit leads.
Reproduction · 2026-08-06 · ork-driver v1.0.7 (267101b) — validation, NOT a new record. Re-ran the same config after #45 removed the blocking mcworker fallback (colsplit is now the sole fp16 multicore path; the ORK_F16_NO_COLSPLIT arm above no longer exists — the fp16 fallback is the single-core reference via ORK_RC_F16_SC). Build: ~/llama-orkasync (feat/ork-async-stream, direct-default), qwen3-1.7b-f16.gguf, llama-bench -p 512 -n 128 -r 2 -t 4, DIRECT ([ork] routing: direct in-process lib), governors=performance; JIT-pack at load (no ORK_PERSIST). Result: pp512 222.17 ± 0.77 / tg128 7.43 ± 0.01 — within noise of the 225.5 / 7.48 record (prefill −1.5 %, decode −0.7 %), on a heavily-used (non-fresh) boot. No regression from the mcworker removal (the fp16 colsplit default path is byte-identical to the record's; #45 only deleted the dead fallback). The 225.5 / 7.48 fresh-boot number stays as the record; this row is the 1.0.7 validation point. (First 1.0.7 attempt failed with bcreate Bb[0] — a stale/partial auto-discovered <model>.orkpack; moving it aside so JIT-pack runs reproduced the number. Record with a clean pack or none.)
int4 via ORK_QUANT=4 on route B: compact int4 storage (i4a8 .orkpack) + W8A8-inflate compute
(int4 weights inflate int4→int8 and run on the robust int8 kernel — coherent, no wedge). This is what
ORK_QUANT=4 now selects by default; native W4A4 (single-row, prefill-wedge-prone) is opt-in only
(ORK_MIXED_W4A4/ORK_HADAMARD). Route B is the fix that un-wedged the P=128 int4 prefill (native W4A4 grouped
soft-reset-stormed at M=128; the int8 kernel handles M=128 fine).
Provenance: ork-driver 20ac059 (main), llama.cpp-rockchip 91e723b94 (diverged-history); harness =
ork_bench (one clock, P=128 G=64), NOT llama-bench — so these are single-run ork_bench numbers, not the
standardized pp512/tg128 of BR-2 (re-run under llama-bench -r 2 for a record-grade claim). DIRECT transport,
governors=performance, single-stream; reads the auto-persisted compact .q4.orkpack in READ mode (loaded 193, packed 0 — no JIT-pack in the timed window). Coherent output confirmed.
| arm (Qwen3-1.7B, ork_bench P=128 G=64) | prefill t/s | decode t/s |
.orkpack size |
|---|---|---|---|
| int4 NF4 — built from F16 (unquantized) source | 215.0 | 6.77 | 674 MiB (~3.4× < int8) |
| int4 uniform — built from Q8_K_XL source | 171.8 | 2.96 | 674 MiB |
int8 W8A8 (reference, BR-2 llama-bench) |
~178 | ~13 | 2.3 GiB |
NF4-from-F16 is both faster and higher-fidelity than uniform-from-Q8 (+25 % prefill, 2.3× decode). The codebook
is routed by the SOURCE, not an env flag: a full-precision (F16/F32/BF16) source auto-selects the NF4 non-uniform
codebook; an already-quantized source falls back to the lossier uniform int4 (and the build emits a warning steering
you to regenerate from unquantized weights). The F16 arm's prefill (215) even edges the int8 reference (~178) —
int4's smaller resident weights ease DMA/IOVA pressure while the compute stays int8-exact; the int4 win is storage
(~⅓ of int8), not compute (int8 speed). Recommended int4 setup: build the 0.7**, W8A8-inflate, verified on the F16 pack with .orkpack from the model's F16 GGUF.
PPL (wikitext-2, **20 chunks — noisy, ±quant_kind=1=NF4):
int4 NF4-from-F16 = 16.40 vs int8 10.67 (+54 %). The gap is dominated by ork's per-output-channel weight
scale (one scale over all K), which the NF4 codebook does NOT address — NF4 improves the weight-quant error
(~0.70× uniform, per test_matmul) but not the scale granularity; finer per-K-block scales (Q4_K-style) would
close more. (A uniform-from-Q8 PPL was not separately measured; a full-wikitext run would tighten these.)