Skip to content

Benchmark Standards and Methodology

Jump to bottom
Michael Fischer edited this page Oct 13, 2026 · 5 revisions

Benchmark Standards & Methodology

Every performance number in this wiki — and every claim in code review, commit messages, or chat — must come from the standardized benchmark below. No number from memory, extrapolation, a different model, or a different code path. This page is the canonical procedure; each Experiment Log entry carries a benchmark run to this standard.

Why this exists

The rule was written after repeated over-optimistic claims that didn't survive a fair test. Two failure modes recur:

  1. Mixed configs (apples-to-oranges). A claimed "ggml-ork prefill 280 beats librkllmrt's 181" was actually a smaller 0.5B model on ggml-ork compared against a 1.7B model on librkllmrt — and the "decode match" was the standalone kernel loop, not the integration. Same-model, same-path measurement showed the closed runtime was 2.6× / 3.9× faster. See Open vs Closed.
  2. Measurement artifacts. Short prompts (10 tokens → avgM≈6) are nothing like real prefill (M=512). And the lazy one-time weight-pack lands inside the timed window unless warmup exercises the same M as the measured op. Both produced false prefill numbers until llama-bench (real warmup + repetition) was used. See Hybrid M-Gate.

The standard

  1. Always compare against the closed runtime — rkllm or rknn, whichever is applicable. This is mandatory, not optional: an open-stack number alone is meaningless without the closed reference on the same work. Pick the one that matches what you're measuring:
    • Full-model throughput (tok/s, an end-to-end experiment) → librkllmrt on the matching .rkllm via rknpu_bench. Same family + size + quant.
    • A single matmul / kernel-level cost (submit latency, tiling, a per-op question) → the RKNN matmul API (librknnrt, rknn_matmul_*) on the same (M,K,N) + dtype, via tools/rknn_vs_ork. rkllm has no matmul API, so this is the only true kernel-level apples-to-apples.
    • If neither closed path can run the exact thing (e.g. a probe the closed runtime doesn't expose), say so explicitly and give the closest closed reference + why it's not exact.
  2. Same model / same shape on both sides. ggml-ork: a GGUF (Q8_0). librkllmrt: the matching .rkllm. RKNN: identical (M,K,N) + dtype to the ork matmul. Same family + size + quant. Never compare across model sizes or paths.
  3. Use llama-bench, NOT short llama-cli prompts — it warms with a real pp512 batch and repeats. Short llama-cli prompts lie on prefill (unrepresentative M + lazy pack in the timed window). For matmul-level work use rknn_vs_ork (warm, N iterations).
  4. Verify governors at max BEFORE timing. Decode is memory-bandwidth-bound; a parked DDR governor ~halves it.
  5. Report BOTH prefill and decode (or, for matmul, the relevant M=1 and M≫1 shapes), for every arm, with ≥2 repetitions (mean ± stddev).
  6. Match the core count when comparing kernels. The RKNN matmul API is single-core (rejects 3-core mask) — pin ork to 1 core for the clean comparison and show multi-core separately.
  7. State the model, the path (integration vs standalone), and warm vs cold — never conflate.

Reference setup

  • Board: RK3588 @ 10.3.0.236. Model: Qwen3-1.7B-w8a8 (/tmp/gguf/qwen3_1.7b.gguf Q8_0 for ggml-ork; matching .rkllm v1.2.3 for librkllmrt at /var/lib/orkllm/models/...w8a8...).
  • Closed-runtime references (mandatory comparison):
    • Full-model: librkllmrt v1.2.3 — prefill 183.9 / decode 11.71 tok/s (rknpu_bench, same .rkllm, 2026-06-13).
    • Per-matmul: RKNN matmul API (librknnrt, headers + lib in /tmp/rknn) — see Matmul apples-to-apples.

Verify governors (must print performance + max freq)

cat /sys/class/devfreq/dmc/governor                        # -> performance
cat /sys/class/devfreq/dmc/cur_freq                        # -> 2112000000
cat /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor   # -> performance
cat /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq   # -> 2400000

If not pinned (oRKLLM's perf_governor does this automatically while a model is loaded):

echo performance | sudo tee /sys/class/devfreq/dmc/governor
for c in 4 5 6 7; do echo performance | sudo tee /sys/devices/system/cpu/cpu$c/cpufreq/scaling_governor; done

Run (ggml-ork)

sudo build/bin/llama-bench -m /tmp/gguf/qwen3_1.7b.gguf -p 512 -n 128 -t 8 -r 2

Run (librkllmrt — closed full-model baseline)

sudo /tmp/rknn/rknpu_bench /var/lib/orkllm/runtimes/librkllmrt-aarch64-v1.2.3.so \
  /var/lib/orkllm/models/Qwen3-1.7B-Base-rk3588-w8a8-normal-v1.2.3-RKLLM/Qwen3-1.7B-Base-rk3588-w8a8-normal-v1.2.3-RKLLM.rkllm 128

Run (RKNN matmul API — closed per-matmul baseline)

# ork-driver tools/rknn_vs_ork: same int8 (M,K,N) via ork_mm_run_i8 vs rknn_matmul_run
make rknn_vs_ork RKNN_DIR=/tmp/rknn
sudo env LD_LIBRARY_PATH=/tmp/rknn ./rknn_vs_ork 200

Env knobs for A/B experiments

The ggml-ork backend exposes knobs so each experiment's "on" and "off" arms can be measured on the same reference model. Hold everything else fixed; change one knob.

Knob Effect Used by
ORK_OFF=1 Force every matmul to CPU (pure-CPU baseline) all (baseline)
ORK_MINM=<n> Offload gate: matmuls with M≥n go to NPU (default 32). 1 = all-NPU; 99999 ≈ off Hybrid M-Gate
ORK_QUANT=4 int4: compact int4 STORAGE + W8A8-inflate compute (route B — coherent, no wedge; native W4A4 is opt-in via ORK_MIXED_W4A4/ORK_HADAMARD). RECOMMENDED: build the .orkpack from an F16/F32/BF16 source so the NF4 codebook auto-selects BR-7 (int4 P=128: NF4-from-F16 215/6.77 vs uniform-from-Q8 172/2.96), int4 W4A4
ORK_FUSE=1 Fuse adjacent same-input matmuls (QKV/gate-up) Fusion
ORK_NOREUSE=1 Disable activation-quant reuse Activation reuse
ORK_NOCACHE=1 Re-pack the weight every matmul (no NPU-resident cache) Weight caching
ORK_PROFILE=1 Print per-section timing (quant / NPU run / dequant; decode vs prefill; run_multicore phase split) Submit-floor, run_multicore

Note on ORK_NOCACHE: re-packing every matmul is pathologically slow (the pre-cache regime), so that one arm is benchmarked with reduced -p/-n and labelled as such — it demonstrates the order-of-magnitude effect, not a serving rate.

A/B matrix template

M=/tmp/gguf/qwen3_1.7b.gguf
run() { echo "### $1"; shift; sudo env "$@" build/bin/llama-bench -m $M -p 512 -n 128 -t 8 -r 2 | grep -E "pp512|tg128"; }
run CPU_baseline    ORK_OFF=1
run hybrid_default  ORK_MINM=32
run all_NPU         ORK_MINM=1
run all_NPU_fuse    ORK_MINM=1 ORK_FUSE=1
# ... one row per arm; change exactly one knob vs the control

Reporting format

Each experiment page reports a table with the arms compared and the control, always including prefill + decode:

Config pp512 (prefill) tg128 (decode)
arm under test … ± … … ± …
control … ± … … ± …

Profiling-only experiments (no tok/s comparison) report the ORK_PROFILE breakdown on the reference model instead, plus the standardized end-to-end numbers for context.

Board-validation attest: which board, and the per-board test_speed guards

The attest records the board (2026-10-13)

tests/sbc_attest.txt used to carry one CORE_SHA and nothing else, so it said a board validated the tree, not which. The two validated boards are not interchangeable — different DRAM type and clock, different kernel, different int4 dispatch envelope — so "ALL TESTS PASSED" on one is not validation on the other. 99771bc landed with "make test ALL TESTS PASSED on .235" in its message and broke every W4A4 inference on the ROCK 5B for six commits.

Each attesting board now adds a line keyed on /proc/device-tree/compatible[0] — product-level, stable, and the same file ork_soc_detect reads. IP address is environmental and the SoC id (rockchip,rk3588) is identical on both:

board compatible[0] DRAM kernel (2026-10-13)
10.3.0.236 Radxa ROCK 5B radxa,rock-5b LPDDR4X, DMC max 2112 6.1.115-vendor-rk35xx-sram
10.3.0.235 Radxa ROCK 5B+ radxa,rock-5b-plus LPDDR5, DMC max 2400 6.1.115-vendor-rk35xx-fix

make test rewrites only its own BOARD= line and carries every other board's through; make check-attest (CI, no NPU) is unchanged — it still compares CORE_SHA — and now also prints which boards hold that hash and which are behind. make check-attest-all is the opt-in full-coverage gate, with an expiring WAIVE=<board> sha=<hash> escape (make attest-waive). It is deliberately not in CI: the suite is ~75 s but the boards are shared and serialise agents, and a gate that cannot be satisfied is a gate nobody runs.

test_speed latency guards are per board

One constant could not serve both. At M=512 K=N=2048 int8 the ROCK 5B runs ~2.8 ms 3-core; the ROCK 5B+ runs slower despite LPDDR5, with a clean bimodal, per-process distribution — best-of-3 reps inside one run does not collapse it, so roughly half of all runs land in the slow mode. The 5B's 3600 µs limit sat inside that band.

ROCK 5B+ (radxa,rock-5b-plus), 6.1.115-vendor-rk35xx-fix, governors at performance (DMC 2400 MHz, A76 2400 MHz), n = 41 runs across four contexts:

context n 3-core min 3-core max 1-core min 1-core max fails vs 3600 µs
standalone, first batch after idle 8 3408 3652 5767 5866 6/8
standalone 20 3357 3616 5706 5880 2/20
after model 12 (the suite's predecessor) 10 3377 3613 5753 5845 1/10
inside a full make test 3 3314 3533 5756 5781 0/3

Modes: fast ≈ 3310–3420 µs, slow ≈ 3530–3655 µs. Worst observed 3652 µs 3-core / 5880 µs 1-core. Thermals are not the cause (51.8–55.5 °C through a whole suite, A76 never below 2400 MHz), and it is pre-existing — it reproduces on a pristine control tree.

Limits are each board's worst observed value +15%:

board 3-core limit 1-core limit
radxa,rock-5b 3600 µs (unchanged) 6500 µs (unchanged)
radxa,rock-5b-plus 4200 µs 6800 µs
uncalibrated widest row, with a printed NOT CALIBRATED warning

What the 5B+ guard still catches. The regression class it exists for is the weight-DMA M-tile cap reverting from mg_max*64 to R-1 (2026-06-30), which was 1.63× on 3-core and 2.3× on 1-core. From the 5B+'s slow mode that lands at ~5900 µs / ~13.3 ms — caught with 40% and 95% of headroom. In general the 4200 µs limit catches any ≥1.15× slowdown from the slow mode, ≥1.26× from the fast mode.

What it would miss. A regression smaller than ~15% on the 5B+ (the 5B's 3600 µs limit, against a ~2.8 ms base, caught ≥1.29×). A regression that only shows in the fast mode, because the slow mode already sets the limit. And — unchanged from before — anything that is not multi-core int8 M=512 K=N=2048: test_speed times exactly one shape. 12/12 runs pass under the new limit.

Divergence observed while doing this (2026-10-13)

At cb0bf06 — a tree whose committed attest says ALL TESTS PASSED — make test on the ROCK 5B+ fails two tests deterministically (3/3 suite runs, and standalone):

  • i4: W4A4 public-API cases M=8 K=512 N=256, M=16 K=2048 N={64,256}, M=32 K=1024 N=128 and the new sat collision case all report maxerr ≈ 34 200–36 200, i.e. ≈ 32767 — output cells still carrying the 0x7fff int16 doorbell seed.
  • test_slice_rescue: three fnv64 golden mismatches.

Same tree, same governors; the two boards differ in DRAM clock and kernel. This is the exact failure class two-board attest exists to make visible, and it was invisible in the single-CORE_SHA format.

Clone this wiki locally