Quantization Paths and Baselines

Quantization Paths & Baselines (don't compare across paths)

The single most common benchmarking mistake on this stack is comparing numbers from different runtime paths as if they were apples-to-apples. They are not. A figure is only meaningful with its path attached.

The paths

Path	Runtime	Format	RK3588 ceiling (Qwen3-1.7B, -t 4, warm)	Notes
w8a8 .rkllm (closed)	librkllmrt	.rkllm (Rockchip w8a8)	184 prefill / 11.71 decode	the proprietary baseline
w8a8 open stack	ggml-ork → ork-driver	.rkllm-equiv w8a8	178 (97%) / 12.8 (109%)	matches+beats closed; the headline result
GGUF Q8_0	ggml-ork → ork-driver	.gguf Q8_0	~83 prefill / ~4 decode	the correct ceiling for this path
int4 W4A4	ggml-ork → ork-driver	int4	research-only (not yet viable)	see Tier 4

The rule

Same model, same format, same runtime — or it's not a comparison.

The 11.71 / 12.8 / 178 / 184 numbers are the w8a8 path. The GGUF Q8_0 path's correct ceiling is ~83 prefill / ~4 decode — its decode is bottlenecked outside ork-driver (~88% in ggml-ork op-routing / CPU fallback + weight bandwidth; ork-driver is ~12% of decode). GGUF Q8_0 decode being ~4 t/s is not a regression — it's that path's ceiling. (This conflation cost real cycles on 2026-06-24: an "83 prefill / decode stalled" reading was mistaken for a w8a8 regression when it was just the wrong baseline + CPU-Big-Core-Contention.)

Before quoting any number

1. State the path (which row above).
2. Verify governors at max (DDR performance@2112MHz, big cores performance).
3. Run with -t 4 (big cores only), warm, ≥2 reps, llama-bench (not short llama-cli).
4. For decode, ensure no other process is on the big cores (stop or taskset co-resident servers — see CPU-Big-Core-Contention).
5. The ggml-ork backend prints the ork-driver version at init ([ork] ork-driver X.Y.Z) — record it; a stale vendored submodule will announce an old version.

See Benchmark Standards & Methodology for the full procedure.