Repository navigation
Exp 2026 06 28 Tiling Autotuner
Commit: ork-driver 33905ff (branch re/rkllm-mtile-decode, local, not pushed) · tools/autotune.c · tools/run_autotune.sh · ORK_KTILE knob in src/npu.c · AUTOTUNE_WIP.md. RK3588 10.3.0.236, governors max, isolated checkout.
Built an NPU matmul tiling autotuner (per-shape config search, bit-exact-gated, cached) and ran it on the Qwen2.5-7B prefill shapes. It found a real matmul-level win — CBUF K-tiling lifts GOPS +36–60% by raising the CBUF row-ceiling R. But end-to-end pp256 was flat (11.96 → 11.46 t/s, ~16% of librkllmrt's 73.6). The decisive result: speeding the NPU matmul +36–60% moved prefill 0% → prefill is host-bound (ggml-ork CPU-side quant/dequant/scheduling), not NPU-matmul-bound. This resolves a long prefill-optimization arc: there is no NPU-side prefill lever left; the K-tiling win is banked for the M>1 (batched-verify / spec-decode) regime, where the matmul actually binds.
tools/autotune.c: for each shape (M,K,N) — pack once, compute a CPU int8 reference once, then sweep the knobs ork exposes (core count 1–3, M/N-subtile via ORK_SMALLTILE, K-tile via ORK_KTILE), bit-exact-gate every config against the reference, time warm ork_mm_run_i8 (median), report effective GOPS, and cache the per-shape min-µs winner. Reusable across models/shapes and re-runnable per SoC.
Finding: the existing knobs are saturated — M-tile is already clamped to R−1 (the CBUF row ceiling), and core-count (1→3 ≈ 2.7×) is the dominant lever and already the default. The only new lever is K-tiling.
ork's prefill row-ceiling is R = pow2_floor(2·cbuf/K) — it shrinks as K grows (R=16 at K=3584). Smaller K-slices → smaller per-row CBUF working set → bigger R → larger M-tile → more M-amortization of the ~80–100 µs per-task submit floor. ORK_KTILE overrides the int8 K-slice (suppressing the full-K Bf, reusing the already-validated wide-K accumulate — no new regcmd, wedge-safe). KT=1024 (R 16→64) was the sweet spot (M=256, 3 cores):
| shape | K×N | default GOPS | KT=1024 | gain |
|---|---|---|---|---|
| q_proj / o_proj | 3584×3584 | 700 | 1118 / 1104 | +60% |
| gate_up | 3584×18944 | 731 | 994 | +36% |
| kv_proj | 3584×512 | 479 | 479 | 0 (N-bound) |
| down | 18944×3584 | 1413 | (already K-split) | — |
Bit-exact vs the CPU reference at every config; no wedges.
Warm pp256, 7B Q8_0, GGML_NO_REPACK=1, -t4, r3, governors max:
| pp256 t/s | % of rkllm 73.6 | |
|---|---|---|
| default | 11.96 | 16.2% |
ORK_KTILE=1024 |
11.46 | 15.6% (flat/−4%) |
A +36–60% matmul-GOPS win produced 0% end-to-end prefill gain. run_i8 is ~9.9 ms/call, but prefill is dominated by host-side work — activation quant/dequant, scheduling, and the per-slice A re-tile + extra accumulation passes K-tiling itself adds. So the matmul compute K-tiling accelerates is not the binding constraint; the host/ggml-ork path is.
✅ The autotuner works and K-tiling is a genuine matmul-level lever (+36–60% GOPS, bit-exact). ⚖️ But it does not move prefill: the 12→73.6 prefill gap is host/ggml-ork-side (CPU quant/dequant/scheduling), not NPU matmul throughput. This closes the NPU-side prefill-optimization arc (submit-floor → quant → tile-size → K-grouping → multi-task-M-offset [doesn't exist] → autotuner): the matmul is already adequate, the knobs are saturated, and faster matmul doesn't help. ORK_KTILE stays gated-off; the matmul-GOPS win is banked for the M>1 batched-verify path (spec-decode), where the matmul binds. The autotuner + K-tiling are durable, reusable assets (re-tune any model/SoC; pay off at M>1).
NEXT: pivot to spec-decode — decode→M>1 (the throughput win the multi-domain residence breakthrough unblocked) and the regime where the banked K-tiling matmul win finally binds.