Repository navigation
Model Recipes
Look here FIRST when bringing up a model. This is the answer to "what should I set for model X", recorded per model, with the measurement that justifies it. It exists because the right config is model-shaped: whether a path pays depends on layer count, head geometry and weight sizes measured against what the NPU actually achieves, and several paths that look like wins in isolation lose end-to-end. Guessing has cost this project repeatedly.
Three sibling pages, easy to confuse:
| page | answers |
|---|---|
| this page | what config should I use for model X |
| Benchmark Configs Registry | what exactly was set when we measured number Y (run provenance) |
| Benchmark Standards & Methodology | how to measure at all (the rules) |
Machine tier (automatic, do not set by hand). The .orkpack footer carries a calibration record
(orkpack_calib_hdr + orkpack_calib in ggml-ork.cpp) that the backend reads at load. It is
provenance-guarded by ork_calib_valid_here() on the five things that actually move the answer —
big-core max freq (so a DTB/OPP change invalidates it), big-core count, the ork-driver pack-format
token, whether all big-core governors were performance, and an FNV-64 of uname release+version
(a kernel change moves the submit floor). Precedence is: a sweep override > ORK_MINM > the pack's
measured value > the built-in default. Today one field is live (min_m); per-shape {K, N, min_m} is
expressible (K=N=0 = global) but unpopulated.
The record carries the resolved decode route — a single ordered decision (0 CPU, 1 dense M=1 on
NPU, 2 + fused FFN chain and M=1 group fusion), not a set of independent switches. Each level implies
the ones below, so a contradictory combination cannot be expressed. ORK_DECODE_ROUTE overrides for A/B;
ORK_M1_NPU/ORK_FFN_DEC remain as deprecated aliases that can only RAISE the route.
Human tier (this page). The recommended recipe, plus the file properties — which matter more than any knob (see the quantization recipe below).
Discipline, or this page rots. A row exists only if a measurement produced it, and it names that measurement — the same rule
OPS_REGISTRY.mdapplies to op status. A recipe with no cited number is a guess, and guesses here are worse than nothing because they read as authority.
-
Decode on CPU, prefill on NPU. Not a knob —
supports_opdeclines denseMUL_MATatM==1. -
-t 4(big cores only; A55s drag the threadpool barrier). - Governors at performance, DDR included — a parked DDR governor roughly halves decode.
- MoE profile is automatic — the model type decides; nothing to set.
llama-quantize --tensor-type attn=q8_0 --tensor-type shexp=q8_0 <f16.gguf> <out.gguf> Q4_K_M
Attention precision in the GGUF trades prefill for decode, and below 5 bits the NPU declines the source
entirely (sbits<5.0), forfeiting the prefill path. Expert precision is irrelevant (experts come from
the pack). Measured on qwen3.6-35B-A3B: f16 attn 38.4 prefill / 5.26 decode; q8_0 36.9 / 6.59 — the
balanced optimum; Q4_K 26.8 / 7.42.
All measured 2026-09-04 on rk3588 (A76 @2400 MHz, kernel 6.1.115 #70, governors performance),
ork_bench P=512 G=32, decode tok/s. Route 0 = decode on CPU, 1 = dense M==1 on NPU, 2 = + fused FFN
chain and M=1 group fusion.
| model | fingerprint (n_layer/n_embd/n_ff/attn_kv) | route 0 | route 1 | route 2 | recipe |
|---|---|---|---|---|---|
| qwen3-0.6b-f16 | 28 / 1024 / 3072 / 1024 | 15.70 | 5.19–5.25 | 5.12–5.58 | 0 |
| qwen3-1.7b f16 and q8_0 | 28 / 2048 / 12288 / 1024 | 10.21 | — | 10.36 | 0 (tie) |
| qwen2.5-1.5b-instruct-q8_0 | 28 / 1536 / 17920 / 256 | 11.51 | — | 11.48 | 0 (tie) |
| qwen3.5-0.8b-bf16 | 24 / 1024 / 3584 / 0 | 2.44 | 3.40 | 3.13 | 1 — NPU WINS 1.40× |
| mamba2-130m-f16 | 24 / 0 / 0 / 0 | 64.88–67.77 | — | 36.30 | 0, unkeyable |
| mamba2-2.7b-q8 | 64 / 0 / 0 / 0 | 6.47 | — | 6.09 | 0, unkeyable |
qwen3.5-0.8b is the first model where decode-on-NPU pays — 2.44 → 3.40 tok/s (was 3.13 before the doorbell big-core fix), reproducible to
±0.4% across 3 trials (far outside the ~9% band seen on the 0.6b; noise is config-dependent). Route 1
== route 2 (3.13 both), so the win is M==1 acceptance alone and the fused FFN chain contributes
nothing — the row says route 1. attn_kv=0 marks the GDN/linear-attention architecture, whose CPU decode
baseline is unusually low (2.44 vs 10–15 for the dense models), leaving headroom the others don't have.
This divergence is exactly what the table exists for; a single global default would have hidden it.
One row covers qwen3-1.7b f16 AND q8_0 — identical fingerprints. That validates keying on shape rather than name: source precision doesn't change the routing answer, so re-quantizing a model inherits its recipe automatically.
The Mamba rows are deliberately absent from the table. Their fingerprint collapses to n_layer alone
(SSM models have no attn_q/ffn_gate/attn_k tensors), so a row would risk colliding with any future
model of the same depth and applying the wrong recipe. A guard rejects such keys and falls through to the
default — which both measured as best anyway, so nothing is lost. Giving them real rows needs SSM-specific
fields in the key.
| prefill | decode | |
|---|---|---|
| librkllmrt (closed) | 184 | 11.71 |
| open stack | 178 (97%) | 12.8 (109%, beats it) |
-t 4, warm, board 10.3.0.236. The canonical parity claim; see Benchmark Configs Registry for the
exact provenance.
| prefill t/s | decode t/s | PPL | |
|---|---|---|---|
| native ggml (Q4_K experts) | 23.3 | 7.5 | 10.21 |
| ork auto-profile | 36.9 (1.59×) | 6.3–6.6 (0.85×) | 10.77 (+5.5%) |
A trade: +5.5% PPL and ~15% slower decode for 1.59× prefill. Prefill-heavy workloads (long prompts,
RAG, batch) win; decode-dominated chat may prefer ORK_MOE_AUTO=0. Nothing else to set — the pack tier
and the CPU/NPU split are chosen from the model type.
Recipe: the standing default. Do not enable the decode-on-NPU paths.
| config | decode t/s | PPL (wikitext, 20×512) |
|---|---|---|
| default (decode on CPU) | 15.70 | 31.4913 |
ORK_FFN_CHAIN=1 (prefill FFN chain) |
— | 31.5320 (+0.13%), wall +28% |
| decode route 1 (dense M=1 on NPU) | 5.19–5.25 | — |
| decode route 2 (+ fused FFN chain, q/k/v+gate/up fusion) | 5.12–5.58 | — |
Every decode-on-NPU config is a 2.8–3× regression.
⚠ decode tok/s here has a ~9% run-to-run spread (5.12–5.58 observed for the same config across runs), so the difference between routes 1 and 2 is NOT resolved by this measurement, and per-change deltas under ~10% are not signal. Earlier revisions of this page quoted "+2.8%" for q/k/v fusion and "+3.4%" for the FFN chain as if they were wins; both sit inside the noise band and are withdrawn. Use
ORK_FFN_PROF's per-hop GB/s (stable) for routing decisions, not end-to-end tok/s. PPL is bit-deterministic here and is safe for quality deltas.ORK_FFN_CHAIN's quality cost is negligible (+0.13%) but it is 28% slower — it is blocked on speed, not the per-channel requant concession its comment describes.
Recipe: the standing default.
| config | PPL (wikitext, 20×512) | wall |
|---|---|---|
| default | 13.9679 | 126.5 s |
ORK_FFN_CHAIN=1 |
14.0169 (+0.35%) | 179.4 s (+42%) |
PPL here is bit-deterministic (a repeat baseline reproduced 13.9679 / NLL 2.636764 exactly), so these deltas are signal, not noise.
State it this way rather than "profitable on some models", because that is what the data supports.
Measured 2026-09-04 (decode_bw_probe, ORK_FFN_PROF, ork_bench):
- Dispatch is not the wall — 48 µs of 1819 µs/layer (2.6%). The doorbell floor (~5 µs dispatch, ~16 µs/task) is already negligible.
- The NPU is not bandwidth-lost: batching S independent M=1 matmuls reaches 8.6 GB/s at S=1 and ~20 GB/s at S=6, against a ~22 GB/s ceiling and the CPU's ~9.4 GB/s effective.
- But the FFN handler gets 7.2 GB/s (gate+up) and 3.5 GB/s (down) on shapes that reach 8.6–8.8 at S=1 in isolation — 1.75–2.4× off, cause unknown.
So the gap is implementation, not hardware, and the per-model calibration record is what would find the
first model where it flips — rather than assuming one exists. Disproven causes for the down gap, do
not re-try: wide-K (its exact shape reaches 8.81 GB/s), missing Bf (8.28 GB/s without it), and the
submit mechanism (the probe's S=1 short-circuits to the same ork_i8_mm_run).
ORK_FFN_DEC was dead for its whole existence in two independent ways: gated on ctx->via_orkd
(unreachable in the default in-process mode) and declined upstream by supports_op at M==1. It
needs ORK_M1_NPU=1 to fire at all. A recipe that lists knobs as independent booleans reproduces that
failure — some combinations are silent no-ops. Record the decision ("where does decode run"), not the
mechanisms. ORK_NO_BF=1 is a second instance: it declines the batched chain entirely (S>=2 returns
nonzero), so it silently disables any batching plan.