Skip to content

Exp 2026 09 03 KV Cache Quantization

Jump to bottom
Michael Fischer edited this page Sep 4, 2026 · 2 revisions

2026-09-03 — KV cache quantization: a 4× attention tax that is NOT bandwidth and NOT the GPU

Verdict: quantising the KV cache costs ~3.6× on the attention slope on this platform, and a mismatched (type_k, type_v) pair costs a further ~1.6-1.8× on top. The cost is dequantisation inside the flash-attention inner loop — it is not memory bandwidth (halving the bytes made it slower) and not Mali-specific (q8_0 with Vulkan disabled shows the same penalty as TurboQuant on the GPU). TurboQuant remains a capacity feature, not a performance one.

Filed as llama.cpp-rockchip#7 and oRKLLM#1.

Hypothesis (refuted)

Attention had been observed at 82-87% of prefill time at 4-6k tokens, with an arithmetic intensity of 2.0 FLOP/byte before query-block reuse against an RK3588 machine balance of ~15 — so the working hypothesis was memory-bound. Prediction: cutting KV bytes 2× (f16 → q8_0) should cut the attention term roughly in half, making KV quantisation a prefill speedup, not just a memory saving.

Measured: it went 4.2× the other way. The hypothesis is dead. FA tiles, so the no-reuse traffic estimate (~6.8 GB/s at n≈6k) was an upper bound the real kernel never approaches; there was no bandwidth wall to hit, and the dequant ALU cost dominates.

Method

Radxa ROCK 5B (RK3588, 10.3.0.236), b10739-ork, ork-driver 1.0.105, governor performance (DDR 2112 MHz / A76 2.4 GHz), cores 4-7, warm. Model: Qwen3-1.7B-UD-Q4_K_XL int4 .orkpack (196 weights), ORK_QUANT=4 ORK_NO_STUB=1.

llama-bench -m <gguf> -p 400,2000,4000 -n 0 -r 3 -t 4 -b 512 -ub 512 \
            -ngl 99 -dev ORK -fa on -ctk <K> -ctv <V>
env: ORK_QUANT=4 ORK_NO_STUB=1 GGML_DISABLE_VULKAN=1

Deviation from the standard config, stated explicitly: this uses -p 400,2000,4000 -n 0 rather than the reference -p 512 -n 128, because the quantity under test is the attention slope — the per-token cost of one extra token of context — which needs multiple lengths and does not need decode. Decode was not measured at all (see Open below). No librkllmrt row: the closed runtime exposes no KV-type control and does not consume .orkpack, so it is not an applicable arm here.

slope below = (ms/token at 2000 − ms/token at 400) / 1600.

Result — CPU path, Vulkan disabled

type_k type_v pp400 pp2000 pp4000 slope vs f16
f16 f16 227.07 ± 1.18 130.19 ± 0.12 84.56 ± 0.04 0.00205 1.0×
q8_0 q8_0 167.20 ± 0.17 56.27 ± 0.04 26.77 ± 0.03 0.00737 3.6×
q8_0 f16 161.25 ± 0.08 40.36 ± 0.02 — 0.01161 5.7×
f16 q8_0 145.72 ± 0.22 35.55 ± 0.08 — 0.01329 6.5×

Two penalties that stack:

  1. Quantising at all: 3.6× (matched q8_0 vs matched f16). Dequant in the FA inner loop.
  2. Mismatching the pair: a further ~1.6-1.8×. Both mixed pairs are worse than both matched pairs — including worse than quantising both sides. Signature of falling off a specialised (type_k, type_v) FA kernel onto a generic path.

At 4000 tokens f16 → q8_0 is 84.56 → 26.77 tok/s, −68%.

TurboQuant — measured through oRKLLM, not llama-bench

turbo needs ggml_vk_set_mode() before model load, which only oRKLLM's addon calls, so llama-bench cannot reach it. Run on a throwaway oRKLLM instance (fresh DB, n_ctx=4096), KV types verified live via [CPAR-DBG] rather than assumed:

tokens q8_0 K + turbo3 V (type_k=8 type_v=43) turbo3/turbo3 (43/43) f16
402 91.60 69.57 (−24%) 244.66
1975 41.10 40.80 136.30
3943 24.48 24.96 85.59
  • The mixed-pair penalty does NOT transfer to turbo. turbo-on-K is much worse at short prompts and a wash at long ones — the turbo codec (inverse WHT + polar dequant) is far heavier than a q8_0 scale-multiply, and K is read in QKᵀ. A 2-point fit projected an 11% matched win at 4000; measured 2%, inside noise. Projection recorded as wrong.
  • oRKLLM's never lead with turbo K policy is therefore right on speed as well as quality and should not be "fixed" on the assumption that matched pairs always win.
  • turbo3 vs f16 served: 7.67 + 0.00842·n vs 3.23 + 0.00214·n ms/token — 2.4× constant, 3.9× slope.
  • CPU load average during the two: 1.3-2.0 (turbo3) vs 3.6 (f16) — the CPU idles while the Mali works.

Controls

control result
n_ctx (q8_0,turbo3) 91.60/41.10 at 4096 vs 90.47/41.04 at 32768 — allocation size is irrelevant
[BATCH-DBG] logging overhead same config with/without ORKLLM_LOGIT_DEBUG: 91.60 vs 90.47 — ≤1%
flash attention actually on flash_attn_type AUTO, resolves to enabled (llama-context.cpp:519-523; the device-mismatch disable branch is commented out)
threads cpar.n_threads = n_threads_batch = 4, matching the pinned big cluster
GPU involvement q8_0 arm runs GGML_DISABLE_VULKAN=1 — same penalty, so not the GPU

Side result: the "serving gap" was this setting

llama.cpp-rockchip#6 reported oRKLLM's worker running the same i8 chain ~2.6× slower than a bare process, with a large exclusion table. That gap was this KV setting — the served path was on (q8_0, turbo3) while every probe ran f16/f16; the exclusion table varied n_ctx but never the KV dtypes. With KV matched:

tokens served f16 llama-bench f16
~400 244.66 227.07
~2000 136.30 130.19
~4000 85.59 84.56

Served is equal or marginally faster; the attention slopes match (0.00214 vs 0.00206). There is no serving overhead. #6 closed. Every mechanism its table excluded (orkd, op routing, addon binary, runtime version, affinity, priority, cgroups, fork+IPC, stdio, stub, node hosting, reload, shape re-warm) was correctly excluded — the variable nobody checked was oRKLLM's own saved per-model KV setting, which is invisible in the logs. Read [CPAR-DBG] before comparing two processes.

Correction to the submit-floor number in circulation

ggml-ork's supports_op comment cites a ~365 µs/matmul NPU submit floor and reasons from it that M=1 decode is a net loss ("~197 submits/token × 365 µs ≈ 72 ms before any compute benefit"). That figure — and Exp-2026-06-12 Submit-Floor Profiling, which measured ~205-496 µs/matmul and cross-checked it against the closed RKNN kernel — describes the blocking mcworker submit path (run_i8).

It does not describe the non-blocking doorbell (ork_submit_async / ork_dyn_begin_mc, ork_mm_run_sliced, run_i4_bchain_db). This wiki already carried the correct figures — they were on record before this experiment and I failed to cite them:

record measurement
Exp-2026-06-21-Mixture-of-Experts-PC-Chaining-and-Scheduler-Programming (tools/chain_probe, RK3588, S=8 int8 256×64, 200 iters) 8 separate submits 58.1 µs/matmul → 1 chained submit 5.2 µs/matmul, 11.21×, "zero CPU intervention"
Exp-2026-07-16-Dynamic-Submit-API-and-Doorbell NONBLOCK doorbell progress overhead is "a few µs/op"; the ~167 µs linger default is the NONBLOCK submit, not a per-op cost
ggml-ork.cpp:398 chained doorbell at M=1 measured at 1.18× the per-channel cost

i.e. ~5 µs per op inside a chain, and no per-op submit cost at all while the doorbell is spinning — against 58.1 µs/matmul unchained and the ~365-496 µs blocking-path floor.

Consequence: any argument that multiplies 365 µs by a submit count is measuring the wrong path — which includes the justification for ORK_MINM and for routing M=1 decode to CPU. Both are calibrated against a cost the doorbell does not pay, and the crossover is worth re-deriving with the doorbell forced. ggml-ork.cpp:6062 already anticipates this ("the thread-free doorbell should overlap for free"; the blocking rendezvous "lost net (profiled)").

It also strengthens the heterogeneous-overlap case (no dispatch tax while spinning, so pipelining layer i's FFN on the NPU against layer i−1's KV work on the GPU costs nothing per op; the blocker is that ggml's scheduler assigns whole ops to backends with a sync at each boundary and has no multi-stream model). It does not change the decomposition case: flash attention cannot be split across processors because its efficiency comes from never materialising the n×n intermediate, and the dequant cost lives inside that fused loop where no dispatch scheme reaches it.

Implementing code

what where
KV type selection pool.js — kv_cache_quant → kv_type_k/kv_type_v; forces kv_type_k=q8_0 whenever V is turbo/q8_0 ("never lead with turbo K")
KV types applied orkllm_llama_napi.cpp init_model — kvTypeFromStr (f16=1, q4_0=2, q5_1=7, q8_0=8, turbo2/3/4=42/43/44) → llama_context_params.type_k/type_v
turbo GPU scoping pool.js sets vk_mode:'turboquant' + n_gpu_layers:0; addon calls ggml_vk_set_mode before model load (nullptr-guarded)
verification knob ORKLLM_LOGIT_DEBUG=1 → [CPAR-DBG] prints resolved n_ctx/type_k/type_v/flash_attn/n_threads
attention op gate ggml-ork.cpp:9749 ork_supports_op_inner — FLASH_ATTN_EXT declined unless ORK_ATTN; batched MUL_MAT (ne[2]>1) and SOFT_MAX likewise
harness scripts/bench-prefill-chunking.mjs (oRKLLM) + llama-bench

Nothing was reverted; every arm is reachable by setting kv_cache_quant / kv_type_k / kv_type_v.

Open

  • Decode is unmeasured. Decode is bandwidth-bound on KV reads, so a 3× smaller cache could be a decode win even while it is a prefill loss. Until measured, "KV compression is slow" is a prefill-only claim.
  • Is the CPU q8_0 FA path missing a NEON specialisation? 4× is high; x86/AVX-512 typically pays 10-30%. May belong upstream in ggml-org/llama.cpp rather than the fork.
  • ORK_ATTN (attention on the NPU) unmeasured. Attention is ~70% of prefill at 4000 tokens even on f16, at ~56 GFLOP/s (~36% of the A76 cluster's fp32 NEON peak). This is the largest remaining prefill lever, well ahead of anything KV-side.

Clone this wiki locally