kvquant: does quantizing the KV cache speed decode up or slow it down?

Quantizing the KV cache (storing keys and values at 8 or 4 bits instead of 16) is a standard way to fit longer contexts in memory. Since decode is memory-bandwidth-bound and each token reads the KV cache, the intuition is that a smaller cache should also be faster. This tests that directly: stream long generations at each KV precision (f16, q8_0, q4_0) and measure per-token decode latency, two model sizes (1.5B and 0.5B, Q4_K_M weights), flash attention on.

Pre-registration

Two predictions were committed to git (PREREG.md) before the sweep: (1) quantizing the KV cache speeds decode up (fewer bytes read); (2) it reduces the decode drift. Both were falsified. Quantizing the KV cache makes decode slower.

Result

Base per-token decode latency (intercept of gap = a + c*pos, robust to the drift):

 model   KV precision   base ms/token   95% CI            vs f16
 1.5B    f16               6.082         [6.03, 6.09]      baseline
 1.5B    q8_0              6.939         [6.71, 7.14]      +14%
 1.5B    q4_0              7.055         [7.05, 7.06]      +16%
 0.5B    f16               3.690         [3.68, 3.70]      baseline
 0.5B    q8_0              4.178         [4.17, 4.18]      +13%
 0.5B    q4_0              4.241         [4.24, 4.25]      +15%

1. Quantizing the KV cache slows decode down, by 13-17%. Every quantized precision has a higher base per-token latency than f16, for both models, and the bootstrap difference in medians excludes zero. A smaller KV cache does save bandwidth on the attention read - but that read is a small part of the forward pass, which is dominated by streaming the (already 4-bit) weights, and the per-step dequantization of the quantized cache back to compute precision costs more than the bandwidth it saves. The 4-bit cache is slightly slower than the 8-bit one, consistent with more unpacking work.

2. KV quantization is a memory-capacity optimization, not a latency one. It lets a longer context fit in the same memory; it does not make generation faster, and in fact trades ~15% of decode speed for that capacity. Anyone reaching for KV quantization to reduce latency is optimizing the wrong resource.

(The per-position drift slope c is reported for completeness but is noisier run-to-run than the base latency; the headline rests on the base per-token cost, which is tight and reproducible. The base latency alone settles the speed question.)

Reproduce

Run once per model; each call relaunches a server at f16, q8_0, and q4_0 in turn:

./reproduce.sh 8001 /path/to/model-1.5b.gguf qwen15b
./reproduce.sh 8001 /path/to/model-0.5b.gguf qwen05b
./scripts/gate.sh               # ruff, mypy --strict, pytest, ASCII, independent verify

tools/verify.py recomputes the base latency per precision as the median of the first 500 tokens (no fit) and asserts, with a bootstrap difference, that every quantized precision is slower than f16

• independent of the fitting code in the analysis.

Limitations and falsifiers

• One backend (llama.cpp on Apple GPU, flash attention on), one build, Q4_K_M weights, two sizes, KV types f16/q8_0/q4_0, ~4000 generated tokens x 3 reps. The magnitude of the slowdown is backend-specific; another attention kernel could dequantize more cheaply. The direction (quant does not speed decode up here) is what is claimed.
• The study measures latency, not the memory saved (which is the real point of KV quantization) or output quality (measured elsewhere). It only refutes the speed rationale.
• Falsifier: if any quantized precision had lower base latency than f16, prediction 1 would stand. None does, for either model.

MIT licensed. The oracle is monotonic-clock inter-token timing; no model-quality judgement involved.