Before a model can generate a single token it must process the entire prompt, and for a long shared prefix (a system prompt, a few-shot block, a retrieved document) that prefill is most of the latency of a short request. llama.cpp caches the prefix across requests to the same slot. This measures exactly what that cache saves, as a function of prefix length.
Everything is measured wall-clock against a real server. No judge.
Qwen2.5-1.5B-Instruct Q4_K_M, real llama-server. A shared prefix of a given length is sent
cold (a unique session id forces a cache miss) then immediately again (a cache hit); prefill
time is the round trip at one token. Full numbers in
bench_results/frontier.md.
| prompt tokens | cold prefill ms | warm prefill ms | cache speedup | prefill eliminated |
|---|---|---|---|---|
| 114 | 67 | 8.1 | 8x | 88% |
| 190 | 93 | 8.3 | 11x | 91% |
| 340 | 160 | 8.6 | 19x | 95% |
| 640 | 294 | 9.3 | 31x | 97% |
| 1240 | 577 | 9.8 | 59x | 98% |
| 1841 | 864 | 10.6 | 81x | 99% |
- Cold prefill grows linearly with prompt length, from 67ms at 114 tokens to 864ms at 1841 tokens: the server processes every token of the prompt.
- Warm prefill is near-flat and tiny, 8 to 11ms at every length: on a cache hit the shared prefix is reused and only the new suffix is processed, so the length of the prefix barely matters.
- The cache speedup climbs with prefix length, from 8x at 114 tokens to 81x at 1841, eliminating 88% to 99% of the prefill. The absolute saving grows because the entire shared prefix is skipped, so the longer and more-shared the prefix, the more caching buys.
So the practical rule this makes exact: a long shared prefix costs its full prefill exactly once, then is nearly free on every later request that shares it. Structuring prompts so the stable, reusable part (system instructions, few-shot examples) comes first and the variable part comes last turns that prefix from a per-request cost into a one-time cost.
The prefill is a forward pass over the prompt that fills the KV cache for every prompt position. llama.cpp keeps that KV cache in the slot and, on the next request, matches the longest common prefix of tokens and reuses its KV entries, so only the diverging suffix is recomputed. The warm time is the fixed request overhead plus the tiny suffix; the cold time is that plus the full prefix forward pass, which scales with prefix length.
- A prefix that changes every request (a timestamp or a unique id at the front) defeats the cache: the longest common prefix is short, so nothing is reused. The win requires the reusable content to come first.
- A larger model has a costlier prefill per token, so the absolute saving grows while the ratios stay similar.
- Slot pressure. With more concurrent distinct prefixes than slots, a prefix can be evicted before reuse, so the realized saving depends on the working set fitting the slots.
Limitations (named, not hidden)
- One model and size (Qwen2.5-1.5B-Instruct), one node (Apple M4 Pro Metal), single-slot reuse (cold immediately followed by warm), five repeats per length. Absolute milliseconds are hardware-specific; the shape (flat warm, linear cold, growing speedup) is the result.
- The warm floor (8 to 11ms) is dominated by request and one-token-decode overhead, not prefill; it is the practical cost of a cache hit on this server, not zero.
- Core (
src/prefill.py). Cache speedup ratio, fraction of prefill eliminated, prefill throughput, and amortized cold cost over reuses. - Sweep (
tools/run_sweep.py). For each prefix length, time a cold request (unique nonce forces a miss) then an identical warm request (cache hit) at one token. - Frontier (
tools/analyze.py). Per-length cold and warm prefill with CIs, the cache speedup, and the fraction of prefill eliminated.
python3 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
python -m pytest -q # 5 tests
./scripts/gate.sh # ruff + mypy --strict + pytest + ASCII
Needs a local llama-server on a Qwen2.5-1.5B-Instruct GGUF:
llama-server -m qwen2.5-1.5b-instruct-q4_k_m.gguf --alias qwen-pf --ctx-size 8192 --port 8081 &
./reproduce.sh 8081
MIT. See LICENSE.