Home

English | Русский | 中文

kv-persistence

Persistent KV cache for vLLM — restarting the server stops being a recount of history.

kv-persistence stores the prompt KV blocks computed by vLLM on NVMe and restores them after restarts. A conversation in any OpenAI-compatible chat client of 100k–1M tokens — which used to be re-prefilled for tens of minutes after every restart — reuses its cache instead of being recomputed.

Chat client → vLLM (TP4) → LMCacheMPConnector → lmcache server (separate process)
       → L2 persistent tier (nixl_store_dynamic, POSIX) → NVMe

Why

• vLLM's built-in prefix cache lives in GPU memory and dies with the process. Every restart of a server holding huge conversations meant re-prefilling them.
• The persistent tier keeps reusable KV chunks on NVMe. A vLLM restart (or even a cache-service restart) does not erase them: a lookup miss falls through to disk, and the index is rebuilt lazily at startup.

What it guarantees

• Persistence across restarts — data survives both a vLLM restart and a full cache-service restart (scenarios A and B in Persistence & Recovery).
• Bounded storage — hard cap (1.5 TB default; bound by free space on the SHARED VM disk) with automatic soft-threshold (90 %) eviction. The cache never wipes itself on startup.
• Compatibility namespaces — model revision, tokenizer, KV dtype and chat template are part of the cache key. Incompatible configurations never read each other's KV data; any uncertainty yields a cache miss, never a wrong hit.
• Lossless storage — the L2 tier is lossless-only. Chunks are stored byte-exact (LMCACHE_L2_SERDE=none, the default) or with lossless zstd/lz4 compression; lossy quantization (fp8 / turboquant / lbq) is refused at startup. Every frame carries a CRC32: corruption is detected and reported as a cache miss, never returned as wrong KV.
• Optional Prometheus + Grafana monitoring — the official LMCache observability stack (OTel Collector + Tempo + Prometheus + Grafana) ships in observability/ and comes up with one command; the cache works fine without it.
• Model-agnostic core — the cache-policy layer is separate from model integration; new models add configuration, not code.

Status

Implementation complete, offline tests green (143 unit tests + 34 more offline integration/config tests = 160 passed, 2 skipped; 177 collected total — plus 302 passed of 307 collected in the sikkatu/recipes repo). Acceptance hardware runs on 4× RTX PRO 6000 with MiMo-V2.6-Pro are scripted in scripts/test-recovery.sh / scripts/benchmark.sh and are run by the operator. Unverified capabilities are labelled honestly — untested things are never claimed as working.

Start here

• Quick Start — install, configure, launch, verify (one linear path)
• Architecture — how it fits together, failure model
• Configuration — every knob of cache.env, identity.env, lmcache-server.env
• Persistence & Recovery — acceptance scenarios, how to prove a disk hit
• Storage & Eviction — budget, quotas, eviction, dedup
• Recipes Integration — KV_PERSIST=1 in the sikkatu/recipes launchers
• Security — permissions, endpoints, data handling
• Troubleshooting — what to check when something looks off
• FAQ — short answers to recurring questions

Repository: github.com/sikkatu/kv-persistence (Apache-2.0 + NOTICE: forks and users must credit sikkatu / kv-persistence; the NOTICE file).