FAQ

English | Русский | 中文

FAQ

Does this replace vLLM's prefix cache? No — it adds a durable layer underneath. In-memory prefix caching stays enabled; the persistent tier rehydrates it after restarts.

Will two different conversations both benefit? Yes, when they share a stable tokenized prefix (common system message + tool definitions): identical chunks are stored once and reused by both. Dynamic values (timestamps, request ids) break prefixes — see Troubleshooting.

Do different models share KV blocks? No, never: namespaces include model id, revision, tokenizer, dtype and layout. "Model-agnostic" means the machinery works for many deployments — their KV data is not interchangeable.

Does the cache survive a host reboot? The data files on NVMe survive. After a reboot start the cache service first (start-cache.sh), then vLLM — index rebuild is lazy and automatic.

Do I have to change my chat client? No. The integration is entirely vLLM-side; your chat client talks to the same /v1/chat/completions endpoint as before.

What happens when the disk fills up? Stores above the hard budget are rejected; serving continues correctly without new cache writes. Eviction at the soft threshold (90 %) normally prevents reaching this state.

Is my prompt text on disk now? Not as text — but KV data is derived from prompts and is treated with the same sensitivity (chmod 700, loopback-only management, docs in Security).

Is the KV on disk lossy-compressed? No. The L2 tier is lossless-only: chunks are stored byte-exact (default) or with lossless zstd/lz4 compression (1.1–1.4x estimate). Lossy quantization (fp8 / turboquant / lbq) is refused at startup. Every frame carries a CRC32 — a corrupt frame is reported as a cache miss, never returned as wrong KV.

Do I need the compression packages if I don't compress? No. A LMCACHE_L2_SERDE=none cache (the default) needs no compression package at all. zstd requires only zstandard, lz4 requires only lz4 — a check before start verifies exactly the selected codec, never an unused one.

Can I watch this in Grafana? Yes — bash scripts/start-observability.sh brings up the official LMCache observability stack (OTel Collector + Tempo + Prometheus + Grafana) with the LMCache dashboards pre-imported. Grafana is at http://localhost:3000, all ports bound to 127.0.0.1 only. Start the cache with USE_OBSERVABILITY=1 so metrics flow; the cache works fine without the stack.

Which parts are tested where? The unit suite (identity, budget, retention, config-locales, lossless serde frames, cache-salt filenames, tmp-file accounting) runs anywhere; integration tests need the lmcache CLI; acceptance/recovery tests require the GPU host (KVP_HARDWARE=1). Status labels are honest — see the repository's docs/final-report.md.

Why did the cache "hit" but TTFT barely improved? Cache hits remove prefill compute, not decode time or PCIe transfer of cold experts (hotsplit/MoE offload dominates tail latency here). Expect TTFT improvements to scale with prompt length; short prompts show little absolute gain.

Can I run this next to the hotsplit recipe? Yes — that is precisely the design: KV_PERSIST=1 in the recipe launcher attaches the connector; the recipe's hot/cold expert split is untouched.