Skip to content

Architecture

Jump to bottom
sikkatu edited this page Nov 1, 2026 · 5 revisions

English | Русский | 中文

Architecture

Topology

Chat client (OpenAI-compatible, long message history)
   │  /v1/chat/completions
   ▼
vLLM  (TP4 or any parallelism, any served model; production recipe untouched)
   │  --kv-transfer-config: LMCacheMPConnector
   │  (kv_connector_module_path pins the LMCache-shipped connector)
   ▼
lmcache server  (standalone process; ZMQ request port 5555, HTTP mgmt 8080)
   │  L1: host RAM buffer (KVP_L1_SIZE_GB, default 64 GB)
   ▼
L2 persistent tier: nixl_store_dynamic / POSIX  →  NVMe (KVP_CACHE_ROOT)
   eviction controller: LRU, trigger_watermark 0.9, eviction_ratio 0.1

Data flow

  1. vLLM computes KV chunks and STORES them through the connector into L1 (host RAM).
  2. The store controller asynchronously writes each object to L2 as one file per chunk key (temp file + atomic rename — a crash never leaves a half-written object).
  3. On lookup, L1 is checked first; a miss falls through to L2 on disk. A disk hit is prefetched into L1 and returned to vLLM. The in-memory index starts empty at boot and is populated lazily from the deterministic file names — this is what makes recovery work after both services restart.
  4. On shutdown, persist_enabled=true keeps the L2 files (the default; setting it to false deletes them and breaks the main requirement).

Why nixl_store_dynamic

Only this LMCache L2 adapter family is documented with persist/recover semantics: files are opened per operation (no pre-allocated registry that dies with the process, no fd-count limit at 1.5 TB scale), and a lookup miss falls through to the deterministic file on disk. nixl_store (the pre-allocated variant) does not document restart recovery. POSIX backend is chosen deliberately — the simplest reliable NVMe path — before any GDS/hybrid exoticism is considered.

Component independence

The cache service is a separate process. Restarting vLLM cannot erase persistent KV data; restarting the cache service keeps the data files on disk. A mode where cache data only lives in one process's memory does NOT satisfy persistence and is not used.

Compatibility namespaces

The cache key includes:

  • model id + model revision (full 40-hex commit SHA)
  • tokenizer name + revision
  • effective KV-cache dtype (resolved — auto is not accepted)
  • KV geometry: block size, num KV heads, head dim
  • tensor-parallel size
  • connector name + pinned connector/vLLM build identity
  • max model len
  • chat-template digest (sha256 of the rendered template)

The namespace is kvp-<first 32 hex of sha256(canonical JSON of the fields)>. This is the logical identity id. On the LMCache storage level it is converted into model_name (CacheIdentity.lmcache_model_name()), which is what actually separates files on disk: the nixl_store_dynamic adapter names files <safe_model_name>_<kv_rank>_<group>_<chunkhash>.bin directly under KVP_CACHE_ROOT (flat layout — there are NO kvp-<hash> directories; the prefix of each file name is the sanitized model name carrying this namespace). Any empty/uncertain field ⇒ cache MISS (recompute), never a wrong hit. Changing any field moves you to a NEW namespace; old data becomes unreachable but is never deleted automatically (see Storage & Eviction).

Model-agnostic boundary

Layer Where Model-specific?
Identity / namespaces implementation/kvpersist/identity.py input data only
Budget & quotas implementation/kvpersist/budget.py no
Retention ranking implementation/kvpersist/retention.py no
lmcache server + L2 adapter configs/generic/* no
Identity pins & geometry configs/<model>/identity.env yes
vLLM launcher scripts/run-vllm-with-persist.sh yes

"Model-agnostic" means the mechanism supports multiple deployments — NOT that their KV data is interchangeable. Different models never share KV blocks.

Failure model

Failure Behavior
cache miss / partial hit recompute only the missing part
incompatible entry unreachable by namespace ⇒ miss
corrupt / incomplete file lookup falls through ⇒ recompute; output never wrong
interrupted write atomic rename ⇒ reader sees old or new file, never partial
interrupted eviction same atomicity; missing files are plain misses
disk full / budget hit stores rejected; serving continues without cache writes
cache service down vLLM falls back to uncached serving; correctness unchanged
crash of either process restart ⇒ lazy index rebuild from disk

A cache failure can never corrupt output: the worst case is recomputation.

Process layout

  • lmcache server — background, pid file KVP_CACHE_ROOT/lmcache-server.pid.
  • vLLM container — name mimo-pro-tp4-kvp (kv-persistence launcher) or the recipe's own container when using the KV_PERSIST=1 integration.
  • kvp-admin (console script / python3 -m kvpersist.cli) — reporting and explicit cleanup; never runs automatically.

Clone this wiki locally