Repository navigation
Configuration
Three files, all read by every script. Values already in the environment win over
the files (handy for one-off overrides). Site-local edits go in
configs/generic/cache.env.local (gitignored) — never edit the shipped files.
| Variable | Default | Meaning |
|---|---|---|
KVP_CACHE_BUDGET_GB |
1500 |
Hard cap of the persistent cache in GB (1.5 TB deliberately, frees more prefill history). The real bound is free space on the shared VM disk (same volume holds Docker, the model and the OS) device. |
KVP_CACHE_ROOT |
/var/lib/kv-persistence/l2 |
Cache directory. MUST sit on local NVMe; a network mount breaks atomic renames (preflight warns). |
KVP_EVICT_TRIGGER_WATERMARK |
0.9 |
Soft threshold: eviction begins at this fraction of the budget. Distinct from the hard cap. |
KVP_EVICT_RATIO |
0.1 |
How much is reclaimed per eviction cycle (fraction of used capacity). |
KVP_LMCACHE_HOST |
127.0.0.1 |
lmcache request host (ZMQ). Keep loopback. |
KVP_LMCACHE_PORT |
5555 |
lmcache request port. |
KVP_LMCACHE_HTTP_PORT |
8080 |
HTTP management/metrics frontend. Loopback only by design. |
KVP_L1_SIZE_GB |
64 |
Host-RAM L1 buffer size between vLLM and NVMe. |
KVP_CHUNK_SIZE |
256 |
KV chunk granularity in tokens (documented production default). |
KVP_PERSIST_ENABLED |
true |
Keep data files at shutdown. Setting false deletes them — breaks the main requirement. |
KVP_CACHE_BYPASS |
0 |
1 = persistent cache disabled (A/B comparisons). |
KVP_NAMESPACE_HINT |
(empty) | Human label only; does NOT affect compatibility. |
Every value participates in the cache namespace. Any change makes old entries incompatible (a cache miss — never wrong data). Empty value = error.
| Variable | Value form | Notes |
|---|---|---|
KVP_MODEL_ID |
HF id | e.g. XiaomiMiMo/MiMo-V2.6-Pro-MOPD
|
KVP_MODEL_REVISION |
full 40-hex SHA | the pinned weights commit |
KVP_TOKENIZER_NAME |
name/path | usually same repo |
KVP_TOKENIZER_REVISION |
full 40-hex SHA | same as weights unless deliberately diverged |
KVP_KV_CACHE_DTYPE |
resolved dtype | the ENGINE's KV cache dtype, spelled as in vLLM's CacheDType: float16 / bfloat16 / fp8*; auto is rejected. The aliases fp16/bf16 are NOT accepted by the pinned build (vllm 8e2beaa3) and the engine refuses to start with them (preflight warns). This is the KV format vLLM computes — it is NOT the L2 storage codec (see LMCACHE_L2_SERDE below). fp8 here is a legitimate engine dtype and is NOT related to lossy L2 compression. |
KVP_BLOCK_SIZE |
int | KV block size in tokens (from engine log) |
KVP_NUM_KV_HEADS |
int | from model config |
KVP_HEAD_DIM |
int | from model config |
KVP_TP_SIZE |
int | 4 for this deployment |
KVP_CONNECTOR |
name | LMCacheMPConnector |
KVP_CONNECTOR_VERSION |
pin string | e.g. vllm-<commit>@lmcache-mp
|
KVP_MAX_MODEL_LEN |
int | 1048576 |
KVP_CHAT_TEMPLATE_DIGEST |
64 hex | computed by preflight from the model dir |
KVP_LORA_ADAPTERS |
(optional) | comma-separated <adapter-id>@<revision> actually attached at serve time. Empty = no LoRA. Changes the namespace when set — two deployments with different LoRA must not share cached KV. |
KVP_MULTIMODAL_COMPONENTS |
(optional) | comma-separated <component-id>@<revision> actually served (vision/audio encoders). Empty = none. Changes the namespace when set.
|
KVP_ROPE_CONFIG_DIGEST |
(optional, 64 hex) | sha256 of the RoPE config in effect (scaling/theta/sections). Empty = unchanged/default. Changes the namespace when set — a RoPE change alters token positions, so old KV must not be reused. |
KVP_SPECULATIVE_CONFIG |
(optional) | stable label for active speculative/MTP/draft settings (e.g. mtp-draft@rev, num_speculative_tokens=2). Empty = disabled. Changes the namespace when set.
|
All four advanced fields are read by kvpersist.identity and folded into the
cache namespace and model_name. Leaving them empty is legal ("not
applicable"); setting one splits the namespace (safe cache miss).
| Variable | Default | Meaning |
|---|---|---|
LMCACHE_HOST / LMCACHE_PORT
|
127.0.0.1 / 5555
|
ZMQ request endpoint vLLM connects to |
LMCACHE_HTTP_HOST / LMCACHE_HTTP_PORT
|
127.0.0.1 / 8080
|
management frontend — keep loopback |
LMCACHE_L1_SIZE_GB |
64 |
L1 RAM tier |
LMCACHE_CHUNK_SIZE |
256 |
chunk granularity (tokens) |
LMCACHE_EVICTION_POLICY |
LRU |
both tiers |
LMCACHE_EVICTION_TRIGGER_WATERMARK |
0.9 |
L2 soft threshold |
LMCACHE_EVICTION_RATIO |
0.1 |
reclaim per cycle |
LMCACHE_L2_SERDE |
none |
L2 codec, lossless only: none = byte-exact raw KV; zstd / lz4 = lossless compression (1.1–1.4x estimate). Lossy codecs (fp8 / turboquant / lbq) are refused at startup. The codec is part of the cache-identity namespace. |
LMCACHE_LOG_LEVEL |
DEBUG |
cache-service log verbosity. DEBUG is required for hardware acceptance (scripts/test-recovery.sh): the L2 store/load evidence lines (L2 store completed: adapter N, N succeeded, L2 prefetch load completed: request N, N loaded, Prefetch request completed (L1+L2)) are DEBUG-level in the pinned LMCache 0.5.3 — without it the acceptance regexes find nothing and the run is INCONCLUSIVE. Use INFO for a smaller log when no acceptance run is planned. |
KVP_L2_COMPRESS_LEVEL |
3 |
zstd level (1–19, lz4 ignores it) |
USE_LOSSLESS_SERDE |
(unset) | operator switch: 1/zstd = zstd, lz4 = lz4, 0/empty = keep LMCACHE_L2_SERDE. Only applied when LMCACHE_L2_SERDE is not exported (env wins). |
KVP_OTLP_ENDPOINT |
(unset) | OTLP gRPC endpoint the cache pushes metrics to (e.g. http://localhost:4320). Telemetry is off by default. |
USE_OBSERVABILITY |
(unset) |
1 = shorthand for KVP_OTLP_ENDPOINT=http://localhost:4320
|
LMCACHE_OTLP_ENDPOINT |
(unset) | same endpoint under the LMCACHE_* name; wins over KVP_OTLP_ENDPOINT
|
Codec dependencies (fail-fast before start, only the selected one is
required): none — nothing; zstd — zstandard only; lz4 — lz4 only.
- Edit
KVP_CACHE_BUDGET_GBincache.env.local. -
bash scripts/stop-cache.sh && bash scripts/start-cache.sh— data is kept. -
bash scripts/budget-report.sh statusto confirm the new budget.
Raising it needs free disk first. Lowering it below current usage rejects new stores immediately; eviction then sweeps down to the target.
bash scripts/budget-report.sh quota --set-namespace=NAME --gb 200
bash scripts/budget-report.sh quota # listNamespace names come from budget-report.sh status (field namespaces); use the
--namespace=NAME form because real names often start with -.
What quotas do and do not do. Quotas are a policy and accounting layer:
they record a per-namespace limit and let the reports show which namespace is
exceeding it. They are NOT wired into the production write path — the
BudgetLedger.admit() decision is not invoked by the LMCache adapter, so a
namespace can exceed its quota and nothing will reject the write. The only hard
capacity bound enforced on writes is the adapter's max_capacity_gb
(KVP_CACHE_BUDGET_GB) for L2 as a whole. Treat quotas as planning/accounting
tooling until an integration test proves enforcement.
Exported variables always win over the files. Example for a one-session change:
KVP_CACHE_BUDGET_GB=500 bash scripts/start-cache.shTerminal exports do not persist — repeat them in new terminals (shell behaviour, not a bug).