Storage and Eviction

English | Русский | 中文

Storage & Eviction

Budget model

The 1.5 TB default is deliberately generous: more prefill history stays on NVMe, so reopening a conversation saves recomputation time and energy on first load.

How long can a conversation be?

Do not confuse two limits:

• Per-conversation token limit (~1M) — bounded by the model's context window (max_position_embeddings in the model's config.json), NOT by this cache.
• How much history the cache retains on disk — bounded only by KVP_CACHE_BUDGET_GB. There is no token limit here: beyond 1M tokens, only the per-context reuse boundary matters (see Architecture on prefix reuse), not kv-persistence itself.

Guard	Default	Enforced by
Hard cap	KVP_CACHE_BUDGET_GB=1500 GB (1.5 TB)	max_capacity_gb of the L2 adapter — stores above it are REJECTED
Physical ceiling	whatever the shared VM disk (same volume holds Docker, the model and the OS) device holds (preflight-verified)	KVP_CACHE_DEVICE free-space check
Soft threshold	0.9 of budget	trigger_watermark — automatic eviction starts here
Reclaim target	~0.81 of budget after one cycle (0.9 x 0.9 of used)	eviction_ratio 0.1 of used capacity
Per-namespace quota	optional, quotas.json	kvp-admin quota policy layer

The budget covers ONLY persistent KV-cache data. Model weights, the OS and other files are never accounted here and are never touched by any kv-persistence script.

Capacity accounting

• One file per KV chunk object directly under KVP_CACHE_ROOT (temp file + atomic rename). The nixl_store_dynamic adapter writes flat *.bin files — there are no per-namespace subdirectories; the identity namespace is encoded in the file name prefix (the sanitized model name).
• No separate on-disk index — the deterministic file names ARE the index; it is rebuilt lazily at startup. used_bytes is the sum of the real cache file sizes (plus legacy kvp-* dirs if any); there is no separate "index overhead" line.
• Admission is enforced by the LMCache adapter via max_capacity_gb (the hard L2 bound): stores above it are rejected. The repo's own BudgetLedger computes the same numbers for reporting and planning but is NOT wired into the production write path — do not rely on it to block writes.
• Restart accounting (NEEDS TESTING, review fix): after a cache-service restart the in-memory index is rebuilt LAZILY as files are touched — the disk is not scanned at startup, so used_bytes starts from zero while the real files are still there. Do NOT assume pre-restart files count against max_capacity_gb until a hardware run confirms the pinned build feeds restored sizes back into get_usage(). Required test: fill cache → restart cache service → keep writing → verify used_bytes grows and stays bounded.
• Crash leftovers: after a hard kill (SIGKILL, OOM, power loss) the adapter can leave <final>.bin.tmp.<uuid> files it never got to rename. They are NOT part of the cache index (never returned as KV) but they DO consume disk, so status reports them separately (tmp_files, tmp_files_bytes). Remove them with purge --tmp --yes (keeps every real cache file) or purge --all --yes. In-flight writes are protected by an active-write registry (kvpersist.tmp_guard): every store registers a marker before its tmp file exists and clears it after the atomic rename. A marker is LIVE while its writer process is alive (judged by (pid, process start-time), so PID reuse cannot fake it) — a registered write is never deleted, however long it runs (an hour-plus write stays protected; liveness, not age, is the criterion). When liveness cannot be determined (no /proc, unreadable marker) the marker is kept: safer to retain one tmp file too long than to delete a live store. Markers whose writer provably exited are swept after an hour. As a fallback for writers without the registry, a 15-minute mtime window additionally skips recent tmp files.

Report at any time:

bash scripts/budget-report.sh status

Fields: used_bytes, free_bytes, usage_fraction, soft_pressure, per_namespace_bytes, evictions, reclaimed_bytes, rejected_writes, tmp_files, tmp_files_bytes.

Automatic eviction (mandatory, always on)

• A background thread watches get_usage(). Above trigger_watermark the policy evicts until usage drops by eviction_ratio.
• The policy that actually runs in production is LRU over stored chunk keys (adapter-native, inside LMCache). Long frequently reused histories are protected only by LRU's recency rule — there is no additional value-based guard active yet.
• Value-based ranking (implementation/kvpersist/retention.py) is a separate planning mechanism, NOT integrated into the production eviction path: it orders entries by recency (24 h decay), reuse count, prefix length and storage cost, and is covered by unit tests — but LMCache's own LRU is what deletes chunks today. Do not assume long valuable prefixes survive because of it.
• Storage-pressure warnings: soft_pressure: true in the status report; the launcher also warns at startup when filesystem free space is below the budget.

Manual cleanup — explicit only

bash scripts/budget-report.sh status                    # see real namespace names
bash scripts/budget-report.sh purge --namespace=NAME --yes
bash scripts/budget-report.sh purge --all --yes
bash scripts/budget-report.sh purge --tmp --yes         # crash leftovers only

The cache layout is FLAT: *.bin files sit directly under KVP_CACHE_ROOT, and a namespace is the file-name prefix (the sanitized model name, e.g. --models--MyModel--a84fadf1). Get the actual names from status → namespaces — never guess them. Because real names often START WITH -, pass the value as --namespace=NAME or as a positional argument (purge --namespace=NAME --yes / purge NAME --yes), never as --namespace NAME. Matching is EXACT (no substring matching). Requires --yes. Deletion is confined to cache files inside KVP_CACHE_ROOT; everything else is rejected with exit code 2. Startup NEVER wipes the cache automatically.

Deduplication & shared blocks

LMCache stores chunks under content-derived keys: identical chunk content ⇒ identical key ⇒ one stored object, not two. Two conversations with the same stable prefix share their objects physically:

• Same identity namespace ⇒ shared keys; different namespaces never collide.
• Evicting one entry removes only its keys; shared objects remain while any other key maps to them. If a shared object is eventually evicted (LRU considers the key unused), concurrent correctness is preserved: a query that misses simply recomputes — a liveness trade-off, never a correctness one.
• Storage saved by dedup is measurable from used_bytes before/after: the content-key MODEL is covered by tests/unit/test_dedup_model.py (simulation, no LMCache), and tests/integration/test_l2_dedup_real.py counts the REAL L2 files against a live lmcache server (shared prefix chunks written once).

Namespaces and cleanup policy

The cache layout is FLAT: one *.bin file per KV chunk directly under KVP_CACHE_ROOT, and the identity namespace is the file-name prefix (the sanitized model name, e.g. --models--MyModel--a84fadf1). A revision change creates a NEW namespace (a new file-name prefix); old ones are NEVER auto-deleted (documented policy: cleanup is an explicit admin decision):

bash scripts/budget-report.sh status      # list namespace names (the "namespaces" field)
bash scripts/budget-report.sh purge --namespace=NAME --yes

Use --namespace=NAME (the = form) or the positional form — real namespace names often START WITH -, and --namespace NAME would be parsed as an option.

Changing the limit

1. Edit KVP_CACHE_BUDGET_GB in configs/generic/cache.env.local.
2. bash scripts/stop-cache.sh && bash scripts/start-cache.sh (data kept).
3. Verify with bash scripts/budget-report.sh status.