Skip to content

Persistence and Recovery

Jump to bottom
sikkatu edited this page Nov 2, 2026 · 5 revisions

English | Русский | 中文

Persistence & Recovery

The project's acceptance criterion: after a restart, the same conversation must reuse KV blocks recovered from NVMe — demonstrably faster, and proven by disk evidence, not by speed alone.

Evidence checklist

After each restart of the same prompt, ALL of the following must hold:

  1. usage.prompt_tokens_details.cached_tokens > 0 in the response (the engine reused cached prefill work).
  2. L2/disk read events appear in KVP_CACHE_ROOT/lmcache-server.log after the restart -- lines matching the retrieval pattern (retriev|loaded|restore with L2|disk|secondary|file), not merely any cache hit line (the cache actually serves FROM ITS STORE), corroborated by physical NVMe read counters.
  3. The L2 bytes did NOT grow to re-create the same data (bash scripts/budget-report.sh status before/after).
  4. TTFT/prefill drops vs the comparable cold baseline.
  5. The answer text is still correct (spot-check).

A speedup alone does NOT satisfy the milestone.

Strict protocol (v2) — what test-recovery.sh enforces

The reviewed protocol, implemented mechanically in scripts/test-recovery.sh (its PASS/FAIL is a machine verdict — exit code 0 only when every check holds):

  1. Log-position markers — a byte offset into lmcache-server.log is taken BEFORE each request; only events appended after that marker count as evidence for that scenario. Nothing from earlier runs can leak in.
  2. Store-before-restart — the warm request must demonstrably WRITE blocks to the L2 store (directory bytes increase + store log events). No pre-stored data means later "recovery" proves nothing and the test FAILS.
  3. L2 vs L1 distinction — after each restart the test requires log events of reads FROM THE DISK tier (not generic "Retrieved" lines that could be RAM hits). cached_tokens > 0 alone is explicitly NOT accepted.
  4. Hard-fail readiness — if the API does not return within the wait budget after a restart, the test FAILS immediately; it never continues on a dead service.
  5. Alternative explanations excluded — reuse with zero post-restart L2 read events is classified as RAM/prefix-cache only and marked a failure.
  6. Machine verdict — PASS ⇒ exit 0; any failed check ⇒ FAIL with exit 1 and a failures.txt listing each reason.

Scenario A — vLLM restarts, cache service stays up

bash scripts/test-recovery.sh

The script warms the cache with a long prompt, restarts the vLLM container and re-sends the identical request. Per-phase summaries land in benchmarks/results/<UTC-ts>-recovery/.

Manual equivalent:

# warm
bash scripts/benchmark.sh cold 3000
# restart vLLM only, wait for the API to answer again
docker restart mimo-pro-tp4-kvp
until curl -fsS --max-time 3 -H "Authorization: Bearer $(head -n1 live/api_key | tr -d '[:space:]')" http://127.0.0.1:30007/v1/models >/dev/null 2>&1; do sleep 10; done
bash scripts/benchmark.sh after-vllm-restart 3000     # cached_tokens must be > 0

Scenario B — both services restart, NVMe data kept

test-recovery.sh continues with stop-cache.sh → start-cache.sh → docker restart → the same request. Here BOTH the cache service and vLLM come back from zero: recovery proves the on-disk index lazy rebuild documented for nixl_store_dynamic. Record retrieval/lookup recovery time from the cache log.

Cold vs warm vs recovered — how to tell

Observation Meaning
cached_tokens = 0 cold (no reuse)
cached_tokens > 0, NO L2/disk read events warm — served from L1/RAM (the cache service survived). This is Scenario A's EXPECTED outcome: the request reuses KV still resident in L1, so nothing has to come from NVMe. It is a memory hit, not a persistence failure.
cached_tokens > 0 AND L2 read events after a FULL restart (both services, empty L1) recovered from NVMe (this is what proves persistence)

Scenario A vs B expectations (review round 18): after a vLLM-only restart the cache service and its L1 are still alive, so a healthy system logs Prefetch request completed (L1+L2): N/N retained keys (N L1, 0 L2) — a pure L1 hit. The acceptance script must NOT fail on that; only Scenario B (full restart) requires L2 prefetch load completed / (N L1, N L2) evidence.

What invalidates the cache (by design)

Changing any identity field — model revision, tokenizer, KV dtype, block layout, TP size, chat template — moves the deployment to a NEW namespace: old data is not addressed. That is isolation, not deletion. Reverting the change restores reuse of the old data.

Re-running the tests

scripts/test-recovery.sh appends a fresh results directory per run; nothing is overwritten. Median of ≥3 repeats per configuration is the number to quote (see the benchmark page).

Clone this wiki locally