Skip to content

Quick Start

Jump to bottom
sikkatu edited this page Nov 2, 2026 · 8 revisions

English | Русский | 中文

Quick Start

One linear path: install → configure → launch → verify. Every command works as-is; setup-identity.sh (step 3) fills and verifies all identity values automatically, so there is no manual revision field to paste. The one strongly recommended manual setting is KVP_CACHE_DEVICE (step 2) so the cache provably sits on the intended NVMe device -- without it, verification runs end INCONCLUSIVE rather than PASS, but nothing blocks a first run.

All commands run on the host with the GPUs (and the NVMe that will hold the cache), as the user that runs Docker. Not inside Docker, not on another machine. REPO below means the clone directory — the commands name it explicitly so nothing has to be added.

0. Preconditions

  • Debian host with the 4× RTX PRO 6000, Docker, the pinned vLLM image and the local model directory of MiMo-V2.6-Pro.
  • Local NVMe with at least the cache budget free (default 1.5 TB); the real bound is free space on the cache disk on the shared VM volume.
  • The production recipe of sikkatu/recipes available on disk (the launcher uses from it only tp4.env.example (the IMAGE digest) and patches/ + MANIFEST.sha256 (mounted read-only); the recipe itself is never modified).

1. Get the repository

git clone https://github.com/sikkatu/kv-persistence ~/kv-persistence
cd ~/kv-persistence
REPO=~/kv-persistence

2. Configure the cache

cp "$REPO/configs/generic/cache.env" "$REPO/configs/generic/cache.env.local"

KVP_CACHE_ROOT in the copy (default /var/lib/kv-persistence/l2) is the NVMe path for the cache. Keep it unless your NVMe lives elsewhere; then edit that one line to the NVMe path. KVP_CACHE_BUDGET_GB (default 1500) is the hard budget.

3. Pin the model identity

One command fills and verifies ALL mandatory identity fields (KVP_MODEL_REVISION, KVP_TOKENIZER_REVISION, KVP_CHAT_TEMPLATE_DIGEST):

bash "$REPO/scripts/setup-identity.sh" "$HOME/models/<your-model-dir>"

Deterministic fingerprints are derived from the files actually on disk (weight index, tokenizer files, chat template via the single documented algorithm) and written straight into configs/mimo-v2.6-pro/identity.env; the script ends by verifying the written configuration (the cache namespace must compute). No manual paste step. If it exits non-zero, the identity is not ready — read its message.

Already tracking the upstream commit SHA? Set it first and only the remaining fields are derived:

export KVP_MODEL_REVISION="$(curl -fsSL https://huggingface.co/api/models/XiaomiMiMo/MiMo-V2.6-Pro-MOPD | python3 -c 'import json,sys; print(json.load(sys.stdin)["sha"])')"
bash "$REPO/scripts/setup-identity.sh" "$HOME/models/<your-model-dir>"

4. Install the LMCache CLI

python3 -m venv ~/.venvs/lmcache
~/.venvs/lmcache/bin/pip install --upgrade pip
~/.venvs/lmcache/bin/pip install -r "$REPO/requirements-lmcache.txt"
export PATH="$HOME/.venvs/lmcache/bin:$PATH"

Python 3.12 is required to create this venv (the production image's runtime, and what requirements-lmcache.txt targets). On Debian/Ubuntu (tested on Debian 12) the system Python is 3.11: install a 3.12 interpreter (e.g. from bookworm-backports or pyenv) and use python3.12 -m venv ~/.venvs/lmcache instead. python3-venv must be installed separately on Debian.

Persist PATH only if you want the lmcache command in new terminals; the kv-persistence scripts call lmcache resolved through PATH. In every NEW terminal repeat the exports (REPO, IMAGE, MODEL_DIR, PATH) — shell variables do not survive a terminal change. If the pip wheel mismatches the production image's ABI (import errors), see Troubleshooting — the fallback runs lmcache server from the pinned image itself.

5. Preflight

bash "$REPO/scripts/preflight.sh"

preflight PASSED means go on. Any [!!] tells you exactly what to fix. Run it once with the model directory to get the chat-template digest automatically:

KVP_MODEL_DIR="$HOME/models/<your-model-dir>" bash "$REPO/scripts/preflight.sh"

No manual copying is needed here: setup-identity.sh (step 3) already wrote the digest; the command above simply confirms it matches the model dir.

6. Start the persistent cache service

bash "$REPO/scripts/start-cache.sh"

The service runs detached (setsid nohup), so it survives SSH disconnects and closing this terminal; log and pid live under KVP_CACHE_ROOT. Previously cached data is preserved — startup never wipes the cache.

Optional: lossless compression

By default chunks are stored byte-exact (LMCACHE_L2_SERDE=none) — no transform, no extra dependency. To store them with lossless zstd or lz4 compression instead (roughly 1.1–1.4x smaller on real KV payloads):

USE_LOSSLESS_SERDE=1 bash "$REPO/scripts/start-cache.sh"     # zstd
USE_LOSSLESS_SERDE=lz4 bash "$REPO/scripts/start-cache.sh"   # lz4

The same switch must be used for step 7 (it is part of the cache-identity namespace). zstd needs only the zstandard package, lz4 only lz4 — a check before start requires exactly the selected codec, and a none cache needs no compression package at all. Every frame carries a CRC32: corruption is detected and reported as a cache miss, never returned as wrong KV.

Optional: Prometheus + Grafana monitoring

bash "$REPO/scripts/start-observability.sh"
# Grafana -> http://localhost:3000 (anonymous Admin, no login)

Then start the cache with USE_OBSERVABILITY=1 so metrics flow to the collector. The stack ports are bound to 127.0.0.1 only. The cache works fully without the stack.

7. Launch vLLM with the cache

Preconditions for this step (all three required):

  1. The production recipe must be on disk at the default location the launcher expects (or point RECIPE_DIR at it): 
    git clone https://github.com/sikkatu/recipes ~/recipes
    The launcher reads tp4.env.example only to reuse the recipe's pinned IMAGE digest and its checksum-verified patches/ (mounted read-only); the recipe itself is never modified.
  2. API_KEY is required. The launcher reads it from live/api_key in the current directory (or an already-exported API_KEY). If you have none yet: 
    mkdir -p live && openssl rand -hex 24 > live/api_key && chmod 600 live/api_key
  3. MODEL_DIR must point at the SAME local model directory used in step 3 — the identity built there and the weights loaded here must match.
cd "$REPO"
export IMAGE="$(bash "$REPO/scripts/print-recipe-image.sh" "$HOME/recipes/recipes/rtx-pro-6000/mimo-v2.6-pro-hotsplit")"
export MODEL_DIR="$HOME/models/<your-model-dir>"
bash "$REPO/scripts/run-vllm-with-persist.sh"

The launcher is based on the production recipe's argument baseline and adds the KV connector, the cache-identity namespace mount, --enable-prompt-tokens-details and a pinned --block-size — it is NOT a byte-for-byte mirror (the recipe's KV_PERSIST=1 branch now matches it on those four points; see Recipes Integration). Recipe patches are checksum-verified and mounted read-only.

8. Verify

bash "$REPO/scripts/verify-engine-identity.sh"   # mandatory gate: engine-config == identity pins
bash "$REPO/scripts/test-recovery.sh"            # acceptance: PASS / INCONCLUSIVE / FAIL (exit 0/5/1)

test-recovery.sh invokes verify-engine-identity.sh BEFORE touching the cache: if any identity pin disagrees with the running engine, cache reuse is refused (exit 4). If a value cannot be verified (e.g. KVP_MODEL_DIR unset), the run is INCONCLUSIVE (exit 5) — never a silent PASS. The physical-NVMe check uses KVP_CACHE_DEVICE; set it in cache.env.local so PASS is achievable.

Warm → restart vLLM → re-send, then warm → restart both services → re-send. The run prints per-phase cached_tokens and retrieval evidence into benchmarks/results/. Disk-backed recovery is proven by cached_tokens > 0 plus L2/disk read events in the cache log after each restart, corroborated by physical NVMe read counters — speed alone is never accepted as proof.

Next

Clone this wiki locally