Skip to content

Recipes Integration

Jump to bottom
sikkatu edited this page Nov 2, 2026 · 4 revisions

English | Русский | 中文

Recipes Integration

The sikkatu/recipes production launchers carry an opt-in switch that attaches the persistent cache without changing the recipe's plain behaviour.

How it works

The launchers (scripts/run-tp4-uva.sh, scripts/run-tp4-hotsplit.sh of recipes/rtx-pro-6000/mimo-v2.6-pro-hotsplit) contain a KV-PERSIST block:

KV_PERSIST="${KV_PERSIST:-0}"
if [ "$KV_PERSIST" = "1" ]; then
  ... derives the cache-identity namespace ...
  MODEL_ARGS=(--model "/models/$KV_PERSIST_NAMESPACE")
  MODEL_ARGS+=(--kv-transfer-config "$KV_TRANSFER_CONFIG")
  MODEL_ARGS+=(--enable-prompt-tokens-details)
  MODEL_ARGS+=(--block-size "${BLOCK_SIZE:-16}")
fi
  • OFF by default — with KV_PERSIST unset or 0 the recipe behaves byte-identically to before: no connector, no namespace remount, no new arguments. The plain run needs no MODEL_DIR and no kv-persistence install.
  • ON (KV_PERSIST=1) the block does four things (review fix, round 19):
    1. Cache-identity namespace — MODEL_DIR is mounted at /models/<namespace> (NOT /model). LMCache keys every L2 object by model_config.model (the --model value), so a fixed /model path made EVERY deployment share one namespace: different revisions, KV dtypes or geometries could read each other's KV blocks. With the namespace path, each identity gets its own namespace and an incompatible deployment gets a SAFE cache miss — provided the namespace is derived from the engine's ACTUAL arguments (review fix: real KV_CACHE_DTYPE, MAX_MODEL_LEN, block size and TP size are passed to the CLI as KVP_* overrides, and auto is resolved to the model's effective dtype first). The namespace is derived from python3 -m kvpersist.cli model-name (the verified identity hash) when a kv-persistence checkout is found (KVP_REPO_ROOT or ~/kv-persistence), else from a deterministic hash of the KV-compatibility fields (config.json, weight indexes + shard sizes, tokenizer files, chat template, kv dtype, max-model-len, block size, TP size). MODEL_DIR is therefore required for KV_PERSIST=1. If you set KV_PERSIST_NAMESPACE manually, YOU own the isolation guarantee.
    2. --kv-transfer-config — the LMCache connector (same JSON as kv-persistence's own launcher).
    3. --enable-prompt-tokens-details — without it usage.prompt_tokens_details is never reported, cached_tokens (the only machine-readable reuse counter) is unavailable, and kv-persistence's acceptance run (scripts/test-recovery.sh) can never PASS.
    4. --block-size pinned (16 by default) — the kv-persistence identity gate (verify-engine-identity.sh) needs a verifiable engine argument.

Knobs are documented in tp4.env.example / .ru / .zh (identical names, values and order in all three languages): KV_PERSIST, KV_PERSIST_HOST, KV_PERSIST_PORT, KV_PERSIST_NAMESPACE, BLOCK_SIZE.

Enable it — full flow

  1. Install and start the cache service per Quick Start steps 1–6.
  2. In the recipe's tp4.env add: 
    KV_PERSIST=1
  3. Launch the recipe exactly as usual: 
    bash scripts/run-tp4-uva.sh

The launcher prints two confirmation lines when the block is active: KV_PERSIST=1: attaching LMCache connector … and KV_PERSIST=1: cache-identity namespace -> /models/<hash> ….

Verify the engine identity before reuse

The recipe's block cannot run kv-persistence's verify-engine-identity.sh automatically (the recipe has no kv-persistence checkout path wired in), so run it manually BEFORE trusting any cache reuse:

bash ~/kv-persistence/scripts/verify-engine-identity.sh <container-name>

Exit 0 = identity verified (cache reuse allowed within that scope). Exit 4 = MISMATCH (do NOT reuse the cache — reconcile configs/mimo-v2.6-pro/identity.env.local with the engine args and re-run). Exit 5 = INCONCLUSIVE (fix the value sources first). The gate compares the container's startup args and the served model's config.json; with KV_PERSIST=1 the launcher pins --block-size and the namespace mount, so all fields are checkable. The gate only reads engine config — it does not alter production behaviour.

Port names on both sides

Three different variables name the same endpoint (review note):

Side Host Port
cache service (kv-persistence configs/generic/lmcache-server.env) LMCACHE_HOST LMCACHE_PORT
cache service (kv-persistence configs/generic/cache.env docs alias) KVP_LMCACHE_HOST KVP_LMCACHE_PORT
recipe launcher (tp4.env) KV_PERSIST_HOST KV_PERSIST_PORT

The recipe side is what vLLM connects WITH; the cache side is what the service listens ON. If you change the port on the cache side (LMCACHE_PORT), you MUST set the same value in KV_PERSIST_PORT on the recipe side — otherwise vLLM cannot reach the cache and the connector silently retries a dead endpoint. Same for the host. Defaults match (127.0.0.1:5555).

Two paths, one cache — what matches and what differs

kv-persistence/scripts/run-vllm-with-persist.sh (the specialized launcher) and the recipe's KV_PERSIST=1 branch now agree on everything that determines cache compatibility:

Parameter kv-persistence launcher recipe KV_PERSIST=1 plain recipe run
--model /models/<namespace> /models/<namespace> /model or MODEL_ID
--kv-transfer-config yes (LMCacheMPConnector) yes (same JSON) no
--enable-prompt-tokens-details yes yes no
--block-size 16 (pinned) 16 (pinned via KV_PERSIST_BLOCK_SIZE) not passed (engine default)
MAX_NUM_SEQS 2 2 2 (env; user-verified: no speed loss, more convenient default)
KV_CACHE_DTYPE auto (resolves to model dtype) auto (resolves to model dtype) auto (env)
MAX_MODEL_LEN 1048576 1048576 1048576 (env)
identity gate automatic (refuses on mismatch) manual reminder none
The specialized launcher additionally runs the identity gate automatically and
REFUSES to attach the cache on mismatch; the recipe branch reminds you to run it
manually. **A value difference in any identity field (dtype string included) is
a SAFE cache miss** — the namespace hash changes, so the two paths simply stop
sharing cached blocks. They never read wrong KV; they just stop reusing. If you
want reuse between both paths, keep the values identical.

Note on KV_PERSIST_NAMESPACE override: if you set KV_PERSIST_NAMESPACE explicitly (skipping the derived namespace), YOU own the isolation guarantee — the launcher cannot verify that the namespace matches the engine's actual configuration. Use the override only when you understand the identity fields.

Disable / rollback

Remove the KV_PERSIST=1 line (or set 0) and relaunch. That is the whole rollback: the recipe is untouched by design, and cached data on NVMe is simply ignored while disabled.

Ordering rule

The cache service must be running BEFORE the launcher attaches. Start order: start-cache.sh first, recipe launcher second. If the service is down when vLLM starts, remove KV_PERSIST=1 and use the plain recipe (or start the cache and relaunch) — never leave vLLM retrying a dead connector in production.

Which launcher to choose

Launcher When
run-tp4-uva.sh UVA baseline (verified working recipe path)
run-tp4-hotsplit.sh hot/cold expert split (dynamic/static modes)
kv-persistence/scripts/run-vllm-with-persist.sh when you want the identity gate enforced automatically and the full kv-persistence tooling (preflight, benchmark, test-recovery)

All three accept the same KV-PERSIST knobs (the specialized launcher reads them from kv-persistence's own config instead). Use whichever you run today.

How the launcher block is tested

Contract tests (string checks) plus BEHAVIOURAL tests in sikkatu/recipes/tests/test_kv_persist_behaviour.py: both launchers are actually EXECUTED against stub docker/nvidia-smi/curl/python3 binaries, and the stub python3 records the KVP_* overrides it received from the block. Covered: the namespace follows the engine's real arguments (fp8 vs bfloat16 yield different namespaces), auto resolves to the model's effective dtype, a CLI failure aborts the launch with the real error (no silent fallback), KV_PERSIST_NAMESPACE override is honoured, the plain (KV_PERSIST=0) run derives nothing, KV_PERSIST_BLOCK_SIZE reaches both the CLI and --block-size, and the model-name prefix is stable across launchers (mimo-pro/<hash> on both paths). kv-persistence additionally has tests/unit/test_launcher_namespace_coherence.py, which executes the real default-expression line of run-vllm-with-persist.sh — a regression to basename $MODEL_DIR fails the test.

What is NOT in the recipe

The cache server itself, capacity policy, identity pins and administrative tools live in kv-persistence; its wiki is the reference for setup, persistence verification and rollback. Note that recovery acceptance (scripts/test-recovery.sh) still requires a GPU-host run — see kv-persistence docs/final-report.md for the honest status. The recipe's contract tests verify the launcher carries the machinery; they do not prove real NVMe recovery by themselves.

Clone this wiki locally