Skip to content

Troubleshooting

Jump to bottom
sikkatu edited this page Nov 2, 2026 · 8 revisions

English | Русский | 中文

Troubleshooting

First, look at three things:

bash scripts/budget-report.sh status        # usage, namespaces, disk free
tail -n 100 "$KVP_CACHE_ROOT/lmcache-server.log"   # cache service
docker logs --tail 100 mimo-pro-tp4-kvp     # engine (kv-persistence launcher)

(KVP_CACHE_ROOT comes from configs/generic/cache.env.local.)

Preflight reports identity problems

  • KVP_MODEL_REVISION must be a full 40-hex SHA — paste the commit SHA, not a branch name (see Quick Start step 3).
  • KVP_CHAT_TEMPLATE_DIGEST must be 64 hex — run preflight once with KVP_MODEL_DIR="$HOME/models/MiMo-V2.6-Pro-MOPD" set; copy the printed value.

lmcache CLI missing or ABI mismatch

ERROR: lmcache CLI missing — install per Quick Start step 4. If import lmcache fails inside your environment with undefined-symbol errors, that pip wheel is built for a different PyTorch/CUDA stack. Fallback: run the server from the production image itself:

docker run --rm --entrypoint lmcache "$IMAGE" server --help

If that works, adapt scripts/start-cache.sh's lmcache server line to run inside docker run with the same arguments (ports/paths stay identical).

The cache service starts but vLLM cannot connect

  1. Start order: start-cache.sh BEFORE the vLLM launcher (run-vllm-with-persist.sh refuses otherwise).
  2. Inspect the service log's error line; both processes use the same host network — do not move the request port without updating both sides.
  3. With the recipe integration (KV_PERSIST=1): the launcher prints KV_PERSIST=1: attaching LMCache connector … — if absent, the knob did not reach the launcher (check tp4.env and whether variables were exported).

No cache hits after a restart (no TTFT drop)

Check in order:

  1. cached_tokens in the response is > 0.
  2. grep -c Retrieved "$KVP_CACHE_ROOT/lmcache-server.log" grows during the request; entries after restarts are the disk-recovery proof.
  3. used_bytes did NOT grow to re-create the same data (budget-report.sh status before/after).
  4. Identity unchanged — rerun scripts/preflight.sh; a changed revision/dtype/template silently moves to a NEW namespace (by design).
  5. KVP_PERSIST_ENABLED is true (start-cache.sh warns otherwise).

Disk full / rejected stores

free_bytes: 0 in the report, or the adapter logs rejections at the hard cap — raise KVP_CACHE_BUDGET_GB or purge old namespaces explicitly. The hard cap is max_capacity_gb in the L2 adapter: once usage reaches it, new stores are rejected (serving continues without new cache writes). Eviction at the soft threshold (90 %) usually reclaims space first, but it is NOT a guarantee against filling the DISK itself — the real bound is the free space on the volume that holds KVP_CACHE_ROOT (on the reference host that volume also holds Docker, the model and the OS). Keep a free-space margin, and lower the budget if the disk runs low. Never delete individual files inside the cache root while the service runs.

Eviction seems too eager (or not eager enough)

Raise KVP_EVICT_TRIGGER_WATERMARK (start later) or tune KVP_EVICT_RATIO (reclaim more per cycle). Verify with a tiny isolated budget before expecting effects at 1.5 TB scale. evictions/reclaimed_bytes in the status report show the actual regime.

Little reuse of a returned cache (everything looks cold)

Measure what part of the prompt is actually stable:

python3 "$REPO/implementation/prompt_probe.py" --tokenizer "$MODEL_DIR" \
  --request-file a.json --request-file b.json

with two real request bodies exported from the client. A small stable_prefix_tokens means dynamic values (timestamps, tool-call ids) break the prefix — fix serialization at the source or accept the limit. Truncating history to make caching work is forbidden.

Corrupted file recovery

Corruption is tolerated: a lookup that misses recomputes. To clean deliberately: stop the cache service, bash scripts/budget-report.sh purge --all --yes, start again. Never happens automatically. Frames written with the lossless codec also carry a CRC32: a corrupt or truncated frame is detected at read time and reported as a cache miss — never returned as wrong KV.

Lossless codec problems

  • CodecError: zstandard/lz4 python bindings not installed — the SELECTED codec needs its Python package (zstd → zstandard, lz4 → lz4). Install it in the lmcache venv, or fall back to LMCACHE_L2_SERDE=none (byte-exact raw, no extra dependency at all). The check runs before the server starts and never requires an unused codec: a none cache needs no compression package.
  • LMCACHE_L2_SERDE=... is LOSSY and is not supported — expected: this project is lossless-only. Use none, zstd or lz4.
  • Switched the codec and cache hits disappeared — expected and safe: the codec is part of the cache-identity namespace, so old chunks become a cache miss. Re-store them (or switch back) — never a wrong read.
  • Compressed chunks not smaller on disk — the codec only shrinks what actually compresses; incompressible blocks are stored raw and padded to the 4 KiB page size. On tiny chunks the per-chunk header (28 B) can dominate.

Disk usage higher than expected, no cache files to explain it

After a hard kill (SIGKILL, OOM, power loss) the adapter can leave crash leftovers: <final>.bin.tmp.<uuid> files it never got to atomically rename. Upstream sweeps them only on a CLEAN shutdown, so they linger and consume disk while staying invisible to the cache index (they are never returned as KV). Check with bash scripts/budget-report.sh status (fields tmp_files and tmp_files_bytes) and remove them with bash scripts/budget-report.sh purge --tmp --yes (keeps every real cache file) or purge --all --yes. Both commands protect in-flight writes with an active-write registry (kvpersist.tmp_guard): every store registers a marker before its tmp file is created and clears it after the atomic rename. Liveness works in BOTH launch modes (review round 18):

  • host mode — the marker records (pid, process start-time) and the writer is alive while that exact process exists (/proc/<pid> + start-time guard against PID reuse);
  • docker mode (KVP_LAUNCH_MODE=docker, no --pid=host) — the marker records the writer's PID-NAMESPACE id; when the marker belongs to a foreign namespace the host CLI cannot check its PID, so liveness falls back to the marker's own HEARTBEAT (a running store refreshes it every 30 s): a marker touched within 2 minutes is alive, older = crash leftover. If liveness cannot be determined at all (no /proc, unreadable marker) the marker is kept: safer to retain one tmp file too long than to delete a live store. Markers whose writer provably exited are swept after an hour. As a second net for writers that do not use the registry (e.g. a bare lmcache server), *.tmp.* files modified within the last 15 minutes are skipped — that window is a HEURISTIC, not an absolute lock.

Grafana shows no data

The monitoring stack is passive until the cache service pushes telemetry. Start the cache with USE_OBSERVABILITY=1 (or KVP_OTLP_ENDPOINT=http://localhost:4320) and check KVP_CACHE_ROOT/lmcache-server.log for the [kvp] banner. In docker mode start-cache.sh rewrites localhost to host.docker.internal automatically. All stack ports are bound to 127.0.0.1 only (loopback contract).

Cache Miss Matrix (why did the cache "stop working"?)

Changing ANY identity field rotates the cache namespace: the old cache entries are not deleted, they become unreachable — which looks like "the cache is broken" but is the isolation design working correctly. Every row below invalidates ALL previously cached KV for that deployment:

You changed... Effect on namespace Reusable old cache?
model weights (content swap, even same file names) SHA-256 of shard bytes rotates NO
model revision (HF SHA) rotates NO
tokenizer (tokenizer.json / tokenizer_config.json) rotates NO
chat_template / chat_template.jinja content digest rotates NO
--kv-cache-dtype (bfloat16 -> float16 etc) rotates NO
--max-model-len rotates NO
--tensor-parallel-size rotates NO
KV block size / head config in model config.json rotates NO
vLLM build / LMCache connector version (KVP_CONNECTOR_VERSION) rotates NO
--enforce-eager / other engine flags that change KV computation order may change block layout → rotates NO
Docker image digest (different vLLM build) rotates NO

Practical advice: pin KVP_MODEL_REVISION via setup-identity.sh output and copy identity.env.local alongside each deployment; expect a namespace change whenever you change engine args and treat "cache is cold after my tweak" as the correct, expected outcome — not a bug.

Clone this wiki locally