Complete the v1 environment migration runbook and validation pipeline

[rickisba]

Purpose

Close out the CUDA 13 / vLLM 0.25.1 / LMCache 0.5.2 migration by turning the successfully validated manual workflow into a maintained runbook and a reproducible, safety-gated validation pipeline.

This PR builds on merged PRs #149 and #151. It does not restructure the repository homepage and does not change serving, routing, queueing, or cache runtime behavior.

Scope

Shared v1 shell activation

Add env/docker/cu130/scripts/activate_v1.sh for every modern service terminal. It:

• sets CACHEROUTE_RUNTIME_PROFILE=v1;
• adds the repository root to PYTHONPATH without duplication;
• sets stable PYTHONHASHSEED and OMP_NUM_THREADS defaults;
• clears the legacy LMCACHE_CONFIG_FILE interface;
• clears PYTORCH_CUDA_ALLOC_CONF for the validated MP startup path;
• refuses accidental execution and must be sourced.

Non-destructive environment check

Add env/docker/cu130/scripts/check_v1_environment.py to validate:

• repository/profile selection;
• Python, PyTorch, vLLM, and LMCache versions;
• CUDA runtime, GPU visibility, and expected GPU count;
• model directory and required scripts;
• Redis connectivity;
• vLLM and LMCache MP endpoints and metric namespaces.

Unavailable services are warnings by default and errors with --require-running.

Staged KDN round-trip validator

Add scripts/validate_v1_kdn_roundtrip.py with four explicit stages:

1. build: register a document and build its KV artifact without manual --match or Redis flush;
2. inspect: validate KID, run_meta.json, manifest, dump count, and payload size;
3. inject: restore the artifact through KVCacheInjector and verify key sets, lengths, payload bytes, and sampled SHA256 values;
4. consume: send the exact builder-style system prefix and verify vLLM external-cache plus LMCache lookup/L2 metrics.

Destructive cleanup is guarded: Redis FLUSHDB requires both --flush-redis and --yes. Injection and consumption remain separate because LMCache and vLLM must be restarted between them to clear L1/GPU state before proving an RESP L2 load.

Documentation closeout

• Update docs/quickstart_v1.md with the complete startup and validation pipeline, Python Redis fallback, correct metrics endpoints, normal no---match registration, and staged acceptance criteria.
• Update env/docker/cu130/README.md to use merged main instead of the historical migration branch, document the new scripts, and record the completed KDN validation.
• Update docs/runtime_compatibility_v1.md to replace the obsolete “cache consumption not yet proven” boundary with the validated result.
• Add docs/v1_migration_closeout.md as the migration baseline, evidence record, safety contract, compatibility checklist, and Definition of Done.

Completed runtime evidence

The validated dataset document produced the following result under TP=8, LMCache chunk size 256, and sha256_cbor:

KDN physical keys saved:              96
Manifest records / dump files:        96 / 96
Payload bytes restored:               1,006,632,960
Missing / extra / size mismatches:     0 / 0 / 0
Sample SHA256 mismatches:              0
LMCache L2 lookup hits:                96 / 96 chunks
LMCache L2 loads completed:            96 chunks
LMCache L2 prefetch failures:          0
LMCache requested / hit tokens:        3072 / 3072
LMCache token hit rate:                100%
vLLM externally cached tokens:         3072

The physical layout is consistent:

96 Redis keys / 8 TP ranks = 12 logical chunks
12 chunks * 256 tokens     = 3072 cached tokens

This closes the previous validation gap: the locally saved artifact was restored byte-for-byte, discovered in Redis RESP L2, loaded into LMCache L1, and consumed by vLLM as external prefix cache.

Metrics correction

The documentation now distinguishes:

vLLM metrics:       http://127.0.0.1:8000/metrics
LMCache MP metrics: http://127.0.0.1:8080/metrics

LMCache lookup metrics may not exist before the first request. Direct KDN injection bypasses LMCache store accounting, so lmcache_mp_l2_usage_bytes is not proof of Redis contents.

Compatibility and non-regression

This PR does not modify:

• the root README.md;
• the legacy Dockerfile or env/README.md path;
• root requirements.txt or pyproject.toml;
• Scheduler, Proxy, Instance, KDN builder/injector runtime code;
• routing, prediction, queueing, storage, or request behavior.

The changed-file set is limited to three existing v1 documents, one new migration closeout document, and three v1 activation/validation tools. While this PR was being prepared, main added separate Chinese and English KDN/LMCache architecture-amendment documents; those files do not overlap with this PR.

The legacy path remains available through CACHEROUTE_RUNTIME_PROFILE=legacy; the modern path uses v1; generic compatibility remains auto.

Validation performed for this PR

• bash -n env/docker/cu130/scripts/activate_v1.sh — passed.
• python3 -m py_compile env/docker/cu130/scripts/check_v1_environment.py scripts/validate_v1_kdn_roundtrip.py — passed.
• Reviewed the complete changed-file set and confirmed the root homepage and legacy environment are untouched.
• The underlying manual GPU/Redis/LMCache/vLLM workflow produced the exact runtime evidence listed above.

The newly packaged validator has been syntax-checked in this PR; the full GPU pipeline was validated manually before packaging and is now represented by the staged commands.

Superseded work

PR #150 is fully incorporated by merged PR #149 and this migration closeout. It should remain closed/superseded rather than being merged separately.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3fe246f83f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Extract the quickstart document before registration

On a fresh checkout, this path does not exist: a repo-wide tracked-file search shows only CacheRoute_test_nq.zip and README.md in this directory, with the named text file stored inside the ZIP, and no preceding quickstart step extracts it. Consequently, the first --build-kv-file "$DOC" command fails before registration; add an extraction step or use a document that is present in the working tree.

AGENTS.md reference: AGENTS.md:L91-L95

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reject artifacts without run_meta

When run_meta.json is absent or contains an empty object, this condition skips every metadata check, so inspect reports success as long as the manifest and dump counts match even though the artifact's runtime profile, key format, and build count are unknown. This contradicts the documented promise to verify run_meta.json and can approve an incomplete or incompatible artifact for injection; treat missing or empty metadata as an inspection error.

AGENTS.md reference: AGENTS.md:L227-L232

Useful? React with 👍 / 👎.