KVCR v0.1.1

KVCR v0.1.1 - Release Notes

Experimental release. KVCR v0.1.1 is a patch release of KV Cache Runner. KVCR-Guard now runs on Linux kernels from 5.3, failover can trigger on missed heartbeats, and the NIXL dependency moves to 1.5.0. Interfaces may evolve as SGLang, TensorRT-LLM, and object storage support land.

What Is in 0.1.1

• Older Linux Kernels for Guard: KVCR-Guard runs on Linux 5.3 through 6.4. It previously required 6.5 or newer (#53).
• Guard Failover:
  • Heartbeat Timeout: Set --heartbeat-timeout-ms (minimum 1000 ms) and the Guard stops a stalled worker and takes over its KV cache. Without the flag, the Guard takes over only when the worker exits (#46).
  • Faster Takeover: The Guard registers its memory when it claims a worker, so failover no longer repeats that registration (#57).
  • Shutdown Cleanup: Slow pool-file cleanup no longer uses up the Guard's shutdown deadline (#55).
• Startup Failure Error: When startup fails and KVCR cannot release its native resources, it raises KVCRStartupError. Callers should keep their KV buffers mapped until the process exits, because NIXL may still write to them (#43).
• Pool and Index Addressing: KVCR refers to KV caches by pool and index instead of raw addresses (#60). This release adds strided memory registration descriptors (#59) and labeled partial delivery inside shared memory pools (#61).
• align_sequence API: Frameworks can align cached blocks' recency with sequence order, so LRU eviction keeps prefixes over tails. It includes an optional policy hook, and sequence positions survive recovery and Guard handoff (#50).
• Opt-in Diagnostics: DEBUG logging, KVCR_DIAGNOSTICS=1, and KVCR_KEY_HISTORY=1 add transfer refusal reasons, stage timings, sampled query decisions, and key lifetimes (#58). Guard promotion logs each phase duration (#45).
• NIXL 1.5.0: The runtime dependency moves from nixl==1.3.2 to nixl==1.5.0 (#70).

Installation

pip install kvcr==0.1.1

The wheel and sdist are on PyPI:

File	sha256
kvcr-0.1.1-py3-none-any.whl	6014c1f303e2dc6d6f2817c39ebe0e9634371c65332baaa876642cba20fd0643
kvcr-0.1.1.tar.gz	ea0d22175d6db97b81bfe0aeccac6caddf88eeadf9cecb918eee420996f00596

Python 3.10 or newer on Linux x86-64. Runtime dependencies are nixl==1.5.0, msgspec, and pyzmq, plus the system libatomic for the recovery journal.

Tested Versions

The quick start pins the stack validated with this release. The exact pins are in Dockerfile.quick-start.

Component	Pin
vLLM base image	vllm/vllm-openai:nightly-43b4aaea3e40e20ef53ce2bfecfa1b72060bf2b5
vLLM KVCR adapter	vllm-project/vllm@08567505b09324f891797f4a13c1bd73fc75a827
Dynamo	ai-dynamo/dynamo@ea23e5130c1bfbadec477b99449d0cee5598caa0 (on the Dynamo v1.6.0 release branch)
NIXL	1.5.0

Other combinations are untested.

Framework Support

• vLLM: Supported, through the KVCR secondary-tier adapter. Additional features are tracked in vllm-project/vllm #59708.
• Dynamo KV Router: Supported.
• SGLang: In progress (sgl-project/sglang #32903).
• TensorRT-LLM: In progress (NVIDIA/TensorRT-LLM #18153).

Known Issues

• Shared Memory Not Released on vLLM: During a shutdown (for example, SIGKILL), vLLM's KV offload path can leave the shared mmap in /dev/shm. The fix is upstream in vLLM (vllm-project/vllm #57422, open). Workaround: clear /dev/shm between runs or size it for the run.

What Is Next

KVCR 0.2.0 is planned alongside Dynamo v1.7.0, with SGLang support.

Feedback and contributions are welcome. See CONTRIBUTING.md. Report security issues per SECURITY.md, not in public issues.

KVCR v0.1.0

KVCR v0.1.0 - Release Notes

Initial release (experimental). KVCR v0.1.0 is the first public release of KV Cache Runner. It supports cross-node DRAM sharing, local disk caching, and resiliency for vLLM. Interfaces may evolve as SGLang, TensorRT-LLM, and object storage support land.

KV Cache Runner (KVCR) treats KV cache as a system-wide distributed resource: available across memory and storage tiers, resilient to worker failure, and placed by a KV-aware request router. KVCR is framework and router agnostic; this release ships the vLLM integration with the Dynamo KV Router.

What Is in 0.1.0

• Cross-Node DRAM Sharing: An inference engine worker can fetch cached KV blocks from a remote peer over NIXL instead of recomputing prefill. There are timeouts throughout, so a stalled or dead remote source never blocks the caller.
• Local Disk Caching: A local disk tier below host DRAM extends KV cache capacity beyond host memory.
• KVCR-Guard Resiliency: An opt-in sidecar mirrors a worker's KV cache residency state, serves the cache when the worker dies, and hands it warm to the replacement. One Guard can hold multiple named pools.
• Router Hints: KVCR parses the versioned kv_hint protocol from the Dynamo KV Router, so engine integrations do not need to know the payload shape.
• Policy Interface: Users can customize eviction behaviors through kvcr.policy APIs.
• Public Quick Start: Builds and verifies KVCR with vLLM, Dynamo, and NIXL from pinned public sources, with no private credentials. See the quick start.

Installation

pip install kvcr==0.1.0

Python 3.10 or newer on Linux x86-64. Runtime dependencies are nixl==1.3.2, msgspec, and pyzmq, plus the system libatomic for the recovery journal.

Framework Support

• vLLM: Supported, through the KVCR secondary-tier adapter (vllm-project/vllm #53423).
• Dynamo KV Router: Supported.
• SGLang and TensorRT-LLM: In progress. RFCs are linked from the README.

Known Issues

• Shared Memory Not Released on vLLM: During a shutdown (for example, SIGKILL), vLLM's KV offload path can leave the shared mmap in /dev/shm. The fix is upstream in vLLM (vllm-project/vllm #57422, open). Workaround: clear /dev/shm between runs or size it for the run.

What Is Next

SGLang and TensorRT-LLM integrations and object storage support are planned for future releases. The quick start pins specific vLLM and Dynamo revisions; other combinations are untested.

Feedback and contributions are welcome. See CONTRIBUTING.md. Report security issues per SECURITY.md, not in public issues.

---