LMCache v0.5.1.dev34

PCIe opt: submit+dict pool, PinnedBufferPool, stream-local sync, async commit. Delta-store: COMMIT_STORE_GROUP_DELTA, LRU prefetch. torch.save wire format (2x faster). Fixes: init, server re-pickle, PR LMCache#3813. Benchmark: warm TTFT 2.4s, PP 23k tok/s.

v0.4.8rc2-dev33

Engine-driven multi-group transfer optimizations (Round 3)

E. Delta-Store (prefix-based skip)

• New COMMIT_STORE_GROUP_DELTA request: payload [key, instance_id, group_idx, skip_count, cpu_data]
• Server stores chunks at offset [skip_count, ...] for the named group
• Worker derives skip_count from the prior lookup's prefix hit count
• For 100%-hit groups, the worker sends an empty blob and the server no-ops the storage write -- saving 14 GiB of zmq traffic on the re-run of a cached prompt
• Opt-in via LMCACHE_MP_DELTA_STORE=1 (default ON; skip_count=0 is identical to legacy)

C. Async-IO MQ (callbacks)

• MessagingFuture.add_done_callback() registers fn(result) to fire on set_result
• Runs in the polling loop's thread without blocking worker threads on future.result(timeout=...)
• Exceptions in callbacks are swallowed (one bad callback would otherwise kill the polling loop)

G. Background-Prefetch (LRU predictor)

• PrefetchPredictor module: LRU of (key, hit_count, total_chunks) tuples
• Wired into EngineDrivenTransferContext as self._prefetch_predictor
• Heuristic: most-recently-used key != current is the predicted next
• Useful for chat-session style workloads; same-prompt re-run is already covered by L1

F. Storage Batched Append (observability)

• Added _STATS_FILE_OPS counter in the server's commit paths
• Tracks per-server storage write invocations
• Helps operators verify when the 4-file-per-multi-group-store pattern is being hit
• The actual batched-append optimization would require a new storage backend API (not implemented -- savings <2ms)

Carried from dev31

• torch.save wire format (~2x faster than pickle.HIGHEST_PROTOCOL)
• Multi-Stream D2D opt-in via LMCACHE_MP_MULTI_STREAM_D2D=1
• Backward-compatible deserializer (auto-detect format)

Built on: 2026-06-23T15:29:36Z
Base: v0.4.8rc2.dev31

v0.4.8rc2-dev31

Engine-driven multi-group transfer optimizations

Wire-Format (Tier 1, always on)

• Replaces pickle.HIGHEST_PROTOCOL with torch.save on the worker→server wire
• b'L' magic byte + payload; legacy pickle blobs auto-detected on first byte
• torch.save reaches ~1.6-1.8 GiB/s on production-sized (~50 MiB) bf16 tensors
  vs pickle's ~0.9 GiB/s -- ~2x speedup on serialize + deserialize
• Cumulative effect on the 14 GiB / 4-group workload: ~1.5-2s shaved per store

Multi-Stream D2D (Tier 1, opt-in)

• LMCACHE_MP_MULTI_STREAM_D2D=1 runs each group's GPU→CPU gather on its own
  CUDA stream
• Default is OFF because on consumer GPUs the PCIe bus is shared and
  per-stream sync overhead exceeds the overlap win (4x slower in our test)
• Available for A/B testing on different hardware

Cleanup

• Removed duplicate _serialize_single_group_chunks (was defined twice)
• Removed dead _to_serializable_chunk and tagged-tuple wire format
• Removed duplicate _pinned_pool initialization

Compatibility

• Backward-compatible wire format: dev<=30 pickle blobs still decode
• No L2 cache wipe required when upgrading
• Same COMMIT_STORE_GROUP request shape; the new format is auto-detected

v0.4.8rc2-dev26

Critical fix: COMMIT_STORE_GROUP must re-pickle for storage backend.

The previous dev25 commit_store_group handler passed the worker's
dev24+ wire-format bytes directly to strategy.commit_store, but the
storage backend expects a pickled list of CPU tensors. Without the
re-pickle, chunk_cpu.shape failed with 'list object has no attribute
shape' on the second model call after store.

This build was tested live against vLLM with --enable-prefix-caching
on Qwen3-27B-AWQ with 56k-token prompts:

Cold cache TTFT: 77.35 s
Warm cache TTFT: 21.74 s (3.56× faster, 55.6 s saved per request)
Warm cache PP tps: 2576 vs cold 724 (3.56×)

Benchmarks via /v1/completions streaming with the local fork wheel.

Includes from earlier in dev26:

• per-group COMMIT_STORE_GROUP (4 GiB msgspec bin limit fix)
• bf16 native pickle (1.9× speedup, 50% smaller blobs)
• backward-compatible deserializer (no L2 wipe required)

v0.4.8rc2-dev25

Performance: skip bf16→float32 conversion + pickle.HIGHEST_PROTOCOL.

Wire-format compatibility: backward-compatible deserializer for blobs
written by dev15-dev23. No L2 cache wipe required when upgrading from
any earlier wheel.

Benchmarks (bf16, 10 chunks ~640 MiB):
OLD bf16→float32: 1314 ms, 1280 MiB blob
NEW pickle HIGHEST: 704 ms, 640 MiB blob
Speedup: 1.87×, 50% smaller blob

Includes all earlier fixes:

• msgspec 4 GiB bin limit fix via per-group COMMIT_STORE_GROUP messages
• fp8 dtype support (e4m3fn, e5m2)
• assert → ValueError replacement
• group count validation

v0.4.8rc2-dev23

Fix for msgspec 4 GiB bin limit: per-group COMMIT_STORE_GROUP messages.

Root cause: combined multi-group cpu_data blob exceeded msgspec msgpack bin limit (4 GiB) for large hybrid models like Qwen3-27B with GDN layers.

Fix: New RequestType.COMMIT_STORE_GROUP ([key, instance_id, group_idx, cpu_data]) — worker sends one message per group instead of one combined blob. Each per-group blob fits within the msgspec limit.

Also includes PR LMCache#3813 review fixes:

• fp8 dtype (e4m3fn, e5m2) support in _serialize_multi_group_chunks via view(torch.uint8)
• assert replaced with explicit ValueError check in worker_transfer.py
• group count validation in _commit_store_multi_group before indexing
• 16 unit tests covering serialization, roundtrip, and dtype preservation

v0.4.8rc2-dev19

Build with PR review fixes applied:

• fp8 dtype (e4m3fn, e5m2) support in multi-group serialization
• Replaced assert with ValueError in worker_transfer.py
• Added group count validation in _commit_store_multi_group
• Added fp8 and mixed-dtype roundtrip tests

v0.4.8rc2-dev15 - Engine-Driven Multi-Group

Installation

Download Wheel

lmcache-0.4.8rc2.dev15-cp312-cp312-linux_x86_64.whl

SHA256: ebbf8be0239b47b6b6b2c77a18a4fc763dfb54b99b49240835fdc64fbb08de5c

Quick Install

# Download the wheel from this release
pip install lmcache-0.4.8rc2.dev15-cp312-cp312-linux_x86_64.whl --force-reinstall --no-deps

Build from Source

If you need a different Python version or platform, build from source:

git clone https://github.com/efschu/LMCache.git
cd LMCache
git checkout dev

# Build wheel (requires GPU)
bash scripts/build_manylinux_gpu.sh

# Install
pip install wheelhouse/*.whl --force-reinstall --no-deps

See docs/BUILD.md for detailed build instructions.

---

What's New

This pre-release implements engine-driven multi-group KV cache transfer for hybrid models like Qwen3.6-27B.

Key Features

• CPU-only LMCache Server: No GPU memory allocation (was ~666 MB/GPU)
• Multi-Group Support: Works with hybrid models (Attention + GDN/Mamba groups)
• Prefetch Fix: Properly awaits all group handles before retrieval

Changes

• lmcache/v1/multiprocess/custom_types.py: Added GroupLayoutInfo for per-group metadata
• lmcache/v1/multiprocess/transfer_context/base.py: Multi-group gather/scatter functions
• lmcache/v1/multiprocess/transfer_context/worker_transfer.py: Worker-side multi-group support
• lmcache/v1/multiprocess/modules/engine_driven_transfer.py: Server-side multi-group support
• lmcache/v1/multiprocess/modules/lookup.py: Fixed multi-group prefetch handling

---

Requirements

• Python 3.12
• CUDA 12.x
• PyTorch 2.5+
• NVIDIA GPU (RTX 3080+, A100, H100, etc.)

---

Documentation

• README.md - Overview and quick start
• docs/BUILD.md - Build instructions
• Lmcache_engine_driven_multigroup.md - Full design document
• [example.yml](Use this as a starting point. Drop-in replacement for the lmcache/vllm-openai:v0.4.7-cu129 container.)