Repository navigation
v0.4.8rc2-dev26
Critical fix: COMMIT_STORE_GROUP must re-pickle for storage backend.
The previous dev25 commit_store_group handler passed the worker's
dev24+ wire-format bytes directly to strategy.commit_store, but the
storage backend expects a pickled list of CPU tensors. Without the
re-pickle, chunk_cpu.shape failed with 'list object has no attribute
shape' on the second model call after store.
This build was tested live against vLLM with --enable-prefix-caching
on Qwen3-27B-AWQ with 56k-token prompts:
Cold cache TTFT: 77.35 s
Warm cache TTFT: 21.74 s (3.56× faster, 55.6 s saved per request)
Warm cache PP tps: 2576 vs cold 724 (3.56×)
Benchmarks via /v1/completions streaming with the local fork wheel.
Includes from earlier in dev26:
- per-group COMMIT_STORE_GROUP (4 GiB msgspec bin limit fix)
- bf16 native pickle (1.9× speedup, 50% smaller blobs)
- backward-compatible deserializer (no L2 wipe required)