Skip to content

v0.4.8rc2-dev26

Choose a tag to compare

@efschu efschu released this 23 Jun 10:38
· 31 commits to dev since this release

Critical fix: COMMIT_STORE_GROUP must re-pickle for storage backend.

The previous dev25 commit_store_group handler passed the worker's
dev24+ wire-format bytes directly to strategy.commit_store, but the
storage backend expects a pickled list of CPU tensors. Without the
re-pickle, chunk_cpu.shape failed with 'list object has no attribute
shape' on the second model call after store.

This build was tested live against vLLM with --enable-prefix-caching
on Qwen3-27B-AWQ with 56k-token prompts:

Cold cache TTFT: 77.35 s
Warm cache TTFT: 21.74 s (3.56× faster, 55.6 s saved per request)
Warm cache PP tps: 2576 vs cold 724 (3.56×)

Benchmarks via /v1/completions streaming with the local fork wheel.

Includes from earlier in dev26:

  • per-group COMMIT_STORE_GROUP (4 GiB msgspec bin limit fix)
  • bf16 native pickle (1.9× speedup, 50% smaller blobs)
  • backward-compatible deserializer (no L2 wipe required)