Skip to content

v0.4.8rc2-dev23

Choose a tag to compare

@efschu efschu released this 23 Jun 08:35
· 34 commits to dev since this release

Fix for msgspec 4 GiB bin limit: per-group COMMIT_STORE_GROUP messages.

Root cause: combined multi-group cpu_data blob exceeded msgspec msgpack bin limit (4 GiB) for large hybrid models like Qwen3-27B with GDN layers.

Fix: New RequestType.COMMIT_STORE_GROUP ([key, instance_id, group_idx, cpu_data]) — worker sends one message per group instead of one combined blob. Each per-group blob fits within the msgspec limit.

Also includes PR LMCache#3813 review fixes:

  • fp8 dtype (e4m3fn, e5m2) support in _serialize_multi_group_chunks via view(torch.uint8)
  • assert replaced with explicit ValueError check in worker_transfer.py
  • group count validation in _commit_store_multi_group before indexing
  • 16 unit tests covering serialization, roundtrip, and dtype preservation