Skip to content

v0.4.8rc2-dev31

Choose a tag to compare

@efschu efschu released this 23 Jun 14:54
· 26 commits to dev since this release

Engine-driven multi-group transfer optimizations

Wire-Format (Tier 1, always on)

  • Replaces pickle.HIGHEST_PROTOCOL with torch.save on the worker→server wire
  • b'L' magic byte + payload; legacy pickle blobs auto-detected on first byte
  • torch.save reaches ~1.6-1.8 GiB/s on production-sized (~50 MiB) bf16 tensors
    vs pickle's ~0.9 GiB/s -- ~2x speedup on serialize + deserialize
  • Cumulative effect on the 14 GiB / 4-group workload: ~1.5-2s shaved per store

Multi-Stream D2D (Tier 1, opt-in)

  • LMCACHE_MP_MULTI_STREAM_D2D=1 runs each group's GPU→CPU gather on its own
    CUDA stream
  • Default is OFF because on consumer GPUs the PCIe bus is shared and
    per-stream sync overhead exceeds the overlap win (4x slower in our test)
  • Available for A/B testing on different hardware

Cleanup

  • Removed duplicate _serialize_single_group_chunks (was defined twice)
  • Removed dead _to_serializable_chunk and tagged-tuple wire format
  • Removed duplicate _pinned_pool initialization

Compatibility

  • Backward-compatible wire format: dev<=30 pickle blobs still decode
  • No L2 cache wipe required when upgrading
  • Same COMMIT_STORE_GROUP request shape; the new format is auto-detected