Skip to content

v0.1.11

Choose a tag to compare

View all tags
@0oshowero0 0oshowero0 released this 24 Sep 02:12
v0.1.11
05d4cc3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlight

🚀 New Features

  • SSD Offload for SimpleStorage: Add optional node-local SSD offloading to SimpleStorage to reduce long-lived host-memory pressure for workloads with large training samples. Each (field, global index) payload is placed independently according to its serialized size: samples at or above 1 MiB are stored on SSD, while smaller samples remain in host memory. The hybrid store preserves existing PUT, GET, CLEAR, overwrite, capacity, and checkpoint semantics without changing the public API. SSD files are isolated by run and storage unit and cleaned up on clear or graceful shutdown. The feature is disabled by default and can be enabled through backend.SimpleStorage.ssd_offload.enabled with the offload directory configured by backend.SimpleStorage.ssd_offload.path. In the reported two-node NVMe benchmark, SSD-backed E2E throughput remained within approximately 3.7–8.0% of the in-memory path for 1–4 MiB per-field samples. (#162)

🐛 Fixes & Improvements

  • Pooled and Reusable ZMQ Request Sockets: Replace the create-connect-send-receive-close cycle used for every RPC with reusable ZMQSocketPool instances, avoiding repeated TCP/ZMTP handshakes and bounding socket growth under concurrency. Leases are exclusive so replies cannot be consumed by the wrong request, and any socket involved in a timeout, cancellation, or other failed request is discarded rather than returned to the pool. Pools are isolated by role, event loop or thread, and peer address, preventing unsafe socket reuse across execution contexts or after peer relocation. TQ_SOCKET_POOL_SIZE controls the per-peer concurrency and idle-socket cap, with a default of 4. In the reported local 300-way concurrency test, peak sockets decreased from 300 to 4 while elapsed time decreased from 71 ms to 48 ms. (#167)

  • Resilient Storage-Unit Requests and Accept-Queue Hardening: Recover automatically when a storage-unit request is lost in transit instead of failing the entire job after a receive timeout. Timed-out GETs and idempotent PUTs are retried over a fresh connection up to TQ_SIMPLE_STORAGE_MAX_ATTEMPTS times, defaulting to 3; PUTs carrying a user data_parser are intentionally not replayed because the parser may have side effects. The storage ROUTER backlog is also increased from ZMQ's default 100 to 4096 through TQ_STORAGE_ZMQ_BACKLOG, reducing silent request loss when the TCP accept queue is saturated. New arrival counters and an optional TQ_ACCEPT_PROBE_INTERVAL kernel-level accept-queue probe improve diagnosis by distinguishing requests lost before arrival from requests that arrived but did not complete. (#171, #172)

  • Storage Proxy Peer Filtering and Worker Resilience: Prevent arbitrary TCP/ZMQ traffic from reaching and terminating the storage-unit request worker. The storage proxy now forwards only peers using recognized TransferQueue storage-manager or metrics-collector identities. Undecodable frames from an otherwise valid peer are handled inside the worker's error boundary and receive an explicit error response instead of terminating the request loop and leaving future clients blocked until timeout. (#168)

  • Explicit Concurrent-Clear Key-Miss Handling: Introduce StorageKeyNotFoundError, exported from transfer_queue, so callers can distinguish the expected race where kv_clear removes data after metadata has been resolved but before it is fetched. The error type is preserved across the ZMQ request path instead of being flattened into a generic RuntimeError, and this expected concurrent condition is logged at DEBUG rather than ERROR. Other failures, such as requesting an unknown field, continue to surface as errors. Large routing-table dumps in failure logs are also replaced with a compact storage-unit count. (#160)

  • Updated Mooncake CI Coverage and Documentation: Refresh the Mooncake version used by CI and update the project README with the latest project information. (#161, #170, #174)

What's Changed

New Contributors

Full Changelog: v0.1.10...v0.1.11

Contributors

OutstanderWang, 0oshowero0, and 3 other contributors
Assets 3
Loading