Add peer-to-peer blob sharing with WorkerProxyStore#2210
Open
rejuvenile wants to merge 6 commits intoTraceMachina:mainfrom
Open
Add peer-to-peer blob sharing with WorkerProxyStore#2210rejuvenile wants to merge 6 commits intoTraceMachina:mainfrom
rejuvenile wants to merge 6 commits intoTraceMachina:mainfrom
Conversation
Update tonic, prost, and other dependencies to latest versions. Regenerate protobuf bindings. Add aws-lc-rs and rayon support to digest hasher. Update BUILD.bazel files for LRE toolchain. Co-Authored-By: Claude <noreply@anthropic.com>
Rewrite existence cache to prevent stale positives by bypassing cache on update and cleaning entries on NotFound. Fix BatchUpdateBlobs duplicate digest handling. Add POSIX_FADV_SEQUENTIAL for read-ahead. Pre-set CAS files to 0o555 to avoid redundant chmod on hardlink. Fix LRU eviction ordering at startup by sorting files by atime. Add stall detector for store operations. Replace async Mutex with parking_lot in EvictingMap. Increase gRPC connections_per_endpoint default to 32. Co-Authored-By: Claude <noreply@anthropic.com>
Fix ByteStream protocol compliance for large blobs. Add gRPC error details with proper status codes. Add max_total_batch_size config for BatchReadBlobs/BatchUpdateBlobs. Improve capabilities server to report actual supported features. Add TLS improvements and server startup configuration options. Downgrade per-request transfer logs to debug level. Co-Authored-By: Claude <noreply@anthropic.com>
Add load-aware worker selection with CPU load tracking. Implement directory cache and subtree coverage scoring for locality-aware scheduling. Add stall detection for queued actions. Support SIGKILL retry on worker timeout. Handle FAILED_PRECONDITION for missing inputs. Add fair round-robin dispatch via LRU promotion. Downgrade per-action dispatch logs to debug level. Co-Authored-By: Claude <noreply@anthropic.com>
Implement batch input fetching using GetTree + has_with_results + BatchReadBlobs + ByteStream for concurrent blob downloads. Add hardlink retry on ENOENT (eviction during link). Implement phase timing for action lifecycle. Add upload timeout with stall detection. Support peer blob hints from scheduler. Downgrade per-blob fetch/upload logs to debug level. Co-Authored-By: Claude <noreply@anthropic.com>
Implement WorkerProxyStore for worker-to-worker blob transfers with locality-aware routing. Add BlobLocalityMap for tracking which workers have which blobs. Add BlobsAvailable RPC for workers to report cached digests. Support redirect-based peer discovery and racing peer vs server downloads. Add inner_store() to all store types. Update all config examples with store_type field. Add integration tests for peer sharing. Co-Authored-By: Claude <noreply@anthropic.com>
2 tasks
|
|
rejuvenile
force-pushed
the
pr/06-peer-blob-sharing
branch
from
4b61327 to
b61818d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
cargo checkpasses🤖 Generated with Claude Code
This change is