Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize file uploads when source is file #723

Merged
merged 1 commit into from Mar 5, 2024
Merged

Conversation

allada
Copy link
Collaborator

@allada allada commented Mar 5, 2024

When using nativelink with a local worker/CAS setup, adds optimizations which make it faster to upload files from the worker to the CAS.

This is specifically useful for Buck2 for users that want to build hermetically.

closes: #409


This change is Reviewable

Copy link
Collaborator Author

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+@zbirenbaum
+@adam-singer

Reviewable status: 0 of 2 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Local / ubuntu-22.04, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, pre-commit-checks, publish-image, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable, zig-cc ubuntu-20.04, zig-cc ubuntu-22.04 (waiting on @adam-singer and @zbirenbaum)

When using nativelink with a local worker/CAS setup, adds
optimizations which make it faster to upload files from the worker
to the CAS.

This is specifically useful for Buck2 for users that want to
build hermetically.

closes: TraceMachina#409
Copy link
Contributor

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does feel a bit linear in terms of how optimization is applied via single function/single enum, since its specialized on store and enums can be defined in a combination of names it is probably fine.

:lgtm:

Reviewed 9 of 9 files at r1, all commit messages.
Reviewable status: 1 of 2 LGTMs obtained (waiting on @zbirenbaum)

Copy link
Contributor

@zbirenbaum zbirenbaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 9 of 9 files at r1, all commit messages.
Reviewable status: 1 of 2 LGTMs obtained

Copy link
Contributor

@zbirenbaum zbirenbaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 2 of 2 LGTMs obtained

@allada allada merged commit 7c9a070 into TraceMachina:main Mar 5, 2024
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sparse files gets de-sparsified
3 participants