Package upload more atomic #10719
Merged
Package upload more atomic #10719
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Firobe
added
the
ci-build-me
Firobe
force-pushed
the
deb-s3-atomic-upload
branch
from
157f867
to
fa37fce
Compare
During package uploads, packages are now uploaded as temporary filenames and then moved to their final names in bulk. This prevents: 1. corrupted files with visible paths if the transfer is interrupted 2. a user downloading a corrupted file while it is being transferred Ultimately, this improves (but cannot completely fix) the possibility of race conditions between deb-s3 and other jobs downloading from the repository. Manifests are uploaded as before, since they have to be uploaded as soon as possible, before the lock is released.
Firobe
force-pushed
the
deb-s3-atomic-upload
branch
from
980c4b7
to
3f6b038
Compare
lk86
approved these changes
Apr 11, 2022
There was a problem hiding this comment.
The changes sound good to me in theory, but can you make this same update to the toolchain container in compatible so that both major branches see the benefits? You will need this PR + a new one for compatible otherwise you will hit "merge conflicts with develop"
This was referenced Apr 20, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Explain your changes
This PR uses a modified version of deb-s3 to achieve the following behavior (see the actual changes in deb-s3 here):
During package uploads, packages are now uploaded as temporary filenames
and then moved to their final names in bulk.
This prevents:
Ultimately, this improves (but cannot completely fix) the possibility of
race conditions between deb-s3 and other CI jobs downloading from the
repository.
An example of such a race is https://buildkite.com/o-1-labs-2/mina/builds/21806#19105d06-9d77-4dce-94de-69bb819d17db/116-197
Manifests are uploaded as before, since they have to be uploaded as soon
as possible, before the lock is released.
Explain how you tested your changes