Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use S3 Sync for CI deploys to avoid recopying files #7895

Merged
merged 14 commits into from Jun 21, 2019

Conversation

Projects
None yet
3 participants
@Eric-Arellano
Copy link
Contributor

commented Jun 19, 2019

We currently copy all of dist/deploy into S3, even if the file is an exact copy of something already in S3. This poses a real cost, both monetary and time.

Instead, here we use aws s3 sync to only transfer files that are different from what is currently in the bucket.

Will close #7258.

@Eric-Arellano Eric-Arellano changed the title WIP: Use S3 Sync for CI deploys to avoid recopying files Use S3 Sync for CI deploys to avoid recopying files Jun 20, 2019

@Eric-Arellano Eric-Arellano marked this pull request as ready for review Jun 20, 2019

@Eric-Arellano Eric-Arellano requested review from benjyw, jsirois and stuhood Jun 20, 2019

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Jun 20, 2019

Going to run all CI via the branch build, not the PR build, since this impacts only non-PRs. I ran yesterday one shard on the PR build to confirm that this won't break PRs uploading pants.pex.

@benjyw

benjyw approved these changes Jun 20, 2019

@Eric-Arellano Eric-Arellano merged commit fee10f2 into master Jun 21, 2019

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the s3-sync branch Jun 21, 2019

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Jun 21, 2019

Big thanks to @benjyw for all the help to land this!

blorente added a commit to blorente/pants that referenced this pull request Jun 21, 2019

Eric-Arellano added a commit that referenced this pull request Jun 21, 2019

Fix #7895 breaking CI for pull requests (#7921)
#7895 works for branch builds, but not pull request builds due to the error `Partial credentials found in env, missing: AWS_SECRET_ACCESS_KEY`.

This would happen because we always would define `AWS_SECRET_ACCESS_KEY`, but the value `AWS_SECRET_ACCESS_KEY` would only be defined on branch builds because it's encrypted.

Instead, we should only define `AWS_SECRET_ACCESS_KEY` if we're on a branch build.
@stuhood

This comment has been minimized.

Copy link
Member

commented Jun 21, 2019

Thanks!

Relatively little should already be in S3, because the path is 1) unique to the SHA, 2) unique to the wheel that is being created. I think that only universal wheels could potentially be uploaded multiple times.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.