Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache AWS CLI install in CI #7472

Merged
merged 10 commits into from Apr 2, 2019

Conversation

Projects
None yet
2 participants
@Eric-Arellano
Copy link
Contributor

commented Mar 31, 2019

Problem

We freshly install the AWS CLI program on nearly every CI shard, even though the program does not change between runs. This adds about 30 seconds to most shards.

Solution

Modify the install_aws_cli_for_ci.sh script to first check if the program is installed, and only install if it is not. Also add the install folder to Travis's cache.

Note that Travis does not cache symlinks, only the actual files. So, we must re-symlink the bin every time we run the script.

Result

Overall CI time reduced by ~13.5 minutes. Each individual shard is now an average of ~19.4 seconds faster.

See https://docs.google.com/spreadsheets/d/1KWzpHnJzUpEZwx7_m-Kt_hZ-QNYFnzoFRPbi3dX0t6c/edit#gid=60516952 for a per-shard analysis of the old time and the new time with this cache.

@Eric-Arellano Eric-Arellano changed the title Cache AWS command line tools in CI WIP: Cache AWS command line tools in CI Mar 31, 2019

Eric-Arellano added some commits Mar 31, 2019

Check if the bin exists, rather than the root folder
Travis will auto-create the root folder, even when it cannot be found in the cache. So, we instead must check that the bin within that folder exists.
Can't use the folder /usr/local/aws for cache with Linux
Travis runs into a permissions issue when doing so: https://travis-ci.org/pantsbuild/pants/jobs/513927894#L432.

Further, the only reason we were doing this is the tutorial used this install dir.

Instead, use a custom install dir. Our symlink makes it so that it doesn't really matter where we install it.
Empty commit to test if caching works
We know macOS does, now lets see if Linux does.

@Eric-Arellano Eric-Arellano changed the title WIP: Cache AWS command line tools in CI Cache AWS command line tools in CI Apr 1, 2019

Fix test for unset AWS_CLI_ROOT
It would complain about an unbound variable.

@Eric-Arellano Eric-Arellano changed the title Cache AWS command line tools in CI Cache AWS CLI install in CI Apr 1, 2019

@stuhood

stuhood approved these changes Apr 1, 2019

@Eric-Arellano Eric-Arellano merged commit dd1e64e into pantsbuild:master Apr 2, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:cache-aws branch Apr 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.