Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache pyenv in CI #7470

merged 8 commits into from Apr 2, 2019


None yet
2 participants
Copy link

commented Mar 31, 2019


Every OSX shard that uses Python 3.6 or 3.7, we waste about 2 minutes re-installing Python 3. Thanks to the insights from pantsbuild/setup#39, we can stop wasting this time by using the Travis cache.


Add $PYENV_ROOT to the cache and modify the pyenv code to first check if the relevant folders exist before git cloning and installing. We extract out all of this Pyenv logic into a new script

We can safely add this to the cache for all shards, even if they don't use Pyenv, because Travis will simply upload an empty folder if there is nothing there.


Overall daily CI time reduced by 8 minutes.

This also allows us to add Python 3.6 on all OSX shards, so that we can use Python 3 in our build-support scripts.

Eric-Arellano added some commits Mar 31, 2019

Don't use &&
It messes up the return code when Python 2.7.13 is already installed.

Eric-Arellano added some commits Apr 1, 2019

Semicolons not needed for bash command
Because it's a multiline string, we don't need semicolons to terminate the statement.
Copy link

left a comment

Love the script, thank you for making that change as well! This is yet another wonderful improvement that will accrue dividends in CI time.

@Eric-Arellano Eric-Arellano merged commit 5f33a06 into pantsbuild:master Apr 2, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:pyenv-cache branch Apr 2, 2019


This comment has been minimized.

Copy link
Contributor Author

commented Apr 2, 2019

@cosmicexplorer the script is so much better. Thank you for the idea!

Will upstream this to the setup repo as well.

Eric-Arellano added a commit to pantsbuild/setup that referenced this pull request Apr 2, 2019

Refactor .travis.yml by extracting script (#50)
In pantsbuild/pants#7470, we came up with a better design for interfacing with our git cloned version of Pyenv. By extracting out a script, we keep `.travis.yml` more focused and are able to improve the robustness of Pyenv-related code, such as adding comments.

Eric-Arellano added a commit that referenced this pull request Apr 3, 2019

Ensure Python 3.6 or 3.7 is installed on all CI shards (#7483)
### Problem
Once all CI shards are guaranteed to have Python 3, we can port our more complex `build-support` scripts to Python 3, which will allow us to reduce technical debt and to bring benefits like better argument parsing. 

In particular, we want to be able to use Python 3 in those scripts so that we can use ``.

### Solution
For Linux, Travis pre-installs Python 3.6 and 3.7. We simply tell Travis we want those installed, and then set `pyenv global` to resolve everything for no performance cost.

OSX shards do not have Python 3 installed by default (at least for certain `osx_images`), so we use Pyenv to install on all shards. This is no longer very costly to do so thanks to caching Pyenv through 

### Result
All shards now have Python 3.6 or 3.7.

Once we update the Centos6 Docker image in #7064, every where in CI will have Python 3. We can in the meantime now start porting scripts if we are confident they are never used by Docker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.