Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache pyenv in CI #7470

Merged
merged 8 commits into from Apr 2, 2019

Conversation

Projects
None yet
2 participants
@Eric-Arellano
Copy link
Contributor

commented Mar 31, 2019

Problem

Every OSX shard that uses Python 3.6 or 3.7, we waste about 2 minutes re-installing Python 3. Thanks to the insights from pantsbuild/setup#39, we can stop wasting this time by using the Travis cache.

Solution

Add $PYENV_ROOT to the cache and modify the pyenv code to first check if the relevant folders exist before git cloning and installing. We extract out all of this Pyenv logic into a new script install_python_for_ci.sh.

We can safely add this to the cache for all shards, even if they don't use Pyenv, because Travis will simply upload an empty folder if there is nothing there.

Result

Overall daily CI time reduced by 8 minutes.

This also allows us to add Python 3.6 on all OSX shards, so that we can use Python 3 in our build-support scripts.

Eric-Arellano added some commits Mar 31, 2019

Don't use &&
It messes up the return code when Python 2.7.13 is already installed.

Eric-Arellano added some commits Apr 1, 2019

Semicolons not needed for bash command
Because it's a multiline string, we don't need semicolons to terminate the statement.
@cosmicexplorer
Copy link
Contributor

left a comment

Love the script, thank you for making that change as well! This is yet another wonderful improvement that will accrue dividends in CI time.

@Eric-Arellano Eric-Arellano merged commit 5f33a06 into pantsbuild:master Apr 2, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:pyenv-cache branch Apr 2, 2019

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Apr 2, 2019

@cosmicexplorer the script is so much better. Thank you for the idea!

Will upstream this to the setup repo as well.

Eric-Arellano added a commit to pantsbuild/setup that referenced this pull request Apr 2, 2019

Refactor .travis.yml by extracting install_python_for_ci.sh script (#50)
In pantsbuild/pants#7470, we came up with a better design for interfacing with our git cloned version of Pyenv. By extracting out a script, we keep `.travis.yml` more focused and are able to improve the robustness of Pyenv-related code, such as adding comments.

Eric-Arellano added a commit that referenced this pull request Apr 3, 2019

Ensure Python 3.6 or 3.7 is installed on all CI shards (#7483)
### Problem
Once all CI shards are guaranteed to have Python 3, we can port our more complex `build-support` scripts to Python 3, which will allow us to reduce technical debt and to bring benefits like better argument parsing. 

In particular, we want to be able to use Python 3 in those scripts so that we can use `subprocess.run()`.

### Solution
For Linux, Travis pre-installs Python 3.6 and 3.7. We simply tell Travis we want those installed, and then set `pyenv global` to resolve everything for no performance cost.

OSX shards do not have Python 3 installed by default (at least for certain `osx_images`), so we use Pyenv to install on all shards. This is no longer very costly to do so thanks to caching Pyenv through 
#7470.

### Result
All shards now have Python 3.6 or 3.7.

Once we update the Centos6 Docker image in #7064, every where in CI will have Python 3. We can in the meantime now start porting scripts if we are confident they are never used by Docker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.