Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare Linux UCS2 shard for upgrade to Centos6 base image in #7064 #7418

Merged
merged 16 commits into from Apr 2, 2019

Conversation

Projects
None yet
4 participants
@Eric-Arellano
Copy link
Contributor

commented Mar 22, 2019

Problem

We will in #7064 start using Pyenv to install Python 2.7 in the base Centos6 image.

This will conflict with our Linux UCS2 shard, which currently uses Pyenv to install Python 2.7 with UCS2 instead of the default UCS4.

Solution

Tweak the Dockerfile to avoid any ambiguity with Centos6 image:

  • Rename the Python version env var to not conflict by marking it as UCS2.
  • Check if PYENV_ROOT is already defined.
  • Check if PYENV_ROOT already exists before git cloning to avoid a git failure.

The key line is ${PYENV_BIN} global ${PYTHON_27_VERSION_UCS2}, which will override whatever ${PYENV_ROOT}/shims/python2.7 used to point to.

@Eric-Arellano Eric-Arellano requested a review from cosmicexplorer Mar 22, 2019

Eric-Arellano added some commits Mar 22, 2019

Change solution: use different PYENV_ROOT
Danny had a great point that trying to clone into the original PYENV_ROOT will fail if it already exists.

Instead, we can simply use a new root. This also allows us to keep more of the original code because there is no longer ambiguity within PYENV_ROOT/shims.
No longer use a custom pyenv_root
If there are already pyenv installed Pythons, then we want to be able to use them. Namely, we're going to be porting some of our build scripts to Python 3 and we need to ensure we have Python 3 on every shard. Overriding it means we could not use the Python 3 installed on centos6.

We can point to ~/.pyenv/versions/2.7.15/bin/python to disambiguate.
Check if PYENV_ROOT exists, rather than PYENV_BIN
In normal Travis, we have to check if PYENV_BIN exists because Travis will automatically create the TRAVIS_ROOT folder as a part of caching.

Docker, however, does not have caching, so we can simply check if the root folder exists.

If we were to stick with PYENV_BIN, we would need to `rm -rf PYENV_ROOT` to make sure the git clone would not fail which is more work than we want.
Fix PATH entry
PATH needs to point to the parent folder of the bin you want included, not its grandparent. It won't look at subfolders, so using the grandparent was causing the PATH entry to not work properly.
Go back to using pyenv/shims instead of versions
I forgot that `pyenv global` will rewrite what `shims/python2.7` points to. If there is already a Python 2 interpreter installed there, `pyenv global` will simply overwrite the shim to point to Python 2.7.

This is how Pyenv is supposed to work, and we should not be using the versions/ folder.

@Eric-Arellano Eric-Arellano requested review from cosmicexplorer and jsirois Apr 2, 2019

@jsirois

jsirois approved these changes Apr 2, 2019

@Eric-Arellano Eric-Arellano merged commit 1f23088 into pantsbuild:master Apr 2, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:robust-ucs2-docker branch Apr 2, 2019

cosmicexplorer added a commit that referenced this pull request Apr 5, 2019

Install Python 2.7 and 3.6 on Centos6 base image through Pyenv (#7064)
### Problem

We have used a python 3 interpreter in CI for a while now, using `pyenv` to install this in our Dockerfiles instead of relying on CentOS's `scl` features, which don't allow for multiple pythons at a time. Using `pyenv` to install Python 2.7 as well instead of relying on the package manager allows us to upgrade or modify the image without screwing up our Python installation.

That work began in this PR and ended up in #6981, #7352, #7418, #7483, and more, but it then led to having to duplicate code across `travis_ci` and `travis_ci_py36` Dockerfiles. Using `pyenv` to install python in our `travis_ci` images also involves a significant setup time in bootstrap phases to install our desired pythons. Installing Python in the CentOS 6 base image avoids having to install Python at all in our CI, which saves us minutes in the bootstrap phase.

### Solution

- Use Pyenv to install Python 2.7 and Python 3.6 in the `centos6` Dockerfile.
- Remove `travis_cy_py36`.
- Modify the `travis_ci` Dockerfile to bootstrap pyenv itself if not available.

### Result

We get some CI time back in the bootstrap phase.

### TODO

- [ ] merge this PR
- [ ] publish the new `centos6` image to dockerhub at `pantsbuild/centos6:latest`
- [ ] merge a followup PR removing the manual pyenv bootstrapping from the `travis_ci` image.

@stuhood stuhood added this to the 1.15.x milestone Apr 8, 2019

stuhood added a commit that referenced this pull request Apr 8, 2019

Prepare Linux UCS2 shard for upgrade to Centos6 base image in #7064 (#…
…7418)

### Problem
We will in #7064 start using Pyenv to install Python 2.7 in the base Centos6 image. 

This will conflict with our Linux UCS2 shard, which currently uses Pyenv to install Python 2.7 with UCS2 instead of the default UCS4.

### Solution
Tweak the Dockerfile to avoid any ambiguity with Centos6 image:

* Rename the Python version env var to not conflict by marking it as `UCS2`.
* Check if `PYENV_ROOT` is already defined.
* Check if `PYENV_ROOT` already exists before git cloning to avoid a git failure.

The key line is `${PYENV_BIN} global ${PYTHON_27_VERSION_UCS2}`, which will override whatever `${PYENV_ROOT}/shims/python2.7` used to point to.

stuhood added a commit that referenced this pull request Apr 8, 2019

Install Python 2.7 and 3.6 on Centos6 base image through Pyenv (#7064)
### Problem

We have used a python 3 interpreter in CI for a while now, using `pyenv` to install this in our Dockerfiles instead of relying on CentOS's `scl` features, which don't allow for multiple pythons at a time. Using `pyenv` to install Python 2.7 as well instead of relying on the package manager allows us to upgrade or modify the image without screwing up our Python installation.

That work began in this PR and ended up in #6981, #7352, #7418, #7483, and more, but it then led to having to duplicate code across `travis_ci` and `travis_ci_py36` Dockerfiles. Using `pyenv` to install python in our `travis_ci` images also involves a significant setup time in bootstrap phases to install our desired pythons. Installing Python in the CentOS 6 base image avoids having to install Python at all in our CI, which saves us minutes in the bootstrap phase.

### Solution

- Use Pyenv to install Python 2.7 and Python 3.6 in the `centos6` Dockerfile.
- Remove `travis_cy_py36`.
- Modify the `travis_ci` Dockerfile to bootstrap pyenv itself if not available.

### Result

We get some CI time back in the bootstrap phase.

### TODO

- [ ] merge this PR
- [ ] publish the new `centos6` image to dockerhub at `pantsbuild/centos6:latest`
- [ ] merge a followup PR removing the manual pyenv bootstrapping from the `travis_ci` image.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.