New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Travis finally released xenial (16.04). #6885

Merged
merged 18 commits into from Dec 12, 2018

Conversation

Projects
None yet
4 participants
@OniOni
Copy link
Contributor

OniOni commented Dec 7, 2018

Travis released Xenial (16.04) (see https://changelog.travis-ci.com/ubuntu-xenial-16-04-build-environment-is-here!-79690) This PR moves us to this new image.

The two major changes are:

  • Switch to Python 3.5 in CI.
    This coincides with prior discussions in Slack that we want to target Python 3.5 as the minimum Python 3 version. This is to give us access to native type hints and async/await syntax, along with general improvements such as fixing the shutil bug identified in #6882.

  • Switch to openjdk over oraclejdk. This is mostly because travis only bakes version 10 and 11 of the oracle jdk and has removed all the tooling to get other versions of that jdk. While image comes with version 8 of the openjdk.

@OniOni OniOni changed the title Travis finally released xenial (16.04). [WIP] Travis finally released xenial (16.04). Dec 10, 2018

@OniOni OniOni changed the title [WIP] Travis finally released xenial (16.04). Travis finally released xenial (16.04). Dec 11, 2018

@Eric-Arellano
Copy link
Contributor

Eric-Arellano left a comment

Thank you Mathieu for pushing this through!

John and Stu, if you look at logs from most recent logs you’ll see how Python now resolves, including Pyenv installing 3.6 and 3.7. (I’m afk, sorry I can’t grab those lines directly)

@stuhood
Copy link
Member

stuhood left a comment

Thanks, looks good. Would figure out whether the JAVA_HOME entry is actually necessary, and would wait for @jsirois to sanity check whether build-support/bin/travis-ci.sh is still resulting in relevant python(s).

.travis.yml Outdated
- sudo rm -rf /usr/lib/jvm/java-6-openjdk-amd64
- jdk_switcher use oraclejdk8
- PATH="/usr/lib/jvm/java-8-openjdk-amd64/jre/bin":$PATH
- JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64k

This comment has been minimized.

@stuhood

stuhood Dec 12, 2018

Member

There is a stray k on the end here, which makes me wonder whether this JAVA_HOME entry is actually being used.

This comment has been minimized.

@OniOni

OniOni Dec 12, 2018

Contributor

That is a very good point. I had removed it, believing it was not needed, but it was causing failures. I'll give it a spin without the JAVA_HOME, for sanity's sake.

This comment has been minimized.

@OniOni

OniOni Dec 12, 2018

Contributor

Yeah, absence of JAVA_HOME causes a lot of failures.

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Dec 12, 2018

The log output does show relevant pythons, and travis-ci.sh concocts this output by actually invoking the interpreters in question:

function list_pythons {
which -a python{,2,3} | sort -u | while read python_bin; do
${python_bin} <<EOF
import os
import sys
print('{} -> {}'.format(os.path.realpath(sys.executable),
'.'.join(map(str, sys.version_info[:3]))))
EOF
done
}

Picking a random shard I read:

Visible pythons are now:
/home/travis/virtualenv/python2.7.15/bin/python -> 2.7.15
/opt/python/2.7.15/bin/python2.7 -> 2.7.15
/opt/python/3.6.7/bin/python3.6 -> 3.6.7
/opt/python/3.7.1/bin/python3.7 -> 3.7.1
/usr/bin/python2.7 -> 2.7.12
/usr/bin/python3.5 -> 3.5.2

LGTM

We may no longer need the travis-ci.sh wrapper, but that can be for another change.

OniOni added some commits Dec 12, 2018

@Eric-Arellano Eric-Arellano merged commit 7da3ab5 into pantsbuild:master Dec 12, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

Eric-Arellano added a commit that referenced this pull request Dec 12, 2018

Unblacklist JVM check_style Python 3 integration test thanks to using…
… Python 3.5+ (#6882)

## Problem
shutil.copytree was failing in CI for the `backend/jvm/tasks:checkstyle_integration` target due to a bug with Python 3.4.

```
shutil.Error: [('/home/travis/build/pantsbuild/pants/build-support/bin/native/src', '/tmp/tmpjdehtsq4/build-support/bin/native/src', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/bin/native/src'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/include/python2.7', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/include/python2.7', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/include/python2.7'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/config', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/lib/python2.7/config', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/config'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/encodings', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/lib/python2.7/encodings', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/encodings'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/lib-dynload', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/lib/python2.7/lib-dynload', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/lib-dynload'")]
```

This error is confusing, because those are intentionally directories. Also, it doesn't reproduce on Python 2.7 and other Python 3 versions.

The bug is from the [std lib's handling of symbolic directories](https://bugs.python.org/issue21697). `shutil.copytree` treated all symbolic folders as files, which leads to the error `is a directory`. This was fixed in Python 3.5+.

## Solution
We upgraded Travis to Python 3.5 with #6885. This allows us to raise the floor of minimum Python 3 to 3.5, fixing this bug.

Eric-Arellano added a commit to Eric-Arellano/pants that referenced this pull request Dec 15, 2018

Eric-Arellano added a commit that referenced this pull request Dec 15, 2018

Fix Travis environment variable overriding interpreter_selection test…
…'s config (#6939)

### Problem
In `test_conflict_via_config()`, we try to constraint Python to `< 2.7`. However, this fails to do anything when running on Python 3, because it's overriden by `ci.sh`'s environment variable `export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS='["CPython>=3.5,<4"]'`.

This is expected behavior, that the environment variable overrides the command line argument. So, the issue is not with the source code, but this test should also not be failing.

### Solution
Set the integration test to be hermetic.

### Other related interpreter selection changes
* Allow Python 3.7 now that it's been fixed with #6363 
* In CI, require for Python 3 mode Python 3.5+, now that we use xenial (#6885)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment