Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for releasing Python 3 wheels #7197

Merged
merged 31 commits into from Mar 14, 2019
Merged
Changes from 8 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
1b556c9
Implement #6450
Eric-Arellano Jan 31, 2019
448c898
Remove pants_requirement.py env marker
Eric-Arellano Jan 31, 2019
539f1a3
Add py3 option to packages.py
Eric-Arellano Jan 31, 2019
179acca
Fix bad subprocess32 import for packages.py
Eric-Arellano Jan 31, 2019
9f7f204
Add -3 arg to release.sh & py3 support
Eric-Arellano Jan 31, 2019
d09fdc3
Add Py3 build wheel shards
Eric-Arellano Jan 31, 2019
f1812b8
Change pantsbuild.pants tag to use abi3
Eric-Arellano Feb 1, 2019
dae8385
Fix invalid travis entry
Eric-Arellano Feb 1, 2019
addd127
Remove leftover env_marker in pants_requirement.py
Eric-Arellano Feb 1, 2019
0ea887e
Clarify why --py3 must be passed first
Eric-Arellano Feb 1, 2019
af968c1
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Feb 5, 2019
5c493b0
Modify Pants BUILD to get ABI setting to apply
Eric-Arellano Feb 5, 2019
ce9445c
Constrain subprocesses to Py3 in releases.sh
Eric-Arellano Feb 5, 2019
dc85e1b
Squashed commit of the following:
Eric-Arellano Feb 27, 2019
bcd86ea
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Feb 27, 2019
aae3a5d
Use Pex36, not Pex37
Eric-Arellano Feb 27, 2019
b470c00
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 9, 2019
c2c51d0
Simplify handling of requirements
Eric-Arellano Mar 9, 2019
3d1bca8
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 10, 2019
3c93b79
Update expected number of wheels
Eric-Arellano Mar 10, 2019
c60b030
Move getopt reset to top of file
Eric-Arellano Mar 10, 2019
7f14443
-3 is not mutually exclusive
Eric-Arellano Mar 10, 2019
31de68e
Move usage() back down to bottom
Eric-Arellano Mar 10, 2019
1e1ed4e
Change expected pants.pex abi to cp36m
Eric-Arellano Mar 10, 2019
7beb55e
Improve readability of release.sh expected interpreter
Eric-Arellano Mar 10, 2019
0c3ccca
Improve comment in packages.py
Eric-Arellano Mar 10, 2019
789bb15
Remove unit test check for PantsRequirement interpreter constraints
Eric-Arellano Mar 11, 2019
aa33854
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 11, 2019
a292f73
Fix typo with default env var arg
Eric-Arellano Mar 12, 2019
ea40d0f
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 12, 2019
1090078
Fix bad docker image from #7352
Eric-Arellano Mar 12, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -386,36 +386,63 @@ base_build_wheels: &base_build_wheels
env:
- &base_build_wheels_env RUN_PANTS_FROM_PEX=1 PREPARE_DEPLOY=1

linux_build_wheels: &linux_build_wheels
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize
# compatibility. This is a Py2.7 shard, so it is not subject to #6985.
base_linux_build_wheels: &base_linux_build_wheels
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize compatibility.
<<: *travis_docker_image
<<: *py27_linux_test_config
<<: *base_build_wheels
name: "Build Linux wheels (No PEX)"
env:
- *py27_linux_test_config_env
- *base_build_wheels_env
- CACHE_NAME=linuxwheelsbuild
script:
- *travis_docker_image_launch
- docker run --rm -t
-v "${HOME}:/travis/home"
-v "${TRAVIS_BUILD_DIR}:/travis/workdir"
travis_ci:latest
sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n"
sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh ${RELEASE_ARGS} -n"

osx_build_wheels: &osx_build_wheels
<<: *py27_osx_test_config
py27_linux_build_wheels: &py27_linux_build_wheels
<<: *py27_linux_test_config
<<: *base_linux_build_wheels
name: "Build Linux wheels (Py2.7 PEX)"
env:
- *py27_linux_test_config_env
- *base_build_wheels_env
- RELEASE_ARGS=''
- CACHE_NAME=linuxwheelsbuild.py27

py36_linux_build_wheels: &py36_linux_build_wheels
<<: *py36_linux_test_config
<<: *base_linux_build_wheels
name: "Build Linux wheels (Py3.6 PEX)"
env:
- *py36_linux_test_config_env
- *base_build_wheels_env
- RELEASE_ARGS='-3'
- CACHE_NAME=linuxwheelsbuild.py36

base_osx_build_wheels: &base_osx_build_wheels
<<: *base_build_wheels
name: "Build OSX wheels (No PEX)"
osx_image: xcode8
script:
- ./build-support/bin/release.sh ${RELEASE_ARGS} -n

py27_osx_build_wheels: &py27_osx_build_wheels
<<: *py27_osx_test_config
<<: *base_osx_build_wheels
name: "Build OSX wheels (Py2.7 PEX)"
env:
- *py27_osx_test_config_env
- *base_build_wheels_env
- CACHE_NAME=osxwheelsbuild
script:
- ./build-support/bin/release.sh -n
- RELEASE_ARGS=''
- CACHE_NAME=osxwheelsbuild.py27

py36_osx_build_wheels: &py36_osx_build_wheels
<<: *py36_osx_test_config
<<: *base_osx_build_wheels
name: "Build OSX wheels (Py3.6 PEX)"
env:
- *py36_osx_test_config_env
- *base_build_wheels_env
- RELEASE_ARGS='-3'
- CACHE_NAME=osxwheelsbuild.py36

# -------------------------------------------------------------------------
# Rust tests
@@ -617,8 +644,10 @@ matrix:
- <<: *linux_rust_clippy
- <<: *cargo_audit

- <<: *linux_build_wheels
- <<: *osx_build_wheels
- <<: *py27_linux_build_wheels
- <<: *py36_linux_build_wheels
- <<: *py27_osx_build_wheels
- <<: *py36_osx_build_wheels

- <<: *py27_linux_test_config
name: "Unit tests for pants and pants-plugins (Py2.7 PEX)"
@@ -7,12 +7,64 @@ set -e
ROOT=$(cd $(dirname "${BASH_SOURCE[0]}") && cd "$(git rev-parse --show-toplevel)" && pwd)
source ${ROOT}/build-support/common.sh

PY=$(which python2.7 || exit 0)
[[ -n "${PY}" ]] || die "You must have python2.7 installed and on the path to release."
# Note we define all options here, but only parse some at the top of the script and parse the rest
This conversation was marked as resolved by Eric-Arellano

This comment has been minimized.

Copy link
@stuhood

stuhood Feb 1, 2019

Member

While it does look like the double options parsing is necessary (according to https://stackoverflow.com/questions/15079009/is-it-possible-to-specify-the-order-getopts-conditions-are-executed), it doesn't look like it's necessary for the two parses to be separated within the file. I think it would be preferable to have the two while loops right after one another.

The consumption of the variables defined (py_version_number in particular) happens lazily in functions... so it's fine if they're not set until later in the file.

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Feb 1, 2019

Author Contributor

The challenge is that all the getopts logic at the bottom of the file immediately executes statements like run_packages_script list-owners, rather than setting a variable like execute_list_owners=true which is lazily evaluated.

For this reason, I originally moved all opts parsing to the top of the file in the release-refactor PR. But per discussion with John here #7190 (comment), we decided to adopt this split opt parsing approach instead.

I'm not sold 100% that this was the better decision and am happy to revisit it. If we end up wanting all opt parsing back at the top (see 43cc18f for what this looks like), I'll break it out as a separate precursor PR.

# at the bottom of the script. This is due to execution order. If the option must be used right away,
# we parse at the top of the script, whereas if it depends on functions defined later in the script,
# we parse at the end.
_OPTS="hdnftcloepqw3"
function usage() {
echo "With no options all packages are built, smoke tested and published to"
echo "PyPi. Credentials are needed for this as described in the"
echo "release docs: http://pantsbuild.org/release.html"
echo
echo "Usage: $0 [-d] [-c] [-3] (-h|-n|-f|-t|-l|-o|-e|-p)"
echo " -d Enables debug mode (verbose output, script pauses after venv creation)"
echo " -h Prints out this help message."
echo " -3 Release any non-universal wheels (i.e. pantsbuild.pants) as Python 3. Defaults to Python 2."
echo " -n Performs a release dry run."
echo " All package distributions will be built, installed locally in"
echo " an ephemeral virtualenv and exercised to validate basic"
echo " functioning."
echo " -f Build the fs_util binary."
echo " -t Tests a live release."
echo " Ensures the latest packages have been propagated to PyPi"
echo " and can be installed in an ephemeral virtualenv."
echo " -l Lists all pantsbuild packages that this script releases."
echo " -o Lists all pantsbuild package owners."
echo " -e Check that wheels are prebuilt for this release."
echo " -p Build a pex from prebuilt wheels for this release."
echo " -q Build a pex which only works on the host platform, using the code as exists on disk."
echo
echo "All options (except for '-d') are mutually exclusive."

if (( $# > 0 )); then
die "$@"
else
exit 0
fi
}

while getopts "${_OPTS}" opt; do
case ${opt} in
3) python_three="true" ;;
*) ;; # skip over other args to be parsed later
esac
done

py_version_number="2.7"
if [[ "${python_three}" == "true" ]]; then
py_version_number="3.6"
fi
PY=$(which "python${py_version_number}" || exit 0)
[[ -n "${PY}" ]] || die "You must have python2.7 or python3.6 installed and on the path to release."
export PY

function run_local_pants() {
${ROOT}/pants "$@"
pants_script="${ROOT}/pants"
if [[ "${python_three}" == "true" ]]; then
pants_script="${ROOT}/pants3"
fi
${pants_script} "$@"
}

# NB: Pants core does not have the ability to change its own version, so we compute the
@@ -42,11 +94,16 @@ function requirement() {
grep "^${package}[^A-Za-z0-9]" "${ROOT}/3rdparty/python/requirements.txt" || die "Could not find requirement for ${package}"
}

function run_pex27() {
function run_pex() {
# TODO: Cache this in case we run pex multiple times
(
PEX_VERSION="$(requirement pex | sed -e "s|pex==||")"
PEX_PEX=pex27
if [[ "${python_three}" == "true" ]]; then
This conversation was marked as resolved by Eric-Arellano

This comment has been minimized.

Copy link
@stuhood

stuhood Feb 1, 2019

Member

I don't think that it is necessary to use a particular pex release here... any pex release should be fine to build a pex for either 2 or 3.

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Feb 1, 2019

Author Contributor

I think it does make a difference.

For example, with all the other changes made in this script but removing this section of code to always use pex27 no matter what, ./build-support/bin/release.sh -3n will fail with this error:

Traceback (most recent call last):
  File "/private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/tmpEMUnm5/.bootstrap/_pex/pex.py", line 349, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/tmpEMUnm5/.bootstrap/_pex/pex.py", line 281, in _wrap_coverage
    return runner(*args)
  File "/private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/tmpEMUnm5/.bootstrap/_pex/pex.py", line 312, in _wrap_profiling
    return runner(*args)
  File "/private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/tmpEMUnm5/.bootstrap/_pex/pex.py", line 397, in _execute
    return self.execute_interpreter()
  File "/private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/tmpEMUnm5/.bootstrap/_pex/pex.py", line 456, in execute_interpreter
    self.execute_content(arg, content)
  File "/private/var/folders/sx/pdpbqz4x5cscn9hhfpbsbqvm0000gn/T/tmpEMUnm5/.bootstrap/_pex/pex.py", line 491, in execute_content
    exec_function(ast, globals())
  File "<exec_function>", line 4, in exec_function
  File "/Users/eric/DocsLocal/code/projects/pants/src/python/pants/releases/packages.py", line 13, in <module>
    from configparser import ConfigParser
ImportError: No module named configparser

While we can get around that by passing the backport as a requirement always, whereas now we only pass if releasing with Py2, this failure speaks to the fact that pex27 indeed means most the processes will be run using Py2.

--

I think we do want to execute every process with Py3 when -3 is enabled. For the same reason John agreed over Slack we should require 3.6 instead of 3.7 when building Py3 wheels (like we use xcode8 and Docker travis), we want to use Py3.6 to build the wheel for max compatibility.

# TODO(pex#654). Once PEX is released more flexibly, set this to `pex3` or `pex` (depending
# on what we end up releasing).
PEX_PEX=pex37
fi

pexdir="$(mktemp -d -t build_pex.XXXXX)"
trap "rm -rf ${pexdir}" EXIT
@@ -62,7 +119,14 @@ function run_pex27() {
function run_packages_script() {
(
cd "${ROOT}"
run_pex27 "$(requirement future)" "$(requirement beautifulsoup4)" "$(requirement configparser)" "$(requirement subprocess32)" -- "${ROOT}/src/python/pants/releases/packages.py" "$@"
requirements=("$(requirement future)" "$(requirement beautifulsoup4)")
args=("$@")
if [[ "${python_three}" == "true" ]]; then
args=("--py3" ${args[@]})
else
requirements+=("$(requirement configparser)" "$(requirement subprocess32)")
fi
run_pex "${requirements[@]}" -- "${ROOT}/src/python/pants/releases/packages.py" "${args[@]}"
)
}

@@ -407,12 +471,16 @@ from __future__ import print_function
import sys
import urllib
import xml.etree.ElementTree as ET
try:
from urllib.parse import quote_plus
except ImportError:
from urllib import quote_plus
root = ET.parse("${wheel_listing}")
ns = {'s3': 'http://s3.amazonaws.com/doc/2006-03-01/'}
for key in root.findall('s3:Contents/s3:Key', ns):
# Because filenames may contain characters that have different meanings
# in URLs (namely '+'), # print the key both as url-encoded and as a file path.
print('{}\t{}'.format(key.text, urllib.quote_plus(key.text)))
print('{}\t{}'.format(key.text, quote_plus(key.text)))
EOF
done
}
@@ -508,7 +576,7 @@ function activate_twine() {
}

function execute_pex() {
run_pex27 \
run_pex \
--no-build \
--no-pypi \
--disable-cache \
@@ -614,38 +682,9 @@ function publish_packages() {
end_travis_section
}

function usage() {
echo "With no options all packages are built, smoke tested and published to"
echo "PyPi. Credentials are needed for this as described in the"
echo "release docs: http://pantsbuild.org/release.html"
echo
echo "Usage: $0 [-d] [-c] (-h|-n|-f|-t|-l|-o|-e|-p)"
echo " -d Enables debug mode (verbose output, script pauses after venv creation)"
echo " -h Prints out this help message."
echo " -n Performs a release dry run."
echo " All package distributions will be built, installed locally in"
echo " an ephemeral virtualenv and exercised to validate basic"
echo " functioning."
echo " -f Build the fs_util binary."
echo " -t Tests a live release."
echo " Ensures the latest packages have been propagated to PyPi"
echo " and can be installed in an ephemeral virtualenv."
echo " -l Lists all pantsbuild packages that this script releases."
echo " -o Lists all pantsbuild package owners."
echo " -e Check that wheels are prebuilt for this release."
echo " -p Build a pex from prebuilt wheels for this release."
echo " -q Build a pex which only works on the host platform, using the code as exists on disk."
echo
echo "All options (except for '-d') are mutually exclusive."

if (( $# > 0 )); then
die "$@"
else
exit 0
fi
}

while getopts "hdnftcloepqw" opt; do
# Reset opt parsing's position to start
OPTIND=0
while getopts "${_OPTS}" opt; do
case ${opt} in
h) usage ;;
d) debug="true" ;;
@@ -658,6 +697,7 @@ while getopts "hdnftcloepqw" opt; do
p) build_pex fetch ; exit $? ;;
q) build_pex build ; exit $? ;;
w) list_prebuilt_wheels ; exit $? ;;
3) ;; # already parsed at top of file
*) usage "Invalid option: -${OPTARG}" ;;
esac
done
@@ -365,36 +365,63 @@ base_build_wheels: &base_build_wheels
env:
- &base_build_wheels_env RUN_PANTS_FROM_PEX=1 PREPARE_DEPLOY=1
linux_build_wheels: &linux_build_wheels
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize
# compatibility. This is a Py2.7 shard, so it is not subject to #6985.
base_linux_build_wheels: &base_linux_build_wheels
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize compatibility.
<<: *travis_docker_image
<<: *py27_linux_test_config
<<: *base_build_wheels
name: "Build Linux wheels (No PEX)"
env:
- *py27_linux_test_config_env
- *base_build_wheels_env
- CACHE_NAME=linuxwheelsbuild
script:
- *travis_docker_image_launch
- docker run --rm -t
-v "${HOME}:/travis/home"
-v "${TRAVIS_BUILD_DIR}:/travis/workdir"
travis_ci:latest
sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n"
sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh ${RELEASE_ARGS} -n"
osx_build_wheels: &osx_build_wheels
<<: *py27_osx_test_config
py27_linux_build_wheels: &py27_linux_build_wheels
<<: *py27_linux_test_config
<<: *base_linux_build_wheels
name: "Build Linux wheels (Py2.7 PEX)"
env:
- *py27_linux_test_config_env
- *base_build_wheels_env
- RELEASE_ARGS=''
- CACHE_NAME=linuxwheelsbuild.py27
py36_linux_build_wheels: &py36_linux_build_wheels
<<: *py36_linux_test_config
<<: *base_linux_build_wheels
name: "Build Linux wheels (Py3.6 PEX)"
env:
- *py36_linux_test_config_env
- *base_build_wheels_env
- RELEASE_ARGS='-3'
- CACHE_NAME=linuxwheelsbuild.py36
base_osx_build_wheels: &base_osx_build_wheels
<<: *base_build_wheels
name: "Build OSX wheels (No PEX)"
osx_image: xcode8
script:
- ./build-support/bin/release.sh ${RELEASE_ARGS} -n
py27_osx_build_wheels: &py27_osx_build_wheels
<<: *py27_osx_test_config
<<: *base_osx_build_wheels
name: "Build OSX wheels (Py2.7 PEX)"
env:
- *py27_osx_test_config_env
- *base_build_wheels_env
- CACHE_NAME=osxwheelsbuild
script:
- ./build-support/bin/release.sh -n
- RELEASE_ARGS=''
- CACHE_NAME=osxwheelsbuild.py27
py36_osx_build_wheels: &py36_osx_build_wheels
<<: *py36_osx_test_config
<<: *base_osx_build_wheels
name: "Build OSX wheels (Py3.6 PEX)"
env:
- *py36_osx_test_config_env
- *base_build_wheels_env
- RELEASE_ARGS='-3'
- CACHE_NAME=osxwheelsbuild.py36
# -------------------------------------------------------------------------
# Rust tests
@@ -596,8 +623,10 @@ matrix:
- <<: *linux_rust_clippy
- <<: *cargo_audit
- <<: *linux_build_wheels
- <<: *osx_build_wheels
- <<: *py27_linux_build_wheels
- <<: *py36_linux_build_wheels
- <<: *py27_osx_build_wheels
- <<: *py36_osx_build_wheels
- <<: *py27_linux_test_config
name: "Unit tests for pants and pants-plugins (Py2.7 PEX)"
@@ -50,14 +50,8 @@ def __call__(self, name=None, dist=None):
msg='The {} target only works for pantsbuild.pants '
'distributions, given {}'.format(self.alias, dist))

# Update the environment marker in lockstep with other changes as described in
# https://github.com/pantsbuild/pants/issues/6450
env_marker = "python_version>='2.7' and python_version<'3'"

requirement = PythonRequirement(requirement="{key}=={version} ; {env_marker}"
This conversation was marked as resolved by Eric-Arellano

This comment has been minimized.

Copy link
@stuhood

stuhood Feb 1, 2019

Member

The env_marker isn't set in this format string anymore.

.format(key=dist,
version=pants_version(),
env_marker=env_marker))
.format(key=dist, version=pants_version()))

self._parse_context.create_object('python_requirement_library',
name=name,
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.