Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for releasing Python 3 wheels #7197

Merged
merged 31 commits into from Mar 14, 2019
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
1b556c9
Implement #6450
Eric-Arellano Jan 31, 2019
448c898
Remove pants_requirement.py env marker
Eric-Arellano Jan 31, 2019
539f1a3
Add py3 option to packages.py
Eric-Arellano Jan 31, 2019
179acca
Fix bad subprocess32 import for packages.py
Eric-Arellano Jan 31, 2019
9f7f204
Add -3 arg to release.sh & py3 support
Eric-Arellano Jan 31, 2019
d09fdc3
Add Py3 build wheel shards
Eric-Arellano Jan 31, 2019
f1812b8
Change pantsbuild.pants tag to use abi3
Eric-Arellano Feb 1, 2019
dae8385
Fix invalid travis entry
Eric-Arellano Feb 1, 2019
addd127
Remove leftover env_marker in pants_requirement.py
Eric-Arellano Feb 1, 2019
0ea887e
Clarify why --py3 must be passed first
Eric-Arellano Feb 1, 2019
af968c1
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Feb 5, 2019
5c493b0
Modify Pants BUILD to get ABI setting to apply
Eric-Arellano Feb 5, 2019
ce9445c
Constrain subprocesses to Py3 in releases.sh
Eric-Arellano Feb 5, 2019
dc85e1b
Squashed commit of the following:
Eric-Arellano Feb 27, 2019
bcd86ea
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Feb 27, 2019
aae3a5d
Use Pex36, not Pex37
Eric-Arellano Feb 27, 2019
b470c00
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 9, 2019
c2c51d0
Simplify handling of requirements
Eric-Arellano Mar 9, 2019
3d1bca8
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 10, 2019
3c93b79
Update expected number of wheels
Eric-Arellano Mar 10, 2019
c60b030
Move getopt reset to top of file
Eric-Arellano Mar 10, 2019
7f14443
-3 is not mutually exclusive
Eric-Arellano Mar 10, 2019
31de68e
Move usage() back down to bottom
Eric-Arellano Mar 10, 2019
1e1ed4e
Change expected pants.pex abi to cp36m
Eric-Arellano Mar 10, 2019
7beb55e
Improve readability of release.sh expected interpreter
Eric-Arellano Mar 10, 2019
0c3ccca
Improve comment in packages.py
Eric-Arellano Mar 10, 2019
789bb15
Remove unit test check for PantsRequirement interpreter constraints
Eric-Arellano Mar 11, 2019
aa33854
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 11, 2019
a292f73
Fix typo with default env var arg
Eric-Arellano Mar 12, 2019
ea40d0f
Merge branch 'master' of github.com:pantsbuild/pants into py3-wheels
Eric-Arellano Mar 12, 2019
1090078
Fix bad docker image from #7352
Eric-Arellano Mar 12, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -397,9 +397,8 @@ base_build_wheels: &base_build_wheels
env:
- &base_build_wheels_env PREPARE_DEPLOY=1

py27_linux_build_wheels_no_ucs: &py27_linux_build_wheels_no_ucs
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize
# compatibility. This is a Py2.7 shard, so it is not subject to #6985.
base_linux_build_wheels: &base_linux_build_wheels
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize compatibility.
<<: *travis_docker_image
<<: *base_build_wheels
# Callers of this anchor are expected to provide values in their `env` for
@@ -423,7 +422,7 @@ py27_linux_build_wheels_no_ucs: &py27_linux_build_wheels_no_ucs
py27_linux_build_wheels_ucs2: &py27_linux_build_wheels_ucs2
<<: *py27_linux_config
<<: *py27_linux_build_wheels_no_ucs
<<: *base_linux_build_wheels
<<: *native_engine_cache_config
name: "Build wheels - Linux and cp27m (UCS2)"
env:
@@ -435,7 +434,7 @@ py27_linux_build_wheels_ucs2: &py27_linux_build_wheels_ucs2
- CACHE_NAME=linuxwheelsbuild.ucs2

py27_linux_build_wheels_ucs4: &py27_linux_build_wheels_ucs4
<<: *py27_linux_build_wheels_no_ucs
<<: *base_linux_build_wheels
<<: *py27_linux_test_config
# `py27_linux_test_config` overrides the stage set by `base_build_wheels`, so we re-override it.
stage: *test
@@ -448,13 +447,25 @@ py27_linux_build_wheels_ucs4: &py27_linux_build_wheels_ucs4
&& RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n"
- CACHE_NAME=linuxwheelsbuild.ucs4

py27_osx_build_wheels_no_ucs: &py27_osx_build_wheels_no_ucs
py36_linux_build_wheels: &py36_linux_build_wheels
<<: *base_linux_build_wheels
<<: *py36_linux_test_config
name: "Build wheels - Linux and abi3 (Py3.6+)"
env:
- *py36_linux_test_config_env
- *base_build_wheels_env
- docker_image_name=travis_ci_py36
- docker_run_command="./build-support/bin/check_pants_pex_abi.py cp36m
&& RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -3n"
- CACHE_NAME=linuxwheelsbuild.abi3

base_osx_build_wheels: &base_osx_build_wheels
<<: *base_build_wheels
osx_image: xcode8

py27_osx_build_wheels_ucs2: &py27_osx_build_wheels_ucs2
<<: *py27_osx_test_config
<<: *py27_osx_build_wheels_no_ucs
<<: *base_osx_build_wheels
name: "Build wheels - OSX and cp27m (UCS2)"
env:
- *py27_osx_test_config_env
@@ -466,7 +477,7 @@ py27_osx_build_wheels_ucs2: &py27_osx_build_wheels_ucs2

py27_osx_build_wheels_ucs4: &py27_osx_build_wheels_ucs4
<<: *py27_osx_config
<<: *py27_osx_build_wheels_no_ucs
<<: *base_osx_build_wheels
<<: *native_engine_cache_config
name: "Build wheels - OSX and cp27mu (UCS4)"
addons:
@@ -497,6 +508,18 @@ py27_osx_build_wheels_ucs4: &py27_osx_build_wheels_ucs4
- ./build-support/bin/check_pants_pex_abi.py cp27mu
- RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n

py36_osx_build_wheels: &py36_osx_build_wheels
<<: *py36_osx_test_config
<<: *base_osx_build_wheels
name: "Build wheels - OSX and abi3 (Py3.6+)"
env:
- *py36_osx_test_config_env
- *base_build_wheels_env
- CACHE_NAME=osxwheelsbuild.abi3
script:
- ./build-support/bin/check_pants_pex_abi.py cp36m
- RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -3n

# -------------------------------------------------------------------------
# Rust tests
# -------------------------------------------------------------------------
@@ -720,8 +743,11 @@ matrix:

- <<: *py27_linux_build_wheels_ucs2
- <<: *py27_linux_build_wheels_ucs4
- <<: *py36_linux_build_wheels

- <<: *py27_osx_build_wheels_ucs2
- <<: *py27_osx_build_wheels_ucs4
- <<: *py36_osx_build_wheels

- <<: *py36_linux_test_config
name: "Integration tests for pants - shard 0 (Py3.6 PEX)"
@@ -7,20 +7,40 @@ set -e
ROOT=$(cd $(dirname "${BASH_SOURCE[0]}") && cd "$(git rev-parse --show-toplevel)" && pwd)
source ${ROOT}/build-support/common.sh

# Note we parse some options here, but parse most at the bottom. This is due to execution order.
# If the option must be used right away, we parse at the top of the script, whereas if it
# depends on functions defined later in the script, we parse at the end.
_OPTS="hdnftcloepqw3"

while getopts "${_OPTS}" opt; do
case ${opt} in
3) python_three="true" ;;
*) ;; # skip over other args to be parsed later
esac
done

# Reset opt parsing's position to start
OPTIND=0

# Set the Python interpreter to be used for the virtualenv. Note we allow the user to
# predefine this value so that they may point to a specific interpreter, e.g. 2.7.13 vs. 2.7.15.
export PY="${PY:-python2.7}"
default_interpreter="python2.7";
if [[ "${python_three:-false}" == "true" ]]; then
default_interpreter="python3.6"
fi
export PY="${PY:-${default_interpreter}}"
if ! which "${PY}" >/dev/null; then
die "Python interpreter ${PY} not discoverable on your PATH."
fi
py_major_minor=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))')
if [[ "${py_major_minor}" != "2.7" ]]; then
die "Invalid interpreter. The release script requires python2.7, and you are using python${py_major_minor}."
if [[ "${py_major_minor}" != "2.7" ]] && [[ "${py_major_minor}" != "3.6" ]]; then
die "Invalid interpreter. The release script requires Python 2.7 or 3.6 (you are using ${py_major_minor})."
fi

# Also set PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS. We set this to the exact Python version
# to resolve any potential ambiguity when multiple Python interpreters are discoverable, such as
# Python 2.7.13 vs. 2.7.15.
# Python 2.7.13 vs. 2.7.15. We must also set this when running with Python 3 to ensure
# that spawned subprocesses use Python 3.
py_major_minor_patch=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:3])))')
export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="${PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS:-['CPython==${py_major_minor_patch}']}"

@@ -55,11 +75,14 @@ function requirement() {
grep "^${package}[^A-Za-z0-9]" "${ROOT}/3rdparty/python/requirements.txt" || die "Could not find requirement for ${package}"
}

function run_pex27() {
function run_pex() {
# TODO: Cache this in case we run pex multiple times
(
PEX_VERSION="$(requirement pex | sed -e "s|pex==||")"
PEX_PEX=pex27
if [[ "${python_three:-false}" == "true" ]]; then
PEX_PEX=pex36
fi

pexdir="$(mktemp -d -t build_pex.XXXXX)"
trap "rm -rf ${pexdir}" EXIT
@@ -75,7 +98,12 @@ function run_pex27() {
function run_packages_script() {
(
cd "${ROOT}"
run_pex27 "$(requirement future)" "$(requirement beautifulsoup4)" "$(requirement configparser)" "$(requirement subprocess32)" -- "${ROOT}/src/python/pants/releases/packages.py" "$@"
args=("$@")
if [[ "${python_three:-false}" == "true" ]]; then
args=("--py3" ${args[@]})
fi
requirements=("$(requirement future)" "$(requirement beautifulsoup4)" "$(requirement configparser)" "$(requirement subprocess32)")
run_pex "${requirements[@]}" -- "${ROOT}/src/python/pants/releases/packages.py" "${args[@]}"
)
}

@@ -420,12 +448,16 @@ from __future__ import print_function
import sys
import urllib
import xml.etree.ElementTree as ET
try:
from urllib.parse import quote_plus
except ImportError:
from urllib import quote_plus
root = ET.parse("${wheel_listing}")
ns = {'s3': 'http://s3.amazonaws.com/doc/2006-03-01/'}
for key in root.findall('s3:Contents/s3:Key', ns):
# Because filenames may contain characters that have different meanings
# in URLs (namely '+'), # print the key both as url-encoded and as a file path.
print('{}\t{}'.format(key.text, urllib.quote_plus(key.text)))
print('{}\t{}'.format(key.text, quote_plus(key.text)))
EOF
done
}
@@ -483,9 +515,8 @@ function fetch_and_check_prebuilt_wheels() {
fi
done

# N.B. For platform-specific wheels, we expect 4 wheels: {linux,osx} * {cp27m,cp27mu}.
# Once we release Python 3 wheels, we will expect 6 wheels: {linux,osx} * {cp27m,cp27mu,abi3}.
if [ "${cross_platform}" != "true" ] && [ ${#packages[@]} -ne 4 ]; then
# N.B. For platform-specific wheels, we expect 6 wheels: {linux,osx} * {cp27m,cp27mu,abi3}.
if [ "${cross_platform}" != "true" ] && [ ${#packages[@]} -ne 6 ]; then

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Mar 13, 2019

Author Contributor

@benjyw this is updated, along with the release documentation.

Thanks for the review!

missing+=("${NAME} (expected whls for each platform: had only ${packages[@]})")
continue
fi
@@ -523,7 +554,7 @@ function activate_twine() {
}

function execute_pex() {
run_pex27 \
run_pex \
--no-build \
--no-pypi \
--disable-cache \
@@ -634,9 +665,10 @@ function usage() {
echo "PyPi. Credentials are needed for this as described in the"
echo "release docs: http://pantsbuild.org/release.html"
echo
echo "Usage: $0 [-d] [-c] (-h|-n|-f|-t|-l|-o|-e|-p)"
echo "Usage: $0 [-d] [-c] [-3] (-h|-n|-f|-t|-l|-o|-e|-p)"
echo " -d Enables debug mode (verbose output, script pauses after venv creation)"
echo " -h Prints out this help message."
echo " -3 Release any non-universal wheels (i.e. pantsbuild.pants) as Python 3. Defaults to Python 2."
echo " -n Performs a release dry run."
echo " All package distributions will be built, installed locally in"
echo " an ephemeral virtualenv and exercised to validate basic"
@@ -651,7 +683,7 @@ function usage() {
echo " -p Build a pex from prebuilt wheels for this release."
echo " -q Build a pex which only works on the host platform, using the code as exists on disk."
echo
echo "All options (except for '-d') are mutually exclusive."
echo "All options (except for '-d' and '-3') are mutually exclusive."

This comment has been minimized.

Copy link
@mateor

mateor Mar 14, 2019

Member

Might be useful to capture the mutual exclusive args in a variable so that this CLI help doesn't drift from reality. Fine to punt or do as a followup.

    echo "All options (except for ${_PANTS_OPTS_MUTUAL_EXCLUSIVE}) are mutually exclusive."

or some better label.

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Mar 14, 2019

Author Contributor

Good idea.

Even better will be using Python's argparse to handle this all for us + to allow us to use long-form arguments. Think it will become much more tenable when we drop Py2 and can start using subprocess.run([my, bash, command]) over subprocess.Popen().communicate().


if (( $# > 0 )); then
die "$@"
@@ -660,7 +692,7 @@ function usage() {
fi
}

while getopts "hdnftcloepqw" opt; do
while getopts "${_OPTS}" opt; do
case ${opt} in
h) usage ;;
d) debug="true" ;;
@@ -673,6 +705,7 @@ while getopts "hdnftcloepqw" opt; do
p) build_pex fetch ; exit $? ;;
q) build_pex build ; exit $? ;;
w) list_prebuilt_wheels ; exit $? ;;
3) ;; # already parsed at top of file
*) usage "Invalid option: -${OPTARG}" ;;
esac
done
@@ -361,9 +361,8 @@ base_build_wheels: &base_build_wheels
env:
- &base_build_wheels_env PREPARE_DEPLOY=1
py27_linux_build_wheels_no_ucs: &py27_linux_build_wheels_no_ucs
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize
# compatibility. This is a Py2.7 shard, so it is not subject to #6985.
base_linux_build_wheels: &base_linux_build_wheels
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize compatibility.
<<: *travis_docker_image
<<: *base_build_wheels
# Callers of this anchor are expected to provide values in their `env` for
@@ -376,7 +375,7 @@ py27_linux_build_wheels_no_ucs: &py27_linux_build_wheels_no_ucs
py27_linux_build_wheels_ucs2: &py27_linux_build_wheels_ucs2
<<: *py27_linux_config
<<: *py27_linux_build_wheels_no_ucs
<<: *base_linux_build_wheels
<<: *native_engine_cache_config
name: "Build wheels - Linux and cp27m (UCS2)"
env:
@@ -388,7 +387,7 @@ py27_linux_build_wheels_ucs2: &py27_linux_build_wheels_ucs2
- CACHE_NAME=linuxwheelsbuild.ucs2
py27_linux_build_wheels_ucs4: &py27_linux_build_wheels_ucs4
<<: *py27_linux_build_wheels_no_ucs
<<: *base_linux_build_wheels
<<: *py27_linux_test_config
# `py27_linux_test_config` overrides the stage set by `base_build_wheels`, so we re-override it.
stage: *test
@@ -401,13 +400,25 @@ py27_linux_build_wheels_ucs4: &py27_linux_build_wheels_ucs4
&& RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n"
- CACHE_NAME=linuxwheelsbuild.ucs4
py27_osx_build_wheels_no_ucs: &py27_osx_build_wheels_no_ucs
py36_linux_build_wheels: &py36_linux_build_wheels
<<: *base_linux_build_wheels
<<: *py36_linux_test_config
name: "Build wheels - Linux and abi3 (Py3.6+)"
env:
- *py36_linux_test_config_env
- *base_build_wheels_env
- docker_image_name=travis_ci_py36
- docker_run_command="./build-support/bin/check_pants_pex_abi.py cp36m
&& RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -3n"
- CACHE_NAME=linuxwheelsbuild.abi3
base_osx_build_wheels: &base_osx_build_wheels
<<: *base_build_wheels
osx_image: xcode8
py27_osx_build_wheels_ucs2: &py27_osx_build_wheels_ucs2
<<: *py27_osx_test_config
<<: *py27_osx_build_wheels_no_ucs
<<: *base_osx_build_wheels
name: "Build wheels - OSX and cp27m (UCS2)"
env:
- *py27_osx_test_config_env
@@ -419,7 +430,7 @@ py27_osx_build_wheels_ucs2: &py27_osx_build_wheels_ucs2
py27_osx_build_wheels_ucs4: &py27_osx_build_wheels_ucs4
<<: *py27_osx_config
<<: *py27_osx_build_wheels_no_ucs
<<: *base_osx_build_wheels
<<: *native_engine_cache_config
name: "Build wheels - OSX and cp27mu (UCS4)"
addons:
@@ -444,6 +455,18 @@ py27_osx_build_wheels_ucs4: &py27_osx_build_wheels_ucs4
- ./build-support/bin/check_pants_pex_abi.py cp27mu
- RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n
py36_osx_build_wheels: &py36_osx_build_wheels
<<: *py36_osx_test_config
<<: *base_osx_build_wheels
name: "Build wheels - OSX and abi3 (Py3.6+)"
env:
- *py36_osx_test_config_env
- *base_build_wheels_env
- CACHE_NAME=osxwheelsbuild.abi3
script:
- ./build-support/bin/check_pants_pex_abi.py cp36m
- RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -3n
# -------------------------------------------------------------------------
# Rust tests
# -------------------------------------------------------------------------
@@ -667,8 +690,11 @@ matrix:
- <<: *py27_linux_build_wheels_ucs2
- <<: *py27_linux_build_wheels_ucs4
- <<: *py36_linux_build_wheels
- <<: *py27_osx_build_wheels_ucs2
- <<: *py27_osx_build_wheels_ucs4
- <<: *py36_osx_build_wheels
{{#py3_integration_shards}}
- <<: *py36_linux_test_config
@@ -156,7 +156,7 @@ is not required.
3. Publish to PyPi
------------------

Once the four "Build wheels" Travis shards have completed for your release
Once the six "Build wheels" Travis shards have completed for your release
commit, you can publish to PyPi. First, ensure that you are on your release branch at your version
bump commit. Then, publish the release:

@@ -50,14 +50,7 @@ def __call__(self, name=None, dist=None):
msg='The {} target only works for pantsbuild.pants '
'distributions, given {}'.format(self.alias, dist))

# Update the environment marker in lockstep with other changes as described in
# https://github.com/pantsbuild/pants/issues/6450
env_marker = "python_version>='2.7' and python_version<'3'"

requirement = PythonRequirement(requirement="{key}=={version} ; {env_marker}"
.format(key=dist,
version=pants_version(),
env_marker=env_marker))
requirement = PythonRequirement(requirement="{key}=={version}".format(key=dist, version=pants_version()))

self._parse_context.create_object('python_requirement_library',
name=name,
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.