Skip to content
Permalink
Browse files

Specify ABI for pantsbuild.pants wheel and build with both UCS2 and U…

…CS4 (#7235)

### Problem
We should be marking the [ABI (application binary interface)](https://docs.python.org/3/c-api/stable.html) for the `pantsbuild.pants` wheel because it uses native code. Currently, we mark the ABI as `none`, which is incorrect per https://www.python.org/dev/peps/pep-0513/#ucs-2-vs-ucs-4-builds.

In particular, in Python 2, Python may be installed with either UCS2 (UTF-16) or UCS4 (UTF-8). We should be marking the wheel as either `cp27m` for UCS2 or `cp27mu` for UCS4.

As a result of marking the ABI, we must now produce more wheels. macOS defaults to UCS2. For Linux, "ucs4 is much more widespread among Linux CPython distributions." We do not want to rely on these assumptions, however, when releasing, as some users may not have these default unicode settings. So, instead we must release `pantsbuild.pants` as both a `cp27m` and `cp27mu` wheel, and rely on Pip to resolve which the user should use.

### Solution
At a high level, this PR does two things:
1. Marks that the ABI should be specified, rather than `none`.
1. Sets up 4 Travis shards so that we build both a `cp27m` and `cp27mu` wheel for both Linux and OSX. See https://travis-ci.org/pantsbuild/pants/builds/503639333 for the end result of this.

To setup the new shards, we use Pyenv to install new versions of Python 2 with the appropriate unicode settings, thanks to the env var `PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs{2,4}` (https://stackoverflow.com/a/38930764). 

Because both the OSX UCS4 shard and Linux UCS2 shard already have Python 2.7 installed, we must install a Python 2.7.x version different than what is already there, and use `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS` to ensure Pants and PEX are using the exact interpreter we want. For this reason, this PR was blocked by #7285 to propagate interpreter constraints to PEX.

Specifically, we make these changes to achieve these two high level goals:

1. Modify [`src/python/pants/BUILD`](https://github.com/pantsbuild/pants/pull/7235/files#diff-3ce39309d74098493a1f3c8107292a8d) so that `bdist_wheel` knows it needs to mark the ABI. This achieves goal 1.
1. Change [`release.sh`](https://github.com/pantsbuild/pants/pull/7235/files#diff-9ed7102b7836807dc342cc2246ec4839) to allow pre-setting `$PY` and to also set `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS` in order to use the specific Python interpreter we are targeting.
1. Create [`travis_ci_py27_ucs2/Dockerfile`](https://github.com/pantsbuild/pants/pull/7235/files#diff-b90425bcccb6969a98b9d1c4066422a8) to get Python 2 w/ UCS2 onto Linux.
1. Extract out `.travis.yml` `env` entries to get OpenSSL and Homebrew-installed Python working properly, along with launching a Docker image, in order to avoid duplication: [`env_osx_with_pyenv.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-c2ab029a1887e2e99f2391fc7568cc5f), [`launch_docker_image.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-3d4a0cecc373a624f233521f76d66999), and [`generate_travis_yml.py`](https://github.com/pantsbuild/pants/pull/7235/files#diff-c11e2f109e12527d2e1ac2c62161edf6).
1. Modify [`travis.yml.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-88af3146f5cc486b749ed790399bde46) to set up the 4 distinct wheel building shards. Similar to how we created a Dockerfile to use Pyenv to install Python 2 with UCS2 on Linux, we use Pyenv to install Python 2 with UCS4 on OSX. We also move the wheel building shards below unit tests.

#### Ensuring the correct abi is used
We need to ensure the `pants.pex` used by `release.sh` has the correct abi for its dependencies, and that `release.sh` is using the correct Python interpreter. We introduce a new script [`check_pants_pex_abi.py`](https://github.com/pantsbuild/pants/pull/7235/files#diff-8b857b8cee6cb9784bf37950220c587b) that inspects the pex's `PEX-INFO` to ensure the targeted abi was used.

An even better test would test the result of `release.sh` to ensure the built `pantsbuild.pants` wheel has the correct ABI and can be consumed properly. Currently `release.sh` verifies the wheel is valid, but it does not enforce which ABI it was built with. This could be a good followup PR.

### Result
We now properly mark the ABI for Python 2. Beyond the new script `check_pants_pex_abi.sh` proving this, we performed a run of this PR with verbose PEX logging turned on: https://travis-ci.org/pantsbuild/pants/builds/503639333. Inspecting the logs for the wheel building shards and searching for `Using the current platform` proves the 4 wheel building shards are using the correct interpreter and abi.

In addition to correctness for Python 2, this unblocks releasing Python 3 wheels (#7197).

Note this should have no significant impact on the end user, as Pip will resolve to the current ABI for their interpreter. It will change the name of our `pantsbuild.pants` wheel and will prevent using that wheel with an interpreter that uses a different UCS setting, but all users should be able to pull down whichever wheel they need as we provide wheels for both UCS2 and UCS4 on both OSX and Linux.

#### Downside: wheel building explosion
We currently are building more wheels than necessary. For wheels that are universal / platform-independent, we only need them to be built once, but we build them every time. See #7258.

This PR adds two new shards so adds ~30 unnecessary core wheels we build, in addition to 3rd party wheels that are universal / platform-independent.
  • Loading branch information...
Eric-Arellano committed Mar 9, 2019
1 parent e65723a commit 7b2225e0e0e004db6d99abf14329c6bccac65bf8
@@ -170,7 +170,6 @@ py36_osx_config: &py36_osx_config
packages: &py36_osx_config_brew_packages
- openssl
env:
# Fix Python 3 issue linking to OpenSSL
- &py36_osx_config_env >
PATH="/usr/local/opt/openssl/bin:$PATH"
LDFLAGS="-L/usr/local/opt/openssl/lib"
@@ -234,14 +233,14 @@ travis_docker_image: &travis_docker_image
before_script:
- ulimit -c unlimited
script:
# NB: This script definition is very likely to be overridden in a consumer, so we alias it
# for re-inclusion.
- &travis_docker_image_launch docker build --rm -t travis_ci
- >
docker build
--rm -t ${docker_image_name}
--build-arg "TRAVIS_USER=$(id -un)"
--build-arg "TRAVIS_UID=$(id -u)"
--build-arg "TRAVIS_GROUP=$(id -gn)"
--build-arg "TRAVIS_GID=$(id -g)"
build-support/docker/travis_ci/
build-support/docker/${docker_image_name}/
# -------------------------------------------------------------------------
# Bootstrap engine shards
@@ -252,14 +251,21 @@ base_linux_build_engine: &base_linux_build_engine
<<: *travis_docker_image
stage: *bootstrap
script:
- *travis_docker_image_launch
- >
docker build
--rm -t ${docker_image_name}
--build-arg "TRAVIS_USER=$(id -un)"
--build-arg "TRAVIS_UID=$(id -u)"
--build-arg "TRAVIS_GROUP=$(id -gn)"
--build-arg "TRAVIS_GID=$(id -g)"
build-support/docker/${docker_image_name}/
# Note that:
# * We mount ${HOME} to cache the ${HOME}/.cache/pants/rust-toolchain.
# * We also build fs_util, to take advantage of the rust code built during bootstrapping.
- docker run --rm -t
-v "${HOME}:/travis/home"
-v "${TRAVIS_BUILD_DIR}:/travis/workdir"
travis_ci:latest
${docker_image_name}:latest
sh -c "./build-support/bin/ci.sh ${BOOTSTRAP_ARGS} && ./build-support/bin/release.sh -f"
- aws --no-sign-request --region us-east-1 s3 cp ${TRAVIS_BUILD_DIR}/pants.pex ${BOOTSTRAPPED_PEX_URL_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX}

@@ -268,6 +274,7 @@ py27_linux_build_engine: &py27_linux_build_engine
<<: *base_linux_build_engine
name: "Build Linux native engine and pants.pex (Py2.7 PEX)"
env:
- docker_image_name=travis_ci
# NB: Only the Py2.7 shard sets PREPARE_DEPLOY to cause the fs_util binary to be uploaded to S3:
# either linux shard could upload this binary, since it does not depend on python at all.
- PREPARE_DEPLOY=1
@@ -280,6 +287,7 @@ py36_linux_build_engine: &py36_linux_build_engine
<<: *base_linux_build_engine
name: "Build Linux native engine and pants.pex (Py3.6 PEX)"
env:
- docker_image_name=travis_ci
- CACHE_NAME=linuxpexbuild.py36
- BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux
- BOOTSTRAP_ARGS='-b'
@@ -381,41 +389,128 @@ cargo_audit: &cargo_audit
# Build wheels
# -------------------------------------------------------------------------

# N.B. With Python 2, we must build pantsbuild.pants with both UCS2 and UCS4 to provide full
# compatibility for end users. This is because we constrain our ABI due to the native engine.
# See https://www.python.org/dev/peps/pep-0513/#ucs-2-vs-ucs-4-builds. Note this distinction is
# not necessary with Python 3.3+ due to flexible storage of Unicode strings (https://www.python.org/dev/peps/pep-0393/).
#
# We treat both Linux UCS4 and OSX UCS2 normally, as these are the defaults for those environments.
# The Linux UCS2 and OSX UCS4 shards, however, must rebuild Python with
# `PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs{2,4}` set, along with bootstrapping Pants again rather
# than pulling the PEX from AWS.

base_build_wheels: &base_build_wheels
stage: *test
env:
- &base_build_wheels_env RUN_PANTS_FROM_PEX=1 PREPARE_DEPLOY=1
- &base_build_wheels_env PREPARE_DEPLOY=1

linux_build_wheels: &linux_build_wheels
py27_linux_build_wheels_no_ucs: &py27_linux_build_wheels_no_ucs
# Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize
# compatibility. This is a Py2.7 shard, so it is not subject to #6985.
<<: *travis_docker_image
<<: *py27_linux_test_config
<<: *base_build_wheels
name: "Build Linux wheels (No PEX)"

py27_linux_build_wheels_ucs2: &py27_linux_build_wheels_ucs2
<<: *py27_linux_config
<<: *py27_linux_build_wheels_no_ucs
<<: *native_engine_cache_config
name: "Build wheels - Linux and cp27m (UCS2)"
env:
- *base_build_wheels_env
- docker_image_name=travis_ci_py27_ucs2
- CACHE_NAME=linuxwheelsbuild.ucs2
script:
- >
docker build
--rm -t ${docker_image_name}
--build-arg "TRAVIS_USER=$(id -un)"
--build-arg "TRAVIS_UID=$(id -u)"
--build-arg "TRAVIS_GROUP=$(id -gn)"
--build-arg "TRAVIS_GID=$(id -g)"
build-support/docker/${docker_image_name}/
- docker run --rm -t
-v "${HOME}:/travis/home"
-v "${TRAVIS_BUILD_DIR}:/travis/workdir"
${docker_image_name}:latest
sh -c "./build-support/bin/ci.sh -2b
&& ./build-support/bin/check_pants_pex_abi.py cp27m
&& RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n"

py27_linux_build_wheels_ucs4: &py27_linux_build_wheels_ucs4
<<: *py27_linux_build_wheels_no_ucs
<<: *py27_linux_test_config
# `py27_linux_test_config` overrides the stage set by `base_build_wheels`, so we re-override it.
stage: *test
name: "Build wheels - Linux and cp27mu (UCS4)"
env:
- *py27_linux_test_config_env
- *base_build_wheels_env
- CACHE_NAME=linuxwheelsbuild
- docker_image_name=travis_ci
- CACHE_NAME=linuxwheelsbuild.ucs4
script:
- *travis_docker_image_launch
- >
docker build
--rm -t ${docker_image_name}
--build-arg "TRAVIS_USER=$(id -un)"
--build-arg "TRAVIS_UID=$(id -u)"
--build-arg "TRAVIS_GROUP=$(id -gn)"
--build-arg "TRAVIS_GID=$(id -g)"
build-support/docker/${docker_image_name}/
- docker run --rm -t
-v "${HOME}:/travis/home"
-v "${TRAVIS_BUILD_DIR}:/travis/workdir"
travis_ci:latest
sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n"
${docker_image_name}:latest
sh -c "./build-support/bin/check_pants_pex_abi.py cp27mu
&& RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n"

osx_build_wheels: &osx_build_wheels
<<: *py27_osx_test_config
py27_osx_build_wheels_no_ucs: &py27_osx_build_wheels_no_ucs
<<: *base_build_wheels
name: "Build OSX wheels (No PEX)"
osx_image: xcode8

py27_osx_build_wheels_ucs2: &py27_osx_build_wheels_ucs2
<<: *py27_osx_test_config
<<: *py27_osx_build_wheels_no_ucs
name: "Build wheels - OSX and cp27m (UCS2)"
env:
- *py27_osx_test_config_env
- *base_build_wheels_env
- CACHE_NAME=osxwheelsbuild
- CACHE_NAME=osxwheelsbuild.ucs2
script:
- ./build-support/bin/check_pants_pex_abi.py cp27m
- RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n

py27_osx_build_wheels_ucs4: &py27_osx_build_wheels_ucs4
<<: *py27_osx_config
<<: *py27_osx_build_wheels_no_ucs
<<: *native_engine_cache_config
name: "Build wheels - OSX and cp27mu (UCS4)"
addons:
brew:
packages:
- openssl
env:
- *base_build_wheels_env
- CACHE_NAME=osxwheelsbuild.ucs4
- >
PATH="/usr/local/opt/openssl/bin:$PATH"
LDFLAGS="-L/usr/local/opt/openssl/lib"
CPPFLAGS="-I/usr/local/opt/openssl/include"
PYENV_ROOT="${HOME}/.pyenv"
PATH="${PYENV_ROOT}/shims:${PATH}"
- PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs4
# We set $PY to ensure the UCS4 interpreter is used when bootstrapping the PEX.
- PY=${PYENV_ROOT}/shims/python2.7
before_install:
- curl -L https://github.com/stedolan/jq/releases/download/jq-1.5/jq-osx-amd64 -o /usr/local/bin/jq
- chmod 755 /usr/local/bin/jq
- ./build-support/bin/install_aws_cli_for_ci.sh
- git clone https://github.com/pyenv/pyenv ${PYENV_ROOT}
- ${PYENV_ROOT}/bin/pyenv install 2.7.13
- ${PYENV_ROOT}/bin/pyenv global 2.7.13
script:
- ./build-support/bin/release.sh -n
- ./build-support/bin/ci.sh -2b
- ./build-support/bin/check_pants_pex_abi.py cp27mu
- RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n

# -------------------------------------------------------------------------
# Rust tests
@@ -607,7 +702,6 @@ matrix:
include:
- <<: *py27_linux_build_engine
- <<: *py36_linux_build_engine

- <<: *py27_osx_build_engine
- <<: *py36_osx_build_engine

@@ -617,9 +711,6 @@ matrix:
- <<: *linux_rust_clippy
- <<: *cargo_audit

- <<: *linux_build_wheels
- <<: *osx_build_wheels

- <<: *py27_linux_test_config
name: "Unit tests for pants and pants-plugins (Py2.7 PEX)"
stage: *test
@@ -637,6 +728,11 @@ matrix:
script:
- ./build-support/bin/travis-ci.sh -lp

- <<: *py27_linux_build_wheels_ucs2
- <<: *py27_linux_build_wheels_ucs4
- <<: *py27_osx_build_wheels_ucs2
- <<: *py27_osx_build_wheels_ucs4

- <<: *py36_linux_test_config
name: "Integration tests for pants - shard 0 (Py3.6 PEX)"
env:
@@ -0,0 +1,70 @@
#!/usr/bin/env python2.7
# coding=utf-8
# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

# Check that the ./pants.pex was built using the passed abi specification.

from __future__ import absolute_import, division, print_function, unicode_literals

import argparse
import json
import os.path
import zipfile


RED = "\033[31m"
BLUE = "\033[34m"
RESET = "\033[0m"


def main():
if not os.path.isfile("pants.pex"):
die("pants.pex not found! Ensure you are in the repository root, then run " \
"'./build-support/bin/ci.sh -b' to bootstrap pants.pex with Python 3 or " \
"'./build-support/bin/ci.sh -2b' to bootstrap pants.pex with Python 2.")
expected_abi = create_parser().parse_args().abi
with zipfile.ZipFile("pants.pex", "r") as pex:
with pex.open("PEX-INFO", "r") as pex_info:
pex_info_content = str(pex_info.readline())
parsed_abis = {
parse_abi_from_filename(filename)
for filename in json.loads(pex_info_content)["distributions"].keys()
if parse_abi_from_filename(filename) != "none"
}
if len(parsed_abis) < 1:
die("No abi tag found. Expected: {}.".format(expected_abi))
elif len(parsed_abis) > 1:
die("Multiple abi tags found. Expected: {}, found: {}.".format(expected_abi, parsed_abis))
found_abi = list(parsed_abis)[0]
if found_abi != expected_abi:
die("pants.pex was built with the incorrect ABI. Expected: {}, found: {}.".format(expected_abi, found_abi))
success("Success. As expected, pants.pex was built with the ABI {}.".format(expected_abi))


def create_parser():
parser = argparse.ArgumentParser(
description="Check that ./pants.pex was built using the passed abi specification."
)
parser.add_argument("abi", help="The expected abi, e.g. `cp27m` or `abi3`")
return parser


def parse_abi_from_filename(filename):
"""This parses out the abi from a wheel filename.
For example, `configparser-3.5.0-py2-abi3-any.whl` would return `abi3`.
See https://www.python.org/dev/peps/pep-0425/#use for how wheel filenames are defined."""
return filename.split("-")[-2]


def success(message):
print("{}{}{}".format(BLUE, message, RESET))


def die(message):
raise SystemExit("{}{}{}".format(RED, message, RESET))


if __name__ == "__main__":
main()
@@ -7,9 +7,22 @@ set -e
ROOT=$(cd $(dirname "${BASH_SOURCE[0]}") && cd "$(git rev-parse --show-toplevel)" && pwd)
source ${ROOT}/build-support/common.sh

PY=$(which python2.7 || exit 0)
[[ -n "${PY}" ]] || die "You must have python2.7 installed and on the path to release."
export PY
# Set the Python interpreter to be used for the virtualenv. Note we allow the user to
# predefine this value so that they may point to a specific interpreter, e.g. 2.7.13 vs. 2.7.15.
export PY="${PY:-python2.7}"
if ! which "${PY}" >/dev/null; then
die "Python interpreter ${PY} not discoverable on your PATH."
fi
py_major_minor=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))')
if [[ "${py_major_minor}" != "2.7" ]]; then
die "Invalid interpreter. The release script requires python2.7, and you are using python${py_major_minor}."
fi

# Also set PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS. We set this to the exact Python version
# to resolve any potential ambiguity when multiple Python interpreters are discoverable, such as
# Python 2.7.13 vs. 2.7.15.
py_major_minor_patch=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:3])))')
export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="${PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS:-['CPython==${py_major_minor_patch}']}"

function run_local_pants() {
${ROOT}/pants "$@"
@@ -0,0 +1,49 @@
# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

# NB: this file duplicates travis_ci/Dockerfile, except it installs Python 2.7 with UCS2.
# We do not include this change directly in centos6/Dockerfile nor in travis_ci/Dockerfile
# because we only want access to this Python interpreter in the Build Wheels Linux UCS2 shard,
# so it is not helpful to other shards. Rather, it would make those shards more finicky to deal
# with by resulting in two Python 2.7 installs: system vs. pyenv.

# Use our custom Centos6 image for binary compatibility with old linux distros.
FROM pantsbuild/centos6:latest

# Note we use 2.7.15, rather than 2.7.13, as the centos6 image already comes with 2.7.13
# installed, which uses UCS4 instead of UCS2. This allows us to disambiguate which Python 2
# interpreter to use when `ci.sh` and `release.sh` set the interpreter constraints for
# Pants, and thus for the built ./pants.pex. We set $PY to the exact Python 2.7 version we want
# to ensure the PEX is bootstrapped with UCS 2.
ARG PYTHON_2_VERSION=2.7.15
# TODO(7064): remove this yum install line once we update the base Centos6 image to include this dependency.
RUN yum install sqlite-devel -y
ENV PYENV_ROOT /pyenv-docker-build
RUN mkdir ${PYENV_ROOT}
RUN git clone https://github.com/pyenv/pyenv ${PYENV_ROOT}
ENV PYTHON_CONFIGURE_OPTS --enable-unicode=ucs2
RUN /usr/bin/scl enable devtoolset-7 -- bash -c '\
${PYENV_ROOT}/bin/pyenv install ${PYTHON_2_VERSION} \
&& ${PYENV_ROOT}/bin/pyenv global ${PYTHON_2_VERSION}'
ENV PATH "${PYENV_ROOT}/shims:${PATH}"
ENV PY "${PYENV_ROOT}/shims/python2.7"
ENV PEX_PYTHON_PATH "${PYENV_ROOT}/shims/python2.7"

# Setup mount points for the travis ci user & workdir.
VOLUME /travis/home
VOLUME /travis/workdir

# Setup a non-root user to execute the build under (avoids problems with npm install).
ARG TRAVIS_USER=travis_ci
ARG TRAVIS_UID=1000
ARG TRAVIS_GROUP=root
ARG TRAVIS_GID=0

RUN groupadd --gid ${TRAVIS_GID} ${TRAVIS_GROUP} || true
RUN useradd -d /travis/home -g ${TRAVIS_GROUP} --uid ${TRAVIS_UID} ${TRAVIS_USER}
USER ${TRAVIS_USER}:${TRAVIS_GROUP}

# Our newly created user is unlikely to have a sane environment: set a locale at least.
ENV LC_ALL="en_US.UTF-8"

WORKDIR /travis/workdir
@@ -0,0 +1,5 @@
PATH="/usr/local/opt/openssl/bin:$PATH"
LDFLAGS="-L/usr/local/opt/openssl/lib"
CPPFLAGS="-I/usr/local/opt/openssl/include"
PYENV_ROOT="${HOME}/.pyenv"
PATH="${PYENV_ROOT}/shims:${PATH}"

0 comments on commit 7b2225e

Please sign in to comment.
You can’t perform that action at this time.