Skip to content
Permalink
Browse files

Squashed commit of the following:

commit 3bca020
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 25 20:31:14 2019 -0700

    Add global interpreter constraints to Python binary creation

commit 887a8ef
Author: Eric Arellano <ericarellano@me.com>
Date:   Fri Feb 22 21:05:50 2019 -0700

    Constrain ci.sh to the exact Python interpreter version

    Earlier we allowed the patch version to float. We discovered in pantsbuild#7235 with the CI run https://travis-ci.org/pantsbuild/pants/jobs/497208431#L891 that PEX was building with 2.7.10 but running with 2.7.13.

    The fix will require having Pants pass interpreter constraints to Pex. Even with that change though, the CI shard would still have the issue because the constraint would be floating.

    Now, we have the constraint be exact.
  • Loading branch information...
Eric-Arellano committed Feb 26, 2019
1 parent 78a1aa9 commit 48ef4dd2342a1dec2bccb4a561b727fc9b8eab7c
Showing with 21 additions and 14 deletions.
  1. +18 −13 build-support/bin/ci.sh
  2. +3 −1 src/python/pants/backend/python/tasks/python_binary_create.py
@@ -104,26 +104,31 @@ esac
# We're running against a Pants clone.
export PANTS_DEV=1

# Note that we set PY, and when running with Python 3, also set PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS.
# This would usually not be necessary when developing locally, because the `./pants` and `./pants3`
# scripts set these constraints for us already. However, we must set the values here because in non-bootstrap shards
# we run CI using `./pants.pex` instead of the scripts `./pants` and `./pants3`, so those scripts cannot set
# the relevant environment variables. Without setting these environment variables, the Python 3 shards will try to
# execute subprocesses using Python 2, which results in the _Py_Dealloc error (#6985), and shards that do not
# pull down `./pants.pex` but still use a virtualenv (such as Rust Tests) will fail to execute.
# Determine the Python version to use for bootstrapping pants.pex. This would usually not be
# necessary to set when developing locally, because the `./pants` and `./pants3` scripts set
# these constraints for us already. However, we must set the values here because in
# non-bootstrap shards we run CI using `./pants.pex` instead of the scripts `./pants`
# and `./pants3`, so those scripts cannot set the relevant environment variables.
if [[ "${python_two:-false}" == "false" ]]; then
py_version_number="3.6"
py_major_minor="3.6"
bootstrap_pants_script="./pants3"
export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="['CPython==${py_version_number}.*']"
else
py_version_number="2.7"
py_major_minor="2.7"
bootstrap_pants_script="./pants"
fi
export PY="python${py_version_number}"
banner "Using Python ${py_version_number} to execute spawned subprocesses (e.g. tests)"
export PY="python${py_major_minor}"

# Also set PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS. We set this to the exact Python version
# to resolve any potential ambiguity when multiple Python interpreters are discoverable, such as
# Python 2.7.10 vs. 2.7.13. When running with Python 3, we must also set this constraint to ensure
# all spawned subprocesses use Python 3 rather than the default of Python 2. This is in part
# necessary to avoid the _Py_Dealloc error (#6985).
py_major_minor_patch=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:3])))')
export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="${PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS:-['CPython==${py_major_minor_patch}']}"
banner "Setting interpreter constraints to ${PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS}"

if [[ "${run_bootstrap:-false}" == "true" ]]; then
start_travis_section "Bootstrap" "Bootstrapping pants as a Python ${py_version_number} PEX"
start_travis_section "Bootstrap" "Bootstrapping pants as a Python ${py_major_minor_patch} PEX"
(
if [[ "${run_bootstrap_clean:-false}" == "true" ]]; then
./build-support/python/clean.sh || die "Failed to clean before bootstrapping pants."
@@ -15,6 +15,7 @@
has_python_sources, has_resources,
is_python_target)
from pants.backend.python.subsystems.python_native_code import PythonNativeCode
from pants.backend.python.subsystems.python_setup import PythonSetup
from pants.backend.python.targets.python_binary import PythonBinary
from pants.backend.python.targets.python_requirement_library import PythonRequirementLibrary
from pants.base.build_environment import get_buildroot
@@ -140,8 +141,9 @@ def _create_binary(self, binary_tgt, results_dir):
if is_python_target(tgt):
constraint_tgts.append(tgt)

# Add target's interpreter compatibility constraints to pex info.
# Add global and target-level interpreter compatibility constraints to pex info.
pex_builder.add_interpreter_constraints_from(constraint_tgts)
pex_builder.add_interpreter_constraints(PythonSetup.global_instance().interpreter_constraints)

# Dump everything into the builder's chroot.
for tgt in source_tgts:

0 comments on commit 48ef4dd

Please sign in to comment.
You can’t perform that action at this time.