Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release pants as both a Python 2.7 PEX and a Python 3.6 PEX #7401

Merged
merged 15 commits into from Mar 20, 2019

Conversation

Projects
None yet
3 participants
@Eric-Arellano
Copy link
Contributor

commented Mar 19, 2019

Problem

While wheels are now released with Python 3 support thanks to #7197, this is only one of two ways people consume Pants. The other way is through a Pex released to Github.

We decided to release both a Py27 and Py36 Pex.

We will not be releasing a Py37 Pex, as it is would require Py37 wheel building shards which would require a Centos7 docker image. We decided this is not worth the work for how few people we anticipate would use Py37 with Pex combined.

We will also not be releasing a more flexible Pex that works with either Py27 or Py36, as it is blocked by pantsbuild/pex#654. We also want organizations pinning their Python version, so a flexible release is not particularly useful in this instance.

This implements #6450 (comment).

Solution

Release both pants.${VERSION}.py{27,36}.pex.

This requires changing how we run release.sh -p and adding new CI shards to deploy on both Py2 and Py3.

Result

The next release, we will release both pants.1.15.0.rc1.py27.pex and pants.1.15.0.rc1.py36.pex.

@jsirois

This comment has been minimized.

Copy link
Member

commented Mar 19, 2019

Not sure which is easier. Your proposal is already documented fwiw here: pantsbuild/pex#539

Clearly fixing PEX wrt abi3 is the right long term solution.

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Mar 19, 2019

Not sure which is easier. Your proposal is already documented fwiw here: pantsbuild/pex#539

Clearly fixing PEX wrt abi3 is the right long term solution.

I now see a path forward to getting this short-term fix working. Will pursue it first, as my priority is ensuring we release a pex36 for 1.15.0.

After we get this working, can work on the upstream Pex issue. My goal for 1.16.0 is that we release a fully flexible Pex, and I'll include that issue as part of this work.

Thanks for the suggestions!

Eric-Arellano added some commits Mar 19, 2019

Workaround cryptography resolve issue (pex #539)
First off, we should be marking the expected abi as `cp-36-m`. We cannot mark it as `cp36-abi3`, as this implies every wheel must be abi3 compatible, which is not the case. Instead, we mark as `cp-36-m`, and if a wheel is `abi3` instead of `cp36m`, that's great!

However, Pex currently does not treat abi3 correctly, in that `cp-36-m` will not correctly resolve `cp34-abi3`, even though it is valid. See Pex #539.

This leads to the Pex build failing due to the cryptography wheel, which is `cp34-abi3`. So, we temporarily workaround this by simply building our own `cp36-cp36m` wheel.

@Eric-Arellano Eric-Arellano marked this pull request as ready for review Mar 19, 2019

@Eric-Arellano Eric-Arellano changed the title WIP: Release pants as both a Python 2.7 PEX and a Python 3.6 PEX Release pants as both a Python 2.7 PEX and a Python 3.6 PEX Mar 20, 2019

;;
Linux)
local platforms=("${linux_platform_noabi}")
local platform=("${linux_platform_noabi}")

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Mar 20, 2019

Author Contributor

These were bad renames from #7393. Renaming it to platforms meant the dest and stable_dest had an undefined variable.

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor Author

commented Mar 20, 2019

Reviewers, note we unfortunately cannot verify two parts until this is merged into master:

  1. release.sh -3p won't work until master releases cryptography as a cp36m wheel, which this PR makes that change. This is because we need that new wheel published, and we only publish with branch builds and not PR builds.
  2. The new deploy shards. They only run in master.

I tested this locally by running the similar command release.sh -3q (had to temporarily update the mac platform name to use 10.14 instead of 10.11).

If this doesn't work when merged, we can eagerly revert this / turn off the new deploy shards.

@stuhood stuhood added this to the 1.15.x milestone Mar 20, 2019

@stuhood
Copy link
Member

left a comment

Thanks, looks good.

Let's see if it works... once merged, can you ping me when you've confirmed that it looks ready for a cherry-pick to 1.15.x?

@Eric-Arellano Eric-Arellano merged commit d392a19 into pantsbuild:master Mar 20, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:release-pex36 branch Mar 20, 2019

stuhood added a commit that referenced this pull request Mar 20, 2019

Specify the python version to use in the unstable pex deploy shard. (#…
…7411)

### Problem

As a follow up to #7401: we need to specify an explicit python version to use in pyenv in order to avoid https://gist.github.com/stuhood/8cfae337c99d2bb21ace0305c2c1d736.

### Solution

Specify the pyenv python version.

stuhood added a commit that referenced this pull request Mar 21, 2019

Release pants as both a Python 2.7 PEX and a Python 3.6 PEX (#7401)
### Problem
While wheels are now released with Python 3 support thanks to #7197, this is only one of two ways people consume Pants. The other way is through a Pex released to Github. 

We decided to release both a Py27 and Py36 Pex. 

We will not be releasing a Py37 Pex, as it is would require Py37 wheel building shards which would require a Centos7 docker image. We decided this is not worth the work for how few people we anticipate would use Py37 with Pex combined.

We will also not be releasing a more flexible Pex that works with either Py27 or Py36, as it is blocked by pantsbuild/pex#654. We also want organizations pinning their Python version, so a flexible release is not particularly useful in this instance.

This implements #6450 (comment).

### Solution
Release both `pants.${VERSION}.py{27,36}.pex`.

This requires changing how we run `release.sh -p` and adding new CI shards to deploy on both Py2 and Py3.

### Result
The next release, we will release both `pants.1.15.0.rc1.py27.pex` and  `pants.1.15.0.rc1.py36.pex`.

stuhood added a commit that referenced this pull request Mar 21, 2019

Specify the python version to use in the unstable pex deploy shard. (#…
…7411)

### Problem

As a follow up to #7401: we need to specify an explicit python version to use in pyenv in order to avoid https://gist.github.com/stuhood/8cfae337c99d2bb21ace0305c2c1d736.

### Solution

Specify the pyenv python version.

Eric-Arellano added a commit that referenced this pull request Apr 1, 2019

Fix issues with deploy shards not properly setting Python version (#7471
)

There were several problems with the deploy shards introduced by #7401 and not completely cleaned up by #7411:

- Only setting `pyenv global 3.6.3` for the unstable deploy shard, even though the stable one should have this line too.
- Relying on Travis defaulting to the Trusty image. If they were to change this to Xenial, the shards would fail because the Pyenv versions would not match.
- `base_deploy_unstable_multiplatform_pex` extending itself rather than `base_deploy`.

This is also pre-work for getting Python 3.6 on all shards, so that we can use Python 3 in our `build-support` scripts.

stuhood added a commit that referenced this pull request Apr 1, 2019

Fix issues with deploy shards not properly setting Python version (#7471
)

There were several problems with the deploy shards introduced by #7401 and not completely cleaned up by #7411:

- Only setting `pyenv global 3.6.3` for the unstable deploy shard, even though the stable one should have this line too.
- Relying on Travis defaulting to the Trusty image. If they were to change this to Xenial, the shards would fail because the Pyenv versions would not match.
- `base_deploy_unstable_multiplatform_pex` extending itself rather than `base_deploy`.

This is also pre-work for getting Python 3.6 on all shards, so that we can use Python 3 in our `build-support` scripts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.