New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support manylinux1 wheels #281

Closed
rouge8 opened this Issue Jul 7, 2016 · 17 comments

Comments

Projects
None yet
@rouge8
Contributor

rouge8 commented Jul 7, 2016

Pex doesn't seem to support manylinux1 wheels:

$ pex --version && pex --no-build -vvv numpy
pex 1.1.11
pex: crawling link i=0 link=Link('file:///home/andy/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/numpy/') follow_links=False
Could not satisfy all requirements for numpy:
    numpy
$ pip install --only-binary :all: numpy
Collecting numpy
  Downloading numpy-1.11.1-cp27-cp27mu-manylinux1_x86_64.whl (15.3MB)
    100% |████████████████████████████████| 15.3MB 102kB/s
Installing collected packages: numpy
Successfully installed numpy-1.11.1

This was mentioned briefly in #250, but that was closed.

@rouge8

This comment has been minimized.

Show comment
Hide comment
@rouge8

rouge8 Jul 7, 2016

Contributor

PEP 513 has an algorithm for detecting manylinux support, although pip has modified it slightly to deal with weird distributions: pypa/pip#3497, pypa/pip#3590

Contributor

rouge8 commented Jul 7, 2016

PEP 513 has an algorithm for detecting manylinux support, although pip has modified it slightly to deal with weird distributions: pypa/pip#3497, pypa/pip#3590

@kwlzn kwlzn added the enhancement label Jul 12, 2016

@rouge8

This comment has been minimized.

Show comment
Hide comment
@rouge8

rouge8 Jul 13, 2016

Contributor

I've been thinking about tackling this, but I can't decide how/whether to handle manylinux compatibility at buildtime. At runtime, you can just use the PEP513 (or modified pip) algorithm, but that won't work for building a Linux PEX on an OS X host. The best I've thought of is using _manylinux.manylinux1_compatible and only applying that portion of the compatibility check:

To handle potential future incompatibilities, we standardize a mechanism for a Python distributor to signal that a particular Python install definitely is or is not compatible with manylinux1 : this is done by installing a module named _manylinux , and setting its manylinux1_compatible attribute. We do not propose adding any such module to the standard library -- this is merely a well-known name by which distributors and installation tools can rendezvous. However, if a distributor does add this module, they should add it to the standard library rather than to a site-packages/ directory, because the standard library is inherited by virtualenvs (which we want), and site-packages/ in general is not.

Similar to pypa/pip#3689.

Or is it reasonable to assume manylinux1 compatibility when building?

Contributor

rouge8 commented Jul 13, 2016

I've been thinking about tackling this, but I can't decide how/whether to handle manylinux compatibility at buildtime. At runtime, you can just use the PEP513 (or modified pip) algorithm, but that won't work for building a Linux PEX on an OS X host. The best I've thought of is using _manylinux.manylinux1_compatible and only applying that portion of the compatibility check:

To handle potential future incompatibilities, we standardize a mechanism for a Python distributor to signal that a particular Python install definitely is or is not compatible with manylinux1 : this is done by installing a module named _manylinux , and setting its manylinux1_compatible attribute. We do not propose adding any such module to the standard library -- this is merely a well-known name by which distributors and installation tools can rendezvous. However, if a distributor does add this module, they should add it to the standard library rather than to a site-packages/ directory, because the standard library is inherited by virtualenvs (which we want), and site-packages/ in general is not.

Similar to pypa/pip#3689.

Or is it reasonable to assume manylinux1 compatibility when building?

@kwlzn

This comment has been minimized.

Show comment
Hide comment
@kwlzn

kwlzn Jul 13, 2016

Member

how about making it explicitly user controlled at pex creation time - e.g. with a feature flag like --manylinux to enable resolving against manylinux artifacts (when --platform contains the string linux)?

then if the user wants to build a manylinux compatible pex for running on a separate host, they'd just pass this flag - and then separately be gated by the runtime compatibility checking when running the pex on the separate host where the constraint actually matters.

Member

kwlzn commented Jul 13, 2016

how about making it explicitly user controlled at pex creation time - e.g. with a feature flag like --manylinux to enable resolving against manylinux artifacts (when --platform contains the string linux)?

then if the user wants to build a manylinux compatible pex for running on a separate host, they'd just pass this flag - and then separately be gated by the runtime compatibility checking when running the pex on the separate host where the constraint actually matters.

@rouge8

This comment has been minimized.

Show comment
Hide comment
@rouge8

rouge8 Jul 13, 2016

Contributor

Sounds reasonable. --manylinux/--no-manylinux maybe and default to --manylinux to match the behavior of pip.

Contributor

rouge8 commented Jul 13, 2016

Sounds reasonable. --manylinux/--no-manylinux maybe and default to --manylinux to match the behavior of pip.

@ariscn

This comment has been minimized.

Show comment
Hide comment
@ariscn

ariscn Aug 8, 2016

+1, more and more packages are providing only manylinux1 wheels. It would be great if pex could pull those in instead of requiring linux-x86_64 artifacts.

ariscn commented Aug 8, 2016

+1, more and more packages are providing only manylinux1 wheels. It would be great if pex could pull those in instead of requiring linux-x86_64 artifacts.

@bragan

This comment has been minimized.

Show comment
Hide comment
@bragan

bragan Nov 9, 2016

another +1 on this. We have run into this issue with libsass which is only providing manylinux1 so we had to pin the last version... any update on manylinux pex support?

bragan commented Nov 9, 2016

another +1 on this. We have run into this issue with libsass which is only providing manylinux1 so we had to pin the last version... any update on manylinux pex support?

@rouge8

This comment has been minimized.

Show comment
Hide comment
@rouge8

rouge8 Nov 9, 2016

Contributor

Unfortunately I'm not using PEX at my current job and haven't had much time to spend on this. 😞

Contributor

rouge8 commented Nov 9, 2016

Unfortunately I'm not using PEX at my current job and haven't had much time to spend on this. 😞

@sholsapp

This comment has been minimized.

Show comment
Hide comment
@sholsapp

sholsapp Jun 7, 2017

Contributor

+1 if/when you get to it :) This 🔥'd me.

Contributor

sholsapp commented Jun 7, 2017

+1 if/when you get to it :) This 🔥'd me.

@augray

This comment has been minimized.

Show comment
Hide comment
@augray

augray commented Jun 16, 2017

+1

@fvinas

This comment has been minimized.

Show comment
Hide comment
@fvinas

fvinas commented Jul 2, 2017

+1 ;-)

@evanj

This comment has been minimized.

Show comment
Hide comment
@evanj

evanj Dec 7, 2017

I found a disgusting hack that "works" for me. At the very least, when I build a pex using some manylinux1 wheels, it will allow it to be run.

In package.py, in the function platform_iterator(...), I added the following "hack" which treats all Linux distributions as manylinux1 compatible. Strictly speaking this is not true, and Pip has a bunch of rules in pip.pep425tags that Pex should probably be reusing. However, this was enough to get it to work for me:

  def platform_iterator(cls, platform):
    """Iterate over all compatible platform tags of a supplied platform tag.

       :param platform: the platform tag to iterate over
    """
    if cls.is_macosx_platform(platform):
      for plat in cls.iter_compatible_osx_platforms(platform):
        yield plat
    elif platform == 'linux_x86_64':
      yield 'manylinux1_x86_64'
      yield platform
    else:
      yield platform

evanj commented Dec 7, 2017

I found a disgusting hack that "works" for me. At the very least, when I build a pex using some manylinux1 wheels, it will allow it to be run.

In package.py, in the function platform_iterator(...), I added the following "hack" which treats all Linux distributions as manylinux1 compatible. Strictly speaking this is not true, and Pip has a bunch of rules in pip.pep425tags that Pex should probably be reusing. However, this was enough to get it to work for me:

  def platform_iterator(cls, platform):
    """Iterate over all compatible platform tags of a supplied platform tag.

       :param platform: the platform tag to iterate over
    """
    if cls.is_macosx_platform(platform):
      for plat in cls.iter_compatible_osx_platforms(platform):
        yield plat
    elif platform == 'linux_x86_64':
      yield 'manylinux1_x86_64'
      yield platform
    else:
      yield platform
@anthrotype

This comment has been minimized.

Show comment
Hide comment
@anthrotype

anthrotype Mar 6, 2018

What's the status of this? There seems to be two competing PRs #316 and #399, both in an unmergeable state. It would be great if pex also supported manylinux1. Thank you!

anthrotype commented Mar 6, 2018

What's the status of this? There seems to be two competing PRs #316 and #399, both in an unmergeable state. It would be great if pex also supported manylinux1. Thank you!

@kwlzn kwlzn self-assigned this May 15, 2018

@kwlzn

This comment has been minimized.

Show comment
Hide comment
@kwlzn

kwlzn May 15, 2018

Member

the final implementation is now reviewable here: #480

Member

kwlzn commented May 15, 2018

the final implementation is now reviewable here: #480

@kwlzn kwlzn closed this in #316 May 15, 2018

kwlzn added a commit that referenced this issue May 15, 2018

Add manylinux wheel support and fix a few bugs along the way (#316)
Adds support for manylinux1_x86_64 and manylinux1_i686 platforms. I've treated them as platforms, because that's how PEP-513 describes them. This fixes #281.
Fixes a number of bugs related to the handling of wheels that have hyphens in the package names (e.g., cassandra-driver and msgpack-python).
Fixes a couple places where str.replace was being called without storing the result (thereby accomplishing nothing since str is immutable)

@kwlzn kwlzn referenced this issue May 15, 2018

Closed

Release 1.4.0 #481

@kwlzn

This comment has been minimized.

Show comment
Hide comment
@kwlzn

kwlzn May 16, 2018

Member

this is now available as of pex 1.4.0 (on pypi).

pex will now resolve against manylinux dists both at build and runtime (implicitly, by default) when targeting linux platforms, with no special platform targeting required:

[omerta ~]$ pex --version
pex 1.4.0
[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64 msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep manylinux | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:56   .deps/msgpack_python-0.4.7-cp27-cp27m-manylinux1_x86_64.whl/msgpack/__init__.py

if you'd like to disable manylinux resolves (1.3.x and earlier behavior), you can pass --no-manylinux:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64 --no-manylinux msgpack_python -o /tmp/msgpack.pex
Could not satisfy all requirements for msgpack_python:
    msgpack_python

you can also do more granular platform/tag targeting via an extended --platform mode:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64-cp-27-cp27mu msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep manylinux | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:58   .deps/msgpack_python-0.4.7-cp27-cp27mu-manylinux1_x86_64.whl/msgpack/__init__.py

...which still supports multi-platform usage:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64-cp-27-cp27mu --platform=linux-x86_64-cp-27-cp27m msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep "cp27m-" | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:59   .deps/msgpack_python-0.4.7-cp27-cp27m-manylinux1_x86_64.whl/msgpack/__init__.py
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep "cp27mu-" | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:59   .deps/msgpack_python-0.4.7-cp27-cp27mu-manylinux1_x86_64.whl/msgpack/__init__.py
[omerta ~]$ 

please take a moment to kick the tires with pex 1.4.0 and open issues for any bugs or problems that you might discover. enjoy!

Member

kwlzn commented May 16, 2018

this is now available as of pex 1.4.0 (on pypi).

pex will now resolve against manylinux dists both at build and runtime (implicitly, by default) when targeting linux platforms, with no special platform targeting required:

[omerta ~]$ pex --version
pex 1.4.0
[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64 msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep manylinux | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:56   .deps/msgpack_python-0.4.7-cp27-cp27m-manylinux1_x86_64.whl/msgpack/__init__.py

if you'd like to disable manylinux resolves (1.3.x and earlier behavior), you can pass --no-manylinux:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64 --no-manylinux msgpack_python -o /tmp/msgpack.pex
Could not satisfy all requirements for msgpack_python:
    msgpack_python

you can also do more granular platform/tag targeting via an extended --platform mode:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64-cp-27-cp27mu msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep manylinux | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:58   .deps/msgpack_python-0.4.7-cp27-cp27mu-manylinux1_x86_64.whl/msgpack/__init__.py

...which still supports multi-platform usage:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64-cp-27-cp27mu --platform=linux-x86_64-cp-27-cp27m msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep "cp27m-" | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:59   .deps/msgpack_python-0.4.7-cp27-cp27m-manylinux1_x86_64.whl/msgpack/__init__.py
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep "cp27mu-" | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:59   .deps/msgpack_python-0.4.7-cp27-cp27mu-manylinux1_x86_64.whl/msgpack/__init__.py
[omerta ~]$ 

please take a moment to kick the tires with pex 1.4.0 and open issues for any bugs or problems that you might discover. enjoy!

@AlecBenzer

This comment has been minimized.

Show comment
Hide comment
@AlecBenzer

AlecBenzer Aug 11, 2018

This seems to still not work for me. When I download a manylinux wheel, pex can't seem to find it, but if I just rename manylinux1 to linux, it works.

When using the manylinux wheel for psycopg2==2.7.4 named as psycopg2-2.7.4-cp27-cp27mu-manylinux1_x86_64.whl, my build eventually fails with:

Exception message: Package SourcePackage(u'file:///Users/alec/.cache/pants/python_cache/requirements/CPython-2.7.10/psycopg2-2.7.4.tar.gz') is not translateable by ChainedTranslator(Wh
eelTranslator, EggTranslator, SourceTranslator) 

(the mechanics of what exactly is happening here are still a mystery to me, but I'm guessing it's something like: no suitable wheel is found, so it starts trying to build from the source tar.gz, and can't because I'm not on Linux)

but when I then clear my ~/.cache/pants/python_cache/requirements and rename the wheel to psycopg2-2.7.4-cp27-cp27mu-linux_x86_64.whl, the build works.


My pex --version is 1.4.5, but a wrinkle is that I'm using pex via pants, and I'm not actually sure if the version of pex on my $PATH is the same version pants ends up using.

AlecBenzer commented Aug 11, 2018

This seems to still not work for me. When I download a manylinux wheel, pex can't seem to find it, but if I just rename manylinux1 to linux, it works.

When using the manylinux wheel for psycopg2==2.7.4 named as psycopg2-2.7.4-cp27-cp27mu-manylinux1_x86_64.whl, my build eventually fails with:

Exception message: Package SourcePackage(u'file:///Users/alec/.cache/pants/python_cache/requirements/CPython-2.7.10/psycopg2-2.7.4.tar.gz') is not translateable by ChainedTranslator(Wh
eelTranslator, EggTranslator, SourceTranslator) 

(the mechanics of what exactly is happening here are still a mystery to me, but I'm guessing it's something like: no suitable wheel is found, so it starts trying to build from the source tar.gz, and can't because I'm not on Linux)

but when I then clear my ~/.cache/pants/python_cache/requirements and rename the wheel to psycopg2-2.7.4-cp27-cp27mu-linux_x86_64.whl, the build works.


My pex --version is 1.4.5, but a wrinkle is that I'm using pex via pants, and I'm not actually sure if the version of pex on my $PATH is the same version pants ends up using.

@stuhood

This comment has been minimized.

Show comment
Hide comment
@stuhood

stuhood Aug 11, 2018

Member

My pex --version is 1.4.5, but a wrinkle is that I'm using pex via pants, and I'm not actually sure if the version of pex on my $PATH is the same version pants ends up using.

Pants resolves its own version of pex. Can find-in-page for pex here: https://www.pantsbuild.org/notes-master.html ... to see when various versions were incorporated.

Member

stuhood commented Aug 11, 2018

My pex --version is 1.4.5, but a wrinkle is that I'm using pex via pants, and I'm not actually sure if the version of pex on my $PATH is the same version pants ends up using.

Pants resolves its own version of pex. Can find-in-page for pex here: https://www.pantsbuild.org/notes-master.html ... to see when various versions were incorporated.

@AlecBenzer

This comment has been minimized.

Show comment
Hide comment
@AlecBenzer

AlecBenzer Aug 11, 2018

Ah, ok. Couldn't find it from that page directly, but I'm on pants 1.7.0, and judging from

https://github.com/pantsbuild/pants/blob/1.7.x/3rdparty/python/requirements.txt#L17

I guess I'm on pex 1.3.2 still.

AlecBenzer commented Aug 11, 2018

Ah, ok. Couldn't find it from that page directly, but I'm on pants 1.7.0, and judging from

https://github.com/pantsbuild/pants/blob/1.7.x/3rdparty/python/requirements.txt#L17

I guess I'm on pex 1.3.2 still.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment