Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip doesn't handle PEP600 compatible manylinux wheels when --platform is specified. #10760

Open
1 task done
Clockwork-Muse opened this issue Jan 4, 2022 · 8 comments · May be fixed by #10894
Open
1 task done

pip doesn't handle PEP600 compatible manylinux wheels when --platform is specified. #10760

Clockwork-Muse opened this issue Jan 4, 2022 · 8 comments · May be fixed by #10894
Labels
state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior

Comments

@Clockwork-Muse
Copy link

Description

When pip is asked to install or download a manylinux wheel and the --platform option is specified, it fails to find either tag aliases, or to download an "earlier" manylinux tag, if there was no exact match.

The manylinux project defines the following legacy aliases in PEP 600:

  • manylinux1_x86_64 is now an alias for manylinux_2_5_x86_64
  • manylinux1_i686 is now an alias for manylinux_2_5_i686
  • manylinux2010_x86_64 is now an alias for manylinux_2_12_x86_64
  • manylinux2010_i686 is now an alias for manylinux_2_12_i686
  • manylinux2014_x86_64 is now an alias for manylinux_2_17_x86_64
  • manylinux2014_i686 is now an alias for manylinux_2_17_i686
  • manylinux2014_aarch64 is now an alias for manylinux_2_17_aarch64
  • manylinux2014_armv7l is now an alias for manylinux_2_17_armv7l
  • manylinux2014_ppc64 is now an alias for manylinux_2_17_ppc64
  • manylinux2014_ppc64le is now an alias for manylinux_2_17_ppc64le
  • manylinux2014_s390x is now an alias for manylinux_2_17_s390x

... these tags should be equivalent to each other.

In addition, manylinux tags are supposed to be forward compatible - if a particular platform is specified, it should find the "latest" tag, if no exact match was found.

Expected behavior

pip downloads a compatible manylinux wheel.

pip version

21.3.1

Python version

3.8.10

OS

Ubuntu 20.04

How to Reproduce

$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux2010_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux2014_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_12_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_17_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_24_x86_64 numpy==1.19.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_17_x86_64 numpy==1.22.0
$ python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_24_x86_64 numpy==1.22.0

(The problem affects any wheel on pypi, numpy was just chosen as a well known package)

The first five commands should find the same manylinux2010 wheel, but only the first two commands do so.
The last two commands should find the same manylinux_2_17 wheel, but only the first command does so.

Output

Collecting numpy==1.19.0
  Downloading numpy-1.19.0-cp38-cp38-manylinux2010_x86_64.whl (14.6 MB)
     |████████████████████████████████| 14.6 MB 4.5 MB/s 
  Saved /tmp/d/numpy-1.19.0-cp38-cp38-manylinux2010_x86_64.whl
Successfully downloaded numpy
Collecting numpy==1.19.0
  File was already downloaded /tmp/d/numpy-1.19.0-cp38-cp38-manylinux2010_x86_64.whl
Successfully downloaded numpy
ERROR: Could not find a version that satisfies the requirement numpy==1.19.0 (from versions: 1.20.3, 1.21.0rc1, 1.21.0rc2, 1.21.0, 1.21.1, 1.21.2, 1.21.3, 1.21.4, 1.21.5)
ERROR: No matching distribution found for numpy==1.19.0
ERROR: Could not find a version that satisfies the requirement numpy==1.19.0 (from versions: 1.22.0rc1, 1.22.0rc2, 1.22.0rc3, 1.22.0)
ERROR: No matching distribution found for numpy==1.19.0
ERROR: Could not find a version that satisfies the requirement numpy==1.19.0 (from versions: none)
ERROR: No matching distribution found for numpy==1.19.0
Collecting numpy==1.22.0
  Downloading numpy-1.22.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
     |████████████████████████████████| 16.8 MB 166 kB/s 
  Saved /tmp/d/numpy-1.22.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Successfully downloaded numpy
ERROR: Could not find a version that satisfies the requirement numpy==1.22.0 (from versions: none)
ERROR: No matching distribution found for numpy==1.22.0

Code of Conduct

@Clockwork-Muse Clockwork-Muse added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jan 4, 2022
@Clockwork-Muse
Copy link
Author

Side note: the command is actually being run as part of a deployment tool, pulling from a requirements file, but tricks like downloading the correct packages and using --find-links or explicitly listing the local path in the requirements file still fail with the same error.

@tvalentyn
Copy link

I was looking at PEP-600 and also concur that current behavior does not match my understanding of PEP-600.
My usecase: I will be running a package on a Linux platform that uses glibc 2.31. I'd like to download a wheel that will be compatible for this platform, so I'd want to run

python3 -m pip download -d /tmp/d --only-binary :all: --platform manylinux_2_31_x86_64 numpy==1.22.0

and expect that the latest compatible wheel is fetched.

@tvalentyn
Copy link

cc: @pfmoore

@pradyunsg pradyunsg removed the S: needs triage Issues/PRs that need to be triaged label Jan 15, 2022
@pradyunsg
Copy link
Member

If somone wants to see this addressed, we'd be happy to accept a PR for this (subject to the regular code review process).

The relevant file for this would be https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/compatibility_tags.py, and I imagine that test improvements would be appreciated.

@vanschelven
Copy link
Contributor

My usecase: I will be running a package on a Linux platform that uses glibc 2.31. I'd like to download a wheel that will be compatible for this platform, so I'd want to run [..] and expect that the latest compatible wheel is fetched.

A workaround is to specify all platforms that the target platform supports. Figuring out that list can be achieved using some variant of the below on the target-platform (and ignoring the message about not automatically parsing):

python -m pip debug --verbose | grep cp39-cp39 | awk -F- '{print $3}'

@cdjameson
Copy link

Folks, PEP-600 was accepted almost 3 years ago. Why does pip download not support this functionality yet? Looks like there's been some recent movement with PR #10894 but I do not understand the feet-dragging on this. As always I feel like the bad guy for wanting basic functionality out of pip download, but I can't imagine I'm the only one that wants to download packages for use in an offline foreign environment.

The workaround of listing each platform tag separately (2.5 through 2.28 in my case) is really inconvenient. Maybe if I were really fancy with my bash array skills it wouldn't be, but nothing immediate came to mind for passing all those options.

@ssbarnea
Copy link
Contributor

Sorry to post on an old bug, but I find myself in the impossible attempt to download wheels for a specific linux platform but different architecture and apparently this bug prevents me from composing a download command that would work.

python -m pip debug --verbose | grep many|grep py310
  py310-none-manylinux_2_38_aarch64
  py310-none-manylinux_2_37_aarch64
  ...
  py310-none-manylinux_2_17_aarch64
  py310-none-manylinux2014_aarch64

For latest version of pyyaml the only command that will succeed is pip download -vv -d dist/linux/x86_64 --platform=manylinux_2_17_x86_64 --only-binary=:all: --python-version=310 pyyaml==6.0.1

Unless I fully mention the manylinux_2_17_x86_64, I will not be able to download "whatever blend of manylinux for x86_64 is available.

The grep command above makes me believe that pip is trying ~20 names until it finds that one to be acceptable. Still, I see no way I can build the same fallback list when attempting to perform to call download.

To give some insights, I am trying to download full list of dependencies for two different platforms (arm and intel) while avoiding to run under qemu as that makes the process 10x slower, this being the preparatory task for building a dual-arch container image.

Basically I have the impression that --platform argument is so restrictive that it makes itself unpractical to use as it requires a perfect match.

@tim-stephenson
Copy link

#12466 It feels tangentially related (but a separate issue with pip download --platform)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants