Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip download --platform --python-version does not work for some packages #6121

Open
chadrik opened this issue Jan 8, 2019 · 8 comments

Comments

@chadrik
Copy link

commented Jan 8, 2019

Environment

  • pip version: 18.0
  • Python version: 2.7
  • OS: centos 7

Description

When using pip to download wheels for it does not find them despite the fact that there are obvious matches available on PyPI

How to Reproduce

I tried all of the following commands <and many other permutations) to try to make this work

pip download --python-version 36 --abi cp36 --platform linux_x86_64  "hiredis==0.3.1" --only-binary=:all: --dest ./wheelhouse -v
pip download --python-version 36 --abi cp36 --platform manylinux1_x86_64  "hiredis==0.3.1" --only-binary=:all: --dest ./wheelhouse -v
pip download --python-version 36 --abi cp36 --platform linux_x86_64  --implementation py "hiredis==0.3.1" --only-binary=:all: --dest ./wheelhouse -v
...
    Skipping link https://files.pythonhosted.org/packages/0d/d1/f0346030e4e5fbd931c7d81acdb8c81ea57733f6d87bc15f3dc28a51eb30/hiredis-0.3.1-cp36-cp36m-manylinux1_x86_64.whl#sha256=736a30992ede64b79989869e1a068fec93405dac71d87a316f22ee72e75cb6a5 (from https://pypi.org/simple/hiredis/) (requires-python:>=2.6, !=3.0.*, !=3.1.*); it is not compatible with this Python
...
No matching distribution found for hiredis==0.3.1

The wheel it skips looks perfectly valid to me: hiredis-0.3.1-cp36-cp36m-manylinux1_x86_64.whl . I assume linux_x86_64 matches manylinux1_x86_64, and even if it doesn't, I tried specifiying manylinux1_x86_64 just to be safe.

The same thing happens when trying to download wheels for python 2.7.

The documentation on how these work together is pretty sorely lacking (

  • what is the valid list of platforms and what is the logic for how they match?
  • the provided examples about macosx use the wrong specifier, AFAICT, using a dash instead of an underscore.
  • the pep that is referred to is far too long to grok and it also seems out of date
  • the verbose output about the skipped wheels above provides no information about how pip came to the conclusion that the wheel "is not compatible with this Python". It would be nice if the note provided some more info about why the wheel is not considered compatible.

Along with the other issue I posted #5369 it feels like the intersection of what is required to be useful and what actually works is frustratingly small.

@noonat

This comment has been minimized.

Copy link

commented Jan 28, 2019

I would just like to mention that I am also running into this issue on CentOS while trying to download MacOS wheels. This issue appears to also be reproducible for both Ubuntu and CentOS via Docker. For Ubuntu:

# create a docker container
$ docker run -it ubuntu bash
# install pip and try to download a macos package
$ apt-get update
$ apt-get install -y python3 python3-virtualenv
$ python3.6 -m virtualenv --python=python3.6 env
$ ./env/bin/python3.6 -m pip install -U pip
$ ./env/bin/python3.6 -m pip download --only-binary :all: --platform macosx_10_10_intel --python-version 27 --abi cp27 -v numpy==1.16.0

And the snipped output from pip:

...
    Skipping link https://files.pythonhosted.org/packages/e4/97/167eb80dadcf2905b58d66ada6c128d3ec5e8595beb02457b881e7399be3/numpy-1.16.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#sha256=a80ecac5664f420556a725a5646f2d1c60a7c0489d68a38b5056393e949e27ac (from https://pypi.org/simple/numpy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*); it is not compatible with this Python
...
  Could not find a version that satisfies the requirement numpy==1.16.0 (from versions: 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.1, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.1, 1.10.2, 1.10.4)
...
Removed build tracker '/tmp/pip-req-tracker-0ljej8rn'
No matching distribution found for numpy==1.16.0
Exception information:
Traceback (most recent call last):
  File "/env/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/env/lib/python3.6/site-packages/pip/_internal/commands/download.py", line 164, in run
    resolver.resolve(requirement_set)
  File "/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 269, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/env/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 196, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/env/lib/python3.6/site-packages/pip/_internal/index.py", line 688, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for numpy==1.16.0

These steps on CentOS have the same result:

# create a docker container
$ docker run -it centos bash

# install pip and try to download a macos package
$ yum install -y centos-release-scl
$ yum install -y rh-python36
$ scl enable rh-python36 bash
$ python3.6 -m virtualenv env
$ ./env/bin/python3.6 -m pip install -U pip
$ ./env/bin/python3.6 -m pip download --only-binary :all: --platform macosx_10_10_intel --python-version 27 --abi cp27 -v numpy==1.16.0

Note that it still seems to exhibit this behavior even if you are actually running under CPython 2.7 (but on Linux instead of MacOS). The packages download without issue when running the same command on MacOS.

@AbdealiJK

This comment has been minimized.

Copy link

commented Apr 24, 2019

Was trying pip download and had some very similar issues. The documentation was not super useful, but after some trial and error, the command that got it working for me for hiredis is:

pip download \
  --python-version 36 --abi cp36m --platform manylinux1_x86_64  --implementation cp \
  "hiredis==0.3.1" --only-binary=:all: --dest /tmp/deps

Observations:

  • The --implementation py doesnt look right as it is CPython specific and there is no implementation agnostic wheel available.
  • The api in the wheel says cp36m, while the download command being tried originally was cp36

Similarly, for the numpy example, changing the abi to cp27m works for me:

pip download \
  --python-version 27 --abi cp27m --platform macosx_10_10_intel \
  "numpy==1.16.0" --only-binary=:all: --dest /tmp/deps/pypi

Pretty much got this via trial and error ... Better documentation would be really helpful

@cjerdonek

This comment has been minimized.

Copy link
Member

commented May 26, 2019

FYI, I just posted PR #6540 to address this portion of the original comment:

  • the verbose output about the skipped wheels above provides no information about how pip came to the conclusion that the wheel "is not compatible with this Python". It would be nice if the note provided some more info about why the wheel is not considered compatible.
@cjerdonek

This comment has been minimized.

Copy link
Member

commented May 26, 2019

It would be nice if the note provided some more info about why the wheel is not considered compatible.

I was thinking more about this because #6540 only provides half of the information (namely the wheel's file tags). Currently, I don't think there's any way for the user to know the other half of the information -- namely what tags the current Python version matches (or the requested version in the case of options like --python-version being passed). Since this list can be very long (e.g. ~400 on my system), this info can't simply be logged.

I was thinking then that a new command called something like pip tags could be useful. It could start out simply by printing out a list of tags for the current Python in sorted order, and the command could accept the same environment options like --platform and --python-version that pip install accepts. This would give people a way to see what tags their options are resulting in. Looking at the get_supported() function inside pip's pep425tags.py, I don't think there's any other way for the user to get this information, since you can really only get it empirically by running the code. A command like this should be relatively simple because it would just be a matter of calling get_supported() and printing out the resulting list.

Maybe later the command could accept arguments for things like evaluating certain wheel filenames against the current Python version, with more verbose logging, etc. The command could be thought of as a helper for working with PEP 425 compatibility tags as implemented by pip: https://www.python.org/dev/peps/pep-0425/#details

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

FYI, yesterday I posted PR #6638 that will help with this. It implements an initial pip debug command that displays, among other things, the list of compatible pep 425 tags. It also accepts the same --python-version, --platform, etc. options that pip download and install support so that one would have a way to experiment and see what tags result in those cases, too.

So combined with PR #6540 (already merged), this would give you a better pathway to troubleshoot issues like the ones in this issue. Namely, with PR #6540 merged, the logs will tell you for each wheel what tags the wheel generates. And PR #6638 would give you an interactive way to see whether you can generate any of those tags (and locally, without having to do any actual installs, etc).

@jacobscgc

This comment has been minimized.

Copy link

commented Aug 13, 2019

I have a similar issue but more related to that it does not recognize --python-version 36 as being python >3.5. This leads pip to download an older version, for example when trying to download numpy:

pip download -v -v -v --no-cache-dir --python-version 36 --abi cp36m --platform manylinux1_x86_64 --only-binary=:all: --no-deps --exists-action i numpy

I get a whole list of output, stating that version 1.16.4 is the newest version available for this version of python, despite the fact that version 1.17.0 is available for python 3.6.

Using version 1.16.4 (newest of versions: 1.11.3, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 1.13.3, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4)

It does accept 1.16.4 because the versions accepted are all != tests:

Found link https://files.pythonhosted.org/packages/87/2d/e4656149cbadd3a8a0369fcd1a9c7d61cc7b87b3903b85389c70c989a696/numpy-1.16.4-cp36-cp36m-manylinux1_x86_64.whl#sha256=27e11c7a8ec9d5838bc59f809bfa86efc8a4fd02e58960fa9c49d998e14332d5 (from https://pypi.org/simple/numpy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 1.16.4

However, python version >=3.5 is not accepted when --python-version=36:

The package https://files.pythonhosted.org/packages/19/b9/bda9781f0a74b90ebd2e046fde1196182900bd4a8e1ea503d3ffebc50e7c/numpy-1.17.0-cp36-cp36m-manylinux1_x86_64.whl#sha256=9588c6b4157f493edeb9378788dcd02cb9e6a6aeaa518b511a1c79d06cbd8094 (from https://pypi.org/simple/numpy/) (requires-python:>=3.5) is incompatible with the pythonversion in use. Acceptable python versions are:>=3.5

Which seems like an error to me but maybe I'm mistaken. The same issues occur for pandas (downloads 0.24.2 instead of 0.25.0) and scipy (downloads 1.2.2 instead of 1.3.1).

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

@jacobscgc What version of pip are you using?

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

@jacobscgc Actually, file a new issue for your issue because yours seems different, and then it can be discussed there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.