pip does not ignore yanked version of package in unqualified installation when it's the only non-prerelease choice

[pganssle]

Environment

• pip version: 20.1
• Python version: 3.8.2
• OS: Arch Linux (using pyenv + virtualenv)

Description

If you have a package named mypkg with the following versions:

• 1.0 (Yanked)
• 2.0rc0

Running pip install mypkg will install 1.0 rather than 2.0rc0.

Expected behavior
I would have expected 2.0rc0 to be installed, since that is what would happen if mypkg only published version 2.0rc0. This seems to be an edge case not super well-clarified in PEP 592, which states:

An installer MUST ignore yanked releases, if the selection constraints can be satisfied with a non-yanked version, and MAY refuse to use a yanked release even if it means that the request cannot be satisfied at all. An implementation SHOULD choose a policy that follows the spirit of the intention above, and that prevents "new" dependencies on yanked releases/files.

I believe 2.0rc0 satisfies the constraints specified if you do not consider "didn't include --pre" to be a constraint, so depending on whether --pre or its absence a "constraint", this may be a violation of the spec. I think the fact that pip install will take release candidates if nothing else matches argues in favor of it not being considered a "constraint".

More clearly, though, it does not fit one of the two suggested implementation strategies:

1. Yanked files are always ignored, unless they are the only file that matches a version specifier that "pins" to an exact version using either == (without any modifiers that make it a range, such as .*) or ===. Matching this version specifier should otherwise be done as per PEP 440 for things like local versions, zero padding, etc.
2. Yanked files are always ignored, unless they are the only file that matches what a lock file (such as Pipfile.lock or poetry.lock) specifies to be installed. In this case, a yanked file SHOULD not be used when creating or updating a lock file from some input file or command.

Regardless of "spec lawyering", from a user experience point of view, I am personally in favor of pip refusing to install yanked versions at all without an exact version pin (option 1). Right now it installs them but with a warning; I think the warning could be turned into an error that lists the yanked versions, and end users who want to work around the "can't find a matching version" can add an exact version pin (though admittedly in some circumstances this can have perverse consequences if a package yanks a version because there's something wrong with it and a consumer pins to the exact version as a workaround, thus preventing an update to a later working, non-yanked version).

How to Reproduce

At the moment the tzdata package has only yanked and release candidate packages on PyPI (and only release candidate packages on Test PyPI). Until a proper release is made of tzdata, you can reproduce it with:

• pip install tzdata: To see the bug
• pip install --index-url https://test.pypi.org/simple/ tzdata: To see pip's behavior when only release candidates are available.

[hroncok]

Also note that if you have:

• version 1 (yanked), otherwise installable on Python 3.7
• version 2 (not yanked), has Requires: Python >=3.8

When you install the package via name (no version specified) from Python 3.7, it installs version 1.

Real life example:

https://pypi.org/project/ferrypick/#history

(__venv37__) [tmp]$ python --version
Python 3.7.7
(__venv37__) [tmp]$ pip install ferrypick
WARNING: The candidate selected for download or install is a yanked version: 'ferrypick' candidate (version 0.1.dev1 at https://files.pythonhosted.org/packages/b5/0e/04a44db1f3ce963f489f8582b92471ff60ef3c14bd90b4a5913e8e80aab1/ferrypick-0.1.dev1-py3-none-any.whl#sha256=110adcd8cf71f35c9926388e7b380103e1f22c2ebcdc8bd9acca16e405193782 (from https://pypi.org/simple/ferrypick/))
Reason for being yanked: This release is entirely broken, and it also doesn't block installation from Python version not supported by the later versions. Hence users of e.g. Python 3.7 would always get this broken version without a reasonable information about ferrypick requiring Python 3.8+.
Collecting ferrypick
  Using cached ferrypick-0.1.dev1-py3-none-any.whl (5.3 kB)
Installing collected packages: ferrypick
Successfully installed ferrypick-0.1.dev1

[pradyunsg]

Definitely looks like a bug to me! PRs welcome to fix this one. :)

[joaomarccos]

Similar problem here.. When using more than one index and when the package candidate has yanked versions.. the pip does not ignore the yanked version even there is a version that matches a version specifier. You can simulate installing pyrsistent>=0.14.0, but before ensure you setup some another repository as proxy to pypi like Nexus.

[Jackenmen]

@pradyunsg do you also consider this part from the issue description a bug?:

More clearly, though, it does not fit one of the two suggested implementation strategies:

1. Yanked files are always ignored, unless they are the only file that matches a version specifier that "pins" to an exact version using either == (without any modifiers that make it a range, such as .*) or ===. Matching this version specifier should otherwise be done as per PEP 440 for things like local versions, zero padding, etc.
2. Yanked files are always ignored, unless they are the only file that matches what a lock file (such as Pipfile.lock or poetry.lock) specifies to be installed. In this case, a yanked file SHOULD not be used when creating or updating a lock file from some input file or command.

Regardless of "spec lawyering", from a user experience point of view, I am personally in favor of pip refusing to install yanked versions at all without an exact version pin (option 1). Right now it installs them but with a warning

As in, would it be accepted if the PR made a change so that pip refuses to install the package if it's installed without version specifier (i.e. pip install -U package-name) per the suggested approach from the PEP 592?

To give a concrete example, I maintain a Red-DiscordBot on PyPI which does not have any non-yanked version supporting Python 3.9 but it does have yanked versions that did not have an upper-bound set. Currently, this allows the user to run python3.9 -m pip install -U Red-DiscordBot and successfully install the package (with a warning from pip about the selected candidate version being yanked) but what I would expect (and potentially want to make a PR for) is for it to refuse to install the package per the "suggested approach" from the PEP.

[pradyunsg]

Yes, exactly that. In this example, Python 3.7 should fail with "nope, not got anything to install".

[dlukyanov]

another case
Dockerfile:

FROM registry.access.redhat.com/ubi8/ubi

RUN yum install -y \
        python38 \
        python38-pip \
        python38-devel \
        gcc-c++

# pdoc3 depends on setuptools-scm 
# current version of setuptools-scm 6.2.0 with label=yanked
# and pdoc3 uses the latest even there are stable versions present
RUN pip3 install pdoc3==0.10.0

ENTRYPOINT /bin/bash

cmd to build and fail:

docker build -t test-pdoc3 -f Dockerfile .

Error - note the setuptools_scm-6.2.0 version:

Collecting pdoc3==0.10.0
  Downloading https://files.pythonhosted.org/packages/95/be/69267c988fb7236cd60c452a4e7fb9a7991729476db490b634a07e7dfcdf/pdoc3-0.10.0.tar.gz (86kB)
      ...
      File "/tmp/pip-install-4trtluv9/pdoc3/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 94, in dump_version
        version_fields = parsed_version.release
    AttributeError: 'Version' object has no attribute 'release'

setuptools-scm has version history:

6.2.0 yanked 
6.1.1 yanked 
6.1.0 yanked 
6.1.0.dev0 pre-release + yanked 
6.0.1 stable
...

and pip takes 6.2.0...

https://pypi.org/project/setuptools-scm/6.2.0/

[pradyunsg]

What version of pip is in that image @dlukyanov?

[RonnyPfannschmidt]

"pip" is not involved in the download of setuptools_scm, thats setuptools/easy_install
please update pip and setuptools, the distribution versions are severely outdated and your build practically worked by accident,

the next release of setuptools_scm will warn better for such situations and suggest the fixes such as also installing python-setuptools_scm and/or other means of preinstalling

[notatallshaw]

@dlukyanov Also as you installed pip from Redhat UBI then Redhat are specifically providing commercial support for the packages they distribute and you should reach out to their support. They are responsible for providing bug fixed versions of the packages they are distributing.

Though I suspect you will find that Redhat only supports installing third party packages via yum, and pip is probably provided for install of internal repositories and probably only ever using the the user flag.

[hroncok]

Red Hat engineer here. If we get a customer report saying that our pip happily installs yanked versions, we will definitively do our best to fix that.

[hroncok]

In fact: https://bugzilla.redhat.com/show_bug.cgi?id=2000135

[notatallshaw]

In fact: https://bugzilla.redhat.com/show_bug.cgi?id=2000135

Thanks, just in case you don't already know a major fix for yanked packages was implemented in pip 20.3.2: #9226 It looks like the pip that the user installs from redhat is 19.3.1?

The reason it became such an issue from 20.3 is that the new Resolver was turned on by default which backtracks on packages until a valid set of compatible requirements are found. Which inherently trades performance for correctness, and exposes users to potentially downloading a lot of package versions which is where the "yanked" issue came up.

[dlukyanov]

here the minimized dockerfile:

FROM registry.access.redhat.com/ubi8/ubi
RUN yum install -y python38 python38-pip
ENTRYPOINT /bin/bash

start docker, and run in console:

# python3 --version
Python 3.8.6

# pip3 install pip --upgrade
Successfully installed pip-21.2.4

# python3 -m pip --version
pip 21.2.4 from /usr/local/lib/python3.8/site-packages/pip (python 3.8)

# pip3 install pdoc3==0.10.0
Collecting pdoc3==0.10.0
  ERROR: Command errored out with exit status 1:
  ...
  File "/tmp/pip-install-lu30xwb5/pdoc3_95c8ac/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 94, in dump_version
        version_fields = parsed_version.release
    AttributeError: 'Version' object has no attribute 'release'

[hroncok]

Does it make any difference if you replace pip3 install with python3 -m pip install? In other words, does python3 -m pip --version give the same output as pip3 --version?

[pfmoore]

/tmp/pip-install-lu30xwb5/pdoc3_95c8ac/.eggs/setuptools_scm-6.2.0-py3.8.egg

Pip wouldn't install setuptools_scm as an egg like that. Looks like it's using easy_install, so maybe an old version of setuptools?

Also, pdoc3 doesn't have a pyproject.toml, so it's probably going through the "legacy" install mechanism that does setup.py install, which again will trigger the use of easy_install, I believe.

[Jackenmen]

From the little time I put into trying to fix this issue, I found that the problematic part to me was that filtering of pre-releases happens in a completely different place than checking for yanked releases - the first is done by the indexer and the second is done by the resolver. I didn't really know how to proceed from there without the change potentially affecting a lot of unrelated code.

Before realizing that, I just went for a simple change in the resolver logic that does in fact make it reject the yanked versions but it also completely ignores pre-releases in the situations like the one in the original issue description. I suppose that a better solution might be to completely get rid of the yanking logic in the resolver(s) and just handle it all in the indexer but I haven't really tried looking into how achievable that would be. Indexer would probably have to have its own filtering logic rather than relying on packaging's Specifier.filter() as it does now.

Anyway, I just wanted to share a bit of information in case it's of any help to a potential future contributor but if it's not, that's fine too 😄

[dlukyanov]

Does it make any difference if you replace pip3 install with python3 -m pip install? In other words, does python3 -m pip --version give the same output as pip3 --version?

no diff - there is only one version, i just played with different options

[pradyunsg]

@dlukyanov Could you post the complete output of the pip install run in a GitHub gist? The … in your posts are hiding vital debugging information that would make it easier for everyone else to see what is happening without spinning up a Docker container (eg: I don’t have Docker installed on any of my personal machines).

[dlukyanov]

# pip3 install pdoc3==0.10.0
Collecting pdoc3==0.10.0
  Downloading pdoc3-0.10.0.tar.gz (86 kB)
     |████████████████████████████████| 86 kB 1.3 MB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.8 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/setup.py'"'"'; __file__='"'"'/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-_42ow1r0
         cwd: /tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/
    Complete output (21 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/setup.py", line 15, in <module>
        setup(
      File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib64/python3.8/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 446, in __init__
        _Distribution.__init__(self, {
      File "/usr/lib64/python3.8/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 735, in finalize_options
        ep.load()(self, ep.name, value)
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/integration.py", line 29, in version_keyword
        dist.metadata.version = _get_version(config)
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 188, in _get_version
        dump_version(
      File "/tmp/pip-install-65g_8nml/pdoc3_0a61515b252043c6a3a6b2a5aad7a524/.eggs/setuptools_scm-6.2.0-py3.8.egg/setuptools_scm/__init__.py", line 94, in dump_version
        version_fields = parsed_version.release
    AttributeError: 'Version' object has no attribute 'release'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/95/be/69267c988fb7236cd60c452a4e7fb9a7991729476db490b634a07e7dfcdf/pdoc3-0.10.0.tar.gz#sha256=5f22e7bcb969006738e1aa4219c75a32f34c2d62d46dc9d2fb2d3e0b0287e4b7 (from https://pypi.org/simple/pdoc3/) (requires-python:>= 3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pdoc3==0.10.0 (from versions: 0.3.11, 0.3.12, 0.3.13, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.9.0, 0.9.1, 0.9.2, 0.10.0)
ERROR: No matching distribution found for pdoc3==0.10.0

[uranusjr]

Your issue does not look related to pip at all.

[dlukyanov]

Your issue does not look related to pip at all.

why pip takes setuptools_scm-6.2.0 that is yanked?

[hroncok]

It looks like easy_install does that, not pip.

[RonnyPfannschmidt]

It's a long standing setuptools issue (it can't self replace with a required version)

The pending setuptools_scm 6.3.0 release will support running on outdated setuptools again

Release will happen within the next hour

[uranusjr]

I think #10625 fixes this, but someone should add a test case to make sure.

[befeleme]

I just ran across the "second" part of the original issue here. With pip 22.1.2 and a mypkg having those releases:

1.0.0a1 pre-release
0.1.0 yanked

I would expect pip to install the pre-release 1.0.0a1 version when invoked with pip install mypkg
Instead, pip errors that it can't find any matching distribution for mypkg

When I try to install package that has only pre-releases, no such problem occurs. Pip installs the latest pre-release.

I would expect pip to behave the same way in both cases, ergo that it would install the pre-release.

[ddelange]

Hi 👋 There's a related issue / test case: #11745