Metadata inconsistent error message should contain all metadata caused the inconsistency

[larsoner]

• pip version: 20.3
• Python version: 3.9 or 3.8 (only two I've tested)
• OS: Linux or Windows (only two I've tested)

See numpy/numpy#17885 (comment) -- in 20.2.4 this works:

$ python3.9 -m pip install --progress-bar off --upgrade "pip!=20.3"
...
Successfully installed pip-20.2.4
$ python3.9 -m pip install --progress-bar off --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy
...
Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129035606_d0a457f-cp38-cp38-manylinux2010_x86_64.whl (15.4 MB)

Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.0.dev0+d3cffdc
    Uninstalling numpy-1.21.0.dev0+d3cffdc:
      Successfully uninstalled numpy-1.21.0.dev0+d3cffdc
Successfully installed numpy-1.21.0.dev0+d0a457f

But with 20.4:

$ python3.9 -m pip install --progress-bar off --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy
...
Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129040310_d0a457f-cp39-cp39-manylinux2010_x86_64.whl (15.4 MB)

ERROR: Requested numpy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129040310_d0a457f-cp39-cp39-manylinux2010_x86_64.whl has different version in metadata: '1.21.0.dev0+d0a457f'

[dstufft]

I'm guessing a call to urlunquote needs added.

[uranusjr]

Weird, the new resolver does not reimplement any HTTP logic, it’s all shared with the legacy resolver.

[pradyunsg]

diff --git a/src/pip/_internal/resolution/resolvelib/candidates.py b/src/pip/_internal/resolution/resolvelib/candidates.py
index 1fc2ff479..a50e888c6 100644
--- a/src/pip/_internal/resolution/resolvelib/candidates.py
+++ b/src/pip/_internal/resolution/resolvelib/candidates.py
@@ -210,6 +210,8 @@ def _check_metadata_consistency(self, dist):
         if self._name is not None and self._name != name:
             raise MetadataInconsistent(self._ireq, "name", dist.project_name)
         version = dist.parsed_version
+        print(version)
+        print(self._version)
         if self._version is not None and self._version != version:
             raise MetadataInconsistent(self._ireq, "version", dist.version)

Gives:

1.21.0.dev0+d0a457f
1.21.0.dev0+20201129033659.d0a457f

I don't think there's a functionality bug here. The versions are actually different. We should probably present them both in the error message though, to avoid confusion like this.

[larsoner]

I don't think there's a functionality bug here. The versions are actually different. We should probably present them both in the error message though, to avoid confusion like this.

From this it's clear at least that in the pip internals somewhere they are different in a way that they must not have been (or that did not matter) in 20.2.4. Can you clarify where these two versions get parsed from the wheel/setuptools/etc. so that the NumPy wheels can be fixed at their end?

[charris]

The date of wheel creation is added to the wheel names so that pip will pick up the latest. The wheel generation itself is automated, are you suggesting that the date should be added when setup is run? This used to work, so the question we are asking is: bug or feature.

[pradyunsg]

@larsoner Sorry, I was very much in "triage all the notifications" mode when I looked at this and responded here.

pip's new resolver is stricter across the board for a lot of things related to a package's metadata. One of these is being stricter about the version information that pip sees about a package -- ensuring that what's provided from the METADATA file in that distribution (or as generated from that, in case of sdists) matches what the distribution can/may provide from other mechanisms -- in this case, the wheel filename.

What's happening here is that the wheel file (https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129040310_d0a457f-cp39-cp39-manylinux2010_x86_64.whl) says the version is 1.21.0.dev0+20201129040310_d0a457f but all the metadata within the file says that it is 1.21.0.dev0+d0a457f.

The resolver is noticing this, and aborting since inconsistent metadata => can't rely on it => can't use it => AAAAAA. 😉

[pfmoore]

Also, just to be clear, this behaviour was present in 20.2.4, however it was opt-in only, as it's part of the new resolver. So the change here is actually just that the new resolver is now default. You should be able to demonstrate this by using the --use-feature=2020-resolver flag on pip 20.2.4, and you should see the same behaviour as 20.3. Conversely, you can (for a short period) opt out of the new resolver using --use-deprecated=legacy-resolver to get the old behaviour back. But this option will be getting dropped next release, so you should definitely only view this as a stopgap solution.

[charris]

How does pip handle build tags? Do they need to be internally specified and equivalent to those found in the file name?

[pradyunsg]

@charris What do you mean by build tags? If you're referring to wheel's build tags, pip does exactly what the wheel spec says, as far as I know. https://www.python.org/dev/peps/pep-0427/#file-name-convention

[uranusjr]

They can be present in the version string, as long as the one listed in METADATA matches the one in the wheel’s file name. So the most simplistic fix would be to modify the METADATA content (and its corresponding entry in RECORD) to match.

[pradyunsg]

I just tried pip install --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy on Python 3.8 (MacOS), with pip's master branch and it worked.

❯ pip install --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy 
Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2B438.gbf4ac1f81/numpy-1.21.0.dev0%2B438.gbf4ac1f81-cp38-cp38-macosx_10_9_x86_64.whl (16.1 MB)
     |████████████████████████████████| 16.1 MB 401 kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.21.0.dev0+438.gbf4ac1f81
❯ pip --version                                                                                                     
pip 21.0.dev0 from /Users/pradyunsg/Projects/pip/src/pip (python 3.8)
❯ pwd                                 
/Users/pradyunsg/Projects/pip
❯ git show HEAD                              
commit 7b7469b8454be1c0f968957e93ae23264187ee41 (HEAD -> master, upstream/master, upstream/HEAD, origin/master, origin/HEAD)
Merge: aa089883a a79758c13
Author: Paul Moore <p.f.moore@gmail.com>
Date:   Sat Jan 16 17:55:28 2021 +0000

    Merge pull request #9460 from pfmoore/vendor_21_0
    
    Vendoring changes for release 21.0 (excluding setuptools)

I think this either got fixed in our code, or on Anaconda's end. If someone else could provide a new reproducer for this or confirm that this is no longer an issue, that'll be great. :)

[charris]

@pradyunsg NumPy changed its versioning in master to use versioneer, so that might have fixed some things along with pip 20.3.

[uranusjr]

The numpy wheels now have consistent versions in metadata and filename, so I think it’s the versioneer change that fixed it for numpy. I think we can close this now? I’ll change the title to log the error message improvement instead.

[larsoner]

I see the edit about the improved title, sounds good to me and thanks for the quick information and fixes!