setuptools pkg_resources fails to detect version properly

Originally reported by: Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL)

---

When installing a package:

  File "/home/larsoner/.local/lib/python2.7/site-packages/setuptools/dist.py", line 268, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
  File "/home/larsoner/.local/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/larsoner/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 836, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/home/larsoner/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1074, in best_match
    dist = working_set.find(req)
  File "/home/larsoner/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 711, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (numpy 1.11.0.dev0-2329eae (/home/larsoner/.local/lib/python2.7/site-packages), Requirement.parse('numpy>=1.6.1'))

In other words, my numpy 1.11.0.dev0 is not seen as satisfying a >= 1.6.1 version requirement. It seems like using disutils.version.LooseVersion or so should solve this. I'm happy to look into making a PR if this seems like the right solution.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/419

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

The issue is that with PEP 440, your local version is parsed as a LegacyVersion:

>>> import pkg_resources.packaging as packaging
>>> packaging.version.parse('1.11.0.dev0-2329eae')
<LegacyVersion('1.11.0.dev0-2329eae')>

And all LegacyVersions compare less than a proper Version of 0.

It's actually the post release tag with the dash in it that's forcing the LegacyVersion. If you can omit that tag, then the version will parse as a proper Version and the requirement will be satisfied.

@dstufft: Do you have any recommendations for this issue?

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

Use a + instead of a -.

>>> import pkg_resources._vendor.packaging as packaging
>>> packaging.version.parse('1.11.0.dev0+2329eae')
<Version('1.11.0.dev0+2329eae')>

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

Great, I'll work with the numpy folks to update it. Thanks!

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

Fixing in numpy

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

...maybe not NumPy's problem

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

So I looked, and numpy correctly sets its numpy.__version__ using +. The problem is that the numpy .egg_info is written out to be safe as 1.11.0.dev0_2329eae, i.e. the + has been replaced by _. Eventually (down a bit of a rabbit hole) in the pkg_resources module, the version extracted from this filename gets put through safe_version, which replaces the _ with a -, thus incorrectly reconstructing the module name. Is there a better/safer way to parse the version number, or could safe_version be improved to deal with this case somehow?

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

Or alternatively, Version could be improved to handle the _ being in place of + in the filename somehow.

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

It sounds like somewhere along the lines someone is using an older version of setuptools. Only older versions mangle the + in that way:

>>> import pkg_resources._vendor.packaging as packaging
>>> packaging.version.parse('1.11.0.dev0+2329eae')
<Version('1.11.0.dev0+2329eae')>
>>> import pkg_resources
>>> pkg_resources.safe_version('1.11.0.dev0+2329eae')
'1.11.0.dev0+2329eae'

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

You mean older versions of setuptools replace the + with a _ in the filename...?

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

Yes (see https://caremad.io/s/IaPtgPLQVE/). I think 8.0 might have been when PEP 440 was implemented in setuptools (and thus when + started to be supported in versions) but my memory might be off.

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

I figured that the filename was being changed to be safe... I'll try to track down who is actually changing the filename, but I suspect it is their desired (and not unreasonable) behavior.

I'm not sure I quite follow your example, though. The problem I'm having here is that the egg-info filename is being parsed for the version, the version in that filename has replaced + with _, and then the _ is being changed to - instead of +. In your example, your filename doesn't have a + in it. Your egg info does have the + in the version within the file, but then again so does mine:

larsoner@bunk:~/python/scitran$ cat ~/.local/lib/python2.7/site-packages/numpy-1.11.0.dev0_2329eae.egg-info 
Metadata-Version: 1.1
Name: numpy
Version: 1.11.0.dev0+2329eae
...

But that is a bit immaterial I think, since that file isn't actually parsed in pkg_resources (just the filename is), at least in the debugging I did. Did you expect that the version actually in that file would come into play?

FWIW I'm currently running 18.2 (most recent release) of setuptools, but it's possible somebody is using an older version or something.

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

It works when installed too: https://caremad.io/s/SsluEIrRRO/

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

Turns out numpy and scipy use distutils to do some of their building, including the part where the filename is constructed. I'll bother them about this issue, hopefully they'll have some solution at their end. But given the lengths they've gone to in order to use distutils instead of setuptools in their setup script, I'm not too optimistic. Is changing the parsing of _ in filenames in setuptools off the table?

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

We considered doing that in the PEP, however the problem was a version like 1.0-1 is valid as an alternative spelling of 1.0.post1 (1.0-1 was supported in setuptools previously and had use on PyPI). When you create a wheel the - character is munged to _ because wheel uses - to separate different parts of the filename so that munging makes a - in the filename unambiguously the separator in wheel, but it means that we need to consider _ equivalent to -. So 1.0-1 and 1.0_1 are both already in use to mean something in particular. We toyed with the idea of making it so that 1.0-<anything that's not an integer> (and thus 1.0_<anything that's not an integer>) as an alias for 1.0+<anything that's not an integer>, however we determined that it was far too liberal and too magical, making strange things that are obviously not versions into "valid" versions.

Original comment by Eric89GXL (Bitbucket: Eric89GXL, GitHub: Eric89GXL):

---

Or perhaps setuptools could actually read the version information from within the file instead of just using the filename...?