Alternative source distro names make dependency acquisition fail

[michaelweiser]

My project requires service_identity which in turn requires pyasn1 and pyasn1-modules. Boilt down to a reproducer, setup.py looks like this

from setuptools import setup
setup(name='repro', install_requires=["service_identity"])

When run in a fresh virtualenv (or anywhere else) installation fails with the following error message:

error: The 'pyasn1' distribution was not found and is required by service-identity

Commands to reproduce with above setup.py:

rm -rf ~/p
virtualenv --python=python3 ~/p
~/p/bin/python setup.py install

(Issue also exists with python2.7.)

I've tracked this to interpret_distro_name in setuptools which has a comment warning of this kind of problem:

setuptools/setuptools/package_index.py

Lines 161 to 170 in 83c667e

	# Because some packages are ambiguous as to name/versions split
	# e.g. "adns-python-1.1.0", "egenix-mx-commercial", etc.
	# So, we generate each possible interepretation (e.g. "adns, python-1.1.0"
	# "adns-python, 1.1.0", and "adns-python-1.1.0, no version"). In practice,
	# the spurious interpretations should be ignored, because in the event
	# there's also an "adns" package, the spurious "python-1.1.0" version will
	# compare lower than any numeric version number, and is therefore unlikely
	# to match a request for it. It's still a potential problem, though, and
	# in the long run PyPI and the distutils should go for "safe" names and
	# versions in distribution archive names (sdist and bdist).

In this instance it seems, the alternative name "pyasn1, modules-0.2.5" poisons the distribution cache in the environment for the actual "pyasn1" module, causing it to download the pyasn1-modules-0.2.5 distribution once again, installing it once again and after that realizing that this did not actually install the pyasn1 module.

The following one-liner modifies above reproducer command to patch an exception for pyasn1 into the module as proof that it's this routine causing the problem:

rm -rf ~/p ; virtualenv --python=python3 ~/p
sed -i -e "/p in range(1/i \\
    o = 1 \\
    if parts[0] == 'pyasn1' and parts[1] == 'modules': \\
        o=2" -e "s/p in range(1/p in range(o/" ~/p/lib/python*/*-packages/setuptools/package_index.py
~/p/bin/python setup.py install

The comment suggests that there's no easy way to fix this.
What can I do to work around this in my project in the meantime?

Full failed command log:

running install
running bdist_egg
running egg_info
writing repro.egg-info/PKG-INFO
writing dependency_links to repro.egg-info/dependency_links.txt
writing requirements to repro.egg-info/requires.txt
writing top-level names to repro.egg-info/top_level.txt
reading manifest file 'repro.egg-info/SOURCES.txt'
writing manifest file 'repro.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying repro.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying repro.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying repro.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying repro.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying repro.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/repro-0.0.0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing repro-0.0.0-py3.6.egg
Copying repro-0.0.0-py3.6.egg to /home/michael/p/lib/python3.6/site-packages
Adding repro 0.0.0 to easy-install.pth file

Installed /home/michael/p/lib/python3.6/site-packages/repro-0.0.0-py3.6.egg
Processing dependencies for repro==0.0.0
Searching for service_identity
Reading https://pypi.org/simple/service_identity/
Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl#sha256=001c0707759cb3de7e49c078a7c0c9cd12594161d3bf06b9c254fdcb1a60dc36
Best match: service-identity 18.1.0
Processing service_identity-18.1.0-py2.py3-none-any.whl
Installing service_identity-18.1.0-py2.py3-none-any.whl to /home/michael/p/lib/python3.6/site-packages
writing requirements to /home/michael/p/lib/python3.6/site-packages/service_identity-18.1.0-py3.6.egg/EGG-INFO/requires.txt
Adding service-identity 18.1.0 to easy-install.pth file

Installed /home/michael/p/lib/python3.6/site-packages/service_identity-18.1.0-py3.6.egg
Searching for pyasn1-modules
Reading https://pypi.org/simple/pyasn1-modules/
Downloading https://files.pythonhosted.org/packages/91/f0/b03e00ce9fddf4827c42df1c3ce10c74eadebfb706231e8d6d1c356a4062/pyasn1_modules-0.2.5-py2.py3-none-any.whl#sha256=f309b6c94724aeaf7ca583feb1cc70430e10d7551de5e36edfc1ae6909bcfb3c
Best match: pyasn1-modules 0.2.5
Processing pyasn1_modules-0.2.5-py2.py3-none-any.whl
Installing pyasn1_modules-0.2.5-py2.py3-none-any.whl to /home/michael/p/lib/python3.6/site-packages
writing requirements to /home/michael/p/lib/python3.6/site-packages/pyasn1_modules-0.2.5-py3.6.egg/EGG-INFO/requires.txt
Adding pyasn1-modules 0.2.5 to easy-install.pth file

Installed /home/michael/p/lib/python3.6/site-packages/pyasn1_modules-0.2.5-py3.6.egg
Searching for pyasn1
Downloading https://files.pythonhosted.org/packages/ec/0b/69620cb04a016e4a1e8e352e8a42717862129b574b3479adb2358a1f12f7/pyasn1-modules-0.2.5.tar.gz#sha256=ef721f68f7951fab9b0404d42590f479e30d9005daccb1699b0a51bb4177db96
Best match: pyasn1 modules-0.2.5
Processing pyasn1-modules-0.2.5.tar.gz
Writing /tmp/easy_install-nnuvns4h/pyasn1-modules-0.2.5/setup.cfg
Running pyasn1-modules-0.2.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-nnuvns4h/pyasn1-modules-0.2.5/egg-dist-tmp-kcfpa3ji
no previously-included directories found matching 'doc/build'
removing '/home/michael/p/lib/python3.6/site-packages/pyasn1_modules-0.2.5-py3.6.egg' (and everything under it)
Moving pyasn1_modules-0.2.5-py3.6.egg to /home/michael/p/lib/python3.6/site-packages
pyasn1-modules 0.2.5 is already the active version in easy-install.pth

Installed /home/michael/p/lib/python3.6/site-packages/pyasn1_modules-0.2.5-py3.6.egg
error: The 'pyasn1' distribution was not found and is required by service-identity

Environment:
Ubuntu 18.04 latest updates
distro's python 2.7.15rc1
distro's python 3.6.7
distro's virtualenv 15.1.0

Versions inside reproducer venv:
setuptools 41.0.1
pip 19.1
service-identity 18.1.0
pyasn1 0.4.5
pyasn1-modules 0.2.5

[benoit-pierre]

Duplicate of #498.

[benoit-pierre]

Suggested workaround: use pip install . instead of python setup.py install which uses the deprecated easy_install path.

[michaelweiser]

Thanks for the very quick response. My searches did not turn up that original issue. Sorry for the noise.