fixup_namespace_packages may trigger ValueError in setuptools 19.4 and newer

[ghost]

Originally reported by: kehoste (Bitbucket: kehoste, GitHub: Unknown)

---

We are hitting a problem with setuptools 19.4 and newer, while things are working fine with setuptools 19.3 and earlier. The problem was introduced with the changes in Pull Request #167 .

The problem is that a call to fixup_namespace_packages is triggering a ValueError, due to the way in which we 'flatten' our easybuild.easyblocks namespace (see here).

More details about the issue we're hitting are available here.

A test case is attached, the problem can be triggered with:

export PYTHONPATH=/tmp/testcase/one:$PYTHONPATH
python test.py

Correct output (which we obtain with setuptools 19.3 and earlier) would be:

atlas
boost
blacs

With setuptools 19.4 and later we get:

atlas
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    fixup_namespace_packages(os.path.join(testdir, 'two'))
  File "build/bdist.macosx-10.10-intel/egg/pkg_resources/__init__.py", line 2237, in fixup_namespace_packages
  File "build/bdist.macosx-10.10-intel/egg/pkg_resources/__init__.py", line 2235, in fixup_namespace_packages
  File "build/bdist.macosx-10.10-intel/egg/pkg_resources/__init__.py", line 2194, in _handle_ns
  File "build/bdist.macosx-10.10-intel/egg/pkg_resources/__init__.py", line 2192, in sort_key
ValueError: '/private/tmp/testcase/one/easybuild' is not in list

The problem is that the way in which the parent location of the package is determined in the sort_key inner function is broken, basically because we 'rewrite' a package name like easybuild.easyblocks.a.atlas to easybuild.easyblocks.atlas.
So the 'depth' of the location at which the module is actually available (easybuild/easyblocks/a/atlas.py) doesn't match the depth of the package name it is imported with (easybuild.easyblocks.atlas).

Since this works fine with setuptools 19.3 and earlier, I consider this a bug introduced in setuptools 19.4.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/520

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I recognize that this behavior appears as a regression from 19.3, but it's also a pretty obscure expectation. It's too late to release 19.4 as a backward-incompatible release, but at this point, I would consider the implementation the new expectation.

One could argue that Setuptools should back out this change in order to support this expectation, but given how difficult and painful it has been to get this functionality to (mostly) work, I'm strongly inclined to keep the existing functionality, despite its limitations.

That said, I would be happy to accept a patch that additionally supports easybuild, especially if the patch is suitably encapsulated, limiting the complexity it adds to the implementation when easybuild is not relevant.

I'll take a look at the test case and see if it reveals any additional inspiration.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Using the attached test case, I get a slightly different error. Same error message, but sooner (before 'atlas' is printed).

$ virtualenv --python python2.7 testcase/env
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /Users/jaraco/issue520/testcase/env/bin/python
Installing setuptools, pip, wheel...done.
$ testcase/env/bin/easy_install -q easybuild
Installing with setuptools.setup...
Installing version 2.6.0
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

zip_safe flag not set; analyzing archive contents...
Installing with setuptools.setup...
Installing version 2.6.0 (API version 2)
Installing with setuptools.setup...
Installing version 2.6.0 (required versions: API >= 2)
Installing with setuptools.setup...
Installing version 2.6.0 (required versions: API >= 2, easyblocks >= 2.6)
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

INFO: This is (based on) vsc.install.shared_setup 0.9.19
INFO: run_tests from base dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18 (using executable /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/setup.py)
WARN: cleanup lib/vsc_base.egg-info
INFO: generated list: ['vsc', 'vsc.utils']
INFO: generated packages list: ['vsc', 'vsc.utils']
INFO: makesetupcfg set to True, (re)creating setup.cfg
INFO: found license /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/LICENSE with md5sum 5f30f0716dfdd0d91eb439ebec522ec2
INFO: Found license name LGPLv2+ and classifier License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)
INFO: setting license LGPLv2+
INFO: No name defined, trying to determine it
INFO: found match name vsc-base in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/PKG-INFO
INFO: found match url https://github.com/hpcugent/vsc-base in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/PKG-INFO
INFO: Removing None download_url
INFO: get_name_url returns {'name': 'vsc-base', 'url': 'https://github.com/hpcugent/vsc-base'}
INFO: using long_description Common tools used within our organization. Originally created by the HPC team of Ghent University (http://ugent.be/hpc).
INFO: generated list: ['bin/logdaemon.py', 'bin/optcomplete.bash', 'bin/startlogdaemon.sh']
INFO: generated scripts list: ['bin/logdaemon.py', 'bin/optcomplete.bash', 'bin/startlogdaemon.sh']
INFO: Searching for vsc-install>=0.9.19
INFO: Reading https://pypi.python.org/simple/vsc-install/
INFO: Best match: vsc-install 0.9.19
INFO: Downloading https://pypi.python.org/packages/source/v/vsc-install/vsc-install-0.9.19.tar.gz#md5=db8c042144fe59acce6d5ba61693ff4f
INFO: Processing vsc-install-0.9.19.tar.gz
INFO: Writing /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/temp/easy_install-g1GGRW/vsc-install-0.9.19/setup.cfg
INFO: Running vsc-install-0.9.19/setup.py -q bdist_egg --dist-dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/temp/easy_install-g1GGRW/vsc-install-0.9.19/egg-dist-tmp-0wgZb4
INFO: This is (based on) vsc.install.shared_setup 0.9.19
INFO: run_tests from base dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/temp/easy_install-g1GGRW/vsc-install-0.9.19 (using executable /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/temp/easy_install-g1GGRW/vsc-install-0.9.19/setup.py)
INFO: not enforcing prospector support. run "easy_install propspector" yourself
WARN: cleanup lib/vsc_install.egg-info
INFO: generated list: ['vsc', 'vsc.install']
INFO: generated packages list: ['vsc', 'vsc.install']
INFO: makesetupcfg set to True, (re)creating setup.cfg
INFO: found license /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/temp/easy_install-g1GGRW/vsc-install-0.9.19/LICENSE with md5sum 5f30f0716dfdd0d91eb439ebec522ec2
INFO: Found license name LGPLv2+ and classifier License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)
INFO: setting license LGPLv2+
INFO: No name defined, trying to determine it
INFO: found match name vsc-install in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/temp/easy_install-g1GGRW/vsc-install-0.9.19/PKG-INFO
INFO: found match url https://github.com/hpcugent/vsc-install in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/temp/easy_install-g1GGRW/vsc-install-0.9.19/PKG-INFO
INFO: Removing None download_url
INFO: get_name_url returns {'name': 'vsc-install', 'url': 'https://github.com/hpcugent/vsc-install'}
INFO: using long_description vsc-install provides shared setuptools functions and classes for python libraries developed by UGent's HPC group
INFO: generated list: []
INFO: generated scripts list: []
WARN: zip_safe flag not set; analyzing archive contents...
WARN: vsc.fancylogger: module references __file__
WARN: vsc.install.shared_setup: module references __file__
WARN: vsc.install.shared_setup: module references __path__
WARN: vsc.install.shared_setup: module MAY be using inspect.getsource
INFO: creating /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/.eggs/vsc_install-0.9.19-py2.7.egg
INFO: Extracting vsc_install-0.9.19-py2.7.egg to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/.eggs
INFO: 
Installed /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-wckVyV/vsc-base-2.4.18/.eggs/vsc_install-0.9.19-py2.7.egg
WARN: zip_safe flag not set; analyzing archive contents...
WARN: vsc.utils.exceptions: module MAY be using inspect.getouterframes
WARN: vsc.utils.fancylogger: module MAY be using inspect.stack
WARN: vsc.utils.generaloption: module MAY be using inspect.stack
INFO: This is (based on) vsc.install.shared_setup 0.9.19
INFO: run_tests from base dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-2j34Mw/vsc-install-0.9.19 (using executable /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-2j34Mw/vsc-install-0.9.19/setup.py)
INFO: not enforcing prospector support. run "easy_install propspector" yourself
WARN: cleanup lib/vsc_install.egg-info
INFO: generated list: ['vsc', 'vsc.install']
INFO: generated packages list: ['vsc', 'vsc.install']
INFO: makesetupcfg set to True, (re)creating setup.cfg
INFO: found license /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-2j34Mw/vsc-install-0.9.19/LICENSE with md5sum 5f30f0716dfdd0d91eb439ebec522ec2
INFO: Found license name LGPLv2+ and classifier License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)
INFO: setting license LGPLv2+
INFO: No name defined, trying to determine it
INFO: found match name vsc-install in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-2j34Mw/vsc-install-0.9.19/PKG-INFO
INFO: found match url https://github.com/hpcugent/vsc-install in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-2j34Mw/vsc-install-0.9.19/PKG-INFO
INFO: Removing None download_url
INFO: get_name_url returns {'name': 'vsc-install', 'url': 'https://github.com/hpcugent/vsc-install'}
INFO: using long_description vsc-install provides shared setuptools functions and classes for python libraries developed by UGent's HPC group
INFO: generated list: []
INFO: generated scripts list: []
WARN: zip_safe flag not set; analyzing archive contents...
WARN: vsc.fancylogger: module references __file__
WARN: vsc.install.shared_setup: module references __file__
WARN: vsc.install.shared_setup: module references __path__
WARN: vsc.install.shared_setup: module MAY be using inspect.getsource
$ PYTHONPATH=testcase/one testcase/env/bin/python testcase/test.py
Traceback (most recent call last):
  File "testcase/test.py", line 5, in <module>
    import easybuild.easyblocks.atlas
  File "/Users/jaraco/issue520/testcase/one/easybuild/easyblocks/__init__.py", line 8, in <module>
    pkg_resources.declare_namespace(__name__)
  File "/Users/jaraco/issue520/testcase/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2070, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/Users/jaraco/issue520/testcase/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2021, in _handle_ns
    _rebuild_mod_path(path, packageName, module)
  File "/Users/jaraco/issue520/testcase/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2039, in _rebuild_mod_path
    orig_path.sort(key=position_in_sys_path)
  File "/Users/jaraco/issue520/testcase/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2037, in position_in_sys_path
    return sys_path.index(_normalize_cached(os.sep.join(parts)))
ValueError: '/Users/jaraco/issue520/testcase/one/easybuild' is not in list
$ testcase/env/bin/python -V
Python 2.7.10
$ testcase/env/bin/python -m easy_install --version
setuptools 20.3.1 from /Users/jaraco/issue520/testcase/env/lib/python2.7/site-packages (Python 2.7)

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

From my pdb session:

ValueError: '/Users/jaraco/issue520/testcase/one/easybuild' is not in list
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /Users/jaraco/issue520/testcase/env/lib/python2.7/site-packages/pkg_resources/__init__.py(2037)position_in_sys_path()
-> return sys_path.index(_normalize_cached(os.sep.join(parts)))
(Pdb) l
2032            """
2033            Return the ordinal of the path based on its position in sys.path
2034            """
2035            parts = p.split(os.sep)
2036            parts = parts[:-(package_name.count('.') + 1)]
2037 ->         return sys_path.index(_normalize_cached(os.sep.join(parts)))
2038    
2039        orig_path.sort(key=position_in_sys_path)
2040        module.__path__[:] = [_normalize_cached(p) for p in orig_path]
2041    
2042    
(Pdb) _normalize_cached(os.sep.join(parts))
'/Users/jaraco/issue520/testcase/one/easybuild'
(Pdb) sys.path
['testcase', '/Users/jaraco/issue520/testcase/one', '/Users/jaraco/issue520/testcase/env/lib/python27.zip', ...]

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I'm afraid I'm stuck on this one. I think what we desire is a function that takes the __path__ item of a module and is able to resolve each of the items in that path to the relevant sys.path entry that resolves to that item, so when testcase/one/easybuild/easyblocks/a appears in some __path__, it's resolved to testcase/one in sys.path. The heuristic that Eric has formulated above, stripping path names based on the number of parts in the module is adequate for the simple case, but fails in this use case.

Can someone suggest a better way to resolve the sys.path entry for a given __path__?

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

I agree that just because it happened to work before, this is a really exceptional case that was never intended to work. I still think that the sys_path.index call should be wrapped in a try/except, like:

try:
    return sys_path.index(_normalize_cached(os.sep.join(parts)))
except ValueError:
    return float('inf')

in other words, if a __path__ entry does not appear in sys.path then just sort it last. Normally that shouldn't be happening at all, but at least there's a fallback if it does happen.

As for easybuild, instead of performing unexpected manipulations of __path__, which really is supposed to be internal information for the import system and is not meant to be manipulated by third-party code (pkg_resources being an extremely exceptional case, unfortunately), I would suggest implementing a custom import hook particular to easybuild. This is the "supported" way to change how the import system works.

[Mekk]

I faced this problem (and found this bug) yesterday, after creating fresh virtualenv (which installed setuptools 35.0.1 (and resolved the case by downgrading to 19.3). The backtrace ended in:

      File "/home/marcink/.virtualenvs/sfx_3-0/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2066, in _rebuild_mod_path
        orig_path.sort(key=position_in_sys_path)
      File "/home/marcink/.virtualenvs/sfx_3-0/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2064, in position_in_sys_path
        return sys_path.index(_normalize_cached(os.sep.join(parts)))
    ValueError: '/home/marcink/GAUSS/python/sfx__util/src' is not in list

Funny thing is that my virtualenv contains both lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py (which contains the fix) and lib/python2.7/site-packages/pkg_resources/__init__.py (which does not, and which fails).

The thing above can be easily reproduced on my Ubuntu 16.04 by simple:

mkvirtualenv freshenv

Anyone has suggestion, who is to blame?