#141 introduces regression in importing namespace packages

[ghost]

Originally reported by: embray (Bitbucket: embray, GitHub: embray)

---

Unfortunately the fix to #141 was basically my code :(

The regression is caused by the changes to pkg_resources._handle_ns. That change was added to support the scenario reported in #141 where the setup_requires package in question has a namespace package. For example:

#!python
setup_requires = ['foo.bar > 1.1']

where foo is a namespace package. The fix ensures that the path to the newer version of foo.bar is added earlier on foo.__path__ while still maintaining the original foo.__path__ entries in their original relative order.

Unfortunately this fix breaks the (likely more common case) where there are two copies of foo.bar on sys.path to begin with. For example, if upon starting the interpreter it has the entries:

#!python
sys.path = [
    ...
    'path/to/virtualenv/site-packages/foo.bar-1.1.egg/'
    ...
    'system/site-packages/foo.bar-1.0.egg/'
]

when one performs import foo, the first entry is found first resulting in

#!python
foo.__path__ == [ 'path/to/virtualenv/site-packages/foo.bar-1.1.egg/']

which is fine. But then as declare_namespace continues its march it gets to the later path entry which ends up reordering foo.__path__ so that

#!python
foo.__path__ == ['system/site-packages/foo.bar-1.0.egg/', 'path/to/virtualenv/site-packages/foo.bar-1.1.egg/']

Thus a call to import foo.bar ends up importing the old foo.bar later on the path.

If that's all confusing, I will attach a regression test in a bit. I think the solution is to handle the __path__ reordering in fixup_namespace_packages which is where it is actually necessary, and not in _handle_ns generally.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/207

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Thanks Erik for taking the lead on this; it's most appreciated.

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

Namespace packages are a great idea but kind of silly to get just right considering the history surrounding them :)

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

This first change moves some things around in test_easy_install to support a new test "test_setup_requires_override_nspkg" which specifically tests the use case my original changes to _handle_ns was meant to support. This test should pass currently. It's a little ugly but gets the job done.

[ghost]

Original comment by embray (Bitbucket: embray, GitHub: embray):

---

Encountered this lovely issue again for the first time in a while, when the software stack used by many of my coworkers upgraded setuptools for the first time in a long time.

The fix to #141 handled namespace packages in such a way that whenever a new distribution is activated, if it contains a member of a namespace package, it inserts that package's path first into the namespace package's __path__. This allows a dependency to be "upgraded" by way of setup_requires.

However, this is at odds the normal behavior, where the first time some member of a namespace package is imported, declare_namespace is called and each sys.path entry is searched for members of the namespace package. Those paths are then added to the namespace package's __path__ in the order they're found on sys.path. Because of the fix in #141, this results in the __path__ being in the reverse order of sys.path.

I have a potential fix for this that ensures that a namespace package's __path__ is always ordered according to sys.path. This also required changing how new eggs are activated on sys.path, by requiring them to be inserted at the beginning of sys.path, rather than appending them to the end. This seems to make more sense in general, I think. But I'm not 100% sure what the impact of this change is overall.

[timj]

Did 8a304f3 fix this issue? Should it have been closed?

[jaraco]

Presumably so.