python3.6 can't import installed namespace package if another package in the same namespace is put locally and pkg_resources are imported

[daa]

Recently I've met an issue with python3.6 and pkg_resources-style namespace packages when for one namespace one package is installed into virtualenv (a) and one is located in current directory (b) - after importing pkg_resources or setuptools package a becomes inaccessible. But no such behaviour is observed with python2.7. To be more clear I have an example:

# package a:
a $ tree
.
├── pkgns
│   ├── __init__.py
│   └── pkga
│       └── __init__.py
├── setup.cfg
└── setup.py

a $ cat pkgns/__init__.py 
__import__('pkg_resources').declare_namespace(__name__)

a $ cat setup.py 
from setuptools import find_packages, setup


setup(
    name='pkgns.pkga',
    version='1.0',
    namespace_packages=[
        'pkgns',
    ],
    packages=find_packages(),
)

# package b:
b $ tree
.
├── pkgns
│   ├── __init__.py
│   └── pkgb
│       └── __init__.py
├── setup.cfg
├── setup.py
└── show-bug.py

b $ cat pkgns/__init__.py 
import pkg_resources

pkg_resources.declare_namespace(__name__)

b $ cat setup.py 
from setuptools import find_packages, setup


setup(
    name='pkgns.pkgb',
    version='1.0',
    namespace_packages=[
        'pkgns',
    ],
    packages=find_packages(),
)

# this file will show a bug, pkg_resources import is necessary to see unexpected behaviour
b $ cat show-bug.py 
import pkg_resources

import pkgns.pkga

# create virtualenv with python2.7 in env27 and python3.6 in env36, install pkgns.pkga to both
# and see problem
b $ ./env36/bin/python show-bug.py 
Traceback (most recent call last):
  File "show-bug.py", line 3, in <module>
    import pkgns.pkga
ModuleNotFoundError: No module named 'pkgns.pkga'

b $ ./env27/bin/python show-bug.py 
b $ echo $?
0

As you can see pkgns.pkga installed in virtualenv was successfully imported under python2.7 and failed under python3.6. For me this is an issue because our code is structured similarly and some our packages contain helper functions to use in setup.py but this bug prevents me from using them. I had a workaround by using older setuptools to build helper packages but this is not a very reliable solution.

I investigated an issue further:

# with python2.7
b $ ./env27/bin/python 
Python 2.7.14 (default, Feb 20 2018, 17:13:09) 
[GCC 6.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['pkgns'].__path__
['.../b/env27/lib/python2.7/site-packages/pkgns']
>>> import pkg_resources
>>> sys.modules['pkgns'].__path__
['.../b/pkgns', '.../b/env27/lib/python2.7/site-packages/pkgns']

# and with python3.6
b $ ./env36/bin/python 
Python 3.6.4 (default, Feb 20 2018, 20:29:59) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['pkgns'].__path__
_NamespacePath(['.../b/env36/lib/python3.6/site-packages/pkgns'])
>>> import pkg_resources
>>> sys.modules['pkgns'].__path__
['./pkgns']

So pkg_resources during import changed namespace package __path__ attribute in one case correctly and in second not - old path was lost. Also one may notice that under python3.6 original path is not list but _NamespacePath object and this gives us a key to understanding source of problem. Let's look at nspkg.pth file:

b $ cat env36/lib/python3.6/site-packages/pkgns.pkga-1.0-py2.7-nspkg.pth 
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('pkgns',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('pkgns', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('pkgns', [os.path.dirname(p)])));m = m or sys.modules.setdefault('pkgns', types.ModuleType('pkgns'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)

If python version is greater than 3.5 pkgns module entry is created using importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec(...)) function which gives real pep420 namespace package, pkg_resources on import adjusts namespace packages paths but handles only lists as original path in _rebuild_mod_path() in handle_ns() function (commit 7c0c39e to fix #885):

2141     if not isinstance(orig_path, list):
2142         # Is this behavior useful when module.__path__ is not a list?
2143         return
2144 
2145     orig_path.sort(key=position_in_sys_path)
2146     module.__path__[:] = [_normalize_cached(p) for p in orig_path]

So what happens briefly: pkg_resources tries to fix namespace packages paths on import, and inside _handle_ns() loads module (pkgns.pkgb in my example), after this pkgns' __path__ attribute points to package in current directory, orig_path points to old _NamespacePath object and _rebuild_mod_path does not handle this combination leaving new path not fixed.

Here are versions of software I used: setuptools-39.0.1, pip-9.0.3, python-2.7.14, python-3.6.4 and python-3.6.5.

I can see at least 2 ways to fix the issue - assign module.__path__ to orig_path when the latter is not a list or to convert it to list and proceed with sorting and normalizing, but I cannot currently predict consequences of both decisions.

[daa]

I've found that this is issue is similar to #900 and proposed pull-request should fix that issue too.