pkg_util technique leaves namespace packages undeclared

[jaraco]

As I was investigating pypa/setuptools#1038 and filing pytest-dev/pytest#2419, I realized there's a potential issue with the two recommended mechanisms for declaring namespace packages (PEP-420 and pkg_util). Neither of those techniques provide a mechanism to declare a package as a namespace package, and especially with the pkg_util technique, the namespace package looks indistinguishable from a regular package. Am I right?

This behavior leads to subtle differences in how package managers like pip install the package:

$ pip install -t pkg-util backports.unittest_mock==1.3
Collecting backports.unittest_mock==1.3
  Using cached backports.unittest_mock-1.3-py2.py3-none-any.whl
Installing collected packages: backports.unittest-mock
Successfully installed backports.unittest-mock-1.3
$ pip install -t pkg-res backports.unittest_mock==1.2
Collecting backports.unittest_mock==1.2
  Using cached backports.unittest_mock-1.2-py2.py3-none-any.whl
Installing collected packages: backports.unittest-mock
Successfully installed backports.unittest-mock-1.2
$ ls pkg-util/backports
__init__.py	__pycache__	unittest_mock
$ ls pkg-res/backports
unittest_mock
$ grep __init__.py pkg-util/*.dist-info/RECORD
backports/__init__.py,sha256=jv2YF__bseklT3OWEzlqJ5qE24c4aWd5F4r0TTjOrWQ,65
backports/unittest_mock/__init__.py,sha256=BbmHzzuuzpHjcezoa0q62AEHxbmUkOzmSr_jPOoTGU0,255
$ grep __init__.py pkg-res/*.dist-info/RECORD
backports/unittest_mock/__init__.py,sha256=BbmHzzuuzpHjcezoa0q62AEHxbmUkOzmSr_jPOoTGU0,255
$ ls pkg-*/*.dist-info/namespace_packages.txt
pkg-res/backports.unittest_mock-1.2.dist-info/namespace_packages.txt

I'm left feeling uneasy about namespace packages using the pkg_util technique. It feels to me like it's losing important information, including the pep-420 forward compatibility (where the __init__.py is removed and superseded by a -nspkg.pth file to configure the namespace package).

Most importantly, it seems to me there's no way to detect in code that a package like backports is a namespace package. That could lead to issues, especially if tools like pytest are relying on the absence of the __init__.py or other factors to avoid behavior on namespace packages.

[jaraco]

In light of this issue, is the pkg-util technique an adequate replacement for the pkg_resources technique? Can we expect a package to detect (or care about) the distinction between a namespace package and a non-namespace package?

I also worry that the lack of a -nspkg.pth file may lead to other issues that were addressed through changes to this file (such as pypa/setuptools#250).

[jaraco]

I also worry that the lack of a -nspkg.pth file may lead to other issues

I've run some tests with these pkg-util-based packages and haven't been able to elicit any emergent issues, so I'll presume my worries are unfounded.

[theacodes]

@jaraco okay, so it seems you findings match the test results here which indicate that our samples are fine.

I too was uneasy about the namespace not actually be declared anywhere other than the __init__.py, but you can't argue with results I guess.

Can we close this, or is there something I've missed?

[ghost]

It the pkgutil method an appropriate way to declare namespace packages? Or is pkg_resources preferred?

[theacodes]

@xoviat based on our test matrix pkgutil is the best choice for new namespaces that need to work on 2 & 3. pkg_resources is only recommended for packages that already have other packages in the same namespace using that method.

[theacodes]

(Didn't mean to close until @jaraco confirms)

[jaraco]

I concur. Thanks for taking the time to confirm.

[ncoghlan]

The resolution here makes sense, but adding in the import system level context for future reference:

At the language level, a "namespace package" is formally just a normal package where the only module level behaviour is to set the metadata attributes (including __path__), with all other attributes being submodules. No runtime code is expected to care about how that state was achieved (whether natively via the import system, or via code in an otherwise self-contained package that manipulates __path__ the same way that the import system would if there was no __init__.py file present.

Native namespace packages that are created by leaving out __init__.py can by identified by way ofm.__spec__.loader being None, and m.__spec__.origin being "namespace", but we'd prefer that folks avoid relying on that in general, since it won't pick up namespace package emulations (regardless of whether they're pkgutils or pkg_resources based, or do their own __path__ manipulation).

There's a different question which relates to how this should be represented in the sdist (e.g. by having the build system generate the __init__.py at install or wheel creation time based on declarative metadata the way setuptools does, rather than by actually including the __init__.py file in the sdist), but runtime code still shouldn't be relying on that to decide whether or not a particular module looks like a namespace package or not.

[theacodes]

Thanks for the context, @ncoglan, that was really helpful.

…

On Sat, May 20, 2017, 7:49 AM Nick Coghlan ***@***.***> wrote: Adding in the import system level context for additional context: At the language level, a "namespace package" is formally just a normal package where the only module level behaviour is to set the metadata attributes (including __path__), with all other attributes being submodules. No runtime code is expected to care about how that state was achieved (whether natively via the import system, or via code in an otherwise self-contained package that manipulates __path__ the same way that the import system would if there was no __init__.py file present. Native namespace packages that are created by leaving out __init__.py *can* by identified by way ofm.__spec__.loader being None, and m.__spec__.origin being "namespace", but we'd prefer that folks avoid relying on that in general, since it won't pick up namespace package emulations (regardless of whether they're pkgutils or pkg_resources based, or do their own __path__ manipulation). There's a *different* question which relates to how this should be represented in the sdist (e.g. by having the build system generate the __init__.py at install or wheel creation time based on declarative metadata the way setuptools does, rather than by actually including the __init__.py file in the sdist), but runtime code still shouldn't be relying on that to decide whether or not a particular module looks like a namespace package or not. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#312 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc7yeYPLbl2gfo6fHGtFa0kF2DULhks5r7v2WgaJpZM4Ngzej> .