New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add topic on namespace packages #290

Merged
merged 2 commits into from Apr 9, 2017

Conversation

Projects
None yet
2 participants
@theacodes
Member

theacodes commented Apr 6, 2017

Resolves #265.

@ncoghlan

ncoghlan requested changes Apr 7, 2017 edited

Thanks for this!

The overall structure looks good to me, just two major comments (and a few minor ones inline):

  • since the import system is covered in the language reference these days, I think we should adopt "native namespace packages" as the preferred term over "PEP-420-style namespace packages"
  • at the import system level, the compatibility between native namespace packages and pkgutil-style namespace packages should be solid, and I think we should commit to also maintaining that compatibility across the PyPA tooling so folks can mix and match them freely (leaving pkg_resources-style namespace packages as the only "all-or-nothing" case)
:Last Reviewed: 2017-05-04
Namespace packages allow you to split a single Python package across multiple
distributions. For instance, if you have the following package structure:

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

While distributions is formally correct, it never really caught on in general use, so I suspect this opening sentence might prove confusing. Perhaps:

Namespace packages allow you to share a single Python import namespace between multiple projects or :term:`distribution packages <Distribution Package>`.

This comment has been minimized.

@theacodes

theacodes Apr 7, 2017

Member

Good point. I updated it to reference terms an disambiguate distributions. Let me know what you think. I suspect no matter what we do this sentence will be difficult to grok, but the example below is hopefully illustrative enough to convey understanding.

packages (such as a large corpus of client libraries for multiple products from
a single company). However, namespace packages come with several caveats and
are not appropriate in all cases. A simple alternative is to use a prefix on
all of your distributions such as ``import mynamespace_subpackage_a``.

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

Perhaps import mynamespace_subpackage_a as subpackage_a to get an alternative closer to the from imports above?

This comment has been minimized.

@theacodes

theacodes Apr 7, 2017

Member

Perhaps, but most style guides discourage aliasing. I've added it as an or. :)

The three methods of creating namespace packages are largely not
cross-compatible. It's inadvisable to use different methods in distributions of
the same namespace package.

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

I'd suggest "distributions that contribute to the same namespace package", since most namespaces either have no particular owning distribution (e.g. backports.*), or else one owning project and multiple contributing projects.

This comment has been minimized.

@theacodes

theacodes Apr 7, 2017

Member

Good point. Done.

There are currently three different approaches to creating namespace packages:
#. Use `PEP 420-style namespace packages`_. This is recommended if packages in

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

I'd suggest referring to these as "native namespace packages" and linking to https://docs.python.org/3/reference/import.html#namespace-packages, rather than codifying the PEP number here - if the native namespace behaviour changes in the future, the language reference will be updated, but PEP 420 won't.

However, the "PEP 420 namespace package" term would still be worth mentioning as an alternative name for native namespace packages.

This comment has been minimized.

@theacodes

theacodes Apr 7, 2017

Member

Good point. Done.

subpackage_a/
# Sub-packages have __init__.py.
__init__.py
module.py

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

The current examples may give the impression that namespace packages only support subpackages, and not submodules - it may simplify things to instead only show submodules, and then state somewhere that any form of submodule (including full subpackages) is supported.

This comment has been minimized.

@theacodes

theacodes Apr 7, 2017

Member

Done in the very first section and examples and explicitly called out that sub-packages and modules can participate in namespace packages.

It is extremely important that every distribution that uses the namespace
package omits the ``__init__.py``. If any distribution does not, it will cause
the namespace logic to fail and the other sub-packages will not be importable.

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

It's probably worth stating that the __init__.py files for pkgutil-style explicit namespace packages are also OK here, since they replicate the logic otherwise used by native namespace packages.

This comment has been minimized.

@theacodes
you need compatibility with packages already using this method or if your
package needs to be zip-safe.
The three methods of creating namespace packages are largely not

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

I'd adjust this to state that we do expect native namespace packages and pkgutil-style namespace packages to be cross-compatible. If mixing them in any given Python version fails, then that's a bug in either pkgutil.extend_path() or the import machinery.

(And we should have stdlib tests in place to ensure that cross-compatibility if we don't already)

This comment has been minimized.

@theacodes
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
**Every** distribution that uses the namespace package must include an
identical ``__init__.py``. If any distribution does not, it will cause the

This comment has been minimized.

@ncoghlan

ncoghlan Apr 7, 2017

Member

As above, I think we should soften the recommendation to also allow "native namespace packages" here.

That way namespaces can more easily support a mix of pure Python 3 projects using native namespace packages, and hybrid Python 2/3 ones using pkgutil-style namespace packages, rather than believing that everyone has to use pkgutil-style namespace as long as anyone is.

This comment has been minimized.

@theacodes

This comment has been minimized.

@theacodes

theacodes Apr 7, 2017

Member

Done via a tip in the native section.

@theacodes

This comment has been minimized.

Member

theacodes commented Apr 7, 2017

since the import system is covered in the language reference these days, I think we should adopt "native namespace packages" as the preferred term over "PEP-420-style namespace packages"

SGTM, I'll incorporate that.

at the import system level, the compatibility between native namespace packages and pkgutil-style namespace packages should be solid, and I think we should commit to also maintaining that compatibility across the PyPA tooling so folks can mix and match them freely (leaving pkg_resources-style namespace packages as the only "all-or-nothing" case)

For sure, the problem is that according to my tests there are a few issues with that. Notably, if you install the first package with setuptools and the second with pip it'll fail. I'm hesitate to advertise compatibility until that's resolved. @jaraco Any ideas what's going on in that case?

@theacodes

This comment has been minimized.

Member

theacodes commented Apr 7, 2017

@ncoghlan Thanks for the review, it's ready for another round. :)

@ncoghlan ncoghlan merged commit ff95678 into pypa:master Apr 9, 2017

@ncoghlan

This comment has been minimized.

Member

ncoghlan commented Apr 9, 2017

Very nice, thank you! In reviewing this, it made me realise that now that namespace packages are documented, it might be feasible to also cover the two main options to writing plugin systems for Python applications: #291

@theacodes

This comment has been minimized.

Member

theacodes commented Apr 9, 2017

@ncoghlan my pleasure! One small note - the examples for namespace packages currently live on my personal github, it might make more sense for those to live under the pypa org. I'm happy to populate that repo if someone can create it.

@theacodes theacodes deleted the theacodes:namespace-packages branch Apr 9, 2017

ncoghlan added a commit to ncoghlan/python-packaging-user-guide that referenced this pull request Jun 24, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment