Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add topic on namespace packages #290

Merged
merged 2 commits into from Apr 9, 2017
Merged

Conversation

theacodes
Copy link
Member

@theacodes theacodes commented Apr 6, 2017

Resolves #265.

Copy link
Member

@ncoghlan ncoghlan left a comment

Thanks for this!

The overall structure looks good to me, just two major comments (and a few minor ones inline):

  • since the import system is covered in the language reference these days, I think we should adopt "native namespace packages" as the preferred term over "PEP-420-style namespace packages"
  • at the import system level, the compatibility between native namespace packages and pkgutil-style namespace packages should be solid, and I think we should commit to also maintaining that compatibility across the PyPA tooling so folks can mix and match them freely (leaving pkg_resources-style namespace packages as the only "all-or-nothing" case)

:Last Reviewed: 2017-05-04

Namespace packages allow you to split a single Python package across multiple
distributions. For instance, if you have the following package structure:
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While distributions is formally correct, it never really caught on in general use, so I suspect this opening sentence might prove confusing. Perhaps:

Namespace packages allow you to share a single Python import namespace between multiple projects or :term:`distribution packages <Distribution Package>`.

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I updated it to reference terms an disambiguate distributions. Let me know what you think. I suspect no matter what we do this sentence will be difficult to grok, but the example below is hopefully illustrative enough to convey understanding.

packages (such as a large corpus of client libraries for multiple products from
a single company). However, namespace packages come with several caveats and
are not appropriate in all cases. A simple alternative is to use a prefix on
all of your distributions such as ``import mynamespace_subpackage_a``.
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps import mynamespace_subpackage_a as subpackage_a to get an alternative closer to the from imports above?

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, but most style guides discourage aliasing. I've added it as an or. :)


The three methods of creating namespace packages are largely not
cross-compatible. It's inadvisable to use different methods in distributions of
the same namespace package.
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest "distributions that contribute to the same namespace package", since most namespaces either have no particular owning distribution (e.g. backports.*), or else one owning project and multiple contributing projects.

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Done.


There are currently three different approaches to creating namespace packages:

#. Use `PEP 420-style namespace packages`_. This is recommended if packages in
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest referring to these as "native namespace packages" and linking to https://docs.python.org/3/reference/import.html#namespace-packages, rather than codifying the PEP number here - if the native namespace behaviour changes in the future, the language reference will be updated, but PEP 420 won't.

However, the "PEP 420 namespace package" term would still be worth mentioning as an alternative name for native namespace packages.

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Done.

subpackage_a/
# Sub-packages have __init__.py.
__init__.py
module.py
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current examples may give the impression that namespace packages only support subpackages, and not submodules - it may simplify things to instead only show submodules, and then state somewhere that any form of submodule (including full subpackages) is supported.

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in the very first section and examples and explicitly called out that sub-packages and modules can participate in namespace packages.


It is extremely important that every distribution that uses the namespace
package omits the ``__init__.py``. If any distribution does not, it will cause
the namespace logic to fail and the other sub-packages will not be importable.
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably worth stating that the __init__.py files for pkgutil-style explicit namespace packages are also OK here, since they replicate the logic otherwise used by native namespace packages.

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

you need compatibility with packages already using this method or if your
package needs to be zip-safe.

The three methods of creating namespace packages are largely not
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd adjust this to state that we do expect native namespace packages and pkgutil-style namespace packages to be cross-compatible. If mixing them in any given Python version fails, then that's a bug in either pkgutil.extend_path() or the import machinery.

(And we should have stdlib tests in place to ensure that cross-compatibility if we don't already)

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

**Every** distribution that uses the namespace package must include an
identical ``__init__.py``. If any distribution does not, it will cause the
Copy link
Member

@ncoghlan ncoghlan Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, I think we should soften the recommendation to also allow "native namespace packages" here.

That way namespaces can more easily support a mix of pure Python 3 projects using native namespace packages, and hybrid Python 2/3 ones using pkgutil-style namespace packages, rather than believing that everyone has to use pkgutil-style namespace as long as anyone is.

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Member Author

@theacodes theacodes Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done via a tip in the native section.

@theacodes
Copy link
Member Author

theacodes commented Apr 7, 2017

since the import system is covered in the language reference these days, I think we should adopt "native namespace packages" as the preferred term over "PEP-420-style namespace packages"

SGTM, I'll incorporate that.

at the import system level, the compatibility between native namespace packages and pkgutil-style namespace packages should be solid, and I think we should commit to also maintaining that compatibility across the PyPA tooling so folks can mix and match them freely (leaving pkg_resources-style namespace packages as the only "all-or-nothing" case)

For sure, the problem is that according to my tests there are a few issues with that. Notably, if you install the first package with setuptools and the second with pip it'll fail. I'm hesitate to advertise compatibility until that's resolved. @jaraco Any ideas what's going on in that case?

@theacodes
Copy link
Member Author

theacodes commented Apr 7, 2017

@ncoghlan Thanks for the review, it's ready for another round. :)

@ncoghlan ncoghlan merged commit ff95678 into pypa:master Apr 9, 2017
@ncoghlan
Copy link
Member

ncoghlan commented Apr 9, 2017

Very nice, thank you! In reviewing this, it made me realise that now that namespace packages are documented, it might be feasible to also cover the two main options to writing plugin systems for Python applications: #291

@theacodes
Copy link
Member Author

theacodes commented Apr 9, 2017

@ncoghlan my pleasure! One small note - the examples for namespace packages currently live on my personal github, it might make more sense for those to live under the pypa org. I'm happy to populate that repo if someone can create it.

@theacodes theacodes deleted the namespace-packages branch Apr 9, 2017
ncoghlan pushed a commit to ncoghlan/python-packaging-user-guide that referenced this pull request Jun 24, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants