Implement PEP 420: Implicit Namespace Packages

[ericvsmith]

BPO	14660
Nosy	@warsaw, @brettcannon, @jaraco, @ericvsmith, @merwok, @ericsnowcurrently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2012-05-25.00:23:19.669>
created_at = <Date 2012-04-24.11:08:39.543>
labels = ['type-feature']
title = 'Implement PEP 420: Implicit Namespace Packages'
updated_at = <Date 2012-05-28.05:35:46.953>
user = 'https://github.com/ericvsmith'

bugs.python.org fields:

activity = <Date 2012-05-28.05:35:46.953>
actor = 'python-dev'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2012-05-25.00:23:19.669>
closer = 'eric.smith'
components = []
creation = <Date 2012-04-24.11:08:39.543>
creator = 'eric.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 14660
keywords = []
message_count = 13.0
messages = ['159131', '159147', '159150', '159162', '159165', '159276', '159290', '159291', '159293', '159735', '161541', '161542', '161758']
nosy_count = 8.0
nosy_names = ['barry', 'brett.cannon', 'jaraco', 'eric.smith', 'eric.araujo', 'Yury.Selivanov', 'python-dev', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14660'
versions = ['Python 3.3']

[ericvsmith]

I have created a branch features/pep-420 where I'll be developing a proof of concept implementation of PEP-420.

I've checked in a basic version, but it has these issues:

• We need to decide how finders communicate that they've found part of
  a namespace package. Per Brett's suggestion, I'm currently returning a
  string that means "the returned path is part of a namespace package".
  I think that's an okay design.

• I guess we need to create a "namespace loader", so we can initialize __loader__. It's a little odd because it would be a loader for which there's no need for a find_module method. I'm currently setting __loader__ to the finder, which is completely wrong, but keeps things working.

• test_import and test_importlib both fail because they're checking
  that imports with no __init__.py fail. Those tests need to be removed.

• There are no tests yet.

[brettcannon]

Loaders are not meant to have a find_module method; that is purely for finders which can be distinct objects. So having a namespace loader is expected and there is no expectation that it have a find_module method (actually almost all of the loaders in importlib lack a find_module method).

[ericvsmith]

Right, that's a typo. I meant load_module(). I'm currently working on implementing the loader for namespace modules, so my comment about the loader is premature.

[ericvsmith]

I created the NamespaceLoader in the feature branch. It has a load_module, but it's only ever called by the code in PathFinder.load_module:

                loader = NamespaceLoader(namespace_path)
                return loader.load_module(fullname)

namespace_path is what will become module.__path__. In order to keep the load_module API (single fullname argument), I pass it in to the constructor. There's no particular need for it to follow that API, but it does.

[brettcannon]

Yeah, having to pass in the name to load_module is silly. I'm seriously considering making it optional for some loaders when the name was passed in to the constructor.

One thing I would like to see is that PathFinder take a new, keyword-only argument of 'namespace_loader' or something that takes an object which is called with the module name and the list of paths to use for __path__ (and thus __path__[0] becomes __file__). That way it can be controlled by users if desired. I'm also fine with it having a default value of NamespaceLoader to start since I don't see people needing to comprehend what they are allowing in this case as I do for general file loaders.

[ericvsmith]

I'd really prefer something like:

  return load_ns_module(fullname, namespace_path)

The point being that load_module() as defined in PEP-302 will never be called on NamespaceLoader. The loader only needs to exist to set module.__loader__, and load_module() would never be called from there.

So we could pass in a callable named load_ns_module to PathFinder.

Then NamespaceLoader's load_module function could be named load_ns_module, made a class or static method.

I think I'll experiment with this.

[brettcannon]

What do you mean the loader is only needed to set __loader__? You need the loader to create the module (or find it in sys.modules to reload), and set all the attributes properly. If you do this then reloading namespace modules will become a special case compared to other loaders as imp.reload() calls module.__loader__.load_module().

This also prevents the creation of an importlib.find_module() which would return the loader to replace imp.find_module() since you now split the API.

I realize the finder/loader dichotomy seems superfluous (and most of the time it is), but it has already been heavily exposed and relied on and deviating from it for namespace modules runs the risk of hurting introspection.

[ericvsmith]

Ah. I didn't realize that reload called load_module. I'll back out the change I just made, then.

My point was that the original call to load_module isn't made through the normal "a finder returned me a loader, so I'll call it" code path. It's always made through the "I know I have a NamespaceLoader" path.

[brettcannon]

The joys of trying to shoehorn into an existing API. I mean short of adding a new sys.namespace_loader instead of an explicit keyword argument to FileFinder I can't think of a better solution.

[ericvsmith]

I've modified zipimport to support namespace packages, and checked it in to the feature branch. This completes all of the functionality I think needs to be added. Next up is adding tests.

[python-dev]

New changeset 702009f3c0b1 by Eric V. Smith in branch 'default':
bpo-14660: Implement PEP-420, namespace packages.
http://hg.python.org/cpython/rev/702009f3c0b1

[ericvsmith]

I'm closing this issue. I'll open new issues as needed.