__init__.py required for pkgutil.walk_packages in python3

[asottile]

BPO	29258
Nosy	@ncoghlan, @methane, @ericsnowcurrently, @wm75, @asottile

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-01-13.04:47:54.646>
labels = ['3.8', 'type-bug', '3.7', '3.9', 'docs']
title = '__init__.py required for pkgutil.walk_packages in python3'
updated_at = <Date 2020-01-20.12:16:18.752>
user = 'https://github.com/asottile'

bugs.python.org fields:

activity = <Date 2020-01-20.12:16:18.752>
actor = 'methane'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2017-01-13.04:47:54.646>
creator = 'Anthony Sottile'
dependencies = []
files = []
hgrepos = []
issue_num = 29258
keywords = []
message_count = 6.0
messages = ['285361', '289345', '289347', '360298', '360307', '360310']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'methane', 'docs@python', 'eric.snow', 'wolma', 'Anthony Sottile', 'Eric Wieser']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29258'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[asottile]

PEP-420 makes __init__.py files optional: https://docs.python.org/3/whatsnew/3.3.html#pep-420-implicit-namespace-packages

Though it seems without them, pkgutil.walk_packages does not function as desired: https://docs.python.org/3/library/pkgutil.html#pkgutil.walk_packages

Consider the following example:

$ tree foo
foo
├── bar
│   ├── baz.py
│   └── __init__.py
├── __init__.py
└── womp.py

And a test script

# test.py

import pkgutil

import foo


for _, mod, _ in pkgutil.walk_packages(foo.__path__, foo.__name__ + '.'):
    print(mod)

In both python2 and python3 I get the following output:

$ python2.7 test.py
foo.bar
foo.bar.baz
foo.womp
$ python3.5 test.py
foo.bar
foo.bar.baz
foo.womp

Removing the __init__.py files and only using python3, I get this:

$ find -name '__init__.*' -delete
$ python3.5 test.py
foo.bar

The modules are definitely importable:

$ python3.5 -c 'import foo.bar.baz'
$

Originally asked as a question on stackoverflow: http://stackoverflow.com/questions/41203765/init-py-required-for-pkgutil-walk-packages-in-python3

[wm75]

While it is rather trivial to implement the proposed functionality - all that's required here is to eliminate the check for __init__.py from pkgutil._iter_file_finder_modules - this would have undesired impacts on, e.g., pydoc.apropos:
This function would then recursively report *any* directory/subdirectory on sys.path, which is quite surely not what people want.

I think this is a fundamental problem with namespace packages: they are nice and flexible for specific imports, but they make it impossible to know whether a directory found on the filesystem is *intended* as a Python package or not.

[wm75]

all that's required here is to eliminate the check for __init__.py from pkgutil._iter_file_finder_modules

Ok, I was exaggerating here. To do it right would require a more complex change, but that's all that's needed to get an estimate of the effect the real thing would have.

[methane]

PEP-420 makes __init__.py files optional

This is almost wrong. PEP-420 added a new way for "namespace pacakge."
PEP-420 doesn't make empty __init__.py file in regular package.

(See https://dev.to/methane/don-t-omit-init-py-3hga)

Then, should pkgutil.walk_packages walk into all directories (e.g. node_modules) ? I don't think so.

[EricWieser]

If the resolution here is that this is behaving as intended (which personally I disagree with), I think this issue should remain open as a documentation task - the docs should clearly state that this does not apply to PEP-420 namespace packages.

[methane]

I am totally agree with Wolfgang:

they make it impossible to know whether a directory found on the filesystem is *intended* as a Python package or not.

I think we shouldn't treat normal directory as namespace package until some portion in the directory is imported, or it is specified explicitly.

So walk_packages() should be used like:

  walk_packages("/path/to/namespace", "namespace")

I already rejected similar issue: bpo-29642.

If you can not agree with me, please make an thread in python-dev ML or discuss.python.org.