An os.walk inspired replacement for pkgutil.walk_packages

[ncoghlan]

BPO	17062
Nosy	@ncoghlan, @wm75
Files	issue17062.diff: Implementation of suggested walk_path

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-01-28.11:17:08.865>
labels = ['type-feature', 'library']
title = 'An os.walk inspired replacement for pkgutil.walk_packages'
updated_at = <Date 2017-03-10.17:00:13.803>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2017-03-10.17:00:13.803>
actor = 'BreamoreBoy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-01-28.11:17:08.865>
creator = 'ncoghlan'
dependencies = []
files = ['29934']
hgrepos = []
issue_num = 17062
keywords = ['patch']
message_count = 5.0
messages = ['180847', '180848', '180849', '187311', '221036']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'isoschiz', 'wolma']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17062'
versions = ['Python 3.4']

[ncoghlan]

I recently had occasion to use pkgutil.walk_packages, and my immediate thought was that it would have been a lot easier for me to use if it worked more like os.walk with topdown=True, producing tuples of (pkg, subpackages, modules)

"pkg" would be the package object at the current level (None for the top level)

"packages" would be a dictionary mapping fully qualified module names to loader objects for the subpackages (i.e. subdirectories)

"modules" would be a dictionary mapping fully qualified module names to loader objects for every submodule that wasn't a subpackage

As with editing the "subdirs" list with os.walk, editing the "packages" dictionary with this new API would keep the iterator from loading that subpackage and avoid recursing into it (this is the part I wanted in my current use case).

(This may even be PEP material, guiding some additions to the importer/finder API)

[ncoghlan]

Oops, forgot the proposed call signature:

    def walk_path(path=None, *, pkg=None):
        """Walk a package hierarchy, starting with the given path

    Iterator producing (package, subpackages, submodules) triples.
    The first entry is the package currently being walked, or None
    for the top level path. The subpackages and submodules entries
    are dictionaries mapping from fully qualified module names to
    the appropriate module loaders.

    Entries may be removed from the subpackages dictionary to avoid
    loading those packages and recursing into them.
    
    If both pkg and path are None, walks sys.path

    If path is not None, walks the specified path.

    If pkg is not None, walks pkg.__path__

    Providing both path and pkg results in ValueError
    """

[ncoghlan]

Regarding the PEP comment - the piece that would be missing is the "iter_modules" functionality. Currently pkgutil provides the support for standard filesystem imports and zipimports directly - the generic function based extension mechanism is undocumented.

[isoschiz]

I threw together a function that implements this. The only variation from the proposed signature was adding the onerror argument supported by the other similar functions in the module.

[BreamoreBoy]

Could somebody review the attached patch please.