Scan folders that do not contain an __init__.py

[invisibleroads]

Python 3.3+ no longer requires __init__.py to import a folder as a module.
https://stackoverflow.com/questions/37139786/is-init-py-not-required-for-packages-in-python-3-3

Venusian does not scan a folder if it lacks an __init__.py.

That means a folder must explicitly have an __init__.py for venusian to scan it.

Not sure if this is something that should be fixed, but just making a note for consideration in case future developers stumble on this.

[stevepiercy]

For maintainers: Should we update the documentation to explicitly refer to this change in PEP 420, especially since we dropped support for Python 3.4 and older? Or should we update Venusian itself?

Side note, if you document docstrings in packages with Sphinx, you might want to still use an __init__.py or use --implicit-namespaces.

[digitalresistor]

This is a limitation in pkgutil.iter_modules, it doesn't return namespace packages (those without an __init__.py) unfortunately.

For example:

.
|____folder2
| |____test
| | |____other.py
| |____other.py
|____folder1
| |____test
| | |______init__.py
| | |____other.py
| |______init__.py
| |____other.py

Python 3.8.2 (default, Apr 18 2020, 23:23:51)
[Clang 11.0.3 (clang-1103.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgutil import walk_packages, iter_modules
>>> list(iter_modules(path=['.']))
[ModuleInfo(module_finder=FileFinder('.'), name='folder1', ispkg=True)]
>>> list(walk_packages(path=['.']))
[ModuleInfo(module_finder=FileFinder('.'), name='folder1', ispkg=True), ModuleInfo(module_finder=FileFinder('/Users/xistence/Projects/temp/sample_project/folder1'), name='folder1.other', ispkg=False), ModuleInfo(module_finder=FileFinder('/Users/xistence/Projects/temp/sample_project/folder1'), name='folder1.test', ispkg=True), ModuleInfo(module_finder=FileFinder('/Users/xistence/Projects/temp/sample_project/folder1/test'), name='folder1.test.other', ispkg=False)]

Take note that folder2 is not even listed in that output.

>>> for name, ob in getmembers(folder2):
...     print("{}: {}".format(name, ob))
...
__doc__: None
__file__: None
__loader__: <_frozen_importlib_external._NamespaceLoader object at 0x106072fa0>
__name__: folder2
__package__: folder2
__path__: _NamespacePath(['/Users/xistence/Projects/temp/sample_project/folder2'])
__spec__: ModuleSpec(name='folder2', loader=<_frozen_importlib_external._NamespaceLoader object at 0x106072fa0>, submodule_search_locations=_NamespacePath(['/Users/xistence/Projects/temp/sample_project/folder2']))

Compare that to:

>>> for name, ob in getmembers(folder1):
...     print("{}: {}".format(name, ob))
...
__builtins__: {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>), '__build_class__': <built-in function __build_class__>, '__import__': <built-in function __import__>, 'abs': <built-in function abs>, 'all': <built-in function all>, 'any': <built-in function any>, 'ascii': <built-in function ascii>, 'bin': <built-in function bin>, 'breakpoint': <built-in function breakpoint>, 'callable': <built-in function callable>, 'chr': <built-in function chr>, 'compile': <built-in function compile>, 'delattr': <built-in function delattr>, 'dir': <built-in function dir>, 'divmod': <built-in function divmod>, 'eval': <built-in function eval>, 'exec': <built-in function exec>, 'format': <built-in function format>, 'getattr': <built-in function getattr>, 'globals': <built-in function globals>, 'hasattr': <built-in function hasattr>, 'hash': <built-in function hash>, 'hex': <built-in function hex>, 'id': <built-in function id>, 'input': <built-in function input>, 'isinstance': <built-in function isinstance>, 'issubclass': <built-in function issubclass>, 'iter': <built-in function iter>, 'len': <built-in function len>, 'locals': <built-in function locals>, 'max': <built-in function max>, 'min': <built-in function min>, 'next': <built-in function next>, 'oct': <built-in function oct>, 'ord': <built-in function ord>, 'pow': <built-in function pow>, 'print': <built-in function print>, 'repr': <built-in function repr>, 'round': <built-in function round>, 'setattr': <built-in function setattr>, 'sorted': <built-in function sorted>, 'sum': <built-in function sum>, 'vars': <built-in function vars>, 'None': None, 'Ellipsis': Ellipsis, 'NotImplemented': NotImplemented, 'False': False, 'True': True, 'bool': <class 'bool'>, 'memoryview': <class 'memoryview'>, 'bytearray': <class 'bytearray'>, 'bytes': <class 'bytes'>, 'classmethod': <class 'classmethod'>, 'complex': <class 'complex'>, 'dict': <class 'dict'>, 'enumerate': <class 'enumerate'>, 'filter': <class 'filter'>, 'float': <class 'float'>, 'frozenset': <class 'frozenset'>, 'property': <class 'property'>, 'int': <class 'int'>, 'list': <class 'list'>, 'map': <class 'map'>, 'object': <class 'object'>, 'range': <class 'range'>, 'reversed': <class 'reversed'>, 'set': <class 'set'>, 'slice': <class 'slice'>, 'staticmethod': <class 'staticmethod'>, 'str': <class 'str'>, 'super': <class 'super'>, 'tuple': <class 'tuple'>, 'type': <class 'type'>, 'zip': <class 'zip'>, '__debug__': True, 'BaseException': <class 'BaseException'>, 'Exception': <class 'Exception'>, 'TypeError': <class 'TypeError'>, 'StopAsyncIteration': <class 'StopAsyncIteration'>, 'StopIteration': <class 'StopIteration'>, 'GeneratorExit': <class 'GeneratorExit'>, 'SystemExit': <class 'SystemExit'>, 'KeyboardInterrupt': <class 'KeyboardInterrupt'>, 'ImportError': <class 'ImportError'>, 'ModuleNotFoundError': <class 'ModuleNotFoundError'>, 'OSError': <class 'OSError'>, 'EnvironmentError': <class 'OSError'>, 'IOError': <class 'OSError'>, 'EOFError': <class 'EOFError'>, 'RuntimeError': <class 'RuntimeError'>, 'RecursionError': <class 'RecursionError'>, 'NotImplementedError': <class 'NotImplementedError'>, 'NameError': <class 'NameError'>, 'UnboundLocalError': <class 'UnboundLocalError'>, 'AttributeError': <class 'AttributeError'>, 'SyntaxError': <class 'SyntaxError'>, 'IndentationError': <class 'IndentationError'>, 'TabError': <class 'TabError'>, 'LookupError': <class 'LookupError'>, 'IndexError': <class 'IndexError'>, 'KeyError': <class 'KeyError'>, 'ValueError': <class 'ValueError'>, 'UnicodeError': <class 'UnicodeError'>, 'UnicodeEncodeError': <class 'UnicodeEncodeError'>, 'UnicodeDecodeError': <class 'UnicodeDecodeError'>, 'UnicodeTranslateError': <class 'UnicodeTranslateError'>, 'AssertionError': <class 'AssertionError'>, 'ArithmeticError': <class 'ArithmeticError'>, 'FloatingPointError': <class 'FloatingPointError'>, 'OverflowError': <class 'OverflowError'>, 'ZeroDivisionError': <class 'ZeroDivisionError'>, 'SystemError': <class 'SystemError'>, 'ReferenceError': <class 'ReferenceError'>, 'MemoryError': <class 'MemoryError'>, 'BufferError': <class 'BufferError'>, 'Warning': <class 'Warning'>, 'UserWarning': <class 'UserWarning'>, 'DeprecationWarning': <class 'DeprecationWarning'>, 'PendingDeprecationWarning': <class 'PendingDeprecationWarning'>, 'SyntaxWarning': <class 'SyntaxWarning'>, 'RuntimeWarning': <class 'RuntimeWarning'>, 'FutureWarning': <class 'FutureWarning'>, 'ImportWarning': <class 'ImportWarning'>, 'UnicodeWarning': <class 'UnicodeWarning'>, 'BytesWarning': <class 'BytesWarning'>, 'ResourceWarning': <class 'ResourceWarning'>, 'ConnectionError': <class 'ConnectionError'>, 'BlockingIOError': <class 'BlockingIOError'>, 'BrokenPipeError': <class 'BrokenPipeError'>, 'ChildProcessError': <class 'ChildProcessError'>, 'ConnectionAbortedError': <class 'ConnectionAbortedError'>, 'ConnectionRefusedError': <class 'ConnectionRefusedError'>, 'ConnectionResetError': <class 'ConnectionResetError'>, 'FileExistsError': <class 'FileExistsError'>, 'FileNotFoundError': <class 'FileNotFoundError'>, 'IsADirectoryError': <class 'IsADirectoryError'>, 'NotADirectoryError': <class 'NotADirectoryError'>, 'InterruptedError': <class 'InterruptedError'>, 'PermissionError': <class 'PermissionError'>, 'ProcessLookupError': <class 'ProcessLookupError'>, 'TimeoutError': <class 'TimeoutError'>, 'open': <built-in function open>, 'quit': Use quit() or Ctrl-D (i.e. EOF) to exit, 'exit': Use exit() or Ctrl-D (i.e. EOF) to exit, 'copyright': Copyright (c) 2001-2020 Python Software Foundation.
All Rights Reserved.

Copyright (c) 2000 BeOpen.com.
All Rights Reserved.

Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All Rights Reserved.

Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.
All Rights Reserved., 'credits':     Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
    for supporting Python development.  See www.python.org for more information., 'license': Type license() to see the full license text, 'help': Type help() for interactive help, or help(object) for help about object., '_': ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'test']}
__cached__: /Users/xistence/Projects/temp/sample_project/folder1/__pycache__/__init__.cpython-38.pyc
__doc__: None
__file__: /Users/xistence/Projects/temp/sample_project/folder1/__init__.py
__loader__: <_frozen_importlib_external.SourceFileLoader object at 0x106096040>
__name__: folder1
__package__: folder1
__path__: ['/Users/xistence/Projects/temp/sample_project/folder1']
__spec__: ModuleSpec(name='folder1', loader=<_frozen_importlib_external.SourceFileLoader object at 0x106096040>, origin='/Users/xistence/Projects/temp/sample_project/folder1/__init__.py', submodule_search_locations=['/Users/xistence/Projects/temp/sample_project/folder1'])
test: <module 'folder1.test' from '/Users/xistence/Projects/temp/sample_project/folder1/test/__init__.py'>

The two are treated very different from each other. Unfortunately this is not something that we can paper over in venusian.

[digitalresistor]

Do note that this issue will cause failures in other software too, which uses similar lookup methods. I believe that importlib_resources/pkg_resources is also bit by this if you try to for example get a resource from a path that does not include a __init__.py, and various other scanners also depend on there being a __init__.py. Sphinx for example also requires it.

[stevepiercy]

We might be able to adapt what Sphinx has done with implicit-namespaces:

• Sphinx apidoc does not process PEP-0420 implicit namespaces sphinx-doc/sphinx#2951
• Sphinx-apidoc does not recognize implicit namespace packages correctly sphinx-doc/sphinx#3962

[digitalresistor]

That walks the file system tree manually, instead of relying on the importer to do its thing. I'm wondering if there is any side effects there that we'd have to be aware of if we went down that same path.

Right now venusian has to not care if the package being scanned is a zip file or not, if we go down the route of manually trying to walk the tree, we'd need special handling for dealing with zipped files or other types of importers.