submodule of c-extension module is quirky

[mattip]

BPO	43367
Nosy	@mattip, @skoslowski, @YannickJadoul
Files	test.c: Complete C module to reproduce the issue

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-02.11:41:23.575>
labels = []
title = 'submodule of c-extension module is quirky'
updated_at = <Date 2021-03-02.14:54:05.759>
user = 'https://github.com/mattip'

bugs.python.org fields:

activity = <Date 2021-03-02.14:54:05.759>
actor = 'skoslowski'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-03-02.11:41:23.575>
creator = 'mattip'
dependencies = []
files = ['49844']
hgrepos = []
issue_num = 43367
keywords = []
message_count = 2.0
messages = ['387917', '387935']
nosy_count = 3.0
nosy_names = ['mattip', 'skoslowski', 'YannickJadoul']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43367'
versions = []

[mattip]

If I have a module "parent", and I add another module "child" with a method "f" to it:

child = PyModule_Create(...);
PyModule_AddObject(parent, "child", child);

then I can call

import parent
parent.child.f()

but importing like this

from parent.child import f

raises a ModuleNotFoundError: ... 'parent' is not a package

This came up in PyTorch pytorch/pytorch#38137
and in pybind11 pybind/pybind11#2639,
and in various other places like stackoverflow https://stackoverflow.com/questions/38454852/importerror-with-error-is-not-a-package

A complete example is attached

If this is intentional, it might be nice to emit a warning when calling PyModule_AddObject with a module.

[skoslowski]

>> import parent.child

first imports "parent" (successfully) but then fails, because the import code has no knowledge of were to find ".child". This is because
a) the module "parent" is not marked as a package (hence the error message)
b) even if it were a package, there is no (ext) module file to locate and load.

If you instead run

> from parent import child

only "parent" is imported, and "child" is retrieved as an attribute - which it actually is. The import code itself will add such an attribute, too [1]. However, that is after the submodule was located and loaded. Attribute lookup on the parent is not part of the submodule import itself.

A (hacky) work-around would be to add an entry for "parent.child" in sys.modules. This prevents the import machinery from running.

A (more) proper solution would be to mark "parent" as a package and install some importlib hooks. See [2] for an example from Cython-land.

Finally there is PyImport_AppendInittab() [3], which could possibly be used to register "parent.child". I have never tried that. Even if this worked, it would be unsupported and probably not without side-effects.

[1] https://docs.python.org/3/reference/import.html#submodules
[2] https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-one-cython-extension
[3] https://docs.python.org/3/c-api/import.html?highlight=inittab#c.PyImport_AppendInittab

[LostTime76]

Bump

Can somebody please explain the proper usage of the C API such that we can create actual packages and submodules from C code that are lazy loaded (Py_AppendInittab)?

Edit:

After some spelunking in the cpython code to find where the "is not a package" error was being thrown, it appears it was not happy with the fact that the parent package did not have the path dunder attribute set. As a test on the parent, I put a module exec slot on the parent and set the path attribute in the exec function to None. Then I created another module definition for a few sub modules of the parent package and appended them using the PyImport_AppendInittab function and dot notation.

PyImport_AppendInittab(parent)
PyImport_AppendInittab(parent.sub)
PyImport_AppendInittab(parent.sub.sub)

The imports now work in a test python script -> import parent.sub.sub. No python, pure C code embedding python. Neat. Now I can lazy load with sub module support. Not sure if this is intended.. but unless somebody comes in here and says otherwise, I will run with this.

I guess the recommendation here would be to set an exec slot on every module and set the path dunder to None. Additionally, set other attributes like @mattip's OP to make sure the python accesses work as expected. This can all be done lazily as a python script requests the modules instead of brute force creating all the modules up front. After all, lazy loading (PyImport_AppendInittab) was my goal here.