Issue with numba functions compiled in dynamically created ModuleType class

[lespaul]

Using the latest version of numba, it appears that numba @njit/jit functions that are run via 'exec' in module dictionaries created with the ModuleType class have issues. Functions run via 'exec' in standard dictionaries appear to run correctly as shown below:

import numpy as np
from types import ModuleType

code = """
from numba import jit

@jit
def testfunc():
    return 0
"""

module = ModuleType("test_module")
standard_dict = {}
exec(code, standard_dict, standard_dict)
print standard_dict['testfunc']()

exec(code, module.__dict__, module.__dict__)
print module.testfunc()

---

0
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "REDACTED.py", line 19, in <module>
    module.__dict__['testfunc']()
  File ".local/lib/python2.7/site-packages/numba/dispatcher.py", line 369, in _compile_for_args
    raise e
KeyError: "Failed in object mode pipeline (step: object mode backend)\n'test_module'"

[stuartarchibald]

Thanks for the report. I think the problem is that the dynamically created module is not in sys.modules. When the compiler tries to find the globals for the function being compiled it needs to interrogate the module the function is in but fails to find the module (because it only looks in two places, 1) a special "dynamic" module for generated functions and 2) sys.modules) and this is the root cause of the failure.

This works:

import numpy as np
import sys
from types import ModuleType

code = """
from numba import njit

@njit
def testfunc():
    return 0
"""

module = ModuleType("test_module")
standard_dict = {}
exec(code, standard_dict, standard_dict)
print(standard_dict['testfunc']())

# register dynamic module
sys.modules['test_module'] = module
exec(code, module.__dict__, module.__dict__)
print(module.testfunc())

[sklam]

There's a 1-liner to make it work. https://gist.github.com/sklam/c78bfcd90f54d14dc96e539e3276a62d
But... should it work?

Functions do not hold a reference to their parent module. They store the name of the module under func.__module__. To find the parent module, one uses sys.modules[func.__module__]. The error reported is due to Numba trying to find from the sys.modules. If the module is not stored into sys.modules or anywhere else, the module object will be likely collected. I do not know if it is useful to support dynamic modules that are not in sys.modules.

[stuartarchibald]

Closing this issue as it seems to be resolved. If this is not the case please re-open with a comment about any item that appears to be unresolved. Many thanks.