SystemError: <method 'load_module' of 'zipimport.zipimporter' objects> returned NULL without setting an error

[Monism1]

Hello, I ran python\setup.py py2exe on converter.py which imports rawpy and imageio (and os, glob), it seemed to work but when I ran converter.exe the following error occurred, numpy and Pillow seem to have been correctly imported

`Traceback (most recent call last):
  File "converter.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\rmhan\AppData\Local\Programs\Python\Python37\lib\site-packages\rawpy\__init__.py", line 5, in <module>
    import rawpy._rawpy
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
SystemError: <method 'load_module' of 'zipimport.zipimporter' objects> returned NULL without setting an error

I'm using python 3.7.0 and py2exe-0.9.3.1-cp37-none-win_amd64. The following is my setup.py

from distutils.core import setup
import py2exe

setup( console=[{ "script": 'C:'+'\\'+'Users'+'\\'+'rmhan'+'\\'+'conversion+'\\'+'converter.py'},],
       options={"py2exe": {
            "packages": ['rawpy','imageio'],}})

Thanks in advance.

[nchidsey]

I just got the same error with numpy. Here's a small reproduction. Tried in Python 3.7.2 (64-bit) and 3.7.3 (32-bit), starting with a blank virtualenv.

foo.py:

import numpy
print("ok")

setup.py:

from setuptools import setup
import py2exe
setup(
    name='foo',
    version='1.0',
    console=['foo.py'],
)

build:

pip install numpy
pip install https://github.com/albertosottile/py2exe/releases/download/v0.9.3.1/py2exe-0.9.3.1-cp37-none-win_amd64.whl
python setup.py py2exe

Then running the exe in a new console window:

Traceback (most recent call last):
  File "foo.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\nick\.virtualenvs\tmp-b1da4736-4c9a-4313-808c-4bdae17f2fdb\lib\site-packages\numpy\__init__.py", line 150, in <module>
    from . import random
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\nick\.virtualenvs\tmp-b1da4736-4c9a-4313-808c-4bdae17f2fdb\lib\site-packages\numpy\random\__init__.py", line 180, in <module>
    from . import mtrand
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
SystemError: <method 'load_module' of 'zipimport.zipimporter' objects> returned NULL without setting an error

I have used the same versions of Python, py2exe and numpy before just fine, but I don't know what the difference is yet. Maybe the other project includes more modules that this simple reproduction doesn't include in the build? I don't think the problem is numpy-specific either. I lean toward thinking it has to do with pyd files and the stub pyc's inside the library.zip that import them.

[albertosottile]

I have used the same versions of Python, py2exe and numpy before just fine, but I don't know what the difference is yet.

This is quite puzzling. I was expecting at least an upgrade, maybe one of the numpy (and perhaps rawpy) dependencies? I cannot understand how an error like this could appear if none of the modules was updated. If one of you could find which dependency is causing this, it would be of great help.

[Monism1]

@nchidsey I think your problem is fixed if you replace your setup.py with the following:

from setuptools import setup
import py2exe
setup( console=[{'foo.py'},],
       options={"py2exe": {
            "packages": ['numpy'],}})

I.e. explicitly include the numpy package in setup.py

[nchidsey]

Correction - numpy was a different version. The current is 1.17.0. If I grab 1.16.4 instead, it works okay. I don't think numpy has any dependencies, at least when installed from the binary wheel. And yes specifying the package in the py2exe option seems to have fixed it.

By the way, the same exception occurs with packages other than numpy. In the few cases I've seen, the exception occurs at the line where it imports a module that is a stub pyc inside the zip file which imports a pyd that is outside the zipfile.

[albertosottile]

@Monism1 Interesting though, it appears that also your problem can be fixed in the same way, specifically by changing the setup.py script to

from distutils.core import setup
import py2exe

setup( console=[{ "script": 'converter.py'},],
       options={"py2exe": {
            "packages": ['numpy', 'rawpy','imageio'],}})

I just tested it with a dummy converter.py script that just imports imageio and rawpy on 3.7.4-64 and it worked. Can you please try this on your system? Thanks.

@nchidsey Honestly, I am not sure why that worked on numpy 1.16.4 and now it is apparently broken. But, I always run my numpy tests with the package name explicitely added in the setup.py script. I also encourage everyone to do that with all their dependencies, especially for packages that contain compiled libraries, such as numpy.

[nchidsey]

Adding packages to that option worked for numpy and a few others, but didn't work for another, which is still failing with the same exception and pyc-to-pyd combo. I'm trying to debug the DLL loading in Visual Studio. It gets into some Microsoft DLLs and fails there, but don't know why yet.

[albertosottile]

@nchidsey It could be that a DLL is not imported for some package and so a hook should be written or adjusted accordingly, but usually this leads to a different error message. In any case, let me know what you find. Thanks for your help.

[Monism1]

@albertosottile Yes actually it works fine! My mistake, I originally added all packages and dependencies (including os and glob) which caused an error (since I guess you should only include external libs) and forgot to re-add numpy. Interestingly Imageio also has Pillow as a dependency but it's not necessary to include that. Maybe the advice of including all package dependencies (if it initially doesn't work) could be included somewhere in the readme? Thanks again.

[albertosottile]

@Monism1 I am glad that everything works now, thanks for having confirmed that. This is a fork of py2exe, so all its extensive documentation still applies. In particular, there are some useful tips and tricks available here. Also, I have always found useful all the setup.py scripts that use py2exe and are available in the other open-source repositories.

@nchidsey I am going to close this issue now since the main probem has been deemed solved. Please open a new issue if you find something relevant during your investigations. Thanks for the time you are investing in this project.