Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SystemError raised from zipimporter when importing a simple package #3157

Open
mental32 opened this issue Sep 28, 2019 · 2 comments

Comments

@mental32
Copy link

commented Sep 28, 2019

Cython doesn't seem to want to play nice when being imported. I get this traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/home/mental/.local/lib/python3.7/site-packages/foob-0.0.1-py3.7-linux-x86_64.egg/t/__init__.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
SystemError: <method 'load_module' of 'zipimport.zipimporter' objects> returned NULL without setting an error

I've got a very simple package structure that looks like this:

.
├── r
├── setup.py
└── t
    ├── e
    │   ├── f.pyx
    │   └── __init__.pyx
    └── __init__.py

3 directories, 4 files

Here are the contents of the source files:

# setup.py
import glob
from setuptools import setup
from Cython.Build import cythonize

ext_sources = [fp for fp in glob.glob('t/**/*.pyx', recursive=True)]

setup(
    name='foob',
    version='0.0.1',
    packages=['t', 't.e'],
    ext_modules=cythonize(ext_sources, compiler_directives={'language_level': '3'})
)

# t/__init__.py
from .e import g

print(g()) 

# t/e/__init__.pyx
from .f import g

# t/e/f.pyx
def g() -> int:
    return 3

Running python setup.py install --user completes succesfully with this output: https://gist.github.com/mental32/fe1ecdf5ba6f412e2c868f39582fe27b

I use the r/ directory so I can force Python to import the installed package instead of the local one.

$ cd r
$ python
Python 3.7.4 (default, Jul 16 2019, 07:12:58) 
[GCC 9.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import t
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/home/mental/.local/lib/python3.7/site-packages/foob-0.0.1-py3.7-linux-x86_64.egg/t/__init__.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
SystemError: <method 'load_module' of 'zipimport.zipimporter' objects> returned NULL without setting an error
>>>

My apologies if this is a simple issue but I've been searching through google and the cython docs to try and figure out why this issue is occuring, the closes I could get is this SO post and if I'm understanding the accepted answer correctly this seems to be a codegen issue, so I assume Cython is responsible.

I'm using Python 3.7.4 with Cython 0.29.13 installed from pypi and uname displays:
Linux hydrogen 5.3.1-arch1-1-ARCH #1 SMP PREEMPT Sat Sep 21 11:33:49 UTC 2019 x86_64 GNU/Linux

I've asked a friend if they could reproduce this and it worked successfully on their machine Cython 0.29.13 Python 3.7.3 and with uname: Linux toshiba-inu 5.0.0-29-generic #31-Ubuntu SMP Thu Sep 12 13:05:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

So now I'm wondering what's wrong with my environment that causes this issues to flair up?

@scoder

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2019

I wonder why zipimport gets involved here. From the path, it doesn't look like the package gets installed as zip/egg archive, it seems to get unpacked. Could you verify that?

In any case, installing a package as zip/egg archive that contains binary modules is mostly useless because in order to import the extensions, Python has to unpack them into a temp directory and load them from there, which eats local storage space, messes up paths, etc. You should always declare such packages as zip_safe=False.

@mental32

This comment has been minimized.

Copy link
Author

commented Sep 29, 2019

@scoder The package does get installed as a egg, I re ran the install script and the output is virtually identical to what I uploaded earlier in a gist, note this line.

Out of paranoia I ran file over the egg and it came back identified as an archive:
/home/mental/.local/lib/python3.7/site-packages/foob-0.0.1-py3.7-linux-x86_64.egg: Zip archive data, at least v2.0 to extract

The setuptools docs dictate that a package will be assumed zip unsafe if there are any C extensions or datafiles found. I wanted to test this so I explicitly passed zip_safe=False into the setup calls kwargs and an egg was still produced.

This now seems like this is an issue with setuptools. I'm using 41.2.0.

Can anyone else reproduce this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.