Packaging spacy with pyinstaller: ImportError: cannot import name _custom_kernels

[michaelhochleitner]

Overview

I'm packaging a python script which imports spacy with pyinstaller https://github.com/pyinstaller/pyinstaller into an executable. I get the error in the title when running the executable.

Environment

• Operating System: Linux-4.15.0-88-generic-x86_64-with-Ubuntu-18.04-bionic
• Python Version Used: 3.6.9
• spaCy Version Used: 2.2.3
• Spacy location: /home/user/spacy-pyinstaller/venv/lib/python3.6/site-packages/spacy
• pyinstaller Version Used: 4.0.dev0+a1f92c6a08

Reproduction steps

All necessary files for reproduction are available at https://github.com/michaelhochleitner/spacy-pyinstaller .

I'm packaging the following python script.

import spacy

I use the following command to package the script.

pyinstaller nlp.spec 

Here is my .spec file:

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['nlp.py'],
             pathex=['/home/user/spacy-pyinstaller'],
             binaries=[],
             datas=[],
             hiddenimports=['srsly.msgpack.util','cymem','cymem.cymem','preshed.maps','thinc.linalg','murmurhash'],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='nlp',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='nlp')

When running

./dist/nlp/nlp

I get the following output

Traceback (most recent call last):
  File "nlp.py", line 1, in <module>
    import spacy
  File "/home/mh/spacy-pyinstaller/venv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 489, in exec_module
    exec(bytecode, module.__dict__)
  File "spacy/__init__.py", line 10, in <module>
  File "/home/mh/spacy-pyinstaller/venv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 489, in exec_module
    exec(bytecode, module.__dict__)
  File "thinc/neural/__init__.py", line 4, in <module>
  File "/home/mh/spacy-pyinstaller/venv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 489, in exec_module
    exec(bytecode, module.__dict__)
  File "thinc/neural/_classes/model.py", line 11, in <module>
  File "/home/mh/spacy-pyinstaller/venv/lib/python3.6/site-packages/PyInstaller/loader/pyimod03_importers.py", line 489, in exec_module
    exec(bytecode, module.__dict__)
  File "thinc/neural/train.py", line 7, in <module>
  File "optimizers.pyx", line 14, in init thinc.neural.optimizers
  File "ops.pyx", line 24, in init thinc.neural.ops
ImportError: cannot import name _custom_kernels
[16858] Failed to execute script nlp

I don't know if this issue should be in the pyinstaller project or the spacy project. So I'm opening issues in both.

Related issues

pyinstaller/pyinstaller#4696
#2536
#3831
https://stackoverflow.com/questions/59645155/spacy-2-2-3-filenotfounderror-errno-2-no-such-file-or-directory-thinc-neur

[Legorooj]

@michaelhochleitner this can be closed - it's an issue with PyInstaller, not spacy. I've left an answer on the SO question.

[svlandeg]

Thanks for the note, @Legorooj !

[Legorooj]

@svlandeg you're welcome. All spacy needed is what is called a PyInstaller "hook", which tells PyInstaller how to package the library. I've written up one that's a bit of a hack, and posted that on SO. Hopefully it'll work without one soon; I've got a PR on the way that will massively decrease the need for hooks.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.