Pyinstaller failing on numcodecs blosc import

[timdnewman]

Minimal, reproducible code sample, a copy-pastable example if possible

from aicsimageio import AICSImage

then

pyinstaller -F -w main.py

Problem description

I'm using aicsimageio for loading CZI files and I'm trying to package it up into a stand-alone app for one of our EngD students to use.

The only reference I could find to my exact problem is this:
napari/napari#665 (comment)

And I couldn't find the ticket. So I thought I would raise it.

The issue is that the app works when run as a script, the app functions completely as expected. When I package it up I get the following error:

Traceback (most recent call last):
File "main.py", line 3, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "control_scheme_python.py", line 11, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "chunking_logic.py", line 6, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "aicsimageio_init_.py", line 6, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "aicsimageio\aics_image.py", line 10, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "xarray_init_.py", line 1, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "xarray\tutorial.py", line 13, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "xarray\backends_init_.py", line 17, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "xarray\backends\zarr.py", line 22, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "zarr_init_.py", line 2, in
File "PyInstaller\loader\pyimod03_importers.py", line 476, in exec_module
File "zarr\codecs.py", line 3, in
ImportError: cannot import name 'Blosc' from 'numcodecs' (C:\Users\iris\AppData\Local\Temp_MEI63202\numcodecs_init_.pyc)

I guess the expected behavior is: it runs.

I had a c++ installer (MS VS build tools 2019) and the build from pip for aicsimageio ran without error - including seeing the numcodecs build.

The most desirable and quickest solution I'm hoping for is that someone points out a stupid mistake I've made and I fix it easily :)

Version and installation information

Please provide the following:

• Value of numcodecs.__version__ = 0.9.1
• Python 3.9.7
• Win 10 19042.1348
• using pip into a clean VM spun up exclusively for this
  bug_requirements.txt

I've had it on other versions of python and conda too.

Thanks

[jakirkham]

Does it work for you without PyInstaller?

[timdnewman]

Yes - although I don't know if blosc is being called directly but it's the exact same code so I think the import should be firing.

[jakirkham]

There's also a fair amount of indirection in the traceback above. Can we reproduce this with just Zarr or maybe better just Numcodecs?

[timdnewman]

I'll have a go tomorrow - I'm not actually familiar with numcodecs but I'll cobble something together.

[jakirkham]

Sounds good :)

For clarity am suggesting something like import numcodecs.blosc.

Anyways just trying to cull this reproducer down into something where it is easier to pinpoint where things are going wrong

[timdnewman]

from numcodecs import blosc

Runs no errors but via pyinstaller I get

Traceback (most recent call last):
File "just_blosc.py", line 1, in
File "numcodecs/blosc.pyx", line 1, in init numcodecs.blosc
ModuleNotFoundError: No module named 'numcodecs.compat_ext'

[timdnewman]

in conda and another clean install of vanilla python 3.9.7

[jakirkham]

So it sounds like Blosc did not get packaged in this case, but the data being loaded requires it. Does that sound about right?

[timdnewman]

Well yes and no - in the minimal example it just imports it, that is the whole program. Maybe since it isn't used in the one I just run that isn't a problem, but packaging requires them all? I'm not an expert on either I'm afraid.

But broadly yes, I guess it that is the case, I think they get used in loading CZIs

[joshmoore]

cc: @JacksonMaxfield especially re: napari/napari#665 (comment)

[evamaxfield]

I don't believe we use numcodecs for CZI loading..... We don't even list numcodecs as a dependency for aicsimageio: https://github.com/AllenCellModeling/aicsimageio/blob/main/setup.py#L88

Which means it's coming from zarr I would assume? https://github.com/AllenCellModeling/aicsimageio/blob/main/setup.py#L99

Do I need to update that zarr version pin?

[joshmoore]

Do I need to update that zarr version pin?

Hmmm.... possibly. Or at least to add a matching numcodecs pin. Do you know why the pin is in place? Is it for matching Python versions?

[evamaxfield]

Why the zarr pin is there? Because we need it for chunked tifffile loading.

[joshmoore]

Ah, but >= should be fine unsure what Zarr version was used here. Perhaps something specifically Windows-esque?

[evamaxfield]

from numcodecs import blosc

Runs no errors but via pyinstaller I get

Traceback (most recent call last): File "just_blosc.py", line 1, in File "numcodecs/blosc.pyx", line 1, in init numcodecs.blosc ModuleNotFoundError: No module named 'numcodecs.compat_ext'

From this earlier comment it does seem more like a packaging issue on windows or something similar.

[timdnewman]

Ah, but >= should be fine unsure what Zarr version was used here. Perhaps something specifically Windows-esque?

@joshmoore 2.10.3 if that helps.

[DavidStirling]

Extremely late to the party here, but I believe you can fix the PyInstaller builds by adding the list generated by PyInstaller.utils.hooks.collect_submodules("numcodecs") to the hidden imports in the pyinstaller .spec file. This should force it to package those codecs (including blosc) even if they're not obviously called within your program.

I've not tested the specific example above, but having --hidden-import numcodecs.blosc in your command line call may also be sufficient in that case.

[jakirkham]

Thanks for chiming in David! 😄

Is there anything special we need to do with PyInstaller to make sure the shared objects in Numcodecs are handled correctly?

[DavidStirling]

I'm not familiar with how numcodecs works, but PyInstaller should pack the entire thing if using the hook method above. Looking at the package I don't see anything within numcodecs that would obviously cause problems.

Missing imports are a common theme with PyInstaller, and might be expected in situations where your program could need an ambiguous codec to read a file but it's never explicitly imported within your code. Good luck!

[jakirkham]

Thanks again David! 🙏

[timdnewman]

Thanks for getting back - now I'm back from holiday I'll give this a try and report back (although it might take a while)

[psavery]

Extremely late to the party here, but I believe you can fix the PyInstaller builds by adding the list generated by PyInstaller.utils.hooks.collect_submodules("numcodecs") to the hidden imports in the pyinstaller .spec file. This should force it to package those codecs (including blosc) even if they're not obviously called within your program.

I've not tested the specific example above, but having --hidden-import numcodecs.blosc in your command line call may also be sufficient in that case.

Adding all of the numcodecs submodules as hidden imports fixed the issue for me! Perhaps an official pyinstaller hook should be created for numcodecs?

[psavery]

In fact, it looks like this works for the hook file (numcodecs-hook.py):

from PyInstaller.utils.hooks import collect_submodules

hiddenimports = collect_submodules("numcodecs")

If you put that file in your current directory, and add --additional-hooks-dir . to the pyinstaller command, numcodecs appears to be packaged correctly. At least for my local example.

[jakirkham]

It would be good to understand why that hook is needed. IOW is there something we should be doing in the packaging of Numcodecs to obviate the need for that hook?

[DavidStirling]

In my testing Pyinstaller typically missed .compat_ext. As noted in the docs, the analysis process will pick up imports of compiled .pyx files, but won't follow imports which are in the .pyx files themselves. .compat_ext turns out to be a good example of this scenario and so needs to be included as a hidden import.

I believe Pyinstaller provides some methods for declaring these imports for them within your package, but I've not tried it myself.

[jakirkham]

Interesting. Good to know that compat_ext is the module with issues. No other modules had issues?

FWIW compat_ext is the only one that supplies a .pxd file. Not sure if that is relevant. Maybe worth digging into further.

PyInstaller includes a number of hooks itself. So another option may be for someone interested in this to add the hook upstream.

[DavidStirling]

I can't confirm whether other modules exhibit the same issue, it's one of those things where it was faster to force bundle the whole package than to repeatedly build my app and tweak until every hidden import is discovered. It's not a large package so I wasn't overly concerned about packaging extra content.

[psavery]

Interesting. Good to know that compat_ext is the module with issues. No other modules had issues?

FWIW compat_ext is the only one that supplies a .pxd file. Not sure if that is relevant. Maybe worth digging into further.

PyInstaller includes a number of hooks itself. So another option may be for someone interested in this to add the hook upstream.

I tried having a single hidden import of numcodecs.compat_ext, and it seemed to work. I guess that is the only module that pyinstaller misses, and it is missed because it is only imported in pyx files.

I added the hook as a PR, which should fix this issue: pyinstaller/pyinstaller-hooks-contrib#420

That pyinstaller-hooks-contrib package is automatically installed as well when pyinstaller is installed.

[psavery]

Hi all. pyinstaller/pyinstaller-hooks-contrib#420 was merged. I gave it a try on my local example, and it seems to have fixed the issue!

Until a new release comes out, I guess we can install from master via a command like pip install git+https://github.com/pyinstaller/pyinstaller-hooks-contrib.

[jakirkham]

Thanks @psavery! 😄