No module named 'numba.cloudpickle.cloudpickle_fast'

[Nee-DS]

I am using Pyinstaller to create an exe for my Bertopic model pipeline. I do not have any issues when I create and execute the exe in my VM GPU. Following is the error I am getting when the pipeline runs in the Azure DevOps pipeline. Shall I contact Azue, Pyinstaller, or here? Do not know. Please guide

File "joblib\numpy_pickle.py", line 648, in load
File "joblib\numpy_pickle.py", line 577, in _unpickle
File "pickle.py", line 1212, in load
File "pickle.py", line 1589, in load_reduce
File "numba\core\serialize.py", line 110, in _unpickle__CustomPickled
ModuleNotFoundError: No module named 'numba.cloudpickle.cloudpickle_fast'

Versions:
Pyinstaller 6.4.0
Python 3.9
Windows

[rokm]

Have you tried adding numba.cloudpickle.cloudpickle_fast to hidden imports? (That's assuming the error is being thrown from the frozen application, and not some subprocess-spawned external script).

[Nee-DS]

I just included it, but having the same error.

[rokm]

Is the error thrown from frozen application or external script? What is the complete error message (from the start of traceback to the end of output)?

[Nee-DS]

I suppose it is from the frozen application and not from the script because it is working in the VM. I am now working on the yml file (https://faun.pub/azure-pipelines-and-python-virtual-environments-fdc1089656db) to make sure it gets the same environment as it gets in the VP. I am supposing it is something related to Azurepipeline. Let me know if there is any other thoughts to it.

[rokm]

OK, just to clarify: when you say it works in VM but not on Azure - do you build in VM and that VM-made build works in VM but not on Azure? Or do you build in VM and that VM-made build runs in VM, but if you build on Azure, that Azure-made build ends up raising numba.cloudpickle.cloudpickle_fast' (both if you run it on Azure or in VM)?

[Nee-DS]

Yes, I build in VM. So I have created a VM through the EC2 instance, and all the process is done in it. The virtual environment is created, the list of libraries in the txt file in installed and the pyinstaller makes the exe and it runs successfully.
The same steps when done in Azure pipeline it does not work. I am still working on the yml file so that the build process takes the virtual environment created and not the global variables. Thank you

[Nee-DS]

To answer your previous question -
[Ques]: do you build in VM and that VM-made build works in VM but not on Azure?
[NS]: The VM-made build runs in the VM and the local machine, I am not running the exe in Azure as of now.

[Ques]: Or do you build in VM and that VM-made build runs in VM, but if you build on Azure, that Azure-made build ends up raising numba.cloudpickle.cloudpickle_fast'?
[NS]: Yes when I build in VM and that VM-made build runs in VM, but when I build on Azure(ADO pipeline), that Azure-made build ends up raising numba.cloudpickle.cloudpickle_fast'

Also, I have modified the YML file to specify the Python version which is the same as in my VM, but still the the build made in Azure doesn't work in my local or VM and fails with the same error.

[Nee-DS]

Hi, I am in touch with numba team too and the following is the comment I got [https://github.com/numba/numba/issues/9500]
Could you please have a look into it too? Thanks

[rokm]

Like they said, this will require debugging and comparing a working and non-working build to see what the difference is, and where the difference comes from.

I can look into it if you can provide both a working and non-working build sample - i.e., if you can archive both the build and dist directory from your VM-generated build and your Azure-generated build, and upload the archives somewhere so I can take a look at them. Or, at least the build directory if you cannot share the final executables and the dist directory. Otherwise, you're on your own.

[Nee-DS]

Sure, Thanks! I will try Google Drive.

[rokm]

Thanks for sharing the build directories. This confirms that your working and non-working builds were made in different environments (different versions of python, as well as different versions of python packags), and this leads to different behavior.

Specifically, the working build was made with numba 0.58.1 or earlier, where vendored numba.cloudpickle package was directly importing the numba.cloudpickle.cloudpickle_fast module - therefore, this module was picked up by our import analysis, and collected into frozen application.

The non-working build was made with numba 0.59 or later, where vendored numba.cloudpickle was updated to v3.0.0; this version keeps cloudpickle_fast module around as a compatibility shim for unpickling data that was pickled with earlier versions. However, it is not directly imported from anywhere, and therefore our import analysis has no awareness of it.

Like I said in the very first comment, adding numba.cloudpickle.cloudpickle_fast to hidden imports should ensure that this module is collected and should fix the problem. If it does not work for you, please show how you tried to add the module to hidden imports (i.e., paste your PyInstaller command if you are building directly from .py file, or paste your spec file if you are building from .spec file).

[Nee-DS]

Reducing numba version to 0.58.1 worked :-)
I have extra hooks python file to include hidden imports. But reducing the numba to 0.58.1 worked and I did not had to include numba.cloudpickle.cloudpickle_fast in it.