New numpy hook for intel mkl libraries

[maqifrnswa]

Hello - anaconda and winpython build numpy against Intel's mkl libraries. Building someone that uses numpy will need to have those libraries.

Here is a winpython hook:
http://stackoverflow.com/a/35853001

and I adapted that for anaconda & python3:
https://github.com/maqifrnswa/scimpy/blob/master/pyinstaller-hooks/hook-numpy.py

[matysek]

@maqifrnswa thanks, could you please create a pull request from those two?

• combine them into one
• use if is_darwin to choose the right path for OSX and other OSes

[clacri]

Hi,
I am using pyinstaller 3.1 in conda 4.0.4 to generate a onefile executable for a program in python that needs multiple scientific libraries, such as numpy or scipy. Previously we had all libraries installed directly on the machine, but we have switched to conda for practical reasons and after the machine in which we were compiling had to be reinstalled. However, in this new setup, on some machines, at runtime, the executable produced raises an intel mkl error, previously reported for numpy:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Both libraries are inside the anaconda installation, in the lib folder, but they are not being included in the wrapped executable, and if the program is searching for them in the machine it is not having success. Looking for a solution I found this thread, I guess that what I will need to try is to use the hook provided by maqifrnswa?

[clacri]

Hi, I did try that, using the hook described as runtime-hook, but got the following error:

Traceback (most recent call last):
File "", line 9, in
File "/cri4/claudia/anaconda/envs/arcimboldo/lib/python2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
File "site-packages/PyInstaller/init.py", line 61, in
File "site-packages/pkg_resources.py", line 336, in get_distribution
File "site-packages/pkg_resources.py", line 215, in get_provider
File "site-packages/pkg_resources.py", line 696, in require
File "site-packages/pkg_resources.py", line 594, in resolve
pkg_resources.DistributionNotFound: PyInstaller
hook_numpy returned -1

I see in the log that it has included it (18880 INFO: Including custom run-time hook 'hook_numpy.py'), but the error happens each time and I see many missing modules from numpy in the warnings folder. I read the answer on http://stackoverflow.com/questions/35478526/pyinstaller-numpy-intel-mkl-fatal-error-cannot-load-mkl-intel-thread-dll and one of them suggest that problems when using the hook might come from using different Python package management schemes. In my case, some of the packages where installed directly by conda and have a built string of py27_*, some others with pip, and in particular, biopython, matplotlib and scipy have one called np110py27. Could that be a problem?

[clacri]

I have also tried to include the libraries as binaries in the spec file and then compile from the spec, but it keeps happening. Someone in the scikit-learn repo told me that this could be related with the numpy version, as mentioned in scikit-learn/scikit-learn#6508 (comment). However, I do have the version 1.10.4, so apparently that should not be a problem.

[clacri]

Just for the record, browsing through github scikit-learn project issues, I found the post at Anaconda blog where they describe how to inactivate the mkl versions of the libraries (https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations) and now I can compile a working version. I don't think this is a problem at all with pyinstaller but with some of the libraries I call, so sorry for all the mess! At least I have learnt about other Pyinstaller options that I had never used, such as the binaries addition or the runtime or linktime hooks. I am pretty new to Pyinstaller so that was good :) Thanks anyway!

[maqifrnswa]

@clacri Thanks for those links, that is very helpful! I didn't know about turning off mkl optimizations.

@matysek Looking through the Anaconda EULA https://docs.continuum.io/anaconda/eula it says re-distribution is allowed, as long as it is redistributed "with anaconda." Since an app frozen with PyInstaller includes a subset of the anaconda distribution (and thus is a derivative of anaconda), redistribution of mkl should be acceptable as long as the terms of the anaconda license are also met. Is there some document in PyInstaller to let users of specific distributions know that there are licensing terms they need to follow? If so I can update it to include info on the mkl libraries. I hope to update and test hook within a couple of days.

(edits for clarification are in bold)

[htgoebel]

@maqifrnswa IMHO this EULA is not quite clear about the mkl-library licence Would you please be so kind clarifying this with continum.io and issue a pull request? Thanks

BTW: This is plain wrong:

Since PyInstaller includes a subset of the anaconda distribution (and thus is a derivative of anaconda),

The frozen application does include "a subset of the anaconda distribution", but not PyInstaller. PyInstaller is only the tool or creating this subset.

[maqifrnswa]

Thank you - that's what I meant, the pyinstaller application includes a distribution - not pyinstaller. I've edited my previous statement to be clear. I'll do the pull request and ask for clarification from continuum. Busy with day job, might not get to the pull request for a couple of days.

[pwuertz]

I'm also having trouble bundling numpy from the anaconda distribution due to missing MKL runtime libraries.
For the numpy hook numpy.__config__ / numpy.__config__.show() could be used to detect if numpy is built with MKL support.

[bjones1]

What platform? The development version of Pyinstaller should now bundle these on Windows.

[pwuertz]

The current numpy.core hook in PyInstaller only looks for DLLs in the numpy folder, but it doesn't install the MKL DLLs from the python /Library/bin folder. MKL seems to load those on demand. When calling certain numpy functions a bundled application crashes with the message

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Platform is Windows. Tested with current Anaconda distribution and the PyInstaller develop branch.

[bjones1]

On Windows, MKL is looking for .so files? That makes no sense...

[pwuertz]

Ah sorry, I just lazily copy pasted the error message from one of the first comments above. But the same problem occurs when bundling from Anaconda for windows, the library names are just different of course (mkl_avx2.dll and mkl_def.dll, also located in Library/bin).

[bjones1]

I'm hacking up a bit of code that might help. Would you be able to test when it's ready?

[pwuertz]

I hacked up this numpy.core hook, which is based on your work for fixing the Pandas issue (#1969). Additionally to searching the numpy.core directory, I added a search for MKL libs in Library\bin. This works for Anaconda on Windows and shouldn't break anything this hook has achieved before.

Edit: Proposing a PR with a modified numpy.core hook that should work on windows and linux alike

[pwuertz]

Proposed fix: #2111
Inviting @maqifrnswa @clacri to try the new hook if still interested

[clacri]

Sure, I'll be happy to test it. What should I do, download the latest development version of pyinstaller and try to compile, but this time with numpy using MKL in Anaconda?

[stephenrauch]

@clacri

I think the following will do a pip install at the specific commit

pip install https://github.com/pyinstaller/pyinstaller/archive/13458ec7d74e9665cae14a8a91da7adde5db66e8.zip

[clacri]

Hi, I did install the version @stephenrauch provided, in a Linux system. I did uninstall the nomkl version of the packages affected (numpy, matplotlib, scipy, etc) and reinstall the mkl ones through Anaconda. However, the binary compiled keeps throwing an error when the program is running:

Intel MKL FATAL ERROR: Cannot load libmkl_mc3.so or libmkl_def.so

[pwuertz]

Ok, I see. Just downloaded the anaconda distribution for Linux and the folder layout is different. Instead of "Library/bin" we need "lib" here. I'm wondering if there is a way to detect the dynamic linker search path from within python..

[bjones1]

Does this help?

http://conda.pydata.org/docs/building/shared-libraries.html

[pwuertz]

It does, thank you! So, in conclusion: Finding dynamic libraries automatically is "complicated".

Here is one more try for @clacri. This time with a is_win switch that decides between Libraries/bin or lib for searching. Judging from the layout I've seen here this "should" work now.

Is this pip command sufficient? I remember some upgrade option one had to append for convincing pip to actually do anything. Seems to be gone nowdays..

pip install https://github.com/pwuertz/pyinstaller/archive/66ea18c248bb4f2ba0537a8af13272600efac675.zip

[clacri]

Hi,
@pwuertz yes, I needed to add the upgrade option, at least if the name of the version is the same, which is the case with the second link you gave me.
I have recompiled with this version, and now is working :)

[pwuertz]

Great!

Squashed the commits again in PR #2111. There are some automated test failures, but I assume this is unrelated to the numpy hook change?