Package numpy wheel with openblas

[mgiessing]

I'm trying to create a numpy wheel for ppc64le architecture and directly integrate openblas into it, so the user just needs to pip install numpy. However the problem is that I need to install libopenblas-dev additionally to make it work (e.g. apt-get install libopenblas-dev)

Build the wheel from source

I'm building the wheel from source with the command:

python3 setup.py bdist_wheel

and having a site.cfg like this:

[default]
include_dirs = /opt/openblas/include
library_dirs = /opt/openblas/lib
 
[openblas]
openblas_libs = openblas
library_dirs = /opt/openblas/lib
 
[lapack]
lapack_libs = openblas
library_dirs = /opt/openblas/lib

Openblas is installed under /opt/openblas

The wheel gets build successfully without errors.

Error message (when importing numpy):

With the x86 numpy wheel it seems that everything is already integrated in the numpy wheel (from pypi)

root@e64ff60c5a8f:/# uname -m
x86_64
root@e64ff60c5a8f:/# pip install numpy==1.19.5
Collecting numpy==1.19.5
  Downloading numpy-1.19.5-cp37-cp37m-manylinux2010_x86_64.whl (14.8 MB)
     |████████████████████████████████| 14.8 MB 4.6 MB/s 
Installing collected packages: numpy
Successfully installed numpy-1.19.5
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
root@e64ff60c5a8f:/# python
Python 3.7.10 (default, Apr 10 2021, 16:02:25) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'1.19.5'

On ppc64le:

root@4ff3de8a0ba0:/# uname -m
ppc64le
root@4ff3de8a0ba0:/# pip install numpy==1.19.5
Looking in indexes: https://pypi.org/simple, https://repo.fury.io/mgiessing
Collecting numpy==1.19.5
  Downloading https://repo.fury.io/mgiessing/-/ver_1CFCoQ/numpy-1.19.5-cp37-cp37m-linux_ppc64le.whl (12.5 MB)
     |████████████████████████████████| 12.5 MB 2.7 MB/s 
Installing collected packages: numpy
Successfully installed numpy-1.19.5
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
root@4ff3de8a0ba0:/# python
Python 3.7.10 (default, May 12 2021, 13:30:24) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/usr/local/lib/python3.7/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/local/lib/python3.7/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/numpy/__init__.py", line 140, in <module>
    from . import core
  File "/usr/local/lib/python3.7/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "/usr/local/bin/python"
  * The NumPy version is: "1.19.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libopenblas.so.0: cannot open shared object file: No such file or directory

This can be fixed by apt-get install libopenblas-dev, but ideally I'd like to package everything into the numpy wheel.
I can also see, that the x86 wheel is slightly bigger than teh ppc64le wheel (14,8 MB vs 12,5 MB).

Hope anyone can help me out here!

[mattip]

The wheels we ship are fixed up using the auditwheel tool to pull in all the correct shared objects. It is roughly

python -m pip wheel .
auditwheel repair dist/numpy*.whl

[mgiessing]

Thanks - I got it working, but it seems there are some issues with the ppc64le platform tags.
auditwheel repair --plat manylinux_2_27_ppc64le dist/*.whl was able to produce a wheel that integrates all the required shared objects

[mattip]

if you want a manylinux2014 tag (which is a synonym for manylinux_2_17), you need to build on a centos7 host. If you are using ubuntu 18.04, you will get a manylinux_2_27 tag. This page nicely summarizes the different distros and tags.