BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909

refparo · 2023-06-09T18:11:15Z

Steps to reproduce:

Use pyenv to install python 3.12.0b2.
Clone numpy v1.25.0rc1.
pip install -r requirements
Uncomment build-backend = "mesonpy".
pip install --no-build-isolation ..

The OS I use is Archlinux, and openblas and cblas are installed. In case it is relevant, openblas.pc is present in /usr/lib/pkgconfig/.

Error message:

Traceback (most recent call last):
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: cblas_sgemm

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/__init__.py", line 125, in <module>
    from numpy.__config__ import show as show_config
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy.core._multiarray_umath import (
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.12 from "/home/paro/.pyenv/versions/3.12.0b2/bin/python"
  * The NumPy version is: "1.25.0rc1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: /home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: cblas_sgemm


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/__init__.py", line 130, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

Additional information:

ldd shows that the compiled library is not linking against BLAS at all.

~ $ ldd /home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffcdc3e1000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f0dfa084000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f0dfa05f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f0df9e75000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f0dfa59c000)

The text was updated successfully, but these errors were encountered:

thesamesam · 2023-06-12T10:36:15Z

Hit the same in Gentoo and was trying to minimise it but came across this! Thank you!

mattip · 2023-06-12T11:27:55Z

Meson builds for NumPy are still a WIP: the OpenBLAS interface detection and SIMD support is still in flux. Here are a few hints:

You should see, very early in the meson build, something like

    The Meson build system
    Version: 1.1.1
    Source dir: /project
    Build dir: /project/.mesonpy-xfjik4zm/build
    Build type: native build
    Project name: NumPy
    Project version: 2.0.0.dev0
    C compiler for the host machine: cc (gcc 10.2.1 "cc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)")
    C linker for the host machine: cc ld.bfd 2.35-5
    C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)")
    C++ linker for the host machine: c++ ld.bfd 2.35-5
    Cython compiler for the host machine: cython (cython 0.29.35)
    Host machine cpu family: x86_64
    Host machine cpu: x86_64
    Program python found: YES (/opt/python/cp310-cp310/bin/python)
    Found pkg-config: /usr/bin/pkg-config (0.27.1)
    Has header "Python.h" with dependency python-3.10: YES
    Compiler for C supports arguments -fno-strict-aliasing: YES
    Library m found: YES
    Found CMake: /usr/local/bin/cmake (3.26.3)
    WARNING: CMake Toolchain: Failed to determine CMake compilers state
    WARNING: CMake Toolchain: Failed to determine CMake compilers state
    Run-time dependency openblas found: NO (tried pkgconfig and cmake)
    WARNING: CMake Toolchain: Failed to determine CMake compilers state
    Run-time dependency openblas found: YES 0.3.23
    Dependency openblas found: YES 0.3.23 (cached)

Note the last two lines which show it is finding OpenBLAS via pkgconfig. If you see that, the build should link to OpenBLAS.

Additionally, there is a way to configure NumPy to use a build of OpenBLAS with 64-bit interfaces instead of 32-bit ones. This is called ILP64, and is currently expressed in a few environment variables and C macros (our convention is to add a 64_ suffix to all the exported symbols). It is unlikely that the system OpenBLAS is built with those symbols, so you should make sure you are not using -DBLAS_SYMBOL_SUFFIX=64_, -DHAVE_BLAS_ILP64, or NPY_USE_BLAS_ILP64="1"

thesamesam · 2023-06-12T11:30:58Z

I guess this is related to https://github.com/numpy/numpy/pull/23838/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R150 then, which at least I was applying.

Thanks for the hint, I'll look into it more now.

EDIT: I found what I was doing wrong -- I was passing -Dblas=blas instead of -Dblas=cblas (need the generic name in Gentoo as we have a different BLAS selection method) and my environment also seemed to have something broken which I've now fixed. Thanks!

mattip · 2023-06-12T12:04:17Z

If you really want the 64-bit interfaces like in our wheels, you can get OpenBLAS from https://anaconda.org/multibuild-wheels-staging/openblas-libs/files, expand the tarball, and point PKG_CONFIG_PATH to the pkgconfig directory. Make sure it works by checking that pkg-config openblas --cflags returns the path to the include files

refparo · 2023-06-12T14:21:07Z

I am now using the branch in gh-23838. My meson build does show these two lines:

  Run-time dependency openblas found: YES
  Dependency openblas found: YES unknown (cached)

It seems my system openblas doesn't indicate its version in the pkg-config file.

I am building with just pip install ., without any special flags. However it still doesn't link to openblas. What can I do to fix this?

mattip · 2023-06-12T14:33:10Z

How are you checking for OpenBLAS? Could you be picking up a static library?

refparo · 2023-06-12T15:42:15Z

I have the pkg-config file /usr/lib/pkgconfig/openblas.pc:

libdir=/usr/lib
libsuffix=
includedir=/usr/include/openblas

openblas_config=USE_64BITINT= NO_CBLAS=1 NO_LAPACK=1 NO_LAPACKE=1 DYNAMIC_ARCH=ON DYNAMIC_OLDER=OFF NO_AFFINITY=ON USE_OPENMP=1 ZEN MAX_THREADS=64
Name: OpenBLAS
Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
Version:
URL: https://github.com/xianyi/OpenBLAS
Libs: -fopenmp -L${libdir} -lopenblas${libsuffix}
Cflags: -I${includedir}

(the version is indeed left empty, although the system package manager says it's 0.3.23)

And the dynamic libraries libopenblas.so, libopenblas.so.0, libopenblas.so.0.3 are in /usr/lib/.

mattip · 2023-06-12T15:52:17Z

In build/build.ninja, under

build numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so

I see this

 LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,--start-group numpy/core/libnpymath.a /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libopenblas.so -Wl,--end-group

refparo · 2023-06-12T16:29:08Z

So... this means that I need to change the LINK_ARGS? But build.ninja is generated by meson. How do I change it?

mattip · 2023-06-12T16:49:11Z

My goal was to guide you to how you can debug the build. Do you see those link arguments? You can work backwards from there trying to figure out where blas/lapack detection is failing.

refparo · 2023-06-12T17:08:54Z

In my build meson actually finds the libopenblas.so. This is what I found in build.ninja.

 LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group numpy/core/libnpymath.a -fopenmp /usr/lib/libopenblas.so -Wl,--end-group

(By the way in my system (Archlinux) libcblas is installed separately and is a standalone dynamic library libcblas.so. The build doesn't seem to link to libcblas.so, while the prebuilt numpy for Python 3.11 from my system package manager links to it. Is this relevant?)

Update:

I realized that my system provides openblas as a drop-in replacement for netlib blas. So although I have openblas installed, maybe I actually need to build numpy to link to blas instead of openblas...?

Strangely, after I change openblas to blas in meson.build, the generated ninja file still uses libopenblas.so. The meson build message is now this:

  Run-time dependency blas found: YES
  Run-time dependency cblas found: YES 3.11.0
  Dependency blas found: YES unknown (cached)

But LINK_ARGS in build.ninja remains unchanged.

Update 2:

The content of blas.pc is the same as openblas.pc, so linking to openblas is expected. I still don't know why it doesn't link to openblas despite that LINK_ARGS seems correct. (is it? @mattip)

mattip · 2023-06-12T19:40:08Z

It is hard to debug this. You need to stare at the different options until they start to make sense. If the LINK_ARGS for numpy/core/_multiarray_umath.cpython-*.so look corrrect, then you should try looking at the compile args. Perhaps something is missing there.

refparo · 2023-06-13T06:08:04Z

So... it seems numpy does need to link to cblas. In my system, the symbol cblas_sgemm is exported by libcblas.so, not libopenblas.so. Although the meson build does define the cblas dependency variable, it doesn't actually use it in the module _multiarray_umath. I added cblas to the dependencies of _multiarray_umath and it worked.

mattip · 2023-06-13T07:14:40Z

Although the meson build does define the cblas dependency variable, it doesn't actually use it in the module _multiarray_umath. I added cblas to the dependencies of _multiarray_umath and it worked.

If this is something you think others will run into, it would be nice to summarize your changes and see how we can integrate them into the meson build.

refparo · 2023-06-13T08:05:44Z

I made two changes:

Use BLAS instead of OpenBLAS in meson.py. Maybe we should check if the user has cblas?

@@ -12,8 +12,8 @@ project(
     'b_ndebug=if-release',
     'c_std=c99',
     'cpp_std=c++17',
-    'blas=openblas',
-    'lapack=openblas',
+    'blas=blas',
+    'lapack=blas',
     'pkgconfig.relocatable=true',
   ],
 )

Add cblas to the dependencies of _multiarray_umath in numpy/core/meson.py. Because cblas is [] when there is no cblas, this will also work when there is no cblas.

@@ -859,7 +859,7 @@ py.extension_module('_multiarray_umath',
     'src/npymath',
     'src/umath',
   ],
-  dependencies: blas,
+  dependencies: [blas, cblas],
   link_with: npymath_lib,
   install: true,
   subdir: 'numpy/core',

rgommers · 2023-06-15T10:24:13Z

Yes, the cblas support is fairly incomplete - I'll work on that in main soon. I'll also note that Arch Linux is the only distro where OpenBLAS is built without LAPACK support, and that's not yet handled automatically. So you'll need Netlib LAPACK as a dependency, and build with -Dlapack=lapack.

rgommers · 2023-06-15T10:39:55Z

Add cblas to the dependencies of _multiarray_umath in numpy/core/meson.py. Because cblas is [] when there is no cblas, this will also work when there is no cblas.

I believe we unconditionally require CBLAS wherever we use BLAS, if BLAS is present and we're not using our own C implementation for matmul & co. So we may fix it more robustly by adding something like:

if have_blas
  blas_dep = declare_dependency(
    dependencies: [blas, cblas],
    compile_args: '-DHAVE_CBLAS',
  )
endif

and then everywhere else use dependencies: blas_dep.

refparo · 2023-06-15T12:29:54Z

So Archlinux just released a new version of OpenBLAS that includes cblas symbols and LAPACK support... Seems the modifications are no longer necessary.

However, if the user wants to use the package blas-openblas, the fore mentioned modifications are still useful.

rgommers · 2023-06-15T13:16:34Z

Ah, that is awesome, that solves a pain point - thanks for pointing that out @refparo!

In a conda env: ``` Run-time dependency openblas found: YES 0.3.23 Checking if "CBLAS" with dependency openblas: links: YES ``` On Arch Linux before the just-updated 0.23.23-2 build that included CBLAS and LAPACK in the `openblas` package (see numpygh-23909): ``` $ python -m build --wheel ... Run-time dependency openblas found: YES Checking if "CBLAS" with dependency openblas: links: NO Run-time dependency cblas found: YES 3.11.0 ... $ ldd numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007fff59235000) libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f5cb4b08000) libcblas.so.3 => /usr/lib/libcblas.so.3 (0x00007f5cb62b3000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f5cb4a1b000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f5cb628e000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f5cb4831000) libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007f5cb623d000) /usr/lib64/ld-linux-x86-64.so.2 (0x00007f5cb62f2000) ``` Closes numpygh-23909

rgommers · 2023-06-19T08:27:50Z

@refparo and @thesamesam this issue should be fixed by gh-23984. I tested on Arch Linux, not on Gentoo. There's more to do for the overall BLAS/LAPACK detection to make it >= numpy.distutils in functionality, but I hope this solves the immediate problem. If you could confirm, that would be much appreciated.

In a conda env: ``` Run-time dependency openblas found: YES 0.3.23 Checking if "CBLAS" with dependency openblas: links: YES ``` On Arch Linux before the just-updated 0.23.23-2 build that included CBLAS and LAPACK in the `openblas` package (see gh-23909): ``` $ python -m build --wheel ... Run-time dependency openblas found: YES Checking if "CBLAS" with dependency openblas: links: NO Run-time dependency cblas found: YES 3.11.0 ... $ ldd numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007fff59235000) libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f5cb4b08000) libcblas.so.3 => /usr/lib/libcblas.so.3 (0x00007f5cb62b3000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f5cb4a1b000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f5cb628e000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f5cb4831000) libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007f5cb623d000) /usr/lib64/ld-linux-x86-64.so.2 (0x00007f5cb62f2000) ``` Closes gh-23909

thesamesam · 2023-06-27T08:01:14Z

Thanks, I've got this on my queue to look at now. Will comment again with results.

refparo added the 32 - Installation Problems installing or compiling NumPy label Jun 9, 2023

mattip changed the title ~~undefined symbol: cblas_sgemm when building with meson with pip install~~ BLD: undefined symbol: cblas_sgemm when building with meson with pip install Jun 11, 2023

mattip added 36 - Build Build related PR Meson Items related to the introduction of Meson as the new build system for NumPy labels Jun 11, 2023

rgommers mentioned this issue Jun 18, 2023

Tracking issue: defaulting to Meson - follow-up tasks #23981

Closed

20 tasks

rgommers mentioned this issue Jun 19, 2023

BLD: improve handling of CBLAS, add -Duse-ilp64 build option #23984

Merged

seberg closed this as completed in #23984 Jun 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909

BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909

refparo commented Jun 9, 2023 •

edited

thesamesam commented Jun 12, 2023

mattip commented Jun 12, 2023

thesamesam commented Jun 12, 2023 •

edited

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023 •

edited

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023 •

edited

mattip commented Jun 12, 2023

refparo commented Jun 13, 2023 •

edited

mattip commented Jun 13, 2023

refparo commented Jun 13, 2023

rgommers commented Jun 15, 2023

rgommers commented Jun 15, 2023

refparo commented Jun 15, 2023 •

edited

rgommers commented Jun 15, 2023

rgommers commented Jun 19, 2023

thesamesam commented Jun 27, 2023 •

edited

BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909

BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909

Comments

refparo commented Jun 9, 2023 • edited

Steps to reproduce:

Error message:

Additional information:

thesamesam commented Jun 12, 2023

mattip commented Jun 12, 2023

thesamesam commented Jun 12, 2023 • edited

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023 • edited

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023

mattip commented Jun 12, 2023

refparo commented Jun 12, 2023 • edited

mattip commented Jun 12, 2023

refparo commented Jun 13, 2023 • edited

mattip commented Jun 13, 2023

refparo commented Jun 13, 2023

rgommers commented Jun 15, 2023

rgommers commented Jun 15, 2023

refparo commented Jun 15, 2023 • edited

rgommers commented Jun 15, 2023

rgommers commented Jun 19, 2023

thesamesam commented Jun 27, 2023 • edited

refparo commented Jun 9, 2023 •

edited

thesamesam commented Jun 12, 2023 •

edited

refparo commented Jun 12, 2023 •

edited

refparo commented Jun 12, 2023 •

edited

refparo commented Jun 13, 2023 •

edited

refparo commented Jun 15, 2023 •

edited

thesamesam commented Jun 27, 2023 •

edited