New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: pip install scipy from git fails with "openblas" not found
#16308
Comments
Indeed, just We just switched build systems yesterday, so while our build requirements didn't change (you need BLAS/LAPACK), the way of detecting them did change. Can you try adding |
So how did it previously succeed? I thought NumPy had a fallback for this or is it that SciPy doesn't support that fallback? Except it previously worked and the test suite completed... To be clear there is no need from SymPy's end to support this particular setup. Maybe though if we noticed this change then it might break something significant for someone else. |
We're no longer using
It's expected that packagers will need to change a few things in build scripts, that is unavoidable. And a small price to pay to move away from |
But also this is one of the items at #16293 and the goal is to teach Meson some of these fallback detection methods. :) |
Yes, that fixes it. At the time of writing tests haven't run yet but scipy built successfully: I guess this can be closed then? |
Tests are now complete. Just one failure due to #16042 (comment) |
Great, thanks Oscar!
I'm expecting the same question a bunch more times, so let's leave this open for now so the issue and solution are more discoverable. |
"openblas" not found
in Python 3.11b1"openblas" not found
Okay, I edited the title since I guess the Python version is not relevant and maybe numpy isn't either... |
Got this issue recently as well, but on CentOS 7 instead. |
You probably then have to point |
My concern is more on why cmake didn't find it there, as it's capable of picking up both /usr and /usr/local.
|
Similar build script works on Debian11/Ubuntu18.04, and their |
Steps off of soapbox
|
Yes I agree with that point, we also find which not installed in many small or embedded environment or points to weird things. But on those popular distros which is quite convenient and gradually converges to similar behavior over time.
Yes, it's actually right on top of that error msg, not sure how I even missed that.
So it took the right cmake, but failed to find openblas. Then I run meson manually to check the failure (would be nice if the meson build log can be printed or at least kept on disk after a failed |
Log for cmake hit with OpenBLAS:
|
Also confirmed "OpenBLAS" cannot be used with "openblas.pc", so scipy should add some case-insensitive routing. |
I think
SciPy's current minimum version of Meson supports In the long run, as mentioned above, the idea is to contribute custom detection code to Meson, to smooth over these differences. |
No it doesn't.
Maybe the other way? |
… finding OpenBLAS. - scipy/scipy#16308
It should be CMake second; the log ended up there because it's the first version that actually worked, right? Adding |
Maybe? Not sure. |
This is a fallback detection path, in case pkg-config fails to find `openblas`. CMake is case-sensitive, so we need to try the camelcase version of the name. If CMake is installed and it finds OpenBLAS, the result looks like: ``` Determining dependency 'OpenBLAS' with pkg-config executable '/home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config' env[PKG_CONFIG_PATH]: Called `/home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config --modversion OpenBLAS` -> 1 CMake binary for 1 is cached. Determining dependency 'OpenBLAS' with CMake executable '/home/rgommers/anaconda3/envs/scipy-dev/bin/cmake' Try CMake generator: auto Calling CMake (['/home/rgommers/anaconda3/envs/scipy-dev/bin/cmake']) in /home/rgommers/code/scipy/build/meson-private/cmake_OpenBLAS with: - "-DNAME=OpenBLAS" - "-DARCHS=libpyldb-util.cpython-310-x86-64-linux-gnu.so;libpyldb-util.cpython-310-x86-64-linux-gnu.so.2;libpyldb-util.cpython-310-x86-64-linux-gnu.so.2.5.0;libpytalloc-util.cpython-310-x86-64-linux-gnu.so;libpytalloc-util.cpython-310-x86-64-linux-gnu.so.2;libpytalloc-util.cpython-310-x86-64-linux-gnu.so.2.3.3;libsamba-policy.cpython-310-x86-64-linux-gnu.so;libsamba-policy.cpython-310-x86-64-linux-gnu.so.0;libsamba-policy.cpython-310-x86-64-linux-gnu.so.0.0.1" - "-DVERSION=" - "-DCOMPS=" - "--trace-expand" - "--trace-format=json-v1" - "--no-warn-unused-cli" - "--trace-redirect=cmake_trace.txt" - "-DCMAKE_TOOLCHAIN_FILE=/home/rgommers/code/scipy/build/meson-private/cmake_OpenBLAS/CMakeMesonToolchainFile.cmake" - "." - "-DCMAKE_PREFIX_PATH=/home/rgommers/anaconda3/envs/scipy-dev;/home/rgommers/anaconda3/envs/scipy-dev/x86_64-conda-linux-gnu/sysroot/usr" using old-style CMake variables for dependency OpenBLAS Include Dirs: ['/home/rgommers/anaconda3/envs/scipy-dev/include'] Compiler Definitions: [] Libraries: ['/home/rgommers/anaconda3/envs/scipy-dev/lib/libopenblas.so'] Run-time dependency openblas found: YES 0.3.20 ``` This follows up on comments in scipygh-16308
gh-16750 improves the fallback detection. I've tested that indeed |
I just got this on my windows machine. I know, on plain windows we use wheels, and we should not have this issue, but some user (me) has gotten used to running wsl for some specific tools, but also needs python available from windows it self - so preferred to install my most used modules everywhere, making the python scripts available from all terminals... however:Microsoft Windows [Version 10.0.19044.2604]
(c) Microsoft Corporation. All rights reserved.
C:\Users\redacted>pip install scipy
Collecting scipy
Downloading scipy-1.10.1.tar.gz (42.4 MB)
---------------------------------------- 42.4/42.4 MB 9.0 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [44 lines of output]
+ meson setup --prefix=C:\Users\redacted\AppData\Local\Programs\Python\Python310-32 C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619 C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-hdayg0pn\build --native-file=C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
The Meson build system
Version: 1.0.1
Source dir: C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619
Build dir: C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-hdayg0pn\build
Build type: native build
Project name: SciPy
Project version: 1.10.1
C compiler for the host machine: cc (gcc 12.2.0 "cc (Rev6, Built by MSYS2 project) 12.2.0")
C linker for the host machine: cc ld.bfd 2.39
C++ compiler for the host machine: c++ (gcc 12.2.0 "c++ (Rev6, Built by MSYS2 project) 12.2.0")
C++ linker for the host machine: c++ ld.bfd 2.39
Cython compiler for the host machine: cython (cython 0.29.33)
Host machine cpu family: x86
Host machine cpu: x86
Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Compiler for C supports arguments -Wno-unused-function: YES
Compiler for C supports arguments -Wno-conversion: YES
Compiler for C supports arguments -Wno-misleading-indentation: YES
Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
Library m found: YES
Fortran compiler for the host machine: gfortran (gcc 12.2.0 "GNU Fortran (Rev6, Built by MSYS2 project) 12.2.0")
Fortran linker for the host machine: gfortran ld.bfd 2.39
Compiler for Fortran supports arguments -Wno-conversion: YES
Checking if "-Wl,--version-script" : links: YES
Program cython found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-d0rw0g37\overlay\Scripts\cython.EXE)
Program python found: YES (C:\Users\redacted\AppData\Local\Programs\Python\Python310-32\python.exe)
Program pythran found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-d0rw0g37\overlay\Scripts\pythran.EXE)
WARNING: You should add the boolean check kwarg to the run_command call.
It currently defaults to false,
but it will default to true in future releases of meson.
See also: https://github.com/mesonbuild/meson/issues/9300
Run-time dependency threads found: YES
Library npymath found: YES
Library npyrandom found: YES
Found pkg-config: C:\msys64\usr\bin\pkg-config.EXE (1.8.0)
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
Run-time dependency openblas found: NO (tried pkgconfig)
..\..\scipy\meson.build:134:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig
A full log can be found at C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-hdayg0pn\build\meson-logs\meson-log.txt
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
C:\Users\redacted>pip install --upgrade wheel
Collecting wheel
Downloading wheel-0.40.0-py3-none-any.whl (64 kB)
---------------------------------------- 64.5/64.5 kB 1.8 MB/s eta 0:00:00
Installing collected packages: wheel
Successfully installed wheel-0.40.0
C:\Users\redacted>python -m pip install scipy
Collecting scipy
Using cached scipy-1.10.1.tar.gz (42.4 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [44 lines of output]
+ meson setup --prefix=C:\Users\redacted\AppData\Local\Programs\Python\Python310-32 C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24 C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-ieizbzv8\build --native-file=C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
The Meson build system
Version: 1.0.1
Source dir: C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24
Build dir: C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-ieizbzv8\build
Build type: native build
Project name: SciPy
Project version: 1.10.1
C compiler for the host machine: cc (gcc 12.2.0 "cc (Rev6, Built by MSYS2 project) 12.2.0")
C linker for the host machine: cc ld.bfd 2.39
C++ compiler for the host machine: c++ (gcc 12.2.0 "c++ (Rev6, Built by MSYS2 project) 12.2.0")
C++ linker for the host machine: c++ ld.bfd 2.39
Cython compiler for the host machine: cython (cython 0.29.33)
Host machine cpu family: x86
Host machine cpu: x86
Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Compiler for C supports arguments -Wno-unused-function: YES
Compiler for C supports arguments -Wno-conversion: YES
Compiler for C supports arguments -Wno-misleading-indentation: YES
Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
Library m found: YES
Fortran compiler for the host machine: gfortran (gcc 12.2.0 "GNU Fortran (Rev6, Built by MSYS2 project) 12.2.0")
Fortran linker for the host machine: gfortran ld.bfd 2.39
Compiler for Fortran supports arguments -Wno-conversion: YES
Checking if "-Wl,--version-script" : links: YES
Program cython found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-p_w8jkr7\overlay\Scripts\cython.EXE)
Program python found: YES (C:\Users\redacted\AppData\Local\Programs\Python\Python310-32\python.exe)
Program pythran found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-p_w8jkr7\overlay\Scripts\pythran.EXE)
WARNING: You should add the boolean check kwarg to the run_command call.
It currently defaults to false,
but it will default to true in future releases of meson.
See also: https://github.com/mesonbuild/meson/issues/9300
Run-time dependency threads found: YES
Library npymath found: YES
Library npyrandom found: YES
Found pkg-config: C:\msys64\usr\bin\pkg-config.EXE (1.8.0)
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
Run-time dependency openblas found: NO (tried pkgconfig)
..\..\scipy\meson.build:134:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig
A full log can be found at C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-ieizbzv8\build\meson-logs\meson-log.txt
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
C:\Users\redacted> how does this even happen? |
On WSL you are bounded by the Linux environment so it won't work with the regular installation of SciPy with windows tooling (that's why you are picking up tar.gz and trying to build). Look at the prerequisite Linux packages at the top of this post for adding openblas. For installing scipy on windows, use the wheels (regular pip) as building it for windows is a very involved process with limited guarantee of success when multiple tools are installed. |
It was my understanding that when I called pip from cmd.exe, it would work as normally, but apparently I need to download the wheels, and install them? - will try and report back if it didn't work |
If pip didn't download the wheels for you then there presumably aren't any wheels for your system. But I notice that "your system" is a combination of Windows and Python310-32, which seems like a 32-bit python -- please try installing a 64-bit python if your operating system supports it. |
I think we have now gotten to the point where this issue is no longer helpful - too long and too many different errors being mixed in. We have docs at |
@eli-schwartz - thank you for the assistance. |
Yes, for CentOS 7,
|
I did not read detail.
However I solve by below at the above problem. MyEnv: Ubuntu 20.04, pypy7.3.11(py3.9).
|
Facing same issue while installing SciPy on fedora
i have already installed when i perhaps i need to run some pkg-config step before running |
@vimalk78 have you read #16308 (comment) ? |
@lucascolley , thanks 🙏 that really helped
|
Describe your issue.
This comes from SymPy CI: sympy/sympy#23513
A few days ago it successfully built and tested NumPy and SciPy on Python 3.11b1 from git using pip but now the same setup fails to build SciPy with
Maybe this would work with
apt install libopenblas-dev liblapack-dev
but it was working before so should it be expected to work still?This was the last successful run (the build was successful although a test later failed):
https://github.com/sympy/sympy/runs/6598088659?check_suite_focus=true
Reproducing Code Example
Error message
SciPy/NumPy/Python version information
scipy/numpy from git and python 3.11b1
The text was updated successfully, but these errors were encountered: