Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: pip install scipy from git fails with "openblas" not found #16308

Closed
oscarbenjamin opened this issue May 29, 2022 · 62 comments
Closed

BUG: pip install scipy from git fails with "openblas" not found #16308

oscarbenjamin opened this issue May 29, 2022 · 62 comments
Labels
Build issues Issues with building from source, including different choices of architecture, compilers and OS

Comments

@oscarbenjamin
Copy link

Describe your issue.

This comes from SymPy CI: sympy/sympy#23513

A few days ago it successfully built and tested NumPy and SciPy on Python 3.11b1 from git using pip but now the same setup fails to build SciPy with

      ../../scipy/meson.build:120:0: ERROR: Dependency "openblas" not found, tried pkgconfig and cmake

Maybe this would work with apt install libopenblas-dev liblapack-dev but it was working before so should it be expected to work still?

This was the last successful run (the build was successful although a test later failed):
https://github.com/sympy/sympy/runs/6598088659?check_suite_focus=true

Reproducing Code Example

# Ubuntu 20.04.4 with Python 3.11b1 running in GA
sudo apt install antlr4 libgfortran5 gfortran libmpfr-dev libmpc-dev
pip install --upgrade pip wheel
pip install git+https://github.com/cython/cython@master
pip install git+https://github.com/numpy/numpy@main
pip install git+https://github.com/scipy/scipy@main

Error message

Run pip install git+https://github.com/numpy/numpy@main
  pip install git+https://github.com/numpy/numpy@main
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib
Collecting git+https://github.com/numpy/numpy@main
  Cloning https://github.com/numpy/numpy (to revision main) to /tmp/pip-req-build-a992dsfx
  Running command git clone --filter=blob:none --quiet https://github.com/numpy/numpy /tmp/pip-req-build-a992dsfx
  Resolved https://github.com/numpy/numpy to commit 68f3a12f4cc9310a7d60f9878af72cf98e7233f3
  Running command git submodule update --init --recursive -q
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: numpy
  Building wheel for numpy (pyproject.toml): started
  Building wheel for numpy (pyproject.toml): still running...
  Building wheel for numpy (pyproject.toml): still running...
  Building wheel for numpy (pyproject.toml): still running...
  Building wheel for numpy (pyproject.toml): still running...
  Building wheel for numpy (pyproject.toml): finished with status 'done'
  Created wheel for numpy: filename=numpy-1.24.0.dev0+75.g68f3a12f4-cp311-cp311-linux_x86_64.whl size=22029574 sha256=ed58cdbe5b547632f5a45e8ab48e0c200b8f247a91e8e945c457a7899185724a
  Stored in directory: /tmp/pip-ephem-wheel-cache-jo9owv4z/wheels/ad/e8/55/3b4372ddf4700df1fb35dc513aacd83756396e870c5a6f8ece
Successfully built numpy
Installing collected packages: numpy
Successfully installed numpy-1.24.0.dev0+75.g68f3a12f4

Run pip install git+https://github.com/scipy/scipy@main
Collecting git+https://github.com/scipy/scipy@main
  Cloning https://github.com/scipy/scipy (to revision main) to /tmp/pip-req-build-b76e9_br
  Running command git clone --filter=blob:none --quiet https://github.com/scipy/scipy /tmp/pip-req-build-b76e9_br
  Resolved https://github.com/scipy/scipy to commit 6c51ed7aa4230d9fc08d099773b66f61e9828e97
  Running command git submodule update --init --recursive -q
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [72 lines of output]
      The Meson build system
      Version: 0.62.1
      Source dir: /tmp/pip-req-build-b76e9_br
      Build dir: /tmp/pip-req-build-b76e9_br/.mesonpy-_wsc8qd3/build
      Build type: native build
      Project name: SciPy
      Project version: 1.9.0.dev0
      C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0")
      C linker for the host machine: cc ld.bfd 2.34
      C++ compiler for the host machine: c++ (gcc 9.4.0 "c++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0")
      C++ linker for the host machine: c++ ld.bfd 2.34
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Library m found: YES
      Fortran compiler for the host machine: gfortran (gcc 9.4.0 "GNU Fortran (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0")
      Fortran linker for the host machine: gfortran ld.bfd 2.34
      Program cython found: YES (/tmp/pip-build-env-4bh3e24n/overlay/bin/cython)
      Program pythran found: YES (/tmp/pip-build-env-4bh3e24n/overlay/bin/pythran)
      Program cp found: YES (/usr/bin/cp)
      Program python3 found: YES (/opt/hostedtoolcache/Python/3.11.0-beta.1/x64/bin/python)
      Found pkg-config: /usr/bin/pkg-config (0.29.1)
      Library npymath found: YES
      Library npyrandom found: YES
      Found CMake: /usr/local/bin/cmake (3.23.1)
      Run-time dependency openblas found: NO (tried pkgconfig and cmake)
      
      ../../scipy/meson.build:120:0: ERROR: Dependency "openblas" not found, tried pkgconfig and cmake
      
      A full log can be found at /tmp/pip-req-build-b76e9_br/.mesonpy-_wsc8qd3/build/meson-logs/meson-log.txt
      + meson setup --native-file=/tmp/pip-req-build-b76e9_br/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/opt/hostedtoolcache/Python/3.11.0-beta.1/x64 /tmp/pip-req-build-b76e9_br /tmp/pip-req-build-b76e9_br/.mesonpy-_wsc8qd3/build
      Traceback (most recent call last):
        File "/opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
          ^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4bh3e24n/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 805, in get_requires_for_build_wheel
          with _project(config_settings) as project:
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib/python3.11/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4bh3e24n/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 784, in _project
          with Project.with_temp_working_dir(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib/python3.11/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4bh3e24n/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 479, in with_temp_working_dir
          yield cls(source_dir, tmpdir, build_dir)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4bh3e24n/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 423, in __init__
          self._configure(reconfigure=bool(build_dir) and not native_file_mismatch)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4bh3e24n/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 450, in _configure
          self._meson(
          ^^^^^^^^^^^^
        File "/tmp/pip-build-env-4bh3e24n/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 438, in _meson
          return self._proc('meson', *args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4bh3e24n/overlay/lib/python3.11/site-packages/mesonpy/__init__.py", line 434, in _proc
          subprocess.check_call(list(args))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib/python3.11/subprocess.py", line 402, in check_call
          raise CalledProcessError(retcode, cmd)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      subprocess.CalledProcessError: Command '['meson', 'setup', '--native-file=/tmp/pip-req-build-b76e9_br/.mesonpy-native-file.ini', '-Ddebug=false', '-Dstrip=true', '-Doptimization=2', '--prefix=/opt/hostedtoolcache/Python/3.11.0-beta.1/x64', '/tmp/pip-req-build-b76e9_br', '/tmp/pip-req-build-b76e9_br/.mesonpy-_wsc8qd3/build']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.

SciPy/NumPy/Python version information

scipy/numpy from git and python 3.11b1

@oscarbenjamin oscarbenjamin added the defect A clear bug or issue that prevents SciPy from being installed or used as expected label May 29, 2022
@rgommers
Copy link
Member

Maybe this would work with apt install libopenblas-dev liblapack-dev but it was working before so should it be expected to work still?

Indeed, just apt install libopenblas-dev is enough (no need for liblapack-dev).

We just switched build systems yesterday, so while our build requirements didn't change (you need BLAS/LAPACK), the way of detecting them did change.

Can you try adding libopenblas-dev and seeing if that fixes things for you?

@oscarbenjamin
Copy link
Author

our build requirements didn't change (you need BLAS/LAPACK)

So how did it previously succeed? I thought NumPy had a fallback for this or is it that SciPy doesn't support that fallback? Except it previously worked and the test suite completed...

To be clear there is no need from SymPy's end to support this particular setup. Maybe though if we noticed this change then it might break something significant for someone else.

@rgommers
Copy link
Member

So how did it previously succeed? I thought NumPy had a fallback for this or is it that SciPy doesn't support that fallback?

We're no longer using numpy.distutils to build, but instead Meson. So discovering dependencies is done via pkg-config, rather than looking for specific libraries directly in a set of hardcoded paths.

To be clear there is no need from SymPy's end to support this particular setup. Maybe though if we noticed this change then it might break something significant for someone else.

It's expected that packagers will need to change a few things in build scripts, that is unavoidable. And a small price to pay to move away from distutils:)

@eli-schwartz
Copy link
Contributor

But also this is one of the items at #16293 and the goal is to teach Meson some of these fallback detection methods. :)

@oscarbenjamin
Copy link
Author

Can you try adding libopenblas-dev and seeing if that fixes things for you?

Yes, that fixes it. At the time of writing tests haven't run yet but scipy built successfully:
https://github.com/sympy/sympy/runs/6645335177?check_suite_focus=true

I guess this can be closed then?

@oscarbenjamin
Copy link
Author

tests haven't run yet

Tests are now complete. Just one failure due to #16042 (comment)

@rgommers
Copy link
Member

Great, thanks Oscar!

I guess this can be closed then?

I'm expecting the same question a bunch more times, so let's leave this open for now so the issue and solution are more discoverable.

@oscarbenjamin oscarbenjamin changed the title BUG: pip install numpy, scipy from git fails with "openblas" not found in Python 3.11b1 BUG: pip install scipy from git fails with "openblas" not found May 29, 2022
@oscarbenjamin
Copy link
Author

so the issue and solution are more discoverable.

Okay, I edited the title since I guess the Python version is not relevant and maybe numpy isn't either...

@rgommers rgommers added Build issues Issues with building from source, including different choices of architecture, compilers and OS and removed defect A clear bug or issue that prevents SciPy from being installed or used as expected labels Jun 1, 2022
@xkszltl
Copy link

xkszltl commented Jul 29, 2022

Got this issue recently as well, but on CentOS 7 instead.
The distro openblas doesn't seem to carry pkg-config or cmake config.
We also have self-built openblas installed under /usr/local with both cmake/pkg-config, but scipy doesn't recognize them.

@rgommers
Copy link
Member

You probably then have to point pkg-config to your custom location with export PKG_CONFIG_PATH=/use/lib/pkgconfig (or a similar such path).

@xkszltl
Copy link

xkszltl commented Jul 29, 2022

My concern is more on why cmake didn't find it there, as it's capable of picking up both /usr and /usr/local.
One special thing about CentOS 7 is, by default, it has /usr/bin/cmake -> cmake2, and an old cmake3.
We also have extra /usr/local/bin/cmake for much newer version of cmake3 and available in PATH.
So I'm wondering if scipy somehow pick the cmake2.

[root@86f3a6392d4a roaster]# /usr/bin/cmake --version
cmake version 2.8.12.2
[root@86f3a6392d4a roaster]# /usr/bin/cmake3 --version
cmake3 version 3.17.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[root@86f3a6392d4a roaster]# /usr/local/bin/cmake --version
cmake version 3.21.7

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[root@86f3a6392d4a roaster]# which cmake
/usr/local/bin/cmake
[root@86f3a6392d4a roaster]# which pkg-config
/usr/bin/pkg-config
[root@86f3a6392d4a roaster]# rpm -ql roaster-openblas | grep -v src | grep -i -e 'cmake' -e '\.pc'
/usr/local/lib/cmake/openblas/OpenBLASConfig.cmake
/usr/local/lib/cmake/openblas/OpenBLASConfigVersion.cmake
/usr/local/lib/pkgconfig/openblas.pc

@xkszltl
Copy link

xkszltl commented Jul 29, 2022

Similar build script works on Debian11/Ubuntu18.04, and their /usr/bin/pkg-config is somehow capable of picking up /usr/local/lib/pkgconfig/openblas.pc without PKG_CONFIG_PATH.

@eli-schwartz
Copy link
Contributor

eli-schwartz commented Jul 29, 2022

  • don't use the which program, it doesn't tell you what your shell prompt would find -- among other things, it doesn't report aliases and shell functions. On some systems, the which program is a csh script that ignores your changes to PATH, and (even when run from a bash prompt) loads ~/.cshrc then reports aliases from that file. Again -- regardless of which shell you use.

    The which program isn't installed on many systems, so it's bizarre that the cargo-culting of the internet suggests people to use it. For interactive use, bash and other modern shells have type -a built into the shell. Accept no inferior substitutes. For posix shell scripts, use command -v.

Steps off of soapbox

  • the progress log of actually trying to build SciPy, will include a message stating that it tried to find cmake, whether it was successful, and if successful, what path it found it at. This will be after it fails to detect openblas via pkg-config, and before it reports the overall status of whether it ultimately found openblas at all.

@xkszltl
Copy link

xkszltl commented Jul 29, 2022

don't use the which program ...

Yes I agree with that point, we also find which not installed in many small or embedded environment or points to weird things. But on those popular distros which is quite convenient and gradually converges to similar behavior over time.

the progress log of actually trying to build SciPy, will include a message stating that it tried to find cmake, whether it was successful, and if successful, what path it found it at. This will be after it fails to detect openblas via pkg-config, and before it reports the overall status of whether it ultimately found openblas at all.

Yes, it's actually right on top of that error msg, not sure how I even missed that.

  The Meson build system                                                                                                                                                                              
  Version: 0.62.2                                                                                                                                                                                     
  Source dir: /tmp/scratch/mirrored-pip-62IX72/scipy                                                                                                                                                  
  Build dir: /tmp/scratch/mirrored-pip-62IX72/scipy/.mesonpy-2icz_mg3/build                                                                                                                           
  Build type: native build                                                                                                                                                                            
  Project name: SciPy                                                                                                                                                                                 
  Project version: 1.9.0                                                                                                                                                                              
  C compiler for the host machine: gcc (gcc 11.2.1 "gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)")                                                                                                    
  C linker for the host machine: gcc ld.bfd 2.36.1-1                                                                                                                                                  
  C++ compiler for the host machine: g++ (gcc 11.2.1 "g++ (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)")                                                                                                  
  C++ linker for the host machine: g++ ld.bfd 2.36.1-1                                                                                                                                                
  Host machine cpu family: x86_64                                                                                                                                                                     
  Host machine cpu: x86_64                                                                                                                                                                            
  Compiler for C supports arguments -Wno-unused-but-set-variable: YES                                                                                                                                 
  Library m found: YES                                                                                                                                                                                
  Fortran compiler for the host machine: gfortran (gcc 11.2.1 "GNU Fortran (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)")                                                                                 
  Fortran linker for the host machine: gfortran ld.bfd 2.36.1-1                                                                                                                                       
  Program cython found: YES (/tmp/pip-build-env-f8l83eie/overlay/bin/cython)                                                                                                                          
  Program pythran found: YES (/tmp/pip-build-env-f8l83eie/overlay/bin/pythran)                                                                                                                        
  Program cp found: YES (/usr/bin/cp)                                                                                                                                                                 
  Program python3 found: YES (/opt/rh/rh-python38/root/usr/bin/python3)                                                                                                                               
  Found pkg-config: /usr/bin/pkg-config (0.27.1)                                                                                                                                                      
  Library npymath found: YES                                                                                                                                                                          
  Library npyrandom found: NO                                                                                                                                                                         
  Found CMake: /usr/local/bin/cmake (3.21.7)                                                                                                                                                          
  Run-time dependency openblas found: NO (tried pkgconfig and cmake)

  ../../scipy/meson.build:120:0: ERROR: Dependency "openblas" not found, tried pkgconfig and cmake

So it took the right cmake, but failed to find openblas.

Then I run meson manually to check the failure (would be nice if the meson build log can be printed or at least kept on disk after a failed pip install call), and notice it's looking for openblas.
As you can tell from my log above, the default cmake config is /usr/local/lib/cmake/openblas/OpenBLASConfig.cmake, so if I say OpenBLAS in meson.build, it'll work.

@xkszltl
Copy link

xkszltl commented Jul 29, 2022

Log for cmake hit with OpenBLAS:

[root@86f3a6392d4a scipy]# sudo -E PATH="$PATH" scl enable rh-python38 devtoolset-11 'meson setup --native-file=/tmp/scratch/mirrored-pip-oMKvvZ/scipy/.mesonpy-native-file.ini -Ddebug=false -Doptimi
zation=2 --prefix=/opt/rh/rh-python38/root/usr /tmp/scratch/mirrored-pip-oMKvvZ/scipy /tmp/scratch/mirrored-pip-oMKvvZ/scipy/.mesonpy-tbyr9jh6/build'                                                 
The Meson build system                                                                                                                                                                                
Version: 0.63.0                                                                                                                                                                                       
Source dir: /tmp/scratch/mirrored-pip-oMKvvZ/scipy                                                                                                                                                    
Build dir: /tmp/scratch/mirrored-pip-oMKvvZ/scipy/.mesonpy-tbyr9jh6/build                                                                                                                             
Build type: native build                                                                                                                                                                              
Project name: SciPy                                                                                                                                                                                   
Project version: 1.9.0                                                                                                                                                                                
C compiler for the host machine: ccache cc (gcc 11.2.1 "cc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)")                                                                                                 
C linker for the host machine: cc ld.bfd 2.36.1-1                                                                                                                                                     
C++ compiler for the host machine: ccache c++ (gcc 11.2.1 "c++ (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)")                                                                                             
C++ linker for the host machine: c++ ld.bfd 2.36.1-1                                                                                                                                                  
Host machine cpu family: x86_64                                                                                                                                                                       
Host machine cpu: x86_64                                                                                                                                                                              
Compiler for C supports arguments -Wno-unused-but-set-variable: YES                                                                                                                                   
Library m found: YES                                                                                                                                                                                  
Fortran compiler for the host machine: gfortran (gcc 11.2.1 "GNU Fortran (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)")                                                                                   
Fortran linker for the host machine: gfortran ld.bfd 2.36.1-1                                                                                                                                         
Program cython found: YES (/opt/rh/rh-python38/root/usr/local/bin/cython)                                                                                                                             
Program pythran found: YES (/opt/rh/rh-python38/root/usr/local/bin/pythran)                                                                                                                           
Program cp found: YES (/usr/bin/cp)                                                                                                                                                                   
Program python3 found: YES (/opt/rh/rh-python38/root/usr/bin/python3)                                                                                                                                 
Found pkg-config: /usr/bin/pkg-config (0.27.1)                                                                                                                                                        
Library npymath found: YES                                                                                                                                                                            
Library npyrandom found: YES                                                                                                                                                                          
Found CMake: /usr/local/bin/cmake (3.23.3)                                                                                                                                                            
Run-time dependency openblas found: YES 0.3.20                                                                                                                                                        
Dependency OpenBLAS found: YES 0.3.20 (cached)                                                                                                                                                        
Program _build_utils/cythoner.py found: YES (/opt/rh/rh-python38/root/usr/bin/python3 /tmp/scratch/mirrored-pip-oMKvvZ/scipy/scipy/_build_utils/cythoner.py)                                          
Checking for function "open_memstream" : NO                                                                                                                                                           
Configuring messagestream_config.h using configuration                                                                                                                                                
Checking for size of "void*" : 8                                                                                                                                                                      
Run-time dependency threads found: YES                                                                                                                                                                
Checking for size of "void*" : 8                                                                                                                                                                      
Dependency threads found: YES unknown (cached)                                                                                                                                                        
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)                                                                                                                          
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)                                                                                                                          
Compiler for C++ supports arguments -Wno-format-truncation: YES                                                                                                                                       
Compiler for C++ supports arguments -Wno-class-memaccess: YES                                                                                                                                         
Build targets in project: 201                                                                                                                                                                         
                                                                                                                                                                                                      
SciPy 1.9.0                                                                                                                                                                                           
                                                                                                                                                                                                      
  User defined options                                                                                                                                                                                
    Native files: /tmp/scratch/mirrored-pip-oMKvvZ/scipy/.mesonpy-native-file.ini                                                                                                                     
    debug       : false
    optimization: 2
    prefix      : /opt/rh/rh-python38/root/usr

Found ninja-1.10.2 at /usr/bin/ninja
WARNING: custom_target '_cython_tree' has more than one output! Using the first one.
WARNING: custom_target '_lib_pxd' has more than one output! Using the first one.
WARNING: custom_target '_cython_tree' has more than one output! Using the first one.

@xkszltl
Copy link

xkszltl commented Jul 29, 2022

Also confirmed "OpenBLAS" cannot be used with "openblas.pc", so scipy should add some case-insensitive routing.

@eli-schwartz
Copy link
Contributor

Then I run meson manually to check the failure (would be nice if the meson build log can be printed or at least kept on disk after a failed pip install call),

I think pip install --no-clean could help there?

Also confirmed "OpenBLAS" cannot be used with "openblas.pc", so scipy should add some case-insensitive routing.

SciPy's current minimum version of Meson supports dependency('foo', 'bar') -- it will first look up "foo" with pkg-config, then with cmake, then if it still doesn't find anything it will try to find "bar" with pkg-config, then with cmake. So that's a possibility.

In the long run, as mentioned above, the idea is to contribute custom detection code to Meson, to smooth over these differences.

@xkszltl
Copy link

xkszltl commented Jul 29, 2022

I think pip install --no-clean could help there?

No it doesn't.
I'm using --no-clean already.

it will first look up "foo" with pkg-config, then with cmake

Maybe the other way?
When I try "OpenBLAS", the log stopped at cmake, so I guess it's cmake first and then pkg-config?

xkszltl added a commit to xkszltl/Roaster that referenced this issue Jul 29, 2022
@rgommers
Copy link
Member

rgommers commented Jul 29, 2022

When I try "OpenBLAS", the log stopped at cmake, so I guess it's cmake first and then pkg-config?

It should be CMake second; the log ended up there because it's the first version that actually worked, right?

Adding dependency('openblas', 'OpenBLAS') in SciPy main / 1.9.1 is a very cheap improvement it looks like. @eli-schwartz just to make sure: that works with Meson 0.62.2 as well, right?

@xkszltl
Copy link

xkszltl commented Jul 29, 2022

Maybe? Not sure.
I didn't saw it trying pkg-config and fail, but maybe it's just me missing something in log again.

rgommers added a commit to rgommers/scipy that referenced this issue Jul 31, 2022
This is a fallback detection path, in case pkg-config fails to
find `openblas`. CMake is case-sensitive, so we need to try
the camelcase version of the name.

If CMake is installed and it finds OpenBLAS, the result looks like:
```
Determining dependency 'OpenBLAS' with pkg-config executable '/home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config'
env[PKG_CONFIG_PATH]:
Called `/home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config --modversion OpenBLAS` -> 1

CMake binary for 1 is cached.

Determining dependency 'OpenBLAS' with CMake executable '/home/rgommers/anaconda3/envs/scipy-dev/bin/cmake'
Try CMake generator: auto
Calling CMake (['/home/rgommers/anaconda3/envs/scipy-dev/bin/cmake']) in /home/rgommers/code/scipy/build/meson-private/cmake_OpenBLAS with:
  - "-DNAME=OpenBLAS"
  - "-DARCHS=libpyldb-util.cpython-310-x86-64-linux-gnu.so;libpyldb-util.cpython-310-x86-64-linux-gnu.so.2;libpyldb-util.cpython-310-x86-64-linux-gnu.so.2.5.0;libpytalloc-util.cpython-310-x86-64-linux-gnu.so;libpytalloc-util.cpython-310-x86-64-linux-gnu.so.2;libpytalloc-util.cpython-310-x86-64-linux-gnu.so.2.3.3;libsamba-policy.cpython-310-x86-64-linux-gnu.so;libsamba-policy.cpython-310-x86-64-linux-gnu.so.0;libsamba-policy.cpython-310-x86-64-linux-gnu.so.0.0.1"
  - "-DVERSION="
  - "-DCOMPS="
  - "--trace-expand"
  - "--trace-format=json-v1"
  - "--no-warn-unused-cli"
  - "--trace-redirect=cmake_trace.txt"
  - "-DCMAKE_TOOLCHAIN_FILE=/home/rgommers/code/scipy/build/meson-private/cmake_OpenBLAS/CMakeMesonToolchainFile.cmake"
  - "."
  - "-DCMAKE_PREFIX_PATH=/home/rgommers/anaconda3/envs/scipy-dev;/home/rgommers/anaconda3/envs/scipy-dev/x86_64-conda-linux-gnu/sysroot/usr"
using old-style CMake variables for dependency OpenBLAS
Include Dirs:         ['/home/rgommers/anaconda3/envs/scipy-dev/include']
Compiler Definitions: []
Libraries:            ['/home/rgommers/anaconda3/envs/scipy-dev/lib/libopenblas.so']
Run-time dependency openblas found: YES 0.3.20
```

This follows up on comments in scipygh-16308
@rgommers
Copy link
Member

gh-16750 improves the fallback detection. I've tested that indeed pkg-config gets tried first, and CMake second.

@belm0
Copy link
Contributor

belm0 commented Nov 7, 2022

Defaulting to openblas on Linux may be an especially poor choice, since Debian / Ubuntu seem to prefer building against shared-object libblas-dev, and then selecting the implementation (blas, openblas, etc.) system-wide via the alternatives mechanism.

@xkszltl
Copy link

xkszltl commented Nov 7, 2022

IMO alternatives is more of a user-facing mechanism, not a system-wide selection, libs may not agree on which to pick so it's only a shortcut for desktop user frequently using gcc-10 to say gcc without suffix.

@rgommers
Copy link
Member

rgommers commented Nov 7, 2022

I see https://docs.scipy.org/doc/scipy/dev/contributor/meson_advanced.html#blas-lapack-selection, but I wonder why this is necessary for scipy build, when numpy itself is able to detect the installed blas dev library (it will detect libblas-dev, libopenblas-dev, etc.).

Fair point. It will not remain necessary; the full support for all of what numpy.distutils offered BLAS/LAPACK-wise is still a work in progress. It hasn't been a small task to replace all of numpy.distutils with something better.

Also, the cited doc doesn't mention how to pass the option through from pip install or pip wheel. After a few hours, I haven't been able to figure it out.

It's not possible in a one-liner yet, it will be once mesonbuild/meson-python#167 lands. For now, use:

$ meson setup build -Dblas=blas -Dlapack=lapack
$ python -m pip wheel --config-setting builddir=build

Defaulting to openblas on Linux may be an especially poor choice, since Debian / Ubuntu seem to prefer building against shared-object libblas-dev, and then selecting the implementation (blas, openblas, etc.) system-wide via the alternatives mechanism.

This is different to every distro, there's no reasonable way to support that. And we're not defaulting to plain -lblas -llapack, because that's likely to pick up Netlib reference libraries, which are very slow. OpenBLAS is our default in wheels, CI, and in from source builds.

@belm0
Copy link
Contributor

belm0 commented Nov 7, 2022

$ meson setup build -Dblas=blas -Dlapack=lapack
$ python -m pip wheel --config-setting builddir=build

This requires manually grabbing the scipy source, correct?

And we're not defaulting to plain -lblas -llapack, because that's likely to pick up Netlib reference libraries, which are very slow. OpenBLAS is our default in wheels, CI, and in from source builds.

Doesn't this differ from numpy, where the default wheels seem to be libblas? So it means the typical app using numpy and scipy is using two blas libraries?

@rgommers
Copy link
Member

rgommers commented Nov 7, 2022

This requires manually grabbing the scipy source, correct?

Correct. The meson-python release with support for passing parameters through pip should be ready in the next few weeks. For now you have to grab the source.

Doesn't this differ from numpy, where the default wheels seem to be libblas?

It doesn't, NumPy definitely ships with OpenBLAS, and has for quite a few years (before that, the default was ATLAS).

So it means the typical app using numpy and scipy is using two blas libraries?

Still kinda, because NumPy 1.22 switched to 64-bit OpenBLAS while SciPy uses 32-bit. This is one of the major downsides of using wheels; PyPI + wheels is too simplistic to manage this well. Use conda-forge, Spack, your Linux distro or whatever else works for you if you want to get a setup that is guaranteed to have a single BLAS library.

@oscarbenjamin
Copy link
Author

NumPy 1.22 switched to 64-bit OpenBLAS while SciPy uses 32-bit

Why doesn't SciPy use the 64-bit OpenBLAS?

@rgommers
Copy link
Member

rgommers commented Nov 7, 2022

It's not fully supported yet, see gh-17244. Also in the numpy.distutils build support was partial, we only had one CI job with it and never shipped wheels. It's just work that needs doing. NumPy upgraded fairly recently, SciPy needs a lot more from BLAS and LAPACK so it takes longer.

@wlepera
Copy link

wlepera commented Dec 6, 2022

I'm seeing the same problem, and after going through the comments in this issue several times, it's still not clear what the workaround is (unless conda is the only solution). I'm trying to install scipy 1.9.3 on RHEL 8.6 Linux, ppc64le, using python 3.8. This was originally part of a build of Apache TVM, where my TVM build was successful but the subsequent setup failed while trying to build the scipy dependency. Even if I install the RHEL-packaged python scipy RPM, TVM setup still tries to build the 1.9.3 version on it's own. At that point I started installing scipy apart from the TVM setup script as follows:

pip3.8 install --no-cache-dir --no-clean scipy

which results in

Collecting scipy
  Downloading https://files.pythonhosted.org/packages/0a/2e/44795c6398e24e45fa0bb61c3e98de1cfea567b1b51efd3751e2f7ff9720/scipy-1.9.3.tar.gz (42.1MB) | 42.1MB 2.5MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.8 /usr/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp_x8latq7
         cwd: /tmp/pip-install-9u5e6hic/scipy
    Complete output (76 lines):
    The Meson build system
    Version: 0.64.1
    Source dir: /tmp/pip-install-9u5e6hic/scipy
    Build dir: /tmp/pip-install-9u5e6hic/scipy/.mesonpy-bjix6uyx/build
    Build type: native build
    Project name: SciPy
    Project version: 1.9.3
    C compiler for the host machine: cc (gcc 8.5.0 "cc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10)")
    C linker for the host machine: cc ld.bfd 2.30-113
    C++ compiler for the host machine: c++ (gcc 8.5.0 "c++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10)")
    C++ linker for the host machine: c++ ld.bfd 2.30-113
    Host machine cpu family: ppc64
    Host machine cpu: ppc64le
    Compiler for C supports arguments -Wno-unused-but-set-variable: YES
    Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
    Compiler for C supports arguments -Wno-unused-function: YES
    Compiler for C supports arguments -Wno-conversion: YES
    Compiler for C supports arguments -Wno-misleading-indentation: YES
    Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
    Library m found: YES
    Fortran compiler for the host machine: gfortran (gcc 8.5.0 "GNU Fortran (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10)")
    Fortran linker for the host machine: gfortran ld.bfd 2.30-113
    Compiler for Fortran supports arguments -Wno-conversion: YES
    Program cython found: YES (/tmp/pip-build-env-9wxs3hmm/overlay/bin/cython)
    Program pythran found: YES (/tmp/pip-build-env-9wxs3hmm/overlay/bin/pythran)
    Program cp found: YES (/bin/cp)
    Program python found: YES (/usr/bin/python3.8)
    Found pkg-config: /bin/pkg-config (1.4.2)
    Run-time dependency threads found: YES
    Library npymath found: YES
    Library npyrandom found: NO
    Found CMake: /bin/cmake (3.20.2)
    Run-time dependency openblas found: NO (tried pkgconfig and cmake)
    Run-time dependency openblas found: NO (tried pkgconfig and cmake)
    
    ../../scipy/meson.build:131:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig and cmake
    
    A full log can be found at /tmp/pip-install-9u5e6hic/scipy/.mesonpy-bjix6uyx/build/meson-logs/meson-log.txt

The openblas and openblas-devel packages are installed, as are cmake and pkg-config. The RHEL openblas packages do not ship an openblas.pc file, so I'm guessing that means pkg-config won't know about it. The meson build log that's referenced doesn't exist, even after using the --no-clean option, so there's no opportunity to get additional information there:

# pwd
/tmp/pip-install-9u5e6hic/scipy
# ls -a
.             CONTRIBUTING.rst      do.py                 LICENSE.txt               mypy.ini                       pyproject.toml  scipy             tox.ini
..            dev.py                HACKING.rst.txt       meson.build               mypy_requirements.txt          pytest.ini      setup.py
benchmarks    doc                   INSTALL.rst.txt       meson_options.txt         pip-delete-this-directory.txt  README.rst      site.cfg.example
CITATION.bib  doc_requirements.txt  LICENSES_bundled.txt  .mesonpy-native-file.ini  PKG-INFO                       runtests.py     tools

I'm installing as root so as to make the environment available to multiple users. I did try installing as my local user with the --user option to pip install, but got the same error. PYTHONPATH is set to /usr/lib64/python3.8/site-packages. I am noticing the scipy build is picking up the system versions of cmake (3.20.2) and pkg-config (1.4.2), rather than newer versions I installed via pip (cmake 3.25.0, pkgconfig 1.5.5). I added the pip version of cmake to the system PATH and verified that the scipy install attempt uses the newer version, but that didn't make a difference.

I also checked out the instructions at https://scipy.github.io/devdocs/dev/contributor/building.html for CENTOS/RHEL builds, and all the referenced packages are installed:

If you want to use the system Python and pip, you will need:

C, C++, and Fortran compilers (typically gcc, g++, and gfortran).

Python header files (typically a package named python3-dev or python3-devel)

BLAS and LAPACK libraries. OpenBLAS is the SciPy default; other variants include ATLAS and MKL.

Debian/Ubuntu Linux

Fedora

CentOS/RHEL
To install SciPy build requirements, you can do:

sudo yum install gcc-gfortran python3-devel openblas-devel lapack-devel pkgconfig

# sudo yum install gcc-gfortran python38-devel openblas-devel lapack-devel pkgconfig
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:44:09 ago on Tue 06 Dec 2022 09:54:09 AM EST.
Package gcc-gfortran-8.5.0-10.1.el8_6.ppc64le is already installed.
Package python38-devel-3.8.12-1.module+el8.6.0+12642+c3710b74.ppc64le is already installed.
Package openblas-devel-0.3.15-3.el8.ppc64le is already installed.
Package lapack-devel-3.8.0-8.el8.ppc64le is already installed.
Package pkgconf-pkg-config-1.4.2-1.el8.ppc64le is already installed.
Dependencies resolved.
Nothing to do.
Complete!

Any workaround or debug suggestions would be appreciated. At this point I'm simply trying to get scipy built so I can proceed with the next steps in our work. Thanks.

@rgommers
Copy link
Member

rgommers commented Dec 6, 2022

The RHEL openblas packages do not ship an openblas.pc file, so I'm guessing that means pkg-config won't know about it.

Yes, RHEL/Fedora are a problem, one packager simply refuses to understand the problem.

I don't know why CMake also doesn't find OpenBLAS here. Note that you do not want CMake from PyPI, you want the one from your system, installed through the same package manager that also installed OpenBLAS.

The workaround I'd suggest is to craft a new openblas.pc file, put it in /any/random/dir/ and then set PKG_CONFIG_PATH=/any/random/dir/. The contents of openblas.pc should be:

libdir=/path/to/openblas/sharedlib-dir   # e.g., /usr/lib
includedir=/path/to/openblas/headers  # e.g., /usr/include
version=0.3.21  # set to actual version
extralib=-lm -lpthread -lgfortran   # Check if you use openblas built with pthreads or openmp, if the latter, add `-lgomp`
Name: openblas
Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
Version: ${version}
URL: https://github.com/xianyi/OpenBLAS
Libs: -L${libdir} -lopenblas
Libs.private: ${extralib}
Cflags: -I${includedir}

@eli-schwartz
Copy link
Contributor

I am noticing the scipy build is picking up the system versions of cmake (3.20.2) and pkg-config (1.4.2), rather than newer versions I installed via pip (cmake 3.25.0, pkgconfig 1.5.5).

The CMake version "shouldn't" matter. Installing a newer one from pip is a waste of time, but shouldn't make things worse. (Hopefully. It may lack correct system integration.)

There's no such thing as a pip version of pkg-config. But there is a python module that is a set of thin python functions that run the system pkg-config, translating a python function-y interface into a set of arguments to subprocess.run, then parse the returned output into python lists.

Neither meson nor SciPy uses that, so you can ignore that this wrapper module exists.

...

What matters is if the openblas RHEL package ships with the upstream openblas cmake files. Unfortunately, it doesn't -- for the same reason it doesn't ship with the upstream openblas pkg-config file.

As @rgommers said, you can write your own -- or download openblas yourself and install it by hand, I guess.

Unfortunately there's nothing to do other than to submit a bug report to RHEL, they broke the upstream software.

@wlepera
Copy link

wlepera commented Dec 6, 2022

@rgommers, I added my own openblas.pc based on your template sample, set the PKG_CONFIG_PATH, and was able to build scipy successfully - and got my TVM build and install complete as well. Thank you for the assist!

@belm0
Copy link
Contributor

belm0 commented Mar 20, 2023

latest on how to pip install scipy against libblas (not libopenblas)

The meson-python release with support for passing parameters through pip should be ready in the next few weeks.

unfortunately, it also requires a fix for pip (pypa/pip#11681), which has been merged, but not yet released

Assuming you're willing to install pip from HEAD, it would look like this:

apt install libblas-dev liblapack-dev gfortran
pip install git+https://github.com/pypa/pip.git
pip install --config-settings "setup-args=-Dblas=blas" --config-settings "setup-args=-Dlapack=lapack" scipy

(also works for pip wheel)

@ZcuBa
Copy link

ZcuBa commented Mar 20, 2023

I just got this on my windows machine.

I know, on plain windows we use wheels, and we should not have this issue, but some user (me) has gotten used to running wsl for some specific tools, but also needs python available from windows it self - so preferred to install my most used modules everywhere, making the python scripts available from all terminals...

however:

Microsoft Windows [Version 10.0.19044.2604]
(c) Microsoft Corporation. All rights reserved.

C:\Users\redacted>pip install scipy
Collecting scipy
  Downloading scipy-1.10.1.tar.gz (42.4 MB)
     ---------------------------------------- 42.4/42.4 MB 9.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [44 lines of output]
      + meson setup --prefix=C:\Users\redacted\AppData\Local\Programs\Python\Python310-32 C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619 C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-hdayg0pn\build --native-file=C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
      The Meson build system
      Version: 1.0.1
      Source dir: C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619
      Build dir: C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-hdayg0pn\build
      Build type: native build
      Project name: SciPy
      Project version: 1.10.1
      C compiler for the host machine: cc (gcc 12.2.0 "cc (Rev6, Built by MSYS2 project) 12.2.0")
      C linker for the host machine: cc ld.bfd 2.39
      C++ compiler for the host machine: c++ (gcc 12.2.0 "c++ (Rev6, Built by MSYS2 project) 12.2.0")
      C++ linker for the host machine: c++ ld.bfd 2.39
      Cython compiler for the host machine: cython (cython 0.29.33)
      Host machine cpu family: x86
      Host machine cpu: x86
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Compiler for C supports arguments -Wno-unused-function: YES
      Compiler for C supports arguments -Wno-conversion: YES
      Compiler for C supports arguments -Wno-misleading-indentation: YES
      Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
      Library m found: YES
      Fortran compiler for the host machine: gfortran (gcc 12.2.0 "GNU Fortran (Rev6, Built by MSYS2 project) 12.2.0")
      Fortran linker for the host machine: gfortran ld.bfd 2.39
      Compiler for Fortran supports arguments -Wno-conversion: YES
      Checking if "-Wl,--version-script" : links: YES
      Program cython found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-d0rw0g37\overlay\Scripts\cython.EXE)
      Program python found: YES (C:\Users\redacted\AppData\Local\Programs\Python\Python310-32\python.exe)
      Program pythran found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-d0rw0g37\overlay\Scripts\pythran.EXE)
      WARNING: You should add the boolean check kwarg to the run_command call.
               It currently defaults to false,
               but it will default to true in future releases of meson.
               See also: https://github.com/mesonbuild/meson/issues/9300
      Run-time dependency threads found: YES
      Library npymath found: YES
      Library npyrandom found: YES
      Found pkg-config: C:\msys64\usr\bin\pkg-config.EXE (1.8.0)
      Did not find CMake 'cmake'
      Found CMake: NO
      Run-time dependency openblas found: NO (tried pkgconfig and cmake)
      Run-time dependency openblas found: NO (tried pkgconfig)

      ..\..\scipy\meson.build:134:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig

      A full log can be found at C:\Users\redacted\AppData\Local\Temp\pip-install-93aft9o9\scipy_283fdb1a5ed940a1b421ca74c1efc619\.mesonpy-hdayg0pn\build\meson-logs\meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

C:\Users\redacted>pip install --upgrade wheel
Collecting wheel
  Downloading wheel-0.40.0-py3-none-any.whl (64 kB)
     ---------------------------------------- 64.5/64.5 kB 1.8 MB/s eta 0:00:00
Installing collected packages: wheel
Successfully installed wheel-0.40.0

C:\Users\redacted>python -m pip install scipy
Collecting scipy
  Using cached scipy-1.10.1.tar.gz (42.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [44 lines of output]
      + meson setup --prefix=C:\Users\redacted\AppData\Local\Programs\Python\Python310-32 C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24 C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-ieizbzv8\build --native-file=C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
      The Meson build system
      Version: 1.0.1
      Source dir: C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24
      Build dir: C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-ieizbzv8\build
      Build type: native build
      Project name: SciPy
      Project version: 1.10.1
      C compiler for the host machine: cc (gcc 12.2.0 "cc (Rev6, Built by MSYS2 project) 12.2.0")
      C linker for the host machine: cc ld.bfd 2.39
      C++ compiler for the host machine: c++ (gcc 12.2.0 "c++ (Rev6, Built by MSYS2 project) 12.2.0")
      C++ linker for the host machine: c++ ld.bfd 2.39
      Cython compiler for the host machine: cython (cython 0.29.33)
      Host machine cpu family: x86
      Host machine cpu: x86
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Compiler for C supports arguments -Wno-unused-function: YES
      Compiler for C supports arguments -Wno-conversion: YES
      Compiler for C supports arguments -Wno-misleading-indentation: YES
      Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
      Library m found: YES
      Fortran compiler for the host machine: gfortran (gcc 12.2.0 "GNU Fortran (Rev6, Built by MSYS2 project) 12.2.0")
      Fortran linker for the host machine: gfortran ld.bfd 2.39
      Compiler for Fortran supports arguments -Wno-conversion: YES
      Checking if "-Wl,--version-script" : links: YES
      Program cython found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-p_w8jkr7\overlay\Scripts\cython.EXE)
      Program python found: YES (C:\Users\redacted\AppData\Local\Programs\Python\Python310-32\python.exe)
      Program pythran found: YES (C:\Users\redacted\AppData\Local\Temp\pip-build-env-p_w8jkr7\overlay\Scripts\pythran.EXE)
      WARNING: You should add the boolean check kwarg to the run_command call.
               It currently defaults to false,
               but it will default to true in future releases of meson.
               See also: https://github.com/mesonbuild/meson/issues/9300
      Run-time dependency threads found: YES
      Library npymath found: YES
      Library npyrandom found: YES
      Found pkg-config: C:\msys64\usr\bin\pkg-config.EXE (1.8.0)
      Did not find CMake 'cmake'
      Found CMake: NO
      Run-time dependency openblas found: NO (tried pkgconfig and cmake)
      Run-time dependency openblas found: NO (tried pkgconfig)

      ..\..\scipy\meson.build:134:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig

      A full log can be found at C:\Users\redacted\AppData\Local\Temp\pip-install-fijewgyc\scipy_feb18cc4fa4b4b3d83a9742125d19d24\.mesonpy-ieizbzv8\build\meson-logs\meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

C:\Users\redacted>

how does this even happen?

@ilayn
Copy link
Member

ilayn commented Mar 20, 2023

On WSL you are bounded by the Linux environment so it won't work with the regular installation of SciPy with windows tooling (that's why you are picking up tar.gz and trying to build). Look at the prerequisite Linux packages at the top of this post for adding openblas.

For installing scipy on windows, use the wheels (regular pip) as building it for windows is a very involved process with limited guarantee of success when multiple tools are installed.

@ZcuBa
Copy link

ZcuBa commented Mar 20, 2023

It was my understanding that when I called pip from cmd.exe, it would work as normally, but apparently I need to download the wheels, and install them? - will try and report back if it didn't work

@eli-schwartz
Copy link
Contributor

If pip didn't download the wheels for you then there presumably aren't any wheels for your system.

But I notice that "your system" is a combination of Windows and Python310-32, which seems like a 32-bit python -- please try installing a 64-bit python if your operating system supports it.

@rgommers
Copy link
Member

rgommers commented Mar 21, 2023

I'm expecting the same question a bunch more times, so let's leave this open for now so the issue and solution are more discoverable.

I think we have now gotten to the point where this issue is no longer helpful - too long and too many different errors being mixed in. We have docs at http://scipy.github.io/devdocs/dev/contributor/meson_advanced.html#select-a-different-blas-or-lapack-library http://scipy.github.io/devdocs/building/blas_lapack.html on how to switch to different BLAS/LAPACK libraries, and for new issues it'd be better if folks opened new issues. So I'll close this. Thanks everyone!

knorth55 added a commit to knorth55/jsk_common that referenced this issue Mar 22, 2023
knorth55 added a commit to knorth55/jsk_common that referenced this issue Mar 22, 2023
@ZcuBa
Copy link

ZcuBa commented Mar 23, 2023

@eli-schwartz - thank you for the assistance.
Re-installing python for windows from scratch, in the cpu architectures version, did the trick (64 bit).

@fengerzh
Copy link

Yes, for CentOS 7, yum install openblas not work, you have to manually install it:

 git clone https://github.com/xianyi/OpenBLAS.git
 cd OpenBLAS
 make -j8
 make PREFIX=/usr/local/OpenBLAS install
export PKG_CONFIG_PATH=/usr/local/OpenBLAS/lib/pkgconfig

@bsnresearcher
Copy link

I did not read detail.

Run-time dependency openblas found: NO (tried pkgconfig and cmake)

 ../../scipy/meson.build:120:0: ERROR: Dependency "openblas" not found, tried pkgconfig and cmake

However I solve by below at the above problem. MyEnv: Ubuntu 20.04, pypy7.3.11(py3.9).

sudo apt-get update
sudo apt-get install python3-pkgconfig libopenblas-dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build issues Issues with building from source, including different choices of architecture, compilers and OS
Projects
None yet
Development

No branches or pull requests