WIP: add support for BLAS and LAPACK dependencies

[rgommers]

This is a start on implementing the BLAS and LAPACK support discussed in gh-2835.

This PR in its current form passes the added test case, and dependency('openblas') works for me now with SciPy in 3 cases beyond pkg-config:

• CMake detecting OpenBLAS
• Detecting OpenBLAS installed in the default environment lib/ and include/ dirs, but no pkg-config nor CMake installed
• Using a native or cross file to specify the include and library directories for OpenBLAS

This is already nice to have in its current form, however there's a lot more to do. This includes dealing with 32 vs. 64 bit interfaces, supporting CBLAS and (maybe) LAPACKE, and implementing support for other implementations (MKL, Netlib, ATLAS, etc.). The docs I added try to sketch this out - I'm very much not sure about the API to expose here, some feedback would be great.

This is also the first time I'm implementing a new dependency, so I may have taken a few wrong turns. If there's better ways of doing any of this, I'd love to hear them.

[codecov]

Codecov Report

Merging #10921 (484833b) into master (d573db7) will decrease coverage by 2.78%.
The diff coverage is 54.23%.

❗ Current head 484833b differs from pull request most recent head 78f219e. Consider uploading reports for the commit 78f219e to get more accurate results

@@            Coverage Diff             @@
##           master   #10921      +/-   ##
==========================================
- Coverage   67.56%   64.79%   -2.78%     
==========================================
  Files         414      416       +2     
  Lines       90145    90350     +205     
  Branches    21318    20721     -597     
==========================================
- Hits        60909    58541    -2368     
- Misses      24515    27350    +2835     
+ Partials     4721     4459     -262     

Impacted Files	Coverage Δ
mesonbuild/mesonmain.py	72.83% <0.00%> (-2.17%)	⬇️
mesonbuild/dependencies/blas_lapack.py	51.06% <51.06%> (ø)
mesonbuild/mtest.py	75.83% <81.25%> (-3.72%)	⬇️
mesonbuild/dependencies/__init__.py	100.00% <100.00%> (ø)
mesonbuild/msubprojects.py	30.36% <100.00%> (+0.13%)	⬆️

... and 162 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[lgtm-com]

This pull request introduces 2 alerts when merging 5c2bb31 into 7912901 - view on LGTM.com

new alerts:

• 2 for Unused import

[eli-schwartz]

Thanks for looking at this! Miscellaneous hodgepodge of review remarks that I found when taking a hurried glance (it's the holiday season right now):

[rgommers]

Thanks for the quick feedback @eli-schwartz. Enjoy your holiday season!

[rgommers]

The UNEXSKIP is because OpenBLAS isn't installed in CI yet. Before I mess with that, let me ask in which jobs you want me to add it. Maybe the same ones as hdf5 or scalapack, since they're similar scientific computing focused dependencies?

[rgommers]

As far as I could tell, adding keywords or methods/attributes to a dependency object is not allowed by design, and I should be using modules and .get_variable. modules is not a 100% natural fit, because not everything we're looking for is an optional/add-on component (e.g., the 32/64-bit interface is "give me this one" and symbol suffix is optional data about the library one is building against). However, it seems close enough.

From that follows the current draft API:

openblas_dep = dependency('openblas',
  version : '>=0.3.21',
  language: 'c',          # can be c/cpp/fortran
  modules: [
    'interface: ilp64',   # can be lp64 or ilp64 (or auto?)
    'symbol-suffix: 64_', # check/auto-detect? default to 64_ or no suffix?
    'cblas',
  ]
)
# Query properties as needed:
has_cblas = openblas.get_variable('cblas')
is_ilp64 = openblas_dep.get_variable('interface') == 'ilp64'
blas_symbol_suffix = openblas_dep.get_variable('symbol-suffix')

It'd be great to get some feedback on this.

I'll probably implement Netlib and MKL as well in this PR, because those are both quite different from OpenBLAS. So if the API works for all three, then that's a good sign.

[eli-schwartz]

As far as I could tell, adding keywords or methods/attributes to a dependency object is not allowed by design

I would say returned object methods definitely should be avoided, and really in nearly all cases .get_variable() should work or should be made to work.

Adding keywords to dependency() is more iffy, there are a couple cases where we have pretty boutique kwargs. Here's the full list of kwargs supported by dependency():

meson/mesonbuild/interpreter/interpreter.py

Lines 252 to 272 in b8e53ed

	permitted_dependency_kwargs = {
	'allow_fallback',
	'cmake_args',
	'cmake_module_path',
	'cmake_package_version',
	'components',
	'default_options',
	'fallback',
	'include_type',
	'language',
	'main',
	'method',
	'modules',
	'native',
	'not_found_message',
	'optional_modules',
	'private_headers',
	'required',
	'static',
	'version',
	}

Many of these are even documented. 🤣 The rest are carefully disclaimed in the docs with:

This function supports additional library-specific keyword arguments that may also be accepted

I think my personal opinion on the matter is I'd like it if we could avoid new kwargs, but if it would be painful or gross without them, then we might as well add them.

[rgommers]

Thanks for the feedback!

I think my personal opinion on the matter is I'd like it if we could avoid new kwargs, but if it would be painful or gross without them, then we might as well add them.

I think we can. If we compare, e.g., modules: ['interface: ilp64'] with interface: '64' the latter is a little nicer probably. But I'm happy enough with the former, and I also like that Meson enforces consistency in its API design.

I will continue down the current path then. Just a heads up that I won't have much time to work on this for the next 10 days or so. So I'll probably continue to enhance this PR after that, and then split off pieces that are ready into more digestible new PRs.

[tylerjereddy]

Will NumPy need 64-bit ILP64 OpenBLAS support for 3.12 wheels in the next release cycle? I think Sebastian was thinking they might in the meeting today? Not sure if that increases priority here.

[rgommers]

@tylerjereddy yes, but this one will be quick compared to dealing with the SIMD stuff. I've been too swamped with other topics, but I should revisit this in the near future. We'll probably miss the 1.25.0 release window and will have to do a 1.25.1/2 to complete the switchover, just like for SciPy (IIRC 1.9.3 was the first complete release without setup.py parts).