Feature: MKL dependency

[alyst]

It would be nice to support Intel Math Kernel Library via dependency('mkl'), which would replicate the logic of Intel MKL Link Advisor.

[arteymix]

MKL now provides a pkg-config file that covers static/dynamic linkage and 32/64-bit integer interfaces since 2018.1 release.

https://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl-and-pkg-config-tool

I'll investigate if this issue is still pertinent.

[rgommers]

I'll investigate if this issue is still pertinent.

This issue is still relevant, although it's larger than only MKL - generic BLAS/LAPACK dependency detection is needed. From comments in gh-2847:

FTR the "correct" way of solving this is to add dependency('BLAS') so the fix is implemented only once in upstream, rather than copypasting the detection logic from one meson.build file to another.

...

It would be great if you could work on BLAS/LAPACK/MKL dependency classes in Meson so that everyone benefits from being able to find and use them. The BoostDependency object is a good start in mesonbuild/dependencies/. Poke me if you have questions!

[dcbaker]

Is this different than the scalapack dependency we already have?

That already covers the Intel MKL iiuc

[rgommers]

That's a good start, but it looks different to me. This is what we'd want I think: #5186 (comment). To me it looks like the ScaLAPACK support is a good start, but not the same. ScaLAPACK itself is a subset of LAPACK redesigned for high-performance distributed computing. From reading the scalapack dependency code and the docs for it, it's unclear to me if today dependency('mkl') standalone works.

For more context: I'm looking into what we'd need to use Meson in SciPy. There are multiple versions of BLAS and LAPACK, and the ideal interface is being able to give a list of ones that are accepted, in order of preference:

# There's 64-bit flavors (ILP64, e.g. 'openblas_ilp64') that we'd need for SciPy too
blas_dep = dependency('mkl', 'openblas', 'blis', 'accelerate', 'atlas', 'netlib')
lapack_dep = dependency('mkl', 'openblas', 'libflame', 'accelerate', 'atlas', 'netlib')

Some of these are nice-to-haves, but at least OpenBLAS, MKL and a generic flavor ('netlib') seem essential (users will want to add the rest later). There's also detection logic for whether the detected BLAS supports a CBLAS interface (usually needs checking via a test compile, e.g. like this).

Cc @scivision, @QuLogic you were discussing this in gh-5186, could you comment on the current status?

[scivision]

I use CMake with MKL across operating systems. I use my own FindLapack.cmake and FindScalapack.cmake.
The MKL logic is orthogonal to the non-MKL versions of those packages, so it could be a starting point for a Meson dependency('mkl').
Scalapack function scalapack_mkl
Lapack mkl

I use these on Linux, MacOS and Windows with MKL and oneAPI or GCC.

The reason I veered away from a FindMKL in CMake is that for non-Intel compilers that work with MKL, I wanted to keep that logic in a single FindLapack rather than elevating that logic to the user's CMakeLists.txt. Since Meson has dependency fallback this is less of a bother in Meson.

[rgommers]

The reason I veered away from a FindMKL in CMake is that for non-Intel compilers that work with MKL, I wanted to keep that logic in a single FindLapack rather than elevating that logic to the user's CMakeLists.txt.

Thanks for the details @scivision. That makes sense.

---

I reviewed CMake's FindBLAS.cmake and FindLAPACK.cmake, and compared it with numpy.distutils.system_info:

• CMake's implementation has more comprehensive coverage of MKL, and that makes up over 50% of the code for detecting all BLAS/LAPACK libraries.
• For OpenBLAS, ATLAS and Accelerate the NumPy support is more extensive.
• NumPy also has better end user configurability (CMake only has a global BLA_VENDOR environment variable to select one particular implementation), and better compile checks for common issues.

My impression is that it's certainly worth implementing BLAS/LAPACK support in Meson rather than relying on what CMake offers - aside from MKL details there's not much there. Also, both CMake and NumPy have a lot of historical baggage, e.g. the CMake MKL support would be a lot shorter if the < 10.3 ( a decade old, no longer needed) support was dropped.

[FRidh]

In Nixpkgs we support different blas and lapack alternatives, and we do that through a provider https://nixos.org/manual/nixpkgs/stable/#sec-overlays-alternatives-blas-lapack. The provider creates pkg-config files which then point to the blas and lapack you've chosen to use. It is my understanding that dependency('blas') would then work and you would not need to specify dependency('openblas') or dependency('mkl'), but I have not tested this. Maybe @matthewbauer who implemented this can chime in.

[eli-schwartz]

NixOS specific pkg-config files are an anti-pattern and should not be used by projects.

[eli-schwartz]

Update: in meson git master there is now support for dependency('foo', 'bar', 'baz') which will try all three dependencies and select the first one it found.

See #9330

Hopefully this is enough to get moving on providing first class mkl, blas, lapack, etc. support -- I'm thinking we probably want a dependency for each library and provider?

[rgommers]

I'm thinking we probably want a dependency for each library and provider?

I think so. I'm not 100% sure about the scope there though; one question I had is in how far to support non pkg-config usage. For Windows that's needed for sure I'd think, but for example on Linux something like dependency('openblas') works today most of the time but not always (it depends on the package manager that installed OpenBLAS, and also what if you build a dev version of OpenBLAS locally?).

[GuillaumeQuenneville]

The blas libraries implement an interface. The issue is the providers name their libraries and pkg-config's differently and we would like for end users to choose their own (usually restricted by a custom compiled version the computer cluster they are using is providing). Separating libraries by providers brings us back to the initial issue (requiring custom logic to find which libraries are installed that implements the interface). see https://github.com/libmir/mir-blas/blob/master/meson.build for example.

The dependency('foo', 'bar', 'baz') is an elegant way to solve this. If we do separate library by provider searches maybe provide a utility function that iterates through them all and returns the one that they found (to adapt for minor implementation differences)? But the dependency('foo', 'bar', 'baz') is perfect for my own use cases.

Thanks for the work. Much appreciated.

[eli-schwartz]

@rgommers internally, a meson custom dependency is represented by a "foo_factory" with several different trials that it can use to find the dependency. For example, the preferred trial is generally to instantiate a PkgConfigDependency, but generally anything with a custom lookup will need to cover its bases by probing for clib_compiler.find_library / has_header and assembling the correct link_args based on that, and wrapping this all up in a subclass derived from SystemDependency. Then "FooSystemDependency" will be the second trial.

Other dependency lookup classes which can be added as trials:

• ConfigToolDependency (for software that uses ugly shell scripts called foo-config which we all dread)
• ExtraFrameworkDependency (macOS frameworks)
• CMakeDependency (try to see if a dummy cmake project can detect the dependency and read that information into meson using the cmake tracing API)
• BuiltinDependency (something like iconv or ngettext that may be available in GNU libc on Linux)

The example FooSystemDependency class would end up doing the same thing on either Windows or Linux, so it might as well be available on both. We are not in the habit of gating dependency methods to a platform just because we think other platforms should prefer PkgConfigDependency.

tl;dr once the non-pkg-config case is implemented for Windows it will be the (fallback) choice for Linux as well.

I am more than happy to help implement all this, please ping me any time if you (or anyone else) needs help.

[eli-schwartz]

@GuillaumeQuenneville, it should be possible to have e.g. a generic "blas" dependency factory that internally invokes a number of different lookups for e.g. openblas and other providers, and returns one. It would be a high-level wrapper over dependency('openblas') in that case.

As we already do this for curses (it looks up pkg-config dependencies for ncursesw, ncurses, pdcurses, generic curses, config-tool via ncurses-config, and probes for the headers for all these varieties too) it should be fairly simple to implement within the current existing framework.

So, I think we should be able to support "just give me something implementing the interface" as well as "I want this priority ordering of these implementations only".

You should be able to detect which implementation was found by checking the dependency object's .name() method, and whether it was found by pkg-config, system probes, a config-tool, cmake, etc. by checking the .type_name() method.

[dcbaker]

We do have a few other dependencies (mostly HPC related) that also implement the factories. I wrote the factory stuff, and I'd be happy to answer any questions as well.

The ConfigTool class has grown really to mean "an executable that you call to get information about a dependency." I created it for llvm-config, but we actually call some other interesting things like qmake with it now :)

[rgommers]

Thanks for the explanations @eli-schwartz and @dcbaker!

So, I think we should be able to support "just give me something implementing the interface" as well as "I want this priority ordering of these implementations only".

I agree, we'd like both for SciPy/NumPy. We also have options to make this user-configurable, via a .cfg file or via environment variables (see https://numpy.org/devdocs/user/building.html?highlight=npy_lapack_order#blas). I'm not attached to that interface, but we will have to provide the capability in some form. If you have any thoughts on whether that is possible in a sane way within Meson, or if we need to put it in some helper build script, that'd be great to hear.

The blas libraries implement an interface.

Multiple interfaces actually: BLAS, CBLAS, and also 64-bit flavors (we support ILP64, but LP64 also exists). In addition, there are library-specific APIs (in addition to the Netlib BLAS interface) and performance differences, which may be reasons to require a specific library.

[eli-schwartz]

You could use a meson_options.txt array option to specify the order, e.g. build via meson setup builddir -Dlapack_order=mkl,blis,openblas,atlas,blas.

Then do

dependency(get_option('lapack_order'))

or foreach over each member of the option array and transform user-friendly names into dependency names, in case they differ, and pass the resulting array to dependency() instead.

meson does not support reading environment variables, though it does support .ini files which can (since 0.56.0) contain project options: https://mesonbuild.com/Machine-files.html
Those do still need to be manually specified on the meson command line...