New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: MKL dependency #2835
Comments
MKL now provides a pkg-config file that covers static/dynamic linkage and 32/64-bit integer interfaces since 2018.1 release. https://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl-and-pkg-config-tool I'll investigate if this issue is still pertinent. |
This issue is still relevant, although it's larger than only MKL - generic BLAS/LAPACK dependency detection is needed. From comments in gh-2847: FTR the "correct" way of solving this is to add dependency('BLAS') so the fix is implemented only once in upstream, rather than copypasting the detection logic from one meson.build file to another. ... It would be great if you could work on BLAS/LAPACK/MKL dependency classes in Meson so that everyone benefits from being able to find and use them. The BoostDependency object is a good start in mesonbuild/dependencies/. Poke me if you have questions! |
Is this different than the scalapack dependency we already have? That already covers the Intel MKL iiuc |
That's a good start, but it looks different to me. This is what we'd want I think: #5186 (comment). To me it looks like the ScaLAPACK support is a good start, but not the same. ScaLAPACK itself is a subset of LAPACK redesigned for high-performance distributed computing. From reading the scalapack dependency code and the docs for it, it's unclear to me if today For more context: I'm looking into what we'd need to use Meson in SciPy. There are multiple versions of BLAS and LAPACK, and the ideal interface is being able to give a list of ones that are accepted, in order of preference:
Some of these are nice-to-haves, but at least OpenBLAS, MKL and a generic flavor ('netlib') seem essential (users will want to add the rest later). There's also detection logic for whether the detected BLAS supports a CBLAS interface (usually needs checking via a test compile, e.g. like this). Cc @scivision, @QuLogic you were discussing this in gh-5186, could you comment on the current status? |
I use CMake with MKL across operating systems. I use my own FindLapack.cmake and FindScalapack.cmake. I use these on Linux, MacOS and Windows with MKL and oneAPI or GCC. The reason I veered away from a FindMKL in CMake is that for non-Intel compilers that work with MKL, I wanted to keep that logic in a single FindLapack rather than elevating that logic to the user's CMakeLists.txt. Since Meson has dependency fallback this is less of a bother in Meson. |
Thanks for the details @scivision. That makes sense. I reviewed CMake's
My impression is that it's certainly worth implementing BLAS/LAPACK support in Meson rather than relying on what CMake offers - aside from MKL details there's not much there. Also, both CMake and NumPy have a lot of historical baggage, e.g. the CMake MKL support would be a lot shorter if the |
In Nixpkgs we support different blas and lapack alternatives, and we do that through a provider https://nixos.org/manual/nixpkgs/stable/#sec-overlays-alternatives-blas-lapack. The provider creates pkg-config files which then point to the blas and lapack you've chosen to use. It is my understanding that |
NixOS specific pkg-config files are an anti-pattern and should not be used by projects. |
Update: in meson git master there is now support for See #9330 Hopefully this is enough to get moving on providing first class mkl, blas, lapack, etc. support -- I'm thinking we probably want a dependency for each library and provider? |
I think so. I'm not 100% sure about the scope there though; one question I had is in how far to support non pkg-config usage. For Windows that's needed for sure I'd think, but for example on Linux something like |
The blas libraries implement an interface. The issue is the providers name their libraries and pkg-config's differently and we would like for end users to choose their own (usually restricted by a custom compiled version the computer cluster they are using is providing). Separating libraries by providers brings us back to the initial issue (requiring custom logic to find which libraries are installed that implements the interface). see https://github.com/libmir/mir-blas/blob/master/meson.build for example. The dependency('foo', 'bar', 'baz') is an elegant way to solve this. If we do separate library by provider searches maybe provide a utility function that iterates through them all and returns the one that they found (to adapt for minor implementation differences)? But the dependency('foo', 'bar', 'baz') is perfect for my own use cases. Thanks for the work. Much appreciated. |
@rgommers internally, a meson custom dependency is represented by a "foo_factory" with several different trials that it can use to find the dependency. For example, the preferred trial is generally to instantiate a PkgConfigDependency, but generally anything with a custom lookup will need to cover its bases by probing for clib_compiler.find_library / has_header and assembling the correct link_args based on that, and wrapping this all up in a subclass derived from SystemDependency. Then "FooSystemDependency" will be the second trial. Other dependency lookup classes which can be added as trials:
The example FooSystemDependency class would end up doing the same thing on either Windows or Linux, so it might as well be available on both. We are not in the habit of gating dependency methods to a platform just because we think other platforms should prefer PkgConfigDependency. tl;dr once the non-pkg-config case is implemented for Windows it will be the (fallback) choice for Linux as well. I am more than happy to help implement all this, please ping me any time if you (or anyone else) needs help. |
@GuillaumeQuenneville, it should be possible to have e.g. a generic "blas" dependency factory that internally invokes a number of different lookups for e.g. openblas and other providers, and returns one. It would be a high-level wrapper over dependency('openblas') in that case. As we already do this for curses (it looks up pkg-config dependencies for ncursesw, ncurses, pdcurses, generic curses, config-tool via ncurses-config, and probes for the headers for all these varieties too) it should be fairly simple to implement within the current existing framework. So, I think we should be able to support "just give me something implementing the interface" as well as "I want this priority ordering of these implementations only". You should be able to detect which implementation was found by checking the dependency object's .name() method, and whether it was found by pkg-config, system probes, a config-tool, cmake, etc. by checking the .type_name() method. |
We do have a few other dependencies (mostly HPC related) that also implement the factories. I wrote the factory stuff, and I'd be happy to answer any questions as well. The ConfigTool class has grown really to mean "an executable that you call to get information about a dependency." I created it for llvm-config, but we actually call some other interesting things like qmake with it now :) |
Thanks for the explanations @eli-schwartz and @dcbaker!
I agree, we'd like both for SciPy/NumPy. We also have options to make this user-configurable, via a
Multiple interfaces actually: BLAS, CBLAS, and also 64-bit flavors (we support ILP64, but LP64 also exists). In addition, there are library-specific APIs (in addition to the Netlib BLAS interface) and performance differences, which may be reasons to require a specific library. |
You could use a meson_options.txt array option to specify the order, e.g. build via Then do dependency(get_option('lapack_order')) or foreach over each member of the option array and transform user-friendly names into dependency names, in case they differ, and pass the resulting array to dependency() instead. meson does not support reading environment variables, though it does support .ini files which can (since 0.56.0) contain project options: https://mesonbuild.com/Machine-files.html |
Why that? In pkgsrc, we rely on pkg-config files for different build variants of openblas, for example (serial, openmp, pthread). You can imagine different CPU builds installed in the same prefix and differentiated by library names. There is no standardization on these, distros following different approaches. I see it as vital to be able to have a generic way to tell differing build systems which BLAS, LAPACK, CBLAS or LAPACKE to use (all possibly distinct libraries, as you can put generic libcblas.so on top of a specific libblas.so). Writing (custom) pkg-config files for the variants and having means to point the build system to them seems a rather good way to solve the BLAS mess. This mess explicitly includes having to write custom library detection code in build systems or in extensions in projects (like lots of redundant FindBLAS/FindLAPACK variants for CMake). The perspective I want to emphasize here: Please consider the packager that wants a controlled build with an exactly specified dependency to use without any autodetection of other variants. You might need magic to detect properties of the library. So, detecting if it's really ilp64, if it has OpenBLAS-specific functions, etc. But you first find the lib, then care for which implementation it is. Build systems converging on pkg-config files as means to communicate dependencies would be a big improvement over dozens of differing ways to force package builds into using the correct libraries as intended by the person starting the build. You should not care where the pkg-config file comes from. Without pkg-config files to communicate the choice, next best thing would be build systems agreeing on how all possible BLAS variants are called and then offer a parameter to submit the one and only desired implementation for the build. |
The problem is that if no one agrees on what the name is, and the name only works on some systems to begin with, then projects cannot do If the foobar project upstream, provides a foobar.pc, then it is safe to do I have lost count of the number of projects that had to be locally patched because the developers only supported Debian, because they only ran Debian, and Debian locally added a foobar.pc that other distros didn't have. And then assumed that |
The problem is that there is no standard set of .pc names, and with BLAS, there is incentive for different build variants that are bound to need local names or other means of separation. I agree that you should not hardcode distro-specific pkg-config module names. All I am pleading for is a standard way to be able to say |
Also, I dream of a world where build systems and project configure scripts don't need to know anything about differing BLAS implementations (and LAPACK, etc.) but just say that they need the generic API. Putting code to detect MKL and other frameworks into meson still is duplicated with same efforcts in autotools, CMake etal. If we all could agree that BLAS stuff will come with a .pc file, the custom code could be reduced to a list of common pkg-config module names as default, and code to check if |
This is always possible with. Meson dependencies by default check pkg-config first, and accept any name. If there's custom code for dependency detection like in the linked PR here, that then tends to check for additional names or adds some scheme that upstream uses, but you can always give it a custom and distro-specific name as a packager when you know the exact name.
That makes two of us:) This is really slow-going, but I'm hopeful we're going to see the standardization of ILP64 symbol names at least in reference BLAS/LAPACK next year. |
It would be nice to support Intel Math Kernel Library via
dependency('mkl')
, which would replicate the logic of Intel MKL Link Advisor.The text was updated successfully, but these errors were encountered: