meson fails to locate libraries on macOS

[jeremyhu]

Seen with meson 0.62.1

The default methods for resolving a dependency() should include a trivial build test with -l.

For example, cairo and fontconfig attempt to locate expat with:

expat_dep = dependency('expat',
  fallback: ['expat', 'expat_dep'])

On macOS, this fails. libexpat is part of the system. There are no pkg-config files. It is trivially usable:

$ echo 'int main() {return 0;}' | clang -x c - -lexpat -o /tmp/true && otool -L /tmp/true
/tmp/true:
	/usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 8.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.120.1)

This results in us trying to download libexpat rather than using the system libexpat (which just requires "-lexpat"):

Determining dependency 'expat' with pkg-config executable '/opt/buildX11/bin/pkg-config'
env[PKG_CONFIG_PATH]: /opt/X11/share/pkgconfig:/opt/X11/lib/pkgconfig
Called `/opt/buildX11/bin/pkg-config --modversion expat` -> 1

Finding framework path by running:  cc -v -E - 

Looking for framework expat in /Library/Developer/CommandLineTools/SDKs/MacOSX12.5.sdk/System/Library/Frameworks
CMake binary for 1 is not cached
CMake binary missing from cross or native file, or env var undefined.
Trying a default CMake fallback at cmake
Did not find CMake 'cmake'
Found CMake: NO
CMake binary for machine 1 not found. Giving up.
Run-time dependency expat found: NO (tried pkgconfig, framework and cmake)
Looking for a fallback subproject for the dependency expat
Downloading expat source from https://github.com/libexpat/libexpat/releases/download/R_2_2_6/expat-2.2.6.tar.bz2

Before resorting to downloading and building the dependency, meson should first check if it's available as "-lexpat" (eg: build test, or look for it in $SDKROOT/usr/lib/libexpat.tbd)

[jeremyhu]

Note I explicitly make cmake not available when building with meson because it cause additional problems, such as pulling in content from the SDK that was used to build cmake even when I'm not using that SDK to build the current project.

[jeremyhu]

Based on the documentation, I'd expect to be able to use the 'system' method here with something like:

expat_dep = dependency('expat', required: false)
if not expat_dep.found()
  expat_dep = dependency('expat', method: 'system', fallback: ['expat', 'expat_dep'])
endif

But that's not working, and there's nothing useful in meson-log.txt to explain what the 'system' method is doing in order to debug why it's failing...

[jeremyhu]

It looks like I can use this as a solution...

expat_dep = dependency('expat', required: false)
if not expat_dep.found()
  expat_dep = cc.find_library('expat', required : false)
  if not expat_dep.found()
    expat_dep = dependency('expat', method: 'system', fallback: ['expat', 'expat_dep'])
  endif
endif

But that's overly convoluted. meson should be able to do that by default...

[eli-schwartz]

This is just how dependencies work. A dependency can be called anything, unrelated to the name of the library, and possess arbitrary, or arbitrarily complex, compile / link args in addition to the library.

This information is read from an interface file, which can be pkg-config or cmake or sometimes the output of a "config-tool" program.

As described in https://mesonbuild.com/Dependencies.html#system a "system" type dependency is when the dependency description is in the meson source code as mesonbuild/dependencies/misc.py etc.

It is indeed correct to use cc.find_library yourself for this, if the Apple SDK is broken and malformed and lacks the pkg-config files which upstream expat provides.

[jeremyhu]

The macOS SDK is not malformed. It is designed to not need pkg-config files. Everything "just works" with -isysroot.

This particular issue makes transitioning from autotools based builds to meson based builds overly complex, and it seems that meson should be able to handle this with a bit of extension.

1. Please update the documentation to better reflect what "system" really is becuase it reads as though it should "just work" for this case:

In these cases Meson provides convenience wrappers in the form of system dependencies. Internally these dependencies do exactly what a user would do in the build system DSL or with a script, likely calling compiler.find_library(), setting link_with and include_directories. By putting these in Meson upstream the barrier of using them is lowered, as projects using Meson don't have to re-implement the logic.

If this is only supported for some cases (like zlib), those cases should be documented.

2. Please support finding expat via "system" the same way you do zlib. They're in the exact same boat here, so it would certainly make sense for meson to support expat the same way it does zlib.

3. Please support a method for finding the library with a simple link check (eg: 'find_library') which basically does the same thing as cc.find_library().

4. Please support passing in an array for method.

If all of those improvements were done, the above could be simplified as:

expat_dep = dependency('expat', method: ['auto', 'find_library'], library_name: 'expat', fallback: ['expat', 'expat_dep'])

or if we want to have find_library assume the dependency name is the library_name (which seems reasonable to me):

expat_dep = dependency('expat', method: ['auto', 'find_library'], fallback: ['expat', 'expat_dep'])

and if we want to just include "find_library" in auto by default (like we do for "extraframework" on macOS):

expat_dep = dependency('expat', fallback: ['expat', 'expat_dep'])

Doing this will make using meson much easier and make it easier to write more poratble builds.

[eli-schwartz]

The macOS SDK is not malformed. It is designed to not need pkg-config files. Everything "just works" with -isysroot.

Neither Meson dependency('expat') nor cmake find_package(expat CONFIG) work with that.

Invoking the compiler with -lexpat and assuming it works, does work.

This particular issue makes transitioning from autotools based builds to meson based builds overly complex, and it seems that meson should be able to handle this with a bit of extension.

From this statement of yours, I assume you were NOT using autotools PKG_CHECK_MODULES([EXPAT], [expat]) as that would have the same issue. Perhaps you were using autotools AC_CHECK_LIB([expat]) (equivalent to meson cc.find_library) combined with AC_CHECK_HEADERS([expat.h]) (equivalent to meson cc.has_header). In that case, nothing is stopping you from doing precisely this with Meson too. :)

Please update the documentation to better reflect what "system" really is becuase it reads as though it should "just work" for this case:

It's entirely possible the documentation can be improved. Nevertheless, my interpretation of that page followed this section:

For dependencies without specific detection logic, the dependency method order for auto is:

• pkg-config
• cmake
• extraframework (OSX only)

This clearly states that any other detection method is the domain of "specific detection logic", a.k.a. "dependencies with custom lookup functionality", which must be listed on that page. Which expat is not.

Please support finding expat via "system" the same way you do zlib. They're in the exact same boat here, so it would certainly make sense for meson to support expat the same way it does zlib.

This could indeed be done. zlib is an example of a dependency that is often provided by a base OS, but not all of them actually install all parts of upstream zlib. macOS is one of those systems, but so are the BSDs.

zlib was implemented... by request. In #2654 -- which includes mention of:

Are there other libraries that should be treated in the same way?

I have no idea [...] By randomly comparing fink's and Apple's versions, i suspect also libexpat, libxml2 [...]

Leading to #6421 which mentions:

I wouldn't want to do this for every dependency, but zlib is really popular so having a built-in solution seems better to me.

Personally, I'm okay with adding stuff that might be shipped as part of a base OS without pkg-config files, though I would not be fine with adding "every software out there". We want to encourage projects to use pkg-config, after all. But fighting with Apple is probably fruitless.

Please support a method for finding the library with a simple link check (eg: 'find_library') which basically does the same thing as cc.find_library().

I assume you mean a dependency method. This is not robust, since library names may differ from package names and simply checking linkability does not tell you whether the development headers are installed.

The generic tracking issue for this is in #2945, and a WIP attempt to implement it is in #8699. Some parts of it have landed as part of incremental PRs, for example since 0.60.0 you can now do dependency('expat', 'EXPAT') which will try both names, the second name is there to handle cmake's builtin FindEXPAT.cmake that conveniently has the wrong name compared to the official pkg-config naming, which is "expat", and also the official cmake naming, which is expat-config.cmake. Meson can find either one using the name "expat" and an upstream expat-project-installed config file, but if expat is artificially stripped of those configs, then the second name might still find it.

In general, we want this functionality. It's just not there yet.

Please support passing in an array for method.

auto means try all methods, so your proposed use of an array would feel a bit strange. That being said, there may indeed be a use case for trying multiple methods, but not all methods. I haven't thought deeply on the topic.

[eli-schwartz]

If you are interested in implementing any new features yourself, but I'm particularly thinking of adding an "expat" custom dependency lookup, I am happy to provide help and guidance.

Also if you have proposed tweaks to the documentation to clarify things, I am happy to review those proposals, but it seems clear enough to me that I'm probably not the best judge of how to improve that.

If not, I can probably find time to add an expat system dependency myself.

[dcbaker]

historically, I actually looked for the issues after I wrote the zlib system dep because I saw everyone writing the dependency to find library dance separately but wanted to be a good issue’s citizen :)

I’m happy to help with the other deps that help xorg/fro related projects @jeremyhu if you let me know which ones make sense and can help test on macOS.

[jeremyhu]

Thanks folks. I just got done updating my build scripts for XQuartz to use meson for the fdo projects that support it, and expat was the only dependency in this boat (for cairo and fontconfig). I'll look into wording changes that would have helped me out (now that I understand what "system" is).