macOS linker warnings in macOS ventura

[pablogsal]

Turns out ld shipped with Xcode 14+ emits a warning when using -undefined dynamic_lookup.`

ld: warning: -undefined dynamic_lookup may not work with chained fixups

Some investigation reveals that in fact -undefined dynamic_lookup doesn't work when:

• Link a shared library with the option
• Link it with a program that uses the chained-fixup introduced from macOS 12 and iOS 15

This is because -undefined dynamic_lookup uses lazy-bindings and they won't be bound while dyld fixes-up by traversing chained-fixup info.

However, we build extension modules with as bundles (-bundle) and they are loaded only through
dlopen, so it's safe to use -undefined dynamic_lookup in theory. So the warning produced by ld64 is
likely to be a false-positive.

ld64 also provides the -bundle_loader <executable> option, which allows resolving symbols defined in the executable symbol table while linking. It behaves almost the same with -undefined dynamic_lookup, but it makes the following changes:

• Require that unresolved symbols among input objects must be defined in the executable.
• Lazy symbol binding will lookup only the symbol table of the bundle loader executable. (-undefined dynamic_lookup lookups all symbol tables as flat namespace)

See "New Features" subsection under "Linking" section for chained fixup
developer.apple.com/documentation/xcode-release-notes/xcode-13-release-notes for more information.

Also, check the same problem in ruby where most of this information is taken from.

[pablogsal]

CC @ambv @ned-deily

[pablogsal]

Unfortunately this warning also propagates to all extension modules built with the given Python.

[ned-deily]

@ronaldoussoren

[ronaldoussoren]

-bundle_loader doesn't work with shared library builds (including the framework build), as the option does exactly as described in the documentation: It looks for symbols in the loader executable, not in shared library dependencies of that binary. I found this when looking into -bundle_loader to get link time errors when using non-existing symbols instead of only getting those when using an extension.

Linking with libpython is also problematic because on macOS shared libraries are referenced by absolute path which means linking with libpython ties the extension to a specific interpreter (e.g. makes it impossible to build an extension using the Python.org installer and load it using a homebrew install). For the user this will result in seemingly random crashes (due to libpython being loaded twice while only one is initialised).

I haven't spend time on looking into this particular issue yet, with some luck there's another option beyond disabling chained lookups.

[CsBigDataHub]

FYI.. https://issues.guix.gnu.org/issue/57849

Looks like using -Wl,-w will help.

[ronaldoussoren]

FYI.. https://issues.guix.gnu.org/issue/57849

Looks like using -Wl,-w will help.

That doesn't really help, it just suppresses the warning.

[keith]

You can pass -Wl,-no_fixup_chains to disable them entirely for now, but that flag is undocumented and who knows if it will exist in the future.

[pablogsal]

You can pass -Wl,-no_fixup_chains to disable them entirely for now, but that flag is undocumented and who knows if it will exist in the future.

It will likely won't because chained fixups are the new linker information for the dynamic symbol tables. So we will be just delaying the inevitable

[ronaldoussoren]

Another workaround is to target an older deployment target (before 11.0), but that's also not a long term solution.

[ronaldoussoren]

For static builds using -Wl,-bundle_loader,$(BUILDPYTHON) -Wl,-undefined,error appears to work. I've manually patched the Makefile for some extensions and that does result in a build without warnings and I can load those extensions. What I've not checked yet is if those extensions work with a python installed into a different prefix.

This doesn't work with dynamic builds though, even when using BLDLIBRARY= -L. -Wl,-reexport-lpython$(LDVERSION) when linking $(BUILDPYTHON).

Note that just linking extensions with libpython isn't a realistic option because that results in C extensions that target a specific python prefix and hence would loose interoperability for wheels build with our installer and homebrew.

[ronaldoussoren]

What does work for a dynamic build (again with a tweaked $(BLDLBIRARY): -Wl,-flat_namespace -Wl,-bundle_loader,./python.exe -Wl,-undefined,error, that is add -Wl,-flat_namespace. That's not ideal though because the default two-level namespaces are more efficient (and faster).

Something similar should also work for framework builds (but using a different reexport flag), but I haven't tested this yet. Mostly because I'm not happy with the link flags I've used for a shared build.

All of this was tested by manual patching of the Makefile, I haven't looked into tweaking the configure script and makefile template yet.

[wjakob]

Hi — this issue has also been raised in the pybind11 and CMake projects, including a fix similar to what was suggested above by @ronaldoussoren.

Links:

• pybind11 issue tracker ticket: macOS: disable fixup chains when linking extension modules pybind/pybind11#4301
• CMake issue tracker ticket: https://gitlab.kitware.com/cmake/cmake/-/issues/24044

All of this raises a few questions (pybind/pybind11#4301 (comment)) that would be good to figure out together with somebody who is familiar with the intricacies of Darwin's dynamic linker.

[pablogsal]

Hi @wjakob,

We are currently waiting for some clarifications from Apple. Will follow up with anything we find.

[mathstuf]

Anyone with Apple clout could push this PR into their attention sphere: apple-opensource/ld64#1

It basically says "here's a library; make any symbol that is resolved here dynamic_lookup and in the flat namespace". Which is…exactly what is wanted here.

[wjakob]

It basically says "here's a library; make any symbol that is resolved here dynamic_lookup and in the flat namespace". Which is…exactly what is wanted here.

Correct me if I'm wrong, but it seems to me like this PR targets a different use case: it enables dynamic_lookup selectively for chosen symbols to avoid the "sledgehammer" solution of changing the defaults for all undefined symbols.

While the issue discussed here is that the very mechanism that drives dynamic_lookup no longer works in binaries using the new "chained fixups" linker feature (which is automatically enabled for newer macOS deployment targets).

[wjakob]

One more data point in case it is useful: explicitly enumerating all relevant Python symbols using the -U command line arguments (wjakob/nanobind@bb211d1) also works and the warnings are gone. But it leaves me wondering:

• Does it actually fix the issue? The manpage says that dynamic_lookup is still used in that case.
• The linker gets a huge command line argument (3.5KB), which seems quite ugly and would need to be tweaked for every Python version. I could not find a -U_from_file <filename> argument to batch-inform ld about the symbols to accept as being undefined.

[carlocab]

I could not find a -U_from_file <filename> argument to batch-inform ld about the symbols to accept as being undefined.

ld response files might be useful here (though it is documented under Rarely used Options):

     @response_file_path
             Inserts contents of file at response_file_path into arguments. This allows for linker command line args to be store in a
             file.  Note: ld is normally invoked through clang, and clang also interprets @file on the command line.  To have clang
             ignore the @file and pass it through to ld, use -Wl,@file.

[ambv]

Thanks @wjakob, this is a clear actionable item for us. I think I prefer having an explicit list like this and retaining chained fixups.

However, I imagine we will likely want to create this symbol list ourselves "just-in-time" in our build tooling to future-proof it without creating an external dependency. 3.12 is still in flux so the list might change, and 3.13 is around the corner.

[mathstuf]

A more future-proof way of handling missing symbols would be poking the right people at Apple to support the functionality in this PR which allows one to say "here is a library for which any symbol found should be treated as if given via -U and do not add the library to the load commands".

[pablogsal]

However, I imagine we will likely want to create this symbol list ourselves "just-in-time" in our build tooling to future-proof it without creating an external dependency. 3.12 is still in flux so the list might change, and 3.13 is around the corner.

We can create it and package it (and use when building the interpreter) but extensions need to pick it up using whatever build system they are using, no?

[wjakob]

We can create it and package it (and use when building the interpreter) but extensions need to pick it up using whatever build system they are using, no?

That was the reason for packaging it up in my binding project. Of course, the implication wasn't that CPython should rely on this external link.

Probably the most portable way would be if the Python binary itself could be queried to list its own exported symbols.

[ronaldoussoren]

However, I imagine we will likely want to create this symbol list ourselves "just-in-time" in our build tooling to future-proof it without creating an external dependency. 3.12 is still in flux so the list might change, and 3.13 is around the corner.

We can create it and package it (and use when building the interpreter) but extensions need to pick it up using whatever build system they are using, no?

With some luck most build systems would pick up flags from python3-config or the equivalent information in sysconfig.

#103306 contains a related discussion about using -U flags, although as only one of the options to drop -undefined dynamic_lookup to improve the build experience (e.g. no silent failure due to missing symbols). I'll close that other issue because this issue is a lot clearer about the way forward.

[ronaldoussoren]

Generating the file should be easy enough, e.g.:

libpython.sym: $(LDLIBRARY)
     nm $(LDLIBRARY) | grep ' T ' | awk '{ nm = substr($3, 2); print("-U ", nm) }'  > libpython.sym

I have not yet tried to actually use this though, let alone thought about the correct way to integrate this into the installation and build system.

This will list more symbols than are technically part of the API, but keeps us closer to the current linking behaviour w.r.t. using private but exported symbols. That's not ideal, but does avoid bug reports about link errors on macOS that don't happen on other unix-y platforms.

[Dave-Allured]

I submitted an upstream patch to GNU libtool, to add -Wl,-no_fixup_chains for Mac builds. If accepted, this will hopefully propagate to many packages using libtool, and mitigate that lookup warning. I welcome any feedback, here or on that GNU ticket.

This would mean that packages wanting to use chained fixups would need to override this new libtool default.

[Dave-Allured]

Sorry, I mean to include that link:
https://savannah.gnu.org/support/?111069