macOS: ability to generate library as a "bundle" instead of dylib

[hryx]

macOS has two formats for shared libraries, both of which ld can create:

1. Dylib: File type MH_DYLIB, ends in .dylib, currently supported by Zig. Intended as something to be linked against with e.g. -lfoo.
2. Bundle: File type MH_BUNDLE, conventionally ends in .so. Intended as an application plugin. Instead of linking others against this, you can provide an executable or library that intends to load this bundle (the "loader") as a source of symbols to resolve. This is because the bundle will likely use an API provided by this loader program/library.

On Linux, there is no distinction between these two use cases.

Use case

Lua can load binary extension libraries with require("foo"), and it relies on dlopen() which is able to open both .dylib and .so files on macOS. But Lua's default scheme for locating a library file for module "foo" only looks for foo.so. It's possible to override the search scheme in the Lua application by setting its package.cpath variable, but it would be good if Zig could output a library that targets the default search behavior so the application doesn't have to expect this.

Related to this use case is one more (unfortunate) Lua default, which is that it doesn't look for libfoo.so, but rather foo.so. This is similar to how PAM plugins do not start with lib. That issue is covered by #2231.

Proposal

Add new zig build-lib CLI flags and std.Build.CompileStep fields to support choosing the output format. The choice between bundle and dylib could be represented as either a boolean (true to opt into the "bundle" format output) or an enum of dylib/bundle, defaulting to the current format in both cases. Here, I'll use a boolean for simplicity, but either seems fine.

Add a CompileStep field bundle: bool = false,. (This name matches the ld flag, but maybe there's a better one) When computing the output file name, check this field to determine whether to use the extension .dylib (false) or .so (true).

Add a CompileStep field bundle_loader: ?[]const u8 = null,. When set, symbols in the library are resolved against the Mach-O object at that path.

Add zig build-lib link options (these are named after their ld counterparts):

  -bundle                (Darwin) produce a bundle (.so) shared library instead of a dylib
  -bundle_loader object  (Darwin) check bundle against executable or library to resolve symbols

These option should be passed to zld, which should add defined symbols from the object found from the -bundle_loader argument, and should change the Mach-O headers if the output is a bundle. If using ld, the flags can be passed verbatim.

As for filename, this proposal only covers how to determine the default output filename, not how it should interact with custom/specified filename (#2231). I'm unclear if that proposal affects the entire basename or just the part before the extension, and how it interacts with versioned libraries.

Alternatives and caveats

After exploring, I don't really get what the point of the bundle format is. But I'll leave the proposal here anyway as a place for discussion and further exploration.

The -bundle_loader option is used to resolve symbols, but this can also be done with -l (see my snippet below, from before I discovered this option). The only difference I've been able to find is that using -bundle_loader does not require linking further libraries, e.g. you don't need to also link libSystem.dylib if you don't use libc functions. I speculate that if you are writing a plugin for an executable instead of a library, you have to use -bundle_loader instead of -l, but I can't confirm that.

On macOS, dlopen() is capable of opening both dylibs and bundles. So far the only examples I've seen of plugin loading on macOS use dlopen(), so maybe it's not worth the added complexity to support both formats.

My issue with Lua specifically could be solved by #2231 alone. Are there examples of macOS applications that load bundles in a way that requires the bundle format specifically?

[kubkon]

I have two observations/comments for this: 1) does Apple's ld support generating .so instead of .dylib? And 2) this very much sounds like Lua's problem and not Zig's, especially if the former is false. If the former is true, it would be good if you could provide example invocations of Apple's linker showing it generating both variants.

[kuon]

As a seasoned lua developer I can confirm this particular lua problem is quite well known. It can be fixed on lua's side with altering cpath, like so:

package.cpath = package.cpath .. ";?.dylib"
require("mylib")

But, strickly speaking, it might not be a bad idea to support .so for zig as the correct library type for lua is a "loadable bundle" (a .so).

On macos, the darwin linker can generate both .dylib and .so and they are different in the way that they work. .dylib are intended to be linked to with ld like a static library and .so are intended to be used with dlopen. (this is a bit of a simplification but is the gist of it)

The linked flag for .so is -bundle IIRC.

This SO answer has some more details.

https://stackoverflow.com/questions/2339679/what-are-the-differences-between-so-and-dylib-on-macos

[hryx]

Quick demo of the whole building -> linking -> loading flow, in case it helps.

thing.zig:

// Minimal subset of the Lua C API
const Lua = opaque {
    const pushNumber = lua_pushnumber;
    extern fn lua_pushnumber(*Lua, f64) void;
};

// Called by Lua when the library is loaded with dlopen()
export fn luaopen_libthing(lua: *Lua) c_int {
    lua.pushNumber(1234);
    return 1;
}

pub fn panic(_: []const u8, _: ?*@import("std").builtin.StackTrace, _: ?usize) noreturn {
    @breakpoint();
    unreachable;
}

Build and link both a dylib and a bundle:

$ zig build-obj thing.zig
$ ld thing.o -dylib -o libthing.dylib -llua -lSystem -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
$ ld thing.o -bundle -o libthing.so -llua -lSystem -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
$ file libthing.*
libthing.dylib: Mach-O 64-bit dynamically linked shared library x86_64
libthing.so:    Mach-O 64-bit bundle x86_64
$ otool -hv libthing.*
libthing.dylib:
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64   X86_64        ALL  0x00       DYLIB    15       1016   NOUNDEFS DYLDLINK TWOLEVEL NO_REEXPORTED_DYLIBS
libthing.so:
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64   X86_64        ALL  0x00      BUNDLE    14        976   NOUNDEFS DYLDLINK TWOLEVEL

Load the bundle with Lua's default search patterns:

$ rm libthing.dylib # leave only the .so
$ lua -e 't = require("libthing"); print(t)'
1234.0

[andrewrk]

Before I mark this as accepted, I think this one needs an associated patch as to what exactly is being proposed here. For example, is the proposal to solve this in the CLI, the build system, or both? Also how does it relate to the more general problem of choosing the basename for build artifacts.

[hryx]

I've added some more detail to the description, although I'm mad fuzzy on the overall value now. I don't have much experience in this area and would love if there were some Darwin devs who could shed some more light on when a bundle is actually necessary instead of a dylib.

[kubkon]

Please note that this is also a feature request for the MachO linker which currently cannot generate an MH_BUNDLE.

[kubkon]

Also, I haven't found anywhere a solid link between .so and MH_BUNDLE. In fact, I've seen suggestions that the preferred extension is .bundle. For example stackoverflow thread.

[hryx]

Also, I haven't found anywhere a solid link between .so and MH_BUNDLE.

That is true. To clarify, my main data points are the Lua thing and Wikipedia, from which it sounds like the extension is arbitrary and some people have chosen .so out of familiarity, while .bundle is recommended.

For example stackoverflow thread.

Thanks, that's educational. At this point I'm confused about the point of bundles because it sounds like they're strictly more limited than dylibs. And to reiterate, I'm now fairly certain that #2231 would solve my personal use case.

[urso]

This issue is not Lua only specific, but might be a general problem for plugin authors writing extensions/plugins for Applications that load their extensions as shared libraries/bundles on Mac.

Next to the -bundle flag one should also pass the -bundle_loader flag to the linker to ensure that symbols are resolved correctly. From the linker man page:

     -bundle_loader executable
             This specifies the executable that will be loading the bundle output file being linked.  Undefined symbols from the bundle are checked against the specified executable like it was
             one of the dynamic libraries the bundle was linked with.

For example I'm currently working on Postgres extensions. When writing an extension in C one needs to pass the following options (automatically done pgxs - postgres extension build scripts): -bundle -bundle_loader path/to/postgres.

Now, if you do not use the -bundle_loader option you get a wrongly linked library. The problem at hand is that postgres defines hash functions like hash_create, hash_destroy, hash_search and so on. But libSystem also exports those symbols.
Because the linker as used via zig is not aware of the postgres binary, it will happily link the hash table functions against libSystem, which leads to segmentation faults at runtime.

The suffix used in Postgres is actually .dylib, not .so. But in the end I don't care about the default suffix as I have to rename the linked file anyways. But this is something plugin authors have to deal with anyways, because their target application mandates the naming scheme, not macOS.