Ship versioned libc headers on macOS #10215

kubkon · 2021-11-24T20:24:04Z

Previously, we would ship a singular set of libc headers per macOS CPU architecture. This PR rectifies it by now shipping a separate set of headers per macOS version AND CPU architecture. The headers are deduplicated among themselves following a 3 layer approach such that:

layer 1: x86_64-macos.10 x86_64-macos.11 x86_64-macos.12 aarch64-macos.11 aarch64-macos.12
layer 2: any-macos.10 any-macos.11 any-macos.12
layer 3: any-macos-any

where each subsequent layer contains common headers from the previous layer. More on this can be found in the discussion in ziglang/fetch-them-macos-headers#15.

Changes required to make this work include:

treating macOS target separately in Compilation.detectLibCIncludeDirs (dab6414)
temporarily tweaking std.Target.Os.defaultVersionRange to also accept std.Target.Cpu.Arch enum as there is currently a mismatch between the minimum version of macOS supported by x86_64 and aarch64 (the former is 10.15, while the latter is 11.6) (baf4609)
do not rely on headers shipped with Zig if we are building natively AND we have detected an SDK using xcrun --sdk macosx --show-sdk-path - turns out that it is possible to have SDK 11.1 on macOS 10.15, or SDK 12.0 on macOS 11.6. I'm guessing that Apple offers devs on the older OSes to target newer / latest OS too.
version libSystem.B.tbd shipped with Zig to match the versions that Zig currently supports

kubkon · 2021-11-24T20:24:25Z

cc @slimsag

andrewrk · 2021-11-24T20:41:00Z

Looks good so far! There's one more place you'll need to edit in Compilation.zig for the generated builtin.zig source.

`fetch-them-macos-headers` gitrev 7036517cc6a9aa154e7aef4c4593b5c4a5143ed4

This is mainly because arm64 macOS doesn't support all versions supported by x86_64 macOS. This is just a temporary thing until both architectures support the same set of OSes.

This means that I am purposefully regressing linking iOS and related which will require manual specification of the sysroot path, etc.

This is handled before by detecting and adding SDK path which is a centralised point for the native libc installation on darwin.

We will mimick the same solution as with the headers: * `libSystem.10.tbd` * `libSystem.11.tbd` * `libSystem.12.tbd` and so on...

kubkon · 2021-11-25T16:42:35Z

Looks good so far! There's one more place you'll need to edit in Compilation.zig for the generated builtin.zig source.

Hmm, are you sure anything needs updating in Compilation.zig? I don't see any use of defaultVersionRange in the generated builtin.zig source there. It seems to be using a direct constructor with already inferred version ranges for the OS.

kubkon · 2021-11-25T16:49:35Z

Ready for review! I've fixed one of the oversights I made when adding better iOS, iPadOS, etc. support to Zig. Namely, since all of these represent cross-compilation targets from macOS, if you decide to cross-compile to any such target, you will need to manually specify relevant sysroot and syslib / framework paths on the command line and in your build.zig file. This behaviour is now consistent with any other cross-compilation target in Zig I believe. The auto-inferred SDK paths are only used if we're compiling something natively (from host to host). So, to build for iPhone simulator on the M1 for example, you will need to spell out something like:

$ zig build-exe test.zig -target aarch64-ios-simulator --sysroot $(xcrun --sdk iphonesimulator --show-sdk-path) -L/usr/lib -F/System/Library/Frameworks

But, if you're building the same test.zig natively, this suffices:

$ zig build-exe test.zig

kubkon · 2021-11-25T16:54:39Z

Finally, you are now free to target any of the supported macOS versions for cross-compilation using version ranges. For instance, in order to target Catalina from Monterey, all you need to do is:

$ zig cc hello.c -target x86_64-macos.10

Similarly, if we want to target M1 Monterey from a Catalina-based Intel MBP, we can do:

$ zig cc hello.c -target aarch64-macos.12

Tl;dr versioning is currently based on major macOS revisions - 10 for Catalina, 11 for BigSur, 12 for Monterey. If this is confusing, or if Apple decides to use minor versioning again between different OS cuts (10.14 vs 10.15), we can add that in too.

slimsag · 2021-11-25T17:49:07Z

src/Compilation.zig

        };

        const sysroot = blk: {
            if (options.sysroot) |sysroot| {
                break :blk sysroot;
-            } else if (darwin_can_use_system_sdk) {
+            } else if (darwin_use_system_sdk) {
                break :blk try std.zig.system.darwin.getSDKPath(arena, options.target);


I think that this change might fix an issue I've been meaning to file for a while now, #10217, specifically when specifying a cross-compilation target. WDYT?

Yeah, I hope that's the case too actually. Would you mind checking after this gets merged, or building zig directly from this branch?

slimsag

Looks quite good to me!

andrewrk · 2021-11-25T20:28:18Z

CI failure:

error(link): undefined reference to symbol '___ulock_wait2'
error(link):   first referenced in '/workspace/zig-cache/o/f13b7e43e90bbe6475a7f932d7504a84/test.o'
thread 51767 panic: attempt to unwrap error: UndefinedSymbolReference
/workspace/src/main.zig:216:9: 0x30c4488 in main.mainArgs (zig1)
        return buildOutputType(gpa, arena, args, .zig_test);
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/src/link/MachO.zig:890:13: 0x32a8643 in link.MachO.flushModule (zig1)
            return error.UndefinedSymbolReference;
            ^
/workspace/src/link/MachO.zig:401:5: 0x328b287 in link.MachO.flush (zig1)
    try self.flushModule(comp);
    ^
/workspace/src/link.zig:541:23: 0x3253083 in link.File.flush (zig1)
            .macho => return @fieldParentPtr(MachO, "base", base).flush(comp),
                      ^
/workspace/src/Compilation.zig:1877:5: 0x324b651 in Compilation.update (zig1)
    try self.bin_file.flush(self);
    ^
/workspace/src/main.zig:2689:5: 0x31d90ed in main.updateModule (zig1)
    try comp.update();
    ^
/workspace/src/main.zig:2372:21: 0x30f147a in main.buildOutputType (zig1)
        else => |e| return e,
                    ^
???:?:?: 0x30c5c22 in ??? (???)
/workspace/src/stage1.zig:48:43: 0x30c3d12 in main (zig1)
        stage2.mainArgs(gpa, arena, args) catch unreachable;
                                          ^
???:?:?: 0x8379fe8 in ??? (???)
The following command terminated unexpectedly:
/workspace/_debug/staging/bin/zig test /workspace/lib/std/std.zig --test-name-prefix std-x86_64-macos-gnu-Debug--multi  --cache-dir /workspace/zig-cache --global-cache-dir /root/.cache/zig --name test -target x86_64-macos-gnu -mcpu x86_64 -I /workspace/test --zig-lib-dir /workspace/lib

… version

kubkon · 2021-11-25T21:21:30Z

CI failure:

error(link): undefined reference to symbol '___ulock_wait2'
error(link):   first referenced in '/workspace/zig-cache/o/f13b7e43e90bbe6475a7f932d7504a84/test.o'
thread 51767 panic: attempt to unwrap error: UndefinedSymbolReference
/workspace/src/main.zig:216:9: 0x30c4488 in main.mainArgs (zig1)
        return buildOutputType(gpa, arena, args, .zig_test);
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/lib/std/heap.zig:150:9: 0x30be120 in std.heap.CAllocator.resize (zig1)
        return error.OutOfMemory;
        ^
/workspace/src/link/MachO.zig:890:13: 0x32a8643 in link.MachO.flushModule (zig1)
            return error.UndefinedSymbolReference;
            ^
/workspace/src/link/MachO.zig:401:5: 0x328b287 in link.MachO.flush (zig1)
    try self.flushModule(comp);
    ^
/workspace/src/link.zig:541:23: 0x3253083 in link.File.flush (zig1)
            .macho => return @fieldParentPtr(MachO, "base", base).flush(comp),
                      ^
/workspace/src/Compilation.zig:1877:5: 0x324b651 in Compilation.update (zig1)
    try self.bin_file.flush(self);
    ^
/workspace/src/main.zig:2689:5: 0x31d90ed in main.updateModule (zig1)
    try comp.update();
    ^
/workspace/src/main.zig:2372:21: 0x30f147a in main.buildOutputType (zig1)
        else => |e| return e,
                    ^
???:?:?: 0x30c5c22 in ??? (???)
/workspace/src/stage1.zig:48:43: 0x30c3d12 in main (zig1)
        stage2.mainArgs(gpa, arena, args) catch unreachable;
                                          ^
???:?:?: 0x8379fe8 in ??? (???)
The following command terminated unexpectedly:
/workspace/_debug/staging/bin/zig test /workspace/lib/std/std.zig --test-name-prefix std-x86_64-macos-gnu-Debug--multi  --cache-dir /workspace/zig-cache --global-cache-dir /root/.cache/zig --name test -target x86_64-macos-gnu -mcpu x86_64 -I /workspace/test --zig-lib-dir /workspace/lib

A bug in libstd, fixed in a041401.

andrewrk

I think this is a really big improvement for targeting macOS!

For those who aren't paying super close attention (and for myself when I type up release notes in a couple weeks):

Master branch, before this PR

macOS C headers & .tbd files total installation size: 6.1 MiB
- tarball size: 534 KB
supported triples:
- x86_64-macos.11 (Big Sur)
- aarch64-macos.11 (Big Sur)

After this PR is merged

macOS C headers & .tbd files total installation size: 13.1 MiB
- tarball size: 581 KB
supported triples:
- x86_64-macos.10 (Catalina)
- x86_64-macos.11 (Big Sur)
- x86_64-macos.12 (Monterey)
- aarch64-macos.11 (Big Sur)
- aarch64-macos.12 (Monterey)

…eting The latest Zig master supports specifying a specific macOS version for libc, via the target triple (ziglang/zig#10215): * x86_64-macos.10 (Catalina) * x86_64-macos.11 (Big Sur) * x86_64-macos.12 (Monterey) * aarch64-macos.11 (Big Sur) * aarch64-macos.12 (Monterey) Mach's `system_sdk.zig` can now download the relevant XCode framework stubs for Big Sur (11) and Monterey (12). We do not support Catalina (10) currently. By default, Zig targets the N-3 version (e.g. `x86_64-macos` defaults to `x86_64-macos.10`). Targeting the older version (`<arch>-macos.11`) is useful for compatability, it guarantees the produced binary will run on macOS 11+. Targeting the newer version can be useful if you wish to use newer APIs not available in previous versions. Fixes #102 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>

…eting The latest Zig master supports specifying a specific macOS version for libc, via the target triple (ziglang/zig#10215): * x86_64-macos.10 (Catalina) * x86_64-macos.11 (Big Sur) * x86_64-macos.12 (Monterey) * aarch64-macos.11 (Big Sur) * aarch64-macos.12 (Monterey) Mach's `system_sdk.zig` can now download the relevant XCode framework stubs for Big Sur (11) and Monterey (12). Although we don't have an SDK for Catalina (10) currently, we use the Big Sur (11) SDK in that case and it generally works fine. By default, Zig targets the N-3 version (e.g. `x86_64-macos` defaults to `x86_64-macos.10`). Targeting the minimum supported version is useful for compatability, it guarantees the produced binary will run on any later macOS version. Targeting the newer version can be useful if you wish to use newer APIs not available in previous versions. Fixes #102 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>

…eting The latest Zig master supports specifying a specific macOS version for libc, via the target triple (ziglang/zig#10215): * x86_64-macos.10 (Catalina) * x86_64-macos.11 (Big Sur) * x86_64-macos.12 (Monterey) * aarch64-macos.11 (Big Sur) * aarch64-macos.12 (Monterey) Mach's `system_sdk.zig` can now download the relevant XCode framework stubs for Big Sur (11) and Monterey (12). Although we don't have an SDK for Catalina (10) currently, we use the Big Sur (11) SDK in that case and it generally works fine. By default, Zig targets the N-3 version (e.g. `x86_64-macos` defaults to `x86_64-macos.10`). Targeting the minimum supported version is useful for compatability, it guarantees the produced binary will run on any later macOS version. Targeting the newer version can be useful if you wish to use newer APIs not available in previous versions. Fixes hexops/mach#102 Signed-off-by: Stephen Gutekanst <stephen@hexops.com>

kubkon requested a review from andrewrk November 24, 2021 20:24

kubkon mentioned this pull request Nov 24, 2021

libc: update macOS headers to macOS 12.0 Monterey #10202

Closed

andrewrk mentioned this pull request Nov 24, 2021

fatal error: 'libDER/DERItem.h' file not found ziglang/fetch-them-macos-headers#2

Closed

kubkon added 5 commits November 25, 2021 12:19

add libc headers for all supported macOS versions

2dcfa48

`fetch-them-macos-headers` gitrev 7036517cc6a9aa154e7aef4c4593b5c4a5143ed4

infer and match macos sdk version for libc headers

42a351e

Pass inferred cpu_arch to defaultVersionRange

c42439d

This is mainly because arm64 macOS doesn't support all versions supported by x86_64 macOS. This is just a temporary thing until both architectures support the same set of OSes.

Autofetch macOS SDK if native target on macOS only

1954cdc

This means that I am purposefully regressing linking iOS and related which will require manual specification of the sysroot path, etc.

Skip detecting native libc dirs on darwin

4270f23

This is handled before by detecting and adding SDK path which is a centralised point for the native libc installation on darwin.

kubkon force-pushed the macos-versions branch 2 times, most recently from 9db4d61 to f8eb840 Compare November 25, 2021 15:58

Version libSystem shipped with zig toolchain

3a3576d

We will mimick the same solution as with the headers: * `libSystem.10.tbd` * `libSystem.11.tbd` * `libSystem.12.tbd` and so on...

kubkon force-pushed the macos-versions branch from f8eb840 to 3a3576d Compare November 25, 2021 16:13

kubkon marked this pull request as ready for review November 25, 2021 16:42

slimsag reviewed Nov 25, 2021

View reviewed changes

slimsag approved these changes Nov 25, 2021

View reviewed changes

libstd: fix bug in std.Thread.Futex selecting incorrect default macOS…

a041401

… version

andrewrk approved these changes Nov 25, 2021

View reviewed changes

kubkon mentioned this pull request Nov 25, 2021

(regression) building macOS binary without XCode / Command Line Tools installed fails #10217

Closed

kubkon merged commit 2006add into master Nov 26, 2021

kubkon deleted the macos-versions branch November 26, 2021 00:08

slimsag mentioned this pull request Nov 26, 2021

update system_sdk to reflect new Zig macOS version targeting hexops/mach#102

Closed

3 tasks

slimsag mentioned this pull request Nov 27, 2021

glfw: update system_sdk to match latest Zig master macOS version targeting hexops/mach#107

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ship versioned libc headers on macOS #10215

Ship versioned libc headers on macOS #10215

kubkon commented Nov 24, 2021 •

edited

Loading

kubkon commented Nov 24, 2021

andrewrk commented Nov 24, 2021

kubkon commented Nov 25, 2021

kubkon commented Nov 25, 2021

kubkon commented Nov 25, 2021

slimsag Nov 25, 2021

kubkon Nov 25, 2021

slimsag Nov 25, 2021

slimsag left a comment

andrewrk commented Nov 25, 2021

kubkon commented Nov 25, 2021

andrewrk left a comment •

edited

Loading

Ship versioned libc headers on macOS #10215

Ship versioned libc headers on macOS #10215

Conversation

kubkon commented Nov 24, 2021 • edited Loading

kubkon commented Nov 24, 2021

andrewrk commented Nov 24, 2021

kubkon commented Nov 25, 2021

kubkon commented Nov 25, 2021

kubkon commented Nov 25, 2021

slimsag Nov 25, 2021

Choose a reason for hiding this comment

kubkon Nov 25, 2021

Choose a reason for hiding this comment

slimsag Nov 25, 2021

Choose a reason for hiding this comment

slimsag left a comment

Choose a reason for hiding this comment

andrewrk commented Nov 25, 2021

kubkon commented Nov 25, 2021

andrewrk left a comment • edited Loading

Choose a reason for hiding this comment

Master branch, before this PR

After this PR is merged

kubkon commented Nov 24, 2021 •

edited

Loading

andrewrk left a comment •

edited

Loading