apple_sdks: init with 11.1, 11.3, 12.1, 12.3, 13.1, and 13.3

[ConnorBaker]

What's new?

Adds a new directory for apple-sdks and refactors Xcode directory. Both use a “builder” and “releases” pattern where a nix configuration file stores information about each release and a separate nix file contains the function to build each release. This pattern has proven useful in the CUDA work I’ve done where we need to support multiple SDKs.

Default Apple SDK for aarch64 has been bumped from 11.0 to 11.1.

Apple SDK versions 11.3, 12.1, 12.3, 13.1, and 13.3 are now available.

In addition, we now have a script to track Apple SDK releases (by parsing their sucatalog). The gen-frameworks.py script has been updated to accept command line arguments as well as support generating a public.nix file for each SDK release in the file generated by gen-apple-sdk-releases.py.

Current problems?

• The 13.3 SDK is mean to be paired with Xcode 14.3, which is in turn backed by LLVM/Clang 15, so there are some compile errors when using old versions of LLVM/Clang.
• Cleanly overriding the compiler used by stdenv and ensuring linking against the correct libc/libcxx.
  • See Tracking issue: Update Darwin bootstrap files #222717 and darwin.stdenv: Update to LLVM 15 and clang to 15.0.7 #229786
• There's a chance that jobs which have to rebuild perl for various stages of bootstrapping the stdenv compiler will fail if run in parallel. The specific error mentions miniperl. If encountered, limit the number of jobs to one until the bootstrapping is complete.

Docs

• Please see pkgs/os-specific/darwin/apple-sdks/README.md for the beginnings of some documentation around how frameworks are generated.

Miscellaneous notes

Notes on the SDK's mkStdenv in default.nix, courtesy of @reckenrode:

• While the minimum version of Darwin supported by the 13.3 SDK is 13.0, it’s possible to build against a newer SDK while targeting older platforms as long as you use an older deployment target and avoid (or dynamically check for) APIs specific to the newer SDK, so it's fine to set darwinMinVersion = "10.12".
• Although the 13.3 SDK expects Xcode 14.3, we don't need to specify xcodeVer = "14.3". Nor do we need to override the default Xcode set in pkgs/os-specific/darwin/Xcode/default.nix because it only serves to copy the Xcode into the store and there aren't any derivations in Nixpkgs which seem to use it.
• Because the 13.3 SDK is mean to be paired with Xcode 14.3, which is in turn backed by LLVM/Clang 15, there are some compile errors when using old versions of LLVM/Clang.
  • This is an open problem and requires further investigation
• We specify darwinSdkVersion = "13.3" so our SDK is chosen instead of the default when building stdenv.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[fgaz]

ofborg bot requested review from fgaz

I wonder where did it get my username from. I don't see any change involving me and I don't have any darwin machine

[ConnorBaker]

Okay! Did some fairly major refactoring of the 13.3 SDK to genericize it and make it easier to sub in other versions. I've also added a releases.nix with different versions of the SDK. @reckenrode what do you think? Thoughts on the direction it took?

[reckenrode]

Should releases.nix be in the SDK or up a level? I like the idea of making the SDK more generic, so new versions can be added more easily, but how would this work with other SDKs? Did you have to do a lot of manual manipulate of the files from the generated ones?

[ConnorBaker]

Should releases.nix be in the SDK or up a level?

It should, but I'm hesitant to refactor all of the SDKs at the moment, so I'm trying to keep changes localized to the SDK for 13.

I like the idea of making the SDK more generic, so new versions can be added more easily, but how would this work with other SDKs? Did you have to do a lot of manual manipulate of the files from the generated ones?

Thankfully, all of the 11.0+ releases seem to follow a similar pattern. I imagine each will have its own nix expression for public and private frameworks, as well as fixups, but that they'll have a lot in common.

If we wanted to move away from checking in the auto-generated public dependencies, we could instead make use of lib.optionals to conditionally add different frameworks depending on the SDK version.

I just updated the PR description with a bit of documentation for what the frameworks dir is for currently. Let me know if that's at all coherent!

[ConnorBaker]

I want to note that I was also able to get rid of some recursion and switch over to using callPackage more consistently -- not sure what kind of breakages that will cause (if any at all) because I can't build things for the SDK right now.

[reckenrode]

Should releases.nix be in the SDK or up a level?

It should, but I'm hesitant to refactor all of the SDKs at the moment, so I'm trying to keep changes localized to the SDK for 13.

Can it be at the top level without refactoring the other SDKs? That would allow them to be refactored later. Ideally, the SDKs should share as much of the build infrastructure to make adding new ones as painless as possible.

I like the idea of making the SDK more generic, so new versions can be added more easily, but how would this work with other SDKs? Did you have to do a lot of manual manipulate of the files from the generated ones?

Thankfully, all of the 11.0+ releases seem to follow a similar pattern. I imagine each will have its own nix expression for public and private frameworks, as well as fixups, but that they'll have a lot in common.

If we wanted to move away from checking in the auto-generated public dependencies, we could instead make use of lib.optionals to conditionally add different frameworks depending on the SDK version.

Would each dependency use lib.optionals? I’d fear that could get hard to read. Instead of using lib.optionals, could we adopt the override pattern? Each SDK would have an override function that receives the final set of dependencies and the unmodified set, then it returns a new set with whatever modifications it would need to make.

I just updated the PR description with a bit of documentation for what the frameworks dir is for currently. Let me know if that's at all coherent!

Looks good!

One other thing that comes to mind is making sure the system ncurses is excluded from the SDK. That’s not the case for the 11.0 SDK currently. It’s one of a few things that prevent Apple’s upstream cctools from building on aarch64-darwin. I grepped the current PR and don’t see it, so it appears it’s not excluded. Ideally, the 11.0 SDK would be updated/refactored and drop the system ncurses framework. Everything in nixpkgs should use the packaged ncurses.

There are a few private frameworks that also aren’t included. See #149692 (comment). If those could also be included, it might be possible to build the upstream cctools and fix the linker crash.

[domenkozar]

cc @thefloweringash

[reckenrode]

I like the new organization. If this means adding new SDKs is a lot easier, that would be fantastic.

As discussed on Matrix, MoltenVK is well-behaved and can be built against the latest SDK. It uses dynamic feature detection to run on older systems. It supports a couple of macOS versions (10.11 and 10.12) for which it is impossible to build using those platforms’ SDKs.

[reckenrode]

@ConnorBaker fyi, #241692 has been merged into staging and should hopefully make 23.11.

[reckenrode]

@ConnorBaker #234710 is complete. The Darwin stdenv update has been merged into master.

[heimalne]

Any updates on the >= 12.x versions? This seems to be the new target architecture for many packages in the Python ecosystem and if i remember correctly the first official version to support Arm64 macs.

[reckenrode]

Any updates on the >= 12.x versions?

I will be picking this up after I get the cctools and ld64 updates successfully building the Darwin blockers channel and take a look at the Darwin cross-compilation issue after those and the recent LLVM changes. The plan is to land an SDK refactor into a common pattern first then add the missing SDKs (10.13 through 14.x) with as many source-based components as possible without regressing compatibility.

Once newer SDKs are available, packages that need one can use overrideSDK to change the SDK version used to build.

This seems to be the new target architecture for many packages in the Python ecosystem and if i remember correctly the first official version to support Arm64 macs.

11.0 was the first release of macOS to support Apple Silicon. There are no plans currently to change the default SDK on aarch64-darwin, but x86_64-darwin will be updated to 10.13 or 10.14 (but most likely 10.13).

[gador]

@reckenrode did I read that correctly that you continue on this PR?
If so, that would be awesome! An update to the apple sdk system would be much appreciated! If you need some testing, let me know!

[reckenrode]

@gador That’s right (more or less). The first iteration will be just the current SDKs in the common pattern. I’d like to do it without causing rebuilds, but I’m doubtful it’s possible. It may be a continuation, or it may just incorporate some of the approach. I’ll post an update to the Darwin updates news thread on Discourse once I have something to test.

[grahamc]

Hey @reckenrode, I didn't mean to step on any toes with that flake. It was for an experiment, but I never managed to make it actually work. It'd definitely be better being part of nixpkgs, and we don't have any grand plans to maintain an apple sdk flake long term. Just trying to get out of a jam where we needed a newer SDK for a Swift program, but couldn't wait for a future merge, and it wasn't working as-is in the PR. Fwiw I emailed Connor Baker about this PR last night before I made the repo, but it didn't occur to me to email you first too. I'm sorry again.

[reckenrode]

Hey @reckenrode, I didn't mean to step on any toes with that flake. It was for an experiment, but I never managed to make it actually work. It'd definitely be better being part of nixpkgs, and we don't have any grand plans to maintain an apple sdk flake long term. Just trying to get out of a jam.

It’s not about whether it’s a flake or an alternative. It’s the lack of communication and roadmap, especially in light of recent controversies. Is this going to be the solution? Is DetSys going to drive that? What’s going on?

[grahamc]

There is no roadmap, and we're definitely not the people to decide if this is the solution, and we're not driving it, lol. Like I said: Just trying to get out of a jam where we needed a newer SDK for a Swift program, but couldn't wait for a future merge, and it wasn't working as-is in the PR.

[reckenrode]

@grahamc Thanks for the clarification. I think if I had known the context when I saw the repo, I’d have reacted better. It’s understandable that you needed a solution when the soonest anything could possibly land is after the 24.05 release. The current SDK situation is definitely not great.