darling: init at unstable-2023-05-02

[zhaofengli]

Description of changes

This PR adds Darling which is a Darwin/macOS emulation layer for Linux.

Patches applied:

• wrapgen: Fix address resolution in dynamic section darlinghq/darling#1355: Fixes the incorrect assumption during ELF parsing in the wrapper generator
• cmake: Make Python 2 bytecode pre-compilation optional darlinghq/darling#1356: Makes pre-compiling Python 2 bytecode optional, at the cost of performance
• libm: Remove unused rpath config darlinghq/darling#1358: Removes libsystem_m.dylib's reference to the store
• Fix sysconfdir config darlinghq/darling-nano#1: Fixes incorrect sysconfdir for nano
• Allow overriding xdg-user-dir executable darlinghq/darlingserver#10: Allows overriding the path to xdg-user-dir

Fixes #38628.

Installing

Set programs.darling.enable = true; in your NixOS config. This will install the setuid wrapper necessary to start Darling. Then, run darling shell.

Stuff To Try

See also https://docs.darlinghq.org/what-to-try.html

$ uname -a
Darwin host 19.6.0 Darwin Kernel Version 19.6.0 x86_64

# You can also use a real copy of Xcode (I haven't tried)
$ xcode-select --install
[...]

# Compile some code
$ clang hello.c

# Install Nix (does not fully work yet - more below)
$ curl -LO "https://github.com/zhaofengli/nix-on-darling/releases/download/initial-hack/nixos-system-x86_64-darwin.tar.xz"
[...]
$ tar xpf nixos-system-x86_64-darwin.tar.xz -C /
[...]
$ bootstrap-nix
Bootstrapping Nix...
installing 'nix-2.13.3'
building '/nix/store/ja3kqp1qwz66iqbkg50wzj264rppcnmg-user-environment.drv'...
installing 'nss-cacert-3.86'
building '/nix/store/27fw1kpgngj3q29ndphb30dw1vkhaxdz-user-environment.drv'...
Run the following command to make Nix work in the current shell:
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'

$ . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh

$ nix-instantiate --eval -E 'builtins.currentSystem'
"x86_64-darwin"

# Try other stuff
$ nix run nixpkgs#neofetch
                    c.'          zhaofeng@host 
                 ,xNMM.          -------------- 
               .OMMMMo           OS: macOS Catalina 10.15 Darling x86_64   
               lMM"              Kernel: 19.6.0 
     .;loddo:.  .olloddol;.      Uptime: 86 days, 4 hours, 9 mins 
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Shell: fish /nix/store/jawzvjj5x2iwv8zmz0905ix2v62a9h3r-neofetch- 
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    DE: Aqua 
 XMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor 
;MMMMMMMMMMMMMMMMMMMMMMMM:       WM Theme: Blue (Print: Entry, AppleInterfaceStyle, Does Not Exist 
:MMMMMMMMMMMMMMMMMMMMMMMM:       Terminal: shellspawn 
.MMMMMMMMMMMMMMMMMMMMMMMMX.      Memory: 27141MiB / 32034MiB 
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.
 'XMMMMMMMMMMMMMMMMMMMMMMMMMMk                           
  'XMMMMMMMMMMMMMMMMMMMMMMMMK.                           
    kMMMMMMMMMMMMMMMMMMMMMMd
     ;KMMMMMMMWXXWMMMMMMMk.
       "cooc*"    "*coo'"

Nix

The official Nix install script does not work on Darling due to its attempts to modify partitions and users on the system. Nix also has trouble clearing file attributes in the emulated filesystem.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[zhaofengli]

The previous patchShebangs . was too indiscriminate and caused scripts intended to be run inside the Darling sandbox to be patched as well. Also added a check to make sure that the Darling sysroot doesn't contain references to the Nix store (except for mldr).

[zhaofengli]

During the build, Darling produces a source-built SDK which is now installed into a separate output. This SDK can be used to cross-compile from Linux to macOS (see the NixOS test). I tried the resulting executable and it worked fine on a real M1 Mac mini under Rosetta 2.

Very hacky cross stdenv

let
  lib = import ./lib;

  pkgs = import ./. {
    crossSystem = lib.systems.elaborate (lib.systems.examples.x86_64-darwin // {
      darwinSdkVersion = "10.15";
      darwinMinVersion = "10.15";
    });

    overlays = [ (final: prev: {
      stdenv =
        if (prev.stdenv.hostPlatform == prev.stdenv.buildPlatform) then prev.stdenv
        else prev.stdenv.override {
          overrides = overrideStdenv;
        };
    }) ];
  };

  overrideStdenv = final: prev: let
    nativePkgs = prev.pkgsBuildBuild;

    llvm = nativePkgs.llvmPackages_latest;
    sdk = "${nativePkgs.darling.sdk}/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk";

    nativeStuff = {
      nativeTools = false;
      nativeLibc = false;
      nativePrefix = "";
      shell = nativePkgs.stdenvNoCC.shell;
      inherit (nativePkgs) coreutils gnugrep;
    };

    mergedBintools = nativePkgs.symlinkJoin {
      name = "bintools-unwrapped";
      paths = [ llvm.bintools-unwrapped ];
      postBuild = ''
        ln -sf ${nativePkgs.darling.sdk}/bin/x86_64-apple-darwin19-ld $out/bin/ld
      '';
    };

    bintools = prev.wrapBintoolsWith ({
      bintools = mergedBintools;
      libc = "${sdk}/usr";
    } // nativeStuff);

    cc = prev.wrapCCWith (rec {
      cc = llvm.clang-unwrapped;
      inherit bintools;
      libc = "${sdk}/usr";
      libcxx = null;

      isClang = true;

      gccForLibs = null;
      extraBuildCommands = ''
        echo "--sysroot=${sdk}" >> $out/nix-support/cc-cflags

        rsrc="$out/resource-root"
        mkdir "$rsrc"
        ln -s "${cc.lib or cc}/lib/clang/${cc.version}/include" "$rsrc"
        echo "-resource-dir=$rsrc" >> $out/nix-support/cc-cflags
      '';
    } // nativeStuff);

    stdenv = prev.stdenv.override {
      inherit cc;
      extraBuildInputs = [];
    };
  in {
    inherit stdenv;
  };
in pkgs

[zhaofengli]

@ofborg test darling

[AndersonTorres]

??

[zhaofengli]

If you are asking about the comments, these are supposed to indicate which components the wrapped libraries are used in. I'll make that clearer.