Flake support, project reorganization, asahi overlay

[oati]

Resolves #34 (and #36 #37)
Supersedes #35
Supersedes #46 and closes #45 (probably)

This PR does a couple of things:

• separate the project into 3 directories: packages, nixos-module, and iso-configuration
• define a project-wide asahi overlay in packages/overlay.nix and update the nixos module and the installer to use it
  • this adds a hardware.asahi.overlay option to the module, which the module automatically appends to the system nixpkgs
• add a flake.nix, which replaces default.nix and pins.nix and outputs
  • the new overlay
  • the packages in the new overlay that were previously exposed by default.nix
  • the "m1-support" nixos module
  • an installer-bootstrap package for every system supported by nixpkgs, cross-compiling as needed
• fix an issue with the nixos module for flake-based configurations that access /boot/asahi as a non-flake input

CI and docs would need to be updated.

• CI changes needed:
  • replace nix-build -A installer-bootstrap-cross -j2
    with nix build .#installer-bootstrap -j2
    • (may need --experimental-features 'nix-command flakes' or configuration changes)
• documentation changes needed:
  • update the nix-build commands with flake-based commands
  • make cross-compilation instructions less specific to x86_64-linux
  • add basic flake usage to the installation instructions (maybe to the "m1 support updates" section)
    • access the module with inputs.nixos-m1.nixosModules.default
    • add /boot/asahi as a non-flake input
  • document the asahi overlay

I'm currently using the nixos module output of the flake for my system configuration, but the iso is yet to be tested.

[oati]

Hmm, including a mesa override in the overlay causes rebuilds for everything that depends on mesa, regardless of whether the experimental GPU option is actually enabled. The option ends up being kind of silly, since it doesn't actually change anything. I'll fix this tomorrow.

[oati]

Actually, I think this is good enough. Overriding mesa globally causes so many rebuilds that it probably shouldn't be easy to do so.

The overlay now lets you reference pkgs.mesa-asahi-edge from anywhere, so you could just override packages as needed, or add an overlay yourself to override globally:

nixpkgs.overlays = [ (final: prev: { mesa = final.mesa-asahi-edge; }) ];

(regarding #45)

[tpwrules]

While I appreciate the effort, this is a big change for the project and I'm not sure about it.

Primarily, I'm not ready to go flakes-only. Though popular, they are still experimental and require extra effort from users to set up, and the simplicity of the current install is useful. There are certainly things I could adjust for a better flakes experience but I don't regularly use them in this context so some suggestions here would be helpful.

I do think the code needs reorganization and was hoping #35 would provide a good solution to that, but it doesn't seem to have happened. I will probably put together a PR myself for this which cribs ideas from this PR and #35 and solicit feedback from you and the community on if that meets your needs. It's hard to accept a big PR like this one without having fiddled with it myself first.

As for the Mesa fixes, I have a better solution to that I am working on which avoids so much recompilation and hope to get it out soon.

[sequencer]

wondering is possible for these tricks to be a part of NixOS/nixos-hardware

[oati]

Primarily, I'm not ready to go flakes-only. Though popular, they are still experimental and require extra effort from users to set up, and the simplicity of the current install is useful. There are certainly things I could adjust for a better flakes experience but I don't regularly use them in this context so some suggestions here would be helpful.

The nixos module can still easily be used without using flakes.

I decided to go with flakes-only for the iso and package outputs because flakes solve many of the problems that default.nix tries to solve in more ad-hoc, partial, and/or impure ways.

Namely,

• flakes use pure evaluation, and thus have much better reproducibility. This helped me find and patch up "leaky impurities" in the project. Flakes enforce
  • pure handling of system and the lockfile, ensuring that all impurities are tracked in flake.lock
  • pure nixpkgs (preventing builtins.currentSystem, builtins.Env, etc. from leaking in)
    • better / higher-level abstractions for the installer with nixos.lib.nixosSystem instead of import (pkgs.path + "/nixos/lib/eval-config)"
  • outputs are always reproducible, unless you explicitly override the pinned inputs (e.g. nixos-m1.inputs.nixpkgs.follows = "my-nixpkgs";)
• flake.lock is automatically managed by the nix flake CLI (nix flake lock, nix flake update)
• flakes follow a standard output schema that integrate with nix-commands, so the output is more standarized / predictable
• flake commands, like nix build github:tpwrules/nixos-m1#installer-bootstrap, can be run from anywhere, without the user having to download the repo; so the iso build is a one-liner.

[oati]

If you'd prefer, (and if you end up agreeing that flakes would be the better solution,) I could get rid of the flake-parts dependency to make the simplest "basic nix code" flake, and figure out flake-compat to maintain the current installation instructions.

flake-compat (very popular) would let you use flake.lock in place of pins.nix, without needing to use flakes. This lets you keep one lockfile for both flake and non-flake users.

[oati]

As for just ensuring a better experience for flake users, the nixos module is easy enough to use as a non-flake input (specifically since the nixos module doesn't have any required project depnendencies):

imports = [
  "${inputs.nixos-m1-non-flake}/nixos-module"
];

You'd just have to get rid of the /. + in https://github.com/tpwrules/nixos-m1/blob/main/nix/m1-support/peripheral-firmware/default.nix#L24 so that /boot/asahi doesn't break as a non-flake input.

One small advantage (for the nixos module) for having the project be a flake, however, is that flake users would gain the ability to easily pin the asahi packages to the "working versions" specified by the nixpkgs in the flake lockfile.

[oati]

(to clarify) The flake in this PR does not require users to have a flake-based system configuration. The nixos module can still be imported manually, through a channel, or through a tarball. The PR merely adds a standardized and convenient way to import the module from a flake.

This PR does require either an extra CLI flag --experimental-features 'nix-command flake' (or a nix user config enabling these) in order to build packages or the iso with the pinned nixpkgs and rust-overlay.

However, this can be completely worked around with via flake-compat, in which case users will never have to touch nix flakes for any use case.

[tpwrules]

Thanks for the clarification on your intentions and the outcomes. Yeah, I agree the current situation is rather ad-hoc and clunky. You raise some good points about how flakes would make it cleaner. I'm not aware that there are any actual problems here at the moment in particular with regard to purity, but making it impossible to make those kinds of mistakes is good. I don't think flake-compat is necessary at this point, I'm not a never-flakes person either.

It's very important to me that a user be able to download an ISO off GitHub, boot it, and install quickly without having to download and rebuild a whole kernel on their system. Nix does make it easy, but it's still not a great time on the lower-end machines. Some of the impurities as I recall were there to make this work. AFAIK these should be possible to deal with when involving flakes but they do need to work properly.

Please give me a week or so to review and play with this in depth. Please also rebase on top of the latest release. I might do another release in a couple days once the recent nixpkgs staging merge makes it to the NixOS release channels.

[tpwrules]

wondering is possible for these tricks to be a part of NixOS/nixos-hardware

@sequencer Some people have tried this (and moving support for these machines into nixpkgs too) but I haven't kept much track and I don't think it's feasible until the kernel upstreaming matures a little more.

[oati]

Ok, I'm running into new issues with the recent changes on main: NixOS/nixpkgs#199162 NixOS/nix#5868

system.replaceRuntimeDependencies just fails in pure evaluation due to builtins.storePath, and therefore on flake-based nixos configurations. I haven't found a workaround.

[tpwrules]

Hm, that was another memory around impurity that is now surfacing. I used to have system.replaceRuntimeDependencies in the default config for another package. I don't think there is a workaround either.

I can add an option to overlay the new mesa I suppose but that's rather unfair for flakes users because essentially everything needed to be rebuilt.

[shaunsingh]

Instead of using fetchFromGitHub to grab the source for m1n1/kernel/etc, could we just put those as pinned inputs in the flake.nix, that way if users wanted to override the source for e.g. mesa to a newer version or fork for debugging they easily could

[oati]

I did consider that option while working on this.

One issue is that it would become much more difficult for non-flake users to incorporate the nixos module into their system, since the overlay would depend on this project's flake.lock.

Flake users can already pretty easily override sources with flake inputs via overlays:

final: prev: {
  mesa-asahi-edge = prev.mesa-asahi-edge.overrideAttrs (old: {
    src = inputs.mesa-asahi;
  });
}

[oati]

TODO: Try to get m1n1 and uboot-asahi to cross-compile without using pkgsCross.aarch64-multiplatform. Cross-compilation should be handled using the crossSystem and localSystem arguments of nixpkgs, which we can control from our flake.

This will make m1n1 and uboot-asahi cross-compile "natively" using nixpkgs, in the same way that everything else in the iso is cross-compiled (and other packages in the overlay).

Also, localPkgs in the kernel package is now always pkgs, since builtins.currentSystem is impure.
(I'm not sure if localPkgs does anything for us. The only system that we're building for is aarch64-linux, so we don't need the config to be reusable.)

[tpwrules]

The catch with m1n1 is that it has to be a native build so that the tools like the proxy client and command line come out for x86_64-linux but also a cross build so that the m1n1 stub is built for aarch64-linux. Might be possible to split this into two separate derivations?

I think your solution is workable for U-Boot though, it always expects to be run using an aarch64 stdenv.

The bit with the kernel package is necessary to support the "install a kernel from a cross-compiled ISO without rebuilding it" scenario. The comment might be a little bit incorrect but we do in fact build the config derivation "for" x86_64-linux when cross-building the ISO, even though the contents never change. There might be a better way to do that. The current clause there should be compatible with flakes as it specifically checks for currentSystem to exist and assumes you are running natively if not.

[oati]

In the absolute worst case, we could explicitly re-import nixpkgs with crossSystem and localSystem to avoid problems with pkgsCross.aarch64-multiplatform (which is I think itself a re-import of nixpkgs with crossSystem.config = "aarch64-unknown-linux-gnu", which causes problems because it's not the same as importing with crossSystem.system = "aarch64-linux").

As for the kernel config IFD, we can just replace builtins.currentSystem with localSystem as a pure alternative.

note: nixpkgs uses native builds when crossSystem = localSystem (in some sense). It doesn't (won't) cross-build from the same platform. (although I think there's an open issue to make everything always cross-build?)

[tpwrules]

Okay, this should be the final rebase. I am still interested in this, just had to make sure everybody's builds didn't break.

[oati]

Some notes about the default firmware directory:

builtins.pathExists is impure, and it (I think) always outputs false in pure mode. NixOS/nix#3998 (stale bot....)

Maybe we should set the default to "/boot/asahi" and override this in the installer configuration?

(edit: actually "/boot/asahi" as a default makes no sense in pure mode since it can't be accessed anyway.)

[tpwrules]

Yes, the fact that builtins.pathExists is always false in pure mode is fine. The documentation covers this case and provides the alternate path for the user to point somewhere their flake can access.

The reason for the two paths is evaluation in the installer ISO. The actual installer gets the firmware by running asahi-fwextract automatically after booting, the configuration options don't matter for it. But when the user runs nixos-install, their configuration will be evaluated when the ESP is mounted at /mnt/boot/asahi (assuming they followed the installation instructions) and that's when the second path option comes into play.

[oati]

I've incorporated the recent changes from main. a1e2ef9 8361211 might need testing for cross-builds, but I think it should be correct.

[tpwrules]

Thanks again for this work. I've been able to build and boot the cross ISO, and have built (but not tried to boot) the ISO natively. Actually installing doesn't quite work correctly yet but I am assuming the packages all still function.

We still need to address documentation too.

[tpwrules]

The rename is not complete. If you want I can push to your branch, or you can fix it. Then I will test again.

[tpwrules]

Please address the mentioned issues. I will then squash, merge, and update the documentation.

[tpwrules]

Is it safe to inherit the kernel package here too? Being able to directly build the kernel and the configured-for-boot u-boot/m1n1 was the reason I had the config initially exposed but it sounds like that causes problems in the flake.

If not I will figure it out later.

[oati]

The issue I had with outputting the kernel package was NixOS/nix#4265, which also breaks nix flake show. Luckily we should be able to get rid of the IFD using pure nix, as I've mentioned below.

We could also output the installer config, but I think it's not as clear where the config should go in the output, especially since we need to account for different possible build platforms, so I wanted to avoid it if possible.

[tpwrules]

This has definitely broken hardware.asahi.buildPkgs. I am pretty sure the above comment is wrong. The real issue is that the config.nix file does not make its way to the ISO. Thus, when the configuration is evaluated, the runCommand needs to be executed to get the complete source code, but runCommand expects to run on a hardware.asahi.buildPkgs system and fails.

I want to keep the localPkgs hack for now (we don't need lib = localPkgs.lib though) and fix it later. That option will still be broken as it was for flakes users before, but it will be fine for the install scenario.

[oati]

I think the solution is to replace the "config.nix" IFD with a pure nix regex parser. It seems like it'll be pretty straightforward, and it can be upstreamed to nixpkgs since nixpkgs also uses an IFD currently.

[oati]

opened #51

[oati]

I've added back the localPkgs workaround for now.

[ingenieroariel]

Thanks for all this work, I have replaced my local config with the latest version of this PR, at the end of this comment I paste a copy of my flake.nix on /etc/nixos.

FYI, The overlay strategy with the latest version of this flake gives:

error: infinite recursion encountered

       at /nix/store/mhx0418frlp5p3i20bk6b0m41klps12m-source/apple-silicon-support/packages/mesa-asahi-edge/default.nix:5:2:

            4|
            5| (mesa.override {
             |  ^
            6|   galliumDrivers = [ "swrast" "asahi" ];
(use '--show-trace' to show detailed location information)

{
  description = "A very basic flake";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/master";
    home-manager = {
      url = github:nix-community/home-manager;
      inputs.nixpkgs.follows = "nixpkgs";
    };
    rust-overlay = {
      url = github:oxalica/rust-overlay;
      inputs.nixpkgs.follows = "nixpkgs";
    };

    nixos-m1-overlay = {
      url = github:oati/nixos-m1/flake;
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.rust-overlay.follows = "rust-overlay";
    };

    };

  outputs = {
    self,
    nixpkgs,
    rust-overlay,
    nixos-m1-overlay,
    home-manager,
  }: let
    system = "aarch64-linux";
    
    pkgs = import nixpkgs {
      inherit system;
      config.allowUnfree = true;
      overlays = [ 
          rust-overlay.overlays.default
          nixos-m1-overlay.overlays.default
      ];
    };
    lib = nixpkgs.lib;
  in {
    nixosConfigurations = {
      nixos = lib.nixosSystem {
        inherit system;
        modules = [
          ./configuration.nix
          nixos-m1-overlay.nixosModules.default
          home-manager.nixosModules.home-manager
          {
            home-manager.useGlobalPkgs = true;
            home-manager.useUserPackages = true;
          }
        ];
         };
    };
  };
}

[ingenieroariel]

I have verified the latest version of this PR on a new (flake-based) configuration and it works.

This set up is very flexible for downstream users like me. Here is an example:

I let nix flakes update the rust overlay to latest and nixpkgs to latest, and it worked after adding this to my configuration.nix (blacklist-types is now blocklist-types).

  boot.kernelPatches = [
      {
        name = "edge-config";
        patch = null;
        # derived from
        # https://github.com/AsahiLinux/PKGBUILDs/blob/stable/linux-asahi/config.edge
        extraConfig = ''
          DRM_SIMPLEDRM_BACKLIGHT n
          BACKLIGHT_GPIO n
          DRM_APPLE m
          APPLE_SMC m
          APPLE_SMC_RTKIT m
          APPLE_RTKIT m
          APPLE_MAILBOX m
          GPIO_MACSMC m
          DRM_VGEM n
          DRM_SCHED y
          DRM_GEM_SHMEM_HELPER y
          DRM_ASAHI m
          SUSPEND y
        '';
      }
      {
        name = "blocklist-type";
        patch = builtins.fetchurl "https://patch-diff.githubusercontent.com/raw/AsahiLinux/linux/pull/109.diff";
      }
    ];