initrd: Opt-in bare bones systemd-based initrd #164943

ElvishJerricco · 2022-03-20T08:04:00Z

Third time's the charm.

Motivation

This PR adds an option, boot.initrd.systemd.enable, that causes NixOS to generate an initrd that uses systemd for PID 1. This brings several benefits:

Upstream units can be used to implement almost all of stage 1 logic, eliminating the need for the many lines of shell scripting NixOS has in its traditional stage 1.
Early boot becomes parallelized.
systemd-ask-password can be used to request credentials, providing a more streamlined way for early boot tasks to communicate with users.
Configuration of stage 1 becomes declarative in exactly the same way that systemd in stage 2 is declarative, instead of adding imperative shell scripts in a variety of places.

How this compares to my previous attempts

This version of a systemd-based initrd has three major improvements over the previous attempts.

It is an opt-in alternative to NixOS's traditional initrd, instead of a replacement.
It uses the same systemd option types that NixOS uses for stage 2 systemd configuration.
It will be implemented piecemeal, instead of in one large PR.
The produced initrd appears to be quite a bit smaller than the previous attempts for whatever reason, at just 9.1M (which is quite good compared to NixOS's tradtional initrd).

The third improvement is especially important. This PR only implements the core functionality needed to have a systemd-based initrd. There are many things that NixOS's traditional initrd supports that this does not yet. But with this framework in place, we can begin to add those features to the systemd-initrd one by one in separate, parallel PRs.

Notably, this PR does keep the make-initrd-ng concept from the previous one. You can check the README included with it in this PR for details, but the short version is that we don't copy full closures into initrd, and we don't patch things. Instead, we try to automate copying only the exactly correct things to their original paths to the best of our ability, and manually add what the automation misses in the boot.initrd.systemd.objects option. The benefits are outlined in the README.

This can be tested with nix-build ./nixos -A vm.

What this approach will not do

is include a compatibility layer for options like postDeviceCommands and the like. Although this is possible in principle with special units that run the concatenated script with Before= and After= clauses on the appropriate targets, I think this is a bad idea for several reason.

It is antithetical to the purpose of using systemd. We want stage 1 to be parallelized, and declarative. These scripts are the opposite of both. It would slow down boot times and make the code for systemd-initrd less clean.
These scripts sometimes expect to be run in the user interactive console. Trying to accommodate this would be troublesome.
Maddness this way lies. I can only imagine the plethora of complexity and edge cases involved in trying to keep extraUtils and the *Commands options alive in this initrd.

Next steps

Here are a number of things to do in future PRs, in very loose, decreasing order of how I view their importance.

Testing. Currently this will only boot extremely simple setups. It'd be good to know how simple that is, and it will be important to test the other things we work on as we go to make sure they work as well.
Support for the script style options in boot.initrd.systemd.services. Currently they do not copy the necessary scripts in.
Proper udev rules configuration.
Basic LUKS support.
ZFS / btrfs / LVM / mdadm support.
Plymouth support
Networking / SSH support
Advanced LUKS features (e.g. yubikeys).
Move activation into stage 1 so that we can immediately switch-root to systemd. This is nice because systemd can serialize and deserialize some useful information if it knows it's switch-root'ing to systemd.

Most of these should be doable independently of each other, and likely need little more API than the core functionality provided by this PR.

Things done

SuperSandro2000 · 2022-03-20T17:53:17Z

nixos/lib/utils.nix

    lib = import ./systemd-lib.nix { inherit lib config pkgs; };
    unitOptions = import ./systemd-unit-options.nix { inherit lib systemdUtils; };
+    types = import ./systemd-types.nix { inherit lib systemdUtils; };


Suggested change

lib = import ./systemd-lib.nix { inherit lib config pkgs; };

unitOptions = import ./systemd-unit-options.nix { inherit lib systemdUtils; };

types = import ./systemd-types.nix { inherit lib systemdUtils; };

lib = import ./systemd/lib.nix { inherit lib config pkgs; };

unitOptions = import ./systemd/unit-options.nix { inherit lib systemdUtils; };

types = import ./systemd/types.nix { inherit lib systemdUtils; };

Maybe? I am not sure. Just an idea.

nixos/tests/systemd-initrd.nix

pkgs/build-support/kernel/make-initrd-ng.nix

ckiee

I started this review yesterday, there's a chance you already touched some of the things I reviewed.

Mostly code quality nitpicks, but more test{s,ing} for [this] core system component would be nice to see.

pkgs/build-support/kernel/make-initrd-ng.nix

pkgs/build-support/kernel/make-initrd-ng/src/main.rs

ckiee · 2022-03-21T02:32:58Z

nixos/modules/system/boot/systemd/initrd.nix

+           mapAttrs' (n: v: nameValuePair "${n}.path"    (pathToUnit          n v)) cfg.paths
+        // mapAttrs' (n: v: nameValuePair "${n}.service" (initrdServiceToUnit n v)) cfg.services
+        // mapAttrs' (n: v: nameValuePair "${n}.slice"   (sliceToUnit         n v)) cfg.slices
+        // mapAttrs' (n: v: nameValuePair "${n}.socket"  (socketToUnit        n v)) cfg.sockets
+        // mapAttrs' (n: v: nameValuePair "${n}.target"  (targetToUnit        n v)) cfg.targets
+        // mapAttrs' (n: v: nameValuePair "${n}.timer"   (timerToUnit         n v)) cfg.timers
+        // listToAttrs (map
+                     (v: let n = escapeSystemdPath v.where;
+                         in nameValuePair "${n}.mount" (mountToUnit n v)) cfg.mounts)
+        // listToAttrs (map
+                     (v: let n = escapeSystemdPath v.where;
+                         in nameValuePair "${n}.automount" (automountToUnit n v)) cfg.automounts);


This feels pretty repetitive but it does keep the code simpler vs. a higher level approach. Not sure what to make of that.

Yea it's not the first time this is repeated in nixos. I suppose I could factor it out into a function. Would that be good?

Yeah, that'd be nice! The generic-unit-ification being in what is effectively the consumer of the systemd lib seems weird.

nixos/tests/systemd-initrd.nix

nixos/lib/systemd-lib.nix

ElvishJerricco · 2022-03-21T14:27:45Z

#164016 has been merged, so I've rebased onto its last commit and removed the WIP label from this PR

dasJ

I haven't really looked at the Rust code apart from my one suggestion which helped me in debugging yesterday.

The approach is very promising and apart from the first commit not really splittable anymore. Most things I comment here are so that we nail some details right at the beginning instead of people starting to work on different aspects of the initrd while the core of it is still in flux.

I'm currently discussing stuff with @tfc to make sure we can get better test coverage. While I don't think this test needs more, I'd love future PRs with more features having the ability to add more stage 1 tests.

About the switch to the real system - I don't know if this PR should already contain code to switch to the new systemd directly without the shell script in between. While this approach in this PR already works of course, it's another thing people may make assumptions about ("this worked in systemd stage-1 before!!"). As you know, I'm experimenting with this detail and will provide my findings when they come up.

nixos/modules/system/boot/stage-1.nix

nixos/modules/system/boot/systemd/initrd.nix

nixos/tests/systemd-initrd.nix

pkgs/build-support/kernel/make-initrd-ng-tool.nix

pkgs/build-support/kernel/make-initrd-ng/src/main.rs

nixos/lib/systemd-types.nix

We can just use serviceToUnit here.

NixOS#164943 (comment)

nixos/modules/system/boot/systemd/initrd.nix

nixos/lib/systemd-types.nix

nixos/modules/system/boot/systemd/initrd.nix

roberth

Let's use module composition instead of generated modules.
It keeps the types simple.

nixos/lib/systemd-lib.nix

nixos/modules/system/boot/systemd/initrd.nix

dasJ · 2022-04-01T08:45:11Z

@roberth can you take another look? I added two commits, one that does the type composition as requested and another one that adds type composition to the unit types to get rid of options that are not used

@roberth

As requested by @roberth, we now have an option similar to environment.etc. There's also extra store paths to copy and a way to suppress store paths to make customizations possible. We also link mount and umount to /bin to make recovery easier when something fails

This is more in line with what dracut does (it appends "Initramfs") and makes it clear where the boot is currently at when it hangs.

ElvishJerricco · 2022-04-01T17:30:46Z

@dasJ Those changes look great!

dasJ · 2022-04-02T01:59:17Z

If there are no more objections and pushes, I'll merge this in the next 24-48h

bobvanderlinden · 2022-04-03T19:44:11Z

Awesome! This unlocks so much potential improvements. Thanks a lot for all the work of everyone! 🥳🥳🥳

ncfavier · 2022-06-20T16:33:00Z

Tried this out on my laptop, it's almost usable except for two issues:

Disable systemd as initrd to get impermanence back. Lose TPM unlock in the process. systemd as initrd explicitly does not support `pre*Commands` or `post*Commands` as stated in the PR introducing it: NixOS/nixpkgs#164943 This is evident when trying to make apply changes using those declarations, and resulting in the same generation/outputs. That's because nothing is actually changing. The scripts are never run.

ElvishJerricco requested a review from dasJ as a code owner March 20, 2022 08:04

github-actions bot added 6.topic: kernel 6.topic: nixos 6.topic: systemd 8.has: module (update) labels Mar 20, 2022

ElvishJerricco mentioned this pull request Mar 20, 2022

[WIP]: initrd: Use systemd #120015

Closed

20 tasks

ofborg bot added 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 labels Mar 20, 2022

ElvishJerricco force-pushed the systemd-initrd-reuse-systemd-module branch 2 times, most recently from 57296ec to 5253f1c Compare March 20, 2022 09:03

SuperSandro2000 reviewed Mar 20, 2022

View reviewed changes

ElvishJerricco force-pushed the systemd-initrd-reuse-systemd-module branch from d6f3bd9 to 4b4e589 Compare March 20, 2022 20:05

mweinelt reviewed Mar 20, 2022

View reviewed changes

nixos/tests/systemd-initrd.nix Outdated Show resolved Hide resolved

mweinelt reviewed Mar 20, 2022

View reviewed changes

pkgs/build-support/kernel/make-initrd-ng.nix Show resolved Hide resolved

ElvishJerricco force-pushed the systemd-initrd-reuse-systemd-module branch from f97afcd to 954b88f Compare March 21, 2022 00:50

ckiee requested changes Mar 21, 2022

View reviewed changes

ElvishJerricco force-pushed the systemd-initrd-reuse-systemd-module branch from 55a6600 to 9d86e36 Compare March 21, 2022 03:25

nixos: systemd: Split unit types into separate module

52c98fc

ElvishJerricco force-pushed the systemd-initrd-reuse-systemd-module branch from 5d8aeef to 89ccc7d Compare March 21, 2022 14:25

ElvishJerricco changed the title ~~[WIP]: initrd: Opt-in bare bones systemd-based initrd~~ initrd: Opt-in bare bones systemd-based initrd Mar 21, 2022

dasJ reviewed Mar 21, 2022

View reviewed changes

dasJ added this to To Do in systemd via automation Mar 22, 2022

dasJ moved this from To Do to In Progress in systemd Mar 22, 2022

ElvishJerricco added 2 commits March 22, 2022 07:02

make-initrd-ng: init

d193ef8

nixos: systemd-lib: Make generateUnits general with default args

2511374

ElvishJerricco force-pushed the systemd-initrd-reuse-systemd-module branch from 89ccc7d to c730ab0 Compare March 22, 2022 11:02

initrd: Optional systemd-based initrd

2d4ebf1

flokli added 2 commits March 24, 2022 18:57

systemd-initrd, systemd-lib: drop initrdServiceToUnit

e3083de

We can just use serviceToUnit here.

nixos/lib/systemd-lib.nix: move comment back down to packages

fc91cdb

NixOS#164943 (comment)

dasJ reviewed Mar 24, 2022

View reviewed changes

nixos/modules/system/boot/systemd/initrd.nix Outdated Show resolved Hide resolved

dasJ requested changes Mar 28, 2022

View reviewed changes

nixos/lib/systemd-types.nix Outdated Show resolved Hide resolved

nixos/modules/system/boot/systemd/initrd.nix Outdated Show resolved Hide resolved

systemd in Stage 1 automation moved this from In progress to Ready for review Mar 28, 2022

roberth reviewed Mar 29, 2022

View reviewed changes

nixos/lib/systemd-lib.nix Outdated Show resolved Hide resolved

nixos/lib/systemd-lib.nix Outdated Show resolved Hide resolved

nixos/lib/systemd-lib.nix Outdated Show resolved Hide resolved

roberth reviewed Mar 31, 2022

View reviewed changes

nixos/modules/system/boot/systemd/initrd.nix Outdated Show resolved Hide resolved

roberth reviewed Mar 31, 2022

View reviewed changes

nixos/modules/system/boot/systemd/initrd.nix Outdated Show resolved Hide resolved

nixos/systemd-lib: Use module composition

1e5261f

dasJ force-pushed the systemd-initrd-reuse-systemd-module branch from 1ea78c4 to 1e5261f Compare April 1, 2022 07:57

dasJ force-pushed the systemd-initrd-reuse-systemd-module branch from 22ac161 to 0d69a99 Compare April 1, 2022 09:51

dasJ added 2 commits April 1, 2022 11:58

nixos/systemd-initrd: Remove unit options that don't work

b7c62b8

nixos/systemd-initrd: Make emergency access more flexible

c465c8d

dasJ force-pushed the systemd-initrd-reuse-systemd-module branch from 0d69a99 to c465c8d Compare April 1, 2022 09:58

dasJ added 2 commits April 1, 2022 13:26

nixos/systemd-stage-1: Append (Initrd) to /etc/initrd-release

7ebb4eb

This is more in line with what dracut does (it appends "Initramfs") and makes it clear where the boot is currently at when it hangs.

dasJ merged commit 7cdc4dd into NixOS:master Apr 3, 2022

systemd automation moved this from In Progress to Done Apr 3, 2022

systemd in Stage 1 automation moved this from Ready for review to Done Apr 3, 2022

arianvp mentioned this pull request Apr 3, 2022

PSA: systemd in initramfs #72401

Closed

yorickvP mentioned this pull request Apr 16, 2022

Request ZFS/LUKS passwords via Plymouth #88789

Draft

13 tasks

alyssais mentioned this pull request May 28, 2022

Expressions using builtins.fromTOML + builtins.readFile can't be evaluated on the installer image. #175125

Closed

infinisil mentioned this pull request Jan 31, 2023

nixos: add service to propagate status of activationScripts #208942

Open

13 tasks

infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Apr 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initrd: Opt-in bare bones systemd-based initrd #164943

initrd: Opt-in bare bones systemd-based initrd #164943

ElvishJerricco commented Mar 20, 2022 •

edited

SuperSandro2000 Mar 20, 2022

ckiee left a comment

ckiee Mar 21, 2022

ElvishJerricco Mar 21, 2022

ckiee Mar 21, 2022

ElvishJerricco commented Mar 21, 2022

dasJ left a comment

roberth left a comment

dasJ commented Apr 1, 2022

ElvishJerricco commented Apr 1, 2022

dasJ commented Apr 2, 2022 •

edited

bobvanderlinden commented Apr 3, 2022

ncfavier commented Jun 20, 2022

initrd: Opt-in bare bones systemd-based initrd #164943

initrd: Opt-in bare bones systemd-based initrd #164943

Conversation

ElvishJerricco commented Mar 20, 2022 • edited

Motivation

How this compares to my previous attempts

What this approach will not do

Next steps

Things done

SuperSandro2000 Mar 20, 2022

Choose a reason for hiding this comment

ckiee left a comment

Choose a reason for hiding this comment

ckiee Mar 21, 2022

Choose a reason for hiding this comment

ElvishJerricco Mar 21, 2022

Choose a reason for hiding this comment

ckiee Mar 21, 2022

Choose a reason for hiding this comment

ElvishJerricco commented Mar 21, 2022

dasJ left a comment

Choose a reason for hiding this comment

roberth left a comment

Choose a reason for hiding this comment

dasJ commented Apr 1, 2022

ElvishJerricco commented Apr 1, 2022

dasJ commented Apr 2, 2022 • edited

bobvanderlinden commented Apr 3, 2022

ncfavier commented Jun 20, 2022

ElvishJerricco commented Mar 20, 2022 •

edited

dasJ commented Apr 2, 2022 •

edited