add support for soft-reboot #309911

arianvp · 2024-05-07T18:54:38Z

Description of changes

Usage:

sudo nixos-rebuild boot && sudo systemctl soft-reboot

switch-to-configuration boot will now prepare a symlink /run/next-system
and just before soft-reboot we will automatically call
/run/next-system/activate to activate the next system to be booted when
systemctl soft-reboot is called.

not only can soft-reboot can be used as an alternative to kexec and reboot, but
it can also be used as an alternative to nixos-rebuild switch in many cases.

Unlike kexec and reboot, soft-reboot can keep certain resources around across the reboot.
For example, socket units can be configured to stay around such that reboots can happen
without stopping accepting any connections. Furthermore file descriptors can be passed across
boots using FDSTORE to even keep existing connections intact.

Please see the man-page for more info: https://www.freedesktop.org/software/systemd/man/latest/systemd-soft-reboot.service.html#

from Systemd NEWS:

A "soft reboot" is similar to a regular reboot, except that it affects
userspace only: the service manager shuts down any running services and other
units, then optionally switches into a new root file system (mounted to
/run/nextroot/), and then passes control to a systemd instance in the new file
system which then starts the system up again. The kernel is not rebooted and
neither is the hardware, firmware or boot loader. This provides a fast,
lightweight mechanism to quickly reset or update userspace, without the latency
that a full system reset involves. Moreover, open file descriptors may be
passed across the soft reboot into the new system where they will be passed
back to the originating services. This allows pinning resources across the
reboot, thus minimizing grey-out time further

Things done

Add a 👍 reaction to pull requests you find important.

nixos/modules/system/boot/systemd.nix

arianvp · 2024-05-07T21:13:41Z

This needs a VM test. Also this probably can not be merged anymore until the 24.05 cutoff has happened.

nixos/modules/system/activation/switch-to-configuration.pl

> A "soft reboot" is similar to a regular reboot, except that it affects > userspace only: the service manager shuts down any running services and other > units, then optionally switches into a new root file system (mounted to > /run/nextroot/), and then passes control to a systemd instance in the new file > system which then starts the system up again. The kernel is not rebooted and > neither is the hardware, firmware or boot loader. This provides a fast, > lightweight mechanism to quickly reset or update userspace, without the latency > that a full system reset involves. Moreover, open file descriptors may be > passed across the soft reboot into the new system where they will be passed > back to the originating services. This allows pinning resources across the > reboot, thus minimizing grey-out time further. not only can soft-reboot can be used as an alternative to kexec and reboot, but it can also be used as an alternative to `nixos-rebuild switch` in many cases. Unlike kexec and reboot, soft-reboot can keep certain resources around across the reboot. For example, socket units can be configured to stay around such that reboots can happen without dropping any connections. Furthermore file descriptors can be passed across boots using FDSTORE to even keep existing connections intact. Please see the man-page for more info: https://www.freedesktop.org/software/systemd/man/latest/systemd-soft-reboot.service.html#

switch-to-configuration boot will now create a symlink /run/next-system systemctl soft-reboot will call /run/next-system/activate just before soft-rebooting, making sure that the system we're rebooting into is activated. Thus you can do: nixos-rebuild boot && systemctl soft-reboot now

We want to get rid of specialFileSystems / earlyMountScript eventually and there is no need to run this before systemd anymore now that the wrappers themselves are set up in a systemd unit since NixOS#263203 Also this is needed to make soft-reboot work. We want to make sure that we remount /run/wrappers with the nosuid bit removed on soft-reboot but because @earlyMountScript@ happens in initrd, this wouldn't happen

arianvp · 2024-05-07T21:52:02Z

Neat follow-up; now that soft-reboot can be used as an alternative to switch-to-configuration switch

Make a version of switch-to-configuration.pl that is:

Not perl
Only sets /nix/var/nix/profiles/system and /run/next-system and nothing else
installs bootloader entries
throws away all the systemd service start/stopping code as it's not needed anymore

arianvp · 2024-05-07T22:55:54Z

nixos/modules/system/activation/switch-to-configuration.pl

@@ -110,6 +110,8 @@
 }

 if ($action eq "boot") {
+    # /run/next-system/activate is called when systemctl soft-reboot is called
+    system("@coreutils@/bin/ln", "-sfn", "$toplevel", "/run/next-system") == 0 or exit 1;


Should we do this on "switch" too?

Doesn't seem necessary. The point of next-system is to have a link to activate on soft-reboot, and a switch will have already activated the new system. Even on a soft-reboot, reactivation shouldn't be necessary.

nixos/modules/system/boot/kexec.nix

arianvp · 2024-05-07T23:31:22Z

nixos/modules/system/boot/systemd.nix

+        DefaultDependencies = false;
+        ConditionPathExists = "/run/next-system/activate";
+      };
+      serviceConfig.ExecStart = "/run/next-system/activate";


Should this fall back to /run/current-system/activate if /run/next-system/activate doesn't exist?

Same argument as not creating next-system on switch; the current system is already activated so it shouldn't be necessary.

This fixes the long standing issue NixOS#50300

NickCao · 2024-05-08T03:13:13Z

nixos/modules/system/boot/systemd.nix

+    # make sure /run/next-system isn't garbage collected before the next boot
+    systemd.tmpfiles.rules = [ "L+ /nix/var/nix/gcroots/next-system - - - - /run/next-system" ];
+
+    # if /run/next-system exists, activate it before  rebooting


Why is this necessary?

Activation happens in the initrd (at least if systemd-initrd is enabled) which soft-reboot doesn't transition into. Same reason why we do activation in the initrd before transitioning into stage-2.

If we don't do this the new system will never get activated

NickCao · 2024-05-08T03:36:43Z

It seems the minimal required part of this PR is the two additional upstream units and the wrapper mount point. It's enough for the nixos-rebuild boot then systemctl soft-reboot workflow to work as nixos-rebuild boot already links the system profile to /nix/var/nix/profiles/system. There are no need for the /run/next-system shenanigans.

arianvp · 2024-05-08T08:18:10Z

It seems the minimal required part of this PR is the two additional upstream units and the wrapper mount point. It's enough for the nixos-rebuild boot then systemctl soft-reboot workflow to work as nixos-rebuild boot already links the system profile to /nix/var/nix/profiles/system. There are no need for the /run/next-system shenanigans.

Using /nix/var/nix/profiles/system is incorrect. See long-standing issue #50300 (which the last commit from this PR fixes; but can maybe be split up into a new PR)

/nix/var/nix/profiles/system is not guaranteed to be updated when nixos-rebuild boot is run due to the existence of alternative system profiles and specialisations

arianvp · 2024-05-08T08:39:55Z

Moving this back to draft. This needs to cook a bit more.

But just keeping this PR around for notes and feedback

I just realized a pretty serious defect:

soft-reboot re-execs into SYSTEMD_BINARY_PATH (Or into /run/nextroot/$SYSTEMD_BINARY_PATH if it exists) (https://github.com/systemd/systemd/blob/18303adcd3a9b16339008b5fa909a7c09247072d/src/core/main.c#L1942)

but we set SYSTEMD_BINARY_PATH=/run/current-system/systemd/lib/systemd https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/systemd/default.nix#L799

which means soft-reboot doesn't execute the new systemd version but into the old one.

Maybe we do need to do the /run/nextroot population instead

arianvp · 2024-05-08T09:04:53Z

Wait no.. We call /run/next-sytem/activate just before the soft-reboot which creates the /run/current-system symlink! This actually already works. 🥳

However I want to have a NixOS Test that asserts this

ElvishJerricco · 2024-05-08T18:18:00Z

nixos/modules/system/boot/kexec.nix

+                echo "Loading NixOS system via kexec."
+                exec kexec --load $p/kernel --initrd=$p/initrd --append="$(cat $p/kernel-params) init=$p/init"
+              fi
+            done


I don't think we should mess with kexec at all in this PR. It's actually got some problems as is and this is only going to make it more confusing.

Yeh I'll drop the commit (and maybe open a separate or for it)

ElvishJerricco · 2024-05-08T18:19:46Z

nixos/modules/system/boot/systemd.nix

+    systemd.tmpfiles.rules = [ "L+ /nix/var/nix/gcroots/next-system - - - - /run/next-system" ];
+
+    # if /run/next-system exists, activate it before  rebooting
+    systemd.services.systemd-soft-reboot-activate-next-system = {


What happened to doing this in ExecStartPre in systemd-soft-reboot.service with a - in the front of the command? That seemed quite elegant to me. With a new unit, I'm less sure that the unit ordering is right.

I copied the ordering from prepare-kexec. I'm pretty confident this is right.

Using ExecStartPre= It would complain about it not being able to connect to the journald.

May 07 21:11:02 utm (activate)[6377]: systemd-soft-reboot.service: Failed to connect stdout to the journal socket, ignoring: Connection refused

soft-reboot.target causes systemd-journald.service to exit before it is reached (has a Conflicts on it) to make sure logs are properly flushed before the switch is made. I think that's what causing this.

Well, the ordering from prepare-kexec isn't exactly relevant, because it isn't really ordered against anything at all except kexec itself. I would think we would want activation to happen as late as possible, i.e. after as much as possible has shut down. Maybe this doesn't matter; it would just mimic an actual reboot better if activation happened when everything was shut down.

ElvishJerricco · 2024-05-08T18:20:51Z

nixos/modules/system/activation/switch-to-configuration.pl

@@ -110,6 +110,8 @@
 }

 if ($action eq "boot") {
+    # /run/next-system/activate is called when systemctl soft-reboot is called
+    system("@coreutils@/bin/ln", "-sfn", "$toplevel", "/run/next-system") == 0 or exit 1;


Doesn't seem necessary. The point of next-system is to have a link to activate on soft-reboot, and a switch will have already activated the new system. Even on a soft-reboot, reactivation shouldn't be necessary.

ElvishJerricco · 2024-05-08T18:21:27Z

nixos/modules/system/boot/systemd.nix

+        DefaultDependencies = false;
+        ConditionPathExists = "/run/next-system/activate";
+      };
+      serviceConfig.ExecStart = "/run/next-system/activate";


Same argument as not creating next-system on switch; the current system is already activated so it shouldn't be necessary.

github-actions bot added 6.topic: nixos 8.has: module (update) labels May 7, 2024

arianvp force-pushed the soft-reboot branch 3 times, most recently from 98fd965 to 2dabb7a Compare May 7, 2024 19:09

arianvp commented May 7, 2024

View reviewed changes

nixos/modules/system/boot/systemd.nix Outdated Show resolved Hide resolved

arianvp force-pushed the soft-reboot branch 4 times, most recently from b8c0bb7 to 57e4d6c Compare May 7, 2024 19:25

This comment was marked as resolved.

Sign in to view

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels May 7, 2024

arianvp force-pushed the soft-reboot branch 2 times, most recently from fc6c978 to b619ed4 Compare May 7, 2024 21:01

arianvp requested a review from ElvishJerricco May 7, 2024 21:09

arianvp marked this pull request as ready for review May 7, 2024 21:09

arianvp requested review from a team and dasJ as code owners May 7, 2024 21:09

arianvp force-pushed the soft-reboot branch from b619ed4 to 404451f Compare May 7, 2024 21:10

arianvp commented May 7, 2024

View reviewed changes

nixos/modules/system/activation/switch-to-configuration.pl Show resolved Hide resolved

arianvp force-pushed the soft-reboot branch from 404451f to f00694d Compare May 7, 2024 21:22

arianvp added 3 commits May 7, 2024 23:23

arianvp force-pushed the soft-reboot branch from f00694d to 3db0ec3 Compare May 7, 2024 21:24

arianvp changed the title ~~nixos/systemd: add support for soft-reboot~~ add support for soft-reboot May 7, 2024

arianvp commented May 7, 2024

View reviewed changes

arianvp mentioned this pull request May 7, 2024

Systemctl kexec always boots /nix/var/nix/profiles/system even if custom profile is selected in nixos-rebuild #50300

Open

arianvp commented May 7, 2024

View reviewed changes

nixos/modules/system/boot/kexec.nix Outdated Show resolved Hide resolved

arianvp commented May 7, 2024

View reviewed changes

nixos/kexec: use /run/next-system

a9d471e

This fixes the long standing issue NixOS#50300

arianvp force-pushed the soft-reboot branch from 2c1ce8a to a9d471e Compare May 7, 2024 23:43

NickCao reviewed May 8, 2024

View reviewed changes

arianvp marked this pull request as draft May 8, 2024 08:34

arianvp removed request for a team and dasJ May 8, 2024 08:52

ElvishJerricco reviewed May 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for soft-reboot #309911

add support for soft-reboot #309911

arianvp commented May 7, 2024 •

edited

This comment was marked as resolved.

This comment was marked as resolved.

arianvp commented May 7, 2024

arianvp commented May 7, 2024 •

edited

arianvp May 7, 2024

ElvishJerricco May 8, 2024

arianvp May 7, 2024

ElvishJerricco May 8, 2024

NickCao May 8, 2024

arianvp May 8, 2024 •

edited

NickCao commented May 8, 2024 •

edited

arianvp commented May 8, 2024 •

edited

arianvp commented May 8, 2024 •

edited

arianvp commented May 8, 2024 •

edited

ElvishJerricco May 8, 2024

arianvp May 8, 2024

ElvishJerricco May 8, 2024

arianvp May 8, 2024

ElvishJerricco May 8, 2024

ElvishJerricco May 8, 2024

ElvishJerricco May 8, 2024

add support for soft-reboot #309911

Are you sure you want to change the base?

add support for soft-reboot #309911

Conversation

arianvp commented May 7, 2024 • edited

Description of changes

Things done

This comment was marked as resolved.

This comment was marked as resolved.

arianvp commented May 7, 2024

arianvp commented May 7, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arianvp May 8, 2024 • edited

Choose a reason for hiding this comment

NickCao commented May 8, 2024 • edited

arianvp commented May 8, 2024 • edited

arianvp commented May 8, 2024 • edited

arianvp commented May 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arianvp commented May 7, 2024 •

edited

arianvp commented May 7, 2024 •

edited

arianvp May 8, 2024 •

edited

NickCao commented May 8, 2024 •

edited

arianvp commented May 8, 2024 •

edited

arianvp commented May 8, 2024 •

edited

arianvp commented May 8, 2024 •

edited