pid1: add a new method of rebooting: userspace only under the name "soft-reboot"

[poettering]

This adds a new mechanism for rebooting, a form of "userspace reboot"
hereby dubbed "soft-reboot". It will stop all services as in a usual
shutdown, possibly transition into a new root fs and then issue a fresh
initial transaction. The kernel is not replaced.

File descriptors can be passed over, thus opening the door for leaving
certain resources around between such reboots.

Usecase: this is an extremely quick way to reset userspace fully when
updating image based systems, without going through a full
hardware/firmware/boot loader/kernel reset. It minimizes "grayout time"
for OS updates. (In particular when combined with kernel live patching)

[poettering]

this is not ready for merging yet. lacks all docs, and tests.

But is pretty comprehensive otherwise. Can do a supercharged reboot in an nspawn container and in qemu in no time.

[bluca]

This is great, and should open the door to easier testing as well here, as we can now do reboots without actually terminating qemu/nspawn and thus add special logic to the test harness.

Bikeshedding: I'd really call this just "userspace reboot", "renew" is really confusing if you don't already know what it does

[poettering]

"systemctl userspace-reboot" is impossible to type

[bluca]

That's why god invented bash completion!

[bluca]

As a shorthand, we could also have a "systemctl uexec" alias - would pair well with kexec

[srd424]

"systemctl userspace-reboot" is impossible to type

How about "reinit"?

[orowith2os]

reinit sounds too much like a reboot to me, from a third party (end user) pov.

I prefer userspace-reboot, it feels like it represents this feature best. It's easy to remember. I can't think of any alternative names, though...

[dtardon]

Maybe system-reexec (analogy to daemon-reexec)? Or just reexec?

Btw, launchd apparently supports this already; it's invoked as launchctl reboot userspace.

[bluca]

the main goal is to quickly transition to a new userspace snapshot, not just to re-execute the current one

[mrc0mmand]

You'll need to add

diff --git a/test/units/testsuite-21.sh b/test/units/testsuite-21.sh
index 36f647ca5f..a0df607377 100755
--- a/test/units/testsuite-21.sh
+++ b/test/units/testsuite-21.sh
@@ -28,6 +28,7 @@ systemctl log-level info
 # FIXME: systemd-run doesn't play well with daemon-reexec
 # See: https://github.com/systemd/systemd/issues/27204
 sed -i '/\[org.freedesktop.systemd1\]/aorg.freedesktop.systemd1.Manager:Reexecute FIXME' /etc/dfuzzer.conf
+sed -i '/\[org.freedesktop.systemd1\]/aorg.freedesktop.systemd1.Manager:Renew destructive' /etc/dfuzzer.conf
 
 # TODO
 #   * check for possibly newly introduced buses?

so dfuzzer doesn't keep triggering "renew" when fuzzing. Once this is merged (and the name of the method is settled) I'll update dfuzzer and drop the sed.

[poettering]

Btw, launchd apparently supports this already; it's invoked as launchctl reboot userspace.

btw, anyone has any idea where to find the current sources for launchd/launchtl? did apple take that closed source? only can find a verson from 7 years ago...

[poettering]

```diff

so dfuzzer doesn't keep triggering "renew" when fuzzing. Once this is merged (and the name of the method is settled) I'll update dfuzzer and drop the sed.

apparently people don't like the name "renew". so we are going to change this before merging anyway, i guess

[Winterhuman]

@poettering Out of curiosity, how would switching to a different root work in this case? In other words, how does a non-destructive switch_root work?

EDIT: It's because it's essentially pivot_root, not switch_root, see #27450.

[topimiettinen]

More bikeshedding suggestions:

• Also analogously to systemctl daemon-reexec: systemctl userspace-reexec or less Yoda: systemctl reexec-userspace?
• analogously to systemctl restart xyz.service: systemctl userspace-restart / systemctl restart-userspace
• systemctl userspace-reboot / systemctl reboot-userspace-only/ systemctl reboot-userspace

Also the existing commands could have new, complementing aliases for symmetry:

• systemctl reboot = systemctl system-reboot / systemctl reboot-system / systemctl system-restart / systemctl restart-system
• systemctl kexec = systemctl kernel-reboot / systemctl reboot-kernel / systemctl kernel-restart / systemctl restart-kernel
• systemctl poweroff = systemctl system-poweroff / systemctl poweroff-system (maybe not, what's the userspace equivalent to poweroff?)

[jevinskie]

Btw, launchd apparently supports this already; it's invoked as launchctl reboot userspace.

btw, anyone has any idea where to find the current sources for launchd/launchtl? did apple take that closed source? only can find a verson from 7 years ago...

Sadly they stopped releasing the source years ago.

[poettering]

Note that the Linux kernel currently pins the original root fs, no matter what

Do you have any resources on the status of changing this upstream?

I know that at least one of the Linux kernel VFS maintainers is aware of the issue. I wouldn't hold my breath though.

Would an image-based system like ostree help here?

ostree is a userspace concept. It cannot avoid this kernel limitation. This needs to be fixed in kernel.

[erkinalp]

This should have been called systemctl --user reboot.

[smac89]

This should have been called systemctl --user reboot.

--user commands usually do not require super user permissions, but I have a feeling this feature will require root permissions

[bluca]

needs a rebase

[poettering]

This should have been called systemctl --user reboot.

Generally, if you specify --user on the systemctl command line you will talk to the per-user instance of the service manager (as supposed to the per-system service manager, i.e. PID 1). Hence systemctl --user reboot as user lennartwould indicate "Hey, service manager of user lennart please reboot yourself!". Which doesn't really make sense, since "reboot" is not a concept defined for users, but only for the system as a whole.

[erkinalp]

It would indeed make sense, by forcibly restarting everything run by the user in question.

[bluca]

I wonder what to do about pcrphase and such - I think we do not want to re-run TPM measurements? To achieve this, and probably to help with other actions that might be needed to run only the "real" boot, what about adding a new ConditionIsSoftReboot= or ConditionBootType= or so, that allows to match and skip? To implement it, it should be doable to pass a variable through via serialization in the manager at shutdown, so that the next iteration knows it's been through a soft reboot

[bluca]

include root in the log message?

[bluca]

maybe plug in portable services and nspawn as solutions that work well with this, and link their docs?

[bluca]

"not executed, the executables in"

[bluca]

would be good to beef up this test, to add a normal service that is shut down as expected (and verify that it is), and one that survives (and verify that it does)

[orowith2os]

It would indeed make sense, by forcibly restarting everything run by the user in question.

You can log out and log back in. That should get you what you want. Userspace reboots should probably be under the normal reboot commands.

[poettering]

I wonder what to do about pcrphase and such - I think we do not want to re-run TPM measurements?

So that's a major discussion to be had, but I think we should do that separately. That said I nowadays think the only safe and secure way is that we really should run at least the boot phase stuff, so that the n-th reboot can be securely distinguished from the n-1-th and n+1-th boot. But we need to start maintaining a proper log of measurements sooner rather than later, so that people can verify this. Soft reboots must appear as a new series of boot phases in the measurement logs I think.

Note that the keys for FDE and such are already unlocked on a soft reboot, and we don't need to unlock them again, hence the boot phase stuff should work out quite OK still: you'd bind FDE unlocking to the initrd boot phase of the first boot, and then the FDE will work in perpetuity even though we can never access the key in the TPM anymore.

A different story is the measurement of machine-id and the mount uuids into PCR 15. We probably should not repeat that, and we don't really have to if it stays the same and mounted.

To achieve this, and probably to help with other actions that might be needed to run only the "real" boot, what about adding a new ConditionIsSoftReboot= or ConditionBootType= or so, that allows to match and skip? To implement it, it should be doable to pass a variable through via serialization in the manager at shutdown, so that the next iteration knows it's been through a soft reboot

We can certainly consider that. For the fs uuid measurement an alternative would be to simply leave the service up till the very end, so that on next boot it is still up because serialization, and then we won't rerun it.

[bugaevc]

btw, anyone has any idea where to find the current sources for launchd/launchtl? did apple take that closed source? only can find a verson from 7 years ago...

As far as we (the Darling project) know, launchd has been integrated into the XPC project, largely rewritten, and its sources are nowhere to be found just like the rest of XPC.

We ship launchd-842.92.1 (apparently from OS X 10.9.4, 2014) and our own reimplementation of XPC.

[jluebbe]

Is there a defined interface for the new userspace to find out how was booted ("normally" or via soft-reboot)? Similarly, where it was booted from, analogous to /proc/cmdline (besides inspecting mountinfo in proc)?

Or would the old userspace prepare that information together with /run/nextroot somewhere readable by the new userspace?

I've not seen it in the code, but perhaps I've missed something.

[brotaxt]

This adds a new mechanism for rebooting, a form of "userspace reboot"
hereby dubbed "soft-reboot". It will stop all services as in a usual
shutdown, possibly transition into a new root fs and then issue a fresh
initial transaction. The kernel is not replaced.

File descriptors can be passed over, thus opening the door for leaving
certain resources around between such reboots.

Usecase: this is an extremely quick way to reset userspace fully when
updating image based systems, without going through a full
hardware/firmware/boot loader/kernel reset. It minimizes "grayout time"
for OS updates. (In particular when combined with kernel live patching)

Could you explain what is meant by image based operating system?

[Winterhuman]

@brotaxt For example, using two read-only SquashFS partitions (the disk "images"), one with the current root, and the other with the updated root to reboot into

[safinaskar]

It seems you meant switch-root, not switch-reboot