systemd emergency mode does not work #1433
Seen on CoreOS stable (1010.5.0)
I did something in my cloud-init (using PXE) that systemd didn't like, and it dumped the boot into the systemd emergency mode. No error was logged to the console.
Turns out emergency mode doesn't work - pressing enter to get the login results in the screen flashing and some message that looks like it might be talking about policy kit. No shell is presented. So I have no idea what is wrong with systemd.
I am booting these machines with coreos.autologin on the kernel command line, so my expectation is that emergency mode should drop immediately into a shell with out a login prompt for debugging. A boot generator should probably reconfigure emergency.service to go directly to a shell in this case.
Inconsistently the initrd emergency mode always just dumps into a shell.
Presumably emergency mode is totally useless normally, as is normal on systemd, since it doesn't have any support for a no-password root account. Maybe that should be revisited as well..
(FWIW, I think systemd has this wrong, emergency mode should always be passwordless unless a boot option is given to force a password. The boot option would only be supplied on configurations that have locked down BIOS that prevent defeating the emergency mode password check via grub/usb/etc.)
The text was updated successfully, but these errors were encountered:
I'm not sure how it's possible that cloud-init trigger systemd emergency mode. That usually happens if Ignition fails to execute (because it fails in the initramfs). Can you share you cloud-config?
That's not quite how that works. The emergency shell does not have a password. That kernel parameter is controlling the post-pivot system.
Can you provide some logs from your machine? The behavior you are describing doesn't match what I would expect to happen.
I believe I created a situation in the cloud-init where systemd could not reach its default target, by manipulating unit files related to mounting. I strongly suspect I made local-fs.target enter a failed state - I can't tell the details because there is no way to debug it. I don't know if the cloud-config will be meaningful for you since it relies on local details..
You are thinking of the initrd emergency shell, which does not have a password and runs before pivot root. I've also seen it in action when debugging a PXE boot issue that caused no initrd. (BTW, it is kind of dysfunctional on bare metal as well because USB support is not available before it starts. You need to luck into an old system with a PS/2 keyboard, sigh)
I am talking about the post-pivot-root emergency shell which is controlled by emergency.service - this comes included with systemd and runs if systemd determines it cannot reach its default target.
Prior to pivoting the initrd should configure the emergency.service in the post-pivot environment to have a functional passwordless shell if the coreos.autologin kernel parameter is present (or maybe it should always be that way?).
I expect if you boot CoreOS with systemd.unit=emergency.target on the kernel command line you will be able to see it in action.
Nope, I can think of no way to get access to anything on the system once it hangs up in the emergency shell, there is no ssh access, and it is PXE so there is no disk.
This has been fixed in 1097.0.0.
Ah! I see now. Yes you are right, we should fix this.