New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd emergency mode does not work #1433

Closed
jgunthorpe opened this Issue Jun 30, 2016 · 4 comments

Comments

Projects
None yet
3 participants
@jgunthorpe

jgunthorpe commented Jun 30, 2016

Seen on CoreOS stable (1010.5.0)

I did something in my cloud-init (using PXE) that systemd didn't like, and it dumped the boot into the systemd emergency mode. No error was logged to the console.

Turns out emergency mode doesn't work - pressing enter to get the login results in the screen flashing and some message that looks like it might be talking about policy kit. No shell is presented. So I have no idea what is wrong with systemd.

I am booting these machines with coreos.autologin on the kernel command line, so my expectation is that emergency mode should drop immediately into a shell with out a login prompt for debugging. A boot generator should probably reconfigure emergency.service to go directly to a shell in this case.

Inconsistently the initrd emergency mode always just dumps into a shell.

Presumably emergency mode is totally useless normally, as is normal on systemd, since it doesn't have any support for a no-password root account. Maybe that should be revisited as well..

(FWIW, I think systemd has this wrong, emergency mode should always be passwordless unless a boot option is given to force a password. The boot option would only be supplied on configurations that have locked down BIOS that prevent defeating the emergency mode password check via grub/usb/etc.)

@crawford

This comment has been minimized.

Member

crawford commented Jul 1, 2016

I did something in my cloud-init (using PXE) that systemd didn't like, and it dumped the boot into the systemd emergency mode.

I'm not sure how it's possible that cloud-init trigger systemd emergency mode. That usually happens if Ignition fails to execute (because it fails in the initramfs). Can you share you cloud-config?

I am booting these machines with coreos.autologin on the kernel command line, so my expectation is that emergency mode should drop immediately into a shell with out a login prompt for debugging.

That's not quite how that works. The emergency shell does not have a password. That kernel parameter is controlling the post-pivot system.

Can you provide some logs from your machine? The behavior you are describing doesn't match what I would expect to happen.

@jgunthorpe

This comment has been minimized.

jgunthorpe commented Jul 1, 2016

I'm not sure how it's possible that cloud-init trigger systemd emergency mode. That usually happens if Ignition fails to execute (because it fails in the initramfs). Can you share you cloud-config?

I believe I created a situation in the cloud-init where systemd could not reach its default target, by manipulating unit files related to mounting. I strongly suspect I made local-fs.target enter a failed state - I can't tell the details because there is no way to debug it. I don't know if the cloud-config will be meaningful for you since it relies on local details..

That's not quite how that works. The emergency shell does not have a password. That kernel parameter is controlling the post-pivot system.

You are thinking of the initrd emergency shell, which does not have a password and runs before pivot root. I've also seen it in action when debugging a PXE boot issue that caused no initrd. (BTW, it is kind of dysfunctional on bare metal as well because USB support is not available before it starts. You need to luck into an old system with a PS/2 keyboard, sigh)

I am talking about the post-pivot-root emergency shell which is controlled by emergency.service - this comes included with systemd and runs if systemd determines it cannot reach its default target.

Prior to pivoting the initrd should configure the emergency.service in the post-pivot environment to have a functional passwordless shell if the coreos.autologin kernel parameter is present (or maybe it should always be that way?).

I expect if you boot CoreOS with systemd.unit=emergency.target on the kernel command line you will be able to see it in action.

Can you provide some logs from your machine? The behavior you are describing doesn't match what I would expect to happen.

Nope, I can think of no way to get access to anything on the system once it hangs up in the emergency shell, there is no ssh access, and it is PXE so there is no disk.

@crawford

This comment has been minimized.

Member

crawford commented Jul 6, 2016

(BTW, it is kind of dysfunctional on bare metal as well because USB support is not available before it starts. You need to luck into an old system with a PS/2 keyboard, sigh)

This has been fixed in 1097.0.0.

Prior to pivoting the initrd should configure the emergency.service in the post-pivot environment to have a functional passwordless shell if the coreos.autologin kernel parameter is present (or maybe it should always be that way?).

Ah! I see now. Yes you are right, we should fix this.

@mischief

This comment has been minimized.

mischief commented Jul 20, 2016

fixed by coreos/init#213 and coreos/coreos-overlay#2085. this will be in the next alpha.

@mischief mischief closed this Jul 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment