locksmithd fails at startup: "Cannot get update engine status"

[bgilbert]

Issue Report

Bug

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1548.0.0
VERSION_ID=1548.0.0
BUILD_ID=2017-09-27-0012
PRETTY_NAME="Container Linux by CoreOS 1548.0.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Observed on AWS, GCE, OCI, Packet, QEMU

Expected Behavior

locksmithd starts successfully.

Actual Behavior

locksmithd sometimes fails at startup with Cannot get update engine status: The name com.coreos.update1 was not provided by any .service files. When restarted by systemd, it starts successfully.

Reproduction Steps

1. Boot Container Linux
2. journalctl -b 0 | grep locksmith

Other Information

Oct 04 01:19:40 localhost locksmithd[1729]: No configured reboot window
Oct 04 01:19:40 localhost locksmithd[1729]: Cannot get update engine status: The name com.coreos.update1 was not provided by any .service files
Oct 04 01:19:40 localhost systemd[1]: locksmithd.service: Main process exited, code=exited, status=1/FAILURE
Oct 04 01:19:40 localhost systemd[1]: locksmithd.service: Unit entered failed state.
Oct 04 01:19:40 localhost systemd[1]: locksmithd.service: Failed with result 'exit-code'.
Oct 04 01:19:50 localhost systemd[1]: locksmithd.service: Service hold-off time over, scheduling restart.
Oct 04 01:19:50 localhost locksmithd[1824]: No configured reboot window
Oct 04 01:19:50 localhost locksmithd[1824]: locksmithd starting currentOperation="UPDATE_STATUS_IDLE" strategy="reboot"

I have kola logs showing this behavior as far back as June.

[euank]

Note that due to:

Restart=on-failure
RestartSec=10s

It should eventually come up anyway.

This could be an opportunity to use a D-BUS Service to activate it, but we'd have to figure out if that's properly backwards compatible. At a glance, it looks like it likely isn't :(

[bgilbert]

@euank D-Bus services can't activate on a broadcast signal.

It seems from coreos/locksmith@d01a4c3 that the lack of a dependency on update_engine is intentional. Perhaps locksmith should retry internally rather than exiting and expecting systemd to restart it.

[bgilbert]

It turns out that the update_engine ebuild has always installed the D-Bus service file into the directory for session bus services, so that file was effectively being ignored. Moving the file to the system bus directory and adding a SystemdService key would allow locksmithd's update_engine dependency to be managed at runtime, without interfering with the retry mechanism mentioned in coreos/locksmith@d01a4c3.

[bgilbert]

As discussed OOB, we can't enable bus activation for update_engine because that would change behavior: update_engine_client -check_for_update, or starting locksmithd, would start update_engine if it was not masked. The latter would break the documented mechanism for disabling update_engine for this boot only, systemctl stop update-engine.service. (It should have been systemctl mask --now --runtime update-engine.service.)

coreos/update_engine#145 and coreos/coreos-overlay#2811 drop the useless D-Bus service file from update_engine, and coreos/locksmith#132 fixes the dependency problem by adding an After to locksmithd.service.

[bgilbert]

Fixed in coreos/coreos-overlay#2821, which should be in CL 1576.0.0.