EC2 CoreOS Cloud Init Running when using Ignition

[troyfontaine]

Issue Report

Bug

Bug

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1298.7.0
VERSION_ID=1298.7.0
BUILD_ID=2017-03-31-0215
PRETTY_NAME="Container Linux by CoreOS 1298.7.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

ec2 AMI: ami-fcc4539c
Instance configured using Ignition-not cloud-config

Expected Behavior

Utilizing update-ssh-keys to remove existing keys for the Core user will remove the key chosen at instance creation and prevent it from persisting across restarts. E.g.

$ update-ssh-keys -D coreos-metadata
Disabling coreos-metadata:
2048 SHA256:KEYINFO MY_OLD_KEY(RSA)
Updated /home/core/.ssh/authorized_keys

Actual Behavior

Once restarting after removing the key, the key is re-added to the Core user by CloudInit.

Reproduction Steps

1. Launch instance with ignition configuration file
2. Use update-ssh-keys to add a new key to the Core user
3. Use update-ssh-keys to remove the coreos-metadata key from the Core user
4. Reboot the instance
5. View /home/core/.ssh/authorized_keys and see the old key has been re-added (the new key is not affected and persists).

Other Information

Checking journalctl from the last reboot, I see the following:

Apr 25 18:30:41 mycoreos coreos-cloudinit[1051]: 2017/04/25 18:30:41 Fetching data from http://169.254.169.254/2009-04-04/meta-data/public-keys/0/openssh-key. Attempt #1
Apr 25 18:30:41 mycoreos coreos-cloudinit[1051]: 2017/04/25 18:30:41 Found SSH key for "MY_OLD_KEY"

This behavior is contrary to the information posted on the Google Group

[crawford]

This is a known issue in older versions of Container Linux. It was fixed in 1367.5.0 when we moved away from running coreos-cloudinit for provisioning. You'll either need to reprovision using at least that version or manually mask coreos-cloudinit with the following Container Linux Config:

systemd:
  units:
    - name: oem-cloudinit.service
      mask: true

The reason this is broken is because we decided long ago that coreos-cloudinit would always apply SSH keys, regardless of how broken the userdata was. This was to facilitate debugging. The side effect of this is that if coreos-cloudinit ever runs, it reapplies your SSH keys. It wasn't until the aforementioned version that we stopped using coreos-cloudinit at all. I hate coreos-cloudinit.

[crawford]

To further clarify this behavior, even after the switch to Ignition, coreos-metadata-sshkeys@.service will still apply those keys on every boot. On EC2, since you cannot update those keys, you'll need to systemctl disable coreos-metadata-sshkeys@.service to stop them from being updated on every boot.

Thinking out loud, I wonder if we should change coreos-metadata-sshkeys@.service to use ConditionFirstBoot=true to prevent this behavior. /cc @euank

[troyfontaine]

Thanks @crawford! We're going to give that a spin.

[troyfontaine]

@crawford that work around worked beautifully. Thank you! That would be excellent to have in the documentation somewhere?

[crawford]

@troyfontaine great! Yeah, that might be worth calling out somewhere. This is a non-issue in later releases (starting with 1367.5.0), but those haven't worked their way through all of the channels yet.

[bgilbert]

I've opened #2070 for changing coreos-metadata-sshkeys@.service to run only on first boot. 1367.5.0 has propagated through all channels, so I'll close.