Cloud-Init doesn't print ssh host key keys

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1910261

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = 2021-03-08T04:17:18.736890+00:00
date_created = 2021-01-05T15:32:32.393839+00:00
date_fix_committed = None
date_fix_released = None
id = 1910261
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1910261
milestone = None
owner = karstensiemer
owner_name = Karsten Siemer
private = False
status = expired
submitter = karstensiemer
submitter_name = Karsten Siemer
tags = ['keys', 'ssh']
duplicates = []

Launchpad user Karsten Siemer(karstensiemer) wrote on 2021-01-05T15:32:32.393839+00:00

Hello,

I am running this ami "ami-003ab7b6dd427580e" on aws ec2.
It is the rhel 8 quickstart ami and an arm image running 4.18.0-240.8.1.el8_3.aarch64
I run it on a m6g.xlarge machine and it has cloud-init pre-installed.

Sadly, it isn't printing the ssh host key keys on first boot:

[  OK  ] Reached target Login Prompts.
[FAILED] Failed to start OpenSSH server daemon.
See 'systemctl status sshd.service' for details.
[  115.600243] cloud-init[1097]: Cloud-init v. 19.4 running 'modules:config' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.52 seconds.
[  OK  ] Started Apply the settings specified in cloud-config.
         Starting Execute cloud user/final scripts...
[  OK  ] Started System Logging Service.
[  OK  ] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.
ci-info: +++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user ec2-user++++++++++++++++++++++
ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+
ci-info: | Keytype |                Fingerprint (md5)                | Options |                   Comment                   |
ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+
ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    | packer_5ff2df88-947a-5d37-29ff-4ca186ea3f79 |
ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |    -    |                   jenkins                   |
ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+
<14>Jan  4 15:36:36 ec2:
<14>Jan  4 15:36:36 ec2: #############################################################
<14>Jan  4 15:36:36 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Jan  4 15:36:36 ec2: -----END SSH HOST KEY FINGERPRINTS-----
<14>Jan  4 15:36:36 ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----
[  116.110779] cloud-init[1180]: Cloud-init v. 19.4 running 'modules:final' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.90 seconds.
[  116.112793] cloud-init[1180]: Cloud-init v. 19.4 finished at Mon, 04 Jan 2021 15:36:36 +0000. Datasource DataSourceEc2Local.  Up 116.10 seconds
[  116.190100] echo[1376]: trying to reload or restart NetworkManager.service
[  OK  ] Started Execute cloud user/final scripts.
[  OK  ] Reached target Cloud-init target.

ssh also failes to come up at first, but succeeds on second try like this:

sshd.service log:
Jan 04 13:49:15 xxx systemd[1]: Starting OpenSSH server daemon...
Jan 04 13:49:15 xxx sshd[1092]: Unable to load host key: /etc/ssh/ssh_host_rsa_key
Jan 04 13:49:15 xxx sshd[1092]: Unable to load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 04 13:49:15 xxx sshd[1092]: Unable to load host key: /etc/ssh/ssh_host_ed25519_key
Jan 04 13:49:15 xxx sshd[1092]: sshd: no hostkeys available -- exiting.
Jan 04 13:49:15 xxx systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Jan 04 13:49:15 xxx systemd[1]: sshd.service: Failed with result 'exit-code'.
Jan 04 13:49:15 xxx systemd[1]: Failed to start OpenSSH server daemon.
Jan 04 13:49:57 xxx systemd[1]: sshd.service: Service RestartSec=42s expired, scheduling restart.
Jan 04 13:49:57 xxx systemd[1]: sshd.service: Scheduled restart job, restart counter is at 1.
Jan 04 13:49:57 xxx systemd[1]: Stopped OpenSSH server daemon.
Jan 04 13:49:58 xxx systemd[1]: Starting OpenSSH server daemon...
Jan 04 13:49:58 xxx sshd[4720]: Server listening on 0.0.0.0 port 22.
Jan 04 13:49:58 xxx sshd[4720]: Server listening on :: port 22.
Jan 04 13:49:58 xxx systemd[1]: Started OpenSSH server daemon.
Jan 04 13:50:45 xxx sshd[4722]: Accepted publickey for ec2-user from xxx port 55550 ssh2: RSA xxx
Jan 04 13:50:45 xxx sshd[4722]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)

The /etc/ssh directory also is populated:

-rw-r--r--.  1 root root     577388 Mar 27  2020 moduli
-rw-r--r--.  1 root root       1770 Mar 27  2020 ssh_config
drwxr-xr-x.  2 root root         28 Nov 26 08:58 ssh_config.d
-rw-r-----.  1 root ssh_keys    480 Jan  4 17:18 ssh_host_ecdsa_key
-rw-r--r--.  1 root root        162 Jan  4 17:18 ssh_host_ecdsa_key.pub
-rw-r-----.  1 root ssh_keys    387 Jan  4 17:18 ssh_host_ed25519_key
-rw-r--r--.  1 root root         82 Jan  4 17:18 ssh_host_ed25519_key.pub
-rw-r-----.  1 root ssh_keys   2578 Jan  4 17:18 ssh_host_rsa_key
-rw-r--r--.  1 root root        554 Jan  4 17:18 ssh_host_rsa_key.pub
-rw-------.  1 root root       4268 Jan  4 09:28 sshd_config

[ubuntu-server-builder]

Launchpad user Karsten Siemer(karstensiemer) wrote on 2021-01-05T15:32:32.393839+00:00

Launchpad attachments: collect-logs

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-01-06T17:01:14.113105+00:00

Hi Karsten, thanks for filing this bug! It's not really clear to me what's going on here: cloud-init should be writing out SSH host keys during cc_ssh execution, which we can see happens in the log.

My best guess as to what is going on is that there is either (a) some ssh_keys configuration which doesn't specify any valid keys: this would cause us to take the path which doesn't generate the keys, but also wouldn't write out any keys; or, (b) there is an empty ssh_genkeytypes configuration which means that we take the generation path, but have no key types to generate. By looking at the journal, I can see that sshd-keygen service instances are running: this could mean that (a) or (b) are intentionally configured in the base image you're building from (because they expect sshd-keygen to perform this, so don't want cloud-init to do so). (It could also mean that sshd-keygen is actively removing the keys that cloud-init created; I don't know what the service does, exactly, so I can't be sure.)

I can see from the logs that no user-data is specified to the instance: this means that any such configuration would have to be in /etc/cloud somewhere.

Could you pastebin the contents of /etc/cloud/cloud.cfg and any files in /etc/cloud/cloud.cfg.d/ and include a link here? Once you've done so, please move this bug back to New.

Thanks!

Dan

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2021-03-08T04:17:18.549956+00:00

[Expired for cloud-init because there has been no activity for 60 days.]