/etc/ssh/ssh_host_rsa_key fails to be generated when the sshd service Is restarted after the cloud-Init boot Is completed

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #2004632

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = 2023-02-07T12:05:38.107600+00:00
date_created = 2023-02-03T09:51:24.568811+00:00
date_fix_committed = None
date_fix_released = None
id = 2004632
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/2004632
milestone = None
owner = sxt1001
owner_name = shixuantong
private = False
status = invalid
submitter = sxt1001
submitter_name = shixuantong
tags = []
duplicates = []

Launchpad user shixuantong(sxt1001) wrote on 2023-02-03T09:51:24.568811+00:00

It should be related to the following commit:
b3e31ba

[ubuntu-server-builder]

Launchpad user shixuantong(sxt1001) wrote on 2023-02-03T09:54:46.106831+00:00

[ubuntu-server-builder]

Launchpad user shixuantong(sxt1001) wrote on 2023-02-04T06:26:54.178975+00:00

[ubuntu-server-builder]

Launchpad user shixuantong(sxt1001) wrote on 2023-02-04T07:28:44.351737+00:00

Can I delete this file (/run/systemd/generator.early/multi-user.target.wants/cloud-init.target) after the cloud-init boot is finished?

[ubuntu-server-builder]

Launchpad user shixuantong(sxt1001) wrote on 2023-02-04T08:32:04.138868+00:00

/etc/ssh/ssh_host_rsa_key fails to be generated:

[root@localhost ~]# cloud-init status
status: done
[root@localhost ~]# systemctl status cloud-init
○ cloud-init.service - Initial cloud-init job (metadata service crawler)
Loaded: loaded (/usr/lib/systemd/system/cloud-init.service; enabled; vendor preset: disabled)
Active: inactive (dead)

Feb 04 15:20:21 localhost cloud-init[1909]: ci-info: ... ...
... ...
[root@localhost ~]# rm -rf /etc/ssh/ssh_host_rsa_key
[root@localhost ~]# systemctl restart sshd
[root@localhost ~]# ll /etc/ssh/ssh_host_rsa_key
ls: cannot access '/etc/ssh/ssh_host_rsa_key': No such file or directory
[root@localhost ~]#

[ubuntu-server-builder]

Launchpad user shixuantong(sxt1001) wrote on 2023-02-06T01:10:14.730828+00:00

No matter whether cloud-init is inactive or active, /etc/ssh/ssh_host_rsa_key generation is affected.

[ubuntu-server-builder]

Launchpad user Emanuele Esposito(esposem) wrote on 2023-02-06T12:15:32.637361+00:00

If you use cloud-init, then a drop-in disable-sshd-keygen-if-cloud-init-active.conf will be placed to prevent ssh creating keys and race with cloud-init.
If you then disable cloud-init, but the drop-in is still there, you won't have any process taking care of ssh keys because cloud-init is disabled and sshd is stopped because of the drop-in.

[ubuntu-server-builder]

Launchpad user shixuantong(sxt1001) wrote on 2023-02-07T02:05:00.785010+00:00

but when cloud-init has finished running, sshd is always stopped because of disable-sshd-keygen-if-cloud-init-active.conf. Is that not a problem?

[ubuntu-server-builder]

Launchpad user Alberto Contreras(aciba) wrote on 2023-02-07T12:04:52.735634+00:00

Hello, shixuantong.

cloud-init will create the ssh keys using the cc_ssh module on first boot, and sshd-keygen won't run.

Could you please elaborate on where you think the problem is, and/or what your expectations are?

[ubuntu-server-builder]

Launchpad user shixuantong(sxt1001) wrote on 2023-02-07T12:38:21.819040+00:00

Hello, Alberto Contreras, thank you very much for your reply.

I have such a use scenario here. After the cloud-init boot is complete, ssh_host_rsa_key is lost due to certain reasons (for example, manual deletion). I need to restart the sshd service to generate ssh_host_rsa_key. This scenario cannot be executed successfully because of disable-sshd-keygen-if-cloud-init-active.conf.

[ubuntu-server-builder]

Launchpad user Huijuan Zhao(huijuanzhao) wrote on 2023-02-07T15:02:20.710560+00:00

Hello shixuantong,

Could you please help to check if the below workaround works for you?

Generate ssh host keys manually via the below commands:
/usr/libexec/openssh/sshd-keygen rsa
/usr/libexec/openssh/sshd-keygen ecdsa
/usr/libexec/openssh/sshd-keygen ed25519

Then restart sshd service

Thanks!

[ubuntu-server-builder]

Launchpad user Alberto Contreras(aciba) wrote on 2023-02-08T09:58:45.758250+00:00

Then you could either

1. Manually regenerate the key(s) as Huijuan Zhao suggested or
2. Remove the systemd/disable-sshd-keygen-if-cloud-init-active.conf file and restart the service or the system.

(2) is probably safe, after the first boot has run, as cc_ssh only runs on first boot. Thus, subsequent boots won't race with sshd-keygen.
Additionally, in your use case, you are generating a new set of keys which you are going to manually extract, so it does not matter if the keys are generated by cloud-init or sshd-keygen.