Jammy's cloud-init doesn't work on lxd 4.0

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #2001737

Launchpad details

affected_projects = []
assignee = chad.smith
assignee_name = Chad Smith
date_closed = 2023-02-22T16:52:41.809855+00:00
date_created = 2023-01-05T01:25:15.883101+00:00
date_fix_committed = 2023-01-11T23:29:19.090839+00:00
date_fix_released = 2023-02-22T16:52:41.809855+00:00
id = 2001737
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/2001737
milestone = None
owner = woutervb
owner_name = Wouter van Bommel
private = False
status = fix_released
submitter = woutervb
submitter_name = Wouter van Bommel
tags = []
duplicates = []

Launchpad user Wouter van Bommel(woutervb) wrote on 2023-01-05T01:25:15.883101+00:00

When trying to boot a Jammy container on a host that uses lxd/4.0 the cloud-init there fails to work properly.

The result is, that a ssh key for authentication isn't inserted in the host, denying ssh access.

Upgrading to lxd/5.0 does works around this issue, which provides a newer interface, but isn't a solution as lxd/4.0 is an LTS of only a few years old.

For a complete log see: https://pastebin.canonical.com/p/X9GN5GH8wv/

To reproduce, install the lxd 4.0/stable snap on a machine, configure it to inject a ssh key, and try to launch an jammy container there. One won't be able to ssh into the container.

An extensive thread about this can be found at: https://chat.canonical.com/canonical/pl/c1erobjybt8jzbgxguowxrddrc

[ubuntu-server-builder]

Launchpad user Alberto Contreras(aciba) wrote on 2023-01-10T10:37:53.248896+00:00

Reproduced with:

lxc launch ubuntu-daily:focal fvm --vm
lxc shell fvm

snap list lxd
lxd init --minimal
lxc launch ubuntu-daily:jammy j
lxc exec j -- cloud-init status --wait

$ lxc exec j -- grep -i warning /var/log/cloud-init.log
2023-01-10 10:29:12,936 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceLXD.DataSourceLXD'> failed
2023-01-10 10:29:14,169 - activators.py[WARNING]: Running ['netplan', 'apply'] resulted in stderr output: Failed to connect system bus: No such file or directory
2023-01-10 10:29:27,476 - cc_final_message.py[WARNING]: Used fallback datasource


$ lxc exec j -- grep -i exception /var/log/cloud-init.log
     raise sources.InvalidMetaDataException(
cloudinit.sources.InvalidMetaDataException: Invalid HTTP response [404] from http://lxd/1.0/devices: 404 page not found

[ubuntu-server-builder]

Launchpad user Alberto Contreras(aciba) wrote on 2023-01-10T11:03:06.432123+00:00

The problem is that cloud-init expects /1.0/devices to be available in the lxc instance socket, but this information was introduced in LXD 4.21, see [1].

This issue is present in focal (lxd 4.0/stable) and bionic (lxd 3 deb package).

As a workaround for focal images, one could perform an upgrade to 4.21:

sudo snap refresh --channel=4.21/stable lxd

A solution for cloud-init could be to only fetch /1.0/devices on compatible lxd versions and enable/disable the lxd hotplug feature depending on that.

[1] https://discuss.linuxcontainers.org/t/lxd-4-21-has-been-released/12860

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2023-01-10T14:42:30.438647+00:00

cloudinit.sources.InvalidMetaDataException: Invalid HTTP response [404] from http://lxd/1.0/devices: 404 page not found

Cloud-init can better handle this error in the event that the LXD control plane doesn't fully support the devices endpoint.

Expectation is LXD 4.0 will not get this feature backported, and cloud-init has fallback configuration options when devices don't exist.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2023-01-10T15:07:09.491721+00:00

work in progress pull request which will resolve this problem. Needs unit tests and then cloud-init can support jammy launches from LXD 4.0 hosts.
#1948

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2023-01-12T06:11:35.860369+00:00

This fix will be in the next SRU release of cloud-init version 23.1 (mid-february). It will be in the next interim upload to Lunar (23.04) as version 22.4.2-0ubuntu3

[ubuntu-server-builder]

Launchpad user Alberto Contreras(aciba) wrote on 2023-02-22T16:52:42.479001+00:00

This bug is believed to be fixed in cloud-init in version 23.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.