"Unable to find a system nic" if openstack lists down ports in json config

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1949371

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = 2022-01-08T04:17:17.845783+00:00
date_created = 2021-11-01T14:43:59.359831+00:00
date_fix_committed = None
date_fix_released = None
id = 1949371
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1949371
milestone = None
owner = juliank
owner_name = Julian Andres Klode
private = False
status = expired
submitter = juliank
submitter_name = Julian Andres Klode
tags = []
duplicates = []

Launchpad user Julian Andres Klode(juliank) wrote on 2021-11-01T14:43:59.359831+00:00

We see most ppc64el machines fail to acquire network IPs on scalingstack with errors like

ValueError: Unable to find a system nic for {'mtu': 1458, 'type': 'physical', 'subnets': [{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:6d:95:51'}

In those cases, openstack generated two networks, but only one device is actually present in the VM.

ubuntu@juju-4d1272-prod-proposed-migration-9:$ openstack console log show jak-s390x-impish | grep fa:16
[ 7.113131] cloud-init[439]: ValueError: Unable to find a system nic for {'mtu': 1458, 'type': 'physical', 'subnets': [{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:a4:29:ce'}
[ 7.681157] cloud-init[464]: ci-info: | enp0s1 | False | . | . | . | fa:16:3e:c1:f9:61 |
[ 7.761093] cloud-init[464]: ValueError: Unable to find a system nic for {'mtu': 1458, 'type': 'physical', 'subnets': [{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:a4:29:ce'}
ubuntu@juju-4d1272-prod-proposed-migration-9:$ openstack port list --server jak-s390x-impish
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+
| 7c68497d-348a-4438-99df-4a7bd31734df | | fa:16:3e:a4:29:ce | ip_address='10.44.124.6', subnet_id='694adc9c-c852-4654-937f-b2438a1bad23' | DOWN |
| ec5f644c-6151-4ad9-bd09-7908e8956faa | | fa:16:3e:c1:f9:61 | ip_address='10.44.124.73', subnet_id='694adc9c-c852-4654-937f-b2438a1bad23' | ACTIVE |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+

cloud-init must not abort here, but continue configuring the active network device.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-11-01T15:07:43.387268+00:00

Thanks for filing the bug Julian. Could you also please attach the tarfile from cloud-init collect-logs to aid in triage. I think what openstack is telling cloud init probably needs to take into account optional devices.

[ubuntu-server-builder]

Launchpad user Julian Andres Klode(juliank) wrote on 2021-11-01T15:17:00.825045+00:00

I unfortunately cannot SSH into the machine without having the network setup first, chicken and egg problem. But it's probably similar to the above.

In any case, there should be two devices sent, and it should not hard early error out just because it can't find one of them.

[ubuntu-server-builder]

Launchpad user Julian Andres Klode(juliank) wrote on 2021-11-01T15:45:33.518797+00:00

Presumably it should log the error instead of raising it, and then at the end, before writing the config delete any items without a name.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-11-01T17:09:50.674881+00:00

The problem is that cloud-init is being told there should be two devices associated with this instance, and it isn't being told that one of them is optional for boot. We might need to tease out why that is the case in this deployment

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2022-01-08T04:17:17.480419+00:00

[Expired for cloud-init because there has been no activity for 60 days.]