Openstack Datasource not detected in Oracle cloud

[palash-gandhi]

Bug report

It looks like recent changes from this PR #4426 have caused an issue in cloud-init where the datasource is not detected causing cloud-init configuration to be ignored.
Here's what we see in the logs:

2024-03-19 22:51:34,786 - __init__.py[DEBUG]: Looking for data source in: ['OpenStack', 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM']
2024-03-19 22:51:34,802 - __init__.py[DEBUG]: Searching for local data source in: ['DataSourceOpenStackLocal']
2024-03-19 22:51:34,802 - handlers.py[DEBUG]: start: init-local/search-OpenStackLocal: searching for local data from DataSourceOpenStackLocal
2024-03-19 22:51:34,802 - __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceOpenStack.DataSourceOpenStackLocal'>
2024-03-19 22:51:34,802 - __init__.py[DEBUG]: Update datasource metadata and network config due to events: boot-new-instance
2024-03-19 22:51:34,803 - dmi.py[DEBUG]: querying dmi data /sys/class/dmi/id/product_name
2024-03-19 22:51:34,803 - dmi.py[DEBUG]: querying dmi data /sys/class/dmi/id/chassis_asset_tag
2024-03-19 22:51:34,803 - util.py[DEBUG]: Reading from /proc/1/environ (quiet=False)
2024-03-19 22:51:34,803 - util.py[DEBUG]: Read 358 bytes from /proc/1/environ
2024-03-19 22:51:34,803 - __init__.py[DEBUG]: Datasource type DataSourceOpenStackLocal [net,ver=None] is not detected.
2024-03-19 22:51:34,803 - __init__.py[DEBUG]: Datasource DataSourceOpenStackLocal [net,ver=None] not updated for events: boot-new-instance
2024-03-19 22:51:34,803 - handlers.py[DEBUG]: finish: init-local/search-OpenStackLocal: SUCCESS: no local data found from DataSourceOpenStackLocal

It looks like the ds-identify script creates a list with "None":

$ cat /run/cloud-init/cloud.cfg
datasource_list: [ OpenStack, None ]

In our cloud-init configuration, the last file to be read lexicographically contains:

$ cat /etc/cloud/cloud.cfg.d/99-delphix-datasource.cfg
datasource_list: [ OpenStack ]

The PR mentions:

If users want to force a single datasource regardless of detection, they can do so by removing None from the list.

How are we supposed to do so? The cfg files we have do not have None in the list to begin with.

Steps to reproduce the problem

This should be reproducible by creating a cfg file like the one above and attempting to run cloud-init clean and then cloud-init init. Unfortunately, I do not have an easy way to spin up a vanilla Ubuntu instance in our OCI account but if the issue is not obvious to the maintainers, I can try to spend some time requesting the required access internally.

Environment details

• Cloud-init version: 23.4.4
• Operating System Distribution: Ubuntu 20.04 running kernel 5.15
• Cloud provider, platform or installer type: Oracle cloud

cloud-init logs

$ cat /run/cloud-init/ds-identify.log
[up 5.35s] ds-identify
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
/etc/cloud/cloud.cfg.d/99-delphix-datasource.cfg set datasource_list: [ OpenStack ]
...
DMI_CHASSIS_ASSET_TAG=OracleCloud.com
...
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=5.15.0-1052-dx2024022415-109c29fe5-oracle
UNAME_KERNEL_VERSION=#58~20.04.1 SMP Sat Feb 24 15:13:29 UTC 2024
UNAME_MACHINE=x86_64
UNAME_NODENAME=localhost.localdomain
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=OpenStack
...
...
is_container=false
single entry in datasource_list (OpenStack) use that.
[up 5.45s] returning 0

[palash-gandhi]

cc: @cjp256 could you please shed some light on this?

[TheRealFalcon]

@palash-delphix Sorry for closing earlier, I accidentally hit the "Close with comment" while in the middle of writing my comment.

I'm working on reproducing this, but in the mean time, can you explain what you're trying to achieve with this configuration? Why does /etc/cloud/cloud.cfg.d/99-delphix-datasource.cfg contain datasource_list: [ OpenStack ]? Without any configuration, cloud-init should identify the Oracle datasource and use that. Years ago, the Oracle cloud used the OpenStack datasource, but that generally shouldn't be used anymore. Are you trying to keep some functionality that the new datasource doesn't have, and if so, what? Is this old configuration that has been brought forward that may no longer be necessary? If so, does it work for you to remove /etc/cloud/cloud.cfg.d/99-delphix-datasource.cfg?

[blackboxsw]

Also, if you determine that, for some reason your images/platform needs require OpenStack datasource to be used. This could be resolved without any changes to cloud-init or ds-identify given a simple config file change if you have reasons to retain OpenStack datasource instead of Oracle platform.

From the looks of the DataSourceOpenStack.ds_detect. If Oracle exists in the datasource_list, the OpenStack datasource can be detected and preferred above Oracle DataSource if both OpenStack and Oracle as in datasource_list.

It looks like the following config could be provided in /etc/cloud/cloud.cfg.d/99-delphix-datasource.cfg

datasource_list: [ OpenStack, Oracle ]

This will tell the unique DataSourceOpenStack.ds_detect logic to prefer OpenStack datasource detection on any system which exposes chassis asset tag=OracleCloud.com.

[palash-gandhi]

@TheRealFalcon and @blackboxsw Thank you for the quick response and fix. Much appreciated.