cloud-init incorrectly always uses sysconfig as the Network Renderer for RHEL9

[chewborg]

Bug report

Since RHEL 9 the default storage of network configurations is now NetworkManager keyfiles, not sysconfig network ifcfg files.
https://www.redhat.com/en/blog/rhel-9-networking-say-goodbye-ifcfg-files-and-hello-keyfiles
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-single/9.0_release_notes/index#enhancement_networking

However a change to the decision evaluation on which network renderer to use now causes sysconfig to be used in all cases, rather than the expected network-manager renderer.
Issue 4131
Pull Request 4132
This is because the RHEL9 NetworkManager package also always includes the file /usr/lib64/NetworkManager/*/libnm-settings-plugin-ifcfg-rh.so, now used as the inescapable determination of whether to use the sysconfig or network-manager renderers, though using NetworkManager keyfiles now is the default.

So cloud-init is overriding the default behaviour and the result is a RHEL9 OS configured with cloud-init is in an unexpect state. Which is the Interfaces are managed by NetworkManager as ifcfg files in /etc/sysconfig/network-scripts/ rather than the expected keyfiles in /etc/NetworkManager/system-configurations/

Additional steps then need to be undertaken to migrate these configurations to keyfile format.

We expect to find the default state for RHEL9 Networking when using an unmodified cloud-config, and there does not appear to be a way for overriding this except to add a custom renderer list in /etc/cloud/cloud.cfg.d/ excluding sysconfig in custom images.

Steps to reproduce the problem

1. On a RHEL9 or Rocky9 OS, install the default cloud-init package from the RHEL repos.
2. Use the image created from this instance in Openstack to create a new VM.
3. It will always create the Interfaces backend as /etc/sysconfig/network-scripts/ifcfg-{intf}

Environment details

• Cloud-init version: 23.4-7.el9_4.3
• Operating System Distribution: RHEL9, Rocky9
• Cloud provider, platform or installer type: Openstack

[root@rhel-as9-tc6k log]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.4 (Plow)

[root@rhel-as9 log]# cat /etc/sysconfig/network-scripts/readme-ifcfg-rh.txt 
NetworkManager stores new network profiles in keyfile format in the
/etc/NetworkManager/system-connections/ directory.

Previously, NetworkManager stored network profiles in ifcfg format
in this directory (/etc/sysconfig/network-scripts/). However, the ifcfg
format is deprecated. By default, NetworkManager no longer creates
new profiles in this format
...

cloud-init logs

2024-07-15 20:20:17,714 - subp.py[DEBUG]: Running command ['ip', '-6', 'addr', 'show', 'permanent', 'scope', 'global'] with allowed return codes [0]
 (shell=False, capture=True)
2024-07-15 20:20:17,718 - subp.py[DEBUG]: Running command ['ip', '-4', 'addr', 'show'] with allowed return codes [0] (shell=False, capture=True)
2024-07-15 20:20:17,723 - __init__.py[DEBUG]: Detected interfaces {'lo': {'downable': False, 'device_id': None, 'driver': None, 'mac': '00:00:00:00:
00:00', 'name': 'lo', 'up': True}, 'eth0': {'downable': True, 'device_id': '0x0001', 'driver': 'virtio_net', 'mac': 'fa:16:3e:5c:c0:52', 'name': 'et
h0', 'up': False}}
2024-07-15 20:20:17,723 - __init__.py[DEBUG]: no work necessary for renaming of [['fa:16:3e:5c:c0:52', 'eth0', 'virtio_net', '0x0001']]
2024-07-15 20:20:17,723 - stages.py[INFO]: Applying network configuration from ds bringup=False: {'version': 1, 'config': [{'type': 'physical', 'mtu': 1500, 'subnets': [{'type': 'dhcp4'}], 'mac_address': 'fa:16:3e:5c:c0:52', 'name': 'eth0'}, {'type': 'nameserver', 'address': '10.26.2.2'}, {'type': 'nameserver', 'address': '10.26.3.192'}, {'type': 'nameserver', 'address': '10.26.3.21'}, {'type': 'nameserver', 'address': '10.26.3.144'}]}
2024-07-15 20:20:17,724 - util.py[DEBUG]: Writing to /run/cloud-init/sem/apply_network_config.once - wb: [644] 23 bytes
2024-07-15 20:20:17,725 - util.py[DEBUG]: Restoring selinux mode for /run/cloud-init/sem/apply_network_config.once (recursive=False)
2024-07-15 20:20:17,726 - util.py[DEBUG]: Restoring selinux mode for /run/cloud-init/sem/apply_network_config.once (recursive=False)
2024-07-15 20:20:17,729 - __init__.py[DEBUG]: Selected renderer 'sysconfig' from priority list: ['sysconfig', 'eni', 'netplan', 'network-manager', 'networkd']
2024-07-15 20:20:17,733 - util.py[DEBUG]: Writing to /etc/sysconfig/network-scripts/ifcfg-eth0 - wb: [644] 176 bytes
2024-07-15 20:20:17,734 - util.py[DEBUG]: Restoring selinux mode for /etc/sysconfig/network-scripts/ifcfg-eth0 (recursive=False)
2024-07-15 20:20:17,735 - util.py[DEBUG]: Restoring selinux mode for /etc/sysconfig/network-scripts/ifcfg-eth0 (recursive=False)
2024-07-15 20:20:17,735 - util.py[DEBUG]: Reading from /etc/resolv.conf (quiet=False)
2024-07-15 20:20:17,735 - util.py[DEBUG]: Read 55 bytes from /etc/resolv.conf
2024-07-15 20:20:17,735 - util.py[DEBUG]: Writing to /etc/resolv.conf - wb: [644] 198 bytes
2024-07-15 20:20:17,736 - util.py[DEBUG]: Restoring selinux mode for /etc/resolv.conf (recursive=False)
2024-07-15 20:20:17,737 - util.py[DEBUG]: Restoring selinux mode for /etc/resolv.conf (recursive=False)
2024-07-15 20:20:17,738 - util.py[DEBUG]: Writing to /etc/NetworkManager/conf.d/99-cloud-init.conf - wb: [644] 72 bytes
2024-07-15 20:20:17,738 - util.py[DEBUG]: Restoring selinux mode for /etc/NetworkManager/conf.d/99-cloud-init.conf (recursive=False)
2024-07-15 20:20:17,739 - util.py[DEBUG]: Restoring selinux mode for /etc/NetworkManager/conf.d/99-cloud-init.conf (recursive=False)
2024-07-15 20:20:17,739 - util.py[DEBUG]: Writing to /etc/udev/rules.d/70-persistent-net.rules - wb: [644] 96 bytes
2024-07-15 20:20:17,739 - util.py[DEBUG]: Restoring selinux mode for /etc/udev/rules.d/70-persistent-net.rules (recursive=False)
2024-07-15 20:20:17,740 - util.py[DEBUG]: Restoring selinux mode for /etc/udev/rules.d/70-persistent-net.rules (recursive=False)
2024-07-15 20:20:17,740 - util.py[DEBUG]: Reading from /etc/sysconfig/network (quiet=True)
2024-07-15 20:20:17,740 - util.py[DEBUG]: Read 37 bytes from /etc/sysconfig/network
2024-07-15 20:20:17,740 - util.py[DEBUG]: Writing to /etc/sysconfig/network - wb: [644] 107 bytes
2024-07-15 20:20:17,742 - util.py[DEBUG]: Restoring selinux mode for /etc/sysconfig/network (recursive=False)
2024-07-15 20:20:17,742 - util.py[DEBUG]: Restoring selinux mode for /etc/sysconfig/network (recursive=False)
2024-07-15 20:20:17,743 - __init__.py[DEBUG]: Not bringing up newly configured network interfaces
2024-07-15 20:20:17,743 - main.py[DEBUG]: [local] Exiting. datasource DataSourceOpenStackLocal [net,ver=2] not in local mode.

[blackboxsw]

Thank you @chewborg for filing this bug and improving cloud-init. Can you clarify what behavior is broken by cloud-init's decision to render sysconfig files instead of /etc/NetworkManager/system-connections/. Is network inoperable, does this conflict with other network configuration emitted on the system in NetworkManager/system-connection/*?

My understanding from the bug is that cloud-init is still rendering content in what's considered "deprecated behavior" by the /etc/sysconfig/network-scripts/readme-ifcfg-rh.txt file.

@ani-sinha this request/issue seems at conflict with the RedHat/Rocky specific changes we added in #4132 which was a strict decision to always use sysconfig if you have the plugin ifcfg-rh. I have no visibility to RHBZ: 2194050 so I don't know specifically what bug #4132 was trying to fix.

Do we want to consider either of the following:

• Do we want to alter cloud-init to disregard sysconfig options in certain cases on RedHat even if ifcfg-rh plugin exists?
• Do we want to update /etc/sysconfig/network-scripts/readme-ifcfg-rh.txt delivered by NetworkManager-1.46.0-13.el9_4.x86_64 to document known exceptions for cloud-init behavior to avoid confusion
• Some other indicator in the environment that informs cloud-init default behavior correctly to use sysconfig vs network-manager?

[ani-sinha]

In RHEL 9 use of Sysconfig renderer is the default. That is because the Sysconfig renderer has the higher priority and the ifcfg plug-in is available. We will never make a major decision to flip the switch and always use network manager renderer by default in the middle of the major RHEL version. If users want, they can override the default priority and use network manager renderer with a higher priority so that it takes precedence over Sysconfig renderer.

In RHEL 10 network manager renderer is the default and is the only way to configure the network. Therefore network manager renderer has the higher priority over Sysconfig renderer (and you can't enable Sysconfig renderer as the ifcfg plugin is absent).

[ani-sinha]

In terms of documentation, I will draw the attention of our doc team to check if this has been documented in one of RH KCs , if not add one.

[ani-sinha]

Another data point is this - RHEL 9.2 and below did not support network manager renderer. It's only RHEL 9.3 and above that supports both Sysconfig and network manager renderer. Hence we won't make network manager renderer default for RHEL 9. Instead we will use the RHEL 9 train to stabilize network manager renderer before making the big switch to making it the default in RHEL 10.

[chewborg]

Hi @ani-sinha

Now I'm not sure if we are talking about cloud-init or the RHEL OS in the above
For a RHEL v9.0 installation the default is to use NetworkManager keyfiles, and I just did a quick KVM install via the ISO to verify

[root@rhel90-iso-install patrick]# nmcli -f NAME,TYPE,AUTOCONNECT,ACTIVE,DEVICE,STATE,FILENAME connection show 
NAME    TYPE      AUTOCONNECT  ACTIVE  DEVICE  STATE      FILENAME                                                   
enp1s0  ethernet  yes          yes     enp1s0  activated  /etc/NetworkManager/system-connections/enp1s0.nmconnection 
[root@rhel90-iso-install patrick]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.0 (Plow)

But perhaps I'm misinterpreting and you are telling me that cloud-init should only use sysconfig and not the network-manager renderer. If so I wasn't aware of this, or at least I didn't find this documented.
I'm probably out of my depth knowledgewise so I guess I didn't find what I expected, and I'm probably not alone there.

Honestly, as the nmcli tool will work with either ifcfg or nmconnection files for most of its functionality it's not a big deal I guess. If you want your vm to use keyfiles
For a Openstack VM using cloud-init you can either run
nmcli connection migrate

Or create a custom image with something like

cat /etc/cloud/cloud.cfg.d/92_network_setup.cfg 
system_info:
  network:
    renderers: ['network-manager']

And you'll get the state we wish to use.

In that case I think a solution to the issue I raised is to ensure the documention for cloud-init that the Network Renderer will be sysconfig by default for RHEL 9 versions (and clones) using cloud-init.

[ani-sinha]

@chewborg

Now I'm not sure if we are talking about cloud-init or the RHEL OS in the above

cloud-init.

But perhaps I'm misinterpreting and you are telling me that cloud-init should only use sysconfig and not the network-manager renderer.

No, what I am saying is this. For RHEL 9.0, 9.1 and 9.2, cloud-init can only use syscnfig renderer as support for network manager renderer is absent. From RHEL 9.3 onwards, both sysconfig and network manager renderer can be used.

Or create a custom image with something like

cat /etc/cloud/cloud.cfg.d/92_network_setup.cfg 
system_info:
network:
  renderers: ['network-manager']

So essentially what you are doing here is removing other renderers and keeping only network-manager renderer. You do not need to do that. You can simply give network-manager renderer a higher priority over sysconfig, something like:

  network:
    renderers: ['netplan', 'network-manager', 'networkd', 'sysconfig', 'eni']

and then it will give priority to network-manager renderer. Again, remember that this will only work for RHEL 9.3 and above.
By default, cloud-init in RHEL 9.3 and above will continue to use sysconfig renderer.

I will check with our doc team to see if we can better document this.

[TheRealFalcon]

Thanks for the additional context Ani.

From upstream's perspective, this works as expected. Cloud-init correctly determines that sysconfig is installed and uses the sysconfig renderer if sysconfig is orderered earlier in the renderer list.

We have a templated config file that can be modified to specify renderer ordering when building cloud-init, though it looks like this is already being overridden downstream. If you want to submit a PR to make a change specific to Rocky in this template, we're happy to accept it.

[jorhett]

@ani-sinha can you please clarify this comment?

No, what I am saying is this. For RHEL 9.0, 9.1 and 9.2, cloud-init can only use syscnfig renderer as support for network manager renderer is absent.

As RHEL 8.0 above include fully functional Network Manager features, and in fact I'm using network manager renderer on an (old) 9.1 machine right now... it's really not clear to me what you are saying here.

Is this more a statement that the vendor-packaged cloud-init versions that ship with RHEL < 9.3 don't support this? Or can you be more specific about what is missing?

Part of the reason for my question is because what cloud-init ships in new versions will never, ever impact an OS-stock-only cloud-init version since RHEL doesn't change them. So having a newer version of cloud-init provide NM default for RHEL 9.0+ will again have zero impact. It's not a risk, it's instead a feature. "Update cloud-init in your boot image to...."