udev needs to be triggered if rules are written to a new file

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1817368

Launchpad details

affected_projects = []
assignee = rjschwei
assignee_name = Robert Schweikert
date_closed = 2019-12-19T23:01:23.952943+00:00
date_created = 2019-02-22T20:52:02.558558+00:00
date_fix_committed = 2019-07-17T20:23:46.298183+00:00
date_fix_released = 2019-12-19T23:01:23.952943+00:00
id = 1817368
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1817368
milestone = None
owner = rjschwei
owner_name = Robert Schweikert
private = False
status = fix_released
submitter = rjschwei
submitter_name = Robert Schweikert
tags = []
duplicates = []

Launchpad user Robert Schweikert(rjschwei) wrote on 2019-02-22T20:52:02.558558+00:00

On a VM with many interfaces it is possible that udev is slow. This may lead to the situation that 70-persistent-net.rules is being written to by cloud-init while also being written to by the net rules generator (75-persistent-net-generator.rules). This may lead to interleaving, duplicate definitions or incomplete data.

The user can already specify an optional rules file with the "netrules_path". However this will not have any effect when the file gets written as udev is not re-triggered. When the user configures a custom file (which logically should be something like 71-persistent-net.rules) cloud-init should trigger udev to add any new rules the code might have written.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2019-02-26T14:22:47.124402+00:00

I think we're in conflict with 75-persistent-net-generator.rules; as both cloud-init and that rule are attempting to map a MAC and interface name. This is going to be trouble, as you've seen.

Even if we sorted the race, if users may be confused as to the two files, or the interleaved values etc.

Does the image have systemd and persistent net naming enabled?

In cases like this, cloud-init has introduced logic to check if an in-image file is present and renames, or deletes it if it's taking an expected action (like generating persistent net rules for network config it has discovered).

[ubuntu-server-builder]

Launchpad user Robert Schweikert(rjschwei) wrote on 2019-02-26T15:44:14.706153+00:00

Well, the way I see, based on investigation and input from others at SUSE we have 2 options. We can set the default file name for the netrules_path to something safe (80-pesistent-netrules-cloud-init.rules) for example. That would avoid the issue, the cloud-init generated rules would be last and thus the user supplied network setup would be applied. If we choose this path then IMHO netrules_path should no longer be configurable.

If we stay with the current intend, i.e. netrules_path is configurable then we want the updates in MP.

I have no strong opinion either way.

The image used that created this problem does have persistent names and systemd, yes. It also has a "large" number of network interfaces.

The way the systemd dependencies are setup today cloud-init-local.service and systemd-udevd.service can run at the same time and both write to 70-persistent-net.rules so there will always be a race. The third option is the cloud-init-local.service gets an

"After=systemd-udevd.service"

but I am not certain that will not create a cycle as cloud-init-local.service also has a

"Before=network-pre.target"

If we can avoid cycles, then the best solution, IMHO, is the change in order in cloud-init-local.service.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2019-03-21T14:29:45.132598+00:00

I'm still a little confused around the presence of 75-persistent-net-generator.rules AND having systemd creating persistent nic names. That's going to cause problems unless it's always booted with net.ifnames=0.

Putting that aside, I do think using a higher number (as you suggest 80-persistent-net-cloud-init.rules) is a good fix which avoids the race by writing to different files altogether.

I don't think we need to prevent netrules_path from being configured; in Ubuntu Xenial, we still utilize our on known value. You can be the path in your Distro class as part of the renderer config.

% git diff cloudinit/distros/opensuse.py
diff --git a/cloudinit/distros/opensuse.py b/cloudinit/distros/opensuse.py
index 1bfe047..b48d40e 100644
--- a/cloudinit/distros/opensuse.py
+++ b/cloudinit/distros/opensuse.py
@@ -36,6 +36,8 @@ class Distro(distros.Distro):
tz_local_fn = '/etc/localtime'
renderer_configs = {
'sysconfig': {

•        'netrules_path':
  

•            'etc/udev/rules.d/80-persistent-net-cloud-init.rules',
         'control': 'etc/sysconfig/network/config',
         'iface_templates': '%(base)s/network/ifcfg-%(name)s',
         'route_templates': {
  

[ubuntu-server-builder]

Launchpad user Robert Schweikert(rjschwei) wrote on 2019-04-19T16:34:22.845768+00:00

I have updated the code. The current implementation works in the customer case. Also should not call settle in general, only when we get into the situation where things are "racy".

But I am having difficulties sorting out the tests and could use some help. Ryan, Dan, can you help me with the tests?

Thanks

[ubuntu-server-builder]

Launchpad user Robert Schweikert(rjschwei) wrote on 2019-05-31T15:41:57.498759+00:00

Still need help with the tests, Ryan, Dan?

Thanks

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2019-06-03T14:21:32+00:00

On Fri, May 31, 2019 at 10:56 AM Robert Schweikert <
1817368@bugs.launchpad.net> wrote:

Still need help with the tests, Ryan, Dan?

Thanks for the ping. We've not completely forgotten =)

Ryan

Thanks

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1817368

Title:
udev needs to be triggered if rules are written to a new file

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1817368/+subscriptions

[ubuntu-server-builder]

Launchpad user Server Team CI bot(server-team-bot) wrote on 2019-07-17T20:23:44.945693+00:00

This bug is fixed with commit b3a87fc to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=b3a87fc0

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2019-12-19T23:01:25.105688+00:00

This bug is believed to be fixed in cloud-init in version 19.2-1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.