Race condition: interface renaming, udev vs. networkd

[smurfix]

Submission type

• Bug report

systemd version the issue has been seen with

235

Used distribution

Debian, but irrelevant AFAICT

In case of bug report: Expected behaviour you didn't see

I can rename an interface with a udev rule or a .link file

In case of bug report: Unexpected behaviour you saw

networkd is faster and enables the interface before udev gets around to renaming it.
Unfortunately, an interface that's up cannot be renamed

In case of bug report: Steps to reproduce the problem

# cat /etc/systemd/network/10-en0.link:
[Match]
MACAddress=aa:bb:cc:dd:ee:ff

[Link]
MACAddressPolicy=none
Name=en0

Reboot a couple of times. Sometimes the interface is renamed correctly. Sometimes it is not.

Workaround:

# cat systemd-networkd.service.d/after-udev.conf 
[Service]
ExecStartPre=/sbin/udevadm settle

[poettering]

uh? networkd does not consider network devices until udev reported they are available, and it does so only after all rules ran, and hence all interfaces got renamed fully. Are you sure it's udev itself that renames the links for you? maybe you have some other networking scripts in place that up the interface too early or so?

[smurfix]

journalctl sez:

Nov 07 13:22:10 dispatch systemd-networkd[275]: lo: Configured
Nov 07 13:22:10 dispatch systemd-networkd[275]: eth1: Gained carrier
Nov 07 13:22:10 dispatch systemd-networkd[275]: eth0: Gained carrier
Nov 07 13:22:10 dispatch kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Nov 07 13:22:11 dispatch systemd-udevd[282]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 07 13:22:11 dispatch systemd-udevd[282]: error changing net interface name 'eth1' to 'en0': Device or resource busy
Nov 07 13:22:11 dispatch systemd-udevd[282]: could not rename interface '3' from 'eth1' to 'en0': Device or resource busy
Nov 07 13:22:11 dispatch systemd-udevd[283]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 07 13:22:11 dispatch systemd-udevd[283]: error changing net interface name 'eth0' to 'en1': Device or resource busy
Nov 07 13:22:11 dispatch systemd-udevd[283]: could not rename interface '2' from 'eth0' to 'en1': Device or resource busy

I have de-installed ifupdown. The only other possible culprit is open-vm-tools, but its network config script prefers using systemctl start systemd-networkd to calling ip link or ifconfig, so I don't think that's the problem.

[skuti-is]

Having same issue on Ubuntu 17.10 and Ubuntu 18.04 (development branch).
The workaround works, thanks @smurfix

Ubuntu uses netplan to create the .link and .network files under /run/systemd/network/
from /etc/netplan/*.yaml
If the .yaml file contains set-name and static ipaddress for example:
lan0:
match:
macaddress: 52:54:00:xx:xx:xx
set-name: lan0
addresses: [ 10.0.0.1/24 ]

You get the interface name set or the ipaddress, not both.

[atonkyra]

I just started seeing this update on Gentoo after updating from 233 to 236. Few scenarios I tested:

• Interface IP configured by [Match] MACAddress -> interface gets configured by networkd but is not renamed (stays eth0)
• Interface IP configured by [Match] Name -> interface is never configured, restarting networkd helps in this case

Workaround by @smurfix fixes it for me too.

[smurfix]

Umm, the "needs-reporter-feedback" tag seems no longer appropriate here.
Is there something I should test?

[atonkyra]

Looks like the udevadm settle fix does not work on all systems. For now I just created an unit which restarts systemd-networkd on multi-user target...

journalctl output from the affected system (manual restart at 04:08:05):

-- Reboot --
Feb 15 04:07:50 zenith systemd-networkd[6431]: extbr0: netdev ready
Feb 15 04:07:50 zenith systemd-networkd[6431]: intbr0: netdev ready
Feb 15 04:07:50 zenith systemd-networkd[6431]: Enumeration completed
Feb 15 04:07:50 zenith systemd[1]: Started Network Service.
Feb 15 04:07:51 zenith systemd-networkd[6431]: eth0: Interface name change detected, eth0 has been renamed to enp6s0f0.
Feb 15 04:07:51 zenith systemd-networkd[6431]: eth1: Interface name change detected, eth1 has been renamed to enp6s0f1.
Feb 15 04:07:51 zenith systemd-networkd[6431]: extbr0: IPv6 successfully enabled
Feb 15 04:07:51 zenith systemd-networkd[6431]: intbr0: IPv6 successfully enabled
Feb 15 04:08:05 zenith systemd[1]: Stopping Network Service...
Feb 15 04:08:05 zenith systemd[1]: Stopped Network Service.
Feb 15 04:08:05 zenith systemd[1]: Starting Network Service...
Feb 15 04:08:05 zenith systemd-networkd[9560]: extbr0: netdev ready
Feb 15 04:08:05 zenith systemd-networkd[9560]: intbr0: netdev ready
Feb 15 04:08:05 zenith systemd-networkd[9560]: Enumeration completed
Feb 15 04:08:05 zenith systemd-networkd[9560]: intbr0: netdev exists, using existing without changing its parameters
Feb 15 04:08:05 zenith systemd[1]: Started Network Service.
Feb 15 04:08:05 zenith systemd-networkd[9560]: extbr0: netdev exists, using existing without changing its parameters
Feb 15 04:08:05 zenith systemd-networkd[9560]: enp6s0f0: IPv6 successfully disabled
Feb 15 04:08:06 zenith systemd-networkd[9560]: enp6s0f0: Gained carrier
Feb 15 04:08:06 zenith systemd-networkd[9560]: enp6s0f0: Configured

[mooinglemur]

Ran into this on Gentoo after upgrading to 236 (and still also after 238 as I decided to try that first). The udevadm settle ExecStartPre did not work, but I added a sleep 5 ExecStartPre as well, which does appear to allow systemd-networkd to start after devices have been renamed. This seems to indicate that udevadm settle returns before udev is finished processing its rules.

When systemd-networkd sees a network device renamed, it doesn't seem to consider the configs for it with the new name, which puzzles me as well, unless it's an unfortunate consequence of timing.

On my system, r8169 is a built-in, not a module, so I'm not sure if this makes any difference with the timing behavior.

[jlabath]

Likewise encountered this on gentoo (systemd 236). And this thread was super helpful. The one thing that threw me off (being new to systemd) was where to put the conf file. So for reference.

jakub@host ~ $ cat /etc/systemd/system/systemd-networkd.service.d/01-after-udev.conf 
[Service]
ExecStartPre=/usr/local/sbin/fileexist /sys/class/net/enp5s0 15

PS: the fileexist is a little program that just waits for up to 15 seconds for the /sys/class/net/enp5s0 to exist. As I felt that's better than just arbitrary sleep. But the idea is the same - wait for the rename to happen.

[mmjo]

ubuntu link: https://bugs.launchpad.net/netplan/+bug/1770082

[yuwata]

So, is this still reproducible with v240+? If so, please provide relevant configs and logs.

[yuwata]

As already @poettering commented, networkd does not configure interfaces not initialized by udevd.
So, the possible story why this happens I guess is the following,

• the interface is once initialized by udevd, may be in the stage of initrd.
• I do not know why, but at the time, network interfaces are not renamed, maybe rules or .link files in your initrd disable that.
• after switching root, networkd brings up network interfaces, as they are already initialized.
• host's udevd try to rename the interfaces as it is requested by the host's .link file. But as you can see, it cannot. Because interfaces are already active.

So, please check the rules or .link files in your initrd. I guess some contradiction with host's setting.

Anyway, try to boot with udev.log_priority=debug and systemd.log_level=debug. And please provide logs of udevd and networkd.

[yuwata]

Does this issue appear only on containers? Or does this appear also on bare systems? If only containers, then #11816 may help this issue. If you are interested the PR, please test it. Thank you.

[smurfix]

Seen on bare systems, but I'll test that PR anyway.

[yuwata]

Could you boot the system with systemd.log_level=debug udev.log_priority=debug? Then, please provide journalctl -b -u systemd-networkd.service -u systemd-udevd.service --no-hostname. Thank you.

[yuwata]

I am not sure, but I hope #11881 helps this issue, and the PR is already merged. If the PR does not fix this issue, please paste the relevant logs I requested in the above, then I will reopen this. Thank you.

[hroemer]

@yuwata This issue bugged me for quite some time now, even though my network interface name did not change. But obviously some virtual devices created by docker would use the original name eth0 as well before being renamed, I don't know whether this is really the case, though.

systemd version the issue has been seen with

242 running linux 5.5 or 5.4

Workaround

Similar to the workaround mentioned in the issue description I found another way to handle it by creating a drop-in file based upon NixOS/nixpkgs#39340:

systemctl edit systemd-networkd.service

and adding these lines:

[Unit]
After=systemd-udev-settle.service

Then after reloading the daemon via system-ctl daemon-reload the drop-in override.conf becomes visible:

systemctl status systemd-networkd.service

● systemd-networkd.service - Network Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/systemd-networkd.service.d
           └─override.conf

You can check the dependency tree via

systemctl list-dependencies --after systemd-networkd

systemd-networkd.service
● ├─-.mount
● ├─system.slice
● ├─systemd-journald.socket
● ├─systemd-networkd.socket
● ├─systemd-sysctl.service
● ├─systemd-sysusers.service
● ├─systemd-udev-settle.service
● ├─systemd-udevd.service
● └─network-pre.target
●   ├─ip6tables.service
●   ├─ipset.service
●   └─iptables.service

The systemd-networkd will now wait until udev is settled and upon system startup my interface is consistently up and running again.

Thus I believe there the race condition still applies here.
Could systemd-udev-settle.service be added to the original systemd-networkd.service dependency list?

[keszybz]

Could systemd-udev-settle.service be added to the original systemd-networkd.service dependency list?

No way. systemd-networkd is written to handle hotplug event properly, and udev-settle is a terrible hack for programs that don't understand hotplug.