Skip to content
This repository has been archived by the owner on Oct 16, 2020. It is now read-only.

networkd starts before udev #1767

Open
crawford opened this issue Jan 17, 2017 · 11 comments
Open

networkd starts before udev #1767

crawford opened this issue Jan 17, 2017 · 11 comments
Labels
area/usability component/distro kind/bug kind/support priority/P0 team/os

Comments

@crawford
Copy link
Contributor

Issue Report

Bug

CoreOS Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1284.2.0
VERSION_ID=1284.2.0
BUILD_ID=2017-01-10-0545
PRETTY_NAME="Container Linux by CoreOS 1284.2.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

All

Expected Behavior

udev is able to rename interfaces.

Actual Behavior

Errors like the following are seen because networkd starts before udev:

Jan 16 23:55:16 localhost systemd-udevd[1315]: Error changing net interface name 'eth2' to 'ens802f0': Device or resource busy
Jan 16 23:55:16 localhost systemd-udevd[1315]: could not rename interface '4' from 'eth2' to 'ens802f0': Device or resource busy
@qrpike
Copy link

qrpike commented Feb 28, 2017

I am getting this error on only 1 machine. Latest CoreOS stable. It's an intel NIC if that matters.

@dm0-
Copy link

dm0- commented Mar 21, 2017

Is there a reliable way to reproduce this error? I haven't seen it, and the upstream systemd-networkd.service has After=systemd-udevd.service on all channels.

@ndenev
Copy link

ndenev commented May 5, 2017

We're also seeing this. While systemd-udevd.service does start shortly before systemd-networkd.service, there are Error changing net interface name entries, and from boot to boot I get a different combinations of eth* and eno* interfaces (some renamed some not)

[stg] root@k8s1-5 system # journalctl -o short-precise -u systemd-networkd.service | head -n2
-- Logs begin at Fri 2017-05-05 10:36:45 UTC, end at Fri 2017-05-05 11:32:44 UTC. --
May 05 10:36:45.647795 localhost systemd[1]: Starting Network Service...
[stg] root@k8s1-5 system # journalctl -o short-precise -u systemd-udevd.service | head -n2
-- Logs begin at Fri 2017-05-05 10:36:45 UTC, end at Fri 2017-05-05 11:32:50 UTC. --
May 05 10:36:45.642651 localhost systemd[1]: Starting udev Kernel Device Manager...
[stg] root@k8s1-5 system #

May 05 10:36:45.775588 localhost kernel: tg3 0000:02:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b2
May 05 10:36:45.775763 localhost kernel: tg3 0000:02:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.775922 localhost kernel: tg3 0000:02:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.776117 localhost kernel: tg3 0000:02:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.791031 localhost kernel: tg3 0000:02:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b3
May 05 10:36:45.791126 localhost kernel: tg3 0000:02:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.791217 localhost kernel: tg3 0000:02:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.791295 localhost kernel: tg3 0000:02:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.807288 localhost kernel: tg3 0000:03:00.0 eth2: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b4
May 05 10:36:45.807408 localhost kernel: tg3 0000:03:00.0 eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.807497 localhost kernel: tg3 0000:03:00.0 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.807782 localhost kernel: tg3 0000:03:00.0 eth2: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.829230 localhost kernel: tg3 0000:03:00.1 eth3: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b5
May 05 10:36:45.829326 localhost kernel: tg3 0000:03:00.1 eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.829426 localhost kernel: tg3 0000:03:00.1 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.829535 localhost kernel: tg3 0000:03:00.1 eth3: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.832921 localhost systemd-networkd[404]: eth3: IPv6 enabled for interface: Success
May 05 10:36:45.958443 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
May 05 10:36:45.972112 localhost systemd-networkd[404]: eth0: IPv6 enabled for interface: Success
May 05 10:36:46.101679 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 05 10:36:46.342269 localhost systemd-networkd[404]: eth2: IPv6 enabled for interface: Success
May 05 10:36:46.468596 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
May 05 10:36:46.468763 localhost systemd-networkd[404]: eth1: IPv6 enabled for interface: Success
May 05 10:36:46.593586 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
May 05 10:36:50.273248 localhost systemd-udevd[1953]: Error changing net interface name 'eth2' to 'eno3': Device or resource busy
May 05 10:36:50.385293 localhost kernel: tg3 0000:03:00.1 eno4: renamed from eth3
May 05 10:36:50.385493 localhost kernel: tg3 0000:02:00.0 eno1: renamed from eth0
May 05 10:36:50.273259 localhost systemd-udevd[1953]: could not rename interface '4' from 'eth2' to 'eno3': Device or resource busy
May 05 10:36:50.288118 localhost systemd-udevd[1974]: Error changing net interface name 'eth1' to 'eno2': Device or resource busy
May 05 10:36:50.288124 localhost systemd-udevd[1974]: could not rename interface '3' from 'eth1' to 'eno2': Device or resource busy
May 05 10:36:51.877535 localhost systemd-networkd[3063]: eno4: IPv6 enabled for interface: Success
May 05 10:36:52.003593 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eno4: link is not ready
May 05 10:36:52.024205 localhost systemd-networkd[3063]: eno1: IPv6 enabled for interface: Success
May 05 10:36:52.734042 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
May 05 10:36:55.827881 localhost kernel: tg3 0000:02:00.0 eno1: Link is up at 1000 Mbps, full duplex
May 05 10:36:55.827989 localhost kernel: tg3 0000:02:00.0 eno1: Flow control is off for TX and off for RX
May 05 10:36:55.828127 localhost kernel: tg3 0000:02:00.0 eno1: EEE is disabled
May 05 10:36:55.828760 localhost systemd-networkd[3063]: eno1: Gained carrier
May 05 10:36:55.828987 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready
May 05 10:36:57.473690 localhost systemd-networkd[3063]: eno1: Gained IPv6LL

This is on Stable-1298.7.0

@ndenev
Copy link

ndenev commented May 8, 2017

Of course setting debug in the kernel config line makes this work :(

@alialshamali
Copy link

alialshamali commented May 10, 2017 •
edited

I am getting the same error, when I reboot the server, its doesn't come up. After looking at the log I find

Error changing net interface name 'eth0' to 'eno1' : Device or resource busy

Only after rebooting the server it come up and take the configured IP address.

Is there any fix?

Just to add its a intel NIC too

@ndenev
Copy link

ndenev commented May 10, 2017

I resorted to disabling the "predictable" interface names altogether and use ethX naming.

@alialshamali
Copy link

Thanks @ndenev
Seems to be a workaround, but wondering what is the root cause. I am trying to analyze to figure what in the kernel is causing this behavior. Its very strange.

@alialshamali
Copy link

I notice the issue was resolved when I removed the multiple ip addresses from the interface and restart ed the interface. Can this be related? Once I did that I no longer have this issue.

@paulmcgoldrick
Copy link

paulmcgoldrick commented Aug 3, 2017 •
edited

Same issue.

[Thu Aug  3 16:45:49 2017] IPv6: ADDRCONF(NETDEV_CHANGE): eth7: link becomes ready
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.1 eno2: renamed from eth1
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.2 eno3: renamed from eth2
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.0 eno1: renamed from eth0
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.3 eno4: renamed from eth3
[Thu Aug  3 16:45:51 2017] ixgbe 0000:44:00.0 enp68s0f0: renamed from eth6
[Thu Aug  3 16:45:51 2017] ixgbe 0000:43:00.0 enp67s0f0: renamed from eth4
[Thu Aug  3 16:45:51 2017] ixgbe 0000:43:00.1 enp67s0f1: renamed from eth5

Aug 03 16:45:47 localhost systemd-udevd[1425]: Error changing net interface name 'eth7' to 'enp68s0f1': Device or resource busy
Aug 03 16:45:47 localhost systemd-udevd[1425]: could not rename interface '9' from 'eth7' to 'enp68s0f1': Device or resource busy


coreos-16 etc # cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1409.6.0
VERSION_ID=1409.6.0
BUILD_ID=2017-07-06-0128
PRETTY_NAME="Container Linux by CoreOS 1409.6.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

@awsiv
Copy link

awsiv commented Aug 11, 2017 •
edited

I'm seeing the same issue as well:

Aug 10 09:44:59 systemd-udevd[975]: Error changing net interface name 'eth6' to 'ens1f0': Device or resource busy
Aug 10 09:44:59 systemd-udevd[975]: could not rename interface '8' from 'eth6' to 'ens1f0': Device or resource busy

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1409.7.0
VERSION_ID=1409.7.0
BUILD_ID=2017-07-19-0005
PRETTY_NAME="Container Linux by CoreOS 1409.7.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

@awsiv
Copy link

awsiv commented Sep 25, 2017

Any updates on this?

@xeji xeji mentioned this issue Apr 17, 2018
Closed
@pothos pothos mentioned this issue Feb 13, 2020
Merged
pothos added a commit to flatcar-archive/coreos-overlay that referenced this issue Feb 25, 2020
@pothos
sys-kernel/bootengine: Re-enable persistent ifnames
dee00c0 
In the initramfs persistent ifnames were disabled.
This caused problems because when the renaming was
done after the initramfs, a race made it fail, as
originally reported in
coreos/bugs#1767

Reverts the booteninge commit
"systemd: add module to disable network device renaming"
and widens the networkd match rule for DigitalOcean.
@pothos pothos mentioned this issue Feb 25, 2020
Merged
pothos added a commit to flatcar-archive/coreos-overlay that referenced this issue Feb 26, 2020
@pothos
sys-kernel/bootengine: Re-enable persistent ifnames
f2e386a 
In the initramfs persistent ifnames were disabled.
This caused problems because when the renaming was
done after the initramfs, a race made it fail, as
originally reported in
coreos/bugs#1767

Reverts the booteninge commit
"systemd: add module to disable network device renaming"
and widens the networkd match rule for DigitalOcean.
pothos added a commit to flatcar-archive/coreos-overlay that referenced this issue Feb 26, 2020
@pothos
sys-kernel/bootengine: Re-enable persistent ifnames
49a3b42 
In the initramfs persistent ifnames were disabled.
This caused problems because when the renaming was
done after the initramfs, a race made it fail, as
originally reported in
coreos/bugs#1767

Reverts the booteninge commit
"systemd: add module to disable network device renaming"
and widens the networkd match rule for DigitalOcean.
pothos added a commit to flatcar-archive/coreos-overlay that referenced this issue Feb 26, 2020
@pothos
sys-kernel/bootengine: Re-enable persistent ifnames
52d6d44 
In the initramfs persistent ifnames were disabled.
This caused problems because when the renaming was
done after the initramfs, a race made it fail, as
originally reported in
coreos/bugs#1767

Reverts the booteninge commit
"systemd: add module to disable network device renaming"
and widens the networkd match rule for DigitalOcean.
@goochjj goochjj mentioned this issue Mar 4, 2020
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area/usability component/distro kind/bug kind/support priority/P0 team/os
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@alialshamali @crawford @qrpike @ndenev @awsiv @dm0- @paulmcgoldrick