Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-networkd-wait-online fails with bridged interfaces #2154

Closed
mrschyte opened this issue Dec 13, 2015 · 14 comments
Closed

systemd-networkd-wait-online fails with bridged interfaces #2154

mrschyte opened this issue Dec 13, 2015 · 14 comments

Comments

@mrschyte
Copy link

@mrschyte mrschyte commented Dec 13, 2015

I have a setup where several vlans are created and all interfaces are parts of different bridges.
Unfortunately the wait online binary fails to detect that the network is up in this case.

Error message in journal:

Failed to start Wait for Network to be Configured.

I suspect that this is caused by the binary calling the "manager_all_configured" function which returns true only when the operational state for a link is degraded or routable, however in my case the state is in "carrier".

I've tried to call systemd-networkd-wait-online when the system was already up with the "-i IFACE" flag for each interface, but no matter what interface is specified, the utility waits until the timeout has been reached.

@mrschyte
Copy link
Author

@mrschyte mrschyte commented Dec 13, 2015

The interface are setup similarly to this:

int.network:

[Match]
Name=eth*

[Network]
VLAN=lan0

lan.network:

[Match]
Name=lan*

[Network]
Bridge=brlan0

lan.netdev:

[NetDev]
Name=lan0
Kind=vlan

[VLAN]
Id=7

@juhokuu
Copy link

@juhokuu juhokuu commented Sep 14, 2016

Happens here too: systemd-networkd-wait-online will not detect network on a bridge interface and times out. However, it does work as expected if I boot with ipv6.disable=1. In both cases, the logs show the bridge gaining a carrier. I'm on v231.

My /etc/systemd/network is:

enp1s0.network

[Match]
Name=enp1s0

[Network]
Bridge=br1

br1.netdev

[NetDev]
Name=br1
Kind=bridge

br1.network

[Match]
Name=br1

[Network]
DHCP=ipv4

Systemd-networkd.service and systemd-networkd-wait-online.service in the absence of ipv6.disable=1:

Sep 14 17:06:15 crossbeam systemd[1]: Starting Wait for Network to be Configured...
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:22 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:23 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:33 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:08:15 crossbeam systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 17:08:15 crossbeam systemd[1]: Failed to start Wait for Network to be Configured.
Sep 14 17:08:15 crossbeam systemd[1]: systemd-networkd-wait-online.service: Unit entered failed state.
Sep 14 17:08:15 crossbeam systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Sep 14 17:06:14 crossbeam systemd[1]: Starting Network Service...
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: netdev ready
Sep 14 17:06:15 crossbeam systemd-networkd[262]: Enumeration completed
Sep 14 17:06:15 crossbeam systemd[1]: Started Network Service.
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Renamed to eth0
Sep 14 17:06:15 crossbeam systemd-networkd[262]: eth0: Renamed to enp1s0
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: IPv6 enabled for interface: Success
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: Could not append VLANs: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: Could not set bridge vlan: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: IPv6 disabled for interface: Success
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Could not append VLANs: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Could not set bridge vlan: Operation not permitted
Sep 14 17:06:18 crossbeam systemd-networkd[262]: enp1s0: Gained carrier
Sep 14 17:06:18 crossbeam systemd-networkd[262]: br1: Gained carrier
Sep 14 17:06:18 crossbeam systemd-networkd[262]: br1: DHCPv4 address 192.168.0.104/24 via 192.168.0.1
Sep 14 17:06:19 crossbeam systemd-networkd[262]: br1: Gained IPv6LL
Sep 14 17:06:23 crossbeam systemd-networkd[262]: br1: Configured

And when booted with ipv6.disable=1

Sep 14 17:10:28 crossbeam systemd[1]: Starting Wait for Network to be Configured...
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd[1]: Started Wait for Network to be Configured.
Sep 14 17:10:28 crossbeam systemd[1]: Starting Network Service...
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: netdev ready
Sep 14 17:10:28 crossbeam systemd-networkd[257]: Enumeration completed
Sep 14 17:10:28 crossbeam systemd[1]: Started Network Service.
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Renamed to eth0
Sep 14 17:10:28 crossbeam systemd-networkd[257]: eth0: Renamed to enp1s0
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Cannot disable IPv6 for interface br1: No such file or directory
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Could not append VLANs: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Could not set bridge vlan: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Cannot disable IPv6 for interface enp1s0: No such file or directory
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Could not append VLANs: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Could not set bridge vlan: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Configured
Sep 14 17:10:31 crossbeam systemd-networkd[257]: enp1s0: Gained carrier
Sep 14 17:10:31 crossbeam systemd-networkd[257]: enp1s0: Configured
Sep 14 17:10:31 crossbeam systemd-networkd[257]: br1: Gained carrier
Sep 14 17:10:31 crossbeam systemd-networkd[257]: br1: DHCPv4 address 192.168.0.104/24 via 192.168.0.1
Sep 14 17:10:31 crossbeam systemd-networkd[257]: br1: Configured
@enihcam
Copy link

@enihcam enihcam commented Feb 4, 2017

Same issue here.
I would suggest that, a virtual interface should be 'configured' as long as it has an IP.

@TCB13
Copy link

@TCB13 TCB13 commented Sep 8, 2017

Same problem here. It should be able to inspect if the bridge is working the same way it does with physical connections.

@bobrik
Copy link

@bobrik bobrik commented Oct 13, 2017

Seeing the same failure, but with bonded interface. If I exclude bond slaves, command succeeds:

$ /lib/systemd/systemd-networkd-wait-online --timeout 3 --ignore eth2 --ignore eth3; echo $?
ignoring: eth3
ignoring: eth2
ignoring: lo
0

I have carrier on bond0 that is eth2+eth3.

@mrschyte
Copy link
Author

@mrschyte mrschyte commented Aug 26, 2018

This has been open for a long time now, so I've decided to write a simple workaround script which is available at https://gist.github.com/mrschyte/aa83ddad3c1d4f46308c6bb0a555af44

To set it up run systemctl edit systemd-networkd-wait-online.service and add the following lines:

[Service]                                                                                                     
ExecStart=
ExecStart=/usr/local/bin/wait-online.py brlan0 brdmz0 eth0 eth1 ppp0

Don't forget to add / change the interfaces. Cheers!

@alitvak69
Copy link

@alitvak69 alitvak69 commented Sep 10, 2018

The problem that I see not addressed is the one where you are not configuring IP addresses on purpose.
In my case Opennebula, OpenStack, nomad hosts

@jojoax
Copy link

@jojoax jojoax commented Apr 24, 2019

Also here, ugly bug, make system unusabe and boot process void:
[root@pc3 ~]# /usr/lib/systemd/systemd-networkd-wait-online --timeout=60 --ignore lo --ignore eth0 --ignore eth1 --ignore eth2 -i br0
Event loop failed: Connection timed out

@yuwata
Copy link
Member

@yuwata yuwata commented Aug 22, 2019

Please test with the recent release (v242 or newer) of systemd, and provide debugging logs if it still does not work.

@trini
Copy link
Contributor

@trini trini commented Sep 28, 2019

With v242 as found in OpenEmbedded/master currently I no longer see this problem.

@jojoax
Copy link

@jojoax jojoax commented Sep 30, 2019

Looks good, as far we see.
I presume, this can be closed.

@yuwata yuwata closed this Sep 30, 2019
@yuwata
Copy link
Member

@yuwata yuwata commented Sep 30, 2019

Thank you for testing.

@darkguy2008
Copy link

@darkguy2008 darkguy2008 commented Feb 4, 2021

I'm experiencing this same issue when configuring a simple bridge for some QEMU machines. The Journalctl says this, and boot time is around 3 minutes... wtf:

-- Reboot --
Feb 04 15:52:10 host systemd[1]: Starting Wait for Network to be Configured...
Feb 04 15:52:10 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:10 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:10 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:14 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:18 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:18 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:32 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 18:26:06 host systemd-networkd-wait-online[929]: Event loop failed: Connection timed out
Feb 04 18:26:06 host systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Feb 04 18:26:06 host systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Feb 04 18:26:06 host systemd[1]: Failed to start Wait for Network to be Configured.

Internet works though, but the amount of time to wait is ridiculous. Any help? Ubuntu 20.10 here.
My netplan file is this:

# This is the network config written by 'subiquity'
network:
  ethernets:
    enp2s0:
      dhcp4: true
    enp3s0:
      dhcp4: true
  bridges:
    br0:
      interfaces: [ enp3s0 ]
      addresses: [192.168.0.9/24]
      gateway4: 192.168.0.1
      mtu: 1500
      nameservers:
        addresses: [8.8.8.8,8.8.4.4]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
  version: 2

I only have a single ethernet, but sometimes due to PCI addresses being weird when I change video cards, it gets moved to enp2s0 or enp3s0. Right now it's enp3s0 so I don't see why it should have a delay.

@yuwata
Copy link
Member

@yuwata yuwata commented Feb 4, 2021

Please open a new issue page with debugging logs. Please see #8686 (comment) how to generate debugging logs of networkd.
Also, instead of the netplan config, please provide the generated .network and .netdev files. We do not know netplan well, unfortunately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet