Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

systemd-networkd-wait-online fails with bridged interfaces #2154

Closed
mrschyte opened this issue Dec 13, 2015 · 16 comments
Closed

systemd-networkd-wait-online fails with bridged interfaces #2154

mrschyte opened this issue Dec 13, 2015 · 16 comments
Labels
bug 🐛 Programming errors, that need preferential fixing needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer network

Comments

@mrschyte
Copy link

I have a setup where several vlans are created and all interfaces are parts of different bridges.
Unfortunately the wait online binary fails to detect that the network is up in this case.

Error message in journal:

Failed to start Wait for Network to be Configured.

I suspect that this is caused by the binary calling the "manager_all_configured" function which returns true only when the operational state for a link is degraded or routable, however in my case the state is in "carrier".

I've tried to call systemd-networkd-wait-online when the system was already up with the "-i IFACE" flag for each interface, but no matter what interface is specified, the utility waits until the timeout has been reached.
@mrschyte
Copy link
Author

The interface are setup similarly to this:

int.network:

[Match]
Name=eth*

[Network]
VLAN=lan0

lan.network:

[Match]
Name=lan*

[Network]
Bridge=brlan0

lan.netdev:

[NetDev]
Name=lan0
Kind=vlan

[VLAN]
Id=7

@juhokuu
Copy link

juhokuu commented Sep 14, 2016 •
edited

Happens here too: systemd-networkd-wait-online will not detect network on a bridge interface and times out. However, it does work as expected if I boot with ipv6.disable=1. In both cases, the logs show the bridge gaining a carrier. I'm on v231.

My /etc/systemd/network is:

enp1s0.network

[Match]
Name=enp1s0

[Network]
Bridge=br1

br1.netdev

[NetDev]
Name=br1
Kind=bridge

br1.network

[Match]
Name=br1

[Network]
DHCP=ipv4

Systemd-networkd.service and systemd-networkd-wait-online.service in the absence of ipv6.disable=1:

Sep 14 17:06:15 crossbeam systemd[1]: Starting Wait for Network to be Configured...
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:15 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:18 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:22 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:23 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:06:33 crossbeam systemd-networkd-wait-online[274]: ignoring: lo
Sep 14 17:08:15 crossbeam systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 17:08:15 crossbeam systemd[1]: Failed to start Wait for Network to be Configured.
Sep 14 17:08:15 crossbeam systemd[1]: systemd-networkd-wait-online.service: Unit entered failed state.
Sep 14 17:08:15 crossbeam systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.

Sep 14 17:06:14 crossbeam systemd[1]: Starting Network Service...
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: netdev ready
Sep 14 17:06:15 crossbeam systemd-networkd[262]: Enumeration completed
Sep 14 17:06:15 crossbeam systemd[1]: Started Network Service.
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Renamed to eth0
Sep 14 17:06:15 crossbeam systemd-networkd[262]: eth0: Renamed to enp1s0
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: IPv6 enabled for interface: Success
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: Could not append VLANs: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: br1: Could not set bridge vlan: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: IPv6 disabled for interface: Success
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Could not append VLANs: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:06:15 crossbeam systemd-networkd[262]: enp1s0: Could not set bridge vlan: Operation not permitted
Sep 14 17:06:18 crossbeam systemd-networkd[262]: enp1s0: Gained carrier
Sep 14 17:06:18 crossbeam systemd-networkd[262]: br1: Gained carrier
Sep 14 17:06:18 crossbeam systemd-networkd[262]: br1: DHCPv4 address 192.168.0.104/24 via 192.168.0.1
Sep 14 17:06:19 crossbeam systemd-networkd[262]: br1: Gained IPv6LL
Sep 14 17:06:23 crossbeam systemd-networkd[262]: br1: Configured

And when booted with ipv6.disable=1

Sep 14 17:10:28 crossbeam systemd[1]: Starting Wait for Network to be Configured...
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:28 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd-networkd-wait-online[270]: ignoring: lo
Sep 14 17:10:31 crossbeam systemd[1]: Started Wait for Network to be Configured.

Sep 14 17:10:28 crossbeam systemd[1]: Starting Network Service...
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: netdev ready
Sep 14 17:10:28 crossbeam systemd-networkd[257]: Enumeration completed
Sep 14 17:10:28 crossbeam systemd[1]: Started Network Service.
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Renamed to eth0
Sep 14 17:10:28 crossbeam systemd-networkd[257]: eth0: Renamed to enp1s0
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Cannot disable IPv6 for interface br1: No such file or directory
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Could not append VLANs: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: br1: Could not set bridge vlan: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Cannot disable IPv6 for interface enp1s0: No such file or directory
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Could not append VLANs: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Failed to assign VLANs to bridge port: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Could not set bridge vlan: Operation not permitted
Sep 14 17:10:28 crossbeam systemd-networkd[257]: enp1s0: Configured
Sep 14 17:10:31 crossbeam systemd-networkd[257]: enp1s0: Gained carrier
Sep 14 17:10:31 crossbeam systemd-networkd[257]: enp1s0: Configured
Sep 14 17:10:31 crossbeam systemd-networkd[257]: br1: Gained carrier
Sep 14 17:10:31 crossbeam systemd-networkd[257]: br1: DHCPv4 address 192.168.0.104/24 via 192.168.0.1
Sep 14 17:10:31 crossbeam systemd-networkd[257]: br1: Configured

@enihcam
Copy link

enihcam commented Feb 4, 2017

Same issue here.
I would suggest that, a virtual interface should be 'configured' as long as it has an IP.

@IxDay IxDay mentioned this issue Mar 31, 2017
Closed
@martyg77 martyg77 mentioned this issue Jul 11, 2017
Closed
1 task
@dongsupark dongsupark mentioned this issue Jul 19, 2017
Closed
@TCB13
Copy link

TCB13 commented Sep 8, 2017

Same problem here. It should be able to inspect if the bridge is working the same way it does with physical connections.

@bobrik
Copy link

bobrik commented Oct 13, 2017 •
edited

Seeing the same failure, but with bonded interface. If I exclude bond slaves, command succeeds:

$ /lib/systemd/systemd-networkd-wait-online --timeout 3 --ignore eth2 --ignore eth3; echo $?
ignoring: eth3
ignoring: eth2
ignoring: lo
0

I have carrier on bond0 that is eth2+eth3.

@netixx netixx mentioned this issue Oct 28, 2017
Closed
@mrschyte
Copy link
Author

This has been open for a long time now, so I've decided to write a simple workaround script which is available at https://gist.github.com/mrschyte/aa83ddad3c1d4f46308c6bb0a555af44

To set it up run systemctl edit systemd-networkd-wait-online.service and add the following lines:

[Service]                                                                                                     
ExecStart=
ExecStart=/usr/local/bin/wait-online.py brlan0 brdmz0 eth0 eth1 ppp0

Don't forget to add / change the interfaces. Cheers!

@alitvak69
Copy link

The problem that I see not addressed is the one where you are not configuring IP addresses on purpose.
In my case Opennebula, OpenStack, nomad hosts

@flokli flokli mentioned this issue Dec 6, 2018
Closed
@jojoax
Copy link

jojoax commented Apr 24, 2019

Also here, ugly bug, make system unusabe and boot process void:
[root@pc3 ~]# /usr/lib/systemd/systemd-networkd-wait-online --timeout=60 --ignore lo --ignore eth0 --ignore eth1 --ignore eth2 -i br0
Event loop failed: Connection timed out

@yuwata yuwata added bug 🐛 Programming errors, that need preferential fixing needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer labels Aug 22, 2019
@yuwata
Copy link
Member

yuwata commented Aug 22, 2019 •
edited

Please test with the recent release (v242 or newer) of systemd, and provide debugging logs if it still does not work.

@trini
Copy link
Contributor

trini commented Sep 28, 2019

With v242 as found in OpenEmbedded/master currently I no longer see this problem.

@jojoax
Copy link

jojoax commented Sep 30, 2019

Looks good, as far we see.
I presume, this can be closed.

@yuwata yuwata closed this as completed Sep 30, 2019
@yuwata
Copy link
Member

yuwata commented Sep 30, 2019

Thank you for testing.

@darkguy2008
Copy link

darkguy2008 commented Feb 4, 2021 •
edited

I'm experiencing this same issue when configuring a simple bridge for some QEMU machines. The Journalctl says this, and boot time is around 3 minutes... wtf:

-- Reboot --
Feb 04 15:52:10 host systemd[1]: Starting Wait for Network to be Configured...
Feb 04 15:52:10 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:10 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:10 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:14 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:18 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:18 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 15:52:32 host systemd-networkd-wait-online[929]: managing: br0
Feb 04 18:26:06 host systemd-networkd-wait-online[929]: Event loop failed: Connection timed out
Feb 04 18:26:06 host systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Feb 04 18:26:06 host systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Feb 04 18:26:06 host systemd[1]: Failed to start Wait for Network to be Configured.

Internet works though, but the amount of time to wait is ridiculous. Any help? Ubuntu 20.10 here.
My netplan file is this:

# This is the network config written by 'subiquity'
network:
  ethernets:
    enp2s0:
      dhcp4: true
    enp3s0:
      dhcp4: true
  bridges:
    br0:
      interfaces: [ enp3s0 ]
      addresses: [192.168.0.9/24]
      gateway4: 192.168.0.1
      mtu: 1500
      nameservers:
        addresses: [8.8.8.8,8.8.4.4]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
  version: 2

I only have a single ethernet, but sometimes due to PCI addresses being weird when I change video cards, it gets moved to enp2s0 or enp3s0. Right now it's enp3s0 so I don't see why it should have a delay.

@yuwata
Copy link
Member

yuwata commented Feb 4, 2021

Please open a new issue page with debugging logs. Please see #8686 (comment) how to generate debugging logs of networkd.
Also, instead of the netplan config, please provide the generated .network and .netdev files. We do not know netplan well, unfortunately.

@elyograg
Copy link

elyograg commented May 21, 2023 •
edited

I am having this problem. Ubuntu 22. Systemd version is 249.11-0ubuntu3.9 ... boot waits for two minutes for the network to come up. The network DOES come up quickly but systemd-networkd-wait-online doesn't notice.

This issue is very old. Should I make a new one?

I have a "br0" interface configured in netplan, for network redundancy and so that I can run QEMU vms in bridged mode.

image

Contents of /etc/netplan/00-installer-config.yaml:

network:
  version: 2
  ethernets:
    eno1:
      dhcp4: false
      dhcp6: false
      optional: true
    eno2:
      dhcp4: false
      dhcp6: false
      optional: true
    eno3:
      dhcp4: false
      dhcp6: false
      optional: true
    eno4:
      dhcp4: false
      dhcp6: false
      optional: true
    enp5s0:
      dhcp4: false
      dhcp6: false
      optional: true
  bridges:
    br0:
      dhcp4: false
      dhcp6: false
      interfaces: [eno1, eno2]
      addresses: [192.168.217.202/24]
      routes:
      - to: default
        via: 192.168.217.1
      - to: 192.168.57.0/24
        via: 192.168.217.10
      nameservers:
        addresses: [192.168.217.202,192.168.217.200]
        search: [int.elyograg.org,elyograg.org]
      parameters:
        stp: false
        forward-delay: 5

@yuwata
Copy link
Member

yuwata commented May 21, 2023

@elyograg Yeah, please open a new issue page. And, please include information I requested in the previous comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug 🐛 Programming errors, that need preferential fixing needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer network
Milestone
No milestone
Development

No branches or pull requests