[Bug]: NetworkManager-wait-online.service not enabled when NM used - means services that require working network start prematurely and fail

[daniel-ayers]

What happened?

I have observed this issue running Armbian v25.2.1 for Tinker Board running Armbian Linux 6.12.16-current-rockchip (Ubuntu noble server image).

This image uses NetworkManager for network config. When NM is used the NetworkManager-wait-online.service must be enabled so that network-online.target is not reached until NetworkManager has brought up all the interfaces. Service units that require the network to be up before starting rely on network-online.target working correctly otherwise they will likely fail on boot (which is the problem I observed).

See the systemd documentation, in particular the first two FAQs.

I encountered this problem when trying to set up a dhcp server, using the isc-dhcpd-server package. I correctly configured the dhcp server, enabled its unit and restarted it. It worked. However when I rebooted the Pi the dhcp server failed to start claiming "there was no subnet definition for end0". I figured out that this was because at the time isc-dhcp-server was starting the network had not yet come up, so dhcpd could not obtain the IP address & netmask for the interface to match with the subnet definition in the config file.

Reading the systemd documentation (linked above) I ran the suggested command to see if the wait services were enabled:

$ systemctl is-enabled NetworkManager-wait-online.service systemd-networkd-wait-online.service
disabled
disabled

This reveals that NetworkManager-wait-online.service is disabled, when it should be enabled when NM is used to configure the network.

I enabled NetworkManager-wait-online.service and then rebooted. The dhcp server started correctly.

How to reproduce?

In a running Armbian system execute the following command (from the systemd documentation) and verify that the correct wait unit (depending on how network configuration is done in the image) is enabled:

$ systemctl is-enabled NetworkManager-wait-online.service systemd-networkd-wait-online.service
enabled
disabled

Branch

main (main development branch)

On which host OS are you running the build script and observing this problem?

Other

Are you building on Windows WSL2?

• Yes, my Ubuntu/Debian/OtherOS is running on WSL2

Relevant log URL

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Hey @daniel-ayers !

Looks like it's your first time interacting with Armbian here on GitHub.

Welcome and thank you for taking the time to report an issue ❤️.

Don't forget to star ⭐ the repo.

[github-actions]

Jira ticket: AR-2622

[rpardini]

Hey @daniel-ayers, having fought the -wait-online stuff for years, while I see your point clearly, enabling it by default will cause hangs during boot for people without an ethernet connection. Specially on the first boot, we don't wanna hang on boot. Also there's many folks on wifi only.

I do think there's a good opportunity to (ask to?) enable it during the first-run "wizard". wdyt?

[igorpecovnik]

Probably somewhere after this?
https://github.com/armbian/build/blob/main/packages/bsp/common/usr/lib/armbian/armbian-firstlogin#L793

conditions:

• connected wired network
• network manager stack

[igorpecovnik]

Should be fixed for the future.

[torte71]

As @rpardini predicted, that change means, that boards with only WiFi hang after entering the first root password. One has to wait about 90secs for the *-wait-online.service to time out, before being able to continue passwort entry.

Maybe moving the "systemctl start" stuff before the first password query and printing a warning, that network-init may take 1-2 minutes, is an option? So the user is at least prepared for the delay.

Edit: @daniel-ayers Are you trying to use the same network interface as a dhcp-client and a dhcp-server at the same time? So that the interface gets its IP from exactly the dhcp-server, that shall run on it?
That's quite a strange setup IMO - from all I've seen e.g. from OpenWRT, the interface, that acts as a dhcp-server, has to be set up with a static IP, otherwise dhcpd can't be bound to it.
In that case, I'd propose to set up the interface with a static address (if you've set up a dhcpd config, then you know the allowed address range) and revert 7f32a58

[ig3]

The template for NetworkManager.service has had NetworkManager-wait-online as an Also in its Install section since March 2018:

[Install]
WantedBy=multi-user.target
Also=NetworkManager-dispatcher.service

# We want to enable NetworkManager-wait-online.service whenever this service
# is enabled. NetworkManager-wait-online.service has
# WantedBy=network-online.target, so enabling it only has an effect if
# network-online.target itself is enabled or pulled in by some other unit.
Also=NetworkManager-wait-online.service

Given this, if NetworkManager is enabled then NetworkManager-wait-online should be enabled. If an image has NetworkManager enabled but NetworkManager-wait-online not enabled, something has gone wrong with the setup of the image. While enabling NetworkManager-wait-online in armbian-firstlogin will fix the error, it would be better to fix the image build so it doesn't have this misconfiguration.