23 wireguard peers hang systemd-networkd

[darkk]

systemd version the issue has been seen with

239-7ubuntu10.6

Used distribution

Ubuntu 18.10

Expected behaviour you didn't see

systemctl restart systemd-networkd restarts cleanly and applies network configuration

Unexpected behaviour you saw

systemctl restart systemd-networkd hangs for a while and fails to apply network configuration. If the node is rebooted, it can't apply network configuration on boot as well.

Steps to reproduce the problem

1. launch VM on Linode with Ubuntu 18.04 or 18.10
2. add wireguard PPA: apt-add-repository ppa:wireguard/wireguard
3. install wireguard: apt-get install wireguard
4. download config file generator systemd-wg that reproduces the issue
5. simulate my VPN concentrator configuration with ./systemd-wg 23

For some reason, 23 is the magic number of peers: 22 peers, 24 peers and 1500 peers work okay, but 23 peers trigger the bug.

The bug also affects systemd-237 from Ubuntu 18.04.

[yuwata]

Could you provide relevant logs?
Also, could you try v240 or current git snapshot? We have fixed several bugs around wireguard in v240.

[darkk]

@yuwata please, tell me, what logs are relevant in this case? Is there some way to enable verbose logging in systemd?

Launchpad ticket has tgz with syslog & strace from 237, but it's not clear to me if they're useful: syslog is almost silent, but strace is quite chatty and looks like some infinite loop.

[yuwata]

please, tell me, what logs are relevant in this case?

$ journalctl -u systemd-networkd.service

Is there some way to enable verbose logging in systemd?

Please set SYSTEMD_LOG_LEVEL=debug environment variable. So, please create the following and restart networkd:

# /etc/systemd/system/systemd-networkd.service.d/override.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

Note that, at least, the kernel warning in your log is fixed by 7d0b26a.

[yuwata]

Aha, indeed, the issue is reproduced.

[yuwata]

I've posted #11418. If possible, please test the PR.

[darkk]

I'm unsure what's the best way to test it for me as it's still part of init :-)
I tried to apply the patch to 239 from Ubuntu 18.10, but it failed to apply and I don't have a system that ships 240 to patch (I assume, it should apply more-or-less cleanly on top of 240).
I'll try to test it with Debian/testing, but it'll take a while.

[yuwata]

@darkk OK. No problem. This is already merged. Thanks.