network: Many interfaces lead to "kernel receive buffer overrun" #14417

haslersn · 2019-12-22T04:23:47Z

systemd version the issue has been seen with
241, but the relevant code has not changed since.

Used distribution
Debian 10

Expected behaviour you didn't see
systemd-networkd.service should start successfully.

Unexpected behaviour you saw
With about 200 interfaces (100 VLANs and a bridge on each of them), when starting systemd-networkd I get the following error in the journal:

systemd-networkd[1831]: Could not enumerate addresses: No buffer space available

Running sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd gives a bit more insight:

rtnl: kernel receive buffer overrun
Could not enumerate addresses: No buffer space available

The buffer that is overrun is probably this one somewhere inside this function.

Steps to reproduce the problem

Add 100 VLAN interfaces to one of your interfaces/bridges.
Add a bridge to each of those interfaces
sudo systemctl restart systemd-networkd

The text was updated successfully, but these errors were encountered:

yuwata · 2019-12-22T14:14:32Z

Could you give us a script or .netdev files to generate the interfaces to reproduce the issue?

haslersn · 2019-12-22T14:28:28Z

Attached is a .tar file which I had to rename to .txt in order to upload it here.
systemd-network.txt

It is also weird that, using that configuration, starting systemd-networkd.service works on only one of our hypervisors and doesn't work on the other one, where I discovered the mentioned error.

Even more weird: Running sudo /lib/systemd/systemd-networkd gives the mentioned error on both of our hypervisors.

yuwata · 2019-12-22T15:35:00Z

Hmm, I cannot reproduce the issue with your config. Also, I've tested with 2000 dummy device or 1000 veth interface, but they seem to work fine. Could you test the issue with v244?

haslersn · 2019-12-22T15:45:14Z

Yes, maybe I can test with a newer version on NixOS or in a VM.

Did you only test via the service, or also by running /lib/systemd/systemd-networkd directly?

ssahani · 2019-12-22T15:55:27Z

I think this a very complex situation. Rather than we try to reproduce it if you give us that would be more useful. And it should not restricted to one distribution.

haslersn · 2019-12-22T15:59:01Z

I did not say anything against that. It was a question about your past test.

ssahani · 2019-12-22T16:43:43Z

Surely not we just trying to get a reproducer so that we can help you solve your problem . Do you figure?

poettering · 2019-12-23T13:28:19Z

i figure we should just set the socket buffer for the netlink socket ridiculously high, like udev does. Socket memory is nowadays properly accounted properly to the processes owning them, hence that's not even a bad idea.. If we do that, then the issue should not go away, but be very hard to reproduce

yuwata · 2019-12-23T14:28:12Z

@haslersn I guess a workaround for this issue is extending receive buffer size in systemd-networkd.socket, e.g. by creating the following drop-in file:

[Socket]
ReceiveBuffer=16M

yuwata · 2019-12-23T14:30:25Z

It is also weird that, using that configuration, starting systemd-networkd.service works on only one of our hypervisors and doesn't work on the other one, where I discovered the mentioned error.

I guess one host enables systemd-networkd.socket, and the other one disables it.

Even more weird: Running sudo /lib/systemd/systemd-networkd gives the mentioned error on both of our hypervisors.

In that case, it is almost equivalent that .socket is disabled. So, it is reasonable.

poettering · 2019-12-23T14:51:03Z

Bumping ReceiveBuffer= is something we definitely should do by default btw. The workaround @yuwata is not just a workaround, but part of the solution I think.

haslersn · 2019-12-23T20:07:30Z

Starting systemd-networkd.socket solved the issue! Do you still want to bump the ReceiveBuffer=, or can this be closed?

Prompted by systemd#14417.

Prompted by #14417.

andir · 2020-05-15T22:21:36Z

I don't think this has been fixed (entirely). I am running into a very similar case with 200 VLAN interfaces. This is on v245.5 on both NixOS and Fedora 33 (fedora systemctl --version gives v245.5-2.fc33).

On the first start (during system bootup) I am observing timeouts and assertions: https://gist.githubusercontent.com/andir/616f937189d176da86dbe7aa0f65217e/raw/918c9926f66c5788d9de2795dee5200331daf8c5/l (as mentioned in #15669 (comment))

Manually restarting networkd (systemctl restart systemd-networkd) after bootup leads to networkd just exiting with code 1 and the same logged error as originally posted in this issue:

router # [  652.004288] systemd-networkd[1074]: rtnl: kernel receive buffer overrun
router # [  652.005020] systemd-networkd[1074]: Could not enumerate addresses: No buffer space available

Increased buffer size further (1024MB) didn't change that behavior.

The networkd config files that I used are here: https://gist.github.com/andir/7963aa3edcd5b834224515f826f10579 You might have to change the 00-vlans.network file to match whatever hardlink there already is in your test setups.

andir · 2020-05-22T10:14:31Z

@yuwata This should probable be re-opened as the initial problem isn't gone but only worked around by activating the systemd-networkd.socket unit. I am also not entirely sure the fix introduced in #14434 actually does help at all. From my limited understanding we are running out of buffer space on the kernel side and increasing our receive buffer will probably not help there.

andir · 2020-05-28T00:59:46Z

I spent a bit more time on this and this bug is indeed partially fixed by the increase of buffer size. At least the rtnl: kernel received buffer overrun messages go away if you set an appropriate value (512MB did it for me) for ReceiveBuffer= in systemd-networkd.socket and obviously also enable that unit.

…ger than the kernel limit The commit 10ce2e0 inverts the order of SO_{RCV,SND}BUFFORCE and SO_{RCV,SND}BUF. However, setting buffer size with SO_{RCV,SND}BUF does not fail even if the requested size is larger than the kernel limit. Hence, SO_{RCV,SND}BUFFORCE will not use anymore and the buffer size is always limited by the kernel limit even if we have the priviledge to ignore the limit. This makes the buffer size is checked after configuring it with SO_{RCV,SND}BUF, and if it is still not sufficient, then try to set it with FORCE command. With this commit, if we have enough priviledge, the requested buffer size is correctly set. Hopefully fixes systemd#14417.

…ger than the kernel limit The commit 10ce2e0 inverts the order of SO_{RCV,SND}BUFFORCE and SO_{RCV,SND}BUF. However, setting buffer size with SO_{RCV,SND}BUF does not fail even if the requested size is larger than the kernel limit. Hence, SO_{RCV,SND}BUFFORCE will not use anymore and the buffer size is always limited by the kernel limit even if we have the priviledge to ignore the limit. This makes the buffer size is checked after configuring it with SO_{RCV,SND}BUF, and if it is still not sufficient, then try to set it with FORCE command. With this commit, if we have enough priviledge, the requested buffer size is correctly set. Hopefully fixes systemd#14417. (cherry picked from commit b92f350)

…ger than the kernel limit The commit 10ce2e0 inverts the order of SO_{RCV,SND}BUFFORCE and SO_{RCV,SND}BUF. However, setting buffer size with SO_{RCV,SND}BUF does not fail even if the requested size is larger than the kernel limit. Hence, SO_{RCV,SND}BUFFORCE will not use anymore and the buffer size is always limited by the kernel limit even if we have the priviledge to ignore the limit. This makes the buffer size is checked after configuring it with SO_{RCV,SND}BUF, and if it is still not sufficient, then try to set it with FORCE command. With this commit, if we have enough priviledge, the requested buffer size is correctly set. Hopefully fixes systemd#14417.

tconrado · 2020-11-03T07:26:48Z

by a mistake, instead creating the adapter and moving to its namespace, we created 195 adapters on the host + 3 others that provide the connectivity and we started to get systemd-networkd[2296549]: Could not enumerate addresses: No buffer space available; raising the property ReceiveBuffer=1024M did not help. On the other hand, using namespaces, we can confirm that the system was stable even with 2500 namespaces while all of those had at least 1 IPv6 and some with IPv4. So maybe, this is a "suitable" strategy to deal with multiple IPs (netns)

haslersn · 2020-11-03T11:14:38Z

2500 namespaces while all of those had at least 1 IPv6 and some with IPv4. So maybe, this is a "suitable" strategy

Not really, as long as systemd-networkd doesn't support configuring network namespaces.

andir · 2020-11-03T12:08:08Z

by a mistake, instead creating the adapter and moving to its namespace, we created 195 adapters on the host + 3 others that provide the connectivity and we started to get systemd-networkd[2296549]: Could not enumerate addresses: No buffer space available; raising the property ReceiveBuffer=1024M did not help. On the other hand, using namespaces, we can confirm that the system was stable even with 2500 namespaces while all of those had at least 1 IPv6 and some with IPv4. So maybe, this is a "suitable" strategy to deal with multiple IPs (netns)

@tconrado What version of systemd was this with? IIRC the latest release(s) have somewhat mitigated the issue.

…ger than the kernel limit The commit 10ce2e0 inverts the order of SO_{RCV,SND}BUFFORCE and SO_{RCV,SND}BUF. However, setting buffer size with SO_{RCV,SND}BUF does not fail even if the requested size is larger than the kernel limit. Hence, SO_{RCV,SND}BUFFORCE will not use anymore and the buffer size is always limited by the kernel limit even if we have the priviledge to ignore the limit. This makes the buffer size is checked after configuring it with SO_{RCV,SND}BUF, and if it is still not sufficient, then try to set it with FORCE command. With this commit, if we have enough priviledge, the requested buffer size is correctly set. Hopefully fixes systemd#14417. (cherry picked from commit b92f350) (cherry picked from commit 4dcae66)

DanielWeeber · 2021-02-01T10:25:36Z

Same problem here with one "real" interface and some docker interfaces. ReceiveBuffer=512M did not help

apalacheno · 2021-06-07T19:32:35Z

I just stumbled across this issue on Ubuntu 20.04 using Ubuntu's Netplan.io in version 0.102-0ubuntu1~20.04.2. Systemd is at version 245 (245.4-4ubuntu3.6).

I was unable to create more than 21 bridged VLANs. Trying to create more bridged VLANs resulted in symptoms as described above.

Enabling and activating systemd-networkd.socket seems to mitigate this issue. Initial testing suggests that this solution also survives restarts.

ImmoWetzel · 2021-08-30T10:57:12Z

I also think this should be rewritten or reopened. I had same issues on several up2date machines last week.

yuwata · 2021-08-30T11:42:38Z

@ImmoWetzel and others, if you still have the issue with recent systemd (currently, v248, v249, or newer), then please open a new issue with debugging logs of networkd and your relevant configs (.network, .netdev, and so on).

Also, please try to increase ReceiveBuffer=128M in systemd-networkd.socket.

This commit fixes the intermittent `Recv() error Out of memory [-5]` crashes of worker process by raising the netlink socket's receive buffer size to 1 MiB. A netlink socket's default receive buffer size is 208 KiB (taken from net.core.rmem_default). When incoming messages fill up the receive buffer, the recvmsg(2) call returns -ENOBUFS, which is then translated to the Out of memory error above by libnl. Both NetworkManager and systemd-networkd raise their netlink socket's buffer size to work around the same issue. More details on systemd-networkd's decision can be found at systemd/systemd#14417 and systemd/systemd#14434. Fixes cifsd-team#235. Signed-off-by: database64128 <free122448@hotmail.com>

This commit fixes the intermittent `Recv() error Out of memory [-5]` crashes of worker process by raising the netlink socket's receive buffer size to 1 MiB. A netlink socket's default receive buffer size is 208 KiB (taken from net.core.rmem_default). When incoming messages fill up the receive buffer, the recvmsg(2) call returns -ENOBUFS, which is then translated to the Out of memory error above by libnl. Both NetworkManager and systemd-networkd raise their netlink socket's buffer size to work around the same issue. More details on systemd-networkd's decision can be found at systemd/systemd#14417 and systemd/systemd#14434. Fixes cifsd-team#235. Signed-off-by: database64128 <free122448@hotmail.com> Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>

This commit fixes the intermittent `Recv() error Out of memory [-5]` crashes of worker process by raising the netlink socket's receive buffer size to 1 MiB. A netlink socket's default receive buffer size is 208 KiB (taken from net.core.rmem_default). When incoming messages fill up the receive buffer, the recvmsg(2) call returns -ENOBUFS, which is then translated to the Out of memory error above by libnl. Both NetworkManager and systemd-networkd raise their netlink socket's buffer size to work around the same issue. More details on systemd-networkd's decision can be found at systemd/systemd#14417 and systemd/systemd#14434. Fixes #235. Signed-off-by: database64128 <free122448@hotmail.com> Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>

This commit fixes the intermittent `Recv() error Out of memory [-5]` crashes of worker process by raising the netlink socket's receive buffer size to 1 MiB. A netlink socket's default receive buffer size is 208 KiB (taken from net.core.rmem_default). When incoming messages fill up the receive buffer, the recvmsg(2) call returns -ENOBUFS, which is then translated to the Out of memory error above by libnl. Both NetworkManager and systemd-networkd raise their netlink socket's buffer size to work around the same issue. More details on systemd-networkd's decision can be found at systemd/systemd#14417 and systemd/systemd#14434. Fixes cifsd-team#235. Signed-off-by: database64128 <free122448@hotmail.com> Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>

This commit fixes the intermittent `Recv() error Out of memory [-5]` crashes of worker process by raising the netlink socket's receive buffer size to 1 MiB. A netlink socket's default receive buffer size is 208 KiB (taken from net.core.rmem_default). When incoming messages fill up the receive buffer, the recvmsg(2) call returns -ENOBUFS, which is then translated to the Out of memory error above by libnl. Both NetworkManager and systemd-networkd raise their netlink socket's buffer size to work around the same issue. More details on systemd-networkd's decision can be found at systemd/systemd#14417 and systemd/systemd#14434. Fixes #235. Signed-off-by: database64128 <free122448@hotmail.com> Signed-off-by: Namjae Jeon <linkinjeon@kernel.org>

yuwata added bug 🐛 Programming errors, that need preferential fixing network sd-netlink labels Dec 22, 2019

yuwata added cant-reproduce needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer labels Dec 23, 2019

yuwata removed cant-reproduce needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer labels Dec 23, 2019

yuwata removed the bug 🐛 Programming errors, that need preferential fixing label Dec 23, 2019

yuwata added a commit to yuwata/systemd that referenced this issue Dec 26, 2019

network: bump netlink receive buffer size to 128M

51b9714

Prompted by systemd#14417.

yuwata mentioned this issue Dec 26, 2019

network: bump netlink receive buffer size to 128M #14434

Merged

yuwata added a commit to yuwata/systemd that referenced this issue Dec 28, 2019

network: bump netlink receive buffer size to 128M

f9c9c9e

Prompted by systemd#14417.

yuwata added the has-pr ✨ label Dec 29, 2019

poettering pushed a commit that referenced this issue Jan 2, 2020

network: bump netlink receive buffer size to 128M

48d0248

Prompted by #14417.

yuwata closed this as completed Jan 3, 2020

andir mentioned this issue May 15, 2020

networkd: subnet id support for ipv6 prefix delegation #15669

Merged

yuwata reopened this May 29, 2020

yuwata mentioned this issue Sep 8, 2020

Fixes for socket buffer size #16982

Merged

poettering closed this as completed in #16982 Sep 9, 2020

hoh mentioned this issue Jan 28, 2022

Superviser host system networking stopped working aleph-im/aleph-vm#119

Closed

database64128 mentioned this issue Sep 4, 2022

ksmbd-tools: set netlink socket's receive buffer size to 1 MiB cifsd-team/ksmbd-tools#277

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

network: Many interfaces lead to "kernel receive buffer overrun" #14417

network: Many interfaces lead to "kernel receive buffer overrun" #14417

haslersn commented Dec 22, 2019

yuwata commented Dec 22, 2019 •

edited

haslersn commented Dec 22, 2019 •

edited

yuwata commented Dec 22, 2019

haslersn commented Dec 22, 2019

ssahani commented Dec 22, 2019

haslersn commented Dec 22, 2019 •

edited

ssahani commented Dec 22, 2019 •

edited

poettering commented Dec 23, 2019

yuwata commented Dec 23, 2019

yuwata commented Dec 23, 2019

poettering commented Dec 23, 2019

haslersn commented Dec 23, 2019 •

edited

andir commented May 15, 2020 •

edited

andir commented May 22, 2020

andir commented May 28, 2020

tconrado commented Nov 3, 2020

haslersn commented Nov 3, 2020

andir commented Nov 3, 2020

DanielWeeber commented Feb 1, 2021

apalacheno commented Jun 7, 2021

ImmoWetzel commented Aug 30, 2021

yuwata commented Aug 30, 2021

network: Many interfaces lead to "kernel receive buffer overrun" #14417

network: Many interfaces lead to "kernel receive buffer overrun" #14417

Comments

haslersn commented Dec 22, 2019

yuwata commented Dec 22, 2019 • edited

haslersn commented Dec 22, 2019 • edited

yuwata commented Dec 22, 2019

haslersn commented Dec 22, 2019

ssahani commented Dec 22, 2019

haslersn commented Dec 22, 2019 • edited

ssahani commented Dec 22, 2019 • edited

poettering commented Dec 23, 2019

yuwata commented Dec 23, 2019

yuwata commented Dec 23, 2019

poettering commented Dec 23, 2019

haslersn commented Dec 23, 2019 • edited

andir commented May 15, 2020 • edited

andir commented May 22, 2020

andir commented May 28, 2020

tconrado commented Nov 3, 2020

haslersn commented Nov 3, 2020

andir commented Nov 3, 2020

DanielWeeber commented Feb 1, 2021

apalacheno commented Jun 7, 2021

ImmoWetzel commented Aug 30, 2021

yuwata commented Aug 30, 2021

yuwata commented Dec 22, 2019 •

edited

haslersn commented Dec 22, 2019 •

edited

haslersn commented Dec 22, 2019 •

edited

ssahani commented Dec 22, 2019 •

edited

haslersn commented Dec 23, 2019 •

edited

andir commented May 15, 2020 •

edited