atlantic stops receiving packets after a while under high load

[hoshinolina]

This issue has been happening randomly for a while now, but I only just got around to setting up an alternate network interface to investigate.

After some time with high load/IO, the 10GbE interface on a j274 stops receiving network packets. TX still works, which eventually devolves to just ARP requests (which are received by other hosts). The failure is not instant, but rather seems to take time ranging up to several minutes, during which somehow network connectivity is degraded between certain hosts.

In this instance, the failure happened at 20:02 initially and most traffic flows were interrupted, but the (remote) monitoring still worked. It somehow recovered at 20:17, before failing completely at 20:31.

This is currently only happening on a j274. I don't think I've ever seen it on other Mac Mini variants (M1 Pro, M2). I'm not sure if that's just a coincidence, though (it could be related to CPU performance relative to network IO, for example).

I have a vague feeling this might be IRQ related. If there are multiple RX queues assigned to different network flows, some IRQs or queues failing first would explain the strange failure behavior where it fails partially before it fails fully.

/proc/interrupts dump during the failure:

124:        131          0     316956     321397          0    6301482    5757779    7076594 PCI-MSIX-0000:03:00.0   0 Edge      end0
125:         62          0   22534244          0     180882     109157      19396     163910 PCI-MSIX-0000:03:00.0   1 Edge      end0
126:        178          0          0     150623    9052354    3688564    1227219    4592258 PCI-MSIX-0000:03:00.0   2 Edge      end0
127:   16596356          0          0          0          0          0          0          0 PCI-MSIX-0000:03:00.0   3 Edge      end0
128:         81          0          0     236171      26504    3987434    4310273    5243217 PCI-MSIX-0000:03:00.0   4 Edge      end0
129:         85          0          0         75       8166    7977179   19628093    7158307 PCI-MSIX-0000:03:00.0   5 Edge      end0
130:        141          0          0     740925    8342324    4133795    2094545    1913432 PCI-MSIX-0000:03:00.0   6 Edge      end0
131:         80   27867953          0          0          0          0          0          0 PCI-MSIX-0000:03:00.0   7 Edge      end0
132:          0          0          0          0          0          0          0          0 PCI-MSIX-0000:03:00.0   8 Edge      end0

In this case, IRQ 128 (MSI 4) is ticking up with TX packets, but nothing else is.

ip link shows all RX packets as dropped (dropped and mcast ticking up, as well as TX normally):

2: end0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether a4:77:f3:05:50:1b brd ff:ff:ff:ff:ff:ff
    RX:     bytes    packets errors dropped  missed   mcast
     725823928713  552326168	  0   20032	  0  449035
    TX:     bytes    packets errors dropped carrier collsns
    1818579085663 1247873976	  0	  0	  0	  0
    altname enp3s0
    altname enxa477f305501b

So the PHY is receiving packets, but they aren't making it into the kernel.

Nothing in dmesg when the problem happens. ip link set end0 down && ip link set end0 up fixes the situation.

[hoshinolina]

Somewhat related bug:

• https://bugzilla.kernel.org/show_bug.cgi?id=202651
• https://bugzilla.redhat.com/show_bug.cgi?id=1499321

Possibly unrelated but I'm pretty sure I'm seeing this too sometimes (symptom is, when NetworkManager sees the link and brings up networking, the link drops and comes back again sometimes, and the drops take several seconds).

Note to self, ethtool -S end0 has more stats.

[jannau]

I had occasional issues with atlantic on my M1 Ultra Studio which I never managed to diagnose. I guess this was around 2023. Usage of that machine shifted and don't think I it reoccurred in the last 1-2 years. Since then high load/IO would have been a rare though. Since ip link set end0 down/up recovers network connectivity I assume a cable unplug/re-plug would do as well.
My M1 Mac Mini with atlantic serves as distfile/package proxy and file server. It sees spiky high load/IO when other systems update or when it compiles gentoo updates. There were a few events of lost network connectivity (1 or 2, maybe 3 over years).
The M1 systems doesn't do anything with IRQ affinity and all end0 interrupts are handled CPU core 0.

Is Mac Mini variants (M1 Pro ... a typo for "M2 Pro"? There was no M1 Pro Mac Mini only M1 Max/Ultra Mac Studios.

[hoshinolina]

Yes sorry, M2 pro. I do not believe cable replug fixes it, I'm pretty sure I tried that before. down/up would tear down and set up the packet interface, but I think cable replug only really affects the PHY, so they are different.

I haven't done anything with affinity manually, though irqbalance does move the affinities around (Fedora default). I've turned it off and manually assigned the IRQs to CPUs 0-7 (plus another 7 for the last one since there are 9), let's see if it happens again.

[hoshinolina]

The actual load on the system isn't particularly extreme. You can see the gaps where it crashed in the past couple of days.

I'm going to artificially make it churn a bit more to see if this keeps happening or I can correlate it with the irqbalance change.

Also the crash yesterday lined up eerily precisely with an earthquake... but it wasn't particularly extreme here in Tokyo, and it certainly doesn't crash when I shake it around by hand, so I don't know if that was just a crazy coincidence or what...