Possible tx stall with macb on pi5/cm5 with 6.18(.24)

[nbuchwitz]

Describe the bug

@lukaszraczylo posted an RFC about a possible tx stall in the macb driver on rp1. Ran into the same on a CM5 today and the symptoms match: link comes up but tx_packets stops incrementing, rx still works, ip link set eth0 down/up recovers it. Often after a link bounce.

Will test the RFC series and update this issue, then post a PR with the patches once I've dug a bit deeper. Issue is just meant as info for others with the same symptoms.

Netdev discussion: https://lore.kernel.org/netdev/cover.1777064117.git.lukasz@raczylo.com/

(6.18.22 ran fine, started seeing it after updating our downstream tree to 6.18.24. Per the upstream report it also reproduces on 6.17, so likely latent rather than a 6.18.24 regression.)

Steps to reproduce the behaviour

Run 6.18.24 on Pi5/CM5

Device (s)

Raspberry Pi CM5

System

Linux RevPi150349 6.18.24-revpi0-rpi-v8 #1+deb12+1+260427+g895e84f73e SMP PREEMPT_RT Mon Apr 27 20:46:08 aarch64 GNU/Linux

(also ran into this with stock rpi kernel)

Logs

No response

Additional context

No response

[lukaszraczylo]

As a background - there's bit more of discussions :

• TalOS: Raspberry Pi 5 — silent network death on Talos siderolabs/sbc-raspberrypi#91
• Ubuntu linux-raspi thread: https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2133877
• Cilium: [JFYI] Raspberry Pi 5 + kernel 6.17: Silent network death cilium/cilium#43198

I have prepared PR for you as well ( perfect timing as I just wrapped the siderolabs/pkgs one ) - #7340

[koeqaife]

I just wanted to actually make issue about that. My network got frozen 4 times in 2 days... Had to make watchdog because kernel doesn't even implement TX watchdog, so my all connections get frozen (including wifi on LAN, because it tries to respond through ethernet even if request went from wifi). Also PHY runs on polling instead of interrupts which is weird

[lukaszraczylo]

@koeqaife I've tried it - before digging deeper with the watchdog, unfortunately it never fired - hence my dig into the kernel itself in the search of an issue resulting in the patch. I'm currently on 2400 node-hours ( 24 nodes ) without issues post patching so - I'd say "it works" :)

[koeqaife]

The bug is really annoying cause I have no access to ssh when it dies, so I have to just reboot by raspberry pi 😭

[koeqaife]

I wrote my own watchdog that was just pinging 1.1.1.1, and if it fails 3 times it restarts whole ethernet interface

[lukaszraczylo]

@koeqaife hopefully it'll be end of the trouble after this PR is approved :) My kubernetes cluster had nodes flapping left and right for no reason, which of course neither postgres or longhorn liked so - I've had motivation and enough of caffeine to dig into it last Friday

[koeqaife]

I wasn't home when my network stopped working, so basically it wasn't working for 2 hours, it was crazy. And I thought it was crashing, but no logs in journalctl, no logs in dmesg, postgresql was working fine and everything else was working fine and even shutdown was fine

[koeqaife]

Also I run this on startup

ExecStart=/usr/bin/ethtool -K end0 tso off sg off
ExecStart=-/usr/bin/ethtool --set-eee end0 eee off

It fixes everything but loads CPU more

EDIT: Not really...

[koeqaife]

[mrdan@rasberry-pi ~]$ cat /var/log/net-watchdog.log
Wed Apr 29 12:10:02 AM CEST 2026 ping fail #1
Wed Apr 29 12:10:15 AM CEST 2026 ping fail #2
Wed Apr 29 12:10:28 AM CEST 2026 ping fail #3
Wed Apr 29 12:10:28 AM CEST 2026 resetting end0
Wed Apr 29 12:10:43 AM CEST 2026 ping fail #1
Wed Apr 29 12:10:56 AM CEST 2026 ping fail #2
Wed Apr 29 12:11:09 AM CEST 2026 ping fail #3
Wed Apr 29 12:11:09 AM CEST 2026 resetting end0
...

Omg looks like I have to do bind/unbind instead of down/up 😭
It kills driver completely...