Skip to content

v4.0.0-rc4 — issue #99 root cause fixed (bounded RX poll)

Pre-release
Pre-release

Choose a tag to compare

@jnilo1 jnilo1 released this 30 Jun 12:15
· 5 commits to main since this release

v4.0.0-rc4 — issue #99: the real root cause, fixed

Two field units still soft-locked on rc3 (thanks @frtz13 and @olivluca for the
captures). Their watchdog records revised the diagnosis — and it was not the RUNOUT
desync that rc1–rc3 targeted (the field signature is iisr=0x320e: RX_DONE set, RUNOUT
clear
). The real cause is an unbounded RX poll loop: the NAPI poll bounded itself by
packets delivered, while every drop/error path re-armed and advanced the ring cursor
without counting — so under a flood of droppable descriptors the poll never returned,
pinning the CPU in softirq (100 % softirq / 0 % hardirq) until the watchdog rebooted. rc3's
switch-core recovery was structurally blind to it, because it gates on RUNOUT.

The fix

  • Bounded RX poll. The receive loop now counts descriptors processed (every
    iteration), not packets delivered, and terminates within one NAPI budget by construction —
    a droppable-descriptor flood can no longer spin it forever. This restores the iteration
    bound the 2012 vendor driver always had and the from-scratch rewrite had dropped.
  • RUNOUT-independent stall detector. A budget-saturating poll that delivers nothing, for
    32 consecutive polls (tunable), escalates to the switch-core deep reset — independent of the
    RUNOUT status the earlier recovery relied on (and which the field showed clear).

Better diagnostics if it ever recurs

  • Watchdog record v7 is re-focused on this cause and adds an A-vs-B discriminator
    (intrinsic switch desync vs a real runt flood) plus the switch's own RX descriptor
    pointers, cross-checked against the driver.
  • The same fingerprint is now logged live at every recovery action, so a self-heal that
    never reboots still records what triggered it.

Bench-validated under fault injection: the bounded poll absorbs a sustained forced-drop flood
with the box staying responsive (no soft-lockup), the detector escalates to a deep reset, and
RX returns to line rate afterward. No datapath regression (TCP RX ~94, TX ~70 Mbit/s).

Release candidate, not GA. Both kernel lines (6.18 default, 7.1) and both boards
(Lidl, Sengled G4) are built.

Updating

Kernel-only change — reflash the kernel partition (config and userdata are untouched):

./3-Main-SoC-Realtek-RTL8196E/flash_remote.sh -y kernel <gateway-ip>

After reboot, uname -r reports …-v4.0.0-rc4. If you hit a soft-lockup anyway, the boot
afterward prints a previous boot ended in panic: … eth=[…] line — please attach it.