v4.0.0-rc5 — issue #99: the storm caught on camera (UART stuck-IIR) — fix shipped, field confirmation pending

v4.0.0-rc5 — issue #99: the storm finally caught on camera (UART stuck-IIR) — fix shipped, field confirmation pending

rc4 did not survive the field: a unit soft-locked after 7.8 days with every rc4 detector counter at zero (pollhit=0 — the bounded RX poll never even saturated once; thanks @frtz13 for the record). The bounded poll stays — it fixed a real latent bug — but whatever kills the field units, it was something else, and the final-frame panic record could not name it.

What rc5 adds to see it: the v8 "flight recorder"

The watchdog's panic record (v8) now preserves, across the reboot, the film instead of the final photo: the last 31 seconds of per-interrupt-line activity sampled at 1 Hz, a raw UART register snapshot, interrupt-controller dispatch statistics, and the last ~2 KB of the kernel log — including the kernel's own lockup report and backtrace that headless units could never show before.

What the recorder caught — the same day it was built

Our bench unit crashed with the exact field signature, recorder running. The film is unambiguous for that crash: for the final 21 s, UART1 — the serial link to the Zigbee/Thread radio — fired ~1360 interrupts/s, 100 % of all interrupt dispatches, starving everything else (timer wheel frozen, network stack never scheduled: the ethernet "culprit" of every earlier theory was an innocent bystander). The UART registers were frozen in a contradictory state: IIR says "RX bytes waiting" (character timeout pending) while LSR says "RX FIFO empty". The driver believes LSR, drains nothing, and the level-triggered interrupt re-fires forever — ~700 µs per rotation, ~95 % of the CPU, until the 20 s softlockup reboot. This is a known 16550-clone hardware bug class, with a mainline precedent in the DesignWare UART driver.

The fix — and its built-in proof

The UART1 driver (v1.5) now detects the contradictory state and clears it with the classic remedy: one dummy FIFO read. Each occurrence logs a rate-limited warning:

rtl8196e-uart 18002100.serial: stuck RX-timeout IIR with empty RX FIFO - cleared by dummy read (iir=0xcc lsr=0x60, occurrence #1)

Honest scope — this issue has been declared fixed twice before, and the field disagreed both times. What is established: the bench crash was this UART1 storm, measured at the registers. What is not yet: whether the field units die of the same mechanism (their v7 records could not see UART state — the match is on the crash signature only), and whether the fix stops a live storm (it has never been exercised against one). Confirmation is exactly one thing: that log line appearing in /var/log/messages with unbroken uptime, and no soft-lockup recurrence over several weeks. If a soft-lockup does recur, the v8 record will decode at the next boot and settle it — please attach /userdata/panic/history.

Also in this release (the full backlog since rc4)

• Issue #109 root-cause fix (kernel): the intermittent busybox SIGSEGV/SIGILL/SIGBUS faults were a TLB-flush bug in our RLX4181 port (the invalidation sweep skipped the low TLB slots the Lexra core actually allocates). Fixed at the root; confirmed by hours of fork/exec churn that previously faulted within minutes.
• Switch-core PHY-interface watchdog (eth v2.21): detects the switch silently severing the MAC↔PHY interface and deep-resets it (~3 s recovery), mirroring the vendor's recovery.
• Userdata supervision overhaul: every long-lived daemon (s40button, linkwatch, otbr-agent, otbr-monitor) now runs under the keepalive supervisor; otbr-monitor rewritten from busybox-ash to a small C daemon.
• S70otbr flow-control fix (discussion #142, root-caused by @hlyi): uart-flow-control is a presence flag for otbr-agent — boards without RTS/CTS no longer get CRTSCTS force-enabled.
• Per-board pre-built bootloaders (discussion #140) and per-board EFR32 builds (#141/#143): boot-img/<board>/boot.bin with a devicetree-model guard on flashing; NCP/OT-RCP/Router build per-board (BOARD=), G4 NCP validated on real hardware by @hlyi — whose Sengled G4 port work drove this whole series.

Release candidate, not GA. Both kernel lines (6.18 default, 7.1) and both boards (Lidl, Sengled G4) are built.

Updating — use the full install this time

Unlike rc4, this release is not kernel-only: the userdata carries part of it (the supervision overhaul, otbr-monitor as a C daemon, the S70otbr flow-control fix, and — directly relevant to the #99 soak — the updated S26panicrec that persists the new multi-line v8 crash record to /userdata/panic/history). Update with the full install, which saves and restores your configuration (network settings, Thread network, SSH keys):

./flash_install_rtl8196e.sh <gateway-ip>

After reboot, uname -r reports …-v4.0.0-rc5 and the login banner shows the new version.

What to watch, for the #99 soakers: grep "stuck RX-timeout" /var/log/messages — one line with climbing uptime is the confirmation we have been chasing for months. And if the box still soft-locks: cat /userdata/panic/history after the reboot and attach it.