v4.0.0 — issue #99 and #109 closed, the clockevent wedge, driver audits, on-chip SRAM hot paths, netwatch

The release the whole 4.0.0-rc series was working towards. The last public release was v3.10.0, so none of the rc work has reached a stable release until now — issue #99, the TLB flush bug, the driver audits, the flashing failures, all of it lands here at once.

The independent driver audits land in full — Ethernet, UART1, IRQ/GPIO, watchdog, LED-PWM, UART bridge. The LAN LED finally goes truly off, and the UART bridge gains a real client-replacement lifecycle instead of an advertised one that was dead code. The three code bodies every packet passes through — the checksum routine and the two assembly copy cores — move into on-chip SRAM; the shipped image benches at 70.7 Mbit/s TX and 92.8 Mbit/s RX, where the same suite read 68.5 / 89.5 before.

One new capability stays off until asked for: netwatch, which reboots and, above all, records a gateway isolated on a live link. The gateway also stops being addressed by a hardcoded 192.168.1.88 — addresses are resolved, from your own network if nothing else says otherwise. And the project finally says what it is: renamed rtl8196e-gateway, with five new user guides.

⚠️ Breaking change — EFR32 firmware filenames

Radio firmware files now carry their flow-control type in the name (discussion #145). Any script, bookmark or note pinning an old filename must be updated:

before	after
ncp-uart-hw-7.5.1-<baud>.gbl	ncp-uart-hw-7.5.1-<baud>-hw.gbl
ncp-uart-hw-7.5.1-115200-sengled-e39-g8c.gbl	ncp-uart-hw-7.5.1-115200-sw-sengled-e39-g8c.gbl
rcp-uart-802154-<baud>.gbl	rcp-uart-802154-<baud>-hw.gbl
ot-rcp-460800.gbl	ot-rcp-460800-hw-uartdrv.gbl
z3-router-7.5.1-115200.gbl	z3-router-7.5.1-115200-hw.gbl

The range-testing tooling was reorganised at the same time: range_test.sh and its three phase scripts become a single thread-range-test, and the Home Assistant helpers move under range-testing/gateway/optional/.

The four long investigations this release closes

Issue #99 — the gateway that stopped talking to its radio

Four releases of work, and the ending is not the one anyone expected. rc3 gave the Ethernet switch core the ability to recover itself (ETHDRV-016). rc4 made the RX poll bounded and the recovery independent of the RUNOUT condition. rc5 caught the real thing on camera — a UART1 stuck-IIR interrupt storm, where the controller latches an RX-timeout interrupt over an empty FIFO and re-asserts it forever, pinning the CPU. The dummy-read quirk that clears it is the mainline dw8250 precedent.

v4.0.0 finishes it by removing the instrument. The rc5 fix logged a rate-limited warning on every phantom it cleared, and the field data showed that warning was not free: it is emitted from the interrupt handler, and on a legacy console every record is written with interrupts disabled — about 147 characters at 38400 baud, so ~38 ms of interrupts-off time per line. On a board without RTS/CTS nothing stops the radio during that blackout, so the warning was causing the hardware overrun it was suspected of witnessing; the one field overrun coincident with a phantom landed 42 ms after the warning line. The recovery is now silent, and observability moves to /sys/module/8250_rtl819x/parameters/phantom_count.

Issue #109 — the random crashes were ours, and they were a TLB bug

Long-lived processes on this gateway took intermittent SIGSEGV, SIGILL and SIGBUS under fork/exec pressure. It was blamed on BusyBox ash for months, and it drove real design decisions — several daemons were rewritten from shell into C to move them off the exposed path. Those rewrites were mitigations. The bug was ours.

Under load a process would intermittently read the wrong physical frame for a correctly-mapped virtual address: a plain load returning a stale pointer its own page table never pointed to, corrupting a register mid-sequence. local_flush_tlb_all() began its invalidation sweep at a hardcoded TLB index of 8, inheriting the classic-R3000 assumption that Random never allocates entries 0–7. The Lexra RLX4181 does not honour that convention — it reserves entries through the Wired register, which is 0 on this core — so tlb_write_random installed translations into slots 0–7 and every bulk flush left them untouched. A stale mapping parked in a low slot survived every flush and, once the ASID generation rolled over, spuriously matched and resolved a user address to the wrong frame.

The sweep now starts at read_c0_wired(), matching the vendor arch/rlx code. Confirmed by ~53 minutes of continuous fork/exec churn with zero faults, on a load that crashed every prior build within 200–600 seconds. Both kernel lines. ash is not inherently unsafe on this platform, and nothing needs rewriting in C to be reliable.

The clockevent wedge — found after rc5

A gateway would lose its TC0 clockevent permanently: COUNT0 stuck at zero while the enable, DATA0 and interrupt registers all still read correct, and TC1 continuing at a healthy 25 kHz. The evidence points at TC0's COUNTER-mode TC0_EN: 0 → 1 re-arm edge being missed across its slow clock domain — not Ethernet traffic, not lost MMIO writes.

The remedy is do-not-toggle: TC0 is held enabled in auto-reload mode and every one-shot is armed by writing DATA0 while it runs, so the hazardous transition never happens. NO_HZ idle keeps it enabled but IRQ-masked, so tickless operation does not recreate the edge. The cause-level soak ran 26.3 hours and 54.8 million kernel reprogrammings under bidirectional saturation, with zero wedges and no missed, double or parasitic IRQ. The verified legacy path remains as the boot and fallback path.

Flashing the radio — the failures that ate people's gateways

Two bugs in the radio-flashing path, both surfaced by users (discussions #148 and #149). flash_efr32.sh would wave through a .gbl it could not actually read, and — the consequential one — the Gecko bootloader declines a same-version stage-2 upgrade in silence, after having erased the application. It installs only when the offered version is strictly greater than the running one, and there is no else branch: you get a radio with no firmware and no error message. The pre-flight now refuses unreadable images, and the version gate is handled explicitly.

Boards that wire the pin also gain hardware bootloader entry through GPIO activation, which is what makes a first radio install on a factory Sengled G4 possible at all.

New

• netwatch — the hardware watchdog only catches a stopped CPU; it is blind to a gateway whose userspace is alive but whose network path is dead. netwatch probes from the box and, after a long continuous failure with carrier still up, writes a forensic snapshot to JFFS2 and reboots. Off by default, armed with ENABLED=1 in /userdata/etc/netwatch.conf. The snapshot is the point: the human power-cycle that ends such an outage is also what destroys the evidence.
• Five user guides — getting started, using the gateway, radio options, upgrading, troubleshooting.
• Addresses are resolved, not hardcoded — lib/gwconf.sh derives them from an explicit argument, then gateway.env, then what the last install wrote, then your own LAN. A user on 192.168.0.0/24 or 10.0.0.0/8 is no longer offered an address on somebody else's subnet.
• Sengled G4 gets the full prebuilt radio set at the bauds that board actually runs, and an experimental multiprotocol RCP with a Bluetooth HCI endpoint (build_rcp_blehci.sh, discussion #146).
• WireGuard ships as a complete build recipe under 34-Userdata/wireguard/, not as a built-in driver: CONFIG_WIREGUARD=y costs 3.7 Mbit/s of TX purely through where it lands in the link, even with no tunnel in existence. Building it as a module is now possible — this release is what makes CONFIG_MODULES=y build on this port — and costs nothing measurable while unloaded. See discussion #152.

Changed

• Default hostname is rtl8196e-gw, not zigbeegw — a Thread Border Router announcing itself as a Zigbee gateway was misleading. etc/hostname is preserved across upgrades, so only a full-flash or a first install picks up the new name; revert with echo zigbeegw > /userdata/etc/hostname && reboot.
• Linux 6.18.35 → 6.18.41 on the production line, 7.1.3 → 7.1.7 on the alternate one. The production line deliberately stops at .41: benching every point release of the interval puts the last good one at .41 and the first bad one at .42, which costs about 2.2 Mbit/s of TX.
• Dropbear 2025.89 → 2026.94, OpenThread Border Router pinned to upstream v2026.07.0.
• No RSA host key — the first boot stops stalling for half a minute.
• Login banner is board- and role-neutral, and the boot log no longer goes silent on services that do nothing.

Also fixed

• /var/log/messages was timestamped in UTC while everything else was local, and every line was stamped (none) instead of the gateway's name.
• A failed DHCP lease no longer strands the gateway on the wrong subnet (issue #132).
• The bootloader address is only derived where boothold can hand it over.
• otbr-agent gains real software flow control for boards without RTS/CTS (discussion #134), and the Thread BorderAgent identifies itself per board (discussion #144).
• build_fullflash.sh / create_fullflash.sh no longer require xxd (issue #147).

Upgrading

git pull
BOARD=lidl KERNEL=6.18 ./flash_install_rtl8196e.sh -y <gateway-ip>

Your configuration is preserved — network settings, radio mode, hostname, passwords, SSH host keys, the Thread dataset, and anything you added under /userdata. Use BOARD=sengled-e39-g8c for the Sengled G4, KERNEL=7.1 for the alternate kernel line.

The full changelog, with the measurements and the reasoning behind each change, is in 3-Main-SoC-Realtek-RTL8196E/CHANGELOG.md.

Thanks

Issue #99 ran for ten weeks and 166 comments, and it was not solved from a bench. It was solved because people ran unreleased kernels on hardware they depend on and reported what happened.

@olivluca opened it on 11 May and stayed with it to the close on 24 July, running instrumented builds on a gateway in daily use and capturing serial-console output that a reset would otherwise have destroyed. @frtz13 ran the release candidates on their own gateway and posted panic record after panic record — the pc= / ra= / softirq= dumps the watchdog post-mortem exists to produce, and the raw material the fix was built from. @MaxRower followed the whole rc series through and reported from the field, including the flashing failure that became issue #115. @ish00t contributed the uptime statistics that told us when a build was actually holding — the negative evidence that is easy to forget to ask for and impossible to do without.

Thanks also to @hlyi, whose Sengled G4 work drove the second board through this cycle and whose bootloader findings are behind #148, and to @malcreatuire for the flashing failure that surfaced #149.