v4.0.0 — issue #99 and #109 closed, the clockevent wedge, driver audits, on-chip SRAM hot paths, netwatch

The release the whole 4.0.0-rc series was working towards. The last public release was v3.10.0, so none of the rc work has reached a stable release until now — issue #99, the TLB flush bug, the driver audits, the flashing failures, all of it lands here at once.

The independent driver audits land in full — Ethernet, UART1, IRQ/GPIO, watchdog, LED-PWM, UART bridge. The LAN LED finally goes truly off, and the UART bridge gains a real client-replacement lifecycle instead of an advertised one that was dead code. The three code bodies every packet passes through — the checksum routine and the two assembly copy cores — move into on-chip SRAM; the shipped image benches at 70.7 Mbit/s TX and 92.8 Mbit/s RX, where the same suite read 68.5 / 89.5 before.

One new capability stays off until asked for: netwatch, which reboots and, above all, records a gateway isolated on a live link. The gateway also stops being addressed by a hardcoded 192.168.1.88 — addresses are resolved, from your own network if nothing else says otherwise. And the project finally says what it is: renamed rtl8196e-gateway, with five new user guides.

⚠️ Breaking change — EFR32 firmware filenames

Radio firmware files now carry their flow-control type in the name (discussion #145). Any script, bookmark or note pinning an old filename must be updated:

before	after
ncp-uart-hw-7.5.1-<baud>.gbl	ncp-uart-hw-7.5.1-<baud>-hw.gbl
ncp-uart-hw-7.5.1-115200-sengled-e39-g8c.gbl	ncp-uart-hw-7.5.1-115200-sw-sengled-e39-g8c.gbl
rcp-uart-802154-<baud>.gbl	rcp-uart-802154-<baud>-hw.gbl
ot-rcp-460800.gbl	ot-rcp-460800-hw-uartdrv.gbl
z3-router-7.5.1-115200.gbl	z3-router-7.5.1-115200-hw.gbl

The range-testing tooling was reorganised at the same time: range_test.sh and its three phase scripts become a single thread-range-test, and the Home Assistant helpers move under range-testing/gateway/optional/.

The four long investigations this release closes

Issue #99 — the gateway that stopped talking to its radio

Four releases of work, and the ending is not the one anyone expected. rc3 gave the Ethernet switch core the ability to recover itself (ETHDRV-016). rc4 made the RX poll bounded and the recovery independent of the RUNOUT condition. rc5 caught the real thing on camera — a UART1 stuck-IIR interrupt storm, where the controller latches an RX-timeout interrupt over an empty FIFO and re-asserts it forever, pinning the CPU. The dummy-read quirk that clears it is the mainline dw8250 precedent.

v4.0.0 finishes it by removing the instrument. The rc5 fix logged a rate-limited warning on every phantom it cleared, and the field data showed that warning was not free: it is emitted from the interrupt handler, and on a legacy console every record is written with interrupts disabled — about 147 characters at 38400 baud, so ~38 ms of interrupts-off time per line. On a board without RTS/CTS nothing stops the radio during that blackout, so the warning was causing the hardware overrun it was suspected of witnessing; the one field overrun coincident with a phantom landed 42 ms after the warning line. The recovery is now silent, and observability moves to /sys/module/8250_rtl819x/parameters/phantom_count.

Issue #109 — the random crashes were ours, and they were a TLB bug

Long-lived processes on this gateway took intermittent SIGSEGV, SIGILL and SIGBUS under fork/exec pressure. It was blamed on BusyBox ash for months, and it drove real design decisions — several daemons were rewritten from shell into C to move them off the exposed path. Those rewrites were mitigations. The bug was ours.

Under load a process would intermittently read the wrong physical frame for a correctly-mapped virtual address: a plain load returning a stale pointer its own page table never pointed to, corrupting a register mid-sequence. local_flush_tlb_all() began its invalidation sweep at a hardcoded TLB index of 8, inheriting the classic-R3000 assumption that Random never allocates entries 0–7. The Lexra RLX4181 does not honour that convention — it reserves entries through the Wired register, which is 0 on this core — so tlb_write_random installed translations into slots 0–7 and every bulk flush left them untouched. A stale mapping parked in a low slot survived every flush and, once the ASID generation rolled over, spuriously matched and resolved a user address to the wrong frame.

The sweep now starts at read_c0_wired(), matching the vendor arch/rlx code. Confirmed by ~53 minutes of continuous fork/exec churn with zero faults, on a load that crashed every prior build within 200–600 seconds. Both kernel lines. ash is not inherently unsafe on this platform, and nothing needs rewriting in C to be reliable.

The clockevent wedge — found after rc5

A gateway would lose its TC0 clockevent permanently: COUNT0 stuck at zero while the enable, DATA0 and interrupt registers all still read correct, and TC1 continuing at a healthy 25 kHz. The evidence points at TC0's COUNTER-mode TC0_EN: 0 → 1 re-arm edge being missed across its slow clock domain — not Ethernet traffic, not lost MMIO writes.

The remedy is do-not-toggle: TC0 is held enabled in auto-reload mode and every one-shot is armed by writing DATA0 while it runs, so the hazardous transition never happens. NO_HZ idle keeps it enabled but IRQ-masked, so tickless operation does not recreate the edge. The cause-level soak ran 26.3 hours and 54.8 million kernel reprogrammings under bidirectional saturation, with zero wedges and no missed, double or parasitic IRQ. The verified legacy path remains as the boot and fallback path.

Flashing the radio — the failures that ate people's gateways

Two bugs in the radio-flashing path, both surfaced by users (discussions #148 and #149). flash_efr32.sh would wave through a .gbl it could not actually read, and — the consequential one — the Gecko bootloader declines a same-version stage-2 upgrade in silence, after having erased the application. It installs only when the offered version is strictly greater than the running one, and there is no else branch: you get a radio with no firmware and no error message. The pre-flight now refuses unreadable images, and the version gate is handled explicitly.

Boards that wire the pin also gain hardware bootloader entry through GPIO activation, which is what makes a first radio install on a factory Sengled G4 possible at all.

New

• netwatch — the hardware watchdog only catches a stopped CPU; it is blind to a gateway whose userspace is alive but whose network path is dead. netwatch probes from the box and, after a long continuous failure with carrier still up, writes a forensic snapshot to JFFS2 and reboots. Off by default, armed with ENABLED=1 in /userdata/etc/netwatch.conf. The snapshot is the point: the human power-cycle that ends such an outage is also what destroys the evidence.
• Five user guides — getting started, using the gateway, radio options, upgrading, troubleshooting.
• Addresses are resolved, not hardcoded — lib/gwconf.sh derives them from an explicit argument, then gateway.env, then what the last install wrote, then your own LAN. A user on 192.168.0.0/24 or 10.0.0.0/8 is no longer offered an address on somebody else's subnet.
• Sengled G4 gets the full prebuilt radio set at the bauds that board actually runs, and an experimental multiprotocol RCP with a Bluetooth HCI endpoint (build_rcp_blehci.sh, discussion #146).
• WireGuard ships as a complete build recipe under 34-Userdata/wireguard/, not as a built-in driver: CONFIG_WIREGUARD=y costs 3.7 Mbit/s of TX purely through where it lands in the link, even with no tunnel in existence. Building it as a module is now possible — this release is what makes CONFIG_MODULES=y build on this port — and costs nothing measurable while unloaded. See discussion #152.

Changed

• Default hostname is rtl8196e-gw, not zigbeegw — a Thread Border Router announcing itself as a Zigbee gateway was misleading. etc/hostname is preserved across upgrades, so only a full-flash or a first install picks up the new name; revert with echo zigbeegw > /userdata/etc/hostname && reboot.
• Linux 6.18.35 → 6.18.41 on the production line, 7.1.3 → 7.1.7 on the alternate one. The production line deliberately stops at .41: benching every point release of the interval puts the last good one at .41 and the first bad one at .42, which costs about 2.2 Mbit/s of TX.
• Dropbear 2025.89 → 2026.94, OpenThread Border Router pinned to upstream v2026.07.0.
• No RSA host key — the first boot stops stalling for half a minute.
• Login banner is board- and role-neutral, and the boot log no longer goes silent on services that do nothing.

Also fixed

• /var/log/messages was timestamped in UTC while everything else was local, and every line was stamped (none) instead of the gateway's name.
• A failed DHCP lease no longer strands the gateway on the wrong subnet (issue #132).
• The bootloader address is only derived where boothold can hand it over.
• otbr-agent gains real software flow control for boards without RTS/CTS (discussion #134), and the Thread BorderAgent identifies itself per board (discussion #144).
• build_fullflash.sh / create_fullflash.sh no longer require xxd (issue #147).

Upgrading

git pull
BOARD=lidl KERNEL=6.18 ./flash_install_rtl8196e.sh -y <gateway-ip>

Your configuration is preserved — network settings, radio mode, hostname, passwords, SSH host keys, the Thread dataset, and anything you added under /userdata. Use BOARD=sengled-e39-g8c for the Sengled G4, KERNEL=7.1 for the alternate kernel line.

The full changelog, with the measurements and the reasoning behind each change, is in [`3-Main-SoC-Realte...

v4.0.0-rc5 — issue #99: the storm caught on camera (UART stuck-IIR) — fix shipped, field confirmation pending

v4.0.0-rc5 — issue #99: the storm finally caught on camera (UART stuck-IIR) — fix shipped, field confirmation pending

rc4 did not survive the field: a unit soft-locked after 7.8 days with every rc4 detector counter at zero (pollhit=0 — the bounded RX poll never even saturated once; thanks @frtz13 for the record). The bounded poll stays — it fixed a real latent bug — but whatever kills the field units, it was something else, and the final-frame panic record could not name it.

What rc5 adds to see it: the v8 "flight recorder"

The watchdog's panic record (v8) now preserves, across the reboot, the film instead of the final photo: the last 31 seconds of per-interrupt-line activity sampled at 1 Hz, a raw UART register snapshot, interrupt-controller dispatch statistics, and the last ~2 KB of the kernel log — including the kernel's own lockup report and backtrace that headless units could never show before.

What the recorder caught — the same day it was built

Our bench unit crashed with the exact field signature, recorder running. The film is unambiguous for that crash: for the final 21 s, UART1 — the serial link to the Zigbee/Thread radio — fired ~1360 interrupts/s, 100 % of all interrupt dispatches, starving everything else (timer wheel frozen, network stack never scheduled: the ethernet "culprit" of every earlier theory was an innocent bystander). The UART registers were frozen in a contradictory state: IIR says "RX bytes waiting" (character timeout pending) while LSR says "RX FIFO empty". The driver believes LSR, drains nothing, and the level-triggered interrupt re-fires forever — ~700 µs per rotation, ~95 % of the CPU, until the 20 s softlockup reboot. This is a known 16550-clone hardware bug class, with a mainline precedent in the DesignWare UART driver.

The fix — and its built-in proof

The UART1 driver (v1.5) now detects the contradictory state and clears it with the classic remedy: one dummy FIFO read. Each occurrence logs a rate-limited warning:

rtl8196e-uart 18002100.serial: stuck RX-timeout IIR with empty RX FIFO - cleared by dummy read (iir=0xcc lsr=0x60, occurrence #1)

Honest scope — this issue has been declared fixed twice before, and the field disagreed both times. What is established: the bench crash was this UART1 storm, measured at the registers. What is not yet: whether the field units die of the same mechanism (their v7 records could not see UART state — the match is on the crash signature only), and whether the fix stops a live storm (it has never been exercised against one). Confirmation is exactly one thing: that log line appearing in /var/log/messages with unbroken uptime, and no soft-lockup recurrence over several weeks. If a soft-lockup does recur, the v8 record will decode at the next boot and settle it — please attach /userdata/panic/history.

Also in this release (the full backlog since rc4)

• Issue #109 root-cause fix (kernel): the intermittent busybox SIGSEGV/SIGILL/SIGBUS faults were a TLB-flush bug in our RLX4181 port (the invalidation sweep skipped the low TLB slots the Lexra core actually allocates). Fixed at the root; confirmed by hours of fork/exec churn that previously faulted within minutes.
• Switch-core PHY-interface watchdog (eth v2.21): detects the switch silently severing the MAC↔PHY interface and deep-resets it (~3 s recovery), mirroring the vendor's recovery.
• Userdata supervision overhaul: every long-lived daemon (s40button, linkwatch, otbr-agent, otbr-monitor) now runs under the keepalive supervisor; otbr-monitor rewritten from busybox-ash to a small C daemon.
• S70otbr flow-control fix (discussion #142, root-caused by @hlyi): uart-flow-control is a presence flag for otbr-agent — boards without RTS/CTS no longer get CRTSCTS force-enabled.
• Per-board pre-built bootloaders (discussion #140) and per-board EFR32 builds (#141/#143): boot-img/<board>/boot.bin with a devicetree-model guard on flashing; NCP/OT-RCP/Router build per-board (BOARD=), G4 NCP validated on real hardware by @hlyi — whose Sengled G4 port work drove this whole series.

Release candidate, not GA. Both kernel lines (6.18 default, 7.1) and both boards (Lidl, Sengled G4) are built.

Updating — use the full install this time

Unlike rc4, this release is not kernel-only: the userdata carries part of it (the supervision overhaul, otbr-monitor as a C daemon, the S70otbr flow-control fix, and — directly relevant to the #99 soak — the updated S26panicrec that persists the new multi-line v8 crash record to /userdata/panic/history). Update with the full install, which saves and restores your configuration (network settings, Thread network, SSH keys):

./flash_install_rtl8196e.sh <gateway-ip>

After reboot, uname -r reports …-v4.0.0-rc5 and the login banner shows the new version.

What to watch, for the #99 soakers: grep "stuck RX-timeout" /var/log/messages — one line with climbing uptime is the confirmation we have been chasing for months. And if the box still soft-locks: cat /userdata/panic/history after the reboot and attach it.

v4.0.0-rc4 — issue #99 root cause fixed (bounded RX poll)

v4.0.0-rc4 — issue #99: the real root cause, fixed

Two field units still soft-locked on rc3 (thanks @frtz13 and @olivluca for the
captures). Their watchdog records revised the diagnosis — and it was not the RUNOUT
desync that rc1–rc3 targeted (the field signature is iisr=0x320e: RX_DONE set, RUNOUT
clear). The real cause is an unbounded RX poll loop: the NAPI poll bounded itself by
packets delivered, while every drop/error path re-armed and advanced the ring cursor
without counting — so under a flood of droppable descriptors the poll never returned,
pinning the CPU in softirq (100 % softirq / 0 % hardirq) until the watchdog rebooted. rc3's
switch-core recovery was structurally blind to it, because it gates on RUNOUT.

The fix

• Bounded RX poll. The receive loop now counts descriptors processed (every
  iteration), not packets delivered, and terminates within one NAPI budget by construction —
  a droppable-descriptor flood can no longer spin it forever. This restores the iteration
  bound the 2012 vendor driver always had and the from-scratch rewrite had dropped.
• RUNOUT-independent stall detector. A budget-saturating poll that delivers nothing, for
  32 consecutive polls (tunable), escalates to the switch-core deep reset — independent of the
  RUNOUT status the earlier recovery relied on (and which the field showed clear).

Better diagnostics if it ever recurs

• Watchdog record v7 is re-focused on this cause and adds an A-vs-B discriminator
  (intrinsic switch desync vs a real runt flood) plus the switch's own RX descriptor
  pointers, cross-checked against the driver.
• The same fingerprint is now logged live at every recovery action, so a self-heal that
  never reboots still records what triggered it.

Bench-validated under fault injection: the bounded poll absorbs a sustained forced-drop flood
with the box staying responsive (no soft-lockup), the detector escalates to a deep reset, and
RX returns to line rate afterward. No datapath regression (TCP RX ~94, TX ~70 Mbit/s).

Release candidate, not GA. Both kernel lines (6.18 default, 7.1) and both boards
(Lidl, Sengled G4) are built.

Updating

Kernel-only change — reflash the kernel partition (config and userdata are untouched):

./3-Main-SoC-Realtek-RTL8196E/flash_remote.sh -y kernel <gateway-ip>

After reboot, uname -r reports …-v4.0.0-rc4. If you hit a soft-lockup anyway, the boot
afterward prints a previous boot ended in panic: … eth=[…] line — please attach it.

v4.0.0-rc3 — issue #99 switch-core self-recovery

v4.0.0-rc3 — issue #99 switch-core self-recovery

Release candidate, not GA. Supersedes v4.0.0-rc2.

Headline — issue #99 switch-core self-recovery

The Ethernet driver (rtl8196e-eth v2.18) now detects a wedged switch core and resets it
at runtime instead of riding the soft-lockup to a watchdog reboot — restoring the deep
recovery the original Realtek SDK shipped and our from-scratch rewrite had dropped:

• Deep switch-core reset. When the cheap in-poll ring resync repeatedly fails to clear a
  RUNOUT storm, a worker performs the vendor FullAndSemiReset (switch-core clock cycle +
  full reset) and re-brings-up the datapath — recovering a live unit with no reboot.
• TX-done hang watchdog. A 1-second check catches a TX path the switch stops draining (a
  stall the RX detectors can't see) and triggers the same reset.
• Clear-RUNOUT-on-progress. Transient runouts now self-clear, matching the stock receive
  path.

You'll see it in the log if it engages, e.g. eth0: switch core reset done (#1). The rc2
watchdog post-mortem record (v6) is kept, so a recurrence is either healed visibly
or fully captured in /userdata/panic/history. No throughput regression (TCP RX ~94, TX
~69 Mbit/s).

Also in rc3 (rolls up the rc2 development cycle)

• Dual kernel × two boards — four images. BOARD (lidl / sengled-e39-g8c) ×
  KERNEL (6.18 / 7.1) selection; both kernel lines now carry identical drivers.
• Bootloader V2.9 — closes the post-flash boot loop on a kernel flash and a 16 MiB
  full-flash.
• Sengled G4 NCP radio firmware validated (#130); flash_efr32.sh gains a BOARD=
  selector + board-match guard.
• Flash robustness — user-added files under /userdata survive a userdata reflash.
• Quiet boot, harmonized init-script output, and a portable per-release iperf3 bench.

See 3-Main-SoC-Realtek-RTL8196E/CHANGELOG.md for the full per-component detail.

Install

rc3 uses the per-board image layout, so this is a full reflash (not kernel-only). Pull
the branch and flash:

git fetch origin && git checkout v4.0.0-rc3
# Lidl board, 6.18 kernel (defaults):
./3-Main-SoC-Realtek-RTL8196E/flash_remote.sh -y kernel <gw-ip>
# (or a full install / other BOARD/KERNEL — see 32-Kernel/CLAUDE-less docs)

Reporting

If you hit a lockup anyway, please attach the boot-after line from
dmesg | grep "previous boot ended in panic" (or /userdata/panic/history) and any
switch core reset done / TX-done descriptor … stuck / RX runout not clearing lines
from dmesg.

v3.10.0 — Device-tree board description, bridge software flow control, ntpd daemon, flash_install reliability

Kernel + userdata + host-tooling release. No bootloader or rootfs change — existing installs upgrade with flash_install_rtl8196e.sh (your config is preserved); the flashing fixes arrive with a plain git pull. Issue #99 soak boxes: stay on v3.8.3 (frozen baseline).

Device tree now describes the board wiring (#122/#123/#124)

Board facts move out of code and into the DTS, so porting to an RTL8196E twin (e.g. Sengled G4, #119) no longer means patching drivers:

• gpio-line-names names the SoC GPIO lines (reset-button, status-led, efr32-nrst on the Lidl board).
• New optional /radio-bridge node seeds the UART-bridge defaults at boot: nrst-gpios and flow-control = "hw" | "sw" | "none" (module parameters and runtime sysfs still override).
• One dtb per board: BOARD=<board> ./build_kernel.sh (default lidl); the add-a-board recipe is documented in the dts Makefile and Kconfig.

uart-bridge v1.2 — software flow control (#123)

flow_control is now tri-state: 0/none, 1/hw (RTS/CTS, default), 2/sw (XON/XOFF). In sw mode the bridge strips bare XON/XOFF from the radio stream and gates the TCP→UART direction (pause on XOFF, resume on XON, bounded 1 s fail-open); remote hosts see a clean ASH stream. New xoff/xon/tx_pause_timeouts counters in parameters/stats. Sysfs readback stays numeric — existing tooling unaffected. During an EFR32 flash, flow control must be 0 in all modes.

s40button v2 — button via the GPIO cdev (#122)

The front-panel button daemon no longer pokes GPIO registers through /dev/mem: it claims its line through /dev/gpiochip0 (uAPI v2, no libgpiod), found by DTS name (reset-button), with fallback to line 9 and a -p <line> override. Claiming through the kernel also applies the pad-mux for pads B2–B6 — the bug that kept a B6-wired button dead on other boards. Press logic unchanged (5 s long-press → EFR32 recovery).

S20time — ntpd runs as a daemon (#125)

The one-shot ntpd -q behind a ping gate is gone: if the gateway booted without internet, the clock stayed at the 1970 epoch forever (no RTC). ntpd now runs as a daemon — it retries until internet appears, steps the clock, and keeps disciplining it (no more drift on month-long uptimes). Measured cost: ~100–200 kB private RSS, ~0.03 % CPU in its busiest phase, one 48-byte UDP exchange per poll interval.

flash_install — no more false "manual flash required" (#115)

Bootloader detection after boothold now requires the bootloader's TFTP server to ACK a write probe (a shutting-down Linux answering ARP could fool the old check), the boothold path no longer runs the Tuya/ICMP classification at all, and the auto-flash confirmation is dual-channel: the bootloader's UDP:9999 notification or the gateway coming back up on SSH — a host firewall can no longer turn a successful flash into a scary message. Failure paths now state explicitly that nothing was written, and that pre-V2.7 bootloaders ignore --boot-ip (they always come up at 192.168.1.6).

Documentation

READMEs synced with the above: uart-bridge pages (flow control, DT seeding, stats), kernel build page (BOARD=), userdata page, EFR32 flashing page.

Upgrade: ./flash_install_rtl8196e.sh <gateway-ip> — full image, config preserved. Kernel and userdata both changed in this release.

v3.9.0 — Linux 6.18.35, release-stamped uname, single-GPIO EFR32 nRST

Kernel-only release: Linux 6.18.35 rebase, release-stamped uname -r, and the EFR32 nRST pulse rework from discussion #121. Bootloader, rootfs and userdata payloads are unchanged — existing installs can upgrade with a kernel-only reflash.

Linux 6.18.24 → 6.18.35

The SysRq dispatch series we submitted upstream (serial core guard + 8250/8250_dw IRQ-path dispatch) landed in 6.18.35, so the three provisional patches are gone from patches-6.18/. Two context-drifted patches were refreshed to offset 0; the remaining 51 apply unchanged. build_kernel.sh now aborts loudly on a rejected or fuzzed hunk instead of swallowing it, and iperf3 confirms no throughput regression on the rebase (RX 93.8 / TX 69.9 Mbit/s).

Kernel self-identifies its firmware release (issue #120)

uname -r now reads 6.18.35-rtl8196e-v3.9.0: the firmware release is stamped into the kernel localversion at build time. After a kernel-only reflash the running kernel names its release even though /userdata/etc/version still describes the (unchanged) userdata partition — the mixed state that was confusing in #120 is now visible and accurate.

UART↔TCP bridge v1.1 — nRST pulse reworked to a single open-drain GPIO (discussion #121)

nrst_pulse used to flip three PIN_MUX_SEL_2 fields copied from the chip's reset-default value. Per-pad bench isolation showed the EFR32 nRST is wired to a single pad (B4 = gpio-rtl819x line 12); the other two fields just re-routed unrelated pads during every pulse.

• The pulse now claims that one line through the gpiod consumer API with open-drain semantics: assert drives the pad low, release floats it back to input and the EFR32's internal RESETn pull-up releases the chip (the line is never driven high).
• New runtime-writable nrst_gpio parameter (default 12) so ports to RTL8196E twins with different nRST routing select their line without patching the driver.
• Sysfs interface unchanged (echo 1 > .../nrst_pulse); recover_efr32 and flash_efr32.sh work as before.

Upgrading

Full install (recommended for new users):

./flash_install_rtl8196e.sh <gateway-ip>

Existing v3.8.x installs only need the kernel partition — config, Thread dataset and SSH keys untouched:

./3-Main-SoC-Realtek-RTL8196E/flash_remote.sh -y kernel <gateway-ip>

After upgrading, uname -r should read 6.18.35-rtl8196e-v3.9.0.

Issue #99 soak boxes: please stay on v3.8.3 for now. The panic-capture instrumentation (watchdog v1.4 / record v3) is identical in both releases, so upgrading gains nothing for the investigation — while the 6.18.35 rebase changes the kernel base under test (notably two upstream hrtimer commits in 6.18.33). Keeping the soak fleet on the frozen v3.8.3 baseline keeps the next capture comparable; we'll give the all-clear in #99 once the first capture has been analyzed.

v3.8.3 — status-LED dimming fix + watchdog post-mortem v3 (#99, #120)

Kernel-only release: status-LED dimming fix and watchdog post-mortem v3 (issue #99 instrumentation). Bootloader, rootfs and userdata payloads are unchanged — existing installs can upgrade with a kernel-only reflash.

Status LED flickered at 31 Hz instead of dimming (issue #120)

Since the Linux 4.8 timer-wheel rework the kernel rounds every timer expiry up by one jiffy so a timer can never fire early: the leds-gpio-pwm per-jiffy re-arm at jiffies + 1 actually fired every 2 ticks, doubling the PWM step to 8 ms and halving the PWM frequency to the 31 Hz flicker scoped in #120. Re-arming at jiffies (expire ASAP — bucketed at the next tick by construction) restores the designed 62.5 Hz at all duty levels (bench: 31.5 → 62.5 Hz at brightness 60/128/192). The sysfs scale stays 0–255; the 4-level duty quantization is now documented in the driver header.

Watchdog v1.4 — panic-path arm race fixed, panic record v3 (issue #99)

• v1.3 regression fixed — candidate lists were silently lost. v3.8.1 armed the recovery reset before the best-effort wheel walks with a single WDTCNR=0 write, assuming a ~1.31 s grace window. With the userspace kicker active the up-counter sits far above the OVSEL=0 threshold and the chip resets instantly — every v3.8.1 capture came back timers=[none] (confirmed on the bench with DRAM breadcrumbs, then in the field by frtz13's capture). The arm is now two writes: clear the counter while the watchdog is halted, then enable.
• delayed_work candidates resolved to their work function — e.g. neigh_managed_work instead of four indistinguishable delayed_work_timer_fn wrappers (the blind spot in the first #99 record-v2 field capture).
• Wheel backlog captured. New record fields overdue= (jiffies the earliest queued timer is past expiry) and pending= (total queued timers) discriminate the two storm shapes behind #99: a wheel that never catches up (death spiral — large overdue) vs a softirq re-raised over a caught-up wheel (overdue ≈ 0). Bench: overdue=0j on a healthy crash, overdue=6269j (25.1 s of wheel starvation) under an injected soft-lockup.

Upgrading

Full install (recommended for new users):

./flash_install_rtl8196e.sh <gateway-ip>

Existing v3.8.x installs (including the #99 soak boxes) only need the kernel partition — config, Thread dataset and SSH keys untouched:

./3-Main-SoC-Realtek-RTL8196E/flash_remote.sh -y kernel <gateway-ip>

After upgrading, re-arm the panic capture: rm -f /userdata/panic/history (note: after a kernel-only reflash the login banner still shows the previous version — cosmetic only).

v3.8.2 — flashing reliability + UX

v3.8.2 — flashing reliability + UX

A host-side flashing-tooling release. The on-device firmware is byte-identical
to v3.8.1 (same kernel and bootloader binaries), so there is nothing to re-flash
for functionality — pull the updated scripts and you're set. Re-flashing simply
updates the version banner to v3.8.2.

Fixed

• flash_install_rtl8196e.sh no longer mislabels a successful auto-flash as
  "manual flash required". The pre-upload ICMP probe that classifies the
  bootloader could miss on a single dropped/raced ping and fall through to the
  manual-FLW path — even though a custom (V2.x) bootloader had already auto-flashed
  the image and rebooted. The probe now polls (~10 s) instead of pinging once,
  and the upload now happens immediately after the gateway enters the bootloader
  (the image is built before boothold on the upgrade path, so nothing masks the
  bootloader's settle window). Thanks @MaxRower for the clear report in #115.

• Clearer fallback guidance. When the script does fall back to the manual path,
  it now tells you that a custom bootloader may have auto-flashed on its own — check
  ping / ssh to the gateway (wait ~2 min for the reboot) before re-flashing,
  instead of the previous, misleading "nothing was changed".

Added

• --boot-ip / BOOT_IP accept a hostname. A name is resolved host-side to a
  dotted-quad before use (the on-device handoff needs a literal IPv4). A dotted-quad
  still passes through unchanged. Works in both flash_install_rtl8196e.sh and
  flash_remote.sh.

Notes

• Upgrade path (flash_install_rtl8196e.sh <LINUX_IP>): the 16 MiB image is now
  assembled while Linux is still up, so the gateway no longer sits idle in download
  mode during the build, and declining the final confirmation leaves Linux intact.

v3.8.1 — watchdog notifier ordering + Docker health-check fix

Patch release on top of v3.8.0 — two fixes, no functional change to the Zigbee/Thread data path.

Kernel — hardware watchdog rtl819x_wdt 1.2 → 1.3 (panic-notifier ordering hardened)

The panic notifier now writes the core post-mortem record (uptime, reason, running fn, pc/ra, softirq mask), sets the magic, and arms the ~1.31 s reset before the best-effort timer/hrtimer wheel walks. A diagnostic walk can no longer lose the record or delay recovery; the candidate-callback lists are a bonus, never a dependency. Record format unchanged (v2). The prebuilt kernel-6.18.img is rebuilt with this driver.

Docker — cpcd-zigbeed health check no longer false-flags unhealthy

3.8.0 serializes the zigbeed port with socat ...,fork,max-children=1. Once Z2M holds the single slot, socat stops accepting, so the old connect-based probe (nc -z localhost 9999) piled up unaccepted in the listen backlog and eventually timed out — flapping the container to unhealthy while the stack was perfectly fine. The probe now confirms the port is in LISTEN state via /proc/net/tcp (no connection opened).

Fixed in both places, because a Compose-level healthcheck: overrides the image's:

• the image Dockerfile.multiarch HEALTHCHECK, and
• docker-compose-zigbee.yml (the cpcd-zigbeed health check).

If you run the Compose stack, update your docker-compose-zigbee.yml (or pull the new one) — pulling the :3.8.1 image alone will not change the health check, since the Compose definition takes precedence.

Upgrade

• Zigbee/RCP Docker users: docker compose pull && docker compose up -d with the updated compose file.
• On-device watchdog fix: reflash the kernel partition (kernel-6.18.img).

v3.8.0 — cpcd native TCP bus + soft-lockup post-mortem that names the culprit

v3.8.0 — cpcd native TCP bus + soft-lockup post-mortem that names the culprit

No EFR32 firmware change — the .gbl artefacts are identical to v3.7.0.
This release improves the RCP host stack, hardens the kernel, and turns the
watchdog panic post-mortem into a real diagnostic for issue #99.

RCP host stack — cpcd native TCP bus (drops the socat PTY shim)

cpcd now speaks bus_type: TCP and dials the gateway's in-kernel
UART↔TCP bridge (TCP:8888) directly, owning its own reconnection. The old
path put a socat in front of cpcd to fake a PTY; on any TCP blip that
PTY went stale and cpcd + zigbeed restarted in cascade. Now a bridge
drop loses only the host-side TCP (not the EFR32), and CPC's reliability
layer (sequence numbers + ACK + RTO retransmit) resumes the session with
sequence continuity — no secondary reset, no daemon restart. Validated
against a live EFR32MG1B RCP including a mid-session bridge disarm/re-arm.

• Docker stack (ghcr.io/jnilo1/cpcd-zigbeed) is wired for it: bus_type: TCP, no more socat-cpc program. The native rcp-stack (systemd
  --user) flow is migrated too. Fall back with bus_type: UART +
  uart_device_file if you need the classic socat-PTY path.

Docker — socat-zigbeed serves one Z2M client at a time (#112)

socat tcp-listen:9999,…,fork forked a handler per connection, so two
overlapping Z2M sessions (e.g. a Z2M restart racing the old session) both
relayed onto the same zigbeed PTY and interleaved bytes, corrupting EZSP
and rebooting zigbeed. The fork is dropped (max-children=1) so a single
client owns the PTY at a time.

Kernel — Ethernet driver rtl8196e-eth 2.5 → 2.6

Correctness hardening at no throughput cost (TCP RX 93.9 / TX 71.5 Mbit/s,
zero interface errors across the iperf3 suite):

• RX poll indexes the shadow skb by the hardware mbuf index (guarded by
  mbuf_index < rx_cnt), so a switch that links a pkthdr to a mbuf at a
  different ring index under saturation can't hand the stack the wrong skb.
• rtl8196e_ptr_in_pool() range/alignment-checks descriptor pointers read
  back from the rings; a corrupt TX descriptor returns -EIO instead of a
  wild dereference. 24 ring-anomaly stats via ethtool -S eth0 (zero in
  nominal flow).
• mm/c-lexra D-cache range-flush is bounded to the rounded 16-byte-line
  range — no more 128-byte blast radius spilling onto adjacent buffers on
  this non-coherent platform.

Kernel — Watchdog rtl819x_wdt 1.1 → 1.2: the panic post-mortem now names the soft-lockup culprit

The v3.7.0 post-mortem recorded only the running timer callback, but an
issue-#99 soft-lockup storm sits between timer callbacks where
running_timer is already NULL — so it read running=0x0 and could not
name the culprit (confirmed by the first field captures of #99 on v3.7.0
units). The record (format v2) now also captures, all in the panic path
and resolved to symbols only on the next boot:

previous boot ended in panic: uptime=<s> pc=<fn> ra=<fn> running=<fn> \
  softirq=0x<mask>[NAMES] timers=[...] hrtimers=[...] reason="<msg>"

• pc + ra — the program counter and return address of the stuck
  context (get_irq_regs()), which the watchdog hrtimer interrupted. Names
  the stuck frame even on a console-less gateway.
• softirq — local_softirq_pending() decoded to vector names (e.g.
  softirq=0x102[TIMER|HRTIMER]): which softirq is storming.
• timers / hrtimers — up to 6 candidate callbacks queued near
  expiry on the panicking CPU; the self-rearming culprit is among them, the
  one recurring across captures. These walks run only in the panic path
  (cold) — no normal-operation cost.

It surfaces on the next boot in dmesg and is persisted to
/userdata/panic/history (first occurrence only; re-arm with
rm /userdata/panic/history).

Tooling

• flash_efr32.sh — fixed a false-negative radio.conf-write abort on
  RCP/NCP/Router flashes (empty MODE=); genuine SSH failures still
  surface.
• cpcd.conf — dropped the unrecognized socket_folder key (cpcd v4.5.3
  ignores it; the /dev/shm default already yields the right path).

Upgrading

• Gateway (Zigbee/Thread): the only on-device change vs v3.7.0 is the
  kernel (Ethernet 2.6 + watchdog 1.2) — reflash the kernel
  partition. Userdata/rootfs are unchanged. See 35-Migration/.
• RCP / multi-PAN Docker users: pull the refreshed
  ghcr.io/jnilo1/cpcd-zigbeed:latest (or pin :3.8.0).