Release v3.4.0 - May 1, 2026 #96
Replies: 5 comments 33 replies
|
first of all, thanks for your outstanding work. unfortunately, I ran into a little problem: flash_efr32.sh did not update radio.conf. Info on the gateway. SoC was flashed on May 1st, at 23:42. flash_efr32 was run on May 2nd. radio.conf not yet edited. |
|
Thanks for the detailed transcript — that helped a lot. I think what happened is that the script tries to extract the Gecko Bootloader version from USF's log, and on your run USF didn't print that particular line (it does internally detect it, but only logs it at INFO level in some code paths). The script runs under If you'd like to patch it locally before the next release, edit | tail -1 | sed -E "s/^Detected bootloader version '([^']+)'$/\1/")to: | tail -1 | sed -E "s/^Detected bootloader version '([^']+)'$/\1/" || true)(just adding After that, re-running One small note: your manual fix (setting I'll roll the fix into v3.4.1. |
|
the patch worked fine. radio.conf is now in a consistant state. |
|
Glad the patch fixed The Python traceback you saw is benign — the script handles it on purpose. Here is what happens, line by line: USF probed the chip and found OT-RCP (SPINEL) running at 460800. Good — that's the existing app it needs to kick out of the way before it can write a new image. USF sends the firmware-specific "drop into bootloader" command to the running app, then waits for a Gecko Bootloader handshake on the same UART. With SPINEL/OT-RCP at 460800 the chip does reset into the Gecko Bootloader, but the bootloader runs at 115200/no-flow, so USF — which is still listening at 460800 — never sees the handshake and raises That line is Short version: the traceback is USF's normal way of saying "I lost sight of the chip after enter_bootloader"; the script expects that on the EZSP→OT-RCP and OT-RCP→OT-RCP transitions (different baud / no flow control on the bootloader side) and recovers automatically. Nothing for you to do — flash succeeded. The only cosmetic wart is that the Python traceback is printed before our "Firmware detected — …" line, which makes it look like a fatal error. We could swallow it (redirect USF's stderr through a filter when we know we're going to recover), but that would also hide genuine failures, so we've kept it visible on purpose. |
|
Hello jnilo1, thank you for this outstanding work. I updated one fully stock gateway with usb-serial without any problems and it is working great as router and recognized in Z2M I have problem with my first gateway, I have version 1.x.x on it? Or older version? I can not restart to bootloader to flash version 3. |
|
Thanks for the logs — they're very informative. Good news first: the flash itself is now fully clean. Then the actual problem, which is different from what we thought. The chip is healthyFrom So the EFR32 is running One nuance about "network came up": Z2M restored your previous network from The new problem is on the UART link, not in the firmwareLook at lines around And one second later, in ASH (the framing layer between That's a much better problem to be debugging than "zigbeed crashes on every join". It also matches your "device joins then leaves" symptom: APS-level handshakes during a join are sensitive to a few lost bytes — the join packets land, but the trust-center key exchange a moment later fails because of corruption. Something is also wrong with your
|
|
Thanks for detailed troubleshoot lets go step by step.
I don't get any script output except
But the file is changed and has correct parameters Next I want to give logs during "idle" network, when I'm not joining or doing anything actively. There are still crashes logged cpcd debug Next are logs after reflash and reboot Logs for cpcd and Z2M after reboot And it is the same after switching baud to 115200 |
|
Thanks for the detailed run — those four logs give a much clearer picture, and they basically clear the kernel UART bridge of suspicion. The problem is one level higher, on the host side. What the bridge stats prove
The What's actually crashingLook at Two observations:
In
The Z2M I also see 8 of these in That's a CPC-level checksum complaint from Why zigbeed is probably dyingWhat's striking is that the cycle starts within ~30 s of every Z2M reconnect — long enough for Z2M to come up, load its database, and start polling its 25-ish previously-paired devices. NVM3 is empty, so the coordinator has no link keys, neighbor table, or source routes for any of those devices, but Z2M doesn't know that and tries to talk to them anyway. Some This isn't proven from the logs alone — The cleanest test: start Z2M with an empty databaseIf we move Z2M's persistent state aside and let it form a brand-new network, we'll know within a few minutes whether the problem is "talking to orphaned devices" or something deeper. On the host running the # stop Z2M first (whatever your compose / systemd unit is named)
docker compose stop zigbee2mqtt
# move the state aside (does NOT delete it — keeps a backup in case)
cd /path/to/zigbee2mqtt/data
mv database.db database.db.bak.$(date +%s)
mv coordinator_backup.json coordinator_backup.json.bak.$(date +%s)
# state.json is recreated by Z2M; safe to leave or also move aside
# also clean up any leftover state on the chip — flash the recovery GBL
# and rcp-460800.gbl one more time to be sure (you've already validated
# that procedure works for you)
# restart Z2M
docker compose start zigbee2mqttThen watch
(If you'd rather not touch your existing pairings, an alternative is to point Z2M at a different data directory for the test, so the new run gets a fresh DB while the old one stays untouched on disk.) Summary
|
|
Dear jnilo1, looks like some issue with Z2M. I cleaned coordinator, start new empty Z2M and tried to connect, everything works normally. Here are the logs For cpcd, you can ignore restarts until last one 17:45 due to Z2M misconfiguration (when Z2M logs start) Z2M with 10 minutes idle and adding temperature device Can we investigate further if there a rouge device configuration or something in network that is crashing daemon or to repair everything in new network? Thanks once more. |
|
There was one crash, here are the logs cpcd Z2M stats ~ # cat /sys/module/rtl8196e_uart_bridge/parameters/stats |
|
After upgrading to 3.4 from 2.x, the border router is quite unstable, I have to reboot it after some days, at least in one location. I had to reset the border router completely once as well. |
|
Thanks for the extraction @MaxRower — that gives us the missing piece. You have a partial upgradeThe three pieces of your system don't match:
The userdata partition holds your The "stable for 49 days, then a HardFault, then another 8 minutes later" pattern fits a state-dependent edge case (a specific neighbor joining, a specific Matter commissioning sequence, a specific routing-table update) rather than a constant bug. Version skew is exactly the kind of thing that produces that signature. Recommended fixBring the system to a consistent v3.4 with the proper installer: # from a fresh checkout of v3.4 sources, on the host (not the gateway):
./flash_install_rtl8196e.sh
After it finishes, please confirm: cat /userdata/etc/version # should say v3.4.x - 2026-05-...
cat /userdata/etc/radio.conf # FIRMWARE=otrcp, FIRMWARE_BAUD=460800
ot-ctl state # router or leader, after a minuteAbout your other gatewayYou mentioned a second gateway at "the other location, with more devices" reporting What we hold off on for nowI had mentioned earlier a possible firmware-side Let me know how the re-install goes. |
|
Sorry, those two /etc/version filles are from the backup of the last upgrade, you wanted to know, what version exactly was running without problems, before the last upgrade. |
|
Next crash, with -d 5, see /var/log/messages |
|
Upgraded to 3.5.1, next report will be an issue. I think, the ot-rcp firmware does not need an upgrade? |
|
@sipe That's the confirmation. The fresh-DB log is clean from end to end — zero Direct answer to your A vs B questionGo with B — re-pair from the clean database. Reasons:
One small ask, if you have the patienceWhile you're re-pairing, the only thing that would genuinely help the project is a targeted check on one device. The two units in your inventory most likely to be the trigger are your Either way, no dichotomy, no extra setup — just one device, one window of logs. Don't worry about it if you'd rather just get back to normal. Side curiosityYou have two Thanks for the methodical testing — that's exactly what nailed this down. |
|
Hello @jnilo1 Few questions There was another crash while system was idle, only one coordinator and temperature sensor sending data, no paring no any other activities. There is homeassistant with few automatic scenarions (start lamp when sunset occurs) could this interfere? Second question for Tuya TS1201 IR blasters. Do you want me to pair directly without IKEA routers? Or do you want 1 Ikea router and to pair through them? Or any other scenario to be covered? For the side curiosity I hope this clears up a bit of confusion, I did lot's of parallel activities when this error occured so it may be confusing a bit |
|
Thanks @sipe — that's all useful, especially the idle-crash detail. The idle crash refines the hypothesisIf Z2M only knows about the coordinator and the temperature sensor, you'd expect total silence on the air. But your 26 previously-paired devices are still physically powered, broadcasting on the same channel and PAN ID. They don't know they've been orphaned, so they keep doing what they always do: routers re-broadcast beacons, end devices My working hypothesis — and it's still a hypothesis, I don't have direct proof from the logs you sent — is that If that's right, re-pairing your devices doesn't just populate Z2M — it cleans up the air. Once each device has fresh keys, its frames decrypt normally and the suspect code path stops being exercised. Quick sanity check, if you want to confirm before re-pairingYou can see the orphan chatter without touching anything: cat /sys/module/rtl8196e_uart_bridge/parameters/stats
sleep 60
cat /sys/module/rtl8196e_uart_bridge/parameters/statsIf the Home Assistant automations — almost certainly not the triggerIf HA sends a "turn on lamp at sunset" via MQTT and the target lamp isn't in Z2M's TS1201 test — pair it normally, no special topologyPair the TS1201 the way you would in your real setup: in its usual physical location, with the routers powered up. The mesh path it picks (direct to the coordinator, through an IKEA bulb, through your SONOFF DONGLE-E) doesn't matter for what we're trying to observe. What matters is whether the interview itself crashes Have Side curiosity — noted, thanksSo your lab is:
Three Lidl gateways plus a SONOFF dongle is a properly built setup. After you've re-paired, the mesh density from those four routers (plus your IKEA bulbs and the Tuya relay) should be excellent. Don't worry about the "parallel activities" — your logs were clean enough that we got here. |
|
Dear @jnilo1 Lets go step by step. I caught 4 more crashes during the night. Only temperature sensor is paired and sending data 2 meters from Coordinator. This is radio stats after whole night I paired both IR directly to Coordinator with fresh started CPCD and Z2M, no crashes. Do you propose any other testing before I repair all devices that maybe we find culprit? As always, I appriciate all the help and I hope this is helping you building even better lidl gateway (which is already great!) |
|
Thanks — these four logs are clean enough to draw real conclusions. Crash inventory (Z2MIDLE.txt)All four crashes share the same Z2M-side signature:
In every case the last EZSP frame Z2M saw was the unicast default-response it sends back after a temp/humidity report, and the radio acknowledged it with The variance in delay (0 s to 6 min) is the giveaway: this isn't a single APS-layer event that crashes zigbeed inline; it's an asynchronous handler firing on internal stack state that builds up over time. Orphan-RX hypothesis — refuted
The sleepy ones have given up on their stale parents and gone deep-sleep with minimal polling. The routers in your mesh (the two LidlRouters, SONOFF, the IKEA bulbs) maintain their tables silently in idle. So the air is essentially empty, and yet zigbeed still crashes four times in 18 hours. My orphan-RX picture was too coarse — the trigger is finer-grained than "lots of orphan chatter". That said, there's still one orphan signal in the log: at 20:00:48 your stack reports TS1201 test — clearedThe two startup What this means for re-pairingThe hypothesis was right in spirit but wrong in detail: orphans aren't actively bombing the air, but they may be leaving zigbeed's internal stack in an inconsistent state (stale APS link-key entries, NVM3 fragments, neighbor table referencing devices that no longer respond). Re-pairing everything should clean that up across the board:
So my recommendation is go ahead and re-pair. The test with the temp sensor alone has done what we wanted — it confirmed the crashes are not caused by Z2M activity (you had effectively none) and not caused by ambient orphan RF chatter (there is essentially none). What remains is the internal-state hypothesis, and the only way to test it is to put zigbeed in a known-clean state. Suggested procedureThe clean way, given everything we've learned in this session: 1. Clean both sides before re-pairing
2. Bring the radio back up and verify the baseline
3. Pair in order: routers first, then end-devices
4. After everything is paired Run another idle test like the one you just did (10–30 minutes of If crashes persist after a full re-pairThen we're past "leftover state" territory and into a real zigbeed 8.2.2 bug on this specific RCP build. At that point the next move would be enabling core dumps for the zigbeed process inside the container — that gives us the actual crash site rather than guessing. We can cross that bridge if we get there; let's not pre-empt it. Looking forward to seeing how the post-re-pair behaviour looks. |
Uh oh!
There was an error while loading. Please reload this page.
A hardening release. Closes the
S40buttonSIGSEGV known issue from v3.3.0, reduces the worst-case latency on the Zigbee / Thread radio path under heavy LAN load, and applies four out-of-band kernel driver audits as small, bounded patch sets.S40button— closes the v3.3.0 known issueThe v3.3.0 release notes flagged an intermittent SIGSEGV in
S40buttonafter some hours of idle polling — when it died, you silently lost the long-press →recover_efr32recovery surface until the next reboot. Root cause was the busyboxashinterpreter occasionally crashing on the loop that polled/dev/memwithdevmem.S40buttonis now a static C daemon (~112 KB, Lexra musl), with the same observable behaviour: 100 ms poll on GPIO 9, 3-sample debounce, 5 s long-press firesrecover_efr32 -q, status LED blinks during the hold for visual feedback. No more interpreter-level crash.To verify it's running on a v3.4.0 gateway:
Lower worst-case latency on the Zigbee / Thread radio path
Under simultaneous heavy Ethernet activity (e.g. iperf-class TCP) and Spinel / EZSP burst (Matter commissioning attestation, Z2M device discovery), the kernel's CPU IRQ priority order used to favour Ethernet over UART1 — the bus carrying the radio link. At 460800 baud the UART RX FIFO has roughly 350 µs of latency budget; if an Ethernet ISR ran first and took longer than that, the FIFO could overrun and lose Zigbee bytes.
This release swaps the priorities at the interrupt-controller routing level: UART1 is now serviced before the Ethernet switch IRQ. The asymmetric benefit is:
Validated by an overnight OT-RCP soak at 460800 baud with two paired Sleepy End Devices: 8 h+ stable, zero overruns measured by a periodic LSR sampler, no
HandleRcpTimeout(). Not a fix for an active bug — a margin improvement that makes the gateway slightly more tolerant to bursts during Matter commissioning while LAN traffic is heavy.Ethernet TX no longer leaks slab tailroom on short frames
The Ethernet driver was DMA'ing
max(skb->len, 60)bytes for every transmitted frame, but never extending the SKB itself when sending a short frame. The bytes betweenskb->lenand 60 came from slab tailroom — uninitialised memory that may briefly hold data from another kernel user — and the switch DMA was happily transmitting them on the wire on every short frame (ARP, IPv4 minimal, etc.). Low-impact but undesirable.Fixed: the driver now calls
skb_put_padto(skb, 60)before the cache flush, which extends the SKB and zero-fills the new tail.ip link set eth0 down/upis safe under loadIf you ran an
ip link set eth0 down; ip link set eth0 upcycle while traffic was flowing (e.g. via an init script restart), the nextupwould reuse descriptor rings the previousdownhad left in indeterminate ownership. Symptoms ranged from harmless (a few dropped packets at the moment of the cycle) to noticeable (TX stuck after a watchdog timeout, SKB leaks). Validated with a 50-cycle down/up loop under traffic on the dev gateway: zerorx_errors, zerotx_errorsafter the patch.This is a defensive fix — most users will never trigger the failure mode — but if you ever script
ip linkcycles, it's now safe.Kernel driver hardening pass
Four small out-of-band audits were applied to the custom RTL8196E kernel drivers. Most are robustness / API alignment, not user-visible bugs:
.irq_unmaskcallback. Plus the priority swap above.For traceability, each driver now ships a local
AUDIT.mdthat maps every audit finding ID to its commit SHA and status (fixed, deferred, or rejected — including the rejected ones, with the reasoning, so a future audit pass doesn't re-litigate decisions already made on hardware).No regression measured
Same iperf baseline as v3.3.0 (TCP RX 93.9 / TX 70.2 Mbit/s vs 93.9 / 71 reference), 5-min stress retransmit rate 0.00 %, no new kernel warnings or errors during 5 back-to-back reboots, no UART overrun on the overnight OTBR soak.
Upgrade
In-place upgrade. No EFR32 reflash needed — the
.gblartefacts (NCP / RCP / OT-RCP / Router) are unchanged from v3.3.0.radio.confand sysfs interfaces are unchanged.If you'd rather not type the password during an unattended upgrade:
(Requires
sudo apt install sshpass.)Full changelogs
AUDIT.mdin the kernel treeAll reactions