Sonoff S26R2ZB Smart Plug Zigbee Router devices by ITead disconnect from network after a day or two - Firmware bug or?

[daufinsyd]

Hello :)

What happened

After a day or two running perfectly fine, routers kind of disconnect from the network; trying to execute any command results in timeout.
It happens for all routers I have (6 sonoff s26r2zb). End devices (sonoff snzb02 / 03) (directly connected to the coordinator) don't appear to be affected.

example of failed command:

Zigbee2MQTT:error 2021-12-18 16:30:30: Publish 'set' 'state' to 'PLUG-SF-SALON' failed: 'Error: Command 0x00124b0024c08124/1 genOnOff.on({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Timeout - 22057 - 1 - 184 - 6 - 11 after 10000ms)'

After the router "disconnected", most of the time, simply plugging it off and back solve the issue for a day or two.

I tried increasing the TX power but it didn't help, all devices (with either good or bad LQI disconnect).

What did you expect to happen

Devices are working.

How to reproduce it (minimal and precise)

• Join devices
• Wait a day or two

Debug info

Zigbee2MQTT version: 1.22.1
Adapter hardware: CC2652RB (from slae.sh, as recommended in the official website)
Adapter firmware version: transportrev 2, product 1, majorrel 2, minorrel 7, maintrel 1, revision: 20210708

Edit: typo

[daufinsyd]

It might be related with #10276 but since these are made by another manufacturer I don't know ...

[irakhlin]

I am having the same issue with two different firmwares and multiple different adapters, granted all are CC2652P.

Routers:

1. Tube's Router/Repeater (CC2652P) (two units)
2. CircuitSetup's CC2652P2 USB Coordinator (flashed with router firmware)
3. Sonoff ZigBee 3.0 USB Dongle plus (two units flashed with router firmware)
4. Zigstar v4 (flashed with router firmware)

The two firmwares I have tried flashing all these devices with are:

1. https://github.com/Koenkk/Z-Stack-firmware/blob/master/router/Z-Stack_3.x.0/bin/CC1352P2_CC2652P_launchpad_router_20210128.zip
2. https://ptvo.info/zigbee-configurable-firmware-features/ configured with only the basics in router mode and a ping rate of around 5 minutes.

I will see these devices behave normally for anywhere from 1-3 days consistently pinging in however eventually they will no longer ping in. I usually only notice this when the last seen for all gets toward the 12hour + range. It does seem that they all stop reporting in around the same time but I cannot verify for sure. Running a network map request reports back that the devices are unreachable.
Only error I see for each device:
error 2021-12-20 16:37:27: Failed to execute LQI for 'Basement Router'

I have tried doing a packet capture but in all honesty I am not exactly sure what I should be looking for in this case. Simply power cycling the devices seems to return them to a working state in all cases. I just did this to all of them around 2-3 hours ago, so please let me know if there is anything else I can capture that would help troubleshoot this issue.

It might be related with #10276 but since these are made by another manufacturer I don't know ...

One thing I want to point out, none of my other devices are exhibiting this behavior these include but are not limited to:
KMPCIL_RES005 (KMPCIL)
Xiaomi (MCCGQ11LM, WXKG11LM, WSDCGQ11LM, VOCKQJK11LM)
Philips Hue (9290012607, 9290022166, 9290012573A)
Innr (sp224, sp234)
Smartthings (STS-IRM-250)

The issue only appears with the custom router devices and seems to have been happening fairly consistently across multiple versions of zigbee2mqtt, currently on 1.22.1.

My coordinator is a https://www.tubeszb.com/product/cc2652_coordinator/1?cp=true&sa=false&sbp=false&q=false&category_id=2 with firmware version 20210708

[daufinsyd]

Great to see that we're not the only one ! :)

I usually only notice this when the last seen for all gets toward the 12hour + range. It does seem that they all stop reporting in around the same time but I cannot verify for sure.
I had the issue yesterday and noticed that 2 routers were still connected. So it definitely doesn't happend for all a the same time (but within 12hours as you said ?).

I also get the same LQI error message as you.

I bought a Sonoff ZigBee 3.0 USB Dongle plus and I'll do test once I receive it. I'd like to provide more debug info but on't know where to start.

[daufinsyd]

According to HA, the routers disconnect every two days and 5 hours.

I got a new sonoff dongle plus ; I'll try with it.

[daufinsyd]

I flashed the latest firmware (20211217) for the new sonoff dongle 3.0 plus, 2 days ago and every routers disconnected today. :(
I didn't set the experimental power tx so that shoudln't be the cause but strangely enough, increasing the tx power seems to lower the reability (I made a few test and although the lqi was a little bit better, I had way more 'lqi failed for...' errors).

[daufinsyd]

So I ordered new plug/sonoff dongle, create a new network from scratch with

• 1 sonoff dongle 3 plug (factory coordinator firmware)
• 1 sonoff dongle 3 plug with latest router firmware flashed
• 3 sonoff plug (1 near the coordinator, 2 near the router)
  After two days, all the 3 plug were unresponsive.

[irakhlin]

@daufinsyd
I can report back that I am still having this issue and it does seem to further point to I believe one of a few things..

1. These devices are somehow being crashes by some other rogue device.. if this is possible? However only custom devices are affected.
2. I have tried this with both the router firmwares provided by @Koenkk and by @ptvoinfo any many different devices.
3. The behavior across the devices is slightly different but the end result is always the same, devices with any of the custom "router" firmwares available will work without issue for anywhere from 24hours -> 4/5 Days, eventually they will stop pinging in and ALL become unresponsive and will not ping or respond to commands. When I say all, I mean 5 "stanard router devices" and 4-5 devices I categorize as "custom" below.
4. It does seem that my coordinator and almost all the devices in question are using the cc2652p chip( easier to come by?). Outside of the sonoff zigbee 3.0 dongle plus they are using the RFstar cc2652p2.
5. Unplugging the device from power and plugging it right back in seems to work in resolving the issue until the next crash.

Standard devices tested (using both zstack router firmware and ptvo basic configured router firmware)

• Tube's Router/Repeater (CC2652P) (two units)
  

• CircuitSetup's CC2652P2 USB Coordinator (flashed with router firmware)
  

• Sonoff ZigBee 3.0 USB Dongle plus (two units flashed with router firmware)
  

• Zigstar v4 (flashed with router firmware)
  

Custom devices (using ptvo firmware, tested configured with 60 second ping. "enable watchdog timer was enabled in some, and not in others to check for any difference".

I created some custom PCB boards based on the designs of the popular coordinator sticks available (tubes router/zigstar/zzh) only with intent of using them as router devices loaded with various sensors (temperature/humidity/voc/ect), currently only using temperature/humidity. These were configured with ptvo firmware, additionally as I had added an additional LED to the board I was able to create a virtual switch in ptvo's firmware that would allow me to control the LED on the board. The intent here was to see if the device would respond to an on/off request when it went into a state of no longer pinging in.

result: device works perfectly pinging in every 60 seconds for days, seemingly 4-5 days? At a certain point I will notice that temperature data is not updating or changing in home assistant, I check back in zigbee2mqtt and the devices will show that they havented pinging in for X hours. Attempting to flip the virtual switch to trigger the LED on the board produces a timeout in zigbee2mqtt. Unplugging the device from power and instantly plugging it back in seems to fix everything with that device until the next "crash".

I have setup a second testing instance of home assistant with ZHA, will probably also install zigbee2mqtt. I have plenty of zigbee devices I can throw at the test network but I would really need a better understanding from @Koenkk or someone with more knowledge on this to help collect useful debugging data.

[tuffelh]

I can also confirm this behavior with Sonoff Stick flashed with latest router firmware.

[beezly]

I have a Sonoff usb stick (using the 20220103 firmware) configured as a coordinator, and a bunch of zzh's as routers along with some Sonoff s26r2zb's. Interestingly, I see the same behaviour, but only in the s26r2zb devices. All of the zzhs remain connected to the mesh.

[beezly]

Some additional info about the S26R2ZBs. I've taken one apart and photographed in the insides (mine is the UK version, but I would doubt there's much variation on these things).

Looks like it is powered by a CC2652 and has some handy solder pads for RX, TX, etc.

[daufinsyd]

I contacted the Sonoff Support and linked this case, I got a reply :

Dear buyer, the technical department has tested it, this is the problem of mqtt, they have contacted the mqtt staff to deal with it

(I guess they meant zigbee2mqtt and @Koenkk).

[Koenkk]

@daufinsyd they haven't contacted me about this. Could you make a sniff when command execution fails?

https://www.zigbee2mqtt.io/advanced/zigbee/04_sniff_zigbee_traffic.html#with-cc2531