Devices disconnecting randomly (see bug #19747)?

[Ra72xx]

Disclaimer: This post is not meant to be rude. I very, very much appreciate the work done here for free, and I'm aware that I'm not elegible to some kind of premium support for an (otherwise great) piece of software created by some people in their spare time.

However, as one of the in the meantime many users bitten by bug #19747, I'm getting a bit desperate now because there is no feedback at all. (Not to count all those vaguely similar bug reports which have been filed since then with also zero replies.)

The bug results in silently loosing control over the Zigbee network, as

• endpoints are no longer usable although they are present in the ui,
• whole devices silently (without any hint in the ui) drop out of the network,
• devices are no longer controllable with Z2M, but the ui happily pretends that everything is fine.

All this happens not sometimes, but is daily routine for the affected users.

Common denominators are that

• Lost devices can be re-connected by re-pairing them. This also restores all settings again.
• The devices themselves also still seem to think that they are connected to a controller, no warning of a lost connection.
• Users with "beta"-EZSP coordinators are especially affected, but there are also users with "recommended" chipsets hit by this bug.
• The bug appeared between 1.33.0 and at least 1.33.2, as there are several reports of users who were able to restore database backups from this time and get back a reliable network. (Only downgrading Z2M seems not to work.)
• Aqara SRTS-A01 thermostats seem to be especially prone to this bug.
• Logs are hard to capture as you usually notice the bug only hours or days later, and as Z2M seemingly itself does not even notice that something is wrong. However, the issue thread contains some logs. And as the last working version can be determined, the root of issue should be restricted to few pull requests.

Given the fact that this is quite a severe bug which seems to be not only occur in rare corners cases, but is confirmed by several users, I wonder how to arouse developers' attention in this case. The Github issue list does not seem to be the right way.

[Koenkk]

@kirovilya could something have been introduced in 1.33.0 and 1.33.2 in the EZSP integration?

[kirovilya]

I specifically looked at the changes between versions of z2m
1.33.2 "zigbee-herdsman": "0.21.0",
1.33.0 "zigbee-herdsman": "0.18.5",
Koenkk/zigbee-herdsman@v0.18.5...v0.21.0

Changes in the EZSP adapter code are minimal. Work with the queue has been moved and you can specify its size in the settings; the use of the queue when sending messages to the endpoint has returned.
Previously, I removed the use of the queue for delivery so that there would be no unnecessary waiting, but they asked me to return it.
This was all done here Koenkk/zigbee-herdsman#771
regarding the task Koenkk/zigbee-herdsman#770 the author never wrote anything back.

[Ra72xx]

Is there anything we can do to clarify your last question in Koenkk/zigbee-herdsman#770 which has not yet been answered by the original author? IMHO the source of the problem could well have something to do with these concurrency patches (as not much else seems to have changed otherwise)?!

[Ra72xx]

Thanks for the feedback.
I also suspected the pull requests you cite, however there are also non-ezsp users with this issue. And there are reports that switching the hardware helped, however in other cases not ..

[verticalsoft]

Hello I've experienced this issue quite severe with lots of my Aqara TRV valve's and I'm using a network tube's Poe zigbee coordinator that I've have zero issues previously.

[jmcollin78]

Same with Aqara T1 on my side. I lose not all the device but only the power values, current value, and voltage.

[verticalsoft]

Something to add to my previous comments.

I have removed the Flag on all my TRV's from external sensor back to Internal and stopped sending temprature vlaues to the TRV's and they have remained on the network now for 3-4 days without any communications issues.

As soon as set the sensor to External and begin sending external temprature readings thats when my particular issues occur.

I hope this information can help or assist.

My setup is:-

15 x Xiaomi SRTS-A01
tube_zb_gw_cc2652p2_poe (Coordinator)
Zigbee2MQTT 1.34.0-1

Unfortunatly I can't remember when this started happening but it must have been a couple of zigbee2mqtt versions ago.

All my other zigbee devices work correctly.

[Ra72xx]

In the bug thread somebody noted that the problem started only after the Zigbee mesh had a certain size. So maybe this is all about overwhelming the adapter with communication needs?!
In Koenkk/zigbee-herdsman#770 (comment), there is still a question of @kirovilya open concerning the usage of adapter_concurrent: . As the original poster did not answer, maybe this is something that should be tried?

[akarnakov]

I also faced with this problem, but it appears even if my Zigbee mesh contains only 4 devices (ts011f plugs). They become irresponsive just few minutes after zigbee2mqtt is restarted (they continue to send ON/OFF status if I switch them by physical button, but polling is not working anymore). If I leave only one device it is working fine, but I don't think that 4 plugs can overwhelm the adapter (Sonoff dongle-e stick)...

[Element2]

Same problem here. Have a 30+ devices mesh with a ezsp coordinator (hacked Lidl gateway).
The mesh was always rock solid up until a recent update.
After the update devices randomly lose connection (and ping time outs) and can't be controlled anymore.
re-pairing reconnects them again, but same or other devices lose connection again soon after.

[verticalsoft]

Ok today I have 2 Aqara Tempreture Sensors (WSDCGQ11LM) that have stopped responding, these are generally bomb proof, the normal put things in pairing mode and press and hold the button did not work.

I could not pair the devices, not even if they moved next to the coordinator.
The only way I could get them to pair was through an aqara wall switch, which paired almost instantly.
I rebooted zigbee2mqtt and power cycled the coordinator but could not pair the device to the coordinator.

As an experminent I have another zigbee2mqtt setup on my local pc long with mqtt broker to test things and was able to pair with this instantly.

This is the same chipset and my life installation so I cannot figure out why this would pair on my dev setup as opposed to my lifve setup.

Very frustrating and I think I may be at the point of giving up on all this Zigbee stuff as it's starting to rule my life, especially in the last few updates.

No disrespect to any of the developers, but it's just not stable anymore.

[verticalsoft]

Sorry shoud of added that 3 aqara door sensors ( MCCGQ11LM ) stopped responnding to.

[Ra72xx]

As I just finished running around my flat chasing for "lost" SONOFF and Xiaomi window sensors, I'd like to keep this thread alive.
Apart from everything else, I'm wondering how it is possible that devices drop out of the network list and Z2M simple does not care. I only noticed the lost devices because the thermostats kept heating with open windows and then in the Z2M UI I saw that the number of devices was lower than usually, so I could chase down which devices were gone.
IMHO the Zigbee software should not let something like this happen without even a warning to the user?!

[maciekdnd]

Same here. 40+ devices, some of them are dropping without any cause or warning. Also looks like a random issue, when I try to reconnect one of my T1's Z2M crashes and whole network is down with new devices dropped. SONOFF_Zigbee_3.0_USB_Dongle_Plus_V2_20221202083941-if00

[jmcollin78]

Anyone check with the 1.35.0 new release ? https://github.com/Koenkk/zigbee2mqtt/releases/tag/1.35.0

[Ra72xx]

There are some fixes in the changelog which sound promising... I will install it right away.

[verticalsoft]

Can you point to the fixes so I can take a read?

[Ra72xx]

Koenkk/zigbee-herdsman#841 Fix EZSP initialisation check (@Nerivec)
Koenkk/zigbee-herdsman#830 Fix devices disappearing with zStack adapter when rejoining the network (@slugzero)
I don't know if these fixes are related, I'm no deveoper, but they deal with EZSP and with disappearing devices, and that's my problem ;-) ...

[verticalsoft]

Thanks trusts interesting...

[jmcollin78]

There is no change on ezsp but only for zstack if I understand well. So not sure it will change something.

[verticalsoft]

I use a tube's POE adapter so I'm not sure how this could effect me.

My issue is not so much items disappearing but just stop communicating and going offline.

I consider my network strong with many routers as possible but still get items randomly stop communicating.

I understand that's Aqara end devices can be sticky and if the device they are connected to goes offline then that could effect them but none of my routers stop responding.

[Nerivec]

Koenkk/zigbee-herdsman#841 fixes the initial check after the serial port opens for EZSP. Before the fix, the code would fail further down the line, even if the serial port never properly opened, creating confusion... It should now be clear when the adapter has trouble connecting. It won't change much as far as fixing connection issues, it will however make it easier to help debug for people having these issues.

The other fix is indeed zStack only.

[Nerivec]

I quickly went through the mentioned issues, there seems to be people commenting on zStack and EZSP alike, but it's unlikely the issues are from the same source (since the code is very different), unless the source has nothing to do with Z2M (interference & co)...

In case you have devices that seemingly "cheat" on reporting "offline", you can try setting the availability timeout for each of them. You can also try enabling Last seen in the advanced settings of Z2M, if you then enable the sensors it adds in HA, you can use them in automations...

I'll mention the WIP thread on EZSP, you can start around here, for more recent stuff, but there is a lot above too. It covers adapters' best practices, checking the actual config file (in case something went wrong with HA saving the UI config to file), and a lot more. I also posted a full working EZSP (Sonoff Dongle-E) config file there recently, if you want to compare/troubleshoot.

I only recently started perusing Z2M's codebase (EZSP side since that's what I have), but there are so many other factors that can affect a Zigbee network, sometimes the code can't do much about it... "Devices disconnecting randomly", 3 words a developer hopes to never hear; it's usually a nightmare to debug.
Your neighbor might have changed WiFi router (or created his own Zigbee network!), a crappy mains-powered device might make you think you're extending the network, when in fact it might quickly become a point of failure if it starts dropping messages because of poor signal (lots of cheap Aliexpress mains-powered devices have crap antennas -try adding an adapter flashed as router instead if you need some more range, if there's one that matches your config-)...

On my end, it's been quite the contrary recently (never worked better, since around ~1.33.1-1); it goes to illustrate how different it can be from environment to environment (and I only have cheap Aliexpress devices, currently 21 routers, 13 end devices, with a pair of Dongle-E, coordinator with 7.3.1...).

I'll take a closer look at the logs in the other issues when I have some time, see if I can spot something, but if kirovilya didn't spot anything... I doubt I will.

[PeterKawa]

Hi, thanks for describing possible causes, but I want to state nothing had changed here for months and months, except for updating to z2m v.1.33.2. Devices randomly offline/disconnecting, and pairing or re-pairing was a real PITA, took several times, up to 10
When I restored my system to v1.33.0 again, those issues disappeared right away, and did not occur anymore, not - a - single - hickup
Full report:
#19747 (comment)

[Ra72xx]

I cannot post historical data, as I switched to Z2M / Skyconnect from Deconz at about the time 1.33 was released. The setup went perfect for the first days, then everything deteriorated. As I was still in the configuration phase at this time I can't really say if there was an update or something like that at this point. It would fit, however.
I never had this kind of problems with the previous Deconz setup. I also live in a rural area and there are not too many possible sources of interference around here that I would know of. I followed the best-practices for Zigbee (USB 2.0 hub, long cable, away from wifi router, many routers, different positions, etc.)
There are some changes done in the 1.33 release cycle mentioned in #20323 (comment). However, it seems that the developer does not consider this changes in the adapter concurrency as possible culprits as he unfortunately did not write any follow-up posts or instructions how to play around with the adapter_concurrent: setting.

[Nerivec]

#10664
Some info there.

PS: EZSP has 8 by default (if not specified/null/0) see here.

[Ra72xx]

Thanks for the link. "0" means that Skyconnect tries to send all commands at once? Is this a possible reason for network problems in certain situations, as the concurrency code seems to have changed since 1.33.0?
For me the problem is that I can't really reproduce the situation, sometimes days pass until I even notice that some device has disappeared, or a thermostat has stopped responding to heating cycles. I also have the feeling that the situation is worst when I change something in the mesh (moving devices around etc.), but stabilizes after some time.

[Nerivec]

For other users wondering, as of zigbee-herdsman 0.30.0 adapter_concurrent has:

ezsp default: 8
zigate default: 2
zstack default: 16 if v3.x.0 else 2
deconz default: 2

If config zero, null or omitted, the above according value will be used.

[Ra72xx]

Reading post Koenkk/zigbee-herdsman#319 (comment), it looks as if upgrading coordinator firmware helps? The recommended version for SkyConnect is still 7.1.1.0, however.

[Nerivec]

For recent Skyconnect owners, there apparently was one (or more) bad batch that was incorrectly programmed
Here and here among others. Despite that problem being only about identification of the adapter, one could think some might have been improperly flashed too and forcing a clean re-flash could fix potential issues. But I can't comment on the result since I don't have one.

I'm pinging @Krzysztonek here since he seems to be in the same boat as yours Ra72xx, except all his devices won't obey... between you guys, we might find that darn bug... 😒

7.3.1 works great on the Sonoff Dongle-E though (flashed with darkxst's tool for ease)

[Element2]

Is there anything else what can be done to tackle this problem? It's still an issue with the latest version.
Random timeouts and disconnects still happening.
Changed the channel from 11 to 20 and re-paired all nodes, but no improvement.

Info:
Zigbee2MQTT version
1.35.1 commit: 3c96204
Coordinator type
EZSP v8
Coordinator revision
6.7.8.0 build 373
Coordinator IEEE Address
0xec1bxxxxxxxx
Frontend version
0.6.151
Zigbee-herdsman-converters version
18.9.0
Zigbee-herdsman version
0.30.0

Log:
Error 2024-01-19 10:43:25Publish 'set' 'state' to '0xec1bbdfffe7eeba0' failed: 'Error: Command 0xec1bbdfffe7eeba0/1 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Timeout - 15890 - 1 - 155 - 6 - 11 after 10000ms)'
Warning 2024-01-19 10:43:53Failed to ping 'Tuinkamer plafond A2' (attempt 1/2, Read 0x842e14fffe4054b1/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":true,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Timeout - 29314 - 1 - 167 - 0 - 1 after 10000ms))
Warning 2024-01-19 12:48:06Failed to ping 'Kelder plafond' (attempt 1/2, Read 0x588e81fffeef4e21/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":true,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))
Warning 2024-01-19 12:48:09Failed to ping 'Kelder plafond' (attempt 2/2, Read 0x588e81fffeef4e21/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))
Warning 2024-01-19 12:48:14Failed to ping 'Tuinkamer plafondB1' (attempt 1/2, Read 0x588e81fffee72ec7/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":true,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))
Warning 2024-01-19 12:48:17Failed to ping 'Tuinkamer plafondB1' (attempt 2/2, Read 0x588e81fffee72ec7/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))
Warning 2024-01-19 12:48:59Failed to ping 'Keuken hal plafond' (attempt 1/2, Read 0x588e81fffefea59b/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":true,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))
Warning 2024-01-19 12:49:21Failed to ping 'Eetkamer raamlamp' (attempt 1/2, Read 0xbc33acfffe0ebb93/11 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":true,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))
Warning 2024-01-19 12:49:29Failed to ping 'Buitenlamp achterdeur' (attempt 1/2, Read 0x60a423fffe423cae/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":true,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))
Warning 2024-01-19 12:49:32Failed to ping 'Buitenlamp achterdeur' (attempt 2/2, Read 0x60a423fffe423cae/1 genBasic(["zclVersion"], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (sendZclFrameToEndpointInternal error))

[Ra72xx]

According to the latest comments in #19747, I have hopes that the next stable version will be better?!

[Element2]

Thanks for the heads up!

[PeterKawa]

I think this can be considered fixed, and closed.

[Ra72xx]

I can close it, es the ezsp driver with which the error occurs for me is deprecated. However, for me it is still a problem as the new ember driver does not work for me...