Zigbee network keeps crashing ... feels like I tried everything

[PJBLinkHA]

Hi everyone,

It's a bit despaired that I'm reaching out as I usually get my way around reading over and over posts and issues from others.
But this time, I can't make it work.

Context:
I'm running a small network of devices, a few Aqara Zigbee Switches, a couple Yeelight Bulb and a few plugs, both Zigbee and Wifi. All this on the Rpi 3B+ with SSD and ZZh Zigbee stick.

I used to have my switches paired with the Xiaomi Gateway itself connected to HA and all automation done via nodered. This was more than 2 years ago. And it ran perfect for that long with literally no crash whatsoever.

I recently thought it was time to update HA seeing the most recent development which are amazing (such a long way in 3 years).
So I want for a fresh HA install and it went rather smooth. I took this opportunity to "get rid" of the gateway and pair my switches directly with HA through MQTT (Mosquito and Z2M) and still using Node Red. It worked almost instantly, no fiddling at all.

Problem:
Since then, there wasn't a single day I didn't have to restart Z2M to make my network functional again.
I was working with an old Zigate V1 which is officially not supported. So I changed it for the praised ZZh thinking it would make things better. Nope.

So I tried all i could read:

• extension cable, 1.3m, 5m, several positions far from the router and Rpi. Nothing
• having only the ZZh stick on the Pi (with the SSD). Nothing
• changing Zigbee channel > I'm currently on 25 which is empty around (I can scan the wifi channels with my ISP router)
• unplug and replug the stick and well pushing the little reset button. Still nothing.

Nothing to do, after a few hours, like 6/8, the network is broken and I get the "failed to execute LQI for coordinator" and nothing works. The network map shows up but no link between the devices.
From the Z2M interface, I have to reboot Z2M and it miraculously work again.

If that help the diagnosis, I'd also say that the network is now slower than it used to be. I get a 1sec delay between a push on a switch and the lamp turning on. Sometimes more, sometimes less. It's quite strange.

There is nothing in the HA logs that is related to Z2m, only a couple things related to Xiaomi map extractor.
So I've enabled the Herdsman logging as I've seen asked in other posts, but I have to admit at this point it's a bit Chinese to me....
If anyone has an idea, I'd love to hear it ....

One last thing to note that seems unrelated but adding anyway in case: when I update my HA, the supervisor crashes. HA works fine but I don't have access to the ad-dons and the supervisor. I have to cut the power and reboot everything so that it comes back to life.
Even soft reboot won't work, even through putty with commands.

Thanks for reading through, that's already something ;)

Here is the last bit of the Z2M log in case it helps

022-03-15T19:42:35.577Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - incomingMsg - {"groupid":0,"clusterid":64704,"srcaddr":31227,"srcendpoint":1,"dstendpoint":1,"wasbroadcast":0,"linkquality":105,"securityuse":0,"timestamp":4227057,"transseqnumber":0,"len":70,"data":{"type":"Buffer","data":[28,95,17,23,10,247,0,65,61,100,16,0,3,40,25,152,57,0,0,0,0,149,57,44,233,181,61,150,57,0,192,15,69,151,57,0,0,0,0,5,33,1,0,154,32,0,8,33,22,1,7,39,0,0,0,0,0,0,0,0,9,33,2,4,11,32,0,155,16,0]}}
2022-03-15T19:42:35.579Z zigbee-herdsman:controller:log Received 'zcl' data '{"frame":{"Header":{"frameControl":{"frameType":0,"manufacturerSpecific":true,"direction":1,"disableDefaultResponse":true,"reservedBits":0},"transactionSequenceNumber":23,"manufacturerCode":4447,"commandIdentifier":10},"Payload":[{"attrId":247,"dataType":65,"attrData":{"type":"Buffer","data":[100,16,0,3,40,25,152,57,0,0,0,0,149,57,44,233,181,61,150,57,0,192,15,69,151,57,0,0,0,0,5,33,1,0,154,32,0,8,33,22,1,7,39,0,0,0,0,0,0,0,0,9,33,2,4,11,32,0,155,16,0]}}],"Command":{"ID":10,"name":"report","parameters":[{"name":"attrId","type":33},{"name":"dataType","type":32},{"name":"attrData","type":1000}]}},"address":31227,"endpoint":1,"linkquality":105,"groupID":0,"wasBroadcast":false,"destinationEndpoint":1}'
2022-03-15T19:42:35.585Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-03-15T19:43:29.319Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,3,69,196,188,30,0,32]
2022-03-15T19:43:29.320Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,3,69,196,188,30,0,32]
2022-03-15T19:43:29.325Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 3 - 2 - 5 - 196 - [188,30,0] - 32
2022-03-15T19:43:29.326Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- ZDO - srcRtgInd - {"dstaddr":7868,"relaycount":0,"relaylist":[]}
2022-03-15T19:43:29.326Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-03-15T19:43:29.330Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [254,62,68,129,0,0,0,0,188,30,1,1,0,102,0,115,198,115,0,0,42,28,95,17,4,10,1,255,66,33,100,16]
2022-03-15T19:43:29.331Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,62,68,129,0,0,0,0,188,30,1,1,0,102,0,115,198,115,0,0,42,28,95,17,4,10,1,255,66,33,100,16]
2022-03-15T19:43:29.333Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [1,3,40,29,152,57,195,245,240,64,149,57,42,76,9,65,5,33,12,0,154,32,0,8,33,92,17,9,33,0,2,188]
2022-03-15T19:43:29.334Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,62,68,129,0,0,0,0,188,30,1,1,0,102,0,115,198,115,0,0,42,28,95,17,4,10,1,255,66,33,100,16,1,3,40,29,152,57,195,245,240,64,149,57,42,76,9,65,5,33,12,0,154,32,0,8,33,92,17,9,33,0,2,188]
2022-03-15T19:43:29.336Z zigbee-herdsman:adapter:zStack:unpi:parser <-- [30,29,151]
2022-03-15T19:43:29.337Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext [254,62,68,129,0,0,0,0,188,30,1,1,0,102,0,115,198,115,0,0,42,28,95,17,4,10,1,255,66,33,100,16,1,3,40,29,152,57,195,245,240,64,149,57,42,76,9,65,5,33,12,0,154,32,0,8,33,92,17,9,33,0,2,188,30,29,151]
2022-03-15T19:43:29.338Z zigbee-herdsman:adapter:zStack:unpi:parser --> parsed 62 - 2 - 4 - 129 - [0,0,0,0,188,30,1,1,0,102,0,115,198,115,0,0,42,28,95,17,4,10,1,255,66,33,100,16,1,3,40,29,152,57,195,245,240,64,149,57,42,76,9,65,5,33,12,0,154,32,0,8,33,92,17,9,33,0,2,188,30,29] - 151
2022-03-15T19:43:29.339Z zigbee-herdsman:adapter:zStack:znp:AREQ <-- AF - incomingMsg - {"groupid":0,"clusterid":0,"srcaddr":7868,"srcendpoint":1,"dstendpoint":1,"wasbroadcast":0,"linkquality":102,"securityuse":0,"timestamp":7587443,"transseqnumber":0,"len":42,"data":{"type":"Buffer","data":[28,95,17,4,10,1,255,66,33,100,16,1,3,40,29,152,57,195,245,240,64,149,57,42,76,9,65,5,33,12,0,154,32,0,8,33,92,17,9,33,0,2]}}
2022-03-15T19:43:29.343Z zigbee-herdsman:controller:log Received 'zcl' data '{"frame":{"Header":{"frameControl":{"frameType":0,"manufacturerSpecific":true,"direction":1,"disableDefaultResponse":true,"reservedBits":0},"transactionSequenceNumber":4,"manufacturerCode":4447,"commandIdentifier":10},"Payload":[{"attrId":65281,"dataType":66,"attrData":{"3":29,"5":12,"8":4444,"9":512,"100":1,"149":8.581094741821289,"152":7.53000020980835,"154":0}}],"Command":{"ID":10,"name":"report","parameters":[{"name":"attrId","type":33},{"name":"dataType","type":32},{"name":"attrData","type":1000}]}},"address":7868,"endpoint":1,"linkquality":102,"groupID":0,"wasBroadcast":false,"destinationEndpoint":1}'
2022-03-15T19:43:29.350Z zigbee-herdsman:adapter:zStack:unpi:parser --- parseNext []
2022-03-15T22:01:26.822Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - mgmtLqiReq - {"dstaddr":0,"startindex":0}
2022-03-15T22:01:26.823Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,3,37,49,0,0,0,23]
2022-03-15T22:01:37.835Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - mgmtLqiReq - {"dstaddr":0,"startindex":0}
2022-03-15T22:01:37.836Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,3,37,49,0,0,0,23]
Zigbee2MQTT:error 2022-03-15 23:01:43: Failed to execute LQI for 'Coordinator'
2022-03-15T22:01:44.927Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - mgmtLqiReq - {"dstaddr":31227,"startindex":0}
2022-03-15T22:01:44.928Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,3,37,49,251,121,0,149]
2022-03-15T22:01:55.938Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - mgmtLqiReq - {"dstaddr":31227,"startindex":0}
2022-03-15T22:01:55.939Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,3,37,49,251,121,0,149]
Zigbee2MQTT:error 2022-03-15 23:02:01: Failed to execute LQI for 'Prise Test 1'
2022-03-15T22:02:02.954Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - mgmtLqiReq - {"dstaddr":7868,"startindex":0}
2022-03-15T22:02:02.955Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,3,37,49,188,30,0,181]
2022-03-15T22:02:13.963Z zigbee-herdsman:adapter:zStack:znp:SREQ --> ZDO - mgmtLqiReq - {"dstaddr":7868,"startindex":0}
2022-03-15T22:02:13.964Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,3,37,49,188,30,0,181]
Zigbee2MQTT:error 2022-03-15 23:02:19: Failed to execute LQI for 'Prise murale Chambre'
2022-03-15T22:08:40.682Z zigbee-herdsman:controller:endpoint Command 0x00158d000313e2b7/1 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false})
2022-03-15T22:08:40.683Z zigbee-herdsman:adapter:zStack:adapter sendZclFrameToEndpointInternal 0x00158d000313e2b7:7868/1 (0,0,1)
2022-03-15T22:08:40.684Z zigbee-herdsman:adapter:zStack:znp:SREQ --> AF - dataRequest - {"dstaddr":7868,"destendpoint":1,"srcendpoint":1,"clusterid":6,"transid":4,"options":0,"radius":30,"len":3,"data":{"type":"Buffer","data":[1,4,0]}}
2022-03-15T22:08:40.685Z zigbee-herdsman:adapter:zStack:unpi:writer --> frame [254,13,36,1,188,30,1,1,6,0,4,0,30,3,1,4,0,144]
2022-03-15T22:08:46.690Z zigbee-herdsman:controller:endpoint Command 0x00158d000313e2b7/1 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (SRSP - AF - dataRequest after 6000ms)
Zigbee2MQTT:error 2022-03-15 23:08:46: Publish 'set' 'state' to 'Prise murale Chambre' failed: 'Error: Command 0x00158d000313e2b7/1 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (SRSP - AF - dataRequest after 6000ms)'

[aldur11]

Is your SSD connected directly to the RPI? I had similar problems until I added a powered usb hub to my RPIs. Apparently, it's common for the RPI's PSU to not supply enough power to run an SSD. As soon as I added a powered usb hub and connected the SSD to it, my crashing problems went away.

[PJBLinkHA]

Good point, I hadn't considered that.
Wouldn't the PI be crashing altogether though? I mean not only the Zigbee network?

I don't have a powered hub so I'll order one.
I'll try with a more powerful supply as well. I'm already above the original one from the PI but who know...
It's worth a shot.

Thanks a lot!

[aldur11]

Good point. I had PI crashing and z2m also crashing. I tried setting up a cron job to reboot the PI every day, but that didn't help. Using a powered usb hub fixed both problems for me.

[PJBLinkHA]

I'll give it a shot anyway, it's doesn't hurt :)

[PJBLinkHA]

Bon, I've tried to solve any potential power problem but it doesn't change anything.
What I've done since last time:

• adding a powered USB hub for the SSD as suggested
• changing the power supply for a brand new and official Raspberry one.
• moving the SSD away from the PI and the Zigbee dongle on the other side so now physically it's "Zigbee Dongle < RPi > SSD", with like 15 cm between each (knowing that I already tried moving the Zigbee Dongle 3 meters away from the Pi and SSD and it didn't change anything)

This made absolutely no difference, my Zigbee network is still crashing after 6/8 sometimes 12 hours.

I do have a few error in the log which I couldn't find any source for. Researching them leads to a lot of non conclusive answers as those errors are too broad. I'm including them below in case it's helpful.
The "Can't read supervisor data" seems to be annoying as after this one, I can't access the supervisor and I have to manually reboot the Pi by pulling the plug off (soft reboot doesn't work, not even command line on supervisor, neither do the rebuild and repair commands by the way).
BUT this error is not always there when my Zigbee network crashed and vice versa.

I'm a bit desperate at this point because I tried everything I read and could and not luck... Does anyone have an idea?

---

_Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:254
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 09:41:43 (1 occurrences)
Last logged: 09:41:43

[547516776992] Disconnected: Did not receive auth message within 10 seconds_

---

_Logger: aiohttp.server
Source: /usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py:405
First occurred: 25 mars 2022, 21:48:40 (2 occurrences)
Last logged: 01:38:31

Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 334, in data_received
messages, upgraded, tail = self._request_parser.feed_data(data)
File "aiohttp/_http_parser.pyx", line 551, in aiohttp.http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message='Pause on PRI/Upgrade'

---

_Logger: frontend.js.latest.202203012
Source: components/system_log/init.py:190
First occurred: 25 mars 2022, 20:54:12 (5 occurrences)
Last logged: 25 mars 2022, 20:54:21

:0:0 ResizeObserver loop completed with undelivered notifications._

---

Logger: homeassistant.components.hassio
Source: components/hassio/init.py:569
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 15:45:19 (2 occurrences)
Last logged: 16:16:00
Can't read Supervisor data:

With some detail for this last one:

2022-03-26 15:06:09 ERROR (MainThread) [frontend.js.latest.202203012] https://192.168.1.50:8123/frontend_latest/bfb7ed92.js:1:1608 TypeError: e is undefined
2022-03-26 15:06:13 ERROR (MainThread) [frontend.js.latest.202203012] https://192.168.1.50:8123/frontend_latest/bfb7ed92.js:1:1608 TypeError: e is undefined
2022-03-26 15:11:29 ERROR (MainThread) [frontend.js.latest.202203012] :0:0 ResizeObserver loop completed with undelivered notifications.
2022-03-26 15:45:19 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_nodered/stats request
2022-03-26 15:45:19 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_mosquitto/stats request
2022-03-26 15:45:19 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_duckdns/stats request
2022-03-26 15:45:19 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/45df7312_zigbee2mqtt/stats request
2022-03-26 15:45:19 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_samba/stats request
2022-03-26 15:45:19 WARNING (MainThread) [homeassistant.components.hassio] Can't read Supervisor data:
2022-03-26 16:03:12 WARNING (SyncWorker_4) [custom_components.xiaomi_cloud_map_extractor.camera] Unable to retrieve map data
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_mariadb/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/45df7312_zigbee2mqtt/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_ssh/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_mosquitto/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_nodered/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_configurator/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/cebe7a76_hassio_google_drive_backup/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_duckdns/stats request
2022-03-26 16:16:00 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_samba/stats request
2022-03-26 16:16:00 WARNING (MainThread) [homeassistant.components.hassio] Can't read Supervisor data:   

[PJBLinkHA]

Just found about a new error in the supervisor log :

[supervisor.api.ingress] Ingress error: Cannot connect to host 172.30.33.3:8099 ssl:default [Connect call failed ('172.30.33.3', 8099)].

Could this be related?

[aldur11]

Dang, sorry to hear that you're still having problems. I was hoping the powered USB hub would solve your problems.

I don't have the skillset to figure out what's going on from the logs, hopefully someone else can chime in there.

Are you using the z2m integration in HA? I had some problems with that and moved to having HA by itself and z2m on a separate RPI, that seemed to help.

It also could be something screwed up with HA. It might be worth blowing away the image on your RPI and starting fresh. Also, are you using HACS to get 3rd party integrations? I had an issue were using the Adaptive Lighting 3rd party integration was screwing up my zigbee networks. Removing the integration didn't solve the problem either. I had to redeploy my devices that were using z2m.

[PJBLinkHA]

It was worth a shot! It's good practice anyway, probably saves me from futur trouble ;)
Yes, I do use the Z2M integration for HA. But frankly I don't feel like getting a second PI. In that case, I'll buy a NUC and un everything from it.
And I tried going to a fresh HA install but didn't change single thing. Like it had copied the error along with the backup.
HACS inegration is a good lead, I can try that easily.

I made some observation this week.
My zigbee network was this time, partially down with only the 2 plugs (router) still connected and a simple bouton (the round one).
I left it that way for 3 days and it seems everything disconnected again at some point.
Rebooting the addon doesn't change anything anymore, I don't get back the full network as it use to be the case, only those 3 devices.

Good news is I don't have the 'can't read supervisor data' anymore (for now) in the log.

Does any of this help the diagnosis?

[ghost]

Hi,
I have continuous problems with the stability of the Zigbee Network - did you eventually got it up and running, or are you still dealing with these problems?
Thanks!

[PJBLinkHA]

Yes! I've been meaning to report for a while but didn't take the time.
Yet, I'm not sure it's going to be very helpful for you as it was super specific to my case: my SSD enclosure was faulty, creating errors accumulating, creating lag and corruption on the pi. Too much for the Zigbee network which was going down.

What also helped before and after that issue was:

• moving the Zigbee dongle to a powered USB Hub (even with genuine pi power supply, it's not enough for the Zzh for example)
• increasing the swap file. I have a Pi 3B and HA has become a bit too much for 1go of Ram. This [(https://community.home-assistant.io/t/how-to-increase-the-swap-file-size-on-home-assistant-os/272226?u=pjblinkha)] is super easy and work perfectly. It solved all of the other issue I had on the PI.

Hope this helps anyway!

[ghost]

Thanks - good to hear that it worked out for you, but yes, an SSD is not my issue.
However, when you say, you move the dongle to a powered USB hub, I am not quite sure I understand: you have the USB hub connected to the RPi and separately to the dongle?

[PJBLinkHA]

No, now the only USB left connected to my PI is the powered USB hub. The SSD and the ZZh stick are connected to the powered hub.
So that the Pi has all the power from it's supply for himself and the peripherals draw power from the hub.
Does that make sense?

If you're running a Pi 3B, I would highly suggest to do the same as both HA and and the peripherals have gone up in terms of power consumptions over the past few year.
And the Pi 3 is not very recent therefore not really sized for that it seems.
Now this is only based on my experience.