deconz devices becoming unavailable

[aguilaair]

The problem

My ZigBee devices connected to deConz are becoming constantly unavailable without any apparent reason. I have multiple brands including Phillips and Aqara. They're all bettery-run so none are acting as repeaters.

Environment

docker_version | 19.03.8
host_os | HassOS 4.10
os_version | 4.19.126-v8
python_version | 3.7.7
supervisor | 228
version | 0.112.0
frontend | 20200603.3 - latest
deconz | 5.3.6

Logs

Nothing shows up in the logs when this happens.

Images

[aguilaair]

Duplicate of home-assistant/addons#1438, wasn't quite sure which repo to post it on (sorry)

[Kane610]

Most likely not related to the integration

[aguilaair]

They worked fine a few days ago... I'll continue investigating, thanks!

[ASNNetworks]

I upgraded to latest Deconz just now and have the exact same issue. Restarted Deconz an HA multiple times, all Zigbee devices just disconnect after a while (looks as if it crashes). Gonna restore to a snapshow as we speak...

I do remember Deconz having this exact same issue a few versions ago. It happened multiple times a day. An update fixed that then.

[aguilaair]

If it works please let me know, it's driving me crazy 🤣

[probot-home-assistant]

deconz documentation
deconz source
_{^{(message by IssueLinks)}}

[Fusionalex]

I have exactly the same problem since the latest Deconz update...

[Grodesh]

FWIW, I was having this problem since the latest update, but upon closer inspection of the logs, I saw that a firmware update was available for my Conbee II. All my devices are stable once again with the new firmware.

[Fusionalex]

For me the problem still exists sadly, I already have the latest firmware for my Conbee (not II)

[FirinBlanks]

I’ve tried updating the firmware and rebooting but each time I go back in to deconz it tells me the same firmware update is available. I can confirm that this issue happened prior to the HA core update but I’ve also now updated to the latest core in the hope that it was an incompatibility issue, I can sadly confirm that it wasn’t.

[aguilaair]

I've just updated the firmware from the Windows 10 application. If the issue is fixed I'll let you guys know

[aguilaair]

Well, that did not seem to help

[aguilaair]

For the time being I've migrated to ZHA, once this is sorted I'll probably move back

[ASNNetworks]

If it works please let me know, it's driving me crazy 🤣

I restored my snapshot (and that way downgraded Deconz) I did have some issues with losing connections after that still. Then rebooted the hardware (OS, so not just restart HA server). After that seems stable again like before (no lost connections for 48 hours).

[Kane610]

By account of previous comments this is related to latest deconz release. Closing this issue since it is not related to the integration per se.

[aguilaair]

Where should this issue be posted then?

[Fusionalex]

I dont get where else this should be posted then either.. The issue in the hassio-addons was closed too?

[Kane610]

Deconz github if it's related to an upgrade of deconz.

[aguilaair]

They told me it's related to HA, running it standalone has no issues

[ASNNetworks]

It is definitely related to HA in some way. I downgraded Deconz to 5.3.2 by restoring HA. I noticed a lot of issues still after that, after that I rebooted the OS entirely (not just HA Server) and also upgraded to 112. Zigbee is more stable now than with Deconz update, BUT all my devices still become unavailable for 15 seconds after a certain amount of time (sometimes after an hour, sometimes after 12 hours, very random). Unfortunately sometimes this happens right when I want to trigger something (which is rare, but still unreliable).

I had this exact issues a few HA versions ago. To say it's entirely Deconz is not correct, since I reverted the update.

[Kane610]

Ok then. Share logs showcasing the issue

[aguilaair]

That's the thing, the logs don't really give that much info:
HA Log:
2020-07-02 23:25:30 DEBUG (MainThread) [pydeconz.websocket] {"attr":{"lastannounced":null,"lastseen":"2020-07-02T21:25:30Z","manufacturername":"dresden elektronik","modelid":"RaspBee","name":"Configuration tool 1","swversion":"0x264a0700","type":"Configuration tool","uniqueid":"00:21:2e:ff:ff:05:d7:cf-01"},"e":"changed","id":"1","r":"lights","t":"event","uniqueid":"00:21:2e:ff:ff:05:d7:cf-01"} 2020-07-02 23:25:50 INFO (MainThread) [volvooncall.volvooncall] Updating 2020-07-02 23:25:54 DEBUG (MainThread) [pydeconz.websocket] Websocket starting 2020-07-02 23:25:54 DEBUG (MainThread) [pydeconz.websocket] Reconnecting to deCONZ in 15. 2020-07-02 23:26:09 DEBUG (MainThread) [pydeconz.websocket] Websocket starting 2020-07-02 23:26:09 DEBUG (MainThread) [pydeconz.websocket] Websocket running 2020-07-02 23:26:09 DEBUG (MainThread) [pydeconz.websocket] {"attr":{"lastannounced":null,"lastseen":"2020-07-02T21:26:09Z","manufacturername":"dresden elektronik","modelid":"RaspBee","name":"Configuration tool 1","swversion":"0x264a0700","type":"Configuration tool","uniqueid":"00:21:2e:ff:ff:05:d7:cf-01"},"e":"changed","id":"1","r":"lights","t":"event","uniqueid":"00:21:2e:ff:ff:05:d7:cf-01"}

Deconz log:
15:16:38:909 APS-DATA.indication request id: 188 -> finished 15:16:38:909 APS-DATA.request id: 188 erase from queue 15:16:38:909 ZDP status = 0x00 -> SUCCESS 15:16:38:910 ZDP Mgmt_Lqi_rsp zdpSeq: 45 from 0x00212EFFFF05D7CF total: 5, startIndex: 4, listCount: 1 15:16:38:910 * neighbor: 0x00158D00045D44DE (0x0B66), LQI: 255, relation: 0x01 rxOnWHenIdle: 0 =======AT 15:07:02 devices became unreachable===== 15:17:10:432 don't close database yet, keep open for 900 seconds 15:17:10:434 Remove websocket 172.30.32.1:51086 after error Unknown error 172.30.32.2 - - [03/Jul/2020:15:17:10 +0200] "GET /api/2C916F87F2/groups?_=1593778553416 HTTP/1.1" 200 1155 "https://rlc616egup6ikom710nd2uv6k3qg9m56.ui.nabu.casa/api/hassio_ingress/Y6CjyPLqCkHCWJS8gU3lwF1Iw86AR-NMdch8hfC_JaM/pwa/index.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0" 15:17:10:439 APS-DATA.indication srcAddr: 0x27e9, srcEp: 0x02 dstAddrMode: 2, profile: 0x0104, cluster: 0x0001, lqi: 111, rssi: -80 15:17:10:440 no button map for: RWL021 ep: 0x02 cl: 0x0001 cmd: 0x0A pl[0]: 021 15:17:10:440 ZCL attribute report 0x00178801080CDF75 for cluster: 0x0001, ep: 0x02, frame control: 0x18, mfcode: 0x0000

https://paste.ubuntu.com/p/jmBpXkNd8y/

[Kane610]

The deconz log is unnecessary, the part from the ha log you shared only show the raspbee device, which isn't relevant more than to show that I should probably ignore it completely. New logs please :)

[ASNNetworks]

At the exact time that all Zigbee devices were unavailable, I saw these lines. Perhaps something else causing stuff to hang, which then disconnects HA to Deconz? Configuration files are all ok though... Very strange.

2020-07-08 16:37:38 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.cpu_temperatuur is taking over 10 seconds
2020-07-08 16:37:43 ERROR (SyncWorker_7) [homeassistant.components.command_line.sensor] Timeout for command: cat /sys/class/thermal/thermal_zone0/temp

[Kane610]

Disable those and try again

[vukisz]

Ha version 2021.02.11 was quite bad. As this issue was happening way more often than before. Deconz was even crashing.
After ha update to 2021.03.4 no deconz crashes and way less unavailable episodes.
So it's really related to HA as no other changes was done during that period of a few weeks

[Kane610]

Ha version 2021.02.11 was quite bad. As this issue was happening way more often than before. Deconz was even crashing.
After ha update to 2021.03.4 no deconz crashes and way less unavailable episodes.
So it's really related to HA as no other changes was done during that period of a few weeks

You really have to clarify what you're talking about, this github repository is about HA Core and integrations. If deCONZ crashes it has (hopefully) nothing to do with the integration nor HA Core, but the addon or a standalone installation of deCONZ.

HA Core and HASSOS with addons are two different things.

This issue is convoluted enough as it is :)

[vukisz]

But I am also getting unavailable episodes. Far more often than crashes.
Sorry to hear, that this is hard to solve. What can we do as a users to help you?

[Mimiix]

@vukisz #37372 (comment)

[vukisz]

@vukisz #37372 (comment)

Thanks. I am quite inexperienced in admin field. Can such trace be setuped from HA?

[timovos]

I'm having similar issues (with HA 2021.3.3). All deconz devices become unavailable, and one second after that, they all become active again.

Happened twice this evening.

HA docker is on RPI4 with SSD (via USB hub). HA db is mariadb. Conbee II stick is on RPI3 with only running deconz docker (Conbee stick did not work on RPI4). Before yesterday, all was running on one RPI3 and I had similar issues (causing the sqlite DB to explode).

Upgrading to 2021.3.4 now. Will report back.

[update]
After three days of running, no unavailable reports anymore here.

[shoxcorp]

Since 2021.3.4 I have had no problems with deconz devices becoming unavailable.
Pre 2021.3.4 I had multiple instances daily. Not sure what changed, but please keep it like that :D

[yusijs]

I'm still seeing this with 2021.3.4 (that was when I started seeing the issue actually).

Really annoying :-/

[pdcmoreira]

I have an Intel NUC with an SSD, running everything on docker containers: HA + deCONZ with a ConBee II
I've been having these unavailable states and also occasionally HA is reporting open window sensors when they're closed (have to open and close the window to update the state).
Updated the HA and deCONZ containers today, to their latest stable versions. ConBee II's firmware was already up-to-date. Let's see if there is any difference.

All my sensors are at least within 5 meters of a router (Aqara and BlitzWolf plugs + Heiman siren).

This is a bit frustrating because after a considerable investment, I ended up with a somewhat unreliable/untrustworthy system. I wonder if the Zigbee reliability was always like this. Should've probably gone with Z-Wave instead.

[vukisz]

This is a bit frustrating because after a considerable investment, I ended up with a somewhat unreliable/untrustworthy system. I wonder if the Zigbee reliability was always like this. Should've probably gone with Z-Wave instead.

It's not a zigbee or hardware related. As it started on around last year June for everybody. And previously it was working :-)
I myself upgraded from rpi3 to rpi4 4gb ram with new original raspberry power supply.
No unavailable episodes since then (4 days). But my db got corrupted during migration. So I started with Ann empty. And it feels like bigger db had caused more unavailable episodes. So too early to tell

[yusijs]

I'm using a pi4, and my issues started fairly recently (after upgrading to latest version, was on 2021.1.something). It seems rather sporadic though; yesterday it was flipping between unavailable/on until ~3.30pm, then nothing, and then it went to shit again at 10.30pm and calmed down around 9.30am this morning, and now has been perfectly fine for 2 hours (see the image I attached).

I also went all in for zigbee, and I'm kinda regretting that now, because I've never seen anyone complain about similar issues with z-wave products 🙈

[timovos]

I have no issues anymore regarding this since HA 2021.3.4 (deCONZ on dedicated RPI3, HA on RPI4 with ssd).

It's mandatory to address this issue to either deCONZ or HomeAssistant.

About the z-wave comments: you should've done your research before making investments. Investments in z-wave would be more than double in comparison to zigbee. You chose the cheaper one, and these issues may come with that choice.

[pdcmoreira]

@timovos I've actually done a lot of research and concluded that the general consensus seemed to be that Zigbee required a bit more setup but would be as reliable as Z-Wave once consolidated.
Unfortunately there seems to be no detailed comparison between the two in the perspective of real life user experiences.

[Mimiix]

@pdcmoreira This issue is about the integration not working/dropping the connection. Concluded: just the connection between HA and deCONZ. If your zigbee stops working, then the problem is within deCONZ/your network.

The integration is not necessary for Zigbee to work properly, just for the automations.

[st3v3nFr]

Hy everybody.
I did have LOT OF ISSUES with deconz since an update of HA. like Crazy !

I tried to update deconz (HA integration) AND deconz firmware of the usb CONBEE 2 to their last versions.
Then everything was going even more crazy.

Then i decided to update / flash back these :

• CONBEE 2 firmware to : 26650700 ( realy this was the only one that was working with HA ).
• updated HA to core-2021.3.4
• updated deconz ( HA intégration ) to 6.7.2 ( not willing to try last update, guess you 'll understand why )

With these settings, i don't have anymore issues, and everything is working like a charm ( no deconnection, etc ... ).

Clearly, it's an ISSUE between HA AND DECONZ, but can't figre out what both deconz and HA are missing too.

Hope my config could help some of you.
St3v3n

[pdcmoreira]

@Mimiix If/when I get unavailable sensors again, I will fetch them through the deCONZ API and see what's reported (state/reachable/lastseen). If that's all OK, then it means my issue is about the integration and not the network.
I'll report back.

[yusijs]

I tried deleting my homeassistant db file, and that seems to have helped as it has been stable for 2-3 hours now. Will report back if it regresses. My db file was >1gb

[timovos]

I think it has something to do with the integration losing connection to deCONZ for whatever reason. When I make my deCONZ raspberry pi unavailable (pulling out the ethernet), I get (as expected) same behavior.

Also, things can be related to the hardware (using usb stick as storage instead of ssd), or interference on usb 3.0 interface (ssd disk can interfere badly).

I encountered these issues on a RPI3 with OS running on usb stick. HA db got large and then there were problems.

Try moving the db from file (sqlite) to mysql / mariadb.

[Kane610]

I think it has something to do with the integration losing connection to deCONZ for whatever reason. When I make my deCONZ raspberry pi unavailable (pulling out the ethernet), I get (as expected) same behavior.

Yes that is the symptom, the cause of this and if it is one or multiple issues are not determined

[yusijs]

Just checking back in, deleting the db has helped tons for me. My guess is that home assistant was struggling to keep up due to load, causing it to fail on deconz status checks marking the zigbee devices as unavailable.

In other words, my previous comment that zigbee is bad is void. For now. :D

[Kane610]

Hey I've been doing some refactoring of the retry mechanism, I've done some smaller changes to try to reconnect to deCONZ prior to marking anything as unavailable. I've also prolonged the heart beat time out to 60 seconds from 15 seconds. I hope this will be a part of the 2021.04 release, if it gets merged in time I hope someone who is affected by this issue can try it out.

There might also be a timing aspect that needs to be tweeked here.

This is more of a work around to not mark devices unnecessarily as unavailable rather than fixing the actual disconnect problem, but since it's only a few people having this issue thats gonna be hard.

[corvy]

I just got this problem one one of my setups. Tried all updates to firmware, hass and deconz. Been unstable for weeks with the exact same symptoms. One person suggested it could be network related. Therefore I checked my WiFi. Found that the channel I was using was noisy. Charged the channel and rebooted my AP (for good measure) and not a single incident since. My next move will be to wire my raspberry with a cat6 ethernet cable.

My suggestion is that everyone having this issue tries to wire the network to see if the problem goes away.

[pdcmoreira]

As far as I've noticed, I didn't get any unavailable sensors since my comment above, which I just realised was a month and a half ago. Hopefully the issue is gone. Meanwhile I added some more sensors too.

[Kane610]

Seems everyone is happy with the work around. Closing this issue prior to it celebrating its first birthday. 🎉

[wifiuk]

I'm getting this issue not WiFi interference, using HA as a VM

[Kane610]

Please open a new issue, reference this issue if you'd like.