Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Versions after 2.05.85 Sensor State Changes Lost to Home Assistant #3707

Closed
majorsl opened this issue Nov 19, 2020 · 91 comments
Closed

Versions after 2.05.85 Sensor State Changes Lost to Home Assistant #3707

majorsl opened this issue Nov 19, 2020 · 91 comments

Comments

@majorsl
Copy link

majorsl commented Nov 19, 2020

Describe the bug

I use deCONZ with the Home Assistant integration: https://www.home-assistant.io/integrations/deconz/

Upgrading to the beta 2.6.0 of deCONZ stops all my sensor states from changing in Home Assistant, but the states do still show in Phoscon.

I tried removing/re-adding the integration with no change. Of note, when I re-added the integration I had a device shown by HA that is "null" just () with no data or information. This does not appear in Phoscon or deCONZ. Erroneous data being sent by the API?

Steps to reproduce the behavior

If the problem is reproducable, list the steps here:

  1. Use Home Assistant
  2. Install a version after 2.05.85
  3. Watch your sensors no longer work in Home Assistant

Expected behavior

Sensors toggle states in Home Assistant.

Screenshots

Environment

  • Host system: Raspberry
  • Running method: Marthoc Docker container
  • Firmware version: (26660700)
  • deCONZ version: > greater than 2.05.85
  • Device: ConBee II
  • Do you use an USB extension cable: yes
  • Is there any other USB or serial devices connected to the host system? no

deCONZ Logs

Additional context

Has been working perfectly with Home Assistant up to this point. I have reverted away from the beta, but will be happy to try again if anything is found or a fix is available.

@Kane610
Copy link

Kane610 commented Nov 19, 2020

It would be good if you could share debug logs from hass. Enable debug per deconz integration instructions and share them here

@Kane610
Copy link

Kane610 commented Nov 19, 2020

Typically when no state changes are observed it's the websocket not setting up properly

@majorsl
Copy link
Author

majorsl commented Nov 20, 2020

We have a thread going on reddit, someone did few tests and it seems the last version that is working is 2.05.84 (I think that is 85). I'm about to revert to that if I can figure out how to use a local image I still have - never done that. https://www.reddit.com/r/homeassistant/comments/jx7lxo/0118_and_deconz_phoscon_issues/

@majorsl
Copy link
Author

majorsl commented Nov 20, 2020

Confirmed: 2.05.85 is the last version that worked. Stepping back through the Docker container versions one by one to it finally started sending sensor states to Home Assistant again.

@majorsl majorsl changed the title Beta 2.6.0 Sensor State Changes Lost to Home Assistant Versions after 2.05.85 Sensor State Changes Lost to Home Assistant Nov 20, 2020
@Mimiix
Copy link
Collaborator

Mimiix commented Nov 20, 2020

I have no issues on 2.60.0 with HA and deCONZ. Running deCONZ in a seperate VM.

@manup
Copy link
Member

manup commented Nov 20, 2020

I've looked in the GIT diff between v2.5.85 and v2.5.86, the only REST-API difference here seems to be:

image

Which is related to ELKO Super TR:

{
  "config": {
       "temperaturemeasurement" : "air sensor"
  }
}

Could it be that?

@SwoopX
Copy link
Collaborator

SwoopX commented Nov 20, 2020

@manup Already discussed with @Kane610 . That's why I updated the API documentation and added when something new was introduced. Cannot be sure however, that I caught everything.

I guess a bit more was introduced. We also have one issue where button_maps.json appears to be missing.

@manup
Copy link
Member

manup commented Nov 20, 2020

According to the diff there seems to only the above and one error message which is different in the REST-API output.

image

Otherwise I can't see anything 🤔

@SwoopX
Copy link
Collaborator

SwoopX commented Nov 20, 2020

A bit more. Besides the config part already mentioned, also this as state (both applicable for thermostats only)
grafik

#3329

@manup
Copy link
Member

manup commented Nov 20, 2020

So the question is if these devices are paired in the problematic installations?

@manup
Copy link
Member

manup commented Nov 20, 2020

The PR seems to only add new attributes but doesn't change existing REST-API attributes. This hopefully shouldn't break a client.

@Kane610
Copy link

Kane610 commented Nov 20, 2020

The deconz integration creates standard sensors if there isn't added explicit support for the sensor so it shouldn't break but it might not expose the best information about it

@indomus
Copy link

indomus commented Nov 20, 2020

Same problem here. #3645

@majorsl
Copy link
Author

majorsl commented Nov 20, 2020

Same problem here. #3645

My goal yesterday was to get up and running as quickly as possible. What can I provide that hasn't been discussed in the above issue to try to help?

I'm using Docker so it isn't too much of an issue to create latest or stable to get some information and then I can switch back to a working environment.

Fortunately, I don't have many devices and no repeaters yet, so if it is a device causing it maybe changes there can be a clue. I have these: SNZB-04 (6), Xiaomi Aqara Temp/Humid Sensor (1), CentraLite 3323-G (3)

@adrha
Copy link

adrha commented Nov 20, 2020

Hey @manup first of all thank you for your great support!

I'm having the same issues as others here in this thread but recently figured out that the .85-version is the first bugy-release (and not 86 as reported earlier).
For me a roll-back to 84 fixed it.
The interesting point is, that others in this thread are using Aqara Temp/Humidity sensors, which i also use and they are having issues as well.

@Florian-Schmidt
Copy link

Same issue here: Suddenly HA automations stopped working during the last days.

Tried to update deconz and ConBee II to:
deconz: 2.06.00
firmware: 26670700

Every device was not reachable, over the day some devices came back. Now downgraded back to.
deconz: 2.05.86
firmware: 26660700

Still the same, updated deconz again to 2.06.00 but no changes. Also tried an older version 2.05.73 but still the same devices available.

Unfortunately I don't have a 2.05.84 available.

@Florian-Schmidt
Copy link

Ok, build marthoc/deconz container locally and now running.

deconz: 2.05.84
firmware: 26650700

But still the same, there are some connections shown in VNC viewer but many devices are missing and the zigbee network doesn't build up.

@Florian-Schmidt
Copy link

Ok, now the first lights are coming back again and zigbee network grows up slowly...

Anyone an idea how to speed up the reconnect of known devices?

@SwoopX
Copy link
Collaborator

SwoopX commented Nov 23, 2020

@Florian-Schmidt Please do not hijack other issues. You mentioned that none of your devices are reachable, which is a completely different topic so for this, please raise your own issue.

To answer your question: You can press 0 (zero) in deconz GUI while selecting a routing node, or F5.

@Florian-Schmidt
Copy link

@SwoopX Thanks for you answer. Sorry, thought this could be helpful as these releases seem to have several similar issues...

Actually Zigee network seems to continue growing up more and more. Going to wait some more days and hope to reach complete network again. In case of getting disappointed by incomplete network, I'll open new ticket.

@stale
Copy link

stale bot commented Dec 18, 2020

As there hasn't been any response in 21 days, this issue has been automatically marked as stale. At OP: Please either close this issue or keep it active It will be closed in 7 days if no further activity occurs.

@stale stale bot added the stale label Dec 18, 2020
@majorsl
Copy link
Author

majorsl commented Dec 18, 2020

As there hasn't been any response in 21 days, this issue has been automatically marked as stale. At OP: Please either close this issue or keep it active It will be closed in 7 days if no further activity occurs.

Still an issue.

@stale stale bot removed the stale label Dec 18, 2020
@adrha
Copy link

adrha commented Dec 28, 2020

Has someone got it to work from you guys?
I'm still running on .84 version. The latest version does still not work with HomeAssistant.
Is there a way to export the device-config and import it on the new setup without repairing all nodes?

@majorsl
Copy link
Author

majorsl commented Dec 28, 2020

Has someone got it to work from you guys?
I'm still running on .84 version. The latest version does still not work with HomeAssistant.
Is there a way to export the device-config and import it on the new setup without repairing all nodes?

Not that I'm aware. I was going to try it again today based on #3977 which seems similar. I had not waited hours for the sensors to come back so i thought I'd give that a go.

I agree - rebuilding from scratch just over an upgrade isn't optimal, but the later builds have some device support I want to use. :(

@SwoopX
Copy link
Collaborator

SwoopX commented Dec 31, 2020

@s0ftice No problem. Happy to check whatever I can.

@Kane610 Honestly, what can I do more? Tell me what to do and I do it. I reconfigured my system and the screenshots evidence all is working with the small difference that I do not use HA. And this seems to be toe only deviating detail here. Again, I'm still open it could be anything on the deconz end... However, from my perspective, air is getting a bit thin for the argument "this is on deconz side. If it is with getting devices available and time stamp issue it can't be hass".

Admit, haven't changed the time zone in deconz yet. Will check it now.

@adrha
Copy link

adrha commented Dec 31, 2020

That's funny with the time-zone.
I remember, everything worked after an hour or two. I live in Switzerland and we actually have UTC+1h :)
I've checked the time in both containers (i'm running it dockered) and they both have the localtime as time.

@Kane610
Copy link

Kane610 commented Dec 31, 2020

@s0ftice No problem. Happy to check whatever I can.

@Kane610 Honestly, what can I do more? Tell me what to do and I do it. I reconfigured my system and the screenshots evidence all is working with the small difference that I do not use HA. And this seems to be toe only deviating detail here. Again, I'm still open it could be anything on the deconz end... However, from my perspective, air is getting a bit thin for the argument "this is on deconz side. If it is with getting devices available and time stamp issue it can't be hass".

Admit, haven't changed the time zone in deconz yet. Will check it now.

What I mean is that I don't care about timestamps or time zones or anything on hass side. All I get are events from Deconz, if I don't get events but websocket is setup it should be something on deconz side.

@majorsl
Copy link
Author

majorsl commented Dec 31, 2020

Mine is working. Party due to a Time Zone issue and mostly due to myself.

It seems versions before the ones noted in this ticket didn't care (or possibly use) the Docker TZ env variable. In looking to change my Docker to use Europe/London to test, I noticed I had America/New York... which is a typo on my part; should be America/New_York

As such, Docker was not using it nor throwing an obvious error. Whenever I looked at the logs, deCONZ, Docker, etc all the times were shown to be correct so I just never thought about that one variable.

With that said, there does appear to be something with the Time Zone and sensors for those that may not have a Time Zone configured exactly as expected.

Note to self: get in the habit of using America/Boston instead.

@ebaauw
Copy link
Collaborator

ebaauw commented Dec 31, 2020

Wondering if Homebridge hue is affected then. @ebaauw, can you answer that?

I haven't seen any issues similar to this with Homebridge Hue and deCONZ. Of course, Homebridge Hue, like Phoscon, uses polling next to the web socket (because that's the only way to work with a Hue bridge). However, looking at my logs, the sensor state changes are coming in through the web socket.

As I suggested in Discord, can you please try another web socket client in parallel to HASS, to see if the problem is with generating the events, or sending the events. Homebridge Hue comes with a dc_eventlog utility, but anything similar would do. Also, please check the deCONZ log: it should issue a message for each web socket event sent to each client.

@Florian-Schmidt
Copy link

@Florian-Schmidt, I think yours is different, as you described it suddenly stopped working (or was that related to a restart of deCONZ too?)

@s0ftice It was related to an update. Actually running 2.08.00.

@cmuellner
Copy link

cmuellner commented Jan 3, 2021

I was running into this issue as well today.
My client application is home-assistant running on an external machine.
My deconz instance runs as a container on a RasPi3/Fedora/Podman host system.
My RF interface is a Conbee II.

Observations:

  • Client system could query all devices.
  • Lights (Philips Hue, OSRAM Smart+ Plug) could be controlled by the client system.
  • Sensor (Xiaomi Aqara multi-sensor) updates were forwarded.
  • Switch (TRÅDFRI wireless dimmer, TRÅDFRI remote control, Mi Magic Cube, Hue Smart Button) updates were detected and logged in deconz, but never forwarded to the client.

I set the timezone today on the host machine to Europe/Vienna and restarted the host.
However, I did not expect this to be related to the observations above.
After debugging for a couple of hours what happened, I found this issue (and the workaround in #3707 (comment) from @s0ftice).
I can confirm that setting the host timezone to Europe/London ("timedatectl set-timezone Europe/London") and restarting the container workarounds the issue.

I don't know how the timezone information is provided by Podman to the deconz container, but I guess more or less in a very similar way as Docker does.

@Metus88
Copy link

Metus88 commented Jan 3, 2021

same problem for me...

@JohCo-hassio
Copy link

Same problem for me.

@Mimiix
Copy link
Collaborator

Mimiix commented Jan 4, 2021

For anyone willing to add a #metoo : Please do not. This clutters communication and my mailbox.

It seems to be affected by timezone. I am currently checking if this is something we should fix or HA should fix. Iam checking with Manuel.

@manup
Copy link
Member

manup commented Jan 4, 2021

By looking at the code I think the problem is that the push logic depends very strictly on correct time to be set — at all times.
Even when the time settings are corrected, the former timestamps of sensor states are carried on.

This is not unique to HA or any client. Having incorrect time settings can happen for various reasons, and the bigger problem is that even correcting them won't fix the problem immediately until all timestamps have been refreshed.

I think to protect the push logic from such issues it needs to be completely independent from any absolute timestamps.

@s0ftice
Copy link

s0ftice commented Jan 4, 2021

Thanks @manup. I don't think that's the main issue to be honest. My HA VM as well as my RPi with deCONZ use ntp to sync their time, so they are set correctly and were also set to the correct time zone - now my RPi's time zone is set to GMT (wrongly) to make it work.

Having to wait a full 8h (exactly my UTC offset) after every deCONZ restart for the sensor events to be pushed out, indicates that it's 100% related to the time zone setting (for users who are affected - still no clue why not everyone is facing this problem). And after those 8h of wait time (exact to the second!), it starts working reliably and continuously (until the next deCONZ restart).

@Mimiix yes, the #metoo clutter the Inbox, but also gives a good indication of this being a slightly broader issue.

Thanks again for looking into this.

@Florian-Schmidt
Copy link

Strange yesterday evening suddenly the lights stopped working again. I thought that both containers (deconz and homesssistant) are running with correct timezone.

Now set environment variable to use TZ=Europe/Berlin for sure but still no chance to get the lights controllable again. I'll check in hours to see if there's an relation to the timezone offset.

@Florian-Schmidt
Copy link

@s0ftice Seems as you are right. Yesterday one hour later the lights were controllable via HA again but later they were not. Perhaps there are more reasons as only the timezone.

Docker logs show two messages.

During standby:
0x00158D000418D399 error APSE-DATA.confirm: 0xE1 on task

When pushed HA button:
delay sending request 213 dt 1 ms to 0X00158D000418D399, ep: 0x01 cluster:0x0006 on Air: 1

@Mimiix
Copy link
Collaborator

Mimiix commented Jan 17, 2021

@s0ftice Seems as you are right. Yesterday one hour later the lights were controllable via HA again but later they were not. Perhaps there are more reasons as only the timezone.

Docker logs show two messages.

During standby:
0x00158D000418D399 error APSE-DATA.confirm: 0xE1 on task

When pushed HA button:
delay sending request 213 dt 1 ms to 0X00158D000418D399, ep: 0x01 cluster:0x0006 on Air: 1

The 0xE1 is a error message. It basically means you are suffering from interference issues.

@Florian-Schmidt
Copy link

@Mimiix What should this mean exactly? It stopped working with newer Deconz versions and downgrades don‘t help. Two days ago the lights worked and without changes stopped to react.

VNC shows connections between the Zigbee devices.

I‘ve got no idea where this is actually coming from...

@Mimiix
Copy link
Collaborator

Mimiix commented Jan 17, 2021

Are you using a extension cable? Interference comes from USB3. 0 or wifi.

@Florian-Schmidt
Copy link

Are you using a extension cable? Interference comes from USB3. 0 or wifi.

Added an short extension cable much earlier as the problems started. This seemed to be an advantage in other threads. USB 3.0 ports left earlier because of experiences from other users and strange situations.

Honestly, appreciate every idea and possible solution but this should be the reasons to stop working suddenly... We are in 2021 now and these reasons shouldn't make our life harder, they should improve.

@majorsl
Copy link
Author

majorsl commented Jan 17, 2021

Are you using a extension cable? Interference comes from USB3. 0 or wifi.

Added an short extension cable much earlier as the problems started. This seemed to be an advantage in other threads. USB 3.0 ports left earlier because of experiences from other users and strange situations.

Honestly, appreciate every idea and possible solution but this should be the reasons to stop working suddenly... We are in 2021 now and these reasons shouldn't make our life harder, they should improve.

I don't know if this will help, but it is worth asking. Do you see any new 802.11b/g/n networks around you? Did you change your own? If you can, get a wifi scanner for your phone. I use WiFi analyzer on my Android phone.

I had a conflict with Wifi and I had to move my Zigbee over to channel 25. I found this helpful to sort out where in the spectrum things were: https://www.metageek.com/training/resources/zigbee-wifi-coexistence.html

@Mimiix
Copy link
Collaborator

Mimiix commented Jan 17, 2021

Are you using a extension cable? Interference comes from USB3. 0 or wifi.

Added an short extension cable much earlier as the problems started. This seemed to be an advantage in other threads. USB 3.0 ports left earlier because of experiences from other users and strange situations.

Honestly, appreciate every idea and possible solution but this should be the reasons to stop working suddenly... We are in 2021 now and these reasons shouldn't make our life harder, they should improve.

Blame Intel on the interference issues. Also: the ssd probsbly needs shielding.

@Florian-Schmidt
Copy link

I don't know if this will help, but it is worth asking. Do you see any new 802.11b/g/n networks around you? Did you change your own? If you can, get a wifi scanner for your phone. I use WiFi analyzer on my Android phone.

I had a conflict with Wifi and I had to move my Zigbee over to channel 25. I found this helpful to sort out where in the spectrum things were: https://www.metageek.com/training/resources/zigbee-wifi-coexistence.html

Nothing changed in my own WiFi and networks around us are not that much. But perhaps WiFi issues are an idea to check and I'm going to check your link...

@s0ftice
Copy link

s0ftice commented Jan 18, 2021

@Florian-Schmidt I think yours is different. The issue described here is not about unavailable lights directly, it's about sensors and battery-based switches (that might be triggering lights). Once they become available after the timezone delta to UTC+00 has been reached, they are working reliably. You said "later they were not": this is not the case for us facing this issue (unless we restart deCONZ).

@Florian-Schmidt
Copy link

@s0ftice I'm going to create a new ticket my issues are changing daily at the moment. Actually it seems not to use expected timezone and suddenly web interface is rejecting password.

manup added a commit to manup/deconz-rest-plugin that referenced this issue Jan 24, 2021
* Prevent duplicated Websocket push events;
* Fix skipped Websocket push events when times stamps don't align due system time changes.

Issue: dresden-elektronik#3707
@Mimiix Mimiix added the stale label Feb 24, 2021
@github-actions
Copy link

github-actions bot commented Mar 5, 2021

As there has not been any response in 28 days, this issue will be closed. @ OP: If this issue is solved post what fixed it for you. If it is not solved, request to get this opened again.

@github-actions github-actions bot closed this as completed Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests