[Bug] Devices reports No response status after upgrading to v1.10.0

[bulivlad]

Describe the bug
Just updated the plugin from v1.9.2 where everything was looking good to v1.10.0 and suddenly all devices in Home.app controlled by this plugin started reporting No Response status, however everything is reported as available in Zigbee2MQTT UI. I also tried to add "ignore_availability": true, as a default as well as for every accessory but nothing changed

Related devices
Devices failing to répond are Livolo T0001 switches & plugs covered here and here

To Reproduce
Steps to reproduce the behavior, for example:

1. Start Zigbee2MQTT
2. Start Homebridge
3. Upgrade the plugin to v1.10.0 & restart Homebridge
4. Switch Homebridge to debug mode

Homebridge logs shows

[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Received devices...
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0
[12/15/2022, 3:22:39 PM] [zigbee2mqtt] Update "error" status for characteristics of <accessory_name_here> to 0

Zigbee2MQTT logs

debug 2022-12-15 18:09:50Retrieving state of '[FRIENDLY_NAME]' after reconnect
debug 2022-12-15 18:09:50Received Zigbee message from '[FRIENDLY_NAME]', type 'raw', cluster 'genPowerCfg', data '{"data":[122,209,224,179,54,35,0,75,18,0,7,1,0,3,0,11],"type":"Buffer"}' from endpoint 6 with groupID 0
info 2022-12-15 18:09:50MQTT publish: topic 'zigbee2mqtt/[FRIENDLY_NAME]', payload '{"linkquality":78,"state_bottom_left":"OFF","state_bottom_right":"OFF","state_left":"OFF","state_right":"OFF"}'

Expected behavior
Devices are being responsive in Home.app

MQTT messages
If possible (and applicable), please provide the following:

• The device entry from zigbee2mqtt/bridge/devices (note: this is different from zigbee2mqtt/bridge/config/devices)
• Status update from zigbee2mqtt/[FRIENDLY_NAME]
• Messages published by this plugin that might be related to your issue (can also be seen in the homebridge logs)

Versions used
Please provide the version of the following pieces of software:

• This plugin - v1.10.1
• Homebridge - v1.6.0
• Zigbee2MQTT (can be found in MQTT topic zigbee2mqtt/bridge/info) - running in docker v1.28.4
• Homebridge Config UI X (if applicable) - v4.50.2

Other info
Tried multiple combinations of Zigbee2MQTT availability true/false and Homebridge-z2m plugin ignore_availability true/false but none seems to fix the issue.
It seems that restarting Zigbee2MQTT while Homebridge is running may fix the issue for a while then it's back on No response status

Config snippet

        {
            "mqtt": {
                "base_topic": "zigbee2mqtt",
                "server": "mqtt://localhost:1883",
                "reject_unauthorized": false,
                "keepalive": 60,
                "version": 4,
                "disable_qos": false
            },
            "log": {
                "mqtt_publish": "debug"
            },
            "defaults": {
                "exclude": false
            },
            "exclude_grouped_devices": false,
            "devices": [
                {
                    "id": "0x00124b0023503994",
                    "ignore_availability": true,
                    "excluded_endpoints": [
                        "bottom_right"
                    ]
                },
                {
                    "id": "0x00124b002336c92d",
                    "ignore_availability": true,
                    "excluded_endpoints": [
                        "right",
                        "bottom_left",
                        "bottom_right"
                    ]
                }
            ]
        }

[FWeinb]

I do have the same issue running the 1.11.0-beta.0. On the zigbee2mqtt dashboard everything is online and can be controlled but HomeKit is reporting No Response. I recently updated my Home to iOS 16.2 HomeKit as well, no idea if that could be related.

[itavero]

I think I've noticed this myself now occasionally as well. Most of the time it only shows the devices unavailable that are also unavailable in Zigbee2MQTT (e.g. some sensors that still need fresh batteries or a light that has been physically turned off). A few times I've seen almost all accessories marked as "No Response".

Unfortunately, I still need to figure out a workable way of getting the debug logs of my local installation. (homebridge-unifi-protect, which I also have running on there, puts a lot of information in the debug logs making them hard to view 😞 ).

In my test though, disabling worked fine. I don't remember if changing that option also ignores the global Zigbee2MQTT online/offline state. I should look into that a bit more.

[zegl]

I've been hit by this myself as well. It's a bit hard to track down the problem fully, but it seems to be related to the loss of connectivity to Zigbee2MQTT.

If I restart zigbee2mqtt, some devices end up being incorrectly marked as "No Response" after the reboot. This happens both with the hb-z2m availability feature enabled or disabled.

I suspect that some code-paths are hit to report availability even if the feature has been disabled.

Another issue, that seems to be related to the availability reporting. Is that after upgrading to v1.10.0, homebridge-z2m will send (seemingly random?) commands to z2m after a restart of z2m. And turns devices on/off without any user interaction.

[1/9/2023, 1:43:48 PM] [zigbee2mqtt] Zigbee2MQTT is OFFLINE!
[1/9/2023, 1:43:48 PM] [zigbee2mqtt] adaptive_lighting: Closet: skipped color_temp (new: 181; old: 181
[1/9/2023, 1:44:40 PM] [zigbee2mqtt] Zigbee2MQTT is ONLINE
[1/9/2023, 1:44:40 PM] [zigbee2mqtt] Zigbee2MQTT availability feature is enabled globally: 'false'
[1/9/2023, 1:44:41 PM] [zigbee2mqtt] Received devices...
....
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"ON"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Pending data: {"state":"OFF"}
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x003c84fffeb1547a/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x00178801082ec23c/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x003c84fffeb15921/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x003c84fffeb6e38b/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x9035eafffe62888f/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x0017880108378b7d/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x588e81fffe8fb405/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x588e81fffe768eef/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x0017880108a1a40c/set': '{"state":"OFF"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x0017880108faec55/set': '{"state":"ON"}'
[1/9/2023, 1:44:55 PM] [zigbee2mqtt] Publish to 'zigbee2mqtt/0x001788010b91f5c1/set': '{"state":"OFF"}'

Any idea where this comes from?

I've tried to disable all code paths related to availability reporting (just hacked something together), and so far it seems to fix both issues from above.

[itavero]

@zegl I reckon that's something either HomeKit or Homebridge is doing. Not something in the plugin it self at least.

I've also noticed some inconsistent behavior of this feature, so I will try to make an update that has it disabled by default (and also allows you to completely disable it, so also when Z2M is offline).

[itavero]

Just pushed a new beta version where the Z2M online state/MQTT broker connection can be ignored, as well as some changes to the logging.

[fuzzy01]

I do have the same issue running the 1.11.0-beta.1.
Random I have devices in HomeKit reporting No Response.
On the zigbee2mqtt dashboard everything is online, and on the Homebridge Accessories tab also everything is online. If I turn on the device (lamp) on the Accessories tab in Homebridge the device recovers in the Home app. After that it is online and works in the Home app.

This is the recovery in the logs:

[1/13/2023, 10:47:43 PM] [homebridge-z2m] Pending data: {"state":"ON"} [1/13/2023, 10:47:43 PM] [homebridge-z2m] Publish to 'zigbee2mqtt/0xccccccfffe620b05/set': '{"state":"ON"}' [1/13/2023, 10:47:43 PM] [homebridge-z2m] Handled device update for Night stand right: {"brightness":254,"linkquality":156,"power_on_behavior":"off","state":"ON","update":{"installed_version":587806257,"latest_version":587806257,"state":"idle"}}

[itavero]

In the latest release (v1.11.0-beta.2), I put the code that actually marks an accessory as unreachable behind a feature flag.

[YellowNest]

I had a weird issue last night that I would like to share with you. Since the adaptive lighting feature is all messed up and not working for me anymore, I had to reset all of my bulbs just so I can get rid of the adaptive lighting symbol among the color picker. I had everything up and running just fine except that availability issue every now and then but I did a reset of a Philips Hue Go that I have and suddently all my devices went offline in the home app but Zigbee2mqtt worked just fine. I tried to restart zigbee2mqtt, still the same. I restarted Homebridge several times and still nothing so I restarted the raspberry pi… still nothing. The only thing that ”fixed” the issue was to restart my router and all devices came online.

[stale]

It appears that this issue did not have an update in quite some time. Please check if you can provide any additional information to help resolve this issue. If there isn't any activity in the next two weeks, this issue will be closed automatically. Thank you for your contributions!

[Avamander]

I've had it enabled for a while and for some reason all my devices suddenly ended up as "Not Available". Restarted every component, no dice. Seeing mentions of availability in the changelog I switched to v1.11.0-beta.3, but that didn't improve it much - all devices were still offline except briefly. I then just had to remove the feature flag because it seems broken in the current release.

Zigbee2MQTT itself thinks all the devices are online, I can toggle them and see their readings update. Devices appear online briefly if I change brightness, but not in other cases (quick toggle doesn't actually trigger anything if a device is marked not responsive by z2m it seems). HomeKit fetching current status also doesn't actually force Z2M to fetch it from (Zigbee2)MQTT. But in general I don't know where Z2M is getting the info that everything is offline.

Is some kind of "unknown" (or stale) status being considered as "offline"? I'd actually really like seeing if a battery runs dry or if a lamp has been unplugged.