Devices are disappearing from Device list.

[erzhan89]

What happened?

Devices 4 pc. (RTV https://www.zigbee2mqtt.io/devices/GS361A-H04.html ) disappearing from Device list. after 15-20 minutes. Before they worked without issues.

What did you expect to happen?

To have stable connection.

How to reproduce it (minimal and precise)

No response

Zigbee2MQTT version

1.28.4

Adapter firmware version

6.10.3.0 build 297

Adapter

ITead Sonoff Zigbee 3.0 USB Dongle Plus V2 model "ZBDongle-E"

Debug log

No response

[cloudharps]

Same problem. Devices dissapearing

[terierul]

I have the same issue after updated to 1.28.4
downgraded to 1.28.2 and everything back to normal

[Koenkk]

Are you all using the EZSP adapter?

[erzhan89]

I have the same issue after updated to 1.28.4 downgraded to 1.28.2 and everything back to normal

I did same. so far works fine.

[cloudharps]

Are you all using the EZSP adapter?

Yes.

[terierul]

Are you all using the EZSP adapter?

conbee 2/deconz

[rhvs]

Are you all using the EZSP adapter?

Slaesh stick

[brunolinsalves]

Same thing here. I can't readd the device that left network. What should I do ?
I already updated to version 1.28.4-1 to test and the problem was not solved...

My device was ZM-L03E-Z(TS0003)

[kilski]

@erzhan89 still works with 1.28.2?

[erzhan89]

@erzhan89 still works with 1.28.2?

Yes. Since I deployed v1.28.2 there is no issue.

[Koenkk]

Can someone provide the debug log when a device disappears?

See https://www.zigbee2mqtt.io/guide/usage/debug.html on how to enable debug logging.

[brunolinsalves]

I wasn't been able to simulate the error to post the logs here. But, I tryed again, in version 1.28.4-1 and was successful. The devices was added as expected and didn't left the network anymore.

Thanks

[terierul]

Can someone provide the debug log when a device disappears?

See https://www.zigbee2mqtt.io/guide/usage/debug.html on how to enable debug logging.

Hello @Koenkk . Here you have the file

log.txt

The screen in IE browser

LE: the amazing part is that immediately after I took the printscreen from above the list reappeared....

LE2: I did as restart of the RPI4 where the HASS is installed and the error come back. Here is the new log after restart

log.txt

BTW why the issue is named: "Devices are disappearing from Device list" because in Chrome browser this is what appears:

In IE is 502: Bad Gateway

LE3: downgraded to 1.28.2 and everything back to normal . the new log
log.txt

[Koenkk]

It looks to me that Zigbee2MQTT is not running, can you check the log page of the addon?

[terierul]

I think this is the issue (check the LE2 comment file- log after restart)
The last lines in the log. Something get wrong and forced Zigbee2MQTT to stop

debug 2022-12-07 08:52:16: lumi.vibration.aq1: unknown key 6 with value 0,5
debug 2022-12-07 08:52:16: lumi.vibration.aq1: unknown key 153 with value 107
debug 2022-12-07 08:52:16: lumi.vibration.aq1: Processed data into payload {"voltage":3005,"battery":100,"device_temperature":21,"power_outage_count":8}
debug 2022-12-07 08:52:16: lumi.vibration.aq1: Processed data into payload {"voltage":3005,"battery":100,"device_temperature":21,"power_outage_count":8}
info 2022-12-07 08:52:16: MQTT publish: topic 'zigbee2mqtt/Sensor vibracion 1', payload '{"action":null,"angle":9,"angle_x":0,"angle_x_absolute":90,"angle_y":-1,"angle_y_absolute":91,"angle_z":-89,"battery":100,"device_temperature":21,"linkquality":255,"power_outage_count":8,"sensitivity":"high","strength":2,"vibration":false,"voltage":3005}'
debug 2022-12-07 08:52:20: Saving state to file /config/zigbee2mqtt/state.json
info 2022-12-07 08:52:20: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'offline'
info 2022-12-07 08:52:20: Disconnecting from MQTT server
info 2022-12-07 08:52:20: Stopping zigbee-herdsman...
info 2022-12-07 08:52:22: Stopped zigbee-herdsman
info 2022-12-07 08:52:22: Stopped Zigbee2MQTT

[Koenkk]

Interesting, can you provide the herdsman debug log of this?

See https://www.zigbee2mqtt.io/guide/usage/debug.html on how to enable the herdsman debug logging. Note that this is only logged to STDOUT and not to log files.

[terierul]

Hello @Koenkk
Sorry have been out of town for some days
please give me some more instructions how to extract STDOUT. Is the first time
Thanks

[jerrm]

https://www.zigbee2mqtt.io/guide/usage/debug.html

But it is a little easier now for the HA addon:

[terierul]

well I do not have this button in add-on. could you please activate the button and show me the yaml code please?

Later edit: was in documentation :)

[terierul]

Test 1 - you may jump directly to Test 2 because the problem seems to be there
@Koenkk after I have activated herdsman debug and install again 1.28.4 - 2 logs have been generated

Log 1
log (1) 2022-12-12.00-02-06.txt

Log2
log (2) 2022-12-12.00-02-35.txt

Look again at log 1

debug 2022-12-12 00:02:23: Saving state to file /config/zigbee2mqtt/state.json
info 2022-12-12 00:02:23: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'offline'
info 2022-12-12 00:02:23: Disconnecting from MQTT server
info 2022-12-12 00:02:23: Stopping zigbee-herdsman...
info 2022-12-12 00:02:24: Stopped zigbee-herdsman
info 2022-12-12 00:02:24: Stopped Zigbee2MQTT

and after that in log 2 which seems is the follow up of log1:

debug 2022-12-12 00:02:36: Loaded state from file /config/zigbee2mqtt/state.json
info 2022-12-12 00:02:36: Logging to console and directory: '/config/zigbee2mqtt/log/2022-12-12.00-02-35' filename: log.txt
info 2022-12-12 00:02:36: Starting Zigbee2MQTT version 1.28.4 (commit #unknown)
info 2022-12-12 00:02:36: Starting zigbee-herdsman (0.14.76)

5 min later I was able to enter in the device screen

Test 2 - restart the machine - the log after restart
log.txt

HASS started at 00:20 --> device screen empty

Finally at 00:30 - the list was populated with all the devices - FINAL LOG attached
log.txt

### CONCLUSION: in the end app starts. But was needing 10 min to do the initiation after HASS did - before 1.28.4 the initiation was in the same time with HASS

_HINT: I have some devices (some 220V smart plugs and switch modules) which are not plug in in the network for the moment but they are paired (Look in the log for the devices call: "Enchufe" or "Antena para Siren"). Because they are not plugged in this moment they need a long time to be discovered as offline

Eg with the device call Enchufe 2. Start configuring at 00:22:06 and ending at 00:23:07 with the message "Failed to configure 'Enchufe 2',"_

info 2022-12-12 00:22:06: Configuring 'Enchufe 2'
debug 2022-12-12 00:23:01: Received Zigbee message from 'Humidity sensor1', type 'attributeReport', cluster 'msTemperatureMeasurement', data '{"measuredValue":1800}' from endpoint 1 with groupID null
info 2022-12-12 00:23:01: MQTT publish: topic 'zigbee2mqtt/Humidity sensor1', payload '{"battery":100,"humidity":57.36,"linkquality":255,"power_outage_count":4,"pressure":937.7,"temperature":18,"voltage":3005}'
debug 2022-12-12 00:23:01: Received Zigbee message from 'Humidity sensor1', type 'attributeReport', cluster 'msRelativeHumidity', data '{"measuredValue":5739}' from endpoint 1 with groupID null
info 2022-12-12 00:23:01: MQTT publish: topic 'zigbee2mqtt/Humidity sensor1', payload '{"battery":100,"humidity":57.39,"linkquality":255,"power_outage_count":4,"pressure":937.7,"temperature":18,"voltage":3005}'
debug 2022-12-12 00:23:01: Received Zigbee message from 'Humidity sensor1', type 'attributeReport', cluster 'msPressureMeasurement', data '{"measuredValue":937,"scale":-1,"scaledValue":9379}' from endpoint 1 with groupID null
info 2022-12-12 00:23:01: MQTT publish: topic 'zigbee2mqtt/Humidity sensor1', payload '{"battery":100,"humidity":57.39,"linkquality":255,"power_outage_count":4,"pressure":937.9,"temperature":18,"voltage":3005}'
error 2022-12-12 00:23:07: Failed to configure 'Enchufe 2', attempt 1 (Error: Read 0xa4c1389c498fc01f/1 genBasic(["manufacturerName","zclVersion","appVersion","modelId","powerSource",65534], {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (no response received)
at DeconzAdapter.sendZclFrameToEndpoint (/app/node_modules/zigbee-herdsman/src/adapter/deconz/adapter/deconzAdapter.ts:658:23)
at Endpoint.sendRequest (/app/node_modules/zigbee-herdsman/src/controller/model/endpoint.ts:299:20)
at Endpoint.read (/app/node_modules/zigbee-herdsman/src/controller/model/endpoint.ts:472:28)
at Object.configureMagicPacket (/app/node_modules/zigbee-herdsman-converters/lib/tuya.js:1179:9)
at Object.configure (/app/node_modules/zigbee-herdsman-converters/devices/tuya.js:2301:13)
at Configure.configure (/app/lib/extension/configure.ts:115:13)
at Configure.start (/app/lib/extension/configure.ts:68:13)
at Controller.callExtensions (/app/lib/controller.ts:316:17)
at Controller.start (/app/lib/controller.ts:153:9)
at start (/app/index.js:107:5))

I have 12 devices in this moment which are offline. So 12 devices * 1 min (the time to discover he status) = aprox 10 min the time to initiation for Test 2

[Koenkk]

Found the issue, should be fixed now.

Changes will be available in the dev branch in a few hours from now. (https://www.zigbee2mqtt.io/advanced/more/switch-to-dev-branch.html)

[terierul]

thanks!!

[erzhan89]

@Koenkk ,
after 2 days... lost all devices again...

[terierul]

@erzhan89 I think you have another issue. For you the list is working and suddenly disappear? Or this is happening (like in my case) only after a reboot/restart?
anyway the fix is not in production. Probably will be push with the next version. Right now is in dev branch.

[erzhan89]

I had dev branch...
now reverted to 1.28.2 back.
in 1.28.2 works stable...

[terierul]

well the true is that I did not test the fix because my zigbee network is installed in a location where I am under a CG-NAT network. Is my secondary home (not living there permanently) and cannot connect to it from internet.
But still, after reboot, after 10-15 min your network does not come back alive?

[erzhan89]

I tried twice to join all devices back... but after 2 days... they are not in list again..

[Koenkk]

@erzhan89 what coordinator are you using?

[vignez2991]

Yes But, When i stopped z2m and tried power restart several times database file contains actual data. The database file gets empty only when i do power restart after running z2m service

May I expect a fix for this??

@vignez2991 Try the last Dev version : "in the latest version, processing of one event is specially disabled so that devices are not deleted"

Hii @draggero,

Sorry for late reply, I have tested with the version 1.30.1-dev. It looks fine, everything is working normal. I tried power restarted several times the database file was not corrupted. Thanks for the fix.

When should i expect this update in stable release??

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[almirus]

hi! same for me
the device (FUT036Z) wasn't the last added (old device).
I had to re-pair the device for it to appear in the list.

PS
Power is not supplied continuously to FUT036Z. Switched on by mechanical switch. Before disappearing, I noticed that the light was flashing (for some reason it was in pairing mode). After that flashing stopped, and the device disappeared from the list

Версия Zigbee2MQTT [1.30.4](https://github.com/Koenkk/zigbee2mqtt/releases/tag/1.30.4) commit: [unknown](https://github.com/Koenkk/zigbee2mqtt/commit/unknown) Тип координатора zStack3x0 Версия координатора 20220301 Адрес координатора IEEE 0x00124b0022810ef5 Версия веб интерфейса 0.6.127

[Stooovie]

Same, i have two devices disappear from Z2M just in the last 24 hours. Not disconnected or unavailable. Actually gone from the devices list.

[BeFygo]

Same here, a 'Nous A1Z' disappeared from the device list last night and after a z2m (docker) reboot the entity's where removed in home assistant.
No logs where saved for the time of disappearance.

And with hitting the pair button z2m show that the device left the network en shortly after reconnects with the custom name I gave it... ?

[seb49]

Same here

Device works fine but not list. Also, I have a group with 3 devices (work fine) but on the UI the group is empty.

`Zigbee2MQTT version [1.33.2]

Coordinator type zStack3x0
Coordinator revision 20210708
Coordinator IEEE Address 0x00124b0014d98e09
Frontend version 0.6.142
Zigbee-herdsman-converters version 15.106.0
Zigbee-herdsman version 0.21.0`

[lacrimal]

Same mine - 12 devices disappeared only two thermometers left in z2m interface automation wok just fine. After forced pairing popup say 'device leaving network' and after that it appears again.
1.33.2
EZSP v8
0.6.142
15.106.0
0.21.0

[oywino]

I have exactly the same issue as @lacrimal
It all began a few days back when Z2M suddenly had stopped and restart failed with the following error:
MQTT retention requires protocol version 5
So I added version: 5 to my mqtt section in configuration.yaml and Z2M started again without this error. But all my devices were gone and I had no clue how to retrieve them. So, I decided to pair them all over again, one-by-one but after first updating Z2M to version 1.33.2 commit: 9996c93 During re-pairing, I noticed he same behavior as @lacrimal describes: After forced pairing popup say 'device leaving network' so obviously it was all in there somewhere.
This is the Settings-->About summary:

Zigbee2MQTT version	1.33.2 commit: 9996c93
Coordinator type	EZSP v8
Coordinator revision	6.7.9.0 build 405
Coordinator IEEE Address 0x60a423fffe1f42fa
Frontend version	0.6.142
Zigbee-herdsman-converters version 15.106.0
Zigbee-herdsman version 0.21.0
Stats:
Total 0
By device type
By power source
By vendor
By model

After going through the complete repairing of all devices, all was well again (I thought) and in working order. But no - upon the next restart of Z2M everything was again lost. The Z2M UI dashboard is empty and no device is working.

I'm running Eclipse Mosquitto 2.0.18 in a Docker container. Z2M also run in a Docker container. This setup has worked fine for a long time, until the 'version 5' issue popped up the other day.
Please help

[klibro]

Yes, happened to me a few times now. I'm running Z2M as a Home Assistant add-on connecting to Mosquito which is also been run as an add-on. My zigbee gateway is a Sonoff ZBBridge flashed with Tasmota. All works for a while, but if the gateway loses power, then some (but not all) devices often disappear. As others have reported, re-pairing them first results in a "leaving network" message, suggesting the device hadn't really disappeared in the first place.

[hakong]

I've just noticed this issue as well. All my paired devices are gone. I moved my coordinator to a new IP address, changed the IP address in zigbee2mqtt, restarted and everything is empty.

Edit: I firmware upgraded the coordinator in right before moving to a new IP address.

Edit 2: I copied database.db.backup over database.db and re-started. Now all my devices are back. That's very strange. Why does zigbee2mqtt just wipe its database?

[EuPhobos]

Same here, device just dissapear with no reason.. The device it self working well. It's worked several month and disappear.

Starting Zigbee2MQTT version 1.34.0-dev (commit #87f6b9e)
Starting zigbee-herdsman (0.26.1)
zigbee-herdsman started (resumed)
Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20221226,"transportrev":2},"type":"zStack3x0"}'
Currently 14 devices are joined:

[BuurmanG]

Devices disappeared with no reason..... Every 2~3 days 😒

[mwhrtin]

I think this is happening to me as well. I couldn't control a device in HomeAssistant and when I checked the history I could see that the last time it reported anything to HomeAssistant was 5 days ago. Went over to Zigbee2MQTT to check the status there and to my surprise the device was no longer listed. Not offline or unavailable, it's just missing.

How do I get it back?

I've tried to restart everything, I was running Zigbee2MQTT 1.35.0 so I've updated to 1.35.1, I've tried to re-add the device both with and without resetting the device. But Zigbee2MQTT refuse to acknowledge it.

[berkansezer77]

I am having the exact same issue. My Hue Motion sensor is constantly being deleted from the list. How can I fix that ?

[riversideneyon]

I had same issue in the past but after trying everything issue is finally resolved but not sure what exactly has resolved. Try to remove device from zigbee interface, perform a factory reset on hue motion and pair it again.

I am having the exact same issue. My Hue Motion sensor is constantly being deleted from the list. How can I fix that ?

[berkansezer77]

I had same issue in the past but after trying everything issue is finally resolved but not sure what exactly has resolved. Try to remove device from zigbee interface, perform a factory reset on hue motion and pair it again.

I am having the exact same issue. My Hue Motion sensor is constantly being deleted from the list. How can I fix that ?

I actually did that couple of times. But it still disappears from device list and even not appearing in mqtt as well. Something is going on. I have 2 other devices connected but they work normal.

[x61]

Same here. they just disappeared 👻

[mikezs]

I have this issue too, I'm running everything in docker. Every now and again (sometimes months) the devices just all disappear at the same time and stop working. When I repair them, the status in zigbee2mqtt says "leaving network" first meaning they're just not having the messages received

[transon123A]

I also encountered the problem of losing the device in z2m,,didn't see it in the Z2M devices list, it just flashed in pairing mode! Latest Z2M version 1.35.3-1
​

[FunInfinity]

same here on 1.35 and 1.36.0 version.
suddenly 6 devices (all miboxers controllers) disappeared from the list. when I try to repair - the screen is flushed with messages
" has left the network".
And also is not repairing in the end.

[burak-sayici]

Hello, I am having the same problem. After I updated Zigbee2MQTT and home assitant core today, no devices started to appear and connections were lost. The list is empty.
Will this problem be fixed?

[oywino]

With som many people complaining about the same issue (or similar issues), maybe it's time @Koenkk steps in and shows some concern for us (his users). Or is that too much to ask?

[Koenkk]

If someone can provide debug log starting from the moment the devices are there until they disappear I'm happy to investigate.

See this on how to enable debug logging.

[WillyCat]

If someone can provide debug log starting from the moment the devices are there until they disappear I'm happy to investigate.

See this on how to enable debug logging.

link is broken (404)

[WillyCat]

How I solved the issue (works for me, could work for you, no guarantee)

Looks like a pure gui issue,
when starting z2m, logs indicate devices are known :

[2024-06-02 14:43:26] info: 	z2m: Currently 3 devices are joined:
[2024-06-02 14:43:26] info: 	z2m: 0x70ac08fffe493702 (0x70ac08fffe493702): ZBMINIL2 - SONOFF Zigbee smart switch (no neutral) (EndDevice)
[2024-06-02 14:43:26] info: 	z2m: 0xf4b3b1fffe459015 (0xf4b3b1fffe459015): Not supported (Unknown)
[2024-06-02 14:43:26] info: 	z2m: 0x00158d0001ab2d77 (0x00158d0001ab2d77): WSDCGQ11LM - Aqara Temperature and humidity sensor (EndDevice)

looking into database.db, each device has its line in the json file, all data are correct (ieeeAddr etc.)

state.json has a recent timestamp , data has been updated recently

devices info are published to mqtt whenever there is something to publish

so from an engine perspective, as far as I can tell, all is fine.

When connecting to the interface, there is no device displayed
There is no error message in the the logs saying that the interface is unable to display what can be found in database.db either.

I'm using latest dockerhub docker image running Zigbee2MQTT version 1.38.0 (commit #fe048e6)

Having a look at js console in the browser, I saw these errors
index-3cebf61c.js:61 WebSocket connection to 'ws://(my address)/api' failed:
my understanding is that this is how the js in the browser retrieves the list of devices from the backend

The reason why this happens on my setup is that I am using nginx as a reverse proxy , what required some additional setting to handle websockets
I added these lines :

                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_read_timeout 86400;
                proxy_http_version 1.1;

and now it works !

My guess : either nginx or zigbee2mqtt or both changed
(maybe zigbee2mqtt switched to websockets or nginx changed the way it handles this)
requiring this additional setup

Not sure this can solve the issue for everyone but for those having an nginx reverse proxy, it's worth a try

by the way, an error message on the front end would be welcome ;-)

hope this helps

[oywino]

Your solution seems to indicate that you are accessing your Z2M server via an NGINX Reverse Proxy, right? You probably have a very good reason for doing so, while the rest of us have our Z2M server running on the same network (LAN/WLAN) as everything else we have, thus there is no need for a reverse proxy, and consequently we don't have one installed at all.

[WillyCat]

Your solution seems to indicate that you are accessing your Z2M server via an NGINX Reverse Proxy, right? You probably have a very good reason for doing so, while the rest of us have our Z2M server running on the same network (LAN/WLAN) as everything else we have, thus there is no need for a reverse proxy, and consequently we don't have one installed at all.

Exactly. When having an nginx reverse proxy, things happen exactly as described in this post and applying the changes fixed the issue.
I do agree that the solution only applies to this setup, one among many so this is not "the" solution, but a solution for some around here.
What I also tried to do is explain how I came to that conclusion and there is a reasonable chance that the methodology could also apply to other setups.
Especially, considering the z2m is ok in the backend, tracking what happens between the js front end and the websocket backend api is the point I would check because this seems to be the culprit.
The idea was not to say "hey , I solved the issue" but to provide some hints to help