Broadcasts failing on ember after migration #22453
Comments
|
Any chance you can downgrade to 7.4.1 and see if you still have those problems on the pi? |
|
Same problem with SLZB-06M But I don't have a raspberry pi 4, host is a x86 machine, running unraid and zigbee2mqtt in docker. |
|
Grouping the mentioned broadcasting issue here guys (#22445, #22398) I cannot reproduce this with my Dongle-E. I've tried various firmware, various ways to migrate from |
adapter: ember May need to add 'rtscts' below adapter setting. |
Two things: I recently installed https://www.zigbee2mqtt.io/devices/ZFP-1A-CH.html#siglis-zfp-1a-ch Wich I think is not a very common router. Swiss market only and most likely not very popular. Initially I had problems with it. Also shortly after I installed it, my second Dongle-E that I use as a router had to re-pair and this was one of the first devices in my 2yo network that I never had any problems with. Second: Shortly before my Router Dongle failed I set reporting interval of every lamp to 1-3 seconds because I didn't see lamps status change quickly enough (or at all) when pressing a HW button like the switches mentioned above. After the Dongle failed I reverted this to 1-30 s and had no problems since. But I did the reverting before I saw the error in logs. Also I have to say: I don't recognize bigger problems or misbehavior. I just saw the error in the logs. The only real problem I have is that sometimes (not reproducible) some IKEA Bulbs are starting in maximum dimmed mode even though at least one of them is never dimmed manually. |
As the dongle-e is working using a docker images on an x86 environnement I'm guessing there is no issue with the zigbee Dongle, so if I focus on some specifics configs, here is what's coming to my mind as part of the change that might be different than a regular installation :
everything else is quite standard in my opinion. |
|
Nothing special over here. Had 1.36 running with SLZD-06M running on zigbee FW 20231030. Everything was running OK with adapter: ezsp Did the following steps:
So currently I'm in a state that my network is running, but I can't add any new devices. Is there any more info we can provide? |
|
Oh I should have mentioned that I am running HAOS in a VM on Synology DSM 7.2. Interference should not be a problem as my dongle is in a USB 2 port with a 2 m extension cable. |
|
My setup is HAOS running on a ODROID M1 with 8GB RAM and 512 GB SSD. |
Exactly the same behavior. Plus the problem that no new devices can't be paired with ember. But with ezsp I can add devices. In my case especially all my routers get disconnected. |
|
I do have 4 mmwave presence sensors. Maybe these devices have an influence. |
|
Sorry, posted my follow-up on the wrong ticket... These are the messages I see when I startup Zigbee2MQTT. Maybe they are related. [2024-05-05 11:00:43] info: z2m: Logging to console, file (filename: log.log) [2024-05-05 11:00:51] info: z2m: Zigbee: disabling joining new devices. Whenever I try to start the pairing process, I see these messages: [2024-05-05 11:03:28] info: z2m: Zigbee: allowing new devices to join. |
Ah yes, and I wasn't aware it is related... I have a SLZB-06M as coordinator (groundfloor) and a Sonoff Dongle-E flashed as router (first floor). Yesterday evening my Sonoff router got disconnected. It is while trying to pair it again that I found out I couldn't pair any devices. I have a very small zigbee network (more a test setup here), so I have no other routers, only end devices. |
I already have nearly 70 devices... |
Here at home, HA is a small setup (12 devices) I use mainly for testing. But in our vacation home, everything is controlled by HA and we have 51 zigbee and 33 ESPHome devices. In this second setup, I also have the same SLZB-06M coordinator, but still on the older 20231030 firmware, where the adapter is still defined as 'adapter: ezsp'. Since I ugraded to 1.37, I couldn't pair any new devices too, due to another error: "zh:controller:greenpower: Received undefined command from '0'" And that setup is not a test setup :-( |
In the development-branch channel. The similarity we both have is the same coordinator (I am at the dev Firmware right now). But maybe you can rather rule out the cause if you only have 12 devices in your setup. |
|
Very very simple configuration here. HAOS on qemu VM in low end x86-64 QNAP nas, resources 2 cpu+2 GB ram as suggested by HAOS setup guide. Back to the setup, I can report two setups:
Anyway I see from other posts that the error is happening with a variety of devices and if I look at another common factor, all the variety of networks showing the error have -> a coordinator <- which again spots the light on the coordinator. I see that @Nerivec is not able to reproduce the issue, and, needless to say, also Nerivec is working with a coordinator which should obviously rule out the coordinator itself (unless there is some elusive coordinator hardware common factor), maybe a good starting point for you would be to constrain the system on a low resource/slow host or a VM with limited resources to see what happens with the coordinator handling of Z2M. Maybe another hint maybe found in the first post from @julien-billaud: "I've tried the exact same configuration on a regular x86 computer running debian (using the same zigbee dongle) and didn't face any issue which seems to be a linked with the Raspberry pi 4". |
|
OK, because my setup is a small setup mainly for test, I did the following steps:
[12:01:03] INFO: Preparing to start...
[2024-05-05 12:01:40] info: z2m: Zigbee: allowing new devices to join.
[2024-05-05 12:02:19] info: z2m: Removing device '0x00158d0008083d2a' (block: false, force: true)
[12:06:41] INFO: Preparing to start...
[2024-05-05 12:07:40] info: z2m: Zigbee: allowing new devices to join. so pairing is working and I didn't get the broadcast error now, not while starting up and not while pairing. So starting over with zigbee2mqtt solved it for me, but that is not possible for everyone I think :-) |
No, not completly... after approx 5 minutes, pairing was again not possible. No errors, but the connection / interview didn't start. Tried to restart z2m and reboot the coordinator, nothing helps. Downgraded the coordinator to the 20231030 FW (ESZP12) and switched back to "adapter: ezsp" and I still got the "error: zh:controller:greenpower: Received undefined command from '0' " messages, but pairing is possible again. Will see in about 10 minutes... |
I do also have one Sonoff TRVZB. And I also started fresh with one new zigbee2mqtt config and just the coordinator, and even at start the pairing/broadcast issue appeared immediately. I don't think that it is an issue with raspberry pi as I am using an x86 machine running a zigbee2mqtt container (docker). I also observed that a coordinator reset sometimes helped. @Nerivec recommended to do a hard reset with my device (that includes pushing the physical reset button). This also helped me once starting without any issues, but after restarting again, I again suffered by those errors. |
Just to have a better understanding: what CPU/RAM is your x86 machine? Is it running what OS? Is it on bare metal or on a virtualization environment like Proxmox or other VM of any sort? I agree dockers are less demanding, but performance then is limited by the host so it would be useful to know what kind of host is running your docker and how loaded is your x86 system. |
|
It is a Intel® Core™ i3-9100 system with 64 GB RAM ECC. |
|
I have a low-resource VM that mimics the specs of an average PI 4 to run tests on stuff that I know affect performance. No issue there either. No failed broadcast without any device, nor with devices, and successfully paired & re-paired a dozen devices since it's been running for a couple of hours. But just in case, you can try giving it some breathing room with the advanced:
adapter_delay: 20Default/min is 5, max is 60 (milliseconds). Note that at 60, you are likely to experience some delays when triggering devices rapidly. PS: I created an issue in the firmware repo for the SLZB-06M and the failing config IDs. May or may not be related to the ensuing troubles, but we need to get to the bottom of it nonetheless. darkxst/silabs-firmware-builder#90 |
Added the adapter_delay option, no joy: [2024-05-05 14:42:54] error: zh:ember: Delivery of BROADCAST failed for "65532" [apsFrame={"profileId":0,"clusterId":54,"sourceEndpoint":0,"destinationEndpoint":0,"options":256,"groupId":0,"sequence":170} messageTag=255] at startup of z2m. |
|
I've been doing little more testing and figured out "what was wrong". To conclude, it seems like the ember driver is for some reason little bit more sensitive (I know that using the Dongle without extension cord isn't ideal). |
|
Can't be my problem. USB2 Port with 2m extension cable. |
|
@majorsl Please create a new issue with a debug log attached as your issue appears unrelated to this one. |
|
@Nerivec good news I think: yesterday I have restarted z2m with log debug but in 24h I did not receive any broadcast error. |
|
I also had this problem with my skyconnect on firmware 7.4.2 - and by pure coincidence I noticed the placement of the dongle had something to do with it. I used something like bluetack to hold the dongle and usb extension lead (on a usb 2 port) to the side of the wood cabinet. This has worked well with ezsp for a year or so. After upgrading to 7.4.2 and changing to ember, I immediately had the described issue with broadcasts. However, due to removing the dongle for the firmware upgrade, it wasn't as well connected to the wall as before, and it dropped down. Suddenly the zigbee network started performing without any problem with the ember driver. (though I also upgraded to the latest version of Z2m, so I thought this had solved it). I noticed the dongle had fallen down, and put it back in it's place, and the issue returned! I now upgraded firmware to 7.4.3 and so far haven't been able to reproduce the issue anymore.... Very strange, but who knows this description helps in tracking it down.... |
|
I have the same error. |
Just wondering if it only seems that way, because powered devices are typically pinged much more frequently, and thus show as "offline" much sooner than battery devices.... |
|
Ok I confirm that in the last 48h I haven't had any broadcast error with fw 7.4.3. I'll disable debug now: to me is fixed. |
This is what I see as well. I tried running it for an extended period, and they ultimately all go offline. |
My Skyconnect is currently running on version 7.4.2, and I was able to reproduce the behavior you described. My Skyconnect is also connected with a USB extension and was mounted on a wall surrounded by wood using double-sided tape. As a test, I placed the dongle on the floor. Suddenly, the lamps could be switched on and off without any problems. This behavior is highly interesting. Unfortunately, I cannot find the firmware version 7.4.3. Where can I download this version? |
btw: that one also seems to work with rts/cts: true (hardware flow control) - the previous ones didn't for me. |
|
For the Silicon Labs flasher you need the raw link: https://github.com/darkxst/silabs-firmware-builder/raw/4.4.3/firmware_builds/zbdonglee/ncp-uart-hw-v7.4.3.0-zbdonglee-115200.gbl I just updated, will see if that works better. I also found that at least one, maybe two IKEA Tradfri bulbs are having problems: they stop reacting and when I switch them off and on again they start minimally dimmed. After that they are starting to react normally again - for some time. Then the error happens again. Will see if the new fw helps. |
|
I have it working now. what I changed: swapped all usb ports and configure them again in proxmox. rebooted the system and it's working now. |
|
For anyone that can't seem to fix this, you can try clearing the NVM3 on your adapter. Make sure to follow the procedure properly to ensure you can restore your network config after (otherwise need to re-pair everything). If you don't care about restoring your network config, you can just flash the init file like a regular firmware, then flash the proper firmware for your adapter. |
Thanks, I just flashed to 7.3.3. It is super important to use the USB cable extension to avoid USB interference. Otherwise, you get these broadcast errors. My setup is now working with this USB extension and Skyconnect on 7.3.3 and EmberZNet. I am now getting these errors at startup: Since 14:46, no more errors have occurred. Just a ton of debug info. @Nerivec, if you want, I can send you the debug log. However, I would like to send it directly and not publicly. Of course, only if it will help to fix problems. |
|
@extreme4u Those errors at startup (RESET_WATCHDOG) are due to a hardware flow control bug. It's being tracked, silabs is aware, but it's been around for a long time... both Z2M and ZHA implement a retry mechanism to work around it. It should not prevent proper start, let me know if it ever does. PS: If you ever want to share logs directly, you can find me on zigbee2mqtt discord, then send me a DM. |
|
For me it was fixed by following steps (note that it involves reconnecting all devices):
Quite some work, but it works great now (even better than donglep). I still get some error but no notable effect on the zigbee network. Note that this is only a quick summary of what I did and I am just a casual z2m user. So be careful and take a backup first. |
Meanwhile, having had some more time in use, I must say the Ember driver + skyconnect for some reason seems to be much more prone to interference - I never noticed this with the same dongle in more or less the same location before. Having been swapping out some components such as a switch in the cupboard where the server and thus the dongles (on a usb extension lead on usb 2) are kept, I noticed for my environment the placement of the dongle is much more finicky now then I ever remember it being before. I changed the location and the usb extension lead, and for now it seems stable again, but I don't remember it being that picky before.... |
|
I've had the same issue, and for me adding: in WARNING: This will require re-pairing all devices. |
This also deletes all devices. Atleast it did for me. And still have the error. |
That's right - generating a new key forces you to re-pair your devices. An it makes sense only if you don't have it generated before. I use the latest firmware for the Zigbee Dongle E device: |
|
Hi team! Just adding my own 2 cents here for anyone wondering what they can do in the meantime until this is fixed: I have rolled back my zigbee2mqtt server version while keeping the updated firmware on the Sonoff dongle. Steps were:
|
|
I have to report that checking the logs I still have the broadcast errors. Fw 7.4.3 does not fix the issue: [2024-06-03 22:22:56] debug: zh:ember:uart:ash: <--- [FRAME type=DATA] |
How did you downgrade? I can't figure out how to downgrade without losing the data because I installed z2m through home assistant add-ons. |
|
As I had this issue along with others making the system quite unreliable, I downgraded to 1.36.1 which solved the problems. |
@huotarih yeah sorry mate, I would have no idea on that one. I use Z2M in a docker container so it's easy for me. Surely it can be done though? |
|
Can confirm the main problem is with ember driver. Ever since I changed back from ember to ezsp (Latest Z2M, Dongle E with 7.4.3) the startup error messages (but no others) are back but all actual problems are completely gone. Will stay there for a while. |
|
Figured out how to downgrade. Depends of course if you still have the backup. |
What happened?
Deprecated driver 'ezsp' currently in use, 'ember' will become the officially supported EmberZNet driver in next release. If using Zigbee2MQTT see #21462 I tried to run SMLIGHT SLZB-06M & Zigbee2MQTT with configuration "ember" although SMLIGHT obviously does not yet recommend this. See "SMLIGHT SLZB-06M control panel Zigbee2MQTT config generator" NOT WORKING Zigbee2MQTT configuration.yamlWhat did you expect to happen?Zigbee2MQTT with adapter: ember is running without errors SMLIGHT SLZB-06M control panel Zigbee2MQTT config generatorHow to reproduce it (minimal and precise)Setting in the following environment Zigbee2MQTT versionCurrent version: 1.38.0-1 AdapterSMLIGHT SLZB-06M Adapter firmware version[7.4.2.0 build 0](firmware: v2.3.6) Radio FW version20240510 Coordinator modeLAN SetupGEEKOM Mini IT13 Mini PC 13th Gen Intel® Core i9-13900H 32GB RAM+2TB SSD SLZB-06M System LogZigbee2MQTT Log[2024-06-06 17:09:59] error: zh:ember: Delivery of BROADCAST failed for "65533" [apsFrame={"profileId":0,"clusterId":0,"sourceEndpoint":0,"destinationEndpoint":0,"options":1024,"groupId":0,"sequence":196} messageTag=255] INTERMEDIATE SOLUTIONNO Zigbee2MQTT downgrade OPEN BUG (not sure about that) - Will contact SMLIGHTErrors during Zigbee2MQTT start-up (WATCHDOG ENABLED!) |
and others
What happened?
While I've never been facing any issues for more than a year with the Sonoff Dongle-e + ezsp driver, I've tried to change the driver to ember, but nothing is working (tried multiple time) but sometime losing all the devices, sometime they are still there but impossible to interact with them, and pairing is never working. (for now I returned to the ezsp driver).
I'm not noticing much error in the log (only the broadcast error reported here #22445)
I've tried the exact same configuration on a regular x86 computer running debian (using the same zigbee dongle) and didn't face any issue which seems to be a linked with the Raspberry pi 4
What did you expect to happen?
No response
How to reproduce it (minimal and precise)
switch from eszp to ember driver
Zigbee2MQTT version
1.37.0
Adapter firmware version
7.4.2.0 build 0
Adapter
Sonoff dongle-e
Setup
Raspberry pi 4 using docker image
Debug log
No response
The text was updated successfully, but these errors were encountered: