Zwave mesh grinds to a halt + HA nodes start growing new (incorrect) capabilities

[jdiegmueller]

I've held off on posting this because I have been trying to better-understand the problem, first. But I've been fighting this for awhile and am stumped so I'm posting in hopes of a pointer:

The following issue occurs every 12-72 hours or so:

• It starts with Zwave communication still working, but being slow. For example, I have plenty of motion sensors and door sensors that trigger lights to turn on. I'll notice that it starts to take 1, 3, 5 seconds before the lights activate.
• It escalates to ZwaveJS being able to receive, but not transmit. I have quite a few Inovelli Zwave dimmer switches with the LED bars, but as they are in "Smart Bulb" mode they're basically just acting as Zwave controllers. When I tap "down" on a switch, for example, I'll turn turn of the lights in that room and turn off the automated lighting in that room. Turning off the automated lighting in that room will change the LED bar on the switch to red. When the problem starts getting bad, the "down" tap is received after a small delay (and lights turn off) but the LED bar doesn't change to red.
• If I look in HomeAssistant, I'll see various nodes have grown new capabilities which aren't possible. For example, I'll have an Aeotec Smart Switch 7 reporting motion detected, or an Aeotec Water Sensor 7 reporting a door open.
• If I look at ZwaveJS stats, I will see received messages/commands incrementing very quickly.
• When this is happening, I'll see tons of "transmit status: Fail, took 0 ms" in zwavejs logs.

The solution to the Zwave network being jammed up is simply to restart ZwaveJS and I'll be back to normal for 12-72 hours or so. To clear the nonsense notifications (ie, motion detected from a smart switch) I need to re-interview the nodes with the impossible capabilities.

The issue is happening this morning as I am typing this (ie, right now) and I happened to get up early enough that I was the only person awake -- so rather than fixing it, I was able to let a Zniffer run for a bit. It appears to me that the controller (node 1) isn't transmitting at all, which I guess explains the flood of messages received as nodes keep retransmitting over and over.

I'm going to today's zwavejs log and the Zniffer capture, would love any insight or suggestions on additional data points to look at. I always struggle to pinpoint the "start" of the issue, but it looks like retransmits started overnight sometime shortly after 2023-09-21T07:00:00'ish. I restarted the ZwaveJS instance at 2023-09-21T11:25 so the Zniffer will show signs of life from node 1 after that.

Thanks for any help.

zwavejs_2023-09-21.log.gz
zniffer_20230921A.zlf.gz

[jdiegmueller]

Saw this in 11.14.3 notes:
When the Serial API on a controller restarts unexpectedly, the node ID length is now synchronized again (#6271)

The logs I attached to this one don't show this behavior, but I have seen in past events where ZwaveJS logs were complaining about "node not found" -- and the node ID(s) it was complaining about wasn't even possible in Zwave (node IDs above 232).

I upgraded to 11.14.3 just now in case this is part of my issue.

[AlCalzone]

2., 4., 5. are this issue: #6260

It appears to me that the controller (node 1) isn't transmitting at all, which I guess explains the flood of messages received as nodes keep retransmitting over and over.

Correct! The controller gets stuck in the middle of a transmission and Z-Wave JS's recovery mechanism is not enough to get it back to work normally. I'm not sure which firmware version your stick is on - all reports so far have been on 7.19.x.

For 1., I need a bit more help when this happens. There are a few instances where queries take more than a second, and they are appearing in combination with re-transmitted responses, so likely a temporary connectivity issue.

For 3., I also need more specific information, at least node IDs. Could also just be this: https://zwave-js.github.io/node-zwave-js/#/troubleshooting/nonsensical-values?id=nonsensical-values-appear-randomly

[jdiegmueller]

Wanted to close the loop on this:

I was running 7.20.1, so I migrated to 7.18.8 and indeed haven't seen this issue happen again. Thank you.

[AlCalzone]

Ok now I'm curious. How did you downgrade?

[jdiegmueller]

Ah, yes, I let the word "migrate" do some heavy lifting here didn't I?

Short answer: I cheated. I have a small handful of UZB-7s that I have amassed over the years. I found one running 7.17.x and did an upgrade to 7.18.8 and an NVM backup/restore.