Zigbee2Mqtt - OTA "Over the Air" updates - automating the OTA one-by-one in NodeRed in Home Assistant

[rwunsch]

Hello all,

I am very impressed by Zigbee2mqtt - and was super happy to see the build-in UI for Zigbee2mqtt in Home-Assistant - and super happy to be able to do updates of most of my devices OTA.

However, this is currently a manual process, kicking-off the update one at a time.

I tried to automate the update once the are available - and used NodeRed to do so.

Here is my approach:

1. First - I set a local "flow"-variable "flow.zigbee_update_going_on_state" to "false" on startup (first flow on top).
2. Second Flow: I listen to any event "_update_available == "on"
3. I check if the "flow.zigbee_update_going_on_state"-variable is "false" (to check if an OTA upgdate is currently being done/executed).
4. in the first function, I return the "friendly_name" as payload, to be consumed in the later "svc: mqtt:publish"-node (Service node) in the "data-object" for the MQTT-call.
5. I call MQTT to update the entity in the node ""svc: mqtt:publish", according to the documentation of Zigbee2mqtt: https://www.zigbee2mqtt.io/information/ota_updates.html . This node uses the "entity_id" in the payload as the "target-entity" to execute the service on (if the "entity_id" in the svc-node is empty.) This helps to make the service call to the initially discovered entity of the event. The "Data-object" contains the "friendly_name" of the zigbee-device, coming from the function before.
6. I set the local variable "flow.zigbee_update_going_on_state" to "true" - to only have one update executed at a time.
7. I again use a function to get the "entity_id" as "msg.payload". This helps with the following "Wait-until" node.
8. I use a "wait-until"-node to see when the "_update_available" turns to "Off" (which means the update is executed successfully). One probelm: The device might be offline, or something else does not work - so the "flow.zigbee_update_going_on_state" would never be set to false if the update does not succeed. Therefore the "wait-until" has a time-out-timer of 30min - after which the "flow.zigbee_update_going_on_state" is set to "false" anyway in the next node.
9. I set the "flow.zigbee_update_going_on_state" to false - a new update can start on point 1) above.

The flow is here as json to be imported into node-red:
[{"id":"bb4b97fe.aa45e8","type":"tab","label":"Zigbee2mqtt update","disabled":false,"info":""},{"id":"a0d299e0.8799c8","type":"trigger-state","z":"bb4b97fe.aa45e8","name":"Update Available","server":"5a0908cf.d3bd78","exposeToHomeAssistant":false,"haConfig":[{"property":"name","value":""},{"property":"icon","value":""}],"entityid":"_update_available","entityidfiltertype":"regex","debugenabled":false,"constraints":[{"targetType":"this_entity","targetValue":"","propertyType":"current_state","comparatorType":"is","comparatorValueDatatype":"str","comparatorValue":"on","propertyValue":"new_state.state"}],"outputs":2,"customoutputs":[],"outputinitially":false,"state_type":"str","x":130,"y":140,"wires":[["e82d842e.6db508"],[]]},{"id":"e82d842e.6db508","type":"switch","z":"bb4b97fe.aa45e8","name":"If \"Flow: update going on\" is FALSE","property":"zigbee_update_going_on_state","propertyType":"flow","rules":[{"t":"false"}],"checkall":"true","repair":false,"outputs":1,"x":360,"y":200,"wires":[["2b2da468.eb9c2c"]]},{"id":"8db78272.81801","type":"inject","z":"bb4b97fe.aa45e8","name":"Set \"FLOW: upgrade going on\" to FALSE on startup","props":[{"p":"payload"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"false","payloadType":"bool","x":270,"y":40,"wires":[["d98780ad.6fe93"]]},{"id":"d98780ad.6fe93","type":"change","z":"bb4b97fe.aa45e8","name":"Set \"FLOW: update goind on\" to FALSE","rules":[{"t":"set","p":"zigbee_update_going_on_state","pt":"flow","to":"false","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":680,"y":40,"wires":[[]]},{"id":"72b25be3.9c9b64","type":"api-call-service","z":"bb4b97fe.aa45e8","name":"","server":"5a0908cf.d3bd78","version":1,"debugenabled":false,"service_domain":"mqtt","service":"publish","entityId":"","data":"{\"topic\":\"zigbee2mqtt/bridge/request/device/ota_update/update\",\"payload\":\"{{payload}}\"}","dataType":"json","mergecontext":"","output_location":"","output_location_type":"none","mustacheAltTags":false,"x":430,"y":320,"wires":[["363a6d24.48f812"]]},{"id":"363a6d24.48f812","type":"change","z":"bb4b97fe.aa45e8","name":"Set \"FLOW: update going on\" to \"true\"","rules":[{"t":"set","p":"zigbee_update_going_on_state","pt":"flow","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":490,"y":400,"wires":[["5f9eb160.33258"]]},{"id":"1b324d94.b0d9d2","type":"ha-wait-until","z":"bb4b97fe.aa45e8","name":"Wait-until \"<entity>_update_available\" is OFF","server":"5a0908cf.d3bd78","outputs":2,"entityId":"","entityIdFilterType":"exact","property":"state","comparator":"is","value":"off","valueType":"str","timeout":"30","timeoutType":"num","timeoutUnits":"minutes","entityLocation":"","entityLocationType":"none","checkCurrentState":true,"blockInputOverrides":true,"x":630,"y":540,"wires":[["e89a0468.421658"],["e89a0468.421658"]]},{"id":"e89a0468.421658","type":"change","z":"bb4b97fe.aa45e8","name":"Set \"FLOW: update going on\" to \"false\"","rules":[{"t":"set","p":"zigbee_update_going_on_state","pt":"flow","to":"false","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":600,"wires":[[]]},{"id":"5f9eb160.33258","type":"function","z":"bb4b97fe.aa45e8","name":"add \"entity_id\" into msg.payload","func":"msg.payload = {\n entity_id: msg.data.entity_id,\n}\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":470,"y":480,"wires":[["1b324d94.b0d9d2"]]},{"id":"2b2da468.eb9c2c","type":"function","z":"bb4b97fe.aa45e8","name":"get \"friendly_name\" - remove \"udpdate_available\" - return \"friendly_name\" as payload","func":"var friendly_name = msg.data.event.new_state.attributes.friendly_name;\n//node.warn(\"friendly_name : \" + friendly_name);\nmsg.payload = friendly_name.replace(\" update available\",\"\");\n//node.warn(\"msg.payload = \" + msg.payload);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":600,"y":260,"wires":[["72b25be3.9c9b64"]]},{"id":"5a0908cf.d3bd78","type":"server","name":"Home Assistant","addon":true}]

This might need to be adjusted depending on your set-up - but I would expect this to be working for anyone.

I would be interested about your feedback.

Thanks and cheers
Robert

[mdegat01]

So first you don't need to reset flow.zigbee_update_going_on_state on startup like that. Node RED has different context stores available and the file one will survive restarts. Here's a quick guide a user posted on the HA community forum explaining how to set that up for the add-on if you're using that

Second, unless you have exactly one device of each type I'm pretty sure you're going to have race conditions. Let me explain with an example. Let's say you have 4 Philips Hue lightbulbs and an OTA update is released for that device. All 4 of those are going to have an update available at almost exactly the same time.

• Best case scenario is one light bulb starts updating and then the other 3 just don't because after update_available triggered the flow the message stopped at If "Flow: update going on" is FALSE for the other three. The state won't change again until they are updated so those other 3 just won't update automatically.
• Worst case scenario is all 4 messages pass through the If "Flow: update going on" is FALSE node before any of them reaches Set "FLOW: update going on" to "true". Now you've got 4 light bulbs updating at the same time.

I have a flow similar to this but I don't do automatic updates personally. What I do is send a mobile app notification to myself when an update is available for a device. That notification will have an action which is "update" which when pressed triggers a flow that sends the MQTT message to update the device. I also have a second flow which looks to see if any sensors which match sensor\.\w+_update_state have changed and have a state of updating. If so I send a notification to myself with the percent complete and time remaining, this way I know when the update is complete and can trigger the next one.

Personally I would not want automatic updates here since I'm not aware of any way to rollback a bad update. I am very cautious about updating things within my Home Automation ecosystem as I want to find out what changed and ideally know how to revert if the update didn't work well for me. I have had too many bad updates to trust automatic updates for pretty much when it comes to home automation devices and software.

But if you do want that I would suggest switching from triggering on state change to polling. This way if the flow stops because an update is already in progress its ok and it will keep trying until nothing is updating. I would also use a control node like delay set to rate limit or a trigger node to ensure that only one "update is available" message can get through at one time.

[rwunsch]

Hi @mdegat01,

Thanks a lot for your recommendations!

I saw the possible race condition - but so far did not experience multiple devices updating at the same time. but throttling might be a way to resolve that.

I would not know how to change from "state change" to "polling. Could you elaborate on that?

Could you maybe share your "flow"?

Thanks and cheers
Robert

[Dennisreneholm]

I'm quite interested in a solution like this.

@rwunsch
Did you change your flow so you eliminated the "risk" of race?

i got like 70 zigbee devices, not all of them can be updated, but all my lights in my house are hue lights, and they get updates rather often.

Right now i'm updating them manually with the Zigbee2MqttAssistant interface, which works rather well to be honest, but it's tedious and painfully slow. It takes me a couple of days to get through all my devices when an update hits.

[rwunsch]

hi @Dennisreneholm ,

I actually did not look at this issue, back after @mdegat01 's comments I added a limitation on messages (1msg/30min) - to somewhat mitigate the race-condition:

I still see 2-3 devices updating simultaniously, however - but it works for me (Zigbee network might be a little slow during updates ...).

For the "context stores" I had set the context-variable to "file" already, but wanted to make sure it is being reset on start-up:

I did not find a way to poll for these states, as the polling -node does NOT support "regex"-based polling, and I would have to configure 70-80 nodes for polling (number of zigbee devices, actually these are 116 by now, but some are Xiaomi devices).

I actually do not understand why, as the "State trigger" should only only trigger once when update is happening/state-changed, but eventually all my devices-auto-update (is this fired again on startup?).
As it works for me, I did not look into this too deeply.

[mdegat01]

The Trigger: State node you are using to start that flow is triggered anytime the state of a watched entity changes. State includes attributes, not just the actual on/off state. So even if just an attribute changes its going to run through your conditionals and fire a message if they all evaluate to true.

The way Z2M works every entity has attributes which are basically the state of all the other entities of that device. So the update_available entity for a light has at least brightness as an attribute and probably others as well (linkquality, last_seen, etc.) If you don't account for this in the Trigger: State node then your flow will likely fire quite frequently one the basic state condition is met (basically every time the light reports in to Z2M).

Another thing to keep in mind is Z2M does not set the retain flag on messages it creates to report current state of devices. This means after a restart of HA the sensors start unknown until the next time Z2M sends a message for it. Often this is quite quick but not too quick for automation to pick up as a state change.

The easiest way to fix this is to use an Events: State node instead of a Trigger: State node. Then you can simply check all these boxes and not worry about these situations:

If you need to use a Trigger: State node for other reasons then generally you'll want to make sure you add the following 4 conditionals to it to recreate what those checkboxes are doing:

You can replace the last one with a check for a specific state or list of states (like is true in this example).

Once those checkboxes were added to Events: State I started using that as much as possible since it greatly reduced the amount of boilerplate I needed. Usually I am not interested in these types of state changes so I have to remember to add those conditionals to each Trigger: State node manually.