Behaviour of the 'retain' option, state.json file and publishing state on zigbee2mqtt restart #3496
Comments
|
Retain is an MQTT specific option, see https://mntolia.com/mqtt-retained-messages-explained-example/ What you are probably searching for is the |
|
Thank you, this is clearer now but only a bit. In fact it, it provokes some further questions. The documentation page says: "cache_state: MQTT message payload will contain all attributes, not only changed ones. Has to be true when integrating via Home Assistant (default: true)" Does it follow from this that the state.json file's only purpose is to keep all attributes last values so that they can be sent together (if cache_state is set to true)? Or is state.json used for any other feature of zigbee2mqtt? Because if this file's purprose is only to enable the cache_state: true, then I don't see how and why the existence of this file should force publishing its contents to the topics during zigbee2mqqt cold start. Keeping attributes together (and using state.json file for this purpose) is one thing (and is very useful in certain cases), but unconditional announcement of "some previously cached values" upon startup as "the actual current state" is a rather questionable, especially if this behaviour is only controlled by the mere existence of cached data. Imagine (as it was in my case) that there was a power failure or network connection interruption for the device that runs zigbee2mqtt and then the services are restored after a while. We don't know for how long zigbee2mqtt was down. Could have been seconds, could have been hours. And yet the "random cached state" was published as the current one causing all sorts of wrong automation to kick in. It shouldn't happen this way just because the bridge software component (zigbee2mqtt) is being restarted. Should it? Can we de-couple republishing of "last known state" on zigbee2mqtt startup from the cache_state option? BTW, maybe somebody can explain to me how this publishing on startup is useful at all. |
|
Additional note about the 'retain' option since my confusion started with it. If it is purely MQTT protocol specific option, shouldn't the global setting belong to the mqtt section of the configuration.yaml instead of the device_options ? If the argument here that it is also a device specific option, then perhaps it should be made more obvious either in the documentation, or even better, by renaming these three options to: or even |
It's also used for some other features (e.g. group behaviour, skipping state publish when device is already on).
I"m not sure if this makes sense. Let's say we have an occupancy sensor with illuminance and occupancy and skip the republish on startup and still have
Retain can be set per message, so that's why it's configurable per device. In case you want to set it to true or false by default you can use |
|
Thank you the answers to some of my questions. How is it useful to republish anything during zigbee2mqtt startup (except for maybe announcing zigbee2mqtt's own state)? What problem does this re-publishing solve or what issue does it address? How is it any good for a home automation system to be fed some values of undetermined age (essentially "any old values" once zigbee2mqtt is stopped and started) as if they were the actual / current state? This is a genuine question, not a sarcasm or rhetorical question. With my quite basic set up (Mosquitto + NodeRed + Influx/Telegraf) this feature of dumping cached data out as if it was the current state actually does "occasional harm" and only creates unwanted issues. The potential for wrongly invoked automation is basically there every time I have to stop and start zigbee2mqtt. And I should add that of all components, zigbee2mqtt is the one that is stopped and started most often due to frequent new releases / updates (updates are great thing, thank you!). I really fail to see how it can be any good to take a run-time cache (that is indeed very useful for many other different reasons) and suddenly turn its meaning upside down and treat it as the "source of truth for the system's state" during start up... I would like to avoid getting the answer of a kind "well, if it is not useful for you, then don't use it" for two reasons:
What I am actually going to do now, to avoid this unwanted behaviour, is to use the following directive in systemd config, i.e. always force-delete state.json file on startup. I will still keep cache_state: true in my settings because I still find it useful for other reasons. Regarding this point:
Sure, you're absolutely right. And that is exactly why you don't want ANY cache present during startup, even if it is enabled in the settings. |
|
Don't get me wrong, this is for sure a valid discussion. It puzzles me a bit that these questions haven't been asked before (because this is in from the start). I think both persistent and non-persistent state have its pros. Hopefully this also answers some of your questions. Persistent state
Non-persistent state
I propose to add a new option |
|
My 2 cents in the game, the current behavior is probably fine for a default. 48% humidity from 40m ago is probably still valid and if I missed an update, it's still better than having it uninitialized so it displays 0%... Would be nice to toggle it per device though. |
It puzzles me too! :-) Before posting I really searched and searched to find anybody else with similar concern to no avail.
I agree with all the pros for persistent and non-persistent cache that you've outlined very nicely. I would note that most people who "don't suffer like I do" seem to base their argument on the fact that "it is useful to have persistent cache in case of quick restart". As if zigbee2mqtt is only "quickly restarted" and there is no other scenario when it can be down and then up again for longer than a second. What about power outage? What about system reboot due to OS patches? What about low spec systems such as Raspberry PI where it takes minutes and minutes to update zigbee2mqtt to new version even with fully automated scripts? I am not buying the argument about "quick restart" because I have a few of the systems running where from time to time it is required to bring down one component or the other for longer than a second or a minute or an hour, and when its zigbee2mqtt's turn, it sometimes results in a completely unwanted automation triggered. If you only collect sensors data in a database to build beautiful graphs (I also do this), then of course cache / no cache / state / no state - who cares. But what if you have some "heavy" automation that depends on innocent events or sensor data? What if the automation is "open garage doors" ? Or "start a boiler" or a sprinkler outside? Or turning lights at 3am in your bedroom? Some of this may trigger really heavy actions that depend on innocent events that can get cached and then pop out in most undesired way. It is about system reliability when you start controlling more critical components with it... So yes, please, the proposed cache_state_persistent option would be very welcome. However... To keep the discussion complete, I would like to go back to one point that is still not being addressed. OK, we have cache. Fine. Some people prefer to have it persistent. Also fine. We now even may get an option for it. Great! But why do we want to publish cache contents during zigbee2mqtt start up? Can we mentally separate, at least for the sake of this discussion, having persistent cache and dumping its contents into the broker when zigbee2mqtt starts. One should not automatically imply the other! I do understand that if there is cached data and new attribute comes from a device, zigbee2mqtt will publish a message that would contain all cached attributes anyway. I do get this fully. But WHY FORCING THIS on start up? For all devices. Unconditionally. Let zigbee2mqtt publish when new actual messages arrive (even together with cached attributes). We shall get them when we get there... But who or what requires all attributes from all devices in one go when zigbee2mqtt starts? This potentially takes us to another lengthy discussion about state keeping in a distributed system, I know... And here I would really argue that zigbee2mqtt is a bridge component and it should stay that because it performs its role beautifully. It should not pretend to act as a state keeper in a distributed system (exactly what it tries to do when it does "sending its cache out on startup"). I am talking about most primitive forms of state keeping, such as MQTT broker itself that can retain messages and republish them as "last known state" to each new client that subscribes to a topic (even if the message is not new, it can be kept with 'retain' option). I am talking about more sophisticated forms too, like NodeRed that is perfectly able to keep its state in memory and persist it on disk and even in a proper database that it can connect to. (So I'm not really getting @sjorge argument about "not having any data in NodeRed when you restart NodeRed". It should not be zigbee2mqtt responsibility to keep the idea of state for NodeRed in case NodeRed is restarted, because NodeRed is able to take care about it's data persistence without any help of zigbee2mqtt). I am talking about even more sophisticated software like OpenHab, HomeAssistant, etc. that act as a centrepiece of home automation and often have state coming not only from MQTT but also from other sources and APIs. All those systems have been designed to act as state keepers in one way or another. And it is their responsibility to keep the complete system's state (persistent or not) the best way they see fit. Zigbee2mqtt does not need to fill those shoes. It's role is to pass messages from raw hardware into something that is universally understood (i.e. MQTT) Let other components care about state keeping. And if we agree on this, then the feature of publishing the contents of persistent cache on startup has no use and is in fact rather "evil" in a properly architected distributed system. |
|
This is only done if the Home Assistant integration is enabled (so when Similar to #3496 (comment) having to wait a hour before temperature values come in can be annoying. I just noticed this issue being created which is exactly the reason why this is done: #3511 Edit: I was wrong, it's not only Home Assistant specific. Actually there are points were zigbee2mqtt sends all cached states
|
|
Thanks for confirming what I've been observing and tried to tell all along. It does not matter if homeassistant option is set to true or false. The cache is always sent out on startup. Can we just discuss this a bit further before more options are created? I am not entirely sure that something like cache_state_send_on_startup is actually needed. I understand now that Home Assistant can benefit by getting some cached state from zigbee2mqtt, specifically when Home Assistant restarts with empty state and signals about that via hass/status topic. And it is perfect that there is already option controlling zigbee2mqtt behaviour in this case (homeassistant: true). But this still does not address the "always publish zigbee2mqtt cache on startup". Is it maybe done because HomeAssistant running on the same host as zigbee2mqtt after OS reboot could start earlier than zigbee2mqtt and then it will never have a chance of getting last known state via hass/status? If yes, then this sounds very much like "publishing cache on startup" is yet another Home Assistant specific behaviour and thus it does not need yet another option controlling it. Sending cache out on startup should be just tucked under the already existing option homeassistant: true (just like you thought it should work in the beginning of your last comment). I think this probably was the original intention anyway but somehow it slipped through. Would you agree to that rather than creating yet another option? |
People may still want this for none home assistant setups too. I use it in node-red as well, although I could work around it in node-red if I really wanted to. But it would mean most of my flows will then need load/store logic to be added to them which would get complex very quickly. Probably something I could fix with spending a weekend or so working on my flows though, so not the end of the world. And sure, I could enable homeassistant integration to keep the old behavior but then I'd get a lot of non-interesting mqtt traffic that is specifically meant for homeassitant. |
|
Side note / slightly off topic / big IMHO / maybe food for thought. It saddens me a bit that zigbee2mqtt has way too many Home Assistant specific options and related logic implemented for a software that is titled zigbee2mqtt. It is not zigbee2ha, is it? ;) And yet zigbee2mqtt it is trying quite hard to plug what I would rate as some rather basic design flaws in Home Assistant. I appreciate that I should be careful when I say flaws because I never used Home Assistant (and now have even less desire to try it :-)) so maybe I am missing some core knowledge here. But I am merely approaching this topic from architectural point of view (something that incidentally I do as my daytime job) and I am treating our home automation world as a typical distributed system, because we have at least 3 components with asynchronous message flows (zigbee2mqtt, MQTT broker and some consumer like HA or NodeRed), but often there are even more (I am running NodeRed, InfluxDB, Grafana, Telegraf, my own web UI for it with server-side scripts, etc.). So we do have a distributed system. Sometimes highly distributed system. We have independent message flows between the components (by nature of MQTT). We need to have state in order to act properly. Whose responsibility is to ensure state consistency and, if needed, persistence? Surely, not the piece of software that does message conversion. This is the LAST place to look for anything state-like. And yet, this is exactly where we are looking when we are discussing that HomeAssistant specific stuff. Implementing very particular options and behaviours just to satisfy somebody else's... laziness? Some may say - this is not laziness, the problem is there because many (most?) of the sensors' state cannot be queried directly. They only emit events (battery % reading is still an event) and state should always be constructed from those events. But the whole system design went somewhat weird when we talk about Home Assistant-like approach. It is great that zigbee2mqtt has very smart and responsive developer(s) behind it who try to "make things work" for others. But actually it feels like some border has been stepped over. In a distributed system, it is a design flaw if a system component that is supposed to accumulate events/state and act on them (NodeRed, HomeAssistant, etc) cannot gracefully handle partial or full absence of some data attributes. And it is a design flaw to solve the above design flaw by forcing specific behaviours in another piece of software to "please" the component that is too lazy to do its job right. This is against the core paradigms of microservice driven design. This is against well architected distributed framework, where each component does its job as a little black box without any knowledge of how other black-boxes do their job. The components speak common language (MQTT) and pass events to each other and that should be enough. No, enough is not the right word. Event should be maximum allowed level of abstraction for communication. Otherwise we are creating new application protocols on top of MQTT (bad). When one component starts commanding very specific behaviours and implementations from another component we have what is known as tight-coupling, which is the root of all evil in a distributed system. If Home Assistant (or NodeRed, or any other software) claims to have support for MQTT, and yet it demands that messages should come packaged in a certain way (like having all attributes together) or else it will create "errors" and "warnings", then this is not really a proper "support" for MQTT, this something like "MQTT+" (and software such as zigbee2mqtt has to conform with it too as the result). If software needs to have "state" of other components to function properly and expects state to be served on a plate, then this is wrong design. In a distributed system there is no "state", there are only "events". If a piece of software actually requires state - it's the job of that piece of software itself to construct state from events and hold on to it. And if it is important not to lose the state (because nobody can serve complete actual state of the entire system on demand) then again, it is the job of that piece of software that needs it, to ensure that the state is persistent enough for the needs of that component. |
You might find that with NodeRed this is much easier than you are probably thinking. https://discourse.nodered.org/t/a-guide-to-understanding-persistent-context/4115 https://nodered.org/docs/user-guide/context#saving-context-data-to-the-file-system |
|
To first clear this out, the resending of all cached states on startup is NOT something implemented for Home Assistant, actually Home Assistant even doesn't need this (given that only z2m is restarted home assistant already has the states). Why was it implemented? I don't know. Ideally this shouldn't be the default behaviour, or this shouldn't be in at all. However given that removing it now would introduce a breaking change, an option to disable this seem to be the correct way forward. This is something that can be further addressed in Zigbee2mqtt 2.x.x. The other republishing of all states is done when Home Assistant publishes |
Spend about 3 hours trying to get that working, doesn't look like it works with subflows at all. Yes, I agree that my setup is kind of shit. But pretty sure most setup out there are not well designed and thought out. Doesn't mean it should not be removed from z2m... but it will cause a mass of issues with people with broken setups. |
|
I update one minor version and the behavior nor works in node-red, will still take a while to go over all my flows though. I guess this is probably indeed a good candidate for a 2.0 release. |
Just a shot in the dark, since I have zero knowledge about your setup. What works easiest for persistence, including all sub-flows, is using 'global' context with persistence. You write to global just using |
Forgive me for lack of history knowledge. I joined the Zigbee + MQTT game not so long ago. I thought you were the one and only original author of zigbee2mqtt who might have known why it was done this way :-)
Thank you! Finally we arrive to what I was trying to say in a rather complex way :)
Sure. And until then I'll just delete state.json before starting zigbee2mqtt which will give me the behaviour of the two not yet existing options: |
|
One question about how current state cache works. |
I can answer this one: Line 8 in 23805c2 There are certain attributes that are not cached. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Removing stale status. Hoping to see both discussed options |
|
Sorry for taking so long, implemented both in the dev branch, docs: https://github.com/Koenkk/zigbee2mqtt.io/blob/develop/docs/information/configuration.md Can you confirm & close the issue? |
|
I can confirm that both options function as expected. Thanks a lot! I would change this: to: Because the enabled option |
|
Done, thanks! Assuming this can be closed now. |
I cannot understand the behaviour related to the 'retain' option and find almost no information online about how it is supposed to work to understand whether this is a bug or a feature.
The documentation states: "Retain MQTT messages of this device (default false)." and that's about as much explanation as you get. Retain it how / where? Is it about instructing MQTT broker to keep message "permanently" or is it about zigbee2mqtt's own "persistent cache"?
I am guessing (perhaps wrongly because the actual behaviour seems to contradict my guess) that if the retain option is set to true, then every time you get new message for a device, the values are sent to MQTT broker and additionally most recent values are stored in the state.json file. When you cold-start zigbee2mqtt, it reads the state.json file and basically re-publishes its contents to the broker again. Is this correct?
What should happen if I do NOT want this 'republishing on start up' happening? It seems that no matter what I do (setting 'retain: false' for each device individually, or in the device_options globally, the state.json file is still being written and updated for all of my devices (and it gets re-published to devices' topics on zigbee2mqtt restart).
I tried stopping zigbee2mqtt service, setting retain: false for all devices, erasing the 'state.json' file entirely from disk and starting zigbee2mqtt again. It is true - nothing will be republished first time you start zigbee2mqtt, when the state.json file is not there. But once new messages start arriving the state.json is getting populated again and a subsequent restart causes messages being republished for all of my devices.
Now, usually it would be not a huge deal (most sensors would get overwritten with newer data anyway), but one of the things that gets republished on the start up is the state of the Aqara occupancy sensor that for whatever reason in my case turned out to be true , triggering all kind of automation upon zigbee2mqtt restart (when there is no motion happening in reality).
So, to sum it up, it seems like in my case retain: true does not work as expected (if my expectations are correct, that is). The state.json file is getting written and updated even with retain set to false for all devices (individually or via the device_options section of the config file).
Running the latest released zigbee2mqtt on Raspbian with the Texas Instruments CC1352P-2 board:
{"version":"1.13.0","commit":"55e6283","coordinator":{"type":"zStack30x","meta":{"transportrev":2,"product":2,"majorrel":2,"minorrel":7,"maintrel":2,"revision":20190425}},"log_level":"info","permit_join":false}The text was updated successfully, but these errors were encountered: