Always keep track of when entity state is written

[emontnemery]

Proposal

• Add a new timestamp, last_reported, to state objects which indicates when the integration set the state, regardless of if the state or any state attributes have been changed.
• Update the state in hass.states on every state write, regardless of if the state or any state attributes have been changed.
• Add a new event state_reported which fired on every state write, regardless of if the state or any state attributes have been changed.
• Update the existing event state_changed such that when fired, the old_state will have an updated last_reported timestamp.

Background

Home Assistant currently discards state writes where neither the state nor the state attributes are changed, unless the integration sets the force_update flag. This behavior makes it very difficult for integrations to correctly do time series analysis of numerical sensor state. It also means the user don't know if an integration is updating a sensor or not.

Performance concerns

The main identified performance concern is recording of state changes by the recorder. If the recorder would subscribe to the new event type state_set and record those events, the recorder would have to process much more events and also do much more database writes. Some quick sanity checks on a few systems @bdraco suggests a factor of 20-30, but this will vary wildly depending on user setup.

To address this, the proposal is that the recorder will still only subscribe to state_changed events, and update the last recorded state's last_reported timestamp as needed.

Future improvements

If this proposal is accepted, we should consider deprecating the force_update flag for hass.states.async_set.

We should also consider deprecating the last_updated timestamp on State objects; it's used only be a few integrations, and to my understanding incorrectly, as if the last_updated timestamp is the last_reported timestamp proposed here. That's for example the case in the filter, the generic_hygrostat and integration integrations.

[dgomes]

This proposal would solve 99% of users complains related to the integration integration. :)

[parautenbach]

The issue I see most in that regard is that people aren't using method left for integrating power signals. It might not be the same issue you're referring to, so clarification is needed. If it is, it would be good to understand why you're saying this, since the different integration methods are about practical mathematical trade-offs, which is quite fundamental.

[dgomes]

People use left method to overcome the fact that their source sensor does not update that often (or so it seams because we only have last_updated).
The trapezoidal method is much more accurate to approximate the power curve then the left method. The issue list in the fact that if you stretch the trapezoid to link far away points (last_updated and current) you end up with a bad approximation. If you do in fact get frequent updates with the same values, then trapezoid becomes the best method always, because you will integrate more frequently.

[bdraco]

If we decide to fire the state_reported event on the event bus it likely has the similar performance concerns as the recorder because it could mean significantly more new events (a bad guess is 1 per entity) per minute on busy systems that have to be dispatched and since event firing is always scheduled on the event loop, all of them will generate a call_soon per listener unless its marked as run_immediately https://github.com/home-assistant/core/blob/99e29b75ccc9fa747f3a60d8d39cdac025375f21/homeassistant/core.py#L1396

If we use the dispatcher or a callback subscription to get the data to the recorder this likely mitigates that concern.

[emontnemery]

To address this, I think we should not include state_reported events in wildcard subscriptions.

We could also consider to:

• Not allow directly subscribing to state_reported, only allow it via helpers similar to async_track_state_change
• Not allow subscribing to state_reported events for all entity IDs
• Not allow subscribing to state_reported without also subscribing to run_immediately

It is probably a good idea to start very conservative, to avoid a breaking change which nerfs the API.

[bdraco]

We have a lot of external addons/integrations that subscribe to all events because we don't have a way to subscribe to all but X events or a way to subscribe to a list of events. I've been asked to add this a few times but it never came to the top of the pile.

Another option would be to send the data via dispatcher only since it won't have the overhead the event bus.

[Petro31]

A ton of templates use last_updated properly to catch attributes changing. We should not consider deprecating that property.

[FlorianOosterhof]

Will it also be made possible to add this as a trigger to an automation or trigger-based template sensor on this event?

[JonLaliberte]

I noticed that last_reported has been added. Are there any plans to use this to update how, for example the statistics sensor functions? Right now, a sensor that reports the same value for repeated readings causes this sensor to fail. For example a 15min avg on a temperature sensor will fail (report unknown) when the temp doesn't change for 16 minutes.
This is discussed a bit here: https://community.home-assistant.io/t/improved-handling-of-repeated-sensor-values-which-are-presently-ignored/637736/18

[frenck]

Closing this discussion as the proposal has been implemented.