Unify optimistic states for entities at core level to prevent UI "flipflops" and Voice Assistant errors

[marcelveldt]

So, this is a discussion dragging around for years now and still a lot of (non technical) users get hit by this and while trying to fix this at integration level a simple solution popped in my head to fix this one once and for all at core level.

Currently we have 3 ways of dealing with how the actual state of devices/entities is handled within HA:

1. Fully optimistic: The integration doesn't poll/retrieve a device to get the actual state but just assumes it based on commands sent from HA.

2. Partially optimistic: The integration retrieves the actual state of a device by polling it periodically and/or from some sort of push message from the device itself. When sending a command to a device, the integration updates the state with the values sent in the command. These values can be overwritten later when the actual state comes in by poll/push.

3. Source based, not optimistic: The integration completely relies on state updates from the device itself to update the HA state. Technically the best option as the state is always 100% accurate.

So, what's the big deal ?
Well, in option 2 you can have a situation where the device did not receive your command, just partially accepted it or another user/system controls that same device (e.g. a light controlled by a light switch). So the UI (or voice assistant) confirms the action succeeded but some seconds after, the state changes again .

In option 2 you give a command to a light and kind of expect it to respond very fast with a new state update. In fact the command can suffer from the same issues as mentioned above for option 2 but we've see a lot of cases (especially Z-Wave) where a light is just slow to respond with the updated state.

On the various support channels for HA this is identified as 2 commonly reported issues:

1. The Home Assistant UI "flipflops" when I give a command. For example a light switch is toggled ON, after 2 seconds it switches back OFF and another 1 or 2 seconds later it switches back ON.

2. I give my Voice Assistant a command to turn ON/OFF a light but is responds that the command did not succeed while in fact it did.

The Home Assistant UI is kind of prepared for this and already has a 2 second redemption period to prevent flipflopping:
home-assistant/frontend#10735

In real life this is not enough because a state update can be received later than 2 seconds and another thing wrong about the current approach is that the user has no visual indication that we're waiting for the device to process the command.

Users, especially the non-technical ones do not understand this, imo this is really bad UX.
People do tend to forget that while people installing Home Assistant are most likely the tech savvy but their family members also have to deal with it ;-)

I have a proposal to fix this at core level (state machine or generic entity model) and at the same time have a good look at how integrations handle this and try to unify this a bit more.

Proposed solution

1. Send a command to a device/light from UI, direct service call or voice assistant.
   1a) Service call received (and executed) by the integration's platform.
2. State machine or generic entity model:

• Update the state with the params sent in the service call and mark it as optimistic
• Expect a state update to happen from the device within X seconds

3. If a state is being marked as optimistic, show this in some visual way to the user and prevent a new command until the confirmation is there.
   4a) State update received from device --> remove optimistic mark and update state
   4b) Timeout expired (no state received from device) --> remove optimistic mark and restore state

I think there should be some overridable flag to set the amount of seconds an entity is in the optimistic state.

Alternative to this solution is that integrations implement the optimistic state themselves but can set a boolean state attribute while the state is optimistic so the Ui can act on it.

[marcelveldt]

This issue keeps popping up, this time I'm experiencing the same issue for Matter devices, especially Thread based devices can be slow to report their state. Click the switch in the HA interface to turn on a light, the light turns on almost immediately but the report that the light turned on takes a few seconds to arrive and you see the UI flipflopping causing confusion.

This issue also exists with Zigbee, Z-Wave and many other integrations where some of them already "hacked" around it by setting the state just optimistic (without any fallback or check).

[marcelveldt]

My idea to tackle/fix it in a very generic way, at the base entity model.

• Have an additional attribute in the entity class called "optimistic_state" which is either a dataclass or a tuple of string and dict (to record both state and attributes).
• When the value of the additional attribute is None, there is no optimistic state.
• If the optimistic_state attribute has a value it means the current state is optimistic and we can act on that.
• Proposal would be to actually write the value of the optimistic state into the state machine and set an additional state attribute "optimistic: true" to let the outside world know that the state is optimistic.
• Within a service call for each platform we set the optimistic state attribute to the values of the service call.
• Whenever the state is written from within the integration's implementation, we always clear the optimistic state again.
• When setting the optimistic state attribute we also set a timer to auto clear the optimistic state after 30 seconds.
• Setting the optimistic state should be done with a class method (to handle setting/clearing the timer etc).

--> making use of the optimistic state has to be implemented per platform (within the service call)

[marcelveldt]

frontend-wise we should ideally indicate that we're waiting for something to happen. Like a transitioning phase.Maybe a small loading spinner or any other indicator to let the user know "I have received your command and I'm working on it"

[thecode]

For some reason my previous comment was hidden, although most of it still applies. IMO the first priority should be that a state update should be handled by the integration. e.g. Integration should handle avoiding the flip-flop and "latching" the current state until it has a new update. We do similar in Shelly (when we send a command that succeeds we internally hold the last state we set until we get a new update).

I think optimistic should stay only for optimistic devices (devices that don't report a state and the integration rely on being optimistic that the command succeeded and the state was updated.

However @marcelveldt your last comment did raise something which would be nice to be shown on the UI and may help reducing complexity in core side - the "loading"/"spinner" state. This would be nice that integrations will be able to set the UI to show that the state is waiting to be updated and when the update finally happens the UI will show the updated state.

IMO the best way to implement this would be:

• Do not change the state of the entity - .e.g if the user changed something from off to on and integration is waiting for an update the state of the entity would stay off (this is not critical, if there is a better suggestion it can be discussed later), the idea is that automations will run only after the state update received from the device.
• Allow integration setting a parameter to tell the frontend to show the "loading"/"spinner", a simple solution would be a boolean, or an improved version can be a timeout value which after the timeout the UI can show an error (such as exclamation mark). Another option is to let the integration handle the timeout and update the UI accordingly. Suggested name for such parameter can be operation_timeout or delayed_update ...

[marcelveldt]

My whole idea/proposal is to NOT having it per integration to have a unified user experience, which is probably not always technically correct but at least the user has the same experience regardless of the device. Difficulty is what you bring up with automations and such, hence my proposal to make the whole state as optimistic during that time and probably not update any automations and/or recorder/history states etc ? Well, that is maybe the discussion :-)

You could say that we only want to overcome the UI flipflop AND the voice assistants confirmations.

[marcelveldt]

Maybe optimistic is the wrong wording because looking at your comment about Shelly is exactly what I'm proposing. But instead of doing it in shelly, zwave, matter and whatelse, unify it into a core feature for each integration to leverage.

[balloob]

The state machine in Home Assistant is to store what HA currently thinks the state is. This might be a recent report, or if assumed_state is True, it's what we expect the state to be based on the last sent command.

By storing optimistic state updates in the state machine, we are now giving the states in the state machine 2 meanings: the state it currently is and the state it should become soon. I think that this is a mistake, as it means that a) we cannot see the actual state and b) every time you consume a state, you now need to check what is going on.

We had a similar problem in the past for context. How do we attribute upcoming state changes to a context. We did that by adding Entity.async_set_context() and Entity.context_recent_time. This function is called before we call a service, and it will attribute all states for the next 5 seconds (overridable using Entity.context_recent_time).

What if we move the frontend logic of providing a fake state to the backend by building upon Entity.async_set_context() and turn it into Entity.async_set_expected_state(…). Just like with set_context, it will be valid for a couple of seconds, overridable by the integration. The expected state can be added as a new attribute to the state until the state change comes in.

The impact would be that if you would call light.turn_on, the light entity will always update it's state right away. It would either be the new state or it would have a new expected_state attribute, which would include the context that created the expected state too. Then when a state comes in from the integration, it can be correctly attributed. It also means that if you try to turn on a light but find out that it's unavailable, you won't attribute the unavailable state to a user.

This solves a couple of use cases:

• State machine states have a single meaning
• It's clear when an upcoming state is arriving, allowing frontend/google/alexa etc to communicate that correctly

[marcelveldt]

I really like this approach. I was seeking for a shortcut with extending/abusing the state machine but this approach is much better and cleaner. Also, not a lot of work extra.

[thecode]

@balloob @marcelveldt please review my answer in #740 (reply in thread), I have suggested something similar with more details on the implementation (spinner is can be dropped)

[TheJulianJES]

The state machine in Home Assistant is to store what HA currently thinks the state is.

So when a light is turned on (from off) to 100% brightness with a 60 second transition, what should the state be during those 60 seconds? (perhaps target brightness or slowly updating (every second?))

AFAIK, most integrations (including ESPHome) directly set the light to the target brightness (which is what I'd expect, as up until recently, the brightness slider completely glitched out when you moved it during an update and updating every second would probably cause many state updates if a large amount of lights are turned on/off with a transition).

(Or should lights get a special property with something like the ending time + transition time?)

The same issue with transitioning brightness also applies to color and color temp.
Even worse, I think both Matter and Zigbee lights can only send attribute reports with their current brightness/color/color temp (and not target brightness/color/temp).

[evelant]

I think the proposal by @balloob is very good. I think this is personally one of my biggest pain points with HA right now. Zigbee and cloud devices can be all over the place with their behavior in setting and reporting state. Without the ability to differentiate between expected/commanded state and actual state I frequently get devices that end up in incorrect states when activating scenes with no clear way to prevent it.

[evelant]

I think this is also related to #1067 #1059 and #978

[thecode]

So when a light is turned on (from off) to 100% brightness with a 60 second transition, what should the state be during those 60 seconds? (perhaps target brightness or slowly updating (every second?))

If we go with the approach that HA does not manipulate the states, the frontend should show the target value immediately. I don't support doing tricks such as showing a moving slider if you don't have any real-time update report from the device.
Take for example covers which I have several models, the movement is not linear in time and changes between covers. So a logic that will try to show the cover moving will show incorrect position. It is better to see the cover in the final state. For light brightness if we add a spinner the spinner can show that the action is still running.

[marcelveldt]

I agree on showing the final state and maybe the frontend can come up with something to show that the device is "busy" performing the action but that is more nice to have. Most important would be that the frontend switches to the "target state" immediately to avoid confusion.

[spartandrew18]

I just wanted to chime in this post to state that me being a user who is not development or code savvy has been confused by the behavior of some products. @marcelveldt was kind enough to direct me to this post. I will attach some videos of my user experience issues. I hope this somehow gets resolved in the future.

20231212-1902-23.5203812.mp4

20231214-2043-15.1022987.mp4

The Bedroom LED is a Hue Zigbee Lightstrip connected via Zigbee2MQTT
The Cubes are Yeelight Matter WiFi Devices.
The rest are Nanoleaf bulbs connected via matter over thread. The thread devices are the ones that seem to be having the delayed issue.