MQTT lights stuck in "unavailable" after reload (new format)

[michalk-k]

The problem

When reloading manually configured MQTT entities, MQTT light entities which:

• are declared using a new format (mqtt: domain)
• have availability set (regardless of new or old way)

get stuck with an unavailable state. Such a state doesn't change even by changes in tracked MQTT topic.

Worth to mention that HA restart causes all entities to recover. Granted, those with availability set, require a minute or so more after restart is confirmed by UI.

The code below configures 4 entities. All do use the same settings and MQTT topics.
Two of them are using the old format, and another two are using the new format. Each pair has one entity with and without availability option (suffixes av and nav respectively).

To reproduce:

1. add those entities to the HA
2. restart HA to confirm all entities are available
3. go to Developer Tools / YAML
4. Reload manually configured MQTT entities

The result: 3 of them are available, while test_mqtt_av is not available.
This test case is deterministic and confirmed with every light entity I reconfigured in order to use the new format. Other domains like sensors or covers seem to be free of this bug. But worth checking all possible domains.

What version of Home Assistant Core has the issue?

core-2022.6.7

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

MQTT Light

Link to integration documentation on our website

https://www.home-assistant.io/integrations/light.mqtt/

Diagnostics information

No response

Example YAML snippet

mqtt:
  light:
    - name: "test_mqtt_av"
      command_topic: "shellies/light-bathroom/relay/0/command"
      state_topic: "shellies/light-bathroom/relay/0"
      availability_topic: "shellies/light-bathroom/online"
      qos: 1
      retain: false
      payload_on: "on"
      payload_off: "off"
      payload_available: "true"
      payload_not_available: "false"
      optimistic: false
      
    - name: "test_mqtt_nav"
      command_topic: "shellies/light-bathroom/relay/0/command"
      state_topic: "shellies/light-bathroom/relay/0"
      #availability_topic: "shellies/light-bathroom/online"
      qos: 1
      retain: false
      payload_on: "on"
      payload_off: "off"
      #payload_available: "true"
      #payload_not_available: "false"
      optimistic: false
light:
  - name: "test_light_av"
    platform: "mqtt"
    command_topic: "shellies/light-bathroom/relay/0/command"
    state_topic: "shellies/light-bathroom/relay/0"
    availability_topic: "shellies/light-bathroom/online"
    qos: 1
    retain: false
    payload_on: "on"
    payload_off: "off"
    payload_available: "true"
    payload_not_available: "false"
    optimistic: false
    
  - name: "test_light_nav"
    platform: "mqtt"
    command_topic: "shellies/light-bathroom/relay/0/command"
    state_topic: "shellies/light-bathroom/relay/0"
    #availability_topic: "shellies/light-bathroom/online"
    qos: 1
    retain: false
    payload_on: "on"
    payload_off: "off"
    payload_available: "true"
    payload_not_available: "false"
    optimistic: false

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant]

Hey there @emontnemery, mind taking a look at this issue as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

---

mqtt documentation
mqtt source
_{^{(message by IssueLinks)}}

[emontnemery]

@michalk-k Thanks for the very detailed issue description with configuration for the various permutations and so on 👍

However, the reproduction steps are incomplete; after a reload the entities with availability will be in state unavailable, and the entities without availability will be in state unknown until they receive messages on their respective status and availability topics.

Are your shellies configured to publish retained messages, or do you maybe have some automation which makes the shellies republish the state after a HA reload?

It would be helpful if you clarify the reproduction steps and also collect a log with the settings below which shows what happens when you reload MQTT.

logger:                       
  default: info               
  logs:                      
    homeassistant.core: debug
    homeassistant.components.mqtt: debug

[michalk-k]

Yes. The availability topic is retained for sure. State topic probably too (I have to back home to confirm). I have retained option enabled in all owned Shellies.
The result is that all properly functioning entities are turning available after the reload and show expected states (reflects state of real-life lights)

I'll try to provide requested log.

edit:

Are your shellies configured to publish retained messages, or do you maybe have some automation that makes the Shellies republish the state after a HA reload?

Indeed. I have two automations triggered by system start and event_mqtt_reloaded event. I completely forgot about them.
Both forces Shellies to republish their state.

here is a code of one of them. The second one differs by trigger only:

alias: Shellies Announce MQTT reload
description: ''
trigger:
  - platform: event
    event_type: event_mqtt_reloaded
condition: []
action:
  - delay: '0'
  - data:
      payload: announce
      topic: shellies/command
    service: mqtt.publish
mode: single

[emontnemery]

Thanks @michalk-k 👍

The automation is very likely to cause the problem, MQTT internally also uses the event_mqtt_reloaded trigger - which is generated when the old format has been reloaded - to reload the new format.

Your automation already has a delay in it, as a workaround I suggest you increase that to a few seconds.
A more correct fix would be to make sure the event_mqtt_reloaded is not fired until both old and new format has been reloaded, I'll look into that.

[emontnemery]

Instead of adding a delay, you can change your automation to trigger in event_type mqtt_reloaded instead of event_type event_mqtt_reloaded.

Never mind this, it won't work.

[emontnemery]

The problem should be fixed by #74226 which will be included in Home Assistant Core 2022.7.0

[michalk-k]

Thank you.
I can confirm that adding 10secs of delay to the automation is a working workaround. But after fixing it you are confident about this I'm sure