when hass discovery is turned on listen for hass restart and resend discovery data

[travisghansen]

Have you looked for this feature in other issues and in the docs? Yes

Is your feature request related to a problem? Please describe.
I see lots of people having issues with hass restarting and losing access to devices. After digging around I think the solution should be to resend discovery data when hass is detected as restarted.

Describe the solution you'd like
Listen to hass/status topic and when a payload of online is received resend all the configuration messages (and possibly states).

Describe alternatives you've considered
Currently handling this via Node-RED and publishing the SetOption19 command to the group topic which retriggers at least discovery messages.

Additional context
I've been scratching my head for a bit trying to understand why mqtt discovery fails on restart with various devices but my zwave2mqtt devices come up after every hass restart. After some digging I realized it's listening to hass/status mqtt topic and republishing when hass restarts. From my reading around on forums etc I think this will solve a lot of headaches and misunderstandings with quite a few users.

Thanks!

(Please, remember to close the issue when the problem has been addressed)

[effelle]

I'll have a look, for now you could simply expand the automation for Syncing Power State that is in our docs.
Also, to have hass/status LWT you need to manually add it through YAML in your configuration.
Discovery is not able to generate that message alone.

[travisghansen]

That actually won't work (unless I'm missing something) as the devices won't even exist and therefore topics won't be subscribed to at all.

That is correct that the LWT must be added to the configuration yaml...

[effelle]

First I need a better understanding of a problem that I don't have. After a reboot of HAss the only thing I do is sync the states using the automation, never lost a device.
Also, Tasmota could need up to 5 minutes to show again due to teleperiod time (set to 300 seconds by default), a device could be shown as unavailable/unknown until the next /state message.
( @emontnemery ) Erik, do you think will be possible add hass/status to the MQTT Integration in Hass without manually adding it from yaml configuration file?
I agree with the OP, using the hass/status LWT could be a cool way to keep devices in sync without using automations hence more user friendly.

[travisghansen]

I’m pretty new to both hass and tasmota...let me do some testing a bit more but I’m near certain I have devices that would never show up as available after hass restart (even after the 5 minute window). I’ll confirm with further detail though.

[ascillato]

If Home Assistant is restarted but the broker is not restarted, Tasmota don't have any way of knowing that HA has been restarted. That is why, the automation for HA explained in the docs, is the solution for this.

[travisghansen]

Yeah, confirmed the devices never show up as available (for me at least). Interestingly, I updated the window to 10 seconds and did see status updates coming through the mqtt debug logging info from hass but the device stays as unavailable until I 'rediscover' them. Also of note, after clicking an unavailable device from the lovelace UI the state messages actually stopped coming through (not sure if hass subscribes on stored info during startup but then unsubscribes when you interact with the thing when it's unavailable or what).

In any case, I've never had any issues with the zwave2mqtt stuff, everything rediscovers and becomes available immediately when hass starts. Initially I thought it was because it set the retain flag on the discovery messages but I dug into the code a bit a bit and realized it's doing as requested here.

[ascillato]

Seems that there is something else going on in your case. Some odd config. The discovery message is needed to be sent just once and the mqtt broker stores it in its database (the message is retained by default) and everytime your HA is restarted, the discovery message retained in the broker, is sent to your HA.

[travisghansen]

Ah, just realized some (many?) people use the integrated broker which is probably why some don't see this issue as much (assuming it restarts when hass restarts?). In other cases the results would be sporadic at best.

I run a separate broker which is probably why I've been scratching my head wondering why the out-of-box behavior is so strange/bad with mqtt.

[travisghansen]

I assure you the config messages sent from tasmota to hass are not sent with the retained flag. I confirmed this already.

[ascillato]

I use the addon mosquitto in HA and I don't have that issue. Do you have any other config in configuration.yaml? Which broker do you use?
Have you modified Tasmota?

[ascillato]

I assure you the config messages sent from tasmota to hass are not sent with the retained flag. I confirmed this already.

Ok, that's it. They are by default retained.
Do you have modified Tasmota?

[travisghansen]

I have not. These are some gosund switches that were hacked with tuya-convert and then updated to the latest full version ota (8.3.1). I then applied a template and set names/topics as desired.

Here are crude notes of setup steps:

https://templates.blakadder.com/gosund_WP6.html
{"NAME":"Gosund WP6","GPIO":[0,0,0,17,0,0,0,0,56,57,21,0,0],"FLAG":0,"BASE":18}

configuration -> other -> template -> save

configure mqtt
set host/user/password
remove topic (will set the default auto-generated value)
add 'tasmota/' to the full topic

Set configuration -> other -> Device Name = tasmota_E5C770-1904 (hostname from main menu -> information)

# turn on hass mqtt discovery
SetOption19 1

[travisghansen]

I use rabbitmq as the broker (have an independent install I've used for years before I used HA or tasmota). I do require username/pass for it.

[effelle]

As @ascillato told you, Tasmota Discovery send all the topic with the retained flag, example:

20:18:08 MQT: homeassistant/device_automation/49A3BC_BTN_1_SINGLE/config =  (retained)
20:18:08 MQT: homeassistant/device_automation/49A3BC_BTN_1_DOUBLE/config =  (retained)
20:18:08 MQT: homeassistant/device_automation/49A3BC_BTN_1_TRIPLE/config =  (retained)
20:18:08 MQT: homeassistant/device_automation/49A3BC_BTN_1_QUAD/config =  (retained)
20:18:08 MQT: homeassistant/device_automation/49A3BC_BTN_1_PENTA/config =  (retained)

If the retained flag isn't there the broker have no idea of the device connected and then a new subscribe is needed. Also, Hass has its internal database where all the discovered devices are stored. I don't know rabbitmq broker then I have no idea how it manage the topics. Seems it has some configuration for retained messages you need to look at. Worth a shot.

[travisghansen]

Retained messages work fine with rabbitmq. The console of tasmota also shows (retained) but I can assure the messages are not sent with the retained flag set to true. I've confirmed this both with the hass UI listening to the messages and Node-RED listening to the messages.

example:

00:34:56 MQT: homeassistant/device_automation/565A71_SW_1_TOGGLE/config =  (retained)

In Node-RED (different topic, but it's the same for all of them):

topic: "homeassistant/binary_sensor/E43DC3_SW_6/config"
payload: ""
qos: 0
retain: false
_msgid: "87a2e40f.8f5a98"

Hass UI:

Message 1044 received on homeassistant/switch/565A71_RL_4/config at 5:41 PM:
QoS: 0 - Retain: false

[effelle]

Mine are all retained as expected using Mosquitto broker, both the Mosquitto broker addon and the pure install under linux on my dev machine.

You certainly agree there is something different in your broker since both me and Adrian have not the issue.
And as you also saw in console, Tasmota is sending the topic with the retained flag.

[travisghansen]

I regularly send/use messages with the retained flag with rabbitmq using several different softwares...tasmota is not sending with the retained flag set in my case :( I can subscribe to messages from the HASS UI for example (specific topics that are sending with the retained flag) and get the retained message/value.

Here's an example of a retained message as sent by zwave2mqtt when subscribed via the HASS UI:

Message 1049 received on z2m/23/128/1/0 at 6:00 PM:
{
    "value_id": "23-128-1-0",
    "node_id": 23,
    "class_id": 128,
    "type": "byte",
    "genre": "user",
    "instance": 1,
    "index": 0,
    "label": "Battery Level",
    "units": "%",
    "help": "Current Battery Level",
    "read_only": true,
    "write_only": false,
    "min": 0,
    "max": 255,
    "is_polled": false,
    "value": 93,
    "lastUpdate": 1592722531159
}
QoS: 0 - Retain: true

Also the LWT messages are set as retained by tasmota:

2020-06-21 17:23:44 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on tasmota/tele/tasmota_E43DC3/LWT (retained): b'Online'

But I'll keep digging. Something weird for sure.

[effelle]

Let me know if you find something, it could benefit other users too.

[travisghansen]

Well, I've confirmed rabbitmq works just fine with retained but the Node-RED output for received messages can't be trusted. I've just sent a message in Node-RED to a test topic with the retained flag set and the received message in Node-RED says retained false. However, I the message is retained and subscribing to the topic in HASS UI shows it properly as well...

Message 1050 received on test/foobar at 6:09 PM:
1592784512298
QoS: 0 - Retain: true

Again, retained messages work fine with rmq so something is up with tasmota (or some config option is off for me or the build that's installed).

[effelle]

Ok, for the sake of knowledge I checked under HA logs and all the messages are retained as expected:

2020-06-21 20:54:39 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on homeassistant/switch/DDDE94_RL_2/config (retained): b'{"name":"Luzes Noturnas","stat_t":"tele/Suite2/STATE","avty_t":"tele/Suite2/LWT","pl_avail":"Online","pl_not_avail":"Offline","cmd_t":"cmnd/Suite2/POWER2","val_tpl":"{{value_json.POWER2}}","pl_off":"OFF","pl_on":"ON","uniq_id":"DDDE94_RL_2","dev":{"ids":["DDDE94"]}}'
2020-06-21 20:54:39 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on homeassistant/sensor/0F280B_status/config (retained): b'{"name":"Quarto Camilla status","stat_t":"tele/qt_cam/HASS_STATE","avty_t":"tele/qt_cam/LWT","pl_avail":"Online","pl_not_avail":"Offline","json_attr_t":"tele/qt_cam/HASS_STATE","unit_of_meas":"%","val_tpl":"{{value_json[\'RSSI\']}}","ic":"mdi:information-outline","uniq_id":"0F280B_status","dev":{"ids":["0F280B"],"name":"Quarto Camilla","mdl":"Quarto Camilla Module","sw":"8.2.0.4(tasmota)","mf":"Tasmota"}}'

Even the LWT are ok:

2020-06-21 20:54:43 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on tele/Bridge/LWT (retained): b'Online'

[travisghansen]

OK, some of my comments appear to be misguided. Node-RED is showing if it received the message from a retained status or fresh, so it does show retained correctly :( The same appears to apply to the HASS UI so the stuff sent earlier with retain false is pretty irrelevant I guess.

Does HASS subscribe to a wildcard config topic to retrieve all the config messages? I'm wondering if mosquitto handles retained messages from wildcard topics differently than rabbitmq...as in, rabbitmq only sends retained messages if explicitly subscribed to the topic but mosquitto sends them down even when subscribing with a wildcard.

Maybe you could test this using mosquitto_sub against your mosquitto server using a topic like this? homeassistant/+/+/config and see if you get all the retained messages. I do not against rmq.

[ascillato]

If I subscribe to #, I receive ALL the retained messages from the database in mosquitto.

So, it is good to know that your broker is not working as expected for HA.

[travisghansen]

Yeah ok. That sounds like a bug in the rmq implementation of mqtt then.

That must mean hass uses the wildcards for its startup process as well.

This feature might still be good in general though?

I’ll open a bug with rmq about it as well and link to this.

[ascillato]

Sorry but this request is not possible. That is why the HA startup automation is explained in the Tasmota.

When HA restarts, there is no standard/default way for Tasmota to know that HA have restarted.

And sending again and again the discovery is not as the mqtt discovery standard is made or needed.

Sorry.

Closing as the issue with retained messages in the broker is not related to Tasmota.

[travisghansen]

Ouch.

• Devices unavailable on Home Assistant restart Koenkk/zigbee2mqtt#3740
• https://github.com/rabbitmq/rabbitmq-mqtt/issues/154
• https://github.com/rabbitmq/rabbitmq-mqtt/issues/127

[travisghansen]

There is a way...that’s exactly what zwave2mqtt does.

[travisghansen]

I’m ok if you don’t want to implement this but seems a little premature to close while discussion is still going on. It was even mentioned earlier that it could make the HA automation unnecessary as an added benefit.

[effelle]

Closed doesn't mean we don't take into consideration. I've already ping an HA dev to see if we can implement it.

[ascillato]

The idea of hass/status is really good, we really like it, but until that is not available from home assistant side, we can't do anything from Tasmota.

[travisghansen]

Fair enough. I’m not sure I follow what you need more from HA. What’s not available from the HA side currently? Or what do you want differently from them?

[ascillato]

To make HA publish to hass/status, you need to add that manually in your HA configuration.yaml

If that mqtt message were automatic and be present by default in HA, we will add to Tasmota a subscription (if setoption19 is 1) to that standard topic to listen as you requested because it is a really good idea and good approach.

In fact, this should solve the problem that some users make relays and buttons as retained messages to not use startup automations but making other issues like ghost switching.

[travisghansen]

Ah I see you just want it as the default. Got it.

Thanks for the consideration and working through that craziness!

[ascillato]

Thanks for pointing to this idea.

And if you want to forget of those retained issues, use the super reliable Mosquitto. I really recommend it. Super lightweight and fast.

[travisghansen]

I’m looking into it now. I use amqp as well so it was easy to add it into the mix. I also have auth with ldap which sounds like it may be fun with mosquitto :(

[emontnemery]

Tasmota publishes its discovery messages with retainflag set, so there is absolutely no need for what's requested here.

I don't think bugs in brokers breaking basic functionality such as "retain" should be fixed in Tasmota.
On the other hand, maybe Tasmota documentation should warn against using rabbitmq?

[ascillato]

@emontnemery

Totally agree.
Another thing, what do you think of adding in Home Assistant that HA publish by default its own LWT mqtt message to hass/state with payload Online and Offline?
That LWT of HA could be used for other devices to inform their own status when HA starts, instead of using a HA startup automation for gathering relays and sensors information.
What do you think of this?

[blakadder]

- id: 359be2dd-5d57-427d-b6bb-845b3892c30e
  alias: sync tasmota state
  initial_state: true
  trigger:
    platform: homeassistant
    event: start
  action:
  - service: mqtt.publish
    data:
      topic: cmnd/tasmotas/state
      payload: ''

one automation can do wonders.... if you wish you can spam so19 1 to all devices on HA restart

[emontnemery]

Another thing, what do you think of adding in Home Assistant that HA publish by default its own LWT mqtt message to hass/state with payload Online and Offline?
That LWT of HA could be used for other devices to inform their own status when HA starts, instead of using a HA startup automation for gathering relays and sensors information.

Yes, maybe something like that could be useful, this is the current status:

• LWT (Last Will and Testament) is a broker based feature included in the MQTT spec. LWT is stored in the broker and be published by the broker only when there is an unclean disconnect of the client from the broker, i.e. when the connection is severed without the client sending a disconnect first. This can be configured in Home Assistant, but is disabled by default.
• In addition to this, HA allows configuring a "birth" topic+message, which is published every time HA connects to the MQTT broker. This features is disabled by default.
• HA does NOT allow configuring a disconnect topic+message to be published on an intentional disconnect, i.e. restart or shutdown of HA. It would make sense to add this feature, although it's perfectly possible to implement with an automation. Any suggestion for naming such an addition?

Once home-assistant/core#36537 is merged, it will be possible to configure will and birth message from the HA UI, it's currently configuration.yaml only so the will and birth options are not very easy to find.

[ascillato]

@emontnemery

Thanks for looking at this. I think is going to be very useful.

HA does NOT allow configuring a disconnect topic+message to be published on an intentional disconnect, i.e. restart or shutdown of HA. It would make sense to add this feature, although it's perfectly possible to implement with an automation. Any suggestion for naming such an addition?

what about as @travisghansen's and as zigbee2mqtt ?

hass/status ONLINE
hass/status OFFLINE

[effelle]

In my tests every reboot triggered the topic Hass/status when back online. Em seg, 22 de jun de 2020 21:43, Adrian Scillato <notifications@github.com> escreveu:

…

@emontnemery <https://github.com/emontnemery> Thanks for looking at this. I think is going to be very useful. HA does NOT allow configuring a disconnect topic+message to be published on an intentional disconnect, i.e. restart or shutdown of HA. It would make sense to add this feature, although it's perfectly possible to implement with an automation. Any suggestion for naming such an addition? what about as @travisghansen <https://github.com/travisghansen>'s and as zigbee2mqtt ? hass/status ONLINE hass/status OFFLINE — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8753 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2YR3RB4KQYBCY2JC2HQT3RX73DNANCNFSM4OEBHVRA> .

[travisghansen]

On connect it will but lwt messages only get sent on ungraceful disconnects...so you only get offline messages if things go bad but if you cleanly shut down no.

[emontnemery]

Starting with Home Assistant version 0.113, Home Assistant publishes messages when it goes online and offline: home-assistant/core#37371 without need for user configuration or automations.

MQTT birth message defaults to: {"topic": "homeassistant/status", "payload": "online"}
MQTT will message defaults to: {"topic": "homeassistant/status", "payload": "offline"}
MQTT will published also on clean connect from broker

I think Tasmota can subscribe to homeassistant/status by default and publish a status update when receiving online.
We could also consider publishing MQTT discovery messages when receiving online. In that case there is no longer any need to retain the discovery messages. That would however mean that Home Assistant won't discover any Tasmota device which happens to be offline.

[effelle]

Good one Erik!
I can see just a eventual issue:

MQTT will published also on clean connect from broker
I think Tasmota can subscribe to homeassistant/status by default and publish a status update when receiving online.

What would interest me is the online message instead of the offline but having HAss send the birth also on clean connection from broker could lead on some unwanted results.
I'm thinking on the extreme case when you start having issue with your broker and HAss rapidly connect and disconnect in sequence.
Should not be a problem for the status update but in this (rare) case I'm not sure if is a good idea send out the discovery again.

[ascillato]

Home Assistant version 0.113 is released with this change 😄

[effelle]

Already? Damn, I need to hurry up! :D
After the new TuyaMCU...