MQTT changes on uniq_id and obj_id for dhw entities for HA discovery within 3.7 dev version

[tp1de]

As already discussed the latest dev versions of 3.7. are introducing changes for home assistant discovery for dhw enities.
In a consequence uniq_id and ob_id are changed for mqtt ha discovery. The results are new entity id's:

1. All statistic information is lost
2. Lovelace UI needs to be adjusted manually
3. Scripts / automations based on dhw / ww entities need to be changed manually

Everyone using the energy dashboard with nrgww entity are loosing all historical statistic data unless transferred by own sql-scripts.

I recommend to secure downwards compatibility with 3.6.5 within mqtt setup in respect to home assistant id's for all users with only one dhw circuit.

[proddy]

can you provide some examples of the uniq_id and obj_id changes between 3.6.5 and 3.7-dev?

[tp1de]

can you provide some examples of the uniq_id and obj_id changes between 3.6.5 and 3.7-dev?

3.6.5

3.7.0-dev4

[MichaelDvP]

There are some more changes for dhw circuits in 3.7. introduces with #1379. The new water device-class remap MM100 (dip8/9) and SM100 (dip3..) modules.
Changes are:
mixer/wwc1/.. becomes water/dhw/..
mixer/wwc2/.. becomes water/dhw2/..
solar/wwc3/.. becomes water/dhw3/..
solar/wwc4/.. becomes water/dhw4/..
boiler/ww... becomes boiler/dhw/..
thermostat/ww... becomes thermostat/dhw/..

For the exotic convservative people we can remap in the mqtt Single Instance long names (v3.4) with:

        if (has_tag && (device_type == EMSdevice::DeviceType::BOILER || device_type == EMSdevice::DeviceType::THERMOSTAT) && tag == DeviceValue::DeviceValueTAG::TAG_DHW1) {
            snprintf(uniq_id, sizeof(uniq_id), "%s_ww%s", device_name, Helpers::toLower(uniq_s).c_str());
        if (has_tag && device_type == EMSdevice::DeviceType::WATER && tag >= DeviceValue::DeviceValueTAG::TAG_DHW3) {
            snprintf(uniq_id, sizeof(uniq_id), "solar_wwc%d_%s", tag - DeviceValue::DeviceValueTAG::TAG_DHW1 + 1, Helpers::toLower(uniq_s).c_str());
        if (has_tag && device_type == EMSdevice::DeviceType::WATER && tag >= DeviceValue::DeviceValueTAG::TAG_DHW1) {
            snprintf(uniq_id, sizeof(uniq_id), "mixer_wwc%d_%s", tag - DeviceValue::DeviceValueTAG::TAG_DHW1 + 1, Helpers::toLower(uniq_s).c_str());
        } else if (has_tag) {
            snprintf(uniq_id, sizeof(uniq_id), "%s_%s_%s", device_name, DeviceValue::DeviceValueTAG_s[tag][0], Helpers::toLower(uniq_s).c_str());
        } else {
            snprintf(uniq_id, sizeof(uniq_id), "%s_%s", device_name, Helpers::toLower(uniq_s).c_str());
        }

here:

EMS-ESP32/src/mqtt.cpp

Lines 857 to 861 in 3270139

	if (has_tag) {
	snprintf(uniq_id, sizeof(uniq_id), "%s_%s_%s", device_name, DeviceValue::DeviceValueTAG_s[tag][0], Helpers::toLower(uniq_s).c_str());
	} else {
	snprintf(uniq_id, sizeof(uniq_id), "%s_%s", device_name, Helpers::toLower(uniq_s).c_str());
	}

[tp1de]

Since you introduced the new device water, I ask myself why haven't you put all dhw entities in this new device.
(km200 name: dhwCuircits instead of water ..... and dhw should be dhw1)

[tp1de]

I started to check and test: It would be possible write a script (e.g. within node-red in HA) to rename the entities in the lovelace files and transferring the statistics to the new entities if MariaDB / mySQL is used. But I do not think that this is useful for non-experienced users.

[MichaelDvP]

Because the modules are registered by device_id: 0x28 is mixer/dhw1, 0x20 is mixer/hc1, but 0x08 is boiler, boiler/dhw does not have a own id. Same for thermostat.

[tp1de]

Most important point is for me not to change boiler_nrgww into boiler_dhw_nrgdhw for use within energy dashboard.

[proddy]

I started to check and test: It would be possible write a script (e.g. within node-red in HA) to rename the entities in the lovelace files and transferring the statistics to the new entities if MariaDB / mySQL is used. But I do not think that this is useful for non-experienced users.

pretty sure you can rename entities (and ids) directly from the HA UI, haven't looked. There's also a script in https://emsesp.github.io/docs/Troubleshooting/#ha-has-messed-up-the-names-of-my-entities. But the point is here: we should respect the history somehow. I'll need to think of what to do, either a) build backwards compatibility or b) brute force it and annoy users. If EMS-ESP was commercial and we were financially impacted by user dissatisfaction, then I would think differently. It's still a hobby project outside my 80hr work week. So mny preference is still b.

[tp1de]

If your final decision is b, please make sure to keep boiler_nrgww in mqtt discovery.

[MichaelDvP]

We already have the v3.4 compatibility as mqtt setting, we can add a setting for v3.6 updates.

and set this automatically on upgrade from 3.6.
I've made a 3.7-dev5. @tp1de please check.

• this also registers only one dhw circuit if second is not active. Working with my RC35, other thermostats needs test.
• update from v3.6 should keep the HA entities and switch to new v3.6 compatible mode
• update from 3.7 keeps the setting you had before.
• for multi instances i have no complatible mode,but can be build.

First try, a bit tuning expected.

[tp1de]

I've made a 3.7-dev5. @tp1de please check.

I downloaded from here: https://github.com/MichaelDvP/EMS-ESP32/releases/tag/latest

But the new ID Format is not shown / selectable

[MichaelDvP]

refresh browser cache (reload page)

[tp1de]

Thanks .... works. You are fast Michael :)
First view seems to be ok. Old uniq_id and obj_id is used within new dhw structure.
Shall I check anything more?

[MichaelDvP]

Shall I check anything more?

Is the customization for RC310 still showing dhw2 entities?
All thermostat dhw entities ok? The thermostat dhw was a bigger change.

[tp1de]

Is the customization for RC310 still showing dhw2 entities?

NO - It is ok.

All thermostat dhw entities ok? The thermostat dhw was a bigger change.

As far as I can see for the moment, all entities should be ok by checking with mqtt-explorer.
I will make a clone of my HA system with proxmox and I will connect to the 2nd mqtt-server to check everything.
It will take a while.

[tp1de]

@MichaelDvP HA clone is up and running with a 2nd mqtt server and your 3.7.0-dev.5 firmware on my test gateway.

I checked all lovelace ui's, automations and node-red scripts. Everything is running the same as on my HA production system.

Well done !

[MichaelDvP]

All thermostat dhw entities ok? The thermostat dhw was a bigger change.
As far as I can see for the moment, all entities should be ok by checking with mqtt-explorer.

Also in dashboard and customizations?

HA clone is up and running with a 2nd mqtt server

If youre playing around with 2nd server, downgrade to 3.6.5, set mqtt option to single/short or multi/short, recreate the HA settings.
Then download and use my new updated version (also dev5), and update ems-esp. This should switch to the compatible version (for all users who upgrade).

[tp1de]

Weird behavior within HA.

1. After downgrading to 3.6.5 alle entities are still active within HA.
2. After upgrading to your last version the mqtt settings are correct.
3. Using mqtt explorer all HA discovery items are correct.
4. But now within HA most dhw entities are not active anymore. Restarting HA or the complete system doesn't help.
5. Stopping mqtt in ems-esp and then deleting the the ems-esp devices boiler and thermostat manually within HA
6. All ems-esp entities are rebuild afterwards correctly after starting mqtt within the gateway.

There might be a problem with HA discovery mode. Or do you have a different idea?

[MichaelDvP]

Hm, i think after upgrade you have 2 mqtt topics with config for the same unique_id. eg.
homeassistant/binary_sensor/ems-esp/boiler_ww3wayvalve/config
homeassistant/binary_sensor/ems-esp/boiler_dhw_3wayvalve/config
and you need to delete one to get the other active.
I'm not using HA, can't test.

[proddy]

what I always do I just go to MQTTExplorer, delete everything ems-esp and restart EMS-ESP. Then you get a clean install

[MichaelDvP]

@tp1de I've adapted the topics, if it's not too frustrating you can test again.

[tp1de]

@tp1de I've adapted the topics, if it's not too frustrating you can test again.

Yes this version works, since you use the same mqtt topics as in 3.6.5.
So the steps from installing 3.6.5 to migrating to your last dev version runs without problems.
All HA mqtt entities are kept active as before.

[tp1de]

@proddy Are all changes within 3.7.0-dev.5 from today?
If yes, we can close this issue.

[proddy]

yes, thanks Thomas.