Feature: Modbus Support

[proddy]

Creating this issue to track issues on the Modbus feature.

@mheyse did all the coding, in this PR #1618

Code is in this branch: https://github.com/emsesp/EMS-ESP32/tree/feat_modbus

[proddy]

I've added the Web settings.

Also noticed with modbus enabled it takes up 17kb of heap memory, and 13 when disabled. We need to try and bring that back. I haven't looked into all the code yet.

[mheyse]

How exactly do you compare the heap usage? Build the firmware from dev and from feat_modbus and compare System -> System Memory -> Heap? Or is there a better way?

I suspect that the data I allocate in modbus_entity_parameters.cpp is not stored in Flash (as I thought it would) if it takes up 13kB of heap memory even when disabled.

There are other optimizations I can do (e.g. I register 4 modbus workers each for all ems devices, this could be changed to use ANY_SERVER and ANY_FUNCTION_CODE wildcards)

[proddy]

yes, I actually use a python script for memory profiling but you can also use the WebUI or just read the "free mem" from the JSON output of http://ems-esp.local/api/system

The Modbus is a great feature and easily fits on modern ESP32s like the S3, but it will struggle on 4MB variants. We just need to find ways to reduce the amount of heap memory if Modbus isn't been used. When it's enabled it's fine as that is the price to pay for a new module/feature.

I see the object ModbusServerTCPasync instantiated as default but this can't be the main culprit, something else is eating the memory, even when it's disabled.

[proddy]

could you possibly instantiate ModbusServerTCPasync from within Modbus::start() and just use static pointers? This may save on heap memory.

Also check that LOCAL_LOG_LEVEL is not set, unless EMSESP_DEBUG is.

I'm working on a new C++ Factory class so we can try this as a hot-pluggable module. Could you describe which integration points you have with EMS-ESP? I see the start() but you have nothing in the loop() ?

[proddy]

dev has 190KB free heap, feat_modbus with Modbus disabled and using the Michael's change to static Modbus * modbus_ is 177KB so we're still eating up 13KB of memory even if not using Modbus. We need to bring this down before accepting into the dev branch. Can you see what you can do @mheyse ?

[mheyse]

Yes. The issue is the static data with the Modbus Parameters. It ends up in the .flash.text section but apparently it's copied to RAM anyways, even after I've made everything const. I guess I have to familiarize myself with the ESP32 memory model to debug this.If all fails I can still try to embed the data as a static binary but that will be pretty ugly.Allocating the Modbus client dynamically in start() doesn't help at all by the way.Am 09.05.2024 16:57 schrieb Proddy ***@***.***>: dev has 190KB free heap, feat_modbus with Modbus disabled and using the Michael's change to static Modbus * modbus_ is 177KB so we're still eating up 13KB of memory even if not using Modbus. We need to bring this down before accepting into the dev branch. Can you see what you can do @mheyse ? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

The design pattern is that Modbus should allocate as little to no heap memory if the service is not enabled. So something is still lurking in the headerfiles and eating up the heap during runtime (not static, progmem/flash). I still suspect it's the creation of the ModbusServerTCPasync object in modbus.cpp - see if you can turn that into a pointer and new-it when it needs to be created.

[mheyse]

I did make the ModbusServerTCPasync a pointer and allocate it dynamically in "Modbus::start" using new - no change at all in memory consumption if Modbus is disabled. I'll push the changes anyways. On the other hand, if (just as a test) I remove all entries from modbus_register_mappings this reduces memory consumption by about 10kB if Modbus is not enabled - so it's this static data that somehow allocates heap memory. I have to figure out how to solve that because it should be possible to have this static data in flash only. Unfortunately I have a project to finish at work and not a lot of spare time, but I'll look into it as soon as possible.

…

------ Originalnachricht ------ Von "Proddy" ***@***.***> An "emsesp/EMS-ESP32" ***@***.***> Cc "mheyse" ***@***.***>; "Mention" ***@***.***> Datum 10.05.2024 10:16:27 Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)

The design pattern is that Modbus should allocate as little to no heap memory if the service is not enabled. So something is still lurking in the headerfiles and eating up the heap during runtime (not static, progmem/flash). I still suspect it's the creation of the ModbusServerTCPasync object in modbus.cpp - see if you can turn that into a pointer and new-it when it needs to be created. — Reply to this email directly, view it on GitHub <#1744 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJD5GWEIQNE6BM4B25OCM3ZBR65XAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBUGE2TQNJTHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

that's probably it - missed that. The modbus_register_mappings list and tag_to_type map. Is there a way to populate both these in the start() function using static_ptr? Also watch out for that Serial.print() in the start() function - sending Serial on a live EMS-ESP can cause havok since we're spoofing the UART for the EMS bus.

even better is to make it dynamic, like we do with HA MQTT Discovery so the modbus registry's are only created when a valid ems device entity is found.

[mheyse]

FYI, I submitted another PR addressing the memory iasues: #1770

[proddy]

thanks, I'll take a look and run some more benchmarking. I've been out the last weeks. I still need to fix the Web settings right?

[mheyse]

Yes, I didn't do anything regarding the web settings.Am 28.05.2024 13:45 schrieb Proddy ***@***.***>: thanks, I'll take a look and run some more benchmarking. I've been out the last weeks. I still need to fix the Web settings right? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

I made some small changes to the branch to fix the UI settings.

Your latest changes are good with memory
without modbus code its 195 / 107 (free heap / max alloc)
with modbus disabled its 192 / 107
with modbus enabled its 187 / 107

So this feature only takes up 3kb extra, which is acceptable.

[mheyse]

Great! Let's discuss how to integrate the modbus parameter generation into the build process. Currently you need to do the following: - first, dump all entities to csv by running 'scripts/dump_entities.sh' - then run 'cat ../dump_entities.csv | python3 update_modbus_registers.py > ../src/modbus_entity_parameters.hpp' from the "scripts" folder - build update_modbus_registers.py assigns modbus registers to all entities that do not have a register yet, keeping already assigned registers the same as before. Then it generates the c++ code with the parameters and writes it to stdout. If there are any breaking changes and old regsiter assignments can not be re-used, all parameters must be deleted from modbus_entity_parameters.hpp manually before running dump_entities.sh - this re-assigns registers to all entities (possibly changing registers). EMS-ESP32 doesn't seem to know the size of string entities, so it's not possible to add this to the dump (is that correct?) - that's why the size of all string entities has to be defined manually in update_modbus_registers.py. Is this process acceptable or do you have any suggestions to improve it? (as an alternative you could maintain modbus_entity_parameters.hpp completely by hand, modify it whenever entities change but that is cumbersome)

…

------ Originalnachricht ------ Von "Proddy" ***@***.***> An "emsesp/EMS-ESP32" ***@***.***> Cc "mheyse" ***@***.***>; "Mention" ***@***.***> Datum 04.06.2024 13:49:09 Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)

I made some small changes to the branch to fix the UI settings. Your latest changes are good with memory without modbus code its 195 / 107 (free heap / max alloc) with modbus disabled its 192 / 107 with modbus enabled its 187 / 107 So this feature only takes up 3kb extra, which is acceptable. — Reply to this email directly, view it on GitHub <#1744 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJD5GWIYTUS342PF4CLJTTZFWSTLAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXGMZTENRWGE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

Is there a way to add the Modbus registers dynamically as soon as they show up, like we do with MQTT Discovery?

[mheyse]

I think it is really important to have a stable register assignment so an entity's Modbus register does not change if other entities are added or removed. So the register offset (and the register count, at least for string entities) needs to be stored somewhere which is currently modbus_entity_parameters.hpp. (BTW, generating this file as part of the build process as I suggested earlier makes no sense as it has to be committed to the repo to ensure stable register assignments so it needs to be regenerated after adding or removing entities)It would also be possible to modify register_device_value() and add a modbus register offset and register count parameter, but maintaining this would be a manual process and quite painful.I can't think of a way to dynamically assign stable registers. It works for MQTT because it uses the entity names which are defined anyways with register_device_value.A clean solution might be to define all entities including their name, unit, data type, size etc. and also the Modbus parameters in a separate data file (e.g. a csv that is maybe compiled into a binary representation and embedded into the firmware image - it would be basically the dump_entities.csv file but it's the authority instead of an export). This file could be maintained manually (e.g. for adding new entities) and with tools (add Modbus parameters for new entities). Entries in this data structure would be referenced by the devices to register entities. But this would be a major development effort.So I guess for now it makes the most sense to document that update_modbus_registers.py needs to run after modifying entities. And maybe improve it so that it automatically dumps entities and generates code.Am 04.06.2024 22:58 schrieb Proddy ***@***.***>: Is there a way to add the Modbus registers dynamically as soon as they show up, like we do with MQTT Discovery? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

Ok, I understand. I'll see if I can run the generation of modbus_entity_parameters.hpp as part of the platformio build target

[proddy]

I looked into automating this today, but it wouldn't compile. Seems running the generation of src/modbus_entity_parameters.hpp is not compatible with the modified class signatures. Can you take a look?

[proddy]

@mheyse any progress on this? It would be good to finally include it 3.7.0

[mheyse]

I was on vacation then sick for quite a while. Hope to have time for it soon.Am 21.06.2024 20:07 schrieb Proddy ***@***.***>: @mheyse any progress on this? It would be good to finally include it 3.7.0 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

Sorry to hear you were unwell. There's no rush - let me know how I can help

[mheyse]

OK, I updated update_modbus_registers.py to reflect the changes I did for memory consuption improvements, forgot to do that. How exactly do you plan to integrate it into the build process? As the generated modbus_entity_parameters.hpp needs to be committed to the repository after it has been updated with update_modbus_registers.py I don't think it can go into the build process. (I initially wrote that it should be added to the build process, but later stated that does not make sense in my opinion. Sorry for the confusion.) I think the best way is to add another shell script that automatically dumps the entities and passes them to update_modbus_registers.py similar to dump_entities.sh (e.g. named update_modbus_registers.sh). This script would need to be executed manually whenever entities have been added, and then the updated modbus_entity_parameters.hpp needs to be committed.

…

------ Originalnachricht ------ Von "Proddy" ***@***.***> An "emsesp/EMS-ESP32" ***@***.***> Cc "mheyse" ***@***.***>; "Mention" ***@***.***> Datum 21.06.2024 20:07:51 Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)

@mheyse <https://github.com/mheyse> any progress on this? It would be good to finally include it 3.7.0 — Reply to this email directly, view it on GitHub <#1744 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJD5GQKP5NHOTP6JBKTYT3ZIRTXPAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTGIYDSMZZGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

yes, I think it needs to be updated manually by running the script and the updated modbus_entity_parameters.hpp file checked in.

Is there some documentation you like to write, which explains what it does - which we can add to the wiki?

[proddy]

@mheyse I checked the code I think it's good to go. Can you refresh your local fork of the dev branch, then Git Merge your local copy of the feat_modbus branch? Then test and if it's good create a PR on dev. That way you'll own the feature and the PR.

[mheyse]

Yes I want to write documentation for devs and end users including generated tables with the modbus registers and data format for all entities. So far I haven't gotten much further than checking out the repo though.Am 29.06.2024 08:57 schrieb Proddy ***@***.***>: yes, I think it needs to be updated manually by running the script and the updated modbus_entity_parameters.hpp file checked in. Is there some documentation you like to write, which explains what it does - which we can add to the wiki? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[mheyse]

@proddy I merged with the current dev and created a PR against dev. Please note: - I changed the modbus entity parameters from an array to an initializer_list. In my tests this does not change memory consumption (I compared it to 3.7.0-dev.19 you released yesterday), but please double check. This is required for standalone builds. - There were merge conflicts in the interface which I didn't implement, so please check if everything is correct. I resolved them like this: For these files I used the version from dev and added the ENABLE_MODBUS translations: interface/src/i18n/de/index.ts interface/src/i18n/en/index.ts interface/src/i18n/fr/index.ts interface/src/i18n/it/index.ts interface/src/i18n/nl/index.ts interface/src/i18n/no/index.ts interface/src/i18n/pl/index.ts interface/src/i18n/sk/index.ts interface/src/i18n/sv/index.ts interface/src/i18n/tr/index.ts For these files I used the version from dev without modification: interface/package.json interface/src/project/validators.ts interface/yarn.lock mock-api/rest_server.ts src/version.h - I added a the script "scripts/update_modbus_registers.sh". If you run this script without arguments it performs a standalone build, dumps all entities, processes them and generates a new ./src/modbus_entity_parameters.hpp. So this needs to be run everytime new entities are added. - You can pass the argument "--reset" to update_modbus_registers.sh. This effectively clears modbus_entity_parameters.hpp before regenerating it. This might cause breaking changes for end users (modbus registers may change) and should only be used if modbus_entity_parameters.hpp is broken due to other breaking changes anyways. "--force" skips the confirmation question that "--reset" usually requires. - For testing, I ran a build from a clean working copy with "pio run -e esp32_16M". I tested on my bbqkees GatewayE32 V2 with both Modbus disabled and enabled.

…

------ Originalnachricht ------ Von "Proddy" ***@***.***> An "emsesp/EMS-ESP32" ***@***.***> Cc "mheyse" ***@***.***>; "Mention" ***@***.***> Datum 29.06.2024 11:08:54 Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)

@mheyse <https://github.com/mheyse> I checked the code I think it's good to go. Can you refresh your local fork of the dev branch, then Git Merge your local copy of the feat_modbus branch? Then test and if it's good create a PR on dev. That way you'll own the feature and the PR. — Reply to this email directly, view it on GitHub <#1744 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJD5GQCRX3SOJ3MR53AEQDZJZ2SNAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJYGA2TMOJVGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mheyse]

I submitted a PR for the documentation and also another PR to EMS-ESP32 with a script that auto generates the Modbus entity parameter documentation. Please review, from my point of view everything should be ready to be released.

…

Message ID: ***@***.***>

[proddy]

Thanks for that - we'll review.

[proddy]

merged.

[proddy]

@mheyse - I'm getting compile errors when compiling with -DEMSESP_TEST. It happens here:

#if defined(EMSESP_STANDALONE) || defined(EMSESP_TEST)
#include <modbus_test.h>
#endif

perhaps a namespace clash with Error

[mheyse]

How do I reproduce this? If I run make clean make ARGS=-DEMSESP_TEST the build process runs without error here (Ubuntu 22.04) and I can run the emsesp binary. However, my tests do not seem to work any more, I get "unrecognized path" errors, I need to check why.

…

------ Originalnachricht ------ Von "Proddy" ***@***.***> An "emsesp/EMS-ESP32" ***@***.***> Cc "mheyse" ***@***.***>; "Mention" ***@***.***> Datum 04.07.2024 17:12:26 Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)

@mheyse <https://github.com/mheyse> - I'm getting compile errors when compiling with -DEMSESP_TEST. It happens here: #if defined(EMSESP_STANDALONE) || defined(EMSESP_TEST) #include <modbus_test.h> #endif perhaps a namespace clash with Error — Reply to this email directly, view it on GitHub <#1744 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJD5GSNDLXWSXWHZEZJMETZKVQ5VAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBZGIYDOMZXGE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[proddy]

set my_build_flags = -DEMSESP_TEST in a pio_local.ini or change it directly in platform.ini and the compile breaks.

We use EMSESP_TEST for creating real ESP32 firmware for testing on ESP32's, not just in standalone mode.

[mheyse]

Ah OK, I thought EMSESP_TEST is only used for testing on PCs. In this case the eModbus libraries are available, too, and || defined(EMSESP_TEST) can simply be removed.

Fix: #1853

[wint100]

Is there a standard Modbus register list available?

[mheyse]

I created a PR with the docs here: emsesp/docs#36
It also documents the modbus registers.

[wint100]

I'm having issues writing to the 'Heating Temperature' using Modbus. I can read back all values, but when trying to write to Holding Register 14 with a uint16 value of 50, return an error on my Modbus Master. I've tried different Modbus Master hardware but both have the same result. Is there anything else I need to do when writing to a Holding Register?

[mheyse]

Please look for error messages in the system log and post them here.What error code do you get?Am 22.07.2024 17:12 schrieb wint100 ***@***.***>: I'm having issues writing to the 'Heating Temperature' using Modbus. I can read back all values, but when trying to write to Holding Register 14 with a uint16 value of 50, return an error on my Modbus Master. I've tried different Modbus Master hardware but both have the same result. Is there anything else I need to do when writing to a Holding Register? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[wint100]

It appears to work on the S3 hardware but not on the older hardware. I have 20 of the S3 boards and these work well, but the older E32 boards (ESP32-D0WD-V3) don't seem to allow writing of the Heating Temperature.

Error from older board
2024-07-23 08:42:12.268 E 140: [modbus] Modbus write command failed with error: unrecognized path (Error)

[proddy]

This is in 3.7.0 now, so closing. Any issues/bugs can be reported using the normal channels.