-
-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: Modbus Support #1744
Comments
I've added the Web settings. Also noticed with modbus enabled it takes up 17kb of heap memory, and 13 when disabled. We need to try and bring that back. I haven't looked into all the code yet. |
How exactly do you compare the heap usage? Build the firmware from I suspect that the data I allocate in There are other optimizations I can do (e.g. I register 4 modbus workers each for all ems devices, this could be changed to use |
yes, I actually use a python script for memory profiling but you can also use the WebUI or just read the "free mem" from the JSON output of The Modbus is a great feature and easily fits on modern ESP32s like the S3, but it will struggle on 4MB variants. We just need to find ways to reduce the amount of heap memory if Modbus isn't been used. When it's enabled it's fine as that is the price to pay for a new module/feature. I see the object |
could you possibly instantiate Also check that I'm working on a new C++ Factory class so we can try this as a hot-pluggable module. Could you describe which integration points you have with EMS-ESP? I see the start() but you have nothing in the loop() ? |
dev has 190KB free heap, feat_modbus with Modbus disabled and using the Michael's change to |
Yes. The issue is the static data with the Modbus Parameters. It ends up in the .flash.text section but apparently it's copied to RAM anyways, even after I've made everything const. I guess I have to familiarize myself with the ESP32 memory model to debug this.If all fails I can still try to embed the data as a static binary but that will be pretty ugly.Allocating the Modbus client dynamically in start() doesn't help at all by the way.Am 09.05.2024 16:57 schrieb Proddy ***@***.***>:
dev has 190KB free heap, feat_modbus with Modbus disabled and using the Michael's change to static Modbus * modbus_ is 177KB so we're still eating up 13KB of memory even if not using Modbus. We need to bring this down before accepting into the dev branch. Can you see what you can do @mheyse ?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
The design pattern is that Modbus should allocate as little to no heap memory if the service is not enabled. So something is still lurking in the headerfiles and eating up the heap during runtime (not static, progmem/flash). I still suspect it's the creation of the ModbusServerTCPasync object in |
I did make the ModbusServerTCPasync a pointer and allocate it
dynamically in "Modbus::start" using new - no change at all in memory
consumption if Modbus is disabled. I'll push the changes anyways.
On the other hand, if (just as a test) I remove all entries from
modbus_register_mappings this reduces memory consumption by about 10kB
if Modbus is not enabled - so it's this static data that somehow
allocates heap memory. I have to figure out how to solve that because it
should be possible to have this static data in flash only.
Unfortunately I have a project to finish at work and not a lot of spare
time, but I'll look into it as soon as possible.
…------ Originalnachricht ------
Von "Proddy" ***@***.***>
An "emsesp/EMS-ESP32" ***@***.***>
Cc "mheyse" ***@***.***>; "Mention" ***@***.***>
Datum 10.05.2024 10:16:27
Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)
The design pattern is that Modbus should allocate as little to no heap
memory if the service is not enabled. So something is still lurking in
the headerfiles and eating up the heap during runtime (not static,
progmem/flash). I still suspect it's the creation of the
ModbusServerTCPasync object in modbus.cpp - see if you can turn that
into a pointer and new-it when it needs to be created.
—
Reply to this email directly, view it on GitHub
<#1744 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJD5GWEIQNE6BM4B25OCM3ZBR65XAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBUGE2TQNJTHA>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
that's probably it - missed that. The even better is to make it dynamic, like we do with HA MQTT Discovery so the modbus registry's are only created when a valid ems device entity is found. |
FYI, I submitted another PR addressing the memory iasues: #1770 |
thanks, I'll take a look and run some more benchmarking. I've been out the last weeks. I still need to fix the Web settings right? |
Yes, I didn't do anything regarding the web settings.Am 28.05.2024 13:45 schrieb Proddy ***@***.***>:
thanks, I'll take a look and run some more benchmarking. I've been out the last weeks. I still need to fix the Web settings right?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
I made some small changes to the branch to fix the UI settings. Your latest changes are good with memory So this feature only takes up 3kb extra, which is acceptable. |
Great!
Let's discuss how to integrate the modbus parameter generation into the
build process. Currently you need to do the following:
- first, dump all entities to csv by running 'scripts/dump_entities.sh'
- then run 'cat ../dump_entities.csv | python3
update_modbus_registers.py > ../src/modbus_entity_parameters.hpp' from
the "scripts" folder
- build
update_modbus_registers.py assigns modbus registers to all entities that
do not have a register yet, keeping already assigned registers the same
as before. Then it generates the c++ code with the parameters and writes
it to stdout. If there are any breaking changes and old regsiter
assignments can not be re-used, all parameters must be deleted from
modbus_entity_parameters.hpp manually before running dump_entities.sh -
this re-assigns registers to all entities (possibly changing registers).
EMS-ESP32 doesn't seem to know the size of string entities, so it's not
possible to add this to the dump (is that correct?) - that's why the
size of all string entities has to be defined manually in
update_modbus_registers.py.
Is this process acceptable or do you have any suggestions to improve it?
(as an alternative you could maintain modbus_entity_parameters.hpp
completely by hand, modify it whenever entities change but that is
cumbersome)
…------ Originalnachricht ------
Von "Proddy" ***@***.***>
An "emsesp/EMS-ESP32" ***@***.***>
Cc "mheyse" ***@***.***>; "Mention" ***@***.***>
Datum 04.06.2024 13:49:09
Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)
I made some small changes to the branch to fix the UI settings.
Your latest changes are good with memory
without modbus code its 195 / 107 (free heap / max alloc)
with modbus disabled its 192 / 107
with modbus enabled its 187 / 107
So this feature only takes up 3kb extra, which is acceptable.
—
Reply to this email directly, view it on GitHub
<#1744 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJD5GWIYTUS342PF4CLJTTZFWSTLAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXGMZTENRWGE>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Is there a way to add the Modbus registers dynamically as soon as they show up, like we do with MQTT Discovery? |
I think it is really important to have a stable register assignment so an entity's Modbus register does not change if other entities are added or removed. So the register offset (and the register count, at least for string entities) needs to be stored somewhere which is currently modbus_entity_parameters.hpp. (BTW, generating this file as part of the build process as I suggested earlier makes no sense as it has to be committed to the repo to ensure stable register assignments so it needs to be regenerated after adding or removing entities)It would also be possible to modify register_device_value() and add a modbus register offset and register count parameter, but maintaining this would be a manual process and quite painful.I can't think of a way to dynamically assign stable registers. It works for MQTT because it uses the entity names which are defined anyways with register_device_value.A clean solution might be to define all entities including their name, unit, data type, size etc. and also the Modbus parameters in a separate data file (e.g. a csv that is maybe compiled into a binary representation and embedded into the firmware image - it would be basically the dump_entities.csv file but it's the authority instead of an export). This file could be maintained manually (e.g. for adding new entities) and with tools (add Modbus parameters for new entities). Entries in this data structure would be referenced by the devices to register entities. But this would be a major development effort.So I guess for now it makes the most sense to document that update_modbus_registers.py needs to run after modifying entities. And maybe improve it so that it automatically dumps entities and generates code.Am 04.06.2024 22:58 schrieb Proddy ***@***.***>:
Is there a way to add the Modbus registers dynamically as soon as they show up, like we do with MQTT Discovery?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Ok, I understand. I'll see if I can run the generation of |
I looked into automating this today, but it wouldn't compile. Seems running the generation of |
@mheyse any progress on this? It would be good to finally include it 3.7.0 |
I was on vacation then sick for quite a while. Hope to have time for it soon.Am 21.06.2024 20:07 schrieb Proddy ***@***.***>:
@mheyse any progress on this? It would be good to finally include it 3.7.0
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Sorry to hear you were unwell. There's no rush - let me know how I can help |
OK, I updated update_modbus_registers.py to reflect the changes I did
for memory consuption improvements, forgot to do that.
How exactly do you plan to integrate it into the build process? As the
generated modbus_entity_parameters.hpp needs to be committed to the
repository after it has been updated with update_modbus_registers.py I
don't think it can go into the build process. (I initially wrote that it
should be added to the build process, but later stated that does not
make sense in my opinion. Sorry for the confusion.)
I think the best way is to add another shell script that automatically
dumps the entities and passes them to update_modbus_registers.py similar
to dump_entities.sh (e.g. named update_modbus_registers.sh). This script
would need to be executed manually whenever entities have been added,
and then the updated modbus_entity_parameters.hpp needs to be committed.
…------ Originalnachricht ------
Von "Proddy" ***@***.***>
An "emsesp/EMS-ESP32" ***@***.***>
Cc "mheyse" ***@***.***>; "Mention" ***@***.***>
Datum 21.06.2024 20:07:51
Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)
@mheyse <https://github.com/mheyse> any progress on this? It would be
good to finally include it 3.7.0
—
Reply to this email directly, view it on GitHub
<#1744 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJD5GQKP5NHOTP6JBKTYT3ZIRTXPAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTGIYDSMZZGU>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
yes, I think it needs to be updated manually by running the script and the updated Is there some documentation you like to write, which explains what it does - which we can add to the wiki? |
@mheyse I checked the code I think it's good to go. Can you refresh your local fork of the |
Yes I want to write documentation for devs and end users including generated tables with the modbus registers and data format for all entities. So far I haven't gotten much further than checking out the repo though.Am 29.06.2024 08:57 schrieb Proddy ***@***.***>:
yes, I think it needs to be updated manually by running the script and the updated modbus_entity_parameters.hpp file checked in.
Is there some documentation you like to write, which explains what it does - which we can add to the wiki?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
@proddy I merged with the current dev and created a PR against dev.
Please note:
- I changed the modbus entity parameters from an array to an
initializer_list. In my tests this does not change memory consumption (I
compared it to 3.7.0-dev.19 you released yesterday), but please double
check. This is required for standalone builds.
- There were merge conflicts in the interface which I didn't implement,
so please check if everything is correct. I resolved them like this:
For these files I used the version from dev and added the ENABLE_MODBUS
translations:
interface/src/i18n/de/index.ts
interface/src/i18n/en/index.ts
interface/src/i18n/fr/index.ts
interface/src/i18n/it/index.ts
interface/src/i18n/nl/index.ts
interface/src/i18n/no/index.ts
interface/src/i18n/pl/index.ts
interface/src/i18n/sk/index.ts
interface/src/i18n/sv/index.ts
interface/src/i18n/tr/index.ts
For these files I used the version from dev without modification:
interface/package.json
interface/src/project/validators.ts
interface/yarn.lock
mock-api/rest_server.ts
src/version.h
- I added a the script "scripts/update_modbus_registers.sh". If you run
this script without arguments it performs a standalone build, dumps all
entities, processes them and generates a new
./src/modbus_entity_parameters.hpp. So this needs to be run everytime
new entities are added.
- You can pass the argument "--reset" to update_modbus_registers.sh.
This effectively clears modbus_entity_parameters.hpp before regenerating
it. This might cause breaking changes for end users (modbus registers
may change) and should only be used if modbus_entity_parameters.hpp is
broken due to other breaking changes anyways. "--force" skips the
confirmation question that "--reset" usually requires.
- For testing, I ran a build from a clean working copy with "pio run -e
esp32_16M". I tested on my bbqkees GatewayE32 V2 with both Modbus
disabled and enabled.
…------ Originalnachricht ------
Von "Proddy" ***@***.***>
An "emsesp/EMS-ESP32" ***@***.***>
Cc "mheyse" ***@***.***>; "Mention" ***@***.***>
Datum 29.06.2024 11:08:54
Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)
@mheyse <https://github.com/mheyse> I checked the code I think it's
good to go. Can you refresh your local fork of the dev branch, then Git
Merge your local copy of the feat_modbus branch? Then test and if it's
good create a PR on dev. That way you'll own the feature and the PR.
—
Reply to this email directly, view it on GitHub
<#1744 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJD5GQCRX3SOJ3MR53AEQDZJZ2SNAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJYGA2TMOJVGU>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I submitted a PR for the documentation and also another PR to EMS-ESP32
with a script that auto generates the Modbus entity parameter
documentation.
Please review, from my point of view everything should be ready to be
released.
…Message ID: ***@***.***>
|
Thanks for that - we'll review. |
merged. |
@mheyse - I'm getting compile errors when compiling with
perhaps a namespace clash with |
How do I reproduce this? If I run
make clean
make ARGS=-DEMSESP_TEST
the build process runs without error here (Ubuntu 22.04) and I can run
the emsesp binary.
However, my tests do not seem to work any more, I get "unrecognized
path" errors, I need to check why.
…------ Originalnachricht ------
Von "Proddy" ***@***.***>
An "emsesp/EMS-ESP32" ***@***.***>
Cc "mheyse" ***@***.***>; "Mention" ***@***.***>
Datum 04.07.2024 17:12:26
Betreff Re: [emsesp/EMS-ESP32] Feature: Modbus Support (Issue #1744)
@mheyse <https://github.com/mheyse> - I'm getting compile errors when
compiling with -DEMSESP_TEST. It happens here:
#if defined(EMSESP_STANDALONE) || defined(EMSESP_TEST)
#include <modbus_test.h>
#endif
perhaps a namespace clash with Error
—
Reply to this email directly, view it on GitHub
<#1744 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJD5GSNDLXWSXWHZEZJMETZKVQ5VAVCNFSM6AAAAABHIYXM6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBZGIYDOMZXGE>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
set We use EMSESP_TEST for creating real ESP32 firmware for testing on ESP32's, not just in standalone mode. |
Ah OK, I thought EMSESP_TEST is only used for testing on PCs. In this case the eModbus libraries are available, too, and Fix: #1853 |
Is there a standard Modbus register list available? |
I created a PR with the docs here: emsesp/docs#36 |
I'm having issues writing to the 'Heating Temperature' using Modbus. I can read back all values, but when trying to write to Holding Register 14 with a uint16 value of 50, return an error on my Modbus Master. I've tried different Modbus Master hardware but both have the same result. Is there anything else I need to do when writing to a Holding Register? |
Please look for error messages in the system log and post them here.What error code do you get?Am 22.07.2024 17:12 schrieb wint100 ***@***.***>:
I'm having issues writing to the 'Heating Temperature' using Modbus. I can read back all values, but when trying to write to Holding Register 14 with a uint16 value of 50, return an error on my Modbus Master. I've tried different Modbus Master hardware but both have the same result. Is there anything else I need to do when writing to a Holding Register?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
It appears to work on the S3 hardware but not on the older hardware. I have 20 of the S3 boards and these work well, but the older E32 boards (ESP32-D0WD-V3) don't seem to allow writing of the Heating Temperature. Error from older board |
This is in 3.7.0 now, so closing. Any issues/bugs can be reported using the normal channels. |
Creating this issue to track issues on the Modbus feature.
@mheyse did all the coding, in this PR #1618
Code is in this branch: https://github.com/emsesp/EMS-ESP32/tree/feat_modbus
The text was updated successfully, but these errors were encountered: