Several medium sized improvements #92

jaredmauch · 2025-06-20T03:06:01Z

add json output to ease testing
add influxdb v1 support for ease of use with grafana
improve eg4 handling

* add json output to ease testing * add influxdb v1 support for ease of use with grafana

HotNoob · 2025-06-20T03:13:50Z

very nice.
i'll need some time to review all of this.

HotNoob · 2025-06-20T03:22:34Z

config.influxdb.example

@@ -0,0 +1,22 @@
+[influxdb_output]
+type = influxdb_out


need to verify "type" is an alias of "transport"

Oh, no religion here, change at will :-)

HotNoob · 2025-06-20T03:36:56Z

classes/protocol_settings.py

@@ -626,7 +626,7 @@ def process_row(row):
                                concatenate_registers.append(i)

            if concatenate_registers:
-                r = range(len(concatenate_registers))
+                r = range(1)  # Only create one entry for concatenated variables


hmm. i dont remember enough to know if this is ok... :P

HotNoob · 2025-06-20T03:52:10Z

classes/transports/modbus_base.py

-            return results[entry.variable_name]
+
+            # Special handling for concatenated ASCII variables (like serial numbers)
+            if entry.concatenate and entry.data_type == Data_Type.ASCII:


i'm not sure why this change is needed?
is there a byte order or encoding issue with eg4 devices?

So it returns a word and you need to split it into the two bytes low and high. In the CSV this is done, but maybe the Serial Number thing in the concat isn't quite right as it needs to concat the high/low bytes? Recommendations of better ways to do this are welcome. The intent is to have multiple inverters going to my influxdb and then can match on the serial of the inverter so if the usb/rs485 cables are shuffled at some point the actual inverter serial is reported as the tag to influxdb/json/whatnot for eg4 - have the 12kpv and 18kpv inverters I'm wiring to this

oh ok. yeah it's just how it's done in the csv.

by default, i just do it the same way that it's documented. the eg4 one isnt one that i've personally tested.

this could easily be changed to:

ASCII 115~119

alternatively the serial number can be manually configured via cfg.

if you can send me a dump of the input and holding registers, that would allow me to emulate and test things in detail.

analyze_protocol = true analyze_protocol_save_load = true

run once and exit.
should create two files: input_registry.json, holding_registry.json

then i can use that to test in detail.

4db8362
v1.1.10 branch

should theorectically do the trick...

I also needed to do this for fwcode as well, otherwise it was in the wrong byte order, so I think this was needed. Let me get you what you are asking for though :-)

HotNoob · 2025-06-20T04:01:12Z

your doing well with navigating through ppg :D

the new transports look ok.

but the concatination changes + ascii handling seems a bit off.
i think your trying to fix an ASCII encoding or byte order problem?

that stuff is done here:

PythonProtocolGateway/classes/protocol_settings.py

Line 1079 in 86b7e64

elif entry.data_type == Data_Type.ASCII:

- Fix AttributeError when accessing ModbusIOException.error_code in pymodbus 3.7+ - Simplify modbus transport compatibility to only support pymodbus 3.7+ - Remove unnecessary pymodbus 2.x compatibility code - Fix client initialization order issues in modbus_rtu and modbus_tcp - Add safety checks for addresses list initialization - Update error handling to work with newer pymodbus exception structure This resolves issues when analyze_protocol=true and improves compatibility with modern pymodbus versions.

- Fix UnboundLocalError when ModbusIOException occurs during register reading - Initialize register variable before try block to prevent undefined access - Add safety checks for register.registers access when register is None - Improve enable_write validation with exception handling and retry logic - Add delay before validation to ensure device is ready during initialization - Better error handling for validation failures during analyze_protocol mode This resolves issues when analyze_protocol=true causes validation to fail due to device not being ready or connection issues.

- Skip init_after_connect validation when analyze_protocol is enabled - Prevents validation from running during analyze_protocol initialization - Fixes timing issue where validation was called before client was fully ready - Maintains normal validation behavior when analyze_protocol is false This resolves the core issue where analyze_protocol=true caused validation to fail due to premature execution during initialization.

- Use configured protocol's register ranges instead of maximum from all protocols - Prevents trying to read non-existent registers from other protocols - Fixes 'No response received after 3 retries' error when analyze_protocol=true - Uses the actual configured protocol (eg4_v58) instead of creating new protocol objects - Maintains backward compatibility with fallback to original behavior This resolves the core issue where analyze_protocol was trying to read registers 0-1599 when the actual device only has registers 0-233.

test.py

jaredmauch · 2025-06-20T17:57:24Z

I emailed you (i hope) in private some notes on this. There were some bugs when analyze_protocol=true is set - it would not honor the configured baud rate (19200) and used a default of 9600. There's a lot of individual commits as I developed on a local machine and tested on a remote that has the hardware on it.

I'll try out your csv change as well and see if that works as desired.

HotNoob · 2025-06-20T18:56:18Z

cool. got your email; was in the spam folder :P

didn't have to go so far with the analyze protocol; just the dump.
it needs a ton of work in general.

got the dump!

i'll look at things in detail hopefully later today.

jaredmauch · 2025-06-20T20:29:52Z

cool. got your email; was in the spam folder :P

didn't have to go so far with the analyze protocol; just the dump. it needs a ton of work in general.

got the dump!

i'll look at things in detail hopefully later today.

Yeah, debugging the serial port speed thing was a fun side-quest of the main side-quest.

my primary goal of the serial number stuff is to have the data in influxdb have the serial number promoted out of the individual record to the device_serial_number making it easier to grab comparable values across multiple inverters and observe any oddities. Our electric company here is regularly giving us power that is outside of the spec that is acceptable.

I can split this into different PRs if that's easier, but this did kinda spiral out. I can get you into a device with the inverter attached as well if that helps at all.

HotNoob · 2025-06-20T21:33:56Z

alright.

simulating your dump. i think it's working... ish
using v1.1.10 to start.

look about right?

jaredmauch · 2025-06-20T21:42:24Z

Well it's not quite working right with my most recent change, looking at it as influxdb has:

> show series 
key
---
eg4_data,device_identifier=52,device_manufacturer=hotnoob,device_model=hotnoob,device_name=hotnoob_,device_serial_number=52,transport=transport.0

while the column with the serial has:

> select serial_number from eg4_data
name: eg4_data
time                serial_number
----                -------------
1750455349470840832 45x84
1750455380676626688 45x84

which is the correct value

HotNoob · 2025-06-20T21:52:46Z

Well it's not quite working right with my most recent change, looking at it as influxdb has:
> show series 
key
---
eg4_data,device_identifier=52,device_manufacturer=hotnoob,device_model=hotnoob,device_name=hotnoob_,device_serial_number=52,transport=transport.0
while the column with the serial has:
> select serial_number from eg4_data
name: eg4_data
time                serial_number
----                -------------
1750455349470840832 45x84
1750455380676626688 45x84
which is the correct value

ok, looks like the "register" order is correct, but the byte order is incorrect for the eg4_v58.
hmm...

HotNoob · 2025-06-20T22:20:35Z

Well it's not quite working right with my most recent change, looking at it as influxdb has:
> show series 
key
---
eg4_data,device_identifier=52,device_manufacturer=hotnoob,device_model=hotnoob,device_name=hotnoob_,device_serial_number=52,transport=transport.0
while the column with the serial has:
> select serial_number from eg4_data
name: eg4_data
time                serial_number
----                -------------
1750455349470840832 45x84
1750455380676626688 45x84
which is the correct value

alright, i've put in a clean solution to the problem in v1.1.10

8d3e784

the byteorder can be specified on a per register level... to help fight those bad chinese programmers :P
ie, ASCII_BE for big endien, ASCII_LE for little endien, HEX_BE, HEX_LE ect...

canbus has better support for byteorder, other datatypes \w modbus will need to be updated to account for byteorder in the future.

* improve eg4 handling

44db2f3

* add json output to ease testing * add influxdb v1 support for ease of use with grafana

HotNoob changed the base branch from main to v1.1.10 June 20, 2025 03:10

HotNoob reviewed Jun 20, 2025

View reviewed changes

jaredmauch added 23 commits June 20, 2025 12:15

s/type/trasport

30902c8

revert

8afe3f0

attempt to fix issue with analyze_protocol = true

d5632a5

attempt to fix issue with analyze_protocol = true

f9d5fc2

attempt to fix issue with analyze_protocol = true

a52c418

attempt to fix issue with analyze_protocol = true

0248882

remove testing files from branch

7f6e1ed

address connection issue

e36fa93

address issue with analyze_protocol

663e76d

address issue with analyze_protocol

55e2a68

address issue with analyze_protocol

635e3ba

address issue with analyze_protocol

e8d7c56

address issue with analyze_protocol

5374391

address issue with analyze_protocol

20e18b1

address issue with analyze_protocol

1970e23

address issue with analyze_protocol

d3b4166

address issue with analyze_protocol

bbf19e3

address issue with analyze_protocol baudrate

021736f

address issue with analyze_protocol baudrate

c0510f6

jaredmauch commented Jun 20, 2025

View reviewed changes

test.py Outdated Show resolved Hide resolved

restore file accidentally deleted in 7f6e1ed

224d4ed

jaredmauch changed the title ~~Several small improvements~~ Several medium sized improvements Jun 20, 2025

jaredmauch added 4 commits June 20, 2025 14:01

sync over 4db8362

d6d14d1

Merge branch 'main' of github.com:jaredmauch/PythonProtocolGateway

64fb1ef

cleanup logging, place DEBUG level messages behind debug

af51f7d

cleanup logging, place DEBUG level messages behind debug

df9c9c6

influxdb floating point fixup

a8f606a

promote serial number from inverter to device

fb49a90

jaredmauch added 2 commits June 20, 2025 17:48

promote serial number from inverter to device

e7d78b0

simplify device_serial_number

d03a843

HotNoob self-assigned this Jun 20, 2025

HotNoob changed the base branch from v1.1.10 to v1.1.10-pr92 June 20, 2025 22:58

Merge branch 'v1.1.10-pr92' into main

950fd4b

HotNoob merged commit c5404a7 into HotNoob:v1.1.10-pr92 Jun 20, 2025

HotNoob mentioned this pull request Aug 11, 2025

V1.1.10 #105

Merged

Uh oh!

Several medium sized improvements #92

Several medium sized improvements #92

Uh oh!

Conversation

jaredmauch commented Jun 20, 2025

Uh oh!

HotNoob commented Jun 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ASCII 115~119

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HotNoob commented Jun 20, 2025

Uh oh!

Uh oh!

jaredmauch commented Jun 20, 2025

Uh oh!

HotNoob commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaredmauch commented Jun 20, 2025

Uh oh!

HotNoob commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaredmauch commented Jun 20, 2025

Uh oh!

HotNoob commented Jun 20, 2025

Uh oh!

HotNoob commented Jun 20, 2025

Uh oh!

Uh oh!

HotNoob commented Jun 20, 2025 •

edited

Loading

HotNoob commented Jun 20, 2025 •

edited

Loading