Data not published, CRC error #4

bortek · 2021-02-05T14:59:31Z

I am using Landis E360 meter and see that the data is coming but not published in HASS. I see in the logs that CRC checksum is failing.

[19:01:21][D][data:259]: Program Started
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [22.7.0]: 0000.000 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [41.7.0]: 0001.019 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [42.7.0]: 0000.000 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [61.7.0]: 0001.110 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [62.7.0]: 0000.000 kW
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [23.7.0]: 0000.210 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [24.7.0]: 0000.000 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [43.7.0]: 0000.200 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [44.7.0]: 0000.000 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [63.7.0]: 0000.274 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [64.7.0]: 0000.000 kVAr
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [32.7.0]: 235.9 V
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [52.7.0]: 238.9 V
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [72.7.0]: 240.2 V
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [31.7.0]: 004.7 A
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [51.7.0]: 004.3 A
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [71.7.0]: 004.7 A
[19:01:21][D][data:263]: Read 6 bytes from UART
[19:01:21][D][crc:275]: CRC: 1490 = ADF2. PASS = NO

Sometimes I see this in the logs

[19:01:11][V][app:081]: A component took a long time in a loop() cycle (0.30 s).
[19:01:11][V][app:082]: Components should block for at most 20-30ms in loop().

And this as well but it is discusses in issue #3

[19:00:52][E][uart_esp8266:167]: Reading from UART timed out at byte 0!

Any ide what could be wrong. Perhaps checksum is calculated differently on E360?

I use short cable like 8cm from the port to the board and it is a shielded cable with shield on the ground so the data loss due to is should not be a cause.

bortek · 2021-02-07T12:15:44Z

According to http://www.nodo-domotica.nl/images/8/86/DSMR.pdf

CRC is a CRC16 value calculated over the preceding characters in the data message (from
“/” to “!” using the polynomial: x16+x15+x2
+1). CRC16 uses no XOR in, no XOR out and is
computed with least significant bit first. The value is represented as 4 hexadecimal characters (MSB first).

Is this how the CRC is calculated in the code?

I also read on the forums (http://domoticx.com/p1-poort-slimme-meter-hardware/ )that other use different resistors. 1K instead of 10K and 4.7K. I will git it a try too.

bortek · 2021-02-07T18:54:23Z

Yet another finding on another forum. Quoting it here.

I fixed the cable was not suitable for a Landis E360. For this I had an ISKRA AM550.
So I opened it and put a 1K resistor between + 5v and RXD. There is also an existing resistor (10K) between the RTS and RXD. Now it works like a charm
Yes! Also soldered that 1k resistor. Now it works again. I had the exact same switch from an ISKRA AM550 -> Landis E360. How did you find out this was supposed to be it?

Source: https://gathering.tweakers.net/forum/list_message/60757450#60757450

But it does not really make sense if looking at the schematic on this project since RTS is already shorted to +5V and putting those two resistors (it will be in parallel) gives us one resistor of 9 Ohms.

bortek · 2021-02-07T19:08:28Z

Here they use it like this

To connect my Landys and Gyr E350 (ZCF110), I used a 10K resistor connected between DATA (RXD) and RTS.

https://github.com/fliphess/esp8266_p1meter

bortek · 2021-02-07T19:13:22Z

Yet another schematic just like this one also with inverter but they use both 1K resistors

https://gathering.tweakers.net/forum/list_message/61171042#61171042

bortek · 2021-02-07T19:22:03Z

Another suggestion of using differente serial parameters

SER.bytesize = serial.SEVENBITS
SER.parity = serial.PARITY_EVEN

I am going to beat it, this is just a matter of time and testing...
:)

bortek · 2021-02-08T20:49:27Z

E360 seems to use ESMR5.0 protocol and not DSMR perhaps crc calculation is different om it. Cant find ESMR5 specs.

bortek · 2021-02-09T08:42:28Z

Another finding about CRC from this page and further down in the comments
https://willem.aandewiel.nl/index.php/2019/04/09/dsmr-logger-v4-slimme-meter-uitlezer/

Willy,
I don't know what you're doing or what your lineup is.
I suspect that you yourself copy a telegram and use it as input. If so, remember that each line of a telegram must be terminated with a CR and LF. One wrong character in your telegram changes the CRC. An empty line also consists of at least a CR and LF!

Willy Verbiest says:
20 December 2019 at 19:04
Dear Willem,
I changed the existing const char raw [] definition from the parse example of Matthijs Kooijman to the received telegram from Fluvius and it looks like this:
// Data to parse (Fluvius mail)
const char raw [] =
“ / FLU5 \ 253770234_A \ r \ n ”
“ \ r \ n ”
“ 0-0: 96.1.4 (50213) \ r \ n ”

Willy
Change this line:

“/FLU5\253770234_A\r\n”

to

“/FLU5\\253770234_A\r\n”

And then try parse again.

A backslash has a special meaning.

Willy Verbiest says
21 December 2019 at 18:41
Dear Willem
_Fantastic it works! I used the original crc16_update from ( https://github.com/PaulStoffregen/cores/blob/master/teensy3/util/crc16.h )
and added the extra \ teg and get the same CRC number as sent in the P1 telegram of my digital meter (it is not smart).
Thank you.

I found a companion/specs for DSRM 5 http://www.nodo-domotica.nl/images/8/86/DSMR.pdf and CRC calculation algorithm is the same.

So what about this extra \ in the header of the message. Maybe it is being dropped out due to it is a special character and needs to be escaped and thus we need to add it? How to do that?

bortek · 2021-02-09T10:09:47Z

More ideas to try from https://willem.aandewiel.nl/index.php/2020/02/28/restapis-zijn-hip-nieuwe-firmware-voor-de-dsmr-logger/

Somehow some brilliant mind thought it was a good idea to use backslashes in the Smart Meter Telegram’s. It was/is not!
If your meter sends a backslash (which is for most computer’s an “escape char”) you have to “escape the escape char”. In other words: Place a backslash in from of every backslash in the telegram (so “\” becomes “\” and if there is something like this “\\” place a backslash in front of ever backslash in the telegram so it becomes “\\\”).

bortek · 2021-02-13T21:49:16Z

Today's finding. CRC fail rate is about 80% while 20% of the time CRC is correct. After analyzing lots of debug output i found that Data is coming or being read from the port using chunks. If there data is read in several chunks then the CRC fails. But if the data comes in as one big chunk the CRC is correct. Below is output of two different telegrams.

In the first part you see several chunks separated by =====Reading Data===== headline and thus CRC fail. The second telegram is only one chunk and CRC becomes OK. The data in both still looks ok.

One possible explanation could be that there are some extra characters being added or missing between chunks that make CRC fail. Next question is to find what causes the multiple chunks and if that is normal what are the characters (if any) that garble CRC.

[22:37:51][D][data:259]: =====Reading Data=====
[22:37:51][D][data:263]: Read 10 bytes from UART
[22:37:51][D][data:263]: Read 1 bytes from UART
[22:37:51][D][data:263]: Read 25 bytes from UART
[22:37:51][D][data:263]: Read 28 bytes from UART
[22:37:51][D][data:303]: [1.8.0]: 00000627.435 kWh
[22:37:51][D][data:263]: Read 28 bytes from UART
[22:37:51][D][data:303]: [2.8.0]: 00000000.000 kWh
[22:37:51][D][data:263]: Read 30 bytes from UART
[22:37:51][D][data:303]: [3.8.0]: 00000058.614 kVArh
[22:37:51][D][data:263]: Read 30 bytes from UART
[22:37:51][D][data:303]: [4.8.0]: 00000008.355 kVArh
[22:37:51][D][data:263]: Read 23 bytes from UART
[22:37:51][D][data:303]: [1.7.0]: 0003.025 kW
[22:37:51][D][data:263]: Read 23 bytes from UART
[22:37:51][D][data:303]: [2.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 25 bytes from UART
[22:37:51][D][data:303]: [3.7.0]: 0000.608 kVAr
[22:37:51][D][data:263]: Read 25 bytes from UART
[22:37:51][D][data:303]: [4.7.0]: 0000.000 kVAr
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [21.7.0]: 0001.000 kW
[22:37:51][D][data:259]: =====Reading Data=====
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [22.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [41.7.0]: 0001.058 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [42.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [61.7.0]: 0000.966 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [62.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 26 bytes from UART
[22:37:51][D][data:303]: [23.7.0]: 0000.171 kVAr
[22:37:52][D][data:259]: =====Reading Data=====
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [24.7.0]: 0000.000 kVAr
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [43.7.0]: 0000.193 kVAr
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [44.7.0]: 0000.000 kVAr
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [63.7.0]: 0000.243 kVAr
[22:37:52][D][data:259]: =====Reading Data=====
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [64.7.0]: 0000.000 kVAr
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [32.7.0]: 237.7 V
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [52.7.0]: 235.6 V
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [72.7.0]: 242.8 V
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [31.7.0]: 004.2 A
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [51.7.0]: 004.5 A
[22:37:52][D][data:259]: =====Reading Data=====
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [71.7.0]: 004.1 A
[22:37:52][D][data:263]: Read 6 bytes from UART
[22:37:52][I][crc:285]: CRC: F7C9 = F976. PASS = NO

[22:38:11][D][data:259]: =====Reading Data=====
[22:38:11][D][data:263]: Read 10 bytes from UART
[22:38:11][D][data:263]: Read 1 bytes from UART
[22:38:11][D][data:263]: Read 25 bytes from UART
[22:38:11][D][data:263]: Read 28 bytes from UART
[22:38:11][D][data:303]: [1.8.0]: 00000627.451 kWh
[22:38:11][D][data:263]: Read 28 bytes from UART
[22:38:11][D][data:303]: [2.8.0]: 00000000.000 kWh
[22:38:11][D][data:263]: Read 30 bytes from UART
[22:38:11][D][data:303]: [3.8.0]: 00000058.618 kVArh
[22:38:11][D][data:263]: Read 30 bytes from UART
[22:38:11][D][data:303]: [4.8.0]: 00000008.355 kVArh
[22:38:11][D][data:263]: Read 23 bytes from UART
[22:38:11][D][data:303]: [1.7.0]: 0003.018 kW
[22:38:11][D][data:263]: Read 23 bytes from UART
[22:38:11][D][data:303]: [2.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 25 bytes from UART
[22:38:11][D][data:303]: [3.7.0]: 0000.603 kVAr
[22:38:11][D][data:263]: Read 25 bytes from UART
[22:38:11][D][data:303]: [4.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [21.7.0]: 0000.998 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [22.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [41.7.0]: 0001.055 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [42.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [61.7.0]: 0000.964 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [62.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [23.7.0]: 0000.168 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [24.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [43.7.0]: 0000.192 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [44.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [63.7.0]: 0000.243 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [64.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [32.7.0]: 237.5 V
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [52.7.0]: 237.0 V
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [72.7.0]: 243.1 V
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [31.7.0]: 004.2 A
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [51.7.0]: 004.5 A
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [71.7.0]: 004.1 A
[22:38:11][D][data:263]: Read 6 bytes from UART
[22:38:11][I][crc:285]: CRC: 5CC3 = 5CC3. PASS = YES

psvanstrom · 2021-02-17T17:55:41Z

Great finds @bortek ! I've have not had time to look at this but I will do that now. You're probably on to something with the reading of chunks, seems that the E360 outputs data a bit different than the Sagemcom reader.

I've updated the code so that the full telegram that has been read is logged by debug logging each raw line read. Could you update your microcontroller and post a telegram here? Would make it a bit easier to try to understand what's going on.

bortek · 2021-02-17T19:16:46Z

Thanks for getting back! I though you left us alone hehe. Yes I think there is something fishy about the way E360 data chunks.

My controller that is located "at landet" has hanged after I have tried to add delay in the loop so I cannot do anything at the moment. I need to disconnect and manually flash new code. I will travel till landet on tuesday and will do the tests then.

jimborr · 2021-03-03T20:48:57Z

I am in the same situation using the E360 and getting random CRC error.. Using this version of code seems to work much better for me, no CRC errors at all any more.

Thanks!!

bortek · 2021-03-03T22:32:01Z

I have tried this specific branch called E360 which adds a delay during reading. But that did not make any difference. Therefore I wonder if you have made any other adjustments to the code or device?

Anteus · 2021-03-16T13:26:45Z

Also on a E360 but only getting CRC errors. Tried with 3s delay but no luck maybe something with my wiring?

bortek · 2021-03-16T15:47:52Z

Can you try with 40ms ? I have heard that one guy managed to get it working with it.

Anteus · 2021-03-16T16:42:01Z

Thank you! Seems to be working fine 20 pass and one fail so far.

bortek · 2021-03-16T20:16:18Z

Thats greate news. Thanks for testing.

Will test on my device as soon as it gets alive.

bortek · 2021-03-31T06:06:32Z

Just reconfirming that 40ms delay works. Thus this branch is a working one in terms of CRC https://github.com/psvanstrom/esphome-p1reader/tree/e360-fix

This in combination with the switch automation below that turns on and off the RTS specific periods of time which is best known to work is 5sec on and 25 sec off. Thanks goes to Ingemar from byggahus forum who tested this and came to working values. All this is neccessary since E360 in Sweden has a bug and if RTS is connected permanently to 5V then the meter will hang after few hours to days and only full power recycle helps. What am I doing is pulling RTS from one of the GPIOs, no transistor is needed.

Just add this to p1reader.yaml file. I will make a PR when I have more time.

switch:

platform: gpio
pin: GPIO4
name: rtspoll
id: rtsout
on_turn_on:
- logger.log: "Switch Turned On!"
- delay: 5s
- switch.turn_off: rtsout
  on_turn_off:
- logger.log: "Switch Turned Off!"
- delay: 25s
- switch.turn_on: rtsout

bortek · 2021-03-31T06:07:02Z

since the CRC issue is fixed I will close this issue.

dimme · 2021-12-15T20:57:11Z

@bortek your link on the Mar 31 comment returns 404.

Anteus · 2021-12-15T21:00:41Z

@bortek your link on the Mar 31 comment returns 404.

It's merged to the main branch.

a7d87ec

sheminasalam · 2022-01-11T09:30:13Z

Hi there, i am trying to use this with my Landis Gyr E250 but I think the RJ port pins are different. This is how the RJ port is labeled

Is this how E350 is also labelled or this is also different from it?

eabase · 2023-07-24T13:22:30Z

@bortek

Thanks goes to Ingemar from byggahus forum who tested this and came to working values. All this is neccessary since E360 in Sweden has a bug and if RTS is connected permanently to 5V then the meter will hang after few hours to days and only full power recycle helps.

Do you have any links to this forum, or anything else that documents this FW bug?
(We're seeing similar issues on devices in a different country.)
So are you pulling the RTS down then, using a resistor or how?

UPDATE:
Found the link here:
https://www.byggahus.se/forum/threads/elmaetare-h1-port-p1-port-han-port-manga-namn-verkar-florera.384608/page-17

However, would be good to also get links to the various FW versions, as it seem that not all meters are compatible with the updated FW....

bortek closed this as completed Mar 31, 2021

jeroenjjjdevries mentioned this issue Mar 21, 2022

DSMR Slimme Meter Landis/Gyr E360 data missing home-assistant/core#68472

Closed

GhostNr1 mentioned this issue Oct 4, 2022

No signal D1 mini and Landis+Gyr E360 #56

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data not published, CRC error #4

Data not published, CRC error #4

bortek commented Feb 5, 2021

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 7, 2021

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 8, 2021

bortek commented Feb 9, 2021 •

edited

Loading

bortek commented Feb 9, 2021 •

edited

Loading

bortek commented Feb 13, 2021

psvanstrom commented Feb 17, 2021

bortek commented Feb 17, 2021

jimborr commented Mar 3, 2021 •

edited

Loading

bortek commented Mar 3, 2021

Anteus commented Mar 16, 2021

bortek commented Mar 16, 2021

Anteus commented Mar 16, 2021

bortek commented Mar 16, 2021

bortek commented Mar 31, 2021

bortek commented Mar 31, 2021

dimme commented Dec 15, 2021

Anteus commented Dec 15, 2021

sheminasalam commented Jan 11, 2022

eabase commented Jul 24, 2023 •

edited

Loading

Data not published, CRC error #4

Data not published, CRC error #4

Comments

bortek commented Feb 5, 2021

bortek commented Feb 7, 2021 • edited Loading

bortek commented Feb 7, 2021 • edited Loading

bortek commented Feb 7, 2021

bortek commented Feb 7, 2021 • edited Loading

bortek commented Feb 7, 2021 • edited Loading

bortek commented Feb 8, 2021

bortek commented Feb 9, 2021 • edited Loading

bortek commented Feb 9, 2021 • edited Loading

bortek commented Feb 13, 2021

psvanstrom commented Feb 17, 2021

bortek commented Feb 17, 2021

jimborr commented Mar 3, 2021 • edited Loading

bortek commented Mar 3, 2021

Anteus commented Mar 16, 2021

bortek commented Mar 16, 2021

Anteus commented Mar 16, 2021

bortek commented Mar 16, 2021

bortek commented Mar 31, 2021

bortek commented Mar 31, 2021

dimme commented Dec 15, 2021

Anteus commented Dec 15, 2021

sheminasalam commented Jan 11, 2022

eabase commented Jul 24, 2023 • edited Loading

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 7, 2021 •

edited

Loading

bortek commented Feb 9, 2021 •

edited

Loading

bortek commented Feb 9, 2021 •

edited

Loading

jimborr commented Mar 3, 2021 •

edited

Loading

eabase commented Jul 24, 2023 •

edited

Loading