Add IDM and NetIDM support

[evilpete]

Protocol support for both IDM and NetIDM

Both Protocols are functional and report the same data as rtlamr in json format.

Currently the code is unable to differentiate between the the two similar protocols thus both will respond to the same packet. As of this time.I am unable to find any documentation on how to differentiate IDM and NetIDM packets as both use identical Sync ID / Packet Type / length / App Version ID and CRC

Superfluous debugging statements have been left intact in the code incase needed in resolving the above mentioned shortcoming.

Test files samples have been submitted in pull request rtl_433_tests # 348

[merbanan]

How about merging both decoders? They seem very similar.

[evilpete]

yea, I was thinking about that. I'll update it now, other then that I've gone as far as I can go (and I am sick of looking for clues by reading patents)

If I had a method to tell them apart they could even co-exist in the same callback,

[merbanan]

And maybe it would be possible to classify the decoder based on the intervals content. I theorise that the msb count will indicate clearly what mode is used.

[merbanan]

Might be good enough heuristic. But we need a good set of samples to see if that holds.

[evilpete]

Ok.... done... files merged

regarding Samples, unfortunately I do not have a local source for data samples, and the samples captured with rtlamr typically need minor processing before they will be read by rtl_433 ( the rtlamr samples need to be amplified and add some leading noise else the signal will not be detected )

[evilpete]

@merbanan When I merged the files I set both decoders use the same output_fields array as a superset for the fields used in both IDM and NetIDM.
I noticed that is the field is not in the callback's output_data entry the entry in output_fields is quietly ignored, If this is not the intended behavior I can split it up again.

[evilpete]

From my notes. (may help when reading the code):

maybe able to detect the difference by looking at byte offset 19. ??

92 byte message including preamble

    preamble = 0x5555
                         length   byte offset    Notes
common:
    Sync Word               2       0           0x16A3
    Packet Type             1       2           0x1C
    Packet Length           1       3           0x5C
    Hamming Code            1       4           0xC6
    Application Version     1       5
    Endpoint Type           1       6           Least significant nibble is equivalent to SCM's endpoint type field.
    Endpoint ID             4       7
    Consumption Interval    1       11
    Mod Programming State   1       12
    Tamper Count            6       13

idm: (65 bytes in length)
    Async Count             2       19
    Power Outage Flags      6       21
    Last Consumption        4       27          Equivalent to SCM's consumption field.
    Diff Consumption        53      31          47 intervals of 9-bit unsigned integers


netidm: (65 bytes in length)
    Unknown_1               7       19
    Last Generation Count   3       26          Total power generated.
    Unknown_2               3       29
    Last Consumption Count  4       32          Total power consumed.
    Differential Cons       48      36          27 intervals of 14-bit unsigned integers.

common:
    Transmit Time Offset    2       84          1/16ths of a second since the first transmission for this interval.
    Meter ID Checksum       2       86          CRC-16-CCITT of Meter ID.
    Packet Checksum         2       88          CRC-16-CCITT of packet starting at Packet Type. poly=1021 init=0xD895

[merbanan]

Can you post some output from IDM and NetIDM signals. I am convinced there is some kind of heuristic that can be used to differentiate between the types.

[evilpete]

@merbanan.

I do not have access to a "live" signal source, you may want to ask in the #1378 thread

My PG&E meter uses zigbee and something else (Berkeley, California)

[merbanan]

@zuckschwerdt should we merge the code with the logic of emitting 2 messages? Or maybe disable both and let people enable the one they need?

[zuckschwerdt]

I didn't follow the IDM topic. If I understand correctly there are no false positives but (currently) there is no obvious way to decide if it's IDM or NetIDM? Then I guess it can reasonably be default enabled with both outputs, perhaps we'll discover a way to discern the two or maybe consumers do a plausibility check.
At least for me, I'd like to see if there is (Net)IDM data and then decide which output to handle after the fact.

[evilpete]

@zuckschwerdt

I get the impression the rtlamr seems to do it but I do not see logic in the code to do so (unless there is sone field unseen validation )

take a look in at a comment in 1378, there is a line to a zip'ed sample file form rtlamr with attached json that where it appears to be able to differentiate the idm and netidm without duplicates

my meter does not generate idm reports, I've been relying on @phrrngtn to make samples and link them to the #1378 thread

[merbanan]

I found these messages in the source. My theory is that if you add all the differential values together based on the 14bit vs 9 bit layout. The one with the lowest sum would be type used. The IDM message below has 3 bits set. The 9bit sum would be 3. The 14 bit sum would be more. Then the type is likely IDM.

Anyway I agree with @zuckschwerdt, it is best to enable it by default. Can you do a review of the code?

@evilpete can you rebase the code against the main tree?

IDM: "AsynchronousCounters":0,"PowerOutageFlags":"QUgmCEEF","LastConsumptionCount":339972,"DifferentialConsumptionIntervals":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,1,0,0],"TransmitTimeOffset":476,"SerialNumberCRC":60090,"PacketCRC":31799}}

NetIDM:
"LastGeneration":125,"LastConsumption":0,"LastConsumptionNet":2223120656,"DifferentialConsumptionIntervals":[7695,545,2086,1475,6240,2180,4240,4616,240,7191,609,7224,1603,96,2052,12464,6152,8480,9226,352,12312,833,10292,1795,4248,4613,8416],"TransmitTimeOffset":2145,"SerialNumberCRC":61178,"PacketCRC":37271}}



[evilpete]

@merbanan done
merge'd upstream/master

[zuckschwerdt]

For future commits and merges please make sure the commit message is sensible, e.g. "Add LDM support".

[merbanan]

Ok, I get it now. The PR name will be the squash commit subject and the squash edit will be the body. That was not obvious to me before.