Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TFA 30.3233.01 support #1240

Closed
Fietspomp86 opened this issue Dec 27, 2019 · 29 comments
Closed

TFA 30.3233.01 support #1240

Fietspomp86 opened this issue Dec 27, 2019 · 29 comments

Comments

@Fietspomp86
Copy link

Hi guys,

I have a new rain-sensor, the TFA 30.3233.01.
Website:

Could I please get some guidance how-to add this?
I tried capturing the raw packages but I didn't succeed.
Running a Pi3 with an RTL dongle.
I see plenty of sensors from the neighbours to it's working.
Just need some help with this new sensor.

@zuckschwerdt
Copy link
Collaborator

Capturing the signal is indeed the first step.

  • Use -S all to capture a few packets, then read them back with -r mode and sort out the known ones (or maybe use -S unknown).
  • Verify the samples visually with http://triq.net/iqs
  • Use -A mode to get an idea what's in the samples.

Post your findings and upload a zip with some samples known to be from the device.
A good way to only capture that device is to remove the antenna and place the device about 10 cm from the recevier.

@Fietspomp86
Copy link
Author

I did as you suggested, placed the device very close to the pi, removed antenna of the dongle.
I believe I caught something, but I'm no expert.
Without the device near the dongle I tried to scan and found nothing, so I guess it's really only the rainmeter which was in range.

RTL-433.zip

@zuckschwerdt
Copy link
Collaborator

It is always good to add some length of USB cable between the Raspi and the dongle. Plugged directly into the ports it will get noticable noise.

@Fietspomp86
Copy link
Author

Ok, I used an extension USB cable and tried again.
Here's the sample zip:
sample.rain.zip

@mhaas
Copy link
Contributor

mhaas commented Jan 1, 2020

Hi @Fietspomp86, I am also looking into this device! I have many samples and some idea how to decode. I will keep you posted.

@mhaas
Copy link
Contributor

mhaas commented Jan 2, 2020

Hi all,

I gathered some samples and loaded one into sigrok. It seems that the signal has a period of 750us. At the beginning of a period, a pulse is sent at either 250us or 500us length.

annotated_sigrok

The sensor sends the same message 6 times. At the beginning of each message, two sync pulses are sent: 750us high, 750us low, 750us high, 750us low. Note that the screenshot shows the beginning of the very first message, where only one sync pulse (750us hgh, 750us low) is sent.

@zuckschwerdt Does this analysis make sense to you?

The modulation seems to be PWM. Given this data, the following flex decoder:

rtl_433 \
      -X 'name=tfa_drop,modulation=OOK_PWM,short=255,long=510,reset=2500,gap=1300,sync=750,bits=66'

produces output:

{
  "time": "2019-12-31 12:26:13",
  "model": "tfa_drop",
  "count": 6,
  "num_rows": 7,
  "rows": [
    {
      "len": 1,
      "data": "8"
    },
    {
      "len": 66,
      "data": "c65d54ff09550047c"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    }
  ],
  "codes": [
    "{1}8",
    "{66}c65d54ff09550047c",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4"
  ]
}

Note that the first row of len 1 is an artifact of the first sync pulse.

I have already gathered many samples, including a nice rain simulation by putting the sensor under a running sink for an hour or so. I find it most intuitive to look at the data at the crumble level (two crumbles are one nibble).

Here is a histogram:

c6_5d_54_histogram

This shows the relative freqencies of each crumble value at a given index in the data. Note that I am adding a padding of a 0 nibble on the right.

What is immediately obvious here is that the first three bytes (crumbles 0 to 11) are always identical. This is the device ID - it changes after a battery reset. The histogram above was generated from just one run, so the ID is the same.

From looking at the data itself, I know that crumbles 16-19 form the LSB of the rain counter. The data is not entirely clean. I assume some sort of transformation has to be applied - but looking at the bits as they change, it is clear that lower bits roll over and higher bits increase.

Interestingly, after 8 tips of the rocker, crumbles 16, 17 and 24-27 flip from 00 to 11. So 24-27 also seem to be related to the rain counter, which would give us 8 crumbles or 2 bytes to encode the rain data.

I will investigate a bit more. It seems that I need to XOR crumbles 18 and 19 with 0111 to get sensible data, but I don't know what to do with the other bits.

@zuckschwerdt
Copy link
Collaborator

The analysis on crumbles and the histogram is great. I have an old half-finished tool to detect change frequency, roll-over and similar. But it was a rabbit hole. It seemed I always needed to add too many assumptions. Better to just show a picture and squint your eyes ;)

The unexpected double bit flip might be a parity bit. Some protocols have 7-to-8 encoding.

@mhaas
Copy link
Contributor

mhaas commented Jan 2, 2020

@zuckschwerdt Well, I have a script which dumps only the changed crumbles. High-entropy columns are bold, low-entropy columns are light. Sadly, the output does not transport very well except as a screenshot.

significant_data

The leftmost column here is the number of tips of the rocker which 0000 being the initial state and 0001 the first tip. Note that some values are duplicated.

It is easy to see that crumble 19 changes often, then crumble 18. It also seems obvious that 00 is the highest possible value, after which the next significant bit increases. 11 is then the value after rollover, so likely the smallest possible value. This seems to hold true for both crumble 18 and 19.

I am still confued by what's happening after 0009 - and why the initial state at 0000 does not seem to correspondly nicely to something zero-ish.

I was also looking at BCD, but that does not seem to be case. Perhaps some gray code or hamming code. I will check later.

I also have some more data for 50, 75 and 100:

siginificant_data_0100

@zuckschwerdt
Copy link
Collaborator

If I read that right and 00 is assumed to be the biggest and 11 the smallest value that might indicate the bits are flipped? Though I guess you already looked into that.
There is one other decoder that does not roll over a counter to zero. IIRC it divides by a large factor and keeps the remainder.

@mhaas
Copy link
Contributor

mhaas commented Jan 3, 2020

I think I got it. You were right about flipping the bits. The missing transformation is the addition of 10 (as in, ten). This is somewhat obvious in the bitbench (cool tool, btw!) after inverting: Bitbench

Basically, zero is at 65526 - 9 is at 65535, and thus 10 (ten) must be at 0.

The following python code implement this transformation:

def simple_decoder(input_data) -> int:
    rain_a = input_data.get_byte_at_index(4)
    lower = (rain_a.as_int() ^ 0xff)
    rain_b = input_data.get_byte_at_index(6)
    higher = rain_b.as_int() ^ 0xff

    value = (higher << 8) + lower

    value = value + 10
    value = value & 0xffff

    return value

I will make a PR with this. Now I need to figure out two more things: the bit for low battery and the parity/checksum. Basically, I have 5 bytes out of 8.5 figured out by now.

@zuckschwerdt Given that I have to invert bytes 4 and 6, it is reasonable to assume that I have to invert the entire stream?

edit:

Looks like the "low battery" bit is bit 24, i.e. first bit after the ID: bitbench

@mhaas
Copy link
Contributor

mhaas commented Jan 7, 2020

@Fietspomp86 Can you try this code? #1255

@Fietspomp86
Copy link
Author

@Fietspomp86 Can you try this code? #1255

Hmm, it seems my rainmeter has died on me.
It won't show any figures, also not on the display....
Guess I'll send it back to the shop. Sorry guys!

@mhaas
Copy link
Contributor

mhaas commented Jan 11, 2020

@Fietspomp86 Maybe you can also try to reset the batteries - maybe the devices lost sync.

@mhaas
Copy link
Contributor

mhaas commented Jan 11, 2020

I did some work on the checksum today

I noticed some interesting behavior: the sensor will repeatedly submit identical data with increasing transmission counter bits. Once the transmission counter rolls over, byte 7 is identical to previous transmissions with identical rain counter and transmission counter bits.

I assume now that byte 7 is indeed the checksum.

I already tried several approaches: reveng, xor of values, sum of values, two's complement etc. Nothing worked. I also tried leaving out the first nibble, as it is a fixed for all device IDs - no luck.

I should note that the checksum is different for identical rain and transmission counter values across different device IDs (e.g. after battery reset).

One interesting thing: the last bit of byte 7 determines the value of the first bit of byte 8: they will be identical.

I have prepared several bitbenches with different device ids:

@zuckschwerdt Any idea how I can proceed with the checksum?

@zuckschwerdt
Copy link
Collaborator

Looks very thorough, so nothing obvious to recommend. Not sure if revdgst (see https://triq.org/) can help. But it could be useful to quickly try some checksum ideas. Putting all of your BitBench data into a txt file and using revsum (or revdgst) shows somethings like nibble-wide parity. But that really needs some poking around to make sense of.

@zuckschwerdt
Copy link
Collaborator

zuckschwerdt commented Jan 11, 2020

One thing that might be visible in your sorted examples is too little true randomness in the checksum. It feels more like a sum (plain add with carry or xor without carry, or even some Fletcher-style sums).

@zuckschwerdt
Copy link
Collaborator

I do have to double check, but could indeed be a LFSR digest: reverse Galois with byte reflection, generator 31, key f4, final XOR c0).

@mhaas
Copy link
Contributor

mhaas commented Jan 11, 2020 via email

@zuckschwerdt
Copy link
Collaborator

Interesting thing to note: LFSR digests are somewhat weaker checksums than CRC, so no reason to bother really. But a Galois digest might very well be a native feature of the used chips. So maybe the designers just went with that because it's easy.

It might look like a nice security feature to novice designers but 8 bit keystreams are toys from a serious security standpoint. Even 16 bit LFSR are not too hard to crack -- revdgst was made for that and the 8 bit brute-force came with that because it takes no computational power whatsoever ;)

Harder to make sense of are the botched CRC implementations in some protocols -- though when you finally see where the designers went wrong it's a great laugh :)

@zuckschwerdt
Copy link
Collaborator

You can get the keystream with keylst -g 31 -i f4 -G -r -w 8 from revdgst.
Since messages are 64 bit we need 64 keys:
f4 d9 83 37 6e dc 89 23 46 8c 29 52 a4 79 f2 d5 9b 07 0e 1c 38 70 e0 f1 d3 97 1f 3e 7c f8 c1 b3 57 ae 6d da 85 3b 76 ec e9 e3 f7 df 8f 2f 5e bc 49 92 15 2a 54 a8 61 c2 b5 5b b6 5d ba 45 8a 25
Byte reflection then means the keys apply from last byte to first byte (high bit to low bit, i.e. left to right).
Of course this means you can shuffle the keys to go first byte to last byte. And if you also shuffle to go low bit to high bit the progression could perhaps be continuous, i.e. you could create a generator that's more plausible. Best to stuff the keys in a BitBench and follow the shift pattern.

@mhaas
Copy link
Contributor

mhaas commented Jan 12, 2020

@zuckschwerdt Now you have me a bit confused. My goal was to understand the computation of the checksum so that I can re-implement it. With my own implementation, I can compute a checksum from the message and compare with the transmitted checksum. Messages where the checksum does not match are discarded.

Above, you provided the output of your revdgst tool:

reverse Galois with byte reflection, generator 31, key f4, final XOR c0).

With this information, I was able to re-implement (or rather, copy&paste from your code :)) the checksum algorithm and it seems to work: https://github.com/merbanan/rtl_433/pull/1255/files#diff-0391fbab6a3dbb6c0ddbc0e5a1e7d82aR226

The code is rather heavily commented as I was trying to understand what was happening there :)

Is there anything else I am missing here? What could I get from revdgst that I am missing? Or where you simply giving me some pointers how I can validate the parameters given by revdgst?

I have two more questions related to revdgst - I am really ignorant on this stuff.

  • What is the meaning of the final XOR c0? In my code, I do not need a final XOR.
  • In your runner() function, you first compute the digest/checksum of the first data line - and then the checksums of the remaining lines, while comparing these checksums with various operations to the initial checksum from the first line. What does it do? I did not seem to need that information at all.

This is now the second time where you just produced a tool that really solves my problem. Next time, I will ask directly :)

The PR in #1255 should now be ready for a review!

@zuckschwerdt
Copy link
Collaborator

Oh, sorry, just rambling there. That algorithm is just one example which works in producing a stream of keys to use for each bit.
Each bit in the protocol has a fixed key (the list above taken from keylst) and those keys are xored for all set bits. There is no given order of keys just convenient orders where a simple LFSR can generate the keys.
And because "reverse" and "byte reflect" really look wrong I wanted to point out that perhaps another key order also has some generator, which could look more sanely.

The final XOR (c0) should be the fixed point that all valid codes result in. Usually we'd want a zero there but that's arbitrary. Perhaps there is a bug in revdgst and indeed it's zero here, like your code shows.

The runner just tries a number of methods how the checksum could be composited and stored. E.g. each step of the checksum could be xored/added/subtracted/... and the result stored as the directly the final value/addition/substraction/... remainder. In this case it's xor, xor.

@merbanan
Copy link
Owner

@mhaas @zuckschwerdt can this issue be closed now?

@Kwenk
Copy link

Kwenk commented May 6, 2020

@Fietspomp86 Can you try this code? #1255

Hmm, it seems my rainmeter has died on me.
It won't show any figures, also not on the display....
Guess I'll send it back to the shop. Sorry guys!

I bought this sensor because of the support here, but despite the update I dont get any data.
Its a very rare sensor. Is there any feedback that it works correctly? The initiator probably never tested it himself?!

@mhaas
Copy link
Contributor

mhaas commented May 6, 2020 via email

@Kwenk
Copy link

Kwenk commented May 6, 2020

Of course, here are my samples: kwenk_raindrop.zip

@Kwenk
Copy link

Kwenk commented May 25, 2020

Not ok? Anything else I could do?

@zuckschwerdt
Copy link
Collaborator

Looks like your samples are "too loud", the center frequency (DC) might be too close to your signal. It might help to offset e.g. 50kHz: -f 433.97M

@Kwenk
Copy link

Kwenk commented May 30, 2020

Nice hint with 433.97M - I used 433.92M - worked with my different kind of temperature sensors but not with this rain sensor. With -f 433.97M I got it every 45 seconds:

model     : TFA-Drop     id        : d93f5
Battery OK: 1            Rain in MM: 4.826         Integrity : CHECKSUM

I'm picking up my other sensors as well. Sorry for the inconvenience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants