Support for FSK Modulation - WH51 Soil Moisture sensor

[NorthernMan54]

Status on the FSK Project

Backlog

[x] Addition of FSK device decoders to the build
[x] Update build process to allow refreshes of rtl_433 components
[x] Wire FSK device decoders into rtl_433_ESP code base
[x] Create FSK Transmitter to emulate a FSK device, tks @AusGunno ( https://github.com/NorthernMan54/rtl_433_ESP/tree/fsk/example/FSK_Transmitter )
[x] Create FSK Receiver ( https://github.com/NorthernMan54/rtl_433_ESP/tree/fsk/example/FSK_Receiver )
[x] Create FSK Sample ( https://github.com/NorthernMan54/rtl_433_ESP/tree/fsk/example/FSK_Sample )
[x] Demodulate signal pulse train into rtl_433 device decoders

[ ] Add backwards compatibility for old RSSI Signal detection mode
[x] Final tuning of signal reception to improve success rate from 10%
[x] Review change for bad code, simplify compiler definitions
[ ] Add support for verbose mode of a single device decoder
[x] Fix broken debug mode setting
[ ] Update documentation for FSK Decoders and new wiring requirement
[ ] Enable modulation switches in openMQTTGateway to enable switching between FSK and ASK modulation
[ ] Create pull request for OpenMQTTGateway for breaking change on usage of gpio2 and removal of minimumRSSI option

[AusGunno]

@NorthernMan54 Will work on this tonight.
Here is the -A data from RTL_433 and SDR dongle
Detected FSK package 2021-04-06 13:21:09

---

time : 2021-04-06 13:21:09
model : Fineoffset-WH51 ID : 00ca1d
Battery level: 1.111 Battery : 1700 mV Moisture : 0 % Transmission boost: 0 AD raw : 20 Integrity : CRC
Analyzing pulses...
Total count: 48, width: 10.26 ms ( 2564 S)
Pulse width distribution:
[ 0] count: 1, width: 0 us [0;0] ( 0 S)
[ 1] count: 38, width: 56 us [56;68] ( 14 S)
[ 2] count: 3, width: 116 us [116;120] ( 29 S)
[ 3] count: 3, width: 172 us [172;176] ( 43 S)
[ 4] count: 1, width: 404 us [404;404] ( 101 S)
[ 5] count: 1, width: 288 us [288;288] ( 72 S)
[ 6] count: 1, width: 1392 us [1392;1392] ( 348 S)
Gap width distribution:
[ 0] count: 33, width: 56 us [48;60] ( 14 S)
[ 1] count: 6, width: 172 us [172;176] ( 43 S)
[ 2] count: 2, width: 464 us [464;468] ( 116 S)
[ 3] count: 3, width: 112 us [112;116] ( 28 S)
[ 4] count: 2, width: 256 us [232;280] ( 64 S)
[ 5] count: 1, width: 348 us [348;348] ( 87 S)
Pulse period distribution:
[ 0] count: 1, width: 48 us [48;48] ( 12 S)
[ 1] count: 27, width: 116 us [116;120] ( 29 S)
[ 2] count: 9, width: 232 us [232;232] ( 58 S)
[ 3] count: 3, width: 172 us [172;176] ( 43 S)
[ 4] count: 2, width: 576 us [520;636] ( 144 S)
[ 5] count: 3, width: 308 us [292;340] ( 77 S)
[ 6] count: 1, width: 872 us [872;872] ( 218 S)
[ 7] count: 1, width: 1448 us [1448;1448] ( 362 S)
Pulse timing distribution:
[ 0] count: 1, width: 0 us [0;0] ( 0 S)
[ 1] count: 71, width: 56 us [48;68] ( 14 S)
[ 2] count: 6, width: 116 us [112;120] ( 29 S)
[ 3] count: 9, width: 172 us [172;176] ( 43 S)
[ 4] count: 3, width: 444 us [404;468] ( 111 S)
[ 5] count: 3, width: 264 us [232;288] ( 66 S)
[ 6] count: 1, width: 1392 us [1392;1392] ( 348 S)
[ 7] count: 1, width: 348 us [348;348] ( 87 S)
[ 8] count: 1, width: 0 us [0;0] ( 0 S)
Level estimates [high, low]: 15873, 7
RSSI: -0.1 dB SNR: 33.6 dB Noise: -33.7 dB
Frequency offsets [F1, F2]: 7357, -7395 (+28.1 kHz, -28.2 kHz)
Guessing modulation: Pulse Code Modulation (Not Return to Zero)
Attempting demodulation... short_width: 56, long_width: 56, reset_limit: 57344, sync_width: 0
Use a flex decoder with -X 'n=name,m=FSK_PCM,s=56,l=56,r=57344'
pulse_demod_pcm(): Analyzer Device
bitbuffer:: Number of rows: 1
[00] {177} 55 55 55 55 55 51 6e a2 88 06 50 e8 8b f8 07 c0 a7 ff ff fb 9a 20 80

Noting that the bit buffer seems to be out of sync by 1 bit. 55 <<1 = AA (Sync),
55 55 55 55 5 = AA AA AA AA A - sync
5 51 6e a = AA 2D D4 - pre
etc

Given that this actually has a well defined packet
Data format:
00 01 02 03 04 05 06 07 08 09 10 11 12 13
aa aa aa 2d d4 51 00 6b 58 6e 7f 24 f8 d2 ff ff ff 3c 28 8
FF II II II TB YY MM ZA AA XX XX XX CC SS
Sync: aa aa aa ...
Preamble: 2d d4 - actually preamble[] = {0xAA, 0x2D, 0xD4};
FF: Family code 0x51 (ECOWITT/FineOffset WH51)
IIIIII: ID (3 bytes)
T: Transmission period boost: highest 3 bits set to 111 on moisture change and decremented each transmission;
if T = 0 period is 70 sec, if T > 0 period is 10 sec
B: Battery voltage: lowest 5 bits are battery voltage * 10 (e.g. 0x0c = 12 = 1.2V). Transmitter works down to 0.7V (0x07)
YY: ? Fixed: 0x7f
MM: Moisture percentage 0%-100% (0x00-0x64) MM = (AD - 70) / (450 - 70)
Z: ? Fixed: leftmost 7 bit 1111 100
AAA: 9 bit AD value MSB byte[07] & 0x01, LSB byte[08]
XXXXXX: ? Fixed: 0xff 0xff 0xff
CC: CRC of the preceding 12 bytes (Polynomial 0x31, Initial value 0x00, Input not reflected, Result not reflected)
SS: Sum of the preceding 13 bytes % 256

It should be simple enough to set up the parameters in the normal packet TX code. I will do it tonight - I can verify against RTL_433 using the SDR, the Misol receiver and ESP.
Hopefully, the transmitter (WH51) will be stable enough clock wise to use Packet Rx mode on the CC101. I have not tried to date but wanted to start with the RTL_433 approach.
Cheers
Gunno

[AusGunno]

Have been playing with the transmit serial example - modified to send a predefined message. Mixed success so far but the parameters that RTL_433 SDR reports are no way near what is required (tried various bit rates etc). The spectrum seems really messy as well. The WH51 has a clean spectrum with the two frequencies for the 0/1 pulses. The CC1101 in some cases seems to have way more frequencies. So wondering if the config is not optimised and may be unstable. Will get a reg config from RF Studio for what I think the TX parameters should be and modify the RegConfigSettings() directly.
Cheers
Gunno

[NorthernMan54]

Sounds like your getting close

[AusGunno]

OK Getting close now. Used RF Studio to set most register values. Tweaked a few afterwards. RTL_433 shows:

Detected FSK package 2021-04-11 22:59:41
Analyzing pulses...
Total count: 39, width: 9.11 ms ( 2278 S)
Pulse width distribution:
[ 0] count: 26, width: 56 us [52;72] ( 14 S)
[ 1] count: 6, width: 116 us [116;120] ( 29 S)
[ 2] count: 3, width: 172 us [172;176] ( 43 S)
[ 3] count: 1, width: 408 us [408;408] ( 102 S)
[ 4] count: 1, width: 292 us [292;292] ( 73 S)
[ 5] count: 1, width: 1392 us [1392;1392] ( 348 S)
[ 6] count: 1, width: 232 us [232;232] ( 58 S)
Gap width distribution:
[ 0] count: 24, width: 56 us [56;64] ( 14 S)
[ 1] count: 3, width: 172 us [172;172] ( 43 S)
[ 2] count: 1, width: 348 us [348;348] ( 87 S)
[ 3] count: 7, width: 112 us [112;116] ( 28 S)
[ 4] count: 1, width: 520 us [520;520] ( 130 S)
[ 5] count: 2, width: 232 us [232;232] ( 58 S)
Pulse period distribution:
[ 0] count: 18, width: 116 us [112;136] ( 29 S)
[ 1] count: 3, width: 228 us [228;232] ( 57 S)
[ 2] count: 8, width: 172 us [172;176] ( 43 S)
[ 3] count: 4, width: 420 us [348;464] ( 105 S)
[ 4] count: 2, width: 288 us [288;292] ( 72 S)
[ 5] count: 2, width: 548 us [520;580] ( 137 S)
[ 6] count: 1, width: 1508 us [1508;1508] ( 377 S)
Pulse timing distribution:
[ 0] count: 50, width: 56 us [52;72] ( 14 S)
[ 1] count: 13, width: 116 us [112;120] ( 29 S)
[ 2] count: 6, width: 172 us [172;176] ( 43 S)
[ 3] count: 2, width: 376 us [348;408] ( 94 S)
[ 4] count: 2, width: 292 us [292;296] ( 73 S)
[ 5] count: 1, width: 1392 us [1392;1392] ( 348 S)
[ 6] count: 3, width: 232 us [232;232] ( 58 S)
[ 7] count: 1, width: 520 us [520;520] ( 130 S)
Level estimates [high, low]: 5406, 5
RSSI: -4.8 dB SNR: 30.3 dB Noise: -35.2 dB
Frequency offsets [F1, F2]: 24416, 6640 (+93.1 kHz, +25.3 kHz)
Guessing modulation: Pulse Code Modulation (Not Return to Zero)
Attempting demodulation... short_width: 56, long_width: 56, reset_limit: 57344, sync_width: 0
Use a flex decoder with -X 'n=name,m=FSK_PCM,s=56,l=56,r=57344'
pulse_demod_pcm(): Analyzer Device
bitbuffer:: Number of rows: 1
[00] {162} aa aa aa 2d d4 0e 51 00 6b 58 6e 7f 24 f8 d2 ff ff ff 3c 28 00

Problem is the CC1101 is inserting the packet length byte (0e) into the stream, so it will not decode.
Have looked at various datasheets CC1101, CC430 etc. Packet Length byte is described as optional - but documentation varies as to which mode it is optional (may be Variable). Have tried fixed, variable and infinite - no change.
Have loaded the modified Elechouse_CC1101 CPP file and my platformio project main.cpp file.

CC1101 WH51.zip

[NorthernMan54]

I would say that you have nailed it

Detected FSK package	2021-04-11 18:18:51
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
time      : 2021-04-11 18:18:51
model     : Fineoffset-WH51                        ID        : 006b58
Battery level: 0.778     Battery   : 1400 mV       Moisture  : 36 %          Transmission boost: 3     AD raw    : 210           Integrity : CRC
pulse_demod_pcm(): Fine Offset Electronics/ECOWITT WH51 Soil Moisture Sensor
bitbuffer:: Number of rows: 1 
[00] {162} aa aa aa 2d d4 0e 51 00 6b 58 6e 7f 24 f8 d2 ff ff ff 3c 28 00 
Analyzing pulses...
Total count:   40,  width: 9.42 ms		( 2356 S)
Pulse width distribution:
 [ 0] count:   26,  width:   56 us [52;72]	(  14 S)
 [ 1] count:    6,  width:  116 us [112;120]	(  29 S)
 [ 2] count:    3,  width:  172 us [172;176]	(  43 S)
 [ 3] count:    1,  width:  408 us [408;408]	( 102 S)
 [ 4] count:    2,  width:  260 us [232;288]	(  65 S)
 [ 5] count:    1,  width: 1392 us [1392;1392]	( 348 S)
 [ 6] count:    1,  width:   20 us [20;20]	(   5 S)
Gap width distribution:
 [ 0] count:   24,  width:   56 us [56;68]	(  14 S)
 [ 1] count:    3,  width:  172 us [172;172]	(  43 S)
 [ 2] count:    2,  width:  316 us [288;348]	(  79 S)
 [ 3] count:    7,  width:  112 us [112;116]	(  28 S)
 [ 4] count:    1,  width:  520 us [520;520]	( 130 S)
 [ 5] count:    2,  width:  232 us [232;232]	(  58 S)
Pulse period distribution:
 [ 0] count:   18,  width:  116 us [112;140]	(  29 S)
 [ 1] count:    3,  width:  228 us [228;232]	(  57 S)
 [ 2] count:    8,  width:  172 us [172;176]	(  43 S)
 [ 3] count:    5,  width:  404 us [344;464]	( 101 S)
 [ 4] count:    2,  width:  288 us [288;292]	(  72 S)
 [ 5] count:    2,  width:  548 us [520;580]	( 137 S)
 [ 6] count:    1,  width: 1508 us [1508;1508]	( 377 S)
Pulse timing distribution:
 [ 0] count:   50,  width:   56 us [52;72]	(  14 S)
 [ 1] count:   13,  width:  116 us [112;120]	(  29 S)
 [ 2] count:    6,  width:  172 us [172;176]	(  43 S)
 [ 3] count:    2,  width:  376 us [348;408]	(  94 S)
 [ 4] count:    5,  width:  252 us [232;288]	(  63 S)
 [ 5] count:    1,  width: 1392 us [1392;1392]	( 348 S)
 [ 6] count:    1,  width:   20 us [20;20]	(   5 S)
 [ 7] count:    1,  width:  520 us [520;520]	( 130 S)
 [ 8] count:    1,  width:    0 us [0;0]	(   0 S)
Level estimates [high, low]:  15879,     55
RSSI: -0.1 dB SNR: 24.6 dB Noise: -24.7 dB
Frequency offsets [F1, F2]:   -1551, -17242	(-5.9 kHz, -65.8 kHz)
Guessing modulation: No clue...

I went and changed the rtl_443 WH51 device decoder to include the additional '0e' in the preamble, rebuild rtl_433 and it worked on the first try

fineoffset.c

static int fineoffset_WH51_callback(r_device *decoder, bitbuffer_t *bitbuffer)
{
    data_t *data;
    uint8_t const preamble[] = {0xAA, 0x2D, 0xD4, 0x0e};
    uint8_t b[14];
    unsigned bit_offset;

Now onto the next phase, getting this to work within the rtl_433_ESP

[AusGunno]

Agree - for the purposes of testing it will work. But I did solve the issue. The ELECHOUSE driver is inserting the length of the message into the FIFO buffer:

void ELECHOUSE_CC1101::SendData(byte *txBuffer, byte size, int t)
{

SpiWriteReg(CC1101_TXFIFO, size);

SpiWriteBurstReg(CC1101_TXFIFO, txBuffer, size); //write data to send
SpiStrobe(CC1101_SIDLE);
SpiStrobe(CC1101_STX); //start send
delay(t);
SpiStrobe(CC1101_SFTX); //flush TXfifo
trxstate = 1;
}

If we comment out that line it works as intended with the standard definition in RTL_433.

Detected FSK package 2021-04-13 11:56:02

---

time : 2021-04-13 11:56:02
model : Fineoffset-WH51 ID : 006b58
Battery level: 0.778 Battery : 1400 mV Moisture : 36 % Transmission boost: 3 AD raw : 210 Integrity : CRC
Analyzing pulses...
Total count: 42, width: 9.11 ms ( 2278 S)
Pulse width distribution:
[ 0] count: 30, width: 56 us [52;68] ( 14 S)
[ 1] count: 6, width: 116 us [116;120] ( 29 S)
[ 2] count: 2, width: 172 us [172;172] ( 43 S)
[ 3] count: 1, width: 404 us [404;404] ( 101 S)
[ 4] count: 1, width: 292 us [292;292] ( 73 S)
[ 5] count: 1, width: 1392 us [1392;1392] ( 348 S)
[ 6] count: 1, width: 232 us [232;232] ( 58 S)
Gap width distribution:
[ 0] count: 28, width: 56 us [56;68] ( 14 S)
[ 1] count: 4, width: 172 us [172;176] ( 43 S)
[ 2] count: 1, width: 520 us [520;520] ( 130 S)
[ 3] count: 2, width: 228 us [228;232] ( 57 S)
[ 4] count: 6, width: 116 us [116;116] ( 29 S)
Pulse period distribution:
[ 0] count: 22, width: 116 us [112;136] ( 29 S)
[ 1] count: 4, width: 228 us [228;232] ( 57 S)
[ 2] count: 8, width: 172 us [172;176] ( 43 S)
[ 3] count: 4, width: 504 us [460;580] ( 126 S)
[ 4] count: 2, width: 316 us [288;348] ( 79 S)
[ 5] count: 1, width: 1508 us [1508;1508] ( 377 S)
Pulse timing distribution:
[ 0] count: 58, width: 56 us [52;68] ( 14 S)
[ 1] count: 12, width: 116 us [116;120] ( 29 S)
[ 2] count: 6, width: 172 us [172;176] ( 43 S)
[ 3] count: 1, width: 404 us [404;404] ( 101 S)
[ 4] count: 2, width: 292 us [292;296] ( 73 S)
[ 5] count: 1, width: 1392 us [1392;1392] ( 348 S)
[ 6] count: 3, width: 228 us [228;232] ( 57 S)
[ 7] count: 1, width: 520 us [520;520] ( 130 S)
Level estimates [high, low]: 5572, 9
RSSI: -4.7 dB SNR: 27.9 dB Noise: -32.6 dB
Frequency offsets [F1, F2]: 24348, 6356 (+92.9 kHz, +24.2 kHz)
Guessing modulation: Pulse Code Modulation (Not Return to Zero)
view at https://triq.org/pdv/#AAB02A08010038007400AC01940124057000E40208808080808080808080808080808080828090A0808280828755+AAB01708010038007400AC01940124057000E40208908090809655+AAB01D08010038007400AC01940124057000E4020890A1B18181C2908180D1E655+AAB01408010038007400AC01940124057000E40208808455
Attempting demodulation... short_width: 56, long_width: 56, reset_limit: 57344, sync_width: 0
Use a flex decoder with -X 'n=name,m=FSK_PCM,s=56,l=56,r=57344'
pulse_demod_pcm(): Analyzer Device
bitbuffer:: Number of rows: 1
[00] {162} aa aa aa aa 2d d4 51 00 6b 58 6e 7f 24 f8 d2 ff ff ff 3c 28 00

[NorthernMan54]

Tks for finding that

I have updated the example in the FSK branch with this

3f2ca30

I'm still working thru getting the receiver to work with FSK modulation. I had thought it would be just set the modulation and deviation to match your sample to go. But nope.

[DigiH]

Watching this closely, and already tried some testing with the fsk branch to get my (possibly)

memcpy(&cfg->devices[48], &fineoffset_wh1080_fsk, sizeof(r_device));

weather station added. No luck so far ;)

[NorthernMan54]

@DigiH with FSK, I’m currently work thru getting the cc1101 transceiver to receive FSK signals in asynchronous serial mode ( the cc1101 needs to have its packet control functions disabled to function as a simple receiver ). Next step after that is to add a signal decoder for fsk encoding pulse streams to rtl_433 bitbuffers.

As I don’t have a FSK device available I’m using the sample fsk transmitter code from above on one device to transmit a signal to test against. So far after 3 evenings I can see the test signal (Rssi), but no pulses yet. What is frustrating is that I see the occasional ook signal and pulses from my ook devices.

[DigiH]

@NorthernMan54 - my issues here being that I ordered a weather station, which I've always been interested in getting one, after reading some comments in issues in the RTL_433 repo.

There the command to get info from the station is stated as

/usr/local/bin/rtl_433 -f 868.319M -R 32 -d1 -p 28 -Y classic -s 250k

which to me sounds it needs the classic FSK pulse detector mode for its recognition.

In fineoffset_wh1080.c I can also see

.modulation = FSK_PULSE_PCM,

Since these stations seem to be rebranded by many companies I'm also not sure if this particular company might have added some changes to the signals. I might decide to return the station for the time being, and getting another one at a later stage. Also I should finally get a Realtek RTL2832 based DVB dongle to do some direct RTL_433 RTL-SDR testing of the devices I get ;)

I'm more than happy to do any testing with the weather station with your changes, as long I I still have it.

[NorthernMan54]

With the way a cc1101 works, the receiver would not be able to receive ook and fsk signals at the same time. As the signal demodulation is done in the cc1101 chip, it needs to know the modulation first. So if you have devices with fsk and ook you will need multiple receivers.

[DigiH]

I've already made two gateways, one for 433Mhz, all ook devices so far, and one for 868Mhz, where I only have the one, probably fsk, device. So that would not be an issue for me.

BTW, just ordered my Realtek RTL2832 based DVB dongle. Should be here on Saturday for more testing of my weather station.

[DigiH]

As an update. I have received my Realtek RTL2832 dongle and tested my new weather station with the rtl_433 package. Contrary to my expectations it isn't recognised as a Fine Offset 1080/3080 station, but a Fine Offset WH65B, therefor using the same fineoffset.c device file as the WH51 soil moisture sensor, with the same FSK_PULSE_PCM modulation. With rtl_433 I'm getting

time      : 2021-04-17 12:28:47
model     : Fineoffset-WH65B                       ID        : 72
Battery   : 1            Temperature: 20.6 C       Humidity  : 41 %          Wind direction: 27        Wind speed: 0.0 m/s       Gust speed: 0.0 m/s       Rainfall  : 0.0 mm        UV        : 1
UVI       : 0            Light     : 0.0 lux       Integrity : CRC

every 16 seconds, as expected.

So the station stays and I'm ready for more testing with rtl_433_ESP.