Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new model keys option #986

Merged
merged 2 commits into from
Mar 9, 2019

Conversation

zuckschwerdt
Copy link
Collaborator

This PR allows a way to transition to new unique, short, and uniform model "keys" instead of the old often descriptive model "names".

For now early adopter users can opt-in to new model keys using -M newmodel. At some point after the next release we can default to new model keys and offer a -M oldmodel for legancy use.

A few changed model keys are included as example.

@zuckschwerdt zuckschwerdt added the wip Work In Progress (for PRs only) label Feb 20, 2019
@zuckschwerdt
Copy link
Collaborator Author

zuckschwerdt commented Feb 20, 2019

Overview of all current model names and some proposed new keys.
Extraction with e.g.:

perl -ne 'print "$1\n" if m/"model",.*DATA_STRING,\s*(.*)/' src/devices/*.c
grep 'model = ' src/devices/*.c

Also note weather (rain/wind) key changes in #1019, other odd key changes in #1010, and that battery:"OK"|"LOW" is replaced by battery_ok:1|0.

Current model new model
Acurite Rain Gauge Acurite-Rain
Acurite 609TXC Sensor Acurite-609TXC
Acurite Lightning 6045M Acurite-Lightning
Acurite tower sensor Acurite-Tower
Acurite 5n1 sensor Acurite-5n1
Acurite 5n1 sensor Acurite-5n1
Acurite 3n1 sensor Acurite-3n1
Acurite 986 Sensor Acurite-986
Acurite 606TX Sensor Acurite-606TX
00275rm Acurite-00275rm
00276rm Acurite-00276rm
Akhan 100F14 remote keyless entry Akhan-100F14
AlectoV1 Wind Sensor AlectoV1-Wind
AlectoV1 Rain Sensor AlectoV1-Rain
AlectoV1 Temperature Sensor AlectoV1-Temperature
Ambient Weather F007TH Thermo-Hygrometer AmbientWeather-F007TH
AmbientWeather-TX8300
AmbientWeather-WH31E
Blyss-DC5ukwh
Brennenstuhl RCS 2044 Brennenstuhl-RCS2044
Bresser 3CH sensor Bresser-3CH
Bresser-5in1
Biltema-Rain
Calibeur RF-104 Calibeur-RF104
Cardin S466 Cardin-S466
Chuango Security Technology Chuango-Security
CurrentCost TX CurrentCost-TX
CurrentCost Counter CurrentCost-Counter
Danfoss CFR Thermostat Danfoss-CFR
Digitech XC0324 Digitech-XC0324
Dish remote 6.3 Dish-RC63
DSC Contact DSC-Security
Efergy e2 CT Efergy-e2CT
Efergy Optical Efergy-Optical
Elro-DB286A
ELV-EM1000
ELV-WS2000
emonTx emonTx-Energy
ESA-x000
Esperanza EWS Esperanza-EWS
Fine Offset Electronics, WH2 Temperature/Humidity sensor FineOffset-WH2
Fine Offset WH2A sensor FineOffset-WH2A
Fine Offset WH5 sensor FineOffset-WH5
Rosenborg-66796
Telldus/Proove thermometer FineOffset-TelldusProove
Fine Offset WH24 FineOffset-WH24
Fine Offset WH65B FineOffset-WH65B
Fine Offset Electronics, WH25 FineOffset-WH25
Fine Offset Electronics, WH0530 Temperature/Rain sensor FineOffset-WH0530
Fine Offset WH1050 weather station FineOffset-WH1050
Fine Offset Electronics WH1080/WH3080 Weather Station FineOffset-WHx080
Fine Offset Electronics WH1080/WH3080 Weather Station FineOffset-WHx080
Fine Offset Electronics WH3080 Weather Station FineOffset-WHx080
Ford Car Remote Ford-CarRemote
FT-004-B Temperature Sensor FT-004B
GE Color Effects Remote GE-ColorEffects
Generic motion sensor Generic-Motion
Generic Remote Generic-Remote
Generic temperature sensor 1 Generic-Temperature
GT-WT02
HIDEKI TS04 sensor Hideki-TS04
HIDEKI Wind sensor Hideki-Wind
HIDEKI Temperature sensor Hideki-Temperature
HIDEKI Rain sensor Hideki-Rain
Honda Remote Honda-CarRemote
Honeywell Door/Window Sensor Honeywell-Security
Honeywell Wireless Doorbell Honeywell-ActivLink
HT680 Remote control HT680-Remote
IBIS beacon IBIS-Beacon
inFactory sensor inFactory-TH
Inovalley kw9015b Inovalley-kw9015b
Interlogix Interlogix-Security
Intertechno Intertechno-Remote
Kedsum Temperature & Humidity Sensor Kedsum-TH
Kerui Security Kerui-Security
LaCrosse TX Sensor LaCrosse-TX
LaCrosse TX Sensor LaCrosse-TX
LaCrosse TX141-Bv2 sensor LaCrosse-TX141Bv2
LaCrosse TX141TH-Bv2 sensor LaCrosse-TX141THBv2
TX29-IT LaCrosse-TX29IT
TX35DTH-IT LaCrosse-TX35DTHIT
LaCrosse WS LaCrosse-WS
LaCrosse WS LaCrosse-WS
LaCrosse WS LaCrosse-WS
LaCrosse WS LaCrosse-WS
LightwaveRF Lightwave-RF
Wireless M-Bus Wireless-MBus
Maverick ET73 Maverick-ET73
Maverick-ET73x
Mebus/433 Mebus-433
New Template
KlikAanKlikUit Wireless Switch KlikAanKlikUit-Switch
Nexa Nexa-Security
Nexus Temperature Nexus-T
Nexus Temperature/Humidity Nexus-TH
Oil Ultrasonic STANDARD Oil-SonicStd
Oil Watchman Oil-SonicSmart
THGR122N Oregon-THGR122N
THGR968 Oregon-THGR968
WGR968 Oregon-WGR968
BHTR968 Oregon-BHTR968
RGR968 Oregon-RGR968
THR228N Oregon-THR228N
THN132N Oregon-THN132N
RTGN129 Oregon-RTGN129
RTGN318 Oregon-RTGN318
THN129 Oregon-THN129
BTHGN129 Oregon-BTHGN129
Oregon Scientific UVR128 Oregon-UVR128
THGR810 Oregon-THGR810
THN802 Oregon-THN802
UV800 Oregon-UV800
PCR800 Oregon-PCR800
PCR800a Oregon-PCR800a
WGR800 Oregon-WGR800
CM160 Oregon-CM160
CM180 Oregon-CM180
CM180 Oregon-CM180
Oregon Scientific SL109H Oregon-SL109H
OSv1 Temperature Sensor Oregon-v1
Philips outdoor temperature sensor Philips-Temperature
Prologue sensor Prologue-TH
Proove Proove-Security
Quhwa doorbell Quhwa-Doorbell
RadioHead ASK RadioHead-ASK
Sensible Living Plant Moisture SensibleLiving-Moisture
RF-tech
Rubicson Temperature Sensor Rubicson-Temperature
S3318P Temperature & Humidity Sensor Conrad-S3318P
Schrader
Schrader Electronics EG53MA4
Silvercrest Remote Control Silvercrest-Remote
SimpliSafe Sensor SimpliSafe-Sensor
SimpliSafe Keypad SimpliSafe-Keypad
SimpliSafe Keypad SimpliSafe-Keypad
Smoke detector GS 558 Smoke-GS558
Solight TE44 Solight-TE44
Springfield Temperature & Moisture Springfield-Soil
Steelmate
TFA pool temperature sensor TFA-Pool
TFA-Twin-Plus-30.3049 TFA-TwinPlus
Thermopro TP11 Thermometer Thermopro-TP11
Thermopro TP12 Thermometer Thermopro-TP12
Citroen
Ford
PMV-107J
Renault
Toyota
Emos TTX201 Emos-TTX201
Vaillant VRT340f Central Heating Thermostat Vaillant-VRT340f
Vaillant VRT340f Central Heating Thermostat (RF Detection) Vaillant-VRT340f
Waveman Switch Transmitter Waveman-Switch
WG-PB12V1
WS Temperature Sensor Hyundai-WS
WT0124 Pool Thermometer WT0124-Pool
WT450 sensor WT450-TH
X10-RF
X10 Security X10-Security

@roger-
Copy link
Contributor

roger- commented Feb 24, 2019

Curious why not "FineOffset" instead of "Fineoffset"?

@zuckschwerdt
Copy link
Collaborator Author

Any more comments on unique, short, and uniform model "keys"? Otherwise I'll get basic support in and convert keys slowly. Old keys will be supported for at least 2 more releases (a year or so) in any case.

@roger-
Copy link
Contributor

roger- commented Mar 4, 2019

Otherwise looks good to me!

@zuckschwerdt zuckschwerdt removed the wip Work In Progress (for PRs only) label Mar 9, 2019
@zuckschwerdt zuckschwerdt merged commit 1c4ffd1 into merbanan:master Mar 9, 2019
@zuckschwerdt zuckschwerdt deleted the feat-newmodelkeys branch March 9, 2019 09:36
@zuckschwerdt
Copy link
Collaborator Author

All renames are listed in the table above and commited to master. Please everybody review carefully and comment with suggestions to improve.

@kukabu
Copy link
Contributor

kukabu commented Mar 12, 2019

'Schrader Electronics EG53MA4' to 'Schrader-EG53MA4'?

@zuckschwerdt
Copy link
Collaborator Author

Yes, all those TPMS need some attention too ;)

@roger-
Copy link
Contributor

roger- commented Mar 24, 2019

Any reason why you didn't split model into manufacture and model?

I didn't look close enough and though the '-' was the separator, but that's not always true.

@zuckschwerdt
Copy link
Collaborator Author

Can you explain in more detail? The keys should be manufacturer-model (except when one of them is unknown or generic.) Improvements welcome.

@roger-
Copy link
Contributor

roger- commented Mar 24, 2019

There are a few cases where things are missing and it's ambiguous, like Acurite-Lightning (lost the 6045M), Generic-Motion, Silvercrest-Remote.

For MQTT discovery it would be nice to have fields like model, manufacturer, name and type, and it's hard to extract these from the current model string.

@zuckschwerdt
Copy link
Collaborator Author

Yes, that should rather be Acurite-6045M. Adding a type will be a later change if we can establish consensus for that. For now look at the keys. E.g. "temperature" and "humidity" hint weather sensor.
Note that model is just a short unique key for the protocol. Not a description of any kind. You will never reliably detect the brand, manufacturer, reseller, and by extension model-names/ids.

@zuckschwerdt
Copy link
Collaborator Author

S.a. https://github.com/zuckschwerdt/rtl_433/blob/feat-mgmqtt/examples/rtl_433_mqtt_hass.py for a draft how Home Assistant discovery could work.

@klohner
Copy link
Contributor

klohner commented Mar 29, 2019

Would it be possible to change what's been proposed as "Honeywell-Doorbell" to "Honeywell-ActivLink" or "Honeywell-Home" or "Honeywell-Alarm"? ActivLink more accurately describes the signal protocol Honeywell seems to use for devices that support this protocol, but they've trademarked the term "ActivLink". There are over 100 SKU's of devices that use this protocol, which also include older Friedland Response and Friedland Libra+ devices (which I believe Honeywell bought the rights to), and the newest international Honeywell Home devices. These devices not only include doorbells, but also security devices such as motion sensors and door/window sensors.

Perhaps another option is to rename what's been proposed as "Honeywell-Security" to something else (I believe what is currently called the "Honeywell Door/Window Sensor" is an older, less-sophisticated protocol used by previous generation Honeywell devices), and use "Honeywell-Security" to refer to the newer, current ActivLink protocol line of devices?

@zuckschwerdt
Copy link
Collaborator Author

Sure, sounds good. -Security for the old one -ActivLink for the new one. Went in with 897f9db.

@helgew helgew mentioned this pull request Apr 2, 2019
@pite81
Copy link

pite81 commented Apr 14, 2019

This PR allows a way to transition to new unique, short, and uniform model "keys" instead of the old often descriptive model "names".

For now early adopter users can opt-in to new model keys using -M newmodel. At some point after the next release we can default to new model keys and offer a -M oldmodel for legancy use.

A few changed model keys are included as example.

I just installed the RTL_433, several times and I still cannot update it to the new model and when I try to run the app it does not run just pop up this message. Consider using the "-M new model" to transition to new model keys. This will become the default someday.
So how do I install new model?
thanks.

.

@zuckschwerdt
Copy link
Collaborator Author

I should just be

rtl_433 -M newmodel

@pite81
Copy link

pite81 commented Apr 14, 2019

I should just be

rtl_433 -M newmodel

Unfortunately, it does not work. That is all I have in return for that command. see picture.
RTL_433 new model

@zuckschwerdt
Copy link
Collaborator Author

I does work, you just don't have any inputs available. You should add at least RTL-SDR.
Use this commands to abort the compilation until you get librtlsdr-dev installed or something.

cd rtl_433
mkdir build ; cd build ; cmake -ENABLE_RTLSDR=ON .. && make && make install

@pite81
Copy link

pite81 commented Jul 14, 2019

will not work with rtl_433 as it is not supporting my station.

There are sure thousands of protocols out there, and we "only" support 150 or so ;) For now…
If you can capture a sample (-S all): then open a new issue and post some infos there. We might be able to add your type of protocol.

Now it is wired to the rpi, but soon I will eliminate the long wires and use mqtt instead.

Do you use cabled sensors, something like BME280 or DHT22 now? I have those on my Raspis too. All feeding into MQTT. Worth the effort!

Oh wait, your sensor has an ESP in it? That's very high-tech. Lot's of possibilities. What sensor is that? Do you have a link?

As you say, exactly I use BME 280 on my terrace wall so I have o plenty of data coming in. A DHT22 on the other side of the wall which measures the internal temp and hum. I can add whatever sensor I want to. I also have ESP8266 based anemometer from adafruit, which is pretty good. It is under testing now, I placed right next to the existing weather station to see the accuracy. All the data transmitted via mqtt to my raspberry and my nas. Both running node-red server.

My weather station has an internal unit which is sending the packet to a web based dashboard. I opened the case and there is an ESP WT8266-S1board which is connected to my wifi router. I maybe a serial connection is possible. If we could download the sketch and then modify it with an addition of mqtt publish command, then all the data can be captured. Opinion?
Thanks.

This is the unit I use.
https://www.weatherspares.co.uk/ventus-w830-colour-weather-station-with-wifi-internet-connection-542-p.asp

IMG_20190714_174622
IMG_20190714_174608

@zuckschwerdt
Copy link
Collaborator Author

Dumping the binary code from the ESP is possible, but you will have to reverse engineer the functionality (source code). It could be possible to change the web dashboard URL in the firmware and re-upload. Then run your own webserver to capture the data.

@dgrostoto
Copy link
Contributor

Hello,
Do you think that newmodel is now mature enough (and stable) to be integrated in other software like domoticz (which use cvs output) ?

@dgrostoto
Copy link
Contributor

dgrostoto commented Jan 28, 2020

i saw the reponse on #1277 (comment)

@zuckschwerdt
Copy link
Collaborator Author

Yes, newmodel improves the format of model to be consistent and also normalizes a lot of keys (esp. weather stations will mostly show the same keys now).
The grace period to switch to newmodel is now up (it's been around for year!). We'll default to newmodel shortly.
You can then still switch to oldmodel for maybe half a year before we remove the old stuff.

@dgrostoto
Copy link
Contributor

I'm trying to make domoticz (rtl_433 data consumer program) newmodel compliant.
I created some (not so) naïve parser to get all keys used in rtl_433 decoders.
Here is the source code if any other program (than domoticz) want to check the full keys model (and may be used also for checking regression).
extract_keys.awk.txt
example of use :
cat rtl_433-master/src/devices/*.c | gawk -f extract_keys.awk | grep -E -v "^model(" | sort | uniq -c | sort -g
will get this file which are all keys sorted by number of occurence :
finish.txt

@zuckschwerdt
Copy link
Collaborator Author

Thanks! I'm working on a script to extract all useful metadata from our decoders, but it will take some time to finsh that. Having something working right now is great.

@dgrostoto
Copy link
Contributor

Maybe others keys need to be checked (found in my extraction) :
1 "energy"
(instead of energy with a unit), and maybe :
1 "power1"
1 "power2"
1 "power_idx"
1 "power_max"
and
2 "power0"

@zuckschwerdt
Copy link
Collaborator Author

Not sure if there is a unit to power_idx/power_max.
But yes, it should be energy_kWh and e.g. power1_W.

@dgrostoto
Copy link
Contributor

i PR'ed the domoticz project to support the newmodel. Now units should be better handled, (i had several problems with those units), i think that more devices will now report nicely between rtl_433 and domoticz.

@hackery
Copy link
Contributor

hackery commented Feb 10, 2020

Ah, so this is why everything here just stopped working. That'll teach me to track master without reading every commit message :)
I'd kind of like to see an explicit -M newmodel result in an additional field output as oldmodel to ease the transition, but it doesn't look like the selection model in the code is amenable to that. Oh well. Always Forward.

@dgrostoto
Copy link
Contributor

Well newmodel is now 1 year old now. If something broke don't hesitate to point out what is the problem if i did something wrong, i'll try to fix it.

@hackery
Copy link
Contributor

hackery commented Feb 10, 2020

True - but there's only been one release in that time, and no release notes to alert people to the upcoming deprecation; you wouldn't be prepared unless you spot the mention in the help output (-M help even), recognise that it could be important, chase into the commit history, and then update consumers with new keys from the code. You didn't break anything afaic, but maybe we should improve visibility. The change to the default should I think be considered a breaking change - might it be worth a note near the top of README.md ?

@merbanan
Copy link
Owner

As always, patch welcome. We are trying to get a release together and it is things like this I'd like to get listed somewhere. But it's hard to keep track of things and creating a sane api (command line/data) that doesn't break with minor changes.

@zuckschwerdt
Copy link
Collaborator Author

I'll try and compile the ChangeLog tomorrow. Likely I'll put the biggest updates on the README also. Thanks for the hint!

@dgrostoto
Copy link
Contributor

dgrostoto commented Feb 28, 2020

Hello, domoticz project is now newmodel compliant (merged in developpement branch). When i red issue related to rtl_433 i've found one issue that make me feel that integration could better than now. The problem is in csv output format that type of device is not formalized, this output does not tell the type of device it is about, when there is a temperature data it is easy to deduce it is a temperature sensor (that the way domoticz actually handle data coming from rtl_433 csv output) but when it comes to a switch (with multi-state), it is not always possible to use the corresponding sensor/actuator in domoticz. So there is only a portion of device recognized by rtl_433 that can are handled in domoticz.
See issue #1117
on rtl_433 side
reported here on domoticz side
domoticz/domoticz#3144
I suggest that the key model by completed with a new key (maybe a set of keys), that describe type/subtype of device with main characteristics.
At the moment, I don't have a model in mind (i have to think about it, with the list above this issue and my awk script will help much), but if you're OK with the principle of this proposition, i can go ahead and construct a model coherent for the 201 devices actually supported by rtl_433.
If this can be constructed and adhoc key(s) used on domoticz side but that's the easy part, then all devices will be automagically supported on domoticz (that's the cool thing) provided of course every device is flagged correctly on rtl_433. (and every new device be flagged at creation).
What's your thoughts about it ? and could it be interesting for other projects using outputs of rtl_433 ?

@zuckschwerdt
Copy link
Collaborator Author

We did try to classify devices in the past but there are quite some differences with individual outputs. The TPMS for example are tagged as a device class, but e.g. weather sensors will be quite different in capability and it's best to look at the output to infer what is transmitted.

We don't have many decoders for switches, this is best solved with a flex config. Perhaps we could allow adding meta-data with flex configs, not sure.

If you find keys that report the same thing but are represented differently in decoders then try to find and suggest a normalized common key, sure!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants