[FR] Add support for PVVX Encrypted advertising beacon

[ilgrank]

I'd want to avoid telling everyone+their dog what's the temperature in my house 😛

PVVX, the author of the most well-known alternative firmware for the Xiaomi LYWSD03MMC (and most other ATC based sensor) has added some time ago the option to encrypt the advertisement beacons (that have temperature and humidity information among other things),

If I get it correctly, at the moment the encrypted advertising format is not supported by OpenMQTTGateway.. (I've searched all the MD files, but found no mention of it), so I just wanted to ask if there's any chance to add support for it

[joshuakraemer]

+1, I would also like to use the pvvx firmware with encrypted advertising data.

I think this would require:

• A decoder in https://github.com/theengs/decoder to get the ciphertext and mic from the advertising data
• Decrypting the ciphertext in OMG with a user-provided key

See also #779

[DigiH]

Hi @ilgrank and @joshuakraemer

I assume you are both solely talking about the same above mentioned energy efficient encryption format introduced by PVVX, as there are also two other encrypted formats for these devices.

We have just started implementing supporting encrypted formats in Theengs Decoder 1.5.5 , and so far with Theengs Gateway testing the decryption implementation.

Would you both mind posting some example encrypted adversing data so I can get started with a decoder, and possibly also allow you to test with Theengs Gateway for verification.?

Also stating with which device(s) you are using the PVVX firmware, if not obvious from the data.

After the initial testing with Gateway this feature is also planned to be ported to OpenMQTTGateway.

Thanks

[joshuakraemer]

That's great news, @DigiH!

Yes, I was talking about the encrypted, energy-efficient "custom" format of the pvvx firmware. The format and the decryption algorithm are described here: pvvx/ATC_MiThermometer#94 (comment)

I'm using Xiaomi LYWSD03MMC devices. Here is a sample message from Theengs Gateway:

{"servicedatauuid": "181a", "servicedata": "23ef56583dd42050fe8e4d", "name": "ATC_9C58AB", "id": "A4:C1:38:9C:58:AB", "rssi": -60}

Can you tell me how to enable the advanced data in Theengs Gateway?

[DigiH]

Thanks for the example @joshuakraemer, Theengs Gateway already shows the data when no decoder is found, so all is fine.

I'll have a look into it.

[DigiH]

@joshuakraemer

Just to clarify, is your above sample encrypted PVVX format or ATC?

Also do you see other messages from your LYWSD03MMC containing

… "servicedatauuid": "161a" …

Thanks

[joshuakraemer]

@DigiH, my sample is PVVX format. I'm sorry, it seems the link from above doesn't describe the actual format. The relevant declarations can be found here: https://github.com/pvvx/ATC_MiThermometer/blob/62c246d961dc1bc0e3b23a6087b8ff46c53ac696/src/custom_beacon.h#L49

However, I'm confused by the code. I think the comments giving the byte numbers are wrong. According to the code, the message should consist of 5 bytes header (adv_cust_head_t), 6 bytes data (adv_cust_data_t) and 4 bytes mic. This however doesn't match with the reported service data, which consists of 11 bytes.

I only get messages with a uuid of "181a".

By the way, I've disabled BLE advertising flags in the firmware configuration.

[DigiH]

However, I'm confused by the code. I think the comments giving the byte numbers are wrong. According to the code, the message should consist of 5 bytes header (adv_cust_head_t), 6 bytes data (adv_cust_data_t) and 4 bytes mic. This however doesn't match with the reported service data, which consists of 11 bytes.

That servicedata length difference was confusing me too :) but the uuid part of the header is already being reported separately with Gateway as the servicedatauuid. I was initially still expecting 12 bytes in the actual servicedata to account for that.

Possibly I just got distracted by a given example of
Received: b'\x0e\x16\x1a\x18\x7f&\xecSL\x8ba\x8d\x054-'
further down the issue.

Therefore I thought you might have had the ATC option plus encrypted selected when flashing, but from what you say you selected PVVX (Custom) with Encrypted Beacon ticked behind, right?

So probably best to have a sleep over it and look at it again afresh tomorrow. I'm sure it will all come together :)

[DigiH]

Possibly I just got distracted by a given example of
Received: b'\x0e\x16\x1a\x18\x7f&\xecSL\x8ba\x8d\x054-'
further down the issue.

That's exactly what got my in the wrong direction. Looking at it again all is clear now 👍

We'll have more tomorrow …

[joshuakraemer]

Therefore I thought you might have had the ATC option plus encrypted selected when flashing, but from what you say you selected PVVX (Custom) with Encrypted Beacon ticked behind, right?

Yes, "Advertising type" is "PVVX (Custom)", "AdFlags" is off, "Encrypted beacon" is on.

The MiTemperature2 script and the PVVX python interface both decrypt the messages correctly. Relevant code:

• https://github.com/JsBergbau/MiTemperature2/blob/a680630e211faf26fea80a0922155191f3636c99/LYWSD03MMC.py#L659
• https://github.com/pvvx/ATC_MiThermometer/blob/62c246d961dc1bc0e3b23a6087b8ff46c53ac696/python-interface/atc_mi_interface/atc_mi_construct.py#L41

However, looking at those sources confuses me even more. There's no counter byte in both, while it's present in the firmware source.

@pvvx, could you please clarify the message structure?

[koenvervloesem]

@joshuakraemer I figured out the decryption thanks to your code references, and I was able to decrypt my LYWSD03MMC's advertisements in PVVX custom encrypted beacon format. I'm adding it to Theengs Gateway. After testing we can port this to OpenMQTTGateway too.

[joshuakraemer]

@koenvervloesem That's great, thank you very much! Please keep me informed so I can help with testing.

[koenvervloesem]

So to test this, you need:

• Theengs Gateway installed from this PR: Add support for encrypted PVVX advertisements theengs/gateway#146
• Theengs Decoder development version

Then supply the device address and accompanying bindkey to Theengs Gateway with the --bindkeys argument.

This should decrypt and then decode your encrypted advertisements and show your LYWSD03MMC's temperature, humidity and battery percentage. If not, have a look at Theengs Gateway's output for error messages.

[joshuakraemer]

Great, it works for me. I guess the counter byte is not included in the broadcast message? This is an example message from TheengsGateway:

{"name": "ATC_9C58AB", "id": "A4:C1:38:9C:58:AB", "rssi": -59, "brand": "Xiaomi", "model": "TH Sensor", "model_id": "LYWSD03MMC/MJWSD05MMC_PVVX_DECR", "type": "THB", "tempc": 30.92, "tempf": 87.656, "hum": 44.05, "batt": 81}

On a side note, it would be nice if development versions of TheengsDecoder and TheengsGateway could be installed directly from git (didn't work for me because setup.py doesn't include a valid version string).

[DigiH]

I guess the counter byte is not included in the broadcast message?

Thanks for the feedback @joshuakraemer. Do you see any need or advantage for the counter byte being included in the final decrypted and decoded JSON?

[koenvervloesem]

I guess the counter byte is not included in the broadcast message?

This is actually because decoding decrypted advertisements is implemented as a three-step process:

• Decoding the encrypted advertisement results in properties cipher, mic and ctr.
• The gateway recognizes this as an encrypted advertisement and calls the right decryptor for this model ID.
• The decrypted result is sent to Decoder again, which results in the temperature, humidity and battery properties, but then the properties of the original encrypted advertisement aren't available anymore.

We could leave the original properties of the encrypted data in the message, or only the counter, but the question is: would you use this counter? Does this have meaning for the unencrypted data? Interesting side note: the other supported encrypted advertisements, from the Shelly Button1, already have a packet ID (which acts as a counter) in their decrypted payload.

As for installing the development versions from Git, yes, that's a valid suggestion, I have already been thinking that it would be nice, I'll add it to our roadmap.

[joshuakraemer]

The counter is needed to detect duplicate and counterfeit messages. So yes, I think it would be useful to have it.

[koenvervloesem]

Ok that makes sense. I'll make Gateway publish the properties cipher, mic and ctr when PUBLISH_ADVDATA is 1.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.

[ilgrank]

Sorry, I has completely missed this, me bad
Thanks @koenvervloesem !