Feature request question: Very energy efficient encrypted custom format?

[JsBergbau]

Hi Victor,

thanks for all your work with this sensor.
I know you have enabled to send encrypted packages like the original firmware does. However these packets require 25 Bytes transmitted as advertising data. The format I propose would be 11 Bytes, 1 byte packet length, 2 Bytes service UUID and 8 Bytes payload.
These 8 Byte payloads are 64 Bits which is excactly the blocksize of Blowfish encryption. AES always uses 128 Bit (16 Bytes) blocksize, so the double amount of data needed to be transmitted. You could use AES as a streamcipher, then you also could transmit 8 Bytes, but I don't see any way how to keep track of the nedeeded Inialisation Vector IV without sending more data again. So blowfish would be the ideal encryption alogrithm here. Blocksize of 64 Bits is only a problem because of birthday paradoxon. Since we have here so little data we will never have any trouble with this.

So you can configure a static key, derived by a Password via PBKDF2 (or even argon2 function) to encrypt all your Mi Beacons.

I know this is a heavy feature request and thats why I've called it "feature request question".
I perfectly understand if you say "Sorry thats not worth the effort". I liked that idea very much and so I just wrote this feature request.

When you're interested I have some ideas for the design of the unencrypted dataformat, which I'll write happily down.

[pvvx]

It's not hard for me to fit any format, but the changed format requires third party application support. Application support for the new format will be very slow. Usually it takes from 3 months if the format is available for several popular sensors. Those. thousands of users are required for popular software to support the new format.
Delete old formats - exclude work in current smart home applications. The combined version with the transition to a new format will not be supported by third-party programs for many years, since everything works on old formats.
The new formats make sense only in the switching option in the source code, since only those who can assemble the firmware will be able to connect them.

[JsBergbau]

Thanks for your detailed answer.

Wouldn't it be possible just to add a 4th select option beside "Custom", "ATC_1441" and "MiLike" ? I hoped that it is possbile this way. Since new Payloadlength differs from length of previous payloads software should be able to distinguish the new packets and ignore it if they don't have support for that new format.
So there would be plenty of time for custom software to adopt to the new secure encrypted format.

[pvvx]

From the very beginning, I built a format, different in length, so that third-party programs could be corrected. But this is not happening. Users are encouraged to switch the format to the old one and disable the new one :)
In 80% of cases, no package parsing scripts were found. Data from the packet is taken at fixed positions and no parsing, even the packet length is not taken into account. There are checks only for MAC.

Entering an additional format will increase the size of the firmware and will not change anything, except for the questions - why does not it work? :)
To save battery, the ad format is not important. There are other ways to improve energy efficiency ...

[JsBergbau]

I think most developers just didn't know how to parse it properly. It also makes sends to only activate the format you need because then you have more transmits in the desired format and thus a higher reception chance. And especially when you want to activate the encrypted format, users should explicitly not send also the unencrypted packets ;).

But yeah I understand how frustrating this can be when third party software isn't adapted after months. At least the questions "why does ist not work" can be prevented by some kind of checkbox "really enable encrypted format, I know what I am doing" and then "error no encryption password set" if they forget to fill password.

Concerning batterysaving I meant using Blowfish to 8 Bytes. I think there is a natively AES support, or at least already built in. One could also use that but then have 16 Bytes Payload instead of small 8 Bytes.

[pvvx]

In the current version of mijia encryption, only 3..4 bytes are encrypted...

[JsBergbau]

What does that mean exactly? Which reported values are encrypted, which aren't?

[pvvx]

https://github.com/pvvx/ATC_MiThermometer/blob/master/test_adv_key.py

# 0  1  2 3  4 5  6 7  8  9         14 15                    26
# 1a 16 95fe 5858 5b05 a8 ed5e0b38c1a4 0239ff0e350000002f044957
#                 pid  cnt mac         crypt data      mic

Only "crypt data".

Some interfaces do not transmit the MAC address. Therefore, it is inserted into data PDU messages. This makes it impossible to reduce the message length to 8 bytes.

[JsBergbau]

So Bytes 15 .. 26 are encrypted?

This makes it impossible to reduce the message length to 8 bytes.

With a new encryption format it should be possible.
Take ATC_1441 as base, remove MAC, you have then 7 byte. Now add 1 magic byte which is always 0xFF. Encrypt data and on decrypt when magic Byte is 0xFF then you now decryption was sucessfull. This was just an example. As I said I have some ideas in improving that format.
If interface doesn't transmit mac address software has to try all encryption keys. Since the numer is very small it is feasible to check all keys an each paket.

[pvvx]

So Bytes 15 .. 26 are encrypted?

Participation in encryption in this example is taken by: 5 bytes (crypt data) + 4 bytes (mic).
The rest of the bytes are constants.

[pvvx]

Payload: Ad len - 1 byte, ad type = 0x16 - 1 byte, UUID16 - 2 bytes, data - 8 bytes: temperature 2, humidity 2, battery level 1, battery voltage 2, counter 1. Total: 1 + 1 + 2 + 8 = 12 bytes

Whole package: 16 + 12 = 28 bytes
In the current version, these are packages: 16 + [17..27] = 33..43 bytes

43/28 = max x 1.53

Advertising interval * 1.5 will come out even less consumption...

[pvvx]

@JsBergbau - Format 11 bytes.
AD_sturct[11]:

N-bytes	Info	Value
0	AD structure size	0x0A
1	Type =  'UUID16'	0x16
2..3	UUID = 'User Data'	0x181C
4	Temp, С	x0.5 - 40
5	Humidity, %	x0.5
6	Battery , % + flag (bit7)	&0x7F
7..10	mic	32 bit

Used by bindkey, AES.MODE_CCM.

Name	Info	bytes
Nonce	MAC[:6] + AD_struct[:4]	10
Mic	AD_struct[-4:]	4
data	AD_struct[4:-4]	3

Test:

import struct
import binascii
from Crypto.Cipher import AES

def parse_value(hexvalue):
	vlength = len(hexvalue)
	# print("vlength:", vlength, "hexvalue", hexvalue.hex(), "typecode", typecode)
	if vlength >= 3:
		temp = hexvalue[0]/2 - 40
		humi = hexvalue[1]/2
		vbat = hexvalue[2] & 0x7F
		trg =  hexvalue[2] >> 7
		print("Temperature:", temp, "Humidity:", humi, "Battery:", vbat, "Trg:", trg)
		return 1
	print("MsgLength:", vlength, "HexValue:", hexvalue.hex())
	return None

def decrypt_payload(payload, key, nonce):
	token = payload[-4:] # mic
	cipherpayload = payload[:-4] # EncodeData
	print("Nonce: %s" % nonce.hex())
	print("CryptData: %s" % cipherpayload.hex(), "Mic: %s" % token.hex())
	cipher = AES.new(key, AES.MODE_CCM, nonce=nonce, mac_len=4)
	cipher.update(b"\x11")
	data = None
	try:
		data = cipher.decrypt_and_verify(cipherpayload, token)
	except ValueError as error:
		print("Decryption failed: %s" % error)
		return None
	print("DecryptData:", data.hex())
	print()
	if parse_value(data) != None:
		return 1
	print('??')
	return None

def decrypt_aes_ccm(key, mac, data):
	print("MAC:", mac.hex(), "Binkey:", key.hex())
	print()
	adslength = len(data)
	if adslength > 8 and data[0] <= adslength and data[0] > 7 and data[1] == 0x16 and data[2] == 0x1C and data[3] == 0x18:
		pkt = data[:data[0]+1]
		# nonce: mac[6] + head[4] + cnt[1]
		nonce = b"".join([mac, pkt[:4]])
		return decrypt_payload(pkt[4:], key, nonce)
	else:
		print("Error: format packet!")
	return None
#=============================
# main()
#=============================
def main():
	print()
	print("====== Test encode -----------------------------------------")
	temp = 22.5
	humi = 50
	vbat = 99
	trg  = 1
	print("Temperature:", temp, "Humidity:", humi, "Battery:", vbat, "Trg:", trg)
	print()
	data = struct.pack(">BBB", int((temp + 40)*2), int(humi*2), vbat + (trg << 7)) 
	mac = binascii.unhexlify('001122334455') # MAC
	binkey = binascii.unhexlify('00112233445566778899AABBCCDDEEFF')
	print("MAC:", mac.hex(), "Binkey:", binkey.hex())
	adshead = struct.pack(">BBH", len(data) + 7, 0x16, 0x1C18) # ad struct head: len, id, uuid16
	beacon_nonce = b"".join([mac, adshead])
	cipher = AES.new(binkey, AES.MODE_CCM, nonce=beacon_nonce, mac_len=4)
	cipher.update(b"\x11")
	ciphertext, mic = cipher.encrypt_and_digest(data)
	print("Data:", data.hex())
	print("Nonce:", beacon_nonce.hex())
	print("CryptData:", ciphertext.hex(), "Mic:", mic.hex())
	adstruct = b"".join([adshead, ciphertext, mic])
	print()
	print("AdStruct:", adstruct.hex())
	print()
	print("====== Test decode -----------------------------------------")
	decrypt_aes_ccm(binkey, mac, adstruct);

if __name__ == '__main__':
	main()

A counter is also needed to track the message. It allows tracking the measurement number and protects against counterfeiting.
The control of forgery of messages during reception is carried out by monitoring the counter value with checking the value is less or repeating more than the specified number of times. Detecting Fake Sensors in Mijia ...
Then the message grows to 12 bytes.

N-bytes	Info	Value
0	AD structure size	0x0B
1	Type =  'UUID16'	0x16
2..3	UUID = 'User Data'	0x181C
4	counter	0..0xff
5	Temp, С	x0.5 - 40
6	Humidity, %	x0.5
7	Battery , % + flag (bit7)	&0x7F
8..11	mic	32 bit

AES.MODE_CCM:

Name	Info	bytes
Nonce	MAC[:6] + AD_struct[:5]	11
Data	AD_struct[5:-4]	3..
Mic	AD_struct[-4:]	4

[JsBergbau]

Hi @pvvx

thanks for your work.

Is it possible to derive bindkey from a Password via argon2 or pbkdf2? I think this long bind key is very complicated to handle for most users whereas a password like "correct horse battery staple" is much easier to handle. The derivation could be done in the TelinkFlasher HTML page, since there are much more ressources than on the device.

The 4 Bytes from 7 to 10 are needed to transmit a kind of initialization vector to have different ciphertext when temperature, humidty and battery stays the same. Did I get this correctly?

Regarding counterfeiting messages and counter: So this counter should prevent replaying messages? So when having values 0-255 and value is incremented by one after each measurement interval and the interval is 10 seconds, then after about 40 minutes the same packet could be replayed / injected again as it then seems valid, woulnd't it? In addition this is perhaps even much earlier possible because you should also accept packets with a higher counter because you could have missed some messages. Then this time is even much shorter. Don't think this counter would raise security level a lot or am I getting something wrong?
What about the mic / initializaton vector? Coulnd't this just be incremented by 1 each new dataframe? IV just has to be different for each key but not secret. So when having at least 32 Bit IV thats more than 4 Billion. With measurement interval of 10 seconds, that would be more than 1361 years until IV would repeat.

So fake data could be prevented with 3 Bytes IV, leading to more than 5 years until it repeats at 10 seconds measurement interval. To prevent fakes you could use the last 4 bits of temperature and humidity and set a specific pattern. If this pattern is found after decryption packet is original, if not, then data is disregarded as it is forged.
So you would even save at least one bit at the same security level.
The IV (then at length of 4 Bytes) could be initialized with unix timestamp minus seconds at 01.01.2021 or so. That would help to detect replayed data because timestamp is just too old and it has to grow all the time without any rollover. 1300 years should be long enough that never ever old packets needed to be accepted.

Its hard to express in english what I want to say. So if something is unclear, don't hesitate to ask.

[pvvx]

Is it possible to derive bindkey from a Password via argon2 or pbkdf2? I think this long bind key is very complicated to handle for most users whereas a password like "correct horse battery staple" is much easier to handle. The derivation could be done in the TelinkFlasher HTML page, since there are much more ressources than on the device.

bindkey is any 16 bytes.

The 4 Bytes from 7 to 10 are needed to transmit a kind of initialization vector to have different ciphertext when temperature, humidty and battery stays the same. Did I get this correctly?

No.
The CCM terminology "Message authentication code (MAC)" is called the "Message integrity check (MIC)" in Bluetooth terminology.

Nonce - For MODE_CCM, its length must be in the range [7..13]. Bear in mind that with CCM there is a trade-off between nonce length and maximum message size. Recommendation: 11 bytes.
https://pycryptodome.readthedocs.io/en/latest/src/cipher/aes.html

[JsBergbau]

bindkey is any 16 bytes.

Argon2 can be configured of how many bytes to return https://en.wikipedia.org/wiki/Argon2

Is in the Telink-Crypto library support for CFB-Mode? Then it would be possible to use 4 Bytes as IV and proceed as described above
https://pycryptodome.readthedocs.io/en/latest/src/cipher/classic.html#cfb-mode

[pvvx]

Is in the Telink-Crypto library support for CFB-Mode? Then it would be possible to use 4 Bytes as IV and proceed as described above

Using other functions will increase the size of the code.
AES CCM - standard in BLE...

---

If the counter does not change after the calculated time interval, this is the definition of a fake duplicate packet.

Xiaomi rolls back the fake after a few hours. The passed counter is 32-bit. Every 512 steps of the counter, the value is stored in Flash. Recovery requires the phone to be connected to the sensor in Mi-Home. When re-registering again (new bindkey), the counter is immediately added to + 512.
Sample Xiaomi adstruct:

# 0  1  2 3  4 5  6 7  8  9         14 15         20      23    26
# 1a 16 95fe 5858 5b05 a8 ed5e0b38c1a4 0239ff0e35 000000  2f044957
#                 pid  cnt mac         crypt data cnt1..3 mic

cnt, cnt1..3 - 32 bit count

I did not include all this in the code so that there was no support for Mi-Home. So far, no one will describe in an open source the entire Xiaomi identification, inserting the full code conflicts with the user agreement in Mi-Home.
And since the description (in source code) of mi-beacon has already been published in the public domain, in the next version I will insert this amendment... :)

[pvvx]

In ver 3.2 new crypto-advertising packages (AES.CCM) are added, according to the previously described format ...
https://github.com/pvvx/ATC_MiThermometer/blob/master/test_adv_cust.py

• https://github.com/pvvx/ble_monitor
  

Two formats:

1. 
2. 

[JsBergbau]

Thanks for implementing.

I have some troubles to decrypt:

def main():
	mac = binascii.unhexlify('bde39238c1a4')
	binkey = binascii.unhexlify('4651865E80C7B114CA682FB84F4BB9F0')
	print("MAC:", mac.hex(), "Binkey:", binkey.hex())
	print()
	print("====== Test1 ATC decode ----------------------------------------")
	#decrypt_aes_ccm(binkey, mac, adstruct);
	received=bytes.fromhex("0e 16 1a 18 7f 26 ec 53 4c 8b 61 8d 05 34 2d")
	print("Received:",received)
	decrypt_aes_ccm(binkey, mac,received);

Error is always

MAC: bde39238c1a4 Binkey: 4651865e80c7b114ca682fb84f4bb9f0

====== Test1 ATC decode ----------------------------------------
Received: b'\x0e\x16\x1a\x18\x7f&\xecSL\x8ba\x8d\x054-'
MAC: bde39238c1a4 Binkey: 4651865e80c7b114ca682fb84f4bb9f0

Nonce: bde39238c1a40e161a187f
CryptData: 26ec534c8b61 Mic: 8d05342d
Decryption failed: MAC check failed

Current bind key:

Copied data from Wireshark dump:

So whats wrong here?

PS: In #94 (comment) Byte 2..3 is stated as "0x181C" but it is advertised and checked in your code sample as "0x181A"

[pvvx]

https://github.com/pvvx/ble_monitor/blob/master/custom_components/ble_monitor/ble_parser/atc.py#L14
uuid16: 0x181A, msg_length: 12, 15, 17, 19 :)

[JsBergbau]

Setting is like in your screenshot above

Did I miss something to setup?

[pvvx]

Error in ATC format (encrypted). I'll fix it now ...

---

Corrected, checked, now I will update everything ...

---

Everything has been updated.

[pvvx]

https://github.com/pvvx/ble_monitor automatically selects the highest data ad option if all ad package options are enabled.
To switch to the reduced precision format, you need to reboot the HA server. Until I rewrote the select options in ble monitor ...
Therefore, I could not re-check after rewriting and including the short format ...

[pvvx]

PS: In #94 (comment) Byte 2..3 is stated as "0x181C" but it is advertised and checked in your code sample as "0x181A"

For now, I decided to leave the old UUID, and use the new one for notification by button and soil moisture. I am experimenting with the option of soldering simple two isolated electrodes in XiaomiLYWSD03MMC connected to the GPIO on which there will be a PWM-> ADC measurement of soil moisture or leakage. I am already rolling back the option with soldering to the reed switch board ...
XiaomiLYWSD03MMC is able to replace all Xiaomi sensors. :)

[JsBergbau]

Works now, thanks.

Is it possible instead of setting a bindkey directly to use a password to derive it via argon2id? This makes it much easier for people to set up encryption. Instead of handling this ugly bind key they can use a Password like "correct horse battery staple". This feature would have to be implemented in the TelinkFlasher Webpage, so there is no additional code or used capacity on the sensor device itself.
This library is free and opensource published under MIT license and can be used https://github.com/antelle/argon2-browser

A live demo is here https://antelle.net/argon2-browser/

I suggest memory size 10240, 64 Iterations, Has length 16 bytes of course and 4 parallelism.

With theses settings a Galaxy S10 Lite needs about 3,1 seconds.

With this approach even a simple PW is quite secure because only a few passwords can be tested per seconds.

[pvvx]

Perhaps, but it's easier to link to https://antelle.net/argon2-browser/
Usually the bindkey from MiHome is used - it is displayed in the XIAOMI integration of gateway 3 or assigned in flash registration in MiHome for subsequent recovery.