Advertisement messages publication to MQTT.

[robertcsakany]

I have Xiaomi Mijjia Temperature / Humadity sensor devices which are not using beacon protocols, it uses the plain advertisement messages with extended custom data messages which contain sensor data too. The current version of software was not able to handle the delegation of advertismenet messages, because if device was discovered never was republished.

[shmuelzon]

Hey Robert,

First of all, thanks for putting the time and submitting this pull request.

I do have some reservations, though. One of them is the fact that you're publishing the raw BLE advertising data requiring some other entity in the network to be familiar with the advertising packet format. Won't it be better using the current beacon implementation and add the Mijia temperature/humidity sensor to it and have it publish the data already formatted?

I'm actually considering refactoring the beacons implementation to send a single JSON payload instead of the multiple publications. Could also rename "beacon" to "broadcaster" to have it more generic.

What do you think?

Good day

[robertcsakany]

Hi,

Thank you for your fast response and your incredible work. I thinking about the choice that implementing the Mijjia protocol directly and the RAW, but originally for me it this form it is more flexible, because for home automation system easier to decode message when a new type of device arrives - I have other devices: TV, HIFI which sends standard AD messages, which can be used in automation with scripts - no need to compile, OTA which can cause pains for a lot of people. For in my case the ESP32 devices will be inside power plugs.

To achive the device protocol implementations made easy on ESP32 side have to create some compile/plugin system, because all new devices have to be implemented easy. I've spend some hours to understand how the actual implementation works (as a matter of fact I haven't made C coding in the last 10 years)

I propose other solution. What happens if I rename to RawAdvertisement - and if there is no interval on that case there is no message send (as it now) - it helps developers to decode messages, and thinking about some chain where protocol callbacks can be registered which can do the JSON transfrormations itself by device type - or some simple script which can be on the configuration file by users? (I can make some simple script prototype with ANTLR)

I don't know these functions are required or not in the standard service discovery / call mechanism, as a matter of fact I haven't got that deep knowledge of BT. What I see there is an include file with a tons of UUID codes related for some standard codes. I assume there is some device dependent service UUID's too. How can it handle it now when a device specific UUID code is used? But this is very interesting part, so I can dig it deeper in this topic.

[shmuelzon]

Hey,

The standard BLE services and characteristics, those that were defined by the Bluetooth SIG, are included in the application. That's the auto-generated include file you mentioned.
Of course, devices can choose to implement their own and those will be published as is. That means that the service name will be its UUID and the same for the characteristic. The data itself associated with the characteristic will be sent as a byte array.
While using the UUIDs and byte array can be done by an external script/application listening on the MQTT bus and acting upon it or re-transmitting it parsed on some other topic is possible, I wanted to keep all of the Bluetooth low-level stuff enclosed in this project. To that end I added the standard definitions and it can be extended via the configuration file.

This is all good and well until we talk about broadcasters (BLE devices that aren't connectable but only publish advertisement packets). There's also a standard set of options here, but manufactures tend to add their own custom messages.
If we could find a way to model these advertisements in the JSON configuration file, we could implement something that's generic enough. The problem is that I couldn't think of something that could work yet :)
The iBeacon format, for example, is quite simple as it's constant size and fields. The Eddystone beacon is more complex since the contents of the payload depend on the values of previous fields so you need a bit of logic to parse it correctly.
The Mijia temperature/humidity sensor is like that as well, the payload it sends could include both sensor readings, just one of them, or the battery level. You need to read the field indicating the inner payload to parse the rest.

I don't really mind have device-specific code in this project. If a device is common enough for multiple people who use it, then why not add it?

BTW - regarding OTA. My ESP's are a in junction boxes so their pretty much a pain to get to. Thankfully I've yet to brick any of them. Even if the OTA process breaks mid-way, there are 2 application images stored. The ESP will simply reboot and load the original image it was using up until now and you can try OTA again.

Anyways, here's my plan. I'll rename 'beacon' to 'broadcaster' as that would be a bit more generic. I've also already written the code that sends a single JSON instead of multiple MQTT topics. Should be easier to handle that way.
If you would agree to test it, I'll add a Mijia temperature/humidity parser, should be simple enough.
I'll keep thinking of a way to generalize, or at least extend the broadcaster parsing via configuration file, as was done for services and characteristics but I don't know how simple that would be due to the diversity of options.

Sorry for the long reply...

[shmuelzon]

Hey Robert,

Would you mind testing the mijia branch at tell me if you correctly receive the sensor information from it?

Thanks!

[robertcsakany]

Yes sure. I'will cheking it soon. Thanks for it!

[robertcsakany]

I've checked it. Its cool, working fine.

Another. I've updated esp-idf 3.1.2 and my ESP32 was more stable than the recommended one - It runs in the last 5 days without freezing. With the recommended version every 1-6 hour there was a freeze and the watchdog was not able to restart the device and cannot connect back.

[itProfi]

Can you say - what about distance in esp32 and you 3-4 Xiaomi Mijia? I have 4 pcs - and esp32 sensitivity is very poor..

[robertcsakany]

I have the WROOM version which have integrated antenna. I can say its about same coverage as with USB dongles. the longest distance is around 10 meter with 1 wall. I have lot of radio noise, I have 433Mhz devices and 4 Wifi router. (3 floor + outdoor). With 2 walls I had connection problems with the dongles too, but thats the case it is Low Energy. I wanna use ESP32 , I will put one ESP32 in every room inside the wall, so the coverage will be full. Maybe other version of ESP32 with external or cheramic antenna can be better, but the chip with a H-LINK PM-05 is cheaper like an antenna.

[shmuelzon]

Another. I've updated esp-idf 3.1.2 and my ESP32 was more stable than the recommended one - It runs in the last 5 days without freezing. With the recommended version every 1-6 hour there was a freeze and the watchdog was not able to restart the device and cannot connect back.

I’ll look into v3.1.2. I see some BLE bugs were fixed there. I’ve never had the ESP freeze like that, but if v3.1.2 is really stable, should be worth the upgrade and I’ll update the documentation.

[shmuelzon]

Hey Robert,

Thanks for the tip regarding v3.1.2. I've updated the default configuration and documentation for it.
I'm also closing this pull request as I've merged the Mijia sensor support to the main branch.

Good day