Handling advertising using manufacturer specific data

[KipK]

Hi,

Great piece of code so far, thanks for that project.

I was wondering if we could consider adding a generic way to handle manufacturer specific data ( 0xFF ) embedded in advertising frames. Those are broadcastes too.

https://www.silabs.com/community/wireless/bluetooth/knowledge-base.entry.html/2017/11/14/bluetooth_advertisin-zCHh

The idea would be to have it configurable with something like this:
example packet:

[02 01 06 02 0A 00]  [08]     [FF]         [0D 00] [ 01 CD 00 34 32 ]
                     length  manuf data     company id      Data

"ble": {	
	"broadcaster_manufacturer_data": [
            {
            "name": "MySwitch_Temp_Sensor", //broadcasted name
            "whitelist": ["10:CE:A9:E3:A4:E6"],     // whitelist/blacklist array
             "data": [                                           // variables positions in the data field
                             { "name": "actuator", 
                               "from" : 0, //starting byte
                               "to":  1,      //ending byte
                               "type": "bool",
                               "endian": "little"
                              },
                             { "name": "temperature", 
                               "from" : 2, //starting byte
                               "to":  5,      //ending byte
                               "type": "hex",
                               "endian": "little"
                              },
                             { "name": "humidity",
                               "from" : 6,
                                "to": 9,
                                "type": "ascii",
                                "endian": "big"
                              }
                        ]
                  }
              ]
           }

In the example we receive this data 0x01CD003432 in the manufacturer specific data field, would mean:
. actuator is On ( 01 )
. temperature = 20.5 ( 00 CD ) // Hex
. humidity = 42% ( 34 32 ) // in ASCII
type could be bool, int, hex, ascii
Best

[shmuelzon]

Hey,

Thanks!

I'm all for this kind of thing. The problem is that I've yet to come up with a robust solution.
First off, how would you know which configuration to use for the device that you received its advertisement? I'm pretty sure that they must all start with the company ID, don't know if this is really the case. Does it exist in the device you want to use?

Even if we do use the company ID to differentiate between the different configurations, would that be enough? What if the same manufacturer has multiple devices and each has a different format?
You would need to define some filtering logic to the configuration.

Then, supposed you figure out all of that, there's still the part of parsing the data. If it's standard BLE data types, then you're golden, but what about MAC addresses, UUIDs, URLs? Some broadcasters even use different endianness compared to the BLE standard.

I've tried to make it simple to add support for different broadcasters in the code. If they will start to pile on, perhaps I'll separate them to different files just to keep things clean. This is the best solution I have thus far but welcome any idea that might bring us closer to a generic solution.

If you want to add support to your device, please send in a pull request, or give me some more info on what it is and what it publishes and I can code it.

Good day!

[robertcsakany]

+1
I like the idea, because I had to put back my Advertiser code on - I have JPY-10 chips also to make some heartbeat function - detect ON/OFF on lights. And it transmits standard broadcast message without any specific data and with the default implementation there is no way to get the Advertise message only.

robertcsakany@7010590

[KipK]

@shmuelzon
I've edited my first message since, check the configuration example.
Using the device name of the standard advertising message sent should be sufficient to differentiate. If not, the whitelist/blacklist should filter the thing out.
If the device is handled natively by esp32-mqtt ( like xiaomi boradcaster ), then use it, if not then check the generic way.
Or we could move all those one to this new system if it could manage those.
The idea is let people adding their own devices easily without involving you to code them and maintain a big device database. This would help with DIY sensors/actuator too.

By looking at different devices that use manufacturer data field, some send different kind of data in different frame and use & byte identifier before each stored value.

But some don't.
So perhaps I'd change the configuration property by something like this to be able to handle booth:

{ "name": "temperature", 
                               "from" : 2, //starting byte
                               "to":  5,      //ending byte
                               "starting_byte": "0C", // previous byte must be '0x0C', 
                               "type": "hex",
                               "endian": "little"
                              },

Also some seems to mess up the little endian standard so I've added it too. Should default to little.

Thanks for your proposal in adding some device. I'll try to do it myself first and submit you a pull. That will help me understand better how those things works, I'm still new to ble.

[KipK]

If it's standard BLE data types, then you're golden,

I thought manufacturer data was a BLE datatype (0xFF) where you can fit whatever,.is it not?

but what about MAC addresses, UUIDs, URLs?

Are they not just char ascii arrays?

{ "name": "url",
"from" : 0,
"to": 17,
"type": "ascii",
"endian": "big"
}

as example for : FF 0D 00 68 74 74 70 3A 2F 2F 77 77 77 2E 75 72 6C 2E 63 6F 6D
"http://www.url.com"

[shmuelzon]

The manufacturer can put whatever he wants as the payload. It should start with the company ID but after that he's free to put anything.

MAC addresses and UUIDs are usually raw bytes as it consumes a lot less space. As for URLs, Eddystone beacons, for example, have their own convoluted format that mixes both "regular" strings and special shortcut values. Because of theses kinds of values, I find a single generic mechanism is problematic...

[KipK]

Here is the pull request for the Beewi sensor: #22

[ojousima]

Hello,

I'd like to chime in as I happen to work on beacons broadcasting sensor data in manufacturer specific format. We've been looking for solutions like this and even consider rolling our own.

Our approach would be to publish the raw payload to MQTT topic and let the subscriber take care of parsing. In some cases the payload might even be encrypted so parsing it on ESP32 would not be possible.

Is there some way I could contribute to this issue?

[robertcsakany]

@ojousima
There was a pull request about it #8
But in my repo there is an uopdated version which is compatible with the actual master:

https://github.com/robertcsakany/esp32-ble2mqtt/tree/Advertisement

[shmuelzon]

I still have my reservations against publishing raw Bluetooth specific data. The whole point of this project is to handle the BLE specifics and let another entity on the MQTT bus to handle the application level.

May I suggest a compromise? How about implementing a ‘generic’ broadcaster? One the would be at the end of the broadcasters list so it will have the lowest priority. It would only look for non-connectable devices (so it won’t send all advertisements) and only publish the raw data of the manufacturer specific section, if it exists.

Would that suit your use-cases?

Since we can’t filter by MAC address, as the devices could (and probably should) randomize their address, I might make each broadcaster type optional as a build configuration. That would be a separate commit, though.

[ojousima]

@robertcsakany Thanks, I'll check it out later this week
@shmuelzon

May I suggest a compromise? How about implementing a ‘generic’ broadcaster? One the would be at the end of the broadcasters list so it will have the lowest priority. It would only look for non-connectable devices (so it won’t send all advertisements) and only publish the raw data of the manufacturer specific section, if it exists.

Would that suit your use-cases?

It would suit my use-case perfectly

[shmuelzon]

It would suit my use-case perfectly

@robertcsakany, @KipK
How about you?

[KipK]

Should work yes. I'd like to add a generic parser function we could edit ourselves . Also some broadcasters are connectable ( like the Beewi ).

[robertcsakany]

@shmuelzon Yes, it can work for me too. But. there are devices which make broadcast and there is a way to connect too, but only broadcast massages I care. Xiami Mijjia is also that type of device for example, I had to blacklist the normal service discovery to save the battery - which is also okay for me. So if Other low priority broadcaster can work thi�s way Its okay for me.

[ghangaskan]

@robertcsakany

Xiami Mijjia is also that type of device for example, I had to blacklist the normal service discovery to save the battery - which is also okay for me. So if Other low priority broadcaster can work thi�s way Its okay for me.

I picked up an 8 pack of these for monitroing aroud the house. Can you share your config so I can see how to blacklist the normal service discovery?

[shmuelzon]

I think he meant that he blacklisted the devices themselves so this application won't try to connect to them and only listen for broadcasts. That would be something like:

{
  "ble": {
    "blacklist": [
      "00:11:22:33:44:55",
      "11:22:33:44:55:66"
    ]
  }
}

You'll need to add to the list all of the MAC addresses of your devices.
The above will try to connect to anything else, so you may prefer to use an empty whitelist instead:

{
  "ble": {
    "whitelist": []
  }
}