BLE Library - Add service data support

[julesDesjardin]

Hi !
I would like to use the BLE library to manage advertising data, but right now it's only possible to add service UUIDs. Would it be possible to implement service data as well ? I also have the same question for the scanning part (for this part, a direct access to the advertising data via a getAdvertisingData method on the AdvertisedDevice class would also be enough, and easier to implement I think)
Thanks !

[1technophile]

Hi,

Yes could be interesting to be able to read service data by an AdvertisedDevice, I think about examples like Mi flora who advertise sensors values on service data.

[chegewara]

Maybe im lack of knowledge and/or imagination, but how it suppose to looks like? It is implemented in Beacon already, but im guessing its not the same.

[1technophile]

Nop I don't think so : -) more a lack of explanation here is a screenshot of what i would like to be able to read with the library.
https://ibin.co/3lNX93QvWwn3.png

Currently I didn't found a method on BLEAdvertisedDevice.h to do that, maybe i m misding something.
Please note that i m using
https://github.com/nkolban/ESP32_BLE_Arduino

[chegewara]

I think it is implemented already. But it cant be done without connecting, its just implemented to do it out of box. But it is not added in library you are using. You can see this code:

esp32-snippets/cpp_utils/BLEDevice.cpp

Lines 197 to 205 in 3d39b22

	/* STATIC */ std::string BLEDevice::getValue(BLEAddress bdAddress, BLEUUID serviceUUID, BLEUUID characteristicUUID) {
	ESP_LOGD(LOG_TAG, ">> getValue: bdAddress: %s, serviceUUID: %s, characteristicUUID: %s", bdAddress.toString().c_str(), serviceUUID.toString().c_str(), characteristicUUID.toString().c_str());
	BLEClient *pClient = createClient();
	pClient->connect(bdAddress);
	std::string ret = pClient->getValue(serviceUUID, characteristicUUID);
	pClient->disconnect();
	ESP_LOGD(LOG_TAG, "<< getValue");
	return ret;
	} // getValue

[nkolban]

Howdy @julesDesjardin , @1technophile

Great questions and thanks for posting them. Lets see if we can't get this going for you. This is a technical discussion and I don't know how much or how little you know so please forgive me if it is too simple or too complicated ... let me know if I need to adjust.

I am assuming that we are thinking of the ESP32 in its role as a BLE Central (Client) where it is passively scanning/listening for arriving advertisement from BLE Peripherals (Servers) that are broadcasting/advertising.

When a BLE advert arrives at the ESP32, the advert contains up to 31 bytes of payload/advert data. On receipt of an advert, the ESP32 will parse its content and understand what arrived. Included in the advert is the Bluetooth Device Address (BDA) of the partner that sent the advert in the first place. The ESP32 can (should it choose and the partner allow) immediately send a message called a "Scan Request" which is basically "I just saw an advert from you, please send me one follow on message and tell me more" where a Scan Response message will be returned ... which is again just a second advertisement packet but containing more data.

Adverts being sent by BLE are not just arrays of bytes. An advert is composed of identified fields where a field has some set of allowable and understood values. The complete list of possible advertisement fields can be found here:

https://www.bluetooth.com/specifications/assigned-numbers/generic-access-profile

This so far is theory ...

In practice, the BLE C++ library found at this repository receives the incoming packet and decodes it to produce an object that is a logical representation of what was learned from the incoming advert. Since an advert may or may not contain certain fields ... it is 100% up to the advertiser as to what they include/exclude ... the resulting object will have some values filled in and others not set.

The object type we create is BLEAdvertisedDevice. An instance of this object can be asked:

1. Do you have a populated value for a given advertisement field?
2. What is the value for a populated received advertisement field?

So far we have provided support for fields of type:

• appearence
• manufacturer data
• name
• RSSI
• service UUID
• TX Power

However, as you will see in the BLE spec list, there are many more. We simply haven't implemented all of those because we don't know which ones folks need or want. If there is a field a device is advertising and the BLEAdvertisedDevice doesn't yet decode that field, simply let us know and we will go ahead and code it in as quickly as possible.

[chegewara]

http://www.cypress.com/blog/100-projects-100-days/project-008-gap-broadcaster

[1technophile]

@chegewara Thanks for the code extract I will try that and let you know if it answer my needs.

[julesDesjardin]

Hi,
Thanks @1technophile for your screenshot, that's exactly what I was talking about !
I managed to implement it in the library by adding methods to modify m_adv_data.p_service_data and m_adv_data.service_data_len in the BLEAdvertising. I can share it, it's pretty straightforward (only a few lines, it doesn't use the service UUID but instead directly the bytes that will be in the advertising packet) but may lack protections, and consistency with the other functions.
To be able to scan it I only added a getPayload method in the BLEAdvertisedDevice, and managed the parsing on the application side (actually I'm only using one 29 bytes long service data so I don't have to parse that much).
Anyway thank you very much @nkolban , this library works perfectly fine and it's much more easier to use an ESP32 than other bigger boards !

[chegewara]

Ok, now i see it and its easy to implement. Work in progress.

[1technophile]

@nkolban thanks for this detailled explanation !
@chegewara thanks for the PR I will give you some feedback soon.

[1technophile]

@nkolban @chegewara @julesDesjardin , I confirm that the modification made by @chegewara is working as expected and enables me to extract sensor values from a mi flora sensor (humidity, temperature, lux and fertilization).

I have now to industrialize the proof of concept and integrate it into OpenMQTTGateway BLE gateway.

Thanks a lot for the help!

[nkolban]

@1technophile ... About an hour ago I made some updates to the code made by @chegewara ... I haven't had a chance to chat with him about them yet ... however, they changed the exposed information from char* to std::string ... please be cautious as this is new code and the interfaces may change until we settle.

[chegewara]

BLE is like mine field, full of traps. But when you understand structure and data how they are send by peripherals we can provide tools to work with it. You guys gave us some tips, that screenshot was great, thanks for that and we were able to add functionality you are requesting to play with sensors.

[1technophile]

@nkolban no problem I will adapt according to your changes

[nkolban]

Careful ... they just changed again. Hopefully this will be the end. We studied and found that the service data advert has THREE variants. One that contains a 2 byte UUID, one a 4 byte UUID and one a 16 byte UUID. We were only handling the 2 byte UUID. We now handle all three. Secondly, we now see that every service data has the format:

[uuid] [data]

where the uuid is 2, 4 or 16 bytes. We now parse all of them and give you the following methods:

• bool haveServiceData() - True if the advert contains service data
• BLEUUID getServiceDataUUID() - returns the UUID in the service data
• std::string getServiceData() - returns the payload following the UUID in the service data advert

I am currently assuming that any given advert will contain only ONE service data field ... if you find more than 1, we will need to accommodate.

[1technophile]

Thanks for the std::string, easier to handle, here is what I get with some string transformation:

Starting Arduino BLE Client application...
BLE Advertised Device found: Name: Flower care, Address: c4:7c:8d:65:b6:63, serviceUUID: 0000fe95-0000-1000-8000-00805f9b34fb
Get service data 
71209800f363b6658d7cc40d041002d500
Get service data UUID
0000fe95-0000-1000-8000-00805f9b34fb

Thanks again for this work.

[nkolban]

Awesome ... is this what you had hoped to see? :-)

If you are satisfied with the resolution, close out the issue as desired. We can always re-open if needed or if there are new issues, don't hesitate to create new ones.

[1technophile]

Yes that's great

[1technophile]

@julesDesjardin if ok for you, could you close the issue, i don't have access to the close button

[joey1442]

Hi
I have a question which related to the BLEAdvertisedDevice object in the discussion.
While the object can return values on TX power, I am a little confused as surely the client determines the RSSI using the BLEAdvertisedDevice_TXPower and this can be done without connecting.
Even without the TXPower value a default value would be used.
Am I misunderstanding the use of BLEAdvertisedDevice by a client listening to an unconnected advertiser ? Is the RSSI only available after the client has connected to a connectable advertiser ?
BTW I am trying get RSSI from BLEAdvertisedDevice in an unconnected scan, the Arduino BLE scan example does not appear to enable this.

[nkolban]

Sometimes these APIs expose data without a deeper understanding on the part of the authors. A BLE advertisement contains fields. There is a finite set of fields types defined. These are documented/described here.

https://www.bluetooth.com/specifications/assigned-numbers/generic-access-profile

The BLEAdvertisedDevice is a model that represents a received advertisement. It has knowledge of some of the fields types. The only reason that we haven't told the model about ALL field types is that we are lazy and we wait for someone to say "Hey, you don't decode XYZ field ..." and then we add it.

One of the possible field types described in the BLE spec is type 0x0A - TX Power Level. Look at that in the spec document referenced previously. We found that was a commonly received advert payload so we added decoding of it. However ... and this is the punch line ... just because the spec says that data is available and because we have decoded that data and made it available to the programmer, unfortunately does not imply that we know what all these fields mean. What I suggest is go and study the spec in depth, study the meaning of the TX Power Level field in an advert and, when done, paste back here what you find. It will help all of us now and in the future.

[shahabmusic]

Is it possible to extract temperature values from an Eddystone TLM data?

I have a beacon that advertises the temperature and battery level. I can see these data using android eddystone app and would like to see these data in esp32 in Arduino IDE too.

In one of the snippets I saw that you wait until you receive an Adv data type 0x16 and then you look for 0x1809 uuid and then you decode the temperature.

I tried to do the same thing in BLEAdvertisedDevice class, however the esp32 does not find 0x1809 uuid.

Am I in the right path to get temperature readings? could anyone help please?

[chegewara]

You can check what raw data are used to advertise, i like to use nRF connect, and read from it what service UUID is used in advertising packet. If you have issue to read it you can paste here raw advertising packet or screenshot and we will try to help. Screenshot prefered like this one
https://ibin.co/3lNX93QvWwn3.png

[nkolban]

@shahabmusic This looks like it would make a great issue in its own right. Might I encourage you to move your post to its own issue so that we can track it separately?

[shahabmusic]

cool. thanks. I just opened a new question.
here it is:
https://github.com/nkolban/esp32-snippets/issues/345

[julesDesjardin]

Hi,
Thank you very much ! Sorry for not answering earlier, I was in vacations.
Unfortunately I'm getting issues with esp_bt.h, apparently it's caused by the ESP-IDF framework, but the board I use only supports Arduino. I'm just gonna put the additional code for service data into the ES32_BLE_Arduino library. Any chance you can also update that library ?

[chegewara]

We have recently updated library to follow changes in esp-idf. They just renamed bt.h to esp_bt.h.
Quick solution for you to use latest cpp_utils with arduino is to create esp_bt.h file and add just one line to it:
#include <bt.h>
Sorry for inconvenience.
But i have to warn you that latest cpp_utils library has also one more incompatibility with arduino-esp32 which is c++ exceptions. This is harder to fix.

[julesDesjardin]

Thanks for the trick, it works !
About exceptions, for the moment I only commented everything about them, which gives incomplete code, but since I'm not gonna use the concerned functions it should work fine ! (that's quite an ugly solution, but I don't want to lose too much time on switching to esp-idf to solve a problem that doesn't apply to my program)