Provide raw data settings for advertising and scan response

[chegewara]

BLE ibeacon requires to advertise raw data and such it requires to add this functionality(optionaly) to resolve issue #195.

[nkolban]

Howdy, can you clarify some of the wording. What do we mean by "raw data"? Is this a suggested functional enhancement? What would the new/changed API look like? How would it be used by a BLE programmer?

[chegewara]

At the moment advertising data is set "under hood". We can set few parameters with functions like setTXPower, addService, etc. But esp ble stack has capability to set raw value to be advertised with http://esp-idf.readthedocs.io/en/latest/api-reference/bluetooth/esp_gap_ble.html?highlight=config_adv_data_raw#_CPPv231esp_ble_gap_config_adv_data_rawP7uint8_t8uint32_t. In there we dont have known values like name, UUID, TX power etc. just bunch of bytes set by user. More can be found here: https://github.com/espressif/esp-idf/blob/master/examples/bluetooth/gatt_server/main/gatts_demo.c and here https://github.com/espressif/esp-idf/tree/master/examples/bluetooth/ble_ibeacon.

I see it in form one or 2 additional functions:
setAdvertisingRawData(uint8_t *data)
pAdvertising->start(false)
With true we will advertise with standard structure and false for raw data. Parameter is optional to backward compatibility.

[nkolban]

A suggestion, instead of overloading start with a true/false flag ... (and think about that for a second, a casual reader wouldn't have a good "feel" for what advertising.start(true) vs advertising.start(false) means, what about a design that had:

setAdvertisingRawData(advertisingData)
and
setAdvertisingRawData(nullptr)

the first would set the advertising data to be the raw data encapsulated in an instance of BLEAdvertisingData while the second call (the one that is passed a nullptr) would clear the raw advertising data resulting in "default" advertising.

[chegewara]

Its not bad idea. But only one type of data can be advertised, raw data or default and user need to make choice, or it can be done by checking if rawAdvertisingData != nullptr

[nkolban]

As always, I could be horribly wrong on this. It is my understanding that BLE advertised data must be well formed and contains "records" of a finite number of types as described in the BLE specification (Core Specification Supplement - Part A). Each "Record" is composed of:

• 1 byte length
• 1 byte data type
• length - 1 bytes of data

the records are then concatenated together in memory.

The distinct types of records can also be found here:

https://www.bluetooth.com/specifications/assigned-numbers/generic-access-profile

What this means is that we can create an "advertisement" using one or more of these codes plus data. However, we can't just "make up our own" ... which means that there is a finite number of them. Rather than have users compose the records themselves, we can create a class that is a BLEAdvertisingData with methods corresponding to each one of these types.

For example, we see type 0x24 is a browser/web URI ... we could exposed a method on BLEAdvertisingData of the form:

setURI(std::string uri)

The order in which the "setters" are called would be the order that they are exposed in the raw data. I'm not seeing any obvious reason why the BLE programmer needs to compose their own binary blob of raw data to send when we can increase the abstraction by one level and provide setters for this finite number of architected advertising types. It looks like there are 62 distinct types. We could define the most common ones and add the others on demand as and when users request them.

For example:

BLEAdvertisingData advertisingData;
advertisingData.setURL("http://www.kolban.com");
advertisingData.setAppearance(0x03);
advertisingData.setTXPowerLevel(-12);
BLEAdvertiser *pAdvertiser = myBLEServer->getAdvertiser();
pAdvertiser->setAdvertisingData(advertisingData);
pAdvertiser->start();

[chegewara]

You are corect. Advertising is has finite number of records just like you described. In our case its just 2 records. One is flags, with legth 2 bytes and in most cases value 0x06 and the other record is raw data, with legth n bytes 1 to 28 bytes ( we cant exced 31 bytes in total). Raw data is formed by user.
Apple iBeacon has standard structure of records https://en.wikipedia.org/wiki/IBeacon, but it is not the only possible type of beacons. But it may be implemented in this form. This is another example:
https://os.mbed.com/blog/entry/BLE-Beacons-URIBeacon-AltBeacons-iBeacon/

Edit:
Maybe you can do raw data in form of union with structures for few well know beacon types? Just a thought

[ugurerkan]

Is there any way to implement iBeacon yet?
I need to create iBeacon with ESP32 😞

[nkolban]

Work has started on raw advertising data and scan response settings.

Here is an example of the new API:

BLEAdvertising* pAdvertising = pServer->getAdvertising();
BLEAdvertisementData advertisementData;
advertisementData.setName("MyName");
advertisementData.setFlags(ESP_BLE_ADV_FLAG_LIMIT_DISC | ESP_BLE_ADV_FLAG_GEN_DISC | ESP_BLE_ADV_FLAG_BREDR_NOT_SPT);
advertisementData.setPartialServices(BLEUUID("4fafc200-1fb5-459e-8fcc-c5c9c331914b"));
pAdvertising->setAdvertisementData(advertisementData);
pAdvertising->setScanResponseData(advertisementData);
pAdvertising->start();

At a high level, we have a new class called BLEAdvertisementData. Against an instance of this object we can set the possible records in an advert or scan response. Once done, we can associate the object with advertisement data or the scan response data.

Only a couple of the possible records are supported. These currently are:

• ESP_BLE_AD_TYPE_16SRV_CMPL
• ESP_BLE_AD_TYPE_32SRV_CMPL
• ESP_BLE_AD_TYPE_128SRV_CMPL
• ESP_BLE_AD_TYPE_FLAG
• ESP_BLE_AD_TYPE_16SRV_PART
• ESP_BLE_AD_TYPE_32SRV_PART
• ESP_BLE_AD_TYPE_128SRV_PART
• ESP_BLE_AD_TYPE_NAME_CMPL
• ESP_BLE_AD_TYPE_NAME_SHORT

Others will be added on demand or as time permits.

[nkolban]

Added 0xFF - setManufacturerData(std::string)

[nkolban]

Added 0x19 - setAppearance(uint16_t)

[nkolban]

In principle, we now have exposed all the APIs needed to become an iBeacon. At a high level the logic would be:

BLEAdvertising* pAdvertising = pServer->getAdvertising();
BLEAdvertisementData advertisementData;
advertisementData.setFlags(ESP_BLE_ADV_FLAG_LIMIT_DISC | ESP_BLE_ADV_FLAG_GEN_DISC | ESP_BLE_ADV_FLAG_BREDR_NOT_SPT);
// Build the iBeacon data structure and store it in std::string beaconData
advertisementData.setManufacturerData(beaconData);
pAdvertising->setAdvertisementData(advertisementData);
pAdvertising->setScanResponseData(advertisementData);
pAdvertising->start();

see:
https://en.wikipedia.org/wiki/IBeacon

[torntrousers]

Could someone give an example of some valid contents for what could go in std::string beaconData?

[nkolban]

See also #271

[chegewara]

Try this, im not 100% sure, but it should be good:
0x0201061AFF4C0002150FDA50693A4E24FB1AFCFC6EB0764782512345678C5

[torntrousers]

Any tips on a concise way of getting 0x0201061AFF004C15020xFDA50693A4E24FB1AFCFC6EB0764782512345678C5 into std::string beaconData?

[nkolban]

By including BLEAdvertising.h you will now find you have a new class called BLEBeacon(). This is a representation of an BLE beacon advert.

Example:

BLEBeacon myBeacon;
myBeacon.setManufacturerId(...);
myBeacon.setMajor(...);
myBeacon.setMinor(...);
myBeacon.setProximityUUID(...);
myBeacon.setSignalPower(...);
BLEAdvertisementData advertisementData;
advertisementData.setFlags(ESP_BLE_ADV_FLAG_LIMIT_DISC | ESP_BLE_ADV_FLAG_GEN_DISC | ESP_BLE_ADV_FLAG_BREDR_NOT_SPT);
// Build the iBeacon data structure and store it in std::string beaconData
advertisementData.setManufacturerData(myBeacon.getData());
...

[torntrousers]

Thanks for that, I have it running but a beacon scanner app on my phone doesn't detect it. A BLE scanner app does see it though. Has anyone had it working as an ibeacon and willing to share their code?

[torntrousers]

Trying out some different ibeacon scanner apps on my phone, another one (iBeaconDetector) gives some more details. It can see the ESP32 but says its not an iBeacon. It dumps out a packet of something that looks like:
0201070201070000000000000000000000000000000
0000000000000000000000000000000000000000000
00000000000000000000000000000000000000

[nkolban]

By any chance are you running Android on your phone? If so, can you list the apps you are testing with? We can try and recreate/test. I'm afraid I personally have no Apple products to test with ... and have not actually tried to test Beacon technology ... all Ive been running with has been theory.

[torntrousers]

It works now! Change to use the BLEBeacon class instead of the raw bytes and the phone apps now identify it as an iBeacon. Terrific, thanks!

Something isn't perfect yet though as the fields don't match what the app sees.

I am using Android, trying several apps but this one seems ok - https://play.google.com/store/apps/details?id=de.flurp.beaconscanner.app

The code I'm trying with is here: https://github.com/HarringayMakerSpace/ESP32iBeacon/blob/master/esp32iBeacon/esp32iBeacon.ino

With the values in that code for uuid/major/minor the phone app shows different values:
UUID: 4c2a280a4126
Major: 1280
Minor: 22528

[torntrousers]

This app still says its not an iBeacon.

Also the previously mentioned Beacon Scanner app is actually showing the UUID set in the sketch but the bytes are reversed, in the sketch the uuid is:
26410a28-2a4c-4d57-ae2f-67a4d96b1f83
The Beacon Scanner app shows:
831f6bd9-a467-2fae-574d-4c2a280a4126
I wonder if something similar is happening to the major/minor values? The setSignalPower value from the sketch seems to get picked up correctly by the app.

[srlevitt]

Is it possible to send a beacon "on demand"? i.e. when an input pin changes, send a short beacon containing the state of the pin?

[chegewara]

Yes. You can start and stop advertising whenever you want/need. If you learn more about esp32 you will see that you can put esp32 into deep sleep mode and wake it up from pin and advertise that data for some time and get back to deep sleep to conserve battery.

[srlevitt]

If I keep the esp32 awake and running, what is the minimum time possible between the pin change and the beacon actually going out? Would I have to stop & restart the advertising? How soon after advertising start does the first beacon actually get sent?

[chegewara]

Well, you are asking very hard questions i dont know answer. I only can answer on one question:

• yes, you have to stop and restart advertising when you want to update advertised data.

Other questions are hard to answer because it vary and depends on your code. If you turn off debugging then it will be faster, i can only guess it can be between 1-3s when you can receive advertising packet after wake up(just guessing).

[evgenyl-github]

The function setManufacturerData(std::string) has parameter std::string. I need to advertise any binary array like: {0x03, 0x00, 0x00,0x4C}. Is it possible? With std::string only 0x03 will be advertised.

[chegewara]

I think the only option is to modify library.

Few weeks ago i did it myself or i changed advertised device to read manufacturer data (dont remember now).

https://stackoverflow.com/questions/2845769/can-a-stdstring-contain-embedded-nulls

You can always check with getPayload().length() how many bytes has been set in advertisement data object (BLEAdvertisementData).

[evgenyl-github]

I think the library modification will be very useful. The alternative way is to send as 2 ASCII characters for each byte but this doubles payload! чт, 17 окт. 2019 г. в 00:19, chegewara <notifications@github.com>:

…

I think the only option is to modify library. Few weeks ago i did it myself or i changed advertised device to read manufacturer data (dont remember now). https://stackoverflow.com/questions/2845769/can-a-stdstring-contain-embedded-nulls You can always check with getPayload().length() how many bytes has been set in advertisement data object (BLEAdvertisementData). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#196?email_source=notifications&email_token=ANQIAB33WLETMRU3LMNZKRLQO6APLA5CNFSM4EDRILZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBN7BLY#issuecomment-542896303>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANQIABZIWEZDL3TGFCTWTT3QO6APLANCNFSM4EDRILZA> .

[DrKamp]

Hello,
Sorry for the delay, but I wanted to suggest an idea that I implemented in the BLEadvertising.ccp files.
The idea was for me to be able to change the data of my BLE ESP32 on the fly (even the name).
It was not possible with the existing functions since it was only adding data payload.append(data).
Therfore, I created a very simple function :

void BLEAdvertisementData::clearAdvertisementData() {
	m_payload="";
} // getPayload

This way, I'm able in my loop() function to recover the data, and "recreate" it as much as I want

BLEAdvertising *pAdvertising = BLEDevice::getAdvertising();
   advData.clearAdvertisementData();
   advData.setFlags(0x06);
   advData.setServiceData(uint16_t(0xFE95), custom_payload);   // 95fe 363833343735303635303135313231
  
  // Set advertisement data
   pAdvertising->setAdvertisementData(advData);
   pAdvertising->start();

Maybe something you could look at implementing ? (if it does not create to many complications)