Question about non-blocking NimBLEScan::start() and continuous scan

[mbariola]

Hi,

for an application, I need to always be monitoring a set of BLE devices (identified by their MACs), which move around the ESP32 running the NimBLE instance. The ESP32 is stationary.

I would need to figure out in the most real time possible, when one of such devices is close enough to the ESP32, and I thought that RSSI is a good enough approximation. The ESP32 also does other tasks in the mean time, so I thought that scanning in a non blocking way would be the way to go. But I find it difficult to debug, i.e. I see discontinuity or delays in RSSI updates, and I am not sure, even with active scan on, if I am doing it right.

What is the intended behavior of NimBLE in this case, ie. continuous nonblocking scan? when does it call the callback for a specific MAC/device?
Once a while per some throttle timer? as many times as it detects it? as often as the device advertises itself?

Many thanks to anyone who can shed some light on this / can suggest the right way to achieve my needs. As to why Real time is needed, it's a toy for kids of a friend, some kind of Red Light / Green light game. They have a mifit band each and the one which gets within a certain radius of the esp32 scores one point :-)

[h2zero]

Hi @mbariola, the continuous scan is intended to simply keep scanning and provide data through the callback provided. There is an internal cache that is configurable and that will filter recently seen devices until the cache is full, at which time it is cleared and the previous devices will be reported again.

For your purposes I suggest not using active scan. That has the effect of delaying scan reports until scan responses are recieved, unless there is data you're interested in from those scan responses of course.

[mbariola]

Hi @h2zero , thanks for the answer. Yes you're correct, I just need to "see" those devices and have their RSSI values with a temporal resolution as needed.
So I seem to understand that I should then

first of all filter for the needed MACs possibly even before calling the callback (if there's something like a MAC whitelist)
disable such cache to get the "most real time experience", or clear it every tenth of second or so (could you point me to the calls and configs to do so please? this is what I surmise I should do:

NimBLEDevice::whiteListAdd(one call per MAC address of the devices I need to track)
NimBLEScan::setFilterPolicy(BLE_HCI_SCAN_FILT_USE_WL),
call NimBLEScan::clearDuplicateCache() or maybe NimBLEScan::clearResults() periodically (I could not find how to disable the cache itself. I surmise that as long as the callback is called per each new detection, )
disable active caching, I suppose that RSSI levels will still be available
let NimBLEScan::start(0, myCallback, FALSE or TRUE??) run, and I guess that no code will be executed after this line so I'll have to move everything else to the other core
Did I get it all right? thanks

Ah, an additional couple question. I guess that the scan is reading the advertisement of the two bands, so it's also a matter of how frequently those bands advertise and not much can be done about it. I guess also for battery saving reasons. by using a companion app, I could see the bands advertise once a second.
Would the "realtimeness" of it somehow changed if I were to pair the esp32 with such bands and "interact with them" somehow? maybe asking for their service characteristics? thinking about the "realtimeness" of ble audio and ble controls from such bands (ie controlling the paired smartphone). This is more about the BLE protocol, I know; but it's quite a jumble of conflicting google results. Maybe you have some empirical data or knowledge about it? thanks again

[mbariola]

This is the best I could manage, using your whitelist client sketch as basis.
I can see the band I am using to test around once a second, consistent with the advertising interval of the band itself.
Any suggestions on how to improve it are very appreciated. Also, is there a way somewhere to obtain / read the BLE devices' Measured power aka RSSI at 1mt ?

/*
 * NimBLE_Scan_Whitelist demo
 *
 * Created May 16, 2021
 * Author: h2zero
 */
#include <Arduino.h>
#include <NimBLEDevice.h>
#include <PrintEx.h>
StreamEx mySerial = Serial;

NimBLEScan *pBLEScan;
NimBLEAddress a("e8:f5:7d:2f:ab:b4");

    class MyAdvertisedDeviceCallbacks : public BLEAdvertisedDeviceCallbacks
{
    void onResult(BLEAdvertisedDevice *advertisedDevice)
    {
        mySerial.printf("Found %s with RSSI %d \n", advertisedDevice->getAddress().toString().c_str(), advertisedDevice->getRSSI());
        pBLEScan->clearResults();
    }
};

void setup()
{
    Serial.begin(460800);
    NimBLEDevice::init("");
    NimBLEDevice::whiteListAdd(a);
    Serial.println("Scanning...");
    pBLEScan = NimBLEDevice::getScan();
    pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks());
    pBLEScan->setFilterPolicy(BLE_HCI_SCAN_FILT_USE_WL);
    pBLEScan->setMaxResults(0); // do not store the scan results, use callback only.
}

void loop()
{
    pBLEScan->start(0, false);
    Serial.println("This should never be reached");
    delay(2000);
}

[h2zero]

This is a good start, there are a few things to incorporate to make this more robust though.

Have a look at https://github.com/h2zero/NimBLE-Arduino/blob/master/examples/NimBLE_Scan_Continuous/NimBLE_Scan_Continuous.ino
For your use, having the call to NimBLEDevice::setScanDuplicateCacheSize(10); would seem most appropriate.

In the callback you should not call pBLEScan->clearResults(); as it could cause an exception, instead use the setting pBLEScan->setMaxResults(0); to not store the results as shown the that example.

In the callback you can retrieve the RSSI from the advertised device using advertisedDevice->getRSSI()

[mbariola]

Hi @h2zero ,

Thanks for the suggestions, I'll incorporate them right now and report. One thing about the setMaxResults(0) - I already had that as you can see in the sketch, but the detection of it did not seem to work properly. I only received one detection then it went silent. Adding the clearResults() restored the once-per-second (I guess once-per-advertisement)I reporting of the whitelisted band. I did not try in combination of the cache size reduction (will do so now), but could you please explain better what constitutes a "duplicate" and the whole process from detection to putting it in the results list to caching? thanks

Will offer the findings with the modified sketch in a little while

[mbariola]

Hi @h2zero , here's the results of the new sketch including the setScanDuplicateCacheSize(10) and the setMaxResults(0) statements

• If I comment out the clearResults() in the callback, I only get one single reading and nothing else.
• If I leave it commented, and set the duplicate cache size to 10, or 0, I still only get one single reading
• If I uncomment it, I get the once-per-second reading.
  So apparently either I am making some mistake / don't understand something, or I need to clear or remove the specific scan from the results set. The idea of a possible exception is quite scary though! Maybe you can run this sketch with one device, and see if you have different results than mine

A heads up: NimBLEDevice::setScanFilterMode(....) does not appear in the doxygen documentation;

/*
 * NimBLE_Scan_Whitelist demo
 *
 * Created May 16, 2021
 * Author: h2zero
 */
#include <Arduino.h>
#include <NimBLEDevice.h>
#include <PrintEx.h>
StreamEx mySerial = Serial;

NimBLEScan *pBLEScan;
NimBLEAddress a("e8:f5:7d:2f:ab:b4");
uint lastAdv = millis();

    class MyAdvertisedDeviceCallbacks : public BLEAdvertisedDeviceCallbacks
{
    void onResult(BLEAdvertisedDevice *advertisedDevice)
    {
        //Serial.printf("Advertised Device: %s \n", advertisedDevice->toString().c_str());
        /*
         * Here we add the device scanned to the whitelist based on service data but any
         * advertised data can be used for your preffered data.
         */
               mySerial.printf("%d: Found %s with RSSI %d \n", millis() -lastAdv, advertisedDevice->getAddress().toString().c_str(), advertisedDevice->getRSSI());
               lastAdv = millis();
               //pBLEScan->clearResults(); // comment-out 1
    }
};

void setup()
{
    Serial.begin(460800);
    NimBLEDevice::setScanDuplicateCacheSize(10); //comment-out 2
    NimBLEDevice::init("");
    NimBLEDevice::whiteListAdd(a);
    Serial.println("Scanning...");
    pBLEScan = NimBLEDevice::getScan();
    pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks());
    pBLEScan->setFilterPolicy(BLE_HCI_SCAN_FILT_USE_WL);
    pBLEScan->setMaxResults(0); // do not store the scan results, use callback only.
}

void loop()
{
 if(pBLEScan->isScanning() == false)
 {
    pBLEScan->start(0, nullptr, false);
 }
 Serial.println("Looping...");
 delay(2000);
}

[h2zero]

Sorry for the delayed reply.

Some detail of how this works:
When a device advertises its sends advertisement packets at a specified rate for a specified time, the purpose of the caching is to reduce the amount of this "spam" to only trigger the callbacks at a reasonable rate. The device advertisement rate, can be anywhere from 20ms to 10240ms, so the cache setting has the effect that it will store up to X advertisements before erasing itself and if it gets an advertisement from a device already in the cache it will be dropped. This works well when there are many different devices advertising to control the callback rate. For your case this isn't working as expected due to the whitelist as there is only 1 device stored in the cache (sorry about that, I hadn't considered this factor) so the cache is not being filled/erased, thus dropping the advertisements.

To solve this we can simply turn off the duplicate filter, which can be done by adding an extra parameter to pBLEScan->setAdvertisedDeviceCallbacks to request duplicates.

Try changing:

pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks());

To:

pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks(), true); // Set wantDuplicates

[mbariola]

Hi @h2zero , thanks for the additional explanation. So let's say the cache is a bit like an anti-flood or debounce measure. I understand that there's a lot of layering to the workings of the NimBLE library, to accommodate for all possible usage scenarios.
I tried your modification, and it works! thanks so much. So, for everyone that may have a similar need as me, here's the whole of the working code, cleaned of commented-out lines and stuff. When this runs, I get an advertisement every second which is the (unmodifiable) advertising rate of the band.

#include <Arduino.h>
#include <NimBLEDevice.h>
#include <PrintEx.h> // just for some sprintf in the code, it's accessory
StreamEx mySerial = Serial;
NimBLEScan *pBLEScan;
NimBLEAddress a("e8:f5:7d:2f:ab:b4");
uint lastAdv = millis();

class MyAdvertisedDeviceCallbacks : public BLEAdvertisedDeviceCallbacks
{
    void onResult(BLEAdvertisedDevice *advertisedDevice)
    {
        mySerial.printf("%d: Found %s with RSSI %d \n", millis() -lastAdv, advertisedDevice->getAddress().toString().c_str(), advertisedDevice->getRSSI());
        lastAdv = millis();
    }
};

void setup()
{
    Serial.begin(460800);
    NimBLEDevice::init("");
    NimBLEDevice::whiteListAdd(a);
    Serial.println("Scanning...");
    pBLEScan = NimBLEDevice::getScan();
    pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks(), true);
    pBLEScan->setFilterPolicy(BLE_HCI_SCAN_FILT_USE_WL);
    pBLEScan->setMaxResults(0); // do not store the scan results, use callback only.
}

void loop()
{
    if(pBLEScan->isScanning() == false)
    {
        pBLEScan->start(0, nullptr, false);
    }
    Serial.println("Looping..."); // just a heartbeat & a delay to avoid piling up scan start calls 
    delay(2000);
}

Thanks again for walking me through troubleshooting this behavior!

Unfortunately I cannot mark your reply as "answer" as GH does not let me mark the reply as answer, so I'll mark the post containing the reply as such. Thanks again :-)

[h2zero]

Awesome! thanks for the confirmation!

[Zimbu98]

Wow, that did it! So simple. Thanks very much!

[Zimbu98]

Quick question: Your instruction fixed my problem, but I don't really know why. Could you point me to some web link that explains something about the meaning of "the address is a random type (2 MS bits are 1)"?

[h2zero]

No problem 😄. glad it's working. Here is a link with some great info about this https://novelbits.io/bluetooth-address-privacy-ble/

[Zimbu98]

I was googling like crazy for anything to do with "MS bits" but kept getting many results for some company called MSbit Software!

Your link really helps me. Thanks.

[h2zero]

Lol, yeah that wasn't worded well. That link has a lot of good BLE info if you ever need to look stuff up.

[Zimbu98]

Sorry to bother you again. I am stumped again.

I am trying to build a beacon scanner. The user will input the target MAC address, and the scanner sketch will do something when it "sees" the target beacon. I want the target MAC to be placed on the whitelist. OK, so far, so good. mbariola's sketch does that perfectly for public address MACs.

The problem I am having is that the user will not know if their beacon MAC is a public address or random static address. So if they input a random static MAC address such as dd:34:02:06:34:17, mbariola's code won't work because the MAC will be missing the beacon type indicator: , 1.

Searching for clues, I looked at your sample sketch "NimBLE_Scan_Whitelist demo" which is able to scan, see, and add a random static address to the whitelist, but I am not sure why it works there.

This works for random static address beacons in Whitelist Demo sketch:
NimBLEDevice::whiteListAdd(advertisedDevice->getAddress());

But this doesn't work in mbariola sketch:
NimBLEAddress a("dd:34:02:06:34:17");
...
NimBLEDevice::whiteListAdd(a);

Thanks for any suggestions!

[Zimbu98]

The only thing I can think of for this is to add the address to the white list twice, with both public and random address types, but this will limit the list to only 6 devices instead of 12.

Bad news: it won't let me add the same MAC address twice, even if the address types are different. So, this:

NimBLEAddress b1p("dd:33:0a:11:1a:a5", 0); 
NimBLEAddress b1r("dd:33:0a:11:1a:a5", 1);

NimBLEDevice::whiteListAdd(b1p);
NimBLEDevice::whiteListAdd(b1r);

Only adds one single MAC to the whitelist. In this case, it only adds the first one (b1p), and then it ignores the second whiteListAdd. If I check the whitelist count while the scanner is running, it is only 1.

It also doesn't work if I leave off the type code on the first line. For example:

NimBLEAddress b1p("dd:33:0a:11:1a:a5"); 
NimBLEAddress b1r("dd:33:0a:11:1a:a5", 1);

NimBLEDevice::whiteListAdd(b1p);
NimBLEDevice::whiteListAdd(b1r);

So I think I need to keep digging for the bug that is causing the scanner to misjudge the type code. Then your code snippet suggestion should work fine.

[Zimbu98]

My hacky solution:

Commented out these lines in NimBLEDevice.cpp

/**
    if (NimBLEDevice::onWhiteList(address)) {
        return true;
    }
    */

Now it allows me to put two versions of each beacon on the whitelist, one version with type code 1 and another version with type code 0 (or no type code and just let it use the default).

Victory!!!!

[h2zero]

Lol, glad you found that, the comparison code needs some work in this regard to take the type into account, I'm just reluctant to as it will likely break some peoples code.

I did a dive back into the core spec to see what it is that was causing the issue and was reminded of the header in the BLE advertisements. The flag that indicates if the address is public or random resides in this header which is then provided as the address type to the scanner from the controller when a device is detected.

I found a link that describes this more simply than the core document if you're interested: https://medium.com/rtone-iot-security/a-brief-overview-of-bluetooth-low-energy-79be06eab4df

So the code snippet above is invalid anyway for this use, it would only work to confirm the random status of the MAC.

[Zimbu98]

If I understand that link correctly, it is saying that the type code for the beacon's MAC is configured in the beacon's broadcast packet.

So even though you and I (and your code snippet) think that dd:33:0a:11:18:89 is a random static MAC type 1, the beacon's broadcast is incorrectly carrying a TxAddr of 0, i.e. a public MAC type code.

Based on that, it makes sense that when I scan and see this beacon, and print the type code, it shows as 0, since the scanner is probably just getting this type 0 info from the beacon's own broadcast packet. (I had incorrectly assumed that the scanner was doing some incorrect analysis of the MAC to figure out the type code).

I will take a look at the raw broadcast from the beacon to confirm this, but I think it is very likely to be true.

Thanks again for all your help. This has been a fascinating learning experience for me!

[h2zero]

I wouldn't say that it is broadcasting and incorrect MAC type, it is certainly likely correct for it, even though the bits are set (just means it will never change). The trouble being, for your use case, is there is no automatic way to detect this from the data provided. This is always the issue with hard coding BLE addresses into projects, the type is vital.