ESP32 crash with ESP_ERR_NO_MEM in NimBLEDevice

[justoke]

I have found this error which is related to #167

The NimBLEDevice code

The consequence of this error aside from the ESP32 crash is that some part of the flash memory seems to be corrupted. I am storing JSON data in the SPIFFS and often when this crash happens the data is corrupted and I have to persist it again.

[h2zero]

Thanks for reporting this, I believe you will find the cause of the no mem error to be that ARDUINO_ARCH_ESP32 is not defined for some reason. This is rather annoying to need, but without it the arduino startup code releases all the BLE memory.

I don't see how that error could affect the SPIFFS data though, that's quite odd.

[justoke]

I agree that affect on SPIFFS seems unlikely, its just that when these crashes occur this seems to be a symptom and its exact cause might be somewhere else that is not coming up in the logs. Overall this and the other issue are all around supporting BLE and WiFi concurrently. I can't say just yet if there is something inherently wrong with the device or the libraries involved. I have managed though to greatly improve stability by managing tasks using a state machine.

[h2zero]

Overall this and the other issue are all around supporting BLE and WiFi concurrently

I feel your pain lol. Looking through the IDF commit logs regarding the ble-wifi coexist stuff it's a never ending stream of patching, so they are having difficulties with this too.

I have managed though to greatly improve stability by managing tasks using a state machine.

This is what I do as well, wifi, ble, mqtt, webserver, etc all in separate tasks with a few semaphores spread around, then just trigger each task as it's needed. Much more manageable as it takes away some of the randomness.

[justoke]

This is what I do as well, wifi, ble, mqtt, webserver, etc all in separate tasks with a few semaphores spread around, then just trigger each task as it's needed. Much more manageable as it takes away some of the randomness.

The biggest stability improvement I got was to put BLE task on Core 0 and ESPAsyncWebServer on Core 1 - whether this really matters I'm not sure but I have one controller running 21 hours without a crash. I have another controller based on same state machine but with a couple of additional tasks. It's better but still crashes with the same errors. I'm still entertaining the idea of adding a second ESP32 to my PCB and having a dedicated BLE scanner to deliver my solution.

[h2zero]

The biggest stability improvement I got was to put BLE task on Core 0

Yes, that does help. In fact I just added some documentation recommending this as helps with more than just your issue as well.

Unfortunately I have not had time to look into the async webserver + BLE issue lately as I've been trying to push a new release.
Hopefully there is a solution that won't involve needing 2 esp32s for your use case.

[doudar]

I know you don't want to hear it (as it's a huge change for the same functionality) but I'm 100% stable (many units with weeks of uptime) using just the regular webserver + client side javascript for parsing and pulling data. I'm not scanning constantly but I've never had an issue during scans with this setup. I'm not running the webserver lightly either as I'm using it to stream debugging output to the client (webserver does slow when scanning).

I spent many weeks this summer trying to get ESPAsync to play nice with bluetooth and it was nothing but a steady +2 bugs for every 1 fixed.

[justoke]

I know you don't want to hear it (as it's a huge change for the same functionality) but I'm 100% stable (many units with weeks of uptime) using just the regular web server + client side javascript for parsing and pulling data.

Actually I'm glad to hear your success with the standard web server. And that you can confirm that BLE and ESPAsyncWebServer is not stable at present. I'd switch in a minute, but I depend on a framework that would take a lot of work to migrate. It's definitely on the cards though.

[h2zero]

Doing some reading and found me-no-dev/ESPAsyncWebServer#876

Maybe modifying async as described in that thread would work?

[justoke]

Maybe modifying async as described in that thread would work?

Thank you! This looks very promising indeed - just tried the code changes to AsyncTCP.cpp - so far so good, I can see the response difference immediately. I'll leave it running overnight and see how it looks tomorrow.

[h2zero]

Let me know if that works out for you, would be great to give the author of that issue feedback as well.

[justoke]

I left a comment, performance is greatly improved by the suggested changes. Crashes though are not unfortunately:

[justoke]

I captured this in the logs:

[E][WiFiClient.cpp:392] write(): fail on fd 57, errno: 11, "No more processes"

[h2zero]

That's unfortunate, that backtrace looks much different than the ones discussed in #167. I'm no expert with the webserver/async stuff but are using SSL? If so, there are issues with that and async webserver that I have seen, also reports of SSL over MQTT and BLE concurrency problems. I don't have any other suggestions at the moment 😞

[justoke]

I've made some progress and resolved some of the issues around the stability. This issue actually occurs due to out of memory and I have noticed a pattern. I run the BLE scan as part of a state machine using FreeRTOS which gives each task a slot to run in. The BLE scan task runs for 5 seconds and is repeated in a 60 sec window or until a beacon is discovered. However what I notice is that after each scan, the heap is gradually getting consumed to the point where it get's to less than 35k and this is usually where the crash occurs.

Could this be a memory leak and is there something I could try to alleviate this somehow?

[h2zero]

I don't think there is a memory leak here. You are printing the free heap as it is scanning and you can see it is recovered eventually. I suspect over a longer period you would see all of the heap recovered but there would need to be enough time in the idle task to do garbage collection. The amount of change you see is related to the number of devices advertising and after clearing results there is a delay in free heap recovery.

[justoke]

Sorry I didn't make it clear, the heap never recovers and the next thing in the log is a crash. This is repeatable.

[doudar]

Any chance (to prove your point) you could stop scanning if the heap gets below 33000 but keep printing heap messages to see if the garbage collection @h2zero mentions starts clearing it up?

[justoke]

Good suggestion - I'll try that.

[h2zero]

I believe I have an answer for this. It's the scan results vector increasing in capacity as advertisers are found, The more advertisers found the bigger the capacity, but when it is cleared it does not reduce it's capacity back to 0.

I modified the BLE_Scan example to test:

void loop() {
  // put your main code here, to run repeatedly:
  Serial.printf("Free heap begin: %d\n", ESP.getFreeHeap());
  BLEScanResults foundDevices = pBLEScan->start(scanTime, false);
  Serial.print("Devices found: ");
  Serial.println(foundDevices.getCount());
  Serial.println("Scan done!");
  Serial.printf("Free heap done: %d\n", ESP.getFreeHeap());
  pBLEScan->clearResults();   // delete results fromBLEScan buffer to release memory
  Serial.printf("vector capacity before: %d\n", foundDevices.m_advertisedDevicesVector.capacity());
  foundDevices.m_advertisedDevicesVector.clear();
  foundDevices.m_advertisedDevicesVector.resize(0);
  Serial.printf("vector capacity after: %d\n", foundDevices.m_advertisedDevicesVector.capacity());
  delay(1000);
  Serial.printf("Free heap end: %d\n", ESP.getFreeHeap());
  delay(1000);
}

Here are the results:

08:53:16.712 -> Scanning...
08:53:17.014 -> Free heap begin: 266772
08:53:17.183 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:53:17.318 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:53:21.994 -> Devices found: 2
08:53:21.994 -> Scan done!
08:53:21.994 -> Free heap done: 266272
08:53:21.994 -> vector capacity before: 2
08:53:21.994 -> vector capacity after: 2
08:53:22.999 -> Free heap end: 266528
08:53:24.005 -> Free heap begin: 266548
08:53:24.174 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:53:24.242 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:53:29.000 -> Devices found: 2
08:53:29.000 -> Scan done!
08:53:29.000 -> Free heap done: 266272
08:53:29.000 -> vector capacity before: 2
08:53:29.000 -> vector capacity after: 2
08:53:30.005 -> Free heap end: 266528
08:53:31.017 -> Free heap begin: 266548
08:53:31.051 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:53:31.084 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:53:36.002 -> Devices found: 2
08:53:36.002 -> Scan done!
08:53:36.002 -> Free heap done: 266272
08:53:36.002 -> vector capacity before: 2
08:53:36.002 -> vector capacity after: 2
08:53:37.012 -> Free heap end: 266528
08:53:38.017 -> Free heap begin: 266548
08:53:38.050 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:53:38.117 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:53:38.252 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:53:43.031 -> Devices found: 3
08:53:43.031 -> Scan done!
08:53:43.031 -> Free heap done: 266052
08:53:43.031 -> vector capacity before: 3
08:53:43.031 -> vector capacity after: 3
08:53:44.008 -> Free heap end: 266512
08:53:45.019 -> Free heap begin: 266544
08:53:45.086 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:53:45.120 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:53:45.454 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:53:50.005 -> Devices found: 3
08:53:50.005 -> Scan done!
08:53:50.005 -> Free heap done: 266056
08:53:50.005 -> vector capacity before: 3
08:53:50.005 -> vector capacity after: 3
08:53:51.014 -> Free heap end: 266516
08:53:52.025 -> Free heap begin: 266544
08:53:52.059 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:53:52.093 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
08:53:52.229 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:53:52.263 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:53:57.030 -> Devices found: 4
08:53:57.030 -> Scan done!
08:53:57.030 -> Free heap done: 265856
08:53:57.030 -> vector capacity before: 4
08:53:57.030 -> vector capacity after: 4
08:53:58.030 -> Free heap end: 266512
08:53:59.014 -> Free heap begin: 266544
08:53:59.014 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:53:59.081 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:53:59.081 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
08:53:59.149 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:53:59.854 -> Advertised Device: Name: , Address: 7a:3e:1d:e7:32:57, manufacturer data: 4c00100540183d5cbf, txPower: 12 
08:54:04.020 -> Devices found: 5
08:54:04.020 -> Scan done!
08:54:04.020 -> Free heap done: 265636
08:54:04.020 -> vector capacity before: 5
08:54:04.020 -> vector capacity after: 5
08:54:05.002 -> Free heap end: 266480
08:54:06.014 -> Free heap begin: 266524
08:54:06.047 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:54:06.047 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
08:54:06.047 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
08:54:06.114 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:54:06.315 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:08.298 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
08:54:11.029 -> Devices found: 6
08:54:11.029 -> Scan done!
08:54:11.029 -> Free heap done: 265464
08:54:11.029 -> vector capacity before: 6
08:54:11.029 -> vector capacity after: 6
08:54:12.038 -> Free heap end: 266480
08:54:13.014 -> Free heap begin: 266524
08:54:13.048 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
08:54:13.082 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
08:54:13.082 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:54:13.185 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:54:14.261 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:18.033 -> Devices found: 5
08:54:18.033 -> Scan done!
08:54:18.033 -> Free heap done: 265636
08:54:18.033 -> vector capacity before: 5
08:54:18.033 -> vector capacity after: 5
08:54:19.007 -> Free heap end: 266484
08:54:20.017 -> Free heap begin: 266524
08:54:20.050 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
08:54:20.050 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:54:20.050 -> Advertised Device: Name: , Address: 7a:3e:1d:e7:32:57, manufacturer data: 4c001005401c3d5cbf, txPower: 12 
08:54:20.252 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:54:20.286 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:25.028 -> Devices found: 5
08:54:25.028 -> Scan done!
08:54:25.028 -> Free heap done: 265700
08:54:25.028 -> vector capacity before: 5
08:54:25.028 -> vector capacity after: 5
08:54:26.043 -> Free heap end: 266484
08:54:27.021 -> Free heap begin: 266524
08:54:27.056 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
08:54:27.123 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
08:54:27.191 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:27.224 -> Advertised Device: Name: , Address: 7a:3e:1d:e7:32:57, manufacturer data: 4c00100500183d5cbf, txPower: 12 
08:54:27.324 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:54:32.033 -> Devices found: 5
08:54:32.033 -> Scan done!
08:54:32.033 -> Free heap done: 265700
08:54:32.033 -> vector capacity before: 5
08:54:32.033 -> vector capacity after: 5
08:54:33.039 -> Free heap end: 266480
08:54:34.016 -> Free heap begin: 266524
08:54:34.083 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:34.117 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
08:54:34.117 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:54:39.030 -> Devices found: 3
08:54:39.030 -> Scan done!
08:54:39.030 -> Free heap done: 266036
08:54:39.030 -> vector capacity before: 3
08:54:39.030 -> vector capacity after: 3
08:54:40.043 -> Free heap end: 266492
08:54:41.031 -> Free heap begin: 266524
08:54:41.202 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:54:41.546 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:42.097 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
08:54:46.053 -> Devices found: 3
08:54:46.053 -> Scan done!
08:54:46.053 -> Free heap done: 266064
08:54:46.053 -> vector capacity before: 3
08:54:46.053 -> vector capacity after: 3
08:54:47.029 -> Free heap end: 266492
08:54:48.035 -> Free heap begin: 266524
08:54:48.237 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:54:48.677 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:53.046 -> Devices found: 2
08:54:53.046 -> Scan done!
08:54:53.046 -> Free heap done: 266244
08:54:53.046 -> vector capacity before: 2
08:54:53.046 -> vector capacity after: 2
08:54:54.028 -> Free heap end: 266504
08:54:55.041 -> Free heap begin: 266524
08:54:55.307 -> Advertised Device: Name: , Address: 56:c8:10:c3:6a:8d, manufacturer data: 4c00100523180f0cee, txPower: 8 
08:54:55.307 -> Advertised Device: Name: , Address: 71:47:cd:72:2a:23, manufacturer data: 4c00100503186e1d3d, txPower: 12 
08:55:00.052 -> Devices found: 2
08:55:00.052 -> Scan done!
08:55:00.052 -> Free heap done: 266244
08:55:00.052 -> vector capacity before: 2
08:55:00.052 -> vector capacity after: 2
08:55:01.028 -> Free heap end: 266504
08:55:02.036 -> Free heap begin: 266524

Most notable are the Free heap begin: values, they are stable but shrink each time an additional device is found. Even though the vectors are cleared and resized to 0 the capacity remains. I have also modified NimBLEScan to do the same as in the example code and this is still the result.

I have also tried reserving 100 in the NimBLEScanResults constructor and that resulted in no reduction in heap at all except for the fact 100 pointers were reserved right at the start and are wasted ram if not used.

[h2zero]

Here is the log using reserve(100):

09:05:08.937 -> Scanning...
09:05:09.243 -> Free heap begin: 266352
09:05:09.344 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:05:09.412 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:13.581 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
09:05:14.252 -> Devices found: 3
09:05:14.252 -> Scan done!
09:05:14.252 -> Free heap done: 265700
09:05:14.252 -> vector capacity before: 3
09:05:14.252 -> vector capacity after: 3
09:05:15.226 -> Free heap end: 266128
09:05:16.241 -> Free heap begin: 266156
09:05:16.241 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:05:16.511 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:21.251 -> Devices found: 2
09:05:21.251 -> Scan done!
09:05:21.251 -> Free heap done: 265872
09:05:21.251 -> vector capacity before: 2
09:05:21.251 -> vector capacity after: 2
09:05:22.227 -> Free heap end: 266132
09:05:23.237 -> Free heap begin: 266156
09:05:23.305 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:23.372 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:05:28.262 -> Devices found: 2
09:05:28.262 -> Scan done!
09:05:28.262 -> Free heap done: 265872
09:05:28.262 -> vector capacity before: 2
09:05:28.262 -> vector capacity after: 2
09:05:29.232 -> Free heap end: 266132
09:05:30.244 -> Free heap begin: 266156
09:05:30.278 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:05:30.379 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:35.261 -> Devices found: 2
09:05:35.261 -> Scan done!
09:05:35.261 -> Free heap done: 265872
09:05:35.261 -> vector capacity before: 2
09:05:35.261 -> vector capacity after: 2
09:05:36.244 -> Free heap end: 266132
09:05:37.252 -> Free heap begin: 266156
09:05:37.420 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:05:37.420 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:42.258 -> Devices found: 2
09:05:42.258 -> Scan done!
09:05:42.258 -> Free heap done: 265872
09:05:42.258 -> vector capacity before: 2
09:05:42.258 -> vector capacity after: 2
09:05:43.232 -> Free heap end: 266132
09:05:44.245 -> Free heap begin: 266156
09:05:44.514 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:45.156 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:05:46.778 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
09:05:49.267 -> Devices found: 3
09:05:49.267 -> Scan done!
09:05:49.267 -> Free heap done: 265700
09:05:49.267 -> vector capacity before: 3
09:05:49.267 -> vector capacity after: 3
09:05:50.241 -> Free heap end: 266128
09:05:51.251 -> Free heap begin: 266156
09:05:51.284 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:51.520 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:05:56.276 -> Devices found: 2
09:05:56.276 -> Scan done!
09:05:56.276 -> Free heap done: 265872
09:05:56.276 -> vector capacity before: 2
09:05:56.276 -> vector capacity after: 2
09:05:57.252 -> Free heap end: 266132
09:05:58.262 -> Free heap begin: 266156
09:05:58.363 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:05:58.398 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:03.275 -> Devices found: 2
09:06:03.275 -> Scan done!
09:06:03.275 -> Free heap done: 265872
09:06:03.275 -> vector capacity before: 2
09:06:03.275 -> vector capacity after: 2
09:06:04.251 -> Free heap end: 266132
09:06:05.260 -> Free heap begin: 266156
09:06:05.294 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:05.429 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:05.530 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:10.147 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:10.250 -> Devices found: 4
09:06:10.250 -> Scan done!
09:06:10.250 -> Free heap done: 265472
09:06:10.250 -> vector capacity before: 4
09:06:10.250 -> vector capacity after: 4
09:06:11.261 -> Free heap end: 266124
09:06:12.267 -> Free heap begin: 266156
09:06:12.301 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:12.301 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:12.402 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:13.039 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:17.284 -> Devices found: 4
09:06:17.284 -> Scan done!
09:06:17.284 -> Free heap done: 265472
09:06:17.284 -> vector capacity before: 4
09:06:17.284 -> vector capacity after: 4
09:06:18.268 -> Free heap end: 266124
09:06:19.247 -> Free heap begin: 266156
09:06:19.281 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:19.281 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:19.314 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:19.314 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:19.958 -> Advertised Device: Name: , Address: 7a:3e:1d:e7:32:57, manufacturer data: 4c00100540183d5cbf, txPower: 12 
09:06:20.293 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
09:06:24.260 -> Devices found: 6
09:06:24.260 -> Scan done!
09:06:24.260 -> Free heap done: 265164
09:06:24.260 -> vector capacity before: 6
09:06:24.260 -> vector capacity after: 6
09:06:25.274 -> Free heap end: 266116
09:06:26.253 -> Free heap begin: 266156
09:06:26.287 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:26.321 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:26.354 -> Advertised Device: Name: , Address: 7a:3e:1d:e7:32:57, manufacturer data: 4c001005401c3d5cbf, txPower: 12 
09:06:26.455 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:26.655 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:31.290 -> Devices found: 5
09:06:31.290 -> Scan done!
09:06:31.290 -> Free heap done: 265272
09:06:31.290 -> vector capacity before: 5
09:06:31.290 -> vector capacity after: 5
09:06:32.272 -> Free heap end: 266120
09:06:33.252 -> Free heap begin: 266156
09:06:33.319 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
09:06:33.319 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:33.319 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:33.319 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:33.722 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:38.290 -> Devices found: 5
09:06:38.290 -> Scan done!
09:06:38.290 -> Free heap done: 265272
09:06:38.290 -> vector capacity before: 5
09:06:38.290 -> vector capacity after: 5
09:06:39.268 -> Free heap end: 266120
09:06:40.277 -> Free heap begin: 266156
09:06:40.310 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
09:06:40.310 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:40.310 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:40.444 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:40.511 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:45.279 -> Devices found: 5
09:06:45.279 -> Scan done!
09:06:45.279 -> Free heap done: 265272
09:06:45.279 -> vector capacity before: 5
09:06:45.279 -> vector capacity after: 5
09:06:46.260 -> Free heap end: 266120
09:06:47.270 -> Free heap begin: 266156
09:06:47.304 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
09:06:47.304 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:47.708 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:47.843 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:47.910 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:50.433 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
09:06:52.290 -> Devices found: 6
09:06:52.290 -> Scan done!
09:06:52.290 -> Free heap done: 265100
09:06:52.290 -> vector capacity before: 6
09:06:52.290 -> vector capacity after: 6
09:06:53.263 -> Free heap end: 266116
09:06:54.272 -> Free heap begin: 266156
09:06:54.306 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
09:06:54.374 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:06:54.374 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:06:54.374 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:06:54.506 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:06:59.282 -> Devices found: 5
09:06:59.282 -> Scan done!
09:06:59.282 -> Free heap done: 265272
09:06:59.282 -> vector capacity before: 5
09:06:59.282 -> vector capacity after: 5
09:07:00.291 -> Free heap end: 266120
09:07:01.265 -> Free heap begin: 266156
09:07:01.299 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
09:07:01.299 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:01.367 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:07:01.400 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:02.006 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:06.284 -> Devices found: 5
09:07:06.284 -> Scan done!
09:07:06.284 -> Free heap done: 265272
09:07:06.284 -> vector capacity before: 5
09:07:06.284 -> vector capacity after: 5
09:07:07.294 -> Free heap end: 266120
09:07:08.268 -> Free heap begin: 266156
09:07:08.334 -> Advertised Device: Name: NimBLE-Arduino, Address: 7a:3e:1d:e7:32:57, serviceUUID: 0xdead, txPower: 12 
09:07:08.334 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:08.401 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:07:08.501 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:08.534 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:13.306 -> Devices found: 5
09:07:13.306 -> Scan done!
09:07:13.306 -> Free heap done: 265272
09:07:13.306 -> vector capacity before: 5
09:07:13.306 -> vector capacity after: 5
09:07:14.278 -> Free heap end: 266120
09:07:15.283 -> Free heap begin: 266156
09:07:15.318 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:15.318 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:15.318 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:07:15.385 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:20.306 -> Devices found: 4
09:07:20.306 -> Scan done!
09:07:20.306 -> Free heap done: 265472
09:07:20.306 -> vector capacity before: 4
09:07:20.306 -> vector capacity after: 4
09:07:21.279 -> Free heap end: 266124
09:07:22.285 -> Free heap begin: 266156
09:07:22.320 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:07:22.353 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:22.419 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
09:07:22.554 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:22.656 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:27.282 -> Devices found: 5
09:07:27.282 -> Scan done!
09:07:27.282 -> Free heap done: 265300
09:07:27.282 -> vector capacity before: 5
09:07:27.282 -> vector capacity after: 5
09:07:28.294 -> Free heap end: 266120
09:07:29.306 -> Free heap begin: 266156
09:07:29.306 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:07:29.340 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:29.712 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:30.250 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:34.314 -> Devices found: 4
09:07:34.314 -> Scan done!
09:07:34.314 -> Free heap done: 265472
09:07:34.314 -> vector capacity before: 4
09:07:34.314 -> vector capacity after: 4
09:07:35.289 -> Free heap end: 266124
09:07:36.303 -> Free heap begin: 266156
09:07:36.336 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:36.336 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:36.369 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:07:36.538 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:41.317 -> Devices found: 4
09:07:41.317 -> Scan done!
09:07:41.317 -> Free heap done: 265472
09:07:41.317 -> vector capacity before: 4
09:07:41.317 -> vector capacity after: 4
09:07:42.292 -> Free heap end: 266124
09:07:43.303 -> Free heap begin: 266156
09:07:43.403 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:7c:c5:f2, serviceUUID: 0xdead, txPower: 9 
09:07:43.403 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:43.742 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:43.876 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:48.312 -> Devices found: 4
09:07:48.312 -> Scan done!
09:07:48.312 -> Free heap done: 265472
09:07:48.312 -> vector capacity before: 4
09:07:48.312 -> vector capacity after: 4
09:07:49.302 -> Free heap end: 266124
09:07:50.307 -> Free heap begin: 266156
09:07:50.375 -> Advertised Device: Name: NimBLE-Arduino, Address: 30:ae:a4:cc:24:26, serviceUUID: 0xdead, txPower: 9 
09:07:50.375 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:50.409 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:53.708 -> Advertised Device: Name: Temp/Hmid/Pres, Address: cc:b8:6c:d6:a8:82, serviceUUID: 0x181a 
09:07:55.292 -> Devices found: 4
09:07:55.292 -> Scan done!
09:07:55.292 -> Free heap done: 265500
09:07:55.292 -> vector capacity before: 4
09:07:55.292 -> vector capacity after: 4
09:07:56.306 -> Free heap end: 266124
09:07:57.313 -> Free heap begin: 266156
09:07:57.482 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:07:57.515 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:07:57.649 -> Advertised Device: Name: , Address: 6f:f7:fe:9f:4c:84, manufacturer data: 4c00100540183d5cbf, txPower: 12 
09:08:02.319 -> Devices found: 3
09:08:02.319 -> Scan done!
09:08:02.319 -> Free heap done: 265736
09:08:02.319 -> vector capacity before: 3
09:08:02.319 -> vector capacity after: 3
09:08:03.293 -> Free heap end: 266128
09:08:04.301 -> Free heap begin: 266156
09:08:04.334 -> Advertised Device: Name: , Address: 6f:f7:fe:9f:4c:84, manufacturer data: 4c0010050a18cbfa7f, txPower: 12 
09:08:04.368 -> Advertised Device: Name: , Address: 59:12:97:a6:69:de, manufacturer data: 4c00100523189a8d7b, txPower: 8 
09:08:05.076 -> Advertised Device: Name: , Address: 43:4a:96:59:2b:f4, manufacturer data: 4c0010050318a6abf3, txPower: 12 
09:08:09.327 -> Devices found: 3
09:08:09.327 -> Scan done!
09:08:09.327 -> Free heap done: 265736
09:08:09.327 -> vector capacity before: 3
09:08:09.327 -> vector capacity after: 3
09:08:10.303 -> Free heap end: 266128
09:08:11.321 -> Free heap begin: 266156

No change in Free heap begin:

I will think about this in the upcoming advertised device / scan patch that's currently being developed.

If you want to test this just add

NimBLEScanResults(){m_advertisedDevicesVector.reserve(100);}

to the NimBLEScanResults class in NimBLEScan.h.

[justoke]

Thanks so much - your example gave me the clue - if you see my code, I had the delay after the if check to break. This means that I was not allowing any chance for the heap to recover. I moved the delay to just after the clearResults() and now the heap is holding within a safe range.

[justoke]

I've reached the point now, where I just don't think BLE is stable on ESP32. Probably NimBLE library is good enough for most use cases. But running even a modest number of tasks and using a web server just doesn't seem to be a good combination. At least I haven't been able to get the sort of reliability that I think should be possible.

Today I got this for the first time.

[h2zero]

Yes, BLE is tricky to get it playing nice with others, specifically others using the the antenna. I can say from experience that it can be stable, it "just depends", there are always bugs somewhere that can get triggered.

That looks like you're running out of heap.

[doudar]

Yes, BLE is tricky to get it playing nice with others, specifically others using the the antenna. I can say from experience that it can be stable, it "just depends", there are always bugs somewhere that can get triggered.

That looks like you're running out of heap.

Actually coming from a long time amateur radio operator, it's freaking magic and I have absolutely no idea how they can get it to work at all with one radio switching that fast.

I'm just going to continue to point my finger at the asyncwebserver library. No issues I can't solve in an hour of fiddling since I moved away from it.

[justoke]

Point taken, thank you. As I have much invested in the features built on said web server, I'll have to sacrifice BLE for now in the interests of getting a working product.

This issue can be closed.

[h2zero]

That's understandable, sometimes it's best to drop something that doesn't work and just come back to it later. I will close this for now then, please re-open if needed.