ESP32 Min Free Heap Diminished During HTTP Requests

[seansimmons74]

Board

ESP32 DevKit V1

Device Description

ESP32 DevKit V1, from Upload Terminal output: Chip is ESP32-D0WDQ5-V3 (revision v3.1)
Purchased here

Hardware Configuration

For this example, the only connection to the ESP32 is the Micro USB connector. No debugger or other external connections at this time.

Version

v3.0.4

IDE Name

PlatformIO

Operating System

macOS Sonoma 14.3.1

Flash frequency

40MHz

PSRAM enabled

no

Upload speed

460800

Description

While conducting HTTP requests, I am tracking my min free heap and notice that it drops by quite a bit when I conduct either GET or POST requests. In this simplified version of my code, I am just looping an HTTP POST request to grab a JWT that I generate in the cloud. The message size is always less than 250 characters so it isn't an incoming payload size issue.
This simplified version also outputs Free Heap, Heap Integrity, ESP Free Heap, Min Free Heap, and Task Count in a separate function.
I notice that the initial drop in min free heap is the worst (approx. 50k) and then it is followed by many, small drops (maybe 1k at most) with some later requests resulting in some medium sized drops (around 10k) mixed in. There is also a roughly 45k drop when WiFi connects initially but maybe this is expected?

Am I correct in assuming that the min free heap should "return" to previous levels after the http.end() function is called? Shouldn't this free up resources? What is an "acceptable" min free heap level for the aforementioned operations?

I have added a ton of Serial outputs in the main.cpp file and I have also made some changes in the HTTPClient to hopefully find where the issue may be.

In a less simplistic scenario, I will be conducting POST and GET operations every minute to different endpoints in my cloud along with many other things. This issue is a blocker for future plans as I intend to run multiple FreeRTOS tasks at the same time that handle things aside from those HTTP requests such as reading/writing to an SD card, taking sensor readings, and much more so seeing my min free heap drop by a substantial amount and continue to deplete is preventing me from using valuable resources for other duties.

My hunch as of right now: I believe the issue is buried somewhere in the HTTPClient, NetworkClient, or maybe the WiFi library.

I initially commented on a different issue when I noticed they had a similar drop in heap (issue: #10220)
I also updated to V3.0.4 as suggested in that issue but the problems mentioned above continue.

Finally, thank you to @Jason2866 and @me-no-dev for responding promptly on the #10220 issue. I'm still fairly new to all this and appreciate any help I get.

**EDIT: As requested, I've shortened my sketch here to focus mainly on the HTTP POST. The WiFiConnect() is pretty standard and the getHealth() function prints out heap values as you'll see in the Serial output.

Sketch

////See bottom of this sketch for the changes I made to HTTPClient.cpp

#include <Arduino.h>
#include "WiFi.h"
#include <HTTPClient.h>
#include "ArduinoJson.h"

const char* ssid;
const char* password;
String deviceID;
String userEmail;
char* JWT;

void getJWT() {
  Serial.println("Printing millis() at beginning of HTTP Request Function: " + String(millis()));
  Serial.println("Min Free Heap before getJWT: " + String(esp_get_minimum_free_heap_size()));
  
  char output[200];
  char jwtAPI[] = "https://myapi-api.us-west-2.amazonaws.com/production/"; 

  if (WiFi.status() != WL_CONNECTED) {
      WiFiConnect(ssid, password);
  }
  
  JsonDocument doc;
  doc["uuid"] = deviceID;
  doc["email"] = userEmail;

  serializeJson(doc, output);
  doc.clear();
  Serial.println("output for JWT: " + String(output));

  HTTPClient http; 
  http.begin(jwtAPI);
  http.addHeader("Content-Type", "application/json");

  //Problem is between min free heap check 1 and min free heap check 2
  Serial.println("Min Free Heap before HTTP POST: " + String(esp_get_minimum_free_heap_size()));
  int httpCode = http.POST(output);
  Serial.println("Min Free Heap after HTTP POST: " + String(esp_get_minimum_free_heap_size()));
  //End of area of concern - Notice heap tanks on requests performed and remains stable on remaining items in function
  if (httpCode > 0) {
    if (httpCode == HTTP_CODE_OK) {
      http.end();
      Serial.println("Printing millis() after HTTP Request Success: " + String(millis()));
      Serial.println("Min Free Heap after HTTP Request Success: " + String(esp_get_minimum_free_heap_size()));
      return; 
    } else {
      http.end();
      Serial.println("Printing millis() after HTTP Request Failure: " + String(millis()));
      Serial.println("Min Free Heap after HTTP Request Failure: " + String(esp_get_minimum_free_heap_size()));
      return;    
    }
  } else {
    http.end();
    Serial.println("Min Free Heap after HTTP Code < 1: " + String(esp_get_minimum_free_heap_size()));
    return;
  }
}

void setup() {
  delay(1000);
  pinMode(2,OUTPUT); //LED

  Serial.begin(9600);
  delay(500);
  Serial.println("Serial Test");
  delay(5000);
  Serial.println("ESP32 Arduino Core Version: " + String(ESP_ARDUINO_VERSION));

  ssid = "mywifi";
  password = "mywifipass";
  userEmail = "myemail@gmail.com";
  deviceID = "abc-123-xyz";

  while (true) {
    Serial.println("Printing millis() at top of While loop: " + String(millis()));

    getHealth();
    delay(1000);

    getJWT();
    delay(180000); //adding this delay in for @TD-er
  }
}

void loop() {}


///////////////////Below this line are the edits (Serial prints) I made to the HTTPClient.cpp file to narrow down where the heap issues may be: 

//sendRequest(), roughly line 577:
    Serial.println("Min Free Heap in HTTPClient before if !connect(): " + String(esp_get_minimum_free_heap_size()));
    // connect to server
    if (!connect()) {
      return returnError(HTTPC_ERROR_CONNECTION_REFUSED);
    }
    Serial.println("Min Free Heap in HTTPClient after if !connect(): " + String(esp_get_minimum_free_heap_size()));



//connect(), roughly line 1110.  This was the end of the road for me.  Couldn't serial print anything beyond this point in the NetworkClient.cpp...
////////////
Serial.println("Min Free Heap in HTTPClient.connect() before if client connect(): " + String(esp_get_minimum_free_heap_size()));
  if (!_client->connect(_host.c_str(), _port, _connectTimeout)) {
    log_d("failed connect to %s:%u", _host.c_str(), _port);
    return false;
  } 
   Serial.println("Min Free Heap in HTTPClient.connect() after if client connect(): " + String(esp_get_minimum_free_heap_size()));
///////////

Debug Message

Serial Test
ESP32 Arduino Core Version: 196612
Printing millis() at top of While loop: 6555
Free heap: 253436 :: Heap Integrity: 1 :: ESP Free Heap: 299068 :: Minimum Free Heap: 247768 :: Task Count: 6
Printing millis() at beginning of HTTP Request Function: 7679
Min Free Heap before getJWT: 247752
SSID: mywifi
Password: mywifipass
WiFi Connection Attempt #: 1
WiFi Connected!
output for JWT: {"uuid":"abc-123-xyz","email":"myemail@gmail.com"}
Min Free Heap before HTTP POST: 197712
Min Free Heap in HTTPClient before if !connect(): 197712
Min Free Heap in HTTPClient.connect() before if client connect(): 197712
Min Free Heap in HTTPClient.connect() after if client connect(): 151156
Min Free Heap in HTTPClient after if !connect(): 151156
Min Free Heap after HTTP POST: 151156
Printing millis() after HTTP Request Success: 12524
Min Free Heap after HTTP Request Success: 151156
Printing millis() at top of While loop: 192652
Free heap: 206628 :: Heap Integrity: 1 :: ESP Free Heap: 252260 :: Minimum Free Heap: 151156 :: Task Count: 10
Printing millis() at beginning of HTTP Request Function: 193776
Min Free Heap before getJWT: 151156
output for JWT: {"uuid":"abc-123-xyz","email":"myemail@gmail.com"}
Min Free Heap before HTTP POST: 151156
Min Free Heap in HTTPClient before if !connect(): 151156
Min Free Heap in HTTPClient.connect() before if client connect(): 151156
Min Free Heap in HTTPClient.connect() after if client connect(): 147688
Min Free Heap in HTTPClient after if !connect(): 147688
Min Free Heap after HTTP POST: 147688
Printing millis() after HTTP Request Success: 195419
Min Free Heap after HTTP Request Success: 147688
Printing millis() at top of While loop: 375547
Free heap: 206628 :: Heap Integrity: 1 :: ESP Free Heap: 252260 :: Minimum Free Heap: 147688 :: Task Count: 10
Printing millis() at beginning of HTTP Request Function: 376671
Min Free Heap before getJWT: 147688
output for JWT: {"uuid":"abc-123-xyz","email":"myemail@gmail.com"}
Min Free Heap before HTTP POST: 147688
Min Free Heap in HTTPClient before if !connect(): 147688
Min Free Heap in HTTPClient.connect() before if client connect(): 147688
Min Free Heap in HTTPClient.connect() after if client connect(): 147688
Min Free Heap in HTTPClient after if !connect(): 147688
Min Free Heap after HTTP POST: 147688
Printing millis() after HTTP Request Success: 378295
Min Free Heap after HTTP Request Success: 147688
Printing millis() at top of While loop: 558422
Free heap: 206628 :: Heap Integrity: 1 :: ESP Free Heap: 252260 :: Minimum Free Heap: 147688 :: Task Count: 10
Printing millis() at beginning of HTTP Request Function: 559546
Min Free Heap before getJWT: 147688
output for JWT: {"uuid":"abc-123-xyz","email":"myemail@gmail.com"}
Min Free Heap before HTTP POST: 147688
Min Free Heap in HTTPClient before if !connect(): 147688
Min Free Heap in HTTPClient.connect() before if client connect(): 147688
Min Free Heap in HTTPClient.connect() after if client connect(): 147688
Min Free Heap in HTTPClient after if !connect(): 147688
Min Free Heap after HTTP POST: 147688
Printing millis() after HTTP Request Success: 561191
Min Free Heap after HTTP Request Success: 147688
Printing millis() at top of While loop: 741319
Free heap: 206628 :: Heap Integrity: 1 :: ESP Free Heap: 252260 :: Minimum Free Heap: 147688 :: Task Count: 10
Printing millis() at beginning of HTTP Request Function: 742443
Min Free Heap before getJWT: 147688
output for JWT: {"uuid":"abc-123-xyz","email":"myemail@gmail.com"}
Min Free Heap before HTTP POST: 147688
Min Free Heap in HTTPClient before if !connect(): 147688
Min Free Heap in HTTPClient.connect() before if client connect(): 147688
Min Free Heap in HTTPClient.connect() after if client connect(): 147688
Min Free Heap in HTTPClient after if !connect(): 147688
Min Free Heap after HTTP POST: 147688
Printing millis() after HTTP Request Success: 743867
Min Free Heap after HTTP Request Success: 147688

----end of my serial output----

Other Steps to Reproduce

This is not board specific as I've seen the exact same results on different boards. This is also not a memory fragmentation issue as this seems to happen even on the first HTTP request that the program performs (its my understanding that fragmentation happens after long runtimes). I verified that this is not a POST or GET specific issue as both have this problem.
I have also tried disconnecting WiFi after each request finishes and waiting for 3 minutes before reconnecting as well as throwing http.end() everywhere possible.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[TD-er]

Can you print some timestamp (or millis() along with the logs and then try again?
I suspect the amount of free memory will deplete with every call, but if you stop making those calls, it should return about 2 minutes after the call was made that will now free its memory.

I know @me-no-dev has done some fixes in that code, so it should be less of an issue in 3.0.4, but I have not yet tested this myself. At least it should now no longer be possible to DDoS your ESP out of memory anymore.

[me-no-dev]

please make the sketch shorter. Just the minimal necessary to reproduce the problem

[seansimmons74]

@me-no-dev I have shortened the sketch by roughly half and ran it again to give a simplified serial output. @TD-er I added millis() in a few places to show how long the process is taking. I also added a delay(180000) after each run in hopes that it would free up memory but as you can see in my serial output, that is not the case. It seems to be sitting at around 147k (still running it right now) but maybe it will continue to drop after I let this run for a bit longer.

[me-no-dev]

if it stabilizes at some value (147K in your case) that means that everything is OK. Only certain number of connections are kept in memory (and later released too). Log above is OK

[seansimmons74]

I don't understand why it wouldn't free up those resources after finishing operations though. Like if I disconnect wifi too, shouldn't I get those resources back? Also, it's appearing stable now because that was only a few requests over maybe 20 minutes. If I let this run for longer, and shorten the request intervals to more realistic use cases, it will begin to drop again.

[TD-er]

I did a few tests yesterday on some nodes and in my setup the previously observed behavior was no longer visible.
This previous behavior was keeping some memory allocated after the client already closed the connection and fetched new data.
For each call the amount of free memory was less and each of those were released after exactly 2 minutes.

This could allow an ESP to be DDoS'ed out of memory by just fetching some JSON for example. (lots of times, but still...)

As I'm now no longer seeing this behavior, I think we should get back to what you're observing and see if this makes sense or not.

First of all, you are checking the esp_get_minimum_free_heap_size() which is obviously not going to get better ever until you reboot :)
Maybe ESP.getFreeHeap() will help here :)

With the printing of millis(), I meant at each log line, so you could see how long it may take for the memory to return.
Like this:

Serial.print(millis());
Serial.print(F(" : "));
Serial.println(....);

(Maybe best to also make a function of it.)

Also I didn't mean to just wait for a few minutes using delay(), but rather keep track of the millis() in the log so you could see if there was any changes.
This while loop should also be done in the loop() function. Now you are calling it still from the setup() and I honestly have no idea if the code will behave differently if the setup() never returns like in your test code.

Anyway, when starting some services, they will allocate some memory.
Not sure if these are always freed again when those services are stopped. For example some may allocate static memory.

So it could be something like this:

void loop() {
static unsigned long lastLogPrint = 0;
    if ((millis() - lastLogPrint) > 1000) {
        Serial.print(millis());
        Serial.print(F(" : "));
        Serial.print(F("Free Heap: "));
        Serial.println(ESP.getFreeHeap());
        lastLogPrint = millis();
    }
    getHealth();
    getJWT();

}

And maybe you should also remove the other Serial.print.... statements from your other code or else your ESP will run out of memory as the serial buffer cannot be read fast enough.

[seansimmons74]

Hi @TD-er, sorry for the delay in responding. I have copied your code exactly (see new serial output below for results). The ESP Free Heap seems to properly "free up" after each request. I am seeing roughly 250k prior to the HTTP POST and it drops to 200k after, then back to 250k when finished. I have some questions though.

1. I see the Min Free Heap still dropping when I leave this running for a while. Are there any consequences of this? Will this cause the program to hang/crash when it reaches a certain point or should I just be worried about the ESP Free Heap?
2. What is an acceptable "low value" that you would recommend I don't go below for ESP Free Heap? When I add back all my FreeRTOS tasks, I plan on keeping an eye on the free heap and seeing how low it gets but is there a ballpark figure of free heap to keep in reserve that I might need to consider going forward?

Here is the new serial printout:

6555 : Free Heap: 299148
Free heap: 253436 :: Heap Integrity: 1 :: ESP Free Heap: 299068 :: Minimum Free Heap: 247768 :: Task Count: 6
ESP Free Heap before HTTP POST: 252736
ESP Free Heap after HTTP POST: 204316
14444 : Free Heap: 251620
Free heap: 205908 :: Heap Integrity: 1 :: ESP Free Heap: 251540 :: Minimum Free Heap: 152040 :: Task Count: 10
ESP Free Heap before HTTP POST: 251628
ESP Free Heap after HTTP POST: 204100
16256 : Free Heap: 251388
Free heap: 205880 :: Heap Integrity: 1 :: ESP Free Heap: 251512 :: Minimum Free Heap: 151764 :: Task Count: 10
ESP Free Heap before HTTP POST: 251404
ESP Free Heap after HTTP POST: 203864
17793 : Free Heap: 251156
Free heap: 205444 :: Heap Integrity: 1 :: ESP Free Heap: 251076 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 251048
ESP Free Heap after HTTP POST: 203684
19022 : Free Heap: 250944
Free heap: 205232 :: Heap Integrity: 1 :: ESP Free Heap: 250864 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 250792
ESP Free Heap after HTTP POST: 203432
20455 : Free Heap: 250728
Free heap: 205016 :: Heap Integrity: 1 :: ESP Free Heap: 250648 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 250740
ESP Free Heap after HTTP POST: 203196
21888 : Free Heap: 250504
Free heap: 204792 :: Heap Integrity: 1 :: ESP Free Heap: 250424 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 250516
ESP Free Heap after HTTP POST: 202992
23320 : Free Heap: 250280
Free heap: 204568 :: Heap Integrity: 1 :: ESP Free Heap: 250404 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 250292
ESP Free Heap after HTTP POST: 202752
24758 : Free Heap: 250056
Free heap: 204344 :: Heap Integrity: 1 :: ESP Free Heap: 249976 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 249920
ESP Free Heap after HTTP POST: 202528
25980 : Free Heap: 249820
Free heap: 204108 :: Heap Integrity: 1 :: ESP Free Heap: 249740 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 249844
ESP Free Heap after HTTP POST: 202308
27417 : Free Heap: 249608
Free heap: 203896 :: Heap Integrity: 1 :: ESP Free Heap: 249528 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 249472
ESP Free Heap after HTTP POST: 202088
28648 : Free Heap: 249384
Free heap: 203672 :: Heap Integrity: 1 :: ESP Free Heap: 249304 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 249220
ESP Free Heap after HTTP POST: 201856
29876 : Free Heap: 249160
Free heap: 203448 :: Heap Integrity: 1 :: ESP Free Heap: 249068 :: Minimum Free Heap: 147020 :: Task Count: 10
ESP Free Heap before HTTP POST: 249024
ESP Free Heap after HTTP POST: 201648
31111 : Free Heap: 248936
Free heap: 203224 :: Heap Integrity: 1 :: ESP Free Heap: 248856 :: Minimum Free Heap: 144952 :: Task Count: 10
ESP Free Heap before HTTP POST: 248812
ESP Free Heap after HTTP POST: 201420
32538 : Free Heap: 248712
Free heap: 203204 :: Heap Integrity: 1 :: ESP Free Heap: 248836 :: Minimum Free Heap: 144952 :: Task Count: 10
ESP Free Heap before HTTP POST: 248588
ESP Free Heap after HTTP POST: 201176
33992 : Free Heap: 248484
Free heap: 202772 :: Heap Integrity: 1 :: ESP Free Heap: 248404 :: Minimum Free Heap: 144952 :: Task Count: 10
ESP Free Heap before HTTP POST: 248500
ESP Free Heap after HTTP POST: 200960
35406 : Free Heap: 248252
Free heap: 202540 :: Heap Integrity: 1 :: ESP Free Heap: 248172 :: Minimum Free Heap: 144952 :: Task Count: 10
ESP Free Heap before HTTP POST: 248128
ESP Free Heap after HTTP POST: 200712```

[seansimmons74]

Additionally, after a longer runtime, I eventually seem to be having an ssl related issue? When this happens, I see a drop in all my health metrics I am logging. See below:

2084184 : Free Heap: 248248
Free heap: 202536 :: Heap Integrity: 1 :: ESP Free Heap: 248168 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 248136
ESP Free Heap after HTTP POST: 200980
2085401 : Free Heap: 248244
Free heap: 202532 :: Heap Integrity: 1 :: ESP Free Heap: 248164 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 248264
[2205807][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[2205907][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 247700
[2206007][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
2206147 : Free Heap: 250280
Free heap: 204568 :: Heap Integrity: 1 :: ESP Free Heap: 250200 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 249948
[2326538][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[2326637][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 245764
[2326737][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
2326877 : Free Heap: 248340
Free heap: 202628 :: Heap Integrity: 1 :: ESP Free Heap: 248260 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 248012
[2447269][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[2447369][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 243828
[2447469][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
2447609 : Free Heap: 246400
Free heap: 200688 :: Heap Integrity: 1 :: ESP Free Heap: 246320 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 246060
[2568098][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[2568198][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 241840
[2568298][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
2568438 : Free Heap: 244440
Free heap: 198728 :: Heap Integrity: 1 :: ESP Free Heap: 244360 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 244100
[2688933][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[2689032][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 239928
[2689132][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
2689271 : Free Heap: 242500
Free heap: 196788 :: Heap Integrity: 1 :: ESP Free Heap: 242420 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 242172
[2809661][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[2809761][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 237988
[2809861][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
2810001 : Free Heap: 240560
Free heap: 194848 :: Heap Integrity: 1 :: ESP Free Heap: 240480 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 240232
[2930494][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[2930594][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 236036
[2930694][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
2930833 : Free Heap: 238608
Free heap: 192896 :: Heap Integrity: 1 :: ESP Free Heap: 238528 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 238280
[3051224][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3051324][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 234076
[3051424][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
3051563 : Free Heap: 236656
Free heap: 190944 :: Heap Integrity: 1 :: ESP Free Heap: 236576 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 236328
[3171955][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3172054][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 232136
[3172154][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
3172293 : Free Heap: 234708
Free heap: 188996 :: Heap Integrity: 1 :: ESP Free Heap: 234628 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 234380
[3292687][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3292789][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 230180
[3292888][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
3293028 : Free Heap: 232752
Free heap: 187040 :: Heap Integrity: 1 :: ESP Free Heap: 232672 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 232432
[3413515][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3413615][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 228184
[3413715][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
3413855 : Free Heap: 230784
Free heap: 185072 :: Heap Integrity: 1 :: ESP Free Heap: 230704 :: Minimum Free Heap: 136912 :: Task Count: 10
ESP Free Heap before HTTP POST: 230456
[3534348][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3534448][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 226260
[3534548][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
3534688 : Free Heap: 228832
Free heap: 183120 :: Heap Integrity: 1 :: ESP Free Heap: 228752 :: Minimum Free Heap: 120176 :: Task Count: 10
ESP Free Heap before HTTP POST: 228492
[3655077][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3655177][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 224300
[3655277][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
3655416 : Free Heap: 226884
Free heap: 181172 :: Heap Integrity: 1 :: ESP Free Heap: 226804 :: Minimum Free Heap: 120176 :: Task Count: 10
ESP Free Heap before HTTP POST: 226544
[3775910][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3776010][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
ESP Free Heap after HTTP POST: 222356
[3776110][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
3776250 : Free Heap: 224944
Free heap: 179232 :: Heap Integrity: 1 :: ESP Free Heap: 224864 :: Minimum Free Heap: 103440 :: Task Count: 10
ESP Free Heap before HTTP POST: 224604
[3896640][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -1
[3896740][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"

And eventually (around a runtime of 4500000) it seems to just sort itself out and get to what it was doing before??

[TD-er]

The minimum free heap is actually a very descriptive term.
It is the lowest point of free heap that was reached.

There is no general one-size-fits-all amount of free heap which is acceptable.
For example in my project (ESPEasy) on ESP8266 I have situations where I have to keep running with less than 3k of free RAM.

There is more to it than just the amount of free ram.
For example if your memory gets fragmented you can have like 100k of free memory, but the largest memory block could be less than a few kB, so any attempt to allocate more than this largest free block will fail.

Some libraries like for setting up TLS encryption may need quite a lot of memory (e.g. 20k - 30k, just as an example)
So you need to be sure you can successfully keep allocating your runtime data.
Typically I want to keep the free ram on ESP32-xx over 80k, but this is no hard limit.

[seansimmons74]

I imagine "just google it" is the answer here but what methods have you had success with in order to prevent/prolong memory fragmentation issues? Do you encourage restarting the board often to prevent this (or in general)?

[TD-er]

Main trick is to prevent (re)allocation of large memory chunks.
Thus use reserve when possible to prevent reallocation.
And when possible use the move operator instead of copying.
This also makes your code faster.

When you need to have large chunks of memory, it is best to allocate those at startup.
If you really need a lot of memory, then you can also look into those modules with PSRAM.

[seansimmons74]

Hello again @TD-er. Thank you for your help and advice as I navigate the complexities of this project. I have been letting my program run for longer durations to see how it performs. I have decided that I need to get rid of all String usage and will use static allocations as much as possible to hopefully limit fragmentation. I have let my board run for many days at a time multiple times now and I will soon be finished with the modifications necessary to remove all String usage, add additional logging, and a few other optimizations.
However, I do have some other questions, if you're willing to answer.

In the current session, my code is having lots of SSL/Networking related errors come up. I think it is likely due to heap fragmentation but maybe I'm wrong. Here are some of the errors and my serial printouts in the mix:

[367644871][E][ssl_client.cpp:36] _handle_error(): [start_ssl_client():298]: (-32512) SSL - Memory allocation failed
[367645044][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -32512
[367645152][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367645254][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367645355][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
Error on HTTP Request
-1

[367645463][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367645589][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
Uptime: 367645851 :: Free heap: 69600 :: Heap Integrity: 1 :: ESP Free Heap: 114964 :: Minimum Free Heap: 220 :: Task Count: 14
[367646913][E][ssl_client.cpp:36] _handle_error(): [start_ssl_client():298]: (-32512) SSL - Memory allocation failed
Uptime: 367646976 :: Free heap: 66160 :: Heap Integrity: 1 :: ESP Free Heap: 111516 :: Minimum Free Heap: 220 :: Task Count: 14
[367647231][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -32512
[367647338][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367647442][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367647543][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
Error on HTTP Request
-1

[367647650][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367647777][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
Uptime: 367648101 :: Free heap: 69228 :: Heap Integrity: 1 :: ESP Free Heap: 114592 :: Minimum Free Heap: 220 :: Task Count: 14
[367649166][E][ssl_client.cpp:36] _handle_error(): [start_ssl_client():298]: (-32512) SSL - Memory allocation failed
Uptime: 367649226 :: Free heap: 67732 :: Heap Integrity: 1 :: ESP Free Heap: 113100 :: Minimum Free Heap: 220 :: Task Count: 14
[367649484][E][NetworkClientSecure.cpp:159] connect(): start_ssl_client: connect failed: -32512
[367649591][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367649693][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367649795][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
Error on HTTP Request
-1

[367649899][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[367650029][E][NetworkClient.cpp:318] setSocketOption(): fail on 0, errno: 9, "Bad file number"
Uptime: 367650350 :: Free heap: 72288 :: Heap Integrity: 1 :: ESP Free Heap: 117644 :: Minimum Free Heap: 220 :: Task Count: 14
Uptime: 367651475 :: Free heap: 72288 :: Heap Integrity: 1 :: ESP Free Heap: 117644 :: Minimum Free Heap: 220 :: Task Count: 14
Uptime: 371118360 :: Free heap: 134860 :: Heap Integrity: 1 :: ESP Free Heap: 180216 :: Minimum Free Heap: 220 :: Task Count: 14

These errors are happening when I am attempting those same API requests mentioned earlier.

My questions are:

1. Should I keep wifi connected at all times or is that potentially causing the errors above?
2. My code was running consistently but it seemed to have started soaking up Free Heap (dropping down to 30k) when attempting the API requests. It was repeating the errors above multiple times and failing the API request functions every few minutes and dropping Free Heap by 1-2k each failure but it then suddenly seemed to recover itself and the Free Heap is back at 134k/180k (see final line of serial output). How does this make sense? I thought for sure the code was finally heading for a crash or something but it just magically keeps going?
3. Is there any benefit of using the WiFiClientSecure library instead of just the regular WiFi library to establish a connection? Would that help with these SSL errors?
4. Is this purely a heap related issue in your opinion?

[TD-er]

WiFi connected to AP should be perfectly fine.
However a side-effect of disabling WiFi might be that you also close all connections which clears allocated memory.

I think you should only use secure connections when needed, for example to a MQTT broker.
Those are long-lasting connections and thus only need to be reconnected every now and then. Thus if a stale connection keeps its memory allocated for a few minutes, then it still isn't an issue.

It is not really a problem to use String in your code.
Just make sure you do as few (re)allocations as possible.
Thus try to use reserve when possible and try to use the std::move operator as return of functions and const String& as function arguments.

Your errors suggest you're running out of free heap, or at least the largest continuous free block is too small for SSL to allocate its stuff.

[seansimmons74]

Thank you for your help @TD-er