esp8266 ESP-12E memory leak under http POST

[andyb2000]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP-12E (NodeMCU 1.0)
• Core Version: 1.8.9
• Development Env: Arduino IDE
• Operating System: Ubuntu

Settings in IDE

• Module: NodeMCU 1.0 ESP-12E
• Flash Mode: [qio|dio|other]
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• Reset Method: [ck|nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: 80Mhz
• Upload Using: SERIAL
• Upload Speed: 115200

Problem Description

I'm writing every minute to an https web server and observing that after 1hr (Approx) the unit crashes and restarts due to memory error. I've added debug options and a memory watcher and can see the memory drops after each http.post of data (Testing memory before action and after action). It constantly drops and never returns until the unit gets too low and crashes/restarts.

I've tried with v2 Lower Memory, v2 higher bandwidth, v1.4 Higher Bandwidth.

I've also tried different board release versions:
2.6.3
2.5.2
2.5.1
2.4.2
2.4.0

Problem persists in all combinations.
In my sketch it runs for around 1-2hrs and slowly see memory exhausting. It is always after the http post that my code sees the memory reduction (in my code I've got several lines checking memory to try to find the leak point):

after http post (2832335) -> free:  6000 - max:  5624 - frag:   7% <- 
after http post (2892265) -> free:  5712 - max:  5336 - frag:   7% <- 
after http post (2952261) -> free:  5424 - max:  5048 - frag:   7% <- 
after http post (3012266) -> free:  5136 - max:  4760 - frag:   8% <- 
after http post (3072262) -> free:  4848 - max:  4472 - frag:   8% <- 
after http post (3132264) -> free:  4560 - max:  4184 - frag:   9% <- 
after http post (3192289) -> free:  4272 - max:  3896 - frag:   9% <- 
after http post (3252361) -> free:  3984 - max:  3608 - frag:  10% <- 
after http post (3312267) -> free:  3696 - max:  3320 - frag:  11% <- 
after http post (3445539) -> free:  2480 - max:  1536 - frag:  33% <- 

Until finally low memory:

after http post (3445539) -> free:  2480 - max:  1536 - frag:  33% <- 
[HTTP-Client][end] tcp is closed
Phone home to show I'm awake
[HTTP-Client][begin] url: https://myurl.co.uk:443/update.php
[HTTP-Client][begin] host: myurl.co.uk port: 443 url: /update.php
[HTTP-Client][sendRequest] type: 'POST' redirCount: 0
BSSL:_connectSSL: start connection
BSSL:CERT: 30 82 06 - REDACTED - 06
BSSL:_run_until: Timeout
BSSL:_wait_for_handshake: failed
BSSL:Couldn't connect. Error = 'Unknown error code.'
[HTTP-Client] failed connect to myurl.co.uk:443
[HTTP-Client][returnError] error(-1): connection refused
[HTTP-Client][returnError] error(-4): not connected
[HTTP-Client][end] tcp is closed
Phone home to show I'm awake
[HTTP-Client][begin] url: https://myurl.co.uk:443/update.php
[HTTP-Client][begin] host: myurl.co.uk port: 443 url: /update.php
[HTTP-Client][sendRequest] type: 'POST' redirCount: 0
BSSL:_connectSSL: start connection
BSSL:CERT: 30 82 06 - REDACTED - 06
Fatal exception 29(StoreProhibitedCause):
epc1=0x4000df64, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

Exception (29):
epc1=0x4000df64 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

>>>stack>>>

ctx: sys
sp: 3fffec10 end: 3fffffb0 offset: 01a0

(See below for decoded stacktrace)

MCVE Sketch

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>
#include <SPI.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <FS.h>
#include <CertStoreBearSSL.h>
/* #include <time.h> */
#include <Arduino.h>
extern "C" {
#include <user_interface.h>
}

String server = "https://myurl.co.uk:443/update.php";
String apiKey = "83925"; // A unique userid for this unit
const char* MY_SSID = "SSID"; 
const char* MY_PWD = "SSIDPASS";
bool wifi_connected = false;
bool did_wifi_ever_work = false;
String otaupdate_error = "";
String postStr = "";
String payload = "";
String payload_substr = "";
static uint32_t lastmin=65536;
static uint32_t myfree;
static uint16_t mymax;
static uint8_t myfrag;
char resetreason_buff[32];
#define MAX_MILLIS_TO_WAIT 8000  //How long to wait for serial data to avoid hanging code
const unsigned long oneMinute = 60 * 1000UL; // This is the counter timer set at 1 minute to do things (upload)
static unsigned long lastSampleTime = 0 - oneMinute;

#include <ConfigManager.h>
ConfigManager configManager;
struct Config {
    char name[20];
    bool enabled;
    int8 hour;
    char password[20];
} config;

struct Metadata {
    int8_t version;
} meta;

void setup() {
  Serial.begin(115200);
  delay(3000);
  Serial.println(F("Debugger starting"));
  Serial.println(F(" Test by Andy Brown"));
  switch (ESP.getResetInfoPtr()->reason) {
    
    case REASON_DEFAULT_RST: 
      // do something at normal startup by power on
      strcpy_P(resetreason_buff, PSTR("Power on"));
      break;
      
    case REASON_WDT_RST:
      // do something at hardware watch dog reset
      strcpy_P(resetreason_buff, PSTR("Hardware Watchdog"));     
      break;
      
    case REASON_EXCEPTION_RST:
      // do something at exception reset
      strcpy_P(resetreason_buff, PSTR("Exception"));      
      break;
      
    case REASON_SOFT_WDT_RST:
      // do something at software watch dog reset
      strcpy_P(resetreason_buff, PSTR("Software Watchdog"));
      break;
      
    case REASON_SOFT_RESTART: 
      // do something at software restart ,system_restart 
      strcpy_P(resetreason_buff, PSTR("Software/System restart"));
      break;
      
    case REASON_DEEP_SLEEP_AWAKE:
      // do something at wake up from deep-sleep
      strcpy_P(resetreason_buff, PSTR("Deep-Sleep Wake"));
      break;
      
    case REASON_EXT_SYS_RST:
      // do something at external system reset (assertion of reset pin)
      strcpy_P(resetreason_buff, PSTR("External System"));
      break;
      
    default:  
      // do something when reset occured for unknown reason
      strcpy_P(resetreason_buff, PSTR("Unknown"));     
      break;
  }
  Serial.printf("\n\nReason for reboot: %s\n", resetreason_buff);
  WiFi.hostname("TestItem-" + apiKey);
  WiFi.begin(MY_SSID, MY_PWD);
  unsigned long starttime = millis();
  while (WiFi.status() != WL_CONNECTED && ((millis() - starttime) < (MAX_MILLIS_TO_WAIT*5))) {
    delay(50); // urgh, any better way?
  };
  if (WiFi.status() != WL_CONNECTED) {
    Serial.println(F("Error no wifi connect"));
    wifi_connected = false;
  } else {
    Serial.println(F("Wifi connected OK"));
    wifi_connected = true;
    did_wifi_ever_work = true;
  };
  if (!wifi_connected) {
    configManager.setAPName("Access-Point-Test");
  configManager.setAPFilename("/index.html");
  configManager.addParameter("name", config.name, 20);
  configManager.addParameter("enabled", &config.enabled);
  configManager.addParameter("hour", &config.hour);
  configManager.addParameter("password", config.password, 20, set);
  configManager.addParameter("version", &meta.version, get);
  configManager.begin(config);
  };
  Serial.println(F("Completed setup"));

}

void loop() {
  unsigned long now = millis();
  if (WiFi.status() != WL_CONNECTED) {
    wifi_connected=false;
  } else {
    wifi_connected=true;
  };
  if (!wifi_connected && did_wifi_ever_work == false) {
    configManager.loop();
  } else {
    unsigned long now = millis();
    if (now - lastSampleTime >= oneMinute) {
      Serial.println(F("Phone home to show I'm awake"));
      lastSampleTime = now;
      if (WiFi.status() == WL_CONNECTED) {
         ESP.getHeapStats(&myfree, &mymax, &myfrag);
         if ( lastmin > myfree ) {
          Serial.printf("before bearssl (%d) -> free: %5d - max: %5d - frag: %3d%% <- \n", millis(), myfree, mymax, myfrag);
          lastmin = myfree;
         };
        BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure();
        client->setInsecure();
        ESP.getHeapStats(&myfree, &mymax, &myfrag);
         if ( lastmin > myfree ) {
          Serial.printf("after bearssl (%d) -> free: %5d - max: %5d - frag: %3d%% <- \n", millis(), myfree, mymax, myfrag);
          lastmin = myfree;
         };
         HTTPClient http;    //Declare object of class HTTPClient
         http.setTimeout(15000);
         int beginResult = http.begin(dynamic_cast<WiFiClient&>(*client), server);
         http.addHeader("Content-Type", " application/x-www-form-urlencoded; charset=UTF-8");
         http.addHeader("User-Agent", "TestWeb-v1");
         ESP.getHeapStats(&myfree, &mymax, &myfrag);
         if ( lastmin > myfree ) {
            Serial.printf("before http post (%d) -> free: %5d - max: %5d - frag: %3d%% <- \n", millis(), myfree, mymax, myfrag);
            lastmin = myfree;
         };
         int httpCode = http.POST("&testvar=1&testvar2=2&testvar3=3&testvar4=4&testvar5=5");   //Send the request
         payload = http.getString();                  //Get the response payload
         ESP.getHeapStats(&myfree, &mymax, &myfrag);
         if ( lastmin > myfree ) {
            Serial.printf("after http post (%d) -> free: %5d - max: %5d - frag: %3d%% <- \n", millis(), myfree, mymax, myfrag);
            lastmin = myfree;
         };
      http.end();  //Close connection
      client->stop();
      };
    };
  };
}

Debug Messages

Decoding stack results
0x402269bd: tcp_output_alloc_header at core/tcp_out.c line 1863
0x401009a9: check_poison_neighbors(unsigned short) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_local.c line 71
0x40100cdc: umm_malloc(size_t) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_malloc.cpp line 511
0x40100894: get_poisoned(void*, size_t) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_poison.c line 112
0x4020f443: loop_task(ETSEvent*) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 188
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x4022e805: br_sha2small_round at src/hash/sha2small.c line 99
0x4022e758: br_sha2small_round at src/hash/sha2small.c line 85
0x4022ed37: sha2small_out at src/hash/sha2small.c line 249
0x4022ec2b: br_sha224_update at src/hash/sha2small.c line 221
0x4022ed78: br_sha256_out at src/hash/sha2small.c line 305
0x40235bfd: br_hmac_out at src/mac/hmac.c line 120
0x4022ec2b: br_sha224_update at src/hash/sha2small.c line 221
0x40235b6d: br_hmac_init at src/mac/hmac.c line 88
0x40236124: br_hmac_drbg_generate at src/rand/hmac_drbg.c line 88
0x4022ed37: sha2small_out at src/hash/sha2small.c line 249
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x40221c09: glue2esp_linkoutput at glue-esp/lwip-esp.c line 301
0x40221e96: new_linkoutput at glue-lwip/lwip-git.c line 260
0x402222b0: ethernet_output at netif/ethernet.c line 312
0x40221c09: glue2esp_linkoutput at glue-esp/lwip-esp.c line 301
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x402304ff: br_ssl_engine_current_state at src/ssl/ssl_engine.c line 1280
0x40205219: BearSSL::WiFiClientSecure::_run_until(unsigned int, bool) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 453
0x40100c8f: umm_free(void*) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_malloc.cpp line 362
0x40100cac: umm_poison_free_fl(void*, char const*, int) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_local.c line 149
0x40100344: free(void*) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/heap.cpp line 259
0x4022bf10: mem_free at core/mem.c line 237
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x4020f4c3: __yield() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 100
0x401002a2: millis() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_wiring.cpp line 188
0x4020f501: optimistic_yield(uint32_t) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 133
0x40202f8f: WiFiClient::available() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi/src/WiFiClient.cpp line 263
0x402051c2: BearSSL::WiFiClientSecure::_run_until(unsigned int, bool) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 527
0x40230586: br_ssl_engine_hs_reset at src/ssl/ssl_engine.c line 1305
0x4020543c: BearSSL::WiFiClientSecure::_wait_for_handshake() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 564
0x40205647: BearSSL::WiFiClientSecure::_connectSSL(char const*) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 1122
0x4010095a: check_poison_neighbors(unsigned short) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_local.c line 59
0x40100cdc: umm_malloc(size_t) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_malloc.cpp line 511
0x40100894: get_poisoned(void*, size_t) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_poison.c line 112
0x4022b440: ip4_output_if_opt at core/ipv4/ip4.c line 1577
0x4010030c: malloc(size_t) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/heap.cpp line 232
0x4022b466: ip4_output_if at core/ipv4/ip4.c line 1550
0x4022bfff: ip_chksum_pseudo at core/inet_chksum.c line 395
0x401002a2: millis() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_wiring.cpp line 188
0x40227552: tcp_output at core/tcp_out.c line 1361
0x40223b5d: sys_timeout_abs at core/timeouts.c line 189
0x4020f497: __esp_yield() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 107
0x4020fb3d: __delay(unsigned long) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_wiring.cpp line 57
0x4020fb32: __delay(unsigned long) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_wiring.cpp line 54
0x40203b18: WiFiClient::connect(IPAddress, unsigned short) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi/src/include/ClientContext.h line 133
0x402057ab: BearSSL::WiFiClientSecure::connect(char const*, unsigned short) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 231
0x4020713c: HTTPClient::connect() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 1195
0x40100fb1: umm_poison_realloc_fl(void*, size_t, char const*, int) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_local.c line 140
0x40207fd4: HTTPClient::sendRequest(char const*, unsigned char const*, unsigned int) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 668
0x40246fee: umm_info(void*, int) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/umm_malloc/umm_info.c line 164
0x4020e3ab: String::reserve(unsigned int) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/WString.cpp line 146
0x40208194: HTTPClient::POST(unsigned char const*, unsigned int) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 599
0x402081c2: HTTPClient::POST(String const&) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 604
0x40201462: loop() at /media/ABDS2-ssh/Andy_s Documents/arduino/sketchbook/test/test.ino line 161
0x4020d600: fs::FS::open(char const*, char const*) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/FS.cpp line 354
0x401001b4: ets_post(uint8, ETSSignal, ETSParam) at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 160
0x4020f5ac: loop_wrapper() at /home/browna/.arduino15/packages/esp8266/hardware/esp8266/2.6.3/cores/esp8266/core_esp8266_main.cpp line 180

Thanks in advance for your help, I'm trying various things (the above code is a cut down version of my code that I'm doing to test the memory leak and it's present in this cut down version too) and no avail!
Andy.

[earlephilhower]

@andyb2000 you put an apikey in your post. If this is anything critical, change it and edit the MCVE above to obscure it ASAP.

What is the beginning free memory reported in your sketch? Do you start with significantly more free space than the 5.5K shown?

Also, debugging is not terribly optimized, so if you're only running w/debugging enabled you might want to give it a go w/o debugging. There are quite a few strings added when debug is enabled, and not all of them end up in PROGMEM (so they eat into heap).

Finally, I'd move the BearSSL client from a temporary new to a global instantiation:
BearSSL::WiFiClientSecure client();. There is no MMU and heap fragmentation can be a problem.

[andyb2000]

Thanks @earlephilhower apikey not relevant so no worries! Thank you.

I'd previously ran without debug and added it as I found more memory leaks so although debug is heavy it's not contributing in this case.
I've been testing various scenarios out all afternoon and have identified it is the SSL+BearSSL components that seem to be leaking memory.

I've changed to this code:

        // BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure();
        // client->setInsecure();
         HTTPClient http;    //Declare object of class HTTPClient
         http.setTimeout(15000);
         int beginResult = http.begin(server);
         http.addHeader("Content-Type", " application/x-www-form-urlencoded; charset=UTF-8");
         http.addHeader("User-Agent", "TestWeb-v1");
         int httpCode = http.POST("&testvar=1&testvar2=2&testvar3=3&testvar4=4&testvar5=5"); 
         payload = http.getString();                  //Get the response payload

(note removed the mem check to make it clearer to read) and now I'm getting no further memory loss after initial couple of loops setting up variables, etc.

So the memory loss appears to be relating to the BearSSL part unfortunately.

So I think you're on the right lines to global instantiate the BearSSL, so I'll give that a try now and feed back.

[andyb2000]

Success.
as @earlephilhower advised. Moving the:

BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure();

up out of the loop and into the top definitions has solved the memory leak/loss during code operation. I've had it running now for almost 15 minutes with no loss of memory in my test code.

Thanks @earlephilhower is this worth moving to the bearssl-esp8266 issue tracker or just closing off?

[earlephilhower]

No, this is a bug in your example code. I see you newing the client...but I don't see a delete client anywhere. So you've been allocating the SSL contexts without freeing them.

Since the context is a little big, it's a good idea to allocate once (i.e. as a global) and just leave it to avoid memory fragmentation.

[Abedi98]

No, this is a bug in your example code. I see you newing the client...but I don't see a delete client anywhere. So you've been allocating the SSL contexts without freeing them.

Since the context is a little big, it's a good idea to allocate once (i.e. as a global) and just leave it to avoid memory fragmentation.

hi
how to delete client after complete request ??

BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure();

[devyte]

The answer is in your question: delete client;