HTTPS failure with low free heap memory?

[fabianoriccardi]

Hardware:

Board: Lolin D32
Core Installation/update date: commit: 884e417 06/dec/2018
IDE name: Arduino IDE
Flash Frequency: 40Mhz
PSRAM enabled: no
Upload Speed: 921600
Computer OS: Windows 10

Description:

I was testing the HTTPS functionality and it works perfectly. Last week I've added the BLE library, but it cause the failure of HTTPS calls. Actually Bluetooth is just instantiated BLEDevice::init(""); and never used, but it seems enough to make fail https calls. I think that the problem is the huge amount of ram stole by BLE library, but I'm not so confident about that.

This is what I get on serial port and it represent the common behaviour, everything works fine (even if there is that socket error) , you can see the huge (normal) amount of free heap:

[HTTPS] sending POST to myUrlAddress ...
Free Heap: 169876
[D][HTTPClient.cpp:265] beginInternal(): host: myHostAddress port: 443 url: myUrl
[E][WiFiClient.cpp:236] setSocketOption(): 1006 : 9
[I][ssl_client.cpp:151] start_ssl_client(): WARNING: Use certificates for a more secure communication!
[D][HTTPClient.cpp:977] connect():  connected to myHostAddress:443
[D][HTTPClient.cpp:1102] handleHeaderResponse(): code: 200
[D][HTTPClient.cpp:1105] handleHeaderResponse(): size: 45
[HTTPS] POST... code: 200
[D][HTTPClient.cpp:1239] writeToStreamDataBlock(): connection closed or file end (written: 45).
[D][HTTPClient.cpp:358] disconnect(): tcp stop

===========
{"content":null,"message":null,"result":"OK"}
===========
[HTTPS] end
[D][HTTPClient.cpp:369] disconnect(): tcp is closed

This is what I get when BLE is initialized (NOTE low free heap quantity):

[HTTPS] sending POST to myUrlAddress ...
Free Heap: 44952
[D][HTTPClient.cpp:265] beginInternal(): host: myHostAddress port: 443 url: myUrl
[E][WiFiClient.cpp:236] setSocketOption(): 1006 : 9
[I][ssl_client.cpp:151] start_ssl_client(): WARNING: Use certificates for a more secure communication!
[E][ssl_client.cpp:33] handle_error(): SSL - Memory allocation failed
[E][ssl_client.cpp:35] handle_error(): MbedTLS message code: -32512
[E][WiFiClientSecure.cpp:118] connect(): start_ssl_client: -32512
[D][HTTPClient.cpp:973] connect(): failed connect to myHostAddress:443
[W][HTTPClient.cpp:1262] returnError(): error(-1): connection refused
[HTTPS] POST... failed, error: connection refused
[HTTPS] end
[D][HTTPClient.cpp:369] disconnect(): tcp is closed

My questions are:

• are there tricks to save RAM memory taken by ble library?
• how many bytes takes an HTTPS call? Does it depends on message size?

[chegewara]

MbedTLS message code: -32512
mbedtTLS cant start because lack of heap. You need about 70-ish kB free heap i think to even start with mbedTLS.

[fabianoriccardi]

Thank, actually I've tried to allocate memory to see this limit. It seem that it start to fails if free heap is about 100k.

the code in the loop is:

  HTTPClient http;
  byte* buf = (byte*)malloc(step);
  if(buf==NULL) Serial.println("Error");
  i+=step;
  Serial.print("[HTTP] begin...\n");
  Serial.println(ESP.getFreeHeap());
  // This call makes a HTTPS call!
  http.begin("myurl.com", ca); 
  Serial.println(ESP.getFreeHeap());
  Serial.print("[HTTP] GET...\n");
  int httpCode = http.GET();
  if(httpCode > 0) {
    Serial.printf("[HTTP] GET... code: %d\n", httpCode);
    if(httpCode == HTTP_CODE_OK) {
        String payload = http.getString();
        Serial.println(payload);
    }
  } else {
    Serial.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
  }

  http.end();

The result is:

[HTTP] begin...
Free Heap: 104928
[HTTP] GET...
[HTTP] GET... code: 200
Hello World!

[HTTP] begin...
Free Heap: 94912
[HTTP] GET...
[HTTP] GET... failed, error: connection refused

I've repeated this test multiple times changing the memory allocated at each iteration: 5k/10k/20k were tested leading to the same result.

[czuvich]

I see the same issues with a WROVER module. See my referenced thread I posted in ESP32 forums. I have almost 4MB of heap available, but it says memory allocation failed. Any help is greatly appreciated.

https://www.esp32.com/viewtopic.php?t=8475

[chegewara]

Only because you have 4MB free heap it does not means that all can be used in all situations of for all components.
start_ssl_client: -32512 means that mbedTLS does not have enough memory to start, most likely you have +/-70kB internal heap free (or less) and most reported heap is from spiram.
You can ask @tobozo how to make use of spiram, he told me today he resolved problems with ble scanner by using spiram.

[tobozo]

@czuvich Depending on the SDK version ESP.getFreeHeap() will return values calculated differently and may deceive. Until this is stabilized, I'll keep using heap_caps_get_free_size(MALLOC_CAP_INTERNAL) instead which sticks to the historical behaviour.

@fabiuz7 using http.getString() multiple times in a loop will eventually consume all available heap, Arduino Strings are evil :D

I've had the same issues in my own project, and upon @chegewara's suggestion I moved all String() statements to const char* and could eventually use ps_malloc instead of malloc.

Consequently you'll have to use streaming with HTTP.

// get this outside your loop !
HTTPClient http;
http.setReuse(true);
// assuming 'step' is a power of 2 (e.g. 512, 1024, 2048)
byte* buff = (byte*)ps_malloc(step); // only malloc/ps_malloc during setup() !!
// (...)

// modify your loop as follows:
WiFiClient * stream = http.getStreamPtr();
// (...)
if(httpCode == HTTP_CODE_OK) {
  while(http.connected() && (len > 0 || len == -1)) {
    // get available data size
    size_t size = stream->available();
    if(size) {
      // read up to 'step' bytes
      int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
      // write it to SD, Serial or any other resource
      Serial.write(buff, c);
      //SD.write(buff, c);
      if(len-c >= 0) {
        len -= c;
      }
    }
  }
}

If you're using a SD Card there's also the possibility to SD-load certificate data directly to the psram: bonus in sketch size and maintainability as it unlocks the possibility to have the certificate stored in psram.

[czuvich]

Thanks for the helpful responses!

I am getting the error on startup of the chip. If I call BLEDevice::init() followed by the HTTPS call it reports that error. If I omit BLEDevice::init() it’s fine.

So, how could I get that working in Arduino (without omitting one or the other)? Is the error reported to be expected since it sounds like it forces internal allocation? I’m looking at
CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST

Perhaps? I assume to get that working I’d need to recompile the SDK libs right?

[tobozo]

@czuvich maybe BLEDevice::getScan()->stop() and BLEDevice::getScan()->clearResults() before calling HTTPS could help free some space?

Ongoing scans can eat up to 256 bytes per detected device so it can affect your heap differently depending on how long your scan is.

[chegewara]

@czuvich Could you try revert order, first https calls and then ble?

[czuvich]

@chegewara Switching order allows me to execute WiFi and then BLE scans; however, I can't run WiFi afterwards. So, here's what I'm seeing:

BLE then WiFi (fails)
WiFi then BLE (success)
WiFi then BLE then WiFi (fails)

Shouldn't mbed use spiram in the Arduino environment? If not, I'm wondering if there needs to be some other configuration. Here's the full log from the other thread.

12:59:27.190 -> [V][ssl_client.cpp:53] start_ssl_client(): Free heap before TLS 4182376
12:59:27.190 -> [V][ssl_client.cpp:55] start_ssl_client(): Starting socket
[V][ssl_client.cpp:88] start_ssl_client(): Seeding the random number generator
12:59:27.684 -> [V][ssl_client.cpp:97] start_ssl_client(): Setting up the SSL/TLS structure...
12:59:27.684 -> [I][ssl_client.cpp:121] start_ssl_client(): WARNING: Use certificates for a more secure communication!
12:59:27.684 -> [V][ssl_client.cpp:145] start_ssl_client(): Setting hostname for TLS session...
12:59:27.684 -> [E][ssl_client.cpp:33] handle_error(): SSL - Memory allocation failed
12:59:27.684 -> [E][ssl_client.cpp:35] handle_error(): MbedTLS message code: -32512
12:59:27.717 -> [E][WiFiClientSecure.cpp:109] connect(): start_ssl_client: -32512
12:59:27.717 -> [V][ssl_client.cpp:211] stop_ssl_socket(): Cleaning SSL connection.
12:59:27.717 -> [D][HTTPClient.cpp:846] connect(): failed connect to [my backend server]:443
12:59:27.717 -> [W][HTTPClient.cpp:1124] returnError(): error(-1): connection refused
12:59:27.717 -> [I][HTTPAdapter.cpp:135] sync(): Received HTTP response [-1].
12:59:27.750 -> [W][HTTPClient.cpp:1124] returnError(): error(-4): not connected

[tobozo]

@czuvich on the Wrover kit + SDK-rc1 I had to run psramInit() manually before being able to use it, which is odd as it's supposed to be automatically initialized.

[czuvich]

@tobozo Thanks for suggestion, but that doesn't seem to work either. I am currently running v3.2-dev-39-gaaf12390 with the WROVER DEV Kit 4.1.

I think I may open up my own github issue for this problem. I don't think it's anything on my end, but I'll try to create a repo. Just curious, are you using the WROVER DEV Kit for your project? Did you end up getting your own board designed with a WROVER chip?

I am currently using the WROVER Dev kit; however, I don't need all of the extra "stuff".

[tobozo]

@czuvich I have the Wrover Dev Kit v4.1 too (playing with this project) without doing simultaneous WiFi/BLE though.

Not sure what the v3.2 you mention is, I'm on the arduino-esp32-1.0.1-rc2 of Arduino IDE 1.8.7 and using the 1.22.0-80-g6c4433a-5.2.0 xtensa.

[czuvich]

Arduino 1.0.1? Either that's a typo or you may want to update :) I'm on Arduino IDE 1.8.7. The SDK version I printed out was fromesp_get_idf_version(). Are you using a custom build for the ESP SDK, or are you using the Boards Manager build?

Also, thank you for linking your project. I can't release the source for mine, but I'm building a BLE->WiFi gateway using the WROVER dev kit for now. I'm leveraging the SD Card and an external RTC (DS3231).

[tobozo]

@czuvich In my project I have a use case for downloading HTTPS content between two BLE scans, what should I do in my scan loop to replicate your issue ?

btw on my build the RTC module wants to run on core 1 or it dies miserably.

[czuvich]

@tobozo I setup 2 tasks. One task is for BLEScanning and the other task is for posting data to an HTTPS backend. For the repo, I was going to use only 1 task and call BLEDevice::init() in setup since I am getting an error without scanning. I am using the following base class for executing tasks:

https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/Task.h

So something like this in the setup() function:

BLEDevice::init();
HttpPostTask task2 = new HttpPostTask(); // inherits from Task.h, feel free to configure stack size

Then, inside of the HttpPostTask.run(void* data), I use the WiFi and HttpClient.post() methods to establish a WiFi connection and POST data. I was going to just post something to https://www.google.com just to set it up.

That should be enough to replicate the issue. I've tried different stack allocations, but it always fails if BLEDevice::init() is invoked first.

To work around the issue, I just "schedule" the HTTPS POST. If the program detects that it's time to run the POST, it resets the board, executes the POST first, and resets the board again. I keep a persistent variable that tells me the next time to execute the POST (hence the RTC).