Everything is trying to eat my internal ram and not my external!

[zekageri]

Board

ESP32 Wrover

Device Description

ESP32-Wrover-E ( 8mb psram, 16mb flash )

Hardware Configuration

Version

latest master (checkout manually)

IDE Name

Platformio

Operating System

Windows10

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

I want to create an https connection every hour or so.
I noticed that my internal ram is not sufficent for it.
I tought that malloc will first try to allocate from psram and if it fails, it allocates from internal.
But it does not the case.

Here is how my esp is on RAM right now

// External
[SYSTEM] - [EXT][1]
Free:   4075727
Min:    4075727
Max:    4075727
Contig: 3997684
Diff:   808

// Internal
[SYSTEM] - [HEAP][1]
Free:   78680
Min:    78680
Max:    78680
Contig: 38900
Diff:   78680

I have only 38900kb of contignous ram left in my internal!! How is this possible?
My psram is working fine and i try to allocate everything on it in my sketch.
When i try to do a HTTPClient https; it just fails because there is not enough internal ram.

I even call heap_caps_malloc_extmem_enable(0); so everything should be allocated from external ram.

At least the doc says this:

void heap_caps_malloc_extmem_enable(size_t limit)
Enable malloc() in external memory and set limit below which malloc() 
attempts are placed in internal memory. When external memory is in use,
the allocation strategy is to initially try to satisfy smaller allocation requests
with internal memory and larger requests with external memory.
This sets the limit between the two, as well as generally enabling allocation in external memory.


Parameters:
limit – Limit, in bytes.

Sketch

void setup(){
    Serial.begin(115200);
    // What can i do more so things use my psram and not my internal???????
    heap_caps_malloc_extmem_enable(0);
esp_err_t error = heap_caps_register_failed_alloc_callback([](size_t requested_size, uint32_t caps, const char *function_name) {

    Serial.printf("[SYSTEM] - %s was called but failed to allocate %d bytes with 0x%X capabilities. \n",
                     function_name, requested_size, caps);

    });
}

void loop(){
}

// Try to do an https request.

Debug Message

[619840][V][HTTPClient.cpp:252] beginInternal(): url: https://api.github.com/repos/{name}/{repoName}/releases/latest
[619841][D][HTTPClient.cpp:303] beginInternal(): protocol: https, host: api.github.com port: 443 url: /repos/{name}/{repoName}/releases/latest      
[619854][D][HTTPClient.cpp:598] sendRequest(): request type: 'GET' redirCount: 0

[619921][V][ssl_client.cpp:62] start_ssl_client(): Free internal heap before TLS 78680
[619921][V][ssl_client.cpp:68] start_ssl_client(): Starting socket
[619956][V][ssl_client.cpp:146] start_ssl_client(): Seeding the random number generator
[619957][V][ssl_client.cpp:155] start_ssl_client(): Setting up the SSL/TLS structure...
[619960][V][ssl_client.cpp:178] start_ssl_client(): Loading CA cert
[619971][V][ssl_client.cpp:254] start_ssl_client(): Setting hostname for TLS session...
[619974][V][ssl_client.cpp:269] start_ssl_client(): Performing the SSL/TLS handshake...
[SYSTEM] - heap_caps_calloc was called but failed to allocate 1051 bytes with 0x804 capabilities.
[620166][E][ssl_client.cpp:37] _handle_error(): [start_ssl_client():273]: (-10368) X509 - Allocation of memory failed
[620166][E][WiFiClientSecure.cpp:144] connect(): start_ssl_client: -10368   
[620172][V][ssl_client.cpp:321] stop_ssl_socket(): Cleaning SSL connection. 
[620179][D][HTTPClient.cpp:1163] connect(): failed connect to api.github.com:443
[620186][W][HTTPClient.cpp:1483] returnError(): error(-1): connection refused
[Updater] - Failed to check new version. HTTP CODE: -1
[620452][D][HTTPClient.cpp:408] disconnect(): tcp is closed

[620452][V][ssl_client.cpp:321] stop_ssl_socket(): Cleaning SSL connection.

Other Steps to Reproduce

How can i force everything to use my PSRAM??
There is 4MB of psram just sitting there without any use!!

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[zekageri]

What i have done so far:

At setup: heap_caps_malloc_extmem_enable(64);

In platformio.ini

build_flags =   
    -DBOARD_HAS_PSRAM
    -mfix-esp32-psram-cache-issue
    -mfix-esp32-psram-cache-strategy=memw
    -DCORE_DEBUG_LEVEL=0
    -std=gnu++17
    -DCONFIG_ARDUINO_ISR_IRAM=0
    -DCONFIG_SPIRAM_USE_MALLOC=1

[lucasssvaz]

Could you confirm if this also happens in the regular Arduino IDE ? PlatformIO is not officially supported.

[zekageri]

I will check it because it is really frustrating. Is there a way in PIO to force at least the mbedtls to use psram?

[lucasssvaz]

I don't really know how PIO works. It might be better to directly ask that on their repo.

[lbernstone]

PSRAM != RAM. Stack memory is allocated from RAM. Interacting with hardware peripherals generally requires the data in and out of RAM (it can then be moved to psram afterwards). This includes the AES accelerator. Use this to track where your RAM is disappearing:

    const char* fmtMemCk = "Free: %d\tMaxAlloc: %d\t PSFree: %d\n";
    #define MEMCK Serial.printf(fmtMemCk,ESP.getFreeHeap(),ESP.getMaxAllocHeap(),ESP.getFreePsram())

Give some specific examples of where you expect data to go into psram, and we can help you make that happen.

[zekageri]

Thank you for the snippet. Specifically i need mbedtls to use my PSRAM instead of the internal ram. This is the most ram consuming part of my sketch.

[lbernstone]

Any piece of data that you want to be processed has to go through the RAM at some point. PSRAM is a holding location. The CPU moves data in and out of it quickly using DMA, so data in PSRAM shouldn't persist in SRAM, but it still has to be in the local registers when the cpu works on it.

[zekageri]

I see. So the only way for me to reduce a https call ram consumption is not to use it at all or use unsecured http. Which i cant since i want to use it for firmware upgrades. :/

[lbernstone]

It needs working space. I would have to look at the internals of HTTPClient, but you can stream data in and out of the tls function into psram. That way, you should only need 8K (4K in/4K out) for the hardware buffer, and then it will move it into the psram for storage. You are getting to the level of complexity in your code that I would suggest you think about using the IDF functions, or modifying the builtins, so that you have more control over the memory management. C gives you fine-grained control that C++, and especially Arduino, just don't have.

[zekageri]

Oh that would be ideal! I will look into that. If you could give me a head start on that psram magic i would be really greatful.

[lbernstone]

There's only one trick, brother. It's called a pointer. If I allocate a pointer, and it is over 4k (or whatever size you set with the function above), it is going to map that pointer to psram (you can specify in heap_caps_malloc() what memory you want to use). If I pass that object around in functions, it is going to make a whole bunch of unnecessary copies. If I pass the pointer, that is a single word.
If I define this function as a String, it would leak a ton of memory, and a lot of that would be small fragments. By passing a pointer, I can control how that memory is allocated much better, and then I manually clean up the complicated structure that I have built.

[zekageri]

Yeah, i knew that. Sorry my question was a bit misleading. I use psram buffers all over my code. In fact I have almost no normal buffer allocations just small arrays. I try to use my 4mb psram in all cases. Thats why i have a problem. Thats why i want to know how to setup my code properly so other libraries will use my psram too instead of the internal. If i could set something so the http requests would use psram that would be good. I wanted to know about these IDF functions you talking about.