The malloc function seems to have problems under Arduino

[HalfSweet]

Board

ESP32 DEV MOUDLE

Device Description

In the arduino environment, I can't seem to request more than 120k bytes of memory space using the malloc function, either using the malloc function, the heap_caps_malloc function or the c++ style new keyword. And trying to request more memory with the new keyword will cause the whole system to crash and reboot. But when I tested in esp idf environment, I didn't find this problem

Hardware Configuration

The development board is not connected to any peripheral

Version

latest master (checkout manually)

IDE Name

PlatformIO and Arduino IDE

Operating System

Windows11

Flash frequency

80Mhz

PSRAM enabled

no

Upload speed

921600

Description

I want to be able to request more than 120k of memory on the arduino platform, but I can't do it with malloc and heap_caps_malloc, and using the new keyword causes the system to crash and reboot. And this is perfectly supported on the esp idf.

Sketch

void setup()
{
    // put your setup code here, to run once:
    Serial.begin(115200);
    log_i("before malloc,free heap:%.1fkb", (float)esp_get_free_heap_size() / 1024.0);

    void *ptr = malloc(120 * 1024);
    if (ptr == nullptr)
    {
        log_i("malloc failed");
    }
    else
    {
        log_i("after malloc,free heap:%.1fkb", (float)esp_get_free_heap_size() / 1024.0);
        heap_caps_print_heap_info(MALLOC_CAP_INTERNAL);
    }
    
    void *ptr2 = heap_caps_malloc(120 * 1024,MALLOC_CAP_8BIT);
    if (ptr2 == nullptr)
    {
        log_i("malloc failed");
    }
    else
    {
        log_i("after malloc,free heap:%.1fkb", (float)esp_get_free_heap_size() / 1024.0);
        heap_caps_print_heap_info(MALLOC_CAP_INTERNAL);
    }

    uint8_t *ptr3 = new uint8_t[120 * 1024];
    if (ptr3 == nullptr)
    {
        log_i("malloc failed");
    }
    else
    {
        log_i("after malloc,free heap:%.1fkb", (float)esp_get_free_heap_size() / 1024.0);
        heap_caps_print_heap_info(MALLOC_CAP_INTERNAL);
    }
}

void loop()
{
}

Debug Message

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1344
load:0x40078000,len:13516
load:0x40080400,len:3604
entry 0x400805f0
[

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[Ouss4]

is PSRAM enabled and part of the heap?

The size you are trying to allocate is at the limit of the largest free node in the internal memory. You can see in the IDF log its 183 KB. This comes right after static allocation, so if Arduino's static allocation is larger, you won't have a free node that satisfies that amount of memory.

[HalfSweet]

I did not enable PSRAM.I can allocate multiple heap spaces, even if the total size is more than 120k, successfully. My problem is simply that I can't allocate too much space at once (this limit is around 110k)

[Ouss4]

I can allocate multiple heap spaces, even if the total size is more than 120k, successfully. My problem is simply that I can't allocate too much space at once (this limit is around 110k)

You can allocate as much memory as you want as long as there are free nodes that satisfy your request.
When you do a large allocation, there has to be one free node that will satisfy that. You can't combine memory from different regions.

From your first screen, the IDF heap looks like this:

6KB
183KB
126KB
86KB
32KB

These are the different heap regions in the internal memory. malloc can return a pointer to any of these as long as the requested size can be honored. But you can't combine them.

i.e

malloc(100 * 1024); // Can be honored from the second 183KB
malloc(100 * 1024); // Can be honored from the third 126Kb

but

malloc(200 * 1024); // There is no free node that can honor this request.

[HalfSweet]

But I can allocate so much memory in esp-idf environment, just can't do it under arduino. Is this because the arduino platform solidifies certain parameters under precompilation? Can I use it after recompiling by changing the parameters?

[Ouss4]

But I can allocate so much memory in esp-idf environment, just can't do it under arduino

How much? I see you are allocating 170KB. From your screen you shouldn't be able to allocate more than 183KB without PSRAM.

Again, it doesn't mean you run out of memory, it's just there is no free node that satisfies your request.
i.e:

malloc(200 * 1024); // Fails

But

malloc(50 * 1024); // Succeeds
malloc(50 * 1024); // Succeeds
malloc(50 * 1024); // Succeeds
malloc(50 * 1024); // Succeeds

[lbernstone]

I think the real question here is why there is this fragmentation (there appears to be 3 blocks ~ 100k-120k available on an ESP32), and is it possible to align whatever these default allocations are so that they leave a larger memory space after initArduino?

[Ouss4]

I think the real question here is why there is this fragmentation

This "fragmentation" is there because of the ROM bootloader that uses some regions in the middle of DRAM. You can read more here: https://github.com/espressif/esp-idf/blob/master/components/heap/port/esp32/memory_layout.c#L138-L159

Note that the situation has been improved in ESP32-S2 and subsequent chips. In these chips the ROM bootloader uses a region at the end of DRAM and leaves the rest for the application.

and is it possible to align whatever these default allocations are so that they leave a larger memory space

In the screens above you can see that the addresses are not contiguous, so we can't merge them.

In summary, if a large allocation is required, that should not exceed the value reported as "largest_free_block".

[SuGlider]

As @Ouss4 pointed out, this is not related to Arduino, thus there is nothing that can be done to change it for the ESP32.
Think about it as a sort of system/hardware limitation for this SoC.
C3/S2/S3 have different structure for heap segmentation/banks.

Usually when the Arduino application needs a lot of Heap space, it will malloc() several chunks in an array of byte pointers, for instance.

Example:
The application needs to address a LCD with 480x320pixels and each pixel has 16 bits (RGB565).
This is 480x320x2 = 307,200 bytes.
The application will have a screen structure like this: uint16_t *screenLine[320], and then, malloc(480 * sizeof(uint16_t)) for each pointer in the array.
By this way, it will work fine and use almost all the available heap of the ESP32 for the LCD pixels.

[SuGlider]

Closing it. Fell free to reopen it if necessary.