Memory allocation issues (IDFGH-2698)

[enricop]

Dear support,
we're facing some memory allocation issues with our ESP-WROVER based BLE Mesh IoT gateway.

We're currently trying to assess the limits of our system by increasing the number of mesh nodes and devices connected to our ESP32.
Everything is going pretty good so far, except for a few sporadic crashes, whose origin we determined to be related to our chronical shortage of internal memory.

We've seen several times this pattern occurring:

1. With big plants comprised of a large number of devices (> 40), free internal DRAM tends to hover around ~25KiB free, probably highly fragmented. We've already moved everything humanly possible to external SPI-RAM, including setting CONFIG_SPIRAM_USE_MALLOC=y and SPIRAM_MALLOC_ALWAYSINTERNAL=256, plus writing custom allocators for our C++ code that forces the usage of heap_caps_malloc with MALLOC_CAP_SPIRAM.
2. Some parts of our code then try to operate on the NVS, calling nvs_flash_init_partition. This triggers the loading of pages from the NVS, which in turns invokes spi_flash_read;
3. spi_flash_read invokes esp_flash_read, which returns ESP_ERR_NO_MEM (0x101);
4. The application calls abort() after printing "unexpected SPI flash error code: 101" (aka ESP_ERR_NO_MEM).

We have a strong suspicion that this error could origin from the following code:

esp_flash_api.c:

esp_err_t IRAM_ATTR esp_flash_read(esp_flash_t *chip, void *buffer, uint32_t address, uint32_t length)
{
   [..]
    //when the cache is disabled, only the DRAM can be read, check whether we need to receive in another buffer in DRAM.
    bool direct_read = chip->host->supports_direct_read(chip->host, buffer);

    uint8_t* temp_buffer = NULL;
    
    if (!direct_read) {
        uint32_t length_to_allocate = MAX(MAX_READ_CHUNK, length);
        length_to_allocate = (length_to_allocate+3)&(~3);
        temp_buffer = heap_caps_malloc(length_to_allocate, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
        ESP_LOGV(TAG, "allocate temp buffer: %p", temp_buffer);
        if (temp_buffer == NULL) return ESP_ERR_NO_MEM;
    }

    [..]

It seems like that due to the conflict between SPI RAM access and NVS reads if the destination buffer is not in DRAM (and given we've set SPIRAM_MALLOC_ALWAYSINTERNAL=256, almost all big allocations are in SPIRAM) the function attempts to allocate an at least 16KiB temporary buffer from internal memory; this operation always fails because of our 25k of free internal memory are very fragmented.

Do you have any tips about how to avoid this issue? We've already tried freeing up as much internal memory as possible. In particular, we would like to know if you can recommend some methods to prevent or at least mitigate memory fragmentation.

Thank you in advance for your support.

[negativekelvin]

esp-idf/components/spi_flash/esp_flash_api.c

Line 501 in caa59da

uint32_t length_to_allocate = MAX(MAX_READ_CHUNK, length);

esp-idf/components/spi_flash/esp_flash_api.c

Line 519 in caa59da

uint32_t length_to_read = MIN(MAX_READ_CHUNK, length);

Should be MIN not MAX

[enricop]

Hi @negativekelvin , thank you.

Could this be fixed in release/v4.0 branch?

[mahavirj]

@enricop We will fix this ASAP and backport for previous applicable release branches as well.

[nacpem]

+1 opening this issue

following all the threads which have led me here

W (198897) wifi:alloc eb len=24 type=3 fail, heap:0

W (198897) wifi:m f null

[projectgus]

@rebeliousconformist This doesn't seem to be the same as the issue reported above. The log message indicates that Wi-Fi is failing to allocate 24 bytes from internal memory because there is no free heap remaining ("heap:0" indicates zero bytes available).

If you think this is an issue in ESP-IDF, rather than the firmware is running out of free memory, then please open a new issue with the extra details shown in the issue template.

If you suspect a memory leak somewhere, there are some tips for heap memory debugging here: https://docs.espressif.com/projects/esp-idf/en/release-v4.1/api-reference/system/heap_debug.html#

[nacpem]

@projectgus yes. ill report them if i do find any bugs. The dearth of heap memory has occured after updating to the latest commit. In the past it has so occured that i have spent days debugging and after i found the bug, it was fixed in the next commit. Hence my post to make sure im not running blind.
My best Regards.