maximum ram usable is 96000

[manecolooper]

maximum ram usable by my sketch is 96000 , i defined it as:

#define MAX_DELAY 96000
byte sDelayBuffer0[MAX_DELAY];

if i define more than that , error is

region `dram0_0_seg' overflowed by 24064 bytes ( i set MAX_DELAY to 128000)

compiler reports that there is more ram free, how can i use it?

Serial.println(ESP.getFreeHeap());

returns 127336

i also tested malloc , new and dividing the memory needed into two segments, but crashes the same ( reset loop ) when sum of both segments exceed 96000

[beegee-tokyo]

I have no proof, but I guess that FreeRTOS reserves memory for other tasks and temporary variables and doesn't allow you to reserve all of the free RAM.

[manecolooper]

Is true, that's why of the 380 k of ram you have less than 300 left, but i can not go past 96000

[ZinggJM]

This used to work in an earlier version I had installed on 24.7.2017, fortunately I kept it on my development notebook, but on the updated version installed on my test notebook I now have this issue. I need big RAM for buffer for multiple big e-paper displays!

[manecolooper]

ZinggJM any solution to use it today? thanks!

[ZinggJM]

Someone should check if this issue also exists in esp-idf.
And Espressif might tell us if this is a parameter in some configuration file of the compiler.

[manecolooper]

ZinggJM thanks for your help, i wonder which size were you able to achieve in that versión of the compiler

[shirish47]

@ZinggJM Hi , I also once tested ESP32 with E-paper of older Kindle I think I was able to use around 800x500 px of 800x600 px .. but currently my problem is different I want to store accelerometer data in to array or buffers (xyz all 2 byte data ) and I am only able to store 9600 Samples(960032=57600 bytes ).

my code also has AWS IoT Wifi manager on it.
this is what arduino shows up.

Sketch uses 852426 bytes (65%) of program storage space. Maximum is 1310720 bytes.
Global variables use 105452 bytes (35%) of dynamic memory, leaving 189460 bytes for local variables. Maximum is 294912 bytes.

this means I have 65% ram(189460 bytes) left.
how could I use that 65% ? if I increase samples more than this 9600 It crashes.

[shirish47]

when is external RAM support be part of arduino core? or do we have it? what is the challenge for having SPI RAM in arduino core?

[stickbreaker]

@shirish47 Instead of one monolithic buffer try multiple buffers. Depending on your design, burst sampling or continuous upload, you may be able to process smaller buffers continuously.

Chuck.

[shirish47]

Ya At first the design was like one core picks the data puts in to 64 sample message and sends via queue to wifi (core 2) and publish over to AWS IoT .. but the queue size was limited to 100 such messages so after it was filled up we would have a blockage and then after like 2 seconds(sometimes 500milliseconds) it would resume but was also blocking the Core 1 that was getting 1600 samples rate data out of sensor. and that block would corrupt(delays between samples) ..we want continuous 30 seconds samples at 1600 rate.

[shirish47]

Also, can we use the SPI RAM on ESP Wrover-32 as SPI RAM for storage? like directly access it with SPI.h and just store data on to it. not like allocating memory from it.

[shirish47]

can someone test this?link I can't believe I can store so much data. I don't believe this. even esp32 does not have that much of RAM.
Please please please explain this. I am not sure what is this why this runs. and works

[shirish47]

https://esp32.com/viewtopic.php?f=19&t=5308

[atsteich]

can someone test this?link I can't believe I can store so much data. I don't believe this. even esp32 does not have that much of RAM.
Please please please explain this. I am not sure what is this why this runs. and works

@shirish47 The thing in your code is, that the data are never set to any value. Therefore the Compiler can do some optimization, and the ram for the array will never been allocated. As soon as u start filling the array with numbers you will get an compile error "region `dram0_0_seg' overflowed by 63501768 bytes". (You also need to do anything else with the numbers, e.g. printing them...)

[atsteich]

So is there still no solution found to the original problem? Why is the esp reporting a free Heap of (in my case) almost 200k, but still crashing? What would be the "correct" way to determine the left memory for variables? (For example if I want to buffer something as long as possible, but stop before crashing)..

[chegewara]

I dont want to read all posts, but OP. The problem can be that you have 127000 bytes free but its fragmented and maximum heap in one fragment that you can use to allocate is about 100000 bytes.
This is heap info from my esp-idf app:

Heap summary for capabilities 0x00001000:
  At 0x3ffb2730 len 15448 free 0 allocated 15244 min_free 0
    largest_free_block 0 alloc_blocks 43 free_blocks 0 total_blocks 43
  At 0x3ffaff10 len 240 free 0 allocated 172 min_free 0
    largest_free_block 0 alloc_blocks 9 free_blocks 0 total_blocks 9
  At 0x3ffb6388 len 7288 free 0 allocated 7164 min_free 0
    largest_free_block 0 alloc_blocks 23 free_blocks 0 total_blocks 23
  At 0x3ffb9a20 len 16648 free 0 allocated 16248 min_free 0
    largest_free_block 0 alloc_blocks 92 free_blocks 0 total_blocks 92
  At 0x3ffca010 len 90096 free 236 allocated 89084 min_free 0
    largest_free_block 104 alloc_blocks 178 free_blocks 8 total_blocks 186
  At 0x3ffe0440 len 15072 free 0 allocated 15016 min_free 0
    largest_free_block 0 alloc_blocks 6 free_blocks 0 total_blocks 6
  At 0x3ffe4350 len 113840 free 79792 allocated 33636 min_free 61348
    largest_free_block 71972 alloc_blocks 82 free_blocks 13 total_blocks 95
  Totals:
    free 80028 allocated 176564 min_free 61348 largest_free_block 71972

In bottom line you can see that i have 80028 bytes free but largest free block is only 71972, so i cant exceed it with one allocation.

[atsteich]

@chegewara No, this is not to be the problem. I have not tried with esp-idf, but: I can allocate e.g. 1 very big array with 23600 integer values (in total 94400 bytes). The remaining freeHeap is reported bigger than 200Kbytes. As soon as i allocate a second small array with e.g. 20 integer values, the chip crashes.
I really don't get it why this is happening...

[chegewara]

Do you have example code for that?

[atsteich]

@chegewara

#include <Arduino.h>
#include <Esp.h>

int var1[23600];
int var2[30];


void setup() {
  Serial.begin(115200);
  delay(50);
  Serial.printf("FreeHeap: %i bytes\n", ESP.getFreeHeap());
  Serial.printf("Size of var1: %i bytes\n", sizeof(var1));
  var1[0] = 1;
  var2[0] = 10;
}

void loop() {
    Serial.println("main");
    delay(5000);
}

This code will result in an rebooting loop, while it is ok as soon one removes var2....

[chegewara]

Thanks. I did some test and this is what i found (maybe @copercini or @me-no-dev can explain it better):

• after allocating those 2 array variables you still have plenty of free heap, but:
• both variables are global which mean they are allocated in dram.bss which is part of heap but cant use all heap space; here is example from esp-idf startup:

D (712) memory_layout: Available memory region 0x4009e000 - 0x400a0000
I (718) heap_init: Initializing. RAM available for dynamic allocation:
D (726) heap_init: New heap initialised at 0x3ffaff10
I (731) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
D (737) heap_init: New heap initialised at 0x3ffb6388
I (742) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM
D (748) heap_init: New heap initialised at 0x3ffb9a20
I (753) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM
I (759) heap_init: At 3FFBDB5C len 00000004 (0 KiB): DRAM
D (765) heap_init: New heap initialised at 0x3ffcc840
I (770) heap_init: At 3FFCC840 len 000137C0 (77 KiB): DRAM
I (777) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (783) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM

Most likely we are talking here about last D/IRAM segment.
If you move var2 to setup or loop function there will be no more crash even if var2 is 20000*4 bytes in size:

#include <Arduino.h>
#include <Esp.h>

int var1[23250];


void setup() {
  Serial.begin(115200);
  delay(50);

  var1[0] = 1;
  Serial.printf("Size of var1: %i bytes\n", sizeof(var1));
int var2[20000];

  var2[0] = 10;
  Serial.printf("Size of var2: %i bytes\n", sizeof(var2));
      heap_caps_print_heap_info(MALLOC_CAP_INTERNAL);
}

void loop() {
    Serial.println("main");
    delay(5000);
}

Logs:

15:11:02.652 -> Size of var1: 93000 bytes
15:11:02.652 -> Size of var2: 80000 bytes
15:11:02.652 -> Heap summary for capabilities 0x00000800:
15:11:02.652 ->   At 0x3ffb0000 len 65536 free 56072 allocated 9272 min_free 56072
15:11:02.652 ->     largest_free_block 56072 alloc_blocks 9 free_blocks 1 total_blocks 10
15:11:02.652 ->   At 0x3ffae6e0 len 6192 free 6092 allocated 36 min_free 6092
15:11:02.652 ->     largest_free_block 6092 alloc_blocks 1 free_blocks 1 total_blocks 2
15:11:02.652 ->   At 0x3ffaff10 len 240 free 0 allocated 180 min_free 0
15:11:02.652 ->     largest_free_block 0 alloc_blocks 1 free_blocks 1 total_blocks 2
15:11:02.685 ->   At 0x3ffda1d0 len 24112 free 7556 allocated 15652 min_free 1508
15:11:02.685 ->     largest_free_block 4980 alloc_blocks 53 free_blocks 3 total_blocks 56
15:11:02.685 ->   At 0x3ffe0440 len 15296 free 15248 allocated 0 min_free 15248
15:11:02.685 ->     largest_free_block 15248 alloc_blocks 0 free_blocks 1 total_blocks 1
15:11:02.685 ->   At 0x3ffe4350 len 113840 free 113792 allocated 0 min_free 113792
15:11:02.685 ->     largest_free_block 113792 alloc_blocks 0 free_blocks 1 total_blocks 1
15:11:02.685 ->   At 0x4008a6ac len 88404 free 88356 allocated 0 min_free 88356
15:11:02.720 ->     largest_free_block 88356 alloc_blocks 0 free_blocks 1 total_blocks 1
15:11:02.720 ->   Totals:
15:11:02.720 ->     free 287116 allocated 25140 min_free 281068 largest_free_block 113792
15:11:02.720 -> main

I hope its clear to you now.

[atsteich]

@chegewara thanks for testing, i found exactly the same today :D

Is there any reason why global variables are only stored in this part of the ram? And why is there no compile warning (or error), if I increase var2 (in my example) to e.g. the same size than var1, i DO get an compile error (dram0.bss will not fit in region dram0_0seg) but if it is just 'a little bit to big' I dont get one... Is this because of stack size?

[chegewara]

Actually there is compile error, just you have to create one variable that exceeds 111kB.

I dont know why you dont have warnings or errors in your example, im not smart enough, sorry.

[atsteich]

But then the compile error should come not when they exceed 111 Kb, but earlier, or not?

[ZinggJM]

Is there a possibility to change the configuration of the RAM segments? Because with small processors you generally rather avoid huge heap allocation, but rather use global memory for buffers, as then the compiler should be able to complain if you use too much, in favor instead of runtime errors.

[ZinggJM]

How is the situation with D32 WROVER?

[atsteich]

I have no idea... but there is an easy workaround, which prevents you from run-time errors. Just declare a global pointer, and initialize the pointer in setup()... If you pass the setup, you should make it through all the time ;)

[chegewara]

How is the situation with D32 WROVER?

It is the same situation, because the same amount of RAM is in data .bss for global/static variables, but you have more heap for dynamic data. You can use workaround described by @atsteich. This will take only few bytes from .bss and variable will be stored in heap. For wrover in case of variables with size bigger than 15kB it should be in psram, so you could be allowed to init variable up to almost 4MB.

[ZinggJM]

So I have to learn how to use psram on wrover.
And I would need to use heap space for display buffer in my GxEPD2 library for ESP32 for users that want to attach multiple e-paper displays to the same processor board. This doesn't seem practical, as the same library is used from small AVR Arduino up to big STM32 or ESP32.
And there once was a version that allowed more than 96000 bytes, but maybe I didn't test if it worked.

[chegewara]

@ZinggJM I believe you dont have to learn how to use psram. esp-idf stack, so arduino stack too, is configured to allocate memory with malloc/calloc from psram for variables with size bigger than 15kB. Only difference is that in esp-idf we can change this value in menuconfig.

So, i believe that should be done by stack itself without your knowledge. Easy test to check if its true is to allocate big enough variable that cant fit internal memory, let say 300kB:

void setup() {
  Serial.begin(115200);
  delay(50);
  int var1[500000];
  var1[0] = 1;
  Serial.printf("Size of var1: %i bytes\n", sizeof(var1));
  heap_caps_print_heap_info(MALLOC_CAP_SPIRAM);
}

[ZinggJM]

But you are aware of the fact that var1 goes out of scope at the end of setup() ?
Same is true for any variable local to loop().

[chegewara]

@ZinggJM Yes, but the trick is to do this like that:

int* var0;  // <-- pointer to global variable that is stored in data .bss

void setup() {
  Serial.begin(115200);
  delay(50);
  int var1[500000];
var0 = var1; // <-- pointer to var0 will point to the first element of array
  var1[0] = 1;
  Serial.printf("Size of var1: %i bytes\n", sizeof(var1));
  heap_caps_print_heap_info(MALLOC_CAP_SPIRAM);
}

Now you can use var0 in like normal global array variable

[ZinggJM]

Ok, I will try this and call heap_caps_print_heap_info(MALLOC_CAP_SPIRAM); in loop(), to check my understanding of C/C++ and stack or heap allocation and variables getting free after going out of scope.
I know I could use var0 = new int[50000];

[atsteich]

@ZinggJM You are totally right! As soon as you are out of scope, the memory of local variables gets released. You still have a pointer, pointing to any memory address, but this ram is not longer allocated, and can (and will) be overwritten. This is one of the reasons why one should be very careful using pointers!

What i meant is allocating memory using malloc/calloc... See the following example:

#include <Arduino.h>
#include <Esp.h>

int *var1; //global Pointer!

void setup() {
    Serial.begin(115200);
    delay(2000);
    Serial.printf("\n\navailable heap befor allocating %i\n", ESP.getFreeHeap());
    Serial.printf("biggest free block: %i\n\”", heap_caps_get_largest_free_block(MALLOC_CAP_8BIT));
    var1 = (int*) calloc(25000, sizeof(int)); /*alocating memory, and save address
    in the global pointer*/
    for(int i = 0; i < 25000; i++){
        var1[i] = i; //doing just anything...
    }
    Serial.printf("\n\navailable heap after allocating %i\n", ESP.getFreeHeap());
    Serial.printf("biggest free block: %i\n\”", heap_caps_get_largest_free_block(MALLOC_CAP_8BIT));}

void loop() {
    Serial.printf("\n\navailable heap in main %i\n", ESP.getFreeHeap());
    Serial.printf("biggest free block: %i\n\”", heap_caps_get_largest_free_block(MALLOC_CAP_8BIT));    delay(10000);
}

As long as u do not call free(var1) the memory kepps allocated, and you can access it from global scope.

I am still interested if there is a reason why this is so, or if it is a bug. Maybe @me-no-dev can answer this?

[devrim-oguz]

I am still waiting for a solution to this problem. Got any updates?

[atsteich]

@adonizer Is my workaround not working for you?

[devrim-oguz]

Doesn't it cause any crashes during runtime? Also, can you use more than 96k this way?

[ZinggJM]

Is there a possibility to change the configuration of the RAM segments? Because with small processors you generally rather avoid huge heap allocation, but rather use global memory for buffers, as then the compiler should be able to complain if you use too much, in favor instead of runtime errors.

this got totally ignored

[atsteich]

@adonizer have you even read my last post?
Why should dynamically allocsting ram cause crashes? check out also this https://www.esp32.com/viewtopic.php?f=19&t=8130

[devrim-oguz]

Ok, I tried it now and it worked. Thank you.
However, I also agree with @ZinggJM . The ram should be reserved in compile time so the compiler could warn us about possible problems. Also, using pointers is not always a good idea. I didn't like the idea of dynamically reserving memory in runtime.

[me-no-dev]

I would say you better do some dynamic allocation, cuz this aint no computer with gigs of ram ;) comparison with other MCUs is also pointless. ESP32 is a bit different. I see no reason why you should statically allocate that much RAM and starve the rest of the system... why not do dynamic, check if NULL was returned and bail if so.

[me-no-dev]

Also to note. PSRAM in Arduino is NOT allocated using malloc and such. It's allocated using ps_malloc and such ;) Reason is that Arduinoers like to use ISRs for this and that and just the same as Flash is not to be used in ISR, same goes for PSRAM. If you are a pro and know what you are doing, you can spin Arduino and IDF together and set whatever you want.

[devrim-oguz]

But wouldn't the use of dynamic allocation on a system without "gigs of ram" cause the ram to be fragmented? Also, rest of the Arduino boards behave the same way, but ESP32 module on Arduino reports free memory while not being able to allocate on compile time. Maybe that should be fixed at least.

[me-no-dev]

the regions of memory used for RAM are not continuous (Ivan's explanation is linked above). Which means that you have some bytes here and some there and the memory allocator combines them to report free heap. There are other things too... like the compiler sees memory to be available but in reality by the time it goes to allocate it's gone. Still I would stay away from statically allocating such large chunks of memory. You can malloc them in setup and be done with it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This stale issue has been automatically closed. Thank you for your contributions.