Extra PSRAM on the heap is available for malloc() but not for new()

[Jeroen88]

Hardware:

Board: TTGO ESP32-WROVER-B T8 V1.8 ESP32 4MB PSRAM
Core Installation version: Latest git
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: yes, at least I THINK SO, by boards.txt (using board 'ESP32 Wrover Module'
Upload Speed: 921600
Computer OS: Ubuntu

Description:

I can allocate a large block (900,000 uint8_t) with malloc() but not with new()

Sketch: (leave the backquotes for code formatting)

//Change the code below by your sketch
void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  Serial.println("\nStarted\n");
  Serial.printf("Internal Total heap %d, internal Free Heap %d\n", ESP.getHeapSize(), ESP.getFreeHeap());
  Serial.printf("SPIRam Total heap %d, SPIRam Free Heap %d\n", ESP.getPsramSize(), ESP.getFreePsram()); 
  Serial.printf("ChipRevision %d, Cpu Freq %d, SDK Version %s\n",ESP.getChipRevision(), ESP.getCpuFreqMHz(), ESP.getSdkVersion());
  Serial.printf("Flash Size %d, Flash Speed %d\n",ESP.getFlashChipSize(), ESP.getFlashChipSpeed());
#if defined(BOARD_HAS_PSRAM)
  Serial.printf("Board has PSRAM\n");
#else
  Serial.printf("Board doesn't have PSRAM\n");
#endif

  Serial.printf("Test malloc()...\n");
  uint8_t *testMallocBuffer = (uint8_t*)ps_malloc(900000);
  if(testMallocBuffer != NULL) Serial.printf("NOT NULL\n"); else Serial.printf("NULL\n");
  free(testMallocBuffer);

  Serial.printf("Test new()...\n");
  uint8_t *testNewBuffer = new uint8_t[900000];
  if(testNewBuffer) Serial.printf("test NOT NULL\n"); else Serial.printf("test NULL\n");
  delete testNewBuffer;
  
}

void loop() {
  // put your main code here, to run repeatedly:

}

Debug Messages:

Started

Internal Total heap 378424, internal Free Heap 348176
SPIRam Total heap 4194252, SPIRam Free Heap 4194252
ChipRevision 1, Cpu Freq 240, SDK Version v3.2-111-g524308c98
Flash Size 4194304, Flash Speed 80000000
Board has PSRAM
Test malloc()...
NOT NULL
Test new()...
abort() was called at PC 0x400e9163 on core 1

Backtrace: 0x4008dce8:0x3ffb1ea0 0x4008df15:0x3ffb1ec0 0x400e9163:0x3ffb1ee0 0x400e91aa:0x3ffb1f00 0x400e8bd7:0x3ffb1f20 0x400e8dfa:0x3ffb1f40 0x400e8db1:0x3ffb1f60 0x400d0dec:0x3ffb1f80 0x400d209f:0x3ffb1fb0 0x4008c681:0x3ffb1fd0

Rebooting...

Decoding stack results:

0x4008dce8: invoke_abort at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/panic.c line 155
0x4008df15: abort at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/panic.c line 170
0x400e9163: __cxxabiv1::__terminate(void (*)()) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/eh_terminate.cc line 47
0x400e91aa: std::terminate() at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/eh_terminate.cc line 57
0x400e8bd7: __cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*)) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/eh_throw.cc line 87
0x400e8dfa: operator new(unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_op.cc line 54
0x400e8db1: operator new[](unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_opv.cc line 32
0x400d0dec: setup() at /home/jeroen/Arduino/Test_PSRAM_malloc_new/Test_PSRAM_malloc_new.ino line 21
0x400d209f: loopTask(void*) at /home/jeroen/Downloads/arduino-1.8.9-linux64/arduino-1.8.9/hardware/espressif/esp32/cores/esp32/main.cpp line 14
0x4008c681: vPortTaskWrapper at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/port.c line 143

[Jeroen88]

I made an esp-idf issue here first. But they say it is an Arduino ESP issue:

ps_malloc specifically targets psram

arduino-esp32/cores/esp32/esp32-hal-psram.c

Lines 55 to 60 in 010a7c6

	void IRAM_ATTR *ps_malloc(size_t size){
	if(!spiramDetected){
	return NULL;
	}
	return heap_caps_malloc(size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
	}

malloc has to be configured to include psram

esp-idf/components/esp32/Kconfig

Lines 63 to 64 in 2c55fae
config SPIRAM_USE_MALLOC
bool "Make RAM allocatable using malloc() as well"

arduino-esp32/tools/sdk/sdkconfig

Line 271 in 7d78247

CONFIG_SPIRAM_USE_MALLOC=

And:

Hi @Jeroen88 ,

Yes, @negativekelvin's assessment is correct. This is a choice in the Arduino project's configuration of ESP-IDF, not a limitation of ESP-IDF.

Feel free to open an issue on the arduino-esp32 project to ask about changing it. However there are other implications for the change so the Arduino maintainers may not be keen to do so.

You can also build Arduino as a component in ESP-IDF, in which case you can change the configuration yourself.

Because this is C++ another option would be to override the new operator in the places you need, in order to call ps_malloc() internally instead of malloc().

Angus

But I am confused. The first reaction is concerning alloc(), but alloc() is working correctly. I am looking for the same behavior for new()

[Jeroen88]

Closing for now, in my above example I called ps_malloc() in stead of malloc(). Going to investigate further first.

[Jonty]

@Jeroen88 Did you manage to debug this? I appear to be having the same problem with new().

[CwbhX]

I would also like to know if you got it working :)

[Jeroen88]

@CwbhX I did not made time to investigate this further, but I believe Angus suggestion is correct. I did not try yet to use the component as he described. As I understand some work has been done to make this easier with Pio.

[zekageri]

Any news on this?
I would like to allocate an array of classes in external ram but it's just not working. My heap goes lower but the psram size remains the same.

#define TEST_CLASS_ARR_SIZE 100
class testClass {
    long testVar;
    public:
    testClass() {
        //Serial.println("testClass constructor");
    }
    ~testClass() {
        //Serial.println("testClass destructor");
    }
    void sayHello(){
        Serial.println("Hello!");
    }
};
testClass* testClassesArray[TEST_CLASS_ARR_SIZE];

void testExtRamAllocation(){

    // print available external and internal ram size
    Serial.printf("Free Heap Size: %d\n", ESP.getFreeHeap());
    Serial.printf("Free ExtRam Size: %d\n", ESP.getFreePsram());

    // allocate testClasses in external ram with extRam allocator and put them in testClassesArray
    for(int i = 0; i < TEST_CLASS_ARR_SIZE; i++){
        testClassesArray[i] = new testClass();
    }

    // I can use any of the calsses like this: testClassesArray[15]->sayHello();

    // print available external and internal ram size
    Serial.printf("Free Heap Size: %d\n", ESP.getFreeHeap());
    Serial.printf("Free ExtRam Size: %d\n", ESP.getFreePsram());

    // delete testClasses in external ram
    for(int i = 0; i < TEST_CLASS_ARR_SIZE; i++){
        delete testClassesArray[i];
    }

    // print available external and internal ram size
    Serial.printf("Free Heap Size: %d\n", ESP.getFreeHeap());
    Serial.printf("Free ExtRam Size: %d\n", ESP.getFreePsram());
}

void setup(){
    testExtRamAllocation();
}

void loop(){
    vTaskDelete(NULL);
}

Using pio with the following configuration:

[env:esp-wrover-kit]
platform = espressif32

board       = esp-wrover-kit
framework   = arduino
board_build.filesystem = littlefs

board_build.f_cpu   = 240000000L
upload_port     = COM8
upload_speed    = 921600
monitor_speed   = 115200
monitor_filters = esp32_exception_decoder

board_upload.flash_size = 16MB
board_build.flash_mode = qio
board_build.partitions = large_spiffs_16MB.csv
board_build.f_flash    = 80000000L

build_flags =   -DBOARD_HAS_PSRAM
                -mfix-esp32-psram-cache-issue
                -mfix-esp32-psram-cache-strategy=memw
                -DCORE_DEBUG_LEVEL=0