Memory corruptions with std::string in SPI RAM (IDFGH-596)

[dexterbg]

Environment

• Development Kit: none
• Kit version (for WroverKit/PicoKit/DevKitC): none
• Module or chip used: ESP32-WROVER 16MB
• IDF version: v3.3-beta1-288-ga672e4f44 (also with v3.2-dev-1420-g151269458 and earlier versions)
• Build System: make
• Compiler version: (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a) 5.2.0
• Operating System: Linux, macOS
• Power Supply: USB

Problem Description

When building a text message within a task running on core 1 by appending text values to a std::string buffer located in SPI RAM, the buffer randomly gets corrupted by NUL bytes overwriting 1 to 3 bytes of the buffer.

The bug only shows on core 1 and only when using a std::string to build the message and only when the std::string is in SPI RAM.

The bug frequency rises with stress on both cores. On an idle module it may not appear. The attached test project reproduces the bug reliably by starting wifi in apsta mode and running a memory stressing task on each core. Starting wifi alone is sufficient to trigger the bug, as well as starting one of the mem stressers on any core.

The test is performed in four modes of different message part sources:

• mode 1: mixed length constant c strings
• mode 2: mixed length constant std::strings
• mode 3: constant c string
• mode 4: constant std::string

Modes 1 & 2 trigger the bug very frequently (run quickly into error limit), modes 3 & 4 less often (single test runs may not show the bug at all).

If using a char buffer instead of a std::string, neither stpcpy() nor memcpy() trigger the bug.

Expected Behavior

No buffer corruptions.

Actual Behavior

Buffer corruptions.

Steps to repropduce

1. Build and run the attached test project.
2. Watch the output.

Code to reproduce this issue

test-memory-corruption.zip

Debug Logs

Test log also included in ZIP:
test-log-20190129.txt

Other items if possible

• sdkconfig file: see ZIP
• elf file: see ZIP

Project origin

https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/
openvehicles/Open-Vehicle-Monitoring-System-3#189

[ThomasRogg]

Could be this problem: #2892

[markwj]

Maybe the same fundamental issue as #2892

We tried changing CONFIG_SPIRAM_SIZE to 2MB, but that had no improvement.

We then tried adding a kludgy:

void* big2mb = heap_caps_malloc(1024*1024*2, MALLOC_CAP_SPIRAM);

to the start of the code, and then the fault seemed to go away. Could this be related to just the bottom 2MB of SPIRAM? Or are they somehow mapped between cores differently? Our code runs mostly on core 1.

[dexterbg]

Running the test project with that 2 MB allocation and also starting the test task on core 0 shows the corruptions again. It seems core 0 can only work reliably with the lower 2 MB and core 1 only with the higher 2 MB.

[rohansingh]

I'm running into pretty much the exact same issue. Might be related to #2892, but this still happens for me while running in unicore mode. In #2892, most people report that the issue disappears when running with CONFIG_FREERTOS_UNICORE.

[dexterbg]

Issue solved by PSRAM fix in toolchain release 1.22.0-96-g2852398:
#2892 (comment)