LittleFS crashes on ESP32 when using UART

[rsiemens77]

I am running my application on ESP32E, 4MB, with IDFv4.4.6.

My applicatioin is structured with below two tasks running on Core 0.
SerialReceiverTask, priority 2, uses interrupts to read from UART2, and passes data to queue.
WebServerTask, priority 1, serves HTTP requests, some of which use LittleFS

There is a constant stream of data coming in on UART2. But there's still plenty of time to respond to HTTP requests. Requests that don't use LittleFS respond fine, regardless if there is incoming data on the UART or not.

The request that accesses LittleFS also work fine when I disconnect the serial line so there's no incoming data on the UART. However, if there is incoming data, and I make the request, it consistently crashes as below. The first 3 lines only show up on the console, not in the decoded hex backtrace. I'd appreciate any insights on the problem.

0x4011bff8: uart_clear_intr_status at C:/exp/frameworks/esp-idf-v4.4.6/components/driver/uart.c:364
0x40082f50: _xt_lowint1 at C:/exp/frameworks/esp-idf-v4.4.6/components/freertos/port/xtensa/xtensa_vectors.S:1114
0x40089d56: _frxt_int_enter at C:/exp/frameworks/esp-idf-v4.4.6/components/freertos/port/xtensa/portasm.S:119 

0x4011bff5: uart_set_line_inverse at C:/exp/frameworks/esp-idf-v4.4.6/components/driver/uart.c:324
0x40082f4d: _xt_lowint1 at C:/exp/frameworks/esp-idf-v4.4.6/components/freertos/port/xtensa/xtensa_vectors.S:1114
0x4008db1e: spi_flash_ll_set_command at C:/exp/frameworks/esp-idf-v4.4.6/components/hal/esp32/include/hal/spi_flash_ll.h:329
(inlined by) spi_flash_hal_configure_host_io_mode at C:/exp/frameworks/esp-idf-v4.4.6/components/hal/spi_flash_hal_common.inc:120
0x40095691: spi_flash_chip_generic_config_host_io_mode at C:/exp/frameworks/esp-idf-v4.4.6/components/spi_flash/spi_flash_chip_generic.c:531
0x400956fe: spi_flash_chip_generic_read at C:/exp/frameworks/esp-idf-v4.4.6/components/spi_flash/spi_flash_chip_generic.c:229
0x40084d07: esp_flash_read at C:/exp/frameworks/esp-idf-v4.4.6/components/spi_flash/esp_flash_api.c:852 (discriminator 4)
0x4011f965: esp_partition_read at C:/exp/frameworks/esp-idf-v4.4.6/components/spi_flash/partition.c:424
0x4011227f: littlefs_api_read at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/littlefs_api.c:21
0x401126af: lfs_bd_read at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/littlefs/lfs.c:116
0x401132a6: lfs_dir_getslice at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/littlefs/lfs.c:704
0x401133a3: lfs_dir_get at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/littlefs/lfs.c:750
0x401169b0: lfs_dir_getinfo at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/littlefs/lfs.c:1386
0x40116ab1: lfs_dir_rawread at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/littlefs/lfs.c:2744
0x40116dc9: lfs_dir_read at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/littlefs/lfs.c:6160
0x4010eb41: vfs_littlefs_readdir_r at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/esp_littlefs.c:1789 (discriminator 4)
0x4010ec69: vfs_littlefs_readdir at C:/exp/frameworks/esp-idf-v4.4.6/components/esp_littlefs/src/esp_littlefs.c:1774
0x400d43df: esp_vfs_readdir at C:/exp/frameworks/esp-idf-v4.4.6/components/vfs/vfs.c:675 (discriminator 3)
0x4010d441: VFSFileImpl::openNextFile(char const*) at C:/exp/frameworks/esp-idf-v4.4.6/components/arduino/libraries/FS/src/vfs_api.cpp:483
0x4010546e: fs::File::openNextFile(char const*) at C:/exp/frameworks/esp-idf-v4.4.6/components/arduino/libraries/FS/src/FS.cpp:187
0x400dc47a: handleConfig() at C:/exp/dmcs/main/Aardvark.cpp:4678
0x401b1ba5: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at c:\exp\tools\xtensa-esp32-elf\esp-2021r2-patch5-8.4.0\xtensa-esp32-elf\xtensa-esp32-elf\include\c++\8.4.0\bits/std_function.h:297
0x401036ed: std::function<void ()>::operator()() const at c:\exp\tools\xtensa-esp32-elf\esp-2021r2-patch5-8.4.0\xtensa-esp32-elf\xtensa-esp32-elf\include\c++\8.4.0\bits/std_function.h:687
0x40106f71: FunctionRequestHandler::handle(WebServer&, http_method, String) at C:/exp/frameworks/esp-idf-v4.4.6/components/arduino/libraries/WebServer/src/detail/RequestHandlersImpl.h:45
0x40106fcf: WebServer::_handleRequest() at C:/exp/frameworks/esp-idf-v4.4.6/components/arduino/libraries/WebServer/src/WebServer.cpp:651
0x40107113: WebServer::handleClient() at C:/exp/frameworks/esp-idf-v4.4.6/components/arduino/libraries/WebServer/src/WebServer.cpp:318
0x400e6628: wifiServer(void*) at C:/exp/dmcs/main/Aardvark.cpp:3535
0x4008d349: vPortTaskWrapper at C:/exp/frameworks/esp-idf-v4.4.6/components/freertos/port/xtensa/port.c:142

[BrianPugh]

Bit of a long shot, but does Check menuconfig > Component config > Driver config > UART > UART ISR in IRAM fix it?

[rsiemens77]

A good shot! That fixed it! Thanks!

[BrianPugh]

Excellent! Do you know if esp_littlefs is doing anything incorrectly when it comes to uart interrupts? I see two non-mutually-exclusive options:

1. I mention this issue/solution in the README.

2. Is there a proper code fix? Or is this just the way it is?

[rsiemens77]

I don't really understand why the problem was exhibited in this way. My conclusion is this was a misconfiguration problem with my UART usage, that for unknown reason manifested with this LFS symptom. I suspect it's not really relevant to LFS and is probably already covered in the UART documentation. I adapted some example code I found, so probably need to go back and read the UART docs.

I don't think there's anything for you to do. But, the lowest level of the stack trace mentions

0x4011bff5: uart_set_line_inverse at C:/exp/frameworks/esp-idf-v4.4.6/components/driver/uart.c:324

I did briefly try to find that in the source code. I didn't see how LFS had any UART dependancy, and am wondering if that's incorrect? You would know better. How did you know the possible solution? Just your experience with ESP32 and configuration or is there actually some kind of UART usage?

For what it's worth, I also posted the issue in ESP32 support forum and got the same suggestion you made from one of the staff.

[rsiemens77]

Update, I asked for more info to ESP tech. He pointed to this:

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/peripherals/spi_flash/spi_flash_concurrency.html

So that makes more sense...

these APIs disable the caches suspends all the other tasks. Besides, all non-IRAM-safe interrupts will be disabled. The other core will be polling in a busy loop. These will be restored until the Flash operation completes.

I would put a pointer to this in the README and mention the possible contention with other devices such as UART.

[BrianPugh]

makes sense! I'll update the README.

How did you know the possible solution?

My thought was:

1. LittleFS has nothing to do with UART, but you aren't the first person to mention weird issues with UART.

2. Your stack trace had interrupt-like functions in it.

3. At some point in my esp32 development life, I read that the esp32 has something weird about flash access and interrupts and stuff.

Closing this issue.