ESP32 is not stable when pushing it hard with WiFi.

[sblantipodi]

Hi,
the TCP stack of the ESP32 is a mess, it's slower than ESP8266 and it way less stable.

Hardware:

PLATFORM: Espressif 32 (3.2.0) > WEMOS LOLIN D32
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
PACKAGES:

• framework-arduinoespressif32 3.10006.210326 (1.0.6)
• tool-esptoolpy 1.30000.201119 (3.0.0)
• tool-mkspiffs 2.230.0 (2.30)
• toolchain-xtensa32 2.50200.97 (5.2.0)

IDE name: PlatformIO
Flash Frequency: 40 MHz
PSRAM enabled: no
Upload Speed: 500.000
Computer OS: Windows 10

Description:

Using pubsubclient (the well known lib for MQTT) I can receive 110 message per second on the ESP8266 running @160Mhz and 40 message per second on ESP32 running @240MHz.

I wrote a software that receives mqtt messages containing RGB informations and sends them to a LED strip using FastLED (the most famous lib for RGB on Arduino).

Code is pretty simple:
https://github.com/sblantipodi/glow_worm_luciferin/tree/color_correction

but ESP32 continues to randomly crash with those exceptions:

***ERROR*** A stack overflow in task DLE0 has been detected.
abort() was called at PC 0x4008fe70 on core 0
Guru Meditation Error: Core  0 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (DLE0) 

ELF file SHA256: 0000000000000000

Backtrace: 0x4000c2e4:0x3ffca6f0 0x4008fc1d:0x3ffca700 0x4008fd9d:0x3ffca7d0 0x40090085:0x3ffca7f0 0x4008d2b2:0x3ffca810 0x4008fbd9:0x3ffca8d0 0x4008fe59:0x3ffca8f0 0x4008fe70:0x3ffca910 0x40091a63:0x3ffca930 0x40093654:0x3ffca950 0x4009360a:0x00000000
  #0  0x4000c2e4:0x3ffca6f0 in ?? ??:0
  #1  0x4008fc1d:0x3ffca700 in commonErrorHandler_dump at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #2  0x4008fd9d:0x3ffca7d0 in commonErrorHandler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #3  0x40090085:0x3ffca7f0 in xt_unhandled_exception at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #4  0x4008d2b2:0x3ffca810 in _xt_user_exc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:757
  #5  0x4008fbd9:0x3ffca8d0 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #6  0x4008fe59:0x3ffca8f0 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #7  0x4008fe70:0x3ffca910 in vApplicationStackOverflowHook at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #8  0x40091a63:0x3ffca930 in vTaskSwitchContext at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:3507
  #9  0x40093654:0x3ffca950 in _frxt_dispatch at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portasm.S:406
  #10 0x4009360a:0x00000000 in _frxt_int_exit at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portasm.S:206

abort() was called at PC 0x40086509 on core 0

ELF file SHA256: 0000000000000000

Backtrace: 0x4008fbdc:0x3ffd0f60 0x4008fe59:0x3ffd0f80 0x40086509:0x3ffd0fa0 0x40086635:0x3ffd0fd0 0x400f3953:0x3ffd0ff0 0x400efd51:0x3ffd12b0 0x400efcec:0x3ffd1300 0x40091ea1:0x3ffd1330 0x40084df7:0x3ffd1360 0x4008f2c1:0x3ffd1380 0x4013e765:0x3ffd13b0 0x401728fd:0x3ffd13e0 0x401724d9:0x3ffd1400 0x4016cbf5:0x3ffd1420 0x4016cef5:0x3ffd1440 0x4016de66:0x3ffd1470 0x4016dea9:0x3ffd14b0 0x4016ded6:0x3ffd14e0 0x4016a421:0x3ffd1510 0x4016a4cf:0x3ffd1540 0x401676d1:0x3ffd1570 0x4016db47:0x3ffd1590 0x4017242a:0x3ffd15c0 0x4015efd3:0x3ffd15e0 0x40090e6a:0x3ffd1610
  #0  0x4008fbdc:0x3ffd0f60 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #1  0x4008fe59:0x3ffd0f80 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715
  #2  0x40086509:0x3ffd0fa0 in lock_acquire_generic at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:143
  #3  0x40086635:0x3ffd0fd0 in _lock_acquire_recursive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:171
  #4  0x400f3953:0x3ffd0ff0 in _vfiprintf_r at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/vfprintf.c:860 (discriminator 2)
  #5  0x400efd51:0x3ffd12b0 in fiprintf at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/fiprintf.c:50
  #6  0x400efcec:0x3ffd1300 in __assert_func at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:59 (discriminator 8)
  #7  0x40091ea1:0x3ffd1330 in vPortCPUAcquireMutexIntsDisabledInternal at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:3507
      (inlined by) vPortCPUAcquireMutexIntsDisabled at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portmux_impl.h:98
      (inlined by) vTaskEnterCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4201
  #8  0x40084df7:0x3ffd1360 in wifi_int_disable_wrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/esp_adapter.c:194
  #9  0x4008f2c1:0x3ffd1380 in pp_post at ??:?
  #10 0x4013e765:0x3ffd13b0 in esp_wifi_internal_tx at ??:?
  #11 0x401728fd:0x3ffd13e0 in low_level_output at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/port/esp32/netif/wlanif.c:122
  #12 0x401724d9:0x3ffd1400 in ethernet_output at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/netif/ethernet.c:309
  #13 0x4016cbf5:0x3ffd1420 in etharp_output_to_arp_index at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/etharp.c:787
  #14 0x4016cef5:0x3ffd1440 in etharp_output at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/etharp.c:889
  #15 0x4016de66:0x3ffd1470 in ip4_output_if_opt_src at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:1037
  #16 0x4016dea9:0x3ffd14b0 in ip4_output_if_opt at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:849
  #17 0x4016ded6:0x3ffd14e0 in ip4_output_if at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:826
  #18 0x4016a421:0x3ffd1510 in tcp_send_empty_ack at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_out.c:967 (discriminator 4)
  #19 0x4016a4cf:0x3ffd1540 in tcp_output at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_out.c:1028
  #20 0x401676d1:0x3ffd1570 in tcp_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:487
  #21 0x4016db47:0x3ffd1590 in ip4_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:750
  #22 0x4017242a:0x3ffd15c0 in ethernet_input at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/netif/ethernet.c:184
  #23 0x4015efd3:0x3ffd15e0 in tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:483
  #24 0x40090e6a:0x3ffd1610 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x40091b81  PS      : 0x00060033  A0      : 0x40093657  A1      : 0x3ffdf0c0  
A2      : 0x00050023  A3      : 0x00000001  A4      : 0x3ffc4c14  A5      : 0x3ffc4c14  
A6      : 0x3ff000dc  A7      : 0x00000001  A8      : 0x00000001  A9      : 0x00000002  
A10     : 0x00000000  A11     : 0x00000081  A12     : 0x3ffc4c30  A13     : 0x283ffbc1  
A14     : 0x00000000  A15     : 0x00000001  SAR     : 0x0000001a  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x000000c9  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xffffffff  

ELF file SHA256: 0000000000000000

Backtrace: 0x40091b81:0x3ffdf0c0 0x40093654:0x3ffdf0e0 0x4009360a:0x4000bff0
  #0  0x40091b81:0x3ffdf0c0 in vTaskSwitchContext at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:3507
  #1  0x40093654:0x3ffdf0e0 in _frxt_dispatch at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portasm.S:406
  #2  0x4009360a:0x4000bff0 in _frxt_int_exit at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portasm.S:206

ESP8266 works like a charm with the same code and it's even three times faster.

how can I fix those exceptions?
any help would be appreciated. thanks

[CelliesProjects]

I did not read any code so I make some assumptions here.

ERROR A stack overflow in task DLE0 has been detected.
abort() was called at PC 0x4008fe70 on core 0
Guru Meditation Error: Core 0 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (DLE0)

The error seems pretty clear. You need to increase the stacksize of the task doing the connecting.

ESP8266 works like a charm with the same code and it's even three times faster.

Keep in mind that esp8266 non-os has no FreeRTOS underneath.

[sblantipodi]

I did not read any code so I make some assumptions here.

ERROR A stack overflow in task DLE0 has been detected.
abort() was called at PC 0x4008fe70 on core 0
Guru Meditation Error: Core 0 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (DLE0)

The error seems pretty clear. You need to increase the stacksize of the task doing the connecting.

ESP8266 works like a charm with the same code and it's even three times faster.

Keep in mind that esp8266 non-os has no FreeRTOS underneath.

It's not a stack size problem, I have increased it many times but the errors does not improved.
I can transfer data no problem for minutes and then with the same data it crashes.

It works with a stack size of 8Kb and it works the same with a stack of 32Kb.

[sblantipodi]

Just another random exception that happen on esp32 when WiFi is used intensively.

assertion "recv_tcp: recv for wrong pcb!" failed: file "/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esGuru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x40091eb9  PS      : 0x00060033  A0      : 0x4009398f  A1      : 0x3ffee7b0  
A2      : 0x00050023  A3      : 0x00000001  A4      : 0x3ffc5ebc  A5      : 0x3ffbd8ec  
A6      : 0x00000001  A7      : 0x3ffba57e  A8      : 0x00000000  A9      : 0x00000002  
A10     : 0x00000000  A11     : 0xbc3ffbd8  A12     : 0x3ffc5ec4  A13     : 0x3ffcc1ac  
A14     : 0x00000000  A15     : 0x00000001  SAR     : 0x0000001f  EXCCAUSE: 0x0000001c  
EXCVADDR: 0xbc3ffc20  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  

ELF file SHA256: 0000000000000000

Backtrace: 0x40091eb9:0x3ffee7b0 0x4009398c:0x3ffee7d0 0x40093942:0xf61eadd8
  #0  0x40091eb9:0x3ffee7b0 in vTaskSwitchContext at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:3507
  #1  0x4009398c:0x3ffee7d0 in _frxt_dispatch at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portasm.S:406
  #2  0x40093942:0xf61eadd8 in _frxt_int_exit at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portasm.S:206

[TD-er]

LoadProhibited errors are always related to accessing data that was not allocate by the object you try to access.
This can either be due to a failed malloc, or the object has moved.

This last option, where the object has moved, can happen when you use some container type that does its own allocation.
For example std::map, String, std::vector etc.
To give some example I recently had to fix in my code.
I have a std::map data structure which is written to by a callback function and read from by the normal loop thread.
So what happened is that inserting a value could call a (re)balance of the RBtree and erasing too.
So if you were adding something in the loop() task and a callback just happened inbetween, both may trigger a RBtree rebalance which corrupt the tree structure.

Something similar can happen with String when you loop over a string (using a pointer instead of [] or use it via .c_str() to another function) and some other task appends something to the String, the memory pointer may no longer be valid.

In short, just based on the LoadProhibited error, I think some data structure is modified in multiple tasks.

Also given that you mention a significant slowdown in performance from ESP8266 to ESP32, I assume that the locking that is present in either your code or the SDK may also play a role in this performace hit.

[sblantipodi]

@TD-er thanks for the answer but I doubt that it can be a problem inside the code since it runs no problem on ESP8266.
and what about the huge performance drop on ESP32 over ESP8266?

[TD-er]

Like I wrote, the ESP8266 does use the WiFi code different from how the ESP32 does.
ESP32 has its WiFi code running on another task (other core too?) and thus it runs actually multi threaded.
On ESP8266 you must call delay() every now and then or else the WiFi code is not executed at all and you might loose incoming requests etc.
So whenever you run delay() on the ESP8266, you know you're not interacting with data structures. => not multi tasking.

I'm curious too what may cause the performance drop.
Just some idea, not sure if it makes sense.... but what if the ESP32 setup on your project does only send acknowledgements when the data has been processed? Then the time slice assigned to your sketch task may have quite a big impact.
On the ESP8266 you may just call delay() (also called at the end of a loop) which may prepare new data from WiFi to be processed. So then you're more actively steering the network data processing to your typical use case.
To get the same behavior on ESP32, you may need to run a separate RTOS task to handle network IO and prepare your data structs. Only then you must absolutely be sure to add locking to the data structures or else you would see a lot more of those crashes.

[sblantipodi]

I have created a simple sketch that receives "hello world" message from MQTT using an MQTT lib, it does not even print the message, just receives it.

I have sent a lot of messages per seconds on ESP8266 and ESP32 using the same sketch, ESP8266 is four or five times faster.

On ESP32 I have tried moving the callback that reads the message from one core to another, changing the priority of the task but the performance improvements is negligible.

ESP8266 seems to be much faster in doing "WiFi/MQTT things"

[srcastor]

Not read the code but regarding Wifi performance, it is likely that the modem sleep has something to do with it, try setting
WiFi.setSleep(false); and / or esp_wifi_set_ps(WIFI_PS_NONE); before connect to wifi

[sblantipodi]

Not read the code but regarding Wifi performance, it is likely that the modem sleep has something to do with it, try setting
WiFi.setSleep(false); and / or esp_wifi_set_ps(WIFI_PS_NONE); before connect to wifi

I was using the WiFi.setSleep(false); before opening this issue.
Thanks!

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.