Random crashes in asyncHTTPrequest::_onDisconnect

[andreas-provectusalgae]

Occasionally, the ESP32 crashes when trying to delete _client in asyncHTTPrequest::_onDisconnect or when trying to call asyncHTTPrequest::_onDisconnect.

The first scenario seems to happen when for some reason asyncHTTPrequest::_onDisconnect is called for the same asyncHTTPrequest instance and the same AsyncClient instance within very short time (1 or 2 ms).
It seems to happen when a connection times out.
In this scenario I get "CORRUPT HEAP: Bad head at 0x3ffdfa60. Expected 0xabba1234 got 0x3ffdfff8".

Stack trace for scenario 1:

0x4008c4b4: invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c line 155
0x4008c6e5: abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c line 170
0x4013731b: __assert_func at ../../../.././newlib/libc/stdlib/assert.c line 63
0x4008c129: multi_heap_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c line 214
0x40084b5a: heap_caps_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c line 268
0x40084f61: _free_r at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/syscalls.c line 42
0x4016a369: operator delete(void*) at /Volumes/build/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/del_op.cc line 46
0x400e1ae6: asyncHTTPrequest::_onDisconnect(AsyncClient*) at ~/Documents/Arduino/libraries/asyncHTTPrequest/src/asyncHTTPrequest.cpp line 510
0x400e1b29: std::_Function_handler >::_M_invoke(const std::_Any_data &, private/var/folders/22/x3rnxz554qq_0ddwr14cm80c0000gn/T/arduino_build_673379/ProvectusAlgaeBioreactorG5MasterController.ino.elf, CU 0x27474b, DIE 0x27f521>, private/var/folders/22/x3rnxz554qq_0ddwr14cm80c0000gn/T/arduino_build_673379/ProvectusAlgaeBioreactorG5MasterController.ino.elf, CU 0x27474b, DIE 0x27f526>) at ~/Documents/Arduino/libraries/asyncHTTPrequest/src/asyncHTTPrequest.cpp line 334
0x400e3112: std::function::operator()(void*, AsyncClient*) const at ~/Library/Arduino15/packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/xtensa-esp32-elf/include/c++/5.2.0/functional line 2271
0x400e3262: AsyncClient::_fin(tcp_pcb*, signed char) at ~/Documents/Arduino/libraries/AsyncTCP/src/AsyncTCP.cpp line 907
0x400e3278: AsyncClient::_s_fin(void*, tcp_pcb*, signed char) at ~/Documents/Arduino/libraries/AsyncTCP/src/AsyncTCP.cpp line 1214
0x400e3502: _async_service_task(void*) at ~/Documents/Arduino/libraries/AsyncTCP/src/AsyncTCP.cpp line 165
0x40088bc5: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

The second scenario seems to happen when asyncHTTPrequest::_onConnect is called just before asyncHTTPrequest::_onDisconnect for the same asyncHTTPrequest instance and the same AsyncClient instance is supposed to be called.
This seems to happen when a new request is started just before the connection is timing out.
In this scenario I get HTTPCODE_CONNECTION_LOST and then "Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited). Exception was unhandled.".

Stack trace for scenario 2:

0x400e326c: AsyncClient::_s_fin(void*, tcp_pcb*, signed char) at ~/Documents/Arduino/libraries/AsyncTCP/src/AsyncTCP.cpp line 1215
0x400e34f6: _async_service_task(void*) at ~/Documents/Arduino/libraries/AsyncTCP/src/AsyncTCP.cpp line 166
0x40088bc5: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

The code in asyncHTTPrequest seems to be thread-safe. I'm not sure how it is possible that asyncHTTPrequest::_onDisconnect tries to delete the same object twice when the mutex should have locked access to it and changed it to nullptr, before the lock is released and the second call is accessing it?!
Currently, I see three possible problems: a bug in the compiler, a bug in FreeRTOS or a bug (missing locks?) in AsyncTCP, but all I can tell so far is that asyncHTTPrequest has crashes sometimes when asyncHTTPrequest::_onDisconnect is (supposed to be) called.

[boblemaire]

I have had that problem as well. In my application, I was deleting the instance of asyncHTTPrequest after the timeout and then creating a new one for the next operation. When I stopped doing that, the problem went away. It runs for days now with no issues. I'm not inclined to try to debug this as I reached the same conclusion as you that it has to do with AsyncTCP or something below that. I had posted some questions to me-no-dev but got no response.

Right now I'm mulling over changing to using the ESP-IDF client directly to try and get HTTPS. The documentation says it only runs async for HTTPS, but I can just run in a task and let it block for HTTP. The HTTPS is more important to me. I would leave this as is an just make a new client class.

[andreas-provectusalgae]

Thanks for the quick reply!

The asyncHTTPrequest is a long-living one in our case. However, there are actually two instances.
I was actually thinking of trying if recreating the instances would be more stable. Good to know that it won't.

Unfortunately, in its current state the library is not usable for us :(
I've changed the HTTP requests back to be blocking for now, but I'm really hoping that asynchronous calls will work in the future.

We do need HTTP (rather than HTTPS) though.

[julienaltieri]

Did you end up getting https? I just came across this library and found https crashes it.

[boblemaire]

@julienaltieri ,
No, HTTPS is not supported. I suggest using a reverse proxy like nginx on Rpi or equivalent.