Callback during tb.loop causes Guru Meditation Error

[karlvoigt]

PlatformIO project for ESP32 using Arduino framework.
ThingsBoard 0.12.2

I noticed an issue where the ESP32 would crash if a Shared Attribute or RPC callback is received while we are uploading telemetry. I have looked through all the open and closed issues for the Thingsboard CLient SDK but could not find any relevant solutions.

I get the following error output over UART:

2024-05-16 11:29:53	Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x4008b8
2024-05-16 11:29:53	44  PS      : 0x00060330  A0      : 0x801a7348  A1      : 0x3ffb0ce0  
A2      : 0xaf3f4041  A3      : 0xaf3f403d  A4      : 0x000000ff  A5      : 0x0000ff00  
A6      : 0x00ff0000  A7      : 0xff000000  A8      : 0x6469002e  A9      : 0x3ffb2200  
A10     : 0x401bc5d4  A11     : 0xffffcfbe  A12     : 0xfffffffd  A13     : 0x00000001  
A14     : 0xffffd137  A15     : 0x00000000  SAR     : 0x0000000a  EXCCAUSE: 0x0000001c  
EXCVADDR: 0xaf3f4041  LBEG    : 0x4008b865  LEND    : 0x4008b875  LCOUNT  : 0xffffffff  


Backtrace: 0x4008b841:0x3ffb0ce0 0x401a7345:0x3ffb0cf0 0x401ae4da:0x3ffb1000 0x401ae516:0x3ffb1090 0x400e5cde:0x3ffb10d0 0x400e3c82:0x3ffb1120 0x401bc5f5:0x3ffb21e0 0x400e5841:0x3ffb2200 0x400e5a11:0x3ffb2250 0x400db4c5:0x3ffb2270 0x400fa0e1:0x3ffb2290

Whith the following trace:

0x401bc5d4: std::_Function_handler::*(ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(char*, unsigned char*, unsigned int)> >::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) at /Users/karlvoigt/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:295
0x401a7345: _svfprintf_r at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1528
0x401ae4da: _vsnprintf_r at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vsnprintf.c:70
0x401ae516: vsnprintf at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vsnprintf.c:40
0x400e5cde: Helper::detectSize(char const*, ...) at /Users/karlvoigt/Documents/Simplifi.nosync/Nuwerus Revamp/Lora Code/Nuwerus Gateway/.pio/libdeps/gateway/ThingsBoard/src/Helper.cpp:19
0x400e3c82: ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>::onMQTTMessage(char*, unsigned char*, unsigned int) at /Users/karlvoigt/Documents/Simplifi.nosync/Nuwerus Revamp/Lora Code/Nuwerus Gateway/.pio/libdeps/gateway/ThingsBoard/src/ThingsBoard.h:1983
0x401bc5f5: void std::__invoke_impl::*&)(char*, unsigned char*, unsigned int), ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*&, char*, unsigned char*, unsigned int>(std::__invoke_memfun_deref, void (ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>::*&)(char*, unsigned char*, unsigned int), ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*&, char*, unsigned char*, unsigned int&&) at /Users/karlvoigt/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/invoke.h:73
0x401bc5f5: std::__invoke_result::*&)(char*, unsigned char*, unsigned int), ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*&, char*, unsigned char*, unsigned int>::type std::__invoke::*&)(char*, unsigned char*, unsigned int), ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*&, char*, unsigned char*, unsigned int>(void (ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>::*&)(char*, unsigned char*, unsigned int), ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*&, char*, unsigned char*, unsigned int) at /Users/karlvoigt/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/invoke.h:95
0x401bc5f5: void std::_Bind::*(ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(char*, unsigned char*, unsigned int)>::__call(std::tuple&&, std::_Index_tuple<((unsigned int)0, (unsigned int)1, (unsigned int)2, (unsigned int)3)...>) at /Users/karlvoigt/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/functional:400
0x401bc5f5: void std::_Bind::*(ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(char*, unsigned char*, unsigned int)>::operator()(char*, unsigned char*, unsigned int&&) at /Users/karlvoigt/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/functional:484
0x401bc5f5: std::_Function_handler::*(ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(char*, unsigned char*, unsigned int)> >::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) at /Users/karlvoigt/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:297
0x400e5841: std::function::operator()(char*, unsigned char*, unsigned int) const at /Users/karlvoigt/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:687
0x400e5841: PubSubClient::loop() at /Users/karlvoigt/Documents/Simplifi.nosync/Nuwerus Revamp/Lora Code/Nuwerus Gateway/.pio/libdeps/gateway/TBPubSubClient/src/PubSubClient.cpp:416
0x400e5a11: Arduino_MQTT_Client::loop() at /Users/karlvoigt/Documents/Simplifi.nosync/Nuwerus Revamp/Lora Code/Nuwerus Gateway/.pio/libdeps/gateway/ThingsBoard/src/Arduino_MQTT_Client.cpp:47
0x400db4c5: ThingsBoardSized<(unsigned int)256, ThingsBoardDefaultLogger>::loop() at /Users/karlvoigt/Documents/Simplifi.nosync/Nuwerus Revamp/Lora Code/Nuwerus Gateway/.pio/libdeps/gateway/ThingsBoard/src/ThingsBoard.h:470
0x400db4c5: loop() at /Users/karlvoigt/Documents/Simplifi.nosync/Nuwerus Revamp/Lora Code/Nuwerus Gateway/src/main.cpp:609
0x400fa0e1: loopTask(void*) at /Users/karlvoigt/.platformio/packages/framework-arduinoespressif32@3.20014.231204/cores/esp32/main.cpp:50

The following function is called from a task to send the telemetry:

//Function to send a specific PompStasie status updates to thingsboard
void updateThingsboardStatus(BesproeingsPunt* endNode) {
    //Send all the telemetry data of the specific pompstasie
    String telemetryKey = endNode->get_uuid() + "_pressure1";
    tb.sendTelemetryData(telemetryKey.c_str(), endNode->pressure1);
    telemetryKey = endNode->get_uuid() + "_pressure2";
    tb.sendTelemetryData(telemetryKey.c_str(), endNode->pressure2);
    telemetryKey = endNode->get_uuid() + "_batteryVoltage";
    tb.sendTelemetryData(telemetryKey.c_str(), endNode->batteryVoltage);
    telemetryKey = endNode->get_uuid() + "_actuator0_state";
    tb.sendTelemetryData(telemetryKey.c_str(), endNode->valveState);
    telemetryKey = endNode->get_uuid() + "_lastTransmission";
    tb.sendTelemetryData(telemetryKey.c_str(), endNode->lastTransmission);
    telemetryKey = endNode->get_uuid() + "_connectivityStatus";
    tb.sendTelemetryData(telemetryKey.c_str(), endNode->isOnline());
}

With the main loop then being used to call tb.loop:

void loop()
{
    //Main loop runs at lowest priority (Priority = 0) 
    if (!OTA_FLAG){
        // Process received Thingsboard messages messages
        if (tb.loop()) {
            thingsboardDownlinkInProgress = false;
        }
        thingsboardConnect();
    }
}

[karlvoigt]

I managed to fix this issue by implementing the single line fix proposed by arjenhiemstra in issue #832 on the PubSubClient project.

This only requires adding a single line check to the following lines of PubSubClient.cpp :

uint16_t tl = (this->buffer[llen+1]<<8)+this->buffer[llen+2]; /* topic length in bytes */
memmove(this->buffer+llen+2,this->buffer+llen+3,tl); /* move topic inside buffer 1 byte to front */

as follows:

uint16_t tl = (this->buffer[llen+1]<<8)+this->buffer[llen+2]; /* topic length in bytes */
if(!(tl < bufferSize)) return false;
memmove(this->buffer+llen+2,this->buffer+llen+3,tl); /* move topic inside buffer 1 byte to front */

Considering the ThingsboardClientSdk already ships with its own version of the PubSub library, I would suggest implementing this fix while waiting for the PubSub library to be properly updated.

[MathewHDYT]

Thanks a lot for the heads up will immediately start on the aforementioned fix and create a pull request so It can be merged into the custom PubSubClient version.

The own fork was originally made, out of exactly this reason as well because the PubSubClient is complety inactive and has not merged any Pull Requests in over 4 years.