How to get list of task running in individual cores in Arduino?

[yogeshwaran774]

Hardware:

Board: ESP32 Dev Module
Core Installation version: 1.0.3
IDE name: Arduino IDE
Flash Frequency: 40Mhz
PSRAM enabled: no
Upload Speed: 115200
Computer OS: Windows 7

Description:

I am trying find out why interrupt wdt timeout is happening in core 0 even though i am not attaching any interrupt in core 0. So i want to know list of task running in core 1 and core 0 to know the exact reason and does scheduler gives interrupt while switching task?

[atanisoft]

Tasks and ISR WDT are unrelated. The ISR WDT will be triggered anytime any ISR blocks more than 300 microseconds, core doesn't matter.

[yogeshwaran774]

Tasks and ISR WDT are unrelated. The ISR WDT will be triggered anytime any ISR blocks more than 300 microseconds, core doesn't matter.

okay but there is no interrupt attached in core 0 then how can i get reset cause as interrupt wdt timeout core 0 was running in ISR context

[atanisoft]

There are non-user controlled ISRs running from both cores. The primary one being a FreeRTOS tick ISR. If the tick does not increment as expected (missed ISR firing from either core) it will trip the ISR WDT for core 0.

Since you have not provided any code to reproduce this I can't help you isolate the issue in your ISR code.

[yogeshwaran774]

There are non-user controlled ISRs running from both cores. The primary one being a FreeRTOS tick ISR. If the tick does not increment as expected (missed ISR firing from either core) it will trip the ISR WDT for core 0.

Since you have not provided any code to reproduce this I can't help you isolate the issue in your ISR code.

Thankyou@atanisoft Understood i can send you the arduino file but you cannot reproduce this issue i feel because i am using decawave which gives interrupt when it is receiving something. This issue will not happen regularly until unless too many interrupts are generating at a time. I am attaching the core dump and debug message below.

##Debug Message:
Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0)
Core 0 register dump:
PC : 0x4008ff9d PS : 0x00060034 A0 : 0x800907b8 A1 : 0x3ffbe2f0
A2 : 0x3ffbefac A3 : 0x0000cdcd A4 : 0xb33fffff A5 : 0x00000001
A6 : 0x00060021 A7 : 0x0000abab A8 : 0x0000abab A9 : 0x3ffbe2f0
A10 : 0x00000003 A11 : 0x00060023 A12 : 0x00060021 A13 : 0x00000001
A14 : 0x0000002a A15 : 0x3ffd4d20 SAR : 0x00000019 EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000
Core 0 was running in ISR context:
EPC1 : 0x401dcb7a EPC2 : 0x00000000 EPC3 : 0x00000000 EPC4 : 0x4008ff9d

Backtrace: 0x4008ff9d:0x3ffbe2f0 0x400907b5:0x3ffbe320 0x4008e907:0x3ffbe340 0x40091699:0x3ffbe360 0x40081ab2:0x3ffbe370 0x401dcb77:0x3ffbc120 0x4015e586:0x3ffbc140 0x40090781:0x3ffbc160 0x4008e819:0x3ffbc180

PC: 0x4008ff9d: vTaskEnterCritical at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/portmux_impl.inc.h line 88
EXCVADDR: 0x00000000

Decoding stack results
0x4008ff9d: vTaskEnterCritical at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/portmux_impl.inc.h line 88
0x400907b5: xTaskIncrementTick at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/tasks.c line 2492
0x4008e907: xPortSysTickHandler at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/port.c line 289
0x401dcb77: esp_pm_impl_waiti at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/pm_esp32.c line 487
0x4015e586: esp_vApplicationIdleHook at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/esp32/freertos_hooks.c line 63
0x40090781: prvIdleTask at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/tasks.c line 3412
0x4008e819: vPortTaskWrapper at /Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/freertos/port.c line 143

[atanisoft]

The core dump and decode are useless without the code. Please post the code in the issue so it can be reviewed. Very likely the ISR handler is doing something that it shouldn't be (really anything more than setting a flag or waking a background task should be avoided).

[atanisoft]

Your code is doing things inside the hardware timer ISR call that can not be done:

#ifdef ESP32
void IRAM_ATTR resetModule() {
  Software.println("reboot\n");
  reset_attiny_timer();
  detachInterrupt(PIN_IRQ);
  detachInterrupt(GPIO_4);
  if (WiFi.status() == WL_CONNECTED)
  {
    delay(2);
    WiFi.disconnect();
    yield();
  }
#if defined BLE_ANCH && defined ESP32
  if (ble_connected)
  {
    btStop();
    Serial.println("le dis");
  }
#endif
  delay(10);
  esp_restart();
}
#endif

You are calling delay(2) which delays for 2 milliseconds which is longer than an ISR is allowed to block (300 microseconds by default).

This method is set to be called every ~5sec as a software WDT, you should move EVERYTHING out of this method and have a background task that is woken up by this method.

[yogeshwaran774]

Your code is doing things inside the hardware timer ISR call that can not be done:

#ifdef ESP32
void IRAM_ATTR resetModule() {
  Software.println("reboot\n");
  reset_attiny_timer();
  detachInterrupt(PIN_IRQ);
  detachInterrupt(GPIO_4);
  if (WiFi.status() == WL_CONNECTED)
  {
    delay(2);
    WiFi.disconnect();
    yield();
  }
#if defined BLE_ANCH && defined ESP32
  if (ble_connected)
  {
    btStop();
    Serial.println("le dis");
  }
#endif
  delay(10);
  esp_restart();
}
#endif

You are calling delay(2) which delays for 2 milliseconds which is longer than an ISR is allowed to block (300 microseconds by default).

This method is set to be called every ~5sec as a software WDT, you should move EVERYTHING out of this method and have a background task that is woken up by this method.

Okay what you said is correct. I have to resetModule() method and check how it is behaving but the hardware timer which i am using to reset esp32 was feeded regularly, hence it is not even triggered then how does this is related to the issue which i posted?

[atanisoft]

This is just one example of an ISR being used incorrectly, yes the hardware timer is an ISR callback. Anything you have tagged with IRAM_ATTR needs to be reviewed in depth to ensure it does not block whatsoever. I did not look into any of the extra libraries which your code includes as I did not have them or know which ones they might be.

[yogeshwaran774]

This is just one example of an ISR being used incorrectly, yes the hardware timer is an ISR callback. Anything you have tagged with IRAM_ATTR needs to be reviewed in depth to ensure it does not block whatsoever. I did not look into any of the extra libraries which your code includes as I did not have them or know which ones they might be.

I am using DWM1000 arduino library and i don't think it has any interrupts associated with it because the same library we used to compile it for arduino. you can check the files which i added whether any of these has core 0 ISR calls or not. As i checked there is no ISR attached in core 0.

I want to understand why PC is at vTaskCritical since i don't have any ISR attached in core 0. is it going there because of FreeRtos tick ISR?

[yogeshwaran774]

I enable Trace facility in FreeRtosConfig.h of my arduino but i am facing compilation error as undefined reference uxTaskGetSystemState(). Atleast if can see list of task running currently then i can verify each one by one. is there any way to get task list or get it compile?

[atanisoft]

As I mentioned previously, ignore the "core 0" piece of the crash as it is not entirely accurate with regard to ISR WDT. The ISR WDT will monitor BOTH cores and if either core does not fire it's ISRs in a timely manner it will cause an abort on core 0.

DWM1000 does use interrupts (at least https://github.com/thotro/arduino-dw1000/blob/master/src/DW1000.cpp#L182 does) and things it does inside this ISR callback (https://github.com/thotro/arduino-dw1000/blob/master/src/DW1000.cpp#L711-L749) will need to be reviewed further to ensure no delays etc.

I enable Trace facility in FreeRtosConfig.h of my arduino but i am facing compilation error as undefined

You would need to recompile ESP-IDF to enable this feature, it will also not help you with ISR WDT timeouts.

[yogeshwaran774]

As I mentioned previously, ignore the "core 0" piece of the crash as it is not entirely accurate with regard to ISR WDT. The ISR WDT will monitor BOTH cores and if either core does not fire it's ISRs in a timely manner it will cause an abort on core 0.

DWM1000 does use interrupts (at least https://github.com/thotro/arduino-dw1000/blob/master/src/DW1000.cpp#L182 does) and things it does inside this ISR callback (https://github.com/thotro/arduino-dw1000/blob/master/src/DW1000.cpp#L711-L749) will need to be reviewed further to ensure no delays etc.

I enable Trace facility in FreeRtosConfig.h of my arduino but i am facing compilation error as undefined

You would need to recompile ESP-IDF to enable this feature, it will also not help you with ISR WDT timeouts.

Okay i will modify the hardware timer callback along with i will look into DWM1000.cpp and let you know.

[yogeshwaran774]

I removed attachinterrupt in DW1000.CPP file and attached in my setup[xTaskCreatePinnedToCore(dwmInterrupt_Task, "Interrupt_service_task", 2000, NULL, 4, &interrupt_task, 1)] which call the ISR.

`void dwmInterrupt_Task(void *param)
{
for(;;)
{
xSemaphoreTake(semHandle, portMAX_DELAY);
DW1000.handleInterrupt();
}
}

void IRAM_ATTR Interrupt_sevice()
{
xSemaphoreGive(semHandle);
}`

Now the interrupt service execution time reduced but still same run time error i am getting as core 0 was running in ISR context. What to do?

[atanisoft]

Keep digging, something is using an ISR incorrectly still.

[yogeshwaran774]

Keep digging, something is using an ISR incorrectly still.

Yeah i am keeping checking the libraries. Let us say if attach interrupt in core 0 then it won't affect the core 1 functionality right?