ISR Queue overflow when signalling from CAN RX ISR

[rutgervandenberg]

Description of defect

When signalling from CAN using RX ISR, we are seeing the ISG Queue overflowing.
This looks to be related to ARM-software/CMSIS_5#283.

• Type: Bug
• Priority: Major

Expected behavior
The program keeps running.
Actual behavior
The program crashes with the message CMSIS-RTOS error: ISR Queue overflow (status: 0x2, task ID: 0x0, object ID: 0x20001014)

Target(s) affected by this defect ?

NUCLEO_F767ZI

Toolchain(s) (name and version) displaying this defect ?

arm-none-eabi-g++ (GNU Tools for ARM Embedded Processors 6-2017-q2-update) 6.3.1 20170620 (release) [ARM/embedded-6-branch revision 249437]

What version of Mbed-os are you using (tag or sha) ?

f9ee4e8

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed-cli version:
mbed --version
1.2.2

How is this defect reproduced ?

I've connected a Peak PCAN USB through a TJA1051T/3 with proper termination on both ends. I'm periodically sending dummy messages using the PCAN. I've tried frequencies as low as 10hz.

I've looked at the trace and the error happens inside the isrRtxThreadFlagsSet function.

I've sporadically run into this or a very similar issue (the ISR queue overflow) in the past when signaling/releasing a semaphore/inserting into a queue, but was never able to consistently reproduce it before.

#include "mbed.h"

static const uint32_t READ_IRQ = 	1 << 0;
CAN bus(PB_5, PB_6, 500000);

rtos::Thread thread;
void read_isr() {
	thread.signal_set(READ_IRQ);
}

void can_main() {
	CANMessage canmsg;
	while(true) {
		osEvent evt = rtos::Thread::signal_wait(READ_IRQ);
		while (bus.read(canmsg)) {
		}
		break;
		thread.signal_clr(READ_IRQ);
	}
}

// main() runs in its own thread in the OS
int main() {
	printf("Starting\r\n\n");
	thread.start(can_main);
	wait_ms(10);
	bus.attach(read_isr, CAN::RxIrq);
    while (true) {
        wait(100);
    }
}

[0xc0170]

@SenRamakri @bulislaw Please review

[rutgervandenberg]

I did some more testing and it seems that the ISR is called at 5 us intervals and the handling process never becomes active.

#include "mbed.h"

static const uint32_t READ_IRQ = 	1 << 0;
CAN bus(PB_5, PB_6, 500000);
DigitalOut isr_on(PA_3, 0);
DigitalOut processing(PC_0, 0);
rtos::Thread thread;
void read_isr() {
	isr_on = 1;
	thread.signal_set(READ_IRQ);
	isr_on = 0;
}

void can_main() {
	CANMessage canmsg;
	while(true) {
		processing = 0;
		osEvent evt = rtos::Thread::signal_wait(READ_IRQ);
		processing = 1;
		while (bus.read(canmsg)) {
		}
		thread.signal_clr(READ_IRQ);
	}
}

// main() runs in its own thread in the OS
int main() {
	printf("Starting\r\n\n");
	thread.start(can_main);
	wait_ms(10);
	bus.attach(read_isr, CAN::RxIrq);
    while (true) {
        wait(100);
    }
}

[SenRamakri]

@rutgervandenberg - I don't know what your application requirements are, one option is to try increase the depth of ISR queue using OS_ISR_FIFO_QUEUE. But I would suggest you to profile your driver/interrupt behavior to make sure you are not running into spurious interrupts issue here.

[ciarmcom]

ARM Internal Ref: MBOTRIAGE-67

[rutgervandenberg]

The issue was:
CAN.read() uses a mutex, which means it cannot be used in an interrupt, which lead me to believe that it should be used out of interrupt context. The result was that the interrupt was never reset, and therefore kept triggering the interrupt handler, preventing any thread from becoming active.

I 'fixed' the issue by extending the CAN class and disabling the mutex lock, which allows me to use CAN.read() in the interrupt handler, which in turn resets the interrupt. I'm guessing this behaviour is not intended, but I see no other way to use interrupts using (only) the mbed CAN class.

[swansk]

@rutgervandenberg So as I understand it, using the EventQueue was what caused the interrupt to trigger infinitely? Assuming there isn't a way to prevent this.

I am suffering from the same problem currently, surprising that after a year the CAN class has not been improved specifically with can.read(). Have you found any other workarounds as of yet?

[40Grit]

@checked01 How badly do you need it improved?

We have not come across many customer projects in our direction that need CAN, but we know our way around Mbed.

[swansk]

Thanks for the quick response @40Grit!

It would be excellent to have the ability to use CAN::read in the CAN interrupt. I've tried working with mbed-events (EventsQueue specifically), executing the CAN::read outside of the ISR attached to the CAN object, but ended up with an ISR queue overflow. This seems to be the same problem @rutgervandenberg had.

I additionally tried the workaround involving disabling the mutex lock in CAN.h under CAN::read, interestingly this did not resolve the mutex lock error for me.

I've settled on using a separate thread to wait for and read incoming CAN messages on a millisecond delay. While I'm glad to have a working solution, it would certainly be nice if we were able to use the interrupt without spinning up a new thread for reading the CAN.

One other quick note, I am developing with mbed on PlatformIO. I was able to use CAN::read in the interrupt no problem when using the mbed library without building the project with the full mbedOS for the RTOS features.

[rutgervandenberg]

@checked01 The reason the interrupt was constantly called is that I never cleared it in the handler. CAN::read resets it, so calling that in the handler solved the issue. I had to disable the lock to do that though.

Unfortunately I have to this day not been able to get CAN fully reliable using the mbed driver on my custom platform.

I considered your solution as well, but 1ms waits will cause me to miss messages, and to my knowledge mbed does not have sub-ms delays that don't busy wait.