While stress testing the onboard CAN buses on the Systemcore beta hardware, we at CTRE noticed that the Systemcore unexpectedly drops CAN receives past certain bus utilization thresholds, in some cases completely halting all CAN receives. Below is a summary of our test observations in various scenarios. All measurements of bus utilization as well as the health of the CAN bus were confirmed using industry-standard third-party tools. Additionally, CPU utilization on the Systemcore was around 50-60% for all tests on the native buses.
Tested on OS release beta 11 on beta Systemcore.
Baseline:
- At 100% bus utilization, the roboRIO native CAN bus does not drop any received frames, regardless of frame size.
- At 100% bus utilization, a CANivore connected to the Systemcore does not drop any received frames, regardless of frame size, both in its default setup and using custom firmware for SocketCAN default 1 Mbps, 5 Mbps, and 8 Mbps setups.
Systemcore S2 bus
| Bus configuration |
Frame size |
Bus utilization |
Approx. frames dropped |
Approx. longest gap of consecutively dropped frames |
| CAN 2.0B |
8 bytes |
100% |
No drops |
No drops |
| CAN 2.0B |
6 bytes |
100% |
8% |
100 ms |
| CAN 2.0B |
8 bytes, 11-bit ID |
100% |
8% |
100 ms |
|
|
|
|
|
| CAN FD 5 Mbps |
8 bytes |
65% |
61% |
500 ms |
| CAN FD 5 Mbps |
8 bytes |
66% |
Completely frozen |
Completely frozen |
| CAN FD 5 Mbps |
64 bytes |
79% |
<1% (~5 frames/s) |
3 ms |
|
|
|
|
|
| CAN FD 8 Mbps |
8 bytes |
60% |
63% |
500 ms |
| CAN FD 8 Mbps |
8 bytes |
63% |
Completely frozen |
Completely frozen |
| CAN FD 8 Mbps |
64 bytes |
75% |
<1% (~10 frames/s) |
15 ms |
| CAN FD 8 Mbps |
64 bytes |
90% |
Completely frozen |
Completely frozen |
Note: In all "Completely frozen" cases, receives resumed after bus utilization was lowered to one of the non-frozen cases.
Systemcore S3+S4 buses
| Bus configuration |
Frame size |
Bus utilization S3 / S4 |
Approx. frames dropped (S3) |
Approx. longest gap of consecutively dropped frames (S3) |
| CAN 2.0B |
8 bytes |
100% / 0% |
No drops |
No drops |
| CAN 2.0B |
8 bytes |
66% / 66% |
<1% |
2 ms |
| CAN 2.0B |
8 bytes |
59% / 74% |
7% |
70 ms |
| CAN 2.0B |
8 bytes |
74% / 74% |
16% |
100 ms |
| CAN 2.0B |
8 bytes |
88% / 88% |
Completely frozen |
Completely frozen |
|
|
|
|
|
| CAN FD 5 Mbps |
8 bytes |
33% / 33% |
Completely frozen |
Completely frozen |
Testing was stopped at this point, as it is clear that splitting the bus utilization evenly across the two buses (such as 33% on each instead of 66% on one) reproduces the issue in a similar manner as the S2 tests.
Other observations
When this issue occurs, ifconfig can_s2 in the "RX" row shows "errors", "overruns", and "frame" occasionally increasing. Additionally, enabling IRQ coalescing on receives slightly reduces the severity of the issue but does not at all solve it.
As a side note, the communication issues that occur at high bus utilization typically come from arbitration, where some devices cannot transmit their frames before timing out because there are too many other higher-priority frames on the CAN bus. However, a CAN device is expected to receive all frames that successfully make it onto the CAN bus regardless of bus utilization, as has been demonstrated with the roboRIO and the CANivore.
While stress testing the onboard CAN buses on the Systemcore beta hardware, we at CTRE noticed that the Systemcore unexpectedly drops CAN receives past certain bus utilization thresholds, in some cases completely halting all CAN receives. Below is a summary of our test observations in various scenarios. All measurements of bus utilization as well as the health of the CAN bus were confirmed using industry-standard third-party tools. Additionally, CPU utilization on the Systemcore was around 50-60% for all tests on the native buses.
Tested on OS release beta 11 on beta Systemcore.
Baseline:
Systemcore S2 bus
Note: In all "Completely frozen" cases, receives resumed after bus utilization was lowered to one of the non-frozen cases.
Systemcore S3+S4 buses
Testing was stopped at this point, as it is clear that splitting the bus utilization evenly across the two buses (such as 33% on each instead of 66% on one) reproduces the issue in a similar manner as the S2 tests.
Other observations
When this issue occurs,
ifconfig can_s2in the "RX" row shows "errors", "overruns", and "frame" occasionally increasing. Additionally, enabling IRQ coalescing on receives slightly reduces the severity of the issue but does not at all solve it.As a side note, the communication issues that occur at high bus utilization typically come from arbitration, where some devices cannot transmit their frames before timing out because there are too many other higher-priority frames on the CAN bus. However, a CAN device is expected to receive all frames that successfully make it onto the CAN bus regardless of bus utilization, as has been demonstrated with the roboRIO and the CANivore.