Deserialization errors and eventual micro-ROS agent crash with best-effort publishers.

[lucas-tiz]

Issue template

• Hardware description: Arduino Portenta H7
• RTOS: Mbed OS
• Installation type: PlatformIO
• Version or commit hash: Galactic, Humble (I've tried both)

Steps to reproduce the issue

I am running micro-ROS on an Arduino Portenta H7, communicating via UDP with a computer running ROS2. I've been testing with both Galactic and Humble (two different systems, one running Ubuntu 20 and the other 22).

I have four publishers in micro-ROS, each publishing a custom message (containing a builtin_interfaces/Time, a uint8, and two float32s) at 100 Hz using best effort QoS. I also have one publisher publishing a custom message (a builtin_interfaces/Time, and a uint8) at 1 Hz.

After the micro-ROS agent is started, communication works for some time, but deserialization errors start to appear, mostly:

[1736739981.033497] error    | InputMessage.cpp   | log_error                | deserialization error  | buffer: 
0000: 00 00 00 00 00 00

and occasionally:

[1734728823.613239] error    | InputMessage.cpp   | log_error                | deserialization error  | buffer: 
0000: 81 01 49 12 07 01 1F 00 12 AD 00 A5 3E ED 65 67 D3 90 66 22 00 00 00 01 00 00 00 00 00 00 00 00
[1734728823.615098] error    | Processor.cpp      | process_write_data_submessage | deserialization error processing WRITE_DATA submessage | client_key: 0x16BCCD79

Messages continue to publish successfully at around 100 Hz, but eventually the micro-ROS agent crashes with the following message:

terminate called after throwing an instance of 'eprosima::fastcdr::exception::BadParamException'
  what():  Unexpected byte value in Cdr::deserialize(bool), expected 0 or 1

Sometimes it takes a while to crash, and other times it only takes a few minutes. I've also tried publishing with reliable QoS. In that case, the micro-ROS agent doesn't crash or have deserialization errors (at least not as quickly), but each topic is only published at about 1 Hz.

Expected behavior

The micro-ROS agent runs indefinitely with no deserialization errors and doesn't crash.

Actual behavior

Deserialization errors and crash.

Thanks!

[pablogs9]

We will need your application and custom msgs code to take a look.

[lucas-tiz]

Hi Pablo, I've attached main.cpp (as .txt because .cpp extension is not allowed) main.txt. I can add the whole project, but I've found out a few things that you might be able to comment on first:

It appears that the deserialization errors and crashing are related to multi-threading. In my main loop, I call g_system.spin() which just calls rclc_executor_spin_some() inside the system class. Inside a second thread canThread, I check for new CAN data, and if received, I process and publish the CAN data. If I move the publishers out of the canThread and into one of my timer callbacks, I no longer get deserialization errors.

I then found here that I should be setting

        "microxrcedds_client": {
            "cmake-args": [
                "-DUCLIENT_PROFILE_MULTITHREAD=ON"
            ]
        }

in my custom .meta file. But when I try to build, I get

.pio/libdeps/project/micro_ros_platformio/build/mcu/src/Micro-XRCE-DDS-Client/src/c/profile/multithread/multithread.c:23:2: error: #error XRCE multithreading not supported for this platform.

Is multi-threading not supported on the Arduino Portenta, or am I missing some other configuration-related things?

[pablogs9]

Hello @lucas-tiz, unfortunately, micro-ROS is not a multithreaded library, and UCLIENT_PROFILE_MULTITHREAD will only make the middleware thread safe.

If you have written your own CAN transports, you probably need to take care of the multithreading within their scope. I would recommend trying to make it work single-threaded and when it works smoothly try to configure multithreading.

In any case, I do not see a micro-ROS application in your attached file, there are no entities nor executors or spinning code block.

[lucas-tiz]

Hi @pablogs9 , sorry, I think I need to clarify: The CAN data is unrelated to the micro-ROS transport. I am using UDP for the micro-ROS transport. The CAN data is received from a sensor connected to the Portenta and then published to a ROS topic inside a thread.

If publish the data from the main thread instead of the CAN thread, I no longer get deserialization errors. However, if I still call rmw_uros_epoch_nanos() inside the CAN thread, I see time jumps in the values returned by rmw_uros_epoch_nanos(). So my questions are about what is supported with micro-ROS multithreading, since I can't find much documentation on this:

• Is multithreading supported on an Arduino Portenta running Mbed OS? I'm guessing it is not because I got #error XRCE multithreading not supported for this platform. when I set -DUCLIENT_PROFILE_MULTITHREAD=ON
• Is multithreading supported if I switch to Zephyr?
• With multithreading enabled, can I publish from inside two different threads?
• With multithreading enabled can I call rmw_uros_epoch_nanos() from inside two different threads?

All the micro-ROS entities and executors are in another file. I have quite a bit of code for this project organized in multiple files, so if it would be helpful to you, I can simplify the project and attach it.

Thank you

[hippo5329]

Only esp32 has native Freertos and limited pthread like support. You may check the example,

https://github.com/hippo5329/micro-ROS-demos-platformio/blob/rolling/rclc/multithread_publisher_subscriber/main.c

https://github.com/hippo5329/micro-ROS-demos-platformio/blob/rolling/rclc/fibonacci_action_server/main.c

The wiki,

https://github.com/hippo5329/micro-ROS-demos-platformio/wiki

[lucas-tiz]

Thanks for the information @hippo5329 , do you know if there is documentation on multithreading with micro-ROS? I've only found this so far: https://micro.ros.org/blog/2021/07/07/multithreading/