Unstable micro-agent from docker (Foxy/Rolling)

[anaelle-sw]

Hello micro-ROS team!
I have some trouble to open a stable session between my board and the micro-ROS agent, when launched from Foxy docker.

Setup

• Ubuntu 20.04
• ROS2 Rolling
• RMW implementation forced to FastRTPS
• Teensy 3.2

Steps to reproduce

Build micro-ROS Arduino for Foxy:

cd ~/Arduino/libraries
git clone -b foxy https://github.com/micro-ROS/micro_ros_arduino.git
cat ~/path_to_meta_file/colcon_lowmem.meta > ~/Arduino/libraries/micro_ros_arduino/extras/library_generation/colcon_lowmem.meta
cd ~/Arduino/libraries/micro_ros_arduino
sudo docker pull microros/micro_ros_arduino_builder:foxy
sudo docker run -it --rm -v $(pwd):/arduino_project microros/micro_ros_arduino_builder:foxy -p teensy32

The meta file I use is just a bit modified in order to be able to use re-connection features:

{
    "names": {
        "tracetools": {
            "cmake-args": [
                "-DTRACETOOLS_DISABLED=ON",
                "-DTRACETOOLS_STATUS_CHECKING_TOOL=OFF"
            ]
        },
        "rosidl_typesupport": {
            "cmake-args": [
                "-DROSIDL_TYPESUPPORT_SINGLE_TYPESUPPORT=ON"
            ]
        },
        "rcl": {
            "cmake-args": [
                "-DBUILD_TESTING=OFF",
                "-DRCL_COMMAND_LINE_ENABLED=OFF",
                "-DRCL_LOGGING_ENABLED=OFF"
            ]
        },
        "rcutils": {
            "cmake-args": [
                "-DENABLE_TESTING=OFF",
                "-DRCUTILS_NO_FILESYSTEM=ON",
                "-DRCUTILS_NO_THREAD_SUPPORT=ON",
                "-DRCUTILS_NO_64_ATOMIC=ON",
                "-DRCUTILS_AVOID_DYNAMIC_ALLOCATION=ON"
            ]
        },
        "microxrcedds_client": {
            "cmake-args": [
                "-DUCLIENT_PIC=OFF",
                "-DUCLIENT_PROFILE_UDP=OFF",
                "-DUCLIENT_PROFILE_TCP=OFF",
                "-DUCLIENT_PROFILE_DISCOVERY=OFF",
                "-DUCLIENT_PROFILE_SERIAL=OFF",
                "-UCLIENT_PROFILE_STREAM_FRAMING=ON",
                "-DUCLIENT_PROFILE_CUSTOM_TRANSPORT=ON",
                "-DUCLIENT_MAX_SESSION_CONNECTION_ATTEMPTS=3"
            ]
        },
        "rmw_microxrcedds": {
            "cmake-args": [
                "-DRMW_UXRCE_ENTITY_CREATION_DESTROY_TIMEOUT=0",
                "-DRMW_UXRCE_MAX_NODES=1",
                "-DRMW_UXRCE_MAX_PUBLISHERS=3",
                "-DRMW_UXRCE_MAX_SUBSCRIPTIONS=2",
                "-DRMW_UXRCE_MAX_SERVICES=1",
                "-DRMW_UXRCE_MAX_CLIENTS=0",
                "-DRMW_UXRCE_MAX_HISTORY=1",
                "-DRMW_UXRCE_TRANSPORT=custom"
            ]
        }
    }
}

Then I launch the micro-ROS agent with the board connected:

sudo docker run -it --rm --net=host -v /dev:/dev --privileged microros/micro-ros-agent:foxy serial --dev /dev/teensy -v6

Then I upload some examples on the board, for instance re-connection example or the publisher example.

Issue description

I noticed that the behavior of the agent is really unstable, it would often do one of the following:
1 - I sometimes cannot open a session. In this case, the output is just:

[1616083974.580827] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 4
[1616083974.580939] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 6

I would try several times to reload the code on the board, or to re-launch the agent. It can take a dozen of tries before the session is established.
It especially happens with the publisher example. Is it possible that the custom meta file is interfering with applications that don't use the re-connection features?

2 - Once the session is open, everything run smoothly, the agent output shows all the messages being published and I can get them with ros2 topic echo. But after a while, the agent just freezes and no message are passed anymore. There is no error, but the agent output is stopped. May be the first case is also some kind agent freeze.

3 - After a few tries trying to launch the agent, I noticed with ros2 topic node that the node running on the board was present twice. With htop, I noticed an agent running in a docker but I was not able to kill the process. After a computer reboot, it was gone. It only happened once.

4 - I encountered more "specific" errors. For instance: the re-connection example doesn't always work, or a custom node only publishes one topic instead of three. But it may be related to the previous cases, so I can detail this later on if it is still relevant.

Sorry it is a bit vague. Both (1) and (2) cases are really easy to reproduce since it happens really frequently, but the freeze seems to happen randomly and I was not able to find more precisely what can cause this.

Thanks a lot for support!

[pablogs9]

I'm trying to reproduce this but:

docker pull microros/micro-ros-agent:foxy 
docker run -it --rm --net=host -v /dev:/dev --privileged microros/micro-ros-agent:foxy serial --dev /dev/serial/by-id/usb-Teensyduino_USB_Serial_8180040-if00 -v6


[1616400578.002739] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1616400578.002956] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 6
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Could you reproduce this @jamoralp @Acuadros95?
Can we upload the bad_alloc fix asap?

[pablogs9]

Hello @anaelle-sw could you confirm if the agent behaves in the same way with the new docker images:

docker pull microros/micro-ros-agent:foxy
docker pull microros/micro-ros-agent:rolling

We have found that the bad_alloc problem is no the only one that appears because of the versioning problem... @Acuadros95 has reported that he also has had agents frozen because of this error...

Let us know if you can replicate the behavior detailed here with these new images, if so, I will replicate your scenario here with our Teensy boards.

Thanks and sorry for the delay.

[anaelle-sw]

Hi @pablogs9
I still have the bad_alloc issue, when launching a lot of ROS2 nodes on behalf on the agent, but not right after launching the agent as it used to do sometimes.

To test the issues listed before, I loaded minimal publisher example on the board. I tried to use the Rolling and the Foxy dockers, and to let the RMW implementation to CycloneDDS or to force it to FastRTPS. With all of these environments I got the same results:

• The agent keeps freezing (at launch time mainly). I have to launch it several times in a raw, and/or reprogram the board until it works
• Sometimes, I cannot kill the agent. For instance, with command htop, I can see there is a process named micro_ros_agent that has been running for 30 minutes, using 100% of one of the CPU cores, and that I cannot kill.

So yes, unfortunately the errors number 1, 2, and 3 from the first post are still happening with the new images.

[pablogs9]

Some points:

1. Make sure that you do both docker pull on foxy and rolling images
2. I have been able to reproduce the bad_alloc under rolling, because it still on 2.0.0, I'm updating this.
3. I have not been able to reproduce the error on foxy. Can you verify if using the foxy image (even with the rest of the system on rolling), you are having the bad_alloc?

[anaelle-sw]

About the bad_alloc issue, you are right: it only happened with the Rolling image.
But the freezing problems happen on both images

[pablogs9]

Can you confirm how much CPU does the agent consumes during normal operation? Before it freezes.

[anaelle-sw]

With Rolling docker, it uses less than 3% of one CPU core. Do you want me to test it with Foxy dock as well?

[pablogs9]

Yes please, just a measure with htop

[anaelle-sw]

Well with Foxy docker, htop gives around 2.6% CPU usage

[pablogs9]

And, if you update the rolling image as stated in #40, to the newest rolling image (where I hope have fixed the bad_alloc as in Foxy docker) are you still having the freezes?

I want to be sure that both problems are not related. Sorry for being so exhaustive.

[anaelle-sw]

No problem. I try also try on both different setups we can use (dev computers and robots), to be sure the results are consistent, as the bad_alloc issue used to mainly happen on the robot.

So, the Rolling docker doesn't crash with error bad_alloc anymore when launching several ROS2 nodes on behalf of the agent!
For the moment, I am not able to reproduce the freeze problem... I will let you know if it happens again.

However, the Rolling docker presents other problems. For instance, on our custom application, the board should publish on 3 topics, but it actually publishes on only one... I will try to understand better what happen, and will describe better the problem then

[pablogs9]

Ok, let's keep this issue open for reporting your use case.

Thanks a lot for all the feedback Anaelle.

[anaelle-sw]

Sorry but just a quick question: how can you use domain Ids with the new api? This way doesn't work anymore and throw error rcl_node_options_t {aka struct rcl_node_options_t}' has no member named 'domain_id':

  rcl_node_options_t node_ops = rcl_node_get_default_options();
  node_ops.domain_id = 10;
  CHECK_RC_LOG(rclc_node_init_with_options(&node, "micro_ros_teensy_node", "", &support, &node_ops));

Thanks for helping!

[pablogs9]

Check: https://github.com/micro-ROS/micro-ROS-demos/blob/300067eaa082b7a85e4f2e648f8bee1623b4b08f/rclc/configuration_example/configured_publisher/main.c#L49