Performance Issues

[julled]

I wanted to test the performance of the current implementation, if its capable of writing multiple Raw cameradatastreams to my SSD.

Testusecase:

• 1xCamera
• Node to simulate datastream 1280x1040x3 Byte @ 30fps
  • rosidl error when i wanted to create a native image
  • Created data by creating a string with the corresponding size
• using sqlite plugin
• Ubuntu 18.04
• ROS2 packages / rosbag via apt installed

Result:

• 4,6 seconds recording: only 32 of ~138 images got saved

Did i miss something, or is rosbag2 not capable yet to save that amount of data?

[Martin-Idel]

The SQLite plugin is not optimized as of now - rosbag2 still lacks a good performance testing suite to check for bottlenecks. Since no transactions are used as of today in the database backend, I would expect performance to be really bad for many small messages and better for larger messages at lower numbers.

Sending around 100 MB of data per second could be too much already on some systems.

[Karsten1987]

Hmm, looks like we have to dig a bit deeper into performance optimizations for SQLite3. The assumption that it's more suited for larger data seems to be in conflict with this ros-answers question: https://answers.ros.org/question/328814/ros2-bag-issue/

@Martin-Idel Do you know of any good starting point for diving into these performance topics?

[Martin-Idel]

I wanted to come back to this issue, because I find it interesting. I think the first step would be to create a good performance test suite to be sure that any changes actually help.

My starting point would probably be https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite - as you can see there are quite some points to get started. From what I know the actual performance tuning will probably start with:

• Use transactions (this will definitely help, and it might help a lot)
• Use batching within transactions (larger changes required here, but this will help for small messages)
• Look into journal modes (might also require more user settings)

[MCP-TTC]

Hi,

I am facing a similar problem with the following configuration:

• 2xImage msg @~11FPS 1928x1208x4
• 1xPointCloud2 msg @~11FPS 900pts
• sqlite plugin
• Ubuntu 18.04
• USB3.0 External SSD (tested up to 480MB/s)
• default DDS

Here some results:

• ROS2 nodes + rosbag2 from ROS2
  • topic_img_1 - 4.707FPS
  • topic_img_2 - 4.709FPS
  • topic_pts - 4.689FPS
• ROS2 nodes + ros1_bridge + rosbag from ROS1
  • topic_img_1 - 1.524FPS
  • topic_img_2 - 3.352FPS
  • topic_pts - 2.972FPS
• ROS1 nodes + rosbagv2 from ROS1
  • topic_img_1 - 11.256FPS
  • topic_img_2 - 11.250FPS
  • topic_pts - 11.116FPS

Any idea on how to improve rosbag2 performances?
Did anyone find a solution, even a temporary one, for this issue?

[MCP-TTC]

I tried the following setup with Eloquent and Foxy on Ubuntu 18.04, both built from source:

• publish 1920x600 images at 20 FPS with image_tools::cam2image (burger mode)
  ros2 run image_tools cam2image --ros-args -p burger_mode:=true -p width:=1920 -p height:=600 -p frequency:=20.0
• record the topic /image splitting the bag every 2GB
  ros2 bag record -o test-bag -b 2000000000 /image

With Eloquent everything seems fine but with Foxy the recording process continuously allocates memory up to a memory overflow if not stopped. I am wondering if there is any memory leak or if anyone else is getting the same issue.

[Karsten1987]

@MCP-TTC we indeed fixed a memory leak recently in #502. It's yet to be backported and released in the next Foxy patch release.
As of now, you could try to compile rosbag2 in an overlay workspace with this PR: #518

[MCP-TTC]

@Karsten1987 I confirm you it works, no more memory leaks. Thanks.

I have a question for you and the community ofc, still about performances, if I record the following topics:

• image1 (like the above) 20Hz
• pcl2 100Hz
• imu 125Hz
• image2 (1/3 of image1 payload) 30Hz

in a single bag instance I notice a big drop of the small but high frequency messages (delta freq. > 40%).
in multiple bag instances (1 per topic for example) the recorded messages frequencies are almost the produced ones (delta freq. = ~4%).

Did anyone ever notice something like this?

PS: the SSD drive can reach the required write bandwidth

[emersonknapp]

That behavior is probably related to #436 - the queue can fill up and drop messages while rosbag2 is writing

[clalancette]

@Karsten1987 @adamdbrw With the recent work done for performance in rosbag2, do you think we should close out this issue?

[adamdbrw]

I think so, this was one of the background issues for our work and things improved a lot

[emersonknapp]

I'm going to close this issue for now, please reopen if there is anything specific to address.

[MCP-TTC]

I am OK with closing this but I had no way to test the relative improvements yet. Have those been already merged to the Foxy branch? To me it seems the new improvements are only available on the Rolling one.
Am I wrong?

[emersonknapp]

The improvements are merged into the master branch (which is available as a Rolling package). It is my opinion that doing the feature development on the latest branch meets the requirements of an enhancement ticket - there is no given expectation that a new feature is backported to all live distributions. If a backport is desired for specific features, that can be tracked via a separate follow-up ticket.

For discussion about backporting latest features to Foxy, also see https://discourse.ros.org/t/fast-forward-merging-rosbag2-master-api-to-foxy/18927

[IdanAviv89]

Hi,
have you managed to solve this?
I am facing the same issue on ros galactic version. I use the python rosbag tool to convert the euroc mav dataset from ros1 to ros2. And while in ros1 I get that the imu topic is at 200Hz, at ros2 I get that the imu topic is at 120Hz.

[prabinrath]

I have a pointcloud2 dataset recorded with ros1. I converted it to ros2 bag using the rosbags tool. When I try to play the bag file, the RAM consumption ramps up and my system starts lagging. I tried to read the bag file with a custom python script using rosbag2_py and things seem to work without any issues. There is no memory outage. Can anyone please let me know whether this is a known issue with ros2 bag?

[juanluishortelano]

I'm still experiencing this in Ros Humble, ros2 bag record drops a lot of image fps.

[MichaelOrlov]

@juanluishortelano If you are experiencing with the performance issue please create a new issue with an explanation of how to reproduce it.
This issue is very outdated and was confirmed to be fixed.