Subscribe to multiple topics, frame rate drops

[xyxegle]

Required Info
Camera Model	L515
ROS version	Melodic
Firmware Version	Running with LibRealSense v2.49.0 FW version: 01.05.08.01
Operating System & Version	Ubuntu18.04
Kernel Version (Linux Only)	5.4.0-89-generic
Platform	PC
Language	C++/Python
Serial No.	f1062522
Product ID	0B64

Recently I have been recording data sets using RealSense 515, I have choose to subscribe to "/ camera/color/camera_info, / camera/color/image_raw, / camera/the depth/camera_info, / camera/the depth/color/points , /camera/imu. I set the collection frequency as 30Hz in the launch file, but AFTER checking the recorded Rosbag, I found that the frame rate was less than 10 frames. Do you know why?

[xyxegle]

I have repeatedly checked that the device is connected to USB3.0 port.It works fine in RealSense-Viewer.

[MartyG-RealSense]

Hi @xyxegle If you are streaming both depth and color and you have auto-exposure enabled then I would first recommend disabling an RGB option called Auto-Exposure Priority to see whether it makes a positive difference. Disabling it when auto-exposure is enabled can force the FPS to remain constant instead of being allowed to vary.

You could try disabling it in the RealSense ROS wrapper to see whether it makes a posiive difference in your particular case by opening the dynamic reconfigure interface after launch has completed and finding auto_exposure_priority under the RGB options, and unticking 'auto_exposure_priority' to disable it. The interface can be opened with the command rosrun rqt_reconfigure rqt_reconfigure

[xyxegle]

I adjusted it as you said. The framerate has improved, but not significantly. Especially, the camera/the depth/color/points of frame rate is very low. This is my Launch file. What else do you think needs to be optimized?

[xyxegle]

[xyxegle]

<arg name="serial_no"           default=""/>
  <arg name="usb_port_id"         default=""/>
  <arg name="device_type"         default=""/>
  <arg name="json_file_path"      default=""/>
  <arg name="camera"              default="camera"/>
  <arg name="tf_prefix"           default="$(arg camera)"/>
  <arg name="external_manager"    default="false"/>
  <arg name="manager"             default="realsense2_camera_manager"/>

  <arg name="fisheye_width"       default="640"/>
  <arg name="fisheye_height"      default="480"/>
  <arg name="enable_fisheye"      default="false"/>

  <arg name="depth_width"         default="640"/>
  <arg name="depth_height"        default="480"/>
  <arg name="enable_depth"        default="true"/>

  <arg name="infra_width"        default="640"/>
  <arg name="infra_height"       default="480"/>
  <arg name="enable_infra"        default="false"/>
  <arg name="enable_infra1"       default="false"/>
  <arg name="enable_infra2"       default="false"/>

  <arg name="color_width"         default="1280"/>
  <arg name="color_height"        default="720"/>
  <arg name="enable_color"        default="true"/>

  <arg name="fisheye_fps"         default="30"/>
  <arg name="depth_fps"           default="30"/>
  <arg name="infra_fps"           default="30"/>
  <arg name="color_fps"           default="30"/>
  <arg name="gyro_fps"            default="200"/>
  <arg name="accel_fps"           default="200"/>
  <arg name="enable_gyro"         default="true"/>
  <arg name="enable_accel"        default="true"/>

  <arg name="enable_pointcloud"         default="true"/>
  <arg name="pointcloud_texture_stream" default="RS2_STREAM_COLOR"/>
  <arg name="pointcloud_texture_index"  default="0"/>

  <arg name="enable_sync"               default="false"/>
  <arg name="align_depth"               default="false"/>

  <arg name="publish_tf"                default="true"/>
  <arg name="tf_publish_rate"           default="0"/>

  <arg name="filters"                   default="pointcloud"/>
  <arg name="clip_distance"             default="-2"/>
  <arg name="linear_accel_cov"          default="0.01"/>
  <arg name="initial_reset"             default="false"/>
  <arg name="unite_imu_method"          default="copy"/>
  <arg name="topic_odom_in"             default="odom_in"/>
  <arg name="calib_odom_file"           default=""/>
  <arg name="publish_odom_tf"           default="true"/>
  <arg name="allow_no_texture_points"   default="false"/>

[MartyG-RealSense]

The launch file looks fine to me. What improvement (if any) do you get if you set color to be 640x480 instead of 1280x720

[xyxegle]

[xyxegle]

There doesn't seem to be a significant improvement

[MartyG-RealSense]

Is the FPS drop limited to playback of the bag file or does it occur during live streaming too?

[xyxegle]

I don't know how to view the frame rate while running. When I use rqt_imageview observation, there is a clear sense of lag and frame loss.
I think the problem appears when it runs.

[MartyG-RealSense]

You can monitor the rate of a specific topic with the rostopic hz instruction. For example:

rostopic hz /camera/depth/color/points

[xyxegle]

After several attempts, I found that there should be a problem in the recording process. By checking the frame rate in the rosbag, the frame rate remains normal for the first few seconds, but it drops more and more as time goes on. But I'm using a solid-state drive, which should not be affected by the read/write rate.

[MartyG-RealSense]

Could you check whether there is a memory leak occurring, please. This is where the memory capacity of a computer is progressively consumed over time until an application has degraded performance or freezes / crashes. You can check the memory usage using an Ubuntu system monitoring utility such as htop

[xyxegle]

Something strange happened and I tried again several times after your reply. But none of them are ideal. Even just opening launch realsense2_camera rs_camera. Launch frame rate is abnormal. I've also rebooted to eliminate the effects of other programs but it's still abnormal. I also use physical cooling to prevent overheating. I'm falling apart right now. What's the reason?

[xyxegle]

[MartyG-RealSense]

I note in your specification information at the top of this case that you are using librealsense 2.49.0. It is important to match the RealSense ROS wrapper as closely as possible with a specific librealsense SDK version recommended in the wrapper release notes. There is not a wrapper version that matches to SDK 2.49.0 though, and so that introduces the potential for instability.

The ideal matches would be librealsense 2.48.0 and wrapper 2.3.1, or librealsense 2.50.0 and wrapper 2.3.2.

---

Also, may I ask whether you are using an Nvidia Jetson as your computer? There is a known issue with FPS drag and missing color when enabling the pointcloud filter on a Jetson board (the issue does not occur on Jetson if the pointcloud is disabled).

---

As you are using an L515 camera, is there any difference if you unplug the micro-sized end of the USB cable from the base of the L515, turn the connector around the other way and re-insert it into the camera base (USB-C cables are two-way insertion at the micro-sized end).

[xyxegle]

I found that the reason for frame loss was actually due to a version mismatch. Thank you very much for your answer.
My current version is LibrealSense 2.48.0 and Wrapper 2.3.1.

[xyxegle]

I also encountered some minor issues during the installation process, which I hope will be fixed in a later version. I encountered problems related to OpenCV and found that there was no associated CV library in Cmakelist, which resulted in the error shown in the picture below.

[xyxegle]

[xyxegle]

I added this to CMakelists under realSense-ros/RealSense2_camera:

find_package(OpenCV 3 REQUIRED opencv_core opencv_imgcodecs opencv_imgproc opencv_highgui)  
 
Or add OpencV_CALIb3D Opencv_Videoio  

It was eventually resolved. It is hoped that subsequent projects can be added to this part of the repair.

[xyxegle]

Another interesting thing is that when I plug in the USB-Type-C in the reverse direction, the frame rate drops by half. This is very interesting, I think it should be caused by the layout defect in the design. Hopefully, this problem can be avoided on newer hardware.

[xyxegle]

One more unresolved warning is the one shown in this answer.#2250 (comment) It still prompts this warning from time to time, and when I playback this rosbag, I find that several frames are missing at this moment. Is there any way to avoid this problem?

[xyxegle]

This warning appears to occur every 63 seconds.

[xyxegle]

[xyxegle]

The device I'm using to collect the data is i5 7200u + 940mx + 16 g DDR4-2133Mhz + 512g SATA solid-state drive.

[xyxegle]

Another thing I forgot to mention is that when I read these topics at the same time, RealSense became so hot that it would overheat if I used it for more than 30 minutes continuously as I mentioned in this question IntelRealSense/librealsense#10011. So I have to mount an external fan to actively cool it down. I also hope that active heat dissipation can be added in the future version to improve this problem.

[MartyG-RealSense]

Thanks very much for sharing your CMakeLists.txt workaround in #2250 (comment) with the RealSense ROS community!

The reduced performance with the L515 camera model when the micro-sized USB connector is inserted in a certain direction is a known hardware issue with this camera model's USB port. As it is a hardware trait, it cannot be corrected with software.

A RealSense user who recently experienced the No streams matched for pointcloud chosen texture Process - Color error at #2242 (comment) found that they had much improved stability and performance if they built librealsense from source code with the V4L backend (the non-RSUSB method where the kernel has to be patched).

It was stated by Doronhi the RealSense ROS developer at #588 (comment) that the warning No stream match for pointcloud chosen texture Process - Color can occur due to a frame drop (which is what you stated you were experiencing). Doronhi said that it was a playback issue related to the librealsense SDK and not caused by the RealSense ROS wrapper. The issue is referred to at #1826

An official Intel bug report was submitted by Doronhi but the librealsense developers were not able to find a problem after carrying out an investigation and the bug report was therefore closed.

In regard to overheating, I recommend making sure that the ventilation grilles on the casing are not obstructed and that you are not blowing air into the grilles (which can make overheating worse, as it disrupts internal air circulation in the casing).

If the camera is hot to the touch after only a few minutes then this could indicate that there is a glitch on the USB port or the USB cable is bad.

There will not be an updated version of the L515 released in the future as this model is being retired (declared End of Life) by Intel. The 400 Series stereo depth range is continuing.

[MartyG-RealSense]

Hi @xyxegle Do you require further assistance with this case, please? Thanks!

[xyxegle]

I have solved the problem. Thank you very much for your help

[MartyG-RealSense]

It's excellent to hear that you achieved a solution, @xyxegle - thanks very much for the update. As you solved the problem, I will close this case. Thanks again!