Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

D435i IMU recording at inconsistent rate #11987

Closed
ryanalex98 opened this issue Jul 10, 2023 · 5 comments
Closed

D435i IMU recording at inconsistent rate #11987

ryanalex98 opened this issue Jul 10, 2023 · 5 comments

Comments

@ryanalex98
Copy link


Required Info
Camera Model D435i {D400 }
Firmware Version 5.15.0.2
Operating System & Version Ubuntu 20.04
Kernel Version (Linux Only) 5.15.0-76-generic
Platform PC
SDK Version 2.54.1.0 (downloaded via command line, not built from source)
Language C++
Segment Robot

Issue Description

Hi there, I am currently using a D435i camera and attempting to record some IMU data. However, when I record data, I am noticing a very inconsistent rate at which the readings are recorded. I have reviewed several issues in this forum to try and alleviate the problem across different product models but nothing seems to work for my case, hence why I am opening my own issue. I will do my best to go through what I've tried thus far.

First attempt: Gyro 400Hz, Accel 250Hz

USB 3.2 cable, port.

Launch file:
imu_recording.txt

Command line output

ryanalexander@ryanalexander-ThinkPad-P51:~/repos/custom/DataRecording$ roslaunch imu_recording imu_recording.launch 
... logging to /home/ryanalexander/.ros/log/50baf114-1f71-11ee-b0ee-1b5bbb1d9f49/roslaunch-ryanalexander-ThinkPad-P51-12884.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ryanalexander-ThinkPad-P51:34191/

SUMMARY
========

PARAMETERS
 * /camera/realsense2_camera/accel_fps: 250
 * /camera/realsense2_camera/accel_frame_id: camera_accel_frame
 * /camera/realsense2_camera/accel_optical_frame_id: camera_accel_opti...
 * /camera/realsense2_camera/align_depth: False
 * /camera/realsense2_camera/aligned_depth_to_color_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/allow_no_texture_points: False
 * /camera/realsense2_camera/base_frame_id: camera_link
 * /camera/realsense2_camera/calib_odom_file: 
 * /camera/realsense2_camera/clip_distance: -2.0
 * /camera/realsense2_camera/color_fps: 5
 * /camera/realsense2_camera/color_frame_id: camera_color_frame
 * /camera/realsense2_camera/color_height: 720
 * /camera/realsense2_camera/color_optical_frame_id: camera_color_opti...
 * /camera/realsense2_camera/color_width: 1280
 * /camera/realsense2_camera/confidence_fps: 30
 * /camera/realsense2_camera/confidence_height: 480
 * /camera/realsense2_camera/confidence_width: 640
 * /camera/realsense2_camera/depth_fps: 5
 * /camera/realsense2_camera/depth_frame_id: camera_depth_frame
 * /camera/realsense2_camera/depth_height: 720
 * /camera/realsense2_camera/depth_optical_frame_id: camera_depth_opti...
 * /camera/realsense2_camera/depth_width: 1280
 * /camera/realsense2_camera/device_type: 
 * /camera/realsense2_camera/enable_accel: True
 * /camera/realsense2_camera/enable_color: False
 * /camera/realsense2_camera/enable_confidence: True
 * /camera/realsense2_camera/enable_depth: False
 * /camera/realsense2_camera/enable_fisheye1: False
 * /camera/realsense2_camera/enable_fisheye2: False
 * /camera/realsense2_camera/enable_fisheye: False
 * /camera/realsense2_camera/enable_gyro: True
 * /camera/realsense2_camera/enable_infra1: False
 * /camera/realsense2_camera/enable_infra2: False
 * /camera/realsense2_camera/enable_infra: False
 * /camera/realsense2_camera/enable_pointcloud: False
 * /camera/realsense2_camera/enable_pose: False
 * /camera/realsense2_camera/enable_sync: False
 * /camera/realsense2_camera/filters: 
 * /camera/realsense2_camera/fisheye1_frame_id: camera_fisheye1_f...
 * /camera/realsense2_camera/fisheye1_optical_frame_id: camera_fisheye1_o...
 * /camera/realsense2_camera/fisheye2_frame_id: camera_fisheye2_f...
 * /camera/realsense2_camera/fisheye2_optical_frame_id: camera_fisheye2_o...
 * /camera/realsense2_camera/fisheye_fps: 5
 * /camera/realsense2_camera/fisheye_frame_id: camera_fisheye_frame
 * /camera/realsense2_camera/fisheye_height: 480
 * /camera/realsense2_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /camera/realsense2_camera/fisheye_width: 640
 * /camera/realsense2_camera/gyro_fps: 400
 * /camera/realsense2_camera/gyro_frame_id: camera_gyro_frame
 * /camera/realsense2_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /camera/realsense2_camera/imu_optical_frame_id: camera_imu_optica...
 * /camera/realsense2_camera/infra1_frame_id: camera_infra1_frame
 * /camera/realsense2_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /camera/realsense2_camera/infra2_frame_id: camera_infra2_frame
 * /camera/realsense2_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /camera/realsense2_camera/infra_fps: 30
 * /camera/realsense2_camera/infra_height: 480
 * /camera/realsense2_camera/infra_rgb: False
 * /camera/realsense2_camera/infra_width: 640
 * /camera/realsense2_camera/initial_reset: True
 * /camera/realsense2_camera/json_file_path: 
 * /camera/realsense2_camera/linear_accel_cov: 0.01
 * /camera/realsense2_camera/odom_frame_id: camera_odom_frame
 * /camera/realsense2_camera/ordered_pc: False
 * /camera/realsense2_camera/pointcloud_texture_index: 0
 * /camera/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /camera/realsense2_camera/pose_frame_id: camera_pose_frame
 * /camera/realsense2_camera/pose_optical_frame_id: camera_pose_optic...
 * /camera/realsense2_camera/publish_odom_tf: False
 * /camera/realsense2_camera/publish_tf: False
 * /camera/realsense2_camera/reconnect_timeout: 6.0
 * /camera/realsense2_camera/rosbag_filename: 
 * /camera/realsense2_camera/serial_no: 
 * /camera/realsense2_camera/stereo_module/exposure/1: 7500
 * /camera/realsense2_camera/stereo_module/exposure/2: 1
 * /camera/realsense2_camera/stereo_module/gain/1: 16
 * /camera/realsense2_camera/stereo_module/gain/2: 16
 * /camera/realsense2_camera/tf_publish_rate: 0.0
 * /camera/realsense2_camera/topic_odom_in: odom_in
 * /camera/realsense2_camera/unite_imu_method: linear_interpolation
 * /camera/realsense2_camera/usb_port_id: 
 * /camera/realsense2_camera/wait_for_device_timeout: -1.0
 * /camera/stereo_module/emitter_enabled: 2
 * /rosdistro: noetic
 * /rosversion: 1.16.0

NODES
  /
    realsense_recording (rosbag/record)
  /camera/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [12899]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 50baf114-1f71-11ee-b0ee-1b5bbb1d9f49
process[rosout-1]: started with pid [12916]
started core service [/rosout]
process[camera/realsense2_camera_manager-2]: started with pid [12923]
process[camera/realsense2_camera-3]: started with pid [12924]
process[realsense_recording-4]: started with pid [12925]
[ INFO] [1689028204.772858239]: Initializing nodelet with 8 worker threads.
[ INFO] [1689028204.862884523]: RealSense ROS v2.3.2
[ INFO] [1689028204.862946807]: Built with LibRealSense v2.50.0
[ INFO] [1689028204.862958928]: Running with LibRealSense v2.50.0
[ INFO] [1689028204.909855350]:  
[ INFO] [1689028205.198879391]: Device with serial number 033422071349 was found.

[ INFO] [1689028205.198991213]: Device with physical ID 2-5-4 was found.
[ INFO] [1689028205.199047836]: Device with name Intel RealSense D435I was found.
[ INFO] [1689028205.199629143]: Device with port number 2-5 was found.
[ INFO] [1689028205.199766298]: Device USB type: 3.2
[ INFO] [1689028205.199878970]: Resetting device...
[ INFO] [1689028211.332208522]:  
[ INFO] [1689028211.542669251]: Device with serial number 033422071349 was found.

[ INFO] [1689028211.542965430]: Device with physical ID 2-5-5 was found.
[ INFO] [1689028211.543106846]: Device with name Intel RealSense D435I was found.
[ INFO] [1689028211.545710826]: Device with port number 2-5 was found.
[ INFO] [1689028211.545824079]: Device USB type: 3.2
[ INFO] [1689028211.552824986]: getParameters...
[ INFO] [1689028211.612397394]: setupDevice...
[ INFO] [1689028211.612443762]: JSON file is not provided
[ INFO] [1689028211.612487214]: ROS Node Namespace: camera
[ INFO] [1689028211.612509189]: Device Name: Intel RealSense D435I
[ INFO] [1689028211.612528538]: Device Serial No: 033422071349
[ INFO] [1689028211.612553795]: Device physical port: 2-5-5
[ INFO] [1689028211.612591598]: Device FW version: 05.15.00.02
[ INFO] [1689028211.612629482]: Device Product ID: 0x0B3A
[ INFO] [1689028211.612654813]: Enable PointCloud: Off
[ INFO] [1689028211.612679387]: Align Depth: Off
[ INFO] [1689028211.612703859]: Sync Mode: Off
[ INFO] [1689028211.612751740]: Device Sensors: 
[ INFO] [1689028211.751948735]: Stereo Module was found.
[ INFO] [1689028211.778202035]: RGB Camera was found.
[ INFO] [1689028211.778513481]: Motion Module was found.
[ INFO] [1689028211.778582602]: (Confidence, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1689028211.778647161]: num_filters: 0
[ INFO] [1689028211.778732099]: Setting Dynamic reconfig parameters.
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
[ INFO] [1689028215.377397569]: Done Setting Dynamic reconfig parameters.
[ INFO] [1689028215.377601674]: gyro stream is enabled - fps: 400
[ INFO] [1689028215.377669049]: accel stream is enabled - fps: 250
[ INFO] [1689028215.377714555]: setupPublishers...
[ INFO] [1689028215.379232134]: Start publisher IMU
[ INFO] [1689028215.380230355]: setupStreams...
[ INFO] [1689028215.749108067]: SELECTED BASE:Depth, 0
[ WARN] [1689028215.750000353]: 
**[ WARN] [1689028215.750313720]: frame's time domain is HARDWARE_CLOCK. Timestamps may reset periodically.**
[ INFO] [1689028215.789152209]: RealSense Node Is Up!
 10/07 23:30:51,933 **WARNING [140112033740544] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11**

Rosbag investigation:
As you can see, the timestamps are consistently inconsistent, with drops occurring around every second.
Screenshot from 2023-07-10 23-34-52

Second attempt with camera enabled

When I enable the infra1 and infra2 cameras at 30Hz (640x480), I get the following output:
Screenshot from 2023-07-10 23-45-31
On the surface this output looks correct, but it should be operating at 400Hz, or every 0.0025 seconds. Instead, its alternating between producing a single message and then a couplet of two messages roughly every 0.0037 seconds. So on average, it produces three messages every 0.0075 seconds which lines up with 400Hz - the distribution of the messages is the problem.

Issues I referenced while attempting debug:

I tried decreasing the rates to 200Hz (Gyro) and 63Hz (Accel), but this didn't help. I also toggled global_time_enabled:=false (and true), neither of which solved the issue.

Please let me know if there is any more information I can provide in resolving this case. Thank you very much!

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Jul 11, 2023

Hi @ryanalex98 Setting unite_imu_method to 'copy' instead of 'linear_interpolation' may provide more stable IMU data. This can be done by including the command unite_imu_method:=copy in your roslaunch instruction.

As well as setting global time to false, you could also try setting enable_sync to true with the roslaunch command enable_sync:=true if you have not done so already. When true, it gathers the closest frames of different sensors to be sent with the same timetag.

I note that you are using camera firmware driver version 5.15.0.2 in your camera. This is the recommended firmware for librealsense 2.54.1 and using it with older SDK versions such as 2.50.0 may cause errors. The recommended firmware version for use with 2.50.0 is 5.13.0.50.

@ryanalex98
Copy link
Author

ryanalex98 commented Jul 12, 2023

Thanks for the quick reply @MartyG-RealSense.

I updated the launch file with the unite_imu_method set to copy and launched with the following args:

roslaunch imu_recording imu_recording.launch global_time_enabled:=false enable_sync:=true

Still getting curious output behavior:
Screenshot from 2023-07-12 21-43-11

I think I have the versions aligned, I'm using sdk v2.54.1.0 and firmware v5.15.0.2, which I believe are the most recent release pair?

To give some context, I am using this for a VIO application, which necessitates a high level of consistency from the IMU readings.

Thanks for your help.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Jul 13, 2023

Thanks very much for the update. Yes, SDK 2.54.1 and firmware 5.15.0.2 are the correct pairing. If you are using ROS1 Noetic though then the ideal configuration would be SDK 2.51.1, ROS wrapper 2.3.2 and firmware 5.13.0.50. This is because development of the ROS1 wrapper has ceased and so it has not been updated for compatibility with librealsense versions newer than 2.51.1.

IntelRealSense/realsense-ros#898 is an interesting discussion about IMU data stability and the importance of it for VIO, which led to the original creation of the 'copy' mode of unite_imu_method to provide additional IMU data stability.

Raw RealSense IMU data is inherently 'noisy'. One RealSense ROS user decided to use a high-quality external IMU instead of the camera's built-in one for their IMU topic's data.

@MartyG-RealSense
Copy link
Collaborator

Hi @ryanalex98 Do you require further assistance with this case, please? Thanks!

@MartyG-RealSense
Copy link
Collaborator

Case closed due to no further comments received.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants