Issues of D435i with Intel Up Board

[mzahana]

Hello. I am opening this issue to share issues of using D435/i with Intel Up board.

My setup is as follows:
1. Camera: D435i with firmware v2.19.1
2. Intel Up board with 4GB RAM and 32GB eMMC
3. Camera is connected to the USB 3.0 OTG port.
4. OS: Ubuntu 16.04
5. RealSense ROS v2.2.3
6. LibRealSense v2.19.1

am getting pointcloud at rate of 20hz, some times dropping to ~16hz. Is it possible to get higher rates e.g. ~30hz? Is the compute power of Up board not enough to achieve that?

I am also getting some warnings and errors as shown below. I would like to know if those are of concern, and must be fixed before consuming the pointcloud and IMU data, or not.

Thanks.

robot@robot:~/catkin_ws$ roslaunch realsense2_camera rs_camera.launch filters:=pointcloud
... logging to /home/robot/.ros/log/a68ec832-56a2-11e9-a4ce-0007323e721b/roslaunch-robot-7990.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://robot:46353/

SUMMARY
========

PARAMETERS
 * /camera/realsense2_camera/accel_fps: 250
 * /camera/realsense2_camera/accel_frame_id: camera_accel_frame
 * /camera/realsense2_camera/accel_optical_frame_id: camera_accel_opti...
 * /camera/realsense2_camera/align_depth: False
 * /camera/realsense2_camera/aligned_depth_to_color_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/base_frame_id: camera_link
 * /camera/realsense2_camera/calib_odom_file: 
 * /camera/realsense2_camera/clip_distance: -2.0
 * /camera/realsense2_camera/color_fps: 30
 * /camera/realsense2_camera/color_frame_id: camera_color_frame
 * /camera/realsense2_camera/color_height: 480
 * /camera/realsense2_camera/color_optical_frame_id: camera_color_opti...
 * /camera/realsense2_camera/color_width: 640
 * /camera/realsense2_camera/depth_fps: 30
 * /camera/realsense2_camera/depth_frame_id: camera_depth_frame
 * /camera/realsense2_camera/depth_height: 480
 * /camera/realsense2_camera/depth_optical_frame_id: camera_depth_opti...
 * /camera/realsense2_camera/depth_width: 640
 * /camera/realsense2_camera/enable_accel: True
 * /camera/realsense2_camera/enable_color: True
 * /camera/realsense2_camera/enable_depth: True
 * /camera/realsense2_camera/enable_fisheye1: True
 * /camera/realsense2_camera/enable_fisheye2: True
 * /camera/realsense2_camera/enable_fisheye: True
 * /camera/realsense2_camera/enable_gyro: True
 * /camera/realsense2_camera/enable_infra1: True
 * /camera/realsense2_camera/enable_infra2: True
 * /camera/realsense2_camera/enable_pointcloud: False
 * /camera/realsense2_camera/enable_sync: False
 * /camera/realsense2_camera/filters: pointcloud
 * /camera/realsense2_camera/fisheye1_frame_id: camera_fisheye1_f...
 * /camera/realsense2_camera/fisheye1_optical_frame_id: camera_fisheye1_o...
 * /camera/realsense2_camera/fisheye2_frame_id: camera_fisheye2_f...
 * /camera/realsense2_camera/fisheye2_optical_frame_id: camera_fisheye2_o...
 * /camera/realsense2_camera/fisheye_fps: 30
 * /camera/realsense2_camera/fisheye_frame_id: camera_fisheye_frame
 * /camera/realsense2_camera/fisheye_height: 480
 * /camera/realsense2_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /camera/realsense2_camera/fisheye_width: 640
 * /camera/realsense2_camera/gyro_fps: 400
 * /camera/realsense2_camera/gyro_frame_id: camera_gyro_frame
 * /camera/realsense2_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /camera/realsense2_camera/infra1_frame_id: camera_infra1_frame
 * /camera/realsense2_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /camera/realsense2_camera/infra2_frame_id: camera_infra2_frame
 * /camera/realsense2_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /camera/realsense2_camera/infra_fps: 30
 * /camera/realsense2_camera/infra_height: 480
 * /camera/realsense2_camera/infra_width: 640
 * /camera/realsense2_camera/initial_reset: False
 * /camera/realsense2_camera/json_file_path: 
 * /camera/realsense2_camera/linear_accel_cov: 0.01
 * /camera/realsense2_camera/odom_frame_id: camera_odom_frame
 * /camera/realsense2_camera/pointcloud_texture_index: 0
 * /camera/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /camera/realsense2_camera/pose_frame_id: camera_pose_frame
 * /camera/realsense2_camera/pose_optical_frame_id: camera_pose_optic...
 * /camera/realsense2_camera/publish_odom_tf: True
 * /camera/realsense2_camera/rosbag_filename: 
 * /camera/realsense2_camera/serial_no: 
 * /camera/realsense2_camera/topic_odom_in: odom_in
 * /camera/realsense2_camera/unite_imu_method: 
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES
  /camera/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [8000]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to a68ec832-56a2-11e9-a4ce-0007323e721b
process[rosout-1]: started with pid [8013]
started core service [/rosout]
process[camera/realsense2_camera_manager-2]: started with pid [8030]
process[camera/realsense2_camera-3]: started with pid [8031]
[ INFO] [1554359222.218164459]: Initializing nodelet with 4 worker threads.
[ INFO] [1554359222.413014163]: RealSense ROS v2.2.3
[ INFO] [1554359222.413113383]: Running with LibRealSense v2.19.1
[ INFO] [1554359222.507840131]: getParameters...
[ INFO] [1554359222.838516012]: setupDevice...
[ INFO] [1554359222.838693466]: JSON file is not provided
[ INFO] [1554359222.838782474]: ROS Node Namespace: camera
[ INFO] [1554359222.838916467]: Device Name: Intel RealSense D435I
[ INFO] [1554359222.838963377]: Device Serial No: 843112071742
[ INFO] [1554359222.839010575]: Device FW version: 05.11.01.100
255.255.255.255
[ INFO] [1554359222.839070609]: Device Product ID: 0x0B3A
[ INFO] [1554359222.839114820]: Enable PointCloud: On
[ INFO] [1554359222.839158680]: Align Depth: Off
[ INFO] [1554359222.839205440]: Sync Mode: On
[ INFO] [1554359222.839385981]: Device Sensors: 
[ INFO] [1554359222.839507363]: Stereo Module was found.
[ INFO] [1554359222.839576697]: RGB Camera was found.
[ INFO] [1554359222.839633057]: Motion Module was found.
[ INFO] [1554359222.839735602]: (Fisheye, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1554359222.839796061]: (Fisheye, 1) sensor isn't supported by current device! -- Skipping...
[ INFO] [1554359222.839853021]: (Fisheye, 2) sensor isn't supported by current device! -- Skipping...
[ INFO] [1554359222.839954216]: (Pose, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1554359222.840354983]: setupPublishers...
[ INFO] [1554359222.859825170]: Expected frequency for depth = 30.00000
[ INFO] [1554359222.881193799]: Expected frequency for infra1 = 30.00000
[ INFO] [1554359222.890276087]: Expected frequency for infra2 = 30.00000
[ INFO] [1554359222.898646198]: Expected frequency for color = 30.00000
[ INFO] [1554359222.933376254]: setupStreams...
 04/04 09:27:02,936 WARNING [139749987571584] (sensor.cpp:366) Unregistered Media formats : [ UYVY ]; Supported: [ ]
[ INFO] [1554359222.939105982]: depth stream is enabled - width: 640, height: 480, fps: 30
[ INFO] [1554359222.939818610]: infra1 stream is enabled - width: 640, height: 480, fps: 30
[ INFO] [1554359222.940416467]: infra2 stream is enabled - width: 640, height: 480, fps: 30
 04/04 09:27:02,941 WARNING [139749987571584] (backend-v4l2.cpp:1248) Pixel format 36315752-1a66-a242-9065-d01814a likely requires patch for fourcc code RW16!
 04/04 09:27:02,942 WARNING [139749987571584] (sensor.cpp:366) Unregistered Media formats : [ RW16 ]; Supported: [ ]
[ INFO] [1554359222.947478868]: color stream is enabled - width: 640, height: 480, fps: 30
[ INFO] [1554359222.951432910]: insert Depth to Stereo Module
[ INFO] [1554359222.951625250]: insert Color to RGB Camera
[ INFO] [1554359222.951707833]: insert Infrared to Stereo Module
[ INFO] [1554359222.951779180]: insert Infrared to Stereo Module
[ INFO] [1554359222.951837752]: insert Gyro to Motion Module
[ INFO] [1554359222.951913036]: insert Accel to Motion Module
 04/04 09:27:02,956 WARNING [139749134149376] (backend-hid.cpp:662) HID set_power 1 failed for /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.5/0003:8086:0B3A.0003/HID-SENSOR-200073.1.auto/iio:device1/buffer/enable
 04/04 09:27:02,957 WARNING [139749286852352] (backend-hid.cpp:662) HID set_power 1 failed for /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.5/0003:8086:0B3A.0003/HID-SENSOR-200076.2.auto/iio:device2/buffer/enable
 04/04 09:27:02,957 WARNING [139749142542080] (ds5-timestamp.cpp:174) HID timestamp not found! please apply HID patch.
[ WARN] [1554359222.959820907]: Frame metadata isn't available! (frame_timestamp_domain = RS2_TIMESTAMP_DOMAIN_SYSTEM_TIME)
[ INFO] [1554359222.998625186]: Add Filter: pointcloud
[ INFO] [1554359223.000365725]: num_filters: 1
[ INFO] [1554359223.000508168]: SELECTED BASE:Depth, 0
[ INFO] [1554359223.005561843]: RealSense Node Is Up!
[ INFO] [1554359223.005670050]: Setting Dynamic reconfig parameters.
 04/04 09:27:03,128 WARNING [139748756674304] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to installation chapter for details.
[ INFO] [1554359223.236626027]: Done Setting Dynamic reconfig parameters.
 04/04 09:27:03,439 WARNING [139748773459712] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to installation chapter for details.
[ WARN] [1554359270.803575147]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1554359272.742917648]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1554359274.569500711]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1554359283.215169597]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1554359284.751846664]: No stream match for pointcloud chosen texture Process - Color
[ERROR] [1554359292.320996890]: An error has occurred during frame callback: Error occured during execution of the processing block! See the log for more info
[ WARN] [1554359303.892353362]: No stream match for pointcloud chosen texture Process - Color

[doronhi]

All the warnings that you get seem to be caused by the low power up board. They just mean missing frames. You can try reducing the resolution to have a better frame rate or disable some of the sensors if you don't need them (If you don't use infra2 for example, set enable_infra2:=false. It may save some CPU.

[mzahana]

Thanks. What are the supported resolutions?

[doronhi]

try typing rs-enumerate-devices
you can add --help to see some more options.

[lucasjinreal]

@doronhi I got the same issue, even using 640x480 resolution same issue still comes sometime.

If I disable infra2, will it effect depth quality?

[doronhi]

It will have no effect on the depth quality. It only disable the infra2 images' transmission via the USB port. They are still taken and the depth is calculated using them inside the device.

[lucasjinreal]

@doronhi If so, how does it improve fps and performance ? It still working with infras

[doronhi]

It can save on the usb transportation and kernel work.

[Alex-Beh]

It will have no effect on the depth quality. It only disable the infra2 images' transmission via the USB port. They are still taken and the depth is calculated using them inside the device.

Can I set both infra1 and infra 2 to false? The depth is calculated using them inside the device but only the data didn't transmit via the USB port?

[doronhi]

It will have no effect on the depth quality. It only disable the infra2 images' transmission via the USB port. They are still taken and the depth is calculated using them inside the device.

Can I set both infra1 and infra 2 to false? The depth is calculated using them inside the device but only the data didn't transmit via the USB port?

You are correct. In fact, this is the default settings in rs_camera.launch. You can set both infrared streams off and still have the depth stream.

[MikeHallettUK]

I am using an ODROID N2+ with USB 3.2 and D455 : It is at less than 50% cpu when running 2 streams, getting this same ERROR message.

depth stream is enabled - width: 848, height: 100, fps: 100, Format: Z16
color stream is enabled - width: 424, height: 240, fps: 5, Format: RGB8

Agreed, the 100 fps is impressive, and runs ok without the minimal color stream: the ERROR msg start with both.

1. In my view, skipping a frame or two is not an "ERROR", at worst a WARN, maybe even just INFO ?
2. It would be useful to know which stream is causing the error.
3. I looked, but could not find anything in the ROS logs eg "See the log for more info" ? where is this please ?
4. When running this in just "realsense-viewer" there is no problem running all these streams together: why does the ROS wrapper have this problem, especially when there is plenty of CPU.
5. Could there be an option to "throttle" certain streams in "realsense2_camera", to purposefully skip frames ? : e.g. the 848x100 setting only does 100fps from RS, but could be useful at 50 or even 25 Hz as a ros topic.

Thanks

[wuh15]

Did you solved the problem@MikeHallettUK