Isaac ROS Image Pipeline

Overview

This metapackage offers similar functionality as the standard, CPU-based image_pipeline metapackage, but does so by leveraging NVIDIA GPUs and the Jetson platform's specialized computer vision hardware. Considerable effort has been made to ensure that replacing image_pipeline with isaac_ros_image_pipeline on a Jetson device is as painless a transition as possible.

Table of Contents

• Isaac ROS Image Pipeline
  • Overview
  • Table of Contents
  • Latest Update
  • Performance
  • Supported Platforms
    • Docker
    • Package Dependencies
  • Quickstart
    • Replacing image_pipeline with isaac_ros_image_pipeline
  • Supported Packages
• ROS2 Package API
  • isaac_ros_image_proc
    • Overview
    • Available Components
  • isaac_ros_stereo_image_proc
    • Overview
    • Available Components
  • Troubleshooting
    • RealSense camera issue with 99-realsense-libusb.rules
      • Symptom
      • Solution
  • Updates
• References

Latest Update

Update 2022-08-31: Image flip support and update to be compatible with JetPack 5.0.2

Performance

The following are the benchmark performance results of Image_Proc Nodes and pipeline in this package, by supported platform:

Pipeline	AGX Orin	AGX Xavier	x86_64 w/ RTX 3060 Ti
Disparity Node (540p)	166 fps	80 fps	424 fps
Rectify Node (1080p)	193 fps	127 fps	--

Supported Platforms

This package is designed and tested to be compatible with ROS2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.

Note: Versions of ROS2 earlier than Humble are not supported. This package depends on specific ROS2 implementation features that were only introduced beginning with the Humble release.

Platform	Hardware	Software	Notes
Jetson	Jetson Orin Jetson Xavier	JetPack 5.0.2	For best performance, ensure that power settings are configured appropriately.
x86_64	NVIDIA GPU	Ubuntu 20.04+ CUDA 11.6.1+

Docker

To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.

Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.

Package Dependencies

• isaac_ros_common
• image_common
• vision_cv
• OpenCV 4.5+
• scikit-learn
• Eigen3

Quickstart

1. Set up your development environment by following the instructions here.

2. Clone this repository and its dependencies under ~/workspaces/isaac_ros-dev/src.

   cd ~/workspaces/isaac_ros-dev/src

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline

3. Launch the Docker container using the run_dev.sh script:

   cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
     ./scripts/run_dev.sh

4. Inside the container, build and source the workspace:

   cd /workspaces/isaac_ros-dev && \
     colcon build --symlink-install && \
     source install/setup.bash

5. (Optional) Run tests to verify complete and correct installation:

   colcon test --executor sequential

6. Start isaac_ros_image_proc using the prebuilt executable (Using Realsense camera as an example):

   ros2 run isaac_ros_image_proc isaac_ros_image_proc --ros-args -r /image_raw:=/camera/color/image_raw --ros-args -r /camera_info:=/camera/color/camera_info

7. In a separate terminal, spin up a calibrated camera publisher to /image_raw and /camera_info using any package(for example, realsense2_camera):

   ros2 launch realsense2_camera rs_launch.py

8. Observe the image output in grayscale and color on /image_mono and /image_rect_color, respectively:

   ros2 run image_view image_view --ros-args -r image:=image_mono  
   ros2 run image_view image_view --ros-args -r image:=image_rect_color  

Note: To build the RealSense camera package for Humble, please refer to the section here.

Other supported cameras can be found here.

For camera calibration, please refer to this guide.

Replacing image_pipeline with isaac_ros_image_pipeline

1. Add a dependency on isaac_ros_image_pipeline to your_package/package.xml and your_package/CMakeLists.txt. If all desired packages under an existing image_pipeline dependency have Isaac ROS alternatives (see Supported Packages), then the original image_pipeline dependency may be removed entirely.
2. Change the package and plugin names in any *.launch.py launch files to use [package name] and nvidia::isaac_ros::image_proc::[component_name] respectively. For a list of all packages, see Supported Packages. For a list of all ROS2 Components made available, see the per-package detailed documentation below.

Supported Packages

At this time, the packages under the standard image_pipeline have the following support:

Existing Package	Isaac ROS Alternative
image_pipeline	See isaac_ros_image_pipeline
image_proc	See isaac_ros_image_proc
stereo_image_proc	See isaac_ros_stereo_image_proc
depth_image_proc	Continue using existing package
camera_calibration	Continue using existing package
image_publisher	Continue using existing package
image_view	Continue using existing package
image_rotate	Continue using existing package

ROS2 Package API

isaac_ros_image_proc

Overview

The isaac_ros_image_proc package offers functionality for rectifying/undistorting images from a monocular camera setup, resizing the image, and changing the image format. It largely replaces the image_proc package, though the image format conversion facility also functions as a way to replace the CPU-based image format conversion in cv_bridge. The rectify node can also resize the image; if resizing is not needed, specify the output width/height same as input.

Available Components

Component	Topics Subscribed	Topics Published	Parameters
ImageFormatConverterNode	image_raw, camera_info: The input camera stream	image: The converted image	encoding_desired: Target encoding to convert to.
RectifyNode	image_raw, camera_info: The input camera stream	image_rect, camera_info_rect: The rectified camera stream	output_height: The absolute height to resize to output_width: The absolute width to resize to keep_aspect_ratio: The flag to keep the aspect_ratio when set to true
ResizeNode	image, camera_info: The input camera stream	resize/image, resize/camera_info: The resized camera stream	output_height: The absolute height to resize to output_width: The absolute width to resize to
ImageFlipNode	image: The input image data	image_flipped: The flipped image	flip_mode: Supports 3 modes - HORIZONTAL, VERTICAL, and BOTH

Limitation: Image proc nodes require even number dimensions for images.

isaac_ros_stereo_image_proc

Overview

The isaac_ros_stereo_image_proc package offers functionality for handling image pairs from a binocular/stereo camera setup, calculating the disparity between the two images, and producing a point cloud with depth information. It largely replaces the stereo_image_proc package.

Available Components

Component	Topics Subscribed	Topics Published	Parameters
DisparityNode	left/image_rect, left/camera_info: The left camera stream right/image_rect, right/camera_info: The right camera stream	disparity: The disparity between the two cameras	max_disparity: The maximum value for disparity per pixel, which is 64 by default. With ORIN backend, this value must be 128 or 256. backends: The VPI backend to use, which is CUDA by default (options: "CUDA", "XAVIER", "ORIN")
PointCloudNode	left/image_rect_color: The coloring for the point cloud left/camera_info: The left camera info right/camera_info: The right camera info disparity The disparity between the two cameras	points2: The output point cloud	use_color: Whether or not the output point cloud should have color. The default value is false. unit_scaling: The amount to scale the xyz points by

Troubleshooting

RealSense camera issue with 99-realsense-libusb.rules

Some RealSense camera users have experienced issues with libusb rules.

Symptom

admin@workstation:/workspaces/isaac_ros-dev$  ros2 launch realsense2_camera rs_launch.py
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2021-10-11-20-13-00-110633-UBUNTU-piyush-3480
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [realsense2_camera_node-1]: process started with pid [3482]
[realsense2_camera_node-1] [INFO] [1633983180.596460523] [RealSenseCameraNode]: RealSense ROS v3.2.2
[realsense2_camera_node-1] [INFO] [1633983180.596526058] [RealSenseCameraNode]: Built with LibRealSense v2.48.0
[realsense2_camera_node-1] [INFO] [1633983180.596543343] [RealSenseCameraNode]: Running with LibRealSense v2.48.0
[realsense2_camera_node-1]  11/10 20:13:00,624 ERROR [139993561417472] (handle-libusb.h:51) failed to open usb interface: 0, error: RS2_USB_STATUS_NO_DEVICE
[realsense2_camera_node-1] [WARN] [1633983180.626359282] [RealSenseCameraNode]: Device 1/1 failed with exception: failed to set power state
[realsense2_camera_node-1] [ERROR] [1633983180.626456541] [RealSenseCameraNode]: The requested device with  is NOT found. Will Try again.
[realsense2_camera_node-1]  11/10 20:13:00,624 ERROR [139993586595584] (sensor.cpp:517) acquire_power failed: failed to set power state
[realsense2_camera_node-1]  11/10 20:13:00,626 WARNING [139993586595584] (rs.cpp:306) null pointer passed for argument "device"

Solution

1. Check if the 99-realsense-libusb.rules file exists in /etc/udev/rules.d/
2. If not, disconnect the camera, copy this file to /etc/udev/rules.d/, then reconnect the camera.

Updates

Date	Changes
2022-08-31	Image flip support and update to be compatible with JetPack 5.0.2
2022-06-30	Migrated to NITROS based implementation
2021-10-20	Migrated to NVIDIA-ISAAC-ROS. Fixed handling of extrinsics in Rectify and Disparity nodes.
2021-08-11	Initial release to NVIDIA-AI-IOT

References

[1] D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nesic, X. Wang, and P. Westling. High-resolution stereo datasets with subpixel-accurate ground truth. In German Conference on Pattern Recognition (GCPR 2014), Münster, Germany, September 2014.