NVIDIA-accelerated, deep learned semantic image segmentation
Isaac ROS Image Segmentation contains ROS packages for semantic image segmentation.
These packages provide methods for classification of an input image at the pixel level by running GPU-accelerated inference on a DNN model. Each pixel of the input image is predicted to belong to a set of defined classes. The output prediction can be used by perception functions to understand where each class is spatially in a 2D image or fuse with a corresponding depth location in a 3D scene.
| Package | Model Architecture | Description |
|---|---|---|
| Isaac ROS U-NET | U-NET | Convolutional network popular for biomedical imaging segmentation models |
| Isaac ROS Segformer | Segformer | Transformer-based network that works well for objects of varying scale |
| Isaac ROS Segment Anything | Segment Anything | Segments any object in an image when given a prompt as to which one |
Input images may need to be cropped and resized to maintain the aspect ratio and match the input resolution expected by the DNN model; image resolution may be reduced to improve DNN inference performance, which typically scales directly with the number of pixels in the image.
Image segmentation provides more information and uses more compute than object detection to produce classifications per pixel, whereas object detection classifies a simpler bounding box rectangle in image coordinates. Object detection is used to know if, and where spatially in a 2D image, the object exists. On the other hand, image segmentation is used to know which pixels belong to the class. One application is using the segmentation result, and fusing it with the corresponding depth information in order to know an object location in a 3D scene.
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
| Sample Graph |
Input Size |
AGX Orin |
Orin NX |
Orin Nano 8GB |
x86_64 w/ RTX 4060 Ti |
x86_64 w/ RTX 4090 |
|---|---|---|---|---|---|---|
| SAM Image Segmentation Graph Full SAM |
720p |
2.22 fps 470 ms @ 30Hz |
– |
– |
– |
14.6 fps 79 ms @ 30Hz |
| SAM Image Segmentation Graph Mobile SAM |
720p |
10.8 fps 880 ms @ 30Hz |
5.13 fps 1500 ms @ 30Hz |
2.22 fps 360 ms @ 30Hz |
27.0 fps 62 ms @ 30Hz |
60.3 fps 27 ms @ 30Hz |
| TensorRT Graph PeopleSemSegNet |
544p |
371 fps 19 ms @ 30Hz |
250 fps 20 ms @ 30Hz |
163 fps 23 ms @ 30Hz |
670 fps 11 ms @ 30Hz |
688 fps 9.3 ms @ 30Hz |
Please visit the Isaac ROS Documentation to learn how to use this repository.
Update 2024-05-30: Add SegFormer and Segment Anything packages