Roscon 2025 Smarter Vision Pipelines

This is the Roscon 2025 Smarter Vision Pipelines for ROS 2: Compress, Transport, and Sync at Scale Demo workspace

Overview

This project is a ROS 2 + GStreamer demo showcasing two complementary technologies for handling high-resolution image data in robotics:

1. skinny_vision – Efficient intra- and inter-process image transport using:

   • Type adaptation for cv::mat and cv::cuda::GpuMat
   • Fallback to compressed images
   • Optional GPU-accelerated compression (inter-process)

2. gst-bridge with timestamp preservation – Improvements to the ROS <-> GStreamer integration that:

   • Preserve original sensor timestamps over unreliable networks
   • Support ONVIF / Raw data streaming

Why This Matters

Robotics systems increasingly rely on multiple high-resolution video streams (e.g., 4K, multi-camera rigs). This creates challenges in:

• Bandwidth & latency – Moving and processing large image frames efficiently
• Synchronization – Keeping multi-camera and sensor data aligned across networks

This demo addresses both by:

• Reducing bandwidth and CPU usage via skinny_vision’s optimized transport
• Preserving original capture times across network boundaries with enhanced gst-bridge

What’s in the Demo

This repository contains launch scripts to:

• Generate synthetic image + metadata streams in ROS 2
• Stream them over GStreamer/RTSP with ONVIF timestamps
• Receive and visualize synchronized video + metadata in a ROS 2 client

You can:

• Run it out-of-the-box in server/client mode to see the full pipeline
• Tinker in advanced mode to experiment with your own GStreamer pipelines, network settings, or ROS 2 nodes

Running Docker with Vscode

Modify the mount points in .devcontainer/devcontainer.json

"workspaceMount": "source=<local workspace location>,target=/home/ros/workspace,type=bind,consistency=cached",

Use build and reopen in container option

Running the demo

Building

build_all

Or without the alias

colcon build --cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo --symlink-install --event-handlers console_direct+ --packages-select audio_msgs gst_handler gst_bridge gst_msgs skinny_vision data_source data_msgs pipeline_launcher

You can also build them individually with

build_dev gst_handler

Running the scripts

To run the scripts after building you will need to source the environment with:

source install/setup.zsh

Several launch scripts are provided in the package pipeline_launcher for different needs.

Launch all will launch all the nodes relevant to this demo.

ros2 launch pipeline_launcher launch_all.py

Launch server / client each launch the nodes that are expected to be on the server / client side of things.

The server side consists of:

• Data generation
• Serialization of metadata to ONVIF XML
• Running the RTSP server

While the client side:

• RTSP client
• Deserialization of metadata
• Visualization in Pygame of synchronized image / metadata pairs

ros2 launch pipeline_launcher launch_server.py

ros2 launch pipeline_launcher launch_client.py

Data node launcher that omits the RTSP server / client and visualization:

• Data generation
• Serialization of metadata to ONVIF XML
• Deserialization of metadata

ros2 launch pipeline_launcher launch_data_nodes.py

The rest needed to be ran manually:

RTSP server

install/gst_handler/bin/ros_2_gst

RTSP client

gst-launch-1.0 rtspsrc location=rtsp://localhost:8554/test_video onvif-mode=1 onvif-rate-control=0 latency=200 buffer-mode=0 ntp-sync=false name=src \
  src. ! application/x-rtp,media=application,encoding-name=VND.ONVIF.METADATA \
       ! queue \
       ! rtponvifparse \
       ! rtponvifmetadatadepay \
       ! queue \
       ! rosrawsink ros-topic=/received/meta_raw timestamp-mode=pts timestamp-conversion-mode=ntp2unix sync=false \
  src. ! application/x-rtp,media=video \
       ! queue \
       ! rtponvifparse \
       ! rtph264depay \
       ! avdec_h264 \
       ! videoconvert \
       ! timeoverlay time-mode=0 show-times-as-dates=true datetime-format="%d.%m.%Y %H:%M:%S.%f" halignment=right valignment=bottom \
       ! rosimagesink ros-topic=/received/image timestamp-mode=pts timestamp-conversion-mode=ntp2unix sync=false

Synchronized image viewer

ros2 run data_source synced_image_detection