Workshop 4 ‐ Vision Sensors

Preparations

Check back to week 1 if you need an update on your working environment.

Task 0: Catch-up

Make sure you have worked through the main workshops to date, in particular, make sure you managed to make the robot move around (you must understand what mover.py does). If you are unsure, talk to the module delivery team.

Task 1: OpenCV and ROS

Streaming and viewing images

• launch the simulator or turn the LIMO robot on
• view image streams whilst driving the robot around: rqt_image_view (e.g. visualise the camera's image from topic /camera/color/image_raw)
• measure the frequency of an image topic: ros2 topic hz <topic_name>

Coding practice in Python using cv_bridge

• create a catkin package my_opencv_test, which should depend on cv_bridge and rclpy (remember how to do that? - HINT)
• be inspired by the following example opencv_test.py and code some small piece of python code that subscribes to the camera of your LIMO robot, and e.g. masks out any specific color in the image. For masking out, the OpenCV function inRange can be quite handy as also used in this code
• Publish and visualise the result from the above operation (e.g. publish your own ROS image from your OpenCV image, and display it in something like rqt_image_view)

Task 2: More fun with Computer Vision (optional)

These tasks benefit from a live camera stream. Therefore attempt these tasks if you either have access to the camera or a suitable video stream.

2D Object Detection

Local feature-based

• read http://introlab.github.io/find-object/ and http://wiki.ros.org/find_object_2d(this is documentation for ROS1 but still applies). Also refer to https://github.com/ros2-gbp/find_object_2d-release for the usage on ROS2.
• read about the feature detectors and descriptors used in the OpenCV Documentation
• install with sudo apt-get install ros-humble-find-object-2d
• run as ros2 run find_object_2d find_object_2d image:=<image_topic>
• train different objects and try different features

YOLO

First, read about YOLO, and even the original paper.

• clone into your workspace git clone --recursive https://github.com/leggedrobotics/darknet_ros.git
• switch to the correct branch git branch foxy
• install its depepndencies: rosdep update; rosdep install --from-paths . -i -y
• build it: colcon build -DCMAKE_BUILD_TYPE=Release
• edit ./darknet_ros/darknet_ros/config/ros.yaml to use the correct image topic, e.g.:

    camera_reading:
      topic: /camera/color/image_raw
  	queue_size: 1
  

• source your workspace and run as ros2 launch darknet_ros darknet_ros.launch