3D Object Pose Estimation - ROS - Realsense d435

[Bilibili_Demo]

Development Environment

• Ubuntu 16.04.2

• ROS Kinetic

• Ubuntu 18.04.1

• ROS Melodic

---

How can you get the datasets object in the real world

In Datasets_obj folder you can printing the object texture onto a box or can of the exact size.

How to make custom synthetic data using UE4

[Bilibili_Demo]

Please check the NVIDIA Deep learning Dataset Synthesizer (Synthetic-Data-UE4)

DOPE Installing

Step 1: Download the DOPE code

cd ~/catkin_ws/src
git clone https://github.com/yehengchen/DOPE-ROS-Realsense.git

Step 2: Install python dependencies

cd ~/catkin_ws/src/dope
pip install -r requirements.txt

Step 3: Install ROS dependencies

cd ~/catkin_ws
rosdep install --from-paths src -i --rosdistro kinetic
sudo apt-get install ros-kinetic-rosbash ros-kinetic-ros-comm
Build

cd ~/catkin_ws
catkin_make

Step 4: Download the weights and save them to the weights folder, i.e., ~/catkin_ws/src/dope/weights/.

---

ROS Wrapper for Intel® RealSense D435 - Ubuntu 16.04_ROS Kinetic

Step 1: Install the latest Intel® RealSense™ SDK 2.0

Install from Debian Package - In that case treat yourself as a developer. Make sure you follow the instructions to also install librealsense2-dev and librealsense-dkms packages. OR Build from sources by downloading the latest Intel® RealSense™ SDK 2.0 and follow the instructions under Linux Installation

Step 2: Install the ROS distribution Install ROS Kinetic, on Ubuntu 16.04

Step 3: Install Intel® RealSense™ ROS from Sources

cd ~/catkin_ws/src/

Clone the latest Intel® RealSense™ ROS from here into 'catkin_ws/src/'

git clone https://github.com/IntelRealSense/realsense-ros.git
cd realsense-ros/
git checkout `git tag | sort -V | grep -P "^\d+\.\d+\.\d+" | tail -1`
cd ..

Make sure all dependent packages are installed. You can check .travis.yml file for reference. Specifically, make sure that the ros package ddynamic_reconfigure is installed. If ddynamic_reconfigure cannot be installed using APT, you may clone it into your workspace 'catkin_ws/src/' from here (Version 0.2.0)

catkin_init_workspace
cd ..
catkin_make clean
catkin_make -DCATKIN_ENABLE_TESTING=False -DCMAKE_BUILD_TYPE=Release
catkin_make install
echo "source ~/catkin_ws/devel/setup.bash" >> ~/.bashrc
source ~/.bashrc

ROS Wrapper for Intel® RealSense D435 - Ubuntu 18.04_ROS Melodic

IntelRealSense to install the librealsense Debian package.

After installing these packages I was able to compile the ros package from source as described in RealSense™ Devices

---

Running

1. Start ROS master

cd ~/catkin_ws
source devel/setup.bash
roscore

2. Start camera node (or start your own camera node)

Realsense D435 & usb_cam node (./dope/config/config_pose.yaml):

topic_camera: "/camera/color/image_raw"            #"/usb_cam/image_raw"
topic_camera_info: "/camera/color/camera_info"     #"/usb_cam/camera_info"

Start camera node:

roslaunch realsense2_camera rs_rgbd.launch  # Publishes RGB images to `/camera/color/image_raw`

3. Start DOPE node

roslaunch dope dope.launch [config:=/path/to/my_config.yaml]  # Config file is optional; default is `config_pose.yaml`

4. Start rviz node

rosrun rviz rviz

Debugging

• The following ROS topics are published (assuming topic_publishing == 'dope'):

    /dope/webcam_rgb_raw       # RGB images from camera
    /dope/dimension_[obj_name] # dimensions of object
    /dope/pose_[obj_name]      # timestamped pose of object
    /dope/rgb_points           # RGB images with detected cuboids overlaid
    /dope/detected_objects     # vision_msgs/Detection3DArray of all detected objects
    /dope/markers              # RViz visualization markers for all objects

*Note:* `[obj_name]` is in {cracker, gelatin, meat, mustard, soup, sugar}

• To debug in RViz, run rviz, then add one or more of the following displays:

  • Add > Image to view the raw RGB image or the image with cuboids overlaid
  • Add > Pose to view the object coordinate frame in 3D.
  • Add > MarkerArray to view the cuboids, meshes etc. in 3D.
  • Add > Camera to view the RGB Image with the poses and markers from above.

Citing

If you use this tool in a research project, please cite as follows:

@inproceedings{tremblay2018corl:dope,
 author = {Jonathan Tremblay and Thang To and Balakumar Sundaralingam and Yu Xiang and Dieter Fox and Stan Birchfield},
 title = {Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects},
 booktitle = {Conference on Robot Learning (CoRL)},
 url = "https://arxiv.org/abs/1809.10790",
 year = 2018
}

License

Copyright (C) 2018 NVIDIA Corporation. All rights reserved. Licensed under the CC BY-NC-SA 4.0 license.

Reference

DOPE - Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)