This is the OCRTOC software package. For more information on OCRTOC please visit the homepage: www.ocrtoc.org
To take part in OCRTOC, you need to develop your own solution using this software package. After uploading your solution to the competition platform, the performance of your solution will be evaluated.
For the simulation stage, we support two simulators: Gazebo and Sapien. You can choose either of them to test your solution on your local machine. On the competition platform we use both simulators to evaluate your solution. As long as your solution works fine with either of the two simulators, it is good enough for the qualification of the real robot stage.
For the real robot stage, hardware drivers will be provided in this software package (around the end of August). Your solution will be tested on real robot hardware. The software interfaces for sensor readings and robot control are the same for both simulation and the real robot hardware. So you will not encounter interface issues when transferring your solution from simulation to the real robot.
In order for your solution to be executable both on your local machine and our competition platform (with cloud computing), we provide you a docker image with a lot of pre-installed software. You need to develop and test your solution within this docker image.
Operation system
We highly recommend you to use Ubuntu 18.04 as your operation system, because the provided software package and docker image are guaranteed to work on this OS.
Install dependencies
- Install docker engine on Ubuntu. https://docs.docker.com/engine/install/ubuntu/
- Install Nvidia container runtime https://www.celantur.com/blog/run-cuda-in-docker-on-linux/
Get the docker image
The docker image is not part of the OCRTOC software package and needs be downloaded separately. You can use either of the following addresses to download the docker image:
# Mirror address of China (ShangHai)
sudo docker pull registry.cn-shanghai.aliyuncs.com/tcc-public/ocrtoc:release1.3
# Mirror address of the United States (Silicon Valley)
sudo docker pull registry.us-west-1.aliyuncs.com/tcc-public/ocrtoc:release1.3The content of the docker image is as follows:
- Operation System: Ubuntu 18.04
- ROS melodic-desktop-full
- CUDA 10.2
- Gazebo 9.14.0
- Sapien 0.6
- A lot of simulated scenes for testing purposes, available at /root/ocrtoc_materials/scenes
- Object mesh models with texture information, available at /root/ocrtoc_materials/models
- description: The description files for hardware simulation.
- docker: The Dockerfile and build script. You can modify the Dockerfile and rebuild your docker image.
- gazebo_simulator: Setups and scripts for the Gazebo simluator.
- sapien_simluator: Setups and scripts for the Sapien simluator.
- ocrtoc_task: Scripts for task execution and evaluation. (No modifiction allowed)
- ocrtoc_solution: Setups and scripts for building the solution. Sample code is given in ocrtoc_solution/scripts/commit_solution.py (Do not change the file name!). You can modify this file to develop your own solution. In addition you can also add any new software modules to this folder. Make sure that your commit_solution.py serves as the main function of your solution and it incorporates all the needed software modules to do the task. After your solution is uploaded to the competition platform, we will run your solution by ocrtoc_solution/launch/commit_solution.launch (Do not change the file name!).
ROS topic
- /clock
- /tf*
- /arm_controller/*
- /gripper_controller/*
- /kinect/*
- /realsense/*
- /joint_group_position_controller/command
- /joint_states
ROS service
- /arm_controller/query_state
- /controller_manager/*
Note: In the simulation contest of OCRTOC, you are not allowed to use simulator topics to read the 6D pose of the objects in the simulated scenes. You must perform perception using the given object models and visual sensor readings. Violation of this rule leads to disqualification.
The target configuration of a task is defined as a list of 6D poses in a given coordinate system (frame) with corresponding object models and scale factors.
In general, we manage tasks using the ROS actionlib functionality. The script ocrtoc_solution/scripts/commit_solution.py will receive tasks in form of a callback function. Each time when a task is received, the corresponding callback function (execute_callback in commit_solution.py) is called.
Task format
string scene_id // the id of the loaded scene
string frame_id // the coordinate system, in which the 6D poses of the objects are defined
string[] object_list // the list of object model names, by which you can get the model from the file system
float64[] scale_list // the list of scale factors, using which the object model is scaled isotropically to generate the scene, typical values are: 0.5, 1, 2
geometry_msgs/Pose[] pose_list // the list of 6D object poses (in x,y,z and quaternion) defined in frame_id
geometry_msgs/Point position
float64 x
float64 y
float64 z
geometry_msgs/Quaternion orientation
float64 x
float64 y
float64 z
float64 w
---
# result definition
string status
---
# feedback
string text
Your solution
After you obtained the task information, you need to implement the core functionality of your solution in the commit_solution.py script to finish the task (object organization). An intuitive way of doing this would be like: 1) recognize the objects in the scene so that you know where to place them; 2) plan the needed motion; 3) execute the planned motion. For some tasks, you may need some kind of active percepion method due to clutters and occlusions in the scene.
Evaluation
After you finished the task, you need to publish the actionlib result topic. The format of this result is a string. We do not parse the content of this string. Instead, it is only used to activate our callback function for evaluation. So you can write anything reasonable into this string, such as "done", "finished" and so on. If you do not publish the actionlib result at all, your solution will be terminated after a predefined timeout (e.g. 10 minutes), and then the evaluation will start automatically. We highly recommend you to publish the actionlib result topic, once your solution has finished the execution. This helps us compute the execution time of your solution. For each object in the target configuration, a distance error will be calculated based on its actual pose and its target pose. There will be an upper bound for the distance error of a single object. For each task, distance errors of all the objects are summed up. The performance of the solution is ranked according to the summed distance error of all tasks. If two teams have the same performance, the team consuming less execution time will be ranked higher.
- Download the OCRTOC software package
# For example, save at $HOME dir.
cd $HOME
git clone git@github.com:OCRTOC/OCRTOC_software_package.git- Create a docker container
sudo docker run -i -d --gpus all --name ocrtoc_container \
-e DISPLAY=$DISPLAY -e QT_X11_NO_MITSHM=1 \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-v $HOME/OCRTOC_software_package:/root/ocrtoc_ws/src \
registry.cn-shanghai.aliyuncs.com/tcc-public/ocrtoc:release1.3
sudo xhost +local:`sudo docker inspect --format='{{ .Config.Hostname }}' ocrtoc_container`- Launch a simulator and a testing scene
# Load a testing scene, e.g. scene 1-1/input.world
sudo docker exec -it ocrtoc_container bash
cd /root/ocrtoc_ws
rm -rf build devel
catkin_make
# You can choose gazebo or sapien for testing
# Start gazebo
roslaunch ocrtoc_task bringup_simulator.launch simulator:=gazebo gui:=true scene:=1-1
# Or start sapien
roslaunch ocrtoc_task bringup_simulator.launch simulator:=sapien gui:=true scene:=1-1- Launch your solution
sudo docker exec -it ocrtoc_container bash
roslaunch ocrtoc_solution commit_solution.launch- Trigger task and evaluation
sudo docker exec -it ocrtoc_container bash
# For gazebo
roslaunch ocrtoc_task trigger_and_evaluation.launch simulator:=gazebo scene:=1-1
# For sapien
roslaunch ocrtoc_task trigger_and_evaluation.launch simulator:=sapien scene:=1-1You are required to submit your solution by uploading a docker image to the competition platform.
Please make sure
- Install all the dependencies into the docker image you submit.
- Disable GUI in your solution, and delete unnecessary print.
- Remove your source code and leave the "catkin_make install" results in you docker image.
- Your solution can be fully launched by the ocrtoc_solution/launch/commit_solution.launch
Submission example
# 1. Install anything you need in your docker image, such as pytorch.
sudo docker exec -it ocrtoc_container bash
pip install torch torchvision
# 2. After your code is compiled, remove unnecessary files in the container.
cd /root/ocrtoc_ws
catkin_make install
rm -rf build devel
# 3. Verify whether your solution can be launched in the following way
# In terminal 1
sudo docker exec -it ocrtoc_container bash
source /root/ocrtoc_ws/install/setup.bash
roslaunch ocrtoc_task bringup_simulator.launch simulator:=gazebo gui:=true scene:=1-1
# In terminal 2
sudo docker exec -it ocrtoc_container bash
source /root/ocrtoc_ws/install/setup.bash
roslaunch ocrtoc_solution commit_solution.launch
# In terminal 3
sudo docker exec -it ocrtoc_container bash
source /root/ocrtoc_ws/install/setup.bash
roslaunch ocrtoc_task trigger_and_evaluation.launch simulator:=gazebo scene:=1-1
# 4. Export a docker image by the docker container.
sudo docker commit ocrtoc_container your_submission_docker_image_name
# 5.Submit you docker image.
# We will provide a submission instruction on the competition platform later.
-
The reset time button in the GUI of the gazebo simulator should not be used. It will cause the robot to explode. Resetting can be done by ROS service /gazebo/reset_simulation instead.
-
In general, the physics engine used in Sapien works more stable than that in Gazebo (open dynamics engine). If you encounter weird issues while using Gazebo, please use Sapien instead.
Hardware safety configuration for the robot arm
- Joint position limits (min: -pi, max: pi).
- Joint velocity limits (max: 45 deg/s).
- A safety plane is set at 2cm above the table.
Differences between the real robot stage and the simulation stage
- The topic_name/frame_id/encoding/camera_info of cameras may changed, check the camera_image_info.md. The follwing topics are recommended. You can also make changes on the camera launch/prarameters.
- /kinect/color/image_rect_color
- /kinect/depth_to_color/image_raw
- /realsense/color/image_raw
- /realsense/aligned_depth_to_color/image_raw
- Some contents of the tf tree, check the tf.png.
- The "arm_controller" won't be launched as the deafult controller. The default controllers are the pos_traj_controller and the scaled_pos_traj_controller. If you are using moveit, you will not encounter the controller issues.
- The control values of /gripper_controller/gripper_cmd/goal. In the real robot stage, "position = 0.0" means openning the gripper to its maximum, and "position = 0.085" means closing the gripper. In addtion, the "max_effort" must be in range 30.0N - 100.0N.
Sample data collected from the real robot
- tf.bag: A ROS bag including the whole tf tree.
- realsense.bag : A ROS bag including all topics of realsense.
- kinect.bag : A ROS bag including all topics of kinect.
- topic_list of arm/cameras/gripper (/docs)
Important rules for real robot trials
Before you begin a real robot trial, make sure you follow the following rules:
- (If applicable) Get the latest release of OCRTOC software package and merge it into your code.
- (If applicable) Update your docker image by rebuilding the Dockerfile or by manual installing dependencies.
- Make sure you use motion planning before you move the robot arm, otherwise your code may result in hardware damage.
- Make sure the joint velocities generated by your code are within the allowed joint velocity limitis (max: 45 deg/s). Otherwise the hardware safety system will stop the execution of the arm movement.
Procedure of a real robot trial
For the first real robot trial you need to finish uploading your docker image at least 12 hours before your trial appointment, so that we have enough time to download your docker image. For later trials, you need to finish the uploading at least 2 hours before your trial appointment.
- Email us the following information:
- The storage link of your docker image
- If you use a private registry, you need to provide us the username and password, so that we can get access to your registry.
- Before your trial starts, we will create a docker container and launch all needed drivers.
# Create container
sudo docker run -i -d --gpus all --name trial_container --network host \
--privileged -v /dev:/dev -e DISPLAY=$DISPLAY -e QT_X11_NO_MITSHM=1 \
-v /tmp/.X11-unix:/tmp/.X11-unix ${your_image_name}
sudo xhost +local:`docker inspect --format='{{ .Config.Hostname }}' trial_container`
# Launch drivers.
sudo docker exec -it trial_container bash
source /root/ocrtoc_ws/install/setup.bash
roslaunch ocrtoc_task all_devices.launch robotip:=${robot_ip}
- During the trial, we will launch your solution and triger the task.
# Launch solution
docker exec -it trial_container bash
source /root/ocrtoc_ws/install/setup.bash
roslaunch ocrtoc_solution commit_solution_real_stage.launch
# Trigger
docker exec -it trial_container bash
source /root/ocrtoc_ws/install/setup.bash
roslaunch ocrtoc_task trigger_real_stage.launch scene:=1-1
-
During the trial, you can watch the whole procedure using zoom (we will provide the link). We will share the output of the terminal and rviz through screen sharing. You can choose the topics that you want to monitor in rviz and put your rviz launch command in "commit_solution_real_stage.launch".
-
You are allowed to update your docker image during the trial.
-
You can save the data (up to 5GB) that you want to keep under "/root/upload". After the trial we will send you the data and the log file of your solution.
If you find our work helpful, please cite the following paper （https://arxiv.org/pdf/2104.11446.pdf）
“OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation”, Ziyuan Liu, Wei Liu, Yuzhe Qin, Fanbo Xiang, Songyan Xin, Maximo A Roa, Berk Calli, Hao Su, Yu Sun, Ping Tan.
If you encounter technical issues regarding this software package, please contact us at info@ocrtoc.org