OASC-Benchmark
This benchmark is an accompaniment to our SIGGRAPH'18 paper Object-Aware Guidance for Autonomous Scene Reconstruction. You can find more information from our paper and project website.
Citation
If you find our work useful in your research, please consider citing:
@article {liu_nbo_sig18,
title = {Object-Aware Guidance for Autonomous Scene Reconstruction},
author = {Ligang Liu and Xi Xia and Han Sun and Qi Shen and Juzhan Xu and Bin Chen and Hui Huang and Kai Xu},
journal = {ACM Transactions on Graphics},
volume = {37},
number = {4},
pages = {Article 104},
year = {2018}
}
Introduction
The Object-Aware Scanning Benchmark (OASC), which aims to facilitate quantitative evaluation of object-aware scene scanning and reconstruction, is a collection of Gazebo virtual scenes, based on existing scene dataset SUNCG. We select 66 appropriate human-modeled synthetic scenes which are suitable for robot autoscanning and slightly adjust them to avoid collision. The collection contains 5 scene categories, including bedroom, living room, kitchen, bathroom and office.
Evaluation metrics
You can evaluate the performance of object-aware scanning from several aspects:
Object recognition
For object recognition task, each object class label can be found in SUNCG/ModelCategoryMapping.csv
. We use SUNCG model id as Gazebo model name, so the correspondence between model name and object category is straightforward.
Single-view object detection
You can use both original rendering RGBD images from SUNCG or simulated color and depth images from Gazebo, however the latter lacks of ground truth and you may have to annotate by yourself.
Object-level segmentation
You can measure the Rand Index of the segmentation against its ground-truth. See tools/RandIndex.cpp
for details.
Object coverage rate & coverage quality
Requirements
Our benchmark and tools are mainly tested in Ubuntu 14.04 and ROS Indigo (with Gazebo 2.2.3), although other environmnet may still work. If you are a novice to ROS, you can get more information from here.
Usage
Each scene directory include a number of model directories and a room.world
file. Copy these models into your .gazebo/models/
and run gazebo room.world
to carry out a virtual scene in Gazebo. You can refer to tools/json_2_world.py
on how to transfer a SUNCG house.json
file into a Gazebo scene, however you may have to add walls by yourself.