ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging

This repository contains code to create the human grasp contact maps, presented in the paper

ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging - Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays, CVPR 2019

Paper (CVPR 2019 Oral) | Explore the dataset | Poster | Slides

Please see contactdb_prediction for code to perform the contactmap prediction experiments presented in the paper.

This code is in the process of being documented. Feel free to open an issue if you need urgent help.

Dataset Download:

We offer both processed and raw forms of the data.

Processed Data

Contact Maps (Textured Meshes) (11.5 GB). If you also need the RGBD-Thermal images, 6-DOF object poses, and image masks, use this Dropbox link (91 GB) instead. We acknowledge the creators of the 3D models (spreadsheet). The models were modified in terms of geometry and texture to create the contact maps. Contact maps are released for non-commercial use.

Raw Data

ROS bagfiles (1.46 TB): Compressed 30 Hz RGB-D-Thermal data streams.

3D Printing

3D models and licensing information

This spreadsheet has links to original 3D models, along with scaling information. We printed the objects at 15% hexagonal honeycomb infill with PLA material. Contact Samarth Brahmbhatt at samarth.robo@gatech.edu if links in that spreadsheet are dead.

We designed a mount for rigidly attaching the FLIR Boson camera to the Kinect v2, you can find it here.

Note: these models are suitable for 3D printing, but some characteristics like sharp triangles make them unsuitable for texture mapping. Please use the re-meshed .ply models from the Processed Data section above for texture mapping. All the object poses, camera poses etc. are defined with respect to those re-meshed models.

This article is a good resource for getting files ready for 3D printing. Netfabb has a free student license.

Setup

Using Processed Data

The repository includes some handy Python scripts in the scripts directory. You can ignore all the C++ code.

1. Install Open3D, numpy, and matplotlib.
2. Download this repository: git clone https://github.com/samarth-robo/contactdb_utils.
3. Download the Processed Data, and make the symlink: ln -s DOWNLOAD-DIR data/contactdb_data.
4. If you want to perform machine learning experiments, check out the contactdb_prediction repository.

Using Raw Data

Our code is a ROS package that has been tested on Ubuntu 16.04 LTS with ROS Kinetic Kame.

1. Install dependencies
2. Set up a Catkin workspace in ~/catkin_ws.
3. Download this repository and my fork of cv_camera and compile the ROS package:

$ cd ~/catkin_ws/src
$ git clone https://github.com/samarth-robo/contactdb_utils
$ git clone https://github.com/samarth-robo/cv_camera
$ cd ..
$ catkin_make

4. Download Raw Data and 3D Models, and make symlinks: ln -s DATA-DOWNLOAD-DIR data/contactdb_data, ln -s 3D-MODEL-DOWNLOAD-DIR data/contactdb_3d_models.
5. Download and setup the ICP code for object pose estimation:

$ git clone https://github.com/samarth-robo/ICP-gui
$ cd ICP-gui
$ mkdir build && cd build && make -j

6. Create a symlink to the build directory:

$ cd contactdb_utils
$ ln -s <ICP-gui dir>/build ICP_build

Visualizing Contact Maps

This script applies our post-processing to the contact map, and shows it in interactive 3D. You need to either download the processed data, or process the raw data before running it.

cd contactdb_utils/scripts
python show_contactmap.py --object_name <object name> --session <participant number 1-50> --instruction <use | handoff>

Documents:

• Recording your own data
• Cleaning up 3D models of objects
• Processing the recorded data
• Various files produced during data processing

Citation

@inproceedings{brahmbhatt2018contactdb,
  title={{ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging}},
  author={Samarth Brahmbhatt and Cusuh Ham and Charles C. Kemp and James Hays},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019},
  note={\url{https://contactdb.cc.gatech.edu}}
}