Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
.make
.readme
examples/grasp_fusion
grasp_fusion_lib
ros/grasp_fusion
tests
.gitignore
MANIFEST.in
Makefile
README.md
requirements.txt
setup.cfg
setup.py

README.md

grasp-fusion

GraspFusion: Realizing Complex Motion by Learning and Fusing Grasp Modalities with Instance Segmentation (ICRA2019)

Shun Hasegawa*, Kentaro Wada*, Shingo Kitagawa, Yuto Uchimi, Kei Okada, Masayuki Inaba (* equal contribution)

Paper | Video | Poster | Code

Installation

Recommended

  1. Install whole jsk_apc
  2. catkin build grasp_fusion just in case

Minimum (Some features may not work)

For Examples

# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src

rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101  # CUDA10.1, cupy-cuda92 for 9.2

cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build grasp_fusion --no-deps

For Full Demonstration

# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src

wstool init
cat jsk_apc/.travis.rosinstall >> .rosinstall
cat jsk_apc/.travis.rosinstall.kinetic >> .rosinstall
wstool update -j -1

rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101  # CUDA10.1, cupy-cuda92 for 9.2

cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build grasp_fusion

Examples

Without ROS

# item data (originally provided at Amazon Picking Challenge)
cd examples/grasp_fusion/instance_segmentation
./view_item_data_all.py

# training data (synthetic), testing data (real)
cd examples/grasp_fusion/instance_segmentation
./view_dataset.py --split train
./view_dataset.py --split test

# object-class-agnostic instance segmentation with Mask R-CNN
cd examples/grasp_fusion/instance_segmentation
./train.py --gpu 0


Figure: Visualization on Test Data (Top: ground truth, Bottom: prediction)

cd examples/grasp_fusion/affordance_segmentation
./get_heightmaps.py
./view_pinch_dataset.py
./view_suction_dataset.py

cd examples/grasp_fusion/affordance_segmentation
./train.py pinch --modal rgb+depth --resolution 15 --gpu 0


Figure: Visualization on Test Data (Top: ground truth, Bottom: prediction)

cd examples/grasp_fusion/affordance_segmentation
./train.py suction --modal rgb+depth --gpu 0


Figure: Visualization on Test Data (Top: ground truth, Bottom: prediction)

cd examples/grasp_fusion/primitive_matching
./get_primitives_poses.py


Figure: Visualization of Primitives (Red: suction, Green: pinch, Yellow: parallel-graspfusion)

With ROS

roslaunch grasp_fusion sample_instance_segmentation.launch


Figure: Predicted Instance Masks

roslaunch grasp_fusion sample_affordance_segmentation.launch affordance:=pinch


Figure: Predicted Pinch Points

roslaunch grasp_fusion sample_affordance_segmentation.launch affordance:=suction


Figure: Predicted Suction Points

roslaunch grasp_fusion sample_sole_affordance_segmentation.launch


Figure: Predicted Pinch Points Ignoring Inter-Object Relationship

roslaunch grasp_fusion sample_primitive_matching.launch


Figure: 3D Suction Points Predicted

# Holistic Integration for Demonstration
roslaunch grasp_fusion baxter.launch
roslaunch grasp_fusion setup_for_stow.launch
roslaunch grasp_fusion stow.launch
# Experiments of Grasp Stability
roslaunch grasp_fusion baxter.launch
roslaunch grasp_fusion setup_for_stow.launch
roslaunch grasp_fusion stow.launch main:=false
roseus `rospack find grasp_fusion`/euslisp/eval-picking.l

# in Euslisp interpreter
(eval-picking-init :ctype :larm-head-controller :moveit t)
(eval-picking-mainloop :larm)
## Please see warn messages and source codes for optional settings

Citation

@inproceedings{Hasegawa:etal:ICRA2019,
  title={{GraspFusion}: Realizing Complex Motion by Learning and Fusing Grasp Modalities with Instance Segmentation},
  author={Shun Hasegawa, Kentaro Wada, Shingo Kitagawa, Yuto Uchimi, Kei Okada, Masayuki Inaba},
  booktitle={{Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)}},
  year={2019},
}

Advanced Usage (At Your Own Risk)

Installation

make install  # Python3
# make install2  # Python2

Usage

source .anaconda/bin/activate
python -c 'import grasp_fusion_lib'

Testing

make lint
make test
You can’t perform that action at this time.