Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
.make
.readme
examples
instance_occlsegm_lib
ros
tests
.gitignore
MANIFEST.in
Makefile
README.md
requirements.txt
setup.cfg
setup.py

README.md

instance-occlsegm

Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter (ICRA2019)

Kentaro Wada, Kei Okada, Masayuki Inaba

Paper | Video | Poster | Supplimentary

Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects (IROS2018)

Kentaro Wada, Shingo Kitagawa, Kei Okada, Masayuki Inaba

Paper | Video | Slides | Poster

Installation

For Examples

# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src

rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101  # CUDA10.1, cupy-cuda92 for 9.2

cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build instance_occlsegm --no-deps

For Full Demonstration

# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src

wstool init
cat jsk_apc/.travis.rosinstall >> .rosinstall
cat jsk_apc/.travis.rosinstall.kinetic >> .rosinstall
wstool update -j -1

rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101  # CUDA10.1, cupy-cuda92 for 9.2

cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build instance_occlsegm

Example

Without ROS

# Dataset for instance (roi-level) occlusion segmentation
cd examples/instance_occlsegm/instance_occlusion_segmentation
./view_dataset_occlusion.py

# Dataset for semantic (image-level) occlusion segmentation
cd examples/instance_occlsegm/semantic_segmentation
./view_dataset.py

# Dataset for joint learning
cd examples/instance_occlsegm/panoptic_occlusion_segmentation
./view_dataset.py

# Training script for instance-only vs. joint-learning
cd examples/instance_occlsegm/panoptic_occlusion_segmentation
./train.py --gpu 0 --notrain pix  # instnce-only
./train.py --gpu 0                # joint-learning

# multi-gpu training for faster training with larger dataset
# mpirun -n 4 ./train.py --multinode --notrain pix --dataset occlusion+synthetic
# mpirun -n 4 ./train.py --multinode --pix-loss-scale 0.25 --dataset occlusion+synthetic

./demo.py logs/<log_dir>


Comparison: instance-only vs. joint-learning (included in the supplimentary)

Results

BackboneModelDatasetLambdamPQ
ResNet50instance-onlyocclusion-41.0
joint-learning0.2542.2
instance-onlyocclusion+synthetic-47.3
joint-learning0.2548.9
ResNet101instance-onlyocclusion-43.5
joint-learning0.2544.5
instance-onlyocclusion+synthetic-50.0
joint-learning0.2550.9

Dataset

The real-world dataset annotated by human can be downloaded from following links:

With ROS

# Find occluded target and plan the next target
roslaunch instance_occlsegm sample_panoptic_segmentation.launch


Figure: Picking Order Planning for the White Binder (From left: RGB, visible regions, occluded regions, next target)

# Pick-and-Place demo
roslaunch instance_occlsegm baxter.launch

roslaunch instance_occlsegm setup.launch
# for target picking, change the ~context and ~target params in setup.lauch

roscd instance_occlsegm/euslisp
> (upick-upick)  # random picking
> (pick-pick)    # target picking

Random / Target Picking

In-Bin Manipulation

Citation

@inproceedings{Wada:etal:ICRA2019,
  title={Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter},
  author={Kentaro Wada, Kei Okada, Masayuki Inaba},
  booktitle={{Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)}},
  year={2019},
}
@inproceedings{Wada:etal:IROS2018,
  title={Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects},
  author={Kentaro Wada, Shingo Kitagawa, Kei Okada, Masayuki Inaba},
  booktitle={{Proceedings of the IEEE/RSJ International Conference on Robotics and Intelligent Systems (IROS)}},
  year={2018},
}

Advanced Usage (At Your Own Risk)

Installation

make install  # Python3
# make install2  # Python2

Usage

source .anaconda/bin/activate
python -c 'import instance_occlsegm_lib'

Testing

make lint
make test
You can’t perform that action at this time.