instance_occlsegm

instance-occlsegm

Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter (ICRA2019)

Kentaro Wada, Kei Okada, Masayuki Inaba

Paper | Video | Poster | Supplimentary

Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects (IROS2018)

Kentaro Wada, Shingo Kitagawa, Kei Okada, Masayuki Inaba

Paper | Video | Slides | Poster

Installation

For Examples

# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src

rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101  # CUDA10.1, cupy-cuda92 for 9.2

cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build instance_occlsegm --no-deps

For Full Demonstration

# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src

wstool init
cat jsk_apc/.travis.rosinstall >> .rosinstall
cat jsk_apc/.travis.rosinstall.kinetic >> .rosinstall
wstool update -j -1

rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101  # CUDA10.1, cupy-cuda92 for 9.2

cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build instance_occlsegm

Example

Without ROS

# Dataset for instance (roi-level) occlusion segmentation
cd examples/instance_occlsegm/instance_occlusion_segmentation
./view_dataset_occlusion.py

# Dataset for semantic (image-level) occlusion segmentation
cd examples/instance_occlsegm/occlusion_segmentation
./view_dataset.py

# Dataset for joint learning
cd examples/instance_occlsegm/panoptic_occlusion_segmentation
./view_dataset.py

# Training script for instance-only vs. joint-learning
cd examples/instance_occlsegm/panoptic_occlusion_segmentation
./train.py --gpu 0 --notrain pix  # instnce-only
./train.py --gpu 0                # joint-learning

# multi-gpu training for faster training with larger dataset
# mpirun -n 4 ./train.py --multinode --notrain pix --dataset occlusion+synthetic
# mpirun -n 4 ./train.py --multinode --pix-loss-scale 0.25 --dataset occlusion+synthetic

./demo.py logs/<log_dir>

Comparison: instance-only vs. joint-learning (included in the supplimentary)

• Full log (ResNet50, joint-learning)
• Full log (ResNet50, instance-only)

Results

Backbone	Model	Dataset	Lambda	mPQ
ResNet50	instance-only	occlusion	-	41.0
	joint-learning		0.25	42.2
	instance-only	occlusion+synthetic	-	47.3
	joint-learning		0.25	48.9
ResNet101	instance-only	occlusion	-	43.5
	joint-learning		0.25	44.5
	instance-only	occlusion+synthetic	-	50.0
	joint-learning		0.25	50.9

Dataset

The real-world dataset annotated by human can be downloaded from following links:

• 20180204_splits.zip
• 20180730_splits.zip

With ROS

# Find occluded target and plan the next target
roslaunch instance_occlsegm sample_panoptic_segmentation.launch

Figure: Picking Order Planning for the White Binder (From left: RGB, visible regions, occluded regions, next target)

# Pick-and-Place demo
roslaunch instance_occlsegm baxter.launch

roslaunch instance_occlsegm setup.launch
# for target picking, change the ~context and ~target params in setup.lauch

roscd instance_occlsegm/euslisp
> (upick-upick)  # random picking
> (pick-pick)    # target picking

Random / Target Picking

In-Bin Manipulation

Citation

@inproceedings{Wada:etal:ICRA2019,
  title={Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter},
  author={Kentaro Wada, Kei Okada, Masayuki Inaba},
  booktitle={{Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)}},
  year={2019},
}
@inproceedings{Wada:etal:IROS2018,
  title={Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects},
  author={Kentaro Wada, Shingo Kitagawa, Kei Okada, Masayuki Inaba},
  booktitle={{Proceedings of the IEEE/RSJ International Conference on Robotics and Intelligent Systems (IROS)}},
  year={2018},
}

---

Advanced Usage (At Your Own Risk)

Installation

make install  # Python3
# make install2  # Python2

Usage

source .anaconda/bin/activate
python -c 'import instance_occlsegm_lib'

Testing

make lint
make test