Skip to content
Siamese Mask R-CNN model for one-shot instance segmentation
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
experiments fixed gpu count Apr 10, 2019
figures visualizations added Mar 18, 2019
lib fixed indents Apr 10, 2019
.gitignore
LICENSE chmod diff Sep 11, 2018
README.md
evaluate.ipynb remove deprecated active_classes settings Apr 10, 2019
install_requirements.ipynb
train.ipynb remove deprecated active_classes settings Apr 10, 2019

README.md

Siamese Mask R-CNN

This is the official implementation of Siamese Mask R-CNN from One-Shot Instance Segmentation. It is based on the Mask R-CNN implementation by Matterport.

The repository includes:

  • Source code of Siamese Mask R-CNN
  • Training code for MS COCO
  • Evaluation on MS COCO metrics (AP)
  • Training and evaluation of one-shot splits of MS COCO
  • Training code to reproduce the results from the paper
  • Pre-trained weights for ImageNet
  • Pre-trained weights for all models from the paper
  • Code to evaluate all models from the paper
  • Code to generate result figures

One-Shot Instance Segmentation

One-shot instance segmentation can be summed up as: Given a query image and a reference image showing an object of a novel category, we seek to detect and segment all instances of the corresponding category (in the image above ‘person’ on the left, ‘car’ on the right). Note that no ground truth annotations of reference categories are used during training. This type of visual search task creates new challenges for computer vision algorithms, as methods from metric and few-shot learning have to be incorporated into the notoriously hard tasks ofobject identification and segmentation. Siamese Mask R-CNN extends Mask R-CNN - a state-of-the-art object detection and segmentation system - with a Siamese backbone and a matching procedure to perform this type of visual search.

Installation

  1. Clone this repository
  2. Prepare COCO dataset as described below
  3. Run the install_requirements.ipynb notebook to install all relevant dependencies.

Requirements

Linux, Python 3.4+, Tensorflow, Keras 2.1.6, cython, scikit_image 0.13.1, h5py, imgaug and opencv_python

Prepare COCO dataset

The model requires MS COCO and the CocoAPI to be added to /data.

mkdir data
cd data
git clone https://github.com/cocodataset/cocoapi.git

It is recommended to symlink the dataset root of MS COCO.

ln -s $PATH_TO_COCO$/coco coco

If unsure follow the instructions of the Matterport Mask R-CNN implementation.

Get pretrained weights

Get the pretrained weights from the releases menu and save them to /checkpoints.

Training

To train a small version of siamese mask r-cnn on MS COCO simply follow the instructions in the training.ipynb notebook. This model runs on a single GPU with 12GB memory.

To train the models reported in the paper run the notebooks provided in experiments. Those models need 4 GPUs with 12GB memory each.

Evaluation

To evaluate and visualize a models results run the evaluation.ipynb notebook. Make sure to use the same config as used for training the model.

To evaluate the models reported in the paper run the evaluation notebook provided in experiments. Each model will be evaluated 5 times to compensate for the stochastic effects introduced by randomly choosing the reference instances. The final result is the mean of those five runs.

Model description

Siamese Mask R-CNN is designed as a minimal variation of Mask R-CNN which can perform the visual search task described above. For more details please read the paper.

Citation

If you use this repository or want to reference our work please cite our paper:

@article{michaelis_one-shot_2018,
    title = {One-Shot Instance Segmentation},
    author = {Michaelis, Claudio and Ustyuzhaninov, Ivan and Bethge, Matthias and Ecker, Alexander S.},
    year = {2018},
    journal = {arXiv},
    url = {http://arxiv.org/abs/1811.11507}
}
You can’t perform that action at this time.