Skip to content
Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation (ACM MM2019)
Jupyter Notebook Python Other
Branch: master
Clone or download
Jack12xl Update README.md
Add baiduyunpan
Latest commit 01fcf21 Nov 4, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
checkpoints add git keep Aug 14, 2019
cocoapi push Aug 24, 2019
data
datasets add gitkeep for datasets Aug 24, 2019
evaluate
modal clean version Aug 16, 2019
nms add nmsl Aug 24, 2019
results clean version Aug 16, 2019
roialign git add .so file Aug 24, 2019
scripts init with gitignore Aug 14, 2019
.gitignore add gitkeep for datasets Aug 24, 2019
LICENSE
README.md
amodal_test.py clean version Aug 16, 2019
amodal_train.py correct data-type Aug 19, 2019
config.py init with gitignore Aug 14, 2019
model.py clean version Aug 16, 2019
utils.py init with gitignore Aug 14, 2019

README.md

This is the code repo of SLN-Amodal. Paper could be downloaded here.

This repository is in large parts based on Multimodallearning's Mask_RCNN, we also borrow the amodal evaluation code from AmodalMask and COCO API. The training and evaluation dataset are referenced from COCOA and D2SA. We would like to thank each of them for their kindly work.

SLN

In this work, we demonstrate yet another approach to tackle the amodal segmentation problem. Specifically, we first introduce a new representation, namely a semantics-aware distance map (sem-dist map), to serve as our target for amodal segmentation instead of the commonly used masks and heatmaps. The sem-dist map is a kind of level-set representation, of which the different regions of an object are placed into different levels on the map according to their visibility. It is a natural extension of masks and heatmaps, where modal, amodal segmentation, as well as depth order information, are all well-described. Then we also introduce a novel convolutional neural network (CNN) architecture, which we refer to as semantic layering network, to estimate sem-dist maps layer by layer, from the global-level to the instance-level, for all objects in an image. Extensive experiments on the COCOA and D2SA datasets have demonstrated that our framework can predict amodal segmentation, occlusion and depth order with state-of-the-art performance.

Authors:

Ziheng Zhang*, Anpei Chen*, Ling Xie, Jingyi Yu, Shenghua Gao

Set up environment

  1. We specify pytorch==0.4.0

    Or you can use Ananconda to create new environment(strongly recommanded) in root directory by

    conda create -n SLN-env
    conda install pytorch=0.4.0 cuda90 -c pytorch
    conda install -c conda-forge scikit-image
    conda install -c anaconda cudatoolkit==9.0
    conda install tqdm
    pip install tensorboardX
  2. Configure COCOAPI

    We modify the COCOAPI to meet our need. Here you have to soft link the pycocotool to the root directory for invoking.

    ln -s /path/to/pycocotool /path/to/our/root/diectory

Datasets

  • Download pre-trained weights and unzip the package to the root directory.
  • we provide layer based D2SA and COCOA dataset. We also provide some scripts that you can convert the original amodal annotation [COCOA,D2SA] to our layer based annotation. If you find those two dataset are useful, please cite their COCOA and D2SA papers.
  • BaiduYunPan link with verify code:yr2i
  • Folder structure
      ├──  datasets                       - dataset folder
      │    └── coco_amodal 
      |        └── annotations    
      |            └── COCO_amodal_val[train]2014.json
      |        └── val2014             
      |            └── ###.jpg ###.npz 
      |        └── train2014           
      |            └── ###.jpg ###.npz 
      │    └── D2S                
      |        └── annotations    
      |            └── COCO_amodal_val[train]2014.json
      |        └── val2014             
      |            └── ###.jpg ###.npz 
      |        └── train2014           
      |            └── ###.jpg ###.npz 
      ├──  checkpoints                 
      |    └──COCOA[D2SA,deeplabv2,mask_rcnn_coco].pth     
      ├──  pycocotool                     - soft link to cocoapi/PythonAPI/pycocotool              
    

Usage

  • For training,

    python amodal_train.py train --dataset ./datasets/coco_amodal --model coco
    python amodal_train.py train --dataset ./datasets/D2S --model D2S
  • For evaluate,

    python amodal_train.py evaluate --dataset ./datasets/coco_amodal --model ./checkpoints/COCOA.pth --data_type COCOA
    python amodal_train.py evaluate --dataset ./datasets/D2S --model ./checkpoints/D2SA.pth --data_type D2SA
  • For test images,
    python amodal_test.py 
    you can modify the path to your image folder inside the script.

Citation

If you find this code useful to your research, please consider citing:

@inproceedings{zhang2019amodal,
  title={Learning Semantics-aware Distance Map with Semantics Layering Network for Amodal Instance Segmentation},
  author={Zhang, Zi-Heng and Chen, An-Pei and Xie, Ling and Yu, Jing-Yi and Gao, Sheng-Hua},
  booktitle={2019 ACM Multimedia Conference on Multimedia Conference},
  pages={1--9},
  year={2019},
  organization={ACM}
}
You can’t perform that action at this time.