Demo code of the paper: "Learning to Segment Instances in Videos with Spatial Propagation Network", in CVPR'17 Workshop on DAVIS Challenge
Branch: master
Clone or download
Latest commit 3effc2c Mar 23, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ResNetF release Seg-with-SPN Sep 13, 2017
data/DAVIS release Seg-with-SPN Sep 13, 2017
fcn_python release Seg-with-SPN Sep 13, 2017
matlab_code first_commit Aug 15, 2017
pretrained first_commit Aug 7, 2017
python_layers Update davis_1st_frame_specific_object_layer.py Oct 13, 2017
training release Seg-with-SPN Sep 13, 2017
README.md Update README.md Mar 23, 2018
cvpr17_workshop_git.png Add files via upload Dec 24, 2017
infer_test_fgbg.py release Seg-with-SPN Sep 13, 2017
infer_test_perobj.py release Seg-with-SPN Sep 13, 2017

README.md

Learning to Segment Instances in Videos with Spatial Propagation Network

alt text

This paper is available at the 2017 DAVIS Challenge website.

Check our results in this video.

Contact: Jingchun Cheng (chengjingchun at gmail dot com)

Cite the Paper

If you find that our method is useful in your research, please cite:

@article{DAVIS2017-6th,
  author = {J. Cheng and S. Liu and Y.-H. Tsai and W.-C. Hung and S. Gupta and J. Gu and J. Kautz and S. Wang and M.-H. Yang}, 
  title = {Learning to Segment Instances in Videos with Spatial Propagation Network}, 
  journal = {The 2017 DAVIS Challenge on Video Object Segmentation - CVPR Workshops}, 
  year = {2017}
}

About the Code

  • The code released here mainly consistes of two parts in the paper: foreground segmentation and instance recognition.

  • It contains the parent net for foreground segmentation and training codes for instance recognition networks.

  • The matlab_code folder contains a simple version of our CRAF step for segmentation refinement.

Requirements

Training

  • Train the per-object recognition model.
    cd training
    python solve.py PATH_OF_MODEL PATH_OF_SOLVER
    Foe example, on the 'choreography' video for the 1st object, run:
    python solve.py ../pretrained/PN_ResNetF.caffemodel ../ResNetF/testnet_per_obj/choreography/solver_1.prototxt

Testing

  • Test the general foreground/backgroung model.
    python infer_test_fgbg.py PATH_OF_MODEL PATH_OF_RESULT VIDEO_NAME
    Foe example, on the 'lions' video, run:
    python infer_test_fgbg.py pretrained/PN_ResNetF.caffemodel results/fgbg lions

  • Test the object instance model.
    python infer_test_perobj.py MODEL_ITERATION VIDEO_NAME OBJECT_ID
    For example, on the 'lions' video for the 2nd object, run:
    python infer_test_perobj.py 3000 lions 2

  • Run example_CRAF.m in the matlab_code folder for a demo on CRAF segmentation refinement.

Download Our Segmentation Results on 2017 DAVIS Challenge

  • General foreground/background segmentation here
  • Instance-level object segmentation without refinement here
  • Final instance-level object segmentation with refinement here

Note

The model and code are available for non-commercial research purposes only.

  • 09/2017: code and model released
  • 03/2018: pre-trained model updated