By Yunhang Shen, Rongrong Ji, Yan Wang, Yongjian Wu, Liujuan Cao.
CVPR 2019 Paper.
This project is based on Detectron.
WS-JDS is released under the Apache 2.0 license. See the NOTICE file for additional details.
If you find WS-JDS useful in your research, please consider citing:
@inproceedings{WS-JDS_2019_CVPR,
author = {Shen, Yunhang and Ji, Rongrong and Wang, Yan and Wu, Yongjian and Cao, Liujuan},
title = {Cyclic Guidance for Weakly Supervised Joint Detection and Segmentation},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2019},
}
Requirements:
- NVIDIA GPU, Linux, Python2
- Caffe2 in pytorch v1.0.1, various standard Python packages, and the COCO API; Instructions for installing these dependencies are found below
Clone the pytorch repository:
# pytorch=/path/to/clone/pytorch
git clone https://github.com/pytorch/pytorch.git $pytorch
cd $pytorch
git checkout v1.0.1
git submodule update --init --recursive
Install Python dependencies:
pip install -r $pytorch/requirements.txt
Build caffe2:
cd $pytorch
sudo USE_OPENCV=On USE_LMDB=On BUILD_BINARY=On python2 setup.py install
Install the COCO API:
# COCOAPI=/path/to/clone/cocoapi
git clone https://github.com/cocodataset/cocoapi.git $COCOAPI
cd $COCOAPI/PythonAPI
# Install into global site-packages
make install
# Alternatively, if you do not have permissions or prefer
# not to install the COCO API into global site-packages
python setup.py install --user
Note that instructions like # COCOAPI=/path/to/install/cocoapi indicate that you should pick a path where you'd like to have the software cloned and then set an environment variable (COCOAPI in this case) accordingly.
Install the pycococreator:
pip install git+git://github.com/waspinator/pycococreator.git@0.2.0
Clone the WS-JDS repository:
# WS-JDS=/path/to/clone/WS-JDS
git clone https://github.com/shenyunhang/WS-JDS.git $WS-JDS
cd $WS-JDS
git submodule update --init --recursive
Install Python dependencies:
pip install -r requirements.txt
Set up Python modules:
make
Build the custom C++ operators library:
mkdir -p build && cd build
cmake .. -DCMAKE_CXX_FLAGS="-isystem $pytorch/third_party/eigen -isystem $/pytorch/third_party/cub"
make
Please follow this to creating symlinks for PASCAL VOC.
Download MCG proposal from here to detectron/datasets/data, and transform it to pickle serialization format:
cd detectron/datasets/data
tar xvzf MCG-Pascal-Main_trainvaltest_2007-boxes.tgz
cd ../../../
python tools/convert_mcg.py voc_2007_train detectron/datasets/data/MCG-Pascal-Main_trainvaltest_2007-boxes detectron/datasets/data/mcg_voc_2007_train.pkl
python tools/convert_mcg.py voc_2007_val detectron/datasets/data/MCG-Pascal-Main_trainvaltest_2007-boxes detectron/datasets/data/mcg_voc_2007_val.pkl
python tools/convert_mcg.py voc_2007_test detectron/datasets/data/MCG-Pascal-Main_trainvaltest_2007-boxes detectron/datasets/data/mcg_voc_2007_test.pkl
Download VGG16 model (VGG_ILSVRC_16_layers.caffemodel and VGG_ILSVRC_16_layers_deploy.prototxtt) and transform it to pickle serialization format:
cd $WS-JDS
mkdir -p model/
cd model
wget http://www.robots.ox.ac.uk/~vgg/software/very_deep/caffe/VGG_ILSVRC_16_layers.caffemodel
wget https://gist.githubusercontent.com/ksimonyan/211839e770f7b538e2d8/raw/ded9363bd93ec0c770134f4e387d8aaaaa2407ce/VGG_ILSVRC_16_layers_deploy.prototxt
cd ../
./scripts/convert_vgg16.sh
Download DeepLabv2_VGG16 and transform it to pickle serialization format:
cd $WS-JDS
cd model
wget http://liangchiehchen.com/projects/released/deeplab_aspp_vgg16/prototxt_and_model.zip
unzip prototxt_and_model.zip
cd ..
python tools/pickle_caffe_blobs.py --prototxt model/train.prototxt --caffemodel model/init.caffemodel --output model/init.pkl
python tools/combine_deeplab_and_original_vgg16.py model/VGG_ILSVRC_16_layers_v1.pkl model/init.pkl model/vgg16_init.pkl
Noted that this requires to instal caffe1 separately, as caffe1 specific proto is removed in pytorch v1.0.1. See this.
You can download vgg16_init.pkl from this link.
You may also need to modify the below config files to point TRAINING.WEIGHTS to vgg16_init.pkl.
./scripts/train_wsl.sh --cfg configs/voc_2007/ws-jds_VGG16-C5D_1x.yaml OUTPUT_DIR experiments/ws-jds_vgg16_voc2007_`date +'%Y-%m-%d_%H-%M-%S'`
The final model and log can be downloaded from here.
Noted that the results reported in the paper are based on Caffe2 in pytorch v0.4.1, while this repository is based on v1.0.1.
And upgrade it to v1.2.0 may reduce the performance by ~2% mAP on PASCAL VOC, which is weird.