Deformable vistr: Spatio temporal deformable attention for video instance segmentation

This is the official implementation of the DefVisTR paper:

Installation

We provide instructions how to install dependencies via conda. First, clone the repository locally:

git clone https://github.com/skrya/DefVIS.git

Then, install PyTorch 1.6 and torchvision 0.7:

conda install pytorch==1.6.0 torchvision==0.7.0

Install pycocotools

conda install cython scipy
pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
pip install git+https://github.com/youtubevos/cocoapi.git#"egg=pycocotools&subdirectory=PythonAPI"

Compile DCN module(requires GCC>=5.3, cuda>=10.0)

cd models/dcn
python setup.py build_ext --inplace

Compiling CUDA operators

cd ./models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.py

Preparation

Download and extract 2019 version of YoutubeVIS train and val images with annotations from CodeLab or YoutubeVIS. We expect the directory structure to be the following:

VisTR
├── data
│   ├── train
│   ├── val
│   ├── annotations
│   │   ├── instances_train_sub.json
│   │   ├── instances_val_sub.json
├── models
...

Compile CUDA operators

cd ./models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.py

Download the pretrained deformable DETR models [Defomable DeTR Repository (44.5 AP)](https://github.com/fundamentalvision/Deformable-DETR/tree/main) on COCO and save it to the pretrained path.

Training

Training of the model requires 4 GPU cards with each 15GB.

To train baseline VisTR on a single node with 4 gpus for 50 epochs (Bsz 1):

python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --backbone resnet50 --ytvos_path /mnt/data/ytvis/ --masks --pretrained_weights ../VisTR/<deformable_detr_coco_r50>.pth

Inference

python inference.py --masks --model_path /mnt/data/exps/r50_def_enc_VisTR/checkpoint0059.pth --save_path /mnt/data/exps/results.json --img_path /mnt/data/ytvis/valid/JPEGImages/ --ann_path /mnt/data/ytvis/valid_vis_codelab.json  --backbone resnet50

License

DefVIS is released under the Apache 2.0 license. Please see the LICENSE file for more information.

Acknowledgement

We would like to thank the Deformable DETR open-source project for its awesome work, part of the code are modified from its project.

Citation

Please consider citing our paper in your publications if the project helps your research. BibTeX reference is as follow.

@inproceedings{yarram2022deformable,
  title={Deformable vistr: Spatio temporal deformable attention for video instance segmentation},
  author={Yarram, Sudhir and Wu, Jialian and Ji, Pan and Xu, Yi and Yuan, Junsong},
  booktitle={ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={3303--3307},
  year={2022},
  organization={IEEE}
}