by Kwanyong Park, Sanghyun Woo, Seoung Wug Oh, In So Kweon, and Joon-Young Lee
CVPR 2022
[arXiv] [PDF] [YouTube] [Poster]
Recently, memory-based approaches show promising results on semi-supervised video object segmentation. These methods predict object masks frame-by-frame with the help of frequently updated memory of the previous mask. Different from this per-frame inference, we investigate an alternative perspective by treating video object segmentation as clip-wise mask propagation. In this per-clip inference scheme, we update the memory with an interval and simultaneously process a set of consecutive frames (i.e. clip) between the memory updates. The scheme provides two potential benefits: accuracy gain by clip-level optimization and efficiency gain by parallel computation of multiple frames. To this end, we propose a new method tailored for the perclip inference, namely PCVOS.
The following tables summarize the results of PCVOS under different clip lengths. The inference speed (FPS) was measured using a single NVIDIA RTX A6000. We also provide Youtube video for visual comparison between PCVOS and other methods.
| Model | Clip Length | FPS | Mean | J Seen | F Seen | J Unseen | F Unseen | Pre-computed Results |
|---|---|---|---|---|---|---|---|---|
| PCVOS | 5 | 11.5 | 84.6 | 82.6 | 87.3 | 80.0 | 88.3 | Google Drive |
| PCVOS | 10 | 24.4 | 84.1 | 82.3 | 87.0 | 79.5 | 87.5 | Google Drive |
| PCVOS | 15 | 30.7 | 83.6 | 81.9 | 86.4 | 79.1 | 87.1 | Google Drive |
| PCVOS | 25 | 33.8 | 83.0 | 81.4 | 85.8 | 78.6 | 86.2 | Google Drive |
| Model | Clip Length | FPS | Mean | J Seen | F Seen | J Unseen | F Unseen | Pre-computed Results |
|---|---|---|---|---|---|---|---|---|
| PCVOS | 5 | 13.4 | 84.6 | 83.0 | 88.0 | 79.6 | 87.9 | Google Drive |
| PCVOS | 10 | 27.7 | 84.0 | 82.7 | 87.7 | 78.7 | 86.8 | Google Drive |
| PCVOS | 15 | 33.9 | 83.8 | 82.6 | 87.4 | 78.4 | 86.6 | Google Drive |
| PCVOS | 25 | 36.9 | 83.3 | 82.2 | 86.9 | 78.1 | 85.9 | Google Drive |
This repository is tested in the following environment:
- Python
3.7 - PyTorch
1.8.1 - torchvision
0.9.1 - timm
0.3.2 - OpenCV
4.2.0
Download the validation split of YouTube-VOS 2018/2019 and place them under ./data/.
You can either manually download it from the official website or use the provided download_datasets.py at STCN.
The resulting folder structure should look like below:
PCVOS
├── ...
├── data
│ ├── YouTube
│ │ ├── all_frames
│ │ │ ├── valid_all_frames
│ │ ├── valid
│ ├── YouTube2018
│ │ ├── all_frames
│ │ │ ├── valid_all_frames
│ │ ├── valid
├── ...Please download the pre-trained weights and put it in ./saves/.
Then, you can run the provided inference script (inference_pretrained_pcvos.py) and it will produce the predictions under different clip lengths.
We also provide other pre-computed results.
If you find our work or code useful for your research, please cite our paper.
@inproceedings{park2022per,
title={Per-Clip Video Object Segmentation},
author={Park, Kwanyong and Woo, Sanghyun and Oh, Seoung Wug and Kweon, In So and Lee, Joon-Young},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={1352--1361},
year={2022}
}This repository is based on the following code bases. We thank all the contributors.
The source code is released under the GNU General Public License v3.0 Licence (please refer here for details.)