A Toolbox for Video Restoration

Authors

Jing Lin*, Yuanhao Cai*, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool

News

• 2022.12.08 : Pretrained model, training/testing log, visual results of FGST on GoPro and DVD dataset are released. S2SVR will be provided later.🔥

• 2022.11.30 : Data preparation codes of GoPro and DVD are provided. 🔆

• 2022.08.05 : Pretrained model of FGST on GOPRO dataset is released. 💫

• 2022.05.14 : Our FGST and S2SVR are accepted by ICML2022. 🚀

Super-Resolution	Deblur	Compressed Video Enhancement

Papers

• Flow-Guided Sparse Transformer for Video Deblurring (ICML 2022)
• Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration (ICML 2022)

Method	Dataset	Pretrained Model	Training Log	Testing Log	Visual Result	Quantitative Result
FGST	GoPro	Google Drive / Baidu Disk	Google Drive / Baidu Disk	Google Drive / Baidu Disk	Google Drive / Baidu Disk	33.02 / 0.947
FGST	DVD	Google Drive / Baidu Disk	Google Drive / Baidu Disk	Google Drive / Baidu Disk	Google Drive / Baidu Disk	33.50 / 0.945

Note: access code for Baidu Disk is VR11

1. Create Environment:

• Python 3 (Recommend to use Anaconda)

• NVIDIA GPU + CUDA

• Python packages:

pip install torchvision==0.9.0  torch==1.8.0  torchaudio==0.8.0
pip install -r requirements.txt
pip install openmim
mim install mmcv-full==1.5.0
pip install -v -e .
pip install cupy-cuda101==7.7.0

2. Prepare Dataset:

Download the datasets (GOPRO,DVD,REDS,VIMEO,MFQE-v2) and and recollect them as the following form:

|--VR-Baseline
    |--data
    	|-- GoPro
    	    |-- test
    	    |-- train
    	|-- DVD
    	    |-- quantitative_datasets
    	      |-- GT
    	      |-- LQ
    	    |-- qualitative_datasets
    	|-- REDS
    	    |-- train_sharp_bicubic
    	    |-- train_sharp
    	|-- VIMEO
    	    |-- BIx4
    	    |-- GT
    	|-- MFQEV2
    	    |-- test
    	    |-- train

You can run the following command to recollect GoPro and DVD dataset:

cd VR-Baseline/data_preparation

# recollect GoPro dataset
python GoPro_Util.py --input_path INPUT_PATH --save_path SAVE_PATH

# recollect DVD dataset
python DVD_Util.py --input_path INPUT_PATH --save_path SAVE_PATH

You need to replace INPUT_PATH and SAVE_PATH with your own path.

3. Training:

cd VR_Baseline

# training FGST on GoPro dataset
bash tools/dist_train.sh configs/FGST_deblur_gopro.py 8

# training FGST on DVD dataset
bash tools/dist_train.sh configs/DVD_deblur_gopro.py 8

# training S2SVR on GoPro dataset
bash tools/dist_train.sh configs/S2SVR_deblur_gopro.py 8

# training S2SVR on REDS dataset
bash tools/dist_train.sh configs/S2SVR_sr_reds4.py 8

# training S2SVR on VIMEO dataset
bash tools/dist_train.sh configs/S2SVR_sr_vimeo.py 8

# training S2SVR on MFQEv2 dataset
bash tools/dist_train.sh configs/S2SVR_vqe_mfqev2.py 8

The training log, trained model will be available in VR-Baseline/experiments/ .

4. Testing:

Download pretrained model and run the following command.

To test on benchmark:

cd VR_Baseline

# testing FGST on GoPro dataset
bash tools/dist_train.sh configs/FGST_deblur_gopro_test.py 8

# testing FGST on DVD dataset
bash tools/dist_train.sh configs/FGST_deblur_dvd_test.py 8

5. TODO

These works are mostly done during the internship at HUAWEI Noah's Ark Lab. Due to the limitation of company regulations, the original pre-trained models can not be transferred and published here. We will retrain more models and open-source them when we have enough GPUs as soon as possible.

• More data preparation codes
• More Pretrained Models
• Inference Results
• MFQEv2 dataloader

6. Acknowledgement.

We refer to codes from BasicVSR++ and mmediting. Thanks for their awesome works.

7. Citation

If this repo helps you, please consider citing our works:

# FGST
@inproceedings{fgst,
  title={Flow-Guided Sparse Transformer for Video Deblurring},
  author={Lin, Jing and Cai, Yuanhao and Hu, Xiaowan and Wang, Haoqian and Yan, Youliang and Zou, Xueyi and Ding, Henghui and Zhang, Yulun and Timofte, Radu and Van Gool, Luc},
  booktitle={ICML},
  year={2022}
}


# S2SVR
@inproceedings{seq2seq,
  title={Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration},
  author={Lin, Jing  and Hu, Xiaowan and Cai, Yuanhao and Wang, Haoqian and Yan, Youliang and Zou, Xueyi and Zhang, Yulun and Van Gool, Luc},
  booktitle={ICML},
  year={2022}
}