Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution
This is a pytorch project for the paper Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution by Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia presented at AAAI2023.
This paper presents a new parametric representation called the Deep Parametric 3D Filters (DP3DF), which incorporates local spatiotemporal information to enable simultaneous denoising, illumination enhancement, and SR efficiently in a single encoder-and-decoder network. Also, a dynamic residual frame is jointly learned with the DP3DF via a shared backbone to further boost the SR quality.
The SDSD dataset is collected as dynamic video pairs containing low-light and normal-light videos. This dataset is consists of two parts, i.e., the indoor subset and the outdoor subset. All data is hosted on baidu pan (验证码: zcrb) and google pan There should contain "indoor_np" and "outdoor_np".
For SMID, we use its full images and also transfer the RAWdata to RGB, since our work explores low-light image enhancement in the RGB domain. You can download our processed datasets for SMID from baidu pan (验证码: btux) (the dataset can also be downloaded from google pan), and there should contain "SMID_Long_np and SMID_LQ_np".
First install Python 3. We advise you to install Python 3 and PyTorch with Anaconda:
conda create --name py36 python=3.6
source activate py36
Clone the repo and install the complementary requirements:
cd $HOME
git clone --recursive git@github.com:xiaogang00/DP3DF.git
cd DP3DF
pip install -r requirements.txt
Train the model on the corresponding dataset using the train config.
The training on indoor subset of SDSD:
python -m torch.distributed.launch --nproc_per_node 1 --master_port 4328 train.py -opt options/train/SDSD_indoor.yml --launcher pytorch
The training on outdoor subset of SDSD:
python -m torch.distributed.launch --nproc_per_node 1 --master_port 4332 train.py -opt options/train/SDSD_outdoor.yml --launcher pytorch
The training on SMID:
python -m torch.distributed.launch --nproc_per_node 1 --master_port 4317 train.py -opt options/train/SMID.yml --launcher pytorch
If you want to resume the training, you should write the location of checkpoint in "pretrain_model_G" and the path of the saved training state in "resume_state" of config file (e.g., options/test/SDSD_indoor.yml).
We use PSNR and SSIM as the metrics for evaluation. Evaluate the model on the corresponding dataset using the test config.
For the evaluation on indoor subset of SDSD, you should write the location of checkpoint in "pretrain_model_G" of options/test/SDSD_indoor.yml use the following command line:
python test.py -opt options/test/SDSD_indoor.yml
For the evaluation on outdoor subset of SDSD, you should write the location of checkpoint in "pretrain_model_G" of options/test/SDSD_outdoor.yml use the following command line:
python test.py -opt options/test/SDSD_outdoor.yml
For the evaluation on SMID, you should write the location of checkpoint in "pretrain_model_G" of options/test/SMID.yml use the following command line:
python test.py -opt options/test/SMID.yml
You can download our trained model using the following links: https://drive.google.com/file/d/12Xtgpxt9Q-3oQDH4l_yONOWhm8lPr90Q/view?usp=share_link
the model trained with indoor subset in SDSD: indoor_G.pth
the model trained with outdoor subset in SDSD: outdoor_G.pth
the model trained with SMID: smid_G.pth
If you find the project useful, please cite:
@inproceedings{xu2023dp3df,
title={Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution},
author={Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia},
booktitle={AAAI},
year={2023}
}
This source code is inspired by EDVR.
If you have any questions/comments/bug reports, feel free to e-mail the author Xiaogang Xu (xiaogangxu00@gmail.com).