Paddle-DSR-Lab：A depth map super-resolution toolkit

English| 简体中文

Paddle-DSR-Lab is a toolkit for depth map super-resolution based on PaddlePaddle, a member of the PaddleDepth project. It is scalable and easy to use, and in addition it provides a fair comparison of algorithms in the field of deep graph super-resolution and SOTA (state-of-the-art) in image super-resolution with the same dataset and environment, in a training configuration consistent with the original paper.

cones	tskuba	teddy	venus

Implemented Algorithms

As initial version, we support the following algoirthms. We are working on more algorithms. Of course, you are welcome to add your algorithms here.

1. WAFP-Net (IEEE Transactions on Multimedia 2021)[1]
2. PMBANet (IEEE Transactions on Image Processing 2019)[2]
3. RCAN (ECCV 2018)[3]
4. DRN (CVPR 2020)[4]

Installation

You can install the Paddle-DSR-Lab toolbox by following these steps:

• Install PaddlePaddle

  • Version requirements: PaddlePaddle>=2.3.0, Python>=3.7

• Paddle-DSR-Lab is installed with the following command

git clone https://github.com/kongdebug/Paddle-DSR-Lab.git
cd Paddle-DSR-Lab
pip install -r requirements.txt

Dataset

see guidance in dataset_prepare for dataset preparation.

Usage

Train

python -u tools/main.py --config-file $file_path$

• The config-file parameter is the path to the configuration file for the training model
• If there are pre-trained weights for finetune, run the following command to start training, with the load parameter being the path to the pre-trained weights

python -u tools/main.py --config-file $file_path$ --load $weight_path$

• If training is interrupted and needs to be resumed, run the following command, with the resume parameter as the checkpoint path

python -u tools/main.py --config-file $file_path$ --resume $checkpoint_path$

Test

python -u tools/main.py --config-file $file_path$ --evaluate-only --load $weight_path$

Customization

The file structure of Paddle-DSR-Lab is shown below:

Paddle-DSR-Lab
    │  README.md                
    │  README_cn.md             
    │  requirements.txt         
    ├─configs                   
    ├─data                      
    │  ├─process_DocumentIMG    
    │  ├─process_pmba           
    │  └─process_wafp           
    ├─docs                      
    ├─ppdsr 
    │  ├─datasets               
    │  │  └─preprocess          
    │  ├─engine                 
    │  ├─metrics                
    │  ├─models                 
    │  │  ├─backbones           
    │  │  ├─criterions          
    │  │  └─generators          
    │  ├─modules                
    │  ├─solver                 
    │  └─utils                  
    └─tools                     

You can develop your own algorithm by following these steps:

1. Check if your model needs a new loss function for training, and if so add the loss function to ppdsr/models/criterions.
2. Check if you need to add new models for training, if so add them to ppdsr/models.
3. check if you need to add new datasets for training, and if so add the datasets to ppdsr/datasets
4. Add your own configuration file (.yaml) to configs

Results

We evaluated the algorithms already implemented in Paddle-DSR-Lab using the four depth maps teddy, cones, tskuba and venus as a test set DSR-TestData.

Note: We did not optimize the model results by additional tricks, so you can directly use the .yaml configuration file to reproduce the accuracy we report in the table.

DSR-TestData

Model	PSNR	SSIM	RMSE	MAD	size
WAFP-Net [1]	42.0344	0.9834	2.5561	0.9246	3M
PMBANet [2]	41.0418	0.9825	2.4728	0.6278	94.9M
RCAN [3]	42.5297	0.9850	2.4401	0.6685	59.6M
DRN [4]	42.4906	0.9850	2.4634	0.6506	18.4M

Contribution

The toolkit is under active development and contributions are welcome! Feel free to submit issues or emails to ask questions or contribute your code. If you would like to implement new features, please submit a issue or emails to discuss with us first.

References

[1] Song, Xibin, Dingfu Zhou, Wei Li, Yuchao Dai, Liu Liu, Hongdong Li, Ruigang Yang, and Liangjun Zhang. ‘WAFP-Net: Weighted Attention Fusion Based Progressive Residual Learning for Depth Map Super-Resolution’. IEEE Transactions on Multimedia 24 (2022): 4113–27. https://doi.org/10.1109/TMM.2021.3118282. .

[2] Ye, Xinchen, Baoli Sun, Zhihui Wang, Jingyu Yang, Rui Xu, Haojie Li, and Baopu Li. ‘PMBANet: Progressive Multi-Branch Aggregation Network for Scene Depth Super-Resolution’. IEEE Transactions on Image Processing 29 (2020): 7427–42. https://doi.org/10.1109/TIP.2020.3002664.

[3] Zhang, Yulun, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. ‘Image Super-Resolution Using Very Deep Residual Channel Attention Networks’. In Computer Vision – ECCV 2018, edited by Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, 11211:294–310. Lecture Notes in Computer Science. Cham: Springer International Publishing, 2018. https://doi.org/10.1007/978-3-030-01234-2_18.

[4] Guo, Yong, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, and Mingkui Tan. ‘Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution’. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5406–15. Seattle, WA, USA: IEEE, 2020. https://doi.org/10.1109/CVPR42600.2020.00545.

Contact

• Yuanhang Kong: 2111330@tongji.edu.cn