Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image Reconstruction

Miaoyu Li ¹   Ying Fu¹  Ji Liu ²   Yulun Zhang ³ 

¹ Beijing Institute of Technology   ² Baidu Inc.   ³ ETH Zurich  

ICCV 2023

Arxiv

1. Comparison with State-of-the-art Methods

Method	Params (M)	FLOPS (G)	PSNR	SSIM	Model Zoo	Result
DAUHST-L	6.15	79.50	38.36	0.967	Repo	Repo
PADUT-3stg	1.35	22.91	36.95	0.962	Google Driver	Google Driver
PADUT-5stg	2.24	37.90	37.84	0.967	Google Driver	Google Driver
PADUT-7stg	3.14	52.90	38.41	0.971	Google Driver	Google Driver
PADUT-12stg	5.38	90.46	38.89	0.972	Google Driver	Google Driver

2. Create Environment

pip install -r requirements.txt

3. Data Preparation

Download cave_1024_28 (Baidu Disk, code: fo0q | One Drive), CAVE_512_28 (Baidu Disk, code: ixoe | One Drive), KAIST_CVPR2021 (Baidu Disk, code: 5mmn | One Drive), TSA_simu_data (Baidu Disk, code: efu8 | One Drive), TSA_real_data (Baidu Disk, code: eaqe | One Drive), and then put them into the corresponding folders of datasets/ and recollect them as the following form:

    |--real
    	|-- test_code
    	|-- train_code
    |--simulation
    	|-- test_code
    	|-- train_code
    |--datasets
        |--cave_1024_28
            |--scene1.mat
            |--scene2.mat
            ：  
            |--scene205.mat
        |--CAVE_512_28
            |--scene1.mat
            |--scene2.mat
            ：  
            |--scene30.mat
        |--KAIST_CVPR2021  
            |--1.mat
            |--2.mat
            ： 
            |--30.mat
        |--TSA_simu_data  
            |--mask.mat   
            |--Truth
                |--scene01.mat
                |--scene02.mat
                ： 
                |--scene10.mat
        |--TSA_real_data  
            |--mask.mat   
            |--Measurements
                |--scene1.mat
                |--scene2.mat
                ： 
                |--scene5.mat

Following TSA-Net and DGSMP, we use the CAVE dataset (cave_1024_28) as the simulation training set. Both the CAVE (CAVE_512_28) and KAIST (KAIST_CVPR2021) datasets are used as the real training set.

4. Simulation Experiement

4.1　Training

cd simulation
python train.py --template dauhst --outf ./exp/padut_3stg/ --method padut_3

python train.py --template dauhst --outf ./exp/padut_5stg/ --method padut_5

python train.py --template dauhst --outf ./exp/padut_7stg/ --method padut_7

python train.py --template dauhst --outf ./exp/padut_12stg/ --method padut_12

4.2　Testing

python test.py --template dauhst --outf ./exp/padut_3stg/ --method padut_3 --pretrained_model_path ./checkpoints/3.pth

python test.py --template dauhst --outf ./exp/padut_5stg/ --method padut_5 --pretrained_model_path ./checkpoints/5.pth

python test.py --template dauhst --outf ./exp/padut_7stg/ --method padut_7 --pretrained_model_path ./checkpoints/7.pth

python test.py --template dauhst --outf ./exp/padut_12stg/ --method padut_12 --pretrained_model_path ./checkpoints/12.pth

5. Real Experiement

5.1　Training

python train.py  --template dauhst --outf ./exp/padut_3stg/ --method padut_3  

5.2　Testing

python test.py  --template dauhst --outf ./exp/padut_3stg/ --method padut_3    --pretrained_model_path ./checkpoints/3.pth

6. Acknowledgements

This code repository's implementation is based on these two works:

@inproceedings{mst,
  title={Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction},
  author={Yuanhao Cai and Jing Lin and Xiaowan Hu and Haoqian Wang and Xin Yuan and Yulun Zhang and Radu Timofte and Luc Van Gool},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}
@inproceedings{res,
  title={Residual Degradation Learning Unfolding Framework with Mixing Priors across Spectral and Spatial for Compressive Spectral Imaging},
  author={Yubo Dong and Dahua Gao and Tian Qiu and Yuyan Li and Minxi Yang and Guangming Shi},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023}
}

Citation

@inproceedings{PADUT,
  title={Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image Reconstruction},
  author={Miaoyu Li and Ying fu and Ji Liu and Yulun Zhang},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision(ICCV)},
  year={2023}
}