Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising

Miaoyu Li, Ji Liu, Ying Fu, Yulun Zhang, and Dejing Dou, "Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising" CVPR 2023

Arxiv

---

Abstract: Denoising is a crucial step for Hyperspectral image (HSI) applications. Though witnessing the great power of deep learning, existing HSI denoising methods suffer from limitations in capturing the non-local self-similarity. Transformers have shown potential in capturing long-range dependencies, but few attempts have been made with specifically designed Transformer to model the spatial and spectral correlation in HSIs. In this paper, we address these issues by proposing a spectral enhanced rectangle Transformer, driving it to explore the non-local spatial similarity and global spectral low-rank property of HSIs. For the former, we exploit the rectangle self-attention horizontally and vertically to capture the non-local similarity in the spatial domain. For the latter, we design a spectral enhancement module that is capable of extracting global underlying low-rank property of spatial-spectral cubes to suppress noise, while enabling the interactions among non-overlapping spatial rectangles. Extensive experiments have been conducted on both synthetic noisy HSIs and real noisy HSIs, showing the effectiveness of our proposed method in terms of both objective metric and subjective visual quality.

---

Network Architecture

Contents

1. Models
2. Datasets
3. Training and Testing
4. Results

Models

We also provide the checkpoints of our competing methods.

Quantitative Comparison on ICVL Dataset

Method	Publication	Params (M)	GFLOPs (512×512×31)	Model Zoo (Gaussian noise)	Model Zoo (Complex noise)
GRNet	TGRS 2022	41.44	610.7	Google Drive	Google Drive
QRNN3D	TNNLS 2020	0.86	2513.7	Google Drive	Google Drive
T3SC	NeurIPS 2021	0.83	-	Google Drive	Google Drive
MACNet	TGRS 2022	0.43	-	Google Drive	Google Drive
SST	AAAI 2023	4.10	2082.4`	Google Drive	Google Drive
SERT	CVPR 2023	1.91	1018.9	Google Drive	Google Drive

Quantitative Comparison on Realistic Dataset

Method	Params (M)	GFLOPs(512×512×34)	PSNR	Model Zoo
GRNet	44.40	611.9		Google Drive
QRNN3D	0.86	2756.9		Google Drive
T3SC	0.83	-		Google Drive
MACNet	0.43	-		Google Drive
SST	2.14			Google Drive
SERT	1.91	1021.9		Google Drive

Quantitative Comparison on Apex/Urban Dataset

Method	Params (M)	GFLOPs	Model Zoo
GRNet	44.53		Google Drive
QRNN3D	0.86		Google Drive
T3SC	0.83		Google Drive
MACNet	0.43		Google Drive
SST	22.76		GoogleDriver
SERT	8		Google Drive

Datasets

ICVL

• The entire ICVL dataset download link: https://icvl.cs.bgu.ac.il/hyperspectral/

1. split the entire dataset into training samples, testing samples and validating samples. The files used in training are listed in utility/icvl_partition/icvl_train_list.txt.
2. generate lmdb dataset for training

python utility/lmdb_data.py

3. download the testing data from BaiduDisk or generate them by yourself through

python utility/mat_data.py

Realistic Dataset

• Please refer to [github-link] for "Hyperspectral Image Denoising with Realistic Data in ICCV, 2021" to download the dataset

Testing HSIs are listed in utility/realistic_partition/test.txt. Other HSIs in the dataset are used for training.

Urban dataset

• The training dataset are from link: https://apex-esa.org/. The origin Urban dataset are from link: https://rslab.ut.ac.ir/data.

1. Run the create_big_apex_dataset() funtion in utility/mat_data.py to generate training samples.

2. Run the createDCmall() function in utility/lmdb_data.py to generate training lmdb dataset.

Training and Testing

ICVL Dataset

#for gaussian noise
#----training----
python hside_simu.py -a sert_base -p sert_base_gaussian

#----testing---- The results are shown in Table 1 in the main paper.
python hside_simu_test.py -a sert_base -p sert_base_gaussian_test -r -rp checkpoints/icvl_gaussian.pth --test-dir /icvl_noise/512_50

#for comlpex noise
#----training----
python hside_simu_complex.py -a sert_base -p sert_base_complex

#----testing---- The results are shown in Table 2 in the main paper.
python hside_simu_test.py -a sert_base -p sert_base_complex_test -r -rp checkpoints/icvl_complex.pth --test-dir  /icvl_noise/512_mix

Urban Dataset

#----training----
python hside_urban.py -a sert_urban -p sert_urban 

#----testing----  The results are shown in Figure 4 in the main paper.
python hside_urban_test.py -a sert_urban -p sert_urban_test -r -rp ./checkpoints/real_urban.pth

Realistic Dataset

#----training----
python hside_real.py -a sert_real -p sert_real

#----testing---- The results are shown in Table 3 in the main paper.
python hside_real_test.py -a sert_real -p sert_real_test -r -rp ./checkpoints/real_realistic.pth

Results

Denoising on Random noise (ICVL) (click to expand)

Denoising on Complex noise (ICVL) (click to expand)

Denoising on Realistic noise (click to expand)

Denoising on Urban dataset (click to expand)

Citations

If you find the code helpful in your resarch or work, please cite the following paper(s).

@inproceedings{li2023spatial,
    title={Spatial-Spectral Transformer for Hyperspectral Image Denoising},
    author={Li, Miaoyu and Fu, Ying and Zhang, Yulun},
    booktitle={AAAI},
    year={2023}
}
@inproceedings{li2023spectral,
    title={Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising},
    author={Miaoyu Li and Ji Liu and Ying Fu and Yulun Zhang and Dejing Dou},
    booktitle={CVPR},
    year={2023}
}

Acknowledgement

The codes are based on QRNN3D.