Bridging Sensor Gaps via Attention Gated Tuning for Hyperspectral Image Classification

Xizhe Xue, Haokui Zhang, Zongwen Bai, Ying Li* [arXiv] [BibTeX]

Features

• Attention-Gated tuning approach for HSI Classification.
• Highly efficient triplet-structured HSI classification transformer
• Support cross-sensor or cross-modality tuning, allowing for the utilization of abundant RGB labeled datasets to enhance HSI classification performance

Datasets preparing

The current version AGT has support for a few datasets. Due to Policy constraints, we are not able to directly provide and host HSI images. However, we share the pre-processed HSI images in .h5 and .mat files. Datasets can be downloaded by accessing Google Drive.

The data preprocessing files are in the data directory. Running these scripts can complete the data preprocessing.

Tri-Former Model training

Indian pines

• python train_dist.py --dataset Indian_pines --model_name trihit_cth --sample_list cp --batch_size 12 --epochs 300

Pavia University

• python train_dist.py --dataset PaviaU --model_name trihit_cth --sample_list cp --batch_size 12 --epochs 300

Pavia Center

• python train_dist.py --dataset PaviaC --model_name trihit_cth --sample_list cp --batch_size 12 --epochs 300

houstonu

• python train_dist.py --dataset HoustonU --model_name trihit_cth --sample_list cp --batch_size 12 --epochs 300

Salinas

• python train_dist.py --dataset Salinas --model_name trihit_cth --sample_list cp --batch_size 12 --epochs 300

Cifar

• python train_dist.py --dataset cifar10 --datatype PHSI --model_name trihit_cth --sample_list cp --batch_size 12 --epochs 150
• python train_dist.py --dataset cifar100 --datatype PHSI --model_name trihit_cth --sample_list cp --batch_size 12 --epochs 150

Inference Tri-Form Models

The pre-trained model can be found in Tri-Former model.

AGT Tuning

Salinas 2 Indian pines

• python train_adapter.py --dataset Indian_pines --freeze_model_dir '/home/disk1/result/trihit/Salinas/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

Salinas 2 HoustonU

• python train_adapter.py --dataset HoustonU --freeze_model_dir '/home/disk1/result/trihit/Salinas/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

Salinas 2 PaviaU

• python train_adapter.py --dataset PaviaU --freeze_model_dir '/home/disk1/result/trihit/Salinas/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

PaviaC 2 indian pines

• python train_adapter.py --dataset Indian_pines --freeze_model_dir '/home/disk1/result/trihit/PaviaC/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

PaviaC 2 HoustonU

• python train_adapter.py --dataset HoustonU --freeze_model_dir '/home/disk1/result/trihit/PaviaC/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

PaviaC 2 PaviaU

• python train_adapter.py --dataset PaviaU --freeze_model_dir '/home/disk1/result/trihit/PaviaC/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

Cifar 2 Indian pines

• python train_adapter.py --dataset Indian_pines --freeze_model_dir '/home/disk1/result/trihit/cifar/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

Cifar 2 HoustonU

• python train_adapter.py --dataset HoustonU --freeze_model_dir '/home/disk1/result/trihit/cifar/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

Cifar 2 PaviaU

• python train_adapter.py --dataset PaviaU --freeze_model_dir '/home/disk1/result/trihit/cifar/trihit_cth_cp/trihit_cth_best.pth' --model_name trihit_cth_sdt_r5 --sample_list ab_50 --batch_size 24 --epochs 150

Inference with Models

Please edit eval_dp.py, enter the model path and test data set in the parameter section, and run eval.py to test the model results.

Pick the fintuned-model from model zoo. More models trained with different samples are coming !

Model Zoo

Name	Dataset	Number of training samples	OA%	AA%	K%	Download
AGT+Tri-Former	Salinas 2 Indian Pines	50 pixel/class	94.83	97.62	94.07	model
AGT+Tri-Former	Salinas 2 Pavia University	50 pixel/class	98.51	98.40	98.22	model
AGT+Tri-Former	Salinas 2 Houston University	50 pixel/class	92.65	93.86	92.05	model

Citing HyT-NAS

If you use AGT in your research or wish to refer to the baseline results published in the Model Zoo, please use the following BibTeX entry.

@misc{xue2023bridgingsensorgapssingledirection,
      title={Bridging Sensor Gaps via Attention Gated Tuning for Hyperspectral Image Classification}, 
      author={Xizhe Xue and Haokui Zhang and Zongwen Bai and Ying Li},
      year={2023},
      eprint={2309.12865},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2309.12865}, 
}

If you have any questions, please feel free to contact me via e-mail .