VisTA-SR: Improving the Accuracy and Resolution of Low-Cost Thermal Imaging Cameras for Agriculture

Official PyTorch implementation of VisTA-SR (Visual & Thermal Alignment and Super Resolution Enhancement), published at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 2024.

Authors: Heesup Yun, Sassoum Lo, Christine H. Diepenbrock, Brian N. Bailey, J. Mason Earles
University of California, Davis
[Paper HTML] | [PDF] | [Hugging Face Dataset]

---

🌟 Overview

Thermal cameras are essential in agricultural research for non-invasive plant temperature measurement, crop water stress indexing, and biophysical modeling. However, high-resolution industrial thermal cameras cost over $10,000, limiting accessibility.

VisTA-SR solves this limitation by combining low-cost consumer thermal imagery (e.g. FLIR One Pro) with visible light RGB images. The framework consists of two main stages:

1. Stage 1: Domain Transfer and Alignment: Translates RGB images into the thermal domain using CycleGAN and aligns the images via normalized cross-correlation template matching.
2. Stage 2: Super-Resolution Fusion Network: Fuses aligned RGB, domain-translated thermal, and low-resolution thermal inputs through a ResNet generator to enhance thermal resolution and sharpness.

---

🛠️ Software Environment & Setup

Requirements

• Python >= 3.9
• PyTorch >= 2.0.0
• torchvision >= 0.15.0
• OpenCV, NumPy, SciPy, Pillow, tqdm, PyYAML

Setup Options

Option A: Using Conda (Recommended)

conda env create -f environment.yml
conda activate vista-sr

Option B: Using Pip

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -e .

---

📊 Dataset (Training_T4_1_2_3)

The paired RGB and thermal dataset is hosted on Hugging Face: heesup/VisTA-SR.

Download Dataset

from datasets import load_dataset

dataset = load_dataset("heesup/VisTA-SR")
print(dataset)

Alternatively, place your local dataset folder at data/Training_T4_1_2_3.

---

🚀 Quickstart

1. Training VisTA-SR

python scripts/train.py \
  --data-dir data/Training_T4_1_2_3 \
  --exp-name vista_sr_run \
  --epochs 200 \
  --batch-size 8 \
  --lr 1e-4

2. Evaluation & Metric Benchmark

python scripts/test.py \
  --data-dir data/Training_T4_1_2_3 \
  --weights results/vista_sr_run/g-best.pth

3. Qualitative Visualization

python scripts/visualize.py \
  --data-dir data/Training_T4_1_2_3 \
  --weights results/vista_sr_run/g-best.pth \
  --output-dir results/visualizations

---

📈 Quantitative Results

Low-Cost Thermal Camera Temperature Calibration (Table 3 from paper)

Parameter Mode	All Data RMSE (°C)	All Data R²	15°C - 30°C RMSE (°C)	15°C - 30°C R²
Factory Parameters	1.52	0.86	1.52	0.83
Calibrated Parameters (Ours)	1.40	0.89	1.39	0.86

Super-Resolution Performance Comparison (Table 4 from paper)

Technique	RMSE (°C)	SSIM	PSNR
Bilinear Interpolation	2.84	0.74	23.84
SRGAN [31]	2.74	0.63	24.26
VisTA-SR (Ours)	2.75	0.63	23.67

---

📝 Citation

If you find this work or repository useful for your research, please cite our paper:

@inproceedings{yun2024vista,
  title={VisTA-SR: Improving the Accuracy and Resolution of Low-Cost Thermal Imaging Cameras for Agriculture},
  author={Yun, Heesup and Lo, Sassoum and Diepenbrock, Christine H and Bailey, Brian N and Earles, J Mason},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  pages={5470--5479},
  year={2024}
}

---

📄 License

This repository is released under the Apache 2.0 License.