WeatherStream: Light Transport Automation of Single Image Deweathering (CVPR'23)

Howard Zhang^1*, Yunhao Ba^1*, Ethan Yang¹, Varan Mehra¹, Blake Gella¹, Akira Suzuki¹, Arnold Pfahnl¹, Chethan Chinder Chandrappa¹, Alex Wong², Achuta Kadambi¹

University of California, Los Angeles¹, Yale University²

Project Webpage

https://visual.ee.ucla.edu/wstream.htm/

Abstract

Today single image deweathering is arguably more sensitive to the dataset type, rather than the model. We introduce WeatherStream, an automatic pipeline capturing all real-world weather effects (rain, snow, and rain fog degradations), along with their clean image pairs. Previous state-of-the-art methods that have attempted the all-weather removal task train on synthetic pairs, and are thus limited by the Sim2Real domain gap. Recent work has attempted to manually collect time multiplexed pairs, but the use of human labor limits the scale of such a dataset. We introduce a pipeline that uses the power of light-transport physics and a model trained on a small, initial seed dataset to reject approximately 99.6% of unwanted scenes. The pipeline is able to generalize to new scenes and degradations that can, in turn, be used to train existing models just like fully human-labeled data. Training on a dataset collected through this procedure leads to significant improvements on multiple existing weather removal methods on a carefully human-collected test set of real-world weather effects.

Citation

@inproceedings{zhang2023weatherstream,
  title={WeatherStream: Light Transport Automation of Single Image Deweathering},
  author={Zhang, Howard and Ba, Yunhao and Yang, Ethan and Mehra, Varan and Gella, Blake and Suzuki, Akira and Pfahnl, Arnold and Chandrappa, Chethan Chinder and Wong, Alex and Kadambi, Achuta},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2023}
}

Dataset

The dataset can be found here.

Model

Initial seed model uses GT-RAIN here.

Requirements

All code was tested on Google Colab with the following:

• Ubuntu 18.04.6
• CUDA 11.2
• Python 3.7.13
• OpenCV-Python 4.6.0
• PyTorch 1.12.1
• scikit-image 0.18.3

Setup

Download initial seed model from link above. Set the model and pipeline parameters. Set the folder name in the main pipeline loop. Make sure directory structure follows code comment.

Disclaimer

Please only use the code and dataset for research purposes.

Contact

Yunhao Ba
UCLA, Electrical and Computer Engineering Department
yhba@ucla.edu

Howard Zhang
UCLA, Electrical and Computer Engineering Department
hwdz15508@g.ucla.edu