A end-to-end system of cloud removal for satellite images.
1. System Usage Guidelines
1.1 Environment requirements The system operates efficiently on the Microsoft Azure Linux (ubuntu 20.04) Virtual Machine with one GPU Tesla T4 (16384 MB). To ensure the optimal functioning of the system, the following software versions are recommended:
earthnet==0.3.9
gradio==3.41.2
gradio-client==0.5.0
matplotlib==3.7.2
numpy==1.24.4
pandas==2.0.3
torch==2.0.1
torchvision==0.15.2
Python version 3.8.10
To start the system you can simply run “run_gradio.py” in codes/Clear_Vision.
1.2 Input Requirements For seamless operation of the AI-based system, adhere to the following input specifications:Spatial Satellite Image: RGB Images with 128 x 128 pixels. The image utilized for training and testing possesses dimensions of 128x128 pixels with a resolution of 20 meters. It is strongly recommended that the input images maintain this resolution (Inadequate input image quality, such as blurring after resizing one small resolution image, may render the cloud mask categorizing regular scenes as entirely cloud-covered, impacting model performance).Sequence Length: The length of the input image sequence should fall within the range of 5 to 10 images.Cloud Coverage: To attain optimal system performance, refrain from having clouds present in more than 50% of the images within the provided time sequence.Time Interval: There is no strict constraint on the time interval between images. However, it is imperative that images are arranged in the correct order. For enhanced system efficiency, it is recommended to maintain a relatively smaller time interval between consecutive images.
1.3 System Performance Evaluation The performance metrics of the system are as follows: PSNR (Peak Signal-to-Noise Ratio): The system achieves a PSNR of 34.894.RMSE (Root Mean Square Error): The RMSE achieved by the system is 0.021. These results are based on an evaluation conducted using simulated data generated from the Earthnet2021 iid test dataset. The generation process adheres rigorously to the guidelines outlined in the simulation of data gaps as described in [1]. Furthermore, the system's efficacy is demonstrated through its performance on time series extracted from the EarthNet2021 ood test dataset. Notably, these time series contain real clouds, and the model has had no prior exposure to this data during training or testing.
Examples from EarthNet2021 ood Test Dataset with Real Clouds Example 1:Input cloudy images
Cloud-free images generated by Clear Vision
Example 2:Input cloudy images
Cloud-free images generated by Clear Vision
2. System Architecture
The system is constructed as a two-stage cohesive model integrated within the Gradio framework. The two stages are cloud detection and cloud removal executed by separate models. Stage 1: Cloud DetectionIn this stage, the input image undergoes processing through a U-Net-based semantic segmentation model [2]. The outcome is the generation of a cloud mask that identifies cloud-covered areas. Stage 2: Cloud RemovalThe subsequent stage involves two inputs: the generated cloud mask and the original input image. These inputs are directed into a symmetric multi-scale encoding and decoding model. The cloud removal model integrates temporal attention and attention-weighted skip connections, which collectively contribute to the removal of clouds. The cloud removal model is based on the architecture proposed in [1], enhanced with an augmented model capacity in both downsampling and upsampling channel depth.
3. File Structure
“Clear_Vision” Folder:Contains all the code for system execution."examples" folder: This directory houses illustrative cloudy images showcased within the Gradio interface."networks" folder: Within this repository resides the model code for cloud detection and cloud removal."model_dict" folder: This repository contains the pre-trained model parameters for cloud detection and cloud removal functionalities.
“Preprocess” Folder:Contained within the "Preprocess" folder is the preprocessing pipeline for the Earthnet2021 dataset. This entails tasks such as RGB channel extraction and cloud mask generation. For both training and evaluation datasets, critical steps involve selecting valid time series and generating simulated data by merging meticulously selected cloud-free frames with authentic cloud masks.
“Cloud_mask_pretrain” Folder:The "Cloud_mask_pretrain" folder encompasses the training code for the cloud mask generation model. Should you desire to tailor the model structure, modifications can be made in "network.py". Additionally, the "dataloader.py" file can be adjusted to accommodate custom data for training or evaluation. To initiate model training and evaluation, execute "run.py".
“Cloud_removal_pretrain” Folder:Similarly, the "Cloud_removal_pretrain" folder houses the training code for the cloud removal model. "network.py" can be edited to adjust the model's architecture, and "dataloader.py" can be modified to incorporate specific training or evaluation data. Model training and evaluation are executed using "run.py".
Reference
[1] Stucker C, Sainte Fare Garnot V, Schindler K. U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series[J]. arXiv e-prints, 2023: arXiv: 2305.13277.
[2] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[C]//Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015: 234-241.