RaVAEn

The RaVÆn system We introduce the RaVÆn system, a lightweight, unsupervised approach for change detection in satellite data based on Variational Auto-Encoders (VAEs) with the specific purpose of on-board deployment. It flags changed areas to prioritise for downlink, shortening the response time. We show that the proposed method outperforms pixel-wise baselines and we test it on resource-limited hardware. We also release the annotated dataset of extreme events. Work conducted at the FDL Europe 2021.

Nature Scientific Reports Paper • NeurIPS workshop paper • in media 1, 2 • video from AI+HADR'21 • Quick Colab Example

---

Unsupervised Change Detection of Extreme Events Using ML On-Board

Abstract: In this paper, we introduce RaVAEn, a lightweight, unsupervised approach for change detection in satellite data based on Variational Auto-Encoders (VAEs) with the specific purpose of on-board deployment. Applications such as disaster management enormously benefit from the rapid availability of satellite observations. Traditionally, data analysis is performed on the ground after all data is transferred - downlinked - to a ground station. Constraint on the downlink capabilities therefore affects any downstream application. In contrast, RaVAEn pre-processes the sampled data directly on the satellite and flags changed areas to prioritise for downlink, shortening the response time. We verified the efficacy of our system on a dataset composed of time series of catastrophic events - which we plan to release alongside this publication - demonstrating that RaVAEn outperforms pixel-wise baselines. Finally we tested our approach on resource-limited hardware for assessing computational and memory limitations.

Dataset

The full annotated dataset used for evaluation is hosted on Google Drive here. It contains 5 locations for each of the Landslide, Hurricane, Fire events and 4 locations for Floods events. For more details see the paper (we use the Sentinel-2 mission, level L1C data).

For dataset inspection use the prepared Colab Dataset Exploration demo .

Code examples

Install

# This will create a ravaen_env conda environment:
make requirements
conda activate ravaen_env
# Add these to open the prepared notebooks:
conda install nb_conda
jupyter notebook
# This will open an interactive notebook in your browser where you can navigate to the training or inference demo

Inference

To start using our model for inference, it's best to start with the prepared Colab Inference demo , which downloads our annotated dataset and evaluates a pre-trained model on a selected event type.

# Check possible parameters with:
!python3 -m scripts.evaluate_model --help 

# Example evaluation script used for the paper results for "small size" VAE model (remeber to adjust paths to the dataset and to the saved model checkpoints)
./bash/eval_run_papers_v3_VAE_128_D_small.sh

Training

For a fast demo on how to train these models on a custom folder of locations, check the Training demo as that presents an easy entry point to this repository. To reproduce the same training process as reported in the paper, you will need to download the whole WorldFloods dataset (see here) and prepare the same folder structure as we chose for the validation datasets.

# Check possible parameters with:
!python3 -m scripts.train_model --help

# Run the same training script used for the paper results for "small size" VAE model (remember to adjust the paths to the training datasets)
./bash/train_run_papers_v3_VAE_128_small_D.sh

Generality of the solution

Name "RaVAEn"

Our project is named after the two ravens in Norse mythology who are helping spirits of the god Odin and also highlights the usage of a Variational Auto-Encoder (VAE) as the main model:

Two ravens sit on his (Odin’s) shoulders and whisper all the news which they see and hear into his ear; they are called Huginn and Muninn. He sends them out in the morning to fly around the whole world, and by breakfast they are back again. Thus, he finds out many new things and this is why he is called ‘raven-god’ (hrafnaguð). (source)

Citation

If you find RaVAEn useful in your research, please consider citing the following papers:

@article{ruzicka_ravaen_2022,
	title = {{RaVÆn}: unsupervised change detection of extreme events using {ML} on-board satellites},
	volume = {12},
	issn = {2045-2322},
	url = {https://www.nature.com/articles/s41598-022-19437-5},
	doi = {10.1038/s41598-022-19437-5},
	number = {1},
	journal = {Scientific Reports},
	author = {Růžička, Vít and Vaughan, Anna and De Martini, Daniele and Fulton, James and Salvatelli, Valentina and Bridges, Chris and Mateo-García, Gonzalo and Zantedeschi, Valentina},
	month = oct,
	year = {2022},
	pages = {16939},
}

@inproceedings{ravaen2023fast,
  title={Fast model inference and training on-board of Satellites},
  url = {https://arxiv.org/abs/2307.08700},
  booktitle = {International Geoscience and Remote Sensing Symposium (IGARSS) 2023, {Pasadena}, {California}, {USA}},
  author={Růžička, Vít and Mateo-García, Gonzalo and Bridges, Chris and Brunskill, Chris and Purcell, Cormac and Longépé, Nicolas and Markham, Andrew},
  month = jul,
  year={2023},
  note = {arXiv: 2307.08700}
}