Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery

Real-World Distribution Shifts (RWDS) Benchmark

📄 CVPR 2025 | 🔗 Paper | 🎥 Video | 📋 Poster | 🌐 Project Page | 💻 Code

Abstract

Object detectors have achieved remarkable performance in many applications; however, these deep learning models are typically designed under the i.i.d. assumption, meaning they are trained and evaluated on data sampled from the same (source) distribution. In real-world deployment, however, target distributions often differ from source data, leading to substantial performance degradation. Domain Generalisation (DG) seeks to bridge this gap by enabling models to generalise to Out-Of-Distribution (OOD) data without access to target distributions during training, enhancing robustness to unseen conditions. In this work, we examine the generalisability and robustness of state-of-the-art object detectors under real-world distribution shifts, focusing particularly on spatial domain shifts. Despite the need, a standardised benchmark dataset specifically designed for assessing object detection under realistic DG scenarios is currently lacking. To address this, we introduce Real-World Distribution Shifts (RWDS), a suite of three novel DG benchmarking datasets that focus on humanitarian and climate change applications. These datasets enable the investigation of domain shifts across (i) climate zones and (ii) various disasters and geographic regions. To our knowledge, these are the first DG benchmarking datasets tailored for object detection in real-world, high-impact contexts. We aim for these datasets to serve as valuable resources for evaluating the robustness and generalisation of future object detection models.

🛰️ TL;DR: RWDS is a comprehensive suite of benchmarks for evaluating the robustness of object detection models under real-world distribution shifts in satellite imagery when conditions change in real-world scenarios. It includes three datasets focused on disaster assessment and climate applications.

📊 Dataset Download

The RWDS datasets are available on both HuggingFace and Zenodo:

🤗 HuggingFace Hub

Direct download:

# Using HuggingFace Hub
pip install huggingface_hub
huggingface-cli download RWGAI/RWDS --repo-type dataset

📁 Installation and Setup

1️⃣ Framework Requirements

Download MMDetection Framework: Install MMDetection from the official repository by following the official MMDetection installation guide.

2️⃣ Directory Structure Setup

# Recommended directory structure:
your_project/
├── mmdetection/                   # Official MMDetection installation
│   └── configs/                   # MMDetection config directory
└── RWDS_Dataset/                 # RWDS datasets (outside mmdetection)
    ├── RWDS-CZ/
    ├── RWDS-FR/
    └── RWDS-HE/

3️⃣ Configuration Setup

After installing MMDetection, copy the RWDS configuration files:

# Copy RWDS config files to MMDetection
cp -r RWDS_mmdetection/configs/* mmdetection/configs/

⚠️ Important:

• Place RWDS_Dataset directory outside the mmdetection directory
• Copy all files from RWDS_Dataset/configs/ to mmdetection/configs/ upon installation
• Verify the directory structure matches the recommended layout above

🔬 Experiments

Stay-tuned: We willl release detailed training and evaluation instructions which we used in RWDS soon.

📝 Citation

@inproceedings{alemadi2025rwds,
  title={Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery},
  author={Al-Emadi, Sara A. and Yang, Yin and Ofli, Ferda},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2025}
}

---

💻 Code and detailed documentation coming soon.