ESP-Dataset (ICRA 2024)

ESP: Extro-Spective Prediction for Long-term Behavior Reasoning in Emergency Scenarios

Introduction | Video | News | Dataset | Download | Website | Paper

Introduction

Emergent-scene safety is the key milestone for fully autonomous driving, and reliable on-time prediction is essential to maintain safety in emergency scenarios. However, these emergency scenarios are long-tailed and hard to collect, which restricts the system from getting reliable predictions. In this paper, we build a new dataset, which aims at the long-term prediction with the inconspicuous state variation in history for the emergency event, named the Extro-Spective Prediction (ESP) problem.

• The ESP-Dataset with semantic environment information is collected over 2k+ kilometers focusing on emergency-event-based challenging scenarios.

• A new metric named CTE is proposed for comprehensive evaluation of prediction performance in time-sensitive emergency scenarios.

• ESP feature extraction and network encoder are introduced, which can be used to enhance existing backbones/algorithms seamlessly.

Video

Click the following Graphical Abstract for the illustration video!

News

• [Jul 24, 2024] The full dataset is released.
• [Jun 10, 2024] A mini split of the dataset is released.

Dataset

The dataset structure of tokens is shown below:

tokens/
├── train/
│   ├── token1/
│   ├── token2/
│   └── ...
├── val/
│   ├── token1/
│   ├── token2/
│   └── ...
└── test/
    ├── token1/
    ├── token2/
    └── ...

The dataset structure of tokens_by_mons is shown below:

tokens_by_mons/
├── mon1/
│   ├── token1/
│   ├── token2/
│   └── ...
├── mon2/
│   ├── token1/
│   ├── token2/
│   └── ...
└── ...

For each samplem, the structure is shown as below:

token
├── MomentId
├── Timestamp
├── TokenId
├── MapId
├── SceneInformation
│   ├── lane_type
│   ├── road_type
│   ├── time_of_day
│   ├── weather_conditions
│   └── ...
├── SemanticInfrastructure
│   ├── speed_monitor
│   ├── near_junction
│   ├── rare_road_objects
│   └── ...
├── EgoVehicleInformation
│   ├── vehicle_id
│   ├── vehicle_type
│   └── ...
├── TvInformation
│   ├── vehicle_id
│   ├── vehicle_type
│   └── ...
├── OtherVehiclesInformation
│   ├── vehicle1
│   ├── vehicle2
│   └── ...
└── ExtroSpectivePredictionFeatures
    ├── tv_dist_to_ev
    ├── tv_speed_to_ev
    └── ...

Download

This section provides a link to the Mini Split ESP-Dataset:

Download ESP-Dataset Mini Split

Download ESP-Dataset Full Dataset

(The Full Dataset contains two separate files: one is "tokens" and the other is "tokens_by_mons". The latter contains samples arranged by their respective moments, while the former contains samples randomly put together.)

Citation

If using our data in your research work, please cite the following paper:

@article{dingrui2024esp,
      author    = {Wang, Dingrui and Lai, Zheyuan and Li, Yuda and Wu, Yi and Ma, Yuexin and Betz, Johannes and Yang, Ruigang and Li, Wei},
      title     = {ESP: Extro-Spective Prediction for Long-term Behavior Reasoning in Emergency Scenarios},
      journal   = {ICRA},
      year      = {2024},
    }