METEOR

This repository contains the official code for the paper: METEOR: A Massive Dense & Heterogeneous Behavior Dataset for Autonomous Driving.

Rohan Chandra, Mridul Mahajan, Rahul Kala, Rishitha Palugulla, Chandrababu Naidu, Alok Jain, and Dinesh Manocha

Under Review at ICRA 2022 | Preprint | Project Page

Pre-requisites

This repository is based on MMDetection and MMAction2.

Download the Dataset

Download the dataset here.

Dataset Preparation

1. Run the following command to extract frames from the videos:

   python videos2frames.py --videos_directory /path/to/videos_dir --output_directory /path/to/output_dir

2. Run the following command to reorganize the dataset into COCO format for object detection:

   python xml2coco.py --dataset_directory /path/to/dataset_dir --output_directory /path/to/output_dir

3. Run the following command to reorganize the dataset into rawframe annotation format for behavior prediction:

   python xml2rawframe.py --dataset_directory /path/to/dataset_dir --output_directory /path/to/output_dir

Config Preparation

1. Download the pre-trained DETR model from link.

2. Download the pre-trained TSN model from link.

3. Update the paths to the base config file, train annotations file, the test annotations file, and the pre-trained model in detr_config.py and tsn_config.py.

Training

1. Run the following command to initiate the training process for DETR:

   python mmdetection/tools/train.py detr_config.py

2. Run the following command to initiate the training process for TSN:

   python mmaction2/tools/train.py tsn_config.py

Pre-trained Models

• Object Detection

• Behavior Prediction

Testing

1. Run the following command to initiate the testing process for DETR:

   python mmdetection/tools/test.py detr_config.py detr_pretrained.pth --eval bbox mAP recall

2. Run the following command to initiate the testing process for TSN:

   python mmaction2/tools/test.py tsn_config.py tsn_pretrained.pth --eval mmit_mean_average_precision

Swin Detection (Scripts/Swin-Transformer-Object-Detection)

1. Installation (https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md)

2. DATA (Json file): Change the frame path in json file to your own path: e.g. "file_name": "/scratch0/xijunwang/data/METEOR_Dataset/Frame_XML_Annotations/REC_2020_10_12_01_26_57_F/frame_000000.JPG" -> "file_name": "/xxx/REC_2020_10_12_01_26_57_F/frame_000000.JPG"

3. Train:

   tools/dist_train.sh configs/swin/mask_rcnn_swin_tiny_patch4_window7_mstrain_480-800_adamw_1x_metor.py 8 --cfg-options model.pretrained=<PRETRAIN_MODEL>

4. Reference

   • Swin-Transformer-Object-Detection