RadarMOTR: Multi-Object Tracking with Transformers on Range-Doppler Maps

RadarMOTR is a transformer neural network based on MOTRv2 able to track multiple objects in Range-Doppler maps. Objects are first detected by an external detector based on CFAR & DBSCAN clustering. Detected objects are then passed to RadarMOTR for association, i.e. tracking. Todays Radar trackers are built with handcrafted methods like Kalman Filter to perform association across frames. The parameters of such methods are difficult to configure and optimize. RadarMOTR solves this problems by associating objects using an neural network instead of a Kalman Filter.

Note

If you're looking for the RADTrack dataset, you can find it in this repository: https://github.com/madeit07/RADTrack.

Paper

• Our paper is accepted at International Conference Radar 2024 (RADAR2024).
• Accepted Preprint

Installation

The codebase is built on top of

• MOTRv2
• MOTR
• Deformable DETR
• DETR

Requirements

• Linux (Tested on WSL Ubuntu 20.04.5 LTS)
• At least one CUDA capable GPU
• GNU G++ Compiler <11 (Tested with version 9.4.0)
• Anaconda or Miniconda

1. Clone repository with submodules

   git clone --recurse-submodules https://github.com/madeit07/RadarMOTR.git
   cd RadarMOTR

2. Install PyTorch using Conda

   conda create -n radarmotr python=3.11
   conda activate radarmotr
   conda install pytorch=2.1 torchvision=0.16 pytorch-cuda=11.8 -c pytorch -c nvidia

3. Install CUDA Toolkit 11.8 for compiling MultiScaleDeformableAttention (optional if already installed on system)

   conda install cuda-libraries-dev cuda-nvcc cuda-nvtx cuda-cupti -c nvidia/label/cuda-11.8.0

4. Install other requirements

   pip install -r requirements.txt

5. Build MultiScaleDeformableAttention

   cd ./models/ops
   ./make.sh

Prepare dataset

Download the RDTrack dataset from Google Drive, unzip it and save it into data/dataset/RDTrack. RDTrack is a preprocessed version of RADTrack that only includes Range-Doppler data and is formatted in MOT20 format.

All paths are only recommendations. If you choose a custom path, make sure to update the dataset config (configs/base/dataset.yaml)

Prepare pretrained weights

All weights are stored in Google Drive

• If you train with ResNet18 backbone, download motrv2_base_r18_v2.pth and save it in data/models/.
• If you train with ResNet50 backbone, download motrv2_base_v2.pth and save it in data/models/.
• If you want to evaluate our model, download radarmotr_r18.pth and save it in data/models/.

All paths are only recommendations. If you choose a custom path, make sure to update pretrained path or model path in the configs (configs/resnet50.yaml, configs/resnet18.yaml, configs/trackers/radarmotr.yaml) or as command line arguments:

# ResNet18
python main.py with resnet18 pretrained=<MY_PATH_TO_MODELS>/motrv2_base_r18_v2.pth
# ResNet50
python main.py with resnet50 pretrained=<MY_PATH_TO_MODELS>/motrv2_base_v2.pth

# Eval
python main.py eval.py with radarmotr resnet18 model_path=<MY_PATH_TO_MODELS>/radarmotr_r18.pth

Training

Adapt the parameters if needed and run the training with e.g. 4 GPUs in data distributed mode.

# Single GPU
python main.py with resnet18

# 4 GPUs
torchrun --standalone --nnodes 1 --nproc_per_node 4 --max_restarts 2 main.py with resnet18

If you use the model with ResNet50 backbone, run:

# Single GPU
python main.py with resnet50

# 4 GPUs
torchrun --standalone --nnodes 1 --nproc_per_node 4 --max_restarts 2 main.py with resnet50

You can change model hyper parameters by either passing them to main.py or changing the dedicated config YAML files in config/. Sacred is used for managing runs and the configuration. Refer to their documentation for more information.

Information about the run itself and logs will be stored in runs/. Model weights will be saved in data/RadarMOTR/ or the custom path specified in output_dir config entry.

Evaluation

Run inference with model. Results will be saved to data/trackers/rdtrack-<split>/<name>.

python eval.py with radarmotr resnet18 split=val model_path=<path> tracker_dirname=<name> 

eval.log will contain tables of all HOTA, CLEAR and Identity metrics.

Omit the resnet18 if you use ResNet50 backbone.

Evaluate with KF tracker:

python eval.py with kf split=val tracker_dirname=kf

Citation

Please use the following citation:

@inproceedings{RadarMOTR2024,
    author = {Dell, Martin and Bradfisch, Wolfgang and Schober, Steffen and Klöck, Clemens},
    title = {{RadarMOTR: Multi-Object Tracking with Transformers on Range-Doppler Maps}},
    booktitle = {International Conference Radar 2024 (RADAR2024)},
    year = {2024}
}