[IROS 2023] Streaming Motion Forecasting for Autonomous Driving

Ziqi Pang, Deva Ramanan, Mengtian Li, Yu-Xiong Wang

1. Introduction

This is the official code for our IROS 2023 paper: "Streaming Motion Forecasting for Autonomous Driving." We propose to view the motion forecasting from a streaming perspective, where the predictions are made on continuous frames, instead of the conventional snapshot-based forecasting. As shown in the figure below, we highlight predicting trajectories on continuous frames, introducing the new challenges of occlusion reasoning and temporal coherence for motion forecasting.

If you find our code or paper useful, please cite by:

@inproceedings{pang2023streaming,
  title={Streaming motion forecasting for autonomous driving},
  author={Pang, Ziqi and Ramanan, Deva and Li, Mengtian and Wang, Yu-Xiong},
  booktitle={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2023}
}

2. Getting Started with Benchmark (Argoverse-SF)

If you wish to evaluate using the protocol of our streaming forecasting, please pay most attention to the Result Format and Evaluation section.

2.1 Install streaming_forecasting Toolkit

• We recommend install our streaming_forecasting toolkit locally via:

pip install -e ./

• Then install the argoverse-api following their instructions.

2.2 Data Preparation

Click to view the details

You will prepare the argoverse dataset, both tracking and forecasting splits included. We recommend you putting the tracking and forecasting into separate directories. For instance, I use directories ~/datasets/argoverse/tracking and ~/datasets/argoverse/forecasting.

Remember to soft-link your data location to ./data/argoverse_forecasting and ./data/argoverse_tracking. The final file structure would be similar to:

./data
    -- argoverse_tracking
        -- train
           xxx   xxx   xxx ...
        -- val
           xxx   xxx   xxx ...
        
    -- argoverse_forecasting
        -- train
            -- data
               xxx.csv, xxx.csv, xxx.csv
        -- val
            -- data
                xxx.csv, xxx.csv, xxx.csv

2.2.1 Tracking data

Argoverse-SF models streaming forecasting by re-purposing the tracking data from Argoverse. Please skip this step if you already have it.

• Download the tracking split from Argoverse Link. You will see 4 *.tar.gz for the training set and 1 *.tar.gz for the validation set.
• Extract the data from compressed files locally. Take tracking_val_v1.1.tar.gz for example:

# Normal extraction
tar -xvf tracking_val_v1.1.tar.gz -C ./

# Exclude the images if you have limited disk space
tar -xvf tracking_val_v1.1.tar.gz --exclude="*.jpg" -C ./

• Move everything out of argoverse_tracking/ and merge the training files

# move everything out
mv argoverse_tracking/* ./

# merge training set
mkdir train
mv train1/* train
mv train2/* train
mv train3/* train
mv train4/* train

2.2.2 Forecasting data

In the pretraining step, we will use the forecasting data to train a snapshot-based forecasting model, just like normal forecasters on Argoverse.

• Download the forecasting split from Argoverse Link. You will see 1 *.tar.gz for the training set and 1 *.tar.gz for the validation set.
• Extract the forecasting file locally. Take forecasting_val_v1.1.tar.gz for example, the sript is as below.

tar -xvf forecasting_val_v1.1.tar.gz

2.3 Benchmark (Argoverse-SF) Creation

Click to view the details

We will walk you through:

• Generating the Argoverse-SF benchmark files for evaluation and visualization.
• Generating the information files for dataloading during training and inference, recording the ground truth trajectories and orders of frames.

2.3.1 Benchmark Creation

Please use our ./tools/benchmark/argoverse_sf_creation.py to create the Argoverse-SF benchmark, which supports evaluation. The commands is as below, if you follow our instructions on softlinking the argoverse datasets to ./data as described in [Dataset Preparation]. After this step, you will see eval_cat_val.pkl and eval_cat_train.pkl popping up in ./data/streaming_forecasting.

mkdir ./data/streaming_forecasting

# training set
python tools/benchmark/argoverse_sf_creation.py --data_dir ./data/argoverse_tracking/train --output_dir ./data/streaming_forecasting --save_prefix eval_cat_train --hist_length 20 --fut_length 30 

# validation set
python tools/benchmark/argoverse_sf_creation.py --data_dir ./data/argoverse_tracking/val --output_dir ./data/streaming_forecasting --save_prefix eval_cat_val --hist_length 20 --fut_length 30

If you want any customization, please follow the template below.

# training set
python tools/benchmark/argoverse_sf_creation.py --data_dir $path_to_tracking_train --output_dir $path_to_save_streaming_benchmark --save_prefix eval_cat_train --hist_length $history_length_of_forecasting --fut_length $prediction_horizon 

# validation set
python tools/benchmark/argoverse_sf_creation.py --data_dir $path_to_tracking_val --output_dir $path_to_save_streaming_benchmark --save_prefix eval_cat_val --hist_length $history_length_of_forecasting --fut_length $prediction_horizon 

2.3.2 Information File Creation

Please use our ./tools/benchmark/info_file.py to create the information files for Argoverse-SF. We mimic the style in mmdetection and mmdetection3d in organizing the information needed for training, inference, and evaluation.

The command is as below, if you follow our instructions to prepare the paths of data. After running these scripts, you will see infos_train.pkl and infos_val.pkl in ./data/streaming_forecasting.

# training set
python tools/benchmark/info_file.py --save_prefix infos_train --benchmark_file eval_cat_train.pkl --data_dir ./data/argoverse_tracking/train

# validation set
python tools/benchmark/info_file.py --save_prefix infos_val --benchmark_file eval_cat_val.pkl --data_dir ./data/argoverse_tracking/val

If you want any customization, please follow the template below.

# training set
python tools/benchmark/info_file.py --data_dir $path_to_tracking_train --output_dir $path_to_save_streaming_benchmark --save_prefix infos_train --benchmark_file eval_cat_train.pkl --hist_length $history_length_of_forecasting --fut_length $prediction_horizon  

# validation set
python tools/benchmark/info_file.py --data_dir $path_to_tracking_validation --output_dir $path_to_save_streaming_benchmark --save_prefix infos_val --benchmark_file eval_cat_val.pkl --hist_length $history_length_of_forecasting --fut_length $prediction_horizon  

2.4 Format of Results and Evaluation Protocol

Evaluation. Click to view details.

2.4.1 Format of Results

To evaluate on Argoverse-SF, the result file is .pkl file compressing a python list, where each item in the list is the result of one frame. For reference, you can check our VectorNet result file for a rough sense.

Specifically, you would use the information file generated before infos_val.pkl. With each item in infos_val.pkl, your result file should also be a list corresponding to each sample in infos_val.pkl. Please note that the trajectories are in the world coordinate.

result = 
[
  # Result for sample[0] in infos_val.pkl 
  {
    ...
  },
  ...
  # Result for sample[i] in infos_val.pkl
  {
    # sequence name in Argoverse, 
    # you should directly copy from sample[i]['seq_name']
    'seq_name': str, 
    # city_name in Argoverse,
    # you should directly copy from sample[i]['city_name']
    'city_name': str,
    # frame number in the sequence,
    # you should directly copy from sample[i]['frame_index']
    'frame_index': int,
    # results of your predictions
    # dictonary with keys being every agent id specified in sample[i]['query_keys']
    'results': {
      # key0 in sample[i]['query_keys]
      sample[i]['query_keys'][0]: ...,
      ...
      # keyj in sample[i]['query_keys]
      sample[i]['query_keys'][j]: {'trajs': numpy array shaped [6, 30, 2], 'confidences': numpy array shaped [6]},
      ...
      sample[i]['query_keys'][-1]: ...
    }
  },
  ...
  # Result for sample[-1] in infos_val.pkl
  {
    ...
  },
]

2.4.2 Evaluation

To evaluate the inference results, run the following command and you will find the metrics.json in you specified $directory_to_save_metric_file

python tools/evaluation.py --result_file $path_streaming_inference.pkl --metric_file_prefix $directory_to_save_metric_file

For example, I want to evaluate my VectorNet results and save to ./results/VectorNet, then:

python tools/evaluation.py --result_file ./results/VectorNet/streaming_inference.pkl --metric_file_prefix ./results/VectorNet/

We structure the metric results in the following way:

• The items of ade6, fde6, and mr6 means the minADE, minFDE and miss rate for K=6, where the forecaster predicts 6 trajectories for each agent.
• We divide 4 groups of agents to acquire the above metrics. The groups are organized according to moving (move) and static (stay) motion, and visible (vis) and occluded (occ) states.
• Finally, the items begin with "ave-" are the major metrics that come from averaging all the groups.

3. Getting Started with Streaming Algorithm (Predictive Streamer)

3.1 HD-Map Preprocessing

HD-Map preprocessing. Click to view.

The first step is to preprocess the HD-Maps to accelerate the development. This part is mainly designed to assist the VectorNet and MMTransformer used in the paper. If you have followed the previous steps in data preparation, especially the soft linking of directories, running the commands below is enough.

python tools/benchmark/hd_map_preprocess.py

3.2 Streaming Inference on Argoverse-SF

Inference details. Click to view.

We use tools/inference.py to run streaming forecasting iteratively on every frame. Its template is as below. After running the command, a streaming_inference.pkl will appear at the directory of $path_to_save_results, which is the result file.

python tools/inference.py --config $config_path --weight_path $your_checkpoint --save_prefix $path_to_save_results

For example, if you use our provided VectorNet checkpoint and save the results to ./results/VectorNet, you could simply run as below.

python tools/inference.py --config configs/streamer/config.yaml --weight_path ./ckpts/vectornet.ckpt --save_prefix ./results/VectorNet/

3.3 Pretraining on the Forecasting Split

Before deploying on the streaming forecasting setup, we leverage the forecasting split to pretrain a strong forecasting model. You can skip this step by directly downloading our pretrained checkpoint. [link]

Pretraining details. Click to view.

3.3.1 Pretraining commands

If you have followed the previous steps, especially the paths to data. Training and evaluating VectorNet on Argoverse's forecasting training/validation sets are as simple as:

# training
python tools/pretrain/train_forecaster_vectornet.py --exp_name $your_experiment_name --model_save_dir $directory_to_save

# validation
python tools/pretrain/eval_forecaster_vectornet.py --weight_path $path_to_trained_model

For example, my command is as simple as:

# training
# use wandb for logging
python tools/pretrain/train_forecaster_vectornet.py --exp_name pretrain --model_save_dir ./results --wandb

# validation
python tools/pretrain/eval_forecaster_vectornet.py --weight_path vectornet.ckpt

The expected results of the validation process is similar to below. We focus on minADE, minFDE, and MR.

------------------------------------------------
Prediction Horizon : 30, Max #guesses (K): 6
------------------------------------------------
{'minADE': 0.7742552296892377, 'minFDE': 1.1925503502421884, 'MR': 0.12593737332792865, 'p-minADE': nan, 'p-minFDE': nan, 'p-MR': 0.8569636313846993, 'brier-minADE': 6.351643563315018, 'brier-minFDE': 6.769938683867964, 'DAC': 0.9879999324415611}
------------------------------------------------
------------------------------------------------
Prediction Horizon : 30, Max #guesses (K): 1
------------------------------------------------
{'minADE': 1.53590666902302, 'minFDE': 3.373533234667678, 'MR': 0.5553810295905959, 'p-minADE': 1.53590666902302, 'p-minFDE': 3.373533234667678, 'p-MR': 0.5553810295905959, 'brier-minADE': 1.53590666902302, 'brier-minFDE': 3.373533234667678, 'DAC': 0.9887515200648561}
------------------------------------------------

Suppose you need to customize the training process, such as path to data or optimization details, change the configuration files:

Customizing pretraining. Click to view

We provide configuration via yaml-based files.

• ./configs/forecaster/VectorNet.yaml specify the sub-configuration files controlling the behaviors of data loading, model architecture, and optimization.

• Data loading:

# batch size
batch_size: 32 
val_batch_size: 32 

# number of workers in the dataloader
workers: 4
val_workers: 4

# path to the forecasting data
train_dir: ./data/argoverse_forecasting/train/data/
val_dir: ./data/argoverse_forecasting/val/data/
train_map_dir: null
val_map_dir: null

# use all the training data
ratio: 1.0

# perception range [xxyy]
pred_range:
  - -100.0
  - 100.0
  - -100.0
  - 100.0

• Model architecture, please use our default VectorNet.

• Optimization:

# beginning epoch
epoch: 0
optimizer: adam
# total epoch
num_epochs: 24
# frequency of saving checkpoints
save_freq: 1.0

# base learning rate
lr: 0.0005
weight_decay: 1.0e-4
# iterations for warmup
warmup_iters: 1000
warmup_ratio: 1.0e-3
grad_clip: 1.0e-1

# when to drop the learning rate
lr_decay_epoch:
  - 16
  - 20
# ratio of dropping learning rate
lr_decay_rate: 0.25

3.4 Finetuning Forecasters for Streaming Forecasting

We describe how to finetune any forecasting model for the streaming setup.

Finetuning instructions. Click to view.

Suppose you have a pretrained forecaster, immitate how our streaming_vectornet.py turn a regular VectorNet in VectorNet.py into a streaming version of VectorNet. The key part is to use instances for storing the temporal information. Then we can finetune the new streaming model by:

# use --wandb if you have a wandb logger to use
python tools/train_streaming.py --config $config_path --weight_path $pretrained_forecaster --name $exp_name --save_prefix $directory_to_save_results

Specifically, if you follow the previous paths, you can finetune from the pretrained VectorNet provided by us with the following link:

python tools/train_streaming.py --config ./configs/streamer/config.yaml --weight_path ./ckpts/vectornet.ckpt --name finetune --save_prefix ./results/

After this, you can see several checkpoints popping up in ./results/finetune/ckpts/. The final one is the finetuned model.

3.4 Training and Evaluating Our Predictive Streamers

• TBD

4. Planerary Sections

Due to tight schedules, I am cleaning the code and writing documentations for the following parts.

• Model zoo.
• Easy way to use config file.
• Design docs for the implementations and system design.