This repository contains the code for the ICCV 2023 paper: Joint Metrics Matter: A Better Standard for Trajectory Forecasting.
SOTA Trajectory Forecasting baselines like AgentFormer optimize for per-agent minimum displacement error metrics such as ADE. Our method, Joint AgentFormer is optimized for multi-agent minimum displacement error metrics such as JADE -- Joint ADE.
We evaluate 6 baselines with respect to joint metrics (described in Sec. 3 of the paper)
and perform improvements with respect to 2 (AgentFormer and View Vertically).
The code for all evaluated methods (S-GAN, Trajectron++, View Vertically, MemoNet,
AgentFormer, Joint AgentFormer) are added as submodules in the methods/
directory.
Y-Net and PECNet are not yet included.
If you consider this work useful in your research, please cite our paper:
@inproceedings{weng2023joint,
title={Joint Metrics Matter: A Better Standard for Trajectory Forecasting},
author={Weng, Erica and Hoshino, Hana and Ramanan, Deva and Kitani, Kris},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2023}
}
To evaluate methods with respect to our joint metrics, first we save the trajectories to text files in a standardized format. Then, we read from the files and evaluate them.
To evaluate your own trajectory forecasting method, add your method to methods/
as a subdirectory
and write code to save all trajectories to results/trajectories
in the standardized format
which is described in the next section.
Ensuring that your saved trajectory files follows the standardized format will
ensure that you are evaluating the exact same dataset splits for fair comparison.
To create the training and evaluation datasets, we follow the leave-one-out training / evaluation setup that was used in the original S-GAN. That is, first we subdivide each dataset scene into sliding windows of length 20 (8 observation steps and 12 prediction steps) using stride of length 1 timestep. We use all sequences with at least 1 pedestrian
For each 20-timestep sequence, we save the observation (past) positions,
ground-truth future positions, and predicted future positions to separate files.
We save the observation to a file obs.txt
, the ground-truth future to gt.txt
,
and the predicted futures to sample_{sample_num}.txt
,
where {sample_num}
is the predicted sample number formatted with 3 digits, e.g. {:03d}
.
There are up to 20 predicted samples per sequence.
Thus, there are up to 22 trajectory files saved per length-20 sequence.
We save these trajectory files to the path: results/{method_name}/{dataset_environment_name}/{frame_id}/
where {frame_id}
is the frame number of the last observation step (t = 7) and serves as an identifying number for the length \delta t = 20 sequence. This format serves to ensure the dataset split being evaluated is the same across all methods.
each file is in the following format:
frame_id agent_id pos_x pos_y
...
In observation files there are 8 * num_agents
lines; and in ground-truth and prediction
files there are 12 * num_agents
lines. num_agents
is the number of agents present in
that observation sequence.
The positions are unnormalized and in world coordinates (the same frame as the ground truth positions for ETH and our included version of Trajnet-SDD).
After saving trajectories to the format described, please evaluate your method by running the following script from the repository root directory:
python evaluate.py --method <trajectory_save_relative_path> --dataset <dataset>
trajectory_save_relative_path
is the relative path of the directory in which the
trajectories for the method are saved, inside results/trajectories
;
e.g. trajectron
, sgan
, view_vertically
, etc.
Evaluation results will be saved to results/evaluations/<method_name>/<dataset>
as pickle files.
You can also run a script that will evaluate all methods in parallel:
python evaluate_all.py
After all pickles are saved, then run:
python evaluate_all.py
The final results table with all methods, datasets, metrics, and aggregations will be saved
to results/results_tables/<metric>.tsv
, one file for each metric.
For ETH, we use the split used by S-GAN, which many trajectory forecasting works use. We include instructions for downloading the dataset for in the README for each method, as each method may have different preprocessing steps.
For SDD, we use the Trajnet-SDD split released by the Trajnet Challenge.
The original test data is no longer available publicly,
so we include the data in our repo in datasets/raw_data/trajnet_sdd
.
Note that the original data is in pixel-coordinates, but we processed the data according
to the estimated scales available in the OpenTraj repo
to get the trajectories in approximate world coordinates.
However, in the paper, we report the pixel-coordinate results by inverse-transforming using the estimated scales.
Pre-trained models are publicly available for S-GAN, View Vertically, AgentFormer, Joint AgentFormer, and MemoNet via the submodules; see instructions for downloading them in the respective READMEs within the submodule for each method.
Joint View Vertically models will also be made available.