InD dataset results

[mkofinas]

Hello Colin,

Thank you for releasing the source code of this excellent work. I have some trouble replicating your results for the InD traffic trajectory dataset. I am running the default script you provide to train and evaluate the model and using the results from eval_results_test_last_driver5burnin.txt. The error that I get from these files is ~0.23 at 40 steps (averaged across 5 seeds). However, in your figures, the reported number is ~0.025. Do the reported numbers come from a different experimental setting and if so, how can I access them?

Thank you in advance.

[cgraber]

Hi Miltiadis,

Thanks for your interest in this work! I took a look at the settings used to generate the plot for the paper, and I realized the wrong eval args were being used in the script uploaded here. This has been changed (you'll want to look at eval_results_test_shortseq_driver5burnin.txt). Hopefully this resolves the issue - let me know how it goes.

[mkofinas]

Hi Colin,

Thank you for your immediate response and for solving this issue. I am wondering, though, why is there such a gap in performance? From what I understand, the reported numbers in both scenarios are 40 steps in the future, even if in one case we are predicting the whole future of the scene. Is my assumption correct and if so, why is there such a performance gap?

Thanks again for your time.

[cgraber]

This is a good question, and I don't know the answer for sure. A few possibilities are:

• There are fewer overall sequences in the other setting, so average performance is hurt more by sequences with bad performance.
• There is a larger proportion of "easy" sequences in the new setting.

Of course, there could be another reason I'm missing, but these are the reasons I would investigate first.

[mkofinas]

Ok, thanks a lot for your help. These were my first thoughts as well. I will investigate these directions further.