Results table1

[Mirorrn]

Hi,

first of all thanks for your work. In table 1, the result of social gan is much better than the improved version reported in,
https://github.com/agrimgupta92/sgan/blob/master/MODEL_ZOO.md.
Are you sure that you evaluation is the same?

[cschoeller]

Hello @Mirorrn, that is a good question, but difficult to answer. The author provides his pre-trained models via a download link form his dropbox. It could be he re-trained or tuned hyperparameters after his post, without making new commits to the repository. I used his pre-trained models and as far as I remember I evaluated them in his code-base (but made sure, embedded in his code, my own models perform the same as in my own evaluation pipeline). I will have a quick look into that again.

[cschoeller]

Ok, I checked it again. The difference between my evaluation and the original sgan evaluation should be, that I also consider partial trajectories of the target pedestrian i. Lets say pedestrian i is visible in the scene from timestep t0 to tn, then I also considered him/her when this observation period is not a full sequence (20 timesteps). In the social gan they would have skipped this pedestrian. Hope that helps!

[Mirorrn]

Ok, thank you for this quick response. This of course explains why the result is so much better. Because the first time steps are relatively easy to predict. It would have been good to see this directly in the paper, since SGAN, for example, only evaluates the predictions that are 20 time steps long in total.

Thank you for your work and confirmation that the data set is not sufficient. I also find it interesting that so far no paper addresses the collisions in the predictions. Most architectures are based on LSTM which produce independent results during the prediction, which is a guarantee for collisions.

[cschoeller]

Yes, in particular the maximum FDE is smaller when a short trajectory is predicted. This benefitted all models.

I agree, but even in the interaction-aware models collisions seem to happen, which agrees with my conclusions that interaction-aware behavior is not learend. Did you see the issue about collisions in the sgan repo? A student showed clearly that no collision avoidance is happening in sgan. But in general, my hypothesis is that people move rather chaotic and no relevant statistical patterns can be extracted to model interaction-awareness.

[Mirorrn]

Not only the FDE but the ADE too i have tested it with my code!

[cschoeller]

Thats reasonable too, because with shorter prediction horizons the spread between predicted and true trajectory is smaller for all timesteps.