About normalization

[sjtuxcx]

Hi, I have noticed that in the code (

AgentFormer/model/agentformer.py

Line 535 in cf13e40

self.data['scene_orig'] = torch.cat([self.data['pre_motion'], self.data['fut_motion']]).view(-1, 2).mean(dim=0)

) the center of both the past trajectory and the future trajectory is used to normalize the input data. However, the future trajectory should not be available in the test. Is this data snooping? Please let me know if there is anything wrong with my understanding. Many thanks!

[Khrylx]

Thanks for spotting this! This is indeed an oversight on my part.
I think you could try to normalize with the positions at t=0 (self.data['pre_motion'][[0]]), retrain the model, and rerun the evaluation. I will also try to do this later.

[Khrylx]

I'm retraining with the correct scene_orig. It is worth noting that all the other models in the ablation study also use this scene_orig, so it does not give our method an edge in the ablations.

[Mirorrn]

Hello all,

first of all, thank you for publishing the code, which is very helpful for the community!

I can only partially agree with this, as the calculated anchor point for the proposed normalisation and calculation of the prediction does have an influence on the prediction. For example, the ETH scene has many 1-person trajectories. Considering that the history consists of 8 time steps and the prediction of 12, the anchor point may well contain information about where the person might be in the future. This is not the case with the Univ dataset with many people in one scene.

I have retrained the models with

AgentFormer/model/agentformer.py

Line 535 in cf13e40

self.data['scene_orig'] = torch.cat([self.data['pre_motion'], self.data['fut_motion']]).view(-1, 2).mean(dim=0)

changed to:

self.data['scene_orig'] = self.data['pre_motion'].view(-1, 2).mean(dim=0)
All models are trained exactly like in the given config files.
Here are the results for the ETH/UCY data set:

Scene | ADE/FDE
ETH | 0.454/0.762
HOTEL | 0.144/0.233
UNIV | 0.252/0.45
Zara1 | 0.18/0.31
Zara2 | 0.14/0.241
Avg | 0.234/ 0.4

I still find the results very good and very similar to the Trajectron++. I think that the result on the dataset is almost perfect and I don't think that the result can be improved by more than 5%.

For the NuScenes dataset I have the problem that I could not reproduce the published results with the proposed (wrong anchor point) using the available script, which i had not for ETH/UTC. I followed all the instructions and downloaded the entire NuScenes dataset and converted it to a file of about 56 MB with the preprocessing script. When I use the model published by you, I get the result in the paper, but unfortunately not when I train it myself. Am I doing something wrong or is there something wrong with the config file? Here are the results for NuScenes-5samples:

Model | ADE\FDE
original | 1.73 / 3.56
correct Anchor | 1.87/ 3.92

Again very similar to the Trajectron, but I hope that there is something wrong with the config file and that I can get much better results.

Many greetings,
Mirrorn

[Khrylx]

Hi @Mirorrn ,

Thank you for your detailed analysis! I will keep investigating this error, and update the paper with the correct numbers.

One thing you may want to try is to change the anchor point to

self.data['scene_orig'] = self.data['pre_motion'][[-1]].view(-1, 2).mean(dim=0)

So that it will use t=0 as the scene origin.

For the nuScenes dataset, I haven't retrained the model yet, but I will retrain it soon with the correct anchor. It could be there is something wrong with the configs or something is changed during the code cleaning process.

Thanks!

[Mirorrn]

Hi,

unfortunately I have no time this week to test it with t=0 as anchor point.

Many greetings
Mirrorn

[Khrylx]

Hi Mirrorn,

I’m actively running experiments and hopefully will have updates soon.

Thanks again for your help!

[Khrylx]

Sorry for the late update. I was busy with a deadline.
I have modified the code using the correct scene_orig computation. Indeed, the performance has dropped quite a bit 😢. Still, our current correct model is a bit better than Trajectron++ on nuScenes. I have updated the pretrained models and the numbers both in the repo and the arXiv paper.
Thanks again for everyone's participation and help!

[Khrylx]

I will close this issue for now, but if anything else comes up, please let me know.