Complete separation in time of scenes (like test set in trajnetplusplusdata)

[pedro-mgb]

Hi. Thank you for having the conversion code available for use.

Currently the test set available in data repository does not have the same configuration as the train one. Particularly, the start and end instants of a scene are always different, even though there are portions of tracks that are common to more than one scene.

I was wondering if there was an existing option to convert datasets in that format too, however I failed to find it. Is it publicly available in this repository?

When start and end instants are not completely separate for different scenes (which is why I get when converting data using this code), I have some troubles in running the evaluation, because it's retrieving the last 12 instants for some tracks (because they are apart of the observation portion for other tracks). I can circumvent it if I force what goes in to the models as only the first obs_len positions, so it's not really a big issue. But I feel like this could affect other people that aren't aware of that behaviour.

Thank you for reading. Have a good day!

[theDebugger811]

Hello,
Yes, the test set data generation process is kept different from the training process for the TrajNet++ dataset. You do not have access to the test set generation process for the TrajNet++ challenge. The reason for this is because we do not want researchers to make use of common frames within different scenes to boost their prediction performance.

However, you can easily use the test set generator present in the current repository to generate the testing split for other external datasets. Hope this helps!

[pedro-mgb]

You do not have access to the test set generation process for the TrajNet++ challenge
Of course, I understand that.

So when we create other test sets from external datasets, we have to make sure that these common frames are not an issue. Correct?

[theDebugger811]

Correct. Note: The current Predictor code (for eg. LSTMPredictor) already does that by inputting only obs_length frames as input.

Hope this addresses the concern.

[pedro-mgb]

Thank you for your response and help so far!
I've tested with LSTM, and it indeed seems to make that separation.

However, for LSTM I'm still getting some "-1" on Col-I metric for certain categories of trajectories (type I - static). The same actually happens on Kalman Filter, even when performing the fix you mentioned on this recent issue. However, the Col-II metric has a > 0 value.
More importantly, the overall Col-I metric is not at -1 though.

What does it mean to give -1 on Col-I metric for some trajectory types, but the global Col-I metric is not -1? I feel like this might also be the same issue, but I'll have to investigate further.

[theDebugger811]

Hi Pedro,
I haven't come across this issue except for the particular case when the number of trajectories corresponding to a particular category is 0 in the test set. Then, by default, the Col-I metric of -1. But this won't affect the global performance.

The '-1' Col-I metric indicates that you are not providing prediction of all the neighbors in the observed scene. If this was not the case, a submission won't provide any prediction of neighbors and end up getting ideal Col-I value of 0. To remove this corner case, the '-1' value was introduced. Note: On the AICrowd challenge, instead of '-1', the evaluator outputs '100'

[pedro-mgb]

Thank you for the quick response!

when the number of trajectories corresponding to a particular category is 0 in the test set. Then, by default, the Col-I metric of -1

If there are several dataset files, and one of them does not have trajectories of a certain category (but the others do), that shouldn't result in the Col-I metric being -1 right? I feel like that's what is happening at the moment.
Also, what about partial trajectories? They could have NaN's as prediction (e.g., because most of the observation has NaN's), and perhaps that could affect the calculation of percentages of collisions? I doubt that is the case though.

I will do some more experiments in this regard, with a small subset of data.
I will post a new comment when I reach any conclusions.

Thank you, once again!

[pedro-mgb]

I was now experimenting with kalman filter, and just realized a possible problem with the available implementation in trajnetppbaselines. It relates to the neighbour predictions being 'lost'.

First, a neighbour_tracks variable is created, to which the neighbour predictions are appended to.
Then, a new neighbour_tracks (same name) is initialized with an empty list, eliminating the predictions there. What this means is that the only trajectory being outputted is that of the primary pedestrian, which gives the collision metrics issues.
This line needs to be removed for the kalman to work.

I opened a PR for it on trajnetppbaselines

[theDebugger811]

Yes, this is indeed a bug. Thanks for fixing it. Merged the PR.