How many samples are used for evaluation in End-to-End leaderboard?

[foolhard]

Hello,

I have question about the frames number used for evalution in End-to-End Forecasting Leaderboard.

1. Submission file of test set will contain 1/10 frames of a sequence according to 1Hz sampling rate, totally about ~2400 frames. Is it correct?
2. Will all frames in the submission file be used to calculated the metrics?

Thanks.

[neeharperi]

1. Yes, this is correct
2. Yes, all frames in the submission file will be used to calculate metrics

[foolhard]

Thanks for kindly feedback.

[foolhard]

@neeharperi
I am confused about the conversion code of forecast_labels by convert_forecast_labels() in av2.evaluation.eval as below:
for future_frame in frames[frame_idx + 1 : frame_idx + 1 + constants.NUM_TIMESTEPS]:

Here you use 6 timesteps, which means future 6s trajectories, because the dataset is 1Hz sampled. However, the competition requires to forecast next 3s.

Is there anything misaligned here?

[neeharperi]

Great catch! I think our configuration changed during debugging, we will push an update to EvalAI. All you need to do is change this line from 10 to 5 and re-run create_data.py.

The original sensor dataset is collected at 10 Hz. (After the fix described above) We subsample this by a factor of 5 to match nuScenes 2Hz sampling rate. 6 timesteps sampled at 2 Hz is 3 seconds.

[foolhard]

Very clear. Then I understand how to subsample the sensor dataset for forecasting.

Another quesiton about data:
I find out that you extract the 1st frame of each sequence through this line. Actually, 1st frame does not have the historical frames, thus it may decrease the model forecasting accuracy.
What do you think?

[neeharperi]

Yes, you are correct that the first frame does not have any history. I don't believe it will have a significant impact on forecasting accuracy. It is up to the method to determine the right way to address this case when there is no history.