NuScenes Evaluation

[codyreading]

Hello and thanks for the release of this code!

I would like to evaluate the tracking results using the NuScenes evaluation. I noticed that this code currently only evaluates one sequence at a time, and produces a single JSON file for each sequence. Would you recommend I just evaluate all sequences and combine them in a single JSON file for evaluation?

[aleksandrkim61]

Hi, thanks for checking out our work!

The code outputs sequence-level evaluations for KITTI, but for NuScenes it produces only one final JSON file after processing the whole data split - you can trace the call of this function. The final result's format should be perfect for NuScenes Devkit evaluation. I am not aware of an easy way to do sequence-level evaluations.

When running on NuScenes script does produce .txt files for each sequence, but they are empty. This functionality is a by-product of KITTI-specific code that I wrote, which I did not remove because it was useful to see how far the run has gone without ssh-ing into the session :). I also attempted to use those .txt files to save progress for each sequence by it became too cumbersome.
Perhaps this led you to this confusion, but those files are completely empty, so you can remove them if you'd like.

[codyreading]

Hello!

So currently the code only produces one JSON file for one sequence, I believe what was happening was due to the break here, which would exit the loop after processing a single sequence. I was able to remove this break and get the functionality that I needed.

Thanks for the help!

[aleksandrkim61]

Oops, I must have left it there after a recent test run.

Thanks for the flag! Fixed it and gave you a shoutout in the commit message :)