What is a reasonable training time on kitti dataset?

[g1y5x3]

Hi, thank you for sharing the implementation and I have been testing the script as well as referencing the paper to have a better understanding of the entire implementations. Just wondering what is a reasonable expected training time on the 80G Kitti dataset? So far on my 64G memory machine with NVIDIA 2060 GPU, it takes around 25 minutes per epoch. I have tried with different batch_size but it didn't seem to improve the training time either.

I understand to train with 50G worth of data would not be any where quick but 25 minutes per epoch for 10000 epochs seem to be really out of the normality.

Thanks for your help!

[nubertj]

Hi @g1y5x3 ,
It is hard to say what the right duration is, as it highly depends on the chosen learning-rate.
But for sure I never trained it for more than 200 epochs (you will also see that the loss kinda converges after that amount of time).
Best,

[g1y5x3]

Thank you for your clarification. So the 10,000 epochs that was in deployer/trainer.py was just a generic configuration but not necessarily the one used for reproducing the results right?

[g1y5x3]

Actually, would you mind share the training parameters that you used in the paper?

[nubertj]

yeah, usually I deployed it on servers and wanted to run it as long as possible rather than being killed by an internal epoch limit.

[nubertj]

Which parameters do you mean? model parameters (i.e. weights)?

[nubertj]

There is an example checkpoint for kitti provided here: https://github.com/leggedrobotics/DeLORA/tree/main/checkpoints