Training Time Required

[kevaldoshi17]

Hi,

I was trying to train the Timesformer model from scratch on Kinetics-600 and the estimated time was shown as ~9 days. In the paper it was mentioned that the training time is roughly 440 V100 GPU hours. My setup is 8x Titan V GPUs, so I assumed that the training time would be closer to 50 hours. What am I missing here?

[gberta]

The numbers in the paper are reported on Kinetics-400, which is smaller than Kinetics-600. I haven't tested the code with Titan V GPUs so I can't really comment on that.

[kevaldoshi17]

Thanks for the quick reply. Just to confirm, Kinetics-400 on 8 V100 GPUs for 15 epochs should take around 50 hours right?

[gberta]

It should be around ~55 hours, yes. Note that the training process will be significantly slower if you don't assign enough CPU processes for data loading. To the best of my knowledge, it shouldn't be a problem unless you are using SLURM.

[kevaldoshi17]

Yes, I was using SLURM and didn't set enough CPU processes. Thanks for the help!