Estimate Training Time

[rakashi]

Hi,
I have successfully started training using ILSVRC2012 Train and Validation datasets. The training dataset contains 12,81,167 images and Validation dataset contains 50,000 images. I am running this code on Intel i5 Processor with 16GB RAM and 2TB HDD. I have kept default 90 epochs. How much time it will take to complete 90 epochs. I am expecting for one epoch may be it will take one and half day time. Can we reduce the time to complete training faster ?

[matteo-dunnhofer]

Hello @rakashi,

for my training I used a machine with 32 CPU cores and an NVIDIA K40c GPU. It took circa 4-5 days for roughly 46 epochs. With your setup the training would take months 😅 consider use some AWS instance that has a GPU, it would not cost too much.

[qihongl]

@dontfollowmeimcrazy Just out of curiosity how many epochs it took to converge? Are 90 epochs enough?

[matteo-dunnhofer]

Hi @qihongl

I trained the model for about half of them (40-45 epochs) and I had pretty decent results. You can find them in the README file.

[qihongl]

I see! I didn't realize that you stopped at 45 epochs because it converged! Thank you very much!

By the way, do you think using batch norm or fancier optimizers, such as Adam can make learning faster? I saw you commented the batch norm code. Did it make the training process worse?