-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How long you train 800 epochs? #37
Comments
As for me, it takes about 17 hours per 1 epoch (batch size 13 (if more then it will be CUDA out of memory error), 1m images, 2080ti). Pretty slow fitting. Can't understand where is a bottleneck. |
It's too slow to fit, as you mention it spend 17h/1 epoch, so to train as authors that train 800 epochs to fit you have to spend 13600h. It is unacceptable |
Agreed, the model training is quite slow. What is the reason? @ZitongYu |
Hi @ZitongYu
I'd like to know how long you train the model with 800 epochs?
In my experiment, I trained 1 epoch (batch size is 8 and 20000 steps) spending 12 hours on a single GPU (P100).
It's so long and I think something that went wrong, any suggestion for me?
The text was updated successfully, but these errors were encountered: