New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The total training time #20
Comments
Hi @csuzhuzhuxia , If you consider 1- or 3-plane model on the ShapeNet dataset, you should be able to at least run 300K iterations in a day. It is strange that it takes so long for you to train. Maybe you can check when the overhead is. I doubt that it might be in the data loading. Best, |
Thanks for the quick reply!! |
@csuzhuzhuxia you have closed the issue but I am still curious where the overhead was? Thanks! |
Sorry for the late reply. I am busy in other things in these days, so I just close the issue although I don't find the reason. And thank you for your concern. Thank you! |
Still strange because I was also keeping the dataset in HDD (or actually SSD?) and it worked well. But yup, thanks for letting me know! |
Hello, I train the model in another machine which installed SSD disk to verify the reason. The speed in SSD disk can reach to nearly 300K iteration a day using shapnet-1-plane setting. zhu |
I just checked the machine I used to run the model and I also kept the ShapeNet dataset in HDD. Therefore, this should not be the problem. For the 1-plane model on ShapeNet, when I used GTX 1080ti with HDD disk, I just checked again my output files and it took really only 8 hours to get 300K iterations. I suggest you checking the following:
Best, |
Thanks for your quick reply. -Zhu |
Hi:
thanks for making this great work available here!
I want to know the total training time for 3000 epochs for the model in this paper (or if you use a single GPU ? ). I find it's so long for me to train the model on a single 2080Ti.
The text was updated successfully, but these errors were encountered: