Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About train! #4

Closed
XLR-man opened this issue Sep 26, 2023 · 6 comments
Closed

About train! #4

XLR-man opened this issue Sep 26, 2023 · 6 comments

Comments

@XLR-man
Copy link

XLR-man commented Sep 26, 2023

Hi
My GPU is V100 with 32GB memory. And I have only one gpu. But I found with your default setting, the training speed is very slow. For example, with the default setting(batch_size=1024.......), it takes about half an hour to finish 100 steps, so if i want to finish 10w steps,i need twenty days!
My question is why the speed is so slow? Can I change the batch_size to speed up? Or other variants need to change?Can you give me some suggestions?

@onpix
Copy link
Owner

onpix commented Sep 27, 2023

Can you please check how long it takes to train the multinerf using the default settings? If it also takes 20 days, there might be an issue with your environment or machine. As far as I remember, I need around 12 hours on V100S or 8 hours on A100 to train our model or multinerf.

@XLR-man
Copy link
Author

XLR-man commented Sep 27, 2023

I found the code isn't train with GPU though i distributed one gpu.
When i change the batch_size from 1024 to 1, it can run fastly.
But both are not use gpu, they use cpu
image
image
Do you know what is the problem?

@onpix
Copy link
Owner

onpix commented Sep 27, 2023

Again, it's a good idea to try running multinerf. If you found you can not use GPU while running multinerf, it is likely due to your environment, like an incorrect jaxlib version. But if you found you can run multinerf well but failed to run our code, please let me know as it's possibly because of our bugs.

@XLR-man
Copy link
Author

XLR-man commented Sep 27, 2023

image
Maybe is environment. How can i fix it?

@XLR-man
Copy link
Author

XLR-man commented Sep 27, 2023

the problem above i have fixed.But there is another problem:
image

@onpix
Copy link
Owner

onpix commented Oct 1, 2023

It also looks like an environment problem. You may need to check the dependeny of packages.

@onpix onpix closed this as completed Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants