New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Training and OOM #8
Comments
MANet training doesn't take much memory. Did you turn on
|
Thanks for reply.
|
It's strange because MANet is a tiny model and consumes little memory. Do you have any problems testing the model? Can you try to set |
Thanks for reply. |
is that to run |
I think it's the problem of your GPU. Can you train other models normally? Can you test the MANet on your GPU? |
My Gpu is 2080 Ti only get 11GB. Is that need a gpu with bigger ram to train it? |
I don't think so. 2080 should at least be enough when manet_nf=32. Can you try to monitor the gpu usage by |
Thanks a lot. The problem is solved. I can run the training now. |
Thanks for your code.
I tried to train the model with train_stage1.yml, and the Cuda OOM.
I am using 2080 Ti, I tried to reduce the batch size from 16 to 2 and the GT_size from 192 to 48.
However, the training still OOM.
May I know is there anything I missed?
Thanks.
The text was updated successfully, but these errors were encountered: