-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sipmask:mmdetection CUDA out of memory error #32
Comments
Generally, it may need some more memory if for image instance segmentation. If you want to reduce the memory, maybe you can try use 2x upsampling about the basis mask. |
@JialeCao001 I see.Thanks for your reply!I noticed that you used 8GPUs during training.Could you tell me what the memory size of per GPU? |
I remember about maybe 20G per GPU. If we limit the number of proposals for mask prediction per GPU, it maybe less than 20G. |
Thank you very much for your prompt reply!It really helps me a lot. |
Hi,when training with the original code,I got RuntimeError as below:
There was no one using GPU when I trained it. First I thought it might be a batchsize issue. But even though I changed the batch size from 16 to 1,the same error still occured .The only difference between batchsize 16 and batchsize 1 is ,when changed to batchsize 1,the training was able to run for half epoch,then the error occurred.
I wonder if the code didn't clear the grad during training .
BTW,my GPU is GeForce RTX 2080 Ti ,11019M.During training I used 2 GPUs.
My environment is
pytorch = 1.1.0 torchvision = 0.3.0 mmcv = 0.4.3
Please tell me how to deal with it.Thanks a lot!
The text was updated successfully, but these errors were encountered: