-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to avoid "CUDA out of memory" #11
Comments
Hi, |
Hi , |
Great. Thanks for the update. |
Hello @Lilyo Could you please share what hardware are you training on? I am trying on a machine with 48GB VRAM, generating class images of size 768 x 968 and I get the same error as you even with the suggested optimisation flags. Works fine if I use 512 x 512 sized images |
Hello, I use the same GPU as you, and still have memory overflow according to your configuration. Can you help me? |
Hi,
Thank you for your great work! Now when I run train.py using the example data(creature) I get the following error and I want to know if this script can be run using a single RTX3090 with 24G VRAM?
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 320.00 MiB (GPU 0; 23.70 GiB total capacity; 22.26 GiB already allocated; 89.44 MiB free; 22.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Look forward to your reply! Thnaks!
The text was updated successfully, but these errors were encountered: