How to avoid "CUDA out of memory"

[Lilyo]

Hi,
Thank you for your great work! Now when I run train.py using the example data(creature) I get the following error and I want to know if this script can be run using a single RTX3090 with 24G VRAM?

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 320.00 MiB (GPU 0; 23.70 GiB total capacity; 22.26 GiB already allocated; 89.44 MiB free; 22.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Look forward to your reply! Thnaks!

[omriav]

Hi,
I did not try it on RTX myself, but, can you please try to use xformers by adding --enable_xformers_memory_efficient_attention flag when executing the training script?
You may need to install the package first first via pip install xformers.

[Lilyo]

Hi ,
Eventually I solved it by (1) using 8-bit Adam, (2) set none grads, now it works, thank you!

[omriav]

Great. Thanks for the update.

[whiterose199187]

Hello @Lilyo

Could you please share what hardware are you training on? I am trying on a machine with 48GB VRAM, generating class images of size 768 x 968 and I get the same error as you even with the suggested optimisation flags.

Works fine if I use 512 x 512 sized images

[wdy321]

Hi , Eventually I solved it by (1) using 8-bit Adam, (2) set none grads, now it works, thank you!

Hello, I use the same GPU as you, and still have memory overflow according to your configuration. Can you help me?