CUDA out of memory error

[svsambandam]

Hello :)

When running python experiment_scripts/train_sdf_ibr.py --config_filepath configs/nlrpp_dtu.txt I run out of memory quickly. If I understand correctly, I am running on a GeForce RTX 2080 SUPER GPU with about 6000MiB available. Is there a way to reduce the batch_size? I think the dataloader in train_sdf_ibr.py already has a batch_size of 1. Thank you in advance!!

[alexanderbergman7]

Yes, so 6G of VRAM is going to be quite low for MetaNLR++ (we trained on GPUs with either 24G or 48G). The simplest place to start is to decrease the size of models, especially the image encoder / decoder (the MLP is not that large in comparison). Operating on lower resolution images is also a possibility to significantly reduce the computational overhead; you can change this parameter (load_im_scale) in the config to automatically load in the DTU at a lower resolution.

[alexanderbergman7]

I'll leave it open in case anyone else has this same question and would like to see the discussion.

[ghost]

I have a question
I have eight gpus and each 32G memory, also have the error "RuntimeError:CUDA out of memory, thied to allocate 352.00MB(GPU 0; 31.75GiB total capacity:29.93GiB already allocated;83.69MiB free; 30.19GiB reserved in total by Pytorch)",
It look like doesn't using other Gpus
@alexanderbergman7