Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: CUDA out of memory #12

Closed
CozmoxJC opened this issue Mar 16, 2022 · 1 comment
Closed

RuntimeError: CUDA out of memory #12

CozmoxJC opened this issue Mar 16, 2022 · 1 comment

Comments

@CozmoxJC
Copy link

I was trying to train the model of the owl figure data by
python train.py /home/mea303/Documents/Python/NeRF_project/unisurf-main/configs/DTU/scan0122.yaml

but ran into the error message:
Traceback (most recent call last):
File "/home/mea303/Documents/Python/NeRF_project/unisurf-main/train.py", line 112, in <module> loss_dict = trainer.train_step(batch, it)
...
File "/home/mea303/Documents/Python/NeRF_project/unisurf-main/model/network.py", line 112, in gradient gradients = torch.autograd.grad(
File "/home/mea303/anaconda3/envs/unisurf/lib/python3.10/site-packages/torch/autograd/__init__.py", line 275, in grad return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 7.79 GiB total capacity; 2.13 GiB already allocated; 23.94 MiB free; 2.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I'm using RTX 3060 Ti with cuda version 11.3 on Ubuntu 20.04

Is that graphic card not good enough? How do I fix this problem?

@CozmoxJC
Copy link
Author

CozmoxJC commented Mar 24, 2022

nvm, I'm just being dumb.
Just find the process engaging gpu memory and kill it or reboot the PC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant