Question About GPU Memory Requirements for Model Training

[sinkers-lan]

Hi, thanks for your inspiring work.

I am attempting to train thte model using an NVIDIA GeForce RTX 3090 GPU, which has approximately 24 GB of VRAM. However, I am encountering a "CUDA out of memory" issue, causing the training process to terminate prematurely.

what is the typical amount of VRAM required for successful training?

[nikitakaraevv]

Hi @sinkers-lan, you can reduce the number of tracks during training from 768 to 324 or 256 by passing --traj_per_sample 256 as an argument.

[sinkers-lan]

thanks for your nice reply