Is an 8x NVIDIA RTX 3090 GPU Setup Sufficient for Training Models in the Predictive World Model 2024 Competition?

[Liury99]

I am excited about participating in the Predictive World Model 2024 competition and have been preparing my environment accordingly. My current setup includes a system with 8 NVIDIA RTX 3090 GPUs, which I believed would be more than capable of handling the training demands of the competition's models.

However, even after adjusting the configuration settings to the minimum requirements as per the competition guidelines, I'm encountering a persistent issue where I run out of memory. The error I receive is as follows:

RuntimeError: CUDA out of memory. Tried to allocate 60.00 MiB (GPU 7; 23.70 GiB total capacity; 21.79 GiB already allocated; 18.81 MiB free; 21.97 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Is an 8x RTX 3090 GPU setup insufficient for training the competition models, or might there be an issue with my configuration or approach?

[tomztyang]

Hi,

You can try using smaller image scale, set supervise_all_future to False, larger voxel size for reducing number of points for supervision, other backbones, or with_cp to True in backbone to reduce memory cost.

Hope these work for you. ViDAR indeed requires a large GPU memory cost and OpenScene dataset has 8 input cameras.

Thanks,
Zetong

[qiuqc1]

In the past few days, I have been trying to use a single 3090 running vidar to reproduce on nuscenes. How can I reduce the memory of the nuscenes data set?
I have tried using mem_efficient_vidar_1_8_nusc_3future.py, future_queue_length_train=1, future_pred_frame_num_train=1, future_queue_length_test=1, future_pred_frame_num_test=1, queue_length=1, but still not working

[qiuqc1]

Hi,

You can try using smaller image scale, set supervise_all_future to False, larger voxel size for reducing number of points for supervision, other backbones, or with_cp to True in backbone to reduce memory cost.

Hope these work for you. ViDAR indeed requires a large GPU memory cost and OpenScene dataset has 8 input cameras.

Thanks, Zetong

In the past few days, I have been trying to use a single 3090 running vidar to reproduce on nuscenes. How can I reduce the memory of the nuscenes data set?
I have tried using mem_efficient_vidar_1_8_nusc_3future.py, future_queue_length_train=1, future_pred_frame_num_train=1, future_queue_length_test=1, future_pred_frame_num_test=1, queue_length=1, but still not working

[qiuqc1]

Hi,
You can try using smaller image scale, set supervise_all_future to False, larger voxel size for reducing number of points for supervision, other backbones, or with_cp to True in backbone to reduce memory cost.
Hope these work for you. ViDAR indeed requires a large GPU memory cost and OpenScene dataset has 8 input cameras.
Thanks, Zetong

In the past few days, I have been trying to use a single 3090 running vidar to reproduce on nuscenes. How can I reduce the memory of the nuscenes data set? I have tried using mem_efficient_vidar_1_8_nusc_3future.py, future_queue_length_train=1, future_pred_frame_num_train=1, future_queue_length_test=1, future_pred_frame_num_test=1, queue_length=1, but still not working

我修改了image scale，在config里面加入了一行

即可解决

[tomztyang]

Great to hear. I think reducing the computational & GPU memory cost of ViDAR is of great importance :_).