How can I speed up the training for 3D derenderer?

[czsc123]

The training for 3D derenderer is very slow. It maybe need two months or more.How can I speed up the training?

[ysymyth]

Could you please share more info, like what hardware you're using?

[czsc123]

the picture is about the hardware I use.
when pretrain , I use the command : CUDA_VISIBLE_DEVICES=4,6 python geometric/scripts/main.py --do train --dataset vkitti --mode pretrain --num_epochs 256 --batch_size 128 --lr 1e-3 --lr_decay_epochs 16 --lr_decay_rate 0.5 --working_dir_root ./models --name vkitti-geometric-derender3d-pretrain --image_size 128 --render_size 192

[czsc123]

I use NVIDIA TeslaV100. It has 8 GPU, 16G per GPU graphics card. GPU driver version is 396.26.

[stmharry]

Hi @czsc123,

From the command you are running, it seems like you are pre-training the model with MSE losses and rendering is not required. I suggest you make sure the CPU usage is not capped (htop command) while you are training as pytorch can be quite input-bound. Also it would be great to provide a snapshot while training where we can see estimated time for completion.

[czsc123]

@stmharry Thank you for your suggestions. I will try it.

[czsc123]

Hello @stmharry,

I have tried to install htop command. But I don't have root permission to install it. I can only use top command. So can I use top command to make sure the CPU usage is not capped? And what operation should I do to make sure the CPU usage is not capped?
Please pardon me as I am new to these operations. I google your suggestions but I can't find suitable answers.