Training got stuck

[Linya-Peng]

Hi, ywyue!
Thank you for your wonderful work.
I tried to train on the Structured3D dataset, however, the training got stuck midway without any error being reported.

I tried to set --num_workers=0, but the problem hasn't been resolved.
I've tried terminating and resuming training multiple times, but the epoch at which it gets stuck varies each time. Do you have any suggestions for a solution?

I'm using 1.9.0+cu111 and running the main.py in a WSL2 Ubuntu 20.04

[ywyue]

Hi @Linya-Peng, thank you for your interest in our work! The problem you said is weird to me (I didn't meet that). What's your GPU card (and memory)? Could you just try a smaller batch size (e.g. 2) and see whether it will solve your problem?

[Linya-Peng]

Hi @Linya-Peng, thank you for your interest in our work! The problem you said is weird to me (I didn't meet that). What's your GPU card (and memory)? Could you just try a smaller batch size (e.g. 2) and see whether it will solve your problem?

Hi ywyue, thank you for your reply!
I've tried set batch size as 2, but the problem still existed. After several attemps and searching, I have not solve this problem.
I'm working on TRX 4070 Ti GPU, with 12GB GPU memory. Do you have any suggestions for the hardware capacity needed for training?

[ywyue]

Hi @Linya-Peng, our code is tested on NVIDIA TITAN RTX with 24 GB memory! Do you also meet similar issues when running other programs with your current GPU?

[Linya-Peng]

Hi @Linya-Peng, our code is tested on NVIDIA TITAN RTX with 24 GB memory! Do you also meet similar issues when running other programs with your current GPU?

Thank you for your reply! I didn't meet similar issues when running other programs. I'm not sure if it's because I'm using WSL, but I've run other projects successfully with the same configuration.

[ywyue]

Hey, sorry for the late follow-up. I cannot reproduce your issue without WSL. Have you tried training on native Ubuntu or other Linux systems (not WSL)?

[ywyue]

Close for now. Feel free to reopen it if you have any concerns.