Some problems encountered during training

[strawberrieszd]

Hello, thank you very much for your research, when I trained the 16,000 images dataset with 2 GPUs with 24G memory, the training parameters were as follows. After a week of training the test images are as follows, I am wondering if any parameter is set wrong.

[yuval-alaluf]

Most of the parameters seem fine. I would turn off the ID loss since its designed for human faces and will therefore probably not work well on your domain.
However, you definitely have some other problems in the training. The code as is doesn't support multi-GPU training and you mentioned that you ran on two GPUs. Does this mean that you made changes to the code?
Did you try running a sanity check by trying to overfit on say 10 images? To make sure that everything works as expected?

[strawberrieszd]

Thank you very much for your reply., When training with one GPU, I was able to complete inference on 16 images. When the dataset was changed to 3w, the validation results during training showed the images, but when the training weights file was used alone for inference, the results were all noisy, whether it was because my training model had not yet converged and the training loss was still 0.4. I just modified the coach.py file as follows。 [image: image.png] yuval-alaluf ***@***.***> 于2022年8月20日周六 18:55写道：

…

Most of the parameters seem fine. I would turn off the ID loss since its designed for human faces and will therefore probably not work well on your domain. However, you definitely have some other problems in the training. The code as is doesn't support multi-GPU training and you mentioned that you ran on two GPUs. Does this mean that you made changes to the code? Did you try running a sanity check by trying to overfit on say 10 images? To make sure that everything works as expected? — Reply to this email directly, view it on GitHub <#283 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZSJBSQMYLDKBAM2STMHMQLV2C2SVANCNFSM56UC62CQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[yuval-alaluf]

I don't see any image attached.
In any case, I would try doing what I recommended and try overfitting on a very small set of images.
Another important thing to verify is that the average image of your generator looks reasonable.