Models breaking after several hours

[Picslook]

It's happened many times now. After a few hours the preview will suddenly change from attempting to merge the face to a changing blob of solid colors. After a while the colors seem to settle on red and from then on the model becomes broken and, as far as I know, unfixable unless you have a backup. Sometimes it's the third row of each image which goes completely red and sometimes it's the second, regardless attempting the merge process while the models are on this broken red state results in blank faces in the final output.

[kellurian]

I've see this happen too with the original trainer as well. Not sure what is going on. Have had some luck with lowering the batch sizes. Looking into it. I had a Issue loaded on the main branch of faceswap, but I closed it as I thought I had fixed the issue, but no luck as of yet as I am still getting the issue at times.

[Picslook]

@kellurian I've now found, though I am not completely certain about this, that if you rerun alignments on an alignments directory the likelihood you'll get the "red screen of death" decreases. It might be that in certain situations, if any images are missings in a directory and according to alignments.json those images are suppossed to be there, when the training process tries to get to those missing images, the process fails and ruins the model. I didn't check the code for exceptions - my programming abilities are fairly limited- so I don't know if these situations are in fact being handled properly, but up until now my anecdotal evidence seems to indicate it is helping.

[Picslook]

@ And just as I type this the model breaks. Wow, incredible timing. It had been running for well over 24 hours, this is the longest I've had it work. I guess you should completely disregard what I just said. I honestly have no clue why this might be.