training suddenly get slower and slower

[Wangzhuoying0716]

Hi, I have met a problem when I tried to train on 2 RTX 8000 GPU card. The first epoch is normal and it prints the log about every 1 minutes. But when it comes to the 2 or 3 epoch, the time becomes 4~5 minutes and the extra presented training time is getting longer. I killed it and then train with 'resume'. It's normal again but after 1 epoch and some iterations, it again gets slower and slower. I'm confused and grateful to get some help !