Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
FloatingPointError: Minimum loss scale reached (0.0001). #1529
The loss is overflowing repeatedly, which causes batches to be thrown away. fairseq eventually terminates training so that you don't waste computation indefinitely.
There are a few options: