This release is the new version for torch 2.1. The code is nicer to read, has fewer dependencies (no more flash attention installations), data can now be easily streamed, and training is faster.
The new checkpoints are about 2% better on GLUE with the same budget.