Changed CheckpointManager to write the latest checkpoint to a consistent filename

[tgaddair]

This changes the checkpoint format from encoding the global step in the filename to having a single latest.ckpt file.

The previous approach required having a background process that would cleanup old checkpoints as training progressed. However, since we only retain the latest checkpoint, this added extra overhead for no benefit, and also caused synchronization issues when background processes would attempt to copy checkpoint files while they were being deleted by the training process.

In this implementation, the global step is instead put into the state dict along with model and optimizer weights. Additionally, checkpoints are first written to a temp directory to guarantee atomicity and prevent background processes from syncing tempfiles that will later be deleted.

[justinxzhao]

This looks like a nice simplification.

[github-actions]

Unit Test Results

       6 files  +       2         6 suites  +2   2h 21m 9s ⏱️ + 48m 25s
2 813 tests ±       0  2 781 ✔️ ±       0    32 💤 ±  0  0 ❌ ±0 
8 439 runs  +2 813  8 339 ✔️ +2 780  100 💤 +33  0 ❌ ±0 

Results for commit dc6ac83. ± Comparison against base commit dd026ca.