SaveModelCallback every nth epoch

[KeremTurgutlu]

Minor modification to allow saving a model every nth epoch by passing an integer to the existing every_epoch parameter. Main motivation is to reduce the disk space occupied by saving models especially when using with_opt=True during long runs. For example, a training run with 300 epochs can create ~100 GB+ checkpoint files but usually people are fine saving every 10/20/50 epoch during training.

Also, added a test with synthetic learner which trains 4 epoch and saves every 2nd epoch starting from epoch 0.

P.S. If desired, we can also extend this callback to allow every nth iteration. It would be particularly useful when training set is large and users would like to save checkpoints before waiting for a single epoch to complete. I made a similar modification during training CLIP with 40m image-text pairs, simply by adding something like:

def after_batch(self): if self.train_iter % self.every_iter == 0: self._save(f'{self.fname}_iter{self.train_iter}')

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[jph00]

Thanks Kerem :)