Training auto resume

[supersergiy]

Re-running the same spec will now automatically resume from the last checkpoint, if present.

Unfortunately it wasn't as easy as passing ckpt_path="last" to trainer.fit because that is actually not properly implemented for GCS checkpoints.

[codecov]

Codecov Report

Merging #154 (36e7507) into main (2b75dce) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main      #154   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           66        66           
  Lines         1905      1905           
=========================================
  Hits          1905      1905           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[nkemnitz]

And in case I want a new experiment based on the state of the last one, I should specify #ENCODER_CKPT and #DECODER_CKPT?

[nkemnitz]

@supersergiy Is the last checkpoint always the end of an epoch, or is that name also used for checkpoints in the middle of an epoch? I saw a warning when I explicitely tried to restart in the middle of an epoch: https://pytorch-lightning.readthedocs.io/en/stable/clouds/fault_tolerant_training_basic.html

[supersergiy]

Yes, if you want to resume with a change of learning rate, the easiest way is to load model weights rather than the full state. There are other ways to do it that involve specific modifications of the training regime. If this becomes a problem, we can make a base zetta regime class that does this. However, if you're changing train/val data and not hyper parameters, full state checkpoint should work

CC Lightning-AI/pytorch-lightning#5339 Lightning-AI/pytorch-lightning#12118

last checkpoint is not the end of the last epoch, but the last checkpoint written out. I think the warning is caused by the fact that the checkpoint doesn't remember where it is in the epoch, as it doesn't save anything about the dataset. That means training is resumed from the beginning of the epoch, and so some samples are going to be used more than others. This is a minor concern in our usecase, and resuming only from epoch start is impractical.