Add version suffix to last.ckpt

[carmocca]

🚀 Feature

If you use ModelCheckpoint(save_last=True) and you run an experiment twice in the same directory, then this set of checkpoints is generated:

file-v0.ckpt
file-v1.ckpt
...
last.ckpt (the last of the second run)

the idea is to add a version also to last.ckpt if it would get overwritten:

file-v0.ckpt
file-v1.ckpt
...
last-v0.ckpt (last of the first run)
last-v1.ckpt (last of the second run)

Motivation

Avoid overwriting existing checkpoints

Alternatives

Modifying the default CHECKPOINT_NAME_LAST to avoid the conflict

Additional context

Discussed in #5000 (comment)

To be done after #5008 is merged.

cc @Borda @carmocca @awaelchli @ninginthecloud @jjenniferdai @rohitgr7

[carmocca]

@awaelchli answering #5000 (comment) here

How do you propose we fix this then?

[awaelchli]

@carmocca Before I make a suggestion could you help me understand the context a bit more. Is this issue a) about damage control for users that accidentally write to the same directory, or b) is this about a particular use case where a user is doing this on purpose?

I am asking this because of two reasons. 1) I notice in your description you didn't append epoch numbers to the filenames and the primary reason why the save_last option is exists is for sequential checkpoints numbered by some metric like epoch or step. 2) last.ckpt is always a copy of the last epoch, i.e. epoch_N.ckpt is identical to last.ckpt. Therefore it would be save to overwrite last.ckpt in a subsequent run.

Note that situation a) cannot be solved by appending version affixes to the files. Consider the case where user changes epoch size between runs so they would get a mix of files and you wouldn't be able to tell if epoch=x_step=y.ckpt or epoch=x_step=z.ckpt belonged to the first or second run. One would only be able to tell by inspecting the individual checkpoints for hyperparameters.

[awaelchli]

One may consider this option:

file-v0.ckpt
file-v1.ckpt
...
last-v0.ckpt (last of the first run)
last-v1.ckpt (last of the second run)
last.ckpt (copy of last-v1)

[carmocca]

It is about a)

I notice in your description you didn't append epoch numbers

Sorry, I didn't for brevity

2. last.ckpt is always a copy of the last epoch, i.e. epoch_N.ckpt is identical to last.ckpt. Therefore it would be save to overwrite last.ckpt in a subsequent run.

That is not the case if you are monitoring something. For example, if you run an experiment for 10 epochs and your best performance is epoch_4.ckpt, last.ckpt corresponds to the tenth epoch. We discussed this in #4335 (comment)

Consider the case where user changes epoch size between runs so they would get a mix of files and you wouldn't be able to tell if epoch=x_step=y.ckpt or epoch=x_step=z.ckpt belonged to the first or second run. One would only be able to tell by inspecting the individual checkpoints for hyperparameters.

This is impossible already in master, version suffixes do not imply that all -v0 files necessarily correspond to the first run, and -v1 to the second run, etc. It is just a means of avoiding overwritting files.

[carmocca]

One may consider this option:

file-v0.ckpt
file-v1.ckpt
...
last-v0.ckpt (last of the first run)
last-v1.ckpt (last of the second run)
last.ckpt (copy of last-v1)

if last.ckpt must stay, then I would keep it as is and make version suffixes start at 1.

However, this would be inconsistent with what was discussed in #5000.

So should we just start versions at 1 and forget about renaming files?

file.ckpt (instead of file-v0.ckpt)
file-v1.ckpt
...
last.ckpt (last of the first run)
last-v1.ckpt (last of the second run)

cc @rohitgr7 @Borda

[awaelchli]

For example, if you run an experiment for 10 epochs and your best performance is epoch_4.ckpt, last.ckpt corresponds to the tenth epoch.

This is correct behaviour. last.ckpt as the name suggests should point to the last saved checkpoint in a run, in this case it would be epoch 10 (epoch=9.ckpt). Again, last.ckpt is there for convenience if the user expects their run to be interrupted (for example manually) they can easily restore and continue training with last.ckpt without having to look up the exact filename. Having a file best.ckpt is a separate feature one could look into :)

[carmocca]

Exactly, but you might not have saved on your tenth epoch because your model wasnt in the top-k

last.ckpt is always a copy of the last epoch, i.e. epoch_N.ckpt is identical to last.ckpt. Therefore it would be save to overwrite last.ckpt in a subsequent run.

So epoch_N.ckpt doesnt exist but last.ckpt does. Then it is not safe to overwrite last.ckpt

[awaelchli]

These edge cases are terrible. Looks like the versioning is necessary then.

[rohitgr7]

No strong opinion here. I am fine with both #5030 (comment) renaming or not, 0 or 1 as the first prefix version.

[edenlightning]

@carmocca is this done?

[carmocca]

@carmocca is this done?

No