Add version suffix to saved model checkpoint with a version

[rohitgr7]

🚀 Feature

Currently, if a duplicate filename is found in the checkpoint directory saved by the ModelCheckpoint, then a version is added to it and the old file is kept as it is. Here is an enhancement request to update the old file too with a version.

Before

file.ckpt
file-v0.ckpt
file-v1.ckpt

After

file-v0.ckpt
file-v1.ckpt
file-v2.ckpt

cc @carmocca

[carmocca]

On it 💪

[carmocca]

Should I include adding version suffixes to last.ckpt files?

[rohitgr7]

Should I include adding version suffixes to last.ckpt files?

Isn't it saved only once?

[carmocca]

It is for one run, but if you do a re-run, it gets overwritten

[rohitgr7]

oh... it's a bug then. Yep version suffixes should be added. If I have a file already present in the directory, not saved by pl.ModelCheckpoint, then it should remain untouched.

[carmocca]

oh... it's a bug then. Yep version suffixes should be added. If I have a file already present in the directory, not saved by pl.ModelCheckpoint, then it should remain untouched.

Okay, let's do it after #5008 is merged then. Created #5030 to track it.

[awaelchli]

@carmocca the filename last.ckpt needs to be kept for easy access, which is what it is for (save_last=True) :)

[ananthsub]

agreed w/ @awaelchli that we shouldn't change the last.ckpt name by adding versions to it. it needs to be deterministic for restarting

[Borda]

@carmocca the filename last.ckpt needs to be kept for easy access, which is what it is for (save_last=True) :)

but if you have in the folder already one last.ckpt so the new one (which you want to use) is last-v0.ckpt
cc: @ananthsub

oh... it's a bug then. Yep version suffixes should be added. If I have a file already present in the directory, not saved by pl.ModelCheckpoint, then it should remain untouched.

Okay, let's do it after #5008 is merged then. Created #5030 to track it.

@rohitgr7 @carmocca seems we have two issues on the same topic, right? let's close one of them...

[carmocca]

@rohitgr7 @carmocca seems we have two issues on the same topic, right? let's close one of them...

This is about version suffixes for normal checkpoints.
#5030 is the same for last checkpoints.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[carmocca]

Closing as we don't plan to do this since it would be inconsistent with the requirements of #5030 where the original last.ckpt couldn't be renamed for backwards compatibility