`ModelCheckpoint` with `save_best_only=True` shouldn't delete previous directory

[ibeltagy]

ModelCheckpoint with save_best_only=True deletes everything in the checkpointing directory before saving the new checkpoint. If the checkpointing directory is not set correctly, it might delete other files that it shouldn't be deleting. While seems unlikely, it is not, because the documentation here [https://github.com/williamFalcon/pytorch-lightning/blob/e0dbc8ab46450fc485f8f2a5c2c996cd1f442a92/pytorch_lightning/trainer/trainer_io.py#L19] uses filepath=os.getcwd(),, which means it will delete the current working directory with the first checkpoint.

Two things need to be fixed:

• Instead of deleting everything, just overwrite old checkpoint files.
• Updating the documentation to point to a different default directory so that people don't delete their cwd by mistake.
  It would also be nice to show an example how to use TestTubeLogger with ModelCheckpoint, something like filepath=os.path.join(logger.name, f'version_{logger.version}', "checkpoints")

[neggert]

See also #394, #413

[neggert]

You ought to be seeing a warning if there are files in the directory. Is that happening?

[ibeltagy]

I agree filepath=os.getcwd() was removed from the code, but it is still in the documentation, so if you want to customize the checkpointing criteria and copy code from the documentation, you end up with your cwd deleted.

Yes, the warning is there but it is easy to miss or interpret as deleting "checkpointing" files not "all" files. This should definitely be fixed regardless of updating the documentation or not. You shouldn't delete any files, just overwriting old ones.

[williamFalcon]

@ibeltagy new docs fix this