Make checkpoint saving fully atomic

[radomirgr]

Bug description

Checkpoint is no longer atomic. I was implemented here to create checkpoints with ".part" and the rename them, but it is no longer implemented that way looking at code here.

Could we implemented it again in that way? If you kill a job that is training during checkpoint it will corrupted the file

What version are you seeing the problem on?

master

How to reproduce the bug

No response

Error messages and logs

# Error messages and logs here please

Environment

Current environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info

No response

cc @Borda @awaelchli

[awaelchli]

@radomirgr Feel free to implement this if you'd like. The contribution is welcome.

[corwinjoy]

This looks straightforward. I will go ahead and create a PR to restore this behavior.

[corwinjoy]

I have created a PR to address this which uses a fsspec transaction context. This should make saves atomic again.
#20011