If you copy checkpoints from HOME to gcs they can get deleted

[sshleifer]

because of this line

orbax/checkpoint/orbax/checkpoint/utils.py

Line 669 in c7a7fd4

if is_gcs_path(path) and not (path / _COMMIT_SUCCESS_FILE).exists():

They don't have success file but are in GCS so orbax thinks its tmp and cleans it up.

I would suggest always or never saving COMMIT_SUCCESS file.

This is not blocking me (easy to just write extra commit success files once I found this) but it felt like I should report because it was very unexpected behavior and moving around checkpoints is super common.

[cpgaffney1]

Thanks for the report, we currently have different behavior for ensuring atomicity on GCS vs. other filesystems. This was sort of a practice that we inherited from earlier code. I will look into standardizing this.