[REP] Consolidated persistence API for Ray Train/Tune

[ericl]

This REP proposes standardizing on the recently introduced storage_path option for Ray Train/Tune, deprecating other legacy persistence paths.

[pcmoritz]

Thanks for putting this together and adding the implementation recommendations!

In the motivation you should consider adding the prevalence of larger and larger models in generative AI and LLMs.

[ericl]

In the motivation you should consider adding the prevalence of larger and larger models in generative AI and LLMs.

Good point, indeed that is one of the reasons why persistence has become increasingly a pain point over the past year.

[krfricke]

Generally I'm very much in favor of simplifying this.

Note that removing the support for automatic syncing will change the way users interact with the code. Until now, code "just worked" in distributed setups without specifying a storage path.

Now, fault-tolerant distributed training will not work out of the box on distributed setups without additional configuration from the user side.

This means that the code that works locally may have to be changed to work on the cloud. We remove complexity from the library setup and push the responsibility to the cluster setup and configuration, i.e. to the user.

In many cases, this is preferred - synchronization of large model checkpoints was very inefficient, and was thus a footgun that could lead to problems. By making the storage configuration explicit we can remove inefficient behavior (syncing large checkpoints via the object store).

It's just something to be aware of - our examples will all have to include a storage path definition, or, if users rely on the RAY_STORAGE environment variable, they have to be aware of this in the docs to avoid a frustrating onboarding experience.

[ericl]

Thanks for the comments @krfricke @woshiyyya , they should be incorporated now.

[krfricke]

Thanks for the update! One last question remaining from my side.

[krfricke]

Thanks for the clarifications

[ericl]

@zhe-thoughts this can be merged