[train][doc] Configuring persistent storage user guide #39428
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
justinvyu
requested review from
krfricke,
xwjiang2010,
amogkam,
matthewdeng,
Yard1,
maxpumperla and
a team
as code owners
Signed-off-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
matthewdeng
reviewed
Sep 8, 2023
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
woshiyyya
reviewed
Sep 8, 2023
|
|
||
| If you save checkpoints with :meth:`ray.train.report(..., checkpoint=...) <ray.train.report>` | ||
| and run on a multi-node cluster, Ray Train will raise an error if NFS or cloud storage is not setup. | ||
| This is because Ray Train expects all workers to be able to write the checkpoint to |
There was a problem hiding this comment.
I like the idea of Ray Train expects all workers to be able to write the checkpoint to the same persistent storage location., that should be the core concept of persistent story. What about moving it to the very top of this user guide?
But logically using head node as persistent storage is also the same persistent storage location. The reason why we no longer support it is because of network communication bottleneck?
There was a problem hiding this comment.
yeah, the deprecation of head node as persistnet storage is described in the issue.
Comment on lines
+152
to
+153
| When providing a custom filesystem, the associated ``storage_path`` is expected | ||
| to be a qualified filesystem path *without the protocol prefix*. |
There was a problem hiding this comment.
Did we check this and raise an error in the code?
There was a problem hiding this comment.
yep, pyarrow will raise an error directly
| network device, such as NFS. | ||
| .. code-block:: text | ||
|
|
||
| s3://bucket-name/sub-path (RunConfig.storage_path) |
…persistence Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
matthewdeng
approved these changes
Sep 9, 2023
matthewdeng
added a commit
to matthewdeng/ray
that referenced
this pull request
Sep 9, 2023
…9428) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matt@anyscale.com>
GeneDer
pushed a commit
that referenced
this pull request
Sep 10, 2023
…39515) * [docs] update doc links in Use Cases (#39445) Signed-off-by: Matthew Deng <matt@anyscale.com> * [train][release] Fix `tune_worker_fault_tolerance` release test node killing (#39233) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] Do not raise warning when no results were reported (#39454) Signed-off-by: Kai Fricke <kai@anyscale.com> * [train] add diagram in overview page (#39512) Signed-off-by: Matthew Deng <matt@anyscale.com> * [train] New persistence mode: Sanity-check release test (#39354) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Co-authored-by: matthewdeng <matt@anyscale.com> * [train][doc] Configuring persistent storage user guide (#39428) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matt@anyscale.com> * [train][doc] New checkpointing user guide (#39505) Signed-off-by: Justin Yu <justinvyu@anyscale.com> --------- Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>
jimthompson5802
pushed a commit
to jimthompson5802/ray
that referenced
this pull request
Sep 12, 2023
…9428) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matt@anyscale.com> Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
…9428) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> Co-authored-by: matthewdeng <matt@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR updates the persistent storage user guide to reflect the improvements in 2.7.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.