-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[train] Fix local storage path for windows #39951
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu could you add to the PR description explaining the problem/fix? |
@matthewdeng Done, added a description |
matthewdeng
approved these changes
Sep 30, 2023
justinvyu
added
the
tests-ok
The tagger certifies test failures are unrelated and assumes personal liability.
label
Oct 2, 2023
justinvyu
added a commit
to justinvyu/ray
that referenced
this pull request
Oct 2, 2023
This PR fixes setting `storage_path` on windows by using `Path(...).as_posix()` to join paths, so that it's always of the format `C:/a/b/c/exp_dir/trial_dir` rather than a combination of `/` and `\`. This PR also adds a minimal windows test to make sure it works and doesn't give this error anymore. Ray Tune/Train support for windows is not very comprehensive, since no train/tune CI is ported to windows CI. Signed-off-by: Justin Yu <justinvyu@anyscale.com>
8 tasks
GeneDer
pushed a commit
that referenced
this pull request
Oct 2, 2023
This PR fixes setting `storage_path` on windows by using `Path(...).as_posix()` to join paths, so that it's always of the format `C:/a/b/c/exp_dir/trial_dir` rather than a combination of `/` and `\`. This PR also adds a minimal windows test to make sure it works and doesn't give this error anymore. Ray Tune/Train support for windows is not very comprehensive, since no train/tune CI is ported to windows CI. Signed-off-by: Justin Yu <justinvyu@anyscale.com>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
This PR fixes setting `storage_path` on windows by using `Path(...).as_posix()` to join paths, so that it's always of the format `C:/a/b/c/exp_dir/trial_dir` rather than a combination of `/` and `\`. This PR also adds a minimal windows test to make sure it works and doesn't give this error anymore. Ray Tune/Train support for windows is not very comprehensive, since no train/tune CI is ported to windows CI. Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Setting a
storage_path
when running with Tune/Train on Windows resulted in an error:The reason for this is that
pyarrow.fs.FileSystem.from_uri("C:\\a\b\c")
outputs:LocalFileSystem, "C:/a/b/c"
. Notice that the path got converted to a posix-style path.However, in the
StorageContext
, when we continue joining directories to construct the experiment path, trial path, etc.,os.path.join
will start adding the native\
path separator again on windows, resulting in directories such asC:/a/b/c\exp_dir\trial_dir
with mismatched separators.This causes the error later on when trying to "upload" a file to the correct location at the storage path, since there's some prefix check that fails due to pyarrow expecting something like
C:/a/b/c/exp_dir/trial_dir
.To fix this, I use
Path(...).as_posix()
to join paths, so that it's always of the formatC:/a/b/c/exp_dir/trial_dir
.This PR also adds a minimal windows test to make sure it works and doesn't give this error anymore. Ray Tune/Train support for windows is not very comprehensive, since no train/tune CI is ported to windows CI.
Related issue number
Closes https://github.com/anyscale/runtime/issues/201
#36448
https://discuss.ray.io/t/disable-ray-synching-to-cloud-in-ray-tune/12263
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.