Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tune] New persistence mode cleanup: Delete legacy utilities (air.remote_storage, etc.) #40207

Merged
merged 28 commits into from
Nov 9, 2023
Merged

[tune] New persistence mode cleanup: Delete legacy utilities (air.remote_storage, etc.) #40207

merged 28 commits into from
Nov 9, 2023

Conversation

justinvyu
Copy link
Contributor

@justinvyu justinvyu commented Oct 9, 2023 •
edited

Why are these changes needed?

This PR removes some legacy utilities, including air._internal.remote_storage, TrainableUtil, and more.

TODO for follow-up

  • Fix the HDFS storage path test.
  • Add new filesystem util unit tests.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@justinvyu
remove TrainableUtil
c5e6591 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove delete node util
6bbcd99 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
add back delete path fn
bda09b8 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove old air stoage telemetry
4c61770 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove air remote storage utils from tune util
e7758f3 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove unused sync down code in tuner
c4b79ba 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove unused util
102b797 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Remove broken hdfs test
bb0b34b 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove telemetry util
cbf84bc 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
move windows path util to data
b43627e 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove remaining deps on remote storage utils
50559ee 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove remote storage test
e3b838f 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
add todo in hdfs test
e296e83 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
add skeleton test
fe620fc 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove legacy remote storage utils
ef2e91f 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove unused utils
62c024c 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
fix lint
c0ae979 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
fix find newest exp ckpt behavior
130105b 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove unused exp_checkpoint_dir arg
72ce683 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into remo…
55f982d 
…ve_unused_code

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into remo…
aa10a50 
…ve_unused_code
@justinvyu
Add hdfs e2e test
ea17ac2 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into remo…
bd1a1b1 
…ve_unused_code

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove hdfs test for now (cannot debug it easily)
4472e94 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
isort
f42cdf6 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into remo…
f83ac98 
…ve_unused_code
@justinvyu
skip in the test instead
a44a59f 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into remo…
63ae4f7 
…ve_unused_code
bveeramani
bveeramani approved these changes Nov 9, 2023
View reviewed changes
Copy link
Member

@bveeramani bveeramani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data changes LGTM

@justinvyu justinvyu merged commit 837ec26 into ray-project:master Nov 9, 2023
23 of 26 checks passed
@justinvyu justinvyu deleted the remove_unused_code branch November 9, 2023 20:12
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Nov 29, 2023
@justinvyu @ujjawal-khare
[tune] New persistence mode cleanup: Delete legacy utilities (`air.re…
9fd9b8d 
…mote_storage`, etc.) (ray-project#40207)

This PR removes some legacy utilities, including `air._internal.remote_storage`, `TrainableUtil`, and more.

---------

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matthewdeng matthewdeng matthewdeng approved these changes

@bveeramani bveeramani bveeramani approved these changes

@ericl ericl Awaiting requested review from ericl ericl is a code owner

@scv119 scv119 Awaiting requested review from scv119 scv119 is a code owner

@c21 c21 Awaiting requested review from c21 c21 is a code owner

@amogkam amogkam Awaiting requested review from amogkam amogkam is a code owner

@scottjlee scottjlee Awaiting requested review from scottjlee scottjlee is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang stephanie-wang is a code owner

@Zandew Zandew Awaiting requested review from Zandew

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@justinvyu @matthewdeng @bveeramani