Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[train] New persistence mode: Migrate 🐙 Tune tests and examples (medium) #39081

Merged
merged 8 commits into from
Aug 30, 2023
Merged

[train] New persistence mode: Migrate 🐙 Tune tests and examples (medium) #39081

merged 8 commits into from
Aug 30, 2023

Conversation

justinvyu
Copy link
Contributor

Why are these changes needed?

This PR migrates all Tune tests + examples on medium instances.

TODO: chdir_to_trial_dir doesn't work at the moment. To fix in a followup.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@justinvyu
update pipeline file
66018d7 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
test_sample
dd001e2 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Fix test_searchers
aac2414 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
skip hebo tests for pandas>=2.0
1502423 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
test_tuner
23bb444 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
fix lint
388d49f 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
remove new_storage tag for medium tests
9c9767b 
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into tune…
37b6031 
…_tests_medium
@matthewdeng matthewdeng merged commit 7cce140 into ray-project:master Aug 30, 2023
56 of 61 checks passed
@justinvyu justinvyu deleted the tune_tests_medium branch August 30, 2023 00:54
matthewdeng pushed a commit to matthewdeng/ray that referenced this pull request Aug 30, 2023
@justinvyu @matthewdeng
[train] New persistence mode: Migrate 🐙 `Tune tests and examples (med…
926b8b1 
…ium)` (ray-project#39081)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
GeneDer pushed a commit that referenced this pull request Aug 30, 2023
@matthewdeng @justinvyu
@woshiyyya @krfricke @ericl @xwjiang2010
[train][2.7] cherry-picks for documentations, tests, examples (#39105)
9e71973 
* [train] enable new persistence mode for core and serve tests (#38938)

Signed-off-by: Matthew Deng <matt@anyscale.com>

* [train] New persistence mode: Update 🐠 `ML Libraries w/ Ray Client Examples (Python 3.7)` (#38923)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

* [train] remove non-URI assertion (#38944)

Signed-off-by: Matthew Deng <matt@anyscale.com>

* [train] New persistence mode: Update 📖 `Doc tests and examples (excluding Ray AIR examples)` (#38940)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Matthew Deng <matt@anyscale.com>
Co-authored-by: Matthew Deng <matt@anyscale.com>

* disable legacy sync config logic in trainable (#38952)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

* [2.7 CI][New Persistent Mode][6/n] 📖 ✈️ Ray AIR examples (#38918)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>

* [2.7 CI][New Persistent Mode][2/n] 📺 📖 Doc GPU tests and examples (#38905)


Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>

* [2.7 CI][New Persistent Mode][4/n] 📺 🚂 Train GPU tests & 🚂 Datasets Train Integration GPU Tests and Examples (#38910)



Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Co-authored-by: Justin Yu <justinvyu@anyscale.com>

* [2.7 CI][New Persistent Mode][1/n] 📺 ✈️ AIR GPU tests (ray/air) & ⚡ :python: Lightning 2.0 Train GPU tests  (#38903)


Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>

* [train] Fix broken tune tests and support ray storage (#38950)

This PR re-introduces support for ray storage ray.init(storage="s3://...") and fixes a broken tune controller test.

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

* [train] New persistence mode: Finish migrating `xgb`, `lgbm` and `sklearn` trainers, checkpoints + tests (#38959)


Signed-off-by: Justin Yu <justinvyu@anyscale.com>

* [2.7 CI][New Persistent Mode][5/n] 📖 Doc examples for external code (#38915)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>

* [train][rllib] temporarily disable new persistence mode for rllib tests (#38965)

Signed-off-by: Matthew Deng <matt@anyscale.com>

* [2.7 CI][New Persistent Mode][8/n] ✈️ AIR tests (ray/air) (#38932)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>

* [tune] Storage: 🐙 🧠 Tune tests and examples {using RLlib} migration (#38895)

Signed-off-by: Kai Fricke <kai@anyscale.com>
Co-authored-by: matthewdeng <matt@anyscale.com>

* [train] Fix MosaicTrainer example and unit test (#38970)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

* [air/release] Fix dreambooth example image preprocessing logic (#39020)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

* [train] clean up ray.train._checkpoint imports (#38951)

Signed-off-by: Matthew Deng <matt@anyscale.com>

* [train] high level cleanup of Ray Train docs (#38971)

Signed-off-by: Matthew Deng <matt@anyscale.com>

* [wip][docs] update FrameworkPredictor examples (#38634)


Signed-off-by: Matthew Deng <matt@anyscale.com>
Signed-off-by: matthewdeng <matt@anyscale.com>

* [train] Add documentation for using metadata argument to save preprocessors (#38701)

* [Train] Restructure Ray Train Example Page (#38814)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>

* [air] Deprecate some fields/classes that are supposed to be gone in 2.6. (#38794)

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* [tune/storage] Fix Tune multinode tests (#39050)

Fixes multinode tests by using the new train.report() API.

Signed-off-by: Kai Fricke <kai@anyscale.com>

* [tune] Fix BOHB example for new storage (#38983)

The new storage path does not create "empty" checkpoints per default anymore. Previously, when no checkpoint is saved, PAUSEing a trial would create a dummy checkpoint that only contains trial metadata (such as the iteration number). This is not the case anymore.

Examples now have to implement checkpointing to properly restore previous state. This was also true previously - but some of our simple examples (e.g. the one in this PR) didn't implement it and still "worked".

I think it's fine to keep the functionality as is and require our examples to show checkpointing implementations. This will ensure that users don't shoot their feet trying to use e.g. BOHB.

Separately, BOHB was malfunctioning as trials were repeatedly PAUSED and restarted as they've never been removed from `bracket.trials_to_unpause`. @justinvyu mentioned this in the review where it was introduced and I believed at the time it wasn't necessary - turns out it is, as we can end up in a situation where a bracket is never finished because trials are constantly running. This was not caught by any tests. We should add one in a follow-up - for now we can proceed with this PR to pick onto Ray 2.7.

Signed-off-by: Kai Fricke <kai@anyscale.com>

* [Release Test] Fix `long_running_horovod_tune_test`. (#39012)

Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>

* [train] New persistence mode: `StorageContext` unit tests (#39023)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

* [train] enable train + tune tests and examples (#39021)

Signed-off-by: Matthew Deng <matt@anyscale.com>

* [rllib] Fix storage-path related tests (#38947)

This PR fixes rllib-related tests that didn't pass changes related to the new storage context.

Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: matthewdeng <matt@anyscale.com>
Co-authored-by: matthewdeng <matt@anyscale.com>

* [train] New persistence mode: Migrate 🐙 `Tune tests and examples (medium)` (#39081)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

---------

Signed-off-by: Matthew Deng <matt@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: matthewdeng <matt@anyscale.com>
Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Co-authored-by: Justin Yu <justinvyu@anyscale.com>
Co-authored-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com>
Co-authored-by: Eric Liang <ekhliang@gmail.com>
Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com>
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
@justinvyu @arvind-chandra
[train] New persistence mode: Migrate 🐙 `Tune tests and examples (med…
3f8b8c2 
…ium)` (ray-project#39081)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
LeonLuttenberger pushed a commit to jaidisido/ray that referenced this pull request Sep 5, 2023
@justinvyu @LeonLuttenberger
[train] New persistence mode: Migrate 🐙 `Tune tests and examples (med…
7f01712 
…ium)` (ray-project#39081)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
jimthompson5802 pushed a commit to jimthompson5802/ray that referenced this pull request Sep 12, 2023
@justinvyu @jimthompson5802
[train] New persistence mode: Migrate 🐙 `Tune tests and examples (med…
56baa2e 
…ium)` (ray-project#39081)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>
vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023
@justinvyu
[train] New persistence mode: Migrate 🐙 `Tune tests and examples (med…
f2af45f 
…ium)` (ray-project#39081)

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matthewdeng matthewdeng matthewdeng approved these changes

@krfricke krfricke Awaiting requested review from krfricke

Assignees

@matthewdeng matthewdeng

@krfricke krfricke

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@justinvyu @matthewdeng @krfricke