New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[air] `pyarrow.fs` persistence (8/n): `ray.train.Checkpoint` restore: `resume_from_checkpoint` #38143

Merged

ericl merged 113 commits into ray-project:master from justinvyu:air/persistence/restore_new_checkpoint_rfc

Aug 9, 2023

Contributor

justinvyu commented Aug 5, 2023 •

edited

Why are these changes needed?

This PR supports Trainer(resume_from_checkpoint) with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

justinvyu added 30 commits

July 27, 2023 14:24


          Pipe storage context to Trainable (used now for Trainable syncing)

abb1307

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Don't use the storage context in the trial/trainable

f6ff90a

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Disable all trainable syncing in new codepath

562369f

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Pipe storage context to Train workers (not actually used yet)

95a3d20

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Update to use the new checkpoint id attribute

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Add todo comment to remove legacy path

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix race condition for setting checkpoint_uri

484e67f

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix cyclical import

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add simple trainer test

8c856b8

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add legacy prefix to train session checkpoint uri

78c525f

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add new checkpoint class

e97f471

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          New train session report implementation using new checkpoint

64945be

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Simplify checkpoint propagation from user code (in worker) -> trainer…

c6480c9

… -> driver

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          New tune session.report

c681ccb

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Save direction works with new checkpoint API

795bafe

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Update test with e2e trainer test

8a084bc

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Make callback supporting new checkpoint a todo for now

725d802

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Remove unnecessary comment

877acb9

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

ee4ccbd

…persistence/storage_context_to_worker_temp


          Separate out the new set checkpoint id from the old set checkpoint uri

88042b3

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

a5eeab2

…persistence/new_checkpoint

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Update id -> index

a6cd9dc

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Address comments on error to raise with old ckpt type

01f34bb

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Move checkpoint upload logic to a helper fn of storage ctx

65e7a27

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint for session.py

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Drop a checkpoint marker after uploading

f2a4c36

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

Fix lint for storage.py

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add a simplified checkpoint manager

49ee126

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fixes to checkpoint manager

ffa0dd4

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add unit test for simplified checkpoint manager

15553f7

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Full test coverage

00cc9d7

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add a simplified checkpoint manager

cb5990e

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fixes to checkpoint manager

2db9aae

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Add unit test for simplified checkpoint manager

a2067b7

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

justinvyu added 15 commits

August 4, 2023 18:10


          Support + test resume_from_checkpoint

c7de72a

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix result grid bug when no checkpoints saved

db016da

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

bcbcec9

…persistence/restore_new_checkpoint_autoft

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          fix merge conflict remainder

d7497e1

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Recover trainable metadata from last_result rather than .tune_metadata

aeb89ba

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix restore info log

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Keep current checkpoint index synchronized on the driver

e897eaa

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Remove checkpoint dirname parsing

3eef417

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

0e52384

…persistence/restore_new_checkpoint_autoft


          Update todo comment

89631ab

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix lint

d4e20f2

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Merge branch 'air/persistence/restore_new_checkpoint_autoft' into air…

877c4ae

…/persistence/restore_new_checkpoint_rfc


          Merge branch 'master' of https://github.com/ray-project/ray into air/…

d367e8d

…persistence/restore_new_checkpoint_rfc

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Fix e2e test for storage_path=None case

d329e1b

Signed-off-by: Justin Yu <justinvyu@anyscale.com>


          Remove unused code

e382e29

Signed-off-by: Justin Yu <justinvyu@anyscale.com>

justinvyu assigned ericl

justinvyu requested a review from ericl

August 8, 2023 20:13

justinvyu marked this pull request as ready for review

August 8, 2023 20:13

ericl approved these changes

View reviewed changes

justinvyu added the tests-ok label

Contributor Author

justinvyu commented Aug 8, 2023

@ericl Good to merge. #38128 ready for review

ericl merged commit 55acb74 into ray-project:master

67 of 72 checks passed

yutsai84 pushed a commit to yutsai84/ray that referenced this pull request


          [air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…

42ca6fe

… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: Yu-Cheng Tsai <yucheng.tsai@sage.com>

justinvyu deleted the air/persistence/restore_new_checkpoint_rfc branch

August 9, 2023 01:15

ericl mentioned this pull request

[tune/train] Implement new persistence strategy and roll out as default option #38294

Closed

NripeshN pushed a commit to NripeshN/ray that referenced this pull request


          [air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…

f795d0b

… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: NripeshN <nn2012@hw.ac.uk>

harborn pushed a commit to harborn/ray that referenced this pull request


          [air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…

3de6bb6

… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: harborn <gangsheng.wu@intel.com>

harborn pushed a commit to harborn/ray that referenced this pull request


          [air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…

35f398f

… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

arvind-chandra pushed a commit to lmco/ray that referenced this pull request


          [air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…

39925a9

… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

vymao pushed a commit to vymao/ray that referenced this pull request


          [air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…

a82afa9

… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: Victor <vctr.y.m@example.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment