[train/tune] Refactor trial metadata organization #38165
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 18 commits
August 4, 2023 12:33
Signed-off-by: Kai Fricke <kai@anyscale.com>
# Conflicts: # python/ray/tune/experiment/trial.py
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
added 2 commits
August 8, 2023 19:31
Signed-off-by: Kai Fricke <kai@anyscale.com>
# Conflicts: # python/ray/tune/experiment/trial.py # python/ray/tune/result_grid.py
xwjiang2010
reviewed
Aug 9, 2023
| from ray.tune.utils.serialization import TuneFunctionEncoder, TuneFunctionDecoder | ||
|
|
||
|
|
||
| class _TrainingRunMetadata: |
There was a problem hiding this comment.
this is just a lump of some of the fields under Trial right? And do we still think everything under Trial is driver's view of things?
I feel like putting this under trainable folder is kinda misleading or are we planning to have trainable take charge of this eventually?
There was a problem hiding this comment.
Talked offline, the idea is to eventually have trainables taking care of the management of these data (low priority for 2.7)
xwjiang2010
approved these changes
Aug 10, 2023
added 5 commits
August 11, 2023 10:12
# Conflicts: # python/ray/tune/analysis/experiment_analysis.py
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
added 3 commits
August 14, 2023 10:21
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
NripeshN
pushed a commit
to NripeshN/ray
that referenced
this pull request
Aug 15, 2023
The Trial object currently keeps properties with different scopes: 1. Static properties that are set **on init** 2. Static properties that are set on init but can be overwritten **on restore** 3. **Temporary** properties that are not saved, e.g. the trial location, 4. **run metadata** that is updated during training, such as the last result, the available checkpoints, etc. This PR refactors the Trial class to explicitly capture 3) and 4) in sub classes. Specifically, it introduces a `_TemporaryTrialState` class that contains temporary properties, and a `_TrainingRunMetadata` class that contains run metadata such as the last result, error files, and available checkpoints. It also changes the way experiment checkpoints are saved. Specifically, we save the trial state (which contains the static properties as well as select runtime metadata (e.g. trial status) separately from the run metadata. This allows us to split these two sources of information in the future. The changes in this PR mean that loading experiment checkpoints from runs before this change will not be possible. However, this is true for any changes to the `Trial` class. Support for backwards compatibility to resume experiments is only guaranteed on a patch level basis, so these changes should be fine. A few other improvements have been made (`Trial.runner` is now `Trial.temporary_state.ray_actor`), and a lot of the changes in this PR are changes to tests. Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: NripeshN <nn2012@hw.ac.uk>
harborn
pushed a commit
to harborn/ray
that referenced
this pull request
Aug 17, 2023
The Trial object currently keeps properties with different scopes: 1. Static properties that are set **on init** 2. Static properties that are set on init but can be overwritten **on restore** 3. **Temporary** properties that are not saved, e.g. the trial location, 4. **run metadata** that is updated during training, such as the last result, the available checkpoints, etc. This PR refactors the Trial class to explicitly capture 3) and 4) in sub classes. Specifically, it introduces a `_TemporaryTrialState` class that contains temporary properties, and a `_TrainingRunMetadata` class that contains run metadata such as the last result, error files, and available checkpoints. It also changes the way experiment checkpoints are saved. Specifically, we save the trial state (which contains the static properties as well as select runtime metadata (e.g. trial status) separately from the run metadata. This allows us to split these two sources of information in the future. The changes in this PR mean that loading experiment checkpoints from runs before this change will not be possible. However, this is true for any changes to the `Trial` class. Support for backwards compatibility to resume experiments is only guaranteed on a patch level basis, so these changes should be fine. A few other improvements have been made (`Trial.runner` is now `Trial.temporary_state.ray_actor`), and a lot of the changes in this PR are changes to tests. Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: harborn <gangsheng.wu@intel.com>
harborn
pushed a commit
to harborn/ray
that referenced
this pull request
Aug 17, 2023
The Trial object currently keeps properties with different scopes: 1. Static properties that are set **on init** 2. Static properties that are set on init but can be overwritten **on restore** 3. **Temporary** properties that are not saved, e.g. the trial location, 4. **run metadata** that is updated during training, such as the last result, the available checkpoints, etc. This PR refactors the Trial class to explicitly capture 3) and 4) in sub classes. Specifically, it introduces a `_TemporaryTrialState` class that contains temporary properties, and a `_TrainingRunMetadata` class that contains run metadata such as the last result, error files, and available checkpoints. It also changes the way experiment checkpoints are saved. Specifically, we save the trial state (which contains the static properties as well as select runtime metadata (e.g. trial status) separately from the run metadata. This allows us to split these two sources of information in the future. The changes in this PR mean that loading experiment checkpoints from runs before this change will not be possible. However, this is true for any changes to the `Trial` class. Support for backwards compatibility to resume experiments is only guaranteed on a patch level basis, so these changes should be fine. A few other improvements have been made (`Trial.runner` is now `Trial.temporary_state.ray_actor`), and a lot of the changes in this PR are changes to tests. Signed-off-by: Kai Fricke <kai@anyscale.com>
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
The Trial object currently keeps properties with different scopes: 1. Static properties that are set **on init** 2. Static properties that are set on init but can be overwritten **on restore** 3. **Temporary** properties that are not saved, e.g. the trial location, 4. **run metadata** that is updated during training, such as the last result, the available checkpoints, etc. This PR refactors the Trial class to explicitly capture 3) and 4) in sub classes. Specifically, it introduces a `_TemporaryTrialState` class that contains temporary properties, and a `_TrainingRunMetadata` class that contains run metadata such as the last result, error files, and available checkpoints. It also changes the way experiment checkpoints are saved. Specifically, we save the trial state (which contains the static properties as well as select runtime metadata (e.g. trial status) separately from the run metadata. This allows us to split these two sources of information in the future. The changes in this PR mean that loading experiment checkpoints from runs before this change will not be possible. However, this is true for any changes to the `Trial` class. Support for backwards compatibility to resume experiments is only guaranteed on a patch level basis, so these changes should be fine. A few other improvements have been made (`Trial.runner` is now `Trial.temporary_state.ray_actor`), and a lot of the changes in this PR are changes to tests. Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
The Trial object currently keeps properties with different scopes: 1. Static properties that are set **on init** 2. Static properties that are set on init but can be overwritten **on restore** 3. **Temporary** properties that are not saved, e.g. the trial location, 4. **run metadata** that is updated during training, such as the last result, the available checkpoints, etc. This PR refactors the Trial class to explicitly capture 3) and 4) in sub classes. Specifically, it introduces a `_TemporaryTrialState` class that contains temporary properties, and a `_TrainingRunMetadata` class that contains run metadata such as the last result, error files, and available checkpoints. It also changes the way experiment checkpoints are saved. Specifically, we save the trial state (which contains the static properties as well as select runtime metadata (e.g. trial status) separately from the run metadata. This allows us to split these two sources of information in the future. The changes in this PR mean that loading experiment checkpoints from runs before this change will not be possible. However, this is true for any changes to the `Trial` class. Support for backwards compatibility to resume experiments is only guaranteed on a patch level basis, so these changes should be fine. A few other improvements have been made (`Trial.runner` is now `Trial.temporary_state.ray_actor`), and a lot of the changes in this PR are changes to tests. Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
The Trial object currently keeps properties with different scopes:
This PR refactors the Trial class to explicitly capture 3) and 4) in sub classes.
Specifically, it introduces a
_TemporaryTrialStateclass that contains temporary properties, and a_TrainingRunMetadataclass that contains run metadata such as the last result, error files, and available checkpoints.It also changes the way experiment checkpoints are saved. Specifically, we save the trial state (which contains the static properties as well as select runtime metadata (e.g. trial status) separately from the run metadata. This allows us to split these two sources of information in the future.
The changes in this PR mean that loading experiment checkpoints from runs before this change will not be possible. However, this is true for any changes to the
Trialclass. Support for backwards compatibility to resume experiments is only guaranteed on a patch level basis, so these changes should be fine.A few other improvements have been made (
Trial.runneris nowTrial.temporary_state.ray_actor), and a lot of the changes in this PR are changes to tests.Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.