Added ability to stop and resume hyperopt / automl runs #2108
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
justinxzhao
reviewed
Jun 7, 2022
| @@ -835,29 +835,25 @@ def test_frequency_vs_f1_vis_api(experiment_to_use): | |||
| assert 2 == len(figure_cnt) | |||
|
|
|||
|
|
|||
| def test_hyperopt_report_vis_api(hyperopt_results): | |||
| def test_hyperopt_report_vis_api(hyperopt_results, tmpdir): | |||
There was a problem hiding this comment.
Thanks for cleaning up these tests!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds two new API features that work with both hyperopt and automl:
should_stop_hyperoptcallback functionresumeparameter to automl and hyperopt entrypoint functionsThe idea is that the user can configure a callback that will trigger stopping of the entire job when some criteria is met. Then at a later point in time, the user can re-run the job and the state will be picked up from where it left off.
This PR also makes a small change to the output name of hyperopt results. Previously the directory name was randomized like
trainable_func_12345. Now the directory name matches that of theexperiment_nameparam, so that if the user calls the same hyperopt function twice, the default behavior will be to resume the previous job from where it left off. This can be disabled by settingresume=False, which will create a new job and write trial results to the same directory.