model_history -> save_best_model_per_estimator #283
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
sonichi
commented
Nov 15, 2021
•
sonichi
commented
- if save_best_model_per_estimator is False and retrain_final is True, unfit the model after evaluation in HPO.
- retrain if using ray.
- update ITER_HP in config after a trial is finished.
- change prophet logging level.
- example and notebook update.
- allow settings to be passed to AutoML constructor. Are you planning to add multi-output-regression capability to FLAML #192 Is multi-tasking allowed? #277 can pass the auotml setting to the constructor instead of requiring a derived class.
- remove model_history.
- checkpoint bug fix.
sonichi
requested review from
qingyun-wu,
int-chaos,
liususan091219 and
a team
Not every test. Only if you need to save the best model per estimator. |
int-chaos
approved these changes
Nov 15, 2021
| the states can best_trial.get_best_checkpoint return a non-empty value. | ||
| """ | ||
| import torch | ||
| def evaluate(self, eval_dataset=None, ignore_keys=None, metric_key_prefix="eval"): |
liususan091219
approved these changes
Nov 17, 2021
qingyun-wu
approved these changes
Nov 18, 2021
flaml/automl.py
Outdated
| return self._model_history | ||
| self._settings = settings | ||
| settings["time_budget"] = settings.get("time_budget", 60) | ||
| settings["task"] = settings.get("task", "regression") |
There was a problem hiding this comment.
The default task is regression? Used to be classification.
flaml/automl.py
Outdated
| ensemble=None, | ||
| eval_method=None, | ||
| log_type=None, | ||
| save_best_model_per_estimator=None, |
There was a problem hiding this comment.
Is it necessary to change 'model_history' to save_best_model_per_estimator? Won't be compatible with code using older versions.
flaml/automl.py
Outdated
| self._n_concurrent_trials = n_concurrent_trials | ||
| self._early_stop = early_stop | ||
| self._use_ray = use_ray or n_concurrent_trials > 1 | ||
| self._state.save_best_model_per_estimator = save_best_model_per_estimator | ||
| # use the following condition if we have an estimation of average_trial_time and average_trial_overhead |
There was a problem hiding this comment.
Move these two commented lines to line #1861
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.