Fix BOHB, change n_iter -> n_trials, fix up early stopping #81
Conversation
| space = { | ||
| "n_estimators": CS.UniformIntegerHyperparameter("n_estimators", 100, 200), | ||
| "min_weight_fraction_leaf": (0.0, 0.5), | ||
| "min_samples_leaf": CS.UniformIntegerHyperparameter( | ||
| "min_samples_leaf", 1, 5) | ||
| } | ||
|
|
||
| tune_search = TuneSearchCV( | ||
| RandomForestClassifier(), | ||
| space, | ||
| search_optimization="bohb", | ||
| n_iter=3, | ||
| max_iters=10) |
There was a problem hiding this comment.
@Yard1 I ended up removing this example because:
- BOHB requires a form of iterative updating but
- random forests don't have an iterative update ability (from what I can read - please let me know if this is wrong!)
Happy to add this back, but would want to understand how to make this work practically.
There was a problem hiding this comment.
I'm fine with it being removed, thanks for working on this.
|
@richardliaw So just to confirm, BOHB requires partial fit or warm start to work correctly, yes? |
|
Yeah, same with other early stopping mechanisms. |
| return hasattr(self.main_estimator, "partial_fit") | ||
|
|
||
| @property | ||
| def main_estimator(self): |
There was a problem hiding this comment.
Thanks for the catch on partial_fit not found for list objects!
| @@ -143,8 +143,8 @@ class TuneSearchCV(TuneBaseSearchCV): | |||
| this parameter is ignored for ``"bohb"``, as it requires | |||
| ``HyperBandForBOHB``. | |||
|
|
|||
| n_iter (int): Number of parameter settings that are sampled. | |||
| n_iter trades off runtime vs quality of the solution. | |||
| n_trials (int): Number of parameter settings that are sampled. | |||
There was a problem hiding this comment.
This would clear confusion with max_iters, but do you think it'd be helpful to specify that this is the equivalent of n_iter from scikit-learn? I remember we discussed naming max_iters before too so I'm wondering if it's a problem about this parameter's name or max_iters's.
There was a problem hiding this comment.
Hm, good point. I think the reference to sklearn doesn't make sense since sklearn doesn't support "iterative models / leveraging partial fit". Since we do, it becomes a source of confusion.
You can see the diff here: inventormc/tune-sklearn-1@warm-start...richardliaw:bobh
Here are the main changes incorporated in the PR: