Add EvalMLAlgorithm - #2525
Add EvalMLAlgorithm#2525
EvalMLAlgorithm#2525Conversation
Codecov Report
@@ Coverage Diff @@
## main #2525 +/- ##
=======================================
+ Coverage 99.9% 99.9% +0.1%
=======================================
Files 291 293 +2
Lines 26664 26881 +217
=======================================
+ Hits 26621 26838 +217
Misses 43 43
Continue to review full report at Codecov.
|
| class EvalMLAlgorithm(AutoMLAlgorithm): | ||
| """An automl algorithm that consists of two modes: fast and long. Where fast is a subset of long. | ||
|
|
||
| 1. Naive pipelines: |
There was a problem hiding this comment.
Thought it would be nice to list it out here - can remove as well.
There was a problem hiding this comment.
Note: this current algorithm doesn't have any feature engineering steps as that is blocked by #2439.
| pipeline_params=None, | ||
| custom_hyperparameters=None, | ||
| n_jobs=-1, | ||
| number_features=None, |
There was a problem hiding this comment.
Not sure if number_features is still necessary anymore. I still see it in IterativeAlgorithm but don't see it mentioned anywhere else in the codebase.
| self._custom_hyperparameters = custom_hyperparameters or {} | ||
| self._selected_cols = None | ||
| self._top_n_pipelines = None | ||
| if custom_hyperparameters and not isinstance(custom_hyperparameters, dict): |
There was a problem hiding this comment.
There's some duplicate logic from IterativeAlgorithm that we can refactor into a base class later on.
There was a problem hiding this comment.
Please file an issue with specifics!
bchen1116
left a comment
bchen1116
left a comment
There was a problem hiding this comment.
Did a quick look-over and it's looking pretty good! Left a few suggestions on ways to clean up the code.
| for pipeline_dict in self._best_pipeline_info.values() | ||
| ] | ||
| estimators.sort(key=lambda pipeline: pipeline[1]) | ||
| estimators = estimators[:n] |
There was a problem hiding this comment.
I see that this is choosing the best estimators based on the mean_cv_score, which makes sense. I think the sorting should be done based on what objective is being used, since R2==1 is the best for regression problems but log loss == 0 is the best for classification problems.
There was a problem hiding this comment.
all the scores coming in should have been converted to "lower is better" so this order should be ok!
| next_batch = self._create_long_top_n(n=3) | ||
| elif self.batch_number % 2 != 0: | ||
| next_batch = self._create_ensemble() | ||
| else: |
There was a problem hiding this comment.
Nice! I like this if/elif/else logic
There was a problem hiding this comment.
100%. I think this is really making it clear how this is working. Huge fan.
chukarsten
left a comment
chukarsten
left a comment
There was a problem hiding this comment.
Jeremy, so sorry I delayed you for such a long time. This is really good work. Not going to lie - I was intimidated by the IterativeAlgorithm, but this really feels and looks a lot cleaner and easier to understand. I think there's a few places that can use some cleanup or refactor.
- add_result() maybe needs a breakup. It's a little thick and could use a bit more addition to the docstring.
- The hardcoded 10 and 50 should be replaced with variables and possibly exposed as params for the EvalMLAlgorithm
- EvalMLAlgorithm should be renamed "JeremyAlgorithm"
- The use of the pipeline's new function in various places should be explored and determined whether it's necessary or not. I'd recommend doing a commit that just pulls out the new calls and see what happens. I feel like those are done because they're an ancient and hallowed tradition or we're uncertain whether we need yet another copy of the pipeline.
- File an issue for creating another common baseclass with Iterative Algorithm.
But yea, this is really good work and super clean. I can see why it took such a long time as it's polished. Well done.
| self._custom_hyperparameters = custom_hyperparameters or {} | ||
| self._selected_cols = None | ||
| self._top_n_pipelines = None | ||
| if custom_hyperparameters and not isinstance(custom_hyperparameters, dict): |
There was a problem hiding this comment.
Please file an issue with specifics!
| {"Select Columns Transformer": {"columns": self._selected_cols}} | ||
| ) | ||
| next_batch.append( | ||
| pipeline.new(parameters=parameters, random_seed=self.random_seed) |
There was a problem hiding this comment.
I'm curious how necessary the new pipeline is in all cases throughout this code.
There was a problem hiding this comment.
it's not necessary in this case as @bchen1116 pointed out but should be when we're creating new pipelines from a tuner proposed parameter set.
| next_batch = self._create_long_top_n(n=3) | ||
| elif self.batch_number % 2 != 0: | ||
| next_batch = self._create_ensemble() | ||
| else: |
There was a problem hiding this comment.
100%. I think this is really making it clear how this is working. Huge fan.
| if self.batch_number == 2 and self._selected_cols is None: | ||
| if is_regression(self.problem_type): | ||
| self._selected_cols = pipeline.get_component( | ||
| "RF Regressor Select From Model" | ||
| ).get_names() | ||
| else: | ||
| self._selected_cols = pipeline.get_component( | ||
| "RF Classifier Select From Model" | ||
| ).get_names() | ||
|
|
||
| current_best_score = self._best_pipeline_info.get( | ||
| pipeline.model_family, {} | ||
| ).get("mean_cv_score", np.inf) | ||
| if ( | ||
| score_to_minimize is not None | ||
| and score_to_minimize < current_best_score | ||
| and pipeline.model_family != ModelFamily.ENSEMBLE | ||
| ): | ||
| self._best_pipeline_info.update( | ||
| { | ||
| pipeline.model_family: { | ||
| "mean_cv_score": score_to_minimize, | ||
| "pipeline": pipeline, | ||
| "parameters": pipeline.parameters, | ||
| "id": trained_pipeline_results["id"], | ||
| } | ||
| } | ||
| ) |
There was a problem hiding this comment.
I'm not sure how I feel about all this being in add_result(), not like you really have much other choice, but I think maybe some of what's happening might want to make its way into the docstring. I'm not 100% sure what _selected_cols is, or why it's being updated and it's also good to know that the _best_pipeline_info is being updated here.
chukarsten
left a comment
chukarsten
left a comment
There was a problem hiding this comment.
LGTM! Thanks for making the changes!
3 participants
Fixes #2347.
Still need do the following:
Saving custom_hyperparameter and pipeline params for AutoMLSearch integration.