Add callback for error handling in AutoMLSearch #1403
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1403 +/- ##
=========================================
+ Coverage 100.0% 100.0% +0.1%
=========================================
Files 213 214 +1
Lines 13975 14040 +65
=========================================
+ Hits 13968 14033 +65
Misses 7 7
Continue to review full report at Codecov.
|
angela97lin
requested review from
dsherry,
freddyaboulton,
bchen1116,
christopherbunn,
eccabay and
jeremyliweishih
There was a problem hiding this comment.
Looks good! I agree with @freddyaboulton that we should add the traceback as part of saving the errors. Could be done in this PR or another but would be great for looking glass.
There was a problem hiding this comment.
@angela97lin this looks great!
I left a few comments, but there was nothing major.
RE Protocol subclassing, I think we're ok to skip that for now, but its a good idea.
evalml/automl/automl_search.py
Outdated
| @@ -674,22 +680,14 @@ def _compute_cv_scores(self, pipeline, X, y): | |||
| logger.debug(f"\t\t\tFold {i}: {self.objective.name} score: {scores[self.objective.name]:.3f}") | |||
| score = scores[self.objective.name] | |||
| except Exception as e: | |||
| if self.error_callback is not None: | |||
| self.error_callback(e, self, fold_num=i, pipeline=pipeline) | |||
There was a problem hiding this comment.
Cool, so because we pass self back in the callback, someone could theoretically write a callback which responds to failure and calls automl.search again. So this is implementing one part of an asynchronous API (#1047 ). Neat!
Closes #1217
Questions:
If all objectives fail and
AutoMLExceptionis raised, we will raise since it indicates that something is likely wrong. This is the same behavior as what is currently implemented.