Update AutoML to use objective decision function during scoring for custom objectives - #1934
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1934 +/- ##
=========================================
+ Coverage 100.0% 100.0% +0.1%
=========================================
Files 273 274 +1
Lines 22381 22418 +37
=========================================
+ Hits 22375 22412 +37
Misses 6 6
Continue to review full report at Codecov.
|
| predictions = self._predict_with_objective(X, ypred_proba, objective) | ||
| return infer_feature_types(predictions) | ||
|
|
||
| def _predict_with_objective(self, X, ypred_proba, objective): |
There was a problem hiding this comment.
Helper function that takes in predict_proba, so we don't have to recalculate it for each objective.
bchen1116
left a comment
bchen1116
left a comment
There was a problem hiding this comment.
Looks good. Left a question for my understanding, but implementation looks good to me
chukarsten
left a comment
chukarsten
left a comment
There was a problem hiding this comment.
I think there's something funky with the double anys on L77 of binary_classification_pipeline.py. Besides that, this looks pretty solid.
freddyaboulton
left a comment
freddyaboulton
left a comment
There was a problem hiding this comment.
@angela97lin This looks good to me! I left some scattered comments throughout. One thing that would be nice to address before merge is adding a time series test to test_binary_predict_pipeline_use_objective
| _threshold = None | ||
| problem_type = ProblemTypes.BINARY | ||
|
|
||
| @property |
There was a problem hiding this comment.
Moved all of this to BinaryClassificationPipelineMixin
| @patch('evalml.pipelines.MulticlassClassificationPipeline.fit') | ||
| @patch('evalml.pipelines.MulticlassClassificationPipeline.score') | ||
| @patch('evalml.pipelines.MulticlassClassificationPipeline.predict') | ||
| def test_pipeline_thresholding_errors(mock_multi_predict, mock_multi_score, mock_multi_fit, |
There was a problem hiding this comment.
Removed multiclass tests and moved to test_binary_classification_pipelines.py
4 participants
Closes #1868
BinaryClassificationPipelineMixinto share logic between binary classification / time series binary classification pipelines regarding the pipeline threshold.optimize_thresholdfrom ClassificationPipeline to BinaryClassificationPipelineMixin. I believe this makes sense since only binary classification pipelines, not multiclass classification pipelines, know about thresholds.