Remove recall objective #784
Conversation
Codecov Report
@@ Coverage Diff @@
## master #784 +/- ##
=======================================
Coverage 99.40% 99.40%
=======================================
Files 150 150
Lines 5578 5588 +10
=======================================
+ Hits 5545 5555 +10
Misses 33 33
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
@eccabay this is looking great! I left a few comments. My main request is that we add a unit test which ensures we can't use recall as an objective in the automl search. Check out this test which is a basic example of how to create an automl search with a particular objective, and this test which is an example of how to expect errors.
| automl = AutoClassificationSearch(objective=Recall(), max_pipelines=1) | ||
| automl.search(X, y) | ||
| assert isinstance(automl.objective, Recall) | ||
| assert automl.best_pipeline.threshold is not None |
There was a problem hiding this comment.
Why add this check? We set binary classification thresholds to 0.5 here, for all binary classification problems, so I'm not sure this check is the right thing to do here.
There was a problem hiding this comment.
I wasn't sure what aspects to check for this test, so I just added elements I saw in other tests. I can definitely remove this if it's unnecessary.
There was a problem hiding this comment.
Got it. Yeah, I'd ask the following: what is this test trying to cover? I think it's checking that we can pass an instance of the recall objective into automl search. In that case, all we really need to do is run the search and make sure there aren't errors. If that seems correct, perhaps all the other lines can be deleted.
| X, y = X_y | ||
| error_msg = 'Could not find the specified objective.' | ||
| with pytest.raises(ObjectiveNotFoundError, match=error_msg): | ||
| AutoClassificationSearch(objective='recall', max_pipelines=1) |
There was a problem hiding this comment.
Awesome, this test looks good.
| def test_recall_object(X_y): | ||
| X, y = X_y | ||
| automl = AutoClassificationSearch(objective=Recall(), max_pipelines=1) | ||
| automl.search(X, y) |
There was a problem hiding this comment.
Looks good, thanks.
Have you ever used test mocking? Unfortunately we don't have a comprehensive pattern in place, but if you search through the auto* tests you'll see some usages of the patch decorator from unittest.mock. In particular, for some automl search tests, we're able to mock the pipelines' fit and predict, or the objectives score. This can be nice because a) most of the time spent during search goes to fitting the pipelines, and making that a no-op saves time, and b) mocking the output of predict and score means we can test for specific edge cases.
In this particular case, I wonder if a good thing to do would be to mock BinaryClassificationPipeline.fit to be a no-op, mock Recall.score to return a predetermined number (say 0.314159), and then mock BinaryClassificationPipeline.predict to return zeros (of the same length as the input) to prevent errors.
You don't need to do that in this PR. But I wanted to throw it out there as food for thought for the future. We're gradually trying to transition most of our automl search tests to use mocking, because it saves a lot of unnecessary unit testing runtime.
| @@ -8,11 +8,13 @@ | |||
|
|
|||
| from evalml import AutoClassificationSearch | |||
| from evalml.automl.pipeline_search_plots import SearchIterationPlot | |||
| from evalml.exceptions import ObjectiveNotFoundError | |||
Closes #476: removes all the recall objectives from the
OPTIONSinobjectives/utils.py. The classes still exist inobjectives/standard_metrics.py, but all uses of any recall classes in tests or docs are removed or replaced.