Add thresholding_objective argument to AutoMLSearch #2320

bchen1116 · 2021-06-01T15:33:50Z

Adds an additional thresholding_objective argument to AutoMLSearch. We use this to threshold binary classification problems when the original objective isn't thresholdable.

Original design doc here
Perf test results HERE

codecov · 2021-06-03T19:57:54Z

Codecov Report

Merging #2320 (bf1e71a) into main (0277fba) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2320     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        281     281             
  Lines      24858   24907     +49     
=======================================
+ Hits       24825   24874     +49     
  Misses        33      33

Impacted Files	Coverage Δ
evalml/automl/engine/dask_engine.py	`100.0% <ø> (ø)`
evalml/automl/engine/sequential_engine.py	`100.0% <ø> (ø)`
evalml/automl/utils.py	`100.0% <ø> (ø)`
evalml/tests/automl_tests/test_dask_engine.py	`100.0% <ø> (ø)`
evalml/automl/automl_search.py	`99.9% <100.0%> (+0.1%)`	⬆️
evalml/automl/engine/engine_base.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/dask_test_utils.py	`98.8% <100.0%> (+0.1%)`	⬆️
evalml/tests/automl_tests/test_automl.py	`99.7% <100.0%> (+0.1%)`	⬆️
.../automl_tests/test_automl_search_classification.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_engine_base.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0277fba...bf1e71a. Read the comment docs.

bchen1116 · 2021-06-03T20:01:46Z

evalml/tests/automl_tests/test_automl.py

@@ -144,37 +144,37 @@ def test_pipeline_limits(mock_fit_binary, mock_score_binary,
    mock_score_multi.return_value = {'Log Loss Multiclass': 1.0}
    mock_score_regression.return_value = {'R2': 1.0}

-    automl = AutoMLSearch(X_train=X, y_train=y, problem_type=automl_type, max_iterations=1)
+    automl = AutoMLSearch(X_train=X, y_train=y, problem_type=automl_type, optimize_thresholds=False, max_iterations=1)


For most of these tests, I choose to set optimize_threhsholds=False rather than patch predict_proba, optimize_thresholds, and _encode_targets.

bchen1116 · 2021-06-03T20:02:42Z

evalml/tests/automl_tests/test_automl_search_classification.py

+            mock_optimize_threshold.assert_not_called()
+            assert automl.best_pipeline.threshold is None
+            mock_split_data.assert_not_called()
+        else:


Check that we optimize the threshold even if the main objective isn't optimizable

bchen1116 · 2021-06-03T20:04:32Z

evalml/automl/automl_search.py

+        self.threshold_automl_config = self.automl_config
+        if is_binary(self.problem_type) and self.optimize_thresholds and self.objective.score_needs_proba:
+            # use the thresholding_objective
+            self.threshold_automl_config = AutoMLConfig(self.data_splitter, self.problem_type,


This allows us to use the thresholding_objective as the objective to train the best pipeline with if we allow it.

bchen1116 · 2021-06-03T20:22:22Z

evalml/tests/automl_tests/test_automl_search_classification.py

-@patch('evalml.pipelines.BinaryClassificationPipeline.score')
-@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
-@patch('evalml.pipelines.BinaryClassificationPipeline.predict_proba')
-def test_tuning_threshold_objective(mock_predict, mock_fit, mock_score, mock_encode_targets, mock_optimize_threshold, objective, X_y_binary):


The other tests cover this condition already, so removing this

bchen1116 · 2021-06-07T18:25:30Z

@freddyaboulton updated this with the perf test results!

ParthivNaresh

Excellent work on this!

docs/source/release_notes.rst

evalml/automl/automl_search.py

evalml/tests/automl_tests/test_automl.py

… bc_2301_thresholding

dsherry

@bchen1116 looking good! I had one code change request, some comments on naming/docs and a couple points on the tests.

evalml/automl/automl_search.py

evalml/automl/engine/engine_base.py

dsherry · 2021-06-07T22:01:06Z

evalml/automl/automl_search.py

+        if (
+            is_binary(self.problem_type)
+            and self.optimize_thresholds
+            and self.objective.score_needs_proba
+        ):
+            # use the thresholding_objective
+            self.threshold_automl_config = AutoMLConfig(
+                self.data_splitter,
+                self.problem_type,
+                self.thresholding_objective,
+                self.additional_objectives,
+                self.thresholding_objective,
+                self.optimize_thresholds,
+                self.error_callback,
+                self.random_seed,
+                self.X_train.ww.schema,
+                self.y_train.ww.schema,
+            )


@bchen1116 hang on, I think @chukarsten 's point still stands. The logic you added in engine_base.py looks great. And I see you've added the alternate threshold tuning objective as an additional argument to AutoMLConfig above this code block, which looks great. So, why can't you delete this block entirely?

evalml/automl/automl_search.py

dsherry · 2021-06-07T22:07:11Z

evalml/tests/automl_tests/test_automl_search_classification.py

+        if objective == "Log Loss Binary":
+            assert automl.best_pipeline.threshold is None
+        else:
+            assert automl.best_pipeline.threshold == 0.5


Ah, so when "optimize" is true, that means automl will have run in this test with optimize_thresholds true, in which case the mock optimization value is expected, regardless of what the objective was. 👍

If you wanna go above and beyond haha, I bet we could move most of this test to cover engine_base.py code directly instead of being an automl-level test. We wrote this test before the engine concept existed.

@dsherry What do you mean here?

Sorry lol. First part was me explaining to myself what this test does so that I understand it. Second part was a suggested improvement which you can certainly ignore if you'd like.

My point was that technically this test is checking the behavior of EngineBase.train_and_score_pipelines and doesn't have anything to do with AutoMLSearch itself, right? I guess its also checking that the threshold values get saved and attached to AutoMLSearch.best_pipeline, but we could write a simpler test to check that using mocking.

@dsherry I might just leave this test as is since it is particular to time series as well, and it seems to be a thorough enough test that shouldn't take much time. Seems like this test is just making sure that AutoMLSearch can threshold time series problems properly when needed.

evalml/tests/automl_tests/test_automl.py

… bc_2301_thresholding

dsherry

I left one request for refactor in the engine code, to avoid duplicated code. After that, let's 🚢 !

evalml/automl/automl_search.py

evalml/automl/engine/dask_engine.py

evalml/automl/engine/engine_base.py

dsherry · 2021-06-08T00:06:43Z

evalml/tests/automl_tests/test_automl_search_classification.py

@@ -715,6 +715,7 @@ def test_automl_allowed_pipelines_specified_allowed_pipelines_binary(
        X_train=X,
        y_train=y,
        problem_type="binary",
+        optimize_thresholds=False,


Following up on my outdated comment on another one of these similar lines:

Why set these to False?

If you did it to avoid having the tests waste time calling the threshold optimizer, then, lol I agree we shouldn't run the optimizer in every test. I do wonder if there's a different way to accomplish this though. Could we mock BinaryClassificationObjective.optimize_threshold instead, in the same way we mock pipeline fit and score in many tests?

I wasn't just referring to that specific test though heh, I was referring to every automl test where you've added optimize_thresholds=False in this PR.

Ah, I added optimize_thresholds to false to avoid tuning the thresholds. The other way would've been to patch predict_proba, optimize_thresholds, and encode_targets since so many of the tests patch the pipeline fit.

Oh damn yeah I follow. Basically, now that we default to enabling threshold optimization, even if we mock pipeline fit/predict/score, optimization will still run and consume a bunch of test runtime.

@freddyaboulton FYI this could be relevant for #1815 #2298

Thanks for explaining @bchen1116 SGTM

dsherry · 2021-06-08T00:09:51Z

evalml/tests/automl_tests/test_automl_search_classification.py

+        if objective == "Log Loss Binary":
+            assert automl.best_pipeline.threshold is None
+        else:
+            assert automl.best_pipeline.threshold == 0.5


Sorry lol. First part was me explaining to myself what this test does so that I understand it. Second part was a suggested improvement which you can certainly ignore if you'd like.

My point was that technically this test is checking the behavior of EngineBase.train_and_score_pipelines and doesn't have anything to do with AutoMLSearch itself, right? I guess its also checking that the threshold values get saved and attached to AutoMLSearch.best_pipeline, but we could write a simpler test to check that using mocking.

evalml/tests/automl_tests/test_automl.py

… bc_2301_thresholding

evalml/automl/engine/engine_base.py

dsherry

@bchen1116 yep, well done! I left some comments about one last refactor, to minimize duplicated code.

Its great that our binary classification models will now stay tuned

bchen1116 added 12 commits May 13, 2021 12:45

change elastic net classifier defaults

527205e

Merge branch 'main' of github.com:alteryx/evalml into main

2f9eb05

Merge branch 'main' of github.com:alteryx/evalml into main

d281c40

Merge branch 'main' of github.com:alteryx/evalml into main

714d1ae

Merge branch 'main' of github.com:alteryx/evalml into main

7b86cc7

Merge branch 'main' of github.com:alteryx/evalml into main

cc2dac9

Merge branch 'main' of github.com:alteryx/evalml into main

6aab40c

Merge branch 'main' of github.com:alteryx/evalml into main

8068578

Merge branch 'main' of github.com:alteryx/evalml into main

39a33a7

Merge branch 'main' of github.com:alteryx/evalml into main

9edac9b

working changes

3ed9c52

fixing tests

28d06a1

bchen1116 self-assigned this Jun 1, 2021

bchen1116 added 5 commits June 2, 2021 16:25

Merge branch 'main' into bc_2301_thresholding

48c341a

working changeS

afee389

thresholding objective implementaiton

1117f48

Merge branch 'main' into bc_2301_thresholding

fa91a30

fix tests

1da8ce1

bchen1116 commented Jun 3, 2021

View reviewed changes

bchen1116 requested review from dsherry, chukarsten and freddyaboulton and removed request for dsherry, chukarsten and freddyaboulton June 3, 2021 20:38

ParthivNaresh approved these changes Jun 7, 2021

View reviewed changes

docs/source/release_notes.rst Outdated Show resolved Hide resolved

evalml/automl/automl_search.py Outdated Show resolved Hide resolved

evalml/tests/automl_tests/test_automl.py Show resolved Hide resolved

bchen1116 added 5 commits June 7, 2021 17:40

fix message

67da672

Merge branch 'main' into bc_2301_thresholding

4574abb

linting

e2de86c

Merge branch 'bc_2301_thresholding' of github.com:alteryx/evalml into…

8938840

… bc_2301_thresholding

Merge branch 'main' into bc_2301_thresholding

025b5c1

dsherry suggested changes Jun 7, 2021

View reviewed changes

bchen1116 added 3 commits June 7, 2021 19:03

fix tests and address comments

3fa6d63

Merge branch 'bc_2301_thresholding' of github.com:alteryx/evalml into…

4978212

… bc_2301_thresholding

Merge branch 'main' into bc_2301_thresholding

c60da3f

dsherry suggested changes Jun 8, 2021

View reviewed changes

bchen1116 added 3 commits June 8, 2021 12:10

address comments and refactor code

10ac839

update files

c907c68

Merge branch 'bc_2301_thresholding' of github.com:alteryx/evalml into…

8a5eb17

… bc_2301_thresholding

bchen1116 requested a review from dsherry June 8, 2021 17:24

bchen1116 added 2 commits June 8, 2021 15:20

Merge branch 'main' into bc_2301_thresholding

4409d79

rerun tests

7dbaf45

dsherry reviewed Jun 8, 2021

View reviewed changes

evalml/automl/engine/engine_base.py Outdated Show resolved Hide resolved

dsherry reviewed Jun 8, 2021

View reviewed changes

evalml/automl/engine/engine_base.py Outdated Show resolved Hide resolved

dsherry approved these changes Jun 8, 2021

View reviewed changes

bchen1116 added 5 commits June 8, 2021 18:10

address changes

a4e0c1c

Merge branch 'main' into bc_2301_thresholding

062891b

fix release notes

c6d574c

Merge branch 'main' into bc_2301_thresholding

92f8eb0

Merge branch 'main' into bc_2301_thresholding

bf1e71a

bchen1116 merged commit ca42b84 into main Jun 14, 2021

chukarsten mentioned this pull request Jun 22, 2021

Release v0.27.0 #2428

Merged

freddyaboulton mentioned this pull request Sep 27, 2021

Best pipeline trained by AutoMLSearch gets different score than cloned version trained on X_train #2844

Closed

freddyaboulton deleted the bc_2301_thresholding branch May 13, 2022 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add thresholding_objective argument to AutoMLSearch #2320

Add thresholding_objective argument to AutoMLSearch #2320

bchen1116 commented Jun 1, 2021 •

edited

Loading

codecov bot commented Jun 3, 2021 •

edited

Loading

bchen1116 Jun 3, 2021

bchen1116 Jun 3, 2021

bchen1116 Jun 3, 2021

bchen1116 Jun 3, 2021

bchen1116 commented Jun 7, 2021

ParthivNaresh left a comment

dsherry left a comment

dsherry Jun 7, 2021

dsherry Jun 7, 2021

bchen1116 Jun 7, 2021

dsherry Jun 8, 2021

bchen1116 Jun 8, 2021

dsherry left a comment •

edited

Loading

dsherry Jun 8, 2021

bchen1116 Jun 8, 2021

dsherry Jun 8, 2021

dsherry Jun 8, 2021

dsherry left a comment

Add thresholding_objective argument to AutoMLSearch #2320

Add thresholding_objective argument to AutoMLSearch #2320

Conversation

bchen1116 commented Jun 1, 2021 • edited Loading

codecov bot commented Jun 3, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bchen1116 commented Jun 7, 2021

ParthivNaresh left a comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

bchen1116 commented Jun 1, 2021 •

edited

Loading

codecov bot commented Jun 3, 2021 •

edited

Loading

dsherry left a comment •

edited

Loading