Default optimize_thresholds to True in AutoMLSearch #1943

bchen1116 · 2021-03-08T17:22:41Z

Perf test here

In-depth dive added here

codecov · 2021-03-08T17:35:06Z

Codecov Report

Merging #1943 (ae727a5) into main (7e8f082) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1943     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         273      273             
  Lines       22279    22356     +77     
=========================================
+ Hits        22273    22350     +77     
  Misses          6        6

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`100.0% <ø> (ø)`
evalml/automl/engine/engine_base.py	`100.0% <100.0%> (ø)`
evalml/automl/utils.py	`100.0% <100.0%> (ø)`
evalml/pipelines/classification_pipeline.py	`100.0% <100.0%> (ø)`
evalml/pipelines/pipeline_base.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl.py	`100.0% <100.0%> (ø)`
.../automl_tests/test_automl_search_classification.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl_utils.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_engine_base.py	`100.0% <100.0%> (ø)`
evalml/tests/pipeline_tests/test_pipelines.py	`100.0% <100.0%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7e8f082...ae727a5. Read the comment docs.

docs/source/demos/cost_benefit_matrix.ipynb

bchen1116 · 2021-03-08T20:55:12Z

evalml/tests/automl_tests/test_automl.py

@@ -83,7 +83,7 @@ def test_search_results(X_y_regression, X_y_binary, X_y_multi, automl_type, obje
        expected_pipeline_class = MulticlassClassificationPipeline
        X, y = X_y_multi

-    automl = AutoMLSearch(X_train=X, y_train=y, problem_type=automl_type, objective=objective, max_iterations=2, n_jobs=1)
+    automl = AutoMLSearch(X_train=X, y_train=y, problem_type=automl_type, objective=objective, optimize_thresholds=False, max_iterations=2, n_jobs=1)


needed to set optimize_thresholds as False in order to avoid issues during training best_pipeline with predict_proba. Otherwise, I could have mocked predict_proba for all the tests but this was a faster approach with similar results.

freddyaboulton

@bchen1116 I have a couple design questions before taking a closer look!

docs/source/demos/cost_benefit_matrix.ipynb

evalml/automl/utils.py

bchen1116 · 2021-03-10T18:56:49Z

evalml/tests/automl_tests/test_automl_search_classification.py

@@ -227,20 +227,21 @@ def test_additional_objectives(X_y_binary):


 @patch('evalml.objectives.BinaryClassificationObjective.optimize_threshold')
+@patch('evalml.pipelines.BinaryClassificationPipeline._encode_targets', side_effect=lambda y: y)


need to patch encode_targets in order to optimize the threshold

Yep, otherwise sklearn will throw an exception. The encoder is fit during pipeline.fit but since we're mocking fit, that doesn't happen.

freddyaboulton

@bchen1116 This looks great!

docs/source/user_guide/automl.ipynb

evalml/automl/engine/engine_base.py

evalml/pipelines/classification_pipeline.py

freddyaboulton · 2021-03-11T21:27:41Z

evalml/tests/automl_tests/test_automl_search_classification.py

@@ -227,20 +227,21 @@ def test_additional_objectives(X_y_binary):


 @patch('evalml.objectives.BinaryClassificationObjective.optimize_threshold')
+@patch('evalml.pipelines.BinaryClassificationPipeline._encode_targets', side_effect=lambda y: y)


Yep, otherwise sklearn will throw an exception. The encoder is fit during pipeline.fit but since we're mocking fit, that doesn't happen.

evalml/tests/pipeline_tests/test_pipelines.py

evalml/tests/automl_tests/test_automl.py

angela97lin

Looks good! Mostly left some nit picky comments heh

evalml/pipelines/classification_pipeline.py

evalml/tests/pipeline_tests/test_pipelines.py

evalml/pipelines/classification_pipeline.py

evalml/tests/pipeline_tests/test_pipelines.py

…_1940_threshold

change optimize to true

9caf42a

bchen1116 self-assigned this Mar 8, 2021

update release notes

677af43

bchen1116 added 9 commits March 8, 2021 14:00

fix doc:

8380df3

Merge branch 'main' into bc_1940_threshold

70a3f07

fix doc

07a8c40

revert

8a432d9

fix doc

be3e7aa

fix doc

07b820b

fix doc

fb86f48

Merge branch 'main' into bc_1940_threshold

b66bd9e

lint

dbedf0f

bchen1116 commented Mar 8, 2021

View reviewed changes

docs/source/demos/cost_benefit_matrix.ipynb Outdated Show resolved Hide resolved

bchen1116 commented Mar 8, 2021

View reviewed changes

bchen1116 marked this pull request as ready for review March 8, 2021 21:15

bchen1116 requested review from dsherry, freddyaboulton, chukarsten, angela97lin, christopherbunn, jeremyliweishih and ParthivNaresh and removed request for dsherry and freddyaboulton March 8, 2021 21:15

freddyaboulton reviewed Mar 8, 2021

View reviewed changes

docs/source/demos/cost_benefit_matrix.ipynb Outdated Show resolved Hide resolved

evalml/automl/utils.py Outdated Show resolved Hide resolved

bchen1116 added 2 commits March 8, 2021 17:48

fix merge

82e599b

linting

fae2d21

bchen1116 marked this pull request as ready for review March 10, 2021 18:54

bchen1116 commented Mar 10, 2021

View reviewed changes

bchen1116 added 3 commits March 11, 2021 09:24

Merge branch 'main' into bc_1940_threshold

46b2e3e

fix release notes

42e1f24

Merge branch 'main' into bc_1940_threshold

060c6c9

bchen1116 requested review from freddyaboulton and chukarsten March 11, 2021 16:36

bchen1116 added 2 commits March 11, 2021 11:37

update

87c8a52

Merge branch 'main' into bc_1940_threshold

353fa44

freddyaboulton approved these changes Mar 11, 2021

View reviewed changes

address comments

b705b38

angela97lin approved these changes Mar 12, 2021

View reviewed changes

angela97lin reviewed Mar 12, 2021

View reviewed changes

evalml/tests/pipeline_tests/test_pipelines.py Outdated Show resolved Hide resolved

angela97lin reviewed Mar 12, 2021

View reviewed changes

evalml/tests/pipeline_tests/test_pipelines.py Outdated Show resolved Hide resolved

angela97lin reviewed Mar 12, 2021

View reviewed changes

evalml/pipelines/classification_pipeline.py Show resolved Hide resolved

angela97lin reviewed Mar 12, 2021

View reviewed changes

evalml/tests/pipeline_tests/test_pipelines.py Outdated Show resolved Hide resolved

bchen1116 added 9 commits March 12, 2021 10:39

add encoding for predict and pred proba

cd4ef35

mocking

aad33b5

Merge branch 'main' into bc_1940_threshold

78e09af

simplify test

6bf2197

Merge branch 'main' into bc_1940_threshold

e1c6235

fix release ntoes

80867c1

Merge branch 'bc_1940_threshold' of github.com:alteryx/evalml into bc…

85eab11

…_1940_threshold

run tests

20d2a90

Merge branch 'main' into bc_1940_threshold

ae727a5

bchen1116 merged commit 96ed697 into main Mar 15, 2021

angela97lin mentioned this pull request Mar 24, 2021

tune_binary_threshold() docstring #1971

Closed

dsherry mentioned this pull request Mar 24, 2021

Release v0.21.0 #2029

Merged

freddyaboulton mentioned this pull request Sep 27, 2021

Best pipeline trained by AutoMLSearch gets different score than cloned version trained on X_train #2844

Closed

freddyaboulton deleted the bc_1940_threshold branch May 13, 2022 15:01

Default optimize_thresholds to True in AutoMLSearch #1943

Default optimize_thresholds to True in AutoMLSearch #1943

bchen1116 commented Mar 8, 2021 •

edited

codecov bot commented Mar 8, 2021 •

edited

bchen1116 Mar 8, 2021

freddyaboulton left a comment

bchen1116 Mar 10, 2021

freddyaboulton Mar 11, 2021

freddyaboulton left a comment

freddyaboulton Mar 11, 2021

angela97lin left a comment

		@@ -227,20 +227,21 @@ def test_additional_objectives(X_y_binary):


		@patch('evalml.objectives.BinaryClassificationObjective.optimize_threshold')
		@patch('evalml.pipelines.BinaryClassificationPipeline._encode_targets', side_effect=lambda y: y)

Default optimize_thresholds to True in AutoMLSearch #1943

Default optimize_thresholds to True in AutoMLSearch #1943

Conversation

bchen1116 commented Mar 8, 2021 • edited

codecov bot commented Mar 8, 2021 • edited

Codecov Report

bchen1116 Mar 8, 2021

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

bchen1116 Mar 10, 2021

Choose a reason for hiding this comment

freddyaboulton Mar 11, 2021

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

freddyaboulton Mar 11, 2021

Choose a reason for hiding this comment

angela97lin left a comment

Choose a reason for hiding this comment

bchen1116 commented Mar 8, 2021 •

edited

codecov bot commented Mar 8, 2021 •

edited