Skip to content

Fix threshold return in AutoMLSearch #3360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bchen1116 merged 3 commits into from
Mar 7, 2022
Merged

Fix threshold return in AutoMLSearch #3360

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6282fc2
implement threshold reset
bchen1116 Mar 4, 2022
9c722cb
update release notes
bchen1116 Mar 4, 2022
8bd11ac
add test
bchen1116 Mar 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/release_notes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
* Enhancements
* Added caching capability for ensemble training during ``AutoMLSearch`` :pr:`3257`
* Fixes
* Fixed ``get_pipelines`` to reset pipeline threshold for binary cases :pr:`3360`
* Changes
* Documentation Changes
* Testing Changes
Expand Down
2 changes: 1 addition & 1 deletion evalml/automl/automl_search.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1276,7 +1276,7 @@ def get_pipeline(self, pipeline_id):
)
new_pipeline = pipeline.new(parameters, random_seed=self.random_seed)
if is_binary(self.problem_type):
new_pipeline.threshold = pipeline.threshold
new_pipeline.threshold = None
Copy link
Contributor

@freddyaboulton freddyaboulton Mar 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be better to take the average threshold over none? My concern with setting it to None is that it won’t reflect the threshold used to compute the scores in the leaderboard. In my opinion, taking the average threshold is closer to matching the mean_cv_score we use to sort the leaderboard.

Copy link
Contributor

@freddyaboulton freddyaboulton Mar 7, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding this comment, @bchen1116 brings up the good point that since get_pipeline returns untrained pipelines, None is a valid return value since it will be "replaced" once the pipeline is trained.

My concern with that reasoning is that it only holds if users train the pipeline with automl.train_pipelines. In fact, we decided to carry over the threshold in get_pipeline because a user was confused as to why the score from automl.best_pipeline was different from the score they got when they called fit and score themselves. #2844

It now seems clear to me that the underlying issue here is that AutoMLSearch is tuning the threshold without the user being aware of it. This makes it hard to recreate the training procedure that AutoMLSearch did once the pipelines are exported.

I will approve this PR and I will file an issue to track the problem around users not being aware of the threshold tuning. @bchen1116 Can you write a unit test (if it's not already present) that automl.best_pipeline.score will produce the score as automl.train_pipelines followed by score ? Something like this maybe

from evalml.demos import load_breast_cancer
from evalml.automl import AutoMLSearch
from evalml.preprocessing import split_data

X, y = load_breast_cancer()
X_train, X_valid, y_train, y_valid = split_data(X, y, "binary")

automl = AutoMLSearch(X_train, y_train, "binary", max_batches=4, ensembling=True, verbose=True,
                      automl_algorithm="default")
automl.search()

best_pipeline_score = automl.best_pipeline.score(X_valid, y_valid, objectives=["F1"])

pl = automl.get_pipeline(1)
pl = automl.train_pipelines([pl])[pl.name]
manual_score = pl.score(X_valid, y_valid, objectives=["F1"])

assert best_pipeline_score == manual_score

return new_pipeline

def describe_pipeline(self, pipeline_id, return_dict=False):
Expand Down
30 changes: 28 additions & 2 deletions evalml/tests/automl_tests/test_automl_search_classification.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -479,7 +479,7 @@ def test_non_optimizable_threshold(AutoMLTestEnv, X_y_binary):
)


def test_optimize_threshold_maintained(AutoMLTestEnv, X_y_binary):
def test_optimize_threshold_get_pipeline_reset(AutoMLTestEnv, X_y_binary):
X, y = X_y_binary
automl = AutoMLSearch(
X_train=X,
Expand All @@ -502,7 +502,7 @@ def test_optimize_threshold_maintained(AutoMLTestEnv, X_y_binary):

best_pipeline_id = automl.rankings["id"][0]
best_get = automl.get_pipeline(best_pipeline_id)
assert best_get.threshold == 0.8
assert best_get.threshold is None


def test_describe_pipeline_objective_ordered(X_y_binary, caplog):
Expand Down Expand Up @@ -1270,3 +1270,29 @@ def test_automl_passes_allow_long_running_models(
estimators.extend(["CatBoost Classifier", "XGBoost Classifier"])

assert "Dropping estimators {}".format(", ".join(sorted(estimators))) in caplog.text


def test_automl_threshold_score(fraud_100):
X, y = fraud_100
X_train, X_valid, y_train, y_valid = split_data(X, y, "binary")

automl = AutoMLSearch(
X_train,
y_train,
"binary",
max_batches=4,
ensembling=True,
verbose=False,
automl_algorithm="default",
train_best_pipeline=True,
)
automl.search()

bp = automl.best_pipeline
best_pipeline_score = bp.score(X_valid, y_valid, objectives=["F1"])
bp_ranking_id = automl.rankings.iloc[0].id
pl = automl.get_pipeline(bp_ranking_id)
pl = automl.train_pipelines([pl])[pl.name]
manual_score = pl.score(X_valid, y_valid, objectives=["F1"])
assert bp.name == pl.name
assert best_pipeline_score == manual_score