Dont allow duplicate pipeline names in AutoMLSearch #1932

freddyaboulton · 2021-03-04T21:51:31Z

Pull Request Description

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

codecov · 2021-03-04T22:04:26Z

Codecov Report

Merging #1932 (2d666f0) into main (5051e67) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1932     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         267      267             
  Lines       21700    21730     +30     
=========================================
+ Hits        21694    21724     +30     
  Misses          6        6

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`100.0% <100.0%> (ø)`
evalml/automl/utils.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5051e67...2d666f0. Read the comment docs.

freddyaboulton · 2021-03-05T15:30:03Z

evalml/automl/utils.py

@@ -85,3 +85,29 @@ def tune_binary_threshold(pipeline, objective, problem_type, X_threshold_tuning,
            y_predict_proba = pipeline.predict_proba(X_threshold_tuning)
            y_predict_proba = y_predict_proba.iloc[:, 1]
            pipeline.threshold = objective.optimize_threshold(y_predict_proba, y_threshold_tuning, X=X_threshold_tuning)
+
+
+def check_all_pipeline_names_unique(pipelines):


This is also in my #1913 PR

classic stacked branches. disappointed this was only one on another and not 5

angela97lin

LGTM! 🥳

If we want to be super sure maybe it'd be good to add test cases for when multiple (>3) pipelines have the same name to make sure its printed out once (Custom, Custom, Custom), or multiple pipelines with the same names (Custom, Custom, Custom1, Custom1) but probably not super necessary since at that point we're just testing set functionality 😂

angela97lin · 2021-03-05T15:50:23Z

evalml/tests/automl_tests/test_automl.py

+
+    with pytest.raises(ValueError,
+                       match="All pipeline names must be unique. The names 'Custom Pipeline' were repeated."):
+        AutoMLSearch(X, y, problem_type="binary", allowed_pipelines=[MyPipeline1, MyPipeline2, MyPipeline3])


Super duper nit-pick suggestion but I wonder if there's a way to phrase this so that it gramatically makes sense for the case with 1 or multiple duplicates :P

I should've known @angela97lin was driving the tense calculator

I had to heavily debate whether it was worth it or not to comment about this LOL 😅

jeremyliweishih

looks good but I like Angela's suggestion on fixing the grammar in the one duplicate case!

chukarsten

Proposed alternate impl for the check to reduce lines, but no need to accept. Great work.

chukarsten · 2021-03-05T19:54:35Z

evalml/automl/utils.py

@@ -85,3 +85,29 @@ def tune_binary_threshold(pipeline, objective, problem_type, X_threshold_tuning,
            y_predict_proba = pipeline.predict_proba(X_threshold_tuning)
            y_predict_proba = y_predict_proba.iloc[:, 1]
            pipeline.threshold = objective.optimize_threshold(y_predict_proba, y_threshold_tuning, X=X_threshold_tuning)
+
+
+def check_all_pipeline_names_unique(pipelines):


classic stacked branches. disappointed this was only one on another and not 5

chukarsten · 2021-03-05T19:55:12Z

evalml/automl/utils.py

+          None
+
+    Raises:
+        ValueError if any pipeline names are duplicated.


I think I mentioned in your other PR that normally I think we do: ValueError: if....

chukarsten · 2021-03-05T19:55:49Z

evalml/automl/utils.py

+            seen_names.add(pipeline.name)
+
+    if duplicate_names:
+        plural, tense = ("s", "were") if len(duplicate_names) > 1 else ("", "was")


I respect this dedication to verb tense calculation.

chukarsten · 2021-03-05T20:05:07Z

evalml/automl/utils.py

+    seen_names = set()
+    duplicate_names = set()
+
+    for pipeline in pipelines:
+        if pipeline.name in seen_names:
+            duplicate_names.add(pipeline.name)
+        else:
+            seen_names.add(pipeline.name)


I have twice proposed a pandas impl and twice deleted it thinking I was being overbearing. Here it is...not blocking, feel free to reject.

name_count = pd.Series([p.name for p in pipelines]).value_counts() duplicate_names = name_count[name_count > 1] # I don't think this line is quite right.

Ended up taking it! 👏

chukarsten · 2021-03-05T20:05:39Z

evalml/tests/automl_tests/test_automl.py

+
+    with pytest.raises(ValueError,
+                       match="All pipeline names must be unique. The names 'Custom Pipeline' were repeated."):
+        AutoMLSearch(X, y, problem_type="binary", allowed_pipelines=[MyPipeline1, MyPipeline2, MyPipeline3])


I should've known @angela97lin was driving the tense calculator

freddyaboulton added 2 commits March 4, 2021 15:18

Check if there are duplicate pipeline names in AutoMLSearch init.

e5a8873

Updating release notes for 1932.

bbe42d8

freddyaboulton changed the title ~~Check if there are duplicate pipeline names in AutoMLSearch init.~~ Dont allow duplicate pipeline names in AutoMLSearch Mar 4, 2021

freddyaboulton commented Mar 5, 2021

View reviewed changes

freddyaboulton marked this pull request as ready for review March 5, 2021 15:30

auto-assign bot assigned freddyaboulton Mar 5, 2021

freddyaboulton requested review from dsherry, bchen1116, angela97lin, chukarsten, jeremyliweishih and ParthivNaresh March 5, 2021 15:30

angela97lin approved these changes Mar 5, 2021

View reviewed changes

jeremyliweishih approved these changes Mar 5, 2021

View reviewed changes

freddyaboulton added 2 commits March 5, 2021 12:40

Grammatical correctness.

25fa3a2

Pipeline names are in alphabetical order.

28fe813

chukarsten approved these changes Mar 5, 2021

View reviewed changes

Using series in check_all_pipeline_names_unique.

2d666f0

freddyaboulton merged commit cfc601d into main Mar 5, 2021

freddyaboulton deleted the 1858-dont-allow-duplicate-pipeline-names-in-automl-search branch March 5, 2021 21:17

dsherry mentioned this pull request Mar 11, 2021

v0.20.0 #1961

Merged

Dont allow duplicate pipeline names in AutoMLSearch #1932

Dont allow duplicate pipeline names in AutoMLSearch #1932

freddyaboulton commented Mar 4, 2021

codecov bot commented Mar 4, 2021 •

edited

freddyaboulton Mar 5, 2021

chukarsten Mar 5, 2021

angela97lin left a comment

angela97lin Mar 5, 2021

freddyaboulton Mar 5, 2021

chukarsten Mar 5, 2021

angela97lin Mar 5, 2021

jeremyliweishih left a comment

chukarsten left a comment

chukarsten Mar 5, 2021

chukarsten Mar 5, 2021

freddyaboulton Mar 5, 2021

chukarsten Mar 5, 2021

chukarsten Mar 5, 2021

freddyaboulton Mar 5, 2021

chukarsten Mar 5, 2021

Dont allow duplicate pipeline names in AutoMLSearch #1932

Dont allow duplicate pipeline names in AutoMLSearch #1932

Conversation

freddyaboulton commented Mar 4, 2021

Pull Request Description

codecov bot commented Mar 4, 2021 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

chukarsten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 4, 2021 •

edited