Training and scoring pipelines with AutoMLSearch #1913

freddyaboulton · 2021-03-02T16:21:01Z

Pull Request Description

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

codecov · 2021-03-03T15:20:00Z

Codecov Report

Merging #1913 (3437792) into main (22e158e) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1913     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         265      265             
  Lines       21737    21936    +199     
=========================================
+ Hits        21731    21930    +199     
  Misses          6        6

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`100.0% <100.0%> (ø)`
evalml/automl/engine/engine_base.py	`100.0% <100.0%> (ø)`
evalml/automl/engine/sequential_engine.py	`100.0% <100.0%> (ø)`
evalml/pipelines/pipeline_base.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl.py	`100.0% <100.0%> (ø)`
.../automl_tests/test_automl_search_classification.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_engine_base.py	`100.0% <100.0%> (ø)`
...valml/tests/automl_tests/test_sequential_engine.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 22e158e...3437792. Read the comment docs.

freddyaboulton · 2021-03-03T15:27:23Z

evalml/automl/automl_search.py

-                                                                                          random_seed=self.random_seed)
-                self._best_pipeline.fit(X_train, y_train)
-                tune_binary_threshold(self._best_pipeline, self.objective, self.problem_type, X_threshold_tuning, y_threshold_tuning)
+                best_pipeline = self._engine.train_pipeline(best_pipeline, self.X_train, self.y_train,


This line will change once #1814 is merged. We'll have to subset to the ensemble indices before calling train_pipeline

bchen1116

Left some questions and other tests that I think would be good to add! Definitely would be nice to discuss the implications of passing None to train and score.

bchen1116 · 2021-03-03T22:17:31Z

evalml/automl/engine/engine_base.py

+            objective (ObjectiveBase): Objective used in threshold tuning.
+
+        Returns:
+            PipelineBase - trained pipeline.


nitpick: can we do

""" Returns: pipeline (PipelineBase): Trained pipeline. """

Or something similar?

bchen1116 · 2021-03-03T22:35:21Z

evalml/automl/engine/sequential_engine.py

+            y (ww.DataTable, pd.DataFrame): Data to score on.
+            objectives (list(ObjectiveBase), list(str)): Objectives to score on.
+        Returns:
+            Dict: Dict containining scores for all objectives for all pipelines. Keyed by pipeline name.


typo: containing

bchen1116 · 2021-03-03T22:39:32Z

evalml/automl/utils.py

+        ValueError if any pipeline names are duplicated.
+    """
+    seen_names = set([])
+    duplicate_names = set([])


is there a reason to do set([]) versus set()? Just curious, I've only ever done set()

I believe set() is better hehe I'll make the change.

bchen1116 · 2021-03-03T22:41:09Z

evalml/tests/automl_tests/test_automl.py

+
+@patch('evalml.pipelines.BinaryClassificationPipeline.score')
+@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
+def test_train_batch_score_batch(mock_fit, mock_score, dummy_binary_pipeline_class, X_y_binary):


What happens if we try to train or score before AutoMLSearch has been run? Or if we try to score before we trained a pipeline batch? can we add test cov?

Also, what happens if the user passes in a stacked ensemble pipeline (in both cases where ensembling runs and ensembling doesn't run)?

Great questions @bchen1116 ! train_batch and score_batch are adjacent to search and mainly help users leverage the underlying engine for fitting and scoring. In my mind it makes the most sense to run this after search with pipelines you get via get_pipeline but I don't think we need to enforce that.

So to answer your questions:

Should work as expected

If you score a pipeline that hasn't been fit the exception will be in the log and the score will be nan for all objectives.

Should work as expected, i.e. the pipeline is either fit/scored successfully.

I'll add coverage 😄

Thanks for the clarification, sounds good to me!

evalml/tests/automl_tests/test_engine_base.py

bchen1116 · 2021-03-03T22:45:28Z

evalml/automl/utils.py

+            seen_names.add(pipeline.name)
+
+    if duplicate_names:
+        raise ValueError(f"All pipeline names must be unique. The names {', '.join(duplicate_names)} were repeated.")


Nit pick, but can we add quotation marks around each name? Maybe something like

duplicates = "', '".join(duplicate_names) f"All pipeline names must be unique. The names {duplicates} were repeated."

just for easier readability

bchen1116

@freddyaboulton thanks for the fixes! Everything looks great! I just left a comment on adding quotation marks around the name for clarity, since it seems like my original suggestion didn't work, but looks good otherwise!

bchen1116 · 2021-03-04T18:25:49Z

evalml/tests/automl_tests/test_engine_base.py

+    class Pipeline2(dummy_binary_pipeline_class):
+        custom_name = "My Pipeline"
+
+    with pytest.raises(ValueError, match="All pipeline names must be unique. The names My Pipeline were repeated."):


Oh sad, did the quotation marks not make it in. Can we add quotes to the pipeline names? Might need to do \" or something

bchen1116 · 2021-03-04T18:32:42Z

evalml/tests/automl_tests/test_sequential_engine.py

+                assert exception in caplog.text
+
+    # Test training before search is run
+    train_batch_and_check()


jeremyliweishih · 2021-03-05T16:07:12Z

docs/source/release_notes.rst

@@ -10,6 +10,8 @@ Release Notes
        * Added utility method to create list of components from a list of ``DataCheckAction`` :pr:`1907`
        * Updated ``validate`` method to include a ``action`` key in returned dictionary for all ``DataCheck``and ``DataChecks`` :pr:`1916`
        * Aggregating the shap values for predictions that we know the provenance of, e.g. OHE, text, and date-time. :pr:`1901`
+        * Added ``score_batch`` and ``train_batch`` methods to ``AutoMLSearch`` :pr:`1913`


think this should be train_pipelines and score_pipelines

You're totally right 🙈

chukarsten

I like this PR and I think it's really going to start to push some solid refactoring of the Engines and AutoMLSearch. I left a few comments, I think the only blocking ones are:

Resolution of placement of EngineBase.tune_threshold() static method.
Some docstring stuff.
Pipeline cloning discussion and resolution.

evalml/automl/engine/engine_base.py

chukarsten · 2021-03-05T16:37:28Z

evalml/automl/engine/engine_base.py

+        """
+        X_threshold_tuning = None
+        y_threshold_tuning = None
+        if EngineBase.tune_threshold(pipeline, optimize_thresholds, objective):


Yea, I dunno, tune_threshold seems like it might better belong in the utils from whence tune_binary_threshold came. What do you think?

evalml/automl/engine/engine_base.py

evalml/automl/utils.py

evalml/tests/automl_tests/test_automl.py

evalml/tests/automl_tests/test_engine_base.py

evalml/tests/automl_tests/test_sequential_engine.py

… before fitting.

…ned pipelines.

chukarsten

Looks good!

freddyaboulton mentioned this pull request Mar 2, 2021

Spike: New Engine Design #1914

Closed

freddyaboulton force-pushed the 1729-score-fit-pipelines-on-holdout-with-automl branch from a083332 to 435330a Compare March 3, 2021 15:11

freddyaboulton commented Mar 3, 2021

View reviewed changes

freddyaboulton force-pushed the 1729-score-fit-pipelines-on-holdout-with-automl branch from 3d0e31d to b49aee4 Compare March 3, 2021 19:26

freddyaboulton marked this pull request as ready for review March 3, 2021 20:04

freddyaboulton requested a review from dsherry March 3, 2021 20:04

auto-assign bot assigned freddyaboulton Mar 3, 2021

freddyaboulton requested review from bchen1116, chukarsten, angela97lin, jeremyliweishih and ParthivNaresh March 3, 2021 20:04

freddyaboulton mentioned this pull request Mar 3, 2021

Add Balanced Classification Data Splitter #1875

Merged

freddyaboulton force-pushed the 1729-score-fit-pipelines-on-holdout-with-automl branch from b49aee4 to 18bb3ef Compare March 3, 2021 22:59

bchen1116 requested changes Mar 3, 2021

View reviewed changes

freddyaboulton force-pushed the 1729-score-fit-pipelines-on-holdout-with-automl branch from 231d4dd to 6fe721c Compare March 4, 2021 16:30

freddyaboulton requested a review from bchen1116 March 4, 2021 18:18

bchen1116 approved these changes Mar 4, 2021

View reviewed changes

freddyaboulton force-pushed the 1729-score-fit-pipelines-on-holdout-with-automl branch from e96ce50 to 3501ac5 Compare March 4, 2021 19:07

freddyaboulton mentioned this pull request Mar 5, 2021

Dont allow duplicate pipeline names in AutoMLSearch #1932

Merged

jeremyliweishih reviewed Mar 5, 2021

View reviewed changes

chukarsten suggested changes Mar 5, 2021

View reviewed changes

freddyaboulton force-pushed the 1729-score-fit-pipelines-on-holdout-with-automl branch from 4464df1 to b71837c Compare March 8, 2021 16:29

freddyaboulton requested a review from chukarsten March 8, 2021 17:30

freddyaboulton added 5 commits March 8, 2021 15:40

Adding unit tests for new engine methods.

9aaf88d

Adding unit test checking that all the pipeline names were unique.

4d17929

Training best pipeline with engine.

ec8593d

Adding test for train_batch and score_batch in AutoMLSearch.

7b07d80

Adding PR 1913 to release notes.

1b2d2c1

freddyaboulton added 11 commits March 8, 2021 15:40

Fixing coverage in test_engine_base.

b700b93

Importing split_data in automl_search.py

f01fd1b

Adding coverage for training/scoring before and after search. Scoring…

c0fd3bf

… before fitting.

Adding single quotes around pipeline names.

493a70d

Fixing typo in release notes.

f7b4ff0

Only cloning once. Add test to make sure train_pipelines returns trai…

d8626f6

…ned pipelines.

Adding can_tune_threshold automl util.

7ac3c7e

Linting automl utils.

e8968ea

Updating error message in test_engines_check_pipeline_names_unique

bb1f209

Linting engine base.

31dffc6

Making can_tune_threshold_with_objective a pipeline method.

3437792

freddyaboulton force-pushed the 1729-score-fit-pipelines-on-holdout-with-automl branch from 9511aa5 to 3437792 Compare March 8, 2021 20:42

chukarsten approved these changes Mar 8, 2021

View reviewed changes

freddyaboulton merged commit c4306f6 into main Mar 8, 2021

freddyaboulton deleted the 1729-score-fit-pipelines-on-holdout-with-automl branch March 8, 2021 22:07

dsherry mentioned this pull request Mar 11, 2021

v0.20.0 #1961

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Training and scoring pipelines with AutoMLSearch #1913

Training and scoring pipelines with AutoMLSearch #1913

freddyaboulton commented Mar 2, 2021 •

edited

Loading

codecov bot commented Mar 3, 2021 •

edited

Loading

freddyaboulton Mar 3, 2021 •

edited

Loading

bchen1116 left a comment

bchen1116 Mar 3, 2021

bchen1116 Mar 3, 2021

bchen1116 Mar 3, 2021

freddyaboulton Mar 3, 2021

bchen1116 Mar 3, 2021

bchen1116 Mar 3, 2021

freddyaboulton Mar 3, 2021 •

edited

Loading

bchen1116 Mar 3, 2021

bchen1116 Mar 3, 2021

freddyaboulton Mar 3, 2021

bchen1116 left a comment

bchen1116 Mar 4, 2021

freddyaboulton Mar 5, 2021

bchen1116 Mar 4, 2021

jeremyliweishih Mar 5, 2021

freddyaboulton Mar 5, 2021

chukarsten left a comment

chukarsten Mar 5, 2021

chukarsten left a comment

Training and scoring pipelines with AutoMLSearch #1913

Training and scoring pipelines with AutoMLSearch #1913

Conversation

freddyaboulton commented Mar 2, 2021 • edited Loading

Pull Request Description

codecov bot commented Mar 3, 2021 • edited Loading

Codecov Report

freddyaboulton Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

bchen1116 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bchen1116 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chukarsten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chukarsten left a comment

Choose a reason for hiding this comment

freddyaboulton commented Mar 2, 2021 •

edited

Loading

codecov bot commented Mar 3, 2021 •

edited

Loading

freddyaboulton Mar 3, 2021 •

edited

Loading

freddyaboulton Mar 3, 2021 •

edited

Loading