Updated _evaluate_pipelines to consolidate side effects #1337

christopherbunn · 2020-10-22T14:55:10Z

Taking on a different approach with _evaluate side effects by restructuring and renaming to _evaluate_pipelines. See this comment for more info.
Resolves #1295

codecov · 2020-10-26T21:23:43Z

Codecov Report

Merging #1337 into main will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1337      +/-   ##
==========================================
- Coverage   99.95%   99.95%   -0.00%     
==========================================
  Files         213      213              
  Lines       13938    13934       -4     
==========================================
- Hits        13931    13927       -4     
  Misses          7        7

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`99.62% <100.00%> (ø)`
evalml/tests/automl_tests/test_automl.py	`100.00% <100.00%> (ø)`
evalml/utils/logger.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 86a39b0...88e0ee5. Read the comment docs.

freddyaboulton

@christopherbunn Looks great!

freddyaboulton · 2020-10-28T19:20:35Z

evalml/tests/automl_tests/test_automl.py

    mock_next_batch.side_effect = [[dummy_binary_pipeline_class(parameters={}), dummy_binary_pipeline_class(parameters={})]]
    automl = AutoMLSearch(problem_type='binary', allowed_pipelines=[dummy_binary_pipeline_class])
-    automl.search(X, y)
+    # Mock rankings so `best_pipeline` setting does not error out
+    with patch('evalml.automl.AutoMLSearch.rankings', new_callable=PropertyMock) as mock_rankings:


Why did you change mock_evaluate_pipelines from .side_effect to .return_value?

So we need this this new PropertyMock because you're mocking _evaluate_pipelines which is what calls _add_result (which is what populates the data for the ranking table)?

I wonder if we even need this code block in the test. I think the point of the test is to verify that the search ends when it encounters a batch of all nan (which is what happens in the code block that doesn't have the property mock).

This would be fine to merge as is but I'm guessing we can simplify this test a bit without losing any coverage.

Why did you change mock_evaluate_pipelines from .side_effect to .return_value?

Good question. I changed it to .return_value as I needed the mock _evaluate_pipelines to return the entire list of pipeline scores. Passing in a list to .side_effect cause it to iterate through the list and only return each element.

RE: the code block, you're right about the reason why we have to populate the ranking table manually. My impression of the intent of this section was to show that having a np.nan score as one of the pipeline results won't result in raising the AutoMLSearchException. If this isn't a necessary check, then I'm good with cutting out this section.

Maybe we can delete that code block then? The second code block returns a Nan in the second batch and the search doesn't terminate then but we can check that explicitly with assert mock_evaluate_pipelines.call_count == 3.

This isn't blocking merge but my vote is to remove code that provides redundant coverage while we're at it.

freddyaboulton · 2020-10-28T19:22:25Z

evalml/automl/automl_search.py

+                if add_single_pipeline:
+                    add_single_pipeline = False
+
+            except KeyboardInterrupt:


From what I can tell there isn't any change to the keyboard interrupt feature! What do you think @christopherbunn ?

I think technically if the user terminates the search while the next batch is being generated, it wouldn't get caught by this KeyboardInterrupt. In practice, getting the next batch takes so little time it's very unlikely that this will occur.

dsherry · 2020-10-30T21:01:40Z

evalml/automl/automl_search.py

-                    return True
-
+        scores = self._evaluate_pipelines(pipelines, X, y, baseline=True)
+        if scores == []:


return len(scores) == 0 ?

dsherry

@christopherbunn LGTM!

I left one question about show_batch_output. I also left a comment about deleting an old docstring. Otherwise, nothing blocking

dsherry · 2020-11-03T14:41:30Z

evalml/automl/automl_search.py

-        Returns:
-            self
+            feature_types (list, optional): list of feature types, either numerical or categorical.
+                Categorical features will automatically be encoded


@christopherbunn I think we deleted this deprecated feature_types field last week

dsherry · 2020-11-03T14:59:15Z

evalml/automl/automl_search.py

+
+            except KeyboardInterrupt:
+                current_pipeline_batch = self._handle_keyboard_interrupt(pipeline, current_pipeline_batch)
+                if current_pipeline_batch == []:


Style nit-pick: if len(current_pipeline_batch) == 0 I don't think there's much significant functional difference here lol, I just think checking length is more clear.

dsherry · 2020-11-03T15:00:15Z

evalml/automl/automl_search.py

@@ -425,6 +427,7 @@ def search(self, X, y, data_checks="auto", show_iteration_plot=True):
        if self.allowed_pipelines == []:
            raise ValueError("No allowed pipelines to search")
        if self.max_batches and self.max_iterations is None:
+            self.show_batch_output = True


@christopherbunn what's this? Why do we need it?

Currently, we show the batch number only if the user specifies a number of max_batches. Since we use batching internally even if only max_iterations is specified, there isn't really a clean way to infer whether or not we want to show the number of batches other than setting a variable at the beginning.

CLAassistant · 2020-11-03T16:16:57Z

All committers have signed the CLA.

dsherry · 2020-11-05T19:53:49Z

@christopherbunn and I saw intermittent failures in the linux CI tests on his branch. Since they were coming from pipeline tests and this PR only changes automl code, we conclude those failures aren't introduced by this PR. Will keep debugging.

This reverts commit 9451546.

* Revert "Updated _evaluate_pipelines to consolidate side effects (#1337)" This reverts commit 9451546. * Updated changelog

christopherbunn force-pushed the 1295_`_evaluate`_changes branch from 7f53df8 to 48b340f Compare October 23, 2020 20:14

christopherbunn changed the title ~~Draft: Alternative to 1295~~ Updated _evaluate_pipelines to consolidate side effects Oct 26, 2020

christopherbunn force-pushed the 1295_`_evaluate`_changes branch from ba2c499 to edf598b Compare October 26, 2020 21:37

christopherbunn marked this pull request as ready for review October 26, 2020 22:08

christopherbunn requested review from dsherry, freddyaboulton, angela97lin, bchen1116, eccabay and jeremyliweishih October 27, 2020 17:17

freddyaboulton approved these changes Oct 28, 2020

View reviewed changes

dsherry assigned christopherbunn Oct 28, 2020

christopherbunn force-pushed the 1295_`_evaluate`_changes branch 2 times, most recently from b2d304b to 3bf80f0 Compare October 30, 2020 15:10

dsherry reviewed Oct 30, 2020

View reviewed changes

christopherbunn force-pushed the 1295_`_evaluate`_changes branch from 3bf80f0 to 028fa66 Compare November 2, 2020 16:23

dsherry approved these changes Nov 3, 2020

View reviewed changes

christopherbunn force-pushed the 1295_`_evaluate`_changes branch from 028fa66 to f67ea11 Compare November 3, 2020 16:16

christopherbunn force-pushed the 1295_`_evaluate`_changes branch 3 times, most recently from 8abf084 to 41d8163 Compare November 4, 2020 16:53

christopherbunn added 7 commits November 5, 2020 12:17

Initial pass of _evaluate_pipelines

71fe54d

Added check for max_iterations in evaluate_pipelines

93dd870

Updated pipeline output

b23a723

Updated testing and added interrupt check for Baseline

495086f

Updated release notes and fixed lint errors

146fa00

Updated logger show batch output only when max_batches is set

da473cf

Removed redundant test coverage in test_pipelines_in_batch_return_nan

f71ca67

christopherbunn added 3 commits November 5, 2020 12:17

Removed accidental quotes in docstring

79eb92b

Cleaned up _add_baseline_logic

02df0db

Removed extra docstring

88e0ee5

christopherbunn force-pushed the 1295_`_evaluate`_changes branch from 41d8163 to 88e0ee5 Compare November 5, 2020 17:17

dsherry merged commit 9451546 into main Nov 5, 2020

dsherry mentioned this pull request Nov 5, 2020

Unit test flake: pipeline tests failing intermittently with timeout/error #1408

Closed

christopherbunn added a commit that referenced this pull request Nov 5, 2020

Revert "Updated _evaluate_pipelines to consolidate side effects (#1337)"

a0a2f85

This reverts commit 9451546.

christopherbunn mentioned this pull request Nov 5, 2020

Revert "Updated _evaluate_pipelines to consolidate side effects" #1409

Merged

christopherbunn added a commit that referenced this pull request Nov 5, 2020

Revert "Updated _evaluate_pipelines to consolidate side effects" (#1409)

9d43af7

* Revert "Updated _evaluate_pipelines to consolidate side effects (#1337)" This reverts commit 9451546. * Updated changelog

christopherbunn mentioned this pull request Nov 5, 2020

Updated _evaluate_pipelines to consolidate side effects #1410

Merged

dsherry mentioned this pull request Nov 24, 2020

Release v0.16.0 #1468

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated _evaluate_pipelines to consolidate side effects #1337

Updated _evaluate_pipelines to consolidate side effects #1337

christopherbunn commented Oct 22, 2020 •

edited

Loading

codecov bot commented Oct 26, 2020 •

edited

Loading

freddyaboulton left a comment

freddyaboulton Oct 28, 2020

christopherbunn Oct 28, 2020

freddyaboulton Oct 29, 2020 •

edited

Loading

freddyaboulton Oct 28, 2020

christopherbunn Oct 28, 2020

dsherry Oct 30, 2020 •

edited

Loading

dsherry left a comment

dsherry Nov 3, 2020

dsherry Nov 3, 2020

dsherry Nov 3, 2020

christopherbunn Nov 3, 2020

CLAassistant commented Nov 3, 2020 •

edited

Loading

dsherry commented Nov 5, 2020

Updated _evaluate_pipelines to consolidate side effects #1337

Updated _evaluate_pipelines to consolidate side effects #1337

Conversation

christopherbunn commented Oct 22, 2020 • edited Loading

codecov bot commented Oct 26, 2020 • edited Loading

Codecov Report

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Nov 3, 2020 • edited Loading

dsherry commented Nov 5, 2020

christopherbunn commented Oct 22, 2020 •

edited

Loading

codecov bot commented Oct 26, 2020 •

edited

Loading

freddyaboulton Oct 29, 2020 •

edited

Loading

dsherry Oct 30, 2020 •

edited

Loading

CLAassistant commented Nov 3, 2020 •

edited

Loading