Stored `predict_proba` results in `.x` for intermediate estimators in ComponentGraph by christopherbunn · Pull Request #2629 · alteryx/evalml

christopherbunn · 2021-08-13T21:39:48Z

As part of the component graph changes necessary to support building our new ensembler (#1930), this PR stores the prediction probabilities (when available) for non-final estimators in a component graph. If predict_proba is not available or if the final estimator in a component graph is being evaluated, then predict is used.

codecov · 2021-08-13T21:45:06Z

Codecov Report

Merging #2629 (b849cc6) into main (0485d7c) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2629     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        298     298             
  Lines      27024   27086     +62     
=======================================
+ Hits       26980   27042     +62     
  Misses        44      44

Impacted Files	Coverage Δ
evalml/pipelines/component_graph.py	`99.7% <100.0%> (+0.1%)`	⬆️
...valml/tests/pipeline_tests/test_component_graph.py	`100.0% <100.0%> (ø)`
evalml/tests/pipeline_tests/test_pipelines.py	`99.9% <100.0%> (+0.1%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0485d7c...b849cc6. Read the comment docs.

angela97lin · 2021-08-16T22:32:23Z

evalml/tests/pipeline_tests/test_component_graph.py

    mock_fit_transform.return_value = mock_X_t
    mock_fit.return_value = Estimator
-    mock_predict.return_value = pd.Series(y)
+    mock_predict_proba.return_value = pd.Series(y)


Should we make this a DataFrame instead to mimic and make sure that when we have an output with more than one column?

(oops, pressed add single comment too quickly!)

angela97lin

LGTM! 😁

I had a suggestion: we should convert our mock_predict_proba to a Dataframe to closer mimic real output. I commented directly on some but not all the places 😂

I'm curious if we have tests for calling _compute_features for a RegressionPipeline (via fit/predict/compute_features/or other public methods) and for MulticlassClassificationPipeline (since predict_proba would not drop a column).

Are there tests that already exist (and hence that test not erroring on this PR is confirmation) for multiclass/regression? Otherwise, it could be good to write some :)

evalml/tests/pipeline_tests/test_pipelines.py

angela97lin · 2021-08-16T22:45:32Z

evalml/tests/pipeline_tests/test_component_graph.py

    X, y = X_y_binary
    mock_fit_transform.return_value = pd.DataFrame(X)
-    mock_predict.return_value = pd.Series(y)
+    mock_predict_proba.return_value = pd.Series(y)


Same comment as below, let's change this to a DataFrame?

angela97lin · 2021-08-16T22:47:19Z

evalml/tests/pipeline_tests/test_component_graph.py

    mock_ohe.return_value = pd.DataFrame(X)
-    mock_en_predict.return_value = pd.Series(np.ones(X.shape[0]))
-    mock_rf_predict.return_value = pd.Series(np.zeros(X.shape[0]))
+    mock_en_predict_proba.return_value = pd.Series(np.ones(X.shape[0]))


Same comment, let's return dfs!

chukarsten

I think this looks good and addressing Angela's concerns and perhaps changing the naming scheme might be a good move.

chukarsten · 2021-08-17T15:12:42Z

evalml/tests/pipeline_tests/test_component_graph.py

    assert input_feature_names["Logistic Regression"] == [
-        "Random Forest.x",
-        "Elastic Net.x",
+        "1_Random Forest.x",


Do we perhaps have any other thoughts for options to name this? The prepending with the underscore is kinda weird, especially with the space after it. I think an appen....sion? might look better and adding like a "col" or something to it might be a bit clearer.

I chose to do column_name_component scheme since to my knowledge we aren't using underscores anywhere else for the pipeline name. However, in general I don't really have any strong preferences for what the naming convention should be and I'm definitely open to ideas. Your idea of something like [Col 0 Random Forest.x, Col 1 Random Forest.x, etc.] makes sense to me and I'm good to move forward with this if there aren't any other suggestions.

christopherbunn · 2021-08-17T22:10:25Z

LGTM! 😁

I had a suggestion: we should convert our mock_predict_proba to a Dataframe to closer mimic real output. I commented directly on some but not all the places 😂

Thanks! I covered most of these test cases, lmk if I missed some but we should be using dfs as the output for intermediate estimators in tests now.

I'm curious if we have tests for calling _compute_features for a RegressionPipeline (via fit/predict/compute_features/or other public methods) and for MulticlassClassificationPipeline (since predict_proba would not drop a column).

Are there tests that already exist (and hence that test not erroring on this PR is confirmation) for multiclass/regression? Otherwise, it could be good to write some :)

For RegressionPipeline, there's some coverage with tests that use the nonlinear_regression_pipeline_class fixture (like test_score_nonlinear_regression) since they use the example_regression_graph that calculates predict_proba for intermediate estimators. This the same with MulticlassClassificationPipeline, nonlinear_multiclass_pipeline_class, and test_score_nonlinear_multiclass. I think the score() test case covers the fit/predict test case for these pipeline types, but lmk if you think it would be worth having a more explicit case for these.

angela97lin

LGTM, thanks for making the changes!

RE the output as dataframes, it'd be nice if they had multiple cols like the output of predict_proba usually would, but not a blocking change 😁

angela97lin · 2021-08-18T16:37:45Z

evalml/tests/pipeline_tests/test_component_graph.py

-def test_predict(mock_predict, mock_fit, example_graph, X_y_binary):
+def test_predict(mock_predict, mock_predict_proba, mock_fit, example_graph, X_y_binary):
    X, y = X_y_binary
+    mock_predict_proba.return_value = pd.DataFrame(y)


Bahaha RE comment to change output to dataframes, I'm curious about when it's a df and has multiple columns! but not a big change :)

Oops I see 😅. I added the two test cases with different predict_proba outputs to address this.

christopherbunn changed the title ~~Store predict_proba results in .x for intermediate estimators in ComponentGraph~~ Stored predict_proba results in .x for intermediate estimators in ComponentGraph Aug 13, 2021

christopherbunn force-pushed the predict_proba_component_graph branch from 84dc8e3 to fbfe8c7 Compare August 13, 2021 21:45

christopherbunn marked this pull request as ready for review August 16, 2021 15:03

auto-assign bot assigned christopherbunn Aug 16, 2021

christopherbunn requested review from a team and angela97lin August 16, 2021 15:03

angela97lin reviewed Aug 16, 2021

View reviewed changes

angela97lin suggested changes Aug 16, 2021

View reviewed changes

chukarsten approved these changes Aug 17, 2021

View reviewed changes

christopherbunn force-pushed the predict_proba_component_graph branch from abd3947 to ca82cff Compare August 17, 2021 22:26

christopherbunn requested review from angela97lin and chukarsten August 17, 2021 22:47

angela97lin approved these changes Aug 18, 2021

View reviewed changes

christopherbunn force-pushed the predict_proba_component_graph branch from f19d66e to 65a54f0 Compare August 18, 2021 19:33

Changed behavior of intermediate estimators to calculate predict_proba

c788fd8

christopherbunn force-pushed the predict_proba_component_graph branch from 65a54f0 to da45cfa Compare August 18, 2021 19:34

christopherbunn added 4 commits August 18, 2021 16:44

Updated naming conventions and tests

1a82faf

Updated tests to use dfs for predict_probas

befde65

Updated naming scheme

23d188d

Added multiclass and regression test case

b849cc6

christopherbunn force-pushed the predict_proba_component_graph branch from da45cfa to b849cc6 Compare August 18, 2021 20:44

christopherbunn merged commit 7231a38 into main Aug 18, 2021

freddyaboulton deleted the predict_proba_component_graph branch May 13, 2022 15:02

Conversation

christopherbunn commented Aug 13, 2021

Uh oh!

codecov bot commented Aug 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

angela97lin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christopherbunn commented Aug 17, 2021

Uh oh!

angela97lin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Aug 13, 2021 •

edited

Loading