Added support for Individual Conditional Expectation plots to `graph_partial_dependencies()` by christopherbunn · Pull Request #2386 · alteryx/evalml

christopherbunn · 2021-06-16T19:45:08Z

This implementation extends graph_partial_dependencies() by adding support for plotting Individual Conditional Expectations (ICE) behind a Partial Dependency plot. In addition, partial_dependence() has been updated to be able to generate the ICE data (as graph_partial_dependencies() uses this function to get the plot data).

Single Class:

Multiclass:

Multiclass ICE only:

By default, ICE plots are not shown when partial dependency plots are created. Setting kind='individual' or kind='both' will show the ICE plot.

Caveats:

Does not work for creating two-way partial dependency plots as there is no clean way to display ICE plots over a Contour plot. partial_dependence() is able to calculate the data behind the two-way partial dependency plot and return it as a list of DataFrames (with each DataFrame representing one sample). For now, a ValueError is thrown when an ICE plot is attempted for a two-way partial dependency.

Once the implementation clears, I can update the documentation example to show an ICE plot.

Resolves #2025

CLAassistant · 2021-06-16T19:45:14Z

All committers have signed the CLA.

codecov · 2021-06-16T19:48:28Z

Codecov Report

Merging #2386 (c7932f9) into main (f5a17c4) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2386     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        283     283             
  Lines      25316   25478    +162     
=======================================
+ Hits       25216   25378    +162     
  Misses       100     100

Impacted Files	Coverage Δ
evalml/model_understanding/graphs.py	`100.0% <100.0%> (ø)`
...lml/tests/model_understanding_tests/test_graphs.py	`100.0% <100.0%> (ø)`
...del_understanding_tests/test_partial_dependence.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5a17c4...c7932f9. Read the comment docs.

christopherbunn

Some implementation details to consider when reviewing:

christopherbunn · 2021-06-16T21:17:19Z

evalml/model_understanding/graphs.py


 def partial_dependence(
-    pipeline, X, features, percentiles=(0.05, 0.95), grid_resolution=100
+    pipeline, X, features, percentiles=(0.05, 0.95), grid_resolution=100, kind="average"


We could potentially make kind="both" by default, but it would throw an error when doing a two-way PD. If we changed the behavior to be a silent change or a warning, I would be okay with doing this as a default

christopherbunn · 2021-06-16T21:18:54Z

evalml/model_understanding/graphs.py

                    y=part_dep["partial_dependence"],
                    name="Partial Dependence",
-                    line=dict(width=3),
+                    line=dict(width=3, color="rgb(99,110,250)"),


The RGB number corresponds to the default blue that I extracted from the default MacOS color picker. I don't think there's like an alias to this exact color in plotly, but I could be missing it from the docs

christopherbunn · 2021-06-16T21:19:28Z

evalml/tests/model_understanding_tests/test_graphs.py

    with pytest.warns(None) as graph_valid:
        graph_partial_dependence(clf, X, features=0, grid_resolution=20)
-        assert len(graph_valid) == 1
+        assert len(graph_valid) == 0


Is there a particular reason that graph_partial_dependence was set to 1 here while the other graphing functions were 0?

I think we should move the grid_resolution back to 5. That was part of @freddyaboulton 's unit test speed up MR! But sorry, don't know the answer to that question.

christopherbunn · 2021-06-16T21:21:03Z

evalml/model_understanding/graphs.py

+            _go.Scatter(
+                x=x,
+                y=ice_data[sample],
+                line=dict(width=0.5, color="gray"),


There's some leeway to adjust this but from tweaking it a bit this is the best color and thickness combo for the ICE plot lines.

christopherbunn · 2021-06-16T21:24:26Z

evalml/model_understanding/graphs.py

+        if kind == "individual" or kind == "both":
+            raise ValueError(
+                "Individual conditional expectation plot can only be created with a one-way partial dependence plot"
+            )


Currently, I'm leaving out the ICE plot when plotting a two-way PD. I think the current behavior of raising an error is the best move, although I've considered putting in a warning message in the log. Open to suggestions for this behavior 🙂

christopherbunn

Some more testing specific notes:

christopherbunn · 2021-06-18T16:26:24Z

evalml/tests/model_understanding_tests/test_partial_dependence.py

+            data = np.concatenate([fig_dict["data"][i]["y"][:window_length],
+                                   fig_dict["data"][i + len(X) + 1]["y"][:window_length],
+                                   fig_dict["data"][i + 2 * len(X) + 2]["y"][:window_length]])


For this particular test, I ended up doing some weird matrix math to wrangle the plotly data. For each sample, this snippet essentially combines the ICE data for each class to one array and compares it to the data for that particular sample in the partial_dependence() output data. The +1 and +2 accounts for the additional PD trace that is in each plot.

christopherbunn · 2021-06-18T16:26:47Z

evalml/tests/model_understanding_tests/test_partial_dependence.py

+            data = np.concatenate([fig_dict["data"][i]["y"][:window_length],
+                                   fig_dict["data"][i + len(X)]["y"][:window_length],
+                                   fig_dict["data"][i + 2 * len(X)]["y"][:window_length]])


Same as the other comment, but the +1 and +2 aren't here since we're only plotting the ICE data.

christopherbunn · 2021-06-18T16:27:42Z

evalml/tests/model_understanding_tests/test_partial_dependence.py

        assert len(data["x"]) == 5
        assert len(data["y"]) == 5
-        assert data["name"] == label
+        assert data["name"] == "Partial Dependence: " + label


Ended up changing the PD trace name to mention that it is the partial dependence line to differentiate it from the ICE plots.

chukarsten

Awesome, great work. I need to do some more due dilligence as I can't run this at the Pentagon and visualize it, but the code and tests look good. I would just say let's stick with the lower grid resolutions, if possible, to try and reduce partial dependence run time. If it can all be tied to one variable, that would be better but I can do that in another PR. Also, I think shying away from the "if not" logic would be a bit more readable. Really great work though for your first week back!

evalml/model_understanding/graphs.py

chukarsten · 2021-06-19T18:50:28Z

evalml/tests/model_understanding_tests/test_graphs.py

    with pytest.warns(None) as graph_valid:
        graph_partial_dependence(clf, X, features=0, grid_resolution=20)
-        assert len(graph_valid) == 1
+        assert len(graph_valid) == 0


I think we should move the grid_resolution back to 5. That was part of @freddyaboulton 's unit test speed up MR! But sorry, don't know the answer to that question.

evalml/tests/model_understanding_tests/test_partial_dependence.py

chukarsten

Thanks for the changes, friendo!

christopherbunn force-pushed the 2025_ice_plots branch from 62f693c to 8632015 Compare June 16, 2021 19:45

christopherbunn commented Jun 16, 2021

View reviewed changes

christopherbunn force-pushed the 2025_ice_plots branch from c959aff to 582fa79 Compare June 18, 2021 13:57

christopherbunn commented Jun 18, 2021

View reviewed changes

christopherbunn force-pushed the 2025_ice_plots branch 2 times, most recently from cc15700 to a662a6d Compare June 18, 2021 17:49

christopherbunn marked this pull request as ready for review June 18, 2021 17:49

auto-assign bot assigned christopherbunn Jun 18, 2021

christopherbunn requested review from ParthivNaresh, angela97lin and freddyaboulton June 18, 2021 17:50

christopherbunn added enhancement An improvement to an existing feature. and removed enhancement An improvement to an existing feature. labels Jun 18, 2021

christopherbunn requested a review from chukarsten June 18, 2021 19:05

chukarsten suggested changes Jun 20, 2021

View reviewed changes

christopherbunn force-pushed the 2025_ice_plots branch 2 times, most recently from 2c1b70a to 490af18 Compare June 22, 2021 01:16

christopherbunn requested a review from chukarsten June 22, 2021 14:08

chukarsten approved these changes Jun 22, 2021

View reviewed changes

christopherbunn added 9 commits June 23, 2021 11:02

First pass of adding support for ICE plots

4687cbf

Lint and existing test fixes

aefbeb5

Added release notes and black lint fixes

19b15c1

Added tests and fixed graph to drop plotting of feature values

91f33d3

Fixed release notes

c2012c1

Fixed bug with multiclass pipelines and added test cases

5e20f05

Fixed the multiclass test case

d113268

Fixed two-way multiclass for partial_dependence

932d634

Cleaned up graph code and sped up model_understanding tests

cd1351b

christopherbunn force-pushed the 2025_ice_plots branch from 490af18 to cd1351b Compare June 23, 2021 15:03

christopherbunn and others added 4 commits June 23, 2021 11:17

Updated to use local datasets

8256afb

Black lint fixes

91c3940

Fixed local breast cancer ds typo

7949a97

Merge branch 'main' into 2025_ice_plots

c7932f9

christopherbunn merged commit 1f285fc into main Jun 24, 2021

christopherbunn deleted the 2025_ice_plots branch June 24, 2021 14:13

dsherry mentioned this pull request Jul 2, 2021

Release v0.28.0 #2467

Merged

Conversation

christopherbunn commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

christopherbunn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christopherbunn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

christopherbunn commented Jun 16, 2021 •

edited

Loading

CLAassistant commented Jun 16, 2021 •

edited

Loading

codecov bot commented Jun 16, 2021 •

edited

Loading