Add ROC/confusion graphing methods #720

dsherry · 2020-04-25T15:43:15Z

Fixes #697

Adds back in the capability to generate ROC plots and confusion matrices (removed in #615), now as standalone methods rather than relying on automl to compute the plot data during CV.

Changes

Update roc_curve to return a dict
Define graph_roc_curve which takes predicted vs actual and makes a graph
Update confusion_matrix to optionally call normalize_confusion_matrix internally
Define graph_confusion_matrix which takes predicted vs actual and makes a graph
Basic unit test coverage, at the level of what we had before, could still be improved for the plotting!

What's currently missing

Docs example of how to use this, will add!

Open questions

Should we say graph or plot here? We use plot in SearchIterationPlot. But we also moved the feature importance chart to graph_feature_importance, and this aligns with that. My default is to keep that pattern.

codecov · 2020-04-25T15:50:07Z

Codecov Report

Merging #720 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #720      +/-   ##
==========================================
+ Coverage   99.34%   99.35%   +0.01%     
==========================================
  Files         148      148              
  Lines        5175     5299     +124     
==========================================
+ Hits         5141     5265     +124     
  Misses         34       34

Impacted Files	Coverage Δ
evalml/pipelines/__init__.py	`100.00% <100.00%> (ø)`
evalml/pipelines/graph_utils.py	`100.00% <100.00%> (ø)`
evalml/tests/pipeline_tests/test_graph_utils.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7357bcb...8177ea5. Read the comment docs.

dsherry · 2020-04-25T15:50:49Z

evalml/tests/pipeline_tests/test_plot_utils.py

+    assert np.array_equal(conf_mat_expected, conf_mat)
+    conf_mat = confusion_matrix(y_true, y_predicted, normalize_method='pred')
+    conf_mat_expected = np.array([[2 / 3.0, np.nan, 0], [0, np.nan, 1 / 3.0], [1 / 3.0, np.nan, 2 / 3.0]])
+    assert np.allclose(conf_mat_expected, conf_mat, equal_nan=True)


np.allclose is cool! Setting equal_nan=True means it handles nans. I'm liking this more than array_almost_equal. I think they've deprecated that in recent versions actually, should check.

jeremyliweishih

Looks good to me - just some clarifying questions.

jeremyliweishih · 2020-04-25T16:35:26Z

evalml/pipelines/plot_utils.py

+    Arguments:
+        y_true (pd.Series or np.array): true binary labels.
+        y_pred (pd.Series or np.array): predictions from a binary classifier.
+        normalize_method ({'true', 'pred', 'all'}): Normalization method. Supported options are: 'true' to normalize by row, 'pred' to normalize by column, or 'all' to normalize by all values. Defaults to 'true'.


Curious about your thoughts on 'true', 'pred' and 'all'. It seems great if we're following the sklearn API but it always seemed confusing to me. IMO axis would be more clear.

I'd argue that using 'true' / 'pred' is more helpful because users don't have to figure out which axis corresponds to what (x-axis == true? x-axis == predicted values?); they get direct access to what they want to normalize.

Unless I'm misunderstanding what you mean by axis 😅

hmm when you put it like that it makes sense 😄

jeremyliweishih · 2020-04-25T16:37:23Z

evalml/pipelines/plot_utils.py

-    return sklearn_roc_curve(y_true, y_pred_proba)
+    fpr_rates, tpr_rates, thresholds = sklearn_roc_curve(y_true, y_pred_proba)
+    auc_score = sklearn_auc(fpr_rates, tpr_rates)
+    return {'fpr_rates': fpr_rates,


I like this change - much more clear for users!

jeremyliweishih · 2020-04-25T16:39:11Z

evalml/pipelines/plot_utils.py

+                            name='ROC (w/ AUC {:06f})'.format(roc_curve_data['auc_score']),
+                            line=dict(width=3)))
+    data.append(_go.Scatter(x=[0, 1], y=[0, 1],
+                            name='Random Chance',


Little nitpick but I like Random Guess over Random Chance 😄

Thanks for the suggestion! I actually prefer "Trivial Model" or something like that -- lmk what you think

Thats good as well!

angela97lin · 2020-04-26T03:55:47Z

evalml/pipelines/plot_utils.py

+    labels = conf_mat.columns
+    reversed_labels = labels[::-1]
+
+    title = 'Confusion matrix {}{}'.format(


Nit-picking but there's an extra space between "Confusion matrix" and the rest of the title!

Thanks wil fix

angela97lin · 2020-04-26T03:57:50Z

evalml/pipelines/plot_utils.py

+        '' if normalize_method is None else (' normalized using method "' + normalize_method + '"'))
+    z_data, custom_data = (conf_mat, conf_mat_normalized) if normalize_method is None else (conf_mat_normalized, conf_mat)
+    primary_heading, secondary_heading = ('Raw', 'Normalized') if normalize_method is None else ('Normalized', 'Raw')
+    hover_text = '<br><b>' + primary_heading + ' Count</b>: %{z}<br><b>' + secondary_heading + ' Count</b>: %{customdata:.3f} <br>'


Also nit-picking but it'd be nice if "Raw Count" was always an integer?

Sure, good idea!

angela97lin · 2020-04-26T03:58:57Z

evalml/pipelines/plot_utils.py

    return conf_mat
+
+
+def graph_confusion_matrix(y_true, y_pred, normalize_method='true', title_addition=None):


title_addition is a cool addition to both graphing methods! Would it be possible to add a test for them?

Sure! Good point :)

angela97lin

LGTM, thanks for adding this back in! I just had a few suggestions :)

dsherry · 2020-04-27T15:28:59Z

@angela97lin @jeremyliweishih thanks for the great comments! And I apologize for disturbing your weekends--this was work I did Friday before signing off, but in the future I'll wait until Monday 😂

I've addressed all the comments. Outstanding:

I'd like to move plot_utils.py to graph_utils.py. We can chat about this at the team meeting
I need to add an example to the docs. The "search results" page is where we had these plots before. I did run into a potential issue with confusion matrix while doing that though so I have to figure that out.

IMO, this shouldn't block the release; it's ok if this doesn't get into v0.9.0.

…tra title space

dsherry · 2020-05-08T14:21:16Z

I've addressed the TODOs above. This is ready to go!

dsherry · 2020-05-08T14:24:01Z

evalml/pipelines/graph_utils.py

+                                      colorscale='Blues'),
+                     layout=layout)
+    # plotly Heatmap y axis defaults to the reverse of what we want: https://community.plotly.com/t/heatmap-y-axis-is-reversed-by-default-going-against-standard-convention-for-matrices/32180
+    fig.update_yaxes(autorange="reversed")


@angela97lin : idk if you remember, but a couple weeks ago I mentioned I was having trouble getting the confusion matrix to plot out in the right order. Turns out that's because the y axis on plotly.Heatmap is the reverse of the input data by default! As you can see in the link I posted in the code comment above, they did that because in the field of image processing, images are typically stored in matrices with the y axis inverted.

Long story short, this inversion fixes the problem without the need for us to invert the labels or data :) lmk if you spot anything funky with this code.

dsherry requested review from angela97lin and jeremyliweishih April 25, 2020 15:43

dsherry changed the title ~~Ds 697 add back plot methods~~ Add ROC/confusion graphing methods Apr 25, 2020

auto-assign bot assigned dsherry Apr 25, 2020

dsherry commented Apr 25, 2020

View reviewed changes

jeremyliweishih approved these changes Apr 25, 2020

View reviewed changes

angela97lin reviewed Apr 26, 2020

View reviewed changes

dsherry force-pushed the ds_697_add_back_plot_methods branch from 914b8c9 to f533a81 Compare April 27, 2020 14:28

dsherry added 14 commits May 7, 2020 23:01

Impl

b39104d

Got existing test coverage passing

314a216

Added coverage of graph methods

9e9e955

Lint

a93841f

Changelog

1363fca

Changed a word

43ad1bd

Simplify numeric formatting in confusion matrix hover text. Delete ex…

78fe54f

…tra title space

Add test coverage for plot data

ce1e222

Display dotted line in ROC as 'trivial model'

8940182

Moved plot_utils.py to graph_utils.py

8242a26

Update docstring

37b002b

Reverse y axis order in confusion matrix

26d614c

Update unit tests with axis order fix

2b230ab

Update plotting example

8514d07

dsherry force-pushed the ds_697_add_back_plot_methods branch from f533a81 to 8514d07 Compare May 8, 2020 14:19

dsherry commented May 8, 2020

View reviewed changes

dsherry added 5 commits May 8, 2020 10:43

Add to api docs

160335d

Add new methods to module import

dd90b0d

Update changelog

ec9ee7e

Missed a comma

fc597aa

Docs fix

8177ea5

dsherry merged commit 6724d10 into master May 8, 2020

dsherry deleted the ds_697_add_back_plot_methods branch May 8, 2020 16:01

angela97lin mentioned this pull request May 29, 2020

Release v0.10.0 May 29, 2020 #822

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ROC/confusion graphing methods #720

Add ROC/confusion graphing methods #720

dsherry commented Apr 25, 2020 •

edited

Loading

codecov bot commented Apr 25, 2020 •

edited

Loading

dsherry Apr 25, 2020

jeremyliweishih left a comment

jeremyliweishih Apr 25, 2020

angela97lin Apr 26, 2020

angela97lin Apr 26, 2020

jeremyliweishih Apr 26, 2020

jeremyliweishih Apr 25, 2020

jeremyliweishih Apr 25, 2020

dsherry Apr 27, 2020

jeremyliweishih Apr 27, 2020

angela97lin Apr 26, 2020

dsherry Apr 27, 2020

angela97lin Apr 26, 2020

dsherry Apr 27, 2020

angela97lin Apr 26, 2020

dsherry Apr 27, 2020

angela97lin left a comment

dsherry commented Apr 27, 2020

dsherry commented May 8, 2020

dsherry May 8, 2020

		return conf_mat


		def graph_confusion_matrix(y_true, y_pred, normalize_method='true', title_addition=None):

Add ROC/confusion graphing methods #720

Add ROC/confusion graphing methods #720

Conversation

dsherry commented Apr 25, 2020 • edited Loading

codecov bot commented Apr 25, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin left a comment

Choose a reason for hiding this comment

dsherry commented Apr 27, 2020

dsherry commented May 8, 2020

Choose a reason for hiding this comment

dsherry commented Apr 25, 2020 •

edited

Loading

codecov bot commented Apr 25, 2020 •

edited

Loading