Plot for time series regression by freddyaboulton · Pull Request #1483 · alteryx/evalml

freddyaboulton · 2020-12-01T19:49:46Z

Pull Request Description

Example with a year's worth of (synthetic) data:

Example with a month's worth of data:

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

codecov · 2020-12-01T20:06:16Z

Codecov Report

Merging #1483 (de127f0) into main (e3acef6) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1483     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         223      223             
  Lines       15100    15139     +39     
=========================================
+ Hits        15093    15132     +39     
  Misses          7        7

Impacted Files	Coverage Δ
evalml/model_understanding/__init__.py	`100.0% <ø> (ø)`
evalml/model_understanding/graphs.py	`100.0% <100.0%> (ø)`
...lml/tests/model_understanding_tests/test_graphs.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3acef6...de127f0. Read the comment docs.

jeremyliweishih

Sweet looks good to me. Only extra thing I would mention would that we should use this in the docs when we get to it.

angela97lin

Sweet, the graphs look amazing! Just left a tiny comment about the axis name :d

angela97lin · 2020-12-01T21:15:48Z

evalml/model_understanding/graphs.py

+    # Let plotly pick the best date format.
+    layout = _go.Layout(title={'text': "Prediction vs Target over time"},
+                        xaxis={'title': 'Time'},
+                        yaxis={'title': 'Target Values'})


Really nit-picky: could we name the y-axis something else since we're graphing both target and prediction values?

Hehe good point. I'll change it to "Target Values and Predictions"!

freddyaboulton · 2020-12-01T22:07:33Z

Good point @jeremyliweishih ! I updated #1384 to mention showing how to use the model understanding module for time series.

dsherry

@freddyaboulton I left a couple suggestions and some thoughts, but once those are resolved LGTM!

dsherry · 2020-12-02T01:17:15Z

evalml/model_understanding/__init__.py

    graph_partial_dependence,
-    graph_prediction_vs_actual
+    graph_prediction_vs_actual,
+    graph_prediction_vs_target_over_time


@freddyaboulton do you like "graph_prediction_vs_actual" or "graph_prediction_vs_target" more? We should use the same name for both of these.

I don't feel strongly about it, but I am more used to "prediction vs actual"

Now that I've seen predicted vs actual, I agree that that sounds much better :o

Sounds good! I preferred using "target" in the function name since that's what we mention in the docstrings and axis labels but I'll change the name to actual for consistency.

dsherry · 2020-12-02T01:19:18Z

evalml/model_understanding/graphs.py

    return _go.Figure(layout=layout, data=data)
+
+
+def graph_prediction_vs_target_over_time(pipeline, X, y, dates):


I commented about this on the issue, but to reiterate: its fine to leave this as-is and expect dates as a 4th arg, but heads up that @angela97lin is in the process of standardizing all the util/graph methods to use DataTables, in which case X.time_index should get you the dates.

dsherry · 2020-12-02T01:21:23Z

evalml/model_understanding/graphs.py

+        y (pd.Series): Target values to compare predictions against.
+
+    Returns:
+        plotly.Figure showing the prediction vs actual over time.


@freddyaboulton could we separate this into a data generation method and a graphing method? That way callers with a different UI can use the data generation method, and people who want plotly figs can use this method.

For this particular plot the data generated is fairly simple, just three vectors: predictions, actual and dates

Yep I thought about doing that at first but thought it wasn't worth it since it was a one-liner. I'll push this up now!

dsherry · 2020-12-02T01:21:37Z

evalml/tests/model_understanding_tests/test_graphs.py

+    class NotTSPipeline:
+        problem_type = ProblemTypes.REGRESSION
+
+    error_msg = "graph_prediction_vs_target_over_time only supports time series regression pipelines! Received regression."


…anding.

…sion pipelines

…not installed.

…init__

…n_vs_actual_over_time_data

freddyaboulton changed the title ~~Adding graph_prediction_vs_target_over_time function to model_underst…~~ Plot for time series regression Dec 1, 2020

freddyaboulton marked this pull request as ready for review December 1, 2020 20:48

freddyaboulton requested review from ParthivNaresh, angela97lin, bchen1116, christopherbunn, dsherry, eccabay and jeremyliweishih December 1, 2020 20:49

jeremyliweishih approved these changes Dec 1, 2020

View reviewed changes

angela97lin approved these changes Dec 1, 2020

View reviewed changes

freddyaboulton force-pushed the 1258-plot-for-ts-regression branch from 6c120e8 to c2d1b05 Compare December 1, 2020 22:03

dsherry approved these changes Dec 2, 2020

View reviewed changes

freddyaboulton force-pushed the 1258-plot-for-ts-regression branch from c2d1b05 to fd2732c Compare December 2, 2020 15:53

freddyaboulton added 9 commits December 2, 2020 12:00

Adding graph_prediction_vs_target_over_time function to model_underst…

5793258

…anding.

Adding PR 1483 to release notes.

81b0457

graph_prediction_vs_target_over_time only supports time series regres…

b9472c0

…sion pipelines

Skip test_graph_prediction_vs_target_over_time_value_error if plotly …

5c0d62f

…not installed.

Adding graph_prediction_vs_target_over_time to api ref.

0d042b3

Adding graph_prediction_vs_target_over_time to model_understanding __…

455eb3f

…init__

Changing y-axis label in graph_prediction_vs_target_over_time.

ab7311b

Using actual instead of target in function name. Adding get_predictio…

43308ea

…n_vs_actual_over_time_data

Adding PR 1483 to future releases section of the release notes.

6164e95

freddyaboulton force-pushed the 1258-plot-for-ts-regression branch from af55d46 to 6164e95 Compare December 2, 2020 17:02

Changing typo in error_msg for graph_prediction_vs_actual_over_time

de127f0

freddyaboulton merged commit 3d69967 into main Dec 2, 2020

freddyaboulton deleted the 1258-plot-for-ts-regression branch December 2, 2020 18:01

freddyaboulton mentioned this pull request Dec 2, 2020

Fix custom index bug in get_prediction_vs_actual_over_time_data #1494

Merged

dsherry mentioned this pull request Dec 29, 2020

Release v0.17.0 #1623

Merged

ParthivNaresh mentioned this pull request Jul 31, 2021

Add a model understanding function for visualizing target variable over time for ts regression #2509

Closed

		return _go.Figure(layout=layout, data=data)


		def graph_prediction_vs_target_over_time(pipeline, X, y, dates):

Conversation

freddyaboulton commented Dec 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Uh oh!

codecov bot commented Dec 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jeremyliweishih left a comment

Choose a reason for hiding this comment

Uh oh!

angela97lin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton commented Dec 1, 2020

Uh oh!

dsherry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

freddyaboulton commented Dec 1, 2020 •

edited

Loading

codecov bot commented Dec 1, 2020 •

edited

Loading