-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added demo for prediction intervals #3954
Conversation
Codecov Report
@@ Coverage Diff @@
## main #3954 +/- ##
=====================================
Coverage 99.7% 99.7%
=====================================
Files 347 347
Lines 36790 36790
=====================================
Hits 36670 36670
Misses 120 120 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
bccdbd1
to
054dda3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Just a couple small notes.
}, | ||
"vscode": { | ||
"interpreter": { | ||
"hash": "fb5a3afe2d0dd7ad0fd9e1ff89de4e0e95804490c629f36065bf8d930a66d311" | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should really have an automation or at least a check to strip these out.
"source": [ | ||
"While predictions that are generated by EvalML pipelines aim to be accurate as possible, it is very rarely the case that future results are the exact same values as predicted. Prediction intervals can help to contextualize a prediction by showing the range a future prediction is expected to fall within a certain likelihood. \n", | ||
"\n", | ||
"Given a set of predictions and a **trained** EvalML pipeline, the prediction intervals for this set of predictions is generated by calling `get_prediction_intervals()` on the pipeline's estimator:" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an important callout, but could you rephrase it slightly to mention that we don't just need the trained pipeline, we specifically need the transformed, ready for prediction features?
I'm also very out of the loop on prediction intervals, why do we need the transformed features?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to clarify that only a fitted estimator is needed and that the example is using the fitted estimator in a trained pipeline.
We need the transformed features as we're calling get_prediction_intervals()
from the estimator directly. It expects to take input in that has already been through the preprocessing steps of the pipeline. We currently don't have a way to get prediction intervals at the pipeline level.
86ed01c
to
536ad7c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great! Thanks @christopherbunn
536ad7c
to
cbff41a
Compare
Also includes a demo for forecasting that plots the prediction intervals.