Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added demo for prediction intervals #3954

Merged
merged 6 commits into from
Jan 27, 2023
Merged

Added demo for prediction intervals #3954

merged 6 commits into from
Jan 27, 2023

Conversation

christopherbunn
Copy link
Contributor

@christopherbunn christopherbunn commented Jan 24, 2023

Also includes a demo for forecasting that plots the prediction intervals.

@codecov
Copy link

codecov bot commented Jan 24, 2023

Codecov Report

Merging #3954 (cbff41a) into main (4346ccd) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##            main   #3954   +/-   ##
=====================================
  Coverage   99.7%   99.7%           
=====================================
  Files        347     347           
  Lines      36790   36790           
=====================================
  Hits       36670   36670           
  Misses       120     120           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Contributor

@eccabay eccabay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just a couple small notes.

Comment on lines 1111 to 1115
},
"vscode": {
"interpreter": {
"hash": "fb5a3afe2d0dd7ad0fd9e1ff89de4e0e95804490c629f36065bf8d930a66d311"
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should really have an automation or at least a check to strip these out.

"source": [
"While predictions that are generated by EvalML pipelines aim to be accurate as possible, it is very rarely the case that future results are the exact same values as predicted. Prediction intervals can help to contextualize a prediction by showing the range a future prediction is expected to fall within a certain likelihood. \n",
"\n",
"Given a set of predictions and a **trained** EvalML pipeline, the prediction intervals for this set of predictions is generated by calling `get_prediction_intervals()` on the pipeline's estimator:"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an important callout, but could you rephrase it slightly to mention that we don't just need the trained pipeline, we specifically need the transformed, ready for prediction features?

I'm also very out of the loop on prediction intervals, why do we need the transformed features?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to clarify that only a fitted estimator is needed and that the example is using the fitted estimator in a trained pipeline.

We need the transformed features as we're calling get_prediction_intervals() from the estimator directly. It expects to take input in that has already been through the preprocessing steps of the pipeline. We currently don't have a way to get prediction intervals at the pipeline level.

@christopherbunn christopherbunn force-pushed the TML-5875_PI_demo branch 2 times, most recently from 86ed01c to 536ad7c Compare January 26, 2023 18:33
Copy link
Collaborator

@jeremyliweishih jeremyliweishih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Thanks @christopherbunn

@christopherbunn christopherbunn enabled auto-merge (squash) January 27, 2023 16:47
@christopherbunn christopherbunn merged commit da10108 into main Jan 27, 2023
@christopherbunn christopherbunn deleted the TML-5875_PI_demo branch January 27, 2023 17:19
@chukarsten chukarsten mentioned this pull request Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants