Update cost-benefit tutorial to use a holdout/test set #1159
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #1159 +/- ##
==========================================
+ Coverage 91.52% 99.93% +8.41%
==========================================
Files 210 210
Lines 13247 13247
==========================================
+ Hits 12124 13239 +1115
+ Misses 1123 8 -1115
Continue to review full report at Codecov.
|
angela97lin
commented
Sep 11, 2020
| @@ -58,7 +58,7 @@ def split_data(X, y, regression=False, test_size=.2, random_state=None): | |||
| if regression: | |||
| CV_method = ShuffleSplit(n_splits=1, | |||
| test_size=test_size, | |||
| random_state=0) | |||
There was a problem hiding this comment.
Unrelated but I think small enough change!
angela97lin
requested review from
freddyaboulton,
dsherry,
bchen1116,
eccabay,
jeremyliweishih and
christopherbunn
and removed request for
freddyaboulton
bchen1116
reviewed
Sep 16, 2020
bchen1116
reviewed
Sep 16, 2020
bchen1116
approved these changes
Sep 16, 2020
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1123 by updating tutorial to use a holdout set. This caused changes in performance for both pipelines being compared, so I updated the results and provided analysis of the updated results (esp confusion matrix which is less black-and-white in terms of performance now).
Unfortunately, I was not able to provide consistent numbers for the scores, even after setting
random_state. Hence, I updated the docs to calculate the the profit difference dynamically instead of writing it in text. (Did a brief look into embedding the calculated value in markdown but it looks like it would require another package)Updated docs here: https://evalml.alteryx.com/en/1123_holdout/demos/cost_benefit_matrix.html