Add tutorial for cost-benefit matrix #1088
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1088 +/- ##
=======================================
Coverage 99.91% 99.91%
=======================================
Files 191 192 +1
Lines 10701 10719 +18
=======================================
+ Hits 10692 10710 +18
Misses 9 9
Continue to review full report at Codecov.
|
angela97lin
requested review from
dsherry,
freddyaboulton,
jeremyliweishih,
kmax12,
eccabay and
bchen1116
and removed request for
dsherry and
freddyaboulton
There was a problem hiding this comment.
@angela97lin Great example! I left some minor suggestions. The only ones that I think would be nice to fix before merge are the formatting ones.
| "source": [ | ||
| "## Cost Benefit Matrix\n", | ||
| "\n", | ||
| "Although the confusion matrix is an incredibly useful visual for understanding our model, each prediction that is correctly or incorrectly classified is treated equally. For example, for detecting breast cancer, the confusion matrix does not take into consideration that it could be much more costly to incorrectly classify a malignant tumor as benign than it is to incorrectly classify a benign tumor as malignant. This is where the cost-benefit matrix shines: it uses the cost of each of the four possible outcomes to weigh each outcome differently. By scoring using the cost-benefit matrix, we can measure the score of the model by a concrete unit that is more closely related to the goal of the model. In the below example, we will show how the cost-benefit matrix objective can be used, and how it can give us better real-world impact when compared to using other standard machine learning objectives." |
There was a problem hiding this comment.
Might be nice to stick with churn instead of bringing up breast cancer?
There was a problem hiding this comment.
Hmmm yeah, I had mentioned breast cancer since it's probably easier to understand even without explaining the dataset and why something might be more costly. :o
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "In this example, let's say that correctly identifying customers who will churn (true positive case) will give us a net profit of $\\$$400, because it allows us to intervene, incentivize the customer to stay, and sign a new contract. Incorrectly classifying customers who were not going to churn as customers who will churn (false positive case) will cost $\\$$100 to represent the marketing and effort used to try to retain the user. Not identifying customers who will churn (false negative case) will cost us $\\$$200 to represent the lost in revenue from losing a customer. Finally, correctly identifying customers who will not churn (true negative case) will not cost us anything ($\\$$0), as nothing needs to be done for that customer." |
There was a problem hiding this comment.
The // is making the formatting weird.
There was a problem hiding this comment.
Argh this is so frustrating since it works differently locally when I run the notebooks. Thanks for pointing this out though, will fix!
There was a problem hiding this comment.
Yep, I'm still seeing that too.
There was a problem hiding this comment.
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now, if we calculate the cost-benefit matrix score on our best pipeline, we see that with this pipeline optimized for our cost-benefit matrix objective, we are able to generate $\\$$302400--much more than the profit returned from our previous pipeline! This is just one example of how using the cost-benefit matrix can help find pipelines that can perform better on real-world problems, rather than on arbitrary standard statistical metrics." |
There was a problem hiding this comment.
Not-blocking: Also in search the best pipeline score is 54300.000 but on the entire dataset it's 302400? Big gap no?
There was a problem hiding this comment.
@angela97lin looks great! I like the layout. This reads really well.
One suggestion would be to move the confusion matrix section into the Model Understanding section (or just delete it), and then link to it. It would help keep the guide short too.
Let's get that one weird formatting error fixed before we merge. Lmk if you get blocked! Must be some issue with the currency symbol, right?
| @@ -0,0 +1,217 @@ | |||
| { | |||
There was a problem hiding this comment.
@angela97lin the cost benefit matrix is cumulative right? like if you scored a holdout set of 10 examples vs 1000 examples you’d expect different ranges of possible score scores?
because of this, im wondering if the objective should be normalized by the number of examples scored. that way the size of the set doesn't affect the objectives score
| "outputs": [], | ||
| "source": [ | ||
| "from evalml.objectives import CostBenefitMatrix\n", | ||
| "cost_benefit_matrix = CostBenefitMatrix(true_positive_cost=400, true_negative_cost=0,\n", |
There was a problem hiding this comment.
sorry, as this was probably cover in the PR where you merged CostBenefitMatrix, but the names of these arguments are confusing in this context. what if we just dropped the _cost off each argument? i.etrue_positive
There was a problem hiding this comment.
Aha interesting, I had changed them from true_positive to true_positive_cost because someone (I think @dsherry) suggested that would be less confusing! I'm indifferent to either as long as the docstrings are clear!
There was a problem hiding this comment.
@kmax12 @angela97lin yep that was me haha. I felt adding "_cost" was more clear because just saying true_negative" was less specific. I'm fine either way as long as our docs are complete.
There was a problem hiding this comment.
Okay, I was thinking again and also feel like @freddyaboulton had mentioned 'cost' being confusing so I'm going to update to remove _cost for now!
Closes #1027, see docs here: https://evalml.alteryx.com/en/1027_cbm_tutorial/demos/cost_benefit_matrix.html