Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tutorial for cost-benefit matrix #1088

Merged
merged 17 commits into from Aug 25, 2020
Merged

Add tutorial for cost-benefit matrix #1088

merged 17 commits into from Aug 25, 2020

Conversation

angela97lin
Copy link
Contributor

@angela97lin angela97lin commented Aug 20, 2020

@angela97lin angela97lin added this to the August 2020 milestone Aug 20, 2020
@angela97lin angela97lin self-assigned this Aug 20, 2020
@codecov
Copy link

codecov bot commented Aug 20, 2020

Codecov Report

Merging #1088 into main will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1088   +/-   ##
=======================================
  Coverage   99.91%   99.91%           
=======================================
  Files         191      192    +1     
  Lines       10701    10719   +18     
=======================================
+ Hits        10692    10710   +18     
  Misses          9        9           
Impacted Files Coverage Δ
evalml/demos/__init__.py 100.00% <100.00%> (ø)
evalml/demos/churn.py 100.00% <100.00%> (ø)
evalml/objectives/cost_benefit_matrix.py 100.00% <100.00%> (ø)
evalml/tests/demo_tests/test_datasets.py 100.00% <100.00%> (ø)
...lml/tests/model_understanding_tests/test_graphs.py 100.00% <100.00%> (ø)
.../tests/objective_tests/test_cost_benefit_matrix.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4870c83...e9de104. Read the comment docs.

@angela97lin angela97lin marked this pull request as ready for review Aug 21, 2020
Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

@angela97lin Great example! I left some minor suggestions. The only ones that I think would be nice to fix before merge are the formatting ones.

"source": [
"## Cost Benefit Matrix\n",
"\n",
"Although the confusion matrix is an incredibly useful visual for understanding our model, each prediction that is correctly or incorrectly classified is treated equally. For example, for detecting breast cancer, the confusion matrix does not take into consideration that it could be much more costly to incorrectly classify a malignant tumor as benign than it is to incorrectly classify a benign tumor as malignant. This is where the cost-benefit matrix shines: it uses the cost of each of the four possible outcomes to weigh each outcome differently. By scoring using the cost-benefit matrix, we can measure the score of the model by a concrete unit that is more closely related to the goal of the model. In the below example, we will show how the cost-benefit matrix objective can be used, and how it can give us better real-world impact when compared to using other standard machine learning objectives."
Copy link
Contributor

@freddyaboulton freddyaboulton Aug 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be nice to stick with churn instead of bringing up breast cancer?

Copy link
Contributor Author

@angela97lin angela97lin Aug 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm yeah, I had mentioned breast cancer since it's probably easier to understand even without explaining the dataset and why something might be more costly. :o

"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, let's say that correctly identifying customers who will churn (true positive case) will give us a net profit of $\\$$400, because it allows us to intervene, incentivize the customer to stay, and sign a new contract. Incorrectly classifying customers who were not going to churn as customers who will churn (false positive case) will cost $\\$$100 to represent the marketing and effort used to try to retain the user. Not identifying customers who will churn (false negative case) will cost us $\\$$200 to represent the lost in revenue from losing a customer. Finally, correctly identifying customers who will not churn (true negative case) will not cost us anything ($\\$$0), as nothing needs to be done for that customer."
Copy link
Contributor

@freddyaboulton freddyaboulton Aug 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The // is making the formatting weird.

image

Copy link
Contributor Author

@angela97lin angela97lin Aug 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Argh this is so frustrating since it works differently locally when I run the notebooks. Thanks for pointing this out though, will fix!

Copy link
Collaborator

@dsherry dsherry Aug 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I'm still seeing that too.

Copy link
Contributor Author

@angela97lin angela97lin Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs/source/demos/cost_benefit_matrix.ipynb Outdated Show resolved Hide resolved
docs/source/demos/cost_benefit_matrix.ipynb Outdated Show resolved Hide resolved
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, if we calculate the cost-benefit matrix score on our best pipeline, we see that with this pipeline optimized for our cost-benefit matrix objective, we are able to generate $\\$$302400--much more than the profit returned from our previous pipeline! This is just one example of how using the cost-benefit matrix can help find pipelines that can perform better on real-world problems, rather than on arbitrary standard statistical metrics."
Copy link
Contributor

@freddyaboulton freddyaboulton Aug 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not-blocking: Also in search the best pipeline score is 54300.000 but on the entire dataset it's 302400? Big gap no?

Copy link
Contributor Author

@angela97lin angela97lin Aug 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahaha I think this was brought up in a comment by @kmax12 about how the score currently scales based on the number of samples. Will fix in #1098.

docs/source/demos/cost_benefit_matrix.ipynb Show resolved Hide resolved
Copy link
Collaborator

@dsherry dsherry left a comment

@angela97lin looks great! I like the layout. This reads really well.

One suggestion would be to move the confusion matrix section into the Model Understanding section (or just delete it), and then link to it. It would help keep the guide short too.

Let's get that one weird formatting error fixed before we merge. Lmk if you get blocked! Must be some issue with the currency symbol, right?

@@ -0,0 +1,217 @@
{
Copy link
Contributor

@kmax12 kmax12 Aug 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@angela97lin the cost benefit matrix is cumulative right? like if you scored a holdout set of 10 examples vs 1000 examples you’d expect different ranges of possible score scores?

because of this, im wondering if the objective should be normalized by the number of examples scored. that way the size of the set doesn't affect the objectives score

@angela97lin
Copy link
Contributor Author

angela97lin commented Aug 24, 2020

@kmax12 Ooo you're right; the way it's currently implemented is affected by the size of the set. Agreed that this should be fixed, will add a new issue to fix this! (#1098)

"outputs": [],
"source": [
"from evalml.objectives import CostBenefitMatrix\n",
"cost_benefit_matrix = CostBenefitMatrix(true_positive_cost=400, true_negative_cost=0,\n",
Copy link
Contributor

@kmax12 kmax12 Aug 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, as this was probably cover in the PR where you merged CostBenefitMatrix, but the names of these arguments are confusing in this context. what if we just dropped the _cost off each argument? i.etrue_positive

Copy link
Contributor Author

@angela97lin angela97lin Aug 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha interesting, I had changed them from true_positive to true_positive_cost because someone (I think @dsherry) suggested that would be less confusing! I'm indifferent to either as long as the docstrings are clear!

Copy link
Collaborator

@dsherry dsherry Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kmax12 @angela97lin yep that was me haha. I felt adding "_cost" was more clear because just saying true_negative" was less specific. I'm fine either way as long as our docs are complete.

Copy link
Contributor Author

@angela97lin angela97lin Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I was thinking again and also feel like @freddyaboulton had mentioned 'cost' being confusing so I'm going to update to remove _cost for now!

@angela97lin angela97lin merged commit 554848c into main Aug 25, 2020
@angela97lin angela97lin deleted the 1027_cbm_tutorial branch Aug 25, 2020
@dsherry dsherry mentioned this pull request Aug 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Write a tutorial on how to use Cost-Benefit Matrix objective
4 participants