Repository navigation
Fix Fraud Cost - #2450
Fix Fraud Cost#2450
Conversation
Codecov Report
@@ Coverage Diff @@
## main #2450 +/- ##
=======================================
- Coverage 99.7% 99.7% -0.0%
=======================================
Files 283 283
Lines 25568 25566 -2
=======================================
- Hits 25466 25464 -2
Misses 102 102
Continue to review full report at Codecov.
|
|
|
||
| Arguments: | ||
| ypred_proba (pd.Series): Predicted probablities | ||
| threshold (float): Dollar threshold to determine if transaction is fraud |
There was a problem hiding this comment.
This threshold parameter didn't really make sense to me. For our fraud prediction models, we are predicting whether or not a transaction is fraud, so the predicted_proba output should be a probability on whether the transaction was fraud or not. Multiplying this probability by the amount spent shouldn't be helpful in determine if there was fraud or not, so I don't think we should threshold based on some spent amount.
I removed this so that we use the probabilities like our other objectives. Let me know if I'm misinterpreting!
There was a problem hiding this comment.
I guess the idea was to label as fraud the transactions whose expected loss due to fraud was larger than some threshold.
Maybe this would be helpful in prioritizing larger transactions?
I'm just thinking out loud. I think this is ok from my point of view.
There was a problem hiding this comment.
Yeah, that seemed reasonable. This also definitely didn't work well with the Bounded method, since we bound search from [0, 1], but it also didn't work well with Brent and Golden methods since those resulting thresholds generally stayed <2, and oftentimes went negative!
It makes sense to prioritize larger transactions, but that should hopefully be something the model can find rather than relying on the objective, imo
|
|
||
| loss = false_negatives.sum() + false_positives.sum() | ||
| # add a penalty if we output naive predictions | ||
| all_one_prediction_cost = (2 - len(set(y_predicted))) * fraud_cost.sum() |
There was a problem hiding this comment.
Add a penalty to naive predictions. Without this penalty, we could incur small fraud costs if we predict every transaction as fraudulent.
There was a problem hiding this comment.
And this "small fraud cost" is currently not captured by the objective, right? I guess I'm wondering why this isn't currently captured by how the objective is computed.
There was a problem hiding this comment.
It is currently captured, but that would only be the cost coming from false_positive cases, which small, especially when interchange_cost is small. I wanted to add a big penalization to prevent us from choosing a naive estimator.
| is declined. Between 0 and 1. Defaults to .5 | ||
|
|
||
| interchange_fee (float): How much of each successful transaction you can collect. | ||
| interchange_fee (float): How much of each successful transaction you pay. |
There was a problem hiding this comment.
interchange fees are fees that a merchant pays when a customer uses their card
freddyaboulton
left a comment
There was a problem hiding this comment.
@bchen1116 I think this looks good! I have a question on the penalty you're adding.
|
|
||
| optimal = minimize_scalar(cost, bounds=(0, 1), method="Bounded") | ||
| optimal = minimize_scalar( | ||
| cost, bracket=(0, 1), method="Golden", options={"maxiter": 250} |
There was a problem hiding this comment.
Apologies if you've shared this with us before, but do you have plots/data for the claim - "I did this because I found that Brent and Bounded methods didn't perform well with the optimization process." ? If you shared it before, it definitely got lost in my inbox 🙈 I'm just curious what the process was to arrive at this conclusion!
There was a problem hiding this comment.
No worries, I was referencing the old perf test I ran when I switched from Golden to Bounded. I saw the perforamcne was the same, but the time took a little longer with these new minimization methods. I assumed the reverse would hold true here
There was a problem hiding this comment.
As a general rule of thumb - I would say optimization methods and their performance on different objective functions and datasets may vary depending on the characteristics of the dataset and luck as well (there's generally some randomness). Might be a good idea to run performance tests just to make sure but I don't think performance would be greatly impacted.
There was a problem hiding this comment.
Might be cool to play around with maxiter as well! My hunch is that 250 might be a little low but thats just a guess 😄
|
|
||
| Arguments: | ||
| ypred_proba (pd.Series): Predicted probablities | ||
| threshold (float): Dollar threshold to determine if transaction is fraud |
There was a problem hiding this comment.
I guess the idea was to label as fraud the transactions whose expected loss due to fraud was larger than some threshold.
Maybe this would be helpful in prioritizing larger transactions?
I'm just thinking out loud. I think this is ok from my point of view.
|
|
||
| loss = false_negatives.sum() + false_positives.sum() | ||
| # add a penalty if we output naive predictions | ||
| all_one_prediction_cost = (2 - len(set(y_predicted))) * fraud_cost.sum() |
There was a problem hiding this comment.
And this "small fraud cost" is currently not captured by the objective, right? I guess I'm wondering why this isn't currently captured by how the objective is computed.
|
@jeremyliweishih @freddyaboulton @chukarsten perf test has been updated using |
chukarsten
left a comment
There was a problem hiding this comment.
Looks good to me! Nice work!
fix #2318
In this PR, I revert the minimization objective back to
Golden. I did this because I found thatBrentandBoundedmethods didn't perform well with the optimization process. WhileBrentandBoundedare preferred and suggested byscipydue to the speed increase they have over theGoldenmethod, I found that these two weren't able to find the threshold properly when I was testing fraud. As we can see from the previous perf test, these methods performed similarly here, althoughBoundeddid take longer to run compared toGolden.Fraud Doc:

Lead scoring doc build passes without triggering asserts!
Quick examples of thresholding here
Perf test here