Feature/sensitivity low alert

[skvorekn]

Sensitivity at low alert rates is an evaluation metric used in the final round of the Centers for Medicare & Medicaid Services AI Health Outcomes Challenge.

It is an important clinical accuracy metric because it gives insight into how resource reallocation will impact outcomes. For example, if we are predicting mortality of the Medicare population, a hospital may want to allocate more resources to people with the top, say, 1% of risk scores (predictions). This 1% is the alert rate. As a measure of accuracy, the hospital may want to know what percent of all deaths are captured by focusing on the top 1% (sensitivity).

The sensitivity metric is calculated by classifying the top 1% of predictions as the 'True' class (we predict they will die), and the remaining 99% are classified as 'False' (we predict they will not die). Sensitivity is measured using this alert rate (# predicted true/# actual true).

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

Merging #2001 (14e80ed) into main (20ddcd6) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #2001     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         278      280      +2     
  Lines       22770    22886    +116     
=========================================
+ Hits        22761    22877    +116     
  Misses          9        9             

Impacted Files	Coverage Δ
evalml/objectives/__init__.py	100.0% <100.0%> (ø)
evalml/objectives/sensitivity_low_alert.py	100.0% <100.0%> (ø)
...tive_tests/test_binary_classification_objective.py	100.0% <100.0%> (ø)
evalml/tests/objective_tests/test_objectives.py	100.0% <100.0%> (ø)
evalml/tests/objective_tests/test_sla.py	100.0% <100.0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20ddcd6...14e80ed. Read the comment docs.

[skvorekn]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ skvorekn
❌ Kelsey Skvoretz

Kelsey Skvoretz seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Please advise on how to resolve this. I only have one GitHub account, so not sure what is going on here! Thanks!

[dsherry]

Hi @skvorekn , thanks for contributing! Regarding your question about the CLA, it appears the email you've made your commits with doesn't match one of the emails in your GitHub account. Here's a guide from GitHub on how to set your email address on the GitHub site, including a section on how to set it on the command-line so git will use it.

My recommendation would be to set this to your GitHub account email and then squash your changes:

git config --global user.email "email@example.com"
# squash your branch down to one commit. instructions here: https://stackoverflow.com/a/5189600/841003
git fetch && git rebase -i origin/main
git push -f

If that doesn't work please let us know.

[chukarsten]

Solid. I like this PR and the testing framework. Thanks a lot for the contribution!

[skvorekn]

I misunderstood the comment about squashing changes, so did that out of order. Looks like things are passing now, though! Also made a few minor updates based on @chukarsten's comments. Thanks for your feedback - I really appreciate it! Let me know if there is anything else.

[dsherry]

This is wonderful! The code looks great. My only comment was about the class name.

[chukarsten]

@skvorekn I think you might need to resign our CLA to pass our CI check. You were this close 🤏 to making it into the last release . I also rebased your remote branch so you'll probably have to pull.

[skvorekn]

Done! Looks like the CLA needs a resign by @chukarsten as well!

[chukarsten]

@skvorekn merged! We'll release it next week with 0.22.0.