-
Notifications
You must be signed in to change notification settings - Fork 89
Updating percent better than baseline computation #1809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1809 +/- ##
=========================================
+ Coverage 100.0% 100.0% +0.1%
=========================================
Files 252 252
Lines 20052 20117 +65
=========================================
+ Hits 20044 20109 +65
Misses 8 8
Continue to review full report at Codecov.
|
fe22f68
to
7f4febe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
@@ -14,6 +14,8 @@ class CostBenefitMatrix(BinaryClassificationObjective): | |||
greater_is_better = True | |||
score_needs_proba = False | |||
perfect_score = np.inf | |||
# Range (-Inf, Inf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit-pick but I thought this was an accidental comment rather than on purpose. Maybe we can move it to be next to is_bounded_like_percentage
rather than being above?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't this go next to perfect_score
?
evalml/objectives/lead_scoring.py
Outdated
@@ -9,6 +9,8 @@ class LeadScoring(BinaryClassificationObjective): | |||
greater_is_better = True | |||
score_needs_proba = False | |||
perfect_score = math.inf | |||
# Range (-Inf, Inf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same comment
@@ -243,6 +263,8 @@ class LogLossBinary(BinaryClassificationObjective): | |||
greater_is_better = False | |||
score_needs_proba = True | |||
perfect_score = 0.0 | |||
# Range [0, Inf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for including these range comments. Super helpful 😁
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work, LGTM! 🥳
Just out of curiosity, what did we decide for 'edge cases' such as the correlation matrix? I guess it's not really what we currently consider as an objective... but was wondering if we decided on anything :3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me.
@@ -14,6 +14,8 @@ class CostBenefitMatrix(BinaryClassificationObjective): | |||
greater_is_better = True | |||
score_needs_proba = False | |||
perfect_score = np.inf | |||
# Range (-Inf, Inf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't this go next to perfect_score
?
@angela97lin Correlation is just like MCC in the sense that it's bounded [-1, 1]. I think it's best to not treat like a percentage in that case. If baseline gets a negative score and our pipeline gets a positive score, then I'd want the percent better to be above 100% which isn't always the case if we treat it like a percentage. |
7f4febe
to
7ed3582
Compare
Pull Request Description
Fixes #1449
After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of
docs/source/release_notes.rst
to include this pull request by adding :pr:123
.