Modify calculate_percent_difference so that it can handle negative values #1100
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #1100 +/- ##
==========================================
+ Coverage 94.06% 99.91% +5.84%
==========================================
Files 192 192
Lines 10736 10751 +15
==========================================
+ Hits 10099 10742 +643
+ Misses 637 9 -628
Continue to review full report at Codecov.
|
angela97lin
reviewed
Aug 25, 2020
evalml/objectives/objective_base.py
Outdated
Comment on lines
121
to
128
| elif baseline_score < score and cls.greater_is_better: | ||
| increase = True | ||
| elif baseline_score > score and cls.greater_is_better: | ||
| increase = False | ||
| elif baseline_score < score and not cls.greater_is_better: | ||
| increase = False | ||
| else: | ||
| increase = True |
There was a problem hiding this comment.
Hmm, I wonder if this can be simplified a little? Suggestion:
increase = True
if (baseline_score > score and cls.greater_is_better) or (baseline_score < score and not cls.greater_is_better):
increase = False
There was a problem hiding this comment.
Good call! Really makes things cleaner. I think I was just being paranoid I would miss a case hehe
There was a problem hiding this comment.
Hahaha I don't blame you, I had to write it all out and make sure too 😂
angela97lin
approved these changes
Aug 25, 2020
freddyaboulton
force-pushed
the
percent-better-baseline-negative-values
branch
from
0e6d9f0
to
0850cf9
Compare
bchen1116
approved these changes
Aug 25, 2020
freddyaboulton
force-pushed
the
percent-better-baseline-negative-values
branch
from
0850cf9
to
f9dba67
Compare
dsherry
approved these changes
Aug 25, 2020
evalml/objectives/objective_base.py
Outdated
|
|
||
| increase = True | ||
| if (baseline_score > score and cls.greater_is_better) or (baseline_score < score and not cls.greater_is_better): | ||
| increase = False |
There was a problem hiding this comment.
Cool. My only thought is maybe you can invert it so you don't need the not in not increase
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
Let's say for CostBenefitMatrix, the model score is 5 and the baseline score is -10.
Right now, we compute 100 * (5 - (-10))/ -10 = -150%, indicating a decrease in performance when the right answer should be a 150% increase in performance.
This PR fixes that and adds coverage.
After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of
docs/source/release_notes.rstto include this pull request by adding :pr:123.