Skip to content

Updating percent better than baseline computation #1809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 11, 2021
Merged

Updating percent better than baseline computation #1809

merged 3 commits into from
Feb 11, 2021

Conversation

freddyaboulton
Copy link
Contributor

@freddyaboulton freddyaboulton commented Feb 9, 2021 •
edited
Loading

Pull Request Description

Fixes #1449

  • For objectives bounded strictly in the range [0, 1], the percent difference is the absolute difference (not normalized by the baseline)
  • For other objectives, division by 0 is now inf rather than nan

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

@codecov
Copy link

codecov bot commented Feb 9, 2021 •
edited
Loading

Codecov Report

Merging #1809 (7ed3582) into main (5c35743) will increase coverage by 0.1%.
The diff coverage is 100.0%.

Impacted file tree graph

@@            Coverage Diff            @@
##             main    #1809     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         252      252             
  Lines       20052    20117     +65     
=========================================
+ Hits        20044    20109     +65     
  Misses          8        8
Impacted Files Coverage Δ
evalml/objectives/cost_benefit_matrix.py 100.0% <100.0%> (ø)
evalml/objectives/fraud_cost.py 100.0% <100.0%> (ø)
evalml/objectives/lead_scoring.py 100.0% <100.0%> (ø)
evalml/objectives/objective_base.py 100.0% <100.0%> (ø)
evalml/objectives/standard_metrics.py 100.0% <100.0%> (ø)
evalml/tests/automl_tests/test_automl.py 100.0% <100.0%> (ø)
...lml/tests/objective_tests/test_standard_metrics.py 100.0% <100.0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c35743...7ed3582. Read the comment docs.

@freddyaboulton freddyaboulton force-pushed the 1449-percent-better-improvements branch from fe22f68 to 7f4febe Compare February 9, 2021 20:32
@freddyaboulton freddyaboulton marked this pull request as ready for review February 10, 2021 16:36
@freddyaboulton freddyaboulton added the enhancement An improvement to an existing feature. label Feb 10, 2021
bchen1116
bchen1116 approved these changes Feb 10, 2021
View reviewed changes
Copy link
Contributor

@bchen1116 bchen1116 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

evalml/objectives/cost_benefit_matrix.py Outdated
@@ -14,6 +14,8 @@ class CostBenefitMatrix(BinaryClassificationObjective):
greater_is_better = True
score_needs_proba = False
perfect_score = np.inf
# Range (-Inf, Inf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit-pick but I thought this was an accidental comment rather than on purpose. Maybe we can move it to be next to is_bounded_like_percentage rather than being above?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this go next to perfect_score?

evalml/objectives/lead_scoring.py Outdated
@@ -9,6 +9,8 @@ class LeadScoring(BinaryClassificationObjective):
greater_is_better = True
score_needs_proba = False
perfect_score = math.inf
# Range (-Inf, Inf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment

angela97lin
angela97lin reviewed Feb 11, 2021
View reviewed changes
evalml/objectives/standard_metrics.py Outdated
@@ -243,6 +263,8 @@ class LogLossBinary(BinaryClassificationObjective):
greater_is_better = False
score_needs_proba = True
perfect_score = 0.0
# Range [0, Inf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for including these range comments. Super helpful 😁

angela97lin
angela97lin approved these changes Feb 11, 2021
View reviewed changes
Copy link
Contributor

@angela97lin angela97lin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, LGTM! 🥳

Just out of curiosity, what did we decide for 'edge cases' such as the correlation matrix? I guess it's not really what we currently consider as an objective... but was wondering if we decided on anything :3

chukarsten
chukarsten approved these changes Feb 11, 2021
View reviewed changes
Copy link
Contributor

@chukarsten chukarsten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

evalml/objectives/cost_benefit_matrix.py Outdated
@@ -14,6 +14,8 @@ class CostBenefitMatrix(BinaryClassificationObjective):
greater_is_better = True
score_needs_proba = False
perfect_score = np.inf
# Range (-Inf, Inf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this go next to perfect_score?

@freddyaboulton
Copy link
Contributor Author

@angela97lin Correlation is just like MCC in the sense that it's bounded [-1, 1]. I think it's best to not treat like a percentage in that case. If baseline gets a negative score and our pipeline gets a positive score, then I'd want the percent better to be above 100% which isn't always the case if we treat it like a percentage.

@freddyaboulton freddyaboulton force-pushed the 1449-percent-better-improvements branch from 7f4febe to 7ed3582 Compare February 11, 2021 18:59
@freddyaboulton freddyaboulton merged commit cb09781 into main Feb 11, 2021
@freddyaboulton freddyaboulton deleted the 1449-percent-better-improvements branch February 11, 2021 19:31
@chukarsten chukarsten mentioned this pull request Feb 23, 2021
Merged
@dsherry dsherry mentioned this pull request Mar 10, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@angela97lin angela97lin angela97lin approved these changes

@bchen1116 bchen1116 bchen1116 approved these changes

@chukarsten chukarsten chukarsten approved these changes

@dsherry dsherry Awaiting requested review from dsherry

@jeremyliweishih jeremyliweishih Awaiting requested review from jeremyliweishih

@ParthivNaresh ParthivNaresh Awaiting requested review from ParthivNaresh

Assignees

@freddyaboulton freddyaboulton

Labels
enhancement An improvement to an existing feature.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

For some objectives where baseline was 0, "pct better than baseline" is nan
4 participants
@freddyaboulton @angela97lin @bchen1116 @chukarsten