Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ROC/confusion matrix if user passes own objectives #276

Merged
merged 6 commits into from Dec 26, 2019
Merged

Conversation

angela97lin
Copy link
Contributor

@angela97lin angela97lin commented Dec 11, 2019

Closes #246.

Somewhat hacky / hard-coded though, could be worthwhile to do the refactor necessary for metrics that don't return single numerical values.

@codecov
Copy link

codecov bot commented Dec 11, 2019

Codecov Report

Merging #276 into master will increase coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #276      +/-   ##
==========================================
+ Coverage   96.99%   97.01%   +0.01%     
==========================================
  Files          95       95              
  Lines        2932     2950      +18     
==========================================
+ Hits         2844     2862      +18     
  Misses         88       88
Impacted Files Coverage Δ
...ts/automl_tests/test_auto_classification_search.py 100% <100%> (ø) ⬆️
evalml/automl/auto_classification_search.py 100% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6970bfc...facd6c3. Read the comment docs.

@angela97lin angela97lin requested a review from kmax12 Dec 11, 2019
@angela97lin angela97lin self-assigned this Dec 12, 2019
Copy link
Contributor

@kmax12 kmax12 left a comment

implementation looks fine, just missing test case

@@ -52,6 +57,18 @@ def __init__(self, problem_type, tuner, cv, objective, max_pipelines, max_time,
if existing_main_objective is not None:
additional_objectives.remove(existing_main_objective)

# hacky, disallows non-numeric metrics from being primary objective
if isinstance(self.objective, ConfusionMatrix) or isinstance(self.objective, ROC):
Copy link
Contributor

@kmax12 kmax12 Dec 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be in auto classifier? i dont think these metrics are valid for regression problems

@angela97lin angela97lin requested a review from kmax12 Dec 26, 2019
kmax12
kmax12 approved these changes Dec 26, 2019
Copy link
Contributor

@kmax12 kmax12 left a comment

LGTM

@angela97lin angela97lin merged commit d56a33c into master Dec 26, 2019
@angela97lin angela97lin mentioned this pull request Mar 9, 2020
@angela97lin angela97lin deleted the plot_bug branch Apr 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ROC and confusion matrix not calculated if user passes in own additional objectives
2 participants