Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix partial dependence graph method failing on multiclass problems when the label is numeric #2372

Merged
merged 8 commits into from
Jun 15, 2021

Conversation

angela97lin
Copy link
Contributor

Closes #2114

@angela97lin angela97lin self-assigned this Jun 10, 2021
@codecov
Copy link

codecov bot commented Jun 10, 2021

Codecov Report

Merging #2372 (e37c756) into main (a096a7c) will increase coverage by 10.4%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff            @@
##            main   #2372      +/-   ##
========================================
+ Coverage   89.3%   99.7%   +10.4%     
========================================
  Files        281     281              
  Lines      24907   24926      +19     
========================================
+ Hits       22222   24829    +2607     
+ Misses      2685      97    -2588     
Impacted Files Coverage Δ
evalml/model_understanding/graphs.py 100.0% <100.0%> (+44.4%) ⬆️
...del_understanding_tests/test_partial_dependence.py 100.0% <100.0%> (+42.8%) ⬆️
evalml/automl/automl_search.py 99.4% <0.0%> (+0.2%) ⬆️
evalml/tests/utils_tests/test_logger.py 100.0% <0.0%> (+0.6%) ⬆️
evalml/pipelines/utils.py 99.1% <0.0%> (+1.0%) ⬆️
evalml/tests/conftest.py 99.6% <0.0%> (+1.1%) ⬆️
...derstanding/prediction_explanations/_algorithms.py 98.9% <0.0%> (+1.2%) ⬆️
...s/prediction_explanations_tests/test_explainers.py 100.0% <0.0%> (+1.8%) ⬆️
evalml/tests/component_tests/test_estimators.py 100.0% <0.0%> (+1.9%) ⬆️
evalml/tests/component_tests/test_components.py 100.0% <0.0%> (+2.8%) ⬆️
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a096a7c...e37c756. Read the comment docs.

@@ -287,6 +287,45 @@ def test_partial_dependence_multiclass(logistic_regression_multiclass_pipeline_c
assert len(two_way_part_dep.columns) == grid_resolution + 1


def test_partial_dependence_multiclass_numeric_labels(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@freddyaboulton I'm sure you might have some opinions about me adding yet another long running test 😂 I wasn't confident in a way to mock things but definitely would love suggestions/opinions!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the runtime now? My takeaway is that using grid_resolution=5 can trim the test time to a couple seconds down from like ~10. The next step would be using fewer rows.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~30-40 seconds before, 8 seconds after setting grid_resolution to 5 😁

Copy link
Contributor

@chukarsten chukarsten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I think dropping the grid res is a good idea to make it a shorter test.

Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me @angela97lin !

@angela97lin angela97lin merged commit 1f7ccf9 into main Jun 15, 2021
@angela97lin angela97lin deleted the 2114_partial_dep branch June 15, 2021 00:34
@chukarsten chukarsten mentioned this pull request Jun 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Partial Dependence Graph fails on multiclass problems when the label is numeric
3 participants