InvalidDataTargetCheck additions by chukarsten · Pull Request #1665 · alteryx/evalml

chukarsten · 2021-01-07T22:25:56Z

For regression, error if the problem type was not "numeric"
For binary, error if the problem type was "categorical" (but first we should confirm that if a categorical column has only two unique values (other than nan) woodwork will label it as binary, not as "categorical" (this was a little unclear - I think this already is the case and is tested.)
For multiclass, error if the problem type was binary
For multiclass, warn if the problem type had a high number of unique values--perhaps set a max cap at over 5%.

codecov · 2021-01-11T15:38:44Z

Codecov Report

Merging #1665 (9da1f50) into main (80daa66) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1665     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         240      240             
  Lines       18577    18625     +48     
=========================================
+ Hits        18569    18617     +48     
  Misses          8        8

Impacted Files	Coverage Δ
evalml/data_checks/no_variance_data_check.py	`100.0% <ø> (ø)`
...lml/tests/model_understanding_tests/test_graphs.py	`100.0% <ø> (ø)`
evalml/data_checks/data_check_message_code.py	`100.0% <100.0%> (ø)`
evalml/data_checks/invalid_targets_data_check.py	`100.0% <100.0%> (ø)`
evalml/tests/data_checks_tests/test_data_checks.py	`100.0% <100.0%> (ø)`
...ta_checks_tests/test_invalid_targets_data_check.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80daa66...9da1f50. Read the comment docs.

chukarsten · 2021-01-12T02:45:25Z

evalml/tests/data_checks_tests/test_invalid_targets_data_check.py


    # with 1 class not having min 2 instances
-    assert invalid_targets_check.validate(X, y=pd.Series([0, 1, 1, 2, 2])) == {
+    assert invalid_targets_check.validate(X, y=pd.Series([0] + [1] * 19 + [2] * 80)) == {


This is an edit to make the targets pass the test that looks for targets with large numbers of unique values relative to the total number of target values. I do this in a few places.

bchen1116

Just left a question

bchen1116 · 2021-01-12T17:11:59Z

evalml/data_checks/invalid_targets_data_check.py

+                    details=details).to_dict())
+
+            num_class_to_num_value_ratio = len(unique_values) / len(y)
+            if num_class_to_num_value_ratio >= self.multiclass_continuous_threshold:


Not sure I'm understanding this. If there's 3 unique values in a list of length 50, this would warn that the problem could be regression? 3/50 == 0.06 > 0.05. If so, this default seems really high, but definitely open to discuss.

Good point, I think that's why this is a warning rather than an error, but maybe there could be an additional check taking into account the size of the target data when calculating this ratio?

Yea I agree that I wouldn't expect a warning on a problem with 3 unique classes across 50 observations. Maybe we should bump it higher. As a follow up, it'd be nice if the detect_problem_type util used this same logic for recommending multiclass vs regression. Right now that has a hard cap on 10 classes for multiclass. That can be tackled in a separate issue though.

Yup, I think this is a good place for a discussion. I just implemented what was in the issue as I wasn't present for the discussion - a 5% threshold. I'm certainly down for a smarter method of doing it.

evalml/data_checks/no_variance_data_check.py

ParthivNaresh

Looks excellent, just a few wording changes!

evalml/data_checks/data_check_message_code.py

evalml/data_checks/invalid_targets_data_check.py

ParthivNaresh · 2021-01-12T18:39:39Z

evalml/tests/data_checks_tests/test_invalid_targets_data_check.py

                                                                         "objective": None}})
+
+
+def test_invalid_target_data_check_regression_problem_nonnumeric_data():


You could parametrize with a non-regression problem type to check the differences if you want to

So I went ahead and parametrized it...I guess I'm not very familiar with what we ultimately want. I know in some of my tests I use non-numerics for a categorical, but I don't think we're setup to do that, right? ["Happy", "Birthday", "Birthday"] isn't considered a multiclass target, but [0, 1, 2, 2, 2, 1] is, right?

evalml/tests/data_checks_tests/test_invalid_targets_data_check.py

freddyaboulton

@chukarsten Thanks for the improvements!

evalml/data_checks/invalid_targets_data_check.py

freddyaboulton · 2021-01-12T18:54:03Z

evalml/data_checks/invalid_targets_data_check.py

+                    details=details).to_dict())
+
+            num_class_to_num_value_ratio = len(unique_values) / len(y)
+            if num_class_to_num_value_ratio >= self.multiclass_continuous_threshold:


Yea I agree that I wouldn't expect a warning on a problem with 3 unique classes across 50 observations. Maybe we should bump it higher. As a follow up, it'd be nice if the detect_problem_type util used this same logic for recommending multiclass vs regression. Right now that has a hard cap on 10 classes for multiclass. That can be tackled in a separate issue though.

… Need to refactor to handle the details checks.

…ulticlass problem with binary target data and tests. Need to update the docstrings and also look at how the previous X and y were updated to not trigger the large amount of classes warning. Many of these X/y combos can probably be merged.

…e issue.

Shortened the unique class message.

Refactored to accomodate : TARGET_MULTICLASS_HIGH_UNIQUE_CLASS

Updated tests to accomodate : TARGET_MULTICLASS_HIGH_UNIQUE_CLASS change.

chukarsten force-pushed the 1548-make_invalidatargetdatacheck_smarter branch from 8d064b0 to a0a5f39 Compare January 8, 2021 21:43

chukarsten force-pushed the 1548-make_invalidatargetdatacheck_smarter branch from 645b9fa to f0989dd Compare January 11, 2021 17:14

chukarsten commented Jan 12, 2021

View reviewed changes

chukarsten marked this pull request as ready for review January 12, 2021 15:33

chukarsten requested review from ParthivNaresh, angela97lin, bchen1116, dsherry, freddyaboulton and jeremyliweishih January 12, 2021 15:33

auto-assign bot assigned chukarsten Jan 12, 2021

bchen1116 reviewed Jan 12, 2021

View reviewed changes

ParthivNaresh approved these changes Jan 12, 2021

View reviewed changes

freddyaboulton approved these changes Jan 12, 2021

View reviewed changes

chukarsten and others added 15 commits January 12, 2021 16:19

Updated the tests and class to cover the regression/non-numeric case.…

89be984

… Need to refactor to handle the details checks.

Forgot a period which wasted a lot of time.

8dbe8f4

Autopep8

38e31a8

Release notes.

91b2d15

Made the test_data_checks pass.

c9caf2d

Updated whitespace lint error.

53c1fa0

Moved the module variable into the class.

0bd3b6f

Updated some of the error messages and formatting.

5796520

Before I try the binary target and multiclass problem type part of th…

e417ac3

…e issue.

Removed the 4th test.

27d5c61

Update data_check_message_code.py

f52bcf4

Shortened the unique class message.

Update invalid_targets_data_check.py

2aaa2ea

Refactored to accomodate : TARGET_MULTICLASS_HIGH_UNIQUE_CLASS

Update test_data_checks.py

4032413

Updated tests to accomodate : TARGET_MULTICLASS_HIGH_UNIQUE_CLASS change.

Addressed the comment about _WARNING.

8142f6b

chukarsten added 2 commits January 12, 2021 16:19

Updated a few more wording issues with messages.

d130fec

Addressed all comments.

9da1f50

chukarsten force-pushed the 1548-make_invalidatargetdatacheck_smarter branch from 1f410af to 9da1f50 Compare January 12, 2021 21:20

chukarsten merged commit ba7590f into main Jan 12, 2021

chukarsten deleted the 1548-make_invalidatargetdatacheck_smarter branch January 12, 2021 21:55

bchen1116 mentioned this pull request Jan 26, 2021

Release v0.18.0 #1743

Merged

		"objective": None}})


		def test_invalid_target_data_check_regression_problem_nonnumeric_data():

Conversation

chukarsten commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bchen1116 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ParthivNaresh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

freddyaboulton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chukarsten commented Jan 7, 2021 •

edited

Loading

codecov bot commented Jan 11, 2021 •

edited

Loading