Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Fix failing tests in tabular #2619

Closed
nirhutnik opened this issue Jun 29, 2023 · 1 comment · Fixed by #2620
Closed

[BUG] Fix failing tests in tabular #2619

nirhutnik opened this issue Jun 29, 2023 · 1 comment · Fixed by #2620
Assignees
Labels
bug linear needs triage Issue needs to be labeled and prioritized

Comments

@nirhutnik
Copy link
Contributor

tests fail on main:

=========================== short test summary info ============================
FAILED tests/tabular/checks/model_evaluation/simple_model_comparison_test.py::test_regression_tree - IndexError: invalid index to scalar variable.
FAILED tests/tabular/checks/model_evaluation/simple_model_comparison_test.py::test_regression_tree_random_state - IndexError: invalid index to scalar variable.
FAILED tests/tabular/checks/model_evaluation/simple_model_comparison_test.py::test_regression_tree_max_depth - IndexError: invalid index to scalar variable.
FAILED tests/tabular/suites/test_suites.py::test_kiss_dataset_suite - AssertionError: 
Expected: a sequence containing items matching (an instance of DeepchecksBaseError)
     but: was <[NotEnoughSamplesError("Not enough samples to calculate drift score. Minimum 10 samples required. Note that for numerical labels, None values do not count as samples.Use the 'min_samples' parameter to change this requirement."), DeepchecksProcessError('WeakSegmentsPerformance was unable to train an error model to find weak segments. Try increasing n_samples or supply additional features.'), DeepchecksValueError('Label observed values [1, 3] contain values that are not found in the model classes: [1, 2, 5].'), ModelValidationError('Check is irrelevant for classification tasks'), ModelValidationError('Check is irrelevant for classification tasks'), IndexError('invalid index to scalar variable.'), IndexError('invalid index to scalar variable.'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain an index'), NotEnoughSamplesError("Not enough samples to calculate drift score. Minimum 10 samples required. Note that for numerical columns, None values do not count as samples.Use the 'min_samples' parameter to change this requirement."), NotEnoughSamplesError("Not enough samples to calculate drift score. Minimum 10 samples required. Note that for numerical labels, None values do not count as samples.Use the 'min_samples' parameter to change this requirement."), NotEnoughSamplesError('There are not enough samples to run this check, found only 4 samples.'), NotEnoughSamplesError('There are not enough samples to run this check, found only 2 samples.'), DatasetValidationError('Dataset does not contain an index or a datetime'), DatasetValidationError('Dataset does not contain an index or a datetime')]>
FAILED tests/tabular/suites/test_suites.py::test_weird_classification_suite - AssertionError: 
Expected: a sequence containing items matching (an instance of DeepchecksBaseError)
     but: was <[ModelValidationError('Check is irrelevant for classification tasks'), ModelValidationError('Check is irrelevant for classification tasks'), IndexError('invalid index to scalar variable.'), IndexError('invalid index to scalar variable.'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain a datetime')]>
FAILED tests/tabular/suites/test_suites.py::test_weird_regression_suite - AssertionError: 
Expected: a sequence containing items matching (an instance of DeepchecksBaseError)
     but: was <[ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), IndexError('invalid index to scalar variable.'), IndexError('invalid index to scalar variable.'), ModelValidationError('Check is irrelevant for regression tasks'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain a datetime'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks')]>
FAILED tests/tabular/suites/test_suites.py::test_missing_test_classes_suite - AssertionError: 
Expected: a sequence containing items matching (an instance of DeepchecksBaseError)
     but: was <[ModelValidationError('Check is irrelevant for classification tasks'), ModelValidationError('Check is irrelevant for classification tasks'), IndexError('invalid index to scalar variable.'), IndexError('invalid index to scalar variable.'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain a datetime')]>
FAILED tests/tabular/suites/test_suites.py::test_diabetes_dataset_suite - AssertionError: 
Expected: a sequence containing items matching (an instance of DeepchecksBaseError)
     but: was <[ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), IndexError('invalid index to scalar variable.'), IndexError('invalid index to scalar variable.'), ModelValidationError('Check is irrelevant for regression tasks'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain an index'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), DatasetValidationError('Dataset does not contain an index or a datetime'), DatasetValidationError('Dataset does not contain an index or a datetime')]>
FAILED tests/tabular/suites/test_suites.py::test_generic_boost - AssertionError: 
Expected: a sequence containing items matching (an instance of DeepchecksBaseError)
     but: was <[ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), IndexError('invalid index to scalar variable.'), IndexError('invalid index to scalar variable.'), ModelValidationError('Check is irrelevant for regression tasks'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain an index'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), DatasetValidationError('Dataset does not contain an index or a datetime'), DatasetValidationError('Dataset does not contain an index or a datetime')]>
FAILED tests/tabular/suites/test_suites.py::test_generic_custom - AssertionError: 
Expected: a sequence containing items matching (an instance of DeepchecksBaseError)
     but: was <[ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), IndexError('invalid index to scalar variable.'), IndexError('invalid index to scalar variable.'), ModelValidationError("Check is relevant for Boosting models of type ('AdaBoostClassifier', 'GradientBoostingClassifier', 'LGBMClassifier', 'XGBClassifier', 'CatBoostClassifier', 'AdaBoostRegressor', 'GradientBoostingRegressor', 'LGBMRegressor', 'XGBRegressor', 'CatBoostRegressor'), but received model of type MyModel"), ModelValidationError('Check is irrelevant for regression tasks'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain a datetime'), DatasetValidationError('Dataset does not contain an index'), ModelValidationError('Check is irrelevant for regression tasks'), ModelValidationError('Check is irrelevant for regression tasks'), DatasetValidationError('Dataset does not contain an index or a datetime'), DatasetValidationError('Dataset does not contain an index or a datetime')]>
==== 10 failed, 1264 passed, 2 skipped, 7433 warnings in 1466.45s (0:24:26) ====
@harsh-deepchecks
Copy link
Contributor

Created an issue with scikilt-learn: scikit-learn/scikit-learn#26750

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug linear needs triage Issue needs to be labeled and prioritized
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants