[MRG+1] Adding multi output checks to common tests #13392

rok · 2019-03-05T15:33:46Z

Reference Issues/PRs

Changes

This implements new classifier and regressor checks to test for multi-output support in common tests.
Some random forest and tree classifier test are removed as they are duplicating this functionality.

sklearn/utils/estimator_checks.py

TomDLT

Not sure about these tag changes (#13392 (comment)). ping @amueller

sklearn/neighbors/regression.py

sklearn/utils/estimator_checks.py

sklearn/base.py

sklearn/linear_model/coordinate_descent.py

sklearn/linear_model/least_angle.py

TomDLT

Thanks for the update !
I like much more your Mixin fix now.

You will also need to add en entry in https://github.com/scikit-learn/scikit-learn/blob/master/doc/whats_new/v0.21.rst#changes-to-estimator-checks, writing the change, this PR number and your GitHub name.

sklearn/utils/tests/test_estimator_checks.py

jnothman · 2019-03-19T11:50:18Z

I wonder if requires_positive_data is implied by accepting pairwise data in all cases.

jnothman · 2019-03-19T11:50:42Z

But there's certainly nothing wrong with conditional tags

rok · 2019-03-19T12:14:04Z

I wonder if requires_positive_data is implied by accepting pairwise data in all cases.

@jnothman - right, is there some other way to know?

TomDLT

LGTM

We just need to wait for a second review.

doc/whats_new/v0.21.rst

sklearn/ensemble/tests/test_forest.py

sklearn/utils/estimator_checks.py

rok · 2019-04-03T20:44:13Z

Ping! :)

rok · 2019-05-10T00:16:19Z

I've rebased and maybe we could get it merged in v0.22? :)

rok · 2019-07-20T22:06:53Z

Rebased and fixed new issues. Since things changed it might be good to do another review.

amueller · 2019-09-04T19:35:59Z

ok this has a chance to be merged soon and would make this much easier: #14884

jnothman · 2019-09-18T11:31:42Z

With #14884 merged, and a bunch of nitpicks above, are you going to bring this to conclusion, @rok? Thanks.

rok · 2019-09-20T08:20:10Z

@jnothman - will try to finish this tonight. Sorry for the delay :)

rok · 2019-09-21T13:18:16Z

I'm not sure if I got the general approach right but I have:

moved mixins to the left
replaced intermediate classes by adding _more_tags to classes that inherit multioutput tag as True but should have it set to False. E.g. Lars is a multioutput class but LarsCV that inherits tags from Lars is not. We adjust LarsCV tags.

Please review :).

Thanks for pushing this @glemaitre @jnothman @amueller!

glemaitre

There is a couple of missing assert in the tests and I would add meaningful error message.
It would ease debugging when rolling your own estimator and that check_estimator is failing.

sklearn/utils/estimator_checks.py

glemaitre · 2019-09-24T08:42:58Z

sklearn/utils/estimator_checks.py

+    estimator.fit(X, y)
+    y_pred = estimator.predict(X)
+
+    assert y_pred.dtype == np.dtype('float')


Let's add an error message

"Multi-output predictions by a regressor are expected to be floating-point precision. Got {} instead".format(y_pred.dtype)

We can also have a similar style as before

assert y_pred.dtype.kind == 'f'

Had to set this to 'float64'.

sklearn/utils/estimator_checks.py

glemaitre

So a couple of changes required.

rok · 2019-09-24T09:38:54Z

Thanks @glemaitre. I'll do this tonight.

[MRG] Adding multi output checks to common tests (scikit-learn#13187) Removing redundant tests. Adding tests to check_classifier_multioutput and check_regressor_multioutput.

Co-Authored-By: Guillaume Lemaitre <g.lemaitre58@gmail.com>

rok · 2019-09-24T16:19:13Z

@glemaitre done, please review. :)

rok · 2019-09-30T11:34:09Z

@glemaitre ping :)

glemaitre · 2019-10-01T12:20:21Z

Thanks @rok

rok · 2019-10-01T13:32:25Z

Thanks all! :)

rok mentioned this pull request Mar 5, 2019

Missing multi-output checks in common tests #13187

Closed

TomDLT requested changes Mar 5, 2019

View changes

rok force-pushed the adding_multi-output_checks_to_common_tests branch from 5328660 to b720e06 Compare Mar 7, 2019

rok changed the title ~~Adding multi output checks to common tests~~ [MRG] Adding multi output checks to common tests Mar 7, 2019

TomDLT reviewed Mar 13, 2019

View changes

rok force-pushed the adding_multi-output_checks_to_common_tests branch from 33aa2ab to 8618a0e Compare Mar 17, 2019

TomDLT requested changes Mar 18, 2019

View changes

sklearn/utils/tests/test_estimator_checks.py Outdated Show resolved Hide resolved

rok force-pushed the adding_multi-output_checks_to_common_tests branch from 9450697 to 499fc5b Compare Mar 18, 2019

TomDLT approved these changes Mar 22, 2019

View changes

doc/whats_new/v0.21.rst Outdated Show resolved Hide resolved

thomasjpfan reviewed Mar 28, 2019

View changes

sklearn/ensemble/tests/test_forest.py Show resolved Hide resolved

rok force-pushed the adding_multi-output_checks_to_common_tests branch from 85cf3c0 to c53179d Compare Mar 28, 2019

TomDLT reviewed Mar 28, 2019

View changes

sklearn/utils/estimator_checks.py Outdated Show resolved Hide resolved

TomDLT approved these changes Apr 3, 2019

View changes

rok force-pushed the adding_multi-output_checks_to_common_tests branch from 6a222f3 to 9a482b6 Compare Apr 11, 2019

rok force-pushed the adding_multi-output_checks_to_common_tests branch 5 times, most recently from c9ceb6e to 3f3671b Compare May 9, 2019

TomDLT added this to the 0.22 milestone May 10, 2019

rth self-requested a review Jun 25, 2019

rok force-pushed the adding_multi-output_checks_to_common_tests branch 3 times, most recently from 14df99b to 86e6baf Compare Jul 20, 2019

glemaitre added this to GETTING STALLED in Guillaume's pet Sep 10, 2019

rok force-pushed the adding_multi-output_checks_to_common_tests branch 3 times, most recently from 0f23831 to 1969e70 Compare Sep 21, 2019

rok force-pushed the adding_multi-output_checks_to_common_tests branch from 1969e70 to fd6c04d Compare Sep 23, 2019

glemaitre reviewed Sep 24, 2019

View changes

glemaitre self-requested a review Sep 24, 2019

glemaitre requested changes Sep 24, 2019

View changes

glemaitre moved this from GETTING STALLED to REVIEWED AND WAITING FOR CHANGES in Guillaume's pet Sep 24, 2019

rok and others added 3 commits Sep 24, 2019

Adding multi-output checks for estimators.

1d3c819

[MRG] Adding multi output checks to common tests (scikit-learn#13187) Removing redundant tests. Adding tests to check_classifier_multioutput and check_regressor_multioutput.

Switching to overwriting inherited tags.

b4070db

Update sklearn/utils/estimator_checks.py

2b7dab1

Co-Authored-By: Guillaume Lemaitre <g.lemaitre58@gmail.com>

rok force-pushed the adding_multi-output_checks_to_common_tests branch from c1457ad to 8cef195 Compare Sep 24, 2019

rok force-pushed the adding_multi-output_checks_to_common_tests branch 2 times, most recently from bb8f4eb to 78f3938 Compare Sep 24, 2019

Review feedback.

8eabe3b

rok force-pushed the adding_multi-output_checks_to_common_tests branch from 78f3938 to 8eabe3b Compare Sep 24, 2019

Update v0.22.rst

c12bb6d

glemaitre approved these changes Oct 1, 2019

View changes

glemaitre merged commit 5e4b275 into scikit-learn:master Oct 1, 2019
13 of 15 checks passed

glemaitre moved this from REVIEWED AND WAITING FOR CHANGES to MERGED in Guillaume's pet Oct 1, 2019

scikit-learn / scikit-learn Public

[MRG+1] Adding multi output checks to common tests #13392

[MRG+1] Adding multi output checks to common tests #13392

rok commented Mar 5, 2019

TomDLT left a comment

TomDLT left a comment

jnothman commented Mar 19, 2019

jnothman commented Mar 19, 2019

rok commented Mar 19, 2019

TomDLT left a comment

rok commented Apr 3, 2019

rok commented May 10, 2019

rok commented Jul 20, 2019

amueller commented Sep 4, 2019

jnothman commented Sep 18, 2019

rok commented Sep 20, 2019

rok commented Sep 21, 2019 •

edited

glemaitre left a comment

glemaitre Sep 24, 2019

glemaitre Sep 24, 2019

rok Sep 24, 2019

glemaitre left a comment

rok commented Sep 24, 2019

rok commented Sep 24, 2019

rok commented Sep 30, 2019

glemaitre commented Oct 1, 2019

rok commented Oct 1, 2019

scikit-learn / scikit-learn Public

[MRG+1] Adding multi output checks to common tests #13392

[MRG+1] Adding multi output checks to common tests #13392

Conversation

rok commented Mar 5, 2019

Reference Issues/PRs

Changes

TomDLT left a comment

TomDLT left a comment

jnothman commented Mar 19, 2019

jnothman commented Mar 19, 2019

rok commented Mar 19, 2019

TomDLT left a comment

rok commented Apr 3, 2019

rok commented May 10, 2019

rok commented Jul 20, 2019

amueller commented Sep 4, 2019

jnothman commented Sep 18, 2019

rok commented Sep 20, 2019

rok commented Sep 21, 2019 • edited

glemaitre left a comment

glemaitre Sep 24, 2019

Choose a reason for hiding this comment

glemaitre Sep 24, 2019

Choose a reason for hiding this comment

rok Sep 24, 2019

Choose a reason for hiding this comment

glemaitre left a comment

rok commented Sep 24, 2019

rok commented Sep 24, 2019

rok commented Sep 30, 2019

glemaitre commented Oct 1, 2019

rok commented Oct 1, 2019

rok commented Sep 21, 2019 •

edited