[MRG+1] cross_val_predict handles multi-label predict_proba #8773

stephen-hoover · 2017-04-21T14:26:53Z

What does this implement/fix? Explain your changes.

The following fails under v0.18.1:

from sklearn.datasets import make_multilabel_classification
from sklearn.model_selection import cross_val_predict
from sklearn.ensemble import RandomForestClassifier

X, y = make_multilabel_classification(n_samples=100, n_labels=3, n_classes=4, n_features=5, random_state=42)
y[:, 0] += y[:, 1]  # Put three classes in the first column
est = RandomForestClassifier(n_estimators=5, random_state=0)
cross_val_predict(est, X, y, 'predict_proba')

The error is

ValueError: Found input variables with inconsistent numbers of samples: [100, 100, 13]

This PR modifies the cross_val_predict and _fit_and_predict functions so that they handle multi-label (and also multi-class multi-label) classification problems with the predict_proba, predict_log_proba, and decision_function methods.

I've found two different kinds of multi-label outputs from scikit-learn estimators. The OneVersusRestClassifier handles multi-label tasks with binary indicator target arrays (no multi-label targets). It outputs 2D arrays from predict_proba, etc. methods. The RandomForestClassifier handles multi-class multi-label problems. It outputs a list of 2D arrays from predict_proba, etc.

I recognize the RandomForest-like outputs by type-checking. Lists of 2D arrays require slightly different code for keeping track of indices than single 2D output arrays.

Any other comments?

I didn't make any modifications to handle sparse outputs for these cases. I don't know if it's necessary. Do any estimators return sparse outputs for multi-class multi-label classification problems?

Modify the `cross_val_predict` and `_fit_and_predict` functions so that they handle multi-label (and multi-class multi-label) classification problems with `predict_proba`, `predict_log_proba`, and `decision_function` methods. There's two different kinds of multi-label outputs from scikit-learn estimators. The `OneVersusRestClassifier` handles multi-label tasks with binary indicator target arrays (no multi-label targets). It outputs 2D arrays from `predict_proba`, etc. methods. The `RandomForestClassifier` handles multi-class multi-label problems. It outputs a list of 2D arrays from `predict_proba`, etc. Recognize the RandomForest-like outputs by type-checking. Lists of 2D arrays require slightly different code for keeping track of indices.

jnothman · 2017-04-27T06:43:16Z

Sparse outputs aren't so relevant to predict_proba

jnothman · 2017-05-25T11:55:47Z

sklearn/model_selection/_validation.py

@@ -419,9 +428,20 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None, n_jobs=1,
    # Check for sparse predictions


Remove this, or move it.

jnothman · 2017-05-25T11:56:07Z

sklearn/model_selection/_validation.py

@@ -419,9 +428,20 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None, n_jobs=1,
    # Check for sparse predictions
    if sp.issparse(predictions[0]):
        predictions = sp.vstack(predictions, format=predictions[0].format)
+    elif do_manual_encoding and isinstance(predictions[0], list):
+        n_labels = y.shape[1]


I think a comment here is deserved to remind us what we've got and where it's going

Added a comment.

jnothman · 2017-05-25T11:56:55Z

sklearn/model_selection/_validation.py

    else:
        predictions = np.concatenate(predictions)
-    return predictions[inv_test_indices]
+
+    if do_manual_encoding and isinstance(predictions, list):


I don't see why you need do_manual_encoding here...

Ah, you're right. Checking if predictions is a list is enough. Changed.

jnothman · 2017-05-25T12:01:38Z

sklearn/model_selection/_validation.py

-    if method in ['decision_function', 'predict_proba', 'predict_log_proba']:
-        le = LabelEncoder()
-        y = le.fit_transform(y)
+    do_manual_encoding = method in ['decision_function', 'predict_proba',


I don't like this variable name. Perhaps just encode, encoded or is_encoded would do

Changed to encode.

…val-predict

stephen-hoover · 2017-05-31T18:21:30Z

@jnothman , apologies for taking a long time to respond here. I believe I've addressed your comments.

stephen-hoover · 2017-07-26T20:05:53Z

@jnothman , would you like additional changes here?

jnothman · 2017-07-27T10:38:34Z

Oh, I thought we'd merged this fix at some point :(

jnothman

I think this is good, but I'm too tired to be sure!

jnothman · 2017-07-27T10:38:46Z

doc/whats_new.rst

@@ -178,6 +178,10 @@ Enhancements
     removed by setting it to `None`.
     :issue:`7674` by :user:`Yichuan Liu <yl565>`.

+   - Added ability for :func:`model_selection.cross_val_predict` to handle multi-label
+     (and multi-class multi-label) targets with `predict_proba`-type methods.


double backticks

jnothman · 2017-07-27T11:28:43Z

sklearn/model_selection/_validation.py

+    elif encode and isinstance(predictions[0], list):
+        # `predictions` is a list of method outputs from each fold.
+        # If each of those is also a list, then treat this as a
+        # multi-class multi-label task. We need to separately concatenate


do you mean multi-output multiclass?

stephen-hoover · 2017-07-27T16:43:46Z

@jnothman , thanks for taking another look at this PR. I added double backticks and changed "multi-class multi-label" to "multioutput-multiclass" (the spelling I found in existing documentation) in that comment and in the What's New.

jnothman · 2017-07-27T22:11:14Z

definitely wasn't a full review yet. All of the class list games are so ugly!

…

On 28 Jul 2017 2:43 am, "Stephen Hoover" ***@***.***> wrote: @jnothman <https://github.com/jnothman> , thanks for taking another look at this PR. I added double backticks and changed "multi-class multi-label" to "multioutput-multiclass" (the spelling I found in existing documentation) in that comment and in the What's New. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8773 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6xTsIRXhSnVb8v6xEUNzcjWRJIcXks5sSL5EgaJpZM4NEX4n> .

jnothman

I'm not convinced that the need for _enforce_prediction_order is tested here. Can we make sure this is tested on data where the set of classes for training each fold will vary, such as y = [1,2,3,4,5] and Y = [[0, 1], [1, 2], [0, 3], [1, 4], [0, 5]]?

It wouldn't hurt if check_cross_val_predict_with_method explicitly checked that the shape was as expected, either...

jnothman · 2017-07-28T05:02:39Z

sklearn/model_selection/tests/test_validation.py

-            expected_predictions[test] = func(X[test])
+            preds = func(X[test])
+            if isinstance(predictions, list):
+                for i_label in range(y.shape[1]):


Call it output rather than label, please

jnothman · 2017-07-28T05:03:23Z

sklearn/model_selection/tests/test_validation.py

        func = getattr(est, method)

        # Naive loop (should be same as cross_val_predict):
        for train, test in kfold.split(X, y):
            est.fit(X[train], y[train])
-            expected_predictions[test] = func(X[test])
+            preds = func(X[test])


I'm a bit confused. Doesn't this mean that sometimes we're going to have mismatched numbers of classes vis-a-vis what we try to solve with _enforce_prediction_order?

Let individual test functions control the for loops.

stephen-hoover · 2017-07-31T19:08:51Z

@jnothman , you're right that the tests weren't covering the case where _enforce_prediction_order was needed. I added two new tests which have a split with fewer classes than the full dataset. I also added some explicit asserts on the output.

jnothman · 2017-08-03T08:37:48Z

sklearn/model_selection/_validation.py

+    then the output prediction array might not have the same
+    columns as other folds. Use the list of class names
+    (assumed to be integers) to enforce the correct column order.
+    """


Is it worth fast-pathing the classes == arange case? Or is that premature optimisation?

That's a good idea. We can return immediately if all classes were present in the subset of data used to train this fold.

jnothman · 2017-08-03T08:37:50Z

sklearn/model_selection/_validation.py

+    predictions_ = np.zeros((predictions.shape[0], n_classes),
+                            dtype=predictions.dtype)
+    if one_col_if_binary and len(classes) == 2:
+        predictions_[:, classes[-1]] = predictions


This leaves one of the columns zeroed?

That's what we have to do if n_classes >= 3. But you're right that this is a problem if n_classes = 2. That should have a single column of output. I added a new test to catch this, test_cross_val_predict_binary_decision_function. It's handled now by having the function return predictions directly when len(classes) == n_classes.

jnothman · 2017-08-03T08:37:55Z

sklearn/model_selection/tests/test_validation.py

+
+
+def test_cross_val_predict_with_method_multilabel_rf():
+    # The RandomForest allows anything for the contents of the labels.


The wording here is unclear. Do you just mean that RF handles multiclass-multioutput and produces predict_proba in that vein?

Changed to "The RandomForest allows multiple classes in each label.".

jnothman · 2017-08-03T08:37:57Z

sklearn/model_selection/tests/test_validation.py

+    X = rng.normal(0, 1, size=(10, 10))
+    y = np.array([0, 1, 0, 1, 0, 1, 0, 1, 0, 2])
+    est = LogisticRegression()
+    for method in ['predict_proba', 'predict_log_proba']:


decision_function also?

I'd left it off because the test code doesn't handle the decision_function case when there's three classes in the full data but only 2 in one of the folds. I think that case is covered by test_cross_val_predict_class_subset. In this test, I modified y to have 4 classes instead of 3 and added "decision_function" to the list.

Also don't do unnecessary work if number of classes in a cross_val_predict fold equals number of classes in the full data.

stephen-hoover · 2017-08-06T21:12:03Z

@jnothman , comments addressed. Thanks for suggesting that optimization and pointing out a bug in _enforce_prediction_order.

amueller · 2017-08-28T21:59:17Z

needs merge fix

…val-predict

codecov · 2017-08-28T22:45:51Z

Codecov Report

Merging #8773 into master will increase coverage by <.01%.
The diff coverage is 97.29%.

@@            Coverage Diff             @@
##           master    #8773      +/-   ##
==========================================
+ Coverage   96.16%   96.17%   +<.01%     
==========================================
  Files         336      335       -1     
  Lines       62144    61953     -191     
==========================================
- Hits        59762    59581     -181     
+ Misses       2382     2372      -10

Impacted Files	Coverage Δ
sklearn/model_selection/_validation.py	`96.73% <94.44%> (+0.27%)`	⬆️
sklearn/model_selection/tests/test_validation.py	`98.75% <98.66%> (-0.02%)`	⬇️
sklearn/utils/tests/test_testing.py	`80.61% <0%> (-0.39%)`	⬇️
sklearn/ensemble/tests/test_gradient_boosting.py	`96.03% <0%> (-0.24%)`	⬇️
sklearn/linear_model/logistic.py	`96.86% <0%> (-0.21%)`	⬇️
sklearn/utils/estimator_checks.py	`93.19% <0%> (-0.2%)`	⬇️
sklearn/preprocessing/data.py	`98.65% <0%> (-0.17%)`	⬇️
...rn/semi_supervised/tests/test_label_propagation.py	`98.91% <0%> (-0.16%)`	⬇️
sklearn/utils/__init__.py	`94.48% <0%> (-0.05%)`	⬇️
sklearn/linear_model/stochastic_gradient.py	`98.13% <0%> (-0.04%)`	⬇️
... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b8a936...764e1fd. Read the comment docs.

stephen-hoover · 2018-06-25T15:08:20Z

Is there anything I can do to help this PR along?

jnothman · 2018-06-25T22:07:26Z

I thought of this PR yesterday. I think it's essential before we merge stacking using cross_val_predict (though there are unresolved conflicts there). @glemaitre? @qinhanmin2014?

glemaitre · 2018-06-25T22:26:57Z

I will look at it tomorrow.

jnothman · 2018-07-15T12:28:06Z

@glemaitre, do you have some time to review this?

amueller · 2018-07-20T18:46:11Z

does someone wanna review? otherwise I say untag.

adrinjalali

LGTM, should we merge this while the other issues/approaches move forward?

jnothman · 2019-04-06T10:29:19Z

I think merging this will be especially valuable if we review and merge StackingClassifier.

jnothman · 2019-04-06T11:21:51Z

Thanks @stephen-hoover, and sorry for the very slow ride!!

stephen-hoover · 2019-04-07T01:21:40Z

Thank you for the merge!

…learn#8773)

…cikit-learn#8773)" This reverts commit 0eaceb6.

…learn#8773)

stephen-hoover force-pushed the multilabel-cross-val-predict branch from 7e21876 to ca2ee0b Compare April 21, 2017 14:31

stephen-hoover changed the title ~~[WIP] cross_val_predict handles multi-label predict_proba~~ [MRG] cross_val_predict handles multi-label predict_proba Apr 21, 2017

jnothman added Enhancement Waiting for Reviewer labels Apr 27, 2017

jnothman reviewed May 25, 2017

View reviewed changes

Stephen Hoover added 2 commits May 31, 2017 11:28

do_manual_encoding -> encode; better commenting

b2adf2c

Merge remote-tracking branch 'upstream/master' into multilabel-cross-…

fdfbc6b

…val-predict

jnothman reviewed Jul 27, 2017

View reviewed changes

Change docs to multioutput-multiclass, backticks

6284761

jnothman reviewed Jul 28, 2017

View reviewed changes

Stephen Hoover added 4 commits July 31, 2017 11:45

TST Move for loop out of check_cross_val_predict_with_method

5a65f4f

Let individual test functions control the for loops.

TST Test shape of cross_val_predict outputs

2e6055b

TST Rename i_label -> i_output

db1e134

TST Test cases where _enforce_prediction_order is needed

6e044fc

jnothman reviewed Aug 3, 2017

View reviewed changes

BUG Output one column for binary decision_function

3a83421

Also don't do unnecessary work if number of classes in a cross_val_predict fold equals number of classes in the full data.

jnothman mentioned this pull request Aug 28, 2017

[MRG+1] Stacking classifier with pipelines API #8960

Closed

7 tasks

Merge remote-tracking branch 'upstream/master' into multilabel-cross-…

764e1fd

…val-predict

flake8

9bf92d4

glemaitre added this to the 0.20 milestone Jun 8, 2018

ogrisel added this to PRs tagged in scikit-learn 0.20 Jul 16, 2018

Merge branch 'master' into multilabel-cross-val-predict

02d53ec

amueller force-pushed the multilabel-cross-val-predict branch 2 times, most recently from 1f1fe77 to 02d53ec Compare July 20, 2018 18:45

actually commit fix

8aa86a9

amueller modified the milestones: 0.20, 0.21 Aug 17, 2018

amueller mentioned this pull request Oct 9, 2018

'cross_val_predict' throws error when estimator is 'OneVsRest Classifier' and method is 'decision_function' #11058

Closed

adrinjalali added 2 commits April 5, 2019 13:59

merge upstream/master

f5455a8

merge upstream/master, again

5b2551e

adrinjalali removed the Waiting for Reviewer label Apr 5, 2019

adrinjalali approved these changes Apr 5, 2019

View reviewed changes

fix lbfgs future warning

1e2f6d0

Merge branch 'master' into multilabel-cross-val-predict

8f0b213

jnothman merged commit 24df999 into scikit-learn:master Apr 6, 2019

stephen-hoover deleted the multilabel-cross-val-predict branch April 7, 2019 01:21

jeremiedbb pushed a commit to jeremiedbb/scikit-learn that referenced this pull request Apr 25, 2019

ENH cross_val_predict now handles multi-output predict_proba (scikit-…

5ebcef0

…learn#8773)

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

ENH cross_val_predict now handles multi-output predict_proba (scikit-…

0eaceb6

…learn#8773)

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "ENH cross_val_predict now handles multi-output predict_proba (s…

c6aa3ea

…cikit-learn#8773)" This reverts commit 0eaceb6.

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "ENH cross_val_predict now handles multi-output predict_proba (s…

82c8132

…cikit-learn#8773)" This reverts commit 0eaceb6.

koenvandevelde pushed a commit to koenvandevelde/scikit-learn that referenced this pull request Jul 12, 2019

ENH cross_val_predict now handles multi-output predict_proba (scikit-…

b4f525a

…learn#8773)

		@@ -419,9 +428,20 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None, n_jobs=1,
		# Check for sparse predictions



		def test_cross_val_predict_with_method_multilabel_rf():
		# The RandomForest allows anything for the contents of the labels.

[MRG+1] cross_val_predict handles multi-label predict_proba #8773

[MRG+1] cross_val_predict handles multi-label predict_proba #8773

Conversation

stephen-hoover commented Apr 21, 2017 • edited by adrinjalali Loading

What does this implement/fix? Explain your changes.

Any other comments?

jnothman commented Apr 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephen-hoover commented May 31, 2017

stephen-hoover commented Jul 26, 2017

jnothman commented Jul 27, 2017

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephen-hoover commented Jul 27, 2017

jnothman commented Jul 27, 2017 via email

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephen-hoover commented Jul 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephen-hoover commented Aug 6, 2017

amueller commented Aug 28, 2017

codecov bot commented Aug 28, 2017 • edited Loading

Codecov Report

stephen-hoover commented Jun 25, 2018

jnothman commented Jun 25, 2018 via email

glemaitre commented Jun 25, 2018 via email • edited Loading

jnothman commented Jul 15, 2018

amueller commented Jul 20, 2018

adrinjalali left a comment

Choose a reason for hiding this comment

jnothman commented Apr 6, 2019

jnothman commented Apr 6, 2019

stephen-hoover commented Apr 7, 2019

stephen-hoover commented Apr 21, 2017 •

edited by adrinjalali

Loading

codecov bot commented Aug 28, 2017 •

edited

Loading

glemaitre commented Jun 25, 2018 via email •

edited

Loading