ValueError raised when using a PLSRegression as the base_estimator in a BaggingRegressor with oob_score set to True

[jb-delafosse]

Description

ValueError raised when using a PLSRegression as the base_estimator in a BaggingRegressor with oob_score set to True

It might be linked to the fact that PLSRegression.predict returns y_pred with shape=(n,1) instead of (n,) as other estimators do.

Steps/Code to Reproduce

import numpy as np
from sklearn.cross_decomposition import PLSRegression
from sklearn.ensemble import BaggingRegressor

x_train=np.random.randn(100,100)
y_train=np.random.randn(100)

plsModel = PLSRegression(n_components=20)
baggingModel = BaggingRegressor(base_estimator = plsModel, 
                                bootstrap = True, 
                                oob_score=True, 
                                n_estimators=10)
baggingModel.fit(x_train,y_train)

Expected Results

No error is thrown

Actual Results

ValueError                                Traceback (most recent call last)
<ipython-input-12-ac7c7bc11194> in <module>()
     12                                 oob_score=True,
     13                                 n_estimators=10)
---> 14 baggingModel.fit(x_train,y_train)

C:\Anaconda3\lib\site-packages\sklearn\ensemble\bagging.py in fit(self, X, y, sample_weight)
    246             Returns self.
    247         """
--> 248         return self._fit(X, y, self.max_samples, sample_weight=sample_weight)
    249 
    250     def _fit(self, X, y, max_samples=None, max_depth=None, sample_weight=None):

C:\Anaconda3\lib\site-packages\sklearn\ensemble\bagging.py in _fit(self, X, y, max_samples, max_depth, sample_weight)
    380 
    381         if self.oob_score:
--> 382             self._set_oob_score(X, y)
    383 
    384         return self

C:\Anaconda3\lib\site-packages\sklearn\ensemble\bagging.py in _set_oob_score(self, X, y)
    980             mask = ~samples
    981 
--> 982             predictions[mask] += estimator.predict((X[mask, :])[:, features])
    983             n_predictions[mask] += 1
    984 

ValueError: non-broadcastable output operand with shape (38,) doesn't match the broadcast shape (38,38)

Versions

Windows-7-6.1.7601-SP1
Python 3.5.3 |Anaconda 2.5.0 (64-bit)| (default, Feb 22 2017, 21:28:42) [MSC v.1900 64 bit (AMD64)]
NumPy 1.11.3
SciPy 0.18.1
Scikit-Learn 0.18.1

[lesteve]

Thanks a lot for posting a stand-alone snippet reproducing the problem! I think that is because PLS-based estimators are slightly peculiar compared to most other scikit-learn estimators: .fit is called on two 2d arrays and .predict returns a 2d array.

I am going to close the issue, please shout if you strongly disagree.

[jb-delafosse]

Thanks for your quick answer !
I strongly disagree with you though :)

From what I understand : the PLS implementation in its current state makes PLS-based estimators differents from other scikit-learn estimators. But ideally, we should strive to have all the estimators behaving in the same way !

I'll try to take a closer look at this issue later.

[lesteve]

This is a fundamental property of PLS not anything related to scikit-learn.

[jb-delafosse]

I'm afraid I don't understand what you are trying to say.

Why is the y returned by PLSRegression.predict() any different from the y returned by LinearRegression.predict() for instance ? Shouldn't it be a 1D array when y_train is a 1D array ?

[bthirion]

Agree that PLSRegression.predict() should behave like LinearRegression.predict().