Spurious cv FutureWarning with cross_val_score and LeaveOneGroupOut

[GaelVaroquaux]

The following code raises a FutureWarning:

from sklearn.model_selection import LeaveOneGroupOut, cross_val_score
cv = LeaveOneGroupOut()
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
from sklearn import datasets
X, y = datasets.load_iris(return_X_y=True)
groups = np.arange(len(y)) % 10
cross_val_score(model, X, y, groups=groups)

Ouput:

/home/varoquau/dev/scikit-learn/sklearn/model_selection/_split.py:1943: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.
  warnings.warn(CV_WARNING, FutureWarning)

This seems spurious to me.

[amueller]

how is that spurious? You're not using cv, right?

[GaelVaroquaux]

Yes. And adding the cv fixes the problem. Sorry for the noise.

@KamalakerDadi : this is not the problem in nilearn.

[KamalakerDadi]

I meant even if you specify cv. For instance, specifying cv=LeaveOneGroupOut()

[KamalakerDadi]

Nope. I don't think so. Sorry.

[GaelVaroquaux]

I meant even if you specify cv. For instance, specifying cv=LeaveOneGroupOut()

I tried, and couldn't reproduce on the code above. Can you can a small example that reproduces?

[KamalakerDadi]

Can you can a small example that reproduces?

Here it is. May be it shouldn't be used this way ?

from sklearn.datasets import make_blobs
from sklearn.model_selection import GridSearchCV, cross_val_score, ShuffleSplit
from sklearn.svm import SVC

X, y = make_blobs(n_samples=40, centers=2, random_state=6)
cv = ShuffleSplit()
svm_cv = GridSearchCV(SVC(C=1., kernel='linear'), param_grid={'C': [.1, .5]})

cross_val_score(svm_cv, X, y, cv=cv)

[adrinjalali]

You're doing a nested cross validation here @KamalakerDadi, for each fold in cross_val_score, GridSearchCV runs a cross validated parameter selection as well, and you're not setting the cv parameter of GridSearchCV here. That's why you're getting the warning.

[KamalakerDadi]

Thanks @adrinjalali it makes sense.