data-independent CV iterators

[mblondel]

In many situations, you don't have a test set so you would like to use CV for both evaluation and hyper-parameter tuning. Therefore, you need to do nested cross-validation:

for train, test in cv1:
   # Find the best hyper-parameters for this split
    for train, val in cv2:
        [...]
   # Retrain using the best hyper-parameters
   [...]
# Return best scores for each split

This is very difficult to implement in a generic way with our current API because CV iterators are tied to a particular data. For example, when doing cv = KFold(n_samples), cv will only work with a dataset of the specified size.

Ideally, we would need something closer to the estimator API: use constructor parameters for data-independent options (n_folds, shuffle, random_state, train / test proportion, etc) and a run method that takes y as argument (the reason to take y is for stratified schemes). This would look something like this:

# deprecated usage
for train, test in KFold(n, n_folds):
    print train, test

# new usage
for train, test in KFold(n_folds).run(y):
    print train, test

[GaelVaroquaux]

deprecated usage

for train, test in KFold(n, n_folds):
print train, test

new usage

for train, test in KFold(n_folds).run(y):
print train, test

+1e6!!!

I would argue for a different name for the method (maybe 'split'), but
the spirit is really the right one.

As this will be a major API change, I would think that it would be good
to good it with #2055. (FIXME: wrong issue number. I am trying to find the one I have in mind).

[GaelVaroquaux]

As this will be a major API change, I would think that it would be good
to good it with #2055.

I actually meant: #1848

[jnothman]

+1

We still have a question of how to handle the labels parameter for LOLO in cross_val_score or GridSearchCV, but in general, I think this is a much better interface.

[mblondel]

split is fine with me.

[agramfort]

I would vote for iter_splits

[GaelVaroquaux]

We still have a question of how to handle the labels parameter for LOLO in
cross_val_score or GridSearchCV,

KFold.split(arrays)

with specific names on arrays? Thus, if one array is 'labels', this work.
The danger is to no capture a typo in an array name. But it's very
generic.

[jnothman]

Naming labels for the CV generator is fine. Getting GridSearchCV to pass
labels onto the CV generator's .split is another matter.

On 27 February 2014 21:21, Gael Varoquaux notifications@github.com wrote:

We still have a question of how to handle the labels parameter for LOLO
in
cross_val_score or GridSearchCV,

KFold.split(arrays)

with specific names on arrays? Thus, if one array is 'labels', this work.
The danger is to no capture a typo in an array name. But it's very
generic.

Reply to this email directly or view it on GitHubhttps://github.com//issues/2904#issuecomment-36228209
.

[GaelVaroquaux]

Naming labels for the CV generator is fine. Getting GridSearchCV to pass
labels onto the CV generator's .split is another matter.

Yes, that's part of the bigger picture problem. You are right that they
need to be tackled together.

@ogrisel and myself have been brainstorming on the idea of allowing y to
be a dictionnary of arrays or a pandas dataframe.

[mblondel]

Please share / elaborate the ideas developed :)

[GaelVaroquaux]

Please share / elaborate the ideas developed :)

Nothing really much more than what I said above. y could be a panda data
frame or a dict of arrays (we would still accept a simple array). These
arrays would all be of length n_samples, and would be sliced and diced
during cross-validation.

They would be useful to add any meta information that describes samples,
such as sample weights, labels for stratification. Open questions are:
how to deal with multi output or multi-label?