[MRG+1] Learning curve: Add an option to randomly choose indices for different training sizes

[NarineK]

Currently, training sizes are chosen sequentially from 0 to n_train_samples:
train[:n_train_samples]

If training data is sorted by the target variable, for small sizes of training data it will choose only one label and this will always lead to failures during model fitting.
For example:

X = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21]], np.int32)
y = np.array(['a', 'a', 'a', 'a', 'b', 'b', 'b']) 
import numpy as np
from sklearn.learning_curve import learning_curve
from sklearn.svm import SVC

The following always fails with: ValueError: The number of classes has to be greater than one; got 1
train_sizes, train_scores, valid_scores = learning_curve(SVC(kernel='linear'), X, y, train_sizes=[0.7, 1.0], cv=3)

The following runs successfully for most tries:

train_sizes, train_scores, valid_scores = learning_curve(SVC(kernel='linear'), X, y, train_sizes=[0.7, 1.0], cv=3, shuffle=True)

If we would have an option to shuffle the indices of training data before choosing ’n_train_samples’, that will increase our chances of not fitting data with the same label into the learner and have more label variety.

In the following pull request I did a small modification and added an option to shuffle and choose randomly the indices. We could do the same also for incremental learning.

Let me know what you think.

Thanks!

---

This change is

[amueller]

Why not just pass a cv with shuffle? Like cv=KFold(5, shuffle=True)?

[amueller]

Why not just pass a cv with shuffle? Like cv=KFold(5, shuffle=True)?

[NarineK]

Thank you, @amueller for the prompt response.
For my specific use case I need to use LabelKFold.
But it seems that there is no support for shuffle.

from sklearn.cross_validation import LabelKFold
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([1, 2, 3, 4])
labels = np.array([0, 0, 2, 2])
label_kfold = LabelKFold(labels, n_folds=2, shuffle=True)
Traceback (most recent call last):
File "", line 1, in
TypeError: init() got an unexpected keyword argument 'shuffle'

[NarineK]

Thank you, @amueller for the prompt response.
For my specific use case I need to use LabelKFold.
But it seems that there is no support for shuffle.

from sklearn.cross_validation import LabelKFold
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([1, 2, 3, 4])
labels = np.array([0, 0, 2, 2])
label_kfold = LabelKFold(labels, n_folds=2, shuffle=True)
Traceback (most recent call last):
File "", line 1, in
TypeError: init() got an unexpected keyword argument 'shuffle'

[amueller]

You are right, there's no shuffle in LabelKFold (which was just renamed GroupKFold).
I think a better solution would be to add optional shuffling to the cross-validation generator.

[amueller]

You are right, there's no shuffle in LabelKFold (which was just renamed GroupKFold).
I think a better solution would be to add optional shuffling to the cross-validation generator.

[NarineK]

I see, do you mean adding shuffle=False here ?
https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/cross_validation.py#L397
Thank you!

[NarineK]

I see, do you mean adding shuffle=False here ?
https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/cross_validation.py#L397
Thank you!

[jnothman]

I was going to say that I don't mind adding a shuffle parameter, but I think I'm coming to agree with @amueller.

(Also, the sklearn.learning_curve module is deprecated. See model_selection)

[jnothman]

I was going to say that I don't mind adding a shuffle parameter, but I think I'm coming to agree with @amueller.

(Also, the sklearn.learning_curve module is deprecated. See model_selection)

[jnothman]

Yes, adding shuffling to the equivalent in model_selection/_split.py

On 29 September 2016 at 07:51, NarineK notifications@github.com wrote:

I see, do you mean adding shuffle=False here ?
https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/cross_
validation.py#L397
Thank you!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#7506 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz6yDNFb3JJA48DDhWLT9FL6KDxu3gks5quuFvgaJpZM4KIS7h
.

[jnothman]

Yes, adding shuffling to the equivalent in model_selection/_split.py

On 29 September 2016 at 07:51, NarineK notifications@github.com wrote:

I see, do you mean adding shuffle=False here ?
https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/cross_
validation.py#L397
Thank you!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#7506 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz6yDNFb3JJA48DDhWLT9FL6KDxu3gks5quuFvgaJpZM4KIS7h
.

[NarineK]

Thank you @amueller and @jnothman!

I see GroupKFold in model_selection/_split.py.
Let me give a try and add shuffle there.

[NarineK]

Thank you @amueller and @jnothman!

I see GroupKFold in model_selection/_split.py.
Let me give a try and add shuffle there.

[amueller]

Great, thanks @NarineK :)

[amueller]

Great, thanks @NarineK :)

[jnothman]

I hope it's okay to start a new PR, thanks @NarineK .

[jnothman]

I hope it's okay to start a new PR, thanks @NarineK .

[jnothman]

I think what you need here is actually a stratify option in learning_curve.

[jnothman]

I think what you need here is actually a stratify option in learning_curve.

[amueller]

@jnothman or leave the subsetting to the cv object and use StratifiedShuffleSplit? Hm that actually conflates the subsetting and the cross-validation somewhat...

What would a stratified learning curve look like? Make sure that for each n_samples amount the data is stratified?

And do we then also add a group option to learning curve? Maybe shuffle is a good enough fix?

[amueller]

@jnothman or leave the subsetting to the cv object and use StratifiedShuffleSplit? Hm that actually conflates the subsetting and the cross-validation somewhat...

What would a stratified learning curve look like? Make sure that for each n_samples amount the data is stratified?

And do we then also add a group option to learning curve? Maybe shuffle is a good enough fix?

[amueller]

Thinking about it more, I feel that the way we are doing the learning curves feels weird. @agramfort do you have literature on cross-validation with learning curves?

[amueller]

Thinking about it more, I feel that the way we are doing the learning curves feels weird. @agramfort do you have literature on cross-validation with learning curves?

[jnothman]

We don't need to add a group option: the nice thing about the learning curve implementation is that any questions of dependency between training and test samples are handled by cv. The only question is then how to sample from the training data. I'm coming to the idea that shuffling it is not so bad. It will mean that there aren't ordering-dependent anomalies in the learning curve.

[jnothman]

We don't need to add a group option: the nice thing about the learning curve implementation is that any questions of dependency between training and test samples are handled by cv. The only question is then how to sample from the training data. I'm coming to the idea that shuffling it is not so bad. It will mean that there aren't ordering-dependent anomalies in the learning curve.

[NarineK]

I prefer this option too. Should leaning_curve in that case have 2 additional arguments ? random_state=None and shuffle=False ?

[NarineK]

I prefer this option too. Should leaning_curve in that case have 2 additional arguments ? random_state=None and shuffle=False ?

[amueller]

@jnothman that's true about correlations between the training and the test set, but that might make for very strange learning curves. I guess we can do the shuffle here. I'm not entirely convinced by the CV based approach but I guess it's too late to change anyhow.

[amueller]

@jnothman that's true about correlations between the training and the test set, but that might make for very strange learning curves. I guess we can do the shuffle here. I'm not entirely convinced by the CV based approach but I guess it's too late to change anyhow.

[NarineK]

I assume the change is needed in sklearn/model_selection/_validation.py instead of sklearn/learning_curve.py
https://github.com/NarineK/scikit-learn/blob/1263e5acb1b9e729fdd740299a1bf5fe73a6c618/sklearn/model_selection/_validation.py#L652
Or maybe in both places ?

[NarineK]

I assume the change is needed in sklearn/model_selection/_validation.py instead of sklearn/learning_curve.py
https://github.com/NarineK/scikit-learn/blob/1263e5acb1b9e729fdd740299a1bf5fe73a6c618/sklearn/model_selection/_validation.py#L652
Or maybe in both places ?

[amueller]

Only the model_selection, I think.

[amueller]

Only the model_selection, I think.

[jnothman]

the overall patch is now nothing...?

[jnothman]

the overall patch is now nothing...?

[NarineK]

I merged to master to fix the conflicts. I'll push my changes in model selection soon.

[NarineK]

I merged to master to fix the conflicts. I'll push my changes in model selection soon.

[NarineK]

It seems that incremental training has the implementation in a separate method:

https://github.com/NarineK/scikit-learn/blob/61f66bb7d6158fd507f30a5a1a02e555e86bdc26/sklearn/model_selection/_validation.py#L847

We could shuffle it there too.

[NarineK]

It seems that incremental training has the implementation in a separate method:

https://github.com/NarineK/scikit-learn/blob/61f66bb7d6158fd507f30a5a1a02e555e86bdc26/sklearn/model_selection/_validation.py#L847

We could shuffle it there too.

[NarineK]

I could have implemented it the following way:

        if shuffle:
            rng = check_random_state(random_state)
            train_test_proportions = [rng.permutation(train)[:n_train_samples], test)
                for train, test in cv for n_train_samples in train_sizes_abs]
        else:
            train_test_proportions = [(train[:n_train_samples], test)
                for train, test in cv for n_train_samples in train_sizes_abs]

but in this case line: for train, test in cv for n_train_samples in train_sizes_abs] is duplicated.

[jnothman]

There are two strategies:

1. shuffle once and take prefixes
2. shuffle for each training set

Option 1. could result in smoother learning curves. Is there any benefit to 2.?

Ideally it should be possible to tell which strategy is being taken from the docstring.

[jnothman]

There are two strategies:

1. shuffle once and take prefixes
2. shuffle for each training set

Option 1. could result in smoother learning curves. Is there any benefit to 2.?

Ideally it should be possible to tell which strategy is being taken from the docstring.

[jnothman]

Yes, you need to handle the incremental case too.

On 10 October 2016 at 12:40, NarineK notifications@github.com wrote:

@NarineK commented on this pull request.

In sklearn/model_selection/_validation.py
#7506 (review)
:

@@ -765,11 +773,20 @@ def learning_curve(estimator, X, y, groups=None,
clone(estimator), X, y, classes, train, test, train_sizes_abs,
scorer, verbose) for train, test in cv.split(X, y, groups))
else:

•    if shuffle:
  

•        rng = check_random_state(random_state)
  

•    train_test_proportions = []
  

•    for train, test in cv_iter:
  

•        for n_train_samples in train_sizes_abs:
  

•            if shuffle:
  

•                train = rng.permutation(train)
  

•            train_test_proportions.append((train[:n_train_samples], test))
  

I could have implemented it the following way:

    if shuffle:
        rng = check_random_state(random_state)
        train_test_proportions = [rng.permutation(train)[:n_train_samples], test)
            for train, test in cv for n_train_samples in train_sizes_abs]
    else:
        train_test_proportions = [(train[:n_train_samples], test)
            for train, test in cv for n_train_samples in train_sizes_abs]

but in this case line: for train, test in cv for n_train_samples in
train_sizes_abs] is duplicated.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7506 (review),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz63H8iRH8iTNhEBlIyMyNrp0uWdn2ks5qyZehgaJpZM4KIS7h
.

[jnothman]

Yes, you need to handle the incremental case too.

On 10 October 2016 at 12:40, NarineK notifications@github.com wrote:

@NarineK commented on this pull request.

In sklearn/model_selection/_validation.py
#7506 (review)
:

@@ -765,11 +773,20 @@ def learning_curve(estimator, X, y, groups=None,
clone(estimator), X, y, classes, train, test, train_sizes_abs,
scorer, verbose) for train, test in cv.split(X, y, groups))
else:

•    if shuffle:
  

•        rng = check_random_state(random_state)
  

•    train_test_proportions = []
  

•    for train, test in cv_iter:
  

•        for n_train_samples in train_sizes_abs:
  

•            if shuffle:
  

•                train = rng.permutation(train)
  

•            train_test_proportions.append((train[:n_train_samples], test))
  

I could have implemented it the following way:

    if shuffle:
        rng = check_random_state(random_state)
        train_test_proportions = [rng.permutation(train)[:n_train_samples], test)
            for train, test in cv for n_train_samples in train_sizes_abs]
    else:
        train_test_proportions = [(train[:n_train_samples], test)
            for train, test in cv for n_train_samples in train_sizes_abs]

but in this case line: for train, test in cv for n_train_samples in
train_sizes_abs] is duplicated.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7506 (review),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz63H8iRH8iTNhEBlIyMyNrp0uWdn2ks5qyZehgaJpZM4KIS7h
.

[amueller]

well method 2 is closer to what we actually want to estimate...

[amueller]

well method 2 is closer to what we actually want to estimate...

[NarineK]

I think both of them could work.
For option 2 we have the overhead of shuffling for each training size in every (train, test) pair.

While for option 1 we shuffle once per (train, test) pair and reuse it for multiple sizes.

That's how I understand @jnothman's comment. I've made the change after his comment.

[NarineK]

I think both of them could work.
For option 2 we have the overhead of shuffling for each training size in every (train, test) pair.

While for option 1 we shuffle once per (train, test) pair and reuse it for multiple sizes.

That's how I understand @jnothman's comment. I've made the change after his comment.

[jnothman]

You've not currently tested that shuffle has any effect, I take it? Can we engineer a case like yours where not shuffling in a group k-fold case would train a bad model?

[jnothman]

*training

[jnothman]

Can we please pull the generator out, perhaps into a separate function, to be shared by the incremental and non-incremental cases?

[jnothman]

I suspect you could simplify that case further, but I've not looked at it in detail.

[jnothman]

I suspect you could simplify that case further, but I've not looked at it in detail.

[NarineK]

Sure, where should I add that test case?
Somewhere here:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_selection/tests/test_validation.py#L575 or in test_split.py ?

[NarineK]

Sure, where should I add that test case?
Somewhere here:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_selection/tests/test_validation.py#L575 or in test_split.py ?

[jnothman]

test_validation, please

On 14 October 2016 at 07:53, NarineK notifications@github.com wrote:

Sure, where should I add that test case?
Somewhere here:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_
selection/tests/test_validation.py#L575 or in test_split.py ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7506 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz63WdKB1FUNK-JsHerX6-PkNxjk2Jks5qzpowgaJpZM4KIS7h
.

[jnothman]

test_validation, please

On 14 October 2016 at 07:53, NarineK notifications@github.com wrote:

Sure, where should I add that test case?
Somewhere here:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_
selection/tests/test_validation.py#L575 or in test_split.py ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7506 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz63WdKB1FUNK-JsHerX6-PkNxjk2Jks5qzpowgaJpZM4KIS7h
.

[amueller]

Cross-validation should then give a measure of the robustness to different sampling approaches.

I don't understand that point. You're saying approach 1 is measuring the increase in performance conditioned on the sample we saw, and approach 2 is trying to estimate performance as a function of training set size, right? I would agree. But I'm not sure what the cross-validation means in the context of 1.

Anyhow, I don't feel strongly either way.

[amueller]

Cross-validation should then give a measure of the robustness to different sampling approaches.

I don't understand that point. You're saying approach 1 is measuring the increase in performance conditioned on the sample we saw, and approach 2 is trying to estimate performance as a function of training set size, right? I would agree. But I'm not sure what the cross-validation means in the context of 1.

Anyhow, I don't feel strongly either way.

[jnothman]

I think you want double-backticks

[jnothman]

"using it based on" could be "taking prefixes of it based on"

[jnothman]

Since this test does nothing but check that shuffle=True gives results akin to shuffle=False (rather than ensuring that it's shuffled), I think you should just add for shuffle in [True, False] to tests above, rather than create a new smoke test.

[jnothman]

If this also fails at master with an estimator supporting partial_fit, such as SGDClassifier, then this test can incorporate tests for incremental and non-incremental approaches. I would rather see both tested.

[NarineK]

Hi @jnothman,
specifically, SGDClassifier with incremental training does not fail without shuffle, but I believe that with shuffle it trains better model (just from looking at the scores):
Without shuffle:

>>> train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, exploit_incremental_learning=True)
>>> train_scores
array([[ 1.        ,  1.        ],
       [ 0.2       ,  0.2       ],
       [ 0.22222222,  0.16666667]])

With shuffle:

>>> train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, shuffle=True, random_state=2, exploit_incremental_learning=True)
>>> train_scores
array([[ 0.5       ,  0.5       ],
       [ 0.4       ,  0.2       ],
       [ 0.22222222,  0.5       ]])

[jnothman]

Actually I think the refactoring that I want looks more like:

if shuffle:
    rng = check_random_state(random_state) 
    cv_iter = ((rng.permutation(train), test) for train, test in cv_iter)

[jnothman]

Add a note here about why the test is designed this way, or perhaps just reference the issue number.

[jnothman]

Even just making clear that you assert in the test that the shuffle=False case breaks will help the maintainer.

[jnothman]

I'd rather the contrast between failure and success. Why are you looking at
train_scores rather than test_scores?

On 16 October 2016 at 11:25, NarineK notifications@github.com wrote:

@NarineK commented on this pull request.

In sklearn/model_selection/tests/test_validation.py
#7506:

@@ -713,6 +738,25 @@ def test_learning_curve_with_boolean_indices():
np.linspace(0.1, 1.0, 10))

+def test_learning_curve_with_shuffle():

• X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [11, 12], [13, 14], [15, 16],

•             [17, 18], [19, 20], [7, 8], [9, 10], [11, 12], [13, 14],
  

•             [15, 16], [17, 18]])
  

• y = np.array([1, 1, 1, 2, 3, 4, 1, 1, 2, 3, 4, 1, 2, 3, 4])
• groups = np.array([1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 4, 4, 4, 4])
• estimator = LogisticRegression(multi_class='multinomial', penalty='l2',

Hi @jnothman https://github.com/jnothman,
specifically, SGDClassifier with incremental training does not fail
without shuffle, but I believe that with shuffle it trains better model
(just from looking at the scores):
Without shuffle:

train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, exploit_incremental_learning=True)
train_scores
array([[ 1. , 1. ],
[ 0.2 , 0.2 ],
[ 0.22222222, 0.16666667]])

With shuffle:

train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, shuffle=True, random_state=2, exploit_incremental_learning=True)
train_scores
array([[ 0.5 , 0.5 ],
[ 0.4 , 0.2 ],
[ 0.22222222, 0.5 ]])

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7506, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAEz68Y76C7mxZENiA23RHTNVE-mjhujks5q0W8IgaJpZM4KIS7h
.

[jnothman]

I'd rather the contrast between failure and success. Why are you looking at
train_scores rather than test_scores?

On 16 October 2016 at 11:25, NarineK notifications@github.com wrote:

@NarineK commented on this pull request.

In sklearn/model_selection/tests/test_validation.py
#7506:

@@ -713,6 +738,25 @@ def test_learning_curve_with_boolean_indices():
np.linspace(0.1, 1.0, 10))

+def test_learning_curve_with_shuffle():

• X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [11, 12], [13, 14], [15, 16],

•             [17, 18], [19, 20], [7, 8], [9, 10], [11, 12], [13, 14],
  

•             [15, 16], [17, 18]])
  

• y = np.array([1, 1, 1, 2, 3, 4, 1, 1, 2, 3, 4, 1, 2, 3, 4])
• groups = np.array([1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 4, 4, 4, 4])
• estimator = LogisticRegression(multi_class='multinomial', penalty='l2',

Hi @jnothman https://github.com/jnothman,
specifically, SGDClassifier with incremental training does not fail
without shuffle, but I believe that with shuffle it trains better model
(just from looking at the scores):
Without shuffle:

train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, exploit_incremental_learning=True)
train_scores
array([[ 1. , 1. ],
[ 0.2 , 0.2 ],
[ 0.22222222, 0.16666667]])

With shuffle:

train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, shuffle=True, random_state=2, exploit_incremental_learning=True)
train_scores
array([[ 0.5 , 0.5 ],
[ 0.4 , 0.2 ],
[ 0.22222222, 0.5 ]])

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7506, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAEz68Y76C7mxZENiA23RHTNVE-mjhujks5q0W8IgaJpZM4KIS7h
.

[NarineK]

I'll add the test score too.

[NarineK]

I'll add the test score too.

[NarineK]

SGDClassifier itself is giving non deterministic scores. It is hard to write test cases for it.
but MultinomialNB is maybe a better option ?

[NarineK]

SGDClassifier itself is giving non deterministic scores. It is hard to write test cases for it.
but MultinomialNB is maybe a better option ?

[NarineK]

MultinomialNB doesn't fail if I set all labels the same. I'm not sure if this is by design, but it is not consistent with other algorithms. The output isn't helpful either.

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [11, 12], [13, 14],[15, 16],[17, 18],[19, 20],[7, 8], [9, 10], [11, 12], [13, 14],[15, 16],[17, 18]])
y= np.array([1,1,1,1,1,1,1,1,1,1,1,1,1,1,1])
groups = np.array([1,1,1,1,1,1,3,3,3,3,3,4,4,4,4])
train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3, 1.0, 3),groups=groups, shuffle=True, exploit_incremental_learning=True)
>>> train_scores
array([[ 1.,  1.],
       [ 1.,  1.],
       [ 1.,  1.]])
>>> test_scores
array([[ 1.,  1.],
       [ 1.,  1.],
       [ 1.,  1.]])

[NarineK]

MultinomialNB doesn't fail if I set all labels the same. I'm not sure if this is by design, but it is not consistent with other algorithms. The output isn't helpful either.

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [11, 12], [13, 14],[15, 16],[17, 18],[19, 20],[7, 8], [9, 10], [11, 12], [13, 14],[15, 16],[17, 18]])
y= np.array([1,1,1,1,1,1,1,1,1,1,1,1,1,1,1])
groups = np.array([1,1,1,1,1,1,3,3,3,3,3,4,4,4,4])
train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3, 1.0, 3),groups=groups, shuffle=True, exploit_incremental_learning=True)
>>> train_scores
array([[ 1.,  1.],
       [ 1.,  1.],
       [ 1.,  1.]])
>>> test_scores
array([[ 1.,  1.],
       [ 1.,  1.],
       [ 1.,  1.]])

[NarineK]

I couldn't find an estimator for which (shuffle=False and exploit_incremental_learning=True) would fail. Tried:

        sklearn.naive_bayes.MultinomialNB
        sklearn.linear_model.SGDClassifier
        sklearn.linear_model.PassiveAggressiveClassifier

[NarineK]

I couldn't find an estimator for which (shuffle=False and exploit_incremental_learning=True) would fail. Tried:

        sklearn.naive_bayes.MultinomialNB
        sklearn.linear_model.SGDClassifier
        sklearn.linear_model.PassiveAggressiveClassifier

[jnothman]

I've realised why: it gets the list of classes from the whole dataset. So performing the test, despite not getting an error in master, is sufficient.

We still have an underlying problem that the metrics are being calculated incorrectly (assuming fewer classes than there should be), but that's a somewhat different issue, closely related to #6231.

[jnothman]

I've realised why: it gets the list of classes from the whole dataset. So performing the test, despite not getting an error in master, is sufficient.

We still have an underlying problem that the metrics are being calculated incorrectly (assuming fewer classes than there should be), but that's a somewhat different issue, closely related to #6231.

[jnothman]

Otherwise LGTM. Please add an entry to what's new (under 0.19/enhancements)

[jnothman]

I think we had a little misunderstanding in the creation of this function. Can you please put it back inline? Thanks.

[jnothman]

LGTM

[jnothman]

this should be

:func:`model_selection.learning_curve`

[jnothman]

need to add link target at bottom of file

[NarineK]

Added the modifications in doc/whats_new.rst
Thanks for reviewing, @jnothman

[NarineK]

Added the modifications in doc/whats_new.rst
Thanks for reviewing, @jnothman

[amueller]

Please use a comment, not a docstring for the test - that makes it easier to find out which test is run.
Also, I'm not sure I understand the test. Can you please add an explanation here?

[amueller]

After reading the discussion again, the point of the test is that it would fail without shuffling, because the first split doesn't contain label 4. Can you please just add that here?

[amueller]

LGTM apart from explaining the test.

[amueller]

After reading the discussion again, the point of the test is that it would fail without shuffling, because the first split doesn't contain label 4. Can you please just add that here?

[amueller]

Any reason to use the mean here instead of everything?

[NarineK]

Thank you for the review, @amueller
I used mean instead of everything in order to be consistent with other test cases for learning curves.

[amueller]

thanks @NarineK

[amueller]

thanks @NarineK

[NarineK]

oh, I haven't addressed this point yet, @amueller

After reading the discussion again, the point of the test is that it would fail without shuffling, because the first split doesn't contain label 4. Can you please just add that here?

Is it too late now? I see you merged it.

[NarineK]

oh, I haven't addressed this point yet, @amueller

After reading the discussion again, the point of the test is that it would fail without shuffling, because the first split doesn't contain label 4. Can you please just add that here?

Is it too late now? I see you merged it.

[amueller]

Damn too quick. I'll add the comment in master. I'm right about the intend, though?

[amueller]

Damn too quick. I'll add the comment in master. I'm right about the intend, though?

[NarineK]

yes, you're right. Thank you.

[NarineK]

yes, you're right. Thank you.

[jnothman]

Either way thanks, @NarineK, for raising the issue and solving it even when we told you to solve it the wrong way at first!

[jnothman]

Either way thanks, @NarineK, for raising the issue and solving it even when we told you to solve it the wrong way at first!

[NarineK]

No problem, my pleasure!

[NarineK]

No problem, my pleasure!