Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MRG+1] Learning curve: Add an option to randomly choose indices for different training sizes #7506

Merged
merged 21 commits into from Oct 19, 2016
Merged

[MRG+1] Learning curve: Add an option to randomly choose indices for different training sizes #7506

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
43ebd81
Chooses randomly the indices for different training sizes
Sep 28, 2016
660db34
Bring back deleted line
Sep 28, 2016
9351553
Rewrote the description of 'shuffle' attribute
Sep 28, 2016
5839808
use random.sample instead of np.random.choice
Sep 28, 2016
1263e5a
replace tabs with spaces
Sep 28, 2016
2cca140
merge to master
Oct 9, 2016
fa40520
merge to master
Oct 9, 2016
abdcc3f
Added shuffle in model-selection's learning_curve method
Oct 10, 2016
cc9b42d
Added shuffle for incremental learning + addressed Joel's comment
Oct 10, 2016
8cae11e
Shorten long lines
Oct 10, 2016
20209e9
Add 2 blank spaces between test cases
Oct 10, 2016
d0ac1ed
Addressed Joel's review comments
Oct 12, 2016
9cc8ad8
Added 2 blank lines between methods
Oct 12, 2016
210a7bb
Added non regression test for learning_curve with shuffle
Oct 15, 2016
60b1531
Fixed indentions
Oct 15, 2016
b2d28c3
Fixed space issues
Oct 15, 2016
a6a63e2
Modified test cases + small code improvements
Oct 16, 2016
7ed713d
Fix some style issues
Oct 16, 2016
1ea3a6e
Addressed Joel's comments - removed _shuffle_train_indices, more test…
Oct 19, 2016
b6af4d1
Merged to master
Oct 19, 2016
c11e29e
Added some modifications in whats_new.rst
Oct 19, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
38 changes: 31 additions & 7 deletions sklearn/model_selection/_validation.py
View file
Open in desktop
Expand Up @@ -642,7 +642,8 @@ def _shuffle(y, groups, random_state):
def learning_curve(estimator, X, y, groups=None,
train_sizes=np.linspace(0.1, 1.0, 5), cv=None, scoring=None,
exploit_incremental_learning=False, n_jobs=1,
pre_dispatch="all", verbose=0):
pre_dispatch="all", verbose=0, shuffle=False,
random_state=None):
"""Learning curve.

Determines cross-validated training and test scores for different training
Expand Down Expand Up @@ -718,7 +719,14 @@ def learning_curve(estimator, X, y, groups=None,
verbose : integer, optional
Controls the verbosity: the higher, the more messages.

Returns
shuffle : boolean, optional
Whether to shuffle training data before taking prefixes of it
based on``train_sizes``.

random_state : None, int or RandomState
When shuffle=True, pseudo-random number generator state used for
shuffling. If None, use default numpy RNG for shuffling.

-------
train_sizes_abs : array, shape = (n_unique_ticks,), dtype int
Numbers of training examples that has been used to generate the
Expand Down Expand Up @@ -759,17 +767,25 @@ def learning_curve(estimator, X, y, groups=None,

parallel = Parallel(n_jobs=n_jobs, pre_dispatch=pre_dispatch,
verbose=verbose)

cv_iter = _shuffle_train_indices(cv_iter, shuffle, random_state)

if exploit_incremental_learning:
classes = np.unique(y) if is_classifier(estimator) else None
out = parallel(delayed(_incremental_fit_estimator)(
clone(estimator), X, y, classes, train, test, train_sizes_abs,
scorer, verbose) for train, test in cv.split(X, y, groups))
clone(estimator), X, y, classes, train,
test, train_sizes_abs, scorer, verbose)
for train, test in cv_iter)
else:
train_test_proportions = []
for train, test in cv_iter:
for n_train_samples in train_sizes_abs:
train_test_proportions.append((train[:n_train_samples], test))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could have implemented it the following way:


        if shuffle:
            rng = check_random_state(random_state)
            train_test_proportions = [rng.permutation(train)[:n_train_samples], test)
                for train, test in cv for n_train_samples in train_sizes_abs]
        else:
            train_test_proportions = [(train[:n_train_samples], test)
                for train, test in cv for n_train_samples in train_sizes_abs]

but in this case line: for train, test in cv for n_train_samples in train_sizes_abs] is duplicated.

out = parallel(delayed(_fit_and_score)(
clone(estimator), X, y, scorer, train[:n_train_samples], test,
clone(estimator), X, y, scorer, train, test,
verbose, parameters=None, fit_params=None, return_train_score=True)
for train, test in cv_iter
for n_train_samples in train_sizes_abs)
for train, test in train_test_proportions)
out = np.array(out)
n_cv_folds = out.shape[0] // n_unique_ticks
out = out.reshape(n_cv_folds, n_unique_ticks, 2)
Expand All @@ -779,6 +795,14 @@ def learning_curve(estimator, X, y, groups=None,
return train_sizes_abs, out[0], out[1]


def _shuffle_train_indices(cv_iter, shuffle, random_state):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we had a little misunderstanding in the creation of this function. Can you please put it back inline? Thanks.

"""Shuffle training indices if ``shuffle=True``. """
if shuffle:
rng = check_random_state(random_state)
cv_iter = ((rng.permutation(train), test) for train, test in cv_iter)
return cv_iter


def _translate_train_sizes(train_sizes, n_max_training_samples):
"""Determine absolute sizes of training subsets and validate 'train_sizes'.

Expand Down
66 changes: 46 additions & 20 deletions sklearn/model_selection/tests/test_validation.py
View file
Open in desktop
Expand Up @@ -560,18 +560,20 @@ def test_learning_curve():
n_redundant=0, n_classes=2,
n_clusters_per_class=1, random_state=0)
estimator = MockImprovingEstimator(20)
with warnings.catch_warnings(record=True) as w:
train_sizes, train_scores, test_scores = learning_curve(
estimator, X, y, cv=3, train_sizes=np.linspace(0.1, 1.0, 10))
if len(w) > 0:
raise RuntimeError("Unexpected warning: %r" % w[0].message)
assert_equal(train_scores.shape, (10, 3))
assert_equal(test_scores.shape, (10, 3))
assert_array_equal(train_sizes, np.linspace(2, 20, 10))
assert_array_almost_equal(train_scores.mean(axis=1),
np.linspace(1.9, 1.0, 10))
assert_array_almost_equal(test_scores.mean(axis=1),
np.linspace(0.1, 1.0, 10))
for shuffle_train in [False, True]:
with warnings.catch_warnings(record=True) as w:
train_sizes, train_scores, test_scores = learning_curve(
estimator, X, y, cv=3, train_sizes=np.linspace(0.1, 1.0, 10),
shuffle=shuffle_train)
if len(w) > 0:
raise RuntimeError("Unexpected warning: %r" % w[0].message)
assert_equal(train_scores.shape, (10, 3))
assert_equal(test_scores.shape, (10, 3))
assert_array_equal(train_sizes, np.linspace(2, 20, 10))
assert_array_almost_equal(train_scores.mean(axis=1),
np.linspace(1.9, 1.0, 10))
assert_array_almost_equal(test_scores.mean(axis=1),
np.linspace(0.1, 1.0, 10))


def test_learning_curve_unsupervised():
Expand Down Expand Up @@ -622,14 +624,15 @@ def test_learning_curve_incremental_learning():
n_redundant=0, n_classes=2,
n_clusters_per_class=1, random_state=0)
estimator = MockIncrementalImprovingEstimator(20)
train_sizes, train_scores, test_scores = learning_curve(
estimator, X, y, cv=3, exploit_incremental_learning=True,
train_sizes=np.linspace(0.1, 1.0, 10))
assert_array_equal(train_sizes, np.linspace(2, 20, 10))
assert_array_almost_equal(train_scores.mean(axis=1),
np.linspace(1.9, 1.0, 10))
assert_array_almost_equal(test_scores.mean(axis=1),
np.linspace(0.1, 1.0, 10))
for shuffle_train in [False, True]:
train_sizes, train_scores, test_scores = learning_curve(
estimator, X, y, cv=3, exploit_incremental_learning=True,
train_sizes=np.linspace(0.1, 1.0, 10), shuffle=shuffle_train)
assert_array_equal(train_sizes, np.linspace(2, 20, 10))
assert_array_almost_equal(train_scores.mean(axis=1),
np.linspace(1.9, 1.0, 10))
assert_array_almost_equal(test_scores.mean(axis=1),
np.linspace(0.1, 1.0, 10))


def test_learning_curve_incremental_learning_unsupervised():
Expand Down Expand Up @@ -713,6 +716,29 @@ def test_learning_curve_with_boolean_indices():
np.linspace(0.1, 1.0, 10))


def test_learning_curve_with_shuffle():
"""Following test case was designed this way to verify the code
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use a comment, not a docstring for the test - that makes it easier to find out which test is run.
Also, I'm not sure I understand the test. Can you please add an explanation here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reading the discussion again, the point of the test is that it would fail without shuffling, because the first split doesn't contain label 4. Can you please just add that here?

changes made in pull request: #7506."""
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [11, 12], [13, 14], [15, 16],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a note here about why the test is designed this way, or perhaps just reference the issue number.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even just making clear that you assert in the test that the shuffle=False case breaks will help the maintainer.

[17, 18], [19, 20], [7, 8], [9, 10], [11, 12], [13, 14],
[15, 16], [17, 18]])
y = np.array([1, 1, 1, 2, 3, 4, 1, 1, 2, 3, 4, 1, 2, 3, 4])
groups = np.array([1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 4, 4, 4, 4])
estimator = LogisticRegression(multi_class='multinomial', penalty='l2',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this also fails at master with an estimator supporting partial_fit, such as SGDClassifier, then this test can incorporate tests for incremental and non-incremental approaches. I would rather see both tested.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jnothman,
specifically, SGDClassifier with incremental training does not fail without shuffle, but I believe that with shuffle it trains better model (just from looking at the scores):
Without shuffle:

>>> train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, exploit_incremental_learning=True)
>>> train_scores
array([[ 1.        ,  1.        ],
       [ 0.2       ,  0.2       ],
       [ 0.22222222,  0.16666667]])

With shuffle:

>>> train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3,1.0,3), groups=groups, verbose=1, shuffle=True, random_state=2, exploit_incremental_learning=True)
>>> train_scores
array([[ 0.5       ,  0.5       ],
       [ 0.4       ,  0.2       ],
       [ 0.22222222,  0.5       ]])

solver='lbfgs')

cv = GroupKFold(n_splits=2)
train_sizes, train_scores, test_scores = learning_curve(
estimator, X, y, cv=cv, n_jobs=1, train_sizes=np.linspace(0.3, 1.0, 3),
groups=groups, shuffle=True, random_state=2)
assert_array_almost_equal(train_scores.mean(axis=1),
np.array([1., 0.9, 0.72222222]))
assert_array_almost_equal(test_scores.mean(axis=1),
np.array([0.19444444, 0.05555556, 0.13888889]))
assert_raises(ValueError, learning_curve, estimator, X, y, cv=cv, n_jobs=1,
train_sizes=np.linspace(0.3, 1.0, 3), groups=groups)


def test_validation_curve():
X, y = make_classification(n_samples=2, n_features=1, n_informative=1,
n_redundant=0, n_classes=2,
Expand Down