[MRG+2] Add a test for sample weights for estimators #11558

sergulaydore · 2018-07-16T15:03:42Z

Reference Issues/PRs

Fixes the invariant test for sample weights as mentioned in issue #11316 (Refactor tests for sample weights).

What is new?

This is a generic test for estimators that makes sure the sample weights yield consistent results.

What does this implement/fix? Explain your changes.

The test checks if the output of the estimators are the same for sample_weight = None and sample_weight=np.ones().

Any other comments?

Pairwise methods are skipped as they require pairwise data.

GaelVaroquaux

Looks good, aside from 2 minor comments.

GaelVaroquaux · 2018-07-16T15:32:58Z

sklearn/utils/estimator_checks.py

+        X = np.array([[1, 3], [1, 3], [1, 3], [1, 3],
+                      [2, 1], [2, 1], [2, 1], [2, 1],
+                      [3, 3], [3, 3], [3, 3], [3, 3],
+                      [4, 1], [4, 1], [4, 1], [4, 1]])


I would force the dtype to be float here.

GaelVaroquaux · 2018-07-16T15:32:58Z

sklearn/utils/estimator_checks.py

+                                 "sample_weight=ones" % name)
+        if hasattr(estimator_orig, "transform"):
+            X_pred1 = estimator1.transform(X)
+            X_pred2 = estimator2.transform(X)


nitpick: I would call these X_trans1 and X_trans2

Changed this back to X_pred1 and X_pred2 because I moved both methods inside a for loop.

GaelVaroquaux · 2018-07-16T15:56:51Z

sklearn/utils/estimator_checks.py

+                      [3, 3], [3, 3], [3, 3], [3, 3],
+                      [4, 1], [4, 1], [4, 1], [4, 1]], dtype=np.dtype('float'))
+        y = np.array([1, 1, 1, 1, 2, 2, 2, 2,
+                      1, 1, 1, 1, 2, 2, 2, 2], dtype=np.dtype('float'))


Actually, I would have kept y as an integer. Only X as float.

glemaitre

I think that it could be great to add an entry in the what's new as well

glemaitre · 2018-07-16T16:54:39Z

sklearn/utils/estimator_checks.py

@@ -553,6 +554,45 @@ def check_sample_weights_list(name, estimator_orig):
        estimator.fit(X, y, sample_weight=sample_weight)


+@ignore_warnings(category=(DeprecationWarning, FutureWarning))
+def check_sample_weight_invariance(name, estimator_orig):


It could be worth to add a one line comment regarding the purpose of the test

glemaitre · 2018-07-16T16:54:39Z

sklearn/utils/estimator_checks.py

+        y = np.array([1, 1, 1, 1, 2, 2, 2, 2,
+                      1, 1, 1, 1, 2, 2, 2, 2], dtype=np.dtype('float'))
+
+        if has_fit_parameter(estimator_orig, "random_state"):


you can replace those using set_random_state from sklearn.utils.testing it will check if the estimators has already the random state

glemaitre · 2018-07-16T16:54:39Z

sklearn/utils/estimator_checks.py

+                      1, 1, 1, 1, 2, 2, 2, 2], dtype=np.dtype('float'))
+
+        if has_fit_parameter(estimator_orig, "random_state"):
+            estimator1.fit(X, y=y, sample_weight=np.ones(shape=len(y)), random_state=0)


so basically:

from sklearn.utils.testing import set_random_state set_random_state(estimator1, random_state=42) set_random_state(estimator2, random_state=42) estimator1.fit(X, y=y, sample_weight=np.ones(shape=len(y))) estimator2.fit(X, y=y, sample_weight=None)

Thanks! Used it as it is.

glemaitre · 2018-07-16T16:54:39Z

sklearn/utils/estimator_checks.py

+            estimator1.fit(X, y=y, sample_weight=np.ones(shape=len(y)))
+            estimator2.fit(X, y=y, sample_weight=None)
+
+        if hasattr(estimator_orig, "predict"):


make a loop here

for method in ('predict', 'transform'): if hasattr(estimator_orig, method): ...

glemaitre · 2018-07-16T16:54:39Z

sklearn/utils/estimator_checks.py

+            X_pred1 = estimator1.predict(X)
+            X_pred2 = estimator2.predict(X)
+            try:
+                assert_allclose(X_pred1, X_pred2, rtol=0.5)


maybe you might want to use assert_allclose_dense_sparse if the output could be sparse.

pass the err_msg to assert_allclose_dense_sparse avoiding the try .. except ... to raise the proper error.

I don't think we expect the output to be sparse.
err_msg is done!

TomDLT · 2018-07-16T17:14:04Z

I did not look into your code, but you might get inspired from the excellent test in #10803.
see here

GaelVaroquaux · 2018-07-17T13:04:17Z

Test are passing! Congratulations.

But you had a PEP8 failure. I fixed it on your branch. We need to wait for the tests to run again.

GaelVaroquaux · 2018-07-17T13:04:44Z

@glemaitre : can you reassess you review. We think that this is good to go.

qinhanmin2014 · 2018-07-17T13:26:09Z

I can't understand why we need to skip "KMeans" and "MiniBatchKMeans". It seems that the test passes locally on my PC with these two classes. Also, at least we have things like v_measure_score to use here?

qinhanmin2014 · 2018-07-17T13:27:17Z

sklearn/utils/estimator_checks.py

+    # unit weights and no weights
+    if (has_fit_parameter(estimator_orig, "sample_weight") and
+            not (hasattr(estimator_orig, "_pairwise")
+                 and estimator_orig._pairwise) and


Also, I'd prefer some comments here to show us what you're doing (why do you skip the test?).

Done! Basically, the data we are testing is not pairwise. Estimator tests with _pariwise fails otherwise.

GaelVaroquaux · 2018-07-17T13:52:38Z

Indeed, I think that it's good to add comments.

I canceled the corresponding travis builds, to save time in the queue.

qinhanmin2014 · 2018-07-17T14:03:39Z

sklearn/utils/estimator_checks.py

+                 and estimator_orig._pairwise) and
+            name not in ["KMeans", "MiniBatchKMeans"]):
+        # We skip pairwise because the data is not pairwise
+        # KMeans and MiniBatchKMeans were unstable; hence skipped.


I'm confused about it. Are they still unstable under a fixed random_state. If so, is there a bug?
(Sorry if I made a mistake :) Go ahead if you have enough confidence)

This is strange. When I initially ran this without using set_random_state, they were failing. After using set_random_state, they seem to be stable. I wonder if estimator.fit(...,random_state=0) is missing something that set_random_state does not. I can include those estimators to the test. @GaelVaroquaux WDYT?

It is weird to pass random_state in the fit. The state should be pass when instantiating an object or by setting the parameter which is the aim of set_random_state.

Regarding KMeans and MiniBatchKMeans, they rely on a random initialization and it might possible that we actually do not pass the random state when passing random state a fit.

sergulaydore · 2018-07-17T14:03:56Z

@qinhanmin2014 : "KMeans" and "MiniBatchKMeans" are skipped because they were unstable.

qinhanmin2014 · 2018-07-17T14:12:05Z

I would expect estimators in scikit-learn to be deterministic with same input and a fixed random_state, or am I wrong?

glemaitre · 2018-07-17T14:26:18Z

sklearn/utils/estimator_checks.py

+                 and estimator_orig._pairwise) and
+            name not in ["KMeans", "MiniBatchKMeans"]):
+        # We skip pairwise because the data is not pairwise
+        # KMeans and MiniBatchKMeans were unstable; hence skipped.


It is weird to pass random_state in the fit. The state should be pass when instantiating an object or by setting the parameter which is the aim of set_random_state.

Regarding KMeans and MiniBatchKMeans, they rely on a random initialization and it might possible that we actually do not pass the random state when passing random state a fit.

glemaitre · 2018-07-17T14:27:27Z

Oh you added back kmeans, cool so this is fine then.
Waiting for the CI to be green.

qinhanmin2014

LGTM if CIs are green. Thanks @sergulaydore

GaelVaroquaux · 2018-07-17T16:59:02Z

OK, travis is green. Merging. Hurray!

jnothman · 2018-07-18T06:08:45Z

Why do we need an rtol as high as 0.5?

jnothman · 2018-07-18T06:09:47Z

This doesn't assert that weighting makes any difference ever, does it?

sergulaydore · 2018-07-18T08:54:11Z

@jnothman Actually we don't need rtol as 0.5. I just tested with the default value and the tests still passed. No, it only makes sure that the no weighting is equal to unit weighting. If you recommend, I can submit another PR with random weights.

qinhanmin2014 · 2018-07-18T10:05:01Z

+1 to use default rtol, @sergulaydore please submit a PR, thanks.

sergulaydore · 2018-07-18T11:07:55Z

Done! Please see #11621.

sergulaydore added 5 commits Jul 16, 2018

test for None and ones for sample_weight added

5f6c614

test for None and ones for sample_weight added

e47a059

skip KMeans based estimators

688be6a

conflict resolved

f0c871f

cleaning

Loading status checks…

3533647

GaelVaroquaux reviewed Jul 16, 2018

View changes

dtype and X_trans

Loading status checks…

0738abc

GaelVaroquaux changed the title ~~Add a test for sample weights for estimators~~ [MRG+1] Add a test for sample weights for estimators Jul 16, 2018

GaelVaroquaux added this to the 0.20 milestone Jul 16, 2018

GaelVaroquaux approved these changes Jul 16, 2018

View changes

glemaitre requested changes Jul 16, 2018

View changes

sergulaydore and others added 5 commits Jul 17, 2018

after second reviews

Loading status checks…

1d93961

pyflake errors

Loading status checks…

6b7f20a

Merge branch 'master' into refactor_sample_weights

Loading status checks…

ad6a365

fixed pairwise error

Loading status checks…

ad50ab9

PEP8: line too long

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

033a84a

qinhanmin2014 reviewed Jul 17, 2018

View changes

add comments and pep8 issues

Loading status checks…

ef0371c

qinhanmin2014 reviewed Jul 17, 2018

View changes

KMeans methods are not skipped anymore

Loading status checks…

1ca44ab

glemaitre requested changes Jul 17, 2018

View changes

glemaitre approved these changes Jul 17, 2018

View changes

qinhanmin2014 approved these changes Jul 17, 2018

View changes

qinhanmin2014 changed the title ~~[MRG+1] Add a test for sample weights for estimators~~ [MRG+2] Add a test for sample weights for estimators Jul 17, 2018

GaelVaroquaux added this to Done in scikit-learn 0.20 via automation Jul 17, 2018

lesteve moved this from Done to PRs tagged in scikit-learn 0.20 Jul 17, 2018

jnothman mentioned this pull request Jul 18, 2018

[WIP] Sample weight consistency #5515

Closed

sergulaydore mentioned this pull request Jul 18, 2018

[MRG+2] TST Using default rtol in test sample_weight invariance #11621

Merged

sergulaydore deleted the sergulaydore:refactor_sample_weights branch Jul 18, 2018

scikit-learn / scikit-learn

Sponsor scikit-learn/scikit-learn

[MRG+2] Add a test for sample weights for estimators #11558

[MRG+2] Add a test for sample weights for estimators #11558

Conversation

sergulaydore commented Jul 16, 2018 • edited

Reference Issues/PRs

What is new?

What does this implement/fix? Explain your changes.

Any other comments?

GaelVaroquaux left a comment

This comment has been minimized.

GaelVaroquaux Jul 16, 2018 Member

This comment has been minimized.

sergulaydore Jul 16, 2018 Author Contributor

This comment has been minimized.

GaelVaroquaux Jul 16, 2018 Member

This comment has been minimized.

sergulaydore Jul 16, 2018 Author Contributor

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

This comment has been minimized.

GaelVaroquaux Jul 16, 2018 Member

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

glemaitre left a comment

This comment has been minimized.

glemaitre Jul 16, 2018 Contributor

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

This comment has been minimized.

glemaitre Jul 16, 2018 Contributor

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

This comment has been minimized.

glemaitre Jul 16, 2018 Contributor

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

This comment has been minimized.

glemaitre Jul 16, 2018 Contributor

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

This comment has been minimized.

glemaitre Jul 16, 2018 Contributor

This comment has been minimized.

glemaitre Jul 16, 2018 Contributor

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

TomDLT commented Jul 16, 2018 • edited

GaelVaroquaux commented Jul 17, 2018

GaelVaroquaux commented Jul 17, 2018

qinhanmin2014 commented Jul 17, 2018

This comment has been minimized.

qinhanmin2014 Jul 17, 2018 Member

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

GaelVaroquaux commented Jul 17, 2018

This comment has been minimized.

qinhanmin2014 Jul 17, 2018 Member

This comment has been minimized.

sergulaydore Jul 17, 2018 Author Contributor

This comment has been minimized.

glemaitre Jul 17, 2018 Contributor

sergulaydore commented Jul 17, 2018

qinhanmin2014 commented Jul 17, 2018

This comment has been minimized.

glemaitre Jul 17, 2018 Contributor

glemaitre commented Jul 17, 2018

qinhanmin2014 left a comment

GaelVaroquaux commented Jul 17, 2018

jnothman commented Jul 18, 2018

jnothman commented Jul 18, 2018

sergulaydore commented Jul 18, 2018

qinhanmin2014 commented Jul 18, 2018

sergulaydore commented Jul 18, 2018

sergulaydore commented Jul 16, 2018 •

edited

GaelVaroquaux Jul 16, 2018
Member

sergulaydore Jul 16, 2018
Author Contributor

GaelVaroquaux Jul 16, 2018
Member

sergulaydore Jul 16, 2018
Author Contributor

sergulaydore Jul 17, 2018
Author Contributor

GaelVaroquaux Jul 16, 2018
Member

sergulaydore Jul 17, 2018
Author Contributor

glemaitre Jul 16, 2018
Contributor

sergulaydore Jul 17, 2018
Author Contributor

glemaitre Jul 16, 2018
Contributor

sergulaydore Jul 17, 2018
Author Contributor

glemaitre Jul 16, 2018
Contributor

sergulaydore Jul 17, 2018
Author Contributor

glemaitre Jul 16, 2018
Contributor

sergulaydore Jul 17, 2018
Author Contributor

glemaitre Jul 16, 2018
Contributor

glemaitre Jul 16, 2018
Contributor

sergulaydore Jul 17, 2018
Author Contributor

TomDLT commented Jul 16, 2018 •

edited

qinhanmin2014 Jul 17, 2018
Member

sergulaydore Jul 17, 2018
Author Contributor

qinhanmin2014 Jul 17, 2018
Member

sergulaydore Jul 17, 2018
Author Contributor

glemaitre Jul 17, 2018
Contributor

glemaitre Jul 17, 2018
Contributor