/scikit-learn

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+2] Add a test for sample weights for estimators #11558

Merged
merged 13 commits into from Jul 17, 2018

Conversation

Reviewers

@GaelVaroquaux GaelVaroquaux

@glemaitre glemaitre

@qinhanmin2014 qinhanmin2014

Assignees
No one assigned
Labels
None yet
Projects
  
PRs tagged in scikit-learn 0.20
Milestone
  0.20
6 participants
@sergulaydore @TomDLT @GaelVaroquaux @qinhanmin2014 @glemaitre @jnothman
@sergulaydore
Contributor

sergulaydore commented Jul 16, 2018 •
edited

Reference Issues/PRs

Fixes the invariant test for sample weights as mentioned in issue #11316 (Refactor tests for sample weights).

What is new?

This is a generic test for estimators that makes sure the sample weights yield consistent results.

What does this implement/fix? Explain your changes.

The test checks if the output of the estimators are the same for sample_weight = None and sample_weight=np.ones().

Any other comments?

Pairwise methods are skipped as they require pairwise data.
@GaelVaroquaux

GaelVaroquaux reviewed Jul 16, 2018
View changes

Looks good, aside from 2 minor comments.

Show outdated Hide outdated sklearn/utils/estimator_checks.py
Show outdated Hide outdated sklearn/utils/estimator_checks.py
@sergulaydore
dtype and X_trans
Loading status checks…
0738abc

@GaelVaroquaux GaelVaroquaux changed the title from Add a test for sample weights for estimators to [MRG+1] Add a test for sample weights for estimators Jul 16, 2018

@GaelVaroquaux GaelVaroquaux added this to the 0.20 milestone Jul 16, 2018

Show outdated Hide outdated sklearn/utils/estimator_checks.py
@glemaitre

glemaitre requested changes Jul 16, 2018
View changes

I think that it could be great to add an entry in the what's new as well

Show outdated Hide outdated sklearn/utils/estimator_checks.py
Show outdated Hide outdated sklearn/utils/estimator_checks.py
Show outdated Hide outdated sklearn/utils/estimator_checks.py
Show outdated Hide outdated sklearn/utils/estimator_checks.py
Show outdated Hide outdated sklearn/utils/estimator_checks.py
@TomDLT

This comment has been minimized.

Show comment
Hide comment
@TomDLT

TomDLT Jul 16, 2018

Member

I did not look into your code, but you might get inspired from the excellent test in #10803.
see here
Member

TomDLT commented Jul 16, 2018 •
edited

I did not look into your code, but you might get inspired from the excellent test in #10803.
see here

sergulaydore and others added some commits Jul 17, 2018

@sergulaydore
after second reviews
Loading status checks…
1d93961
@sergulaydore
pyflake errors
Loading status checks…
6b7f20a
@sergulaydore
Merge branch 'master' into refactor_sample_weights
Loading status checks…
ad6a365
@sergulaydore
fixed pairwise error
Loading status checks…
ad50ab9
@GaelVaroquaux
PEP8: line too long
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
033a84a
@GaelVaroquaux

This comment has been minimized.

Show comment
Hide comment
@GaelVaroquaux

GaelVaroquaux Jul 17, 2018

Member

Test are passing! Congratulations.

But you had a PEP8 failure. I fixed it on your branch. We need to wait for the tests to run again.
Member

GaelVaroquaux commented Jul 17, 2018

Test are passing! Congratulations.

But you had a PEP8 failure. I fixed it on your branch. We need to wait for the tests to run again.
@GaelVaroquaux

This comment has been minimized.

Show comment
Hide comment
@GaelVaroquaux

GaelVaroquaux Jul 17, 2018

Member

@glemaitre : can you reassess you review. We think that this is good to go.
Member

GaelVaroquaux commented Jul 17, 2018

@glemaitre : can you reassess you review. We think that this is good to go.
@qinhanmin2014

This comment has been minimized.

Show comment
Hide comment
@qinhanmin2014

qinhanmin2014 Jul 17, 2018

Member

I can't understand why we need to skip "KMeans" and "MiniBatchKMeans". It seems that the test passes locally on my PC with these two classes. Also, at least we have things like v_measure_score to use here?
Member

qinhanmin2014 commented Jul 17, 2018

I can't understand why we need to skip "KMeans" and "MiniBatchKMeans". It seems that the test passes locally on my PC with these two classes. Also, at least we have things like v_measure_score to use here?
Show outdated Hide outdated sklearn/utils/estimator_checks.py
@GaelVaroquaux

This comment has been minimized.

Show comment
Hide comment
@GaelVaroquaux

GaelVaroquaux Jul 17, 2018

Member

Indeed, I think that it's good to add comments.

I canceled the corresponding travis builds, to save time in the queue.
Member

GaelVaroquaux commented Jul 17, 2018

Indeed, I think that it's good to add comments.

I canceled the corresponding travis builds, to save time in the queue.
@sergulaydore
add comments and pep8 issues
Loading status checks…
ef0371c
Show outdated Hide outdated sklearn/utils/estimator_checks.py
@sergulaydore

This comment has been minimized.

Show comment
Hide comment
@sergulaydore

sergulaydore Jul 17, 2018

Contributor

@qinhanmin2014 : "KMeans" and "MiniBatchKMeans" are skipped because they were unstable.
Contributor

sergulaydore commented Jul 17, 2018

@qinhanmin2014 : "KMeans" and "MiniBatchKMeans" are skipped because they were unstable.
@qinhanmin2014

This comment has been minimized.

Show comment
Hide comment
@qinhanmin2014

qinhanmin2014 Jul 17, 2018

Member

I would expect estimators in scikit-learn to be deterministic with same input and a fixed random_state, or am I wrong?
Member

qinhanmin2014 commented Jul 17, 2018

I would expect estimators in scikit-learn to be deterministic with same input and a fixed random_state, or am I wrong?
Show outdated Hide outdated sklearn/utils/estimator_checks.py
@glemaitre

This comment has been minimized.

Show comment
Hide comment
@glemaitre

glemaitre Jul 17, 2018

Contributor

Oh you added back kmeans, cool so this is fine then.
Waiting for the CI to be green.
Contributor

glemaitre commented Jul 17, 2018

Oh you added back kmeans, cool so this is fine then.
Waiting for the CI to be green.
@qinhanmin2014

qinhanmin2014 approved these changes Jul 17, 2018
View changes

LGTM if CIs are green. Thanks @sergulaydore

@qinhanmin2014 qinhanmin2014 changed the title from [MRG+1] Add a test for sample weights for estimators to [MRG+2] Add a test for sample weights for estimators Jul 17, 2018

@GaelVaroquaux

This comment has been minimized.

Show comment
Hide comment
@GaelVaroquaux

GaelVaroquaux Jul 17, 2018

Member

OK, travis is green. Merging. Hurray!
Member

GaelVaroquaux commented Jul 17, 2018

OK, travis is green. Merging. Hurray!

@GaelVaroquaux GaelVaroquaux merged commit bcd6ff3 into scikit-learn:master Jul 17, 2018
3 of 5 checks passed

3 of 5 checks passed

ci/circleci: deploy Your tests are queued behind your running builds
Details
continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
ci/circleci: python2 Your tests passed on CircleCI!
Details
ci/circleci: python3 Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@GaelVaroquaux GaelVaroquaux added this to Done in scikit-learn 0.20 via automation Jul 17, 2018

@lesteve lesteve moved this from Done to PRs tagged in scikit-learn 0.20 Jul 17, 2018

@jnothman jnothman referenced this pull request Jul 18, 2018

Open

[WIP] Sample weight consistency #5515

@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Jul 18, 2018

Member

Why do we need an rtol as high as 0.5?
Member

jnothman commented Jul 18, 2018

Why do we need an rtol as high as 0.5?
@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Jul 18, 2018

Member

This doesn't assert that weighting makes any difference ever, does it?
Member

jnothman commented Jul 18, 2018

This doesn't assert that weighting makes any difference ever, does it?
@sergulaydore

This comment has been minimized.

Show comment
Hide comment
@sergulaydore

sergulaydore Jul 18, 2018

Contributor

@jnothman Actually we don't need rtol as 0.5. I just tested with the default value and the tests still passed. No, it only makes sure that the no weighting is equal to unit weighting. If you recommend, I can submit another PR with random weights.
Contributor

sergulaydore commented Jul 18, 2018

@jnothman Actually we don't need rtol as 0.5. I just tested with the default value and the tests still passed. No, it only makes sure that the no weighting is equal to unit weighting. If you recommend, I can submit another PR with random weights.
@qinhanmin2014

This comment has been minimized.

Show comment
Hide comment
@qinhanmin2014

qinhanmin2014 Jul 18, 2018

Member

+1 to use default rtol, @sergulaydore please submit a PR, thanks.
Member

qinhanmin2014 commented Jul 18, 2018

+1 to use default rtol, @sergulaydore please submit a PR, thanks.

@sergulaydore sergulaydore referenced this pull request Jul 18, 2018

Merged

[MRG+2] TST Using default rtol in test sample_weight invariance #11621

@sergulaydore

This comment has been minimized.

Show comment
Hide comment
@sergulaydore

sergulaydore Jul 18, 2018

Contributor

Done! Please see #11621.
Contributor

sergulaydore commented Jul 18, 2018

Done! Please see #11621.

@sergulaydore sergulaydore deleted the sergulaydore:refactor_sample_weights branch Jul 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment