[MRG + 2] add sample_weight support to Pipeline.score #7723

kmike · Oct 21, 2016

Many estimators support sample_weight argument in their .score function - e.g. default .score implementation in ClassifierMixin provides this feature. In this PR kwargs support is added to Pipeline.score, similar to **fit_params in Pipeline.fit; this allows to pass sample_weight to pipeline.score(...).

raghavrv · Oct 23, 2016

raghavrv requested changes Oct 23, 2016

View changes

raghavrv · Oct 23, 2016

sklearn/tests/test_pipeline.py

+    pipe.fit(X, y=None)
+    assert_equal(pipe.score(X), 3)
+    assert_equal(pipe.score(X, y=None), 3)
+    assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8)


Could you also check if it raises a sensible error message if the underlying estimator does not support sample_weight in it's scoring?

A good call, I've added a check for TypeError.

I haven't added error message check because the error is a standard TypeError for missing keyword arguments - it may potentially vary across Python versions. Error messages for similar TypeErrors are not checked in other test_pipeline.py tests.

That makes sense but are there snippets of the error message that would be common across the versions? Something like unexpected.*sample_weight that would surely be present in all the versions?

(assert_raises_regexp)?

raghavrv · Oct 23, 2016

raghavrv requested changes Oct 23, 2016

View changes

Besides, looks okay to me. @GaelVaroquaux @amueller @jnothman Do you approve this change?

raghavrv · Oct 23, 2016

sklearn/tests/test_pipeline.py

@@ -227,6 +227,7 @@ def test_pipeline_score_params():
    assert_equal(pipe.score(X), 3)
    assert_equal(pipe.score(X, y=None), 3)
    assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8)
+    assert_raises(TypeError, pipe.score, X, foo='bar')


I think testing for an explicit error message would be better... (assert_raise_message)

amueller · Oct 24, 2016

amueller reviewed Oct 24, 2016

View changes

amueller · Oct 24, 2016

sklearn/tests/test_pipeline.py

+    assert_equal(pipe.score(X), 3)
+    assert_equal(pipe.score(X, y=None), 3)
+    assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8)
+    assert_raises(TypeError, pipe.score, X, foo='bar')


please use assert_raise_message. Otherwise LGTM

raghavrv · Oct 24, 2016

raghavrv approved these changes Oct 24, 2016

View changes

LGTM

amueller · Oct 24, 2016

LGTM

Done! I've also added a similar test for pipe.fit(X,y, **fit_params), errors were untested for them. Is it OK to have it in the same PR?

kmike · Oct 24, 2016

Done! I've also added a similar test for pipe.fit(X,y, **fit_params), errors were untested for them. Is it OK to have it in the same PR?

Is it OK to have it in the same PR?

Thanks for that. Please do!

raghavrv · Oct 24, 2016

Is it OK to have it in the same PR?

Thanks for that. Please do!

raghavrv · Oct 24, 2016

raghavrv reviewed Oct 24, 2016

View changes

raghavrv · Oct 24, 2016

sklearn/tests/test_pipeline.py

+    assert_raise_message(
+        TypeError,
+        "fit() got an unexpected keyword argument 'bad'",
+        pipe.fit, None, None, clf__bad=True


I think this could be mentioned in whatsnew.

raghavrv · Oct 24, 2016

I think this could be mentioned in whatsnew.

jnothman · Oct 26, 2016

jnothman reviewed Oct 26, 2016

View changes

jnothman · Oct 26, 2016

sklearn/pipeline.py

@@ -481,7 +481,7 @@ def _inverse_transform(self, X):
        return Xt

    @if_delegate_has_method(delegate='_final_estimator')
-    def score(self, X, y=None):
+    def score(self, X, y=None, **score_params):


I'd be tempted to be more conservative and make this only about sample_weight. Is there no prospect of allowing other kwargs with the __ magic to be passed to the transformers along the way?

Sorry, I'm not sure I'm following.

What I'm saying is that, despite the PR description referring to fit_params, this behaves quite differently. fit_params is routed to each step according to __-delimited namespaces. By introducing **score_params that are all routed to the score function only, we are promising that we will never do anything more sophisticated with any named arguments to score. I'm not convinced we have a use case that motivates such a general implementation, when we could just accept sample_weight=None.

Makes sense. I agree.

The difference with **fit_params is that transformers can have .fit method, but there is only a single .score method - that's why I went with **score_params without __ handling. But I see the problem now.

So did I get it right: add sample_weight=None argument, don't pass sample_weight to .score if it is None (as not all score methods suport sample_weight), and in the future scikit-learn can add support for __-delimeted keyword arguments which are passed to .transform methods?

score(X, y=None, sample_weight=None) signature has a small issue: it is not possible to distinguish between passing sample_weight=None and not passing sample_weight; it is technically different if estimator.score doesn't support sample_weight argument. Should I handle this case by making signature .score(X, y=None, **kwargs), but documenting it as sample_weight=None?

Or we could have the signature as ..., sample_weight=None, pass **score_params to score method and have it documented that it is applicable only if the underlying estimator supports it...

If we pass **score_params to score function people will start using it even if it is not documented, why should we do that?

If we pass **score_params to score function people

Ah wait we are talking about different score functions ;)

I suggested

passing **score_params to self.steps[-1][-1].score(Xt, y, **score_params) so it doesn't raise an error if sample_weights is not supported by the underlying estimator's score.

passing sample_weights=None to Pipeline.score(..., sample_weights=None) to make it clear that we do not support passing any other parameters, except sample_weights.

documenting at Pipeline.score that sample_weights can only be passed if the underlying estimator supports it...

I think this is exactly what you suggested already. Sorry for the noise :)

I think using sample_weight and also passing it if it is not None is a good solution.

amueller · Oct 27, 2016

I think using sample_weight and also passing it if it is not None is a good solution.

I've implemented the simplest solution - don't pass sample_weight if it is None. Does it look ok?

kmike · Oct 28, 2016

I've implemented the simplest solution - don't pass sample_weight if it is None. Does it look ok?

Could we bother you for a what's new entry? LGTM

jnothman · Oct 29, 2016

Could we bother you for a what's new entry? LGTM

@jnothman yep!

kmike · Oct 29, 2016

@jnothman yep!

thanks!

jnothman · Oct 29, 2016

thanks!

amueller changed the title from add sample_weight support to Pipeline.score to [MRG + 1] add sample_weight support to Pipeline.score Oct 24, 2016

amueller added this to the 0.19 milestone Oct 24, 2016

jnothman changed the title from [MRG + 1] add sample_weight support to Pipeline.score to [MRG + 2] add sample_weight support to Pipeline.score Oct 29, 2016

scikit-learn/scikit-learn

Join GitHub today

[MRG + 2] add sample_weight support to Pipeline.score #7723

Conversation

kmike commented Oct 21, 2016

raghavrv requested changes Oct 23, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

raghavrv requested changes Oct 23, 2016 View changes

This comment has been minimized.

amueller reviewed Oct 24, 2016 View changes

This comment has been minimized.

raghavrv approved these changes Oct 24, 2016 View changes

amueller changed the title from add sample_weight support to Pipeline.score to [MRG + 1] add sample_weight support to Pipeline.score Oct 24, 2016

amueller added this to the 0.19 milestone Oct 24, 2016

This comment has been minimized.

amueller commented Oct 24, 2016

This comment has been minimized.

kmike commented Oct 24, 2016 • edited Edited 1 time kmike edited Oct 24, 2016 (most recent)

This comment has been minimized.

raghavrv commented Oct 24, 2016

raghavrv reviewed Oct 24, 2016 View changes

This comment has been minimized.

This comment has been minimized.

raghavrv commented Oct 24, 2016

jnothman reviewed Oct 26, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kmike Oct 27, 2016 • edited Edited 1 time kmike edited Oct 27, 2016 (most recent)

This comment has been minimized.

This comment has been minimized.

kmike Oct 27, 2016 • edited Edited 1 time kmike edited Oct 27, 2016 (most recent)

This comment has been minimized.

This comment has been minimized.

amueller commented Oct 27, 2016

This comment has been minimized.

kmike commented Oct 28, 2016

This comment has been minimized.

jnothman commented Oct 29, 2016

jnothman changed the title from [MRG + 1] add sample_weight support to Pipeline.score to [MRG + 2] add sample_weight support to Pipeline.score Oct 29, 2016

This comment has been minimized.

kmike commented Oct 29, 2016

kmike added some commits Oct 21, 2016

Hide details View details jnothman merged commit 10323f5 into scikit-learn:master Oct 29, 2016 2 of 3 checks passed

2 of 3 checks passed

This comment has been minimized.

jnothman commented Oct 29, 2016

sergeyf added a commit to sergeyf/scikit-learn that referenced this pull request Feb 28, 2017

afiodorov added a commit to unravelin/scikit-learn that referenced this pull request Apr 25, 2017

Sundrique added a commit to Sundrique/scikit-learn that referenced this pull request Jun 14, 2017

paulha added a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017

maskani-moh added a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017

raghavrv requested changes Oct 23, 2016

View changes

raghavrv requested changes Oct 23, 2016

View changes

amueller reviewed Oct 24, 2016

View changes

raghavrv approved these changes Oct 24, 2016

View changes

kmike commented Oct 24, 2016 •

edited

Edited 1 time

kmike edited Oct 24, 2016 (most recent)

raghavrv reviewed Oct 24, 2016

View changes

jnothman reviewed Oct 26, 2016

View changes

kmike Oct 27, 2016 •

edited

Edited 1 time

kmike edited Oct 27, 2016 (most recent)

kmike Oct 27, 2016 •

edited

Edited 1 time

kmike edited Oct 27, 2016 (most recent)

jnothman merged commit `10323f5` into scikit-learn:master Oct 29, 2016
2 of 3 checks passed