[MRG + 2] add sample_weight support to Pipeline.score #7723

kmike · 2016-10-21T20:34:12Z

Many estimators support sample_weight argument in their .score function - e.g. default .score implementation in ClassifierMixin provides this feature. In this PR kwargs support is added to Pipeline.score, similar to **fit_params in Pipeline.fit; this allows to pass sample_weight to pipeline.score(...).

raghavrv · 2016-10-23T15:56:32Z

sklearn/tests/test_pipeline.py

+    pipe.fit(X, y=None)
+    assert_equal(pipe.score(X), 3)
+    assert_equal(pipe.score(X, y=None), 3)
+    assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8)


Could you also check if it raises a sensible error message if the underlying estimator does not support sample_weight in it's scoring?

A good call, I've added a check for TypeError.

I haven't added error message check because the error is a standard TypeError for missing keyword arguments - it may potentially vary across Python versions. Error messages for similar TypeErrors are not checked in other test_pipeline.py tests.

That makes sense but are there snippets of the error message that would be common across the versions? Something like unexpected.*sample_weight that would surely be present in all the versions?

(assert_raises_regexp)?

raghavrv

Besides, looks okay to me. @GaelVaroquaux @amueller @jnothman Do you approve this change?

raghavrv · 2016-10-23T18:58:18Z

sklearn/tests/test_pipeline.py

@@ -227,6 +227,7 @@ def test_pipeline_score_params():
    assert_equal(pipe.score(X), 3)
    assert_equal(pipe.score(X, y=None), 3)
    assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8)
+    assert_raises(TypeError, pipe.score, X, foo='bar')


I think testing for an explicit error message would be better... (assert_raise_message)

amueller · 2016-10-24T16:08:03Z

sklearn/tests/test_pipeline.py

+    assert_equal(pipe.score(X), 3)
+    assert_equal(pipe.score(X, y=None), 3)
+    assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8)
+    assert_raises(TypeError, pipe.score, X, foo='bar')


please use assert_raise_message. Otherwise LGTM

amueller · 2016-10-24T19:08:51Z

LGTM

kmike · 2016-10-24T19:09:07Z

Done! I've also added a similar test for pipe.fit(X,y, **fit_params), errors were untested for them. Is it OK to have it in the same PR?

raghavrv · 2016-10-24T19:12:40Z

Is it OK to have it in the same PR?

Thanks for that. Please do!

raghavrv · 2016-10-24T19:13:34Z

sklearn/tests/test_pipeline.py

+    assert_raise_message(
+        TypeError,
+        "fit() got an unexpected keyword argument 'bad'",
+        pipe.fit, None, None, clf__bad=True


raghavrv · 2016-10-24T19:14:17Z

I think this could be mentioned in whatsnew.

jnothman · 2016-10-26T12:15:33Z

sklearn/pipeline.py

@@ -481,7 +481,7 @@ def _inverse_transform(self, X):
        return Xt

    @if_delegate_has_method(delegate='_final_estimator')
-    def score(self, X, y=None):
+    def score(self, X, y=None, **score_params):


I'd be tempted to be more conservative and make this only about sample_weight. Is there no prospect of allowing other kwargs with the __ magic to be passed to the transformers along the way?

Sorry, I'm not sure I'm following.

What I'm saying is that, despite the PR description referring to fit_params, this behaves quite differently. fit_params is routed to each step according to __-delimited namespaces. By introducing **score_params that are all routed to the score function only, we are promising that we will never do anything more sophisticated with any named arguments to score. I'm not convinced we have a use case that motivates such a general implementation, when we could just accept sample_weight=None.

Makes sense. I agree.

The difference with **fit_params is that transformers can have .fit method, but there is only a single .score method - that's why I went with **score_params without __ handling. But I see the problem now.

So did I get it right: add sample_weight=None argument, don't pass sample_weight to .score if it is None (as not all score methods suport sample_weight), and in the future scikit-learn can add support for __-delimeted keyword arguments which are passed to .transform methods?

score(X, y=None, sample_weight=None) signature has a small issue: it is not possible to distinguish between passing sample_weight=None and not passing sample_weight; it is technically different if estimator.score doesn't support sample_weight argument. Should I handle this case by making signature .score(X, y=None, **kwargs), but documenting it as sample_weight=None?

Or we could have the signature as ..., sample_weight=None, pass **score_params to score method and have it documented that it is applicable only if the underlying estimator supports it...

If we pass **score_params to score function people will start using it even if it is not documented, why should we do that?

If we pass **score_params to score function people

Ah wait we are talking about different score functions ;)

I suggested

passing **score_params to self.steps[-1][-1].score(Xt, y, **score_params) so it doesn't raise an error if sample_weights is not supported by the underlying estimator's score.

passing sample_weights=None to Pipeline.score(..., sample_weights=None) to make it clear that we do not support passing any other parameters, except sample_weights.

documenting at Pipeline.score that sample_weights can only be passed if the underlying estimator supports it...

I think this is exactly what you suggested already. Sorry for the noise :)

amueller · 2016-10-27T17:02:42Z

I think using sample_weight and also passing it if it is not None is a good solution.

kmike · 2016-10-28T09:15:03Z

I've implemented the simplest solution - don't pass sample_weight if it is None. Does it look ok?

jnothman · 2016-10-29T11:21:35Z

Could we bother you for a what's new entry? LGTM

kmike · 2016-10-29T12:52:48Z

@jnothman yep!

jnothman · 2016-10-29T13:38:09Z

thanks!

…rn#7723) * add sample_weight support to Pipeline.score * TST check that pipe.score raises TypeError for unsorrported arguments * TST check error message for invalid score_params in pipeline.score * TST check that pipe.fit raises TypeError when argument is not supported * only support sample_weight argument * whats_new: mention sample_weight in Pipeline.score

raghavrv suggested changes Oct 23, 2016

View changes

amueller reviewed Oct 24, 2016

View changes

raghavrv approved these changes Oct 24, 2016

View changes

amueller changed the title ~~add sample_weight support to Pipeline.score~~ [MRG + 1] add sample_weight support to Pipeline.score Oct 24, 2016

amueller added this to the 0.19 milestone Oct 24, 2016

raghavrv reviewed Oct 24, 2016

View changes

sklearn/tests/test_pipeline.py

assert_raise_message(

TypeError,

"fit() got an unexpected keyword argument 'bad'",

pipe.fit, None, None, clf__bad=True

This comment has been minimized.

Sign in to view

raghavrv Oct 24, 2016
Member

+1 thanks!

jnothman reviewed Oct 26, 2016

View changes

jnothman changed the title ~~[MRG + 1] add sample_weight support to Pipeline.score~~ [MRG + 2] add sample_weight support to Pipeline.score Oct 29, 2016

kmike added 6 commits Oct 21, 2016

add sample_weight support to Pipeline.score

206e397

TST check that pipe.score raises TypeError for unsorrported arguments

f6e76c4

TST check error message for invalid score_params in pipeline.score

ecded73

TST check that pipe.fit raises TypeError when argument is not supported

8c4d180

only support sample_weight argument

bb94713

whats_new: mention sample_weight in Pipeline.score

Loading status checks…

71d486d

kmike force-pushed the kmike:pipeline-score-sample-weight branch from fd72352 to 71d486d Oct 29, 2016

scikit-learn / scikit-learn

[MRG + 2] add sample_weight support to Pipeline.score #7723

[MRG + 2] add sample_weight support to Pipeline.score #7723

kmike commented Oct 21, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

raghavrv left a comment

This comment has been minimized.

This comment has been minimized.

amueller commented Oct 24, 2016

kmike commented Oct 24, 2016 •

edited

raghavrv commented Oct 24, 2016

This comment has been minimized.

raghavrv commented Oct 24, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

amueller commented Oct 27, 2016

kmike commented Oct 28, 2016

jnothman commented Oct 29, 2016

kmike commented Oct 29, 2016

jnothman commented Oct 29, 2016

scikit-learn / scikit-learn

Sponsor scikit-learn/scikit-learn

[MRG + 2] add sample_weight support to Pipeline.score #7723

[MRG + 2] add sample_weight support to Pipeline.score #7723

Conversation

kmike commented Oct 21, 2016

This comment has been minimized.

raghavrv Oct 23, 2016 Member

This comment has been minimized.

kmike Oct 23, 2016 Author Contributor

This comment has been minimized.

raghavrv Oct 23, 2016 Member

raghavrv left a comment

This comment has been minimized.

raghavrv Oct 23, 2016 Member

This comment has been minimized.

amueller Oct 24, 2016 Member

amueller commented Oct 24, 2016

kmike commented Oct 24, 2016 • edited

raghavrv commented Oct 24, 2016

This comment has been minimized.

raghavrv Oct 24, 2016 Member

raghavrv commented Oct 24, 2016

This comment has been minimized.

jnothman Oct 26, 2016 Member

This comment has been minimized.

amueller Oct 26, 2016 Member

This comment has been minimized.

jnothman Oct 26, 2016 Member

This comment has been minimized.

amueller Oct 27, 2016 Member

This comment has been minimized.

kmike Oct 27, 2016 • edited Author Contributor

This comment has been minimized.

raghavrv Oct 27, 2016 Member

This comment has been minimized.

kmike Oct 27, 2016 • edited Author Contributor

This comment has been minimized.

raghavrv Oct 27, 2016 Member

amueller commented Oct 27, 2016

kmike commented Oct 28, 2016

jnothman commented Oct 29, 2016

kmike commented Oct 29, 2016

jnothman commented Oct 29, 2016

raghavrv Oct 23, 2016
Member

kmike Oct 23, 2016
Author Contributor

raghavrv Oct 23, 2016
Member

raghavrv Oct 23, 2016
Member

amueller Oct 24, 2016
Member

kmike commented Oct 24, 2016 •

edited

raghavrv Oct 24, 2016
Member

jnothman Oct 26, 2016
Member

amueller Oct 26, 2016
Member

jnothman Oct 26, 2016
Member

amueller Oct 27, 2016
Member

kmike Oct 27, 2016 •

edited

Author Contributor

raghavrv Oct 27, 2016
Member

kmike Oct 27, 2016 •

edited

Author Contributor

raghavrv Oct 27, 2016
Member