[MRG + 2] add sample_weight support to Pipeline.score #7723
Conversation
pipe.fit(X, y=None) | ||
assert_equal(pipe.score(X), 3) | ||
assert_equal(pipe.score(X, y=None), 3) | ||
assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8) |
raghavrv
Oct 23, 2016
Member
Could you also check if it raises a sensible error message if the underlying estimator does not support sample_weight
in it's scoring?
Could you also check if it raises a sensible error message if the underlying estimator does not support sample_weight
in it's scoring?
kmike
Oct 23, 2016
Author
Contributor
A good call, I've added a check for TypeError.
I haven't added error message check because the error is a standard TypeError for missing keyword arguments - it may potentially vary across Python versions. Error messages for similar TypeErrors are not checked in other test_pipeline.py tests.
A good call, I've added a check for TypeError.
I haven't added error message check because the error is a standard TypeError for missing keyword arguments - it may potentially vary across Python versions. Error messages for similar TypeErrors are not checked in other test_pipeline.py tests.
raghavrv
Oct 23, 2016
Member
That makes sense but are there snippets of the error message that would be common across the versions? Something like unexpected.*sample_weight
that would surely be present in all the versions?
(assert_raises_regexp
)?
That makes sense but are there snippets of the error message that would be common across the versions? Something like unexpected.*sample_weight
that would surely be present in all the versions?
(assert_raises_regexp
)?
Besides, looks okay to me. @GaelVaroquaux @amueller @jnothman Do you approve this change? |
@@ -227,6 +227,7 @@ def test_pipeline_score_params(): | |||
assert_equal(pipe.score(X), 3) | |||
assert_equal(pipe.score(X, y=None), 3) | |||
assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8) | |||
assert_raises(TypeError, pipe.score, X, foo='bar') |
raghavrv
Oct 23, 2016
Member
I think testing for an explicit error message would be better... (assert_raise_message
)
I think testing for an explicit error message would be better... (assert_raise_message
)
assert_equal(pipe.score(X), 3) | ||
assert_equal(pipe.score(X, y=None), 3) | ||
assert_equal(pipe.score(X, sample_weight=np.array([2, 3])), 8) | ||
assert_raises(TypeError, pipe.score, X, foo='bar') |
amueller
Oct 24, 2016
Member
please use assert_raise_message
. Otherwise LGTM
please use assert_raise_message
. Otherwise LGTM
LGTM |
Done! I've also added a similar test for |
Thanks for that. Please do! |
assert_raise_message( | ||
TypeError, | ||
"fit() got an unexpected keyword argument 'bad'", | ||
pipe.fit, None, None, clf__bad=True |
raghavrv
Oct 24, 2016
Member
+1 thanks!
+1 thanks!
I think this could be mentioned in whatsnew. |
@@ -481,7 +481,7 @@ def _inverse_transform(self, X): | |||
return Xt | |||
|
|||
@if_delegate_has_method(delegate='_final_estimator') | |||
def score(self, X, y=None): | |||
def score(self, X, y=None, **score_params): |
jnothman
Oct 26, 2016
Member
I'd be tempted to be more conservative and make this only about sample_weight
. Is there no prospect of allowing other kwargs with the __
magic to be passed to the transformers along the way?
I'd be tempted to be more conservative and make this only about sample_weight
. Is there no prospect of allowing other kwargs with the __
magic to be passed to the transformers along the way?
amueller
Oct 26, 2016
Member
Sorry, I'm not sure I'm following.
Sorry, I'm not sure I'm following.
jnothman
Oct 26, 2016
Member
What I'm saying is that, despite the PR description referring to fit_params
, this behaves quite differently. fit_params
is routed to each step according to __
-delimited namespaces. By introducing **score_params
that are all routed to the score function only, we are promising that we will never do anything more sophisticated with any named arguments to score. I'm not convinced we have a use case that motivates such a general implementation, when we could just accept sample_weight=None
.
What I'm saying is that, despite the PR description referring to fit_params
, this behaves quite differently. fit_params
is routed to each step according to __
-delimited namespaces. By introducing **score_params
that are all routed to the score function only, we are promising that we will never do anything more sophisticated with any named arguments to score. I'm not convinced we have a use case that motivates such a general implementation, when we could just accept sample_weight=None
.
amueller
Oct 27, 2016
Member
Makes sense. I agree.
Makes sense. I agree.
kmike
Oct 27, 2016
•
Author
Contributor
The difference with **fit_params
is that transformers can have .fit
method, but there is only a single .score method - that's why I went with **score_params
without __
handling. But I see the problem now.
So did I get it right: add sample_weight=None argument, don't pass sample_weight to .score if it is None (as not all score methods suport sample_weight), and in the future scikit-learn can add support for __
-delimeted keyword arguments which are passed to .transform methods?
score(X, y=None, sample_weight=None)
signature has a small issue: it is not possible to distinguish between passing sample_weight=None
and not passing sample_weight
; it is technically different if estimator.score doesn't support sample_weight argument. Should I handle this case by making signature .score(X, y=None, **kwargs)
, but documenting it as sample_weight=None?
The difference with **fit_params
is that transformers can have .fit
method, but there is only a single .score method - that's why I went with **score_params
without __
handling. But I see the problem now.
So did I get it right: add sample_weight=None argument, don't pass sample_weight to .score if it is None (as not all score methods suport sample_weight), and in the future scikit-learn can add support for __
-delimeted keyword arguments which are passed to .transform methods?
score(X, y=None, sample_weight=None)
signature has a small issue: it is not possible to distinguish between passing sample_weight=None
and not passing sample_weight
; it is technically different if estimator.score doesn't support sample_weight argument. Should I handle this case by making signature .score(X, y=None, **kwargs)
, but documenting it as sample_weight=None?
raghavrv
Oct 27, 2016
Member
Or we could have the signature as ..., sample_weight=None
, pass **score_params
to score
method and have it documented that it is applicable only if the underlying estimator supports it...
Or we could have the signature as ..., sample_weight=None
, pass **score_params
to score
method and have it documented that it is applicable only if the underlying estimator supports it...
kmike
Oct 27, 2016
•
Author
Contributor
If we pass **score_params
to score function people will start using it even if it is not documented, why should we do that?
If we pass **score_params
to score function people will start using it even if it is not documented, why should we do that?
raghavrv
Oct 27, 2016
Member
If we pass **score_params to score function people
Ah wait we are talking about different score
functions ;)
I suggested
- passing
**score_params
to self.steps[-1][-1].score(Xt, y, **score_params)
so it doesn't raise an error if sample_weights
is not supported by the underlying estimator's score
.
- passing
sample_weights=None
to Pipeline.score(..., sample_weights=None)
to make it clear that we do not support passing any other parameters, except sample_weights
.
- documenting at
Pipeline.score
that sample_weights
can only be passed if the underlying estimator supports it...
I think this is exactly what you suggested already. Sorry for the noise :)
If we pass **score_params to score function people
Ah wait we are talking about different score
functions ;)
I suggested
- passing
**score_params
toself.steps[-1][-1].score(Xt, y, **score_params)
so it doesn't raise an error ifsample_weights
is not supported by the underlying estimator'sscore
. - passing
sample_weights=None
toPipeline.score(..., sample_weights=None)
to make it clear that we do not support passing any other parameters, exceptsample_weights
. - documenting at
Pipeline.score
thatsample_weights
can only be passed if the underlying estimator supports it...
I think this is exactly what you suggested already. Sorry for the noise :)
I think using |
I've implemented the simplest solution - don't pass sample_weight if it is None. Does it look ok? |
Could we bother you for a what's new entry? LGTM |
@jnothman yep! |
fd72352
to
71d486d
thanks! |
…rn#7723) * add sample_weight support to Pipeline.score * TST check that pipe.score raises TypeError for unsorrported arguments * TST check error message for invalid score_params in pipeline.score * TST check that pipe.fit raises TypeError when argument is not supported * only support sample_weight argument * whats_new: mention sample_weight in Pipeline.score
…rn#7723) * add sample_weight support to Pipeline.score * TST check that pipe.score raises TypeError for unsorrported arguments * TST check error message for invalid score_params in pipeline.score * TST check that pipe.fit raises TypeError when argument is not supported * only support sample_weight argument * whats_new: mention sample_weight in Pipeline.score
…rn#7723) * add sample_weight support to Pipeline.score * TST check that pipe.score raises TypeError for unsorrported arguments * TST check error message for invalid score_params in pipeline.score * TST check that pipe.fit raises TypeError when argument is not supported * only support sample_weight argument * whats_new: mention sample_weight in Pipeline.score
…rn#7723) * add sample_weight support to Pipeline.score * TST check that pipe.score raises TypeError for unsorrported arguments * TST check error message for invalid score_params in pipeline.score * TST check that pipe.fit raises TypeError when argument is not supported * only support sample_weight argument * whats_new: mention sample_weight in Pipeline.score
…rn#7723) * add sample_weight support to Pipeline.score * TST check that pipe.score raises TypeError for unsorrported arguments * TST check error message for invalid score_params in pipeline.score * TST check that pipe.fit raises TypeError when argument is not supported * only support sample_weight argument * whats_new: mention sample_weight in Pipeline.score
Many estimators support
sample_weight
argument in their .score function - e.g. default .score implementation in ClassifierMixin provides this feature. In this PR kwargs support is added to Pipeline.score, similar to**fit_params
in Pipeline.fit; this allows to pass sample_weight to pipeline.score(...).