[MRG + 2] Print train scores in _fit_and_score #12613

marctorsoc · 2018-11-18T20:23:42Z

This small feature was first mentioned (afaik) in Issue #12598

What does this implement/fix? Explain your changes.

It is a simple change in _fit_and_score allowing to print both train and test scores. This is used for different purposes: cross-validation (with any kind of splitter: Kfold, etc), grid/random search, learning curves. I also modified the format so that 3 decimals places are printed only

Any other comments?

There was no documentation for the different verbosity levels, so I left the docstring untouched. Happy to explain what the different levels print if required

marctorsoc · 2018-11-18T23:19:04Z

Sorry guys, it's my first PR and have no idea how to solve that code coverage error...

jnothman

These cases are already not covered by tests, so it might not be essential to cover them (although it would be better if they were). Tests can be added in sklearn/model_selection/tests/test_validation.py.

I don't see why we shouldn't just output this data if verbose=2 also, as long as return_train_score=True...

sklearn/model_selection/_validation.py

test coverage increased

amueller · 2018-11-20T01:12:25Z

looks good. maybe a test for multimetric would be nice? Don't have a strong opinion, though.

tests cleaned using pytest parametrize

jnothman

Otherwise LGTM

jnothman · 2018-11-20T04:27:58Z

sklearn/model_selection/_validation.py

-            try:
-                # e.g. unwrap memmapped scalars
-                score = score.item()
-            except ValueError:


I don't get what this has to do with the present pull request

I was trying to improve test coverage. Not sure in what situation the item() method can fail (we're checking before that if the object has this method before calling it), that's why I removed that try-except. But can revert that change if you feel was correct (and add a test if you give me an example of how to make it fail)

I don't know why it's useful really, and I don't know why it's not tested, but you can't just remove existing functionality...

yeah sorry

@marctorrellas if you need help with test tell me I can help =)

@eamanu I would actually appreciate if you can revert my change and add a test for that .item() failure. Or just giving me an example of how to make that to fail in purpose. I mean, can do it totally artificially, but there's a comment there "unwrap memapped scalars" and probably the tests should use that but no clue what it is

Blame shows this commit:
amueller@a08555a
which is not very helpful.
I had assumed this was for dask support, but it would be weird if it was introduced in that PR.
If neither @jnothman nor I know why it's there it's a bit suspect but I wouldn't worry about it for this PR.

(i.e. undo the change)

eamanu

LGTM

eamanu · 2018-11-20T13:30:24Z

@marctorrellas if you need help with test tell me I can help =)

marctorsoc · 2018-11-21T01:19:47Z

I think this is ready for merge

jnothman · 2018-11-21T03:59:13Z

Please add a brief entry to the change log at doc/whats_new/v0.21.rst. Like the other entries there, please reference this pull request with :issue: and credit yourself (and other contributors if applicable) with :user:

marctorsoc · 2018-11-21T20:55:28Z

Please add a brief entry to the change log at doc/whats_new/v0.21.rst. Like the other entries there, please reference this pull request with :issue: and credit yourself (and other contributors if applicable) with :user:

Not really sure if I took the correct labels. Are API and Enhancement ok?

amueller · 2018-11-21T21:06:03Z

I don't think API is right, just Enhancement probably?

amueller · 2018-11-21T21:11:21Z

doc/whats_new/v0.21.rst

+  `return_train_scores` is True and `verbose` > 2.
+  :issue:`12613` by :user:`Marc Torrellas <marctorrellas>`.
+
+- |API| Additional tests for :func:`_fit_and_score` and :func:`_score`.


I would just remove this item tbh.

amueller · 2018-11-22T01:22:19Z

sweet, thanks!

jnothman · 2018-11-25T21:31:23Z

The what's new entry would avoid referring to private API. It should instead mention affected estimators or public functions. Please submit a pr fixing this. Thanks

marctorsoc · 2018-11-26T00:01:49Z

The what's new entry would avoid referring to private API. It should instead mention affected estimators or public functions. Please submit a pr fixing this. Thanks

sorry for that, see #12669

* print train scores when verbose > 3 in _fit_and_score * train_scores computed if verbose > 3, and 3 decimals places only * flake8 warnings solved * print train score if return_train_score test coverage increased * added test for multimetric; tests cleaned using pytest parametrize * fixing failed tests for python2... * revert changes in _scorer * modified whats_new * modified whats_new again

…12613)" This reverts commit 67c6c37.

* print train scores when verbose > 3 in _fit_and_score * train_scores computed if verbose > 3, and 3 decimals places only * flake8 warnings solved * print train score if return_train_score test coverage increased * added test for multimetric; tests cleaned using pytest parametrize * fixing failed tests for python2... * revert changes in _scorer * modified whats_new * modified whats_new again

marctorsoc added 2 commits November 18, 2018 19:28

print train scores when verbose > 3 in _fit_and_score

ef2b924

train_scores computed if verbose > 3, and 3 decimals places only

fdc8d2e

marctorsoc mentioned this pull request Nov 18, 2018

Improve GridSearchCV and splitter classes #12598

Closed

flake8 warnings solved

6820c34

jnothman reviewed Nov 19, 2018

View reviewed changes

sklearn/model_selection/_validation.py Outdated Show resolved Hide resolved

print train score if return_train_score

afe4e36

test coverage increased

added test for multimetric;

5aac1b5

tests cleaned using pytest parametrize

amueller approved these changes Nov 20, 2018

View reviewed changes

fixing failed tests for python2...

2d099f5

marctorsoc changed the title ~~[MRG] Print train scores in _fit_and_score~~ [MRG + 1] Print train scores in _fit_and_score Nov 20, 2018

jnothman approved these changes Nov 20, 2018

View reviewed changes

eamanu approved these changes Nov 20, 2018

View reviewed changes

revert changes in _scorer

cbf7c4b

marctorsoc changed the title ~~[MRG + 1] Print train scores in _fit_and_score~~ [MRG + 2] Print train scores in _fit_and_score Nov 21, 2018

modified whats_new

3b9a299

amueller reviewed Nov 21, 2018

View reviewed changes

modified whats_new again

ea6f756

amueller merged commit f6f7e3c into scikit-learn:master Nov 22, 2018

marctorsoc mentioned this pull request Nov 25, 2018

[MRG] Fixed whatsnew and tiny docstring fix #12669

Merged

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "[MRG + 2] Print train scores in _fit_and_score (scikit-learn#…

66fabc0

…12613)" This reverts commit 67c6c37.

xhluca pushed a commit to xhluca/scikit-learn that referenced this pull request Apr 28, 2019

Revert "[MRG + 2] Print train scores in _fit_and_score (scikit-learn#…

6845874

…12613)" This reverts commit 67c6c37.

marctorsoc deleted the print_train_scores_fit_and_score branch February 26, 2022 02:23

Uh oh!

[MRG + 2] Print train scores in _fit_and_score #12613

[MRG + 2] Print train scores in _fit_and_score #12613

Uh oh!

Conversation

marctorsoc commented Nov 18, 2018

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

marctorsoc commented Nov 18, 2018

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amueller commented Nov 20, 2018

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eamanu left a comment

Choose a reason for hiding this comment

Uh oh!

eamanu commented Nov 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marctorsoc commented Nov 21, 2018

Uh oh!

jnothman commented Nov 21, 2018

Uh oh!

marctorsoc commented Nov 21, 2018

Uh oh!

amueller commented Nov 21, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller commented Nov 22, 2018

Uh oh!

jnothman commented Nov 25, 2018 via email

Uh oh!

marctorsoc commented Nov 26, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eamanu commented Nov 20, 2018 •

edited

Loading