Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weighted average scores no nan #14595

Merged

Conversation

@amueller
Copy link
Member

amueller commented Aug 7, 2019

Finishes #10891 by adding a regression test.
If there's no positives the code raises a division by zero warning, and there's a test that the warning is raised. I'm not sure if that's useful, but I'm keeping with the current behavior. For the case in #10891 I'm still raising the warning but providing a useful number.

kyu-sz and others added 5 commits Mar 29, 2018
Some metrics can produce NaN scores which are however 0-weighted (e.g. PR curve score calculated on class with no positive ground truth samples in the batch). When directly taking weighted average of them, the NaN scores can cause the average score to be NaN, even if they are 0-weighted.

To prevent this, before taking the average, we force the scores with 0 weight to be 0, so that they will not pose effect on the final average score.
@amueller

This comment has been minimized.

Copy link
Member Author

amueller commented Aug 12, 2019

These failures are somewhat disconcerning, in particular because they are version-dependent?!

@amueller

This comment has been minimized.

Copy link
Member Author

amueller commented Aug 12, 2019

I fix

amueller added 2 commits Aug 12, 2019
@amueller

This comment has been minimized.

Copy link
Member Author

amueller commented Aug 13, 2019

actually good for reviews now.

Copy link
Member

jnothman left a comment

A small change log entry?

@amueller

This comment has been minimized.

Copy link
Member Author

amueller commented Aug 13, 2019

What should it say? Allowing averaging metrics in the absence of true positives? Sure.

@jnothman

This comment has been minimized.

Copy link
Member

jnothman commented Aug 13, 2019

@adrinjalali adrinjalali merged commit 1a14920 into scikit-learn:master Aug 14, 2019
17 checks passed
17 checks passed
LGTM analysis: C/C++ No code changes detected
Details
LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python No new or fixed alerts
Details
ci/circleci: deploy Your tests passed on CircleCI!
Details
ci/circleci: doc Your tests passed on CircleCI!
Details
ci/circleci: doc-min-dependencies Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
codecov/patch 100% of diff hit (target 96.87%)
Details
codecov/project 96.88% (+<.01%) compared to 3eacf94
Details
scikit-learn.scikit-learn Build #20190813.75 succeeded
Details
scikit-learn.scikit-learn (Linux py35_conda_openblas) Linux py35_conda_openblas succeeded
Details
scikit-learn.scikit-learn (Linux py35_ubuntu_atlas) Linux py35_ubuntu_atlas succeeded
Details
scikit-learn.scikit-learn (Linux pylatest_conda_mkl_pandas) Linux pylatest_conda_mkl_pandas succeeded
Details
scikit-learn.scikit-learn (Linux32 py35_ubuntu_atlas_32bit) Linux32 py35_ubuntu_atlas_32bit succeeded
Details
scikit-learn.scikit-learn (Windows py35_pip_openblas_32bit) Windows py35_pip_openblas_32bit succeeded
Details
scikit-learn.scikit-learn (Windows py37_conda_mkl) Windows py37_conda_mkl succeeded
Details
scikit-learn.scikit-learn (macOS pylatest_conda_mkl) macOS pylatest_conda_mkl succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.