fix(areaScores): consistency b/w aupr auroc #26176

a-parida12 · 2023-04-14T04:29:56Z

Reference Issues/PRs

Fixes #24381

What does this implement/fix? Explain your changes.

replace ValueError with Userwarning
update -0.0 with 0.0

glevv · 2023-07-28T05:06:29Z

Hi! You need to update tests so that it will expect warning and not value error

lucyleeow · 2023-10-25T00:46:56Z

@a-parida12 would you still like to work on this PR?

a-parida12 · 2023-10-25T18:39:32Z

@lucyleeow could you please help me out, by pointing to me what needs to be done to be able to merge it? I updated the changes from the main to the current branch.

github-actions · 2023-10-25T18:40:34Z

❌ Linting issues

This PR is introducing linting issues. Here's a summary of the issues. Note that you can avoid having linting issues by enabling pre-commit hooks. Instructions to enable them can be found here.

You can see the details of the linting issues under the lint job here

`black`

black detected issues. Please run black . locally and push the changes. Here you can see the detected issues. Note that running black might also fix some of the issues which might be detected by ruff. Note that the installed black version is black=23.3.0.


error: cannot format /home/runner/work/scikit-learn/scikit-learn/sklearn/metrics/_ranking.py: Cannot parse: 387:8:         warnings.warn(

Oh no! 💥 💔 💥
906 files would be left unchanged, 1 file would fail to reformat.

`ruff`

ruff detected issues. Please run ruff --fix --show-source . locally, fix the remaining issues, and push the changes. Here you can see the detected issues. Note that the installed ruff version is ruff=0.1.11.


error: Failed to parse sklearn/metrics/_ranking.py:387:9: Unexpected token 'warnings'
sklearn/metrics/_ranking.py:387:9: E999 SyntaxError: Unexpected token 'warnings'
    |
385 |             "Only one class present in y_true. ROC AUC score "
386 |             "is not defined in that case."
387 |         warnings.warn(
    |         ^ E999
388 |             "Only one class present in y_true. ROC AUC score is not defined in that "
389 |             "case. The value 0.0 is returned.",
    |

Found 1 error.

`mypy`

mypy detected issues. Please fix them locally and push the changes. Here you can see the detected issues. Note that the installed mypy version is mypy=1.3.0.


sklearn/metrics/_ranking.py:384: error: '(' was never closed  [syntax]
Found 1 error in 1 file (errors prevented further checking)

_{Generated for commit: 5757d7e. Link to the linter CI: here}

lucyleeow · 2023-10-26T01:42:32Z

As suggested above, the tests need to pass, test_auc_score_non_binary_class is failing, see CI: https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=60356&view=logs&j=7d13af21-9cb9-5d40-483b-ea0f074409c6&t=97bcaf17-ac30-535a-f717-a6f6663c5884&l=1529

a-parida12 · 2023-11-01T22:08:29Z

@lucyleeow thanks for pointing it out to me... I will get working on it.

a-parida12 · 2023-11-05T14:28:47Z

@lucyleeow @glevv this pull request is ready!

glevv

I think you should add a UndefinedMetricWarning type here instead of UserWarning

a-parida12 · 2023-11-15T16:32:51Z

@glevv cool I will update the UserWarning to the UndefinedMetricWarning

glevv · 2023-11-15T17:10:40Z

Will need a reviewer on this one
ping @Micky774 @adrinjalali @ogrisel @glemaitre

Micky774 · 2023-11-15T17:17:27Z

sklearn/metrics/tests/test_common.py

-            with pytest.raises(ValueError):
-                metric(y1_row, y2_row)
+            if "roc_auc" not in name:
+                with pytest.raises(ValueError):


Out of curiosity, which metrics currently hit this branch?

The following:

adjusted_balanced_accuracy_score

balanced_accuracy_score

cohen_kappa_score

d2_tweedie_score

hinge_loss

matthews_corrcoef_score

max_error

mean_compound_poisson_deviance

mean_gamma_deviance

mean_normal_deviance

mean_poisson_deviance

normalized_confusion_matrix

ovo_roc_auc

ovr_roc_auc

top_k_accuracy_score

unnormalized_confusion_matrix

weighted_ovo_roc_auc

weighted_ovr_roc_auc

Just to sanity check then, do we want to propogate the preference for a warning over an error to any of these as well?

glemaitre · 2023-11-15T17:21:09Z

I would prefer to have to separate pull-request:

one with the issue related to the negative zero
one related to the warning

Indeed, we will need two different entries in the changelog

sklearn/metrics/_ranking.py

glemaitre · 2023-11-15T17:26:13Z

sklearn/metrics/tests/test_ranking.py

@@ -363,7 +363,10 @@ def test_roc_curve_toydata():
    with pytest.warns(UndefinedMetricWarning, match=expected_message):
        tpr, fpr, _ = roc_curve(y_true, y_score)

-    with pytest.raises(ValueError):
+    expected_message = (


You will need to update the message and type of warning.

glemaitre · 2023-11-15T17:34:19Z

sklearn/metrics/tests/test_common.py

-            with pytest.raises(ValueError):
-                metric(y1_row, y2_row)
+            if "roc_auc" not in name:
+                with pytest.raises(ValueError):


The following:

adjusted_balanced_accuracy_score

balanced_accuracy_score

cohen_kappa_score

d2_tweedie_score

hinge_loss

matthews_corrcoef_score

max_error

mean_compound_poisson_deviance

mean_gamma_deviance

mean_normal_deviance

mean_poisson_deviance

normalized_confusion_matrix

ovo_roc_auc

ovr_roc_auc

top_k_accuracy_score

unnormalized_confusion_matrix

weighted_ovo_roc_auc

weighted_ovr_roc_auc

Co-authored-by: Guillaume Lemaitre <g.lemaitre58@gmail.com>

glemaitre · 2024-01-09T15:13:51Z

The CIs are failing.

fix(areaScores): consistency b/w aupr auroc

29e7ea3

github-actions bot added the module:metrics label Apr 14, 2023

Merge branch 'main' into api-consistency-aupr-auroc

5a43a14

glevv mentioned this pull request Sep 20, 2023

FIX make roc_auc_score consistent with roc_curve with a single class #27412

Open

Merge branch 'main' into api-consistency-aupr-auroc

8e0a39e

a-parida12 added 9 commits November 4, 2023 04:17

Merge branch 'scikit-learn:main' into api-consistency-aupr-auroc

e4d5fd5

test(aucscore): update to check warning

537564a

test(warning): check for warning

aac4f81

test(warning): warnign for auc only

2294e1a

lint(test): update black

d2cd0f4

linting and warning checking

a5e4cdd

more update value error to warn

065d107

test(toy data): update warnings

2258f88

Merge branch 'main' into api-consistency-aupr-auroc

7adee60

glevv reviewed Nov 9, 2023

View reviewed changes

Micky774 reviewed Nov 15, 2023

View reviewed changes

glemaitre self-requested a review November 15, 2023 17:19

glemaitre reviewed Nov 15, 2023

View reviewed changes

a-parida12 and others added 2 commits January 5, 2024 09:30

Merge branch 'scikit-learn:main' into api-consistency-aupr-auroc

ea7af55

Update sklearn/metrics/_ranking.py

27f1643

Co-authored-by: Guillaume Lemaitre <g.lemaitre58@gmail.com>

Update sklearn/metrics/_ranking.py

5757d7e

Co-authored-by: Guillaume Lemaitre <g.lemaitre58@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(areaScores): consistency b/w aupr auroc #26176

fix(areaScores): consistency b/w aupr auroc #26176

a-parida12 commented Apr 14, 2023

glevv commented Jul 28, 2023

lucyleeow commented Oct 25, 2023

a-parida12 commented Oct 25, 2023

github-actions bot commented Oct 25, 2023 •

edited

lucyleeow commented Oct 26, 2023

a-parida12 commented Nov 1, 2023

a-parida12 commented Nov 5, 2023

glevv left a comment •

edited

a-parida12 commented Nov 15, 2023

glevv commented Nov 15, 2023

Micky774 Nov 15, 2023

glemaitre Nov 15, 2023

Micky774 Nov 15, 2023

glemaitre commented Nov 15, 2023

glemaitre Nov 15, 2023

glemaitre Nov 15, 2023

glemaitre commented Jan 9, 2024

fix(areaScores): consistency b/w aupr auroc #26176

Are you sure you want to change the base?

fix(areaScores): consistency b/w aupr auroc #26176

Conversation

a-parida12 commented Apr 14, 2023

Reference Issues/PRs

What does this implement/fix? Explain your changes.

glevv commented Jul 28, 2023

lucyleeow commented Oct 25, 2023

a-parida12 commented Oct 25, 2023

github-actions bot commented Oct 25, 2023 • edited

❌ Linting issues

black

ruff

mypy

lucyleeow commented Oct 26, 2023

a-parida12 commented Nov 1, 2023

a-parida12 commented Nov 5, 2023

glevv left a comment • edited

Choose a reason for hiding this comment

a-parida12 commented Nov 15, 2023

glevv commented Nov 15, 2023

Micky774 Nov 15, 2023

Choose a reason for hiding this comment

glemaitre Nov 15, 2023

Choose a reason for hiding this comment

Micky774 Nov 15, 2023

Choose a reason for hiding this comment

glemaitre commented Nov 15, 2023

glemaitre Nov 15, 2023

Choose a reason for hiding this comment

glemaitre Nov 15, 2023

Choose a reason for hiding this comment

glemaitre commented Jan 9, 2024

github-actions bot commented Oct 25, 2023 •

edited

`black`

`ruff`

`mypy`

glevv left a comment •

edited