Skip to content

fix: Compute correct ROC-AUC score with link_prediction labels in one class#3845

Merged
colinbarry merged 5 commits intomasterfrom
fix/mage-roc-auc-score
Mar 25, 2026
Merged

fix: Compute correct ROC-AUC score with link_prediction labels in one class#3845
colinbarry merged 5 commits intomasterfrom
fix/mage-roc-auc-score

Conversation

@colinbarry
Copy link
Copy Markdown
Contributor

@colinbarry colinbarry commented Feb 27, 2026

Fixes a flaky MAGE tests where occasionally, the graph - which is randomly generated - can have all labels in a single class. This is because ROC-AUC is undefined without both positives and negatives. This change returns a neutral 0.5 AUC in that edge case. Also adds deterministic tests for all-positive, all-negative, and mixed-label scenarios.


Note

Low Risk
Small, localized change to metric computation with added tests; behavior only differs in edge cases where prior code errored or produced inconsistent confusion-matrix shapes.

Overview
Fixes flaky link_prediction_util.evaluate() metric computation when a batch contains only one label class.

roc_auc_score is now guarded to return 0.5 for single-class labels, and confusion-matrix values are computed lazily with labels=[0,1] only when TP/FP/TN/FN metrics are requested. Adds pytest coverage for AUC on all-positive, all-negative, and mixed-label cases.

Written by Cursor Bugbot for commit ac8ed5f. This will update automatically on new commits. Configure here.

@colinbarry
Copy link
Copy Markdown
Contributor Author

colinbarry commented Feb 27, 2026

Tracking

  • [Link to Epic/Issue]

Standard development

CI Testing Labels

  • Select the appropriate CI test labels (CI -build=build-name -test=test-suite)

Documentation checklist

  • Add the documentation label
  • Add the bug / feature label
  • Add the milestone for which this feature is intended
    • If not known, set for a later milestone
  • Write a release note, including added/changed clauses
    • link_prediction metric evaluation now handles single-class label batches safely, returning a neutral AUC_SCORE of 0.5 when a batch contains only one class. #3845

@colinbarry colinbarry force-pushed the fix/mage-roc-auc-score branch from 018f3d6 to ac8ed5f Compare March 5, 2026 09:29
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented Mar 5, 2026

@colinbarry
Copy link
Copy Markdown
Contributor Author

BugBot run

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

@colinbarry colinbarry added the Docs - changelog only Docs - changelog only label Mar 5, 2026
@colinbarry colinbarry requested a review from DavIvek March 5, 2026 11:03
@colinbarry colinbarry marked this pull request as ready for review March 5, 2026 11:03
@colinbarry colinbarry modified the milestones: mg-v3.9.0, mg-v3.10.0 Mar 9, 2026
@colinbarry colinbarry added this pull request to the merge queue Mar 25, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Mar 25, 2026
@colinbarry colinbarry added this pull request to the merge queue Mar 25, 2026
Merged via the queue into master with commit 55d4e0d Mar 25, 2026
45 checks passed
@colinbarry colinbarry deleted the fix/mage-roc-auc-score branch March 25, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants