Aggregation

[csudre]

Creation of relevant strategies when having to deal with NaN values in particular for the final aggregation.

Summary by CodeRabbit

• New Features

  • Added worst-case distance calculation for edge-case scenarios
  • Added NaN-replacement columns for segmentation metrics with configurable fallback values (best/worst based on empty reference/prediction conditions)

• Bug Fixes

  • Fixed dimension detection when prediction and reference have matching size but different dimensionality
  • Improved handling of empty prediction/reference cases across multiple metrics

• Behavior Changes

  • Metrics now return NaN instead of default numeric values when reference lacks required positives or negatives (affects Youden index, balanced accuracy, false positive rate, Dice, F-beta, and distance-based measures)

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ca194f1d-2c98-43d3-b7c4-a88fc5d83272

📥 Commits

Reviewing files that changed from the base of the PR and between cb38dfc and 2bf9c91.

📒 Files selected for processing (6)

• MetricsReloaded/metrics/pairwise_measures.py
• MetricsReloaded/metrics/prob_pairwise_measures.py
• MetricsReloaded/processes/mixed_measures_processes.py
• MetricsReloaded/processes/overall_process.py
• test/test_metrics/test_pairwise_measures.py
• test/test_processes/test_overall_process.py

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to data retention organization setting

---

Walkthrough

The changes implement comprehensive handling of empty predictions and references across the metrics library. BinaryPairwiseMeasures now calculates a worse_dist value and returns NaN instead of numeric fallbacks in several metrics when predictions or references lack required positives/negatives. ProbabilityPairwiseMeasures tracks empty flags, while MultiLabelPairwiseMeasures integrates worse_dist reporting. ProcessEvaluation adds NaN replacement columns that conditionally substitute NaN values based on which prediction/reference was empty.

Changes

Cohort / File(s)	Summary
Core Metrics Infrastructure MetricsReloaded/metrics/pairwise_measures.py	Added calculate_worse_dist() method; modified __init__ to cast sum to int and compute worse_dist; updated 13 metrics (youden_index, balanced_accuracy, false_positive_rate, normalised_expected_cost, positive_predictive_value, recall, dsc, fbeta, negative_predictive_value, centreline_dsc, boundary_iou, normalised_surface_distance, measured_distance) to return NaN instead of numeric fallbacks for empty cases; added caching decorator to measured_distance.
Probability Measures MetricsReloaded/metrics/prob_pairwise_measures.py	Added cached n_pos_pred() method; added instance attributes flag_ref_empty and flag_pred_empty in __init__ to track empty states.
Multi-Label Process Integration MetricsReloaded/processes/mixed_measures_processes.py	Added module-level list_distance constant; introduced squeeze_ref_and_pred_to_size() method to align dimensionality; updated per_label_dict() to include worse_dist and check_empty fields tracking which prediction/reference was empty.
Overall Process NaN Replacement MetricsReloaded/processes/overall_process.py	Added module-level LIST_DISTANCE, BEST, and NAN_LIST constants; added create_mapping_column_nan_replaced_seg() method that creates <measure>_nanrep columns replacing NaN values conditionally based on empty flags; modified process_data() to call the new NaN replacement method; added stub identify_empty_ref() method.
Test Updates test/test_metrics/test_pairwise_measures.py, test/test_processes/test_overall_process.py	Commented out and updated distance empty-case tests to expect NaN; added test_calculate_worse_dist() verifying worse_dist computation; updated empty reference/prediction assertions to expect NaN returns; updated process test to verify NaN replacement columns with new data_nan dataset and test_op_nanreplacement() test.

Sequence Diagram

sequenceDiagram
    participant Metrics as BinaryPairwiseMeasures
    participant MultiLabel as MultiLabelPairwiseMeasures
    participant Process as ProcessEvaluation
    participant Output as NaN-Replaced Columns

    Metrics->>Metrics: Compute empty flags (flag_empty_pred/ref)
    Metrics->>Metrics: Calculate metrics (returns NaN if empty)
    Metrics->>Metrics: Compute worse_dist from ref shape
    
    MultiLabel->>Metrics: Aggregate per-label metrics
    Metrics-->>MultiLabel: Return metrics + empty flags + worse_dist
    MultiLabel->>MultiLabel: Store check_empty field
    
    Process->>MultiLabel: Retrieve resseg data
    MultiLabel-->>Process: Return resseg with check_empty, worse_dist
    
    Process->>Process: For each NaN metric in resseg
    alt check_empty == 'Both'
        Process->>Output: Replace NaN with BEST value
    else check_empty == 'Ref' or 'Pred'
        Process->>Output: Replace NaN with WORSE value
    end
    
    alt metric in LIST_DISTANCE
        Process->>Output: Replace NaN with worse_dist
    end
    
    Output-->>Process: Create <measure>_nanrep columns

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 Whiskers twitching with glee:
When metrics go empty and NaN appears,
This code hops to handle it clear,
With worse_dist computed and flags all set,
NaN replacements—the best safety net!
Thump thump! 🐇

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch aggregation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}