Add equal opportunity metric to fairness analysis

[SalesforceRocks]

User Story

As a data scientist evaluating model fairness,
I want to compute the equal opportunity metric across protected groups in my model's predictions
so that I can assess whether qualified individuals (true positive cases) are treated equitably across groups, regardless of differences in false positive rates.

Context

Equal opportunity measures whether true positive rates (TPR) are equal across protected groups. It is a deliberate relaxation of equalized odds — equalized odds requires both TPR and FPR equality, while equal opportunity requires only TPR equality.

• Formal definition: P(Y_hat=1 | Y=1, A=a) = P(Y_hat=1 | Y=1, A=b) for all groups a, b
• TPR (recall) per group: (true positives for group) / (total actual positives for group)
• Ratio version: min(TPR_a, TPR_b, ...) / max(TPR_a, TPR_b, ...) >= threshold (default 0.8)
• When to use: When the primary concern is that creditworthy, qualified, or eligible individuals have an equal chance of receiving a positive outcome (e.g., loan approval for creditworthy applicants, job offers for qualified candidates)

The metric requires actual outcomes (ground truth labels). It only examines the positive class (Y=1); individuals with Y=0 do not contribute to this metric.

Acceptance Criteria

AC1: Compute equal opportunity for two groups

• Given a Dataset with binary predictions, binary actuals, and two distinct group labels (e.g., "group_a", "group_b") where each group contains at least one actual positive (actual=1),
  When I run the equal opportunity metric,
  Then I receive a FairnessReport where metric_name is "equal_opportunity",
  And the FairnessReport contains a GroupMetric for each group where metric_value is the true positive rate (TPR) for that group,
  And overall_pass is True when min(TPR) / max(TPR) >= 0.8 (default threshold),
  And overall_pass is False when min(TPR) / max(TPR) < 0.8

AC2: Compute equal opportunity for three or more groups

• Given a Dataset with binary predictions, binary actuals, and three or more distinct group labels where each group has at least one actual positive,
  When I run the equal opportunity metric,
  Then I receive a FairnessReport with a GroupMetric for each group,
  And overall_pass is determined by min(TPR across all groups) / max(TPR across all groups) >= threshold

AC3: Custom threshold support

• Given a Dataset with two or more groups,
  When I run the equal opportunity metric with a custom threshold of 0.9,
  Then overall_pass is evaluated against 0.9 instead of the default 0.8,
  And the threshold field in the FairnessReport reflects 0.9

AC4: Threshold validation

• Given a threshold value less than 0.0 or greater than 1.0,
  When I attempt to run the equal opportunity metric,
  Then a validation error is raised before any computation occurs,
  And the error message indicates the threshold must be between 0.0 and 1.0 (inclusive)

AC5: Edge case — group with no actual positives

• Given a Dataset where one group has zero actual positives (all actuals are 0 for that group),
  When I run the equal opportunity metric,
  Then that group is excluded from the TPR calculation and from the FairnessReport,
  And a warning is surfaced indicating which group(s) were excluded due to no actual positives,
  And no division-by-zero error occurs

AC6: Edge case — group with actual positives but no true positives (TPR = 0.0)

• Given a Dataset where one group has actual positives but the model predicted 0 for all of them (TPR = 0.0 for that group),
  When I run the equal opportunity metric,
  Then the TPR for that group is 0.0,
  And overall_pass is False (since 0.0 / any_positive_TPR < any threshold > 0),
  And no division-by-zero error occurs

AC7: Edge case — all groups have identical TPR

• Given a Dataset where all groups have the same true positive rate,
  When I run the equal opportunity metric,
  Then overall_pass is True,
  And the ratio equals 1.0

AC8: Edge case — single group with actual positives

• Given a Dataset with only one group that has actual positives (either one group total, or all other groups excluded due to no actual positives),
  When I run the equal opportunity metric,
  Then overall_pass is True (no disparity is possible across a single group),
  And the FairnessReport contains one GroupMetric with that group's TPR

AC9: CLI integration

• Given a CSV file with prediction, actual, and group columns where at least one group has actual positives,
  When I run fairness-checker equal-opportunity --file data.csv --threshold 0.8,
  Then the FairnessReport is printed to stdout in a human-readable format showing each group's TPR and the overall pass/fail result,
  And the exit code is 0 when overall_pass is True,
  And the exit code is 1 when overall_pass is False

AC10: CLI error — file not found

• Given a file path that does not exist,
  When I run fairness-checker equal-opportunity --file nonexistent.csv,
  Then an error message is displayed indicating the file was not found,
  And the exit code is non-zero

AC11: CLI error — missing required columns

• Given a CSV file that is missing the prediction, actual, or group column,
  When I run fairness-checker equal-opportunity --file data.csv,
  Then an error message is displayed naming the missing column(s),
  And the exit code is non-zero

Out of Scope

• Equalized odds (joint TPR + FPR equality) — a separate metric
• Continuous/probabilistic predictions — binary predictions (0/1) only
• Statistical significance testing for observed TPR differences
• Intersectional groups (e.g., race × gender combinations)
• Changes to the Dataset, GroupMetric, or FairnessReport Pydantic models — use existing models as-is; propose changes during design if needed

Technical Notes

These notes describe existing patterns to follow. They do not prescribe implementation — the developer selects the approach during design.

• Existing models (src/fairness_checker/models.py): Dataset, GroupMetric, and FairnessReport are the established data contracts. Dataset.to_dataframe() converts to a pandas DataFrame with columns prediction, actual, group.
• CLI skeleton (src/fairness_checker/cli.py): The cli click group is where the new equal-opportunity subcommand registers. See issue Add demographic parity metric to fairness analysis #3 (demographic parity) for the expected CLI command structure and exit code behaviour.
• actuals are available: Unlike demographic parity, this metric uses the actual column. Dataset already carries actuals.
• TPR formula: For a group, TPR = sum(prediction == 1 AND actual == 1) / sum(actual == 1). Groups where sum(actual == 1) == 0 must be handled explicitly.
• Test pattern: Each AC maps to one or more pytest test functions. Tests use Dataset directly (unit) and the click test runner (CLI). Follow the test patterns established for demographic parity once that issue is implemented.

Dependencies

• Blocks: None
• Blocked by: None
• Related to: Add demographic parity metric to fairness analysis #3 (Add demographic parity metric — establishes the CLI command pattern and metric module structure this story follows)

Test Plan

All ACs are automatically testable:

AC	Test type	Description
AC1	Unit	Two-group dataset, verify GroupMetric values and overall_pass
AC2	Unit	Three-group dataset, verify all groups present and ratio logic
AC3	Unit	Pass threshold=0.9, verify FairnessReport.threshold and pass/fail
AC4	Unit	Pass threshold=-0.1 and 1.1, verify ValidationError raised
AC5	Unit	One group has no actual positives, verify exclusion and warning
AC6	Unit	One group has actual positives but TPR=0.0, verify overall_pass=False
AC7	Unit	All groups identical TPR, verify ratio=1.0 and overall_pass=True
AC8	Unit	Single eligible group, verify overall_pass=True
AC9	CLI (click test runner)	Valid CSV, verify stdout format and exit codes
AC10	CLI (click test runner)	Non-existent file path, verify error message and non-zero exit
AC11	CLI (click test runner)	CSV missing columns, verify error message and non-zero exit