ENH Array API support for f1_score and multilabel_confusion_matrix

[OmarManzoor]

Reference Issues/PRs

Towards #26024

What does this implement/fix? Explain your changes.

• Adds array api support for f1_score and the functions related to it.

Any other comments?

CC: @ogrisel @betatim

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 3b3555b. Link to the linter CI: here}

[OmarManzoor]

@ogrisel Could you kindly have a look at this PR?

[ogrisel]

Overall this looks good to me. I am surprised it works without being very specific about device and dtypes, but as long as the tests (and they do), I am happy.

[ogrisel]

We should probably make sure that it matches the device of the inputs, no? It's curious that existing tests do not fail with PyTorch and MPS device (or cuda devices).

I am also wondering of whether we should convert to a specific dtype. However looking at the tests I never see any case where we pass non-integer sample weights. And even for integer weights, it's only done to check an error message, not to check an actual computation. So I am not sure our sample_weight support is correct, even outside of array API concerns.

I guess this is only indirectly tested by classification metrics that rely on multilabel_confusion_matrix internally. But then the array API compliance tests for F1 score do not fail with floating point weights (I just checked) and I am not sure why.

[ogrisel]

Here is the output of my cuda run on this PR (updated to check that boolean array indexing also works, but this should be orthogonal):

$ pytest -vlx -k "array_api and f1_score" sklearn/ 

================================================================================================== test session starts ===================================================================================================
platform linux -- Python 3.10.12, pytest-7.4.2, pluggy-1.3.0
collected 34881 items / 34863 deselected / 2 skipped / 18 selected                                                                                                                                                       

sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-numpy-None-None] PASSED                                                                      [  5%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-array_api_strict-None-None] PASSED                                                           [ 11%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-cupy-None-None] PASSED                                                                       [ 16%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-cupy.array_api-None-None] PASSED                                                             [ 22%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cpu-float64] PASSED                                                                    [ 27%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cpu-float32] PASSED                                                                    [ 33%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cuda-float64] PASSED                                                                   [ 38%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cuda-float32] PASSED                                                                   [ 44%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-mps-float32] SKIPPED (Skipping MPS device test because PYTORCH_ENABLE_MPS_FALLBACK...) [ 50%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-numpy-None-None] PASSED                                                                  [ 55%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-array_api_strict-None-None] PASSED                                                       [ 61%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-cupy-None-None] PASSED                                                                   [ 66%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-cupy.array_api-None-None] PASSED                                                         [ 72%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cpu-float64] PASSED                                                                [ 77%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cpu-float32] PASSED                                                                [ 83%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cuda-float64] PASSED                                                               [ 88%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cuda-float32] PASSED                                                               [ 94%]
sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-mps-float32] SKIPPED (Skipping MPS device test because PYTORCH_ENABLE_MPS_FALL...) [100%]

============================================================================= 16 passed, 4 skipped, 34863 deselected, 105 warnings in 15.59s =============================================================================

[OmarManzoor]

@ogrisel @betatim Does this look okay now?

[ogrisel]

I do not have time to finish the review today but here is some quick feedback:

[ogrisel]

I merged main to be able to launch the new CUDA GPU CI workflow on this PR:

• https://github.com/scikit-learn/scikit-learn/actions/runs/9387261905

EDIT: tests are green.

[ogrisel]

LGTM once the following is addressed:

[ogrisel]

@betatim this is ready for a second review.

[ogrisel]

I launched the CUDA GPU CI at:

• https://github.com/scikit-learn/scikit-learn/actions/runs/9402808393

EDIT: CUDA tests are green.

[OmarManzoor]

@betatim Could you kindly review this PR?

[OmarManzoor]

@ogrisel @betatim Since multilabel confusion matrix seems to work fine I added that in this PR. I think it might be possible to handle precision, recall and fbeta_score in one PR as well, once this is merged. I also added a generic function in the tests to get any additional params combinations.

[OmarManzoor]

@betatim I added the handling for the multi-label case as well. Could you kindly have a look?
I verified that the cuda and mps tests pass.

[ogrisel]

LGTM again with the extra commits after my last review.

[ogrisel]

Note: this was introduced in #27381 but was missing from the array API doc.

[ogrisel]

Merged main and labeled with "CUDA CI" to check that the tests still pass with the recent CI config and dependency updates.

[OmarManzoor]

@ogrisel I think the tests are failing because in the error messages we have numpy._str values that arise because numpy 2.0 supports the array api by default. Do we want to fix such errors by turning such objects to strs before creating the error message or do we need to place a check at the place where the creation of numpy._str objects is done?