DOC Separate predefined scorer names from the ones requiring make_scorer

[marijavlajic]

Reference Issues/PRs

Related to #28671

What does this implement/fix? Explain your changes.

Table of scorer names currently in 3.3.1.1. includes also some that haven't been implemented / can't be passed as strings to the scoring parameter directly. These have now been removed from the table; a new table has been created in 3.3.1.2. (where the usage of make_scorer is explained) including the scoring functions which need to be wrapped with make_scorer.

Any other comments?

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: c9bce0c. Link to the linter CI: here}

[ogrisel]

Thanks for the PR. I think the new table should be put right after the main one and introduce this table with with a sentence that explains that those scoring functions need to be wrapped as a scorer by the user using the make_scorer function to set the main parameter of interest and link to the section about make_scorer usage for details.

[glemaitre]

@ogrisel For the first table, we have the "Comments" column. I'm wondering if for the metric that we have a comment, we should actually have the make_scorer(..) equivalent call.

This could help in terms of discoverability of make_scorer maybe?

[ogrisel]

Once the 2 comments below are addressed, I think this PR will be a net improvement. +1 for merge.

Note: the style of the table is a bit crammed but this will be addressed in other PRs, in particular in @Charlie-XIAO's work on adopting a new website theme.

Note I took this screenshot from the HTML page rendered by our CI, when clicking on the "details" link of the "Check the rendered docs here!" in the CI report below.

[ogrisel]

I like the suggestion in #28750 (comment).

Maybe add a new column named "Example usage" with the make_scorer call. If there is not enough horizontal screen real estate, we can either:

• Shorten the title of the "User-settable parameter name" column to "Parameter"
• or even drop that column entirely, because the parameter name should be explicit enough from the example snippet.

[marijavlajic]

I used your neat trick @ogrisel to look at HTML but looks like the second table is now not rendering at all?

[glemaitre]

One last thing before to merge: I think that we should move the example below above the table that you added because it does not rely on make_scorer. However, we could add a similar example to show that we can pass a scorer object obtained via make_scorer.

In this case, we have a usage example for each configuration.

[marijavlajic]

We had some back and forth already about the exact position of the table. I feel if it is below the example, and just above "Defining your scoring strategy from metric functions" section, then it might as well be within that section. Perhaps @ogrisel can chime in too?

And if I'm understanding correctly what you mean by "similar example to show that we can pass a scorer object obtained via make_scorer", then that example is in the 3.3.1.2. section already too?

[glemaitre]

So the example right now is:

Usage examples:

    >>> from sklearn import svm, datasets
    >>> from sklearn.model_selection import cross_val_score
    >>> X, y = datasets.load_iris(return_X_y=True)
    >>> clf = svm.SVC(random_state=0)
    >>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
    array([0.96..., 0.96..., 0.96..., 0.93..., 1.        ])

I feel strange that it comes right after the table that mentioning that you should make_scorer

Would it not be more friendly to have:

====
Table showing string metric
====


    >>> from sklearn import svm, datasets
    >>> from sklearn.model_selection import cross_val_score
    >>> X, y = datasets.load_iris(return_X_y=True)
    >>> clf = svm.SVC(random_state=0)
    >>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
    array([0.96..., 0.96..., 0.96..., 0.93..., 1.        ])

Narration about `make_scorer` only metric

===
Table for `make_scorer` metric
===

Usage example:

    >>> from sklearn import svm, datasets
    >>> from sklearn.metrics import fbeta_score, make_scorer
    >>> from sklearn.model_selection import cross_val_score
    >>> X, y = datasets.load_iris(return_X_y=True)
    >>> clf = svm.SVC(random_state=0)
    >>> scorer = make_scorer(fbeta_score, beta=2)
    >>> cross_val_score(clf, X, y, cv=5, scoring=scorer)
    array([...])

It might be slightly redundant with the section below but this is just a short example usage.

But indeed @ogrisel had maybe another opinion, so let's see what I think?

[marijavlajic]

Agree that this would be a more logical way to structure it, and yes, also agree on the redundancy since the next section talks about make_scorer in more detail and gives that exact same example.

[ogrisel]

Alright, let's move back the second table at the beginning of the "3.3.1.2. Defining your scoring strategy from metric functions" section. We can start the section with a short sentence such as the one you wrote:

The following metrics functions are not implemented as named scorers. They cannot be passed to the scoring parameters; instead their callable needs to be passed to make_scorer together with the value of the user-settable parameters

(without link to the section since we are already in it).

Then you can simplify the paragraph that starts with the "Many metrics are not given names to ..." to remove redundancies but keep introducing the usage example with the snippet that shows how to combine make_scorer with user settable parameters for a grid search.

Then, after the example snippet, finally insert the paragraph with the two bullet points:

This is not as important as the rest so I would move it to the end of the section but it's still useful information to speak about the naming conventions of the metric functions and how to set the higher_is_better keyword parameter in a consistent way.

[glemaitre]

I like this proposal.

[ogrisel]

LGTM! Thanks very much for the PR.

[ArturoAmorQ]

I think we are missing d2_absolute_error_score here. But this PR made me wonder if we should actually accept it as a named scorer. Thoughts on this @glemaitre, @ogrisel ?

Of course, that would be done in another PR. As this one already looks in good shape.

[glemaitre]

Yep this would be for another PR. I'm not sure that there is a meaningful default for all use-case and it might be better to make sure that a user choose the parameter each time.

@ogrisel will have better insights on these metrics indeed.

[betatim]

There was a discussion in: #28750 (comment)

I think it was removed because you do not have to provide a parameter to make_scorer. I also first thought that d2_absolute_error_score got forgotten. Maybe that is a sign we should add a sentence (in a future PR?) to the table to say "there are more metrics, this is just a subset"?

[ArturoAmorQ]

The thing is that d2_absolute_error_score is already equivalent to make_scorer(d2_pinball_score, alpha=0.5). Accepting it as a named scorer (a string to be passed to scoring) is redundant but could be a common enough practice to be worth the shortcut. Indeed, its presence in model_evaluation.rst L105 makes me think it was meant to be the case, but the OP from #22118 forgot to add it to the _SCORERS dict.

[ogrisel]

Let's do this in a dedicated follow-up PR (and document it as part of the first table for named scorers that do not require setting parameters).

[ArturoAmorQ]

Just a small change to reflect the decision in https://github.com/scikit-learn/scikit-learn/pull/28750/files#r1559656259. Otherwise LGTM!

[betatim]

I committed the change and enabled auto-merge