Feature: standardize inputs/outputs of metrics

[lvwerra]

Currently there are several different inputs/output formats possible in Metrics. We should standardize them as much as possible and respecting the following principle:

• inputs/outputs are easy to understand and use
• outputs are compatible with other frameworks

For the output standardization: probably a dictionary structure, even if nested would be ok. Also a dedicated output class could be considered like in transformer models but this is probably not necessary here. To make it compatible with e.g. keras we could add a postprocess function at initialization similar a transform in datasets.

There are three options we could implement:

load_metric(..., postprocess="metric_key") # equivalent result to `metric.compute()["metric_key"]`
load_metric(..., postprocess="flatten") # equivalent flattening the output dict: `flatten(metric.compute())`
load_metric(..., postprocess=func) # equivalent result to `func(metric.compute())`

[sashavor]

As per our meeting today, we proposed to have standardized structure for inputs, in dictionary form.

An initial proposal of that structure can be:

     {
        "references":  ,
        "predictions": ,
     }

With references and predictions being of any format (strings, images, numbers, vectors, etc.). I was looking at examples of computer vision metrics and this should work for those as well.

Edge cases:

• COMET, WikiSplit and SARI -- take an additional input, sources
• F1, Precision and Recall -- require average parameter for multiclass labels, but we could define a default if needed
• Perplexity -- needs an input string and a model

I think we could have additional, optional, source and average inputs, but I don't really know what to so for perplexity 😅 (I mean, in any case, the metrics will not function without these arguments, but I guess waiting for them to crash isn't the best solution)

CC @lvwerra @lhoestq @apsdehal

[lhoestq]

Just to distinguish the two cases: sources is part of the input data to the metric (one source per sample), and average is a mandatory parameter of the metric in the multilabel setting.

I think it's fine if we allow certain metrics to have extra inputs like sources - but it must be possible to know that in advance, maybe we can provide a property to the Metric object that lists the expected inputs, and extra inputs should have standard names as well.

It's also fine to require certain metric parameters depending on the task like for F1, we can define that when we start to map metrics to tasks.

Regarding perplexity I guess it will depend on how it's used:

• Is it a tool to evaluate a model based on its predictions ? Does it have to fit the other metrics formats to be used in automated evaluations ? In this case it should probably take a model output as metric input, maybe the logits as "predictions" ?
• it is a measurement tool for data ? In this case the current implementation with "input_texts" is the most convenient.

[sashavor]

Sorry, indeed, I should have distinguished inputs and parameters :)

…

On Mon, Apr 11, 2022 at 6:00 AM Quentin Lhoest ***@***.***> wrote: Just to distinguish the two cases: sources is part of the input data to the metric (one source per sample), and average is a mandatory parameter of the metric in the multilabel setting. I think it's fine if we allow certain metrics to have extra inputs like sources - but it must be possible to know that in advance, maybe we can provide a property to the Metric object that lists the expected inputs, and extra inputs should have standard names as well. It's also fine to require certain metric parameters depending on the task like for F1, we can define that when we start to map metrics to tasks. Regarding perplexity I guess it will depend on how it's used: - Is it a tool to evaluate a model based on its predictions ? Does it have to fit the other metrics formats to be used in automated evaluations ? In this case it should probably take a model output as metric input, maybe the logits as "predictions" ? - it is a measurement tool for data ? In this case the current implementation with "input_texts" is the most convenient. — Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADMMIIWGLXFIHBVFA7GKZE3VEPZ2TANCNFSM5SWIFFNQ> . You are receiving this because you commented.Message ID: ***@***.***>

-- Sasha Luccioni, PhD Postdoctoral Researcher (Mila, Université de Montréal) Chercheure postdoctorale (Mila, Université de Montréal) https://www.sashaluccioni.com/ [image: Image result for universite de montreal logo]

[lvwerra]

Also regarding perplexity:
The currently implemented version aims at being a data measurement tool if I understand correctly - you provide text and model. Although one could evaluate a model in that setting it would be less efficient as it requires an additional forward pass. So we could move this version of perplexity to a new folder (e.g. measurements) and load it with load_measurement("perplexity") in the future. Since these tools will measure different modalities should we call it input_data to make it agnostic?

[sashavor]

Sure, that could work! I mean, having two separate versions of the code seems a bit redundant, but I agree that the way it's implemented now makes it stand out from the other metrics :)

…

On Tue, Apr 12, 2022 at 8:05 AM Leandro von Werra ***@***.***> wrote: Also regarding perplexity: The currently implemented version aims at being a data measurement tool if I understand correctly - you provide text and model. Although one could evaluate a model in that setting it would be less efficient as it requires an additional forward pass. So we could move this version of perplexity to a new folder (e.g. measurements) and load it with load_measurement("perplexity") in the future. Since these tools will measure different modalities should we call it input_data to make it agnostic? — Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADMMIIQCGMJ6D5YXY4FTGWTVEVRHLANCNFSM5SWIFFNQ> . You are receiving this because you commented.Message ID: ***@***.***>

-- Sasha Luccioni, PhD Postdoctoral Researcher (Mila, Université de Montréal) Chercheure postdoctorale (Mila, Université de Montréal) https://www.sashaluccioni.com/ [image: Image result for universite de montreal logo]

[lvwerra]

Postprocessing

IIRC we discussed for the postprocessing step that a more flexible alternative is to add a method to the class such that we could do:

>>> metric = load_metric("bleu")
>>> metric.output_transform(lambda x: x["bleu"])
...
>>> metric.compute()
7.4 # instead of default {"bleu": 7.4, "precisions": 0.4, "brevity_penalty": ...}

Is that what you had in mind @lhoestq?

Input format

@sashavor did you see many metrics that violated the proposed format? I think most of them are in that format since it used to be enforced? Then the next thing is to go one level deeper and look at what's in references and predictions in the individual metrics to make them uniform, too?

[sashavor]

The only metrics that violate the proposed format are in the Edge cases in my comment above -- all the other metrics that we currently have are compatible with it.

But yes, the next step would be standardizing the references and predictions format to... a dictionary as well?

[lhoestq]

IIRC we discussed for the postprocessing step that a more flexible alternative is to add a method to the class such that we could do:

metric = load_metric("bleu")
metric.output_transform(lambda x: x["bleu"])
...
metric.compute()
7.4 # instead of default {"bleu": 7.4, "precisions": 0.4, "brevity_penalty": ...}
Is that what you had in mind @lhoestq?

Yup, we can even name the method .postprocess(...) ^^

But yes, the next step would be standardizing the references and predictions format to... a dictionary as well?

We not keep lists of strings or floats directly. Or use dictionaries if we feel like having explicit named fields like texts, tokenized_texts, labels could help - but maybe it would make it less convenient to use

[sashavor]

As per @yjernite 's proposal, maybe it makes sense to have two categories of metrics: the ones with references and the ones without references (like perplexity and stuff like n-gram diversity).
Do you think that makes sense, @lvwerra and @lhoestq ?

[lvwerra]

Would these be the metrics that you call measurement? Could we set references=None in these cases so they don't throw an error if a reference is passed? That way they would be easily compatible with the other metrics like BLEU or ROUGE.

[sashavor]

Sure, that could work! I think @yjernite preferred the term "referenceless metrics" because they actually measure model performance, but if we implement them with references = None by default, that would work!

[lhoestq]

I like referenceless metrics / metrics for unsupervised learning.

In terms of implementation if possible I would remove any mention of "references" to not make things confusing

[sashavor]

So you would have a different input structure for the referenceless metrics?

[lhoestq]

Yes I think it's fine