Add InfoLM

[stancld]

🚀 Feature

Add InfoLM

Sources:

• Paper: InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation ((accepted at AAAI '22))
• Repo

Motivation

The recent NLG metrics are more often based on BERT (or related) embeddings. As such, I believe, we should also start adding such metrics into TorchMetrics with an extra dependency on transformers if a user wants to use any of these metrics. The InfoLM metric is from a family of untrained metrics (i.e. the model is not fine-tuned on any specific task) so it should be easier for us to begin with it. (Any opinion on this? @Borda :] )

Abstract:

Assessing the quality of natural language generation systems through human annotation is very expensive. Additionally, human annotation campaigns are time-consuming and include non-reusable human labour. In practice, researchers rely on automatic metrics as a proxy of quality. In the last decade, many string-based metrics (e.g., BLEU) have been introduced. However, such metrics usually rely on exact matches and thus, do not robustly handle synonyms. In this paper, we introduce InfoLM a family of untrained metrics that can be viewed as a string-based metric that addresses the aforementioned flaws thanks to a pre-trained masked language model. This family of metrics also makes use of information measures allowing the adaptation of InfoLM to various evaluation criteria. Using direct assessment, we demonstrate that InfoLM achieves statistically significant improvement and over 10 points of correlation gains in many configurations on both summarization and data2text generation.

[Borda]

@stancld ho is it doing, seems that this is the key for two additional metrics where we have volunteers to help... 🐰

[stancld]

@Borda Sorry, was a bit busy this week, but it's already wip. I'm gonna try to finish it this weekend so that the other two PRs can be opened.