notebooks/nlg_metricverse_demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJpU-5zdsrjQ"
      },
      "source": [
        "# NLG Metricverse Demo\n",
        "\n",
        "This notebook is an introduction to **nlg-metricverse**. It contains simple examples to apply Natural Language Generation (NLG) evaluation metrics, analyze them, compute metric-metric and metric-human correlations.\n",
        "\n",
        "Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.\n",
        "\n",
        "Developed by\n",
        "*   Giacomo Frisoni @ University of Bologna, Italy (giacomo.frisoni[at]unibo.it)\n",
        "*   Andrea Zammarchi @ University of Bologna, Italy (andrea.zammarchi3[at]studio.unibo.it)\n",
        "*   Valentina Pieri @ University of Bologna, Italy (valentina.pieri5[at]studio.unibo.it)\n",
        "*   Marco Avagnano @ University of Bologna, Italy (marco.avagnano[at]studio.unibo.it)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FvN-jbx9VK9M"
      },
      "source": [
        "## Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TIeC1h9JQBsZ"
      },
      "source": [
        "To start off, we have to install the nlg-metricverse package from PyPI or build the library from source. Select one:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9mCe1rc7sGB0"
      },
      "outputs": [],
      "source": [
        "# FROM PYPI\n",
        "!pip install nlg-metricverse --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQes9uk3H7gZ"
      },
      "source": [
        "or"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eQIaRDxNFU0n"
      },
      "outputs": [],
      "source": [
        "# FROM GITHUB SOURCE\n",
        "import os\n",
        "!git clone https://github.com/disi-unibo-nlp/nlg-metricverse.git\n",
        "os.chdir(\"/content/nlg-metricverse/\")\n",
        "!pip install -v . --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_yUSUBVsNpi"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IGel3ny_sZox"
      },
      "source": [
        "We start with required imports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 341
        },
        "id": "xER7hqCusQVA",
        "outputId": "66fadc92-8f4f-44b7-9160-2ac0f6fe86e2"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertModel: ['vocab_layer_norm.bias', 'vocab_transform.weight', 'vocab_transform.bias', 'vocab_projector.bias', 'vocab_layer_norm.weight']\n",
            "- This IS expected if you are initializing DistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing DistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ],
      "source": [
        "import json # Just for pretty printing the output metric dicts\n",
        "from nlgmetricverse import NLGMetricverse, load_metric\n",
        "base_path = \"/content/nlg-metricverse/nlgmetricverse/metrics/\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qfE_REAsg_Z"
      },
      "source": [
        "## Usage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QWaOoBedZSW4"
      },
      "source": [
        "The use of NLG Metricverse follows two simple steps:\n",
        "1. creation and activation a virtual environment of the desired metric;\n",
        "2. the definition of a scorer object (formed by one or more metrics);\n",
        "3. its application to input texts (with a pipeline or parallel execution strategy in case of more metrics).\n",
        "\n",
        "**NOTE:** to be able to use several metrics (e.g., SacreBLEU, BERTScore, COMET, BLEURT), you need to install the related package(s). When you try to use a metric with an implementation based on external packages, NLG Metricverse will throw an exception indicating the installation need for these packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#create virtual environment\n",
        "python -m venv nlgmetricverse\\env\\'name_metric'\n",
        "\n",
        "#activate the virtual environment on Command Prompt\n",
        "nlgmetricverse\\env\\'name_metric'\\Scripts\\activate.bat\n",
        "\n",
        "#or else on powershell\n",
        "nlgmetricverse\\env\\'name_metric'\\Scripts\\activate.ps1\n",
        "\n",
        "#install all the requirements\n",
        "!pip install -v . --quiet\n",
        "!pip install -r requirements-dev.txt\n",
        "#if present, install the specific requirements for the metric\n",
        "!pip install -r nlgmetricverse\\env\\'name_metric'\\requirements.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7bM2mQDsuVHh"
      },
      "outputs": [],
      "source": [
        "# Feel free to play around with the samples below\n",
        "\n",
        "# 1:1\n",
        "predictions_1_1 = [\"Peace in the dormitory, peace in the world.\", \"There is a cat on the mat.\"]\n",
        "references_1_1 = [\"Peace at home, peace in th world.\", \"The cat is playing on the mat.\"]\n",
        "\n",
        "# 1:N\n",
        "predictions_1_n = [\"Evaluating artificial text has never been so simple\", \"the cat is on the mat\"]\n",
        "references_1_n = [\n",
        "    [\"Evaluating artificial text is not difficult\", \"Evaluating artificial text is simple\"],\n",
        "    [\"The cat is playing on the mat.\", \"The cat plays on the mat.\"]\n",
        "]\n",
        "\n",
        "# M:N\n",
        "predictions_m_n = [\n",
        "    [\"Evaluating artificial text has never been so simple\", \"The evaluation of automatically generated text is simple.\"],\n",
        "    [\"the cat is on the mat\", \"the cat likes playing on the mat\"]\n",
        "]\n",
        "references_m_n = [\n",
        "    [\"Evaluating artificial text is not difficult\", \"Evaluating artificial text is simple\"],\n",
        "    [\"The cat is playing on the mat.\", \"The cat plays on the mat.\"]\n",
        "]\n",
        "\n",
        "#@title Select the Hypothesis/Reference setting\n",
        "\n",
        "N_ARITY = \"N:M\" #@param[\"1:1\", \"1:N\", \"N:M\"]\n",
        "\n",
        "REDUCTION_FUNCTION = \"mean\" #@param[\"mean\", \"max\"]\n",
        "\n",
        "predictions, references = None, None\n",
        "if (N_ARITY == \"1:1\"):\n",
        "  predictions, references = predictions_1_1, references_1_1\n",
        "elif (N_ARITY == \"1:N\"):\n",
        "  predictions, references = predictions_1_n, references_1_n\n",
        "else:\n",
        "  predictions, references = predictions_m_n, references_m_n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Ms4kJXG3UTd"
      },
      "source": [
        "### Basic features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mKRAWbpU3dl_"
      },
      "source": [
        "###### Get metrics by applying filters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oGUHUhdh3h2k"
      },
      "outputs": [],
      "source": [
        "from nlgmetricverse import filter_metrics, Categories, ApplTasks, QualityDims\n",
        "\n",
        "print(filter_metrics(category=Categories.Embedding))\n",
        "print(filter_metrics(appl_task=ApplTasks.MachineTranslation))\n",
        "print(filter_metrics(quality_dim=QualityDims.Fluency))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5eArlrLk4pID"
      },
      "source": [
        "##### Load Hypothesis/References from files"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lmHPrfQl4yCh"
      },
      "outputs": [],
      "source": [
        "from nlgmetricverse import DataLoaderStrategies\n",
        "\n",
        "predictions_dir = \"/content/data_loading/predictions\"\n",
        "references_dir = \"/content/data_loading/references\"\n",
        "\n",
        "scorer = NLGMetricverse(metrics=\"meteor\")\n",
        "scores = scorer(predictions=predictions_dir, references=references_dir, \n",
        "                strategy=DataLoaderStrategies.OneRecordPerLine)\n",
        "print(scores)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h-uRvt6VuCrg"
      },
      "source": [
        "### Metrics\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7tVLlZH8I9R0"
      },
      "source": [
        "##### Abstractness"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JNm-S46JI3bi"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"abstractness\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6wrsnL2KLl0"
      },
      "source": [
        "##### Accuracy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U6CmSmg6KR0n"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"accuracy\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0FP_l_1KYjA"
      },
      "source": [
        "##### AUN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1OjuRDrqKaqP"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"aun\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hGyQw2wf0Bkg"
      },
      "source": [
        "##### BARTScore"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v05yHiL80Dms"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"bartscore\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eZqtUbjT63_a"
      },
      "outputs": [],
      "source": [
        "# Author samples\n",
        "# https://github.com/neulab/BARTScore\n",
        "\n",
        "auth_predictions = [\"I'm super happy today.\", \"This is a good idea.\"]\n",
        "auth_references = [\n",
        "  [\"I feel good today.\", \"I feel sad today.\"],\n",
        "  [\"Not bad.\", \"Sounds like a good idea.\"]\n",
        "]\n",
        "scorer = NLGMetricverse(metrics=load_metric(\n",
        "    base_path + \"bartscore\",\n",
        "    compute_kwargs={\"segment_scores\": True}))\n",
        "scores = scorer(predictions=auth_predictions, references=auth_references) # max aggregation (default)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KwpWToi8vw0b"
      },
      "source": [
        "##### BERTScore"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jgBE71DOv0Ow"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"bertscore\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xQpP_mhExiS2"
      },
      "outputs": [],
      "source": [
        "# HF samples\n",
        "# https://github.com/huggingface/datasets/tree/master/metrics/bertscore\n",
        "\n",
        "# Maximal values with the distilbert-base-uncased model:\n",
        "hf_predictions = [\"hello world\", \"general kenobi\"]\n",
        "hf_references = [\"hello world\", \"general kenobi\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(\n",
        "    base_path + \"bertscore\",\n",
        "    compute_kwargs={\"model_type\": \"distilbert-base-uncased\"}))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references)\n",
        "print(json.dumps(scores, indent=4))\n",
        "\n",
        "# Partial match with the bert-base-uncased model and 5 layers:\n",
        "hf_predictions = [\"hello world\", \"general kenobi\"]\n",
        "hf_references = [\"goodnight moon\", \"the sun is shining\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(\n",
        "    base_path + \"bertscore\",\n",
        "    compute_kwargs={\n",
        "        \"model_type\": \"bert-base-uncased\",\n",
        "        \"num_layers\": 5\n",
        "    }))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gj20rxQLfHtz"
      },
      "source": [
        "##### BLEU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gIc06Ub4fNim"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"bleu\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "knjueOucghhy"
      },
      "outputs": [],
      "source": [
        "# HF samples\n",
        "# https://raw.githubusercontent.com/huggingface/datasets/master/metrics/bleu\n",
        "\n",
        "hf_predictions = [\"hello there general kenobi\", \"foo bar foobar\"]\n",
        "hf_references = [\"hello there general kenobi\", \"foo bar foobar\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(base_path + \"bleu\"))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2WNFKYs6Leia"
      },
      "source": [
        "##### BLEURT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0bKpB6kdMVlK"
      },
      "outputs": [],
      "source": [
        "!pip install --upgrade pip  # ensures that pip is current\n",
        "!git clone https://github.com/google-research/bleurt.git\n",
        "!pip install ./bleurt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ynfReXA-mvTT"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"bleurt\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Carburacy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\n",
        "    base_path + \"carburacy\",\n",
        "    compute_kwargs={\"co2_val\": \"1\"}\n",
        "))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGm1BwTXLvG8"
      },
      "source": [
        "##### CER"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JLm354-lTq4K"
      },
      "outputs": [],
      "source": [
        "!pip install jiwer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bdueI9KSLxSU"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"cer\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YiZWCxViTRwA"
      },
      "source": [
        "##### ChrF(++)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Iv-GHUz1MK65"
      },
      "outputs": [],
      "source": [
        "!pip install sacrebleu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aO96p_5PTUtd"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"chrf\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x9iCWMmRTrq-"
      },
      "outputs": [],
      "source": [
        "# HF samples (do not match)\n",
        "# https://github.com/huggingface/datasets/blob/master/metrics/chrf/README.md\n",
        "\n",
        "hf_predictions = [\"The relationship between cats and dogs is not exactly friendly.\", \"a good bookshop is just a genteel black hole that knows how to read.\"]\n",
        "hf_references = [\"The relationship between dogs and cats is not exactly friendly.\", \"A good bookshop is just a genteel Black Hole that knows how to read.\"]\n",
        "\n",
        "hf_predictions = [\"The relationship between cats and dogs is not exactly friendly.\", \"a good bookshop is just a genteel black hole that knows how to read.\"]\n",
        "hf_references = [[\"The relationship between dogs and cats is not exactly friendly.\", ], [\"A good bookshop is just a genteel Black Hole that knows how to read.\"]]\n",
        "\n",
        "# A simple example of calculating chrF\n",
        "scorer = NLGMetricverse(metrics=load_metric(base_path + \"chrf\"))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references)\n",
        "print(json.dumps(scores, indent=4))\n",
        "\n",
        "# The same example, but with the argument word_order=2, to calculate chrF++ instead of chrF\n",
        "scorer = NLGMetricverse(metrics=load_metric(\n",
        "    base_path + \"chrf\",\n",
        "    compute_kwargs={\"word_order\": 2}))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references)\n",
        "print(json.dumps(scores, indent=4))\n",
        "\n",
        "# The same chrF++ example as above, but with lowercase=True to normalize all case\n",
        "scorer = NLGMetricverse(metrics=load_metric(\n",
        "    base_path + \"chrf\",\n",
        "    compute_kwargs={\"word_order\": 2, \"lowercase\": True}))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7IQhmQATJ6M"
      },
      "source": [
        "##### Cider"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9CEBGGj8TPoF"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"cider\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvlapmQPT4_8"
      },
      "source": [
        "##### Coleman-Liau"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jBFLwh_QT7-r"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"coleman_liau\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dM7vryN2maxF"
      },
      "source": [
        "##### COMET"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hYrY6NICmwwQ"
      },
      "outputs": [],
      "source": [
        "!pip install unbabel-comet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RGCnvENZmcOo"
      },
      "outputs": [],
      "source": [
        "# HF samples\n",
        "# Note: COMET is MT-specific and also requires source sentences\n",
        "\n",
        "# Full match\n",
        "# hf_sources = [\"Dem Feuer konnte Einhalt geboten werden\", \"Schulen und Kindergärten wurden eröffnet.\"]\n",
        "# hf_predictions = [\"They were able to control the fire.\", \"Schools and kindergartens opened\"]\n",
        "# hf_references = [\"They were able to control the fire.\", \"Schools and kindergartens opened\"]\n",
        "\n",
        "# Partial match\n",
        "hf_sources = [\"Dem Feuer konnte Einhalt geboten werden\", \"Schulen und Kindergärten wurden eröffnet.\"]\n",
        "hf_predictions = [\"The fire could be stopped\", \"Schools and kindergartens were open\"]\n",
        "hf_references = [\"They were able to control the fire\", \"Schools and kindergartens opened\"]\n",
        "\n",
        "# No match\n",
        "# hf_sources = [\"Dem Feuer konnte Einhalt geboten werden\", \"Schulen und Kindergärten wurden eröffnet.\"]\n",
        "# hf_predictions = [\"The girl went for a walk\", \"The boy was sleeping\"]\n",
        "# hf_references = [\"They were able to control the fire\", \"Schools and kindergartens opened\"]\n",
        "\n",
        "scorer = NLGMetricverse(metrics=load_metric(\n",
        "    \"comet\",\n",
        "    config_name=\"wmt21-cometinho-da\", # smaller model than wmt20-comet-da (default)\n",
        "    compute_kwargs={\"gpus\": 0, \"num_workers\": 0, \"progress_bar\": True, \"batch_size\": 2}))\n",
        "scores = scorer(sources=hf_sources, predictions=hf_predictions, references=hf_references)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Compression"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"compression\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Coverage"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"coverage\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Density"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"density\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5QtY-DNvUk8i"
      },
      "source": [
        "##### F1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFoy8LN9Umbw"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"f1\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kyA2BWUdeSy6"
      },
      "source": [
        "##### Flesch-Kincaid"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OfoXgkBkeSy7"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"flesch_kincaid\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JZjp-OLceSy7"
      },
      "source": [
        "##### Gunning-Fog"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZBCvU3w8eSy7"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"gunning_fog\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CAQLyfPCeSy8"
      },
      "source": [
        "##### Mauve"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C2BtHHNHeSy8"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"mauve\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wXKCKmg9LwdZ"
      },
      "source": [
        "##### METEOR"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "puNS5uaTLx_M"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"meteor\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "03COchrePxDh"
      },
      "outputs": [],
      "source": [
        "# Popular tutorial sample\n",
        "tut_predictions = [\"the cat sat on the mat\"]\n",
        "tut_references = [\"on the mat sat the cat\"]\n",
        "\n",
        "#P = 1, R = 1, F_mean = 1.0000\n",
        "#p = 0.5*(6/6)^3 = 0.5000\n",
        "#M = 1.0000*(1-0.5000) = 0.5000\n",
        "scorer = NLGMetricverse(metrics=load_metric(base_path + \"meteor\"))\n",
        "scores = scorer(predictions=tut_predictions, references=tut_references)\n",
        "print(json.dumps(scores, indent=4))\n",
        "\n",
        "# HF sample\n",
        "hf_predictions = [\"It is a guide to action which ensures that the military always obeys the commands of the party\"]\n",
        "hf_references = [\"It is a guide to action that ensures that the military will forever heed Party commands\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(base_path + \"meteor\"))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Lo6D7KaZerH"
      },
      "source": [
        "##### MoverScore"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g0bC8j5B7lUi"
      },
      "outputs": [],
      "source": [
        "!pip install git+https://github.com/AIPHES/emnlp19-moverscore.git\n",
        "!pip install transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mdqRx40NZsbo"
      },
      "outputs": [],
      "source": [
        "from moverscore_v2 import get_idf_dict, word_mover_score \n",
        "from collections import defaultdict\n",
        "\n",
        "refs = [\"Peace at home, peace in th world.\", \"The cat is playing on the mat.\"]\n",
        "sys = [\"Peace in the dormitory, peace in the world.\", \"There is a cat on the mat.\"]\n",
        "\n",
        "idf_dict_hyp = get_idf_dict(sys) # idf_dict_hyp = defaultdict(lambda: 1.)\n",
        "idf_dict_ref = get_idf_dict(refs) # idf_dict_ref = defaultdict(lambda: 1.)\n",
        "\n",
        "scores = word_mover_score(refs, sys, idf_dict_ref, idf_dict_hyp, \\\n",
        "                          stop_words=[], n_gram=1, remove_subwords=True)\n",
        "print(scores)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "m0uVigyeZhK_"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"moverscore\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### NID"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"nid\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jvns5hOTRSbo"
      },
      "source": [
        "##### NIST"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7djJzIdDQuta"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"nist\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TOWeaBLmeSy_"
      },
      "source": [
        "##### Nubia"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "waBgpzQqeSy_"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"nubia\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eaYJC9S4eSy_"
      },
      "source": [
        "##### Perplexity"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VeqfhRN3eSzA"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"perplexity\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dxQn7oFceSzA"
      },
      "source": [
        "##### Prism"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w8qBEGdAFZFx"
      },
      "outputs": [],
      "source": [
        "!pip install sentencepiece\n",
        "!pip install fairseq"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rv-vtbqLeSzA"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"prism\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8H-fRle8eSzB"
      },
      "source": [
        "##### Repetitiveness"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YErhg-oMeSzB"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"repetitiveness\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### Recall"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"recall\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_zr9obGNd9JT"
      },
      "source": [
        "##### Rouge"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7r-Bi9yyd50J"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"rouge\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DeP5-wvrfJm8"
      },
      "outputs": [],
      "source": [
        "# TO ROUGE OR NOT TO ROUGE samples\n",
        "# https://towardsdatascience.com/to-rouge-or-not-to-rouge-6a5f3552ea45\n",
        "\n",
        "# 1)\n",
        "hf_predictions = [\"The quick brown fox jumped over the lazy dog.\"]\n",
        "hf_references = [\"The fox jumped over the dog.\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(\"rouge\"))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references,\n",
        "                rouge_types=[\"rougeL\"],\n",
        "                use_aggregator=False, use_stemmer=False,\n",
        "                metric_to_select=\"fmeasure\")\n",
        "print(json.dumps(scores, indent=4))\n",
        "\n",
        "# 2)\n",
        "hf_predictions = [\"The quick brown fox jumped over the lazy dog.\"]\n",
        "hf_references = [\"The quick brown dog jumped over the lazy fox.\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(\"rouge\"))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references,\n",
        "                rouge_types=[\"rougeL\"],\n",
        "                use_aggregator=False, use_stemmer=False,\n",
        "                metric_to_select=\"fmeasure\")\n",
        "print(json.dumps(scores, indent=4))\n",
        "\n",
        "# 3)\n",
        "hf_predictions = [\"The quick brown fox jumped over the lazy dog.\"]\n",
        "hf_references = [\"The fast wood-coloured fox hopped over the lethargic dog.\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(\"rouge\"))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references,\n",
        "                rouge_types=[\"rougeL\"],\n",
        "                use_aggregator=False, use_stemmer=False,\n",
        "                metric_to_select=\"fmeasure\")\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KKnvqhK5C6CS"
      },
      "outputs": [],
      "source": [
        "hf_predictions = [\"The quick brown fox jumped over the lazy dog.\"]\n",
        "hf_references = [\"The quick brown dog jumped over the lazy fox.\"]\n",
        "scorer = NLGMetricverse(metrics=load_metric(\"rouge\", compute_kwargs={\"rouge_types\": [\"rougeL\"]}))\n",
        "scores = scorer(predictions=hf_predictions, references=hf_references,\n",
        "                use_aggregator=False, use_stemmer=False,\n",
        "                metric_to_select=\"fmeasure\")\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lopsa1S7eSzC"
      },
      "source": [
        "##### SacreBLEU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DiNpDGUdeSzC"
      },
      "outputs": [],
      "source": [
        "!pip install sacrebleu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pj3BeunKeSzC"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"prism\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nvOrVFaveSzD"
      },
      "source": [
        "##### TER"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MZ0ppci-eSzD"
      },
      "outputs": [],
      "source": [
        "!pip install jiwer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KRQoi_mteSzD"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"ter\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##### UNR"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"unr\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P6P3S-DAeSzD"
      },
      "source": [
        "##### WER"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fZAmMNwEeSzE"
      },
      "outputs": [],
      "source": [
        "!pip install jiwer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E1eyXbyseSzE"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"wer\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EiXzzroteSzE"
      },
      "source": [
        "##### WMD"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LMmqUpureSzE"
      },
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"wmd\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Missing Metric"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here is an example of what happens when a metric is not available in NLGmetricverse. In this case we are trying to load the metric glue, that is not present on NLGmetricverse, when runned the following code it will appear a warning message that explains that the metric is not available and the metric will be loaded through the HuggingFace/evaluate.\n",
        "\n",
        "**NOTE:** at the moment, this function is not available due to a network problem to HuggingFace/evaluate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scorer = NLGMetricverse(metrics=load_metric(\"glue\"))\n",
        "scores = scorer(predictions=predictions, references=references, reduce_fn=REDUCTION_FUNCTION)\n",
        "print(json.dumps(scores, indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YEwPBxquuHcg"
      },
      "source": [
        "### Meta-Eval"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NNNRD7_KeSzF"
      },
      "source": [
        "##### Metric-Human Correlation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7EVanZvTfEkp"
      },
      "outputs": [],
      "source": [
        "!pip install jiwer\n",
        "!pip install sacrebleu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JfDM0y14eSzF"
      },
      "outputs": [],
      "source": [
        "from matplotlib import pyplot as plt\n",
        "from nlgmetricverse import metric_human_correlation\n",
        "\n",
        "metrics = [\"meteor\", \"ter\", \"wer\"]\n",
        "predictions_large = [\n",
        "    [\"Evaluating artificial text has never been so simple\", \"The evaluation of automatically generated text is simple.\", \"Evaluating artificial text is really easy.\"],\n",
        "    [\"the cat is on the mat\", \"the cat likes playing on the mat\", \"the cat is laying on the mat\"],\n",
        "    [\"The weather outside is cold\", \"It's freezing today\", \"Look! It's not warm at all today\"]\n",
        "]\n",
        "references_large = [\n",
        "    [\"Evaluating artificial text is not difficult\", \"Evaluating artificial text is simple\", \"The evaluation of artificial text is easy\"],\n",
        "    [\"The cat is playing on the mat.\", \"The cat plays on the mat.\", \"Look! The cat plays on the mat\"],\n",
        "    [\"Outside is cold today\", \"It's freezing today outside\", \"The temperature is low ouside\"]\n",
        "]\n",
        "\n",
        "metric_human_correlation(predictions=predictions_large, references=references_large, metrics=metrics, human_scores=[0.5, 0.6, 0.7])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ga4trgMFeSzF"
      },
      "source": [
        "##### Metric-Metric Correlation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZDpbkG7Expje"
      },
      "outputs": [],
      "source": [
        "!pip install jiwer\n",
        "!pip install sacrebleu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LI4wttmheSzF"
      },
      "outputs": [],
      "source": [
        "from matplotlib import pyplot as plt\n",
        "from nlgmetricverse import metrics_correlation\n",
        "\n",
        "metrics = [\"meteor\", \"ter\", \"wer\"]\n",
        "predictions_large = [\n",
        "    [\"Evaluating artificial text has never been so simple\", \"The evaluation of automatically generated text is simple.\", \"Evaluating artificial text is really easy.\"],\n",
        "    [\"the cat is on the mat\", \"the cat likes playing on the mat\", \"the cat is laying on the mat\"],\n",
        "    [\"The weather outside is cold\", \"It's freezing today\", \"Look! It's not warm at all today\"]\n",
        "]\n",
        "references_large = [\n",
        "    [\"Evaluating artificial text is not difficult\", \"Evaluating artificial text is simple\", \"The evaluation of artificial text is easy\"],\n",
        "    [\"The cat is playing on the mat.\", \"The cat plays on the mat.\", \"Look! The cat plays on the mat\"],\n",
        "    [\"Outside is cold today\", \"It's freezing today outside\", \"The temperature is low ouside\"]\n",
        "]\n",
        "\n",
        "metrics_correlation(predictions=predictions_large, references=references_large, metrics=metrics)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kWF6rMNqeSzG"
      },
      "source": [
        "##### Performance comparison"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lUjSbKHmeSzG"
      },
      "outputs": [],
      "source": [
        "from matplotlib import pyplot as plt\n",
        "from nlgmetricverse import times_correlation\n",
        "\n",
        "metrics = [\"meteor\", \"ter\", \"wer\"]\n",
        "times_correlation(predictions=predictions, references=references, metrics=metrics)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "talqNnKXIwcJ"
      },
      "source": [
        "### Visualization\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6TE2Vlq1zNSk"
      },
      "source": [
        "##### N-Gram distance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tP0oRu9wzWDh"
      },
      "outputs": [],
      "source": [
        "from nlgmetricverse import n_gram_distance_visualization\n",
        "\n",
        "n_gram_distance_visualization(predictions[0][0], references[0][0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pBxIqk-Q1jF7"
      },
      "source": [
        "##### Similarity Word Matching"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lAFPEWIj1nPq"
      },
      "outputs": [],
      "source": [
        "from nlgmetricverse import similarity_word_matching\n",
        "\n",
        "similarity_word_matching(predictions[1][0], references[1][0], lang=\"en\")"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "7_yUSUBVsNpi",
        "mKRAWbpU3dl_",
        "YEwPBxquuHcg",
        "NNNRD7_KeSzF",
        "Ga4trgMFeSzF",
        "kWF6rMNqeSzG",
        "talqNnKXIwcJ"
      ],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.9.13 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "vscode": {
      "interpreter": {
        "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}