homework4/mt2022_hw4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.4"
    },
    "colab": {
      "name": "mt2022_hw4.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DcmmZwMfL8Te"
      },
      "source": [
        "# Homework 4"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fiwof0fQL8Tn"
      },
      "source": [
        "### Task 1: Importance of data (2 points)\n",
        "\n",
        "As we have discussed before, language is very diverse. Machine translation models work quite well in limited settings, but a single model is rarely good at dealing with very diverse kinds of texts. Therefore, it is important to keep in mind what data the model is trained on and what data it is applied to. In this task, you will see how a model trained on a particular type of texts behaves when translating texts of different domains.\n",
        "\n",
        "**Subtask 1 (1 point).** You have two ET$\\rightarrow$EN Fairseq models (`/gpfs/space/projects/mt2022/data/model-comparison`): one was trained on a corpus of legislative documents (DGT), and the other on a corpus of movie and TV subtitles (OpenSubtitles). You also have two test sets, one from the DGT corpus and one from OpenSubtitles. Preprocess the sets (the SentencePiece model is also provided), translate each of the test sets with each of the models, postprocess. Measure BLEU score for each of the 4 translations (using sacreBLEU, the same way you did in Homework 3). Report the results, explain what you see.\n",
        "\n",
        "**Hint.** Translation may be faster if you increase batch size, e.g. `--batch-size 32`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zYqw2g7xL8To"
      },
      "source": [
        "**Your answer:** ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZdPDQtdL8To"
      },
      "source": [
        "**Subtask 2 (0.5 points).** You can probably observe that the OpenSubtitles model has a much lower BLEU score on the OpenSubtitles test set than the DGT model does on the DGT test set. The models were trained in exactly the same way: the preprocessing pipeline is the same, the architecture is the same, the number of training sentence pairs is the same, the number of epochs is the same, only the corpora the sentences come from are different. How would you explain this?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VW-lQVVQL8Tp"
      },
      "source": [
        "**Your answer:** ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VBifdHtwhpjQ"
      },
      "source": [
        "**Subtask 3 (0.5 points).** Look at 3-5 examples of sentences from the DGT test set translated with the OpenSubtitles model and vice versa. Compare them with the references. How bad are the translations? Do you see a pattern in what goes wrong?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UGYul2s_nANe"
      },
      "source": [
        "**Your answer:** ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jakzuACdL8Tp"
      },
      "source": [
        "### Task 2: Beam search (2 points)\n",
        "\n",
        "During inference (translation), the model generates a probability distribution over the vocabulary at each step. For each word in the sentence that distribution depends on the previously generated words. To choose the most promising next step, beam search is used.\n",
        "\n",
        "**Subtask 1 (0.5 points).** Explain (in  4-5 sentences) what is beam search and why we need it for text generation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-6en62bwL8Tp"
      },
      "source": [
        "**Your answer:** ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3B6szYh8L8Tq"
      },
      "source": [
        "Beam size is a parameter that does not depend on the model and can be set during translation. Typically, the default beam size is 5. \n",
        "\n",
        "**Subtask 2 (1.5 points).** Experiment with the external test set that you used in Homework 3. Translate the test set with your baseline model from Homework 2 using different beam sizes (for example, 1, 3, 5 and 10). Don't forget to postrocess your translations. Plot BLEU and translation time with different beam sizes (Fairseq reports translation time, which you can divide by the number of sentences in your test set). Describe what you see.\n",
        "\n",
        "If you do not have a well trained baseline model right now (if your model showed BLEU score less than 10 on the test set in Homework 3), use one of the models provided for Task 1.\n",
        "\n",
        "**Hint.** Do all translations from one SLURM script, then all translations will be done on the same node. Use the same batch size for all translations. This will ensure that the only thing that brings in the difference is the beam size."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1NVNpWV-L8Tq"
      },
      "source": [
        "**Your answer:** ..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qpZHYbFfL8Tr"
      },
      "source": [
        "### Task 3: Ideas for improving the baseline (1 point)\n",
        "\n",
        "In Homework 3 you summarized what kinds of mistakes your baseline model tends to make. In this task, you should come up with some ideas how to fix those mistakes.\n",
        "\n",
        "**Note.** If your model showed BLEU less than 10 on the test set from the previous homework, it means that something was wrong with the training. You should find what went wrong, retrain your model properly and redo the analysis. Otherwise it makes little sense for you to do this task. \n",
        "\n",
        "**Subtask 1 (1 point).** Propose some modifications to your model that you think could fix the typical mistakes your baseline model makes. Those modifications can be changes to the architecture, changes to the preprocessing pipeline, or other tricks you can come up with. Explain why you think your proposed modifications could fix the particular mistakes you mentioned."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZnuzhXekL8Tr"
      },
      "source": [
        "**Your answer:** ..."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Task 4: Improving the baseline (5 points)\n",
        "\n",
        "**Subtask 1 (3 points)**. Now, apply one of the proposed changes and retrain your baseline model. If your proposed modification does not cause the vocabulary and the architecture and size of the model to change (for example, if you want to do some data manipulation which does not change preprocessing), continue training your exisiting model instead of starting from scratch. Describe what exactly you changed. Include the training log with your submission. (You can also include scripts or hyperparameter configurations if you wish.)\n",
        "\n",
        "Calculate the new BLEU (and, if you want to, also BERTScore or some other metric) on the held-out and external test sets. How much did it change?\n"
      ],
      "metadata": {
        "id": "6xTPNpS5iSIK"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hvCOp0RlN7ac"
      },
      "source": [
        "**Your answer:** ..."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Subtask 2 (2 points).** Inspect some of the example translations which had the specific problem you were trying to fix with your modifications. How did the translation of those examples change? Did your modifications fix the problem?"
      ],
      "metadata": {
        "id": "Pj1cCuVFlygV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Your answer:** ..."
      ],
      "metadata": {
        "id": "GM6KnzlBm5Za"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iJso_NHyL8Ts"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}