homeworks/hw6/Homework6.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g3yhk6NlB5P4",
        "colab_type": "text"
      },
      "source": [
        "# Homework 6. Sequence Tagging with BERT\n",
        "\n",
        "Welcome to Homework 6! \n",
        "\n",
        "The homework contains several tasks. You can find the amount of points that you get for the correct solution in the task header. Maximum amount of points for each homework is _six_.\n",
        "\n",
        "The **grading** for each task is the following:\n",
        "- correct answer - **full points**\n",
        "- insufficient solution or solution resulting in the incorrect output - **half points**\n",
        "- no answer or completely wrong solution - **no points**\n",
        "\n",
        "Even if you don't know how to solve the task, we encourage you to write down your thoughts and progress and try to address the issues that stop you from completing the task.\n",
        "\n",
        "When working on the written tasks, try to make your answers short and accurate. Most of the times, it is possible to answer the question in 1-3 sentences.\n",
        "\n",
        "When writing code, make it readable. Choose appropriate names for your variables (`a = 'cat'` - not good, `word = 'cat'` - good). Avoid constructing lines of code longer than 100 characters (79 characters is ideal). If needed, provide the commentaries for your code, however, a good code should be easily readable without them :)\n",
        "\n",
        "Finally, all your answers should be written only by yourself. If you copy them from other sources it will be considered as an academic fraud. You can discuss the tasks with your classmates but each solution must be individual.\n",
        "\n",
        "<font color='red'>**Important!:**</font> **before sending your solution, do the `Kernel -> Restart & Run All` to ensure that all your code works.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ohBaJxUz4-8z",
        "colab_type": "text"
      },
      "source": [
        "## Task 1. Prepare the environment and download the data (1 point)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sTzC1mgRmWdK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "from torch.utils.data.sampler import SubsetRandomSampler\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "import numpy as np\n",
        "\n",
        "from pathlib import Path\n",
        "import time\n",
        "import random\n",
        "from collections import Counter\n",
        "\n",
        "from typing import List, Dict\n",
        "\n",
        "# Check if we are running on a CPU or GPU\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HWmhlyh7DJK0",
        "colab_type": "text"
      },
      "source": [
        "We need to install the transformers library first, if we use Google Colab.\n",
        "\n",
        "Another useful library to make the data loading cleaner and easier is [conllu](https://github.com/EmilStenstrom/conllu/). In particular, we are going to use the `parse()` function from this package. You can consult the documentation if you want to know more about this package."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DoyOsBDdmpHl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install transformers\n",
        "!pip install conllu"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GNiuqmnMDTG3",
        "colab_type": "text"
      },
      "source": [
        "In this Homework, we are going to use `AutoModelForTokenClassification`, `AutoConfig` and `AutoTokenizer` to treat our data. Please, read more about them on the official documentation page: https://huggingface.co/transformers/model_doc/auto.html.\n",
        "\n",
        "In short, these classes will create a specific model class for us. We just need to specify the name for the model. For example, if we call `AutoTokenizer.from_pretrained('bert-base-multilingual-cased')` it is going to create a `BertTokenizer` for us. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pI2RRh0anNai",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from transformers import AutoModelForTokenClassification, AdamW, AutoConfig\n",
        "from transformers import AutoTokenizer, PreTrainedTokenizer\n",
        "from transformers import get_linear_schedule_with_warmup\n",
        "\n",
        "from conllu import parse"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7_o-eY4DyEd",
        "colab_type": "text"
      },
      "source": [
        "As in the previous Homework, we are going to use [Universal Dependencies](https://universaldependencies.org/) data. It has labelled corpora for morphological tagging and syntax parsing for over than 70 languages. You need to choose your language from the official UD page, choose the treebank that you like and follow the GitHub link to it. Then, from GitHub, copy the link from the green \"Clone or download\" button and replace it in the cell below. \n",
        "\n",
        "Also, replace the name of your treebank in the `!mv` command.\n",
        "\n",
        "For example, if I choose the EDT treebank for Estonian from [here](https://universaldependencies.org/#estonian-treebanks), the GitHub link is going to be `https://github.com/UniversalDependencies/UD_Estonian-EDT.git` and the name of the treebank is `UD_Estonian-EDT`, which is the name of the repository.\n",
        "\n",
        "Replace the `...` in `!git clone` with the GitHub link to the repository that you've chosen.\n",
        "\n",
        "Replace the `...` in `!mv` with the name of the treebank that you've chosen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ox9JncIamvf2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone ...\n",
        "!mkdir data/\n",
        "!mv .../ data/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OYEcwVlHD2Jv",
        "colab_type": "text"
      },
      "source": [
        "Here, you will need to choose the model to suit your data. The list of all available models can be found here: https://huggingface.co/transformers/pretrained_models.html\n",
        "\n",
        "The common advice is, that if you can use a language-specific model, this is the best way to go. Otherwise, check if a `multilingual` model has your language (https://github.com/google-research/bert/blob/master/multilingual.md) and go with it. Another important thing is that your model must support token classification. You can see the list of available models here: https://huggingface.co/transformers/model_doc/auto.html#automodelfortokenclassification\n",
        "\n",
        "In this Homework, you are free to choose any suitable model. To do that, paste the appropriate name into the `MODEL_NAME` variable.\n",
        "\n",
        "Finally, don't forget to put the name of your UD treebank into the `DATA_PATH` variable.\n",
        "\n",
        "__What model did you choose? Briefly explain your choice.__\n",
        "\n",
        "<font color='red'>Your answer here</font>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IXQiZKhtns-p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "PAD = '[PAD]'\n",
        "PAD_ID = 0\n",
        "UNK = '[UNK]'\n",
        "UNK_ID = 1\n",
        "CLS = '[CLS]'\n",
        "CLS_ID = 2\n",
        "SEP = '[SEP]'\n",
        "SEP_ID = 3\n",
        "VOCAB_PREFIX = [PAD, UNK, CLS, SEP]\n",
        "# Setting label padding to -100 since this is a default value for ignore_index\n",
        "# in Pytorch CrossEntropyLoss:\n",
        "# (https://pytorch.org/docs/stable/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss)\n",
        "LABEL_PAD_ID = -100\n",
        "DATA_PATH = Path('data') / '...'\n",
        "\n",
        "MODEL_NAME = '...'\n",
        "\n",
        "batch_size = 16\n",
        "random_seed = 42"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5-YzLvr4o2SO",
        "colab_type": "text"
      },
      "source": [
        "Initialize your tokenizer with the `AutoTokenizer.from_pretrained()`. Pay attention to the `do_lower_case` parameter. It should be set in appropriately depending on if your model is `cased` or `uncased`.\n",
        "\n",
        "You can read about other available parameters here: https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4EfPzA8Hm-0Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tokenizer = AutoTokenizer.from_pretrained(...)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u_ByAQXDplgm",
        "colab_type": "text"
      },
      "source": [
        "You can test if the tokenizer is working here"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QIK-URPGncKv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(tokenizer.tokenize(\"This is the BERT tokenizer that we're going to use today.\"))\n",
        "print(tokenizer.convert_tokens_to_ids(tokenizer.tokenize(\"This is the BERT tokenizer that we're going to use today.\")))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJiv2OgvpqJY",
        "colab_type": "text"
      },
      "source": [
        "Another usefull thing is to know how to access the ids of special tokens (`[CLS]`, `[SEP]`)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iq0XN6AWFlv1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(f'Token: {tokenizer.cls_token}\\tID: {tokenizer.cls_token_id}')\n",
        "print(f'Token: {tokenizer.sep_token}\\tID: {tokenizer.sep_token_id}')\n",
        "print(f'Token: {tokenizer.unk_token}\\tID: {tokenizer.unk_token_id}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "45zCLDbSEBh_",
        "colab_type": "text"
      },
      "source": [
        "## Task 2. Load the data (4 points)\n",
        "\n",
        "For loading the data, we can benifit from the `WordVocab` from Homework 4 to create the vocab for the labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tSGs_2F46n_b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class BaseVocab:\n",
        "    def __init__(self, data, idx=0, lower=False):\n",
        "        self.data = data\n",
        "        self.lower = lower\n",
        "        self.idx = idx\n",
        "        self.build_vocab()\n",
        "        \n",
        "    def normalize_unit(self, unit):\n",
        "        if self.lower:\n",
        "            return unit.lower()\n",
        "        else:\n",
        "            return unit\n",
        "        \n",
        "    def unit2id(self, unit):\n",
        "        unit = self.normalize_unit(unit)\n",
        "        if unit in self._unit2id:\n",
        "            return self._unit2id[unit]\n",
        "        else:\n",
        "            return self._unit2id[UNK]\n",
        "    \n",
        "    def id2unit(self, id):\n",
        "        return self._id2unit[id]\n",
        "    \n",
        "    def map(self, units):\n",
        "        return [self.unit2id(unit) for unit in units]\n",
        "\n",
        "    def unmap(self, ids):\n",
        "        return [self.id2unit(idx) for idx in ids]\n",
        "        \n",
        "    def build_vocab(self):\n",
        "        NotImplementedError()\n",
        "        \n",
        "    def __len__(self):\n",
        "        return len(self._unit2id)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kgga4EWs6pNV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class WordVocab(BaseVocab):\n",
        "    def build_vocab(self):\n",
        "        if self.lower:\n",
        "            counter = Counter([w[self.idx].lower() for sent in self.data for w in sent])\n",
        "        else:\n",
        "            counter = Counter([w[self.idx] for sent in self.data for w in sent])\n",
        "\n",
        "        self._id2unit = VOCAB_PREFIX + list(sorted(list(counter.keys()), key=lambda k: counter[k], reverse=True))\n",
        "        self._unit2id = {w:i for i, w in enumerate(self._id2unit)}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m8jBdJiNqvT-",
        "colab_type": "text"
      },
      "source": [
        "You might have seen in the Lab 6 that the data for the BERT transformers must follow a set convention. In particular, each sequence must start with the `[CLS]` token and end with the `[SEP]` token. Even though we are not going to use the `[CLS]` token in this task, it is still required to be in your data.\n",
        "\n",
        "Another important thing that we need to take into account is the word tokenization. In the UD data, the sentences are already pre-tokenized. We could used the existing tokenization but then we would have a lot of `[UNK]` tokens, especially for a language other than English. BERT tokenizer tends to split unknown tokens into sub-words to get useful information from them. \n",
        "\n",
        "Let's see the following example:\n",
        "\n",
        "If we tokenize the sentence \"This is the BERT tokenizer that we're going to use today.\" with the tokenizer from `bert-base-multilingual-cased` model, we get these tokens: `['This', 'is', 'the', 'BE', '##RT', 'tok', '##eni', '##zer', 'that', 'we', \"'\", 're', 'going', 'to', 'use', 'today', '.']`. The problem is that we need to somehow allign the labels since now since, for example, the word `tokenizer` is split into three sub-words. \n",
        "\n",
        "One way to overcome this is not to use the BERT tokenizer and just convert already pre-tokenized words to ids. However, then we get the following mapping:\n",
        "\n",
        "Tokens: `['[CLS]', 'This', 'is', 'the', 'BERT', 'tokenizer', 'that', 'we', \"'\", 're', 'going', 'to', 'use', 'today', '.', '[SEP]']`\n",
        "\n",
        "IDs: `[[CLS], 10747, 10124, 10105, [UNK], [UNK], 10189, 11951, 112, 11639, 19090, 10114, 11760, 18745, 119, [SEP]]`\n",
        "\n",
        "You can see that we now have two `[UNK]` tokens since `BERT` and `tokenizer` were not in the vocabulary. We also lost five sub-words that could give the model useful features from the text. What is more, for languages other than English, the number of `[UNK]` is going to be even higher.\n",
        "\n",
        "One way to overcome this is to use the real label for the first sub-word and the padding labels for the following sub-words. For example, for the same sentence:\n",
        "\n",
        "`['[CLS]', 'This', 'is', 'the', 'BE', '##RT', 'tok', '##eni', '##zer', 'that', 'we', \"'\", 're', 'going', 'to', 'use', 'today', '.', '[SEP]']`\n",
        "\n",
        "We will have the following labels:\n",
        "\n",
        "`['[CLS]', 'DET', 'VERB', 'DET', 'NOUN', '[PAD]', 'NOUN', '[PAD]', '[PAD], 'DET', 'PRON', 'PUNCT', 'AUX', 'VERB', 'PART', 'VERB', 'NOUN', 'PUNCT', '[SEP]']`\n",
        "\n",
        "The next step is to pad all the sequences to the max length. For the input IDs, we are going to pad them with `0` and for the labels with `-100`. We use `-100` for the labels since this is the default value for the `ignore_index` in `CrossEntropyLoss`. This means that the labels with the ID `-100` are not going to contribute to the loss.\n",
        "\n",
        "Finally, we need to create an attention mask which holds `1` in the place of meaningful tokens and `0` in the place of paddings. You can read more about this here: https://huggingface.co/transformers/glossary.html#attention-mask.\n",
        "\n",
        "Now you should be ready to create the dataset by completing the `preprocess` method below. Read the data word by word, see if a word from the data is split into sub-words, in this case add the correct amount of paddings to the labels. The final output should look similar to this:\n",
        "\n",
        "```\n",
        "Tokend ids : tensor([  101,   146, 10483, 64254, 42430, 10107, 50302, 15938,    \n",
        "                     17802, 20509, 10410, 60400, 26419, 10123, 10124, 25151, \n",
        "                     15636, 10123,   119,   102,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0,\n",
        "                         0,     0,     0,     0,     0,     0,     0,     0])\n",
        "Attention mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,   \n",
        "                        1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  \n",
        "                        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \n",
        "                        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
        "                        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \n",
        "                        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \n",
        "                        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,   \n",
        "                        0, 0])\n",
        "Label ids: tensor([   2,   16,    0,    8,    0,    0,    4,    0,    6,   \n",
        "                      8,    0,    0,    0,    0,    4,    0,    0,    0,    \n",
        "                      5,    3, -100, -100, -100, -100, -100, -100, -100, -100, \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,  \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, \n",
        "                   -100, -100, -100, -100, -100, -100, -100, -100, -100, -100])\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "_Some tips:_\n",
        "\n",
        "- To get the ID of the tag, use `vocab['upos'].unit2id(upostag)`.\n",
        "- You can extend lists with `+=`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3nGj7Z0g61iV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TaggerDataset(Dataset):\n",
        "    def __init__(self, data_path: str, tokenizer: PreTrainedTokenizer,\n",
        "                 vocab: BaseVocab = None, max_length: int = 128,\n",
        "                 label_pad_id: int = -100):\n",
        "        self.pretrained_tokenizer = tokenizer\n",
        "        self.max_length = max_length\n",
        "        self.label_pad_id = label_pad_id\n",
        "        data = self.load_doc(data_path)\n",
        "\n",
        "        if vocab is None:\n",
        "            self.vocab = self.init_vocab(data)\n",
        "        else:\n",
        "            self.vocab = vocab\n",
        "\n",
        "        self.data = self.preprocess(data, self.vocab)\n",
        "\n",
        "    def init_vocab(self, data: List) -> Dict[str, BaseVocab]:\n",
        "        uposvocab = WordVocab(data, idx=1)\n",
        "        vocab = {'upos': uposvocab}\n",
        "        return vocab\n",
        "\n",
        "    def preprocess(self, data: List, vocab: Dict[str, BaseVocab]) -> List[List[int]]:\n",
        "        processed = []\n",
        "        for sent in data:\n",
        "            # Put the [CLS] token in the beginning of each sentence\n",
        "            input_ids = [self.pretrained_tokenizer.convert_tokens_to_ids(CLS)]\n",
        "            attention_mask = [1]\n",
        "            # Put the [CLS] label to match the sequence\n",
        "            tokenized_labels = [CLS_ID]\n",
        "            for word in sent:\n",
        "                form = word[0]\n",
        "                upostag = word[1]\n",
        "                token_ids = ...\n",
        "                if len(token_ids) > 1:\n",
        "                    ...\n",
        "                else:\n",
        "                    ...\n",
        "            \n",
        "            # Trim the sequences to max_length - 1 since we will add [SEP] token\n",
        "            # in the end later\n",
        "            if len(input_ids) + 1 > self.max_length:\n",
        "                input_ids = ...\n",
        "                attention_mask = ...\n",
        "                tokenized_labels = ...\n",
        "\n",
        "            # Adding [SEP] token to mark the end of sequence\n",
        "            input_ids += [self.pretrained_tokenizer.convert_tokens_to_ids(SEP)]\n",
        "            attention_mask += [1]\n",
        "            tokenized_labels += [SEP_ID]\n",
        "\n",
        "            # Padding the rest to the max_length\n",
        "            input_ids += ...\n",
        "            attention_mask += ...\n",
        "            tokenized_labels += ...\n",
        "\n",
        "            # Check that all the inputs have equal lengths\n",
        "            assert len(input_ids) == self.max_length, f\"Input length is {len(input_ids)} while max length is {self.max_length}\"\n",
        "            assert len(attention_mask) == self.max_length, f\"Attention mask length is {len(attention_mask)} while max length is {self.max_length}\"\n",
        "            assert len(tokenized_labels) == self.max_length, f\"Labels length is {len(tokenized_labels)} while max length is {self.max_length}\"\n",
        "\n",
        "            # Converting python lists to pytorch long tensors\n",
        "            input_ids = torch.tensor(input_ids, dtype=torch.long)\n",
        "            attention_mask = torch.tensor(attention_mask, dtype=torch.long)\n",
        "            tokenized_labels = torch.tensor(tokenized_labels, dtype=torch.long)\n",
        "\n",
        "            processed.append([input_ids, attention_mask, tokenized_labels])\n",
        "        return processed\n",
        "        \n",
        "    def load_doc(self, data_path: str) -> List:\n",
        "        doc_text = open(data_path, encoding='utf-8').read()\n",
        "        data = [[[token['form'], token['upostag']] for token in sent] for sent in parse(doc_text)]\n",
        "        return data\n",
        "            \n",
        "    def __len__(self):\n",
        "        return len(self.data)\n",
        "    \n",
        "    def __getitem__(self, idx):\n",
        "        return self.data[idx][0], self.data[idx][1], self.data[idx][2]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qH8QLb3d2iro",
        "colab_type": "text"
      },
      "source": [
        "Don't forget to put the correct names for the train, dev, and test files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H-QurcZpHBR1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_path = DATA_PATH / '...'\n",
        "dev_path = DATA_PATH / '...'\n",
        "test_path = DATA_PATH / '...'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TFM42wwFpCVx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_data = TaggerDataset(train_path, tokenizer)\n",
        "vocab = train_data.vocab\n",
        "\n",
        "dev_data = TaggerDataset(dev_path, tokenizer, vocab)\n",
        "test_data = TaggerDataset(test_path, tokenizer, vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y9hSPwf4du41",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(f\"Tokend ids (size = {train_data[0][0].size()}): {train_data[0][0]}\")\n",
        "print(f\"Attention mask (size = {train_data[0][1].size()}): {train_data[0][1]}\")\n",
        "print(f\"Label ids (size = {train_data[0][2].size()}): {train_data[0][2]}\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fl11K6H4FmVT",
        "colab_type": "text"
      },
      "source": [
        "Last modification that we make is adding `attention_masks`. This vector is basically telling the model which characters are meaningful and which one are used for padding. To do that, we put `1` in the position of meaningful tokens and `0` in the position of paddings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "As_arcISsOzj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_loader = DataLoader(train_data, batch_size=batch_size)\n",
        "validation_loader = DataLoader(dev_data, batch_size=batch_size)\n",
        "test_loader = DataLoader(test_data, batch_size=batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zzU2bxyd2uTg",
        "colab_type": "text"
      },
      "source": [
        "## Task 3. Initialize and train the model (1 point)\n",
        "\n",
        "Initialize the `AutoConfing` for your model. Put the correct number of labels to the `num_labels` parameters. This will be the output size of the last linear layer of the model.\n",
        "\n",
        "More information by the following link: https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoConfig\n",
        "\n",
        "_Hint_: You can access your label vocab with `vocab['upos']`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kV0LOnq5LAv7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "config = AutoConfig.from_pretrained(...)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ohw4__63fNU",
        "colab_type": "text"
      },
      "source": [
        "Initialize the `AutoModelForTokenClassification` for your model. Don't forget to use the `config`. \n",
        "\n",
        "More information by the following link: https://huggingface.co/transformers/model_doc/auto.html#automodelfortokenclassification"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_-Eae5oXsUak",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = AutoModelForTokenClassification.from_pretrained(...)\n",
        "\n",
        "model.to(device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pGtGdb_23x38",
        "colab_type": "text"
      },
      "source": [
        "You can see the model structure by running this cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LGhKQ6jsswJA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get all of the model's parameters as a list of tuples.\n",
        "params = list(model.named_parameters())\n",
        "\n",
        "print('The {} model has {:} different named parameters.\\n'.format(MODEL_NAME, len(params)))\n",
        "\n",
        "print('==== Embedding Layer ====\\n')\n",
        "\n",
        "emb_params = [p for p in params if 'embeddings' in p[0]]\n",
        "for p in emb_params:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== First Transformer ====\\n')\n",
        "\n",
        "first_transformer_params = [p for p in params if '.0.' in p[0]]\n",
        "for p in first_transformer_params:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== Output Layer ====\\n')\n",
        "\n",
        "cls_params = [p for p in params if 'classifier' in p[0]]\n",
        "for p in cls_params:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5V0U9TcK34R8",
        "colab_type": "text"
      },
      "source": [
        "Choose the appropriate learning rate and number of epochs. It is recommeded to train for 2-4 epochs with learning rate `{2e-5, 3e-5, 5e-5}`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A5gyvqU9s-ze",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = AdamW(model.parameters(), lr = ..., eps = 1e-8)\n",
        "\n",
        "epochs = ...\n",
        "# Total number of training steps is number of batches * number of epochs.\n",
        "total_steps = len(train_loader) * epochs\n",
        "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
        "                                            num_warmup_steps = 0,\n",
        "                                            num_training_steps = total_steps)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sU2GouCy4ToX",
        "colab_type": "text"
      },
      "source": [
        "Pay attention to the modified accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ClW0hk8ZtgQ4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def flat_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=2).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    # Trim the paddings\n",
        "    pred_flat = pred_flat[(labels_flat != LABEL_PAD_ID) & (labels_flat != PAD_ID)]\n",
        "    labels_flat = labels_flat[(labels_flat != LABEL_PAD_ID) & (labels_flat != PAD_ID)]\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G5G9QWPUtpcF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUB6saIt4XlC",
        "colab_type": "text"
      },
      "source": [
        "Finally, you can start the training. Make sure your loss is going down and accuracy is going up."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yfUd-Wr3t7rO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Taken from this tutorial: https://github.com/aniruddhachoudhury/BERT-Tutorials/blob/master/Blog%202/BERT_Fine_Tuning_Sentence_Classification.ipynb\n",
        "# The code was modified\n",
        "\n",
        "random.seed(random_seed)\n",
        "np.random.seed(random_seed)\n",
        "torch.manual_seed(random_seed)\n",
        "torch.cuda.manual_seed_all(random_seed)\n",
        "\n",
        "loss_values = []\n",
        "\n",
        "# For each epoch...\n",
        "for epoch_i in range(0, epochs):\n",
        "    \n",
        "    # ========================================\n",
        "    #               Training\n",
        "    # ========================================\n",
        "    \n",
        "    # Perform one full pass over the training set.\n",
        "\n",
        "    print(\"\")\n",
        "    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))\n",
        "    print('Training...')\n",
        "\n",
        "    # Measure how long the training epoch takes.\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Reset the total loss for this epoch.\n",
        "    total_loss = 0\n",
        "\n",
        "    # Put the model into training mode. Don't be mislead--the call to \n",
        "    # `train` just changes the *mode*, it doesn't *perform* the training.\n",
        "    # `dropout` and `batchnorm` layers behave differently during training\n",
        "    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)\n",
        "    model.train()\n",
        "\n",
        "    # For each batch of training data...\n",
        "    for step, batch in enumerate(train_loader):\n",
        "\n",
        "        # Progress update every 40 batches.\n",
        "        if step % 40 == 0 and not step == 0:\n",
        "            # Calculate elapsed time in minutes.\n",
        "            elapsed_mins, elapsed_secs = epoch_time(t0, time.time())\n",
        "            \n",
        "            # Report progress.\n",
        "            print(f'  Batch {step:>5,}  of  {len(train_loader):>5,}.    Elapsed: {elapsed_mins:}m {elapsed_secs:}s.')\n",
        "\n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "\n",
        "        # Always clear any previously calculated gradients before performing a\n",
        "        # backward pass. PyTorch doesn't do this automatically because \n",
        "        # accumulating the gradients is \"convenient while training RNNs\". \n",
        "        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)\n",
        "        model.zero_grad()        \n",
        "\n",
        "        # Perform a forward pass (evaluate the model on this training batch).\n",
        "        # This will return the loss (rather than the model output) because we\n",
        "        # have provided the `labels`.\n",
        "        # The documentation for this `model` function is here: \n",
        "        # https://huggingface.co/transformers/model_doc/bert.html#bertfortokenclassification\n",
        "        outputs = model(b_input_ids, \n",
        "                    attention_mask=b_input_mask,\n",
        "                    labels=b_labels)\n",
        "        \n",
        "        # The call to `model` always returns a tuple, so we need to pull the \n",
        "        # loss value out of the tuple.\n",
        "        loss = outputs[0]\n",
        "\n",
        "        # Accumulate the training loss over all of the batches so that we can\n",
        "        # calculate the average loss at the end. `loss` is a Tensor containing a\n",
        "        # single value; the `.item()` function just returns the Python value \n",
        "        # from the tensor.\n",
        "        total_loss += loss.item()\n",
        "\n",
        "        # Perform a backward pass to calculate the gradients.\n",
        "        loss.backward()\n",
        "\n",
        "        # Clip the norm of the gradients to 1.0.\n",
        "        # This is to help prevent the \"exploding gradients\" problem.\n",
        "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
        "\n",
        "        # Update parameters and take a step using the computed gradient.\n",
        "        # The optimizer dictates the \"update rule\"--how the parameters are\n",
        "        # modified based on their gradients, the learning rate, etc.\n",
        "        optimizer.step()\n",
        "\n",
        "        # Update the learning rate.\n",
        "        scheduler.step()\n",
        "\n",
        "    # Calculate the average loss over the training data.\n",
        "    avg_train_loss = total_loss / len(train_loader)            \n",
        "    \n",
        "    # Store the loss value for plotting the learning curve.\n",
        "    loss_values.append(avg_train_loss)\n",
        "\n",
        "    print(\"\")\n",
        "    print(f\"  Average training loss: {avg_train_loss:.2f}\")\n",
        "    print(\"  Training epcoh took: {:}m {:}s\".format(*epoch_time(t0, time.time())))\n",
        "        \n",
        "    # ========================================\n",
        "    #               Validation\n",
        "    # ========================================\n",
        "    # After the completion of each training epoch, measure our performance on\n",
        "    # our validation set.\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"Running Validation...\")\n",
        "\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Put the model in evaluation mode--the dropout layers behave differently\n",
        "    # during evaluation.\n",
        "    model.eval()\n",
        "\n",
        "    # Tracking variables \n",
        "    eval_loss, eval_accuracy = 0, 0\n",
        "    nb_eval_steps, nb_eval_examples = 0, 0\n",
        "\n",
        "    # Evaluate data for one epoch\n",
        "    for batch in validation_loader:\n",
        "        \n",
        "        # Add batch to GPU\n",
        "        batch = tuple(t.to(device) for t in batch)\n",
        "        \n",
        "        # Unpack the inputs from our dataloader\n",
        "        b_input_ids, b_input_mask, b_labels = batch\n",
        "        \n",
        "        # Telling the model not to compute or store gradients, saving memory and\n",
        "        # speeding up validation\n",
        "        with torch.no_grad():        \n",
        "\n",
        "            # Forward pass, calculate logit predictions.\n",
        "            # This will return the logits rather than the loss because we have\n",
        "            # not provided labels.\n",
        "            # The documentation for this `model` function is here: \n",
        "            # https://huggingface.co/transformers/model_doc/bert.html#bertfortokenclassification\n",
        "            outputs = model(b_input_ids, \n",
        "                            attention_mask=b_input_mask)\n",
        "        \n",
        "        # Get the \"logits\" output by the model. The \"logits\" are the output\n",
        "        # values prior to applying an activation function like the softmax.\n",
        "        logits = outputs[0]\n",
        "\n",
        "        # Move logits and labels to CPU\n",
        "        logits = logits.detach().cpu().numpy()\n",
        "        label_ids = b_labels.to('cpu').numpy()\n",
        "        \n",
        "        # Calculate the accuracy for this batch of test sentences.\n",
        "        tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n",
        "        \n",
        "        # Accumulate the total accuracy.\n",
        "        eval_accuracy += tmp_eval_accuracy\n",
        "\n",
        "        # Track the number of batches\n",
        "        nb_eval_steps += 1\n",
        "\n",
        "    # Report the final accuracy for this validation run.\n",
        "    print(\"  Accuracy: {0:.2f}\".format((eval_accuracy/nb_eval_steps) * 100))\n",
        "    print(\"  Validation took: {:}m {:}s\".format(*epoch_time(t0, time.time())))\n",
        "\n",
        "print(\"\")\n",
        "print(\"Training complete!\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wtG3p25Z4fvC",
        "colab_type": "text"
      },
      "source": [
        "See how the model performs on the test set. I managed to get `97.52` for `UD_English-EWT` and `97.55` for `UD_Estonian-EDT`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f33nbvQ1v2A6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"\")\n",
        "print(\"Running Testing...\")\n",
        "\n",
        "t0 = time.time()\n",
        "\n",
        "# Put the model in evaluation mode--the dropout layers behave differently\n",
        "# during evaluation.\n",
        "model.eval()\n",
        "\n",
        "# Tracking variables \n",
        "test_loss, test_accuracy = 0, 0\n",
        "nb_test_steps, nb_test_examples = 0, 0\n",
        "\n",
        "# Evaluate data for one epoch\n",
        "for batch in test_loader:\n",
        "    \n",
        "    # Add batch to GPU\n",
        "    batch = tuple(t.to(device) for t in batch)\n",
        "    \n",
        "    # Unpack the inputs from our dataloader\n",
        "    b_input_ids, b_input_mask, b_labels = batch\n",
        "    \n",
        "    # Telling the model not to compute or store gradients, saving memory and\n",
        "    # speeding up validation\n",
        "    with torch.no_grad():        \n",
        "\n",
        "        # Forward pass, calculate logit predictions.\n",
        "        # This will return the logits rather than the loss because we have\n",
        "        # not provided labels.\n",
        "        # The documentation for this `model` function is here: \n",
        "        # https://huggingface.co/transformers/model_doc/distilbert.html#distilbertforsequenceclassification\n",
        "        outputs = model(b_input_ids, \n",
        "                        attention_mask=b_input_mask)\n",
        "    \n",
        "    # Get the \"logits\" output by the model. The \"logits\" are the output\n",
        "    # values prior to applying an activation function like the softmax.\n",
        "    logits = outputs[0]\n",
        "\n",
        "    # Move logits and labels to CPU\n",
        "    logits = logits.detach().cpu().numpy()\n",
        "    label_ids = b_labels.to('cpu').numpy()\n",
        "    \n",
        "    # Calculate the accuracy for this batch of test sentences.\n",
        "    tmp_test_accuracy = flat_accuracy(logits, label_ids)\n",
        "    \n",
        "    # Accumulate the total accuracy.\n",
        "    test_accuracy += tmp_test_accuracy\n",
        "\n",
        "    # Track the number of batches\n",
        "    nb_test_steps += 1\n",
        "\n",
        "# Report the final accuracy for this test run.\n",
        "print(\"  Accuracy: {0:.2f}\".format((test_accuracy/nb_test_steps) * 100))\n",
        "print(\"  Testing took: {:}m {:}s\".format(*epoch_time(t0, time.time())))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IB3v9m2P4wAg",
        "colab_type": "text"
      },
      "source": [
        "## Task 4. Save the model (0 points)\n",
        "\n",
        "Study how to save and load your model to use it later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kiPRjiUT421s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "\n",
        "# Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()\n",
        "\n",
        "output_dir = './model_save/'\n",
        "\n",
        "# Create output directory if needed\n",
        "if not os.path.exists(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "\n",
        "print(\"Saving model to %s\" % output_dir)\n",
        "\n",
        "# Save a trained model, configuration and tokenizer using `save_pretrained()`.\n",
        "# They can then be reloaded using `from_pretrained()`\n",
        "model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training\n",
        "model_to_save.save_pretrained(output_dir)\n",
        "tokenizer.save_pretrained(output_dir)\n",
        "\n",
        "# Good practice: save your training arguments together with the trained model\n",
        "# torch.save(args, os.path.join(output_dir, 'training_args.bin'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sh7qN9t-6WIT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load a trained model and vocabulary that you have fine-tuned\n",
        "model = AutoModelForTokenClassification.from_pretrained(output_dir)\n",
        "tokenizer = AutoTokenizer.from_pretrained(output_dir)\n",
        "\n",
        "# Copy the model to the GPU.\n",
        "model.to(device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-8NxSSfP6mrK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "name": "Homework6_Transformers_SequenceTagging.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}