123028_Assignment3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Dependency Parser (Assignment)\n",
    "Everything should be done ON MY code, no new code.\n",
    "\n",
    "\n",
    "1. Read https://aclanthology.org/D14-1082.pdf and maybe just write one paragraph summary in your README.md in your github\n",
    "\n",
    "2. Do something called ablation study (meaning try to delete something so we know the impact of that deleted thing - very common in NLP)\n",
    "- Recall that we have 18 word + 18 pos + 12 dep features\n",
    "- Try to delete only the 12 dep features and check UAS\n",
    "- Try to delete only the 18 pos features and check UAS\n",
    "3. Do another comparison study testing the embedding\n",
    "Chaky uses some embedding\n",
    "Try to use (1) glove embedding (smallest), (2) nn.Embedding (train from scratch) and compare with Chaky's embedding - on how it affects the UAS\n",
    "4. Do some testing, compare 2-3 sentences with spaCy and see whether our neural network gives the same dependency.\n",
    "\n",
    "Criteria:\n",
    "0: not done\n",
    "1: ok\n",
    "2: with comments/explanation like how Chaky does his tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1. Read https://aclanthology.org/D14-1082.pdf and maybe just write one paragraph summary in your README.md in your github\n",
    "//Already created readme in my github"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Do something called ablation study (meaning try to delete something so we know the impact of that deleted thing - very common in NLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.1 Recall that we have 18 word + 18 pos + 12 dep features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "import time\n",
    "import os\n",
    "import logging\n",
    "from collections import Counter\n",
    "from datetime import datetime\n",
    "import math\n",
    "\n",
    "from tqdm import tqdm\n",
    "import pickle\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch import nn, optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the dependency parser, there will be lot of component.\n",
    "\n",
    "1. The transition parsing function\n",
    "2. Loading the data (conll)\n",
    "3. The actual parser\n",
    "4. The embedding lookup\n",
    "5. The deep learning model\n",
    "6. The training and evaluation\n",
    "\n",
    "Alert:  there will be a lot of coding today!\n",
    "\n",
    "**Note**:  In real life (production), you probably just use the existing tool such as `spaCy` (which we will learn).  Here we just learn so we can develop a deep understanding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing function\n",
    "\n",
    "We gonna start with a class `Parsing`, representing a parser for each sentence.  For each sentence, we need the `stack`, `buffer`, and the `dependencies`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Parsing(object):\n",
    "    def __init__(self, sentence):\n",
    "        self.sentence = sentence #['The', 'cat', 'sat']\n",
    "        \n",
    "        # The current stack represented as a list with the top of the stack as the\n",
    "        # last element of the list.\n",
    "        self.stack = ['ROOT']\n",
    "        \n",
    "        # The current buffer represented as a list with the first item on the\n",
    "        # buffer as the first item of the list\n",
    "        self.buffer = sentence[:]\n",
    "        \n",
    "        # The list of dependencies produced so far. Represented as a list of\n",
    "        # tuples where each tuple is of the form (head, dependent).\n",
    "        self.dep = []\n",
    "    \n",
    "    #Let's first create a function to perform \"Shift\", \"Left-Arc\", and \"Right-Arc\"\n",
    "    def parse_step(self, transition):\n",
    "        if transition == 'S':\n",
    "            buffer_head = self.buffer.pop(0)\n",
    "            self.stack.append(buffer_head)\n",
    "        #stack: [ROOT, I, parsed]\n",
    "        elif transition == 'LA':\n",
    "            dependent = self.stack.pop(-2) #I\n",
    "            self.dep.append((self.stack[-1], dependent)) #(parsed, I)\n",
    "        #stack: [ROOT, parsed, sentence]\n",
    "        elif transition == 'RA':\n",
    "            dependent = self.stack.pop() #sentence\n",
    "            self.dep.append((self.stack[-1], dependent)) #(parsed, sentence)\n",
    "        else:\n",
    "            print(f\"Unknown transition: {transition}\")\n",
    "            \n",
    "    #loop all transitions\n",
    "    def parse(self, transitions):\n",
    "        for transition in transitions:\n",
    "            self.parse_step(transition)\n",
    "        return self.dep\n",
    "    \n",
    "    #an utility function to check that the parsing is done.\n",
    "    def is_completed(self):\n",
    "        return (len(self.buffer) == 0) and (len(self.stack) == 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the parse step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['ROOT'], ['The', 'cat', 'sat'], [])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsing = Parsing([\"The\", \"cat\", \"sat\"])\n",
    "parsing.stack, parsing.buffer, parsing.dep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['ROOT', 'The'], ['cat', 'sat'], [])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#let's try shift\n",
    "parsing.parse_step(\"S\")\n",
    "parsing.stack, parsing.buffer, parsing.dep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['The'], ['cat', 'sat'], [('The', 'ROOT')])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#let's try shift\n",
    "parsing.parse_step(\"LA\")\n",
    "parsing.stack, parsing.buffer, parsing.dep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['The', 'cat'], ['sat'], [('The', 'ROOT')])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#let's try shift\n",
    "parsing.parse_step(\"S\")\n",
    "parsing.stack, parsing.buffer, parsing.dep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['The'], ['sat'], [('The', 'ROOT'), ('The', 'cat')])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#let's try shift\n",
    "parsing.parse_step(\"RA\")\n",
    "parsing.stack, parsing.buffer, parsing.dep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the parse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['The'], ['sat'], [('The', 'ROOT'), ('The', 'cat')])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsing = Parsing([\"The\", \"cat\", \"sat\"])\n",
    "parsing.parse([\"S\", \"LA\", \"S\", \"RA\"])\n",
    "parsing.stack, parsing.buffer, parsing.dep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Minibatch parsing\n",
    "\n",
    "We gonna create a minibatch loader that loads a bunch of sentences, and perform parse accordingly.  For now, we will assume a very dump model to predict the transitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def minibatch_parse(sentences, model, batch_size):\n",
    "    dep = []\n",
    "\n",
    "    # Initialize a list of DepParser, one for each sentence.\n",
    "    partial_parses = [Parsing(sentence) for sentence in sentences]\n",
    "    \n",
    "    # Shallow copy partial_parses.\n",
    "    unfinished_parses = partial_parses[:]\n",
    "        \n",
    "    # While unfinished_parses is not empty do.\n",
    "    while unfinished_parses:\n",
    "        # Take first batch_size parses in unfinished_parses as minibatch.\n",
    "        minibatch = unfinished_parses[:batch_size]\n",
    "        # Use model to predict next transition for each partial parse in the minibatch.\n",
    "        transitions = model.predict(minibatch)\n",
    "        # Perform parse step for each partial_parse_minibatch with their predicted transition.\n",
    "        for transition, partial_parse in zip(transitions, minibatch):\n",
    "            partial_parse.parse_step(transition)\n",
    "        # Remove completed parses.\n",
    "        unfinished_parses[:] = [\n",
    "            p for p in unfinished_parses if not p.is_completed()]\n",
    "    dep = [parse.dep for parse in partial_parses]\n",
    "\n",
    "    return dep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DummyModel(object):\n",
    "    \"\"\"\n",
    "    Dummy model for testing the minibatch_parse function\n",
    "    First shifts everything onto the stack and then does exclusively right arcs if the first word of\n",
    "    the sentence is \"right\", \"left\" if otherwise.\n",
    "    \"\"\"\n",
    "\n",
    "    def predict(self, partial_parses):\n",
    "        return [(\"RA\" if pp.stack[1] == \"right\" else \"LA\") if len(pp.buffer) == 0 else \"S\"\n",
    "                for pp in partial_parses]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the minibatch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [[\"right\", \"arcs\", \"only\"],\n",
    "             [\"right\", \"arcs\", \"only\", \"again\"],\n",
    "             [\"left\", \"arcs\", \"only\"],\n",
    "             [\"left\", \"arcs\", \"only\", \"again\"]]\n",
    "deps = minibatch_parse(sentences, DummyModel(), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('arcs', 'only'), ('right', 'arcs'), ('ROOT', 'right')],\n",
       " [('only', 'again'), ('arcs', 'only'), ('right', 'arcs'), ('ROOT', 'right')],\n",
       " [('only', 'arcs'), ('only', 'left'), ('only', 'ROOT')],\n",
       " [('again', 'only'), ('again', 'arcs'), ('again', 'left'), ('again', 'ROOT')]]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data\n",
    "\n",
    "We used English Penn Treebank dataset in CoNLL format.\n",
    "\n",
    "CoNLL is the conventional name for TSV formats in NLP (TSV - tab-separated values, i.e., CSV with <TAB> as separator).\n",
    "It originates from a series of shared tasks organized at the Conferences of Natural Language Learning (hence the name)\n",
    "\n",
    "In CoNLL formats,\n",
    "- every word (token) is represented in one line\n",
    "- every sentence is separated from the next by an empty line\n",
    "- every column represents one annotation\n",
    "\n",
    "There are many formats, in our case, our conll file has 10 columns, the important columns are:\n",
    "- 1:  word\n",
    "- 4:  pos\n",
    "- 6:  head of the dependency\n",
    "- 7:  type of dependency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define the config for easy management\n",
    "class Config(object):\n",
    "    data_path = ''\n",
    "    train_file = 'train.conll'\n",
    "    dev_file = 'dev.conll'\n",
    "    test_file = 'test.conll'\n",
    "    embedding_file = 'en-cw.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for reading the conll file\n",
    "def read_conll(filename, max_example=5):\n",
    "    examples = []\n",
    "    with open(filename) as f:\n",
    "        i = 0\n",
    "        word, pos, head, dep = [], [], [], []\n",
    "        for line in f.readlines():\n",
    "            i = i + 1\n",
    "            wa = line.strip().split('\\t') #wa = word annotations\n",
    "            \n",
    "            #if all 10 columns are there....\n",
    "            if len(wa) == 10:  \n",
    "                word.append(wa[1].lower())\n",
    "                pos.append(wa[4])\n",
    "                head.append(int(wa[6]))\n",
    "                dep.append(wa[7])\n",
    "            \n",
    "            #otherwise, it means the line is empty, thus we consider finish for one example\n",
    "            elif len(word) > 0:\n",
    "                examples.append({'word': word, 'pos': pos, 'head': head, 'dep': dep})\n",
    "                word, pos, head, dep = [], [], [], []\n",
    "                    \n",
    "        #last example\n",
    "        if len(word) > 0:\n",
    "            examples.append({'word': word, 'pos': pos, 'head': head, 'dep': dep})\n",
    "            \n",
    "    return examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the load function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "#sample_size = (train, dev, test)\n",
    "def load_data(sample_size=(1000, 500, 500)):\n",
    "    config = Config()\n",
    "\n",
    "    print(\"1. Loading data...\",)\n",
    "    start = time.time()\n",
    "    train_set = read_conll(os.path.join(config.data_path, config.train_file))\n",
    "    dev_set   = read_conll(os.path.join(config.data_path, config.dev_file))\n",
    "    test_set  = read_conll(os.path.join(config.data_path, config.test_file))\n",
    "\n",
    "    #for shorter preprocessing time.....just for education, ok guys?\n",
    "    train_set = train_set[:sample_size[0]]\n",
    "    dev_set   = dev_set[:sample_size[1]]\n",
    "    test_set  = test_set[:sample_size[2]]\n",
    "    print(\"Example 1: \", train_set[1]) #i choose one because it's short\n",
    "\n",
    "    print(\"took {:.2f} seconds\".format(time.time() - start))\n",
    "    \n",
    "    return train_set, dev_set, test_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the preprocessing data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "INITIALIZING\n",
      "================================================================================\n",
      "1. Loading data...\n",
      "Example 1:  {'word': ['ms.', 'haag', 'plays', 'elianti', '.'], 'pos': ['NNP', 'NNP', 'VBZ', 'NNP', '.'], 'head': [2, 3, 0, 3, 3], 'dep': ['compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
      "took 3.00 seconds\n"
     ]
    }
   ],
   "source": [
    "print(80 * \"=\")\n",
    "print(\"INITIALIZING\")\n",
    "print(80 * \"=\")\n",
    "train_set, dev_set, test_set = load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand, we can draw these in a dependency tree, with the help of spaCy.  **Note** that spaCy do not draw the ROOT for us, but imagine the head of \"plays\" is ROOT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"e3b930f9b1d4495988b20d3315865eae-0\" class=\"displacy\" width=\"925\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Ms.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">Haag</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">plays</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">Elianti</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">PUNCT</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e3b930f9b1d4495988b20d3315865eae-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,89.5 220.0,89.5 220.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e3b930f9b1d4495988b20d3315865eae-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e3b930f9b1d4495988b20d3315865eae-0-1\" stroke-width=\"2px\" d=\"M245,177.0 C245,89.5 395.0,89.5 395.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e3b930f9b1d4495988b20d3315865eae-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M245,179.0 L237,167.0 253,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e3b930f9b1d4495988b20d3315865eae-0-2\" stroke-width=\"2px\" d=\"M420,177.0 C420,89.5 570.0,89.5 570.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e3b930f9b1d4495988b20d3315865eae-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M570.0,179.0 L578.0,167.0 562.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-e3b930f9b1d4495988b20d3315865eae-0-3\" stroke-width=\"2px\" d=\"M420,177.0 C420,2.0 750.0,2.0 750.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-e3b930f9b1d4495988b20d3315865eae-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">punct</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M750.0,179.0 L758.0,167.0 742.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "compound\n",
      "nsubj\n",
      "ROOT\n",
      "dobj\n",
      "punct\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Ms. Haag plays Elianti .\")\n",
    "options = {\"collapse_punct\": False}\n",
    "displacy.render(doc, options = options, style=\"dep\", jupyter=True)\n",
    "\n",
    "for token in doc:\n",
    "    print(token.dep_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Parser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "We gonna create a parser that gonna help us:\n",
    "- create a `tok2id` dictionary in the `__init__` function\n",
    "- numercalize `numercalize` the words, dependencies, and pos tags\n",
    "- create training data, `create_instances` by leveraging the ground truth of the dependencies\n",
    "- finally the `parse` function\n",
    "\n",
    "This feature vector consists of a list of tokens. They can be represented as a list of integers $\\mathbf{w} = [w_1, w_2, \\cdots, w_m]$ where $m$ is the number of features and each $0 \\leq w_i \\leq |V|$ is the index of a token in the vocabulary ($|V|$ is the vocabulary size).  Then our network looks up an embedding for each word and concatenates them into a single input vector:\n",
    "\n",
    "$$\\mathbf{x} = [\\mathbf{E}_{w_1}, \\cdots, \\mathbf{E}_{w_m}] \\in \\mathbb{R}^{dm}$$\n",
    "\n",
    "where $\\mathbf{E} \\in \\mathbb{R}^{|V| \\times d}$ is an embedding matrix with each row $\\mathbf{E}_w$ as the vector for a particular word $w$\n",
    "\n",
    "#### Features\n",
    "\n",
    "A total of 18 + 18 + 12 features were used in the paper.\n",
    "\n",
    "- Feature 1: 18 features\n",
    "  - (a). 6 - top 3 words on buffer, top 3 words on stack, \n",
    "  - (b). 6 - the first and second left most/rightmost children and the leftmost/rightmost of the **top word** (i.e., `stack[-1]`) on the stack,  - `(leftmost(0), rightmost(0), secondleftmost(0), secondrightmost(0), leftmost(leftmost(0)), rightmost(rightmost(0)))`\n",
    "  - (c). 6 - the first and second left most/rightmost children and the leftmost/rightmost of the **second top word** (i.e., `stack[-2]`) on the stack, - `(leftmost(0), rightmost(0), secondleftmost(0), secondrightmost(0), leftmost(leftmost(0)), rightmost(rightmost(0)))`\n",
    "- Feature 2: 18 pos - basically corresponding POS tags\n",
    "- Feature 3: 12 dep - corresponding ARC, excluding 6 words on the stack/buffer..\n",
    "\n",
    "**NOTE**: There is no 3a because each word itself does not have dependency\n",
    "\n",
    "**NOTE**: For brevity, the table skipped 2b, 2c\n",
    "\n",
    "**NOTE**: Here, in the dependency column, I used LABELED dependency.   But in my code, it is unlabeled, thus e.g., COMPOUND will not be needed.\n",
    "\n",
    "| Stack | Buffer | Feature 1a | Feature 1b | Feature 1c | Feature 2a | Feature 3b | Feature 3c | Dependency (ARC) | Transition |\n",
    "| :--   |  :--   | :--        | :--        | :--        | :--        | :--        | :--        | :--               | :--        |\n",
    "| [ROOT] | [Ms., Haag, plays, Elianti, .] | [NULL, NULL, ROOT, Ms., Haag, plays] | [NULL, NULL, NULL, NULL, NULL, NULL] | [NULL, NULL, NULL, NULL, NULL, NULL] | [P-NULL, P-NULL, P-ROOT, PROPN, PROPN, VERB] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL]  | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL]| | Init |\n",
    "| [ROOT, Ms.] | [Haag, plays, Elianti, .] | [NULL, ROOT, Ms., Haag, plays, Elianti] | [NULL, NULL, NULL, NULL, NULL, NULL]  | [NULL, NULL, NULL, NULL, NULL, NULL] | [P-NULL, P-ROOT, PROPN, PROPN, VERB, PROPN] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL]  | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL]| | SHIFT |\n",
    "| [ROOT, Ms., Haag] | [plays, Elianti, .] | [ROOT, Ms., Haag, plays, Elianti, .] | [NULL, NULL, NULL, NULL, NULL, NULL]  | [NULL, NULL, NULL, NULL, NULL, NULL] | [P-ROOT, PROPN, PROPN, VERB, PROPN, PUNCT] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL]  | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL]|  | SHIFT |\n",
    "| [ROOT, Haag] | [plays, Elianti, .] | [NULL, ROOT, Haag, plays, Elianti, .] | [Ms., NULL, NULL, NULL, NULL, NULL, NULL] | [NULL, NULL, NULL, NULL, NULL, NULL] | [P-NULL, P-ROOT, PROPN, VERB, PROPN, PUNCT] | [COMPOUND, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | (Haag, Ms., COMPOUND) | LEFTARC |\n",
    "| [ROOT, Haag, plays] | [Elianti, .] | [ROOT, Haag, plays, Elianti, ., NULL] | [NULL, NULL, NULL, NULL, NULL, NULL] | [Ms., NULL, NULL, NULL, NULL, NULL] | [P-ROOT, PROPN, VERB, PROPN, PUNCT, P-NULL] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | [COMPOUND, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | (Haag, Ms., COMPOUND) | SHIFT |\n",
    "| [ROOT, plays] | [Elianti, .] | [NULL, ROOT, plays, Elianti, ., NULL] | [Haag, NULL, NULL, NULL, Ms., NULL] | [NULL, NULL, NULL, NULL, NULL, NULL] | [P-NULL, P-ROOT, VERB, PROPN, PUNCT, P-NULL] | [NSUBJ, D-NULL, D-NULL, D-NULL, COMPOUND, D-NULL] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | (Haag, Ms., COMPOUND), (plays, Haag, NSUBJ) | LEFTARC |\n",
    "| [ROOT, plays, Elianti] | [.] | [ROOT, plays, Elianti, ., NULL, NULL] | [NULL, NULL, NULL, NULL, NULL, NULL] | [Haag, NULL, NULL, NULL, Ms., NULL] | [P-ROOT, VERB, PROPN, PUNCT, P-NULL, P-NULL] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | [NSUBJ, D-NULL, D-NULL, D-NULL, COMPOUND, D-NULL] | (Haag, Ms., COMPOUND), (plays, Haag, NSUBJ) | SHIFT |\n",
    "| [ROOT, plays] | [.] | [NULL, ROOT, plays, ., NULL, NULL] | [Haag, Elianti, NULL, NULL, Ms., NULL] | [NULL, NULL, NULL, NULL, NULL, NULL] | [P-NULL, P-ROOT, VERB, PUNCT, P-NULL, P-NULL] | [NSUBJ, DOBJ, D-NULL, D-NULL, COMPOUND, D-NULL] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | (Haag, Ms., COMPOUND), (plays, Haag, NSUBJ), (plays, Elianti, DOBJ) | RIGHTARC |\n",
    "| [ROOT, plays, .] | [] | [ROOT, plays, ., NULL, NULL, NULL] | [NULL, NULL, NULL, NULL, NULL, NULL] | [Haag, Elianti, NULL, NULL, Ms., NULL] | [P-NULL, P-ROOT, VERB, PUNCT, P-NULL, P-NULL] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | [NSUBJ, DOBJ, D-NULL, D-NULL, COMPOUND, D-NULL] | (Haag, Ms., COMPOUND), (plays, Haag, NSUBJ), (plays, Elianti, DOBJ) | SHIFT |\n",
    "| [ROOT, plays] | [] | [NULL, ROOT, plays, NULL, NULL, NULL] | [Haag, Elianti, NULL, plays, Ms., NULL] | [Haag, Elianti, NULL, NULL, Ms., NULL] | [P-NULL, P-ROOT, VERB, P-NULL, P-NULL, P-NULL] | [NSUBJ, PUNCT, D-NULL, DOBJ, COMPOUND, D-NULL] | [D-NULL, D-NULL, D-NULL, D-NULL, D-NULL, D-NULL] | (Haag, Ms., COMPOUND), (plays, Haag, NSUBJ), (plays, Elianti, DOBJ), (plays, ., PUNCT) | RIGHT-ARC |\n",
    "| [ROOT] | [] |  | |  |  |  |  | (Haag, Ms., COMPOUND), (plays, Haag, NSUBJ), (plays, Elianti, DOBJ), (plays, ., PUNCT), (ROOT, plays, ROOT) | RIGHT-ARC |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "P_PREFIX = '<p>:' #indicating pos tags\n",
    "D_PREFIX = '<d>:' #indicating dependency tags\n",
    "UNK      = '<UNK>'\n",
    "NULL     = '<NULL>'\n",
    "ROOT     = '<ROOT>'\n",
    "\n",
    "class Parser(object):\n",
    "\n",
    "    def __init__(self, dataset):\n",
    "        \n",
    "        #set the root dep\n",
    "        self.root_dep = 'root'\n",
    "                \n",
    "        #get all the dep of the dataset as list, e.g., ['root', 'acl', 'nmod', 'nmod:npmod']\n",
    "        all_dep = [self.root_dep] + list(set([w for ex in dataset\n",
    "                                               for w in ex['dep']\n",
    "                                               if w != self.root_dep]))\n",
    "        \n",
    "        #1. put dep into tok2id lookup table, with D_PREFIX so we know it is dependency\n",
    "        #{'D_PREFIX:root': 0, 'D_PREFIX:acl': 1, 'D_PREFIX:nmod': 2, ..., 'D_PREFIX:<NULL>': 30}\n",
    "        tok2id = {D_PREFIX + l: i for (i, l) in enumerate(all_dep)}\n",
    "        tok2id[D_PREFIX + NULL] = self.D_NULL = len(tok2id)\n",
    "        \n",
    "        #we are using \"unlabeled\" where we do not label with the dependency\n",
    "        #thus the number of dependency relation is 1\n",
    "        trans = ['L', 'R', 'S']\n",
    "        self.n_deprel = 1\n",
    "        \n",
    "        #create a simple lookup table mapping action and id\n",
    "        #e.g., tran2id: {'L': 0, 'R': 1, 'S': 2}\n",
    "        #e.g., id2tran: {0: 'L', 1: 'R', 2: 'S'}\n",
    "        self.n_trans = len(trans)\n",
    "        self.tran2id = {t: i for (i, t) in enumerate(trans)}\n",
    "        self.id2tran = {i: t for (i, t) in enumerate(trans)}\n",
    "\n",
    "        #2. put pos tags into tok2id lookup table, with P_PREFIX so we know it is pos\n",
    "        tok2id.update(build_dict([P_PREFIX + w for ex in dataset for w in ex['pos']],\n",
    "                                  offset=len(tok2id)))\n",
    "        tok2id[P_PREFIX + UNK]  = self.P_UNK  = len(tok2id)  #also remember the pos tags of unknown\n",
    "        tok2id[P_PREFIX + NULL] = self.P_NULL = len(tok2id)\n",
    "        tok2id[P_PREFIX + ROOT] = self.P_ROOT = len(tok2id)\n",
    "        \n",
    "        #now tok2id:  {'P_PREFIX:root': 0, 'P_PREFIX:acl': 1, ..., 'P_PREFIX:JJR': 62, 'P_PREFIX:<UNK>': 63, 'P_PREFIX:<NULL>': 64, 'P_PREFIX:<ROOT>': 65}\n",
    "        \n",
    "        #3. put word into tok2id lookup table\n",
    "        tok2id.update(build_dict([w for ex in dataset for w in ex['word']],\n",
    "                                  offset=len(tok2id)))\n",
    "        tok2id[UNK]  = self.UNK = len(tok2id)\n",
    "        tok2id[NULL] = self.NULL = len(tok2id)\n",
    "        tok2id[ROOT] = self.ROOT = len(tok2id)\n",
    "        \n",
    "        #now tok2id: {'D_PREFIX:root': 0, 'D_PREFIX:acl': 1, 'D_PREFIX:nmod': 2, ..., 'memory': 340, 'mr.': 341, '<UNK>': 342, '<NULL>': 343, '<ROOT>': 344}\n",
    "        \n",
    "        #create id2tok\n",
    "        self.tok2id = tok2id\n",
    "        self.id2tok = {v: k for (k, v) in tok2id.items()}\n",
    "        \n",
    "        #why 18 normal features + 18 (pos) + 12 (dep)\n",
    "        #18 features - top 3 words on buffer, top 3 words on stack, \n",
    "        # the first and second left most/rightmost children of the top two words on the stack\n",
    "        # the leftmost of leftmost/rightmost of rightmost children of the top two words on the stack\n",
    "        #18 pos - basically corresponding POS tags\n",
    "        #12 dep - corresponding ARC, excluding 6 words on hte stack/buffer..\n",
    "        self.n_features = 18 + 18 + 12\n",
    "        self.n_tokens = len(tok2id)\n",
    "    \n",
    "    #function to turn train set with words to train set with id instead using tok2id\n",
    "    def numericalize(self, examples):\n",
    "        numer_examples = []\n",
    "        for ex in examples:\n",
    "            word = [self.ROOT] + [self.tok2id[w] if w in self.tok2id\n",
    "                                  else self.UNK for w in ex['word']]\n",
    "            pos  = [self.P_ROOT] + [self.tok2id[P_PREFIX + w] if P_PREFIX + w in self.tok2id\n",
    "                                   else self.P_UNK for w in ex['pos']]\n",
    "            head = [-1] + ex['head']\n",
    "            dep  = [-1] + [self.tok2id[D_PREFIX + w] if D_PREFIX + w in self.tok2id\n",
    "                            else -1 for w in ex['dep']]\n",
    "            numer_examples.append({'word': word, 'pos': pos,\n",
    "                                 'head': head, 'dep': dep})\n",
    "        return numer_examples\n",
    "\n",
    "    #function to extract features to form a feature embedding matrix\n",
    "    def extract_features(self, stack, buf, arcs, ex):\n",
    "             \n",
    "        #ex['word']:  [55, 32, 33, 34, 35, 30], i.e., ['root', 'ms.', 'haag', 'plays', 'elianti', '.']\n",
    "        #ex['pos']:   [29, 14, 14, 16, 14, 17], i.e., ['NNP', 'NNP', 'VBZ', 'NNP', '.']\n",
    "        #ex['head']:  [-1, 2, 3, 0, 3, 3]  or ['root', 'compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
    "        #ex['dep']:   [-1, 1, 2, 0, 6, 12] or ['compound', 'nsubj', 'root', 'dobj', 'punct']\n",
    "\n",
    "        #stack     :  [0]\n",
    "        #buffer    :  [1, 2, 3, 4, 5]\n",
    "        \n",
    "        if stack[0] == \"ROOT\":\n",
    "            stack[0] = 0  #start the stack with [ROOT]\n",
    "\n",
    "        #get leftmost children based on the dependency arcs\n",
    "        def get_lc(k):\n",
    "            return sorted([arc[1] for arc in arcs if arc[0] == k and arc[1] < k])\n",
    "\n",
    "        #get right most children based on the dependency arcs\n",
    "        def get_rc(k):\n",
    "            return sorted([arc[1] for arc in arcs if arc[0] == k and arc[1] > k],\n",
    "                          reverse=True)\n",
    "\n",
    "        p_features = [] #pos features (2a, 2b, 2c) - 18\n",
    "        d_features = [] #dep features (3b, 3c) - 12\n",
    "        \n",
    "        #last 3 things on the stack as features\n",
    "        #if the stack is less than 3, then we simply append NULL from the left\n",
    "        features = [self.NULL] * (3 - len(stack)) + [ex['word'][x] for x in stack[-3:]]\n",
    "        \n",
    "        # next 3 things on the buffer as features\n",
    "        #if the buffer is less than 3, simply append NULL\n",
    "        #the reason why NULL is appended on end because buffer is read left to right\n",
    "        features += [ex['word'][x] for x in buf[:3]] + [self.NULL] * (3 - len(buf))\n",
    "        \n",
    "        #corresponding pos tags\n",
    "        p_features = [self.P_NULL] * (3 - len(stack)) + [ex['pos'][x] for x in stack[-3:]]\n",
    "        p_features += [ex['pos'][x] for x in buf[:3]] + [self.P_NULL] * (3 - len(buf))\n",
    "        \n",
    "        #get the leftmost and rightmost children of the top two words, thus we loop 2 times\n",
    "        for i in range(2):\n",
    "            if i < len(stack):\n",
    "                k = stack[-i-1] #-1, -2 last two in the stack\n",
    "                \n",
    "                #the first and second lefmost/rightmost children of the top two words (i=1, 2) on the stack\n",
    "                lc = get_lc(k)  \n",
    "                rc = get_rc(k)\n",
    "                \n",
    "                #the leftmost of leftmost/rightmost of rightmost children of the top two words on the stack:\n",
    "                llc = get_lc(lc[0]) if len(lc) > 0 else []\n",
    "                rrc = get_rc(rc[0]) if len(rc) > 0 else []\n",
    "\n",
    "                #(leftmost of first word on stack, rightmost of first word, \n",
    "                # leftmost of the second word on stack, rightmost of second, \n",
    "                # leftmost of leftmost, rightmost of rightmost\n",
    "                features.append(ex['word'][lc[0]] if len(lc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][rc[0]] if len(rc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][lc[1]] if len(lc) > 1 else self.NULL)\n",
    "                features.append(ex['word'][rc[1]] if len(rc) > 1 else self.NULL)\n",
    "                features.append(ex['word'][llc[0]] if len(llc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][rrc[0]] if len(rrc) > 0 else self.NULL)\n",
    "\n",
    "                #corresponding pos\n",
    "                p_features.append(ex['pos'][lc[0]] if len(lc) > 0 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][rc[0]] if len(rc) > 0 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][lc[1]] if len(lc) > 1 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][rc[1]] if len(rc) > 1 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][llc[0]] if len(llc) > 0 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][rrc[0]] if len(rrc) > 0 else self.P_NULL)\n",
    "            \n",
    "                #corresponding dep\n",
    "                d_features.append(ex['dep'][lc[0]] if len(lc) > 0 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][rc[0]] if len(rc) > 0 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][lc[1]] if len(lc) > 1 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][rc[1]] if len(rc) > 1 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][llc[0]] if len(llc) > 0 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][rrc[0]] if len(rrc) > 0 else self.D_NULL)\n",
    "                \n",
    "            else:\n",
    "                #attach NULL when they don't exist\n",
    "                features += [self.NULL] * 6\n",
    "                p_features += [self.P_NULL] * 6\n",
    "                d_features += [self.D_NULL] * 6\n",
    "\n",
    "        features += p_features + d_features\n",
    "        assert len(features) == self.n_features  #assert they are 18 + 18 + 12\n",
    "        return features\n",
    "\n",
    "    #decide whether to shift, leftarc, or rightarc, based on gold parse trees\n",
    "    #this is needed to create training examples which contain samples and ground truth\n",
    "    def get_oracle(self, stack, buf, ex):\n",
    "\n",
    "        #leave if the stack is only 1, thus nothing to predict....\n",
    "        if len(stack) < 2:\n",
    "            return self.n_trans - 1\n",
    "\n",
    "        #predict based on the last two words on the stack\n",
    "        i0 = stack[-1]\n",
    "        i1 = stack[-2]\n",
    "\n",
    "        #get the head and dependency\n",
    "        h0 = ex['head'][i0]\n",
    "        h1 = ex['head'][i1]\n",
    "        d0 = ex['dep'][i0]\n",
    "        d1 = ex['dep'][i1]\n",
    "\n",
    "        #either shift, left arc or right arc\n",
    "        #\"Shift\" = 2; \"LA\" = 0; \"RA\" = 1\n",
    "        #if head of the second last word is the last word, then leftarc\n",
    "        if (i1 > 0) and (h1 == i0):\n",
    "            return 0\n",
    "        #if head of the last word is the second last word, then rightarc\n",
    "        #make sure nothing in the buffer has head with the last word on the stack\n",
    "        #otherwise, we lose the last word.....\n",
    "        elif (i1 >= 0) and (h0 == i1) and \\\n",
    "                (not any([x for x in buf if ex['head'][x] == i0])):\n",
    "            return 1\n",
    "        #otherwise shift, if something is left in buffer, otherwise, do nothing....\n",
    "        else:\n",
    "            return None if len(buf) == 0 else 2\n",
    "\n",
    "    #generate training examples\n",
    "    #from the training sentences and their gold parse trees \n",
    "    def create_instances(self, examples):\n",
    "        all_instances = []\n",
    "        \n",
    "        for i, ex in enumerate(examples):\n",
    "            #e.g., ex['word]: [344, 163, 99, 164, 165, 68]\n",
    "            n_words = len(ex['word']) - 1  #excluding the root\n",
    "\n",
    "            #arcs = {(head, tail, dependency label)}\n",
    "            stack = [0]\n",
    "            buf = [i + 1 for i in range(n_words)]  #[1, 2, 3, 4, 5]\n",
    "            arcs = []\n",
    "            instances = []\n",
    "            \n",
    "            #because that's the maximum number of shift, leftarcs, rightarcs you can have\n",
    "            #this will determine the sample size of each training example\n",
    "            #if given five words, we will get a sample of (10, 48) where 10 comes from 5 * 2, and 48 is n_features\n",
    "            #but this for loop can be break if there is nothing left....\n",
    "            for i in range(n_words * 2):\n",
    "\n",
    "                #get the gold transition based on the parse trees\n",
    "                #gold_t can be either shift(2), leftarc(0), or rightarc(1)\n",
    "                gold_t = self.get_oracle(stack, buf, ex)\n",
    "                \n",
    "                #if gold_t is None, no need to extract features.....\n",
    "                if gold_t is None:\n",
    "                    break\n",
    "                \n",
    "                #make sure when the model predicts, we inform the current state of stack and buffer, so\n",
    "                #the model is not allowed to make any illegal action, e.g., buffer is empty but trying to pop\n",
    "                legal_labels = self.legal_labels(stack, buf)                \n",
    "                assert legal_labels[gold_t] == 1\n",
    "               \n",
    "                #extract all the 48 features \n",
    "                features = self.extract_features(stack, buf, arcs, ex)\n",
    "                instances.append((features, legal_labels, gold_t))\n",
    "            \n",
    "                #shift \n",
    "                if gold_t == 2:\n",
    "                    stack.append(buf[0])\n",
    "                    buf = buf[1:]\n",
    "                #left arc \n",
    "                elif gold_t == 0:\n",
    "                    arcs.append((stack[-1], stack[-2], gold_t))\n",
    "                    stack = stack[:-2] + [stack[-1]]\n",
    "                #right arc\n",
    "                else:\n",
    "                    arcs.append((stack[-2], stack[-1], gold_t - self.n_deprel))\n",
    "                    stack = stack[:-1]\n",
    "            else:\n",
    "                all_instances += instances\n",
    "\n",
    "        return all_instances\n",
    "\n",
    "    #provide an one hot encoding of the labels\n",
    "    def legal_labels(self, stack, buf):\n",
    "        labels =  ([1] if len(stack) > 2  else [0]) * self.n_deprel   #left arc   But cannot ROOT <----He thus 3\n",
    "        labels += ([1] if len(stack) >= 2 else [0]) * self.n_deprel   #right arc  ROOT--->He\n",
    "        labels += [1] if len(buf) > 0 else [0]   #shift\n",
    "        return labels\n",
    "    \n",
    "    #a simple function to check punctuation POS tags\n",
    "    def punct(self, pos):\n",
    "        return pos in [\"''\", \",\", \".\", \":\", \"``\", \"-LRB-\", \"-RRB-\"]\n",
    "\n",
    "    def parse(self, dataset, eval_batch_size=5000):\n",
    "        sentences = []\n",
    "        sentence_id_to_idx = {}\n",
    "                \n",
    "        for i, example in enumerate(dataset):\n",
    "            \n",
    "            #example['word']=[188, 186, 186, ..., 59]\n",
    "            #n_words=37\n",
    "            #sentence=[1, 2, 3, 4, 5,.., 37]\n",
    "                        \n",
    "            n_words = len(example['word']) - 1\n",
    "            sentence = [j + 1 for j in range(n_words)]            \n",
    "            sentences.append(sentence)\n",
    "            \n",
    "            #mapping the object unique id to the i            \n",
    "            #The id is the object's memory address\n",
    "            sentence_id_to_idx[id(sentence)] = i\n",
    "            \n",
    "        model = ModelWrapper(self, dataset, sentence_id_to_idx)\n",
    "        dependencies = minibatch_parse(sentences, model, eval_batch_size)\n",
    "                \n",
    "        UAS = all_tokens = 0.0\n",
    "        with tqdm(total=len(dataset)) as prog:\n",
    "            for i, ex in enumerate(dataset):\n",
    "                head = [-1] * len(ex['word'])\n",
    "                for h, t, in dependencies[i]:\n",
    "                    head[t] = h\n",
    "                for pred_h, gold_h, gold_l, pos in \\\n",
    "                        zip(head[1:], ex['head'][1:], ex['dep'][1:], ex['pos'][1:]):\n",
    "                        assert self.id2tok[pos].startswith(P_PREFIX)\n",
    "                        pos_str = self.id2tok[pos][len(P_PREFIX):]\n",
    "                        if (not self.punct(pos_str)):\n",
    "                            UAS += 1 if pred_h == gold_h else 0\n",
    "                            all_tokens += 1\n",
    "                prog.update(i + 1)\n",
    "        UAS /= all_tokens\n",
    "        return UAS, dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelWrapper(object):\n",
    "    def __init__(self, parser, dataset, sentence_id_to_idx):\n",
    "        self.parser = parser\n",
    "        self.dataset = dataset\n",
    "        self.sentence_id_to_idx = sentence_id_to_idx\n",
    "\n",
    "    def predict(self, partial_parses):\n",
    "        mb_x = [self.parser.extract_features(p.stack, p.buffer, p.dep,\n",
    "                                             self.dataset[self.sentence_id_to_idx[id(p.sentence)]])\n",
    "                for p in partial_parses]\n",
    "        mb_x = np.array(mb_x).astype('int32')\n",
    "        mb_x = torch.from_numpy(mb_x).long()\n",
    "        mb_l = [self.parser.legal_labels(p.stack, p.buffer) for p in partial_parses]\n",
    "\n",
    "        pred = self.parser.model(mb_x)\n",
    "        pred = pred.detach().numpy()\n",
    "        \n",
    "        #we need to multiply 10000 with legal labels, to force the model not to make any impossible prediction\n",
    "        #other, when we parse sequentially, sometimes there is nothing in the buffer or stack, thus error....        \n",
    "        pred = np.argmax(pred + 10000 * np.array(mb_l).astype('float32'), 1)\n",
    "        pred = [\"S\" if p == 2 else (\"LA\" if p == 0 else \"RA\") for p in pred]\n",
    "        \n",
    "        return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#a simple function to create ids.....\n",
    "def build_dict(keys, offset=0):\n",
    "    #keys = ['P_PREFIX:IN', 'P_PREFIX:DT', 'P_PREFIX:NNP', 'P_PREFIX:CD', so on...]\n",
    "    #offset is needed because this tok2id has something already inside....\n",
    "    count = Counter()\n",
    "    for key in keys:\n",
    "        count[key] += 1\n",
    "    \n",
    "    #most_common = [('P_PREFIX:NN', 70), ('P_PREFIX:IN', 57), ... , ('P_PREFIX:JJR', 1)]\n",
    "    #we use most_common in case we only want some maximum pos tags....\n",
    "    mc = count.most_common()\n",
    "    \n",
    "    #{'P_PREFIX:NN': 31, 'P_PREFIX:IN': 32, .., 'P_PREFIX:JJR': 62} \n",
    "    return {w[0]: index + offset for (index, w) in enumerate(mc)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the parser `__init__`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2. Building parser...\n",
      "took 0.03 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"2. Building parser...\",)\n",
    "start = time.time()\n",
    "parser = Parser(train_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the `numericalize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word:  ['ms.', 'haag', 'plays', 'elianti', '.']\n",
      "Pos:   ['NNP', 'NNP', 'VBZ', 'NNP', '.']\n",
      "Head:  [2, 3, 0, 3, 3]\n",
      "Dep:   ['compound', 'nsubj', 'root', 'dobj', 'punct']\n"
     ]
    }
   ],
   "source": [
    "#before numericalize\n",
    "print(\"Word: \",  train_set[1]['word'])\n",
    "print(\"Pos:  \",  train_set[1]['pos'])\n",
    "print(\"Head: \",  train_set[1]['head'])\n",
    "print(\"Dep:  \",  train_set[1]['dep'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3. Numericalizing data...\n",
      "took 0.07 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"3. Numericalizing data...\",)\n",
    "start = time.time()\n",
    "train_set = parser.numericalize(train_set)\n",
    "dev_set   = parser.numericalize(dev_set)\n",
    "test_set  = parser.numericalize(test_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5156, 304, 1364, 1002, 2144, 87]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (rootis added in front)\n",
    "train_set[1]['word']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<ROOT>\n",
      "ms.\n",
      "haag\n",
      "plays\n",
      "elianti\n",
      ".\n"
     ]
    }
   ],
   "source": [
    "for i in train_set[1]['word']:\n",
    "    print(parser.id2tok[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[84, 42, 42, 55, 42, 46]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (pos tag root is added in front)\n",
    "train_set[1]['pos']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<p>:<ROOT>\n",
      "<p>:NNP\n",
      "<p>:NNP\n",
      "<p>:VBZ\n",
      "<p>:NNP\n",
      "<p>:.\n"
     ]
    }
   ],
   "source": [
    "for i in train_set[1]['pos']:\n",
    "    print(parser.id2tok[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-1, 2, 3, 0, 3, 3]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (root has no head, so we assign -1)\n",
    "train_set[1]['head']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-1, 25, 12, 0, 28, 9]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (root has no dep, so we assign -1)\n",
    "train_set[1]['dep']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Embedding\n",
    "\n",
    "Word embedding length of 50.  In the paper, they applied a custom 50-embedding to all the words, pos, and dependencies.  For pos and dependencies, they claimed that there are some similarities that can be learned as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4. Loading pretrained embeddings...\n",
      "Embedding matrix shape (vocab, emb size):  (5157, 50)\n",
      "took 2.69 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"4. Loading pretrained embeddings...\",)\n",
    "config = Config()\n",
    "start = time.time()\n",
    "word_vectors = {}\n",
    "for line in open(config.embedding_file).readlines():\n",
    "    we = line.strip().split() #we = word embeddings - first column: word;  the rest is embedding\n",
    "    word_vectors[we[0]] = [float(x) for x in we[1:]] #{word: [list of 50 numbers], nextword: [another list], so on...}\n",
    "    \n",
    "#create an empty embedding matrix holding the embedding lookup table (vocab size, embed dim)\n",
    "#we use random.normal instead of zeros, to keep the embedding matrix arbitrary in case word vectors don't exist....\n",
    "embeddings_matrix = np.asarray(np.random.normal(0, 0.9, (parser.n_tokens, 50)), dtype='float32')\n",
    "\n",
    "for token in parser.tok2id:\n",
    "        i = parser.tok2id[token]\n",
    "        if token in word_vectors:\n",
    "            embeddings_matrix[i] = word_vectors[token]\n",
    "        elif token.lower() in word_vectors:\n",
    "            embeddings_matrix[i] = word_vectors[token.lower()]\n",
    "print(\"Embedding matrix shape (vocab, emb size): \", embeddings_matrix.shape)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5. Preprocessing training data...\n",
      "took 1.85 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"5. Preprocessing training data...\",)\n",
    "start = time.time()\n",
    "train_examples = parser.create_instances(train_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Minibatch loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_minibatches(data, minibatch_size, shuffle=True):\n",
    "    data_size = len(data[0])\n",
    "    indices = np.arange(data_size)\n",
    "    if shuffle:\n",
    "        np.random.shuffle(indices)\n",
    "    for minibatch_start in np.arange(0, data_size, minibatch_size):\n",
    "        minibatch_indices = indices[minibatch_start:minibatch_start + minibatch_size]\n",
    "        yield [_minibatch(d, minibatch_indices) for d in data]\n",
    "\n",
    "def _minibatch(data, minibatch_idx):\n",
    "    return data[minibatch_idx] if type(data) is np.ndarray else [data[i] for i in minibatch_idx]\n",
    "\n",
    "def minibatches(data, batch_size):\n",
    "    x = np.array([d[0] for d in data])\n",
    "    y = np.array([d[2] for d in data])\n",
    "    one_hot = np.zeros((y.size, 3))\n",
    "    one_hot[np.arange(y.size), y] = 1\n",
    "    return get_minibatches([x, one_hot], batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing your minibatch loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i, (train_x, train_y) in enumerate(minibatches(train_examples, 1024)):\n",
    "#     print(train_x.shape)  #batch size, features\n",
    "#     print(train_y.shape)        #one hot encoding of 3 actions - shift, la, ra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network\n",
    "\n",
    "Let's train a neural network to predict, given the state of the stack, buffer, and dependencies, which transition should be applied next.\n",
    "\n",
    "Recall that our input vector is:\n",
    "\n",
    "$$\\mathbf{x} = [\\mathbf{E}_{w_1}, \\cdots, \\mathbf{E}_{w_m}] \\in \\mathbb{R}^{dm}$$\n",
    "\n",
    "where $\\mathbf{E} \\in \\mathbb{R}^{|V| \\times d}$ is an embedding matrix with each row $\\mathbf{E}_w$ as the vector for a particular word $w$\n",
    "\n",
    "We then compute our prediction as:\n",
    "\n",
    "$$\\mathbf{h} = \\text{ReLU}(\\mathbf{xW} + \\mathbf{b}_1)$$\n",
    "$$\\mathbf{l} = \\mathbf{hU} + \\mathbf{b}_2$$\n",
    "$$\\hat{\\mathbf{y}} = \\text{softmax}(l)$$\n",
    "\n",
    "where $\\mathbf{h}$ is referred to as the hidden layer, $\\mathbf{l}$ is the logits, $\\hat{\\mathbf{y}}$ is the predictions, and $\\text{ReLU}(z) = \\text{max}(z, 0))$.  We will then train the model to minimize cross-entropy (CE) loss:\n",
    "\n",
    "$$J(\\theta) = \\text{CE}(\\mathbf{y}, \\hat{\\mathbf{y}}) = -\\sum_{i=1}^{3}y_i \\log \\hat{y}_i$$\n",
    "\n",
    "To compute the loss for the training set, we average this $J(\\theta)$ across all training examples.  We will use UAS (Unlabeled Attachment Score) as main metric, which is computed as the ratio between number of correctly predicted dependencies and the number of total dependencies despite of the relations (which our model doesn't predict the relation since is unlabeled). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ParserModel(nn.Module):\n",
    "\n",
    "    def __init__(self, embeddings, n_features=48,\n",
    "                 hidden_size=400, n_classes=3, dropout_prob=0.5):\n",
    "\n",
    "        super(ParserModel, self).__init__()\n",
    "        self.n_features   = n_features\n",
    "        self.n_classes    = n_classes\n",
    "        self.dropout_prob = dropout_prob\n",
    "        self.embed_size   = embeddings.shape[1]\n",
    "        self.hidden_size  = hidden_size\n",
    "        self.pretrained_embeddings = nn.Embedding(embeddings.shape[0], self.embed_size)\n",
    "        self.pretrained_embeddings.weight = nn.Parameter(torch.tensor(embeddings))\n",
    "\n",
    "        self.embed_to_hidden = nn.Linear(n_features * self.embed_size, hidden_size)\n",
    "        nn.init.xavier_uniform_(self.embed_to_hidden.weight, gain=1.)\n",
    "        self.dropout = nn.Dropout(p=dropout_prob)\n",
    "        self.hidden_to_logits = nn.Linear(hidden_size, n_classes)\n",
    "        nn.init.xavier_uniform_(self.hidden_to_logits.weight)\n",
    "\n",
    "    def embedding_lookup(self, t):\n",
    "        #t:  batch_size, n_features\n",
    "        batch_size = t.size()[0]\n",
    "                    \n",
    "        x = self.pretrained_embeddings(t)        \n",
    "        x = x.reshape(-1, self.n_features * self.embed_size)\n",
    "        # x = (1024, 48 * 50)\n",
    "\n",
    "        return x\n",
    "\n",
    "    def forward(self, t):\n",
    "        # t: (1024, 48)\n",
    "        embeddings = self.embedding_lookup(t)  \n",
    "    \n",
    "        # embeddings: (1024, 48 * 50)\n",
    "        hidden = self.embed_to_hidden(embeddings)\n",
    "    \n",
    "        # hidden: (1024, 200)\n",
    "        hidden_activations = F.relu(hidden)\n",
    "        # hidden_activations: (1024, 200)\n",
    "        thin_net = self.dropout(hidden_activations)\n",
    "        # thin_net: (1024, 200)\n",
    "        logits = self.hidden_to_logits(thin_net)\n",
    "        # logits: (1024, 3)\n",
    "\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now complete the <code>train_for_epoch</code> and <code>train</code> functions to actually train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#just a class to get the average.....\n",
    "class AverageMeter(object):\n",
    "    \"\"\"Computes and stores the average and current value\"\"\"\n",
    "    def __init__(self):\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self):\n",
    "        self.val = 0\n",
    "        self.avg = 0\n",
    "        self.sum = 0\n",
    "        self.count = 0\n",
    "\n",
    "    def update(self, val, n=1):\n",
    "        self.val = val\n",
    "        self.sum += val * n\n",
    "        self.count += n\n",
    "        self.avg = self.sum / self.count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(parser, train_data, dev_data, output_path, batch_size=1024, n_epochs=10, lr=0.0005):\n",
    "    \n",
    "    best_dev_UAS = 0\n",
    "    \n",
    "    optimizer = optim.Adam(parser.model.parameters(), lr=0.001)\n",
    "    loss_func = nn.CrossEntropyLoss()\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        print(\"Epoch {:} out of {:}\".format(epoch + 1, n_epochs))\n",
    "        dev_UAS = train_for_epoch(\n",
    "            parser, train_data, dev_data, optimizer, loss_func, batch_size)\n",
    "        if dev_UAS > best_dev_UAS:\n",
    "            best_dev_UAS = dev_UAS\n",
    "            print(\"New best dev UAS! Saving model.\")\n",
    "            torch.save(parser.model.state_dict(), output_path)\n",
    "        print(\"\")\n",
    "\n",
    "\n",
    "def train_for_epoch(parser, train_data, dev_data, optimizer, loss_func, batch_size):\n",
    "    \n",
    "    parser.model.train()  # Places model in \"train\" mode, i.e. apply dropout layer\n",
    "    n_minibatches = math.ceil(len(train_data) / batch_size)\n",
    "    loss_meter = AverageMeter()\n",
    "\n",
    "    with tqdm(total=(n_minibatches)) as prog:\n",
    "        for i, (train_x, train_y) in enumerate(minibatches(train_data, batch_size)):\n",
    "            \n",
    "            #train_x:  batch_size, n_features\n",
    "            #train_y:  batch_size, target(=3)\n",
    "            \n",
    "            optimizer.zero_grad() \n",
    "            loss = 0.\n",
    "            train_x = torch.from_numpy(train_x).long()  #long() for int so embedding works....\n",
    "            train_y = torch.from_numpy(train_y.nonzero()[1]).long()  #get the index with 1 because torch expects label to be single integer\n",
    "\n",
    "            # Forward pass: compute predicted logits.\n",
    "            logits = parser.model(train_x)\n",
    "            # Compute loss\n",
    "            loss = loss_func(logits, train_y)\n",
    "            # Compute gradients of the loss w.r.t model parameters.\n",
    "            loss.backward()\n",
    "            # Take step with optimizer.\n",
    "            optimizer.step()\n",
    "\n",
    "            prog.update(1)\n",
    "            loss_meter.update(loss.item())\n",
    "\n",
    "    print(\"Average Train Loss: {}\".format(loss_meter.avg))\n",
    "    print(\"Evaluating on dev set\",)\n",
    "    parser.model.eval()  # Places model in \"eval\" mode, i.e. don't apply dropout layer\n",
    "        \n",
    "    dev_UAS, _ = parser.parse(dev_data)\n",
    "    print(\"- dev UAS: {:.2f}\".format(dev_UAS * 100.0))\n",
    "    return dev_UAS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "TRAINING\n",
      "================================================================================\n",
      "Epoch 1 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 24.66it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.8773038300375143\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6686393.65it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 53.49\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 2 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 24.50it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.34603272813061875\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6559082.27it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 62.65\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 3 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 25.02it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.2739281163861354\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6622167.86it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 65.09\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 4 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 26.09it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.23824124100307623\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6585145.61it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 68.17\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 5 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 26.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.2107957989598314\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6734393.60it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 70.84\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 6 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 25.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.18975919050474963\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6413348.02it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 71.59\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 7 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 25.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.172836784273386\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6702174.91it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 71.74\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 8 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 25.93it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.1532217807446917\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6498312.46it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 73.29\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 9 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 25.91it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.14028875467677912\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6461783.98it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 74.82\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 10 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 25.05it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.13193910320599875\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6692697.23it/s]       "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 75.48\n",
      "New best dev UAS! Saving model.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#create directory if it does not exist for saving the weights...\n",
    "output_dir = \"output/{:%Y%m%d_%H%M%S}/\".format(datetime.now())\n",
    "output_path = output_dir + \"model.weights\"\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "    \n",
    "print(80 * \"=\")\n",
    "print(\"TRAINING\")\n",
    "print(80 * \"=\")\n",
    "    \n",
    "model = ParserModel(embeddings_matrix)\n",
    "parser.model = model\n",
    "\n",
    "start = time.time()\n",
    "train(parser, train_examples, dev_set, output_path,\n",
    "      batch_size=1024, n_epochs=10, lr=0.0005)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "TESTING\n",
      "================================================================================\n",
      "Restoring the best model weights found on the dev set\n",
      "Final evaluation on test set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6741307.05it/s]       "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- test UAS: 76.46\n",
      "Done!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print(80 * \"=\")\n",
    "print(\"TESTING\")\n",
    "print(80 * \"=\")\n",
    "\n",
    "print(\"Restoring the best model weights found on the dev set\")\n",
    "parser.model.load_state_dict(torch.load(output_path))\n",
    "print(\"Final evaluation on test set\",)\n",
    "parser.model.eval()\n",
    "UAS, dependencies = parser.parse(test_set)\n",
    "print(\"- test UAS: {:.2f}\".format(UAS * 100.0))\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Try to delete only the 12 dep features and check UAS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "P_PREFIX = '<p>:' #indicating pos tags\n",
    "D_PREFIX = '<d>:' #indicating dependency tags\n",
    "UNK      = '<UNK>'\n",
    "NULL     = '<NULL>'\n",
    "ROOT     = '<ROOT>'\n",
    "\n",
    "class Parser(object):\n",
    "\n",
    "    def __init__(self, dataset):\n",
    "        \n",
    "        #set the root dep\n",
    "        self.root_dep = 'root'\n",
    "                \n",
    "        #get all the dep of the dataset as list, e.g., ['root', 'acl', 'nmod', 'nmod:npmod']\n",
    "        all_dep = [self.root_dep] + list(set([w for ex in dataset for w in ex['dep'] if w != self.root_dep]))\n",
    "        \n",
    "        #1. put dep into tok2id lookup table, with D_PREFIX so we know it is dependency\n",
    "        #{'D_PREFIX:root': 0, 'D_PREFIX:acl': 1, 'D_PREFIX:nmod': 2, ..., 'D_PREFIX:<NULL>': 30}\n",
    "        tok2id = {D_PREFIX + l: i for (i, l) in enumerate(all_dep)}\n",
    "        tok2id[D_PREFIX + NULL] = self.D_NULL = len(tok2id)\n",
    "        \n",
    "        #we are using \"unlabeled\" where we do not label with the dependency\n",
    "        #thus the number of dependency relation is 1\n",
    "        trans = ['L', 'R', 'S']\n",
    "        self.n_deprel = 1\n",
    "        \n",
    "        #create a simple lookup table mapping action and id\n",
    "        #e.g., tran2id: {'L': 0, 'R': 1, 'S': 2}\n",
    "        #e.g., id2tran: {0: 'L', 1: 'R', 2: 'S'}\n",
    "        self.n_trans = len(trans)\n",
    "        self.tran2id = {t: i for (i, t) in enumerate(trans)}\n",
    "        self.id2tran = {i: t for (i, t) in enumerate(trans)}\n",
    "\n",
    "        #2. put pos tags into tok2id lookup table, with P_PREFIX so we know it is pos\n",
    "        tok2id.update(build_dict([P_PREFIX + w for ex in dataset for w in ex['pos']],\n",
    "                                  offset=len(tok2id)))\n",
    "        tok2id[P_PREFIX + UNK]  = self.P_UNK  = len(tok2id)  #also remember the pos tags of unknown\n",
    "        tok2id[P_PREFIX + NULL] = self.P_NULL = len(tok2id)\n",
    "        tok2id[P_PREFIX + ROOT] = self.P_ROOT = len(tok2id)\n",
    "        \n",
    "        #now tok2id:  {'P_PREFIX:root': 0, 'P_PREFIX:acl': 1, ..., 'P_PREFIX:JJR': 62, 'P_PREFIX:<UNK>': 63, 'P_PREFIX:<NULL>': 64, 'P_PREFIX:<ROOT>': 65}\n",
    "        \n",
    "        #3. put word into tok2id lookup table\n",
    "        tok2id.update(build_dict([w for ex in dataset for w in ex['word']],\n",
    "                                  offset=len(tok2id)))\n",
    "        tok2id[UNK]  = self.UNK = len(tok2id)\n",
    "        tok2id[NULL] = self.NULL = len(tok2id)\n",
    "        tok2id[ROOT] = self.ROOT = len(tok2id)\n",
    "        \n",
    "        #now tok2id: {'D_PREFIX:root': 0, 'D_PREFIX:acl': 1, 'D_PREFIX:nmod': 2, ..., 'memory': 340, 'mr.': 341, '<UNK>': 342, '<NULL>': 343, '<ROOT>': 344}\n",
    "        \n",
    "        #create id2tok\n",
    "        self.tok2id = tok2id\n",
    "        self.id2tok = {v: k for (k, v) in tok2id.items()}\n",
    "        \n",
    "        #why 18 normal features + 18 (pos) + 12 (dep)\n",
    "        #18 features - top 3 words on buffer, top 3 words on stack, \n",
    "        # the first and second left most/rightmost children of the top two words on the stack\n",
    "        # the leftmost of leftmost/rightmost of rightmost children of the top two words on the stack\n",
    "        #18 pos - basically corresponding POS tags\n",
    "        #12 dep - corresponding ARC, excluding 6 words on hte stack/buffer..\n",
    "        self.n_features = 18 + 18\n",
    "        self.n_tokens = len(tok2id)\n",
    "    \n",
    "    #function to turn train set with words to train set with id instead using tok2id\n",
    "    def numericalize(self, examples):\n",
    "        numer_examples = []\n",
    "        for ex in examples:\n",
    "            word = [self.ROOT] + [self.tok2id[w] if w in self.tok2id\n",
    "                                  else self.UNK for w in ex['word']]\n",
    "            pos  = [self.P_ROOT] + [self.tok2id[P_PREFIX + w] if P_PREFIX + w in self.tok2id\n",
    "                                   else self.P_UNK for w in ex['pos']]\n",
    "            head = [-1] + ex['head']\n",
    "            numer_examples.append({'word': word, 'pos': pos,\n",
    "                                 'head': head})\n",
    "        return numer_examples\n",
    "\n",
    "    #function to extract features to form a feature embedding matrix\n",
    "    def extract_features(self, stack, buf, arcs, ex):\n",
    "             \n",
    "        #ex['word']:  [55, 32, 33, 34, 35, 30], i.e., ['root', 'ms.', 'haag', 'plays', 'elianti', '.']\n",
    "        #ex['pos']:   [29, 14, 14, 16, 14, 17], i.e., ['NNP', 'NNP', 'VBZ', 'NNP', '.']\n",
    "        #ex['head']:  [-1, 2, 3, 0, 3, 3]  or ['root', 'compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
    "        #ex['dep']:   [-1, 1, 2, 0, 6, 12] or ['compound', 'nsubj', 'root', 'dobj', 'punct']\n",
    "\n",
    "        #stack     :  [0]\n",
    "        #buffer    :  [1, 2, 3, 4, 5]\n",
    "        \n",
    "        if stack[0] == \"ROOT\":\n",
    "            stack[0] = 0  #start the stack with [ROOT]\n",
    "\n",
    "        #get leftmost children based on the dependency arcs\n",
    "        def get_lc(k):\n",
    "            return sorted([arc[1] for arc in arcs if arc[0] == k and arc[1] < k])\n",
    "\n",
    "        #get right most children based on the dependency arcs\n",
    "        def get_rc(k):\n",
    "            return sorted([arc[1] for arc in arcs if arc[0] == k and arc[1] > k],\n",
    "                          reverse=True)\n",
    "\n",
    "        p_features = [] #pos features (2a, 2b, 2c) - 18\n",
    "        d_features = [] #dep features (3b, 3c) - 12\n",
    "        \n",
    "        #last 3 things on the stack as features\n",
    "        #if the stack is less than 3, then we simply append NULL from the left\n",
    "        features = [self.NULL] * (3 - len(stack)) + [ex['word'][x] for x in stack[-3:]]\n",
    "        \n",
    "        # next 3 things on the buffer as features\n",
    "        #if the buffer is less than 3, simply append NULL\n",
    "        #the reason why NULL is appended on end because buffer is read left to right\n",
    "        features += [ex['word'][x] for x in buf[:3]] + [self.NULL] * (3 - len(buf))\n",
    "        \n",
    "        #corresponding pos tags\n",
    "        p_features = [self.P_NULL] * (3 - len(stack)) + [ex['pos'][x] for x in stack[-3:]]\n",
    "        p_features += [ex['pos'][x] for x in buf[:3]] + [self.P_NULL] * (3 - len(buf))\n",
    "        \n",
    "        #get the leftmost and rightmost children of the top two words, thus we loop 2 times\n",
    "        for i in range(2):\n",
    "            if i < len(stack):\n",
    "                k = stack[-i-1] #-1, -2 last two in the stack\n",
    "                \n",
    "                #the first and second lefmost/rightmost children of the top two words (i=1, 2) on the stack\n",
    "                lc = get_lc(k)  \n",
    "                rc = get_rc(k)\n",
    "                \n",
    "                #the leftmost of leftmost/rightmost of rightmost children of the top two words on the stack:\n",
    "                llc = get_lc(lc[0]) if len(lc) > 0 else []\n",
    "                rrc = get_rc(rc[0]) if len(rc) > 0 else []\n",
    "\n",
    "                #(leftmost of first word on stack, rightmost of first word, \n",
    "                # leftmost of the second word on stack, rightmost of second, \n",
    "                # leftmost of leftmost, rightmost of rightmost\n",
    "                features.append(ex['word'][lc[0]] if len(lc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][rc[0]] if len(rc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][lc[1]] if len(lc) > 1 else self.NULL)\n",
    "                features.append(ex['word'][rc[1]] if len(rc) > 1 else self.NULL)\n",
    "                features.append(ex['word'][llc[0]] if len(llc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][rrc[0]] if len(rrc) > 0 else self.NULL)\n",
    "\n",
    "                #corresponding pos\n",
    "                p_features.append(ex['pos'][lc[0]] if len(lc) > 0 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][rc[0]] if len(rc) > 0 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][lc[1]] if len(lc) > 1 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][rc[1]] if len(rc) > 1 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][llc[0]] if len(llc) > 0 else self.P_NULL)\n",
    "                p_features.append(ex['pos'][rrc[0]] if len(rrc) > 0 else self.P_NULL)\n",
    "                \n",
    "            else:\n",
    "                #attach NULL when they don't exist\n",
    "                features += [self.NULL] * 6\n",
    "                p_features += [self.P_NULL] * 6\n",
    "\n",
    "        features += p_features\n",
    "        assert len(features) == self.n_features  #assert they are 18 + 18\n",
    "        return features\n",
    "\n",
    "    #decide whether to shift, leftarc, or rightarc, based on gold parse trees\n",
    "    #this is needed to create training examples which contain samples and ground truth\n",
    "    def get_oracle(self, stack, buf, ex):\n",
    "\n",
    "        #leave if the stack is only 1, thus nothing to predict....\n",
    "        if len(stack) < 2:\n",
    "            return self.n_trans - 1\n",
    "\n",
    "        #predict based on the last two words on the stack\n",
    "        i0 = stack[-1]\n",
    "        i1 = stack[-2]\n",
    "\n",
    "        #get the head and dependency\n",
    "        h0 = ex['head'][i0]\n",
    "        h1 = ex['head'][i1]\n",
    "\n",
    "        #either shift, left arc or right arc\n",
    "        #\"Shift\" = 2; \"LA\" = 0; \"RA\" = 1\n",
    "        #if head of the second last word is the last word, then leftarc\n",
    "        if (i1 > 0) and (h1 == i0):\n",
    "            return 0\n",
    "        #if head of the last word is the second last word, then rightarc\n",
    "        #make sure nothing in the buffer has head with the last word on the stack\n",
    "        #otherwise, we lose the last word.....\n",
    "        elif (i1 >= 0) and (h0 == i1) and \\\n",
    "                (not any([x for x in buf if ex['head'][x] == i0])):\n",
    "            return 1\n",
    "        #otherwise shift, if something is left in buffer, otherwise, do nothing....\n",
    "        else:\n",
    "            return None if len(buf) == 0 else 2\n",
    "\n",
    "    #generate training examples\n",
    "    #from the training sentences and their gold parse trees \n",
    "    def create_instances(self, examples):\n",
    "        all_instances = []\n",
    "        \n",
    "        for i, ex in enumerate(examples):\n",
    "            #e.g., ex['word]: [344, 163, 99, 164, 165, 68]\n",
    "            n_words = len(ex['word']) - 1  #excluding the root\n",
    "\n",
    "            #arcs = {(head, tail, dependency label)}\n",
    "            stack = [0]\n",
    "            buf = [i + 1 for i in range(n_words)]  #[1, 2, 3, 4, 5]\n",
    "            arcs = []\n",
    "            instances = []\n",
    "            \n",
    "            #because that's the maximum number of shift, leftarcs, rightarcs you can have\n",
    "            #this will determine the sample size of each training example\n",
    "            #if given five words, we will get a sample of (10, 48) where 10 comes from 5 * 2, and 48 is n_features\n",
    "            #but this for loop can be break if there is nothing left....\n",
    "            for i in range(n_words * 2):\n",
    "\n",
    "                #get the gold transition based on the parse trees\n",
    "                #gold_t can be either shift(2), leftarc(0), or rightarc(1)\n",
    "                gold_t = self.get_oracle(stack, buf, ex)\n",
    "                \n",
    "                #if gold_t is None, no need to extract features.....\n",
    "                if gold_t is None:\n",
    "                    break\n",
    "                \n",
    "                #make sure when the model predicts, we inform the current state of stack and buffer, so\n",
    "                #the model is not allowed to make any illegal action, e.g., buffer is empty but trying to pop\n",
    "                legal_labels = self.legal_labels(stack, buf)                \n",
    "                assert legal_labels[gold_t] == 1\n",
    "               \n",
    "                #extract all the 48 features \n",
    "                features = self.extract_features(stack, buf, arcs, ex)\n",
    "                instances.append((features, legal_labels, gold_t))\n",
    "            \n",
    "                #shift \n",
    "                if gold_t == 2:\n",
    "                    stack.append(buf[0])\n",
    "                    buf = buf[1:]\n",
    "                #left arc \n",
    "                elif gold_t == 0:\n",
    "                    arcs.append((stack[-1], stack[-2], gold_t))\n",
    "                    stack = stack[:-2] + [stack[-1]]\n",
    "                #right arc\n",
    "                else:\n",
    "                    arcs.append((stack[-2], stack[-1], gold_t - self.n_deprel))\n",
    "                    stack = stack[:-1]\n",
    "            else:\n",
    "                all_instances += instances\n",
    "\n",
    "        return all_instances\n",
    "\n",
    "    #provide an one hot encoding of the labels\n",
    "    def legal_labels(self, stack, buf):\n",
    "        labels =  ([1] if len(stack) > 2  else [0]) * self.n_deprel   #left arc   But cannot ROOT <----He thus 3\n",
    "        labels += ([1] if len(stack) >= 2 else [0]) * self.n_deprel   #right arc  ROOT--->He\n",
    "        labels += [1] if len(buf) > 0 else [0]   #shift\n",
    "        return labels\n",
    "    \n",
    "    #a simple function to check punctuation POS tags\n",
    "    def punct(self, pos):\n",
    "        return pos in [\"''\", \",\", \".\", \":\", \"``\", \"-LRB-\", \"-RRB-\"]\n",
    "\n",
    "    def parse(self, dataset, eval_batch_size=5000):\n",
    "        sentences = []\n",
    "        sentence_id_to_idx = {}\n",
    "                \n",
    "        for i, example in enumerate(dataset):\n",
    "            \n",
    "            #example['word']=[188, 186, 186, ..., 59]\n",
    "            #n_words=37\n",
    "            #sentence=[1, 2, 3, 4, 5,.., 37]\n",
    "                        \n",
    "            n_words = len(example['word']) - 1\n",
    "            sentence = [j + 1 for j in range(n_words)]            \n",
    "            sentences.append(sentence)\n",
    "            \n",
    "            #mapping the object unique id to the i            \n",
    "            #The id is the object's memory address\n",
    "            sentence_id_to_idx[id(sentence)] = i\n",
    "            \n",
    "        model = ModelWrapper(self, dataset, sentence_id_to_idx)\n",
    "        dependencies = minibatch_parse(sentences, model, eval_batch_size)\n",
    "                \n",
    "        UAS = all_tokens = 0.0\n",
    "        with tqdm(total=len(dataset)) as prog:\n",
    "            for i, ex in enumerate(dataset):\n",
    "                head = [-1] * len(ex['word'])\n",
    "                for h, t, in dependencies[i]:\n",
    "                    head[t] = h\n",
    "                for pred_h, gold_h, pos in \\\n",
    "                        zip(head[1:], ex['head'][1:], ex['pos'][1:]):\n",
    "                        assert self.id2tok[pos].startswith(P_PREFIX)\n",
    "                        pos_str = self.id2tok[pos][len(P_PREFIX):]\n",
    "                        if (not self.punct(pos_str)):\n",
    "                            UAS += 1 if pred_h == gold_h else 0\n",
    "                            all_tokens += 1\n",
    "                prog.update(i + 1)\n",
    "        UAS /= all_tokens\n",
    "        return UAS, dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelWrapper(object):\n",
    "    def __init__(self, parser, dataset, sentence_id_to_idx):\n",
    "        self.parser = parser\n",
    "        self.dataset = dataset\n",
    "        self.sentence_id_to_idx = sentence_id_to_idx\n",
    "\n",
    "    def predict(self, partial_parses):\n",
    "        mb_x = [self.parser.extract_features(p.stack, p.buffer, p.dep,\n",
    "                                             self.dataset[self.sentence_id_to_idx[id(p.sentence)]])\n",
    "                for p in partial_parses]\n",
    "        mb_x = np.array(mb_x).astype('int32')\n",
    "        mb_x = torch.from_numpy(mb_x).long()\n",
    "        mb_l = [self.parser.legal_labels(p.stack, p.buffer) for p in partial_parses]\n",
    "\n",
    "        pred = self.parser.model(mb_x)\n",
    "        pred = pred.detach().numpy()\n",
    "        \n",
    "        #we need to multiply 10000 with legal labels, to force the model not to make any impossible prediction\n",
    "        #other, when we parse sequentially, sometimes there is nothing in the buffer or stack, thus error....        \n",
    "        pred = np.argmax(pred + 10000 * np.array(mb_l).astype('float32'), 1)\n",
    "        pred = [\"S\" if p == 2 else (\"LA\" if p == 0 else \"RA\") for p in pred]\n",
    "        \n",
    "        return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#a simple function to create ids.....\n",
    "def build_dict(keys, offset=0):\n",
    "    #keys = ['P_PREFIX:IN', 'P_PREFIX:DT', 'P_PREFIX:NNP', 'P_PREFIX:CD', so on...]\n",
    "    #offset is needed because this tok2id has something already inside....\n",
    "    count = Counter()\n",
    "    for key in keys:\n",
    "        count[key] += 1\n",
    "    \n",
    "    #most_common = [('P_PREFIX:NN', 70), ('P_PREFIX:IN', 57), ... , ('P_PREFIX:JJR', 1)]\n",
    "    #we use most_common in case we only want some maximum pos tags....\n",
    "    mc = count.most_common()\n",
    "    \n",
    "    #{'P_PREFIX:NN': 31, 'P_PREFIX:IN': 32, .., 'P_PREFIX:JJR': 62} \n",
    "    return {w[0]: index + offset for (index, w) in enumerate(mc)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ParserModel(nn.Module):\n",
    "\n",
    "    def __init__(self, embeddings, n_features=36,\n",
    "                 hidden_size=400, n_classes=3, dropout_prob=0.5):\n",
    "\n",
    "        super(ParserModel, self).__init__()\n",
    "        self.n_features   = n_features\n",
    "        self.n_classes    = n_classes\n",
    "        self.dropout_prob = dropout_prob\n",
    "        self.embed_size   = embeddings.shape[1]\n",
    "        self.hidden_size  = hidden_size\n",
    "        self.pretrained_embeddings = nn.Embedding(embeddings.shape[0], self.embed_size)\n",
    "        self.pretrained_embeddings.weight = nn.Parameter(torch.tensor(embeddings))\n",
    "\n",
    "        self.embed_to_hidden = nn.Linear(n_features * self.embed_size, hidden_size)\n",
    "        nn.init.xavier_uniform_(self.embed_to_hidden.weight, gain=1.)\n",
    "        self.dropout = nn.Dropout(p=dropout_prob)\n",
    "        self.hidden_to_logits = nn.Linear(hidden_size, n_classes)\n",
    "        nn.init.xavier_uniform_(self.hidden_to_logits.weight)\n",
    "\n",
    "    def embedding_lookup(self, t):\n",
    "        #t:  batch_size, n_features\n",
    "        batch_size = t.size()[0]\n",
    "                    \n",
    "        x = self.pretrained_embeddings(t)        \n",
    "        x = x.reshape(-1, self.n_features * self.embed_size)\n",
    "        # x = (1024, 48 * 50)\n",
    "\n",
    "        return x\n",
    "\n",
    "    def forward(self, t):\n",
    "        # t: (1024, 48)\n",
    "        embeddings = self.embedding_lookup(t)  \n",
    "    \n",
    "        # embeddings: (1024, 48 * 50)\n",
    "        hidden = self.embed_to_hidden(embeddings)\n",
    "    \n",
    "        # hidden: (1024, 200)\n",
    "        hidden_activations = F.relu(hidden)\n",
    "        # hidden_activations: (1024, 200)\n",
    "        thin_net = self.dropout(hidden_activations)\n",
    "        # thin_net: (1024, 200)\n",
    "        logits = self.hidden_to_logits(thin_net)\n",
    "        # logits: (1024, 3)\n",
    "\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the parser `__init__`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "INITIALIZING\n",
      "================================================================================\n",
      "1. Loading data...\n",
      "Example 1:  {'word': ['ms.', 'haag', 'plays', 'elianti', '.'], 'pos': ['NNP', 'NNP', 'VBZ', 'NNP', '.'], 'head': [2, 3, 0, 3, 3], 'dep': ['compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
      "took 2.62 seconds\n"
     ]
    }
   ],
   "source": [
    "print(80 * \"=\")\n",
    "print(\"INITIALIZING\")\n",
    "print(80 * \"=\")\n",
    "train_set, dev_set, test_set = load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2. Building parser...\n",
      "took 0.03 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"2. Building parser...\",)\n",
    "start = time.time()\n",
    "parser = Parser(train_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the `numericalize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word:  ['ms.', 'haag', 'plays', 'elianti', '.']\n",
      "Pos:   ['NNP', 'NNP', 'VBZ', 'NNP', '.']\n",
      "Head:  [2, 3, 0, 3, 3]\n"
     ]
    }
   ],
   "source": [
    "#before numericalize\n",
    "print(\"Word: \",  train_set[1]['word'])\n",
    "print(\"Pos:  \",  train_set[1]['pos'])\n",
    "print(\"Head: \",  train_set[1]['head'])\n",
    "# print(\"Dep:  \",  train_set[1]['dep'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3. Numericalizing data...\n",
      "took 0.05 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"3. Numericalizing data...\",)\n",
    "start = time.time()\n",
    "train_set = parser.numericalize(train_set)\n",
    "dev_set   = parser.numericalize(dev_set)\n",
    "test_set  = parser.numericalize(test_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5156, 304, 1364, 1002, 2144, 87]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (rootis added in front)\n",
    "train_set[1]['word']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<ROOT>\n",
      "ms.\n",
      "haag\n",
      "plays\n",
      "elianti\n",
      ".\n"
     ]
    }
   ],
   "source": [
    "for i in train_set[1]['word']:\n",
    "    print(parser.id2tok[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[84, 42, 42, 55, 42, 46]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (pos tag root is added in front)\n",
    "train_set[1]['pos']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<p>:<ROOT>\n",
      "<p>:NNP\n",
      "<p>:NNP\n",
      "<p>:VBZ\n",
      "<p>:NNP\n",
      "<p>:.\n"
     ]
    }
   ],
   "source": [
    "for i in train_set[1]['pos']:\n",
    "    print(parser.id2tok[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-1, 2, 3, 0, 3, 3]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (root has no head, so we assign -1)\n",
    "train_set[1]['head']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Embedding\n",
    "\n",
    "Word embedding length of 50.  In the paper, they applied a custom 50-embedding to all the words, pos, and dependencies.  For pos and dependencies, they claimed that there are some similarities that can be learned as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4. Loading pretrained embeddings...\n",
      "Embedding matrix shape (vocab, emb size):  (5157, 50)\n",
      "took 2.47 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"4. Loading pretrained embeddings...\",)\n",
    "config = Config()\n",
    "start = time.time()\n",
    "word_vectors = {}\n",
    "for line in open(config.embedding_file).readlines():\n",
    "    we = line.strip().split() #we = word embeddings - first column: word;  the rest is embedding\n",
    "    word_vectors[we[0]] = [float(x) for x in we[1:]] #{word: [list of 50 numbers], nextword: [another list], so on...}\n",
    "    \n",
    "#create an empty embedding matrix holding the embedding lookup table (vocab size, embed dim)\n",
    "#we use random.normal instead of zeros, to keep the embedding matrix arbitrary in case word vectors don't exist....\n",
    "embeddings_matrix = np.asarray(np.random.normal(0, 0.9, (parser.n_tokens, 50)), dtype='float32')\n",
    "\n",
    "for token in parser.tok2id:\n",
    "        i = parser.tok2id[token]\n",
    "        if token in word_vectors:\n",
    "            embeddings_matrix[i] = word_vectors[token]\n",
    "        elif token.lower() in word_vectors:\n",
    "            embeddings_matrix[i] = word_vectors[token.lower()]\n",
    "print(\"Embedding matrix shape (vocab, emb size): \", embeddings_matrix.shape)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5. Preprocessing training data...\n",
      "took 1.62 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"5. Preprocessing training data...\",)\n",
    "start = time.time()\n",
    "train_examples = parser.create_instances(train_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Minibatch loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_minibatches(data, minibatch_size, shuffle=True):\n",
    "    data_size = len(data[0])\n",
    "    indices = np.arange(data_size)\n",
    "    if shuffle:\n",
    "        np.random.shuffle(indices)\n",
    "    for minibatch_start in np.arange(0, data_size, minibatch_size):\n",
    "        minibatch_indices = indices[minibatch_start:minibatch_start + minibatch_size]\n",
    "        yield [_minibatch(d, minibatch_indices) for d in data]\n",
    "\n",
    "def _minibatch(data, minibatch_idx):\n",
    "    return data[minibatch_idx] if type(data) is np.ndarray else [data[i] for i in minibatch_idx]\n",
    "\n",
    "def minibatches(data, batch_size):\n",
    "    x = np.array([d[0] for d in data])\n",
    "    y = np.array([d[2] for d in data])\n",
    "    one_hot = np.zeros((y.size, 3))\n",
    "    one_hot[np.arange(y.size), y] = 1\n",
    "    return get_minibatches([x, one_hot], batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing your minibatch loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i, (train_x, train_y) in enumerate(minibatches(train_examples, 1024)):\n",
    "#     print(train_x.shape)  #batch size, features\n",
    "#     print(train_y.shape)        #one hot encoding of 3 actions - shift, la, ra"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(parser, train_data, dev_data, output_path, batch_size=1024, n_epochs=10, lr=0.0005):\n",
    "    \n",
    "    best_dev_UAS = 0\n",
    "    \n",
    "    optimizer = optim.Adam(parser.model.parameters(), lr=0.001)\n",
    "    loss_func = nn.CrossEntropyLoss()\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        print(\"Epoch {:} out of {:}\".format(epoch + 1, n_epochs))\n",
    "        dev_UAS = train_for_epoch(\n",
    "            parser, train_data, dev_data, optimizer, loss_func, batch_size)\n",
    "        if dev_UAS > best_dev_UAS:\n",
    "            best_dev_UAS = dev_UAS\n",
    "            print(\"New best dev UAS! Saving model.\")\n",
    "            torch.save(parser.model.state_dict(), output_path)\n",
    "        print(\"\")\n",
    "\n",
    "\n",
    "def train_for_epoch(parser, train_data, dev_data, optimizer, loss_func, batch_size):\n",
    "    \n",
    "    parser.model.train()  # Places model in \"train\" mode, i.e. apply dropout layer\n",
    "    n_minibatches = math.ceil(len(train_data) / batch_size)\n",
    "    loss_meter = AverageMeter()\n",
    "    \n",
    "    with tqdm(total=(n_minibatches)) as prog:\n",
    "        for i, (train_x, train_y) in enumerate(minibatches(train_data, batch_size)):\n",
    "            \n",
    "            #train_x:  batch_size, n_features\n",
    "            #train_y:  batch_size, target(=3)\n",
    "            optimizer.zero_grad() \n",
    "            loss = 0.\n",
    "            train_x = torch.from_numpy(train_x).long()  #long() for int so embedding works....\n",
    "            train_y = torch.from_numpy(train_y.nonzero()[1]).long()  #get the index with 1 because torch expects label to be single integer\n",
    "\n",
    "            # Forward pass: compute predicted logits.\n",
    "            logits = parser.model(train_x)\n",
    "            # Compute loss\n",
    "            # print(logits.shape, train_y.shape, )\n",
    "            loss = loss_func(logits, train_y)\n",
    "            # Compute gradients of the loss w.r.t model parameters.\n",
    "            loss.backward()\n",
    "            # Take step with optimizer.\n",
    "            optimizer.step()\n",
    "\n",
    "            prog.update(1)\n",
    "            loss_meter.update(loss.item())\n",
    "\n",
    "    print(\"Average Train Loss: {}\".format(loss_meter.avg))\n",
    "    print(\"Evaluating on dev set\",)\n",
    "    parser.model.eval()  # Places model in \"eval\" mode, i.e. don't apply dropout layer\n",
    "        \n",
    "    dev_UAS, _ = parser.parse(dev_data)\n",
    "    print(\"- dev UAS: {:.2f}\".format(dev_UAS * 100.0))\n",
    "    return dev_UAS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "TRAINING\n",
      "================================================================================\n",
      "Epoch 1 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.5276538319885731\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6605514.60it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 60.54\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 2 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.66it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.27448750132073957\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6867144.78it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 66.16\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 3 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.72it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.21784652241816124\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6828938.44it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 69.61\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 4 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.18455509934574366\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6916600.74it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 72.05\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 5 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.1582772887001435\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6702345.93it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 73.96\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 6 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.31it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.14128117387493452\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6888575.91it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 75.14\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 7 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.12391945455844204\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6846917.29it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 76.52\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 8 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.61it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.11013054832195242\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6857016.12it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 75.16\n",
      "\n",
      "Epoch 9 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 32.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.10083270414421956\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6880906.60it/s]       \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 76.03\n",
      "\n",
      "Epoch 10 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 33.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.08860842961197098\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6901154.39it/s]       "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 77.53\n",
      "New best dev UAS! Saving model.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#create directory if it does not exist for saving the weights...\n",
    "output_dir = \"output/{:%Y%m%d_%H%M%S}/\".format(datetime.now())\n",
    "output_path = output_dir + \"model.weights\"\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "    \n",
    "print(80 * \"=\")\n",
    "print(\"TRAINING\")\n",
    "print(80 * \"=\")\n",
    "    \n",
    "model = ParserModel(embeddings_matrix)\n",
    "parser.model = model\n",
    "\n",
    "start = time.time()\n",
    "train(parser, train_examples, dev_set, output_path,\n",
    "      batch_size=1024, n_epochs=10, lr=0.0005)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "TESTING\n",
      "================================================================================\n",
      "Restoring the best model weights found on the dev set\n",
      "Final evaluation on test set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 6868940.59it/s]       "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- test UAS: 79.23\n",
      "Done!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print(80 * \"=\")\n",
    "print(\"TESTING\")\n",
    "print(80 * \"=\")\n",
    "\n",
    "print(\"Restoring the best model weights found on the dev set\")\n",
    "parser.model.load_state_dict(torch.load(output_path))\n",
    "print(\"Final evaluation on test set\",)\n",
    "parser.model.eval()\n",
    "UAS, dependencies = parser.parse(test_set)\n",
    "print(\"- test UAS: {:.2f}\".format(UAS * 100.0))\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Try to delete only the 18 pos features and check UAS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "P_PREFIX = '<p>:' #indicating pos tags\n",
    "D_PREFIX = '<d>:' #indicating dependency tags\n",
    "UNK      = '<UNK>'\n",
    "NULL     = '<NULL>'\n",
    "ROOT     = '<ROOT>'\n",
    "\n",
    "class Parser(object):\n",
    "\n",
    "    def __init__(self, dataset):\n",
    "        \n",
    "        #set the root dep\n",
    "        self.root_dep = 'root'\n",
    "                \n",
    "        #get all the dep of the dataset as list, e.g., ['root', 'acl', 'nmod', 'nmod:npmod']\n",
    "        all_dep = [self.root_dep] + list(set([w for ex in dataset\n",
    "                                               for w in ex['dep']\n",
    "                                               if w != self.root_dep]))\n",
    "        \n",
    "        #1. put dep into tok2id lookup table, with D_PREFIX so we know it is dependency\n",
    "        #{'D_PREFIX:root': 0, 'D_PREFIX:acl': 1, 'D_PREFIX:nmod': 2, ..., 'D_PREFIX:<NULL>': 30}\n",
    "        tok2id = {D_PREFIX + l: i for (i, l) in enumerate(all_dep)}\n",
    "        tok2id[D_PREFIX + NULL] = self.D_NULL = len(tok2id)\n",
    "        \n",
    "        #we are using \"unlabeled\" where we do not label with the dependency\n",
    "        #thus the number of dependency relation is 1\n",
    "        trans = ['L', 'R', 'S']\n",
    "        self.n_deprel = 1\n",
    "        \n",
    "        #create a simple lookup table mapping action and id\n",
    "        #e.g., tran2id: {'L': 0, 'R': 1, 'S': 2}\n",
    "        #e.g., id2tran: {0: 'L', 1: 'R', 2: 'S'}\n",
    "        self.n_trans = len(trans)\n",
    "        self.tran2id = {t: i for (i, t) in enumerate(trans)}\n",
    "        self.id2tran = {i: t for (i, t) in enumerate(trans)}\n",
    "\n",
    "        #2. put pos tags into tok2id lookup table, with P_PREFIX so we know it is pos\n",
    "        tok2id.update(build_dict([P_PREFIX + w for ex in dataset for w in ex['pos']],\n",
    "                                  offset=len(tok2id)))\n",
    "        tok2id[P_PREFIX + UNK]  = self.P_UNK  = len(tok2id)  #also remember the pos tags of unknown\n",
    "        tok2id[P_PREFIX + NULL] = self.P_NULL = len(tok2id)\n",
    "        tok2id[P_PREFIX + ROOT] = self.P_ROOT = len(tok2id)\n",
    "        \n",
    "        #now tok2id:  {'P_PREFIX:root': 0, 'P_PREFIX:acl': 1, ..., 'P_PREFIX:JJR': 62, 'P_PREFIX:<UNK>': 63, 'P_PREFIX:<NULL>': 64, 'P_PREFIX:<ROOT>': 65}\n",
    "        \n",
    "        #3. put word into tok2id lookup table\n",
    "        tok2id.update(build_dict([w for ex in dataset for w in ex['word']],\n",
    "                                  offset=len(tok2id)))\n",
    "        tok2id[UNK]  = self.UNK = len(tok2id)\n",
    "        tok2id[NULL] = self.NULL = len(tok2id)\n",
    "        tok2id[ROOT] = self.ROOT = len(tok2id)\n",
    "        \n",
    "        #now tok2id: {'D_PREFIX:root': 0, 'D_PREFIX:acl': 1, 'D_PREFIX:nmod': 2, ..., 'memory': 340, 'mr.': 341, '<UNK>': 342, '<NULL>': 343, '<ROOT>': 344}\n",
    "        \n",
    "        #create id2tok\n",
    "        self.tok2id = tok2id\n",
    "        self.id2tok = {v: k for (k, v) in tok2id.items()}\n",
    "        \n",
    "        #why 18 normal features + 18 (pos) + 12 (dep)\n",
    "        #18 features - top 3 words on buffer, top 3 words on stack, \n",
    "        # the first and second left most/rightmost children of the top two words on the stack\n",
    "        # the leftmost of leftmost/rightmost of rightmost children of the top two words on the stack\n",
    "        #18 pos - basically corresponding POS tags\n",
    "        #12 dep - corresponding ARC, excluding 6 words on hte stack/buffer..\n",
    "        self.n_features = 18 + 12\n",
    "        self.n_tokens = len(tok2id)\n",
    "    \n",
    "    #function to turn train set with words to train set with id instead using tok2id\n",
    "    def numericalize(self, examples):\n",
    "        numer_examples = []\n",
    "        for ex in examples:\n",
    "            word = [self.ROOT] + [self.tok2id[w] if w in self.tok2id\n",
    "                                  else self.UNK for w in ex['word']]\n",
    "            head = [-1] + ex['head']\n",
    "            dep  = [-1] + [self.tok2id[D_PREFIX + w] if D_PREFIX + w in self.tok2id\n",
    "                            else -1 for w in ex['dep']]\n",
    "            numer_examples.append({'word': word,\n",
    "                                 'head': head, 'dep': dep})\n",
    "        return numer_examples\n",
    "\n",
    "    #function to extract features to form a feature embedding matrix\n",
    "    def extract_features(self, stack, buf, arcs, ex):\n",
    "             \n",
    "        #ex['word']:  [55, 32, 33, 34, 35, 30], i.e., ['root', 'ms.', 'haag', 'plays', 'elianti', '.']\n",
    "        #ex['pos']:   [29, 14, 14, 16, 14, 17], i.e., ['NNP', 'NNP', 'VBZ', 'NNP', '.']\n",
    "        #ex['head']:  [-1, 2, 3, 0, 3, 3]  or ['root', 'compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
    "        #ex['dep']:   [-1, 1, 2, 0, 6, 12] or ['compound', 'nsubj', 'root', 'dobj', 'punct']\n",
    "\n",
    "        #stack     :  [0]\n",
    "        #buffer    :  [1, 2, 3, 4, 5]\n",
    "        \n",
    "        if stack[0] == \"ROOT\":\n",
    "            stack[0] = 0  #start the stack with [ROOT]\n",
    "\n",
    "        #get leftmost children based on the dependency arcs\n",
    "        def get_lc(k):\n",
    "            return sorted([arc[1] for arc in arcs if arc[0] == k and arc[1] < k])\n",
    "\n",
    "        #get right most children based on the dependency arcs\n",
    "        def get_rc(k):\n",
    "            return sorted([arc[1] for arc in arcs if arc[0] == k and arc[1] > k],\n",
    "                          reverse=True)\n",
    "\n",
    "        p_features = [] #pos features (2a, 2b, 2c) - 18\n",
    "        d_features = [] #dep features (3b, 3c) - 12\n",
    "        \n",
    "        #last 3 things on the stack as features\n",
    "        #if the stack is less than 3, then we simply append NULL from the left\n",
    "        features = [self.NULL] * (3 - len(stack)) + [ex['word'][x] for x in stack[-3:]]\n",
    "        \n",
    "        # next 3 things on the buffer as features\n",
    "        #if the buffer is less than 3, simply append NULL\n",
    "        #the reason why NULL is appended on end because buffer is read left to right\n",
    "        features += [ex['word'][x] for x in buf[:3]] + [self.NULL] * (3 - len(buf))\n",
    "        \n",
    "        #get the leftmost and rightmost children of the top two words, thus we loop 2 times\n",
    "        for i in range(2):\n",
    "            if i < len(stack):\n",
    "                k = stack[-i-1] #-1, -2 last two in the stack\n",
    "                \n",
    "                #the first and second lefmost/rightmost children of the top two words (i=1, 2) on the stack\n",
    "                lc = get_lc(k)  \n",
    "                rc = get_rc(k)\n",
    "                \n",
    "                #the leftmost of leftmost/rightmost of rightmost children of the top two words on the stack:\n",
    "                llc = get_lc(lc[0]) if len(lc) > 0 else []\n",
    "                rrc = get_rc(rc[0]) if len(rc) > 0 else []\n",
    "\n",
    "                #(leftmost of first word on stack, rightmost of first word, \n",
    "                # leftmost of the second word on stack, rightmost of second, \n",
    "                # leftmost of leftmost, rightmost of rightmost\n",
    "                features.append(ex['word'][lc[0]] if len(lc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][rc[0]] if len(rc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][lc[1]] if len(lc) > 1 else self.NULL)\n",
    "                features.append(ex['word'][rc[1]] if len(rc) > 1 else self.NULL)\n",
    "                features.append(ex['word'][llc[0]] if len(llc) > 0 else self.NULL)\n",
    "                features.append(ex['word'][rrc[0]] if len(rrc) > 0 else self.NULL)\n",
    "            \n",
    "                #corresponding dep\n",
    "                d_features.append(ex['dep'][lc[0]] if len(lc) > 0 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][rc[0]] if len(rc) > 0 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][lc[1]] if len(lc) > 1 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][rc[1]] if len(rc) > 1 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][llc[0]] if len(llc) > 0 else self.D_NULL)\n",
    "                d_features.append(ex['dep'][rrc[0]] if len(rrc) > 0 else self.D_NULL)\n",
    "                \n",
    "            else:\n",
    "                #attach NULL when they don't exist\n",
    "                features += [self.NULL] * 6\n",
    "                p_features += [self.P_NULL] * 6\n",
    "                d_features += [self.D_NULL] * 6\n",
    "\n",
    "        features += d_features\n",
    "        assert len(features) == self.n_features  #assert they are 18 + 18 + 12\n",
    "        return features\n",
    "\n",
    "    #decide whether to shift, leftarc, or rightarc, based on gold parse trees\n",
    "    #this is needed to create training examples which contain samples and ground truth\n",
    "    def get_oracle(self, stack, buf, ex):\n",
    "\n",
    "        #leave if the stack is only 1, thus nothing to predict....\n",
    "        if len(stack) < 2:\n",
    "            return self.n_trans - 1\n",
    "\n",
    "        #predict based on the last two words on the stack\n",
    "        i0 = stack[-1]\n",
    "        i1 = stack[-2]\n",
    "\n",
    "        #get the head and dependency\n",
    "        h0 = ex['head'][i0]\n",
    "        h1 = ex['head'][i1]\n",
    "        d0 = ex['dep'][i0]\n",
    "        d1 = ex['dep'][i1]\n",
    "\n",
    "        #either shift, left arc or right arc\n",
    "        #\"Shift\" = 2; \"LA\" = 0; \"RA\" = 1\n",
    "        #if head of the second last word is the last word, then leftarc\n",
    "        if (i1 > 0) and (h1 == i0):\n",
    "            return 0\n",
    "        #if head of the last word is the second last word, then rightarc\n",
    "        #make sure nothing in the buffer has head with the last word on the stack\n",
    "        #otherwise, we lose the last word.....\n",
    "        elif (i1 >= 0) and (h0 == i1) and \\\n",
    "                (not any([x for x in buf if ex['head'][x] == i0])):\n",
    "            return 1\n",
    "        #otherwise shift, if something is left in buffer, otherwise, do nothing....\n",
    "        else:\n",
    "            return None if len(buf) == 0 else 2\n",
    "\n",
    "    #generate training examples\n",
    "    #from the training sentences and their gold parse trees \n",
    "    def create_instances(self, examples):\n",
    "        all_instances = []\n",
    "        \n",
    "        for i, ex in enumerate(examples):\n",
    "            #e.g., ex['word]: [344, 163, 99, 164, 165, 68]\n",
    "            n_words = len(ex['word']) - 1  #excluding the root\n",
    "\n",
    "            #arcs = {(head, tail, dependency label)}\n",
    "            stack = [0]\n",
    "            buf = [i + 1 for i in range(n_words)]  #[1, 2, 3, 4, 5]\n",
    "            arcs = []\n",
    "            instances = []\n",
    "            \n",
    "            #because that's the maximum number of shift, leftarcs, rightarcs you can have\n",
    "            #this will determine the sample size of each training example\n",
    "            #if given five words, we will get a sample of (10, 48) where 10 comes from 5 * 2, and 48 is n_features\n",
    "            #but this for loop can be break if there is nothing left....\n",
    "            for i in range(n_words * 2):\n",
    "\n",
    "                #get the gold transition based on the parse trees\n",
    "                #gold_t can be either shift(2), leftarc(0), or rightarc(1)\n",
    "                gold_t = self.get_oracle(stack, buf, ex)\n",
    "                \n",
    "                #if gold_t is None, no need to extract features.....\n",
    "                if gold_t is None:\n",
    "                    break\n",
    "                \n",
    "                #make sure when the model predicts, we inform the current state of stack and buffer, so\n",
    "                #the model is not allowed to make any illegal action, e.g., buffer is empty but trying to pop\n",
    "                legal_labels = self.legal_labels(stack, buf)                \n",
    "                assert legal_labels[gold_t] == 1\n",
    "               \n",
    "                #extract all the 48 features \n",
    "                features = self.extract_features(stack, buf, arcs, ex)\n",
    "                instances.append((features, legal_labels, gold_t))\n",
    "            \n",
    "                #shift \n",
    "                if gold_t == 2:\n",
    "                    stack.append(buf[0])\n",
    "                    buf = buf[1:]\n",
    "                #left arc \n",
    "                elif gold_t == 0:\n",
    "                    arcs.append((stack[-1], stack[-2], gold_t))\n",
    "                    stack = stack[:-2] + [stack[-1]]\n",
    "                #right arc\n",
    "                else:\n",
    "                    arcs.append((stack[-2], stack[-1], gold_t - self.n_deprel))\n",
    "                    stack = stack[:-1]\n",
    "            else:\n",
    "                all_instances += instances\n",
    "\n",
    "        return all_instances\n",
    "\n",
    "    #provide an one hot encoding of the labels\n",
    "    def legal_labels(self, stack, buf):\n",
    "        labels =  ([1] if len(stack) > 2  else [0]) * self.n_deprel   #left arc   But cannot ROOT <----He thus 3\n",
    "        labels += ([1] if len(stack) >= 2 else [0]) * self.n_deprel   #right arc  ROOT--->He\n",
    "        labels += [1] if len(buf) > 0 else [0]   #shift\n",
    "        return labels\n",
    "    \n",
    "    #a simple function to check punctuation POS tags\n",
    "    def punct(self, pos):\n",
    "        return pos in [\"''\", \",\", \".\", \":\", \"``\", \"-LRB-\", \"-RRB-\"]\n",
    "\n",
    "    def parse(self, dataset, eval_batch_size=5000):\n",
    "        sentences = []\n",
    "        sentence_id_to_idx = {}\n",
    "                \n",
    "        for i, example in enumerate(dataset):\n",
    "            \n",
    "            #example['word']=[188, 186, 186, ..., 59]\n",
    "            #n_words=37\n",
    "            #sentence=[1, 2, 3, 4, 5,.., 37]\n",
    "                        \n",
    "            n_words = len(example['word']) - 1\n",
    "            sentence = [j + 1 for j in range(n_words)]            \n",
    "            sentences.append(sentence)\n",
    "            \n",
    "            #mapping the object unique id to the i            \n",
    "            #The id is the object's memory address\n",
    "            sentence_id_to_idx[id(sentence)] = i\n",
    "            \n",
    "        model = ModelWrapper(self, dataset, sentence_id_to_idx)\n",
    "        dependencies = minibatch_parse(sentences, model, eval_batch_size)\n",
    "                \n",
    "        UAS = all_tokens = 0.0\n",
    "        with tqdm(total=len(dataset)) as prog:\n",
    "            for i, ex in enumerate(dataset):\n",
    "                head = [-1] * len(ex['word'])\n",
    "                for h, t, in dependencies[i]:\n",
    "                    head[t] = h\n",
    "                for pred_h, gold_h, gold_l in \\\n",
    "                        zip(head[1:], ex['head'][1:], ex['dep'][1:]):\n",
    "                        UAS += 1 if pred_h == gold_h else 0\n",
    "                        all_tokens += 1\n",
    "                prog.update(i + 1)\n",
    "        UAS /= all_tokens\n",
    "        return UAS, dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelWrapper(object):\n",
    "    def __init__(self, parser, dataset, sentence_id_to_idx):\n",
    "        self.parser = parser\n",
    "        self.dataset = dataset\n",
    "        self.sentence_id_to_idx = sentence_id_to_idx\n",
    "\n",
    "    def predict(self, partial_parses):\n",
    "        mb_x = [self.parser.extract_features(p.stack, p.buffer, p.dep,\n",
    "                                             self.dataset[self.sentence_id_to_idx[id(p.sentence)]])\n",
    "                for p in partial_parses]\n",
    "        mb_x = np.array(mb_x).astype('int32')\n",
    "        mb_x = torch.from_numpy(mb_x).long()\n",
    "        mb_l = [self.parser.legal_labels(p.stack, p.buffer) for p in partial_parses]\n",
    "\n",
    "        pred = self.parser.model(mb_x)\n",
    "        pred = pred.detach().numpy()\n",
    "        \n",
    "        #we need to multiply 10000 with legal labels, to force the model not to make any impossible prediction\n",
    "        #other, when we parse sequentially, sometimes there is nothing in the buffer or stack, thus error....        \n",
    "        pred = np.argmax(pred + 10000 * np.array(mb_l).astype('float32'), 1)\n",
    "        pred = [\"S\" if p == 2 else (\"LA\" if p == 0 else \"RA\") for p in pred]\n",
    "        \n",
    "        return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "#a simple function to create ids.....\n",
    "def build_dict(keys, offset=0):\n",
    "    #keys = ['P_PREFIX:IN', 'P_PREFIX:DT', 'P_PREFIX:NNP', 'P_PREFIX:CD', so on...]\n",
    "    #offset is needed because this tok2id has something already inside....\n",
    "    count = Counter()\n",
    "    for key in keys:\n",
    "        count[key] += 1\n",
    "    \n",
    "    #most_common = [('P_PREFIX:NN', 70), ('P_PREFIX:IN', 57), ... , ('P_PREFIX:JJR', 1)]\n",
    "    #we use most_common in case we only want some maximum pos tags....\n",
    "    mc = count.most_common()\n",
    "    \n",
    "    #{'P_PREFIX:NN': 31, 'P_PREFIX:IN': 32, .., 'P_PREFIX:JJR': 62} \n",
    "    return {w[0]: index + offset for (index, w) in enumerate(mc)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ParserModel(nn.Module):\n",
    "\n",
    "    def __init__(self, embeddings, n_features=30,\n",
    "                 hidden_size=400, n_classes=3, dropout_prob=0.5):\n",
    "\n",
    "        super(ParserModel, self).__init__()\n",
    "        self.n_features   = n_features\n",
    "        self.n_classes    = n_classes\n",
    "        self.dropout_prob = dropout_prob\n",
    "        self.embed_size   = embeddings.shape[1]\n",
    "        self.hidden_size  = hidden_size\n",
    "        self.pretrained_embeddings = nn.Embedding(embeddings.shape[0], self.embed_size)\n",
    "        self.pretrained_embeddings.weight = nn.Parameter(torch.tensor(embeddings))\n",
    "\n",
    "        self.embed_to_hidden = nn.Linear(n_features * self.embed_size, hidden_size)\n",
    "        nn.init.xavier_uniform_(self.embed_to_hidden.weight, gain=1.)\n",
    "        self.dropout = nn.Dropout(p=dropout_prob)\n",
    "        self.hidden_to_logits = nn.Linear(hidden_size, n_classes)\n",
    "        nn.init.xavier_uniform_(self.hidden_to_logits.weight)\n",
    "\n",
    "    def embedding_lookup(self, t):\n",
    "        #t:  batch_size, n_features\n",
    "        batch_size = t.size()[0]\n",
    "                    \n",
    "        x = self.pretrained_embeddings(t)        \n",
    "        x = x.reshape(-1, self.n_features * self.embed_size)\n",
    "        # x = (1024, 48 * 50)\n",
    "\n",
    "        return x\n",
    "\n",
    "    def forward(self, t):\n",
    "        # t: (1024, 48)\n",
    "        embeddings = self.embedding_lookup(t)  \n",
    "    \n",
    "        # embeddings: (1024, 48 * 50)\n",
    "        hidden = self.embed_to_hidden(embeddings)\n",
    "    \n",
    "        # hidden: (1024, 200)\n",
    "        hidden_activations = F.relu(hidden)\n",
    "        # hidden_activations: (1024, 200)\n",
    "        thin_net = self.dropout(hidden_activations)\n",
    "        # thin_net: (1024, 200)\n",
    "        logits = self.hidden_to_logits(thin_net)\n",
    "        # logits: (1024, 3)\n",
    "\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the parser `__init__`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "INITIALIZING\n",
      "================================================================================\n",
      "1. Loading data...\n",
      "Example 1:  {'word': ['ms.', 'haag', 'plays', 'elianti', '.'], 'pos': ['NNP', 'NNP', 'VBZ', 'NNP', '.'], 'head': [2, 3, 0, 3, 3], 'dep': ['compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
      "took 2.56 seconds\n"
     ]
    }
   ],
   "source": [
    "print(80 * \"=\")\n",
    "print(\"INITIALIZING\")\n",
    "print(80 * \"=\")\n",
    "train_set, dev_set, test_set = load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2. Building parser...\n",
      "took 0.03 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"2. Building parser...\",)\n",
    "start = time.time()\n",
    "parser = Parser(train_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the `numericalize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word:  ['ms.', 'haag', 'plays', 'elianti', '.']\n",
      "Dep:   ['compound', 'nsubj', 'root', 'dobj', 'punct']\n",
      "Head:  [2, 3, 0, 3, 3]\n"
     ]
    }
   ],
   "source": [
    "#before numericalize\n",
    "print(\"Word: \",  train_set[1]['word'])\n",
    "print(\"Dep:  \",  train_set[1]['dep'])\n",
    "print(\"Head: \",  train_set[1]['head'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3. Numericalizing data...\n",
      "took 0.05 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"3. Numericalizing data...\",)\n",
    "start = time.time()\n",
    "train_set = parser.numericalize(train_set)\n",
    "dev_set   = parser.numericalize(dev_set)\n",
    "test_set  = parser.numericalize(test_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5156, 304, 1364, 1002, 2144, 87]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (rootis added in front)\n",
    "train_set[1]['word']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-1, 2, 3, 0, 3, 3]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#after numericalize (root has no head, so we assign -1)\n",
    "train_set[1]['head']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Embedding\n",
    "\n",
    "Word embedding length of 50.  In the paper, they applied a custom 50-embedding to all the words, pos, and dependencies.  For pos and dependencies, they claimed that there are some similarities that can be learned as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4. Loading pretrained embeddings...\n",
      "Embedding matrix shape (vocab, emb size):  (5157, 50)\n",
      "took 2.46 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"4. Loading pretrained embeddings...\",)\n",
    "config = Config()\n",
    "start = time.time()\n",
    "word_vectors = {}\n",
    "for line in open(config.embedding_file).readlines():\n",
    "    we = line.strip().split() #we = word embeddings - first column: word;  the rest is embedding\n",
    "    word_vectors[we[0]] = [float(x) for x in we[1:]] #{word: [list of 50 numbers], nextword: [another list], so on...}\n",
    "    \n",
    "#create an empty embedding matrix holding the embedding lookup table (vocab size, embed dim)\n",
    "#we use random.normal instead of zeros, to keep the embedding matrix arbitrary in case word vectors don't exist....\n",
    "embeddings_matrix = np.asarray(np.random.normal(0, 0.9, (parser.n_tokens, 50)), dtype='float32')\n",
    "\n",
    "for token in parser.tok2id:\n",
    "        i = parser.tok2id[token]\n",
    "        if token in word_vectors:\n",
    "            embeddings_matrix[i] = word_vectors[token]\n",
    "        elif token.lower() in word_vectors:\n",
    "            embeddings_matrix[i] = word_vectors[token.lower()]\n",
    "print(\"Embedding matrix shape (vocab, emb size): \", embeddings_matrix.shape)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5. Preprocessing training data...\n",
      "took 1.42 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"5. Preprocessing training data...\",)\n",
    "start = time.time()\n",
    "train_examples = parser.create_instances(train_set)\n",
    "print(\"took {:.2f} seconds\".format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Minibatch loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_minibatches(data, minibatch_size, shuffle=True):\n",
    "    data_size = len(data[0])\n",
    "    indices = np.arange(data_size)\n",
    "    if shuffle:\n",
    "        np.random.shuffle(indices)\n",
    "    for minibatch_start in np.arange(0, data_size, minibatch_size):\n",
    "        minibatch_indices = indices[minibatch_start:minibatch_start + minibatch_size]\n",
    "        yield [_minibatch(d, minibatch_indices) for d in data]\n",
    "\n",
    "def _minibatch(data, minibatch_idx):\n",
    "    return data[minibatch_idx] if type(data) is np.ndarray else [data[i] for i in minibatch_idx]\n",
    "\n",
    "def minibatches(data, batch_size):\n",
    "    x = np.array([d[0] for d in data])\n",
    "    y = np.array([d[2] for d in data])\n",
    "    one_hot = np.zeros((y.size, 3))\n",
    "    one_hot[np.arange(y.size), y] = 1\n",
    "    return get_minibatches([x, one_hot], batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing your minibatch loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i, (train_x, train_y) in enumerate(minibatches(train_examples, 1024)):\n",
    "#     print(train_x.shape)  #batch size, features\n",
    "#     print(train_y.shape)        #one hot encoding of 3 actions - shift, la, ra"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(parser, train_data, dev_data, output_path, batch_size=1024, n_epochs=10, lr=0.0005):\n",
    "    \n",
    "    best_dev_UAS = 0\n",
    "    \n",
    "    optimizer = optim.Adam(parser.model.parameters(), lr=0.001)\n",
    "    loss_func = nn.CrossEntropyLoss()\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        print(\"Epoch {:} out of {:}\".format(epoch + 1, n_epochs))\n",
    "        dev_UAS = train_for_epoch(\n",
    "            parser, train_data, dev_data, optimizer, loss_func, batch_size)\n",
    "        if dev_UAS > best_dev_UAS:\n",
    "            best_dev_UAS = dev_UAS\n",
    "            print(\"New best dev UAS! Saving model.\")\n",
    "            torch.save(parser.model.state_dict(), output_path)\n",
    "        print(\"\")\n",
    "\n",
    "\n",
    "def train_for_epoch(parser, train_data, dev_data, optimizer, loss_func, batch_size):\n",
    "    \n",
    "    parser.model.train()  # Places model in \"train\" mode, i.e. apply dropout layer\n",
    "    n_minibatches = math.ceil(len(train_data) / batch_size)\n",
    "    loss_meter = AverageMeter()\n",
    "    \n",
    "    with tqdm(total=(n_minibatches)) as prog:\n",
    "        for i, (train_x, train_y) in enumerate(minibatches(train_data, batch_size)):\n",
    "            \n",
    "            #train_x:  batch_size, n_features\n",
    "            #train_y:  batch_size, target(=3)\n",
    "            optimizer.zero_grad() \n",
    "            loss = 0.\n",
    "            train_x = torch.from_numpy(train_x).long()  #long() for int so embedding works....\n",
    "            train_y = torch.from_numpy(train_y.nonzero()[1]).long()  #get the index with 1 because torch expects label to be single integer\n",
    "\n",
    "            # Forward pass: compute predicted logits.\n",
    "            logits = parser.model(train_x)\n",
    "            # Compute loss\n",
    "            # print(logits.shape, train_y.shape, )\n",
    "            loss = loss_func(logits, train_y)\n",
    "            # Compute gradients of the loss w.r.t model parameters.\n",
    "            loss.backward()\n",
    "            # Take step with optimizer.\n",
    "            optimizer.step()\n",
    "\n",
    "            prog.update(1)\n",
    "            loss_meter.update(loss.item())\n",
    "\n",
    "    print(\"Average Train Loss: {}\".format(loss_meter.avg))\n",
    "    print(\"Evaluating on dev set\",)\n",
    "    parser.model.eval()  # Places model in \"eval\" mode, i.e. don't apply dropout layer\n",
    "        \n",
    "    dev_UAS, _ = parser.parse(dev_data)\n",
    "    print(\"- dev UAS: {:.2f}\".format(dev_UAS * 100.0))\n",
    "    return dev_UAS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "TRAINING\n",
      "================================================================================\n",
      "Epoch 1 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 36.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.7487162550290426\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 19461955.91it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 44.30\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 2 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 37.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.35841394277910393\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 19556867.55it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 51.05\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 3 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 37.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.2881408945346872\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 20468987.96it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 54.15\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 4 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 36.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.24969061774512133\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 20462609.59it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 55.59\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 5 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 37.53it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.22014790773391724\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 18531698.04it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 58.09\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 6 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 36.87it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.19378843096395335\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 20237165.38it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 60.38\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 7 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 37.05it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.17362946520249048\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 19762125.27it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 60.80\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 8 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 37.49it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.1587189588074883\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 17601573.95it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 61.89\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 9 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 38.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.14229372081657252\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 20610325.08it/s]      \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 62.18\n",
      "New best dev UAS! Saving model.\n",
      "\n",
      "Epoch 10 out of 10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:01<00:00, 38.29it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Train Loss: 0.13082516503830752\n",
      "Evaluating on dev set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 20203698.79it/s]      "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- dev UAS: 61.06\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#create directory if it does not exist for saving the weights...\n",
    "output_dir = \"output/{:%Y%m%d_%H%M%S}/\".format(datetime.now())\n",
    "output_path = output_dir + \"model.weights\"\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "    \n",
    "print(80 * \"=\")\n",
    "print(\"TRAINING\")\n",
    "print(80 * \"=\")\n",
    "    \n",
    "model = ParserModel(embeddings_matrix)\n",
    "parser.model = model\n",
    "\n",
    "start = time.time()\n",
    "train(parser, train_examples, dev_set, output_path,\n",
    "      batch_size=1024, n_epochs=10, lr=0.0005)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "TESTING\n",
      "================================================================================\n",
      "Restoring the best model weights found on the dev set\n",
      "Final evaluation on test set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "125250it [00:00, 20559509.08it/s]      "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- test UAS: 62.97\n",
      "Done!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "print(80 * \"=\")\n",
    "print(\"TESTING\")\n",
    "print(80 * \"=\")\n",
    "\n",
    "print(\"Restoring the best model weights found on the dev set\")\n",
    "parser.model.load_state_dict(torch.load(output_path))\n",
    "print(\"Final evaluation on test set\",)\n",
    "parser.model.eval()\n",
    "UAS, dependencies = parser.parse(test_set)\n",
    "print(\"- test UAS: {:.2f}\".format(UAS * 100.0))\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "|  |  | Train UAS | Test UAS |\n",
    "|---|---|---|---|\n",
    "| 1 | Words, POS, DEP | 77 | 78.65 |\n",
    "| 2 | Words, DEP | 77.58 | 78.20 |\n",
    "| 3 | Words, POS | 58.51 | 59.79 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Do another comparison study testing the embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 glove embedding (smallest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define the config for easy management\n",
    "class Config(object):\n",
    "    data_path = ''\n",
    "    train_file = 'train.conll'\n",
    "    dev_file = 'dev.conll'\n",
    "    test_file = 'test.conll'\n",
    "    embedding_file = 'en-cw.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for reading the conll file\n",
    "def read_conll(filename, max_example=5):\n",
    "    examples = []\n",
    "    with open(filename) as f:\n",
    "        i = 0\n",
    "        word, pos, head, dep = [], [], [], []\n",
    "        for line in f.readlines():\n",
    "            i = i + 1\n",
    "            wa = line.strip().split('\\t') #wa = word annotations\n",
    "            \n",
    "            #if all 10 columns are there....\n",
    "            if len(wa) == 10:  \n",
    "                word.append(wa[1].lower())\n",
    "                pos.append(wa[4])\n",
    "                head.append(int(wa[6]))\n",
    "                dep.append(wa[7])\n",
    "            \n",
    "            #otherwise, it means the line is empty, thus we consider finish for one example\n",
    "            elif len(word) > 0:\n",
    "                examples.append({'word': word, 'pos': pos, 'head': head, 'dep': dep})\n",
    "                word, pos, head, dep = [], [], [], []\n",
    "                    \n",
    "        #last example\n",
    "        if len(word) > 0:\n",
    "            examples.append({'word': word, 'pos': pos, 'head': head, 'dep': dep})\n",
    "            \n",
    "    return examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the load function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "#sample_size = (train, dev, test)\n",
    "def load_data(sample_size=(1000, 500, 500)):\n",
    "    config = Config()\n",
    "\n",
    "    print(\"1. Loading data...\",)\n",
    "    start = time.time()\n",
    "    train_set = read_conll(os.path.join(config.data_path, config.train_file))\n",
    "    dev_set   = read_conll(os.path.join(config.data_path, config.dev_file))\n",
    "    test_set  = read_conll(os.path.join(config.data_path, config.test_file))\n",
    "\n",
    "    #for shorter preprocessing time.....just for education, ok guys?\n",
    "    train_set = train_set[:sample_size[0]]\n",
    "    dev_set   = dev_set[:sample_size[1]]\n",
    "    test_set  = test_set[:sample_size[2]]\n",
    "    print(\"Example 1: \", train_set[1]) #i choose one because it's short\n",
    "\n",
    "    print(\"took {:.2f} seconds\".format(time.time() - start))\n",
    "    \n",
    "    return train_set, dev_set, test_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the preprocessing data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "INITIALIZING\n",
      "================================================================================\n",
      "1. Loading data...\n",
      "Example 1:  {'word': ['ms.', 'haag', 'plays', 'elianti', '.'], 'pos': ['NNP', 'NNP', 'VBZ', 'NNP', '.'], 'head': [2, 3, 0, 3, 3], 'dep': ['compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
      "took 2.60 seconds\n"
     ]
    }
   ],
   "source": [
    "print(80 * \"=\")\n",
    "print(\"INITIALIZING\")\n",
    "print(80 * \"=\")\n",
    "train_set, dev_set, test_set = load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load tokeniser from SpaCy. I will use only small version.\n",
    "\n",
    "tokeniser = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define corpus //I will use data that posted in classroom one.\n",
    "\n",
    "corpus = []\n",
    "\n",
    "for _, w in enumerate(train_set):\n",
    "    corpus.append(w['word'])\n",
    "corpus_tokenised = []\n",
    "for i in range(len(corpus)):\n",
    "    corpus_tokenised.append([str(token) for token in np.array(corpus[i])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['himself', 'consumer', 'changes', 'mistakenly', 'soften']"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get word sequences and unique words\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "vocab = list(set(flatten(corpus_tokenised)))\n",
    "vocab[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "#numericalization\n",
    "word2index = {w: i for i, w in enumerate(vocab)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5069\n"
     ]
    }
   ],
   "source": [
    "#vocab size\n",
    "voc_size = len(vocab)\n",
    "print(voc_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "#append UNK\n",
    "vocab.append('<UNK>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2index['<UNK>'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "#just in case we need to use\n",
    "index2word = {v:k for k, v in word2index.items()} "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Build Co-occurence Matrix X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "X_i = Counter(flatten(corpus)) # X_i\n",
    "# X_i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('oct.', 'in'),\n",
       " ('oct.', 'an'),\n",
       " ('oct.', '19'),\n",
       " ('oct.', 'review'),\n",
       " ('19', 'an')]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make skip gram of one size window\n",
    "skip_grams = []\n",
    "\n",
    "for sent in corpus_tokenised:\n",
    "    #for each sent [\"apple\", \"banana\", \"fruit\"]\n",
    "    for i in range(2, len(sent) - 2): # start from 1 to second last but we use window size of 2, then first centre word should be at index 2\n",
    "        target = sent[i]\n",
    "        context = np.ravel([sent[i - 2:i],sent[i+1: i + 3]])  # window_size = 2\n",
    "        for w in context:\n",
    "            skip_grams.append((target, w))\n",
    "\n",
    "skip_grams[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_ik_skipgram = Counter(skip_grams) # Co-occurece in window size 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "#simply a normalized function...don't worry too much\n",
    "def weighting(w_i, w_j, X_ik):\n",
    "        \n",
    "    #check whether the co-occurrences exist between these two words\n",
    "    try:\n",
    "        x_ij = X_ik[(w_i, w_j)]\n",
    "    except:\n",
    "        x_ij = 1  #if does not exist, set it to 1\n",
    "                \n",
    "    x_max = 100 #100 # fixed in paper  #cannot exceed 100 counts\n",
    "    alpha = 0.75\n",
    "    \n",
    "    #if co-occurrence does not exceed 100, scale it based on some alpha\n",
    "    if x_ij < x_max:\n",
    "        result = (x_ij/x_max)**alpha  #scale it\n",
    "    else:\n",
    "        result = 1  #if is greater than max, set it to 1 maximum\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub data rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_data_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from itertools import combinations_with_replacement\n",
    "\n",
    "X_ik = {}  #for keeping the co-occurences\n",
    "weighting_dic = {} #scaling the percentage of sampling\n",
    "\n",
    "for bigram in combinations_with_replacement(vocab, 2):\n",
    "    if X_ik_skipgram.get(bigram) is not None:  #matches \n",
    "        co_occer = X_ik_skipgram[bigram]  #get the count from what we already counted\n",
    "        X_ik[bigram] = co_occer + 1 # + 1 for stability issue\n",
    "        X_ik[(bigram[1],bigram[0])] = co_occer+1   #count also for the opposite\n",
    "    else:\n",
    "        pass\n",
    "        \n",
    "    weighting_dic[bigram] = weighting(bigram[0], bigram[1], X_ik)\n",
    "    weighting_dic[(bigram[1], bigram[0])] = weighting(bigram[1], bigram[0], X_ik)\n",
    "\n",
    "print(f\"{X_ik=}\")\n",
    "print(f\"{weighting_dic=}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def random_batch(batch_size, word_sequence, skip_grams, X_ik, weighting_dic):\n",
    "    \n",
    "    #convert to id since our skip_grams is word, not yet id\n",
    "    skip_grams_id = [(word2index[skip_gram[0]], word2index[skip_gram[1]]) for skip_gram in skip_grams]\n",
    "    \n",
    "    random_inputs = []\n",
    "    random_labels = []\n",
    "    random_coocs  = []\n",
    "    random_weightings = []\n",
    "    random_index = np.random.choice(range(len(skip_grams_id)), batch_size, replace=False) #randomly pick without replacement\n",
    "        \n",
    "    for i in random_index:\n",
    "        random_inputs.append([skip_grams_id[i][0]])  # target, e.g., 2\n",
    "        random_labels.append([skip_grams_id[i][1]])  # context word, e.g., 3\n",
    "        \n",
    "        #get cooc\n",
    "        pair = skip_grams[i]\n",
    "        try:\n",
    "            cooc = X_ik[pair]\n",
    "        except:\n",
    "            cooc = 1\n",
    "        random_coocs.append([math.log(cooc)])\n",
    "        \n",
    "        #get weighting\n",
    "        weighting = weighting_dic[pair]\n",
    "        random_weightings.append([weighting])\n",
    "                    \n",
    "    return np.array(random_inputs), np.array(random_labels), np.array(random_coocs), np.array(random_weightings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  [[1925]\n",
      " [1525]]\n",
      "Target:  [[4244]\n",
      " [2888]]\n",
      "Cooc:  [[2.48490665]\n",
      " [0.69314718]]\n",
      "Weighting:  [[0.20388531]\n",
      " [0.05318296]]\n"
     ]
    }
   ],
   "source": [
    "#testing the method\n",
    "batch_size = 2 # mini-batch size\n",
    "input_batch, target_batch, cooc_batch, weighting_batch = random_batch(batch_size, corpus, skip_grams, X_ik, weighting_dic)\n",
    "\n",
    "print(\"Input: \", input_batch)\n",
    "print(\"Target: \", target_batch)\n",
    "print(\"Cooc: \", cooc_batch)\n",
    "print(\"Weighting: \", weighting_batch)\n",
    "\n",
    "#we will convert them to tensor during training, so don't worry..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GloVe(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size,embed_size):\n",
    "        super(GloVe,self).__init__()\n",
    "        self.embedding_v = nn.Embedding(vocab_size, embed_size) # center embedding\n",
    "        self.embedding_u = nn.Embedding(vocab_size, embed_size) # out embedding\n",
    "        \n",
    "        self.v_bias = nn.Embedding(vocab_size, 1)\n",
    "        self.u_bias = nn.Embedding(vocab_size, 1)\n",
    "        \n",
    "    def forward(self, center_words, target_words, coocs, weighting):\n",
    "        center_embeds = self.embedding_v(center_words) # [batch_size, 1, emb_size]\n",
    "        target_embeds = self.embedding_u(target_words) # [batch_size, 1, emb_size]\n",
    "        \n",
    "        center_bias = self.v_bias(center_words).squeeze(1)\n",
    "        target_bias = self.u_bias(target_words).squeeze(1)\n",
    "        \n",
    "        inner_product = target_embeds.bmm(center_embeds.transpose(1, 2)).squeeze(2)\n",
    "        #[batch_size, 1, emb_size] @ [batch_size, emb_size, 1] = [batch_size, 1, 1] = [batch_size, 1]\n",
    "        \n",
    "        #note that coocs already got log\n",
    "        loss = weighting*torch.pow(inner_product +center_bias + target_bias - coocs, 2)\n",
    "        \n",
    "        return torch.sum(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size     = 2 # mini-batch size\n",
    "embedding_size = 2 #so we can later plot\n",
    "modelGl          = GloVe(voc_size, embedding_size)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(modelGl.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 100 | cost: 0.565653 | time: 0m 0s\n",
      "Epoch: 200 | cost: 8.280432 | time: 0m 0s\n",
      "Epoch: 300 | cost: 0.527364 | time: 0m 0s\n",
      "Epoch: 400 | cost: 0.326449 | time: 0m 0s\n",
      "Epoch: 500 | cost: 0.062437 | time: 0m 0s\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# Training\n",
    "num_epochs = 500\n",
    "for epoch in range(num_epochs):\n",
    "    \n",
    "    start = time.time()\n",
    "    \n",
    "    input_batch, target_batch, cooc_batch, weighting_batch = random_batch(batch_size, corpus, skip_grams, X_ik, weighting_dic)\n",
    "    input_batch  = torch.LongTensor(input_batch)         #[batch_size, 1]\n",
    "    target_batch = torch.LongTensor(target_batch)        #[batch_size, 1]\n",
    "    cooc_batch   = torch.FloatTensor(cooc_batch)         #[batch_size, 1]\n",
    "    weighting_batch = torch.FloatTensor(weighting_batch) #[batch_size, 1]\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    loss = modelGl(input_batch, target_batch, cooc_batch, weighting_batch)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    end = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "\n",
    "    if (epoch + 1) % 100 == 0:\n",
    "        print(f\"Epoch: {epoch + 1} | cost: {loss:.6f} | time: {epoch_mins}m {epoch_secs}s\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['himself',\n",
       " 'consumer',\n",
       " 'changes',\n",
       " 'mistakenly',\n",
       " 'soften',\n",
       " 'pushed',\n",
       " '90',\n",
       " 'dollar',\n",
       " 'accused',\n",
       " 'inadvertence']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#list of vocabs\n",
    "vocab[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = vocab[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#numericalization\n",
    "id = word2index[word]\n",
    "id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0])"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_tensor = torch.LongTensor([id])\n",
    "id_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[-0.1113, -0.0486]], grad_fn=<EmbeddingBackward0>),\n",
       " tensor([[0.4702, 0.5522]], grad_fn=<EmbeddingBackward0>))"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get the embedding by averaging\n",
    "v_embed = modelGl.embedding_v(id_tensor)\n",
    "u_embed = modelGl.embedding_u(id_tensor)\n",
    "\n",
    "v_embed, u_embed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.2518, grad_fn=<SelectBackward0>)"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#average to get the word embedding\n",
    "word_embed = (v_embed + u_embed) / 2\n",
    "word_embed[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "#let's write a function to get embedding given a word\n",
    "def get_embed(word):\n",
    "    id_tensor = torch.LongTensor([word2index[word]])\n",
    "    v_embed = modelGl.embedding_v(id_tensor)\n",
    "    u_embed = modelGl.embedding_u(id_tensor) \n",
    "    word_embed = (v_embed + u_embed) / 2 \n",
    "    x, y = word_embed[0][0].item(), word_embed[0][1].item()\n",
    "\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAADFCAYAAACcom94AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA/iklEQVR4nO3deVzVxf748deACLihhguaifJVUHYEBRXEyKXcWjQ1TdFuZqWp3SzN65JpeZOflVaaLZJlrul1zbi4JJobIOKGO6aC5ZIICAo4vz8OnAsKCHHY9P18PHh4znzmzLzPUXmfz8x8PqO01gghhBCmYFbeAQghhHhwSFIRQghhMpJUhBBCmIwkFSGEECYjSUUIIYTJSFIRQghhMpJUhBDiAaSUClRKtc/1/GmlVOu/0U6wUuqz7MfTlFJvFVZfkooQQjyYAoH2uZ4/DeSbVJRSVUzVqUmSilLqW6XUn0qpwwUcD1RKJSmlYrJ/ppiiXyGEeNgopYYopWKVUgeVUt8rpXoppfYqpQ4opcKVUg2UUvbASGBc9u/cTkBvYHb2cwel1Hal1AdKqV+BMfm183fiM1V2CgU+AxYXUidCa92zOI3a2tpqe3v7EoQlhBAPjrS0NCwtLXFycqJKlSpkZma6AYPNzc1RSnHlyhXOnTt3VmtdTSm1AEjRWocAKKXWARu01quynwPU1lp3yn5eB/DVWmul1D+At4F/FjdGkyQVrfWO7MxoUvb29kRGRpq6WSGEqJTmzZvHpUuXmDlzprHs0KFD/POf/yQxMRFLS0uAjGI0uTzX40eB5UopO6AqcPbvxFiWcyp+2adrPyulnMuwXyGEeCBorXPOMIxGjx7NqFGjOHToEF9++SUU7/d6aq7H84DPtNauwCuA1d+JsaySSjTQVGvtjiHw/xRUUSk1QikVqZSKvHz5chmFJ4QQFV9QUBArVqzg6tWrAFy7do2kpCQaN24MwHfffZe7ejJQs5Dnd7MBLmY/Hvp3YzTZjH9htNY3cj3epJT6Qillq7W+kk/dhcBCAG9vb7mFshDioXEsYhsRyxaTfPUKNR+xxX/AEFr5dzYed3Z2ZtKkSXTq1Alzc3M8PT2ZNm0a/fr1o3Hjxvj6+uZubj2wSinVBxgNLAO+Ukq9AfTNp/tpwEql1EVgD9Ds77wHZapb32fPqWzQWrvkc6wh8Ef2BFBbYBWGM5dCO/f29tYypyIqsho1apCSkmKy9kJDQ+natSuNGjUyWZuicjgWsY2whZ+RefuWsaxKVUu6jhiVJ7Hcj1IqSmvtXRoxFoWplhQvBXYDjkqpC0qpl5RSI5VSI7Or9AUOK6UOAnOBAfdLKEI8jEJDQ0lISCjvMEQ5iFi2OE9CAci8fYuIZYUtqq14TLX6a+B9jn+GYcmxEA+s2bNns2LFCm7dusUzzzzDe++9B8DTTz/N+fPnSU9PZ8yYMYwYMYKsrCxeeuklIiMjUUoxfPhwmjRpQmRkJIMGDcLa2prdu3djbW1dzu9KlJXkq/fMBhRaXlGVyZyKEA+6sLAwTp48yb59+9Ba07t3b3bs2EFAQADffvstdevWJS0tDR8fH5577jni4+O5ePEihw8brhe+fv06tWvX5rPPPiMkJARv73IbvRDlpOYjtiRfuXdxUs1HbMshmr9PbtMihAmEhYURFhaGp6cnXl5exMXFcfLkSQDmzp2Lu7s7vr6+nD9/npMnT9K8eXPOnDnD6NGj2bx5M7Vq1SrndyDKm/+AIVSpapmnrEpVS/wHDCmniP4eOVMRwgS01kycOJFXXnklT/n27dsJDw9n9+7dVKtWjcDAQNLT06lTpw4HDx7kl19+4fPPP2fFihV8++235RS9qAhyJuMLW/1VGUhSEcIEunXrxuTJkxk0aBA1atTg4sWLWFhYkJSURJ06dahWrRpxcXHs2bMHgCtXrlC1alWee+45HBwcCA4OBqBmzZokJyeX4zsR5amVf+dKl0TuJklFiEIkrV/Pnx9/QmZiIlXs7Kg/biw2vXrdU69r164cO3YMPz8/wLDU+IcffqB79+4sWLAANzc3HB0djdcRXLx4kWHDhnHnzh0APvzwQwCCg4MZOXKkTNSLSstk16mUBrlORZSnpPXrSZw8BZ2ebixTVlbYvT8938QiREXwQFynIsSD6M+PP8mTUAB0ejp/fvxJ+QQkRCUgSUWIAmQmJharXAghSUWIAlWxsytWuRBCkooQBao/bizKKu/dv5WVFfXHjS2fgISoBGT1lxAFyJmML8rqLyGEgSQVIQph06uXJJFKKjg4mJ49e9K3b353eRelRYa/hBBCmIwkFSHEA2Hx4sW4ubnh7u7Oiy++CMCOHTto3749zZs3Z9WqVQCkpKQQFBSEl5cXrq6urF27FoD4+HhatWrFyy+/jLOzM127diUtLQ2A/fv34+bmhp+fH+PHj8fFxbBtVFZWFuPHj8fHxwc3N7ec7XxJTEwkICAADw8PXFxciIiIKOuPo/xorSvsT5s2bbQQQtzP4cOHdcuWLfXly5e11lpfvXpVDx06VPft21dnZWXpI0eOaAcHB6211hkZGTopKUlrrfXly5e1g4ODvnPnjj579qw2NzfXBw4c0Fpr3a9fP/39999rrbV2dnbWu3bt0lpr/c4772hnZ2ettdZffvmlfv/997XWWqenp+s2bdroM2fO6JCQED1jxgyttdaZmZn6xo0bZfNBaK2BSF2Ov7dlTkUIUelt3bqVvn37YmtruE183bp1AcNeNmZmZrRu3Zo//vgDMHyRfvfdd9mxYwdmZmZcvHjReKxZs2Z4eHgA0KZNG+Lj47l+/TrJycm0b98egBdeeIENGzYAhrtTx8bGGs+CkpKSOHnyJD4+PgwfPpyMjAyefvppY5sPAxn+EkJUelprlFL3lFtaWuapA7BkyRIuX75MVFQUMTExNGjQgPTsOyfkrm9ubk5mZqbxdfk5efIkt2/fJiMjg+DgYM6ePYu3tzfvv/8+WVlZrFixghdeeIHFiyvX7o0lIUlFCFHpBQUFsWLFCq5evQrAtWvXCqyblJRE/fr1sbCwYNu2bZw7d67QtuvUqUPNmjWxs7PjypUrLFu2jGPHjnH48GGuXLmCi4sLkZGRbNiwgbCwMN5//328vLw4efIkQ4YMwc7OjujoaJO+34pMhr+EEBXexjMb+TT6Uy6lXqJh9YaM8RpDj+Y9jMednZ2ZNGkSnTp1wtzcHE9PzwLbGjRoEL169cLb2xsPDw+cnJzu2/8333yDv78/3bt3p0uXLgAcO3aM7t2706BBA9q2bcsff/zBq6++ipmZGaNGjTIOeZ05c8Y4gf9QMMXEDPAt8CdwuIDjCpgLnAJiAa+itCsT9eVj0aJF+vXXX//br9+2bZvu0aPHfetVr169WO1OnTpVz549+++GJSqpDac3aO/vvbVLqIvxx/t7b73h9IZS67NPnz7ay8tLt27dWn/55Zc6OTlZN23aVF++fFl/+OGH2sLCQh89elQ3btxYd+nSRaempmpfX1/t6uqqra2ttdZaN23aVL/33nva3NxcL126VP/yyy/a19dXe3p66r59++rk5ORSiZ1ynqg31fBXKNC9kONPAi2yf0YA803UrxDiAfdp9KekZ+W9W3R6VjqfRn9aan1+++23REVFERkZydy5c1m2bBkJCQl07NiRiIgILCwsaNWqFQMHDmTv3r10794dd3d3zMzy/kq1srKiZs2aPPHEE8yYMYPw8HCio6Px9vZmzpw5pRZ/eTJJUtFa7wAKHsSEPsDi7ES6B6itlJK78pWx1NRUevTogbu7Oy4uLixfvpz9+/fTvn173N3dadu2rXHXwYSEBLp3706LFi14++23jW2EhYXh5+eHl5cX/fr1IyUlBYDNmzfj5OREx44dWb16tbH+tGnTCAkJMT53cXEhPj7+nthmz55tXOs/depUY/nMmTNxdHTkiSee4Pjx46b+SEQlcCn1UrHKTWHu3Lm4u7vj6+vL+fPncXFxoVGjRuzcuZONGzcaFwX06NEDf39/duzYQd26dbGxsaFWrVokZt/JunPnztSvX589e/Zw9OhROnTogIeHB999991953Iqq7KaU2kMnM/1/EJ2mdxDvAxt3ryZRo0asXHjRsAwYenp6cny5cvx8fHhxo0bxp0GY2JiOHDgAJaWljg6OjJ69Gisra2N37aqV6/Ov//9b+bMmcPbb7/Nyy+/zNatW/m///s/+vfvX6y4wsLCOHnyJPv27UNrTe/evdmxYwfVq1dn2bJlHDhwgMzMTLy8vGjTpo3JPxdRsTWs3pDE1Ht/VTSs3rBU+tu+fTvh4eHs3r2batWqERgYaFwddrfk5GTu3LnD77//zurVq/Hx8aFatWp89913AKxbt44+ffqgtaZLly4sXbq0VGKuSMpq9de9a/0g33V6SqkRSqlIpVTk5cuXSzmsh4urqyvh4eG88847RERE8Pvvv2NnZ4ePjw8AtWrVokoVw/eMoKAgbGxssLKyonXr1pw7d67Ab1txcXE0a9aMFi1aoJRi8ODBxYorLCyMsLAwPD098fLyIi4ujpMnTxIREcEzzzxDtWrVqFWrFr179zb5ZyIqvjFeY7Ayz3u3aCtzK8Z4jSmV/pKSkqhTpw7VqlUjevUu9uzazeWFsWQl3SY1Nu/vpPfff5/w8HB69OjBv//9b3bu3EmPHj3473//y8WLF9m+fTsTJkzA19eXXbt2cerUKQBu3rzJiRMnSiX+8lZWSeUC0CTX80eBhPwqaq0Xaq29tdbe9erVK9Wg7h6auVtwcLDxoqbAwEAq+9bGLVu2JCoqCldXVyZOnMiaNWvyXdsPBa/X79KlCzExMcTExHD06FG++eYbgALbqVKlinEfdiDfb3xaayZOnGhs99SpU7z00kuFtiseHj2a92Ba+2nYVbdDobCrbse09tPyrP4qttgV8LELTKtt+DN2hfFQ9+7dyczMxKVFa6b861942rU2HLijubHxDKkH/jTW3bdvH2PHjuX27dt89dVXeHp6UqNGDbZs2ULjxo1Zs2YNdevWpV69eoSGhjJw4EDc3Nzw9fUlLi7u78dfgZXV8Nc6YJRSahnQDkjSWj/QQ19ZWVmYm5uXdxh5JCQkULduXQYPHkyNGjVYuHAhCQkJ7N+/Hx8fH5KTk43DX/nx9fXl9ddf59SpU/zf//0fN2/e5MKFCzg5OXH27FlOnz6Ng4NDnlN8e3t749XH0dHRnD179p52u3XrxuTJkxk0aBA1atTg4sWLWFhYEBAQQHBwMBMmTCAzM5P169fzyiuvmP6DEaVu+/bthISEGP8tFJezmTMJ0xI4fPhwyYOJXQHr34AMw329SDpveA7g9jyWlpb8/PPPJM7aR9b1W8aX7X7VkHhu/BJvnEsE+Oijj/joo4/u6ebuucPHH3+c/fv3lzz+Cs4kSUUptRQIBGyVUheAqYAFgNZ6AbAJeArDkuKbwDBT9Pt3zJw5k8WLF9OkSRPq1atHmzZtiImJYeTIkdy8eRMHBwe+/fZb6tSpU2Abr776Kvv37yctLY2+ffvy3nvvAYZfoMOHDycsLIxRo0YxYMCAsnpbAPx06Rofnknk4q0MGltaMLG5Hc81rGs8fujQIcaPH4+ZmRkWFhbMnz8frTWjR48mLS0Na2trwsPDC2w/97etW7cM/9lmzJhBy5YtWbhwIT169MDW1paOHTsa//M/99xzLF68GA8PD3x8fGjZsuU97Xbt2pVjx47h5+cHQI0aNfjhhx/w8vKif//+eHh40LRpU/z9/U35cYmH1Zbp/0soOTLSDOVuzxuLcieU3AoqFwZKF3ILgvLm7e2tTTnkFBUVRXBwMHv37jVO/I4cOZLFixczb948OnXqxJQpU7hx4waffPJJnv0YAgMDCQkJwdvbm2vXrlG3bl2ysrIICgpi7ty5uLm5YW9vz2uvvZZntVRZ+enSNd46fp60O//7+7Q2U4Q4NsmTWIQorvj4eLp37067du04cOAALVu2ZPHixbRu3ZrIyEhsbW2JjIzkrbfeYvv27fz666+MGWOY71BKsWPHDqKiopg2bRq2trYcPnyYNm3a8MMPP6CUIioqijfffJOUlBRsbW0JDQ3Fzs6OqKgohg8fTrVq1ejYsSM///yzac5UptUm/yldBdOuG5/dfaaSw7y2JXYT2pY8jlKilIrSWnuXV/8P1W1a8pv4TU1N5fr163Tq1AmAoUOHsmPHjkLbWbFiBV5eXnh6enLkyBGOHj1qPFbclU+m8uGZxDwJBSDtjubDMw/0KKMoI8ePH2fEiBHExsZSq1YtvvjiiwLrhoSE8PnnnxMTE0NERIRxSPXAgQN88sknHD16lDNnzrBr1y4yMjIYPXo0q1atMiaRSZMmATBs2DDmzp3L7t27TftmbB4tUnmtbvYoi7y/IpWFGbW62Zs2ngfMQ5VUoOQTv2fPniUkJIQtW7YQGxtLjx498kw+V69evaQh/i0Xb2UUq1yI4mjSpAkdOnQAYPDgwezcubPAuh06dODNN99k7ty5XL9+3biisG3btjz66KOYmZnh4eFBfHw8x48f5/Dhw3Tp0gUPDw9mzJjBhQsXSEpKyvNlL2d/FJMImgIWd80dWlgbynOp7lmf2s+2wLy2YdGKeW1Laj/bguqe9U0XywPooUoqAQEBrFmzhrS0NJKTk1m/fj3Vq1enTp06xk10vv/+e+M/5PzcuHGD6tWrY2Njwx9//MHPP/9cVuEXqrGlRbHKhSiOu7+MKaXyrOzL/cVqwoQJfP3116SlpeVZ5VTQikJnZ2fjyr9Dhw4RFhZW4F2HTcLteeg1F2yaAMrwZ6+5eeZTclT3rI/dhLY8OssfuwltJaEUwQN3Q8nY2Fi2bNlCUlISNjY2BAUF4ebmBlDgxO93331nnKhv3rw5ixYtKrB9d3d3PD09cXZ2pnnz5sZvb+VtYnO7fOdUJjaXGxcUx/Xr1/nxxx957bXXSrxi6UHy+++/s3v3bvz8/Fi6dCkdO3YkOTmZqKgonnzySX766Sdj3dOnT+Pq6oqrqyu7d+8mLi6O2rVr59uuo6Mjly9fNradkZHBiRMncHZ2xsbGhp07d9KxY0eWLFli2jfk9ny+SUSU3AOVVGJjY1m/fj0ZGYYhn6SkJNavXw9gTCyTJk0yjtnmtmfPnnvKQkNDjY+3b9+eb3lu+d1+pKzkTMYXtvpL3N/169f54osveO2118o7lDJ1LGIbEcsWk3z1CjUfscV/wBBa+Xc2Hm/VqhXfffcdr7zyCi1atODVV1+lbdu2vPTSS3zwwQe0a9fOWPeTTz5h27ZtmJub07p1a5588skC50WqVq3KqlWreOONN0hKSiIzM5OxY8fi7OzMokWLjBP13bp1K/XPQJjGA7X66+OPPyYpKemechsbG8aNG2fK0MQDasCAAaxduxZHR0csLCyoXr16viuW7O3ti7zyqWbNmuX8rgp3LGIbYQs/I/P2/1Y6ValqSdcRo2jl35n4+Hh69uxpmpVXotTJ6i8Tyi+hFFYuxN1mzZqFg4MDMTExzJ49O98VS4UpaOVTRRaxbHGehAKQefsWEcsent0Khek8UEnFxsamWOVC3E9+K5YKU9DKp4os+eqVQsvt7e3lLEUU2QOVVIKCgrCwyLvaycLCgqCgoHKKSFR2+a1YAoq98qkiq/mIbbHKhShMxf8aVQw5k/EFrf4SAgpfIVizZk3jnjKFsbe3L/LKp6JsV1ue/AcMyXdOxX/AkHKMSlRWD1RSAUNikSQiCnK/FYKPPPIIHTp0wMXFBWtraxo0aJBvO1OnTi3yyqeKLmeVV2Grv4Qoqgdq9ZcQ9yMrBMWDTlZ/CVGGZIWgEKVLkop4qMgKQSFKlyQV8VCRFYJClK4HbqJeiMLICkEhSpckFfHQkRWCQpQeGf4SQghhMpJUhBBCmIxJkopSqrtS6rhS6pRSakI+xwOVUklKqZjsnyn5tSOEEKJyK/GcilLKHPgc6AJcAPYrpdZprY/eVTVCa92zpP0JIYSouExxptIWOKW1PqO1vg0sA/qYoF0hhBCVjCmSSmPgfK7nF7LL7uanlDqolPpZKeVsgn6FEEJUMKZYUqzyKbv7hmLRQFOtdYpS6ingP0CLfBtTagQwAuCxxx4zQXhCCCHKiinOVC4ATXI9fxRIyF1Ba31Da52S/XgTYKGUynezBq31Qq21t9bau169eiYITwgBMHz4cOrXr4+Li4uxbPLkybi5ueHh4UHXrl1JSEi453Xp6em0bdsWd3d3nJ2dmTp1almGLSoZUySV/UALpVQzpVRVYACwLncFpVRDpZTKftw2u9+rJuhbCFFEwcHBbN68OU/Z+PHjiY2NJSYmhp49ezJ9+vR7XmdpacnWrVs5ePAgMTExbN68mT179pRV2KKSKfHwl9Y6Uyk1CvgFMAe+1VofUUqNzD6+AOgLvKqUygTSgAG6It9zX4gHUEBAwD3bIdeqVcv4ODU1lezvfnkopahRowYAGRkZZGRk5FtPCDDRdSpa601a65Zaawet9czssgXZCQWt9Wdaa2ettbvW2ldr/Zsp+hVCFM+FCxc4depUnrJJkyZhY2PDwoUL8z1TAcjKysLDw4P69evTpUuXPBuTFUVgYCA5eyOtXLmSVq1a0bmzbAL2IJIr6oV4yM2cOZOkpCRGjBjBZ599lm8dc3NzYmJiuHDhAvv27ePw4cN/u79vvvmGL774gm3btv3tNkTFJUlFiIeM1pqXX34ZZ2dnunbtSlpaGsHBwdSuXZuffvoJe3t73n33Xfz8/PD29iY6Oppu3brh4ODAsmXLCAwMZMWKFQQEBODh4YGLiwsREREAhIWF4efnh5eXF/369SMlJSVP39OnT2fnzp2MHDmS8ePHl8fbF6VMkooQD5nbt2/z+uuvc+TIEczNzfnpp58AiIyMxMnJCYAmTZqwe/du/P39GTx4MF9//TV79uxh8uTJhIeHc/HiRbp160ZMTAwHDx7Ew8ODK1euMGPGDMLDw4mOjsbb25s5c+bk6XvKlCl4e3uzZMkSZs+eXebvXZQ+ufW9EA+IjWc28mn0p1xKvUTD6g0Z4zWGHs17GI8PHDiQLVu2ANCzZ0/ee+89zp8/zz//+U8yMjJo3rw5a9eupUOHDrRt25annnqKvn37cu7cOXr37k1WVhbXr1/H39+f7t27M3z4cDIyMnj66afx8PDg119/5ejRo3To0AEwJC8/P79y+SxE+ZGkUobat2/Pb7+VfI1CfHw8PXv2LNG4du62fvvtN1544YUStyXKz8YzG5n22zTSs9IBSExNZNpv0wCMiWXp0qX3/Nv566+/SElJMZY3bmy4GUbTpk3ZtGkToaGhNGrUiNWrVwNgb2/Pm2++ia2tLTt27GDjxo28+OKLjB8/njp16tClSxeWLl1axu9eVCQy/FWGTJFQTCkzM5P4+Hh+/PHH8g5FlNCn0Z8aE0qO9Kx0Po3+tFT6O3fuHPXr1+fll1/mpZdeIjo6Gl9fX3bt2mVcXXbz5k1OnDhRKv2LikuSShnKWeu/fft2AgMD6du3L05OTgwaNIicy3amT5+Oj48PLi4ujBgxwlgeFRWFu7s7fn5+fP7558Y227Vrx5EjR4zPAwMDiYqKIjU1leHDh+Pj44Onpydr164FIDQ0lH79+tGrVy+6du3KhAkTiIiIwMPDg48//pisrCzGjx+Pj48Pbm5ufPnll/eNef/+/bRv3x53d3fatm1LcnJyge2I0nEp9VK+5YkFlJfU9u3b8fDwwNPTk59++okxY8ZQr149QkNDGThwIG5ubvj6+hIXF1cq/YsKTGtdYX/atGmjHyTVq1fXWmu9bds2XatWLX3+/HmdlZWlfX19dUREhNZa66tXrxrrDx48WK9bt05rrbWrq6vevn271lrrt956Szs7O2uttZ4zZ46eMmWK1lrrhIQE3aJFC6211hMnTtTff/+91lrrv/76S7do0UKnpKToRYsW6caNGxv72bZtm+7Ro4exzy+//FK///77Wmut09PTdZs2bfSZM2cKjPnWrVu6WbNmet++fVprrZOSknRGRkaB7YjS0WVlF+0S6nLPj/NXHfWa6AvlHZ4oQ0CkLsff23KmUk7atm3Lo48+ipmZGR4eHsYrnbdt20a7du1wdXVl69atHDlyhKSkJK5fv06nTp0AePHFF43tPP/886xcuRKAFStW0K9fP8CwtHPWrFl4eHgQGBhIeno6v//+OwBdunShbt26+cYVFhbG4sWL8fDwoF27dly9epWTJ08WGPPx48exs7PDx8cHMFyhXaVKlULbEaY3xmsM3LHIU6bvWJD+Zzdm/3K8nKISDyOZqC8nlpaWxsfm5uZkZmaSnp7Oa6+9RmRkJE2aNGHatGmkp6ejtS7wthiNGzfmkUceITY2luXLlxuHmbTW/PTTTzg6Ouapv3fvXqpXr15gXFpr5s2bR7du3fKUb9++Pd+YC4qtoHZE6ejRvAdjlx2gar1fUBbX0Rm1uXW5G5k3PEkgrbzDEw8ROVOpQNLTDROttra2pKSksGrVKgBq166NjY0NO3fuBGDJkiV5XjdgwAA++ugjkpKScHV1BaBbt27MmzfPOO9x4MCBfPusWbMmycnJxufdunVj/vz5ZGRkAHDixAlSU1MLjNnJyYmEhAT2798PQHJyMpmZmcVuR5RcfbP2pJ6eQErcLFJPTyDzhicAjWpbl3Nk4mEiZyom9NOla3x4JpGLtzJobGnBxOZ2PNcw/2Gm/NSuXZuXX34ZV1dX7O3tjUNKAIsWLWL48OFUq1btnm//ffv2ZcyYMUyePNlYNnnyZMaOHYubmxtaa+zt7dmwYcM9fbq5uVGlShXc3d0JDg5mzJgxxMfH4+XlhdaaevXq8Z///KfAmM3NzVm+fDmjR48mLS0Na2trwsPD+cc//lGsdkTJje/myMTVh0jLyDKWWVuYM76bYyGvEsK0VM432YrI29tb59yErqL76dI13jp+nrQ7//s8rc0UIY5N7ptYFi9eTEhICEop3NzcmDFjBsOHD+fy5cvUq1ePRYsW8dhjjxEcHEytWrWIjIzk0qVLfPTRR/Tt25fExET69+/PjRs3yMzMZP78+fj7+1OjRg3jbTJWrVrFhg0bCA0NJTg4GGtra+Li4jh37hyLFi3iu+++Y/fu3bRr147Q0FDAML8ydepUbt26hYODA4sWLaJGjRrY29szfPhwwsLCGDVqFAMGDCi1z1UUz38OXGT2L8dJuJ5Go9rWjO/myNOe+W3EKh5USqkorbV3efUvZyom8uGZxDwJBSDtjubDM4mFJpUjR44wc+ZMdu3aha2tLdeuXWPo0KEMGTKEoUOH8u233/LGG28Yv+UnJiayc+dO4uLi6N27N3379uXHH3+kW7duTJo0iaysLG7evHnfeP/66y+2bt3KunXr6NWrF7t27eLrr7/Gx8eHmJgYHn30UeMtN6pXr86///1v5syZw5QpUwCwsrIyDseVteDgYHr27Enfvn3Lpf+K7GnPxpJERLmSpGIiF29lFKs8x9atW+nbty+2toaNMOvWrcvu3buNVzC/+OKLvP3228b6Tz/9NGZmZrRu3Zo//vgDAB8fn3tumXE/vXr1QimFq6srDRo0MM7FODs7Ex8fz4ULFwq95Ub//v3v24cQ4uEjE/Um0tjSoljlOQpb2ZUj9/HcK7Byhi4DAgLYsWMHjRs35sUXX2Tx4sX3vC5nEcDd7ZiZmeVp08zMzLiqq0uXLtjb22Nubo5SyriHRlpaGr169cLd3Z2goCAAUlJSGDZsGK6urri5uRlvUphzwScYhuCCg4MBw54aLi4uuLu7ExAQAFDgBZNaa0aNGkXr1q3p0aMHf/75Z6GflxCi/MiZiolMbG6X75zKxOZ2hb4uKCiIZ555hnHjxvHII49w7do12rdvz7Jly3jxxRdZsmQJHTt2LLSNc+fO0bhxY15++WVSU1OJjo5myJAhNGjQgGPHjuHo6MiaNWuoWbNmkd+Pr68vr7/+OqtXr8bb25urV6/i5+dHnz59uHr1Kps2baJNmzZcu3YNgPfffx8bGxsOHToEGIbXCjN9+nR++eUXGjduzPXr1wHDPhs2Njbs37+fW7du0aFDB7p27cqBAwc4fvw4hw4d4o8//qB169YMHz68yO9FCFF2JKmYSM68yT2rv/4Mh6XTIekC2DwKQVPA7Xnj65ydnZk0aRKdOnXC3NwcT09P5s6dy/Dhw5k9e7Zxor4w27dvZ/bs2VhYWFCjRg3jmcqsWbPo2bMnTZo0wcXF5Z69LQqTc8uN3r17GxOEmZkZCxcuxNLSkqZNmwIYL6IMDw9n2bJlxtfXqVOn0PY7dOhAcHAwzz//PM8++yxgWBgQGxtrXEqdlJTEyZMn2bFjBwMHDsTc3JxGjRrx+OOPF/l9CCHKliQVE3quYd28k/KxK2D9G5CRffFZ0nnDc8iTWIYOHcrQoUPztLV169Z72s9ZlZUjJ0nk93owLDXObzI7dzv29vZ57nac+5iZmRnNmzcnLCyMatWqERgYiLu7O88884xxDihHQcN4BQ3BLViwgL1797Jx40Y8PDyIiYkp8ILJTZs2yZ7oQlQSJplTUUp1V0odV0qdUkpNyOe4UkrNzT4eq5TyMkW/Fd6W6f9LKDky0gzllUBSUhJ16tShWrVqxMXFsWfPHm7dusWvv/7K2bNnAYzDX127ds2zFW3O2U3OENydO3dYs2aN8fjp06dp164d06dPx9bWlvPnzxd4wWRAQADLli0jKyuLxMRE2YZWiAqsxGcqSilz4HOgC3AB2K+UWqe1Ppqr2pNAi+yfdsD87D8fbEkXildexhIvreXM6RDSbyViZWlHc4e3sGvYx3i8e/fuLFiwADc3NxwdHfH19aVevXosXLiQZ599ljt37lC/fn3++9//8q9//YvXX38dFxcXzM3NmTp1Ks8++2yBQ3Djx4/n5MmTaK0JCgrC3d0dNze3PBdMVq1alZs3b2JpaUmLFi1o1qwZXl5exnugCSEqnhJf/KiU8gOmaa27ZT+fCKC1/jBXnS+B7VrrpdnPjwOBWuvEwtquTBc/5utjF8OQ191smsC4km+wVRKJl9YSFzeJO3f+dyZlZmaNk9PMPImlPI0cOZJ27doxbNgwQkNDiYyMzHM2JIS4V3lf/GiK4a/GQO7fnBeyy4pb58ETNAUs7rrvkoW1obycnTkdkiehANy5k8aZ0yGl2m9qaio9evTA3d0dFxcXli9fzpYtW/D09MTV1ZXhw4dz69Ytvv76a1asWMH06dMZOHAgU6ZMYfny5Xh4eLB8+fJC94t59tln6d69Oy1atMhzjY8QovSZYqI+vxnUu09/ilLHUFGpEcAIgMcee6xkkZW3nMn4LQWv/iov6bfyP0ksqNxUNm/eTKNGjdi4cSNgmLdxcXFhy5YttGzZkiFDhjB//nzGjh3Lzp07jVfO332m8u677/L444/z7bffcv36ddq2bcsTTzwBQExMDAcOHMDS0hJHR0dGjx5NkyZNSvV9CSEMTHGmcgHI/T/2USDhb9QBQGu9UGvtrbX2rlevngnCK2duzxuGuqZdN/xZARIKgJVl/tfPFFRuKq6uroSHh/POO+8QERFBfHw8zZo1o2XLloBhJduOHTvu205h+8UEBQVhY2ODlZUVrVu35ty5c6X6noQQ/2OKpLIfaKGUaqaUqgoMANbdVWcdMCR7FZgvkHS/+RRRupo7vIWZWd6hOTMza5o7vFWq/bZs2ZKoqChcXV2ZOHGicdiquHL2i4mJiSEmJobff/+dVq1aAfnvVSOEKBslTipa60xgFPALcAxYobU+opQaqZQamV1tE3AGOAV8BbxW0n5Fydg17IOT00ysLBsBCivLRmUySZ+QkEC1atUYPHgwb731Fr/99hvx8fGcOnUKgO+//z7f1V357ftSlP1ihBBlyyQXP2qtN2FIHLnLFuR6rIHXTdGXMB27hn1MnkRO7L3E7rWnSbl2ixp1LfHr40DLdg2Nxw8dOsT48eMxMzPDwsKC+fPnk5SURL9+/cjMzMTHx4eRI0fe027nzp2Nw10TJ04s8n4xwnTat2/Pb7/9Vmb9xcfH89tvv/HCCy+UWZ+i5GQ/FWEyJ/ZeYtuSODJv3zGWValqRudBTnkSi6gYtm/fTkhISLGScVxcHAMGDEApxapVq3BwcCjS6z755BNGjBhBtWrVAHjqqaf48ccfqV27dp59f3JkZmayc+fOYscnyn9JsdymRZjM7rWn8yQUgMzbd9i99rQklQdAjRo1+Ne//oW7uzvx8fFMnDiRmJgYnn32WVxdXfn0009JS0vjP//5Dw4ODgQHB2NlZcXhw4fZu3cv9erVY9CgQaSnp9OgQQP8/f2pUqUKWVmGnSpDQ0PZuHEj6enppKamcvPmTY4dO4aHhwdDhw5l3Lhx5fwJiKKQpCJMJuXarWKVi5KLj4+ne/fudOzYkT179uDu7s6wYcOYOnUqf/75J0uWLAFg7Nixxu2eFy1ahKNj3i2GU1NTGT16NIcOHSIzM5Np06bRp0/eodHMzEw++eQTsrKy+Ouvvzh//jybN2/mlVdeoW7duvTp0wcnJyfmzZvH119/jYODA+fOnWPcuHHs27eP4OBgFi5cSO/evVm5ciXx8fFcuXKF1q1bG+8LFxYWRvPmzcnKysLT0xNbW1s5U6lkZD8VYTI16loWq1yYxqlTpxgzZgyxsbHExcXx448/GoeOPvjgA5ycnNixYwcHDhxg+vTpvPvuu/e0MXPmTB5//HH279/Ptm3bGD9+PKmpqXnqVKlShZEjR9K3b186d+7M9evXWb16NW3btuWnn37C3NycP/74g/j4eFJTU6lTpw5z5sxh6tSpNGrUCG9vbz799FN27txp3GfHyckJpRQnTpzg8OHDNGzYkOjoaGJiYjhx4gRXr14tk89QmI6cqQiT8evjkO+cil+foo27i7+nWbNmeXbuDAoKMu7qGR8fT1JSEkOHDuXkyZMopYw37MwtLCyMdevWERJiuKNCznU/Ocu072ZpacmWLVuIiorixo0bDBs2DKUUfn5+ZGZmYm5uzmOPPXbP3aWVUhQ0j3v48GESExPx9PQE4PLlyzzyyCN/+3MR5UPOVITJtGzXkM6DnIxnJjXqWsokfRm4e+fO3Lt6ZmZmMnnyZDp37szhw4dZv379PbuAQsHX/QwbNgwPDw+eeuqpfF8zdOhQvL29+fHHHzl+/LhxZ08rKyvMzMxYuXIld+7cITMzk/j4eBwdHQkICDCeBZ04cQKttXE4zsvLyxjDunXr7rsvj6h45ExFmFTLdg0liZjQ/ZZoF0VSUhKNGxtutXf3njw5unXrxrgpH3Ld40USk9KpdfMC7w3rmWeDuNxbQ4PhzgV9+vQh584X165d49KlS3nqODo60qlTJy5fvsznn3+OlZUVr732GlOmTMHf3x8rKyssLS2xtLTExcWF77//npSUFGrUqMEjjzzCnTt3cHd3Jzg4WCbqKwlJKkJUUHcv0U65dottS+IAipVY3n77bYYOHcqcOXMK3DXTvddw5r/+BqlbXwI0f9g0YKKNYXfPpz0NCSklJYVp06bh4ODA/PnzAZgxYwYffvghw4cPx8LCgs8//5wNGzYYE1CHDh34+OOPmTdvHiEhISxZsoRt27Zha2tLREQEtra2xrohISE0adIEPz8/wJDEfvjhhyIvWxYVg1ynIkQF9d27u/JdOVejriVDP+hg0r46zNrKxetp95Q3rm3Nrgl/b/vm4OBg4w1BRdmR61SEEPkqyyXaCfkklMLKi6KgoTbxYJOJeiEqqLJcot2otnWxyoUoiCQVISoovz4OVKma979oaS3RHt/NEWsL8zxl1hbmjO/mWMArhMifDH8JUUHlTMaXdPVXUeRMxs/+5TgJ19NoVNua8d0cjeVCFJVM1AshxAOkvCfqZfhLCCGEyUhSEUIIYTKSVIQQQpiMJBUhhBAmU6LVX0qpusBywB6IB57XWv+VT714IBnIAjLLcxJJCCFE6SnpmcoEYIvWugWwJft5QTprrT0koQghKoO5c+fSqlUrBg0aVKT6gYGBFHe16pQpUwgPD/874VVYJb1OpQ8QmP34O2A78E4J2xRCiHL3xRdf8PPPP9OsWbNSaT8rK4vp06eXStvlqaRnKg201okA2X/WL6CeBsKUUlFKqREl7FMIIUrVyJEjOXPmDL1792bmzJkMHz4cHx8fPD09Wbt2LQBpaWkMGDAANzc3+vfvT1ra/+6TFhYWhp+fH15eXvTr14+UlBQA7O3tmT59Oh07dmTlypUEBwezatUqAPbv30/79u1xd3enbdu2JCcnk56ezrBhw3B1dcXT05Nt27aV/YdRTPdNKkqpcKXU4Xx++tzvtbl00Fp7AU8CryulAgrpb4RSKlIpFXn58uVidCGEEKaxYMECGjVqxLZt20hNTc13q+X58+dTrVo1YmNjmTRpElFRUQBcuXKFGTNmEB4eTnR0NN7e3syZM8fYtpWVFTt37mTAgAHGstu3b9O/f38+/fRTDh48SHh4ONbW1nz++ecAHDp0iKVLlzJ06NB8N1mrSO47/KW1fqKgY0qpP5RSdlrrRKWUHfBnAW0kZP/5p1JqDdAW2FFA3YXAQjBcUX//tyCEEKWnoK2Wd+zYwRtvvAGAm5sbbm5uAOzZs4ejR4/SoYNhe4Lbt28b94gB6N+//z19HD9+HDs7O3x8fACoVasWADt37mT06NEAODk50bRpU06cOGHsqyIq6ZzKOmAoMCv7z7V3V1BKVQfMtNbJ2Y+7Ag/eQKIQotILDAwkJCQEb29vLly4wNWrV41bLedseZybUuqeMq01Xbp0YenSpfn2Ub169XxfU1BblU1J51RmAV2UUieBLtnPUUo1Ukptyq7TANiplDoI7AM2aq03l7BfIYQokdQDf5I4ax8XJkSQOGsfqQfyHWihW7duzJs3z/gL/sCBAwAEBASwZMkSAA4fPkxsbCwAvr6+7Nq1i1OnTgFw8+ZNTpw4UWgsTk5OJCQksH//fgCSk5PJzMzM08eJEyf4/fff801uFUmJzlS01leBoHzKE4Cnsh+fAdxL0o8QQhTHRx99hJWVFW+88Qbjxo3j4MGDbN26lS1btrBo0SKeb9+b9z6Yzu3M2zSt3Zj/99QE7qzOICsl45623nzzTby8vPjmm2/QWtO6dWuio6N59dVXGTZsGG5ubnh4eNC2bVsA6tWrR2hoKAMHDuTWLcOGajNmzKBly5YFxlu1alWWL1/O6NGjSUtLw9ramvDwcF577TVGjhyJq6srVapUITQ0FEtL0++nY0pyl2IhxANnz549/L//9/9YuXIl/v7+3Lp1i127dvHBBx9gZWXFmgXLWPzMv6lW1Zov9izhdlYGYzsE8/yKscxd8SXe3t7Y29sTGRnJr7/+yubNm/nqq68ASEpKwsbGppzfYcHkLsVCCGFibdq0ISoqiuTkZCwtLfHz8yMyMpKIiAisra058cdZnlnyOt0WDWfV4c1cSLoEgM68c09brq6uhIeH88477xAREVGhE0pFIJt0CSEeOBYWFtjb27No0SLat2+Pm5sb27Zt4/Tp0zRr1oyAFm357MnJ97xOVbn3e3bLli2Jiopi06ZNTJw4ka5duzJlypSyeBuVkpypCCEeSAEBAYSEhBAQEIC/vz8LFizAw8MDX19fov48SnxyAgBpGemcuXYeZWGGeV2re9pJSEigWrVqDB48mLfeeovo6OiyfiuVipypCCEqncRLazlzOoT0W4lYWdrR3OEt7BrmvR7b39+fmTNn4ufnR/Xq1bGyssLf35969erx3ZLFjBn9T9KupaCz7vBO91dp8+wTmEdY3NPXoUOHGD9+PGZmZlhYWDB//vyyepuVkkzUCyEqlcRLa4mLm8SdO/+7LYqZmTVOTjPvSSwPI5moF0KIYjhzOiRPQgG4cyeNM6dDyikikZskFSFEpZJ+K7FY5aJsSVIRQlQqVpZ2xSoXZUuSihCiUmnu8BZmZtZ5yszMrGnu8FY5RSRyk9VfQohKJWcy/n6rv0T5kKQihKh07Br2kSRSQcnwlxBCCJORpCKEEMJkJKkIIYQwGUkqQgghTEaSihBCCJORpCKEEMJkJKkIIYQwmRIlFaVUP6XUEaXUHaVUgXfFVEp1V0odV0qdUkpNKEmfQgghKq6SnqkcBp4FdhRUQSllDnwOPAm0BgYqpVqXsF8hhBAVUImuqNdaHwNQShVWrS1wSmt9JrvuMqAPcLQkfQshhKh4ymJOpTFwPtfzC9llQgghHjD3PVNRSoUDDfM5NElrvbYIfeR3GlPgdpNKqRHACIDHHnusCM0LIYSoKO6bVLTWT5SwjwtAk1zPHwUSCulvIbAQDNsJl7BvIYQQZagshr/2Ay2UUs2UUlWBAcC6MuhXCCFEGSvpkuJnlFIXAD9go1Lql+zyRkqpTQBa60xgFPALcAxYobU+UrKwhSmtW7eOWbNmFXg8JiaGTZs23bed0NBQRo0aVeJ4TNWOEKLslXT11xpgTT7lCcBTuZ5vAu7/W0mUi969e9O7d+8Cj8fExBAZGclTTz1VYB0hhAC5ov6BFx8fj5OTE//4xz9wcXFh0KBBhIeH06FDB1q0aMG+ffvynBmsXLkSFxcX3N3dCQgI4Pbt20yZMoXly5fj4eHB8uXL2bdvH+3bt8fT05P27dtz/Pjxe/rduHEjfn5+XLlyhbCwMPz8/PDy8qJfv36kpKQAYG9vz9SpU/Hy8sLV1ZW4uLg8bSQnJ9OsWTMyMjIAuHHjBvb29sbnQoiKR5LKQ+DUqVOMGTOG2NhY4uLi+PHHH9m5cychISF88MEHeepOnz6dX375hYMHD7Ju3TqqVq3K9OnT6d+/PzExMfTv3x8nJyd27NjBgQMHmD59Ou+++26eNtasWcOsWbOMQ2YzZswgPDyc6OhovL29mTNnjrGura0t0dHRvPrqq4SEhORpp2bNmgQGBrJx40YAli1bxnPPPYeFhUVpfExCCBOQ7YQfAs2aNcPV1RUAZ2dngoKCUErh6upKfHx8nrodOnQgODiY559/nmeffTbf9pKSkhg6dCgnT55EKZXnzGHbtm1ERkYSFhZGrVq12LBhA0ePHqVDhw4A3L59Gz8/P2P9nD7atGnD6tWr7+nrH//4Bx999BFPP/00ixYt4quvvirRZyGEKF2SVB4ClpaWxsdmZmbG52ZmZmRmZuapu2DBAvbu3cvGjRvx8PAgJibmnvYmT55M586dWbNmDfHx8QQGBhqPNW/enDNnznDixAm8vb3RWtOlSxeWLl1aaGzm5ub3xAKGJBcfH8+vv/5KVlYWLi4uxX37QogyJMNfIo/Tp0/Trl07pk+fjq2tLefPn6dmzZokJycb6yQlJdG4seGmCKGhoXle37RpU1avXs2QIUM4cuQIvr6+7Nq1i1OnTgFw8+ZNTpw4UayYhgwZwsCBAxk2bFjJ3pwQotRJUqnkktav5+TjQRxr1ZqTjweRtH59idobP348rq6uuLi4EBAQgLu7O507d+bo0aPGifq3336biRMn0qFDB7Kysu5pw9HRkSVLltCvXz9u3LhBaGgoAwcOxM3NDV9f33sm5O9n0KBB/PXXXwwcOLBE700IUfqU1hX3onVvb28dGRlZ3mFUWEnr15M4eQo6Pd1YpqyssHt/Oja9epVjZKa1atUq1q5dy/fff1/eoQhR4SmlorTWBW5FUtpkTqUS+/PjT/IkFACdns6fH3/ywCSV0aNH8/PPPxfp4kshRPmTpFKJZSYmFqu8Mpo3b155hyCEKAaZU6nEqtjZFatcCCFKmySVSqz+uLEoK6s8ZcrKivrjxpZPQEKIh54Mf1ViOfMmf378CZmJiVSxs6P+uLEPzHyKEKLykaRSydn06iVJRAhRYcjwlxBCCJORpCKEEMJkJKkIIYQwmQp9Rb1S6jJwrhgvsQWulFI4piDxlUxFjq8ixwYSX0lVpviaaq3rlVcgFTqpFJdSKrI8b09wPxJfyVTk+CpybCDxlZTEV3Qy/CWEEMJkJKkIIYQwmQctqSws7wDuQ+IrmYocX0WODSS+kpL4iuiBmlMRQghRvh60MxUhhBDlqFInFaVUXaXUf5VSJ7P/rFNAvXFKqSNKqcNKqaVKKav86pVjfLWVUquUUnFKqWNKKb+KFF92XXOl1AGl1IayiK2o8SmlmiiltmV/bkeUUmNKOabuSqnjSqlTSqkJ+RxXSqm52cdjlVJepRnP34hvUHZcsUqp35RS7hUpvlz1fJRSWUqpvhUpNqVUoFIqJvvf2q9lFVtR4lNK2Sil1iulDmbHVz77b2utK+0P8BEwIfvxBODf+dRpDJwFrLOfrwCCK0p82ce+A/6R/bgqULsixZd9/E3gR2BDBfv7tQO8sh/XBE4ArUspHnPgNNA8++/p4N19AU8BPwMK8AX2luHnVZT42gN1sh8/WdHiy1VvK7AJ6FtRYgNqA0eBx7Kf169Inx3wbs7/EaAecA2oWlYx5vxU6jMVoA+GX8hk//l0AfWqANZKqSpANSCh9EMDihCfUqoWEAB8A6C1vq21vl5R4gNQSj0K9AC+LpuwjO4bn9Y6UWsdnf04GTiG4YtEaWgLnNJan9Fa3waWZcd4d8yLtcEeoLZSqqw2uLlvfFrr37TWf2U/3QM8WkaxFSm+bKOBn4A/K1hsLwCrtda/A2itK1p8GqiplFJADQxJJbMMYwQq+fAX0EBrnQiGXy5A/bsraK0vAiHA70AikKS1Dqso8WH45nEZWJQ9vPS1Uqp6BYoP4BPgbeBOGcWVo6jxAaCUsgc8gb2lFE9j4Hyu5xe4N4EVpU5pKW7fL2E4qyor941PKdUYeAZYUIZxQdE+u5ZAHaXUdqVUlFJqSJlFV7T4PgNaYfjSfAgYo7Uu6/+zFf/W90qpcKBhPocmFfH1dTBk9GbAdWClUmqw1vqHihAfhr8DL2C01nqvUupTDEM9kytCfEqpnsCfWusopVSgKWK6q/2Sfn457dTA8O12rNb6hiliy6+bfMruXj5ZlDqlpch9K6U6Y0gqHUs1oru6zafs7vg+Ad7RWmcZvnCXmaLEVgVoAwQB1sBupdQerfWJ0g6OosXXDYgBHgccgP8qpSJK8f9Dvip8UtFaP1HQMaXUH0opO611YvYQQ36no08AZ7XWl7NfsxrDuLJJkooJ4rsAXNBa53y7XoUhqZiECeLrAPRWSj0FWAG1lFI/aK0HV5D4UEpZYEgoS7TWq00RVwEuAE1yPX+Ue4dSi1KntBSpb6WUG4ahzCe11lfLKDYoWnzewLLshGILPKWUytRa/6cCxHYBuKK1TgVSlVI7AHcM83ilrSjxDQNmacOkyiml1FnACdhXBvEZVfbhr3XA0OzHQ4G1+dT5HfBVSlXLHmsMwjDuXiHi01pfAs4rpRyzi4IwTAaWhaLEN1Fr/ajW2h4YAGw1VUIxRXzZf6ffAMe01nNKOZ79QAulVDOlVFUMn8e6u+qsA4ZkrwLzxTDcmljKcRU5PqXUY8Bq4MUy+oZdrPi01s201vbZ/95WAa+VQUIpUmwY/v35K6WqKKWqAe0ou98lRYnvdwy/P1BKNQAcgTNlFN//lPXKAFP+AI8AW4CT2X/WzS5vBGzKVe89IA44DHwPWFaw+DyASCAW+A/Zq3MqSny56gdStqu/7hsfhuEbnf3ZxWT/PFWKMT2F4ZvpaWBSdtlIYGT2YwV8nn38EOBdVp9XEeP7Gvgr12cVWZHiu6tuKGW0+quosQHjMXzpO4xhqLXCfHbZ/y/Csv/dHQYGl2V8OT9yRb0QQgiTqezDX0IIISoQSSpCCCFMRpKKEEIIk5GkIoQQwmQkqQghhDAZSSpCCCFMRpKKEEIIk5GkIoQQwmT+P7eTueQk9N9EAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(6,3))\n",
    "for i, word in enumerate(vocab[:20]): #loop each unique vocab\n",
    "    x, y = get_embed(word)\n",
    "    plt.scatter(x, y)\n",
    "    plt.annotate(word, xy=(x, y), xytext=(5, 2), textcoords='offset points')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Skipgram\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "#delete dupliAthense word define as vocabs, also include unknow as <UNK>.\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "vocabs  = list(set(flatten(corpus_tokenised)))\n",
    "vocabs.append('<UNK>') # to let '<UNK>' become the last.\n",
    "word2index = {v: idx+1 for idx, v in enumerate(vocabs)} #plus one to shift value in word index to assign <UNK>'s value to 0 without dupliAthension with others.\n",
    "word2index['<UNK>'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'laughs'"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create index to word\n",
    "index2word = {v:k for k, v in word2index.items()}\n",
    "\n",
    "index2word[50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2. Prepare train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['oct.', 'in'],\n",
       " ['oct.', 'an'],\n",
       " ['oct.', '19'],\n",
       " ['oct.', 'review'],\n",
       " ['19', 'an'],\n",
       " ['19', 'oct.'],\n",
       " ['19', 'review'],\n",
       " ['19', 'of'],\n",
       " ['review', 'oct.'],\n",
       " ['review', '19'],\n",
       " ['review', 'of'],\n",
       " ['review', '``'],\n",
       " ['of', '19'],\n",
       " ['of', 'review'],\n",
       " ['of', '``'],\n",
       " ['of', 'the'],\n",
       " ['``', 'review'],\n",
       " ['``', 'of'],\n",
       " ['``', 'the'],\n",
       " ['``', 'misanthrope'],\n",
       " ['the', 'of'],\n",
       " ['the', '``'],\n",
       " ['the', 'misanthrope'],\n",
       " ['the', \"''\"],\n",
       " ['misanthrope', '``'],\n",
       " ['misanthrope', 'the'],\n",
       " ['misanthrope', \"''\"],\n",
       " ['misanthrope', 'at'],\n",
       " [\"''\", 'the'],\n",
       " [\"''\", 'misanthrope'],\n",
       " [\"''\", 'at'],\n",
       " [\"''\", 'chicago'],\n",
       " ['at', 'misanthrope'],\n",
       " ['at', \"''\"],\n",
       " ['at', 'chicago'],\n",
       " ['at', \"'s\"],\n",
       " ['chicago', \"''\"],\n",
       " ['chicago', 'at'],\n",
       " ['chicago', \"'s\"],\n",
       " ['chicago', 'goodman'],\n",
       " [\"'s\", 'at'],\n",
       " [\"'s\", 'chicago'],\n",
       " [\"'s\", 'goodman'],\n",
       " [\"'s\", 'theatre'],\n",
       " ['goodman', 'chicago'],\n",
       " ['goodman', \"'s\"],\n",
       " ['goodman', 'theatre'],\n",
       " ['goodman', '-lrb-'],\n",
       " ['theatre', \"'s\"],\n",
       " ['theatre', 'goodman']]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skipgrams = []\n",
    "\n",
    "#for each corpus\n",
    "for sent in corpus_tokenised:\n",
    "    #for each sent [\"apple\", \"banana\", \"fruit\"]\n",
    "    for i in range(2, len(sent) - 2): # start from 1 to second last but we use window size of 2, then first centre word should be at index 2\n",
    "        center_word = sent[i]\n",
    "        outside_words = np.ravel([sent[i - 2:i],sent[i+1: i + 3]])  # window_size = 2\n",
    "        for o in outside_words:\n",
    "            skipgrams.append([center_word, o])\n",
    "\n",
    "skipgrams[:50] #shows only sample 50 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(batch_size, corpus):\n",
    "    \n",
    "    skipgrams = []\n",
    "\n",
    "    for sent in corpus:\n",
    "        for i in range(2, len(sent) - 2): #start from 2 to third last for window size = 2\n",
    "            centre_word = word2index[sent[i]]\n",
    "            outside_words = []\n",
    "            for j in np.ravel([sent[i - 2:i],sent[i + 1: i + 3]]):\n",
    "                outside_words.append(word2index[j])  #window_size = 2\n",
    "            for o in outside_words:\n",
    "                skipgrams.append([centre_word, o])\n",
    "                \n",
    "    #only get a batch, not the entire list\n",
    "    random_index = np.random.choice(range(len(skipgrams)), batch_size, replace=False)\n",
    "    #appending some list of inputs and labels\n",
    "    random_inputs, random_labels = [], []   \n",
    "    for index in random_index:\n",
    "        random_inputs.append([skipgrams[index][0]])  #center words, this will be a shape of (1, ) --> (1, 1) for modeling\n",
    "        random_labels.append([skipgrams[index][1]])\n",
    "        \n",
    "    return np.array(random_inputs), np.array(random_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 1)\n",
      "label=array([[4023],\n",
      "       [4497],\n",
      "       [3591],\n",
      "       [  19],\n",
      "       [3500],\n",
      "       [1058],\n",
      "       [2827],\n",
      "       [4646],\n",
      "       [ 502],\n",
      "       [4310]])\n"
     ]
    }
   ],
   "source": [
    "input, label = random_batch(10, corpus_tokenised)\n",
    "\n",
    "print(f\"{input.shape}\")\n",
    "print(f\"{label=}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Unigram distribution\n",
    "\n",
    "$$P(w)=U(w)^{3/4}/Z$$\n",
    "\n",
    "Defining the probability of sampling negative words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "word_count = Counter(flatten(corpus_tokenised))\n",
    "# word_count # Showing word count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24358"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_total_words = sum([c for w, c in word_count.items()])\n",
    "num_total_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "z = 0.001  #scaling up low frequency terms\n",
    "unigram_table = []\n",
    "\n",
    "# create loop for the P(w) formula\n",
    "for v in vocabs:\n",
    "    uw = word_count[v]/num_total_words\n",
    "    uw_alpha = uw ** (3/4)\n",
    "    uw_alpha_dividebyz = int(uw_alpha / z)\n",
    "    unigram_table.extend([v] * uw_alpha_dividebyz)\n",
    "    \n",
    "# Counter(unigram_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Negative sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_sequence(seq, word2index):\n",
    "    idxs = list(map(lambda w: word2index[w] if word2index.get(w) is not None else word2index[\"<UNK>\"], seq))\n",
    "    return torch.LongTensor(idxs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def negative_sampling(targets, unigram_table, k):\n",
    "    \n",
    "    batch_size = targets.shape[0]\n",
    "    neg_samples = []\n",
    "    \n",
    "    for i in range(batch_size):\n",
    "        #randomly pick k negative words from unigram_table\n",
    "        target_index = targets[i].item()  #looping each of the batch....\n",
    "        nsample = []\n",
    "        while len(nsample) < k:\n",
    "            neg = random.choice(unigram_table)\n",
    "            #if this word == target, skip this word\n",
    "            if word2index[neg] == target_index:\n",
    "                continue\n",
    "            nsample.append(neg)\n",
    "        #append this word to some list\n",
    "        neg_samples.append(prepare_sequence(nsample, word2index).reshape(1, -1))  #tensor[], tensor[]\n",
    "    return torch.cat(neg_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[3250],\n",
       "        [3250]]),\n",
       " array([[502],\n",
       "        [ 96]]))"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 2\n",
    "input_batch, label_batch = random_batch(batch_size, corpus_tokenised)\n",
    "\n",
    "input_batch, label_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to Tensor\n",
    "\n",
    "input_batch = torch.LongTensor(input_batch)\n",
    "label_batch = torch.LongTensor(label_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_neg = 10  # I will try 10 as in the paper.\n",
    "neg_samples = negative_sampling(label_batch, unigram_table, num_neg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 10])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neg_samples.shape # shape is (batch_size, num_neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Model\n",
    "\n",
    "$$\\mathbf{J}_{\\text{neg-sample}}(\\mathbf{v}_c,o,\\mathbf{U})=-\\log(\\sigma(\\mathbf{u}_o^T\\mathbf{v}_c))-\\sum_{k=1}^K\\log(\\sigma(-\\mathbf{u}_k^T\\mathbf{v}_c))$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SkipgramNeg(nn.Module):\n",
    "    \n",
    "    def __init__(self, voc_size, emb_size):\n",
    "        super(SkipgramNeg, self).__init__()\n",
    "        self.embedding_centre_word  = nn.Embedding(voc_size, emb_size)\n",
    "        self.embedding_outside_word = nn.Embedding(voc_size, emb_size)\n",
    "        self.logsigmoid = nn.LogSigmoid()\n",
    "        \n",
    "    def forward(self, centre_words, outside_words, negative_words):\n",
    "        #center_words, outside_words: (batch_size, 1)\n",
    "        #negative_words:  (batch_size, k)\n",
    "        \n",
    "        centre_embed  = self.embedding_centre_word(centre_words)    #(batch_size, 1, emb_size)\n",
    "        outside_embed = self.embedding_outside_word(outside_words)  #(batch_size, 1, emb_size)\n",
    "        neg_embed     = self.embedding_outside_word(negative_words) #(batch_size, k, emb_size)\n",
    "        \n",
    "        uovc          =  outside_embed.bmm(centre_embed.transpose(1, 2)).squeeze(2)  #(batch_size, 1)\n",
    "        ukvc          = -neg_embed.bmm(centre_embed.transpose(1, 2)).squeeze(2)  #(batch_size, k)\n",
    "        ukvc_sum      =  torch.sum(ukvc, 1).view(-1, 1) #(batch_size, 1)\n",
    "        \n",
    "        loss = self.logsigmoid(uovc) + self.logsigmoid(ukvc_sum)  #(batch_size, 1) + (batch_size, 1)\n",
    "                \n",
    "        return -torch.mean(loss)  #scalar, loss should be scalar, to call backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "input, label = random_batch(batch_size, corpus_tokenised)\n",
    "input_tensor = torch.LongTensor(input)  \n",
    "label_tensor = torch.LongTensor(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb_size = 10 # I will try on 10.\n",
    "voc_size = len(vocabs)\n",
    "modelSkNeg = SkipgramNeg(voc_size, emb_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_tensor = negative_sampling(label_tensor, unigram_table, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 1]), torch.Size([2, 1]))"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_tensor.shape, label_tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.1989, grad_fn=<NegBackward0>)"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = modelSkNeg(input_tensor, label_tensor, neg_tensor)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "voc_size   = len(vocabs)\n",
    "batch_size = 2\n",
    "emb_size   = 10\n",
    "modelSkNeg      = SkipgramNeg(voc_size, emb_size)\n",
    "\n",
    "optimizer  = optim.Adam(modelSkNeg.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 100 | Loss: 0.357088 | Time: 0m 0s\n",
      "Epoch 200 | Loss: 1.945285 | Time: 0m 0s\n",
      "Epoch 300 | Loss: 2.073411 | Time: 0m 0s\n",
      "Epoch 400 | Loss: 0.889838 | Time: 0m 0s\n",
      "Epoch 500 | Loss: 4.167721 | Time: 0m 0s\n",
      "Process Time: 3m 6s\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 500\n",
    "startp = time.time()\n",
    "for epoch in range(num_epochs):\n",
    "\n",
    "    #get random batch\n",
    "    start = time.time()\n",
    "    input_batch, label_batch = random_batch(batch_size, corpus_tokenised)\n",
    "    input_batch = torch.LongTensor(input_batch)\n",
    "    label_batch = torch.LongTensor(label_batch)\n",
    "    neg_batch   = negative_sampling(label_batch, unigram_table, 5)    \n",
    "    \n",
    "    #loss = modelSkNeg\n",
    "    loss = modelSkNeg(input_batch, label_batch, neg_batch)\n",
    "    \n",
    "    end = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start, end)\n",
    "    #backpropagate\n",
    "    loss.backward()\n",
    "    \n",
    "    #update alpha\n",
    "    optimizer.step()\n",
    "    \n",
    "    #print epoch loss\n",
    "    if (epoch + 1) % 100 == 0:\n",
    "        print(f\"Epoch {epoch+1} | Loss: {loss:.6f} | Time: {epoch_mins}m {epoch_secs}s\")\n",
    "endp = time.time()\n",
    "mins, secs = epoch_time(startp, endp)\n",
    "print(f\"Process Time: {mins}m {secs}s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Plot the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_embed(word):\n",
    "    try:\n",
    "        index = word2index[word]\n",
    "    except:\n",
    "        index = word2index['<UNK>']\n",
    "    \n",
    "    word = torch.LongTensor([index])\n",
    "\n",
    "    center_embed  = modelSkNeg.embedding_centre_word(word)\n",
    "    outside_embed = modelSkNeg.embedding_outside_word(word)\n",
    "    \n",
    "    embed = (center_embed + outside_embed) / 2\n",
    "    \n",
    "    return  embed[0][0].item(), embed[0][1].item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(-0.447147399187088, 0.7554438710212708)\n"
     ]
    }
   ],
   "source": [
    "print(get_embed('Athens'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAADFCAYAAABO8mUgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA/YUlEQVR4nO3deVhV1f748fcCERANNSccrqgpKHA4iDiLKCnenMohNSuHsmuDX/P7aGleh2v6jcpfmVaaXceyHDM1rUgcUNIEBXEWB5zAnAJFQRnW7w/gJHJAkMMgfF7Pw+M5a++99lr71PmcvdektNYIIYQQD7Iq6QIIIYQonSRACCGEMEsChBBCCLMkQAghhDBLAoQQQgizJEAIIYQwSwKEEKJEKaX+Ryl1TCm1Ip/771BKtSrgOWYopZ5+tBKWXxVKugBCiHLvDeCfWuuzRZG5Uspaaz21KPIu6+QOQghRYpRSC4DGwEal1GSl1GKlVJhSKkIp1TdzH3ul1EqlVJRSahVgf9/x3ZVSe5RSB5RSa5RSlTPTY5RSU5VSu4GBSqmlSqkBmdt8lFK/K6UOKqX2KaWqKKXslFJLlFKHMs/dpfivRumjSvNI6ho1amhnZ+eSLoYQoggdOnSI5s2b8+eff2JnZ8eTTz5Jamoqx48fp3nz5ly7do2kpCScnZ25c+cOx44dw9XVFVtbW06fPs1TTz2FtbU1ly9fJj09nbp163Lo0CFq1qxJnTp1AIiJicHR0RFHR0eOHDlC48aNcXBwIC0tDSsrK65cuWI6R3JyMidPnsTd3R0rq8fvN/T+/fuvaa1rWiQzrXWp/fP29tZCiLKtYcOG+urVq9rb21u7ublpT09P7enpqRs0aKCPHj2q+/btq4ODg037e3l56bCwML1p0yb95JNPmvZv3ry5HjlypCnPmJgY0zHDhg3Ta9as0VFRUbp9+/Y5yvDss89mO0fHjh31wYMHi7DWRQcI1xb6DpY2CCFEqaC1Zt26dbi4uOTYppQyu3+3bt34/vvvzebn4OBg9pjc8hI5PX73T0KIMikgIIB58+aZvqwjIiIA8PX1ZcWKjA5Ohw8fJioqCoC2bdsSGhrKqVOnALhz5w4nT57M8xyurq7ExsYSFhYGwK1bt0hNTc12jpMnT3L+/Hmzgaq8kTsIIUSRuh1xhZu/xpAWfxfrqrY8EeCMg1etHPtNmTKFt99+G4PBgNYaZ2dnfvrpJ15//XVGjBiBwWDAaDTSunVrAGrWrMnSpUsZMmQId+/eBWDmzJk0a9Ys17JUrFiRVatWMWbMGJKSkrC3t2fr1q288cYbjB49Gg8PDypUqMDSpUuxtbUtmgvyGCnVjdStWrXS4eHhJV0MIcQjuh1xhfgfotEp6aY0ZWNF1X5NzQYJUXhKqf1a6wKNE8mNPGISQhSZm7/GZAsOADolnZu/xpRMgUSBSIAQQhSZtPi7BUoXpYsECCFEkbGuav45fm7ponSRACGEKDJPBDijbLJ/zSgbK54IcC6ZAokCkV5MQogik9UQnZ9eTKL0kQAhhChSDl61JCA8puQRkxBCCLMkQAghhDBLAoQQQgizJEAIIYQwSwKEEEIIsyRACCGEMEsChBBCCLMkQAghhDBLAoQQQgizLBIglFKLlVJXlFKHc9mulFJzlVKnlFJRSqmWljivEEKIomOpO4ilQI88tv8TaJr59xow30LnFUIIUUQsEiC01iHAjTx26Qss1xn2AlWVUk6WOLcQQoiiUVxtEPWAC/e9v5iZJoQQopQqrgChzKSZXQxbKfWaUipcKRV+9erVIi6WEEKI3BRXgLgINLjvfX0g1tyOWuuFWutWWutWNWvWLJbCCSGEyKm4AsRG4OXM3kxtgQStdVwxnVsIIcQjsMiCQUqp7wE/oIZS6iIwDbAB0FovALYAzwCngDvACEucVwghRNGxSIDQWg95yHYNvGmJcwkhhCgeMpJa5ODs7My1a9cAqFy5cgmXRghRUiRACItKS0sr6SIIISxEAkQ59+yzz+Lt7Y2bmxsLFy7Mdb8dO3bQq1cv0/u33nqLpUuXAhl3HDNmzKBjx46sWbOGoKAg2rVrR8uWLRk4cCCJiYlFXQ0hRBGwSBuEeHwtXryY6tWrk5SUhI+PD/3793+kfOzs7Ni9ezfXrl2jX79+bN26FQcHBz788EM++eQTpk6dauGSCyGKmtxBlHNz587F09OTtm3bcuHCBaKjox8pn0GDBgGwd+9ejh49SocOHTAajSxbtoxz585ZssiiDFm6dClvvfXWIx//4J1tbgraljZ9+nRmz579qMUqM+QOohzbsWMHW7duZc+ePVSqVAk/Pz+Sk5PN7luhQgXS09NN7x/cz8HBAQCtNd26deP7778vuoILIYqF3EGUYwkJCVSrVo1KlSpx/Phx9u7dm+u+DRs25OjRo9y9e5eEhASCg4PN7te2bVtCQ0M5deoUAHfu3OHkyZNFUv7SaOPGjQQGBua6PTIyki1btjw0n8L+srZ0PgV1+/ZtevbsiaenJ+7u7qxatYqwsDDat2+Pp6cnrVu35tatWwDExsbSo0cPmjZtyjvvvGPKI7e2rF9++QVXV1c6duzIDz/8YNr/wV/97u7uxMTE5Cjbxx9/jI+PDwaDgWnTppnSZ82ahYuLC08//TQnTpyw9CV5LMkdRFkWtRqCZ0DCRXCsD/5TwfC8aXOPHj1YsGABBoMBFxcX2rZtm2tWDRo04Pnnn8dgMNC0aVO8vLzM7lezZk2WLl3KkCFDuHv3LgAzZ86kWbNmlq1bKdWnTx/69OmT6/bIyEjCw8N55plnirFUxe+XX36hbt26bN68Gcj4MeLl5cWqVavw8fHh5s2b2NvbAxnXJCIiAltbW1xcXBgzZgz29vbMnDkzR1vWO++8w6hRo9i2bRtPPfWU6dFmfgUFBREdHc2+ffvQWtOnTx9CQkJwcHBg5cqVREREkJqaSsuWLfH29rb4dXncSIAoq6JWw6b/gZSkjPcJFzLegylI2Nra8vPPP+c49P5fXff3QProo4/46KOP8twfoGvXroSFhRWu/KVQTEwMPXr0oGPHjuzduxdPT09GjBjBtGnTuHLlCitWrODo0aOEh4fz+eefs2bNGv7zn/9gbW2No6MjW7duZerUqSQlJbF7924mTZpEo0aNePvtt0lKSsLe3p4lS5bg4uKS7bybN29m5syZbNq0iQMHDjBt2jTu3r1LkyZNWLJkCZUrV8bZ2Zlhw4axadMmUlJSWLNmDa6urqY8bt26hcFg4OTJk9jY2HDz5k0MBgPR0dHY2NhY/Fp5eHgwfvx43n33XXr16kXVqlVxcnLCx8cHgCeeeMK0r7+/P46OjgC0aNGCc+fOER8fb2rLArh37x7t2rXj+PHjNGrUiKZNmwLw4osv5tn77kFBQUEEBQWZfuAkJiYSHR3NrVu3eO6556hUqRJAnkG+PJFHTGVV8Iy/g0OWlKSMdPHITp06xdixY4mKiuL48eN899137N69m9mzZ/N///d/2fadMWMGv/76KwcPHmTjxo1UrFiRGTNmMGjQICIjIxk0aBCurq6EhIQQERHBjBkzeO+997LlsX79egIDA02PpbJ+VR84cIBWrVrxySefmPatUaMGBw4c4PXXX8/RwFqlShX8/PxMv+hXrlxJ//79iyQ4ADRr1oz9+/fj4eHBpEmTWL9+PUqZm9Q544dKFmtra1JTU01tWZGRkURGRnL06FEWLVoEkGs+D2sng4w2skmTJpnyPXXqFK+88kqe+ZZnEiDKqoSLBUsX+dKoUSM8PDywsrLCzc0Nf39/lFJ4eHjkuJPq0KEDw4cP5+uvv851AGFCQgIDBw7E3d2dcePGceTIEdO27du38+GHH7J582aqVav20B5i/fr1A8Db29vss/dXX32VJUuWALBkyRJGjCi6KdFiY2OpVKkSL774IuPHj2fv3r3Exsaa7ixv3bpFampqrsfn1pbl6urK2bNnOX36NEC2zhDOzs4cOHAAgAMHDnD27Nkc+QYEBLB48WLTnfGlS5e4cuUKvr6+rF+/nqSkJG7dusWmTZsscyEec/KIqaxyrJ/xWMlcunhk9//atbKyMr23srLK8YW3YMEC/vjjDzZv3ozRaCQyMjJHflOmTKFLly6sX7+emJgY/Pz8TNsaN27MmTNnOHnyJK1atXpoD7GssmT9Cn9Qhw4diImJYefOnaSlpeHu7l7Q6pusu3yDD87EceluCvVsbZjU2In+daqbth86dIgJEyZgZWWFjY0N8+fPR2vNmDFjTI/Ttm7dmmv+ebVlLVy4kJ49e1KjRg06duzI4cOHAejfvz/Lly/HaDTi4+Njtt2re/fuHDt2jHbt2gEZ3V+//fZbWrZsyaBBgzAajTRs2JBOnTo98rUpSyRAlFX+U7O3QQDY2Geki2Jx+vRp2rRpQ5s2bdi0aRMXLlygSpUqpt47kHEHUa9exuKKWSPTszRs2JDZs2fz3HPPsWbNGtq2bcubb77JqVOneOqpp7hz5w4XL14sUAeAl19+mSFDhjBlypRHrte6yzcYf+ICSekZa35dvJvC+BMZP0aygkRAQAABAQE5jn2wp9zw4cMZPny46f1PP/1kep1bW1aPHj04fvx4jnR7e3uCgoLMlvn+trSxY8cyduzYHPtMnjyZyZMnmz2+vJJHTGWV4XnoPRccGwAq49/ec7P1YhLZJWzaRHRXf441b0F0V38SCvmYYcKECXh4eODu7o6vry+enp506dKFo0ePYjQaWbVqFe+88w6TJk2iQ4cOZh9Dubi4sGLFCgYOHMjNmzdNv6oNBgNt27Y1+0WZl6FDh/LXX38xZEieEzDn6YMzcabgkCUpXfPBGVnipaxRGTNxl06tWrXS4eHhJV0MUQ4kbNpE3JSp6PsaNpWdHU7vz8Cxd+8SLJllrV27lg0bNvDNN988ch5O2yPNrhesgLguxkfOV1iGUmq/1rqVJfKSR0xCAFc+nZMtOADo5GSufDqnzASIMWPG8PPPP+droF5e6tnacPFuitl0UbbIIyYhgNQ4849Hckt/HM2bN49Tp04VetDipMZO2Ftl7xJqb6WY1NipUPmKvMXHx/Pll18C+Z+DqrAkQAgBVHAy/+WWW3p51r9OdWa7NKC+rQ0KqG9rw2yXBtl6MQnLuz9AFBeLtEEopXoAnwHWwH+11oEPbPcDNgBZHZN/0Fo/dMSWtEGI4lJe2iDE42vw4MFs2LABFxcXbGxscHBwoEaNGhw+fBhvb2++/fZblFIope4B9bTW15RSrYDZWms/pVRnMr6nATTgq7W+lesJsUAbhFLKGvgC6AZcBMKUUhu11kcf2HWX1rro74mEeARZQeDKp3NIjYujgpMTtca9LcFBlBqBgYEcPnyYyMhIduzYQd++fTly5Ah169alQ4cOhIaG0rFjx7yyGA+8qbUOVUpVBsxP3XwfSzxiag2c0lqf0VrfA1YCfS2QrxDFyrF3b5puC6b5saM03RYswaGAhg8fztq1a0u6GOVG69atqV+/PlZWVhiNRrOj5x8QCnyilPofoKrWOveh7JksESDqAfcP2b2Ymfagdkqpg0qpn5VSbhY4rxBClFvm5rDKpPn7u93OlJjx6P9VwB7Yq5T6ezbHXFgiQJib4erBho0DQEOttScwD/gx18yUek0pFa6UCr969aoFiidE2WNuLfFffvmFli1b4unpib+/P5AxgnjEiBF4eHhgMBhYt24dkH2FtbVr15pGM69ZswZ3d3c8PT3x9fUFIC0tjQkTJpjWUPjqq6+AjInv3nrrLVq0aEHPnj25cuVKcVW/XHpwFH4e7gFZc5Wb1hBWSjXRWh/SWn8IhAMPDRCWGAdxEWhw3/v6QOz9O2itb973eotS6kulVA2t9bUHM9NaLwQWQkYjtQXKJ0SZ8+Ba4n379mXUqFGEhITQqFEjbty4AcD777+Po6Mjhw4dAuCvv/7KM9+sGWjr1atHfHw8AIsWLcLR0ZGwsDDu3r1Lhw4d6N69OxEREZw4cYJDhw7x559/0qJFC0aOHFmk9S7roqKiCA4OJiEhAUdHR/z9/TEYDAA8+eSTdOjQAXd3d+zt7aldu3Zu2cQCnyml3gP+uC/9baVUFyANOArknOv/AZYIEGFAU6VUI+ASMBh44f4dlFJ1gD+11lop1ZqMO5frFji3EOXS3LlzWb9+PQAXLlxg4cKF+Pr60qhRIwCqV8/ocrp161ZWrlxpOq5atWp55ps1A+3zzz9vmh02KCiIqKgoU/tCQkIC0dHRhISEMGTIEKytralbty5du3a1eD3Lk6ioKNN6HpBxnbNmlc0KEt99953ZYz///PP73yaaG0mttR5T0DIVOkBorVOVUm8Bv5LRzXWx1vqIUmp05vYFwADgdaVUKpAEDNaleY4PIUoxc2uJe3p6ml0mU2ttdp2D+9PuXzfB3Ay0WmvmzZuXY/K9LVu2yBoKFhQcHGwKDllSUlIIDg42BYjiZpGBclrrLVrrZlrrJlrrWZlpCzKDA1rrz7XWblprT611W63175Y4rxBlUdzlDYSGdiJ421OEhnYi7vKGbNvNrSV+9+5ddu7caVoDIesRU/fu3bP9usx6xFS7dm2OHTtGenq66U4E/p6BdsaMGdSoUYMLFy4QEBDA/PnzTV9eJ0+e5Pbt2/j6+rJy5UrS0tKIi4tj+/btRXpdyrqEhIQCpRcHmYtJiFIk7vIGjh+fTHp6xjTtyXdjOX48YwpqpzoZvcfNrSVes2ZNFi5cSL9+/UhPT6dWrVr89ttv/Pvf/+bNN9/E3d0da2trpk2bRr9+/QgMDKRXr140aNAAd3d303TYEyZMIDo6Gq01/v7+eHp6YjAYiImJoWXLlmitqVmzJj/++CPPPfcc27Ztw8PDg2bNmtG5c+eSuWhlhKOjo9lgkLUca0mQ2VyFKEVCQzuRfDc2R7qdbV06dNhVAiUSxeXBNggAGxsbevfuXaBHTDKbqxBlVPJd85MD5pYuyo6sIJBbL6aSIAFCiFLEztYplzsImTSwPDAYDCUaEB4ks7kKUYo0bjIeKyv7bGlWVvY0bjK+hEokyjO5gxCiFMlqiD5zejbJd+Ows3WicZPxpnQhipMECCFKGac6fSUgiFJBHjEJIYQwSwKEEEIIsyRACCGEMEsChBBCCLMkQAghhDBLAoQQQgizJEAIIYQwSwKEEEIIsyRACCGEMEsChBBCCLMkQAhhITt27KBXr14FOub48eMYjUa8vLw4ffp0vo+bM2cOd+7cMb1/5plniI+PB6By5coFKoMQuZEAIUQJ+vHHH+nbty8RERE0adIkX8ekpaXlCBBbtmyhatWqRVRKUV5ZJEAopXoopU4opU4ppSaa2a6UUnMzt0cppVpa4rxCWFpMTAyurq68+uqruLu7M3ToULZu3UqHDh1o2rQp+/btY9++fbRv3x4vLy/at2/PiRMncuRz+/ZtRo4ciY+PD15eXmzYsCHHPlu2bGHOnDn897//pUuXLgB8++23tG7dGqPRyL/+9S/S0tKAjLuCqVOn0qZNG2bNmkVsbCxdunQxHefs7My1a9dynOPjjz/Gx8cHg8HAtGnTLHmpRHmgtS7UH2ANnAYaAxWBg0CLB/Z5BvgZUEBb4I/85O3t7a2FKE5nz57V1tbWOioqSqelpemWLVvqESNG6PT0dP3jjz/qvn376oSEBJ2SkqK11vq3337T/fr101prvX37dt2zZ0+ttdaTJk3S33zzjdZa67/++ks3bdpUJyYm5jjftGnT9Mcff6y11vro0aO6V69e+t69e1prrV9//XW9bNkyrbXWgF61apXpuIYNG+qrV6+afe/g4KC11vrXX3/Vo0aN0unp6TotLU337NlT79y503IXS5RKQLgu5Pd61p8lpvtuDZzSWp8BUEqtBPoCR+/bpy+wPLPwe5VSVZVSTlprWUdRlDqNGjXCw8MDADc3N/z9/VFK4eHhQUxMDAkJCQwbNozo6GiUUtnWEM4SFBTExo0bmT17NgDJycmcP3+e5s2b53re4OBg9u/fj4+PDwBJSUnUqlULAGtra/r371+gegQFBREUFISXlxcAiYmJREdH4+vrW6B8RPlliQBRD7hw3/uLQJt87FMPkAAhSh1bW1vTaysrK9N7KysrUlNTmTJlCl26dGH9+vXExMTg5+eXIw+tNevWrcPFxSVb+ogRI4iIiKBu3bps2bIlxzHDhg3jgw8+yJGfnZ0d1tbWBaqH1ppJkybxr3/9q0DHCZHFEm0QykyafoR9MnZU6jWlVLhSKvzq1auFLpwQ5owcOZJatWrh7u5uSpsyZQo9evTg9OnTdO/endjYnGtDa63ZsGEDn3zyCW5ubgwbNsxs/gEBAcybNy/rESsREREALFmyhMjIyBzBAcDf35+1a9dy5coVAG7cuMG5c+fM5l+lShVu3bqVZx0DAgJYvHgxiYmJAFy6dMmUtxD5YYkAcRFocN/7+sCD/2flZx8AtNYLtdattNatatasaYHiCfG3k39cZtl7oTheM/Bmr0DuJaWatk2YMIFffvmFJk2a0KtXL2bMmJHjeKUUa9eupWLFijg6OnLq1Cnu3r2bY78pU6aQkpKCwWDA3d2dKVOmPLRsLVq0YObMmXTv3h2DwUC3bt2IizN/k/3aa6/xz3/+09RIbU737t154YUXaNeuHR4eHgwYMOChQaWkTZ8+3fRYzpzhw4ezdu1aAPz8/AgPDy+uopVLKusXziNnoFQF4CTgD1wCwoAXtNZH7tunJ/AWGY3VbYC5WuvWD8u7VatWWv4DEJZy8o/LbF9xnNR76QBcv3WZr36ZzPYte2jWpk62fT/44APOnz/P/Pnzc83vzp07dOzYkfnz59OmzYNPVcWjmD59OpUrV2b8+PFmtw8fPpxevXoxYMAA/Pz8mD17Nq1atcpX3mlpaQV+TPc4Ukrt11rn76I8RKHvILTWqWR8+f8KHANWa62PKKVGK6VGZ+62BTgDnAK+Bt4o7HmFKKg9G06bgkMWrTPSs0yePJkGDRqwYsUKs3cQkPFFYzQaqVWrFt26dZPgUEizZs3CxcWFp59+2tRlODIykrZt22IwGHjuuef466+/8szj9ddfp1WrVri5uWXrzuvs7MyMGTPo2LEja9asKdJ6lEUWGQehtd6itW6mtW6itZ6VmbZAa70g87XWWr+Zud1Day23BaLYJd7I+SjowfRZs2Zx4cIFhg4dyueff252f2trayIjI7l48SL79u3j8OHDRVLe8mD//v2sXLmSiIgIfvjhB8LCwgB4+eWX+fDDD4mKisLDw4P//Oc/eeYza9YswsPDiYqKYufOnURFRZm22dnZsXv3bgYPHlykdSmLZCS1KDcqV7fNd/oLL7zAunXr8syvatWq+Pn58csvv1ikfOXRrl27eO6556hUqRJPPPEEffr04fbt28THx9O5c2cAhg0bRkhISJ75rF69mpYtW+Ll5cWRI0c4evTvXvaDBg0q0jqUZRIgRJm0+cxmuq/tjmGZge5ru7P5zGba9W1ChYrZ/5NXCtr1zZjiIjo62pS+ceNGXF1dc+R79epV05xHSUlJbN261ex+Iv+UMtfJMf/Onj3L7NmzCQ4OJioqip49e5KcnGza7uDgUNgillsSIESZs/nMZqb/Pp2423FoNHG345j++3Sia+6ny1BXKle3ZcnWmXyy4X+4knCRrv1bsWjRIiZOnIi7uzsGg4GgoCA+++wzAGJjY3nmmWcAiIuLo0uXLhgMBnx8fOjWrVuBJ+gTf/P19WX9+vUkJSVx69YtNm3ahIODA9WqVWPXrl0AfPPNN6a7CXNu3ryJg4MDjo6O/Pnnn/z888/FVfwyzxID5YQoVT478BnJacnZ0pLTkvnswGcEDQiiWZs6DPu/bTmOe+WVV8zmd/+gNoPBYBrTIB4uKiqK4OBgEhIScHR0xN/fH4PBYNresmVLBg0ahNFopGHDhnTq1AmAZcuWMXr0aO7cuUPjxo1ZsmRJrufw9PTEy8sLNzc3GjduTIcOHYq8XuVFobu5FiXp5ioehWGZAW1mHKZCETUsyswRoihERUWxadOmbFOR2NjY0Lt372xBQlhWqermKkRp0L59e9PrOg51zO6TW/qjiImJ4bvvvrNYfmVRcHBwjnmqUlJSCA4OLqESiYKSACHKhN9//930emzLsdhZ22Xbbmdtx9iWYy1yrtTU1DITIGJiYrJNN5Jl6tSpbN26tVB5JyQk5Jp+/yjoNWvW0Lx58zxHhVuCpRdSWrp0qdnpWMoSaYMQZULlypVJTExkx44dfDTtI+wq23E04igOXg7Ufqo2CTsSmJg+EdcfXWnSpAnDhw/Hzs6OI0eO8Oeff/LJJ5/Qq1cvkpOTef311wkPD6dChQp88skndOnShaVLl7J582aSk5O5ffs2d+7c4dixYxiNRoYNG8a4ceNK+hJYVG6DBAvC0dHRbJBwdHTM9n7RokV8+eWXRR4gLG3p0qW4u7tTt27dki5KkZE7CFHmHDx4kNX/Xc2NszewibShX9V+nDx4kldffZV58+aZ9ouJiWHnzp1s3ryZ0aNHk5yczBdffAHAoUOH+P777xk2bJipy+SePXtYtmwZ27ZtIzAwkE6dOhEZGfnYB4e0tDRGjRqFm5sb3bt3JykpKducR87Ozrz33nu0a9eOVq1aceDAAQICAmjSpAkLFiwAMnp3+fr6YjQacXd3Z9euXfj7+xMTE8OiRYv46quvWLNmDenp6fj7+5vOPWPGDHbv3s3o0aOZMGFCsdU5t4WUnn32Wby9vXFzc2PhwoWm6zN8+HDc3d3x8PDg008/Ze3atYSHhzN06FCMRiNJSUnFVvbiJHcQoszx8fHByckJgCZNmtC9e3cAPDw82L59u2m/559/HisrK5o2bUrjxo05fvw4u3fvZsyYMQC4urrSsGFDTp48CUC3bt2oXr16Mdem6EVHR/P999/z9ddf8/zzz5sdINigQQP27NnDuHHjGD58OKGhoSQnJ+Pm5sbo0aP57rvvCAgIYPLkyaSlpXHnzh3u3r3LoUOHePPNN0lOTiY8PJxr165la6CeOnUq27ZtK9CcSoUVFBREdHQ0+/btQ2tNnz59CAkJwdfXl8WLF1O9enWSkpLw8fGhf//+xMTEcOnSJdOI+fj4eKpWrcrnn39erOUuCRIgRJnzsPUcsjw4QEspRV69+srqgKtGjRphNBoB8Pb2JiYmJsc+ffr0ATKCbGJiIlWqVKFKlSrY2dkRHx+Pj48PI0eOJCUlhWeffRaj0cjOnTs5d+4cK1euBODevXvUrl27uKqVq7wWUpo7dy7r168H4MKFC0RHR+Pi4sKZM2cYM2YMPXv2NP3gKA/kEZMot7IeeZw+fZozZ87g4uKCr68vK1asAODkyZOcP38+x6I/kL/1GB4X9wdUa2vrbEH0wX3uD7hZ71NTU/H19SUkJIR69erx0ksvsXz5crTWdOvWjcjISCIjIzl69CiLFi165HJmrc9dWFkLKWWV69SpU7zyyivs2LGDrVu3smfPHg4ePIiXlxfJyclUq1aNgwcP4ufnxxdffMGrr75qkXI8DiRAiMfCjxGX6BC4jUYTN9MhcBs/RlwqdJ4uLi507tyZf/7znyxYsAA7OzveeOMN0tLS8PDwYNCgQSxdujTbF2IWg8FAhQoV8PT05NNPPy10WYrL8uXLMRgMeHp68tJLL3Hx4kViYmIwGAz4+/ubZk3dvXs3ixcvpn379ly6dImNGzcCGY9XfvjhB1NbQ1b7jIODA7Vq1WLUqFF4e3sTGBhI27Zt2bhxIy+88AJdunShUaNGfPvtt4wcOZKwsDCmT59uKtfevXtp164dLVu2ZODAgaZFjopiNtbcFlJKSEigWrVqVKpUiePHj7N3714Arl27Rnp6Ov379+f999/nwIEDQNn6kZArSy1uXRR/3t7e+ViiW5R16w9c1K7//lk3fPcn05/rv3/W6w9cfOQ8hw0bptesWWPBUpZO6w9c1O0/CNbO7/6kjW8v0nUbNtZXr17VWmt9/fp13bVrV12vXj2ttdaLFi3Sbm5uetq0abpJkya6bdu2Oi0tTTs5OWlnZ2ettdaDBg3Sbdq00VprnZqaqhs0aKCvXr2qbW1ttZubmzYajdrV1VX3799fa6119+7ddbVq1bS7u7tu2LChtre311FRUdrX11e7urrqiIgI3b59e+3l5aUTExO11loHBgbq//znP1prrRs2bKg//PDDfNU1fuNGfbJLV33Utbk+2aWrjt+4Mdt2BwcH0+s5c+Zod3d37e7urtu2batPnTqlk5OTdY8ePbSHh4ceMGCA7ty5s96+fbuOjIzUXl5e2tPTU3t6euotW7ZorbVeu3atbtasmfb09NR37tx5pM+nKADh2kLfwTKSWpR6HQK3cSk+Zy+RelXtCZ3Y9ZHyvH/hmbLqx4hLTPrhEEkpGY9mbu7fhFVSPP+d9/941qseADVq1CAuLg4bGxtSUlJwcnLi2rVrDB8+nG7dujF06FDg71/LISEhjBw5khdffNHU1gB/dzMGWLt2LT/99BNLly7Nls+ZM2cICAgwTYr48ssv069fPypUqMDw4cOpX78+kNFW0a5dOxYtWoSzszM7d+6kYcOGedY1YdMm4qZMRd83SZ+ys8Pp/Rk49u5tuYv6GJCR1KJciTUTHPJKz4+lS5eW6eAA8PGvJ0zBAQCtSUnPSM/N/Q339z9ay/ohaa6t4cHj7p9J9f58cmu/0A9pq8hP54Arn87JFhwAdHIyVz6d89BjRe4kQIhSr25V+wKliwwPBlC7hp7cOb6LC3F/AnDjxg3at29v6mW0YsUKOnbsmGee586dM7U1vPLKK6bn8bVr1+bYsWOkp6ebegHlV9u2bQkNDeXUqVNAxlKuWV2L8ys1l7W7c0sX+SMBQpR6EwJcsLfJvpawvY01EwJy9i4qKz777DPc3d1xc3Njzpw5QMYXerdu3WjatCndunXLuQxn1Gr41B2mV4VP3RlWeV+2zRVrNsSx3SCurXoPT09P/vd//5e5c+eyZMkSDAYD33zzjWmK89zs2LEDo9GIl5cX69atY+zYjOlLAgMD6dWrF127djWNQcmvmjVrsnTpUoYMGYLBYKBt27YcP368QHlUyOWcuaXn5f5pQJydnbl27VqB8ygrCtUGoZSqDqwCnIEY4HmtdY7FY5VSMcAtIA1Ize/zMWmDEFl+jLjEx7+eIDY+ibpV7ZkQ4GJ6jl7WHD58mMGDB7Nv3z4qVqxIjx49mD9/Pl9//TXVq1dn4sSJBAYG8tdff/Hhhx9mHBS1Gjb9D6T8fdeQam3HxJRXWXvv74kM7W2secn5Dis+mYKNjQ3z58/nr7/+Mq138biyZBuEn5+faQCcs7Mz4eHh1KhRw9JFLjKlqQ1iIhCstW4KBGe+z00XrbXRUgUX5cuzXvUIndiVs4E9CZ3YtcwGB4Bjx47Rtm1bKlWqRIUKFejcuTPr169nw4YNDBs2DMhYhvPHH3/8+6DgGdmCA0CFtGRmOKyjXlV7FBmN+h/08+B8WBDjx48nMjKSEydOmNa6eJw59u6N0/szqFC3LihFhbp1WW30ZNnZswCMGzeOrl0zOjQEBwfz4osvEhQUZLZrrTm3b9+mZ8+eeHp64u7uzqpVq4qlXiWtsAGiL7As8/Uy4NlC5idEuefu7k5ISAjXr1/nzp07bNmyhQsXLvDnn3+aHt84OTlx5coV0zG3r16g53d38FyQiPuXiaw6nELwmVQ6zInm5oqxdLmyjm3jOnBt/8+sXr2aGTNmMGTIEKZOncqqVaswGo2sWrWK27dvM3LkSHx8fPDy8mLDhg1ARqN+v3796NGjB02bNuWdd94pkWuTF8fevWm6LZjmx47SdFsw3UePNq1KFx4eTmJiIikpKezevRsPDw9mzpzJ1q1bOXDgAK1ateKTTz7JNe9ffvmFunXrcvDgQQ4fPkyPHj2Kq1olqrBTbdTWWscBaK3jlFK1ctlPA0FKKQ18pbVeWMjzClFmNW/enHfffZdu3bpRuXJlPD09qVAh7/9Vf4l1pG7lJDa/kDGldUKyxn1+IsFvNKHZB4d4+eWXmT9/Pm+//Ta7d+82dfFdunQp4eHhfP755wC89957dO3alcWLFxMfH0/r1q15+umnAYiMjCQiIgJbW1tcXFwYM2YMDRo0KNqLUQje3t7s37+fW7duYWtrS8uWLQkPD2fXrl306dOHo0ePmlafy+pamxsPDw/Gjx/Pu+++S69evUwr35V1D72DUEptVUodNvPXtwDn6aC1bgn8E3hTKeWbx/leU0qFK6XCr169WoBTCFF2ZPUQCgkJoXr16jRt2pTatWsTl9krJy4ujlq1/v495tHvf9l6No13f0tm17lUYuLTaVStAs2GzAIyHkmFhIQ89LxBQUEEBgZiNBrx8/MjOTmZ8+fPA+Dv74+joyN2dna0aNGCc+fOFUHNLcfGxgZnZ2eWLFlC+/bt6dSpE9u3b+f06dM0atSoQNOANGvWjP379+Ph4cGkSZMsMh364+ChAUJr/bTW2t3M3wbgT6WUE0Dmv1dyySM2898rwHqgdR7nW6i1bqW1blWzZs1HqZMQj72sx0fnz5/nhx9+YMiQIfTp04dlyzKe6C5btoy+ff/+jdasz9vs3/g1Hs41mRR8lw0xleDJJmB4vkDn1Vqzbt060xfn+fPnad68OZC/OZtKG19fX2bPno2vry+dOnViwYIFGI3GAnetjY2NpVKlSrz44ouMHz/e1L23rCtsG8RGYFjm62HAhgd3UEo5KKWqZL0GugOHC3leIR5btyOuEBe4j4sTdxEXuI/bETl/V/Xv358WLVrQu3dvvvjiC6pVq8bEiRP57bffaNq0Kb/99hsTJ/7dJyQ2NpZKPi/w4oqLjJ+7lt9pSczV26YvwG+++YbOnTvnOM+D8wkFBAQwb94808C4iIgIS1ffouIubyA0tBPB254iNLQTcZezfwV16tSJuLg42rVrR+3atbGzs6NTp04F7lp76NAhWrdujdFoZNasWfz73/8u6qqVCoVtgwgEViulXgHOAwMBlFJ1gf9qrZ8BagPrM0daVgC+01r/UsjzCvFYuh1xhfgfotEp6QCkxd8l/oeMqSccvP5+ZJTVuJrl5B+X2bPhNP2aTqZyG1va9W2SbW2KQ4cOMWHCBKysrEzdVxMSEhg4cCCpqan4+PgwevToHOXp0qWL6ZHSpEmTmDJlCm+//TYGgwGtNc7Ozvz0009FcSkKLe7yBo4fn0x6ekbvreS7sRw/PhkApzoZd1f+/v7Z1sW+/y6ha9euhIWF5ch3x44dptdZU58HBAQQEBBg6SqUejIXkxDFKC5wH2nxd3OkW1e1xWmi+SevJ/+4zPYVx0m9l25Kq1DRii5DXWnWpk6RlbW0Cw3tRPLdnGtC29nWpUOHXWaOKB9K0zgIYcaOHTvo1avXIx+f20Ly4vFnLjjklQ6wZ8PpbMEBIPVeOns2nLZo2R43yXfNT6ORW7ooOAkQQhQj66o515bIKx0g8Yb54JFbenlhZ2t+Go3c0kXBSYB4QExMDK6urgwbNgyDwcCAAQO4c+dOtjlZwsPD8fPzA2Dnzp0YjUbT/DRZDX6JiYkMGDAAV1dXhg4damr0279/P507d8bb25uAgABTt8X9+/fj6elJu3bt+OKLL4q/4qJYPBHgjLLJ/r+dsrHiiQDnXI+pXN188Mgtvbxo3GQ8VlbZJ2y0srKncZPxJVSiskcChBknTpygVatWLFiwgCeeeIIvv/ySO3fucOJEzmmSZ8+ezRdffEFkZCS7du3C3j7jP9iIiAjmzJnDO++8Q3BwMKGhoUyZMoX+/fuzdu1a9u/fz8iRI5k8OaNRbcSIEcydO5c9e/YUa11F8XLwqkXVfk1NdwzWVW2p2q9ptgbqB7Xr24QKFbP/r1qhohXt+jYp0rKWdk51+uLqOgs727qAws62Lq6us0wN1KLwCtuLqUxq0KAB169f5/fff+fFF19k7ty5pgCRNfIyS4cOHRg3bhwvvvgi/fr1My160rp1a+rXr4+VlRU1atQgJiaG69evc/nyZbp16wZkrLHr5OREQkIC8fHxpm6IL730Ej///HPxVloUGwevWnkGhAdlNUTv2XCaxBt3qVw9oxdTeW6gzuJUp68EhCJU7gLE8uXLmT17NkopDAYDzz//PDNnzuTevXs8+eSTfPDBB6SlpbFgwQKsra2xt7fHycmJpKQkpk+fzty5c00Lnr/33nuEhobi6+vL/v37GT9+PE899RT29vbZuiDevzBK7dq1iYyMzFam+Pj4bAuuCPGgZm3qSEAQxa5cBYgjR44wa9YsQkNDqVGjBjdu3EApxd69e1FK8d///pevvvqK2NhYRo4cSfPmzTlx4gSurq6cPn2al156iVmzZjFu3Dgg44t98eLFNGnShL/++ov4+HhGjBhBaGgo69aty3H+GjVqcPv2bfbs2UO7du1ISUnh5MmTuLm54ejoyO7du+nYsSMrVqwo7ksjhBA5lKsAsW3bNvza+PDDtAncun6NKk/WoLZPR774bhVxcXHcu3ePOnXq0Lx5cw4ePMiWLVto3749r7/+Olu3bmXx4sWEhITQpk0bAAYNGsScOXPYvn079+7d4+bNm5w+fZqbN2/mWHYRMqYnePnll3n33XdJSEggNTWVt99+Gzc3N5YsWcLIkSOpVKlSuRyQI4QofcpVgIg7dZKzEeE0a/EUALeuXeWjdycy7u2xvDnp3+zYsYOJEydiZWVFr169qFy5MuPHZ/SIqF27NvPmzTOtYxweHo6DgwPz5s0D/l5kpE+fPuzYsYPp06ebzuvr68vw4cOZPn06devWNTtpmre3NwcPHjS9v/94IYQoCeWqF1PFKxeIiLnI7bv3ALhz9x5J9+4Re+APANNEaJBzjpoH3z8oISGBevXq5chHCCEeV2UuQBzbtZ2Fb47g/w3uzcI3R3Bs13bTtio6Df8WT/Hl9j38v19D2HjwGN3dmrLg52106tSJGjVqYGdnx+HDh+nduzfr16/HaDSya9cuBg8ezMcff4yXlxenT+ccwTp9+nQGDhxoykcIIR53ZWoupmO7thO08HNS7/09wrRCRVu6v/YWzTt1YeGbI7h1LecaE1Vq1OS1L5ZYpMxCCFGSZC6mXOxauTxbcABIvXeXXSuXA9Bp8MtUqJh99GmFirZ0GvxysZVRCCEeF2WqkfrW9Wt5pjfv1AXICCRZvZg6DX7ZlC6EEOJvZSpAVHmyhvlHSE/+3SbQvFMXCQhCCJEPZeoRkzxCEkIIyylTdxDyCEkIISynTAUIkEdIQghhKWXqEZMQQgjLKVSAUEoNVEodUUqlK6Vy7XerlOqhlDqhlDqllJpYmHMKIYQoHoW9gzgM9ANyTi6USSllDXwB/BNoAQxRSrUo5HmFEEIUsUK1QWitjwEPW8ugNXBKa30mc9+VQF/gaGHOLYQQomgVRxtEPeDCfe8vZqYJIYQoxR56B6GU2gqYW8pqstZ6Qz7OYe72ItcJoJRSrwGvAfzjH//IR/ZCCCGKwkMDhNb66UKe4yLQ4L739YHYPM63EFgIGZP1FfLcQgghHlFxPGIKA5oqpRoppSoCg4GNxXBeYUHDhw9n7dq1JV0MIUQxKmw31+eUUheBdsBmpdSvmel1lVJbALTWqcBbwK/AMWC11vpI4YothBCiqBUqQGit12ut62utbbXWtbXWAZnpsVrrZ+7bb4vWupnWuonWelZhCy2K3vLlyzEYDHh6evLSSy8BEBISQvv27WncuLHpbiIxMRF/f39atmyJh4cHGzZkNEvFxMTQvHlzRo0ahZubG927dycpKQmAsLAwDAYD7dq1Y8KECbi7uwOQlpbGhAkT8PHxwWAw8NVXXwEQFxeHr68vRqMRd3d3du3aVdyXQ4jySWtdav+8vb21KH6HDx/WzZo101evXtVaa339+nU9bNgwPWDAAJ2WlqaPHDmimzRporXWOiUlRSckJGittb569apu0qSJTk9P12fPntXW1tY6IiJCa631wIED9TfffKO11trNzU2HhoZqrbV+9913tZubm9Za66+++kq///77Wmutk5OTtbe3tz5z5oyePXu2njlzptZa69TUVH3z5s3iuRBCPIaAcG2h7+AyNxeTKLxt27YxYMAA09Kp1atXB+DZZ5/FysqKFi1a8OeffwIZPzDee+89QkJCsLKy4tKlS6ZtjRo1wmg0AuDt7U1MTAzx8fHcunWL9u3bA/DCCy/w008/ARAUFERUVJTp7iQhIYHo6Gh8fHwYOXIkKSkpPPvss6Y8hRBFSwKEyEFrbXbwo62tbbZ9AFasWMHVq1fZv38/NjY2ODs7k5ycnGN/a2trkpKSTMfldt558+YREBCQY1tISAibN2/mpZdeYsKECbz8skzhLkRRk8n6RA7+/v6sXr2a69evA3Djxo1c901ISKBWrVrY2Niwfft2zp07l2fe1apVo0qVKuzduxeAlStXmrYFBAQwf/58UlJSADh58iS3b9/m3Llz1KpVi1GjRvHKK69w4MCBwlZRCJEPcgdRDm0+s5nPDnzG5duXqeNQh7Etx9KzcU/Tdjc3NyZPnkznzp2xtrbGy8sr17yGDh1K7969adWqFUajEVdX14eef9GiRYwaNQoHBwf8/PxwdHQE4NVXXyUmJoaWLVuitaZmzZr8+OOP7Nixg48//hgbGxsqV67M8uXLC38RhBAPpfK65S9prVq10uHh4SVdjDJl85nNTP99OslpyaY0O2s7prefni1IFKXExEQqV64MQGBgIHFxcXz22WfFcu5H0b59e37//fdC5xMTE0OvXr04fPiwRfL6/fffeeGFFwqdlyhblFL7tda5zq5dEPKIqZz57MBn2YIDQHJaMp8dKL4v6M2bN2frsvrvf/+72M79KCwRHCwpNTWVmJgYvvvuu5IuiijjJECUM5dvXy5QelEYNGgQkZGRHD58mM2bN1OzZs1iO/ejyLrb2bFjB35+fgwYMABXV1eGDh1qanSfMWMGPj4+uLu789prr5nS9+/fj6enJ+3ateOLL74w5dmmTRuOHPl7vKifnx/79+/n9u3bjBw5Eh8fH7y8vEzjSpYuXcrAgQPp3bs33bt3Z+LEiezatQuj0cinn36a6xiSvMocFhZG+/bt8fT0pHXr1ty6dSvXfET5JAGinKnjYG7exdzTRXYRERHMmTOHo0ePcubMGUJDQwF46623CAsL4/DhwyQlJZm67o4YMYK5c+eyZ8+ebPkMHjyY1atXAxkDAWNjY/H29mbWrFl07dqVsLAwtm/fzoQJE7h9+zYAe/bsYdmyZWzbto3AwEA6depEZGQk48aNY9GiRTg6OhIWFkZYWBhff/01Z8+ezbXM9+7dY9CgQXz22WccPHiQrVu3Ym9vn2c+ovyRAFHOjG05Fjtru2xpdtZ2jG05toRK9Hhp3bo19evXx8rKCqPRSExMDADbt2+nTZs2eHh4sG3bNo4cOUJCQgLx8fF07twZwDQiHeD5559nzZo1AKxevZqBAwcCGWNBAgMDMRqN+Pn5kZyczPnz5wHo1q2baUzKg4KCgli+fDlGo5E2bdpw/fp1oqOjcy3ziRMncHJywsfHB4AnnniCChUq5JmPKH+kF1M5k9UQnVcvJpG7B8d2pKamkpyczBtvvEF4eDgNGjRg+vTpJCcn5zqeBKBevXo8+eSTREVFsWrVKtOjHK0169atw8XFJdv+f/zxBw4ODrmWK7cxJDt27DBb5tzKltdYFFH+yB1EOdSzcU+CBgQRNSyKoAFBEhwKKWtgYI0aNUhMTDSNBK9atSqOjo7s3r0byBhUeL/Bgwfz0UcfkZCQgIeHB5AxFmTevHmmdoKIiAiz56xSpQq3bt0yvc9tDEluXF1diY2NJSwsDIBbt26Rmppa4HxE2SZ3EKLcW3f5Bh+ciePS3RTq2dowqbET/euYf5RjTtWqVRk1ahQeHh44OzubHtsALFmyhJEjR1KpUqUcv8oHDBjA2LFjmTJliiltypQpvP322xgMBrTWODs7m9oz7mcwGKhQoQKenp4MHz6csWPHmh1DkpuKFSuyatUqxowZQ1JSEvb29mzdujXXsSiifJJxEKJcW3f5BuNPXCAp/e//D+ytFLNdGhQoSAhRWsg4CCEs5IMzcdmCA0BSuuaDM3ElVCIhSg8JEKJcu3Q3pUDpQpQnEiBEuVbP1qZA6UKUJxIgRLk2qbET9lbZu3vaWykmNXYqoRIJUXoUdk3qgUqpI0qpdKVUro0iSqkYpdQhpVSkUkpanUWp0b9OdWa7NKC+rQ0KqG9rIw3UQmQqbDfXw0A/ID8TtnTRWl8r5PmEsLj+dapLQBDCjEIFCK31MSDX0aJCCCEeX8XVBqGBIKXUfqXUa8V0TiGEEIXw0DsIpdRWwNxUn5O11hvyeZ4OWutYpVQt4Del1HGtdUgu53sNeA3gH//4Rz6zF0IIYWkWGUmtlNoBjNdaP7QBWik1HUjUWs/Ox75XgbwXOc5dDaAstXmUtfpA2atTWasPlL06lbX6QM46NdRaW2SRlSKfi0kp5QBYaa1vZb7uDszIz7GFqaRSKtxSw81Lg7JWHyh7dSpr9YGyV6eyVh8o2joVtpvrc0qpi0A7YLNS6tfM9LpKqS2Zu9UGdiulDgL7gM1a618Kc14hhBBFr7C9mNYD682kxwLPZL4+A3gW5jxCCCGKX1keSb2wpAtgYWWtPlD26lTW6gNlr05lrT5QhHUq1dN9CyGEKDll+Q5CCCFEIZSJAFEW54QqQJ16KKVOKKVOKaUmFmcZC0opVV0p9ZtSKjrz32q57FeqP6eHXXOVYW7m9iilVMuSKGd+5aM+fkqphMzPI1IpNbUkylkQSqnFSqkrSqnDuWx/3D6jh9WnaD4jrfVj/wc0B1yAHUCrPPaLAWqUdHktVSfAGjgNNAYqAgeBFiVd9jzq9BEwMfP1RODDx+1zys81J6ODxs+AAtoCf5R0uQtZHz/gp5IuawHr5Qu0BA7nsv2x+YzyWZ8i+YzKxB2E1vqY1vpESZfDkvJZp9bAKa31Ga31PWAl0LfoS/fI+gLLMl8vA54tuaI8svxc877Acp1hL1BVKVVa5w9/3P4byhedMVPDjTx2eZw+o/zUp0iUiQBRAGVtTqh6wIX73l/MTCutamut4wAy/62Vy36l+XPKzzV/nD6X/Ja1nVLqoFLqZ6WUW/EUrUg9Tp9Rfln8MyrykdSWUtxzQhUHC9TJ3DS6JdotLa86FSCbUvU5PSA/17zUfS55yE9ZD5AxfUOiUuoZ4EegaVEXrIg9Tp9RfhTJZ/TYBAit9dMWyCM2898rSqn1ZNxel9gXjwXqdBFocN/7+kBsIfMslLzqpJT6UynlpLWOy7ydv5JLHqXqc3pAfq55qftc8vDQsmqtb973eotS6kulVA39eK/v8jh9Rg9VVJ9RuXnEpJRyUEpVyXpNxpxQZnsEPEbCgKZKqUZKqYrAYGBjCZcpLxuBYZmvhwE57pIeg88pP9d8I/ByZk+ZtkBC1qO1Uuih9VFK1VEqY9EXpVRrMr43rhd7SS3rcfqMHqrIPqOSbp23UAv/c2T8IrgL/An8mpleF9iS+boxGT00DgJHyHiMU+JlL0ydMt8/A5wkoydKaa/Tk0AwEJ35b/XH8XMyd82B0cDozNcK+CJz+yHy6FlXGv7yUZ+3Mj+Lg8BeoH1JlzkfdfoeiANSMv8/euUx/4weVp8i+YxkJLUQQgizys0jJiGEEAUjAUIIIYRZEiCEEEKYJQFCCCGEWRIghBBCmCUBQgghhFkSIIQQQpglAUIIIYRZ/x/vvNbjEXnpvwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(6,3))\n",
    "for i, word in enumerate(vocabs[:20]): #loop each unique vocab\n",
    "    x, y = get_embed(word)\n",
    "    plt.scatter(x, y)\n",
    "    plt.annotate(word, xy=(x, y), xytext=(5, 2), textcoords='offset points')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Do some testing, compare 2-3 sentences with spaCy and see whether our neural network gives the same dependency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Loading data...\n",
      "Example 1:  {'word': ['ms.', 'haag', 'plays', 'elianti', '.'], 'pos': ['NNP', 'NNP', 'VBZ', 'NNP', '.'], 'head': [2, 3, 0, 3, 3], 'dep': ['compound', 'nsubj', 'root', 'dobj', 'punct']}\n",
      "took 2.49 seconds\n"
     ]
    }
   ],
   "source": [
    "train_set, dev_set, test_set = load_data()\n",
    "\n",
    "sentence = [sent for sent in test_set]\n",
    "sentences = [sentence[1], sentence[2], sentence[3]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['but', 'while', 'the', 'new', 'york', 'stock', 'exchange', 'did', \"n't\", 'fall', 'apart', 'friday', 'as', 'the', 'dow', 'jones', 'industrial', 'average', 'plunged', '190.58', 'points', '--', 'most', 'of', 'it', 'in', 'the', 'final', 'hour', '--', 'it', 'barely', 'managed', 'to', 'stay', 'this', 'side', 'of', 'chaos', '.']\n",
      "['some', '``', 'circuit', 'breakers', \"''\", 'installed', 'after', 'the', 'october', '1987', 'crash', 'failed', 'their', 'first', 'test', ',', 'traders', 'say', ',', 'unable', 'to', 'cool', 'the', 'selling', 'panic', 'in', 'both', 'stocks', 'and', 'futures', '.']\n",
      "['the', '49', 'stock', 'specialist', 'firms', 'on', 'the', 'big', 'board', 'floor', '--', 'the', 'buyers', 'and', 'sellers', 'of', 'last', 'resort', 'who', 'were', 'criticized', 'after', 'the', '1987', 'crash', '--', 'once', 'again', 'could', \"n't\", 'handle', 'the', 'selling', 'pressure', '.']\n"
     ]
    }
   ],
   "source": [
    "for s in sentences:\n",
    "    print(s['word'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, Numericalize its!\n",
    "sentences_n = parser.numericalize(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SpaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"26dd622bb8e742479b717d625681936d-0\" class=\"displacy\" width=\"225\" height=\"137.0\" direction=\"ltr\" style=\"max-width: none; height: 137.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"47.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">but</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">CCONJ</tspan>\n",
       "</text>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"059dcf697a08486a9c507c4990925928-0\" class=\"displacy\" width=\"225\" height=\"137.0\" direction=\"ltr\" style=\"max-width: none; height: 137.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"47.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">some</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"8d436049f0a640bfa56dd28be71c0d9d-0\" class=\"displacy\" width=\"225\" height=\"137.0\" direction=\"ltr\" style=\"max-width: none; height: 137.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"47.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">the</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "for s in sentences:\n",
    "    doc = nlp(s['word'][0])\n",
    "    options = {\"collapse_punct\": False}\n",
    "    displacy.render(doc, options = options, style=\"dep\", jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ParserModel(\n",
       "  (pretrained_embeddings): Embedding(5157, 50)\n",
       "  (embed_to_hidden): Linear(in_features=1500, out_features=400, bias=True)\n",
       "  (dropout): Dropout(p=0.5, inplace=False)\n",
       "  (hidden_to_logits): Linear(in_features=400, out_features=3, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load model\n",
    "model = ParserModel(embeddings_matrix)\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Restoring the best model weights found on the dev set\n"
     ]
    },
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'output/20230202_162112/model.weights'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_25290/4142429317.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Restoring the best model weights found on the dev set\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Final evaluation on test set\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mUAS\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdependencies\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.9/site-packages/torch/serialization.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(f, map_location, pickle_module, weights_only, **pickle_load_args)\u001b[0m\n\u001b[1;32m    769\u001b[0m         \u001b[0mpickle_load_args\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'encoding'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    770\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 771\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0m_open_file_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'rb'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mopened_file\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    772\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0m_is_zipfile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopened_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    773\u001b[0m             \u001b[0;31m# The zipfile reader is going to advance the current file position.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.9/site-packages/torch/serialization.py\u001b[0m in \u001b[0;36m_open_file_like\u001b[0;34m(name_or_buffer, mode)\u001b[0m\n\u001b[1;32m    268\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_open_file_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    269\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0m_is_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 270\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_open_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    271\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    272\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;34m'w'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.9/site-packages/torch/serialization.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, name, mode)\u001b[0m\n\u001b[1;32m    249\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0m_open_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_opener\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    250\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 251\u001b[0;31m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_open_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    252\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    253\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__exit__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'output/20230202_162112/model.weights'"
     ]
    }
   ],
   "source": [
    "print(\"Restoring the best model weights found on the dev set\")\n",
    "parser.model.load_state_dict(torch.load(output_path))\n",
    "print(\"Final evaluation on test set\",)\n",
    "parser.model.eval()\n",
    "UAS, dependencies = parser.parse(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'dependency' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_25290/606377867.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdependency\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'dependency' is not defined"
     ]
    }
   ],
   "source": [
    "dependency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "0f2c79af21be9d001248940c049b6176cf8bfb45cabf7aa85848f5cea0f590f6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}