Drop_Dataset_with_Seq2Seq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7-BF6VfBruup"
   },
   "source": [
    "Drop dataset if a Question Answering datset focusing on reading comprehension of models. Each datapoint has:\n",
    "\n",
    "1.   passage: which we refer as context in this notebook. A passage is derived from wikipedia and its corresponding wikipedia url is mentioned in the key 'wiki_url.\n",
    "2.   Qa_pairs: For a single passage, the dataset has mutiple questions. Each question has the answer of the following type:\n",
    "\n",
    "  *   Number: When the answer is a specific number\n",
    "  *   Date: When the answer is a date\n",
    "  *   Span: When the answer is a text like names or places\n",
    "\n",
    "What makes this datset interesting is the presence of multiple questions for the same context. So lets say a batch of 32 contexts might have total 900 questions on them total or 560. This varying number of number of questions for each passage brings up design issues.\n",
    "\n",
    "A goto choice could be that for each question you create an example like\n",
    "<context,question,answer>\n",
    "\n",
    "But this would mean passing the same context again and again through the encoder for questions belonging to the same context which would take time for longer contexts since we are modelling the problem as a seq2seq problem. To avoid this, we create an example like the dataset itself.\n",
    "<context,(question1..question_n),(answer1..answer_n)>\n",
    "\n",
    "This required making customField and a customExample class. To have a batch of contexts and yet allow different number of questions in each batch.\n",
    "\n",
    "\n",
    "The architecture contains three RNN cells. The procedure looks like this:\n",
    "1. We pass context to an encoder.\n",
    "2. We then pass the question through a different encoder cell which takes care of different number of questions for each batch of contexts. We use the last hidden vector of a context passed through context encoder as an input to cell for its question.\n",
    "3. We then use the hidden vectors of the question encoder as an input to the 'answer decoder'.\n",
    "\n",
    "We are initially only focussing on loss and perplexity and not on exact matches of the answer.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9iEKPzfocn5K"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import json\n",
    "from torchtext import data\n",
    "from itertools import chain\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import time\n",
    "from torch.nn import Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "R__vGEjydJeD",
    "outputId": "3e616eda-42df-4dc4-ee52-03d1c75a841d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  drop_dataset.zip\n",
      "  inflating: drop_dataset/drop_dataset_dev.json  \n",
      "  inflating: drop_dataset/drop_dataset_train.json  \n",
      "  inflating: drop_dataset/license.txt  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "--2020-12-23 15:15:08--  https://s3-us-west-2.amazonaws.com/allennlp/datasets/drop/drop_dataset.zip\n",
      "Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.160.92\n",
      "Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.160.92|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8308692 (7.9M) [application/zip]\n",
      "Saving to: ‘drop_dataset.zip’\n",
      "\n",
      "     0K .......... .......... .......... .......... ..........  0%  303K 27s\n",
      "    50K .......... .......... .......... .......... ..........  1%  605K 20s\n",
      "   100K .......... .......... .......... .......... ..........  1% 77.3M 13s\n",
      "   150K .......... .......... .......... .......... ..........  2%  610K 13s\n",
      "   200K .......... .......... .......... .......... ..........  3%  129M 10s\n",
      "   250K .......... .......... .......... .......... ..........  3% 91.2M 9s\n",
      "   300K .......... .......... .......... .......... ..........  4%  111M 7s\n",
      "   350K .......... .......... .......... .......... ..........  4%  146M 6s\n",
      "   400K .......... .......... .......... .......... ..........  5%  617K 7s\n",
      "   450K .......... .......... .......... .......... ..........  6%  136M 6s\n",
      "   500K .......... .......... .......... .......... ..........  6%  118M 6s\n",
      "   550K .......... .......... .......... .......... ..........  7%  111M 5s\n",
      "   600K .......... .......... .......... .......... ..........  8%  137M 5s\n",
      "   650K .......... .......... .......... .......... ..........  8%  141M 4s\n",
      "   700K .......... .......... .......... .......... ..........  9%  114M 4s\n",
      "   750K .......... .......... .......... .......... ..........  9%  140M 4s\n",
      "   800K .......... .......... .......... .......... .......... 10%  175M 4s\n",
      "   850K .......... .......... .......... .......... .......... 11%  626K 4s\n",
      "   900K .......... .......... .......... .......... .......... 11%  134M 4s\n",
      "   950K .......... .......... .......... .......... .......... 12%  128M 4s\n",
      "  1000K .......... .......... .......... .......... .......... 12%  137M 3s\n",
      "  1050K .......... .......... .......... .......... .......... 13%  140M 3s\n",
      "  1100K .......... .......... .......... .......... .......... 14%  142M 3s\n",
      "  1150K .......... .......... .......... .......... .......... 14%  146M 3s\n",
      "  1200K .......... .......... .......... .......... .......... 15%  156M 3s\n",
      "  1250K .......... .......... .......... .......... .......... 16%  184M 3s\n",
      "  1300K .......... .......... .......... .......... .......... 16%  149M 3s\n",
      "  1350K .......... .......... .......... .......... .......... 17%  142M 2s\n",
      "  1400K .......... .......... .......... .......... .......... 17%  141M 2s\n",
      "  1450K .......... .......... .......... .......... .......... 18%  150M 2s\n",
      "  1500K .......... .......... .......... .......... .......... 19%  181M 2s\n",
      "  1550K .......... .......... .......... .......... .......... 19%  227M 2s\n",
      "  1600K .......... .......... .......... .......... .......... 20%  262M 2s\n",
      "  1650K .......... .......... .......... .......... .......... 20%  268M 2s\n",
      "  1700K .......... .......... .......... .......... .......... 21%  274M 2s\n",
      "  1750K .......... .......... .......... .......... .......... 22%  643K 2s\n",
      "  1800K .......... .......... .......... .......... .......... 22% 99.8M 2s\n",
      "  1850K .......... .......... .......... .......... .......... 23%  103M 2s\n",
      "  1900K .......... .......... .......... .......... .......... 24%  160M 2s\n",
      "  1950K .......... .......... .......... .......... .......... 24% 68.5M 2s\n",
      "  2000K .......... .......... .......... .......... .......... 25%  161M 2s\n",
      "  2050K .......... .......... .......... .......... .......... 25%  160M 2s\n",
      "  2100K .......... .......... .......... .......... .......... 26%  186M 2s\n",
      "  2150K .......... .......... .......... .......... .......... 27%  173M 2s\n",
      "  2200K .......... .......... .......... .......... .......... 27%  184M 2s\n",
      "  2250K .......... .......... .......... .......... .......... 28%  179M 1s\n",
      "  2300K .......... .......... .......... .......... .......... 28%  279M 1s\n",
      "  2350K .......... .......... .......... .......... .......... 29%  317M 1s\n",
      "  2400K .......... .......... .......... .......... .......... 30%  237M 1s\n",
      "  2450K .......... .......... .......... .......... .......... 30%  131M 1s\n",
      "  2500K .......... .......... .......... .......... .......... 31%  228M 1s\n",
      "  2550K .......... .......... .......... .......... .......... 32%  187M 1s\n",
      "  2600K .......... .......... .......... .......... .......... 32%  217M 1s\n",
      "  2650K .......... .......... .......... .......... .......... 33%  220M 1s\n",
      "  2700K .......... .......... .......... .......... .......... 33%  233M 1s\n",
      "  2750K .......... .......... .......... .......... .......... 34%  196M 1s\n",
      "  2800K .......... .......... .......... .......... .......... 35%  242M 1s\n",
      "  2850K .......... .......... .......... .......... .......... 35%  231M 1s\n",
      "  2900K .......... .......... .......... .......... .......... 36%  229M 1s\n",
      "  2950K .......... .......... .......... .......... .......... 36%  205M 1s\n",
      "  3000K .......... .......... .......... .......... .......... 37%  207M 1s\n",
      "  3050K .......... .......... .......... .......... .......... 38%  223M 1s\n",
      "  3100K .......... .......... .......... .......... .......... 38%  212M 1s\n",
      "  3150K .......... .......... .......... .......... .......... 39%  192M 1s\n",
      "  3200K .......... .......... .......... .......... .......... 40%  224M 1s\n",
      "  3250K .......... .......... .......... .......... .......... 40%  233M 1s\n",
      "  3300K .......... .......... .......... .......... .......... 41%  231M 1s\n",
      "  3350K .......... .......... .......... .......... .......... 41%  206M 1s\n",
      "  3400K .......... .......... .......... .......... .......... 42%  262M 1s\n",
      "  3450K .......... .......... .......... .......... .......... 43%  246M 1s\n",
      "  3500K .......... .......... .......... .......... .......... 43%  276M 1s\n",
      "  3550K .......... .......... .......... .......... .......... 44%  678K 1s\n",
      "  3600K .......... .......... .......... .......... .......... 44%  214M 1s\n",
      "  3650K .......... .......... .......... .......... .......... 45%  208M 1s\n",
      "  3700K .......... .......... .......... .......... .......... 46%  203M 1s\n",
      "  3750K .......... .......... .......... .......... .......... 46%  141M 1s\n",
      "  3800K .......... .......... .......... .......... .......... 47%  217M 1s\n",
      "  3850K .......... .......... .......... .......... .......... 48%  200M 1s\n",
      "  3900K .......... .......... .......... .......... .......... 48%  203M 1s\n",
      "  3950K .......... .......... .......... .......... .......... 49%  172M 1s\n",
      "  4000K .......... .......... .......... .......... .......... 49%  134M 1s\n",
      "  4050K .......... .......... .......... .......... .......... 50%  198M 1s\n",
      "  4100K .......... .......... .......... .......... .......... 51%  232M 1s\n",
      "  4150K .......... .......... .......... .......... .......... 51%  188M 1s\n",
      "  4200K .......... .......... .......... .......... .......... 52%  231M 1s\n",
      "  4250K .......... .......... .......... .......... .......... 52%  163M 1s\n",
      "  4300K .......... .......... .......... .......... .......... 53%  226M 1s\n",
      "  4350K .......... .......... .......... .......... .......... 54%  179M 1s\n",
      "  4400K .......... .......... .......... .......... .......... 54%  220M 1s\n",
      "  4450K .......... .......... .......... .......... .......... 55%  205M 1s\n",
      "  4500K .......... .......... .......... .......... .......... 56%  161M 1s\n",
      "  4550K .......... .......... .......... .......... .......... 56%  182M 1s\n",
      "  4600K .......... .......... .......... .......... .......... 57%  229M 0s\n",
      "  4650K .......... .......... .......... .......... .......... 57%  200M 0s\n",
      "  4700K .......... .......... .......... .......... .......... 58%  225M 0s\n",
      "  4750K .......... .......... .......... .......... .......... 59%  154M 0s\n",
      "  4800K .......... .......... .......... .......... .......... 59%  223M 0s\n",
      "  4850K .......... .......... .......... .......... .......... 60%  200M 0s\n",
      "  4900K .......... .......... .......... .......... .......... 61%  202M 0s\n",
      "  4950K .......... .......... .......... .......... .......... 61%  205M 0s\n",
      "  5000K .......... .......... .......... .......... .......... 62%  225M 0s\n",
      "  5050K .......... .......... .......... .......... .......... 62%  179M 0s\n",
      "  5100K .......... .......... .......... .......... .......... 63%  217M 0s\n",
      "  5150K .......... .......... .......... .......... .......... 64%  148M 0s\n",
      "  5200K .......... .......... .......... .......... .......... 64%  229M 0s\n",
      "  5250K .......... .......... .......... .......... .......... 65%  184M 0s\n",
      "  5300K .......... .......... .......... .......... .......... 65%  145M 0s\n",
      "  5350K .......... .......... .......... .......... .......... 66%  199M 0s\n",
      "  5400K .......... .......... .......... .......... .......... 67%  242M 0s\n",
      "  5450K .......... .......... .......... .......... .......... 67%  161M 0s\n",
      "  5500K .......... .......... .......... .......... .......... 68%  222M 0s\n",
      "  5550K .......... .......... .......... .......... .......... 69%  190M 0s\n",
      "  5600K .......... .......... .......... .......... .......... 69%  214M 0s\n",
      "  5650K .......... .......... .......... .......... .......... 70%  212M 0s\n",
      "  5700K .......... .......... .......... .......... .......... 70%  242M 0s\n",
      "  5750K .......... .......... .......... .......... .......... 71%  202M 0s\n",
      "  5800K .......... .......... .......... .......... .......... 72%  213M 0s\n",
      "  5850K .......... .......... .......... .......... .......... 72%  212M 0s\n",
      "  5900K .......... .......... .......... .......... .......... 73%  228M 0s\n",
      "  5950K .......... .......... .......... .......... .......... 73%  185M 0s\n",
      "  6000K .......... .......... .......... .......... .......... 74%  236M 0s\n",
      "  6050K .......... .......... .......... .......... .......... 75%  223M 0s\n",
      "  6100K .......... .......... .......... .......... .......... 75%  223M 0s\n",
      "  6150K .......... .......... .......... .......... .......... 76%  203M 0s\n",
      "  6200K .......... .......... .......... .......... .......... 77%  209M 0s\n",
      "  6250K .......... .......... .......... .......... .......... 77%  225M 0s\n",
      "  6300K .......... .......... .......... .......... .......... 78%  213M 0s\n",
      "  6350K .......... .......... .......... .......... .......... 78%  192M 0s\n",
      "  6400K .......... .......... .......... .......... .......... 79%  206M 0s\n",
      "  6450K .......... .......... .......... .......... .......... 80%  269M 0s\n",
      "  6500K .......... .......... .......... .......... .......... 80%  266M 0s\n",
      "  6550K .......... .......... .......... .......... .......... 81%  733K 0s\n",
      "  6600K .......... .......... .......... .......... .......... 81%  216M 0s\n",
      "  6650K .......... .......... .......... .......... .......... 82%  203M 0s\n",
      "  6700K .......... .......... .......... .......... .......... 83%  212M 0s\n",
      "  6750K .......... .......... .......... .......... .......... 83%  191M 0s\n",
      "  6800K .......... .......... .......... .......... .......... 84%  226M 0s\n",
      "  6850K .......... .......... .......... .......... .......... 85%  226M 0s\n",
      "  6900K .......... .......... .......... .......... .......... 85%  234M 0s\n",
      "  6950K .......... .......... .......... .......... .......... 86%  202M 0s\n",
      "  7000K .......... .......... .......... .......... .......... 86%  240M 0s\n",
      "  7050K .......... .......... .......... .......... .......... 87%  222M 0s\n",
      "  7100K .......... .......... .......... .......... .......... 88%  120M 0s\n",
      "  7150K .......... .......... .......... .......... .......... 88%  176M 0s\n",
      "  7200K .......... .......... .......... .......... .......... 89%  248M 0s\n",
      "  7250K .......... .......... .......... .......... .......... 89%  224M 0s\n",
      "  7300K .......... .......... .......... .......... .......... 90%  177M 0s\n",
      "  7350K .......... .......... .......... .......... .......... 91%  161M 0s\n",
      "  7400K .......... .......... .......... .......... .......... 91%  225M 0s\n",
      "  7450K .......... .......... .......... .......... .......... 92%  215M 0s\n",
      "  7500K .......... .......... .......... .......... .......... 93%  228M 0s\n",
      "  7550K .......... .......... .......... .......... .......... 93%  183M 0s\n",
      "  7600K .......... .......... .......... .......... .......... 94%  173M 0s\n",
      "  7650K .......... .......... .......... .......... .......... 94%  159M 0s\n",
      "  7700K .......... .......... .......... .......... .......... 95%  207M 0s\n",
      "  7750K .......... .......... .......... .......... .......... 96%  191M 0s\n",
      "  7800K .......... .......... .......... .......... .......... 96%  212M 0s\n",
      "  7850K .......... .......... .......... .......... .......... 97%  199M 0s\n",
      "  7900K .......... .......... .......... .......... .......... 97%  226M 0s\n",
      "  7950K .......... .......... .......... .......... .......... 98%  160M 0s\n",
      "  8000K .......... .......... .......... .......... .......... 99%  224M 0s\n",
      "  8050K .......... .......... .......... .......... .......... 99%  155M 0s\n",
      "  8100K .......... ...                                        100% 33.0M=0.8s\n",
      "\n",
      "2020-12-23 15:15:10 (10.5 MB/s) - ‘drop_dataset.zip’ saved [8308692/8308692]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "directory=/content/drop_dataset/\n",
    "if [ ! -d \"$directory\" ]; then\n",
    "  # Download drop dataset\n",
    "  wget -c \"https://s3-us-west-2.amazonaws.com/allennlp/datasets/drop/drop_dataset.zip\"\n",
    "  # unzip the file\n",
    "  unzip drop_dataset.zip\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1-z9FK_AeZiQ",
    "outputId": "446c14ba-57c7-45a6-a05c-2efae57613a0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ".vector_cache/glove.6B.zip: 862MB [06:28, 2.22MB/s]                           \n",
      "100%|█████████▉| 399552/400000 [00:16<00:00, 22837.03it/s]"
     ]
    }
   ],
   "source": [
    "train_file = \"/content/drop_dataset/drop_dataset_train.json\"\n",
    "test_file = \"/content/drop_dataset/drop_dataset_dev.json\"\n",
    "\n",
    "train = json.load(open(train_file))\n",
    "test = json.load(open(test_file))\n",
    "\n",
    "train_data = []\n",
    "for i, j in enumerate(train):\n",
    "    ques_ = []\n",
    "    ans_ = []\n",
    "    for l in train[j]['qa_pairs']:\n",
    "        ques_.append(l['question'])\n",
    "        m = l['answer']\n",
    "        if len(m['spans']) > 0:\n",
    "            ans_.append(\" \".join(m['spans']))\n",
    "        elif len(m['number']) > 0:\n",
    "            ans_.append(m['number'])\n",
    "        else:\n",
    "            ans_.append(\" \".join([m['date'][o] for o in m['date']]))\n",
    "    train_data.append([train[j]['passage'], ques_, ans_])\n",
    "\n",
    "test_data = []\n",
    "for i, j in enumerate(test):\n",
    "    ques_ = []\n",
    "    ans_ = []\n",
    "    for l in test[j]['qa_pairs']:\n",
    "        ques_.append(l['question'])\n",
    "        m = l['answer']\n",
    "        if len(m['spans']) > 0:\n",
    "            ans_.append(\" \".join(m['spans']))\n",
    "        elif len(m['number']) > 0:\n",
    "            ans_.append(m['number'])\n",
    "        else:\n",
    "            ans_.append(\" \".join([m['date'][o] for o in m['date']]))\n",
    "    test_data.append([test[j]['passage'], ques_, ans_])\n",
    "\n",
    "class CustomExample(data.Example):\n",
    "    # a problem might happen while trying to convert to tensors or while making vocab\n",
    "    @classmethod\n",
    "    def fromCustomList(cls, data, fields):\n",
    "        ex = cls()\n",
    "        for index, ((name, field), val) in enumerate(zip(fields, data)):\n",
    "            if index == 0:\n",
    "                if isinstance(val, str):\n",
    "                    val = val.rstrip('\\n')\n",
    "                setattr(ex, name, field.preprocess(val))\n",
    "            else:\n",
    "                field_processed_val = []\n",
    "                if not isinstance(val, list):\n",
    "                    val = [val]\n",
    "                for i in val:\n",
    "                    if isinstance(i, str):\n",
    "                        i = i.rstrip('\\n')\n",
    "                    field_processed_val.append(field.preprocess(i))\n",
    "                setattr(ex, name, field_processed_val)\n",
    "        return ex\n",
    "\n",
    "\n",
    "class CustomField(data.Field):\n",
    "    # Creating this class for custom padding and numercalizing of batches of questions\n",
    "    # problem was batch_size number of passages will have way more number of questions on them\n",
    "    # so append all question to make 1 batch for the second RNN cell.\n",
    "\n",
    "    def pad(self, minibatch):\n",
    "        # If I get OOM, restrict number of questions per passage here\n",
    "        self.lengths = [len(i) for i in minibatch]\n",
    "        m = chain.from_iterable(minibatch)\n",
    "        return super(CustomField, self).pad(m)\n",
    "\n",
    "\n",
    "context = data.Field(sequential=True, tokenize='spacy', init_token='<sos>', eos_token='<eos>')\n",
    "question = CustomField(sequential=True, tokenize='spacy', init_token='<sos>', eos_token='<eos>')\n",
    "answer = CustomField(sequential=True, tokenize='spacy', init_token='<sos>', eos_token='<eos>')\n",
    "\n",
    "fields = [('context', context), ('question', question), ('answer', answer)]\n",
    "train_Examples = [\n",
    "    CustomExample.fromCustomList([i[0], i[1], i[2]], fields) for i\n",
    "    in train_data]\n",
    "test_Examples = [\n",
    "    CustomExample.fromCustomList([i[0], i[1], i[2]], fields) for i\n",
    "    in test_data]\n",
    "\n",
    "train_dataset = data.Dataset(train_Examples, fields)\n",
    "test_dataset = data.Dataset(test_Examples,fields)\n",
    "context.build_vocab(train_dataset,min_freq=2,max_size = 30000,vectors = \"glove.6B.100d\", \n",
    "                 unk_init = torch.Tensor.normal_)\n",
    "question.build_vocab(train_dataset,min_freq=2,max_size = 10000,vectors = \"glove.6B.100d\", \n",
    "                 unk_init = torch.Tensor.normal_)\n",
    "answer.build_vocab(train_dataset,min_freq=2,max_size = 5000,vectors = \"glove.6B.100d\", \n",
    "                 unk_init = torch.Tensor.normal_)\n",
    "\n",
    "BATCH_SIZE = 128\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "train_iterator, test_iterator = data.BucketIterator.splits((train_dataset, test_dataset), batch_size=BATCH_SIZE,\n",
    "                                                            sort_key=lambda x: len(x.context), sort_within_batch=True,device=device)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "K1Q129_le3KS",
    "outputId": "12cb6397-e0d5-4ef6-f901-b399edcd3ce7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context:\n",
      "['To', 'start', 'the', 'season', ',', 'the', 'Lions', 'traveled', 'south', 'to', 'Tampa', ',', 'Florida', 'to', 'take', 'on', 'the', 'Tampa', 'Bay', 'Buccaneers', '.', 'The', 'Lions', 'scored', 'first', 'in', 'the', 'first', 'quarter', 'with', 'a', '23-yard', 'field', 'goal', 'by', 'Jason', 'Hanson', '.', 'The', 'Buccaneers', 'tied', 'it', 'up', 'with', 'a', '38-yard', 'field', 'goal', 'by', 'Connor', 'Barth', ',', 'then', 'took', 'the', 'lead', 'when', 'Aqib', 'Talib', 'intercepted', 'a', 'pass', 'from', 'Matthew', 'Stafford', 'and', 'ran', 'it', 'in', '28', 'yards', '.', 'The', 'Lions', 'responded', 'with', 'a', '28-yard', 'field', 'goal', '.', 'In', 'the', 'second', 'quarter', ',', 'Detroit', 'took', 'the', 'lead', 'with', 'a', '36-yard', 'touchdown', 'catch', 'by', 'Calvin', 'Johnson', ',', 'and', 'later', 'added', 'more', 'points', 'when', 'Tony', 'Scheffler', 'caught', 'an', '11-yard', 'TD', 'pass', '.', 'Tampa', 'Bay', 'responded', 'with', 'a', '31-yard', 'field', 'goal', 'just', 'before', 'halftime', '.', 'The', 'second', 'half', 'was', 'relatively', 'quiet', ',', 'with', 'each', 'team', 'only', 'scoring', 'one', 'touchdown', '.', 'First', ',', 'Detroit', \"'s\", 'Calvin', 'Johnson', 'caught', 'a', '1-yard', 'pass', 'in', 'the', 'third', 'quarter', '.', 'The', 'game', \"'s\", 'final', 'points', 'came', 'when', 'Mike', 'Williams', 'of', 'Tampa', 'Bay', 'caught', 'a', '5-yard', 'pass', '.', ' ', 'The', 'Lions', 'won', 'their', 'regular', 'season', 'opener', 'for', 'the', 'first', 'time', 'since', '2007']\n",
      "Questions :\n",
      "[['How', 'many', 'points', 'did', 'the', 'buccaneers', 'need', 'to', 'tie', 'in', 'the', 'first', '?'], ['How', 'many', 'field', 'goals', 'did', 'the', 'Lions', 'score', '?'], ['How', 'long', 'was', 'the', 'Lion', \"'s\", 'longest', 'field', 'goal', '?'], ['Who', 'caught', 'the', 'touchdown', 'for', 'the', 'fewest', 'yard', '?'], ['Who', 'caught', 'the', 'shortest', 'touchdown', 'pass', '?'], ['How', 'many', 'field', 'goals', 'were', 'scored', 'in', 'the', 'first', 'quarter', '?'], ['How', 'many', 'yards', 'was', 'the', 'shortest', 'touchdown', 'scoring', 'play', '?'], ['How', 'many', 'touchdowns', 'were', 'scored', 'in', 'the', 'second', 'quarter', '?'], ['Which', 'player', 'scored', 'the', 'first', 'points', 'of', 'the', 'game', 'for', 'Tampa', 'Bay', '?'], ['How', 'many', 'yards', 'Hanson', 'score', 'with', 'in', 'the', 'first', '?'], ['How', 'many', 'field', 'goals', 'were', 'made', 'in', 'the', 'first', 'quarter', '?'], ['Who', 'threw', 'the', 'first', 'touchdown', 'pass', 'of', 'the', 'game', '?'], ['How', 'many', 'touchdowns', 'were', 'scored', 'in', 'the', '2nd', 'half', '?'], ['Who', 'caught', 'the', 'longest', 'touchdown', 'reception', 'of', 'the', 'game', '?'], ['How', 'many', 'points', 'were', 'scored', 'first', '?']]\n",
      "Answers:\n",
      "[['3'], ['2'], ['28-yard'], ['Mike', 'Williams'], ['Calvin', 'Johnson'], ['3'], ['1'], ['3'], ['Connor', 'Barth'], ['23'], ['3'], ['Matthew', 'Stafford'], ['2'], ['Calvin', 'Johnson'], ['3']]\n"
     ]
    }
   ],
   "source": [
    "print(\"Context:\")\n",
    "print(train_Examples[0].context)\n",
    "print(\"Questions :\")\n",
    "print(train_Examples[0].question)\n",
    "print(\"Answers:\")\n",
    "print(train_Examples[0].answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 167
    },
    "id": "cD8KKuwPe5Sk",
    "outputId": "d8f85e02-259f-43ef-ce30-ac475fb159ae"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-7-e12899fa8f26>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_examples\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'train_examples' is not defined"
     ]
    }
   ],
   "source": [
    "print(len(train_Examples))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tuVd5BFdmvWF"
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "  def __init__(self,context_dim,emb_dim,hid_dim,n_layers,dropout,bidirectional):\n",
    "    super().__init__()\n",
    "    self.hid_dim = hid_dim\n",
    "    \n",
    "    self.embedding = nn.Embedding(context_dim,emb_dim)\n",
    "\n",
    "    self.rnn = nn.LSTM(input_size=emb_dim,hidden_size = hid_dim,num_layers= n_layers,dropout= dropout,bidirectional = bidirectional)\n",
    "\n",
    "    self.dropout = nn.Dropout(dropout)\n",
    "  \n",
    "  def forward(self,context):\n",
    "    \n",
    "    embedded = self.dropout(self.embedding(context))\n",
    "\n",
    "    outputs, (hidden,cell_state) = self.rnn(embedded)\n",
    "\n",
    "    return hidden,cell_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B-1WjaBQm0TH"
   },
   "outputs": [],
   "source": [
    "class Encoder1(nn.Module):\n",
    "  def __init__(self,question_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout):\n",
    "    super().__init__()\n",
    "    self.hid_dim = hid_dim\n",
    "\n",
    "    self.embedded = nn.Embedding(question_dim,emb_dim)\n",
    "\n",
    "    self.rnn = nn.LSTM(input_size=emb_dim,hidden_size = hid_dim,num_layers= n_layers,dropout= dropout,bidirectional = bidirectional)\n",
    "\n",
    "    self.dropout = nn.Dropout(dropout)    \n",
    "    \n",
    "\n",
    "  def fill_tensor(self,hidden, cell_state, question):\n",
    "    # The idea is to make a new tensor that fills the hidden and cell state according to the lengths of the question\n",
    "    \n",
    "    hidden_shape = hidden.shape\n",
    "    question_shape = question.shape\n",
    "    t_hidden = torch.zeros(hidden_shape[0],question_shape[-1],hidden_shape[-1]).to(device)\n",
    "    t_cell_state = torch.zeros(hidden_shape[0],question_shape[-1],hidden_shape[-1]).to(device)\n",
    "    pos = 0\n",
    "    for i in range(hidden_shape[1]):\n",
    "      t_hidden[:,pos:pos+self.lengths[i],:] = hidden[:,i,:].unsqueeze(1)\n",
    "      t_cell_state[:,pos:self.lengths[i],:] = cell_state[:,i,:].unsqueeze(1)\n",
    "      pos += self.lengths[i]\n",
    "    return t_hidden,t_cell_state\n",
    "\n",
    "  def forward(self, question, hidden, cell_state,lengths):\n",
    "    self.lengths = lengths\n",
    "\n",
    "    assert sum(lengths) == question.shape[-1]\n",
    "    input = self.dropout(self.embedded(question))\n",
    "\n",
    "    # filled tensor\n",
    "    filled_hidden, filled_cell_state = self.fill_tensor(hidden,cell_state,question)\n",
    "\n",
    "    outputs , (hidden, cell_state) = self.rnn(input,(filled_hidden,filled_cell_state))\n",
    "\n",
    "    return hidden, cell_state\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "k3ay_sB4y0FK"
   },
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "  def __init__(self,answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout):\n",
    "    super().__init__()\n",
    "\n",
    "    ## I wont be passing the hidden state from last LSTM to each input and fc_out of eaxh time step and emb dimension to fc_out\n",
    "    self.hid_dim = hid_dim\n",
    "\n",
    "    self.embedded = nn.Embedding(answer_dim,emb_dim)\n",
    "\n",
    "    self.rnn = nn.LSTM(input_size=emb_dim,hidden_size = hid_dim,num_layers= n_layers,dropout= dropout,bidirectional = bidirectional)\n",
    "\n",
    "    self.dropout = nn.Dropout(dropout)\n",
    "    \n",
    "    no_of_directions = 2 if bidirectional else 1\n",
    "\n",
    "    self.fc_out = nn.Linear(no_of_directions*hid_dim,answer_dim)\n",
    "\n",
    "  def forward(self, answer, hidden, cell_state):\n",
    "    # this forward will be 1 step at a time\n",
    "\n",
    "    # emdeb answer with dropout\n",
    "    answer = answer.unsqueeze(0)\n",
    "    embedded = self.dropout(self.embedded(answer))\n",
    "\n",
    "    # pass it through cell\n",
    "    output , (hidden,cell_state) = self.rnn(embedded,(hidden,cell_state))\n",
    "\n",
    "    # change dimension of output for fc_out\n",
    "    # output should be [1,batch_size,no_of_directions*hid_dim]\n",
    "    output = output.squeeze(0)\n",
    "    \n",
    "    # return hidden and cell state\n",
    "    prediction = self.fc_out(output)\n",
    "\n",
    "    return prediction, hidden, cell_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "o0In0cetqfVR"
   },
   "outputs": [],
   "source": [
    "\n",
    "import random\n",
    "class Seq2Seq(nn.Module):\n",
    "  def __init__(self,context_dim,question_dim,answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout):\n",
    "    super().__init__()\n",
    "    self.context_encoder = Encoder(context_dim,emb_dim,hid_dim,n_layers,dropout,bidirectional)\n",
    "    self.question_encoder = Encoder1(question_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout)\n",
    "    self.answer_decoder = Decoder(answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout)\n",
    "\n",
    "    self.answer_dim = answer_dim\n",
    "  def forward(self,context,question,answer,lengths,teacher_forcing_ratio = 0.5):\n",
    "\n",
    "    # encode context\n",
    "    hidden,cell_state = self.context_encoder(context)\n",
    "\n",
    "    # encode question\n",
    "    hidden,cell_state = self.question_encoder(question,hidden,cell_state,lengths)\n",
    "\n",
    "    # pass the answer one by one\n",
    "    answer_len = len(answer)\n",
    "    batch_size = answer.shape[1]\n",
    "\n",
    "    outputs = torch.zeros(answer_len,batch_size,self.answer_dim).to(device)\n",
    "\n",
    "    for i,j in enumerate(range(answer_len)):\n",
    "      k = answer[j]\n",
    "      if not i == 0:\n",
    "        k = prediction.argmax(1) if random.random() < teacher_forcing_ratio else k\n",
    "      prediction,hidden,cell_state = self.answer_decoder(k,hidden,cell_state)\n",
    "      outputs[j] = prediction\n",
    "\n",
    "    return outputs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-nczYUvTzMnk",
    "outputId": "db2f71a0-48f0-4fb0-e72b-b5186b4d170f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:61: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1\n",
      "  \"num_layers={}\".format(dropout, num_layers))\n"
     ]
    }
   ],
   "source": [
    "context_dim = len(context.vocab)\n",
    "question_dim = len(question.vocab)\n",
    "answer_dim = len(answer.vocab)\n",
    "emb_dim = 100\n",
    "hid_dim = 100\n",
    "n_layers = 1\n",
    "bidirectional = True\n",
    "dropout = 0.5\n",
    "\n",
    "model = Seq2Seq(context_dim,question_dim,answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout).to(device)\n",
    "\n",
    "def init_weights(m):\n",
    "    for name, param in m.named_parameters():\n",
    "      if not isinstance(m, Embedding):\n",
    "        nn.init.normal_(param.data, mean=0, std=0.01)\n",
    "        \n",
    "model.apply(init_weights)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "\n",
    "TRG_PAD_IDX = answer.vocab.stoi[answer.pad_token]\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index = TRG_PAD_IDX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SZHL7hR14A5S"
   },
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion, clip):\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    for i, batch in enumerate(iterator):\n",
    "        \n",
    "        context_ = batch.context\n",
    "        question_ = batch.question\n",
    "        answer_ = batch.answer\n",
    "        lengths = question.lengths # I know this is not the right way to do..maybe create a new filed for lenghts but they depend of the chosen batch??\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        output = model(context_, question_,answer_,lengths)\n",
    "        \n",
    "        trg = answer_\n",
    "        #trg = [trg len, batch size]\n",
    "        #output = [trg len, batch size, output dim]\n",
    "        \n",
    "        output_dim = output.shape[-1]\n",
    "        \n",
    "        output = output[1:].view(-1, output_dim)\n",
    "        trg = trg[1:].view(-1)\n",
    "        \n",
    "        #trg = [(trg len - 1) * batch size]\n",
    "        #output = [(trg len - 1) * batch size, output dim]\n",
    "        \n",
    "        loss = criterion(output, trg)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nCvWtM2J5BeO"
   },
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for i, batch in enumerate(iterator):\n",
    "\n",
    "            context_ = batch.context\n",
    "            question_ = batch.question\n",
    "            answer_ = batch.answer\n",
    "            lengths = question.lengths\n",
    "        \n",
    "\n",
    "            output = model(context_, question_,answer_,lengths,0) #turn off teacher forcing\n",
    "\n",
    "            trg = answer_\n",
    "            #trg = [trg len, batch size]\n",
    "            #output = [trg len, batch size, output dim]\n",
    "\n",
    "            output_dim = output.shape[-1]\n",
    "            \n",
    "            output = output[1:].view(-1, output_dim)\n",
    "            trg = trg[1:].view(-1)\n",
    "\n",
    "            #trg = [(trg len - 1) * batch size]\n",
    "            #output = [(trg len - 1) * batch size, output dim]\n",
    "\n",
    "            loss = criterion(output, trg)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PGBoT91y5Ln6"
   },
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HoDqVqra5NpP",
    "outputId": "e5d11860-ef27-499d-c1bc-af9f75a818aa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Time: 0m 45s\n",
      "\tTrain Loss: 3.957 | Train PPL:  52.296\n",
      "\t Val. Loss: 3.313 |  Val. PPL:  27.456\n",
      "Epoch: 02 | Time: 0m 45s\n",
      "\tTrain Loss: 3.849 | Train PPL:  46.937\n",
      "\t Val. Loss: 3.258 |  Val. PPL:  25.992\n",
      "Epoch: 03 | Time: 0m 45s\n",
      "\tTrain Loss: 3.855 | Train PPL:  47.220\n",
      "\t Val. Loss: 3.213 |  Val. PPL:  24.847\n",
      "Epoch: 04 | Time: 0m 45s\n",
      "\tTrain Loss: 3.703 | Train PPL:  40.556\n",
      "\t Val. Loss: 3.152 |  Val. PPL:  23.379\n",
      "Epoch: 05 | Time: 0m 45s\n",
      "\tTrain Loss: 3.632 | Train PPL:  37.801\n",
      "\t Val. Loss: 3.063 |  Val. PPL:  21.387\n",
      "Epoch: 06 | Time: 0m 45s\n",
      "\tTrain Loss: 3.619 | Train PPL:  37.291\n",
      "\t Val. Loss: 3.043 |  Val. PPL:  20.977\n",
      "Epoch: 07 | Time: 0m 45s\n",
      "\tTrain Loss: 3.568 | Train PPL:  35.433\n",
      "\t Val. Loss: 2.987 |  Val. PPL:  19.818\n",
      "Epoch: 08 | Time: 0m 45s\n",
      "\tTrain Loss: 3.484 | Train PPL:  32.576\n",
      "\t Val. Loss: 2.912 |  Val. PPL:  18.403\n",
      "Epoch: 09 | Time: 0m 45s\n",
      "\tTrain Loss: 3.439 | Train PPL:  31.147\n",
      "\t Val. Loss: 2.850 |  Val. PPL:  17.286\n",
      "Epoch: 10 | Time: 0m 45s\n",
      "\tTrain Loss: 3.376 | Train PPL:  29.265\n",
      "\t Val. Loss: 2.780 |  Val. PPL:  16.127\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "N_EPOCHS = 10\n",
    "CLIP = 1\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss = train(model, train_iterator, optimizer, criterion, CLIP)\n",
    "    valid_loss = evaluate(model, test_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut2-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Drop Dataset with Seq2Seq.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}