Irony_Detection_Traditional_and_NN_approach.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Irony Detection - Traditional and NN approach",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/KrisSandy/ExMachineLearning/blob/master/Irony_Detection_Traditional_and_NN_approach.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "tx6n_xxbj6dV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# IRONY DETECTION - Traditional and NN approach"
      ]
    },
    {
      "metadata": {
        "id": "qndnYAQpL6I8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We will use the data from the SemEval-2018 task on irony detection. The file `SemEval2018-T3-train-taskA.txt` consists of examples as follows:"
      ]
    },
    {
      "metadata": {
        "id": "vEvckziyL6I9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "```csv\n",
        "Tweet index     Label   Tweet text\n",
        "1       1       Sweet United Nations video. Just in time for Christmas. #imagine #NoReligion  http://t.co/fej2v3OUBR\n",
        "2       1       @mrdahl87 We are rumored to have talked to Erv's agent... and the Angels asked about Ed Escobar... that's hardly nothing    ;)\n",
        "3       1       Hey there! Nice to see you Minnesota/ND Winter Weather \n",
        "4       0       3 episodes left I'm dying over here\n",
        "```\n"
      ]
    },
    {
      "metadata": {
        "id": "oc7hUIIjL6I9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Data Extraction\n",
        "\n",
        "Read all the data and find the size of vocabulary of the dataset (ignoring case) and the number of positive and negative examples."
      ]
    },
    {
      "metadata": {
        "id": "GY3XqnhZkF0r",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### import required libraries"
      ]
    },
    {
      "metadata": {
        "id": "hyEdGlJHzN8P",
        "colab_type": "code",
        "outputId": "fd56d2a3-98a5-4083-ec50-724f34c8c910",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import nltk\n",
        "from nltk.tokenize import word_tokenize\n",
        "from collections import Counter\n",
        "from sklearn.model_selection import train_test_split\n",
        "import string\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "metadata": {
        "id": "8i9uEnyGkMw1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Downloading the data file"
      ]
    },
    {
      "metadata": {
        "id": "jYI46bjcQvwO",
        "colab_type": "code",
        "outputId": "cba008eb-fc26-47cb-8357-690ca581f810",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        }
      },
      "cell_type": "code",
      "source": [
        "!wget https://transfer.sh/DnXTx/SemEval2018-T3-train-taskA.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-03-02 13:45:02--  https://transfer.sh/DnXTx/SemEval2018-T3-train-taskA.txt\n",
            "Resolving transfer.sh (transfer.sh)... 144.76.136.153\n",
            "Connecting to transfer.sh (transfer.sh)|144.76.136.153|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 380455 (372K) [text/plain]\n",
            "Saving to: ‘SemEval2018-T3-train-taskA.txt’\n",
            "\n",
            "SemEval2018-T3-trai 100%[===================>] 371.54K   892KB/s    in 0.4s    \n",
            "\n",
            "2019-03-02 13:45:04 (892 KB/s) - ‘SemEval2018-T3-train-taskA.txt’ saved [380455/380455]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "noHsxIfKkUFl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Calculate stats"
      ]
    },
    {
      "metadata": {
        "id": "pBeJkhuNvImm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_stats(data):\n",
        "\n",
        "  n = len(data)\n",
        "  n_pos = sum(data['Label'])\n",
        "  n_neg = n - n_pos\n",
        "  vocab = list()\n",
        "\n",
        "  for text in data['Tweet text']:\n",
        "    words = word_tokenize(text.lower())\n",
        "    vocab = vocab + words\n",
        " \n",
        "  vocab = set(vocab)\n",
        "\n",
        "  print(\"Total number of examples : \", n)\n",
        "  print(\"Number of positive examples : \", n_pos)\n",
        "  print(\"Number of negative examples : \", n_neg)\n",
        "  print(\"Size of Vocabulary : \", len(vocab))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4rnKw-u-lrFg",
        "colab_type": "code",
        "outputId": "44917d69-1882-4763-e63b-61ffa0e722c0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "cell_type": "code",
      "source": [
        "data = pd.read_csv('SemEval2018-T3-train-taskA.txt', sep='\\t')\n",
        "get_stats(data)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total number of examples :  3817\n",
            "Number of positive examples :  1901\n",
            "Number of negative examples :  1916\n",
            "Size of Vocabulary :  13460\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qVdS5s6yL6I-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Naive Bayes Model\n",
        "\n",
        "Develop a classifier using the Naive Bayes model to predict if an example is ironic. The model should convert each Tweet into a bag-of-words and calculate\n",
        "\n",
        "$p(\\text{Ironic}|w_1,\\ldots,w_n) \\propto \\prod_{i=1,\\ldots,n} p(w_i \\in \\text{tweet}| \\text{Ironic}) p(\\text{Ironic})$\n",
        "\n",
        "$p(\\text{NotIronic}|w_1,\\ldots,w_n) \\propto \\prod_{i=1,\\ldots,n} p(w_i \\in \\text{tweet}| \\text{NotIronic}) p(\\text{NotIronic})$\n",
        "\n",
        "You should use add-alpha smoothing to calculate probabilities"
      ]
    },
    {
      "metadata": {
        "id": "UROTQD-yngIk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Naive Bayes"
      ]
    },
    {
      "metadata": {
        "id": "TIo2oMh0KFm6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the Naive Bayes model, we calculated the posterior probabilities are calculated for each class (Ironic and NotIronic) and the class is predicted as the class with highest probability. The probability od a sentence being Ironic or not is calculated as shown in the above equation.\n",
        "\n",
        "For calculating the probability of each word being Ironic or not, a vocabulary is created for both the classes using the training examples and correcponding word counts are maintained.\n",
        "\n",
        "Now the probability of word is calculated as \n",
        "\n",
        "$p(w_i|Ironic) = \\frac{c(w_{i, Ironic})}{c(w_{Ironic})}$\n",
        "\n",
        "$\\text{where }c(w_{i, Ironic}) = \\text{count of word } w_i \\text{ in Ironic vocabulary}$\n",
        "$\\text{where }c(w_{Ironic}) = \\text{count of all words in Ironic vocabulary}$\n",
        "\n",
        "$p(w_i|NotIronic) = \\frac{c(w_{i, NotIronic})}{c(w_{NotIronic})}$\n",
        "\n",
        "$\\text{where }c(w_{i, NotIronic}) = \\text{count of word } w_i \\text{ in NotIronic vocabulary}$\n",
        "$\\text{where }c(w_{NotIronic}) = \\text{count of all words in NotIronic vocabulary}$"
      ]
    },
    {
      "metadata": {
        "id": "-wXxNxKxOj3S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Add $\\alpha$ Smoothing"
      ]
    },
    {
      "metadata": {
        "id": "wbF3Ln7eOu5x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the above calculations, for missing words / new words, the probability is calculated as zero. To avoid this we will use alpha smoothing.\n",
        "\n",
        "Probability calculation using add alpha smoothing:\n",
        "\n",
        "\n",
        "$p(w_i|Ironic) = \\frac{c(w_{i, Ironic}) + \\alpha}{c(w_{Ironic})+\\alpha|v|}$\n",
        "\n",
        "$p(w_i|NotIronic) = \\frac{c(w_{i, NotIronic}) + \\alpha}{c(w_{NotIronic})+\\alpha|v|}$"
      ]
    },
    {
      "metadata": {
        "id": "QgiQwtsb8diq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The above classifier is implemented in below class.\n",
        "\n",
        "\n",
        "*   The fit method takes the training data and clacluates $P(Ironic)$ and $P(NonIronic)$, creates two vocabularies, one with ironic words and other with non ironic words, and the corresponding word counts\n",
        "*   The predict function takes in the list of words in the sentence and using add $\\alpha$ smoothing, calculates the posterior probabiities $p(\\text{Ironic}|w_1,\\ldots,w_n)$ and $p(\\text{NotIronic}|w_1,\\ldots,w_n)$. It returns the probabilities ad a flag indicating the class of the sentence\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "ycgaGTRYY-vB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Implementation"
      ]
    },
    {
      "metadata": {
        "id": "2dAnjOHtZEYE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Below fit method takes the training data and clacluates P(Ironic) and P(NonIronic), creates two vocabularies, one with ironic words and other with non ironic words, and the corresponding word counts"
      ]
    },
    {
      "metadata": {
        "id": "EvRAEYRgZIVA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def fit(train):\n",
        "  n = len(train)\n",
        "  n_pos = 0\n",
        "  n_neg = 0\n",
        "  vocab_pos = list()\n",
        "  vocab_neg = list()\n",
        "  for row in train:\n",
        "    if row[1] == 1:\n",
        "      n_pos += 1\n",
        "      vocab_pos = vocab_pos + row[2]\n",
        "    elif row[1] == 0:\n",
        "      n_neg += 1\n",
        "      vocab_neg = vocab_neg + row[2]\n",
        "    else:\n",
        "      raise Exception(\"Unknown Label\")\n",
        "\n",
        "  cache = dict()\n",
        "  cache['f_vocab_pos'] = Counter(vocab_pos)\n",
        "  cache['f_vocab_neg'] = Counter(vocab_neg)\n",
        "  cache['p_pos'] = n_pos/n\n",
        "  cache['p_neg'] = 1 - cache['p_pos']\n",
        "  return cache"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xJArmCwHbXKo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The predict function takes in the list of words in the sentence and using add $\\alpha$ smoothing, calculates the posterior probabiities $p(\\text{Ironic}|w_1,\\ldots,w_n)$ and $p(\\text{NotIronic}|w_1,\\ldots,w_n)$. It returns the probabilities ad a flag indicating the class of the sentence"
      ]
    },
    {
      "metadata": {
        "id": "8kJeTmg-bYMl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def predict(cache, test, alpha=1):\n",
        "\n",
        "  f_vocab_pos = cache['f_vocab_pos']\n",
        "  f_vocab_neg = cache['f_vocab_neg']\n",
        "  n_words_pos = sum(f_vocab_pos.values())\n",
        "  n_words_neg = sum(f_vocab_neg.values())\n",
        "  size_vocab_pos = len(f_vocab_pos)\n",
        "  size_vocab_neg = len(f_vocab_neg)\n",
        "\n",
        "  predictions = list()\n",
        "  for row in test:\n",
        "    p_text_pos = 1\n",
        "    p_text_neg = 1\n",
        "    for word in row[2]:\n",
        "      if word not in f_vocab_pos:\n",
        "        f_vocab_pos[word] = 0\n",
        "      if word not in f_vocab_neg:\n",
        "        f_vocab_neg[word] = 0\n",
        "      p_text_pos *= (((f_vocab_pos[word] + alpha) /\n",
        "                      (n_words_pos + alpha * size_vocab_pos)) * \n",
        "                       cache['p_pos'])\n",
        "      p_text_neg *= (((f_vocab_neg[word] + alpha) / \n",
        "                      (n_words_neg + alpha * size_vocab_neg)) * \n",
        "                       cache['p_neg'])\n",
        "    predictions.append(int(p_text_pos > p_text_neg))\n",
        "      \n",
        "  return predictions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "w5LivWxOL6JA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Train and Test above Model\n",
        "\n",
        "Divide the data into a training and test set and justify your split.\n",
        "\n",
        "Choose a suitable evaluation metric and implement it. Explain why you chose this evaluation metric.\n",
        "\n",
        "Evaluate the method in Task 2 according to this metric."
      ]
    },
    {
      "metadata": {
        "id": "-q_QkiH82wM7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Split the dataset into test and train"
      ]
    },
    {
      "metadata": {
        "id": "AIIAwXt93MxP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Data has been split into 80% training and 20% testing. Extracted the counts from both the datasets reveal that number of Ironic and NonIronic examples are distributed evenly in both the datasets. This proves that the distribution of examples is not biased and have enough examples in both the classes for training and testing the model"
      ]
    },
    {
      "metadata": {
        "id": "cPuSyMjhfMTf",
        "colab_type": "code",
        "outputId": "e6364da9-508a-40c4-c9a6-8ac19c88e164",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 243
        }
      },
      "cell_type": "code",
      "source": [
        "train_raw, test_raw = train_test_split(data, test_size=0.20, random_state = 20)\n",
        "print(\"Train dataset stats\")\n",
        "print(\"-------------------\")\n",
        "get_stats(train_raw)\n",
        "print(\"\\nTest dataset stats\")\n",
        "print(\"------------------\")\n",
        "get_stats(test_raw)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train dataset stats\n",
            "-------------------\n",
            "Total number of examples :  3053\n",
            "Number of positive examples :  1513\n",
            "Number of negative examples :  1540\n",
            "Size of Vocabulary :  11466\n",
            "\n",
            "Test dataset stats\n",
            "------------------\n",
            "Total number of examples :  764\n",
            "Number of positive examples :  388\n",
            "Number of negative examples :  376\n",
            "Size of Vocabulary :  4058\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1xAjaWeI49B1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Convert the training and test set to list of (index, label, list of words) format\n"
      ]
    },
    {
      "metadata": {
        "id": "R4vIE4xlvV3g",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train = list()\n",
        "test = list()\n",
        "for index, row in train_raw.iterrows():\n",
        "  words = word_tokenize(row['Tweet text'].lower())\n",
        "  train.append((index, row['Label'], words))\n",
        "test_list = list()\n",
        "for index, row in test_raw.iterrows():\n",
        "  words = word_tokenize(row['Tweet text'].lower())\n",
        "  test.append((index, row['Label'], words))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VYcMe0XYA44M",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Evaluation Metric\n",
        "\n",
        "For evaluating the classifier, I have used accuracy and F1 score to measure the performance the classifier. As the dataset is equally distributed, accuracy can be used to check the overall performance for both the classes, while the F1 score give the harmonic mean of precision and recall.\n",
        "\n",
        "$Accuracy = \\frac{\\text{tp + tn}}{\\text{total examples}}$\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "v0iC7C5SepCB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluate(y_predict, y):\n",
        "  \n",
        "  n_correct = 0\n",
        "  \n",
        "  if len(y_predict) != len(y):\n",
        "    raise Exception(\"Input vectors are of different lengths\")\n",
        "    \n",
        "  for i in range(len(y)):\n",
        "    if y[i] == y_predict[i]:\n",
        "      n_correct += 1\n",
        "      \n",
        "  return n_correct/len(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "T0Ba8nRWDBqL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Training and Evaluate the model"
      ]
    },
    {
      "metadata": {
        "id": "YfSANq1bf7PG",
        "colab_type": "code",
        "outputId": "fe0444b5-f0c4-4267-f012-d225b6a9ed86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "cell_type": "code",
      "source": [
        "cache = fit(train)\n",
        "print(\"Probability of Ironic : \", cache['p_pos'])\n",
        "print(\"Probability of non Ironic : \", cache['p_neg'])\n",
        "print(\"Vacab size of Ironic : \", len(cache['f_vocab_pos']))\n",
        "print(\"Vacab size of non Ironic : \", len(cache['f_vocab_neg']))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Probability of Ironic :  0.4955781198820832\n",
            "Probability of non Ironic :  0.5044218801179168\n",
            "Vacab size of Ironic :  6338\n",
            "Vacab size of non Ironic :  7217\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "3B6h3xw_LhbQ",
        "colab_type": "code",
        "outputId": "4230a880-7920-487e-df3b-d1e080ba6226",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "y_test = [r[1] for r in test]\n",
        "y_nb_predict = predict(cache, test)\n",
        "accuracy = evaluate(y_nb_predict, y_test)\n",
        "print(\"Accuracy : \", accuracy)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy :  0.6544502617801047\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "82JnhmgBL6JA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Train and Test NN with LSTM\n",
        "\n",
        "Run the following code to generate a model from your training set. The training set should be in a variable  called `train` and is assumed to be of the form:\n",
        "\n",
        "```\n",
        "[(1, 1, ['sweet', 'united', 'nations', 'video', '.', 'just', 'in', 'time', 'for', 'christmas', '.', '#', 'imagine', '#', 'noreligion', 'http', ':', '//t.co/fej2v3oubr']), \n",
        " (2, 1, ['@', 'mrdahl87', 'we', 'are', 'rumored', 'to', 'have', 'talked', 'to', 'erv', \"'s\", 'agent', '...', 'and', 'the', 'angels', 'asked', 'about', 'ed', 'escobar', '...', 'that', \"'s\", 'hardly', 'nothing', ';', ')']), \n",
        " (3, 1, ['hey', 'there', '!', 'nice', 'to', 'see', 'you', 'minnesota/nd', 'winter', 'weather']), \n",
        " (4, 0, ['3', 'episodes', 'left', 'i', \"'m\", 'dying', 'over', 'here']), \n",
        " ...\n",
        "]\n",
        " ```\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "iV207juZL6JB",
        "colab_type": "code",
        "outputId": "698f5c17-6ce7-4d24-b28c-85cab6eab616",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1999
        }
      },
      "cell_type": "code",
      "source": [
        "from keras.models import Sequential, load_model\n",
        "from keras.layers import Dense, Activation, Embedding, Dropout, TimeDistributed\n",
        "from keras.layers import LSTM\n",
        "from keras.optimizers import Adam\n",
        "from keras.utils import to_categorical\n",
        "from keras.callbacks import ModelCheckpoint\n",
        "import numpy as np\n",
        "\n",
        "## These values should be set from Task 3\n",
        "# train, test = task3()\n",
        "\n",
        "def make_dictionary(train, test):\n",
        "    dictionary = {}\n",
        "    for d in train+test:\n",
        "        for w in d[2]:\n",
        "            if w not in dictionary:\n",
        "                dictionary[w] = len(dictionary)\n",
        "    return dictionary\n",
        "\n",
        "class KerasBatchGenerator(object):\n",
        "    def __init__(self, data, num_steps, batch_size, vocabulary, skip_step=5):\n",
        "        self.data = data\n",
        "        self.num_steps = num_steps\n",
        "        self.batch_size = batch_size\n",
        "        self.vocabulary = vocabulary\n",
        "        self.current_idx = 0\n",
        "        self.current_sent = 0\n",
        "        self.skip_step = skip_step\n",
        "\n",
        "    def generate(self):\n",
        "        x = np.zeros((self.batch_size, self.num_steps))\n",
        "        y = np.zeros((self.batch_size, self.num_steps, 2))\n",
        "        while True:\n",
        "            for i in range(self.batch_size):\n",
        "                # Choose a sentence and position with at lest num_steps more words\n",
        "                while self.current_idx + self.num_steps >= len(self.data[self.current_sent][2]):\n",
        "                    self.current_idx = self.current_idx % len(self.data[self.current_sent][2])\n",
        "                    self.current_sent += 1\n",
        "                    if self.current_sent >= len(self.data):\n",
        "                        self.current_sent = 0\n",
        "                # The rows of x are set to values like [1,2,3,4,5]\n",
        "                x[i, :] = [self.vocabulary[w] for w in self.data[self.current_sent][2][self.current_idx:self.current_idx + self.num_steps]]\n",
        "                # The rows of y are set to values like [[1,0],[1,0],[1,0],[1,0],[1,0]]\n",
        "                y[i, :, :] = [[self.data[self.current_sent][1], 1-self.data[self.current_sent][1]]] * self.num_steps\n",
        "                self.current_idx += self.skip_step\n",
        "            yield x, y\n",
        "\n",
        "# Hyperparameters for model\n",
        "vocabulary = make_dictionary(train, test)\n",
        "num_steps = 5\n",
        "batch_size = 20\n",
        "num_epochs = 50 # Reduce this if the model is taking too long to train (or increase for performance)\n",
        "hidden_size = 50 # Increase this to improve perfomance (or increase for performance)\n",
        "use_dropout=True\n",
        "\n",
        "# Create batches for RNN\n",
        "train_data_generator = KerasBatchGenerator(train, num_steps, batch_size, vocabulary,\n",
        "                                           skip_step=num_steps)\n",
        "valid_data_generator = KerasBatchGenerator(test, num_steps, batch_size, vocabulary,\n",
        "                                           skip_step=num_steps)\n",
        "\n",
        "# A double stacked LSTM with dropout and n hidden layers\n",
        "model = Sequential()\n",
        "model.add(Embedding(len(vocabulary), hidden_size, input_length=num_steps))\n",
        "model.add(LSTM(hidden_size, return_sequences=True))\n",
        "model.add(LSTM(hidden_size, return_sequences=True))\n",
        "if use_dropout:\n",
        "    model.add(Dropout(0.5))\n",
        "model.add(TimeDistributed(Dense(2)))\n",
        "model.add(Activation('softmax'))\n",
        "\n",
        "# Set optimizer and build model\n",
        "optimizer = Adam()\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['categorical_accuracy'])\n",
        "\n",
        "# Train the model\n",
        "model.fit_generator(train_data_generator.generate(), len(train)//(batch_size*num_steps), num_epochs,\n",
        "                        validation_data=valid_data_generator.generate(),\n",
        "                        validation_steps=len(test)//(batch_size*num_steps))\n",
        "\n",
        "# Save the model\n",
        "model.save(\"final_model.hdf5\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n",
            "Epoch 1/50\n",
            "30/30 [==============================] - 2s 76ms/step - loss: 0.6933 - categorical_accuracy: 0.5213 - val_loss: 0.6840 - val_categorical_accuracy: 0.6643\n",
            "Epoch 2/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6895 - categorical_accuracy: 0.5533 - val_loss: 0.6884 - val_categorical_accuracy: 0.5571\n",
            "Epoch 3/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6800 - categorical_accuracy: 0.5817 - val_loss: 0.6788 - val_categorical_accuracy: 0.5714\n",
            "Epoch 4/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6875 - categorical_accuracy: 0.5417 - val_loss: 0.6732 - val_categorical_accuracy: 0.6186\n",
            "Epoch 5/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6704 - categorical_accuracy: 0.5740 - val_loss: 0.6969 - val_categorical_accuracy: 0.5214\n",
            "Epoch 6/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6784 - categorical_accuracy: 0.5717 - val_loss: 0.6720 - val_categorical_accuracy: 0.5843\n",
            "Epoch 7/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6579 - categorical_accuracy: 0.6207 - val_loss: 0.7168 - val_categorical_accuracy: 0.5200\n",
            "Epoch 8/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6322 - categorical_accuracy: 0.6543 - val_loss: 0.7162 - val_categorical_accuracy: 0.5414\n",
            "Epoch 9/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6253 - categorical_accuracy: 0.6700 - val_loss: 0.6989 - val_categorical_accuracy: 0.5714\n",
            "Epoch 10/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6219 - categorical_accuracy: 0.6810 - val_loss: 0.7126 - val_categorical_accuracy: 0.5414\n",
            "Epoch 11/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.5882 - categorical_accuracy: 0.6977 - val_loss: 0.7338 - val_categorical_accuracy: 0.5414\n",
            "Epoch 12/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.6043 - categorical_accuracy: 0.6900 - val_loss: 0.7392 - val_categorical_accuracy: 0.5514\n",
            "Epoch 13/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.5676 - categorical_accuracy: 0.7130 - val_loss: 0.7872 - val_categorical_accuracy: 0.5357\n",
            "Epoch 14/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.5158 - categorical_accuracy: 0.7420 - val_loss: 0.8284 - val_categorical_accuracy: 0.5014\n",
            "Epoch 15/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.4876 - categorical_accuracy: 0.7703 - val_loss: 0.7398 - val_categorical_accuracy: 0.6029\n",
            "Epoch 16/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.4893 - categorical_accuracy: 0.7713 - val_loss: 0.7743 - val_categorical_accuracy: 0.5857\n",
            "Epoch 17/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.4405 - categorical_accuracy: 0.7903 - val_loss: 0.8360 - val_categorical_accuracy: 0.5471\n",
            "Epoch 18/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.4620 - categorical_accuracy: 0.7787 - val_loss: 0.8143 - val_categorical_accuracy: 0.5843\n",
            "Epoch 19/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.4496 - categorical_accuracy: 0.7830 - val_loss: 0.8242 - val_categorical_accuracy: 0.5471\n",
            "Epoch 20/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3735 - categorical_accuracy: 0.8170 - val_loss: 1.0482 - val_categorical_accuracy: 0.5143\n",
            "Epoch 21/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3823 - categorical_accuracy: 0.8180 - val_loss: 1.0087 - val_categorical_accuracy: 0.5114\n",
            "Epoch 22/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3953 - categorical_accuracy: 0.8080 - val_loss: 0.8924 - val_categorical_accuracy: 0.6057\n",
            "Epoch 23/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3847 - categorical_accuracy: 0.8080 - val_loss: 0.7972 - val_categorical_accuracy: 0.6114\n",
            "Epoch 24/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3926 - categorical_accuracy: 0.8120 - val_loss: 0.9526 - val_categorical_accuracy: 0.5586\n",
            "Epoch 25/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3889 - categorical_accuracy: 0.8127 - val_loss: 0.8916 - val_categorical_accuracy: 0.5486\n",
            "Epoch 26/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3553 - categorical_accuracy: 0.8170 - val_loss: 0.9512 - val_categorical_accuracy: 0.5571\n",
            "Epoch 27/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3446 - categorical_accuracy: 0.8217 - val_loss: 1.0654 - val_categorical_accuracy: 0.5129\n",
            "Epoch 28/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3533 - categorical_accuracy: 0.8143 - val_loss: 1.0878 - val_categorical_accuracy: 0.4943\n",
            "Epoch 29/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3351 - categorical_accuracy: 0.8373 - val_loss: 0.9186 - val_categorical_accuracy: 0.6100\n",
            "Epoch 30/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3636 - categorical_accuracy: 0.8190 - val_loss: 0.9378 - val_categorical_accuracy: 0.5857\n",
            "Epoch 31/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3519 - categorical_accuracy: 0.8177 - val_loss: 0.9284 - val_categorical_accuracy: 0.5329\n",
            "Epoch 32/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3052 - categorical_accuracy: 0.8413 - val_loss: 1.0869 - val_categorical_accuracy: 0.5214\n",
            "Epoch 33/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3244 - categorical_accuracy: 0.8300 - val_loss: 1.0664 - val_categorical_accuracy: 0.5571\n",
            "Epoch 34/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3321 - categorical_accuracy: 0.8250 - val_loss: 1.0438 - val_categorical_accuracy: 0.5486\n",
            "Epoch 35/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3070 - categorical_accuracy: 0.8463 - val_loss: 1.0322 - val_categorical_accuracy: 0.5671\n",
            "Epoch 36/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3372 - categorical_accuracy: 0.8310 - val_loss: 1.1525 - val_categorical_accuracy: 0.5400\n",
            "Epoch 37/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3740 - categorical_accuracy: 0.8133 - val_loss: 0.8038 - val_categorical_accuracy: 0.5900\n",
            "Epoch 38/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.2938 - categorical_accuracy: 0.8433 - val_loss: 1.0889 - val_categorical_accuracy: 0.5100\n",
            "Epoch 39/50\n",
            "30/30 [==============================] - 1s 19ms/step - loss: 0.3165 - categorical_accuracy: 0.8320 - val_loss: 1.1252 - val_categorical_accuracy: 0.5500\n",
            "Epoch 40/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3223 - categorical_accuracy: 0.8303 - val_loss: 1.1029 - val_categorical_accuracy: 0.5457\n",
            "Epoch 41/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.2804 - categorical_accuracy: 0.8590 - val_loss: 1.1621 - val_categorical_accuracy: 0.5271\n",
            "Epoch 42/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3228 - categorical_accuracy: 0.8320 - val_loss: 1.0745 - val_categorical_accuracy: 0.5529\n",
            "Epoch 43/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3230 - categorical_accuracy: 0.8353 - val_loss: 1.0535 - val_categorical_accuracy: 0.5500\n",
            "Epoch 44/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.2823 - categorical_accuracy: 0.8470 - val_loss: 0.9354 - val_categorical_accuracy: 0.5914\n",
            "Epoch 45/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3066 - categorical_accuracy: 0.8357 - val_loss: 1.1344 - val_categorical_accuracy: 0.5257\n",
            "Epoch 46/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3057 - categorical_accuracy: 0.8297 - val_loss: 1.1487 - val_categorical_accuracy: 0.5171\n",
            "Epoch 47/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.2945 - categorical_accuracy: 0.8490 - val_loss: 1.1963 - val_categorical_accuracy: 0.5314\n",
            "Epoch 48/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3062 - categorical_accuracy: 0.8453 - val_loss: 1.2615 - val_categorical_accuracy: 0.5414\n",
            "Epoch 49/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.3182 - categorical_accuracy: 0.8330 - val_loss: 1.1241 - val_categorical_accuracy: 0.5471\n",
            "Epoch 50/50\n",
            "30/30 [==============================] - 1s 20ms/step - loss: 0.2749 - categorical_accuracy: 0.8480 - val_loss: 1.0339 - val_categorical_accuracy: 0.5814\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qLYwZTAVL6JH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now consider the following code:"
      ]
    },
    {
      "metadata": {
        "id": "wQfP5qylL6JH",
        "colab_type": "code",
        "outputId": "b0136dec-442c-48a8-9a29-99ccfdf9624a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "cell_type": "code",
      "source": [
        "model = load_model(\"final_model.hdf5\")\n",
        "\n",
        "x = np.zeros((1,num_steps))\n",
        "x[0,:] = [vocabulary[\"this\"],vocabulary[\"is\"],vocabulary[\"an\"],vocabulary[\"easy\"],vocabulary[\"test\"]]\n",
        "print(model.predict(x))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[[0.32178226 0.67821777]\n",
            "  [0.35342664 0.64657336]\n",
            "  [0.37172422 0.62827575]\n",
            "  [0.26450235 0.73549765]\n",
            "  [0.65818816 0.34181184]]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zh8vY22gL6JL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Using the code above write a function that can predict the label using the LSTM model above and compare it with the evaluation performed in Task 3"
      ]
    },
    {
      "metadata": {
        "id": "I3xifEODQQID",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Below function uses the model created above and gets the Ironic and Non Ironic probabilities of each word. As the model accepts 5 words at a time, predictions are made looping through 5 words at a time and making the predictions. When the sentence has less than 5 words or when the last part of the sentence has less than 5 words, fullstop is padded at the end to make the length 5. This is a hack to make the size of words to 5. The probabilities are then multiplied to calculate the total probability of sentence being ironic or not. "
      ]
    },
    {
      "metadata": {
        "id": "57s6phLgZL3V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def predict_using_keras_model(test):\n",
        "  \n",
        "  y_predict = list()\n",
        "  \n",
        "  for row in test:\n",
        "    current_pos = 0\n",
        "    to_pos=0\n",
        "    p_predict = np.ones(2)\n",
        "    max_i = int(len(row[2])/num_steps)+1\n",
        "    for i in range(max_i):\n",
        "      x = np.zeros((1, num_steps))\n",
        "      to_pos = to_pos+num_steps\n",
        "      if to_pos > len(row[2]):\n",
        "        to_pos = len(row[2])\n",
        "        current_pos = to_pos-num_steps\n",
        "      if current_pos < 0:\n",
        "        x_temp = [vocabulary['.']]*5\n",
        "        x_temp[0:len(row[2])] = [vocabulary[w] for w in row[2]]\n",
        "        x[0, :] = x_temp\n",
        "      else:\n",
        "        x[0, :] = [vocabulary[w] for w in row[2][current_pos:to_pos]]\n",
        "      p_temp = model.predict(x)\n",
        "      p_predict = p_predict * np.prod(p_temp[0], axis=0)\n",
        "      current_pos += num_steps\n",
        "      \n",
        "    y_predict.append(int(p_predict[0] > p_predict[1]))\n",
        "  \n",
        "  return y_predict\n",
        "\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2i33hD3vqu1r",
        "colab_type": "code",
        "outputId": "d9ab7d18-1038-4785-8a0a-270423342ab6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "y_lstm_predict = predict_using_keras_model(test)\n",
        "accuracy = evaluate(y_lstm_predict, y_test)\n",
        "print(\"Accuracy : \", accuracy)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy :  0.5589005235602095\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "9yZucGZmL6JM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Improvements to above models\n",
        "\n",
        "Suggest an improvement to either the system developed in Task 2 or 4 and show that it improves according to your evaluation metric.\n",
        "\n",
        "Please note this task is marked according to: demonstration of knowledge from the lecutures (10), originality and appropriateness of solution (10), completeness of description (10), technical correctness (5) and improvement in evaluation metric (5)."
      ]
    },
    {
      "metadata": {
        "id": "QZiww5b6pLtL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Preprocessing"
      ]
    },
    {
      "metadata": {
        "id": "EjKFNPT1YmX5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Tweets are generally written in informal language for example \"im, ur, rofl...\" etc. These also contain emojis, urls, user names etc. Preprocessing can be to \n",
        "\n",
        "\n",
        "*   Convert the emojis to acual emotion (sad, happy etc)\n",
        "*   Replace the words with lemmas using lemmatisation\n",
        "*  Replace URL's with a keyword `<URL>` as these doesn't add any value in the calculation.\n",
        "* Replace user names with `<USER>` as these doesn't add any value in the calculation.\n",
        "\n",
        "Along with above, a model can be built to expands like rofl etc to its full text.\n",
        "\n",
        "In the below implementation, I have changed urls and user names to keywords `<URL>` and `<USER>` as part of preprocessing.\n"
      ]
    },
    {
      "metadata": {
        "id": "7QTMkaAsppmb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import re\n",
        "\n",
        "def preprocessing(data):\n",
        "  \n",
        "  tweets = list()\n",
        "  \n",
        "  for index, row in data.iterrows():\n",
        "    tweet = row[2]\n",
        "    \n",
        "    #Replace URL's with token <URL>\n",
        "    tweet = re.sub(r'((http(s)?:\\/\\/[\\S]+)|www\\.[\\S]+)', '<URL>', tweet)\n",
        "    \n",
        "    #Replace user name with <USER>\n",
        "    tweet = re.sub(r'(@[\\S]+)', '<USER>', tweet)\n",
        "    \n",
        "    tweets.append(tweet)\n",
        "    \n",
        "  return tweets"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vRbiggWlcVS_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Model "
      ]
    },
    {
      "metadata": {
        "id": "HAdtdwzmcYt6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the above model, two LTSM are used with 5 words in a tweet processed at a time. Each time, a tweet is divided into chunks of 5 words and are sent as seperate sentences to the model for training and predictions. Some of the words at the end of the sentence are ignored due to the restriction of 5 input words. Because of this, the context of the tweets are not captured completely and few words are ignored in out training. \n"
      ]
    },
    {
      "metadata": {
        "id": "1emMiW1eemlK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Below improvements are done for the model created in Task 4"
      ]
    },
    {
      "metadata": {
        "id": "dDdRrJSSeuE_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Considering full tweet"
      ]
    },
    {
      "metadata": {
        "id": "NQx_d3nYdv6n",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As an improvement to the model, I have used all the words in the tweet as input to the model. To mitigate the difference in length of (count of words) each tweet, zeros has been padded infront of the sentence sequence vector to make the length equal to tweet with most number of words. This method considers all the words in the tweets capturing the context in a better way."
      ]
    },
    {
      "metadata": {
        "id": "r0pEmJbzVB_J",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# gets the maximum number of words in a tweet using all the input tweets\n",
        "\n",
        "def get_maxlen(tweets):\n",
        "  return max([len(tweet.split()) for tweet in tweets])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KzVtHMVhfImO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "tweets = preprocessing(data)\n",
        "max_len = get_maxlen(tweets)\n",
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts(tweets)\n",
        "X = tokenizer.texts_to_sequences(tweets)\n",
        "X = pad_sequences(X, maxlen=max_len)\n",
        "y = data['Label'].values\n",
        "vocab_size = len(tokenizer.word_index) + 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fSO7ZeG0fhFO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Using Bidirectional RNN"
      ]
    },
    {
      "metadata": {
        "id": "q2rWDHDKflEu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In Natural Language, sometimes the context words are present ahead in the sentence with respect to current word. LTSM are good in memorizing the context which is already processed, but not the context which is unprocessed ot lie ahead of current word. Hence we use bidirectional LSTM to capture this information."
      ]
    },
    {
      "metadata": {
        "id": "24dyc_UMgMg-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Using POS tags"
      ]
    },
    {
      "metadata": {
        "id": "84JwDE6ZgTq2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The input to the LSTM are the word embeddings using which LTSM predicts whether the tweet is ironic or not. POS tag of the words also provide information in about whether the tweet is ironic or not. This information can also be used in the model for predictions.\n",
        "\n",
        "For using POS tags, nltk's pos_tag method is used to extract the pos_tags from the sentence. As NN accepts inly numerical data as inputs, we need to convert this POS tags information to numbers. In order to do this, a dictionary of all possible POS tags are created along with its index as its value. Using this dictionary all the POS tags in the tweet in converted to numbers. Each POS sequence is also padded with 0 to maximum length to make the length of each vector same. \n",
        "\n",
        "Finally, the numbers are normalised by dividing with number of POS tags which will give each tag a number between 0 and 1. This is done to improve to the performance of the NN"
      ]
    },
    {
      "metadata": {
        "id": "AY5b4NE53ZbM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from nltk import pos_tag\n",
        "\n",
        "def tweet_pos_tagger(tweets):\n",
        "  tweets_pos_tags = [pos_tag(word_tokenize(tweet)) for tweet in tweets]\n",
        "  all_pos = set([tag_pair[1] for tweet_pos in tweets_pos_tags for tag_pair in tweet_pos])\n",
        "  pos_dict = {pos_tag: i for i, pos_tag in enumerate(all_pos)}\n",
        "  total_tags = len(pos_dict)\n",
        "  max_len = get_maxlen(tweets)\n",
        "  \n",
        "  tweets_tag_vector = list()\n",
        "  for tweet_pos in tweets_pos_tags:\n",
        "    tweet_pos_vector = np.zeros(max_len)\n",
        "    tweet_pos_vector = np.asarray([pos_dict[word_tag[1]] for word_tag in tweet_pos])\n",
        "    tweets_tag_vector.append(tweet_pos_vector)\n",
        "    \n",
        "  tweets_tag_vector = pad_sequences(tweets_tag_vector, maxlen=max_len)\n",
        "  return tweets_tag_vector / total_tags"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dXDuk1DRh5_S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "T = np.stack(tweet_pos_tagger(tweets))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pYS5NAwKh9Bx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Both the inputs are merged, to split the data into test and train datasets. When the split is done both the inputs are seperated as they are processed seperately.\n",
        "\n",
        "Note: In below test train split, same random seed is used to produced the same dataset as above used in Task 2 and Task 3"
      ]
    },
    {
      "metadata": {
        "id": "1aiJPlCMiOok",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "XT = np.concatenate((X, T), axis=1)\n",
        "\n",
        "# Same random state is used to generate same test train split as above \n",
        "XT_train, XT_test, y_train, y_test = train_test_split(XT, y, test_size=0.20, random_state = 20)\n",
        "X_train, T_train = np.hsplit(XT_train, 2)\n",
        "X_test, T_test = np.hsplit(XT_test, 2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lkjdj-KZidYH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Model building, Training"
      ]
    },
    {
      "metadata": {
        "id": "Xo0q7fRhio0m",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Below model uses the input tweet data (X) and builds the LTSM using word embeddings of the input. The output of LSTM is combined with POS information and passed to a deep dense conneted network with ReLu activation functions. The output of (LTSB,POS) is combined with LSTM and returned.\n",
        "\n",
        "Reference: https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models"
      ]
    },
    {
      "metadata": {
        "id": "g5H6NFUVJoNQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from keras.layers import concatenate, Input, Bidirectional, Model\n",
        "\n",
        "tweet_words = Input(shape=(max_len,), name='tweet_words')\n",
        "tweet_embd = Embedding(output_dim=512, input_dim=vocab_size, input_length=max_len)(tweet_words)\n",
        "lstm_out = Bidirectional(LSTM(32))(tweet_embd)\n",
        "lstm_temp_out = Dense(1, activation='sigmoid', name='aux_output')(lstm_out)\n",
        "pos_info = Input(shape=(max_len,), name='aux_input')\n",
        "combined_out = concatenate([lstm_out, pos_info])\n",
        "combined_out = Dense(64, activation='relu')(combined_out)\n",
        "combined_out = Dense(64, activation='relu')(combined_out)\n",
        "combined_output = Dense(1, activation='sigmoid', name='combined_output')(combined_out)\n",
        "model_multi = Model(inputs=[tweet_words, pos_info], outputs=[combined_output, lstm_temp_out])\n",
        "model_multi.compile(optimizer='adam', loss='binary_crossentropy',\n",
        "              loss_weights=[1., 0.2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "duQULczsMl3L",
        "colab_type": "code",
        "outputId": "cf9cdb42-607f-4a64-e1c9-b0f9843cec4d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 382
        }
      },
      "cell_type": "code",
      "source": [
        "epochs = 10\n",
        "model_multi.fit([X_train, T_train], [y_train, y_train], epochs = epochs, batch_size=batch_size)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "3053/3053 [==============================] - 66s 22ms/step - loss: 0.7958 - combined_output_loss: 0.6634 - aux_output_loss: 0.6621\n",
            "Epoch 2/10\n",
            "3053/3053 [==============================] - 64s 21ms/step - loss: 0.4771 - combined_output_loss: 0.3951 - aux_output_loss: 0.4103\n",
            "Epoch 3/10\n",
            "3053/3053 [==============================] - 64s 21ms/step - loss: 0.1704 - combined_output_loss: 0.1390 - aux_output_loss: 0.1572\n",
            "Epoch 4/10\n",
            "3053/3053 [==============================] - 63s 21ms/step - loss: 0.0599 - combined_output_loss: 0.0474 - aux_output_loss: 0.0625\n",
            "Epoch 5/10\n",
            "3053/3053 [==============================] - 63s 21ms/step - loss: 0.0292 - combined_output_loss: 0.0222 - aux_output_loss: 0.0347\n",
            "Epoch 6/10\n",
            "3053/3053 [==============================] - 63s 21ms/step - loss: 0.0248 - combined_output_loss: 0.0195 - aux_output_loss: 0.0269\n",
            "Epoch 7/10\n",
            "3053/3053 [==============================] - 63s 21ms/step - loss: 0.0119 - combined_output_loss: 0.0082 - aux_output_loss: 0.0184\n",
            "Epoch 8/10\n",
            "3053/3053 [==============================] - 64s 21ms/step - loss: 0.0022 - combined_output_loss: 8.6669e-04 - aux_output_loss: 0.0068\n",
            "Epoch 9/10\n",
            "3053/3053 [==============================] - 64s 21ms/step - loss: 0.0010 - combined_output_loss: 2.5176e-04 - aux_output_loss: 0.0040\n",
            "Epoch 10/10\n",
            "3053/3053 [==============================] - 64s 21ms/step - loss: 7.2093e-04 - combined_output_loss: 1.6205e-04 - aux_output_loss: 0.0028\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f5bf9fdb4a8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "metadata": {
        "id": "5w6uxEfSPTIr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Improvement in evaluation Metric\n",
        "\n",
        "For evaluating the classifier, I have used accuracy and F1 score to measure the performance the classifier. As the dataset is equally distributed, accuracy can be used to check the overall performance for both the classes, while the F1 score give the harmonic mean of precision and recall.\n",
        "\n",
        "$Accuracy = \\frac{\\textit{true positive + true negative}}{\\textit{total examples}}$\n",
        "\n",
        "$Precision = \\frac{\\textit{true positive}}{\\textit{true positive + false positive}}$\n",
        "\n",
        "$Recall = \\frac{\\textit{true positive}}{\\textit{true positive + false negative}}$\n",
        "\n",
        "$Precision = \\frac{\\textit{2} \\times \\textit{precision} \\times \\textit{recall}}{\\textit{precision + recall}}$"
      ]
    },
    {
      "metadata": {
        "id": "TjzEzABJPXSa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluate_new(y_predict, y):\n",
        "  \n",
        "  tp, fn, fp, tn = 0, 0, 0, 0\n",
        "  \n",
        "  if len(y_predict) != len(y):\n",
        "    raise Exception(\"Input vectors are of different lengths\")\n",
        "    \n",
        "  for i in range(len(y)):\n",
        "    if y[i] == 0 and y_predict[i] == 0:\n",
        "      tn += 1\n",
        "    elif y[i] == 1 and y_predict[i] == 1:\n",
        "      tp += 1\n",
        "    elif y[i] == 0 and y_predict[i] == 1:\n",
        "      fp += 1\n",
        "    elif y[i] == 1 and y_predict[i] == 0:\n",
        "      fn += 1\n",
        "  \n",
        "  accuracy = (tp + tn) / len(y)\n",
        "  precision = tp / (tp + fp)\n",
        "  recall = tp / (tp + fn)\n",
        "  f1 = (2 * precision * recall) / (precision + recall)\n",
        "  \n",
        "  return accuracy, precision, recall, f1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yiI4KUjkkEyX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Evaluating the model"
      ]
    },
    {
      "metadata": {
        "id": "-0TaBvNkllNw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Evaluation metric of improved model"
      ]
    },
    {
      "metadata": {
        "id": "2qRuSncUOF7p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "y_multi_predict = model_multi.predict([X_test, T_test])\n",
        "y_final_predict = np.where(y_multi_predict[0] > 0.5, 1, 0)\n",
        "y_final_predict = np.ravel(y_final_predict)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wgk6bokHYYER",
        "colab_type": "code",
        "outputId": "4dbccc91-ba4d-40a0-f98f-11f20fbccc66",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "cell_type": "code",
      "source": [
        "accuracy, precision, recall, f1 = evaluate_new(y_final_predict, y_test)\n",
        "print(\"Accuracy: \", accuracy)\n",
        "print(\"Precision: \", precision)\n",
        "print(\"Recall: \", recall)\n",
        "print(\"F1 score: \", f1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy:  0.6178010471204188\n",
            "Precision:  0.638728323699422\n",
            "Recall:  0.5695876288659794\n",
            "F1 score:  0.6021798365122615\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "q-P1Ex6Sl9is",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Evaluation metric of LSTM model (Task 3)"
      ]
    },
    {
      "metadata": {
        "id": "xcGOAe27kune",
        "colab_type": "code",
        "outputId": "2b145691-304c-43e5-d1ab-11fe219656fd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "cell_type": "code",
      "source": [
        "accuracy, precision, recall, f1 = evaluate_new(y_lstm_predict, y_test)\n",
        "print(\"Accuracy: \", accuracy)\n",
        "print(\"Precision: \", precision)\n",
        "print(\"Recall: \", recall)\n",
        "print(\"F1 score: \", f1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy:  0.5589005235602095\n",
            "Precision:  0.580952380952381\n",
            "Recall:  0.47164948453608246\n",
            "F1 score:  0.5206258890469417\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mnTo684-mEA0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Evaluation metric of Naive Bayes classifier (Task 2)"
      ]
    },
    {
      "metadata": {
        "id": "wigfnJ9dgBu2",
        "colab_type": "code",
        "outputId": "4348f5db-6b70-4794-cb79-8e31f2fcde2d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "cell_type": "code",
      "source": [
        "accuracy, precision, recall, f1 = evaluate_new(y_nb_predict, y_test)\n",
        "print(\"Accuracy: \", accuracy)\n",
        "print(\"Precision: \", precision)\n",
        "print(\"Recall: \", recall)\n",
        "print(\"F1 score: \", f1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy:  0.6544502617801047\n",
            "Precision:  0.6353711790393013\n",
            "Recall:  0.75\n",
            "F1 score:  0.6879432624113475\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "g6RROS4Om0oi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "An improvement of 6% is observed in the improved model when compared to LSTM model in Task 3. However, the accuracy and F1 score is still behinf Naive Bayes model. Training the model longer might increase the accuracy as the model is trainied for only 3 epochs."
      ]
    },
    {
      "metadata": {
        "id": "GI8pzqV6nFgV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Further improvements"
      ]
    },
    {
      "metadata": {
        "id": "jGvE71JxnIem",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Some more features can be added to the model to improve the performance like capturing the sentiment (counts of positive and negative words) etc. Improvements can also be done with respect to pos_tagger as it is not customized for tweets, the error percentage in tagging will be high."
      ]
    },
    {
      "metadata": {
        "id": "95q4c-PDl4mc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Improvising Naive Bayes"
      ]
    },
    {
      "metadata": {
        "id": "WH8VBVDDmBCW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Interpolation"
      ]
    },
    {
      "metadata": {
        "id": "kw9LhT1pmFH_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Along with using unigram probaility, bigram probabilities can be used to improv the model. Both these probabilities can be combined as below \n",
        "\n",
        "$P(Ironic) = \\lambda_1P_{unigram}(Ironic) + \\lambda_2P_{bigram}(Ironic)$\n",
        "\n",
        "$\\textit{where } \\lambda_1 + \\lambda_2 = 1$"
      ]
    },
    {
      "metadata": {
        "id": "WHnCkxN5n7fi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def fit_new(train):\n",
        "    n = len(train)\n",
        "    n_pos = 0\n",
        "    n_neg = 0\n",
        "    unigram_vocab_pos = list()\n",
        "    unigram_vocab_neg = list()\n",
        "    bigram_vocab_pos = list()\n",
        "    bigram_vocab_neg = list()\n",
        "    for row in train:\n",
        "        words = row[2]\n",
        "        bigrams = list(nltk.bigrams(words))\n",
        "        if row[1] == 1:\n",
        "            n_pos += 1\n",
        "            unigram_vocab_pos = unigram_vocab_pos + words\n",
        "            bigram_vocab_pos = bigram_vocab_pos + bigrams\n",
        "        elif row[1] == 0:\n",
        "            n_neg += 1\n",
        "            unigram_vocab_neg = unigram_vocab_neg + words\n",
        "            bigram_vocab_neg = bigram_vocab_neg + bigrams\n",
        "        else:\n",
        "            raise Exception(\"Unknown Label\")\n",
        "\n",
        "    cache = dict()\n",
        "    cache['f_unigram_pos'] = Counter(unigram_vocab_pos)\n",
        "    cache['f_unigram_neg'] = Counter(unigram_vocab_neg)\n",
        "    cache['f_bigram_pos'] = ConditionalFreqDist(bigram_vocab_pos)\n",
        "    cache['f_bigram_neg'] = ConditionalFreqDist(bigram_vocab_neg)\n",
        "    cache['n_unigram_pos'] = len(unigram_vocab_pos)\n",
        "    cache['n_unigram_neg'] = len(unigram_vocab_neg)\n",
        "    cache['n_bigram_pos'] = len(bigram_vocab_pos)\n",
        "    cache['n_bigram_neg'] = len(bigram_vocab_neg)\n",
        "    cache['p_pos'] = n_pos / n\n",
        "    cache['p_neg'] = 1 - cache['p_pos']\n",
        "    return cache"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "C3zshHgent5I",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_unigram_prob(cache, words, alpha=1):\n",
        "    f_unigram_pos = cache['f_unigram_pos']\n",
        "    f_unigram_neg = cache['f_unigram_neg']\n",
        "    n_unigram_pos = cache['n_unigram_pos']\n",
        "    n_unigram_neg = cache['n_unigram_neg']\n",
        "    s_unigram_pos = len(f_unigram_pos)\n",
        "    s_unigram_neg = len(f_unigram_neg)\n",
        "    p_unigram_pos = 1\n",
        "    p_unigram_neg = 1\n",
        "\n",
        "    for word in words:\n",
        "        if word not in f_unigram_pos:\n",
        "            f_unigram_pos[word] = 0\n",
        "        if word not in f_unigram_neg:\n",
        "            f_unigram_neg[word] = 0\n",
        "        p_unigram_pos *= (((f_unigram_pos[word] + alpha) /\n",
        "                        (n_unigram_pos + alpha * s_unigram_pos)))\n",
        "        p_unigram_neg *= (((f_unigram_neg[word] + alpha) /\n",
        "                        (n_unigram_neg + alpha * s_unigram_neg)))\n",
        "\n",
        "    p_unigram_pos *= cache['p_pos']\n",
        "    p_unigram_neg *= cache['p_neg']\n",
        "    return p_unigram_pos, p_unigram_neg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "979yAaWpnw1O",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from nltk.probability import ConditionalFreqDist\n",
        "\n",
        "def get_bigram_prob(cache, words, alpha=1):\n",
        "    f_bigram_pos = cache['f_bigram_pos']\n",
        "    f_bigram_neg = cache['f_bigram_neg']\n",
        "    n_bigram_pos = cache['n_bigram_pos']\n",
        "    n_bigram_neg = cache['n_bigram_neg']\n",
        "    s_bigram_pos = len(f_bigram_pos)\n",
        "    s_bigram_neg = len(f_bigram_neg)\n",
        "    p_bigram_pos = 1\n",
        "    p_bigram_neg = 1\n",
        "\n",
        "    bigrams = list(nltk.bigrams(words))\n",
        "\n",
        "    for bigram in bigrams:\n",
        "        if bigram[0] not in f_bigram_pos:\n",
        "            c_bigram_pos = 0\n",
        "        elif bigram[1] not in f_bigram_pos[bigram[0]]:\n",
        "            c_bigram_pos = 0\n",
        "        else:\n",
        "            c_bigram_pos = f_bigram_pos[bigram[0]].get(bigram[1])\n",
        "\n",
        "        if bigram[0] not in f_bigram_neg:\n",
        "            c_bigram_neg = 0\n",
        "        elif bigram[1] not in f_bigram_neg[bigram[0]]:\n",
        "            c_bigram_neg = 0\n",
        "        else:\n",
        "            c_bigram_neg = f_bigram_neg[bigram[0]].get(bigram[1])\n",
        "\n",
        "        p_bigram_pos *= (((c_bigram_pos + alpha) /\n",
        "                          (n_bigram_pos + alpha * s_bigram_pos)))\n",
        "        p_bigram_neg *= (((c_bigram_neg + alpha) /\n",
        "                          (n_bigram_neg + alpha * s_bigram_neg)))\n",
        "\n",
        "    p_bigram_pos *= cache['p_pos']\n",
        "    p_bigram_neg *= cache['p_neg']\n",
        "    return p_bigram_pos, p_bigram_neg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fvdUfe2moHY-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def predict_new(cache, test, alpha=1):\n",
        "    predictions = list()\n",
        "    lambda1 = 0.3\n",
        "    lambda2 = 0.7\n",
        "    for row in test:\n",
        "        p_unigram_pos, p_unigram_neg = get_unigram_prob(cache, row[2], alpha)\n",
        "        p_bigram_pos, p_bigram_neg = get_bigram_prob(cache, row[2], alpha)\n",
        "\n",
        "        p_predict_pos = lambda1*p_unigram_pos + lambda2*p_bigram_pos\n",
        "        p_predict_neg = lambda1*p_unigram_neg + lambda2*p_bigram_neg\n",
        "\n",
        "        predictions.append(int(p_predict_pos > p_predict_neg))\n",
        "\n",
        "    return predictions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oavP_5VnoQCt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cache = fit_new(train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3hxrl92IomFS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "y_new_nv = predict_new(cache, test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Qzu5-SzXouHI",
        "colab_type": "code",
        "outputId": "8fe062fe-d6c6-4c92-9907-431584edf397",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "cell_type": "code",
      "source": [
        "accuracy, precision, recall, f1 = evaluate_new(y_new_nv, y_test)\n",
        "print(\"Accuracy: \", accuracy)\n",
        "print(\"Precision: \", precision)\n",
        "print(\"Recall: \", recall)\n",
        "print(\"F1 score: \", f1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy:  0.6518324607329843\n",
            "Precision:  0.6270833333333333\n",
            "Recall:  0.7757731958762887\n",
            "F1 score:  0.6935483870967742\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "NydmJdQRoxuf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The accuracy is not improved by introducing bigrams"
      ]
    },
    {
      "metadata": {
        "id": "vgLzNdL0o3c_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Further improvements can be done by tuning alpha, ad extending the model to trigrams, getting more data"
      ]
    }
  ]
}