Jigsaw4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8EokSNxD3O7m"
   },
   "source": [
    "# Problem Statement.\n",
    "## Given a comment classify it into toxic or non-toxic. Here toxic means comments that are unacceptable to a person or a community.\n",
    "\n",
    "## This project was inspired from the jigsaw unintended bias toxic comment classification which was hosted on kaggle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H0p9huIa3086"
   },
   "source": [
    "# The problem that we are solving here.\n",
    "## Background : \n",
    "### This problem was also solved earlier but there was a problem with the system . The system was a bit biased towards the use of unparliamentary/abusive words.\n",
    "### For example a sentence \"I am a gay\" or \"I am a black man.\" were classified as toxic comments. \n",
    "### Therefore the challenge was to get rid of this bias. To solve this problem a new metric was introduced by jigsaw. This metric also considers the identity of the person about whom the comment is made. You can get into  the details of the metric on the below link.\n",
    "<a href=\"https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/overview/evaluation\">Click Here</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2WHF2FAsYj8L"
   },
   "source": [
    "# Exploratory  Data Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "id": "en1qQYZT29rn",
    "outputId": "5af387f5-aeff-4903-cd8e-add43a13a6a5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
      "\n",
      "Enter your authorization code:\n",
      "··········\n",
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4Qa8JRLoKqQ7"
   },
   "source": [
    "# Loading Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 462
    },
    "colab_type": "code",
    "id": "HtMXpvJcYj8N",
    "outputId": "4c5c7910-53b7-479c-c74a-68fda689d19b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>severe_toxicity</th>\n",
       "      <th>obscene</th>\n",
       "      <th>identity_attack</th>\n",
       "      <th>insult</th>\n",
       "      <th>threat</th>\n",
       "      <th>asian</th>\n",
       "      <th>atheist</th>\n",
       "      <th>bisexual</th>\n",
       "      <th>black</th>\n",
       "      <th>buddhist</th>\n",
       "      <th>christian</th>\n",
       "      <th>female</th>\n",
       "      <th>heterosexual</th>\n",
       "      <th>hindu</th>\n",
       "      <th>homosexual_gay_or_lesbian</th>\n",
       "      <th>intellectual_or_learning_disability</th>\n",
       "      <th>jewish</th>\n",
       "      <th>latino</th>\n",
       "      <th>male</th>\n",
       "      <th>muslim</th>\n",
       "      <th>other_disability</th>\n",
       "      <th>other_gender</th>\n",
       "      <th>other_race_or_ethnicity</th>\n",
       "      <th>other_religion</th>\n",
       "      <th>other_sexual_orientation</th>\n",
       "      <th>physical_disability</th>\n",
       "      <th>psychiatric_or_mental_illness</th>\n",
       "      <th>transgender</th>\n",
       "      <th>white</th>\n",
       "      <th>created_date</th>\n",
       "      <th>publication_id</th>\n",
       "      <th>parent_id</th>\n",
       "      <th>article_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>funny</th>\n",
       "      <th>wow</th>\n",
       "      <th>sad</th>\n",
       "      <th>likes</th>\n",
       "      <th>disagree</th>\n",
       "      <th>sexual_explicit</th>\n",
       "      <th>identity_annotator_count</th>\n",
       "      <th>toxicity_annotator_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59848</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>This is so cool. It's like, 'would you want yo...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-09-29 10:50:41.987077+00</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2006</td>\n",
       "      <td>rejected</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59849</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Thank you!! This would make my life a lot less...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-09-29 10:50:42.870083+00</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2006</td>\n",
       "      <td>rejected</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>59852</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>This is such an urgent design problem; kudos t...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-09-29 10:50:45.222647+00</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2006</td>\n",
       "      <td>rejected</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>59855</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Is this something I'll be able to install on m...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2015-09-29 10:50:47.601894+00</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2006</td>\n",
       "      <td>rejected</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59856</td>\n",
       "      <td>0.893617</td>\n",
       "      <td>haha you guys are a bunch of losers.</td>\n",
       "      <td>0.021277</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.021277</td>\n",
       "      <td>0.87234</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2015-09-29 10:50:48.488476+00</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2006</td>\n",
       "      <td>rejected</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id    target  ... identity_annotator_count  toxicity_annotator_count\n",
       "0  59848  0.000000  ...                        0                         4\n",
       "1  59849  0.000000  ...                        0                         4\n",
       "2  59852  0.000000  ...                        0                         4\n",
       "3  59855  0.000000  ...                        0                         4\n",
       "4  59856  0.893617  ...                        4                        47\n",
       "\n",
       "[5 rows x 45 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(\"/content/drive/My Drive/Project/train.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "QqDZlqmdYj8V",
    "outputId": "3ffe95e1-1e1c-4459-e004-fb3653bbb801"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1804874, 45)\n"
     ]
    }
   ],
   "source": [
    "#print(df.shape)\n",
    "#df = df.sample(500000)\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "4UwqAY0BwEDN",
    "outputId": "45e37635-ee0b-431b-b829-e28cb35edf83"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1780823, 45)"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop_duplicates(subset={'comment_text'},keep='first')\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zFY0EQn2Yj9K"
   },
   "source": [
    "### Looking the absuive words used in the comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Hwfe7u52Yj9M",
    "outputId": "0c3b4981-283b-4602-c05e-34febcec74ff"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1780823/1780823 [01:58<00:00, 15073.35it/s]\n"
     ]
    }
   ],
   "source": [
    "# Lets find the words that contains letters or character that are not present in english\n",
    "import re\n",
    "from tqdm import tqdm\n",
    "\n",
    "non_eng_words = {}\n",
    "pattern = re.compile(r'([fadpbit]{1,4}[*]{1,4}[ingkscyhabd]{1,4})')\n",
    "\n",
    "for sent in tqdm(df['comment_text'].values):\n",
    "    sent = re.sub(r'[^a-zA-Z* ]',' ',sent)\n",
    "    for wrd in sent.split():\n",
    "        if(re.match(pattern,wrd) and len(wrd)>2):\n",
    "            if(non_eng_words.get(wrd,-1)<0):\n",
    "                non_eng_words[wrd] = 1\n",
    "            else:\n",
    "                non_eng_words[wrd] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "Zm68pnCKYj9a",
    "outputId": "bb2c563b-2859-436a-a7f8-c3a11775b968"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of aabusive words wused are \n",
      "p***y -> 73\n",
      "p*ssy -> 65\n",
      "f**k -> 45\n",
      "a**hole -> 38\n",
      "f*ck -> 29\n",
      "a**holes -> 27\n",
      "p*ss -> 25\n",
      "f*cking -> 18\n",
      "f***ing -> 17\n",
      "f**king -> 15\n",
      "p*ssed -> 13\n",
      "p**sy -> 13\n",
      "a*s -> 10\n",
      "f**ked -> 10\n",
      "b***h -> 9\n",
      "f*cked -> 8\n",
      "d*ck -> 7\n",
      "p**s -> 7\n",
      "p*ssie -> 6\n",
      "p*ssing -> 6\n",
      "a**clown -> 6\n",
      "bit*h -> 5\n",
      "d**k -> 5\n",
      "b**ch -> 5\n",
      "f*cks -> 4\n",
      "p**y -> 4\n",
      "d****d -> 3\n",
      "p***ing -> 3\n",
      "pi*s -> 3\n",
      "p***ies -> 3\n",
      "p*nis -> 3\n",
      "b*itch -> 3\n",
      "d**n -> 3\n",
      "d**ks -> 3\n",
      "b*stards -> 3\n",
      "f**cker -> 3\n",
      "p****y -> 3\n",
      "a**clowns -> 3\n",
      "p****d -> 2\n",
      "pi**ing -> 2\n",
      "bi*ch -> 2\n",
      "a*sholes -> 2\n",
      "f**ks -> 2\n",
      "f*k -> 2\n",
      "a*se -> 2\n",
      "f***in -> 2\n",
      "f***n -> 2\n",
      "d*amn -> 2\n",
      "f**kin -> 2\n",
      "p*ss* -> 2\n",
      "p*ssies -> 2\n",
      "f****d -> 2\n",
      "f****ing -> 2\n",
      "a**hats -> 2\n",
      "b***s -> 2\n",
      "p*sses -> 2\n",
      "p*ss*ng -> 2\n",
      "f***king -> 2\n",
      "b*stard -> 2\n",
      "p*s*y -> 2\n",
      "f*ckers -> 2\n",
      "d*ckhead -> 2\n",
      "f****s -> 2\n",
      "a***holes -> 2\n",
      "a*holes -> 2\n",
      "t**d -> 2\n",
      "f*ckin -> 1\n",
      "b**ching -> 1\n",
      "a*a* -> 1\n",
      "f*c*ing -> 1\n",
      "bat*hit -> 1\n",
      "f***g -> 1\n",
      "d***s -> 1\n",
      "a*shole -> 1\n",
      "p****s -> 1\n",
      "b*sta*ds -> 1\n",
      "p**s* -> 1\n",
      "p****ies -> 1\n",
      "b*gg*r -> 1\n",
      "f*c*king -> 1\n",
      "f***s -> 1\n",
      "a*ss -> 1\n",
      "bat****crazy -> 1\n",
      "p*sspot -> 1\n",
      "p***ys -> 1\n",
      "ba**ards -> 1\n",
      "f**kc -> 1\n",
      "di*k -> 1\n",
      "f*ck*d -> 1\n",
      "b*alls -> 1\n",
      "di*ks -> 1\n",
      "f*c* -> 1\n",
      "b****h -> 1\n",
      "f*kcyouCanada -> 1\n",
      "t*g -> 1\n",
      "a**h*** -> 1\n",
      "f**ken -> 1\n",
      "f**g -> 1\n",
      "f**ing -> 1\n",
      "a**holery -> 1\n",
      "f**kem -> 1\n",
      "i*iot -> 1\n",
      "a**hat -> 1\n",
      "f**cking -> 1\n",
      "f**ching -> 1\n",
      "d*icks -> 1\n",
      "t*i*t -> 1\n",
      "a**h**e -> 1\n",
      "f***ked -> 1\n",
      "b***s**t -> 1\n",
      "b*st*rd -> 1\n",
      "b***ch -> 1\n",
      "p*as -> 1\n",
      "a****and -> 1\n",
      "f*ggots -> 1\n",
      "da*n -> 1\n",
      "a**h* -> 1\n",
      "f**cked -> 1\n",
      "t*h -> 1\n",
      "id*iot -> 1\n",
      "b****s -> 1\n",
      "bit*hes -> 1\n",
      "b***ches -> 1\n",
      "fa**in -> 1\n",
      "b**hes -> 1\n",
      "da*ned -> 1\n",
      "p**a -> 1\n",
      "t**s -> 1\n",
      "di*cks -> 1\n",
      "d**ned -> 1\n",
      "f*ckup -> 1\n",
      "a***hole -> 1\n",
      "p*sssy -> 1\n",
      "a*hole -> 1\n",
      "f***k -> 1\n",
      "p****grabber -> 1\n",
      "f***able -> 1\n",
      "bi**h -> 1\n",
      "a**ing -> 1\n",
      "d*do -> 1\n",
      "p**ck -> 1\n",
      "di**s -> 1\n",
      "f*ggot -> 1\n",
      "d*ke -> 1\n",
      "f*g -> 1\n"
     ]
    }
   ],
   "source": [
    "# Now we will store these words and their counts in a list.\n",
    "wrds = []\n",
    "counts = []\n",
    "for wrd in sorted(non_eng_words, key=non_eng_words.get, reverse=True):\n",
    "    if(len(wrd)<30):\n",
    "        wrds.append(wrd)\n",
    "        counts.append(non_eng_words[wrd])\n",
    "print(\"Number of aabusive words wused are \".format(len(non_eng_words)))\n",
    "        \n",
    "## Abusive words present in the comments.\n",
    "for i in range(len(wrds)):\n",
    "    print(wrds[i],\"->\",counts[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gWuYp_sL-kKB"
   },
   "source": [
    "### Data Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XqwmrvE-Yj9_"
   },
   "outputs": [],
   "source": [
    "\n",
    "# Ref: https://www.kaggle.com/haqishen/jigsaw-predict\n",
    "# We are creating a dict with shortened word as key and actual word as value.\n",
    "apostophe_dict = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\",\n",
    "                  \"couldn't\": \"could not\", \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\",\n",
    "                  \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\",\n",
    "                  \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",  \"I'd\": \"I would\",\n",
    "                  \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\",\n",
    "                  \"i'd\": \"i would\", \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\",\n",
    "                  \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \"it'll\": \"it will\",\n",
    "                  \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\",\n",
    "                  \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\",\n",
    "                  \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\n",
    "                  \"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\",\n",
    "                  \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\",\n",
    "                  \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\",\n",
    "                  \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\",\n",
    "                  \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\",\n",
    "                  \"there'd\": \"there would\", \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\n",
    "                  \"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\",\n",
    "                  \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\",\n",
    "                  \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\",\n",
    "                  \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\",\n",
    "                  \"what're\": \"what are\",  \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\",\n",
    "                  \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\",\n",
    "                  \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n",
    "                  \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\",\n",
    "                  \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\",\n",
    "                  \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\n",
    "                  \"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\",\n",
    "                  \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \"you're\": \"you are\",\n",
    "                  \"you've\": \"you have\" }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "IEeOYrHj9B7i",
    "outputId": "a62b6041-905d-4ab9-ecd3-5004bc79edd3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "from nltk.corpus import stopwords\n",
    "stopwords = list(stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "C7rKRNfa9CLe"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def process_sent(sent):\n",
    "    line = ''\n",
    "    for wrd in sent.split():\n",
    "        if(len(apostophe_dict.get(wrd.lower(),'n'))>1 and wrd.lower() not in stopwords and len(wrd)>2):\n",
    "            wrd = apostophe_dict[wrd.lower()]\n",
    "            n_wrd = ''\n",
    "            for w in wrd.split():\n",
    "                if(w not in stopwords):\n",
    "                    n_wrd += \" \"+w\n",
    "            line += \" \"+n_wrd\n",
    "        else:\n",
    "            line += \" \"+ wrd.lower()    \n",
    "    line = re.sub(r'[^a-zA-Z* ]',' ',line)\n",
    "    return line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "xAkmxr6Y-2or",
    "outputId": "69f18be2-237e-43ac-e588-2ec091f138de"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1780823/1780823 [01:52<00:00, 15806.98it/s]\n"
     ]
    }
   ],
   "source": [
    "# Preprocessing the comments.\n",
    "preprocessed_data = []\n",
    "from tqdm import tqdm\n",
    "for sent in tqdm(df['comment_text'].values):\n",
    "    sent = process_sent(sent)\n",
    "    line = ''\n",
    "    for wrd in sent.split():\n",
    "        if(len(wrd)>2):\n",
    "            line += \" \" +wrd.lower()\n",
    "    line = re.sub(r\"[']\",'',line)\n",
    "    preprocessed_data.append(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aBoBrIuz-2iN"
   },
   "outputs": [],
   "source": [
    "df['comment_text'] = preprocessed_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mHh3Vpfo_jji"
   },
   "source": [
    "### Splitting data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "kbd5sTCL7l0s",
    "outputId": "059a524a-f459-48dc-d8f3-75662e9911e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1442466, 45) (178083, 45)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "# To train and validate classical machine models use cv datset also. \n",
    "Y = df['target'].values\n",
    "train,test,Y_train,Y_test = train_test_split(df,Y,test_size=0.1,random_state=42)\n",
    "train,cv,Y_train,Y_cv = train_test_split(train,Y_train,test_size=0.1,random_state=42)\n",
    "print((train.shape),(test.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QBeovSv8GiZd"
   },
   "outputs": [],
   "source": [
    "identity_columns = [\n",
    "        'male', 'female', 'homosexual_gay_or_lesbian', 'christian', 'jewish',\n",
    "        'muslim', 'black', 'white', 'psychiatric_or_mental_illness']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dk9s2VwFHNqc"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "seed = 1029\n",
    "\n",
    "train_x = train['comment_text'].fillna('_##_').values\n",
    "test_x = test['comment_text'].fillna('_##_').values\n",
    "cv_x = cv['comment_text'].fillna('_##_').values\n",
    "\n",
    "# For gtrain\n",
    "weights = np.ones((len(train),))\n",
    "weights += train[identity_columns].fillna(0).values.sum(axis=1) * 3\n",
    "weights += train['target'].values * 8\n",
    "weights /= weights.max()\n",
    "train_y = np.vstack([train['target'], weights]).T   \n",
    "train_y_identity = train[identity_columns].values\n",
    "\n",
    "# For test\n",
    "weights = np.ones((len(test),))\n",
    "weights += test[identity_columns].fillna(0).values.sum(axis=1) * 3\n",
    "weights += test['target'].values * 8\n",
    "weights /= weights.max()\n",
    "test_y = np.vstack([test['target'], weights]).T   \n",
    "test_y_identity = test[identity_columns].values\n",
    "\n",
    "# For cv\n",
    "weights = np.ones((len(cv),))\n",
    "weights += cv[identity_columns].fillna(0).values.sum(axis=1) * 3\n",
    "weights += cv['target'].values * 8\n",
    "weights /= weights.max()\n",
    "cv_y = np.vstack([cv['target'], weights]).T    \n",
    "cv_y_identity = cv[identity_columns].values\n",
    "\n",
    "# shuffling the data\n",
    "np.random.seed(seed)\n",
    "train_idx = np.random.permutation(len(train_x))\n",
    "test_idx = np.random.permutation(len(test_x))\n",
    "cv_idx = np.random.permutation(len(cv_x))\n",
    "\n",
    "train_x = train_x[train_idx]\n",
    "train_y = train_y[train_idx]\n",
    "train_y_identity = train_y_identity[train_idx]\n",
    "\n",
    "\n",
    "test_x = test_x[test_idx]\n",
    "test_y = test_y[test_idx]\n",
    "test_y_identity = test_y_identity[test_idx]\n",
    "\n",
    "\n",
    "cv_x = cv_x[cv_idx]\n",
    "cv_y = cv_y[cv_idx]\n",
    "cv_y_identity = cv_y_identity[cv_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "gVMR8tQVSRNE",
    "outputId": "6a32eab2-e30e-4ef6-8bf8-e6b0ed7bc8e9"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in greater_equal\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(178083, 9)"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_binary = (test_y[:, 0] >= 0.5).astype(int)\n",
    "y_identity_binary = (test_y_identity >= 0.5).astype(int)\n",
    "y_identity_binary.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "1R-S5srPCmTS",
    "outputId": "e66c18f2-7e91-40f0-aea4-d74c815b0902"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1442466,) (160274,)\n"
     ]
    }
   ],
   "source": [
    "Y_train = (train_y[:,0]>=0.5).astype(int)\n",
    "Y_cv = (cv_y[:,0]>=0.5).astype(int)\n",
    "Y_test = (test_y[:,0]>=0.5).astype(int)\n",
    "print(Y_train.shape, Y_cv.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fZlZwaRiI4Zf"
   },
   "source": [
    "### Custom metric given by *Kaggle*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "nmbsUXNavCwt",
    "outputId": "fbe93b10-8088-4365-d5e9-4436a06c1786"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# This metric code below was taken from kaggle \n",
    "from sklearn.metrics import roc_auc_score\n",
    "import keras.backend as K\n",
    "class JigsawEvaluator:\n",
    "    \n",
    "    def __init__(self, y_binary, y_identity_binary, power=-5, overall_model_weight=0.25):\n",
    "        self.y = y_binary\n",
    "        self.y_i = y_identity_binary\n",
    "        self.n_subgroups = self.y_i.shape[1]\n",
    "        self.power = power\n",
    "        self.overall_model_weight = overall_model_weight\n",
    "        \n",
    "    @staticmethod\n",
    "    def _compute_auc(y_true, y_pred):\n",
    "        try:\n",
    "            return roc_auc_score(y_true, y_pred)\n",
    "        except ValueError:\n",
    "            return np.nan\n",
    "        \n",
    "    def _compute_subgroup_auc(self, i, y_pred):\n",
    "        mask = self.y_i[:, i] == 1\n",
    "        return self._compute_auc(self.y[mask], y_pred[mask])\n",
    "        \n",
    "    def _compute_bpsn_auc(self, i, y_pred):\n",
    "        mask = self.y_i[:, i] + self.y == 1\n",
    "        return self._compute_auc(self.y[mask], y_pred[mask])\n",
    "        \n",
    "    def _compute_bnsp_auc(self, i, y_pred):\n",
    "        mask = self.y_i[:, i] + self.y != 1\n",
    "        return self._compute_auc(self.y[mask], y_pred[mask])\n",
    "      \n",
    "    def compute_bias_metrics_for_model(self, y_pred):\n",
    "        records = np.zeros((3, self.n_subgroups))\n",
    "        for i in range(self.n_subgroups):\n",
    "            records[0, i] = self._compute_subgroup_auc(i, y_pred)\n",
    "            records[1, i] = self._compute_bpsn_auc(i, y_pred)\n",
    "            records[2, i] = self._compute_bnsp_auc(i, y_pred)\n",
    "        return records\n",
    "        \n",
    "    def _calculate_overall_auc(self, y_pred):\n",
    "        return roc_auc_score(self.y, y_pred)\n",
    "        \n",
    "    def _power_mean(self, array):\n",
    "        total = sum(np.power(array, self.power))\n",
    "        return np.power(total / len(array), 1 / self.power)\n",
    "        \n",
    "    def get_final_metric(self, y_pred):\n",
    "        #y_pred = K.flatten(y_pred)\n",
    "        bias_metrics = self.compute_bias_metrics_for_model(y_pred)\n",
    "        bias_score = np.average([\n",
    "            self._power_mean(bias_metrics[0]),\n",
    "            self._power_mean(bias_metrics[1]),\n",
    "            self._power_mean(bias_metrics[2])\n",
    "        ])\n",
    "        overall_score = self.overall_model_weight * self._calculate_overall_auc(y_pred)\n",
    "        bias_score = (1 - self.overall_model_weight) * bias_score\n",
    "        return overall_score + bias_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0ukVSuPsymiH"
   },
   "outputs": [],
   "source": [
    "def custom_loss(y_true, y_pred):\n",
    "    return binary_crossentropy(K.reshape(y_true[:,0],(-1,1)), y_pred) * y_true[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cMIWJJa1PYRA"
   },
   "source": [
    "## Vectorizing datset using tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qeNjR0XTVYK6"
   },
   "source": [
    "#### vectorizing train dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dCavHB4HUe3E"
   },
   "outputs": [],
   "source": [
    "# target with value greater than equal to 0.5 will be assigned 1 and rest 0.\n",
    "def fun(x):\n",
    "    if(x>=0.5):\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4rh7cbCMUfBH"
   },
   "outputs": [],
   "source": [
    "labels = df['target']\n",
    "Y = labels.map(fun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "n5AS38gZYj_A",
    "outputId": "92e3eb15-3a6c-4b87-862d-18e9a3dae905"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1282192 356165 142466\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train,X_test,Y_train,Y_test = train_test_split(preprocessed_data,Y,test_size=0.2,random_state=42)\n",
    "X_train,X_cv,Y_train,Y_cv = train_test_split(X_train,Y_train,test_size=0.1,random_state=42)\n",
    "print(len(X_train),len(X_test),len(X_cv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1vEFdbPoU3wl"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer(min_df=5, max_features=50000)\n",
    "train_vect1 = vectorizer.fit_transform(X_train)\n",
    "test_vect1 = vectorizer.transform(X_test)\n",
    "cv_vect1 = vectorizer.transform(X_cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "g_BfZFUdYj_F",
    "outputId": "ba43e748-f71c-4951-e85f-2fc9ffc9cbe2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1282192, 50000)\n",
      "(356165, 50000)\n",
      "(142466, 50000)\n"
     ]
    }
   ],
   "source": [
    "print(train_vect1.shape)\n",
    "print(test_vect1.shape)\n",
    "print(cv_vect1.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DJlRT7XxVcdf"
   },
   "source": [
    "#### vectorizing test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "9qXCQhK0VdHp",
    "outputId": "c50ce8d8-da96-4f68-8ce0-353b92bc8597"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7097320</td>\n",
       "      <td>[ Integrity means that you pay your debts.]\\n\\...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7097321</td>\n",
       "      <td>This is malfeasance by the Administrator and t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7097322</td>\n",
       "      <td>@Rmiller101 - Spoken like a true elitist. But ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7097323</td>\n",
       "      <td>Paul: Thank you for your kind words.  I do, in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7097324</td>\n",
       "      <td>Sorry you missed high school. Eisenhower sent ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                       comment_text\n",
       "0  7097320  [ Integrity means that you pay your debts.]\\n\\...\n",
       "1  7097321  This is malfeasance by the Administrator and t...\n",
       "2  7097322  @Rmiller101 - Spoken like a true elitist. But ...\n",
       "3  7097323  Paul: Thank you for your kind words.  I do, in...\n",
       "4  7097324  Sorry you missed high school. Eisenhower sent ..."
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.read_csv(\"/content/drive/My Drive/Project/test.csv\")\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "gaFdiBUIVdEH",
    "outputId": "49fd9a36-9e44-4128-ea80-ff03da64b5b0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 97320/97320 [00:05<00:00, 16720.14it/s]\n"
     ]
    }
   ],
   "source": [
    "# Preprocessing the comments.\n",
    "preprocessed_data_test = []\n",
    "from tqdm import tqdm\n",
    "for sent in tqdm(test_df['comment_text'].values):\n",
    "    sent = process_sent(sent)\n",
    "    line = ''\n",
    "    for wrd in sent.split():\n",
    "        if(len(wrd)>2):\n",
    "            line += \" \" +wrd.lower()\n",
    "    #line = re.sub(r\"[']\",'',line)\n",
    "    preprocessed_data_test.append(line)\n",
    "    \n",
    "# Preparing the test data\n",
    "test_X = vectorizer.transform(preprocessed_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eCbzpZf8Yj_W"
   },
   "source": [
    "# Training Logistic Regression on tfidf vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p6ljlSUAScUg"
   },
   "source": [
    "### On word unigrams "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "iIS0fQ5uYj_X",
    "outputId": "e0d9669e-8abb-42f7-a00d-0bd8417481d3"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "  0%|          | 0/10 [00:00<?, ?it/s]\u001b[A\n",
      " 10%|█         | 1/10 [06:34<59:07, 394.17s/it]\u001b[A\n",
      " 20%|██        | 2/10 [09:16<43:17, 324.68s/it]\u001b[A\n",
      " 30%|███       | 3/10 [10:48<29:44, 254.91s/it]\u001b[A\n",
      " 40%|████      | 4/10 [11:52<19:46, 197.69s/it]\u001b[A\n",
      " 50%|█████     | 5/10 [12:39<12:41, 152.30s/it]\u001b[A\n",
      " 60%|██████    | 6/10 [13:24<08:00, 120.14s/it]\u001b[A\n",
      " 70%|███████   | 7/10 [13:57<04:42, 94.05s/it] \u001b[A\n",
      " 80%|████████  | 8/10 [14:30<02:31, 75.80s/it]\u001b[A\n",
      " 90%|█████████ | 9/10 [15:05<01:03, 63.53s/it]\u001b[A\n",
      "100%|██████████| 10/10 [15:36<00:00, 53.68s/it]\u001b[A\n",
      "\u001b[A"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.calibration import CalibratedClassifierCV\n",
    "from sklearn.metrics import roc_auc_score\n",
    "params = [0.0000001,0.000001,0.00001,0.0001,0.001,0.01,0.1,1.0,10,100]\n",
    "train_auc = []\n",
    "cv_auc = []\n",
    "for i in tqdm(params):\n",
    "    lr = SGDClassifier(loss='log',penalty='l2',tol=0.0001,alpha=i,n_jobs=-1,class_weight='balanced')\n",
    "    model = CalibratedClassifierCV(lr,cv=5)\n",
    "    model.fit(train_vect1,Y_train)\n",
    "    predict_y_train = model.predict_proba(train_vect1)[:,1] # Taking probability for positive class\n",
    "    t_auc = roc_auc_score(Y_train,predict_y_train)\n",
    "    predict_y_cv = model.predict_proba(cv_vect1)[:,1]# Taking probability for positve class\n",
    "    c_auc = roc_auc_score(Y_cv,predict_y_cv)\n",
    "    train_auc.append(t_auc)\n",
    "    cv_auc.append(c_auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 283
    },
    "colab_type": "code",
    "id": "rmjtJl98Yj_b",
    "outputId": "63d312fa-bb76-4541-e78b-ab4c81563bf0"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4FWX2wPHvSYckhBQIJZRIh1AC\nEVBQKRbEFRVQYEUXXWVVdG1bcIuyrv50FevqYkVFUUQQRETZVYmIitI7SO9IhyRAQpLz+2MmyU2A\n5EJyc1PO53nmuXOnnnmN9zDzvvO+oqoYY4wx5yrA3wEYY4yp3CyRGGOMKRVLJMYYY0rFEokxxphS\nsURijDGmVCyRGGOMKRVLJMYYY0rFEokxxphSsURijDGmVIL8HUB5iIuL06ZNm/o7jFLJyMggPDzc\n32FUCFYWhVl5FGblUaC0ZbFo0aL9qlqnpO18mkhEpB/wAhAIvKGqTxZZ3wQYD9QBDgLDVXWHiPQG\nnvPYtDUwVFWni8jbwCXAEXfdCFVdWlwcTZs2ZeHChWVxSX6TmppKr169/B1GhWBlUZiVR2FWHgVK\nWxYistWb7XyWSEQkEHgZuAzYASwQkRmqutpjs7HABFV9R0T6AE8AN6nqHKCTe5wYYAPwX4/9/qiq\nU3wVuzHGGO/5so6kK7BBVTepahYwCbimyDZtga/d+TmnWQ8wGPhcVY/5LFJjjDHnzJeJpCGw3eP7\nDneZp2XAQHf+OiBSRGKLbDMU+KDIssdFZLmIPCcioWUVsDHGmLMnvupGXkQGA/1U9Tb3+01AN1W9\n22ObBsBLQCIwFxgEJKnqYXd9fWA50EBVT3os2wOEAK8BG1X10dOcfyQwEiA+Pr7LpEmTfHKd5SU9\nPZ2IiAh/h1EhWFkUVpHLQ0QIDw8nMDCw3M6pqohIuZ2vIvO2LHJycsjIyKBoPujdu/ciVU0paX9f\nVrbvBBp5fE9wl+VT1V24dyQiEgEMyksirhuAaXlJxN1ntzubKSJvAX843clV9TWcRENKSopW9so3\nq0AsYGVRWEUuj82bNxMZGUlsbGy5/binpaURGRlZLueq6LwpC1XlwIEDpKWlkZiYeE7n8eWjrQVA\nCxFJFJEQnEdUMzw3EJE4EcmL4SGcFlyehlHksZZ7R4I4f5XXAit9ELsxpgycOHGiXJOIOXsiQmxs\nLCdOnDjnY/gskahqNnA3MBtYA0xW1VUi8qiIDHA36wWsE5GfgXjg8bz9RaQpzh3NN0UOPVFEVgAr\ngDjgMV9dgzGm9CyJVHyl/W/k0/dIVHUWMKvIsoc95qcAp23Gq6pbOLVyHlXtU7ZRntnHi3dw6NhJ\n2jWoRdsGtagVFlxepzbGmEqjWrzZfq4+W76br9buzf/eKKYG7epH0bZBrfzkUq9WmP2Ly5gK6vDh\nw7z//vvcddddZ71v//79ef/996ldu7YPIqtaLJEU480R57M37QSrdx1l1a6jrN59lNW7jvLFqj35\n28SEh9C2fkFiadegFolxEQQGWHIxxt8OHz7Mf/7zn9MmkuzsbIKCzvwTOGvWrDOuM4VZIilB3cgw\n6rYKo1eruvnL0jOzWbvbSSyrdjqfb323haycXADCggNoXa8gsbRrEEWr+EhqhJRfE0hjDIwePZqN\nGzfSqVMnLrvsMq666ir+/ve/Ex0dzdq1a/n555+59tpr2b59OydOnODee+9l5MiRQEHXSunp6Vx5\n5ZX07NmT77//noYNG/LJJ59Qo0aNQuf69NNPeeyxx8jKyiI2NpaJEycSHx/PmDFjiIiI4A9/cBqY\nJiUlMXPmTJo2bcqECRMYO3YsIkKHDh149913y72MyoIlknMQERpEStMYUprG5C87mZPLhr3pHncv\nR/h02S7e/3EbAAECzepEFDwWqx9Fuwa1iA4P8ddlGFOu/vHpKlbvOlqmx2zboBaPXN3ujOuffPJJ\nVq5cydKlTnd8qampLF68mJUrV+Y3dR0/fjwxMTEcP36c888/n0GDBhEbW/i96PXr1/PBBx/w+uuv\nc8MNNzB16lSGDx9eaJuePXsyf/58RIQ33niDp556imeeeeaMsa1atYrHHnuM77//nri4OA4ePHiu\nxeB3lkjKSHBgAG3q16JN/VoM6uIsU1V2HDru8VjsCD9tPsgnS3fl71c/KsxNLLVo28BJLgnRNaze\nxRgf6dq1a6H3JV588UWmTZsGwPbt21m/fv0piSQxMZFOnToB0KVLF7Zs2XLKcXfs2MGQIUPYvXs3\nWVlZJb6T8fXXX3P99dcTFxcHQExMTLHbV2SWSHxIRGgUU5NGMTXpl1Qvf/nBjCxWu3ctq3Y59S5f\nr91LrvtSaWRYkFvvUlCxn+ujHgiMKS/F3TmUJ89u1VNTU/nyyy/54YcfqFmzJr169Trt+xShoQU9\nMQUGBnL8+PFTtrnnnnt44IEHGDBgAKmpqYwZMwaAoKAgcnNz87crzfsaFZUlEj+ICQ+hZ4s4eraI\ny192PCuHdb+ksWrXkfzHY+//tJUTJ50/wDo1hPtqbmVQ5wTCgq2uxRhvREZGkpaWdsb1R44cITo6\nmpo1a7J27Vrmz59/zuc6cuQIDRs6byy88847+cubNm3KzJkzAVi8eDGbN28GoE+fPlx33XU88MAD\nxMbGcvDgwUp7V2KJpIKoERJIp0a16dSooKlhTq6yeX86S7cf4T//XcFfp63kuf+t57c9E7mxe2N7\nr8WYEsTGxtKjRw+SkpK48sorueqqqwqt79evH6+88gpt2rShVatWdO/e/ZzPNWbMGK6//nqio6Pp\n06dPfsIYNGgQEyZMoF27dnTr1o2WLVsC0K5dO/76179yySWXEBgYSHJyMm+//fY5n9+ffNZpY0WS\nkpKilX1gqzlz5hDauD3jUjfy7fr9RIYGcdMFTbilRyJ1IqtXB8gVuW8pf6jI5bFmzRratGlTrue0\nvrYKnE1ZnO6/lYj4vdNGU4ZEhAubxXFhszhW7jzCuNSNjPtmI2/M28wNKQmMvKgZjWNr+jtMY0w1\nZImkEkpqGMXLN3Zm8/4MXv1mIx8u2M77P27j6o4NuOOSZrSpX8vfIRpjqhFf9v5rfCwxLpwnB3Xg\n2z/14baLzuPL1b9w5QvfcstbP7FgS+Vtk26MqVwskVQB9aLC+Ev/Nnw/ui8PXtaSZTuOcP0rPzB4\n3Pd8teaXUwarMcaYsmSJpAqJqhnMPX1b8N2f+/CPAe3YfeQEv31nIf2e/5bpS3aSnZNb8kGMMeYs\nWSKpgmqEBPKbC5uS+sdePHtDRxTlvg+X0mtsKhN+2MKJkzn+DtEYU4VYIqnCggMDGNg5gS/uvZg3\nbk6hbmQoD3+yih5Pfs3LczZw5PjJkg9iTDUTEREBwK5duxg8ePBpt+nVqxclvVLw/PPPc+zYsfzv\n/fv35/Dhw8XsUXlZIqkGAgKES9vGM/XOC/lwZHfaJ0Tx9Ox19Hjya56YtYa9R6telw3GlFaDBg2Y\nMuW04+55pWgimTVrVpUd28QSSTUiInQ7L5a3b+nKZ7/vSe/WdXn92030/NccHvp4BVv2Z/g7RGPK\n1OjRo3n55Zfzv48ZM4axY8eSnp5O37596dy5M+3bt+eTTz45Zd8tW7aQlJQEwPHjxxk6dCht2rTh\nuuuuK9TX1p133klKSgrt2rXjkUceAZyOIHft2kXv3r3p3bs34HSVsn//fgCeffZZkpKSSEpK4vnn\nn88/X5s2bbj99ttp164dl19++Wn79Pr000/p1q0bycnJXHrppfzyyy+Fri1PUlISW7duBWDChAl0\n6NCBjh07ctNNN517gZ6BT98jEZF+wAtAIPCGqj5ZZH0TYDxQBzgIDFfVHe66HJxx2QG2qeoAd3ki\nMAmIBRYBN6lqli+voypq1yCKfw9L5g+Xt+S1uZv4aNEOPlywjf7t63PHJc1Iahjl7xBNVfP5aNiz\nouTtzka99nDlk2dcPWTIEO677z5GjRoFwOTJk5k9ezZhYWFMmzaNWrVqsX//frp3786AAQPO2Ov2\nuHHjqFmzJmvWrGH58uV07tw5f93jjz9OTEwMOTk59O3bl+XLl/P73/+eZ599ljlz5uT37ptn0aJF\nvPXWW/z444+oKt26deOSSy4hOjq60nZX77M7EhEJBF4GrgTaAsNEpG2RzcYCE1S1A/Ao8ITHuuOq\n2smdBngs/xfwnKo2Bw4Bv/XVNVQHTWLDefy69sz7U29GXtyM1HX7+NW/53Hz+J/4YeMBazpsKrXk\n5GT27t3Lrl27WLZsGdHR0TRq1AhV5S9/+QsdOnTg0ksvZefOnfn/sj+duXPn5v+gd+jQgQ4dOuSv\nmzx5Mp07dyY5OZlVq1axevXqYmOaN28e1113HeHh4URERDBw4EC+/fZbwPvu6q+44grat2/P008/\nzapVq4o9X3l0V+/LO5KuwAZV3QQgIpOAawDPUm4LPODOzwGmF3dAcf650Af4tbvoHWAMMK7Moq6m\n6tYKY/SVrbmzVzMm/riV8fM2M+z1+SQ3rs2dlzTj0jbxBNjwwaY0irlz8KXrr7+eKVOmsGfPHoYM\nGQLAxIkT2bdvH4sWLSI4OJimTZueU/fumzdvZuzYsSxYsIDo6GhGjBhRqm7iK2t39b6sI2kIbPf4\nvsNd5mkZMNCdvw6IFJG8EWXCRGShiMwXkWvdZbHAYVXNLuaYphSiagRzV6/mzPtzH/55bRL70zMZ\n+e4iLn9+LlMW7SAn1+5QTOUyZMgQJk2axJQpU7j++usBp8v3unXrEhwczJw5c/LrEs7k4osv5v33\n3wdg5cqVLF++HICjR48SHh5OVFQUv/zyC59//nn+Pmfqwv6iiy5i+vTpHDt2jIyMDKZNm8ZFF13k\n9fUU11394sWLgVO7q//oo484cOAAgE8ebfm7r60/AC+JyAhgLrATyHvJoYmq7hSR84CvRWQFcMTb\nA4vISGAkQHx8PKmpqWUZd7lLT08v92toBIxJEX7aE8pnmzL4w0fLmPbdKm5rH+LXERz9URYVWUUu\nj6ioqGLHA/GFnJycQuds3LgxR44coV69ekRERJCWlsY111zDDTfcQLt27UhOTqZly5akp6fn75eW\nlkZ6ejq5ubmkpaUxfPhw7rzzTlq1akWrVq3o1KkTGRkZdO7cmaSkJFq2bElCQgLdunXjxIkTpKWl\ncfPNN3P55ZdTv359PvvsM1SV9PR0WrRowbBhw0hJcTrVvfnmm2nevDlbt27NPx9AZmYmmZmZp5Tf\nn//8ZwYPHkzt2rW5+OKL86/38ssvZ/z48bRp04aUlBSaN29Obm4uiYmJPPDAA1x00UUEBgbSoUMH\nXnnllVPK7cSJE+f8d+SzbuRF5AJgjKpe4X5/CEBVnzjD9hHAWlVNOM26t4GZwFRgH1BPVbOLnuNM\nKmU38qpweCts/wl2LGDrngM06X4tNEiGqAQo5x9yVeW5//3Mi19v4MHLWnJP3xblen5PFbnbdH+o\nyOVh3cj7V1XoRn4B0MJtZbUTGEpB3QYAIhIHHFTVXOAhnBZciEg0cExVM91tegBPqaqKyBxgME7L\nrd8Ap7bbq4yyM2H3ctg+H7b/6CSQdLfyLzicRtknYNtU53t4HWjQGRp2dj4bJENEHZ+GJyLcf1lL\nth86zjP/+5nGsTW5ppM9VTTG+DCRuHcMdwOzcZr/jlfVVSLyKLBQVWcAvYAnRERxHm2NcndvA7wq\nIrk49ThPqmpeJf2fgUki8hiwBHjTV9fgU+l7nWSRlzR2LYGcTGdd7SZwXi9o1BUadYO6bZk35ysu\nbhULuxbDzsXO5/r/Au4dZVQjJ6HkJ5dOEFa2TXhFhCcHtWfn4eP8ccpyEqJr0KVJ5Rwa1BhTdnxa\nR6Kqs4BZRZY97DE/BTjl1VFV/R5of4ZjbsJpEVZ55ObAvrUFSWPbfDjkVIQRGAL1O0HX252k0agr\nRNY79RCBIZDQxZnyZKbD7mWFk8uaGQXrY5t73LkkQ70OEFK6wa9CgwJ5dXgXBo77ntsnLGLaXRfS\nJDa8VMc0VZuq+rVOzZSstFUc/q5sr5pOHIWdi9zE8SPsWAiZR5114XWchJFyCzTqDvU7QnDYuZ0n\nNAKa9nCmPMcOOnc3uxbDziWw5VtYMdlZJ4FQt43HnUsy1G0HQSFnddro8BDGjzif6/7zHbe8vYBp\nd/YgqqaNH29OFRYWxoEDB4iNjbVkUkGpKgcOHCAs7Bx/h7BEUnqqcGhL4cdUe1eB5gIC8e2g/eCC\nu43oRN9WlNeMgeZ9nSnP0d0eyWUxrJ0JS9511gWGQr2kwncucS0hILDY0yTGhfPq8C4Mf/NHfvfe\nQibc2o2QIOtxxxSWkJDAjh072LdvX7md88SJE6X6UaxKvC2LsLAwEhJOaefkNUskZys703mclHe3\n4VkpHhIJCSlw8Z+cpJGQUub1FOekVn1nat3f+Z6X/DzvXJZ9AAted9aHRDh3Sg2SC+5eTpMAu50X\ny1ODO3D/h8v4y7QVPD24g/2r0xQSHBxMYmJiuZ4zNTWV5OTkcj1nRVVeZWGJpCTpewsnjV1LIMft\n2iu6qUeleHfnsVEJ/5KvEEQgJtGZktz3QXNzYP/6wncuP71e0ACgRrRz19L3705ycV2XnMCW/cd4\n4av1JMaFM6p3cz9ckDHGnyyRFGfSjc5jIHAqxRskQ7ffOY+pErpCZLx/4ytLAYFQt7UzdRrmLMvO\ngr2rC5LLz7Ph3evgls+dpOm679IWbD2QwdOz19E4piZXd2zgp4swxviDJZLitOzn1m10K12leGUV\nFOI0I27QCbgFDm6G8f2cZHLrF84dGU6z4H8N7sDOw8d58KNlNKhdgy5Nov0aujGm/FjtaHE63wQ9\nfg+Nu1W/JHI6MYlw0zQ4eRwmXAtpe/JXhQYF8upNKdSPCmPkhIVsO3CsmAMZY6oSSyTm7MS3hRun\nOHVH7w6E44fyV8WEh/DWiPPJzlVuefsnjhyzoXyNqQ4skZiz1+h8GDoRDqyHiTdAVsHIiufVieDV\nm7qw7eAx7py4iKzs3GIOZIypCiyRmHPTrDcMehN2LoQPhzvNol3dz4vlyYEd+H7jAf42fYUNjmVM\nFWeJxJy7tgPg6hdh49fw8UinCbFrUJcEft+nOZMX7mDcNxv9GKQxxtes1ZYpnc43wYkj8N+/wsxa\nTmJxX0q8/7KWbDlwjKe+WEeTmHCu6lDfz8EaY3zBEokpvQvvdirdvx3rvLh42aOA0yz4qcEd2HX4\nOA9MXkr92mF0bmzNgo2pauzRlikbff4GKb+F716Aec/lLw4LDuTVm7oQX8tpFrz9oDULNqaqsURi\nyoYI9B8LSYPhyzGw8K38VbERoYwfcT5Z2bnc+vYCjhy3ZsHGVCWWSEzZCQiA616BFpfDzPth5dT8\nVc3rRvDKTV3YvD+DuyYu4mSONQs2pqqwRGLKVmAwXP8ONL4APv4drP8yf9WFzeJ4YmB7vttwgL9P\nX2nNgo2pIiyRmLIXUhN+PcnpAPLD4c6IkK7rUxpxd+/mTFqwnVfnbvJjkMaYsuLTRCIi/URknYhs\nEJHRp1nfRES+EpHlIpIqIgnu8k4i8oOIrHLXDfHY520R2SwiS92pky+vwZyjsCgYPg2iGjpvv+9Z\nkb/qgcta8qsO9Xny87V8vmK3H4M0xpQFnyUSEQkEXgauBNoCw0SkbZHNxgITVLUD8CjwhLv8GHCz\nqrYD+gHPi0htj/3+qKqd3Gmpr67BlFJEHbhpujMk8LsD4YDzYmJAgDD2+o50blyb+z5cytLth/0c\nqDGmNHx5R9IV2KCqm1Q1C5gEXFNkm7bA1+78nLz1qvqzqq5353cBe4E6PozV+ErtRk4y0Rynx+Cj\nuwCnWfDrN6dQt1Yot72zkB2HrFmwMZWVLxNJQ2C7x/cd7jJPywB3iD6uAyJFJNZzAxHpCoQAnv1s\nPO4+8npORELLNmxT5uq0hOFTnZcWJ1wLGQcAp1nwWyPOJzM7h1vfXsDRE9Ys2JjKSLxpOSMiVwHt\ngPxBOVT10RL2GQz0U9Xb3O83Ad1U9W6PbRoALwGJwFxgEJCkqofd9fWBVOA3qjrfY9kenOTyGrDx\ndLGIyEhgJEB8fHyXSZMmlXidFVl6ejoRERH+DqNUog6vpOOyMaRHNGVZx0fJCaoJwOoDOTyz8ASt\nYwK4v0sYQQHFj/teFcqiLFl5FGblUaC0ZdG7d+9FqppS4oaqWuwEvAJMwLm7eARYAbzpxX4XALM9\nvj8EPFTM9hHADo/vtYDFwOBi9ukFzCwpli5dumhlN2fOHH+HUDbWzlIdE6361lWqWcfzF3+4YJs2\n+fNMHT11uebm5hZ7iCpTFmXEyqMwK48CpS0LYKGW8Puqql492rpQVW8GDqnqP9wE0dKL/RYALUQk\nUURCgKHADM8NRCRORPJieAgY7y4PAabhVMRPKbJPffdTgGuBlV7EYiqKVlfCteNgy7cw5VbIyQbg\nhpRG3NWrGR/8tI3Xv7VmwcZUJt4kkuPu5zH3UdRJoMRuXFU1G7gbmA2sASar6ioReVREBrib9QLW\nicjPQDzwuLv8BuBiYMRpmvlOFJEVOHdGccBjXlyDqUg6DoErn4Z1n8GMeyDXecv9D5e34qr29Xni\n87V8sXJPCQcxxlQU3vT+O9Ntevs0zqMmBd7w5uCqOguYVWTZwx7zU4App9nvPeC9MxyzjzfnNhVc\nt5Fw4jDMedx556TfEwQECM/c0JFdR45z34dL+DDqAjo2ql3ysYwxflXiHYmq/lNVD6vqVKAJ0FpV\n/+770EyVd/Efoftd8OM4+OYpoKBZcFxEKLdNWMjOw8dLOIgxxt/OeEciIgOLWYeqfuybkEy1IQKX\nPw7HD0Pq/0GN2tDtd8S5zYIHjvueW99awJQ7LyAyLNjf0RpjzqC4O5Kr3em3wJvAje70BnCr70Mz\n1UJAAAz4N7T+FXz+J1j2IQAt4iN5ZXgXNu5LZ9T7S8i23oKNqbDOmEhU9RZVvQUIBtqq6iBVHYTz\nPon989CUncAgGPQmJF4M0++EdZ8D0KN5HI9fl8Tcn/fxyIxV1luwMRWUN622GqmqZ896vwCNfRSP\nqa6Cw2Do+1C/I0z+DWz+FoAh5zfmjkuaMfHHbbw5b7OfgzTGnI43ieQrEZktIiNEZATwGfBlCfsY\nc/ZCI52uVGIS4YNhsHMxAH+6ohX929fj8VlrmL3KmgUbU9F402rrbpy32zu602uqeo+vAzPVVM0Y\nuGka1IiG9wbBvnUEBAjP3tCJDgm1uW/SUrYcyfF3lMYYD9522vgDTp9XX7vzxvhOrQZw83QICIJ3\nr4PD2wgLDuSNm1OICQ/hucWZ7EvL9HeUxhhXiYlERG4DfsLpnXcwMF9ErNWW8a3YZs6dSVa602Nw\n+j7qRIby5ogUjmYqb8yzblSMqSi8uSP5I5CsqiNU9TdAF+DPvg3LGKBeEvz6I2cMk/eugxNHaF2v\nFl3rBTJx/jaOHLdu542pCLxJJAeANI/vae4yY3yvcTcY+h7sXQvvD4GsY1x1XjDpmdm8N3+rv6Mz\nxlBMIhGRB0TkAWAD8KOIjBGRR4D5wM/lFaAxNL8UBr4G2+bD5JtpEpFLr1Z1GD9vM8ezrOLdGH8r\n7o4k0p02AtNxOmsE+ASwBv2mfCUNhKufhw3/o/XaF7jrkvM4kJHFR4u2l7yvMcanztjXljv2iDEV\nR5cRcOwg8V/9g7pH/0eXJk149ZtNDOvamOBAX44abYwpjjettlJEZJqILHbHSV8uIsvLIzhjTtHj\nPtLDmyDfPc+dFyey8/BxZi7f5e+ojKnWvPln3ETgLZzx1K/2mIwpfwEBbGs8CPatpU/AIlrFRzIu\ndSO5udYPlzH+4k0i2aeqM1R1s6puzZt8HpkxZ7CvTk+IbkrAvOe445JEfv4lna/X7vV3WMZUW94k\nkkdE5A0RGSYiA/Mmn0dmzBloQCBc+HvYuZABUZtIiK7Bf1I3WO/AxviJN4nkFqAT0I+Cx1q/8ubg\nItJPRNaJyAYRGX2a9U1E5Cu33iVVRBI81v1GRNa70288lncRkRXuMV8UEfEmFlPFdLoRIuIJ/O5Z\nRl58Hou3HeanzQf9HZUx1ZI3ieR8VU1R1d/kjVGiqiV2kSIigcDLwJVAW2CYiLQtstlYYIKqdgAe\nBZ5w940BHgG6AV1x7oqi3X3GAbcDLdypnxfXYKqa4DC4YBRsSmVIg/3Ehocw7puN/o7KmGrJm0Ty\n/WkSgDe6AhtUdZOqZgGTgGuKbNMWpyNIgDke668A/qeqB1X1EPA/oJ+I1Adqqep8dZ5jTACuPYfY\nTFWQciuERRE6/3lu7ZlI6rp9rN511N9RGVPteJNIugNL3UdUy93HSt40/20IeL4ttsNd5mkZkFff\nch0QKSKxxezb0J0v7pimugiNhK4jYc1Mbm6eSURokN2VGOMHZ3wh0YMvHx39AXjJHTBrLrATKJM+\nL0RkJDASID4+ntTU1LI4rN+kp6dX+msoK55lEZzdnu4BwWTM/BsXN7iTmct2cVHUIerWrD4vKNrf\nRmFWHgXKqyxKTCR5TX1FpC4QdhbH3gk08vie4C7zPPYu3DsSEYkABqnqYRHZCfQqsm+qu39CkeWF\njulx7NeA1wBSUlK0V69ep9us0khNTaWyX0NZOaUscn6g3oLXefTWp/nylfUszazD//Vv77f4ypv9\nbRRm5VGgvMrCmzfbB4jIepz+tb4BtgCfe3HsBUALEUkUkRBgKDCjyLHjRCQvhoeA8e78bOByEYl2\nK9kvB2a7Y8cfFZHubmutm3H6/jLV2YV3A0Lc8tcY3CWBKQt3sPfoCX9HZUy14c39/z9x6kl+VtVE\noC9OD8DFUtVs4G6cpLAGmKyqq0TkUREZ4G7WC1gnIj8D8cDj7r4H3fMucKdH3WUAdwFv4PRKvBHv\nkpqpyqISoMMQWDyBO7rUIjs3l/HfbfF3VMZUG97UkZxU1QMiEiAiAao6R0Se9+bgqjoLmFVk2cMe\n81OAKWfYdzwFdyieyxcCSd6c31QjPe+DpRNpvP4d+rf/Fe/N38qdvZoRVSPY35EZU+V5c0dy2K2/\nmAtMFJEXgAzfhmXMWYprAW0HwE9vMOrCujbwlTHlyJtEcg1wHLgf+ALncZJ12mgqnp4PQOYR2uz4\niEta1uGt7zZz4qQNfGWMr5Wb9bJ3AAAgAElEQVSYSFQ1Q1VzVDVbVd9R1RdV1YbaNRVPg07QrA/8\n8DKjejZkf3oWHy20ga+M8bXihtpNE5Gjp5nSRMReHzYVU88HIGMf5x+eRefGtXl17iayc3L9HZUx\nVdoZE4mqRqpqrdNMkapaqzyDNMZrTXtCQlfk+xcZdXFTdhw6zszlu/0dlTFVWvV5/ddUDyJw0QNw\neBu9s7+lZXwE41I3WhfzxviQJRJT9bS4Auq2JeC757jj4kTW/ZJmA18Z40OWSEzVExAAPe+HfWsZ\nUGM5DWvXYFyqdeZojK9YIjFVU7uBULsJQd89x8iLElm49ZANfGWMj3jT19ZAd5TCI9Zqy1QagUHQ\n417YuZChdbY6A1+lbvB3VMZUSd7ckTwFDFDVKGu1ZSoVdzje0PnPcUuPpsyxga+M8QlvEskvqrrG\n55EYU9Y8huMd0eQQ4SGBvGIDXxlT5op7IXGgiAwEForIhyIyLG+Zu9yYis8djjdi4YsM796Emct3\nse3AMX9HZUyVUtwdydXuVAs4hjMmSN6yX/k+NGPKQP5wvJ9ye5uTBAUE8OpcuysxpiydsRt5Vb2l\nPAMxxme63QHfv0Tc0nEM6nIXHy3awb2XtqBu5NkM+GmMORNvWm29IyK1Pb5Hi8gp44QYU2GFx0GX\nEbBiMnclh5Cdk8v4eVv8HZUxVYY3le0dVPVw3hdVPQQk+y4kY3zAHY630Zo36d++PhPnb+XoiZP+\njsqYKsGbRBLgjpsOgIjE4N3IisZUHPnD8b7DqK5RpNnAV8aUGW8SyTPADyLyTxF5DPge592SEolI\nPxFZJyIbRGT0adY3FpE5IrJERJaLSH93+Y0istRjyhWRTu66VPeYeevqen+5plrreR9kZ9Jm60Qu\nblmH8fNs4CtjyoI3A1tNAAYBvwB7gIGq+m5J+4lIIPAycCXQFhgmIm2LbPY3YLKqJgNDgf+455yo\nqp1UtRNwE7BZVZd67Hdj3npVtd74jHc8huO958K6zsBXi3b4OypjKj2v+tpS1VXAZGAGkC4ijb3Y\nrSuwQVU3qWoWMAln2N5Ch8ZpXgwQBew6zXGGufsaU3rucLwp+6aR3Lg2r83daANfGVNK3rTaGiAi\n64HNwDfAFuBzL47dEPAc53SHu8zTGGC4iOwAZgH3nOY4Q4APiix7y32s9XcRES9iMcbhDscr81/m\n7p4JbD94nM9W2MBXxpSGN5Xm/wS6A1+qarKI9AaGl9H5hwFvq+ozInIB8K6IJKlqLoCIdAOOqepK\nj31uVNWdIhIJTMV59DWh6IFFZCQwEiA+Pp7U1NQyCtk/0tPTK/01lJXSlkXtiD50yviahOUv0SCi\nN0/PXEatQz9TWf9NYn8bhVl5FCivsvAmkZxU1QMiEiAiAao6R0Se92K/nUAjj+8J7jJPvwX6Aajq\nDyISBsQBefUeQylyN6KqO93PNBF5H+cR2imJRFVfA14DSElJ0V69enkRcsWVmppKZb+GslLqstBL\n4MAMWu3/gj9cMYoHpq5C67eld+v4MouxPNnfRmFWHgXKqyy8qSM5LCIRwLfARBF5AcjwYr8FQAsR\nSRSREJykMKPINtuAvgAi0gYIA/a53wOAG/CoHxGRIBGJc+eDcbpqWYkxZ8NjON4BQT/YwFfGlJI3\nieQanL627gO+ADbi9LdVLFXNBu4GZgNrcFpnrRKRR0VkgLvZg8DtIrIM585jhBYMrn0xsF1VN3kc\nNhSYLSLLgaU4dzive3ENxhTmDscb9P3z3N6zCQu2HGLBFhv4yphzUeKjLVXNEJEmQAtVfUdEagKB\n3hxcVWfhVKJ7LnvYY3410OMM+6bi1M0UigXo4s25jSlW3nC8H9/OsNqreTE8jHGpGzl/RIy/IzOm\n0vGm1dbtwBTgVXdRQ2C6L4Myply4w/GG/vAct1zQhK/X7mXNbhv4ypiz5c2jrVE4dw1HAVR1PWBv\nk5vKL3843kXc0nC7DXxlzDnyJpFkui8UAk6FN86LhMZUfu5wvBELXuTG7k34dJkNfGXM2fImkXwj\nIn8BaojIZcBHwKe+DcuYchIcBt3vgk2p/K75EYICAnjtW7srMeZseJNIRuM0yV0B/A6n8vxvvgzK\nmHLlDscbu+QlBnVpyOSFO9iXlunvqIypNLzptDFXVV9X1etVdbA7b4+2TNURVit/ON5RSbnOwFff\nbfZ3VMZUGl512mhMldftDgiqQcKqV7myfX3e+8EGvjLGW5ZIjAF3ON7fwIrJ/L5LmA18ZcxZKDaR\niEigiIwtr2CM8asLnc6nW218m4taxDF+3hYb+MoYLxSbSFQ1B+hZTrEY419RCdBhKCx+h993q83+\n9Eym2MBXxpTIm0dbS0RkhojcJCID8yafR2aMP7jD8abs+ZBOjWrzqg18ZUyJvEkkYcABoA9OZ41X\n4/S6a0zV4w7HKwve4Pc96trAV8Z4wZtOG28pj0CMqTB6PgCrP6HX0U9pUbcL41I3MqBjg0o78JUx\nvuZNp40JIjJNRPa601QRSSiP4IzxC3c43oAf/8NdPRuydk8aqev2+TsqYyosbx5tvYUzIFUDd/rU\nXWZM1dXzAcjYxwD9mgZRYfwndYO/IzKmwvImkdRR1bdUNdud3gbq+DguY/yraU9I6ErgD/9mZM9G\nNvCVMcXwJpEcEJHh7jslgSIyHKfy3Ziqy2M43l/XXEhMeAiv2HC8xpyWN4nkVpyx0/cAu4HBgFXA\nm6rPHY43ZP4LjLigMV+t3cvaPTbwlTFFnTGRiMi/3NmuqjpAVeuoal1VvVZVt3lzcBHpJyLrRGSD\niIw+zfrGIjJHRJaIyHIR6e8ubyoix0VkqTu94rFPFxFZ4R7zRbGmNMZX8obj3beW39ZZ6wx8ZXcl\nxpyiuDuS/u6P9EPncmARCQReBq4E2gLDRKRtkc3+BkxW1WRgKPAfj3UbVbWTO93hsXwccDvQwp36\nnUt8xnjFHY43/KcX+HXXRny6fDfbD9rAV8Z4Ki6RfAEcAjqIyFERSfP89OLYXYENqrrJHWFxEnBN\nkW0UqOXORwG7ijugiNQHaqnqfLcr+wnAtV7EYsy58RiO984muwgQeG3uJn9HZUyFcsZEoqp/VNXa\nwGeqWktVIz0/vTh2Q2C7x/cd7jJPY4DhIrIDZ8CsezzWJbqPvL4RkYs8junZ+dHpjmlM2XKH441Z\n8hKDOicweeF2G/jKGA/evNle9C6iLA0D3lbVZ0TkAuBdEUnCqdRvrKoHRKQLMF1E2p3NgUVkJDAS\nID4+ntTU1DIOvXylp6dX+msoK/4oi0Z1+9Fs0zv0afMDH2Yn8I8PvmFwy5ByjeFM7G+jMCuPAuVV\nFiUmklLYCTTy+J7gLvP0W9w6DlX9QUTCgDhV3QtkussXichGoKW7v+db9ac7Ju5+rwGvAaSkpGiv\nXr1Kez1+lZqaSmW/hrLil7I40Rmen87lOpf+7R/km5/38X8396BWWHD5xnEa9rdRmJVHgfIqC18O\nbLUAaCEiiSISglOZPqPINtuAvgAi0gang8h9IlLHraxHRM7DqVTfpKq7gaMi0t1tCHAz8IkPr8EY\nR95wvGtncl/HXNIys5k436vGi8ZUed70tRUuIgEe3wNEpGZJ+6lqNnA3MBtYg9M6a5WIPCoiA9zN\nHgRuF5FlwAfACLcS/WJguYgsBaYAd6hq3mvFdwFvABuAjcDnXl6rMaXjDsfbYv2bXNQijjfnbbbh\neI3Bu0dbXwGXAunu95rAf4ELS9pRVWfhVKJ7LnvYY3410OM0+00Fpp7hmAuBJC/iNqZs5Q3Hu+AN\nRg8exYD3DjB66nJe/nVn6xnYVGtejUeiqnlJBHe+xDsSY6okdzjedlve4Y9XtGLWij02trup9rxJ\nJBki0jnvi9uK6rjvQjKmAssfjncCIztH0rtVHf45cw0rdx7xd2TG+I03ieQ+4CMR+VZE5gEf4tR9\nGFM9ucPxBvz0Cs/c0ImY8BBGvb+YNKsvMdVUiYlEVRcArYE7gTuANqq6yNeBGVNhxbWANlfDT68T\nk3OAf/86mR2HjjP64xU4bUWMqV68abV1M86Lg53daZi7zJjqq+/DkJsNU2/j/Ea1+MPlrfhs+W7e\n+9GaBJvqx5tHW+d7TBfhdGsyoLgdjKny4lrAVc/C1nnwzZP87uLz6NWqDv+cudrqS0y1482jrXs8\npttx7koifB+aMRVcp2HQaTjMHUvA5jk8c31HYmqGcLfVl5hq5lzebM8AEss6EGMqpf5PQ53WMPV2\nYnMP8uKwZLYfOs5DVl9iqhFv6kg+FZEZ7jQTWAdM931oxlQCITXh+rfh5DGYehtdG9figctaMnP5\nbt7/yepLTPXgzZvtYz3ms4GtqrrjTBsbU+3Ube3Ul0y/A775F3f2+gs/bj7IPz5dTadGtWnXIMrf\nERrjU97UkXzjMX0HNBWRl8shNmMqj/z6kqcJ2DyH527oSHTNYO5+fwnpmdn+js4Yn/KqjkREkkXk\naRHZAvwTWOvTqIypjIrWlwxNZuuBDP5i9SWmijtjIhGRliLyiIisBf6N0+W7qGpvVf13uUVoTGVR\npL6kW5MoHry8FTOW7eKDn7aXuLsxlVVxdyRrgT7Ar1S1p5s8csonLGMqqbqt4apn3PdL/sWdlzTj\nohZxjPl0Fat3HfV3dMb4RHGJZCDOkLdzROR1EekLWF/ZxpSk06+dcd7nPk3A5lSeG9KJ2jWCufv9\nxVZfYqqkMyYSVZ2uqkNx+tmag9N5Y10RGScil5dXgMZUSv2fhjqt4OPbidNDvDgsmS0HMvjrNKsv\nMVWPN622MlT1fVW9GmeM9CXAn30emTGVWUg4XP8OZGXA1Nvo3rQ291/akk+W7uLDBVZfYqqWs3qz\nXVUPqeprqtrXVwEZU2Xk1Zds+Ra++Rd39W7ORS3ieGTGKtbstvoSU3WcSxcpXhORfiKyTkQ2iMjo\n06xvLCJzRGSJiCwXkf7u8stEZJGIrHA/+3jsk+oec6k71fXlNRhTKnn1Jd88ReDmVJ69oRO1agQz\n6v3FZFh9iakifJZIRCQQeBm4EmiL0/182yKb/Q2YrKrJwFDgP+7y/cDVqtoe+A3wbpH9blTVTu60\n11fXYEyZ8KgvqcMhXhjaiS37M/jb9JVWX2KqBF/ekXQFNqjqJlXNAiYB1xTZRoFa7nwUsAtAVZeo\n6i53+SqghoiE+jBWY3wnJNx5vyQzHabexoWJ0dx3aUumLdnJRwuttyFT+fkykTQEPGsVd7jLPI0B\nhovIDmAWcM9pjjMIWKyqmR7L3nIfa/1dRKxJsqn46rbxqC95ilG9m9OjeSwPz1jJuj1p/o7OmFIR\nX91ai8hgoJ+q3uZ+vwnopqp3e2zzgBvDMyJyAfAmkKSque76dsAM4HJV3egua6iqO0UkEpgKvKeq\nE05z/pHASID4+PgukyZN8sl1lpf09HQiImwYGKjcZdF6zQvE/zKHZR3/wdaaHXj4++PUDIJHLqhB\nWNC5/ZuoMpeHL1h5FChtWfTu3XuRqqaUuKGq+mQCLgBme3x/CHioyDargEYe3zcBdd35BOBnoEcx\n5xgBvFRSLF26dNHKbs6cOf4OocKo1GWRma767/NVn2quenSPfrd+nzYdPVPvn7REc3Nzz+mQlbo8\nfMDKo0BpywJYqF783vvy0dYCoIWIJIpICE5l+owi22wD+gKISBsgDNgnIrWBz4DR6vQ4jLtNkIjE\nufPBwK+AlT68BmPKVkg43PAOZKbB1N9y4XnR3Nu3BR8v2clHi6y+xFROPkskqpoN3A3MBtbgtM5a\nJSKPikjemO8PAreLyDLgA2CEmwXvBpoDDxdp5hsKzBaR5cBSYCfwuq+uwRifqNsGrhqbX19yT58W\nXNgsloc/WcnPv1h9ial8vBnY6pyp6iycSnTPZQ97zK8Gepxmv8eAx85w2C5lGaMxftHpRtjidOwY\n2OQCnh/anf4vzOOuiYuZcXcPaob49H9NY8qUT19INMacgYjTiiuuJUy9nboc4YWhndi4L52HP1nl\n7+iMOSuWSIzxl/z3S9Lg49vocV409/RpwZRFO/hoofXHZSoPSyTG+FN8W6e+ZPNcmPs09/ZtQffz\nYnj4k1Wst/oSU0lYIjHG3zrdCB2HQeqTBG6Zy4tDkwkPDeSuiYs5lmX9cZmKzxKJMf5WqL7kNurK\nEZ4fksyGfek8YvUlphKwRGJMRVCkvqRns2ju6d2cjxbtYKq9X2IqOEskxlQU8W2dnoLz6ksubUn3\n82L42/SVbNhr9SWm4rJEYkxFkjwcOgzNry95YWgyNUOc+pLjWTn+js6Y07JEYkxFkl9f0gKm3ka8\nHOG5IZ1YvzedMTOsvsRUTJZIjKloQiOc8d4z0+Dj27m4eQyjejXnw4XbmbbE6ktMxWOJxJiKKL++\n5BuYO5b7Lm1B18QY/jptJRv2pvs7OmMKsURiTEWVX1/yBEFbv+Xfw5KpERzIKKsvMRWMJRJjKirP\n+pKPbyc+4CjPDunEul/S+MenVl9iKg5LJMZUZKERzvslJ47Ax7dzSfMYRvVuxqQF25m+ZKe/ozMG\nsERiTMUX386pL9mUCt8+w/2XtqRr0xj+Mm0FG/dZfYnxP0skxlQGyTdBhyFOfcm2ebwwrBNhbn1J\nVo76OzpTzVkiMaYyEIGrnoWYZjD1NuoHpvHsDR1ZuyeNccsyWbT1ELm5llCMf1giMaayCI1wxnt3\n60t6tYjlj1e0Yvm+HAaN+54Ln/yaRz5Zyfcb95Odk+vvaE014tNEIiL9RGSdiGwQkdGnWd9YROaI\nyBIRWS4i/T3WPeTut05ErvD2mMZUaUXqS0b1bs6/+9Tk+SGd6Ngoig8XbufXr/9I1//7itFTlzNn\n3V6ysi2pGN/y2cDQIhIIvAxcBuwAFojIDHec9jx/Ayar6jgRaYszvntTd34o0A5oAHwpIi3dfUo6\npjFVW/JNsPlbSH0CGl9AzWChf3JDrk1uyLGsbL5Zt48vVu1h5vLdTFqwnciwIPq2rku/pPpc0rIO\nNUIC/X0FporxWSIBugIbVHUTgIhMAq4BPH/0FajlzkcBu9z5a4BJqpoJbBaRDe7x8OKYxlRtIvCr\n52DXEpj6W4I7PJW/qmZIEFe2r8+V7euTmZ3Ddxv288XKPfxv9S9MX7qLGsGB9GpVh35J9ejTui6R\nYcF+vBBTVfgykTQEPAee3gF0K7LNGOC/InIPEA5c6rHv/CL7NnTnSzqmMVVf3vslb/Sl47IxEHsY\nWv8KwuMKNgkKpE/rePq0jic7J5cfNx/ki5V7mL1qD5+v3ENIYAA9W8TRL6kel7WJJzo8xG+XYyo3\nXyYSbwwD3lbVZ0TkAuBdEUkqiwOLyEhgJEB8fDypqallcVi/SU9Pr/TXUFasLArEtbqXxA1vw6f3\nop/ez+HaSeyrcyH747qTFRp9yvZ9a0PvCwPZeDiMhb9ks3DLPr5eu5cAgdYxAaTEB9G5biC1wypv\nOxz7+yhQXmXhy0SyE2jk8T3BXebpt0A/AFX9QUTCgLgS9i3pmLjHew14DSAlJUV79ep1ThdRUaSm\nplLZr6GsWFl46kXqnAvo1ToWWf0J0aunE73+FVqufxWa9IC210Cbq6FW/UJ79QFuB1SVlTuP8sWq\n3Xy+cg8TVmfw7hro0jiafkn1uKJdPRrF1PTLlZ0r+/soUF5l4ctEsgBoISKJOD/2Q4FfF9lmG9AX\neFtE2gBhwD5gBvC+iDyLU9neAvgJEC+OaUz1IgL1OzhTn7/B3jWw+hNn+vyPztSou5NU2g6AqASP\nXYX2CVG0T4jiD5e3YsPedD5f6Tz6euyzNTz22RraN4yiX1I9+iXVo1mdCD9eqKmofJZIVDVbRO4G\nZgOBwHhVXSUijwILVXUG8CDwuojcj1PxPkJVFVglIpNxKtGzgVGqmgNwumP66hqMqXREnC7o49tC\n74dg3zpYPQNWT4fZDzlTwvnuncoAiG7isavQIj6SFvGR/L5vC7YeyOALN6k8PXsdT89eR8v4CPq1\nq0e/pPq0qR+JiPjxYk1F4dM6ElWdhdOk13PZwx7zq4EeZ9j3ceBxb45pjDmDOq3gkj860/4NsMa9\nU/nv35ypQbJ7p3INxJxXaNcmseH87pJm/O6SZuw+cpzZblJ5ac4GXvx6A01ia7pJpR4dE2oTEGBJ\npbryd2W7Maa8xDWHix50poObYc0MJ6l8OcaZ6rWHttc6U1zzQrvWj6rBiB6JjOiRyP70TP63+hc+\nX7mHN+dt5tW5m6hXK4ykhrUIDQokJCiAkMAAQoM9P53loUEBp3zm71No+WmOExRgd0AVlCUSY6qj\nmEToca8zHd4Gaz51ksrX/3Smuu0K7lTqti60a1xEKMO6NmZY18YcOX6Sr9b8wuxVe9h+8DhZOblk\nZueQlZ1LVnYume5ndhn1AxYSWDjhnJJ8AgM4nnaCaXuWEB4aRGRoEOGhQUS4U3hoEBFhQUSEBhIR\nGkx4aGD+8uDAytNSLTdXOZGdw7GsHI5nOZ8ZWdn588fc+eVbTpJ8/CRRNXz7vpAlEmOqu9qN4YJR\nznRkZ0FSSX0CUv8P4lo5CaXdtVC3rVMP44qqEczAzgkM7JxQzAkgJ1c9kkuOk2Bycsk86XzmLfdM\nPmfa1vk83bbO9ocylcPbD5OemU16ZjYnTnrXRUxoUACRYU5SCQ/JSzgeCcgj+eRtV3h9wXxIUAC5\nucrxkx4/9ieznR/5TPeH/mTej34Ox7OyPeZzOHbSWZaRWTDvmTSOn/R+hMxbjp6wRGKMKUdRDaH7\nHc6UtqcgqXw7FuY+BbHNC+5U6nUolFSKExgg1AgJdLtn8e2PWtEmr9k5uWRk5pCWeZKMzJz8BJOR\nmU36iezC3z3m005kszftBJv2ZZOemUNGZrbXP+DBgcLJs+zeP8gto5ohgdQMCaJGsDNfKyyIerVC\nnWUhgYSHBFIjJMjdLtDdzvlesL+zzZKf5nNeObS0s0RijDm9yHrQ9XZnSt8Ha2c6rb/mPQ/fPgPR\nTd2kcq1TaV9B6y+CAgOIqhlAVM3SJ7DsnFwysnIKJZuMvHmP5HTsZA4hgQGFftTzf+jdH/4aIYGE\nhwZSM9iZDwkq+0dr60OEwHJoBGGJxBhTsog6kHKLM2UcgHWfOXcqP7wM370AUY2hQUcICoOgUAgM\ndedDnM/AkIJ1QaFFlnmxTWBwhUhUQYEBRNUI8PmjosrGEokx5uyEx0Lnm53p+CFY97nzrsr+DZB9\nAnKynM9s9zP3ZNmct8Qk5Xy2OZwBadMgNBJCa7mfnlNU4e8h4RUiSZ2T3BzIyiiYTubNH4OsdOrt\nXgQnOkNYrZKPVQqWSIwx565GNHT6tTOdSW4u5GQWTi5Fk83pluV4rDvjfpnusTOdH89jB4hMOwBr\n10FmGmQf9+Ii5AwJJ7KYZHSG7QPO0EV/TjZkpcPJY0V++J0ffM8f/4Jt0t1lRRNERsF22SeKvbLW\nAGnDLZEYYyq5gAAIqAHBNcrldD95VrbnnHQSyinT0TMsc5efOAxHthesy0r37uTB4QV3OTlZBckg\nJ/MsrkAgJMI5RkhN5zM4HMJqQ60GBeuCa7rzHtuEFJ7mL15B9+jEsy3Cs2aJxBhTdQUGQ80YZyqN\n3BwnKZSYjNzlWRnOI7giP+yn/PiHRLjLPLYJCiuzR20nauxzHvn5mCUSY4wpSUAghEU5kzlF5XmV\n0xhjTIVkicQYY0ypWCIxxhhTKpZIjDHGlIolEmOMMaViicQYY0ypWCIxxhhTKpZIjDHGlIqols3I\nZRWZiOwDtvo7jlKKA/b7O4gKwsqiMCuPwqw8CpS2LJqoap2SNqoWiaQqEJGFqpri7zgqAiuLwqw8\nCrPyKFBeZWGPtowxxpSKJRJjjDGlYomk8njN3wFUIFYWhVl5FGblUaBcysLqSIwxxpSK3ZEYY4wp\nFUsklYyI3CMia0VklYg85e94/E1EHhQRFZE4f8fiTyLytPt3sVxEpolIbX/HVN5EpJ+IrBORDSIy\n2t/x+JOINBKROSKy2v2tuNeX57NEUomISG/gGqCjqrYDxvo5JL8SkUbA5cA2f8dSAfwPSFLVDsDP\nwEN+jqdciUgg8DJwJdAWGCYibf0blV9lAw+qalugOzDKl+VhiaRyuRN4UlUzAVR1r5/j8bfngD8B\n1b6iT1X/q6rZ7tf5QII/4/GDrsAGVd2kqlnAJJx/dFVLqrpbVRe782nAGqChr85niaRyaQlcJCI/\nisg3InK+vwPyFxG5Btipqsv8HUsFdCvwub+DKGcNge0e33fgwx/OykREmgLJwI++OoeN2V7BiMiX\nQL3TrPorzn+vGJxb1fOBySJynlbRpncllMVfcB5rVRvFlYeqfuJu81ecxxoTyzM2UzGJSAQwFbhP\nVY/66jyWSCoYVb30TOtE5E7gYzdx/CQiuTh96ewrr/jK05nKQkTaA4nAMhEB5zHOYhHpqqp7yjHE\nclXc3waAiIwAfgX0rar/uCjGTqCRx/cEd1m1JSLBOElkoqp+7Mtz2aOtymU60BtARFoCIVTDzulU\ndYWq1lXVpqraFOcxRueqnERKIiL9cOqLBqjqMX/H4wcLgBYikigiIcBQYIafY/Ibcf6F9SawRlWf\n9fX5LJFULuOB80RkJU5l4m+q4b88zem9BEQC/xORpSLyir8DKk9uQ4O7gdk4FcuTVXWVf6Pyqx7A\nTUAf9+9hqYj099XJ7M12Y4wxpWJ3JMYYY0rFEokxxphSsURijDGmVCyRGGOMKRVLJMYYY0rFEokp\nMyKS7uPjt3abMS4RkWblee7KSkRGiEiDMjjGS2UUzxZvemq2/56ViyUSU5lcC0xR1WTV/2/v3EKs\nKqM4/vtniql5qYZJCDMIekjIUIrSTB968MGMHBiovEQXiCgLxKeIIUtQqQxjkgzLahrEwjAxxUtj\nNhnedcZMKDQMuhCZZBGJs3r41sbt8cx4dM+xSdcPNmed77LX2nsOZ+3vW3PWsu/+KyMkdWtGiO4+\nXwkzgHNyJFW2J7gICXT96YsAAAR8SURBVEcSdDtKLJDULqlNUr23Xyap0etmrJe0RlJdmfkjJX2V\nq60xxH9M9QzwhKTPOtH7kqS9PrdW0pWSDnmqCCQNzN5LapH0mq9w2iXd5mP6S1oqaZuvfCZ7+wxJ\nqyRtAjaW6B3u19Qk6YCkDyX1877nJW13HW/6L45x/Qsl7QBmSprkyTh3S9ogqdbHNUhaJmmLpO8l\n3S9pvt/XtblrG+WJPHdKWidpqN/b0UCTX+cV5caVs6eLv20hO53Z3r5N0o0+/wZJW739xZy+AZI2\nStrlfZdsRt8ejZnFEUe3HMBxf51Cqo/RC6gl1QsZCtQBa0gPMNcCR4G6MufZB9zt8gvAQpcbgFmd\n6DZgksvzgedcfhu4z+XHgZddbgGWuDwOaHd5LvCQy4NJtT36k57sfwCuKqN7uOsf4++XZnbmxwPv\n5WxsARpzfUM49QPhR3N2NgBfAL2BW4C/gInet5K0SusNfAnUeHs9sDSnZ7TLZxvX2Mm9nQG8XtRO\nlw+TkkwCTANWu7wKmObyk5z6LF0ODHT5GuDbTH8cPeeIJWxQDcYCzWZ2EvhZ0mZStuKxwAoz6wB+\nKreykDQIGGxmm71pGbCiAp3/AKtd3gnc4/JbpBxUHwMPA4/l5jQDmNnnvloZTMoofK+kWT6mLzDM\n5fVm9lsn+o+YWavL7wNPkwqPTZA0G+hHyty8H/jExy3Pzb8OWO4rhD7AoVzfp2Z2QlIbyTmv9fY2\nkhO7CRhBSo+Cj/mxjI1nG7e8zJxSitiZ0Zx7fdXlMaQHEEgOd57LAuZKGgd0kFLD1wKXbF61nkg4\nkuBi4YT5YytwEv9sm1mrbz2NB3qZWXtuTml+ICN9cU0xs4P5Dkm3A392of+Mc0nqCzSSVgRHJDWQ\nHFNG/nyLgFfMbJXb2pDrywqZdUjKX2eHX6eA/WZ2Rxf2UcG4rq6vO+zMsArkjAeBGmCUO6nDnH4P\ngx5AxEiCarAFqJfUS1INaetoG9AKTFGKldQC40snmtkx4Kiku7xpKrC5dNw58i7wAWmbK08WuxkL\nHHPd64CncrGMWyvUMUxS9gX9AGmbJ/vC+1WpLsQZ8aAcgziV9nx6hTozDgI1mX6lGNDN3vcHKZnj\n2cZVShE7M+pzr1tdbiVl7IXkPPL6fnEnMgG4/jx1BlUkHElQDVaS4hx7gU3AbEsp3j8ixRm+Jm3/\n7AKOlZk/HVggaR8wkhQnKUITaW+/uaT9b0m7gcXAI942h7TPv0/Sfn9fCQdJdbEPuK43zOx3YAnQ\nTnJQ27uY3wCskLSTcywNYKm0bB0wT9JeYA9wp3e/AyyWtIe03dTZuEo5bztzDPG/7UzgWW+bSbp/\nbZxe2bAJGO3t04BvzlNnUEUi+29wQZE0wMyOS7qatEoZY1WuI+L/vTTZzKbm2lpIAfEd3XD+4aSg\n8Yii5wqC/yMRIwkuNKs9qN0HmHMBnMgiYCJQtVoMQXCpEyuSIAiCoBARIwmCIAgKEY4kCIIgKEQ4\nkiAIgqAQ4UiCIAiCQoQjCYIgCAoRjiQIgiAoxL+pHmoOecCRCgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import math as m\n",
    "para = [m.log10(x) for x in params]\n",
    "plt.plot(para,train_auc,label='train auc')\n",
    "plt.plot(para,cv_auc,label='validation auc')\n",
    "plt.xlabel(\"log of hyper parameter lambda\")\n",
    "plt.ylabel(\"Auc for each lambda\")\n",
    "plt.grid()\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "id": "rhP3kFHlYj_i",
    "outputId": "95ce4051-90ac-4824-fd53-546b99c83abf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CalibratedClassifierCV(base_estimator=SGDClassifier(alpha=1e-06, average=False,\n",
       "                                                    class_weight=None,\n",
       "                                                    early_stopping=False,\n",
       "                                                    epsilon=0.1, eta0=0.0,\n",
       "                                                    fit_intercept=True,\n",
       "                                                    l1_ratio=0.15,\n",
       "                                                    learning_rate='optimal',\n",
       "                                                    loss='log', max_iter=1000,\n",
       "                                                    n_iter_no_change=5,\n",
       "                                                    n_jobs=-1, penalty='l2',\n",
       "                                                    power_t=0.5,\n",
       "                                                    random_state=None,\n",
       "                                                    shuffle=True, tol=0.0001,\n",
       "                                                    validation_fraction=0.1,\n",
       "                                                    verbose=0,\n",
       "                                                    warm_start=False),\n",
       "                       cv=5, method='sigmoid')"
      ]
     },
     "execution_count": 116,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = SGDClassifier(loss='log',penalty='l2',tol=0.0001,alpha=0.000001,n_jobs=-1)\n",
    "model = CalibratedClassifierCV(lr,cv=5)\n",
    "model.fit(train_vect1,Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "NwsLz3ypkHcr",
    "outputId": "f0455d4b-7b24-40c1-8897-74dd31a92470"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc for logistic regression is 0.9465540641624216\n"
     ]
    }
   ],
   "source": [
    "predict_y_test = model.predict_proba(test_vect1)[:,1] # Taking probability for positive class\n",
    "eval = JigsawEvaluator(y_binary, y_identity_binary)\n",
    "auc = eval.get_final_metric(predict_y_test)\n",
    "print(\"Auc for logistic regression is {}\".format(roc_auc_score(Y_test,predict_y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rKYF4x5uYj_n"
   },
   "source": [
    "### Training Naive bayes model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "32kER6MhYj_o",
    "outputId": "d53bfad1-3867-4577-b794-aea4615b172e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "  0%|          | 0/8 [00:00<?, ?it/s]\u001b[A\n",
      " 12%|█▎        | 1/8 [00:01<00:11,  1.61s/it]\u001b[A\n",
      " 25%|██▌       | 2/8 [00:03<00:09,  1.61s/it]\u001b[A\n",
      " 38%|███▊      | 3/8 [00:04<00:08,  1.61s/it]\u001b[A\n",
      " 50%|█████     | 4/8 [00:06<00:06,  1.61s/it]\u001b[A\n",
      " 62%|██████▎   | 5/8 [00:08<00:04,  1.61s/it]\u001b[A\n",
      " 75%|███████▌  | 6/8 [00:09<00:03,  1.61s/it]\u001b[A\n",
      " 88%|████████▊ | 7/8 [00:11<00:01,  1.60s/it]\u001b[A\n",
      "100%|██████████| 8/8 [00:12<00:00,  1.60s/it]\u001b[A\n",
      "\u001b[A"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "param = [0.00001,0.0001,0.001,0.01,0.1,1.0,10,100]\n",
    "train_auc = []\n",
    "cv_auc = []\n",
    "\n",
    "for i in tqdm(param):\n",
    "    model = MultinomialNB(alpha=i)\n",
    "    model.fit(train_vect1, Y_train)\n",
    "    predict_y_train = model.predict_log_proba(train_vect1)[:,1] # Taking probability for positive class\n",
    "    t_auc = roc_auc_score(Y_train,predict_y_train)\n",
    "    predict_y_cv = model.predict_log_proba(cv_vect1)[:,1]# Taking probability for positve class\n",
    "    c_auc = roc_auc_score(Y_cv,predict_y_cv)\n",
    "    train_auc.append(t_auc)\n",
    "    cv_auc.append(c_auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 283
    },
    "colab_type": "code",
    "id": "jG73kybSYj_u",
    "outputId": "86f56712-0eb9-46fd-f2fe-8705bcf0ebcc"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VvX5//HXdSchAZJACHsGEGQK\nyBbQgAsXqKBi3RYHddQ6WuwQqm1/fuuoraNWrS1YFRVFUVFAIA72EJQpGwKIEFYCgazr98c5SW5C\nkvuGcOfcd3I9H4/zyBmfc877DnpfOetzRFUxxhhjyuPzOoAxxpjwZ8XCGGNMQFYsjDHGBGTFwhhj\nTEBWLIwxxgRkxcIYY0xAViyMMcYEZMXCGGNMQFYsjDHGBBTtdYDTpX79+pqSknLK6x8+fJjatWuf\nvkAhFElZIbLyRlJWiKy8kZQVIitvRbIuXbp0r6o2CNhQVavE0LNnT62IOXPmVGj9yhRJWVUjK28k\nZVWNrLyRlFU1svJWJCuwRIP4jrXTUMYYYwKyYmGMMSYgKxbGGGMCsmJhjDEmICsWxhhjArJiYYwx\nJiArFsYYYwKqMg/lnaojOXm8nLaRLVtzWJazDkSKlolfu8LZ4jfXr2mpbZ1x/yWlrXfi9srclrtk\nw+ZcNkRtKtq++LUVv30WTnNCGzluX4XzOG5eiXVKrOe/b//fjRRvxp0vrN6dR97q3fh8xW18Iu7g\n7NB/2tlf8bTP3YFPBJ+P4vUoXu4TKcp5/Lb891e43eKfReu521J7zbAxpar2xSI7J5/n52wABd24\nwes4wVu3xusEJ+fbJV4nCJpvxqdER/mI9glRPiEmykeUT4j2CdFRQrTvxOmAbaPEbeMjJkr82pSx\nrk+IcpeVtZ+YKB8Z2QWoaql/lBhzOoW0WIjIUODvQBTwmqo+WWJ5K+B1oAGwD7hRVdPdZbcAv3eb\n/klVJ4QiY3J8LJv/32WkpaWRmppaapvCvzb9/+jUUpafOL9wnp4w78R9lN/Wf7Wvv/6aAQMHOsv9\n1lMtbqfqbMmZV9SoaF7JdUp+xnLbFC1Xv3bH70u1eHrJkiWcfXYvClQpKMqlFCgUFDg/C/dT4M53\nnhotni5wp9VvusDdeYEqBQV+bfBvc+K2tDCH4rdtZ9nGTZtp0bIVeQVKfkEBuflKfoGSV1BAnjue\n6y7Ly1fyCvS4ttm5+c68/AJ3PWfc+VnctnhcyS0oKPO/i2A8sXgmnZokOkNTZ2jbIJ6YKDvLbE6f\nkBULEYkCXgQuBNKBxSIyVVVX+zV7GpioqhNEZAjw/4CbRKQeMA7ohfM9tNRdd3+o8pbH/7ROGS0q\nLQtAzWghMS6mUvdZEXsSo+javI7XMYKSFrWD1NQzK32/BYWFpcApLPn5ThHJL1FkiouXkp2Tz7S5\ny8iLb8TqnYd4Y8FWjuUVAFAj2seZjRKOKyAdGieQEEH/3ZjwEsojiz7ABlXdBCAik4DhgH+x6AQ8\n6I7PAT50xy8GZqrqPnfdmcBQ4O0Q5jXGMz6fUMMn1DjJe06ObY8hNfUsAPLyC9i89zCrdh5i9a5D\nrN55iBmrf+SdJduL2rdKrkWnJol0dgtIpyZ1aJQYa6exTEASqgt6IjISGKqqo93pm4C+qnqvX5u3\ngIWq+ncRuRp4H6gP3AbEqeqf3HZ/ALJV9ekS+7gTuBOgUaNGPSdNmnTKebOysoiPjz/l9StTJGWF\nyMobSVkhcF5V5cAxZeuhArZlFrDN/fnTkeL/7xNioGWijxYJUbRM9NEqwUfj2s61kcrMGm4iKW9F\nsg4ePHipqvYK1M7rC9wPAy+IyK3AV8AOID/YlVX1FeAVgF69emlZ1xyCUd41i3ATSVkhsvJGUlY4\n9byZR3NZ+2Mmq3c6RyCrdx1i9vZMcvJzAYiN9tGhcYJ79FF4GiuR2rGn/pVRXX63XqiMrKEsFjuA\nFn7Tzd15RVR1J3A1gIjEAyNU9YCI7ABSS6ybFsKsxlQrCXEx9E6pR++UekXzcvML2LTnMKt2Hiwq\nIJ+t/JG3FzmnsUQgJbn2cQWkc5NEGiTYaazqIJTFYjHQTkRa4xSJUcDP/BuISH1gn6oWAI/i3BkF\nMB34i4gkudMXucuNMSESE+XjzMYJnNk4gavPduapKrsOHi0qHqt3HuK79AN8+t2uovXqx9egY+GF\n9CaJdG5ah9b1a5/201jGWyErFqqaJyL34nzxRwGvq+oqEXkc52UbU3GOHv6fiCjOaah73HX3icgT\nOAUH4PHCi93GmMojIjStW5OmdWtyQadGRfMPZueydldxAVm96xCvf7OZ3HznWkhcjI8OjROPOwrJ\nLbAHHiNZSK9ZqOo0YFqJeY/5jU8GJpex7usUH2kYY8JInZox9G2TTN82yUXzcvIK2Lgni9U7D7l3\nZB3kkxU7eWvhNgBa1/ExJFXtiCNCeX2B2xhTRdSI9tGxSSIdmyQyoqczT1XZcSCbj1fs4v8+X8vH\nK3ZyZY9m3gY1p8Qe8TTGhIyI0DypFned24YWCT7+9sUP5OYXeB3LnAIrFsaYkPP5hKvbxbA14wiT\nl6Z7HcecAisWxphK0b1BFN1b1OUfs9ZzNDfox6lMmLBiYYypFCLCIxefya6DR4suepvIYcXCGFNp\nBpxRn/5tknkpbQNHcvK8jmNOgt0NZUwkObIPMjY4w/4tpGzZAr4lEB0LUTWcobTx6FiIioWomBLz\nSrSrhCexH774TEb8cx7/mbuFewafEfL9mdPDioUx4SbnMGRsdIvCxuLikLEBjh7wayikoLD1NO7b\nV7KYxDhF5rjCUiPAvMJ13Hnu8ppHnK+bnq2SGNKhIf/6ciM39mtFnZrWbXoksGJhjBfyc2H/1uML\nQWFxyNx5fNvE5pDcFrpcDclnFA91W5L21TekDjoH8nMgLwfyj0HeMWf7+ceK552wPOck1ykx71im\nO6/k8lxnWwW5J3zks6PjYfBlUDOJhy5qz2X/+IbXvt7EQxdV/vtDzMmzYmFMqBQUQOau4wuB3ykk\n1O+OoJr1nALQJtUpDIUFoV4bqFGr7H2IOH/NR8dCbIg/z8koKCguJPm5sGcd0f+9DL5+Fi56gs5N\n63BZ1ya8/s1mbj0nheT4cApvSmPFwpiKOrLvxNNFGRth30bIPVLcLrqmUwAad4XOV/kdJbSFWvXK\n3n4k8vnAFwcxcc507frsbjSYxgv/BX3uhLot+NWF7fls5S7+mbaR31/eydu8JiArFsYEI+cw7NtU\n+lFCtt/bfiUKklKcItD63OOPEhKaOF+i1dTm1j+j8d65MOfPcNXLnNEwnqt6NOeNBVsZPagNjevE\neR3RlMOKhTGF8nKoeWQn/DD9xKOEQzuOb5vQ1CkEna48/jpCUivnAq85wbG4BtDvbpj7D+h/DzTu\nyi/Pb8dHy3fwwpz1/OnKrl5HNOWwYmGqh2NZzvWDQzvgUOHPnX7zdsLhPfQFWOSuE1cX6rdzjhDq\ntS0+SqjXBmIj43WbYWfgg7BsIswcBzd9QMvkWlzXuwWTFm3nrnPb0qJeOddnjKesWJjIpuqcBjq0\n0/3yd38WFQV3+tjBE9etmQSJzZzTQ026Q2Iz1u7KosPAK5yiUNWuI4SDmnVh0MMw43ewcTa0HcJ9\nQ9oxeWk6z32xnmeu7eZ1QlMGKxYmfBXkQ9ZPfgWgxFA4P+9oiRUF4htBonuqqPUgZzyhqfMzsalT\nIEq5y+jHtDQ6tOhTOZ+vuupzByz6l3N00TqVxnXiuKlfK16fu5kxqW04o2GC1wlNKaxYGG/kHXNP\nAZVxSujQLmdaS3Q454uBxCbOEUGT7nDmpc544bzEpk6hsOsG4Ss6FoY8Bh+MhpWT4axrGZPalrcX\nbeNvM9fz4g1ne53QlMKKhTl98vOcU0LZ+yF7n/PzSAYtt86DT6YWnx7K3AWH95y4fkzt4r/8C48G\njjsiaAa1kqv1HUVVRpcRMP95mPUEdBxGcnwctw9szfOzNzBmx0G6NKvjdUJTghULc6KCAqdbicIv\n/iP7ShSA0qYPlH5dAGgD8GNS8V/+TXsUj/sfEcQmVkrfRCYM+Hxw4eMwcTgsfhXOuY/Rg9owYd4W\nnp35A6/f2tvrhKYEKxZVmSocO1TiC96/AJTx5Z99ANAyNioQV8e5+FszCWrVh+R2xdM1C+cnFU1/\n9e0PnHv+xZX5yU0kaJMKZ1wAXz0NPW6kTs0k7jqvLU9NX8fSrfvp2SrJ64TGjxWLcJefBzlZzkNh\nOYchJ4u6+1fASv8v+ANl//Vf8py/v9hE9ws9yfmyT2rl92Vf8svf/RlXB3xRJ/URCqJOZ093pkq5\n4I/w8sCibkBuG5DCf+Zu5unp63j7zn5epzN+rFicTnk5Jb7YD5eYzixjfonpY1nF0/nHTthNd4AV\nfjNiah//13zDTuV/4des59zCaBeBjdcad4Fu14PbDUitui34ReoZPP7JauZu2MuAM+p7ndC4Qlos\nRGQo8HcgCnhNVZ8ssbwlMAGo67YZq6rTRCQFWAOsc5suUNW7QxIyPxf2rifx4DrYqGV8sZf8ci/x\nRX/MbVtKT5tliq7pPNhVozbUcH/GJkBC4+LpGrWhRoLfuNP22zUb6NF/SPGXf7R1wmYi2ODfwsr3\ni7oB+Vnflrz69Saemr6Oc9omI3YdKyyErFiISBTwInAhkA4sFpGpqrrar9nvgXdV9Z8i0gmYBqS4\nyzaqavdQ5SuSvR/+2Z+zAb4to03M8V/W1HD/kq/TvMQXe+1yvuhLtDvJUzn+Du5Kg0bW8ZqpIuq2\nOK4bkLjGXbn//HY8+sH3zF77E+d3bOR1QkNojyz6ABtUdROAiEwChgP+xUKBRHe8DlCiI/9KEFcX\nrvkv363dxFm9+vt9sbtf7jG17FZNY0KtRDcgI3s25+UvN/L0jB8YfGZDfD47uvBaKL8FmwHb/abT\n3Xn+xgM3ikg6zlHFfX7LWovItyLypYgMClnK6BrQ+Sr2JfeEVudAk27OU78JjZzTRFYojAm9wm5A\nNs6CjbOJifLxwAXtWLPrENNW7vI6nQFEtaxbJCu4YZGRwFBVHe1O3wT0VdV7/do86GZ4RkT6A/8G\nugAxQLyqZohIT+BDoLOqHiqxjzuBOwEaNWrUc9KkSaecNysri/j4yOgcLpKyQmTljaSsEFl5A2WV\nglz6LPoFedHxLO35DAUIv5+bTYHCnwfUJKqSjy6q0u+2PIMHD16qqr0CNlTVkAxAf2C63/SjwKMl\n2qwCWvhNbwIalrKtNKBXefvr2bOnVsScOXMqtH5liqSsqpGVN5KyqkZW3qCyrnhXdVyi6op3VFX1\ns+93aqvffKLvLt4W2nClqHK/2zIASzSI7/RQnmNZDLQTkdYiUgMYBUwt0WYbcD6AiHQE4oA9ItLA\nvUCOiLQB2rmFxBhTlXUZ4ZwKnvUE5B7l4s6N6dqsDn+ftZ6cvAKv01VrISsWqpoH3AtMx7kN9l1V\nXSUij4vIMLfZQ8AdIrICeBu41a105wLfichyYDJwt6ruC1VWY0yYKOwG5OA2WPwaIsJDF7UnfX82\n7yze5nW6ai2kz1mo6jScC9f+8x7zG18NDChlvfeB90OZzRgTptqkQtvz4aunoMcNnNe+Ab1Tknh+\n9gZG9mxBzRqnftu5OXV2q48xJvxc+Ec4ehC+fhYR4eGLzuSnzGO8sWCL18mqLSsWxpjw07hrcTcg\nB7bTt00yg9rV559pG8k8ehI9JZjTxoqFMSY8Df6t83POnwF4+KIz2X8kl9e/2eJdpmrMioUxJjwV\ndgOyYhL8+D3dWtTlok6NeO3rTRw4kuN1umrHioUxJnwNfNB5unvmOAAeuuhMsnLy+NdXdid9ZbNi\nYYwJX8d1AzKHMxsnMKxbU/47dws/ZR71Ol21YsXCGBPe+twBdVvCzMegoIAHLmhPTn4BL83Z6HWy\nasWKhTEmvEXHwpA/wI/fwcrJtK5fm5FnN+ethdvYcSDb63TVRlDFQkQuE5Ffi8hjhUOogxljTJEu\nI6HxWUXdgNx/QTsAnp+13uNg1UfAYiEiLwPX4XQfLsA1QKsQ5zLGmGI+H1z0RFE3IM3q1uRnfVvy\n3tJ0Nu897HW6aiGYI4tzVPVmYL+q/hGnN9n2oY1ljDEltEkt7gYkez+/GNyWmCjhuS9+8DpZtRBM\nsSg8KXhERJoCuUCT0EUyxpgyFHYD8s3faJgQx63ntGbqip2s+zHT62RVXjDF4hMRqQs8BSwDtuD0\nEGuMMZWrsBuQBS/Dge3cfV4b4mtE88yMdV4nq/ICFgtVfUJVD7g9wbYCOqjqH0IfzRhjSuHXDUjd\nWjUYPagNM1bvZsX2A97mquLKLBYicnXJAbgMON8dN8aYyleiG5DbB6aQVCuGp+3oIqTKO7K4wh1+\njvNu7Bvc4TXg9tBHM8aYMgz8FcTVgZnjSIiLYUxqW75ev5eFmzK8TlZllVksVPU2Vb0NiAE6qeoI\nVR0BdHbnGWOMN2omwbmPFHUDcnP/FBomxPLMjB9wXrZpTrdgLnC3UNVdftO7gZYhymOMMcHx6wYk\nLkq4b8gZLNqyj6/W7/U6WZUUTLGYJSLTReRWEbkV+BT4IrSxjDEmgBLdgFzXuyXNk2ryzIx1dnQR\nAsHcDXUv8DLQzR1eUdX7Qh3MGGMC8usGpAa5/PL8dnyXfpDpq3Z7nazKCbYjwflAGjDbHTfGGO/5\ndwOy6FWu6tGMNg1q8+zMdeQX2NHF6RRM31CjgUXAVcBIYIGI2N1Qxpjw0Ca1qBuQ6JyD/OqC9vyw\nO4uPV+z0OlmVEsyRxSNAD1W9VVVvAXoCvwlm4yIyVETWicgGERlbyvKWIjJHRL4Vke9E5FK/ZY+6\n660TkYuD/UDGmGrIrxuQy7o2oWOTRP72xQ/k5hd4nazKCKZYZAD+Ha9kuvPKJSJRwIvAJUAn4HoR\n6VSi2e+Bd1W1BzAKeMldt5M73RkYCrzkbs8YY07UuCt0GwULXsZ3KJ2HLmzP1owjTF6a7nWyKqO8\nJ7gfFJEHgQ3AQhEZLyLjgAVAMN089gE2qOomVc0BJgHDS7RRINEdrwMUHjcOByap6jFV3exm6BPs\nhzLGVEODf+f8nPNnzu/YkO4t6vKPWes5mpvvba4qorwjiwR32Ah8iPPFDvARsDmIbTcDtvtNp7vz\n/I0HbhSRdGAazjszgl3XGGOK+XUDIrtX8sjFZ7Lr4FHeWrjN62RVQnRZC9x3V4Ta9cB/VfUZEekP\nvCEiXYJdWUTuBO4EaNSoEWlpaaccJCsrq0LrV6ZIygqRlTeSskJk5a2MrNH0oW90bTLfuY/cbuPp\nWM/HczNW0+zoFmKj5aS2Zb/bElS13AHoBUzB6Z78u8IhiPX6A9P9ph8FHi3RZhXOE+KF05uAhiXb\nAtOB/uXtr2fPnloRc+bMqdD6lSmSsqpGVt5IyqoaWXkrLevc51XHJapumK1LtuzTVr/5RF+Yvf6k\nN1NdfrfAEg3wfa6qQV3gfhP4DzCC4s4FrwhivcVAOxFpLSI1cC5YTy3RZhtwPoCIdATigD1uu1Ei\nEisirYF2OLfvGmNM+fy6AenZog5DOjTkX19u5GB2rtfJIlowxWKPqk5V1c2qurVwCLSSquYB9+Ic\nFazBuetplYg8LiLD3GYPAXeIyAqcFyrd6ha7VcC7wGrgc+AeVbWrVMaYwEp0A/LQRe05dDSPf3+9\nyetkEa3MaxZ+xonIa8As4FjhTFX9INCKqjoN58K1/7zH/MZXAwPKWPfPwJ+DyGeMMcfrMhLmPQ+z\nnqDzfcO5rGsT/v3NZm45J4Xk+Fiv00WkYI4sbgO64zzvUHgK6vJQhjLGmArx+eDCx4u6AfnVhe3J\nzs3n5S83ep0sYgVzZNFbVc8MeRJjjDmd2g4u6gbkjB43cFWP5kycv5XRg9rQKDHO63QRJ5gji3ml\nPHltjDHhz68bkF+e3478AuX52eu9ThWRgikW/YDlbh9N34nI9yLyXaiDGWNMhfl1A9IyKoPrerdg\n0qLtbN93xOtkESeYYjEU59bViyi+XhHMrbPGGOO9om5A/sJ9Q9oR5ROe+8KOLk5WMC8/KrxVNhun\ny4/CwRhjwl9RNyBv0zh7PTf1a8WUb9PZ8FNm4HVNkWDeZzFMRNbj9Af1JbAF+CzEuYwx5vQZ+CuI\nqwMzxzEmtS01Y6L420w7ujgZwZyGegLnusUPqtoa54nrBSFNZYwxp1PNJDj3Edg4i+Td87h9YGs+\n/X4XK3cc9DpZxAimWOSqagbgExGfqs7B6S/KGGMiR587oI7TDcjogSkkxkXz7Mxg3rZgILhicUBE\n4oGvgDdF5O/A4dDGMsaY0yw6Fs53ugGps2Eqd53Xltlrf2Lp1v1eJ4sIwRSL4TgXt3+F00/TRuxu\nKGNMJOoyEhqfBbMe57a+TagfX4NnZqzzOlVECOZuqMOqmq+qeao6QVX/4Z6WMsaYyOLXDUit5f/h\nF6lnMG9jBvM27PU6Wdgr77WqmSJyqJQhU0QOVWZIY4w5bfy6AfnZWYk0qRPHUzPWFb47x5ShzGKh\nqgmqmljKkKCqiWWtZ4wxYc/tBiRu4d+5//x2fLvtALPX/uR1qrAWzDULY4ypWvy6ARl5htIquRZP\nz/iBggI7uiiLFQtjTPXkdgMS8+WTPHBBO9bsOsS0lbs8DhW+rFgYY6qnui2g712w4m2GNdpHu4bx\nPDvzB/LyC7xOFpasWBhjqq9BD0JcHaJm/5GHLmrPpj2HmfLtDq9ThaVg+oa6WkTWi8hBuxvKGFOl\nFHYDsuELLq65lq7N6vD3WevJybOji5KCObL4KzBMVevY3VDGmCrH7QZEZj7GQxeeQfr+bN5ZvM3r\nVGEnmGKxW1XXhDyJMcZ4wa8bkPNyvqJ3ShLPz97AsXy7M8pfeQ/lXS0iVwNLROQdEbm+cJ473xhj\nqga3GxCZ9QQPn5/CT5nHmLsjz+tUYaW8I4sr3CEROELxm/IK35ZnjDFVg183IH32vE/XZnX4Yluu\nPdXtJ7qsBap6W0U3LiJDgb8DUcBrqvpkieV/Awa7k7WAhqpa112WD3zvLtumqsMqmscYY8rkdgMi\nXz3Nz8/9lAemHmT+pgzOaVvf62RhIZi7oSaISF2/6SQReT2I9aKAF4FLgE7A9SLSyb+Nqv5KVbur\nanfgeeADv8XZhcusUBhjKoXbDcjlme8QHwMT5231OlHYCOYC91mqeqBwQlX3Az2CWK8PsEFVN6lq\nDjAJp7vzslwPvB3Edo0xJjTcbkCiF/2L4U0OMmP1j+w4kO11qrAggc7JicgKINUtEohIPeBLVe0a\nYL2RwFBVHe1O3wT0VdV7S2nbCudVrc1VNd+dlwcsB/KAJ1X1w1LWuxO4E6BRo0Y9J02aFODjli0r\nK4v4+PhTXr8yRVJWiKy8kZQVIitvpGSNPfoT/Rbcxer6l3B5+g1c1iaGke1reB2rXBX53Q4ePHip\nqgZ8+2mZ1yz8PAPMF5H3AAFGAn8+pVRlGwVMLiwUrlaqukNE2gCzReR7Vd3ov5KqvgK8AtCrVy9N\nTU095QBpaWlUZP3KFElZIbLyRlJWiKy8kZSVzM84c+10Lu8whrnbj/L0bYOIi4nyOlWZKuN3G8zL\njyYCI4DdwI/A1ar6RhDb3gG08Jtu7s4rzShKnIJS1R3uz01AGsGd+jLGmIrr9wui8w/zQIOl7Duc\nw6ffWQeDQfUNpaqrgHeBqUCWiLQMYrXFQDsRaS0iNXAKwtSSjUSkA5AEzPeblyQise54fWAAsDqY\nrMYYU2EtenMooR1tNr3BGfVrMnH+Fq8TeS6Yu6GGich6YDPwJbAF+CzQeqqaB9wLTAfWAO+q6ioR\neVxE/O9uGgVM0uMvnnTEeRhwBTAH55qFFQtjTKVJbz4MydjA2PY7WZF+kG+37fc6kqeCuWbxBNAP\n+EJVe4jIYODGYDauqtOAaSXmPVZienwp680Dyr2AbowxobSnwTmQ/hap+ycTHzuGifO30qNlktex\nPBPMaahcVc0AfCLiU9U5QMAr58YYE8nUFw29RxO9eQ5jOuXy6Xe72JN5zOtYngmmWBwQkXjga+BN\nEfk7cDi0sYwxJgz0vA2i47hRPiMnv6Ba90YbTLEYjtM31APA58BGnP6hjDGmaqudDGddS50f3mdo\nmxr8b8G2avsmvWBunT2McwtsqqpOAF4DckIdzBhjwkLfuyEvm4eSF/DjoaPMWL3b60SeCOZuqDuA\nycC/3FnNgBOepjbGmCqpUWdofR5nbH2bVnVjmDBvi9eJPBHMaah7cJ5zOASgquuBhqEMZYwxYaXf\nGOTQDn7fdiMLN+9j7Y/V783SwRSLY25HgACISDRgnbwbY6qPdhdDUmsG73+f2GgfE+dXv95ogykW\nX4rIb4GaInIh8B7wcWhjGWNMGPH5oO/dRO9czD3tDzJl2Q4OHsn1OlWlCqZYjAX24LyI6C6ch+x+\nH8pQxhgTdnrcALGJ3Oz7nOzcfN5but3rRJUqmLuhClT1VVW9RlVHuuN2GsoYU73EJkCPG6m76WMu\nbF7AGwu2UlBQfb4Kg+pI0BhjDNDnTijI5+Hkb9iacYQv1+/xOlGlsWJhjDHBqtcazryU9tvfo1m8\nMLEa3UZbbrEQkSgRebqywhhjTNjrNwY5ksG4lNWk/bCHLXurR+9H5RYL9811AyspizHGhL+UgdCo\nC0MOTCZK4I0F1eM22mBOQ30rIlNF5CYRubpwCHkyY4wJRyLQbwzRe9dwf9vdvLtkO0dy8rxOFXLB\nFIs4IAMYgtOB4BXA5aEMZYwxYa3LSKhVn5v4lMyjeXz47U6vE4VcwJcfqeptlRHEGGMiRkwc9Lqd\nul89xZCG1zFx/hau79MCEfE6WcgE05FgcxGZIiI/ucP7ItK8MsIZY0zY6v1zxBfN2HpfsvbHTBZu\n3ud1opAK5jTUf4CpQFN3+NidZ4wx1VdCY+hyNe12fkizmrlMnL/F60QhFUyxaKCq/1HVPHf4L9Ag\nxLmMMSb89b0byclifIvlTF+1m10Hs71OFDLBFIsMEbnRfeYiSkRuxLngbYwx1Vuzs6FFPwYf+AA0\nn7cWVt3XrgZTLG4HrgV+BHYBIwG76G2MMQD97ib60FYeaLGJtxdt41hevteJQqLMYiEi/+eO9lHV\nYaraQFUbquqVqhpU+RSRoSJSf132AAAdGklEQVSyTkQ2iMjYUpb/TUSWu8MPInLAb9ktIrLeHW45\n6U9mjDGVocMVkNicm2Qae7NymPb9Lq8ThUR5RxaXinMf2KOnsmERiQJeBC4BOgHXi0gn/zaq+itV\n7a6q3YHngQ/cdesB44C+QB9gnIgknUoOY4wJqaho6HMHdXcv4IJ6e5gwr2o+0V1esfgc2A+cJSKH\nRCTT/2cQ2+4DbFDVTe6b9iYBw8tpfz3wtjt+MTBTVfep6n5gJjA0iH0aY0zlO/tmiKnFr+vOYfn2\nA6zYfiDwOhGmzGKhqo+oal3gU1VNVNUE/59BbLsZ4P92kHR33glEpBXQGph9susaY4znatWDbqNo\nt/szmtc4XCVfuxrME9zlHQ2cLqOAyW7HhUETkTuBOwEaNWpEWlraKQfIysqq0PqVKZKyQmTljaSs\nEFl5IykrnHzeWnI2ffJf58GEL/j18tqcV3cfiTUq54nuyvjdBiwWFbADaOE33dydV5pRwD0l1k0t\nsW5ayZVU9RXgFYBevXppampqySZBS0tLoyLrV6ZIygqRlTeSskJk5Y2krHCKefdNYdiuNH5TcBnb\nY1pwT+oZIclWUmX8bkP58qPFQDsRaS0iNXAKwtSSjUSkA5AEzPebPR24SESS3AvbF7nzjDEmfPX7\nBdFHdvNA01W8uWArefkFXic6bYLpG6q2iPj8pn0iUivQeqqaB9yL8yW/BnhXVVeJyOMiMsyv6Shg\nkv97vVV1H/AETsFZDDzuzjPGmPDVdggkt+NG/ZSdB7P5Ys1PXic6bYI5DTULuADIcqdrATOAcwKt\nqKrTgGkl5j1WYnp8Geu+DrweRD5jjAkPPh/0u5s6nz7ExYnbmDi/PkO7NPY61WkR1PssVLWwUOCO\nBzyyMMaYaqnb9RBXh0fqzGbexgx+2J3pdaLTIphicVhEzi6cEJGeQNXtLcsYYyqiRm04+xba7p1N\ny+h9VaY32mCKxQPAeyLytYh8A7yDcy3CGGNMafrcgaCMbzyPD5bt4NDRXK8TVVjAYqGqi4EOwBjg\nbqCjqi4NdTBjjIlYdVtCxys4N/NTNOcw7y9N9zpRhQVzN9TNOF1xnO0O17vzjDHGlKXvGKKPHeT+\nBsuYOH8rBQUaeJ0wFsxpqN5+wyBgPDCsvBWMMabaa9kPmnTnBp3G5r1ZfL1hr9eJKiSY7j7u858W\nkbo4nQIaY4wpiwj0G0PilLu4rNZaJs5rxHntI/clo6fyBPdhnE7/jDHGlKfzVRDfiIcTZzF73U9s\nyzjidaJTFsw1i49FZKo7fAKsAz4MfTRjjIlw0bHQ6+e0PjCPtrKL/y2M3N5og3mC+2m/8Txgq6pG\n/qV9Y4ypDL1uh6+f5g8NvuL+xS351QXtqVkjyutUJy2YW2e/9BvmAiki8mIlZDPGmMgX3wC6XsPA\nwzPQ7P18tLyszrfDW1DXLESkh4g8JSJbcDr4WxvSVMYYU5X0vZuovGzuqzufCfO34tdvasQos1iI\nSHsRGScia3Hej70NEFUdrKrPV1pCY4yJdE3OglYDuV6m88Ou/SzZut/rRCetvCOLtcAQ4HJVHegW\niJN6k50xxhhXvzHEZ+9keNy3TJi3xes0J628YnE1sAuYIyKvisj5QOW8I9AYY6qaMy+Buq14IH42\nn6/8kd2Hjnqd6KSUWSxU9UNVHYXTL9QcnA4FG4rIP0XkosoKaIwxVYIvCvreRcus5XRgE28u3OZ1\nopMSzN1Qh1X1LVW9Audd2N8Cvwl5MmOMqWp63Ag14nk0KY23Fm4jJy9yXrt6Uk9wq+p+VX1FVc8P\nVSBjjKmy4upA9xvon52GZO3ms5W7vE4UtFPp7sMYY8yp6nsXUpDHPQlfRtSFbisWxhhTmZLbIu0v\n5jqZycpte/g+/aDXiYJixcIYYypbvzHUzNnHiBoLI+a1q1YsjDGmsrU+Dxp24r5aM/loxQ72H87x\nOlFAIS0WIjJURNaJyAYRGVtGm2tFZLWIrBKRt/zm54vIcneYGsqcxhhTqUSg7900Pbqe7vmreWfJ\ndq8TBRSyYiEiUcCLwCVAJ5zXsXYq0aYd8CgwQFU74zzLUShbVbu7g72ZzxhTtZx1LdSsx8N1ZvHG\n/K3kh/lrV0N5ZNEH2KCqm1Q1B+ftesNLtLkDeFFV9wOo6k8hzGOMMeEjpib0uo3eRxcgB7cya81u\nrxOVK5TFohngf2yV7s7z1x5oLyJzRWSBiAz1WxYnIkvc+VeGMKcxxnij92jw+fhFrVlMnB/eL0aS\nUHWVKyIjgaGqOtqdvgnoq6r3+rX5BMgFrsV5OvwroKuqHhCRZqq6Q0TaALOB81V1Y4l93AncCdCo\nUaOekyad+qvBs7KyiI+PP+X1K1MkZYXIyhtJWSGy8kZSVqi8vB1XP0PinsX0yn6B3w2sR9P4k/8b\nviJZBw8evFRVewVsqKohGYD+wHS/6UeBR0u0eRm4zW96FtC7lG39FxhZ3v569uypFTFnzpwKrV+Z\nIimramTljaSsqpGVN5KyqlZi3u2LVccl6h9/f78+9uH3p7SJimQFlmgQ3+mhPA21GGgnIq1FpAYw\nCih5V9OHQCqAiNTHOS21SUSSRCTWb/4AYHUIsxpjjDea94Lmvbm75he8v3QbmUdzvU5UqpAVC1XN\nA+4FpgNrgHdVdZWIPC4ihXc3TQcyRGQ1Ts+2j6hqBtARWCIiK9z5T6qqFQtjTNXU924a5qTTO28Z\nHywLz9euRody46o6DZhWYt5jfuMKPOgO/m3mAV1Dmc0YY8JGp+Ew4w/88uhMHpw/kJv7t0IkvF4f\nZE9wG2OM16JioM9ouucuJ2rvOuZuyPA60QmsWBhjTDjoeRsaHceYuBlMmL/F6zQnsGJhjDHhoFY9\n5KzruIKvWbpmA9v3HfE60XGsWBhjTLjoN4YYPcb1UbP538LwekgvpBe4vZabm0t6ejpHjwZ+MXqd\nOnVYs2ZNJaSquEjKCoHzxsXF0bx5c2JiYioxlTFhqGFHaJPK6K2zuHDRlfzqgvbExUR5nQqo4sUi\nPT2dhIQEUlJSAt5ZkJmZSUJCQiUlq5hIygrl51VVMjIySE9Pp3Xr1pWczJgw1O8XJG26lv45c5m6\noivX9mrhdSKgip+GOnr0KMnJyWF3C5opJiIkJycHdfRnTLVwxoVovbaMqTmTCfO2FPZi4bkqXSwA\nKxQRwP6NjPHj8yF976ZT/jpidi1j2bb9XicCqkGx8NKBAwd46aWXTmndSy+9lAMHDpzmRMaYiND9\nejQ2gTtjpzNhXnhc6LZiEULlFYu8vLxy1502bRp169YNRSxjTLiLTUDOvoWLWcDS71fx0yHvT9Na\nsQihsWPHsnHjRrp3784jjzxCWloagwYNYtiwYXTq5Lw08Morr6Rnz5507tyZV155pWjdlJQU9u7d\ny5YtW+jYsSN33HEHnTt35qKLLiI7O/uEfX388cf07duXHj16cMEFF7B7t/MilfHjx/P0008XtevS\npQtbtmwBYOLEiZx11ll069aNm266KYS/CWPMSetzBz5RrvfN4O1F3r92tUrfDeXvjx+vYvXOQ2Uu\nz8/PJyrq5G5R69Q0kXFXdC5z+ZNPPsnKlStZvnw5AGlpaSxbtoyVK1cW3fnz+uuvU69ePbKzs+nd\nuzcjRowgOTn5uO2sX7+et99+m1dffZVrr72Wjz76iDvuuOO4NgMHDmTBggWICK+99hp//etfeeaZ\nZ8rMtmrVKv70pz8xb9486tevz759+07qsxtjQiwpBTnzUm75YQ6XLriOMaltqRHt3d/31aZYhIs+\nffocd4voP/7xD6ZMmQLA9u3bWb9+/QnFonXr1nTv3h2Anj17sm3bthO2m56eznXXXceuXbvIyckJ\neBvq7Nmzueaaa6hfvz4A9erVq9DnMsaEQL8xJKz9hHOOzWH6qh5c0a2pZ1GqTbEo7wgAKu/Zhdq1\naxeNp6Wl8cUXXzB//nxq1apFampqqbeQxsbGFo1HRUWVer3jvvvu48EHH2TYsGGkpaUxfvx4AKKj\noykoKChqZ7eoGhNBWg1AG3flrt3T+c28qzwtFnbNIoQSEhLIzMwsc/nBgwdJSkqiVq1arF27lgUL\nFpzyvg4ePEizZs4rzidMmFA0PyUlhWXLlgGwbNkyNm/eDMCQIUN47733yMhwere001DGhCERpO8Y\n2ug2YrZ/w6qdBz2LYsUihJKTkxkwYABdunThkUceOWH50KFDycvLo2PHjowdO5Z+/fqd8r7Gjx/P\nNddcQ8+ePYtOLQGMGDGCffv20blzZ1544QXat28PQOfOnfnd737HeeedR7du3XjwwQfL2rQxxktd\nRlBQqwF3xHzOG/O9u4222pyG8spbb7113HRqamrReGxsLJ999lmp6xXesVS/fn1WrlxZNP/hhx8u\n9Whl+PDhDB8+/IT5NWvWZMaMGaXu45ZbbuGWW24J9BGMMV6KicPX+3bO+/KvPLl8CQcu6UDdWjUq\nPYYdWRhjTLjr9XPEF80o/Zx3l3hzG60VC2OMCXcJjZAuIxgV8xUfzF9NfkHl9xdlxcIYYyJBv7up\nqdkMOPQ5aet+qvTdW7EwxphI0LQHBS36cXvMDCbM3Vjpu7diYYwxEcLX/xc04yfiNs1g456syt13\npe7NGGPMqTvzMvITmnN79PRKv402pMVCRIaKyDoR2SAiY8toc62IrBaRVSLylt/8W0RkvTtUm/s7\n4+PjAdi5cycjR44stc2ll17KkiVLyt3Oc889x5EjxS98ty7PjakCoqKJ6ncX/XyrWbn0G7KOld97\n9ekUsmIhIlHAi8AlQCfgehHpVKJNO+BRYICqdgYecOfXA8YBfYE+wDgRSQpV1nDUtGlTJk+efMrr\nlywW1uW5MVXE2TeRH12Ta/M/Zcq3Oyptt6E8sugDbFDVTaqaA0wCSj41dgfwoqruB1DVwkv8FwMz\nVXWfu2wmMDSEWUNi7NixvPjii0XThd2FZ2Vlcf7553P22WfTtWtXPvrooxPW3bJlC126dAEgOzub\nUaNG0bFjR6666qrjuigfM2YMvXr1onPnzowbNw5wOifcuXMngwcPZvDgwUBxl+cAzz77LF26dKFL\nly4899xzRfsLVVfob731lnWFbszpUjMJX/cbuCp6LlO/WV5pr10N5RPczQD/p0fScY4U/LUHEJG5\nQBQwXlU/L2PdZiV3ICJ3AncCNGrUiLS0tOOW16lTp+hp59g54/D9tKrMsDUV8k7y7Z4FDTtzbPAf\ny1x++eWXM3bsWG6++WYAJk2axJQpU8jNzWXixIkkJiaSkZHBkCFDGDx4cNHrRTMzM8nKyqKgoIDM\nzExeeOEFYmJiWLRoEStXrmTQoEEcPnyYzMxMxo4dS7169cjPz+eKK65g6NCh3HbbbTzzzDN8/PHH\nJCcnk5mZiaqSlZXF6tWr+fe//82sWbNQVYYMGUKvXr2oW7cu69ev57XXXuPZZ5/llltu4X//+x+j\nRo067jN169aNmTNnIiJMmDCBP/3pT/zlL3/h2LFjxMTEFP2+CwoKyMrKYtGiRTz11FN88cUXJCcn\ns2/fvlKfQD969OgJ/35eyMrKCoscwYqkvJGUFcI7b01fD/ryGv32T+WfH8TTMjY75Fm97u4jGmgH\npALNga9EpGuwK6vqK8ArAL169VL/rjQA1qxZU9yTbEwNiCr74+bl5xFdzvJSxdSgRjk91Q4cOJCM\njAwyMzPZs2cPycnJdOzYkdzcXP7whz/w1Vdf4fP52LVrF0eOHKFx48aA0wFhfHw8Pp+PhIQEFi5c\nyP33309CQgL9+/enS5cu1K5dm4SEBN58801eeeUV8vLy2LVrF1u3bqV///6ICPHx8UWfv3D622+/\nZcSIEUX7GjlyJMuWLWPYsGG0bt2aAQMGANC3b1927959Qk+8W7ZsYfTo0cd1hZ6QkEBsbCyxsbFF\n7X0+H/Hx8cyZM4errrqKlJSUos9Wmri4OHr06HFyv/8QSEtLo+R/R+EskvJGUlYI/7z5GVO4eeMX\njDs8mk7JUSHPGspisQNo4Tfd3J3nLx1YqKq5wGYR+QGneOzAKSD+66ZVKM0lT5a7ODtEXZRfc801\nTJ48mR9//JHrrrsOgDfffJM9e/awdOlSYmJiSElJOaWuwzdv3szTTz/N4sWLSUpK4tZbb61QF+Ql\nu0Iv7TSUdYVuTHiI6j+GBhtHELtuKhkNLgz5/kJ5zWIx0E5EWotIDWAUMLVEmw9xi4KI1Mc5LbUJ\nmA5cJCJJ7oXti9x5Eee6665j0qRJTJ48mWuuuQZwuhNv2LAhMTExzJkzh61by78F7txzzy3qkHDl\nypVFHQseOnSI2rVrU6dOHXbv3n1cp4RldY8+aNAgPvzwQ44cOcLhw4eZMmUKgwYNCvrznEpX6FOm\nTLGu0I053dqeT269dtwe9RlztuWGfHchKxaqmgfci/MlvwZ4V1VXicjjIjLMbTYdyBCR1cAc4BFV\nzVDVfcATOAVnMfC4Oy/idO7cmczMTJo1a0aTJk0AuOGGG1iyZAldu3Zl4sSJdOjQodxtjBkzhqys\nLDp27Mhjjz1W9Na8bt260aNHDzp06MDPfvazolNIAHfeeSdDhw4tusBd6Oyzz+bWW2+lT58+9O3b\nl9GjR5/U6Z9T6Qr94Ycftq7QjTndRIjpP4auvs3I3jUhv9AtlXUlPdR69eqlJZ89WLNmDR07dgxq\n/cp6U97pEElZIbi8J/NvFUrhfp66pEjKG0lZIULy5hym4NlO7I3vRMN7S3/dQSAislRVewVqZ09w\nG2NMpKpRG9+A+8mu1QxC/Ie/13dDGWOMqYhBD7E5P41WcpL3/p8kO7IwxhgTUJUvFlXlmkxVZv9G\nxoS/Kl0s4uLiyMjIsC+jMKaqZGRkEBcX53UUY0w5qvQ1i+bNm5Oens6ePXsCtj169GjEfGFFUlYI\nnDcuLo7mzZtXYiJjzMmq0sUiJiaG1q1bB9U2LS0tLLqbCEYkZYXIy2uMOVGVPg1ljDHm9LBiYYwx\nJiArFsYYYwKqMt19iMgeoCIvpa0P7D1NcUItkrJCZOWNpKwQWXkjKStEVt6KZG2lqg0CNaoyxaKi\nRGRJMP2jhINIygqRlTeSskJk5Y2krBBZeSsjq52GMsYYE5AVC2OMMQFZsSj2itcBTkIkZYXIyhtJ\nWSGy8kZSVoisvCHPatcsjDHGBGRHFsYYYwKyYuESkfEiskNElrvDpV5nCoaIPCQi6r7DPCyJyBMi\n8p37e50hIk29zlQeEXlKRNa6maeISF2vM5VFRK4RkVUiUiAiYXvnjogMFZF1IrJBRMZ6nacsIvK6\niPwkIiu9zhIMEWkhInNEZLX738EvQ7UvKxbH+5uqdneHaV6HCUREWgAXAdu8zhLAU6p6lqp2Bz4B\nHvM6UAAzgS6qehbwA/Cox3nKsxK4GvjK6yBlEZEo4EXgEqATcL2IdPI2VZn+Cwz1OsRJyAMeUtVO\nQD/gnlD9bq1YRLa/Ab8GwvrCk6oe8pusTfjnnaGqee7kAiBsu8RV1TWqus7rHAH0ATao6iZVzQEm\nAcM9zlQqVf0K2Od1jmCp6i5VXeaOZwJrgGah2JcVi+Pd6556eF1EkrwOUx4RGQ7sUNUVXmcJhoj8\nWUS2AzcQ/kcW/m4HPvM6RIRrBmz3m04nRF9o1ZmIpAA9gIWh2H6V7qK8JBH5AmhcyqLfAf8EnsD5\nq/cJ4BmcLwrPBMj7W5xTUGGhvKyq+pGq/g74nYg8CtwLjKvUgCUEyuu2+R3OYf6blZmtpGCymupN\nROKB94EHShzJnzbVqlio6gXBtBORV3HOrXuqrLwi0hVoDawQ5yXtzYFlItJHVX+sxIhFgv3d4nzx\nTsPjYhEor4jcClwOnK8e319+Er/bcLUDaOE33dydZ04DEYnBKRRvquoHodqPnYZyiUgTv8mrcC4c\nhiVV/V5VG6pqiqqm4BzWn+1VoQhERNr5TQ4H1nqVJRgiMhTnWtAwVT3idZ4qYDHQTkRai0gNYBQw\n1eNMVYI4fy3+G1ijqs+GdF/2UJ5DRN4AuuOchtoC3KWquzwNFSQR2QL0UtWw7CFTRN4HzgQKcHoG\nvltVw/YvSxHZAMQCGe6sBap6t4eRyiQiVwHPAw2AA8ByVb3Y21Qncm9Ffw6IAl5X1T97HKlUIvI2\nkIrTi+tuYJyq/tvTUOUQkYHA18D3OP9/Afw2FHdzWrEwxhgTkJ2GMsYYE5AVC2OMMQFZsTDGGBOQ\nFQtjjDEBWbEwxhgTkBULc9JEJCvE2+/g9lD7rYi0rcx9RyoRubWivfm623jhNOXZEkxPyPbvGTms\nWJhwdCUwWVV7qOpGr0KIyGnt4eB0b6+EW4GTKhYhzmOqGCsW5pSJ4ykRWSki34vIde58n4i85L4T\nYqaITBORkaWs311EFvi9NyLJfXjrAWCMiMwpY79/FpEV7rqNRCRBRDa73R4gIomF0yKSJiJ/d49U\nVopIH7dNbbfDyEXuEcxwd/6tIjJVRGYDs0rsN8X9TG+KyBoRmSwitdxlj4nIYncfr7hP1uLu/zkR\nWQL8UkSuEJGF7j6/EJFGbrvxIjJBRL4Wka0icrWI/NX9vX7u99l6isiXIrJURKaLSBP3d9sLeNP9\nnDVLa1dannL+bSuU0/Vrd/4iETnDXb+1iMx35//Jb3/xIjJLRJa5y8KyV9pqTVVtsOGkBiDL/TkC\n590PUUAjnPdqNAFG4vT/5MPpAG8/MLKU7XwHnOeOPw48546PBx4uY98KXOGO/xX4vTv+H+BKd/xO\n4Bl3PA141R0/F1jpjv8FuNEdr4vz3oraOH+hpwP1Stl3irv/Ae7064U5/dsDb/hlTANe8luWRPHD\nsKP9co4HvgFigG7AEeASd9kUnKOtGGAe0MCdfx3O09CF++nljgdq91IZv9tbgRcqmtMd34LT0SHA\nzcAn7vhU4GZ3/B6K/1uKBhLd8frAhsL92xAegx2GmooYCLytqvnAbhH5Eujtzn9PVQuAH0s7QhCR\nOkBdVf3SnTUBeC+IfeZQ3MnjUuBCd/w1nP6cPgRuA+7wW+dtcN5V4B511MXpsXeYiDzstokDWrrj\nM1W1rHcabFfVue74/4D7gaeBwSLya6AWUA9YBXzstnvHb/3mwDvuX/o1gM1+yz5T1VwR+R6nAH/u\nzv8ep1CdCXQBZroHLlFAaV3SBGr3TinrlFSRnIXe9vv5N3d8AM4fGeAU1f9zxwX4i4ici9NtRTOc\nP0DCsr+z6siKhYk0uer++Qnk4/43rKpz3dNEqUCUqvp3BFmyTxvF+XIaoSVeHCQifYHD5ez/hG2J\nSBzwEs5f9ttFZDxO8Snkv73ngWdVdaqbdbzfsmPuZykQEf/PWeB+TgFWqWr/cvIRRLvyPt/pyFlI\ngxgvdANO/1Y93UK0heN/h8Zjds3CVMTXwHUiEiUiDXBO8ywC5gIjxLl20QinY7bjqOpBYL+IDHJn\n3QR8WbLdSZoIvIVzSspf4bWUgcBBd9/Tgfv8ri30CHIfLUWk8Ev4ZzinZAq/1PaK816BE67P+KlD\ncffctwS5z0LrgAaF+xfnmkxnd1kmkBBEu2BVJGeh6/x+znfH5+L0OgtOgfDf309uoRgMtDrFfZoQ\nsWJhKmIKznWHFcBs4NfqdJP+Ps55/9U4p2qWAQdLWf8W4CkR+Q6nx9/HK5jnTZxz7W+XmH9URL4F\nXgZ+7s57Aue8+3cissqdDsY6nPccr3H39U9VPQC8itOt/XScLrnLMh54T0SWAifVS7A6ryQdCfyf\niKwAlgPnuIv/C7wsIstxTg2V1S5Yp5zTT5L7b/tL4FfuvF/i/P6+5/i35b0J9HLn30yYd2NfHVmv\nsyYkRCReVbNEJBnnaGOAhvh9G+5dQcNV9Sa/eWk4F6GXnIbtp+BcqO1S0W0ZE2nsmoUJlU/cC8k1\ngCcqoVA8D1wCXBrK/RhTXdmRhTHGmIDsmoUxxpiArFgYY4wJyIqFMcaYgKxYGGOMCciKhTHGmICs\nWBhjjAno/wP/fXQXyKlcMwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import math as m\n",
    "para = [m.log10(x) for x in param]\n",
    "plt.plot(para,train_auc,label='train auc')\n",
    "plt.plot(para,cv_auc,label='validation auc')\n",
    "plt.xlabel(\"log of hyper parameter lambda\")\n",
    "plt.ylabel(\"Auc for each lambda\")\n",
    "plt.grid()\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "yMaVQWZaYj_0",
    "outputId": "caf9fd52-68cb-4c25-9c52-582176858f40"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test auc for naive bayes model is 0.8755791924694993\n"
     ]
    }
   ],
   "source": [
    "model = MultinomialNB(alpha=1)\n",
    "model.fit(train_vect1, Y_train)\n",
    "predict_y_test = model.predict_log_proba(test_vect1)[:,1] # Taking probability for positive class\n",
    "print(\"Test auc for naive bayes model is {}\".format(roc_auc_score(Y_test,predict_y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Jbotoes8YkB_"
   },
   "source": [
    "## Using LSTM Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2WhBWo9WcO-9"
   },
   "outputs": [],
   "source": [
    "#downloaded = drive.CreateFile({'id':id}) \n",
    "#glove_file = downloaded.GetContentFile('glove.6B.100d.txt') \n",
    "\n",
    "f = open(\"/content/drive/My Drive/Project/glove.6B.300d.txt\",'r',encoding=\"utf-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zIRU0rsTYkB_"
   },
   "outputs": [],
   "source": [
    "# Tokenizing the essay text data for train dataset\n",
    "# We will train the vectorizer on train data and will use the same on test and cv data.\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "# Tokenizing the essay text data for train dataset\n",
    "# We will train the vectorizer on train data and will use the same on test and cv data.\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.layers import Dense,Input\n",
    "from numpy import asarray\n",
    "t  = Tokenizer(num_words=40000)\n",
    "t.fit_on_texts(train_x)\n",
    "vocab_size = len(t.word_index)+1\n",
    "# Integer coding all the words.\n",
    "encoded_essay = t.texts_to_sequences(train_x)\n",
    "# defining a max size for padding.\n",
    "max_len = 150\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "train_sequence = pad_sequences(encoded_essay,maxlen = max_len,padding='post')\n",
    "\n",
    "# Vectorizing test data\n",
    "# Integer coding all the words.\n",
    "encoded_essay = t.texts_to_sequences(test_x)\n",
    "# defining a max size for padding.\n",
    "max_len = 150\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "test_sequence = pad_sequences(encoded_essay,maxlen = max_len,padding='post')\n",
    "\n",
    "# to use cv datset un-comment this.\n",
    "\n",
    "# Vectorizing cv data\n",
    "# Integer coding all the words.\n",
    "#encoded_essay = t.texts_to_sequences(cv_x)\n",
    "# defining a max size for padding.\n",
    "#max_len = 150\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "#cv_sequence = pad_sequences(encoded_essay,maxlen = max_len,padding='post')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "WFLf1fXSYkCC",
    "outputId": "29778341-47c4-44c2-e016-9b79770a9f8e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "# we will load the whole glove vectors .\n",
    "embeddings_index1 = {}\n",
    "# Opening the file\n",
    "f = open(\"/content/drive/My Drive/Project/glove.6B.300d.txt\",'r',encoding=\"utf-8\")\n",
    "for line in f:\n",
    "\tvalues = line.split()\n",
    "\tword = values[0]\n",
    "\tcoefs = asarray(values[1:], dtype='float32')\n",
    "\tembeddings_index1[word] = coefs\n",
    "f.close()\n",
    "\n",
    "\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "import numpy as np\n",
    "embedding_matrix1 = np.zeros((vocab_size, 300))\n",
    "for word, i in t.word_index.items():\n",
    "  for wrd in [word, word.lower()]:\n",
    "    if wrd in embedding_matrix1:\n",
    "\t    embedding_matrix1[i] = embeddings_index1.get(wrd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "Ox3erkv-an7h",
    "outputId": "bc590372-30ba-4220-ab66-846128cf9b61"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:16: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "  app.launch_new_instance()\n"
     ]
    }
   ],
   "source": [
    "#  Loading crawl vectors\n",
    "from gensim.models import KeyedVectors\n",
    "path = \"/content/drive/My Drive/Project/crawl-300d-2M.gensim\"\n",
    "embeddings_index2 = KeyedVectors.load(path, mmap='r')\n",
    "embedding_matrix2 = np.zeros((len(t.word_index) + 1, 300))\n",
    "for word, i in t.word_index.items():\n",
    "    for candidate in [word, word.lower()]:\n",
    "        if candidate in embeddings_index2:\n",
    "            embedding_matrix2[i] = embeddings_index2[candidate]\n",
    "            break\n",
    "            \n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "import numpy as np\n",
    "embedding_matrix2 = np.zeros((vocab_size, 300))\n",
    "for word, i in t.word_index.items():\n",
    "  for wrd in [word, word.lower()]:\n",
    "    if wrd in embedding_matrix2:\n",
    "\t    embedding_matrix2[i] = embeddings_index2.get(wrd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l7HvsMTBZSHx"
   },
   "outputs": [],
   "source": [
    "embedding_matrix = np.concatenate([embedding_matrix1, embedding_matrix2], axis= -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "yxqo9dvjZSQi",
    "outputId": "be77559e-32cb-4f07-e703-be2ccb873fe4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(252210, 600)"
      ]
     },
     "execution_count": 39,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "fda3L5s7sZ1q",
    "outputId": "243d8732-9d6d-4dd3-8889-ebb39394fac1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7097320</td>\n",
       "      <td>[ Integrity means that you pay your debts.]\\n\\...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7097321</td>\n",
       "      <td>This is malfeasance by the Administrator and t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7097322</td>\n",
       "      <td>@Rmiller101 - Spoken like a true elitist. But ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7097323</td>\n",
       "      <td>Paul: Thank you for your kind words.  I do, in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7097324</td>\n",
       "      <td>Sorry you missed high school. Eisenhower sent ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                       comment_text\n",
       "0  7097320  [ Integrity means that you pay your debts.]\\n\\...\n",
       "1  7097321  This is malfeasance by the Administrator and t...\n",
       "2  7097322  @Rmiller101 - Spoken like a true elitist. But ...\n",
       "3  7097323  Paul: Thank you for your kind words.  I do, in...\n",
       "4  7097324  Sorry you missed high school. Eisenhower sent ..."
      ]
     },
     "execution_count": 36,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#from google.colab import drive\n",
    "#drive.mount('/content/drive')\n",
    "test_df = pd.read_csv(\"/content/drive/My Drive/Project/test.csv\")\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "nnMbfg7gsZxb",
    "outputId": "b1013db2-2b1a-4769-877b-e97b81192c87"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 97320/97320 [00:05<00:00, 16647.45it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7097320</td>\n",
       "      <td>integrity means that you pay your debts does ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7097321</td>\n",
       "      <td>this malfeasance the administrator and the bo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7097322</td>\n",
       "      <td>rmiller spoken like true elitist but look out...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7097323</td>\n",
       "      <td>paul thank you for your kind words indeed hav...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7097324</td>\n",
       "      <td>sorry you missed high school eisenhower sent ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                       comment_text\n",
       "0  7097320   integrity means that you pay your debts does ...\n",
       "1  7097321   this malfeasance the administrator and the bo...\n",
       "2  7097322   rmiller spoken like true elitist but look out...\n",
       "3  7097323   paul thank you for your kind words indeed hav...\n",
       "4  7097324   sorry you missed high school eisenhower sent ..."
      ]
     },
     "execution_count": 37,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preprocessing the comments.\n",
    "test_preprocessed = []\n",
    "bad_found = []\n",
    "from tqdm import tqdm\n",
    "for sent in tqdm(test_df['comment_text'].values):\n",
    "    sent = process_sent(sent)\n",
    "    line = ''\n",
    "    for wrd in sent.split():\n",
    "        if(len(wrd)>2):\n",
    "            line += \" \" +wrd.lower()\n",
    "    line = re.sub(r\"[']\",'',line)\n",
    "    test_preprocessed.append(line)\n",
    "\n",
    "test_df['comment_text'] = test_preprocessed\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wcpFgf1SddiJ"
   },
   "outputs": [],
   "source": [
    "\n",
    "# Vectorizing test data\n",
    "# Integer coding all the words.\n",
    "encoded_essay1 = t.texts_to_sequences(test_df['comment_text'])\n",
    "# defining a max size for padding.\n",
    "max_len = 150\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "test_sequence1 = pad_sequences(encoded_essay1,maxlen = max_len,padding='post')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NI6ijwfIs9qW"
   },
   "source": [
    "# First Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "id": "k5jhe5y7ekrq",
    "outputId": "1f05d16b-8165-4dc5-f72e-4d9e1fed0175"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_6 (InputLayer)         (None, 150)               0         \n",
      "_________________________________________________________________\n",
      "embedding_6 (Embedding)      (None, 150, 100)          4000000   \n",
      "_________________________________________________________________\n",
      "bidirectional_6 (Bidirection (None, 150, 256)          234496    \n",
      "_________________________________________________________________\n",
      "seq_self_attention_5 (SeqSel (None, 150, 256)          16449     \n",
      "_________________________________________________________________\n",
      "flatten_5 (Flatten)          (None, 38400)             0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1)                 38401     \n",
      "=================================================================\n",
      "Total params: 4,289,346\n",
      "Trainable params: 4,289,346\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM, GRU\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.layers import MaxPooling1D\n",
    "from keras.layers import Flatten, GlobalMaxPooling1D, GlobalAveragePooling1D\n",
    "from keras.layers import Dropout, BatchNormalization, Input, SpatialDropout1D\n",
    "from keras.layers import Dense, Bidirectional, concatenate\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n",
    "from keras_self_attention import SeqSelfAttention\n",
    "import keras.backend as K\n",
    "\n",
    "embedding_vector_length = 100\n",
    "input1 = Input(shape=(150,))\n",
    "e1 = Embedding(40000,embedding_vector_length, input_length=150)(input1)\n",
    "x1 = Bidirectional(LSTM(128, return_sequences=True, dropout=0.2, recurrent_dropout=0.2))(e1)\n",
    "att = SeqSelfAttention(attention_activation='sigmoid')(x1)\n",
    "\n",
    "x1 = Flatten()(att)\n",
    "output = Dense(1, activation='sigmoid')(x1)\n",
    "model = Model(inputs=[input1,], outputs=[output])\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy',])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "id": "bdFfNlk1e7au",
    "outputId": "06844879-852b-48eb-d05d-db35d0a50128"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1282192 samples, validate on 142466 samples\n",
      "Epoch 1/5\n",
      "1282192/1282192 [==============================] - 2665s 2ms/step - loss: 0.1483 - acc: 0.9454 - val_loss: 0.1334 - val_acc: 0.9491\n",
      "Epoch 2/5\n",
      "1282192/1282192 [==============================] - 2657s 2ms/step - loss: 0.1256 - acc: 0.9515 - val_loss: 0.1307 - val_acc: 0.9495\n",
      "Epoch 3/5\n",
      "1282192/1282192 [==============================] - 2654s 2ms/step - loss: 0.1155 - acc: 0.9552 - val_loss: 0.1330 - val_acc: 0.9494\n",
      "Epoch 4/5\n",
      "1282192/1282192 [==============================] - 2659s 2ms/step - loss: 0.1037 - acc: 0.9600 - val_loss: 0.1407 - val_acc: 0.9477\n",
      "Epoch 5/5\n",
      "1282192/1282192 [==============================] - 2665s 2ms/step - loss: 0.0924 - acc: 0.9646 - val_loss: 0.1522 - val_acc: 0.9463\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f0bd2deaf98>"
      ]
     },
     "execution_count": 67,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_sequence, Y_train, nb_epoch=5, batch_size=512, validation_data=(cv_sequence, Y_cv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "p1W7ZufTe7Wt",
    "outputId": "8f04495f-0790-4150-d9a8-88d3c53fa7aa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9358755310733398\n"
     ]
    }
   ],
   "source": [
    "# Roc auc\n",
    "predicted_y = model.predict(test_sequence,batch_size=512)\n",
    "from sklearn.metrics import roc_auc_score\n",
    "print(roc_auc_score(Y_test,predicted_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "smCQ902La-cO",
    "outputId": "1caba7ca-3263-4e86-cd3c-0a00b67e7974"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8823309859591929\n"
     ]
    }
   ],
   "source": [
    "eval = JigsawEvaluator(y_binary, y_identity_binary)\n",
    "print(eval.get_final_metric(predicted_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Al19mijyZq9i"
   },
   "outputs": [],
   "source": [
    "y_pre = model.predict(test_sequence1,batch_size=512)\n",
    "submission = pd.DataFrame({'id': test_df['id'].values})\n",
    "submission['prediction'] = y_pre\n",
    "submission.to_csv(\"submissionNew2.csv\",index=False) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Mled4ZmNZrLu"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download( \"/content/submissionNew2.csv\" )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s_DWPQ8JNOGY"
   },
   "source": [
    "## First Bi-LSTM architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 454
    },
    "colab_type": "code",
    "id": "qyOSj3q2YkCL",
    "outputId": "d348acb4-f330-4c5d-895d-f586a0639b0e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_11 (Embedding)     (None, 200, 64)           6223104   \n",
      "_________________________________________________________________\n",
      "bidirectional_10 (Bidirectio (None, 256)               197632    \n",
      "_________________________________________________________________\n",
      "dropout_15 (Dropout)         (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 256)               1024      \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 768)               197376    \n",
      "_________________________________________________________________\n",
      "dropout_16 (Dropout)         (None, 768)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 768)               3072      \n",
      "_________________________________________________________________\n",
      "dense_16 (Dense)             (None, 1)                 769       \n",
      "=================================================================\n",
      "Total params: 6,622,977\n",
      "Trainable params: 6,620,929\n",
      "Non-trainable params: 2,048\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.layers import MaxPooling1D\n",
    "from keras.layers import Flatten, GlobalMaxPool1D\n",
    "from keras.layers import Dropout, BatchNormalization\n",
    "from keras.layers import Dense, Bidirectional\n",
    "from keras.models import Model\n",
    "\n",
    "embedding_vecor_length = 64\n",
    "model = Sequential()\n",
    "model.add(Embedding(vocab_size, embedding_vecor_length, input_shape=(150,)))\n",
    "model.add(Bidirectional(LSTM(128)))\n",
    "model.add(Dropout(0.3))\n",
    "model.add(GlobalMaxPllo1D())\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dense(768,activation='relu'))\n",
    "model.add(Dropout(0.50))\n",
    "model.add(GlobalMaxPllo1D())\n",
    "model.add(BatchNormalization())\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "id": "MiMeplYRYkCN",
    "outputId": "3742c13c-e815-4020-cdb1-648b419f403b"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 190911 samples, validate on 47728 samples\n",
      "Epoch 1/5\n",
      "190911/190911 [==============================] - 266s 1ms/step - loss: 0.2945 - acc: 0.8903 - val_loss: 0.2760 - val_acc: 0.8859\n",
      "Epoch 2/5\n",
      "190911/190911 [==============================] - 259s 1ms/step - loss: 0.1441 - acc: 0.9477 - val_loss: 0.1511 - val_acc: 0.9442\n",
      "Epoch 3/5\n",
      "190911/190911 [==============================] - 258s 1ms/step - loss: 0.1232 - acc: 0.9523 - val_loss: 0.1696 - val_acc: 0.9411\n",
      "Epoch 4/5\n",
      "190911/190911 [==============================] - 257s 1ms/step - loss: 0.1061 - acc: 0.9584 - val_loss: 0.1763 - val_acc: 0.9383\n",
      "Epoch 5/5\n",
      "190911/190911 [==============================] - 253s 1ms/step - loss: 0.0857 - acc: 0.9663 - val_loss: 0.1948 - val_acc: 0.9394\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f28de5322e8>"
      ]
     },
     "execution_count": 72,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_sequence, Y_train, nb_epoch=5, batch_size=512, validation_data=(cv_sequence, Y_cv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "FKNc-qIudkz1",
    "outputId": "2c83af66-177a-4150-acce-4724449bb946"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9038698075698687\n"
     ]
    }
   ],
   "source": [
    "# Roc auc\n",
    "predicted_y = model.predict(test_sequence,batch_size=512)\n",
    "from sklearn.metrics import roc_auc_score\n",
    "print(roc_auc_score(Y_test,predicted_y))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vdos0aE0NJdS"
   },
   "source": [
    "## Second Bi-LSTM architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 612
    },
    "colab_type": "code",
    "id": "0feWATevhwF6",
    "outputId": "973a4754-0ff6-4137-8d28-422df2502a7d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_3 (InputLayer)            (None, 150)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 150, 100)     40000000    input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 150, 100)     40000000    input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_5 (Bidirectional) (None, 150, 400)     481600      embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_6 (Bidirectional) (None, 150, 400)     481600      embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dropout_7 (Dropout)             (None, 150, 400)     0           bidirectional_5[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dropout_8 (Dropout)             (None, 150, 400)     0           bidirectional_6[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_5 (GlobalM (None, 400)          0           dropout_7[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_6 (GlobalM (None, 400)          0           dropout_8[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_3 (Concatenate)     (None, 800)          0           global_max_pooling1d_5[0][0]     \n",
      "                                                                 global_max_pooling1d_6[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 75)           60075       concatenate_3[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dropout_9 (Dropout)             (None, 75)           0           dense_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 1)            76          dropout_9[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 81,023,351\n",
      "Trainable params: 81,023,351\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.layers import MaxPooling1D\n",
    "from keras.layers import Flatten, GlobalMaxPool1D\n",
    "from keras.layers import Dropout, BatchNormalization, Input\n",
    "from keras.layers import Dense, Bidirectional, concatenate\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "embedding_vector_length = 100\n",
    "input1 = Input(shape=(150,))\n",
    "x1 = Embedding(vocab_size,embedding_vector_length, input_length=150)(input1)\n",
    "x1 = Bidirectional(LSTM(200,return_sequences=True))(x1)\n",
    "x1 = Dropout(0.3)(x1)\n",
    "x1 = GlobalMaxPool1D()(x1)\n",
    "\n",
    "#input2 = Input(shape=(150,))\n",
    "x2 = Embedding(vocab_size,embedding_vector_length, input_length=150)(input1)\n",
    "x2 = Bidirectional(LSTM(200,return_sequences=True))(x2)\n",
    "x2 = Dropout(0.5)(x2)\n",
    "x2 = GlobalMaxPool1D()(x2)\n",
    "\n",
    "x = concatenate([x1, x2])\n",
    "x = Dense(75, activation='relu')(x)\n",
    "x = Dropout(0.2)(x)\n",
    "output = Dense(1, activation='sigmoid')(x)\n",
    "model3 = Model(inputs=[input1,], outputs = output)\n",
    "#adam = Adam(lr=0.0001)\n",
    "model3.compile(loss='binary_crossentropy', optimizer= 'adam', metrics=['accuracy'])\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "QFz3kfS0hwEA",
    "outputId": "30e05a7f-ced7-4d8a-c25e-38c6b7b35d2d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1282192 samples, validate on 142466 samples\n",
      "Epoch 1/4\n",
      "1282192/1282192 [==============================] - 4279s 3ms/step - loss: 0.1440 - acc: 0.9459 - val_loss: 0.1568 - val_acc: 0.9472\n",
      "Epoch 2/4\n",
      "1282192/1282192 [==============================] - 4267s 3ms/step - loss: 0.1159 - acc: 0.9536 - val_loss: 0.1421 - val_acc: 0.9489\n",
      "Epoch 3/4\n",
      "1282192/1282192 [==============================] - 4268s 3ms/step - loss: 0.0956 - acc: 0.9610 - val_loss: 0.1463 - val_acc: 0.9430\n",
      "Epoch 4/4\n",
      "1282192/1282192 [==============================] - 4270s 3ms/step - loss: 0.0711 - acc: 0.9714 - val_loss: 0.1569 - val_acc: 0.9380\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f9dcd6d9b38>"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model3.fit(train_sequence, Y_train, epochs=4, batch_size=512, validation_data=[cv_sequence, Y_cv])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "k6amyC__cyws",
    "outputId": "eac82e49-bb03-453d-84bc-ffc8427aede3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9373771542809413\n"
     ]
    }
   ],
   "source": [
    "# Roc auc\n",
    "predicted_y = model3.predict(test_sequence,batch_size=512)\n",
    "from sklearn.metrics import roc_auc_score\n",
    "print(roc_auc_score(Y_test,predicted_y))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uwbZScEQ2Pdk"
   },
   "outputs": [],
   "source": [
    "y_pre = model3.predict(test_sequence1,batch_size=512)\n",
    "submission = pd.DataFrame({'id': test_df['id'].values})\n",
    "submission['prediction'] = y_pre\n",
    "submission.to_csv(\"submission11.csv\",index=False) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PHOY6RKX7k0V"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download( \"/content/submission11.csv\" )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Fkuzww6cN1-s"
   },
   "source": [
    "## Third Bi-LSTM architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 887
    },
    "colab_type": "code",
    "id": "V8oZknWteWTD",
    "outputId": "c140ad09-5915-4258-f795-07fde471b1ea"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0830 05:13:28.912689 139808314402688 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n",
      "W0830 05:13:28.966059 139808314402688 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "W0830 05:13:28.973107 139808314402688 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "W0830 05:13:30.026833 139808314402688 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:148: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
      "\n",
      "W0830 05:13:30.035221 139808314402688 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "W0830 05:13:30.064623 139808314402688 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "W0830 05:13:30.086747 139808314402688 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3657: The name tf.log is deprecated. Please use tf.math.log instead.\n",
      "\n",
      "W0830 05:13:30.092208 139808314402688 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 150)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 150, 100)     40000000    input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 150, 100)     40000000    input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional) (None, 150, 256)     234496      embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_2 (Bidirectional) (None, 150, 256)     234496      embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 256)          0           bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           bidirectional_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 512)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 64)           32832       concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 64)           0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 1)            65          dropout_1[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 80,501,889\n",
      "Trainable params: 80,501,889\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.layers import MaxPooling1D\n",
    "from keras.layers import Flatten, GlobalMaxPool1D\n",
    "from keras.layers import Dropout, BatchNormalization, Input\n",
    "from keras.layers import Dense, Bidirectional, concatenate\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "embedding_vector_length = 100\n",
    "input1 = Input(shape=(150,))\n",
    "x1 = Embedding(vocab_size,embedding_vector_length, input_length=150)(input1)\n",
    "x1 = Bidirectional(LSTM(128,return_sequences=True))(x1)\n",
    "x1 = GlobalMaxPool1D()(x1)\n",
    "\n",
    "\n",
    "x2 = Embedding(vocab_size,embedding_vector_length, input_length=150)(input1)\n",
    "x2 = Bidirectional(LSTM(128,return_sequences=True))(x2)\n",
    "x2 = GlobalMaxPool1D()(x2)\n",
    "\n",
    "x = concatenate([x1, x2])\n",
    "x = Dense(64, activation='relu')(x)\n",
    "x = Dropout(0.1)(x)\n",
    "output = Dense(1, activation='sigmoid')(x)\n",
    "model4 = Model(inputs=[input1,], outputs = output)\n",
    "#adam = Adam(lr=0.0001)\n",
    "model4.compile(loss='binary_crossentropy', optimizer= 'adam', metrics=['accuracy'])\n",
    "model4.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "AmWZsrnveWXr",
    "outputId": "e9634ddb-025d-4ca6-f97e-2409dae86cbd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1282192 samples, validate on 142466 samples\n",
      "Epoch 1/4\n",
      "1282192/1282192 [==============================] - 3407s 3ms/step - loss: 0.1435 - acc: 0.9465 - val_loss: 0.1267 - val_acc: 0.9496\n",
      "Epoch 2/4\n",
      "1282192/1282192 [==============================] - 3402s 3ms/step - loss: 0.1150 - acc: 0.9541 - val_loss: 0.1284 - val_acc: 0.9495\n",
      "Epoch 3/4\n",
      "1282192/1282192 [==============================] - 3397s 3ms/step - loss: 0.0936 - acc: 0.9622 - val_loss: 0.1432 - val_acc: 0.9477\n",
      "Epoch 4/4\n",
      "1282192/1282192 [==============================] - 3387s 3ms/step - loss: 0.0662 - acc: 0.9737 - val_loss: 0.1730 - val_acc: 0.9422\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f2704459c50>"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model4.fit(train_sequence, Y_train, epochs=4, batch_size=512, validation_data=[cv_sequence, Y_cv])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "I7rLqgNKDf-5",
    "outputId": "e72956f1-71b4-439a-d43c-bab9ff8dba47"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9968805599682502\n"
     ]
    }
   ],
   "source": [
    "# Train auc\n",
    "predicted_y = model4.predict(train_sequence,batch_size=512)\n",
    "from sklearn.metrics import roc_auc_score\n",
    "print(roc_auc_score(Y_train,predicted_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "oZe8EaOlE5MS",
    "outputId": "393f33fc-b55b-4aac-f5d1-9626ebd8c5f8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.937064717518582\n"
     ]
    }
   ],
   "source": [
    "# Roc auc\n",
    "predicted_y = model4.predict(test_sequence,batch_size=512)\n",
    "from sklearn.metrics import roc_auc_score\n",
    "print(roc_auc_score(Y_test,predicted_y))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "whFv3tqHTB0P"
   },
   "outputs": [],
   "source": [
    "y_pre = model4.predict(test_sequence1,batch_size=512)\n",
    "submission = pd.DataFrame({'id': test_df['id'].values})\n",
    "submission['prediction'] = y_pre\n",
    "submission.to_csv(\"submission12.csv\",index=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qFIT6L52X6Cr"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download( \"/content/submission12.csv\" )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "udNfnvCt77z9"
   },
   "source": [
    "# Fourth Arcitecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 326
    },
    "colab_type": "code",
    "id": "-kNfyrgT43rj",
    "outputId": "76075791-f108-4bd4-acaa-b9c254ca7a78"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting keras-self-attention\n",
      "  Downloading https://files.pythonhosted.org/packages/44/3e/eb1a7c7545eede073ceda2f5d78442b6cad33b5b750d7f0742866907c34b/keras-self-attention-0.42.0.tar.gz\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-self-attention) (1.16.5)\n",
      "Requirement already satisfied: Keras in /usr/local/lib/python3.6/dist-packages (from keras-self-attention) (2.2.5)\n",
      "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-self-attention) (1.3.1)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras->keras-self-attention) (3.13)\n",
      "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-self-attention) (1.0.8)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-self-attention) (1.12.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-self-attention) (1.1.0)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras->keras-self-attention) (2.8.0)\n",
      "Building wheels for collected packages: keras-self-attention\n",
      "  Building wheel for keras-self-attention (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for keras-self-attention: filename=keras_self_attention-0.42.0-cp36-none-any.whl size=17296 sha256=f2ca30cf1ecfb1b0c27d993423542039fcdd4b2f8610622d52962d7c473f17df\n",
      "  Stored in directory: /root/.cache/pip/wheels/7b/05/a0/99c0cf60d383f0494e10eca2b238ea98faca9a1fe03cac2894\n",
      "Successfully built keras-self-attention\n",
      "Installing collected packages: keras-self-attention\n",
      "Successfully installed keras-self-attention-0.42.0\n"
     ]
    }
   ],
   "source": [
    "!pip install keras-self-attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 731
    },
    "colab_type": "code",
    "id": "9l43xwPS43kK",
    "outputId": "f39eb9b1-50a7-479b-c444-d157f7d79b7a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"flatten_10/Reshape:0\", shape=(?, ?), dtype=float32)\n",
      "Model: \"model_4\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_18 (InputLayer)           (None, 150)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_18 (Embedding)        (None, 150, 300)     71461200    input_18[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_30 (Bidirectional (None, 150, 240)     404160      embedding_18[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_31 (Bidirectional [(None, 150, 120), ( 108360      bidirectional_30[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_22 (Concatenate)    (None, 120)          0           bidirectional_31[0][1]           \n",
      "                                                                 bidirectional_31[0][2]           \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_7 (Glo (None, 120)          0           bidirectional_31[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_14 (Global (None, 120)          0           bidirectional_31[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "reshape_12 (Reshape)            (None, 1, 120)       0           concatenate_22[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "reshape_13 (Reshape)            (None, 1, 120)       0           global_average_pooling1d_7[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "reshape_14 (Reshape)            (None, 1, 120)       0           global_max_pooling1d_14[0][0]    \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_23 (Concatenate)    (None, 1, 360)       0           reshape_12[0][0]                 \n",
      "                                                                 reshape_13[0][0]                 \n",
      "                                                                 reshape_14[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_7 (Dense)                 (None, 1, 20)        7220        concatenate_23[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "dropout_21 (Dropout)            (None, 1, 20)        0           dense_7[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "flatten_10 (Flatten)            (None, 20)           0           dropout_21[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 1)            21          flatten_10[0][0]                 \n",
      "==================================================================================================\n",
      "Total params: 71,980,961\n",
      "Trainable params: 519,761\n",
      "Non-trainable params: 71,461,200\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM, GRU\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.layers import MaxPooling1D, Reshape\n",
    "from keras.layers import Flatten, GlobalMaxPool1D, GlobalAveragePooling1D\n",
    "from keras.layers import Dropout, BatchNormalization, Input, SpatialDropout1D\n",
    "from keras.layers import Dense, Bidirectional, concatenate\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n",
    "from keras_self_attention import SeqSelfAttention\n",
    "\n",
    "\n",
    "embedding_vector_length = 300\n",
    "input1 = Input(shape=(150,))\n",
    "e1 = Embedding(vocab_size,embedding_vector_length, input_length=150, weights=[embedding_matrix], trainable=False)(input1)\n",
    "x1 = Dropout(0.2)(e1)\n",
    "x1 = Bidirectional(LSTM(120, return_sequences=True))(e1)\n",
    "#lstm_att = SeqSelfAttention(attention_activation='sigmoid')(x1)\n",
    "\n",
    "gru1, fh_state, bh_state = Bidirectional(GRU(60, return_sequences= True, return_state=True))(x1)\n",
    "#x2 = SeqSelfAttention(attention_activation='sigmoid')(x2)\n",
    "\n",
    "h_state = concatenate([fh_state, bh_state])\n",
    "h_state = Reshape((-1,120))(h_state )\n",
    "\n",
    "h_avg = GlobalAveragePooling1D()(gru1)\n",
    "h_max = GlobalMaxPool1D()(gru1)\n",
    "\n",
    "h_avg = Reshape((-1,120))(h_avg )\n",
    "h_max = Reshape((-1,120))(h_max )\n",
    "\n",
    "\n",
    "x = concatenate([h_state, h_avg, h_max])\n",
    "x = Dense(20, activation='relu')(x)\n",
    "x = Dropout(0.1)(x)\n",
    "x = Flatten()(x)\n",
    "#print(x)\n",
    "output = Dense(1, activation='sigmoid')(x)\n",
    "model5 = Model(inputs=[input1,], outputs = output)\n",
    "#adam = Adam(lr=0.001)\n",
    "model5.compile(loss= 'binary_crossentropy' , optimizer= 'adam', metrics=['accuracy',])\n",
    "model5.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "id": "nWCvIKgc43hW",
    "outputId": "cef172db-22c2-40f2-dd4a-ef0a51df406e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1282192 samples, validate on 142466 samples\n",
      "Epoch 1/6\n",
      "1282192/1282192 [==============================] - 2724s 2ms/step - loss: 0.1427 - acc: 0.9465 - val_loss: 0.1264 - val_acc: 0.9503\n",
      "Epoch 2/6\n",
      "1282192/1282192 [==============================] - 2730s 2ms/step - loss: 0.1237 - acc: 0.9516 - val_loss: 0.1253 - val_acc: 0.9507\n",
      "Epoch 3/6\n",
      "1282192/1282192 [==============================] - 2724s 2ms/step - loss: 0.1171 - acc: 0.9535 - val_loss: 0.1219 - val_acc: 0.9519\n",
      "Epoch 4/6\n",
      "1282192/1282192 [==============================] - 2713s 2ms/step - loss: 0.1104 - acc: 0.9556 - val_loss: 0.1253 - val_acc: 0.9505\n",
      "Epoch 5/6\n",
      "1282192/1282192 [==============================] - 2713s 2ms/step - loss: 0.1026 - acc: 0.9581 - val_loss: 0.1263 - val_acc: 0.9506\n",
      "Epoch 6/6\n",
      "1282192/1282192 [==============================] - 2724s 2ms/step - loss: 0.0933 - acc: 0.9612 - val_loss: 0.1340 - val_acc: 0.9492\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f0cc1757d68>"
      ]
     },
     "execution_count": 51,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.fit(train_sequence, Y_train, epochs=6, batch_size=512, validation_data=[cv_sequence, Y_cv])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9lfqtrN-Ggkm"
   },
   "outputs": [],
   "source": [
    "# Creating loss metric\n",
    "j_eval = JigsawEvaluator(y_binary, y_identity_binary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HeO14m8dGz4B"
   },
   "outputs": [],
   "source": [
    "predicted_y = model5.predict(test_sequence,batch_size=512)\n",
    "final_auc = j_eval.get_final_metric(predicted_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "k7k952YEFukt",
    "outputId": "03b89787-3d2e-4e2c-c535-954d80687861"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc of the model is 0.9118993070137469\n"
     ]
    }
   ],
   "source": [
    "# This is the auc metric that was given by kaggle.\n",
    "print(\"Auc of the model is {}\".format(final_auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oB6Y70mXOUwH"
   },
   "outputs": [],
   "source": [
    "predicted_y1 = (predicted_y >= 0.5).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "8B6xpRPmOIjq",
    "outputId": "31fe59ae-6392-4247-f019-e4abc49295b4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9550143824173905\n"
     ]
    }
   ],
   "source": [
    "# This is the simple auc on test data\n",
    "print(roc_auc_score(Y_test, predicted_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gRxZjb0lKNb7"
   },
   "outputs": [],
   "source": [
    "predict_y = model5.predict(test_sequence1,batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mhQ4i1ur7QnY"
   },
   "outputs": [],
   "source": [
    "# saving model to google drive\n",
    "from keras.models import load_model\n",
    "model5.save(\"/content/drive/My Drive/Project/my_model5.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0zSl7Cnz43U5"
   },
   "outputs": [],
   "source": [
    "#y_pre = model5.predict(test_sequence1,batch_size=32)\n",
    "submission = pd.DataFrame({'id': test_df['id'].values})\n",
    "submission['prediction'] = predict_y\n",
    "submission.to_csv(\"submission18.csv\",index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z4K80WM42s9q"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download( \"/content/submission18.csv\" )  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jyIQ_6Ri1u9L"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9FekWD7s1wZs"
   },
   "source": [
    "# Fifth Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 751
    },
    "colab_type": "code",
    "id": "MAvMQ9451vGm",
    "outputId": "e5aac6e2-b1ff-45fb-f89c-8596214aa61d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_2 (InputLayer)            (None, 150)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 150, 300)     71461200    input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d_2 (SpatialDro (None, 150, 300)     0           embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_3 (Bidirectional) (None, 150, 256)     440320      spatial_dropout1d_2[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_4 (Bidirectional) (None, 150, 256)     395264      bidirectional_3[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 256)          0           bidirectional_4[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_2 (Glo (None, 256)          0           bidirectional_4[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 512)          0           global_max_pooling1d_3[0][0]     \n",
      "                                                                 global_average_pooling1d_2[0][0] \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 512)          262656      concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "add_1 (Add)                     (None, 512)          0           concatenate_2[0][0]              \n",
      "                                                                 dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 512)          262656      add_1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "add_2 (Add)                     (None, 512)          0           add_1[0][0]                      \n",
      "                                                                 dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 1)            513         add_2[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 72,822,609\n",
      "Trainable params: 1,361,409\n",
      "Non-trainable params: 71,461,200\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Embedding, SpatialDropout1D, Dropout, add, concatenate\n",
    "from keras.layers import CuDNNLSTM, Bidirectional, GlobalMaxPooling1D, GlobalAveragePooling1D\n",
    "from keras.preprocessing import text, sequence\n",
    "from keras.callbacks import LearningRateScheduler\n",
    "from keras.losses import binary_crossentropy\n",
    "from keras import backend as K\n",
    "\n",
    "DENSE_HIDDEN_UNITS = 4*128\n",
    "\n",
    "embedding_vector_length = 300\n",
    "input1 = Input(shape=(150,))\n",
    "e1 = Embedding(vocab_size,embedding_vector_length, input_length=150, weights=[embedding_matrix], trainable=False)(input1)\n",
    "x = SpatialDropout1D(0.3)(e1)\n",
    "x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)\n",
    "x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)\n",
    "\n",
    "max_pool = GlobalMaxPooling1D()(x)\n",
    "avg_pool = GlobalAveragePooling1D()(x)\n",
    "x = concatenate([max_pool, avg_pool])\n",
    "\n",
    "x = add([x, Dense(DENSE_HIDDEN_UNITS, activation='relu')(x)])\n",
    "x = add([x, Dense(DENSE_HIDDEN_UNITS, activation='relu')(x)])\n",
    "output = Dense(1, activation='sigmoid')(x)\n",
    "\n",
    "model6 = Model(inputs=[input1,], outputs=[output])\n",
    "model6.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model6.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "OsZCXpXr1vPO",
    "outputId": "f8b8a5e8-06c8-4111-f8b9-df7223e90e2e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1282192 samples, validate on 142466 samples\n",
      "Epoch 1/4\n",
      "1282192/1282192 [==============================] - 984s 767us/step - loss: 0.1478 - acc: 0.9443 - val_loss: 0.1283 - val_acc: 0.9499\n",
      "Epoch 2/4\n",
      "1282192/1282192 [==============================] - 979s 764us/step - loss: 0.1293 - acc: 0.9495 - val_loss: 0.1236 - val_acc: 0.9514\n",
      "Epoch 3/4\n",
      "1282192/1282192 [==============================] - 979s 764us/step - loss: 0.1240 - acc: 0.9512 - val_loss: 0.1264 - val_acc: 0.9488\n",
      "Epoch 4/4\n",
      "1282192/1282192 [==============================] - 980s 764us/step - loss: 0.1201 - acc: 0.9525 - val_loss: 0.1237 - val_acc: 0.9505\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fafd66ab438>"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model6.fit(train_sequence, Y_train, epochs=4, batch_size=512, validation_data=[cv_sequence, Y_cv])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "colab_type": "code",
    "id": "Qu1SZCyks9IE",
    "outputId": "857a5282-a1be-445b-9906-da7993d3bcb1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:148: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    }
   ],
   "source": [
    "from keras.models import load_model\n",
    "model6 = load_model(\"/content/drive/My Drive/Project/my_model6.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "m1C_dpDw5ltj",
    "outputId": "74f83a0a-5131-409e-8c1d-21cb8e16ce30"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc of the model is 0.9192013466030311\n"
     ]
    }
   ],
   "source": [
    "# Creating loss metric\n",
    "j_eval = JigsawEvaluator(y_binary, y_identity_binary)\n",
    "\n",
    "predicted_y = model6.predict(test_sequence,batch_size=512)\n",
    "final_auc = j_eval.get_final_metric(predicted_y)\n",
    "# This is the auc metric that was given by kaggle.\n",
    "print(\"Auc of the model is {}\".format(final_auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IFvn9OZ15l14"
   },
   "outputs": [],
   "source": [
    "predict_y = model6.predict(test_sequence1,batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i2-mXRhI1vXk"
   },
   "outputs": [],
   "source": [
    "# saving model to google drive\n",
    "from keras.models import load_model\n",
    "model6.save(\"/content/drive/My Drive/Project/my_model6.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OLlSl_Uysece"
   },
   "outputs": [],
   "source": [
    "model6 = load_model(\"/content/drive/My Drive/Project/my_model6.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CFt0RTQF1vf4"
   },
   "outputs": [],
   "source": [
    "#y_pre = model5.predict(test_sequence1,batch_size=32)\n",
    "submission = pd.DataFrame({'id': test_df['id'].values})\n",
    "submission['prediction'] = predict_y\n",
    "submission.to_csv(\"submission19.csv\",index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "26JxU3tB1voE"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download( \"/content/submission19.csv\" )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6jpSUYQvYwbK"
   },
   "source": [
    "### Attention layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1GAQBSCev7b5"
   },
   "outputs": [],
   "source": [
    "# Code taken from kaggle\n",
    "from keras import backend as K\n",
    "from keras.engine.topology import Layer\n",
    "from keras import initializers, regularizers, constraints, optimizers, layers\n",
    "\n",
    "class Attention(Layer):\n",
    "    def __init__(self, step_dim,\n",
    "                 W_regularizer=None, b_regularizer=None,\n",
    "                 W_constraint=None, b_constraint=None,\n",
    "                 bias=True, **kwargs):\n",
    "        self.supports_masking = True\n",
    "        self.init = initializers.get('glorot_uniform')\n",
    "\n",
    "        self.W_regularizer = regularizers.get(W_regularizer)\n",
    "        self.b_regularizer = regularizers.get(b_regularizer)\n",
    "\n",
    "        self.W_constraint = constraints.get(W_constraint)\n",
    "        self.b_constraint = constraints.get(b_constraint)\n",
    "\n",
    "        self.bias = bias\n",
    "        self.step_dim = step_dim\n",
    "        self.features_dim = 0\n",
    "        super(Attention, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape) == 3\n",
    "\n",
    "        self.W = self.add_weight((input_shape[-1],),\n",
    "                                 initializer=self.init,\n",
    "                                 name='{}_W'.format(self.name),\n",
    "                                 regularizer=self.W_regularizer,\n",
    "                                 constraint=self.W_constraint)\n",
    "        self.features_dim = input_shape[-1]\n",
    "\n",
    "        if self.bias:\n",
    "            self.b = self.add_weight((input_shape[1],),\n",
    "                                     initializer='zero',\n",
    "                                     name='{}_b'.format(self.name),\n",
    "                                     regularizer=self.b_regularizer,\n",
    "                                     constraint=self.b_constraint)\n",
    "        else:\n",
    "            self.b = None\n",
    "\n",
    "        self.built = True\n",
    "\n",
    "    def compute_mask(self, input, input_mask=None):\n",
    "        return \n",
    "      \n",
    "    def call(self, x, mask=None):\n",
    "        features_dim = self.features_dim\n",
    "        step_dim = self.step_dim\n",
    "\n",
    "        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n",
    "                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n",
    "\n",
    "        if self.bias:\n",
    "            eij += self.b\n",
    "\n",
    "        eij = K.tanh(eij)\n",
    "\n",
    "        a = K.exp(eij)\n",
    "\n",
    "        if mask is not None:\n",
    "            a *= K.cast(mask, K.floatx())\n",
    "\n",
    "        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n",
    "\n",
    "        a = K.expand_dims(a)\n",
    "        weighted_input = x * a\n",
    "        return K.sum(weighted_input, axis=1)\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return input_shape[0],  self.features_dim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SfZckLoQgoV6"
   },
   "source": [
    "## We are vectorizing train data and will train the model on whole data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XLMeH0L8gXMB"
   },
   "outputs": [],
   "source": [
    "# Tokenizing the essay text data for train dataset\n",
    "# We will train the vectorizer on train data and will use the same on test and cv data.\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "# Tokenizing the essay text data for train dataset\n",
    "# We will train the vectorizer on train data and will use the same on test and cv data.\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.layers import Dense,Input\n",
    "from numpy import asarray\n",
    "t  = Tokenizer(num_words=80000)\n",
    "t.fit_on_texts(df['comment_text'])\n",
    "vocab_size = len(t.word_index)+1\n",
    "# Integer coding all the words.\n",
    "encoded_essay = t.texts_to_sequences(df['comment_text'])\n",
    "# defining a max size for padding.\n",
    "max_len = 220\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "train_sequence = pad_sequences(encoded_essay,maxlen = max_len,padding='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "RrmNUbWnihI8",
    "outputId": "35a0e74a-6e58-436d-f151-a15d793e0f92"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:17: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n"
     ]
    }
   ],
   "source": [
    "# we will load the whole glove vectors .\n",
    "embeddings_index1 = {}\n",
    "# Opening the file\n",
    "f = open(\"/content/drive/My Drive/Project/glove.6B.300d.txt\",'r',encoding=\"utf-8\")\n",
    "for line in f:\n",
    "\tvalues = line.split()\n",
    "\tword = values[0]\n",
    "\tcoefs = asarray(values[1:], dtype='float32')\n",
    "\tembeddings_index1[word] = coefs\n",
    "f.close()\n",
    "\n",
    "\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "import numpy as np\n",
    "embedding_matrix1 = np.zeros((vocab_size, 300))\n",
    "for word, i in t.word_index.items():\n",
    "  for wrd in [word, word.lower()]:\n",
    "    if wrd in embedding_matrix1:\n",
    "\t    embedding_matrix1[i] = embeddings_index1.get(wrd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "E2oipq77ifVD",
    "outputId": "49004075-db3d-4a23-cfe2-54dca22256eb"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:16: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "  app.launch_new_instance()\n"
     ]
    }
   ],
   "source": [
    "#  Loading crawl vectors\n",
    "from gensim.models import KeyedVectors\n",
    "path = \"/content/drive/My Drive/Project/crawl-300d-2M.gensim\"\n",
    "embeddings_index2 = KeyedVectors.load(path, mmap='r')\n",
    "embedding_matrix2 = np.zeros((len(t.word_index) + 1, 300))\n",
    "for word, i in t.word_index.items():\n",
    "    for candidate in [word, word.lower()]:\n",
    "        if candidate in embeddings_index2:\n",
    "            embedding_matrix2[i] = embeddings_index2[candidate]\n",
    "            break\n",
    "            \n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "import numpy as np\n",
    "embedding_matrix2 = np.zeros((vocab_size, 300))\n",
    "for word, i in t.word_index.items():\n",
    "  for wrd in [word, word.lower()]:\n",
    "    if wrd in embedding_matrix2:\n",
    "\t    embedding_matrix2[i] = embeddings_index2.get(wrd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QUHJ1jA8m5TK"
   },
   "outputs": [],
   "source": [
    "# Concatenating embedding vectors of glove and crawl together\n",
    "embedding_matrix = np.concatenate([embedding_matrix1, embedding_matrix2], axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "R0EqQLv8nLPv",
    "outputId": "48ff0da2-6508-4bc2-8450-dcff3a57fc58"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(280377, 600)"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "jvzULe_6khtF",
    "outputId": "f217cee3-2649-4acb-aedb-51797a1067fe"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 97320/97320 [00:06<00:00, 15763.02it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7097320</td>\n",
       "      <td>integrity means that you pay your debts does ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7097321</td>\n",
       "      <td>this malfeasance the administrator and the bo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7097322</td>\n",
       "      <td>rmiller spoken like true elitist but look out...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7097323</td>\n",
       "      <td>paul thank you for your kind words indeed hav...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7097324</td>\n",
       "      <td>sorry you missed high school eisenhower sent ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                       comment_text\n",
       "0  7097320   integrity means that you pay your debts does ...\n",
       "1  7097321   this malfeasance the administrator and the bo...\n",
       "2  7097322   rmiller spoken like true elitist but look out...\n",
       "3  7097323   paul thank you for your kind words indeed hav...\n",
       "4  7097324   sorry you missed high school eisenhower sent ..."
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#from google.colab import drive\n",
    "#drive.mount('/content/drive')\n",
    "test_df = pd.read_csv(\"/content/drive/My Drive/Project/test.csv\")\n",
    "\n",
    "# Preprocessing the comments.\n",
    "test_preprocessed = []\n",
    "bad_found = []\n",
    "from tqdm import tqdm\n",
    "for sent in tqdm(test_df['comment_text'].values):\n",
    "    sent = process_sent(sent)\n",
    "    line = ''\n",
    "    for wrd in sent.split():\n",
    "        if(len(wrd)>2):\n",
    "            line += \" \" +wrd.lower()\n",
    "    line = re.sub(r\"[']\",'',line)\n",
    "    test_preprocessed.append(line)\n",
    "    \n",
    "test_df['comment_text'] = test_preprocessed\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3S7P0SAggXS9"
   },
   "outputs": [],
   "source": [
    "# Vectorizing test data\n",
    "# Integer coding all the words.\n",
    "encoded_essay1 = t.texts_to_sequences(test_df['comment_text'])\n",
    "# defining a max size for padding.\n",
    "max_len = 220\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "test_sequence1 = pad_sequences(encoded_essay1,maxlen = max_len,padding='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "du9M0cXyapBD"
   },
   "outputs": [],
   "source": [
    "def binarize_label(x):\n",
    "  if(x>=0.5):\n",
    "    return 1\n",
    "  else:\n",
    "    return 0\n",
    "Y = df['target'].map(binarize_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YEHSLzm4ZnJE"
   },
   "source": [
    "# Sixth Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YSZlQEcxDjPO"
   },
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Embedding, SpatialDropout1D, Dropout, add, concatenate, Flatten, Reshape\n",
    "from keras.layers import CuDNNLSTM, Bidirectional, GlobalMaxPooling1D, GlobalAveragePooling1D, GRU, GlobalMaxPool1D\n",
    "from keras.preprocessing import text, sequence\n",
    "from keras.callbacks import LearningRateScheduler\n",
    "from keras.losses import binary_crossentropy\n",
    "from keras import backend as K\n",
    "#from keras_self_attention import SeqSelfAttention\n",
    "#import tensorflow\n",
    "#from tensorflow.keras.layers import Attention\n",
    "\n",
    "max_len = 220\n",
    "embedding_vector_length = 600\n",
    "\n",
    "def build_model():\n",
    "  input1 = Input(shape=(max_len,))\n",
    "  e1 = Embedding(vocab_size,embedding_vector_length, input_length=max_len, weights=[embedding_matrix], trainable=False)(input1)\n",
    "  x = SpatialDropout1D(0.2)(e1)\n",
    "  x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)\n",
    "  #x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)\n",
    "  att_lstm = Attention(max_len)(x)\n",
    "\n",
    "  gru1, fh_state, bh_state = Bidirectional(GRU(64, return_sequences= True, return_state=True))(x)\n",
    "  #att_gru = Attention(max_len)(gru1)\n",
    "  h_state = concatenate([fh_state, bh_state])\n",
    "\n",
    "  h_avg = GlobalAveragePooling1D()(gru1)\n",
    "  h_max = GlobalMaxPool1D()(gru1)\n",
    "\n",
    "  x = concatenate([att_lstm, h_state, h_avg, h_max])\n",
    "  x = Dropout(0.4)(x)\n",
    "  # Skip connection . It helps in better flow of gradients.\n",
    "  x=  add([x,Dense(640, activation='relu')(x)])\n",
    "  x = add([x,Dense(640, activation='relu')(x)])\n",
    "  #x = Dropout(0.1)(x)\n",
    "  output = Dense(1, activation='sigmoid')(x)\n",
    "\n",
    "  model8 = Model(inputs=[input1,], outputs=[output])\n",
    "  model8.compile(loss=['binary_crossentropy'] , optimizer='adam', metrics=['accuracy'])\n",
    "  model8.summary()\n",
    "  return model8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Atkf8QlSJH6O"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fIbZO4LfJH29"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ifQWjQQdJHzW"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "oznXPJJeF9jo",
    "outputId": "c27bdb57-3c26-427a-9011-1b2dd51b3b8f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 220)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 220, 600)     168226200   input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d_1 (SpatialDro (None, 220, 600)     0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional) (None, 220, 256)     747520      spatial_dropout1d_1[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_2 (Bidirectional) [(None, 220, 128), ( 123264      bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "attention_1 (Attention)         (None, 256)          476         bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 128)          0           bidirectional_2[0][1]            \n",
      "                                                                 bidirectional_2[0][2]            \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_1 (Glo (None, 128)          0           bidirectional_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 128)          0           bidirectional_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 640)          0           attention_1[0][0]                \n",
      "                                                                 concatenate_1[0][0]              \n",
      "                                                                 global_average_pooling1d_1[0][0] \n",
      "                                                                 global_max_pooling1d_1[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 640)          0           concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 640)          410240      dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_1 (Add)                     (None, 640)          0           dropout_1[0][0]                  \n",
      "                                                                 dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 640)          410240      add_1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "add_2 (Add)                     (None, 640)          0           add_1[0][0]                      \n",
      "                                                                 dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 1)            641         add_2[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 169,918,581\n",
      "Trainable params: 1,692,381\n",
      "Non-trainable params: 168,226,200\n",
      "__________________________________________________________________________________________________\n",
      "Train on 1424658 samples, validate on 356165 samples\n",
      "Epoch 1/10\n"
     ]
    },
    {
     "ename": "InternalError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mInternalError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-30-4fd27a1f56bd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mmodel8\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuild_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmodel8\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_sequence\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2048\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidation_split\u001b[0m\u001b[0;34m=\u001b[0m \u001b[0;36m0.2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[1;32m   1176\u001b[0m                                         \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1177\u001b[0m                                         \u001b[0mvalidation_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidation_steps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1178\u001b[0;31m                                         validation_freq=validation_freq)\n\u001b[0m\u001b[1;32m   1179\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1180\u001b[0m     def evaluate(self,\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/keras/engine/training_arrays.py\u001b[0m in \u001b[0;36mfit_loop\u001b[0;34m(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps, validation_freq)\u001b[0m\n\u001b[1;32m    202\u001b[0m                     \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 204\u001b[0;31m                 \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfit_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    205\u001b[0m                 \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mto_list\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout_labels\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mouts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2977\u001b[0m                     \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_legacy_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2978\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2979\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2980\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2981\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mpy_any\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mis_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2935\u001b[0m             \u001b[0mfetched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_metadata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2936\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2937\u001b[0;31m             \u001b[0mfetched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2938\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfetched\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2939\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1456\u001b[0m         ret = tf_session.TF_SessionRunCallable(self._session._session,\n\u001b[1;32m   1457\u001b[0m                                                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_handle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1458\u001b[0;31m                                                run_metadata_ptr)\n\u001b[0m\u001b[1;32m   1459\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1460\u001b[0m           \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mInternalError\u001b[0m: 2 root error(s) found.\n  (0) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size]: [1, 600, 128, 1, 220, 2048] \n\t [[{{node bidirectional_1/CudnnRNN_1}}]]\n\t [[metrics/acc/Mean_1/_247]]\n  (1) Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size]: [1, 600, 128, 1, 220, 2048] \n\t [[{{node bidirectional_1/CudnnRNN_1}}]]\n0 successful operations.\n0 derived errors ignored."
     ]
    }
   ],
   "source": [
    "model8 = build_model()\n",
    "model8.fit(train_sequence, Y, epochs=10, batch_size=2048, validation_split= 0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gSLmywU2F8ov"
   },
   "outputs": [],
   "source": [
    "# saving model to google drive\n",
    "from keras.models import load_model\n",
    "model8.save(\"/content/drive/My Drive/Project/my_model1New8.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "fh9dFlVYF8lg",
    "outputId": "099d2773-c691-4d28-f076-1518133333cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Auc of the model is 0.919981223216495\n"
     ]
    }
   ],
   "source": [
    "# Creating loss metric\n",
    "j_eval = JigsawEvaluator(y_binary, y_identity_binary)\n",
    "\n",
    "predicted_y = model8.predict(test_sequence,batch_size=512)\n",
    "final_auc = j_eval.get_final_metric(predicted_y)\n",
    "# This is the auc metric that was given by kaggle.\n",
    "print(\"Auc of the model is {}\".format(final_auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U01J-0qjF8iW"
   },
   "outputs": [],
   "source": [
    "predict_y = model8.predict(test_sequence1,batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5NEeGxHPF8ei"
   },
   "outputs": [],
   "source": [
    "#y_pre = model5.predict(test_sequence1,batch_size=32)\n",
    "submission = pd.DataFrame({'id': test_df['id'].values})\n",
    "submission['prediction'] = predict_y\n",
    "submission.to_csv(\"/content/drive/My Drive/Project/Newsubmission23.csv\",index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MPPQPxVQhUlY"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download( \"/content/submission23.csv\" )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oUeYkpDvJXM9"
   },
   "source": [
    "# Seventh Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "U8w1_pCxNJOg",
    "outputId": "27e161d5-11e7-4b8a-c594-31d7518f83e0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# Code taken from kaggle\n",
    "from keras import backend as K\n",
    "from keras.engine.topology import Layer\n",
    "from keras import initializers, regularizers, constraints, optimizers, layers\n",
    "\n",
    "class Attention(Layer):\n",
    "    def __init__(self, step_dim,\n",
    "                 W_regularizer=None, b_regularizer=None,\n",
    "                 W_constraint=None, b_constraint=None,\n",
    "                 bias=True, **kwargs):\n",
    "        self.supports_masking = True\n",
    "        self.init = initializers.get('glorot_uniform')\n",
    "\n",
    "        self.W_regularizer = regularizers.get(W_regularizer)\n",
    "        self.b_regularizer = regularizers.get(b_regularizer)\n",
    "\n",
    "        self.W_constraint = constraints.get(W_constraint)\n",
    "        self.b_constraint = constraints.get(b_constraint)\n",
    "\n",
    "        self.bias = bias\n",
    "        self.step_dim = step_dim\n",
    "        self.features_dim = 0\n",
    "        super(Attention, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        assert len(input_shape) == 3\n",
    "\n",
    "        self.W = self.add_weight((input_shape[-1],),\n",
    "                                 initializer=self.init,\n",
    "                                 name='{}_W'.format(self.name),\n",
    "                                 regularizer=self.W_regularizer,\n",
    "                                 constraint=self.W_constraint)\n",
    "        self.features_dim = input_shape[-1]\n",
    "\n",
    "        if self.bias:\n",
    "            self.b = self.add_weight((input_shape[1],),\n",
    "                                     initializer='zero',\n",
    "                                     name='{}_b'.format(self.name),\n",
    "                                     regularizer=self.b_regularizer,\n",
    "                                     constraint=self.b_constraint)\n",
    "        else:\n",
    "            self.b = None\n",
    "\n",
    "        self.built = True\n",
    "\n",
    "    def compute_mask(self, input, input_mask=None):\n",
    "        return \n",
    "      \n",
    "    def call(self, x, mask=None):\n",
    "        features_dim = self.features_dim\n",
    "        step_dim = self.step_dim\n",
    "\n",
    "        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n",
    "                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n",
    "\n",
    "        if self.bias:\n",
    "            eij += self.b\n",
    "\n",
    "        eij = K.tanh(eij)\n",
    "\n",
    "        a = K.exp(eij)\n",
    "\n",
    "        if mask is not None:\n",
    "            a *= K.cast(mask, K.floatx())\n",
    "\n",
    "        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n",
    "\n",
    "        a = K.expand_dims(a)\n",
    "        weighted_input = x * a\n",
    "        return K.sum(weighted_input, axis=1)\n",
    "\n",
    "    def compute_output_shape(self, input_shape):\n",
    "        return input_shape[0],  self.features_dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "116Z6ZPSJbxP"
   },
   "outputs": [],
   "source": [
    "def preprocess(data):\n",
    "    '''\n",
    "    Credit goes to https://www.kaggle.com/gpreda/jigsaw-fast-compact-solution\n",
    "    '''\n",
    "    punct = \"/-'?!.,#$%\\'()*+-/:;<=>@[\\\\]^_`{|}~`\" + '\"\"“”’' + '∞θ÷α•à−β∅³π‘₹´°£€\\×™√²—–&'\n",
    "    def clean_special_chars(text, punct):\n",
    "        for p in punct:\n",
    "            text = text.replace(p, ' ')\n",
    "        return text\n",
    "\n",
    "    data = data.astype(str).apply(lambda x: clean_special_chars(x, punct))\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J2PqkfY5JbuA"
   },
   "outputs": [],
   "source": [
    "identity_columns = [\n",
    "        'male', 'female', 'homosexual_gay_or_lesbian', 'christian', 'jewish',\n",
    "        'muslim', 'black', 'white', 'psychiatric_or_mental_illness']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UYNeS3XxKV2M"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(\"/content/drive/My Drive/Project/train.csv\")\n",
    "test_df = pd.read_csv(\"/content/drive/My Drive/Project/test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ytd4ksoLJbqt"
   },
   "outputs": [],
   "source": [
    "x_train = preprocess(df['comment_text'])\n",
    "x_test = preprocess(test_df['comment_text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nuBB3dzwJbnM"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# calculating weights for each data points. This will be used during loss calculation.\n",
    "# Each of the 4 sections can contribute 25% to the overall weight.\n",
    "\n",
    "# Overall\n",
    "weights = np.ones((len(x_train),)) / 4\n",
    "\n",
    "# Subgroup\n",
    "weights += (df[identity_columns].fillna(0).values>=0.5).sum(axis=1).astype(bool).astype(np.int) / 4\n",
    "\n",
    "# Background Positive, Subgroup Negative\n",
    "weights += (( (df['target'].values>=0.5).astype(bool).astype(np.int) +\n",
    "   (df[identity_columns].fillna(0).values<0.5).sum(axis=1).astype(bool).astype(np.int) ) > 1 ).astype(bool).astype(np.int) / 4\n",
    "\n",
    "# Background Negative, Subgroup Positive\n",
    "weights += (( (df['target'].values<0.5).astype(bool).astype(np.int) +\n",
    "   (df[identity_columns].fillna(0).values>=0.5).sum(axis=1).astype(bool).astype(np.int) ) > 1 ).astype(bool).astype(np.int) / 4\n",
    "loss_weight = 1.0 / weights.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-BLxmSQRJbkM"
   },
   "outputs": [],
   "source": [
    "y_train = np.vstack([(df['target'].values>=0.5).astype(np.int),weights]).T\n",
    "y_aux_train = df[['target', 'severe_toxicity', 'obscene', 'identity_attack', 'insult', 'threat']].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qLvrJ56EJqbr"
   },
   "outputs": [],
   "source": [
    "# Tokenizing the essay text data for train dataset\n",
    "# We will train the vectorizer on train data and will use the same on test and cv data.\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "# Tokenizing the essay text data for train dataset\n",
    "# We will train the vectorizer on train data and will use the same on test and cv data.\n",
    "# Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.layers import Dense,Input\n",
    "from numpy import asarray\n",
    "t  = Tokenizer()\n",
    "t.fit_on_texts(list(x_train)+list(x_test))\n",
    "vocab_size = len(t.word_index)+1\n",
    "# Integer coding all the words.\n",
    "encoded_essay = t.texts_to_sequences(x_train)\n",
    "# defining a max size for padding.\n",
    "max_len = 220\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "train_sequence = pad_sequences(encoded_essay,maxlen = max_len,padding='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JRYHZ8AjJtXy"
   },
   "outputs": [],
   "source": [
    "# Vectorizing test data\n",
    "# Integer coding all the words.\n",
    "encoded_essay1 = t.texts_to_sequences(x_test)\n",
    "# defining a max size for padding.\n",
    "max_len = 220\n",
    "# padding the vectors of each datapoint to fixed length of 600.\n",
    "test_sequence1 = pad_sequences(encoded_essay1,maxlen = max_len,padding='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Kx3OTTiYJtUs"
   },
   "outputs": [],
   "source": [
    "\n",
    "# Opening the file\n",
    "def get_embedding(path):\n",
    "  f = open(path,'r',encoding=\"utf-8\")\n",
    "  embeddings_index1 = {}\n",
    "  for line in f:\n",
    "\t  values = line.split()\n",
    "\t  word = values[0]\n",
    "\t  coefs = asarray(values[1:], dtype='float32')\n",
    "\t  embeddings_index1[word] = coefs\n",
    "  f.close()\n",
    "\n",
    "\n",
    "  # Credit : https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/\n",
    "  import numpy as np\n",
    "  embedding_matrix1 = np.zeros((vocab_size, 100))\n",
    "  for word, i in t.word_index.items():\n",
    "    for wrd in [word, word.lower()]:\n",
    "      if wrd in embedding_matrix1:\n",
    "\t      embedding_matrix1[i] = embeddings_index1.get(wrd)\n",
    "  return embedding_matrix1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8I6S2cX6JtRz"
   },
   "outputs": [],
   "source": [
    "#  Loading crawl vectors\n",
    "from gensim.models import KeyedVectors\n",
    "def return_embedding(path):\n",
    "    embeddings_index = KeyedVectors.load(path, mmap='r')\n",
    "    embedding_matrix = np.zeros((len(t.word_index) + 1, 300))\n",
    "    for word, i in t.word_index.items():\n",
    "        for candidate in [word, word.lower()]:\n",
    "            if candidate in embeddings_index:\n",
    "                embedding_matrix[i] = embeddings_index[candidate]\n",
    "                break\n",
    "    return embedding_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 88
    },
    "colab_type": "code",
    "id": "TvSl-Us1Jqhf",
    "outputId": "73d6d964-7b8e-4fed-e43a-743c9fe4a334"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:17: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "path1 = \"/content/drive/My Drive/Project/crawl-300d-2M.gensim\"\n",
    "path2 = \"/content/drive/My Drive/Project/glove.6B.100d.txt\"\n",
    "embedding_matrix = np.concatenate([return_embedding(path1), get_embedding(path2)], axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6Br8piX9mF2X"
   },
   "outputs": [],
   "source": [
    "def custom_loss(y_true, y_pred):\n",
    "    return binary_crossentropy(K.reshape(y_true[:,0],(-1,1)), y_pred) * y_true[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "reJvCRmSJ5Pb"
   },
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Embedding, SpatialDropout1D, Dropout, add, concatenate, Flatten, Reshape\n",
    "from keras.layers import CuDNNLSTM, Bidirectional, GlobalMaxPooling1D, GlobalAveragePooling1D, GRU, GlobalMaxPool1D\n",
    "from keras.preprocessing import text, sequence\n",
    "from keras.callbacks import LearningRateScheduler\n",
    "from keras.losses import binary_crossentropy\n",
    "from keras import backend as K\n",
    "\n",
    "max_len = 220\n",
    "embedding_vector_length = 400\n",
    "\n",
    "def build_model(embedding_matrix,loss_weight):\n",
    "  input1 = Input(shape=(max_len,))\n",
    "  e1 = Embedding(vocab_size,embedding_vector_length, input_length=max_len, weights=[embedding_matrix], trainable=False)(input1)\n",
    "  x = SpatialDropout1D(0.2)(e1)\n",
    "  x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)\n",
    "  x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(x)\n",
    "  att_lstm = Attention(max_len)(x)\n",
    "\n",
    "\n",
    "  #h_avg = GlobalAveragePooling1D()(x)\n",
    "  h_max = GlobalMaxPooling1D()(x)\n",
    "\n",
    "  x = concatenate([att_lstm, h_max])\n",
    "  # Skip connection . It helps in better flow of gradients.\n",
    "  x=  add([x,Dense(128*4, activation='relu')(x)])\n",
    "  x = add([x,Dense(128*4, activation='relu')(x)])\n",
    "  output = Dense(1, activation='sigmoid')(x)\n",
    "  aux_output = Dense(6, activation='sigmoid')(x)\n",
    "\n",
    "  model = Model(inputs=[input1,], outputs=[output, aux_output])\n",
    "  model.compile(loss=[custom_loss,'binary_crossentropy'] , optimizer='adam', loss_weights=[loss_weight, 1.0] )\n",
    "  #model.summary()\n",
    "  return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 785
    },
    "colab_type": "code",
    "id": "Tx8KM6r_KC7f",
    "outputId": "110f0782-03f5-4c5e-e3bd-9e2078104f42"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "Model: \"model_2\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_3 (InputLayer)            (None, 220)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 220, 400)     130803600   input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d_2 (SpatialDro (None, 220, 400)     0           embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_3 (Bidirectional) (None, 220, 256)     542720      spatial_dropout1d_2[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_4 (Bidirectional) (None, 220, 256)     395264      bidirectional_3[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "attention_2 (Attention)         (None, 256)          476         bidirectional_4[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           bidirectional_4[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 512)          0           attention_2[0][0]                \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 512)          262656      concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "add_3 (Add)                     (None, 512)          0           concatenate_2[0][0]              \n",
      "                                                                 dense_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 512)          262656      add_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "add_4 (Add)                     (None, 512)          0           add_3[0][0]                      \n",
      "                                                                 dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_7 (Dense)                 (None, 1)            513         add_4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 6)            3078        add_4[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 132,270,963\n",
      "Trainable params: 1,467,363\n",
      "Non-trainable params: 130,803,600\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "build_model(embedding_matrix, loss_weight).summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "id": "Mp8ejVxbKC4S",
    "outputId": "2d40b236-9440-417c-f8b5-f415ff40c408"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2227s 1ms/step - loss: 0.3051 - dense_11_loss: 0.0612 - dense_12_loss: 0.1089\n",
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2215s 1ms/step - loss: 0.2758 - dense_11_loss: 0.0537 - dense_12_loss: 0.1036\n",
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2212s 1ms/step - loss: 0.2671 - dense_11_loss: 0.0513 - dense_12_loss: 0.1026\n",
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2216s 1ms/step - loss: 0.2612 - dense_11_loss: 0.0496 - dense_12_loss: 0.1021\n",
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2214s 1ms/step - loss: 0.3048 - dense_15_loss: 0.0611 - dense_16_loss: 0.1088\n",
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2207s 1ms/step - loss: 0.2755 - dense_15_loss: 0.0536 - dense_16_loss: 0.1035\n",
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2215s 1ms/step - loss: 0.2667 - dense_15_loss: 0.0512 - dense_16_loss: 0.1025\n",
      "Epoch 1/1\n",
      "1804874/1804874 [==============================] - 2218s 1ms/step - loss: 0.2606 - dense_15_loss: 0.0494 - dense_16_loss: 0.1020\n"
     ]
    }
   ],
   "source": [
    "from keras.callbacks import LearningRateScheduler\n",
    "checkpoint_predictions = []\n",
    "weights = []\n",
    "\n",
    "for model_idx in range(2):\n",
    "    model = build_model(embedding_matrix, loss_weight)\n",
    "    for global_epoch in range(4):\n",
    "        model.fit(\n",
    "            train_sequence,\n",
    "            [y_train, y_aux_train],\n",
    "            batch_size=512,\n",
    "            epochs=1,\n",
    "            verbose=1,\n",
    "            callbacks=[\n",
    "                LearningRateScheduler(lambda epoch: 1e-3 * (0.6 ** global_epoch))\n",
    "            ]\n",
    "        )\n",
    "        checkpoint_predictions.append(model.predict(test_sequence1, batch_size=2048)[0].flatten())\n",
    "        weights.append(2 ** global_epoch)\n",
    "\n",
    "predictions = np.average(checkpoint_predictions, weights=weights, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8NLcVGi_KC0v"
   },
   "outputs": [],
   "source": [
    "test_df = pd.read_csv(\"/content/drive/My Drive/Project/test.csv\")\n",
    "submission = pd.DataFrame.from_dict({\n",
    "    'id': test_df.id,\n",
    "    'prediction': predictions\n",
    "})\n",
    "submission.to_csv('/content/drive/My Drive/Project/FinalSubmission.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dMCNTgarJqYJ"
   },
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "# Saving the model to gdrive\n",
    "model.save('/content/drive/My Drive/Project/FinalMOdel.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eanGYB2fDNVk"
   },
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "colab_type": "code",
    "id": "DdSq7i99e70t",
    "outputId": "418fb88c-242c-4d27-da9c-a4e0cd0fe626"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------------------------------------------+------------------------------+----------+\n",
      "|                           model                           |          vectorizer          | test auc |\n",
      "+-----------------------------------------------------------+------------------------------+----------+\n",
      "|                    Logistic Regression                    |        tfidf unigram         |  0.9465  |\n",
      "|                        Naive Bayes                        |        tfidf unigram         |  0.8755  |\n",
      "|          1 layered Bi-LSTM + self attention layer         |        glove 100dim          |  0.8823  |\n",
      "|                     1 layered Bi-LSTM                     |         glove 100dim         |  0.9038  |\n",
      "|                2 layered Bi-LSTM(200 units)               |        glove 100dim          |  0.9373  |\n",
      "|                2 layered Bi-LSTM(128 units)               |        glove 100dim          |  0.9370  |\n",
      "| 1 layered Bi-LSTM(120 units) + 1 layered Bi-gru(60) units |        glove 300dim          |  0.9550  |\n",
      "|                        2 lstm layer                       |        glove 300 dim         |  0.9505  |\n",
      "|                 2 lstm layer + 1 gru layer                |        glove 300 dim         |  0.9469  |\n",
      "|               2 lstm layer + attention layer              | glove 100dim + crawl 300 dim | 0.93309  |\n",
      "+-----------------------------------------------------------+------------------------------+----------+\n"
     ]
    }
   ],
   "source": [
    "from prettytable import PrettyTable\n",
    "\n",
    "x = PrettyTable(['model','vectorizer','test auc'])\n",
    "x.add_row(['Logistic Regression','tfidf unigram','0.9465'])\n",
    "x.add_row(['Naive Bayes','tfidf unigram','0.8755'])\n",
    "x.add_row(['1 layered Bi-LSTM + self attention layer','glove 100dim ','0.8823'])\n",
    "x.add_row(['1 layered Bi-LSTM','glove 100dim','0.9038'])\n",
    "x.add_row(['2 layered Bi-LSTM(200 units)','glove 100dim ','0.9373'])\n",
    "x.add_row(['2 layered Bi-LSTM(128 units)','glove 100dim ','0.9370'])\n",
    "x.add_row(['1 layered Bi-LSTM(120 units) + 1 layered Bi-gru(60) units','glove 300dim ','0.9550'])\n",
    "x.add_row(['2 lstm layer','glove 300 dim', '0.9505'])\n",
    "x.add_row(['2 lstm layer + 1 gru layer','glove 300 dim', '0.9469'])\n",
    "x.add_row(['2 lstm layer + attention layer','glove 100dim + crawl 300 dim', '0.93309'])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HX3UKgS38vov"
   },
   "source": [
    "## The best model that we had was the seventh and the last model that gave us an auc of 0.93309 on the kaggle private leader board."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o35GGrmJaAjO"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "4Qa8JRLoKqQ7",
    "eCbzpZf8Yj_W",
    "NI6ijwfIs9qW",
    "udNfnvCt77z9",
    "9FekWD7s1wZs",
    "YEHSLzm4ZnJE"
   ],
   "machine_shape": "hm",
   "name": "Jigsaw4.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}