Ch_05_Sentence_Classification_CNN/02_CNN_Sentence_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "597487a4-d412-4a54-8ad7-7196f12da585",
   "metadata": {},
   "source": [
    "# Using CNN for Sentence Classification\n",
    "\n",
    "- [Paper](https://arxiv.org/pdf/1408.5882.pdf): Convolutional Neural Networks for Sentence Classification by Yoon Kim\n",
    "\n",
    "- [Convolution in NLP - Code Emporium](https://youtu.be/AGWieLbom_g)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7521a20-37f3-46f2-926e-c060d4cd7bc0",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "773237cb-e2c1-43cb-a643-acf4e9575db2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import collections\n",
    "import math\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import random\n",
    "import tensorflow as tf\n",
    "import zipfile\n",
    "from matplotlib import pylab\n",
    "from six.moves import range\n",
    "from six.moves.urllib.request import urlretrieve\n",
    "import tensorflow as tf\n",
    "\n",
    "seed = 54321"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1541c2e-fae0-4f53-8a2e-82dfdf6d4644",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## How data is transformed for sentence classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9893eb07-9188-45c8-a599-97f027263dd9",
   "metadata": {},
   "source": [
    "- Let's assume a sentence of $p$ words. \n",
    "\n",
    "- First, we will pad the sentence with some special words (if the length of the sentence is $< n$) to set the sentence length to $n$ words, where $n \\geq p$. \n",
    "\n",
    "- Next, we will represent each word in the sentence by a vector of size $k$, where this vector can either be a one-hot-encoded representation, or Word2vec word vectors learned using skip-gram, CBOW, or GloVe. \n",
    "\n",
    "- Then a batch of sentences of size b can be represented by a $b \\times n \\times k$ matrix. \n",
    "\n",
    "* **\n",
    "\n",
    " Let's walk through an example. Let's consider the following three sentences:\n",
    "- *Bob and Mary are friends*\n",
    "- *Bob plays soccer*\n",
    "- *Mary likes to sing*\n",
    "\n",
    "In this ex., the third sentence has the most words, so let's set $n=7$, which is the num. of words in the third sentence.\n",
    "Next, we create the One-Hot-Encoded rep. for each word. Here we have $13$ distinct words. Thus we get:\n",
    "```\n",
    "Bob: 1,0,0,0,...\n",
    "and: 0,1,0,0,...\n",
    "Mary:0,0,1,0,...\n",
    "...so on...\n",
    "```\n",
    "Also, $k = 13$ i.e. the vector size of each word, for the same reason. We finally can represent the three sentences as 3-D matrix of size $3 \\times 7 \\times 13$ as shown below:\n",
    "\n",
    "<div align='center'>\n",
    "    <img src='images/sentence_matrix.png'/>\n",
    "</div>\n",
    "\n",
    "* **\n",
    "\n",
    "You could also utilize word embeddings instead of one-hot encoding here. Representing each word as a one-hot-encoded feature introduces sparsity and wastes computational memory. By using embeddings, we are enabling the model to learn more compact and powerful word representations than one-hot-encoded representations. This also means that $k$ becomes a hyperparameter (i.e. the embedding size), as opposed to being driven by the size of the vocabulary. This means that, in above fig., each column will be a distributed continuous vector, not a combination of Os and Is."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf5fbc5-fedb-4614-87cd-bae280c6bb8d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Get the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0d3406a9-ad8c-4ba8-a1c2-b269d6147035",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found and verified data\\train_5500.label\n",
      "Found and verified data\\TREC_10.label\n"
     ]
    }
   ],
   "source": [
    "url = 'http://cogcomp.org/Data/QA/QC/'\n",
    "dir_name = 'data'\n",
    "\n",
    "def download_data(dir_name, filename, expected_bytes):\n",
    "    \"\"\"Download a file if not present, and make sure it's the right size.\"\"\"\n",
    "  \n",
    "    os.makedirs(dir_name, exist_ok=True)\n",
    "    if not os.path.exists(os.path.join(dir_name,filename)):\n",
    "        filepath, _ = urlretrieve(url + filename, os.path.join(dir_name,filename))\n",
    "    else:\n",
    "        filepath = os.path.join(dir_name, filename)\n",
    "    \n",
    "    statinfo = os.stat(filepath)\n",
    "    if statinfo.st_size == expected_bytes:\n",
    "        print('Found and verified %s' % filepath)\n",
    "    else:\n",
    "        print(statinfo.st_size)\n",
    "        raise Exception(\n",
    "          'Failed to verify ' + filepath + '. Can you get to it with a browser?')\n",
    "        \n",
    "    return filepath\n",
    "\n",
    "train_filename = download_data(dir_name, 'train_5500.label', 335858)\n",
    "test_filename = download_data(dir_name, 'TREC_10.label', 23354)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ecba69d-8dc0-4138-b9cd-4487b656e88d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Read & Preprocess Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f4ba1e80-16c4-4522-aadc-cecc626e97c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def read_data(filename):\n",
    "    '''\n",
    "    Read data from a file with given filename\n",
    "    Returns a list of strings where each string is a lower case word\n",
    "    '''\n",
    "\n",
    "    # Holds question strings, categories and sub categories\n",
    "    # category/sub_cateory definitions: https://cogcomp.seas.upenn.edu/Data/QA/QC/definition.html\n",
    "    questions, categories, sub_categories = [], [], []     \n",
    "    \n",
    "    with open(filename,'r',encoding='latin-1') as f:        \n",
    "        # Read each line\n",
    "        for row in f:   \n",
    "            # Each string has format <cat>:<sub cat> <question>\n",
    "            # Split by : to separate cat and (sub_cat + question)\n",
    "            row_str = row.split(\":\")        \n",
    "            cat, sub_cat_and_question = row_str[0], row_str[1]\n",
    "            tokens = sub_cat_and_question.split(' ')\n",
    "            # The first word in sub_cat_and_question is the sub category\n",
    "            # rest is the question\n",
    "            sub_cat, question = tokens[0], ' '.join(tokens[1:])        \n",
    "            \n",
    "            questions.append(question.lower().strip())\n",
    "            categories.append(cat)\n",
    "            sub_categories.append(sub_cat)\n",
    "            \n",
    "\n",
    "    return questions, categories, sub_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e0f80ab9-ac7f-4b6c-849f-dbfcfac00bf7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_questions, train_categories, train_sub_categories = read_data(train_filename)\n",
    "test_questions, test_categories, test_sub_categories = read_data(test_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a6e8b41c-924b-4428-b893-b822b7f78476",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_questions has 5452 questions / 5452 labels\n",
      "Some samples\n",
      "\thow did serfdom develop in and then leave russia ? / cat - DESC / sub_cat - manner\n",
      "\twhat films featured the character popeye doyle ? / cat - ENTY / sub_cat - cremat\n",
      "\thow can i find a list of celebrities ' real names ? / cat - DESC / sub_cat - manner\n",
      "\twhat fowl grabs the spotlight after the chinese year of the monkey ? / cat - ENTY / sub_cat - animal\n",
      "\twhat is the full form of .com ? / cat - ABBR / sub_cat - exp\n",
      "\twhat contemptible scoundrel stole the cork from my lunch ? / cat - HUM / sub_cat - ind\n",
      "\twhat team did baseball 's st. louis browns become ? / cat - HUM / sub_cat - gr\n",
      "\twhat is the oldest profession ? / cat - HUM / sub_cat - title\n",
      "\twhat are liver enzymes ? / cat - DESC / sub_cat - def\n",
      "\tname the scar-faced bounty hunter of the old west . / cat - HUM / sub_cat - ind\n",
      "\n",
      "test_questions has 500 questions / 500 labels\n",
      "Some samples\n",
      "\thow far is it from denver to aspen ? / cat - NUM / sub_cat - dist\n",
      "\twhat county is modesto , california in ? / cat - LOC / sub_cat - city\n",
      "\twho was galileo ? / cat - HUM / sub_cat - desc\n",
      "\twhat is an atom ? / cat - DESC / sub_cat - def\n",
      "\twhen did hawaii become a state ? / cat - NUM / sub_cat - date\n",
      "\thow tall is the sears building ? / cat - NUM / sub_cat - dist\n",
      "\tgeorge bush purchased a small interest in which baseball team ? / cat - HUM / sub_cat - gr\n",
      "\twhat is australia 's national flower ? / cat - ENTY / sub_cat - plant\n",
      "\twhy does the moon turn orange ? / cat - DESC / sub_cat - reason\n",
      "\twhat is autism ? / cat - DESC / sub_cat - def\n"
     ]
    }
   ],
   "source": [
    "n_samples = 10\n",
    "print(f\"train_questions has {len(train_questions)} questions / {len(train_categories)} labels\")\n",
    "print(\"Some samples\")\n",
    "for question, cat, sub_cat in zip(train_questions[:n_samples], train_categories[:n_samples], train_sub_categories[:n_samples]):    \n",
    "    print(f\"\\t{question} / cat - {cat} / sub_cat - {sub_cat}\")\n",
    "          \n",
    "print(f\"\\ntest_questions has {len(test_questions)} questions / {len(test_categories)} labels\")\n",
    "print(\"Some samples\")\n",
    "for question, cat, sub_cat in zip(test_questions[:n_samples], test_categories[:n_samples], test_sub_categories[:n_samples]):    \n",
    "    print(f\"\\t{question} / cat - {cat} / sub_cat - {sub_cat}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56afdf98-2c42-4b87-8b88-bf825b60385a",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Converting train-test text data to `pd.DataFrame` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "96794f7d-9627-45a8-a83c-827be5cf1e2f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>category</th>\n",
       "      <th>sub_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>how did serfdom develop in and then leave russ...</td>\n",
       "      <td>DESC</td>\n",
       "      <td>manner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>what films featured the character popeye doyle ?</td>\n",
       "      <td>ENTY</td>\n",
       "      <td>cremat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>how can i find a list of celebrities ' real na...</td>\n",
       "      <td>DESC</td>\n",
       "      <td>manner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>what fowl grabs the spotlight after the chines...</td>\n",
       "      <td>ENTY</td>\n",
       "      <td>animal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>what is the full form of .com ?</td>\n",
       "      <td>ABBR</td>\n",
       "      <td>exp</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>what contemptible scoundrel stole the cork fro...</td>\n",
       "      <td>HUM</td>\n",
       "      <td>ind</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>what team did baseball 's st. louis browns bec...</td>\n",
       "      <td>HUM</td>\n",
       "      <td>gr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>what is the oldest profession ?</td>\n",
       "      <td>HUM</td>\n",
       "      <td>title</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>what are liver enzymes ?</td>\n",
       "      <td>DESC</td>\n",
       "      <td>def</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>name the scar-faced bounty hunter of the old w...</td>\n",
       "      <td>HUM</td>\n",
       "      <td>ind</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question category sub_category\n",
       "0  how did serfdom develop in and then leave russ...     DESC       manner\n",
       "1   what films featured the character popeye doyle ?     ENTY       cremat\n",
       "2  how can i find a list of celebrities ' real na...     DESC       manner\n",
       "3  what fowl grabs the spotlight after the chines...     ENTY       animal\n",
       "4                    what is the full form of .com ?     ABBR          exp\n",
       "5  what contemptible scoundrel stole the cork fro...      HUM          ind\n",
       "6  what team did baseball 's st. louis browns bec...      HUM           gr\n",
       "7                    what is the oldest profession ?      HUM        title\n",
       "8                           what are liver enzymes ?     DESC          def\n",
       "9  name the scar-faced bounty hunter of the old w...      HUM          ind"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define training and testing\n",
    "train_df = pd.DataFrame(\n",
    "    {'question': train_questions, 'category': train_categories, 'sub_category': train_sub_categories}\n",
    ")\n",
    "test_df = pd.DataFrame(\n",
    "    {'question': test_questions, 'category': test_categories, 'sub_category': test_sub_categories}\n",
    ")\n",
    "\n",
    "train_df.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b9c25353-8b3e-493b-8a39-819adf48c565",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Shuffle the data for better randomization\n",
    "train_df = train_df.sample(frac=1.0, random_state=seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65052c6c-9ae8-4985-a029-447515de64a4",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Convert string labels to integer IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e398a130-bec3-498f-a349-bf84f43089dd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label->ID mapping: {'DESC': 0, 'ENTY': 1, 'LOC': 2, 'NUM': 3, 'HUM': 4, 'ABBR': 5}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>category</th>\n",
       "      <th>sub_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5267</th>\n",
       "      <td>what is an aurora ?</td>\n",
       "      <td>0</td>\n",
       "      <td>def</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>what articles of clothing are tokens in monopo...</td>\n",
       "      <td>1</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3258</th>\n",
       "      <td>what causes rust ?</td>\n",
       "      <td>0</td>\n",
       "      <td>reason</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1356</th>\n",
       "      <td>what does an irate car owner call iron oxide ?</td>\n",
       "      <td>1</td>\n",
       "      <td>termeq</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1529</th>\n",
       "      <td>what do we call the imaginary line along the t...</td>\n",
       "      <td>2</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3631</th>\n",
       "      <td>why is hockey so violent ?</td>\n",
       "      <td>0</td>\n",
       "      <td>reason</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4802</th>\n",
       "      <td>how many characters makes up a word for typing...</td>\n",
       "      <td>3</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2288</th>\n",
       "      <td>what peter blatty novel recounts the horrors o...</td>\n",
       "      <td>1</td>\n",
       "      <td>cremat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>803</th>\n",
       "      <td>what is measured in curies ?</td>\n",
       "      <td>0</td>\n",
       "      <td>def</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4472</th>\n",
       "      <td>what does seccession mean ?</td>\n",
       "      <td>0</td>\n",
       "      <td>def</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               question  category sub_category\n",
       "5267                                what is an aurora ?         0          def\n",
       "21    what articles of clothing are tokens in monopo...         1        other\n",
       "3258                                 what causes rust ?         0       reason\n",
       "1356     what does an irate car owner call iron oxide ?         1       termeq\n",
       "1529  what do we call the imaginary line along the t...         2        other\n",
       "3631                         why is hockey so violent ?         0       reason\n",
       "4802  how many characters makes up a word for typing...         3        count\n",
       "2288  what peter blatty novel recounts the horrors o...         1       cremat\n",
       "803                        what is measured in curies ?         0          def\n",
       "4472                        what does seccession mean ?         0          def"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate the label to ID mapping\n",
    "unique_cats = train_df[\"category\"].unique()\n",
    "\n",
    "labels_map = dict(zip(unique_cats, np.arange(unique_cats.shape[0])))\n",
    "\n",
    "print(f\"Label->ID mapping: {labels_map}\")\n",
    "\n",
    "n_classes = len(labels_map)\n",
    "\n",
    "# Convert all string labels to IDs\n",
    "train_df[\"category\"] = train_df[\"category\"].map(labels_map)\n",
    "test_df[\"category\"] = test_df[\"category\"].map(labels_map)\n",
    "\n",
    "# View some data\n",
    "train_df.head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4937c9ee-eb5c-461b-a4af-9221f387e330",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Split training data to train and valid subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "81548e79-af92-4d68-993f-51e587061e86",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: (4906, 3)\n",
      "Valid size: (546, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>category</th>\n",
       "      <th>sub_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3400</th>\n",
       "      <td>what was franklin roosevelt 's program for eco...</td>\n",
       "      <td>1</td>\n",
       "      <td>event</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2630</th>\n",
       "      <td>how many megawatts will the power project in i...</td>\n",
       "      <td>3</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3449</th>\n",
       "      <td>what is a fear of money ?</td>\n",
       "      <td>1</td>\n",
       "      <td>dismed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1640</th>\n",
       "      <td>what dog was dubbed the mortgage lifter ?</td>\n",
       "      <td>1</td>\n",
       "      <td>animal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5194</th>\n",
       "      <td>the kentucky horse park is close to which amer...</td>\n",
       "      <td>2</td>\n",
       "      <td>city</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               question  category sub_category\n",
       "3400  what was franklin roosevelt 's program for eco...         1        event\n",
       "2630  how many megawatts will the power project in i...         3        count\n",
       "3449                          what is a fear of money ?         1       dismed\n",
       "1640          what dog was dubbed the mortgage lifter ?         1       animal\n",
       "5194  the kentucky horse park is close to which amer...         2         city"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_df, valid_df = train_test_split(train_df, test_size=0.1)\n",
    "print(f\"Train size: {train_df.shape}\")\n",
    "print(f\"Valid size: {valid_df.shape}\")\n",
    "\n",
    "# Print data\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9929722b-07e9-4732-9cc2-3841b7a1f321",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Tokenizer & Padding Sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d1320ce-f182-432b-8930-e54c7e6c4012",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cc5c05ce-497f-4386-b6eb-73d281f53ea5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabluary size: 7849\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(train_df.question.tolist())\n",
    "\n",
    "# Vocab size\n",
    "n_vocab = len(tokenizer.index_word) + 1\n",
    "print(f\"Vocabluary size: {n_vocab}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93433391-68fc-490c-8ab3-586f18b693b9",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Find the sequence length\n",
    "\n",
    "Here we analyze the `1%` and `99%` percentiles of the sequence lengths. We will use the `99%` percentile as our maximum sequence length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0e9eedb1-6a15-4283-84bd-dd84491ad482",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    4906.000000\n",
       "mean       10.060742\n",
       "std         3.771990\n",
       "min         2.000000\n",
       "1%          4.000000\n",
       "50%        10.000000\n",
       "99%        22.000000\n",
       "max        37.000000\n",
       "Name: question, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split each string by \" \", compute length of the list, get the percentiles\n",
    "train_df[\"question\"].str.split(\" \").str.len().describe(percentiles=[0.01, 0.5, 0.99])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d011de0-e457-4696-be80-6e9f2014c51e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Padding Shorter Sentences\n",
    "\n",
    "We use padding to pad short sentences so that all the sentences are of the same length."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2a3b04e-91d6-467b-be1c-c346c0b53f51",
   "metadata": {},
   "source": [
    "It's important to understand that we are feeding our model a batch of questions at a given time. It is very unlikely that all of the questions have the same number of tokens. If all questions do not have the same number of tokens, we cannot form a tensor due to the uneven lengths of different questions. To solve this, we have to pad shorter sequences with special tokens and truncate sequences longer than a specified length. To achieve this we can easily use the `tf.keras.preprocessing.sequence.pad_sequences()` function. The arguments accepted by this function:\n",
    "- `sequences` - list of list integers; each list of integers is a sequence\n",
    "- `maxlen` - maximum padding length\n",
    "- `padding` - wheather to pad at the beginning(`pre`) or end (`post`)\n",
    "- `truncating` - wheather to truncate at the beginning(`pre`) or end (`post`)\n",
    "- `value` - what value to be used for padding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b924b5fb-c1ce-401c-b6b4-67bd88f1e50f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "# Convert each list of tokens to a list of IDs, using tokenizer's mapping\n",
    "train_sequences = tokenizer.texts_to_sequences(train_df[\"question\"].tolist())\n",
    "train_labels = train_df[\"category\"].values\n",
    "\n",
    "valid_sequences = tokenizer.texts_to_sequences(valid_df[\"question\"].tolist())\n",
    "valid_labels = valid_df[\"category\"].values\n",
    "\n",
    "test_sequences = tokenizer.texts_to_sequences(test_df[\"question\"].tolist())\n",
    "test_labels = test_df[\"category\"].values\n",
    "\n",
    "\n",
    "# 99% perecentile of the text sequence lengths of training corpus is 22.\n",
    "# that's we picked 22 as the max_seq_length \n",
    "max_seq_length = 22\n",
    "\n",
    "# Pad shorter sentences and truncate longer ones (max length: max_seq_length)\n",
    "preprocessed_train_sequences = pad_sequences(train_sequences, \n",
    "                                             maxlen=max_seq_length, \n",
    "                                             padding='post', \n",
    "                                             truncating='post')\n",
    "\n",
    "preprocessed_valid_sequences = pad_sequences(valid_sequences, \n",
    "                                             maxlen=max_seq_length, \n",
    "                                             padding='post', \n",
    "                                             truncating='post')\n",
    "\n",
    "preprocessed_test_sequences = pad_sequences(test_sequences, \n",
    "                                            maxlen=max_seq_length, \n",
    "                                            padding='post', \n",
    "                                            truncating='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a719fafa-a4e9-47c1-8d69-cc4d3166836b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4906, 22)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_train_sequences.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4b055a5-7d5b-4c06-8a22-9a6810b9b6bc",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Sentence Classifying CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68292412-7482-4043-971f-edafc4620066",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Convolution Operation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e81e81-0b4e-48f3-be91-46c052f9c9b1",
   "metadata": {},
   "source": [
    "To learn a rich set of features, we have parallel layers with different convolution filter sizes. Each convolution layer outputs a hidden vector of size $1 \\times n$, [where $n = $ number of words per sentence after padding] and we will concatenate these outputs to form the input to the next layer of size $q \\times n$, where $q$ is the number of parallel layers we will use. The larger $q$ is, the better the performance of the model.\n",
    "\n",
    "* **\n",
    "\n",
    "The value of convolving can be understood in the following manner. Think about the movie rating learning problem (with two classes, positive or negative), and we have the following sentences:\n",
    "\n",
    "- *I like the movie, not too bad*\n",
    "- *I did not like the movie, bad*\n",
    "\n",
    "Now imagine a convolution window of size 5. Let’s bin the words according to the movement of the convolution window.\n",
    "\n",
    "The sentence *I like the movie, not too bad* gives:\n",
    "- [I, like, the, movie, ‘,’]\n",
    "- [like, the, movie, ‘,’, not]\n",
    "- [the, movie, ‘,’, not, too]\n",
    "- [movie, ‘,’, not, too, bad]\n",
    "\n",
    "The sentence *I did not like the movie, bad* gives:\n",
    "- [I, did, not, like, the]\n",
    "- [did, not ,like, the, movie]\n",
    "- [not, like, the, movie, ‘,’]\n",
    "- [like, the, movie, ‘,’, bad]\n",
    "\n",
    "\n",
    "For the first sentence, windows such as the following convey that the rating is positive:\n",
    "- [I, like, the, movie, ‘,’] ; [movie, ‘,’, not, too, bad]\n",
    "\n",
    "However, for the second sentence, windows such as the following convey negativity in the rating:\n",
    "- [did, not, like, the, movie]\n",
    "\n",
    "* **\n",
    "\n",
    "- We are able to see such patterns that help to classify ratings thanks to the preserved spatiality.\n",
    "\n",
    "  - For example, if you use a technique such as bag-of-words to calculate sentence representations that lose spatial information, the sentence representations of the above two sentences would be highly similar.<br></br> \n",
    "\n",
    "- The convolution operation plays an important role in preserving the spatial information of the sentences. \n",
    "\n",
    "- Having q different layers with different filter sizes, the network learns to extract the rating with different size phrases, leading to an improved performance.\n",
    "\n",
    "\n",
    "<div align='center'>\n",
    "    <img src='images/conv_op.png'/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "718ae597-e9e0-426b-8dbd-251c2c51b52f",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Pooling Over Time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b08b020-f4fd-4fd0-96ac-0ab912bd34ca",
   "metadata": {},
   "source": [
    "The pooling operation is designed to subsample the outputs produced by the previously discussed parallel convolution layers.\n",
    "\n",
    "Let’s assume the output of the last layer $h$ is of size $q \\times n$. The pooling over time layer would produce an output $h’$ of size $q \\times 1$ output.\n",
    "\n",
    "Simply put, the pooling over time operation creates a\n",
    "vector by concatenating the maximum element of each convolution layer. \n",
    "\n",
    "<div align='center'>\n",
    "    <img src='images/pooling_over_time.png'/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c26d802-b1e0-4435-a9db-ad582d651ae0",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Model Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b8fe43-105a-45d7-854d-3bff992cd929",
   "metadata": {},
   "source": [
    "<div align='center'>\n",
    "    <img src='./images/sentence_classification_cnn_architecture.png'/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57a7cfb-ba68-4e6d-bfa1-0f69c62c44d3",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Build Sentence Classifying CNN\n",
    "\n",
    "We are going to implement a very simple CNN to classify sentences. However you will see that even with this simple structure we achieve good accuracies. \n",
    "\n",
    "**Our CNN will have one layer (with 3 different parallel layers). This will be followed by a pooling-over-time layer and finally a fully connected layer that produces the logits.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6bf95bc4-49a6-4759-b87c-6ce2eff0e2a7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow.keras.backend as K\n",
    "import tensorflow.keras.layers as layers\n",
    "import tensorflow.keras.regularizers as regularizers\n",
    "from tensorflow.keras.models import Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "949b0adb-ad66-46c8-b03f-68fd80dfe6be",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_1 (InputLayer)           [(None, 22)]         0           []                               \n",
      "                                                                                                  \n",
      " embedding (Embedding)          (None, 22, 64)       502336      ['input_1[0][0]']                \n",
      "                                                                                                  \n",
      " conv1d (Conv1D)                (None, 22, 100)      19300       ['embedding[0][0]']              \n",
      "                                                                                                  \n",
      " conv1d_1 (Conv1D)              (None, 22, 100)      25700       ['embedding[0][0]']              \n",
      "                                                                                                  \n",
      " conv1d_2 (Conv1D)              (None, 22, 100)      32100       ['embedding[0][0]']              \n",
      "                                                                                                  \n",
      " concatenate (Concatenate)      (None, 22, 300)      0           ['conv1d[0][0]',                 \n",
      "                                                                  'conv1d_1[0][0]',               \n",
      "                                                                  'conv1d_2[0][0]']               \n",
      "                                                                                                  \n",
      " max_pooling1d (MaxPooling1D)   (None, 1, 300)       0           ['concatenate[0][0]']            \n",
      "                                                                                                  \n",
      " flatten (Flatten)              (None, 300)          0           ['max_pooling1d[0][0]']          \n",
      "                                                                                                  \n",
      " dense (Dense)                  (None, 6)            1806        ['flatten[0][0]']                \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 581,242\n",
      "Trainable params: 581,242\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "K.clear_session()\n",
    "\n",
    "# Input layer takes word IDs as inputs\n",
    "word_id_inputs = layers.Input(shape=(max_seq_length,), dtype='int32')\n",
    "\n",
    "# Get the embeddings of the inputs / out [batch_size, sent_length, output_dim]\n",
    "embedding_out = layers.Embedding(input_dim=n_vocab, output_dim=64)(word_id_inputs)\n",
    "\n",
    "# For all layers: in [batch_size, sent_length, emb_size] / out [batch_size, sent_length, 100]\n",
    "conv1_1 = layers.Conv1D(100, kernel_size=3, \n",
    "                        strides=1, padding='same', \n",
    "                        activation='relu')(embedding_out)\n",
    "\n",
    "conv1_2 = layers.Conv1D(100, kernel_size=4,\n",
    "                        strides=1, padding='same', \n",
    "                        activation='relu')(embedding_out)\n",
    "\n",
    "conv1_3 = layers.Conv1D(100, kernel_size=5,\n",
    "                        strides=1, padding='same',\n",
    "                        activation='relu')(embedding_out)\n",
    "\n",
    "# in previous conve outputs / out [batch_size, sent_length, 300]\n",
    "conv_out = layers.Concatenate(axis=-1)([conv1_1, conv1_2, conv1_3])\n",
    "\n",
    "# Pooling over time operation. This is doing the max pooling over sequence length\n",
    "# in other words, each feature map results in a single output\n",
    "# in [batch_size, sent_length, 300] / out [batch_size, 1, 300]\n",
    "pool_over_time_out = layers.MaxPool1D(pool_size=max_seq_length,\n",
    "                                      padding='valid')(conv_out)\n",
    "\n",
    "# Flatten the unit length dimension\n",
    "flatten_out = layers.Flatten()(pool_over_time_out)\n",
    "\n",
    "# Compute the final output\n",
    "out = layers.Dense(n_classes, activation='softmax',\n",
    "                   kernel_regularizer=regularizers.l2(l2=0.001))(flatten_out)\n",
    "\n",
    "\n",
    "# Define the model\n",
    "cnn_model = Model(inputs=word_id_inputs, outputs=out)\n",
    "\n",
    "\n",
    "# Compile the model with loss/optimzier/metrics\n",
    "cnn_model.compile(loss='sparse_categorical_crossentropy', \n",
    "                  optimizer='adam', \n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "cnn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "119a4cea-9bd7-4813-8396-dff4799ed2bc",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Training the model\n",
    "\n",
    "- `ReduceLROnPlateau` - Reduces the learning rate when no improvement detected\n",
    "\n",
    "    - The technique we'll be using is known as \"decaying the learning rate.\" The idea is to reduce the learning rate (by some fraction) whenever the model has stopped to improve performance. The following callback assists us to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1e1bed52-6351-4d60-b717-e016c1294873",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "39/39 [==============================] - 7s 14ms/step - loss: 1.6203 - accuracy: 0.3934 - val_loss: 1.3731 - val_accuracy: 0.6044 - lr: 0.0010\n",
      "Epoch 2/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 1.0843 - accuracy: 0.6924 - val_loss: 0.8045 - val_accuracy: 0.7564 - lr: 0.0010\n",
      "Epoch 3/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.6154 - accuracy: 0.8223 - val_loss: 0.5615 - val_accuracy: 0.8388 - lr: 0.0010\n",
      "Epoch 4/25\n",
      "39/39 [==============================] - 0s 7ms/step - loss: 0.3430 - accuracy: 0.9164 - val_loss: 0.4540 - val_accuracy: 0.8645 - lr: 0.0010\n",
      "Epoch 5/25\n",
      "39/39 [==============================] - 0s 7ms/step - loss: 0.1919 - accuracy: 0.9653 - val_loss: 0.4147 - val_accuracy: 0.8736 - lr: 0.0010\n",
      "Epoch 6/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.1181 - accuracy: 0.9855 - val_loss: 0.4144 - val_accuracy: 0.8718 - lr: 0.0010\n",
      "Epoch 7/25\n",
      "39/39 [==============================] - 0s 7ms/step - loss: 0.0829 - accuracy: 0.9923 - val_loss: 0.4251 - val_accuracy: 0.8663 - lr: 0.0010\n",
      "Epoch 8/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0655 - accuracy: 0.9969 - val_loss: 0.4309 - val_accuracy: 0.8645 - lr: 0.0010\n",
      "Epoch 9/25\n",
      "38/39 [============================>.] - ETA: 0s - loss: 0.0561 - accuracy: 0.9990\n",
      "Epoch 9: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0560 - accuracy: 0.9990 - val_loss: 0.4380 - val_accuracy: 0.8645 - lr: 0.0010\n",
      "Epoch 10/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0514 - accuracy: 0.9994 - val_loss: 0.4364 - val_accuracy: 0.8645 - lr: 1.0000e-04\n",
      "Epoch 11/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0508 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-04\n",
      "Epoch 12/25\n",
      "32/39 [=======================>......] - ETA: 0s - loss: 0.0498 - accuracy: 0.9998\n",
      "Epoch 12: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0503 - accuracy: 0.9996 - val_loss: 0.4357 - val_accuracy: 0.8645 - lr: 1.0000e-04\n",
      "Epoch 13/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0500 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-05\n",
      "Epoch 14/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0499 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-05\n",
      "Epoch 15/25\n",
      "31/39 [======================>.......] - ETA: 0s - loss: 0.0502 - accuracy: 0.9995\n",
      "Epoch 15: ReduceLROnPlateau reducing learning rate to 1.0000000656873453e-06.\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0499 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-05\n",
      "Epoch 16/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 17/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 18/25\n",
      "30/39 [======================>.......] - ETA: 0s - loss: 0.0502 - accuracy: 0.9995\n",
      "Epoch 18: ReduceLROnPlateau reducing learning rate to 1e-06.\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 19/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 20/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 21/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 22/25\n",
      "39/39 [==============================] - 0s 7ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 23/25\n",
      "39/39 [==============================] - 0s 7ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 24/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n",
      "Epoch 25/25\n",
      "39/39 [==============================] - 0s 6ms/step - loss: 0.0498 - accuracy: 0.9996 - val_loss: 0.4356 - val_accuracy: 0.8645 - lr: 1.0000e-06\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x254ba090ac0>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# callbacks\n",
    "lr_reduce_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss',\n",
    "                                                         patience=3, verbose=1,\n",
    "                                                         min_delta=0.0001, min_lr=0.000001)\n",
    "\n",
    "\n",
    "# Train the model\n",
    "cnn_model.fit(preprocessed_train_sequences, train_labels,\n",
    "              validation_data=(preprocessed_valid_sequences, valid_labels),\n",
    "              batch_size=128,\n",
    "              epochs=25,\n",
    "              callbacks=[lr_reduce_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6908e1-1aaf-40e8-b861-f8f749eec4d5",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Test the model on test data\n",
    "\n",
    "> Test Accuracy: 88.6% for 500 test sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "dce4a52a-bd19-49f6-9500-76dfbebaa95b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16/16 [==============================] - 0s 14ms/step - loss: 0.3922 - accuracy: 0.8860\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'loss': 0.39220285415649414, 'accuracy': 0.8859999775886536}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnn_model.evaluate(preprocessed_test_sequences, test_labels, return_dict=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "361e0902-db64-43a4-adad-79ed9af53af7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}