Code/RNN/2. NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Domain:\n",
    "* Natural language processing (NLP)\n",
    "\n",
    "## Data source:\n",
    "* PubMed abstracts (n = 10,000)\n",
    "* From PubMed 200k RCT: https://arxiv.org/pdf/1710.06071.pdf\n",
    "* https://github.com/Franck-Dernoncourt/pubmed-rct\n",
    "\n",
    "\n",
    "## Prediction task:\n",
    "* Given a random sentence from a PubMed abstract, predict which abstract section the sentence came from:\n",
    "    * Background\n",
    "    * Objective\n",
    "    * Methods\n",
    "    * Results\n",
    "    * Conclusions\n",
    "* Multiclass (5) classification\n",
    "\n",
    "## Model:\n",
    "* Recurrent neural network (RNN) with gated recurrent units (GRU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "import sklearn.utils\n",
    "from sklearn.metrics import classification_report, accuracy_score, roc_auc_score\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import GRU, Dense,  Masking, Dropout, Embedding\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Source: https://raw.githubusercontent.com/Franck-Dernoncourt/pubmed-rct/master/PubMed_20k_RCT/train.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examine raw text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "###24845963\n",
      "\n",
      "BACKGROUND\tThis study analyzed liver function abnormalities in heart failure patients admitted with severe acute decompensated heart failure ( ADHF ) .\n",
      "\n",
      "RESULTS\tA post hoc analysis was conducted with the use of data from the Evaluation Study of Congestive Heart Failure and Pulmonary Artery Catheterization Effectiveness ( ESCAPE ) .\n",
      "\n",
      "RESULTS\tLiver function tests ( LFTs ) were measured at 7 time points from baseline , at discharge , and up to 6 months follow-up .\n",
      "\n",
      "RESULTS\tSurvival analyses were used to assess the association between admission Model of End-Stage Liver Disease Excluding International Normalized Ratio ( MELD-XI ) scores and patient outcome.There was a high prevalence of abnormal baseline ( admission ) LFTs ( albumin 23.8 % , aspartate transaminase 23.5 % , alanine transaminase 23.8 % , and total bilirubin 36.1 % ) .\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('data/pubmed.txt', 'r') as f:\n",
    "    for line in f.readlines()[:5]:\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load into dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LABEL</th>\n",
       "      <th>TEXT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>This study analyzed liver function abnormaliti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>A post hoc analysis was conducted with the use...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>Liver function tests ( LFTs ) were measured at...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>Survival analyses were used to assess the asso...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>The percentage of patients with abnormal LFTs ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>When mean hemodynamic profiles were compared i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>Multivariable analyses revealed that patients ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>CONCLUSIONS</td>\n",
       "      <td>Abnormal LFTs are common in the ADHF populatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>CONCLUSIONS</td>\n",
       "      <td>Elevated MELD-XI scores are associated with po...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>###24469619</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>Minimally invasive endovascular aneurysm repai...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>The aim of this study was to analyse the cost-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          LABEL                                               TEXT\n",
       "0    BACKGROUND  This study analyzed liver function abnormaliti...\n",
       "1       RESULTS  A post hoc analysis was conducted with the use...\n",
       "2       RESULTS  Liver function tests ( LFTs ) were measured at...\n",
       "3       RESULTS  Survival analyses were used to assess the asso...\n",
       "4       RESULTS  The percentage of patients with abnormal LFTs ...\n",
       "5       RESULTS  When mean hemodynamic profiles were compared i...\n",
       "6       RESULTS  Multivariable analyses revealed that patients ...\n",
       "7   CONCLUSIONS  Abnormal LFTs are common in the ADHF populatio...\n",
       "8   CONCLUSIONS  Elevated MELD-XI scores are associated with po...\n",
       "9   ###24469619                                                NaN\n",
       "10   BACKGROUND  Minimally invasive endovascular aneurysm repai...\n",
       "11   BACKGROUND  The aim of this study was to analyse the cost-..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/pubmed.txt',\n",
    "                 sep='\\t',\n",
    "                 header=None,\n",
    "                 skiprows=1,\n",
    "                 names=['LABEL', 'TEXT'])\n",
    "df[:12]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Remove metadata rows ('#24845963', etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LABEL</th>\n",
       "      <th>TEXT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>This study analyzed liver function abnormaliti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>A post hoc analysis was conducted with the use...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>Liver function tests ( LFTs ) were measured at...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>Survival analyses were used to assess the asso...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>The percentage of patients with abnormal LFTs ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        LABEL                                               TEXT\n",
       "0  BACKGROUND  This study analyzed liver function abnormaliti...\n",
       "1     RESULTS  A post hoc analysis was conducted with the use...\n",
       "2     RESULTS  Liver function tests ( LFTs ) were measured at...\n",
       "3     RESULTS  Survival analyses were used to assess the asso...\n",
       "4     RESULTS  The percentage of patients with abnormal LFTs ..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df[df['LABEL'].isin(['BACKGROUND', 'OBJECTIVE', 'METHODS', 'RESULTS', 'CONCLUSIONS'])].reset_index(drop=True)\n",
    "df[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examine distribution of labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "METHODS        9897\n",
       "RESULTS        9713\n",
       "CONCLUSIONS    4571\n",
       "BACKGROUND     3621\n",
       "OBJECTIVE      2333\n",
       "Name: LABEL, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['LABEL'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build example balanced dataset (2,000 sentences per label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LABEL</th>\n",
       "      <th>TEXT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5626</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>In recent years , the surgical step-up approac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17991</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>Peripheral opioid receptor targeting has been ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3338</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>A constructive safety culture is essential for...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13095</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>NCT00679809 .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19278</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>We aimed to compare gefitinib with placebo in ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            LABEL                                               TEXT\n",
       "5626   BACKGROUND  In recent years , the surgical step-up approac...\n",
       "17991  BACKGROUND  Peripheral opioid receptor targeting has been ...\n",
       "3338   BACKGROUND  A constructive safety culture is essential for...\n",
       "13095  BACKGROUND                                      NCT00679809 .\n",
       "19278  BACKGROUND  We aimed to compare gefitinib with placebo in ..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.concat([df[df['LABEL'] == label].sample(2000) for label in ['BACKGROUND', 'OBJECTIVE', 'METHODS', 'RESULTS', 'CONCLUSIONS']])\n",
    "df[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shuffle rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LABEL</th>\n",
       "      <th>TEXT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BACKGROUND</td>\n",
       "      <td>One hypothesis suggests that the differential ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OBJECTIVE</td>\n",
       "      <td>To investigate whether learning basic life sup...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>All patients received six doses of study medic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RESULTS</td>\n",
       "      <td>No significant differences were found between ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OBJECTIVE</td>\n",
       "      <td>The authors studied the immediate and long-ter...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        LABEL                                               TEXT\n",
       "0  BACKGROUND  One hypothesis suggests that the differential ...\n",
       "1   OBJECTIVE  To investigate whether learning basic life sup...\n",
       "2     RESULTS  All patients received six doses of study medic...\n",
       "3     RESULTS  No significant differences were found between ...\n",
       "4   OBJECTIVE  The authors studied the immediate and long-ter..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.sample(frac=1).reset_index(drop=True)\n",
    "df[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extract text (documents) and corresponding labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"One hypothesis suggests that the differential response to ondansetron - and serotonin-specific re-uptake inhibitors ( SSRIs ) may be due to a functional polymorphism of the 5 ' - HTTLPR promoter region in SLC6A4 , the gene that codes for the serotonin transporter ( 5-HTT ) .\",\n",
       " 'To investigate whether learning basic life support ( BLS ) and cardiopulmonary resuscitation ( CPR ) from video produce higher learning outcomes compared to pictures in reciprocal learning .',\n",
       " 'All patients received six doses of study medication .',\n",
       " 'No significant differences were found between the two study groups .',\n",
       " 'The authors studied the immediate and long-term performance and complications of two twin-catheter systems , the Tesio catheter ( TC ) and the LifeCath Twin ( LC ) , to inform clinical practice .']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents = list(df['TEXT'].values)\n",
    "documents[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BACKGROUND', 'OBJECTIVE', 'RESULTS', 'RESULTS', 'OBJECTIVE']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = list(df['LABEL'].values)\n",
    "labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert string label to integer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 3, 3, 1]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "UNIQUE_LABELS = ['BACKGROUND', 'OBJECTIVE', 'METHODS', 'RESULTS', 'CONCLUSIONS']\n",
    "y = [UNIQUE_LABELS.index(label) for label in labels]\n",
    "y[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Preprocess data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lowercase all documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"one hypothesis suggests that the differential response to ondansetron - and serotonin-specific re-uptake inhibitors ( ssris ) may be due to a functional polymorphism of the 5 ' - httlpr promoter region in slc6a4 , the gene that codes for the serotonin transporter ( 5-htt ) .\",\n",
       " 'to investigate whether learning basic life support ( bls ) and cardiopulmonary resuscitation ( cpr ) from video produce higher learning outcomes compared to pictures in reciprocal learning .',\n",
       " 'all patients received six doses of study medication .',\n",
       " 'no significant differences were found between the two study groups .',\n",
       " 'the authors studied the immediate and long-term performance and complications of two twin-catheter systems , the tesio catheter ( tc ) and the lifecath twin ( lc ) , to inform clinical practice .']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents = [document.lower() for document in documents]\n",
    "documents[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenize each document (naive whitespace tokenization)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['one',\n",
       " 'hypothesis',\n",
       " 'suggests',\n",
       " 'that',\n",
       " 'the',\n",
       " 'differential',\n",
       " 'response',\n",
       " 'to',\n",
       " 'ondansetron',\n",
       " '-',\n",
       " 'and',\n",
       " 'serotonin-specific',\n",
       " 're-uptake',\n",
       " 'inhibitors',\n",
       " '(',\n",
       " 'ssris',\n",
       " ')',\n",
       " 'may',\n",
       " 'be',\n",
       " 'due',\n",
       " 'to',\n",
       " 'a',\n",
       " 'functional',\n",
       " 'polymorphism',\n",
       " 'of',\n",
       " 'the',\n",
       " '5',\n",
       " \"'\",\n",
       " '-',\n",
       " 'httlpr',\n",
       " 'promoter',\n",
       " 'region',\n",
       " 'in',\n",
       " 'slc6a4',\n",
       " ',',\n",
       " 'the',\n",
       " 'gene',\n",
       " 'that',\n",
       " 'codes',\n",
       " 'for',\n",
       " 'the',\n",
       " 'serotonin',\n",
       " 'transporter',\n",
       " '(',\n",
       " '5-htt',\n",
       " ')',\n",
       " '.']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_documents = [document.split() for document in documents]\n",
    "tokenized_documents[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Other optional methods (not included in this example):\n",
    "* Preserving original capitalization  \n",
    "  \n",
    "* Removing punctuation\n",
    "\n",
    "* Splitting hyphenated words\n",
    "    * \"health-based\" --> \"health\", \"based\"\n",
    "    \n",
    "* Replacing numbers with special tokens\n",
    "    * \"1.4 mcg\" --> \"&lt;NUM&gt; mcg\"\n",
    "    \n",
    "* Replacing URL links with special tokens\n",
    "    \n",
    "* Removing stop words\n",
    "    * \"the\", \"a\", \"to\", \"of\", etc.\n",
    "\n",
    "* Stemming\n",
    "    * \"working\" --> \"work\"\n",
    "    * \"studies\" --> \"studi\"\n",
    "    * \"studying\" --> \"study\"\n",
    "* Lemmatizing\n",
    "    * \"am\", \"are\", \"is\" --> \"be\"\n",
    "    * \"studies\" --> \"study\"\n",
    "    * \"studying\" --> \"study\"\n",
    "    * \"better\" --> \"good\"\n",
    "    \n",
    "* More complex tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Convert text to word indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Unique tokens in vocabulary = 20331\n"
     ]
    }
   ],
   "source": [
    "token2idx = {'<PAD_TOKEN>': 0}\n",
    "idx2token = ['<PAD_TOKEN>']\n",
    "\n",
    "for tokenized_document in tokenized_documents:\n",
    "    for token in tokenized_document:\n",
    "        if token not in token2idx:\n",
    "            token2idx[token] = len(token2idx)\n",
    "            idx2token.append(token)\n",
    "            \n",
    "print('# Unique tokens in vocabulary = %d' % len(token2idx))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<PAD_TOKEN>': 0,\n",
       " 'one': 1,\n",
       " 'hypothesis': 2,\n",
       " 'suggests': 3,\n",
       " 'that': 4,\n",
       " 'the': 5,\n",
       " 'differential': 6,\n",
       " 'response': 7,\n",
       " 'to': 8,\n",
       " 'ondansetron': 9}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{k: v for (k,v) in list(token2idx.items())[:10]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'that'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx2token[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ondansetron'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx2token[9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<PAD_TOKEN>'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx2token[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replace text tokens with integer indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = []\n",
    "for tokenized_document in tokenized_documents:\n",
    "    X.append([token2idx[token] for token in tokenized_document])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1,\n",
       "  2,\n",
       "  3,\n",
       "  4,\n",
       "  5,\n",
       "  6,\n",
       "  7,\n",
       "  8,\n",
       "  9,\n",
       "  10,\n",
       "  11,\n",
       "  12,\n",
       "  13,\n",
       "  14,\n",
       "  15,\n",
       "  16,\n",
       "  17,\n",
       "  18,\n",
       "  19,\n",
       "  20,\n",
       "  8,\n",
       "  21,\n",
       "  22,\n",
       "  23,\n",
       "  24,\n",
       "  5,\n",
       "  25,\n",
       "  26,\n",
       "  10,\n",
       "  27,\n",
       "  28,\n",
       "  29,\n",
       "  30,\n",
       "  31,\n",
       "  32,\n",
       "  5,\n",
       "  33,\n",
       "  4,\n",
       "  34,\n",
       "  35,\n",
       "  5,\n",
       "  36,\n",
       "  37,\n",
       "  15,\n",
       "  38,\n",
       "  17,\n",
       "  39],\n",
       " [8,\n",
       "  40,\n",
       "  41,\n",
       "  42,\n",
       "  43,\n",
       "  44,\n",
       "  45,\n",
       "  15,\n",
       "  46,\n",
       "  17,\n",
       "  11,\n",
       "  47,\n",
       "  48,\n",
       "  15,\n",
       "  49,\n",
       "  17,\n",
       "  50,\n",
       "  51,\n",
       "  52,\n",
       "  53,\n",
       "  42,\n",
       "  54,\n",
       "  55,\n",
       "  8,\n",
       "  56,\n",
       "  30,\n",
       "  57,\n",
       "  42,\n",
       "  39],\n",
       " [58, 59, 60, 61, 62, 24, 63, 64, 39],\n",
       " [65, 66, 67, 68, 69, 70, 5, 71, 63, 72, 39],\n",
       " [5,\n",
       "  73,\n",
       "  74,\n",
       "  5,\n",
       "  75,\n",
       "  11,\n",
       "  76,\n",
       "  77,\n",
       "  11,\n",
       "  78,\n",
       "  24,\n",
       "  71,\n",
       "  79,\n",
       "  80,\n",
       "  32,\n",
       "  5,\n",
       "  81,\n",
       "  82,\n",
       "  15,\n",
       "  83,\n",
       "  17,\n",
       "  11,\n",
       "  5,\n",
       "  84,\n",
       "  85,\n",
       "  15,\n",
       "  86,\n",
       "  17,\n",
       "  32,\n",
       "  8,\n",
       "  87,\n",
       "  88,\n",
       "  89,\n",
       "  39]]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Prepare data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examine distribution of sequence lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10000.000000\n",
       "mean        25.417900\n",
       "std         14.265715\n",
       "min          1.000000\n",
       "25%         16.000000\n",
       "50%         23.000000\n",
       "75%         32.000000\n",
       "max        211.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_lengths = [len(x) for x in X]\n",
    "pd.Series(sequence_lengths).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split into train, validation, and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Train documents = 6400\n",
      "# Validation documents = 1600\n",
      "# Test documents = 2000\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True)\n",
    "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, shuffle=True)\n",
    "\n",
    "print('# Train documents = %d' % len(y_train))\n",
    "print('# Validation documents = %d' % len(y_val))\n",
    "print('# Test documents = %d' % len(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. RNN (GRU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model using Keras Sequential API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(vocab_size=len(token2idx), n_classes=5, embedding_dim=100, gru_dim=32, dropout=0.2, optimizer='adam'):\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(input_dim=vocab_size,\n",
    "                        output_dim=embedding_dim,\n",
    "                        mask_zero=True))\n",
    "    model.add(GRU(units=gru_dim))\n",
    "    model.add(Dropout(rate=dropout))\n",
    "    model.add(Dense(n_classes, activation='softmax'))\n",
    "    model.compile(loss='sparse_categorical_crossentropy',\n",
    "                  optimizer=optimizer,\n",
    "                  metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Batch generator for batch-wise padding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batch_generator(X, y, batch_size=64, shuffle=False):\n",
    "    while True:\n",
    "        if shuffle:\n",
    "            X, y = sklearn.utils.shuffle(X, y)\n",
    "            \n",
    "        for idx in range(0, len(X), batch_size):\n",
    "            X_batch = X[idx:idx + batch_size]\n",
    "            y_batch = y[idx:idx + batch_size]\n",
    "            \n",
    "            batch_sequence_lengths = [len(x) for x in X_batch]\n",
    "            X_batch = pad_sequences(sequences=X_batch,\n",
    "                                    maxlen=max(batch_sequence_lengths),\n",
    "                                    padding='post',\n",
    "                                    value=0)\n",
    "            yield X_batch, y_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# train batches = 100\n",
      "# val batches = 25\n",
      "# test batches = 32\n"
     ]
    }
   ],
   "source": [
    "batch_size = 64\n",
    "\n",
    "n_train_batches = int(np.ceil(len(X_train) / batch_size))\n",
    "n_val_batches = int(np.ceil(len(X_val) / batch_size))\n",
    "n_test_batches = int(np.ceil(len(X_test) / batch_size))\n",
    "\n",
    "train_generator = batch_generator(X_train, y_train)\n",
    "val_generator = batch_generator(X_val, y_val)\n",
    "test_generator = batch_generator(X_test, y_test)\n",
    "\n",
    "print('# train batches = %d' % n_train_batches)\n",
    "print('# val batches = %d' % n_val_batches)\n",
    "print('# test batches = %d' % n_test_batches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "100/100 [==============================] - 13s 132ms/step - loss: 1.4941 - acc: 0.3237 - val_loss: 1.2374 - val_acc: 0.4469\n",
      "Epoch 2/5\n",
      "100/100 [==============================] - 13s 130ms/step - loss: 1.0086 - acc: 0.5588 - val_loss: 1.0840 - val_acc: 0.5413\n",
      "Epoch 3/5\n",
      "100/100 [==============================] - 13s 131ms/step - loss: 0.6441 - acc: 0.7319 - val_loss: 1.1431 - val_acc: 0.5669\n",
      "Epoch 4/5\n",
      "100/100 [==============================] - 13s 130ms/step - loss: 0.3946 - acc: 0.8517 - val_loss: 1.3342 - val_acc: 0.5962\n",
      "Epoch 5/5\n",
      "100/100 [==============================] - 13s 128ms/step - loss: 0.2304 - acc: 0.9311 - val_loss: 1.5378 - val_acc: 0.5881\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fa4f0477be0>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = build_model()\n",
    "\n",
    "model.fit_generator(generator=train_generator,\n",
    "                    steps_per_epoch=n_train_batches,\n",
    "                    validation_data=val_generator,\n",
    "                    validation_steps=n_val_batches,\n",
    "                    epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get class prediction probabilities for test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_proba = model.predict_generator(generator=test_generator,\n",
    "                                       steps=n_test_batches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.6741875e-05, 2.1852116e-04, 9.8095691e-01, 3.9991997e-03,\n",
       "        1.4728604e-02],\n",
       "       [1.2877173e-03, 2.3125400e-04, 7.8454393e-01, 1.9392265e-03,\n",
       "        2.1199793e-01],\n",
       "       [8.5129309e-04, 8.2725929e-03, 8.9508820e-01, 7.7318639e-02,\n",
       "        1.8469281e-02],\n",
       "       [6.2861266e-03, 7.9566188e-02, 5.1076740e-01, 3.8030130e-01,\n",
       "        2.3078980e-02],\n",
       "       [2.6253971e-01, 7.1709919e-01, 5.9131352e-04, 2.1120450e-03,\n",
       "        1.7657692e-02]], dtype=float32)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_proba[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert class probabilities to integer label prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(y_pred_proba, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 2, 2, 2, 1])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set accuracy = 0.577\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "  BACKGROUND       0.45      0.55      0.49       391\n",
      "   OBJECTIVE       0.57      0.52      0.54       394\n",
      "     METHODS       0.59      0.72      0.65       381\n",
      "     RESULTS       0.80      0.63      0.70       413\n",
      " CONCLUSIONS       0.53      0.48      0.51       421\n",
      "\n",
      "   micro avg       0.58      0.58      0.58      2000\n",
      "   macro avg       0.59      0.58      0.58      2000\n",
      "weighted avg       0.59      0.58      0.58      2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print('Test set accuracy = %.3f' % accuracy)\n",
    "print()\n",
    "print(classification_report(y_test, y_pred, target_names=UNIQUE_LABELS))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}