# Unfolding Naïve Bayes from Scratch! Take-3 🎬.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unfolding Naïve Bayes from Scratch! Take-3 🎬\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cdn-images-1.medium.com/max/1000/1*sjet9qSO4O8fX2-FXvxflw.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Outcome of this Tutorial - A Hands-On Scikit-learn Implementation of NB \n",
    "A complete walk-through of NB implementation of NB using Python's Holy Grail of Machine Learning - Scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](pic2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with a few imports..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "from collections import defaultdict\n",
    "import re "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets first write a handy text preprocessing function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def preprocess_string(str_arg):\n",
    "    \n",
    "    \"\"\"\"\n",
    "        Parameters:\n",
    "        ----------\n",
    "        str_arg: example string to be preprocessed\n",
    "        \n",
    "        What the function does?\n",
    "        -----------------------\n",
    "        Preprocess the string argument - str_arg - such that :\n",
    "        1. everything apart from letters is excluded\n",
    "        2. multiple spaces are replaced by single space\n",
    "        3. str_arg is converted to lower case \n",
    "        \n",
    "        Example:\n",
    "        --------\n",
    "        Input :  Menu is absolutely perfect,loved it!\n",
    "        Output:  ['menu', 'is', 'absolutely', 'perfect', 'loved', 'it']\n",
    "        \n",
    "\n",
    "        Returns:\n",
    "        ---------\n",
    "        Preprocessed string \n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    cleaned_str=re.sub('[^a-z\\s]+',' ',str_arg,flags=re.IGNORECASE) #every char except alphabets is replaced\n",
    "    cleaned_str=re.sub('(\\s+)',' ',cleaned_str) #multiple spaces are replaced by single space\n",
    "    cleaned_str=cleaned_str.lower() #converting the cleaned string to lower case\n",
    "    \n",
    "    return cleaned_str # returning the preprocessed string in tokenized form"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading the 20newsgroup Dataset  \n",
    "#### The same dataset that was used earlier in [Unfolding Naïve Bayes from Scratch! Take-2 🎬](https://towardsdatascience.com/na%C3%AFve-bayes-from-scratch-using-python-only-no-fancy-frameworks-a1904b37222d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Number of Training Examples:  2257\n",
      "Total Number of Training Labels:  2257\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "######################### Loading Training Dataset ############################\n",
    "\n",
    "categories=['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med'] \n",
    "newsgroups_train=fetch_20newsgroups(subset='train',categories=categories)\n",
    "\n",
    "train_data=newsgroups_train.data #getting all training examples\n",
    "train_labels=newsgroups_train.target #getting training labels\n",
    "\n",
    "print (\"Total Number of Training Examples: \",len(train_data))\n",
    "print (\"Total Number of Training Labels: \",len(train_labels))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here is what the training dataset looks like in it's raw form .....  🤔 \n",
    "Training Examples : <br>\n",
    "    The 20 newsgroups dataset comprises around 18000 newsgroups posts on 20 topics \n",
    "    \n",
    "Training Labels : <br>\n",
    "    Training Labels are ['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian'] - where each training     label has its own unique integer id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Training Examples</th>\n",
       "      <th>Training Labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier)\\nSubj...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson)\\nSu...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart)\\nSubjec...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: stanly@grok11.columbiasc.ncr.com (stanly...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   Training Examples Training Labels\n",
       "0  From: sd345@city.ac.uk (Michael Collier)\\nSubj...               1\n",
       "1  From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\...               1\n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson)\\nSu...               3\n",
       "3  From: s0612596@let.rug.nl (M.M. Zwart)\\nSubjec...               3\n",
       "4  From: stanly@grok11.columbiasc.ncr.com (stanly...               3"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data=np.column_stack([train_data,train_labels]),columns=[\"Training Examples\",\"Training Labels\"]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's begin with the preprocessing of the training dataset that includes\n",
    "1. Text Cleaning\n",
    "2. Creating the BoW representation of our training Dataset (would need the same for test dataset as well)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Text Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Cleaning Done\n",
      "Total Number of Training Examples:  2257\n"
     ]
    }
   ],
   "source": [
    "train_data=[preprocess_string(train_str) for train_str in train_data]\n",
    "print (\"Data Cleaning Done\")\n",
    "print (\"Total Number of Training Examples: \",len(train_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here's what the processed training dataset looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Training Examples</th>\n",
       "      <th>Training Labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>from sd city ac uk michael collier subject con...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>from ani ms uky edu aniruddha b deglurkar subj...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>from djohnson cs ucsd edu darin johnson subjec...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>from s let rug nl m m zwart subject catholic c...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>from stanly grok columbiasc ncr com stanly sub...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   Training Examples Training Labels\n",
       "0  from sd city ac uk michael collier subject con...               1\n",
       "1  from ani ms uky edu aniruddha b deglurkar subj...               1\n",
       "2  from djohnson cs ucsd edu darin johnson subjec...               3\n",
       "3  from s let rug nl m m zwart subject catholic c...               3\n",
       "4  from stanly grok columbiasc ncr com stanly sub...               3"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data=np.column_stack([train_data,train_labels]),columns=[\"Training Examples\",\"Training Labels\"]).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Creating the BoW representation of our training Dataset (would need the same for test dataset as well)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2257, 31159)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer #simply import CountVectorizer\n",
    "count_vect = CountVectorizer() #instantiate it's object\n",
    "X_train_counts = count_vect.fit_transform(train_data) #builds a term-document matrix ands return it\n",
    "print (X_train_counts.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regarding CountVectorizer - as explained on [Scikit_learn](http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html)\n",
    "\n",
    "What the Countvectorizer Does?\n",
    "Takes in the text corpus, builds it's term document matrix (i.e BoW), and returns it\n",
    "\n",
    "Every word is assigned a fixed unique integer id and vale of each cell of this matrix represents the word\n",
    "count - BoW\n",
    "\n",
    "So for example X_train_counts[ i , j ]- where i refers to a document which in our case each document specifies a training example and j refers to the index of a word w in it's respective document i- would return count of word j \n",
    "\n",
    "X_train_counts[0,12048] will retrieve the word count of word with the integer id = 12048 and domcent/example \n",
    "id 0\n",
    "\n",
    "You can read more about Sklearn CountVectorizer at [CountVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "print (X_train_counts[0,12048])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 12048)\t1\n",
      "  (0, 8371)\t1\n",
      "  (0, 9880)\t1\n",
      "  (0, 16147)\t1\n",
      "  (0, 27392)\t1\n",
      "  (0, 29019)\t1\n",
      "  (0, 5237)\t1\n",
      "  (0, 21703)\t1\n",
      "  (0, 484)\t1\n",
      "  (0, 13293)\t1\n",
      "  (0, 27608)\t1\n",
      "  (0, 11615)\t1\n",
      "  (0, 5831)\t1\n",
      "  (0, 27740)\t1\n",
      "  (0, 14221)\t1\n",
      "  (0, 23397)\t1\n",
      "  (0, 1283)\t1\n",
      "  (0, 8664)\t2\n",
      "  (0, 20880)\t1\n",
      "  (0, 20903)\t1\n",
      "  (0, 12679)\t1\n",
      "  (0, 24151)\t1\n",
      "  (0, 7862)\t1\n",
      "  (0, 15887)\t1\n",
      "  (0, 898)\t1\n",
      "  :\t:\n",
      "  (2256, 28544)\t1\n",
      "  (2256, 2499)\t1\n",
      "  (2256, 19646)\t1\n",
      "  (2256, 17923)\t1\n",
      "  (2256, 31008)\t2\n",
      "  (2256, 27703)\t1\n",
      "  (2256, 30555)\t1\n",
      "  (2256, 1027)\t1\n",
      "  (2256, 30326)\t1\n",
      "  (2256, 1780)\t1\n",
      "  (2256, 22605)\t2\n",
      "  (2256, 10403)\t1\n",
      "  (2256, 16988)\t1\n",
      "  (2256, 3703)\t1\n",
      "  (2256, 8455)\t2\n",
      "  (2256, 5237)\t1\n",
      "  (2256, 13293)\t1\n",
      "  (2256, 27740)\t1\n",
      "  (2256, 14221)\t2\n",
      "  (2256, 19212)\t2\n",
      "  (2256, 15943)\t1\n",
      "  (2256, 27615)\t6\n",
      "  (2256, 19515)\t1\n",
      "  (2256, 26575)\t1\n",
      "  (2256, 10690)\t1\n"
     ]
    }
   ],
   "source": [
    "print (X_train_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# That's it!!! Let's Move to Training! ⛸⛸⛸"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Completed\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB #importing the Sklearn's NB Fucntionality\n",
    "clf = MultinomialNB() #simply instantiate a Multinomial Naive Bayes object\n",
    "clf.fit(X_train_counts, train_labels)  #calling the fit method trains it\n",
    "print (\"Training Completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So Now That We Have Trained NB Model - Let's Move to Testing! 🏄🏽🏄🏽🏄🏽"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Test Examples:  1502\n",
      "Number of Test Labels:  1502\n",
      "Number of Test Examples:  1502\n"
     ]
    }
   ],
   "source": [
    "newsgroups_test=fetch_20newsgroups(subset='test',categories=categories) #loading test data\n",
    "test_data=newsgroups_test.data #get test set examples\n",
    "test_labels=newsgroups_test.target #get test set labels\n",
    "\n",
    "print (\"Number of Test Examples: \",len(test_data))\n",
    "print (\"Number of Test Labels: \",len(test_labels))\n",
    "\n",
    "test_data=[preprocess_string(test_str) for test_str in test_data] #need to preporcess the test set as well!!\n",
    "print (\"Number of Test Examples: \",len(test_data))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same count_vect object that was instantiated for training dataset will be used for test dataset.\n",
    "But remeber that we are not calling fit_transform(since we only want to transform the test data into a term-document matrix whereas fit_transform fit_transform learns the vocabulary dictionary first and then returns a term-document matrix. We are supposed to learn the vocabulary on training dataset only\n",
    "\n",
    "fit_transform : learns the vocabulary dictionary and returns term-document matrix\n",
    "transform : transforms documents to document-term matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1502, 31159)\n"
     ]
    }
   ],
   "source": [
    "X_test_counts=count_vect.transform(test_data) #transforms test data to numerical form\n",
    "print (X_test_counts.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now we can test on the transformed version of test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Set Accuracy :  0.936085219707\n"
     ]
    }
   ],
   "source": [
    "predicted=clf.predict(X_test_counts)\n",
    "print (\"Test Set Accuracy : \",np.sum(predicted==test_labels)/float(len(predicted))) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The above code can be further reduced to literally 3 lines of code by using the pipeline functionality of sklearn!\n",
    "\n",
    "# It's truly the ML Holy Toolkit!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline #importing the pipeline functionality\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "    We simply build a pipeline object by specifying the pipeline actions and once that pipeline object is\n",
    "    used for the trainign purpose, it will automatically perform the pipeline steps int he specified order.\n",
    "    In our case, as we first want to build a CountVectorizer for the purpose of BoW, and then fit/train a \n",
    "    NB model, so in exectly the same manner, we will speicify these actions in our pipeline. \n",
    "    \n",
    "    Do note that, now when calling the fit method, we will pass the original textual data as now\n",
    "    the count_vect in pipeline will itself convert it to numeric form. So it's important here that we\n",
    "    pass the textual data or else nasty errros will pop out. Same is the case for test data as well. No need\n",
    "    to count vectorize it separately :) But we do need to preprocess the test data from cleaning point of view\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "clf=Pipeline([('count_vect', CountVectorizer()),('clf', MultinomialNB())])\n",
    "clf.fit(train_data,train_labels)  \n",
    "print (\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1502\n",
      "Test Set Accuracy :  0.936085219707\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print (len(test_data))\n",
    "predicted=clf.predict(test_data)\n",
    "print (\"Test Set Accuracy : \",np.sum(predicted==test_labels)/float(len(predicted))) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}