news_data_mining.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Mining \n",
    "\n",
    "For Data Mining Lab (2017 Fall)\n",
    "\n",
    "Lectured by Elvis Saravia ([Website](http://www.elvissaravia.com) | [Twitter](http://www.twitter.com/omarsar0))\n",
    "\n",
    "Accompanying slides here ([Link](https://github.com/omarsar/data_mining_2017_fall_lab/blob/master/Data%20Mining%20Lab%201%20-%202017.pdf))\n",
    "\n",
    "Reference slides (Session 1 Handout - Please obtain from iLMS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of Contents\n",
    "1. Data Source\n",
    "2. Data Preparation\n",
    "3. So What's Next\n",
    " - 3.1 Converting Dictionary into Pandas dataframe\n",
    " - 3.2 Familiarizing yourself with the Data\n",
    "4. Data Mining using Pandas\n",
    " - 4.1 Dealing with Missing Values\n",
    " - 4.2 Dealing with Duplicate Data\n",
    "5. Data Preprocessing\n",
    " - 5.1 Sampling\n",
    " - 5.2 Feature Creation\n",
    " - 5.3 Feature Subset Selection\n",
    " - 5.4 Dimensionality Reduction\n",
    " - 5.5 Atrribute Transformation / Aggregation\n",
    " - 5.6 Discretization and Binarization\n",
    "6. Conclusion\n",
    "7. References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "In this notebook I will explore a text-based, document-based [dataset](http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html) using scientific computing tools such as Pandas and Numpy. In addition, several fundamental Data Mining concepts will be explored and explained in details, ranging from calculating distance measures to computing term frequency vectors to visualizing embeddings. Coding examples, visualizations and demonstrations will be provided where necessary. Furthermore, additional exercises are provided after special topics. These exercises are geared towards testing the understanding of students and also for students to explore beyond, through programming practises, the techniques covered in the notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements\n",
    "Here are the computing and software requirements\n",
    "\n",
    "#### Computing Resources\n",
    "- Operating system: Preferably Linux or MacOS\n",
    "- RAM: 8GB\n",
    "- Disk space: Mininium 8GB\n",
    "\n",
    "#### Software Requirements\n",
    "Here is a list of the required programs and libraries necessary for this lab session:\n",
    "- [Python 3+](https://www.python.org/download/releases/3.0/) (Note: coding will be done strictly on Python 3)\n",
    "    - Install latest version of Python 3\n",
    "- [Anaconda](https://www.anaconda.com/download/) environemnt or any other environement (recommended but not required)\n",
    "    - Install anaconda environment\n",
    "- [Jupyter](http://jupyter.org/) (Strongly recommended but not required)\n",
    "    - Install jupyter\n",
    "- [Scikit Learn](http://scikit-learn.org/stable/index.html)\n",
    "    - Install `sklearn` latest python library\n",
    "- [Pandas](http://pandas.pydata.org/)\n",
    "    - Install `pandas` python library\n",
    "- [Numpy](http://www.numpy.org/)\n",
    "    - Install `numpy` python library\n",
    "- [Matplotlib](https://matplotlib.org/)\n",
    "    - Install `maplotlib` for python\n",
    "- [Plotly](https://plot.ly/)\n",
    "    - Install and signup for `plotly`\n",
    "- [NLTK](http://www.nltk.org/)\n",
    "    - Install `nltk` library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# necessary for when working with external scripts\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "import math\n",
    "%matplotlib inline\n",
    "\n",
    "# my functions\n",
    "import helpers.data_mining_helpers as dmh\n",
    "import helpers.text_analysis as ta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data source\n",
    "In this notebook we will explore the popular 20 newsgroup dataset, originally provided [here](http://qwone.com/~jason/20Newsgroups/). The dataset is called \"Twenty Newsgroups\", which means there are 20 categories of news articles available in the entire dataset. A short description of the dataset, provided by the authors, is provided below:\n",
    "\n",
    "- *The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of our knowledge, it was originally collected by Ken Lang, probably for his paper “Newsweeder: Learning to filter netnews,” though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.*\n",
    "\n",
    "If you need more information about the dataset please refer to the reference provided above. Below is a snapshot of the dataset already converted into a table. Keep in mind that the original dataset is not in this nice pretty format. That work is left to us. That is one of the tasks that will be covered in this notebook: how to convert raw data into convenient tabular formats using Pandas. \n",
    "\n",
    "![atl txt](https://docs.google.com/drawings/d/e/2PACX-1vRd845nNXa1x1Enw6IoEbg-05lB19xG3mfO2BjnpZrloT0pSnY89stBV1gS9Iu6cgRCTq3E5giIT5ZI/pub?w=835&h=550)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Data Preparation\n",
    "Now let us begin to explore the data. The original dataset can be found on the link provided above or you can directly use the version provided by scikit learn. Here we will use the scikit learn version. \n",
    "\n",
    "In this demonstration we are only going to look at 4 categories. This means we will not make use of the complete dataset, but only a subset of it, which includes the 4 categories defined below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# categories\n",
    "categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# obtain the documents containing the categories provided\n",
    "twenty_train = fetch_20newsgroups(subset='train', categories=categories, \\\n",
    "                                  shuffle=True, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take at look some of the records that are contained in our subset of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert tif/img/tga files into LaserJet III format.  We would also like to\\ndo the same, converting to HPGL (HP plotter) files.\\n\\nPlease email any response.\\n\\nIs this the correct group?\\n\\nThanks in advance.  Michael.\\n-- \\nMichael Collier (Programmer)                 The Computer Unit,\\nEmail: M.P.Collier@uk.ac.city                The City University,\\nTel: 071 477-8000 x3769                      London,\\nFax: 071 477-8565                            EC1V 0HB.\\n',\n",
       " \"From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\nSubject: help: Splitting a trimming region along a mesh \\nOrganization: University Of Kentucky, Dept. of Math Sciences\\nLines: 28\\n\\n\\n\\n\\tHi,\\n\\n\\tI have a problem, I hope some of the 'gurus' can help me solve.\\n\\n\\tBackground of the problem:\\n\\tI have a rectangular mesh in the uv domain, i.e  the mesh is a \\n\\tmapping of a 3d Bezier patch into 2d. The area in this domain\\n\\twhich is inside a trimming loop had to be rendered. The trimming\\n\\tloop is a set of 2d Bezier curve segments.\\n\\tFor the sake of notation: the mesh is made up of cells.\\n\\n\\tMy problem is this :\\n\\tThe trimming area has to be split up into individual smaller\\n\\tcells bounded by the trimming curve segments. If a cell\\n\\tis wholly inside the area...then it is output as a whole ,\\n\\telse it is trivially rejected. \\n\\n\\tDoes any body know how thiss can be done, or is there any algo. \\n\\tsomewhere for doing this.\\n\\n\\tAny help would be appreciated.\\n\\n\\tThanks, \\n\\tAni.\\n-- \\nTo get irritated is human, to stay cool, divine.\\n\"]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.data[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** the `twenty_train` is just a bunch of objects that can be accessed as python dictionaries; so, you can do the following operations on `twenty_train`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2257"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(twenty_train.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2257"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(twenty_train.filenames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can also print an example from the subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: sd345@city.ac.uk (Michael Collier)\n",
      "Subject: Converting images to HP LaserJet III?\n",
      "Nntp-Posting-Host: hampton\n",
      "Organization: The City University\n",
      "Lines: 14\n",
      "\n",
      "Does anyone know of a good way (standard PC application/PD utility) to\n",
      "convert tif/img/tga files into LaserJet III format.  We would also like to\n",
      "do the same, converting to HPGL (HP plotter) files.\n",
      "\n",
      "Please email any response.\n",
      "\n",
      "Is this the correct group?\n",
      "\n",
      "Thanks in advance.  Michael.\n",
      "-- \n",
      "Michael Collier (Programmer)                 The Computer Unit,\n",
      "Email: M.P.Collier@uk.ac.city                The City University,\n",
      "Tel: 071 477-8000 x3769                      London,\n",
      "Fax: 071 477-8565                            EC1V 0HB.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# An example of what the subset contains\n",
    "print(\"\\n\".join(twenty_train.data[0].split(\"\\n\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and determine the label of the example via `target_names` key value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "comp.graphics\n"
     ]
    }
   ],
   "source": [
    "print(twenty_train.target_names[twenty_train.target[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... we can also get the category of 10 documents via `target` key value "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 3, 3, 3, 3, 3, 2, 2, 2])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# category of first 10 documents.\n",
    "twenty_train.target[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: As you can observe, both approaches above provide two different ways of obtaining the `category` value for the dataset. Ideally, we want to have access to both types -- numerical and nominal -- in the event some particular library favors a particular type. \n",
    "\n",
    "As you may have already noticed as well, there is no **tabular format** for the current version of the data. As data miners, we are interested in having our dataset in the most convenient format as possible; something we can manipulate easily and is compatible with our algorithms, and so forth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is one way to get access to the *text* version of the label of a subset of our training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "comp.graphics\n",
      "comp.graphics\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "soc.religion.christian\n",
      "sci.med\n",
      "sci.med\n",
      "sci.med\n"
     ]
    }
   ],
   "source": [
    "for t in twenty_train.target[:10]:\n",
    "    print(twenty_train.target_names[t])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. So What's next?\n",
    "So we want to explore and understand our data a little bit better. Before we do that we definitely need to make some conversions here and there just so we can have our dataset in a nice format to be able to explore it freely. Lucky for us, there are efficient scientific tools to transform our data into that tabular format we are so farmiliar with. So that is what we will do in the next section--transform our data into a nice table to play with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Converting Dictionary into Pandas dataframe\n",
    "**Note:** This part is not provided in the original scikit learn tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['From: sd345@city.ac.uk (Michael Collier)\\nSubject: Converting images to HP LaserJet III?\\nNntp-Posting-Host: hampton\\nOrganization: The City University\\nLines: 14\\n\\nDoes anyone know of a good way (standard PC application/PD utility) to\\nconvert tif/img/tga files into LaserJet III format.  We would also like to\\ndo the same, converting to HPGL (HP plotter) files.\\n\\nPlease email any response.\\n\\nIs this the correct group?\\n\\nThanks in advance.  Michael.\\n-- \\nMichael Collier (Programmer)                 The Computer Unit,\\nEmail: M.P.Collier@uk.ac.city                The City University,\\nTel: 071 477-8000 x3769                      London,\\nFax: 071 477-8565                            EC1V 0HB.\\n',\n",
       " \"From: ani@ms.uky.edu (Aniruddha B. Deglurkar)\\nSubject: help: Splitting a trimming region along a mesh \\nOrganization: University Of Kentucky, Dept. of Math Sciences\\nLines: 28\\n\\n\\n\\n\\tHi,\\n\\n\\tI have a problem, I hope some of the 'gurus' can help me solve.\\n\\n\\tBackground of the problem:\\n\\tI have a rectangular mesh in the uv domain, i.e  the mesh is a \\n\\tmapping of a 3d Bezier patch into 2d. The area in this domain\\n\\twhich is inside a trimming loop had to be rendered. The trimming\\n\\tloop is a set of 2d Bezier curve segments.\\n\\tFor the sake of notation: the mesh is made up of cells.\\n\\n\\tMy problem is this :\\n\\tThe trimming area has to be split up into individual smaller\\n\\tcells bounded by the trimming curve segments. If a cell\\n\\tis wholly inside the area...then it is output as a whole ,\\n\\telse it is trivially rejected. \\n\\n\\tDoes any body know how thiss can be done, or is there any algo. \\n\\tsomewhere for doing this.\\n\\n\\tAny help would be appreciated.\\n\\n\\tThanks, \\n\\tAni.\\n-- \\nTo get irritated is human, to stay cool, divine.\\n\"]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.data[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 3, ..., 2, 2, 2])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = pd.DataFrame.from_records(dmh.format_rows(twenty_train), columns= ['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2257"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add category to the dataframe\n",
    "X['category'] = twenty_train.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add category label also\n",
    "X['category_name'] = X.category.apply(lambda t: dmh.format_labels(t, twenty_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can print screen and see what our table looks like. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier) Subje...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart) Subject...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: stanly@grok11.columbiasc.ncr.com (stanly...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>From: jodfishe@silver.ucs.indiana.edu (joseph ...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>From: aldridge@netcom.com (Jacquelin Aldridge)...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>From: geb@cs.pitt.edu (Gordon Banks) Subject: ...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>From: libman@hsc.usc.edu (Marlena Libman) Subj...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  category  \\\n",
       "0  From: sd345@city.ac.uk (Michael Collier) Subje...         1   \n",
       "1  From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...         1   \n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...         3   \n",
       "3  From: s0612596@let.rug.nl (M.M. Zwart) Subject...         3   \n",
       "4  From: stanly@grok11.columbiasc.ncr.com (stanly...         3   \n",
       "5  From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B...         3   \n",
       "6  From: jodfishe@silver.ucs.indiana.edu (joseph ...         3   \n",
       "7  From: aldridge@netcom.com (Jacquelin Aldridge)...         2   \n",
       "8  From: geb@cs.pitt.edu (Gordon Banks) Subject: ...         2   \n",
       "9  From: libman@hsc.usc.edu (Marlena Libman) Subj...         2   \n",
       "\n",
       "            category_name  \n",
       "0           comp.graphics  \n",
       "1           comp.graphics  \n",
       "2  soc.religion.christian  \n",
       "3  soc.religion.christian  \n",
       "4  soc.religion.christian  \n",
       "5  soc.religion.christian  \n",
       "6  soc.religion.christian  \n",
       "7                 sci.med  \n",
       "8                 sci.med  \n",
       "9                 sci.med  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! Isn't it? With this format of the data we can conduct many operations, easily and efficiently, since Pandas dataframes provide us with a wide range of built-in features. These features are operations which can directly and quickly be applied to the dataset. These operations may include standard operations like removing records with missing values and aggregating new fields to the current state of the table (hereinafter referred to as a dataframe), which is desirable in almost every data mining project. If you are not covinced head on over to Kaggle -- one of the largest data science community on the web -- and see for yourself some of the top kernels and what they make use of. First thing you will notice is the use of Pandas to transform and explore data. Take a look at this [kernel](https://www.kaggle.com/arthurtok/introduction-to-ensembling-stacking-in-python) for example.\n",
    "\n",
    "I want to also add here that in this notebook we are not following conventions, but we are very interested in using tools and techniques that make our lives easier. Go Pandas!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Familiarizing yourself with the Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin to show you the magnificence of Pandas dataframes, let us look at how to run a simple query on our dataset. We want to query for the first 10 rows (news articles), and we only want to keep the `text` and `category_name` attributes or fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier) Subje...</td>\n",
       "      <td>comp.graphics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...</td>\n",
       "      <td>comp.graphics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart) Subject...</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: stanly@grok11.columbiasc.ncr.com (stanly...</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B...</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>From: jodfishe@silver.ucs.indiana.edu (joseph ...</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>From: aldridge@netcom.com (Jacquelin Aldridge)...</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>From: geb@cs.pitt.edu (Gordon Banks) Subject: ...</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>From: libman@hsc.usc.edu (Marlena Libman) Subj...</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text           category_name\n",
       "0  From: sd345@city.ac.uk (Michael Collier) Subje...           comp.graphics\n",
       "1  From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...           comp.graphics\n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...  soc.religion.christian\n",
       "3  From: s0612596@let.rug.nl (M.M. Zwart) Subject...  soc.religion.christian\n",
       "4  From: stanly@grok11.columbiasc.ncr.com (stanly...  soc.religion.christian\n",
       "5  From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B...  soc.religion.christian\n",
       "6  From: jodfishe@silver.ucs.indiana.edu (joseph ...  soc.religion.christian\n",
       "7  From: aldridge@netcom.com (Jacquelin Aldridge)...                 sci.med\n",
       "8  From: geb@cs.pitt.edu (Gordon Banks) Subject: ...                 sci.med\n",
       "9  From: libman@hsc.usc.edu (Marlena Libman) Subj...                 sci.med"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:10][[\"text\", \"category_name\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at a few more interesting queries to familiarize ourselves with the efficiency and conveniency of Pandas dataframes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We want the last 10 records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2246</th>\n",
       "      <td>From: lmvec@westminster.ac.uk (William Hargrea...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2247</th>\n",
       "      <td>From: daniels@math.ufl.edu (TV's Big Dealer) S...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2248</th>\n",
       "      <td>From: \"danny hawrysio\" &lt;danny.hawrysio@canrem....</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2249</th>\n",
       "      <td>From: shellgate!llo@uu4.psi.com (Larry L. Over...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2250</th>\n",
       "      <td>From: ingles@engin.umich.edu (Ray Ingles) Subj...</td>\n",
       "      <td>0</td>\n",
       "      <td>alt.atheism</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2251</th>\n",
       "      <td>From: Mark-Tarbell@suite.com Subject: Amniocen...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2252</th>\n",
       "      <td>From: roos@Operoni.Helsinki.FI (Christophe Roo...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2253</th>\n",
       "      <td>From: mhollowa@ic.sunysb.edu (Michael Holloway...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2254</th>\n",
       "      <td>From: sasghm@theseus.unx.sas.com (Gary Merrill...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2255</th>\n",
       "      <td>From: Dan Wallach &lt;dwallach@cs.berkeley.edu&gt; S...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   text  category  \\\n",
       "2246  From: lmvec@westminster.ac.uk (William Hargrea...         3   \n",
       "2247  From: daniels@math.ufl.edu (TV's Big Dealer) S...         3   \n",
       "2248  From: \"danny hawrysio\" <danny.hawrysio@canrem....         1   \n",
       "2249  From: shellgate!llo@uu4.psi.com (Larry L. Over...         3   \n",
       "2250  From: ingles@engin.umich.edu (Ray Ingles) Subj...         0   \n",
       "2251  From: Mark-Tarbell@suite.com Subject: Amniocen...         2   \n",
       "2252  From: roos@Operoni.Helsinki.FI (Christophe Roo...         2   \n",
       "2253  From: mhollowa@ic.sunysb.edu (Michael Holloway...         2   \n",
       "2254  From: sasghm@theseus.unx.sas.com (Gary Merrill...         2   \n",
       "2255  From: Dan Wallach <dwallach@cs.berkeley.edu> S...         2   \n",
       "\n",
       "               category_name  \n",
       "2246  soc.religion.christian  \n",
       "2247  soc.religion.christian  \n",
       "2248           comp.graphics  \n",
       "2249  soc.religion.christian  \n",
       "2250             alt.atheism  \n",
       "2251                 sci.med  \n",
       "2252                 sci.med  \n",
       "2253                 sci.med  \n",
       "2254                 sci.med  \n",
       "2255                 sci.med  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[-11:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ready for some sourcery? Brace yourselves! Let us see if we can query every 10th record in our dataframe. In addition, our query must only contain the first 10 records. For this we will use the build-in function ``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier) Subje...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>From: anasaz!karl@anasazi.com (Karl Dussik) Su...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>From: dotsonm@dmapub.dma.org (Mark Dotson) Sub...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>From: vgwlu@dunsell.calgary.chevron.com (greg ...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>From: david-s@hsr.no (David A. Sjoen) Subject:...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>From: ab@nova.cc.purdue.edu (Allen B) Subject:...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>From: Nanci Ann Miller &lt;nm0w+@andrew.cmu.edu&gt; ...</td>\n",
       "      <td>0</td>\n",
       "      <td>alt.atheism</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>From: weaver@chdasic.sps.mot.com (Dave Weaver)...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>From: annick@cortex.physiol.su.oz.au (Annick A...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>Subject: Vonnegut/atheism From: dmn@kepler.unh...</td>\n",
       "      <td>0</td>\n",
       "      <td>alt.atheism</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 text  category  \\\n",
       "0   From: sd345@city.ac.uk (Michael Collier) Subje...         1   \n",
       "10  From: anasaz!karl@anasazi.com (Karl Dussik) Su...         3   \n",
       "20  From: dotsonm@dmapub.dma.org (Mark Dotson) Sub...         3   \n",
       "30  From: vgwlu@dunsell.calgary.chevron.com (greg ...         2   \n",
       "40  From: david-s@hsr.no (David A. Sjoen) Subject:...         3   \n",
       "50  From: ab@nova.cc.purdue.edu (Allen B) Subject:...         1   \n",
       "60  From: Nanci Ann Miller <nm0w+@andrew.cmu.edu> ...         0   \n",
       "70  From: weaver@chdasic.sps.mot.com (Dave Weaver)...         3   \n",
       "80  From: annick@cortex.physiol.su.oz.au (Annick A...         2   \n",
       "90  Subject: Vonnegut/atheism From: dmn@kepler.unh...         0   \n",
       "\n",
       "             category_name  \n",
       "0            comp.graphics  \n",
       "10  soc.religion.christian  \n",
       "20  soc.religion.christian  \n",
       "30                 sci.med  \n",
       "40  soc.religion.christian  \n",
       "50           comp.graphics  \n",
       "60             alt.atheism  \n",
       "70  soc.religion.christian  \n",
       "80                 sci.med  \n",
       "90             alt.atheism  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.iloc[::10, :][0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:** Do more querying here using pandas dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Data Mining using Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's stop playing around and let's do some serious work. Let's learn to program some of the ideas and concepts related to the data mining course. This is the only way we can be convinced of the true power of Pandas dataframes for data mining tasks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Missing Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let us consider that our dataset has some *missing values* and we want to remove those values. In its current state our dataset has no missing values, but for practice sake we will add some records with missing values and then write to code to deal with these object that contain missing value. You will see for yourself how easy it is to deatl with missing values once you have your data transformed into a Pandas dataframe.\n",
    "\n",
    "First let us do a quick review of what we learned in the Data Mining course. Specifically, how to deal with missing values and how they come about. \n",
    "\n",
    "The most common of cases for having missing values in datasets has to do with how the data was initially collected. A good example of this is when a patients comes into the ER room, the data is collected as quickly as possible and depending on the conditions of the patients, the personal data being collected it either incomplete or partially complete. In the former and latter cases, we are presented with a case of \"missing values\". Knowing that patients data is particularly critical and can be used by the health authorities to conduct some interesting analysis, we as the data miners are left with the tough task of deciding what to do with these missing and incomplete records. We need to deal with these records because they are definitely going to affect our analysis or algorithms. So what do we do? There are several ways to handle missing values, and some of the more effective ways are presented below (Note: You can reference the slides - Session 1 Handout for the additional information).\n",
    "\n",
    "- **Eliminate Data Objects** - Here we completely discard records once they contain some missing values. This is the easiest approach and the one we will be using in this notebook. The immediate drawback of going with this approach is that you loose information. Now imagine that half of the records have at least one or more missing values. Here you are presented with the tough decision of quantity vs quality. In any event, this decision must be made carefully, hence the reason for emphasizing it here in this notebook. \n",
    "\n",
    "- **Estimate Missing Values** - Here we try to estimate the missing values based on some criteria. Although this approach may be proven to be effective, it is not always the case, especially when we are dealing with sensitive data, like **Gender** or **Names**. For fields like **Address**, there could be ways to obtain these missing addresses using some data aggregation technique or obtain the information directly from other databases or public data sources.\n",
    "\n",
    "- **Ignore the missing value during analysis** - Here we basically ignore the missing values and proceed with our analysis. Although this is the most naive way to handle missing values it may proof effective, especially when the missing values includes information that is not important to the analysis being conducted. But think about it for a while. Would you be ignore missing values, especially when in this day and age it is difficult to obtain high quality databases. Again, there are some tradeoffs, which will talk about later in the notebook.\n",
    "\n",
    "- **Replace with all possible values** - As an efficient and responsible data miner, we sometimes just need to put in the hard hours of work and find ways to makes up for these missing values. This last option is a very wise option for cases where data is scarce (which is almost always) or when dealing with sensitive data. Imagine that our dataset has an **Age** field, which contains many missing values. Since **Age** is a continuous variable, it means that we can build a separate model for calculating the age for the incomplete records based on some rule-based appraoch or probabilistic approach.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned earlier, we are going to go with the first option but you may be asked to compute missing values, using a different approach, as an exercise. So let's get to it!\n",
    "\n",
    "First we want to add the dummy records with missing values since the dataset we have is perfectly composed and cleaned that it contains no missing values. First let us check for ourselves that indeed the dataset doesn't contain any missing values. We can do that easily by using the following built-in function provided by Pandas.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2227</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2228</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2229</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2230</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2231</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2232</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2233</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2234</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2235</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2236</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2237</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2238</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2239</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2240</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2241</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2242</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2243</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2244</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2245</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2246</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2247</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2248</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2249</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2250</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2251</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2252</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2253</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2254</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2255</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2256</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2257 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       text  category  category_name\n",
       "0     False     False          False\n",
       "1     False     False          False\n",
       "2     False     False          False\n",
       "3     False     False          False\n",
       "4     False     False          False\n",
       "5     False     False          False\n",
       "6     False     False          False\n",
       "7     False     False          False\n",
       "8     False     False          False\n",
       "9     False     False          False\n",
       "10    False     False          False\n",
       "11    False     False          False\n",
       "12    False     False          False\n",
       "13    False     False          False\n",
       "14    False     False          False\n",
       "15    False     False          False\n",
       "16    False     False          False\n",
       "17    False     False          False\n",
       "18    False     False          False\n",
       "19    False     False          False\n",
       "20    False     False          False\n",
       "21    False     False          False\n",
       "22    False     False          False\n",
       "23    False     False          False\n",
       "24    False     False          False\n",
       "25    False     False          False\n",
       "26    False     False          False\n",
       "27    False     False          False\n",
       "28    False     False          False\n",
       "29    False     False          False\n",
       "...     ...       ...            ...\n",
       "2227  False     False          False\n",
       "2228  False     False          False\n",
       "2229  False     False          False\n",
       "2230  False     False          False\n",
       "2231  False     False          False\n",
       "2232  False     False          False\n",
       "2233  False     False          False\n",
       "2234  False     False          False\n",
       "2235  False     False          False\n",
       "2236  False     False          False\n",
       "2237  False     False          False\n",
       "2238  False     False          False\n",
       "2239  False     False          False\n",
       "2240  False     False          False\n",
       "2241  False     False          False\n",
       "2242  False     False          False\n",
       "2243  False     False          False\n",
       "2244  False     False          False\n",
       "2245  False     False          False\n",
       "2246  False     False          False\n",
       "2247  False     False          False\n",
       "2248  False     False          False\n",
       "2249  False     False          False\n",
       "2250  False     False          False\n",
       "2251  False     False          False\n",
       "2252  False     False          False\n",
       "2253  False     False          False\n",
       "2254  False     False          False\n",
       "2255  False     False          False\n",
       "2256  False     False          False\n",
       "\n",
       "[2257 rows x 3 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `isnull` function looks through the entire dataset for null values and return `True` wherever it finds any missing field or record. As you will see above, and as we anticipated, our dataset looks clean and all values are present, since `isnull` return **False** for all fields and records. But let us start to get our hands dirty and build a nice little function to check each of the records, column by column, and return a nice little message telling us the amount of missing records collected. This excerice will also encourage us to explore the possibilities of pandas dataframes. In reality, the build function are good enough, but as you saw above when the entire table was printed, it is impossible really to tell if there are missing records just by looking at the records manually, especially in cases where the dataset is huge. Let's get to it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "text             (The amoung of missing records is: , 0)\n",
       "category         (The amoung of missing records is: , 0)\n",
       "category_name    (The amoung of missing records is: , 0)\n",
       "dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.isnull().apply(lambda x: dmh.check_missing_values(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, a lot happened there in that one line of code, so let's break it down. First, with the `isnull` we tranformed our table into the **True/False** table you see above, where **True** in this case means that the data is missing and **False** means that the data is present. We then take the transformed table and apply a function to each that essentially counts to see if there are missing values in each records and how much missing values are there. In other words the `check_missing_values` function looks through each field in the dataset and counts how many missing values there are. \n",
    "\n",
    "There are many other clever ways to check for missing data, and that is what makes Pandas so beautiful to work with. You get the control you need as a data scientist or just a person working in data mining projects. Indeed, Pandas makes your life easy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:** Insert excercise here on how to calculate the missing values for every record instead of every column. Hint `axis` parameter. Check the documentation for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have our function to check for missing records, now let us do some bad deed and insert some dummy data into the dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy_series = pd.Series([\"dummy_record\", 1], index=[\"text\", \"category\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "text        dummy_record\n",
       "category               1\n",
       "dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_with_series = X.append(dummy_series, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2258"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if the records was commited into result\n",
    "len(result_with_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we that we have added the record (specifically a `Series` or what we would usually now as lists) with some missing values. Let try our function and see if it can detect that there is a missing value on the resulting dataframe called ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "text             (The amoung of missing records is: , 0)\n",
       "category         (The amoung of missing records is: , 0)\n",
       "category_name    (The amoung of missing records is: , 1)\n",
       "dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_with_series.isnull().apply(lambda x: dmh.check_missing_values(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed there is a missing value in this new dataframe. Specificall, the missing value comes from the `category_name` attribute. As I mentioned before, there are many ways to conduct specific operations on the dataframes. In this case let us use a simple dictionary and try to insert it into our original dataframe `X`. Notice also that we are note changing the `X` dataframe as results are directly applied to the assignment variable provided. But in the event that we just want to keep things simple, we can just directly apply the changes to `X` and assign it to itself as we will do below. This exercise will challenge to us for the need to remove this dummy records later on, which means that we need to learn more about Pandas dataframes. This is getting intense! But just relax, everything will be fine!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy_dict = [{'text': 'dummy_record',\n",
    "               'category': 1\n",
    "              }]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.append(dummy_dict,ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2258"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "text             (The amoung of missing records is: , 0)\n",
       "category         (The amoung of missing records is: , 0)\n",
       "category_name    (The amoung of missing records is: , 1)\n",
       "dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.isnull().apply(lambda x: dmh.check_missing_values(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now that we can see that our data has missing values, we want to remove the records with missing values. The code to drop the record with missing that we just added, is the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and now let us test to see if we gotten rid of the records with missing values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "text             (The amoung of missing records is: , 0)\n",
       "category         (The amoung of missing records is: , 0)\n",
       "category_name    (The amoung of missing records is: , 0)\n",
       "dtype: object"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.isnull().apply(lambda x: dmh.check_missing_values(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we are back with our original dataset, clean and tidy as we want it. That's enough on how to deal with missing values, let us now move unto something more fun. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But just in case you want to learn more about how to deal with missing data, go to reference the official [Pandas documentation](http://pandas.pydata.org/pandas-docs/stable/missing_data.html#missing-data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Dealing with Duplicate Data\n",
    "Dealing with duplicate data is just as painful as dealing with missing data. The only difference between the two is that now there is more of the same. The worst case is that you have duplicate data that has missing values. But let us not get carried away. Let us keep it to the basics. As we have learned in our Data Mining course, duplicated data can occur because of many reasons. The majority of the times it has to do with how we store data or how we collect and merge data. For instance, we may collected and stored a tweet, and retweet of that same tweet as two different records. The result is case of data duplication; the only difference being that one is the original tweet and the other the retweeted one. Here you will learn that dealing with duplicate data is not as challenging as missing values. But this also all depends on what you consider as duplicates; this all depends on your criteria for what is considered as a duplicate and also what type of data you are dealing with. For text it may not be so trivial as it is for numerical values. Anyhow let us look at some code on how to deal with duplicate records in our `X` dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let us check how many duplicates we have in our current dataset. Here is the line of code that checks for duplicates; it is very similar to the `isnull` operation that we used to check for missing values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       False\n",
       "1       False\n",
       "2       False\n",
       "3       False\n",
       "4       False\n",
       "5       False\n",
       "6       False\n",
       "7       False\n",
       "8       False\n",
       "9       False\n",
       "10      False\n",
       "11      False\n",
       "12      False\n",
       "13      False\n",
       "14      False\n",
       "15      False\n",
       "16      False\n",
       "17      False\n",
       "18      False\n",
       "19      False\n",
       "20      False\n",
       "21      False\n",
       "22      False\n",
       "23      False\n",
       "24      False\n",
       "25      False\n",
       "26      False\n",
       "27      False\n",
       "28      False\n",
       "29      False\n",
       "        ...  \n",
       "2227    False\n",
       "2228    False\n",
       "2229    False\n",
       "2230    False\n",
       "2231    False\n",
       "2232    False\n",
       "2233    False\n",
       "2234    False\n",
       "2235    False\n",
       "2236    False\n",
       "2237    False\n",
       "2238    False\n",
       "2239    False\n",
       "2240    False\n",
       "2241    False\n",
       "2242    False\n",
       "2243    False\n",
       "2244    False\n",
       "2245    False\n",
       "2246    False\n",
       "2247    False\n",
       "2248    False\n",
       "2249    False\n",
       "2250    False\n",
       "2251    False\n",
       "2252    False\n",
       "2253    False\n",
       "2254    False\n",
       "2255    False\n",
       "2256    False\n",
       "Length: 2257, dtype: bool"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.duplicated()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also check the sum of duplicated by simply doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(X.duplicated())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on that output, you may be asking why did the `duplicated` operation only returned one single column that indicates whether there is a duplicate record or not. So yes, all the operation does is to check per records instead of per column. That is why the operation only returns on value instead of three values for each columns, for each record as before. It appears that we don't have any duplicates since none of our records resulted in `True`. If we want to check for duplicates as we did above for some particular columns instead of all columns we do something as below. As you may have noticed, in this case where we select some columns instead of checking by all columns, we are kind of lowering the criteria of what is considered as a duplicate record. So let us only check for duplicateS by only the text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(X.duplicated('text'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us create some dummy duplicate data and append it to the main dataframe `X`. And subsequenlty, let us try to get rid of the duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy_duplicate_dict = [{\n",
    "                             'text': 'dummy record',\n",
    "                             'category': 1, \n",
    "                             'category_name': \"dummy category\"\n",
    "                        },\n",
    "                        {\n",
    "                             'text': 'dummy record',\n",
    "                             'category': 1, \n",
    "                             'category_name': \"dummy category\"\n",
    "                        }]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.append(dummy_duplicate_dict, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2259"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(X.duplicated('text'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have added the dummy duplicate to `X`. Now we are faced with the decision as to what to do with the duplicate after we have found it. In our case we want to get rid of all the duplicated records without preserving a copy. So we simple do that with the following line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.drop_duplicates(keep=False, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2257"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out the Pandas [documentation](http://pandas.pydata.org/pandas-docs/stable/indexing.html?highlight=duplicate#duplicate-data) for more information on dealing with duplicate data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.  Data Preprocessing\n",
    "In the Data Mining course we learned about the many ways of performing data preprocessing. In reality, the list is quiet general as the specifics of what data preprocessing involves is too much to cover in one course. This is especially true when you are dealing with unstructured data, as we are dealing with in this particular notebook. But let us look at some examples for each data preprocessing technique that we learned in the class. We will cover each item one by one, and provide example code for each category. You will learn how to peform each of the operations, using Pandas, that cover the essentials to Preprocessing in Data Mining. We are not going to follow any strict order, but the items we will cover in the preprocessing section of this notebook are as follows:\n",
    "\n",
    "- Aggregation\n",
    "- Sampling\n",
    "- Dimensionality Reduction\n",
    "- Feature Subset Selection\n",
    "- Feature Creation\n",
    "- Discretization and Binarization\n",
    "- Attribute Transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Sampling\n",
    "The first concept that we are going to cover from the above list is sampling. Sampling refers to the technique used for selecting data. The functionalities that we use to  selected data through queries provided by Pandas are actually methods for sampling. The reasons for sampling are sometimes due to the size of data; we want a smaller subset of the data that still best represents the complete dataset. We don't have a problem of size in our dataset since our dataset is just a couple thousand records long. But if we pay attention to how much content is included in the `text` field of each of those records, you will realize that sampling may not be a bad idea. In fact, we have already done some sampling by just reducing the records we are using here in this notebook; we only used four categories from the all the 20 categories available. Let us get an idea of what sampling involves in terms of using pandas code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_sample = X.sample(n=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do something cool here while we are in sampling! Let us look at the distribution of categories in both the sample and original dataset. Let us visually analyze the disparity between the two datasets. To generate some visualizations, we are going to use `Plotly` python library. Plotly is an excellent visualization tool for data scientist to generate interactive charts and to create portable dashboards. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_category_counts = ta.get_tokens_and_frequency(list(X.category_name))\n",
    "X_sample_category_counts = ta.get_tokens_and_frequency(list(X_sample.category_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~omarsar/190.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py.iplot(ta.plot_word_frequency(X_category_counts, \"Category distribution\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~omarsar/192.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py.iplot(ta.plot_word_frequency(X_sample_category_counts, \"Category distribution\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** We can also do a side by side comparison of the distribution between the two datasets, but maybe you can try that as an excerise. Look at the [Plotly documents](https://plot.ly/python/bar-charts/) for tons of examples and ways to visualizing bar charts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I must also make mention that there are other libraries for visualizing, but we are going to focus on that in the later sections. For now, we just show you what are the ways to sample and to verify the distributions of your samplings. I can't resits it, but actually, there is an easier to generate the chart we generated above using a library called `matplotlib`. With matplotlib, things are faster and compatability-wise it may just be the best visualization library for visualizing content extracted from dataframes. Let's quickly take a loot at the magic of `matplotlib` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fcc28984908>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFgBJREFUeJzt3Xu0ZnV93/H3h0HBKzIynTVyyVA76oIkknQkWq2LiBGM\nSYbVJjjW6mhpJjSImtRUsNYuuzpZJKZNYgym441JvNBRE5mopU7GC5HIZVBgGBCZcilDgZl4RyMK\nfPvH/h19OJwz5zlzzuEwP9+vtc56fvu3f3vv3759nv3s53JSVUiS+nXQYndAkrSwDHpJ6pxBL0md\nM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5w5e7A4AHHHEEbVy5crF7oYkHVCuuuqqv6+qZTO1\ne0QE/cqVK9m+fftid0OSDihJbhunnbduJKlzBr0kdc6gl6TOGfSS1DmDXpI6N1bQJ3lSko8k+XKS\nG5I8J8nSJFuT3NQeDx9pf26SXUluTHLKwnVfkjSTca/o/xi4uKqeATwTuAE4B9hWVauAbW2YJMcB\na4HjgVOB85Msme+OS5LGM2PQJzkMeD7wHoCq+n5VfQNYA2xqzTYBp7XyGuDCqrq3qm4BdgEnznfH\nJUnjGeeK/lhgL/C+JF9K8u4kjwOWV9Wdrc1dwPJWPhK4fWT63a3uQZKsT7I9yfa9e/fu/xpIkvZp\nnG/GHgz8LHB2VV2e5I9pt2kmVFUlmdV/Ga+qjcBGgNWrV8/7fyhfec4n5nuWC+LW816y2F2Q1Llx\ngn43sLuqLm/DH2EI+ruTrKiqO5OsAPa08XcAR49Mf1Sr0wHMJ07pwDXjrZuqugu4PcnTW9XJwPXA\nFmBdq1sHXNTKW4C1SQ5JciywCrhiXnstSRrbuD9qdjbwgSSPBm4GXs3wJLE5yRnAbcDpAFW1M8lm\nhieD+4Czqur+ee+5JGksYwV9VV0NrJ5i1MnTtN8AbJhDvyRJ8+QR8TPF0o8b3/PQw8mfQJCkzhn0\nktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9J\nnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHVurKBPcmuSHUmu\nTrK91S1NsjXJTe3x8JH25ybZleTGJKcsVOclSTObzRX9z1fVCVW1ug2fA2yrqlXAtjZMkuOAtcDx\nwKnA+UmWzGOfJUmzMJdbN2uATa28CThtpP7Cqrq3qm4BdgEnzmE5kqQ5GDfoC/ibJFclWd/qllfV\nna18F7C8lY8Ebh+ZdnerkyQtgoPHbPe8qrojyT8Ctib58ujIqqokNZsFtyeM9QDHHHPMbCaVJM3C\nWFf0VXVHe9wD/BXDrZi7k6wAaI97WvM7gKNHJj+q1U2e58aqWl1Vq5ctW7b/ayBJ2qcZgz7J45I8\nYaIMvAi4DtgCrGvN1gEXtfIWYG2SQ5IcC6wCrpjvjkuSxjPOrZvlwF8lmWj/waq6OMmVwOYkZwC3\nAacDVNXOJJuB64H7gLOq6v4F6b0kaUYzBn1V3Qw8c4r6rwInTzPNBmDDnHsnSZozvxkrSZ0z6CWp\nc+N+vFKSHpFWnvOJxe7CWG497yWLtmyv6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxB\nL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS\n1DmDXpI6Z9BLUucMeknq3NhBn2RJki8l+XgbXppka5Kb2uPhI23PTbIryY1JTlmIjkuSxjObK/rX\nATeMDJ8DbKuqVcC2NkyS44C1wPHAqcD5SZbMT3clSbM1VtAnOQp4CfDukeo1wKZW3gScNlJ/YVXd\nW1W3ALuAE+enu5Kk2Rr3iv6PgP8APDBSt7yq7mzlu4DlrXwkcPtIu92tTpK0CGYM+iS/BOypqqum\na1NVBdRsFpxkfZLtSbbv3bt3NpNKkmZhnCv65wK/kuRW4ELgBUneD9ydZAVAe9zT2t8BHD0y/VGt\n7kGqamNVra6q1cuWLZvDKkiS9mXGoK+qc6vqqKpayfAm66er6l8DW4B1rdk64KJW3gKsTXJIkmOB\nVcAV895zSdJYDp7DtOcBm5OcAdwGnA5QVTuTbAauB+4Dzqqq++fcU0nSfplV0FfVZ4HPtvJXgZOn\nabcB2DDHvkmS5oHfjJWkzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLU\nOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z\n6CWpcwa9JHXOoJekzs0Y9EkOTXJFkmuS7Ezy1la/NMnWJDe1x8NHpjk3ya4kNyY5ZSFXQJK0b+Nc\n0d8LvKCqngmcAJya5NnAOcC2qloFbGvDJDkOWAscD5wKnJ9kyUJ0XpI0sxmDvgb3tMFHtb8C1gCb\nWv0m4LRWXgNcWFX3VtUtwC7gxHnttSRpbGPdo0+yJMnVwB5ga1VdDiyvqjtbk7uA5a18JHD7yOS7\nW50kaRGMFfRVdX9VnQAcBZyY5CcnjS+Gq/yxJVmfZHuS7Xv37p3NpJKkWZjVp26q6hvAZxjuvd+d\nZAVAe9zTmt0BHD0y2VGtbvK8NlbV6qpavWzZsv3puyRpDON86mZZkie18mOAXwC+DGwB1rVm64CL\nWnkLsDbJIUmOBVYBV8x3xyVJ4zl4jDYrgE3tkzMHAZur6uNJvgBsTnIGcBtwOkBV7UyyGbgeuA84\nq6ruX5juS5JmMmPQV9W1wM9MUf9V4ORpptkAbJhz7yRJc+Y3YyWpcwa9JHXOoJekzhn0ktQ5g16S\nOmfQS1LnDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalz\nBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHVuxqBPcnSSzyS5PsnOJK9r\n9UuTbE1yU3s8fGSac5PsSnJjklMWcgUkSfs2zhX9fcC/r6rjgGcDZyU5DjgH2FZVq4BtbZg2bi1w\nPHAqcH6SJQvReUnSzGYM+qq6s6q+2MrfBm4AjgTWAJtas03Aaa28Briwqu6tqluAXcCJ891xSdJ4\nZnWPPslK4GeAy4HlVXVnG3UXsLyVjwRuH5lsd6uTJC2CsYM+yeOBjwKvr6pvjY6rqgJqNgtOsj7J\n9iTb9+7dO5tJJUmzMFbQJ3kUQ8h/oKr+slXfnWRFG78C2NPq7wCOHpn8qFb3IFW1sapWV9XqZcuW\n7W//JUkzGOdTNwHeA9xQVf99ZNQWYF0rrwMuGqlfm+SQJMcCq4Ar5q/LkqTZOHiMNs8FXgHsSHJ1\nq3sTcB6wOckZwG3A6QBVtTPJZuB6hk/snFVV9897zyVJY5kx6Kvq80CmGX3yNNNsADbMoV+SpHni\nN2MlqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmD\nXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdM+gl\nqXMGvSR1bsagT/LeJHuSXDdStzTJ1iQ3tcfDR8adm2RXkhuTnLJQHZckjWecK/oLgFMn1Z0DbKuq\nVcC2NkyS44C1wPFtmvOTLJm33kqSZm3GoK+qS4CvTapeA2xq5U3AaSP1F1bVvVV1C7ALOHGe+ipJ\n2g/7e49+eVXd2cp3Actb+Ujg9pF2u1vdQyRZn2R7ku179+7dz25IkmYy5zdjq6qA2o/pNlbV6qpa\nvWzZsrl2Q5I0jf0N+ruTrABoj3ta/R3A0SPtjmp1kqRFsr9BvwVY18rrgItG6tcmOSTJscAq4Iq5\ndVGSNBcHz9QgyYeAk4AjkuwG/jNwHrA5yRnAbcDpAFW1M8lm4HrgPuCsqrp/gfouSRrDjEFfVS+b\nZtTJ07TfAGyYS6ckSfPHb8ZKUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1Ln\nDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6g\nl6TOGfSS1DmDXpI6Z9BLUucWLOiTnJrkxiS7kpyzUMuRJO3bggR9kiXAnwIvBo4DXpbkuIVYliRp\n3xbqiv5EYFdV3VxV3wcuBNYs0LIkSfuwUEF/JHD7yPDuVidJepgdvFgLTrIeWN8G70ly42L1ZRaO\nAP5+PmeY35vPuR1w3J7zy+05fw6UbfkT4zRaqKC/Azh6ZPioVvdDVbUR2LhAy18QSbZX1erF7kcv\n3J7zy+05f3rblgt16+ZKYFWSY5M8GlgLbFmgZUmS9mFBruir6r4krwH+N7AEeG9V7VyIZUmS9m3B\n7tFX1SeBTy7U/BfJAXWr6QDg9pxfbs/509W2TFUtdh8kSQvIn0CQpM51G/RJPptkdSt/MsmTZmj/\nX5K88OHozwztnpLkI/sY/6Qkvzlu+8WSZHWStz8My7k1yRELvZwfB0kuSPKrU9Q/Io+xCRPHwORz\nYxbTvz7JY0eG75nl9L/ySP+ZlwP21k2SMPT/gWnGfxZ4Q1Vtf1g7No1x+pPk4Kq6b4b5rAQ+XlU/\nOa8dPEAluRVYXVXz+pnnA9lM58Y+pruA4dh6xIb6VCaOAeDx7Me5MfkYSnJPVT1+vvu5mBb8ij7J\n45J8Isk1Sa5L8tIkJyf5UpIdSd6b5JDW9llJ/q61vSLJEybNa2X7obQ/B64Djk7yoiRfSPLFJB9O\n8pAdNHrVl+Q/tXl8PsmHkryh1f/wamYf/bs1yVvbsnYkecY06/zGNv6aJOeNjPq1tl5fSfLPW9tX\nJdmS5NPAtraO17Vxx7f2Vye5Nskq4Dzgqa3ubZPar0zyt61/X0zyz1r9Se0VxUeSfDnJB1oYzNf+\nfMh+a8v8+BTTn5Tkc0kuSnJzkvOSvLxNtyPJU1u7ZUk+muTK9vfcVv/kJJ9KsjPJu4H9Wo/5kOSV\nbb9ck+Qv2vb/dKvbluSY1u6CJO9Mcllb55PacXVDC9eJ+d2T5A/bum1LsmyKZS5LsnVi/ZPcluFq\ndqpz451Jtre2bx2Zx61Jfr9t7yuS/JORRTy/7cubR86H0WNsSZI/aPv+2iRnt/rzklzf6v5gIbZ3\nW87HklzV1mn9pNEPOjemmPYh2yPJa4GnAJ9J8pmRthvafr0syfJWN90x+aok72jlX2vb5pokl4yM\n/1jbb7cmeU2S386QMZclWboQ2+pBqmpB/4B/CbxrZPgwhp9HeFob/nPg9cCjgZuBZ7X6JwIHT5rX\nSuAB4Nlt+AjgEuBxbfiNwFta+bMMz9IAt7a2zwKuBg4FngDcxHCVDXAB8Ktt3EP6NzKfs1v5N4F3\nT7G+Lwb+DnhsG1460p//1sq/CPxNK7+K4Scilo6s43Wt/CfAy1v50cBjRsdP0f6xwKGtvArY3son\nAd9k+OLaQcAXgOfN4/58yH5ry/z4FNOfBHwDWAEcwvBFure2ca8D/qiVPzjRR+AY4IZWfvvIPn4J\nUMARC30cT7EexwNfmVg2sBT4a2BdG/43wMdGjq0LGZ6U1gDfAn6q7YurgBNauxrZ328B3jHFct8B\nnNvKp06sP5POjUnH3pJ2/P30yHH8H1v5lRP7qfXzw61fxzH8XtXkY+zfAR+hnZttvZ8M3MiP7hA8\naQG3+8Q6PYbhCe3J/Oj8/mE/Z5h2qu1xxEi7An65lX8fePMMx+SrJvYVsAM4cnQ7tPG7GDJnGcO5\neGYb94e0fFnIv4fjHv0O4BeS/F6Gq9iVwC1V9ZU2fhPwfODpwJ1VdSVAVX2rpr6NcVtVXdbKz2Y4\nIC9NcjWwjn1/Jfi5wEVV9b2q+jbDiTnZ06fp34S/bI9XtXWZ7IXA+6rqu209vjbGtFsntZvwBeBN\nSd4I/ERV/cN0K9Y8CnhXkh0MJ+zoL4ZeUVW7a3g5f/U0fR/H5P15DOPtt1FXVtWdVXUv8H+AT43M\ne6JfLwTe0fbrFuCJGV6tPR94f1vWJ4Cv7+d6zNULgA9Xe7nf9t9zGMIA4C+A5420/+sazuwdwN1V\ntaPti538aJ0fAP5nK79/0vQTnsfwpEFVXcyD13/03AA4PckXgS8xPDGNHg8fGnl8zkj9x6rqgaq6\nHlg+xfJfCPyPiX3c1vubwPeA9yT5F8B3p5huvrw2yTXAZQzfvl81i2n3tT1GfR+YeDU6eq5Od0yO\nuhS4IMmvMzyhTPhMVX27qvYybK+J7Bk95hfMgv/WTVV9JcnPMlzF/lfg03Oc5XdGymEIyZfNcZ6z\ncW97vJ/Zb7/ppv3OFG2pqg8muZzhyvWTSX6D4ep5Or8F3A08k+Gq7HtTLHuq5Y9tnvbnaF8eGBl+\nYKRfBzFcnY6uA9m/O06PBKPrOHn9p9sXs30D7YfHUZJjgTcwvNL6ertFdOg08x4tj/ZtrI1dwxck\nTwROZnhV/BqGJ8J5leQkhrB9TlV9N8P7Xofuc6IfTTvT9hj1g/akDA8+V2Y8JqvqzCQ/x3DOXpXk\nn7ZR4xzzC+bhuEf/FOC7VfV+4G0MVw8rR+4LvgL4HMNLvxVJntWme0KSmTbAZcBzJ+aV4f7x0/bR\n/lLgl5Mc2p6Jf2mKNjdO079xbQVenfYu/lzuvyX5x8DNVfV24CLgp4FvM7wEnMphDFfXD7R+L5mm\n3X6bYn/+HLPfb+P4FHD2yHJPaMVLgH/V6l4MHD4Py9ofn2Z4z+XJrS9LGW7ZrW3jXw787SzneRBD\nUMKwjp+fos2lwOltmS9i+vV/IkPwf7PdY37xpPEvHXn8wiz6uBX4jYl9nGRpO5cOq+FLkr/FcKGx\nEA4Dvt5C/hkMr+hH7evc2Nf22Nd0o6Y7Jhmpe2pVXV5VbwH28uDf/Fo0D8evV/4U8LYkDwA/YLjH\ndxjw4XawXAn8WVV9P8lLgT9J8hjgH4AXJnkiw73wX5w846ram+RVwIfS3jAF3sxw7/QhqurKJFuA\naxmufHcwvIwabfO9JK+e3L99rWCGj02eWVX/tqoubgfA9iTfZ/h28Jtm2kjTOB14RZIfAHcBv1tV\nX0tyaXtz7H8x/IOXCecDH03ySuBipnmlMEdT7c8wab+NTjC6fWaxnNcCf5rkWobj9BLgTOCtDPt7\nJ0Ow/t85rs9+qaqdSTYAn0tyP8PtgLOB9yX5HYaT/NWznO13gBOTvBnYQwvjJGe2Zf4ZP1r/VzAE\n9F0MQfWgWwhVdU2SLwFfZnjP6dJJyzq8bdt7gdm8In438DTg2nZcvgv4KHBRkkMZjoXfnsX8ZuNi\n4MwkNzBckI3epqKqvjp6blTV7yS5uqpOmGF7bAQuTvL/qurn97H86Y7JUW/L8KGJANuAa4CHPCE8\n3A7Yj1furySPr6p72hX3JcD6qvriYvdLyhgf62sXNPe32yXPAd5ZVbMKkviR1B87i/Z79ItoY4Z/\na3gosMmQ1wHmGGBzkoMY3jT89UXujw4AP3ZX9JL046bbn0CQJA0MeknqnEEvSZ0z6CWpcwa9JHXO\noJekzv1/KJN5AzjaZkYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcc2a12fac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X.category_name.value_counts().plot(kind=\"bar\", rot=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fcc68cab4e0>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFT9JREFUeJzt3X20ZXV93/H3R1DwARWc21nIQwbNmCxIE9JeiVabNalG\nUZti0ohYFwFrM9IgahKzBGtN7CpdJD51GR9SUAKJETI+RIhSKiKGgCDMIE8DolMYCmSASTQKGlGG\nb//YvyuHy30494krP9+vte66+/z277f3b//O3p+zzz773JuqQpLUr8esdgckSSvLoJekzhn0ktQ5\ng16SOmfQS1LnDHpJ6pxBL0mdmzfok+yZ5Iok1yTZmuQdrXyfJBck+Xr7vfdIm5OSbEtyU5IXr+QG\nSJLmlvm+MJUkwBOr6t4kjwUuAd4I/Brwjao6JcmJwN5V9ZYkBwNnAYcBTwc+Dzyrqnat5IZIkma2\n+3wVangluLc9fGz7KeAIYEMrPxP4IvCWVn52Vd0H3JJkG0PoXzbbOtasWVPr1q1b1AZI0o+rLVu2\n/H1VTcxXb96gB0iyG7AF+EngA1X15SRrq2pHq3InsLZN7wdcPtL89lY2fZkbgY0ABx54IJs3bx6n\nK5KkJsmt49Qb68PYqtpVVYcC+wOHJfmZafOL4Sx/bFV1alVNVtXkxMS8L0iSpEVa0F03VfWPwEXA\n4cBdSfYFaL/vbtXuAA4YabZ/K5MkrYJx7rqZSPLUNv144JeBrwLnAse0ascA57Tpc4GjkuyR5CBg\nPXDFcndckjSeca7R7wuc2a7TPwbYVFWfSXIZsCnJa4FbgSMBqmprkk3ADcD9wPHecSNJq2fe2ysf\nCZOTk+WHsZK0MEm2VNXkfPX8Zqwkdc6gl6TOGfSS1DmDXpI6N9Y3Yx+N1p342dXuwli2n/Ky1e6C\npM55Ri9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJek\nzhn0ktQ5g16SOmfQS1LnDHpJ6ly3/3hEy8t/5CI9enlGL0mdM+glqXMGvSR1zqCXpM4Z9JLUuXmD\nPskBSS5KckOSrUne2Mr/IMkdSa5uPy8daXNSkm1Jbkry4pXcAEnS3Ma5vfJ+4Her6qokewFbklzQ\n5r23qt41WjnJwcBRwCHA04HPJ3lWVe1azo5LksYzb9BX1Q5gR5u+J8mNwH5zNDkCOLuq7gNuSbIN\nOAy4bBn6K3XB7yXokbSga/RJ1gE/D3y5FZ2Q5NokpyfZu5XtB9w20ux2ZnhhSLIxyeYkm3fu3Lng\njkuSxjN20Cd5EvBJ4E1V9W3gQ8AzgEMZzvjfvZAVV9WpVTVZVZMTExMLaSpJWoCxgj7JYxlC/i+q\n6lMAVXVXVe2qqgeA0xguzwDcARww0nz/ViZJWgXj3HUT4CPAjVX1npHyfUeq/SpwfZs+FzgqyR5J\nDgLWA1csX5clSQsxzl03zwOOBq5LcnUreyvwqiSHAgVsB14HUFVbk2wCbmC4Y+d477iRpNUzzl03\nlwCZYdZ5c7Q5GTh5Cf2SJC0TvxkrSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxB\nL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS\n1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHVu3qBPckCSi5LckGRrkje28n2SXJDk6+333iNt\nTkqyLclNSV68khsgSZrbOGf09wO/W1UHA88Bjk9yMHAicGFVrQcubI9p844CDgEOBz6YZLeV6Lwk\naX7zBn1V7aiqq9r0PcCNwH7AEcCZrdqZwMvb9BHA2VV1X1XdAmwDDlvujkuSxrOga/RJ1gE/D3wZ\nWFtVO9qsO4G1bXo/4LaRZre3sunL2phkc5LNO3fuXGC3JUnjGjvokzwJ+CTwpqr69ui8qiqgFrLi\nqjq1qiaranJiYmIhTSVJC7D7OJWSPJYh5P+iqj7Viu9Ksm9V7UiyL3B3K78DOGCk+f6tTJKW3boT\nP7vaXRjL9lNetmrrHueumwAfAW6sqveMzDoXOKZNHwOcM1J+VJI9khwErAeuWL4uS5IWYpwz+ucB\nRwPXJbm6lb0VOAXYlOS1wK3AkQBVtTXJJuAGhjt2jq+qXcvec0nSWOYN+qq6BMgss18wS5uTgZOX\n0C9J0jLxm7GS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1Ln\nDHpJ6pxBL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6g\nl6TOGfSS1DmDXpI6Z9BLUufmDfokpye5O8n1I2V/kOSOJFe3n5eOzDspybYkNyV58Up1XJI0nnHO\n6M8ADp+h/L1VdWj7OQ8gycHAUcAhrc0Hk+y2XJ2VJC3cvEFfVRcD3xhzeUcAZ1fVfVV1C7ANOGwJ\n/ZMkLdFSrtGfkOTadmln71a2H3DbSJ3bW5kkaZUsNug/BDwDOBTYAbx7oQtIsjHJ5iSbd+7cuchu\nSJLms6igr6q7qmpXVT0AnMaDl2fuAA4Yqbp/K5tpGadW1WRVTU5MTCymG5KkMSwq6JPsO/LwV4Gp\nO3LOBY5KskeSg4D1wBVL66IkaSl2n69CkrOADcCaJLcDvw9sSHIoUMB24HUAVbU1ySbgBuB+4Piq\n2rUyXZckjWPeoK+qV81Q/JE56p8MnLyUTkmSlo/fjJWkzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0md\nM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmD\nXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOjdv0Cc5PcndSa4fKdsnyQVJ\nvt5+7z0y76Qk25LclOTFK9VxSdJ4xjmjPwM4fFrZicCFVbUeuLA9JsnBwFHAIa3NB5Pstmy9lSQt\n2LxBX1UXA9+YVnwEcGabPhN4+Uj52VV1X1XdAmwDDlumvkqSFmGx1+jXVtWONn0nsLZN7wfcNlLv\n9lYmSVolS/4wtqoKqIW2S7IxyeYkm3fu3LnUbkiSZrHYoL8ryb4A7ffdrfwO4ICRevu3soepqlOr\narKqJicmJhbZDUnSfBYb9OcCx7TpY4BzRsqPSrJHkoOA9cAVS+uiJGkpdp+vQpKzgA3AmiS3A78P\nnAJsSvJa4FbgSICq2ppkE3ADcD9wfFXtWqG+S5LGMG/QV9WrZpn1glnqnwycvJROSZKWj9+MlaTO\nGfSS1DmDXpI6Z9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxB\nL0mdM+glqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS\n1DmDXpI6t/tSGifZDtwD7ALur6rJJPsAfwmsA7YDR1bVN5fWTUnSYi3HGf0vVdWhVTXZHp8IXFhV\n64EL22NJ0ipZiUs3RwBntukzgZevwDokSWNaatAX8PkkW5JsbGVrq2pHm74TWLvEdUiSlmBJ1+iB\n51fVHUn+GXBBkq+OzqyqSlIzNWwvDBsBDjzwwCV2Q5I0myWd0VfVHe333cBfAYcBdyXZF6D9vnuW\ntqdW1WRVTU5MTCylG5KkOSw66JM8McleU9PAi4DrgXOBY1q1Y4BzltpJSdLiLeXSzVrgr5JMLedj\nVXV+kiuBTUleC9wKHLn0bkqSFmvRQV9VNwM/N0P5PwAvWEqnJEnLx2/GSlLnDHpJ6pxBL0mdM+gl\nqXMGvSR1zqCXpM4Z9JLUOYNekjpn0EtS5wx6SeqcQS9JnTPoJalzBr0kdc6gl6TOGfSS1DmDXpI6\nZ9BLUucMeknqnEEvSZ0z6CWpcwa9JHXOoJekzhn0ktQ5g16SOmfQS1LnDHpJ6pxBL0mdW7GgT3J4\nkpuSbEty4kqtR5I0txUJ+iS7AR8AXgIcDLwqycErsS5J0txW6oz+MGBbVd1cVd8HzgaOWKF1SZLm\nsFJBvx9w28jj21uZJOkRtvtqrTjJRmBje3hvkptWqy8LsAb4++VcYP5wOZf2qON4Li/Hc/k8Wsby\nJ8aptFJBfwdwwMjj/VvZD1XVqcCpK7T+FZFkc1VNrnY/euF4Li/Hc/n0NpYrdenmSmB9koOSPA44\nCjh3hdYlSZrDipzRV9X9SV4P/B9gN+D0qtq6EuuSJM1txa7RV9V5wHkrtfxV8qi61PQo4HguL8dz\n+XQ1lqmq1e6DJGkF+ScQJKlzBn2TZDLJ+x6B9WxPsmal17OcknwxyWSbPi/JU+ep/9+SvPCR6M88\n9Z6e5BNzzH9qkt8at37vkpyR5NdnKP+RHpepY2r687mA9m9K8oSRx/cusP2/+1H/My9eunmEJdkO\nTFbVst6juxRJwrAvPDDL/C8Cb66qzY9ox2YxTn+S7F5V98+znHXAZ6rqZ5a1g6tsvudzjnZnMIzH\nj2yoz2TqmAKexCKez+nHZJJ7q+pJy93P1dT9GX2SJyb5bJJrklyf5JVJnp3kS63siiR7JdmQ5DMz\ntN+Q5G+SnJPk5iSnJHl1a3ddkme2ehNJPpnkyvbzvFb+tCSfS7I1yYeBLPO2vCDJV1pfTk+yR6v7\nsG2ctqx17Y/O/RlwPXBAkhcluSzJVUk+nuRhO/voO5Ik/7Ut45IkZyV5cyv/4ZnhHP3bnuQdbV3X\nJfnpWbb5LW3+NUlOGZn1irZdX0vyr1vdY5Ocm+QLwIVtG69v8w5p9a9Ocm2S9cApwDNb2Tun1V+X\n5G9b/65K8q9a+YYM7yg+keTvknyz9e3PW5svtOVfmOTAkfH4UJLL2z60oY3FjS1cp7b13iTvbfvK\nhUkmZhiPiSQXTO1PSW7NcDY70/P5oSSbW913THsO/6iN6xVJfnJkFb/Y9pubR57D0XHZLcm72v53\nbZITWvkpSW5oZe+a6blcDkk+nWRL26aN02Y/5Pmcoe3DxiPJG4CnAxcluWik7snteb08ydpWNtsx\nfmyS97fpV7SxuSbJxSPzP92et+1JXp/kd9pxcXmSfVZirB6iqrr+Af49cNrI46cANwPPbo+fzHD3\n0QaGs4Hp7TcA/wjsC+zB8MWvd7R5bwT+Z5v+GPD8Nn0gcGObfh/w9jb9MqCANcu4LbcBz2qP/wx4\nE/C4mbZx2rLWAQ8Az2mP1wAXA09sj98y0u8vMpzxAGxvdZ8NXA3sCewFfJ3hLBvgDODX27yH9W9k\nOSe06d8CPjzD9r4E+BLwhPZ4n5H+vLtNvxT4fJs+luHPbewzso3Xt+k/Bl7dph8HPH50/gz1nwDs\n2abXA5tH9odvAf8G+BrDd0aeD+wD/DVwTKv3H4FPj4zH2Qwv8kcA3wb+OcOJ1hbg0FavRvr4duD9\nM4zJ+4GT2vThrc2a6c/ntPHarY3Zz46M/X9p079B2+9bPz/e+nUww9+rmj4u/xn4BG1/atv9NOAm\nHrxC8NQVPJ6ntunxDC9oT+PBffIhz+ccbWcajzUj9Qr4lTb9R8Db5jnGj516roDrgP1Gx6HN38Zw\nnEy0/ee4Nu+9tGNiJX+6P6NnGPhfTvKH7czvQGBHVV0JUFXfrnne4gNXVtWOqroP+L/A50aWva5N\nvxB4f5KrGb4c9uQMZ8S/CHy0reuzwDeXcVvWAbdU1dfa/DPb+n5qzG28taoub9PPYTi4L23bcAxz\nf736ecA5VfW9qrqHIeSm+6lZ+jflU+33Fh4cx1EvBP60qr7btuMbY7S9YFq9KZcBb03yFuAnquqf\nZtuw5rHAaUmuYwi/0b++egVwSCvfDKxr63wuQxgA/DnDC8CUv67hyL4OuKuqrqvh0srWkf4/APxl\nm/7otPZTns/wokFVnc9D96fR5xPgyCRXAV9p/R3dhrNGfj93pPzTVfVAVd0ArJ1h/S8E/tfU/tS2\n+1vA94CPJPk14LsztFsub0hyDXA5w7fv1y+g7VzjMer7wNS7+9H9a7ZjfNSlwBlJfpPhBWXKRVV1\nT1XtZBivqeNlNENWzKr9rZtHSlV9Lcm/YDjz++/AFxaxmPtGph8YefwAD47hYxjOpr432jBZ9JWa\nh1mmbRn1nZHpMITkq5a4zIWYGsddLHxfnK3td2aoS1V9LMmXGd5VnZfkdQzvembz28BdwM8xPLej\nz+vo/jBu30f3men702ztF/oB2g+3PclBwJsZ3tV9s10i2nOWZY9Oj/ZtrJ23hi9IHga8gOGd3OsZ\n3vEsqyQbGML2uVX13Qyf1ew5Z6MH2843HqN+0F6U4aHP77zHeFUdl+QXGPazLUn+ZZs1ToasmO7P\n6JM8HfhuVX0UeCfwC8C+SZ7d5u+VZDkG+nPACSPrPbRNXgz8h1b2EmDvxa5ghm15LrBu5Brr0cDf\nMLyNXug2Xg48b2pZGT4PeNYc9S8FfiXJnu2s5t/OUOemWfo3rguA16TdEbGUa5lJngHcXFXvA84B\nfha4h+Ht9EyewvCu6IHW792mzf8C8ApaWLS+fYnhz30AvBr42wV28zEMQQnDPnPJDHUuBY5s63wR\ns+9PT2YI/m+1a8wvmTb/lSO/L1tAHy8AXje1PyXZpz3/T6nhS5K/zfDiuBKeAnyzhfxPM7wLHTXX\n8znXeMzVbtRsxzgjZc+sqi9X1duBnTz0b36tmu7P6Bmuhb4zyQPADxiuMQb44ySPB/6J4SzhhzLc\nundcVf2nBaznDcAHklzLMK4XA8cB7wDOSrKVIQj+3zJvy1OAj7cD70rgT6rq+0leOX0bkzyZ4Vr4\nS6cvuKp2Jjm29XWPVvw2huvQD1NVVyY5F7iW4cz3Ooa3pKN1vpfkNdP7N9cGjo59VZ3fDqbNSb7P\n8E3rt843SLM4Ejg6yQ+AO4H/UVXfSHJp+6DxfzP8s5wpHwQ+meQ3gPOZ9k6hqrYmORl4D/Aihmv3\nJwB/muT3GA7y1yywj98BDkvyNuBuWhgnOa6t8094cH86miGg72QIqodcQqiqa5J8Bfgqw+ckl05b\n195tX70PWMi7uA8DzwKubWN5GvBJ4JwkezIcW7+zgOUtxPnAcUluZDiJGL1MRVX9w+jzWVW/l+Tq\nqjp0nvE4FTg/yd9V1S/Nsf7ZjvFR78zwQX+AC4FrgIe9IDzSvL1Si5bkSVV1bzvjvhjYWFVXrXa/\nHq0yxm197UV4V7tc8lzgQ1W1oCDJj+AtvlpZPw5n9Fo5p2b4F5F7Amca8o+IA4FNSR7D8KHhb65y\nf/Qo4Bm9JHWu+w9jJenHnUEvSZ0z6CWpcwa9JHXOoJekzhn0ktS5/w/ib1ABuhk87gAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcc2940a748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_sample.category_name.value_counts().plot(kind=\"bar\", rot=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing that stood out from the both datasets, is that the distribution of the categories remain relatively the same, which is a good sign for us data scientist. For now that is. Actually there are many ways to conduct sampling on the dataset and still have a representative enough dataset. That is not the main focus in this notebook, but if you would like to know more about sampling and how the `sample` feature works, just reference the Pandas documentation and you will find interesting ways to conduct more advanced sampling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Feature Creation\n",
    "The other operation from the list above that we are going to practise on is the so-called feature creation. As the name suggests, in feature creation we are looking at creating new interesting and useful features from the original dataset; a feature which captures the most important information from the raw information we already have access to. In our `X` table, we would like to create some features from the `text` field, but we are still not clear what kind of features we want to create. We can think of an interesting problem we want to solve, or something we want to analyze from the data, or some questions we want to answer. This is one process to come up with features -- this process is usually called `feature engineering` in the data science community. \n",
    "\n",
    "We know what feature creation is so let us get real involved with our dataset and make it more interesting by adding some special features or attributes if you will. First we are going to obtain the **unigrams** for each text. (Unigram is just a fancy word we use in Text Mining which stands for 'individual words'.) Yes, we want to extract all the words found in each text and append it as a new feature. The reason for extracting unigrams is not so clear yet, but we can start to think of obtaining some statistics about the articles we have: something like **word distribution** or **word frequency**.\n",
    "\n",
    "Before going into any further coding, we will also introduce some basic library called [NLTK](http://www.nltk.org/). The NLTK library is a natural language processing tool used for text mining tasks, so might as well we start to familiarize ourselves with it from now (It may come in handy for the final project!). In partcular, we are going to use the NLTK library to conduct tokenization because we are interested in splitting a sentence into its individual components, which can be words, emojis, emails, etc. So let us go for it! We can call the `nltk` library as follows:\n",
    "\n",
    "```python\n",
    "import nltk\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# takes a like a minute or two to process\n",
    "X['unigrams'] = X['text'].apply(lambda x: dmh.tokenize_text(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_name</th>\n",
       "      <th>unigrams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier) Subje...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "      <td>[From, :, sd345, @, city.ac.uk, (, Michael, Co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "      <td>[From, :, ani, @, ms.uky.edu, (, Aniruddha, B....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, djohnson, @, cs.ucsd.edu, (, Darin, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart) Subject...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, s0612596, @, let.rug.nl, (, M.M, ., ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  category  \\\n",
       "0  From: sd345@city.ac.uk (Michael Collier) Subje...         1   \n",
       "1  From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...         1   \n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...         3   \n",
       "3  From: s0612596@let.rug.nl (M.M. Zwart) Subject...         3   \n",
       "\n",
       "            category_name                                           unigrams  \n",
       "0           comp.graphics  [From, :, sd345, @, city.ac.uk, (, Michael, Co...  \n",
       "1           comp.graphics  [From, :, ani, @, ms.uky.edu, (, Aniruddha, B....  \n",
       "2  soc.religion.christian  [From, :, djohnson, @, cs.ucsd.edu, (, Darin, ...  \n",
       "3  soc.religion.christian  [From, :, s0612596, @, let.rug.nl, (, M.M, ., ...  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you take at the `X` table now, you will see the new columns `unigrams` that we have added. You will notice that it contains an array of tokens, which were extracted from the original `text` field. On first impression, you will notice that the tokenizer is not doing a great job, let us take a closer at a single records and see what was the exact result of the tokenization using the `nltk` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['From',\n",
       "  ':',\n",
       "  'sd345',\n",
       "  '@',\n",
       "  'city.ac.uk',\n",
       "  '(',\n",
       "  'Michael',\n",
       "  'Collier',\n",
       "  ')',\n",
       "  'Subject',\n",
       "  ':',\n",
       "  'Converting',\n",
       "  'images',\n",
       "  'to',\n",
       "  'HP',\n",
       "  'LaserJet',\n",
       "  'III',\n",
       "  '?',\n",
       "  'Nntp-Posting-Host',\n",
       "  ':',\n",
       "  'hampton',\n",
       "  'Organization',\n",
       "  ':',\n",
       "  'The',\n",
       "  'City',\n",
       "  'University',\n",
       "  'Lines',\n",
       "  ':',\n",
       "  '14',\n",
       "  'Does',\n",
       "  'anyone',\n",
       "  'know',\n",
       "  'of',\n",
       "  'a',\n",
       "  'good',\n",
       "  'way',\n",
       "  '(',\n",
       "  'standard',\n",
       "  'PC',\n",
       "  'application/PD',\n",
       "  'utility',\n",
       "  ')',\n",
       "  'to',\n",
       "  'convert',\n",
       "  'tif/img/tga',\n",
       "  'files',\n",
       "  'into',\n",
       "  'LaserJet',\n",
       "  'III',\n",
       "  'format',\n",
       "  '.',\n",
       "  'We',\n",
       "  'would',\n",
       "  'also',\n",
       "  'like',\n",
       "  'to',\n",
       "  'do',\n",
       "  'the',\n",
       "  'same',\n",
       "  ',',\n",
       "  'converting',\n",
       "  'to',\n",
       "  'HPGL',\n",
       "  '(',\n",
       "  'HP',\n",
       "  'plotter',\n",
       "  ')',\n",
       "  'files',\n",
       "  '.',\n",
       "  'Please',\n",
       "  'email',\n",
       "  'any',\n",
       "  'response',\n",
       "  '.',\n",
       "  'Is',\n",
       "  'this',\n",
       "  'the',\n",
       "  'correct',\n",
       "  'group',\n",
       "  '?',\n",
       "  'Thanks',\n",
       "  'in',\n",
       "  'advance',\n",
       "  '.',\n",
       "  'Michael',\n",
       "  '.',\n",
       "  '--',\n",
       "  'Michael',\n",
       "  'Collier',\n",
       "  '(',\n",
       "  'Programmer',\n",
       "  ')',\n",
       "  'The',\n",
       "  'Computer',\n",
       "  'Unit',\n",
       "  ',',\n",
       "  'Email',\n",
       "  ':',\n",
       "  'M.P.Collier',\n",
       "  '@',\n",
       "  'uk.ac.city',\n",
       "  'The',\n",
       "  'City',\n",
       "  'University',\n",
       "  ',',\n",
       "  'Tel',\n",
       "  ':',\n",
       "  '071',\n",
       "  '477-8000',\n",
       "  'x3769',\n",
       "  'London',\n",
       "  ',',\n",
       "  'Fax',\n",
       "  ':',\n",
       "  '071',\n",
       "  '477-8565',\n",
       "  'EC1V',\n",
       "  '0HB',\n",
       "  '.']]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(X[0:1]['unigrams'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nltk` library does a pretty decent job of tokenizing our text. There are many other tokenizers online, such as [spacy](https://spacy.io/), and the built in libraries provided by [scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html). We are making use of the NLTK library because it is open source and because it simple does a good job of segmentating text-based data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Feature subset selection\n",
    "Okay, so we are making some headway here. Let us now make things a litte bit more interesting. We are going to do something different from what we have been doing thus far. We are going use a bit of everything that we have learned so far. Briefly speaking, we are going to move away from our main dataset (one form of feature subset selection), and we are going to generate a document-term matrix from the original dataset. In other words we are going to be creating something like this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt txt](https://docs.google.com/drawings/d/e/2PACX-1vS01RrtPHS3r1Lf8UjX4POgDol-lVF4JAbjXM3SAOU-dOe-MqUdaEMWwJEPk9TtiUvcoSqTeE--lNep/pub?w=748&h=366)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially, it won't have the same shape as the table above, but we will get into that later. For now, let us use scikit learn built in functionalities to generate this document. You will see for yourself how easy it is to generate this table without much coding. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_vect = CountVectorizer()\n",
    "X_counts = count_vect.fit_transform(X.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we did with those two lines of code is that we transorfmed the articles into a term-document matrix. Those lines tokenize each article using a built-in, default tokenizer (often referred to as an `analzyer`) and then produces the word frequency vector for each document. We can create our own analyzers or even use the nltk analyzer that we previously built. To keep things tidy and minimal we are going to use the default analyzer provided by `CountVectorizer`. Let us look closely at this analyzer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['from',\n",
       " 'stanly',\n",
       " 'grok11',\n",
       " 'columbiasc',\n",
       " 'ncr',\n",
       " 'com',\n",
       " 'stanly',\n",
       " 'subject',\n",
       " 're',\n",
       " 'elder',\n",
       " 'brother',\n",
       " 'organization',\n",
       " 'ncr',\n",
       " 'corp',\n",
       " 'columbia',\n",
       " 'sc',\n",
       " 'lines',\n",
       " '15',\n",
       " 'in',\n",
       " 'article',\n",
       " 'apr',\n",
       " '00',\n",
       " '57',\n",
       " '41',\n",
       " '1993',\n",
       " '28246',\n",
       " 'athos',\n",
       " 'rutgers',\n",
       " 'edu',\n",
       " 'rexlex',\n",
       " 'fnal',\n",
       " 'gov',\n",
       " 'writes',\n",
       " 'in',\n",
       " 'article',\n",
       " 'apr',\n",
       " '01',\n",
       " '56',\n",
       " '56',\n",
       " '1993',\n",
       " '22824',\n",
       " 'athos',\n",
       " 'rutgers',\n",
       " 'edu',\n",
       " 'shrum',\n",
       " 'hpfcso',\n",
       " 'fc',\n",
       " 'hp',\n",
       " 'com',\n",
       " 'matt',\n",
       " '22',\n",
       " '14',\n",
       " 'go',\n",
       " 'therefore',\n",
       " 'to',\n",
       " 'the',\n",
       " 'main',\n",
       " 'highways',\n",
       " 'and',\n",
       " 'as',\n",
       " 'many',\n",
       " 'as',\n",
       " 'you',\n",
       " 'find',\n",
       " 'there',\n",
       " 'invite',\n",
       " 'to',\n",
       " 'the',\n",
       " 'wedding',\n",
       " 'feast',\n",
       " 'hmmmmmm',\n",
       " 'sounds',\n",
       " 'like',\n",
       " 'your',\n",
       " 'theology',\n",
       " 'and',\n",
       " 'christ',\n",
       " 'are',\n",
       " 'at',\n",
       " 'odds',\n",
       " 'which',\n",
       " 'one',\n",
       " 'am',\n",
       " 'to',\n",
       " 'believe',\n",
       " 'in',\n",
       " 'this',\n",
       " 'parable',\n",
       " 'jesus',\n",
       " 'tells',\n",
       " 'the',\n",
       " 'parable',\n",
       " 'of',\n",
       " 'the',\n",
       " 'wedding',\n",
       " 'feast',\n",
       " 'the',\n",
       " 'kingdom',\n",
       " 'of',\n",
       " 'heaven',\n",
       " 'is',\n",
       " 'like',\n",
       " 'unto',\n",
       " 'certain',\n",
       " 'king',\n",
       " 'which',\n",
       " 'made',\n",
       " 'marriage',\n",
       " 'for',\n",
       " 'his',\n",
       " 'son',\n",
       " 'so',\n",
       " 'the',\n",
       " 'wedding',\n",
       " 'clothes',\n",
       " 'were',\n",
       " 'customary',\n",
       " 'and',\n",
       " 'given',\n",
       " 'to',\n",
       " 'those',\n",
       " 'who',\n",
       " 'chose',\n",
       " 'to',\n",
       " 'attend',\n",
       " 'this',\n",
       " 'man',\n",
       " 'refused',\n",
       " 'to',\n",
       " 'wear',\n",
       " 'the',\n",
       " 'clothes',\n",
       " 'the',\n",
       " 'wedding',\n",
       " 'clothes',\n",
       " 'are',\n",
       " 'equalivant',\n",
       " 'to',\n",
       " 'the',\n",
       " 'clothes',\n",
       " 'of',\n",
       " 'righteousness',\n",
       " 'when',\n",
       " 'jesus',\n",
       " 'died',\n",
       " 'for',\n",
       " 'our',\n",
       " 'sins',\n",
       " 'those',\n",
       " 'clothes',\n",
       " 'were',\n",
       " 'then',\n",
       " 'provided',\n",
       " 'like',\n",
       " 'that',\n",
       " 'man',\n",
       " 'it',\n",
       " 'is',\n",
       " 'our',\n",
       " 'decision',\n",
       " 'to',\n",
       " 'put',\n",
       " 'the',\n",
       " 'clothes',\n",
       " 'on']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analyze = count_vect.build_analyzer()\n",
    "analyze(\" \".join(list(X[4:5].text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 35788)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can check the shape of this matrix by:\n",
    "X_counts.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['00',\n",
       " '000',\n",
       " '0000',\n",
       " '0000001200',\n",
       " '000005102000',\n",
       " '0001',\n",
       " '000100255pixel',\n",
       " '00014',\n",
       " '000406',\n",
       " '0007']"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can obtain the feature names of the vectorizer, i.e., the terms\n",
    "count_vect.get_feature_names()[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we can see the features found in the all the documents `X`, which are basically all the terms found in all the documents. As I said earlier, the transformation is not in the pretty format (table) we saw above -- the term-document matrix. We can do many things with the `count_vect` vectorizer and its transformation `X_counts`. You can find more information on other cool stuff you can do with the [CountVectorizer](http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction). \n",
    "\n",
    "Now let us try to obtain something that is as close to the pretty table I provided above. Before jumping into the code for doing just that, it is important to mention that the reason for choosing the `fit_transofrm` for the `CountVectorizer` is that it efficiently learns the vocabulary dictionary and returns a term-document matrix.\n",
    "\n",
    "In the next bit of code, we want to extract the first five articles and transform them into document-term matrix, or in this case a 2-dimensional array. Here it goes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_name</th>\n",
       "      <th>unigrams</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier) Subje...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "      <td>[From, :, sd345, @, city.ac.uk, (, Michael, Co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "      <td>[From, :, ani, @, ms.uky.edu, (, Aniruddha, B....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, djohnson, @, cs.ucsd.edu, (, Darin, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart) Subject...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, s0612596, @, let.rug.nl, (, M.M, ., ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: stanly@grok11.columbiasc.ncr.com (stanly...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, stanly, @, grok11.columbiasc.ncr.com...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  category  \\\n",
       "0  From: sd345@city.ac.uk (Michael Collier) Subje...         1   \n",
       "1  From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...         1   \n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...         3   \n",
       "3  From: s0612596@let.rug.nl (M.M. Zwart) Subject...         3   \n",
       "4  From: stanly@grok11.columbiasc.ncr.com (stanly...         3   \n",
       "\n",
       "            category_name                                           unigrams  \n",
       "0           comp.graphics  [From, :, sd345, @, city.ac.uk, (, Michael, Co...  \n",
       "1           comp.graphics  [From, :, ani, @, ms.uky.edu, (, Aniruddha, B....  \n",
       "2  soc.religion.christian  [From, :, djohnson, @, cs.ucsd.edu, (, Darin, ...  \n",
       "3  soc.religion.christian  [From, :, s0612596, @, let.rug.nl, (, M.M, ., ...  \n",
       "4  soc.religion.christian  [From, :, stanly, @, grok11.columbiasc.ncr.com...  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [1, 0, 0, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_counts[0:5].toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see the result is just this huge sparse matrix, which is computationally intensive to generate and difficult to visualize. But we can see that the fifth record, specifically, contain a `1` in the beginning, which from our feature names we can deduce that this article contains exactly one `00` term.\n",
    "\n",
    "We can also use the vectorizer to generate word frequency vector for new documents or articles. Let us try that below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect.transform(['Something completely new.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us put a `00` in the document to see if it is detected as we expect. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect.transform(['00 Something completely new.']).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Impressive, huh!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get you started in thinking about how to better analyze your data or transformation, let us look at this nice little heat map of our term-document matrix. It may come to your suprise to things that you can see or the gems you can mine when you start to look at the data from a different perspective>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_x = [\"term_\"+str(i) for i in count_vect.get_feature_names()[0:20]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['term_00',\n",
       " 'term_000',\n",
       " 'term_0000',\n",
       " 'term_0000001200',\n",
       " 'term_000005102000',\n",
       " 'term_0001',\n",
       " 'term_000100255pixel',\n",
       " 'term_00014',\n",
       " 'term_000406',\n",
       " 'term_0007',\n",
       " 'term_000usd',\n",
       " 'term_0010',\n",
       " 'term_001004',\n",
       " 'term_0010580b',\n",
       " 'term_001125',\n",
       " 'term_001200201pixel',\n",
       " 'term_0014',\n",
       " 'term_001642',\n",
       " 'term_00196',\n",
       " 'term_002']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_y = [\"doc_\"+ str(i) for i in list(X.index)[0:20]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_z = X_counts[0:20, 0:20].toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~omarsar/194.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to plot\n",
    "py.iplot(ta.plot_heat_map(plot_x, plot_y, plot_z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** From the chart above, we can see how sparse the term-document matrix is; i.e., there is only one terms with frequency of `1` in the subselection of the matrix. By the way, you may have noticed that we only selected 20 articles and 20 terms to plot the histrogram. As an excersise you can try to modify the code above to plot the entire term-document matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The great thing about what we have done so far is that we now open doors to new problems. Let us be optimistic. Even though we have the problem of sparsity and a very high dimensional data, we are now closer to uncovering wonders from the data. You see, the price you pay for the hard work is worth it because now you are gaining a lot of knowledge from what was just a list of what appeared to be irrelevant articles. Just the fact that you can blow up the data and find out interesting characteristics about the dataset in just a couple lines of code, is something that truly inspires me to practise Data Science. That's the motivation right there!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 Dimensionality Reduction\n",
    "Since we have just touched on the concept of sparsity most naturally the problem of \"curse of dimentionality\" comes up. I am not going to get into the full details of what dimensionality reduction is and what it is good for just the fact that is an excellent technique for visualizing data efficiently (please refer to notes for more information). All I can say is that we are going to deal with the issue of sparsity with a few lines of code. And we are going to try to visualize our data more efficiently with the results.\n",
    "\n",
    "We are going to make use of Principal Component Analysis to efficeintly reduce the dimensions of our data, with the main goal of \"finding a projection that captures the largest amounty of variation in the data.\" This concept is important as it is very useful for visualizing and observing the characteristics of our dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[PCA Algorithm](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)\n",
    "\n",
    "**Input:** Raw term-vector matrix\n",
    "\n",
    "**Output:** Projections "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_reduced = PCA(n_components=3).fit_transform(X_counts.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 3)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_reduced.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "trace1 = ta.get_trace(X_reduced, X[\"category_name\"], \"alt.atheism\", \"rgb(71,233,163)\")\n",
    "trace2 = ta.get_trace(X_reduced, X[\"category_name\"], \"soc.religion.christian\", \"rgb(52,133,252)\")\n",
    "trace3 = ta.get_trace(X_reduced, X[\"category_name\"], \"comp.graphics\", \"rgb(229,65,136)\")\n",
    "trace4 = ta.get_trace(X_reduced, X[\"category_name\"], \"sci.med\", \"rgb(99,151,68)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [trace1, trace2, trace3, trace4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~omarsar/4.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layout = go.Layout(\n",
    "    margin=dict(\n",
    "        l=0,\n",
    "        r=0,\n",
    "        b=0,\n",
    "        t=0\n",
    "    )\n",
    ")\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='simple-3d-scatter')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the 3D visualization above, we see some \"hint of separation in the data\"; i.e., they have some special grouping by category, but it is not immediately clear. The PCA was applied to the raw frequencies and this is considered a very naive approach as some words are not really unique to a document. Only categorizing by word frequency is considered a \"bag of word\" approach. Later on in the course you will learn about different approaches on how to create better features from the term-vector matrix, such as term-frequency inverse document frequency so-called [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5 Atrribute Transformation / Aggregation\n",
    "We can do other things with the term-vector matrix besides applying dimensionalaity reduction technique to deal with sparsity problem. Here we are going to generate a simple distribution of the words found in all the entire set of articles. Intuitively, this may not make any sense, but in data science sometimes we take some things for granted, as we just have to explore the data first to make any premature conclusions. On the topic of attribute transformation, we will take the distribution and put the distribution in a scale that makes the it easy to analyze patterns in the distrubution of words. Let us get into it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to compute these frequencies for each term in all documents. Visually speaking, we are seeking to add values of the 2D matrix, vertically; i.e., sum of columns. You can also refer to this process as aggregation, which we won't explore further in this notebook because of the type of data we are dealing with. But I believe you get the idea of what that includes.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt txt](https://docs.google.com/drawings/d/e/2PACX-1vTMfs0zWsbeAl-wrpvyCcZqeEUf7ggoGkDubrxX5XtwC5iysHFukD6c-dtyybuHnYigiRWRlRk2S7gp/pub?w=750&h=412)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note this takes time to compute. You may want to reduce the amount of terms you want to compute frequencies for\n",
    "term_frequencies = []\n",
    "for j in range(0,X_counts.shape[1]):\n",
    "    term_frequencies.append(sum(X_counts[:,j].toarray()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([134])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "term_frequencies[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The draw time for this plot will be slow for clients without much RAM.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ellfae/anaconda3/envs/tensorflow/lib/python3.6/site-packages/plotly/api/v1/clientresp.py:40: UserWarning:\n",
      "\n",
      "Estimated Draw Time Slow\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~omarsar/196.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py.iplot(ta.plot_word_frequency([count_vect.get_feature_names(), term_frequencies], \"Term Frequency Distribution\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** The chart above contains all the vocabulary, and it's computationally intensive to both compute and visualize. You can try to reduce the number of terms you want to visualize as an exercise. Additionally, you can attempt to sort the terms on the x-axis by frequency instead of in alphabetical order."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have those term frequencies, we can also transform the values in that vector into the log distribution. All we need is to import the `math` library provided by python and apply it to the array of values of the term frequency vector. This is a typical example of attribute transformation. Let's go for it. The log distribution is a technique to visualize the term frequency into a scale that makes you easily visualize the distribution in a more readable format. In other words, the discrepancies between the term frequencies are now better observable. Let us try it out. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "term_frequencies_log = [math.log(i) for i in term_frequencies]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The draw time for this plot will be slow for clients without much RAM.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ellfae/anaconda3/envs/tensorflow/lib/python3.6/site-packages/plotly/api/v1/clientresp.py:40: UserWarning:\n",
      "\n",
      "Estimated Draw Time Slow\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~omarsar/198.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py.iplot(ta.plot_word_frequency([count_vect.get_feature_names(), term_frequencies_log], \"Term Frequency Distribution\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides observing a complete transformation on the disrtibution, notice the scale on the y-axis. The log distribution in our unsorted example has no meaning, but try to properly sort the terms by their frequency, and you will see an interesting effect. Go for it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.6 Discretization and Binarization\n",
    "In this section we are going to discuss a very important pre-preprocessing technique used to transform the data, specifically categorical values, into a format that satisfies certain criteria required by particular algorithms. Given our current original dataset, we would like to transform one of the attributes, `category_name`, into four binary attributes. In other words, we are taking the category name and replacing it with a `n` asymmetric binary attributes. The logic behind this transformation is dicussed in detail in the required text book (please refer to it on page 58). People from the Deep Learning community also refer to this transformation as one-hot encoding, but as you may become aware later in the course, these concepts are all the same, we just have different prefrence on how we refer to the concetps. Let us take a look at what we want to achieve in code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import preprocessing, metrics, decomposition, pipeline, dummy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlb = preprocessing.LabelBinarizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LabelBinarizer(neg_label=0, pos_label=1, sparse_output=False)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlb.fit(X.category)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlb.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "X['bin_category'] = mlb.transform(X['category']).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>category_name</th>\n",
       "      <th>unigrams</th>\n",
       "      <th>bin_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: sd345@city.ac.uk (Michael Collier) Subje...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "      <td>[From, :, sd345, @, city.ac.uk, (, Michael, Co...</td>\n",
       "      <td>[0, 1, 0, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...</td>\n",
       "      <td>1</td>\n",
       "      <td>comp.graphics</td>\n",
       "      <td>[From, :, ani, @, ms.uky.edu, (, Aniruddha, B....</td>\n",
       "      <td>[0, 1, 0, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, djohnson, @, cs.ucsd.edu, (, Darin, ...</td>\n",
       "      <td>[0, 0, 0, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: s0612596@let.rug.nl (M.M. Zwart) Subject...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, s0612596, @, let.rug.nl, (, M.M, ., ...</td>\n",
       "      <td>[0, 0, 0, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: stanly@grok11.columbiasc.ncr.com (stanly...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, stanly, @, grok11.columbiasc.ncr.com...</td>\n",
       "      <td>[0, 0, 0, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, vbv, @, lor.eeap.cwru.edu, (, Virgil...</td>\n",
       "      <td>[0, 0, 0, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>From: jodfishe@silver.ucs.indiana.edu (joseph ...</td>\n",
       "      <td>3</td>\n",
       "      <td>soc.religion.christian</td>\n",
       "      <td>[From, :, jodfishe, @, silver.ucs.indiana.edu,...</td>\n",
       "      <td>[0, 0, 0, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>From: aldridge@netcom.com (Jacquelin Aldridge)...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "      <td>[From, :, aldridge, @, netcom.com, (, Jacqueli...</td>\n",
       "      <td>[0, 0, 1, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>From: geb@cs.pitt.edu (Gordon Banks) Subject: ...</td>\n",
       "      <td>2</td>\n",
       "      <td>sci.med</td>\n",
       "      <td>[From, :, geb, @, cs.pitt.edu, (, Gordon, Bank...</td>\n",
       "      <td>[0, 0, 1, 0]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  category  \\\n",
       "0  From: sd345@city.ac.uk (Michael Collier) Subje...         1   \n",
       "1  From: ani@ms.uky.edu (Aniruddha B. Deglurkar) ...         1   \n",
       "2  From: djohnson@cs.ucsd.edu (Darin Johnson) Sub...         3   \n",
       "3  From: s0612596@let.rug.nl (M.M. Zwart) Subject...         3   \n",
       "4  From: stanly@grok11.columbiasc.ncr.com (stanly...         3   \n",
       "5  From: vbv@lor.eeap.cwru.edu (Virgilio (Dean) B...         3   \n",
       "6  From: jodfishe@silver.ucs.indiana.edu (joseph ...         3   \n",
       "7  From: aldridge@netcom.com (Jacquelin Aldridge)...         2   \n",
       "8  From: geb@cs.pitt.edu (Gordon Banks) Subject: ...         2   \n",
       "\n",
       "            category_name                                           unigrams  \\\n",
       "0           comp.graphics  [From, :, sd345, @, city.ac.uk, (, Michael, Co...   \n",
       "1           comp.graphics  [From, :, ani, @, ms.uky.edu, (, Aniruddha, B....   \n",
       "2  soc.religion.christian  [From, :, djohnson, @, cs.ucsd.edu, (, Darin, ...   \n",
       "3  soc.religion.christian  [From, :, s0612596, @, let.rug.nl, (, M.M, ., ...   \n",
       "4  soc.religion.christian  [From, :, stanly, @, grok11.columbiasc.ncr.com...   \n",
       "5  soc.religion.christian  [From, :, vbv, @, lor.eeap.cwru.edu, (, Virgil...   \n",
       "6  soc.religion.christian  [From, :, jodfishe, @, silver.ucs.indiana.edu,...   \n",
       "7                 sci.med  [From, :, aldridge, @, netcom.com, (, Jacqueli...   \n",
       "8                 sci.med  [From, :, geb, @, cs.pitt.edu, (, Gordon, Bank...   \n",
       "\n",
       "   bin_category  \n",
       "0  [0, 1, 0, 0]  \n",
       "1  [0, 1, 0, 0]  \n",
       "2  [0, 0, 0, 1]  \n",
       "3  [0, 0, 0, 1]  \n",
       "4  [0, 0, 0, 1]  \n",
       "5  [0, 0, 0, 1]  \n",
       "6  [0, 0, 0, 1]  \n",
       "7  [0, 0, 1, 0]  \n",
       "8  [0, 0, 1, 0]  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the new attribute we have added to the `X` table. You can see that the new attribute, which is called `bin_category`, contains an array of 0's and 1's. The `1` is basically to indicate the position of the label or category we binarized. If you look at the first two records, the one is places in slot 2 in the array; this helps to indicate to any of the algorithms which we are feeding this data to, that the record belong to that specific category. \n",
    "\n",
    "Attributes with continuous values also have strategies to tranform the data; this is usually called **Discretization** (please refer to the text book for more inforamation)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Concluding Remarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! We have come a long way! We can now call ourselves experts of Data Preprocessing. You should feel excited and proud because the process of Data Mining usually involves 70% preprocessing and 30% training algorithms. You will learn this as you progress in the Data Mining course. I really feel that if you go through the exercise and challenge yourself, you have just become an expert. \n",
    "\n",
    "From here the possibilities for you are endless. You know how to use almost every technique for preprocessing with state-of-the-art tools, such as as Pandas, which makes even more an expert. You are now with the trend! \n",
    "\n",
    "After completing this notebook you can do a lot with the results we have generated. You can train algorithms and models that are able to classify articles into certain categories and much more. You can also try to experiment with different datasets, or venture further into text analytics by using new deep learning techniques such as word2vec. All of this will be presented in the next lab session. Until then, go teach machines how to be intelligent to make the world a better place. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Pandas cook book ([Recommended for starters](http://pandas.pydata.org/pandas-docs/stable/cookbook.html))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}