2 - Text data preprocessing/Text data preprocessing.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"><img alt=\"Creative Commons License\" align=\"left\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png\" /></a>&nbsp;| [Emmanuel Rachelson](https://personnel.isae-supaero.fr/emmanuel-rachelson?lang=en) | <a href=\"https://supaerodatascience.github.io/machine-learning/\">https://supaerodatascience.github.io/machine-learning/</a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size:22pt; line-height:25pt; font-weight:bold; text-align:center;\">Text data pre-processing</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercice, we shall load a database of email messages and pre-format them so that we can design automated classification methods or use off-the-shelf classifiers. The general purpose of this notebook is to give a practical notion (through this example) of how important data pre-processing can be in a Machine Learning workflow, and generalize it to other situations.\n",
    "\n",
    "\"What is there to pre-process?\" you might ask. Well, actually, text data comes in a very noisy form that we, humans, have become accustomed to and filter out effortlessly to grasp the core meaning of the text. It has a lot of formatting (fonts, colors, typography...), punctuation, abbreviations, common words, grammatical rules, etc. that we might wish to discard before even starting the data analysis.\n",
    "\n",
    "Here are some pre-processing steps that can be performed on text:\n",
    "1. loading the data, removing attachements, merging title and body;\n",
    "2. tokenizing - splitting the text into atomic \"words\";\n",
    "3. removal of stop-words - very common words;\n",
    "4. removal of non-words - punctuation, numbers, gibberish;\n",
    "3. lemmatization - merge together \"find\", \"finds\", \"finder\".\n",
    "\n",
    "The final goal is to be able to represent a document as a mathematical object, e.g. a vector, that our machine learning black boxes can process."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<img id=\"fig1\" src=\"https://imgs.xkcd.com/comics/constructive.png\"> \n",
    "\n",
    "A tech company comes to you to create a moderation system for their social network : they want to detect spam comments, and later on also detect offensive contents to remove them automatically.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#1.-Load-the-data\" data-toc-modified-id=\"1.-Load-the-data-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>1. Load the data</a></span></li><li><span><a href=\"#2.-Filtering-out-the-noise\" data-toc-modified-id=\"2.-Filtering-out-the-noise-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>2. Filtering out the noise</a></span></li><li><span><a href=\"#3.-Even-better-filtering\" data-toc-modified-id=\"3.-Even-better-filtering-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>3. Even better filtering</a></span></li><li><span><a href=\"#4.-Term-frequency-times-inverse-document-frequency\" data-toc-modified-id=\"4.-Term-frequency-times-inverse-document-frequency-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>4. Term frequency times inverse document frequency</a></span></li><li><span><a href=\"#5.-Utility-function\" data-toc-modified-id=\"5.-Utility-function-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>5. Utility function</a></span></li></ul></div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Load the data\n",
    "\n",
    "To showcase our proposed system, we load a database of email messages and pre-format them so that we can design automated classification methods or use off-the-shelf classifiers.\n",
    "\n",
    "\n",
    "**Questions** :\n",
    "- What simple statistics could you print on the dataset ?\n",
    "- Why is there multiple folders on the dataset ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!git clone git@github.com:SupaeroDataScience/machine-learning.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "train_dir = '../data/lingspam_public/bare/'\n",
    "# train_dir = 'machine-learning/data/ling-spam/train-mails/'\n",
    "\n",
    "email_path = []\n",
    "email_label = []\n",
    "for d in os.listdir(train_dir):\n",
    "    folder = os.path.join(train_dir,d)\n",
    "    email_path += [os.path.join(folder,f) for f in os.listdir(folder)]\n",
    "    email_label += [f[0:3]=='spm' for f in os.listdir(folder)]\n",
    "print(\"number of emails\",len(email_path))\n",
    "email_nb = 8 # try 8 for a spam example\n",
    "print(\"email file:\", email_path[email_nb])\n",
    "print(\"email is a spam:\", email_label[email_nb])\n",
    "print(open(email_path[email_nb]).read())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Filtering out the noise\n",
    "\n",
    "One nice thing about scikit-learn is that is has lots of preprocessing utilities. Like [`CountVectorizer`](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) for instance, that converts a collection of text documents to a matrix of token counts.\n",
    "\n",
    "- To remove stop-words, we set: `stop_words='english'`\n",
    "- To convert all words to lowercase: `lowercase=True`\n",
    "- The default tokenizer in scikit-learn removes punctuation and only keeps words of more than 2 letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "countvect = CountVectorizer(input='filename', stop_words='english', lowercase=True)\n",
    "word_count = countvect.fit_transform(email_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of documents:\", len(email_path))\n",
    "words = countvect.get_feature_names_out()\n",
    "print(\"Number of words:\", len(words))\n",
    "print(\"Document - words matrix:\", word_count.shape)\n",
    "print(\"First words:\", words[0:100])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Even better filtering\n",
    "\n",
    "That's already quite ok, but this pre-processing does not perform lemmatization, the list of stop-words could be better and we could wish to remove non-english words (misspelled, with numbers, etc.).\n",
    "\n",
    "A slightly better preprocessing uses the [Natural Language Toolkit](https://www.nltk.org/https://www.nltk.org/). The one below:\n",
    "- tokenizes;\n",
    "- removes punctuation;\n",
    "- removes stop-words;\n",
    "- removes non-English and misspelled words (optional);\n",
    "- removes 1-character words;\n",
    "- removes non-alphabetical words (numbers and codes essentially)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run only of nltk is not installed\n",
    "#%pip install nltk\n",
    "#import nltk\n",
    "#nltk.download('stopwords')\n",
    "#nltk.download('words')\n",
    "#nltk.download('wordnet')\n",
    "#nltk.download('omw-1.4')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import wordpunct_tokenize          \n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.corpus import words\n",
    "from string import punctuation\n",
    "class LemmaTokenizer(object):\n",
    "    def __init__(self, remove_non_words=True):\n",
    "        self.wnl = WordNetLemmatizer()\n",
    "        self.stopwords = set(stopwords.words('english'))\n",
    "        self.words = set(words.words())\n",
    "        self.remove_non_words = remove_non_words\n",
    "    def __call__(self, doc):\n",
    "        # tokenize words and punctuation\n",
    "        word_list = wordpunct_tokenize(doc)\n",
    "        # remove stopwords\n",
    "        word_list = [word for word in word_list if word not in self.stopwords]\n",
    "        # remove non words\n",
    "        if(self.remove_non_words):\n",
    "            word_list = [word for word in word_list if word in self.words]\n",
    "        # remove 1-character words\n",
    "        word_list = [word for word in word_list if len(word)>1]\n",
    "        # remove non alpha\n",
    "        word_list = [word for word in word_list if word.isalpha()]\n",
    "        return [self.wnl.lemmatize(t) for t in word_list]\n",
    "\n",
    "countvect = CountVectorizer(input='filename',tokenizer=LemmaTokenizer(remove_non_words=True))\n",
    "word_count = countvect.fit_transform(email_path)\n",
    "feat2word = {v: k for k, v in countvect.vocabulary_.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Number of documents:\", len(email_path))\n",
    "words = countvect.get_feature_names_out()\n",
    "print(\"Number of words:\", len(words))\n",
    "print(\"Document - words matrix:\", word_count.shape)\n",
    "print(\"First words:\", words[0:100])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Term frequency times inverse document frequency\n",
    "\n",
    "After this first preprocessing, each document is summarized by a vector of size \"number of words in the extracted dictionnary\". For example, the first email in the list has become:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "mail_number = 0\n",
    "text = open(email_path[mail_number]).read()\n",
    "print(\"Original email:\")\n",
    "print(text)\n",
    "#print(LemmaTokenizer()(text))\n",
    "#print(len(set(LemmaTokenizer()(text))))\n",
    "#print(len([feat2word[i] for i in word_count2[mail_number, :].nonzero()[1]]))\n",
    "#print(len([word_count2[mail_number, i] for i in word_count2[mail_number, :].nonzero()[1]]))\n",
    "#print(set([feat2word[i] for i in word_count2[mail_number, :].nonzero()[1]])-set(LemmaTokenizer()(text)))\n",
    "emailBagOfWords = {feat2word[i]: word_count[mail_number, i] for i in word_count[mail_number, :].nonzero()[1]}\n",
    "print(\"Bag of words representation (\", len(emailBagOfWords), \" words in dict):\", sep='')\n",
    "print(emailBagOfWords)\n",
    "print(\"\\nVector reprensentation (\", word_count[mail_number, :].nonzero()[1].shape[0], \" non-zero elements):\", sep='')\n",
    "print(word_count[mail_number, :])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Questions** : \n",
    "- What is a bag-of-word representation ?\n",
    "- What kind of feature selection or feature engineering could you derivate from this bag of word representation ?\n",
    "\n",
    "Counting words is a good start but there is an issue: longer documents will have higher average count values than shorter documents, even though they might talk about the same topics.\n",
    "\n",
    "To avoid these potential discrepancies it suffices to divide the number of occurrences of each word in a document by the total number of words in the document: these new features are called `tf` for Term Frequencies.\n",
    "\n",
    "Another refinement on top of `tf` is to downscale weights for words that occur in many documents in the corpus and are therefore less informative than those that occur only in a smaller portion of the corpus.\n",
    "\n",
    "This downscaling is called `tf–idf` for “Term Frequency times Inverse Document Frequency” and again, scikit-learn does the job for us with the [TfidfTransformer](scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "tfidf = TfidfTransformer().fit_transform(word_count)\n",
    "tfidf.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now every email in the corpus has a vector representation that filters out unrelevant tokens and retains the significant information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"email 0:\")\n",
    "print(tfidf[0,:])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Utility function\n",
    "\n",
    "Let's put all this loading process into a separate file so that we can reuse it in other experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import load_spam\n",
    "spam_data = load_spam.spam_data_loader()\n",
    "spam_data.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spam_data.print_email(8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}