jupyter_notebooks/week_7/ex6_spam.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spam Classification with SVMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "# needed modules\n",
    "path_to_modules = '../../algorithms_in_python/week_7/ex6/'\n",
    "sys.path.append(path_to_modules)\n",
    "import numpy as np\n",
    "import scipy.io as sio\n",
    "# we use scikit-learn's svm since the simple SMO algorithm is only meant for small datasets\n",
    "# from svmModel import SVMModel\n",
    "from sklearn import svm\n",
    "\n",
    "from emailFeatures import email_features\n",
    "from getVocabList import get_vocab_list\n",
    "from processEmail import process_email\n",
    "from readFile import read_file\n",
    "from svmModel import SVMModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Email Pre-processing\n",
    "  To use an SVM to classify emails into Spam v.s. Non-Spam, you first need\n",
    "  to convert each email into a vector of features. In this part, you will\n",
    "  implement the pre-processing steps for each email. You should\n",
    "  complete the code in processEmail.py to produce a word indices vector\n",
    "  for a given email."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Pre-processing sample email (emailSample1.txt)\n",
      "\n",
      "\n",
      "==== Processed Email ====\n",
      "\n",
      "  anyone knows how much it costs to host a web portal well it depends on how many visitors youre expectingthis can be anywhere from less than number bucks a month to a couple of dollarnumberyou should checkout httpaddr or perhaps amazon ecnumberif youre running something bigto unsubscribe yourself from this mailing list send an email toemailaddr\n",
      "Word Indices: \n",
      "\n",
      "[86, 916, 794, 1077, 883, 370, 1699, 790, 1822, 1831, 883, 431, 1171, 794, 1002, 1895, 238, 162, 89, 688, 945, 1663, 1120, 1062, 1699, 375, 1162, 1510, 799, 1182, 1237, 1895, 1440, 1547, 1758, 1896, 688, 1676, 992, 961, 1477, 71, 530]\n"
     ]
    }
   ],
   "source": [
    "print('\\nPre-processing sample email (emailSample1.txt)\\n')\n",
    "\n",
    "# Extract Features\n",
    "file_contents = read_file(path_to_modules + 'emailSample1.txt')\n",
    "word_indices = process_email(file_contents, path_to_modules + 'vocab.txt')\n",
    "\n",
    "# Print Stats\n",
    "print('Word Indices: \\n')\n",
    "print(word_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Feature Extraction\n",
    "  Convert each email into a vector of features in R^n."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Extracting features from sample email (emailSample1.txt)\n",
      "\n",
      "Length of feature vector:  1899\n",
      "Number of non-zero entries:  38\n"
     ]
    }
   ],
   "source": [
    "print('\\nExtracting features from sample email (emailSample1.txt)\\n')\n",
    "\n",
    "# Extract Features\n",
    "features = email_features(word_indices)\n",
    "\n",
    "# Print Stats\n",
    "print('Length of feature vector: ', len(features))\n",
    "print('Number of non-zero entries: ', sum(features > 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Train Linear SVM for Spam Classification \n",
    "  Train a linear classifier to determine if an email is Spam or Not-Spam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training Linear SVM (Spam Classification)\n",
      "\n",
      "(this may take 1 to 2 minutes) ...\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\hamed\\Anaconda3\\lib\\site-packages\\sklearn\\svm\\base.py:224: ConvergenceWarning: Solver terminated early (max_iter=200).  Consider pre-processing your data with StandardScaler or MinMaxScaler.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy:  99.6\n"
     ]
    }
   ],
   "source": [
    "# Load the Spam Email dataset\n",
    "# You will have X, y in your environment\n",
    "mat_contents = sio.loadmat(path_to_modules + 'spamTrain.mat')\n",
    "X = mat_contents['X']\n",
    "y = mat_contents['y'].flatten()\n",
    "\n",
    "print('\\nTraining Linear SVM (Spam Classification)\\n')\n",
    "print('(this may take 1 to 2 minutes) ...\\n')\n",
    "\n",
    "C = 0.1\n",
    "#model = svm.LinearSVC(C=C)\n",
    "model = svm.SVC(C=C, kernel='linear', tol=1e-3, max_iter=200)\n",
    "model.fit(X, y)\n",
    "#model = SVMModel()\n",
    "#model.train(X, y, C, kernel_type='lnr')\n",
    "\n",
    "p = model.predict(X)\n",
    "\n",
    "print('Training Accuracy: ', np.multiply(np.mean((p == y).astype(int)), 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Test Spam Classification\n",
    "  After training the classifier, we can evaluate it on a test set. We have\n",
    "  included a test set in spamTest.mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Evaluating the trained Linear SVM on a test set ...\n",
      "\n",
      "Test Accuracy:  98.1\n"
     ]
    }
   ],
   "source": [
    "# Load the test dataset\n",
    "# You will have Xtest, ytest in your environment\n",
    "mat_contents = sio.loadmat(path_to_modules + 'spamTest.mat')\n",
    "Xtest = mat_contents['Xtest']\n",
    "ytest = mat_contents['ytest'].flatten()\n",
    "\n",
    "print('\\nEvaluating the trained Linear SVM on a test set ...\\n')\n",
    "p = model.predict(Xtest)\n",
    "\n",
    "print('Test Accuracy: ', np.multiply(np.mean((p == ytest).astype(int)), 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: Top Predictors of Spam \n",
    "  Since the model we are training is a linear SVM, we can inspect the\n",
    "  weights learned by the model to understand better how it is determining\n",
    "  whether an email is spam or not. The following code finds the words with\n",
    "  the highest weights in the classifier. Informally, the classifier\n",
    "  'thinks' that these words are the most likely indicators of spam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Top predictors of spam: \n",
      "\n",
      "Word        Weight      \n",
      "___         ___         \n",
      "\n",
      "our         0.391337    \n",
      "click       0.379293    \n",
      "remov       0.365469    \n",
      "visit       0.335558    \n",
      "guarante    0.327508    \n",
      "basenumb    0.292663    \n",
      "dollar      0.258750    \n",
      "bodi        0.233422    \n",
      "ga          0.222452    \n",
      "below       0.212933    \n",
      "price       0.209824    \n",
      "most        0.201052    \n",
      "will        0.199862    \n",
      "al          0.198814    \n",
      "am          0.189730    \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Sort the weights, result a list of tuples (index, value)\n",
    "weights = sorted(enumerate(model.coef_[0]), key=lambda x: x[1], reverse=True)\n",
    "vocabList = get_vocab_list(path_to_modules + 'vocab.txt')\n",
    "\n",
    "print('\\nTop predictors of spam: \\n')\n",
    "print('%-12s%-12s' % (\"Word\", \"Weight\"))\n",
    "print('%-12s%-12s\\n' % (\"___\", \"___\"))\n",
    "for i in range(15):\n",
    "    print ('%-12s%-12f' % (vocabList[weights[i][0]], weights[i][1]))\n",
    "\n",
    "print('\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: Try Your Own Emails \n",
    "  Now that we've trained the spam classifier, we can use it on your own\n",
    "  emails! In the starter code, we have included spamSample1.txt,\n",
    "  spamSample2.txt, emailSample1.txt and emailSample2.txt as examples. \n",
    "  The following code reads in one of these emails and then uses your \n",
    "  learned SVM classifier to determine whether the email is Spam or \n",
    "  Not Spam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def classify_email(filename):\n",
    "    x = email_features(process_email(read_file(filename), path_to_modules + 'vocab.txt')).reshape(1, -1)\n",
    "    pred = model.predict(x)\n",
    "    print('\\nProcessed {}\\n\\nSpam Classification: {}\\n'.format(filename, pred))\n",
    "    print('(1 indicates spam, 0 indicates not spam)\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "==== Processed Email ====\n",
      "\n",
      "  anyone knows how much it costs to host a web portal well it depends on how many visitors youre expectingthis can be anywhere from less than number bucks a month to a couple of dollarnumberyou should checkout httpaddr or perhaps amazon ecnumberif youre running something bigto unsubscribe yourself from this mailing list send an email toemailaddr\n",
      "\n",
      "Processed ../../algorithms_in_python/week_7/ex6/emailSample1.txt\n",
      "\n",
      "Spam Classification: [0]\n",
      "\n",
      "(1 indicates spam, 0 indicates not spam)\n",
      "\n",
      "\n",
      "\n",
      "==== Processed Email ====\n",
      "\n",
      " folks my first time posting  have a bit of unix experience but am new to linux just got a new pc at home  dell box with windows xp added a second hard diskfor linux partitioned the disk and have installed suse numbernumber from cd which wentfine except it didnt pick up my monitor i have a dell branded enumberfpp number lcd flat panel monitor and a nvidia geforcenumbertinumber video card both of which are probably too new to feature in suses defaultset i downloaded a driver from the nvidia website and installed it using rpmthen i ran saxnumber as was recommended in some postings i found on the net butit still doesnt feature my video card in the available list what next another problem i have a dell branded keyboard and if i hit capslock twicethe whole machine crashes in linux not windows  even the onoff switch isinactive leaving me to reach for the power cable instead if anyone can help me in any way with these probs id be really grateful ive searched the net but have run out of ideas or should i be going for a different version of linux such as redhat opinionswelcome thanks a lotpeter irish linux users group emailaddrhttpaddr for unsubscription informationlist maintainer emailaddr\n",
      "\n",
      "Processed ../../algorithms_in_python/week_7/ex6/emailSample2.txt\n",
      "\n",
      "Spam Classification: [0]\n",
      "\n",
      "(1 indicates spam, 0 indicates not spam)\n",
      "\n",
      "\n",
      "\n",
      "==== Processed Email ====\n",
      "\n",
      " click here to gain money visit this website httpaddr to gain many dollar below guarantee now\n",
      "\n",
      "Processed ../../algorithms_in_python/week_7/ex6/spamSample1.txt\n",
      "\n",
      "Spam Classification: [1]\n",
      "\n",
      "(1 indicates spam, 0 indicates not spam)\n",
      "\n",
      "\n",
      "\n",
      "==== Processed Email ====\n",
      "\n",
      " best buy viagra generic onlineviagra numbermg x number pills dollarnumber free pills  reorder discount top selling number quality  satisfaction guaranteedwe accept visa master  echeck payments number satisfied customershttpaddr\n",
      "\n",
      "Processed ../../algorithms_in_python/week_7/ex6/spamSample2.txt\n",
      "\n",
      "Spam Classification: [1]\n",
      "\n",
      "(1 indicates spam, 0 indicates not spam)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "classify_email(path_to_modules + 'emailSample1.txt')\n",
    "classify_email(path_to_modules + 'emailSample2.txt')\n",
    "classify_email(path_to_modules + 'spamSample1.txt')\n",
    "classify_email(path_to_modules + 'spamSample2.txt')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}