assignment3 - Spam Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7109c4020fb6a8bcaef37e41e738e03d",
     "grade": false,
     "grade_id": "cell-dcb6ef72e08dd5fe",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Assignment 3\n",
    "\n",
    "In this assignment you will explore text message data and create models to predict if a message is spam or not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "97714be9ccaf89bbff57d64a249680b8",
     "grade": false,
     "grade_id": "cell-f403cc37f510e3b4",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>FreeMsg Hey there darling it's been 3 week's n...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Even my brother is not like to speak with me. ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>As per your request 'Melle Melle (Oru Minnamin...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>WINNER!! As a valued network customer you have...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Had your mobile 11 months or more? U R entitle...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  target\n",
       "0  Go until jurong point, crazy.. Available only ...       0\n",
       "1                      Ok lar... Joking wif u oni...       0\n",
       "2  Free entry in 2 a wkly comp to win FA Cup fina...       1\n",
       "3  U dun say so early hor... U c already then say...       0\n",
       "4  Nah I don't think he goes to usf, he lives aro...       0\n",
       "5  FreeMsg Hey there darling it's been 3 week's n...       1\n",
       "6  Even my brother is not like to speak with me. ...       0\n",
       "7  As per your request 'Melle Melle (Oru Minnamin...       0\n",
       "8  WINNER!! As a valued network customer you have...       1\n",
       "9  Had your mobile 11 months or more? U R entitle...       1"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "spam_data = pd.read_csv('assets/spam.csv')\n",
    "\n",
    "spam_data['target'] = np.where(spam_data['target']=='spam',1,0)\n",
    "spam_data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8458ef46cc0956f75d602cfadbb125f8",
     "grade": false,
     "grade_id": "cell-73f5a763723ca5b1",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(spam_data['text'], \n",
    "                                                    spam_data['target'], \n",
    "                                                    random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "aa1acdd18515eabb22a61270e7d9aed5",
     "grade": false,
     "grade_id": "cell-687d4f4eb100a178",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 1\n",
    "What percentage of the documents in `spam_data` are spam?\n",
    "\n",
    "*This function should return a float, the percent value (i.e. $ratio * 100$).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2793aeb56f0b9f5455a8022808ce2c68",
     "grade": false,
     "grade_id": "cell-2707cd8ee068d8d3",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.406317300789663"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    ratio = spam_data['target'].sum()/spam_data.shape[0]\n",
    "    # raise NotImplementedError()\n",
    "    return ratio*100\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b439a749e0a4074aeacbe58049c5bf35",
     "grade": true,
     "grade_id": "cell-35ee2f1c33047f8c",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a1d3d36cf3d3ae8a7d2e0334eb0a7b9c",
     "grade": false,
     "grade_id": "cell-a86c7b0db64621be",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 2\n",
    "\n",
    "Fit the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "What is the longest token in the vocabulary?\n",
    "\n",
    "*This function should return a string.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a99146357529f0924de5432298a2a73b",
     "grade": false,
     "grade_id": "cell-ed0fec2012295ea3",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'com1win150ppmx3age16subscription'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "def answer_two():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    vect = CountVectorizer().fit(X_train)\n",
    "    features = vect.get_feature_names()\n",
    "    feature_lens = [(feature,len(feature)) for feature in features]\n",
    "    feature_lens_sorted = sorted(feature_lens, key=lambda x:x[1], reverse=True)\n",
    "    longest = feature_lens_sorted[0][0]\n",
    "    # raise NotImplementedError()\n",
    "    return longest\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "50c106c6e36fbba1eac00269b3215cef",
     "grade": true,
     "grade_id": "cell-6eb97e449cd12bee",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "77778225d05ba447bf388c0884703333",
     "grade": false,
     "grade_id": "cell-bf601421a6965e8f",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 3\n",
    "\n",
    "Fit and transform the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "Next, fit a fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1`. Find the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "978982d101ec080d93ae0fbceb856df2",
     "grade": false,
     "grade_id": "cell-d6823cd2d4b78595",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.991545422134696"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "def answer_three():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    vect = CountVectorizer().fit(X_train)\n",
    "    X_train_vect = vect.transform(X_train)\n",
    "    X_test_vect = vect.transform(X_test)\n",
    "    mnb = MultinomialNB(alpha=0.1).fit(X_train_vect,y_train)\n",
    "    y_pred = mnb.predict(X_test_vect)\n",
    "    predictions = mnb.predict_proba(X_test_vect)\n",
    "    auc_score = roc_auc_score(y_test, predictions[:,1])\n",
    "    # raise NotImplementedError()\n",
    "    return auc_score\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1e5d06cc86ab90502c72075822f243aa",
     "grade": true,
     "grade_id": "cell-c9dd3c3ec0b63773",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "18f455d16422cc21ef23082784d54d23",
     "grade": false,
     "grade_id": "cell-881103829277a63e",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 4\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer with default parameters. The transformed data will be a compressed sparse row matrix where the number of rows is the number of documents in `X_train`, the number of columns is the number of features found by the vectorizer in each document, and each value in the sparse matrix is the tf-idf value. First find the **max** tf-idf value for every feature.\n",
    "\n",
    "What 20 features have the smallest tf-idf and what 20 have the largest tf-idf among the **max** tf-idf values?\n",
    "\n",
    "Put these features in two series where each series is sorted by tf-idf value. The index of the series should be the feature name, and the data should be the tf-idf.\n",
    "\n",
    "The series of 20 features with smallest tf-idfs should be sorted smallest tfidf first, the list of 20 features with largest tf-idfs should be sorted largest first. Any entries with identical tf-ids should appear in lexigraphically increasing order by their feature name in boh series. For example, if the features \"a\", \"b\", \"c\" had the tf-idfs 1.0, 0.5, 1.0 in the series with the largest tf-idfs, then they should occur in the returned result in the order \"a\", \"c\", \"b\" with values 1.0, 1.0, 0.5.\n",
    "\n",
    "*This function should return a tuple of two series\n",
    "`(smallest tf-idfs series, largest tf-idfs series)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "398b7634e65c7fdec4e98f8966139517",
     "grade": false,
     "grade_id": "cell-76cb5ae962a7a6f5",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(feature\n",
       " aaniye          0.074475\n",
       " athletic        0.074475\n",
       " chef            0.074475\n",
       " companion       0.074475\n",
       " courageous      0.074475\n",
       " dependable      0.074475\n",
       " determined      0.074475\n",
       " exterminator    0.074475\n",
       " healer          0.074475\n",
       " listener        0.074475\n",
       " organizer       0.074475\n",
       " pest            0.074475\n",
       " psychiatrist    0.074475\n",
       " psychologist    0.074475\n",
       " pudunga         0.074475\n",
       " stylist         0.074475\n",
       " sympathetic     0.074475\n",
       " venaam          0.074475\n",
       " afternoons      0.091250\n",
       " approaching     0.091250\n",
       " Name: tf-idf, dtype: float64,\n",
       " feature\n",
       " 146tf150p    1.000000\n",
       " 645          1.000000\n",
       " anything     1.000000\n",
       " anytime      1.000000\n",
       " beerage      1.000000\n",
       " done         1.000000\n",
       " er           1.000000\n",
       " havent       1.000000\n",
       " home         1.000000\n",
       " lei          1.000000\n",
       " nite         1.000000\n",
       " ok           1.000000\n",
       " okie         1.000000\n",
       " thank        1.000000\n",
       " thanx        1.000000\n",
       " too          1.000000\n",
       " where        1.000000\n",
       " yup          1.000000\n",
       " tick         0.980166\n",
       " blank        0.932702\n",
       " Name: tf-idf, dtype: float64)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "def answer_four():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    vect = TfidfVectorizer().fit(X_train)\n",
    "    matrix = vect.transform(X_train)\n",
    "    features = vect.get_feature_names_out()\n",
    "    rows, _ = matrix.shape\n",
    "    matrix = matrix.todense()\n",
    "    df = pd.DataFrame(matrix, columns=features).T\n",
    "    df['tf-idf'] = df.max(axis=1)\n",
    "    df.reset_index(inplace=True)\n",
    "    df.rename(columns={'index':'feature'}, inplace=True)\n",
    "    df = df[['feature','tf-idf']]\n",
    "    df_high_df = df.sort_values(by=['tf-idf','feature'], ascending=[False,True])\n",
    "    df_high_df.set_index('feature', inplace=True)\n",
    "    df_high = df_high_df.squeeze()\n",
    "    df_low_df = df.sort_values(by=['tf-idf','feature'], ascending=True)\n",
    "    df_low_df.set_index('feature', inplace=True)\n",
    "    df_low = df_low_df.squeeze()\n",
    "\n",
    "    # raise NotImplementedError()\n",
    "    return df_low.iloc[:20], df_high[:20]\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "65b6827daca8dc89566fe6731234e202",
     "grade": true,
     "grade_id": "cell-09a122df96d70683",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e40c91cd89e25cecdcb123bb447dadd1",
     "grade": false,
     "grade_id": "cell-cf833ad4436ff594",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 5\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **3**.\n",
    "\n",
    "Then fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1` and compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8679a4b2f417981c7f2163a0fa692866",
     "grade": false,
     "grade_id": "cell-4f8dc958e73ebf9e",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9954968337775665"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    vect = TfidfVectorizer(min_df=3).fit(X_train)\n",
    "    X_train_vect = vect.transform(X_train)\n",
    "    X_test_vect = vect.transform(X_test)\n",
    "    mnb = MultinomialNB(alpha=0.1).fit(X_train_vect,y_train)\n",
    "    y_pred = mnb.predict(X_test_vect)\n",
    "    predictions = mnb.predict_proba(X_test_vect)\n",
    "    auc_score = roc_auc_score(y_test, predictions[:,1])\n",
    "    # raise NotImplementedError()\n",
    "    return auc_score\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bcf549a0e74e7e1818c9accf7cd6556d",
     "grade": true,
     "grade_id": "cell-09607be9c976cae0",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "01decd9375604608e5ee425fd9f57f08",
     "grade": false,
     "grade_id": "cell-8140300bc6d127ff",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 6\n",
    "\n",
    "What is the average length of documents (number of characters) for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average length not spam, average length spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "94051f9c02aab671e630b67e818fa490",
     "grade": false,
     "grade_id": "cell-9ef7d7820ba0644d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(71.02362694300518, 138.8661311914324)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    spam_data['Length'] = spam_data['text'].apply(len)\n",
    "    new_data = pd.DataFrame(np.where(spam_data['target'] == 1, spam_data['text'].apply(len), np.nan), columns=['Spam'])\n",
    "    new_data['Ham'] = np.where(spam_data['target'] == 0, spam_data['text'].apply(len), np.nan)\n",
    "    spam_ave_len = new_data['Spam'].mean()\n",
    "    ham_ave_len = new_data['Ham'].mean()\n",
    "    # raise NotImplementedError()\n",
    "    return ham_ave_len, spam_ave_len\n",
    "\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7d0200c29dd8c642e469627c9ec80d0d",
     "grade": true,
     "grade_id": "cell-3cc7f12d3457b034",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9ae20c6a5ea3577daaff67026ada729d",
     "grade": false,
     "grade_id": "cell-142fa11bdbd6c041",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "<br>\n",
    "<br>\n",
    "The following function has been provided to help you combine new features into the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d03fff196791b604effa8f3738498b59",
     "grade": false,
     "grade_id": "cell-b1b59ae1895c1675",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def add_feature(X, feature_to_add):\n",
    "    \"\"\"\n",
    "    Returns sparse feature matrix with added feature.\n",
    "    feature_to_add can also be a list of features.\n",
    "    \"\"\"\n",
    "    from scipy.sparse import csr_matrix, hstack\n",
    "    return hstack([X, csr_matrix(feature_to_add).T], 'csr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "4388967b5944a88ddb4ba6ce1ee787bc",
     "grade": false,
     "grade_id": "cell-157cfd481d53666e",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 7\n",
    "\n",
    "Fit and transform the training data X_train using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5**.\n",
    "\n",
    "Using this document-term matrix and an additional feature, **the length of document (number of characters)**, fit a Support Vector Classification model with regularization `C=10000`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*Hint: Since probability is set to false, use the model's `decision_function` on the test data when calculating the target scores to use in roc_auc_score*\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3308c695dc06f58ad0471f643e787794",
     "grade": false,
     "grade_id": "cell-8bd2ea8cadce6a40",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9963202213809143"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "def answer_seven():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    vect = TfidfVectorizer(min_df=5).fit(X_train)\n",
    "    X_train_vect = vect.transform(X_train)\n",
    "    X_test_vect = vect.transform(X_test)\n",
    "    X_lengths = spam_data['text'].apply(len)\n",
    "    X_train_lens = X_lengths.loc[X_train.index]\n",
    "    X_test_lens = X_lengths.loc[X_test.index]\n",
    "    X_train_vect = add_feature(X_train_vect,X_train_lens)\n",
    "    X_test_vect = add_feature(X_test_vect,X_test_lens)\n",
    "    svc = SVC(C=10000).fit(X_train_vect,y_train)\n",
    "    predictions = svc.decision_function(X_test_vect)\n",
    "    auc_score = roc_auc_score(y_test, predictions)\n",
    "    # raise NotImplementedError()\n",
    "    return auc_score\n",
    "\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c32d3ed51b28e5570648f91f13f78778",
     "grade": true,
     "grade_id": "cell-3627e3b7549e1a87",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5450b9d6e5473f80506df772fcdeece5",
     "grade": false,
     "grade_id": "cell-47de3c090d6cc28c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 8\n",
    "\n",
    "What is the average number of digits per document for not spam and spam documents?\n",
    "\n",
    "*Hint: Use `\\d` for digit class*\n",
    "\n",
    "*This function should return a tuple (average # digits not spam, average # digits spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f0796b63b394927cbb4f24bc1abfad05",
     "grade": false,
     "grade_id": "cell-bad49f3617c66d86",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.2992746113989637, 15.759036144578314)"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import regex as re\n",
    "def answer_eight():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    counts = spam_data['text'].str.extractall(r'(\\d{1})')\n",
    "    counts.reset_index(inplace=True)\n",
    "    counts.rename(columns={'level_0':'OrigInd',0:'counts'}, inplace=True)\n",
    "    counts.drop(columns='match', inplace=True)\n",
    "    counts = counts.groupby('OrigInd').agg({'counts':'count'})\n",
    "    spam_data['NumDigs'] = np.zeros(spam_data.shape[0])\n",
    "    spam_data.loc[counts.index,\"NumDigs\"] = counts['counts']\n",
    "    spam_data['spam_count'] = np.where(spam_data['target'] == 1, spam_data['NumDigs'], np.nan)\n",
    "    spam_data['ham_count'] = np.where(spam_data['target'] == 0, spam_data['NumDigs'], np.nan)\n",
    "    ham_ave1 = float(spam_data.agg({'ham_count':np.mean}))\n",
    "    # ham_ave2 = spam_data['ham_count'].mean()\n",
    "    spam_ave1 = float(spam_data.agg({'spam_count':np.mean}))\n",
    "    # spam_ave2 = spam_data['spam_count'].mean()\n",
    "    spam_data.drop(columns=['spam_count','ham_count'], inplace=True)\n",
    "    # raise NotImplementedError()\n",
    "    return ham_ave1, spam_ave1\n",
    "\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2443b6f2fae61333d4cb3d50ed6ee321",
     "grade": true,
     "grade_id": "cell-501bf7c435747a23",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c12a31611d9f66f872fd93daf94889a8",
     "grade": false,
     "grade_id": "cell-1ddf2c6c108f6a85",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 9\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5** and using **word n-grams from n=1 to n=3** (unigrams, bigrams, and trigrams).\n",
    "\n",
    "Using this document-term matrix and the following additional features:\n",
    "* the length of document (number of characters)\n",
    "* **number of digits per document**\n",
    "\n",
    "fit a Logistic Regression model with regularization `C=100` and `max_iter=1000`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "752f6ba55e9ea2c42030a0bef42d7bc0",
     "grade": false,
     "grade_id": "cell-bbd73da89641f28d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9972921582941445"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "def answer_nine():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    vect = TfidfVectorizer(min_df=5,ngram_range=(1,3))\n",
    "    vect.fit(X_train)\n",
    "    X_train_vect = vect.transform(X_train)\n",
    "    X_test_vect = vect.transform(X_test)\n",
    "    X_lengths = spam_data['text'].apply(len)\n",
    "    X_train_lens = X_lengths.loc[X_train.index]\n",
    "    X_test_lens = X_lengths.loc[X_test.index]\n",
    "    X_train_vect = add_feature(X_train_vect,X_train_lens)\n",
    "    X_test_vect = add_feature(X_test_vect,X_test_lens)\n",
    "    X_counts = pd.Series(np.zeros(X_lengths.shape[0],))\n",
    "    dig_cts = spam_data['text'].str.extractall(r'(\\d{1})')\n",
    "    dig_cts.reset_index(inplace=True)\n",
    "    dig_cts.rename(columns={'level_0':'OrigInd',0:'counts'}, inplace=True)\n",
    "    dig_cts = dig_cts.groupby('OrigInd').agg({'counts':'count'})\n",
    "    X_counts.loc[dig_cts.index] = dig_cts.values.T[0]\n",
    "    X_train_cts = X_counts.loc[X_train.index]\n",
    "    X_test_cts = X_counts.loc[X_test.index]\n",
    "    X_train_vect = add_feature(X_train_vect,X_train_cts)\n",
    "    X_test_vect = add_feature(X_test_vect,X_test_cts)\n",
    "    logreg = LogisticRegression(C=100, max_iter=1000)\n",
    "    logreg.fit(X_train_vect,y_train)\n",
    "    predictions = logreg.predict_proba(X_test_vect)\n",
    "    auc_score = roc_auc_score(y_test, predictions[:,1])\n",
    "    # raise NotImplementedError()\n",
    "    return auc_score\n",
    "\n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "307ca68d7f0c4af478cfaca06c59f4f8",
     "grade": true,
     "grade_id": "cell-c7d3dd647af1574e",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d3d2d857dbe6d218394aff1a073463bf",
     "grade": false,
     "grade_id": "cell-822483537e34697a",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 10\n",
    "\n",
    "What is the average number of non-word characters (anything other than a letter, digit or underscore) per document for not spam and spam documents?\n",
    "\n",
    "*Hint: Use `\\w` and `\\W` character classes*\n",
    "\n",
    "*This function should return a tuple (average # non-word characters not spam, average # non-word characters spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dcd6cd538aa54ac0ec7b03365bd74689",
     "grade": false,
     "grade_id": "cell-8d2212cb850f907d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17.29181347150259, 29.041499330655956)"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_ten():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    counts = spam_data['text'].str.extractall(r'(\\W{1})')\n",
    "    counts.reset_index(inplace=True)\n",
    "    counts.rename(columns={'level_0':'OrigInd',0:'counts'}, inplace=True)\n",
    "    counts.drop(columns='match', inplace=True)\n",
    "    counts = counts.groupby('OrigInd').agg({'counts':'count'})\n",
    "    spam_data['NumNonWordC'] = np.zeros(spam_data.shape[0])\n",
    "    spam_data.loc[counts.index,\"NumNonWordC\"] = counts['counts']\n",
    "    spam_data['spam_count'] = np.where(spam_data['target'] == 1, spam_data['NumNonWordC'], np.nan)\n",
    "    spam_data['ham_count'] = np.where(spam_data['target'] == 0, spam_data['NumNonWordC'], np.nan)\n",
    "    ham_ave1 = float(spam_data.agg({'ham_count':np.mean}))\n",
    "    spam_ave1 = float(spam_data.agg({'spam_count':np.mean}))\n",
    "    spam_data.drop(columns=['spam_count','ham_count'], inplace=True)\n",
    "    # raise NotImplementedError()\n",
    "    return ham_ave1, spam_ave1\n",
    "\n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b7901a0e4f5b169e1305280bf4a7a2c8",
     "grade": true,
     "grade_id": "cell-1d6ac20393ffe9ff",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "4aad0115bf7fe21b470680010fde6246",
     "grade": false,
     "grade_id": "cell-733de6a9548fe648",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### Question 11\n",
    "\n",
    "Fit and transform the **first 2000 rows** of training data X_train using a Count Vectorizer ignoring terms that have a document frequency strictly lower than **5** and using **character n-grams from n=2 to n=5.**\n",
    "\n",
    "To tell Count Vectorizer to use character n-grams pass in `analyzer='char_wb'` which creates character n-grams only from text inside word boundaries. This should make the model more robust to spelling mistakes.\n",
    "\n",
    "Using this document-term matrix and the following additional features:\n",
    "* the length of document (number of characters)\n",
    "* number of digits per document\n",
    "* **number of non-word characters (anything other than a letter, digit or underscore.)**\n",
    "\n",
    "fit a Logistic Regression model with regularization C=100 and max_iter=1000. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "Also **find the 10 smallest and 10 largest coefficients from the model** and return them along with the AUC score in a tuple.\n",
    "\n",
    "The list of 10 smallest coefficients should be sorted smallest first, the list of 10 largest coefficients should be sorted largest first.\n",
    "\n",
    "The three features that were added to the document term matrix should have the following names should they appear in the list of coefficients:\n",
    "['length_of_doc', 'digit_count', 'non_word_char_count']\n",
    "\n",
    "*This function should return a tuple `(AUC score as a float, smallest coefs list, largest coefs list)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a5efa62b1d9cf26ce2beb28db30c1b83",
     "grade": false,
     "grade_id": "cell-1233becfd19fff26",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.997568035583926,\n",
       " ['n ', ' i', 'at', 'he', ' m', '..', 'us', 'go', ' lo', ' bu'],\n",
       " ['digit_count', 'ne', ' st', 'co', 's ', 'xt', 'lt', 'xt ', ' ne', 'der'])"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eleven():\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    vect = CountVectorizer(min_df=5, ngram_range=(2,5), analyzer='char_wb')\n",
    "    X_train_11 = X_train[:2000]\n",
    "    X_test_11 = X_test[:2000]\n",
    "    y_train_11 = y_train[:2000]\n",
    "    vect.fit(X_train_11)\n",
    "    X_train_vect = vect.transform(X_train_11)\n",
    "    X_test_vect = vect.transform(X_test_11)\n",
    "    X_train_len = spam_data['Length'].loc[X_train_11.index]\n",
    "    X_test_len = spam_data['Length'].loc[X_test_11.index]\n",
    "    X_train_digct = spam_data['NumDigs'].loc[X_train_11.index]\n",
    "    X_test_digct = spam_data['NumDigs'].loc[X_test_11.index]\n",
    "    X_train_nwct = spam_data['NumNonWordC'].loc[X_train_11.index]\n",
    "    X_test_nwct = spam_data['NumNonWordC'].loc[X_test_11.index]\n",
    "    X_train_vect = add_feature(X_train_vect,X_train_len)\n",
    "    X_train_vect = add_feature(X_train_vect,X_train_digct)\n",
    "    X_train_vect = add_feature(X_train_vect,X_train_nwct)\n",
    "    X_test_vect = add_feature(X_test_vect,X_test_len)\n",
    "    X_test_vect = add_feature(X_test_vect,X_test_digct)\n",
    "    X_test_vect = add_feature(X_test_vect,X_test_nwct)\n",
    "    logreg = LogisticRegression(C=100, max_iter=1000)\n",
    "    logreg.fit(X_train_vect,y_train_11)\n",
    "    predictions = logreg.predict_proba(X_test_vect)\n",
    "    auc_score = roc_auc_score(y_test, predictions[:,1])\n",
    "    features = np.array(vect.get_feature_names())\n",
    "    features = np.append(features, values=['length_of_doc', 'digit_count', 'non_word_char_count'])\n",
    "    sorted_coef_index = logreg.coef_[0].argsort()\n",
    "    smallest_coefs = list(features[sorted_coef_index[:10]])\n",
    "    largest_coefs = list(features[sorted_coef_index[:-11:-1]])\n",
    "    # raise NotImplementedError()\n",
    "    return auc_score, smallest_coefs, largest_coefs\n",
    "\n",
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "478c0c9b96fa606218d12419d35cb768",
     "grade": true,
     "grade_id": "cell-477ea85f5bcd7cef",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "Pn19K",
   "launcher_item_id": "y1juS",
   "part_id": "ctlgo"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}