index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 0
   },
   "source": [
    "# Natural Language Processing\n",
    "![](oprah-everyone.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 1
   },
   "source": [
    "Today we will be doing some EDA with the nltk library, and fitting machine learning models using text as predictors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 2
   },
   "outputs": [],
   "source": [
    "# Base Libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import string\n",
    "\n",
    "# NLP\n",
    "import nltk\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk import FreqDist\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "# Visualization\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Machine Learning\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 3
   },
   "source": [
    "In the cell below, we import the policy proposal by 2020 Democratic Presidential Candidates Bernie Sanders and Elizabeth Warren."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 4
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/2020_policies_feb_24.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-14T21:21:46.525652Z",
     "start_time": "2020-07-14T21:21:46.522008Z"
    },
    "index": 5
   },
   "source": [
    "**We need to do some processing to make this text usable.** \n",
    "\n",
    "In the cell below, define a function called `prepocessing` that receives a single parameter called `text`.\n",
    "\n",
    "<u><b>This function should:</b></u>\n",
    "1. Lower the text so all letters are the same case\n",
    "2. Use nltk's `word_tokenize` function to convert the string into a list of tokens.\n",
    "3. Remove stop words from the data using nltk's english stopwords corpus.\n",
    "4. Use nltk's `PortStemmer` to stem the text data\n",
    "5. Remove punctuation from the data \n",
    "    - *(You can use the [string](https://www.journaldev.com/23788/python-string-module) library for this)*\n",
    "6. Convert the list of tokens into a string\n",
    "7. Remove opening and trailing spaces, and replace all double spaces with a single space.\n",
    "8. Return the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 6
   },
   "outputs": [],
   "source": [
    "## The below code may need to be run in for\n",
    "## you to use the nltk PortStemmer\n",
    "## and word_tokenize\n",
    "\n",
    "# import nltk\n",
    "# nltk.download('stopwords')\n",
    "# nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 7
   },
   "outputs": [],
   "source": [
    "def preprocessing(text):\n",
    "    # Initialize a PortStemmer object\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Initialize nltk's stopwords\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Lower the text\n",
    "    # YOUR CODE HERE  \n",
    "    \n",
    "    # Remove punctuation\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Tokenize the no-punctuation text\n",
    "    # YOUR CODE HERE    \n",
    "    \n",
    "    # Remove stop words\n",
    "    # YOUR CODE HERE  \n",
    "    \n",
    "    # Convert the tokens into their stem\n",
    "    # YOUR CODE HERE  \n",
    "    \n",
    "    # Convert the list of words back into\n",
    "    # a string by joining each word with a space\n",
    "    # YOUR CODE HERE   \n",
    "    \n",
    "    # Remove double spaces\n",
    "    # YOUR CODE HERE   \n",
    "    \n",
    "    # Remove opening and trailing spaces\n",
    "    # YOUR CODE HERE   \n",
    "    \n",
    "    # Return the cleaned text data\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 9
   },
   "source": [
    "**For this warmup, tests are not provided.** \n",
    "\n",
    "Instead, examine the output for the following cell. \n",
    "- Was your code successful? \n",
    "- Are there words in the output that should be added to our list of stopwords?\n",
    "- Should we remove numbers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 10
   },
   "outputs": [],
   "source": [
    "preprocessing(df.policy[0])[:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 11
   },
   "source": [
    "**Let's apply our preprocessing to every policy.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 12
   },
   "outputs": [],
   "source": [
    "df.policy = df.policy.apply(preprocessing)\n",
    "\n",
    "print(df.policy[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 13
   },
   "source": [
    "Now, we can explore our text data.\n",
    "\n",
    "In the cell below define a function called `average_word_length` that receives a single parameter called `text`, and outputs the average word length.\n",
    "\n",
    "<u><b>This function should:</b></u>\n",
    "1. Split the text into a list of tokens\n",
    "2. Find the length of every word in the list\n",
    "3. Sum the word lengths and divide by the number of words in the list of tokens.\n",
    "4. Return the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 14
   },
   "outputs": [],
   "source": [
    "def average_word_length(text):\n",
    "    # Split the text\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Calculate the sum of each word length\n",
    "    # and divide by the total number of words\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "    # Return the calculation\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 16
   },
   "source": [
    "Now, we apply our function to every policy and add the output as column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 17
   },
   "outputs": [],
   "source": [
    "df['average_word_length'] = df.policy.apply(average_word_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 18
   },
   "source": [
    "Sweet let's take a look at the documents with the highest average word length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 19
   },
   "outputs": [],
   "source": [
    "df.sort_values(by='average_word_length', ascending=False).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 20
   },
   "source": [
    "An average measurement can be a bit misleading. \n",
    "\n",
    "Let's also write a function that finds the word count for a given document.\n",
    "\n",
    "In the cell below, define a function called `word_count` that receives a single `text` parameter.\n",
    "\n",
    "<u><b>This function should:</b></u>\n",
    "1. Split the text data\n",
    "2. Return the length of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 21
   },
   "outputs": [],
   "source": [
    "def word_count(text):\n",
    "    # Split the text\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Find the number of words\n",
    "    # in the split text\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 23
   },
   "source": [
    "Nice. Now we apply the function to our entire dataset, and save the output as a column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 24
   },
   "outputs": [],
   "source": [
    "df['word_count'] = df.policy.apply(word_count)\n",
    "\n",
    "df.sort_values(by='average_word_length', ascending=False).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 25
   },
   "source": [
    "Interesting. Let's take a look at the distribution for the `word_count` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 26
   },
   "outputs": [],
   "source": [
    "warren_df = df[df.candidate=='warren']\n",
    "sanders_df = df[df.candidate=='sanders']\n",
    "\n",
    "plt.figure(figsize=(15,6))\n",
    "plt.hist(warren_df.word_count, alpha=.6, label='Warren')\n",
    "plt.hist(sanders_df.word_count, alpha=.6, label='Sanders')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 27
   },
   "source": [
    "It looks like the average length of a policy is about 1,000 words.\n",
    "\n",
    "Let's print the mean and median for the `word_count` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 28
   },
   "outputs": [],
   "source": [
    "print('Mean Word Count: ',df.word_count.mean())\n",
    "print('Median Word Count: ',df.word_count.median())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 29
   },
   "source": [
    "*There are some outliers in this data that in a full data science project would would be worth exploring!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 30
   },
   "source": [
    "Let's find out what the most frequent words are for each candidate.\n",
    "\n",
    "First, we use list comprehension to create a list of token-lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 31
   },
   "outputs": [],
   "source": [
    "token_warren= [word_tokenize(policy) for policy in warren_df.policy] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 32
   },
   "source": [
    "Next, we want to create a bag of words. AKA a single list containing every token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-15T02:02:40.284467Z",
     "start_time": "2020-07-15T02:02:40.262652Z"
    },
    "index": 33
   },
   "outputs": [],
   "source": [
    "warren_bow = []\n",
    "for doc in token_warren:\n",
    "    warren_bow.extend([word.lower() for word in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 34
   },
   "source": [
    "Now, we use the `FreqDist` object to find the 10 most frequent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 35
   },
   "outputs": [],
   "source": [
    "fd_warren = FreqDist(warren_bow)\n",
    "print(fd_warren.most_common(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 36
   },
   "source": [
    "Are there any words here that should be added to our list of stopwords?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 37
   },
   "source": [
    "*In the cell below* define a function called `word_frequency` that receives a series of documents, and outputs a fitted FreqDist object.\n",
    "\n",
    "<u><b>This function should be</b></u> a generalized version of the code we just wrote, only instead of printing out the most frequent words, the function should return an Initialized `FreqDist` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 38
   },
   "outputs": [],
   "source": [
    "def word_frequency(documents):\n",
    "    # Tokenize each of the documents\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Create an empty list\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Loop over each tokenized document\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "        # Add all of the tokens to the empty list\n",
    "        # YOUR CODE HERE\n",
    "        \n",
    "    # Initialize a FreqDist object\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    # Return the FreqDist object\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 40
   },
   "source": [
    "Now, we can feed all of sanders policies into our `word_frequency` functions and receive a fitted `FreqDist` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 41
   },
   "outputs": [],
   "source": [
    "fd_sanders = word_frequency(sanders_df.policy)\n",
    "fd_sanders.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 42
   },
   "source": [
    "`FreqDist` objects come with a handy `.plot` method :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-15T01:53:50.258743Z",
     "start_time": "2020-07-15T01:53:50.102070Z"
    },
    "index": 43
   },
   "outputs": [],
   "source": [
    "fd_sanders.plot(10, title='Bernie Sanders - Most Common Words');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 44
   },
   "outputs": [],
   "source": [
    "fd_warren.plot(10, title='Elizabeth Warren - Most Common Words');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 45,
    "tags": []
   },
   "source": [
    "## Classification\n",
    "\n",
    "It looks like there are some more words we could probably add as stopwords. This is a pretty common iteration that is seen in Natural Language projects. It's pretty typical to  drop initial stopwords, evaluate the frequency distribution of the cleaned text, fit models to the text, and evaluate what words are most important/most common. Depending on your modeling goal, it can oftentimes take several iterations to ensure that your data does not contain obvious indicators for your target. For instance, in this data it would probably be a good idea to remove the names of the candidate from the text if we really wanted to see how a model differentiates between the two candidates based on the content of their policies.\n",
    "\n",
    "Let's see how many policies for each candidate mention the candidate by name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 46
   },
   "outputs": [],
   "source": [
    "warren_perc = warren_df[warren_df.policy.str.contains('warren')].shape[0]/warren_df.shape[0]\n",
    "sanders_perc = sanders_df[sanders_df.policy.str.contains('berni')].shape[0]/sanders_df.shape[0]\n",
    "string_template = '{} percent: {:.2%}'\n",
    "print(string_template.format('Sanders', sanders_perc))\n",
    "print(string_template.format('Warren', warren_perc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 47
   },
   "source": [
    "Let's see if we can remove references to the candidates themselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 48
   },
   "outputs": [],
   "source": [
    "# Helper function to remove specific words from the dataset\n",
    "def remove_new_stopwords(text, words):\n",
    "    new_text = str(text)\n",
    "    for word in words:\n",
    "        new_text = new_text.replace(word, '')\n",
    "    return new_text\n",
    "\n",
    "# Remove the names of the candidates from the policies\n",
    "warren_df = warren_df.assign(policy = warren_df.policy.apply(lambda x: remove_new_stopwords(x, ['warren', 'elizabeth'])))\n",
    "sanders_df = sanders_df.assign(policy = sanders_df.policy.apply(lambda x: remove_new_stopwords(x, ['berni', 'sander'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 49
   },
   "source": [
    "The percentages should now be at 0%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 50
   },
   "outputs": [],
   "source": [
    "warren_perc = warren_df[warren_df.policy.str.contains('warren')].shape[0]/warren_df.shape[0]\n",
    "sanders_perc = sanders_df[sanders_df.policy.str.contains('berni')].shape[0]/sanders_df.shape[0]\n",
    "string_template = '{} percent: {:.2%}'\n",
    "print(string_template.format('Sanders', sanders_perc))\n",
    "print(string_template.format('Warren', warren_perc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 51
   },
   "source": [
    "Let's concatenate these two tables together and put together a target columm for modeling.\n",
    "\n",
    "In this case, we will create a target column that indicates the name of the candidate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 52
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "# Concatenate the two datasets\n",
    "modeling_data = pd.concat([warren_df, sanders_df])\n",
    "\n",
    "# Fit a label encode to the column \n",
    "# indicating the name of the candidate\n",
    "target_encoder = LabelEncoder()\n",
    "\n",
    "# Transform to candidate column to an array of [0,1]\n",
    "target = target_encoder.fit_transform(modeling_data.candidate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 53
   },
   "source": [
    "Now that we have our target column, let's create a train test split of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 54
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "                    \n",
    "X_train, X_test, y_train, y_test = train_test_split(modeling_data[['policy']], # Isolating the policy column\n",
    "                                                    target, random_state=2021)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 55
   },
   "source": [
    "Good, now let's create some pipelines for different tokenization strategies.\n",
    "\n",
    "In the cell below, import `CountVectorizer` and `TfidfVectorizer` from sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 56
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 58
   },
   "source": [
    "In the cell below, initialize two vectorizers using the following variable names\n",
    "1. `count`\n",
    "2. `tfidf`\n",
    "\n",
    "> Small note: Please take a second to notice that both of these vectorizors have a `stop_words` hyperparameter! Using this hyperparameter, we can pass in a list of stopwords to the the vectorizer and the vecotrizer will vectorize *and* remove stopwords all at once. This is not important to us, given that we have already removed stopwords, but it is important to recognize that this can dramatically streamline your preprocessing with text data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 59
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 61
   },
   "source": [
    "Now we will create a dictionary containing different pipelines for each of our vectorization strategies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 62
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "random_state = 2021\n",
    "\n",
    "models = {'lr_count': make_pipeline(count, LogisticRegression(random_state=random_state)),\n",
    "          'dt_count': make_pipeline(count, DecisionTreeClassifier(random_state=random_state)),\n",
    "          'rf_count': make_pipeline(count, RandomForestClassifier(random_state=random_state)),\n",
    "          'lr_tfidf': make_pipeline(tfidf, LogisticRegression(random_state=random_state)),\n",
    "          'dt_tfidf': make_pipeline(tfidf, DecisionTreeClassifier(random_state=random_state)),\n",
    "          'rf_tfidf': make_pipeline(tfidf, RandomForestClassifier(random_state=random_state))}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 63
   },
   "source": [
    "And then we can run our models!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 64
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "baseline_scores = {}\n",
    "\n",
    "for model in models:\n",
    "    score = cross_val_score(models[model], X_train.iloc[:,0], y_train, scoring='f1')\n",
    "    baseline_scores[model] = score.mean()\n",
    "    \n",
    "baseline_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 65
   },
   "source": [
    "It looks like our best performing model is the RandomForest model using tfidf vectorization. Let's see how this modeling is doing.\n",
    "\n",
    "To do that, we will pull the model out of our models dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 66
   },
   "outputs": [],
   "source": [
    "rf_pipeline = models['rf_tfidf']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 67
   },
   "source": [
    "Now fit the model to the training data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 68
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 70
   },
   "source": [
    "Let's inspect the confusion matrix for our two sets of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 71
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import plot_confusion_matrix\n",
    "\n",
    "fig, ax = plt.subplots(1,2, figsize=(15,6))\n",
    "labels = target_encoder.inverse_transform([0,1])\n",
    "\n",
    "plot_confusion_matrix(rf_pipeline, X_train.iloc[:,0], y_train, ax=ax[0], display_labels=labels)\n",
    "ax[0].set_title('Training Data')\n",
    "\n",
    "plot_confusion_matrix(rf_pipeline, X_test.iloc[:,0], y_test, ax=ax[1], display_labels=labels)\n",
    "ax[1].set_title('Testing Data');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 72
   },
   "source": [
    "100% accuracy on our training data and then it looks like we're predicting Bernie Sanders with about a 50% recall score.\n",
    "\n",
    "Let's inspect what features the model is using for prediction by using `permutation_importance`. Because we used a pipeline, we will need pull out the individual objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 73
   },
   "outputs": [],
   "source": [
    "# the fit tfidf vectorizer\n",
    "transformer = rf_model.steps[0][-1]\n",
    "# the fit random forest model\n",
    "rf_model = rf_model.steps[-1][-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 74
   },
   "source": [
    "Next we will import `permutation_importance` from sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 75
   },
   "outputs": [],
   "source": [
    "from sklearn.inspection import permutation_importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 76
   },
   "source": [
    "Transform our our testing data with the fit tfidf vectorizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 77
   },
   "outputs": [],
   "source": [
    "X_inspect = transformer.transform(X_test.iloc[:,0]).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 78
   },
   "source": [
    "And pass our model, the transformed data, and the target into `permutation_importance`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 79
   },
   "outputs": [],
   "source": [
    "importance = permutation_importance(rf_model, X_inspect, y_test, random_state=2021, scoring='f1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 80
   },
   "source": [
    "The cell above will take a moment to run, so while it runs, we may as well talk about what `permutation_importance` is doing. \n",
    "\n",
    "Researchers have found that the feature importances given from `.get_feature_importance` returns bias information that does not accurately reflect how how much predictive information a feature provides to the model. You can read more about this [here](https://explained.ai/rf-importance/#:~:text=Permutation%20importance,-Breiman%20and%20Cutler&text=Permute%20the%20column%20values%20of,caused%20by%20permuting%20the%20column.). The recommended solution to this problem is to use `permutation importance`. **Permutation Importance** loops over every feature in your dataset and for each iteration will randomly shuffle the data in a feature's column. By doing so, the relationship between the target and the feature is severed. Once this has been done for every feature, features are given a weight based on how poorly the model did when the feature's data was scrambled. If the model did a lot worse, that suggests that model really needs that feature, thus it is importance. If the model did exactly the same, then the feature is marked as unimportant.\n",
    "\n",
    "Let's take a look at what words were considered most importance by our Random Forest model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 81
   },
   "outputs": [],
   "source": [
    "# Zip the names of the features \n",
    "# with the features permutation importance\n",
    "importance_weights = list(zip(transformer.get_feature_names(), importance['importances_mean']))\n",
    "\n",
    "# Sort the weights in descending order\n",
    "sorted(importance_weights, key=lambda x: x[1], reverse=True)[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 82
   },
   "source": [
    "Looking at the feature importances above, it looks like after the top 73 features, the remaining features are not considered importance. Let's drop them and see how the model does. We will also drop features where the word is a number, as this seems sort of nonsensical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 83
   },
   "outputs": [],
   "source": [
    "# Sort the features in descending order based on their\n",
    "# permutation importance\n",
    "top_features = sorted(importance_weights, key=lambda x: x[1], reverse=True)[:73]\n",
    "\n",
    "# Isolate the name of the feature\n",
    "top_features = [x[0] for x in top_features if not x[0].isdigit()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 84
   },
   "source": [
    "Cool cool, now we will... \n",
    "1. Transform the training and testing data with out tfidf vectorizer\n",
    "2. Set the feature names as the column for the transformed data\n",
    "3. Isolate the features with the most predictive power."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 85
   },
   "outputs": [],
   "source": [
    "# Transform the training and testing data with out tfidf vectorizer\n",
    "X_train_transformed = pd.DataFrame(tfidf.transform(X_train.iloc[:,0]).toarray())\n",
    "X_test_transformed = pd.DataFrame(tfidf.transform(X_test.iloc[:,0]).toarray())\n",
    "\n",
    "# Set the feature names as the column for the transformed data\n",
    "X_train_transformed.columns = tfidf.get_feature_names()\n",
    "X_test_transformed.columns = tfidf.get_feature_names()\n",
    "\n",
    "# Isolate the features with the most predictive power.\n",
    "X_train_top_features = X_train_transformed[top_features]\n",
    "X_test_top_features = X_test_transformed[top_features]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 86
   },
   "source": [
    "And then we fit the model to our filtered training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 87
   },
   "outputs": [],
   "source": [
    "rf_model.fit(X_train_top_features, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 88
   },
   "source": [
    "And now let's plot confusion matrices for both data splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 89
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,2, figsize=(15,6))\n",
    "\n",
    "plot_confusion_matrix(estimator, X_train_top_features, y_train, ax=ax[0])\n",
    "plot_confusion_matrix(estimator, X_test_top_features, y_test, ax=ax[1])\n",
    "ax[0].set_title('Training')\n",
    "ax[1].set_title('Testing');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 90
   },
   "source": [
    "Much better performance! The last thing to note about feature importance is that it tells us absolutely nothing about the *relationship* of the feature, other than \"it is informative\". For example, from this modeling process we have learned that the word \"Moratorium\" contains predictive information, but we have no idea whether or not \"Moratorium\" is predictive of Warren or if it predictive of Sanders. \n",
    "\n",
    "A first step in analyzing our feature importances is to visualize their relationship with the target column\n",
    "\n",
    "Let's visualize the percentage of policies that contain the top 25 words for each candidate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "index": 91
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(5,5, figsize=(20,20))\n",
    "\n",
    "for idx in range(25):\n",
    "    word = top_features[idx]\n",
    "    row, col = idx//5, idx%5\n",
    "    ax = axes[row, col]\n",
    "    w_count = warren_df[warren_df.policy.str.contains(word)].shape[0]/warren_df.shape[0]\n",
    "    s_count = sanders_df[sanders_df.policy.str.contains(word)].shape[0]/sanders_df.shape[0]\n",
    "    ax.bar(['Warren', 'Sanders'], [w_count, s_count], color=['#b7e4d0','#0370cc'])\n",
    "    ax.set_title(word.title(), fontsize=20)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "index": 92
   },
   "source": [
    "If we look at the above, we can see that `Detail` is in 100% of the Sanders Policies. This is because every Sanders policy began with bullet points, and a title containing the word \"Detail\". This is likely something we would want to from our data."
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}