1-Obtain-Clean-Structure.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mic check / Introduction\n",
    "Rap music has been one of the most influential music genres for the biggest part of the last quarter century. It all started by the name of \"Rhythm and Poetry\", meaning, words mean the world here. This data analysis will try to shed more light on the way the most influential rappers, well, rap.\n",
    "\n",
    "The purpose of this project is to analyze 5 songs per rapper. We analyze 7 rappers, meaning our data will contain lyrics from 35 songs. The goal of our analysis is to make conclusions on the positivity and the subjectivity of the rappers, as well as the size of their vocabulary.\n",
    "\n",
    "This project, as hopefully every other data science project, will go through the notorious process of \"Data Cleaning\". It's boring and all that, yet it's absoulutely vital for the sanity of our results. If you put sour food in the juicer, you can't really expect not to get sour juice, right? Feed a model with bad data, it feeds you with bad results. In the world of data science, it's almost impossible to completely clean your data, yet we should always try to minimize the garbage, so we can maximize the correctness of our output.\n",
    "\n",
    "Clean data is not enough. We need properly structured data. Why? Well, algorithms have a thing for properly structured data. Organizing data in the right way makes it easier for algorithms to extract better information.\n",
    "\n",
    "It's worth noting that the choice of rappers and songs is totally subjective. You could to this analysis for any musician, author, poet or any other text content producer. Good luck with cleaning data for mumble rap lyrics though.\n",
    "\n",
    "That said, let's see the final list of steps we will go through before we actually do some really cool stuff with our data:\n",
    "\n",
    "1. Optain data\n",
    "2. Clean data\n",
    "3. Structure data\n",
    "    1. **Corpus** - collection of song lyrics from every rapper\n",
    "   \n",
    "    2. **Word Matrix** - matrix format of every word used in the Corpus\n",
    "    \n",
    "As we will notice, the \"Clean data\" and \"Structure data\" steps are dependent and could be executed in parallel.\n",
    "\n",
    "Finally, two days ago, on June 16, it was Tupac's birthday. So this is my way of saying \"Happy birthday, man\". Legends never die.\n",
    "\n",
    "## 1. Optain data\n",
    "Here is the list of rappers and songs we will optain data for:\n",
    "1. Tupac Amaru Shakur\n",
    "    1. \"Life Goes On\"\n",
    "    2. \"Unconditional Love\"\n",
    "    3. \"Changes\"\n",
    "    4. \"Until The End Of Time\"\n",
    "    5. \"Dear Mama\"\n",
    "2. The Notorious B.I.G.\n",
    "    1. \"Ten Crack Commandments\"\n",
    "    2. \"Hypnotize\"\n",
    "    3. \"Juicy\"\n",
    "    4. \"One More Chance\"\n",
    "    5. \"Big Poppa\"\n",
    "3. Snoop Doggy Dogg\n",
    "    1. \"What's My Name?\"\n",
    "    2. \"Gin And Juice\"\n",
    "    3. \"That's That Shit\"\n",
    "    4. \"The Doggfather\"\n",
    "    5. \"Eastside Party\"\n",
    "4. Nas\n",
    "    1. \"N.Y. State Of Mind\"\n",
    "    2. \"Nas Is Like\"\n",
    "    3. \"Surviving The Times\"\n",
    "    4. \"The Message\"\n",
    "    5. \"Memory Lane\"\n",
    "5. The Game\n",
    "    1. \"Where I'm From\"\n",
    "    2. \"One Night\"\n",
    "    3. \"Don't Need That Love\"\n",
    "    4. \"Laugh\"\n",
    "    5. \"Too Much\"\n",
    "6. 50 Cent\n",
    "    1. \"Hustler's Ambition\"\n",
    "    2. \"Window Shopper\"\n",
    "    3. \"If I Can't\"\n",
    "    4. \"Many Men\"\n",
    "    5. \"When It Rains It Pours\"\n",
    "7. Eminem\n",
    "    1. \"Rap God\"\n",
    "    2. \"Phenomenal\"\n",
    "    3. \"Wicked Ways\"\n",
    "    4. \"Kings Never Die\"\n",
    "    5. \"The Ringer\"\n",
    "\n",
    "We will find the lyrics on ********** and store the URLs in variables:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will use Requests and BeatufiulSoup for scraping\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "# We will use Pickle and OS to create folders and and store files\n",
    "import pickle\n",
    "import os\n",
    "import shutil\n",
    "# We will use Pandas to structure and maintain our data\n",
    "import pandas\n",
    "# We will use Re and String modules when we start cleaning our data\n",
    "import re\n",
    "import string\n",
    "# We will use this module in the 'Cleaning Data' and 'Structure Data' steps\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Storing URLs to the Tupac songs lyrics\n",
    "tupac_urls = []\n",
    "\n",
    "# Storing URLs to the Biggie songs lyrics\n",
    "biggie_urls = []\n",
    "\n",
    "# Storing URLs to the Snoop songs lyrics\n",
    "snoop_urls = []\n",
    "\n",
    "# Storing URLs to the Nas songs lyrics\n",
    "nas_urls = []\n",
    "\n",
    "# Storing URLs to the The Game songs lyrics\n",
    "game_urls = []\n",
    "\n",
    "# Storing URLs to the 50 Cent songs lyrics\n",
    "fifty_urls = []\n",
    "\n",
    "eminem_urls = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need a function that will take a URL and return the scraped text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that takes a URL as parameter, scrapes the ***** page at that URL, extracts and returns the lyrics\n",
    "def fetch_lyrics(url):\n",
    "    print(url)\n",
    "    \n",
    "    page = requests.get(url)\n",
    "    html = BeautifulSoup(page.text, \"html.parser\")\n",
    "    lyrics = html.find(\"div\", class_=\"lyrics\").get_text()\n",
    "    \n",
    "    return lyrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a function, we will use it to fetch lyrics. This could take a couple of minutes. You will see the URL of the scraped page printed out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
     
     ]
    }
   ],
   "source": [
    "# We will store the lyrics in this dictionary\n",
    "lyrics = {}\n",
    "lyrics['tupac'] = [fetch_lyrics(url) for url in tupac_urls]\n",
    "lyrics['biggie'] = [fetch_lyrics(url) for url in biggie_urls]\n",
    "lyrics['snoop'] = [fetch_lyrics(url) for url in snoop_urls]\n",
    "lyrics['nas'] = [fetch_lyrics(url) for url in nas_urls]\n",
    "lyrics['game'] = [fetch_lyrics(url) for url in game_urls]\n",
    "lyrics['fifty'] = [fetch_lyrics(url) for url in fifty_urls]\n",
    "lyrics['eminem'] = [fetch_lyrics(url) for url in eminem_urls]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could print out the lyrics collection for one rapper just to see that our scraping was successful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(lyrics['tupac'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    
    "Now that we have scraped our data, we need to organize it. We will save the lyrics collection for every rapper in a separate file using Pickle, which we imported at the start. Pickle is used in Python for object serialization. The files will be saved in a directory that we will create, if it doesn't already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the directory where we will keep the lyrics files\n",
    "if os.path.exists('lyrics'):\n",
    "    shutil.rmtree('lyrics')\n",
    "\n",
    "os.makedirs('lyrics')\n",
    "    \n",
    "rappers = ['tupac', 'biggie', 'snoop', 'nas', 'game', 'fifty', 'eminem']\n",
    "\n",
    "# Create named files\n",
    "for i, rapper in enumerate(rappers):\n",
    "    with open(\"lyrics/\" + rapper + \".txt\", \"wb\") as file:\n",
    "        pickle.dump(lyrics[rapper], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will read the data from the files and apply a few cleaning techniques to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {}\n",
    "\n",
    "for i, rapper in enumerate(rappers):\n",
    "    with open(\"lyrics/\" + rapper + \".txt\", \"rb\") as file:\n",
    "        data[rapper] = pickle.load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point you could check if data was loaded correctly. We are done with obtaining data, now we proceed to the second step, cleaning data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data['tupac']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we actually have a list of song lyrics instead of one single text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Clean Data \n",
    "The techniques used to clean textual data are called \"text pre-processing techniques\". We will use few of those, but won't go to any advanced data cleaning. We will go through the following techniques:\n",
    "1. Remove garbage characters (like \"/n\", \"[]\" and so on)\n",
    "2. Remove punctuation\n",
    "3. Remove numerical values\n",
    "4. Make all text lowercase\n",
    "5. Remove stop words\n",
    "\n",
    "First, we will structure the data in a better way. As noticed before, our data per rapper is a list of lyrics. So, we have to transform it into one single text. We will write a function that will accept a list of lyrics and will return one single text. That can only mean one thing...\n",
    "\n",
    "## 3. Structure data\n",
    "So, first we will merge our data into single text, and then we will put it in the following formats:\n",
    "- Corpus - collection of song lyrics from every rapper\n",
    "- Word Matrix - matrix format of every word used in the Corpus\n",
    "\n",
    "We will move back and forth between cleaning and structuring data, as we can't efficiently clean badly structured data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def merge_lyrics(list_of_lyrics):\n",
    "    merged_text = ' '.join(list_of_lyrics)\n",
    "    return merged_text\n",
    "\n",
    "merged_data = {key: [merge_lyrics(value)] for (key, value) in data.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it out. We will print the data before and after executing this function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data['tupac']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#merged_data['tupac']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's better. Our data in the moment is in the dictionary format. We will put it in pandas dataframe, or we will re-structure it. \n",
    "\n",
    "We kinda already have our data in the Corpus format, now we will create Pandas dataframe from it. Why? Because it will be easier to clean our data if it is structured in a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas.set_option('max_colwidth',150)\n",
    "\n",
    "lyrics_dataframe = pandas.DataFrame.from_dict(merged_data).transpose()\n",
    "lyrics_dataframe.columns = ['lyrics']\n",
    "lyrics_dataframe = lyrics_dataframe.sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will check to see if our dataframe is what we expect it to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lyrics_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we could take a look at Tupac's (or any other rapper's) data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lyrics_dataframe.lyrics.loc['eminem']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our data restructured we move on to the second data cleaning technique that we will apply: remove garbage characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_garbage_characters(lyrics):\n",
    "    # remove data in brackets\n",
    "    lyrics = re.sub('\\[.*?\\]', '', lyrics)\n",
    "    # remove multiple \"\\n\" characters with ''\n",
    "    lyrics = re.sub('\\n{2,}', '', lyrics)\n",
    "    # remove single \"\\n\" character with ' '\n",
    "    lyrics = re.sub('\\n', ' ', lyrics)\n",
    "    return lyrics\n",
    "\n",
    "garbage_characters_removal_technique = lambda x: remove_garbage_characters(x)\n",
    "\n",
    "clean_data = pandas.DataFrame(lyrics_dataframe.lyrics.apply(garbage_characters_removal_technique))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could check to see if there are any changes in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean_data.lyrics.loc['eminem']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We move on to the third data cleaning step: remove all punctuation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_punctuation(lyrics):\n",
    "    lyrics = re.sub('[%s]' % re.escape(string.punctuation), '', lyrics)\n",
    "    lyrics = re.sub('[‘’“”…]', '', lyrics)\n",
    "    return lyrics\n",
    "\n",
    "remove_punctuation_technique = lambda x: remove_punctuation(x)\n",
    "\n",
    "clean_data = pandas.DataFrame(clean_data.lyrics.apply(remove_punctuation_technique))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See where we are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean_data.lyrics.loc['tupac']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's beautiful. If you run the last line of code, you will notice that we still have numbers in the text. We don't need numbers, so let's remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_numerical_values(lyrics):\n",
    "    lyrics = re.sub('\\w*\\d\\w*', '', lyrics)\n",
    "    return lyrics\n",
    "\n",
    "remove_numerical_values_technique = lambda x: remove_numerical_values(x)\n",
    "\n",
    "clean_data = pandas.DataFrame(clean_data.lyrics.apply(remove_numerical_values_technique))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean_data.lyrics.loc['tupac']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you see any numbers? I don't. So we are doing good. Let's make all text lowercase now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_to_lowercase(lyrics):\n",
    "    lyrics = lyrics.lower()\n",
    "    return lyrics\n",
    "\n",
    "transform_to_lowercase_technique = lambda x: transform_to_lowercase(x)\n",
    "\n",
    "clean_data = pandas.DataFrame(clean_data.lyrics.apply(transform_to_lowercase_technique))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean_data.lyrics.loc['eminem']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now add the full artist name of the rappers in our dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lyrics</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>biggie</th>\n",
       "      <td>the ten crack commandments what nigga cant tell me nothing about this coke cant tell me nothing about this crack this weed for my hustlin...</td>\n",
       "      <td>The Notorious B.I.G.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eminem</th>\n",
       "      <td>look i was gonna go easy on you not to hurt your feelings but im only going to get this one chance somethings wrong i can feel it six minutes six ...</td>\n",
       "      <td>Eminem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fifty</th>\n",
       "      <td>man we gotta go get somethin to eat man im hungry as a motherfucker ayo man damn whats takin homie so long son  calm down he coming  gunshots ahh ...</td>\n",
       "      <td>50 Cent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>game</th>\n",
       "      <td>im a bldoubleod been on songs with sndoubleop inside a ferrari with the dre run up i let it sing like nate dodoubleg walk through  mile gunits on ...</td>\n",
       "      <td>The Game</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nas</th>\n",
       "      <td>but thats the whole tragic point my friends what would i do if i could suddenly feel and to know once again that what i feel is real i could cry i...</td>\n",
       "      <td>Nas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>snoop</th>\n",
       "      <td>ugh hahaha im serious nigga one of yall niggas got some bad motherfuckin breath oh man aye baby aye baby shit aye baby get some bubblegum in this ...</td>\n",
       "      <td>Snoop Doggy Dogg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tupac</th>\n",
       "      <td>how many brothers fell victim to the streets rest in peace young nigga theres a heaven for a g be a lie if i told you that i never thought of deat...</td>\n",
       "      <td>Tupac Amaru</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                       lyrics  \\\n",
       "biggie           the ten crack commandments what nigga cant tell me nothing about this coke cant tell me nothing about this crack this weed for my hustlin...   \n",
       "eminem  look i was gonna go easy on you not to hurt your feelings but im only going to get this one chance somethings wrong i can feel it six minutes six ...   \n",
       "fifty   man we gotta go get somethin to eat man im hungry as a motherfucker ayo man damn whats takin homie so long son  calm down he coming  gunshots ahh ...   \n",
       "game    im a bldoubleod been on songs with sndoubleop inside a ferrari with the dre run up i let it sing like nate dodoubleg walk through  mile gunits on ...   \n",
       "nas     but thats the whole tragic point my friends what would i do if i could suddenly feel and to know once again that what i feel is real i could cry i...   \n",
       "snoop   ugh hahaha im serious nigga one of yall niggas got some bad motherfuckin breath oh man aye baby aye baby shit aye baby get some bubblegum in this ...   \n",
       "tupac   how many brothers fell victim to the streets rest in peace young nigga theres a heaven for a g be a lie if i told you that i never thought of deat...   \n",
       "\n",
       "                        name  \n",
       "biggie  The Notorious B.I.G.  \n",
       "eminem                Eminem  \n",
       "fifty                50 Cent  \n",
       "game                The Game  \n",
       "nas                      Nas  \n",
       "snoop       Snoop Doggy Dogg  \n",
       "tupac            Tupac Amaru  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rappers_full_names = ['The Notorious B.I.G.', 'Eminem', '50 Cent', 'The Game', 'Nas', 'Snoop Doggy Dogg', 'Tupac Amaru']\n",
    "\n",
    "clean_data['name'] = rappers_full_names\n",
    "\n",
    "clean_data.to_pickle(\"corpus.pkl\")\n",
    "\n",
    "clean_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's move on to the last step of our data cleaning process: remove stop words. But, before we do that, we have to organize our data in a word matrix. We will use CountVectorizer to tokenize the lyrics and we can remove the stop words with the same module. Every row will contain data for a rapper and every column will be a different word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer(stop_words='english')\n",
    "tokenized_data = cv.fit_transform(clean_data.lyrics)\n",
    "word_matrix = pandas.DataFrame(tokenized_data.toarray(), columns=cv.get_feature_names())\n",
    "word_matrix.index = clean_data.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see our word matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aaaahif</th>\n",
       "      <th>abandoned</th>\n",
       "      <th>able</th>\n",
       "      <th>abuse</th>\n",
       "      <th>ac</th>\n",
       "      <th>accident</th>\n",
       "      <th>accolades</th>\n",
       "      <th>accomplishments</th>\n",
       "      <th>account</th>\n",
       "      <th>accountant</th>\n",
       "      <th>...</th>\n",
       "      <th>yous</th>\n",
       "      <th>youthful</th>\n",
       "      <th>youve</th>\n",
       "      <th>youyeah</th>\n",
       "      <th>zags</th>\n",
       "      <th>zeros</th>\n",
       "      <th>zod</th>\n",
       "      <th>zombie</th>\n",
       "      <th>zone</th>\n",
       "      <th>zé</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>biggie</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eminem</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fifty</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>game</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nas</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>snoop</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tupac</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7 rows × 3890 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        aaaahif  abandoned  able  abuse  ac  accident  accolades  \\\n",
       "biggie        0          0     0      0   1         0          0   \n",
       "eminem        0          0     2      1   1         1          2   \n",
       "fifty         0          0     0      0   0         0          0   \n",
       "game          1          1     0      0   0         0          0   \n",
       "nas           0          0     0      0   0         0          0   \n",
       "snoop         0          0     0      0   0         0          0   \n",
       "tupac         0          0     0      0   0         0          0   \n",
       "\n",
       "        accomplishments  account  accountant  ...  yous  youthful  youve  \\\n",
       "biggie                0        0           1  ...     3         0      0   \n",
       "eminem                1        1           0  ...     0         1      1   \n",
       "fifty                 0        0           0  ...    12         0      0   \n",
       "game                  0        0           0  ...     0         0      0   \n",
       "nas                   0        0           0  ...     0         0      0   \n",
       "snoop                 0        0           0  ...     0         0      0   \n",
       "tupac                 0        0           0  ...     0         0      2   \n",
       "\n",
       "        youyeah  zags  zeros  zod  zombie  zone  zé  \n",
       "biggie        0     0      0    0       0     0   0  \n",
       "eminem        0     0      0    1       1     1   0  \n",
       "fifty         1     0      0    0       0     0   0  \n",
       "game          0     0      0    0       0     0   0  \n",
       "nas           0     0      1    0       0     0   1  \n",
       "snoop         0     1      0    0       0     1   0  \n",
       "tupac         0     0      0    0       0     1   0  \n",
       "\n",
       "[7 rows x 3890 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will save all data to files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_matrix.to_pickle(\"word_matrix.pkl\")\n",
    "\n",
    "clean_data.to_pickle('clean_data.pkl')\n",
    "pickle.dump(cv, open(\"cv.pkl\", \"wb\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}