code/Poll_data_2016.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Reading-data-from-FiveThirtyEight.com\" data-toc-modified-id=\"Reading-data-from-FiveThirtyEight.com-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Reading data from FiveThirtyEight.com</a></span><ul class=\"toc-item\"><li><span><a href=\"#Class-objects\" data-toc-modified-id=\"Class-objects-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Class objects</a></span></li><li><span><a href=\"#Data\" data-toc-modified-id=\"Data-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Data</a></span></li><li><span><a href=\"#References\" data-toc-modified-id=\"References-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>References</a></span></li></ul></li><li><span><a href=\"#Establish-programming-components\" data-toc-modified-id=\"Establish-programming-components-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Establish programming components</a></span><ul class=\"toc-item\"><li><span><a href=\"#Import-libraries\" data-toc-modified-id=\"Import-libraries-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Import libraries</a></span></li><li><span><a href=\"#Establish-classes\" data-toc-modified-id=\"Establish-classes-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Establish classes</a></span></li><li><span><a href=\"#Establish-functions\" data-toc-modified-id=\"Establish-functions-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Establish functions</a></span></li></ul></li><li><span><a href=\"#Scrape-data\" data-toc-modified-id=\"Scrape-data-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Scrape data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Instantiate-objects\" data-toc-modified-id=\"Instantiate-objects-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Instantiate objects</a></span></li><li><span><a href=\"#Get-candidates,-poll-data-and-create-a-DataFrame\" data-toc-modified-id=\"Get-candidates,-poll-data-and-create-a-DataFrame-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Get candidates, poll data and create a DataFrame</a></span></li><li><span><a href=\"#Assign-dates\" data-toc-modified-id=\"Assign-dates-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Assign dates</a></span></li><li><span><a href=\"#Create-a-modeling-DataFrame\" data-toc-modified-id=\"Create-a-modeling-DataFrame-3.4\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Create a modeling DataFrame</a></span></li><li><span><a href=\"#Save-to-a-CSV-file\" data-toc-modified-id=\"Save-to-a-CSV-file-3.5\"><span class=\"toc-item-num\">3.5&nbsp;&nbsp;</span>Save to a CSV file</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-28T19:01:21.315970Z",
     "start_time": "2019-04-28T19:01:18.297352Z"
    }
   },
   "source": [
    "# Reading data from FiveThirtyEight.com\n",
    "\n",
    "This module runs a script to scrape poll data from a specific page at FiveThirtyEight.com.  It stores these data in a DataFrame and outputs them to a comma separated value (CSV) file.  \n",
    "\n",
    "\n",
    "## Class objects\n",
    "\n",
    "To retrieve the poll data, this code employs a class object designed to interact with this particular FiveThirtyEight [page](https://projects.fivethirtyeight.com/election-2016/national-primary-polls/republican/) providing presidential primary polling data from the 2016 presidential election.  The class object is Scrape538PollData and it is developed to scrape and store in memory poll results.  It is built using the [Selenium](https://www.seleniumhq.org/) Web Driver application programming interface (API) which can handle Javascript web pages and the [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) API for extracting data from HTML objects. \n",
    "\n",
    "   * Scrape538PollData attributes:\n",
    "       * url - the URL of the FiveThirtyPage \n",
    "       * chrome_driver - the location on the local machine of the Chrome driver needed by Selenium\n",
    "       * status_code_ - set to None unless there is an error in retrieving the page\n",
    "       * html_ - the HTML format of the web page\n",
    "       * soup_ - A Beautiful Soup object representation of the page (using 'xlml')\n",
    "       * candidates_ - a list of candidates\n",
    "       * polls_ - a list of dictionaries containing individual poll data\n",
    "\n",
    "   * Scrape538PollData methods:\n",
    "       * collect_page_data() - collect page data and update the html_ and soup_ attributes\n",
    "       * extract_polls() - extract poll data from the page\n",
    "    \n",
    "\n",
    "\n",
    "## Data\n",
    "\n",
    "The data below represent the FiveThirtyEight polling data as of April 28, 2019.  These elements are stored in the raw_poll_data_2016.csv file.\n",
    "\n",
    "A new DataFrame is created using the end_time as the key index.  In this case, the poll data are grouped by end_date with any polls having the same end date being averaged.  In this process, the string and categorical variables are dropped leaving only the candidate poll results and the end_date index column.  These data are stored in the pure_poll_data_2016.csv file.\n",
    "\n",
    "|Column        |Description    |\n",
    "|-----------------|--------------------|\n",
    "|Bush  | poll results|\n",
    "|Carson| poll results|\n",
    "|Christie| poll results|\n",
    "|Cruz  | poll results|\n",
    "|Fiorina| poll results|\n",
    "|Huckabee| poll results|\n",
    "|Kasich  | poll results|\n",
    "|Paul| poll results|\n",
    "|Rubio| poll results|\n",
    "|Santorum  | poll results|\n",
    "|Trump| poll results|\n",
    "|dates| date poll was conducted|\n",
    "|leader| poll leader |\n",
    "|pollster| Polling organization|\n",
    "|pollster_url| URL for the polling organization|\n",
    "|sample| size of the poll |\n",
    "|weight| FiveThirtyEight weighting score |\n",
    "|start_time| datetime object for the start of the poll |\n",
    "|end_time| datetime object for the end of the poll |\n",
    "\n",
    "\n",
    "## References\n",
    "\n",
    "- https://stackoverflow.com/questions/48477688/scrape-page-with-load-more-results-button\n",
    "- https://projects.fivethirtyeight.com/election-2016/national-primary-polls/republican/\n",
    "- https://songhuiming.github.io/pages/2017/04/02/jupyter-and-pandas-display/\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Establish programming components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:42:34.752263Z",
     "start_time": "2019-05-14T20:42:33.086986Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import libaries\n",
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup as bs\n",
    "from selenium import webdriver\n",
    "import time\n",
    "from datetime import datetime, timedelta\n",
    "import regex as re\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Establish classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:42:36.433757Z",
     "start_time": "2019-05-14T20:42:36.420867Z"
    }
   },
   "outputs": [],
   "source": [
    "class Scrape538PollData:\n",
    "    # Attributes of the data retrieval\n",
    "    url = 'https://projects.fivethirtyeight.com/election-2016/national-primary-polls/republican/'\n",
    "    chrome_driver = '/Users/stephengodfrey/anaconda3/envs/dsi/bin/chromedriver'\n",
    "    status_code_ = None\n",
    "    html_ = None\n",
    "    soup_ = None\n",
    "    candidates_ = []\n",
    "    polls_ = []\n",
    "    \n",
    "    # Initialization method\n",
    "    def __init__(self, url = None, chrome_driver = None):\n",
    "        # If url != None then reset self.url\n",
    "        if url:\n",
    "            self.url = url\n",
    "        # If chrome_driver != None reset self.chrome_driver\n",
    "        if chrome_driver:\n",
    "            self.chrome_driver = chrome_driver\n",
    "    \n",
    "    # method to collect data from posts\n",
    "    def collect_page_data(self):\n",
    "        # Set the selenium driver\n",
    "        try:\n",
    "            driver = webdriver.Chrome(self.chrome_driver)\n",
    "            driver.get(self.url)\n",
    "        except:\n",
    "            self.status_code_ = driver.error_handler.check_response\n",
    "            raise ValueError('Error retrieving the web page; see WebExceptionError for details.')\n",
    "            \n",
    "        # Find the read more polls button and click it\n",
    "        driver.find_element_by_css_selector('.more-polls').click()\n",
    "        \n",
    "        # Return the html of the new page\n",
    "        self.html_ = driver.page_source.encode('utf-8')\n",
    "        \n",
    "        # Return the soup version of the page\n",
    "        self.soup_ = bs(self.html_, 'lxml')\n",
    "\n",
    "    # method to collect data from posts\n",
    "    def extract_polls(self):\n",
    "        # Check to see that self.soup_ has data, if not run collect_page_data\n",
    "        if self.soup_ == None or len(self.soup_) == 0:\n",
    "            self.collect_page_data()\n",
    "            \n",
    "        # Find the list of candidates\n",
    "        table = self.soup_.find('table')\n",
    "        self.candidates_ = [c.text for c in table.find_all('th', {'class':'th th-rotate'})]\n",
    "\n",
    "        # Find the poll data\n",
    "        body = self.soup_.find('tbody')\n",
    "        polls = body.find_all('tr',{'class': 't-row'})\n",
    "\n",
    "        # For each poll extract information related to the poll\n",
    "        for poll in polls:\n",
    "            pl_d = {}\n",
    "            pl_d['dates'] = poll.find('td', {'class':'t-dates'}).text\n",
    "            pl_d['pollster_url'] = poll.find('a', href = True)\n",
    "            pl_d['pollster'] = poll.find('td', {'class': 't-pollster t-left-margin'}).text\n",
    "            pl_d['sample'] = poll.find('td', {'class': 't-sample t-left-margin t-right-align only-full'}).text\n",
    "            pl_d['weight'] = poll.find('td', \\\n",
    "                                        {'class': 't-weight t-left-margin t-right-margin double-l-margin t-right-border-dark'}).text\n",
    "            try:\n",
    "                pl_d['leader'] = poll.find('td', \\\n",
    "                                        {'class':'t-leader t-left-margin t-right-margin only-full color-text-rep'}).text\n",
    "            except:\n",
    "                pl_d['leader'] = ''\n",
    "\n",
    "            # Get the odds for each candidate except for the last candidate in the table\n",
    "            for i, odds in enumerate(poll.find_all('td', {'class':'t-center-align td-cand-odds td-block t-right-border'})):\n",
    "                # This tag is present if a value exists in the poll for that candidate\n",
    "                if odds.find('div', {'class':'t-cand-odds heat-map-blocks'}):\n",
    "                    pl_d[self.candidates_[i]] = float(odds.text.replace('%','').strip())/100\n",
    "#                 else:\n",
    "#                     pl_d[self.candidates_[i]] = 0\n",
    "\n",
    "            # Get the odds for the last candidate \n",
    "            odds = poll.find('td', {'class':'t-center-align td-cand-odds td-block'})\n",
    "            # This tag is present if a value exists in the poll for that candidate\n",
    "            if odds.find('div', {'class':'t-cand-odds heat-map-blocks'}):\n",
    "                pl_d[self.candidates_[len(self.candidates_) - 1]] = float(odds.text.replace('%','').strip())/100\n",
    "#             else:\n",
    "#                 pl_d[self.candidates_[len(self.candidates_) - 1]] = 0\n",
    "\n",
    "            self.polls_.append(pl_d)\n",
    "  \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Establish functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:42:37.979559Z",
     "start_time": "2019-05-14T20:42:37.973653Z"
    }
   },
   "outputs": [],
   "source": [
    "# Walk through DataFrame to assign dates\n",
    "def assign_dates(df = None):\n",
    "    # Loop through each row in this DataFrame to create \n",
    "    #  a datetime object for the beginnging and end of the poll\n",
    "    start_year = '2016'\n",
    "    end_year = '2016'\n",
    "    \n",
    "    for idx in df.index:\n",
    "        # Find the months and set them equal to start and end month\n",
    "        months = re.findall(r'[\\w]{3}',df.loc[idx,'dates'])\n",
    "        if len(months)==1:\n",
    "            start_month = months[0]\n",
    "            end_month = months[0]\n",
    "        else:\n",
    "            start_month = months[0]\n",
    "            end_month = months[1]\n",
    "            \n",
    "        # Find the months and set them equal to start and end month            \n",
    "        dates = re.findall(r'[\\d]{1,}',df.loc[idx,'dates'])\n",
    "        if len(dates)==1:\n",
    "            start_date = dates[0]\n",
    "            end_date = dates[0]\n",
    "        else:\n",
    "            start_date = dates[0]\n",
    "            end_date = dates[1]       \n",
    "        \n",
    "        # Figure out if the year needs to be changed\n",
    "        if start_month=='Dec' and end_month=='Jan':\n",
    "            start_year = '2015'\n",
    "            end_year = '2016'\n",
    "        elif start_month=='Jan' and end_month=='Jan':\n",
    "            start_year = '2016'\n",
    "            end_year = '2016'           \n",
    "        elif start_month=='Dec' and end_month=='Dec':\n",
    "            start_year = '2015'\n",
    "            end_year = '2015'\n",
    "        \n",
    "        start_time=start_year+' '+start_month+' '+start_date\n",
    "        end_time=end_year+' '+end_month+' '+end_date\n",
    "        \n",
    "        dt_start = datetime.strptime(start_time,'%Y %b %d')\n",
    "        dt_end = datetime.strptime(end_time,'%Y %b %d')\n",
    "        \n",
    "        df.loc[idx,'start_time'] = dt_start \n",
    "        df.loc[idx,'end_time'] = dt_end \n",
    "        \n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrape data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instantiate objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:42:55.157992Z",
     "start_time": "2019-05-14T20:42:40.801911Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Instantiate and a poll object and get poll data\n",
    "rep_2016_polls = Scrape538PollData()\n",
    "rep_2016_polls.extract_polls()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get candidates, poll data and create a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:43:06.578404Z",
     "start_time": "2019-05-14T20:43:06.526410Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Trump', 'Kasich', 'Cruz', 'Rubio', 'Carson', 'Bush', 'Christie', 'Fiorina', 'Santorum', 'Paul', 'Huckabee']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bush</th>\n",
       "      <th>Carson</th>\n",
       "      <th>Christie</th>\n",
       "      <th>Cruz</th>\n",
       "      <th>Fiorina</th>\n",
       "      <th>Huckabee</th>\n",
       "      <th>Kasich</th>\n",
       "      <th>Paul</th>\n",
       "      <th>Rubio</th>\n",
       "      <th>Santorum</th>\n",
       "      <th>Trump</th>\n",
       "      <th>dates</th>\n",
       "      <th>leader</th>\n",
       "      <th>pollster</th>\n",
       "      <th>pollster_url</th>\n",
       "      <th>sample</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.56</td>\n",
       "      <td>Apr. 25-May 1</td>\n",
       "      <td>Trump +34</td>\n",
       "      <td>SurveyMonkey</td>\n",
       "      <td>&lt;a href=\"http://www.nbcnews.com/politics/2016-...</td>\n",
       "      <td>3,479</td>\n",
       "      <td>1.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.49</td>\n",
       "      <td>Apr. 22-26</td>\n",
       "      <td>Trump +21</td>\n",
       "      <td>YouGov</td>\n",
       "      <td>&lt;a href=\"https://today.yougov.com/news/2016/04...</td>\n",
       "      <td>499</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.49</td>\n",
       "      <td>Apr. 28-May 1</td>\n",
       "      <td>Trump +24</td>\n",
       "      <td>Opinion Research Corporation</td>\n",
       "      <td>&lt;a href=\"https://assets.documentcloud.org/docu...</td>\n",
       "      <td>406</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.56</td>\n",
       "      <td>Apr. 29-May 2</td>\n",
       "      <td>Trump +36</td>\n",
       "      <td>Morning Consult</td>\n",
       "      <td>&lt;a href=\"https://morningconsult.com/2016/05/do...</td>\n",
       "      <td>723</td>\n",
       "      <td>0.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.56</td>\n",
       "      <td>Apr. 29-May 3</td>\n",
       "      <td>Trump +30</td>\n",
       "      <td>Ipsos, online</td>\n",
       "      <td>&lt;a href=\"http://polling.reuters.com/#poll/TR13...</td>\n",
       "      <td>244</td>\n",
       "      <td>0.94</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Bush  Carson  Christie  Cruz  Fiorina  Huckabee  Kasich  Paul  Rubio  \\\n",
       "0   NaN     NaN       NaN  0.22      NaN       NaN    0.14   NaN    NaN   \n",
       "1   NaN     NaN       NaN  0.28      NaN       NaN    0.19   NaN    NaN   \n",
       "2   NaN     NaN       NaN  0.25      NaN       NaN    0.19   NaN    NaN   \n",
       "3   NaN     NaN       NaN  0.20      NaN       NaN    0.13   NaN    NaN   \n",
       "4   NaN     NaN       NaN  0.26      NaN       NaN    0.17   NaN    NaN   \n",
       "\n",
       "   Santorum  Trump           dates     leader                      pollster  \\\n",
       "0       NaN   0.56  Apr. 25-May 1   Trump +34                  SurveyMonkey   \n",
       "1       NaN   0.49     Apr. 22-26   Trump +21                        YouGov   \n",
       "2       NaN   0.49  Apr. 28-May 1   Trump +24  Opinion Research Corporation   \n",
       "3       NaN   0.56  Apr. 29-May 2   Trump +36               Morning Consult   \n",
       "4       NaN   0.56  Apr. 29-May 3   Trump +30                 Ipsos, online   \n",
       "\n",
       "                                        pollster_url  sample weight  \n",
       "0  <a href=\"http://www.nbcnews.com/politics/2016-...  3,479    1.30  \n",
       "1  <a href=\"https://today.yougov.com/news/2016/04...    499    0.99  \n",
       "2  <a href=\"https://assets.documentcloud.org/docu...    406    0.98  \n",
       "3  <a href=\"https://morningconsult.com/2016/05/do...    723    0.96  \n",
       "4  <a href=\"http://polling.reuters.com/#poll/TR13...    244    0.94  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the candidate list and put the poll data in a DataFrame\n",
    "print(rep_2016_polls.candidates_)\n",
    "df = pd.DataFrame(rep_2016_polls.polls_)\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:45:13.546306Z",
     "start_time": "2019-05-14T20:45:13.521715Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bush</th>\n",
       "      <th>Carson</th>\n",
       "      <th>Christie</th>\n",
       "      <th>Cruz</th>\n",
       "      <th>Fiorina</th>\n",
       "      <th>Huckabee</th>\n",
       "      <th>Kasich</th>\n",
       "      <th>Paul</th>\n",
       "      <th>Rubio</th>\n",
       "      <th>Santorum</th>\n",
       "      <th>Trump</th>\n",
       "      <th>dates</th>\n",
       "      <th>leader</th>\n",
       "      <th>pollster</th>\n",
       "      <th>pollster_url</th>\n",
       "      <th>sample</th>\n",
       "      <th>weight</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>665</th>\n",
       "      <td>0.16</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mar. 13-15</td>\n",
       "      <td>Bush +3*</td>\n",
       "      <td>Opinion Research Corporation</td>\n",
       "      <td>&lt;a href=\"http://i2.cdn.turner.com/cnn/2015/ima...</td>\n",
       "      <td>450</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-03-13</td>\n",
       "      <td>2015-03-15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>666</th>\n",
       "      <td>0.16</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Feb. 26-Mar. 2</td>\n",
       "      <td>Walker +2*</td>\n",
       "      <td>Quinnipiac University</td>\n",
       "      <td>&lt;a href=\"http://www.quinnipiac.edu/news-and-ev...</td>\n",
       "      <td>554</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-02-26</td>\n",
       "      <td>2015-03-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>667</th>\n",
       "      <td>0.16</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Feb. 21-23</td>\n",
       "      <td>Walker +2*</td>\n",
       "      <td>YouGov</td>\n",
       "      <td>&lt;a href=\"http://d25d2506sfb94s.cloudfront.net/...</td>\n",
       "      <td>255</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-02-21</td>\n",
       "      <td>2015-02-23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>668</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Feb. 12-15</td>\n",
       "      <td>Huckabee +5</td>\n",
       "      <td>Opinion Research Corporation</td>\n",
       "      <td>&lt;a href=\"http://www.realclearpolitics.com/docs...</td>\n",
       "      <td>385</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-02-12</td>\n",
       "      <td>2015-02-15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>669</th>\n",
       "      <td>0.15</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.04</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jan. 25-27</td>\n",
       "      <td>Bush +2</td>\n",
       "      <td>Fox News</td>\n",
       "      <td>&lt;a href=\"http://www.foxnews.com/politics/inter...</td>\n",
       "      <td>394</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2016-01-25</td>\n",
       "      <td>2016-01-27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Bush  Carson  Christie  Cruz  Fiorina  Huckabee  Kasich  Paul  Rubio  \\\n",
       "665  0.16    0.09      0.07  0.04     0.00      0.10    0.02  0.12   0.07   \n",
       "666  0.16    0.07      0.08  0.06      NaN      0.08    0.01  0.06   0.05   \n",
       "667  0.16    0.06      0.07  0.05     0.00      0.10    0.01  0.10   0.08   \n",
       "668  0.12    0.09      0.07  0.03     0.01      0.17    0.02  0.11   0.06   \n",
       "669  0.15    0.10      0.06  0.04      NaN      0.13    0.02  0.13   0.05   \n",
       "\n",
       "     Santorum  Trump            dates       leader  \\\n",
       "665      0.01    NaN      Mar. 13-15      Bush +3*   \n",
       "666      0.02    NaN  Feb. 26-Mar. 2    Walker +2*   \n",
       "667      0.02    NaN      Feb. 21-23    Walker +2*   \n",
       "668      0.02    NaN      Feb. 12-15   Huckabee +5   \n",
       "669      0.02    NaN      Jan. 25-27       Bush +2   \n",
       "\n",
       "                         pollster  \\\n",
       "665  Opinion Research Corporation   \n",
       "666         Quinnipiac University   \n",
       "667                        YouGov   \n",
       "668  Opinion Research Corporation   \n",
       "669                      Fox News   \n",
       "\n",
       "                                          pollster_url sample  \\\n",
       "665  <a href=\"http://i2.cdn.turner.com/cnn/2015/ima...   450    \n",
       "666  <a href=\"http://www.quinnipiac.edu/news-and-ev...   554    \n",
       "667  <a href=\"http://d25d2506sfb94s.cloudfront.net/...   255    \n",
       "668  <a href=\"http://www.realclearpolitics.com/docs...   385    \n",
       "669  <a href=\"http://www.foxnews.com/politics/inter...   394    \n",
       "\n",
       "                                                weight start_time   end_time  \n",
       "665  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-03-13 2015-03-15  \n",
       "666  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-02-26 2015-03-02  \n",
       "667  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-02-21 2015-02-23  \n",
       "668  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-02-12 2015-02-15  \n",
       "669  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2016-01-25 2016-01-27  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:44:05.185642Z",
     "start_time": "2019-05-14T20:44:05.181433Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(670, 17)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the size of this dataframe\n",
    "df.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assign dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:44:07.898794Z",
     "start_time": "2019-05-14T20:44:07.319815Z"
    }
   },
   "outputs": [],
   "source": [
    "# Assign datetime objects to the start and end of the poll\n",
    "assign_dates(df=df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:44:08.798037Z",
     "start_time": "2019-05-14T20:44:08.770132Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bush</th>\n",
       "      <th>Carson</th>\n",
       "      <th>Christie</th>\n",
       "      <th>Cruz</th>\n",
       "      <th>Fiorina</th>\n",
       "      <th>Huckabee</th>\n",
       "      <th>Kasich</th>\n",
       "      <th>Paul</th>\n",
       "      <th>Rubio</th>\n",
       "      <th>Santorum</th>\n",
       "      <th>Trump</th>\n",
       "      <th>dates</th>\n",
       "      <th>leader</th>\n",
       "      <th>pollster</th>\n",
       "      <th>pollster_url</th>\n",
       "      <th>sample</th>\n",
       "      <th>weight</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>665</th>\n",
       "      <td>0.16</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mar. 13-15</td>\n",
       "      <td>Bush +3*</td>\n",
       "      <td>Opinion Research Corporation</td>\n",
       "      <td>&lt;a href=\"http://i2.cdn.turner.com/cnn/2015/ima...</td>\n",
       "      <td>450</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-03-13</td>\n",
       "      <td>2015-03-15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>666</th>\n",
       "      <td>0.16</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.06</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Feb. 26-Mar. 2</td>\n",
       "      <td>Walker +2*</td>\n",
       "      <td>Quinnipiac University</td>\n",
       "      <td>&lt;a href=\"http://www.quinnipiac.edu/news-and-ev...</td>\n",
       "      <td>554</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-02-26</td>\n",
       "      <td>2015-03-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>667</th>\n",
       "      <td>0.16</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Feb. 21-23</td>\n",
       "      <td>Walker +2*</td>\n",
       "      <td>YouGov</td>\n",
       "      <td>&lt;a href=\"http://d25d2506sfb94s.cloudfront.net/...</td>\n",
       "      <td>255</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-02-21</td>\n",
       "      <td>2015-02-23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>668</th>\n",
       "      <td>0.12</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Feb. 12-15</td>\n",
       "      <td>Huckabee +5</td>\n",
       "      <td>Opinion Research Corporation</td>\n",
       "      <td>&lt;a href=\"http://www.realclearpolitics.com/docs...</td>\n",
       "      <td>385</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2015-02-12</td>\n",
       "      <td>2015-02-15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>669</th>\n",
       "      <td>0.15</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.04</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.02</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jan. 25-27</td>\n",
       "      <td>Bush +2</td>\n",
       "      <td>Fox News</td>\n",
       "      <td>&lt;a href=\"http://www.foxnews.com/politics/inter...</td>\n",
       "      <td>394</td>\n",
       "      <td>\\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n...</td>\n",
       "      <td>2016-01-25</td>\n",
       "      <td>2016-01-27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Bush  Carson  Christie  Cruz  Fiorina  Huckabee  Kasich  Paul  Rubio  \\\n",
       "665  0.16    0.09      0.07  0.04     0.00      0.10    0.02  0.12   0.07   \n",
       "666  0.16    0.07      0.08  0.06      NaN      0.08    0.01  0.06   0.05   \n",
       "667  0.16    0.06      0.07  0.05     0.00      0.10    0.01  0.10   0.08   \n",
       "668  0.12    0.09      0.07  0.03     0.01      0.17    0.02  0.11   0.06   \n",
       "669  0.15    0.10      0.06  0.04      NaN      0.13    0.02  0.13   0.05   \n",
       "\n",
       "     Santorum  Trump            dates       leader  \\\n",
       "665      0.01    NaN      Mar. 13-15      Bush +3*   \n",
       "666      0.02    NaN  Feb. 26-Mar. 2    Walker +2*   \n",
       "667      0.02    NaN      Feb. 21-23    Walker +2*   \n",
       "668      0.02    NaN      Feb. 12-15   Huckabee +5   \n",
       "669      0.02    NaN      Jan. 25-27       Bush +2   \n",
       "\n",
       "                         pollster  \\\n",
       "665  Opinion Research Corporation   \n",
       "666         Quinnipiac University   \n",
       "667                        YouGov   \n",
       "668  Opinion Research Corporation   \n",
       "669                      Fox News   \n",
       "\n",
       "                                          pollster_url sample  \\\n",
       "665  <a href=\"http://i2.cdn.turner.com/cnn/2015/ima...   450    \n",
       "666  <a href=\"http://www.quinnipiac.edu/news-and-ev...   554    \n",
       "667  <a href=\"http://d25d2506sfb94s.cloudfront.net/...   255    \n",
       "668  <a href=\"http://www.realclearpolitics.com/docs...   385    \n",
       "669  <a href=\"http://www.foxnews.com/politics/inter...   394    \n",
       "\n",
       "                                                weight start_time   end_time  \n",
       "665  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-03-13 2015-03-15  \n",
       "666  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-02-26 2015-03-02  \n",
       "667  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-02-21 2015-02-23  \n",
       "668  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2015-02-12 2015-02-15  \n",
       "669  \\n\\n0.00\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n... 2016-01-25 2016-01-27  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a modeling DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-14T20:44:13.638990Z",
     "start_time": "2019-05-14T20:44:13.592886Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Bush</th>\n",
       "      <th>Carson</th>\n",
       "      <th>Christie</th>\n",
       "      <th>Cruz</th>\n",
       "      <th>Fiorina</th>\n",
       "      <th>Huckabee</th>\n",
       "      <th>Kasich</th>\n",
       "      <th>Paul</th>\n",
       "      <th>Rubio</th>\n",
       "      <th>Santorum</th>\n",
       "      <th>Trump</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>end_time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2015-02-15</th>\n",
       "      <td>0.120</td>\n",
       "      <td>0.090</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.030000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.170</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.020</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-02-23</th>\n",
       "      <td>0.160</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.020</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-02</th>\n",
       "      <td>0.160</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.060000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.020</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-15</th>\n",
       "      <td>0.160</td>\n",
       "      <td>0.090</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.010</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-19</th>\n",
       "      <td>0.160</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.090000</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.065</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.035</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-20</th>\n",
       "      <td>0.170</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.090000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.060</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-21</th>\n",
       "      <td>0.170</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.090000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.080</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.040000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-22</th>\n",
       "      <td>0.180</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.050000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-23</th>\n",
       "      <td>0.155</td>\n",
       "      <td>0.105</td>\n",
       "      <td>0.065</td>\n",
       "      <td>0.095000</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.065</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.025</td>\n",
       "      <td>0.060000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-24</th>\n",
       "      <td>0.170</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.090000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.060000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-25</th>\n",
       "      <td>0.160</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.090000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.120</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.060000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-26</th>\n",
       "      <td>0.150</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.040000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-27</th>\n",
       "      <td>0.160</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-28</th>\n",
       "      <td>0.170</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-29</th>\n",
       "      <td>0.190</td>\n",
       "      <td>0.075</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.075</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.025</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-30</th>\n",
       "      <td>0.180</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-03-31</th>\n",
       "      <td>0.155</td>\n",
       "      <td>0.085</td>\n",
       "      <td>0.075</td>\n",
       "      <td>0.115000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.095</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.085</td>\n",
       "      <td>0.065</td>\n",
       "      <td>0.025</td>\n",
       "      <td>0.035000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-01</th>\n",
       "      <td>0.180</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.120000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.040000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-02</th>\n",
       "      <td>0.135</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.115000</td>\n",
       "      <td>0.015</td>\n",
       "      <td>0.105</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.065</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.015</td>\n",
       "      <td>0.055000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-03</th>\n",
       "      <td>0.220</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-04</th>\n",
       "      <td>0.240</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-05</th>\n",
       "      <td>0.230</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-06</th>\n",
       "      <td>0.200</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-07</th>\n",
       "      <td>0.200</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.180000</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-08</th>\n",
       "      <td>0.170</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-09</th>\n",
       "      <td>0.150</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-10</th>\n",
       "      <td>0.160</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.090</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-11</th>\n",
       "      <td>0.170</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.080</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.050</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-12</th>\n",
       "      <td>0.180</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.090</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.070</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015-04-13</th>\n",
       "      <td>0.150</td>\n",
       "      <td>0.075</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.065</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>0.135</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.020000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-04</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.370000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.280000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.350000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-05</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.390000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.230000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.370000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-06</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.460000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.210000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.330000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-07</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.230000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.370000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-08</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.370000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.230000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-09</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.190000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.410000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-10</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.330000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.185000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.445000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.293333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.163333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.483333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-12</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.315000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.190000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.435000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-13</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.285000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.235000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.460000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-14</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.310000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.230000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-15</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.290000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.220000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.480000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-16</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.280000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.520000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-17</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.283333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.156667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.483333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-18</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.340000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.510000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-19</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.380000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.480000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-20</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.340000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.150000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.490000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-21</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.310000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-22</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.270000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.165000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.495000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-23</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.280000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.180000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.520000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-24</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.270000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.173333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-25</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.280000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.540000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-26</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.290000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.180000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.505000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-27</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.520000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-28</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.315000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.160000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.490000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-29</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.310000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.485000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-04-30</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.520000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.253333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.163333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.530000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-02</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-05-03</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.260000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.170000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.560000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>415 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Bush  Carson  Christie      Cruz  Fiorina  Huckabee    Kasich  \\\n",
       "end_time                                                                     \n",
       "2015-02-15  0.120   0.090     0.070  0.030000    0.010     0.170  0.020000   \n",
       "2015-02-23  0.160   0.060     0.070  0.050000    0.000     0.100  0.010000   \n",
       "2015-03-02  0.160   0.070     0.080  0.060000      NaN     0.080  0.010000   \n",
       "2015-03-15  0.160   0.090     0.070  0.040000    0.000     0.100  0.020000   \n",
       "2015-03-19  0.160   0.095     0.070  0.090000    0.020     0.065  0.020000   \n",
       "2015-03-20  0.170   0.110     0.080  0.090000    0.010     0.060       NaN   \n",
       "2015-03-21  0.170   0.100     0.080  0.090000    0.010     0.080       NaN   \n",
       "2015-03-22  0.180   0.110     0.070  0.100000    0.010     0.090       NaN   \n",
       "2015-03-23  0.155   0.105     0.065  0.095000    0.020     0.065  0.020000   \n",
       "2015-03-24  0.170   0.110     0.070  0.090000    0.010     0.090       NaN   \n",
       "2015-03-25  0.160   0.140     0.050  0.090000    0.010     0.120       NaN   \n",
       "2015-03-26  0.150   0.150     0.050  0.130000    0.010     0.110       NaN   \n",
       "2015-03-27  0.160   0.120     0.070  0.130000    0.000     0.100       NaN   \n",
       "2015-03-28  0.170   0.100     0.080  0.140000    0.000     0.110       NaN   \n",
       "2015-03-29  0.190   0.075     0.080  0.130000    0.005     0.095  0.010000   \n",
       "2015-03-30  0.180   0.070     0.100  0.130000    0.000     0.110       NaN   \n",
       "2015-03-31  0.155   0.085     0.075  0.115000    0.010     0.095  0.010000   \n",
       "2015-04-01  0.180   0.050     0.120  0.120000    0.010     0.100       NaN   \n",
       "2015-04-02  0.135   0.060     0.080  0.115000    0.015     0.105  0.010000   \n",
       "2015-04-03  0.220   0.030     0.100  0.140000    0.020     0.110       NaN   \n",
       "2015-04-04  0.240   0.040     0.070  0.160000    0.010     0.090       NaN   \n",
       "2015-04-05  0.230   0.050     0.060  0.170000    0.020     0.090       NaN   \n",
       "2015-04-06  0.200   0.060     0.080  0.160000    0.010     0.100       NaN   \n",
       "2015-04-07  0.200   0.070     0.070  0.180000    0.020     0.090       NaN   \n",
       "2015-04-08  0.170   0.080     0.070  0.170000    0.020     0.090       NaN   \n",
       "2015-04-09  0.150   0.080     0.080  0.160000    0.030     0.100       NaN   \n",
       "2015-04-10  0.160   0.070     0.080  0.150000    0.020     0.090       NaN   \n",
       "2015-04-11  0.170   0.060     0.070  0.170000    0.040     0.080       NaN   \n",
       "2015-04-12  0.180   0.070     0.090  0.140000    0.040     0.070       NaN   \n",
       "2015-04-13  0.150   0.075     0.060  0.130000    0.040     0.065  0.020000   \n",
       "...           ...     ...       ...       ...      ...       ...       ...   \n",
       "2016-04-04    NaN     NaN       NaN  0.370000      NaN       NaN  0.280000   \n",
       "2016-04-05    NaN     NaN       NaN  0.390000      NaN       NaN  0.230000   \n",
       "2016-04-06    NaN     NaN       NaN  0.460000      NaN       NaN  0.210000   \n",
       "2016-04-07    NaN     NaN       NaN  0.400000      NaN       NaN  0.230000   \n",
       "2016-04-08    NaN     NaN       NaN  0.370000      NaN       NaN  0.230000   \n",
       "2016-04-09    NaN     NaN       NaN  0.400000      NaN       NaN  0.190000   \n",
       "2016-04-10    NaN     NaN       NaN  0.330000      NaN       NaN  0.185000   \n",
       "2016-04-11    NaN     NaN       NaN  0.293333      NaN       NaN  0.163333   \n",
       "2016-04-12    NaN     NaN       NaN  0.315000      NaN       NaN  0.190000   \n",
       "2016-04-13    NaN     NaN       NaN  0.285000      NaN       NaN  0.235000   \n",
       "2016-04-14    NaN     NaN       NaN  0.310000      NaN       NaN  0.230000   \n",
       "2016-04-15    NaN     NaN       NaN  0.290000      NaN       NaN  0.220000   \n",
       "2016-04-16    NaN     NaN       NaN  0.280000      NaN       NaN  0.200000   \n",
       "2016-04-17    NaN     NaN       NaN  0.283333      NaN       NaN  0.156667   \n",
       "2016-04-18    NaN     NaN       NaN  0.340000      NaN       NaN  0.150000   \n",
       "2016-04-19    NaN     NaN       NaN  0.380000      NaN       NaN  0.140000   \n",
       "2016-04-20    NaN     NaN       NaN  0.340000      NaN       NaN  0.150000   \n",
       "2016-04-21    NaN     NaN       NaN  0.310000      NaN       NaN  0.170000   \n",
       "2016-04-22    NaN     NaN       NaN  0.270000      NaN       NaN  0.165000   \n",
       "2016-04-23    NaN     NaN       NaN  0.280000      NaN       NaN  0.180000   \n",
       "2016-04-24    NaN     NaN       NaN  0.270000      NaN       NaN  0.173333   \n",
       "2016-04-25    NaN     NaN       NaN  0.280000      NaN       NaN  0.170000   \n",
       "2016-04-26    NaN     NaN       NaN  0.290000      NaN       NaN  0.180000   \n",
       "2016-04-27    NaN     NaN       NaN  0.300000      NaN       NaN  0.170000   \n",
       "2016-04-28    NaN     NaN       NaN  0.315000      NaN       NaN  0.160000   \n",
       "2016-04-29    NaN     NaN       NaN  0.310000      NaN       NaN  0.140000   \n",
       "2016-04-30    NaN     NaN       NaN  0.300000      NaN       NaN  0.170000   \n",
       "2016-05-01    NaN     NaN       NaN  0.253333      NaN       NaN  0.163333   \n",
       "2016-05-02    NaN     NaN       NaN  0.250000      NaN       NaN  0.140000   \n",
       "2016-05-03    NaN     NaN       NaN  0.260000      NaN       NaN  0.170000   \n",
       "\n",
       "             Paul  Rubio  Santorum     Trump  \n",
       "end_time                                      \n",
       "2015-02-15  0.110  0.060     0.020       NaN  \n",
       "2015-02-23  0.100  0.080     0.020       NaN  \n",
       "2015-03-02  0.060  0.050     0.020       NaN  \n",
       "2015-03-15  0.120  0.070     0.010       NaN  \n",
       "2015-03-19  0.095  0.070     0.035  0.010000  \n",
       "2015-03-20  0.100  0.080     0.030  0.020000  \n",
       "2015-03-21  0.110  0.070     0.040  0.040000  \n",
       "2015-03-22  0.100  0.060     0.040  0.050000  \n",
       "2015-03-23  0.060  0.060     0.025  0.060000  \n",
       "2015-03-24  0.080  0.060     0.040  0.060000  \n",
       "2015-03-25  0.070  0.050     0.050  0.060000  \n",
       "2015-03-26  0.050  0.070     0.030  0.040000  \n",
       "2015-03-27  0.070  0.060     0.040  0.030000  \n",
       "2015-03-28  0.070  0.050     0.040  0.030000  \n",
       "2015-03-29  0.075  0.070     0.025  0.030000  \n",
       "2015-03-30  0.080  0.060     0.030  0.030000  \n",
       "2015-03-31  0.085  0.065     0.025  0.035000  \n",
       "2015-04-01  0.080  0.050     0.030  0.040000  \n",
       "2015-04-02  0.065  0.060     0.015  0.055000  \n",
       "2015-04-03  0.050  0.060     0.030  0.020000  \n",
       "2015-04-04  0.040  0.060     0.030  0.030000  \n",
       "2015-04-05  0.070  0.070     0.030  0.010000  \n",
       "2015-04-06  0.060  0.080     0.020  0.010000  \n",
       "2015-04-07  0.070  0.060     0.030  0.020000  \n",
       "2015-04-08  0.070  0.070     0.030  0.020000  \n",
       "2015-04-09  0.080  0.070     0.030  0.010000  \n",
       "2015-04-10  0.080  0.070     0.030  0.010000  \n",
       "2015-04-11  0.100  0.050     0.030  0.020000  \n",
       "2015-04-12  0.120  0.040     0.020  0.020000  \n",
       "2015-04-13  0.135  0.060     0.020  0.020000  \n",
       "...           ...    ...       ...       ...  \n",
       "2016-04-04    NaN    NaN       NaN  0.350000  \n",
       "2016-04-05    NaN    NaN       NaN  0.370000  \n",
       "2016-04-06    NaN    NaN       NaN  0.330000  \n",
       "2016-04-07    NaN    NaN       NaN  0.370000  \n",
       "2016-04-08    NaN    NaN       NaN  0.400000  \n",
       "2016-04-09    NaN    NaN       NaN  0.410000  \n",
       "2016-04-10    NaN    NaN       NaN  0.445000  \n",
       "2016-04-11    NaN    NaN       NaN  0.483333  \n",
       "2016-04-12    NaN    NaN       NaN  0.435000  \n",
       "2016-04-13    NaN    NaN       NaN  0.460000  \n",
       "2016-04-14    NaN    NaN       NaN  0.450000  \n",
       "2016-04-15    NaN    NaN       NaN  0.480000  \n",
       "2016-04-16    NaN    NaN       NaN  0.520000  \n",
       "2016-04-17    NaN    NaN       NaN  0.483333  \n",
       "2016-04-18    NaN    NaN       NaN  0.510000  \n",
       "2016-04-19    NaN    NaN       NaN  0.480000  \n",
       "2016-04-20    NaN    NaN       NaN  0.490000  \n",
       "2016-04-21    NaN    NaN       NaN  0.500000  \n",
       "2016-04-22    NaN    NaN       NaN  0.495000  \n",
       "2016-04-23    NaN    NaN       NaN  0.520000  \n",
       "2016-04-24    NaN    NaN       NaN  0.500000  \n",
       "2016-04-25    NaN    NaN       NaN  0.540000  \n",
       "2016-04-26    NaN    NaN       NaN  0.505000  \n",
       "2016-04-27    NaN    NaN       NaN  0.520000  \n",
       "2016-04-28    NaN    NaN       NaN  0.490000  \n",
       "2016-04-29    NaN    NaN       NaN  0.485000  \n",
       "2016-04-30    NaN    NaN       NaN  0.520000  \n",
       "2016-05-01    NaN    NaN       NaN  0.530000  \n",
       "2016-05-02    NaN    NaN       NaN  0.550000  \n",
       "2016-05-03    NaN    NaN       NaN  0.560000  \n",
       "\n",
       "[415 rows x 11 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Average the polls by end date\n",
    "df_m = df.groupby(['end_time']).mean()\n",
    "df_m\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save to a CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-30T12:35:42.136573Z",
     "start_time": "2019-04-30T12:35:42.082003Z"
    }
   },
   "outputs": [],
   "source": [
    "df.to_csv('../data/raw_poll_data_2016.csv', index = False)\n",
    "df_m.to_csv('../data/pure_poll_data_2016.csv', index = True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}