Assignment 3 - Jon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right;font-weight:bold\">Jon Kaykin</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Clustering Analysis of Posts By The White House"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We are interested in understanding whether the White House Administration can influence how people feel about the economy. One good indicator of the economy's strength are the top index funds in the stock market: S&P 500, Nasdaq, and Dow Jones (each represents a good mix of businesses within many different industries). Collecting just the blog posts for the corpus was quite easy: using BeautifulSoup, I scraped the latest 500 posts from the White House Blog (https://www.whitehouse.gov/blog). I first grabbed the date, title, and url of each page (10 posts per page). I then went through each url and, using BeautifulSoup again, scraped the body of each blog post. I then combined the date, title, and body and added each one to a row of a CSV. I now had a clean CSV that I could import into Pandas to start my analysis.\n",
    "\n",
    "I was curious to do some topic modeling of the collected blog posts. President Trump is known as a businessman, so it comes as no surprise that one of his promises was to reduce regulations, cut taxes and government spending, bring back jobs, and create a strong economy. So first, has the adminstration made any announcments about the economy/deregulation and if so, can we use a topic modeling algorithm (LDA) to pull this topic out? Let's find out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploratory Technique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "from itertools import combinations\n",
    "from scipy.stats import ttest_ind\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Title</th>\n",
       "      <th>Body</th>\n",
       "      <th>Nasdaq Value</th>\n",
       "      <th>Nasdaq Delta</th>\n",
       "      <th>Nasdaq Proportion</th>\n",
       "      <th>Dow Jones Value</th>\n",
       "      <th>Dow Jones Delta</th>\n",
       "      <th>Dow Jones Proportion</th>\n",
       "      <th>S&amp;P 500 Value</th>\n",
       "      <th>S&amp;P 500 Delta</th>\n",
       "      <th>S&amp;P 500 Proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-04-06</td>\n",
       "      <td>Notice Regarding the Continuation of the Natio...</td>\n",
       "      <td>notice continuation of the national emergency ...</td>\n",
       "      <td>5878.950195</td>\n",
       "      <td>-1.140136</td>\n",
       "      <td>-0.000194</td>\n",
       "      <td>20662.949219</td>\n",
       "      <td>-6.849610</td>\n",
       "      <td>-0.000331</td>\n",
       "      <td>2357.489990</td>\n",
       "      <td>-1.949951</td>\n",
       "      <td>-0.000827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-04-06</td>\n",
       "      <td>Message to the Congress Regarding the Continua...</td>\n",
       "      <td>to the congress of the united states section d...</td>\n",
       "      <td>5878.950195</td>\n",
       "      <td>-1.140136</td>\n",
       "      <td>-0.000194</td>\n",
       "      <td>20662.949219</td>\n",
       "      <td>-6.849610</td>\n",
       "      <td>-0.000331</td>\n",
       "      <td>2357.489990</td>\n",
       "      <td>-1.949951</td>\n",
       "      <td>-0.000827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-04-06</td>\n",
       "      <td>1600 Daily: Everything White House for 4/6/17</td>\n",
       "      <td>summary get news events and updates from the w...</td>\n",
       "      <td>5878.950195</td>\n",
       "      <td>-1.140136</td>\n",
       "      <td>-0.000194</td>\n",
       "      <td>20662.949219</td>\n",
       "      <td>-6.849610</td>\n",
       "      <td>-0.000331</td>\n",
       "      <td>2357.489990</td>\n",
       "      <td>-1.949951</td>\n",
       "      <td>-0.000827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-04-06</td>\n",
       "      <td>Vice President Mike Pence to Travel to the Rep...</td>\n",
       "      <td>washington dc vice president mike pence will t...</td>\n",
       "      <td>5878.950195</td>\n",
       "      <td>-1.140136</td>\n",
       "      <td>-0.000194</td>\n",
       "      <td>20662.949219</td>\n",
       "      <td>-6.849610</td>\n",
       "      <td>-0.000331</td>\n",
       "      <td>2357.489990</td>\n",
       "      <td>-1.949951</td>\n",
       "      <td>-0.000827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-04-06</td>\n",
       "      <td>White House History: White House Easter Egg Roll</td>\n",
       "      <td>continuing the timeless tradition of the white...</td>\n",
       "      <td>5878.950195</td>\n",
       "      <td>-1.140136</td>\n",
       "      <td>-0.000194</td>\n",
       "      <td>20662.949219</td>\n",
       "      <td>-6.849610</td>\n",
       "      <td>-0.000331</td>\n",
       "      <td>2357.489990</td>\n",
       "      <td>-1.949951</td>\n",
       "      <td>-0.000827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2017-04-06</td>\n",
       "      <td>Remarks by President Trump and Vice President ...</td>\n",
       "      <td>east room a m edtthe vice president to all the...</td>\n",
       "      <td>5878.950195</td>\n",
       "      <td>-1.140136</td>\n",
       "      <td>-0.000194</td>\n",
       "      <td>20662.949219</td>\n",
       "      <td>-6.849610</td>\n",
       "      <td>-0.000331</td>\n",
       "      <td>2357.489990</td>\n",
       "      <td>-1.949951</td>\n",
       "      <td>-0.000827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2017-04-06</td>\n",
       "      <td>Photo of the Day: April 6, 2017</td>\n",
       "      <td>president donald trump and king abdullah ii of...</td>\n",
       "      <td>5878.950195</td>\n",
       "      <td>-1.140136</td>\n",
       "      <td>-0.000194</td>\n",
       "      <td>20662.949219</td>\n",
       "      <td>-6.849610</td>\n",
       "      <td>-0.000331</td>\n",
       "      <td>2357.489990</td>\n",
       "      <td>-1.949951</td>\n",
       "      <td>-0.000827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Readout of President Donald J. Trump’s Call wi...</td>\n",
       "      <td>president donald j trump spoke today with prim...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Statement from the Press Secretary on Gold Sta...</td>\n",
       "      <td>on this day in nearing the end of world war ii...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Memorandum: Implementing Executive Order 13771...</td>\n",
       "      <td>m memorandum for regulatory policy officers at...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Readout of President Donald J. Trump’s Call wi...</td>\n",
       "      <td>president donald j trump spoke today by teleph...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>A Proclamation by President Donald J. Trump Ho...</td>\n",
       "      <td>honoring the memory of john glenn by the presi...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Statement from the Press Secretary on the Decl...</td>\n",
       "      <td>statement from the press secretarytodays annou...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Remarks by President Trump and His Majesty Kin...</td>\n",
       "      <td>rose garden p m edtpresident trump thank you v...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Readout of First Lady Melania Trump’s Visit to...</td>\n",
       "      <td>after a brief visit to the oval office with th...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Press Briefing on the Congressional Review Act</td>\n",
       "      <td>james s brady press briefing room a m edtmr sp...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>1600 Daily: Everything White House for 4/5/17</td>\n",
       "      <td>summary get news events and updates from the w...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>First Lady Melania Trump to Host Queen Rania o...</td>\n",
       "      <td>first lady melania trump along with her husban...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>2017-04-05</td>\n",
       "      <td>Photo of the Day: April 5, 2017</td>\n",
       "      <td>president donald trump responds to a question ...</td>\n",
       "      <td>5864.479980</td>\n",
       "      <td>14.470215</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>20648.150391</td>\n",
       "      <td>14.798828</td>\n",
       "      <td>0.000717</td>\n",
       "      <td>2352.949951</td>\n",
       "      <td>4.540039</td>\n",
       "      <td>0.001930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>Two Nominations Delivered to the Senate Today</td>\n",
       "      <td>nominations sent to the senate sigal mandelker...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>Background Briefing by Senior Administration O...</td>\n",
       "      <td>james s brady press briefing room p m edtsenio...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>Statement from President Donald J. Trump</td>\n",
       "      <td>todays chemical attack in syria against innoce...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>UPDATE: Confirming Judge Neil Gorsuch to the S...</td>\n",
       "      <td>on monday judge neil gorsuch was confirmed by ...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>President Donald J. Trump Announces Intent to ...</td>\n",
       "      <td>president donald j trump today announced his i...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>Remarks by President Trump at 2017 North Ameri...</td>\n",
       "      <td>washington hiltonwashington d c p m edtthe pre...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>Press Gaggle by Press Secretary Sean Spicer, 4...</td>\n",
       "      <td>james s brady press briefing room a m edtmr sp...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>Remarks by President Trump and Vice President ...</td>\n",
       "      <td>south court auditorium a m edtthe vice preside...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>1600 Daily: Everything White House for 4/4/17</td>\n",
       "      <td>summary get news events and updates from the w...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>2017-04-04</td>\n",
       "      <td>Photo of the Day: April 4, 2017</td>\n",
       "      <td>moments before egyptian president abdel fattah...</td>\n",
       "      <td>5898.609863</td>\n",
       "      <td>-34.129883</td>\n",
       "      <td>-0.005786</td>\n",
       "      <td>20689.240234</td>\n",
       "      <td>-41.089843</td>\n",
       "      <td>-0.001986</td>\n",
       "      <td>2360.159912</td>\n",
       "      <td>-7.209961</td>\n",
       "      <td>-0.003055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>2017-04-03</td>\n",
       "      <td>Readout of President Donald J. Trump’s Meeting...</td>\n",
       "      <td>president donald j trump today welcomed presid...</td>\n",
       "      <td>5894.680176</td>\n",
       "      <td>3.929687</td>\n",
       "      <td>0.000667</td>\n",
       "      <td>20650.210938</td>\n",
       "      <td>39.029296</td>\n",
       "      <td>0.001890</td>\n",
       "      <td>2358.840088</td>\n",
       "      <td>1.319824</td>\n",
       "      <td>0.000560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>433</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Remarks by the Vice President at the Henry O. ...</td>\n",
       "      <td>the united states military academywest point n...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>434</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Readout of the President's Call with President...</td>\n",
       "      <td>president trump today spoke with president gha...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>435</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Providing an Order of Succession Within the De...</td>\n",
       "      <td>executive order providing an order of successi...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>436</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Press Briefing by Press Secretary Sean Spicer,...</td>\n",
       "      <td>james s brady press briefing room p m estmr sp...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>437</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>President Trump Signs Pro-Law Enforcement and ...</td>\n",
       "      <td>today the president signedthree executive orde...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>438</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Background Press Call on Prime Minister Abe's ...</td>\n",
       "      <td>background press callby senior administration ...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>439</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Remarks by President Trump Before SCOTUS Liste...</td>\n",
       "      <td>roosevelt room p m estthe president thank you ...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>440</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Presidential Executive Order on Enforcing Fede...</td>\n",
       "      <td>executive order enforcing federal law with res...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>441</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Presidential Executive Order on Preventing Vio...</td>\n",
       "      <td>executive order preventing violence against fe...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>442</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Remarks by President Trump in Meeting with the...</td>\n",
       "      <td>state dining room a m estthe president thank y...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>443</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Presidential Executive Order on a Task Force o...</td>\n",
       "      <td>executive order task force on crime reduction ...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>444</th>\n",
       "      <td>2017-02-09</td>\n",
       "      <td>Remarks by President Trump at Swearing-In of A...</td>\n",
       "      <td>oval office a m estthe president thank you ver...</td>\n",
       "      <td>5715.180176</td>\n",
       "      <td>18.949707</td>\n",
       "      <td>0.003316</td>\n",
       "      <td>20172.400391</td>\n",
       "      <td>96.968750</td>\n",
       "      <td>0.004807</td>\n",
       "      <td>2307.870117</td>\n",
       "      <td>8.229981</td>\n",
       "      <td>0.003566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>445</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>Statement from the Press Secretary</td>\n",
       "      <td>president donald j trump today provided a lett...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>446</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>First Lady Melania Trump Announces the Appoint...</td>\n",
       "      <td>first lady melania trumpis pleased to announce...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>447</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>Widespread Support for Judge Neil Gorsuch</td>\n",
       "      <td>all across the country editorial boards are pr...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>448</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>Press Briefing by Press Secretary Sean Spicer,...</td>\n",
       "      <td>james s brady press briefing room p m estmr sp...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>449</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>Remarks by President Trump and Intel CEO Brian...</td>\n",
       "      <td>oval office p m estthe president you ve never ...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>450</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>Intel Announces $7 Billion Investment Creating...</td>\n",
       "      <td>the past years have taken a toll on american w...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>451</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>Judge Neil Gorsuch is the Mainstream Choice fo...</td>\n",
       "      <td>gorsuch in the mainstreamhe was upheld at the ...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>452</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>Remarks by President Trump at MCCA Winter Conf...</td>\n",
       "      <td>j w marriottwashington d c a m estthe presiden...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>453</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>ICYMI: CNBC: “CEO Confidence At Near-Record Hi...</td>\n",
       "      <td>confidence among chief executives of u s based...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>454</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>President Trump's War on Regulation Results in...</td>\n",
       "      <td>ceo confidence at near record high why trump s...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>455</th>\n",
       "      <td>2017-02-08</td>\n",
       "      <td>President Donald J. Trump Announces His Cabinet</td>\n",
       "      <td>president donald j trump today formally announ...</td>\n",
       "      <td>5682.450195</td>\n",
       "      <td>32.729981</td>\n",
       "      <td>0.005760</td>\n",
       "      <td>20054.339844</td>\n",
       "      <td>118.060547</td>\n",
       "      <td>0.005887</td>\n",
       "      <td>2294.669922</td>\n",
       "      <td>13.200195</td>\n",
       "      <td>0.005753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>456</th>\n",
       "      <td>2017-02-07</td>\n",
       "      <td>Readout of the President's Call with Prime Min...</td>\n",
       "      <td>president donald j trump today spoke with prim...</td>\n",
       "      <td>5674.220215</td>\n",
       "      <td>8.229980</td>\n",
       "      <td>0.001450</td>\n",
       "      <td>20090.289062</td>\n",
       "      <td>-35.949218</td>\n",
       "      <td>-0.001789</td>\n",
       "      <td>2293.080078</td>\n",
       "      <td>1.589844</td>\n",
       "      <td>0.000693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>457</th>\n",
       "      <td>2017-02-07</td>\n",
       "      <td>Readout of the Vice President's Meeting with R...</td>\n",
       "      <td>vice president mike pence today met with right...</td>\n",
       "      <td>5674.220215</td>\n",
       "      <td>8.229980</td>\n",
       "      <td>0.001450</td>\n",
       "      <td>20090.289062</td>\n",
       "      <td>-35.949218</td>\n",
       "      <td>-0.001789</td>\n",
       "      <td>2293.080078</td>\n",
       "      <td>1.589844</td>\n",
       "      <td>0.000693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458</th>\n",
       "      <td>2017-02-07</td>\n",
       "      <td>Readout of the President's Call with President...</td>\n",
       "      <td>president donald j trump today spoke by phone ...</td>\n",
       "      <td>5674.220215</td>\n",
       "      <td>8.229980</td>\n",
       "      <td>0.001450</td>\n",
       "      <td>20090.289062</td>\n",
       "      <td>-35.949218</td>\n",
       "      <td>-0.001789</td>\n",
       "      <td>2293.080078</td>\n",
       "      <td>1.589844</td>\n",
       "      <td>0.000693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>2017-02-07</td>\n",
       "      <td>White House Director of Legislative Affairs An...</td>\n",
       "      <td>washington d c president trumps director of le...</td>\n",
       "      <td>5674.220215</td>\n",
       "      <td>8.229980</td>\n",
       "      <td>0.001450</td>\n",
       "      <td>20090.289062</td>\n",
       "      <td>-35.949218</td>\n",
       "      <td>-0.001789</td>\n",
       "      <td>2293.080078</td>\n",
       "      <td>1.589844</td>\n",
       "      <td>0.000693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>460</th>\n",
       "      <td>2017-02-07</td>\n",
       "      <td>Vice President Pence Casts Historic Vote to Co...</td>\n",
       "      <td>friends today i cast the tie breaking vote in ...</td>\n",
       "      <td>5674.220215</td>\n",
       "      <td>8.229980</td>\n",
       "      <td>0.001450</td>\n",
       "      <td>20090.289062</td>\n",
       "      <td>-35.949218</td>\n",
       "      <td>-0.001789</td>\n",
       "      <td>2293.080078</td>\n",
       "      <td>1.589844</td>\n",
       "      <td>0.000693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>2017-02-07</td>\n",
       "      <td>HJR 58    Disapproving the Rule Submitted by t...</td>\n",
       "      <td>february house statement of administration pol...</td>\n",
       "      <td>5674.220215</td>\n",
       "      <td>8.229980</td>\n",
       "      <td>0.001450</td>\n",
       "      <td>20090.289062</td>\n",
       "      <td>-35.949218</td>\n",
       "      <td>-0.001789</td>\n",
       "      <td>2293.080078</td>\n",
       "      <td>1.589844</td>\n",
       "      <td>0.000693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>462</th>\n",
       "      <td>2017-02-07</td>\n",
       "      <td>HJR  57   Disapproving the Rule Submitted by t...</td>\n",
       "      <td>february house statement of administration pol...</td>\n",
       "      <td>5674.220215</td>\n",
       "      <td>8.229980</td>\n",
       "      <td>0.001450</td>\n",
       "      <td>20090.289062</td>\n",
       "      <td>-35.949218</td>\n",
       "      <td>-0.001789</td>\n",
       "      <td>2293.080078</td>\n",
       "      <td>1.589844</td>\n",
       "      <td>0.000693</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>463 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date                                              Title  \\\n",
       "0    2017-04-06  Notice Regarding the Continuation of the Natio...   \n",
       "1    2017-04-06  Message to the Congress Regarding the Continua...   \n",
       "2    2017-04-06      1600 Daily: Everything White House for 4/6/17   \n",
       "3    2017-04-06  Vice President Mike Pence to Travel to the Rep...   \n",
       "4    2017-04-06  White House History: White House Easter Egg Roll    \n",
       "5    2017-04-06  Remarks by President Trump and Vice President ...   \n",
       "6    2017-04-06                    Photo of the Day: April 6, 2017   \n",
       "7    2017-04-05  Readout of President Donald J. Trump’s Call wi...   \n",
       "8    2017-04-05  Statement from the Press Secretary on Gold Sta...   \n",
       "9    2017-04-05  Memorandum: Implementing Executive Order 13771...   \n",
       "10   2017-04-05  Readout of President Donald J. Trump’s Call wi...   \n",
       "11   2017-04-05  A Proclamation by President Donald J. Trump Ho...   \n",
       "12   2017-04-05  Statement from the Press Secretary on the Decl...   \n",
       "13   2017-04-05  Remarks by President Trump and His Majesty Kin...   \n",
       "14   2017-04-05  Readout of First Lady Melania Trump’s Visit to...   \n",
       "15   2017-04-05     Press Briefing on the Congressional Review Act   \n",
       "16   2017-04-05      1600 Daily: Everything White House for 4/5/17   \n",
       "17   2017-04-05  First Lady Melania Trump to Host Queen Rania o...   \n",
       "18   2017-04-05                    Photo of the Day: April 5, 2017   \n",
       "19   2017-04-04      Two Nominations Delivered to the Senate Today   \n",
       "20   2017-04-04  Background Briefing by Senior Administration O...   \n",
       "21   2017-04-04           Statement from President Donald J. Trump   \n",
       "22   2017-04-04  UPDATE: Confirming Judge Neil Gorsuch to the S...   \n",
       "23   2017-04-04  President Donald J. Trump Announces Intent to ...   \n",
       "24   2017-04-04  Remarks by President Trump at 2017 North Ameri...   \n",
       "25   2017-04-04  Press Gaggle by Press Secretary Sean Spicer, 4...   \n",
       "26   2017-04-04  Remarks by President Trump and Vice President ...   \n",
       "27   2017-04-04      1600 Daily: Everything White House for 4/4/17   \n",
       "28   2017-04-04                    Photo of the Day: April 4, 2017   \n",
       "29   2017-04-03  Readout of President Donald J. Trump’s Meeting...   \n",
       "..          ...                                                ...   \n",
       "433  2017-02-09  Remarks by the Vice President at the Henry O. ...   \n",
       "434  2017-02-09  Readout of the President's Call with President...   \n",
       "435  2017-02-09  Providing an Order of Succession Within the De...   \n",
       "436  2017-02-09  Press Briefing by Press Secretary Sean Spicer,...   \n",
       "437  2017-02-09  President Trump Signs Pro-Law Enforcement and ...   \n",
       "438  2017-02-09  Background Press Call on Prime Minister Abe's ...   \n",
       "439  2017-02-09  Remarks by President Trump Before SCOTUS Liste...   \n",
       "440  2017-02-09  Presidential Executive Order on Enforcing Fede...   \n",
       "441  2017-02-09  Presidential Executive Order on Preventing Vio...   \n",
       "442  2017-02-09  Remarks by President Trump in Meeting with the...   \n",
       "443  2017-02-09  Presidential Executive Order on a Task Force o...   \n",
       "444  2017-02-09  Remarks by President Trump at Swearing-In of A...   \n",
       "445  2017-02-08                 Statement from the Press Secretary   \n",
       "446  2017-02-08  First Lady Melania Trump Announces the Appoint...   \n",
       "447  2017-02-08          Widespread Support for Judge Neil Gorsuch   \n",
       "448  2017-02-08  Press Briefing by Press Secretary Sean Spicer,...   \n",
       "449  2017-02-08  Remarks by President Trump and Intel CEO Brian...   \n",
       "450  2017-02-08  Intel Announces $7 Billion Investment Creating...   \n",
       "451  2017-02-08  Judge Neil Gorsuch is the Mainstream Choice fo...   \n",
       "452  2017-02-08  Remarks by President Trump at MCCA Winter Conf...   \n",
       "453  2017-02-08  ICYMI: CNBC: “CEO Confidence At Near-Record Hi...   \n",
       "454  2017-02-08  President Trump's War on Regulation Results in...   \n",
       "455  2017-02-08    President Donald J. Trump Announces His Cabinet   \n",
       "456  2017-02-07  Readout of the President's Call with Prime Min...   \n",
       "457  2017-02-07  Readout of the Vice President's Meeting with R...   \n",
       "458  2017-02-07  Readout of the President's Call with President...   \n",
       "459  2017-02-07  White House Director of Legislative Affairs An...   \n",
       "460  2017-02-07  Vice President Pence Casts Historic Vote to Co...   \n",
       "461  2017-02-07  HJR 58    Disapproving the Rule Submitted by t...   \n",
       "462  2017-02-07  HJR  57   Disapproving the Rule Submitted by t...   \n",
       "\n",
       "                                                  Body  Nasdaq Value  \\\n",
       "0    notice continuation of the national emergency ...   5878.950195   \n",
       "1    to the congress of the united states section d...   5878.950195   \n",
       "2    summary get news events and updates from the w...   5878.950195   \n",
       "3    washington dc vice president mike pence will t...   5878.950195   \n",
       "4    continuing the timeless tradition of the white...   5878.950195   \n",
       "5    east room a m edtthe vice president to all the...   5878.950195   \n",
       "6    president donald trump and king abdullah ii of...   5878.950195   \n",
       "7    president donald j trump spoke today with prim...   5864.479980   \n",
       "8    on this day in nearing the end of world war ii...   5864.479980   \n",
       "9    m memorandum for regulatory policy officers at...   5864.479980   \n",
       "10   president donald j trump spoke today by teleph...   5864.479980   \n",
       "11   honoring the memory of john glenn by the presi...   5864.479980   \n",
       "12   statement from the press secretarytodays annou...   5864.479980   \n",
       "13   rose garden p m edtpresident trump thank you v...   5864.479980   \n",
       "14   after a brief visit to the oval office with th...   5864.479980   \n",
       "15   james s brady press briefing room a m edtmr sp...   5864.479980   \n",
       "16   summary get news events and updates from the w...   5864.479980   \n",
       "17   first lady melania trump along with her husban...   5864.479980   \n",
       "18   president donald trump responds to a question ...   5864.479980   \n",
       "19   nominations sent to the senate sigal mandelker...   5898.609863   \n",
       "20   james s brady press briefing room p m edtsenio...   5898.609863   \n",
       "21   todays chemical attack in syria against innoce...   5898.609863   \n",
       "22   on monday judge neil gorsuch was confirmed by ...   5898.609863   \n",
       "23   president donald j trump today announced his i...   5898.609863   \n",
       "24   washington hiltonwashington d c p m edtthe pre...   5898.609863   \n",
       "25   james s brady press briefing room a m edtmr sp...   5898.609863   \n",
       "26   south court auditorium a m edtthe vice preside...   5898.609863   \n",
       "27   summary get news events and updates from the w...   5898.609863   \n",
       "28   moments before egyptian president abdel fattah...   5898.609863   \n",
       "29   president donald j trump today welcomed presid...   5894.680176   \n",
       "..                                                 ...           ...   \n",
       "433  the united states military academywest point n...   5715.180176   \n",
       "434  president trump today spoke with president gha...   5715.180176   \n",
       "435  executive order providing an order of successi...   5715.180176   \n",
       "436  james s brady press briefing room p m estmr sp...   5715.180176   \n",
       "437  today the president signedthree executive orde...   5715.180176   \n",
       "438  background press callby senior administration ...   5715.180176   \n",
       "439  roosevelt room p m estthe president thank you ...   5715.180176   \n",
       "440  executive order enforcing federal law with res...   5715.180176   \n",
       "441  executive order preventing violence against fe...   5715.180176   \n",
       "442  state dining room a m estthe president thank y...   5715.180176   \n",
       "443  executive order task force on crime reduction ...   5715.180176   \n",
       "444  oval office a m estthe president thank you ver...   5715.180176   \n",
       "445  president donald j trump today provided a lett...   5682.450195   \n",
       "446  first lady melania trumpis pleased to announce...   5682.450195   \n",
       "447  all across the country editorial boards are pr...   5682.450195   \n",
       "448  james s brady press briefing room p m estmr sp...   5682.450195   \n",
       "449  oval office p m estthe president you ve never ...   5682.450195   \n",
       "450  the past years have taken a toll on american w...   5682.450195   \n",
       "451  gorsuch in the mainstreamhe was upheld at the ...   5682.450195   \n",
       "452  j w marriottwashington d c a m estthe presiden...   5682.450195   \n",
       "453  confidence among chief executives of u s based...   5682.450195   \n",
       "454  ceo confidence at near record high why trump s...   5682.450195   \n",
       "455  president donald j trump today formally announ...   5682.450195   \n",
       "456  president donald j trump today spoke with prim...   5674.220215   \n",
       "457  vice president mike pence today met with right...   5674.220215   \n",
       "458  president donald j trump today spoke by phone ...   5674.220215   \n",
       "459  washington d c president trumps director of le...   5674.220215   \n",
       "460  friends today i cast the tie breaking vote in ...   5674.220215   \n",
       "461  february house statement of administration pol...   5674.220215   \n",
       "462  february house statement of administration pol...   5674.220215   \n",
       "\n",
       "     Nasdaq Delta  Nasdaq Proportion  Dow Jones Value  Dow Jones Delta  \\\n",
       "0       -1.140136          -0.000194     20662.949219        -6.849610   \n",
       "1       -1.140136          -0.000194     20662.949219        -6.849610   \n",
       "2       -1.140136          -0.000194     20662.949219        -6.849610   \n",
       "3       -1.140136          -0.000194     20662.949219        -6.849610   \n",
       "4       -1.140136          -0.000194     20662.949219        -6.849610   \n",
       "5       -1.140136          -0.000194     20662.949219        -6.849610   \n",
       "6       -1.140136          -0.000194     20662.949219        -6.849610   \n",
       "7       14.470215           0.002467     20648.150391        14.798828   \n",
       "8       14.470215           0.002467     20648.150391        14.798828   \n",
       "9       14.470215           0.002467     20648.150391        14.798828   \n",
       "10      14.470215           0.002467     20648.150391        14.798828   \n",
       "11      14.470215           0.002467     20648.150391        14.798828   \n",
       "12      14.470215           0.002467     20648.150391        14.798828   \n",
       "13      14.470215           0.002467     20648.150391        14.798828   \n",
       "14      14.470215           0.002467     20648.150391        14.798828   \n",
       "15      14.470215           0.002467     20648.150391        14.798828   \n",
       "16      14.470215           0.002467     20648.150391        14.798828   \n",
       "17      14.470215           0.002467     20648.150391        14.798828   \n",
       "18      14.470215           0.002467     20648.150391        14.798828   \n",
       "19     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "20     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "21     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "22     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "23     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "24     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "25     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "26     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "27     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "28     -34.129883          -0.005786     20689.240234       -41.089843   \n",
       "29       3.929687           0.000667     20650.210938        39.029296   \n",
       "..            ...                ...              ...              ...   \n",
       "433     18.949707           0.003316     20172.400391        96.968750   \n",
       "434     18.949707           0.003316     20172.400391        96.968750   \n",
       "435     18.949707           0.003316     20172.400391        96.968750   \n",
       "436     18.949707           0.003316     20172.400391        96.968750   \n",
       "437     18.949707           0.003316     20172.400391        96.968750   \n",
       "438     18.949707           0.003316     20172.400391        96.968750   \n",
       "439     18.949707           0.003316     20172.400391        96.968750   \n",
       "440     18.949707           0.003316     20172.400391        96.968750   \n",
       "441     18.949707           0.003316     20172.400391        96.968750   \n",
       "442     18.949707           0.003316     20172.400391        96.968750   \n",
       "443     18.949707           0.003316     20172.400391        96.968750   \n",
       "444     18.949707           0.003316     20172.400391        96.968750   \n",
       "445     32.729981           0.005760     20054.339844       118.060547   \n",
       "446     32.729981           0.005760     20054.339844       118.060547   \n",
       "447     32.729981           0.005760     20054.339844       118.060547   \n",
       "448     32.729981           0.005760     20054.339844       118.060547   \n",
       "449     32.729981           0.005760     20054.339844       118.060547   \n",
       "450     32.729981           0.005760     20054.339844       118.060547   \n",
       "451     32.729981           0.005760     20054.339844       118.060547   \n",
       "452     32.729981           0.005760     20054.339844       118.060547   \n",
       "453     32.729981           0.005760     20054.339844       118.060547   \n",
       "454     32.729981           0.005760     20054.339844       118.060547   \n",
       "455     32.729981           0.005760     20054.339844       118.060547   \n",
       "456      8.229980           0.001450     20090.289062       -35.949218   \n",
       "457      8.229980           0.001450     20090.289062       -35.949218   \n",
       "458      8.229980           0.001450     20090.289062       -35.949218   \n",
       "459      8.229980           0.001450     20090.289062       -35.949218   \n",
       "460      8.229980           0.001450     20090.289062       -35.949218   \n",
       "461      8.229980           0.001450     20090.289062       -35.949218   \n",
       "462      8.229980           0.001450     20090.289062       -35.949218   \n",
       "\n",
       "     Dow Jones Proportion  S&P 500 Value  S&P 500 Delta  S&P 500 Proportion  \n",
       "0               -0.000331    2357.489990      -1.949951           -0.000827  \n",
       "1               -0.000331    2357.489990      -1.949951           -0.000827  \n",
       "2               -0.000331    2357.489990      -1.949951           -0.000827  \n",
       "3               -0.000331    2357.489990      -1.949951           -0.000827  \n",
       "4               -0.000331    2357.489990      -1.949951           -0.000827  \n",
       "5               -0.000331    2357.489990      -1.949951           -0.000827  \n",
       "6               -0.000331    2357.489990      -1.949951           -0.000827  \n",
       "7                0.000717    2352.949951       4.540039            0.001930  \n",
       "8                0.000717    2352.949951       4.540039            0.001930  \n",
       "9                0.000717    2352.949951       4.540039            0.001930  \n",
       "10               0.000717    2352.949951       4.540039            0.001930  \n",
       "11               0.000717    2352.949951       4.540039            0.001930  \n",
       "12               0.000717    2352.949951       4.540039            0.001930  \n",
       "13               0.000717    2352.949951       4.540039            0.001930  \n",
       "14               0.000717    2352.949951       4.540039            0.001930  \n",
       "15               0.000717    2352.949951       4.540039            0.001930  \n",
       "16               0.000717    2352.949951       4.540039            0.001930  \n",
       "17               0.000717    2352.949951       4.540039            0.001930  \n",
       "18               0.000717    2352.949951       4.540039            0.001930  \n",
       "19              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "20              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "21              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "22              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "23              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "24              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "25              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "26              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "27              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "28              -0.001986    2360.159912      -7.209961           -0.003055  \n",
       "29               0.001890    2358.840088       1.319824            0.000560  \n",
       "..                    ...            ...            ...                 ...  \n",
       "433              0.004807    2307.870117       8.229981            0.003566  \n",
       "434              0.004807    2307.870117       8.229981            0.003566  \n",
       "435              0.004807    2307.870117       8.229981            0.003566  \n",
       "436              0.004807    2307.870117       8.229981            0.003566  \n",
       "437              0.004807    2307.870117       8.229981            0.003566  \n",
       "438              0.004807    2307.870117       8.229981            0.003566  \n",
       "439              0.004807    2307.870117       8.229981            0.003566  \n",
       "440              0.004807    2307.870117       8.229981            0.003566  \n",
       "441              0.004807    2307.870117       8.229981            0.003566  \n",
       "442              0.004807    2307.870117       8.229981            0.003566  \n",
       "443              0.004807    2307.870117       8.229981            0.003566  \n",
       "444              0.004807    2307.870117       8.229981            0.003566  \n",
       "445              0.005887    2294.669922      13.200195            0.005753  \n",
       "446              0.005887    2294.669922      13.200195            0.005753  \n",
       "447              0.005887    2294.669922      13.200195            0.005753  \n",
       "448              0.005887    2294.669922      13.200195            0.005753  \n",
       "449              0.005887    2294.669922      13.200195            0.005753  \n",
       "450              0.005887    2294.669922      13.200195            0.005753  \n",
       "451              0.005887    2294.669922      13.200195            0.005753  \n",
       "452              0.005887    2294.669922      13.200195            0.005753  \n",
       "453              0.005887    2294.669922      13.200195            0.005753  \n",
       "454              0.005887    2294.669922      13.200195            0.005753  \n",
       "455              0.005887    2294.669922      13.200195            0.005753  \n",
       "456             -0.001789    2293.080078       1.589844            0.000693  \n",
       "457             -0.001789    2293.080078       1.589844            0.000693  \n",
       "458             -0.001789    2293.080078       1.589844            0.000693  \n",
       "459             -0.001789    2293.080078       1.589844            0.000693  \n",
       "460             -0.001789    2293.080078       1.589844            0.000693  \n",
       "461             -0.001789    2293.080078       1.589844            0.000693  \n",
       "462             -0.001789    2293.080078       1.589844            0.000693  \n",
       "\n",
       "[463 rows x 12 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import complete CSV\n",
    "df = pd.read_csv('data/dataset.csv')\n",
    "# remove all non-alphabet characters and convert to lowercase letters\n",
    "regex = re.compile('[^a-zA-Z]')\n",
    "df['Body'] = df['Body'].apply(lambda post: regex.sub(' ', post).lower())\n",
    "# remove spaces\n",
    "df['Body'] = df['Body'].apply(lambda post: \" \".join([word for word in post.split()]))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Begin cluster analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting tf features for LDA...\n",
      "Fitting LDA models with tf features, n_samples=20000 and n_topics=4...\n",
      "\n",
      "Topics in LDA model:\n",
      "\n",
      "Topic #1:\n",
      "order federal regulatory executive agency agencies actions regulations rule law action act cost state costs review states department agenda government regulation general information year director\n",
      "\n",
      "Topic #2:\n",
      "trump states united security house american law national secretary white donald vice minister mr prime economic administration america court office today leaders new americans countries\n",
      "\n",
      "Topic #3:\n",
      "going thank people great applause know just want mr country good right american like jobs really im think laughter lot years say time said job\n",
      "\n",
      "Topic #4:\n",
      "think mr going people just said know house question administration im want right look make dont white theres thats sure did say does time lot\n",
      "\n"
     ]
    }
   ],
   "source": [
    "n_samples = 20000\n",
    "n_topics = 4\n",
    "n_top_words = 25\n",
    "\n",
    "def print_top_words(model, feature_names, n_top_words):\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        print(\"\\nTopic #{}:\".format(topic_idx + 1))\n",
    "        print(\" \".join([feature_names[i]\n",
    "                        for i in topic.argsort()[:-n_top_words - 1:-1]]))\n",
    "    print()\n",
    "    \n",
    "# Vectorize our text using CountVectorizer\n",
    "print(\"Extracting tf features for LDA...\")\n",
    "tf_vectorizer = CountVectorizer(max_df=0.80, min_df=50,\n",
    "                                max_features=None,\n",
    "                                stop_words='english'\n",
    "                                )\n",
    "\n",
    "tf = tf_vectorizer.fit_transform(df['Body'])\n",
    "\n",
    "print(\"Fitting LDA models with tf features, \"\n",
    "      \"n_samples=%d and n_topics=%d...\"\n",
    "      % (n_samples, n_topics))\n",
    "\n",
    "#define the lda function, with desired options\n",
    "#Check the documentation, linked above, to look through the options\n",
    "lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=20,\n",
    "                                learning_method='online',\n",
    "                                learning_offset=80.,\n",
    "                                total_samples=n_samples,\n",
    "                                random_state=0)\n",
    "#fit the model\n",
    "lda.fit(tf)\n",
    "\n",
    "print(\"\\nTopics in LDA model:\")\n",
    "tf_feature_names = tf_vectorizer.get_feature_names()\n",
    "print_top_words(lda, tf_feature_names, n_top_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### It's hard to tell what some of these topics are, so let's print out the top 5 most correlated titles for each topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topic_dist = lda.transform(tf)\n",
    "topic_dist_df = pd.DataFrame(topic_dist)\n",
    "df_w_topics = topic_dist_df.join(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sorted_topic_1 = df_w_topics[['Title', 'Date', 0]].sort_values(by=[0], ascending=False)\n",
    "sorted_topic_2 = df_w_topics[['Title', 'Date', 1]].sort_values(by=[1], ascending=False)\n",
    "sorted_topic_3 = df_w_topics[['Title', 'Date', 2]].sort_values(by=[2], ascending=False)\n",
    "sorted_topic_4 = df_w_topics[['Title', 'Date', 3]].sort_values(by=[3], ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 titles for topic 1: \n",
      "['Memorandum: Implementing Executive Order 13771, Titled \"Reducing Regulation and Controlling Regulatory Costs\"', 'Memorandum: Spring 2017 Data Call for the Unified Agenda of Federal Regulatory and Deregulatory Actions', 'Presidential Executive Order on Promoting Energy Independence and Economic Growth', 'Presidential Executive Order on Enforcing the Regulatory Reform Agenda', 'Presidential Executive Order on Preventing Violence Against Federal, State, Tribal, and Local Law Enforcement Officers']\n",
      "\n",
      "Top 5 titles for topic 2: \n",
      "['President Donald J. Trump Announces Intent to Nominate Key Administration Posts', 'Joint Statement from President Donald J. Trump and Prime Minister Shinzo Abe', 'President Donald J. Trump Announces Key Administration Posts', 'March: Your Favorite Photos of the Day', 'Joint Readout of Meeting Between President Donald J. Trump and Iraqi Prime Minister Haider al-Abadi']\n",
      "\n",
      "Top 5 titles for topic 3: \n",
      "['Remarks by the President in Meeting with Truckers and CEOS on Healthcare', 'Remarks by President Trump at Unveiling of Boeing 787 Dreamliner Aircraft', 'Remarks by President Trump at Parent-Teacher Conference Listening Session', \"Remarks by President Trump at Women's Empowerment Panel\", 'Remarks by President Trump at the National Museum of African American History and Culture']\n",
      "\n",
      "Top 5 titles for topic 4: \n",
      "['Daily Press Briefing by the Press Secretary -- #27', 'Press Gaggle by Press Secretary Sean Spicer, 2/24/2017', 'Press Briefing by Press Secretary Sean Spicer, 3/20/2017', 'Background Press Briefing on Financial Disclosure Forms', 'Press Gaggle by Press Secretary Sean Spicer']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def top5_titles(lst_sorted_topics):\n",
    "    for i in range(len(lst_sorted_topics)):\n",
    "        print('Top 5 titles for topic {}: '.format(i + 1))\n",
    "        print([topic[0] for topic in lst_sorted_topics[i][:5].values])\n",
    "        print()\n",
    "\n",
    "top5_titles([sorted_topic_1, sorted_topic_2, sorted_topic_3, sorted_topic_4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluation of clusters\n",
    "\n",
    "- The first cluster looks to be about financial regulations and agencies/departments (dampening the powers of certain financial regulatory bodies).\n",
    "\n",
    "- The second cluster could be about strengthening the administration through nominations of officials and meetings with important leaders.\n",
    "\n",
    "- The third cluser is hard to tell. It doesn't seem to be about one specific topic but rather around remarks made by the president.\n",
    "\n",
    "- The fourth topic is also quite hard to tell. You wouldn't be able to tell from the topic modeling but looking at the top 5 titles, the topic seems to surround press briefings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion of Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I am quite surprised by, and impressed with, how well LDA worked for picking out topics, especially the first topic about the economy and regulation. It was pretty obvious from the top words that it was centered around the economy and regulation. The second, third, and fourth were a bit less obvious but intriguing nonetheless. The second topic didn't make much sense to me and looking at the 5 most correlated articles didn't help much either. What I find absolutely fascinating, though, is the third and fourth topic: when looking at the 5 most correlated titles for the third topic, they were all remarks made by the president on different issues while the titles for the fourth topic were all centered around press briefings. \n",
    "\n",
    "This makes me wonder what the topic modeling algorithm is picking up, I see two possibilities:\n",
    "\n",
    "1. The presidential remark posts and the press briefing posts follow the same structure (i.e the posts follow the same structure that usually include some speaking followed by questions and answers).\n",
    "\n",
    "2. Potentially, both President Trump and Sean Spicer have a limited vocabulary that they use when speaking (President Trump has been criticized about a limited vocabulary before - https://www.washingtonpost.com/news/morning-mix/wp/2016/03/18/trumps-grammar-in-speeches-just-below-6th-grade-level-study-finds/)\n",
    "\n",
    "While I think these two topics are fascinating and have generated some hypotheses, I don't believe they are within the scope of our project. It would be interesting to dive deeper into the vocabulary found in President Trump's remarks and press briefings but these don't seem to matter, as the first topic regarding regulations and the economy seems to be what we want to look into more deeply."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What this exploration has done is:\n",
    "\n",
    "1. shown that the administration is, in fact, making an attempt to remove regulations that stifle business and dampen economic growth\n",
    "2. generated a new hypothesis that can be tested: there is a statistically significant difference between the mean of the proportion changes of the top 3 index funds when the administration posts about deregulation/the economy and the mean of the other 3 posts."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python 3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}