twitter_NLP_backup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "The goal of this project is to use twitter streaming API to collect text data, and perform Natural Language Processing (NLP) for sentiment analysis, and do a statistical analysis to see if a tweet in reply to different gender / affiliation /  shows statistically meaningful difference in terms of aggression /insult. The degree of \"aggression/ insult\" in a text are modeled based on https://arxiv.org/ftp/arxiv/papers/1604/1604.06648.pdf\n",
    "https://arxiv.org/pdf/1604.06650.pdf\n",
    "https://arxiv.org/pdf/1702.06877.pdf\n",
    "and references therein.\n",
    "As a pilot survey, we only include 50 significant figures on twitter according to wikipedia (whose gender is known). Accounts for groups / organizations are hand-picked and removed. We only collect tweets that have replies. \n",
    "\n",
    "# Data\n",
    "Uses of twitter APIata collecting and preprocessing step reference:\n",
    "https://marcobonzanini.com/2015/03/09/mining-twitter-data-with-python-part-2/\n",
    "\n",
    "## Training data\n",
    "For training purposes, we started by collecting aggressive users on twitter and their tweets, provided by Despoina Chatzakou (Mean Birds): https://arxiv.org/pdf/1702.06877.pdf\n",
    "However many of the tweets are unaccessible due to user suspension / authorization issues.\n",
    "http://www.yichang-cs.com/yahoo/WWW16_Abusivedetection.pdf and dataset provided therein (e.g. Kaggle challenge) provides insulting comments with verification set.\n",
    "The list of Google-banned bad words are obtained via https://www.freewebheaders.com/full-list-of-bad-words-banned-by-google/\n",
    "We also apply GloVe and see how the result appears.\n",
    "## Test data\n",
    "Using Twitter streaming API, we collect tweets in reply to top 50 most followed users on twitter according to Wikipedia. The size of the dataset is 10k tweets to start with. \n",
    "\n",
    "# Processing tweets\n",
    "We perform standard pre-processing of the tweeter text data, which involves:\n",
    "tokenize, removing stop words, twitter-specific features (e.g. RT, @, ...).\n",
    "\n",
    "# Analysis\n",
    "We test various sentiment analysis here. We use Vader as a starter, to assess the performance of a typical and easy-to-use sentiment analyzer on our training / verification data. We then test various widely used word embedding and algorithm to assess the performance of themover Vader. Finally we apply the algorithm to collected tweets, visualize and understand the result.\n",
    "\n",
    "## Word embedding\n",
    "### TfidfVectorizer\n",
    "CountVectorizer (simple token count)-> TfidfTransformer. Probably more suitable for a large corpus with consistent context.\n",
    "### GloVe\n",
    "pre-trained unsupervised word clustering / vetorization of words provided by Stanford group.\n",
    "\n",
    "## Classification\n",
    "### Vader \n",
    "provides pre-trained positive / negtaive sentiment analyzer. Tweets can be classifies and the intensity of the sentiment is returned. \n",
    "### Logistic Regression\n",
    "### NaiveBayse\n",
    "### RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TIP: Install a pip package in the current Jupyter kernel\n",
    "#import sys\n",
    "#!{sys.executable} -m pip install numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import json\n",
    "except ImportError:\n",
    "    import simplejson as json\n",
    "\n",
    "# Import the necessary methods from \"twitter\" library\n",
    "# from twitter import Twitter, OAuth, TwitterHTTPError, TwitterStream"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['katyperry', 'justinbieber', 'BarackObama', 'rihanna', 'taylorswift13', 'ladygaga', 'TheEllenShow', 'Cristiano', 'YouTube', 'jtimberlake', 'twitter', 'KimKardashian', 'britneyspears', 'ArianaGrande', 'ddlovato', 'selenagomez', 'cnnbrk', 'realDonaldTrump', 'shakira', 'jimmyfallon', 'BillGates', 'JLo', 'narendramodi', 'BrunoMars', 'Oprah', 'nytimes', 'KingJames', 'MileyCyrus', 'CNN', 'NiallOfficial', 'neymarjr', 'instagram', 'BBCBreaking', 'Drake', 'iamsrk', 'SportsCenter', 'KevinHart4real', 'SrBachchan', 'LilTunechi', 'espn', 'wizkhalifa', 'BeingSalmanKhan', 'Louis_Tomlinson', 'Pink', 'LiamPayne', 'Harry_Styles', 'onedirection', 'aliciakeys', 'realmadrid', 'KAKA']\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "# Obtain list of 50 most followed people worldwide from wikipedia\n",
    "#import wikipedia as wp\n",
    " \n",
    "#Get the html source\n",
    "#html = wp.page(\"List of most-followed Twitter accounts\").html().encode(\"UTF-8\")\n",
    "#df = pd.read_html(html)[0]\n",
    "#df.to_csv('twitter_list_of_influencers.csv',header=0,index=False)\n",
    "#user_list = list(df[2])[1:]\n",
    "# remove '@' infront of the screen names and save the list as user_list\n",
    "#user_list = [a[1:] for a in user_list]\n",
    "#print(user_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Variables that contains the user credentials to access Twitter API \n",
    "#ACCESS_TOKEN = '187549975-3LD41YaLCw3XnOvRUJVkqXrjt6gMsuT1HrEUlqDi'\n",
    "#ACCESS_SECRET = 'ZykwgJpFYQaZdP6vEFHtbqBfkQfgBo9mV0LM3MkmEp5Oj'\n",
    "#CONSUMER_KEY = 'qQTzJ4OceyUFMLcWkn7ZZ5Wrp'\n",
    "#CONSUMER_SECRET = 'z6aV6zjg2yac1TiZw5ERUvZ2XLXAGBQJ1OD0yUCFtCczWNKjik'\n",
    "\n",
    "#oauth = OAuth(ACCESS_TOKEN, ACCESS_SECRET, CONSUMER_KEY, CONSUMER_SECRET)\n",
    "\n",
    "#t = Twitter(auth=oauth)\n",
    "#id_list = t.users.lookup(screen_name=','.join(user_list))\n",
    "# user_list is the list of screen names: we need to convert this to user id using twitter API for filtering purposes\n",
    "# filter out non-person accounts, non-english accounts and obtain tweets that have replies by hand\n",
    "# maybe I need to pick out overly controvertial accounts as well.........."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['21447363', '27260086', '813286', '79293791', '17919972', '14230524', '15846407', '155659213', '10228272', '26565946', '783214', '25365536', '16409683', '34507480', '21111883', '23375688', '428333', '25073877', '44409004', '15485441', '50393960', '85603854', '18839785', '100220864', '19397785', '807095', '23083404', '268414482', '759251', '105119490', '158487331', '180505807', '5402612', '27195114', '101311381', '26257166', '23151437', '145125358', '116362700', '2557521', '20322929', '132385468', '84279963', '28706024', '158314798', '181561712', '209708391', '35094637', '14872237', '60865434']\n"
     ]
    }
   ],
   "source": [
    "#user_id = [id_list[x]['id_str'] for x in range(50)]\n",
    "#print(user_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file exists\n"
     ]
    }
   ],
   "source": [
    "# Initiate the connection to Twitter Streaming API\n",
    "#twitter_stream = TwitterStream(auth=oauth)\n",
    "#iterator = twitter_stream.statuses.filter(lang='en',follow=','.join(user_id))\n",
    "\n",
    "# Get a sample of the public data following through Twitter\n",
    "# iterator = twitter_stream.statuses.sample()\n",
    "# As a pilot survey we set it to stop after getting 1000 tweets. \n",
    "# You don't have to set it to stop, but can continue running \n",
    "# the Twitter API to collect data for days or even longer. \n",
    "\n",
    "# Collecting data can take a long time, so I separately implemented a notebook for stream twitter.\n",
    "# This code reads file generated from Collect_tweets_50mostpop_users.ipynb\n",
    "tweets_filename = 'twitter_savereplies_nsamp10000.json'\n",
    "#tweets_file = open(tweets_filename, \"r\")\n",
    "import os.path\n",
    "# find out a way to filter replies only:\n",
    "# save to json file\n",
    "tweet_count = 100\n",
    "tweet_cnt = tweet_count\n",
    "try:\n",
    "    os.path.isfile(tweets_filename)\n",
    "    print(\"file exists\")\n",
    "    with open(tweets_filename, 'r') as f:\n",
    "        tweets = json.load(f) # readline only the first tweet/line\n",
    "    pass\n",
    "except:\n",
    "    print(\"file does not exist, stream twitter for collecting tweets\")\n",
    "    tweets = []\n",
    "    for tweet in iterator:\n",
    "    # select only \"replies\" to top 50 followed users\n",
    "        if str(tweet['in_reply_to_user_id']) in user_id:\n",
    "        #print(tweet['in_reply_to_user_id'])\n",
    "            tweet_count -= 1\n",
    "            tweets.append(tweet)\n",
    "    # Twitter Python Tool wraps the data returned by Twitter \n",
    "    # as a TwitterDictResponse object.\n",
    "    # We convert it back to the JSON format to print/score\n",
    "    # loads converts json format to python dictionary\n",
    "    # dumps converts python dictionary to json format\n",
    "    # The command below will do pretty printing for JSON data, try it out\n",
    "    # print json.dumps(tweet, indent=4)\n",
    "            if tweet_count <= 0:\n",
    "                break     \n",
    "    with open(tweets_filename, 'w') as outfile:\n",
    "        json.dump(tweets,outfile,indent=4)\n",
    "text = []\n",
    "for i0 in range(tweet_cnt):\n",
    "    text.append(tweets[i0]['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25073877\n"
     ]
    }
   ],
   "source": [
    "print(tweets[1]['in_reply_to_user_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@realDonaldTrump when are you going to pay the troops in harms way a visit. Cannot think of a single POTUS who hasn… https://t.co/3nWb4cyJlB\n"
     ]
    }
   ],
   "source": [
    "print(tweets[50]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing the text\n",
    "Using nltk, we pre-process the text here. Tokenized words will be feed into multiple classifier to determine the degree of aggression in the text. We compare their performances and decide which algorithm to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "#print(word_tokenize(text[0]))\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "tweet_tokenizer = TweetTokenizer(preserve_case=False, strip_handles=True, reduce_len=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    " \n",
    "emoticons_str = r\"\"\"\n",
    "    (?:\n",
    "        [:=;] # Eyes\n",
    "        [oO\\-]? # Nose (optional)\n",
    "        [D\\)\\]\\(\\]/\\\\OpP] # Mouth\n",
    "    )\"\"\"\n",
    " \n",
    "regex_str = [\n",
    "    emoticons_str,\n",
    "    r'<[^>]+>', # HTML tags\n",
    "    r'(?:@[\\w_]+)', # @-mentions\n",
    "    r\"(?:\\#+[\\w_]+[\\w\\'_\\-]*[\\w_]+)\", # hash-tags\n",
    "    r'http[s]?://(?:[a-z]|[0-9]|[$-_@.&amp;+]|[!*\\(\\),]|(?:%[0-9a-f][0-9a-f]))+', # URLs\n",
    " \n",
    "    r'(?:(?:\\d+,?)+(?:\\.?\\d+)?)', # numbers\n",
    "    r\"(?:[a-z][a-z'\\-_]+[a-z])\", # words with - and '\n",
    "    r'(?:[\\w_]+)', # other words\n",
    "    r'(?:\\S)' # anything else\n",
    "]\n",
    "    \n",
    "tokens_re = re.compile(r'('+'|'.join(regex_str)+')', re.VERBOSE | re.IGNORECASE)\n",
    "emoticon_re = re.compile(r'^'+emoticons_str+'$', re.VERBOSE | re.IGNORECASE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import string\n",
    "import re\n",
    " \n",
    "from nltk.corpus import stopwords \n",
    "stopwords_english = stopwords.words('english')\n",
    " \n",
    "from nltk.stem import PorterStemmer, WordNetLemmatizer\n",
    "stemmer = PorterStemmer()\n",
    "wnl = WordNetLemmatizer()\n",
    "\n",
    "from nltk.tokenize import TweetTokenizer\n",
    " \n",
    "# Happy Emoticons\n",
    "emoticons_happy = set([\n",
    "    ':-)', ':)', ';)', ':o)', ':]', ':3', ':c)', ':>', '=]', '8)', '=)', ':}',\n",
    "    ':^)', ':-D', ':D', '8-D', '8D', 'x-D', 'xD', 'X-D', 'XD', '=-D', '=D',\n",
    "    '=-3', '=3', ':-))', \":'-)\", \":')\", ':*', ':^*', '>:P', ':-P', ':P', 'X-P',\n",
    "    'x-p', 'xp', 'XP', ':-p', ':p', '=p', ':-b', ':b', '>:)', '>;)', '>:-)',\n",
    "    '<3'\n",
    "    ])\n",
    " \n",
    "# Sad Emoticons\n",
    "emoticons_sad = set([\n",
    "    ':L', ':-/', '>:/', ':S', '>:[', ':@', ':-(', ':[', ':-||', '=L', ':<',\n",
    "    ':-[', ':-<', '=\\\\', '=/', '>:(', ':(', '>.<', \":'-(\", \":'(\", ':\\\\', ':-c',\n",
    "    ':c', ':{', '>:\\\\', ';('\n",
    "    ])\n",
    " \n",
    "# all emoticons (happy + sad)\n",
    "emoticons = emoticons_happy.union(emoticons_sad)\n",
    " \n",
    "def clean_tweets(tweet):\n",
    "    # remove stock market tickers like $GE\n",
    "    tweet = re.sub(r'\\$\\w*', '', tweet)\n",
    "    # remove old style retweet text \"RT\"\n",
    "    tweet = re.sub(r'^RT[\\s]+', '', tweet)\n",
    "    # remove hyperlinks\n",
    "    tweet = re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', tweet)\n",
    "    # remove hashtags\n",
    "    # only removing the hash # sign from the word\n",
    "    tweet = re.sub(r'#', '', tweet)    \n",
    "    # remove abbreviation marks?\n",
    "    tweet = re.sub(r'/[.]{2,}/g','',tweet)\n",
    "    # tokenize tweets\n",
    "    tokenizer = TweetTokenizer(preserve_case=False, strip_handles=True, reduce_len=True)\n",
    "    tweet_tokens = tokenizer.tokenize(tweet)\n",
    " \n",
    "    tweets_clean = []    \n",
    "    for word in tweet_tokens:\n",
    "        if (word not in stopwords_english and # remove stopwords\n",
    "              word not in emoticons and # remove emoticons\n",
    "                word not in string.punctuation): # remove punctuation\n",
    "            #tweets_clean.append(word)\n",
    "            #stem_word = stemmer.stem(word) \n",
    "            # stemming word : tend to remove 'e' from ending of some words. replaced by WordNetLemmatizer\n",
    "            wnl_word = wnl.lemmatize(word)\n",
    "            #tweets_clean.append(stem_word)\n",
    "            tweets_clean.append(wnl_word)\n",
    " \n",
    "    return tweets_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def preprocess(s, lowercase=False):\n",
    "    tokens = tokenize(s)\n",
    "    if lowercase:\n",
    "        tokens = [token if emoticon_re.search(token) else token.lower() for token in tokens]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['@realDonaldTrump', 'when', 'are', 'you', 'going', 'to', 'pay', 'the', 'troops', 'in', 'harms', 'way', 'a', 'visit', '.', 'Cannot', 'think', 'of', 'a', 'single', 'POTUS', 'who', 'hasn', '…', 'https://t.co/3nWb4cyJlB']\n",
      "['going', 'pay', 'troop', 'harm', 'way', 'visit', 'cannot', 'think', 'single', 'potus', '…']\n"
     ]
    }
   ],
   "source": [
    "print(preprocess(text[50]))\n",
    "print(clean_tweets(text[50]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.text import text_to_word_sequence\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.models import Sequential\n",
    "\n",
    "from keras.layers import Input, Dense, Embedding, Conv1D, Conv2D, MaxPooling1D, MaxPool2D\n",
    "from keras.layers import Reshape, Flatten, Dropout, Concatenate\n",
    "from keras.layers import SpatialDropout1D, concatenate\n",
    "from keras.layers import GRU, Bidirectional, GlobalAveragePooling1D, GlobalMaxPooling1D\n",
    "\n",
    "from keras.callbacks import Callback\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from keras.models import load_model\n",
    "from keras.utils.vis_utils import plot_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_coefs(word, *arr):\n",
    "    try:\n",
    "        return word, np.asarray(arr, dtype='float32')\n",
    "    except:\n",
    "        return None, None\n",
    "    \n",
    "embeddings_index = dict(get_coefs(*o.strip().split()) for o in open('./GloVe/glove.twitter.27B.200d.txt'))\n",
    "\n",
    "embed_size=200\n",
    "for k in list(embeddings_index.keys()):\n",
    "    v = embeddings_index[k]\n",
    "    try:\n",
    "        if v.shape != (embed_size, ):\n",
    "            embeddings_index.pop(k)\n",
    "    except:\n",
    "        pass\n",
    "            \n",
    "#embeddings_index.pop(None)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "values = list(embeddings_index.values())\n",
    "all_embs = np.stack(values)\n",
    "\n",
    "emb_mean, emb_std = all_embs.mean(), all_embs.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#def tokenize(s):\n",
    "#    return tokens_re.findall(s)\n",
    "def tokenize(tweet):\n",
    "    tweet = re.sub(r'http\\S+', '', tweet)\n",
    "    tweet = re.sub(r\"#(\\w+)\", '', tweet)\n",
    "    tweet = re.sub(r\"@(\\w+)\", '', tweet)\n",
    "    tweet = re.sub(r'[^\\w\\s]', '', tweet)\n",
    "    tweet = tweet.strip().lower()\n",
    "    tokens = word_tokenize(tweet)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "for some reason u sound retarded lol damn where u been negro\n",
      "[[17, 82, 419, 56, 138, 195, 141, 305, 162, 56, 154, 823]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ijee/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "/home/ijee/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:6: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n"
     ]
    }
   ],
   "source": [
    "MAX_NB_WORDS = 80000\n",
    "tokenizer = Tokenizer(num_words=MAX_NB_WORDS,char_level=False)\n",
    "\n",
    "\n",
    "insult['tokens'] = insult.Comment.map(tokenize)\n",
    "insult['cleaned_text'] = insult['tokens'].map(lambda tokens: ' '.join(tokens))\n",
    "print(insult['cleaned_text'][15])\n",
    "tokenizer.fit_on_texts(insult['cleaned_text'])\n",
    "print(tokenizer.texts_to_sequences([insult['cleaned_text'][15]]))\n",
    "#print(insult['cleaned_text'][15])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_sequences = tokenizer.texts_to_sequences(train_x)\n",
    "test_sequences = tokenizer.texts_to_sequences(ver_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MAX_LENGTH = 35\n",
    "\n",
    "padded_train_sequences = pad_sequences(train_sequences, maxlen=MAX_LENGTH)\n",
    "padded_test_sequences = pad_sequences(test_sequences, maxlen=MAX_LENGTH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1089\n",
      "Train on 3947 samples, validate on 2235 samples\n",
      "Epoch 1/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.6243 - acc: 0.7012Epoch 00000: val_acc improved from -inf to 0.51812, saving model to ./models/rnn_with_embeddings/weights-improvement-00-0.5181-200.hdf5\n",
      "3947/3947 [==============================] - 24s - loss: 0.6128 - acc: 0.7097 - val_loss: 0.8499 - val_acc: 0.5181\n",
      "Epoch 2/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.5471 - acc: 0.7436Epoch 00001: val_acc improved from 0.51812 to 0.51902, saving model to ./models/rnn_with_embeddings/weights-improvement-01-0.5190-200.hdf5\n",
      "3947/3947 [==============================] - 19s - loss: 0.5450 - acc: 0.7426 - val_loss: 0.7477 - val_acc: 0.5190\n",
      "Epoch 3/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.5009 - acc: 0.7416Epoch 00002: val_acc improved from 0.51902 to 0.52573, saving model to ./models/rnn_with_embeddings/weights-improvement-02-0.5257-200.hdf5\n",
      "3947/3947 [==============================] - 20s - loss: 0.4993 - acc: 0.7406 - val_loss: 0.7259 - val_acc: 0.5257\n",
      "Epoch 4/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.4522 - acc: 0.7793Epoch 00003: val_acc improved from 0.52573 to 0.60895, saving model to ./models/rnn_with_embeddings/weights-improvement-03-0.6089-200.hdf5\n",
      "3947/3947 [==============================] - 20s - loss: 0.4498 - acc: 0.7836 - val_loss: 0.6588 - val_acc: 0.6089\n",
      "Epoch 5/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.4004 - acc: 0.8136Epoch 00004: val_acc improved from 0.60895 to 0.66756, saving model to ./models/rnn_with_embeddings/weights-improvement-04-0.6676-200.hdf5\n",
      "3947/3947 [==============================] - 20s - loss: 0.3994 - acc: 0.8138 - val_loss: 0.6241 - val_acc: 0.6676\n",
      "Epoch 6/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.3506 - acc: 0.8613Epoch 00005: val_acc improved from 0.66756 to 0.68770, saving model to ./models/rnn_with_embeddings/weights-improvement-05-0.6877-200.hdf5\n",
      "3947/3947 [==============================] - 20s - loss: 0.3484 - acc: 0.8629 - val_loss: 0.6388 - val_acc: 0.6877\n",
      "Epoch 7/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.3094 - acc: 0.8722Epoch 00006: val_acc improved from 0.68770 to 0.71812, saving model to ./models/rnn_with_embeddings/weights-improvement-06-0.7181-200.hdf5\n",
      "3947/3947 [==============================] - 20s - loss: 0.3064 - acc: 0.8736 - val_loss: 0.6218 - val_acc: 0.7181\n",
      "Epoch 8/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.2863 - acc: 0.8772Epoch 00007: val_acc improved from 0.71812 to 0.72304, saving model to ./models/rnn_with_embeddings/weights-improvement-07-0.7230-200.hdf5\n",
      "3947/3947 [==============================] - 20s - loss: 0.2807 - acc: 0.8802 - val_loss: 0.6540 - val_acc: 0.7230\n",
      "Epoch 9/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.2617 - acc: 0.8898Epoch 00008: val_acc improved from 0.72304 to 0.72886, saving model to ./models/rnn_with_embeddings/weights-improvement-08-0.7289-200.hdf5\n",
      "3947/3947 [==============================] - 20s - loss: 0.2581 - acc: 0.8926 - val_loss: 0.6807 - val_acc: 0.7289\n",
      "Epoch 10/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.2385 - acc: 0.8984Epoch 00009: val_acc did not improve\n",
      "3947/3947 [==============================] - 19s - loss: 0.2404 - acc: 0.8969 - val_loss: 0.6769 - val_acc: 0.7266\n",
      "Epoch 11/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.2212 - acc: 0.9076Epoch 00010: val_acc did not improve\n",
      "3947/3947 [==============================] - 19s - loss: 0.2228 - acc: 0.9070 - val_loss: 0.7968 - val_acc: 0.7038\n",
      "Epoch 12/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.2066 - acc: 0.9169Epoch 00011: val_acc did not improve\n",
      "3947/3947 [==============================] - 22s - loss: 0.2077 - acc: 0.9164 - val_loss: 0.7169 - val_acc: 0.7262\n",
      "Epoch 13/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1938 - acc: 0.9266Epoch 00012: val_acc improved from 0.72886 to 0.72886, saving model to ./models/rnn_with_embeddings/weights-improvement-12-0.7289-200.hdf5\n",
      "3947/3947 [==============================] - 21s - loss: 0.1935 - acc: 0.9263 - val_loss: 0.7180 - val_acc: 0.7289\n",
      "Epoch 14/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1736 - acc: 0.9339Epoch 00013: val_acc did not improve\n",
      "3947/3947 [==============================] - 19s - loss: 0.1713 - acc: 0.9336 - val_loss: 0.7764 - val_acc: 0.7204\n",
      "Epoch 15/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1629 - acc: 0.9403Epoch 00014: val_acc did not improve\n",
      "3947/3947 [==============================] - 19s - loss: 0.1613 - acc: 0.9405 - val_loss: 0.7832 - val_acc: 0.7190\n",
      "Epoch 16/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1440 - acc: 0.9453Epoch 00015: val_acc did not improve\n",
      "3947/3947 [==============================] - 19s - loss: 0.1473 - acc: 0.9427 - val_loss: 0.8663 - val_acc: 0.7110\n",
      "Epoch 17/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1359 - acc: 0.9520Epoch 00016: val_acc did not improve\n",
      "3947/3947 [==============================] - 20s - loss: 0.1373 - acc: 0.9508 - val_loss: 0.8261 - val_acc: 0.7230\n",
      "Epoch 18/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1273 - acc: 0.9542Epoch 00017: val_acc did not improve\n",
      "3947/3947 [==============================] - 19s - loss: 0.1226 - acc: 0.9564 - val_loss: 0.8392 - val_acc: 0.7221\n",
      "Epoch 19/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1171 - acc: 0.9593Epoch 00018: val_acc did not improve\n",
      "3947/3947 [==============================] - 19s - loss: 0.1175 - acc: 0.9595 - val_loss: 0.8499 - val_acc: 0.7253\n",
      "Epoch 20/20\n",
      "3584/3947 [==========================>...] - ETA: 1s - loss: 0.1089 - acc: 0.9615Epoch 00019: val_acc did not improve\n",
      "3947/3947 [==============================] - 21s - loss: 0.1061 - acc: 0.9622 - val_loss: 0.9276 - val_acc: 0.7101\n"
     ]
    }
   ],
   "source": [
    "\n",
    "word_index = tokenizer.word_index\n",
    "nb_words = MAX_NB_WORDS\n",
    "embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n",
    "\n",
    "oov = 0\n",
    "for word, i in word_index.items():\n",
    "    if i >= MAX_NB_WORDS: continue\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        embedding_matrix[i] = embedding_vector\n",
    "    else:\n",
    "        oov += 1\n",
    "\n",
    "print(oov)\n",
    "\n",
    "embedding_dim = 200\n",
    "def get_rnn_model_with_glove_embeddings():\n",
    "    inp = Input(shape=(MAX_LENGTH, ))\n",
    "    x = Embedding(MAX_NB_WORDS, embedding_dim, weights=[embedding_matrix], input_length=MAX_LENGTH, trainable=True)(inp)\n",
    "    x = SpatialDropout1D(0.3)(x)\n",
    "    x = Bidirectional(GRU(100, return_sequences=True))(x)\n",
    "    avg_pool = GlobalAveragePooling1D()(x)\n",
    "    max_pool = GlobalMaxPooling1D()(x)\n",
    "    conc = concatenate([avg_pool, max_pool])\n",
    "    outp = Dense(1, activation=\"sigmoid\")(conc)\n",
    "    \n",
    "    model = Model(inputs=inp, outputs=outp)\n",
    "    model.compile(loss='binary_crossentropy',\n",
    "                  optimizer='adam',\n",
    "                  metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "rnn_model_with_embeddings = get_rnn_model_with_glove_embeddings()\n",
    "\n",
    "filepath=\"./models/rnn_with_embeddings/weights-improvement-{epoch:02d}-{val_acc:.4f}-%03d.hdf5\"%(embedding_dim)\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')\n",
    "\n",
    "batch_size = 512\n",
    "epochs = 20\n",
    "\n",
    "\n",
    "history = rnn_model_with_embeddings.fit(x=padded_train_sequences, \n",
    "                    y=train_y, \n",
    "                    validation_data=(padded_test_sequences, ver_y), \n",
    "                    batch_size=batch_size, \n",
    "                    callbacks=[checkpoint], \n",
    "                    epochs=epochs, \n",
    "                    verbose=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2235/2235 [==============================] - 4s     \n"
     ]
    }
   ],
   "source": [
    "best_rnn_model_with_glove_embeddings = load_model('./models/rnn_with_embeddings/weights-improvement-12-0.7289-200.hdf5')\n",
    "\n",
    "y_pred_rnn_with_glove_embeddings = best_rnn_model_with_glove_embeddings.predict(\n",
    "    padded_test_sequences, verbose=1, batch_size=2048)\n",
    "\n",
    "y_pred_rnn_with_glove_embeddings = pd.DataFrame(y_pred_rnn_with_glove_embeddings, columns=['prediction'])\n",
    "y_pred_rnn_with_glove_embeddings['prediction'] = y_pred_rnn_with_glove_embeddings['prediction'].map(lambda p: \n",
    "                                                                                                    1 if p >= 0.5 else 0)\n",
    "y_pred_rnn_with_glove_embeddings.to_csv('./predictions/y_pred_rnn_with_glove_embeddings.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.728859060403\n"
     ]
    }
   ],
   "source": [
    "y_pred_rnn_with_glove_embeddings = pd.read_csv('./predictions/y_pred_rnn_with_glove_embeddings.csv')\n",
    "print(accuracy_score(ver_y, y_pred_rnn_with_glove_embeddings))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "##### Initiate the connection to Twitter REST API\n",
    "#twitter = Twitter(auth=oauth)\n",
    "            \n",
    "# Search for latest tweets about \"#nlproc\"\n",
    "#twitter.search.tweets(q='#WorldCup') \n",
    "#a = twitter.search.tweets(q='#WorldCup',geocode='30.357245,-97.7611217,1000km',result_type='recent')  #Austin\n",
    "#print('Austin searches: '+str(len(a['statuses'][:])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#a = twitter.search.tweets(q='instagram',geocode='30.357245,-97.7611217,100km',count=100,since='2016-06-01')  #Austin\n",
    "#print('Austin searches: '+str(len(a['statuses'][:])))\n",
    "#print(a['statuses'][-1]['created_at'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#import wget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#wget.download(a['statuses'][3]['entities']['urls'][0]['expanded_url'],'/home/ijee/2018_fall/twitter_region_language_timestamp')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#twitter = Twitter(auth=oauth)\n",
    "#WhitePower = twitter.search.tweets(q='#WhitePower',count=100,lang='en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#print(WhitePower['statuses'][4]['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "training_file = './dataset/data'\n",
    "training_data = pd.read_csv(training_file,delim_whitespace=True,names=['user','category','tweet_id'])\n",
    "ID = []\n",
    "for i in range(len(training_data)):\n",
    "    ID.append(training_data['tweet_id'][i].split(\",\"))\n",
    "#print(training_data)\n",
    "# category is divided into four: aggressor, bully, normal and spammer\n",
    "# each up to (43, 101, 883, 1303) id indices and 5-10 tweets.\n",
    "# It can take long time to collect these tweets, so I save them as json files with their indices provided in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# saved aggressive / bullying tweets for later runs\n",
    "agg_filename = 'agg_tweets.json'\n",
    "bull_filename= 'bull_tweets.json'\n",
    "norm_filename = 'norm_tweets.json' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "agg_file exist: read and pass\n"
     ]
    }
   ],
   "source": [
    "#t.statuses.oembed(_id=672282716436467714)\n",
    "try:\n",
    "    os.path.isfile(agg_filename)\n",
    "    with open(agg_filename, 'r') as f:\n",
    "        agg_tweets = json.load(f) # readline only the first tweet/line\n",
    "    print(\"agg_file exist: read and pass\")\n",
    "    pass\n",
    "except:\n",
    "    print(\"agg_file does not exist: collect tweets from twitter database\")\n",
    "    agg_tweets = []\n",
    "    agg_index = []\n",
    "    ID_idx = [len(x) for x in ID]\n",
    "    for i in range(0,43):\n",
    "        for j in range(ID_idx[i]):\n",
    "            try:\n",
    "                agg_tweets.append(t.statuses.show(_id=ID[i][j]))\n",
    "                agg_index.append(i)\n",
    "                print(i,j)\n",
    "            except:\n",
    "                pass\n",
    "    with open(agg_filename, 'w') as outfile:\n",
    "        json.dump(agg_tweets,outfile,indent=4)\n",
    "    with open('agg_tweets_idx.txt','w') as outfile:\n",
    "        json.dump(agg_index,outfile)\n",
    "    print(\"aggressive tweets collected and saved\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bull_file exist: read and pass\n"
     ]
    }
   ],
   "source": [
    "#t.statuses.oembed(_id=672282716436467714)\n",
    "try:\n",
    "    os.path.isfile(bull_filename)\n",
    "    with open(bull_filename, 'r') as f:\n",
    "        bull_tweets = json.load(f) # readline only the first tweet/line\n",
    "    print(\"bull_file exist: read and pass\")\n",
    "    pass\n",
    "except:\n",
    "    print(\"bull_file does not exist: collect tweets from twitter database\")\n",
    "    ID_idx = [len(x) for x in ID]\n",
    "    bull_tweets = []\n",
    "    bull_index = []\n",
    "    for i in range(43,101):\n",
    "        for j in range(ID_idx[i]):\n",
    "            try:\n",
    "                bull_tweets.append(t.statuses.show(_id=ID[i][j]))\n",
    "                bull_index.append(i)\n",
    "                print(i,j)\n",
    "            except:\n",
    "                pass\n",
    "    with open(bull_filename, 'w') as outfile:\n",
    "        json.dump(bull_tweets,outfile,indent=4)\n",
    "    with open('bull_tweets_idx.txt','w') as outfile:\n",
    "        json.dump(bull_index,outfile)\n",
    "    print(\"bully tweets collected and saved\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "norm_file exist: read and pass\n"
     ]
    }
   ],
   "source": [
    "#t.statuses.oembed(_id=672282716436467714)\n",
    "try:\n",
    "    os.path.isfile(norm_filename)\n",
    "    with open(norm_filename, 'r') as f:\n",
    "        norm_tweets = json.load(f) # readline only the first tweet/line\n",
    "    print(\"norm_file exist: read and pass\")\n",
    "    pass\n",
    "except:\n",
    "    print(\"norm_file does not exist: collect tweets from twitter database\")\n",
    "    norm_tweets = []\n",
    "    norm_index = []\n",
    "    ID_idx = [len(x) for x in ID]\n",
    "    for i in range(101,883):\n",
    "        for j in range(ID_idx[i]):\n",
    "            try:\n",
    "                norm_tweets.append(t.statuses.show(_id=ID[i][j]))\n",
    "                norm_index.append(i)\n",
    "                print(i,j)\n",
    "            except:\n",
    "                pass\n",
    "    with open(norm_filename, 'w') as outfile:\n",
    "        json.dump(norm_tweets,outfile,indent=4)\n",
    "    with open('norm_tweets_idx.txt','w') as outfile:\n",
    "        json.dump(norm_index,outfile)\n",
    "    print(\"normal tweets collected and saved\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print((bull_tweets[10]['in_reply_to_user_id']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "norm_tweets_text = [norm_tweets[i]['text'] for i in range(len(norm_tweets))]\n",
    "# tweets from aggressive + bullying users currently available are about 100... :x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# tweeter seems to have worked well and suspended most of the accounts the authors classified as aggressors / bullies,\n",
    "# which is unfortunate for our purpose but we will proceed with the currently available data...?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# External dataset\n",
    "Due to the lack of twitter data, we use external data to train the network. One is from Kaggle challenge, and the other is from banned words by Google (not a official list, obtained from https://www.freewebheaders.com/full-list-of-bad-words-banned-by-google/). Kaggle data also provide verification set with labels separatly provided, so we will proceed and test our model on the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "banwords_file = './Google_BanWords/full-list-of-bad-words-text-file_2018_03_26.txt'\n",
    "insult_corpus_file = './kaggle_insult/train.csv'\n",
    "insult_train = pd.read_csv(insult_corpus_file)\n",
    "banwords = pd.read_csv(banwords_file,sep='\\r\\n',engine='python',names = ['banword'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"You fuck your dad.\"\n",
      "2898\n",
      "['either', 'fake', 'extremely', 'stupid', '...', 'maybe', '...']\n"
     ]
    }
   ],
   "source": [
    "#insult_train.head(4)\n",
    "#banwords['word'][462]\n",
    "insult = insult_train.loc[insult_train['Insult']==1]\n",
    "normal = insult_train.loc[insult_train['Insult']==0]\n",
    "print((insult['Comment'][0]))\n",
    "print(len(normal))\n",
    "print(clean_tweets(insult['Comment'][8]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   id  Insult             Date  \\\n",
      "0   1       0  20120603163526Z   \n",
      "1   2       1  20120531215447Z   \n",
      "2   3       1  20120823164228Z   \n",
      "3   4       1  20120826010752Z   \n",
      "4   5       1  20120602223825Z   \n",
      "\n",
      "                                             Comment        Usage  \n",
      "0                 \"like this if you are a tribe fan\"  PrivateTest  \n",
      "1              \"you're idiot.......................\"  PrivateTest  \n",
      "2  \"I am a woman Babs, and the only \"war on women...  PrivateTest  \n",
      "3  \"WOW & YOU BENEFITTED SO MANY WINS THIS YEAR F...  PrivateTest  \n",
      "4  \"haha green me red you now loser whos winning ...  PrivateTest  \n"
     ]
    }
   ],
   "source": [
    "# let's import verification data from the Kaggle challenge\n",
    "ver_file = './kaggle_insult/impermium_verification_labels.csv'\n",
    "ver_data = pd.read_csv(ver_file)\n",
    "print(ver_data.head(5))\n",
    "\n",
    "insult = ver_data.loc[ver_data['Insult']==1]\n",
    "normal = ver_data.loc[ver_data['Insult']==0]\n",
    "\n",
    "insult_ver_set = []\n",
    "for tweet in insult['Comment']:\n",
    "    insult_ver_set.append((bag_of_words(tweet), 'ins'))\n",
    "    \n",
    "norm_ver_set = []\n",
    "for tweet in normal['Comment']:\n",
    "    norm_ver_set.append((bag_of_words(tweet), 'norm'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we try uni-/bigram (word / character) approach here following \n",
    "# https://ahmedbesbes.com/overview-and-benchmark-of-traditional-and-deep-learning-models-in-text-classification.html\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score, auc, roc_auc_score\n",
    "from sklearn.externals import joblib\n",
    "\n",
    "import scipy\n",
    "from scipy.sparse import hstack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Try TFIDF \n",
    "train_x = insult_train['Comment'][:]\n",
    "train_y = insult_train['Insult'][:]\n",
    "ver_x = ver_data['Comment'][:]\n",
    "ver_y = ver_data['Insult'][:]\n",
    "vectorizer_word = TfidfVectorizer(max_features=40000,\n",
    "                             min_df=5, \n",
    "                             max_df=0.5, \n",
    "                             analyzer='word', \n",
    "                             stop_words='english', \n",
    "                             ngram_range=(1, 2))\n",
    "\n",
    "vectorizer_word.fit(train_x)\n",
    "\n",
    "tfidf_matrix_word_train = vectorizer_word.transform(train_x)\n",
    "tfidf_matrix_word_test = vectorizer_word.transform(ver_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "convergence after 19 epochs took 0 seconds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='sag', tol=0.0001,\n",
       "          verbose=2, warm_start=False)"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_word = LogisticRegression(solver='sag', verbose=2)\n",
    "lr_word.fit(tfidf_matrix_word_train, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "joblib.dump(lr_word, 'lr_word_ngram.pkl')\n",
    "\n",
    "y_pred_word = lr_word.predict(tfidf_matrix_word_test)\n",
    "pd.DataFrame(y_pred_word, columns=['y_pred']).to_csv('lr_word_ngram.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.656823266219\n"
     ]
    }
   ],
   "source": [
    "y_pred_word = pd.read_csv('lr_word_ngram.csv')\n",
    "print(accuracy_score(ver_y, y_pred_word))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "convergence after 17 epochs took 0 seconds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.3s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.3s finished\n"
     ]
    }
   ],
   "source": [
    "vectorizer_char = TfidfVectorizer(max_features=40000,\n",
    "                             min_df=5, \n",
    "                             max_df=0.5, \n",
    "                             analyzer='char', \n",
    "                             ngram_range=(1, 4))\n",
    "\n",
    "vectorizer_char.fit(train_x);\n",
    "\n",
    "tfidf_matrix_char_train = vectorizer_char.transform(train_x)\n",
    "tfidf_matrix_char_test = vectorizer_char.transform(ver_x)\n",
    "\n",
    "lr_char = LogisticRegression(solver='sag', verbose=2)\n",
    "lr_char.fit(tfidf_matrix_char_train, train_y)\n",
    "\n",
    "y_pred_char = lr_char.predict(tfidf_matrix_char_test)\n",
    "joblib.dump(lr_char, 'lr_char_ngram.pkl')\n",
    "\n",
    "pd.DataFrame(y_pred_char, columns=['y_pred']).to_csv('lr_char_ngram.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.663982102908\n"
     ]
    }
   ],
   "source": [
    "y_pred_char = pd.read_csv('lr_char_ngram.csv')\n",
    "print(accuracy_score(ver_y, y_pred_char))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'morning': True, 'hello': True, 'good': True, 'day': True, 'great': True}\n",
      "2298 5793\n"
     ]
    }
   ],
   "source": [
    "# feature extractor function\n",
    "def bag_of_words(tweet):\n",
    "    words = clean_tweets(tweet)\n",
    "    words_dictionary = dict([word, True] for word in words)    \n",
    "    return words_dictionary\n",
    " \n",
    "custom_tweet = \"RT @Twitter @chapagain Hello There! Have a great day. :) #good #morning http://chapagain.com.np\"\n",
    "print (bag_of_words(custom_tweet))\n",
    "'''\n",
    "Output:\n",
    " \n",
    "{'great': True, 'good': True, 'morning': True, 'hello': True, 'day': True}\n",
    "'''\n",
    "# insult tweets feature set\n",
    "insult_tweets_set = []\n",
    "for tweet in insult['Comment']:\n",
    "    insult_tweets_set.append((bag_of_words(tweet), 'ins'))    \n",
    "for words in banwords['banword']:\n",
    "    insult_tweets_set.append((bag_of_words(words),'ins'))\n",
    "# normal tweets feature set\n",
    "normal_tweets_set = []\n",
    "for tweet in normal['Comment']:\n",
    "    normal_tweets_set.append((bag_of_words(tweet), 'norm'))\n",
    "for tweet in norm_tweets_text:\n",
    "    normal_tweets_set.append((bag_of_words(tweet), 'norm'))\n",
    "print(len(insult_tweets_set), len(normal_tweets_set))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(clean_tweets(insult['Comment'][200]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "({'tell': True, 'emabiggestfans': True, 'mbf': True, 'u': True, 'notifs': True, 'vamp': True, 'dm': True, 'rt': True, '1d': True, 'turn': True, 'want': True, 'solo': True}, 'norm')\n"
     ]
    }
   ],
   "source": [
    "print(normal_tweets_set[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2235 8091\n"
     ]
    }
   ],
   "source": [
    "# try n-fold (minibatch) later\n",
    "from random import shuffle, seed\n",
    "seed(10)\n",
    "shuffle(insult_tweets_set)\n",
    "shuffle(normal_tweets_set)\n",
    " \n",
    "#test_set = insult_tweets_set[:1000] + normal_tweets_set[:1000]\n",
    "#train_set = insult_tweets_set[1000:] + normal_tweets_set[1000:]\n",
    "train_set = insult_tweets_set + normal_tweets_set\n",
    "test_set = insult_ver_set + norm_ver_set\n",
    "print(len(test_set),  len(train_set)) # Output: (2000, 8000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word embedding: efficient vectorization of words\n",
    "By vectorizing words, we can reduce the sparcity and dimensionality of the feature space. \n",
    "The representation is normally ~ 100-1000 dimension. The basic concept behind it is that words used in\n",
    "similar context have similar meanings.\n",
    "## Embedding layer\n",
    "Using NN for language modeling or document classification. \n",
    "## Word2vec\n",
    "probably not easy with twitter texts as individual tweets are very short and thus predicting word from the surrounding (both for CBoW and Cont. skip-gram) would be hard? : not much freedom in choosing the sliding window size...\n",
    "## GloVe\n",
    "Word2Vec with additional information about the context from the whole corpus. Provides \"Unsupervised\" clustering of vocab, purely based on a large corpus of data. We use pre-trained GloVe vocabulary vector library collected from Twitter data from https://github.com/stanfordnlp/GloVe (2B tweets, 27B tokens, 1.2M vocab, uncased, 25d, 50d, 100d, & 200d vectors, 1.42 GB).\n",
    "\n",
    "Reference : https://machinelearningmastery.com/what-are-word-embeddings/\n",
    "\n",
    "# Vader sentiment analysis (pos,neg)- how useful is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'neu': 0.462, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.538}\n"
     ]
    }
   ],
   "source": [
    "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
    "analyser = SentimentIntensityAnalyzer()\n",
    "snt = analyser.polarity_scores(insult['Comment'][0])\n",
    "print(snt)\n",
    "#AttributeError: 'SentimentIntensityAnalyzer' object has no attribute 'polarity_score'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 {'neu': 0.462, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.538}\n",
      "1 {'neu': 0.577, 'compound': -0.7003, 'pos': 0.119, 'neg': 0.304}\n",
      "2 {'neu': 0.693, 'compound': -0.4767, 'pos': 0.0, 'neg': 0.307}\n",
      "3 {'neu': 0.769, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.231}\n",
      "4 {'neu': 0.432, 'compound': -0.5574, 'pos': 0.173, 'neg': 0.395}\n",
      "7 {'neu': 0.702, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.298}\n",
      "8 {'neu': 0.421, 'compound': -0.6705, 'pos': 0.0, 'neg': 0.579}\n",
      "11 {'neu': 0.761, 'compound': -0.8767, 'pos': 0.0, 'neg': 0.239}\n",
      "13 {'neu': 0.81, 'compound': -0.7506, 'pos': 0.04, 'neg': 0.15}\n",
      "14 {'neu': 0.619, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.381}\n",
      "16 {'neu': 0.481, 'compound': -0.8238, 'pos': 0.0, 'neg': 0.519}\n",
      "18 {'neu': 0.948, 'compound': -0.1027, 'pos': 0.024, 'neg': 0.028}\n",
      "20 {'neu': 0.701, 'compound': -0.7906, 'pos': 0.068, 'neg': 0.231}\n",
      "21 {'neu': 0.637, 'compound': -0.8395, 'pos': 0.0, 'neg': 0.363}\n",
      "23 {'neu': 0.645, 'compound': -0.8775, 'pos': 0.094, 'neg': 0.261}\n",
      "26 {'neu': 0.364, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.636}\n",
      "28 {'neu': 0.713, 'compound': -0.7213, 'pos': 0.0, 'neg': 0.287}\n",
      "30 {'neu': 0.844, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.156}\n",
      "31 {'neu': 0.698, 'compound': -0.8271, 'pos': 0.072, 'neg': 0.23}\n",
      "32 {'neu': 0.448, 'compound': -0.9568, 'pos': 0.055, 'neg': 0.497}\n",
      "34 {'neu': 0.64, 'compound': -0.1531, 'pos': 0.163, 'neg': 0.198}\n",
      "35 {'neu': 0.741, 'compound': -0.891, 'pos': 0.0, 'neg': 0.259}\n",
      "36 {'neu': 0.701, 'compound': -0.954, 'pos': 0.084, 'neg': 0.214}\n",
      "37 {'neu': 0.592, 'compound': -0.1027, 'pos': 0.191, 'neg': 0.217}\n",
      "38 {'neu': 0.548, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.452}\n",
      "39 {'neu': 0.602, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.398}\n",
      "40 {'neu': 0.736, 'compound': -0.6542, 'pos': 0.071, 'neg': 0.194}\n",
      "42 {'neu': 0.795, 'compound': -0.658, 'pos': 0.0, 'neg': 0.205}\n",
      "44 {'neu': 0.49, 'compound': -0.9299, 'pos': 0.046, 'neg': 0.464}\n",
      "45 {'neu': 0.685, 'compound': -0.3182, 'pos': 0.0, 'neg': 0.315}\n",
      "49 {'neu': 0.625, 'compound': -0.6597, 'pos': 0.0, 'neg': 0.375}\n",
      "51 {'neu': 0.728, 'compound': -0.9732, 'pos': 0.024, 'neg': 0.248}\n",
      "52 {'neu': 0.71, 'compound': -0.7482, 'pos': 0.104, 'neg': 0.186}\n",
      "53 {'neu': 0.69, 'compound': -0.8929, 'pos': 0.0, 'neg': 0.31}\n",
      "54 {'neu': 0.638, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.362}\n",
      "55 {'neu': 0.907, 'compound': -0.1655, 'pos': 0.0, 'neg': 0.093}\n",
      "57 {'neu': 0.392, 'compound': -0.7351, 'pos': 0.0, 'neg': 0.608}\n",
      "58 {'neu': 0.686, 'compound': -0.6705, 'pos': 0.0, 'neg': 0.314}\n",
      "60 {'neu': 0.58, 'compound': -0.9709, 'pos': 0.0, 'neg': 0.42}\n",
      "63 {'neu': 0.693, 'compound': -0.7351, 'pos': 0.058, 'neg': 0.248}\n",
      "64 {'neu': 0.602, 'compound': -0.891, 'pos': 0.0, 'neg': 0.398}\n",
      "65 {'neu': 0.462, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.538}\n",
      "66 {'neu': 0.486, 'compound': -0.3578, 'pos': 0.185, 'neg': 0.329}\n",
      "67 {'neu': 0.764, 'compound': -0.8625, 'pos': 0.0, 'neg': 0.236}\n",
      "68 {'neu': 0.614, 'compound': -0.6597, 'pos': 0.0, 'neg': 0.386}\n",
      "70 {'neu': 0.5, 'compound': 0.0, 'pos': 0.208, 'neg': 0.292}\n",
      "71 {'neu': 0.767, 'compound': -0.4588, 'pos': 0.071, 'neg': 0.163}\n",
      "74 {'neu': 0.484, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.516}\n",
      "75 {'neu': 0.904, 'compound': -0.7783, 'pos': 0.0, 'neg': 0.096}\n",
      "76 {'neu': 0.552, 'compound': -0.8442, 'pos': 0.0, 'neg': 0.448}\n",
      "77 {'neu': 0.407, 'compound': -0.4515, 'pos': 0.216, 'neg': 0.377}\n",
      "79 {'neu': 0.743, 'compound': -0.8061, 'pos': 0.074, 'neg': 0.183}\n",
      "81 {'neu': 0.845, 'compound': -0.7241, 'pos': 0.039, 'neg': 0.116}\n",
      "82 {'neu': 0.793, 'compound': -0.7184, 'pos': 0.063, 'neg': 0.144}\n",
      "83 {'neu': 0.625, 'compound': -0.8375, 'pos': 0.107, 'neg': 0.268}\n",
      "87 {'neu': 0.808, 'compound': -0.8105, 'pos': 0.0, 'neg': 0.192}\n",
      "88 {'neu': 0.221, 'compound': -0.8904, 'pos': 0.0, 'neg': 0.779}\n",
      "89 {'neu': 0.588, 'compound': -0.8553, 'pos': 0.0, 'neg': 0.412}\n",
      "90 {'neu': 0.661, 'compound': -0.9337, 'pos': 0.073, 'neg': 0.266}\n",
      "93 {'neu': 0.68, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.32}\n",
      "94 {'neu': 0.462, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.538}\n",
      "98 {'neu': 0.548, 'compound': -0.765, 'pos': 0.0, 'neg': 0.452}\n",
      "99 {'neu': 0.533, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.467}\n",
      "101 {'neu': 0.571, 'compound': -0.4588, 'pos': 0.0, 'neg': 0.429}\n",
      "102 {'neu': 0.608, 'compound': -0.7214, 'pos': 0.078, 'neg': 0.314}\n",
      "104 {'neu': 0.671, 'compound': -0.8834, 'pos': 0.0, 'neg': 0.329}\n",
      "107 {'neu': 0.222, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.778}\n",
      "108 {'neu': 0.657, 'compound': -0.6597, 'pos': 0.091, 'neg': 0.252}\n",
      "109 {'neu': 0.776, 'compound': -0.9062, 'pos': 0.0, 'neg': 0.224}\n",
      "110 {'neu': 0.37, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.63}\n",
      "111 {'neu': 0.6, 'compound': -0.8923, 'pos': 0.068, 'neg': 0.332}\n",
      "113 {'neu': 0.816, 'compound': -0.3089, 'pos': 0.0, 'neg': 0.184}\n",
      "114 {'neu': 0.575, 'compound': -0.34, 'pos': 0.144, 'neg': 0.282}\n",
      "115 {'neu': 0.523, 'compound': -0.807, 'pos': 0.0, 'neg': 0.477}\n",
      "118 {'neu': 0.805, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.195}\n",
      "119 {'neu': 0.837, 'compound': -0.5882, 'pos': 0.058, 'neg': 0.105}\n",
      "120 {'neu': 0.644, 'compound': -0.8555, 'pos': 0.0, 'neg': 0.356}\n",
      "121 {'neu': 0.747, 'compound': -0.5279, 'pos': 0.0, 'neg': 0.253}\n",
      "124 {'neu': 0.759, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.241}\n",
      "125 {'neu': 0.563, 'compound': -0.935, 'pos': 0.0, 'neg': 0.437}\n",
      "127 {'neu': 0.677, 'compound': -0.9348, 'pos': 0.0, 'neg': 0.323}\n",
      "129 {'neu': 0.515, 'compound': -0.765, 'pos': 0.0, 'neg': 0.485}\n",
      "130 {'neu': 0.675, 'compound': 0.0875, 'pos': 0.146, 'neg': 0.178}\n",
      "131 {'neu': 0.703, 'compound': -0.5859, 'pos': 0.0, 'neg': 0.297}\n",
      "132 {'neu': 0.784, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.216}\n",
      "135 {'neu': 0.523, 'compound': -0.857, 'pos': 0.102, 'neg': 0.374}\n",
      "136 {'neu': 0.364, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.636}\n",
      "137 {'neu': 0.658, 'compound': -0.9005, 'pos': 0.0, 'neg': 0.342}\n",
      "139 {'neu': 0.462, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.538}\n",
      "140 {'neu': 0.84, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.16}\n",
      "141 {'neu': 0.502, 'compound': -0.8523, 'pos': 0.094, 'neg': 0.405}\n",
      "142 {'neu': 0.651, 'compound': -0.8442, 'pos': 0.0, 'neg': 0.349}\n",
      "146 {'neu': 0.88, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.12}\n",
      "148 {'neu': 0.684, 'compound': -0.8599, 'pos': 0.133, 'neg': 0.183}\n",
      "149 {'neu': 0.677, 'compound': -0.7667, 'pos': 0.0, 'neg': 0.323}\n",
      "151 {'neu': 0.453, 'compound': -0.5808, 'pos': 0.233, 'neg': 0.314}\n",
      "153 {'neu': 0.311, 'compound': -0.9565, 'pos': 0.0, 'neg': 0.689}\n",
      "154 {'neu': 0.821, 'compound': -0.6404, 'pos': 0.052, 'neg': 0.127}\n",
      "156 {'neu': 0.64, 'compound': -0.8345, 'pos': 0.0, 'neg': 0.36}\n",
      "157 {'neu': 0.519, 'compound': -0.873, 'pos': 0.138, 'neg': 0.343}\n",
      "161 {'neu': 0.541, 'compound': -0.6249, 'pos': 0.0, 'neg': 0.459}\n",
      "162 {'neu': 0.732, 'compound': -0.4449, 'pos': 0.0, 'neg': 0.268}\n",
      "163 {'neu': 0.716, 'compound': -0.7542, 'pos': 0.12, 'neg': 0.164}\n",
      "164 {'neu': 0.691, 'compound': -0.9792, 'pos': 0.084, 'neg': 0.225}\n",
      "166 {'neu': 0.665, 'compound': -0.8607, 'pos': 0.0, 'neg': 0.335}\n",
      "167 {'neu': 0.768, 'compound': -0.9197, 'pos': 0.083, 'neg': 0.149}\n",
      "168 {'neu': 0.744, 'compound': -0.6705, 'pos': 0.0, 'neg': 0.256}\n",
      "169 {'neu': 0.577, 'compound': -0.8829, 'pos': 0.0, 'neg': 0.423}\n",
      "172 {'neu': 0.53, 'compound': -0.3818, 'pos': 0.174, 'neg': 0.295}\n",
      "173 {'neu': 0.404, 'compound': -0.0258, 'pos': 0.293, 'neg': 0.303}\n",
      "174 {'neu': 0.556, 'compound': -0.8225, 'pos': 0.082, 'neg': 0.363}\n",
      "175 {'neu': 0.744, 'compound': -0.6271, 'pos': 0.055, 'neg': 0.201}\n",
      "176 {'neu': 0.213, 'compound': -0.5709, 'pos': 0.0, 'neg': 0.787}\n",
      "177 {'neu': 0.675, 'compound': -0.7442, 'pos': 0.063, 'neg': 0.262}\n",
      "178 {'neu': 0.629, 'compound': -0.0772, 'pos': 0.175, 'neg': 0.196}\n",
      "179 {'neu': 0.89, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.11}\n",
      "181 {'neu': 0.702, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.298}\n",
      "183 {'neu': 0.647, 'compound': -0.25, 'pos': 0.147, 'neg': 0.206}\n",
      "188 {'neu': 0.907, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.093}\n",
      "193 {'neu': 0.602, 'compound': -0.765, 'pos': 0.0, 'neg': 0.398}\n",
      "194 {'neu': 0.75, 'compound': -0.2732, 'pos': 0.118, 'neg': 0.133}\n",
      "196 {'neu': 0.741, 'compound': -0.0516, 'pos': 0.123, 'neg': 0.136}\n",
      "197 {'neu': 0.548, 'compound': -0.9136, 'pos': 0.073, 'neg': 0.379}\n",
      "199 {'neu': 0.794, 'compound': -0.0772, 'pos': 0.0, 'neg': 0.206}\n",
      "200 {'neu': 0.798, 'compound': -0.4199, 'pos': 0.0, 'neg': 0.202}\n",
      "201 {'neu': 0.607, 'compound': -0.9062, 'pos': 0.0, 'neg': 0.393}\n",
      "202 {'neu': 0.652, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.348}\n",
      "203 {'neu': 0.536, 'compound': -0.0387, 'pos': 0.229, 'neg': 0.235}\n",
      "204 {'neu': 0.698, 'compound': -0.7783, 'pos': 0.0, 'neg': 0.302}\n",
      "206 {'neu': 0.733, 'compound': -0.9131, 'pos': 0.027, 'neg': 0.24}\n",
      "208 {'neu': 0.647, 'compound': -0.9227, 'pos': 0.067, 'neg': 0.286}\n",
      "209 {'neu': 0.405, 'compound': -0.6597, 'pos': 0.0, 'neg': 0.595}\n",
      "212 {'neu': 0.417, 'compound': -0.9042, 'pos': 0.042, 'neg': 0.542}\n",
      "214 {'neu': 0.221, 'compound': -0.9726, 'pos': 0.069, 'neg': 0.71}\n",
      "215 {'neu': 0.884, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.116}\n",
      "216 {'neu': 0.711, 'compound': -0.7516, 'pos': 0.043, 'neg': 0.246}\n",
      "218 {'neu': 0.752, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.248}\n",
      "219 {'neu': 0.114, 'compound': -0.8316, 'pos': 0.0, 'neg': 0.886}\n",
      "220 {'neu': 0.323, 'compound': -0.128, 'pos': 0.312, 'neg': 0.366}\n",
      "222 {'neu': 0.663, 'compound': -0.765, 'pos': 0.0, 'neg': 0.337}\n",
      "223 {'neu': 0.503, 'compound': -0.81, 'pos': 0.057, 'neg': 0.439}\n",
      "227 {'neu': 0.829, 'compound': -0.2023, 'pos': 0.075, 'neg': 0.097}\n",
      "228 {'neu': 0.783, 'compound': -0.802, 'pos': 0.0, 'neg': 0.217}\n",
      "229 {'neu': 0.707, 'compound': -0.9385, 'pos': 0.033, 'neg': 0.26}\n",
      "230 {'neu': 0.43, 'compound': -0.9812, 'pos': 0.136, 'neg': 0.434}\n",
      "231 {'neu': 0.621, 'compound': -0.7269, 'pos': 0.0, 'neg': 0.379}\n",
      "232 {'neu': 0.763, 'compound': -0.1531, 'pos': 0.106, 'neg': 0.131}\n",
      "233 {'neu': 0.793, 'compound': -0.6597, 'pos': 0.05, 'neg': 0.156}\n",
      "237 {'neu': 0.595, 'compound': -0.8468, 'pos': 0.0, 'neg': 0.405}\n",
      "238 {'neu': 0.671, 'compound': -0.3257, 'pos': 0.136, 'neg': 0.192}\n",
      "241 {'neu': 0.757, 'compound': -0.9689, 'pos': 0.053, 'neg': 0.19}\n",
      "242 {'neu': 0.664, 'compound': -0.9657, 'pos': 0.077, 'neg': 0.26}\n",
      "244 {'neu': 0.59, 'compound': -0.4402, 'pos': 0.162, 'neg': 0.248}\n",
      "245 {'neu': 0.896, 'compound': -0.1027, 'pos': 0.0, 'neg': 0.104}\n",
      "246 {'neu': 0.364, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.636}\n",
      "249 {'neu': 0.827, 'compound': -0.3182, 'pos': 0.0, 'neg': 0.173}\n",
      "250 {'neu': 0.596, 'compound': -0.9929, 'pos': 0.102, 'neg': 0.302}\n",
      "251 {'neu': 0.559, 'compound': -0.9398, 'pos': 0.093, 'neg': 0.348}\n",
      "252 {'neu': 0.696, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.304}\n",
      "254 {'neu': 0.738, 'compound': -0.8631, 'pos': 0.042, 'neg': 0.22}\n",
      "255 {'neu': 0.814, 'compound': -0.7906, 'pos': 0.0, 'neg': 0.186}\n",
      "256 {'neu': 0.786, 'compound': -0.5473, 'pos': 0.0, 'neg': 0.214}\n",
      "257 {'neu': 0.792, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.208}\n",
      "258 {'neu': 0.581, 'compound': -0.5574, 'pos': 0.0, 'neg': 0.419}\n",
      "259 {'neu': 0.612, 'compound': -0.5859, 'pos': 0.0, 'neg': 0.388}\n",
      "260 {'neu': 0.714, 'compound': -0.25, 'pos': 0.117, 'neg': 0.168}\n",
      "262 {'neu': 0.658, 'compound': -0.8611, 'pos': 0.07, 'neg': 0.272}\n",
      "264 {'neu': 0.421, 'compound': -0.6705, 'pos': 0.0, 'neg': 0.579}\n",
      "265 {'neu': 0.786, 'compound': -0.504, 'pos': 0.0, 'neg': 0.214}\n",
      "268 {'neu': 0.677, 'compound': -0.9231, 'pos': 0.0, 'neg': 0.323}\n",
      "269 {'neu': 0.343, 'compound': -0.128, 'pos': 0.314, 'neg': 0.343}\n",
      "271 {'neu': 0.581, 'compound': -0.5574, 'pos': 0.0, 'neg': 0.419}\n",
      "272 {'neu': 0.697, 'compound': -0.9702, 'pos': 0.061, 'neg': 0.242}\n",
      "273 {'neu': 0.401, 'compound': -0.9917, 'pos': 0.033, 'neg': 0.566}\n",
      "274 {'neu': 0.634, 'compound': -0.8558, 'pos': 0.063, 'neg': 0.302}\n",
      "282 {'neu': 0.836, 'compound': -0.2263, 'pos': 0.07, 'neg': 0.095}\n",
      "286 {'neu': 0.455, 'compound': -0.9726, 'pos': 0.043, 'neg': 0.502}\n",
      "287 {'neu': 0.552, 'compound': -0.9188, 'pos': 0.066, 'neg': 0.382}\n",
      "290 {'neu': 0.805, 'compound': -0.594, 'pos': 0.0, 'neg': 0.195}\n",
      "292 {'neu': 0.593, 'compound': -0.5106, 'pos': 0.119, 'neg': 0.289}\n",
      "293 {'neu': 0.813, 'compound': -0.5707, 'pos': 0.0, 'neg': 0.187}\n",
      "294 {'neu': 0.582, 'compound': -0.9089, 'pos': 0.128, 'neg': 0.29}\n",
      "295 {'neu': 0.727, 'compound': -0.34, 'pos': 0.105, 'neg': 0.168}\n",
      "296 {'neu': 0.776, 'compound': -0.8834, 'pos': 0.06, 'neg': 0.164}\n",
      "298 {'neu': 0.655, 'compound': -0.685, 'pos': 0.107, 'neg': 0.239}\n",
      "303 {'neu': 0.495, 'compound': -0.6597, 'pos': 0.144, 'neg': 0.361}\n",
      "304 {'neu': 0.888, 'compound': -0.796, 'pos': 0.0, 'neg': 0.112}\n",
      "305 {'neu': 0.664, 'compound': -0.8625, 'pos': 0.0, 'neg': 0.336}\n",
      "308 {'neu': 0.695, 'compound': -0.7616, 'pos': 0.074, 'neg': 0.231}\n",
      "311 {'neu': 0.527, 'compound': -0.8934, 'pos': 0.095, 'neg': 0.379}\n",
      "312 {'neu': 0.241, 'compound': -0.2023, 'pos': 0.325, 'neg': 0.434}\n",
      "313 {'neu': 0.687, 'compound': -0.6705, 'pos': 0.079, 'neg': 0.234}\n",
      "314 {'neu': 0.613, 'compound': -0.3612, 'pos': 0.177, 'neg': 0.21}\n",
      "316 {'neu': 0.734, 'compound': -0.5622, 'pos': 0.0, 'neg': 0.266}\n",
      "317 {'neu': 0.599, 'compound': -0.338, 'pos': 0.176, 'neg': 0.225}\n",
      "318 {'neu': 0.672, 'compound': -0.9349, 'pos': 0.045, 'neg': 0.283}\n",
      "319 {'neu': 0.842, 'compound': -0.296, 'pos': 0.064, 'neg': 0.094}\n",
      "320 {'neu': 0.817, 'compound': -0.7096, 'pos': 0.048, 'neg': 0.135}\n",
      "324 {'neu': 0.374, 'compound': -0.7717, 'pos': 0.0, 'neg': 0.626}\n",
      "325 {'neu': 0.652, 'compound': -0.7059, 'pos': 0.0, 'neg': 0.348}\n",
      "326 {'neu': 0.809, 'compound': -0.838, 'pos': 0.063, 'neg': 0.128}\n",
      "329 {'neu': 0.52, 'compound': -0.6956, 'pos': 0.19, 'neg': 0.29}\n",
      "330 {'neu': 0.667, 'compound': -0.7906, 'pos': 0.0, 'neg': 0.333}\n",
      "331 {'neu': 0.838, 'compound': -0.7352, 'pos': 0.0, 'neg': 0.162}\n",
      "332 {'neu': 0.839, 'compound': -0.2869, 'pos': 0.063, 'neg': 0.098}\n",
      "333 {'neu': 0.804, 'compound': -0.3976, 'pos': 0.0, 'neg': 0.196}\n",
      "335 {'neu': 0.634, 'compound': -0.9136, 'pos': 0.0, 'neg': 0.366}\n",
      "336 {'neu': 0.458, 'compound': -0.2023, 'pos': 0.258, 'neg': 0.284}\n",
      "337 {'neu': 0.763, 'compound': -0.4215, 'pos': 0.0, 'neg': 0.237}\n",
      "338 {'neu': 0.676, 'compound': -0.34, 'pos': 0.0, 'neg': 0.324}\n",
      "339 {'neu': 0.438, 'compound': -0.9186, 'pos': 0.24, 'neg': 0.322}\n",
      "340 {'neu': 0.523, 'compound': -0.9081, 'pos': 0.073, 'neg': 0.404}\n",
      "341 {'neu': 0.723, 'compound': -0.3976, 'pos': 0.0, 'neg': 0.277}\n",
      "343 {'neu': 0.494, 'compound': -0.6249, 'pos': 0.0, 'neg': 0.506}\n",
      "344 {'neu': 0.754, 'compound': -0.9403, 'pos': 0.0, 'neg': 0.246}\n",
      "345 {'neu': 0.771, 'compound': -0.8316, 'pos': 0.0, 'neg': 0.229}\n",
      "347 {'neu': 0.707, 'compound': -0.6597, 'pos': 0.0, 'neg': 0.293}\n",
      "348 {'neu': 0.75, 'compound': -0.9739, 'pos': 0.08, 'neg': 0.17}\n",
      "349 {'neu': 0.853, 'compound': -0.3412, 'pos': 0.0, 'neg': 0.147}\n",
      "350 {'neu': 0.761, 'compound': -0.6486, 'pos': 0.06, 'neg': 0.179}\n",
      "351 {'neu': 0.643, 'compound': -0.6444, 'pos': 0.087, 'neg': 0.27}\n",
      "352 {'neu': 0.848, 'compound': -0.4015, 'pos': 0.0, 'neg': 0.152}\n",
      "354 {'neu': 0.739, 'compound': -0.2614, 'pos': 0.118, 'neg': 0.143}\n",
      "355 {'neu': 0.893, 'compound': -0.6072, 'pos': 0.0, 'neg': 0.107}\n",
      "356 {'neu': 0.367, 'compound': -0.7841, 'pos': 0.0, 'neg': 0.633}\n",
      "357 {'neu': 0.873, 'compound': -0.7599, 'pos': 0.0, 'neg': 0.127}\n",
      "361 {'neu': 0.518, 'compound': -0.9706, 'pos': 0.058, 'neg': 0.424}\n",
      "364 {'neu': 0.377, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.623}\n",
      "366 {'neu': 0.687, 'compound': -0.9742, 'pos': 0.027, 'neg': 0.286}\n",
      "367 {'neu': 0.439, 'compound': -0.7688, 'pos': 0.0, 'neg': 0.561}\n",
      "368 {'neu': 0.73, 'compound': -0.5984, 'pos': 0.073, 'neg': 0.197}\n",
      "369 {'neu': 0.752, 'compound': -0.644, 'pos': 0.089, 'neg': 0.16}\n",
      "370 {'neu': 0.637, 'compound': -0.9651, 'pos': 0.0, 'neg': 0.363}\n",
      "371 {'neu': 0.5, 'compound': -0.1531, 'pos': 0.235, 'neg': 0.265}\n",
      "373 {'neu': 0.816, 'compound': -0.8506, 'pos': 0.053, 'neg': 0.132}\n",
      "374 {'neu': 0.507, 'compound': -0.7783, 'pos': 0.0, 'neg': 0.493}\n",
      "375 {'neu': 0.726, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.274}\n",
      "377 {'neu': 0.769, 'compound': -0.4019, 'pos': 0.0, 'neg': 0.231}\n",
      "379 {'neu': 0.443, 'compound': -0.4574, 'pos': 0.184, 'neg': 0.373}\n",
      "380 {'neu': 0.334, 'compound': -0.4912, 'pos': 0.211, 'neg': 0.455}\n",
      "381 {'neu': 0.814, 'compound': -0.7599, 'pos': 0.044, 'neg': 0.142}\n",
      "382 {'neu': 0.811, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.189}\n",
      "385 {'neu': 0.533, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.467}\n",
      "388 {'neu': 0.5, 'compound': -0.875, 'pos': 0.0, 'neg': 0.5}\n",
      "389 {'neu': 0.704, 'compound': -0.9739, 'pos': 0.025, 'neg': 0.27}\n",
      "390 {'neu': 0.718, 'compound': -0.6249, 'pos': 0.0, 'neg': 0.282}\n",
      "391 {'neu': 0.655, 'compound': -0.836, 'pos': 0.0, 'neg': 0.345}\n",
      "393 {'neu': 0.857, 'compound': -0.5267, 'pos': 0.038, 'neg': 0.106}\n",
      "394 {'neu': 0.498, 'compound': -0.6166, 'pos': 0.0, 'neg': 0.502}\n",
      "395 {'neu': 0.661, 'compound': -0.6124, 'pos': 0.137, 'neg': 0.203}\n",
      "397 {'neu': 0.556, 'compound': -0.4926, 'pos': 0.0, 'neg': 0.444}\n",
      "399 {'neu': 0.518, 'compound': -0.9735, 'pos': 0.0, 'neg': 0.482}\n",
      "400 {'neu': 0.838, 'compound': -0.4404, 'pos': 0.0, 'neg': 0.162}\n",
      "401 {'neu': 0.667, 'compound': -0.7579, 'pos': 0.0, 'neg': 0.333}\n",
      "402 {'neu': 0.424, 'compound': -0.7783, 'pos': 0.0, 'neg': 0.576}\n",
      "403 {'neu': 0.362, 'compound': -0.5859, 'pos': 0.181, 'neg': 0.457}\n",
      "404 {'neu': 0.876, 'compound': -0.34, 'pos': 0.0, 'neg': 0.124}\n",
      "405 {'neu': 0.574, 'compound': -0.5994, 'pos': 0.176, 'neg': 0.25}\n",
      "406 {'neu': 0.635, 'compound': -0.765, 'pos': 0.062, 'neg': 0.303}\n",
      "408 {'neu': 0.545, 'compound': -0.962, 'pos': 0.113, 'neg': 0.342}\n",
      "409 {'neu': 0.874, 'compound': -0.8555, 'pos': 0.0, 'neg': 0.126}\n",
      "411 {'neu': 0.759, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.241}\n",
      "412 {'neu': 0.543, 'compound': -0.6486, 'pos': 0.131, 'neg': 0.326}\n",
      "413 {'neu': 0.687, 'compound': -0.4926, 'pos': 0.095, 'neg': 0.217}\n",
      "414 {'neu': 0.55, 'compound': -0.0571, 'pos': 0.22, 'neg': 0.23}\n",
      "416 {'neu': 0.455, 'compound': -0.5562, 'pos': 0.0, 'neg': 0.545}\n",
      "422 {'neu': 0.295, 'compound': -0.8008, 'pos': 0.0, 'neg': 0.705}\n",
      "423 {'neu': 0.748, 'compound': -0.658, 'pos': 0.0, 'neg': 0.252}\n",
      "424 {'neu': 0.643, 'compound': -0.8139, 'pos': 0.149, 'neg': 0.208}\n",
      "425 {'neu': 0.705, 'compound': -0.6808, 'pos': 0.0, 'neg': 0.295}\n",
      "427 {'neu': 0.602, 'compound': -0.969, 'pos': 0.0, 'neg': 0.398}\n",
      "432 {'neu': 0.494, 'compound': -0.9059, 'pos': 0.0, 'neg': 0.506}\n",
      "434 {'neu': 0.657, 'compound': -0.3832, 'pos': 0.0, 'neg': 0.343}\n",
      "436 {'neu': 0.725, 'compound': -0.7506, 'pos': 0.06, 'neg': 0.215}\n",
      "438 {'neu': 0.665, 'compound': -0.5892, 'pos': 0.134, 'neg': 0.202}\n",
      "439 {'neu': 0.571, 'compound': -0.9331, 'pos': 0.0, 'neg': 0.429}\n",
      "440 {'neu': 0.706, 'compound': -0.7691, 'pos': 0.0, 'neg': 0.294}\n",
      "443 {'neu': 0.354, 'compound': -0.8074, 'pos': 0.0, 'neg': 0.646}\n",
      "444 {'neu': 0.856, 'compound': -0.1027, 'pos': 0.067, 'neg': 0.078}\n",
      "446 {'neu': 0.658, 'compound': -0.9273, 'pos': 0.05, 'neg': 0.292}\n",
      "447 {'neu': 0.556, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.444}\n",
      "450 {'neu': 0.465, 'compound': 0.0516, 'pos': 0.244, 'neg': 0.291}\n",
      "451 {'neu': 0.781, 'compound': -0.1531, 'pos': 0.098, 'neg': 0.121}\n",
      "452 {'neu': 0.671, 'compound': -0.8928, 'pos': 0.035, 'neg': 0.294}\n",
      "453 {'neu': 0.779, 'compound': -0.7783, 'pos': 0.0, 'neg': 0.221}\n",
      "454 {'neu': 0.775, 'compound': -0.7278, 'pos': 0.0, 'neg': 0.225}\n",
      "455 {'neu': 0.736, 'compound': -0.6869, 'pos': 0.0, 'neg': 0.264}\n",
      "456 {'neu': 0.811, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.189}\n",
      "457 {'neu': 0.703, 'compound': -0.6062, 'pos': 0.084, 'neg': 0.213}\n",
      "458 {'neu': 0.122, 'compound': -0.802, 'pos': 0.0, 'neg': 0.878}\n",
      "459 {'neu': 0.767, 'compound': -0.6068, 'pos': 0.0, 'neg': 0.233}\n",
      "460 {'neu': 0.09, 'compound': -0.8779, 'pos': 0.0, 'neg': 0.91}\n",
      "461 {'neu': 0.727, 'compound': -0.8555, 'pos': 0.0, 'neg': 0.273}\n",
      "462 {'neu': 0.788, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.212}\n",
      "463 {'neu': 0.626, 'compound': -0.4562, 'pos': 0.0, 'neg': 0.374}\n",
      "464 {'neu': 0.721, 'compound': -0.7351, 'pos': 0.0, 'neg': 0.279}\n",
      "467 {'neu': 0.377, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.623}\n",
      "468 {'neu': 0.327, 'compound': -0.9612, 'pos': 0.0, 'neg': 0.673}\n",
      "469 {'neu': 0.665, 'compound': -0.5892, 'pos': 0.134, 'neg': 0.202}\n",
      "472 {'neu': 0.469, 'compound': -0.8265, 'pos': 0.085, 'neg': 0.446}\n",
      "474 {'neu': 0.796, 'compound': -0.8922, 'pos': 0.068, 'neg': 0.136}\n",
      "476 {'neu': 0.365, 'compound': -0.8658, 'pos': 0.0, 'neg': 0.635}\n",
      "477 {'neu': 0.345, 'compound': -0.5859, 'pos': 0.0, 'neg': 0.655}\n",
      "478 {'neu': 0.589, 'compound': -0.8011, 'pos': 0.092, 'neg': 0.319}\n",
      "483 {'neu': 0.526, 'compound': -0.8402, 'pos': 0.0, 'neg': 0.474}\n",
      "486 {'neu': 0.645, 'compound': -0.6597, 'pos': 0.0, 'neg': 0.355}\n",
      "489 {'neu': 0.694, 'compound': -0.6597, 'pos': 0.0, 'neg': 0.306}\n",
      "490 {'neu': 0.789, 'compound': -0.5859, 'pos': 0.0, 'neg': 0.211}\n",
      "491 {'neu': 0.83, 'compound': -0.3818, 'pos': 0.059, 'neg': 0.111}\n",
      "493 {'neu': 0.641, 'compound': -0.4588, 'pos': 0.115, 'neg': 0.244}\n",
      "494 {'neu': 0.553, 'compound': -0.9609, 'pos': 0.124, 'neg': 0.322}\n",
      "495 {'neu': 0.377, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.623}\n",
      "496 {'neu': 0.712, 'compound': -0.0772, 'pos': 0.139, 'neg': 0.149}\n",
      "498 {'neu': 0.527, 'compound': -0.9567, 'pos': 0.06, 'neg': 0.413}\n",
      "499 {'neu': 0.78, 'compound': -0.4767, 'pos': 0.0, 'neg': 0.22}\n",
      "500 {'neu': 0.581, 'compound': -0.5574, 'pos': 0.0, 'neg': 0.419}\n",
      "501 {'neu': 0.741, 'compound': -0.2883, 'pos': 0.118, 'neg': 0.141}\n",
      "503 {'neu': 0.705, 'compound': -0.9094, 'pos': 0.051, 'neg': 0.244}\n",
      "505 {'neu': 0.667, 'compound': -0.6124, 'pos': 0.0, 'neg': 0.333}\n",
      "507 {'neu': 0.415, 'compound': -0.6249, 'pos': 0.207, 'neg': 0.378}\n",
      "509 {'neu': 0.742, 'compound': -0.7906, 'pos': 0.0, 'neg': 0.258}\n",
      "510 {'neu': 0.852, 'compound': -0.7825, 'pos': 0.03, 'neg': 0.118}\n",
      "512 {'neu': 0.785, 'compound': -0.4019, 'pos': 0.085, 'neg': 0.13}\n",
      "514 {'neu': 0.811, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.189}\n",
      "516 {'neu': 0.571, 'compound': -0.4588, 'pos': 0.0, 'neg': 0.429}\n",
      "518 {'neu': 0.577, 'compound': -0.8316, 'pos': 0.0, 'neg': 0.423}\n",
      "519 {'neu': 0.702, 'compound': -0.9115, 'pos': 0.0, 'neg': 0.298}\n",
      "520 {'neu': 0.612, 'compound': -0.1779, 'pos': 0.17, 'neg': 0.218}\n",
      "521 {'neu': 0.754, 'compound': -0.7345, 'pos': 0.0, 'neg': 0.246}\n",
      "522 {'neu': 0.469, 'compound': -0.5255, 'pos': 0.0, 'neg': 0.531}\n",
      "524 {'neu': 0.615, 'compound': -0.7845, 'pos': 0.0, 'neg': 0.385}\n",
      "525 {'neu': 0.484, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.516}\n",
      "527 {'neu': 0.469, 'compound': -0.7783, 'pos': 0.0, 'neg': 0.531}\n",
      "529 {'neu': 0.82, 'compound': 0.402, 'pos': 0.077, 'neg': 0.103}\n",
      "530 {'neu': 0.393, 'compound': -0.8689, 'pos': 0.0, 'neg': 0.607}\n",
      "533 {'neu': 0.617, 'compound': -0.357, 'pos': 0.0, 'neg': 0.383}\n",
      "536 {'neu': 0.906, 'compound': -0.6904, 'pos': 0.0, 'neg': 0.094}\n",
      "537 {'neu': 0.749, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.251}\n",
      "540 {'neu': 0.617, 'compound': -0.9595, 'pos': 0.04, 'neg': 0.343}\n",
      "541 {'neu': 0.229, 'compound': -0.9256, 'pos': 0.0, 'neg': 0.771}\n",
      "543 {'neu': 0.508, 'compound': -0.9385, 'pos': 0.114, 'neg': 0.378}\n",
      "544 {'neu': 0.601, 'compound': -0.9056, 'pos': 0.0, 'neg': 0.399}\n",
      "545 {'neu': 0.633, 'compound': -0.4404, 'pos': 0.0, 'neg': 0.367}\n",
      "546 {'neu': 0.759, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.241}\n",
      "547 {'neu': 0.736, 'compound': -0.9442, 'pos': 0.0, 'neg': 0.264}\n",
      "548 {'neu': 0.767, 'compound': -0.8062, 'pos': 0.029, 'neg': 0.204}\n",
      "552 {'neu': 0.831, 'compound': -0.0572, 'pos': 0.0, 'neg': 0.169}\n",
      "554 {'neu': 0.828, 'compound': -0.3612, 'pos': 0.0, 'neg': 0.172}\n",
      "555 {'neu': 0.604, 'compound': -0.8493, 'pos': 0.094, 'neg': 0.302}\n",
      "556 {'neu': 0.89, 'compound': -0.5526, 'pos': 0.036, 'neg': 0.074}\n",
      "557 {'neu': 0.688, 'compound': -0.4098, 'pos': 0.113, 'neg': 0.199}\n",
      "560 {'neu': 0.632, 'compound': -0.3094, 'pos': 0.14, 'neg': 0.229}\n",
      "561 {'neu': 0.594, 'compound': -0.743, 'pos': 0.109, 'neg': 0.297}\n",
      "563 {'neu': 0.738, 'compound': -0.296, 'pos': 0.082, 'neg': 0.18}\n",
      "564 {'neu': 0.222, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.778}\n",
      "569 {'neu': 0.727, 'compound': -0.707, 'pos': 0.119, 'neg': 0.153}\n",
      "570 {'neu': 0.456, 'compound': -0.5043, 'pos': 0.189, 'neg': 0.355}\n",
      "571 {'neu': 0.774, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.226}\n",
      "573 {'neu': 0.889, 'compound': -0.5972, 'pos': 0.0, 'neg': 0.111}\n",
      "574 {'neu': 0.659, 'compound': -0.3802, 'pos': 0.0, 'neg': 0.341}\n",
      "575 {'neu': 0.331, 'compound': -0.8442, 'pos': 0.0, 'neg': 0.669}\n",
      "576 {'neu': 0.602, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.398}\n",
      "577 {'neu': 0.778, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.222}\n",
      "578 {'neu': 0.581, 'compound': -0.5647, 'pos': 0.19, 'neg': 0.229}\n",
      "579 {'neu': 0.357, 'compound': -0.5574, 'pos': 0.0, 'neg': 0.643}\n",
      "581 {'neu': 0.553, 'compound': -0.687, 'pos': 0.0, 'neg': 0.447}\n",
      "582 {'neu': 0.737, 'compound': -0.7579, 'pos': 0.063, 'neg': 0.2}\n",
      "584 {'neu': 0.507, 'compound': -0.8274, 'pos': 0.0, 'neg': 0.493}\n",
      "587 {'neu': 0.772, 'compound': -0.4344, 'pos': 0.086, 'neg': 0.142}\n",
      "588 {'neu': 0.712, 'compound': -0.9791, 'pos': 0.057, 'neg': 0.231}\n",
      "590 {'neu': 0.425, 'compound': -0.872, 'pos': 0.097, 'neg': 0.479}\n",
      "591 {'neu': 0.712, 'compound': -0.902, 'pos': 0.077, 'neg': 0.211}\n",
      "592 {'neu': 0.472, 'compound': -0.6808, 'pos': 0.0, 'neg': 0.528}\n",
      "593 {'neu': 0.441, 'compound': -0.5859, 'pos': 0.0, 'neg': 0.559}\n",
      "594 {'neu': 0.753, 'compound': -0.6351, 'pos': 0.089, 'neg': 0.159}\n",
      "596 {'neu': 0.581, 'compound': -0.7438, 'pos': 0.079, 'neg': 0.341}\n",
      "597 {'neu': 0.539, 'compound': -0.2484, 'pos': 0.192, 'neg': 0.269}\n",
      "598 {'neu': 0.27, 'compound': -0.8442, 'pos': 0.0, 'neg': 0.73}\n",
      "600 {'neu': 0.64, 'compound': -0.8353, 'pos': 0.075, 'neg': 0.285}\n",
      "601 {'neu': 0.645, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.355}\n",
      "604 {'neu': 0.645, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.355}\n",
      "605 {'neu': 0.508, 'compound': -0.8487, 'pos': 0.142, 'neg': 0.35}\n",
      "606 {'neu': 0.473, 'compound': -0.9377, 'pos': 0.0, 'neg': 0.527}\n",
      "607 {'neu': 0.326, 'compound': -0.8507, 'pos': 0.0, 'neg': 0.674}\n",
      "608 {'neu': 0.761, 'compound': -0.6971, 'pos': 0.079, 'neg': 0.16}\n",
      "609 {'neu': 0.476, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.524}\n",
      "611 {'neu': 0.683, 'compound': -0.7579, 'pos': 0.0, 'neg': 0.317}\n",
      "612 {'neu': 0.592, 'compound': -0.7351, 'pos': 0.0, 'neg': 0.408}\n",
      "614 {'neu': 0.284, 'compound': -0.8777, 'pos': 0.0, 'neg': 0.716}\n",
      "615 {'neu': 0.623, 'compound': -0.3098, 'pos': 0.118, 'neg': 0.259}\n",
      "617 {'neu': 0.809, 'compound': -0.2244, 'pos': 0.0, 'neg': 0.191}\n",
      "618 {'neu': 0.922, 'compound': -0.4019, 'pos': 0.0, 'neg': 0.078}\n",
      "621 {'neu': 0.417, 'compound': -0.25, 'pos': 0.24, 'neg': 0.344}\n",
      "622 {'neu': 0.447, 'compound': -0.9396, 'pos': 0.121, 'neg': 0.432}\n",
      "623 {'neu': 0.634, 'compound': -0.9134, 'pos': 0.049, 'neg': 0.317}\n",
      "624 {'neu': 0.4, 'compound': -0.8176, 'pos': 0.0, 'neg': 0.6}\n",
      "625 {'neu': 0.665, 'compound': -0.5892, 'pos': 0.134, 'neg': 0.202}\n",
      "629 {'neu': 0.748, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.252}\n",
      "631 {'neu': 0.68, 'compound': -0.9465, 'pos': 0.0, 'neg': 0.32}\n",
      "634 {'neu': 0.633, 'compound': -0.4404, 'pos': 0.0, 'neg': 0.367}\n",
      "635 {'neu': 0.764, 'compound': -0.5267, 'pos': 0.087, 'neg': 0.149}\n",
      "636 {'neu': 0.88, 'compound': -0.0749, 'pos': 0.05, 'neg': 0.07}\n",
      "637 {'neu': 0.61, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.39}\n",
      "639 {'neu': 0.598, 'compound': -0.7326, 'pos': 0.086, 'neg': 0.316}\n",
      "641 {'neu': 0.351, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.649}\n",
      "644 {'neu': 0.438, 'compound': -0.9113, 'pos': 0.0, 'neg': 0.562}\n",
      "646 {'neu': 0.476, 'compound': -0.4215, 'pos': 0.155, 'neg': 0.369}\n",
      "647 {'neu': 0.366, 'compound': -0.9022, 'pos': 0.0, 'neg': 0.634}\n",
      "649 {'neu': 0.833, 'compound': -0.296, 'pos': 0.0, 'neg': 0.167}\n",
      "650 {'neu': 0.702, 'compound': -0.1739, 'pos': 0.137, 'neg': 0.162}\n",
      "652 {'neu': 0.742, 'compound': -0.0644, 'pos': 0.106, 'neg': 0.152}\n",
      "654 {'neu': 0.092, 'compound': -0.872, 'pos': 0.0, 'neg': 0.908}\n",
      "655 {'neu': 0.764, 'compound': -0.7772, 'pos': 0.097, 'neg': 0.139}\n",
      "656 {'neu': 0.54, 'compound': -0.8439, 'pos': 0.113, 'neg': 0.347}\n",
      "659 {'neu': 0.527, 'compound': -0.9746, 'pos': 0.048, 'neg': 0.425}\n",
      "661 {'neu': 0.693, 'compound': -0.6885, 'pos': 0.101, 'neg': 0.205}\n",
      "663 {'neu': 0.632, 'compound': -0.7845, 'pos': 0.063, 'neg': 0.305}\n",
      "664 {'neu': 0.485, 'compound': -0.4885, 'pos': 0.17, 'neg': 0.345}\n",
      "665 {'neu': 0.421, 'compound': -0.773, 'pos': 0.161, 'neg': 0.418}\n",
      "667 {'neu': 0.52, 'compound': -0.5697, 'pos': 0.0, 'neg': 0.48}\n",
      "668 {'neu': 0.708, 'compound': -0.7906, 'pos': 0.0, 'neg': 0.292}\n",
      "669 {'neu': 0.656, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.344}\n",
      "670 {'neu': 0.501, 'compound': -0.8957, 'pos': 0.105, 'neg': 0.394}\n",
      "672 {'neu': 0.429, 'compound': -0.7691, 'pos': 0.184, 'neg': 0.387}\n",
      "674 {'neu': 0.884, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.116}\n",
      "676 {'neu': 0.182, 'compound': -0.926, 'pos': 0.0, 'neg': 0.818}\n",
      "678 {'neu': 0.32, 'compound': -0.9686, 'pos': 0.106, 'neg': 0.575}\n",
      "679 {'neu': 0.469, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.531}\n",
      "681 {'neu': 0.365, 'compound': -0.5149, 'pos': 0.265, 'neg': 0.371}\n",
      "682 {'neu': 0.657, 'compound': -0.3612, 'pos': 0.08, 'neg': 0.263}\n",
      "684 {'neu': 0.718, 'compound': -0.6808, 'pos': 0.059, 'neg': 0.223}\n",
      "690 {'neu': 0.708, 'compound': -0.7523, 'pos': 0.0, 'neg': 0.292}\n",
      "691 {'neu': 0.536, 'compound': -0.8255, 'pos': 0.0, 'neg': 0.464}\n",
      "692 {'neu': 0.745, 'compound': -0.5803, 'pos': 0.0, 'neg': 0.255}\n",
      "693 {'neu': 0.545, 'compound': -0.6597, 'pos': 0.161, 'neg': 0.294}\n",
      "694 {'neu': 0.564, 'compound': -0.7579, 'pos': 0.177, 'neg': 0.259}\n",
      "696 {'neu': 0.577, 'compound': -0.8825, 'pos': 0.0, 'neg': 0.423}\n",
      "697 {'neu': 0.569, 'compound': -0.7661, 'pos': 0.102, 'neg': 0.33}\n",
      "698 {'neu': 0.761, 'compound': -0.8555, 'pos': 0.0, 'neg': 0.239}\n",
      "699 {'neu': 0.592, 'compound': -0.8989, 'pos': 0.099, 'neg': 0.308}\n",
      "701 {'neu': 0.68, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.32}\n",
      "703 {'neu': 0.744, 'compound': -0.2866, 'pos': 0.099, 'neg': 0.157}\n",
      "704 {'neu': 0.262, 'compound': -0.8832, 'pos': 0.126, 'neg': 0.612}\n",
      "706 {'neu': 0.486, 'compound': -0.8602, 'pos': 0.0, 'neg': 0.514}\n",
      "707 {'neu': 0.41, 'compound': -0.4939, 'pos': 0.205, 'neg': 0.385}\n",
      "709 {'neu': 0.303, 'compound': -0.784, 'pos': 0.0, 'neg': 0.697}\n",
      "710 {'neu': 0.365, 'compound': -0.8652, 'pos': 0.0, 'neg': 0.635}\n",
      "711 {'neu': 0.781, 'compound': -0.7236, 'pos': 0.0, 'neg': 0.219}\n",
      "713 {'neu': 0.637, 'compound': -0.6908, 'pos': 0.0, 'neg': 0.363}\n",
      "714 {'neu': 0.521, 'compound': -0.9151, 'pos': 0.047, 'neg': 0.431}\n",
      "715 {'neu': 0.543, 'compound': -0.6486, 'pos': 0.131, 'neg': 0.326}\n",
      "716 {'neu': 0.632, 'compound': -0.5826, 'pos': 0.117, 'neg': 0.252}\n",
      "717 {'neu': 0.613, 'compound': -0.8015, 'pos': 0.0, 'neg': 0.387}\n",
      "719 {'neu': 0.377, 'compound': -0.9628, 'pos': 0.0, 'neg': 0.623}\n",
      "720 {'neu': 0.611, 'compound': -0.7184, 'pos': 0.0, 'neg': 0.389}\n",
      "721 {'neu': 0.756, 'compound': -0.4767, 'pos': 0.096, 'neg': 0.149}\n",
      "722 {'neu': 0.65, 'compound': -0.9064, 'pos': 0.115, 'neg': 0.235}\n",
      "724 {'neu': 0.47, 'compound': -0.836, 'pos': 0.0, 'neg': 0.53}\n",
      "725 {'neu': 0.682, 'compound': -0.9534, 'pos': 0.101, 'neg': 0.217}\n",
      "726 {'neu': 0.914, 'compound': -0.6269, 'pos': 0.0, 'neg': 0.086}\n",
      "727 {'neu': 0.556, 'compound': -0.8395, 'pos': 0.0, 'neg': 0.444}\n",
      "728 {'neu': 0.889, 'compound': -0.8952, 'pos': 0.0, 'neg': 0.111}\n",
      "730 {'neu': 0.545, 'compound': -0.9945, 'pos': 0.017, 'neg': 0.438}\n",
      "731 {'neu': 0.541, 'compound': -0.5267, 'pos': 0.0, 'neg': 0.459}\n",
      "732 {'neu': 0.73, 'compound': -0.0943, 'pos': 0.117, 'neg': 0.153}\n",
      "733 {'neu': 0.719, 'compound': -0.6908, 'pos': 0.0, 'neg': 0.281}\n",
      "737 {'neu': 0.527, 'compound': -0.717, 'pos': 0.127, 'neg': 0.346}\n",
      "739 {'neu': 0.764, 'compound': -0.2942, 'pos': 0.095, 'neg': 0.141}\n",
      "740 {'neu': 0.796, 'compound': -0.6569, 'pos': 0.0, 'neg': 0.204}\n",
      "741 {'neu': 0.364, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.636}\n",
      "743 {'neu': 0.6, 'compound': -0.6369, 'pos': 0.136, 'neg': 0.264}\n",
      "746 {'neu': 0.91, 'compound': -0.2924, 'pos': 0.0, 'neg': 0.09}\n",
      "747 {'neu': 0.508, 'compound': -0.4404, 'pos': 0.0, 'neg': 0.492}\n",
      "748 {'neu': 0.737, 'compound': -0.8955, 'pos': 0.063, 'neg': 0.2}\n",
      "752 {'neu': 0.554, 'compound': -0.8017, 'pos': 0.163, 'neg': 0.283}\n",
      "753 {'neu': 0.49, 'compound': -0.5363, 'pos': 0.194, 'neg': 0.316}\n",
      "755 {'neu': 0.789, 'compound': -0.2538, 'pos': 0.105, 'neg': 0.106}\n",
      "756 {'neu': 0.741, 'compound': -0.2732, 'pos': 0.0, 'neg': 0.259}\n",
      "757 {'neu': 0.654, 'compound': -0.8439, 'pos': 0.101, 'neg': 0.246}\n",
      "759 {'neu': 0.74, 'compound': -0.8292, 'pos': 0.0, 'neg': 0.26}\n",
      "760 {'neu': 0.817, 'compound': -0.6166, 'pos': 0.0, 'neg': 0.183}\n",
      "761 {'neu': 0.545, 'compound': -0.4019, 'pos': 0.176, 'neg': 0.279}\n",
      "762 {'neu': 0.625, 'compound': -0.68, 'pos': 0.123, 'neg': 0.252}\n",
      "763 {'neu': 0.662, 'compound': -0.8976, 'pos': 0.045, 'neg': 0.293}\n",
      "764 {'neu': 0.408, 'compound': -0.4404, 'pos': 0.0, 'neg': 0.592}\n",
      "765 {'neu': 0.629, 'compound': -0.6808, 'pos': 0.076, 'neg': 0.295}\n",
      "767 {'neu': 0.449, 'compound': -0.5994, 'pos': 0.0, 'neg': 0.551}\n",
      "768 {'neu': 0.756, 'compound': -0.7527, 'pos': 0.06, 'neg': 0.184}\n",
      "769 {'neu': 0.507, 'compound': -0.9345, 'pos': 0.088, 'neg': 0.405}\n",
      "770 {'neu': 0.36, 'compound': -0.7964, 'pos': 0.0, 'neg': 0.64}\n",
      "771 {'neu': 0.504, 'compound': -0.7845, 'pos': 0.0, 'neg': 0.496}\n",
      "772 {'neu': 0.364, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.636}\n",
      "774 {'neu': 0.679, 'compound': -0.6369, 'pos': 0.0, 'neg': 0.321}\n",
      "775 {'neu': 0.631, 'compound': -0.3818, 'pos': 0.126, 'neg': 0.243}\n",
      "776 {'neu': 0.645, 'compound': -0.296, 'pos': 0.0, 'neg': 0.355}\n",
      "777 {'neu': 0.662, 'compound': -0.5542, 'pos': 0.0, 'neg': 0.338}\n",
      "778 {'neu': 0.736, 'compound': -0.2241, 'pos': 0.114, 'neg': 0.15}\n",
      "786 {'neu': 0.235, 'compound': -0.7579, 'pos': 0.0, 'neg': 0.765}\n",
      "789 {'neu': 0.602, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.398}\n",
      "790 {'neu': 0.685, 'compound': -0.4257, 'pos': 0.147, 'neg': 0.168}\n",
      "791 {'neu': 0.318, 'compound': -0.8934, 'pos': 0.0, 'neg': 0.682}\n",
      "792 {'neu': 0.735, 'compound': -0.9719, 'pos': 0.065, 'neg': 0.2}\n",
      "793 {'neu': 0.694, 'compound': -0.296, 'pos': 0.0, 'neg': 0.306}\n",
      "794 {'neu': 0.768, 'compound': -0.453, 'pos': 0.098, 'neg': 0.133}\n",
      "795 {'neu': 0.71, 'compound': -0.886, 'pos': 0.074, 'neg': 0.216}\n",
      "796 {'neu': 0.227, 'compound': -0.7783, 'pos': 0.0, 'neg': 0.773}\n",
      "798 {'neu': 0.828, 'compound': -0.4019, 'pos': 0.056, 'neg': 0.116}\n",
      "802 {'neu': 0.833, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.167}\n",
      "803 {'neu': 0.484, 'compound': -0.8761, 'pos': 0.089, 'neg': 0.426}\n",
      "804 {'neu': 0.794, 'compound': -0.6597, 'pos': 0.0, 'neg': 0.206}\n",
      "805 {'neu': 0.903, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.097}\n",
      "806 {'neu': 0.583, 'compound': -0.9769, 'pos': 0.0, 'neg': 0.417}\n",
      "808 {'neu': 0.598, 'compound': -0.5216, 'pos': 0.0, 'neg': 0.402}\n",
      "810 {'neu': 0.629, 'compound': -0.7946, 'pos': 0.0, 'neg': 0.371}\n",
      "811 {'neu': 0.694, 'compound': -0.8971, 'pos': 0.061, 'neg': 0.246}\n",
      "813 {'neu': 0.494, 'compound': -0.7345, 'pos': 0.142, 'neg': 0.364}\n",
      "815 {'neu': 0.46, 'compound': -0.7925, 'pos': 0.0, 'neg': 0.54}\n",
      "816 {'neu': 0.222, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.778}\n",
      "818 {'neu': 0.621, 'compound': -0.9118, 'pos': 0.0, 'neg': 0.379}\n",
      "820 {'neu': 0.718, 'compound': -0.8859, 'pos': 0.0, 'neg': 0.282}\n",
      "821 {'neu': 0.596, 'compound': -0.7531, 'pos': 0.0, 'neg': 0.404}\n",
      "825 {'neu': 0.334, 'compound': -0.6115, 'pos': 0.0, 'neg': 0.666}\n",
      "826 {'neu': 0.678, 'compound': -0.1779, 'pos': 0.141, 'neg': 0.181}\n",
      "828 {'neu': 0.377, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.623}\n",
      "834 {'neu': 0.515, 'compound': -0.765, 'pos': 0.0, 'neg': 0.485}\n",
      "839 {'neu': 0.846, 'compound': -0.4767, 'pos': 0.0, 'neg': 0.154}\n",
      "840 {'neu': 0.886, 'compound': -0.4003, 'pos': 0.0, 'neg': 0.114}\n",
      "842 {'neu': 0.658, 'compound': -0.7205, 'pos': 0.116, 'neg': 0.226}\n",
      "843 {'neu': 0.644, 'compound': -0.891, 'pos': 0.0, 'neg': 0.356}\n",
      "848 {'neu': 0.58, 'compound': -0.8653, 'pos': 0.0, 'neg': 0.42}\n",
      "849 {'neu': 0.741, 'compound': -0.1027, 'pos': 0.0, 'neg': 0.259}\n",
      "850 {'neu': 0.796, 'compound': -0.9701, 'pos': 0.089, 'neg': 0.116}\n",
      "851 {'neu': 0.574, 'compound': -0.6705, 'pos': 0.105, 'neg': 0.321}\n",
      "852 {'neu': 0.722, 'compound': -0.5544, 'pos': 0.089, 'neg': 0.189}\n",
      "853 {'neu': 0.799, 'compound': -0.6731, 'pos': 0.0, 'neg': 0.201}\n",
      "854 {'neu': 0.532, 'compound': -0.923, 'pos': 0.0, 'neg': 0.468}\n",
      "855 {'neu': 0.606, 'compound': -0.9536, 'pos': 0.052, 'neg': 0.342}\n",
      "856 {'neu': 0.708, 'compound': -0.7906, 'pos': 0.0, 'neg': 0.292}\n",
      "857 {'neu': 0.765, 'compound': -0.3094, 'pos': 0.116, 'neg': 0.119}\n",
      "858 {'neu': 0.712, 'compound': -0.8777, 'pos': 0.0, 'neg': 0.288}\n",
      "859 {'neu': 0.205, 'compound': -0.7906, 'pos': 0.192, 'neg': 0.603}\n",
      "860 {'neu': 0.46, 'compound': -0.6103, 'pos': 0.155, 'neg': 0.384}\n",
      "863 {'neu': 0.824, 'compound': -0.128, 'pos': 0.0, 'neg': 0.176}\n",
      "864 {'neu': 0.412, 'compound': -0.6458, 'pos': 0.0, 'neg': 0.588}\n",
      "865 {'neu': 0.755, 'compound': -0.3818, 'pos': 0.0, 'neg': 0.245}\n",
      "866 {'neu': 0.575, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.425}\n",
      "867 {'neu': 0.484, 'compound': -0.2003, 'pos': 0.226, 'neg': 0.29}\n",
      "869 {'neu': 0.882, 'compound': -0.0972, 'pos': 0.054, 'neg': 0.064}\n",
      "870 {'neu': 0.748, 'compound': -0.4015, 'pos': 0.0, 'neg': 0.252}\n",
      "872 {'neu': 0.875, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.125}\n",
      "873 {'neu': 0.653, 'compound': -0.7951, 'pos': 0.12, 'neg': 0.227}\n",
      "877 {'neu': 0.497, 'compound': -0.9344, 'pos': 0.0, 'neg': 0.503}\n",
      "878 {'neu': 0.73, 'compound': -0.8271, 'pos': 0.041, 'neg': 0.229}\n",
      "880 {'neu': 0.646, 'compound': -0.5481, 'pos': 0.133, 'neg': 0.221}\n",
      "882 {'neu': 0.727, 'compound': -0.4588, 'pos': 0.0, 'neg': 0.273}\n",
      "883 {'neu': 0.703, 'compound': -0.6942, 'pos': 0.128, 'neg': 0.169}\n",
      "884 {'neu': 0.596, 'compound': -0.9802, 'pos': 0.04, 'neg': 0.364}\n",
      "885 {'neu': 0.736, 'compound': -0.9028, 'pos': 0.0, 'neg': 0.264}\n",
      "886 {'neu': 0.717, 'compound': -0.7545, 'pos': 0.066, 'neg': 0.217}\n",
      "888 {'neu': 0.729, 'compound': -0.9385, 'pos': 0.029, 'neg': 0.242}\n",
      "889 {'neu': 0.601, 'compound': -0.157, 'pos': 0.179, 'neg': 0.22}\n",
      "891 {'neu': 0.829, 'compound': -0.4019, 'pos': 0.0, 'neg': 0.171}\n",
      "892 {'neu': 0.677, 'compound': -0.953, 'pos': 0.107, 'neg': 0.216}\n",
      "894 {'neu': 0.286, 'compound': -0.8176, 'pos': 0.0, 'neg': 0.714}\n",
      "895 {'neu': 0.317, 'compound': -0.6486, 'pos': 0.0, 'neg': 0.683}\n",
      "899 {'neu': 0.476, 'compound': -0.937, 'pos': 0.09, 'neg': 0.433}\n",
      "900 {'neu': 0.8, 'compound': -0.5803, 'pos': 0.0, 'neg': 0.2}\n",
      "904 {'neu': 0.686, 'compound': -0.6239, 'pos': 0.109, 'neg': 0.206}\n",
      "906 {'neu': 0.484, 'compound': -0.5204, 'pos': 0.231, 'neg': 0.285}\n",
      "908 {'neu': 0.821, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.179}\n",
      "909 {'neu': 0.519, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.481}\n",
      "910 {'neu': 0.473, 'compound': -0.9971, 'pos': 0.077, 'neg': 0.45}\n",
      "911 {'neu': 0.694, 'compound': -0.296, 'pos': 0.0, 'neg': 0.306}\n",
      "912 {'neu': 0.226, 'compound': -0.9081, 'pos': 0.173, 'neg': 0.602}\n",
      "913 {'neu': 0.377, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.623}\n",
      "914 {'neu': 0.798, 'compound': -0.2944, 'pos': 0.069, 'neg': 0.133}\n",
      "917 {'neu': 0.586, 'compound': -0.9829, 'pos': 0.031, 'neg': 0.383}\n",
      "918 {'neu': 0.704, 'compound': -0.5848, 'pos': 0.0, 'neg': 0.296}\n",
      "919 {'neu': 0.823, 'compound': -0.9514, 'pos': 0.0, 'neg': 0.177}\n",
      "922 {'neu': 0.691, 'compound': -0.9367, 'pos': 0.036, 'neg': 0.273}\n",
      "923 {'neu': 0.602, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.398}\n",
      "925 {'neu': 0.654, 'compound': -0.9026, 'pos': 0.0, 'neg': 0.346}\n",
      "927 {'neu': 0.504, 'compound': -0.5479, 'pos': 0.224, 'neg': 0.273}\n",
      "928 {'neu': 0.606, 'compound': -0.9077, 'pos': 0.0, 'neg': 0.394}\n",
      "931 {'neu': 0.41, 'compound': -0.9444, 'pos': 0.102, 'neg': 0.488}\n",
      "932 {'neu': 0.506, 'compound': -0.0943, 'pos': 0.234, 'neg': 0.26}\n",
      "933 {'neu': 0.548, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.452}\n",
      "935 {'neu': 0.688, 'compound': -0.4404, 'pos': 0.115, 'neg': 0.198}\n",
      "936 {'neu': 0.548, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.452}\n",
      "938 {'neu': 0.784, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.216}\n",
      "940 {'neu': 0.509, 'compound': -0.9262, 'pos': 0.0, 'neg': 0.491}\n",
      "941 {'neu': 0.688, 'compound': -0.6124, 'pos': 0.0, 'neg': 0.312}\n",
      "943 {'neu': 0.8, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.2}\n",
      "944 {'neu': 0.775, 'compound': -0.2449, 'pos': 0.094, 'neg': 0.132}\n",
      "945 {'neu': 0.787, 'compound': -0.4767, 'pos': 0.066, 'neg': 0.147}\n",
      "946 {'neu': 0.755, 'compound': -0.8681, 'pos': 0.0, 'neg': 0.245}\n",
      "948 {'neu': 0.75, 'compound': -0.4588, 'pos': 0.0, 'neg': 0.25}\n",
      "949 {'neu': 0.638, 'compound': -0.8122, 'pos': 0.0, 'neg': 0.362}\n",
      "950 {'neu': 0.426, 'compound': -0.7184, 'pos': 0.154, 'neg': 0.42}\n",
      "951 {'neu': 0.611, 'compound': -0.7906, 'pos': 0.0, 'neg': 0.389}\n",
      "952 {'neu': 0.523, 'compound': -0.7579, 'pos': 0.154, 'neg': 0.323}\n",
      "953 {'neu': 0.686, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.314}\n",
      "956 {'neu': 0.136, 'compound': -0.9287, 'pos': 0.0, 'neg': 0.864}\n",
      "958 {'neu': 0.716, 'compound': -0.7717, 'pos': 0.053, 'neg': 0.231}\n",
      "959 {'neu': 0.586, 'compound': -0.7182, 'pos': 0.103, 'neg': 0.31}\n",
      "961 {'neu': 0.655, 'compound': -0.7003, 'pos': 0.0, 'neg': 0.345}\n",
      "962 {'neu': 0.136, 'compound': -0.9287, 'pos': 0.0, 'neg': 0.864}\n",
      "963 {'neu': 0.453, 'compound': -0.7212, 'pos': 0.0, 'neg': 0.547}\n",
      "965 {'neu': 0.707, 'compound': -0.8717, 'pos': 0.044, 'neg': 0.249}\n",
      "967 {'neu': 0.596, 'compound': -0.1779, 'pos': 0.179, 'neg': 0.225}\n",
      "969 {'neu': 0.778, 'compound': -0.5574, 'pos': 0.06, 'neg': 0.162}\n",
      "970 {'neu': 0.579, 'compound': -0.5267, 'pos': 0.14, 'neg': 0.281}\n",
      "971 {'neu': 0.571, 'compound': -0.926, 'pos': 0.0, 'neg': 0.429}\n",
      "976 {'neu': 0.612, 'compound': -0.7906, 'pos': 0.075, 'neg': 0.313}\n",
      "978 {'neu': 0.718, 'compound': -0.6486, 'pos': 0.092, 'neg': 0.19}\n",
      "979 {'neu': 0.867, 'compound': -0.4696, 'pos': 0.0, 'neg': 0.133}\n",
      "981 {'neu': 0.57, 'compound': -0.91, 'pos': 0.067, 'neg': 0.363}\n",
      "982 {'neu': 0.741, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.259}\n",
      "983 {'neu': 0.842, 'compound': -0.128, 'pos': 0.0, 'neg': 0.158}\n",
      "985 {'neu': 0.558, 'compound': -0.8516, 'pos': 0.0, 'neg': 0.442}\n",
      "986 {'neu': 0.504, 'compound': -0.5479, 'pos': 0.224, 'neg': 0.273}\n",
      "987 {'neu': 0.848, 'compound': -0.6124, 'pos': 0.0, 'neg': 0.152}\n",
      "988 {'neu': 0.373, 'compound': -0.5216, 'pos': 0.0, 'neg': 0.627}\n",
      "989 {'neu': 0.448, 'compound': -0.5709, 'pos': 0.0, 'neg': 0.552}\n",
      "990 {'neu': 0.387, 'compound': -0.4043, 'pos': 0.265, 'neg': 0.348}\n",
      "992 {'neu': 0.356, 'compound': -0.9457, 'pos': 0.0, 'neg': 0.644}\n",
      "994 {'neu': 0.791, 'compound': -0.8035, 'pos': 0.075, 'neg': 0.133}\n",
      "995 {'neu': 0.713, 'compound': -0.858, 'pos': 0.0, 'neg': 0.287}\n",
      "996 {'neu': 0.906, 'compound': -0.6652, 'pos': 0.0, 'neg': 0.094}\n",
      "997 {'neu': 0.654, 'compound': -0.2869, 'pos': 0.154, 'neg': 0.192}\n",
      "998 {'neu': 0.702, 'compound': -0.8361, 'pos': 0.0, 'neg': 0.298}\n",
      "1000 {'neu': 0.448, 'compound': -0.9537, 'pos': 0.0, 'neg': 0.552}\n",
      "1002 {'neu': 0.893, 'compound': -0.6072, 'pos': 0.0, 'neg': 0.107}\n",
      "1003 {'neu': 0.69, 'compound': -0.5574, 'pos': 0.0, 'neg': 0.31}\n",
      "1004 {'neu': 0.816, 'compound': -0.128, 'pos': 0.085, 'neg': 0.099}\n",
      "1005 {'neu': 0.61, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.39}\n",
      "1006 {'neu': 0.634, 'compound': -0.9485, 'pos': 0.0, 'neg': 0.366}\n",
      "1007 {'neu': 0.799, 'compound': -0.6476, 'pos': 0.0, 'neg': 0.201}\n",
      "1009 {'neu': 0.556, 'compound': -0.101, 'pos': 0.208, 'neg': 0.236}\n",
      "1010 {'neu': 0.932, 'compound': -0.2263, 'pos': 0.0, 'neg': 0.068}\n",
      "1013 {'neu': 0.625, 'compound': -0.2023, 'pos': 0.167, 'neg': 0.208}\n",
      "1014 {'neu': 0.516, 'compound': -0.8583, 'pos': 0.083, 'neg': 0.401}\n",
      "1017 {'neu': 0.616, 'compound': -0.4767, 'pos': 0.118, 'neg': 0.265}\n",
      "1019 {'neu': 0.845, 'compound': -0.6575, 'pos': 0.035, 'neg': 0.12}\n",
      "1020 {'neu': 0.742, 'compound': -0.4824, 'pos': 0.0, 'neg': 0.258}\n",
      "1021 {'neu': 0.816, 'compound': -0.5815, 'pos': 0.051, 'neg': 0.134}\n",
      "1022 {'neu': 0.775, 'compound': -0.4939, 'pos': 0.0, 'neg': 0.225}\n",
      "1026 {'neu': 0.924, 'compound': -0.357, 'pos': 0.0, 'neg': 0.076}\n",
      "1027 {'neu': 0.908, 'compound': -0.3869, 'pos': 0.0, 'neg': 0.092}\n",
      "1028 {'neu': 0.426, 'compound': -0.1027, 'pos': 0.266, 'neg': 0.309}\n",
      "1029 {'neu': 0.791, 'compound': -0.4404, 'pos': 0.0, 'neg': 0.209}\n",
      "1030 {'neu': 0.114, 'compound': -0.8316, 'pos': 0.0, 'neg': 0.886}\n",
      "1033 {'neu': 0.575, 'compound': -0.5719, 'pos': 0.0, 'neg': 0.425}\n",
      "1034 {'neu': 0.757, 'compound': -0.8402, 'pos': 0.0, 'neg': 0.243}\n",
      "1035 {'neu': 0.82, 'compound': -0.5106, 'pos': 0.0, 'neg': 0.18}\n",
      "1037 {'neu': 0.658, 'compound': -0.9574, 'pos': 0.094, 'neg': 0.248}\n",
      "1041 {'neu': 0.698, 'compound': -0.9638, 'pos': 0.113, 'neg': 0.189}\n",
      "1043 {'neu': 0.533, 'compound': -0.5423, 'pos': 0.0, 'neg': 0.467}\n",
      "1044 {'neu': 0.78, 'compound': -0.8515, 'pos': 0.033, 'neg': 0.187}\n",
      "1046 {'neu': 0.901, 'compound': -0.5574, 'pos': 0.0, 'neg': 0.099}\n",
      "1048 {'neu': 0.777, 'compound': -0.3182, 'pos': 0.0, 'neg': 0.223}\n",
      "0.6301239275500476\n",
      "\"fucking behave then you prick!\"\n"
     ]
    }
   ],
   "source": [
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "analyser = SentimentIntensityAnalyzer()\n",
    "cnt = 0\n",
    "insult_cmt = insult.reset_index()\n",
    "for i in range(len(insult_cmt)):\n",
    "    snt = analyser.polarity_scores(insult_cmt['Comment'][i])\n",
    "    if snt.get('neg') > snt.get('pos'):\n",
    "        cnt += 1\n",
    "#        print(i,snt)\n",
    "print(cnt/len(insult))\n",
    "print(insult_cmt['Comment'][10])\n",
    "# only about 11% of the ban words are categorized as having negative connotation according to Vader.\n",
    "# only 9% of the insult comments from Kaggle competition are negative if neg > .5 is counted.\n",
    "# (when negative > positive: 63% : this is similar to our NBC result)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!!!NaiveBayesClassifier!!!\n",
      "0.6335570469798658\n",
      "Most Informative Features\n",
      "                       � = True              ins : norm   =     48.6 : 1.0\n",
      "               direction = True             norm : ins    =     30.3 : 1.0\n",
      "                    cunt = True              ins : norm   =     22.7 : 1.0\n",
      "                    game = True             norm : ins    =     22.6 : 1.0\n",
      "                     win = True             norm : ins    =     19.7 : 1.0\n",
      "                  result = True             norm : ins    =     18.4 : 1.0\n",
      "                      sa = True             norm : ins    =     18.1 : 1.0\n",
      "                   crawl = True              ins : norm   =     17.6 : 1.0\n",
      "                   happy = True             norm : ins    =     16.8 : 1.0\n",
      "                 hundred = True             norm : ins    =     16.8 : 1.0\n",
      "None\n",
      "!!!MaxEntropyClassifier!!!\n",
      "0.5552572706935123\n",
      "  -0.000 direction==True and label is 'ins'\n",
      "  -0.000 game==True and label is 'ins'\n",
      "  -0.000 win==True and label is 'ins'\n",
      "  -0.000 result==True and label is 'ins'\n",
      "  -0.000 sa==True and label is 'ins'\n",
      "  -0.000 hundred==True and label is 'ins'\n",
      "  -0.000 happy==True and label is 'ins'\n",
      "  -0.000 abortion==True and label is 'ins'\n",
      "  -0.000 thank==True and label is 'ins'\n",
      "  -0.000 4==True and label is 'ins'\n",
      "None\n",
      "!!!SVM!!!\n",
      "0.6219239373601789\n"
     ]
    }
   ],
   "source": [
    "from nltk import classify\n",
    "import nltk.classify.util, nltk.metrics\n",
    "from nltk.classify import NaiveBayesClassifier, MaxentClassifier, SklearnClassifier, DecisionTreeClassifier\n",
    "from sklearn import cross_validation\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "import itertools\n",
    "from nltk.collocations import BigramCollocationFinder\n",
    "from nltk.metrics import BigramAssocMeasures\n",
    "\n",
    "print(\"!!!NaiveBayesClassifier!!!\")\n",
    "classifier = NaiveBayesClassifier.train(train_set)\n",
    "accuracy = classify.accuracy(classifier, test_set)\n",
    "print(accuracy) # Output: 0.765\n",
    "print (classifier.show_most_informative_features(10)) \n",
    "\n",
    "print(\"!!!MaxEntropyClassifier!!!\")\n",
    "classifier = MaxentClassifier.train(train_set, 'GIS', trace=0, encoding=None, labels=None, gaussian_prior_sigma=0, max_iter = 1)\n",
    "accuracy = nltk.classify.util.accuracy(classifier, test_set)\n",
    "print(accuracy)\n",
    "print(classifier.show_most_informative_features(10))\n",
    "\n",
    "print(\"!!!SVM!!!\")\n",
    "svmclassifier = SklearnClassifier(SVC(kernel='linear',probability=True), sparse=False)\n",
    "svmclassifier.train(train_set)\n",
    "accuracy = nltk.classify.util.accuracy(svmclassifier, test_set)\n",
    "print(accuracy)\n",
    "#print(classifier.show_most_informative_features(10))\n",
    "\n",
    "# Decision Tree takes forever to run... need to find an alternative!\n",
    "#print(\"!!!DecisionTree!!!\")\n",
    "#dtclassifier = DecisionTreeClassifier.train(train_set)\n",
    "#accuracy = nltk.classify.util.accuracy(dtclassifier, test_set)\n",
    "#print(accuracy)\n",
    "#print(dtclassifier.show_most_informative_features(10))\n",
    "#pos_precision = nltk.metrics.precision(refsets['pos'], testsets['pos'])\n",
    "#pos_recall = nltk.metrics.recall(refsets['pos'], testsets['pos'])\n",
    "#pos_fmeasure = nltk.metrics.f_measure(refsets['pos'], testsets['pos'])\n",
    "#neg_precision = nltk.metrics.precision(refsets['neg'], testsets['neg'])\n",
    "#neg_recall = nltk.metrics.recall(refsets['neg'], testsets['neg'])\n",
    "#neg_fmeasure =  nltk.metrics.f_measure(refsets['neg'], testsets['neg'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'parliament': True, 'racist': True, 'university': True, '–': True, 'mp': True, 'racism': True, 'lash': True, 'cape': True, 'academ': True, 'town': True, 'academic': True, '...': True, 'uct': True}\n",
      "norm\n",
      "<ProbDist with 2 samples>\n",
      "norm\n",
      "0.0463495889777\n",
      "0.953650411022\n",
      "norm\n",
      "<ProbDist with 2 samples>\n",
      "norm\n",
      "0.0126036422799\n",
      "0.98739635772\n"
     ]
    }
   ],
   "source": [
    "custom_tweet = \"I hated the film. It was a disaster. Poor direction, bad acting.\"\n",
    "custom_tweet_set = bag_of_words(agg_tweets[30]['text'])\n",
    "print(custom_tweet_set)\n",
    "print (svmclassifier.classify(custom_tweet_set)) # Output: neg\n",
    "# Negative tweet correctly classified as negative\n",
    "\n",
    "# probability result\n",
    "prob_result = svmclassifier.prob_classify(custom_tweet_set)\n",
    "print (prob_result) # Output: <ProbDist with 2 samples>\n",
    "print (prob_result.max()) # Output: neg\n",
    "print (prob_result.prob(\"ins\")) # Output: 0.941844352481\n",
    "print (prob_result.prob(\"norm\")) # Output: 0.0581556475194\n",
    " \n",
    " \n",
    "custom_tweet = \"It was a wonderful and amazing movie. I loved it. Best direction, good acting.\"\n",
    "custom_tweet_set = bag_of_words(custom_tweet)\n",
    " \n",
    "print (svmclassifier.classify(custom_tweet_set)) # Output: pos\n",
    "# Positive tweet correctly classified as positive\n",
    " \n",
    "# probability result\n",
    "prob_result = svmclassifier.prob_classify(custom_tweet_set)\n",
    "print (prob_result) # Output: <ProbDist with 2 samples>\n",
    "print (prob_result.max()) # Output: pos\n",
    "print (prob_result.prob(\"ins\")) # Output: 0.00131055449755\n",
    "print (prob_result.prob(\"norm\")) # Output: 0.998689445502"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [anaconda3]",
   "language": "python",
   "name": "Python [anaconda3]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}