notebooks/naive_bayes_nltk_sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment analysis with NLTK Naive Bayes (Bernoulli?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes with NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Naive Bayes classifier determines the probability that an input text belongs to one of a set of classes, eg. predicting if a review is positive or negative.\n",
    "\n",
    "It is ‘Naive’ because it assumes the words in the text are independent (even though in reality, in natural human language, the order of words convey contextual information).  Despite these assumptions, Naive Bayes has a high degree of accuracy when predicting classes with only a small training set.\n",
    "\n",
    "- Baines, O., Naive Bayes: Machine Learning and Text Classification Application of Bayes’ Theorem. https://journals.le.ac.uk/ojs1/index.php/lumj/article/download/3484/3110"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.metrics.scores import precision, recall, f_measure\n",
    "import pandas as pd\n",
    "import collections\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"..\") # Adds higher directory to python modules path.\n",
    "from NLPmoviereviews.data import load_data_sent\n",
    "from NLPmoviereviews.utilities import preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-30 13:16:08.477119: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.2\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-08-30 13:16:08.554841: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)\n"
     ]
    }
   ],
   "source": [
    "# load data\n",
    "X_train, y_train, X_test, y_test = load_data_sent(percentage_of_sentences=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Prepare text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove custom stop-words\n",
    "def rm_custom_stops(sentence):\n",
    "    '''\n",
    "    Custom stop word remover\n",
    "    Parameters:\n",
    "        sentence (str): a string of words\n",
    "    Returns:\n",
    "        list_of_words (list): cleaned sentence as a list of words\n",
    "    '''\n",
    "    words = sentence.split()\n",
    "    stop_words = {'br', 'movie', 'film'}\n",
    "    \n",
    "    return [w for w in words if not w in stop_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform preprocessing (cleaning) & transform to dataframe\n",
    "def process_df(X, y):\n",
    "    '''\n",
    "    Transform texts and labels into dataframe of \n",
    "    cleaned texts (as list of words) and human readable target labels\n",
    "    \n",
    "    Parameters:\n",
    "        X (list): list of strings (reviews)\n",
    "        y (list): list of target labels (0/1)\n",
    "    Returns:\n",
    "        df (dataframe): dataframe of processed reviews (as list of words)\n",
    "                        and corresponding sentiment label (positive/negative)\n",
    "    '''\n",
    "    # create dataframe from data\n",
    "    d = {'text': X, 'sentiment': y}\n",
    "    df = pd.DataFrame(d)\n",
    "    \n",
    "    # make sentiment human-readable\n",
    "    df['sentiment'] = df.sentiment.map(lambda x: 'positive' if x==1 else 'negative')\n",
    "\n",
    "    # clean and split text into list of words\n",
    "    df['text'] = df.text.apply(preprocessing)\n",
    "    df['text'] = df.text.apply(rm_custom_stops)\n",
    "\n",
    "    # Generate the feature sets for the movie review documents one by one\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# process data\n",
    "train_df = process_df(X_train, y_train)\n",
    "test_df = process_df(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[absolutely, terrible, dont, lure, christopher...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[know, fall, asleep, usually, due, combination...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[mann, photograph, alberta, rocky, mountain, s...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[kind, snowy, sunday, afternoon, rest, world, ...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[others, mention, woman, go, nude, mostly, abs...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text sentiment\n",
       "0  [absolutely, terrible, dont, lure, christopher...  negative\n",
       "1  [know, fall, asleep, usually, due, combination...  negative\n",
       "2  [mann, photograph, alberta, rocky, mountain, s...  negative\n",
       "3  [kind, snowy, sunday, afternoon, rest, world, ...  positive\n",
       "4  [others, mention, woman, go, nude, mostly, abs...  positive"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inspect dataframe\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Create list of most common words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get frequency distribution of words in corpus & select 2000 most common words\n",
    "def most_common(df, n=2000):\n",
    "    '''\n",
    "    Get n most common words from data frame of text reviews\n",
    "    \n",
    "    Parameters:\n",
    "        df (dataframe): dataframe with column of processed text reviews\n",
    "        n (int): number of most common words to get\n",
    "    Returns:\n",
    "        most_common_words (list): list of n most common words\n",
    "    '''\n",
    "    # create list of all words in the train data\n",
    "    complete_corpus = df.text.sum()\n",
    "    \n",
    "    # Construct a frequency dict of all words in the overall corpus \n",
    "    all_words = nltk.FreqDist(w.lower() for w in complete_corpus)\n",
    "\n",
    "    # select the 2,000 most frequent words (incl. frequency)\n",
    "    most_common_words = all_words.most_common(n)\n",
    "    \n",
    "    return [item[0] for item in most_common_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['one',\n",
       " 'make',\n",
       " 'like',\n",
       " 'see',\n",
       " 'get',\n",
       " 'time',\n",
       " 'good',\n",
       " 'go',\n",
       " 'watch',\n",
       " 'character']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get 2000 most common words\n",
    "most_common_2000 = most_common(train_df)\n",
    "\n",
    "# inspect first 10 most common words\n",
    "most_common_2000[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Create nltk featuresets from train/test\n",
    "\n",
    "For the nltk naive bayes classifier, we must tokenize the sentence and figure out which words the sentence shares with all_words/most_common_words. These constitute the sentence's features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for a given text, create a featureset (dict of features - {'word': True/False})\n",
    "def review_features(review, most_common_words):\n",
    "    '''\n",
    "    Feature extractor that checks whether each of the most\n",
    "    common words is present in a given review\n",
    "    \n",
    "    Parameters:\n",
    "        review (list): text reviews as list of words\n",
    "        most_common_words (list): list of n most common words\n",
    "    Returns:\n",
    "        features (dict): dict of most common words & corresponding True/False\n",
    "    '''\n",
    "    review_words = set(review)\n",
    "    features = {}\n",
    "    for word in most_common_words:\n",
    "        features['contains(%s)' % word] = (word in review_words)\n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create featureset for each text in a given dataframe\n",
    "def make_set(df, most_common_words):\n",
    "    '''\n",
    "    Generates nltk featuresets for each movie review in dataframe.\n",
    "    Feature sets are composed of a dict describing whether each of the most \n",
    "    common words is present in the text review or not\n",
    "\n",
    "    Parameters:\n",
    "        df (dataframe): processed dataframe of text reviews\n",
    "        most_common_words (list): list of most common words\n",
    "    Returns:\n",
    "        feature_set (list): list of dicts of most common words & corresponding True/False\n",
    "    '''\n",
    "    return [(review_features(df.text[i], most_common_words), df.sentiment[i]) for i in range(len(df.sentiment))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make data into featuresets (for nltk naive bayes classifier)\n",
    "train_set = make_set(train_df, most_common_2000)\n",
    "test_set = make_set(test_df, most_common_2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'contains(one)': False,\n",
       "  'contains(make)': True,\n",
       "  'contains(like)': True,\n",
       "  'contains(see)': False,\n",
       "  'contains(get)': False,\n",
       "  'contains(time)': False,\n",
       "  'contains(good)': True,\n",
       "  'contains(go)': False,\n",
       "  'contains(watch)': False,\n",
       "  'contains(character)': False,\n",
       "  'contains(even)': True,\n",
       "  'contains(think)': False,\n",
       "  'contains(story)': False,\n",
       "  'contains(would)': False,\n",
       "  'contains(really)': False,\n",
       "  'contains(show)': False,\n",
       "  'contains(scene)': True,\n",
       "  'contains(well)': False,\n",
       "  'contains(look)': False,\n",
       "  'contains(end)': False,\n",
       "  'contains(say)': False,\n",
       "  'contains(much)': False,\n",
       "  'contains(give)': False,\n",
       "  'contains(great)': True,\n",
       "  'contains(first)': False,\n",
       "  'contains(bad)': False,\n",
       "  'contains(act)': True,\n",
       "  'contains(way)': False,\n",
       "  'contains(know)': False,\n",
       "  'contains(also)': False,\n",
       "  'contains(people)': False,\n",
       "  'contains(dont)': True,\n",
       "  'contains(take)': False,\n",
       "  'contains(thing)': False,\n",
       "  'contains(play)': False,\n",
       "  'contains(find)': False,\n",
       "  'contains(love)': True,\n",
       "  'contains(come)': False,\n",
       "  'contains(could)': True,\n",
       "  'contains(plot)': False,\n",
       "  'contains(actor)': True,\n",
       "  'contains(seem)': False,\n",
       "  'contains(want)': False,\n",
       "  'contains(best)': False,\n",
       "  'contains(two)': False,\n",
       "  'contains(work)': False,\n",
       "  'contains(many)': False,\n",
       "  'contains(life)': False,\n",
       "  'contains(try)': False,\n",
       "  'contains(year)': False,\n",
       "  'contains(never)': False,\n",
       "  'contains(man)': False,\n",
       "  'contains(little)': False,\n",
       "  'contains(better)': False,\n",
       "  'contains(ever)': False,\n",
       "  'contains(still)': False,\n",
       "  'contains(director)': False,\n",
       "  'contains(part)': False,\n",
       "  'contains(back)': False,\n",
       "  'contains(something)': False,\n",
       "  'contains(real)': True,\n",
       "  'contains(im)': False,\n",
       "  'contains(feel)': False,\n",
       "  'contains(interest)': False,\n",
       "  'contains(use)': False,\n",
       "  'contains(performance)': False,\n",
       "  'contains(cast)': False,\n",
       "  'contains(doesnt)': False,\n",
       "  'contains(star)': False,\n",
       "  'contains(write)': False,\n",
       "  'contains(didnt)': False,\n",
       "  'contains(point)': False,\n",
       "  'contains(old)': False,\n",
       "  'contains(though)': False,\n",
       "  'contains(lot)': False,\n",
       "  'contains(role)': True,\n",
       "  'contains(funny)': False,\n",
       "  'contains(new)': False,\n",
       "  'contains(every)': False,\n",
       "  'contains(leave)': False,\n",
       "  'contains(nothing)': True,\n",
       "  'contains(guy)': False,\n",
       "  'contains(another)': False,\n",
       "  'contains(actually)': False,\n",
       "  'contains(woman)': False,\n",
       "  'contains(set)': False,\n",
       "  'contains(tell)': False,\n",
       "  'contains(day)': False,\n",
       "  'contains(start)': False,\n",
       "  'contains(girl)': False,\n",
       "  'contains(bite)': False,\n",
       "  'contains(minute)': False,\n",
       "  'contains(pretty)': False,\n",
       "  'contains(world)': False,\n",
       "  'contains(live)': False,\n",
       "  'contains(turn)': False,\n",
       "  'contains(happen)': False,\n",
       "  'contains(fact)': False,\n",
       "  'contains(horror)': False,\n",
       "  'contains(mean)': True,\n",
       "  'contains(cant)': False,\n",
       "  'contains(quite)': False,\n",
       "  'contains(u)': True,\n",
       "  'contains(young)': False,\n",
       "  'contains(thats)': False,\n",
       "  'contains(become)': False,\n",
       "  'contains(fan)': False,\n",
       "  'contains(around)': False,\n",
       "  'contains(kill)': False,\n",
       "  'contains(family)': False,\n",
       "  'contains(enough)': False,\n",
       "  'contains(series)': False,\n",
       "  'contains(long)': False,\n",
       "  'contains(may)': False,\n",
       "  'contains(do)': False,\n",
       "  'contains(right)': False,\n",
       "  'contains(action)': False,\n",
       "  'contains(however)': False,\n",
       "  'contains(keep)': False,\n",
       "  'contains(original)': False,\n",
       "  'contains(always)': False,\n",
       "  'contains(line)': False,\n",
       "  'contains(run)': False,\n",
       "  'contains(believe)': False,\n",
       "  'contains(big)': False,\n",
       "  'contains(script)': False,\n",
       "  'contains(saw)': False,\n",
       "  'contains(music)': False,\n",
       "  'contains(kid)': False,\n",
       "  'contains(book)': False,\n",
       "  'contains(without)': False,\n",
       "  'contains(ive)': False,\n",
       "  'contains(enjoy)': False,\n",
       "  'contains(isnt)': False,\n",
       "  'contains(begin)': False,\n",
       "  'contains(need)': False,\n",
       "  'contains(let)': False,\n",
       "  'contains(friend)': False,\n",
       "  'contains(whole)': False,\n",
       "  'contains(far)': False,\n",
       "  'contains(put)': False,\n",
       "  'contains(might)': False,\n",
       "  'contains(lead)': False,\n",
       "  'contains(kind)': False,\n",
       "  'contains(comedy)': False,\n",
       "  'contains(least)': False,\n",
       "  'contains(there)': False,\n",
       "  'contains(tv)': False,\n",
       "  'contains(must)': True,\n",
       "  'contains(last)': False,\n",
       "  'contains(since)': False,\n",
       "  'contains(shoot)': False,\n",
       "  'contains(place)': False,\n",
       "  'contains(screen)': False,\n",
       "  'contains(name)': True,\n",
       "  'contains(away)': False,\n",
       "  'contains(almost)': False,\n",
       "  'contains(bore)': False,\n",
       "  'contains(reason)': False,\n",
       "  'contains(fun)': False,\n",
       "  'contains(help)': False,\n",
       "  'contains(rather)': False,\n",
       "  'contains(especially)': False,\n",
       "  'contains(moment)': False,\n",
       "  'contains(anything)': False,\n",
       "  'contains(hard)': False,\n",
       "  'contains(read)': False,\n",
       "  'contains(call)': False,\n",
       "  'contains(laugh)': False,\n",
       "  'contains(audience)': False,\n",
       "  'contains(effect)': False,\n",
       "  'contains(sure)': False,\n",
       "  'contains(three)': False,\n",
       "  'contains(bring)': False,\n",
       "  'contains(expect)': False,\n",
       "  'contains(he)': False,\n",
       "  'contains(worst)': True,\n",
       "  'contains(american)': False,\n",
       "  'contains(anyone)': False,\n",
       "  'contains(child)': False,\n",
       "  'contains(second)': False,\n",
       "  'contains(maybe)': False,\n",
       "  'contains(course)': False,\n",
       "  'contains(night)': False,\n",
       "  'contains(probably)': False,\n",
       "  'contains(yet)': False,\n",
       "  'contains(although)': False,\n",
       "  'contains(money)': False,\n",
       "  'contains(eye)': False,\n",
       "  'contains(wasnt)': False,\n",
       "  'contains(idea)': False,\n",
       "  'contains(everything)': False,\n",
       "  'contains(dvd)': False,\n",
       "  'contains(job)': False,\n",
       "  'contains(episode)': False,\n",
       "  'contains(sense)': False,\n",
       "  'contains(true)': False,\n",
       "  'contains(high)': False,\n",
       "  'contains(instead)': False,\n",
       "  'contains(sound)': False,\n",
       "  'contains(someone)': False,\n",
       "  'contains(war)': False,\n",
       "  'contains(understand)': False,\n",
       "  'contains(fall)': False,\n",
       "  'contains(version)': False,\n",
       "  'contains(direct)': False,\n",
       "  'contains(boy)': False,\n",
       "  'contains(main)': False,\n",
       "  'contains(later)': False,\n",
       "  'contains(miss)': False,\n",
       "  'contains(different)': False,\n",
       "  'contains(mind)': False,\n",
       "  'contains(hour)': False,\n",
       "  'contains(lose)': False,\n",
       "  'contains(father)': False,\n",
       "  'contains(move)': False,\n",
       "  'contains(wife)': False,\n",
       "  'contains(le)': False,\n",
       "  'contains(death)': False,\n",
       "  'contains(head)': False,\n",
       "  'contains(follow)': False,\n",
       "  'contains(production)': False,\n",
       "  'contains(else)': False,\n",
       "  'contains(face)': False,\n",
       "  'contains(hand)': False,\n",
       "  'contains(together)': False,\n",
       "  'contains(short)': False,\n",
       "  'contains(house)': False,\n",
       "  'contains(worth)': False,\n",
       "  'contains(problem)': False,\n",
       "  'contains(black)': False,\n",
       "  'contains(recommend)': False,\n",
       "  'contains(die)': False,\n",
       "  'contains(home)': False,\n",
       "  'contains(waste)': False,\n",
       "  'contains(special)': False,\n",
       "  'contains(word)': False,\n",
       "  'contains(view)': False,\n",
       "  'contains(couple)': False,\n",
       "  'contains(attempt)': False,\n",
       "  'contains(meet)': False,\n",
       "  'contains(half)': False,\n",
       "  'contains(fight)': False,\n",
       "  'contains(include)': False,\n",
       "  'contains(top)': False,\n",
       "  'contains(camera)': False,\n",
       "  'contains(viewer)': False,\n",
       "  'contains(surprise)': False,\n",
       "  'contains(mr)': False,\n",
       "  'contains(beautiful)': False,\n",
       "  'contains(hollywood)': False,\n",
       "  'contains(youre)': False,\n",
       "  'contains(appear)': True,\n",
       "  'contains(classic)': False,\n",
       "  'contains(suppose)': False,\n",
       "  'contains(john)': False,\n",
       "  'contains(excellent)': False,\n",
       "  'contains(rat)': False,\n",
       "  'contains(simply)': True,\n",
       "  'contains(involve)': False,\n",
       "  'contains(talk)': False,\n",
       "  'contains(writer)': False,\n",
       "  'contains(either)': False,\n",
       "  'contains(actress)': False,\n",
       "  'contains(stupid)': False,\n",
       "  'contains(poor)': False,\n",
       "  'contains(perfect)': False,\n",
       "  'contains(base)': False,\n",
       "  'contains(decide)': False,\n",
       "  'contains(budget)': False,\n",
       "  'contains(dead)': False,\n",
       "  'contains(wrong)': False,\n",
       "  'contains(murder)': False,\n",
       "  'contains(hear)': False,\n",
       "  'contains(care)': False,\n",
       "  'contains(nice)': False,\n",
       "  'contains(couldnt)': False,\n",
       "  'contains(drama)': False,\n",
       "  'contains(change)': False,\n",
       "  'contains(feature)': False,\n",
       "  'contains(style)': False,\n",
       "  'contains(remember)': False,\n",
       "  'contains(completely)': False,\n",
       "  'contains(id)': False,\n",
       "  'contains(case)': True,\n",
       "  'contains(picture)': False,\n",
       "  'contains(entertain)': False,\n",
       "  'contains(along)': False,\n",
       "  'contains(rest)': False,\n",
       "  'contains(men)': False,\n",
       "  'contains(sort)': False,\n",
       "  'contains(lack)': False,\n",
       "  'contains(open)': False,\n",
       "  'contains(awful)': False,\n",
       "  'contains(everyone)': False,\n",
       "  'contains(comment)': False,\n",
       "  'contains(school)': False,\n",
       "  'contains(perhaps)': False,\n",
       "  'contains(release)': False,\n",
       "  'contains(add)': False,\n",
       "  'contains(truly)': False,\n",
       "  'contains(art)': False,\n",
       "  'contains(small)': False,\n",
       "  'contains(dialogue)': False,\n",
       "  'contains(piece)': True,\n",
       "  'contains(age)': False,\n",
       "  'contains(finally)': False,\n",
       "  'contains(song)': False,\n",
       "  'contains(human)': False,\n",
       "  'contains(felt)': False,\n",
       "  'contains(killer)': False,\n",
       "  'contains(low)': False,\n",
       "  'contains(close)': False,\n",
       "  'contains(wonderful)': False,\n",
       "  'contains(full)': False,\n",
       "  'contains(mother)': False,\n",
       "  'contains(wait)': False,\n",
       "  'contains(light)': False,\n",
       "  'contains(review)': False,\n",
       "  'contains(\\x96)': False,\n",
       "  'contains(early)': True,\n",
       "  'contains(video)': False,\n",
       "  'contains(joke)': False,\n",
       "  'contains(hope)': False,\n",
       "  'contains(consider)': False,\n",
       "  'contains(save)': False,\n",
       "  'contains(sit)': True,\n",
       "  'contains(fail)': False,\n",
       "  'contains(title)': False,\n",
       "  'contains(side)': False,\n",
       "  'contains(brother)': False,\n",
       "  'contains(terrible)': True,\n",
       "  'contains(others)': False,\n",
       "  'contains(stop)': False,\n",
       "  'contains(god)': False,\n",
       "  'contains(worse)': False,\n",
       "  'contains(break)': False,\n",
       "  'contains(example)': False,\n",
       "  'contains(mention)': False,\n",
       "  'contains(wonder)': False,\n",
       "  'contains(ask)': False,\n",
       "  'contains(stand)': False,\n",
       "  'contains(hit)': False,\n",
       "  'contains(cut)': False,\n",
       "  'contains(next)': False,\n",
       "  'contains(yes)': False,\n",
       "  'contains(voice)': False,\n",
       "  'contains(buy)': False,\n",
       "  'contains(often)': False,\n",
       "  'contains(daughter)': False,\n",
       "  'contains(sequence)': False,\n",
       "  'contains(city)': False,\n",
       "  'contains(lady)': False,\n",
       "  'contains(create)': False,\n",
       "  'contains(absolutely)': True,\n",
       "  'contains(type)': False,\n",
       "  'contains(totally)': False,\n",
       "  'contains(number)': False,\n",
       "  'contains(power)': False,\n",
       "  'contains(certainly)': False,\n",
       "  'contains(dark)': False,\n",
       "  'contains(twist)': False,\n",
       "  'contains(theyre)': False,\n",
       "  'contains(town)': False,\n",
       "  'contains(experience)': False,\n",
       "  'contains(wont)': False,\n",
       "  'contains(overall)': False,\n",
       "  'contains(evil)': False,\n",
       "  'contains(definitely)': False,\n",
       "  'contains(cinema)': False,\n",
       "  'contains(sex)': False,\n",
       "  'contains(disappoint)': True,\n",
       "  'contains(speak)': False,\n",
       "  'contains(guess)': False,\n",
       "  'contains(flick)': False,\n",
       "  'contains(favorite)': False,\n",
       "  'contains(relationship)': False,\n",
       "  'contains(fine)': False,\n",
       "  'contains(white)': False,\n",
       "  'contains(entire)': False,\n",
       "  'contains(already)': False,\n",
       "  'contains(catch)': False,\n",
       "  'contains(throw)': False,\n",
       "  'contains(portray)': False,\n",
       "  'contains(history)': True,\n",
       "  'contains(ill)': False,\n",
       "  'contains(particularly)': False,\n",
       "  'contains(stay)': False,\n",
       "  'contains(present)': False,\n",
       "  'contains(throughout)': False,\n",
       "  'contains(person)': False,\n",
       "  'contains(oh)': False,\n",
       "  'contains(hold)': False,\n",
       "  'contains(direction)': False,\n",
       "  'contains(slow)': False,\n",
       "  'contains(shes)': False,\n",
       "  'contains(matter)': False,\n",
       "  'contains(wish)': False,\n",
       "  'contains(etc)': False,\n",
       "  'contains(event)': False,\n",
       "  'contains(stuff)': False,\n",
       "  'contains(deal)': False,\n",
       "  'contains(unfortunately)': False,\n",
       "  'contains(sequel)': False,\n",
       "  'contains(humor)': False,\n",
       "  'contains(anyway)': False,\n",
       "  'contains(element)': False,\n",
       "  'contains(force)': False,\n",
       "  'contains(score)': False,\n",
       "  'contains(support)': False,\n",
       "  'contains(dream)': False,\n",
       "  'contains(quality)': False,\n",
       "  'contains(heart)': False,\n",
       "  'contains(walk)': False,\n",
       "  'contains(behind)': False,\n",
       "  'contains(despite)': False,\n",
       "  'contains(theme)': False,\n",
       "  'contains(genre)': False,\n",
       "  'contains(pick)': False,\n",
       "  'contains(drive)': False,\n",
       "  'contains(deserve)': False,\n",
       "  'contains(spend)': False,\n",
       "  'contains(annoy)': False,\n",
       "  'contains(chance)': False,\n",
       "  'contains(pay)': False,\n",
       "  'contains(novel)': False,\n",
       "  'contains(highly)': False,\n",
       "  'contains(able)': False,\n",
       "  'contains(late)': False,\n",
       "  'contains(robert)': False,\n",
       "  'contains(return)': False,\n",
       "  'contains(amaze)': False,\n",
       "  'contains(today)': False,\n",
       "  'contains(past)': False,\n",
       "  'contains(blood)': False,\n",
       "  'contains(b)': False,\n",
       "  'contains(forget)': False,\n",
       "  'contains(manage)': False,\n",
       "  'contains(youll)': False,\n",
       "  'contains(build)': False,\n",
       "  'contains(please)': False,\n",
       "  'contains(hero)': False,\n",
       "  'contains(soon)': False,\n",
       "  'contains(decent)': False,\n",
       "  'contains(king)': False,\n",
       "  'contains(brilliant)': False,\n",
       "  'contains(dance)': False,\n",
       "  'contains(son)': False,\n",
       "  'contains(realize)': False,\n",
       "  'contains(draw)': False,\n",
       "  'contains(police)': False,\n",
       "  'contains(michael)': True,\n",
       "  'contains(figure)': False,\n",
       "  'contains(car)': False,\n",
       "  'contains(game)': False,\n",
       "  'contains(final)': False,\n",
       "  'contains(obvious)': False,\n",
       "  'contains(jam)': False,\n",
       "  'contains(several)': False,\n",
       "  'contains(rent)': False,\n",
       "  'contains(level)': False,\n",
       "  'contains(mystery)': False,\n",
       "  'contains(credit)': False,\n",
       "  'contains(body)': False,\n",
       "  'contains(ok)': False,\n",
       "  'contains(hell)': False,\n",
       "  'contains(except)': False,\n",
       "  'contains(opinion)': False,\n",
       "  'contains(learn)': False,\n",
       "  'contains(david)': False,\n",
       "  'contains(musical)': False,\n",
       "  'contains(comic)': False,\n",
       "  'contains(stick)': False,\n",
       "  'contains(animal)': False,\n",
       "  'contains(edit)': False,\n",
       "  'contains(female)': False,\n",
       "  'contains(sometimes)': False,\n",
       "  'contains(tale)': False,\n",
       "  'contains(strong)': False,\n",
       "  'contains(alone)': False,\n",
       "  'contains(talent)': False,\n",
       "  'contains(question)': False,\n",
       "  'contains(group)': False,\n",
       "  'contains(documentary)': False,\n",
       "  'contains(possible)': False,\n",
       "  'contains(result)': False,\n",
       "  'contains(pace)': False,\n",
       "  'contains(somewhat)': False,\n",
       "  'contains(touch)': False,\n",
       "  'contains(offer)': False,\n",
       "  'contains(convince)': False,\n",
       "  'contains(arent)': False,\n",
       "  'contains(english)': False,\n",
       "  'contains(explain)': False,\n",
       "  'contains(single)': False,\n",
       "  'contains(country)': False,\n",
       "  'contains(jane)': False,\n",
       "  'contains(whose)': False,\n",
       "  'contains(horrible)': False,\n",
       "  'contains(husband)': False,\n",
       "  'contains(middle)': False,\n",
       "  'contains(thriller)': False,\n",
       "  'contains(complete)': False,\n",
       "  'contains(ago)': False,\n",
       "  'contains(scary)': False,\n",
       "  'contains(oscar)': False,\n",
       "  'contains(check)': False,\n",
       "  'contains(simple)': False,\n",
       "  'contains(compare)': False,\n",
       "  'contains(air)': False,\n",
       "  'contains(state)': False,\n",
       "  'contains(room)': False,\n",
       "  'contains(seriously)': False,\n",
       "  'contains(hop)': False,\n",
       "  'contains(fit)': False,\n",
       "  'contains(wouldnt)': False,\n",
       "  'contains(red)': False,\n",
       "  'contains(dialog)': False,\n",
       "  'contains(across)': False,\n",
       "  'contains(crap)': False,\n",
       "  'contains(message)': False,\n",
       "  'contains(street)': False,\n",
       "  'contains(serious)': False,\n",
       "  'contains(usually)': False,\n",
       "  'contains(york)': False,\n",
       "  'contains(mostly)': False,\n",
       "  'contains(sister)': False,\n",
       "  'contains(marry)': False,\n",
       "  'contains(sad)': False,\n",
       "  'contains(rock)': False,\n",
       "  'contains(grow)': False,\n",
       "  'contains(value)': False,\n",
       "  'contains(deliver)': False,\n",
       "  'contains(spoiler)': False,\n",
       "  'contains(local)': False,\n",
       "  'contains(order)': False,\n",
       "  'contains(situation)': False,\n",
       "  'contains(zombie)': False,\n",
       "  'contains(self)': False,\n",
       "  'contains(imagine)': False,\n",
       "  'contains(lee)': False,\n",
       "  'contains(filmmaker)': False,\n",
       "  'contains(gore)': False,\n",
       "  'contains(happy)': False,\n",
       "  'contains(effort)': False,\n",
       "  'contains(finish)': False,\n",
       "  'contains(cool)': False,\n",
       "  'contains(cop)': False,\n",
       "  'contains(obviously)': False,\n",
       "  'contains(baby)': False,\n",
       "  'contains(basically)': False,\n",
       "  'contains(british)': False,\n",
       "  'contains(pas)': False,\n",
       "  'contains(important)': False,\n",
       "  'contains(thank)': False,\n",
       "  'contains(pull)': False,\n",
       "  'contains(class)': False,\n",
       "  'contains(prove)': False,\n",
       "  'contains(extremely)': False,\n",
       "  'contains(richard)': False,\n",
       "  'contains(avoid)': False,\n",
       "  'contains(due)': False,\n",
       "  'contains(detail)': False,\n",
       "  'contains(hilarious)': False,\n",
       "  'contains(note)': False,\n",
       "  'contains(none)': False,\n",
       "  'contains(predictable)': False,\n",
       "  'contains(theater)': False,\n",
       "  'contains(shot)': False,\n",
       "  'contains(provide)': False,\n",
       "  'contains(huge)': False,\n",
       "  'contains(straight)': False,\n",
       "  'contains(ten)': False,\n",
       "  'contains(dull)': False,\n",
       "  'contains(career)': False,\n",
       "  'contains(lame)': False,\n",
       "  'contains(jack)': False,\n",
       "  'contains(reality)': False,\n",
       "  'contains(monster)': False,\n",
       "  'contains(western)': False,\n",
       "  'contains(allow)': False,\n",
       "  'contains(screenplay)': False,\n",
       "  'contains(storyline)': True,\n",
       "  'contains(suffer)': False,\n",
       "  'contains(near)': False,\n",
       "  'contains(cartoon)': False,\n",
       "  'contains(clearly)': False,\n",
       "  'contains(entertainment)': False,\n",
       "  'contains(peter)': False,\n",
       "  'contains(warn)': False,\n",
       "  'contains(remind)': False,\n",
       "  'contains(charm)': False,\n",
       "  'contains(lover)': False,\n",
       "  'contains(romantic)': False,\n",
       "  'contains(suck)': False,\n",
       "  'contains(attention)': False,\n",
       "  'contains(television)': False,\n",
       "  'contains(date)': False,\n",
       "  'contains(usual)': False,\n",
       "  'contains(ridiculous)': True,\n",
       "  'contains(parent)': False,\n",
       "  'contains(bear)': False,\n",
       "  'contains(choose)': False,\n",
       "  'contains(attack)': False,\n",
       "  'contains(season)': False,\n",
       "  'contains(fast)': False,\n",
       "  'contains(fire)': False,\n",
       "  'contains(member)': False,\n",
       "  'contains(violence)': False,\n",
       "  'contains(similar)': False,\n",
       "  'contains(beyond)': False,\n",
       "  'contains(havent)': False,\n",
       "  'contains(win)': False,\n",
       "  'contains(apparently)': False,\n",
       "  'contains(cause)': False,\n",
       "  'contains(cry)': False,\n",
       "  'contains(japanese)': False,\n",
       "  'contains(animation)': False,\n",
       "  'contains(whether)': False,\n",
       "  'contains(sing)': False,\n",
       "  'contains(agree)': False,\n",
       "  'contains(future)': False,\n",
       "  'contains(doubt)': False,\n",
       "  'contains(chase)': False,\n",
       "  'contains(continue)': False,\n",
       "  'contains(exactly)': False,\n",
       "  'contains(strange)': False,\n",
       "  'contains(drug)': False,\n",
       "  'contains(whats)': False,\n",
       "  'contains(possibly)': False,\n",
       "  'contains(typical)': False,\n",
       "  'contains(focus)': False,\n",
       "  'contains(third)': False,\n",
       "  'contains(non)': False,\n",
       "  'contains(discover)': False,\n",
       "  'contains(realistic)': False,\n",
       "  'contains(cinematography)': False,\n",
       "  'contains(somehow)': False,\n",
       "  'contains(sorry)': False,\n",
       "  'contains(excite)': False,\n",
       "  'contains(who)': False,\n",
       "  'contains(indeed)': False,\n",
       "  'contains(earth)': False,\n",
       "  'contains(major)': False,\n",
       "  'contains(nature)': False,\n",
       "  'contains(producer)': False,\n",
       "  'contains(modern)': False,\n",
       "  'contains(atmosphere)': False,\n",
       "  'contains(bunch)': False,\n",
       "  'contains(aspect)': False,\n",
       "  'contains(villain)': False,\n",
       "  'contains(control)': False,\n",
       "  'contains(steal)': False,\n",
       "  'contains(material)': False,\n",
       "  'contains(image)': False,\n",
       "  'contains(produce)': False,\n",
       "  'contains(nearly)': False,\n",
       "  'contains(escape)': False,\n",
       "  'contains(box)': False,\n",
       "  'contains(wear)': False,\n",
       "  'contains(de)': False,\n",
       "  'contains(emotion)': False,\n",
       "  'contains(christmas)': False,\n",
       "  'contains(standard)': False,\n",
       "  'contains(victim)': False,\n",
       "  'contains(fantastic)': False,\n",
       "  'contains(develop)': False,\n",
       "  'contains(hate)': False,\n",
       "  'contains(easily)': False,\n",
       "  'contains(imdb)': False,\n",
       "  'contains(student)': False,\n",
       "  'contains(cat)': False,\n",
       "  'contains(enjoyable)': False,\n",
       "  'contains(choice)': False,\n",
       "  'contains(band)': False,\n",
       "  'contains(adult)': False,\n",
       "  'contains(accent)': False,\n",
       "  'contains(cheap)': False,\n",
       "  'contains(team)': False,\n",
       "  'contains(fantasy)': False,\n",
       "  'contains(sadly)': False,\n",
       "  'contains(period)': False,\n",
       "  'contains(easy)': False,\n",
       "  'contains(four)': False,\n",
       "  'contains(remain)': False,\n",
       "  'contains(emotional)': True,\n",
       "  'contains(within)': False,\n",
       "  'contains(george)': False,\n",
       "  'contains(shock)': False,\n",
       "  'contains(send)': False,\n",
       "  'contains(scream)': False,\n",
       "  'contains(five)': False,\n",
       "  'contains(form)': False,\n",
       "  'contains(clear)': False,\n",
       "  'contains(french)': False,\n",
       "  'contains(depth)': False,\n",
       "  'contains(fill)': False,\n",
       "  'contains(silly)': False,\n",
       "  'contains(mark)': False,\n",
       "  'contains(hot)': False,\n",
       "  'contains(describe)': False,\n",
       "  'contains(crime)': False,\n",
       "  'contains(suspense)': False,\n",
       "  'contains(doctor)': False,\n",
       "  'contains(battle)': False,\n",
       "  'contains(mistake)': False,\n",
       "  'contains(cover)': False,\n",
       "  'contains(difficult)': False,\n",
       "  'contains(fly)': False,\n",
       "  'contains(fear)': False,\n",
       "  'contains(upon)': False,\n",
       "  'contains(copy)': False,\n",
       "  'contains(beat)': False,\n",
       "  'contains(girlfriend)': False,\n",
       "  'contains(train)': False,\n",
       "  'contains(admit)': False,\n",
       "  'contains(stewart)': False,\n",
       "  'contains(capture)': False,\n",
       "  'contains(believable)': False,\n",
       "  'contains(weird)': False,\n",
       "  'contains(premise)': False,\n",
       "  'contains(tear)': False,\n",
       "  'contains(famous)': False,\n",
       "  'contains(mix)': False,\n",
       "  'contains(dramatic)': False,\n",
       "  'contains(concern)': False,\n",
       "  'contains(hat)': False,\n",
       "  'contains(space)': False,\n",
       "  'contains(jump)': False,\n",
       "  'contains(reveal)': False,\n",
       "  'contains(roll)': False,\n",
       "  'contains(plan)': False,\n",
       "  'contains(subject)': False,\n",
       "  'contains(confuse)': False,\n",
       "  'contains(among)': False,\n",
       "  'contains(hide)': False,\n",
       "  'contains(notice)': False,\n",
       "  'contains(trouble)': False,\n",
       "  'contains(gay)': False,\n",
       "  'contains(romance)': False,\n",
       "  'contains(forward)': False,\n",
       "  'contains(bill)': False,\n",
       "  'contains(society)': False,\n",
       "  'contains(free)': False,\n",
       "  'contains(perfectly)': False,\n",
       "  'contains(hair)': False,\n",
       "  'contains(amuse)': False,\n",
       "  'contains(blow)': False,\n",
       "  'contains(youve)': False,\n",
       "  'contains(bother)': False,\n",
       "  'contains(color)': False,\n",
       "  'contains(adventure)': False,\n",
       "  'contains(poorly)': False,\n",
       "  'contains(project)': False,\n",
       "  'contains(odd)': False,\n",
       "  'contains(social)': False,\n",
       "  'contains(issue)': False,\n",
       "  'contains(flaw)': False,\n",
       "  'contains(particular)': False,\n",
       "  'contains(appreciate)': False,\n",
       "  'contains(dress)': False,\n",
       "  'contains(remake)': False,\n",
       "  'contains(otherwise)': False,\n",
       "  'contains(shame)': False,\n",
       "  'contains(soldier)': False,\n",
       "  'contains(potential)': False,\n",
       "  'contains(fake)': False,\n",
       "  'contains(mary)': False,\n",
       "  'contains(eventually)': False,\n",
       "  'contains(adaptation)': False,\n",
       "  'contains(track)': False,\n",
       "  'contains(eat)': False,\n",
       "  'contains(award)': False,\n",
       "  'contains(spirit)': False,\n",
       "  'contains(stage)': False,\n",
       "  'contains(male)': False,\n",
       "  'contains(drink)': False,\n",
       "  'contains(paul)': False,\n",
       "  'contains(previous)': False,\n",
       "  'contains(answer)': False,\n",
       "  'contains(whatever)': False,\n",
       "  'contains(beauty)': False,\n",
       "  'contains(list)': False,\n",
       "  'contains(earlier)': False,\n",
       "  'contains(general)': False,\n",
       "  'contains(maker)': False,\n",
       "  'contains(reach)': False,\n",
       "  'contains(cute)': False,\n",
       "  'contains(carry)': False,\n",
       "  'contains(frank)': False,\n",
       "  'contains(purpose)': False,\n",
       "  'contains(design)': False,\n",
       "  'contains(party)': False,\n",
       "  'contains(wood)': False,\n",
       "  'contains(basic)': False,\n",
       "  'contains(kelly)': False,\n",
       "  'contains(burn)': False,\n",
       "  'contains(co)': False,\n",
       "  'contains(america)': False,\n",
       "  'contains(company)': False,\n",
       "  'contains(water)': False,\n",
       "  'contains(portrayal)': False,\n",
       "  'contains(badly)': False,\n",
       "  'contains(brain)': False,\n",
       "  'contains(creepy)': False,\n",
       "  'contains(tone)': False,\n",
       "  'contains(apart)': False,\n",
       "  'contains(ben)': False,\n",
       "  'contains(th)': False,\n",
       "  'contains(average)': False,\n",
       "  'contains(inspire)': False,\n",
       "  'contains(pathetic)': True,\n",
       "  'contains(sci)': False,\n",
       "  'contains(fi)': False,\n",
       "  'contains(honestly)': False,\n",
       "  'contains(greatest)': False,\n",
       "  'contains(treat)': False,\n",
       "  'contains(unless)': False,\n",
       "  'contains(strike)': False,\n",
       "  'contains(claim)': False,\n",
       "  'contains(contain)': False,\n",
       "  'contains(trash)': False,\n",
       "  'contains(gun)': False,\n",
       "  'contains(hole)': False,\n",
       "  'contains(werewolf)': False,\n",
       "  'contains(hardly)': False,\n",
       "  'contains(sweet)': False,\n",
       "  'contains(older)': False,\n",
       "  'contains(secret)': False,\n",
       "  'contains(store)': False,\n",
       "  'contains(weak)': False,\n",
       "  'contains(paint)': False,\n",
       "  'contains(science)': False,\n",
       "  'contains(tom)': False,\n",
       "  'contains(certain)': False,\n",
       "  'contains(master)': False,\n",
       "  'contains(week)': False,\n",
       "  'contains(batman)': False,\n",
       "  'contains(la)': False,\n",
       "  'contains(front)': False,\n",
       "  'contains(law)': False,\n",
       "  'contains(destroy)': False,\n",
       "  'contains(german)': False,\n",
       "  'contains(disney)': False,\n",
       "  'contains(soundtrack)': False,\n",
       "  'contains(background)': False,\n",
       "  'contains(footage)': False,\n",
       "  'contains(hurt)': False,\n",
       "  'contains(joe)': False,\n",
       "  'contains(crazy)': False,\n",
       "  'contains(conflict)': False,\n",
       "  'contains(business)': False,\n",
       "  'contains(truth)': False,\n",
       "  'contains(rise)': False,\n",
       "  'contains(soul)': False,\n",
       "  'contains(trip)': False,\n",
       "  'contains(shop)': False,\n",
       "  'contains(actual)': False,\n",
       "  'contains(million)': False,\n",
       "  'contains(cheesy)': False,\n",
       "  'contains(dog)': False,\n",
       "  'contains(rip)': False,\n",
       "  'contains(barely)': True,\n",
       "  'contains(tire)': False,\n",
       "  'contains(e)': False,\n",
       "  'contains(serve)': False,\n",
       "  'contains(sexual)': False,\n",
       "  'contains(dumb)': False,\n",
       "  'contains(scott)': False,\n",
       "  'contains(mess)': False,\n",
       "  'contains(perform)': False,\n",
       "  'contains(step)': False,\n",
       "  'contains(appeal)': False,\n",
       "  'contains(teen)': False,\n",
       "  'contains(creature)': False,\n",
       "  'contains(detective)': False,\n",
       "  'contains(mood)': False,\n",
       "  'contains(recently)': False,\n",
       "  'contains(total)': False,\n",
       "  'contains(suit)': False,\n",
       "  'contains(development)': False,\n",
       "  'contains(superb)': False,\n",
       "  'contains(lie)': False,\n",
       "  'contains(sign)': False,\n",
       "  'contains(haunt)': False,\n",
       "  'contains(location)': False,\n",
       "  'contains(genius)': False,\n",
       "  'contains(plain)': False,\n",
       "  'contains(nudity)': False,\n",
       "  'contains(powerful)': False,\n",
       "  'contains(crew)': False,\n",
       "  'contains(stun)': False,\n",
       "  'contains(buddy)': False,\n",
       "  'contains(torture)': False,\n",
       "  'contains(okay)': False,\n",
       "  'contains(camp)': False,\n",
       "  'contains(rich)': False,\n",
       "  'contains(familiar)': False,\n",
       "  'contains(memorable)': False,\n",
       "  'contains(f)': False,\n",
       "  'contains(pop)': False,\n",
       "  'contains(marriage)': False,\n",
       "  'contains(critic)': False,\n",
       "  'contains(intrigue)': False,\n",
       "  'contains(suggest)': False,\n",
       "  'contains(neither)': False,\n",
       "  'contains(political)': False,\n",
       "  'contains(popular)': False,\n",
       "  'contains(c)': False,\n",
       "  'contains(slasher)': False,\n",
       "  'contains(unbelievable)': False,\n",
       "  'contains(plenty)': False,\n",
       "  'contains(exist)': False,\n",
       "  'contains(extra)': False,\n",
       "  'contains(month)': False,\n",
       "  'contains(sleep)': False,\n",
       "  'contains(disturb)': False,\n",
       "  'contains(kick)': False,\n",
       "  'contains(towards)': False,\n",
       "  'contains(office)': False,\n",
       "  'contains(deep)': False,\n",
       "  'contains(outside)': False,\n",
       "  'contains(london)': False,\n",
       "  'contains(channel)': False,\n",
       "  'contains(fiction)': False,\n",
       "  'contains(wind)': False,\n",
       "  'contains(van)': False,\n",
       "  'contains(costume)': False,\n",
       "  'contains(violent)': False,\n",
       "  'contains(ultimately)': False,\n",
       "  'contains(studio)': False,\n",
       "  'contains(flat)': False,\n",
       "  'contains(drag)': False,\n",
       "  'contains(thrill)': False,\n",
       "  'contains(memory)': False,\n",
       "  'contains(island)': False,\n",
       "  'contains(travel)': False,\n",
       "  'contains(incredibly)': False,\n",
       "  'contains(unique)': False,\n",
       "  'contains(entirely)': False,\n",
       "  'contains(utterly)': False,\n",
       "  'contains(ruin)': True,\n",
       "  'contains(folk)': False,\n",
       "  'contains(rate)': False,\n",
       "  'contains(positive)': False,\n",
       "  'contains(regard)': False,\n",
       "  'contains(blue)': False,\n",
       "  'contains(harry)': False,\n",
       "  'contains(fair)': False,\n",
       "  'contains(bottom)': False,\n",
       "  'contains(personal)': False,\n",
       "  'contains(struggle)': False,\n",
       "  'contains(judge)': False,\n",
       "  'contains(culture)': False,\n",
       "  'contains(grant)': False,\n",
       "  'contains(cold)': False,\n",
       "  'contains(tension)': False,\n",
       "  'contains(fashion)': False,\n",
       "  'contains(spot)': False,\n",
       "  'contains(count)': False,\n",
       "  'contains(taste)': False,\n",
       "  'contains(sell)': False,\n",
       "  'contains(heavy)': False,\n",
       "  'contains(artist)': False,\n",
       "  'contains(quickly)': False,\n",
       "  'contains(pure)': False,\n",
       "  'contains(reviewer)': False,\n",
       "  'contains(amount)': False,\n",
       "  'contains(dub)': False,\n",
       "  'contains(handle)': False,\n",
       "  'contains(accept)': False,\n",
       "  'contains(unlike)': False,\n",
       "  'contains(super)': False,\n",
       "  'contains(machine)': False,\n",
       "  'contains(introduce)': False,\n",
       "  'contains(suddenly)': False,\n",
       "  'contains(computer)': False,\n",
       "  'contains(ability)': False,\n",
       "  'contains(expression)': False,\n",
       "  'contains(smart)': False,\n",
       "  'contains(various)': False,\n",
       "  'contains(indian)': False,\n",
       "  'contains(army)': False,\n",
       "  'contains(land)': False,\n",
       "  'contains(william)': False,\n",
       "  'contains(masterpiece)': False,\n",
       "  'contains(tragic)': False,\n",
       "  'contains(suspect)': False,\n",
       "  'contains(conclusion)': False,\n",
       "  'contains(boyfriend)': False,\n",
       "  'contains(superman)': False,\n",
       "  'contains(mad)': False,\n",
       "  'contains(player)': False,\n",
       "  'contains(public)': False,\n",
       "  'contains(younger)': False,\n",
       "  'contains(dad)': False,\n",
       "  'contains(expectation)': False,\n",
       "  'contains(slightly)': False,\n",
       "  'contains(werent)': False,\n",
       "  'contains(key)': False,\n",
       "  ...},\n",
       " 'negative')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inspect first train featureset\n",
    "train_set[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Train & evaluate model (naive bayes classifier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train a naive bayes classifier with train set by nltk\n",
    "classifier = nltk.NaiveBayesClassifier.train(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8332"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the accuracy of the naive bayes classifier with test set\n",
    "accuracy = nltk.classify.accuracy(classifier, test_set)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build reference and test set of observed values (for each label)\n",
    "refsets = collections.defaultdict(set)\n",
    "testsets = collections.defaultdict(set)\n",
    " \n",
    "for i, (feats, label) in enumerate(train_set):\n",
    "    refsets[label].add(i)\n",
    "    observed = classifier.classify(feats)\n",
    "    testsets[observed].add(i)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pos precision: 0.8739232576350823\n",
      "pos recall: 0.9036437246963562\n",
      "pos F-measure: 0.8885350318471338\n",
      "neg precision: 0.902698282910875\n",
      "neg recall: 0.8727272727272727\n",
      "neg F-measure: 0.8874598070739551\n"
     ]
    }
   ],
   "source": [
    "# print precision, recall, and f-measure\n",
    "print('pos precision:', precision(refsets['positive'], testsets['positive']))\n",
    "print('pos recall:', recall(refsets['positive'], testsets['positive']))\n",
    "print('pos F-measure:', f_measure(refsets['positive'], testsets['positive']))\n",
    "print('neg precision:', precision(refsets['negative'], testsets['negative']))\n",
    "print('neg recall:', recall(refsets['negative'], testsets['negative']))\n",
    "print('neg F-measure:', f_measure(refsets['negative'], testsets['negative']))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "     contains(underrate) = True           positi : negati =     16.7 : 1.0\n",
      "    contains(ridiculous) = True           negati : positi =     16.4 : 1.0\n",
      "       contains(unfunny) = True           negati : positi =     13.3 : 1.0\n",
      "          contains(hong) = True           positi : negati =      9.9 : 1.0\n",
      "        contains(unfold) = True           positi : negati =      9.6 : 1.0\n",
      "   contains(wonderfully) = True           positi : negati =      8.6 : 1.0\n",
      "         contains(appal) = True           negati : positi =      8.4 : 1.0\n",
      "          contains(lame) = True           negati : positi =      8.0 : 1.0\n",
      "          contains(dumb) = True           negati : positi =      7.9 : 1.0\n",
      "         contains(awful) = True           negati : positi =      7.8 : 1.0\n"
     ]
    }
   ],
   "source": [
    "# show top n most informative features\n",
    "classifier.show_most_informative_features(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Make prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict on new review (from mubi.com)\n",
    "new_review = \"Surprisingly effective and moving, The Balcony Movie takes the Front Up\\\n",
    "            concept of talking to strangers, but here attaches it to a fixed perspective \\\n",
    "            in order to create a strong sense of the stream of life passing us by. \\\n",
    "            It's possible to not only witness the subtle changing of seasons\\\n",
    "            but also the gradual opening of trust and confidence in Lozinski's \\\n",
    "            repeating characters. A Pandemic movie, pre-pandemic. 3.5 stars\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform preprocessing (cleaning & featureset transformation)\n",
    "processed_review = rm_custom_stops(preprocessing(new_review))\n",
    "processed_review = review_features(processed_review, most_common_2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'positive'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# predict label\n",
    "classifier.classify(processed_review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "negative:\n",
      "   contains(underrate)(True): 0.12%\n",
      "   contains(ridiculous)(True): 5.96%\n",
      "   contains(unfunny)(True): 1.62%\n",
      "   contains(hong)(True): 0.12%\n",
      "   contains(unfold)(True): 0.20%\n",
      "   contains(wonderfully)(True): 0.28%\n",
      "   contains(appal)(True): 1.70%\n",
      "   contains(lame)(True): 5.49%\n",
      "   contains(dumb)(True): 3.52%\n",
      "   contains(awful)(True): 10.39%\n",
      "   contains(waste)(True): 12.68%\n",
      "   contains(laughable)(True): 2.65%\n",
      "   contains(worst)(True): 14.89%\n",
      "   contains(rubbish)(True): 2.01%\n",
      "   contains(ninja)(True): 0.83%\n",
      "   contains(gon)(True): 1.94%\n",
      "   contains(pointless)(True): 3.04%\n",
      "   contains(redeem)(True): 3.04%\n",
      "   contains(magnificent)(True): 0.28%\n",
      "   contains(giallo)(True): 0.12%\n",
      "   contains(asleep)(True): 1.78%\n",
      "   contains(subtle)(True): 0.43%\n",
      "   contains(horrible)(True): 6.04%\n",
      "   contains(kong)(True): 0.20%\n",
      "   contains(crap)(True): 5.96%\n",
      "   contains(garbage)(True): 2.57%\n",
      "   contains(horribly)(True): 2.09%\n",
      "   contains(wooden)(True): 2.09%\n",
      "   contains(gritty)(True): 0.28%\n",
      "   contains(werewolf)(True): 1.15%\n",
      "   contains(absurd)(True): 2.49%\n",
      "   contains(prince)(True): 0.20%\n",
      "   contains(insult)(True): 2.88%\n",
      "   contains(pile)(True): 1.54%\n",
      "   contains(annoy)(True): 7.54%\n",
      "   contains(poorly)(True): 4.07%\n",
      "   contains(suck)(True): 4.46%\n",
      "   contains(finest)(True): 0.36%\n",
      "   contains(gorgeous)(True): 0.43%\n",
      "   contains(dangerous)(True): 0.36%\n",
      "   contains(trial)(True): 0.36%\n",
      "   contains(idiot)(True): 2.17%\n",
      "   contains(crappy)(True): 1.38%\n",
      "   contains(desperate)(True): 1.78%\n",
      "   contains(terrible)(True): 8.73%\n",
      "   contains(stewart)(True): 0.43%\n",
      "   contains(jackie)(True): 0.20%\n",
      "   contains(epic)(True): 0.36%\n",
      "   contains(stupid)(True): 9.60%\n",
      "   contains(embarrass)(True): 2.80%\n",
      "\n",
      "\n",
      "positive:\n",
      "   contains(underrate)(True): 1.98%\n",
      "   contains(ridiculous)(True): 0.36%\n",
      "   contains(unfunny)(True): 0.12%\n",
      "   contains(hong)(True): 1.17%\n",
      "   contains(unfold)(True): 1.90%\n",
      "   contains(wonderfully)(True): 2.39%\n",
      "   contains(appal)(True): 0.20%\n",
      "   contains(lame)(True): 0.69%\n",
      "   contains(dumb)(True): 0.44%\n",
      "   contains(awful)(True): 1.33%\n",
      "   contains(waste)(True): 1.74%\n",
      "   contains(laughable)(True): 0.36%\n",
      "   contains(worst)(True): 2.06%\n",
      "   contains(rubbish)(True): 0.28%\n",
      "   contains(ninja)(True): 0.12%\n",
      "   contains(gon)(True): 0.28%\n",
      "   contains(pointless)(True): 0.44%\n",
      "   contains(redeem)(True): 0.44%\n",
      "   contains(magnificent)(True): 1.82%\n",
      "   contains(giallo)(True): 0.77%\n",
      "   contains(asleep)(True): 0.28%\n",
      "   contains(subtle)(True): 2.63%\n",
      "   contains(horrible)(True): 1.01%\n",
      "   contains(kong)(True): 1.17%\n",
      "   contains(crap)(True): 1.01%\n",
      "   contains(garbage)(True): 0.44%\n",
      "   contains(horribly)(True): 0.36%\n",
      "   contains(wooden)(True): 0.36%\n",
      "   contains(gritty)(True): 1.58%\n",
      "   contains(werewolf)(True): 0.20%\n",
      "   contains(absurd)(True): 0.44%\n",
      "   contains(prince)(True): 1.09%\n",
      "   contains(insult)(True): 0.53%\n",
      "   contains(pile)(True): 0.28%\n",
      "   contains(annoy)(True): 1.42%\n",
      "   contains(poorly)(True): 0.77%\n",
      "   contains(suck)(True): 0.85%\n",
      "   contains(finest)(True): 1.82%\n",
      "   contains(gorgeous)(True): 2.14%\n",
      "   contains(dangerous)(True): 1.74%\n",
      "   contains(trial)(True): 1.74%\n",
      "   contains(idiot)(True): 0.44%\n",
      "   contains(crappy)(True): 0.28%\n",
      "   contains(desperate)(True): 0.36%\n",
      "   contains(terrible)(True): 1.82%\n",
      "   contains(stewart)(True): 2.06%\n",
      "   contains(jackie)(True): 0.93%\n",
      "   contains(epic)(True): 1.66%\n",
      "   contains(stupid)(True): 2.06%\n",
      "   contains(embarrass)(True): 0.61%\n"
     ]
    }
   ],
   "source": [
    "# to get individual probability for each label and word, taken from:\n",
    "# https://stackoverflow.com/questions/20773200/python-nltk-naive-bayes-probabilities\n",
    "for label in classifier.labels():\n",
    "  print(f'\\n\\n{label}:')\n",
    "  for (fname, fval) in classifier.most_informative_features(50):\n",
    "    print(f\"   {fname}({fval}): \", end=\"\")\n",
    "    print(\"{0:.2f}%\".format(100*classifier._feature_probdist[label, fname].prob(fval)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 64-bit ('nlpreviews')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "ecbb573cc41fe6baf34354607df718ebf4ab136e8601a96e7a6da1a1db011e8e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}