5. Neural BOW Model for Sentiments..ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Develop a Neural Bag-of-Words Model for Sentiment Analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Movie Review Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the process and dataset as described in this notebook: http://localhost:8888/notebooks/Documents/Programming%23/AMMI%20Basics/Natural%20Language%20Processing%20Keras/4.%20Sentiment%20Analysis%20Data%20Preparation..ipynb.\n",
    "\n",
    "Note, all processes in the above link can be run from the terminal using the main.py script. Though one may have to download the file manually and run the process of unzipping and extracting the file. We'll fix this in the script later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1. Split data into training and testing sets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll split the data into 90% for train and 10% for test. We can do this without shuffling the data as there is no time factor in the collection of our data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2. Loading and cleaning data to remove punctuation and numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luckily we already have functions designed to clean our data, we can load them and use to:\n",
    "    \n",
    "    1. Split tokens on whitespace.\n",
    "    2. Remove all punctuation from words.\n",
    "    3. Remove all words that are not purely comprised of alphabetical characters.\n",
    "    4. Remove all words that are known stop words.\n",
    "    5. Remove all words that have a length <= 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# Reuse our saved modules.\n",
    "from text_prep import *\n",
    "from save_processed import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable            Type                Data/Info\n",
      "-------------------------------------------------\n",
      "Tokenizer           type                <class 'keras_preprocessing.text.Tokenizer'>\n",
      "add_doc_to_vocab    function            <function add_doc_to_vocab at 0x7f4a497029d8>\n",
      "clean_doc           function            <function clean_doc at 0x7f4abce541e0>\n",
      "create_tokenizer    function            <function create_tokenizer at 0x7f4a49702bf8>\n",
      "doc_to_line         function            <function doc_to_line at 0x7f4a49702c80>\n",
      "load_doc            function            <function load_doc at 0x7f4abce34f28>\n",
      "os                  module              <module 'os' from '/usr/lib/python3.7/os.py'>\n",
      "process_docs        function            <function process_docs at 0x7f4a49702a60>\n",
      "process_docs_full   function            <function process_docs_full at 0x7f4a49702d08>\n",
      "re                  module              <module 're' from '/usr/lib/python3.7/re.py'>\n",
      "remove_min          function            <function remove_min at 0x7f4a49702ae8>\n",
      "save_list           function            <function save_list at 0x7f4a49702b70>\n",
      "stopwords           LazyCorpusLoader    <WordListCorpusReader in <...>pwords' (not loaded yet)>\n",
      "string              module              <module 'string' from '/u<...>lib/python3.7/string.py'>\n"
     ]
    }
   ],
   "source": [
    "%whos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['films', 'adapted', 'comic', 'books', 'plenty', 'success', 'whether', 'theyre', 'superheroes', 'batman', 'superman', 'spawn', 'geared', 'toward', 'kids', 'casper', 'arthouse', 'crowd', 'ghost', 'world', 'theres', 'never', 'really', 'comic', 'book', 'like', 'hell', 'starters', 'created', 'alan', 'moore', 'eddie', 'campbell', 'brought', 'medium', 'whole', 'new', 'level', 'mid', 'series', 'called', 'watchmen', 'say', 'moore', 'campbell', 'thoroughly', 'researched', 'subject', 'jack', 'ripper', 'would', 'like', 'saying', 'michael', 'jackson', 'starting', 'look', 'little', 'odd', 'book', 'graphic', 'novel', 'pages', 'long', 'includes', 'nearly', 'consist', 'nothing', 'footnotes', 'words', 'dont', 'dismiss', 'film', 'source', 'get', 'past', 'whole', 'comic', 'book', 'thing', 'might', 'find', 'another', 'stumbling', 'block', 'hells', 'directors', 'albert', 'allen', 'hughes', 'getting', 'hughes', 'brothers', 'direct', 'seems', 'almost', 'ludicrous', 'casting', 'carrot', 'top', 'well', 'anything', 'riddle', 'better', 'direct', 'film', 'thats', 'set', 'ghetto', 'features', 'really', 'violent', 'street', 'crime', 'mad', 'geniuses', 'behind', 'menace', 'ii', 'society', 'ghetto', 'question', 'course', 'whitechapel', 'londons', 'east', 'end', 'filthy', 'sooty', 'place', 'whores', 'called', 'unfortunates', 'starting', 'get', 'little', 'nervous', 'mysterious', 'psychopath', 'carving', 'profession', 'surgical', 'precision', 'first', 'stiff', 'turns', 'copper', 'peter', 'godley', 'robbie', 'coltrane', 'world', 'enough', 'calls', 'inspector', 'frederick', 'abberline', 'johnny', 'depp', 'blow', 'crack', 'case', 'abberline', 'widower', 'prophetic', 'dreams', 'unsuccessfully', 'tries', 'quell', 'copious', 'amounts', 'absinthe', 'opium', 'upon', 'arriving', 'whitechapel', 'befriends', 'unfortunate', 'named', 'mary', 'kelly', 'heather', 'graham', 'say', 'isnt', 'proceeds', 'investigate', 'horribly', 'gruesome', 'crimes', 'even', 'police', 'surgeon', 'cant', 'stomach', 'dont', 'think', 'anyone', 'needs', 'briefed', 'jack', 'ripper', 'wont', 'go', 'particulars', 'say', 'moore', 'campbell', 'unique', 'interesting', 'theory', 'identity', 'killer', 'reasons', 'chooses', 'slay', 'comic', 'dont', 'bother', 'cloaking', 'identity', 'ripper', 'screenwriters', 'terry', 'hayes', 'vertical', 'limit', 'rafael', 'yglesias', 'les', 'mis', 'rables', 'good', 'job', 'keeping', 'hidden', 'viewers', 'end', 'funny', 'watch', 'locals', 'blindly', 'point', 'finger', 'blame', 'jews', 'indians', 'englishman', 'could', 'never', 'capable', 'committing', 'ghastly', 'acts', 'hells', 'ending', 'whistling', 'stonecutters', 'song', 'simpsons', 'days', 'holds', 'back', 'electric', 'carwho', 'made', 'steve', 'guttenberg', 'star', 'dont', 'worry', 'itll', 'make', 'sense', 'see', 'onto', 'hells', 'appearance', 'certainly', 'dark', 'bleak', 'enough', 'surprising', 'see', 'much', 'looks', 'like', 'tim', 'burton', 'film', 'planet', 'apes', 'times', 'seems', 'like', 'sleepy', 'hollow', 'print', 'saw', 'wasnt', 'completely', 'finished', 'color', 'music', 'finalized', 'comments', 'marilyn', 'manson', 'cinematographer', 'peter', 'deming', 'dont', 'say', 'word', 'ably', 'captures', 'dreariness', 'victorianera', 'london', 'helped', 'make', 'flashy', 'killing', 'scenes', 'remind', 'crazy', 'flashbacks', 'twin', 'peaks', 'even', 'though', 'violence', 'film', 'pales', 'comparison', 'blackandwhite', 'comic', 'oscar', 'winner', 'martin', 'childs', 'shakespeare', 'love', 'production', 'design', 'turns', 'original', 'prague', 'surroundings', 'one', 'creepy', 'place', 'even', 'acting', 'hell', 'solid', 'dreamy', 'depp', 'turning', 'typically', 'strong', 'performance', 'deftly', 'handling', 'british', 'accent', 'ians', 'holm', 'joe', 'goulds', 'secret', 'richardson', 'dalmatians', 'log', 'great', 'supporting', 'roles', 'big', 'surprise', 'graham', 'cringed', 'first', 'time', 'opened', 'mouth', 'imagining', 'attempt', 'irish', 'accent', 'actually', 'wasnt', 'half', 'bad', 'film', 'however', 'good', 'strong', 'violencegore', 'sexuality', 'language', 'drug', 'content']\n"
     ]
    }
   ],
   "source": [
    "# load the document\n",
    "filename = 'Data/txt_sentoken/pos/cv000_29590.txt'\n",
    "text = load_doc(filename)\n",
    "tokens = clean_doc(text)\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3. Defining a vocabulary of preferred words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44276\n"
     ]
    }
   ],
   "source": [
    "# Load the counter module to create a vocabulary of all documents\n",
    "from collections import Counter\n",
    "# Define vocab\n",
    "vocab = Counter()\n",
    "# Add all docs to vocab using our process_docs() function\n",
    "process_docs('Data/txt_sentoken/pos', vocab)\n",
    "process_docs('Data/txt_sentoken/neg', vocab)\n",
    "# print size of vocab\n",
    "print(len(vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** we created our vocabulary from our train set not the train and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25767\n"
     ]
    }
   ],
   "source": [
    "# Remove minimum occurences using the remove_min() function\n",
    "tokens = remove_min(2, vocab)\n",
    "print(len(tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done saving vocabulary to Data/vocab_25767.txt ...\n"
     ]
    }
   ],
   "source": [
    "# Finally lets save this new vocab to later use it using our save_list() function\n",
    "save_list(tokens, 'Data/vocab_25767.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Bag-of-Words Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we need now is a way of representing our data so we can provide it to a multilayer perceptron model. BOW is a way of extracting features from text so the text input can be used with Machine learning algorithms like neural networks.\n",
    "\n",
    "Words in a document are scored and the scores are placed in the corresponding locations in the representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.1. Convert reviews to lines of tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll use our doc_to_line() function.\n",
    "# then we'll walkthrough all the files we have in our directory and encode each into our vocab using the process_docs_full() function\n",
    "\n",
    "def load_clean_dataset(vocab, is_train):\n",
    "    # load documents\n",
    "    neg = process_docs_full('Data/txt_sentoken/neg', vocab, is_train)\n",
    "    pos = process_docs_full('Data/txt_sentoken/pos', vocab, is_train)\n",
    "    docs = neg + pos\n",
    "    \n",
    "    # Prepare the labels\n",
    "    labels = [0 for _ in range(len(neg))] + [1 for _ in range(len(pos))]\n",
    "    return docs, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's load our vocabulary we had created and saved.\n",
    "vocab_filename = 'Data/vocab_25767.txt'\n",
    "vocab = load_doc(vocab_filename)\n",
    "vocab = set(vocab.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1800 1800\n"
     ]
    }
   ],
   "source": [
    "# Finally we can load all our training reviews\n",
    "document, labels = load_clean_dataset(vocab, True)\n",
    "# Summarize what we have\n",
    "print(len(document), len(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200 200\n"
     ]
    }
   ],
   "source": [
    "# Load our testing reviews.\n",
    "test_document, test_labels = load_clean_dataset(vocab, False)\n",
    "# Summarize what we have\n",
    "print(len(test_document), len(test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.2. Encode reviews with a bag-of-words model representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the Keras API to convert reviews to encoded document vectors. The tokenizer class is convenient and will easily transform documents into encoded vectors. First, the tokenizer must be created, then fit on the text documents in the training dataset. Inthis case, these are the aggregation of the positive_lines and negative_lines arrays developed in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load tokenizer module\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "# Create a tokenizer function (Already created in scripts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Putting it all together.\n",
    "# Load vocab created from the words (Loaded above)\n",
    "# Load all reviews\n",
    "train_docs, y_train = load_clean_dataset(vocab, True)\n",
    "test_docs, y_test = load_clean_dataset(vocab, False)\n",
    "# Create the tokenizer\n",
    "tokenizer = create_tokenizer(train_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1800, 25768) (200, 25768)\n"
     ]
    }
   ],
   "source": [
    "# Encode data\n",
    "X_train = tokenizer.texts_to_matrix(train_docs, mode='freq') # You can change the mode to any other.\n",
    "X_test = tokenizer.texts_to_matrix(test_docs, mode='freq')\n",
    "print(X_train.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Sentiment Analysis Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have our train and test data with the same sized encoding vocabulary (vector length), we can develop a Multilayer Perceptron model to classify encoded documents as wither positive or negative.\n",
    "\n",
    "Our MLP should have an input layer that equals the number of words in the vocabulary, and in turn the length of the input documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the number of words. note this is same for the test set too.\n",
    "n_words = X_train.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports for the model\n",
    "from keras.layers import Input, Dense\n",
    "from keras.models import Model\n",
    "from keras.utils import plot_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the model\n",
    "def sen_model1(n_words):\n",
    "    # Define network\n",
    "    input_layer = Input(shape = (n_words,))\n",
    "    hidden1 = Dense(50, activation='relu')(input_layer)\n",
    "    output = Dense(1, activation='sigmoid')(hidden1)\n",
    "    model = Model(inputs = input_layer, outputs = output)\n",
    "    # Get model summary    \n",
    "    # model.summary()\n",
    "    # Plot the model\n",
    "    # plot_model(model, to_file='Model_Images/sent_model1.png', show_shapes=True)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = sen_model1(n_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      " - 1s - loss: 0.6917 - accuracy: 0.6017\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6832 - accuracy: 0.8389\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6670 - accuracy: 0.8800\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6418 - accuracy: 0.8800\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6092 - accuracy: 0.9178\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5692 - accuracy: 0.9422\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5266 - accuracy: 0.9456\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4833 - accuracy: 0.9500\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4400 - accuracy: 0.9556\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3991 - accuracy: 0.9628\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7f4a300ebda0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compile the model\n",
    "model.compile('adam', loss = 'binary_crossentropy', metrics=['accuracy'])\n",
    "# fit the network\n",
    "model.fit(X_train, y_train, epochs=10, verbose = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 87.000000\n"
     ]
    }
   ],
   "source": [
    "# Evaluate network\n",
    "loss, acc = model.evaluate(X_test, y_test, verbose=0)\n",
    "print('Test Accuracy: %f'% (acc*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Comparing Word Scoring Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the above scores are pretty much alright for the dataset we are working with and reflect the paper and text-book for the BOW model.\n",
    "\n",
    "The texts_to_matrix() function in Tokenizer in the Keras API provides 4 different methods for scoring words:\n",
    "    \n",
    "    1. binary: where words are marked as present (1) or absent (0).\n",
    "    2. count: where the occurence count for each word is marked as an integer.\n",
    "    3. tfidf: where each word is scored based on their frequency, where words that are common across all documents are penalized.\n",
    "    4. freq: where words are scored based on their frequency of occurence within the document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because neural network models are stochastic, we will get various results if we run our model several times. This is good practive as we want to get an average accuracy for our models when building. We'll define a function to allow us make multiple runs on a given network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate network\n",
    "def evaluate_mode(trainX, trainY, testX, testY):\n",
    "    scores = list()\n",
    "    n_repeats = 10\n",
    "    n_words = trainX.shape[1]\n",
    "    for i in range(n_repeats):\n",
    "        #call our model\n",
    "        model = sen_model1(n_words)\n",
    "        # compile it\n",
    "        model.compile(loss='binary_crossentropy', optimizer = 'adam', metrics=['accuracy'])\n",
    "        # fit the model\n",
    "        model.fit(trainX, trainY, epochs=10, verbose=2)\n",
    "        # evaluate\n",
    "        loss, acc = model.evaluate(testX, testY, verbose=0)\n",
    "        scores.append(acc)\n",
    "        print('%d accuracy: %s'% ((i+1), acc))\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_12\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_12 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_23 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_24 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6921 - accuracy: 0.5389\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6850 - accuracy: 0.8806\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6702 - accuracy: 0.9050\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6448 - accuracy: 0.8667\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6096 - accuracy: 0.9333\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5685 - accuracy: 0.9417\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5234 - accuracy: 0.9467\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4775 - accuracy: 0.9517\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4328 - accuracy: 0.9544\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3923 - accuracy: 0.9594\n",
      "1 accuracy: 0.8650000095367432\n",
      "Model: \"model_13\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_13 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_25 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_26 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6921 - accuracy: 0.5961\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6845 - accuracy: 0.7500\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6676 - accuracy: 0.8950\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6406 - accuracy: 0.9250\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6042 - accuracy: 0.9250\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5621 - accuracy: 0.9394\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5174 - accuracy: 0.9478\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4721 - accuracy: 0.9533\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4284 - accuracy: 0.9567\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3870 - accuracy: 0.9600\n",
      "2 accuracy: 0.8700000047683716\n",
      "Model: \"model_14\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_14 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_27 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_28 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6924 - accuracy: 0.5194\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6861 - accuracy: 0.6606\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6700 - accuracy: 0.7422\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6429 - accuracy: 0.8506\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6074 - accuracy: 0.9067\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5664 - accuracy: 0.9333\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5208 - accuracy: 0.9506\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4748 - accuracy: 0.9556\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4301 - accuracy: 0.9622\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3878 - accuracy: 0.9656\n",
      "3 accuracy: 0.8899999856948853\n",
      "Model: \"model_15\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_15 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_29 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_30 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6919 - accuracy: 0.5794\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6840 - accuracy: 0.7156\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6678 - accuracy: 0.8556\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6416 - accuracy: 0.9256\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6078 - accuracy: 0.9217\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5673 - accuracy: 0.9450\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5254 - accuracy: 0.9411\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4818 - accuracy: 0.9528\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4399 - accuracy: 0.9539\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3995 - accuracy: 0.9628\n",
      "4 accuracy: 0.8700000047683716\n",
      "Model: \"model_16\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_16 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_31 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_32 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6915 - accuracy: 0.5600\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6815 - accuracy: 0.7856\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6610 - accuracy: 0.9183\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6300 - accuracy: 0.9239\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.5901 - accuracy: 0.9372\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5463 - accuracy: 0.9383\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.4997 - accuracy: 0.9494\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4538 - accuracy: 0.9578\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4092 - accuracy: 0.9617\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3688 - accuracy: 0.9633\n",
      "5 accuracy: 0.8650000095367432\n",
      "Model: \"model_17\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_17 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_33 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_34 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6914 - accuracy: 0.5489\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6807 - accuracy: 0.8889\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6611 - accuracy: 0.8994\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6307 - accuracy: 0.9072\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.5918 - accuracy: 0.9344\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5484 - accuracy: 0.9389\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5022 - accuracy: 0.9483\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4571 - accuracy: 0.9528\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4132 - accuracy: 0.9556\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3719 - accuracy: 0.9639\n",
      "6 accuracy: 0.8650000095367432\n",
      "Model: \"model_18\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_18 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_35 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_36 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      " - 1s - loss: 0.6915 - accuracy: 0.6067\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6811 - accuracy: 0.7122\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6620 - accuracy: 0.8867\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6339 - accuracy: 0.9033\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.5980 - accuracy: 0.9311\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5565 - accuracy: 0.9378\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5127 - accuracy: 0.9456\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4679 - accuracy: 0.9556\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4245 - accuracy: 0.9594\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3845 - accuracy: 0.9622\n",
      "7 accuracy: 0.8600000143051147\n",
      "Model: \"model_19\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_19 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_37 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_38 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.6921 - accuracy: 0.6089\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6851 - accuracy: 0.8428\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6709 - accuracy: 0.9139\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6476 - accuracy: 0.9272\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6166 - accuracy: 0.9294\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5798 - accuracy: 0.9422\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5389 - accuracy: 0.9456\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4982 - accuracy: 0.9517\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4575 - accuracy: 0.9567\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.4182 - accuracy: 0.9600\n",
      "8 accuracy: 0.8650000095367432\n",
      "Model: \"model_20\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_20 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_39 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_40 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6921 - accuracy: 0.5217\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6850 - accuracy: 0.6878\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6705 - accuracy: 0.8656\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6480 - accuracy: 0.8983\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6185 - accuracy: 0.9161\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5835 - accuracy: 0.9367\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5453 - accuracy: 0.9383\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.5058 - accuracy: 0.9467\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4656 - accuracy: 0.9528\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.4268 - accuracy: 0.9561\n",
      "9 accuracy: 0.8700000047683716\n",
      "Model: \"model_21\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_21 (InputLayer)        (None, 25768)             0         \n",
      "_________________________________________________________________\n",
      "dense_41 (Dense)             (None, 50)                1288450   \n",
      "_________________________________________________________________\n",
      "dense_42 (Dense)             (None, 1)                 51        \n",
      "=================================================================\n",
      "Total params: 1,288,501\n",
      "Trainable params: 1,288,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6921 - accuracy: 0.4983\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6853 - accuracy: 0.6733\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6698 - accuracy: 0.8600\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6470 - accuracy: 0.8433\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6175 - accuracy: 0.9272\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5838 - accuracy: 0.9183\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5467 - accuracy: 0.9289\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.5076 - accuracy: 0.9561\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4680 - accuracy: 0.9589\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.4301 - accuracy: 0.9606\n",
      "10 accuracy: 0.8650000095367432\n"
     ]
    }
   ],
   "source": [
    "scores = evaluate_mode(X_train, y_train, X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.8650000095367432,\n",
       " 0.8700000047683716,\n",
       " 0.8899999856948853,\n",
       " 0.8700000047683716,\n",
       " 0.8650000095367432,\n",
       " 0.8650000095367432,\n",
       " 0.8600000143051147,\n",
       " 0.8650000095367432,\n",
       " 0.8700000047683716,\n",
       " 0.8650000095367432]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Accuracy: 86.85%\n"
     ]
    }
   ],
   "source": [
    "# get the average accuracy\n",
    "print('Average Accuracy: %.2f%%'% (sum(scores)*100/len(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Maximum accuracy: 89.00%, Minimum accuracy: 86.00%\n"
     ]
    }
   ],
   "source": [
    "# Get the maximum and minimum accuracy.\n",
    "print('Maximum accuracy: %.2f%%, Minimum accuracy: %.2f%%'% (max(scores)*100, min(scores)*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next:** We need to try out various modes for our BOW model on our multi-run neural network to test how well each mode and model works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Before we loose track of our steps, lets put it all together in one big script**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First let's define a data_prep function for the tokenizer modes.\n",
    "# Prepare bag of words encoding of docs\n",
    "def prepare_bow(train_doc, test_doc, mode):\n",
    "    # Call the create_tokenizer function and fit on train docs\n",
    "    tokenizer = create_tokenizer(train_doc)\n",
    "    # Encode the train data\n",
    "    X_train = tokenizer.texts_to_matrix(train_doc, mode=mode)\n",
    "    # Encode the test data\n",
    "    X_test = tokenizer.texts_to_matrix(test_doc, mode=mode)\n",
    "    return X_train, X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For binary mode, training...\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4502 - accuracy: 0.7872\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0478 - accuracy: 0.9950\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0147 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0078 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0048 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0030 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.0020 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.0010 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 8.0031e-04 - accuracy: 1.0000\n",
      "1 accuracy: 0.9300000071525574\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4992 - accuracy: 0.7361\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0633 - accuracy: 0.9933\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0155 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0070 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0040 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0025 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 9.4126e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 7.3508e-04 - accuracy: 1.0000\n",
      "2 accuracy: 0.9350000023841858\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4811 - accuracy: 0.7833\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0672 - accuracy: 0.9939\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0191 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0085 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0042 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0022 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 7.9090e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 5.4920e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 3.9666e-04 - accuracy: 1.0000\n",
      "3 accuracy: 0.9049999713897705\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4846 - accuracy: 0.7772\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0681 - accuracy: 0.9933\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0184 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0079 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0043 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0025 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 2s - loss: 7.9136e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 5.9430e-04 - accuracy: 1.0000\n",
      "4 accuracy: 0.9300000071525574\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4552 - accuracy: 0.7933\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0419 - accuracy: 0.9978\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0140 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0075 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0048 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.0034 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.0025 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.0015 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "5 accuracy: 0.9350000023841858\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4634 - accuracy: 0.7900\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0567 - accuracy: 0.9956\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0151 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0068 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0038 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0015 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0010 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 7.6600e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 5.8837e-04 - accuracy: 1.0000\n",
      "6 accuracy: 0.925000011920929\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4656 - accuracy: 0.7933\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0584 - accuracy: 0.9950\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0171 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0079 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0043 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0027 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 8.3337e-04 - accuracy: 1.0000\n",
      "7 accuracy: 0.925000011920929\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4901 - accuracy: 0.7700\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0696 - accuracy: 0.9906\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0173 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0076 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0043 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0026 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.0010 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 7.1073e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 2s - loss: 5.0626e-04 - accuracy: 1.0000\n",
      "8 accuracy: 0.9300000071525574\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.4713 - accuracy: 0.7767\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.0538 - accuracy: 0.9950\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.0154 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0070 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.0040 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0026 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 8.0330e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 5.8153e-04 - accuracy: 1.0000\n",
      "9 accuracy: 0.925000011920929\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.4677 - accuracy: 0.7922\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.0616 - accuracy: 0.9928\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.0163 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0083 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0049 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0030 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.0013 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 2s - loss: 9.1460e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 6.9673e-04 - accuracy: 1.0000\n",
      "10 accuracy: 0.925000011920929\n",
      "For count mode, training...\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4620 - accuracy: 0.7806\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0498 - accuracy: 0.9939\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0134 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0062 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0035 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.0022 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 9.7856e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 2s - loss: 7.1152e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 2s - loss: 5.4357e-04 - accuracy: 1.0000\n",
      "1 accuracy: 0.9150000214576721\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4538 - accuracy: 0.7722\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0523 - accuracy: 0.9922\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0148 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0062 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0032 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.0013 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 8.9397e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 6.7133e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 5.2377e-04 - accuracy: 1.0000\n",
      "2 accuracy: 0.8949999809265137\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4843 - accuracy: 0.7639\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0674 - accuracy: 0.9883\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0156 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.0061 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.0031 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 7.6156e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 5.5780e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 4.2657e-04 - accuracy: 1.0000\n",
      "3 accuracy: 0.9150000214576721\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4878 - accuracy: 0.7589\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0691 - accuracy: 0.9894\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0181 - accuracy: 0.9994\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0069 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.0038 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.0024 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 8.8912e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 6.9438e-04 - accuracy: 1.0000\n",
      "4 accuracy: 0.9049999713897705\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.4451 - accuracy: 0.7889\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.0456 - accuracy: 0.9944\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.0134 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0070 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0042 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0028 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0020 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.0015 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 2s - loss: 0.0011 - accuracy: 1.0000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10/10\n",
      " - 2s - loss: 9.1354e-04 - accuracy: 1.0000\n",
      "5 accuracy: 0.8949999809265137\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4545 - accuracy: 0.7856\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.0614 - accuracy: 0.9861\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0137 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.0066 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.0041 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0028 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.0020 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0015 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 9.5503e-04 - accuracy: 1.0000\n",
      "6 accuracy: 0.9100000262260437\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.4537 - accuracy: 0.7689\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.0468 - accuracy: 0.9933\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0131 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0060 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0034 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0022 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0015 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 8.5661e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 6.7568e-04 - accuracy: 1.0000\n",
      "7 accuracy: 0.9049999713897705\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4569 - accuracy: 0.7839\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0519 - accuracy: 0.9939\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0143 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0060 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0031 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0020 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 9.9967e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 7.6848e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 6.0677e-04 - accuracy: 1.0000\n",
      "8 accuracy: 0.9049999713897705\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4719 - accuracy: 0.7822\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0525 - accuracy: 0.9944\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0140 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0067 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0039 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 9.8068e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 7.1193e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 5.4774e-04 - accuracy: 1.0000\n",
      "9 accuracy: 0.9150000214576721\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4630 - accuracy: 0.7756\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0480 - accuracy: 0.9972\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0137 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0060 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0034 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0022 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0015 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 8.4303e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 6.6290e-04 - accuracy: 1.0000\n",
      "10 accuracy: 0.8799999952316284\n",
      "For tfidf mode, training...\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4793 - accuracy: 0.7617\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0157 - accuracy: 0.9989\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0035 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0018 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 7.7525e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 5.5745e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 4.1294e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.1851e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 2s - loss: 2.5318e-04 - accuracy: 1.0000\n",
      "1 accuracy: 0.8700000047683716\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4665 - accuracy: 0.7706\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0160 - accuracy: 0.9983\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0039 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 7.7464e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 5.5507e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 4.0646e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.0700e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 2.3445e-04 - accuracy: 1.0000\n",
      "2 accuracy: 0.8849999904632568\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4629 - accuracy: 0.7817\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0130 - accuracy: 0.9989\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0037 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 8.2276e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 6.0987e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 4.6896e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.7210e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 3.0222e-04 - accuracy: 1.0000\n",
      "3 accuracy: 0.8899999856948853\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4794 - accuracy: 0.7583\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0173 - accuracy: 1.0000\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0045 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 9.1327e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 6.4032e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 4.6875e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 2s - loss: 3.5630e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 2s - loss: 2.7989e-04 - accuracy: 1.0000\n",
      "4 accuracy: 0.8849999904632568\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.4721 - accuracy: 0.7700\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.0184 - accuracy: 0.9967\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.0036 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 7.6989e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 5.6221e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 4.2639e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 2s - loss: 3.3338e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 2.6788e-04 - accuracy: 1.0000\n",
      "5 accuracy: 0.8849999904632568\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4529 - accuracy: 0.7817\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0133 - accuracy: 0.9983\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.0034 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 2s - loss: 8.1754e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 5.8852e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 4.3902e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.3790e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 2.6933e-04 - accuracy: 1.0000\n",
      "6 accuracy: 0.8450000286102295\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4654 - accuracy: 0.7728\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0179 - accuracy: 0.9983\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0036 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0013 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 8.6605e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 6.1541e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 2s - loss: 4.5586e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.4636e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 2.6836e-04 - accuracy: 1.0000\n",
      "7 accuracy: 0.8700000047683716\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4953 - accuracy: 0.7550\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0260 - accuracy: 0.9956\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0056 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.0024 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 9.1433e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 6.3303e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 4.5523e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.4792e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 2s - loss: 2.7289e-04 - accuracy: 1.0000\n",
      "8 accuracy: 0.8799999952316284\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4687 - accuracy: 0.7594\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0187 - accuracy: 0.9989\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0040 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0019 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0012 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 8.1476e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 2s - loss: 5.9498e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 4.5524e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.5577e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 2.8531e-04 - accuracy: 1.0000\n",
      "9 accuracy: 0.8849999904632568\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.4519 - accuracy: 0.7922\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0133 - accuracy: 0.9983\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0032 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.0017 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.0011 - accuracy: 1.0000\n",
      "Epoch 6/10\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - 2s - loss: 7.5056e-04 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 5.2937e-04 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 3.9853e-04 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 3.0660e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 2.4440e-04 - accuracy: 1.0000\n",
      "10 accuracy: 0.8849999904632568\n",
      "For freq mode, training...\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6921 - accuracy: 0.6133\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6857 - accuracy: 0.8767\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.6732 - accuracy: 0.9028\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6546 - accuracy: 0.8783\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6274 - accuracy: 0.9294\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.5978 - accuracy: 0.9378\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5633 - accuracy: 0.9461\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.5272 - accuracy: 0.9461\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4911 - accuracy: 0.9517\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.4557 - accuracy: 0.9544\n",
      "1 accuracy: 0.8650000095367432\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6916 - accuracy: 0.6456\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6826 - accuracy: 0.6244\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6655 - accuracy: 0.7517\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6387 - accuracy: 0.8756\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.6046 - accuracy: 0.9206\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5644 - accuracy: 0.9394\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5214 - accuracy: 0.9433\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4779 - accuracy: 0.9494\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4353 - accuracy: 0.9583\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3942 - accuracy: 0.9678\n",
      "2 accuracy: 0.8600000143051147\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6910 - accuracy: 0.6578\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6795 - accuracy: 0.7083\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6572 - accuracy: 0.8950\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6238 - accuracy: 0.9211\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.5820 - accuracy: 0.9356\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5344 - accuracy: 0.9439\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.4861 - accuracy: 0.9506\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.4388 - accuracy: 0.9472\n",
      "Epoch 9/10\n",
      " - 2s - loss: 0.3929 - accuracy: 0.9611\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3518 - accuracy: 0.9650\n",
      "3 accuracy: 0.8700000047683716\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6914 - accuracy: 0.5800\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6817 - accuracy: 0.7333\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6630 - accuracy: 0.8961\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6339 - accuracy: 0.8783\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.5944 - accuracy: 0.8978\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5496 - accuracy: 0.9433\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5013 - accuracy: 0.9456\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4527 - accuracy: 0.9589\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4082 - accuracy: 0.9578\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3646 - accuracy: 0.9711\n",
      "4 accuracy: 0.8650000095367432\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6916 - accuracy: 0.6039\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6824 - accuracy: 0.6744\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.6643 - accuracy: 0.8194\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.6371 - accuracy: 0.8128\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.6023 - accuracy: 0.9150\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.5621 - accuracy: 0.9378\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.5189 - accuracy: 0.9489\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.4760 - accuracy: 0.9556\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4333 - accuracy: 0.9644\n",
      "Epoch 10/10\n",
      " - 2s - loss: 0.3937 - accuracy: 0.9689\n",
      "5 accuracy: 0.8700000047683716\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.6919 - accuracy: 0.5250\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.6840 - accuracy: 0.7311\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.6697 - accuracy: 0.7600\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6485 - accuracy: 0.8478\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.6201 - accuracy: 0.9128\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.5865 - accuracy: 0.9333\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.5493 - accuracy: 0.9439\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.5112 - accuracy: 0.9539\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4723 - accuracy: 0.9550\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.4351 - accuracy: 0.9639\n",
      "6 accuracy: 0.875\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.6919 - accuracy: 0.5367\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6839 - accuracy: 0.7828\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6643 - accuracy: 0.8967\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.6321 - accuracy: 0.9139\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.5908 - accuracy: 0.9206\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.5420 - accuracy: 0.9389\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.4922 - accuracy: 0.9394\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.4416 - accuracy: 0.9533\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.3946 - accuracy: 0.9583\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3513 - accuracy: 0.9689\n",
      "7 accuracy: 0.8700000047683716\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6918 - accuracy: 0.6200\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6834 - accuracy: 0.8639\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.6653 - accuracy: 0.8878\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.6356 - accuracy: 0.9117\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.5974 - accuracy: 0.9283\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.5546 - accuracy: 0.9422\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.5093 - accuracy: 0.9467\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.4639 - accuracy: 0.9522\n",
      "Epoch 9/10\n",
      " - 1s - loss: 0.4212 - accuracy: 0.9544\n",
      "Epoch 10/10\n",
      " - 1s - loss: 0.3807 - accuracy: 0.9606\n",
      "8 accuracy: 0.8700000047683716\n",
      "Epoch 1/10\n",
      " - 1s - loss: 0.6911 - accuracy: 0.5383\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.6805 - accuracy: 0.6811\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.6604 - accuracy: 0.9200\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.6310 - accuracy: 0.8911\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.5933 - accuracy: 0.9322\n",
      "Epoch 6/10\n",
      " - 2s - loss: 0.5500 - accuracy: 0.9433\n",
      "Epoch 7/10\n",
      " - 2s - loss: 0.5051 - accuracy: 0.9428\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.4591 - accuracy: 0.9494\n",
      "Epoch 9/10\n",
      " - 2s - loss: 0.4157 - accuracy: 0.9589\n",
      "Epoch 10/10\n",
      " - 2s - loss: 0.3753 - accuracy: 0.9644\n",
      "9 accuracy: 0.8650000095367432\n",
      "Epoch 1/10\n",
      " - 2s - loss: 0.6919 - accuracy: 0.4994\n",
      "Epoch 2/10\n",
      " - 2s - loss: 0.6837 - accuracy: 0.5539\n",
      "Epoch 3/10\n",
      " - 2s - loss: 0.6677 - accuracy: 0.7339\n",
      "Epoch 4/10\n",
      " - 2s - loss: 0.6429 - accuracy: 0.8333\n",
      "Epoch 5/10\n",
      " - 2s - loss: 0.6109 - accuracy: 0.9089\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.5739 - accuracy: 0.9217\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.5341 - accuracy: 0.9467\n",
      "Epoch 8/10\n",
      " - 2s - loss: 0.4926 - accuracy: 0.9533\n",
      "Epoch 9/10\n",
      " - 2s - loss: 0.4514 - accuracy: 0.9617\n",
      "Epoch 10/10\n",
      " - 2s - loss: 0.4114 - accuracy: 0.9694\n",
      "10 accuracy: 0.875\n",
      "          binary      count      tfidf       freq\n",
      "count  10.000000  10.000000  10.000000  10.000000\n",
      "mean    0.926500   0.904000   0.878000   0.868500\n",
      "std     0.008515   0.011255   0.013375   0.004743\n",
      "min     0.905000   0.880000   0.845000   0.860000\n",
      "25%     0.925000   0.897500   0.872500   0.865000\n",
      "50%     0.927500   0.905000   0.885000   0.870000\n",
      "75%     0.930000   0.913750   0.885000   0.870000\n",
      "max     0.935000   0.915000   0.890000   0.875000\n"
     ]
    }
   ],
   "source": [
    "# Step 1: Load modules from all locations\n",
    "from text_prep import * # All our text prep functions\n",
    "from save_processed import * # all our save routines.\n",
    "# Supporting modules\n",
    "from keras.layers import Input, Dense\n",
    "from keras.models import Model\n",
    "from pandas import DataFrame\n",
    "from matplotlib import pyplot\n",
    "\n",
    "# Load the vocabulary\n",
    "vocab_filename = 'Data/vocab_25767.txt'\n",
    "vocab = load_doc(vocab_filename)\n",
    "vocab = set(vocab.split())\n",
    "\n",
    "# Load all reviews\n",
    "train_doc, y_train = load_clean_dataset(vocab, True)\n",
    "test_doc, y_test = load_clean_dataset(vocab, False)\n",
    "\n",
    "# Run experiment on all modes\n",
    "modes = ['binary', 'count', 'tfidf', 'freq']\n",
    "results = DataFrame()\n",
    "for mode in modes:\n",
    "    print('For %s mode, training...'%(mode))\n",
    "    # Prepare data for mode\n",
    "    X_train, X_test = prepare_bow(train_doc, test_doc, mode)\n",
    "    # Evaluate model on data for mode\n",
    "    results[mode] = evaluate_mode(X_train, y_train, X_test, y_test)\n",
    "# Summarize results\n",
    "print(results.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>binary</th>\n",
       "      <th>count</th>\n",
       "      <th>tfidf</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.926500</td>\n",
       "      <td>0.904000</td>\n",
       "      <td>0.878000</td>\n",
       "      <td>0.868500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.008515</td>\n",
       "      <td>0.011255</td>\n",
       "      <td>0.013375</td>\n",
       "      <td>0.004743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.905000</td>\n",
       "      <td>0.880000</td>\n",
       "      <td>0.845000</td>\n",
       "      <td>0.860000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.925000</td>\n",
       "      <td>0.897500</td>\n",
       "      <td>0.872500</td>\n",
       "      <td>0.865000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.927500</td>\n",
       "      <td>0.905000</td>\n",
       "      <td>0.885000</td>\n",
       "      <td>0.870000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.930000</td>\n",
       "      <td>0.913750</td>\n",
       "      <td>0.885000</td>\n",
       "      <td>0.870000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.935000</td>\n",
       "      <td>0.915000</td>\n",
       "      <td>0.890000</td>\n",
       "      <td>0.875000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          binary      count      tfidf       freq\n",
       "count  10.000000  10.000000  10.000000  10.000000\n",
       "mean    0.926500   0.904000   0.878000   0.868500\n",
       "std     0.008515   0.011255   0.013375   0.004743\n",
       "min     0.905000   0.880000   0.845000   0.860000\n",
       "25%     0.925000   0.897500   0.872500   0.865000\n",
       "50%     0.927500   0.905000   0.885000   0.870000\n",
       "75%     0.930000   0.913750   0.885000   0.870000\n",
       "max     0.935000   0.915000   0.890000   0.875000"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the evaluation accuracies for the various modes.\n",
    "results.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAE5JJREFUeJzt3X+QnVdh3vHvU8mOjRECLKIEDJLbMdNVJIcW1SmDmuzGE9eEgBtIiLdJkZNtRZsippk4tZhlwHFnJyKBTNriplVYx8JOlwHNpHUsxTZRdyGixrEdkI29GFzXYOHMxECiWMZTW8rpH/cVulrJ3rvSK93de76fmTt633PPe/bco7vPffe8P25KKUiS6vB3+t0BSdLZY+hLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKrK83x2Ya9WqVWXt2rX97sa8nnnmGS644IJ+d2NgOJ7tcjzbs1TG8v777/9WKeVV89VbdKG/du1a7rvvvn53Y14zMzMMDw/3uxsDw/Fsl+PZnqUylkm+3ks9p3ckqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFVl0F2ctBklabc/vIZa0WLinfxKllHkfa667vad6Br6kxcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0JekilR1G4Yf/vW7OPjs8621t3bb7lbaWXn+Oez/0BWttCVJL6aq0D/47PM8vv2trbTV5pclt/XhIUnzcXpHkipi6EtSRQx9SaqIoS9JFanqQO6KoW1s2LmtvQZ3ttPMiiGAdg4wS9KLqSr0n57d7tk7kqrm9I4kVcTQl6SKGPqSVJGq5vSht/nzr3/4p1r9mWuuu/1Fn195/jmt/jxJeiFVhX7PB3G3l3mrtHkgV5LOFqd3JKkihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkV6Cv0kVyZ5JMmjSU64Y1mSNUn2JnkgyUySi5ryNyS5O8lDzXM/1/YLkCT1bt7QT7IMuBF4C7AOGE2ybk61jwCfKKVcCtwA/EZT/l3g3aWUHwKuBH4nycvb6rwkaWF62dO/DHi0lPJYKeU54JPAVXPqrAP2NsvTR58vpXy1lPK1ZvlJ4C+BV7XRcUnSwvVyRe5rgCe61g8APzKnzn7gncB/BH4aWJHkwlLKt49WSHIZcC7wf+b+gCRbgC0Aq1evZmZmZgEvoT8OHTq0JPq5VDie7XI82zNoY9lL6OckZXPvU3At8LEk1wCfA74JHP5eA8kPArcAm0spf3tCY6XsAHYAbNy4sSyF2xt4G4Z2OZ7tcjzbM2hj2UvoHwBe27V+EfBkd4Vm6uYdAEleCryzlHKwWX8ZsBv4QCnlC210WpJ0anqZ078XuCTJxUnOBa4GbuuukGRVkqNtvR+4qSk/F/hDOgd5P91etyVJp2Le0C+lHAbeC9wJzAKfKqU8lOSGJG9vqg0DjyT5KrAamGjK3wX8KHBNki81jze0/SIkSb3p6dbKpZQ9wJ45ZR/sWt4F7DrJdrcCt55mHyVJLfGKXEmqiKEvSRUx9CWpIoa+JFXE0JekilT1xejqj+RkF3WfmlLm/9J6SS/MPX2dcaWUeR9rrru9p3qSTo+hL0kVcXpHp+yHf/0uDj77fGvtrd22u5V2Vp5/Dvs/dEUrbUmDxtDXKTv47PM8vv2trbTV5p0M2/rwkAaR0zuSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKG/gJNTU2xfv16Lr/8ctavX8/U1FS/uyRJPfN++gswNTXF+Pg4k5OTHDlyhGXLljE2NgbA6Ohon3snSfNzT38BJiYmmJycZGRkhOXLlzMyMsLk5CQTExP97pok9cQ9/QWYnZ1l06ZNx5Vt2rSJ2dnZPvWov1YMbWPDzm3tNbiznWZWDAG0841e0qAx9BdgaGiIffv2MTIy8r2yffv2MTQ01Mde9c/Ts9v9ukRpiXF6ZwHGx8cZGxtjenqaw4cPMz09zdjYGOPj4/3umiT1xD39BTh6sHbr1q3Mzs4yNDTExMSEB3ElLRmG/gKNjo4yOjra6nSEJJ0tTu9IUkUMfUmqSE+hn+TKJI8keTTJCefoJVmTZG+SB5LMJLmo67nNSb7WPDa32XlJ0sLMG/pJlgE3Am8B1gGjSdbNqfYR4BOllEuBG4DfaLZ9JfAh4EeAy4APJXlFe92XJC1EL3v6lwGPllIeK6U8B3wSuGpOnXXA3mZ5uuv5fwp8ppTynVLKXwGfAa48/W5Lkk5FL6H/GuCJrvUDTVm3/cA7m+WfBlYkubDHbSVJZ0kvp2zmJGVlzvq1wMeSXAN8DvgmcLjHbUmyBdgCsHr1amZmZnroVn8dOnRoSfTzTGtrDNoez9r/b3x/tmfQxrKX0D8AvLZr/SLgye4KpZQngXcAJHkp8M5SysEkB4DhOdvOzP0BpZQdwA6AjRs3lqVw/rvn6QN37G5tDFodzxb7tVT5/mzPoI1lL9M79wKXJLk4ybnA1cBt3RWSrEpytK33Azc1y3cCVyR5RXMA94qmTJLUB/OGfinlMPBeOmE9C3yqlPJQkhuSvL2pNgw8kuSrwGpgotn2O8B/oPPBcS9wQ1MmSeqDnm7DUErZA+yZU/bBruVdwK4X2PYmju35S5L6yCtyJaki3nBNp6XVe9ff0U5bK88/p5V2pEFk6OuUtfUFKtD58GizPUkn5/SOJFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIt1aWlpAkrbZXSmm1PS1+7ulLS0gppafHmutu76me6mPoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRbwNg7RIbNi5obW2VgzBhp3bWmvvwc0PttaW+svQlxaJp2e38/j2t7bS1szMDMPDw620tXbb7lba0eLg9I4kVcTQl6SKGPqSVBFDX5IqYuhLUkV6Cv0kVyZ5JMmjSU44DyzJ65JMJ/likgeS/GRTfk6SnUkeTDKb5P1tvwBJUu/mDf0ky4AbgbcA64DRJOvmVPsA8KlSyj8Argb+S1P+s8D3lVI2AG8E3pNkbTtdlyQtVC97+pcBj5ZSHiulPAd8ErhqTp0CvKxZXgk82VV+QZLlwPnAc8DfnHavJUmnpJfQfw3wRNf6gaas2/XALyQ5AOwBtjblu4BngL8AvgF8pJTyndPpsCTp1PVyRW5OUjb3G5VHgZtLKR9N8ibgliTr6fyVcAR4NfAK4E+T/Ekp5bHjfkCyBdgCsHr1amZmZhb2Kvrg0KFDS6Kfi8HIyEhP9fLh+etMT0+fZm8Wt7beU22/P2t+rw/a73ovoX8AeG3X+kUcm745agy4EqCUcneS84BVwD8H7iilPA/8ZZLPAxuB40K/lLID2AGwcePG0tbl42dSm5e5D7pS5u4jnMjxBO7Y3doYtDqeLfZrKRq092Yv0zv3ApckuTjJuXQO1N42p843gMsBkgwB5wFPNeU/no4LgH8MfKWtzkuSFmbe0C+lHAbeC9wJzNI5S+ehJDckeXtT7VeBf5VkPzAFXFM6u3c3Ai8Fvkznw+P3SykPnIHXIUnqQU932Syl7KFzgLa77INdyw8Dbz7JdofonLYpSVoEvCJXkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcQvRpdUreRkd5k5db1cfd5v7ulLqlYpZd7Hmutu76neUgh8MPQlqSqGviRVxNCXpIoY+pJUEc/ekRaRtdt2t9fYHe20tfL8c1ppR4uDoS8tEo9vf2trba3dtrvV9jQ4nN6RpIoY+pJUEUNfkirinL60hCzktgH58Px1lspVpKdiw84NrbSzYgg27NzWSlsAD25+sLW2ToWhLy0hvYb0zMwMw8PDZ7Yzi9zTs9tbOZjd5li2enbWKXJ6R5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKuIVuZIGVmtXwA7QdxMY+pIGUlvfJzBo303g9I4kVcTQl6SKGPqSVBFDX5IqYuhLUkV6Cv0kVyZ5JMmjSU74Cpkkr0syneSLSR5I8pNdz12a5O4kDyV5MMl5bb4ASVLv5j1lM8ky4EbgJ4ADwL1JbiulPNxV7QPAp0opv5tkHbAHWJtkOXAr8C9KKfuTXAg83/qrkCT1pJc9/cuAR0spj5VSngM+CVw1p04BXtYsrwSebJavAB4opewHKKV8u5Ry5PS7LUk6Fb2E/muAJ7rWDzRl3a4HfiHJATp7+Vub8tcDJcmdSf48yb8/zf5Kkk5DL1fk5iRlc7+deRS4uZTy0SRvAm5Jsr5pfxPwj4DvAnuT3F9K2XvcD0i2AFsAVq9ezczMzMJeRR8cOnRoSfRzqXA82+V49mZkZKSnevlwb+1NT0+fRm/Ojl5C/wDw2q71izg2fXPUGHAlQCnl7uZg7apm28+WUr4FkGQP8A+B40K/lLID2AGwcePG0tY3z59JMzMzLIV+LhWOZ7scz96UMnf/9USDNpa9TO/cC1yS5OIk5wJXA7fNqfMN4HKAJEPAecBTwJ3ApUle0hzU/THgYSRJfTHvnn4p5XCS99IJ8GXATaWUh5LcANxXSrkN+FXg95L8Cp2pn2tK5yP0r5L8Np0PjgLsKaW0dNs7SdJC9XSXzVLKHjoHaLvLPti1/DDw5hfY9lY6p21KkvrMK3IlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGvjRApqamWL9+PZdffjnr169namqq313SIrO83x2Q1I6pqSnGx8eZnJzkyJEjLFu2jLGxMQBGR0f73DstFu7pSwNiYmKCyclJRkZGWL58OSMjI0xOTjIxMdHvrmkRMfSlATE7O8umTZuOK9u0aROzs7N96pEWI0NfGhBDQ0Ps27fvuLJ9+/YxNDTUpx5pMTL0pQExPj7O2NgY09PTHD58mOnpacbGxhgfH+9317SIeCBXGhBHD9Zu3bqV2dlZhoaGmJiY8CCujmPoSwNkdHSU0dFRZmZmGB4e7nd3tAg5vSNJFTH0Jakihr4kVcTQl6SKGPqSVJGUUvrdh+MkeQr4er/70YNVwLf63YkB4ni2y/Fsz1IZyzWllFfNV2nRhf5SkeS+UsrGfvdjUDie7XI82zNoY+n0jiRVxNCXpIoY+qduR787MGAcz3Y5nu0ZqLF0Tl+SKuKeviRVpOrQT7I2yZdPUv7xJOv60Se9uCT/LslL+t2Pfkny8iS/3LX+W0keav7910nefZJtjnufJ5lK8kCSXzlb/V7MkrwvyWySP+h3X86Gqqd3kqwFbi+lrD9D7S8vpRw+E23XKsnjwMZSylI4b7p1c9+zSf4GeFUp5f/1sk2SHwDuKaWsOfO9XRqSfAV4Synl/3aVDezvbtV7+o3lSXY2ez67krwkyUySjQBJDiWZSLI/yReSrG7K35bkniRfTPInXeXXJ9mR5C7gE0n+NMkbjv6wJJ9PcmlfXulZkuTdzXjuT3JLkjVJ9jZle5O8rql3c5Kf6druUPPvcPN/sCvJV5L8QTreB7wamE4y3Z9X13fbgb+X5EtJPgNcANyT5Oea9961AEne2Iz/3cC/7dr+LuD7m+3/ydnv/uKS5L8Cfxe4LcnBOb+7y5q/oO5t3rvvabZJko8leTjJ7iR7ut/Hi14ppdoHsBYowJub9ZuAa4EZOnuTNM+/rVn+TeADzfIrOPaX0r8EPtosXw/cD5zfrG8GfqdZfj1wX79f9xke0x8CHgFWNeuvBP4I2Nys/xLwP5rlm4Gf6dr2UPPvMHAQuIjOjsndwKbmucePtl3jo3nPfnnumDXL1wPXNssPAD/WLP/W0W3mbu/j2HvqJL+7W7p+378PuA+4GHgH8BlgGZ2dkL/ufh8v9od7+vBEKeXzzfKtwKY5zz8H3N4s30/nlwY6gXRnkgeBX6MTdkfdVkp5tln+NPBTSc6hE3g3t9r7xefHgV2lmX4ppXwHeBPw35vnb+HEMT6ZPyulHCil/C3wJY6Nu+aRZCXw8lLKZ5uiW/rZnyWm+3f3CuDdSb4E3ANcCFwC/CgwVUo5Ukp5Evhf/enqqTH0O3vyL7b+fGk+6oEjHPu2sf8MfKyUsgF4D3Be1zbPfK+xUr5LZ6/gKuBdHAu/QRVOHMO5jj5/mOY9mCTAuV11uueou8dd8+vl/0An90zXcoCtpZQ3NI+LSyl3Nc8t2fE19OF1Sd7ULI8C+3rcbiXwzWZ58zx1Pw78J+DeZs93kO0F3pXkQoAkrwT+N3B18/zPc2yMHwfe2CxfBZzTQ/tPAyva6uwSNO/rL6X8NXAwydG/qH7+jPdqMN0J/Jvmr3SSvD7JBcDngKubOf8fBEb62cmFcu8JZoHNSf4b8DXgd4G39bDd9cCnk3wT+AKdub6TKqXc35xl8fun393FrZTyUJIJ4LNJjgBfBN4H3JTk14CngF9sqv8e8D+T/BmdD4tnTtbmHDuAP07yF6WUJfXL1oZSyrebkwG+DPzxi1T9RTpj/l064aWF+zidacU/b/4SfQr4Z8Af0pnGfBD4KvDZF2pgMar6lM2zJcmr6Rwc/vvNHLWkAZHkZjqnxO7qd1964fTOGdZcLHMPMG7gS+o39/QlqSLu6UtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SK/H+v28aJA/xGIQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Show the plot of results\n",
    "results.boxplot()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Predicting Sentiment for New Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see from the previous section and boxplot that the binary mode gives us the best scoring for the bag of words model.\n",
    "\n",
    "Finally we will use this binary model to predict on new textual reviews. This is why we wanted the model in the first place.\n",
    "\n",
    "To predict on a new data, we have to follow the process for preparing the test set to prepare the new text.\n",
    "\n",
    "We can then call our model.predict directly and get an integer of 0 for a negative review and 1 for a positive review. We can put all these steps into a function called predict_sentiment() that requires the review text, the vocabulary, the tokenizer and the fit model and returns the predicted sentiment and associated level of confidence-like output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Classify a review as negative or positive\n",
    "def predict_sentiment(review, vocab, tokenizer, model):\n",
    "    \"\"\"\n",
    "    params: \n",
    "    review: raw text to predict\n",
    "    vocab: the text vocabulary from corpus\n",
    "    tokenizer: the tokenizer with mode choice\n",
    "    model: a classifier model\n",
    "    \n",
    "    return:\n",
    "    percent_pos: percentage probability\n",
    "    'REVIEW': Negative or positive\n",
    "    \"\"\"\n",
    "    # Clean\n",
    "    tokens = clean_doc(review)\n",
    "    # filter by vocab\n",
    "    tokens = [w for w in tokens if w in vocab]\n",
    "    # Convert to line\n",
    "    line = ' '.join(tokens)\n",
    "    # Encode\n",
    "    encoded = tokenizer.texts_to_matrix([line], mode='binary')\n",
    "    # Predict sentiment\n",
    "    yhat = model.predict(encoded, verbose=0)\n",
    "    # Retrieve predicted percentage and label\n",
    "    percent_pos = yhat[0, 0]\n",
    "    if round(percent_pos) == 0:\n",
    "        return (1-percent_pos), 'NEGATIVE'\n",
    "    return percent_pos, 'POSITIVE'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      " - 1s - loss: 0.4779 - accuracy: 0.7933\n",
      "Epoch 2/10\n",
      " - 1s - loss: 0.0605 - accuracy: 0.9933\n",
      "Epoch 3/10\n",
      " - 1s - loss: 0.0155 - accuracy: 1.0000\n",
      "Epoch 4/10\n",
      " - 1s - loss: 0.0076 - accuracy: 1.0000\n",
      "Epoch 5/10\n",
      " - 1s - loss: 0.0044 - accuracy: 1.0000\n",
      "Epoch 6/10\n",
      " - 1s - loss: 0.0027 - accuracy: 1.0000\n",
      "Epoch 7/10\n",
      " - 1s - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 8/10\n",
      " - 1s - loss: 0.0010 - accuracy: 1.0000\n",
      "Epoch 9/10\n",
      " - 1s - loss: 6.8785e-04 - accuracy: 1.0000\n",
      "Epoch 10/10\n",
      " - 1s - loss: 4.2322e-04 - accuracy: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7f49e8591358>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We will train our model again then predict\n",
    "# Load vocab\n",
    "vocab_filename = 'Data/vocab_25767.txt'\n",
    "vocab = load_doc(vocab_filename)\n",
    "vocab = set(vocab.split())\n",
    "\n",
    "# Load all reviews\n",
    "train_doc, y_train = load_clean_dataset(vocab, True)\n",
    "test_doc, y_test = load_clean_dataset(vocab, False)\n",
    "\n",
    "# tokenize and encode data\n",
    "X_train, X_test = prepare_bow(train_doc, test_doc, mode='binary')\n",
    "\n",
    "# define model\n",
    "n_words = X_train.shape[1]\n",
    "model = sen_model1(n_words)\n",
    "\n",
    "# Compile and fit model\n",
    "model.compile(loss='binary_crossentropy', optimizer = 'adam', metrics=['accuracy'])\n",
    "model.fit(X_train, y_train, epochs=10, verbose = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: [Best movie ever! it was great, I recommend it.]\n",
      "Sentiment: POSITIVE (57.179%)\n",
      "Review: [This is a very bad and terrible movie.]\n",
      "Sentiment: NEGATIVE (73.734%)\n"
     ]
    }
   ],
   "source": [
    "# Define some test cases\n",
    "text = 'Best movie ever! it was great, I recommend it.'\n",
    "percent, sentiment = predict_sentiment(text, vocab, tokenizer, model)\n",
    "print('Review: [%s]\\nSentiment: %s (%.3f%%)'%(text, sentiment, percent*100))\n",
    "# Another\n",
    "text2 = 'This is a very bad and terrible movie.'\n",
    "percent, sentiment = predict_sentiment(text2, vocab, tokenizer, model)\n",
    "print('Review: [%s]\\nSentiment: %s (%.3f%%)'%(text2, sentiment, percent*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally!!**\n",
    "\n",
    "Ideally, we would fit the model on all available data (train and test) to create a final model and save the model and tokenizer to file so that they can be loaded and used in new software."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking to go deeper using this methodology? Try these:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Manage Vocabulary: Explore using a larger or smaller vocabulary, reduce or increase the occurence words. Perhaps you can get better performances with a smaller set of words.\n",
    "\n",
    "2. Tune the Network Topology: Explore alternate network topologies such as deeper or wider networks. You might achieve better performances by doing this.\n",
    "\n",
    "3. Use Regularization: Explore the use of regularization techniques, such as dropout. Perhaps the convergence may be delayed on the model and achieve better test performance.\n",
    "    \n",
    "4. More Data Cleaning: Explore more or less cleaning of the review text and see how it impacts the model skill.\n",
    "    \n",
    "5. Training Diagnostics: Use the test dataset as a validation dataset during training and create plots of train and test loss. use these diagnostics to tune the batch size and number of training epochs.\n",
    "    \n",
    "6. Trigger Words: Explore whether there are specific words in reviews that are highly predictive of the sentiment.\n",
    "    \n",
    "7. Use Bigrams: Prepare the model to score bigrams of words and evaluate the performance under different scoring schemes.\n",
    "    \n",
    "8. Truncated Reviews. Explore how using a truncated version of the movie reviews results impacts model skill, try truncating the start, end and middle of reviews.\n",
    "\n",
    "9. Ensemble Models: Create models with different word scoring schemes and see if using ensembles of the models results in improvements to model skill.\n",
    "    \n",
    "10. Real reviews: Train a final model on all data and evaluate the model on real movie reviews taken from the internet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}