CMSE202_GroupG_Final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting the Sentiment of Movie Reviews\n",
    "Carson Honkala, Hannah Striebel, Foster Krause, Kevin Hong, Ben Sinko"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Links\n",
    "\n",
    "**Github repo:** https://github.com/carsonh079/CMSE202_Final_Project\n",
    "\n",
    "**Presentation Recording:** https://mediaspace.msu.edu/media/t/1_0wnpu1qs\n",
    "\n",
    "**Presentation Slides:** https://docs.google.com/presentation/d/1ygOKZUn32GHreyUx3oSXRdqKGoI-LmKoX_e518aOp9k/edit?usp=sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.datasets import load_svmlight_file\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.linear_model import Perceptron\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn import datasets, linear_model\n",
    "from sklearn.tree import DecisionTreeClassifier, plot_tree\n",
    "import time\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of our project is to analyze movie review data in order to predict their sentiment. To do this, we will train various machine learning algorithms to determine which algorithm works best. We will also attempt to improve performance by applying `GirdSearchCV` to tune the hyper parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this project, we are using the *Large Movie Review Dataset* from the 2011 publication, *Learning Word Vectors for Sentiment Analysis*. The dataset contains the text from 50,000 movie reviews labeled 0 through 10, with 25,000 designated for training and the other 25,000 for testing. Also included is a tokenized bag of words (BOW) that contains a count of the number of times a given word appears in a review. The inclusion of this BOW removed the need for us to process the text ourselves, and can be directly used as the input for our various machine learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bag of Words Reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we read in the \"imdb.vocab\" file, which acts as a reference for the bag of words described above. The bag of words contains a bunch of key-value pairs, in which the key is the index of the word in this reference array, and the value is the number of times that word that appears in the reviews. For example, if the first element in a sample of input data is 12, that means the word 'the' (the first element in the reference) appears 12 times in that review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the' 'and' 'a' ... 'profanity-laced' 'sarah-michelle' 'copywrite']\n"
     ]
    }
   ],
   "source": [
    "# bow_reference contains the list of words that correspond to the bag of words representation\n",
    "bow_reference = pd.read_table('aclImdb/imdb.vocab', header = None)[0].values\n",
    "\n",
    "# drop the last four words to keep in line with test data\n",
    "bow_reference = bow_reference[:-4]\n",
    "print(bow_reference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we import the training bag of words which is stored in a `.feat` file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load in and format the training and testing data\n",
    "\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_svmlight_file.html\n",
    "# X is a Sparse Matrix that contains the word counts for each review\n",
    "# y is a NumPy array that contains the review score \n",
    "X, y = load_svmlight_file('aclImdb/train/labeledBow.feat')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: We dropped the final 4 columns of the training data to ensure that the data format was identical for both the training and testing sets. These four columns represented words that occurred in only one sample of the training set, so this should have no noticable effect on the accuracy.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Samples: 25000\n",
      "Unique Words: 89523\n"
     ]
    }
   ],
   "source": [
    "# the X.toarray() converts the sparse matrix to a 2D NumPy array, \n",
    "# which is then converted to a Pandas DF\n",
    "Xdata = pd.DataFrame(X.toarray())\n",
    "Xdata = Xdata.drop(columns=[89523, 89524, 89525, 89526])\n",
    "\n",
    "print(\"Samples:\", Xdata.shape[0])\n",
    "print(\"Unique Words:\", Xdata.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we convert the all review scores 0-4 to negative (0) and 5-10 to positive (1). Since we are only using simple word counts to classify the data, it seemed very unrealistic that the machine learning models would be able to accurately predict the score of a particular review. By changing our classification to only positive or negative, we hope to increase the chance of creating an accurate prediction model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# changes all review scores 0-4 to negative (0) and 5-10 to positive(1)\n",
    "ydata = pd.DataFrame(y)\n",
    "ydata = ydata.replace({0:0,1:0,2:0,3:0,4:0,5:1,6:1,7:1,8:1,9:1,10:1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process of importing the testing data is exactly the same as the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = load_svmlight_file('aclImdb/test/labeledBow.feat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Samples: 25000\n",
      "Unique Words: 89523\n"
     ]
    }
   ],
   "source": [
    "Xtest = pd.DataFrame(X.toarray())\n",
    "\n",
    "print(\"Samples:\", Xtest.shape[0])\n",
    "print(\"Unique Words:\", Xtest.shape[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "ytest = pd.DataFrame(y)\n",
    "ytest = ytest.replace({0:0,1:0,2:0,3:0,4:0,5:1,6:1,7:1,8:1,9:1,10:1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a visual representation of the data format. Every row represents a unique sample, and each column represents how many time the corresponding word occurred in that sample. For example, the 0th element of row 0 is 9.0, meaning that the first word in the bow_reference ('the') appears 9 times in that sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>89513</th>\n",
       "      <th>89514</th>\n",
       "      <th>89515</th>\n",
       "      <th>89516</th>\n",
       "      <th>89517</th>\n",
       "      <th>89518</th>\n",
       "      <th>89519</th>\n",
       "      <th>89520</th>\n",
       "      <th>89521</th>\n",
       "      <th>89522</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 89523 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0      1      2      3      4      5      6      7      8      9      ...  \\\n",
       "0    9.0    1.0    4.0    4.0    6.0    4.0    2.0    2.0    4.0    0.0  ...   \n",
       "1    7.0    4.0    2.0    2.0    0.0    4.0    1.0    0.0    2.0    2.0  ...   \n",
       "2    4.0    4.0    4.0    7.0    2.0    1.0    1.0    1.0    0.0    1.0  ...   \n",
       "3   10.0    2.0    2.0    0.0    3.0    2.0    4.0    2.0    0.0    1.0  ...   \n",
       "\n",
       "   89513  89514  89515  89516  89517  89518  89519  89520  89521  89522  \n",
       "0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "1    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "2    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "3    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0    0.0  \n",
       "\n",
       "[4 rows x 89523 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prints first four rows of Xdata\n",
    "Xdata.head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Citation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the reference for the paper the dataset is from, as requested by the authors.\n",
    "\n",
    " - Author: *Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher*\n",
    " - Title: *Learning Word Vectors for Sentiment Analysis*\n",
    " - Booktitle: *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*\n",
    " - Month: *June*\n",
    " - Year: *2011*\n",
    " - Address: *Portland, Oregon, USA*\n",
    " - Publisher: *Association for Computational Linguistics*\n",
    " - Pages: *142--150*\n",
    " - URL: *http://www.aclweb.org/anthology/P11-1015*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Training and Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLPClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MLPClassifier is a supervised machine learning function that uses a Multilevel Perceptron (MLP) model to make binary classifications of input data. MLP models work by passing data through a series of hidden, weighted \"neuron\" layers which are then used as the input for an activation function that classifies the data. This is very similar to the Perceptron model, with the key difference being the addition of hidden neuron layers. A visual representation of the MLP model can be found below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://t1.daumcdn.net/cfile/tistory/2115D1495530A29336\" width=600px>\n",
    "<p style=\"text-align: right;\">MLP Visualization is from: https://goodtogreate.tistory.com/262 </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to optimize the accuracy of the MLPClassifier, various inputs for the model parameters of `hidden_layer_sizes`, `activation`, and `solver` were placed into a Grid Search. `hidden_layer_sizes` determines the number of hidden layers present and the number of neurons in each layer, `activation` determines which activation function is being used, and `solver` determines which function is being used to determine the weight of each neuron. The code for determining the optimal combination of parameters, as well as the output, can be found in the cell below. It is placed in a markdown cell, as the code took just over 12 hours to run. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Input\n",
    "```\n",
    "# the grid of parameters being searched through to find the best combination\n",
    "param_grid = {'hidden_layer_sizes': [(20,), (20, 20,), (20, 20, 20), (50,), (50, 50,), (50, 50, 50)],\n",
    "              'activation': ['relu', 'tanh'],\n",
    "              'solver': ['lbfgs','adam']}\n",
    "\n",
    "# creates a GridSearch object that is then fitted to determine the best combination\n",
    "mlp = GridSearchCV(MLPClassifier(random_state = 22), param_grid)\n",
    "\n",
    "# fits all combinations and returns the model with the best combination\n",
    "mlp = mlp.fit(Xdata, ydata.values.reshape(1,-1)[0])\n",
    "\n",
    "print(\"Best estimator found by grid search:\")\n",
    "print(mlp.best_estimator_)\n",
    "print(\"Best parameters found by grid search:\")\n",
    "print(mlp.best_params_)\n",
    "```\n",
    "\n",
    "#### Output\n",
    "```\n",
    "Best estimator found by grid search:\n",
    "MLPClassifier(activation = 'relu', hidden_layer_sizes = (50, 50), solver = 'adam')\n",
    "Best parameters found by grid search:\n",
    "{'activation': 'relu', 'hidden_layer_sizes': (50, 50), 'solver': 'adam'}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime:  415.3508803844452\n"
     ]
    }
   ],
   "source": [
    "# trains the model with the best parameters found from the grid search above\n",
    "start = time.time()\n",
    "best_mlp = MLPClassifier(activation = 'relu', hidden_layer_sizes = (50, 50),\n",
    "                         solver = 'adam', random_state = 22)\n",
    "best_mlp.fit(Xdata, ydata[0].values)\n",
    "end = time.time()\n",
    "print(\"Runtime: \", end - start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime:  2.329716682434082\n"
     ]
    }
   ],
   "source": [
    "# generates predictions for every sample of the test data\n",
    "start = time.time()\n",
    "model_test = best_mlp.predict(Xtest)\n",
    "end = time.time()\n",
    "print(\"Runtime: \", end - start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.85      0.87      0.86     12500\n",
      "         1.0       0.87      0.85      0.86     12500\n",
      "\n",
      "    accuracy                           0.86     25000\n",
      "   macro avg       0.86      0.86      0.86     25000\n",
      "weighted avg       0.86      0.86      0.86     25000\n",
      "\n",
      "[[10889  1611]\n",
      " [ 1935 10565]]\n"
     ]
    }
   ],
   "source": [
    "# prints a classification report and confusion matrix to analyze results\n",
    "print(classification_report(ytest, model_test))\n",
    "print(confusion_matrix(ytest, model_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pros\n",
    "- High accuracy, with a fairly even recall and precision on errors\n",
    "- Fast prediction time -- once model is trained, predictions take seconds\n",
    "\n",
    "#### Cons\n",
    "- Long fit time -- back propogation through multiple layers takes time\n",
    "- Black box -- Not at all clear how decisions are being made in the model\n",
    "\n",
    "#### Sources\n",
    "- https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier\n",
    "- https://goodtogreate.tistory.com/262"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perceptron is a single layer neural network that takes in all of the input variables and then multiplies them by their weights, it then adds all these values together and gets a value between 0 and 1 or -1 and 1 depending on what you want to use as your labels.\n",
    "\n",
    "The two parameters that I choose to modify were alpha and tol. Alpha is the constant that multiplies the regularization term if regularization is used. And tol is The stopping criterion. If it is not None, the iterations will stop when (loss > previous_loss - tol)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.simplilearn.com/ice9/free_resources_article_thumb/general-diagram-of-perceptron-for-supervised-learning.jpg\" width=600px>\n",
    "<p style=\"text-align: right;\">This image is from: https://www.simplilearn.com/what-is-perceptron-tutorial </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Perceptron()"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Initalize a perceptron object with the default parameters\n",
    "clf = Perceptron(tol=1e-3, random_state=0)\n",
    "clf.fit(Xdata,ydata.values.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use the modle to predict values\n",
    "pred_labels = clf.predict(Xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.83      0.89      0.85     12500\n",
      "         1.0       0.88      0.81      0.84     12500\n",
      "\n",
      "    accuracy                           0.85     25000\n",
      "   macro avg       0.85      0.85      0.85     25000\n",
      "weighted avg       0.85      0.85      0.85     25000\n",
      "\n",
      "[[11078  1422]\n",
      " [ 2336 10164]]\n"
     ]
    }
   ],
   "source": [
    "#Printing the results to check how good the model did\n",
    "print(classification_report(ytest, pred_labels))\n",
    "print(confusion_matrix(ytest, pred_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#running this Grid Search worked fine but only gave the default parameters as the best ones\n",
    "param_grid = {'tol': [1e-3, 5e-3],\n",
    "              'alpha': [0.0001, 0.0005]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Grid Search crashed with this many parameters\n",
    "param_grid = {'tol': [1e-3, 5e-3, 1e-4, 5e-4, 1e-5],\n",
    "              'alpha': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1]}\n",
    "clf = GridSearchCV(Perceptron(), param_grid)\n",
    "clf = clf.fit(Xtraindata,ytrain.values.ravel())\n",
    "\n",
    "\n",
    "print(\"Best parameters found by grid search:\")\n",
    "print(clf.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the grid search that did not crash, which only had the two options for each parameter, the grid search returned that the best parameters were the default parameters, with the default parameters the accuracy of the model was 85% which is not too bad for the simplest neural network model.\n",
    "\n",
    "For this model the pro is also the con because it is such a simple model, depending on the situation you may want a simple model but there are more complex models that could give you better results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creates a linear model of SGD using training values\n",
    "model = linear_model.SGDClassifier(random_state=0)\n",
    "fit = model.fit(X_train,y_train.values.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating predictor values using the test values given in the data\n",
    "predict_vectors = X_test\n",
    "true_labels = y_test\n",
    "\n",
    "pred_labels = fit.predict(predict_vectors)\n",
    "#Printing the classification report and confusion matrix of the tested model\n",
    "print(classification_report(true_labels, pred_labels))\n",
    "print(confusion_matrix(true_labels, pred_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output:\n",
    "```\n",
    "precision    recall  f1-score   support\n",
    "\n",
    "         0.0       0.81      0.91      0.86     12500\n",
    "         1.0       0.90      0.78      0.84     12500\n",
    "\n",
    "    accuracy                           0.85     25000\n",
    "   macro avg       0.85      0.85      0.85     25000\n",
    "weighted avg       0.85      0.85      0.85     25000\n",
    "\n",
    "[[11424  1076]\n",
    " [ 2716  9784]]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "\n",
    "tmp_vectors = X_test\n",
    "tmp_labels = y_test\n",
    "#Find the best parameters using the selected parameters\n",
    "#Tuning to find the hyperparamters given for SGD\n",
    "print(\"Fitting the classifier to the training set\")\n",
    "param_grid = {'tol': [1e-4, 1e-3, 0.01, 0.1],\n",
    "              'alpha': [0.00001, 0.0001, 0.001, 0.01],\n",
    "              'learning_rate': ['constant','optimal']}\n",
    "\n",
    "clf = GridSearchCV(SGDClassifier(random_state=1,eta0=0.1), param_grid)\n",
    "clf = clf.fit(tmp_vectors, tmp_labels.values.ravel())\n",
    "print(\"Best estimator found by grid search:\")\n",
    "print(clf.best_estimator_)\n",
    "print(\"Best parameters found by grid search:\")\n",
    "print(clf.best_params_)\n",
    "\n",
    "end = time.time()\n",
    "print(\"Runtime\",end - start)\n",
    "\n",
    "#Runtime 11723.690677165985"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output:\n",
    "```\n",
    "Fitting the classifier to the training set\n",
    "Best estimator found by grid search:\n",
    "SGDClassifier(alpha=0.01, eta0=0.1, random_state=1, tol=0.0001)\n",
    "Best parameters found by grid search:\n",
    "{'alpha': 0.01, 'learning_rate': 'optimal', 'tol': 0.0001}\n",
    "Runtime 11723.690677165985\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is model above is using a Stochastic Gradient Descent model which uses a iterative algorithm, that starts from a random point on a function and travels down its slope in steps until it reaches the lowest point of that function.\n",
    "So it basically first:\n",
    "\n",
    "1. Finds the slope of the objective function with respect to each parameter/feature. In other words, compute the gradient of the function.\n",
    "2. Picks a random initial value for the parameters. \n",
    "3. Update the gradient function by plugging in the parameter values.\n",
    "4. Calculate the step sizes for each feature as : step size = gradient * learning rate.\n",
    "5. Calculate the new parameters as : new params = old params -step size\n",
    "6. Repeat steps 3 to 5 until gradient is almost 0.\n",
    "\n",
    "Image below demonstrates this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.researchgate.net/profile/Bhaskar-Ghosh/publication/344544069/figure/fig4/AS:944366457729024@1602165917164/Stochastic-Gradient-Descent.ppm\" width=600px>\n",
    "<p style=\"text-align: right;\">This image is from: https://www.researchgate.net/figure/Stochastic-Gradient-Descent_fig4_344544069 </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The grid search used the same parameters that is used from the beginning which means that default mode of gradient algorithm was already optimized as it was defaulted to optimized the most. This made the accuracy of the model to be around 85% which is high end of the spectrum of accuracys.\n",
    "\n",
    "Its pros can contribute to the fact that it is computationally fast and provides faster converging as it causes updates to the parameters more frequently. Disadvantages could be that the frequent updates are computationally expensive in this model which makes the model spend all resources on one sample/training.\n",
    "\n",
    "Citations/Websites:\n",
    "\n",
    "https://towardsdatascience.com/stochastic-gradient-descent-clearly-explained-53d239905d31\n",
    "\n",
    "https://www.researchgate.net/figure/Stochastic-Gradient-Descent_fig4_344544069\n",
    "\n",
    "https://scikit-learn.org/stable/modules/sgd.html#classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree\n",
    "\n",
    "A decision tree is a supervised learning model that predicts the class of an input for a set of binary decision rules. Each node in the tree contains one such a rule - for example, if a movie review contains the word 'great' more than 5 times, branch left, otherwise branch right. A decision tree is built by chaining these rules together into a branching structure. The leaf nodes of the tree contain the decision of how to classify the input, such as \"positive\" or \"negative\".\n",
    "\n",
    "The parameters tested for this decision tree are `criterion`, `splitter`, `max_depth`, and `min_samples_split`. `criterion` and `splitter` are used to specify how to make training decisions at each node. `max_depth` and `min_samples_split` are used to limit overfitting. A Grid Search was run to determine the combination of parameters that yields the best peformance, the code for which and resulting output are shown below. (Note: this is included as markdown instead of a code block because it took upwards of 10 hours to run.)\n",
    "\n",
    "```\n",
    "start = time.time()\n",
    "\n",
    "param_grid = {'criterion': ['gini', 'entropy'],\n",
    "              'splitter': ['best', 'random'],\n",
    "              'max_depth': [None, 100, 1000, 10000],\n",
    "              'min_samples_split': [2, 10, 100, 1000]}\n",
    "\n",
    "# make a classifier by searching over a classifier and the parameter grid\n",
    "tree = GridSearchCV(DecisionTreeClassifier(), param_grid)\n",
    "\n",
    "# we have a \"good\" classifier (according to GridSearchCV), how's it look\n",
    "tree = tree.fit(Xdata, ydata)\n",
    "\n",
    "print(\"Best estimator found by grid search:\")\n",
    "print(tree.best_estimator_)\n",
    "print(\"Best parameters found by grid search:\")\n",
    "print(tree.best_params_)\n",
    "\n",
    "end = time.time()\n",
    "print(\"Runtime\",end - start)\n",
    "```\n",
    "\n",
    "<br><br>\n",
    "\n",
    "Output:\n",
    "\n",
    "```\n",
    "Best estimator found by grid search:\n",
    "DecisionTreeClassifier(criterion='entropy', max_depth=100,\n",
    "                       min_samples_split=1000, splitter='random')\n",
    "Best parameters found by grid search:\n",
    "{'criterion': 'entropy', 'max_depth': 100, 'min_samples_split': 1000, 'splitter': 'random'}\n",
    "Runtime 46619.721442461014\n",
    "```\n",
    "\n",
    "<br><br>\n",
    "\n",
    "Below, a decision tree is trained using the best parameters selected by the grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime 157.15520572662354\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "\n",
    "# train the decision tree model on the training data\n",
    "tree = DecisionTreeClassifier(criterion=\"entropy\", max_depth=100, min_samples_split=1000,\n",
    "                              splitter=\"random\")\n",
    "tree.fit(Xdata, ydata)\n",
    "\n",
    "end = time.time()\n",
    "print(\"Runtime\",end - start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime 3.3699564933776855\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "\n",
    "# use the trained models to predict the values for the test data\n",
    "pred_labels = tree.predict(Xtest)\n",
    "\n",
    "end = time.time()\n",
    "print(\"Runtime\",end - start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[9707 2793]\n",
      " [3852 8648]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.72      0.78      0.75     12500\n",
      "         1.0       0.76      0.69      0.72     12500\n",
      "\n",
      "    accuracy                           0.73     25000\n",
      "   macro avg       0.74      0.73      0.73     25000\n",
      "weighted avg       0.74      0.73      0.73     25000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# compare the predictions to the actual test labels to evaluate performance\n",
    "print(confusion_matrix(ytest, pred_labels,))\n",
    "print(classification_report(ytest, pred_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV8AAADpCAYAAACDUTRdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmy0lEQVR4nO2dS3Ab59Wm3wZBdFOCSEKmKFwJSGbRNF2RL6KupCQu4qosUlllIe9m9ddstJyazWym/uXMYsr/Ymq2sRM7TiJX5VqVy8SUSZXKiW1VKfHICg2CFEgTDC/gRQQIguxZkF+n2ewGupsgru9T1WWTwOnvAy0df/3yPedIqqqCEEJIdfHUegOEENKKMPkSQkgNYPIlhJAawORLCCE1gMmXEEJqAJMvIaQp6OjoWJAkSXVydXR0LNRqvxKtZoSQZkCSJNVpPpMkCaqqSie0pZJ4a7EoIYScFJOTkwCAfD6PwcFBLC0t4fTp05ifn4eqqlAUBdeuXavxLnnyJYQ0CeLku7q6iidPnkBVVWxsbODMmTPo7u6GoijIZrN44403IMuyiKnZyZeaLyGkalRSl5UkqUOSpNclSborSdJ/F9//+9//jn/+85+QJAmdnZ0YHBxET08PFhcXkc/n8eWXXxrv858kSbouSVJXpfZuR0vmyZcQUjX0uuzk5CS8Xi+KxSLOnDkDVVUhyzLW1taQz+dx6dIlBAKBQ6dTSZL+DcB/AJgDEALwDYD/B+ApgP/mRvMF8GMAgwfXxsH9ggD+t6qq/6Hf+8TEBLzefbU2Go0ekjRkWcbe3h5u3Lhh60TN5EsIqRr65Lu6uorTp0/jxYsXCAQCpWL0yfcygH8D8D8BTKuqWtTfO5VKIZ1Ow+/3o6enB1NTU1AUBe3t7Xj55ZeRSqWQz+ehqipisRii0aj+3h4AEQCvAvivAH6lqur/0t9/ZWUFgUAAhUKh5L6ZfAkhdYU++T548AD6/NPd3Q1VVbG9vQ0ACAaDiMfjtnXZjo6OhXw+f97JfhRFyeRyuaCTvVvtW5IkKIqCV155xdaeqfkSQmxxHL1WkqSIJElJca+JiQlNl5UkSdNlRQIDgHg8rq0tSdKEJEneUnsplXgVRcmoqioZr1KJ17hGuX1ns1lt73bgyZcQYgtx8pucnISiKOjs7MSpU6dKWrnECVCSpNcA/BpAwqUuuwRgUFXVZeNeAoEAuru7sbu7C5/Ph3w+j4WF/d93GfdxnM87PDwMu3s/2DMA69M1ky8hxBZ6K9fp06fh8/mwurpqW68V9/jkk0+QSCQOabMbGxtYW1uDx+PB4OAgkskk8vk8CoUCbt++fSRx6veiKAp2dnbQ2dlpex9OP6+iKDh16hSMmvLi4iIKhQIURUGxWITX60VPTw+i0SgePXqE3t5eXLx40XRtFlkQQkyx0lCFh1ag12rz+TwSicQhyUCPoiiZsbExR7qsz+eDeOx3updIJIL+/n4AsLyH/mRq9pnFGqFQCIlEwtaeo9Eorl69eugEbITJlxBiSj6fP69PbCKReDweZDIZnDt3DsD+L8aKxSJmZ2dx6tQpLC8vWyZfu7/c0q2pil/AGb5fci+ZTAb5fB7t7e1ajNVTviRJWrLVf2bjGh9++CEA4JVXXkGxWBROCScf5/C6lB0IIWYYPbmjo6OOk41IYE5cBcY9TExMoK2tDW1tbQiHw1haWsIbb7zhaC+SJOHx48fY29tDoVDQSo/Pnz9vtLKpExMTkGUZV65cKbmGJEkwk1DS6TT6+vowNTWFixcvHrKz6eHJlxBSlmw2CwCYmZk54qMFAK/Xi5deekl73C8WixgZGQEAYcNyJDUY1/b7/QCAL7/8ErFYDAAwPj5+JPEtLi5ifX0dXq8XZ86cQbFYxOnTp7V76f5ngK2trbLrma0hvMPhcBhjY2Nl968oSsbs+0y+hLQATj2w+oQxMTGBfD7vSPMUGOxirjTXrq4uTVro7OxEMBhELBazlfgE58+f1ySJbDaLzc1NeDz/ctrq9ybWC4VCjtaIx+NIpVLifiyyIIQctk319PRAURSsra1BVVXkcjntRNjX14dQKKR9XS4/SJJ0xAGQTqehKAqCwSCePn2K/v5+RKNRPH78GKdPn8bW1ha2t7ext7eHa9eumT72F4v7hWtjY2OO7F0TExPw+Xy4evWqo7iHDx8iEAjg1VdfLaUNm8oMc3NzyOVySCQSWFhYQDQatZQaDt2PyZeQ5sdoE7NT0gvANNlsbW1hfn4ePT09+N73vod0Ol12/b6+PszMzFiupU++KysrmpXNKrkD+79cS6fTKBaLuHPnDiRJgoiVZflInLCznTp1Cj6fD5ubm+jp6UEikYCqqigUCqZxy8vL2NnZwQ9+8APMz8/b+nnb0bgpOxDSQpSyZgH7j+dCWnBjCxNJx6wUd3x8XFuvo6MDW1tbePPNN0vu0YnUEY/HXdnC4vG4tk8ncZFIBLOzs2hra3PVlpLlxYTUOU7LevXtDCVJ+r+SJH0jvvZ4PFhaWtLKY4PBIM6dOwe/3498Po/19XVt3Xw+n9aX4lr94khPPp8/X64UV1Sg6U/e+hJe/R4//PBDLCwsIJ1O4/HjxwD2pRDjBQCpVEqL/fDDDzE+Pq7Fuo17+PChZWw6nT6kGzuFsgMhdY5er1VVFe3t7UfaGerLeg2P8f8DwBqAf3dqzQLwf1RV/c9m+8jlcmhvb0cgENBaQb799tuHJAinWq0kSRgZGSlr7zKrMnvrrbdc2cLseHXN1lxZWdGq3uzou2ZQdiCkQRAWqN3dXc1ytbm5CVmWUSgUEAwelRhVVf0vACBJ0r8bbWLpdBr5fB4ejwddXV3Y2dlBsVjU7qNPvGb72NjYALAvXeTzeaTTaS2RSZJkadMC9vXabDar/WLNrr0rGo1aygLl4qycC2Zxy8vL2ucLh8MlpQg7TwSmmB2nefHiVblLUZQFAKrdS1GUBRELYHj/r6l9Du7TbtjDrps9GPdebt1PP/1U/fnPf66GQiHba8myrN0jHo9XJc7NPqPRqDoxMaF+9tln4mdxrD8XlB0IOWH0lWI23w91vxNYCMA8sH9ImpiYOFRKK8pck8mk1s1L/NYf+43Af+BmD0b3gYg7+L7lPqLR6JHHc/1Jd3Bw8Mjrs7OzuHnzJj799FPTe4pqNmPc3Nwcrl27ZhmXTCZx+/btI3GZTAaXL1+2rEy7fv266Wvicwg72fXr17WfkVuYfAk5YfTeVa/Xi0QiYasN48G/KwByVknN6/Vq1+bmJl555RWEQiEAkFVVLRj3IEkSdnZ20N/ff2gPkiTB6/UeGYEj4vx+v2kSXF5exurqKjwej+g+pn1uY8czkbyNr5tpqlNTU/D5fNojv904oTl3d3djcHDQMs6Y9+y8ZvhveuzkS82XkCowNDSEJ0+eoFgs4ssvv8SZM2cAAJ2dnZrNa2pqSuvAJVBVNd/R0ZFJJBKOqtNyuVzB+H2zMl2hGXu9XnR1dZneT2iz5WxYsiyX7OJl9bosy67ue5w4q32W+wwC1zqvDiZfQiqMWYmsmKgrSmTF4/Hs7KzWD9aYeAVuGtLYLdMVj/b6xAyYl9sep6uXJEnI5/PaadpKLnj+/Dlu3Lhhel87ske5/ej3UWsoOxBSYfSWLK/XK/RBJ/Havx+3G1ixWERHRweuXbvmKFGK2FLlvVb2rampKe17xWJRKy/WU2ovbm1hVnHJZBJ+vx/r6+vi89RF8uXJl5ATYmhoSOuoZWbzAoBCoYC9vT0A+522RCOaZDKJCxcuHKsb2NDQkDZp12wP6+vrWF9f16xi6+vr6OvrK7v35eVl7O7uIhKJ2G48I8syfv/730NVVYyNjZnau4T1zaktLJ1Ow+fzlYwTVEIuqBRMvoSY4KYLmPGE6qbUFdj3lV64cEH72qobmHFd454DgYCrslm3ZbqRSAQfffQRbt68aTo+SDQld9ItTJ+033nnHdtxkUgEf/rTn2xPEq4FlB0IMcGtPUwfexxt0nBfR+vq45zuoRK6qnFfxr05+YySJFnaydzupV7gyZeQEuhLaY32LEVRkMvlcOfOHdPY0dFRy/saH+Xn5uagqiq6u7u1ybvivs+fP4fX68WzZ8+00eSi9aPVfu3swewR3s3ehd1MeGDz+bytOH0psvE10YHMzWcQscFgEHNzc5axtYbJl5ASWNmzRJWS0EiNmCWFhYUFbG5uIhKJ2H6UDwaDSCaT6O7uRldXl2ZLEzpuqf2a7SGVSmmnRie66vT0NCRJsr13o7aqKIqlXa6cZcxqeoYdjffg/otl31QDmHwJOcCJPevFixcA9oscjDhtxSjL8t729rbWHsvssV+sK3oV7OzsaPGVmMIQDofx7rvv4t69e47jfvKTn2iVdVaP96UcGwdd2LSfl15muHv3ru3/UYXDYfzyl7/E5cuX61Zq0EPNl5AD9PYsJxMUDmJd/2UX6549exZDQ0OO13348CF6enowMDBQVh82VoR98803uHPnTknN1ayqbW1tDb29vQCApaUljIyMuP4ZiM/f09NzpCrNuBczK1k2m8XKygp6e3uRzWaRz+frylJmBU++hOgQ9izA/PF7fn4eW1tb6O3t1QY1is5c4gTqxptbbt319XUsLi5qlWhra2vaqfvGjRuWdrLp6Wm0tbXh7NmzJZ0Lxjgx56xcRy89x7Fx6a1tZp8/mUw6Os3Xk6XMkuN25uHFq1kuHHTtGh8fd9TtCoB6/vx59fPPP1fV/Rup1Vo3Foup4+Pj6ieffOIoNh6Pu4qLRCLqs2fPXH3OUp/fzV5isZg6MTGhFgqFiu2lmhdlB9KymGm8qmpuswoGg7D7d6Xc43el1q03O5lb3JYbn8ReqgmTL2lZ9GXAgUAAr732WslEZ1VKqygKEokEnj59Cq/Xi9HR0ZKJQK/xLi4uuirh3djYKDlpt1SsnTJdqyGSfr9fa7RT7nNaYfU/Hyf7SSaT6O/vx7Nnz8r+sq9eoeZLWp6hoSHNP2vUPhcXF1EoFByV0trRG4XG+/LLL5uuKyxh4XC45LpmNiy/3w+/319SI7Xyx2YyGdd2Mrvk8/nz4+PjWsIdGxsz9QB7PB7Nz1xuPw2h8Rqpte7Bi1elLqcTI6DTWp3qjTjQP1VVtXxdP5HCbCKE23VDoVBV48LhcEV1VfH53UyTiEQi6qNHj7Sfe63/zB3nouxAmgaXJcGm2mc+nz/SxLvcfSy+X7HSXxFbbuqD1V6s4sQ0iXKftZIab6m1ypVUV3o/tYKyA2kqJicncfbsWczNzeHVV189VA4sy7I2akcgHr8lSTr0yG983ezxPJFIYHl5GcB+I3QxlQKAacnx5OSkZkszW/cf//iH1k7STE4Q0kgsFjsUt7S0hM3NTXR3d1vu2SxudXUVmUxG8+sa10wmkzh37hwymYxlCbVbzPYoSqzN9iLkn+7ubmSz2SMtKhsRnnxJ0yBJkrqysqJ15NrY2MCZM2fQ3d0NVVWxvb2NfD6PixcvaonI+OdfPzImkUhgZmam7LqhUAi//e1v0dHRgb29PZw6dQrxePzIyVfszayowcm6+ljjiJtSsW7jBG57Cxsp1zFOlmWtjLoa+6kVPPmShsXsL3EgEIBoXWhVmiv+YtsdGSMQSaHUY38+n0dHR4cWoy/9FdMsQqGQo3Xj8Tjef/99ZDIZ3Lt3z3bsceLE6f8kHu0bOWFWEp58ScOit4q5LQkWWJ2i9Gt0dnbi0qVLttcQEyHsTLOopC3sODY0MYViaWkJPT09QvduWF21nuHJlzQ8paYuCK3Q7/dje3sbqqqiWCxqEyNmZ2eFBFGyEU6pNebm5pDL5RAMBpHNZuHxeBAOh8vGiZN4sVgsaWWzarfY3d1dsvzX7bQIPQ1p4WoQmHxJw3OciRHz8/OIxWInsobbiRDAvo78wQcf4J133rEd53bqQywWw/379zE8PNzwDoJGgrIDqWvK/XLGrWXLiD7pGNd0s0a92clKxVn9HMjJwuRL6hq95qqqKtrb2xGNRm0lE6sWil1dXQgGg1o5sLEdoij/lSQJIyMjjtdIpVKi9LZknJnmmk6nT1QfNoubnZ2Fx+NBNBoV92DyrQKUHUhDICY07O7uahMlAHNdU0w6cNIOUZZl0/Ws1lhYWMDW1hai0agjzVWU/zotG06n0+jr68Pa2lrJWLM4UTBSbk2AGm9VqXWJHS9exsusFNeMeDzuqEw2FotZ3sv4XrdrxOPxY+/NSaxYz2lcKBRSd3d3tc9e6//mrXhRdiB1h5NSXOGt7e3txYULFw4VTpTTQPP5PMLh8JGCCzvaqVFqmJqawtjYWNm4UnJBKX3YbZzZPvv7+7G4uIizZ88eKQYh1YOyA6lLRJkwYF6Ku7KygvX1dRSLRSiKgpWVFczOzgIA/vKXv2iTb40ltc+ePYPX60U4HMbCwoJmNRNrigRsjPv666/R3t6Os2fPlpQarEp/FUWxNbRSH/v06VMkEglXcT6fr+Q+9VBqqA1MvqQuGRoawpMnTxwPgoxGo9qo92Aw6MgWJnReJ7YwWZbxi1/8An6/35EtDPiXnezu3bu2P6NxPad2sitXrvCUWydQdiA1x8lkh5Owd52ULawe7WSUGOoHJl9Sc/R2slOnTuGtt94qm0AqabVyG+dm8q8YhFmuFLrSFjZhJzuwsTH51gGUHUjdUKoUVz+9t5Rlysre5fV6bWmnbkpxzexdwP6jfikZwqzceGBgQLOT2Z00PDU1BZ/Ph3A4XNZOJsvyouWLpLrU2m7BqzWvWk92kGX52Os5idWvV61Jw+FwWM3n87SS1elF2YHUhEad7HASceX26fZnY7gHpYY6g8mX1AS9zmtHxzTqn+l0Gnt7e2XH31jprkNDQ67WtFP6a+UBLhdX6ZLhQqGA/v5+lgzXKdR8SU3JZrMAzDXX6elp7O7ulp2ma9VGMhgMlrSNmY3N2dvbQzAYdNyqUZQ0l9J5reLeeustx3q0WM9Oa0j6eOuUWusevFrjMpssrKr7E2zPnz/vSHONxWKuJt/G43FXcbIsN0RcLBZT//rXv1LjbZCLsgOpCk6nTlR6QoPbx/dyax63O1klJQrayRoLyg6kqpSb7OD3+7WeC04e35eXl7G6uurq8b1c6a/VXhVFQSwWcxS3ubmJzs5OV3YyALa6k9FO1hjw5Euqgjj5PnjwAKq6P2nh22+/tR2vn+xgN8442cHJevF4HD/60Y8cxerXA+BoTTfrAfttM+fm5uhoaECYfMmJUYmJEMDJ2MlqUTK8ublp6uyw07mNZcPNB5MvOTHERAi703vN9E+/34/h4eGK67ylpveW0lyt4paXl/H666+X3KfV57Tr1TXGcQJFY0PNl5woQ0NDCAQCAMw112w2i5WVFcd2sm+++QaqqpbVQI1xCwsL2NzcLKvxWpUbl9NbrXTlRCKBXC5nqfOWspOJz1jq50M7WQNSa7sFr+a5rOxk4+Pjjkt4Y7GYq5Jat6W4bsuNqx0Xi8XUzz77TN3b26OlrMEvyg6kYuhLhg++hqpWvsWjlQbaLK0hnz9/jhs3bliuafi8lBoaFMoOpOLoJ0LMzMxAkiT09/drvtlUKgVFUQCYP27rv2+cJqGqqvb4bZQUrOLE9Aqr9dbX1wEcnUKRSqW0z2QWt7GxYRkXDoe1oZxmlXRmcU+fPoWiKAiFQqZxYgTQ1NQU7ty5c1L/+UiV4MmXVAxx8v3Nb34Dv99/5BdXq6urmv4L7GuZMzMzpvcy/rm0G1sqrtR6xthKxJWLdRsnUBQlk8vlgiXfROoWnnzJsTCbQtHV1YVMJoNQKARJsv9UHA6H8e677+LevXu242RZxh//+EdkMhnXcefOncPdu3dtxVY7Dtj3AKdSKcoMTQZPvuRYCDuZoiiuLWF2S3GrPb3CaO1KJpO4fft2VfcpnBLpdFp4hJl8mwSefMmx0ZcMW2mjmUymrE3LbHrF5uYment7HdnJ9JMrnMYpioLu7u6S3dCsbGEDAwOuypuFV5fdyVqMWtsteDX2BcD1RIhGspO57U7m1k72+PFj2sma/KLsQBxhpvFaWaZKPaZX24YmYt3aydy+5mY9k/tTamhCmHyJI/StIVVVxa1bt0omkWprtSLWaatGq/XE1Auz+w0MDCAcDlu2dyw1ZaOUxtvf34/l5WW89NJLLBtuYqj5Eldks1n4/X4A5tN7FUXRPKtONND19XWsr6871k4XFhZQLBbLTqGwamNZaj1Zli3vJ8uyq1aUdjRegDpvU1Nr3YNX/V9mk4YF8XjckQZaqzgnsUJz3d3ddaS5ip+NG304HA6rjx49UtX9G9lek1fjXpQdSFmcTho2e5yen5/H1atXy5bUWkkN5eKsbGHlSn/N4vr7+5FMJjEyMuJIc5UkSbWSEhRFwfXr14+sNz09DVVV0dvbi/X1dYTDYUoNLQJlB2KLyclJ5HI5AOYlvE+fPgVQfqCjsaQ2k8lge3sbgUCgpERhjBPTi0WclSxgVcLb3d1dtlMY4OyxX1GUzNjY2Hmr10vJF27XJI0Lky+xhdB4SyVIM+LxON5//32tAs1O8gGAvr4+vPfee1BVFXfv3j3xuHA4jJmZGXi9XtcOA5b6EidQdiBHqLSdzAqr10/KhkZ7F6knmHzJEUTJsMfjgSzLuHz5ctkkamXtMtNAU6mUNsHYGCesXeXWq3QJb39/P1KplGOdlxC3MPmSI0iSpK6srCAQCKBQKECWZdPkKrTT7373u0in00fuI8sytre3Ldcp9bpZklxcXEShUMAPf/hD0/Ws4oT17fvf/75lnB52CyPVgMmXADAfdulm0nAsFsMHH3yAq1evwufzOXIKCGnj3r17jiYU67uaOdnnT3/6U3i9Xly9epUnXVJ1mHwJAGd2MjvdvXT3sZ18j6MPO4k5zj4JqRR0OxCNycnJQz1mR0dHTd8XiUQAmFe2DQ0N4YsvvnA1acF4v8XFRayvr2NwcND0dTERwmwf+XweQOmBlvqJGoRUGyZfoqEvGbZKWkLnjUajFffIJhIJxx7Z45T+6t63aHefhFQKJt8WpdwECide3lAohOnpaSiK4vrxnb/gIq0GNd8WRd+drLOzE5cuXXJl0xINz7e2tkTXMGqnhNiAJ98WRz+FwqwDFwCtUxgnLRBSOZh8W5wnT55AVdWS/RGMxGIxvPfee7h16xba2tp42iXEBZQdWggzLy9LcQmpDUy+LYQoG/b5fKKwoNR7S5bibmxsoKOjA4lEgsmXEBdQdmgxyk0aFjovJy0QcrIw+TYxZnYyvcbrxE4WDoeRSqUclQwTQqyh7NDEGEt2S03TLTX113BPJl9CKgBPvi2AmDQMmE928Hq96OnpAWBeqtvX14e1tTV0dHTU8mMQ0lQw+bYA+rLheDyOeDwOAFhdXT3UgyEej1e0ZJgQYg2Tb5NRrmxY3zinFPF4HKlUCgClBkJOAmq+TYawk3m9Xly/ft3VBIpEIoHNzU1sbm4iGo1ymi4hJwBPvk2IHTuZoii2OpMBlBoIOQmYfJsQt3ayUCiEdDrNkmFCqgBlhwbHyaRh2skIqR+YfBscfWtIVVVx69YtxzrvwsICBgYG8PjxY1y8eJEaLyFVgLJDk6C3kxm9ultbW1hYWEBvby/C4TDtZITUATz5NiBm3ckEiUQCMzMztu4TDoeRTqchSRKlBkKqDJNvA+Jk0nA0GjXtTpZKpRCNRpFOpzEyMsLkS0iVoezQoOgnDQs7mb5sOJVKIZ/PszsZIXUKk2+DIjReN93J7t+/j2vXrvG0S0gNoezQADixk9mdQAHQUkZILWHybQBEyXCxWEQ4HMbAwIBlgjWzkqVSKRSLRQwODuLp06cAgDt37jD5ElJDKDs0CENDQwgEAigUCgDMWz8KjdfuIEzqvITUDibfOsUoNQQCATx48MDxpOFIJILZ2Vl4PB6edAmpIyg71ClO7WScNExIY8GTbx2jn0ABAKOjo5bvtepe1t/fr/2TEFI/MPnWMfqSYbPkqtd56eUlpLFg8q0Tyk2gcOPlPWimTpmBkDqEmm+doLeTeTwe3L59u6SdzKxkWHQnm56exs7ODoaHh5l8CalTePKtI8rZyfQTKOychGVZXjzhLRNCXMLkW0e4tZOFw2GkUin4fD6edAlpECg71BCz1pDHsZPRSkZI48DkW0P0Ou/Y2FjZCRRmOu/y8jLOnTuHZDKJ0dFRJl9CGgTKDjWm3KThdDoNj8djS+ellYyQxoHJt4qY2cncTBqOxWL48Y9/jKtXr0JRFJ52CWlAKDtUEX3J8MHXlq0hZ2dncfPmTeq8hDQpPPlWmcnJSRSLRbS1tQHYP8UaJ1B4PB6cPXsWwFG72ezsLBRFwdbWFkZGRmr5UQghx4An3yoiSZK6srKiSQ3GX7Ktrq4iEAhoX9sZhqkoSiaXywVPbNOEkBOBJ98Txqw1pMfj0cqGxRy2csTjcaRSKcoMhDQJPPmeMMJO1tnZiUuXLtFORggBwJNvVShnJ3NSNkw7GSHNAZNvFXBrJ/vDH/6AgYEBeDwennYJaTIoO1QYJ5OGaScjpHVh8q0wQuP1er3Y2NjA22+/XVbnNU4bzmaz6Ovrw1dffYVgMIhEIsHkS0iTQdnhBBgaGsKTJ0/Q3t4O4KhXd25uDrlcDsFgEOFwuGz3Muq8hDQfPPlWALPuZAI7Xl1BJBLB8+fPqfES0gIw+VYAp5OGzexkGxsb6OjoQDqdpp2MkBaAskOFGB8fx87OjvbviUTiUNnwzMwMcrkc7WSEEABMvhVDkiS0t7cjGAw6spN99NFHePPNN9mdjJAWg7KDC5zYyZaWlvDGG2/QTkYIOQSTrwv0EygURREj2ku9/4idTK/xjoyMMPkS0mJQdnCJKBn2+XwAjtrJFhYWUCwW0d3dTTsZIeQITL42MUoNomQYgKNJw9FoFM+fP+dJl5AWh7KDTYx2slIabzabtTUQk8mXkNaFJ18HPHr0CMViEQAwOjpq+p5IJKL9u1GKWFlZwYsXLxCLxaqyX0JI/cLk64BCoaCdZs1aQ87NzUFVVVs6LzVeQlobJl8LzOxk+gkU9PISQo4DNV8LhJ1sb28PPp/PlZ1samoK58+fx5kzZxCJRKjzEkI0ePItQTk72dzcHHZ3d3H27FnayQghjmDyLQHtZISQk4Kygw6jzks7GSHkpGDy1SFJkvq3v/0Nm5ubZTXeg/cf0Xn1drJoNMrkSwgxhbKDgddeew2FQgGAuZ1sYWEBm5ubtqZQUOMlhFjR0snXzE724MEDV5OG79+/jytXrvCkSwixRUvLDvqS4YOvj9UakhovIcQuLX3yBYDJyUm0t7fD693/UcRisUMTKL7++mtIkoTe3l4AR+1mU1NTkGWZJcOEEEe0fPLNZrPw+/3a1/F4HPF4HACwurqK27dvH3qNGi8hpBK0XPI16rxdXV2azBAKhSBJ9lSDeDyOVCpFqYEQ4oqW03xF2bDX67VVMmw2aZh2MkLIcWm5ky/wr7JhwNxO9s0336C9vZ2ThgkhJ0ZLJl9RNuzUTvazn/0MV69ehcfj4WmXEHIsml52cDJp+EBCKHtP6ryEkOPS9MlXaLwdHR3Y2dkpqfNaabxTU1Po7+9HMpnExYsXqfMSQo5NS8gO2WxWG/8DmHt1gX1pwY4MQZ2XEHJcmvLka5Qa9J8xkUhgZmam7D1isRh+97vfYWhoiBovIaTiNGXylSRJffjwIXp6ejAwMFC2ZNhMapiensbLL7+MZDKJ0dFRJl9CSEVpWtnhxo0bWney3d1dDA8Pa8m1WCwinU4jn8/bkhooMxBCKk3TJl92JyOE1DNNITs4sZOxOxkhpB5oiuQr7GTFYhGKorieNCy6k9FKRgg5aZpGdig3aVhovIODg2WHYVLjJYScNA2bfI1Sg5tJw0LjHR4eptRACKkqDSs76KdQHHcChbgHky8hpFo07MkXAD7//HOsrq4CAEZHR03fE4lEAJh3L+MUCkJIrWjo5PvixQu0t7cDKK/x0stLCKknGib5mtnJPB4PMpmMbY0X2Nd5P/74YwwPD1NmIITUjIbRfJ10Jzt4P+1khJC6pWFOvkB5O9n8/Dy2trZoJyOE1D0NlXzd2sk+/vhjXL58mY4GQkjdUNeyg1HnpZ2MENIs1HXylSRJffToEXZ3dzEyMuJq0jB1XkJIPVL3ssP29raWdM00XgDo6+ujnYwQ0lDU1cnXzE4m9md3AgVAOxkhpP6pq+SrLxk++JqtIQkhTUndyQ6Tk5NQVRW7u7sA9k+xkiRpUkMqldIq1wDzYZgsGSaE1Dt1l3yz2Sz8fr/2dTweRzweBwCsrq7i2rVrh16jl5cQ0ojUNPmaabxdXV2azBAKhSBJ5VWDeDyOVCoFgFIDIaQxqKnmK0qGvV4vVFXFjRs3HJcMp1Ip+Hw+bG1t4c6dO0y+hJCGoOayw9DQEAKBgDZp2Kw7mdfrRTQatVXVRqmBENII1Dz5BgKBQ5OGnXQn+/Of/4z+/n6edAkhDUfVZQejzquqKiYmJmgnI4S0FFVPvkLnlSSpbMnwwfuP6LwrKyt48eIFS4YJIQ1LTWQHvZ3MbLwPJw0TQpqdEz/5VrJk+Fe/+hW+853voK2tjaddQkhDc+LJVy8z7O3t4datW5Ylw8+fP8eNGzeOyAzZbBZ9fX344osvaCcjhDQFVZEdjHYyY8nws2fPsLe3Z0tmACg1EEIan6ok3+PYydidjBDSjJyI7ODETiakBtrJCCGtxIkkX6HzFotFjI2NubKTcQIFIaSZOTHZQUwaBsztZMlkEl6vFxcvXqSdjBDSclQk+ZrZycSkYTvjfQTUeAkhrUJFZAcnEyio8RJCSIVlBzGFAgBGR0dN3xOJRACYSxFC5yWEkGanoslXXzZsNt5HURQkEomyUoQsy4uV3BchhNQbrpNvqSkUTr28v/71r/H6669TaiCEtAyuNV9JktSvvvoKi4uLru1kqVQKbW1ttJMRQlqOYyVfVVVRKBQgyzI++eQTUw3X7/cjGo3izTffxLfffmt5P0VRMrlcLuj2gxBCSCNxLM1XXzLsxE723nvv4ebNm/D5fDztEkJaEtsnXzON9zh2MoCWMkJI62I7+YqS4b29PWxvb+Ptt992NWm4o6MDuVwOIyMjTL6EkJbFkewwNDSEJ0+eoL29HYD5pGExgSISibA1JCGEWODo5Kt/r9MpFPfv38eVK1d40iWEENhMvpIk+QBsszUkIYRUhpKyg/GXbKIkWD+FIpVKwePx4MKFCwDMK9tEa0hCCCH7lDz5SpKkCkfDvXv3Svp0AUCWZWxvb1u+Ti8vIYTsUzb5Wr1+ICGUX4BSAyGEHKGs28GqQQ5g3ZlMfC8ajZ74ByCEkEak5MnXrLBCUE5iEFBqIISQo5zIDDdCCCGl8dR6A4QQ0oow+RJCSA1g8iWEkBrA5EsIITWAyZcQQmrA/wccpAsWbpgLIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#visualize the decision tree\n",
    "plot_tree(tree)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pros\n",
    "- very quick to predict (logarithmic, just have to traverse the tree)\n",
    "- simple to understand and interpret - can visualize the tree\n",
    "    - not a black box like neural networks - very clear how decision is being made\n",
    "    \n",
    "cons\n",
    "- overfitting - can create over-complex models that do not generalise to the data well\n",
    "- lower test performance compared to the other models tested\n",
    "\n",
    "Sources\n",
    "- https://scikit-learn.org/stable/modules/tree.html#tree\n",
    "- https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Odinary Least Squares with Ridge Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Overview\n",
    "Ridge Regression is a way to ensure you do not overfit your training data - essentially, we are desensitizing (normalizing) the model to the training data. A line is fit using the least squares technique, but the coefficients are not estimated by ordinary least squares, but by an estimator called ridge estimator that is biased but has lower variance than the OLS estimator. In other words, by starting with a slightly worse fit, Ridge Regression can provide better long-term predictions with lower variance. \n",
    "\n",
    "#### How it works\n",
    "\n",
    "When Ridge Regression determines values for the parameters of its fit:\n",
    "\n",
    "__Size__ = y-axis intercept + slope x __Weight__\n",
    "\n",
    "It minimizes the __sum of the squared residuals + λ x slope^2__\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.stack.imgur.com/s71QZ.png\" width=600px>\n",
    "<p style=\"text-align: right;\">Source: https://stats.stackexchange.com/questions/402889/why-ridge-regression-only-decreases-slope-and-not-increases-it </p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The red line in the visualization above is Ordinary Least Squares minimized with the sum of the squared residuals, and __the blue line is Ridge Regression minimized with the sum of the squared residuals +  λ x slope^2__. The value for lambda reduces the slope, making the prediction for size less sensitive to weight in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RidgeClassifier()"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initalize object with the default parameters\n",
    "ols = linear_model.RidgeClassifier() \n",
    "ols.fit(Xdata, ydata[0].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Runtime:  9.361006736755371\n"
     ]
    }
   ],
   "source": [
    "# generates predictions for every sample of the test data\n",
    "start = time.time()\n",
    "model_test = ols.predict(Xtest)\n",
    "end = time.time()\n",
    "print(\"Runtime: \", end - start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.76      0.78      0.77     12500\n",
      "         1.0       0.77      0.76      0.76     12500\n",
      "\n",
      "    accuracy                           0.77     25000\n",
      "   macro avg       0.77      0.77      0.77     25000\n",
      "weighted avg       0.77      0.77      0.77     25000\n",
      "\n",
      "[[9706 2794]\n",
      " [3056 9444]]\n"
     ]
    }
   ],
   "source": [
    "# printing model performance results\n",
    "print(classification_report(ytest, model_test))\n",
    "print(confusion_matrix(ytest, model_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pros\n",
    "* Very fast prediction time\n",
    "* Simple model thats easy to understand\n",
    "\n",
    "#### Cons\n",
    "* Unable to change lambda value \n",
    "* Not the most accurate\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results\n",
    "- MLP - 86%\n",
    "- Perceptron 85%\n",
    "- SGD - 85%\n",
    "- OLS - 77%\n",
    "- Decision Tree - 73%\n",
    "\n",
    "MLP was the most accurate model at 86%. \n",
    "\n",
    "# Potential Future Improvements\n",
    "\n",
    "The accuracy of our models appears to be capped around 85%, which is most likely a function of the bag of words format of our input. We could potentially improve the performance in the future by implementing other natural language processing strategies, such as those implemented in the Natural Language Toolkit. In addition, we could apply dimensionality reduction techniques such as PCA to reduce the complexity of our input feature space.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}