text_vectorization_and_sentiment_analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Text_vectorization_and_sentiment_analysis.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1jmiKqCViBd1-CdXoMlMsXziFm7Ysp9SF",
      "authorship_tag": "ABX9TyPrKilus23NNkqwAeYy1qLx",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/Melvinchen0404/1866f59fef2a9e53bf532b744202f1bd/text_vectorization_and_sentiment_analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##NLP Technique 5: Text Vectorization\n",
        "**Text vectorization** refers to the process of converting text into **numerical representation** \\\n",
        "\\\n",
        "2 METHODS for **text vectorization** are: \\\n",
        "**METHOD 1:** The **bag-of-words (BoW)** model; \\\n",
        "**METHOD 2:** **TF-IDF** or **Term Frequency-Inverse Document Frequency** \\\n",
        "\n",
        "**Word embedding** is the representation of words for text analysis, typically in the form of **real-valued vectors** that encode the meaning of the words such that the **words that are closer in the vector space are expected to be similar in meaning** \\\n",
        "$\\therefore$ Both **METHOD 1** and **METHOD 2** are **word embedding** techniques \\\n",
        "\n",
        "SOURCES: \\\n",
        "https://sites.pitt.edu/~naraehan/presentation/Movie%20Reviews%20sentiment%20analysis%20with%20Scikit-Learn.html \\\n",
        "https://www.analyticsvidhya.com/blog/2020/02/quick-introduction-bag-of-words-bow-tf-idf/ \\\n",
        "https://towardsdatascience.com/sentiment-analysis-a-how-to-guide-with-movie-reviews-9ae335e6bcb2"
      ],
      "metadata": {
        "id": "46VE5hJyMBGX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**METHOD 1:** The **bag-of-words (BoW)** model \\\n",
        "The **bag-of-words (BoW)** model is the simplest form of text representation in numbers \\\n",
        "Each text (e.g., a sentence or a document) is represented as the **bag (multiset) of its words**\n",
        "\n",
        "**STEP 1 of METHOD 1:** Import `nltk` and `CountVectorizer` \\\n",
        "`CountVectorizer` allows us to convert a collection of text documents to a **matrix of token counts**"
      ],
      "metadata": {
        "id": "D7k77DkwNDwV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# import CountVectorizer, nltk\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "import nltk"
      ],
      "metadata": {
        "id": "BS1FmIPqNdmr"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Turn off pretty printing of jupyter notebook... it generates long lines\n",
        "%pprint"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DbnKLBX3ObYD",
        "outputId": "1e7e70a4-0f27-43dc-ce2c-c38e03bc5052"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pretty printing has been turned OFF\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 2 of METHOD 1:** Initialize the `CountVectorizer` \\\n",
        "Its default mode will remove **punctuation** (non-alphabetic characters) and **stopwords** (a set of very common words like 'the', 'a', 'and', etc) \\\n",
        "It will also convert all letters into **lowercase form** \\\n",
        "Its minimum document frequency can be set to 1 with `min_df=1`"
      ],
      "metadata": {
        "id": "mSLXmiNEO1QU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize a CountVectorizer to use NLTK's tokenizer instead of its \n",
        "#    default one (which ignores punctuation and stopwords). \n",
        "# Minimum document frequency set to 1. \n",
        "convert = CountVectorizer(min_df=1)"
      ],
      "metadata": {
        "id": "3bm73GdKO9VC"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 3 of METHOD 1:** Set the **corpus** \\\n",
        "Here, our **corpus** comprises 3 movie reviews: \\\n",
        "REVIEW 1: This movie is very scary and long. \\\n",
        "REVIEW 2: This movie is not scary and is slow. \\\n",
        "REVIEW 3: This movie is spooky and good. \\\n",
        "\n",
        "**STEP 4 of METHOD 1:** Use the `.fit_transform` method to adapt `convert` to the supplied text data (or `corpus`) and create and return a **count-vectorized output**"
      ],
      "metadata": {
        "id": "_KrhLl-WPmc7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "corpus = ['This movie is very scary and long.',\n",
        "        'This movie is not scary and is slow.',\n",
        "        'This movie is spooky and good.']"
      ],
      "metadata": {
        "id": "TXfR5k8bQQVj"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "corpus_counts = convert.fit_transform(corpus)\n",
        "\n",
        "# convert now contains vocabulary dictionary which maps unique words to index numbers\n",
        "convert.vocabulary_"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NTxq62KHQal6",
        "outputId": "0890fe7e-e203-4613-b22b-cd0eadc7962e"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'this': 9, 'movie': 4, 'is': 2, 'very': 10, 'scary': 6, 'and': 0, 'long': 3, 'not': 5, 'slow': 7, 'spooky': 8, 'good': 1}"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 5 of METHOD 1:** Use the `.shape` function to determine that we have a dimension of 3 (we have REVIEWS 1-3 in our `corpus`) by 11 (we have 11 **unique words**) \\\n",
        "**STEp 6 of METHOD 1:** Generate the **vector** for each of REVIEWS 1-3 using the `.toarray()` method \\\n",
        "Our vocabulary of **11 unique words** is as follows (in accordance with the index numbering from the previous step): \\\n",
        "'and', 'good', 'is', 'long', 'movie', 'not', 'scary', 'slow', 'spooky', 'this', 'very' \\\n",
        "If a particular **unique word** is present in a review, it will be marked with 1. Otherwise, it will be marked with 0 in the **vector** \\\n",
        "**Vector** of REVIEW 1 (This movie is very scary and long.): [1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1] \\\n",
        "**Vector** of REVIEW 2 (This movie is not scary and is slow.): [1, 0, 2, 0, 1, 1, 1, 1, 0, 1, 0] \\\n",
        "**Vector** of REVIEW 3 (This movie is spooky and good.): [1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0]  "
      ],
      "metadata": {
        "id": "4AylqTaFQ20i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# corpus_counts has a dimension of 3 (there are 3 reviews) by 11 (there are 11 unique words)\n",
        "corpus_counts.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7bbOTqlPQtJy",
        "outputId": "a34b663b-1996-40ce-9be7-0bf0ef583192"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3, 11)"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# this vector is small enough to view in a full, non-sparse form! \n",
        "corpus_counts.toarray()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZlQ3EzK_RJK8",
        "outputId": "af65cce7-4d25-435a-a37b-7ce42353f97c"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1],\n",
              "       [1, 0, 2, 0, 1, 1, 1, 1, 0, 1, 0],\n",
              "       [1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0]])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**METHOD 2:** **TF-IDF** or **Term Frequency-Inverse Document Frequency** \\\n",
        "**Term Frequent (TF)** is a measure of how frequently a term, $t$, appears in a document, $d$ \\\n",
        "$tf_{t,d} = \\frac{n_{t,d}}{\\text{Number of terms in document $d$}}$ \\\n",
        "\n",
        "Consider REVIEW 2: This movie is not scary and is slow. \\\n",
        "Number of terms in REVIEW 2 = 8 \\\n",
        "'and', 'good', 'is', 'long', 'movie', 'not', 'scary', 'slow', 'spooky', 'this', 'very'\n",
        "\n",
        "TF(and) = $\\frac{1}{8}$ \\\n",
        "TF(good) = $\\frac{0}{8} = 0$ \\\n",
        "TF(is) = $\\frac{2}{8} = \\frac{1}{4}$ \\\n",
        "TF(long) = $\\frac{0}{8} = 0$ \\\n",
        "TF(movie) = $\\frac{1}{8}$ \\\n",
        "TF(not) = $\\frac{1}{8}$ \\\n",
        "TF(scary) = $\\frac{1}{8}$ \\\n",
        "TF(slow) = $\\frac{1}{8}$ \\\n",
        "TF(spooky) = $\\frac{0}{8} = 0$ \\\n",
        "TF(this) = $\\frac{1}{8}$ \\\n",
        "TF(very) = $\\frac{0}{8} = 0$ \\\n",
        "\n",
        "**IDF (Inverse Document Frequency)** is a measure of how important a term is \\\n",
        "$idf_t = \\log{\\frac{\\text{Number of documents}}{\\text{Number of documents with the term $t$}}}$ \\\n",
        "IDF('this') = $\\log{3/3} = 0$ \\\n",
        "IDF('movie') = $\\log{3/3} = 0$ \\\n",
        "IDF('is') = $\\log{3/3} = 0$ \\\n",
        "IDF('not') = $\\log{3/1} = 0.48$ \\\n",
        "IDF('scary') = $\\log{3/2} = 0.18$ \\\n",
        "IDF('and') = $\\log{3/3} = 0$ \\\n",
        "IDF('slow') = $\\log{3/1} = 0.48$ \\\n",
        "\n",
        "NOTE: Terms like 'is', 'this', 'and', etc have little importance and their **IDF** score is reduced to 0, whereas other terms like 'long', 'scary', 'slow', etc are more important and enjoy a higher **IDF** score \\\n",
        "\n",
        "We can compute the **TF-IDF** score from the **TF** and **IDF** scores of each term \\\n",
        "$tfidf_{t,d} = tf_{t,d} \\times idf_t$ \\\n",
        "TF-IDF('this', REVIEW 2) = $\\frac{1}{8} \\times 0 = 0$ \\\n",
        "TF-IDF('movie', REVIEW 2) =  $\\frac{1}{8} \\times 0 = 0$ \\\n",
        "TF-IDF('is', REVIEW 2) = $\\frac{1}{4} \\times 0 = 0$ \\\n",
        "TF-IDF('not', REVIEW 2) = $\\frac{1}{8} \\times 0.48 = 0.06$ \\\n",
        "TF-IDF('scary', REVIEW 2) = $\\frac{1}{8} \\times 0.18 = 0.023$ \\\n",
        "TF-IDF('and', REVIEW 2) = $\\frac{1}{8} \\times 0 = 0$ \\\n",
        "TF-IDF('slow', REVIEW 2) = $\\frac{1}{8} \\times 0.48 = 0.06$ \\\n",
        "\n",
        "NOTE: **TF-IDF** gives higher values for less frequent words \\\n",
        "The **TF-IDF** score is high when both the **IDF** score (i.e.,  the term is **rare in the corpus** or all the documents combined) and the **TF** score (i.e., the term is **frequent in a single document**) \\\n",
        "\n",
        "The **bag-of-words (BoW)** model (**METHOD 1**) merely creates a set of vectors containing the **frequency count of word occurrences** in the corpus \\\n",
        "On the other hand, the **TF-IDF** model (**METHOD 2**) contains information on the more important terms and the less important ones. $\\therefore$ **METHOD 2** typically fares better in **machine learning** approaches"
      ],
      "metadata": {
        "id": "5jrlRt60TcKV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 1 of METHOD 2:** Import the `TfidfTransformer` \\\n",
        "This will allow us to **transform** a **count matrix** to a normalized **TF-IDF** representation \\\n",
        "**STEP 2 of METHOD 2:** Use the `fit_transform()` method to convert the raw **frequency counts** of words into **TF-IDF** scores \\\n",
        "NOTE: Under the **standard textbook** method: \\\n",
        "$idf_t = \\log{\\frac{\\text{Number of documents}}{\\text{Number of documents with the term $t$}}}$ \\\n",
        "Under the **Scikit-Learn (Sklearn)** method: \\\n",
        "$idf_t = \\log{\\frac{\\text{Number of documents} \\ + \\ 1}{\\text{Number of documents with the term $t$} \\ + \\ 1} + 1}$ \\\n",
        "The **Scikit-Learn (Sklearn)** method ensures that terms that occur in all documents in the training set or corpus (i.e., terms with a zero **IDF** score under the **standard textbook** approach) will not be completely ignored \\"
      ],
      "metadata": {
        "id": "4hINIzIIYZh_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert raw frequency counts into TF-IDF (Term Frequency -- Inverse Document Frequency) values\n",
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "fooTfmer = TfidfTransformer()\n",
        "\n",
        "# Again, fit and transform\n",
        "corpus_tfidf = fooTfmer.fit_transform(corpus_counts)"
      ],
      "metadata": {
        "id": "f9fpJzjBYeQf"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# TF-IDF values are returned\n",
        "# Raw counts have been normalized against document length \n",
        "# Terms that are found across many documents are weighted down ('a' vs. 'scary')\n",
        "corpus_tfidf.toarray()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FCedJ5nLZAtu",
        "outputId": "1af82744-07ed-4076-b9b8-a4d4d52171e4"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.29628336, 0.        , 0.29628336, 0.50165133, 0.29628336,\n",
              "        0.        , 0.38151877, 0.        , 0.        , 0.29628336,\n",
              "        0.50165133],\n",
              "       [0.26359985, 0.        , 0.5271997 , 0.        , 0.26359985,\n",
              "        0.44631334, 0.3394328 , 0.44631334, 0.        , 0.26359985,\n",
              "        0.        ],\n",
              "       [0.32052772, 0.54270061, 0.32052772, 0.        , 0.32052772,\n",
              "        0.        , 0.        , 0.        , 0.54270061, 0.32052772,\n",
              "        0.        ]])"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 3 of METHOD 2:** **Transform** new data or reviews into **count-vectorized** form \\\n",
        "\n",
        "Suppose we introduce a new review (REVIEW 4) \\\n",
        "REVIEW 4: This movie is filled with good actors and is very good. \\\n",
        "**STEP 3A of METHOD 2:** Define REVIEW 4 as `newdata` \\\n",
        "**STEP 3B of METHOD 2:** **Transform** new data or reviews into **count-vectorized** form using the `transform()` method. No fitting is needed \\\n",
        "This will yield the **vector** of REVIEW 4 under the **BoW** model \\\n",
        "Unseen words (e.g., 'actors', 'filled', 'with') are ignored"
      ],
      "metadata": {
        "id": "HEd3DXaNZnPB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# A list of new documents\n",
        "newdata = [\"This movie is filled with good actors and is very good.\"]"
      ],
      "metadata": {
        "id": "9mDQUMBkZ1hm"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "newdata_counts = convert.transform(newdata)\n",
        "newdata_counts.toarray()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "byiwtLy2anKP",
        "outputId": "8e657b42-7ae2-47a5-c201-2a3491770619"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 2, 2, 0, 1, 0, 0, 0, 0, 1, 1]])"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 3C of METHOD 2:** **Transform** new data or reviews using `tfidf` to derive the **TF-IDF** scores of the terms \\"
      ],
      "metadata": {
        "id": "K0E5mJSza7By"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Again, transform using tfidf \n",
        "newdata_tfidf = fooTfmer.transform(newdata_counts)\n",
        "newdata_tfidf.toarray()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0Fpimvxua7ir",
        "outputId": "812ffd5d-ade3-4041-98e1-0fe038e46fc3"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.2165043 , 0.7331473 , 0.43300861, 0.        , 0.2165043 ,\n",
              "        0.        , 0.        , 0.        , 0.        , 0.2165043 ,\n",
              "        0.36657365]])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Sentiment Analysis (SA) of Movie Reviews\n",
        "**STEP 1 of SA:** Import `sklearn` and use the `load_files` package \\\n",
        "**STEP 2 of SA:** Use the `movie_reviews` dataset co-authored by Bo Pang and Lillian Lee (2004): http://www.cs.cornell.edu/people/pabo/movie-review-data/ \\\n",
        "\n",
        "Here is the documentation on **dataset loading** for Scikit-learn (or `sklearn`): https://scikit-learn.org/stable/datasets.html \\\n",
        "NOTE: I have saved the `movie_reviews` dataset (containing `neg` and `pos` subfolders) on Google Drive: https://drive.google.com/drive/folders/1da0zAlewgCUxYgsqfuiOYHSTSYwqrvRh?usp=sharing \\\n",
        "i) Import `drive` from Google Colab and mount Google Drive. This will allow you to interact with your files from Google Drive; \\\n",
        "ii) Define `moviedir` as the `movie_reviews` folder on Google Drive (using the relevant **file path**: in my case, it is `/content/drive/MyDrive/movie_reviews`)"
      ],
      "metadata": {
        "id": "9sADzk8rlM8n"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sklearn\n",
        "from sklearn.datasets import load_files"
      ],
      "metadata": {
        "id": "pDOu2jmGGStM"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XcLby0uejnxq",
        "outputId": "6c5d3944-56f0-4d2c-82ec-282ca986d123"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "moviedir = \"/content/drive/MyDrive/movie_reviews\"\n",
        "# loading all files. \n",
        "movie = load_files(moviedir, shuffle=True)"
      ],
      "metadata": {
        "id": "RKZPUlIuj-UL"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 3 of SA:** Split the data from `movie_reviews` into **training sets** and **test sets** \\\n",
        "**STEP 4 of SA:** Initialize `CountVectorizer`. The instructions `mxa_features=3000` are intended to confine our analysis to the **top 3,000 words** only. If we use all 25,000 words, we will get a higher level of accuracy \\\n",
        "**STEP 5 of SA:** **Fit and transform** `movieVzer` using the `fit_transform()` method"
      ],
      "metadata": {
        "id": "LXuOdZnEnR_h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "docs_train, docs_test, y_train, y_test = train_test_split(movie.data, movie.target, \n",
        "                                                          test_size = 0.20, random_state = 12)"
      ],
      "metadata": {
        "id": "QcDOP6y4kbdn"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('punkt')\n",
        "movieVzer= CountVectorizer(min_df=2, tokenizer=nltk.word_tokenize, max_features=3000) # use top 3000 words only. 78.25% acc.\n",
        "# movieVzer = CountVectorizer(min_df=2, tokenizer=nltk.word_tokenize)         # use all 25K words. Higher accuracy\n",
        "\n",
        "# fit and tranform using training text \n",
        "docs_train_counts = movieVzer.fit_transform(docs_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DoC17t4fliG7",
        "outputId": "703e76dd-e011-4104-c4b4-b7fc6971de36"
      },
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 6 of SA:** Check the index number of unique terms in the corpus using the `vocabulary_.get()` function \\\n",
        "The term 'screen' is in the `movie_reviews` corpus and has been mapped to index number 2291 \\\n",
        "The term 'seagal' is in the `movie_reviews` corpus and has been mapped to index number 2298 \\\n",
        "**STEP 7 of SA:** Determine the **dimensions** using the `.shape` function \\\n",
        "The **dimensions** here are a whopping 1,600 (number of documents) by 3,000 (number of **unique terms**)"
      ],
      "metadata": {
        "id": "teqep7k6oBVN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movieVzer.vocabulary_.get('screen')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mcb8owi0lzPR",
        "outputId": "bbdf857b-a70d-4b31-aa73-27b125fc20fd"
      },
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2291"
            ]
          },
          "metadata": {},
          "execution_count": 98
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "movieVzer.vocabulary_.get('seagal')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "atcbq5-3l2xa",
        "outputId": "a6c5eb35-0f70-4674-adde-4d5f1084ec94"
      },
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2298"
            ]
          },
          "metadata": {},
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "docs_train_counts.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wL-pX39Bl5ng",
        "outputId": "f511c47f-00e1-44e2-bfb6-4ae205ff2a53"
      },
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1600, 3000)"
            ]
          },
          "metadata": {},
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 8 of SA:** Use the `TfidfTransformer` and the `.fit_transform()` method to convert **raw frequency counts** under the **BoW** model (**METHOD 1**) into **TF-IDF** values (**METHOD 2**) \\\n",
        "$\\therefore$ The **vectorizer** and **transformer** have been built into the **training data** \\\\\n",
        "The **training data** text has also been turned into **TF-IDF** vector form"
      ],
      "metadata": {
        "id": "0-CEAnSCosNY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "movieTfmer = TfidfTransformer()\n",
        "docs_train_tfidf = movieTfmer.fit_transform(docs_train_counts)"
      ],
      "metadata": {
        "id": "VeXiRwZRl8Gg"
      },
      "execution_count": 101,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Same dimensions, now with tf-idf values instead of raw frequency counts\n",
        "docs_train_tfidf.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_iO7iAl0l-4Y",
        "outputId": "dd4a330c-c98b-4238-d1b1-a05009ab3458"
      },
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1600, 3000)"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 9 of SA:** Prepare the **test data** using the same **feature extraction** scheme"
      ],
      "metadata": {
        "id": "EMWPUfhjpHKg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Using the fitted vectorizer and transformer, tranform the test data\n",
        "docs_test_counts = movieVzer.transform(docs_test)\n",
        "docs_test_tfidf = movieTfmer.transform(docs_test_counts)"
      ],
      "metadata": {
        "id": "TiD2vqzEpH6S"
      },
      "execution_count": 103,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 10 of SA:** Train and test a **naive Bayes classifier** \\\n",
        "We will use **Multinominal Naive Bayes** (or `Multinomial NB`) as our model \\\n",
        "We will train our **naive Bayes classifier** by using the `.fit()` method \\\n"
      ],
      "metadata": {
        "id": "99wi7_cdpqi2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.naive_bayes import MultinomialNB\n",
        "\n",
        "clf = MultinomialNB()\n",
        "clf.fit(docs_train_tfidf, y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UkkOjYRvpua2",
        "outputId": "ed8fb1a7-f62b-4293-b91d-71cac47f7db6"
      },
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MultinomialNB()"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 11 of SA:** Predict the **test dataset** results and determine the accuracy of the **naive Bayes classifier**"
      ],
      "metadata": {
        "id": "SBfZlfrWqVFT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Predict the Test set results, find accuracy\n",
        "y_pred = clf.predict(docs_test_tfidf)\n",
        "sklearn.metrics.accuracy_score(y_test, y_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4OXM9MlBqJ5u",
        "outputId": "3d5c045c-1572-4524-f464-93cd842654a3"
      },
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7825"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Making the Confusion Matrix\n",
        "from sklearn.metrics import confusion_matrix\n",
        "cm = confusion_matrix(y_test, y_pred)\n",
        "cm"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NUSzSYxrqiI8",
        "outputId": "983d8a33-55cb-488a-dd7d-2eb48a72c9dd"
      },
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[164,  42],\n",
              "       [ 45, 149]])"
            ]
          },
          "metadata": {},
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**STEP 12 of SA:** We can now try the **naive Bayes classifier** on fake movie reviews \\\n",
        "Here is the text for the fake movie reviews (`reviews_new`): \\\n",
        "'This movie was excellent', \\\n",
        "'Absolute joy ride', \\\n",
        "'Steven Seagal was terrible', \\\n",
        "'Steven Seagal shone through.', \\\n",
        "'This was certainly a movie', \\\n",
        "'Two thumbs up', \\\n",
        "'I fell asleep halfway through', \\\n",
        "\"We can't wait for the sequel!!\", \\\n",
        "'!', \\\n",
        "'?', \\\n",
        "'I cannot recommend this highly enough', \\\n",
        "'instant classic.', \\\n",
        "'Steven Seagal was amazing. His performance was Oscar-worthy.'"
      ],
      "metadata": {
        "id": "kk11_4ROqmGv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reviews_new = ['This movie was excellent', 'Absolute joy ride', \n",
        "            'Steven Seagal was terrible', 'Steven Seagal shone through.', \n",
        "              'This was certainly a movie', 'Two thumbs up', 'I fell asleep halfway through', \n",
        "              \"We can't wait for the sequel!!\", '!', '?', 'I cannot recommend this highly enough', \n",
        "              'instant classic.', 'Steven Seagal was amazing. His performance was Oscar-worthy.']\n",
        "reviews_new_counts = movieVzer.transform(reviews_new)         # turn text into count vector\n",
        "reviews_new_tfidf = movieTfmer.transform(reviews_new_counts)  # turn into tfidf vector"
      ],
      "metadata": {
        "id": "0jX0ymKOqsyB"
      },
      "execution_count": 107,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the naive Bayes classifier to make a prediction\n",
        "pred = clf.predict(reviews_new_tfidf)"
      ],
      "metadata": {
        "id": "FmThM00zqvPZ"
      },
      "execution_count": 108,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print out the results\n",
        "for review, category in zip(reviews_new, pred):\n",
        "    print('%r => %s' % (review, movie.target_names[category]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gi1yiccCqzgj",
        "outputId": "91ca083d-d3c3-4742-f535-6d5b49a9bf84"
      },
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "'This movie was excellent' => pos\n",
            "'Absolute joy ride' => pos\n",
            "'Steven Seagal was terrible' => neg\n",
            "'Steven Seagal shone through.' => neg\n",
            "'This was certainly a movie' => neg\n",
            "'Two thumbs up' => neg\n",
            "'I fell asleep halfway through' => neg\n",
            "\"We can't wait for the sequel!!\" => neg\n",
            "'!' => neg\n",
            "'?' => neg\n",
            "'I cannot recommend this highly enough' => pos\n",
            "'instant classic.' => pos\n",
            "'Steven Seagal was amazing. His performance was Oscar-worthy.' => neg\n"
          ]
        }
      ]
    }
  ]
}