unsupervised-learning/NLP_about_amazon_reviews.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLP_about_amazon_reviews.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "bdlqerpH59cc",
        "_dfjExSo-6X3",
        "i3Dm6aUO-5bM",
        "tsssvpzhAf06"
      ],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNNJEqZJmeymSGOMtqjMnE7",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wenjunsun/personal-machine-learning-projects/blob/master/unsupervised-learning/NLP_about_amazon_reviews.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "34y0lPoL5o3c",
        "colab_type": "text"
      },
      "source": [
        "In this notebook I will explore Amazon's reviews data, and do some clustering analysis - what kind of comments tend to lump together? Maybe I can do some elementary sentiment analysis about which comments are positive and which comments are negative as well. Pretty exciting! Let's get started! This is my first time doing an NLP project, I used to always work with extracting insights from numbers, so let's see how far I can get with doing the same thing twith words and sentences.\n",
        "\n",
        "This whole notebook is done in Google Colab, which provides 12 GB of RAM and 100 GB hard drive. I don't see a better free platform than Google Colab. But of course you can do all of the following on your own local machine. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdlqerpH59cc",
        "colab_type": "text"
      },
      "source": [
        "# Step 1. download and load data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_dfjExSo-6X3",
        "colab_type": "text"
      },
      "source": [
        "## 1.1 find the data, download it onto Colab drive, explore in bash"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CdUoXS_B6Vfc",
        "colab_type": "text"
      },
      "source": [
        "First we need to download review data from Aamzon. There are many choices about how to get that data of course. [I choose the data source from this website](https://s3.amazonaws.com/amazon-reviews-pds/readme.html). Amazon releases its own data on this web, which is a better source than other third-party dataset hosts. \n",
        "\n",
        "If you have trouble finding where the data is, [go to this index file](https://s3.amazonaws.com/amazon-reviews-pds/tsv/index.txt), in it you will see the download links for all kinds of review datasets, in URL links with .tsv.gz at the end. There are many kinds of reviews as we can see. There are reviews about video games, watches, and books. I like books, so I am going to use that dataset, but of course you can use any dataset you want."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "75oMy1rF8goA",
        "colab_type": "text"
      },
      "source": [
        "Anyway, assuming you are on colab, you can copy the link to your favorite dataset, and do the following command in bash. Then you will have the data sitting in your colab drive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CmazcQnS8VVE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "wget https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Books_v1_02.tsv.gz # replace by your own data link here."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gYmnX5jU80An",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# since the data is in .gz formate (compressed like .zip file),\n",
        "# we need to unzip it. This is how you unzip a file in bash:\n",
        "%%bash\n",
        "gunzip amazon_reviews_us_Books_v1_02.tsv.gz "
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F4raXvoH9qn2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5443763f-8e9c-4e73-8022-c6fdc7d135ea"
      },
      "source": [
        "ls"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "amazon_reviews_us_Books_v1_02.tsv  \u001b[0m\u001b[01;34msample_data\u001b[0m/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rTdaw0ON9uFb",
        "colab_type": "text"
      },
      "source": [
        "Now we can see our data in .tsv file. (3 GB). tsv means separated by tabs. (csv is separated by comma)\n",
        "\n",
        "Before we even load this file into a pandas dataframe, let's just do a little bit of exploration in bash. For example we can find out how many lines this file has, and we can see its first 5 lines, etc."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KsIPA76O9-jq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "fec2b2ec-effc-4bc1-9357-49374249f5aa"
      },
      "source": [
        "# count how many lines this file has.\n",
        "%%bash\n",
        "wc -l amazon_reviews_us_Books_v1_02.tsv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "3105521 amazon_reviews_us_Books_v1_02.tsv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nv5q9Vwg-esF",
        "colab_type": "text"
      },
      "source": [
        "We can see that we have about 3 million lines of data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zLbasomo-ip9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "a5e71136-def9-4a94-d110-ab18e3cdc756"
      },
      "source": [
        "# print out first 5 lines.\n",
        "%%bash\n",
        "head -5 amazon_reviews_us_Books_v1_02.tsv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "marketplace\tcustomer_id\treview_id\tproduct_id\tproduct_parent\tproduct_title\tproduct_category\tstar_rating\thelpful_votes\ttotal_votes\tvine\tverified_purchase\treview_headline\treview_body\treview_date\n",
            "US\t12076615\tRQ58W7SMO911M\t0385730586\t122662979\tSisterhood of the Traveling Pants (Book 1)\tBooks\t4\t2\t3\tN\tN\tthis book was a great learning novel!\tthis boook was a great one that you could learn from. it not only teaches the imponrtance of family and their values but it also deals with basic issues that teens and some kids even deal with.  this book is about 4 best friends who are for the first time in their lives spending their summer apart. one day they are all in one of the girls rooms and finds a pair of pants that were tucked away in her closet.  once all four  of them try them on they realize that there is really something special about these pants.  seeming as how all 4 girls are differnt shapes and sizes and somehow the pants fit all of them,  they realize that these pants were the start of something special.  immediatley following they decided to make up certian rules abut the pants such as you must write the best thing u did while wearing the pants over your summer on the right leg and also some silly things such as to \\\\\"never pick yuor nose while wearing the pants.\\\\\"  this book follows the girls throuh their summers in differnt places of the world and through all of the different obstacles that life takes them through. it can really teach you alot not only about what is going on around you but most imporntantly about yuorself.  i would give this book 4 stars and would reccommend it to anyone who seems the slihgtest bit interested.\t2005-10-14\n",
            "US\t12703090\tRF6IUKMGL8SF\t0811828964\t56191234\tThe Bad Girl's Guide to Getting What You Want\tBooks\t3\t5\t5\tN\tN\tFun Fluff\tIf you are looking for something to stimulate your brain, this isn't it.  However, if you are just looking for a good laugh, you'll enjoy The Bad Girl's Guide.  It's funny and light, and definitely a good way to pass a little bit of time.\t2005-10-14\n",
            "US\t12257412\tR1DOSHH6AI622S\t1844161560\t253182049\tEisenhorn (A Warhammer 40,000 Omnibus)\tBooks\t4\t1\t22\tN\tN\tthis isn't a review\tnever read it-a young relative idicated he liked it and somehow my name popped upon this--no more to say\t2005-10-14\n",
            "US\t50732546\tRATOTLA3OF70O\t0373836635\t348672532\tColby Conspiracy (Colby Agency)\tBooks\t5\t2\t2\tN\tN\tfine author on her A-game\tThough she is honored to be Chicago Woman of the Year, Victoria Colby-Camp is more euphoric over the mental improvement that her son Jim has shown recently especially since he and Tasha North fell in love.  Jim was snatched almost twenty years ago when he was seven and turned into the killing Seth whose goal was to murder Victoria for abandoning him.  However, her elation would turn to despair if she knew Seth resurfaced and started to rape a pregnant Tasha.<br /><br />Former military strategist Daniel Marks is in town complements of the Colby Agency that wants to hire him.  Also in Chicago is Emily Hastings whose father a veteran homicide detective was murdered.  She finds letters linking her dad to Victoria, the woman's long ago murdered first husband James, and her dad's first partner Marelyn Rutland that confuses her.  Soon she will meet Daniel and they will be embroiled in the COLBY CONSPIRACY that goes back almost two decades ago.<br /><br />Though the subplots can become confusing at first, once the audience comprehends how this complex superb suspense thriller starts to come together, they will want more Colby Agency tales; (see FILES FROM THE COLBY AGENCY: THE BODYGUARD'S BABY PROTECTIVE CUSTODY).  The ensemble cast is solid as fans will feel with Victoria who has overcome so much tragedy, hope Jim \\\\\"defeats\\\\\" Seth with Tasha at his side, and root for Daniel and Emily to make it while wondering what really happened two decades ago.  A final twist marks a strong Webb of deceit tale that showcases a fine author on her A-game.<br /><br />Harriet Klausner\t2005-10-14\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "98evnOaD-rwM",
        "colab_type": "text"
      },
      "source": [
        "we can see that first row is the column names of the data, we have the name of the product, and the rating, and of course the reviews and the date of the review. Looks super interesting!!!! Now let's load this data into python dataframe."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i3Dm6aUO-5bM",
        "colab_type": "text"
      },
      "source": [
        "## 1.2 load data into python dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BQnn2JeIKsjc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "5db08017-8672-4d45-8abb-2b9260b42928"
      },
      "source": [
        "# load necessary packages.\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qeacXV95_a6b",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3e7d13b0-3d96-4378-832e-263006693753"
      },
      "source": [
        "ls"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "amazon_reviews_us_Books_v1_02.tsv  \u001b[0m\u001b[01;34msample_data\u001b[0m/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ne8FB6L1MdEe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 477
        },
        "outputId": "17fe638a-9296-4d83-b12c-27a4e739ac81"
      },
      "source": [
        "book_review_data = pd.read_csv(\"amazon_reviews_us_Books_v1_02.tsv\", sep = \"\\t\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ParserError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mParserError\u001b[0m                               Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-11-a4d6864a5537>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mbook_review_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"amazon_reviews_us_Books_v1_02.tsv\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\t\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    674\u001b[0m         )\n\u001b[1;32m    675\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 676\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    678\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    452\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    453\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 454\u001b[0;31m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    455\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    456\u001b[0m         \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, nrows)\u001b[0m\n\u001b[1;32m   1131\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnrows\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1132\u001b[0m         \u001b[0mnrows\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_validate_integer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"nrows\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1133\u001b[0;31m         \u001b[0mret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1135\u001b[0m         \u001b[0;31m# May alter columns / col_dict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, nrows)\u001b[0m\n\u001b[1;32m   2035\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnrows\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2036\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2037\u001b[0;31m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2038\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2039\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_first_chunk\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.read\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._read_low_memory\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._read_rows\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._tokenize_rows\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.raise_parser_error\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mParserError\u001b[0m: Error tokenizing data. C error: Expected 15 fields in line 1680001, saw 22\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7MhoOarI_pDW",
        "colab_type": "text"
      },
      "source": [
        "Okay we see that we have some lines that correspond to \"bad data\", some lines have 22 fields while we are only expecting 15. For now let's just not load those bad lines into our dataframe. There is a way to do this with *read_csv* function with a extra parameter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uNCGRgJkAHgr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "11bbd5e9-42bb-4e68-91c2-8f93489dbb14"
      },
      "source": [
        "book_review_data = pd.read_csv(\"amazon_reviews_us_Books_v1_02.tsv\", sep = \"\\t\", error_bad_lines = False) \n",
        "# error_bad_lines = False just tells parser to ignore bad lines in .tsv file.\n",
        "# only read in the lines that make sense."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "b'Skipping line 1680001: expected 15 fields, saw 22\\n'\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qOTSvdxiHNtL",
        "colab_type": "text"
      },
      "source": [
        "# Step 2: explore data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tsssvpzhAf06",
        "colab_type": "text"
      },
      "source": [
        "## 2.1 Play with our data!!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2GuyD07tR6_L",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        },
        "outputId": "95312fa7-6b5e-4557-fd41-c170aa288ff9"
      },
      "source": [
        "# look at our data.\n",
        "book_review_data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>marketplace</th>\n",
              "      <th>customer_id</th>\n",
              "      <th>review_id</th>\n",
              "      <th>product_id</th>\n",
              "      <th>product_parent</th>\n",
              "      <th>product_title</th>\n",
              "      <th>product_category</th>\n",
              "      <th>star_rating</th>\n",
              "      <th>helpful_votes</th>\n",
              "      <th>total_votes</th>\n",
              "      <th>vine</th>\n",
              "      <th>verified_purchase</th>\n",
              "      <th>review_headline</th>\n",
              "      <th>review_body</th>\n",
              "      <th>review_date</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>US</td>\n",
              "      <td>12076615</td>\n",
              "      <td>RQ58W7SMO911M</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>this book was a great learning novel!</td>\n",
              "      <td>this boook was a great one that you could lear...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>US</td>\n",
              "      <td>12703090</td>\n",
              "      <td>RF6IUKMGL8SF</td>\n",
              "      <td>0811828964</td>\n",
              "      <td>56191234</td>\n",
              "      <td>The Bad Girl's Guide to Getting What You Want</td>\n",
              "      <td>Books</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Fun Fluff</td>\n",
              "      <td>If you are looking for something to stimulate ...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>US</td>\n",
              "      <td>12257412</td>\n",
              "      <td>R1DOSHH6AI622S</td>\n",
              "      <td>1844161560</td>\n",
              "      <td>253182049</td>\n",
              "      <td>Eisenhorn (A Warhammer 40,000 Omnibus)</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>22.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>this isn't a review</td>\n",
              "      <td>never read it-a young relative idicated he lik...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>US</td>\n",
              "      <td>50732546</td>\n",
              "      <td>RATOTLA3OF70O</td>\n",
              "      <td>0373836635</td>\n",
              "      <td>348672532</td>\n",
              "      <td>Colby Conspiracy (Colby Agency)</td>\n",
              "      <td>Books</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>fine author on her A-game</td>\n",
              "      <td>Though she is honored to be Chicago Woman of t...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>US</td>\n",
              "      <td>51964897</td>\n",
              "      <td>R1TNWRKIVHVYOV</td>\n",
              "      <td>0262181533</td>\n",
              "      <td>598678717</td>\n",
              "      <td>The Psychology of Proof: Deductive Reasoning i...</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Execellent cursor examination</td>\n",
              "      <td>Review based on a cursory examination by Unive...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  marketplace  ...  review_date\n",
              "0          US  ...   2005-10-14\n",
              "1          US  ...   2005-10-14\n",
              "2          US  ...   2005-10-14\n",
              "3          US  ...   2005-10-14\n",
              "4          US  ...   2005-10-14\n",
              "\n",
              "[5 rows x 15 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lpqMRgyOR_aO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "outputId": "eda7bc56-834e-4239-cc1e-45364e5c99de"
      },
      "source": [
        "# print out the first review we have.\n",
        "# this looks like a positive review.\n",
        "book_review_data.loc[0,'review_body']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'this boook was a great one that you could learn from. it not only teaches the imponrtance of family and their values but it also deals with basic issues that teens and some kids even deal with.  this book is about 4 best friends who are for the first time in their lives spending their summer apart. one day they are all in one of the girls rooms and finds a pair of pants that were tucked away in her closet.  once all four  of them try them on they realize that there is really something special about these pants.  seeming as how all 4 girls are differnt shapes and sizes and somehow the pants fit all of them,  they realize that these pants were the start of something special.  immediatley following they decided to make up certian rules abut the pants such as you must write the best thing u did while wearing the pants over your summer on the right leg and also some silly things such as to \\\\\\\\\"never pick yuor nose while wearing the pants.\\\\\\\\\"  this book follows the girls throuh their summers in differnt places of the world and through all of the different obstacles that life takes them through. it can really teach you alot not only about what is going on around you but most imporntantly about yuorself.  i would give this book 4 stars and would reccommend it to anyone who seems the slihgtest bit interested.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WeMdPou4BRYy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d3998609-57c4-45c3-daa8-53e01dfefd70"
      },
      "source": [
        "# as we can see, we have 3 million data points about book reviews.\n",
        "# don't think 3 million is \"big data\", but every row contains\n",
        "# rich information about reviews, which is not just some numbers.\n",
        "book_review_data.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3105370, 15)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TfaqXQEXA4Nn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "b130b5e8-7618-45d0-8b2b-f18f75a94a97"
      },
      "source": [
        "# look at data types. - in some datasets ints looks like ints\n",
        "# but are actually strings.\n",
        "book_review_data.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 3105370 entries, 0 to 3105369\n",
            "Data columns (total 15 columns):\n",
            " #   Column             Dtype  \n",
            "---  ------             -----  \n",
            " 0   marketplace        object \n",
            " 1   customer_id        int64  \n",
            " 2   review_id          object \n",
            " 3   product_id         object \n",
            " 4   product_parent     int64  \n",
            " 5   product_title      object \n",
            " 6   product_category   object \n",
            " 7   star_rating        float64\n",
            " 8   helpful_votes      float64\n",
            " 9   total_votes        float64\n",
            " 10  vine               object \n",
            " 11  verified_purchase  object \n",
            " 12  review_headline    object \n",
            " 13  review_body        object \n",
            " 14  review_date        object \n",
            "dtypes: float64(3), int64(2), object(10)\n",
            "memory usage: 355.4+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hcXL54q0BA6D",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "b1e86e08-7d9a-4062-ef6a-2462bf12d878"
      },
      "source": [
        "# there are some null rows, not that much\n",
        "# compared to our 3 million data points\n",
        "book_review_data.isnull().sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "marketplace            0\n",
              "customer_id            0\n",
              "review_id              0\n",
              "product_id             0\n",
              "product_parent         0\n",
              "product_title          0\n",
              "product_category       0\n",
              "star_rating            4\n",
              "helpful_votes          4\n",
              "total_votes            4\n",
              "vine                   4\n",
              "verified_purchase      4\n",
              "review_headline       42\n",
              "review_body            4\n",
              "review_date          133\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JoYc_hkLBDU0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "fb6e66b1-df1f-42b8-cf93-de6519f66ca6"
      },
      "source": [
        "book_review_data.head()['review_date']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    2005-10-14\n",
              "1    2005-10-14\n",
              "2    2005-10-14\n",
              "3    2005-10-14\n",
              "4    2005-10-14\n",
              "Name: review_date, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "omi60coQB2zE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "d27eaf07-f6d1-4b43-9310-c82c471f9a4f"
      },
      "source": [
        "book_review_data.tail()['review_date']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3105365    1995-07-19\n",
              "3105366    1995-07-18\n",
              "3105367    1995-07-07\n",
              "3105368    1995-06-24\n",
              "3105369    1995-06-24\n",
              "Name: review_date, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HxFceBt7CJCF",
        "colab_type": "text"
      },
      "source": [
        "From above we can see this dataset's reviews are from year 1995 to 2005, pretty ancient lol... (Right now it is 2020.) Will be interesting to find out 20 years ago people's taste in books!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o6cotL69DEYo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "b9821287-2a29-4a20-eeb4-a7c1983cf119"
      },
      "source": [
        "book_review_data"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>marketplace</th>\n",
              "      <th>customer_id</th>\n",
              "      <th>review_id</th>\n",
              "      <th>product_id</th>\n",
              "      <th>product_parent</th>\n",
              "      <th>product_title</th>\n",
              "      <th>product_category</th>\n",
              "      <th>star_rating</th>\n",
              "      <th>helpful_votes</th>\n",
              "      <th>total_votes</th>\n",
              "      <th>vine</th>\n",
              "      <th>verified_purchase</th>\n",
              "      <th>review_headline</th>\n",
              "      <th>review_body</th>\n",
              "      <th>review_date</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>US</td>\n",
              "      <td>12076615</td>\n",
              "      <td>RQ58W7SMO911M</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>this book was a great learning novel!</td>\n",
              "      <td>this boook was a great one that you could lear...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>US</td>\n",
              "      <td>12703090</td>\n",
              "      <td>RF6IUKMGL8SF</td>\n",
              "      <td>0811828964</td>\n",
              "      <td>56191234</td>\n",
              "      <td>The Bad Girl's Guide to Getting What You Want</td>\n",
              "      <td>Books</td>\n",
              "      <td>3.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Fun Fluff</td>\n",
              "      <td>If you are looking for something to stimulate ...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>US</td>\n",
              "      <td>12257412</td>\n",
              "      <td>R1DOSHH6AI622S</td>\n",
              "      <td>1844161560</td>\n",
              "      <td>253182049</td>\n",
              "      <td>Eisenhorn (A Warhammer 40,000 Omnibus)</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>22.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>this isn't a review</td>\n",
              "      <td>never read it-a young relative idicated he lik...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>US</td>\n",
              "      <td>50732546</td>\n",
              "      <td>RATOTLA3OF70O</td>\n",
              "      <td>0373836635</td>\n",
              "      <td>348672532</td>\n",
              "      <td>Colby Conspiracy (Colby Agency)</td>\n",
              "      <td>Books</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>fine author on her A-game</td>\n",
              "      <td>Though she is honored to be Chicago Woman of t...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>US</td>\n",
              "      <td>51964897</td>\n",
              "      <td>R1TNWRKIVHVYOV</td>\n",
              "      <td>0262181533</td>\n",
              "      <td>598678717</td>\n",
              "      <td>The Psychology of Proof: Deductive Reasoning i...</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Execellent cursor examination</td>\n",
              "      <td>Review based on a cursory examination by Unive...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3105365</th>\n",
              "      <td>US</td>\n",
              "      <td>53096582</td>\n",
              "      <td>R1U2RAML67OKWM</td>\n",
              "      <td>0471114251</td>\n",
              "      <td>574261099</td>\n",
              "      <td>Bitter Winds: A Memoir of My Years in China's ...</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Harry Wu could have taught Kafka a thing or two.</td>\n",
              "      <td>&amp;quot;Bitter Winds&amp;quot; is at once fascinatin...</td>\n",
              "      <td>1995-07-19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3105366</th>\n",
              "      <td>US</td>\n",
              "      <td>53011769</td>\n",
              "      <td>R33HOJ2OWJIDQI</td>\n",
              "      <td>089145537X</td>\n",
              "      <td>836948045</td>\n",
              "      <td>The Collector's Guide to Harker Pottery U.S.A....</td>\n",
              "      <td>Books</td>\n",
              "      <td>5.0</td>\n",
              "      <td>27.0</td>\n",
              "      <td>27.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>The definitive volume on Harker Pottery</td>\n",
              "      <td>OK, I'm biased. My mother wrote this one. Many...</td>\n",
              "      <td>1995-07-18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3105367</th>\n",
              "      <td>US</td>\n",
              "      <td>53096573</td>\n",
              "      <td>RPLV77JZXG575</td>\n",
              "      <td>047194128X</td>\n",
              "      <td>377091465</td>\n",
              "      <td>Object-Oriented Type Systems</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Good techniques, well written.</td>\n",
              "      <td>The best (and possibly only) book I've seen on...</td>\n",
              "      <td>1995-07-07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3105368</th>\n",
              "      <td>US</td>\n",
              "      <td>53096571</td>\n",
              "      <td>RHL4UW17ZK72A</td>\n",
              "      <td>0521314925</td>\n",
              "      <td>980601331</td>\n",
              "      <td>Invention and Evolution:Design in Nature and E...</td>\n",
              "      <td>Books</td>\n",
              "      <td>5.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>BUY THIS BOOK!</td>\n",
              "      <td>This is a beautiful book.  French talks about ...</td>\n",
              "      <td>1995-06-24</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3105369</th>\n",
              "      <td>US</td>\n",
              "      <td>53096571</td>\n",
              "      <td>R34N4QWDXX58WB</td>\n",
              "      <td>0870210092</td>\n",
              "      <td>442607382</td>\n",
              "      <td>Arming and Fitting of English Ships of War, 16...</td>\n",
              "      <td>Books</td>\n",
              "      <td>4.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>good enough to understand all of Pat O'brien</td>\n",
              "      <td>Nice diags, lucid explanations of rigging, gun...</td>\n",
              "      <td>1995-06-24</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3105370 rows × 15 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "        marketplace  ...  review_date\n",
              "0                US  ...   2005-10-14\n",
              "1                US  ...   2005-10-14\n",
              "2                US  ...   2005-10-14\n",
              "3                US  ...   2005-10-14\n",
              "4                US  ...   2005-10-14\n",
              "...             ...  ...          ...\n",
              "3105365          US  ...   1995-07-19\n",
              "3105366          US  ...   1995-07-18\n",
              "3105367          US  ...   1995-07-07\n",
              "3105368          US  ...   1995-06-24\n",
              "3105369          US  ...   1995-06-24\n",
              "\n",
              "[3105370 rows x 15 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R7soZtXmCIbr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "b08c1d90-a72b-43a6-b486-6a7d3fccd6bd"
      },
      "source": [
        "# look at bad reviews for the book \"Sisterhood of the Traveling Pants (Book 1)\"\n",
        "book_review_data[(book_review_data['product_title'] == 'Sisterhood of the Traveling Pants (Book 1)')& (book_review_data['star_rating'] < 3)]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>marketplace</th>\n",
              "      <th>customer_id</th>\n",
              "      <th>review_id</th>\n",
              "      <th>product_id</th>\n",
              "      <th>product_parent</th>\n",
              "      <th>product_title</th>\n",
              "      <th>product_category</th>\n",
              "      <th>star_rating</th>\n",
              "      <th>helpful_votes</th>\n",
              "      <th>total_votes</th>\n",
              "      <th>vine</th>\n",
              "      <th>verified_purchase</th>\n",
              "      <th>review_headline</th>\n",
              "      <th>review_body</th>\n",
              "      <th>review_date</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>132278</th>\n",
              "      <td>US</td>\n",
              "      <td>12799825</td>\n",
              "      <td>RLMD20PYH5VJL</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>25.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>The Fall of the Sisterhood</td>\n",
              "      <td>I recently read The Sisterhood of the Travelin...</td>\n",
              "      <td>2005-08-03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>161578</th>\n",
              "      <td>US</td>\n",
              "      <td>17580189</td>\n",
              "      <td>RCMW8NARCCMBL</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>2.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Too mature for young girls</td>\n",
              "      <td>I have to say that I am a little amazed at the...</td>\n",
              "      <td>2005-07-18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>210661</th>\n",
              "      <td>US</td>\n",
              "      <td>24822026</td>\n",
              "      <td>RPGJKHGFQKXX9</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>2.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Do not read this!</td>\n",
              "      <td>I spent more than 7 dollors to buy this book a...</td>\n",
              "      <td>2005-06-09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>278735</th>\n",
              "      <td>US</td>\n",
              "      <td>36610202</td>\n",
              "      <td>R22G6JK4SOEKY4</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>26.0</td>\n",
              "      <td>38.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Clichéd</td>\n",
              "      <td>I'm a fifteen year old reader, and I typically...</td>\n",
              "      <td>2005-04-13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>293886</th>\n",
              "      <td>US</td>\n",
              "      <td>23349377</td>\n",
              "      <td>RSQWUFUGP3VMU</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>26.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>What's all the fuss?</td>\n",
              "      <td>I just don't get why this book is so popular. ...</td>\n",
              "      <td>2005-04-01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>369134</th>\n",
              "      <td>US</td>\n",
              "      <td>14702007</td>\n",
              "      <td>R104D8BEQTESWJ</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>27.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>The Sisterhood of the Traveling Pants</td>\n",
              "      <td>This book is not worth your time. It took me m...</td>\n",
              "      <td>2005-01-29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>426232</th>\n",
              "      <td>US</td>\n",
              "      <td>35949446</td>\n",
              "      <td>R2UXGS8KLE2YKQ</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>2.0</td>\n",
              "      <td>41.0</td>\n",
              "      <td>50.0</td>\n",
              "      <td>N</td>\n",
              "      <td>Y</td>\n",
              "      <td>Not appropriate for young girls- or older ones</td>\n",
              "      <td>I bought this book for my 12 year old niece af...</td>\n",
              "      <td>2004-12-12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>630493</th>\n",
              "      <td>US</td>\n",
              "      <td>17899762</td>\n",
              "      <td>RHKL3VT2ZTYXG</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>2.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Disappointed - overrated and overhyped novel!</td>\n",
              "      <td>I incessantly heard about how \\\\\"great\\\\\" and ...</td>\n",
              "      <td>2004-06-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>709347</th>\n",
              "      <td>US</td>\n",
              "      <td>19451524</td>\n",
              "      <td>R1OXNMG4I56UBB</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Sisterhood Of the traveling Jeans</td>\n",
              "      <td>Too be honest, this is only my opinion! This w...</td>\n",
              "      <td>2004-04-12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>710655</th>\n",
              "      <td>US</td>\n",
              "      <td>19440744</td>\n",
              "      <td>R31JKWJRJGFYE</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>17.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>The Sisterhood of Utter Crap</td>\n",
              "      <td>This is by far one the worst books I've ever r...</td>\n",
              "      <td>2004-04-12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>785199</th>\n",
              "      <td>US</td>\n",
              "      <td>20994710</td>\n",
              "      <td>R2QJPGUGPR0N8I</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>If it's the only thing on your shelf...</td>\n",
              "      <td>In all honesty, this book is lame.  If it's th...</td>\n",
              "      <td>2004-02-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>942080</th>\n",
              "      <td>US</td>\n",
              "      <td>51169446</td>\n",
              "      <td>R26WO158AFTX5A</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>40.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>The Sisterhood of the Traveling Pants</td>\n",
              "      <td>I am totally disgusted that anyone would recom...</td>\n",
              "      <td>2003-10-13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1083612</th>\n",
              "      <td>US</td>\n",
              "      <td>51694414</td>\n",
              "      <td>RYN4V9I480ZTJ</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>1.0</td>\n",
              "      <td>176.0</td>\n",
              "      <td>201.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>The Sisterhood of the Travelling Pants</td>\n",
              "      <td>While overall, I thought this book was fine an...</td>\n",
              "      <td>2003-06-18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1092726</th>\n",
              "      <td>US</td>\n",
              "      <td>32737494</td>\n",
              "      <td>RF4DS0YNTLVF7</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Not the best book</td>\n",
              "      <td>This book was ok, but most of the time I was b...</td>\n",
              "      <td>2003-06-11</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        marketplace  ...  review_date\n",
              "132278           US  ...   2005-08-03\n",
              "161578           US  ...   2005-07-18\n",
              "210661           US  ...   2005-06-09\n",
              "278735           US  ...   2005-04-13\n",
              "293886           US  ...   2005-04-01\n",
              "369134           US  ...   2005-01-29\n",
              "426232           US  ...   2004-12-12\n",
              "630493           US  ...   2004-06-14\n",
              "709347           US  ...   2004-04-12\n",
              "710655           US  ...   2004-04-12\n",
              "785199           US  ...   2004-02-14\n",
              "942080           US  ...   2003-10-13\n",
              "1083612          US  ...   2003-06-18\n",
              "1092726          US  ...   2003-06-11\n",
              "\n",
              "[14 rows x 15 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cczrsLL9DIAW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# look at the most helpful reviews rated by others about this book\n",
        "most_helpful_review = book_review_data[(book_review_data['product_title'] == 'Sisterhood of the Traveling Pants (Book 1)') \\\n",
        "                                       & (book_review_data['helpful_votes'] > 100)]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LBt5fXEvEzmA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "bdec2b14-88b9-4d15-bece-c9850d6111bc"
      },
      "source": [
        "most_helpful_review['review_headline'].array[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'The Sisterhood of the Travelling Pants'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7M6AOB1FFHK7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 188
        },
        "outputId": "782f0b36-72d2-44e4-8b6a-38b0af40f8d4"
      },
      "source": [
        "most_helpful_review['review_body'].array[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'While overall, I thought this book was fine and possibly engaging for young women, I strongly think the 12 year old age guideline is too young, both for some of the contents of the plot and for the deeper and more meaningful aspects of what a girl could discover from the book.  In other words, I thought the chararacters were interesting to an older teen aged girl, but 12 year olds are not teenagers.  In fact, these characters are 15 and almost 16, all about to be juniors in high school.  And those ages, I think, as a mother, are the appropriate ages for a girl to be reading this book. Not only is there mature sexual content and encounter (albeit, which one of the characters cannot handle), the characters themselves, and their metamorphises, what they go through that summer, what they learn about themselves, will be much more understandable and appropriate to a 14 or 15 or 16 year old, then  to an impressionable 12 or 13 year old.  I would liken it to reading Jane Eyre, or Little Women, before a girl is old enough to understand the levels, the deep meaning,  the beauty of these books.  Sure, girls of 11 and 12 are capable of reading these books, but it is the parents\\' jobs to say \\\\\\\\\"not yet\\\\\\\\\".   I also am disturbed that none of the other reviewers that I read on line seem to understand that one of the characters is self-destructively manic depressive and that would be something ( in fact I would urge any mother who lets her daughter read this book) to discuss the behaviors with her daughter.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WH3ro2P6Fp2X",
        "colab_type": "text"
      },
      "source": [
        "We can see that this reviewer said that the book is too mature for young girlds. This is interesting. Although I never read the book, this is a great insight to have as Amazon, or as the book writer. Amazon should suggest this book for >= 16 olds, and not 12 year olds."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3EJ4x_3gHE8J",
        "colab_type": "text"
      },
      "source": [
        "# Step 3: define our problem"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tB61RmGtHVTp",
        "colab_type": "text"
      },
      "source": [
        "Obviously there are so many paths we can explore with this dataset! For example:\n",
        "\n",
        "1. what books are gaining the most positive reviews? What kind of books are those (relevant for book writers and Amazon to think about what they want to sell more in the future. For example if we found out that now people are really into fantasy/romance, this is a useful signal for book writers to produce more of these books.)\n",
        "2. who are the most active book reviewers? Which user(s) leave the most comments? Whose comments are deemed most helpful to others? Can potentially hire them to be book critics to judge a new book when it comes out.\n",
        "3. cluster similar comments together to find patterns (think of k-means)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C3S9plBLKbfD",
        "colab_type": "text"
      },
      "source": [
        "For now let's stick with idea 3, finding clusters of similar comments. This is called topic modeling in NLP world. The upshot is that given a set of documents/sentences, we want to group the ones that are semantically similar. For example this is how Google News group similar news together. (of course Google's clustering algorithms might be far more complex than what we will use here.)\n",
        "\n",
        "Of course we immediately hit a wall here: **all ML models including simple K-means clustering expects numbers. How do we turn words and sentences into numbers???**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qTQrcVtp_mlF",
        "colab_type": "text"
      },
      "source": [
        "There are many ways to convert a sentence/document into a 'mathematical vector'. \n",
        "\n",
        "- The simplest way is to use a dictionary of words of vocabs. For example, if our vocab is $[hello, world, mother, dad]$ then a sentence like 'hello mother' would be represented by vector $[1, 0, 1, 0]$, and 'hello dad' would be represented by $[1, 0, 0, 1]$. I will call this approach bag of words approach.\n",
        "  - this approach is very simple, but downside is order of words is lost. For example in the above example 'mother hello' and 'hello mother' would be the same vector\n",
        "\n",
        "- another approach is the word2vec approach. word2vec uses a neural network to learn vector representation of words from a corpus of texts so that words with similar meanings are close together. Exactly how that happens is a topic for another time. \n",
        "- There is also another way of assigning \"importance\" to each word within a document in a corpus of documents using TF-IDF encoding. The essense is if a word appears more frequently within the document it is more important, and if the word appears less frequently outside of this document, then more important it is.\n",
        "  - TF stands for term frequency, and IDF stands for inverse document frequency.\n",
        "  - the logic for assinging a word more importance when it appears multiple times within one document is because if say \"scientist\" appears a lot in a newspaper, we can be more confident that this document is about \"science\". \n",
        "  - the logic for assigning a word less importance when it appears a lot across a lot of documents is because say the most frequent work in one document is \"the\", but \"the\" appears a lot in other documents as well, which means this word doesn't carry the meaning of this document, but more of grammatical choices. So we want these kinds of words to carry less weight.\n",
        "  - [example of calculating TF-IDF](http://www.tfidf.com/)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQCgFhU9IP28",
        "colab_type": "text"
      },
      "source": [
        "After some indecision I decide to use TF-IDF to convert words into numbers and do our clustering analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P83f3dZXJ0sD",
        "colab_type": "text"
      },
      "source": [
        "## 3.1 demonstrate calculating TF-IDF with python using a small example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D_NRe6NTJ-JQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "64b4d836-6e8f-483d-b0ee-1196568ab8f5"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "corpus = [\n",
        "          'I love python python is the best language',\n",
        "          'I love milkshake it is definitely the best',\n",
        "          'I am Hugh Sun I am the best'\n",
        "]\n",
        "\n",
        "tfIdfVectorizer=TfidfVectorizer(use_idf=True)\n",
        "tfIdf = tfIdfVectorizer.fit_transform(corpus)\n",
        "df = pd.DataFrame(tfIdf[0].T.todense(), index=tfIdfVectorizer.get_feature_names(), columns=[\"TF-IDF\"])\n",
        "df = df.sort_values('TF-IDF', ascending=False)\n",
        "print (df.head(25))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              TF-IDF\n",
            "python      0.763912\n",
            "language    0.381956\n",
            "is          0.290488\n",
            "love        0.290488\n",
            "best        0.225589\n",
            "the         0.225589\n",
            "am          0.000000\n",
            "definitely  0.000000\n",
            "hugh        0.000000\n",
            "it          0.000000\n",
            "milkshake   0.000000\n",
            "sun         0.000000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Ly7wsgP_dGh",
        "colab_type": "text"
      },
      "source": [
        "# Step 4. calculate TF-IDF for all comments."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwrNI-RhJFR4",
        "colab_type": "text"
      },
      "source": [
        "Let's start doing analysis only using a small portion of our 3 million data points, for the demonstration of how things work. Then we can scale our method to 3 million data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m86kpfKtIiHz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "small_data = book_review_data.loc[:10000, 'review_body'].tolist() # pick the first 10000 reviews for training"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q1iCUql3JVos",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "c16cc1b6-0ebf-4d56-8c1a-2c97cf76cb8e"
      },
      "source": [
        "small_data[:5] # look at first 5 reviews"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['this boook was a great one that you could learn from. it not only teaches the imponrtance of family and their values but it also deals with basic issues that teens and some kids even deal with.  this book is about 4 best friends who are for the first time in their lives spending their summer apart. one day they are all in one of the girls rooms and finds a pair of pants that were tucked away in her closet.  once all four  of them try them on they realize that there is really something special about these pants.  seeming as how all 4 girls are differnt shapes and sizes and somehow the pants fit all of them,  they realize that these pants were the start of something special.  immediatley following they decided to make up certian rules abut the pants such as you must write the best thing u did while wearing the pants over your summer on the right leg and also some silly things such as to \\\\\\\\\"never pick yuor nose while wearing the pants.\\\\\\\\\"  this book follows the girls throuh their summers in differnt places of the world and through all of the different obstacles that life takes them through. it can really teach you alot not only about what is going on around you but most imporntantly about yuorself.  i would give this book 4 stars and would reccommend it to anyone who seems the slihgtest bit interested.',\n",
              " \"If you are looking for something to stimulate your brain, this isn't it.  However, if you are just looking for a good laugh, you'll enjoy The Bad Girl's Guide.  It's funny and light, and definitely a good way to pass a little bit of time.\",\n",
              " 'never read it-a young relative idicated he liked it and somehow my name popped upon this--no more to say',\n",
              " 'Though she is honored to be Chicago Woman of the Year, Victoria Colby-Camp is more euphoric over the mental improvement that her son Jim has shown recently especially since he and Tasha North fell in love.  Jim was snatched almost twenty years ago when he was seven and turned into the killing Seth whose goal was to murder Victoria for abandoning him.  However, her elation would turn to despair if she knew Seth resurfaced and started to rape a pregnant Tasha.<br /><br />Former military strategist Daniel Marks is in town complements of the Colby Agency that wants to hire him.  Also in Chicago is Emily Hastings whose father a veteran homicide detective was murdered.  She finds letters linking her dad to Victoria, the woman\\'s long ago murdered first husband James, and her dad\\'s first partner Marelyn Rutland that confuses her.  Soon she will meet Daniel and they will be embroiled in the COLBY CONSPIRACY that goes back almost two decades ago.<br /><br />Though the subplots can become confusing at first, once the audience comprehends how this complex superb suspense thriller starts to come together, they will want more Colby Agency tales; (see FILES FROM THE COLBY AGENCY: THE BODYGUARD\\'S BABY PROTECTIVE CUSTODY).  The ensemble cast is solid as fans will feel with Victoria who has overcome so much tragedy, hope Jim \\\\\\\\\"defeats\\\\\\\\\" Seth with Tasha at his side, and root for Daniel and Emily to make it while wondering what really happened two decades ago.  A final twist marks a strong Webb of deceit tale that showcases a fine author on her A-game.<br /><br />Harriet Klausner',\n",
              " 'Review based on a cursory examination by University of Phoenix students in Philosophy 251.  <br />We have found the book to be well organized, and detailed. The structure placed the information in an easy to read and presentable manner. <br />The use of strong and sound proofs by experiment and examples supported the overall logic of reasoning in a clear and concise manner. <br />We have found the book to be well thought out and laid out so the reader can learn and follow what is being explained. <br />']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xZIVAuiOYFe4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# max_df : maximum document frequency a given word\n",
        "# min_df : minimum document frequency a given word\n",
        "# max_features: maximum number of words we want to extract.\n",
        "# stop_words : like \"the\", \"I\", ... this will allow tfidfVectorizer to ignore\n",
        "#     such meaningless words in its computation.\n",
        "# ngram_range: (1, 3) means the result will include 1-gram, 2-gram, 3-gram\n",
        "#     for this experiment we will just use ngram of 1, which means we only\n",
        "#     analyze in terms of single words.\n",
        "\n",
        "# create tfidf model\n",
        "tfidf = TfidfVectorizer(max_df=0.99, max_features=1000,\n",
        "                        min_df=0.01, stop_words='english',\n",
        "                        use_idf=True, ngram_range=(1,1))\n",
        "\n",
        "# fit model to our data, get back a matrix, each vector represents\n",
        "# one review. The numbers in the matrix are gotten by the tf-idf\n",
        "# calculation.\n",
        "tfidf_matrix = tfidf.fit_transform(small_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ChZNN6h9gcf_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "acd5bd95-fd70-449b-97ba-6299ecc7772b"
      },
      "source": [
        "# words that tfidf thinks are important in these documents.\n",
        "tfidf.get_feature_names()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['10',\n",
              " '100',\n",
              " '11',\n",
              " '12',\n",
              " '20',\n",
              " '2005',\n",
              " '30',\n",
              " '50',\n",
              " 'ability',\n",
              " 'able',\n",
              " 'absolutely',\n",
              " 'academic',\n",
              " 'accept',\n",
              " 'according',\n",
              " 'account',\n",
              " 'accurate',\n",
              " 'act',\n",
              " 'action',\n",
              " 'actual',\n",
              " 'actually',\n",
              " 'add',\n",
              " 'added',\n",
              " 'addition',\n",
              " 'admit',\n",
              " 'adult',\n",
              " 'adults',\n",
              " 'advanced',\n",
              " 'adventure',\n",
              " 'adventures',\n",
              " 'advice',\n",
              " 'age',\n",
              " 'ago',\n",
              " 'agree',\n",
              " 'alive',\n",
              " 'amazing',\n",
              " 'amazon',\n",
              " 'america',\n",
              " 'american',\n",
              " 'americans',\n",
              " 'analysis',\n",
              " 'ancient',\n",
              " 'answer',\n",
              " 'answers',\n",
              " 'anti',\n",
              " 'apparently',\n",
              " 'appear',\n",
              " 'appears',\n",
              " 'appreciate',\n",
              " 'approach',\n",
              " 'area',\n",
              " 'areas',\n",
              " 'aren',\n",
              " 'art',\n",
              " 'artist',\n",
              " 'aside',\n",
              " 'ask',\n",
              " 'aspects',\n",
              " 'attempt',\n",
              " 'attention',\n",
              " 'audience',\n",
              " 'author',\n",
              " 'authors',\n",
              " 'available',\n",
              " 'average',\n",
              " 'avoid',\n",
              " 'away',\n",
              " 'baby',\n",
              " 'background',\n",
              " 'bad',\n",
              " 'based',\n",
              " 'basic',\n",
              " 'basics',\n",
              " 'battle',\n",
              " 'beautiful',\n",
              " 'beautifully',\n",
              " 'beauty',\n",
              " 'began',\n",
              " 'begin',\n",
              " 'beginning',\n",
              " 'begins',\n",
              " 'behavior',\n",
              " 'believe',\n",
              " 'best',\n",
              " 'better',\n",
              " 'bible',\n",
              " 'big',\n",
              " 'biography',\n",
              " 'bit',\n",
              " 'black',\n",
              " 'body',\n",
              " 'book',\n",
              " 'books',\n",
              " 'boring',\n",
              " 'born',\n",
              " 'bought',\n",
              " 'boy',\n",
              " 'br',\n",
              " 'break',\n",
              " 'brief',\n",
              " 'brilliant',\n",
              " 'bring',\n",
              " 'brings',\n",
              " 'brother',\n",
              " 'brought',\n",
              " 'build',\n",
              " 'building',\n",
              " 'business',\n",
              " 'buy',\n",
              " 'buying',\n",
              " 'called',\n",
              " 'came',\n",
              " 'care',\n",
              " 'career',\n",
              " 'case',\n",
              " 'cases',\n",
              " 'caught',\n",
              " 'cause',\n",
              " 'cd',\n",
              " 'century',\n",
              " 'certain',\n",
              " 'certainly',\n",
              " 'chance',\n",
              " 'change',\n",
              " 'changed',\n",
              " 'changes',\n",
              " 'changing',\n",
              " 'chapter',\n",
              " 'chapters',\n",
              " 'character',\n",
              " 'characters',\n",
              " 'check',\n",
              " 'child',\n",
              " 'childhood',\n",
              " 'children',\n",
              " 'choice',\n",
              " 'christian',\n",
              " 'church',\n",
              " 'city',\n",
              " 'claims',\n",
              " 'class',\n",
              " 'classic',\n",
              " 'clear',\n",
              " 'clearly',\n",
              " 'close',\n",
              " 'cold',\n",
              " 'collection',\n",
              " 'college',\n",
              " 'color',\n",
              " 'come',\n",
              " 'comes',\n",
              " 'coming',\n",
              " 'comments',\n",
              " 'common',\n",
              " 'community',\n",
              " 'company',\n",
              " 'compelling',\n",
              " 'complete',\n",
              " 'completely',\n",
              " 'complex',\n",
              " 'comprehensive',\n",
              " 'concept',\n",
              " 'concepts',\n",
              " 'conclusion',\n",
              " 'consider',\n",
              " 'considered',\n",
              " 'contains',\n",
              " 'content',\n",
              " 'context',\n",
              " 'continue',\n",
              " 'continues',\n",
              " 'control',\n",
              " 'copy',\n",
              " 'correct',\n",
              " 'couldn',\n",
              " 'country',\n",
              " 'couple',\n",
              " 'course',\n",
              " 'cover',\n",
              " 'covered',\n",
              " 'covers',\n",
              " 'create',\n",
              " 'created',\n",
              " 'creating',\n",
              " 'creative',\n",
              " 'crime',\n",
              " 'cross',\n",
              " 'cultural',\n",
              " 'culture',\n",
              " 'current',\n",
              " 'cut',\n",
              " 'daily',\n",
              " 'dark',\n",
              " 'data',\n",
              " 'date',\n",
              " 'daughter',\n",
              " 'david',\n",
              " 'day',\n",
              " 'days',\n",
              " 'dead',\n",
              " 'deal',\n",
              " 'dealing',\n",
              " 'death',\n",
              " 'decided',\n",
              " 'decision',\n",
              " 'deep',\n",
              " 'deeply',\n",
              " 'definitely',\n",
              " 'depth',\n",
              " 'described',\n",
              " 'describes',\n",
              " 'description',\n",
              " 'descriptions',\n",
              " 'design',\n",
              " 'desire',\n",
              " 'despite',\n",
              " 'detailed',\n",
              " 'details',\n",
              " 'developed',\n",
              " 'development',\n",
              " 'dialogue',\n",
              " 'did',\n",
              " 'didn',\n",
              " 'died',\n",
              " 'difference',\n",
              " 'different',\n",
              " 'difficult',\n",
              " 'disappointed',\n",
              " 'discussion',\n",
              " 'does',\n",
              " 'doesn',\n",
              " 'dog',\n",
              " 'doing',\n",
              " 'don',\n",
              " 'doubt',\n",
              " 'dr',\n",
              " 'drawn',\n",
              " 'earlier',\n",
              " 'early',\n",
              " 'earth',\n",
              " 'easily',\n",
              " 'easy',\n",
              " 'eat',\n",
              " 'edition',\n",
              " 'editor',\n",
              " 'education',\n",
              " 'effective',\n",
              " 'effort',\n",
              " 'elements',\n",
              " 'emotional',\n",
              " 'emotions',\n",
              " 'end',\n",
              " 'ended',\n",
              " 'ending',\n",
              " 'ends',\n",
              " 'engaging',\n",
              " 'english',\n",
              " 'enjoy',\n",
              " 'enjoyable',\n",
              " 'enjoyed',\n",
              " 'entertaining',\n",
              " 'entire',\n",
              " 'entirely',\n",
              " 'era',\n",
              " 'errors',\n",
              " 'especially',\n",
              " 'essential',\n",
              " 'europe',\n",
              " 'event',\n",
              " 'events',\n",
              " 'eventually',\n",
              " 'everyday',\n",
              " 'evidence',\n",
              " 'evil',\n",
              " 'exactly',\n",
              " 'example',\n",
              " 'examples',\n",
              " 'excellent',\n",
              " 'exciting',\n",
              " 'expect',\n",
              " 'expected',\n",
              " 'experience',\n",
              " 'experienced',\n",
              " 'experiences',\n",
              " 'explain',\n",
              " 'explained',\n",
              " 'explains',\n",
              " 'explanation',\n",
              " 'extremely',\n",
              " 'eye',\n",
              " 'eyes',\n",
              " 'face',\n",
              " 'fact',\n",
              " 'facts',\n",
              " 'fair',\n",
              " 'fairly',\n",
              " 'faith',\n",
              " 'fall',\n",
              " 'familiar',\n",
              " 'family',\n",
              " 'famous',\n",
              " 'fan',\n",
              " 'fans',\n",
              " 'fantastic',\n",
              " 'fantasy',\n",
              " 'far',\n",
              " 'fascinating',\n",
              " 'fast',\n",
              " 'father',\n",
              " 'favorite',\n",
              " 'fear',\n",
              " 'feel',\n",
              " 'feeling',\n",
              " 'feelings',\n",
              " 'feels',\n",
              " 'felt',\n",
              " 'female',\n",
              " 'fiction',\n",
              " 'field',\n",
              " 'fight',\n",
              " 'figure',\n",
              " 'figures',\n",
              " 'filled',\n",
              " 'final',\n",
              " 'finally',\n",
              " 'finding',\n",
              " 'finds',\n",
              " 'fine',\n",
              " 'finish',\n",
              " 'finished',\n",
              " 'fit',\n",
              " 'focus',\n",
              " 'follow',\n",
              " 'following',\n",
              " 'follows',\n",
              " 'food',\n",
              " 'force',\n",
              " 'forced',\n",
              " 'form',\n",
              " 'format',\n",
              " 'forward',\n",
              " 'free',\n",
              " 'friend',\n",
              " 'friends',\n",
              " 'fully',\n",
              " 'fun',\n",
              " 'funny',\n",
              " 'future',\n",
              " 'game',\n",
              " 'gave',\n",
              " 'general',\n",
              " 'genius',\n",
              " 'genre',\n",
              " 'george',\n",
              " 'gets',\n",
              " 'getting',\n",
              " 'gift',\n",
              " 'girl',\n",
              " 'girls',\n",
              " 'given',\n",
              " 'gives',\n",
              " 'giving',\n",
              " 'glad',\n",
              " 'god',\n",
              " 'goes',\n",
              " 'going',\n",
              " 'gone',\n",
              " 'good',\n",
              " 'got',\n",
              " 'government',\n",
              " 'great',\n",
              " 'greater',\n",
              " 'greatest',\n",
              " 'ground',\n",
              " 'group',\n",
              " 'grow',\n",
              " 'growing',\n",
              " 'guess',\n",
              " 'guide',\n",
              " 'guy',\n",
              " 'guys',\n",
              " 'half',\n",
              " 'hand',\n",
              " 'hands',\n",
              " 'happen',\n",
              " 'happened',\n",
              " 'happens',\n",
              " 'happy',\n",
              " 'hard',\n",
              " 'hate',\n",
              " 'haven',\n",
              " 'having',\n",
              " 'head',\n",
              " 'health',\n",
              " 'hear',\n",
              " 'heard',\n",
              " 'heart',\n",
              " 'help',\n",
              " 'helped',\n",
              " 'helpful',\n",
              " 'helps',\n",
              " 'hero',\n",
              " 'high',\n",
              " 'highly',\n",
              " 'historical',\n",
              " 'history',\n",
              " 'hit',\n",
              " 'hold',\n",
              " 'home',\n",
              " 'honest',\n",
              " 'hope',\n",
              " 'hot',\n",
              " 'hours',\n",
              " 'house',\n",
              " 'huge',\n",
              " 'human',\n",
              " 'humor',\n",
              " 'husband',\n",
              " 'idea',\n",
              " 'ideas',\n",
              " 'ii',\n",
              " 'illustrations',\n",
              " 'images',\n",
              " 'imagination',\n",
              " 'imagine',\n",
              " 'immediately',\n",
              " 'important',\n",
              " 'include',\n",
              " 'included',\n",
              " 'includes',\n",
              " 'including',\n",
              " 'individual',\n",
              " 'industry',\n",
              " 'info',\n",
              " 'information',\n",
              " 'informative',\n",
              " 'inside',\n",
              " 'insight',\n",
              " 'insights',\n",
              " 'inspired',\n",
              " 'instead',\n",
              " 'instructions',\n",
              " 'intelligent',\n",
              " 'interested',\n",
              " 'interesting',\n",
              " 'introduction',\n",
              " 'involved',\n",
              " 'isn',\n",
              " 'issue',\n",
              " 'issues',\n",
              " 'james',\n",
              " 'jesus',\n",
              " 'job',\n",
              " 'john',\n",
              " 'journey',\n",
              " 'just',\n",
              " 'keeps',\n",
              " 'kept',\n",
              " 'key',\n",
              " 'kid',\n",
              " 'kids',\n",
              " 'kind',\n",
              " 'king',\n",
              " 'knew',\n",
              " 'know',\n",
              " 'knowing',\n",
              " 'knowledge',\n",
              " 'known',\n",
              " 'knows',\n",
              " 'lack',\n",
              " 'land',\n",
              " 'language',\n",
              " 'large',\n",
              " 'late',\n",
              " 'later',\n",
              " 'laugh',\n",
              " 'law',\n",
              " 'lead',\n",
              " 'leads',\n",
              " 'learn',\n",
              " 'learned',\n",
              " 'learning',\n",
              " 'leave',\n",
              " 'leaves',\n",
              " 'left',\n",
              " 'lessons',\n",
              " 'let',\n",
              " 'level',\n",
              " 'library',\n",
              " 'life',\n",
              " 'light',\n",
              " 'like',\n",
              " 'liked',\n",
              " 'likely',\n",
              " 'limited',\n",
              " 'line',\n",
              " 'lines',\n",
              " 'list',\n",
              " 'literary',\n",
              " 'literature',\n",
              " 'little',\n",
              " 'live',\n",
              " 'lived',\n",
              " 'lives',\n",
              " 'living',\n",
              " 'll',\n",
              " 'local',\n",
              " 'long',\n",
              " 'longer',\n",
              " 'look',\n",
              " 'looked',\n",
              " 'looking',\n",
              " 'looks',\n",
              " 'lord',\n",
              " 'loss',\n",
              " 'lost',\n",
              " 'lot',\n",
              " 'lots',\n",
              " 'love',\n",
              " 'loved',\n",
              " 'loves',\n",
              " 'magic',\n",
              " 'main',\n",
              " 'major',\n",
              " 'make',\n",
              " 'makes',\n",
              " 'making',\n",
              " 'male',\n",
              " 'man',\n",
              " 'manner',\n",
              " 'mark',\n",
              " 'market',\n",
              " 'marriage',\n",
              " 'married',\n",
              " 'master',\n",
              " 'material',\n",
              " 'matter',\n",
              " 'maybe',\n",
              " 'mean',\n",
              " 'meaning',\n",
              " 'means',\n",
              " 'medical',\n",
              " 'meet',\n",
              " 'meets',\n",
              " 'members',\n",
              " 'men',\n",
              " 'mention',\n",
              " 'mentioned',\n",
              " 'message',\n",
              " 'met',\n",
              " 'method',\n",
              " 'methods',\n",
              " 'michael',\n",
              " 'middle',\n",
              " 'military',\n",
              " 'mind',\n",
              " 'miss',\n",
              " 'missing',\n",
              " 'modern',\n",
              " 'mom',\n",
              " 'moment',\n",
              " 'moments',\n",
              " 'money',\n",
              " 'month',\n",
              " 'months',\n",
              " 'mother',\n",
              " 'moves',\n",
              " 'movie',\n",
              " 'moving',\n",
              " 'mr',\n",
              " 'ms',\n",
              " 'murder',\n",
              " 'music',\n",
              " 'mystery',\n",
              " 'named',\n",
              " 'names',\n",
              " 'narrative',\n",
              " 'national',\n",
              " 'natural',\n",
              " 'nature',\n",
              " 'near',\n",
              " 'nearly',\n",
              " 'necessary',\n",
              " 'need',\n",
              " 'needed',\n",
              " 'needs',\n",
              " 'negative',\n",
              " 'new',\n",
              " 'news',\n",
              " 'nice',\n",
              " 'night',\n",
              " 'non',\n",
              " 'note',\n",
              " 'notes',\n",
              " 'novel',\n",
              " 'novels',\n",
              " 'number',\n",
              " 'numerous',\n",
              " 'obvious',\n",
              " 'obviously',\n",
              " 'offer',\n",
              " 'offers',\n",
              " 'oh',\n",
              " 'ok',\n",
              " 'old',\n",
              " 'older',\n",
              " 'ones',\n",
              " 'open',\n",
              " 'opinion',\n",
              " 'opportunity',\n",
              " 'order',\n",
              " 'organized',\n",
              " 'original',\n",
              " 'outside',\n",
              " 'overall',\n",
              " 'overview',\n",
              " 'page',\n",
              " 'pages',\n",
              " 'pain',\n",
              " 'paper',\n",
              " 'parents',\n",
              " 'particular',\n",
              " 'particularly',\n",
              " 'parts',\n",
              " 'party',\n",
              " 'pass',\n",
              " 'passion',\n",
              " 'past',\n",
              " 'path',\n",
              " 'pay',\n",
              " 'people',\n",
              " 'perfect',\n",
              " 'period',\n",
              " 'person',\n",
              " 'personal',\n",
              " 'personally',\n",
              " 'perspective',\n",
              " 'philosophy',\n",
              " 'photographs',\n",
              " 'photos',\n",
              " 'physical',\n",
              " 'pick',\n",
              " 'picked',\n",
              " 'picture',\n",
              " 'pictures',\n",
              " 'piece',\n",
              " 'place',\n",
              " 'places',\n",
              " 'plan',\n",
              " 'play',\n",
              " 'playing',\n",
              " 'plenty',\n",
              " 'plot',\n",
              " 'plus',\n",
              " 'point',\n",
              " 'points',\n",
              " 'police',\n",
              " 'political',\n",
              " 'politics',\n",
              " 'poor',\n",
              " 'popular',\n",
              " 'position',\n",
              " 'positive',\n",
              " 'possible',\n",
              " 'possibly',\n",
              " 'post',\n",
              " 'potential',\n",
              " 'power',\n",
              " 'powerful',\n",
              " 'practical',\n",
              " 'practice',\n",
              " 'present',\n",
              " 'presented',\n",
              " 'presents',\n",
              " 'pretty',\n",
              " 'previous',\n",
              " 'price',\n",
              " 'print',\n",
              " 'probably',\n",
              " 'problem',\n",
              " 'problems',\n",
              " 'process',\n",
              " 'product',\n",
              " 'professional',\n",
              " 'professor',\n",
              " 'program',\n",
              " 'project',\n",
              " 'prose',\n",
              " 'prove',\n",
              " 'provide',\n",
              " 'provided',\n",
              " 'provides',\n",
              " 'public',\n",
              " 'published',\n",
              " 'purchase',\n",
              " 'purchased',\n",
              " 'purpose',\n",
              " 'quality',\n",
              " 'question',\n",
              " 'questions',\n",
              " 'quick',\n",
              " 'quickly',\n",
              " 'quite',\n",
              " 'range',\n",
              " 'rate',\n",
              " 'read',\n",
              " 'reader',\n",
              " 'readers',\n",
              " 'reading',\n",
              " 'reads',\n",
              " 'ready',\n",
              " 'real',\n",
              " 'realistic',\n",
              " 'reality',\n",
              " 'realize',\n",
              " 'really',\n",
              " 'reason',\n",
              " 'reasons',\n",
              " 'received',\n",
              " 'recent',\n",
              " 'recently',\n",
              " 'recipes',\n",
              " 'recommend',\n",
              " 'recommended',\n",
              " 'red',\n",
              " 'reference',\n",
              " 'references',\n",
              " 'regarding',\n",
              " 'relate',\n",
              " 'related',\n",
              " 'relationship',\n",
              " 'relationships',\n",
              " 'religion',\n",
              " 'religious',\n",
              " 'remains',\n",
              " 'remember',\n",
              " 'required',\n",
              " 'research',\n",
              " 'resource',\n",
              " 'respect',\n",
              " 'rest',\n",
              " 'result',\n",
              " 'results',\n",
              " 'return',\n",
              " 'review',\n",
              " 'reviewer',\n",
              " 'reviewers',\n",
              " 'reviews',\n",
              " 'rich',\n",
              " 'right',\n",
              " 'robert',\n",
              " 'role',\n",
              " 'romance',\n",
              " 'room',\n",
              " 'run',\n",
              " 'running',\n",
              " 'sad',\n",
              " 'said',\n",
              " 'save',\n",
              " 'saw',\n",
              " 'say',\n",
              " 'saying',\n",
              " 'says',\n",
              " 'scene',\n",
              " 'scenes',\n",
              " 'school',\n",
              " 'science',\n",
              " 'scientific',\n",
              " 'search',\n",
              " 'second',\n",
              " 'secret',\n",
              " 'section',\n",
              " 'sections',\n",
              " 'seeing',\n",
              " 'seen',\n",
              " 'sees',\n",
              " 'self',\n",
              " 'sense',\n",
              " 'sequel',\n",
              " 'series',\n",
              " 'seriously',\n",
              " 'service',\n",
              " 'set',\n",
              " 'setting',\n",
              " 'sex',\n",
              " 'sexual',\n",
              " 'share',\n",
              " 'short',\n",
              " 'showing',\n",
              " 'shows',\n",
              " 'similar',\n",
              " 'simple',\n",
              " 'simply',\n",
              " 'single',\n",
              " 'sister',\n",
              " 'situation',\n",
              " 'situations',\n",
              " 'size',\n",
              " 'skills',\n",
              " 'slow',\n",
              " 'small',\n",
              " 'social',\n",
              " 'society',\n",
              " 'somewhat',\n",
              " 'son',\n",
              " 'soon',\n",
              " 'sorry',\n",
              " 'sort',\n",
              " 'soul',\n",
              " 'sound',\n",
              " 'source',\n",
              " 'sources',\n",
              " 'space',\n",
              " 'speak',\n",
              " 'special',\n",
              " 'specific',\n",
              " 'spend',\n",
              " 'spent',\n",
              " 'spirit',\n",
              " 'spiritual',\n",
              " 'stand',\n",
              " 'standard',\n",
              " 'star',\n",
              " 'stars',\n",
              " 'start',\n",
              " 'started',\n",
              " 'starting',\n",
              " 'starts',\n",
              " 'state',\n",
              " 'states',\n",
              " 'stay',\n",
              " 'step',\n",
              " 'steps',\n",
              " 'stop',\n",
              " 'stories',\n",
              " 'story',\n",
              " 'straight',\n",
              " 'strength',\n",
              " 'strong',\n",
              " 'struggle',\n",
              " 'student',\n",
              " 'students',\n",
              " 'studies',\n",
              " 'study',\n",
              " 'stuff',\n",
              " 'style',\n",
              " 'subject',\n",
              " 'success',\n",
              " 'successful',\n",
              " 'suggest',\n",
              " 'support',\n",
              " 'supposed',\n",
              " 'sure',\n",
              " 'surprise',\n",
              " 'surprised',\n",
              " 'suspense',\n",
              " 'table',\n",
              " 'taken',\n",
              " 'takes',\n",
              " 'taking',\n",
              " 'tale',\n",
              " 'tales',\n",
              " 'talk',\n",
              " 'talking',\n",
              " 'taught',\n",
              " 'teach',\n",
              " 'teacher',\n",
              " 'teaching',\n",
              " 'team',\n",
              " 'technical',\n",
              " 'techniques',\n",
              " 'technology',\n",
              " 'tell',\n",
              " 'telling',\n",
              " 'tells',\n",
              " 'terms',\n",
              " 'test',\n",
              " 'text',\n",
              " 'thank',\n",
              " 'thanks',\n",
              " 'theme',\n",
              " 'theory',\n",
              " 'thing',\n",
              " 'things',\n",
              " 'think',\n",
              " 'thinking',\n",
              " 'thoroughly',\n",
              " 'thought',\n",
              " 'thoughts',\n",
              " 'time',\n",
              " 'times',\n",
              " 'tips',\n",
              " 'title',\n",
              " 'today',\n",
              " 'told',\n",
              " 'tone',\n",
              " 'took',\n",
              " 'tool',\n",
              " 'tools',\n",
              " 'topic',\n",
              " 'topics',\n",
              " 'totally',\n",
              " 'touch',\n",
              " 'town',\n",
              " 'traditional',\n",
              " 'training',\n",
              " 'travel',\n",
              " 'tried',\n",
              " 'tries',\n",
              " 'trip',\n",
              " 'trouble',\n",
              " 'true',\n",
              " 'truly',\n",
              " 'truth',\n",
              " 'try',\n",
              " 'trying',\n",
              " 'turn',\n",
              " 'turned',\n",
              " 'turning',\n",
              " 'turns',\n",
              " 'tv',\n",
              " 'twists',\n",
              " 'type',\n",
              " 'types',\n",
              " 'typical',\n",
              " 'ultimately',\n",
              " 'understand',\n",
              " 'understanding',\n",
              " 'unfortunately',\n",
              " 'unique',\n",
              " 'united',\n",
              " 'university',\n",
              " 'unless',\n",
              " 'unlike',\n",
              " 'use',\n",
              " 'used',\n",
              " 'useful',\n",
              " 'uses',\n",
              " 'using',\n",
              " 'usual',\n",
              " 'usually',\n",
              " 'valuable',\n",
              " 'value',\n",
              " 'various',\n",
              " 've',\n",
              " 'version',\n",
              " 'view',\n",
              " 'views',\n",
              " 'visit',\n",
              " 'voice',\n",
              " 'volume',\n",
              " 'wait',\n",
              " 'waiting',\n",
              " 'want',\n",
              " 'wanted',\n",
              " 'wanting',\n",
              " 'wants',\n",
              " 'war',\n",
              " 'wasn',\n",
              " 'waste',\n",
              " 'way',\n",
              " 'ways',\n",
              " 'web',\n",
              " 'week',\n",
              " 'weeks',\n",
              " 'went',\n",
              " 'west',\n",
              " 'western',\n",
              " 'white',\n",
              " 'wife',\n",
              " 'wisdom',\n",
              " 'wish',\n",
              " 'woman',\n",
              " 'women',\n",
              " 'won',\n",
              " 'wonder',\n",
              " 'wonderful',\n",
              " 'word',\n",
              " 'words',\n",
              " 'work',\n",
              " 'worked',\n",
              " 'working',\n",
              " 'works',\n",
              " 'world',\n",
              " 'worse',\n",
              " 'worst',\n",
              " 'worth',\n",
              " 'wouldn',\n",
              " 'write',\n",
              " 'writer',\n",
              " 'writers',\n",
              " 'writes',\n",
              " 'writing',\n",
              " 'written',\n",
              " 'wrong',\n",
              " 'wrote',\n",
              " 'year',\n",
              " 'years',\n",
              " 'yes',\n",
              " 'york',\n",
              " 'young']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cBT3Gl-pgkNU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 459
        },
        "outputId": "f5283b2e-931c-4aee-f9ab-fcaeb9499957"
      },
      "source": [
        "# print the most important 25 words.\n",
        "df = pd.DataFrame(tfidf_matrix[0].T.todense(), index= tfidf.get_feature_names(), columns=[\"TF-IDF\"])\n",
        "df = df.sort_values('TF-IDF', ascending=False)\n",
        "print (df.head(25))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              TF-IDF\n",
            "girls       0.477401\n",
            "special     0.292233\n",
            "realize     0.291920\n",
            "best        0.181828\n",
            "really      0.163492\n",
            "fit         0.159627\n",
            "follows     0.159134\n",
            "teach       0.148231\n",
            "decided     0.146912\n",
            "following   0.146116\n",
            "places      0.141082\n",
            "finds       0.141082\n",
            "pick        0.140299\n",
            "kids        0.138189\n",
            "deal        0.132640\n",
            "basic       0.131954\n",
            "issues      0.130536\n",
            "stars       0.128833\n",
            "try         0.126572\n",
            "interested  0.123658\n",
            "start       0.122937\n",
            "write       0.121546\n",
            "lives       0.120348\n",
            "friends     0.120153\n",
            "takes       0.117081\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z-49PFwihLUe",
        "colab_type": "text"
      },
      "source": [
        "As we can see, the most \"important\" words in the first 10000 reviews about books are \"girls\", \"special\", ... This matches our intuition that a lot of the books are about teenage girls, like the first book we saw earlier. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HJhrB6xSihvJ",
        "colab_type": "text"
      },
      "source": [
        "## 4.1 find the most similar review to the first review"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7M8ej4mivt_",
        "colab_type": "text"
      },
      "source": [
        "We can use cosinge similarity function and our TF-IDF matrix to calculate the most similar review to a given review."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R2C95ktshBVc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "similarity_matrix = cosine_similarity(tfidf_matrix)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rLE1ONfmi9Zb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "78eb8028-5615-474a-be46-f779e578671e"
      },
      "source": [
        "similarity_matrix"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1.        , 0.0347547 , 0.        , ..., 0.04832078, 0.0561037 ,\n",
              "        0.00930388],\n",
              "       [0.0347547 , 1.        , 0.        , ..., 0.02462948, 0.        ,\n",
              "        0.04230847],\n",
              "       [0.        , 0.        , 1.        , ..., 0.        , 0.        ,\n",
              "        0.        ],\n",
              "       ...,\n",
              "       [0.04832078, 0.02462948, 0.        , ..., 1.        , 0.15238882,\n",
              "        0.03626608],\n",
              "       [0.0561037 , 0.        , 0.        , ..., 0.15238882, 1.        ,\n",
              "        0.00859655],\n",
              "       [0.00930388, 0.04230847, 0.        , ..., 0.03626608, 0.00859655,\n",
              "        1.        ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gVzxtFcqi_GA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "37a49de6-15e0-4753-a9ed-8dfa223e56dc"
      },
      "source": [
        "# similarity of first review to all the other reviews.\n",
        "similarity_matrix[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1.        , 0.0347547 , 0.        , ..., 0.04832078, 0.0561037 ,\n",
              "       0.00930388])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "twneR_rjj-zU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# given an array like [1, 0.8, 0.9], return 2\n",
        "# because 0.9 is the greatest. Exclude the\n",
        "# entry with 1.\n",
        "def findTheIndexOfLargestSimilarity(array):\n",
        "  largestSimilarity, index = -float('inf'), -1\n",
        "  for i in range(len(array)):\n",
        "    if array[i] >= 1:\n",
        "      continue\n",
        "    if array[i] > largestSimilarity:\n",
        "      largestSimilarity = array[i]\n",
        "      index = i\n",
        "  return index"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hJa7IDaok0Vc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d2dd7310-e9a9-43ba-d682-510727529f51"
      },
      "source": [
        "findTheIndexOfLargestSimilarity(similarity_matrix[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2991"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cejIdIC8k38a",
        "colab_type": "text"
      },
      "source": [
        "So we found the most simlar comment to 0th comment is 2991 comment. Let's look at them and see if they are really similar."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i-diWjZlk7Eo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "69c2b772-6822-4d97-df98-c02b37366ed1"
      },
      "source": [
        "small_data[2991]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Ideal for ages 8 and up.  Helps girls to deal with friendships and difficulties which arise.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kh3xUIEJlNcr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "outputId": "2eca46ce-9e8a-464f-d377-15cb64ffe678"
      },
      "source": [
        "small_data[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'this boook was a great one that you could learn from. it not only teaches the imponrtance of family and their values but it also deals with basic issues that teens and some kids even deal with.  this book is about 4 best friends who are for the first time in their lives spending their summer apart. one day they are all in one of the girls rooms and finds a pair of pants that were tucked away in her closet.  once all four  of them try them on they realize that there is really something special about these pants.  seeming as how all 4 girls are differnt shapes and sizes and somehow the pants fit all of them,  they realize that these pants were the start of something special.  immediatley following they decided to make up certian rules abut the pants such as you must write the best thing u did while wearing the pants over your summer on the right leg and also some silly things such as to \\\\\\\\\"never pick yuor nose while wearing the pants.\\\\\\\\\"  this book follows the girls throuh their summers in differnt places of the world and through all of the different obstacles that life takes them through. it can really teach you alot not only about what is going on around you but most imporntantly about yuorself.  i would give this book 4 stars and would reccommend it to anyone who seems the slihgtest bit interested.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S8TTon-BlaMk",
        "colab_type": "text"
      },
      "source": [
        "Interesting, don't really see how these two comments relate to each other, but they are both positive reviews."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BP2WG82Mlfdt",
        "colab_type": "text"
      },
      "source": [
        "Let's take look on another one, let's find the most similar review to review #2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_9pT2rNflKQz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "997e2340-be0d-46a5-9cf9-0762fa9e5325"
      },
      "source": [
        "findTheIndexOfLargestSimilarity(similarity_matrix[2])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5860"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v9g9OEFBl0I2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b52cf8ee-0b3d-4b09-9ec5-cbcc9408bb67"
      },
      "source": [
        "small_data[2]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'never read it-a young relative idicated he liked it and somehow my name popped upon this--no more to say'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XAdXf7uRl6VU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "2d6ad004-30ac-4721-f5a7-488972055e3f"
      },
      "source": [
        "small_data[5860]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'This was my first book that I have read about the Beatles.  All I have to say is: very well done, Cynthia!'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZfTUjwK4mR6n",
        "colab_type": "text"
      },
      "source": [
        "These two comments are somewhat similar? althuogh first one's emotion is neutral and second one is positive. Not saying this cosine similarity thing is working the best, but it is something right?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TOYlg3l2mns-",
        "colab_type": "text"
      },
      "source": [
        "# 5. K-means"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6yEVF8ZkmvTt",
        "colab_type": "text"
      },
      "source": [
        "Now let's do a k-means clustering algorithm on our TF-IDF matrix, to find what kinds of comments \"lump\" together."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rF2jr5wul7ai",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.cluster import KMeans\n",
        "\n",
        "# this takes quite a while... 1 minute ish\n",
        "km = KMeans(n_clusters = 5)\n",
        "km.fit(tfidf_matrix)\n",
        "\n",
        "clusters = km.labels_.tolist()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IVUUSQK7nuhf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d0d22f1d-90a1-4479-b76e-4d393678c3fc"
      },
      "source": [
        "clusters[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[4, 4, 0, 3, 3, 2, 2, 4, 4, 4]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DORCGFeloQk-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "bfccc4e4-afd0-4d71-8c44-ca61fafbd229"
      },
      "source": [
        "# these 5 1000-vectors are the centroids of our kmeans\n",
        "# it is a 1000-vector because we chose 1000 numbers\n",
        "# to describe a document.\n",
        "km.cluster_centers_"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.00274416, 0.00132513, 0.00206833, ..., 0.00306006, 0.00086566,\n",
              "        0.00573727],\n",
              "       [0.00381079, 0.00446254, 0.0040167 , ..., 0.0045068 , 0.00355097,\n",
              "        0.00915503],\n",
              "       [0.00183441, 0.00128342, 0.0015219 , ..., 0.00222807, 0.00265311,\n",
              "        0.01469605],\n",
              "       [0.00432678, 0.00291198, 0.00262661, ..., 0.00514683, 0.00335108,\n",
              "        0.00988328],\n",
              "       [0.00303853, 0.00165133, 0.00124466, ..., 0.00354047, 0.00184363,\n",
              "        0.00517786]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PnL2WUbNo5hI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's append the cluster number to each comment\n",
        "clusteredComments = pd.DataFrame(data = {'review_body': small_data, 'cluster_num': clusters})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e3zd97LTpP_H",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "outputId": "68838df2-7ad6-42fe-ce6b-7cb6ad419250"
      },
      "source": [
        "clusteredComments"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review_body</th>\n",
              "      <th>cluster_num</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>this boook was a great one that you could lear...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>If you are looking for something to stimulate ...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>never read it-a young relative idicated he lik...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Though she is honored to be Chicago Woman of t...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Review based on a cursory examination by Unive...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>Even though Carole went through some painful t...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>My twin boys love this book....especially the ...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>As a writer aspiring to be published myself th...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>I enjoyed enormously this book: it doesnt try ...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10000</th>\n",
              "      <td>This is a good book for beginners as well as t...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>10001 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                             review_body  cluster_num\n",
              "0      this boook was a great one that you could lear...            4\n",
              "1      If you are looking for something to stimulate ...            4\n",
              "2      never read it-a young relative idicated he lik...            0\n",
              "3      Though she is honored to be Chicago Woman of t...            3\n",
              "4      Review based on a cursory examination by Unive...            3\n",
              "...                                                  ...          ...\n",
              "9996   Even though Carole went through some painful t...            0\n",
              "9997   My twin boys love this book....especially the ...            4\n",
              "9998   As a writer aspiring to be published myself th...            3\n",
              "9999   I enjoyed enormously this book: it doesnt try ...            4\n",
              "10000  This is a good book for beginners as well as t...            4\n",
              "\n",
              "[10001 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IMPqCT-bp06k",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "outputId": "73645281-82c3-4295-b3cd-7ce5a972ddf0"
      },
      "source": [
        "# look at all reviews inside of 0th cluster\n",
        "clusteredComments[clusteredComments['cluster_num'] == 0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review_body</th>\n",
              "      <th>cluster_num</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>never read it-a young relative idicated he lik...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>The Outlander series are my favorite books of ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>Genuinely witty and fun.  I really liked the i...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>I just want to say, that for a used book, this...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>\\\\\"Last Chance to See\\\\\" is Douglas Adams' hil...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9980</th>\n",
              "      <td>Charles and Melanie Fraser seem like a couple ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9988</th>\n",
              "      <td>I read this in 2 hours. It's a boring, short b...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9990</th>\n",
              "      <td>I found this book to be an inspiration. In a w...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9994</th>\n",
              "      <td>My 3 year old daughter enjoys this book.  We r...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>Even though Carole went through some painful t...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1367 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            review_body  cluster_num\n",
              "2     never read it-a young relative idicated he lik...            0\n",
              "23    The Outlander series are my favorite books of ...            0\n",
              "30    Genuinely witty and fun.  I really liked the i...            0\n",
              "42    I just want to say, that for a used book, this...            0\n",
              "43    \\\\\"Last Chance to See\\\\\" is Douglas Adams' hil...            0\n",
              "...                                                 ...          ...\n",
              "9980  Charles and Melanie Fraser seem like a couple ...            0\n",
              "9988  I read this in 2 hours. It's a boring, short b...            0\n",
              "9990  I found this book to be an inspiration. In a w...            0\n",
              "9994  My 3 year old daughter enjoys this book.  We r...            0\n",
              "9996  Even though Carole went through some painful t...            0\n",
              "\n",
              "[1367 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CX3SSFIBqp0X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "79bcf42e-0a86-45f0-9311-f8c4eeb78484"
      },
      "source": [
        "# print out the 5 reviews in 0-th cluster\n",
        "for i in range(5):\n",
        "  print(clusteredComments[clusteredComments['cluster_num'] == 0].iloc[i][0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "never read it-a young relative idicated he liked it and somehow my name popped upon this--no more to say\n",
            "The Outlander series are my favorite books of all time, period.  But after reading the Fiery Cross, I was a bit let down, as many people were, and afraid that the rest of the series would just continue to go downhill.  So when A Breath of Snow and Ashes came out, I was so glad to see that it was getting good reviews.  I finally got to read it this week and was not disappointed in the least. <br />All the characters we know and love are back, and Brianna seems to have finally grow up.  She has been my least favorite character since the second book, but this time around she was actually likeable (for the most part).  At certain points this book became so intense that I actually had to put it down and take a break for awhile, but not all readers will have this experience, only ones that are practically obsessed with this series (and who's friends have to remind them, \\\\\"Jamie is NOT a real person!).    <br />So basically, if you have read the previous books, just read this one. Please.  You won't regret it.  The only reason I didn't give this one five stars is that I think nothing can top Outlander and Dragonfly in Amber.  If I could give this 4.5, I would. <br /> <br />One thing I have wondered about - if Claire dies back in the 1700's, then she'll be born again in 1918, go back in time in 1946, die in the 1700's, be born in 1918, and just keep repeating this forever.  For all she knows this could be her tenth time doing this.  I haven't seen this matter dealt with in the books.  Did I just miss it or something?\n",
            "Genuinely witty and fun.  I really liked the incorporation of the cartoon and quotations into the book, this made it more interesting and gave a better perspective on the chapters.  An uplifting read from beginning to end.\n",
            "I just want to say, that for a used book, this book arrived in great condition.  I think that it even came in shrink wrap!  I'm pleased with my purchase and am glad that I don't have to read through other people's notes and marks as I study for my class.  Good job!\n",
            "\\\\\"Last Chance to See\\\\\" is Douglas Adams' hilarious and though-provoking non-fiction, eco-travel narrative about circling the globe in search of some of the Earth's most endangered animals. We get to see a side of Adams that is deeply concerned with the plight of the beautiful, unique, and often ludicrously implausible lifeforms on our planet -- the same side that prompted him to climb Mount Kilimanjaro in a rhino suit.     What made this book all the more meaningful to me was that I'd thought I'd finished all of Douglas Adams' books long ago. I loved them all, but I never fully appreciated the depth of his genius as a humanist and social satirist until, sadly, he had passed away.  What's more, I never realized how much Adams' writings meant to me until I was hit with the cold, hard reality that he'd written all he ever would.  Douglas Adams had hitched his last ride across the galaxy, and I'd never hear from him again.  Or so I thought.    Infinitely improbable as it may seem, I heard from Douglas Adams again about two weeks ago.  I was browsing around in Borders, long since resigned to the fact that I'd read the last \\\\\"new\\\\\" Douglas Adams book I ever would, when I stumbled upon \\\\\"Last Chance to See.\\\\\" I felt like I'd gone into the pet store to buy a parakeet and, instead, found a dodo bird.      The endangered species Douglas Adams criss-crossed the globe in search of are all, in their own ways, remarkable and priceless. He urges us not to wait until they're gone to fully appreciate them, and he drives this point home to today's readers, in his own ironic style, by now being himself \\\\\"extinct.\\\\\"    If you're a Douglas Adams fan, and you've never read this book, here it is -- your own \\\\\"last chance to see\\\\\" (or, rather, read) him.  His own voice, his own words, his own experiences.  For me, it was a fond farewell from an old friend I'd never met, who, from his new home -- far beyond life, the universe, and everything -- telling me, \\\\\"Yes, I'm dead. But there are still a few rhinos and gorillas and dolphins that aren't. Do me a favor; make sure they stay that way.\\\\\"\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a0fzLJP9reQC",
        "colab_type": "text"
      },
      "source": [
        "The comments are above seem to be positive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h9ubEb5zq7jF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "ceca0f97-c8b1-46b9-e3f8-0b8f5c772da7"
      },
      "source": [
        "# let's look at 5 commments in cluster 1\n",
        "for i in range(5):\n",
        "  print(clusteredComments[clusteredComments['cluster_num'] == 1].iloc[i][0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Hands down, the funniest book I've ever read.  It's touching, frustrating, and poignant.  I even related to this character, and I'm a 35 year-old woman.    <br /> <br />Read it, you won't regret it! <br /> <br />\n",
            "\\\\\"Informative and entertaining.\\\\\" --\\\\\"Home\\\\\" <br /> <br />\\\\\"It inspires.  As you turn the pages, you can see that nothing is impossible.\\\\\" <br />--\\\\\"House Beautiful\\\\\"\n",
            "The issue with opinion like his is that atheism is easy. IN complete arrogance any 8th grader can clearly refute the possibilites of the scriptures using 8th grade science! <br /> <br />What Earl misses is the basis for all believers.  <br /> <br />                      \\\\\"Faith\\\\\" <br /> <br />Scientists have no faith seeking earthy proof. The ultimate arrogance is the man who can refute the smallest evidance of something he does not understand. Something that requires \\\\\"faith\\\\\" above all.  <br /> <br />Without faith this book is just as moot as the book by Sobel. Without faith. Neither is relavant. <br /> <br />Tom B\n",
            "Isabella Zophie & Le Cirque de Magique is an excellent book. I would say that it is one of the best books that I have ever read. It is very exciting and I enjoyed the mystery that was in the story. This is one of the most adventurous books that I have ever read.  <br /> <br />This book is about a ten year old girl, Isabella Zophie, who finds the Locket of Fire and Water in her great aunt's house and when she puts it on she is transported to another world - Zartarbia. She meets a circus which has lots of great characters including Mr Krock O'Dile who is a crocodile that runs the circus.  Isabella is chased by one of the syndicate, Mistress Sectica.  The Syndicate are a group of villains who are trying to control Zartarbia.  Fighting against the syndicate is the Federation Against Dark Forces (FADF) with agents trained in some of the coolest fighting techniques.   <br /> <br />Zartarbia is a magical world where nothing is as it seems. There is a crazy character called the Thinker who is a complete lunatic and very funny.  The first chapter is a bit confusing but you need to keep reading as it is important for later in the book.  <br /> <br />I think that all genders will like this book. Some parts in this book are very funny. If you are interested in fantasy and adventure, then you will enjoy this book.  <br /> <br />I would recommend this book to anyone.  <br />\n",
            "I've read almost every book written on the Beatles and found Cynthia's story poignant and from the heart.  She's had a hard time finding her true self in the shadows of her larger-than-life men and marriages and she persists despite the setbacks and enjoys the adventures. <br /> <br />The first half of the book is almost identical to her earlier book, Twist of Lennon in 1989.  I've not read much about her or Julian from 1980 on and it is interesting to look at behind-the-scenes reflections vs. what the public sees.  Julian: Sing again in your own key and forget about Them. <br /> <br />If you are a '42-years And Counting' Beatles fan like me, then  you will enjoy this book.  There are some unreleased pictues you'll relish from Cyn's family album. <br /> <br />All You Need Is Love, oh yeah, yet there wuz a lot of hurtin' souls along Penny Lane besides Eleanor Rigby or Sgt. Pepper. The price of fame which is not fleeting here. <br /> <br />Donna Coulson\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tBKzkDn3rvGR",
        "colab_type": "text"
      },
      "source": [
        "these comments seem also to be positive.. Pretty sure k-means is not about sentiment analysis, so not a big deal that clusters don't have clear separataion."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-uvZCjgstlW",
        "colab_type": "text"
      },
      "source": [
        "# 6. Conclusion and Future analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ins9bnAPs2-h",
        "colab_type": "text"
      },
      "source": [
        "So this is it! we have converted documents into vectors, and pass that into the k-means algorithm to see which comments are grouped together. We also did cosine similarity to see which comments are related to each other.\n",
        "\n",
        "For sure the analysis I did is very elementary, and I didn't dig deep enough to find any truly meaningful results, but I found that \n",
        "- The most helpful comment to the book \"Sisterhood of the Traveling Pants (Book 1)\" is that it is not suitable for girls younger than 16 years old.\n",
        "- The most \"important\" word that differentiates one document from another in the first 10000 reviews is the word \"girl\", based on tf-idf calculation\n",
        "- 0th review is most similar in cosine similarity metrics to review 2991.\n",
        "- K-means is pretty slow.\n",
        "\n",
        "Future analysis:\n",
        "- do sentiment analysis. Given a bunch of comments about a book, the model should give how much people like/dislike this book. How does people's perception of this book change over time? Sometimes people don't like a book at first, but later come to like it. (generation shift)\n",
        "- what kind of books do young people like and what kind does old people like? (this requires more information about customer ages, which might not be easy to get)\n",
        "\n",
        "Please get a clone of this notebook, and do your own analysis if you are so inclined. If not, hope you learned something from this notebook!!!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OyOLl-JeyNJZ",
        "colab_type": "text"
      },
      "source": [
        "# 7. Further Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oAV1v6p7yTeH",
        "colab_type": "text"
      },
      "source": [
        "There are a couple of areas that I want to improve:\n",
        "- want to do some preprocessing like stemming - so TF-IDF doesn't think \"do\" and \"did\" are two different words.\n",
        "- want to find out the best value of k for k-means clustering using elbow method. plot the inertia of k-means against k and see where it becomes leveled off.\n",
        "- want to see what are the promient words in each cluster, so we can begin to figure out the \"topic\" of each cluster. This might not make sense at all but let's try."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lFH0nH-Ry7lz",
        "colab_type": "text"
      },
      "source": [
        "## 7.1 stemming and write our own tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dchdXPuWzM0r",
        "colab_type": "text"
      },
      "source": [
        "stemming is the process of finding the root of a word that represents this word's content and meaning. (Future tense/past tense of a verb conveys the same meaning.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rgnhUS4eyQWA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# in python there is a stemming library called snowballStemmer.\n",
        "from nltk.stem.snowball import SnowballStemmer"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YiElrVdB6koP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import nltk"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ke-fKEv87fZU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "c240a9cc-e05f-4687-9201-efb642fcfc69"
      },
      "source": [
        "nltk.download('punkt')"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Wi6DLpvBXx9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "06199206-b72a-449a-e828-211afd419209"
      },
      "source": [
        "nltk.download('stopwords')"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bIk7zfS10SDt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "stemmer = SnowballStemmer(language='english')"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8f4LN9t60V3J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "words = ['compute', 'computer', 'computed', 'computing']"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3AHFnCCW0Ypr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "476223c1-1a8a-4b96-8971-540d8fe576a4"
      },
      "source": [
        "for word in words:\n",
        "  print(f'stem of {word} is {stemmer.stem(word)}')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "stem of compute is comput\n",
            "stem of computer is comput\n",
            "stem of computed is comput\n",
            "stem of computing is comput\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "10mxpxbq0feo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# get first 10000 rows as our data.\n",
        "data = pd.read_csv(\"amazon_reviews_us_Books_v1_02.tsv\", sep = \"\\t\", nrows = 10000)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UECEMk6i2lQl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        },
        "outputId": "966b4569-fbca-494a-c818-d0a04bd15786"
      },
      "source": [
        "data.head()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>marketplace</th>\n",
              "      <th>customer_id</th>\n",
              "      <th>review_id</th>\n",
              "      <th>product_id</th>\n",
              "      <th>product_parent</th>\n",
              "      <th>product_title</th>\n",
              "      <th>product_category</th>\n",
              "      <th>star_rating</th>\n",
              "      <th>helpful_votes</th>\n",
              "      <th>total_votes</th>\n",
              "      <th>vine</th>\n",
              "      <th>verified_purchase</th>\n",
              "      <th>review_headline</th>\n",
              "      <th>review_body</th>\n",
              "      <th>review_date</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>US</td>\n",
              "      <td>12076615</td>\n",
              "      <td>RQ58W7SMO911M</td>\n",
              "      <td>0385730586</td>\n",
              "      <td>122662979</td>\n",
              "      <td>Sisterhood of the Traveling Pants (Book 1)</td>\n",
              "      <td>Books</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>this book was a great learning novel!</td>\n",
              "      <td>this boook was a great one that you could lear...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>US</td>\n",
              "      <td>12703090</td>\n",
              "      <td>RF6IUKMGL8SF</td>\n",
              "      <td>0811828964</td>\n",
              "      <td>56191234</td>\n",
              "      <td>The Bad Girl's Guide to Getting What You Want</td>\n",
              "      <td>Books</td>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Fun Fluff</td>\n",
              "      <td>If you are looking for something to stimulate ...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>US</td>\n",
              "      <td>12257412</td>\n",
              "      <td>R1DOSHH6AI622S</td>\n",
              "      <td>1844161560</td>\n",
              "      <td>253182049</td>\n",
              "      <td>Eisenhorn (A Warhammer 40,000 Omnibus)</td>\n",
              "      <td>Books</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>22</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>this isn't a review</td>\n",
              "      <td>never read it-a young relative idicated he lik...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>US</td>\n",
              "      <td>50732546</td>\n",
              "      <td>RATOTLA3OF70O</td>\n",
              "      <td>0373836635</td>\n",
              "      <td>348672532</td>\n",
              "      <td>Colby Conspiracy (Colby Agency)</td>\n",
              "      <td>Books</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>fine author on her A-game</td>\n",
              "      <td>Though she is honored to be Chicago Woman of t...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>US</td>\n",
              "      <td>51964897</td>\n",
              "      <td>R1TNWRKIVHVYOV</td>\n",
              "      <td>0262181533</td>\n",
              "      <td>598678717</td>\n",
              "      <td>The Psychology of Proof: Deductive Reasoning i...</td>\n",
              "      <td>Books</td>\n",
              "      <td>4</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>N</td>\n",
              "      <td>N</td>\n",
              "      <td>Execellent cursor examination</td>\n",
              "      <td>Review based on a cursory examination by Unive...</td>\n",
              "      <td>2005-10-14</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  marketplace  ...  review_date\n",
              "0          US  ...   2005-10-14\n",
              "1          US  ...   2005-10-14\n",
              "2          US  ...   2005-10-14\n",
              "3          US  ...   2005-10-14\n",
              "4          US  ...   2005-10-14\n",
              "\n",
              "[5 rows x 15 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "viK5vHx13rpH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# get only the reviews data.\n",
        "reviews = data['review_body']"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P5DHXKvx3vsy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "outputId": "776ca0e3-c914-47a6-a1b8-aeb294e053f9"
      },
      "source": [
        "reviews[0]"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'this boook was a great one that you could learn from. it not only teaches the imponrtance of family and their values but it also deals with basic issues that teens and some kids even deal with.  this book is about 4 best friends who are for the first time in their lives spending their summer apart. one day they are all in one of the girls rooms and finds a pair of pants that were tucked away in her closet.  once all four  of them try them on they realize that there is really something special about these pants.  seeming as how all 4 girls are differnt shapes and sizes and somehow the pants fit all of them,  they realize that these pants were the start of something special.  immediatley following they decided to make up certian rules abut the pants such as you must write the best thing u did while wearing the pants over your summer on the right leg and also some silly things such as to \\\\\\\\\"never pick yuor nose while wearing the pants.\\\\\\\\\"  this book follows the girls throuh their summers in differnt places of the world and through all of the different obstacles that life takes them through. it can really teach you alot not only about what is going on around you but most imporntantly about yuorself.  i would give this book 4 stars and would reccommend it to anyone who seems the slihgtest bit interested.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "voFCJz-H7WL5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        },
        "outputId": "0b9843b0-9348-4f1d-c290-59c6d466c9fc"
      },
      "source": [
        "# this tokenize all the sentences within a paragraph.\n",
        "nltk.sent_tokenize(reviews[0])"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['this boook was a great one that you could learn from.',\n",
              " 'it not only teaches the imponrtance of family and their values but it also deals with basic issues that teens and some kids even deal with.',\n",
              " 'this book is about 4 best friends who are for the first time in their lives spending their summer apart.',\n",
              " 'one day they are all in one of the girls rooms and finds a pair of pants that were tucked away in her closet.',\n",
              " 'once all four  of them try them on they realize that there is really something special about these pants.',\n",
              " 'seeming as how all 4 girls are differnt shapes and sizes and somehow the pants fit all of them,  they realize that these pants were the start of something special.',\n",
              " 'immediatley following they decided to make up certian rules abut the pants such as you must write the best thing u did while wearing the pants over your summer on the right leg and also some silly things such as to \\\\\\\\\"never pick yuor nose while wearing the pants.\\\\\\\\\"  this book follows the girls throuh their summers in differnt places of the world and through all of the different obstacles that life takes them through.',\n",
              " 'it can really teach you alot not only about what is going on around you but most imporntantly about yuorself.',\n",
              " 'i would give this book 4 stars and would reccommend it to anyone who seems the slihgtest bit interested.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GFGh0KRr_HcW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sentences = nltk.sent_tokenize(reviews[0])"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oDQ-LwQD_Ez5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "30905732-6a54-40a4-dce4-26145764ae4e"
      },
      "source": [
        "# this tokenize all the words out of a sentence.\n",
        "nltk.word_tokenize(sentences[0])"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['this',\n",
              " 'boook',\n",
              " 'was',\n",
              " 'a',\n",
              " 'great',\n",
              " 'one',\n",
              " 'that',\n",
              " 'you',\n",
              " 'could',\n",
              " 'learn',\n",
              " 'from',\n",
              " '.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l82gMYyf5ubL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# we need to write our own tokenizer for tfidfVectorizer to\n",
        "# ignore the tense of the verb and stuff.\n",
        "\n",
        "# what this tokenizer does is convert a string like\n",
        "# 'I love computing and compute'\n",
        "# to a list of stemmed words like [I, love, comput, and]\n",
        "\n",
        "# stopwords = ['the', 'I'] etc\n",
        "stopwords = nltk.corpus.stopwords.words('english')\n",
        "\n",
        "def stemming_tokenizer(text):\n",
        "  tokens = []\n",
        "  sentences = nltk.sent_tokenize(text)\n",
        "  for sentence in sentences:\n",
        "    for word in nltk.word_tokenize(sentence):\n",
        "      if word not in stopwords:\n",
        "        tokens.append(word.lower())\n",
        "\n",
        "  # filter out any tokens that are not words - filter out punctuations and numbers\n",
        "  onlyWordTokens = [word for word in filter(lambda x: x.isalpha(), tokens)]\n",
        "\n",
        "  # replace every word by its stem:\n",
        "  onlyWordTokens = [stemmer.stem(word) for word in onlyWordTokens]\n",
        "\n",
        "  return onlyWordTokens"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wHHvm464Batt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "f6ba78b3-aad3-4b7d-ab6a-45bf265cfee1"
      },
      "source": [
        "# now let's see this tokenizer in action:\n",
        "stemming_tokenizer(reviews[0])"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['boook',\n",
              " 'great',\n",
              " 'one',\n",
              " 'could',\n",
              " 'learn',\n",
              " 'teach',\n",
              " 'imponrt',\n",
              " 'famili',\n",
              " 'valu',\n",
              " 'also',\n",
              " 'deal',\n",
              " 'basic',\n",
              " 'issu',\n",
              " 'teen',\n",
              " 'kid',\n",
              " 'even',\n",
              " 'deal',\n",
              " 'book',\n",
              " 'best',\n",
              " 'friend',\n",
              " 'first',\n",
              " 'time',\n",
              " 'live',\n",
              " 'spend',\n",
              " 'summer',\n",
              " 'apart',\n",
              " 'one',\n",
              " 'day',\n",
              " 'one',\n",
              " 'girl',\n",
              " 'room',\n",
              " 'find',\n",
              " 'pair',\n",
              " 'pant',\n",
              " 'tuck',\n",
              " 'away',\n",
              " 'closet',\n",
              " 'four',\n",
              " 'tri',\n",
              " 'realiz',\n",
              " 'realli',\n",
              " 'someth',\n",
              " 'special',\n",
              " 'pant',\n",
              " 'seem',\n",
              " 'girl',\n",
              " 'differnt',\n",
              " 'shape',\n",
              " 'size',\n",
              " 'somehow',\n",
              " 'pant',\n",
              " 'fit',\n",
              " 'realiz',\n",
              " 'pant',\n",
              " 'start',\n",
              " 'someth',\n",
              " 'special',\n",
              " 'immediatley',\n",
              " 'follow',\n",
              " 'decid',\n",
              " 'make',\n",
              " 'certian',\n",
              " 'rule',\n",
              " 'abut',\n",
              " 'pant',\n",
              " 'must',\n",
              " 'write',\n",
              " 'best',\n",
              " 'thing',\n",
              " 'u',\n",
              " 'wear',\n",
              " 'pant',\n",
              " 'summer',\n",
              " 'right',\n",
              " 'leg',\n",
              " 'also',\n",
              " 'silli',\n",
              " 'thing',\n",
              " 'never',\n",
              " 'pick',\n",
              " 'yuor',\n",
              " 'nose',\n",
              " 'wear',\n",
              " 'book',\n",
              " 'follow',\n",
              " 'girl',\n",
              " 'throuh',\n",
              " 'summer',\n",
              " 'differnt',\n",
              " 'place',\n",
              " 'world',\n",
              " 'differ',\n",
              " 'obstacl',\n",
              " 'life',\n",
              " 'take',\n",
              " 'realli',\n",
              " 'teach',\n",
              " 'alot',\n",
              " 'go',\n",
              " 'around',\n",
              " 'impornt',\n",
              " 'yuorself',\n",
              " 'would',\n",
              " 'give',\n",
              " 'book',\n",
              " 'star',\n",
              " 'would',\n",
              " 'reccommend',\n",
              " 'anyon',\n",
              " 'seem',\n",
              " 'slihgtest',\n",
              " 'bit',\n",
              " 'interest']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WZOmYsrYBg_3",
        "colab_type": "text"
      },
      "source": [
        "Great! Now we wrote a functioning tokenizer extracting all words in a paragraph and ignoring stopwords and stemmed every word. Let's apply this tokenizer while we are doing the Tfidf vectorization process!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_OPw-NSx4Cvf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7a9v2bek3wUj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tfidf = TfidfVectorizer(max_df=0.99, max_features=1000,\n",
        "                        min_df=0.01, stop_words='english',\n",
        "                        use_idf=True, tokenizer = stemming_tokenizer, # our custom tokenizer that ignores the tense of words.\n",
        "                        ngram_range=(1,1))"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vnWa-7BD4QFy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "e7a66793-c34d-444b-efb9-58dfa9757361"
      },
      "source": [
        "tfidf_matrix = tfidf.fit_transform(reviews)"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/feature_extraction/text.py:385: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['afterward', 'alon', 'alreadi', 'alway', 'anoth', 'anyon', 'anyth', 'anywher', 'becam', 'becom', 'besid', 'cri', 'describ', 'els', 'elsewher', 'empti', 'everi', 'everyon', 'everyth', 'everywher', 'fifti', 'forti', 'henc', 'hereaft', 'herebi', 'howev', 'hundr', 'inde', 'mani', 'meanwhil', 'moreov', 'nobodi', 'noon', 'noth', 'nowher', 'otherwis', 'perhap', 'pleas', 'sever', 'sinc', 'sincer', 'sixti', 'someon', 'someth', 'sometim', 'somewher', 'thenc', 'thereaft', 'therebi', 'therefor', 'togeth', 'twelv', 'twenti', 'whatev', 'whenc', 'whenev', 'wherea', 'whereaft', 'wherebi', 'wherev'] not in stop_words.\n",
            "  'stop_words.' % sorted(inconsistent))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o7D4y4WMCWMV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "38966caa-b011-43fb-994d-55dffbbb1bf8"
      },
      "source": [
        "# look at the words tfidf extracted.\n",
        "tfidf.get_feature_names()[:100]"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['abil',\n",
              " 'abl',\n",
              " 'absolut',\n",
              " 'abus',\n",
              " 'academ',\n",
              " 'accept',\n",
              " 'access',\n",
              " 'accomplish',\n",
              " 'accord',\n",
              " 'account',\n",
              " 'accur',\n",
              " 'achiev',\n",
              " 'act',\n",
              " 'action',\n",
              " 'activ',\n",
              " 'actual',\n",
              " 'ad',\n",
              " 'add',\n",
              " 'addit',\n",
              " 'address',\n",
              " 'admir',\n",
              " 'admit',\n",
              " 'adult',\n",
              " 'advanc',\n",
              " 'adventur',\n",
              " 'advic',\n",
              " 'affect',\n",
              " 'age',\n",
              " 'ago',\n",
              " 'agre',\n",
              " 'allow',\n",
              " 'alon',\n",
              " 'alreadi',\n",
              " 'altern',\n",
              " 'alway',\n",
              " 'amaz',\n",
              " 'amazon',\n",
              " 'america',\n",
              " 'american',\n",
              " 'analysi',\n",
              " 'ancient',\n",
              " 'anim',\n",
              " 'anoth',\n",
              " 'answer',\n",
              " 'anyon',\n",
              " 'anyth',\n",
              " 'apart',\n",
              " 'appar',\n",
              " 'appeal',\n",
              " 'appear',\n",
              " 'appli',\n",
              " 'applic',\n",
              " 'appreci',\n",
              " 'approach',\n",
              " 'appropri',\n",
              " 'area',\n",
              " 'argu',\n",
              " 'argument',\n",
              " 'arriv',\n",
              " 'art',\n",
              " 'articl',\n",
              " 'artist',\n",
              " 'asid',\n",
              " 'ask',\n",
              " 'aspect',\n",
              " 'associ',\n",
              " 'assum',\n",
              " 'attack',\n",
              " 'attempt',\n",
              " 'attent',\n",
              " 'attract',\n",
              " 'audienc',\n",
              " 'author',\n",
              " 'avail',\n",
              " 'averag',\n",
              " 'avoid',\n",
              " 'awar',\n",
              " 'away',\n",
              " 'babi',\n",
              " 'background',\n",
              " 'bad',\n",
              " 'balanc',\n",
              " 'base',\n",
              " 'basic',\n",
              " 'battl',\n",
              " 'beauti',\n",
              " 'becam',\n",
              " 'becom',\n",
              " 'began',\n",
              " 'begin',\n",
              " 'beginn',\n",
              " 'behavior',\n",
              " 'belief',\n",
              " 'believ',\n",
              " 'benefit',\n",
              " 'best',\n",
              " 'better',\n",
              " 'bibl',\n",
              " 'big',\n",
              " 'biographi']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eK4uxHn2Cfwq",
        "colab_type": "text"
      },
      "source": [
        "These words are clearly better than the ones we had before (which inclueded numbers and punctuations.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9noWrEr9CvpB",
        "colab_type": "text"
      },
      "source": [
        "## 7.2 figure out the best k in k-means using elbow method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hGWX7blJC3OT",
        "colab_type": "text"
      },
      "source": [
        "Now we need to figure out the best number of clusters to do the k-means. We will do so by using the elbow method. Basically we want sum of every point's distance to its cluster to be minimized. (don't know why this metric is called inertia in sklearn kmeans.) Then We will plot inertia of kmeans against k, and find where the k is optimal. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jMog5E8oDOvi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.cluster import KMeans"
      ],
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wGoqHOt35DAI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "inertias = []\n",
        "\n",
        "# this might take a while, k means is very computationally expensive.\n",
        "# ran this for 30 minutes..\n",
        "for k in range(1, 10):\n",
        "  km = KMeans(n_clusters = k)\n",
        "  km.fit(tfidf_matrix)\n",
        "  inertias.append(km.inertia_) # append the inertia"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "24qlZmEyG5oY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "2600f4c7-ba2b-4bf6-e102-105e5bafb6a5"
      },
      "source": [
        "plt.plot([x for x in range(1, len(inertias) + 1)], inertias)\n",
        "plt.xlabel('k')\n",
        "plt.ylabel('inertia')\n",
        "plt.show()"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXiV9Z338fc3CwlhzU5IQsIumyg5sqigdUVs1apQHZ22tlPbPrajT595Ou0s7bSdzvX0mrmmnandbHVax62oXayjuBQFN5CAoOyGPZGQQAIBQsj2ff44NzRa4CSSk/sk+byuKxfJnTvnfPASPty/3/373ebuiIiInElS2AFERCTxqSxERCQmlYWIiMSkshARkZhUFiIiElNK2AHiIScnx0tLS8OOISLSq6xevXq/u+ee6nt9sixKS0spLy8PO4aISK9iZrtO9z0NQ4mISEwqCxERiUllISIiMaksREQkJpWFiIjEpLIQEZGYVBYiIhKTyqKDg43NfP+FrWyubgg7iohIQlFZfMBPXt7GY2/uCTuGiEhCUVl0MDxjAFdNyee3b1XR1NIWdhwRkYShsviARZFiDh1r4cVN+8KOIiKSMFQWH3DRuBxGDktncXll2FFERBKGyuIDkpOMm8uKeOXdWt47eCzsOCIiCUFlcQo3lxXjDk+u1tWFiAioLE5pVHYGc8Zk8/jqStrbPew4IiKhU1mcxqILithd18jKHXVhRxERCZ3K4jTmTylgSFoKj5drzYWIiMriNAYOSOZj543kmfV7aWhqCTuOiEioVBZnsChSTFNLO0+v2xt2FBGRUKkszmB60TAm5A9msYaiRKSfU1mcgZmxKFLM2j0H2brvcNhxRERCo7KI4YbzC0lJMk10i0i/prKIIWdwGpdPyuM3a6poaWsPO46ISChUFp2wKFLMgaPNLN1cE3YUEZFQqCw64ZIJueQOSdNQlIj0WyqLTkhJTuKmGUW8tKWWmoamsOOIiPQ4lUUnLYwU0dbu/OatqrCjiIj0OJVFJ43NHUykJJPF5Xtw1+aCItK/qCy6YFGkmO21R1mzuz7sKCIiPUpl0QULzi0gY0Ayi1fpORci0r+oLLpgcFoK104r4Om33+Po8daw44iI9BiVRRctuqCYo81tPPOONhcUkf5DZdFFkZJMRucM4vFyDUWJSP+hsugiM2NhpIg3d9axvfZI2HFERHqEyuJDuGlGEUkGT6zW1YWI9A9xLQszu9vM1pvZBjO7Jzj2HTN728zWmtnzZjYyOG5m9p9mVhF8f0aH1/mUmb0bfHwqnpk7I39oOpdOzOPJNZW0anNBEekH4lYWZjYV+BwwE5gOfNTMxgH/6u7nuvt5wNPAN4IfuQYYH3zcCfwkeJ0s4JvArOC1vmlmmfHK3VmLIkXsazjOK+/uDzuKiEjcxfPKYhKw0t0b3b0VWAbc6O4NHc4ZBJxYDn098KBHrQCGm1kBcDXwgrvXuXs98AIwP465O+Wyc/LJGjRAT9ETkX4hnmWxHphrZtlmlgEsAIoBzOy7ZrYHuI0/XVkUAh3/5q0Mjp3ueKgGpCTx8fMLeXHTPg4cOR52HBGRuIpbWbj7JuB7wPPAEmAt0BZ87+/dvRh4GPhSd7yfmd1pZuVmVl5bW9sdLxnTokgxLW3O79a+1yPvJyISlrhOcLv7/e5e5u7zgHpg6wdOeRi4Kfi8iuDKI1AUHDvd8Q++133uHnH3SG5ubnf9Fs5o4oghTC8axuPaXFBE+rh43w2VF/w6CrgReMTMxnc45Xpgc/D5U8Ang7uiZgOH3H0v8BxwlZllBhPbVwXHEsLCSDGbqw/zTtWhsKOIiMRNSpxf/0kzywZagLvc/aCZ3W9mE4F2YBfwheDcZ4jOa1QAjcAdAO5eZ2bfAVYF533b3evinLvTPjZ9JN95eiOLy/dwbtHwsOOIiMRFXMvC3eee4thNpznXgbtO870HgAe6N133GDYwlWumjuD3a9/jH66dTHpqctiRRES6nVZwd4NFkWION7Xy3IbqsKOIiMSFyqIbzB6TTVHmQK25EJE+S2XRDZKSjIVlxbxWcYA9dY1hxxER6XYqi25yU1khps0FRaSPUll0k6LMDC4el8MTqytpb9eaCxHpW1QW3WhhpJiqg8d4fduBsKOIiHQrlUU3umpyPkPTUzTRLSJ9jsqiG6WnJnPD+YUs2VDNocaWsOOIiHQblUU3WxQpprm1nafW/dn2VSIivZbKoptNGTmUSQVDWVyuu6JEpO9QWXQzM2NRpIh3qg6x8b2G2D8gItILqCzi4IbzChmQnMTjqzXRLSJ9g8oiDjIHDeDKyfn87q0qjre2hR1HROSsqSziZGGkiPrGFv64qSbsKCIiZ01lESdzx+cyYmi61lyISJ+gsoiT5CTj5rIilm+tZe+hY2HHERE5KyqLOLq5rIh2h9+s0ZoLEendVBZxVJoziFmjs1hcvofogwBFRHonlUWcLYoUs+tAI2/uSJjHhouIdJnKIs6umTaCwWkpWtEtIr2ayiLOMgak8LHpBTzzzl4ON2lzQRHpnVQWPWBhpJhjLW38z9t7w44iIvKhqCx6wPnFwxmXN1hrLkSk11JZ9IATmwuu2X2QiprDYccREekylUUP+fj5RSQnGY9roltEeiGVRQ/JHZLGZefk8eSaKlra2sOOIyLSJSqLHrQoUsz+I8d5eUtt2FFERLpEZdGDLp2YS87gNE10i0ivo7LoQanJSdw0o5Clm2uoOdwUdhwRkU5TWfSwhZEi2tqd372lzQVFpPdQWfSwcXlDmDFqOIvLK7W5oIj0GiqLECyKFFNRc4S39hwMO4qISKeoLEJw7bkFDExN5nFNdItIL6GyCMGQ9FQWTCvgD+v20tjcGnYcEZGYVBYhWRQp4sjxVp59pzrsKCIiMaksQjJzdBal2RlacyEivUJcy8LM7jaz9Wa2wczuCY79q5ltNrO3zey3Zja8w/lfN7MKM9tiZld3OD4/OFZhZl+LZ+aeYmYsjBSzckcdO/cfDTuOiMgZxa0szGwq8DlgJjAd+KiZjQNeAKa6+7nAVuDrwfmTgVuAKcB84MdmlmxmycCPgGuAycCtwbm93o0zCkkyeGK1NhcUkcQWzyuLScBKd29091ZgGXCjuz8ffA2wAigKPr8eeMzdj7v7DqCCaNHMBCrcfbu7NwOPBef2egXDBjJvQi5PrK6krV1rLkQkccWzLNYDc80s28wygAVA8QfO+QzwbPB5IdBxAL8yOHa64+9jZneaWbmZldfW9p6N+hZFiqluaOKVd3tPZhHpf+JWFu6+Cfge8DywBFgLtJ34vpn9PdAKPNxN73efu0fcPZKbm9sdL9kjLp+UR2ZGqp5zISIJLa4T3O5+v7uXufs8oJ7oHAVm9mngo8Bt/qc9L6p4/5VHUXDsdMf7hLSUZG44v5DnN1ZTd7Q57DgiIqcU77uh8oJfRwE3Ao+Y2Xzgq8B17t7Y4fSngFvMLM3MRgPjgTeBVcB4MxttZgOIToI/Fc/cPW1hWTEtbc7v1/aZDhSRPibe6yyeNLONwB+Au9z9IHAvMAR4wczWmtlPAdx9A7AY2Eh02Ooud28LJsO/BDwHbAIWB+f2GZNHDmVa4TB+vWqPNhcUkYSUEs8Xd/e5pzg27gznfxf47imOPwM8073pEsuiSBH/+PsNbHivgamFw8KOIyLyPlrBnSCum17IgJQkregWkYSkskgQwzJSmT9lBL97q4qmlrbYPyAi0oNUFglkUaSYhqZWnt+4L+woIiLvo7JIIBeOzaZw+EA950JEEo7KIoEkJRk3lxXxasV+KusbY/+AiEgP6XRZmNm1ZvZVM/vGiY94Buuvbi4rwh2eXK01FyKSODpVFsFaiE8AXwYMWAiUxDFXv1WclcFF47J5fPUe2rW5oIgkiM5eWVzo7p8E6t39W8AcYEL8YvVviyLFVNYfY8X2A2FHEREBOl8Wx4JfG81sJNACFMQnklw9ZQRD0lO05kJEEkZny+Lp4Il2/wqsAXYCj8YrVH+XnprM9eeN5Nn11Rw61hJ2HBGRzpWFu3/H3Q+6+5NE5yrOcfd/jG+0/m1RpJjjre38Yd17YUcRETnz3lBmdpm7LzWzG0/xPdz9N/GL1r9NKxzGOSOG8Hj5Hm6frXsJRCRcsTYSvARYCnzsFN9zQGURJ2bGwkgx33l6I5urGzhnxNCwI4lIP3bGYSh3/2bw6bfd/Y6OH8B34h+vf7vhvJGkJpueoicioevsBPeTpzj2RHcGkT+XPTiNKybl89u3qmhubQ87joj0Y7HmLM4BpgDDPjBvMRRIj2cwiVoUKebZ9dUs3byP+VN1t7KIhCPWnMVEos/KHs775y0OA5+LVyj5k7njc8gfmsbi8kqVhYiE5oxl4e6/N7Ongb9193/poUzSQUpyEjfNKOKny7axr6GJ/KG6oBORnhdzzsLd24AbeiCLnMbCSDHtDnc+WM47lYfCjiMi/VBnJ7hfM7N7zWyumc048RHXZHLS6JxB/Oet51N1sInrfvQqf/fbd6g/2hx2LBHpR8w99s6mZvbSKQ67u1/W/ZHOXiQS8fLy8rBjdLuGpha+/8JWHnxjF0PSU/i/V0/klgtGkZxkYUcTkT7AzFa7e+SU3+tMWfQ2fbUsTthc3cA3fr+BN3fUMa1wGN++fgrnj8oMO5aI9HJnKovOPs8i38zuN7Nng68nm9lnuzOkdN45I4by6ztn8x+3nMe+hiY+/uPX+eoT6zhw5HjY0USkj+rsnMUvgeeAkcHXW4F74hFIOsfMuP68Qpb+zaV8ft4YfrOmio/828v86vWdtLZpAZ+IdK/OlkWOuy8G2gHcvRVoi1sq6bTBaSl8fcEkltwzl2lFw/jmUxv42L2vUb6zLuxoItKHdLYsjppZNtHNAzGz2YDu4Uwg4/KG8NBnZ/Hj22ZwqLGZm3/6Bl9ZvJaaw01hRxORPiDWCu4TvgI8BYw1s9eAXODmuKWSD8XMWDCtgEsn5nLv0gp+/sp2Xtiwj3uunMAn55SQmtzZfxuIiLxfp++GMrMUott/GLDF3RP2EW59/W6oztpee4Rv/WEjy7bWMjF/CP903RTmjM0OO5aIJKizvhsqMBOYDswAbjWzT3ZHOImfMbmD+eUdF3DfX5ZxtLmVW3++gi8/+hbVhzQ0JSJd06lhKDP7b2AssJY/TWw78GCcckk3MTOumjKCeRNy+fHL2/jpsm0s3bSPv758PHdcNJoBKRqaEpHYOruCexMw2XvJCj4NQ53e7gONfPvpDby4qYaxuYP41nVTuXh8TtixRCQBdMcw1HpgRPdFkrCMys7gF5+6gAc+HaG13bn9/pV88aHVVB08FnY0EUlgnb0bKgfYaGZvAieXCbv7dXFJJXF32Tn5XDg2h58v386PXq7g5S21fOmycfzV3NGkpSSHHU9EEkxnh6EuOdVxd1/W7Ym6gYahuqayvpF/fnoTSzZUU5qdwTevm8JHJuaFHUtEepg2EpROWb61ln/6wwa21x7lysn5fOOjkynOygg7loj0kA89Z2Fmrwa/Hjazhg4fh82soRNvfLeZrTezDWZ2T3BsYfB1u5lFPnD+182swsy2mNnVHY7PD45VmNnXOvOblq6bNyGXJXfP42/nn8NrFfu54t+X8YMXt9LUop1dRPq7uF1ZmNlU4DGi6zOagSXAF4BUontM/Qz4G3cvD86fDDwanD8SeBGYELzcVuBKoBJYBdzq7htP9966sjh7ew8d47v/s4mn395LcdZAvvHRKVwxKQ8zPTtDpK/qrkV5XTUJWOnujcHGg8uAG919k7tvOcX51wOPuftxd98BVBAtjplAhbtvd/dmogV0fRxzC1AwbCD3/sUMHvncLNJTkvncg+V85per2Ln/aNjRRCQE8SyL9cBcM8s2swxgAVB8hvMLgT0dvq4Mjp3u+PuY2Z1mVm5m5bW1tWcdXqIuHJvDM3fP5R+uncSqnfVc9f3l/NtzWzjWrKEpkf4kbmXh7puA7wHPEx2C6rj6Ox7vd5+7R9w9kpubG6+36ZdSk5P4q7ljWPp/LuHacwu496UKrvj3ZTz7zl764g0SIvLn4rrXg7vf7+5l7j4PqCc693A6Vbz/yqMoOHa649LD8oam8/1PnMfiz89hSHoKX3x4DZ984E3W7TkYdjQRibO4loWZ5QW/jgJuBB45w+lPAbeYWZqZjQbGA28SndAeb2ajzWwAcEtwroRk5ugsnv7yxXzruims3XOQ63/0Gh/74av8etVuGptbw44nInEQ13UWZvYKkA20AF9x9z+a2ceBHxJ9JsZBYK27Xx2c//fAZ4BW4B53P/HM7wXAD4Bk4AF3/+6Z3ld3Q/Wcw00t/O6tKh5asZst+w4zJD2Fm2YUcfvsUYzLGxJ2PBHpAi3Kk7hzd8p31fPQil08+041zW3tzBqdxe2zS7h6ygjtbivSC6gspEftP3Kcx8sreeTNXeypO0bO4DQ+cUERt84cRVGmVoSLJCqVhYSivd1Z9m4tD6/YxdLNNQB8ZGIet88uYd6EXJKTtMBPJJGoLCR0VQeP8ejK3Ty2ag/7jxynKHMgt84cxScuKCZncFrY8UQElYUkkObWdp7fWM3DK3bzxvYDpCYb86cWcPusUcwcnaXtRERCpLKQhFRRc4SHV+7iidWVHG5qZUL+YG6bVcLHZxQyND017Hgi/Y7KQhLaseY2/rDuPR5auYu3Kw+RMSCZ688byW2zSphaOCzseCL9hspCeo23Kw/y0IpdPLXuPZpa2jmveDi3zRrFx6aPJD1VT/ATiSeVhfQ6hxpbeHJNJQ+v3MW22qMMG5jKzWVF3DZrFGNyB4cdT6RPUllIr+XurNhex0Mrd/Hc+mpa252LxmVz+6wSrpicT2qyFvuJdJczlUVKT4cR6QozY87YbOaMzabmcBOLV+3h0Tf38MWH15A3JI1bZo7i1pnFFAwbGHZUkT5NVxbS67S1Oy9truGhlbtYtrWWJDMuPye62O/icTkkabGfyIeiKwvpU5KTjCsm53PF5Hz21DXy8MrdPF6+h+c37qMkO4PbZo3i5rJisgYNCDuqSJ+hKwvpE463trFkfXSx35s76xiQksSN5xfyxUvHUpI9KOx4Ir2CJrilX9lSfZgH39jJ46sraWt3rp8+kv/1kXGMy9NdVCJnorKQfqmmoYn7lm/n4ZW7aWptY8G0Ar70kXFMKhgadjSRhKSykH7twJHj3P/qDh58YxdHjrdy5eR8vnzZOM4tGh52NJGEorIQIbrQ779e38EDr+6goamVSybk8uXLxhEpzQo7mkhCUFmIdHC4qYX/XrGLX7yyg7qjzcwek8VfXzaeOWOzteut9GsqC5FTaGxu5dE39/CzZduoOXycspJMvnTZOC6dkKvSkH5JZSFyBk0tbTy+upKfvryNqoPHmFY4jC9dNo4rJ+VrgZ/0KyoLkU5obm3nd29V8aOXK9h1oJFzRgzhro+MY8G0Aj0CVvoFlYVIF7S2tfP023u596UKKmqOMCZ3EHddOo7rzhupjQulT1NZiHwI7e3Okg3V/HBpBZv2NlCcNZAvXjKOm8oKSUvRszWk71FZiJwFd+ePm2r44dJ3WVd5iIJh6Xx+3hhumTlKD2SSPkVlIdIN3J1X3t3PD5e+y6qd9eQMTuPOeaO5bVYJg9K0J6f0fioLkW62YvsB7l1awasV+8nMSOWzF4/mkxeWMjQ9NexoIh+aykIkTlbvqudHL1WwdHMNQ9JTuOPCUu64aDSZ2h5deiGVhUicra86xL1LK1iyoZpBA5K5fU4Jn5s7hpzBaWFHE+k0lYVID9lSfZgfvVTB02+/x4CUJG6dOYrPzxvLiGHpYUcTiUllIdLDttce4ccvb+O3b1WRbMbCSBFfuGQsxVkZYUcTOS2VhUhI9tQ18pNl23iivJJ2d244v5DbZo3i3KLhWhUuCUdlIRKyvYeOcd/y7TyycjfHW9sZnpHK3PG5XDIhl3njc8gbqmEqCZ/KQiRB1B9tZvm7tSzfup9lW2vZf+Q4AJMKhkaLY0IOkZIsBqRoWxHpeSoLkQTU3u5sqm4IiqOG8p31tLY7gwYkM2dsNpdMyOWSCXmMytY8h/QMlYVIL3DkeCtvbDvAsq01LNtay566YwCUZmcEVx25zBmbTcYArRaX+FBZiPQy7s7OA40s21LD8nf388a2AxxraWNAchIXjM5k3vhcLpmYy8T8IXpQk3Sb0MrCzO4GPgcY8HN3/4GZZQG/BkqBncAid6+36P/x/wEsABqBT7v7muB1PgX8Q/Cy/+zuvzrT+6ospK9pammjfGc9y9+tZdmWWrbsOwxA/tC0k8Vx8bgchmdo5bh8eKGUhZlNBR4DZgLNwBLgC8CdQJ27/z8z+xqQ6e5/a2YLgC8TLYtZwH+4+6ygXMqBCODAaqDM3etP994qC+nr9h46xivBJPkr79bS0NRKksH04uEnh6ym6/Zc6aKwymIhMN/dPxt8/Y/AceCzwKXuvtfMCoCX3X2imf0s+PzR4PwtwKUnPtz988Hx9513KioL6U9a29pZV3mIZVtrWb61lnWVB3GHYQNTuXh8TjBRnku+bs+VGM5UFvGcKVsPfNfMsoFjRK8YyoF8d98bnFMN5AefFwJ7Ovx8ZXDsdMffx8zuJHrVwqhRo7rvdyGS4FKSkygryaSsJJOvXDmB+qPNvFKxn+Vba1m2tZb/eTv6x+2cEUNOFkdZaaYe4CRdEreycPdNZvY94HngKLAWaPvAOW5m3XJp4+73AfdB9MqiO15TpDfKHDSA66aP5LrpI3F3NlcfZtnW6FzHA6/t4GfLtzMwNZkLx2YzLyiP0pxBYceWBBfXe/Dc/X7gfgAz+xeiVwX7zKygwzBUTXB6FVDc4ceLgmNVRIeiOh5/OZ65RfoKM2NSwVAmFQzlC5eM5ejJ23OjVx1/3Bz94zcqK4PLzsnjmqkjiJRmaa5D/ky874bKc/caMxtF9ApjNvD3wIEOE9xZ7v5VM7sW+BJ/muD+T3efGUxwrwZmBC+7hugEd93p3ldzFiKds3P/UZa/W8vLW2p5tWI/za3t5AxO46op+VwzdQSzx2STmqzV5P1FmLfOvgJkAy3AV9z9j8EcxmJgFLCL6K2zdcGts/cC84neOnuHu5cHr/MZ4O+Cl/2uu//Xmd5XZSHSdUeOt/LS5hqWrK/mpS01NDa3MTwjlSsmRYvj4vE5mufo47QoT0S6pKmljWVba1myvpoXN+3jcFMrg9NSTg5VXTIxVyvJ+6Cw7oYSkV4qPTWZq6eM4OopI2hubee1bftZ8k41z2+s5ql175GemsQlE3K5ZmoBl03K07PH+wFdWYhIp7W2tfPmjjqWbKhmyfpqag4fZ0ByEheNy+aaqQVcOTlfzx/vxTQMJSLdrr3deWtPPc++U82z66upOniM5CRj9pgs5k8t4Oop+eQN0ULA3kRlISJx5e6sr2rg2fV7WbK+mu37j2IGkZJM5k8tYP7UERQOHxh2TIlBZSEiPcbd2brvyMni2Fwd3fTw3KJhzJ86gmumFjBaiwATkspCREKzY//Rk8XxduUhILr1yInimJA/WNusJwiVhYgkhMr6Rpasj06Or95djzuMyRl0sjimFg5VcYRIZSEiCaemoYnnNkQnx1fuqKOt3SnKHMj8KSO4ZtoIzi/OJEnbjvQolYWIJLS6o828uHEfz67fy6sV+2lpc/KGpHH1lBFcPimPC0qzGJSmZWHxprIQkV6joamFpZtqeHb9XpZtraWppZ2UJGN68XAuHJvNnDHZzCjJJD1VW490N5WFiPRKjc2tlO+s5/VtB3hj+wHeqTxIu8OAlCRmjBrOhWNzmDM2m+lFwxmQog0Pz5bKQkT6hIamFlbtqOONbQd4fdsBNlU34A4DU5OJlGaeLI+pI4eSot1yu0xlISJ9Uv3RZlbuOHCyPN6tOQLAkLQUZo7OYs7YbOaMzWbSiKGaLO8EbSQoIn1S5qABwQrxAgBqDjexYnv0ymPF9gMnH+40PCOV2aOjxXHh2GzG5WltR1fpykJE+qy9h46dvOp4Y9sBqg4eAyBncFr0qmNMtDxKsjNUHmgYSkQEd2dP3THe2L7/ZHnUHD4OQMGw9JPlMWdsNkWZGSGnDYfKQkTkA9yd7fuP8vq2A6wI7raqO9oMRJ9JfmEw3zFnTDZ5Q/vH7rkqCxGRGNrbna01h3m9IlocK7Yf4HBTKwBjcwcF8x05zB6TTVYffWaHykJEpIva2p2N7zXw+rb9vLH9AKt21HG0uQ2IboQ4e0w2kdJMIiVZjBjWN648VBYiImeppa2dtysPsWL7AV7ftp81uw5yrCVaHkWZA4mUZFJWmsUFpZlMyBvSK2/VVVmIiHSzlrZ2Nu1tYNXOelbvqmPVznpqgwnzIekplJVkEinJJFKaxfSi4QwckPjbk6gsRETi7MTdVqt21lG+K1ogW/dFFwmmJBlTCodxQUkmkdJMykqyyB2SFnLiP6eyEBEJwcHGZtbsro9efeysZ23lQZpb2wEozc4gUpp18upjbO6g0Nd6qCxERBLA8dY21lc1UH7y6qP+5O26mRmp0aGroECmFQ0jLaVnh6603YeISAJIS0mmrCSTspJMPs+f1nqs3lnPqp11rN5Vz4uboluUDEhJ4tzCYZSVZnJBSRZlJZlkhnjLrq4sREQSyP4jx1kdXHWs2lnH+qpDtLRF/54elzf45LBVpCSz27cp0TCUiEgv1dTSxro9B08OW5XvrKMhWCyYMzgtKI9ogUwZOZTUs9iaXcNQIiK9VHpqMrPGZDNrTDYQXWleUXskOmy1s55Vu+pYsqE6ODeJKyblc+9fzOj2HCoLEZFeJCnJmJA/hAn5Q7htVgkANQ1NlAfDVhlxWs+hshAR6eXyhqazYFoBC6YVxO099NxBERGJSWUhIiIxqSxERCQmlYWIiMSkshARkZhUFiIiEpPKQkREYlJZiIhITH1ybygzqwV2ncVL5AD7uylOd1KurlGurlGurumLuUrcPfdU3+iTZXG2zKz8dJtphUm5uka5uka5uqa/5dIwlIiIxKSyEBGRmFQWp3Zf2AFOQ7m6Rrm6Rrm6pl/l0pyFiIjEpCsLERGJSWUhIiIxqSw6MLMHzKzGzDLM36EAAARSSURBVNaHneUEMys2s5fMbKOZbTCzu8POBGBm6Wb2ppmtC3J9K+xMHZlZspm9ZWZPh53lBDPbaWbvmNlaM0uYh8Sb2XAze8LMNpvZJjObE3YmADObGPy3OvHRYGb3JECu/x38P7/ezB41s/SwMwGY2d1Bpg3x+O+kOYsOzGwecAR40N2nhp0HwMwKgAJ3X2NmQ4DVwA3uvjHkXAYMcvcjZpYKvArc7e4rwsx1gpl9BYgAQ939o2HngWhZABF3T6iFXGb2K+AVd/+FmQ0AMtz9YNi5OjKzZKAKmOXuZ7Pg9mxzFBL9f32yux8zs8XAM+7+y7AyBbmmAo8BM4FmYAnwBXev6K730JVFB+6+HKgLO0dH7r7X3dcEnx8GNgGF4aYCjzoSfJkafCTEvzzMrAi4FvhF2FkSnZkNA+YB9wO4e3OiFUXgcmBbmEXRQQow0MxSgAzgvZDzAEwCVrp7o7u3AsuAG7vzDVQWvYiZlQLnAyvDTRIVDPWsBWqAF9w9IXIBPwC+CrSHHeQDHHjezFab2Z1hhwmMBmqB/wqG7X5hZoPCDnUKtwCPhh3C3auAfwN2A3uBQ+7+fLipAFgPzDWzbDPLABYAxd35BiqLXsLMBgNPAve4e0PYeQDcvc3dzwOKgJnBpXCozOyjQI27rw47yylc7O4zgGuAu4Jhz7ClADOAn7j7+cBR4GvhRnq/YGjsOuDxBMiSCVxPtGRHAoPM7PZwU4G7bwK+BzxPdAhqLdDWne+hsugFgjmBJ4GH3f03Yef5oGDY4iVgfthZgIuA64L5gceAy8zsoXAjRQX/KsXda4DfEh1fDlslUNnhqvAJouWRSK4B1rj7vrCDAFcAO9y91t1bgN8AF4acCQB3v9/dy9x9HlAPbO3O11dZJLhgIvl+YJO7/3vYeU4ws1wzGx58PhC4Etgcbipw96+7e5G7lxIduljq7qH/y8/MBgU3KBAM81xFdOggVO5eDewxs4nBocuBUG+eOIVbSYAhqMBuYLaZZQR/Ni8nOo8YOjPLC34dRXS+4pHufP2U7nyx3s7MHgUuBXLMrBL4prvfH24qLgL+EngnmB8A+Dt3fybETAAFwK+Cu1SSgMXunjC3qSagfOC30b9fSAEecfcl4UY66cvAw8Fwz3bgjpDznBQU65XA58POAuDuK83sCWAN0Aq8ReJs+/GkmWUDLcBd3X2jgm6dFRGRmDQMJSIiMaksREQkJpWFiIjEpLIQEZGYVBYiIhKTykKkh5hZaSLtaCzSFSoLERGJSWUhEgIzGxNs3HdB2FlEOkMruEV6WLC1xmPAp919Xdh5RDpDZSHSs3KB3wM3hv0AK5Gu0DCUSM86RHQzuovDDiLSFbqyEOlZzcDHgefM7Ii7d+vOoCLxorIQ6WHufjR4SNMLQWE8FXYmkVi066yIiMSkOQsREYlJZSEiIjGpLEREJCaVhYiIxKSyEBGRmFQWIiISk8pCRERi+v8aHBubxrO9NQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZrJ6TRyMHfHZ",
        "colab_type": "text"
      },
      "source": [
        "We can see that our k-means is pretty bad.. with inertia of around 9000.. But we did choose 1000 dimensions, so maybe data are pretty far part."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q9c78AOGHzDp",
        "colab_type": "text"
      },
      "source": [
        "k = 5 seems like a better turning point than 2. Let's just pick k = 5."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VLukwnOIH4M-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "287b777e-7c70-49d6-c6a1-e5ea3dffbde4"
      },
      "source": [
        "km = KMeans(n_clusters = 5)\n",
        "km.fit(tfidf_matrix)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,\n",
              "       n_clusters=5, n_init=10, n_jobs=None, precompute_distances='auto',\n",
              "       random_state=None, tol=0.0001, verbose=0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uUqG0EIAN8qW",
        "colab_type": "text"
      },
      "source": [
        "## 7.3 Now let's see the words that characterize each cluster."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UtgHRHXCJGrZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "8d2b768e-db3e-4de1-89f2-ac0870e29d4a"
      },
      "source": [
        "km.cluster_centers_"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.00514466, 0.00579936, 0.0061077 , ..., 0.00413277, 0.00286362,\n",
              "        0.00889444],\n",
              "       [0.00216298, 0.00504677, 0.0025899 , ..., 0.00336507, 0.00142543,\n",
              "        0.00427739],\n",
              "       [0.00353999, 0.00703205, 0.00432043, ..., 0.00177656, 0.00248713,\n",
              "        0.01323159],\n",
              "       [0.00424007, 0.0059498 , 0.00546244, ..., 0.00431304, 0.00330749,\n",
              "        0.00862123],\n",
              "       [0.00081867, 0.00688584, 0.00825519, ..., 0.00313211, 0.00121768,\n",
              "        0.00687   ]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Gpz6G59OE8U",
        "colab_type": "text"
      },
      "source": [
        "Each of these numbers is a TF-IDF of a word, let's find the top 5 TF-IDF words within each cluster. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QXbfTF-lOC2S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f940cc91-7853-4d91-946d-0a9f6ea63205"
      },
      "source": [
        "# numpy's argsort function returns the array\n",
        "# [index in original array of smallest element, index ... of second smallest element, ...]\n",
        "np.argsort([3,1,2])"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 2, 0])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ckm-s_PFOyBA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "8d5db627-60c3-45d4-b34b-470c69effa02"
      },
      "source": [
        "print('top 5 words that characterize cluster 0:')\n",
        "# reverse because we are finding the highest TF-IDF, not the lowest\n",
        "indices_of_words = np.argsort(km.cluster_centers_[0])[::-1][:5]\n",
        "for index in indices_of_words:\n",
        "  print(tfidf.get_feature_names()[index])"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "top 5 words that characterize cluster 0:\n",
            "br\n",
            "book\n",
            "read\n",
            "stori\n",
            "like\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NRkMFSJ_PYVw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 612
        },
        "outputId": "3b6d09a0-cee0-4b2d-bfda-12309bc59a6a"
      },
      "source": [
        "for i in range(5):\n",
        "  print(f'top 5 words that characterize cluster {i}:')\n",
        "  indices_of_words = np.argsort(km.cluster_centers_[i])[::-1][:5]\n",
        "  for index in indices_of_words:\n",
        "    print(tfidf.get_feature_names()[index])\n",
        "  print()"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "top 5 words that characterize cluster 0:\n",
            "br\n",
            "book\n",
            "read\n",
            "stori\n",
            "like\n",
            "\n",
            "top 5 words that characterize cluster 1:\n",
            "book\n",
            "use\n",
            "great\n",
            "help\n",
            "inform\n",
            "\n",
            "top 5 words that characterize cluster 2:\n",
            "stori\n",
            "charact\n",
            "novel\n",
            "book\n",
            "read\n",
            "\n",
            "top 5 words that characterize cluster 3:\n",
            "br\n",
            "book\n",
            "read\n",
            "like\n",
            "time\n",
            "\n",
            "top 5 words that characterize cluster 4:\n",
            "book\n",
            "read\n",
            "love\n",
            "enjoy\n",
            "great\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XiCPafZtQkjG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "8a4035ed-0eaa-4b28-a74c-288442fcb070"
      },
      "source": [
        "for i in range(5):\n",
        "  print(f'top 10 words that characterize cluster {i}:')\n",
        "  indices_of_words = np.argsort(km.cluster_centers_[i])[::-1][:10]\n",
        "  for index in indices_of_words:\n",
        "    print(tfidf.get_feature_names()[index])\n",
        "  print()"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "top 10 words that characterize cluster 0:\n",
            "br\n",
            "book\n",
            "read\n",
            "stori\n",
            "like\n",
            "love\n",
            "good\n",
            "time\n",
            "make\n",
            "author\n",
            "\n",
            "top 10 words that characterize cluster 1:\n",
            "book\n",
            "use\n",
            "great\n",
            "help\n",
            "inform\n",
            "good\n",
            "read\n",
            "like\n",
            "make\n",
            "recommend\n",
            "\n",
            "top 10 words that characterize cluster 2:\n",
            "stori\n",
            "charact\n",
            "novel\n",
            "book\n",
            "read\n",
            "love\n",
            "like\n",
            "br\n",
            "good\n",
            "life\n",
            "\n",
            "top 10 words that characterize cluster 3:\n",
            "br\n",
            "book\n",
            "read\n",
            "like\n",
            "time\n",
            "work\n",
            "author\n",
            "make\n",
            "mani\n",
            "good\n",
            "\n",
            "top 10 words that characterize cluster 4:\n",
            "book\n",
            "read\n",
            "love\n",
            "enjoy\n",
            "great\n",
            "like\n",
            "realli\n",
            "time\n",
            "good\n",
            "year\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G_MrPF6VQBsa",
        "colab_type": "text"
      },
      "source": [
        "Great! We now have the words that characterize different kinds of reviews. We can see that reviews from cluster 0 is all stories, cluster 1 is about \"inform\", so reader might be looking for informations and guidance in books. Cluster 2 is about characters and novels - so reviwers in this category probably reads fictions. Cluster 3 is about time - time travels? cluster 4 is about love, enjoy and great, probably by people who just enjoys reading for the sake of it. We found out different kinds of people give different reviews on books! This is super interesting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hvHsFdenF_Or",
        "colab_type": "text"
      },
      "source": [
        "# 8. Further Conclusion."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VE8-IncKGBqu",
        "colab_type": "text"
      },
      "source": [
        "1.   K-means is excruciatingly slow. I was just doing k-means with 10000 documents, each document is represented as a 1000- dimensional vector, and it tooke me 30 minutes to do 10 k-means. In the future clustering analysis need to think of a better method that is more computationally efficient.\n",
        "2.   there are more insights when I see what each cluster correspond to in terms of what are the most important words that characterize that cluster in terms of TF-IDF score. \n",
        "  - Found out that some reviews like novels, some read because they just enjoy it, some would recommend books to others, some read books because they like the author.\n",
        "  - these are great insights for Amazon because they can target each cluster of customers and recommend them different things. For example once we know a person loves novels, we can recommend good novels when they come out in the future.\n",
        "3. There are many more ways to do topic modeling and document clustering, such as LDA. I am just scratching the surface here. But it is really fun! \n",
        "\n"
      ]
    }
  ]
}