Topic_Analysis/Topic_modeling_Amazon_Reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling Amazon Reviews\n",
    "\n",
    "We will use the 142.8 million reviews spanning May 1996 – July 2014. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import library\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "from stop_words import get_stop_words\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from gensim import corpora, models\n",
    "import gensim\n",
    "import pandas as pd\n",
    "import gzip\n",
    "import operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "from ipywidgets import interact, interact_manual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one-review-per-line in json\n",
    "def parse(path):\n",
    "    g = gzip.open(path, 'rb')\n",
    "    for l in g:\n",
    "        yield eval(l)\n",
    "        \n",
    "# Get data\n",
    "def getDF(path):\n",
    "    i = 0\n",
    "    df = {}\n",
    "    for d in parse(path):\n",
    "        df[i] = d\n",
    "        i += 1\n",
    "    return pd.DataFrame.from_dict(df, orient='index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading our data\n",
    "df = getDF('../data/reviews_Automotive_5.json.gz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 20473 entries, 0 to 20472\n",
      "Data columns (total 9 columns):\n",
      "reviewerID        20473 non-null object\n",
      "asin              20473 non-null object\n",
      "reviewerName      20260 non-null object\n",
      "helpful           20473 non-null object\n",
      "reviewText        20473 non-null object\n",
      "overall           20473 non-null float64\n",
      "summary           20473 non-null object\n",
      "unixReviewTime    20473 non-null int64\n",
      "reviewTime        20473 non-null object\n",
      "dtypes: float64(1), int64(1), object(7)\n",
      "memory usage: 1.6+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3cf83c898a3a4288b6740a1d7c68a442",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Dropdown(description='column', options=('overall', 'reviewerID'), value='overall'), IntS…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact\n",
    "def show_articles_more_than(column=['overall','reviewerID'], x=(0,10,1)):\n",
    "    return df.loc[df[column] >= x]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_to_timestamp(x): \n",
    "    month, day, year = re.split(' |, ',str(x))\n",
    "    \n",
    "    return pd.Timestamp(year=int(year), month=int(month), day=int(day))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_products_reviewed(start_date, end_date):\n",
    "\n",
    "    date = df['reviewTime'].apply(lambda x: convert_to_timestamp(x))\n",
    "    \n",
    "    start_date = pd.Timestamp(start_date)\n",
    "    end_date = pd.Timestamp(end_date)\n",
    "    \n",
    "    stat_df = df.loc[(date >= start_date) & (date <= end_date)].copy()\n",
    "    total_words = stat_df['reviewTime'].sum()\n",
    "\n",
    "    num_articles = len(stat_df)\n",
    "    print(f'You reviewed {num_articles} products between {start_date.date()} and {end_date.date()}.');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79dd77120d424c9caecf14ac41687184",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(DatePicker(value=Timestamp('2011-01-01 00:00:00'), description='start_date'), DatePicker…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create interactive version of function with DatePickers\n",
    "interact(print_products_reviewed,\n",
    "         start_date=widgets.DatePicker(value=pd.to_datetime('2011-01-01')),\n",
    "         end_date=widgets.DatePicker(value=pd.to_datetime('2014-01-01')));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I needed a set of jumper cables for my new car and these had good reviews and were at a good price.  They have been used a few times already and do what they are supposed to - no complaints there.What I will say is that 12 feet really isn't an ideal length.  Sure, if you pull up front bumper to front bumper they are plenty long, but a lot of times you will be beside another car or can't get really close.  Because of this, I would recommend something a little longer than 12'.Great brand - get 16' version though.\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['reviewText'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a nice corpus of text, lets go through some of the standard preprocessing required for almost any topic modeling or NLP problem.\n",
    "\n",
    "Approach will involve:\n",
    "\n",
    "- Tokenizing: converting a document to its atomic elements\n",
    "- Stopping: removing meaningless words\n",
    "- Stemming: merging words that are equivalent in meaning\n",
    "\n",
    "#### Tokenization\n",
    "We have many ways to segment our document into its atomic elements. To start we'll tokenize the document into words. For this instance we'll use NLTK’s tokenize.regexp module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = RegexpTokenizer(r'\\w+')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running through part of the first review to demonstrate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_1 = df.reviewText[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "516 characters in string vs 103 words in a list\n",
      "['i', 'needed', 'a', 'set', 'of', 'jumper', 'cables', 'for', 'my', 'new']\n"
     ]
    }
   ],
   "source": [
    "# Using one of our docs as an example\n",
    "tokens = tokenizer.tokenize(doc_1.lower())\n",
    "\n",
    "print('{} characters in string vs {} words in a list'.format(len(doc_1), len(tokens)))\n",
    "print(tokens[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stop Words\n",
    "\n",
    "Determiners like \"the\" and conjunctions such as \"or\" and \"for\" do not add value to our simple topic model. We refer to these types of words as stop words and want to remove them from our list of tokens. The definition of a stop work changes depending on the context of the documents we are examining. If considering Product Reviews for children's board games on Amazon.com we would not find \"Chutes and Ladders\" as a token and eventually an entity in some other model if we remove the word \"and\" as we'll end up with a distinct \"chutes\" AND \"ladders\" in our list.\n",
    "\n",
    "Let's make a super list of stop words from the stop_words and nltk package below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/chrisjcc/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "import ssl\n",
    "\n",
    "try:\n",
    "    _create_unverified_https_context = ssl._create_unverified_context\n",
    "except AttributeError:\n",
    "    pass\n",
    "else:\n",
    "    ssl._create_default_https_context = _create_unverified_https_context\n",
    "\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "211\n",
      "['by', 'don', \"wouldn't\", 'ought', 'yourselves', 'isn', 'her', \"hasn't\", \"i'd\", 'each']\n"
     ]
    }
   ],
   "source": [
    "nltk_stpwd = stopwords.words('english')\n",
    "stop_words_stpwd = get_stop_words('en')\n",
    "merged_stopwords = list(set(nltk_stpwd + stop_words_stpwd))\n",
    "\n",
    "print(len(set(merged_stopwords)))\n",
    "print(merged_stopwords[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['needed', 'set', 'jumper', 'cables', 'new', 'car', 'good', 'reviews', 'good', 'price']\n"
     ]
    }
   ],
   "source": [
    "stopped_tokens = [token for token in tokens if not token in merged_stopwords]\n",
    "print(stopped_tokens[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming\n",
    "\n",
    "Stemming allows us to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For instance, running and runner to run. Another example:\n",
    "\n",
    "Amazon's catalog contains bike tires in different sizes and colors $\\Rightarrow$ Amazon catalog contain bike tire in differ size and color\n",
    "\n",
    "Stemming is a basic and crude heuristic compared to Lemmatization which understands vocabulary and morphological analysis instead of lobbing off the end of words. Essentially Lemmatization removes inflectional endings to return the word to its base or dictionary form of a word, which is defined as the lemma. Great illustrative examples from Wikipedia:\n",
    "\n",
    "- The word \"better\" has \"good\" as its lemma. This link is missed by stemming, as it requires a dictionary look-up.\n",
    "- The word \"walk\" is the base form for word \"walking\", and hence this is matched in both stemming and lemmatisation.\n",
    "- The word \"meeting\" can be either the base form of a noun or a form of a verb (\"to meet\") depending on the context, e.g., \"in our last meeting\" or \"We are meeting again tomorrow\". Unlike stemming, lemmatisation can in principle select the appropriate lemma depending on the context.\n",
    "\n",
    "We'll start with the common Snowball stemming method, a successor of sorts of the original Porter Stemmer which is implemented in NLTK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a Snowball stemmer\n",
    "sb_stemmer = SnowballStemmer('english')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that p_stemmer requires all tokens to be type str. p_stemmer returns the string parameter in stemmed form, so we need to loop through our stopped_tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['need', 'set', 'jumper', 'cabl', 'new', 'car', 'good', 'review', 'good', 'price', 'use', 'time', 'alreadi', 'suppos', 'complaint', 'say', '12', 'feet', 'realli', 'ideal', 'length', 'sure', 'pull', 'front', 'bumper', 'front', 'bumper', 'plenti', 'long', 'lot', 'time', 'besid', 'anoth', 'car', 'get', 'realli', 'close', 'recommend', 'someth', 'littl', 'longer', '12', 'great', 'brand', 'get', '16', 'version', 'though']\n"
     ]
    }
   ],
   "source": [
    "stemmed_tokens = [sb_stemmer.stem(token) for token in stopped_tokens]\n",
    "print(stemmed_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting together a document-term matrix\n",
    "\n",
    "In order to create an LDA model we'll need to put the 3 steps from above (tokenizing, stopping, stemming) together to create a list of documents (list of lists) to then generate a document-term matrix (unique terms as rows, documents or reviews as columns). This matrix will tell us how frequently each term occurs with each individual document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 16.2 s, sys: 110 ms, total: 16.3 s\n",
      "Wall time: 16.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "num_reviews = df.shape[0]\n",
    "\n",
    "doc_set = [df.reviewText[i] for i in range(num_reviews)]\n",
    "\n",
    "texts = []\n",
    "\n",
    "for doc in doc_set:\n",
    "    # putting our three steps together\n",
    "    tokens = tokenizer.tokenize(doc.lower())\n",
    "    stopped_tokens = [token for token in tokens if not token in merged_stopwords]\n",
    "    stemmed_tokens = [sb_stemmer.stem(token) for token in stopped_tokens]\n",
    "    \n",
    "    # add tokens to list\n",
    "    texts.append(stemmed_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['need', 'set', 'jumper', 'cabl', 'new', 'car', 'good', 'review', 'good', 'price', 'use', 'time', 'alreadi', 'suppos', 'complaint', 'say', '12', 'feet', 'realli', 'ideal', 'length', 'sure', 'pull', 'front', 'bumper', 'front', 'bumper', 'plenti', 'long', 'lot', 'time', 'besid', 'anoth', 'car', 'get', 'realli', 'close', 'recommend', 'someth', 'littl', 'longer', '12', 'great', 'brand', 'get', '16', 'version', 'though']\n"
     ]
    }
   ],
   "source": [
    "print(texts[0]) # examine review 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform tokenized documents into an id-term dictionary\n",
    "\n",
    "Gensim's Dictionary method encapsulates the mapping between normalized words and their integer ids. \n",
    "\n",
    "Note: a term will have an id of some number and in the subsequent bag of words step we can see that id will have a count associated with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(19216 unique tokens: ['12', '16', 'alreadi', 'anoth', 'besid']...)\n"
     ]
    }
   ],
   "source": [
    "# Gensim's Dictionary encapsulates the mapping between normalized words and their integer ids.\n",
    "texts_dict = corpora.Dictionary(texts)\n",
    "texts_dict.save('auto_review.dict') # lets save to disk for later use\n",
    "# Examine each token’s unique id\n",
    "print(texts_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the mapping between words and their ids we can use the token2id method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IDs 1 through 10: [('12', 0), ('16', 1), ('alreadi', 2), ('anoth', 3), ('besid', 4), ('brand', 5), ('bumper', 6), ('cabl', 7), ('car', 8), ('close', 9)]\n"
     ]
    }
   ],
   "source": [
    "print(\"IDs 1 through 10: {}\".format(sorted(texts_dict.token2id.items(),\n",
    "                                           key=operator.itemgetter(1),\n",
    "                                           reverse = False)[:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to guess the original work and examine the count difference between our #1 most frequent term and our #10 most frequent term:\n",
    "\n",
    "print(df.reviewText.str.contains(\"complaint\").value_counts()) print() print(df.reviewText.str.contains(\"lot\").value_counts())\n",
    "\n",
    "We have a lot of unique tokens, let's see what happens if we ignore tokens that appear in less than 30 documents or more than 15% documents. Granted this is arbitrary but a quick search shows tons of methods for reducing noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(2464 unique tokens: ['12', '16', 'alreadi', 'anoth', 'besid']...)\n",
      "top terms:\n",
      "[('12', 0), ('16', 1), ('alreadi', 2), ('anoth', 3), ('besid', 4), ('brand', 5), ('bumper', 6), ('cabl', 7), ('close', 8), ('complaint', 9)]\n"
     ]
    }
   ],
   "source": [
    "texts_dict.filter_extremes(no_below=30, no_above=0.15) # inlace filter\n",
    "print(texts_dict)\n",
    "print(\"top terms:\")\n",
    "print(sorted(texts_dict.token2id.items(), key=operator.itemgetter(1), reverse = False)[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We went from 19216 unique tokens to 2462 after filtering. Looking at the top 10 tokens it looks like we got more specific subjects opposed to adjectives.\n",
    "\n",
    "#### Creating bag of words\n",
    "Next let's turn texts_dict into a bag of words instead. doc2bow converts a document (a list of words) into the bag-of-words format (list of (token_id, token_count) tuples)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20473"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = [texts_dict.doc2bow(text) for text in texts]\n",
    "len(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corpus is 20473 long, the amount of reviews in our dataset and in our dataframe. Let's dump this bag-of-words into a file to avoid parsing the entire text again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 770 ms, sys: 35.1 ms, total: 805 ms\n",
      "Wall time: 810 ms\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "# Matrix Market format https://radimrehurek.com/gensim/corpora/mmcorpus.html, why exactly? I don't know\n",
    "gensim.corpora.MmCorpus.serialize('amzn_auto_review.mm', corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training an LDA model\n",
    "\n",
    "As a topic modeling newbie this part is unsatisfying to me. In this unsupervised learning application I can see how a lot of people would arbitrarily set a number of topics, similar to centroids in k-means clustering, and then have a human evaluate if the topics \"make sense\". You can go very deep very quickly by researching this online. For now let's plead ignorance and go through with a simple model FULL of assumptions!\n",
    "\n",
    "Training an LDA model using our BOW corpus as training data.\n",
    "\n",
    "\n",
    "The number of topics is arbitrary, I'll use the browse taxonomy visible off https://www.amazon.com/automotive to inform the number we choose:\n",
    "\n",
    "1. Performance Parts & Accessories\n",
    "2. Replacement Parts\n",
    "3. Truck Accessories\n",
    "4. Interior Accessories\n",
    "5. Exterior Accessories\n",
    "6. Tires & Wheels\n",
    "7. Car Care\n",
    "8. Tools & Equipment\n",
    "9. Motorcycle & Powersports Accessories\n",
    "10. Car Electronics\n",
    "11. Enthusiast Merchandise\n",
    "\n",
    "I think these categories could be compressed into 5 general topics. We might consider rolling number 9 into 4 & 5, and rolling the products in number 3 across other accessory categories and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 29s, sys: 252 ms, total: 2min 29s\n",
      "Wall time: 2min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "lda_model = gensim.models.LdaModel(corpus,alpha='auto', \n",
    "                                   num_topics=5,\n",
    "                                   id2word=texts_dict, \n",
    "                                   passes=20)\n",
    "# ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=2, id2word = texts_dict, passes=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Gensim offers a fantastic multicore implementation of LDAModel that reduced my training time by 75%, but it does not have the auto alpha parameter available.\n",
    "\n",
    "#### Inferring Topics\n",
    "Below are the top 5 words associated with 5 random topics. The float next to each word is the weight showing how much the given word influences this specific topic. In this case, we see that for topic 4, light and battery are the most telling words. We might interpret that topic 4 might be close to Amazon's Tools & Equipment category which has a sub-category titled \"Jump Starters, Battery Chargers & Portable Power\". Similarly we might infer topic 1 refers to Car Care, maybe sub category \"Exterior Care\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.017*\"light\" + 0.013*\"blade\" + 0.012*\"instal\" + 0.010*\"wiper\" + 0.009*\"tire\"'),\n",
       " (1,\n",
       "  '0.017*\"towel\" + 0.016*\"clean\" + 0.013*\"dri\" + 0.013*\"wash\" + 0.010*\"wax\"'),\n",
       " (2, '0.016*\"hose\" + 0.009*\"fit\" + 0.009*\"water\" + 0.008*\"rv\" + 0.008*\"tape\"'),\n",
       " (3,\n",
       "  '0.014*\"oil\" + 0.012*\"drive\" + 0.011*\"filter\" + 0.009*\"engin\" + 0.008*\"app\"'),\n",
       " (4,\n",
       "  '0.032*\"batteri\" + 0.019*\"power\" + 0.018*\"charg\" + 0.015*\"light\" + 0.011*\"plug\"')]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For `num_topics` number of topics, return `num_words` most significant words\n",
    "lda_model.show_topics(num_topics=5,num_words=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that LDA is a probabilistic mixture of mixtures (or admixture) model for grouped data. The observed data (words) within the groups (documents) are the result of probabilistically choosing words from a specific topic (multinomial over the vocabulary), where the topic is itself drawn from a document-specific multinomial that has a global Dirichlet prior. This means that words can belong to various topics in various degrees. For example, the word 'pressure' might refer to a category/topic of automotive wash products and a category of tire products (in the case where we think the topics are about classes of products).\n",
    "\n",
    "##### Querying the LDA Model\n",
    "We cannot pass an arbitrary string to our model and evaluate what topics are most associated with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_query = 'portable air compressor'\n",
    "\n",
    "query_words = raw_query.split()\n",
    "query = []\n",
    "for word in query_words:\n",
    "    # ad-hoc reuse steps from above\n",
    "    q_tokens = tokenizer.tokenize(word.lower())\n",
    "    q_stopped_tokens = [word for word in q_tokens if not word in merged_stopwords]\n",
    "    q_stemmed_tokens = [sb_stemmer.stem(word) for word in q_stopped_tokens]\n",
    "    query.append(q_stemmed_tokens[0]) # different frome above, this is not a lists of lists!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['portabl', 'air', 'compressor']\n"
     ]
    }
   ],
   "source": [
    "print(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# translate words in query to ids and frequencies. \n",
    "id2word = gensim.corpora.Dictionary()\n",
    "_ = id2word.merge_with(texts_dict) # garbage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(595, 1), (2140, 1), (2369, 1)]\n"
     ]
    }
   ],
   "source": [
    "# translate this document into (word, frequency) pairs\n",
    "query = id2word.doc2bow(query)\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we run this constructed query against our trained model we will get each topic and the likelihood that the query relates to that topic. Remember we arbitrarily specified 11 topics when we made the model. When we organize this list to find the most relative topics, we see some intuitive results. We see that our query of 'battery powered inflator' relates most to a topic we thought might align to Amazon's Tools & Equipment category which has a sub-category titled \"Jump Starters, Battery Chargers & Portable Power\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(3, 0.026814938), (1, 0.031968977), (2, 0.03333397), (0, 0.04890519), (4, 0.8589769)]\n"
     ]
    }
   ],
   "source": [
    "a = list(sorted(lda_model[query], key=lambda x: x[1])) # sort by the second entry in the tuple\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.014*\"oil\" + 0.012*\"drive\" + 0.011*\"filter\" + 0.009*\"engin\" + 0.008*\"app\" + 0.008*\"chang\" + 0.008*\"price\" + 0.007*\"vehicl\" + 0.007*\"code\" + 0.006*\"amazon\"'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_model.print_topic(a[0][0]) #least related"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.032*\"batteri\" + 0.019*\"power\" + 0.018*\"charg\" + 0.015*\"light\" + 0.011*\"plug\" + 0.011*\"devic\" + 0.010*\"phone\" + 0.010*\"unit\" + 0.010*\"connect\" + 0.009*\"charger\"'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_model.print_topic(a[-1][0]) #most related"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What can we do with this in production?\n",
    "We could take these inferred topics and analyze the sentiment of their corresponding documents (reviews) to find out what customers are saying (or feeling) about specific products. We can also use an LDA model to extract representative statements or quotes, enabling us to summarize customers’ opinions about products, perhaps even displaying them on the site.We could also use LDA to model groups of customers to topics which are groups of products that frequently occur within some customer's orders over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}