DSFromScratch/Chap20/074-Natural_Language.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chap 20: Natural Language Processing\n",
    "\n",
    "#### Topic Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 300,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import math, random, re\n",
    "import numpy as np\n",
    "from collections import defaultdict, Counter\n",
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gibbs Sampling\n",
    "\n",
    "Generating samples from some distributions is easy. We can get uniform random variables with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 324,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.04523406786561235"
      ]
     },
     "execution_count": 324,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.random()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and normal random variables with:\n",
    "`inverse_normal_cdf(random.random())`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But some distributions are harder to sample from. *Gibbs sampling* is a technique for generating samples from multidimensional distributions when we only know some of the conditional distributions.\n",
    "\n",
    "For example, imagine rolling two dice. Let x by value from die 1 and y be the sum of the dice and image you wanted ot generate lots of (x,y) pairs. In this case it's easy to generate the samples directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 325,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def roll_a_die():\n",
    "    return random.choice([1,2,3,4,5,6])\n",
    "\n",
    "def direct_sample():\n",
    "    d1 = roll_a_die()\n",
    "    d2 = roll_a_die()\n",
    "    return d1, d1 + d2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 11)\n"
     ]
    }
   ],
   "source": [
    "print(direct_sample())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but imagine that you only knew the conditional distributions. The distribution of `y` conditional on `x` is easy - if you know the value of `x,y` is equally likely to be `x + 1, x + 2, x + 3, x + 4, x + 5, x + 6`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 327,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def random_y_given_x(x):\n",
    "    \"\"\"equally likely to x + 1, x + 2,.... x + 6\"\"\"\n",
    "    return x + roll_a_die()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 328,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    }
   ],
   "source": [
    "print(random_y_given_x(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other direction is more complicated. For example, if you know that `y` is 2, then necessarily `x` is 1 (since the only way two dice can sum to 2 is if both of them are 1). If you know `y` is 3, then `x` is equally likely to be 1 or 2. Similarly, if `y` is 11, then `x` has to be either 5 or 6:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def random_x_given_y(y):\n",
    "    if y <= 7:\n",
    "        # if the total is 7 or less, the first die is equally likely to be\n",
    "        # 1,2,...(total - 1)\n",
    "        return random.randrange(1, y) # choose randomly from range 1 - 7\n",
    "    else:\n",
    "        # if the total is 7 or more, the first die is equally likely to be\n",
    "        # (total - 6), (total - 5)... 6\n",
    "        return random.randrange(y - 6, 7) # choose randomly from range y -6, 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way Gibbs sampling works is that we start with any (valid) value for x and y and then repeatedly alternate replacing x with a random value picked conditional on y and replacing y with a random value picked conditional on x. After a number of iterations, the resulting values of x and y will represent a sample from the unconditional joint distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 330,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gibbs_sample(num_iters=100):\n",
    "    x,y = 1, 2 # doesn't really matter\n",
    "    for _ in range(num_iters):\n",
    "        x = random_x_given_y(y)\n",
    "        y = random_y_given_x(x)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the results of this to using a direct sample:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 331,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def compare_distributions(num_samples=1000):\n",
    "    counts = defaultdict(lambda: [0, 0])\n",
    "    for _ in range(num_samples):\n",
    "        counts[gibbs_sample()][0] += 1 # returns an x,y tuple based on 100 iterations\n",
    "        if _ < 4: \n",
    "            print(list(counts))\n",
    "            print(gibbs_sample())\n",
    "        counts[direct_sample()][1] += 1 # returns an x,y tuple\n",
    "        #if _ < 4: print(list(counts))\n",
    "    return counts # 2000 tuples where 1000 are from "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(5, 6)]\n",
      "(4, 8)\n",
      "[(5, 6), (4, 5), (4, 10)]\n",
      "(3, 5)\n",
      "[(5, 6), (4, 5), (6, 7), (5, 10), (4, 10)]\n",
      "(6, 9)\n",
      "[(4, 5), (6, 7), (4, 10), (5, 6), (5, 10), (6, 11), (2, 3)]\n",
      "(5, 9)\n",
      "defaultdict(<function compare_distributions.<locals>.<lambda> at 0x7ffadcd82840>, {(4, 10): [37, 24], (4, 7): [34, 28], (2, 6): [31, 35], (3, 7): [23, 24], (4, 8): [22, 31], (4, 5): [22, 35], (2, 8): [31, 25], (2, 7): [31, 31], (1, 4): [27, 26], (5, 9): [22, 26], (5, 6): [27, 20], (3, 9): [32, 30], (1, 6): [22, 23], (6, 12): [28, 28], (5, 10): [34, 33], (2, 5): [29, 31], (5, 8): [25, 27], (4, 6): [24, 35], (1, 2): [33, 24], (6, 9): [21, 39], (6, 7): [19, 28], (6, 8): [33, 29], (1, 5): [25, 37], (6, 10): [36, 20], (5, 7): [29, 33], (4, 9): [31, 25], (3, 8): [20, 18], (6, 11): [36, 22], (3, 6): [22, 34], (1, 7): [24, 29], (1, 3): [31, 23], (2, 3): [34, 30], (5, 11): [19, 23], (3, 5): [33, 26], (3, 4): [24, 19], (2, 4): [29, 29]})\n"
     ]
    }
   ],
   "source": [
    "print(compare_distributions())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Topic Modeling\n",
    "\n",
    "- Fixed number of K topics\n",
    "- p(w|k) : probability of seeing word w, given topic k\n",
    "- Another random variable with a probability distribution of topics for each document, d.\n",
    "- Each word in a document was generated by picking a random topic (from d distribution of topics), then picking a random word (from topics distribution of words - p(w|k))\n",
    "\n",
    "We have a collection of `documents`, each consisting of a `list` of words.    \n",
    "We have a collection of `document_topics` that assigns a topic to each word in each document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 333,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the fifth word in the fourth document is:\n",
    "# documents[3][4]\n",
    "\n",
    "# the topic from which the word was chosen is:\n",
    "# documents_topics[3][4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can estimate the likelihood that topic 1 produces a certain word by comparing how many times topic 1 produces that word with how many times topic 1 produces *any* word.\n",
    "\n",
    "We need Gibbs sampling to generate `document_topics`.\n",
    "\n",
    "We start by assigning every word a random topic. For each word in each document we construct weights for each topic that depend on the (current) distribution of topics in that document and the (current) distribution of words for that topic.  - we create priors for p(t|d) and p(w|t) by assigning random topics to words\n",
    "\n",
    "We then use those weights to sample a new topic for that word. If we iterate this process many times, we will end up with a joint sample from the topic-word distribution (p(t|w)) and the document-topic distribution (p(d|t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start with we'll need a function to randomly choose an index based on an arbitrary set of weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 334,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sample_from(weights):\n",
    "    \"\"\"returns i with probability weights[i]/ sum(weights)\"\"\"\n",
    "    total = sum(weights)\n",
    "    rnd = total * random.random()  # uniform between 0 and total\n",
    "    for i, w in enumerate(weights):\n",
    "        rnd -= w\n",
    "        if rnd <= 0: return i # return the smallest i such that weights[0] + ... + weights[i]\n",
    "                              # >= rnd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, if you give it weights[1,1,3] then one-fifth of the time it will return 0, one fifth of the time it will return 1 and 3/5 of the time it will return 2. i.e. the index is the number 0 = 1/5, 1 = 1/5, 2 = 3/5.\n",
    "\n",
    "Our documents are our users' interests which look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 335,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "documents = [\n",
    "    [\"Hadoop\", \"Big Data\", \"HBase\", \"Java\", \"Spark\", \"Storm\", \"Cassandra\"],\n",
    "    [\"NoSQL\", \"MongoDB\", \"Cassandra\", \"HBase\", \"Postgres\"],\n",
    "    [\"Python\", \"scikit-learn\", \"scipy\", \"numpy\", \"statsmodels\", \"pandas\"],\n",
    "    [\"R\", \"Python\", \"statistics\", \"regression\", \"probability\"],\n",
    "    [\"machine learning\", \"regression\", \"decision trees\", \"libsvm\"],\n",
    "    [\"Python\", \"R\", \"Java\", \"C++\", \"Haskell\", \"programming languages\"],\n",
    "    [\"statistics\", \"probability\", \"mathematics\", \"theory\"],\n",
    "    [\"machine learning\", \"scikit-learn\", \"Mahout\", \"neural networks\"],\n",
    "    [\"neural networks\", \"deep learning\", \"Big Data\", \"artificial intelligence\"],\n",
    "    [\"Hadoop\", \"Java\", \"MapReduce\", \"Big Data\"],\n",
    "    [\"statistics\", \"R\", \"statsmodels\"],\n",
    "    [\"C++\", \"deep learning\", \"artificial intelligence\", \"probability\"],\n",
    "    [\"pandas\", \"R\", \"Python\"],\n",
    "    [\"databases\", \"HBase\", \"Postgres\", \"MySQL\", \"MongoDB\"],\n",
    "    [\"libsvm\", \"regression\", \"support vector machines\"]\n",
    "]\n",
    "K = 4 # we'll try to find 4 topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to calculate the sampling weights, we'll need to keep track of several counts. We will create data structures for the counts.\n",
    "\n",
    "How many times each topic is assigned to each document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 336,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# a list of Counters, one for each document:\n",
    "document_topic_counts = [Counter() for _ in documents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 337,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter(), Counter()]\n"
     ]
    }
   ],
   "source": [
    "print(document_topic_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# a list of Counters, one for each topic\n",
    "topic_word_counts = [Counter() for _ in range(K)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# a list of numbers, one for each topic\n",
    "topic_counts = [0 for _ in range(K)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "print(topic_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 365,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# a list of numbers, one for each document\n",
    "document_lengths = [len(d) for d in documents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 443,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7, 5, 6, 5, 4, 6, 4, 4, 4, 4, 3, 4, 3, 5, 3]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[4, 6, 7, 8, 9, 11]"
      ]
     },
     "execution_count": 443,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print([x for x in document_lengths])\n",
    "\n",
    "# what are the index values where value=4\n",
    "[i for i,x in enumerate(document_lengths) if x == 4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 367,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# number of distinct words\n",
    "# getting word from document and document from documents\n",
    "distinct_words = set(word for document in documents for word in document)\n",
    "W = len(distinct_words)\n",
    "\n",
    "D = len(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 368,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Big Data', 'Cassandra', 'pandas', 'scikit-learn', 'MapReduce', 'Hadoop', 'programming languages', 'neural networks', 'R', 'support vector machines', 'decision trees', 'NoSQL', 'Storm', 'machine learning', 'probability', 'Postgres', 'C++', 'Haskell', 'MySQL', 'scipy', 'Mahout', 'statistics', 'mathematics', 'artificial intelligence', 'libsvm', 'databases', 'Spark', 'MongoDB', 'statsmodels', 'deep learning', 'regression', 'theory', 'Java', 'numpy', 'HBase', 'Python'}\n"
     ]
    }
   ],
   "source": [
    "print(distinct_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once these are populated we can find the number of words in `documents[3]` associated with topic 1 as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're ready to define our conditional probability functions. As with Naive Bayes, each has a smoothing term that ensures every topic has a nonzero chance of being chosen in any document and that every word has a nonzero chance of being chosen for any topic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 371,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def p_topic_given_document(topic, d, alpha=0.1):\n",
    "    \"\"\"the fraction of words in document _d_\n",
    "    that are assigned to _topic_ (plus some smoothing)\"\"\"\n",
    "    \n",
    "    return ((document_topic_counts[d][topic] + alpha)/\n",
    "             document_lengths[d] + K * alpha)\n",
    "\n",
    "def p_word_given_topic(word, topic, beta=0.1):\n",
    "    \"\"\"the fraction of words assigned to _topic_ \n",
    "    that equal _word_ (plus some smoothing)\"\"\"\n",
    "    \n",
    "    return ((topic_word_counts[topic][word] + beta)/\n",
    "             topic_counts[topic] + W * beta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use these to create the weights for updating topics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 372,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def topic_weight(d, word, k):\n",
    "    \"\"\"given a document and a word in that document\n",
    "    return the weight for the kth topic\"\"\"\n",
    "    return p_word_given_topic(word, k) * p_topic_given_document(k, d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 373,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def choose_new_topic(d, word):\n",
    "    return sample_from([topic_weight(d, word, k)\n",
    "                        for k in range(K)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have all the machinery we need. Now we will assign a random topic to every word and will also populate our counters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 374,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "random.seed(0)\n",
    "document_topics = [[random.randrange(K) for word in document] for document in documents]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many times in \"R\" mentioned in topic 0 (not labelled yet)?\n",
    "### An aside looking into exploring the different counters etc we have..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 476,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Big Data',\n",
       " 'Big Data',\n",
       " 'Big Data',\n",
       " 'pandas',\n",
       " 'pandas',\n",
       " 'pandas',\n",
       " 'statistics',\n",
       " 'statistics',\n",
       " 'scipy',\n",
       " 'artificial intelligence']"
      ]
     },
     "execution_count": 476,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is a list of dictionaries. Each dictionary has a key:value pair where it is word: count.\n",
    "topic_word_counts[0][\"R\"] # [0] identies the list of counters. \"R\" is a key value in counter dictionary 0.\n",
    "\n",
    "\n",
    "# This gives us the elements which have been counted. i.e. Big Data will be stored in topic [0] as \"Big Data\": 3\n",
    "list(topic_word_counts[0].elements())[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 405,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({0: 3, 1: 3, 2: 3, 3: 1})"
      ]
     },
     "execution_count": 405,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document_topic_counts[3] # not filled in yet - try this later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 424,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "print(documents[3][1]) # word number 1 in document 3\n",
    "print(document_topics[3][1]) # topic assigned to word number 1 in document 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 471,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(3, 2), (6, 0), (10, 0)]\n"
     ]
    }
   ],
   "source": [
    "# find the index of all values of \"Python\" in documents. To see if they've all been assigned the same topic. \n",
    "\n",
    "positions=[]\n",
    "for x,doc in enumerate(documents):     # We get the indices from x and y and using enumerate.\n",
    "    for y,word in enumerate(doc):\n",
    "        if word == \"statistics\":\n",
    "            tup = x,y\n",
    "            positions.append(tup)\n",
    "\n",
    "print(positions)  # we find \"Python\" attached to the words in document 2, position 0, doc 3, position 1 etc...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 472,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 0, 2]"
      ]
     },
     "execution_count": 472,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If we find that all of these indices have the same topic label then the script is doing what i think it is.\n",
    "# Use positions to search document_topics\n",
    "\n",
    "[document_topics[x][y] for x,y in positions]\n",
    "\n",
    "# these output values are always different no matter what document word we choose. So, why does every word get assigned\n",
    "# a different topic?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back to book code - populating the counters\n",
    "\n",
    "Here we fill in the counters stored in document_topic_counts, topic_word_counts and topic_counts. \n",
    "Topics were initially allocated at random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 375,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# D is the number of documents (15)\n",
    "for d in range(D):\n",
    "    for word, topic in zip(documents[d], document_topics[d]):  # zip together each word with its random topic assignment\n",
    "        document_topic_counts[d][topic] += 1   # everytime you call on a key with topic +1 is added to its count\n",
    "        topic_word_counts[topic][word] += 1    # everytime you call of key \"word\"\n",
    "        topic_counts[topic] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal is to get a joint sample of the topics-words distribution and the documents-topics distribution. We do this using a form of Gibbs sampling that uses the conditional probabilities defined previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for iter in range(1000):\n",
    "    for d in range(D):\n",
    "        for i,  (word, topic) in enumerate(zip(documents[d], document_topics[d])):\n",
    "            # remove this word / topic from the counts so that it doesn't\n",
    "            # influence the weights. undo everything from the previous cell.\n",
    "            document_topic_counts[d][topic] -= 1   \n",
    "            topic_word_counts[topic][word] -= 1\n",
    "            topic_counts[topic] -= 1\n",
    "            document_lengths[d] -= 1\n",
    "            \n",
    "            # choose a new topic based on the weights\n",
    "            new_topic = choose_new_topic(d, word)\n",
    "            document_topics[d][i] = new_topic\n",
    "            \n",
    "            # same procedure as cell above\n",
    "            document_topic_counts[d][new_topic] += 1\n",
    "            topic_word_counts[new_topic][word] += 1\n",
    "            topic_counts[new_topic] += 1\n",
    "            document_lengths[d] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the five most heavily weighted words for each of the topics, labeled 0,1,2 and 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 478,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Big Data 3\n",
      "0 pandas 3\n",
      "0 scikit-learn 3\n",
      "0 statistics 2\n",
      "0 statsmodels 2\n",
      "1 neural networks 4\n",
      "1 databases 2\n",
      "1 MySQL 2\n",
      "1 deep learning 2\n",
      "1 decision trees 2\n",
      "2 R 5\n",
      "2 Python 4\n",
      "2 regression 3\n",
      "2 Java 3\n",
      "2 Cassandra 2\n",
      "3 probability 3\n",
      "3 Python 3\n",
      "3 Big Data 2\n",
      "3 libsvm 2\n",
      "3 support vector machines 2\n"
     ]
    }
   ],
   "source": [
    "for k, word_counts in enumerate(topic_word_counts):\n",
    "    for word, count in word_counts.most_common(5):\n",
    "        if count > 1: print (k, word, count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on these results we will name the topics as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 384,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topic_names = [\"Big Data and programming languages\",\n",
    "                   \"databases\",\n",
    "                   \"machine learning\",\n",
    "                   \"statistics\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see how the model assigns topics to each user's interests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 479,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hadoop', 'Big Data', 'HBase', 'Java', 'Spark', 'Storm', 'Cassandra']\n",
      "Big Data and programming languages 4\n",
      "machine learning 4\n",
      "statistics 3\n",
      "databases 1\n",
      "\n",
      "['NoSQL', 'MongoDB', 'Cassandra', 'HBase', 'Postgres']\n",
      "machine learning 5\n",
      "statistics 3\n",
      "databases 2\n",
      "\n",
      "['Python', 'scikit-learn', 'scipy', 'numpy', 'statsmodels', 'pandas']\n",
      "Big Data and programming languages 5\n",
      "statistics 3\n",
      "databases 2\n",
      "machine learning 2\n",
      "\n",
      "['R', 'Python', 'statistics', 'regression', 'probability']\n",
      "Big Data and programming languages 3\n",
      "databases 3\n",
      "machine learning 3\n",
      "statistics 1\n",
      "\n",
      "['machine learning', 'regression', 'decision trees', 'libsvm']\n",
      "statistics 4\n",
      "databases 2\n",
      "Big Data and programming languages 1\n",
      "machine learning 1\n",
      "\n",
      "['Python', 'R', 'Java', 'C++', 'Haskell', 'programming languages']\n",
      "Big Data and programming languages 5\n",
      "machine learning 4\n",
      "statistics 3\n",
      "\n",
      "['statistics', 'probability', 'mathematics', 'theory']\n",
      "Big Data and programming languages 3\n",
      "machine learning 2\n",
      "statistics 2\n",
      "databases 1\n",
      "\n",
      "['machine learning', 'scikit-learn', 'Mahout', 'neural networks']\n",
      "databases 3\n",
      "machine learning 3\n",
      "Big Data and programming languages 1\n",
      "statistics 1\n",
      "\n",
      "['neural networks', 'deep learning', 'Big Data', 'artificial intelligence']\n",
      "databases 4\n",
      "Big Data and programming languages 2\n",
      "statistics 2\n",
      "\n",
      "['Hadoop', 'Java', 'MapReduce', 'Big Data']\n",
      "Big Data and programming languages 4\n",
      "databases 2\n",
      "machine learning 1\n",
      "statistics 1\n",
      "\n",
      "['statistics', 'R', 'statsmodels']\n",
      "machine learning 4\n",
      "Big Data and programming languages 2\n",
      "\n",
      "['C++', 'deep learning', 'artificial intelligence', 'probability']\n",
      "machine learning 3\n",
      "Big Data and programming languages 2\n",
      "statistics 2\n",
      "databases 1\n",
      "\n",
      "['pandas', 'R', 'Python']\n",
      "machine learning 3\n",
      "Big Data and programming languages 2\n",
      "statistics 1\n",
      "\n",
      "['databases', 'HBase', 'Postgres', 'MySQL', 'MongoDB']\n",
      "databases 8\n",
      "Big Data and programming languages 1\n",
      "machine learning 1\n",
      "\n",
      "['libsvm', 'regression', 'support vector machines']\n",
      "machine learning 2\n",
      "statistics 2\n",
      "Big Data and programming languages 1\n",
      "databases 1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for document, topic_counts in zip(documents, document_topic_counts):\n",
    "    print(document)\n",
    "    for topic, count in topic_counts.most_common():\n",
    "        if count > 0: \n",
    "            print(topic_names[topic], count)\n",
    "    print()\n",
    "\n",
    "    \n",
    "# uncomment to see the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}