docs/notebooks/topic_coherence_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstration of the topic coherence pipeline in Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the `u_mass` and `c_v` coherence for two different LDA models: a \"good\" and a \"bad\" LDA model. The good LDA model will be trained over 50 iterations and the bad one for 1 iteration. Hence in theory, the good LDA model will be able come up with better or more human-understandable topics. Therefore the coherence measure output for the good LDA model should be more (better) than that for the bad LDA model. This is because, simply, the good LDA model usually comes up with better topics that are more human interpretable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/vru959/anaconda2/lib/python2.7/site-packages/scipy/sparse/sparsetools.py:20: DeprecationWarning: `scipy.sparse.sparsetools` is deprecated!\n",
      "scipy.sparse.sparsetools is a private module for scipy.sparse, and should not be used.\n",
      "  _deprecated()\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import os\n",
    "import logging\n",
    "import json\n",
    "import warnings\n",
    "\n",
    "try:\n",
    "    import pyLDAvis.gensim\n",
    "    CAN_VISUALIZE = True\n",
    "    pyLDAvis.enable_notebook()\n",
    "    from IPython.display import display\n",
    "except ImportError:\n",
    "    ValueError(\"SKIP: please install pyLDAvis\")\n",
    "    CAN_VISUALIZE = False\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from gensim.models import CoherenceModel, LdaModel, HdpModel\n",
    "from gensim.models.wrappers import LdaVowpalWabbit, LdaMallet\n",
    "from gensim.corpora import Dictionary\n",
    "\n",
    "warnings.filterwarnings('ignore')  # To ignore all warnings that arise here to enhance clarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As stated in table 2 from [this](http://www.cs.bham.ac.uk/~pxt/IDA/lsa_ind.pdf) paper, this corpus essentially has two classes of documents. First five are about human-computer interaction and the other four are about graphs. We will be setting up two LDA models. One with 50 iterations of training and the other with just 1. Hence the one with 50 iterations (\"better\" model) should be able to capture this underlying pattern of the corpus better than the \"bad\" LDA model. Therefore, in theory, our topic coherence for the good LDA model should be greater than the one for the bad LDA model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "texts = [['human', 'interface', 'computer'],\n",
    "         ['survey', 'user', 'computer', 'system', 'response', 'time'],\n",
    "         ['eps', 'user', 'interface', 'system'],\n",
    "         ['system', 'human', 'system', 'eps'],\n",
    "         ['user', 'response', 'time'],\n",
    "         ['trees'],\n",
    "         ['graph', 'trees'],\n",
    "         ['graph', 'minors', 'trees'],\n",
    "         ['graph', 'minors', 'survey']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dictionary = Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up two topic models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be setting up two different LDA Topic models. A good one and bad one. To build a \"good\" topic model, we'll simply train it using more iterations than the bad one. Therefore the `u_mass` coherence should in theory be better for the good model than the bad one since it would be producing more \"human-interpretable\" topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "goodLdaModel = LdaModel(corpus=corpus, id2word=dictionary, iterations=50, num_topics=2)\n",
    "badLdaModel = LdaModel(corpus=corpus, id2word=dictionary, iterations=1, num_topics=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using U_Mass Coherence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "goodcm = CoherenceModel(model=goodLdaModel, corpus=corpus, dictionary=dictionary, coherence='u_mass')\n",
    "badcm = CoherenceModel(model=badLdaModel, corpus=corpus, dictionary=dictionary, coherence='u_mass')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View the pipeline parameters for one coherence model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following are the pipeline parameters for `u_mass` coherence. By pipeline parameters, we mean the functions being used to calculate segmentation, probability estimation, confirmation measure and aggregation as shown in figure 1 in [this](http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf) paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coherence_Measure(seg=<function s_one_pre at 0x11e3216e0>, prob=<function p_boolean_document at 0x11e334230>, conf=<function log_conditional_probability at 0x11e338c08>, aggr=<function arithmetic_mean at 0x11e33d230>)\n"
     ]
    }
   ],
   "source": [
    "print(goodcm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting the topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we will see below using LDA visualization, the better model comes up with two topics composed of the following words:\n",
    "1. goodLdaModel:\n",
    "    - __Topic 1__: More weightage assigned to words such as \"system\", \"user\", \"eps\", \"interface\" etc which captures the first set of documents.\n",
    "    - __Topic 2__: More weightage assigned to words such as \"graph\", \"trees\", \"survey\" which captures the topic in the second set of documents.\n",
    "2. badLdaModel:\n",
    "    - __Topic 1__: More weightage assigned to words such as \"system\", \"user\", \"trees\", \"graph\" which doesn't make the topic clear enough.\n",
    "    - __Topic 2__: More weightage assigned to words such as \"system\", \"trees\", \"graph\", \"user\" which is similar to the first topic. Hence both topics are not human-interpretable.\n",
    "\n",
    "Therefore, the topic coherence for the goodLdaModel should be greater for this than the badLdaModel since the topics it comes up with are more human-interpretable. We will see this using `u_mass` and `c_v` topic coherence measures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize topic models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el4592648534317606780006507\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el4592648534317606780006507_data = {\"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 2], \"token.table\": {\"Topic\": [1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2], \"Freq\": [0.4692125100483321, 0.4692125100483321, 0.48036588248303724, 0.48036588248303724, 0.6975749408191356, 0.3487874704095678, 0.4784868656682996, 0.4784868656682996, 0.47341949002228034, 0.47341949002228034, 0.9299670287855416, 0.4649835143927708, 0.4674488829453285, 0.4674488829453285, 0.4673208871941226, 0.4673208871941226, 0.2890062037550016, 0.5780124075100032, 0.4687980245217653, 0.4687980245217653, 0.7014349048045118, 0.3507174524022559, 0.7033719592765316, 0.3516859796382658], \"Term\": [\"computer\", \"computer\", \"eps\", \"eps\", \"graph\", \"graph\", \"human\", \"human\", \"interface\", \"interface\", \"minors\", \"minors\", \"response\", \"response\", \"survey\", \"survey\", \"system\", \"system\", \"time\", \"time\", \"trees\", \"trees\", \"user\", \"user\"]}, \"mdsDat\": {\"y\": [0.0, 0.0], \"cluster\": [1, 1], \"Freq\": [55.73087987945022, 44.26912012054978], \"topics\": [1, 2], \"x\": [0.022556279531108087, -0.022556279531108087]}, \"R\": 12, \"lambda.step\": 0.01, \"tinfo\": {\"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\"], \"Term\": [\"system\", \"eps\", \"human\", \"graph\", \"minors\", \"interface\", \"survey\", \"trees\", \"response\", \"time\", \"computer\", \"user\", \"minors\", \"graph\", \"survey\", \"response\", \"trees\", \"time\", \"user\", \"computer\", \"interface\", \"human\", \"eps\", \"system\", \"system\", \"eps\", \"human\", \"interface\", \"computer\", \"user\", \"time\", \"trees\", \"response\", \"survey\", \"graph\", \"minors\"], \"loglift\": [12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 0.2496, 0.2473, 0.1638, 0.1589, 0.1522, 0.1056, 0.1008, 0.0886, -0.1023, -0.3946, -0.5291, -0.635, 0.4648, 0.4169, 0.3439, 0.1155, -0.124, -0.1433, -0.151, -0.2318, -0.2444, -0.2537, -0.4357, -0.4415], \"Freq\": [3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 1.538381346204438, 2.04608932032542, 1.40485597900046, 1.397582624125938, 1.8502393333971954, 1.3211592567144792, 1.752765521959243, 1.2977685923404727, 1.0626736577737315, 0.7849866961420866, 0.683507496342354, 1.0219453407147447, 2.4381879955499923, 1.3982390149438886, 1.3049348288788902, 1.0496179167724229, 0.8334618812395969, 1.0906802162773837, 0.8119555340744314, 1.001058750431974, 0.7416887197248446, 0.7350012955249248, 0.8209861477977274, 0.612232533743361], \"Total\": [3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.150613879947799, 2.867075468123147, 2.1398572745253848, 2.1392713438507824, 2.8512980838291693, 2.1331147907889108, 2.8434457382366265, 2.13123047358007, 2.1122915745461546, 2.089921525020977, 2.0817465112862426, 3.460133336264737, 3.460133336264737, 2.0817465112862426, 2.089921525020977, 2.1122915745461546, 2.13123047358007, 2.8434457382366265, 2.1331147907889108, 2.8512980838291693, 2.1392713438507824, 2.1398572745253848, 2.867075468123147, 2.150613879947799], \"logprob\": [12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -2.3519, -2.0667, -2.4427, -2.4479, -2.1673, -2.5042, -2.2215, -2.522, -2.7219, -3.0247, -3.1632, -2.761, -1.6612, -2.2172, -2.2863, -2.504, -2.7346, -2.4656, -2.7607, -2.5514, -2.8512, -2.8603, -2.7497, -3.0431]}};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el4592648534317606780006507\", ldavis_el4592648534317606780006507_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el4592648534317606780006507\", ldavis_el4592648534317606780006507_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el4592648534317606780006507\", ldavis_el4592648534317606780006507_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if CAN_VISUALIZE:\n",
    "    prepared = pyLDAvis.gensim.prepare(goodLdaModel, corpus, dictionary)\n",
    "    display(pyLDAvis.display(prepared))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el4592645661155367187472854\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el4592645661155367187472854_data = {\"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 2], \"token.table\": {\"Topic\": [1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2], \"Freq\": [0.47127343135554306, 0.47127343135554306, 0.4712030409081992, 0.4712030409081992, 0.7070186647778998, 0.3535093323889499, 0.4712910662410224, 0.4712910662410224, 0.4711614921066636, 0.4711614921066636, 0.47132017677255156, 0.47132017677255156, 0.4712698177027075, 0.4712698177027075, 0.47138034409750634, 0.47138034409750634, 0.5656167397316655, 0.28280836986583274, 0.47126175533036385, 0.47126175533036385, 0.3533589346056535, 0.706717869211307, 0.3533729508860421, 0.7067459017720842], \"Term\": [\"computer\", \"computer\", \"eps\", \"eps\", \"graph\", \"graph\", \"human\", \"human\", \"interface\", \"interface\", \"minors\", \"minors\", \"response\", \"response\", \"survey\", \"survey\", \"system\", \"system\", \"time\", \"time\", \"trees\", \"trees\", \"user\", \"user\"]}, \"mdsDat\": {\"y\": [0.0, 0.0], \"cluster\": [1, 1], \"Freq\": [52.62107962766313, 47.37892037233687], \"topics\": [1, 2], \"x\": [0.0022452603163718794, -0.0022452603163718794]}, \"R\": 12, \"lambda.step\": 0.01, \"tinfo\": {\"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\"], \"Term\": [\"trees\", \"user\", \"interface\", \"eps\", \"survey\", \"system\", \"time\", \"graph\", \"response\", \"computer\", \"human\", \"minors\", \"survey\", \"system\", \"graph\", \"minors\", \"human\", \"computer\", \"response\", \"time\", \"eps\", \"user\", \"interface\", \"trees\", \"trees\", \"interface\", \"user\", \"eps\", \"time\", \"response\", \"computer\", \"human\", \"minors\", \"graph\", \"system\", \"survey\"], \"loglift\": [12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 0.1244, 0.0905, 0.0889, 0.0618, 0.03, 0.0103, 0.0062, -0.003, -0.0726, -0.1218, -0.125, -0.1462, 0.1407, 0.1226, 0.1199, 0.0749, 0.0033, -0.0069, -0.0116, -0.0344, -0.0734, -0.109, -0.1111, -0.1591], \"Freq\": [2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 1.264153321593625, 2.03682725788834, 1.6269062343609695, 1.1876099122291406, 1.150569185712821, 1.1281280371867524, 1.1235293015457002, 1.1132688699679398, 1.0385364388657357, 1.3183346816594803, 0.9856413649269657, 1.2866084860848348, 1.5433751426466535, 1.136773171814538, 1.5115366978352291, 1.0836909518416644, 1.0086941127888087, 0.9983973791712749, 0.9937823729198795, 0.9712618262261813, 0.9340900473348168, 1.2018733999479392, 1.499135982783817, 0.8572758226668903], \"Total\": [2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.121429144260515, 3.5359632406721575, 2.8287796343089084, 2.1216999595639576, 2.1218310119390025, 2.121910410106632, 2.121926680716975, 2.1219629827567488, 2.1222273907074003, 2.8298713794947092, 2.1224145367415037, 2.8299836287314886, 2.8299836287314886, 2.1224145367415037, 2.8298713794947092, 2.1222273907074003, 2.1219629827567488, 2.121926680716975, 2.121910410106632, 2.1218310119390025, 2.1216999595639576, 2.8287796343089084, 3.5359632406721575, 2.121429144260515], \"logprob\": [12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -2.4908, -2.0138, -2.2386, -2.5533, -2.585, -2.6047, -2.6088, -2.6179, -2.6874, -2.4489, -2.7397, -2.4732, -2.1863, -2.4921, -2.2072, -2.5399, -2.6116, -2.6219, -2.6265, -2.6495, -2.6885, -2.4364, -2.2154, -2.7743]}};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el4592645661155367187472854\", ldavis_el4592645661155367187472854_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el4592645661155367187472854\", ldavis_el4592645661155367187472854_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el4592645661155367187472854\", ldavis_el4592645661155367187472854_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if CAN_VISUALIZE:\n",
    "    prepared = pyLDAvis.gensim.prepare(badLdaModel, corpus, dictionary)\n",
    "    display(pyLDAvis.display(prepared))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-13.8029561191\n",
      "-14.1531313765\n"
     ]
    }
   ],
   "source": [
    "print(goodcm.get_coherence())\n",
    "print(badcm.get_coherence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using C_V coherence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "goodcm = CoherenceModel(model=goodLdaModel, texts=texts, dictionary=dictionary, coherence='c_v')\n",
    "badcm = CoherenceModel(model=badLdaModel, texts=texts, dictionary=dictionary, coherence='c_v')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline parameters for C_V coherence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coherence_Measure(seg=<function s_one_set at 0x11e3217d0>, prob=<function p_boolean_sliding_window at 0x11e338938>, conf=<function cosine_similarity at 0x11e338b90>, aggr=<function arithmetic_mean at 0x11e33d230>)\n"
     ]
    }
   ],
   "source": [
    "print(goodcm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print coherence values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.379532110157\n",
      "0.385963126348\n"
     ]
    }
   ],
   "source": [
    "print(goodcm.get_coherence())\n",
    "print(badcm.get_coherence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support for wrappers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This API supports gensim's _ldavowpalwabbit_ and _ldamallet_ wrappers as input parameter to `model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Replace with path to your Vowpal Wabbit installation\n",
    "vw_path = '/usr/local/bin/vw'\n",
    "\n",
    "# Replace with path to your Mallet installation\n",
    "home = os.path.expanduser('~')\n",
    "mallet_path = os.path.join(home, 'mallet-2.0.8', 'bin', 'mallet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model1 = LdaVowpalWabbit(vw_path, corpus=corpus, num_topics=2, id2word=dictionary, passes=50)\n",
    "model2 = LdaVowpalWabbit(vw_path, corpus=corpus, num_topics=2, id2word=dictionary, passes=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-13.226132904\n",
      "-14.3236789858\n"
     ]
    }
   ],
   "source": [
    "cm1 = CoherenceModel(model=model1, corpus=corpus, coherence='u_mass')\n",
    "cm2 = CoherenceModel(model=model2, corpus=corpus, coherence='u_mass')\n",
    "print(cm1.get_coherence())\n",
    "print(cm2.get_coherence())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model1 = LdaMallet(mallet_path, corpus=corpus, num_topics=2, id2word=dictionary, iterations=50)\n",
    "model2 = LdaMallet(mallet_path, corpus=corpus, num_topics=2, id2word=dictionary, iterations=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.37605697523\n",
      "0.393714418809\n"
     ]
    }
   ],
   "source": [
    "cm1 = CoherenceModel(model=model1, texts=texts, coherence='c_v')\n",
    "cm2 = CoherenceModel(model=model2, texts=texts, coherence='c_v')\n",
    "print(cm1.get_coherence())\n",
    "print(cm2.get_coherence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support for other topic models\n",
    "The gensim topics coherence pipeline can be used with other topics models too. Only the tokenized `topics` should be made available for the pipeline. Eg. with the gensim HDP model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hm = HdpModel(corpus=corpus, id2word=dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[u'minors',\n",
       "  u'user',\n",
       "  u'interface',\n",
       "  u'system',\n",
       "  u'survey',\n",
       "  u'response',\n",
       "  u'trees',\n",
       "  u'computer',\n",
       "  u'human',\n",
       "  u'time',\n",
       "  u'graph',\n",
       "  u'eps'],\n",
       " [u'response',\n",
       "  u'trees',\n",
       "  u'human',\n",
       "  u'graph',\n",
       "  u'user',\n",
       "  u'computer',\n",
       "  u'interface',\n",
       "  u'eps',\n",
       "  u'survey',\n",
       "  u'system',\n",
       "  u'minors',\n",
       "  u'time']]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the topic words from the model\n",
    "topics = []\n",
    "for topic_id, topic in hm.show_topics(num_topics=10, formatted=False):\n",
    "    topic = [word for word, _ in topic]\n",
    "    topics.append(topic)\n",
    "topics[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-14.611179327706207"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize CoherenceModel using `topics` parameter\n",
    "cm = CoherenceModel(topics=topics, corpus=corpus, dictionary=dictionary, coherence='u_mass')\n",
    "cm.get_coherence()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence as we can see, the `u_mass` and `c_v` coherence for the good LDA model is much more (better) than that for the bad LDA model. This is because, simply, the good LDA model usually comes up with better topics that are more human interpretable. The badLdaModel however fails to decipher between these two topics and comes up with topics which are not clear to a human. The `u_mass` and `c_v` topic coherences capture this wonderfully by giving the interpretability of these topics a number as we can see above. Hence this coherence measure can be used to compare difference topic models based on their human-interpretability."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}