Term-Document Matrix from Elasticsearch.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:f1297bde3442356ce30e7a40ce708b4346d807dab1880a25910183d17460d9f5"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Creating Term-Document Matrix from Elasticsearch"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "####Tutorial by Animesh Pandey, email: apanimesh061@gmail.com\n",
      "***\n",
      "There have been many questions about how can we create a Term-Document matrix from an ElasticSearch index. I tried out a possible way which I think should work well. I will be starting from scratch and this whole method is divided into three parts\n",
      "\n",
      "1. Initialising Index\n",
      "2. Indexing documents\n",
      "3. Querying for Term-Vectors\n",
      "4. Creating a Term-Document Matrix\n",
      "\n",
      "####Libraries used:\n",
      "\n",
      "1. Python Version: 2.7.4 with Numpy/Scipy\n",
      "2. ElasticSearch Version: 1.6.0\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Initialising Index"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Since, a few of my the python files that I was importing were in a local directory, I had to import the directory in to the Python path using:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import sys\n",
      "sys.path.insert(0, 'C:\\Users\\Animesh\\Documents\\Presentation_07_07')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "***\n",
      "Following is the content of the file _constants.py_ which contains the constants that will be freqentry used in the whole application.\n",
      "Here `ES_CLIENT` is the client of the elasticsearch that I'll be connecting to. `INDEX_NAME`, `TYPE_NAME` etc. are the constants defining the name of the index, document type etc. I have two different type of analyzers which also have been initialzed here."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# This is constants.py\n",
      "from elasticsearch import Elasticsearch\n",
      "\n",
      "ES_HOST = {\"host\" : \"localhost\", \"port\" : 9200}\n",
      "INDEX_NAME = 'social_media'\n",
      "TYPE_NAME = 'tweet'\n",
      "ANALYZER_NAME = 'my_english'\n",
      "ANALYZER_NAME_SHINGLE = \"my_english_shingle\"\n",
      "ES_CLIENT = Elasticsearch(hosts = [ES_HOST], timeout = 180)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "***\n",
      "Now, we will initialize the index. Notice that I have imported the _constants.py_ file to access the required values while creating the index. The function `init_index()` has two parts.\n",
      "1. Settings: This is the set up of the index i.e. configuration of the index. `constants.ES_CLIENT.indices.create` helps in creating an index with soe settings.\n",
      "2. Mapping: `press_mapping` is like the schema of the index. This defines that how the document will look like. Think of this thing like the SQL's `CREATE TABLE ...` where you specify which column will hold what type of data. Purpose of `\"dynamic\": \"strict\"` is raise an error if a document does not comply with the format specified in mapping. `constants.ES_CLIENT.indices.put_mapping` helps us add the mapping to the index."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from elasticsearch import Elasticsearch\n",
      "import constants\n",
      "\n",
      "def init_index():\n",
      "    constants.ES_CLIENT.indices.create(\n",
      "        index = constants.INDEX_NAME,\n",
      "        body = {\n",
      "            \"settings\": {\n",
      "                \"index\": {\n",
      "                    \"number_of_shards\": 3,\n",
      "                    \"number_of_replicas\": 0\n",
      "                },\n",
      "                \"analysis\": {\n",
      "                    \"analyzer\": {\n",
      "                        constants.ANALYZER_NAME: {\n",
      "                            \"type\": \"custom\",\n",
      "                            \"tokenizer\": \"standard\",\n",
      "                            \"filter\": [\n",
      "                                \"lowercase\",\n",
      "                                \"asciifolding\",\n",
      "                                \"cust_stop\",\n",
      "                                \"my_snow\"\n",
      "                            ]\n",
      "                        },\n",
      "                        constants.ANALYZER_NAME_SHINGLE: {\n",
      "                            \"type\": \"custom\",\n",
      "                            \"tokenizer\": \"standard\",\n",
      "                            \"filter\": [\n",
      "                                \"lowercase\",\n",
      "                                \"asciifolding\",\n",
      "                                \"cust_stop\",\n",
      "                                \"my_snow\",\n",
      "                                \"shingle_filter\"\n",
      "                            ]\n",
      "                        }\n",
      "                    },\n",
      "                    \"filter\": {\n",
      "                        \"cust_stop\": {\n",
      "                            \"type\": \"stop\",\n",
      "                            \"stopwords_path\": \"stoplist.txt\",\n",
      "                        },\n",
      "                        \"shingle_filter\" : {\n",
      "                            \"type\" : \"shingle\",\n",
      "                            \"min_shingle_size\" : 2,\n",
      "                            \"max_shingle_size\" : 2,\n",
      "                            \"output_unigrams\": True\n",
      "                        },\n",
      "                        \"my_snow\" : {\n",
      "                            \"type\" : \"snowball\",\n",
      "                            \"language\" : \"English\"\n",
      "                        }\n",
      "                    }\n",
      "                }\n",
      "            }\n",
      "        }\n",
      "    )\n",
      "\n",
      "    press_mapping = {\n",
      "        constants.TYPE_NAME: {\n",
      "            \"dynamic\": \"strict\",\n",
      "            \"properties\": {\n",
      "                \"_id\": {\n",
      "                    \"type\": \"string\",\n",
      "                    \"store\": True,\n",
      "                    \"index\": \"not_analyzed\"\n",
      "                },\n",
      "                \"text\": {\n",
      "                    \"type\": \"multi_field\",\n",
      "                    \"fields\": {\n",
      "                        \"text\": {\n",
      "                            \"include_in_all\": False,\n",
      "                            \"type\": \"string\",\n",
      "                            \"store\": False,\n",
      "                            \"index\": \"not_analyzed\"\n",
      "                        },\n",
      "                        \"_analyzed\": {\n",
      "                            \"type\": \"string\",\n",
      "                            \"store\": True,\n",
      "                            \"index\": \"analyzed\",\n",
      "                            \"term_vector\": \"with_positions_offsets\",\n",
      "                            \"analyzer\": constants.ANALYZER_NAME\n",
      "                        },\n",
      "                        \"_analyzed_shingles\": {\n",
      "                            \"type\": \"string\",\n",
      "                            \"store\": True,\n",
      "                            \"index\": \"analyzed\",\n",
      "                            \"term_vector\": \"with_positions_offsets\",\n",
      "                            \"analyzer\": constants.ANALYZER_NAME_SHINGLE\n",
      "                        }\n",
      "                    }\n",
      "                }\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "\n",
      "    constants.ES_CLIENT.indices.put_mapping (\n",
      "        index = constants.INDEX_NAME,\n",
      "        doc_type = constants.TYPE_NAME,\n",
      "        body = press_mapping\n",
      "    )\n",
      "\n",
      "    print \"The index has been initialised!\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now if you send a CURL command `curl -X GET \"http://localhost:9200/social_media\"?pretty=true` you will get the same `setings` and `mappings` as the response from ElasticSearch."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Indexing Documents"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now that we have our index, we'll have to start indexing data into the index. Tweets are the best thing if you want something short and simple. For getting the tweets I used the `TwitterAPI`. `constants` is the constants file and `index_tweets` is the file where we had `init_index()`. I will talk about `util.py` later."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from TwitterAPI import TwitterAPI\n",
      "from elasticsearch import Elasticsearch\n",
      "import constants, index_tweets, util\n",
      "import cPickle"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Using the `TwitterAPI` I will retireve the top 10 tweets under the handle name `elastic`. For interacting with Twitter API you need the authentication tokens, whic you can get from Twitter's [Developer Portal](https://dev.twitter.com/)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "SCREEN_NAME = 'elastic'\n",
      "\n",
      "CONSUMER_KEY = ''\n",
      "CONSUMER_SECRET = ''\n",
      "ACCESS_TOKEN_KEY = ''\n",
      "ACCESS_TOKEN_SECRET = ''"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "***\n",
      "Here, I initialize the index `INDEX_NAME` and I send a request to the API to give me the tweets. The response JSON has a lot more than just text. You can view the whole respose skeleton [here](https://dev.twitter.com/rest/reference/get/statuses/user_timeline).\n",
      "\n",
      "`TEMP_DOC` is the document that I will index. This document should comply with the schema that I have mentioned in the mapping of the index, otherwise it will raise an error. `constants.ES_CLIENT.index` is used to index the tweet in the index `INDEX_NAME`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "api = TwitterAPI(CONSUMER_KEY,\n",
      "                 CONSUMER_SECRET,\n",
      "                 ACCESS_TOKEN_KEY,\n",
      "                 ACCESS_TOKEN_SECRET)\n",
      "\n",
      "r = api.request('statuses/user_timeline',\n",
      "                {\n",
      "                    'screen_name': SCREEN_NAME,\n",
      "                    'count': 10\n",
      "                })\n",
      "\n",
      "doc_index = 0\n",
      "doc_index_map = dict()\n",
      "\n",
      "# initialise the index\n",
      "index_tweets.init_index()\n",
      "\n",
      "for item in r:\n",
      "    TEMP_DOC = { \"text\": item['text'] }\n",
      "    try:\n",
      "        constants.ES_CLIENT.index(\n",
      "                index=constants.INDEX_NAME,\n",
      "                doc_type=constants.TYPE_NAME,\n",
      "                body=TEMP_DOC,\n",
      "                id=doc_index\n",
      "        )\n",
      "        print \"Successfully indexed Tweet {tweet} with id {tweet_index}\".format(tweet=item['id'], tweet_index=doc_index)\n",
      "        doc_index_map.update({item['id'] : doc_index})\n",
      "        doc_index += 1\n",
      "    except Exception as e:\n",
      "        print e\n",
      "        print \"Error for Tweet {tweet}\".format(tweet=item['id'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "The index has been initialised!\n",
        "Successfully indexed Tweet 629241463797366784 with id 0"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Successfully indexed Tweet 629216039339978752 with id 1"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Successfully indexed Tweet 629060379155431424 with id 2\n",
        "Successfully indexed Tweet 628949758166470656 with id 3\n",
        "Successfully indexed Tweet 628614404196696065 with id 4"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "Successfully indexed Tweet 628601038690480128 with id 5\n",
        "Successfully indexed Tweet 628591924212051968 with id 6\n",
        "Successfully indexed Tweet 628578918396641280 with id 7\n",
        "Successfully indexed Tweet 628267050927063040 with id 8\n",
        "Successfully indexed Tweet 628255353168228352 with id 9\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "E:\\Python27\\Lib\\site-packages\\requests\\packages\\urllib3\\util\\ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.\n",
        "  InsecurePlatformWarning\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "***\n",
      "Now we come to a very important part. If you look at the previous code, you'll notice `doc_index_map` which is actually a hash/dictionary that maps the orignal identifier of the tweet to an index represented by a number e.g. now tweet _629060379155431424_ will now be identified as _2_. This comes handy when you are dealing with creation of matrices which we'll talk about in a while.\n",
      "\n",
      "The following are two functions from the `util.py` which save dictionaries to the hard-disk for future use. `doc_index_map.pkl` is the dictionary of the documnent identifier to a number. `vocab.pkl` is a similar type of hash but terms. i.e. this holds the complete vocabulary/unique terms in the 10 document index."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "util.createPickleFromDict(doc_index_map, \"doc_index_map.pkl\")\n",
      "util.saveVocabulary()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Pickled doc_index_map.pkl\n",
        "Pickled vocab.pkl"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Querying for Term-Vectors"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here we'll talk about the `util.py` file and its importance in the code. This file holds the functions that are required frequently by the application like reading the index, getting the term-vectors of a document, serialising of dictionaries, creation of the vocabulary etc."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import constants\n",
      "from elasticsearch import Elasticsearch\n",
      "import cPickle\n",
      "\n",
      "TERM_INDEX = 0\n",
      "VOCAB = dict()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`loadPickle` loads the serialized pickle file where `name` of the path to the pickle file."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def loadPickle(name):\n",
      "    print \"Loading \" + name\n",
      "    with open(name, 'rb') as f:\n",
      "        return cPickle.load(f)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`createPickleFromDict` serializes a dictionary `d` to the `filename` specified."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def createPickleFromDict(d,  filename):\n",
      "    cPickle.dump(d,  open(filename,  \"wb\"))\n",
      "    print 'Pickled ' + filename"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`getTermVector` returns the term_vector of the documents having document id `doc_id`. We mainly require the term-frequencies of the terms mapped to their original term. We return this mapping(`temp`) along with the document's id(`a[\"_id\"]`)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def getTermVector(doc_id):\n",
      "    temp = dict()\n",
      "    a = constants.ES_CLIENT.termvector(index = constants.INDEX_NAME,\n",
      "                                    doc_type = constants.TYPE_NAME,\n",
      "                                    id = doc_id,\n",
      "                                    field_statistics = True,\n",
      "                                    fields = ['text._analyzed'],\n",
      "                                    term_statistics = True\n",
      "                                )\n",
      "    curr_termvec = a[\"term_vectors\"][\"text._analyzed\"][\"terms\"]\n",
      "    tokens = curr_termvec.keys()\n",
      "    [temp.update({token : {\"tf\": curr_termvec[token][\"term_freq\"]}}) for token in tokens]\n",
      "    return a[\"_id\"], temp"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`scrollIndex` scans through the index without loads all results into memory. So, with every document we get, we get its term_vector using `getTermVector`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def scrollIndex():\n",
      "    ALL_QUERY = {\"query\": {\"match_all\": {}}}\n",
      "\n",
      "    rs = constants.ES_CLIENT.search(\n",
      "                index=constants.INDEX_NAME,\n",
      "                scroll='60s',\n",
      "                size=10,\n",
      "                body=ALL_QUERY\n",
      "            )\n",
      "\n",
      "    data = rs['hits']['hits']\n",
      "    for doc in data:\n",
      "        curr_doc, curr_term_vector = getTermVector(doc[\"_id\"])\n",
      "        yield curr_doc, curr_term_vector\n",
      "\n",
      "    scroll_size = rs['hits']['total']\n",
      "\n",
      "    while scroll_size:\n",
      "        try:\n",
      "            scroll_id = rs['_scroll_id']\n",
      "            rs = constants.ES_CLIENT.scroll(scroll_id=scroll_id, scroll='60s')\n",
      "            data = rs['hits']['hits']\n",
      "            for doc in data:\n",
      "                curr_doc, curr_term_vector = getTermVector(doc[\"_id\"])\n",
      "                yield curr_doc, curr_term_vector\n",
      "\n",
      "            scroll_size = len(rs['hits']['hits'])\n",
      "        except Exception as e:\n",
      "            print e"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`saveVocabulary` makes a pass through the whole index and creates a vocabulary (list of the unique terms) from that index and finally serializes it to the hard-disk as `vocab.pkl`. This explains _In [8]_ entry above."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def getVocabulary(term_vector):\n",
      "    global VOCAB\n",
      "    global TERM_INDEX\n",
      "    for term in term_vector:\n",
      "        if term not in VOCAB.keys():\n",
      "            VOCAB.update({term : TERM_INDEX})\n",
      "            TERM_INDEX += 1\n",
      "\n",
      "def saveVocabulary():\n",
      "    for _, vector in scrollIndex():\n",
      "        getVocabulary(vector.keys())\n",
      "    createPickleFromDict(VOCAB, \"vocab.pkl\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Creating a Term-Document Matrix"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "At this point we have all the documents and dictionaries that we require. We'll now start with how the interpret what ever we have to a Matrix."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import util\n",
      "import numpy as np\n",
      "from scipy.sparse import csr_matrix"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let us now load the dctionaries that we created. If you have a lot of documents I suggest using MongoDB for storing the mappings."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "term_map = util.loadPickle(\"vocab.pkl\")\n",
      "doc_map = util.loadPickle(\"doc_index_map.pkl\")\n",
      "\n",
      "no_of_terms = len(term_map)\n",
      "no_of_docs = len(doc_map)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Loading vocab.pkl\n",
        "Loading doc_index_map.pkl\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I won't be going in detail about what a Sparse representation of a matrix is. The outputs that we get from ElasticSearch can easliy be translated into a Compressed Row Sparse Format. You can read about this represetation from internet. Using `ROW_OFFSETS`, `COLUMN_INDICES` and `VALUES` you can recreate the whole rectaungular Term-Document Matrix."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Creating a Compressed Row Sparse Format of the to be made Term-Document Matrix\n",
      "ROW_OFFSETS = [0]\n",
      "COLUMN_INDICES = []\n",
      "VALUES = []\n",
      "for doc, vector in util.scrollIndex():\n",
      "    prev_offset = ROW_OFFSETS[-1]\n",
      "    ROW_OFFSETS.append(prev_offset + len(vector))\n",
      "    [(COLUMN_INDICES.append(term_map[term]), VALUES.append(count['tf'])) for (term, count) in vector.iteritems()]\n",
      "\n",
      "print \"The Compressed Sparse Row format:\"\n",
      "print\n",
      "print \"The row offsets:\", ROW_OFFSETS # There can be takes as the Document IDs\n",
      "print\n",
      "print \"The index of the term as per the vocab.pkl:\"\n",
      "print COLUMN_INDICES\n",
      "print\n",
      "print \"The term-frequencies of the terms in COLUMN_INDICES on the same indices\"\n",
      "print VALUES"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "The Compressed Sparse Row format:\n",
        "\n",
        "The row offsets: [0, 16, 34, 47, 60, 70, 86, 98, 113, 125, 141]\n",
        "\n",
        "The index of the term as per the vocab.pkl:\n",
        "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 7, 27, 28, 29, 30, 31, 11, 32, 18, 33, 34, 23, 25, 35, 36, 11, 28, 37, 38, 39, 40, 41, 18, 42, 43, 44, 11, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 18, 55, 7, 56, 57, 11, 58, 59, 60, 18, 19, 61, 62, 38, 32, 23, 63, 11, 21, 20, 25, 64, 17, 65, 66, 67, 68, 69, 7, 9, 70, 71, 72, 11, 73, 20, 19, 74, 75, 76, 77, 18, 78, 7, 21, 79, 80, 81, 11, 82, 83, 84, 85, 18, 86, 87, 11, 88, 7, 89, 90, 17, 91, 92, 93, 19, 94, 61, 18, 64, 95, 7, 21, 20, 96, 97, 11]\n",
        "\n",
        "The term-frequencies of the terms in COLUMN_INDICES on the same indices\n",
        "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 2, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I have not used `termVectorFromCSR` anywhere but this function gives you the term_vector for each document where `doc_id` is the document ID, `indices[start : end]` is the list storing the term IDs and `data[start : end]` stores the term-frequencies of the terms in `indices[start : end]`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def termVectorFromCSR(row_offsets, indices, data):\n",
      "    offsets = zip(row_offsets[::], row_offsets[1::])\n",
      "    doc_id = 0\n",
      "    for (start, end) in offsets:\n",
      "        yield doc_id, indices[start : end], data[start : end]\n",
      "        doc_id += 1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Finally, we have the Sparse Matrix and now we will utilise _Numpy/Scipy_ libraries to create a Dense Matrix out of the Sparse representation. The Rows are documents and Columns are the terms. The values are the term-frequencies. Now we can do whatever we want to do with `tdm` i.e. the term-document matrix."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Creating a Dense (Term-Document) Matrix using scipy and numpy\n",
      "indptr = np.asarray(ROW_OFFSETS)\n",
      "indices = np.asarray(COLUMN_INDICES)\n",
      "data = np.asarray(VALUES)\n",
      "tdm = csr_matrix((data, indices, indptr), shape=(no_of_docs, no_of_terms)).toarray()\n",
      "\n",
      "print tdm # This is the Dense Version of the Sparse Matrix from ElasticSearch"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 1 0 0 0 1 1 1 1 1\n",
        "  1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 2 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 1 1 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0\n",
        "  0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 1 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1\n",
        "  1 1 2 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 1 2 1 1 1 1 1 1 1 0 0 0 0 0 0 0]\n",
        " [0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1]]\n"
       ]
      }
     ],
     "prompt_number": 12
    }
   ],
   "metadata": {}
  }
 ]
}