doc/source/bow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with the Bag-of-Words representation\n",
    "\n",
    "The [bow module](api.rst#tmtoolkit-bow) in tmtoolkit contains several functions for working with Bag-of-Words (BoW) representations of documents. It's divided into two sub-modules: [bow.bow_stats](api.rst#module-tmtoolkit.bow.bow_stats) and [bow.dtm](api.rst#module-tmtoolkit.bow.dtm). The former implements several statistics and transformations for BoW representations, the latter contains functions to create and convert sparse or dense document-term matrices (DTMs).\n",
    "\n",
    "Most of the functions in both sub-modules accept and/or return sparse DTMs. The [previous chapter](preprocessing.ipynb) contained a section about what sparse DTMs are and [how they can be generated with tmtoolkit](preprocessing.ipynb#Generating-a-sparse-document-term-matrix-(DTM))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example document-term matrix\n",
    "\n",
    "Before we start with the [bow.dtm](api.rst#module-tmtoolkit.bow.dtm) module, we will generate a sparse DTM from a small example corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type='text/css'>\n",
       ".datatable table.frame { margin-bottom: 0; }\n",
       ".datatable table.frame thead { border-bottom: none; }\n",
       ".datatable table.frame tr.coltypes td {  color: #FFFFFF;  line-height: 6px;  padding: 0 0.5em;}\n",
       ".datatable .boolean { background: #DDDD99; }\n",
       ".datatable .object  { background: #565656; }\n",
       ".datatable .integer { background: #5D9E5D; }\n",
       ".datatable .float   { background: #4040CC; }\n",
       ".datatable .string  { background: #CC4040; }\n",
       ".datatable .row_index {  background: var(--jp-border-color3);  border-right: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  font-size: 9px;}\n",
       ".datatable .frame tr.coltypes .row_index {  background: var(--jp-border-color0);}\n",
       ".datatable th:nth-child(2) { padding-left: 12px; }\n",
       ".datatable .hellipsis {  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .vellipsis {  background: var(--jp-layout-color0);  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .na {  color: var(--jp-cell-editor-border-color);  font-size: 80%;}\n",
       ".datatable .footer { font-size: 9px; }\n",
       ".datatable .frame_dimensions {  background: var(--jp-border-color3);  border-top: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  display: inline-block;  opacity: 0.6;  padding: 1px 10px 1px 5px;}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "random.seed(20191113)   # to make the sampling reproducible\n",
    "\n",
    "import numpy as np\n",
    "np.set_printoptions(precision=5)\n",
    "\n",
    "from tmtoolkit.corpus import Corpus\n",
    "\n",
    "corpus = Corpus.from_builtin_corpus('en-NewsArticles').sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at a sample document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Merkel: 'Only if Europe is doing well, will Germany be doing well'\n",
      "\n",
      "Ahead of meeting her fellow European leaders at a summit in Brussels, German Chancellor Angela Merkel has reiterated her government's call for unity in the EU.\n"
     ]
    }
   ],
   "source": [
    "print(corpus['NewsArticles-2058'][:227])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We employ a preprocessing pipeline that removes a lot of information from our original data in order to obtain a very condensed DTM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='datatable'>\n",
       "  <table class='frame'>\n",
       "  <thead>\n",
       "    <tr class='colnames'><td class='row_index'></td><th>doc</th><th>position</th><th>token</th><th>lemma</th><th>pos</th><th>whitespace</th></tr>\n",
       "    <tr class='coltypes'><td class='row_index'></td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int64'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='boolean' title='bool8'>&#x25AA;</td></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><td class='row_index'>0</td><td>NewsArticles-119</td><td>0</td><td>day</td><td>day</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>1</td><td>NewsArticles-119</td><td>1</td><td>nhs</td><td>NHS</td><td>PROPN</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>2</td><td>NewsArticles-119</td><td>2</td><td>day</td><td>day</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>3</td><td>NewsArticles-119</td><td>3</td><td>nhs</td><td>NHS</td><td>PROPN</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>4</td><td>NewsArticles-119</td><td>4</td><td>pledge</td><td>pledge</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>5</td><td>NewsArticles-119</td><td>5</td><td>prime</td><td>Prime</td><td>PROPN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>6</td><td>NewsArticles-119</td><td>6</td><td>minister</td><td>Minister</td><td>PROPN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>7</td><td>NewsArticles-119</td><td>7</td><td>david</td><td>David</td><td>PROPN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>8</td><td>NewsArticles-119</td><td>8</td><td>cameron</td><td>Cameron</td><td>PROPN</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>9</td><td>NewsArticles-119</td><td>9</td><td>theresa</td><td>Theresa</td><td>PROPN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>10</td><td>NewsArticles-119</td><td>10</td><td>may</td><td>May</td><td>PROPN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>11</td><td>NewsArticles-119</td><td>11</td><td>government</td><td>government</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>12</td><td>NewsArticles-119</td><td>12</td><td>people</td><td>people</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>13</td><td>NewsArticles-119</td><td>13</td><td>access</td><td>access</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>14</td><td>NewsArticles-119</td><td>14</td><td>gps</td><td>gps</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>&#x22EE;</td><td class='hellipsis'>&#x22EE;</td><td class='hellipsis'>&#x22EE;</td><td class='hellipsis'>&#x22EE;</td><td class='hellipsis'>&#x22EE;</td><td class='hellipsis'>&#x22EE;</td><td class='hellipsis'>&#x22EE;</td></tr>\n",
       "    <tr><td class='row_index'>925</td><td>NewsArticles-3665</td><td>351</td><td>article</td><td>article</td><td>NOUN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>926</td><td>NewsArticles-3665</td><td>352</td><td>author</td><td>author</td><td>NOUN</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>927</td><td>NewsArticles-3665</td><td>353</td><td>al</td><td>Al</td><td>PROPN</td><td>1</td></tr>\n",
       "    <tr><td class='row_index'>928</td><td>NewsArticles-3665</td><td>354</td><td>jazeera</td><td>Jazeera</td><td>PROPN</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>929</td><td>NewsArticles-3665</td><td>355</td><td>policy</td><td>policy.-</td><td>NOUN</td><td>0</td></tr>\n",
       "  </tbody>\n",
       "  </table>\n",
       "  <div class='footer'>\n",
       "    <div class='frame_dimensions'>930 rows &times; 6 columns</div>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<Frame [930 rows x 6 cols]>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.preprocess import TMPreproc\n",
    "\n",
    "preproc = TMPreproc(corpus, language='en')\n",
    "preproc.pos_tag() \\\n",
    "    .lemmatize() \\\n",
    "    .filter_for_pos('N') \\\n",
    "    .tokens_to_lowercase() \\\n",
    "    .remove_special_chars_in_tokens() \\\n",
    "    .clean_tokens(remove_shorter_than=2) \\\n",
    "    .remove_common_tokens(5, absolute=True) # remove tokens that occur in all documents\n",
    "preproc.tokens_datatable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 514)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc.n_docs, preproc.vocabulary_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We fetch the document labels and vocabulary and convert them to NumPy arrays, because such arrays allow advanced indexing methods such as boolean indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['NewsArticles-119', 'NewsArticles-1206', 'NewsArticles-2058',\n",
       "       'NewsArticles-3016', 'NewsArticles-3665'], dtype='<U17')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_labels = np.array(preproc.doc_labels)\n",
    "doc_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['70', 'abuse', 'access', 'accession', 'accusation', 'act',\n",
       "       'addition', 'address', 'administration', 'affiliation'],\n",
       "      dtype='<U16')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = np.array(preproc.vocabulary)\n",
    "vocab[:10]  # only showing the first 10 tokens here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we fetch the sparse DTM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<5x514 sparse matrix of type '<class 'numpy.int32'>'\n",
       "\twith 578 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtm = preproc.dtm\n",
    "dtm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a sparse DTM `dtm`, an array of document labels `doc_labels` that represent the rows of the DTM and an array of vocabulary tokens `vocab` that represent the columns of the DTM. We will use this data for the remainder of the chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `bow.dtm` module\n",
    "\n",
    "This module is quite small. There are two functions to convert a DTM to a [datatable](https://github.com/h2oai/datatable/) or [DataFrame](https://pandas.pydata.org/): [dtm_to_datatable()](api.rst#tmtoolkit.bow.dtm.dtm_to_datatable) and [dtm_to_dataframe()](api.rst#tmtoolkit.bow.dtm.dtm_to_dataframe). Note that the generated datatable or DataFrame is *dense*, i.e. it uses up (much) more memory than the input DTM.\n",
    "\n",
    "Let's generate a datatable via [dtm_to_datatable()](api.rst#tmtoolkit.bow.dtm.dtm_to_datatable) from our DTM, the document labels and the vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='datatable'>\n",
       "  <table class='frame'>\n",
       "  <thead>\n",
       "    <tr class='colnames'><td class='row_index'></td><th>_doc</th><th>70</th><th>abuse</th><th>access</th><th>accession</th><th>accusation</th><th>act</th><th>addition</th><th>address</th><th>administration</th><th class='vellipsis'>&hellip;</th><th>world</th><th>wound</th><th>year</th><th>york</th><th>yucel</th></tr>\n",
       "    <tr class='coltypes'><td class='row_index'></td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td></td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><td class='row_index'>0</td><td>NewsArticles-119</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td class=vellipsis>&hellip;</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>1</td><td>NewsArticles-1206</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td class=vellipsis>&hellip;</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>2</td><td>NewsArticles-2058</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td class=vellipsis>&hellip;</td><td>0</td><td>0</td><td>2</td><td>0</td><td>2</td></tr>\n",
       "    <tr><td class='row_index'>3</td><td>NewsArticles-3016</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td class=vellipsis>&hellip;</td><td>3</td><td>1</td><td>0</td><td>1</td><td>0</td></tr>\n",
       "    <tr><td class='row_index'>4</td><td>NewsArticles-3665</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td class=vellipsis>&hellip;</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr>\n",
       "  </tbody>\n",
       "  </table>\n",
       "  <div class='footer'>\n",
       "    <div class='frame_dimensions'>5 rows &times; 515 columns</div>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<Frame [5 rows x 515 cols]>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.dtm import dtm_to_datatable\n",
    "\n",
    "dtm_to_datatable(dtm, doc_labels, vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that a row `_doc` with the document labels was created and that the vocabulary tokens become the column names. [dtm_to_dataframe()](api.rst#tmtoolkit.bow.dtm.dtm_to_dataframe) works the same way.\n",
    "\n",
    "You can combine tmtoolkit with [Gensim](https://radimrehurek.com/gensim/). The `bow.dtm` module provides several functions to convert data between both packages:\n",
    "\n",
    "- [dtm_and_vocab_to_gensim_corpus_and_dict()](api.rst#tmtoolkit.bow.dtm.dtm_and_vocab_to_gensim_corpus_and_dict): converts a (sparse) DTM and a vocabulary list to a *Gensim Corpus* and *Gensim Dictionary*\n",
    "- [dtm_to_gensim_corpus()](api.rst#tmtoolkit.bow.dtm.dtm_to_gensim_corpus): convert a (sparse) DTM only to a *Gensim Corpus*\n",
    "- [gensim_corpus_to_dtm()](api.rst#tmtoolkit.bow.dtm.gensim_corpus_to_dtm): converts a *Gensim Corpus* object to a sparse DTM in COO format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `bow.bow_stats` module\n",
    "\n",
    "This module provides several statistics and transformations for sparse or dense DTMs.\n",
    "\n",
    "### Document lengths, document and term frequencies, token co-occurrences\n",
    "\n",
    "Let's start with the [doc_lengths()](api.rst#tmtoolkit.bow.bow_stats.doc_lengths) function, which simply gives the number of tokens per document (i.e. the row-wise sum of the DTM):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 38,  40, 336, 160, 356])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import doc_lengths\n",
    "\n",
    "doc_lengths(dtm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned array is aligned to the document labels `doc_labels` so we can see that the last document, \"NewsArticles-3665\", is the one with the most tokens. Or to do it computationally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'NewsArticles-3665'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_labels[doc_lengths(dtm).argmax()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While [doc_lengths()](api.rst#tmtoolkit.bow.bow_stats.doc_lengths) gives the row-wise sum across the DTM, [term_frequencies()](api.rst#tmtoolkit.bow.bow_stats.term_frequencies) gives the column-wise sum. This means it returns an array of the length of the vocabulary's size where each entry in that array reflects the number of occurrences of the respective vocabulary token (aka term).\n",
    "\n",
    "Let's calculate that measure, get its maximum and the vocabulary token(s) for that maximum value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21, array(['medium'], dtype='<U16'))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import term_frequencies\n",
    "\n",
    "term_freq = term_frequencies(dtm)\n",
    "(term_freq.max(), vocab[term_freq == term_freq.max()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's also possible to calculate the proportional frequency, i.e. normalize the counts by the overall number of tokens via `proportions=True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['candidate', 'eu', 'macron', 'medium', 'merkel', 'refugee'],\n",
       "      dtype='<U16')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "term_prop = term_frequencies(dtm, proportions=True)\n",
    "vocab[term_prop >= 0.01]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function [doc_frequencies()](api.rst#tmtoolkit.bow.bow_stats.doc_frequencies) returns how often each token in the vocabulary occurs at least *n* times per document. You can control *n* per parameter `min_val` which is set to `1` by default. The returned array is aligned with the vocabulary. Here, we calculate the document frequency with the default value `min_val=1`, extract the maximum document frequency and see which of the tokens in the `vocab` array reach the maximum document frequency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, array(['minister'], dtype='<U16'))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import doc_frequencies\n",
    "\n",
    "df = doc_frequencies(dtm)\n",
    "max_df = df.max()\n",
    "max_df, vocab[df == max_df]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It turns out that the maximum document frequency is 4 and only the token \"minister\" reaches that document frequency. This means only \"minister\" is mentioned across 4 documents at least once (because `min_val` is `1`). Remember that during preprocessing, we removed all tokens that occur across *all* five documents, hence there can't be a vocabulary token with a document frequency of 5.\n",
    "\n",
    "Let's see which vocabulary tokens occur within a single document at least 10 times:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['candidate', 'eu', 'macron', 'medium', 'merkel', 'refugee'],\n",
       "      dtype='<U16')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = doc_frequencies(dtm, min_val=10)\n",
    "vocab[df > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also calculate the *co-document frequency* or *token co-occurrence* matrix via [codoc_frequencies()](api.rst#tmtoolkit.bow.bow_stats.codoc_frequencies). This measures how often each pair of vocabulary tokens occurs at least *n* times together in the same document. Again, you can control *n* per parameter `min_val` which is set to `1` by default. The result is a sparse matrix of shape *vocabulary size* by *vocabulary size*. The columns and rows give the pairs of tokens from the vocabulary.\n",
    "\n",
    "Let's generate a co-document frequency matrix and convert it to a dense representation, because our further operations don't support sparse matrices.\n",
    "\n",
    "A co-document frequency matrix is symmetric along the diagonal, because co-occurrence between a pair `(token1, token2)` is always the same as between `(token2, token1)`. We want to filter out the duplicate pairs and for that use [np.triu()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.triu.html) to take only the upper triangle of the matrix, i.e. set all values in the lower triangle including the matrix diagonal to zero (`k=1` does this):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 1, 0],\n",
       "       [0, 0, 0, ..., 1, 0, 0],\n",
       "       ...,\n",
       "       [0, 0, 0, ..., 0, 0, 1],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import codoc_frequencies\n",
    "\n",
    "codoc_mat = codoc_frequencies(dtm).todense()\n",
    "codoc_upper = np.triu(codoc_mat, k=1)\n",
    "codoc_upper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a list that contains the pairs of tokens that occur together in at least two documents (`codoc_upper > 1`) together with their co-document frequency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('access', 'channel', 2),\n",
       " ('access', 'day', 2),\n",
       " ('access', 'minister', 2),\n",
       " ('access', 'news', 2),\n",
       " ('april', 'author', 2),\n",
       " ('april', 'co', 2),\n",
       " ('april', 'critic', 2),\n",
       " ('april', 'distribution', 2),\n",
       " ('april', 'heart', 2),\n",
       " ('april', 'law', 2)]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interesting_pairs = [(vocab[t1], vocab[t2], codoc_upper[t1, t2])\n",
    "                     for t1, t2 in zip(*np.where(codoc_upper > 1))]\n",
    "interesting_pairs[:10]  # showing only the first ten pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate sorted lists and datatables according to term frequency\n",
    "\n",
    "When working with DTMs, it's often helpful to rank terms per document according to their frequency. This is what [sorted_terms()](api.rst#tmtoolkit.bow.bow_stats.sorted_terms) does for you. It further allows to specify the sorting order (the default is descending order via `ascending=False`) and several limits:\n",
    "\n",
    "- `lo_thresh` for the minimum term frequency\n",
    "- `hi_thresh` for the maximum term frequency\n",
    "- `top_n` for the maximum number of terms per document\n",
    "\n",
    "Let's display the top three tokens per document by frequency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('day', 3), ('nhs', 2), ('bbc', 2)],\n",
       " [('car', 4), ('garda', 4), ('collision', 3)],\n",
       " [('merkel', 14), ('eu', 13), ('refugee', 13)],\n",
       " [('politic', 7), ('party', 6), ('farron', 5)],\n",
       " [('medium', 21), ('candidate', 19), ('macron', 15)]]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import sorted_terms\n",
    "\n",
    "sorted_terms(dtm, vocab, top_n=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output is a list for each document (this means the output is aligned with the document labels `doc_labels`), with three pairs of `(token, frequency)` each. It's also possible to get this data as datatable via [sorted_terms_datatable()](api.rst#tmtoolkit.bow.bow_stats.sorted_terms_datatable), which gives a better overview and also includes labels for the documents. It accepts the same parameters for sorting and limitting the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='datatable'>\n",
       "  <table class='frame'>\n",
       "  <thead>\n",
       "    <tr class='colnames'><td class='row_index'></td><th>doc</th><th>token</th><th>value</th></tr>\n",
       "    <tr class='coltypes'><td class='row_index'></td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><td class='row_index'>0</td><td>NewsArticles-119</td><td>day</td><td>3</td></tr>\n",
       "    <tr><td class='row_index'>1</td><td>NewsArticles-119</td><td>nhs</td><td>2</td></tr>\n",
       "    <tr><td class='row_index'>2</td><td>NewsArticles-119</td><td>bbc</td><td>2</td></tr>\n",
       "    <tr><td class='row_index'>3</td><td>NewsArticles-1206</td><td>car</td><td>4</td></tr>\n",
       "    <tr><td class='row_index'>4</td><td>NewsArticles-1206</td><td>garda</td><td>4</td></tr>\n",
       "    <tr><td class='row_index'>5</td><td>NewsArticles-1206</td><td>collision</td><td>3</td></tr>\n",
       "    <tr><td class='row_index'>6</td><td>NewsArticles-2058</td><td>merkel</td><td>14</td></tr>\n",
       "    <tr><td class='row_index'>7</td><td>NewsArticles-2058</td><td>eu</td><td>13</td></tr>\n",
       "    <tr><td class='row_index'>8</td><td>NewsArticles-2058</td><td>refugee</td><td>13</td></tr>\n",
       "    <tr><td class='row_index'>9</td><td>NewsArticles-3016</td><td>politic</td><td>7</td></tr>\n",
       "    <tr><td class='row_index'>10</td><td>NewsArticles-3016</td><td>party</td><td>6</td></tr>\n",
       "    <tr><td class='row_index'>11</td><td>NewsArticles-3016</td><td>farron</td><td>5</td></tr>\n",
       "    <tr><td class='row_index'>12</td><td>NewsArticles-3665</td><td>medium</td><td>21</td></tr>\n",
       "    <tr><td class='row_index'>13</td><td>NewsArticles-3665</td><td>candidate</td><td>19</td></tr>\n",
       "    <tr><td class='row_index'>14</td><td>NewsArticles-3665</td><td>macron</td><td>15</td></tr>\n",
       "  </tbody>\n",
       "  </table>\n",
       "  <div class='footer'>\n",
       "    <div class='frame_dimensions'>15 rows &times; 3 columns</div>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<Frame [15 rows x 3 cols]>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import sorted_terms_datatable\n",
    "\n",
    "sorted_terms_datatable(dtm, vocab, doc_labels, top_n=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='datatable'>\n",
       "  <table class='frame'>\n",
       "  <thead>\n",
       "    <tr class='colnames'><td class='row_index'></td><th>doc</th><th>token</th><th>value</th></tr>\n",
       "    <tr class='coltypes'><td class='row_index'></td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='integer' title='int32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><td class='row_index'>0</td><td>NewsArticles-2058</td><td>merkel</td><td>14</td></tr>\n",
       "    <tr><td class='row_index'>1</td><td>NewsArticles-2058</td><td>refugee</td><td>13</td></tr>\n",
       "    <tr><td class='row_index'>2</td><td>NewsArticles-2058</td><td>eu</td><td>13</td></tr>\n",
       "    <tr><td class='row_index'>3</td><td>NewsArticles-2058</td><td>germany</td><td>8</td></tr>\n",
       "    <tr><td class='row_index'>4</td><td>NewsArticles-2058</td><td>country</td><td>8</td></tr>\n",
       "    <tr><td class='row_index'>5</td><td>NewsArticles-2058</td><td>turkey</td><td>7</td></tr>\n",
       "    <tr><td class='row_index'>6</td><td>NewsArticles-2058</td><td>europe</td><td>6</td></tr>\n",
       "    <tr><td class='row_index'>7</td><td>NewsArticles-3016</td><td>politic</td><td>7</td></tr>\n",
       "    <tr><td class='row_index'>8</td><td>NewsArticles-3016</td><td>party</td><td>6</td></tr>\n",
       "    <tr><td class='row_index'>9</td><td>NewsArticles-3665</td><td>medium</td><td>21</td></tr>\n",
       "    <tr><td class='row_index'>10</td><td>NewsArticles-3665</td><td>candidate</td><td>19</td></tr>\n",
       "    <tr><td class='row_index'>11</td><td>NewsArticles-3665</td><td>macron</td><td>15</td></tr>\n",
       "    <tr><td class='row_index'>12</td><td>NewsArticles-3665</td><td>france</td><td>9</td></tr>\n",
       "    <tr><td class='row_index'>13</td><td>NewsArticles-3665</td><td>election</td><td>9</td></tr>\n",
       "    <tr><td class='row_index'>14</td><td>NewsArticles-3665</td><td>le</td><td>7</td></tr>\n",
       "    <tr><td class='row_index'>15</td><td>NewsArticles-3665</td><td>coverage</td><td>6</td></tr>\n",
       "  </tbody>\n",
       "  </table>\n",
       "  <div class='footer'>\n",
       "    <div class='frame_dimensions'>16 rows &times; 3 columns</div>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<Frame [16 rows x 3 cols]>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_terms_datatable(dtm, vocab, doc_labels, lo_thresh=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Term frequency–inverse document frequency transformation (tf-idf)\n",
    "\n",
    "[Term frequency–inverse document frequency transformation (tf-idf)](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) is a matrix transformation that is often applied to DTMs in order to reflect the importance of a token to a document. The `bow_stats` module provides the function [tfidf()](api.rst#tmtoolkit.bow.bow_stats.tfidf) for this. When the input is a sparse matrix, and the calculation supports operating on sparce matrices, the output will also be a sparse matrix, which means that the tf-idf transformation is implemented in a very memory-efficient way.\n",
    "\n",
    "Let's apply tf-idf to our DTM using the default way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<5x514 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 578 stored elements in COOrdinate format>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import tfidf\n",
    "\n",
    "tfidf_mat = tfidf(dtm)\n",
    "tfidf_mat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the output is a sparse matrix. Let's have a look at its values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0.     , 0.     , 0.02581, ..., 0.     , 0.     , 0.     ],\n",
       "        [0.03132, 0.     , 0.     , ..., 0.     , 0.     , 0.     ],\n",
       "        [0.     , 0.     , 0.     , ..., 0.00584, 0.     , 0.00746],\n",
       "        [0.     , 0.00783, 0.     , ..., 0.     , 0.00783, 0.     ],\n",
       "        [0.     , 0.     , 0.00276, ..., 0.00276, 0.     , 0.     ]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_mat.todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course we can also pass this matrix to `sorted_terms_datatable()` and observe that some rankings have changed in comparison to the untransformed DTM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='datatable'>\n",
       "  <table class='frame'>\n",
       "  <thead>\n",
       "    <tr class='colnames'><td class='row_index'></td><th>doc</th><th>token</th><th>value</th></tr>\n",
       "    <tr class='coltypes'><td class='row_index'></td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='float' title='float64'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><td class='row_index'>0</td><td>NewsArticles-119</td><td>day</td><td>0.0774339</td></tr>\n",
       "    <tr><td class='row_index'>1</td><td>NewsArticles-119</td><td>nhs</td><td>0.0659349</td></tr>\n",
       "    <tr><td class='row_index'>2</td><td>NewsArticles-119</td><td>victoria</td><td>0.0659349</td></tr>\n",
       "    <tr><td class='row_index'>3</td><td>NewsArticles-1206</td><td>car</td><td>0.125276</td></tr>\n",
       "    <tr><td class='row_index'>4</td><td>NewsArticles-1206</td><td>garda</td><td>0.125276</td></tr>\n",
       "    <tr><td class='row_index'>5</td><td>NewsArticles-1206</td><td>collision</td><td>0.0939572</td></tr>\n",
       "    <tr><td class='row_index'>6</td><td>NewsArticles-2058</td><td>merkel</td><td>0.0521985</td></tr>\n",
       "    <tr><td class='row_index'>7</td><td>NewsArticles-2058</td><td>refugee</td><td>0.04847</td></tr>\n",
       "    <tr><td class='row_index'>8</td><td>NewsArticles-2058</td><td>eu</td><td>0.0379488</td></tr>\n",
       "    <tr><td class='row_index'>9</td><td>NewsArticles-3016</td><td>politic</td><td>0.0548084</td></tr>\n",
       "    <tr><td class='row_index'>10</td><td>NewsArticles-3016</td><td>farron</td><td>0.0391488</td></tr>\n",
       "    <tr><td class='row_index'>11</td><td>NewsArticles-3016</td><td>party</td><td>0.0367811</td></tr>\n",
       "    <tr><td class='row_index'>12</td><td>NewsArticles-3665</td><td>medium</td><td>0.0738989</td></tr>\n",
       "    <tr><td class='row_index'>13</td><td>NewsArticles-3665</td><td>candidate</td><td>0.0668609</td></tr>\n",
       "    <tr><td class='row_index'>14</td><td>NewsArticles-3665</td><td>macron</td><td>0.052785</td></tr>\n",
       "  </tbody>\n",
       "  </table>\n",
       "  <div class='footer'>\n",
       "    <div class='frame_dimensions'>15 rows &times; 3 columns</div>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<Frame [15 rows x 3 cols]>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_terms_datatable(tfidf_mat, vocab, doc_labels, top_n=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tf-idf matrix is calculated from a DTM $D$ as $\\textit{tf}(D) \\cdot \\textit{idf}(D)$.\n",
    "\n",
    "\n",
    "There are different variants for how to calculate the term frequency $\\textit{tf}(D)$ and the inverse document frequency $\\textit{idf(D)}$. tmtoolkit contains several functions that implement some of these variants. For $\\text{tf()}$ these are:\n",
    "\n",
    "- [tf_binary()](api.rst#tmtoolkit.bow.bow_stats.tf_binary): binary term frequency matrix (matrix contains 1 whenever a term occurred in a document, else 0)\n",
    "- [tf_proportions()](api.rst#tmtoolkit.bow.bow_stats.tf_proportions): proportional term frequency matrix (term counts are normalized by document length)\n",
    "- [tf_log()](api.rst#tmtoolkit.bow.bow_stats.tf_log): log-normalized term frequency matrix (by default $\\log(1 + D)$)\n",
    "- [tf_double_norm()](api.rst#tmtoolkit.bow.bow_stats.tf_double_norm): double-normalized term frequency matrix\n",
    "    $K + (1-K) \\cdot \\frac{D}{\\textit{rowmax}(D)}$, where $\\textit{rowmax}(D)$ is a vector containing the maximum term count per document\n",
    "\n",
    "As you can see, all the term frequency functions are prefixed with a `tf_`. There are also two variants for $\\textit{idf()}$:\n",
    "\n",
    "- [idf()](api.rst#tmtoolkit.bow.bow_stats.idf): calculates $\\log(\\frac{a + N}{b + \\textit{df}(D)})$ where $a$ and $b$ are smoothing constants, $N$ is the number of documents and $\\textit{df}(D)$ calculates the [document frequency](#Document-lengths,-document-and-term-frequencies,-token-co-occurrences)\n",
    "- [idf_probabilistic()](api.rst#tmtoolkit.bow.bow_stats.idf_probabilistic): calculates $\\log(a + \\frac{N - \\textit{df}(D)}{\\textit{df}(D)})$\n",
    "\n",
    "The term frequency functions always return a sparse matrix if possible and if the input is sparse. Let's try out two term frequency functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0, 0, 1, ..., 0, 0, 0],\n",
       "        [1, 0, 0, ..., 0, 0, 0],\n",
       "        [0, 0, 0, ..., 1, 0, 1],\n",
       "        [0, 1, 0, ..., 0, 1, 0],\n",
       "        [0, 0, 1, ..., 1, 0, 0]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import tf_binary, tf_proportions\n",
    "\n",
    "tf_binary(dtm).todense()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0.     , 0.     , 0.02632, ..., 0.     , 0.     , 0.     ],\n",
       "        [0.025  , 0.     , 0.     , ..., 0.     , 0.     , 0.     ],\n",
       "        [0.     , 0.     , 0.     , ..., 0.00595, 0.     , 0.00595],\n",
       "        [0.     , 0.00625, 0.     , ..., 0.     , 0.00625, 0.     ],\n",
       "        [0.     , 0.     , 0.00281, ..., 0.00281, 0.     , 0.     ]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_proportions(dtm).todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like the [document frequency](#Document-lengths,-document-and-term-frequencies,-token-co-occurrences) function [doc_frequencies()](api.rst#tmtoolkit.bow.bow_stats.doc_frequencies), the inverse document frequency functions also return a vector with the same length as the vocabulary. Let's use these functions and have a look at the inverse document frequency of certain tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('70', 1.252762968495368),\n",
       " ('abuse', 1.252762968495368),\n",
       " ('access', 0.9808292530117262),\n",
       " ('accession', 1.252762968495368),\n",
       " ('accusation', 1.252762968495368),\n",
       " ('act', 1.252762968495368),\n",
       " ('addition', 1.252762968495368),\n",
       " ('address', 1.252762968495368),\n",
       " ('administration', 1.252762968495368),\n",
       " ('affiliation', 1.252762968495368)]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import idf, idf_probabilistic\n",
    "\n",
    "idf_vec = idf(dtm)\n",
    "list(zip(vocab, idf_vec))[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('70', 1.6094379124341003),\n",
       " ('abuse', 1.6094379124341003),\n",
       " ('access', 0.9162907318741551),\n",
       " ('accession', 1.6094379124341003),\n",
       " ('accusation', 1.6094379124341003),\n",
       " ('act', 1.6094379124341003),\n",
       " ('addition', 1.6094379124341003),\n",
       " ('address', 1.6094379124341003),\n",
       " ('administration', 1.6094379124341003),\n",
       " ('affiliation', 1.6094379124341003)]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probidf_vec = idf_probabilistic(dtm)\n",
    "\n",
    "list(zip(vocab, probidf_vec))[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that due to our very small sample, there's not much variation in the inverse document frequency values.\n",
    "\n",
    "By default, [tfidf()](api.rst#tmtoolkit.bow.bow_stats.tfidf) uses `tf_proportions()` and `idf()` to calculate the tf-idf matrix. You can plug in other functions to get other variants of tf-idf:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.40236, 0.40236, 0.45815, ..., 0.22907, 0.40236, 0.40236],\n",
       "       [0.70413, 0.40236, 0.22907, ..., 0.22907, 0.40236, 0.40236],\n",
       "       [0.40236, 0.40236, 0.22907, ..., 0.32725, 0.40236, 0.5748 ],\n",
       "       [0.40236, 0.5748 , 0.22907, ..., 0.22907, 0.5748 , 0.40236],\n",
       "       [0.40236, 0.40236, 0.2618 , ..., 0.2618 , 0.40236, 0.40236]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tmtoolkit.bow.bow_stats import tf_double_norm\n",
    "\n",
    "# we also set a \"K\" parameter for \"tf_double_norm\"\n",
    "tfidf_mat2 = tfidf(dtm, tf_func=tf_double_norm,\n",
    "                   idf_func=idf_probabilistic, K=0.25)\n",
    "tfidf_mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='datatable'>\n",
       "  <table class='frame'>\n",
       "  <thead>\n",
       "    <tr class='colnames'><td class='row_index'></td><th>doc</th><th>token</th><th>value</th></tr>\n",
       "    <tr class='coltypes'><td class='row_index'></td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='string' title='str32'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td><td class='float' title='float64'>&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;&#x25AA;</td></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><td class='row_index'>0</td><td>NewsArticles-119</td><td>bbc</td><td>1.20708</td></tr>\n",
       "    <tr><td class='row_index'>1</td><td>NewsArticles-119</td><td>victoria</td><td>1.20708</td></tr>\n",
       "    <tr><td class='row_index'>2</td><td>NewsArticles-119</td><td>nhs</td><td>1.20708</td></tr>\n",
       "    <tr><td class='row_index'>3</td><td>NewsArticles-1206</td><td>car</td><td>1.60944</td></tr>\n",
       "    <tr><td class='row_index'>4</td><td>NewsArticles-1206</td><td>garda</td><td>1.60944</td></tr>\n",
       "    <tr><td class='row_index'>5</td><td>NewsArticles-1206</td><td>road</td><td>1.30767</td></tr>\n",
       "    <tr><td class='row_index'>6</td><td>NewsArticles-2058</td><td>merkel</td><td>1.60944</td></tr>\n",
       "    <tr><td class='row_index'>7</td><td>NewsArticles-2058</td><td>refugee</td><td>1.52322</td></tr>\n",
       "    <tr><td class='row_index'>8</td><td>NewsArticles-2058</td><td>germany</td><td>1.09212</td></tr>\n",
       "    <tr><td class='row_index'>9</td><td>NewsArticles-3016</td><td>politic</td><td>1.60944</td></tr>\n",
       "    <tr><td class='row_index'>10</td><td>NewsArticles-3016</td><td>farron</td><td>1.26456</td></tr>\n",
       "    <tr><td class='row_index'>11</td><td>NewsArticles-3016</td><td>putin</td><td>1.09212</td></tr>\n",
       "    <tr><td class='row_index'>12</td><td>NewsArticles-3665</td><td>medium</td><td>1.60944</td></tr>\n",
       "    <tr><td class='row_index'>13</td><td>NewsArticles-3665</td><td>candidate</td><td>1.49448</td></tr>\n",
       "    <tr><td class='row_index'>14</td><td>NewsArticles-3665</td><td>macron</td><td>1.26456</td></tr>\n",
       "  </tbody>\n",
       "  </table>\n",
       "  <div class='footer'>\n",
       "    <div class='frame_dimensions'>15 rows &times; 3 columns</div>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<Frame [15 rows x 3 cols]>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_terms_datatable(tfidf_mat2, vocab, doc_labels, top_n=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have generated a DTM, we can use it for topic modeling. The [next chapter](topic_modeling.ipynb) will show how tmtoolkit can be used to evaluate the quality of your model, export essential information from it and visualize the results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}