notebooks/IntroducingGensim.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topics – Easy Topic Modeling in Python\n",
    "\n",
    "The text mining technique **Topic Modeling** has become a popular statistical method for clustering documents. This [Jupyter notebook](http://jupyter.org/) introduces a step-by-step workflow, basically containing data preprocessing, the actual topic modeling using **latent Dirichlet allocation** (LDA), which learns the relationships between words, topics, and documents, as well as some interactive visualizations to explore the model.\n",
    "\n",
    "LDA, introduced in the context of text analysis in [2003](http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf), is an instance of a more general class of models called **mixed-membership models**. Involving a number of distributions and parameters, the topic model is typically performed using [Gibbs sampling](https://en.wikipedia.org/wiki/Gibbs_sampling) with conjugate priors and is purely based on word frequencies.\n",
    "\n",
    "There have been written numerous introductions to topic modeling for humanists (e.g. [this one](http://scottbot.net/topic-modeling-for-humanists-a-guided-tour/)), which provide another level of detail regarding its technical and epistemic properties.\n",
    "\n",
    "For this workflow, you will need a corpus (a set of texts) as plain text (`.txt`) or [TEI XML](http://www.tei-c.org/index.xml) (`.xml`). Using the `dariah_topics` package, you also have the ability to process the output of [DARIAH-DKPro-Wrapper](https://github.com/DARIAH-DE/DARIAH-DKPro-Wrapper), a command-line tool for *natural language processing*.\n",
    "\n",
    "Topic modeling works best with very large corpora. The [TextGrid Repository](https://textgridrep.org/) is a great place to start searching for text data. Anyway, to demonstrate the technique, we provide one small text collection in the folder `grenzboten_sample` containing 15 diary excerpts, as well as 15 war diary excerpts, which appeared in *Die Grenzboten*, a German newspaper of the late 19th and early 20th century.\n",
    "\n",
    "**Of course, you can work with your own corpus in this notebook.**\n",
    "\n",
    "We're relying on the LDA implementation by [Radim Řehůřek](https://radimrehurek.com/), called [Gensim](https://radimrehurek.com/project/gensim/), which is attractive because of its multi-core support. Aside from that, we provide two more Jupyter notebooks:\n",
    "\n",
    "* [IntroducingMallet](IntroducingMallet.ipynb), using LDA by [MALLET](http://mallet.cs.umass.edu/topics.php), which is known to be very robust. \n",
    "* [IntroducingLda](IntroducingLda.ipynb), using LDA by [lda](http://pythonhosted.org/lda/index.html), which is lightweight.\n",
    "\n",
    "For more information in general, have a look at the [documentation](http://dev.digital-humanities.de/ci/job/DARIAH-Topics/doclinks/1/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First step: Installing dependencies\n",
    "\n",
    "To work within this Jupyter notebook, you will have to import the `dariah_topics` library. As you do, `dariah_topics` also imports a couple of external libraries, which have to be installed first. `pip` is the preferred installer program in Python. Starting with Python 3.4, it is included by default with the Python binary installers. If you are interested in `pip`, have a look at [this website](https://docs.python.org/3/installing/index.html).\n",
    "\n",
    "To install the `dariah_topics` library with all dependencies, open your commandline, go with `cd` to the folder `Topics` and run:\n",
    "\n",
    "```\n",
    "pip install -r requirements.txt\n",
    "```\n",
    "\n",
    "Alternatively, you can do:\n",
    "\n",
    "```\n",
    "python setup.py install\n",
    "```\n",
    "\n",
    "If you get any errors or are not able to install *all* dependencies properly, try [Stack Overflow](https://stackoverflow.com/questions/tagged/pip) for troubleshooting or create a new issue on our [GitHub page](https://github.com/DARIAH-DE/Topics).\n",
    "\n",
    "**Important**: If you are on macOS or Linux, you will have to use `pip3` and `python3`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some final words\n",
    "As you probably already know, code has to be written in the grey cells. You execute a cell by clicking the **Run**-button (or **Ctrl + Enter**). If you want to run all cells of the notebook at once, click **Cell > Run All** or **Kernel > Restart & Run All** respectively, if you want to restart the Python kernel first. On the left side of an (unexecuted) cell stands `In [ ]:`. The empty bracket means, that the cell hasn't been executed yet. By clicking **Run**, a star appears in the brackets (`In [*]:`), which means the process is running. In most cases, you won't see that star, because your computer is faster than your eyes. You can execute only one cell at once, all following executions will be in the waiting line. If the process of a cell is done, a number appears in the brackets (`In [1]:`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting with topic modeling!\n",
    "\n",
    "Execute the following cell to import modules from the `dariah_topics` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dariah_topics import preprocessing\n",
    "from dariah_topics import postprocessing\n",
    "from dariah_topics import visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Furthermore, we will need some additional functions from external libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import LdaMulticore\n",
    "import metadata_toolbox.utils as metadata\n",
    "import pandas as pd\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's not pay heed to any warnings right now and execute the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preprocessing\n",
    "### 1.1. Reading a corpus of documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining the path to the corpus folder\n",
    "\n",
    "In the present example code, we are using the 30 diary excerpts from the folder `grenzboten`. To use your own corpus, change the path accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_corpus = Path('grenzboten_sample')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specifying the pattern of filenames for metadata extraction\n",
    "\n",
    "You have the ability to extract metadata from the filenames. For instance, if your textfiles look like:\n",
    "\n",
    "```\n",
    "goethe_1816_stella.txt\n",
    "```\n",
    "\n",
    "the pattern would look like this:\n",
    "\n",
    "```\n",
    "{author}_{year}_{title}\n",
    "```\n",
    "\n",
    "So, let's try this for the example corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern = '{author}_{year}_{title}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accessing file paths and metadata\n",
    "We begin by creating a list of all the documents in the folder specified above. That list will tell the function `preprocessing.read_from_pathlist` (see below) which text documents to read. Furthermore, based on filenames we can create some metadata, e.g. author and title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta = pd.concat([metadata.fname2metadata(path, pattern=pattern) for path in path_to_corpus.glob('*.txt')])\n",
    "meta[:5] # by adding '[:5]' to the variable, only the first 5 elements will be printed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read listed documents from folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = list(preprocessing.read_from_pathlist(meta.index))\n",
    "corpus[0][:255] # printing the first 255 characters of the first document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your `corpus` contains as much elements (`documents`) as texts in your corpus are. Each element of `corpus` is a list containing exactly one element, the text itself as one single string including all whitespaces and punctuations:\n",
    "\n",
    "```\n",
    "[['This is the content of your first document.'],\n",
    " ['This is the content of your second document.'],\n",
    " ...\n",
    " ['This is the content of your last document.']]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Tokenize corpus\n",
    "Now, your `documents` in `corpus` will be tokenized. Tokenization is the task of cutting a stream of characters into linguistic units, simply words or, more precisely, tokens. The tokenize function `dariah_topics` provides is a simple Unicode tokenizer. Depending on the corpus, it might be useful to use an external tokenizer function, or even develop your own, since its efficiency varies with language, epoch and text type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_corpus = [list(preprocessing.tokenize(document)) for document in corpus]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, each `document` is represented by a list of separate token strings. As above, have a look at the first document (which has the index `0` as Python starts counting at 0) and show its first 14 words/tokens (that have the indices `0:13` accordingly)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_corpus[0][0:13]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4. Create a document-term matrix\n",
    "\n",
    "The LDA topic model is based on a bag-of-words model of the corpus. To improve performance in large corpora, actual words and document titels are replaced by indices in the actual bag-of-words model.\n",
    "\n",
    "**Because of Gensim's API, you will have to choose the large corpus model.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_term_matrix, document_ids, type_ids = preprocessing.create_document_term_matrix(tokenized_corpus,\n",
    "                                                                                         meta['title'],\n",
    "                                                                                         large_corpus=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5. Feature removal\n",
    "\n",
    "*Stopwords* (also known as *most frequent tokens*) and *hapax legomena* are harmful for LDA and have to be removed from the corpus or the document-term matrix respectively. In this example, the 50 most frequent tokens will be categorized as stopwords.\n",
    "\n",
    "**Hint**: Be careful with removing most frequent tokens, you might remove tokens quite important for LDA. Anyway, to gain better results, it is highly recommended to use an external stopwords list.\n",
    "\n",
    "In this notebook, we combine the 50 most frequent tokens, hapax legomena and an external stopwordslist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List the 100 most frequent words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords = preprocessing.find_stopwords(document_term_matrix, most_frequent_tokens=100, type_ids=type_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the five most frequent words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stopwords[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List hapax legomena"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hapax_legomena = preprocessing.find_hapax_legomena(document_term_matrix, type_ids)\n",
    "print(\"Total number of types in corpus:\", len(type_ids))\n",
    "print(\"Total number of hapax legomena:\", len(hapax_legomena))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optional: Use external stopwordlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_stopwordlist = 'tutorial_supplementals/stopwords/de.txt'\n",
    "external_stopwords = [line.strip() for line in open(path_to_stopwordlist, 'r', encoding='utf-8')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combine lists and remove content from `document_term_matrix`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = stopwords + hapax_legomena + external_stopwords\n",
    "document_term_matrix = preprocessing.remove_features(features, document_term_matrix=document_term_matrix, type_ids=type_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6. Creating specific Gensim corpus format\n",
    "\n",
    "With the function below, you can create the specific Gensim corpus format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gensim_corpus = postprocessing.doc2bow(document_term_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Model creation\n",
    "\n",
    "The actual topic modeling is done with external state-of-the-art LDA implementations. In this example, we are relying on the open-source toolkit **Gensim** which was used and cited in over 400 commercial and academic applications since 2008."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Rearrange dictionaries "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc2id = {value : key for key, value in document_ids.items()}\n",
    "type2id = {value : key for key, value in type_ids.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Generate LDA model\n",
    "\n",
    "We use the class `LDA` from the library `lda` (which is basically not the same, because Python is case sensitive) to generate a LDA topic model. To instance a `LDA` object, there have to be specified a couple of parameters.\n",
    "\n",
    "But first, if you are curious about any library, module, class or function, try `help()`. This can be very useful, because (at least in a well documented library) explanations of use and parameters will be printed. We're interested in the class `LDA` of the library `lda`, so let's try:\n",
    "\n",
    "```\n",
    "help(LdaMulticore)\n",
    "```\n",
    "\n",
    "This will print something like this (in fact even more):\n",
    "\n",
    "```\n",
    "Help on class LdaMulticore in module gensim.models.ldamulticore:\n",
    "\n",
    "class LdaMulticore(gensim.models.ldamodel.LdaModel)\n",
    " |  The constructor estimates Latent Dirichlet Allocation model parameters based\n",
    " |  on a training corpus:\n",
    " |  \n",
    " |  >>> lda = LdaMulticore(corpus, num_topics=10)\n",
    " |  \n",
    " |  You can then infer topic distributions on new, unseen documents, with\n",
    " |  \n",
    " |  >>> doc_lda = lda[doc_bow]\n",
    " |  \n",
    " |  The model can be updated (trained) with new documents via\n",
    " |  \n",
    " |  >>> lda.update(other_corpus)\n",
    " |  \n",
    " |  Methods defined here:\n",
    " |  \n",
    " |  __init__(parameter ...)\n",
    " |      If given, start training from the iterable `corpus` straight away. If not given,\n",
    " |      the model is left untrained (presumably because you want to call `update()` manually).\n",
    " |      \n",
    " |      `num_topics` is the number of requested latent topics to be extracted from\n",
    " |      the training corpus.\n",
    " |      \n",
    " |      `id2word` is a mapping from word ids (integers) to words (strings). It is\n",
    " |      used to determine the vocabulary size, as well as for debugging and topic\n",
    " |      printing.\n",
    " |      \n",
    " |      `workers` is the number of extra processes to use for parallelization. Uses\n",
    " |      all available cores by default: `workers=cpu_count()-1`. **Note**: for\n",
    " |      hyper-threaded CPUs, `cpu_count()` returns a useless number -- set `workers`\n",
    " |      directly to the number of your **real** cores (not hyperthreads) minus one,\n",
    " |      for optimal performance.\n",
    " |      \n",
    " |      If `batch` is not set, perform online training by updating the model once\n",
    " |      every `workers * chunksize` documents (online training). Otherwise,\n",
    " |      run batch LDA, updating model only once at the end of each full corpus pass.\n",
    " |      \n",
    " |      `alpha` and `eta` are hyperparameters that affect sparsity of the document-topic\n",
    " |      (theta) and topic-word (lambda) distributions. Both default to a symmetric\n",
    " |      1.0/num_topics prior.\n",
    " |      \n",
    " |      `alpha` can be set to an explicit array = prior of your choice. It also\n",
    " |      support special values of 'asymmetric' and 'auto': the former uses a fixed\n",
    " |      normalized asymmetric 1.0/topicno prior, the latter learns an asymmetric\n",
    " |      prior directly from your data.\n",
    " |      \n",
    " |      `eta` can be a scalar for a symmetric prior over topic/word\n",
    " |      distributions, or a matrix of shape num_topics x num_words,\n",
    " |      which can be used to impose asymmetric priors over the word\n",
    " |      distribution on a per-topic basis. This may be useful if you\n",
    " |      want to seed certain topics with particular words by boosting\n",
    " |      the priors for those words.\n",
    "```\n",
    "\n",
    "So, now you know how to define the number of topics and the number of sampling iterations as well. A higher number of iterations will probably yield a better model, but also increases processing time. `alpha`, `eta` and `random_state` are so-called *hyperparameters*. They influence the model's performance, so feel free to play around with them. In the present example, we will leave the default values. Furthermore, there exist various methods for hyperparameter optimization, e.g. gridsearch or Gaussian optimization.\n",
    "\n",
    "**Warning: This step can take quite a while!** Meaning something between some seconds and some hours depending on corpus size and the number of iterations. Our example corpus should be done within a minute or two at `iterations=1000`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "model = LdaMulticore(corpus=gensim_corpus, id2word=type2id, num_topics=10, passes=10, iterations=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. Create document-topic matrix\n",
    "\n",
    "The generated model object can now be translated into a human-readable document-topic matrix (that is a actually a pandas data frame) that constitutes our principle exchange format for topic modeling results. For generating the matrix from a Gensim model, we can use the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topics = postprocessing.show_topics(model=model)\n",
    "topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Model visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each topic has a certain probability for each document in the corpus (have a look at the cell below). This probability distributions are visualized in an interactive **heatmap** (the darker the color, the higher the probability) which displays the kind of information\n",
    "                that is presumably most useful to literary scholars. Going beyond pure exploration, this visualization can be used to show thematic developments over a set of texts as well as a single text, akin to a dynamic topic model. What might become\n",
    "                apparent here, is that some topics correlate highly with a specific author or group of authors, while other topics correlate highly with a specific text or group of texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualization.notebook_handling()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_topics = postprocessing.show_document_topics(topics=topics,\n",
    "                                                      model=model,\n",
    "                                                      document_labels=meta['title'],\n",
    "                                                      doc2bow=gensim_corpus)\n",
    "document_topics[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Distribution of topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Distribution of topics over all documents\n",
    "\n",
    "The distribution of topics over all documents can now be visualized in a heat map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "PlotDocumentTopics = visualization.PlotDocumentTopics(document_topics)\n",
    "PlotDocumentTopics.interactive_heatmap()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or a static heatmap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "static_heatmap = PlotDocumentTopics.static_heatmap()\n",
    "static_heatmap.show()"
   ]
  }
 ],
 "metadata": {
  "git": {
   "suppress_outputs": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}