Tutorial_Toolboxcor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial for using the fancy_name_toolbox_for_mighty_topic_Modeling\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following tutorial will explain the usage of the python library \"cophi_toolbox\". If you have not done so yet, please follow the instructions for installing jupyter and all necessary python libraries mentioned in readme.txt/installation_instructions.\n",
    "\n",
    "The toolbox provides two different approaches for preprocessing which both lead to tokenized text. After that you can use one of two topic modeling algorithms to create topic models for your corpus. The last step will be to visualize the results.\n",
    "During the preprocessing, the data your topic modeling algorithm will handle is a mayor concern. Different kinds of texts will react differently to certain preprocessing steps, and furthermore, adapting the preprocessing is crucial for a better topic model. Therefore, the cophi_toolbox can either take plain text (Step 1-?) or already nlp enhanced csv files (Step ? to ?) as input. For now, the csv functions here are designed to use the dkproWrapper output. For more information follow this link: https://github.com/DARIAH-DE/DARIAH-DKPro-Wrapper/tree/master/doc\n",
    "\n",
    "\n",
    "Before you start, make sure to have Git and the Topics repository ready to use on your computer:\n",
    "1. Download and install [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)\n",
    "2. Open the [command-line interface](https://en.wikipedia.org/wiki/Command-line_interface), type `git clone https://github.com/DARIAH-DE/Topics.git` and press Enter\n",
    "3. Access the new folder *topics* in your package explorer\n",
    "4. To install the required packages, simply run `setup.py`\n",
    "5. Install [Jupyter](http://jupyter.readthedocs.io/en/latest/install.html) and run it by typing `jupyter notebook` in the command-line\n",
    "5. Access the folder **Topics** through Jupyter in your browser, open **IntroductionTopics.ipynb** and follow the instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Importing packages\n",
    "Firstly, we get access to the functionalities of the toolbox by importing them. For using its functions we use the prefix of the toolbox's submodules (pre, visual and mallet)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from dariah_topics import preprocessing as pre\n",
    "from dariah_topics import visualization as visual\n",
    "from dariah_topics import mallet as mal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Loading txt-Files\n",
    "We start by using plain text files, if you want to use nlp annotated data (csv files), skip to Step ?\n",
    "\n",
    "With the second step we load the plain text corpus into memory for further preprocessing. The Tutorial includes an example set of data. For using your own corpus, change the path accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Creates a list of documents from the directory (in this case 'corpus_txt', a sample text corpus)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_txt = \"corpus_txt\"\n",
    "\n",
    "doclist_txt = pre.create_document_list(path_txt)\n",
    "\n",
    "doclist_txt[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Creates a list of filenames used for the datamodel and the visualization*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc_labels = list(pre.get_labels(doclist_txt))\n",
    "doc_labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Reads/yields the text files of the corpus*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_txt = pre.read_from_txt(doclist_txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Tokenizing\n",
    "In this step we tokenize the text (into words), as topic modeling algorithms usually only work on a bag-of-words model. The tokenize function the toolbox provides is a simple unicode tokenizer. Depending on the corpus it might be useful to import your own tokenizer since its efficiency varies with language, epoch etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Tokenizes the corpus and returns a list of tokens of one text as an example*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc_tokens = [list(pre.tokenize(txt)) for txt in list(corpus_txt)]\n",
    "doc_tokens[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3.5: Working with dkproWrapper output\n",
    "As mentioned above, you need to run the code listed in this Step to incorporate csv-Output created by the dkproWrapper.\n",
    "The output is already tokenized. Therefore it is unnecessary to go through Step 3 \"Tokenizing\".\n",
    "Just like in step 2, we need a list of document names first.\n",
    "With the function provided, the program reads only vital information from the csv file as default. You can pass an additional parameter with a list of column names (to read_from_csv; has to be of type 'list') to specify the information you want to retrieve.\n",
    "As default, the filter_csv selects lemmas with the POS-Tags for adjectives, verbs and nouns. By passing an argument (to filter_csv; has to be of type 'list') you can specify the value of the POS-Tags."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Creates a list of documents from the directory (in this case 'corpus_csv', a sample dkproWrapper output)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_csv = \"corpus_csv\"\n",
    "\n",
    "doclist_csv = pre.create_document_list(path_csv, 'csv')\n",
    "doclist_csv[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Creates a list of filenames used for the datamodel and the visualization*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc_labels = list(pre.get_labels(doclist_txt))\n",
    "doc_labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Reads/yields the text files of the corpus. Only reads columns 'ParagraphId', 'TokenId', 'Lemma', 'CPOS', 'NamedEntity'*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_csv = pre.read_from_csv(doclist_csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# needs editing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Filters the dkproWrapper output to only add the lemmas of certain POS-Tags. (adjectives, verbs and nouns by default)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc_tokens = #transform filter_csv into list of list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Transforming the data\n",
    "Using the code in the following sections, the text corpus is transformed into a data structure similar to the one used in Gensim. This new data model allows for a more efficient way of processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Transforms the corpus into a marketmatrix*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "id_types, doc_ids = pre.create_dictionaries(doc_labels, doc_tokens)\n",
    "sparse_bow = pre.create_mm(doc_labels, doc_tokens, id_types, doc_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When looking at the data structure by running the section below, you can see that each document and each word type of the corpus is represented by a number (column doc_id and token_id). In turn, each token_id has an assigned word count in the adjacent column, showing how often the word with the assigned token_id occurs in the document no. 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Shows the datamodel*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sparse_bow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Removing features\n",
    "Here we show several steps on how to create a list of all words we want to ignore for the actual topic modeling step. One way is  to remove the most frequent words, as these are usually function words with little semantic meaning. Another possibility is to remove hapax legomena (words occuring only once) which are usually considered noise in a topic model. The third is to simply use a stopword list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Reads a txt file (in this case the english stopword list en.txt), one word per line, and creates a list*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os.path\n",
    "basepath = os.path.abspath('.')\n",
    "\n",
    "with open(os.path.join(basepath, \"tutorial_supplementals\", \"stopwords\", \"en\"), 'r', encoding = 'utf-8') as f: \n",
    "    stopword_list = f.read().split('\\n')\n",
    "    \n",
    "stopword_list = set(stopword_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Removes stopwords*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sparse_df_stopwords_removed = pre.remove_features(sparse_bow, id_types, stopword_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Creates a stopword list consisting of the 100 most frequent words*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stopword_list = pre.find_stopwords(sparse_bow, id_types, 100)\n",
    "len(stopword_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Creates a list of words which occur only once per document*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hapax_list = pre.find_hapax(sparse_bow, id_types)\n",
    "len(hapax_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Combines the list of most frequent words with the words occuring only once into one list*\n",
    "*Cleans the corpus with the combined list*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feature_list = set(stopword_list).union(hapax_list)\n",
    "clean_term_frequency = pre.remove_features(sparse_bow, id_types, feature_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Topic modeling with Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following lines of code tranform the data structure we created earlier into the format used by Gensim and saves it on disk. With the next code block we create the topic model. Depending on the size of the corpus used, lean back and get a coffee. Or a pot. A biig pot. Or go home and get some sleep :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Imports Gensim functions*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.models import LdaModel\n",
    "from gensim.corpora import MmCorpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_docs = max(sparse_bow.index.get_level_values(\"doc_id\"))\n",
    "num_types = max(sparse_bow.index.get_level_values(\"token_id\"))\n",
    "sum_counts = sum(sparse_bow[0])\n",
    "\n",
    "header_string = str(num_docs) + \" \" + str(num_types) + \" \" + str(sum_counts) + \"\\n\"\n",
    "\n",
    "with open(\"gb_plain.mm\", 'w', encoding = \"utf-8\") as f:\n",
    "    pass\n",
    "\n",
    "with open(\"gb_plain.mm\", 'a', encoding = \"utf-8\") as f:\n",
    "    f.write(\"%%MatrixMarket matrix coordinate real general\\n\")\n",
    "    f.write(header_string)\n",
    "    sparse_bow.to_csv( f, sep = ' ', header = None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#\n",
    "mm = MmCorpus(\"gb_plain.mm\")\n",
    "doc2id = {value : key for key, value in doc_ids.items()}\n",
    "type2id = {value : key for key, value in id_types.items()}\n",
    "\n",
    "model = LdaModel(corpus=mm, id2word=type2id, num_topics=60, alpha = \"symmetric\", passes = 10)\n",
    "\n",
    "topic_nr_x = model.get_topic_terms(10)\n",
    "\n",
    "[type2id[i[0]] for i in topic_nr_x]\n",
    "\n",
    "topics = model.show_topics(num_topics = 60)\n",
    "\n",
    "topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6.5: Topic Modeling with Mallet\n",
    "Another algorithm for topic modeling is implemented in the java-based software Mallet. For this to work you need to download and install Mallet from http://mallet.cs.umass.edu/download.php.\n",
    "Mallet uses plain text as input, so none of the preprocessing steps of this package are available for Mallet topic modeling as of yet.\n",
    "To use Mallet for topic modeling via this Python Script, you need to pass paths to the Mallet binary, input and output. Again, depending on the size of your corpus, after calling mal.create_MalletMatrix() you should get some coffee... or tea, a lot of tea, a huge pot, believe me, it's great, it's fantastic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Sets path to corpus and Mallet binary and creates an output folder*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_to_corpus = os.path.join(os.path.abspath('.'), 'corpus_txt')\n",
    "path_to_mallet = \"insert_path_here\"\n",
    "malletBinary = mal.create_mallet_binary(path_to_corpus, path_to_mallet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Sets Path to the output folder and works (its magic)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "basepath = os.path.join(os.path.abspath('.'), \"tutorial_supplementals/mallet_output\")\n",
    "doc_topics = os.path.join(basepath, \"doc_topics.txt\")\n",
    "\n",
    "\n",
    "mal.create_MalletMatrix(doc_topics)\n",
    "\n",
    "'''\n",
    "         ,/   *\n",
    "      _,'/_   |\n",
    "      `(\")' ,'/\n",
    "   _ _,-H-./ /\n",
    "   \\_\\_\\.   /\n",
    "     )\" |  (\n",
    "  __/   H   \\__\n",
    "  \\    /|\\    /\n",
    "   `--'|||`--'\n",
    "      ==^==\n",
    "\n",
    "'''\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}