notebooks/introducing-cophi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing text for Natural Language Processing\n",
    "\n",
    "This is a short, user-oriented introduction to the Python library `cophi`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cophi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High-level API\n",
    "\n",
    "There are basically two functions:\n",
    "\n",
    "- `cophi.documet()` to process single text files.\n",
    "- `cophi.corpus()` to process a whole corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about the parameters?\n",
    "\n",
    "You can use Python’s built-in `help()` function to access `cophi`’s short and sweet documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function document in module cophi.api:\n",
      "\n",
      "document(filepath, lemma=False, pos=None, jar='ddw-0.4.6.jar', language='de', **kwargs)\n",
      "    Read a text file and create a Document object.\n",
      "    \n",
      "    Parameter:\n",
      "        filepath (str): Path to the text file.\n",
      "        lemma (bool): If True, lemmatize text (optional).\n",
      "        pos (list): If not None, filter POS tags (optional).\n",
      "        jar (str): Path to DARIAH-DKPro-Wrapper JAR file (optional).\n",
      "        language (str): Language of text (optional).\n",
      "        title (str): Describing title for the document (optional).\n",
      "        lowercase (bool): If True, writes all letters in lowercase (optional).\n",
      "        n (int): Number of tokens per ngram (optional).\n",
      "        token_pattern (str): Regex pattern for one token (optional).\n",
      "        maximum (int): Stop tokenizing after that much tokens (optional).\n",
      "    \n",
      "    Returns:\n",
      "        A Document object.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(cophi.document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing a single text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have now a <class 'cophi.text.model.Document'>.\n"
     ]
    }
   ],
   "source": [
    "dickens = cophi.document(filepath=\"british-fiction-corpus/dickens_bleak.txt\",\n",
    "                         title=\"dickens-bleak\",\n",
    "                         lowercase=True,\n",
    "                         n=2,\n",
    "                         token_pattern=r\"\\p{L}+\\p{P}?\\p{L}+\",\n",
    "                         maximum=1000)\n",
    "\n",
    "print(\"You have now a {}.\".format(type(dickens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `help()` to learn about the properties of your new object (this is only a subset):\n",
    "\n",
    "```python\n",
    ">>> help(dickens)\n",
    "```\n",
    "```\n",
    "Help on Document in module cophi.model object:\n",
    "\n",
    "class Document(builtins.object)\n",
    " |  Model class for a Document.\n",
    " |  \n",
    " |  Attributes:\n",
    " |      text (str): Content of the text file.\n",
    " |      title (str): Describing title for the document.\n",
    " |      lowercase (bool): If True, all letters are lowercase.\n",
    " |      n (int): Number of words per ngram.\n",
    " |      token_pattern (str): Regex pattern for one token.\n",
    " |      maximum (int): Stopped tokenizing after that much tokens.\n",
    " |      tokens (list): Tokenized content of the document.\n",
    " |  ----------------------------------------------------------------------\n",
    " |  Data descriptors defined here:\n",
    " |  \n",
    " |  bow\n",
    " |      Bag-of-words representation.\n",
    " |  \n",
    " |  freq_spectrum\n",
    " |      Frequency spectrum.\n",
    " |  \n",
    " |  hapax\n",
    " |      Hapax legomena.\n",
    " |  \n",
    " |  ngrams\n",
    " |      Constructed ngrams.\n",
    " |  \n",
    " |  num_tokens\n",
    " |      Number of tokens.\n",
    " |  \n",
    " |  num_types\n",
    " |      Number of types.\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Access ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The return value is a <class 'generator'>.\n"
     ]
    }
   ],
   "source": [
    "ngrams = dickens.ngrams\n",
    "\n",
    "print(\"The return value is a {}.\".format(type(ngrams)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can e.g. iterate over this generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bleak house\n",
      "house by\n",
      "by charles\n",
      "charles dickens\n"
     ]
    }
   ],
   "source": [
    "for ngram in ngrams:\n",
    "    print(ngram)\n",
    "    if \"dickens\" in ngram:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing a corpus\n",
    "\n",
    "With `cophi.corpus()`, you can pipe a collection of text files through several NLP tasks, and get a document-term matrix with useful attributes and methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic usage\n",
    "\n",
    "Depending on the size of the text collection, this may take a while. These are the basic steps of the pipeline:\n",
    "\n",
    "1. read,\n",
    "2. get metadata,\n",
    "3. tokenize,\n",
    "4. count tokens,\n",
    "5. arrange in a document-term matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus, metadata = cophi.corpus(directory=\"british-fiction-corpus\",\n",
    "                                filepath_pattern=\"**/*.txt\",\n",
    "                                treat_as=\".txt\",\n",
    "                                encoding=\"utf-8\",\n",
    "                                lowercase=True,\n",
    "                                token_pattern=r\"\\p{L}+\\p{P}?\\p{L}+\",\n",
    "                                maximum=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One useful attribute would be `stats`, which gives you information about some statistics of your corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "documents         10.0\n",
       "types          33050.0\n",
       "tokens       1310189.0\n",
       "hapax          16495.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the document-term matrix with the attribute `dtm`, which returns a pandas `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>83833ed6-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed7-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed8-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a'mighty  a'most  a'n  a'ready  a-bakin  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "\n",
       "                                      a-be  a-beatin  a-bed  a-begging  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f   NaN       NaN    2.0        NaN   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f   NaN       NaN    NaN        NaN   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f   NaN       NaN    NaN        NaN   \n",
       "\n",
       "                                      a-bleatin  \n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f        NaN  \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f        NaN  \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f        NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.dtm.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `cophi.corpus()` returned not only the `Corpus` object but also a `Metadata` object with metadata extracted from the filenames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filepath</th>\n",
       "      <th>parent</th>\n",
       "      <th>suffix</th>\n",
       "      <th>title</th>\n",
       "      <th>uuid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>british-fiction-corpus/fielding_tom.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>fielding_tom</td>\n",
       "      <td>83833ed6-678c-11e9-b708-7c7a9196f28f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>british-fiction-corpus/thackeray_lyndon.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>thackeray_lyndon</td>\n",
       "      <td>83833ed7-678c-11e9-b708-7c7a9196f28f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>british-fiction-corpus/fielding_joseph.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>fielding_joseph</td>\n",
       "      <td>83833ed8-678c-11e9-b708-7c7a9196f28f</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filepath                  parent suffix  \\\n",
       "0      british-fiction-corpus/fielding_tom.txt  british-fiction-corpus   .txt   \n",
       "1  british-fiction-corpus/thackeray_lyndon.txt  british-fiction-corpus   .txt   \n",
       "2   british-fiction-corpus/fielding_joseph.txt  british-fiction-corpus   .txt   \n",
       "\n",
       "              title                                  uuid  \n",
       "0      fielding_tom  83833ed6-678c-11e9-b708-7c7a9196f28f  \n",
       "1  thackeray_lyndon  83833ed7-678c-11e9-b708-7c7a9196f28f  \n",
       "2   fielding_joseph  83833ed8-678c-11e9-b708-7c7a9196f28f  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.iloc[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily add additional metadata to the existing object (which is basically a pandas `DataFrame`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filepath</th>\n",
       "      <th>parent</th>\n",
       "      <th>suffix</th>\n",
       "      <th>title</th>\n",
       "      <th>uuid</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>british-fiction-corpus/fielding_tom.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>fielding_tom</td>\n",
       "      <td>83833ed6-678c-11e9-b708-7c7a9196f28f</td>\n",
       "      <td>1868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>british-fiction-corpus/thackeray_lyndon.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>thackeray_lyndon</td>\n",
       "      <td>83833ed7-678c-11e9-b708-7c7a9196f28f</td>\n",
       "      <td>1853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>british-fiction-corpus/fielding_joseph.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>fielding_joseph</td>\n",
       "      <td>83833ed8-678c-11e9-b708-7c7a9196f28f</td>\n",
       "      <td>1860</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filepath                  parent suffix  \\\n",
       "0      british-fiction-corpus/fielding_tom.txt  british-fiction-corpus   .txt   \n",
       "1  british-fiction-corpus/thackeray_lyndon.txt  british-fiction-corpus   .txt   \n",
       "2   british-fiction-corpus/fielding_joseph.txt  british-fiction-corpus   .txt   \n",
       "\n",
       "              title                                  uuid  year  \n",
       "0      fielding_tom  83833ed6-678c-11e9-b708-7c7a9196f28f  1868  \n",
       "1  thackeray_lyndon  83833ed7-678c-11e9-b708-7c7a9196f28f  1853  \n",
       "2   fielding_joseph  83833ed8-678c-11e9-b708-7c7a9196f28f  1860  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata[\"year\"] = [1868, 1853, 1860, 1799, 1742, 1749, 1844, 1850, 1876, 1848]\n",
    "metadata.iloc[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you can save it with its pandas `to_csv()` method, edit or enrich it with a spreadsheet program like Excel, and read it again with `pd.read_csv()`.\n",
    "\n",
    "> In case you want to map metadata to the document-term matrix, be aware that each line in the document-term matrix must have a unique index value.\n",
    "\n",
    "The `Corpus` object also has a static method for mapping metadata (based on a [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier)) to the document-term matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1868_fielding_tom</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1853_thackeray_lyndon</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1860_fielding_joseph</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       a'mighty  a'most  a'n  a'ready  a-bakin  a-be  \\\n",
       "1868_fielding_tom           NaN     NaN  NaN      NaN      NaN   NaN   \n",
       "1853_thackeray_lyndon       NaN     NaN  NaN      NaN      NaN   NaN   \n",
       "1860_fielding_joseph        NaN     NaN  NaN      NaN      NaN   NaN   \n",
       "\n",
       "                       a-beatin  a-bed  a-begging  a-bleatin  \n",
       "1868_fielding_tom           NaN    2.0        NaN        NaN  \n",
       "1853_thackeray_lyndon       NaN    NaN        NaN        NaN  \n",
       "1860_fielding_joseph        NaN    NaN        NaN        NaN  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.map_metadata(data=corpus.dtm,\n",
    "                    metadata=metadata,\n",
    "                    uuid=\"uuid\",  # connection between metadata and document\n",
    "                    fields=[\"year\", \"title\"],  # year and title as document descriptor\n",
    "                    sep=\"_\").iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced usage\n",
    "\n",
    "The `Corpus` object has some useful methods and attributes. In addition to various normalized or standardized document-term matrices, there are a multitude of lexical complexity measures – by document or corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Corpus vocabulary (or types)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"a'mighty\",\n",
       " \"a'most\",\n",
       " \"a'n\",\n",
       " \"a'ready\",\n",
       " 'a-bakin',\n",
       " 'a-be',\n",
       " 'a-beatin',\n",
       " 'a-bed',\n",
       " 'a-begging',\n",
       " 'a-bleatin']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.types[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Most frequent words\n",
    "\n",
    "You can return a list of the tokens itself (with `as_list=True`, which is default):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'and', 'to', 'of', 'in', 'he', 'was', 'that', 'his', 'it']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.mfw(n=10,\n",
    "           as_list=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or with their frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "the     65281.0\n",
       "and     44016.0\n",
       "to      41217.0\n",
       "of      36234.0\n",
       "in      22514.0\n",
       "he      20581.0\n",
       "was     19609.0\n",
       "that    19295.0\n",
       "his     16007.0\n",
       "it      14827.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mfw = corpus.mfw(n=10,\n",
    "                 as_list=False)\n",
    "mfw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because this is a pandas `Series`, you can easily plot the frequencies (first using `sort_values()`, that most common word is at the top of the plot):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f0faca234e0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD8CAYAAACW/ATfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEmpJREFUeJzt3X+Q7XV93/Hnq4Dgesm9KgwD0emKsWWAKsgJCaIOWjsVQ9IabeOvmZCkuW3RGjSawdFxs51khiaONTaJes0P0w616jWJMyQarY31Dir0rFzgXpUKQkdclWCLARYJ4rt/nO8l52723t1zz/fsOYfv8zGzs5/zOd/z/b4/ly+v/ez3e/Z8UlVIkh7b/t60C5AkTZ5hL0kdYNhLUgcY9pLUAYa9JHWAYS9JHWDYS1IHGPaS1AGGvSR1wPHTLuCQU045pRYXF6ddhiTNlZWVlXuq6tTNtpuZsF9cXKTf70+7DEmaK0n+z1a28zKOJHWAYS9JHWDYS1IHGPaS1AEzc4N2dXWV5eXlaZchSdtqaWlpW47jzF6SOmCksE+yK8kVTfuSJNdOpixJUptGndnvAq6YRCGSpMkZ9Zr91cDTk+wHHgYeSLIXOBdYAV5TVZXkAuCdwA7gHuDyqvpmi3VLkkYw6sz+KuD2qjoPeDNwPnAlcDZwJnBxkhOA/wS8vKouAP4A+PWNdpZkd5J+kv7a2tqxjkGStIlx341zQ1XdBdDM9heBexnM9D+VBOA4YMNZfVXtAfYAnHHGGTVmLZKkIxg37B8aaj/S7C/Awaq6aMx9S5JaMuplnPuAkzfZ5lbg1CQXASQ5Ick5x1KcJKkdI83sq+o7Sa5LcgB4EPj2Btv8TZKXA+9OsrM5xruAg20ULEka3ciXcarqVUfof91Qez/w/DHqkiS1KFWzcV+01+uVn2cvSaNJslJVvc228+MSJKkDDHtJ6gDDXpI6wLCXpA4w7CWpAwx7SeoAw16SOsCwl6QOMOwlqQMMe0nqgHE/4rg1q6urLC8vT7sM6TFnaWlp2iVoBmzLzD7J/dtxHEnSxryMI0kdsOWwT/KnSVaSHEyyu+m7P8mvJ7kpyReSnNb0Py3J55PckuTXJlW8JGlrRpnZ/3yzgHgPeH2SJwNPAL5QVc8CPgv8YrPtbwHvqap/xBHWn5UkbZ9Rwv71SW4CvgA8FXgG8DfAtc3zKwwWHAe4GPhg0/4vR9phkt1J+kn6a2tro9QtSRrBlsI+ySXAi4CLmln8jcBJwMP1t6ufHFpw/JBNV0Wpqj1V1auq3sLCwkiFS5K2bqsz+53A/6uqtSRnAT++yfbXAa9o2q8+1uIkSe3Yath/Ajg+yZeBqxlcyjmaXwJem+QW4IfHqE+S1IIt/VFVVT0EXLrBUzuGttkL7G3adwAXDW33tjFqlCSNyQXHJWmOueC4JOlRhr0kdYBhL0kdYNhLUgcY9pLUAYa9JHWAYS9JHWDYS1IHGPaS1AGGvSR1gAuOS3PExcN1rJzZS1IHtBb2SXYluaKt/UmS2tPmzH4XYNhL0gxq85r91cDTk+wHPtX0XcpgecJfq6oPtXgsSdII2pzZXwXcXlXnMVjJ6jzgWQzWrv3NJKe3eCxJ0ggmdYP2ucAHq+qRqvo28D+BH12/UZLdSfpJ+mtraxMqRZI01XfjVNWequpVVW9hYWGapUjSY1qbYX8fcHLT3gf8TJLjkpwKPB+4ocVjSZJG0NoN2qr6TpLrkhwAPg7cDNzE4Abtr1TVt9o6liRpNK3+BW1VvWpd15vb3L8k6dikqqZdAwC9Xq/6/f60y5CkuZJkpap6m23nxyVIUgcY9pLUAYa9JHWAYS9JHWDYS1IHGPaS1AGGvSR1gGEvSR1g2EtSBxj2ktQBrX42zjhWV1dZXl6edhnSMVtaWpp2CdIRObOXpA6YeNgneX2SLye5ZtLHkiRtbDsu41wBvKiq7tqGY0mSNtDqzD7JG5McaL6uTPJe4Ezg40ne0OaxJElb19rMPskFwM8BPwYEuB54DfBi4AVVdc8Gr9kN7AbYuXNnW6VIktZpc2b/XOBPquqBqrof+GPgeUd7gQuOS9L28N04ktQBbYb9PuCfJ1lI8gTgpU2fJGnKWrtmX1VfTPIB4Iam6/eq6sYkbR1CknSMWn3rZVW9E3jnur7FNo8hSRpdqmraNQDQ6/Wq3+9PuwxJmitJVqqqt9l23qCVpA4w7CWpAwx7SeoAw16SOsCwl6QOMOwlqQMMe0nqAMNekjrAsJekDjDsJakDtmNZwi1ZXV1leXl52mVoREtLS9MuQdIWOLOXpA5oew3az7W5P0lSO1oN+6p6Tpv7kyS1o+2Z/f3N90uSfCbJ3iRfSXJNXMVEkqZmktfszweuBM4GzgQuXr9Bkt1J+kn6a2trEyxFkrptkmF/Q1XdVVU/APYDi+s3qKo9VdWrqt7CwsIES5Gkbptk2D801H6EGXqbpyR1jW+9lKQOMOwlqQNavbRSVTua758BPjPU/7o2jyNJGk2qato1ANDr9arf70+7DEmaK0lWqqq32XZexpGkDjDsJakDDHtJ6gDDXpI6wLCXpA4w7CWpAwx7SeoAw16SOsCwl6QOMOwlqQNm5mOHV1dXWV5ennYZGrK0tDTtEiS1ZOyZfZLFJAfaKEaSNBlexpGkDmgr7I9L8v4kB5N8Msnjkzw9ySeSrCTZl+Sslo4lSRpRW2H/DOB3quoc4F7gZcAe4N9V1QXAm4DfbelYkqQRtXWD9o6q2t+0VxgsLv4c4CNJDm1z4voXJdkN7AbYuXNnS6VIktZrK+zXLy5+GnBvVZ13tBdV1R4GvwFwxhlnzMYqKpL0GDSpG7R/DdyR5F8AZOBZEzqWJGkTk3w3zquBX0hyE3AQ+GcTPJYk6SjGvoxTVXcC5w49fsfQ0y8ed/+SpPG54LgkzTEXHJckPcqwl6QOMOwlqQMMe0nqAMNekjrAsJekDjDsJakDDHtJ6gDDXpI6wLCXpA5wwXEBLi4uPdY5s5ekDjDsJakDDHtJ6oBNwz7Jm5O8vmn/xyT/o2m/MMk1Sd6TpJ/kYJLlodddneRLSW5O8o4j7V+SNHlbuUG7D/hl4N1ADzgxyQnA84DPAh+pqv+b5Djg00meCXwDeClwVlVVkl2TKV+StBVbuYyzAlyQ5IcYLCz+eQah/zwGPwj+ZZIvAjcC5wBnA98Fvgf8fpKfBtY22nGS3c1vBf21tQ03kSS1YNOwr6qHgTuAy4HPMQj4FwA/AjwIvAn4x1X1TODPgJOq6vvAhcBe4DLgE0fY956q6lVVb2FhYfzRSJI2tNUbtPsYhPpnm/a/YTCT/yHgAeC7SU4DLgVIsgPYWVV/DrwBeFbLdUuSRrDVP6raB7wV+HxVPZDke8C+qropyY3AV4CvA9c1258MfCzJSUCAN7ZctyRpBFsK+6r6NHDC0ON/MNS+/Agvu3CsyiRJrUlVTbsGAHq9XvX7/WmXIUlzJclKVfU2284/qpKkDjDsJakDDHtJ6gDDXpI6wLCXpA4w7CWpAwx7SeoAw16SOsCwl6QOMOwlqQO2+kFoE7e6usry8vLmG6p1S0tL0y5B0oQ5s5ekDjhq2CfZleSKpn1JkmtH2XmSy5OcMU6BkqTxbTaz3wVcMcb+LwcMe0mass2u2V8NPD3JfuBh4IEke4FzGaxN+5pmQfG3Az8JPJ7B0oX/GngZg7Vqr0nyIHBRVT04oXFIko5is5n9VcDtVXUe8GbgfOBKBouKnwlc3Gz321X1o1V1LoPAv6yq9gJ94NVVdd5GQe+C45K0PUa9QXtDVd1VVT8A9gOLTf8Lklyf5BbghcA5W9mZC45L0vYY9a2XDw21HwGOb9aZ/V2gV1VfT/KrwEkt1SdJasFmM/v7GCwefjSHgv2eJDuAl4/4eknShB11Zl9V30lyXZIDwIPAtzfY5t4k7wcOAN8C/tfQ0x8A3usNWkmark0v41TVq47Q/7qh9tuAt22wzUeBj45ToCRpfKmqadcAQK/Xq36/P+0yJGmuJFmpqt5m2/lxCZLUAYa9JHWAYS9JHWDYS1IHGPaS1AGGvSR1gGEvSR1g2EtSBxj2ktQBhr0kdcCoH3E8MaurqywvL0+7jLm0tLQ07RIkzThn9pLUAccc9kkWm48+Xt//75O8aLyyJEltav0yTlW9ve19SpLGM+5lnOOSvD/JwSSfTPL4JB9I8nKAJFcn+VKSm5O8o4V6JUnHYNyZ/TOAV1bVLyb5MPCyQ08keTLwUuCsqqoku9a/OMluYDfAzp07xyxFknQk487s76iq/U17BVgceu67wPeA30/y08Da+hdX1Z6q6lVVb2FhYcxSJElHMm7YPzTUfoSh3xSq6vvAhcBe4DLgE2MeS5J0jCb2PvskO4CFqvrzJNcBX5vUsSRJRzfJP6o6GfhYkpOAAG+c4LEkSUdxzGFfVXcC5w493ujdNhce6/4lSe1JVU27BgB6vV71+/1plyFJcyXJSlX1NtvOj0uQpA4w7CWpAwx7SeoAw16SOsCwl6QOMOwlqQMMe0nqAMNekjrAsJekDnDB8TniwuKSjpUze0nqgImFfZLPNd8Xk7xqUseRJG1uYmFfVc9pmouAYS9JUzTJmf39TfNq4HlJ9id5w6SOJ0k6su24QXsV8KaqumwbjiVJ2sBUb9Am2Z2kn6S/tvZ31iOXJLVkqmFfVXuqqldVvYWFhWmWIkmPadsR9vcxWI9WkjQl2xH2NwOPJLnJG7SSNB0Tu0FbVTua7w8DL5zUcSRJm3PBcUmaYy44Lkl6lGEvSR1g2EtSBxj2ktQBhr0kdcDMvBsnyX3ArdOuY0ynAPdMu4gxOYbZ4BhmwzyM4e9X1ambbTQzK1UBt27l7UOzLEnfMUyfY5gNjmG2eBlHkjrAsJekDpilsN8z7QJa4Bhmg2OYDY5hhszMDVpJ0uTM0sxekjQhMxH2SV6c5NYktyW5agbq+YMkdyc5MNT3pCSfSvLV5vsTm/4keXdT+81Jnj30mp9ttv9qkp8d6r8gyS3Na96dJC3X/9Qkf5nkS0kOJvmlORzDSUluaD4a+2CS5ab/aUmub477oSSPa/pPbB7f1jy/OLSvtzT9tyb5p0P923LeJTkuyY1Jrp3HMSS5s/lvvT9Jv+mbm3OpOcauJHuTfCXJl5NcNG9jGFtVTfULOA64HTgTeBxwE3D2lGt6PvBs4MBQ328AVzXtq4D/0LRfAnwcCPDjwPVN/5OArzXfn9i0n9g8d0OzbZrXXtpy/acDz27aJwP/Gzh7zsYQYEfTPgG4vjneh4FXNP3vBf5t074CeG/TfgXwoaZ9dnNOnQg8rTnXjtvO8w54I/BfgWubx3M1BuBO4JR1fXNzLjXH+CPgXzXtxwG75m0MY/8bTL0AuAj4i6HHbwHeMgN1LXJ42N8KnN60T2fwdwEA7wNeuX474JXA+4b639f0nQ58Zaj/sO0mNJaPAf9kXscALABfBH6MwR+4HL/+3AH+ArioaR/fbJf159Oh7bbrvAOeAnyawZoO1zY1zdsY7uTvhv3cnEvATuAOmnuU8ziGNr5m4TLODwNfH3p8V9M3a06rqm827W8BpzXtI9V/tP67NuifiOZSwPkMZsZzNYbm8sd+4G7gUwxmsfdW1fc3OO6jtTbPfxd48iZj2I7z7l3ArwA/aB4/mfkbQwGfTLKSZHfTN0/n0tOAvwL+sLmc9ntJnjBnYxjbLIT93KnBj++ZfxtTkh3AR4Erq+qvh5+bhzFU1SNVdR6D2fGFwFlTLmkkSS4D7q6qlWnXMqbnVtWzgUuB1yZ5/vCTc3AuHc/gsux7qup84AEGl20eNQdjGNsshP03gKcOPX5K0zdrvp3kdIDm+91N/5HqP1r/Uzbob1WSExgE/TVV9cfzOIZDqupe4C8ZXLbYleTQx3wMH/fRWpvndwLfYfSxteli4KeS3An8NwaXcn5rzsZAVX2j+X438CcMfvDO07l0F3BXVV3fPN7LIPznaQzjm/Z1JAY/db/G4FetQzeZzpmBuhY5/Jr9b3L4zZzfaNo/weE3c25o+p/E4DrhE5uvO4AnNc+tv5nzkpZrD/CfgXet65+nMZwK7Grajwf2AZcBH+Hwm5tXNO3XcvjNzQ837XM4/Obm1xjc2NzW8w64hL+9QTs3YwCeAJw81P4c8OJ5OpeaY+wD/mHT/tWm/rkaw9j/BtMuoPmHegmDd4zcDrx1Bur5IPBN4GEGs4JfYHDt9NPAV4H/PvQfOcDvNLXfAvSG9vPzwG3N188N9feAA81rfpt1N45aqP+5DH4lvRnY33y9ZM7G8EzgxmYMB4C3N/1nNv9j3cYgNE9s+k9qHt/WPH/m0L7e2tR5K0PvktjO847Dw35uxtDUelPzdfDQMebpXGqOcR7Qb86nP2UQ1nM1hnG//AtaSeqAWbhmL0maMMNekjrAsJekDjDsJakDDHtJ6gDDXpI6wLCXpA4w7CWpA/4/Y//11v1x6UUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "mfw.sort_values().plot.barh(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hapax legomena\n",
    "\n",
    "In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, or in this case in a single text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"a'mighty\",\n",
       " \"a'n\",\n",
       " 'a-bakin',\n",
       " 'a-beatin',\n",
       " 'a-begging',\n",
       " 'a-bleatin',\n",
       " 'a-bringin',\n",
       " 'a-carrying',\n",
       " 'a-collectin',\n",
       " 'a-comin']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.hapax[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Drop features from document-term matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A total of 16595 features will be dropped.\n"
     ]
    }
   ],
   "source": [
    "features = corpus.mfw(n=100) + corpus.hapax\n",
    "print(\"A total of {} features will be dropped.\".format(len(features)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size before: 33050.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Vocabulary size before: {}\".format(corpus.stats[\"types\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size after: 16455\n"
     ]
    }
   ],
   "source": [
    "dtm = corpus.drop(dtm=corpus.dtm,\n",
    "                  features=features)\n",
    "print(\"Vocabulary size after: {}\".format(dtm.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Culling\n",
    "Remove features that do not appear in a minimum of documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a-going</th>\n",
       "      <th>abandon</th>\n",
       "      <th>abandoned</th>\n",
       "      <th>abated</th>\n",
       "      <th>abhorrence</th>\n",
       "      <th>abide</th>\n",
       "      <th>abilities</th>\n",
       "      <th>ability</th>\n",
       "      <th>able</th>\n",
       "      <th>abode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>83833ed6-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed7-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed8-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>39.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a-going  abandon  abandoned  abated  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f      NaN      2.0       15.0     6.0   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f      NaN      NaN        NaN     1.0   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f      NaN      3.0        5.0     3.0   \n",
       "\n",
       "                                      abhorrence  abide  abilities  ability  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f         8.0    1.0        5.0      1.0   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f         NaN    NaN        2.0      NaN   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f         2.0    NaN        6.0      NaN   \n",
       "\n",
       "                                      able  abode  \n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f  59.0    4.0  \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f   8.0    1.0  \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f  39.0    NaN  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "culled = corpus.cull(dtm=corpus.dtm,\n",
    "                     threshold=5,  # must appear in at least 5 documents\n",
    "                     keepna=True)\n",
    "culled.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Frequency spectrum\n",
    "\n",
    "How often does a frequency occur in the corpus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0     12616\n",
       "2.0      4508\n",
       "3.0      2627\n",
       "4.0      1760\n",
       "5.0      1253\n",
       "6.0      1012\n",
       "7.0       863\n",
       "8.0       586\n",
       "9.0       519\n",
       "10.0      467\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq_spectrum = corpus.freq_spectrum[:10]\n",
    "freq_spectrum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting is easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f0faa6bb3c8>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE9FJREFUeJzt3X+MZWd93/H3p15sGAd5/YNY612na4SFipDAzhVeSoJSHIztIruNjGQrqR3H6ZbWaiGpFNlFYjT0n7qNQmK18maFnRhqHCcGkpVFAy6Qokplk7vg2Otf8fAj7O4YFmyzpIxocfLtH/csvb47s7u+587cO3veL+lqznnOM/d5nntmPnPm3HPPk6pCktQNf2/aHZAkrR9DX5I6xNCXpA4x9CWpQwx9SeoQQ1+SOsTQl6QOMfQlqUMMfUnqkE3T7sCo8847r7Zv3z7tbkjShrJv377vVtVrTlRv5kJ/+/bt9Pv9aXdDkjaUJH99MvU8vSNJHWLoS1KHGPqS1CGGviR1yMy9kbu0tMTCwsK0uyFJ62p+fn5d2vFIX5I6pFXoJ7knyeEk+1fZniR3JllM8miSS9u0J0lqp+2R/u8DVx5n+1XAxc1jJ3BXy/YkSS20Cv2q+iLw/HGqXAt8tAa+BGxOsqVNm5Kk8a31Of2twIGh9YNN2Usk2Zmkn6S/vLy8xl2SpO6aiTdyq2p3VfWqqjc3Nzft7kjSKWutQ/8QcOHQ+ramTJI0BWsd+nuAG5ureHYAR6rq2TVuU5K0ilYfzkpyP/BzwHlJDgLzwCsAqmoX8GngamARWAZubtOeJKmdVqFfVTecYHsBt7ZpQ5I0ORnk8uzo9Xrl/fQl6eVJsq+qeieqNxNX70iS1oehL0kdYuhLUocY+pLUIYa+JHWIoS9JHWLoS1KHGPqS1CGGviR1iKEvSR3S6t47a2FpaYmFhYVpd0Mtzc/PT7sLklYw9pF+kguTfCHJE0keT/K+Feo4MbokzZA2R/ovAv+2qr6c5NXAviQPV9UTQ3WGJ0a/jMHE6Je1aFOS1MLYR/pV9WxVfblZ/hvgSY6d/9aJ0SVphkzkjdwk24FLgL0jm05qYnRJ0vpoHfpJfgL4BPD+qvr+mM+xM0k/SX95ebltlyRJq2gV+klewSDw76uqT65Q5aQmRq+q3VXVq6re3Nxcmy5Jko6jzdU7Ae4Gnqyq31qlmhOjS9IMaXP1ztuAfwY8luSRpuzfAT8FTowuSbNo7NCvqv8J5AR1nBhdkmaIE6NL0inAidElSccw9CWpQwx9SeoQQ1+SOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDnBh9hjm5uKRJ80hfkjqkzf30X5nkz5P8ZZLHkxxzeJ7kjCQPJFlMsreZVlGSNCVtjvT/D/COqnoT8GbgymailGG3AC9U1euADwN3tGhPktTS2KFfA/+7WX1F8xi9T/O1wL3N8oPA5c2MW5KkKWg7R+5pzaxZh4GHq2rvSJWtwAGAqnoROAKc26ZNSdL4WoV+Vf1tVb2ZwYTnb0nyxnGeJ8nOJP0k/eXl5TZdkiQdx0Su3qmq7wFfAK4c2XQIuBAgySbgLOC5Fb5/d1X1qqo3Nzc3iS5JklbQ5uqd1yTZ3Cy/Cngn8NRItT3ATc3ydcDna9bmZ5SkDmnz4awtwL1JTmPwx+MPq+qhJB8C+lW1B7gb+FiSReB54PrWPZYkjW3s0K+qR4FLVij/4NDyD4H3jNuGJGmyMmtnW3q9XvX7/Wl3Q5I2lCT7qqp3onrehkGSOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA5pc8O1NbG0tMTCwjHT7c60+fn5aXdBkk6KR/qS1CGtQ7+ZMvErSR5aYdsZSR5Isphkb5LtbduTJI1vEkf67wOeXGXbLcALVfU64MPAHRNoT5I0prYTo28D/jHwkVWqXAvc2yw/CFyeJG3alCSNr+2R/m8DvwH83SrbtwIHAKrqReAIcO5oJSdGl6T10WaO3HcDh6tqX9tOODG6JK2PNkf6bwOuSfIN4A+AdyT5ryN1DgEXAiTZBJwFPNeiTUlSC2OHflXdXlXbqmo7gwnPP19VvzRSbQ9wU7N8XVNntuZnlKQOmfiHs5J8COhX1R7gbuBjSRaB5xn8cZAkTclEQr+q/gz4s2b5g0PlPwTeM4k2JEntZdbOtvR6ver3+9PuhiRtKEn2VVXvRPW8DYMkdYihL0kdYuhLUocY+pLUIYa+JHWIoS9JHWLoS1KHGPqS1CGGviR1iKEvSR0y8RuutbW0tMTCwsLU2p+fn59a25K01jzSl6QOaTtH7jeSPJbkkSTH3CUtA3cmWUzyaJJL27QnSWpnEqd3/lFVfXeVbVcBFzePy4C7mq+SpClY69M71wIfrYEvAZuTbFnjNiVJq2gb+gV8Nsm+JDtX2L4VODC0frApe4kkO5P0k/SXl5dbdkmStJq2p3d+pqoOJflJ4OEkT1XVF1/uk1TVbmA3wAUXXDBbs7pI0imk1ZF+VR1qvh4GPgW8ZaTKIeDCofVtTZkkaQrGDv0kZyZ59dFl4Apg/0i1PcCNzVU8O4AjVfXs2L2VJLXS5vTO+cCnkhx9no9X1Z8meS9AVe0CPg1cDSwCy8DN7borSWpj7NCvqq8Bb1qhfNfQcgG3jtuGJGmyMsjl2dHr9arfP+ZzXpKk40iyr6p6J6rnbRgkqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6xNCXpA4x9CWpQwx9SeqQSUyXOFFLS0ssLCyseTvz8/Nr3oYkzZq2E6NvTvJgkqeSPJnkrSPbnRhdkmZI2yP93wH+tKquS3I6MDey3YnRJWmGtJlE5Szg7cDdAFX1f6vqeyPVnBhdkmZIm9M7FwHfAX4vyVeSfKSZQWvYSU2MLklaH21CfxNwKXBXVV0C/AC4bZwnSrIzST9Jf3l5uUWXJEnH0yb0DwIHq2pvs/4ggz8Cw05qYvSq2l1Vvarqzc2Nvi0gSZqUsUO/qr4FHEjy+qbocuCJkWpOjC5JM6Tt1Tv/GrivuXLna8DNTowuSbOrVehX1SPA6JyMTowuSTPKidEl6RTgxOiSpGMY+pLUIYa+JHWIoS9JHWLoS1KHGPqS1CGGviR1iKEvSR1i6EtShxj6ktQhnZkY3YnQJckjfUnqlDZz5L4+ySNDj+8nef9InSS5M8likkeTjE6yIklaR2Of3qmqp4E3AyQ5jcGMWJ8aqXYVcHHzuAy4q/kqSZqCSZ3euRz4alX99Uj5tcBHa+BLwOYkWybUpiTpZZpU6F8P3L9C+VbgwND6waZMkjQFrUO/mSrxGuCPWjzHziT9JP3l5eW2XZIkrWISR/pXAV+uqm+vsO0QcOHQ+ram7CWqandV9aqqNzc3N4EuSZJWMonQv4GVT+0A7AFubK7i2QEcqapnJ9CmJGkMrT6cleRM4J3Avxgqey9AVe0CPg1cDSwCy8DNbdqTJLXTKvSr6gfAuSNlu4aWC7i1TRuSpMnJIJdnR6/Xq36/P+1uSNKGkmRfVfVOVM/bMEhShxj6ktQhhr4kdYihL0kdYuhLUocY+pLUIYa+JHWIoS9JHWLoS1KHGPqS1CGt7r2zFpaWllhYWBjre+fn5yfcG0k6tXikL0kd0ir0k/xakseT7E9yf5JXjmw/I8kDSRaT7E2yvU17kqR2xg79JFuBfwP0quqNwGkM5soddgvwQlW9DvgwcMe47UmS2mt7emcT8Kokm4A5YGlk+7XAvc3yg8DlSdKyTUnSmMYO/ao6BPwm8E3gWQZTIX52pNpW4EBT/0XgCCOTroATo0vSemlzeudsBkfyFwEXAGcm+aVxnsuJ0SVpfbQ5vfPzwNer6jtV9SPgk8A/HKlzCLgQoDkFdBbwXIs2JUkttAn9bwI7ksw15+kvB54cqbMHuKlZvg74fM3a/IyS1CFtzunvZfDm7JeBx5rn2p3kQ0muaardDZybZBH4deC2lv2VJLXQ6hO5VTUPjH4M9oND238IvKdNG5KkycmsnW3p9XrV7/en3Q1J2lCS7Kuq3onqeRsGSeoQQ1+SOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6pNW9d9bC0tISCwsLL+t75udHb/8jSVqJR/qS1CGtQj/J+5LsT/J4kvevsD1J7kyymOTRJJe2aU+S1E6b6RLfCPxz4C3Am4B3J3ndSLWrgIubx07grnHbkyS11+ZI/x8Ae6tquZn0/H8AvzBS51rgozXwJWBzki0t2pQktdAm9PcDP5vk3CRzwNU08+EO2QocGFo/2JS9RJKdSfpJ+svLyy26JEk6nrGv3qmqJ5PcAXwW+AHwCPC3Yz7XbmA3wAUXXDBbs7pI0imk1Ru5VXV3Vf10Vb0deAH4q5Eqh3jp0f+2pkySNAVtr975yebrTzE4n//xkSp7gBubq3h2AEeq6tk2bUqSxtf2w1mfSHIu8CPg1qr6XpL3AlTVLuDTDM71LwLLwM0t25MktdAq9KvqZ1co2zW0XMCtbdqQJE1OBrk8O3q9XvX7/Wl3Q5I2lCT7qqp3onrehkGSOsTQl6QOMfQlqUMMfUnqEENfkjrE0JekDjH0JalDDH1J6hBDX5I6ZMNPjO6k6JJ08jzSl6QOOWHoJ7knyeEk+4fKzknycJJnmq9nr/K9NzV1nkly0yQ7Lkl6+U7mSP/3gStHym4DPldVFwOfa9ZfIsk5wDxwGYPJ0+dX++MgSVofJwz9qvoi8PxI8bXAvc3yvcA/WeFb3wU8XFXPV9ULwMMc+8dDkrSOxj2nf/7QDFjfAs5foc5JTYouSVo/rd/IbSZKaXVT/iQ7k/ST9JeXl9t2SZK0inFD/9tJtgA0Xw+vUOekJ0Wvqt1V1auq3tzc3JhdkiSdyLihvwc4ejXOTcCfrFDnM8AVSc5u3sC9oimTJE3JyVyyeT/wv4DXJzmY5BbgPwDvTPIM8PPNOkl6ST4CUFXPA/8e+Ivm8aGmTJI0JSf8RG5V3bDKpstXqNsHfnVo/R7gnrF7J0maKCdGl6RTgBOjS5KOYehLUocY+pLUIYa+JHWIoS9JHTJzV+8k+Rvg6Wn3Y0LOA7477U5MwKkyDnAss+hUGQdMdyx/v6pec6JKMzdzFvD0yVx2tBEk6Z8KYzlVxgGOZRadKuOAjTEWT+9IUocY+pLUIbMY+run3YEJOlXGcqqMAxzLLDpVxgEbYCwz90auJGntzOKRviRpjcxU6Ce5MsnTSRaTHDPZ+rQluTDJF5I8keTxJO9rys9J8nCSZ5qvZzflSXJnM55Hk1w69Fw3NfWfSXLTam2u8XhOS/KVJA816xcl2dv094EkpzflZzTri8327UPPcXtT/nSSd01pHJuTPJjkqSRPJnnrBt4nv9b8bO1Pcn+SV26U/ZLkniSHk+wfKpvYfkjy00kea77nziRZx3H8p+bn69Ekn0qyeWjbiq/1anm22v5cN1U1Ew/gNOCrwGuB04G/BN4w7X6N9HELcGmz/Grgr4A3AP8RuK0pvw24o1m+GvhvQIAdwN6m/Bzga83Xs5vls6cwnl8HPg481Kz/IXB9s7wL+JfN8r8CdjXL1wMPNMtvaPbTGcBFzf47bQrjuBf41Wb5dGDzRtwnDOaQ/jrwqqH98csbZb8AbwcuBfYPlU1sPwB/3tRN871XreM4rgA2Nct3DI1jxdea4+TZavtz3X7O1rOxE7zQbwU+M7R+O3D7tPt1gj7/CfBOBh8m29KUbWHwWQOA3wVuGKr/dLP9BuB3h8pfUm+d+r4N+BzwDuCh5hfpu0M/2D/eHwxmPHtrs7ypqZfRfTRcbx3HcRaDoMxI+UbcJ1uBA03gbWr2y7s20n4Bto+E5UT2Q7PtqaHyl9Rb63GMbPunwH3N8oqvNavk2fF+z9brMUund47+wB91sCmbSc2/0pcAe4Hzq+rZZtO3gPOb5dXGNAtj/W3gN4C/a9bPBb5XVS+u0Kcf97fZfqSpPwvjuAj4DvB7zamqjyQ5kw24T6rqEPCbwDeBZxm8zvvYmPvlqEnth63N8mj5NPwKg/804OWP43i/Z+tilkJ/w0jyE8AngPdX1feHt9Xgz/dMXxKV5N3A4araN+2+TMAmBv+K31VVlwA/YHAa4cc2wj4BaM53X8vgD9kFwJnAlVPt1ARtlP1wPEk+ALwI3DftvoxrlkL/EHDh0Pq2pmymJHkFg8C/r6o+2RR/O8mWZvsW4HBTvtqYpj3WtwHXJPkG8AcMTvH8DrA5ydFbcwz36cf9bbafBTzH9McBgyOlg1W1t1l/kMEfgY22T2Aw3/TXq+o7VfUj4JMM9tVG3C9HTWo/HGqWR8vXTZJfBt4N/GLzBwxe/jieY/X9uS5mKfT/Ari4eWf7dAZvTO2Zcp9eorla4G7gyar6raFNe4CjVxncxOBc/9HyG5srFXYAR5p/dT8DXJHk7Obo7oqmbF1U1e1Vta2qtjN4nT9fVb8IfAG4bpVxHB3fdU39asqvb64iuQi4mMGbbeumqr4FHEjy+qbocuAJNtg+aXwT2JFkrvlZOzqWDbdfhkxkPzTbvp9kR/Pa3Dj0XGsuyZUMTodeU1XLQ5tWe61XzLNm/6y2P9fHer6BcBJvnlzN4IqYrwIfmHZ/VujfzzD49/RR4JHmcTWD83SfA54B/jtwTlM/wH9pxvMY0Bt6rl8BFpvHzVMc08/x/6/eeS2DH9hF4I+AM5ryVzbri8321w59/wea8T3NGl1NcRJjeDPQb/bLHzO46mND7hNgAXgK2A98jMFVIRtivwD3M3gv4kcM/gO7ZZL7Aeg1r8tXgf/MyJv3azyORQbn6I/+3u860WvNKnm22v5cr4efyJWkDpml0zuSpDVm6EtShxj6ktQhhr4kdYihL0kdYuhLUocY+pLUIYa+JHXI/wPdJ6PGyeQHpQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "freq_spectrum.sort_values().plot.barh(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Document-term matrix sorted in descending order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>the</th>\n",
       "      <th>and</th>\n",
       "      <th>to</th>\n",
       "      <th>of</th>\n",
       "      <th>in</th>\n",
       "      <th>he</th>\n",
       "      <th>was</th>\n",
       "      <th>that</th>\n",
       "      <th>his</th>\n",
       "      <th>it</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>83833ed6-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>8171.0</td>\n",
       "      <td>4295.0</td>\n",
       "      <td>5381.0</td>\n",
       "      <td>5195.0</td>\n",
       "      <td>2800.0</td>\n",
       "      <td>2547.0</td>\n",
       "      <td>2040.0</td>\n",
       "      <td>1934.0</td>\n",
       "      <td>2081.0</td>\n",
       "      <td>1598.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed7-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>5987.0</td>\n",
       "      <td>3963.0</td>\n",
       "      <td>2881.0</td>\n",
       "      <td>3432.0</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>877.0</td>\n",
       "      <td>1724.0</td>\n",
       "      <td>1121.0</td>\n",
       "      <td>1025.0</td>\n",
       "      <td>746.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed8-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>4758.0</td>\n",
       "      <td>2474.0</td>\n",
       "      <td>2885.0</td>\n",
       "      <td>2555.0</td>\n",
       "      <td>1512.0</td>\n",
       "      <td>1694.0</td>\n",
       "      <td>1125.0</td>\n",
       "      <td>1022.0</td>\n",
       "      <td>1309.0</td>\n",
       "      <td>770.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         the     and      to      of      in  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f  8171.0  4295.0  5381.0  5195.0  2800.0   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f  5987.0  3963.0  2881.0  3432.0  1956.0   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f  4758.0  2474.0  2885.0  2555.0  1512.0   \n",
       "\n",
       "                                          he     was    that     his      it  \n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f  2547.0  2040.0  1934.0  2081.0  1598.0  \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f   877.0  1724.0  1121.0  1025.0   746.0  \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f  1694.0  1125.0  1022.0  1309.0   770.0  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.sort(dtm=corpus.dtm).iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Document-term matrix with z-scores\n",
    "\n",
    "Used formula is: $z_x = \\frac{x - \\mu}{\\sigma}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>83833ed6-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.707107</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed7-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed8-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a'mighty  a'most  a'n  a'ready  a-bakin  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "\n",
       "                                      a-be  a-beatin     a-bed  a-begging  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f   NaN       NaN  0.707107        NaN   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f   NaN       NaN       NaN        NaN   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f   NaN       NaN       NaN        NaN   \n",
       "\n",
       "                                      a-bleatin  \n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f        NaN  \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f        NaN  \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f        NaN  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.zscores.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Document-term matrix with TF-IDF scores\n",
    "\n",
    "Used formula is: $tf-idf_{t,d} \\; = \\; tf_{t,d} \\times idf_t \\; = \\; tf_{t,d} \\times log(\\frac{N}{df_t})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>83833ed6-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00002</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed7-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83833ed8-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a'mighty  a'most  a'n  a'ready  a-bakin  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f       NaN     NaN  NaN      NaN      NaN   \n",
       "\n",
       "                                      a-be  a-beatin    a-bed  a-begging  \\\n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f   NaN       NaN  0.00002        NaN   \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f   NaN       NaN      NaN        NaN   \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f   NaN       NaN      NaN        NaN   \n",
       "\n",
       "                                      a-bleatin  \n",
       "83833ed6-678c-11e9-b708-7c7a9196f28f        NaN  \n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f        NaN  \n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f        NaN  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.tfidf.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Number of tokens by document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "83833ed6-678c-11e9-b708-7c7a9196f28f    158439.0\n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f     97524.0\n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f     87923.0\n",
       "83833ed9-678c-11e9-b708-7c7a9196f28f     99570.0\n",
       "83833eda-678c-11e9-b708-7c7a9196f28f    203949.0\n",
       "83833edb-678c-11e9-b708-7c7a9196f28f    146348.0\n",
       "83833edc-678c-11e9-b708-7c7a9196f28f    147059.0\n",
       "83833edd-678c-11e9-b708-7c7a9196f28f    127439.0\n",
       "83833ede-678c-11e9-b708-7c7a9196f28f     92012.0\n",
       "83833edf-678c-11e9-b708-7c7a9196f28f    149926.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.num_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Number of types by document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "83833ed6-678c-11e9-b708-7c7a9196f28f    10023\n",
       "83833ed7-678c-11e9-b708-7c7a9196f28f     9298\n",
       "83833ed8-678c-11e9-b708-7c7a9196f28f     7574\n",
       "83833ed9-678c-11e9-b708-7c7a9196f28f     9174\n",
       "83833eda-678c-11e9-b708-7c7a9196f28f    14225\n",
       "83833edb-678c-11e9-b708-7c7a9196f28f     7875\n",
       "83833edc-678c-11e9-b708-7c7a9196f28f    10006\n",
       "83833edd-678c-11e9-b708-7c7a9196f28f    10357\n",
       "83833ede-678c-11e9-b708-7c7a9196f28f     8828\n",
       "83833edf-678c-11e9-b708-7c7a9196f28f     7845\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.num_types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get a document by its UUID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'83833ed6-678c-11e9-b708-7c7a9196f28f'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "uuid = corpus.documents.index[0]  # first document\n",
    "uuid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select it and show its bag-of-words representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "the          8171\n",
       "history        58\n",
       "of           5195\n",
       "tom           117\n",
       "jones         636\n",
       "foundling      16\n",
       "by            900\n",
       "henry           1\n",
       "fielding        1\n",
       "book           43\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document = corpus.documents[uuid]\n",
    "document.bow[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or its tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the',\n",
       " 'history',\n",
       " 'of',\n",
       " 'tom',\n",
       " 'jones',\n",
       " 'foundling',\n",
       " 'by',\n",
       " 'henry',\n",
       " 'fielding',\n",
       " 'book']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.tokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get token lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 7, 2, ..., 4, 2, 5])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the mean token length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.528064428581347"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.lengths.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sliding window\n",
    "\n",
    "You can iterate with a sliding window over your document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'history', 'of', 'tom', 'jones', 'foundling', 'by', 'henry', 'fielding', 'book']\n"
     ]
    }
   ],
   "source": [
    "for chunk in document.window(size=10):\n",
    "    print(chunk)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get paragraphs or segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 14351 paragraphs.\n",
      "And 159 segments, 1000 tokens each segment.\n",
      "First paragraph: THE HISTORY OF TOM JONES, A FOUNDLING\n"
     ]
    }
   ],
   "source": [
    "paragraphs = list(document.paragraphs(sep=\"\\n\"))\n",
    "segments = list(document.segments(size=1000))\n",
    "\n",
    "print(\"There are {} paragraphs.\".format(len(paragraphs)))\n",
    "print(\"And {} segments, 1000 tokens each segment.\".format(len(segments)))\n",
    "print(\"First paragraph: {}\".format(paragraphs[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Complexity metrics\n",
    "\n",
    "There are a plenty of complexity metrics for measuring lexical richness (adapted from [here](https://github.com/tsproisl/Linguistic_and_Stylistic_Complexity)).\n",
    "\n",
    "\n",
    "Measures that use sample size and vocabulary size:\n",
    "\n",
    "  * Type-token ratio\n",
    "  * Guiraud’s $R$\n",
    "  * Herdan’s $C$\n",
    "  * Dugast’s $k$\n",
    "  * Maas’ $a^2$\n",
    "  * Dugast’s $U$\n",
    "  * Tuldava’s $LN$\n",
    "  * Brunet’s $W$\n",
    "  * Carroll’s $CTTR$\n",
    "  * Summer’s $S$\n",
    "\n",
    "Measures that use part of the frequency spectrum:\n",
    "\n",
    "  * Honoré’s $H$\n",
    "  * Sichel’s $S$\n",
    "  * Michéa’s $M$\n",
    "\n",
    "Measures that use the whole frequency spectrum:\n",
    "\n",
    "  * Entropy\n",
    "  * Yule’s $K$\n",
    "  * Simpson’s $D$\n",
    "  * Herdan’s $V_m$\n",
    "\n",
    "Parameters of probabilistic models:\n",
    "\n",
    "  * Orlov’s $Z$\n",
    "\n",
    "\n",
    "You can access them for the _whole corpus_ using the corresponding attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07266508877726802"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.ttr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by _document_ using the `complexity()` method. You _have_ to use a sliding window in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "sttr = corpus.complexity(measure=\"ttr\", window=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course you can also map metadata here again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sttr</th>\n",
       "      <th>ci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>fielding_tom</th>\n",
       "      <td>0.359880</td>\n",
       "      <td>0.004151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thackeray_lyndon</th>\n",
       "      <td>0.378406</td>\n",
       "      <td>0.005436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fielding_joseph</th>\n",
       "      <td>0.363512</td>\n",
       "      <td>0.004977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dickens_bleak</th>\n",
       "      <td>0.359245</td>\n",
       "      <td>0.007024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thackeray_vanity</th>\n",
       "      <td>0.388614</td>\n",
       "      <td>0.003362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trollope_prime</th>\n",
       "      <td>0.313212</td>\n",
       "      <td>0.004565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dickens_david</th>\n",
       "      <td>0.346240</td>\n",
       "      <td>0.003996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eliot_adam</th>\n",
       "      <td>0.374310</td>\n",
       "      <td>0.006497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eliot_mill</th>\n",
       "      <td>0.365087</td>\n",
       "      <td>0.007684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trollope_phineas</th>\n",
       "      <td>0.307230</td>\n",
       "      <td>0.004331</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      sttr        ci\n",
       "fielding_tom      0.359880  0.004151\n",
       "thackeray_lyndon  0.378406  0.005436\n",
       "fielding_joseph   0.363512  0.004977\n",
       "dickens_bleak     0.359245  0.007024\n",
       "thackeray_vanity  0.388614  0.003362\n",
       "trollope_prime    0.313212  0.004565\n",
       "dickens_david     0.346240  0.003996\n",
       "eliot_adam        0.374310  0.006497\n",
       "eliot_mill        0.365087  0.007684\n",
       "trollope_phineas  0.307230  0.004331"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sttr = corpus.map_metadata(sttr, metadata)\n",
    "sttr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And visualize the result e.g. as a barplot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f0fa7eae748>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcEAAAD8CAYAAAAPKB8vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xu8XfOd//HXO6gkEkGpcRTBVN0lsoMRSlotU1UxoulUkU5n8HPnp5r+dJocpUNNf0arpanRUH6tuqQy/NQ9hNLk5OZEKtSlU07KQ+oWh9D4zB/re3Q5zj7Xffbe56z38/E4j73297Y+a3nsfHzXWnt/FRGYmZkV0ZBaB2BmZlYrToJmZlZYToJmZlZYToJmZlZYToJmZlZYToJmZlZYToJmZlZYToJmZlZYToJmZlZY69Y6ACtv0003jdGjR9c6DDOzAWXhwoUvRcRm3WnrJFjHRo8eTVNTU63DMDMbUCT9obttfTnUzMwKy0nQzMwKy0nQzMwKy0nQzMwKyw/G1LGWlhYaGxtrHYaZVcH06dNrHUIheSZoZmaF1eskKGkjSSf1ot/q9Dpa0rLe7r8SJM2VVOqgvCTp+7WIyczMqqcvM8GNgA8kQUkD/hJrRDRFxGm1jsPMzPpXX5LghcD2kpZIWiBpnqQ5wHIASWdJWpb+zuhsIElDJf1UUrOkxZImpvKpkm5JM7YnJU3P9fmypPlp/z+WtE4n46+WdImkxyTdIyn/SwJHpXGekLR/an+gpFvT9gxJV6UYnpZ0Wm7cDmOQdLmkprS/xlz7CyUtl/SopH/vwbk2M7N+0JckOA14KiLGAF8D9gROj4gdJI0DvgLsDewD/IuksZ2MdTIQEbEb8I/A1ZKGprq9gCOB3ckSVknSTsAUYELa/1rg6E7G3wBoiohdgPuB/B3odSNiL+CMduV5OwIHp1imS1qvixjOjYhSivkASbtL+jBwBLBLROwOnN/RjiQdnxJoU2trayeHZGZmfVXJS5fzI+KZtL0fMDsi3gCQdDOwP7C4TN/9gB8ARMTj6Sdvdkh1d0XEqtw4+wF/AcYBCyQBDANe7CS2d4Hr0/a1wM25urbthcDoMv1vi4g1wBpJLwKbA5/qJIYvSDqe7PxuAexMNkN+C/jPNMu8taMdRcRMYCZAQ0NDdHJMZmbWR5VMgm9UcKy89okgAAFXR8Q3KjDmmvS6lvLnY01uu61dhzFI2hY4GxgfES9LmgUMjYi/SNqLLHlOBk4BPtnL+M3MrAL6cjn0dWBkmbp5wCRJwyVtQHYZcF4nY80jXUqUtAOwNbAi1X1a0iaShgGTgIeAe4DJkj6S+mwiaZtOxh9ClngAvgQ82NXBdUO5GDYk+x+CVyVtDvx9qh8BjIqI/w+cCexRgRjMzKwPej0TjIhVkh5KX3N4E3ghV7cozYDmp6IrI6LcpVCAHwGXS2omu9Q5NSLWpMuM84GbgI8C10ZEE4CkbwJ3ShoCvEN2X7HcL4e/AeyV+rxIdi+vTyJieUcxRMQjkhYDjwN/JEvakP0Pwy3pXqeAs/oag5mZ9Y0i6ve2k6SpQCkiTunjOKsjYkRloqqehoaGOOGEE2odhplVgX8xpnIkLUwPJ3ZpwH+nbzBraGjwB8PMrB/VdRKMiFnArO62l/RbYP12xccMxFmgmZn1v7pOgj0VEXvXOgYzMxs4/APaZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWIPqy/KDTUtLC42NjV03NDOrsYH6E4+eCZqZWWEVIglKelbSpmn7N120HS3pSxXa73mSDkrbcyWV2sdjZma1U4gkmBcR+3bRZDTZwruV2Ne3IuLuSoxlZmaVN+iSoKQvS5ovaYmkH0tap1396vQqSRdLWiapWVLbQrsXAvun/meW2cdUSb+SdFea1Z0i6SxJiyU9ImmT1G6WpMkdjWFmZrU3qJKgpJ3IVo2fEBFjgLXA0WWa/wMwBtgDOAi4WNIWwDRgXkSMiYhLOtndrmmM8cAFQGtEjAUeBo7twzEcL6lJUlNra2tvhzEzs24YbE+HfgoYByyQBDAMeLFM2/2An0fEWuAFSfeTJbTXurmv+yLideB1Sa8C/5XKm4Hdexk/ETETmAnZyvK9HcfMzLo22JKggKsj4hvvK5Sm9sO+1uS23829f5fBd17NzAalQXU5FLgHmCzpIwCSNpG0TZm284ApktaRtBnwCWA+8DowsirRmplZTQ2qGUtELJf0TeBOSUOAd4CTyzSfDfwdsBQI4JyI+JOkVcBaSUuBWV3cFzQzswFMEb7tVK9KpVI0NTXVOgwzswFF0sKIKHWn7WC7HGpmZtZtg+pyaKVJOhi4qF3xMxFxRC3iMTOzynIS7ERE3AHcUes4zMysf/hyqJmZFZaToJmZFZaToJmZFZaToJmZFZaToJmZFZaToJmZFZaToJmZFZa/J1jHWlpaaGxsrHUYZmZ9Mn369FqHUJZngmZmVlhOgmZmVlgDOglKelbSpmn7N120HS3pSxXa7yxJkysxlpmZ1c6AToJ5EbFvF01GAxVJgmZmNjgMmCQo6cuS5ktaIunHktZpV786vUrSxZKWSWqWNCU1uRDYP/U/s8w+RkuaJ2lR+ts3N+ZlklZIuhv4SK7PtyQtSPubKUmpfK6kSyQ1SfqdpPGSbpb0pKTz++EUmZlZDw2IJChpJ2AKMCEixgBrgaPLNP8HYAywB3AQcLGkLYBpwLyIGNPJavEvAp+OiD3T/r6fyo8APg7sDBwL5Gedl0XE+IjYFRgGfC5X93Za2PEK4BayVe53BaZK+nCZYz0+Jc6m1tbWMmGamVklDJSvSHwKGAcsSBOtYWQJqyP7AT+PiLXAC5LuB8YDr3VjP+sBl0lqS7Q7pPJP5MZskXRvrs9ESecAw4FNgMeA/0p1c9JrM/BYRKwEkPQ0sBWwqn0AETETmAnQ0NAQ3YjZzMx6aaAkQQFXR8Q33lcoTa3wfs4EXiCbRQ4B3uo0KGko8COgFBF/lDQDGJprsia9vpvbbns/UM69mdmgNSAuhwL3AJMlfQRA0iaStinTdh4wRdI6kjYjm8XNB14HRnaxn1HAyoh4FzgGaLvv+EBuzC2Aiam8LeG9JGkE4CdGzcwGkAExG4mI5ZK+CdwpaQjwDtn9tY7MBv4OWAoEcE5E/EnSKmCtpKXArDL3BX8E3CTpWODXwBu5MT8JLAf+G3g4xfWKpJ8Ay4A/AQv6frRmZlYtivBtp3pVKpWiqamp1mGYmQ0okhamhxK7NFAuh5qZmVXcgLgcWmmSDgYualf8TEQcUYt4zMysNgqZBCPiDuCOWsdhZma15cuhZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWIX8nuBA0dLSQmNjY63DMLMCmD59eq1DqAnPBM3MrLCcBM3MrLBqlgQlzZB0tqTzJB3USbupki6rZmzt9n+gpFt72ffzkqaVqVvdt8jMzKyvan5PMCK+VesY+ktEzAHm1DoOMzPrWFVngpLOlfSEpAeBj6eyWZImp+3xkn4jaamk+ZJGtut/qKSHJW0qaTNJN0lakP4mpDYzJF0laa6kpyWdlso3kHRbGnuZpCmdxHmIpMclLQL+IVe+V9r/4hRn2zE8ImmXXLu5kkr5WaykbVPfZknnd7Lv4yU1SWpqbW3t+Uk2M7Nuq1oSlDQO+CIwBvgsML5d/YeA64HTI2IP4CDgzVz9EcA04LMR8RJwKXBJRIwHjgSuzA23I3AwsBcwXdJ6wCFAS0TsERG7kq0c31GcQ4GfAIcB44C/yVU/DuwfEWOBbwHfSeXXA19I/bcAtoiI9qvhXgpcHhG7ASvLnaeImBkRpYgoDR8+vFwzMzOrgGpeDt0fmB0RrQCS2l8m/DiwMiIWAETEa6kdwCeBEvCZtnKyJLlzqgfYUNKItH1bRKwB1kh6EdgcaAa+J+ki4NaImFcmzh3J1hZ8Mu3/WuD4VDcKuFrSx4AA1kvlvwTuBKaTJcMbOxh3AlmyBvgZH1zP0MzMqmygPB36FDAS2CFXNgTYJyLGpL8tI6LtYZM1uXZrgXUj4glgT7JkeL6k3tyL/DZwX5pJHgYMBYiI54FVknYHppDNDDsSvdinmZn1k2omwQeASZKGpXt9h7WrXwFsIWk8gKSRktpmqn8gm0Vdk7v3didwaltnSWM627mkBqA1Iq4FLiZLiB15HBgtafv0/h9zdaOA59P21Hb9rgfOAUZFxKMdjPsQ2eVggKM7i9XMzKqjakkwIhaRJYqlwO3Agnb1b5PNon4gaSlwF2mmleofJ0seN6QEdRpQkvSopOXAiV2EsBswX9ISssuWHT6cEhFvkV3+vC09GPNirvq7wL9JWswHLyXfSJbkfllm/6cDJ0tqBrbsIlYzM6sCRfgKXb0qlUrR1NT++RozM+uMpIURUepO24FyT9DMzKziav5l+VqSNBvYtl3x1yPijlrEY2Zm1VXoJBgRR9Q6BjMzqx1fDjUzs8JyEjQzs8JyEjQzs8JyEjQzs8JyEjQzs8JyEjQzs8JyEjQzs8Iq9PcE611LSwuNjY21DsPMamT69Om1DmHQ80zQzMwKq9+ToKSNJJ3Ui36r0+toScsqH1nvSTpR0rG1jsPMzPqmGjPBjYAPJMHcWoEDiqR1I+KKiLim1rGYmVnfVCMJXghsL2mJpAWS5kmaAywHkHSWpGXp74zOBpI0VNJPJTVLWixpYiqfKukWSXMlPSlpeq7PlyXNT/v/saR1Ohl/taRLJD0m6R5Jm6XyuZL+Q1ITcLqkGZLOztVdIqlJ0u8kjZd0c4rj/NzY3Y7DzMyqoxpJcBrwVESMAb5GtqL76RGxg6RxwFeAvYF9gH+RNLaTsU4GIiJ2I1vx/WpJbQvv7kW2+vzuwFGSSpJ2Iluod0La/1o6X9V9A6ApInYB7idbfLfNhyKiFBHf66Df22ntqiuAW1KcuwJTJX24J3FIOj4l1KbW1tZOQjUzs76qxSXJ+RHxTNreD5gdEW8ASLoZ2B9YXKbvfsAPIFtpXtIfgB1S3V0RsSo3zn7AX4BxwAJJAMN4/0rx7b0LXJ+2rwVuztVd/8Hm75mTXpuBxyJiZYrjaWCrFEu34oiImcBMgIaGBq94bGbWj2qRBN/op3HbJ4wABFwdEd+owJidxb0mvb6b2257v24F4jAzs35QjcuhrwMjy9TNAyZJGi5pA+CIVFbOPNJlREk7AFsDK1LdpyVtImkYMAl4CLgHmCzpI6nPJpK26WT8IcDktP0l4MGuDq6behqHmZlVQb/PBCNilaSH0tcc3gReyNUtkjQLmJ+KroyIcpdCAX4EXC6pmexS59SIWJMuMc4HbgI+ClwbEU0Akr4J3ClpCPAO2f26P5QZ/w1gr9TnRbL7eH0WEct7GIeZmVWBIgb+bSdJU4FSRJzSx3FWR8SIykTVdw0NDXHCCSfUOgwzqxH/YkzvSFqYHlbs0oD8rl5RNDQ0+ENgZtaPBkUSjIhZwKzutpf0W2D9dsXH1NMs0MzM+t+gSII9FRF71zoGMzOrPf+AtpmZFZaToJmZFZaToJmZFZaToJmZFZaToJmZFZaToJmZFZaToJmZFZaToJmZFVYhvyw/ULS0tNDY2FjrMMzMqqqaPxfpmaCZmRVWt5OgpI0knZS2D5R0a193Xqlx6oWkKyXtnLb/T63jMTOzzvVkJrgRcFJ/BdJTkuruUm5E/HNELE9vnQTNzOpcT5LghcD2kpYAFwMjJN0o6XFJ1ymtbCvpW5IWSFomaWau/G8l3S1pqaRFkrbPDy5pvKTFkraXtIGkqyTNT2WHpzZTJc2RdC9wj6QRku5J4zXn2p0n6Yzc2BdIOr2jg5L0C0mH5t7PkjRZ0mhJ89LYiyTtm+oPlDS3zLHPlVSSdCEwTNKSVN/teMzMrHp6kgSnAU9FxBjga8BY4AxgZ2A7YEJqd1lEjI+IXYFhwOdS+XXADyNiD2BfYGXbwCnBXAEcHhFPAecC90bEXsBE4GJJG6TmewKTI+IA4C3giIjYM7X7XkpIVwHHprGHAF8Eri1zXNcDX0htPwR8CriNbGX5T6expwDfz/Upd+wARMQ04M2IGBMRR/ckHknHS2qS1NTa2lomZDMzq4S+XFKcHxHPAaTZ4WjgQWCipHOA4cAmwGOS5gJbRsRsgIh4K/UD2AmYCXwmIlrS2J8BPi/p7PR+KLB12r4rIv6ctgV8R9IngHeBLYHNI+JZSaskjQU2BxZHxKoyx3E7cKmk9YFDgAci4k1Jo4DLJI0B1gI7dOPYO9STeCJiZjofNDQ0RLkxzcys7/qSBNfkttcC60oaCvwIKEXEHyXNIEtgnVmZ2owF2pKggCMjYkW+oaS9gTdyRUcDmwHjIuIdSc/m9nclMBX4G7KZWIci4q2UpA8mm/H9IlWdCbwA7EE2Y34r1+0Dx97FMXY7HjMzq56eXA59HRjZRZu2BPSSpBHAZICIeB14TtIkAEnrSxqe2r4CHAr8m6QDU9kdwKm5e21jy+xvFPBiSoATgW1ydbPJZnbj03iduR74CrA/8Ovc2Csj4l3gGGCdLsZo7x1J6/UyHjMzq4JuJ8F0+e4hScvIHozpqM0rwE+AZWT/0C/IVR8DnCbpUeA3ZDOitn4vkN07/GGa7X0bWA94VNJj6X1HrgNKkprJ7rk9nhvzbeA+4JcRsbaLw7sTOAC4O/WDbEZ7nKSlwI68fwbaHTNT/Nf1Ih4zM6sCRQzO207pAZRFwFER8eRAjKdUKkVTU1P/BmZmNshIWhgRpe60HZS/GJO+sP574J46SYB1FY+ZmWXq7gvnlZC+sL5dvkzSbsDP2jVdExF71yIeMzOrvUGZBDsSEc3AmFrHYWZm9WNQXg41MzPrDidBMzMrLCdBMzMrLCdBMzMrLCdBMzMrLCdBMzMrLCdBMzMrrMJ8T3AgamlpobGxsdZhmJm9z/Tp02sdQsV4JmhmZoXlJGhmZoVVs8uhacHd1cCGZKu5312m3VSyRXpPqWJsz6Z9vtSufAawOiL+vRdj9rqvmZn1j5rfE4yIb9U6BjMzK6aqXg6VdK6kJyQ9CHw8lc2SNDltj5f0G0lLJc2XNLJd/0MlPSxpU0mbSbpJ0oL0NyG1mSHpKklzJT0t6bRUvoGk29LYyyRN6SLccyQ1pzj+toNj2V7SryUtlDRP0o6p/DBJv5W0WNLdkjbvoO+/SLpd0rDenEczM6uMqs0EJY0Dvki2ksO6ZAvMLszVfwi4HpgSEQskbQi8mas/AjgL+GxEvCzp/wGXRMSDkrYmW8l+p9R8R2AiMBJYIely4BCgJSIOTeON6iLkVyNiN0nHAv9BtvJ93kzgxIh4UtLeZCvRfxJ4ENgnIkLSPwPnAP87dxynAJ8GJkXEmg7O0/HA8QCjRnUVopmZ9UU1L4fuD8yOiFYASXPa1X8cWBkRCwAi4rXUDrLkUgI+01YOHATsnOoBNpQ0Im3flhLMGkkvApsDzcD3JF0E3BoR87qI9+e510vyFWk/+wI35Pa/fnr9KHC9pC2ADwHP5LoeC/yRLAG+09FOI2ImWYKloaEhuojRzMz6YKA8HfoU2axuh1zZELIZ15j0t2VErE51+RnWWmDdiHgC2JMsGZ4vqat7kVFmu23fr+T2PSYi2mahPwAui4jdgBOAobl+zcBoskRpZmY1Vs0k+AAwSdKwdK/vsHb1K4AtJI0HkDRSUttM9Q/AkcA1knZJZXcCp7Z1ltTpgrmSGoDWiLgWuJgsIXZmSu714XxFmo0+I+moNLYk7ZGqRwHPp+3j2o25mCwxzknxmJlZDVUtCUbEIrJ7fkuB24EF7erfJks4P5C0FLiL3CwqIh4Hjia7BLk9cBpQkvSopOXAiV2EsBswX9ISYDpwfhftN5b0KHA6cGYH9UcDX02xPgYcnspnpBgXAi+17xQRDwJnA7dJ2rSLGMzMrB8pwred6lWpVIqmpqZah2FmNqBIWhgRpe60HSj3BM3MzCqu5l+WryVJs4Ft2xV/PSLuqEU8ZmZWXYVOghFxRK1jMDOz2vHlUDMzKywnQTMzKywnQTMzKywnQTMzKywnQTMzKywnQTMzKywnQTMzK6xCf0+w3rW0tNDY2FjrMMzMypo+fXqtQ+gTzwTNzKywnATNzKyw+i0JSjpN0u8kvSxpWhdtp0q6rEzd6vTaIOnGCsd4oqRjKzlmmf2MlrSsv/djZmY905/3BE8CDoqI5yoxWES0AJMrMVZuzCsqOZ6ZmQ0s/TITlHQFsB1wu6Qz22Z5kjaTdJOkBelvQgd9t5X0sKRmSefnyt+bTaWZ482Sfi3pSUnfzbX7qqQnJM2X9JNyM8zUdoaks9P2GEmPpEV6Z0vaOJWfJml5Kv9FKttA0lVpH4slHZ6L6xZJc1Nc+TvG66R4HpN0p6RhZWI6XlKTpKbW1tZun3MzM+u5fkmCEXEi0AJMBF7OVV0KXBIR44EjgSs76H4pcHlE7Aas7GQ3Y8hWot8NmCJpK0kNwL8C+wATgB17EPY1ZMso7Q40k60+DzANGJvK21avPxe4NyL2Ssd4saQNUt1e6dh2B46S1Law48eAH0bELsArqc0HRMTMiChFRGn48OE9CN/MzHqq2g/GHARcJmkJMAfYUNKIdm0mAD9P2z/rZKx7IuLViHgLWA5sQ5aA7o+IP0fEO8AN3QlK0ihgo4i4PxVdDXwibT8KXCfpy8BfUtlngGnpOOYCQ4GtU91dEbEqIt4Ebgb2S+XPRMSStL0QGN2d2MzMrP9U+3uCQ4B9UuJ6j6T27aIbY63Jba+l/47lULKEeBhwrqTdAAFHRsSKfENJe/PB2Nvet4+3w8uhZmZWPdWeCd4JnNr2RtKYDto8BHwxbR/dw/EXAAdI2ljSupS55NheRLwKvCxp/1R0DHC/pCHAVhFxH/B1YBQwArgDOFUpe0samxvu05I2Sff8JqXjMTOzOlTtJHgaUEoPmSznr/fY8k4HTpbUDGzZk8Ej4nngO8B8suTzLPBqV93S63Fk9/YeJbvfeB6wDnBtimUx8P2IeAX4NrAe8Kikx9L7NvOBm8guo94UEU09OQYzM6seRXTnyuPAIWlERKxOM8HZwFURMbtM2x8AiyLipxXa91SgFBGnVGK8UqkUTU3OoWZmPSFpYUSUum45OH8xZkZ6YGUZ8Azwq44aSfo2sDfZAzpmZlZAg+4HtCPi7PZlks4FjmpXfEP6ikMl9z0LmFXJMc3MrP8MuiTYkYi4ALig1nGYmVl9GYyXQ83MzLrFSdDMzArLSdDMzArLSdDMzArLSdDMzArLSdDMzArLSdDMzAqrEN8THKhaWlpobGysdRhmZhUzffr0rhtVkWeCZmZWWJ0mQUkbSTopbR8o6da+7rBS4/QxhtGSllVorBmSPvBTbWZmVv+6mgluBJxUjUC6I60MYWZmVhFdJcELge3TqgwXAyMk3SjpcUnX5RaV/ZakBZKWSZqZK/9bSXdLWippkaTt84NLGi9psaTtJW0g6SpJ81PZ4anNVElzJN0L3CNphKR70njNuXbnSTojN/YFkk7v6gRIeiC/uK+kByXtkWZ4V0maK+lpSafl2pwr6QlJDwIfz5WPkfRIWi9xtqSNU/lcSRelY3sit3ivmZnVUFdJcBrwVESMAb4GjAXOAHYGtgMmpHaXRcT4iNgVGAZ8LpVfB/wwIvYA9gVWtg0saV/gCuDwiHgKOBe4N63sMJFsgdsNUvM9gckRcQDwFnBEROyZ2n0vJd2rgGPT2EPIVqe/thvn4D+BqanfDsDQiFia6nYEDgb2AqZLWk/SuDT2GOCzwPjcWNcAX4+I3YFmIH8HeN10bGe0K38fScdLapLU1Nra2o3wzcyst3r6YMz8iHguIt4FlgCjU/lESb9NK7B/EthF0khgy7YFbSPirYho+1d9J2AmcFhE/Hcq+wwwLc065wJDga1T3V0R8ee0LeA7aQX4u8lWn988Ip4FVkkam8ZaHBGrunFMNwCfk7Qe8E+8fymk2yJiTUS8BLwIbA7sD8yOiNaIeI20HqGkUcBGEXF/6ns18IncWDen14W58/YBETEzIkoRURo+fHg3wjczs97q6T22NbnttcC6koYCPyJbUf2PkmaQJbDOrExtxgItqUzAkRGxIt9Q0t7AG7mio4HNgHER8Y6kZ3P7u5JsVvc3ZDPDLkVEq6S7gMOBLwDjctUfON7ujFlG21h9HcfMzCqkq5ng68DILtq0JaCXJI0AJgNExOvAc5ImAUhaX1Lb1OYV4FDg3yQdmMruAE7N3U8cW2Z/o4AXUwKcCGyTq5sNHEJ2ifKOLuLOuxL4PrAgIl7uou0DwCRJw9Js9zCAiHgVeDl3v+8Y4P4yY5iZWR3odEYSEaskPZS+TvAm8EIHbV6R9BNgGfAnYEGu+hjgx5LOA94ht7p7RLwg6XPA7ZL+Cfg28B/Ao+me3jP89d5i3nXAf6VLr03A47kx35Z0H/BKRKzt+vDf67dQ0mvAT7vRdpGk64GlZJdI88d7HHBFSvZPA1/pbgxmZlZ9iohax1AxKXkuAo6KiCd70K+B7D7kjul+Z11oaGiIE044odZhmJlVTDV+MUbSwogodaftoLk3JWln4Fayh1Z6kgCPBS4AzqqnBAjQ0NBQdz8xZGY2mAyaJBgRy8m+tvEeSbsBP2vXdE1E7J3rdw3ZVxvMzKxgBk0S7EhENJN9n8/MzOwD/APaZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWE6CZmZWWIP6e4IDXUtLC42NjbUOw8zsfQbTL1l5JmhmZoXlJGhmZoVVV0lQ0mmSfifpZUnTumg7VdJlZepWp9cGSTdWOMZJ6ce6zcxsgKu3e4InAQdFxHOVGCwiWkiL/FbQJLLVKpZXeFwzM6uyupkJSrqCbBWI2yWd2TbLk7SZpJskLUh/Ezrou62khyU1Szo/Vz46LQjcNnO8WdKvJT0p6bu5dl+V9ISk+ZJ+0skMc1/g88DFkpZI2l7SGEmPSHpU0mxJG6e2cyVdIqkpzW7Hp/0/mY/RzMxqp26SYEScCLQAE4GXc1WXApdExHjgSODKDrpfClweEbsBKzvZzRhgCrDNqR/XAAAGG0lEQVQbMEXSVmlB3X8F9gEmADt2EuNvgDnA1yJiTEQ8RbYM09cjYnegGcg/NvV2WtjxCuAW4GRgV2CqpA93tA9Jx6fE2dTa2trJoZiZWV/V2+XQjhwE7Cyp7f2Gkka0azOBLEFCtn7gRWXGuiciXgWQtBzYBtgUuD8i/pzKbwB26E5gkkYBG0XE/anoauCGXJM56bUZeCwiVqZ+TwNbAavajxkRM4GZkK0s3504zMysdwZCEhwC7BMRb+ULc0mxTXcSxprc9lr6//jb9vduu32/W4V9m5lZF+rmcmgn7gRObXsjqaNFch8Cvpi2j+7h+AuAAyRtLGld/jqjLOd1YCRAmlW+LGn/VHcMcH+5jmZmVl8GQhI8DSilB0+WAyd20OZ04GRJzcCWPRk8Ip4HvgPMJ0umzwKvdtLlF8DXJC2WtD1wHNmDMo+S3XM8ryf7NzOz2lGEbztJGhERq9NMcDZwVUTMrnVcpVIpmpqaah2GmdmAImlheiixSwNhJlgNMyQtAZYBzwC/qnE8ZmZWBX44A4iIs9uXSToXOKpd8Q0RcUF1ojIzs/7mJFhGSnZOeGZmg5gvh5qZWWE5CZqZWWH56dA6Jul1YEWt4+jCpsBLtQ6iC46xMhxjZTjGyugsxm0iYrPuDOJ7gvVtRXcf860VSU2Ose8cY2U4xsooUoy+HGpmZoXlJGhmZoXlJFjfZtY6gG5wjJXhGCvDMVZGYWL0gzFmZlZYngmamVlhOQnWiKRDJK2Q9HtJ0zqoX1/S9an+t5JG5+q+kcpXSDq43mKUNFrSm5KWpL8rahjjJyQtkvQXSZPb1R0n6cn0d1ydxrg2dx7ntO9bxRjPkrQ8reZyj6RtcnX1ch47i7FezuOJkppTHA9K2jlXVy+f6w5jrKfPda7dkZJCUilX1rPzGBH+q/IfsA7wFLAd8CFgKbBzuzYnAVek7S8C16ftnVP79YFt0zjr1FmMo4FldXIeRwO7A9cAk3PlmwBPp9eN0/bG9RRjqltdJ+dxIjA8bf+v3H/rejqPHcZYZ+dxw9z254Ffp+16+lyXi7FuPtep3UjgAeARoNTb8+iZYG3sBfw+Ip6OiLfJ1ig8vF2bw4Gr0/aNwKckKZX/IiLWRMQzwO/TePUUY7V0GWNEPBsRjwLvtut7MHBXRPw5Il4G7gIOqbMYq6U7Md4XEa3p7SPAR9N2PZ3HcjFWS3difC33dgOg7aGMuvlcdxJjtXTn3x6AbwMXAW/lynp8Hp0Ea2NL4I+598/xwcWA32sTEX8hW+j3w93sW+sYAbZVtvDw/ZL274f4uhtjf/Ttib7uZ6ikJkmPSJpU2dDe09MYvwrc3su+vdWXGKGOzqOkkyU9BXyXbNHwbvetcYxQJ59rSXsCW0XEbT3t255/Mcb6w0pg64hYJWkc8CtJu7T7P0zrnm0i4nlJ2wH3SmqOiKdqFYykLwMl4IBaxdCVMjHWzXmMiB8CP5T0JeCbQL/dR+2tMjHWxeda0hDg/wJTKzGeZ4K18TywVe79R1NZh22UrXg/CljVzb41jTFdilgFEBELya7L71CjGPujb0/0aT8R8Xx6fRqYC4ytZHBJt2KUdBBwLvD5iFjTk741jrGuzmPOL4C2WWldncec92Kso8/1SGBXYK6kZ4F9gDnp4Zien8f+vsnpvw5v/K5L9gDBtvz1xu8u7dqczPsfOvll2t6F99/4fZr+uYHelxg3a4uJ7Ob288AmtYgx13YWH3ww5hmyhzk2Ttv1FuPGwPppe1PgSTp4QKBK/63Hkv2j97F25XVzHjuJsZ7O48dy24cBTWm7nj7X5WKsu891aj+Xvz4Y0+PzWNHg/dej/9CfBZ5IH9pzU9l5ZP8HCzAUuIHsxu58YLtc33NTvxXA39dbjMCRwGPAEmARcFgNYxxPdl/gDbKZ9GO5vv+UYv898JV6ixHYF2hOH+pm4Ks1jPFu4IX033QJMKcOz2OHMdbZebw099m4j9w/7nX0ue4wxnr6XLdrO5eUBHtzHv2LMWZmVli+J2hmZoXlJGhmZoXlJGhmZoXlJGhmZoXlJGhmZoXlJGhmZoXlJGhmZoXlJGhmZoX1P/HNEheeH/GmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sttr[\"sttr\"].plot.barh(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Going back to the document level\n",
    "\n",
    "The complexity measures can be accessed via the `Corpus` object as well as via a `Document` object – even more flexible in the case of `Document`, e.g. using the `bootstrap()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These are the TTRs of the first 10 windows: [0.3765, 0.3865, 0.375, 0.353, 0.3675, 0.364, 0.357, 0.383, 0.3855, 0.3595]\n"
     ]
    }
   ],
   "source": [
    "results = list(document.bootstrap(measure=\"ttr\", window=2000))\n",
    "\n",
    "print(\"These are the TTRs of the first 10 windows: {}\".format(results[:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you put those in a pandas `Series`, you could plot them and see how the TTRs differ over the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f0fa7e63748>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztvWmwZNV17/lbOd15qlsTUCWqBIWgBomhDBhLAtlCwJMNki13I8k2HxSNiRaWXtuOJxytULjVfo6w4jXPdjRhh0K22vFswFi2n7HBIMn2kwyooAqEKIqCRzFIVFHUxK0735vT6g+ZO2vfc885eTLz5HRz/yIqKvPktO/Jk//zP2uvtbaoKg6Hw+HoDRLtHoDD4XA4WocTfYfD4eghnOg7HA5HD+FE3+FwOHoIJ/oOh8PRQzjRdzgcjh7Cib7D4XD0EE70HQ6Ho4dwou9wOBw9RKrdA/Cyfv163bZtW7uH4XA4HF3Fs88+e1pVN1R7XseJ/rZt2zhw4EC7h+FwOBxdhYj8OMrzIoV3RORmEXlFRI6IyD0hz/slEVER2Wtt+53y614RkZuifJ7D4XA4mkNVpy8iSeA+4EbgKLBfRB5W1Zc8zxsBvgg8bW3bCdwO7ALOB74rIpeoaiG+P8HhcDgcUYni9K8Gjqjq66qaBR4EbvN53v8N/AGwZG27DXhQVZdV9Q3gSPn9HA6Hw9EGooj+BcBb1v2j5W0VRORKYKuqPlLra8uvv1NEDojIgVOnTkUauMPhcDhqp+GUTRFJAPcCv1Xve6jq11V1r6ru3bCh6uSzw+FwOOokSvbOMWCrdX9LeZthBNgN/A8RAdgMPCwit0Z4rcPhcDhaSBSnvx/YISLbRSRDaWL2YfOgqk6r6npV3aaq24B9wK2qeqD8vNtFpE9EtgM7gGdi/yscDofDEYmqoq+qeeBu4HHgMPCQqh4Ska+W3XzYaw8BDwEvAY8Bn3eZO83njTfe4PTp0+0ehsPh6ECk09bI3bt3r7rirMb4oz/6Iy644AI+9alPtXsoXcPS0hLLy8uMjY21eygOR12IyLOqurfa81zvnTVINptlenq63cPoKv71X/+Vv/iLv2j3MByOpuNEfw1SKBSc6NfI9PQ0U1NTFArtjT6eOnWK3//93+fs2bNtHYdj7eJEfw2Sz+eZnZ1tu4B1EwsLCwDMzMy0dRxTU1PkcjmmpqbaOg7H2sWJ/hpDVStiPzs72+bRdA+dIvr5fH7F/w5H3DjRX2PYYtFuAesmjOi3OyzmRN/RbJzorzFssWi3gHULxWKRpaVSy6h27zNzleZE39EsnOivMew4vnP60VhcXKzcbvc+c07f0Wyc6K8xnNOvnfn5+crtdou+c/q9xz/8wz/wne98p2Wf13ErZzkaw8X0a8fE8zOZTNtPlM7p9x5Hjx5lYGCgZZ/nRH+N4Zx+7RjR37RpE+1u7e1Ev/dYXl5u6ee58M4aw4jF2NiYc/oRMaK/efNmlpaWyGazbRuLCe/kcrm2jcHRWrLZbOUYbAVO9NcYRvTXrVvHwsKCE48ImB/ceeedB7Q3LOacfm+hqmSzWRYXF2lVHzQn+msMW/TBxfWjsLCwQF9fX2WftTMs5kS/t8jlcqgqqlpJG242TvTXGEYsJicnARfXj8Li4iKDg4OMjo4C7d1nLnunt7BDiXbqcDNxor/GcE6/dubn51eIfjv3mRP93sKexG1VXD+S6IvIzSLyiogcEZF7fB6/S0QOisjzIvKEiOwsb8+IyDfLj/1IRG6IefwOD17Rd06/OgsLCwwODpJMJhkeHnbhHUfLsJ1+x4i+iCSB+4BbgJ3Ap42oW9yvqntU9XLga5QWSgf43wBUdQ9wI/D/lBdSdzQJIxb9/f0MDQ050Y+AEX1of9aTE/3ewnb6nRTeuRo4oqqvq2oWeBC4zX6Cqtq/kiHATEPvBP61/JyTwFmg6soujvox4YFUKtV2AesWFhYWKsUxo6OjLqbvaBmdGt65AHjLun+0vG0FIvJ5EXmNktP/Qnnzj4BbRSRVXhj9KmBrY0N2hGHEIpVKMTo66kS/CrlcjlwuV3H6Zp+1axlR5/R7i44M70RFVe9T1YuALwFfLm/+c0oniQPAHwJPAatW9hCRO0XkgIgcaHdFZLdjxCKZTFZca6etg9xJmEvqoaEhoBTeyeVyLUuf8+Kcfm/RqeGdY6x051vK24J4EPgEgKrmVfX/UNXLVfU2YBz4n94XqOrXVXWvqu7dsGFD9NE7VpHP50kkEiQSCcbGxshmsy0v8+4mjLuyY/rQvglw5/R7C+P0R0dHO0r09wM7RGS7iGSA24GH7SeIyA7r7seBV8vbB0VkqHz7RiCvqi/FMnKHL/l8nlSq1FKp3QLWDZgOm3Z4B9qXtulEv7cwhmxsbKxl4Z2qDddUNS8idwOPA0ngz1X1kIh8FTigqg8Dd4vIR4EcMAXcUX75RuBxESlSujr41Wb8EY5z2KJvC9imTZvaOayOpdOcvuu901tks1kymQyDg4MtWxc5UpdNVX0UeNSz7SvW7S8GvO5N4H0NjM9RI87p14ZX9IeGhkgkEi6842gJy8vLZDIZBgYGePvtt1vymS5nfo1RKBRIJpMADA8PIyJO9EMwot/f3w9AIpFgZGTEhXccLSGbzdLX18fg4CALCwstSbpwor/GsJ1+uwWsGzA5+onEuZ9CO+sb7OydegWgUCjwN3/zN5w4cSLOoTmagHH6g4ODFAqFloT1Olr0i8Ui3//+91csZ+cIxxZ9aH+FaaezuLhYSdc0jI2NtT28AyvXO66FmZkZXnrpJd58882YRuVoFsbpm+LAVkzmdrToHz9+nH/7t3/j4MGD7R5K1+An+i68E4zdgsHQrgItVaVYLNLX1wfUH+Ixr3OTwZ3P8vJyJbwDrcnV72jRN5enZ86caflnz8/Pd2VRk1f0211h2umYDps2o6OjFItF5ubmWjoWu2+Sfb9WjNi3cwUwRzTs7B1wTr9tov+Tn/yEe++9lxdffLGlnxsHfqJfKBRciCwAu++OwWQ9tTosZsI5TvR7Bzt7B5zoc/LkSQBOnz7dss9cXl7m7//+7ykWi5w9e7ZlnxsXfuEdcH31/VBV3/BOu1Jdjcg3Gt5xot89uPCOhapy4sQJRITZ2dmWtRJ49NFHmZ6eRkRaVhYdJ0Gi7+L6q8lmsxSLRd/wDrR+n8Xl9F1MvzsoFAoUCgXn9A1zc3MsLi5y4YUXAq0J8bz44ou88MILfOhDH2J4eLhrRd/k6UP72wp0Mt7CLMPAwACpVKrl+8zF9HsL8/309fWRSCTo7+/vbdE3oZ3LLrsMaH6IZ3p6mkceeYQtW7Zw/fXXMzAw0LZOi41QKBRWOP3BwUFSqVTXOP35+fmWlaObH5g3ZVNE2pLq6sI7vYWJXpjve2BgoLfDO2YS97LLLkNEmur0i8ViJY7/yU9+kkQi0bIvIG684R0RqauvfrFYrDtPvBG+/e1v89d//dct+awgpw/tSXV1E7m9hfl+MpkMUDoOe1r0T548yfDwMCMjI0xMTDTV6b/88sv8+Mc/5uabb66sLdvf39+VTt8r+lCfgP3jP/4jDz74YJxDi8TU1BSzs7Mt+Sxvh02bdqyg5Q3v1BuTdzH97sDr9E0rhmbTsaJ/4sSJSmfIycnJpjp98+M2oSRo3aVWnBSLRYrF4irRr8fpv/POOy1rAGUzOzvL0tJSS+oKwpz+6Ogoc3NzLb3aceGd3sLr9AcGBnpX9IvFIqdOnWLjxo3AOdFvlhAYcTc/Nii5rW4TfXupRJuRkRHm5uZq2n+zs7MsLCy0VDhUlbm5OYrFYktc6sLCAolEovKjs2lHqqsL7/QWLqZvcebMGQqFQsXpr1+/nnw+37TL7cXFRQYGBhCRyraBgQFyuVxb4tr1Yi+KbjM0NESxWIx8QOXz+Uroo5W1CktLSxWha0VozeTo29+74bzzzgPgyJEjTR+HwWXv9BZG9O2YfjabbXqH1UiiLyI3i8grInJERO7xefwuETkoIs+LyBMisrO8PS0if1F+7LCI/E6UzzOZO8bpr1+/HmheBs/S0lLlh2YwebPd5PaDnP7w8DBA5Kpcu/1AK0Xf/txWiP7i4qJvaAdKor9582aee+65lrWwiDtPv1AoUCwW4xmcI3bslE2gZQVaVUVfRJLAfcAtwE7g00bULe5X1T2qejnwNeDe8vZfBvpUdQ9wFfDrIrKt2meaoiyzXm6zRd84fRvzw+umyVx7UXQbk5IYtZeMHdJopejbE7itONn6VePaXHnllbzzzjscP3686WOB+GP64Nx+J+N1+q0q0Iri9K8Gjqjq66qapbTw+W32E1TVDnwOAcYaKTAkIilgAMgCVYOkJ0+eZHJysuJYBwcH6e/vb9pk7tLS0irRX4tOv9NFv9VOf2FhYVWOvs2ePXtIpVI899xzTR8LONHvNbLZLMlkcoXOQQc4feAC4C3r/tHythWIyOdF5DVKTv8L5c3fAuaB48BPgP+iqu9W+0A7c6f83kxOTjbV6bvwzjmM6A8PD7fN6bdC9Ofn51ed7G36+/vZtWsXBw8ejCSeR48e5aGHHqo7pGLCO0YIGg3vgBP9TsY0WzO0qtNmbBO5qnqfql4EfAn4cnnz1UABOB/YDvyWiLzX+1oRuVNEDojIgVOnTnH27NlKPN+wfv36pjl9v/DOWhL9/v5+EolEZKc/OztLOp1m06ZNa9bpm4ntsPAOlEI82WyWQ4cOVX3PN954g8OHD9c9dvv7S6fTdWcw2a9zufqdi1lAxdBJ4Z1jwFbr/pbytiAeBD5Rvv0Z4DFVzanqSeBJYK/3Bar6dVXdq6p7Taqc7fShlLbZjMZrquo7kdvNMX2v6IsIQ0NDNTn9kZERxsfHW+70x8fHgeafbM33Wk30t27dyvr16yOFeMx7NlpUlUqlGnL6uVyuMq/jnH7nYnrpGzopvLMf2CEi20UkA9wOPGw/QUR2WHc/Drxavv0T4GfLzxkCrgVeDvswc6D7OX2Iv/Ha8vIyqho4kbsWnD6UQjW1OP3R0VEmJiZYXFxsWYfTubk5RkdHyWQyTT/ZhhVm2YgIV155JUePHq1klQVh9lO9Yl0oFBAREokEqVSq7nThXC5Xmatwot+5mLbKBnOF13anr6p54G7gceAw8JCqHhKRr4rIreWn3S0ih0TkeeA3gTvK2+8DhkXkEKWTxzdV9YWwz8vlcmQymYrjMzQrg8eIi1f0E4kEfX19a0r0a3H6o6Ojle+gVW5/dnaWkZGRlrTAiCr6AO9///tJJBJV3X6jom93SG00pu9Ev/PxxvShNf13VquDD6r6KPCoZ9tXrNtfDHjdHKW0zcjk83k2bty4qmBmYmICEYld9M0O9pvQ67ZOm2GiPzQ0xDvvvFP1PVS1Ir626HvDbc1gbm6O4eHhjhP9oaEhLrvsMl544QU++tGP+u5fiMfpm/dOpVINxfQnJycBJ/qdTDabXWVuW9GKoeMqcnO53KrQDpR+BBMTE7GHd4zoe2P60H39d6qJfpR1f+fn5ykWiy13+tlslmw2y8jISEtOtkFtlYO48sorWVxc5OWXg6OTRvQbienbot9ITN/8XW4it3Npl9PvONEvFouBrrIZjdeCwjvQfZ02g9owQCm8E6UVg0nXHB0dZXBwkHQ63RLRN+marXL6JtQVlrJps337dgBOnToV+Bwz5kacfhzhnVwuV7mCcU6/c/Fm70CPOn1YnbljaEbjtWrhnbXi9KPm6hvxHRkZQURalsFjf26rwjvpdJp0Oh3p+SJSdVxxxPQbdfqFQqGSmCAiTvQ7FFVdlb0DrWmv3JGi7xfegeY0XgsL73Rbp82gNgwQvRWD7fSBlom+GZdx+o3sd1WtGtaIkqPvJaro1xtSsWP66XS6LtE3n21OaE70O5NcLoeq+jr9paWlpvZM6jjR37hxY+AldzMyeJaWlkgmk76Oz3wBrWq41ShxOP2ZmZlKXj+0TvS9Tt8sWl4Pzz77LH/4h38Y+vpqfXf86O/vD0xfVdWOyN6xj4FMJuNEv0Px9t0xmGOymVe6HSf6QZkR0BzRNy0Y/NrrDgwMUCgUumYyzIiG398S1embzJ1EonRojI+Ps7S01PRwy9zcHMlkkv7+/oYL406cOMHCwkLoCa5e0Q8ak3FuEE94J5lMNuz0M5lM1xy7vYa3w6ahFa0YOk70wzCN1+J2+kFXFt1Wleu3VKJhYGAgUisGk6NvaFUGz9zcXGUewXwf9e538ze2UvTt7e2cyPWKvnP6nUmQ029FK4auEn0zsRjnGqp+fXcM3dZ/J0z0o7ZiaJfoz87OVkJQjZ5szfER9rfOz8/HKvp22CeOlM16e+840e8Oqjn9ZmpOV4k+lHZK1MrSKPh12DR0m+jbTtGPalW5qlrpu2OYmJgASguWNxPj9KFx0TdOP+iqxq4JqIW+vr5Ioh9n9k6t80neiVwX3ulMvEslGpzT9yHulKYwp99t/XfCnD6U4vph4Z3l5WVyudwKp9/f308mk2mL069nv5t1diHY6duZQrVgJnL9JoibEd4Bap7MXqsTuWttBTDvougGF9P3IW7R9+uwaWg0ttxqqol+taZrdgaNQUSYmJho2vrEUHKnS0tLFRFuZL8vLi5WitSaIfqAbwZPM5x+Pe+1FsM73/ve9/iTP/mTrlqvuhpBTj+TyZBIJFx4x2ZwcJDl5eVYFg8uFossLy/3REwfqrdi8OboG8bHx5sa3jEiHEd4xz6pVRP9qC0YDGHjMj9iEWkoT9/r9Gt9r7WYp3/y5ElOnz7N4cOH2z2U2AiayBWRphdodZ3omx9qHEIc1oIBSl+IiKwZ0TetGILENEj0x8bGOHv2bNPqFbyin06nSSQSdYm+PcnfLKfvNy67P79z+vFiBHDfvn1dUzNTjaDwDjS//07Xib6JecUxmRtWjQtU0gfXSninWq6+X3gHSpO52Wy2aQei3XcHzrU8qOfzzHtNTk4G/p3z8/MVR1ULUZx+o6Lvdfq1vpc3pl8oFLo+Hr6wsEAikeDYsWMcPXq03cOJBdNsLag+yDl9izgnOsL67hi6qRVDFKcPwaI/MzPD4ODgqvdodtqm1+lDePVrlPfavHlzqNMfGhqqFKBFpVpMv6+vr+72Caq6qrUyNO70ofubri0sLLBz5076+/vZt29fu4cTC37N1gwd4fRF5GYReUVEjojIPT6P3yUiB0XkeRF5QkR2lrd/trzN/CuKyOWNDNi41ThEv1p4xzy21px+kBiaalwvQaIf16X27OwsiURihfNuxOmbRXiC5i9M3/5aqeb0+/r66i6qMm48LtE3Tt/e1o2oKgsLC4yNjXHVVVdx+PDhpqcPtwK/ZmsG4/QLhULlX5xXa1UXURGRJKUVsG4EjgL7ReRhVX3Jetr9qvqn5effCtwL3KyqfwX8VXn7HuC/q+rzjQy4GeGdaqLf7K53cRGH0/fG88Ff9I8dO8YDDzzAL/7iL/Le965a674mjPO2L3Xr7bRp8v2HhoYq8xfe77eZop9Op+sat7dZnukFVY/op1IpRKTyHt3s9E2K7ODgILt37+YHP/gBzzzzDDfddFO7h9YQ3qUSbUzCxe/93u9VtmUyGW6//fZKi+9GiOL0rwaOqOrrqpqltPD5bfYTVHXGHjPgZwE/XX5tQ8RZvFAtpm8+r1vCO9WKs0wrhqATprcwy2D64RjRf/fdd7n//vuZn5+vum5sFPyuMOq9wjKiH9Zgrl7RNz/SoInc/v7+WBql2f/XE9M3Yr8Wwjv2Cmejo6Ps2rWL5557rmXrNjeLMKe/d+9efu7nfo6PfOQjlX9DQ0M88sgjsaStRhH9C4C3rPtHy9tWICKfF5HXgK8BX/B5n/8VeKCeQdokEonY3HeU8E43LaRSzembVgx+Tj+fz7OwsODr9OFct835+Xn+8i//ElVFRGL5HuxqXENQ9auqsm/fvkqmkRdT5BU0aW2Kt2pN14TS/gsaV6PhHfNjbnQiN5fLrVnRB7j22mvJZrP88Ic/bOewGibM6Y+OjvLBD36QD3/4w5V/t9xyC2fOnIllTiO2iVxVvU9VLwK+BHzZfkxErgEWVPVFv9eKyJ0ickBEDoStTGSIK491cXGRdDpd1R0vLi52RapYNdEHAvvvmKyXMNE/ffo0DzzwALOzs3zmM5+JbZLbrsY1BLW1np6e5vHHH/f90Zv1fW3R9/6tpld5PU4fgk1AoxO5zun7413W8vzzz+c973kPTz/9dFdnJfktlRjGjh07uOSSS/j+97/fcO+xKKJ/DNhq3d9S3hbEg8AnPNtuJ8Tlq+rXVXWvqu7dsGFD1QENDQ3F5vSrLZcXlrFRK4VCoWmXpaoaSfSDqnKDcvQNpkDr7bff5pd+6ZfYsmVLLKGvQqHAwsLCKhHu7++nUCisEj07xOTFFO2FhXfqzdG3xxUU3unr6yOZTNY1cRok+vUUZ5nXroWJXL8F7K+66irOnj3LiRMn2jWsQPL5fKT9HRbeCeKmm26iUCjw3e9+t97hAdFEfz+wQ0S2i0iGkoA/bD9BRHZYdz8OvGo9lgD+F2KI5xviaroW1nfHEGdV7re+9S2++c1vNvw+fhSLRVQ1kuiHOf2gJmSTk5MA3HLLLVx66aVAPKllfumaEDxpajI3/NZKtgXdLBfoPcE1S/QbdfrNCO+shYlcP9E3xqQT4/qPPfYY999/f9XnhYV3gli3bh0//dM/zQsvvMBbb71V/QUBVBV9Vc0DdwOPA4eBh1T1kIh8tZypA3C3iBwSkeeB3wTusN7iw8Bbqvp63aP0EGd4J2wSF+IT/VdffZWXX36Z06dPNyVUFLYouo2J6XvHUM3pf+ADH+Bzn/scP/VTP1XZFofTDxLhoKZrxun7rZVsn7hMCmgrnL65IumEidx2xvQLhQLf+MY3eO2112J7T1OYZbviTj6ZnTlzpup6HyYNs1bRB/jQhz7E6Ogojz76aN3hrUgxfVV9VFUvUdWLVPU/l7d9RVUfLt/+oqruUtXLVfUjqnrIeu3/UNVr6xpdAEb0GxXPWsI7jYhbPp/nscceA0pfeDMmhsOWSrSxUxltZmZmSKfTgQdiOp1my5YtK7bFIfpBVxhBTddM47elpaVVn+0VdL/5izhE3+sw7eZZqVSKYrFY8w/Se9LuRtGfmpri2LFj/PjHP47tPc1iN3Y6byfPVSwuLladAwxrwVCNTCbDjTfeyDvvvMN3v/vdukJ3XVeRCyXRV1Vf8czn8zzzzDORdkYtTr8Rod63bx/vvvsuV1xxBUCsi8AYwhZFtwnK1Z+dnWV0dNS3LDyIOLKo6gnvmDF6QzzeE0iQ6CeTybpcFvhnFXlFH+pvn2C+P/N/Pe/jPXG0ShzNVVi11dlqwW+Fs04WfVNUFaY/QR02o7Jr1y4+8IEP8IMf/ID77ruPgwcP1mSAu1L0w6pyX3/9df75n/85UmpTK2L6MzMzfP/73+fSSy/l8stLxcjNFP0oMX1YPcEZVJgVxsDAANlstqHcYbMvvCmUQaJ/9uxZLriglDHsJ/qpVKryY/Kbv5ifn2d4eLimk5t3XN6sIjPG/v7+isuu1YF5vz8RqStUZDt9EWlp0zUn+ud0IkwvGnH6UPpeP/GJT3DHHXcwMDDA3/3d3/Fnf/ZnkV/flaIf1n/HHHhPPfVU6ERPPp8nn883Pbzzne98B1XlYx/7WMWBtlP0g/LXjdOvhTjmO4L64Pjt90KhwOzsLNu2bUNEVom+vc4u+C8aU29hlndc9rEVh9P3TuRCfevk2tk7QEsXRzeT7L0q+rlcrvJ9hf0mGnX6hm3btnHnnXdy22231bTeRVeLvl8WytmzZxERlpaWQt1+lGpcKMWyU6lUXeGdN998kxdffJHrrruOiYmJith0mtNfXFwM7LsTRhzreQZ9rp/Tn56eRlVZt24dExMTq9I2vYI+NDRELpdbIQ5xib49rjjDO/b316jTB1rq9I3wxHl8+4l+MpkkkUh0nOjbJjSK6Nfr9G1EhMsvv5zf+I3fiPyarhZ9P6c/PT3NxMQE73vf+9i3b1+gWEepxjXUW4T0ne98h7GxMT74wQ8CpRNIf39/W0XfL5XxqaeeolgssmfPnpo+s5rTf/nll/nWt74V+h5BIpxMJlf1sTFXcRMTE0xOTvqGd+wTiF+BVr3VuAY/0fcL78Qh+rWmf5paDa/ot8rpm+9nfn4+lsKpYrHI4uKibwvsTlwrwP4dhM11BS2K3gi1nEC6UvTDYvrT09OMj49zww03hLr9KM3WDPX0gVleXubtt9/myiuvXPEjHBkZaavoe1sxLCws8Mwzz7Br1y42btxY02dWE/1XX32VQ4cOhU4y+VXj2u/vJ/rj4+OsW7eOd999d8V7e08g3quaYrHoWwhWC1Gdfq1CG0d4p1AooKorjrdWrp5lJtlNZ8xGMcdVkOjHfTL74Q9/yJEjR+p+fa1OP07Rr4WuFH2zFFxQeGdsbIzNmzdz6aWXsm/fPt8voFbRr9Xpm0ZkmzZtWrF9ZGQk1pinIarow8oJzqeeeopsNsv1119f82dWa35nPiNsondpaSlwMRNvTrwJ3Y2OjjI5OUkul6ucQLPZLMvLy75O3z7BqeqaDe+Y57YjvJPNZllYWGDz5s1APHF9v8IsQzP+rn//93/nqaeeqvv1tkY0cyK3UbpS9MG/GjSXyzE/P19pBXzDDTewvLzMD37wg1WvjxrTh/hFv51OH86J/vz8PM888wy7d+8mSvsLL9Vi+uaHH+TI/MIRNn6iPzY2RiKRqFQImxCPX/69N7zTaI6+GROsDu8kk0lSqVTDE7mNiL7dS9/QKtE38XxTy9GNop/NZqsWVoURVfSd068Tv2pLc+CNjY0BJcHduXMnTz/99KovodaYfq3hnRMnTpDJZCpjMQwPDzM7Oxt7VW4tom/CO08++ST5fL4ulw/V1xA230+QcFUbs5/omxO6V/T9irxaJfp2SX2jKZt2FlMqlarpfexVswytEn2TudPNom+uHOt9X7s5XDWnn0wmq9bUNIuuFn1vWMGIvhEGgOuvv55sNsv+/ftXPNd8KVHOtvVM5J48eZKNGzeuygcfGRmpxJbjJGobBjgn+vv372fPnj2sX7+jQMj7AAAgAElEQVS+rs80awj77RvTwhiCBdBPpGy8+90W/dHRUVKpVKjTNzn75rE4RN8cL96UTXMyaCS8k0wmVxwv9Tp9b0y/FRO5Zr7FiH4cV7PVRD/Ov0tVK2Lv19cpCouLi2QyGYaHh6s6/XaFdqCLRd+v06Y90WfYuHEj559//qp+IKYaN8o6qbUWIakqJ06cWBXaAZqWq19reKdYLFIoFPjwhz/c0OcGNV3LZrOVMQUJVxTRN446n88zOztb+W5FpDKZC8HtHIaHhyvHiRH9RrJ3TB+YIKffSHjH+911U0z/7NmzpFIpJiYmyGQyXef07f1cb4jHFHtWq1QPWx+3FXSt6PuFd8xEn/eHv23bNo4ePbrCGUTpu2OotRXD7OwsS0tLvtkwzRb9KJeMxum+//3vr4RJ6iXI6ds/+iBHFiW8Y5bL87uKs9M2zTq73u/UzlSam5sjk8k07LK8Yac4RN+vLXZcMf18Pt/03vPmKsz8/uISfZO04SXurCT7veoV/YWFhYroO6ffBAYHB8nlcisEZXp6mtHR0VXufdu2bRSLxRXtSKP03THUWnlq+nx3qtPfsmULW7ZsqTuWbxPkauwTcr3hHbPfl5eXfa/iJicnmZqaolgsVtI1veE0u/+OacHQKF7RN0sl2n9LPSmb3hN2HOGdVvXUt0NvQWs21EpQjj7E7/Tt/dNIeGdwcLCq6DunXyd+BVomR9/Le97zHkSEN954o7KtFqdfaysGk7nj5/SbVZVbi9OfmJjgc5/7HBMTEw1/bhSn30h4B0rfVZDoF4tFzp4967vkIqwU/Uarce1xdbLT98b0ofktC0xmFcQn+vPz81VFP66EiGY4/aCx1dNLP07WlOjbbsOmr6+P888/f0XL1yjN1gy1hndOnDjB6Oio7/unUikGBgZiz9U3olFvI7F6CRL9KE4/SngHSvt9amqKRCKxQtjXrVsHlJxZUDsHM6lWKBRaIvqJRAIRaUtMP8zpN1P0l5eXWVxcrJiIuETfrwWDIZPJoKqxLBQO5/aPCRnWczIxmjI4OEixWAzc5/WsmhUnXSv63qpc04zLmyJp2LZtG8eOHat8Ec0M75jMnSCakasfZanEZjAwMLCi0ZQhTqe/uLjI9PR0JUffYKdtBgm6fZw02oLBHpcRfVVdIfr1dsc02Ts25n2iCpDfSbQVou+9ChseHiabzTb8mWGiH/cVjDkWzzvvPPL5fE0NzIBKq3cT3oFgveiKmL6I3Cwir4jIERG5x+fxu0TkoIg8LyJPiMhO67H3i8gPyitrHRSRaEpbBW/TtZmZGVQ1UPS3b99OsVjkJz/5SeULakZ4p1AocOrUqbaIfjvyfoMKtKJM5EaN6ZvwjvcqbnBwkP7+fk6ePMni4mKo6E9PT7O0tBSL07d76udyOVR1xeV6rfn1EBzeUdXIk7Dtcvp+og+N5+pXc/oQ399l3ue8884Dao/rm3bbJrwD4aLf0eEdEUkC9wG3ADuBT9uiXuZ+Vd2jqpcDXwPuLb82BfwlcJeq7gJuAGKZUfKGd/yyO2y2bt1KIpHgzTffJJvNUiwWa3b6UcI7Z86coVgs+k7iGpoh+n7hgVYQdIDbk6bVwjtRYvpTU1OrvlsRYXJyshK2CwrvwLnJ9bjCO8vLyysW8rGPpXrWyfWbyK21eVsjE7nFYpGnn366rqUevaIfR7JCPp8nm822TPRtpw+1x/Xt9NKw9iSmHqDTnf7VwBFVfV1Vs5QWOL/NfoKqzlh3hwBzPfox4AVV/VH5eWdUNZYgnOkWaXas30SfTSaT4YILLuDNN9+sqRoXzuVmR3H6YZk7BpPSFmcaXTvDO7D6ALfbYVQL71SL6c/Ozq54P5vJyclKrn6Y049b9M2P16+kvt7wjp/TN49FwexP++QRNQzyk5/8hMcee4zXX699KeuzZ8+STqcrAh2H0w9rtgbNc/oTExP09fXVLPp2L6+w9iTmO+popw9cANhLrx8tb1uBiHxeRF6j5PS/UN58CaAi8riIPCci/8nvA0TkThE5ICIHTp06FWngphrUK/phC4Fs27aNt99+u3JVEFX0zXOjOP2TJ0+SSCRCq1xHRkZi60RoaLfo+4V3jEjXG94xbR6MYPuJvpnMBX+n3yzRh9IVSFyiHzSRC9FF3/Qxqmc9WfP9hR2TpleSFztHH+IR/bDCLGie6GcyGdavX19zeMeMt1p4x2hIp4t+JFT1PlW9CPgS8OXy5hTwQeCz5f8/KSI/5/Par6vqXlXdW0vjL7sqd3p6muHh4VDh27ZtG6rKK6+8AtQu+lFE+uTJk6xfvz40vt6MXP12ib6fqzEtGIaHh0kmk3WHd0SE/v5+jh8/DuCbYmoXl/mJfiaTIZVKNU30/cI7cU7kQvQce+8CKlC76IddzX7729/mG9/4xqqJZe98y+DgIIlEoqtE3+xjI/r1Ov1qE7nGcNa6Sl2cRBH9Y8BW6/6W8rYgHgQ+Ub59FPi+qp5W1QXgUeDKegbqh12VG5Sjb2Pi+ocPHwaiddi0PyuK6J84caJqX/pm5Op3ktM3LRjMSTgsvCMioa0w+vv7Q0N3RvRFxFcgzPoBxpHHlb0DwU6/nn43cYR3/DqWxun0T58+zYkTJzh2bOXP3yv63jUb6sH8rlvp9EWEZDLJ5OQks7OzocuterGdfjKZJJPJ+O7LmZlSJDwo4aQVRBH9/cAOEdkuIhngduBh+wkissO6+3Hg1fLtx4E9IjJYntS9Hnip8WGXsIU4KEffJp1Os2XLlkpHwFqcvl+vHy9LS0tMT09XFf215PTT6TTJZHLFvjE/2OHh4VABNOu5htUWmO8omUz6unQT3vFbZ9dgXmd+kI3SrPBOkNOvJaYfdOKodhKK4vTNd3zw4MHKNnO14/3tNZqrH9Xpx1VpbK6STHIA1JbBs7i4WLkyheBwsLcTcDuoKvqqmgfupiTgh4GHVPWQiHxVRG4tP+3uckrm88BvAneUXztFKZNnP/A88JyqPhLX4I3oq2olj7sa27Ztq9yuRfT9ev14Ceqh76URpz8/P1+5UrFpl+j7ddq0G5uFZbKE9dI3mB+RHTO26evrY3h4OHR9X+Pu4wjt2GOywzudMpHr3Z8iEqllgRHZMNE3jx06dKiShBB0FdZohprtnP1ohtM372nm42oVfZNcAsHh4JmZGTKZTOfH9FX1UVW9RFUvUtX/XN72FVV9uHz7i6q6S1UvV9WPqOoh67V/WX5st6r6TuTWi+nwODMzQ7FYrOr0oZSvD6WMnGqCYzM0NEQ2mw11FlEyd6DkWoeGhur6UTz77LM89NBDq36c7RJ9WF2Vazv9sJx1P5HyYot+EBdffDEXXnhh4OPNFP04s3ficPp++zOK6JuTV9jV7MLCAuPj48zPz1eyfIJEv9HwzsLCAv39/YFXZs0ozjKiv27dOkSkpri+t8I/qPvszMwMo6OjLa+ct+nailwoHViqyjvvvANEu2TasmULyWRyxVk56mdB+I/i5MmT9PX1RZqkqbcToYkJek8Y7crTh9WiX4vTrzZmI7Bh3+1tt93GTTfdFPh43KJvBN6Ifl9f34pjqZ6Yvt/3V2ueftCVU5Te89XCO4VCgeXlZfbs2UNfX18lxGNCpd5J9pGRkYYWSA9rtgYl05ZKpWJ1+mbfpVIpxsfHa3L6pu+OIag9SdSIRDPpatE3B8Xbb78NhLtBQyqVYuvWraEHlB/eVZj8CFo4xY96L3+9C4IY2lWRC6tdjVf043D6jTSHi1v0k8kk6XR6hejb1Or0i8UiqhpLeMfvJBqlDXG1iVyzfWRkhJ07d/Lyyy+Ty+U4e/YsmUxmVVKEd1H6WgmrxjXE2WnTdvpAzRk83pNUmOi3M3MH1pjoRz2D/sIv/AKf/OQn6/qsoIPYLJxSbRLXYJZNrBXzGu9r2xne8a5wZbojGjcWlr0TR3inGkaA4sjcMZiq3DhEP6hDaivDO9Wcvp2SuGfPHrLZLK+88kola85rdBrN1Y8q+nFN5NpOH2pvvBbk9O3X5/N55ufnndNvBPMjPn78OIODg5FLm9etW1cpt671s4JEf2ZmhuXl5arxfEO9VblG7P2cfrtE355Qh5UtGMKcfi3hnUZEP26nD+eari0tLfmKfi1iFNRttB2in8/nfcduZ9NceOGFjIyM8OKLLzI1NeV7FdYK0Y9zIRU/p19L4zVvTH9gYKDSjM9gfrvO6TeA7b6bffasFtM3rQCirjdrsk1q+VGYxUJgpdM3lZLtjOkXCoWKONndLBudyL3wwgu56KKLIl9B+bF582a2b98eOtlbK0b07fVxDbU2SjPtgeMozvI7BqqJvhF6c2L1c/u26CcSCXbv3s2rr77Ku+++6/vbayQt2VSrtzK84+2HU0vaptl/3vAOrF7vA9qbrglrRPShMScYhUwmQzKZDHT6QWu0BlHPj8J20/bJwohLO0Ufzh3gdpvjsIncKKK/ceNGfuVXfqWhBlUDAwP82q/9WqzHiC36Xqdf6+pZtTr9bDbrmwNe70SuEXlT8+BnbLx583v27KFYLJLP5333ayNO37TqjlP0bcPkhze8Y8xblLi+X3qpX6V6J1TjQpeLfiqVqohBs8+epsowbtGv5UdhnyDs27UsldgMvFW58/PzkZx+O69OGqVaeAeih2WM0/fuC+P8ve/zyCOP8MADD6zYpqqBJ9FqYRCv6Ic5ffNdb968uSKMfuGdVCpFf39/XaJfrTDLUIvoHzx4kD/+4z8OrLL17ruhoaHIjdf8msP5Vap3QjUudLnow7kd3WynD+FVubOzs6TT6chFF/U4ffPc8fFx30VK2pm9A6UD3NQyxOX0O5Uwp19P+wRY/f0FLchy/PhxvI0JzYmjnpi+ESYT0ghy+qaPkRnbnj17gODfXr1Vuc0Q/TNnzpDL5QLbHXtj+iISufGa3WHT4Cf609PTDAwMtP2Y706bZTE4OLhifc5mEub0g9ZoDXsvEalL9M8//3xeffVVVHXF0nyd4PTtdE0zJrP6kzfDIygG3Q0Y0VdV35g+1C76fvvCK/qqytTUVKXfvLcdQVBMP5/PUywWfVtVRHH6fnnz1157LSMjI4HJC80W/Vomcs3v1i8sZjdbs1m/fn2kVtN+4R2/mP7MzEzbXT6sAadvxKUVTj+sFUPQGq1BJBKJmqtyzXM3b95MLperHPCdIvpmSUJghdOH1QJoJp/b7Xrqpa+vrzK/0mhMPyi8Y7bZ+25ubq5y34QL7M8KCu+EjSdqTN8rwplMhiuuuCKwLqWTnL55T7/wTtC+W7duHbOzs5Ezn6qFdzohRx/WgOibHd1Kp++Xu1ur6EPtBVqzs7MMDQ1V/lbz2k4R/cXFxRUtGCBYcExBUreKvu3umxXeMe9lv4+pgAVWpBOGiX61PjVGmEZGRkin04Ex/VoLGo3o17rIeC2ib5arrIY5+fg5fbuXvo0Jd5nMvGrjtZ1+IpGgr69vVUzfiX4MbNiwgfHx8ZraJNfL0NCQbx6zqrZE9E0IyTsJ3G7RT6fTpFKpwPCOPUZDtVWzOp1miH4Up28LkO30w9YmiCL6ZnW4oEpSb/FRFIaHh1dckUZlYWFhRcfKIGrptBkW3jHj8+67qGmbi4uLpFKpVa+3K9VNxlUnhHe68xdncd1113HNNde0pIGRXRdgu4Ll5WXy+Xxdon/06NHIz5+dnWV4eHhVl852iz6cq0A0+8WIfpDTr7ZqVqdjC5JXnOoN70R1+iJS6SxrqBbTh3DRN72oghqFVeuF44dtTmrpKmmuKqr9pu2/q1pKrxH9sPCO9z1MuCuK6PvtG/sE2ik5+rAGnL7JcGgFQVW5taZrGoaHh1lYWKj86KthriY6zenDOVczNzdXKeCBYAGstmpWp9Mqp+/NfpqammJ0dJTh4eGawzthMX3j4v1aAptma/WEd6D2Aq2ooaSo7ZXtq41anH4mk2F0dDRSeMfvKsgWfXNV5sI7XUZQVW69ol9L2qYpLhkZGaGvr49UKtWRTt9uwWCPyYV3gqllIvfdd99l3bp1jI2N1TyRW83pg39L4Gq97YOot0CrVtGvdkVl/15rcfpQcvuNOH17OVfoIqcvIjeLyCsickRE7vF5/C4ROSgiz4vIEyKys7x9m4gslrc/LyJ/Gvcf0Eridvom48ienAvC/HBGRkYQkRWZEZ0i+iZ7x25s1ovhnWZP5E5MTDA6OhpZ9KOGd8Df6UedWPXSKtGv5vTt32stE7lwrvFaGN6+Owa/8E6tGtEMqoq+iCSB+4BbgJ3Ap42oW9yvqntU9XLga5RWyzK8Vl5c5XJVvSuugbeDoE6bRvRrbehl+smYFbfC8J5Y7EngMKfYKqo5/bUs+s1qw2C2mfdZXl5mYWGBiYkJxsbGmJ6ermSuhL1HraJv6g/sx6F20R8YGKhrgfSok8ZRF1Kxf69+Tj8ovAMl0V9cXKy6uEyQ6C8tLVEsFpmZmWFkZKRtBZQ2UZz+1cARVX1dVbOUFj6/zX6Cqs5Yd4eA2nK0uoRMJkM6nfYV/b6+vpr7wwwPD9Pf37+qutIPP9HvNKdvYvp+Tt/rejthzI1geuonk8mGu2NGDe+YK8J169YxOjpKLperONe4nP7g4CCqusIR1+v0vVekUVDVQOfspVanPzg4WHN4p1raZth4zf5aWlrqmHRNiCb6FwBvWfePlretQEQ+LyKvUXL6X7Ae2i4iPxSR74nIhxoabQfg14qhnnRNKP0oNm7cWJfo2/34292GAUqiXywWyeVyPRHegZLb98tKaVZ4xwiPcfpwLmxQ70RuoVAgm82ucPqwsqioXtGH0vFaywpUuVyOYrHYFNGfnJysaSIXqmfwLC8vo6qBMX0o7ctOWDHLENtErqrep6oXAV8CvlzefBx4j6peQWnB9PtFZNXpTkTuFJEDInIgigC2E7+q3HpFH0p1BidPnqxaYDI7O1tp+gYl0V9eXq50JIT2Z+8YapnI7WbR7+vr8xX9RCKxoj1GNfL5fOU1Xvycvonpw7mskLD9ab4DP3H09o3xax9Q70QuwKWXXsrRo0cr60dXw6+PTRBRRX9ubo50Os3IyEig008kEr4n3YmJCUQkUPTD9o29LzulGheiif4xYKt1f0t5WxAPAp8AUNVlVT1Tvv0s8BpwifcFqvp1Vd2rqns3bNgQdextwa//Tq19d2w2bNjA0tJS1UtgEzYxqZB22mYniL590NuiXy1ls1vDO1By+n4FRCaNuJY8/aD94HX6AwMD9Pf3r3L6+XweEfHtrSMigS0LvCLr1xLY22ytFq666irS6TT79u2L9PxmiP7CwkKla2aQ00+n074n3WQyycTERGB4J2y+w/wN7777Lvl8vquc/n5gh4hsF5EMcDvwsP0EEdlh3f048Gp5+4byRDAi8l5gB1C9g1EH4w3vmGrceldlijqZ672asHOgw37wrcL+kfZKeGf37t3s2rXL97FalkwMazFtN6ybmpqqhBuGh4dJJBIrwjtBwgXBzcmiOP16CrMMAwMDfOADH+DgwYORYvu1iH4tE7lDQ0OVJS69VCvuCsvgCXP6Zp+Zq5yucfqqmgfuBh4HDgMPqeohEfmqiNxaftrdInJIRJ6nFMa5o7z9w8AL5e3fAu5S1fBKhw7HhHdMOGZxcZFCodCQ0weqxvW9om/n+BvRaEVVchBBTj+oJ/xaEP1rrrmGn/mZn/F9LKyltJdCoRA4H2OvwmUvTSgiK9I2q7WpDlpIJarTr1f0odSNs1AosH///qrPrUX0RSRSp00j+n19feRyuVXFkNX2ncnV9wvBRnH677zzDtAZOfoQsQ2Dqj4KPOrZ9hXr9hcDXve3wN82MsBOY2hoqFKh2N/fX3eOvv1+AwMDkZz+BRecmz/3hnfaHSaxD3r7dlCoYy2Ed8KI0+lDyY1OT09XetgDK0S/2jEQFNrwimx/fz8isiqm34joT05Ocskll3DgwAE+9KEPhY7TjCdqL60oi6PPz89z3nnnVd7TW13s7aXvN/5cLucbxg1z+ubzjOh3jdN3rMRblduo6EfJ4CkUCiwsLKz4DJMDbZx+u/N/bafoHYuf6zW99Nt5ddJMahH9ak4fSsv2qWolvANUcvWhulsdHx/3LQL0ir5pdOZ1+vVM4tpce+21LCws8MILL4Q+z5yYon5etfbKqrrC6dufYYgS3gH/DJ6wk5SIVFKZE4lE3SHguHGiXyPeqtxGRR9KIZ5Tp04FZvDY1bgGOwc6bCKwVSSTSTKZzIp4viGdTvvG9Ns95mbi9zcHEebSjZCbK0F7aULj9MOWSjSsW7eOqampVYu1Ly4uIiIrspC8rRgadfoA27ZtY9OmTezbty80U21xcbFSAxGFaqJviqNMTB9WF2h518f1Uk30+/v7A+fTzMlrdHS0YwyOE/0a8Vbl1luNa7Nx40aWl5cDe/AEnVhMgVYnhHegdID77Qc/19vNC6hEoVanXy28Y64EbdEfGxur9GSqtj/XrVtHoVBYdYzZHTYNdvsAk8ffqOiLCNdeey2nTp0KXY3KbzxhVBN98zsNc/rVwjujo6OkUqlA0Q/bN0b0OyWeD070a8bP6ff39zckYGYyNyiuHyT6pkCrU0T/sssu45JLVmXk+rpeJ/rnCAvP2aKfSqVWHAN2rn61K6egIiO/alK7/04jOfpedu/ezfDwcGj6ZtRqXEO1mL4t+vU6fRFh3bp1vmmb1UJf5oTQKfF8cKJfM96YfiM5+oZqaZvdIvo33XQT11577artfgK41sM7cU/knjx5slIoZLBz9aOEd2B1OwE/kbXDO41U43pJpVJcfvnlHDlyJLQlRC0LIsXl9KsZkKC0zahO34l+F5NKpchkMiucfqOiPzg4yNDQUOBkrrca1zAyMsLi4iLLy8sdLaBBMf217PRrielHCe/Mzc2tCO3AOSGJIvomRBFF9P2cfhyiD+dOPkFrTS8tLdXk9KulbEZ1+tX6Zhmn750Tqeb0XXhnjWBX5cYh+kBoBs/c3BzDw8Or4pzmc8+ePdt1ou/CO+eIEt4BVon+wMAA6XQ6UnhHRJiYmFiVwRMk+qa9R70dNoMIak8eNp4wojr9wcFBX6dfLBbJ5/NVRX9ycpJisbhq4Zr5+Xkn+r2AqcpV1VjCOxCewRN0YjGTpvPz8x0t+i68E06U8A6wIl0TVhZoRTmJ+i0IEhTeMY/F7fTtY9aPZoi+SW82WUG2049aJOiXwfPEE0+Qz+e57LLLAl/nwjtrBOP0FxYWKBaLsYm+KcDxEiT69rZOFtBeDO/U2nunHqcP53L1o+xPk7ZpjIUpMvRz+lAKXcQ5kQvhTt9kCtUa08/n86vCLgaTo2/wFqmFtVW28Yr+1NQUTz75JLt37+bCCy8MfN2OHTu44oor6KSeYk7068C0YogjXdMQNplbzelDZ4t+UEXuWhf9OJy+vY/8RH90dJTp6enITj+fz1eO26BCKK/Tr7fZmh9BCxGZz/MbTxjVFqzxir63/05YW2XvuPv6+iqi//jjj5NIJLjxxhtDXzcxMcGtt97a9uJJGyf6dWDCO6YEPi6nD6t78Ji4qt9nDA0NVeL8nXRQeQmryF2rpNNpVDXSovdRnb5ZXtNmdHS0UrxXbX96M3iCRNbuA99IszU/0un0ikQIm1qrcaF6p824nL6IMDk5ybvvvsuRI0d45ZVX+PCHP9xRYZuoONGvg8HBQYrFYkWg4xB9U9jkFf2wit9EIlE5oDtZQP2c/loX/VoWUokS0x8bG/N9jj1BGMXpw7kQRZDoG5E34Z04RR/825OHjSeMWkU/yOlHWfVucnKSU6dO8dhjj7Fu3Trf9ORuwIl+HZiDyLRMjaunxsaNG1eFd6q1eTDbO1lA0+k0xWJxRdy1F2L6UF30TQfNak7fL7QDtYn+6OgoyWSyJqffzaJfKBRYWloKdfpRwztQOmnOzMxw5swZbr755o7+zYXhRL8OzEH0zjvvMDg4GNuXv2HDhkpjLUM10TcnnE4+AL1xV3MCcKJfvduoORkEib4dXqi2PxOJxIq0zSCRTafTpFKpitOPaxLX0CrRN5PQXtH3y96J6vQBLrnkEnbs2FHl2Z2LE/06MAfR6dOnYwntGDZu3Egul+Ps2bOVbWtB9L0CuNbbKkP1CUZD2KLoUIolX3XVVezevdv3cdvpR9mfdjuBMJE1/Xe62embuQ5veKdep79t2zYuvvhibr755sjj60Qiib6I3Cwir4jIERG5x+fxu0TkoIg8LyJPiMhOz+PvEZE5EfntuAbeTsxBpKqxir6ZzDX9t6F04CYSicAfQreEd+CcAK6FBVSqUavTD5uI//mf/3ne+973+j6WyWQqKY5R9qdZ+k9VQ9sCDw4OMjc3F0uzNS92nYuNGY/fusNBhIm+XY1r6OvrI5/PV062tTj9kZERPvvZzwZedXULVZWivNzhfcCNwFFgv4g8rKovWU+7X1X/tPz8W4F7Aft0eC/wz7GNus0ELQLeKBs3biSRSPDQQw9xwQUXcPHFF/P2228zMjIS2HXQfH4nZ+94BdCJ/jniuOoZGxtjaWkp0v60FwQJ62g5MDBQmfBtRnjHnHTs31K1NsV+GLH2u6LyE33vQiq1TOSuFaLs3auBI6r6uqpmKS18fpv9BFWdse4OAZVTuIh8AngDONT4cDuDZDJZOXjidPp9fX3ceeedXH/99YgI3/ve93jjjTdC08K60en3QnjH/G2NhneiYI6PqJORUErbDKt+HRwcrIQZm+H0YXWufq19dyB8ndww0TchnlrCO2uFKEfaBcBb1v2jwDXeJ4nI5ymtj5sBfra8bRj4EqWrhDUR2jEMDg6ytLQUq+gDbNq0iU2bNnHDDTewsLDAG2+8UZlA8qMbRb8XnL752+II71TDiH7UmD5UF/2BgYFK+KWZom9XqtbaggGqh3eSyeSKcJG5bSZzc7lcpUVDrxDbRK6q3qeqF1ES+S+XN/8u8F9ogncAABBUSURBVF9VdS7stSJyp4gcEJED1RYI7xTMgRu36NsMDg6ya9cuNm/eHPiczZs3c91113HRRRc1bRyN4sI7wcQV3oFo+3NsbIxEIsGZM2eqir6hVU6/HtE3S24GZe/YBYzg7/R7KbQD0Zz+MWCrdX9LeVsQDwJ/Ur59DfApEfkaMA4URWRJVf9f+wWq+nXg6wB79+4NXkutg2iF6EchmUxWLQVvN70c3qkm+ia804jTvPjii3nzzTcjHYt22ubi4mLgVWTQQvdxECb6tU6Sikhg0zVvYRawqtPmWq8X8SPKr24/sENEtlMS+9uBz9hPEJEdqvpq+e7HgVcBVPVD1nN+F5jzCn63Yn4I7Rb9bsAb6ugFpx81ZTOOE+B5553Hr/7qr0Z+vknbjOr0457INZPHfqJfS7M1Qy2i7+2pX22pxLVI1SNNVfMicjfwOJAE/lxVD4nIV4EDqvowcLeIfBTIAVPAHc0cdCcwPj5OOp32XQjcsRLvpGYviH6tTr+VVz0TExO8+eab5HK5qqIfZ7M1QyKRqDQtNKhqXRO5ELxgjXfOAFY7fRfeCUBVHwUe9Wz7inX7ixHe43drHVwnc80117Bz586emgCqFxfeCSaOidxaWbduXeW7CMvesf+PG2+B1vLyMqpal+j7OX2z1kVQeMc4/Wrr465FXEVunWQymdCsGsc53ERuMO04AdqLsVRz+q0S/XqqcQ1+op/NZikUCqtE3yykYsf0e83pO9F3NJ1eTNlMJpOISEvy9GvFNiud4vTDqoOr4Sf6fjn6Brv/jnP6DkcT8ApgL4R3INpCKu0I74yNjVXSGINE34hv3JO4Bm9MP26nHyb6dntl5/QdjiZhL6TSKwUxfovHeGnHCTCZTFYWZAkS2UQiweTkJOvXr2/KGIaGhlheXq78/a0WfXsit9ec/tq2Wo6OwV5IZa0voGKI4vTbEd6Bc+vlhonsr//6rzftxGwvkG56B0F9ou+XvVMtvNPL2TvO6Ttagu161/r6uIZODe/AucncsBh6Op2uqflZLXgLtJrl9P3mJIzTLxaLFAqFnjgWbda+3XJ0BLYb65UqyKD8cZtCoYCINE1cg7jmmms4//zzW/65Bj/RT6VSdV3xZDIZCoXCirWG5+fn6evr830/M5FbS1vltYQTfUdLsF1v2Jqwa4moTr8d+2JycrKtKcd+ol/vpLHddM28h181rsGEd3qxwya48I6jRfSi048i+r04kQjNE30oHV8//vGPA09q/f395PP5Sly/15y+E31HS/Bm7/SC0EUR/bm5uZ7s32TW4TWiX28LBli9kMqzzz7L3Nwc1113ne/zTVWuWYrUib7D0QTs7J1eCe9EienPzs7GuvpatyAilWUToTGnby+kksvleOKJJ9i+fTvbtm3zfb6ZvDai3wsGxMaJvqMluPCOP7Ozsz3p9GFlVW69HTZhZXhn//79zM/Pc8MNNwQ+3zl9h6MF2ALoRL+EqjI/P9+TTh9Koj83V1pfKY6Y/tzcHE8++STvfe97ec973hP4fOf0HY4WYDv9Xgnv2CEtPxYWFigWiz3v9HO5HPl8vmHRf+qpp1hYWAh1+bBa9J3TdziagHP6qzGi08tOf35+vqHCLDgn2sePH+eiiy5i69atoc934R2HowUYp6+qPdeGwSww7sWENnrZ6ReLRc6ePQs0LvpAVZcPLrwTSfRF5GYReUVEjojIPT6P3yUiB0XkeRF5QkR2lrdfXd72vIj8SEQ+Gfcf4OgOUqkUqkqhUOiZNgzpdBpVpVgs+j7unH4pV//06dNA46K/Y8cOtmzZUvX5ve70q9otEUkC9wE3AkeB/SLysKq+ZD3tflX90/LzbwXuBW4GXgT2lpdcPA/4kYj8o6qGpzQ41hxG5E1BTC+Ivr2Qil9vHSM6vez0oXHRTyaT/PIv/3IkwYdSB1Fz5ZlMJtvWiqJdRPlrrwaOqOrrqpoFHgRus5+gqjPW3SFAy9sXLIHvN9sdvYcReRO/7ZXwDgSvnjU3N0d/f39P7As/jOifOXMGaKx3/86dOxkdHY38fBPi6TWXD9FE/wLgLev+0fK2FYjI50XkNeBrwBes7deIyCHgIHCXn8sXkTtF5ICIHDh16lStf4OjCzDCZopxesHpe1cM89Kr1bgGr9OvN0+/HkyIpxeOQy+xXdeo6n2qehHwJeDL1vanVXUX8FPA74jIqm9WVb+uqntVda939XrH2sDr9Hvhx1bN6fdqNa7BtD2emppCRCpC3Aqc0w/nGGDnQG0pbwviQeAT3o2qehiYA3bXMkDH2sCFd1bT604/mUwyMDCAqjIwMFBZwrEVOKcfzn5gh4hsF5EMcDvwsP0EEdlh3f048Gp5+3YRSZVvXwhcCrwZw7gdXYYRwF50+n7hHVXteacP50I8zVqLN4hedvpV7VY58+Zu4HEgCfy5qh4Ska8CB1T1YeBuEfkokAOmgDvKL/8gcI+I5IAi8L+r6ulm/CGOzsaIfC/G9P2c/tLSEoVCwYn+0BCnT59uaTwfetvpR7rGVtVHgUc9275i3f5iwOv+G/DfGhmgY23gdfq9Ht7p9XRNg3P6rae3ElQdbcNN5K6k16txDe0SfeP0neg7HE2iF0XfCIspSLPp9WpcQ7udfi8ch16c6DtagjdPvxfCO2NjYySTyUoeuo1z+iWM6LuYfutwou9oCb3o9BOJBOvXr+fkyZOrHpudnSWTyfRkeMGm3U6/F/e/E31HS+jFlE2ADRs24FdlPjc31/OhHXAx/XbgRN/REhKJBMlksjKp6deAbC2yceNGpqenWV5eXrG9l5dJtNm0aRM7duwIXemqGbiYvsPRAozbT6VSLa2+bCemrYjX7fd6Na6hr6+Pz3zmM0xMTLT0c8fGxhgaGmLjxo0t/dxOwIm+o2UYV9VL7sqIilf0XTVuexkYGOC3f/u3W36F0Qk40Xe0jF4U/fHxcVKp1IrJ3OXlZXK5nBN9R1twou9oGSa800uibzJ4bKfv0jUd7cSJvqNlGLHvhRx9m40bN65w+q4wy9FOnOg7WkYvOn0oTebOzs5WKnNd3x1HO3Gi72gZvRjTh9WTuS6842gnTvQdLaNXwzsmbdOEeGZnZ0mlUi1dKcrhMDjRd7SMXg3vjI+Pk06nVzj94eHhnqlVcHQWkURfRG4WkVdE5IiI3OPz+F0iclBEnheRJ0RkZ3n7jSLybPmxZ0XkZ+P+AxzdQ6+KvoisaMfgCrMc7aSq6ItIErgPuAXYCXzaiLrF/aq6R1UvB74G3Fvefhr4BVXdQ2k1LbegSg/Tq+EdKIV47PCOy9xxtIsoTv9q4Iiqvq6qWUoLn99mP0FVZ6y7Q4CWt/9QVd8ubz8EDIiIC2T2KL0u+nNzcywuLjrRd7SVKL++C4C3rPtHgWu8TxKRzwO/CWQAvzDOLwHPqeqy9wERuRO4E+jJsuheoVfDO3Aug+ftt99meXnZhXccbSO2iVxVvU9VLwK+BHzZfkxEdgF/APx6wGu/rqp7VXWvyXRwrD16NWUTzon+66+/Drh0TUf7iCL6x4Ct1v0t5W1BPAh8wtwRkS3A3wO/pqqv1TNIx9qgl8M7o6OjZDIZ3njjDcBV4zraRxTR3w/sEJHtIpIBbgcetp8gIjusux8HXi1vHwceAe5R1SfjGbKjW+nl8I7J4Dl+/DjgnL6jfVQVfVXNA3cDjwOHgYdU9ZCIfFVEbi0/7W4ROSQiz1OK699htgMXA18pp3M+LyK918DaAfR2eAfOFWmBc/qO9hHpOltVHwUe9Wz7inX7iwGv+z3g9xoZoGPtYC+i0ouYuH4ikWBwcLDNo3H0Kq4i19EynNMvOX1XjetoJ070HS3D9Jrp1Z4zxum7eL6jnTjRd7SMrVu38qlPfYoLL7yw3UNpCyMjI/T19bl4vqOt9GZw1dEWRIRdu3a1exhtQ0T42Mc+1vJFwB0OGyf6DkcLufLKK9s9BEeP48I7DofD0UM40Xc4HI4ewom+w+Fw9BBO9B0Oh6OHcKLvcDgcPYQTfYfD4eghnOg7HA5HD+FE3+FwOHoIUdV2j2EFIjILvNLucURkPaXF37sBN9bm4MbaHNxYa+dCVa269GAnVuS+oqp72z2IKIjIATfW+HFjbQ5urM2hm8YKLrzjcDgcPYUTfYfD4eghOlH0v97uAdSAG2tzcGNtDm6szaGbxtp5E7kOh8PhaB6d6PQdDofD0SQ6SvRF5GYReUVEjojIPe0ej42I/LmInBSRF61t60TkOyLyavn/jlgdQ0S2isi/ichLInJIRL5Y3t5x4xWRfhF5RkR+VB7r/1Xevl1Eni4fC38tIpl2jxVARJIi8kMR+afy/U4d55siclBEnheRA+VtHff9A4jIuIh8S0ReFpHDIvLTnThWEXlfeX+afzMi8h87caxhdIzoi0gSuA+4BdgJfFpEdrZ3VCv4/4CbPdvuAf5FVXcA/1K+3wnkgd9S1Z3AtcDny/uyE8e7DPysqn4AuBy4WUSuBf4A+K+qejEwBXyujWO0+SJw2LrfqeME+IiqXm6lE3bi9w/wR8Bjqnop8AFK+7fjxqqqr5T35+XAVcAC8Pd04FhDUdWO+Af8NPC4df93gN9p97g8Y9wGvGjdfwU4r3z7PEo1Bm0fp8+4/wG4sdPHCwwCzwHXUCp2SfkdG20c3xZKP+qfBf4JkE4cZ3ksbwLrPds67vsHxoA3KM8vdvJYPeP7GPBkN4zV+69jnD5wAfCWdf9oeVsns0lVj5dvvwNsaudg/BCRbcAVwNN06HjLIZPngZPAd4DXgLOqmi8/pVOOhT8E/hNQLN+fpDPHCaDAt0XkWRG5s7ytE7//7cAp4JvlsNk3RGSIzhyrze3AA+XbnT7WFXSS6Hc1WjrNd1QqlIgMA38L/EdVnbEf66TxqmpBS5fMW4CrgUvbPKRViMjPAydV9dl2jyUiH1TVKymFSz8vIh+2H+yg7z8FXAn8iapeAczjCY900FgBKM/b3Ar8jfexThurH50k+seArdb9LeVtncwJETkPoPz/yTaPp4KIpCkJ/l+p6t+VN3fseAFU9Szwb5TCJOMiYtqEdMKx8DPArSLyJvAgpRDPH9F54wRAVY+V/z9JKe58NZ35/R8Fjqrq0+X736J0EujEsRpuAZ5T1RPl+5081lV0kujvB3aUsyEylC6fHm7zmKrxMHBH+fYdlGLnbUdEBPgz4LCq3ms91HHjFZENIjJevj1Aae7hMCXx/1T5aW0fq6r+jqpuUdVtlI7Nf1XVz9Jh4wQQkSERGTG3KcWfX6QDv39VfQd4S0TeV970c8BLdOBYLT7NudAOdPZYV9PuSQXP5Mh/AP4npZju/9nu8XjG9gBwHMhRciefoxTT/RfgVeC7wLp2j7M81g9SusR8AXi+/O8/dOJ4gfcDPyyP9UXgK+Xt7wWeAY5Quozua/dYrTHfAPxTp46zPKYflf8dMr+lTvz+y+O6HDhQPgb+OzDRwWMdAs4AY9a2jhxr0D9XketwOBw9RCeFdxwOh8PRZJzoOxwORw/hRN/hcDh6CCf6DofD0UM40Xc4HI4ewom+w+Fw9BBO9B0Oh6OHcKLvcDgcPcT/D+vwKJwWd6WdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.Series(results).plot.line(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Low-level model classes\n",
    "\n",
    "The `cophi.corpus()` function wraps three classes:\n",
    "\n",
    "1. `cophi.model.Textfile` models a document on the file level.\n",
    "2. `cophi.model.Document` models a document on the text level.\n",
    "3. `cophi.model.Corpus` models a collection of documents on the corpus level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Textfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = \"/home/severin/kafka_derprozeß.xml\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = cophi.model.Textfile(filepath=file,\n",
    "                         treat_as=\".xml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kafka_derprozeß'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'.xml'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.suffix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/severin'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.parent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Der Prozeß'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.content.strip()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Context-manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 10 characters of 'kafka_derprozeß.xml', treated as XML: \n",
      "Der Prozeß\n"
     ]
    }
   ],
   "source": [
    "with cophi.model.Textfile(file, treat_as=\".xml\") as document:\n",
    "    print(f\"First 10 characters of '{document.title + document.suffix}', treated as XML: \\n{document.content.strip()[:10]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 39 characters of 'kafka_derprozeß.xml', treated as plain text: \n",
      "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with cophi.model.Textfile(file, treat_as=\".txt\") as document:\n",
    "    print(f\"First 39 characters of '{document.title + document.suffix}', treated as plain text: \\n{document.content.strip()[:39]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Processing XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<lxml.etree._ElementTree at 0x7fedb490ad08>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lxml import etree\n",
    "\n",
    "tree = f.parse_xml(parser=etree.XMLParser())\n",
    "tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Jemand mußte Josef K. verleumdet haben, denn ohne daß er etwas Böses getan hätte, wurde er eines Morgens verhaftet. Die Köchin der Frau Grubach, seiner Zimmervermieterin, die ihm jeden Tag gegen acht Uhr früh das Frühstück brachte, kam diesmal nicht. Das war noch niemals geschehen. K. wartete noch ein Weilchen, sah von seinem Kopfkissen aus die alte Frau, die ihm gegenüber wohnte und die ihn mit einer an ihr ganz ungewöhnlichen Neugierde beobachtete, dann aber, gleichzeitig befremdet und hungrig, läutete er. Sofort klopfte es und ein Mann, den er in dieser Wohnung noch niemals gesehen hatte, trat ein. Er war schlank und doch fest gebaut, er trug ein anliegendes schwarzes Kleid, das, ähnlich den Reiseanzügen, mit verschiedenen Falten, Taschen, Schnallen, Knöpfen und einem Gürtel versehen war und infolgedessen, ohne daß man sich darüber klar wurde, wozu es dienen sollte, besonders praktisch erschien. »Wer sind Sie?« fragte K. und saß gleich halb aufrecht im Bett. Der Mann aber ging über die Frage hinweg, als müsse man seine Erscheinung hinnehmen, und sagte bloß seinerseits: »Sie haben geläutet?« »Anna soll mir das Frühstück bringen«, sagte K. und versuchte, zunächst stillschweigend, durch Aufmerksamkeit und Überlegung festzustellen, wer der Mann eigentlich war. Aber dieser setzte sich nicht allzulange seinen '"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.xpath(\"tei:text//text()\", namespaces={\"tei\": \"http://www.tei-c.org/ns/1.0\"})[31]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use case: Segmenting and tokenizing a textfile\n",
    "\n",
    "Construct a `Document` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = cophi.model.Document(text=f.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Segmenting the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71 segments in 'kafka_derprozeß'.\n"
     ]
    }
   ],
   "source": [
    "segments = list(d.segments(size=1000))\n",
    "print(\"{} segments in '{}'.\".format(len(segments), f.title))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating labels for the segment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = [\"{}_{}\".format(f.title, n) for n in range(len(segments))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "kafka_derprozeß_0    [der, prozeß, hdl, de, textgrid:qmx, der, anno...\n",
       "kafka_derprozeß_1    [vielleicht, noch, besaß, schätzte, er, nicht,...\n",
       "kafka_derprozeß_2    [worte, die, ich, mit, einem, mir, ebenbürtige...\n",
       "kafka_derprozeß_3    [gehen, pflegte, spät, nach, hause, kam, und, ...\n",
       "kafka_derprozeß_4    [er, dann, hinüber, die, drei, wichen, auch, s...\n",
       "kafka_derprozeß_5    [mann, aufmerksam, machte, den, er, selbst, sc...\n",
       "kafka_derprozeß_6    [ja, bloß, die, vermieterin, nun, ich, habe, a...\n",
       "kafka_derprozeß_7    [fenster, lag, und, die, müden, augen, drückte...\n",
       "kafka_derprozeß_8    [den, leib, schickt, da, sie, aber, doch, frei...\n",
       "kafka_derprozeß_9    [hat, eine, größere, summe, von, mir, geliehen...\n",
       "dtype: object"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(segments, index=labels).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Low-level helper functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The return value is a <class 'generator'>.\n",
      "Use it e.g. as a list: list(tokens) -> ['Jemand', 'mußte', 'Josef', 'K']\n",
      "Note that it stopped tokenizing after 4 words (because you set maximum=4).\n"
     ]
    }
   ],
   "source": [
    "tokens = cophi.utils.find_tokens(document=\"Jemand mußte Josef K. verleumdet haben.\",\n",
    "                                 token_pattern=\"\\w+\",\n",
    "                                 maximum=4)\n",
    "\n",
    "print(\"The return value is a {}.\".format(type(tokens)))\n",
    "print(\"Use it e.g. as a list: list(tokens) -> {}\".format(list(tokens)))\n",
    "print(\"Note that it stopped tokenizing after 4 words (because you set maximum=4).\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Constructing ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The return value is also a <class 'generator'>.\n",
      "You can use it as a list, too: list(bigrams) -> ['Jemand mußte', 'mußte Josef', 'Josef K']\n"
     ]
    }
   ],
   "source": [
    "bigrams = cophi.utils.construct_ngrams(tokens=['Jemand', 'mußte', 'Josef', 'K'],\n",
    "                                       n=2,\n",
    "                                       sep=\" \")\n",
    "print(\"The return value is also a {}.\".format(type(bigrams)))\n",
    "print(\"You can use it as a list, too: list(bigrams) -> {}\".format(list(bigrams)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using DARIAH-DKPro-Wrapper\n",
    "\n",
    "You can plug the [DARIAH-DKPro-Wrapper](https://dariah-de.github.io/DARIAH-DKPro-Wrapper/) into the high-level API.\n",
    "\n",
    "> Check out the [short user guide](https://dariah-de.github.io/DARIAH-DKPro-Wrapper/user-guide.html), or the extensive [tutorial](https://dariah-de.github.io/DARIAH-DKPro-Wrapper/tutorial.html) for setting up and using DARIAH-DKPro-Wrapper.\n",
    "\n",
    "All you have to do is putting the JAR file and the configs folder in your current working directory. Setting e.g. the `lemma` arg in `cophi.corpus` to `True` will lemmatize all words, and passing `[\"NN\"]` to `pos` will make `cophi` use only nouns for further processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus, metadata = cophi.corpus(directory=\"british-fiction-corpus\",\n",
    "                                lemma=True,\n",
    "                                pos=[\"NN\"],\n",
    "                                language=\"en\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a-dozen</th>\n",
       "      <th>a-mak</th>\n",
       "      <th>a-pace</th>\n",
       "      <th>a-voot</th>\n",
       "      <th>a-year</th>\n",
       "      <th>abashment</th>\n",
       "      <th>abatement</th>\n",
       "      <th>abbe</th>\n",
       "      <th>abdication</th>\n",
       "      <th>abduction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dc802706-678c-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>03d6e876-678d-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2c85d2dc-678d-11e9-b708-7c7a9196f28f</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a-dozen  a-mak  a-pace  a-voot  a-year  \\\n",
       "dc802706-678c-11e9-b708-7c7a9196f28f      NaN    NaN     1.0     1.0     2.0   \n",
       "03d6e876-678d-11e9-b708-7c7a9196f28f      1.0    NaN     NaN     NaN     NaN   \n",
       "2c85d2dc-678d-11e9-b708-7c7a9196f28f      1.0    NaN     NaN     NaN     NaN   \n",
       "\n",
       "                                      abashment  abatement  abbe  abdication  \\\n",
       "dc802706-678c-11e9-b708-7c7a9196f28f        1.0        1.0   NaN         NaN   \n",
       "03d6e876-678d-11e9-b708-7c7a9196f28f        NaN        NaN   7.0         NaN   \n",
       "2c85d2dc-678d-11e9-b708-7c7a9196f28f        NaN        NaN   7.0         NaN   \n",
       "\n",
       "                                      abduction  \n",
       "dc802706-678c-11e9-b708-7c7a9196f28f        NaN  \n",
       "03d6e876-678d-11e9-b708-7c7a9196f28f        NaN  \n",
       "2c85d2dc-678d-11e9-b708-7c7a9196f28f        NaN  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.dtm.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, these are the 10 most frequent _nouns_ in the corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['lady',\n",
       " 'man',\n",
       " 'friend',\n",
       " 'day',\n",
       " 'woman',\n",
       " 'gentleman',\n",
       " 'house',\n",
       " 'dear',\n",
       " 'time',\n",
       " 'family']"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.mfw(10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}