notebooks/introducing-cophi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing text for Natural Language Processing\n",
    "\n",
    "This is a short, user-oriented introduction to the Python library `cophi`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cophi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High-level API\n",
    "\n",
    "There are basically two functions:\n",
    "\n",
    "- `cophi.documet()` to process single text files.\n",
    "- `cophi.corpus()` to process a whole corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about the parameters?\n",
    "\n",
    "You can use Python’s built-in `help()` function to access `cophi`’s short and sweet documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function document in module cophi.api:\n",
      "\n",
      "document(filepath, **kwargs)\n",
      "    Read a text file and create a Document object.\n",
      "    \n",
      "    Parameter:\n",
      "        filepath (str): Path to the text file.\n",
      "        title (str): Describing title for the document. (optional).\n",
      "        lowercase (bool): If True, writes all letters in lowercase (optional).\n",
      "        n (int): Number of tokens per ngram (optional).\n",
      "        token_pattern (str): Regex pattern for one token (optional).\n",
      "        maximum (int): Stop tokenizing after that much tokens (optional).\n",
      "    \n",
      "    Returns:\n",
      "        A Document object.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(cophi.document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing a single text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have now a <class 'cophi.model.Document'>.\n"
     ]
    }
   ],
   "source": [
    "dickens = cophi.document(filepath=\"british-fiction-corpus/dickens_bleak.txt\",\n",
    "                         title=\"dickens-bleak\",\n",
    "                         lowercase=True,\n",
    "                         n=2,\n",
    "                         token_pattern=r\"\\p{L}+\\p{P}?\\p{L}+\",\n",
    "                         maximum=1000)\n",
    "\n",
    "print(\"You have now a {}.\".format(type(dickens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `help()` to learn about the properties of your new object (this is only a subset):\n",
    "\n",
    "```python\n",
    ">>> help(dickens)\n",
    "```\n",
    "```\n",
    "Help on Document in module cophi.model object:\n",
    "\n",
    "class Document(builtins.object)\n",
    " |  Model class for a Document.\n",
    " |  \n",
    " |  Attributes:\n",
    " |      text (str): Content of the text file.\n",
    " |      title (str): Describing title for the document.\n",
    " |      lowercase (bool): If True, all letters are lowercase.\n",
    " |      n (int): Number of words per ngram.\n",
    " |      token_pattern (str): Regex pattern for one token.\n",
    " |      maximum (int): Stopped tokenizing after that much tokens.\n",
    " |      tokens (list): Tokenized content of the document.\n",
    " |  ----------------------------------------------------------------------\n",
    " |  Data descriptors defined here:\n",
    " |  \n",
    " |  bow\n",
    " |      Bag-of-words representation.\n",
    " |  \n",
    " |  freq_spectrum\n",
    " |      Frequency spectrum.\n",
    " |  \n",
    " |  hapax\n",
    " |      Hapax legomena.\n",
    " |  \n",
    " |  ngrams\n",
    " |      Constructed ngrams.\n",
    " |  \n",
    " |  num_tokens\n",
    " |      Number of tokens.\n",
    " |  \n",
    " |  num_types\n",
    " |      Number of types.\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Access ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The return value is a <class 'generator'>.\n"
     ]
    }
   ],
   "source": [
    "ngrams = dickens.ngrams\n",
    "\n",
    "print(\"The return value is a {}.\".format(type(ngrams)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can e.g. iterate over this generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bleak house\n",
      "house by\n",
      "by charles\n",
      "charles dickens\n"
     ]
    }
   ],
   "source": [
    "for ngram in ngrams:\n",
    "    print(ngram)\n",
    "    if \"dickens\" in ngram:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing a corpus\n",
    "\n",
    "With `cophi.corpus()`, you can pipe a collection of text files through several NLP tasks, and get a document-term matrix with useful attributes and methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic usage\n",
    "\n",
    "Depending on the size of the text collection, this may take a while. These are the basic steps of the pipeline:\n",
    "\n",
    "1. read,\n",
    "2. get metadata,\n",
    "3. tokenize,\n",
    "4. count tokens,\n",
    "5. arrange in a document-term matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus, metadata = cophi.corpus(directory=\"british-fiction-corpus\",\n",
    "                                filepath_pattern=\"**/*.txt\",\n",
    "                                treat_as=\".txt\",\n",
    "                                encoding=\"utf-8\",\n",
    "                                lowercase=True,\n",
    "                                token_pattern=r\"\\p{L}+\\p{P}?\\p{L}+\",\n",
    "                                maximum=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One useful attribute would be `stats`, which gives you information about some statistics of your corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "documents         10\n",
       "types          33050\n",
       "tokens       1310189\n",
       "hapax          16495\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the document-term matrix with the attribute `dtm`, which returns a pandas `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>d478779c-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779d-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779e-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a'mighty  a'most  a'n  a'ready  a-bakin  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64         0       0    0        0        0   \n",
       "d478779d-aa14-11e8-b467-002710199d64         0       1    0        0        0   \n",
       "d478779e-aa14-11e8-b467-002710199d64         1       1    0        2        0   \n",
       "\n",
       "                                      a-be  a-beatin  a-bed  a-begging  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64     0         0      0          0   \n",
       "d478779d-aa14-11e8-b467-002710199d64     0         0      0          0   \n",
       "d478779e-aa14-11e8-b467-002710199d64     0         1      0          0   \n",
       "\n",
       "                                      a-bleatin  \n",
       "d478779c-aa14-11e8-b467-002710199d64          0  \n",
       "d478779d-aa14-11e8-b467-002710199d64          0  \n",
       "d478779e-aa14-11e8-b467-002710199d64          0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.dtm.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `cophi.corpus()` returned not only the `Corpus` object but also a `Metadata` object with metadata extracted from the filenames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filepath</th>\n",
       "      <th>parent</th>\n",
       "      <th>suffix</th>\n",
       "      <th>title</th>\n",
       "      <th>uuid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>british-fiction-corpus/trollope_phineas.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>trollope_phineas</td>\n",
       "      <td>d478779c-aa14-11e8-b467-002710199d64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>british-fiction-corpus/dickens_bleak.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>dickens_bleak</td>\n",
       "      <td>d478779d-aa14-11e8-b467-002710199d64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>british-fiction-corpus/eliot_mill.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>eliot_mill</td>\n",
       "      <td>d478779e-aa14-11e8-b467-002710199d64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filepath                  parent suffix  \\\n",
       "0  british-fiction-corpus/trollope_phineas.txt  british-fiction-corpus   .txt   \n",
       "1     british-fiction-corpus/dickens_bleak.txt  british-fiction-corpus   .txt   \n",
       "2        british-fiction-corpus/eliot_mill.txt  british-fiction-corpus   .txt   \n",
       "\n",
       "              title                                  uuid  \n",
       "0  trollope_phineas  d478779c-aa14-11e8-b467-002710199d64  \n",
       "1     dickens_bleak  d478779d-aa14-11e8-b467-002710199d64  \n",
       "2        eliot_mill  d478779e-aa14-11e8-b467-002710199d64  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.iloc[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily add additional metadata to the existing object (which is basically a pandas `DataFrame`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filepath</th>\n",
       "      <th>parent</th>\n",
       "      <th>suffix</th>\n",
       "      <th>title</th>\n",
       "      <th>uuid</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>british-fiction-corpus/trollope_phineas.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>trollope_phineas</td>\n",
       "      <td>d478779c-aa14-11e8-b467-002710199d64</td>\n",
       "      <td>1868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>british-fiction-corpus/dickens_bleak.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>dickens_bleak</td>\n",
       "      <td>d478779d-aa14-11e8-b467-002710199d64</td>\n",
       "      <td>1853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>british-fiction-corpus/eliot_mill.txt</td>\n",
       "      <td>british-fiction-corpus</td>\n",
       "      <td>.txt</td>\n",
       "      <td>eliot_mill</td>\n",
       "      <td>d478779e-aa14-11e8-b467-002710199d64</td>\n",
       "      <td>1860</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filepath                  parent suffix  \\\n",
       "0  british-fiction-corpus/trollope_phineas.txt  british-fiction-corpus   .txt   \n",
       "1     british-fiction-corpus/dickens_bleak.txt  british-fiction-corpus   .txt   \n",
       "2        british-fiction-corpus/eliot_mill.txt  british-fiction-corpus   .txt   \n",
       "\n",
       "              title                                  uuid  year  \n",
       "0  trollope_phineas  d478779c-aa14-11e8-b467-002710199d64  1868  \n",
       "1     dickens_bleak  d478779d-aa14-11e8-b467-002710199d64  1853  \n",
       "2        eliot_mill  d478779e-aa14-11e8-b467-002710199d64  1860  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata[\"year\"] = [1868, 1853, 1860, 1799, 1742, 1749, 1844, 1850, 1876, 1848]\n",
    "metadata.iloc[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you can save it with its pandas `to_csv()` method, edit or enrich it with a spreadsheet program like Excel, and read it again with `pd.read_csv()`.\n",
    "\n",
    "> In case you want to map metadata to the document-term matrix, be aware that each line in the document-term matrix must have a unique index value.\n",
    "\n",
    "The `Corpus` object also has a static method for mapping metadata (based on a [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier)) to the document-term matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1868_trollope_phineas</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1853_dickens_bleak</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1860_eliot_mill</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       a'mighty  a'most  a'n  a'ready  a-bakin  a-be  \\\n",
       "1868_trollope_phineas         0       0    0        0        0     0   \n",
       "1853_dickens_bleak            0       1    0        0        0     0   \n",
       "1860_eliot_mill               1       1    0        2        0     0   \n",
       "\n",
       "                       a-beatin  a-bed  a-begging  a-bleatin  \n",
       "1868_trollope_phineas         0      0          0          0  \n",
       "1853_dickens_bleak            0      0          0          0  \n",
       "1860_eliot_mill               1      0          0          0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.map_metadata(data=corpus.dtm,\n",
    "                    metadata=metadata,\n",
    "                    uuid=\"uuid\",  # connection between metadata and document\n",
    "                    fields=[\"year\", \"title\"],  # year and title as document descriptor\n",
    "                    sep=\"_\").iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced usage\n",
    "\n",
    "The `Corpus` object has some useful methods and attributes. In addition to various normalized or standardized document-term matrices, there are a multitude of lexical complexity measures – by document or corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Corpus vocabulary (or types)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"a'mighty\",\n",
       " \"a'most\",\n",
       " \"a'n\",\n",
       " \"a'ready\",\n",
       " 'a-bakin',\n",
       " 'a-be',\n",
       " 'a-beatin',\n",
       " 'a-bed',\n",
       " 'a-begging',\n",
       " 'a-bleatin']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.types[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Most frequent words\n",
    "\n",
    "You can return a list of the tokens itself (with `as_list=True`, which is default):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'and', 'to', 'of', 'in', 'he', 'was', 'that', 'his', 'it']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.mfw(n=10,\n",
    "           as_list=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or with their frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "the     0.498177\n",
       "and     0.333683\n",
       "to      0.314256\n",
       "of      0.276484\n",
       "in      0.172018\n",
       "he      0.155096\n",
       "was     0.149235\n",
       "that    0.144934\n",
       "his     0.121167\n",
       "it      0.113929\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mfw = corpus.mfw(n=10,\n",
    "                 as_list=False)\n",
    "mfw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because this is a pandas `Series`, you can easily plot the frequencies (first using `sort_values()`, that most common word is at the top of the plot):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fedb48c5390>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD8CAYAAACW/ATfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAENpJREFUeJzt3X+MZWV9x/H3R6TiCC6JGCNqHKE0BKhiuVIRMf7gD20p/oC2Vk1Ka91atBRRWxKJ07Ga0mpotf7qqo1tQ9S6tpX6g2hVKkUR7+jyY1VaERpxKi20IDCAgN/+MRedbGd37uyce8/uPu9Xsplzz33ueb5P7u5nnznn3PukqpAk7dse1HcBkqTJM+wlqQGGvSQ1wLCXpAYY9pLUAMNekhpg2EtSAwx7SWqAYS9JDXhw3wU84JBDDqnZ2dm+y5CkvcrCwsLNVfXItdrtMWE/OzvLcDjsuwxJ2qsk+Y9x2nkaR5IaYNhLUgMMe0lqgGEvSQ3YYy7QLi4uMj8/33cZkjRVc3NzU+nHmb0kNWBdYZ/k4CRnjrafmeQTkylLktSl9c7sDwbOnEQhkqTJWe85+/OBw5NsA+4F7kyyFTgGWABeVlWV5DjgAuBA4GbgjKr6zw7rliStw3pn9ucC11XVscDrgScDZwNHAYcBJybZH/gL4PSqOg74K+Atqx0syeYkwyTDpaWl3R2DJGkNG70b54qquhFgNNufBW5leab/2SQA+wGrzuqraguwBeDQQw+tDdYiSdqJjYb9PSu27x8dL8D2qjphg8eWJHVkvadxbgcOWqPNtcAjk5wAkGT/JEfvTnGSpG6sa2ZfVbckuSzJNcBdwE2rtPlhktOBdyTZNOrjz4HtXRQsSVq/dZ/GqaqX7GT/q1dsbwOesYG6JEkdStWecV10MBiU32cvSeuTZKGqBmu18+sSJKkBhr0kNcCwl6QGGPaS1ADDXpIaYNhLUgMMe0lqgGEvSQ0w7CWpAYa9JDVgo19x3JnFxUXm5+f7LkPap8zNzfVdgvYQU5nZJ7ljGv1IklbnaRxJasDYYZ/kH5MsJNmeZPNo3x1J3pLkyiSXJ3nUaP8Tknw5ydVJ3jyp4iVJ41nPzP43RwuID4CzkjwCeBhweVU9Cfgi8IpR27cD76mqn2Un689KkqZnPWF/VpIrgcuBxwFHAD8EPjF6foHlBccBTgQ+NNr+250dMMnmJMMkw6WlpfXULUlah7HCPskzgZOBE0az+K8DBwD31k9WP3lgwfEHrLkqSlVtqapBVQ1mZmbWVbgkaXzjzuw3Af9bVUtJjgSeukb7y4AXj7ZfurvFSZK6MW7YXww8OMk3gfNZPpWzK78HvCrJ1cBjNlCfJKkDY32oqqruAZ63ylMHrmizFdg62r4eOGFFu/M2UKMkaYNccFyS9mIuOC5J+jHDXpIaYNhLUgMMe0lqgGEvSQ0w7CWpAYa9JDXAsJekBhj2ktQAw16SGuCC49JexAXEtbuc2UtSAzoL+yQHJzmzq+NJkrrT5cz+YMCwl6Q9UJfn7M8HDk+yDfjsaN/zWF6e8M1V9ZEO+5IkrUOXM/tzgeuq6liWV7I6FngSy2vXvjXJozvsS5K0DpO6QPt04ENVdX9V3QT8C/CUHRsl2ZxkmGS4tLQ0oVIkSb3ejVNVW6pqUFWDmZmZPkuRpH1al2F/O3DQaPtS4FeT7JfkkcAzgCs67EuStA6dXaCtqluSXJbkGuDTwFXAlSxfoP39qvp+V31Jktan00/QVtVLdtj1+i6PL0naPamqvmsAYDAY1HA47LsMSdqrJFmoqsFa7fy6BElqgGEvSQ0w7CWpAYa9JDXAsJekBhj2ktQAw16SGmDYS1IDDHtJaoBhL0kN6PS7cTZicXGR+fn5vsuQdtvc3FzfJUg75cxekhow8bBPclaSbya5cNJ9SZJWN43TOGcCJ1fVjVPoS5K0ik5n9knOSXLN6M/ZSd4LHAZ8OslruuxLkjS+zmb2SY4DfgP4eSDAV4CXAc8FnlVVN6/yms3AZoBNmzZ1VYokaQddzuyfDvxDVd1ZVXcAfw+ctKsXuOC4JE2Hd+NIUgO6DPtLgRckmUnyMOCFo32SpJ51ds6+qr6W5IPAFaNd76+qryfpqgtJ0m7q9NbLqroAuGCHfbNd9iFJWr9UVd81ADAYDGo4HPZdhiTtVZIsVNVgrXZeoJWkBhj2ktQAw16SGmDYS1IDDHtJaoBhL0kNMOwlqQGGvSQ1wLCXpAYY9pLUgGksSziWxcVF5ufn+y5Du2Fubq7vEiStwZm9JDWg6zVov9Tl8SRJ3eg07KvqaV0eT5LUja5n9neMfj4zySVJtib5VpIL4yomktSbSZ6zfzJwNnAUcBhw4o4NkmxOMkwyXFpammApktS2SYb9FVV1Y1X9CNgGzO7YoKq2VNWgqgYzMzMTLEWS2jbJsL9nxfb97EG3eUpSa7z1UpIaYNhLUgM6PbVSVQeOfl4CXLJi/6u77EeStD6pqr5rAGAwGNRwOOy7DEnaqyRZqKrBWu08jSNJDTDsJakBhr0kNcCwl6QGGPaS1ADDXpIaYNhLUgMMe0lqgGEvSQ0w7CWpAXvM1w4vLi4yPz/fdxkamZub67sESR3a8Mw+yWySa7ooRpI0GZ7GkaQGdBX2+yV5X5LtST6T5KFJDk9ycZKFJJcmObKjviRJ69RV2B8BvKuqjgZuBU4DtgC/W1XHAa8D3t1RX5KkderqAu31VbVttL3A8uLiTwM+muSBNg/Z8UVJNgObATZt2tRRKZKkHXUV9jsuLv4o4NaqOnZXL6qqLSz/BsChhx66Z6yiIkn7oEldoP0BcH2SXwbIsidNqC9J0homeTfOS4GXJ7kS2A48f4J9SZJ2YcOncarqBuCYFY/ftuLp5270+JKkjXPBcUnai7nguCTpxwx7SWqAYS9JDTDsJakBhr0kNcCwl6QGGPaS1ADDXpIaYNhLUgMMe0lqgAuOC3CBcWlf58xekhpg2EtSAwx7SWrAmmGf5PVJzhpt/1mSz4+2n53kwiTvSTJMsj3J/IrXnZ/kG0muSvK2nR1fkjR541ygvRR4LfAOYAA8JMn+wEnAF4GPVtX/JNkP+FySJwLfA14IHFlVleTgyZQvSRrHOKdxFoDjkjyc5YXFv8xy6J/E8n8Ev5Lka8DXgaOBo4DbgLuBDyR5EbC02oGTbB79VjBcWlq1iSSpA2uGfVXdC1wPnAF8ieWAfxbw08BdwOuA51TVE4FPAgdU1X3A8cBW4BTg4p0ce0tVDapqMDMzs/HRSJJWNe4F2ktZDvUvjrZfyfJM/uHAncBtSR4FPA8gyYHApqr6FPAa4Ekd1y1JWodxP1R1KfAG4MtVdWeSu4FLq+rKJF8HvgV8F7hs1P4g4ONJDgACnNNx3ZKkdRgr7Kvqc8D+Kx7/zIrtM3bysuM3VJkkqTOpqr5rAGAwGNRwOOy7DEnaqyRZqKrBWu38UJUkNcCwl6QGGPaS1ADDXpIaYNhLUgMMe0lqgGEvSQ0w7CWpAYa9JDXAsJekBoz7RWgTt7i4yPz8/NoN1Zm5ubm+S5A0Jc7sJakBuwz7JAcnOXO0/cwkn1jPwZOckeTQjRQoSdq4tWb2BwNnbuD4ZwCGvST1bK1z9ucDhyfZBtwL3JlkK3AMy2vTvmy0oPgbgV8CHsry0oW/DZzG8lq1Fya5Czihqu6a0DgkSbuw1sz+XOC6qjoWeD3wZOBslhcVPww4cdTunVX1lKo6huXAP6WqtgJD4KVVdexqQe+C45I0Heu9QHtFVd1YVT8CtgGzo/3PSvKVJFcDzwaOHudgLjguSdOx3lsv71mxfT/w4NE6s+8GBlX13SR/CBzQUX2SpA6sNbO/neXFw3flgWC/OcmBwOnrfL0kacJ2ObOvqluSXJbkGuAu4KZV2tya5H3ANcD3ga+uePqDwHu9QCtJ/VrzNE5VvWQn+1+9Yvs84LxV2nwM+NhGCpQkbVyqqu8aABgMBjUcDvsuQ5L2KkkWqmqwVju/LkGSGmDYS1IDDHtJaoBhL0kNMOwlqQGGvSQ1wLCXpAYY9pLUAMNekhpg2EtSA9b7FccTs7i4yPz8fN9l7HXm5ub6LkHSXsCZvSQ1YLfDPsns6KuPd9z/piQnb6wsSVKXOj+NU1Vv7PqYkqSN2ehpnP2SvC/J9iSfSfLQJB9McjpAkvOTfCPJVUne1kG9kqTdsNGZ/RHAr1XVK5L8HXDaA08keQTwQuDIqqokB+/44iSbgc0AmzZt2mApkqSd2ejM/vqq2jbaXgBmVzx3G3A38IEkLwKWdnxxVW2pqkFVDWZmZjZYiiRpZzYa9ves2L6fFb8pVNV9wPHAVuAU4OIN9iVJ2k0Tu88+yYHATFV9KsllwHcm1Zckadcm+aGqg4CPJzkACHDOBPuSJO3Cbod9Vd0AHLPi8Wp32xy/u8eXJHUnVdV3DQAMBoMaDod9lyFJe5UkC1U1WKudX5cgSQ0w7CWpAYa9JDXAsJekBhj2ktQAw16SGmDYS1IDDHtJaoBhL0kNcMHxvYiLi0vaXc7sJakBEwv7JF8a/ZxN8pJJ9SNJWtvEwr6qnjbanAUMe0nq0SRn9neMNs8HTkqyLclrJtWfJGnnpnGB9lzgdVV1yhT6kiStotcLtEk2JxkmGS4t/b/1yCVJHek17KtqS1UNqmowMzPTZymStE+bRtjfzvJ6tJKknkwj7K8C7k9ypRdoJakfE7tAW1UHjn7eCzx7Uv1IktbmguOStBdzwXFJ0o8Z9pLUAMNekhpg2EtSAwx7SWrAHnM3TpLbgWv7rqNHhwA3911Ejxy/43f8u+fxVfXItRrtMStVAdeOc/vQvirJ0PE7/r7r6Ivjn/z4PY0jSQ0w7CWpAXtS2G/pu4CeOf62Of62TXz8e8wFWknS5OxJM3tJ0oRMPeyTPDfJtUm+neTcVZ5/SJKPjJ7/SpLZadc4SWOM/xlJvpbkviSn91HjJI0x/nOSfCPJVUk+l+TxfdQ5KWOM/5VJrh6t2fyvSY7qo85JWWv8K9qdlqSS7FN36Izx/p+R5L9H7/+2JL/VWedVNbU/wH7AdcBhwE8BVwJH7dDmTOC9o+0XAx+ZZo17wPhngScCfwOc3nfNPYz/WcDMaPt3Gnz/H75i+1Tg4r7rnub4R+0OAr4IXA4M+q57yu//GcA7J9H/tGf2xwPfrqrvVNUPgQ8Dz9+hzfOBvx5tbwWekyRTrHGS1hx/Vd1QVVcBP+qjwAkbZ/xfqKoHFiS+HHjslGucpHHG/4MVDx8G7EsX1cb59w/wR8CfAHdPs7gpGHf8EzHtsH8M8N0Vj28c7Vu1TVXdB9wGPGIq1U3eOOPfl613/C8HPj3RiqZrrPEneVWS64A/Bc6aUm3TsOb4k/wc8Liq+uQ0C5uScf/+nzY6jbk1yeO66twLtNojJXkZMADe2nct01ZV76qqw4E/AM7ru55pSfIg4ALgtX3X0qN/Amar6onAZ/nJWY4Nm3bYfw9Y+T/VY0f7Vm2T5MHAJuCWqVQ3eeOMf1821viTnAy8ATi1qu6ZUm3TsN73/8PACyZa0XStNf6DgGOAS5LcADwVuGgfuki75vtfVbes+Dv/fuC4rjqfdth/FTgiyROS/BTLF2Av2qHNRcCvj7ZPBz5foysX+4Bxxr8vW3P8SZ4M/CXLQf9fPdQ4SeOM/4gVD38R+Pcp1jdpuxx/Vd1WVYdU1WxVzbJ8zebUqtpX1isd5/1/9IqHpwLf7Kz3Hq5I/wLwbyxflX7DaN+bWH5TAQ4APgp8G7gCOKzvq+hTHv9TWD6XdyfLv9Fs77vmKY//n4GbgG2jPxf1XfOUx/92YPto7F8Aju675mmOf4e2l7AP3Y0z5vv/x6P3/8rR+39kV337CVpJaoAXaCWpAYa9JDXAsJekBhj2ktQAw16SGmDYS1IDDHtJaoBhL0kN+D90hjqAtXb02AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "mfw.sort_values().plot.barh(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hapax legomena\n",
    "\n",
    "In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, or in this case in a single text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"a'mighty\",\n",
       " \"a'n\",\n",
       " 'a-bakin',\n",
       " 'a-beatin',\n",
       " 'a-begging',\n",
       " 'a-bleatin',\n",
       " 'a-bringin',\n",
       " 'a-carrying',\n",
       " 'a-collectin',\n",
       " 'a-comin']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.hapax[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Drop features from document-term matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A total of 16595 features will be dropped.\n"
     ]
    }
   ],
   "source": [
    "features = corpus.mfw(n=100) + corpus.hapax\n",
    "print(\"A total of {} features will be dropped.\".format(len(features)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size before: 33050\n"
     ]
    }
   ],
   "source": [
    "print(\"Vocabulary size before: {}\".format(corpus.stats[\"types\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size after: 16455\n"
     ]
    }
   ],
   "source": [
    "dtm = corpus.drop(dtm=corpus.dtm,\n",
    "                  features=features)\n",
    "print(\"Vocabulary size after: {}\".format(dtm.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Culling\n",
    "Remove features that do not appear in a minimum of documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a-going</th>\n",
       "      <th>abandon</th>\n",
       "      <th>abandoned</th>\n",
       "      <th>abated</th>\n",
       "      <th>abhorrence</th>\n",
       "      <th>abide</th>\n",
       "      <th>abilities</th>\n",
       "      <th>ability</th>\n",
       "      <th>able</th>\n",
       "      <th>abode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>d478779c-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>NaN</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>45</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779d-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>21</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779e-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a-going  abandon  abandoned  abated  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64      NaN     14.0        1.0     NaN   \n",
       "d478779d-aa14-11e8-b467-002710199d64      4.0      3.0        2.0     1.0   \n",
       "d478779e-aa14-11e8-b467-002710199d64      NaN      NaN        NaN     2.0   \n",
       "\n",
       "                                      abhorrence  abide  abilities  ability  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64         NaN    NaN        NaN      3.0   \n",
       "d478779d-aa14-11e8-b467-002710199d64         1.0    NaN        2.0      1.0   \n",
       "d478779e-aa14-11e8-b467-002710199d64         1.0    1.0        1.0      NaN   \n",
       "\n",
       "                                      able  abode  \n",
       "d478779c-aa14-11e8-b467-002710199d64    45    1.0  \n",
       "d478779d-aa14-11e8-b467-002710199d64    21    2.0  \n",
       "d478779e-aa14-11e8-b467-002710199d64    14    NaN  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "culled = corpus.cull(dtm=corpus.dtm,\n",
    "                     threshold=5,  # must appear in at least 5 documents\n",
    "                     keepna=True)\n",
    "culled.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Frequency spectrum\n",
    "\n",
    "How often does a frequency occur in the corpus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1     12616\n",
       "2      4508\n",
       "3      2627\n",
       "4      1760\n",
       "5      1253\n",
       "6      1012\n",
       "7       863\n",
       "8       586\n",
       "9       519\n",
       "10      467\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "freq_spectrum = corpus.freq_spectrum[:10]\n",
    "freq_spectrum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting is easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fedb349bcc0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEIBJREFUeJzt3X2MZXV9x/H3p6yIg0aelLBAu5gaUmOi4I2B+pAWBJESbBv/gGjFh3aTtmnBNjEQ/5jMf7U1Rk0bdeNjWsQHhGpIfKCINSZ27V1EXFgQfIYBF7SKMkkF/faPe5YOm52dvXPO3Nn58X4lN3PuOWfu7/e7v5nPnPmdc88vVYUkafP7rY2ugCRpGAa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqRFbZlnYCSecUNu2bZtlkZK06e3ateuhqnrWavvNNNC3bdvGeDyeZZGStOkl+cGh7OeQiyQ1wkCXpEYY6JLUCANdkhox05Oii4uLLCwszLJISdpw8/PzMynHI3RJakSvQE/yoSR7k+weqkKSpLXpe4T+EeCCAeohSeqpV6BX1VeAnw5UF0lSD+s+hp5ke5JxkvHS0tJ6FydJT1rrHuhVtaOqRlU1mpubW+/iJOlJy6tcJKkRBrokNaLvZYvXAF8DTk9yb5I3D1MtSdK0en1StKouHaoikqR+UlUzK2w0GpX3Q5ek6STZVVWj1fZzDF2SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSI3rdy2Vai4uLLCwszLJIrYNZzWAuaTprPkJPcmqSm5PckeT2JJcPWTFJ0nT6HKE/Bvx9Vd2S5BnAriQ3VtUdA9VNkjSFNR+hV9X9VXVLt/wLYA9w8lAVkyRNZ5CTokm2AWcAO4d4PUnS9HoHepKnA58Grqiqhw+wfXuScZLx0tJS3+IkSSvoOwXdU5iE+dVVdd2B9qmqHVU1qqrR3Nxcn+IkSQfR5yqXAB8E9lTVO4erkiRpLfocob8E+DPgnCS3do8LB6qXJGlKa75ssaq+CmTAukiSenCSaEk6zDlJtCQ9yRjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCCeJ3gBOsixpPXiELkmN6HM/9KOSfD3JN5PcnsRDb0naQH2GXP4XOKeqftnNXPTVJJ+rqv8aqG6SpCn0uR96Ab/snj6le8zuXrySpCfoO6foEUluBfYCN1bVzmGqJUmaVq9Ar6pfV9ULgVOAFyd5/v77JNmeZJxkvLS01Kc4SdJBDHKVS1X9DLgZuOAA23ZU1aiqRnNzc0MUJ0k6gD5XuTwryTHd8tOA84A7h6qYJGk6fa5yOQn4aJIjmPxh+GRV3TBMtSRJ0+pzlcttwBkD1kWS1EMmVx/Oxmg0qvF4PLPyJKkFSXZV1Wi1/fzovyQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RG9Lk519QWFxdZWNg8U4/Oz89vdBUk6ZB5hC5Jjegd6N00dN9I4q1zJWkDDXGEfjmwZ4DXkST10HeS6FOAPwI+MEx1JElr1fcI/V3AW4HfrLSDk0RL0mz0mVP0ImBvVe062H5OEi1Js9HnCP0lwMVJvg98HDgnyb8NUitJ0tTWHOhVdVVVnVJV24BLgC9V1esGq5kkaSpehy5JjRjkk6JV9WXgy0O8liRpbVJVMytsNBrVeDyeWXmS1IIku6pqtNp+DrlIUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJasQg93I5VIuLiywsLMyyyMfNz89vSLmSNCseoUtSI3odoXeTW/wC+DXw2KHcPEaStD6GGHL5w6p6aIDXkST14JCLJDWib6AX8MUku5JsP9AOSbYnGScZLy0t9SxOkrSSvkMuL62q+5I8G7gxyZ1V9ZXlO1TVDmAHwNatW2c3m4YkPcn0OkKvqvu6r3uB64EXD1EpSdL01hzoSY5O8ox9y8D5wO6hKiZJmk6fIZcTgeuT7Hudj1XV5weplSRpamsO9Kr6LvCCAesiSeohVbM7TzkajWo8Hs+sPElqQZJdh/LBTa9Dl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktSIIaagO2SLi4ssLCysaxnz8/Pr+vqSdLjqdYSe5Jgk1ya5M8meJGcPVTFJ0nT6HqG/G/h8Vb0myZHA3AB1kiStwZoDPckzgZcDbwCoql8BvxqmWpKkafUZcjkNeBD4cJJvJPlAN3ORJGkD9An0LcCZwHur6gzgEeDK/XdKsj3JOMl4aWmpR3GSpIPpE+j3AvdW1c7u+bVMAv4JqmpHVY2qajQ35xC7JK2XNQd6VT0A/CjJ6d2qc4E7BqmVJGlqfa9y+Rvg6u4Kl+8Cb+xfJUnSWvQK9Kq6FVh1njtJ0vpzkmhJOsw5SbQkPckY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1IgmJol2YmhJ8ghdkpqx5kBPcnqSW5c9Hk5yxZCVkyQdujUPuVTVXcALAZIcAdwHXD9QvSRJUxpqyOVc4DtV9YOBXk+SNKWhAv0S4JqBXkuStAa9A72bfu5i4FMrbN+eZJxkvLS01Lc4SdIKhjhCfxVwS1X9+EAbq2pHVY2qajQ3NzdAcZKkAxki0C/F4RZJ2nC9Aj3J0cB5wHXDVEeStFa9PilaVY8Axw9UF0lSD6mqmRU2Go1qPB7PrDxJakGSXVU1Wm0/P/ovSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqRG97uUyrcXFRRYWFtb0vfPz8wPXRpLa4hG6JDWi7+1z35Lk9iS7k1yT5KihKiZJms6aAz3JycDfAqOqej5wBJO5RSVJG6DvkMsW4GlJtgBzwGL/KkmS1mLNgV5V9wHvAH4I3A/8vKq+uP9+ThItSbPRZ8jlWODVwGnAVuDoJK/bfz8niZak2egz5PIK4HtV9WBVPcpkXtHfH6ZakqRp9Qn0HwJnJZlLEuBcYM8w1ZIkTavPGPpO4FrgFuBb3WvtGKhekqQp9fqkaFXNA36EU5IOA6mqmRU2Go1qPB7PrDxJakGSXVU1Wm0/P/ovSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqRG97uUyrcXFRRYWFqb6nvl5bxUjSYfCI3RJakSvQE9yeZLdSW5PcsVQlZIkTa/PFHTPB/4CeDHwAuCiJL87VMUkSdPpc4T+e8DOqlqqqseA/wT+dJhqSZKm1SfQdwMvS3J8kjngQuDU/XdKsj3JOMl4aWmpR3GSpINZ81UuVbUnyduBLwKPALcCvz7AfjvopqbbunXr7GbTkKQnmV4nRavqg1X1oqp6OfA/wLeHqZYkaVq9rkNP8uyq2pvkt5mMn581TLUkSdPq+8GiTyc5HngU+Ouq+tkAdZIkrUGvQK+qlw1VEUlSP6ma3XnK0WhU4/F4ZuVJUguS7Kqq0Wr7+dF/SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqxGE9SbQTREvSofMIXZIasWqgJ/lQkr1Jdi9bd1ySG5Pc3X09dn2rKUlazaEcoX8EuGC/dVcCN1XVc4GbuueSpA20aqBX1VeAn+63+tXAR7vljwJ/PHC9JElTWusY+olVdX+3/ABw4kD1kSStUe+TojW5ofqKN1VPsj3JOMl4aWmpb3GSpBWsNdB/nOQkgO7r3pV2rKodVTWqqtHc3Nwai5MkrWatgf5Z4LJu+TLgM8NUR5K0Vody2eI1wNeA05Pcm+TNwD8A5yW5G3hF91yStIFW/aRoVV26wqZzB66LJKkHJ4mWpMOck0RL0pOMgS5JjTDQJakRBrokNcJAl6RGzPQqlyS/AO6aWYHr5wTgoY2uxEBaaUsr7QDbcjja6Hb8TlU9a7WdZjpjEXDXoVx6c7hLMm6hHdBOW1ppB9iWw9FmaYdDLpLUCANdkhox60DfMePy1ksr7YB22tJKO8C2HI42RTtmelJUkrR+HHKRpEbMJNCTXJDkriT3JDksJ5ROcmqSm5PckeT2JJd3649LcmOSu7uvx3brk+Q9XZtuS3Lmste6rNv/7iSXrVTmOrfniCTfSHJD9/y0JDu7+n4iyZHd+qd2z+/ptm9b9hpXdevvSvLKDWrHMUmuTXJnkj1Jzt6MfZLkLd3P1e4k1yQ5arP0SZIPJdmbZPeydYP1QZIXJflW9z3vSZIZtuOfup+t25Jcn+SYZdsO+F6vlGcr9edMVdW6PoAjgO8AzwGOBL4JPG+9y11DPU8CzuyWnwF8G3ge8I/Ald36K4G3d8sXAp8DApwF7OzWHwd8t/t6bLd87Aa05++AjwE3dM8/CVzSLb8P+Mtu+a+A93XLlwCf6Jaf1/XVU4HTuj48YgPa8VHgz7vlI4FjNlufACcD3wOetqwv3rBZ+gR4OXAmsHvZusH6APh6t2+6733VDNtxPrClW377snYc8L3mIHm2Un/O9PdlBj8MZwNfWPb8KuCqWTd0DfX+DHAekw9CndStO4nJtfQA7wcuXbb/Xd32S4H3L1v/hP1mVPdTgJuAc4Abul+Uh5b94D7eJ8AXgLO75S3dftm/n5bvN8N2PJNJEGa/9ZuqT5gE+o+6MNvS9ckrN1OfANv2C8JB+qDbduey9U/Yb73bsd+2PwGu7pYP+F6zQp4d7Hdslo9ZDLns+2He595u3WGr+xf3DGAncGJV3d9tegA4sVteqV2HQ3vfBbwV+E33/HjgZ1X12AHq9Hh9u+0/7/Y/HNpxGvAg8OFu+OgDSY5mk/VJVd0HvAP4IXA/k/d4F5uzT/YZqg9O7pb3X78R3sTkPwSYvh0H+x2bGU+K7ifJ04FPA1dU1cPLt9XkT+9hfVlQkouAvVW1a6PrMoAtTP5Ffm9VnQE8wuTf+8dtkj45Fng1kz9QW4GjgQs2tFID2gx9sJokbwMeA67e6Lr0MYtAvw84ddnzU7p1h50kT2ES5ldX1XXd6h8nOanbfhKwt1u/Urs2ur0vAS5O8n3g40yGXd4NHJNk360eltfp8fp2258J/ISNbwdMjnLuraqd3fNrmQT8ZuuTVwDfq6oHq+pR4Dom/bQZ+2Sfofrgvm55//Uzk+QNwEXAa7s/TjB9O37Cyv05M7MI9P8GntudAT6SyUmez86g3Kl0Z9Y/COypqncu2/RZYN8Z+cuYjK3vW//67qz+WcDPu39BvwCcn+TY7sjs/G7dTFTVVVV1SlVtY/Jef6mqXgvcDLxmhXbsa99ruv2rW39Jd8XFacBzmZy8mpmqegD4UZLTu1XnAnewyfqEyVDLWUnmup+zfe3YdH2yzCB90G17OMlZ3Xvz+mWvte6SXMBkePLiqlpatmml9/qAedb1z0r9OTuzGKhncub720zODr9t1icKDrGOL2Xyb+NtwK3d40ImY2M3AXcD/wEc1+0f4F+6Nn0LGC17rTcB93SPN25gm/6A/7/K5TlMfiDvAT4FPLVbf1T3/J5u+3OWff/buvbdxTpdeXAIbXghMO765d+ZXCGx6foEWADuBHYD/8rk6olN0SfANUzG/h9l8l/Tm4fsA2DUvS/fAf6Z/U6Cr3M77mEyJr7vd/59q73XrJBnK/XnLB9+UlSSGuFJUUlqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1Ij/g/UUebtAZWSiAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "freq_spectrum.sort_values().plot.barh(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Document-term matrix sorted in descending order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>the</th>\n",
       "      <th>and</th>\n",
       "      <th>to</th>\n",
       "      <th>of</th>\n",
       "      <th>in</th>\n",
       "      <th>he</th>\n",
       "      <th>was</th>\n",
       "      <th>that</th>\n",
       "      <th>his</th>\n",
       "      <th>it</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>d478779c-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>6412</td>\n",
       "      <td>4226</td>\n",
       "      <td>5175</td>\n",
       "      <td>4004</td>\n",
       "      <td>2416</td>\n",
       "      <td>3667</td>\n",
       "      <td>2424</td>\n",
       "      <td>3373</td>\n",
       "      <td>2094</td>\n",
       "      <td>1854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779d-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>4918</td>\n",
       "      <td>3706</td>\n",
       "      <td>2928</td>\n",
       "      <td>2642</td>\n",
       "      <td>1909</td>\n",
       "      <td>1387</td>\n",
       "      <td>1194</td>\n",
       "      <td>1389</td>\n",
       "      <td>1079</td>\n",
       "      <td>1464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779e-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>4053</td>\n",
       "      <td>2979</td>\n",
       "      <td>2701</td>\n",
       "      <td>2216</td>\n",
       "      <td>1389</td>\n",
       "      <td>1135</td>\n",
       "      <td>1456</td>\n",
       "      <td>1139</td>\n",
       "      <td>1065</td>\n",
       "      <td>1129</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       the   and    to    of    in    he  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64  6412  4226  5175  4004  2416  3667   \n",
       "d478779d-aa14-11e8-b467-002710199d64  4918  3706  2928  2642  1909  1387   \n",
       "d478779e-aa14-11e8-b467-002710199d64  4053  2979  2701  2216  1389  1135   \n",
       "\n",
       "                                       was  that   his    it  \n",
       "d478779c-aa14-11e8-b467-002710199d64  2424  3373  2094  1854  \n",
       "d478779d-aa14-11e8-b467-002710199d64  1194  1389  1079  1464  \n",
       "d478779e-aa14-11e8-b467-002710199d64  1456  1139  1065  1129  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.sort(dtm=corpus.dtm).iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Document-term matrix with z-scores\n",
    "\n",
    "Used formula is: $z_x = \\frac{x - \\mu}{\\sigma}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>d478779c-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>-0.474342</td>\n",
       "      <td>-0.650791</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.414039</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.444478</td>\n",
       "      <td>-0.474342</td>\n",
       "      <td>-0.316228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779d-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>-0.474342</td>\n",
       "      <td>0.162698</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.414039</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.444478</td>\n",
       "      <td>-0.474342</td>\n",
       "      <td>-0.316228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779e-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>1.897367</td>\n",
       "      <td>0.162698</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>0.621059</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>-0.316228</td>\n",
       "      <td>2.846050</td>\n",
       "      <td>-0.444478</td>\n",
       "      <td>-0.474342</td>\n",
       "      <td>-0.316228</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a'mighty    a'most       a'n   a'ready  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64 -0.474342 -0.650791 -0.316228 -0.414039   \n",
       "d478779d-aa14-11e8-b467-002710199d64 -0.474342  0.162698 -0.316228 -0.414039   \n",
       "d478779e-aa14-11e8-b467-002710199d64  1.897367  0.162698 -0.316228  0.621059   \n",
       "\n",
       "                                       a-bakin      a-be  a-beatin     a-bed  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64 -0.316228 -0.316228 -0.316228 -0.444478   \n",
       "d478779d-aa14-11e8-b467-002710199d64 -0.316228 -0.316228 -0.316228 -0.444478   \n",
       "d478779e-aa14-11e8-b467-002710199d64 -0.316228 -0.316228  2.846050 -0.444478   \n",
       "\n",
       "                                      a-begging  a-bleatin  \n",
       "d478779c-aa14-11e8-b467-002710199d64  -0.474342  -0.316228  \n",
       "d478779d-aa14-11e8-b467-002710199d64  -0.474342  -0.316228  \n",
       "d478779e-aa14-11e8-b467-002710199d64  -0.474342  -0.316228  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.zscores.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Document-term matrix with TF-IDF scores\n",
    "\n",
    "Used formula is: $tf-idf_{t,d} \\; = \\; tf_{t,d} \\times idf_t \\; = \\; tf_{t,d} \\times log(\\frac{N}{df_t})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a'mighty</th>\n",
       "      <th>a'most</th>\n",
       "      <th>a'n</th>\n",
       "      <th>a'ready</th>\n",
       "      <th>a-bakin</th>\n",
       "      <th>a-be</th>\n",
       "      <th>a-beatin</th>\n",
       "      <th>a-bed</th>\n",
       "      <th>a-begging</th>\n",
       "      <th>a-bleatin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>d478779c-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779d-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d478779e-aa14-11e8-b467-002710199d64</th>\n",
       "      <td>0.000017</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000035</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000025</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      a'mighty    a'most  a'n   a'ready  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64  0.000000  0.000000  0.0  0.000000   \n",
       "d478779d-aa14-11e8-b467-002710199d64  0.000000  0.000009  0.0  0.000000   \n",
       "d478779e-aa14-11e8-b467-002710199d64  0.000017  0.000010  0.0  0.000035   \n",
       "\n",
       "                                      a-bakin  a-be  a-beatin  a-bed  \\\n",
       "d478779c-aa14-11e8-b467-002710199d64      0.0   0.0  0.000000    0.0   \n",
       "d478779d-aa14-11e8-b467-002710199d64      0.0   0.0  0.000000    0.0   \n",
       "d478779e-aa14-11e8-b467-002710199d64      0.0   0.0  0.000025    0.0   \n",
       "\n",
       "                                      a-begging  a-bleatin  \n",
       "d478779c-aa14-11e8-b467-002710199d64        0.0        0.0  \n",
       "d478779d-aa14-11e8-b467-002710199d64        0.0        0.0  \n",
       "d478779e-aa14-11e8-b467-002710199d64        0.0        0.0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.tfidf.iloc[:3,:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Number of tokens by document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d478779c-aa14-11e8-b467-002710199d64    149926\n",
       "d478779d-aa14-11e8-b467-002710199d64     99570\n",
       "d478779e-aa14-11e8-b467-002710199d64     92012\n",
       "d478779f-aa14-11e8-b467-002710199d64    127439\n",
       "d47877a0-aa14-11e8-b467-002710199d64     87923\n",
       "d47877a1-aa14-11e8-b467-002710199d64    158439\n",
       "d47877a2-aa14-11e8-b467-002710199d64     97524\n",
       "d47877a3-aa14-11e8-b467-002710199d64    147059\n",
       "d5606214-aa14-11e8-b467-002710199d64    146348\n",
       "d5606215-aa14-11e8-b467-002710199d64    203949\n",
       "dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.num_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Number of types by document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d478779c-aa14-11e8-b467-002710199d64     7845\n",
       "d478779d-aa14-11e8-b467-002710199d64     9174\n",
       "d478779e-aa14-11e8-b467-002710199d64     8828\n",
       "d478779f-aa14-11e8-b467-002710199d64    10357\n",
       "d47877a0-aa14-11e8-b467-002710199d64     7574\n",
       "d47877a1-aa14-11e8-b467-002710199d64    10023\n",
       "d47877a2-aa14-11e8-b467-002710199d64     9298\n",
       "d47877a3-aa14-11e8-b467-002710199d64    10006\n",
       "d5606214-aa14-11e8-b467-002710199d64     7875\n",
       "d5606215-aa14-11e8-b467-002710199d64    14225\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.num_types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get a document by its UUID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'d478779c-aa14-11e8-b467-002710199d64'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "uuid = corpus.documents.index[0]  # first document\n",
    "uuid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select it and show its bag-of-words representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "phineas      954\n",
       "finn         354\n",
       "by           570\n",
       "anthony        1\n",
       "trollope       1\n",
       "volume         3\n",
       "chapter       51\n",
       "dr            29\n",
       "of          4004\n",
       "killaloe      33\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document = corpus.documents[uuid]\n",
    "document.bow[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or its tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['phineas',\n",
       " 'finn',\n",
       " 'by',\n",
       " 'anthony',\n",
       " 'trollope',\n",
       " 'volume',\n",
       " 'chapter',\n",
       " 'dr',\n",
       " 'finn',\n",
       " 'of']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.tokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get token lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([7, 4, 2, ..., 6, 2, 4])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the mean token length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.271233808678948"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.lengths.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sliding window\n",
    "\n",
    "You can iterate with a sliding window over your document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['phineas', 'finn', 'by', 'anthony', 'trollope', 'volume', 'chapter', 'dr', 'finn', 'of']\n"
     ]
    }
   ],
   "source": [
    "for chunk in document.window(size=10):\n",
    "    print(chunk)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get paragraphs or segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 13684 paragraphs.\n",
      "And 150 segments, 1000 tokens each segment.\n",
      "First paragraph: PHINEAS FINN\n"
     ]
    }
   ],
   "source": [
    "paragraphs = list(document.paragraphs(sep=\"\\n\"))\n",
    "segments = list(document.segments(size=1000))\n",
    "\n",
    "print(\"There are {} paragraphs.\".format(len(paragraphs)))\n",
    "print(\"And {} segments, 1000 tokens each segment.\".format(len(segments)))\n",
    "print(\"First paragraph: {}\".format(paragraphs[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Complexity metrics\n",
    "\n",
    "There are a plenty of complexity metrics for measuring lexical richness (adapted from [here](https://github.com/tsproisl/Linguistic_and_Stylistic_Complexity)).\n",
    "\n",
    "\n",
    "Measures that use sample size and vocabulary size:\n",
    "\n",
    "  * Type-token ratio\n",
    "  * Guiraud’s $R$\n",
    "  * Herdan’s $C$\n",
    "  * Dugast’s $k$\n",
    "  * Maas’ $a^2$\n",
    "  * Dugast’s $U$\n",
    "  * Tuldava’s $LN$\n",
    "  * Brunet’s $W$\n",
    "  * Carroll’s $CTTR$\n",
    "  * Summer’s $S$\n",
    "\n",
    "Measures that use part of the frequency spectrum:\n",
    "\n",
    "  * Honoré’s $H$\n",
    "  * Sichel’s $S$\n",
    "  * Michéa’s $M$\n",
    "\n",
    "Measures that use the whole frequency spectrum:\n",
    "\n",
    "  * Entropy\n",
    "  * Yule’s $K$\n",
    "  * Simpson’s $D$\n",
    "  * Herdan’s $V_m$\n",
    "\n",
    "Parameters of probabilistic models:\n",
    "\n",
    "  * Orlov’s $Z$\n",
    "\n",
    "\n",
    "You can access them for the _whole corpus_ using the corresponding attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.025225368248397752"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.ttr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by _document_ using the `complexity()` method. You _have_ to use a sliding window in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "sttr = corpus.complexity(measure=\"ttr\", window=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course you can also map metadata here again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sttr</th>\n",
       "      <th>ci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>trollope_phineas</th>\n",
       "      <td>0.307230</td>\n",
       "      <td>0.004331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dickens_bleak</th>\n",
       "      <td>0.359245</td>\n",
       "      <td>0.007024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eliot_mill</th>\n",
       "      <td>0.365087</td>\n",
       "      <td>0.007684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>eliot_adam</th>\n",
       "      <td>0.374310</td>\n",
       "      <td>0.006497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fielding_joseph</th>\n",
       "      <td>0.363512</td>\n",
       "      <td>0.004977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fielding_tom</th>\n",
       "      <td>0.359880</td>\n",
       "      <td>0.004151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thackeray_lyndon</th>\n",
       "      <td>0.378406</td>\n",
       "      <td>0.005436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dickens_david</th>\n",
       "      <td>0.346240</td>\n",
       "      <td>0.003996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trollope_prime</th>\n",
       "      <td>0.313212</td>\n",
       "      <td>0.004565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>thackeray_vanity</th>\n",
       "      <td>0.388614</td>\n",
       "      <td>0.003362</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      sttr        ci\n",
       "trollope_phineas  0.307230  0.004331\n",
       "dickens_bleak     0.359245  0.007024\n",
       "eliot_mill        0.365087  0.007684\n",
       "eliot_adam        0.374310  0.006497\n",
       "fielding_joseph   0.363512  0.004977\n",
       "fielding_tom      0.359880  0.004151\n",
       "thackeray_lyndon  0.378406  0.005436\n",
       "dickens_david     0.346240  0.003996\n",
       "trollope_prime    0.313212  0.004565\n",
       "thackeray_vanity  0.388614  0.003362"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sttr = corpus.map_metadata(sttr, metadata)\n",
    "sttr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And visualize the result e.g. as a barplot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fedb2e95eb8>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcEAAAD8CAYAAAAPKB8vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xu8XfOd//HXO6gkEkGpcRRBR91CIjthRIqWMlUVI5pOFel0Bj+XoD/V9JeO5Kh2qOlPtYqmRkP5lbpkZPi5hhBaTXauJ1KXuk05qTykbnEI4jN/7O/R5Tj7XPc5e5+z3s/H4zz22t/b+qzlsfPxXWvt/VVEYGZmlkcDqh2AmZlZtTgJmplZbjkJmplZbjkJmplZbjkJmplZbjkJmplZbjkJmplZbjkJmplZbjkJmplZbm1Y7QCsvC233DKGDx9e7TDMzPqURYsWvRwRW3WkrZNgDRs+fDjFYrHaYZiZ9SmSnu9oW18ONTOz3HISNDOz3HISNDOz3HISNDOz3PKDMTWssbGR+vr6aodhZtarpk+f3mv78kzQzMxyq8NJUNJmkk5N2wdJur27O6/UOLVC0lWSdk/b/6fa8ZiZWds6MxPcDDi1pwLpLEk1dyk3Iv45Ilamt06CZmY1rjNJ8EJgZ0lLgYuBIZJulvS4pOslCUDSeZIWSlohaWam/FOS7pO0TNJiSTtnB5c0RtISSTtL2kTS1ZIWpLKjUpvJkuZIuh+YK2mIpLlpvIZMu/MlnZUZ+/uSzmztoCTdIOmIzPtZkiZKGi5pfhp7saT9U/1BkuaVOfZ5kgqSLgQGSVqa6jscj5mZ9Z7OJMGpwNMRMRL4FjAKOAvYHdgJGJfaXRYRYyJiT2AQ8MVUfj3ws4jYG9gfWNU8cEowVwJHRcTTwDTg/ogYCxwMXCxpk9R8H2BiRBwIvA0cHRH7pHY/SgnpauCENPYA4CvAdWWO60bgy6ntx4DPAXcAq4FD09iTgJ9k+pQ7dgAiYirwVkSMjIjjOhOPpJMkFSUVm5qayoRsZmaV0J1Ligsi4gWANDscDjwMHCzpXGAwsAXwmKR5wLYRMRsgIt5O/QB2A2YCn4+IxjT254EvSTonvR8IbJ+2742Iv6RtAT+Q9BngfWBbYOuIeE7SGkmjgK2BJRGxpsxx3AlcKmlj4HDgoYh4S9Iw4DJJI4H1wC4dOPZWdSaeiJiZzgd1dXVRbkwzM+u+7iTBdZnt9cCGkgYClwOFiPiTpBmUElhbVqU2o4DmJCjgmIh4IttQ0r7Am5mi44CtgNER8a6k5zL7uwqYDPwNpZlYqyLi7ZSkD6M047shVZ0NvATsTWnG/Ham20eOvZ1j7HA8ZmbWezpzOfQNYGg7bZoT0MuShgATASLiDeAFSRMAJG0saXBq+ypwBPBvkg5KZXcDZ2TutY0qs79hwOqUAA8GdsjUzaY0sxuTxmvLjcDXgfHAXZmxV0XE+8DxwAbtjNHSu5I26mI8ZmbWCzqcBNPlu0ckraD0YExrbV4FfgGsoPQP/cJM9fHAFEnLgd9SmhE193uJ0r3Dn6XZ3veAjYDlkh5L71tzPVCQ1EDpntvjmTHfAR4AfhMR69s5vHuAA4H7Uj8ozWhPlLQM2JUPz0A7YmaK//ouxGNmZr1AEf3ztlN6AGUxcGxEPNUX46mrq4uTTz65ZwMzM6sx3f3FGEmLIqLQkbY19127SkhfWL8dmF0jCbBL8dTV1fXqzweZmeVNv0yC6QvrO2XLJI0AftWi6bqI2Lca8ZiZWfX1yyTYmohoAEZWOw4zM6sd/gFtMzPLLSdBMzPLLSdBMzPLLSdBMzPLLSdBMzPLLSdBMzPLLSdBMzPLLSdBMzPLrdx8Wb4vamxspL6+vtphmFmV+GcTe55ngmZmlls9ngQlbSbp1C70W5teh6flm2qGpFMknVDtOMzMrHt6Yya4GfCRJCipT16KlbRhRFwZEddWOxYzM+ue3kiCFwI7S1oqaaGk+ZLmACsBJH1T0or0d1ZbA0kaKOmXkhokLUmrySNpsqTbJM2T9JSk6Zk+X5O0IO3/55LKrhAvaa2kSyQ9JmmupK1S+TxJP5ZUBM6UNEPSOZm6SyQVJf1B0hhJt6Y4LuhKHGZm1jt6IwlOBZ6OiJHAt4B9gDMjYhdJo4GvA/sC+wH/ImlUG2OdBkREjAD+EbhG0sBUNxY4BtgLOFZSQdJuwCRgXNr/euC4NsbfBChGxB7Ag0D2rvTHIqIQET9qpd87aQHHK4HbUpx7ApMlfbwzcUg6KSXUYlNTUxuhmplZd1XjkuSCiHg2bR9AaaHZNwEk3QqMB5aU6XsA8FOAiHhc0vPALqnu3ohYkxnnAOA9YDSwUBLAIGB1G7G9D9yYtq8Dbs3U3fjR5h+Yk14bgMciYlWK4xlguxRLh+KIiJnATCitLN/GPs3MrJuqkQTf7KFxWyaMAARcExHfqcCYbcW9Lr2+n9lufr9hBeIwM7Me0BuXQ98Ahpapmw9MkDRY0ibA0amsnPmky4iSdgG2B55IdYdK2kLSIGAC8AgwF5go6ROpzxaSdmhj/AHAxLT9VeDh9g6ugzobh5mZ9YIenwlGxBpJj6SvObwFvJSpWyxpFrAgFV0VEeUuhQJcDlwhqYHSpc7JEbEuXWJcANwCfBK4LiKKAJK+C9wjaQDwLqX7dc+XGf9NYGzqs5rSfbxui4iVnYzDzMx6gSL6/m0nSZOBQkSc3s1x1kbEkMpE1X2FQiGKxWK1wzAz61MkLUoPK7bLvxhjZma51Se/sN5SRMwCZnW0vaTfAxu3KD6+lmaBZmbW8/pFEuysiNi32jGYmVn1+XKomZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnlVi6/J9hXNDY2Ul9fX+0wzCwHpk+f3n6jfsgzQTMzyy0nQTMzy62qJUFJMySdI+l8SYe00W6ypMt6M7YW+z9I0u1d7PslSVPL1K3tXmRmZtZdVb8nGBHnVTuGnhIRc4A51Y7DzMxa16szQUnTJD0p6WHg06lslqSJaXuMpN9KWiZpgaShLfofIel3kraUtJWkWyQtTH/jUpsZkq6WNE/SM5KmpPJNJN2Rxl4hqeyCuZIOl/S4pMXAP2TKx6b9L0lxNh/Do5L2yLSbJ6mQncVK2jH1bZB0QaXOqZmZdV2vJUFJo4GvACOBLwBjWtR/DLgRODMi9gYOobQSfXP90cBU4AsR8TJwKXBJRIwBjgGuygy3K3AYMBaYLmkj4HCgMSL2jog9gbvKxDkQ+AVwJDAa+JtM9ePA+IgYBZwH/CCV3wh8OfXfBtimeWX7jEuBKyJiBLCqjfN0kqSipGJTU1O5ZmZmVgG9ORMcD8yOiKaIeJ2PXib8NLAqIhYCRMTrEfFeqvss8G3giIh4JZUdAlwmaWkaa1NJzesB3hER61KyXA1sDTQAh0q6SNL4iHitTJy7As9GxFMREcB1mbphwE2SVgCXAM2zv98AE9P2l4GbWxl3HPDrtP2rMvsmImZGRCEiCoMHDy7XzMzMKqCvPB36NDAU2CVTNgDYLyJGpr9tI6L5YZN1mXbrgQ0j4klgH0rJ8AJJXbkX+T3ggTSTPBIYCBARLwJrJO0FTKI0M2xNdGGfZmbWQ3ozCT4ETJA0KN3rO7JF/RPANpLGAEgaKqn5wZ3nKV3yvDZz7+0e4IzmzpJGtrVzSXVAU0RcB1xMKSG25nFguKSd0/t/zNQNA15M25Nb9LsROBcYFhHLWxn3EUqXgwGOaytWMzPrHb2WBCNiMaVEsQy4E1jYov4dSrOon0paBtxLmmml+scpJY+bUoKaAhQkLZe0EjilnRBGAAvS5dPpQKsPp0TE28BJwB3pwZjVmeofAv8maQkffbL2ZkpJ7jdl9n8mcJqkBmDbdmI1M7NeoNJtL6tFhUIhisWWz9eYmVlbJC2KiEJH2vaVe4JmZmYVV/Uvy1eTpNnAji2Kvx0Rd1cjHjMz6125ToIRcXS1YzAzs+rx5VAzM8stJ0EzM8stJ0EzM8stJ0EzM8stJ0EzM8stJ0EzM8stJ0EzM8utXH9PsNY1NjZSX19f7TDMzCpm+vTp1Q7hQzwTNDOz3HISNDOz3GozCUraTNKpafsgSbd3d4eVGqebMQxPq8NXYqwZks6pxFhmZta72psJbgac2huBdERmkV0zM7Nuay8JXgjsnBaivRgYIulmSY9Lul6SACSdJ2mhpBWSZmbKPyXpPknLJC3OrNZOqh8jaYmknSVtIulqSQtS2VGpzWRJcyTdD8yVNETS3DReQ6bd+ZLOyoz9fUlntncCJD2UXZVe0sOS9k4zvKslzZP0jKQpmTbTJD0p6WHg05nykZIeTQv9zpa0eSqfJ+midGxPShrfRjwnSSpKKjY1NbUXvpmZdUN7SXAq8HREjAS+BYwCzgJ2B3YCxqV2l0XEmIjYExgEfDGVXw/8LCL2BvYHVjUPLGl/4ErgqIh4GpgG3B8RY4GDgYslbZKa7wNMjIgDgbeBoyNin9TuRynpXg2ckMYeQGmV9+s6cA7+A5ic+u0CDIyIZaluV+AwYCwwXdJGkkansUcCXwDGZMa6ltJSTHsBDZRWsG+2YTq2s1qUf0hEzIyIQkQUBg8e3IHwzcysqzr7YMyCiHghIt4HlgLDU/nBkn4vqQH4LLCHpKHAthExGyAi3o6I5qnNbsBM4MiI+O9U9nlgapp1zgMGAtununsj4i9pW8APJC0H7gO2BbaOiOeANZJGpbGWRMSaDhzTTcAXJW0E/BMwK1N3R0Ssi4iXgdXA1sB4YHZENEXE68AcAEnDgM0i4sHU9xrgM5mxbk2vizLnzczMqqiz99jWZbbXAxtKGghcDhQi4k+SZlBKYG1ZldqMAhpTmYBjIuKJbENJ+wJvZoqOA7YCRkfEu5Key+zvKkqzur+hNDNsV0Q0SboXOAr4MjA6U/2R4+3ImGU0j9XdcczMrELamwm+AQxtp01zAnpZ0hBgIkBEvAG8IGkCgKSNJTVf33sVOAL4N0kHpbK7gTMy9xNHldnfMGB1SoAHAztk6mYDh1O6RNmZ1eGvAn4CLIyIV9pp+xAwQdKgNNs9EiAiXgNeydzvOx54sMwYZmZWA9qckUTEGkmPpK8TvAW81EqbVyX9AlgB/BlYmKk+Hvi5pPOBd4FjM/1ekvRF4E5J/wR8D/gxsDzd03uWv95bzLoe+K906bUIPJ4Z8x1JDwCvRsT69g//g36LJL0O/LIDbRdLuhFYRukSafZ4TwSuTMn+GeDrHY3BzMx6nyKi2jFUTEqei4FjI+KpTvSro3Qfctd0v7MmFAqFKBaL1Q7DzKxPkbQoIgodadtvfjFG0u7AH4G5nUyAJwC/B6bVUgI0M7Oe128e0IiIlZS+tvEBSSOAX7Voui4i9s30u5bSVxvMzCxn+k0SbE1ENFD6Pp+ZmdlH9JvLoWZmZp3lJGhmZrnlJGhmZrnlJGhmZrnlJGhmZrnlJGhmZrnlJGhmZrnVr78n2Nc1NjZSX19f7TDMzD5k+vSyS6L2OZ4JmplZbtVUEpQ0RdIfJL0iaWo7bSdLuqxM3dr0Wifp5grHOCH9TqmZmfVxtXY59FTgkIh4oRKDRUQjaX3DCpoA3A6srPC4ZmbWy2pmJijpSko/gH2npLObZ3mStpJ0i6SF6W9cK313lPQ7SQ2SLsiUD09rITbPHG+VdJekpyT9MNPuG5KelLRA0i/amGHuD3wJuFjSUkk7Sxop6VFJyyXNlrR5ajtP0iWSiml2Oybt/6lsjGZmVj01kwQj4hSgETgYyK7ufilwSUSMAY6htAp8S5cCV0TECGBVG7sZCUwCRgCTJG2X1hL8V2A/YBywaxsx/haYA3wrIkZGxNOUVqD4dkTsBTQA2TvG76Q1ra4EbgNOA/YEJkv6eGv7kHRSSpzFpqamNg7FzMy6q9Yuh7bmEGB3Sc3vN5U0pEWbcZQSJJSWTrqozFhzI+I1AEkrgR2ALYEHI+IvqfwmYJeOBCZpGLBZRDyYiq4Bbso0mZNeG4DHImJV6vcMsB2wpuWYETETmAlQV1fXf1Y8NjOrQX0hCQ4A9ouIt7OFmaTYrCMJY11mez09f/zN+3u/xb7f74V9m5lZO2rmcmgb7gHOaH4jqbX1AR8BvpK2j+vk+AuBAyVtLmlD/jqjLOcNYChAmlW+Iml8qjseeLBcRzMzqy19IQlOAQrpwZOVwCmttDkTOE1SA7BtZwaPiBeBHwALKCXT54DX2uhyA/AtSUsk7QycSOlBmeWU7jme35n9m5lZ9SjCt50kDYmItWkmOBu4OiJmVzuuurq6OPnkk6sdhpnZh9T6L8ZIWpQeSmy/rZMgSPp3Sg/gDKR0+fXMqIETUygUolgsVjsMM7M+pTNJ0A9nABFxTssySdOAY1sU3xQR3++dqMzMrKc5CZaRkp0TnplZP9YXHowxMzPrEU6CZmaWW06CZmaWW06CZmaWW06CZmaWW06CZmaWW06CZmaWW06CZmaWW/6yfA1rbGykvr6+2mGYmZVV678j2h7PBM3MLLd6LAlKmiLpD5JekTS1nbaTJV1Wpm5teq2TdHOFYzxF0gmVHLPMfoZLWtHT+zEzs87pycuhpwKHRMQLlRgsIhqBiZUYKzPmlZUcz8zM+pYemQlKuhLYCbhT0tnNszxJW0m6RdLC9Deulb47SvqdpAZJF2TKP5hNpZnjrZLukvSUpB9m2n1D0pOSFkj6RbkZZmo7Q9I5aXukpEfT4r2zJW2eyqdIWpnKb0hlm0i6Ou1jiaSjMnHdJmleiit7sXyDFM9jku6RNKgbp9jMzCqgR5JgRJwCNAIHA69kqi4FLomIMcAxwFWtdL8UuCIiRgCr2tjNSGASMAKYJGk7SXXAvwL7AeOAXTsR9rXAtyNiL6ABaE5gU4FRqbx5VftpwP0RMTYd48WSNkl1Y9Ox7QUcK6l5Tau/BX4WEXsAr6Y2HyHpJElFScWmpqZOhG9mZp3V2w/GHAJcJmkpMAfYVNKQFm3GAb9O279qY6y5EfFaRLwNrAR2oJSAHoyIv0TEu8BNHQlK0jBgs4h4MBVdA3wmbS8Hrpf0NeC9VPZ5YGo6jnmUFuPdPtXdGxFrIuIt4FbggFT+bEQsTduLgOGtxRIRMyOiEBGFwYMHdyR8MzProt7+isQAYL+UuD4gqWW7jqzqvi6zvZ6eO5YjKCXEI4FpkkYAAo6JiCeyDSXty0djb37fMl5fDjUzq7LengneA5zR/EbSyFbaPAJ8JW0f18nxFwIHStpc0oaUueTYUkS8BrwiaXwqOh54UNIAYLuIeAD4NjAMGALcDZyhlL0ljcoMd6ikLdI9vwnpeMzMrAb1dhKcAhTSQyYr+es9tqwzgdMkNQDbdmbwiHgR+AGwgFLyeQ54rb1u6fVESvf2llO633g+sAFwXYplCfCTiHgV+B6wEbBc0mPpfbMFwC2ULqPeEhHFzhyDmZn1HkV05Mpj3yFpSESsTTPB2cDVETG7TNufAosj4pcV2vdkoBARp1divEKhEMWic6iZWWdIWhQRhfZb9s9fjJmRHlhZATwL/GdrjSR9D9iX0gM6ZmaWQ/3ut0Mj4pyWZZKmAce2KL4pfcWhkvueBcyq5JhmZtZz+l0SbE1EfB/4frXjMDOz2tIfL4eamZl1iJOgmZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnlVi6+J9hXNTY2Ul9fX+0wzMy6Zfr06e03qhLPBM3MLLecBM3MLLf6dBKU9JykLdP2b9tpO1zSVyu031mSJlZiLDMzq54+nQSzImL/dpoMByqSBM3MrH/oM0lQ0tckLZC0VNLPJW3Qon5tepWkiyWtkNQgaVJqciEwPvU/u8w+hkuaL2lx+ts/M+Zlkp6QdB/wiUyf8yQtTPubmVltfp6kSyQVJf1B0hhJt0p6StIFPXCKzMysk/pEEpS0GzAJGBcRI4H1wHFlmv8DpZXh9wYOobRa/DbAVGB+RIyMiEvK9F0NHBoR+6T9/SSVHw18GtgdOAHIzjovi4gxEbEnMAj4YqbunbSw45XAbcBpwJ7AZEkfL3OsJ6XEWWxqaioTppmZVUJf+YrE54DRwMI00RpEKWG15gDg1xGxHnhJ0oPAGOD1DuxnI+AySc2JdpdU/pnMmI2S7s/0OVjSucBgYAvgMeC/Ul3zgr0NwGMRsQpA0jPAdsCalgFExExgJkBdXV10IGYzM+uivpIEBVwTEd/5UKE0ucL7ORt4idIscgDwdptBSQOBy4FCRPxJ0gxgYKbJuvT6fma7+X1fOfdmZv1Wn7gcCswFJkr6BICkLSTtUKbtfGCSpA0kbUVpFrcAeAMY2s5+hgGrIuJ94Hig+b7jQ5kxtwEOTuXNCe9lSUMAPzFqZtaH9InZSESslPRd4B5JA4B3Kd1fa81s4O+AZUAA50bEnyWtAdZLWgbMKnNf8HLgFkknAHcBb2bG/CywEvhv4Hcprlcl/QJYAfwZWNj9ozUzs96iCN92qlWFQiGKxWK1wzAz61MkLUoPJbarr1wONTMzq7g+cTm00iQdBlzUovjZiDi6GvGYmVl15DIJRsTdwN3VjsPMzKrLl0PNzCy3nATNzCy3nATNzCy3nATNzCy3nATNzCy3nATNzCy3nATNzCy3cvk9wb6isbGR+vr6aodhZtau6dOnVzuELvFM0MzMcstJ0MzMcisXSVDSc5K2TNu/baftcElfrdB+z5d0SNqeJ6nQMh4zM6ueXCTBrIjYv50mw4GKJMGIOC8i7qvEWGZmVnn9LglK+pqkBZKWSvq5pA1a1K9Nr5J0saQVkhokTUpNLgTGp/5nl9nHZEn/KeneNKs7XdI3JS2R9KikLVK7WZI6tdq8pJMkFSUVm5qaOn8CzMysw/pVEpS0GzAJGBcRI4H1wHFlmv8DMBLYGzgEuFjSNsBUYH5EjCyz+nyzPdMYY4DvA00RMYrSqvMndPUYImJmRBQiojB48OCuDmNmZh3Q374i8TlgNLBQEsAgYHWZtgcAv46I9cBLkh6klNBe7+C+HoiIN4A3JL0G/FcqbwD26mL8ZmbWi/pbEhRwTUR850OF0uQe2Ne6zPb7mffv0//Oq5lZv9SvLocCc4GJkj4BIGkLSTuUaTsfmCRpA0lbAZ8BFgBvAEN7JVozM6uqfjVjiYiVkr4L3CNpAPAucFqZ5rOBvwOWAQGcGxF/lrQGWC9pGTCrnfuCZmbWhykiqh2DlVEoFKJYLFY7DDOzPkXSoogodKRtf7scamZm1mH96nJopUk6DLioRfGzEXF0NeIxM7PKchJsQ0TcDdxd7TjMzKxn+HKomZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnllpOgmZnllr8nWMMaGxupr6+vdhhmZh8yffr0aodQMZ4JmplZblVtJihpBrAW2BR4KCLuK9NuMlCIiNN7Mbbn0j5fblE+A1gbEf/ehTG73NfMzHpG1S+HRsR51Y7BzMzyqVcvh0qaJulJSQ8Dn05lsyRNTNtjJP1W0jJJCyQNbdH/CEm/k7SlpK0k3SJpYfobl9rMkHS1pHmSnpE0JZVvIumONPYKSZPaCfdcSQ0pjk+1ciw7S7pL0iJJ8yXtmsqPlPR7SUsk3Sdp61b6/oukOyUN6sp5NDOzyui1maCk0cBXgJFpv4uBRZn6jwE3ApMiYqGkTYG3MvVHA98EvhARr0j6f8AlEfGwpO0p/dD1bqn5rsDBlFaIf0LSFcDhQGNEHJHGG9ZOyK9FxAhJJwA/Br7Yon4mcEpEPCVpX+By4LPAw8B+ERGS/hk4F/jfmeM4HTgUmBAR61o5TycBJwEMG9ZeiGZm1h29eTl0PDA7IpoAJM1pUf9pYFVELASIiNdTOygllwLw+eZy4BBg91QPsKmkIWn7jpRg1klaDWwNNAA/knQRcHtEzG8n3l9nXj+0unzaz/7ATZn9b5xePwncKGkb4GPAs5muJwB/opQA321tpxExk1KCpa6uzisem5n1oL7ydOjTlGZ1u2TKBlCacY1Mf9tGxNpUl51hrQc2jIgngX0oJcMLJLV3LzLKbDfv+9XMvkdGRPMs9KfAZRExAjgZGJjp1wAMp5QozcysynozCT4ETJA0KN3rO7JF/RPANpLGAEgaKql5pvo8cAxwraQ9Utk9wBnNnSWNbGvnkuqApoi4DriYUkJsy6TM6++yFWk2+qykY9PYkrR3qh4GvJi2T2wx5hJKiXFOisfMzKqo15JgRCymdM9vGXAnsLBF/TuUEs5PJS0D7iUzi4qIx4HjKF2C3BmYAhQkLZe0EjilnRBGAAskLQWmAxe0035zScuBM4GzW6k/DvhGivUx4KhUPiPFuAh4uWWniHgYOAe4Q9KW7cRgZmY9SBG+7VSr6urq4uSTT652GGZmH1LrvxgjaVFEFDrU1kmwdhUKhSgWi9UOw8ysT+lMEqz6l+WrSdJsYMcWxd+OiLurEY+ZmfWuXCfBiDi62jGYmVn19JWvSJiZmVWck6CZmeWWk6CZmeWWk6CZmeWWk6CZmeWWk6CZmeWWk6CZmeVWrr8nWOsaGxupr6+vdhhm1gtq/afI+ivPBM3MLLecBM3MLLe6nAQlbSbp1C70W5teh0ta0dX9V4KkeZI+8iOrkgqSflKNmMzMrPd0Zya4GfCRJJhZCLfPiohiREypdhxmZtazupMELwR2lrRU0kJJ8yXNAVYCSPqmpBXp76y2BpI0UNIvJTVIWiLp4FQ+WdJtacb2lKTpmT5fk7Qg7f/nkjZoY/y1ki6R9JikuZK2ylQfm8Z5UtL41P4gSben7RmSrk4xPCNpSmbcVmOQdIWkYtpffab9hZJWpoWA/70T59rMzHpAd5LgVODpiBgJfAvYBzgzInaRNBr4OrAvsB/wL5JGtTHWaUBExAjgH4FrJDWvKj8WOAbYi1LCKkjajdIq9OPS/tdTWum9nE2AYkTsATxIaWX5ZhtGxFjgrBblWbsCh6VYpkvaqJ0YpqW1rPYCDpS0l6SPA0cDe0TEXpRZ2V7SSSmBFpuamto4JDMz665KXrpcEBHPpu0DgNkR8SaApFuB8cCSMn0PAH4KEBGPS3oe2CXV3RsRazLjHAC8B4wGFkoCGASsbiN5g4opAAAGJ0lEQVS294Eb0/Z1wK2ZuubtRcDwMv3viIh1wDpJq4Gtgc+1EcOXJZ1E6fxuA+xOaYb8NvAfaZZ5e2s7ioiZwEworSzfxjGZmVk3VTIJvlnBsbJaJoIABFwTEd+pwJjr0ut6yp+PdZnt5natxiBpR+AcYExEvCJpFjAwIt6TNJZS8pwInA58tovxm5lZBXTncugbwNAydfOBCZIGS9qE0mXA+W2MNZ90KVHSLsD2wBOp7lBJW0gaBEwAHgHmAhMlfSL12ULSDm2MP4BS4gH4KvBwewfXAeVi2JTS/xC8Jmlr4O9T/RBgWET8f+BsYO8KxGBmZt3Q5ZlgRKyR9Ej6msNbwEuZusVpBrQgFV0VEeUuhQJcDlwhqYHSpc7JEbEuXWZcANwCfBK4LiKKAJK+C9wjaQDwLqX7is+XGf9NYGzqs5rSvbxuiYiVrcUQEY9KWgI8DvyJUtKG0v8w3JbudQr4ZndjMDOz7lFE7d52kjQZKETE6d0cZ21EDKlMVL2nUChEsVisdhhmZn2KpEXp4cR2+RdjzMwst2r6i+0RMQuY1dH2kn4PbNyi+Pi+OAs0M7OeV9NJsLMiYt9qx2BmZn2HL4eamVluOQmamVlu1fTToXkn6Q3++n3JWrUl8HK1g2iHY6wMx1gZjrEy2opxh4jYqkzdh/Sre4L90BMdfcy3WiQVHWP3OcbKcIyVkacYfTnUzMxyy0nQzMxyy0mwts2sdgAd4BgrwzFWhmOsjNzE6AdjzMwstzwTNDOz3HISrBJJh0t6QtIfJU1tpX5jSTem+t9LGp6p+04qf0LSYbUWo6Thkt6StDT9XVnFGD8jabGk9yRNbFF3oqSn0t+JNRrj+sx5nFPFGL8paaWk5ZLmZpcuq6Hz2FaMtXIeT5HUkOJ4WNLumbpa+Vy3GmMtfa4z7Y6RFJIKmbLOnceI8F8v/wEbAE8DOwEfA5YBu7docypwZdr+CnBj2t49td8Y2DGNs0GNxTgcWFEj53E4sBdwLTAxU74F8Ex63Txtb15LMaa6tTVyHg8GBqft/5X5b11L57HVGGvsPG6a2f4ScFfarqXPdbkYa+ZzndoNBR4CHqW02lCXzqNngtUxFvhjRDwTEe8ANwBHtWhzFHBN2r4Z+JwkpfIbImJdRDwL/DGNV0sx9pZ2Y4yI5yJiOfB+i76HAfdGxF8i4hXgXuDwGouxt3Qkxgcioim9fZTS+p5QW+exXIy9pSMxvp55uwnQ/FBGzXyu24ixt3Tk3x6A7wEXAW9nyjp9Hp0Eq2NbSgvuNnshlbXaJiLeA14DPt7BvtWOEWBHSUskPShpfA/E19EYe6JvZ3R3PwMlFSU9KmlCZUP7QGdj/AZwZxf7dlV3YoQaOo+STpP0NPBDYEpn+lY5RqiRz7WkfYDtIuKOzvZtyb8YYz1hFbB9RKyRNBr4T0l7tPg/TOuYHSLiRUk7AfdLaoiIp6sVjKSvAQXgwGrF0J4yMdbMeYyInwE/k/RV4LtAj91H7aoyMdbE51rSAOD/ApMrMZ5ngtXxIrBd5v0nU1mrbSRtCAwD1nSwb1VjTJci1gBExCJK1+V3qVKMPdG3M7q1n4h4Mb0+A8wDRlUyuKRDMUo6BJgGfCki1nWmb5VjrKnzmHED0DwrranzmPFBjDX0uR4K7AnMk/QcsB8wJz0c0/nz2NM3Of3X6o3fDSk9QLAjf73xu0eLNqfx4YdOfpO29+DDN36foWduoHcnxq2aY6J0c/tFYItqxJhpO4uPPhjzLKWHOTZP27UW4+bAxml7S+ApWnlAoJf+W4+i9I/e37Yor5nz2EaMtXQe/zazfSRQTNu19LkuF2PNfa5T+3n89cGYTp/Higbvv079h/4C8GT60E5LZedT+j9YgIHATZRu7C4Adsr0nZb6PQH8fa3FCBwDPAYsBRYDR1YxxjGU7gu8SWkm/Vim7z+l2P8IfL3WYgT2BxrSh7oB+EYVY7wPeCn9N10KzKnB89hqjDV2Hi/NfDYeIPOPew19rluNsZY+1y3aziMlwa6cR/9ijJmZ5ZbvCZqZWW45CZqZWW45CZqZWW45CZqZWW45CZqZWW45CZqZWW45CZqZWW45CZqZWW79DwFROsfhR4crAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sttr[\"sttr\"].plot.barh(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Going back to the document level\n",
    "\n",
    "The complexity measures can be accessed via the `Corpus` object as well as via a `Document` object – even more flexible in the case of `Document`, e.g. using the `bootstrap()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These are the TTRs of the first 10 windows: [0.327, 0.312, 0.33, 0.3185, 0.3195, 0.3215, 0.3125, 0.3115, 0.315, 0.307]\n"
     ]
    }
   ],
   "source": [
    "results = list(document.bootstrap(measure=\"ttr\", window=2000))\n",
    "\n",
    "print(\"These are the TTRs of the first 10 windows: {}\".format(results[:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you put those in a pandas `Series`, you could plot them and see how the TTRs differ over the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fedb2df7b00>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztvXuQHGeZ5vu8devqa3VXt9S6y5ItG+QLMpZk7DUMxoYxsMcMMxszNksMfxABTNgxEJyJHSZ2gtglDidiTQQLZ/CZOAyHcyJ2MQbM7CAP2pGZGXuMMIwlW7KFbOtYli1LslotqVt9q+665Xf+qHpTX2XntSqrKjv7/UUo1JV1+6oq88knn+/73o+UUhAEQRBWB4luN0AQBEHoHCL6giAIqwgRfUEQhFWEiL4gCMIqQkRfEARhFSGiLwiCsIoQ0RcEQVhFiOgLgiCsIkT0BUEQVhGpbjfAytjYmLrmmmu63QxBEIQVxQsvvHBJKbXG63GRE/1rrrkGhw8f7nYzBEEQVhREdNrP4yTeEQRBWEWI6AuCIKwiRPQFQRBWESL6giAIqwgRfUEQhFWEiL4gCMIqQkRfEARhFSGiLwgxxDAMHDlyBIZhdLspQsQQ0ReEGPL2229j3759ePvtt7vdFCFiiOgLQgwpFosAgHK53OWWCFFDRF8QYgiLfaVS6XJLhKghoi8IMaRUKgEAqtVql1siRA0RfUGIIeL0BSdE9AUhhrDoi9MXrIjoC0IM4XhHnL5gRURfEGKIOH3BCRF9QYghkukLTojoC0IMEacvOCGiLwgxRJy+4ISIviDEEBmnLzghoi8IMUScvuCEiL4gxBDJ9AUnRPQFIYbIOH3BCRF9QYgh4vQFJ0T0BSGGSKYvOCGiLwgxREbvCE6I6AtCzDAMwxR7cfqCFV+iT0T3EdEJIjpJRF+xuf8LRHSMiI4S0UEi2qnddwsR/ZqIjtcfkw3zAwiC0Ii+WpY4fcGKp+gTURLAowA+CmAngAd1Ua/zmFLqZqXULgCPAPhm/bkpAP8dwBeUUjcC+CAAWb9NENqILvri9AUrfpz+XgAnlVKnlFIlAI8D+IT+AKXUrHazH4Cq//0RAC8rpV6qP+6yUkqshyC0EXH6ght+RH8jgDPa7bP1bQ0Q0UNE9AZqTv9P65uvB6CI6AARvUhE/6HVBguC4A534qZSKXH6wjJC68hVSj2qlLoWwJ8D+Mv65hSAuwD8+/r/nySie6zPJaLPEdFhIjp88eLFsJokCKsSdvq9vb3i9IVl+BH9cwA2a7c31bc58TiA36v/fRbAs0qpS0qpAoD9AN5rfYJS6rtKqd1Kqd1r1qzx13JBEGzRRV+cvmDFj+gfArCDiLYRUQbAAwD26Q8goh3azY8DeL3+9wEANxNRX71T93cAvNJ6swVBcILjnWw2K05fWEbK6wFKqQoRPYyagCcBfF8pdZyIvgbgsFJqH4CHiehe1EbmTAP4TP2500T0TdROHArAfqXUz9v0WQRBgDh9wR1P0QcApdR+1KIZfdtXtb+/6PLc/47asE1BEDqALvqGYUApBSLqcquEqCAzcgUhZujxDiDDNoVGRPQFIWboTh+QCVpCIyL6ghAzWPTF6Qt2iOgLQswolUpIp9NIpWpdduL0BR0RfUGIGeVyGel0GslkEoA4faEREX1BiBnlchmZTEacvmCLiL4gxAyr0xfRF3RE9AUhZrDos9OXeEfQEdEXhJhRKpWQyWTE6Qu2iOgLQswQpy+4IaIvCDHDKvri9AUdEX1BiBnWeEecvqAjoi8IMUOcvuCGiL4gxAyZnCW4IaIvCDHCMAxUKhVx+oIjIvqCECO42Jo4fcEJEX1BiBEs+lKGQXBCRF8QYoQ4fcELEX1BiBG66BMREomEOH2hARF9QYgRvFRiJpMBAKRSKXH6QgMi+oIQI3SnDwDJZFKcvtCAiL4gxAir6IvTF6yI6AtCjLDGO8lkUkRfaEBEXxBihJ3Tl3hH0BHRF4QYYZfpi9MXdET0BSFG2I3eEacv6IjoC0KMYKfPs3Fl9I5gRURfEGIEV9gkIgAyekdYjoi+IMSIUqlk5vmAOH1hOSL6ghAj2Okz4vQFKyL6ghAjyuWy2YkLiNMXluNL9InoPiI6QUQniegrNvd/gYiOEdFRIjpIRDst928honki+rOwGi4IwnLE6QteeIo+ESUBPArgowB2AnjQKuoAHlNK3ayU2gXgEQDftNz/TQD/M4T2CoLgglX0xekLVvw4/b0ATiqlTimlSgAeB/AJ/QFKqVntZj8AxTeI6PcAvAngeOvNFQTBjVKp1BDviNMXrPgR/Y0Azmi3z9a3NUBEDxHRG6g5/T+tbxsA8OcA/rPbGxDR54joMBEdvnjxot+2C4JgQZy+4EVoHblKqUeVUteiJvJ/Wd/8nwD8V6XUvMdzv6uU2q2U2r1mzZqwmiQIqw6nTF8p5fIsYTWR8vGYcwA2a7c31bc58TiAv67/fTuAf0dEjwAYBmAQ0ZJS6jvNNFYQBHfsxukDgGEY5t9CcF599VUMDQ1h48ZlIceKw4/TPwRgBxFtI6IMgAcA7NMfQEQ7tJsfB/A6ACil3q+UukYpdQ2AbwH430XwBaF9WIdsyuLo4fDUU0/hueee63YzQsHT6SulKkT0MIADAJIAvq+UOk5EXwNwWCm1D8DDRHQvgDKAaQCfaWejBUFYjlIKlUrF1ulLZ25rlMtlLC4udrsZoeAn3oFSaj+A/ZZtX9X+/qKP1/hPQRsnCIJ/rGWVAXH6YVGpVGIj+jIjVxBigrWsMiBOPywqlQqWlpa63YxQENGPASdPnsTRo0e73Qyhy4jTbw9KKVSrVXH6QnR4/vnncfDgwW43Q+gydqIvTr91+LsrFoswDKPLrWkdEf0YsLi4aB7wwurFLt4Rp986+ncXh4hHRD8GFAqFWB7UFy5cwFNPPSUTi3wiTr896MdWHCIeEf0YEFen/+qrr+LXv/51LA60TiCZfnsQ0RcihWEYWFxcRKVSiZ0jLhaLAGpXMoI3bk5fRL95JN4RIgXvhEqpWHQy6bDox8FddQK3TF/ineYRpy9ECt0Fxy3iYRETp+8PcfrtQURfiBT6Thi3A1vinWC4Zfor1ekvLi7i8ccfx9zcXNfaIKIvRIo4O/24xjvz8/P4wQ9+EPrJjK+M4uT03377bZw4cQLnzrkV9m0vkukLkUKcfud58skn8dprrzX9/LfeegsnT54MXci4lj4RmdtWutOfmpoCcPWE1g3E6QuRYjU4/SiJvlIKR44cwW9/+9umX2NmZgYAsLCwEFazACxfQAVY+U6fRb+b+zZ/d8lkUpz+SmP//v144YUXut2MUFkNTj9K7qpUKkEphVaW9eyk6IvTbx0+rgYHByO1LzbLqhF9pRReeuklvPzyy91uSqjE1ekrpSLp9Pmgv3z5ctNDZGdnZwGEL/rWRdEBIJFIgIhWrCGIktMfGBgQ0V9JLC0toVQq4cKFC7GaxBRXp69PNovSgcaX99Vq1XTsQemk0wdqscRKdPr6d+zm9J988sm2XsGL01+h8M5TLBZx5cqVLrcmPBYXF9Hb2wsgXk6fXT4QTacPAJcuXWrqNTot+qlUakUagunpafPE77ZvnzhxAm+++Wbb2qE7fcn0VxC6K5uYmOhiS8KlUChgaGgIQLycPov+0NAQFhcXI3N1ph/0zYh+qVQyTxydiHeAmuivRKfP0Q7g7vRLpVJbDY/u9CuVyoo3V7ESfcMwHHNWXfQvXLjQqSa1ncXFRVP0V/rOqMOiPzIyAsMwGpx/N2HBJqKmOnN5P0yn0xLveMCin81mHfdtpRTK5XJbO3orlQqSyaR5Rb3S3X5sRP/MmTP4+te/jrfeesv2/pmZGSSTSeTz+diIvlIKhUIBg4ODAOLp9EdGRgBEJ9fndoyPj+Py5cuBn8+iv2HDBiwsLIR6BRO3eGdqago9PT0YHh52FH3e3m6nn0qlTNGPyr7YLLER/aGhIRiG0XBJqDMzM4NcLof169fHJt4pl8uoVquxdvrDw8MAopPrLy0tgYiwfv36puIdFv3169fDMIxQXWPcnP709DTy+TwymYyjk+ft7Xb6qVQK2WwWgIh+ZBgaGkIymfQU/fHxcVy5cqWrl2hhuTve+VaD04+K6HPH+Zo1a1AoFAK3a2ZmBkSEdevWAQg31y+VSrFz+vl8Hul02tHQdFL0xelHDCJCPp/H9PS07f266APdy/WPHDmC73znO6HspCw4fX19rgfGSiTKTr+3txdjY2MAgnfmzszMYGhoyDxRhyX6SilUKhXbjtyV6PSr1Wogp9/JeEcy/QiRz+dtnX61WsXc3ByGhoZMh9Ut0T9z5gympqZw5MiRll+LHUdvb2/H3dyxY8fwV3/1V22r4R/lTD+bzbYk+rlcDv39/QDCE327CpvMSnT6MzMzUEqZTj8K8Y44/QjCom+NT3gGZC6Xw+DgIHp7e7uW63Om+9xzz7XsvqxOv5MH9vnz5zE1NdW2UTXFYhGJRAIDAwMgosg5/Vwuh1QqtSJEfyU6fTZvfuOddq4cx6Lf09MDIhLRjxL5fB6VSmVZ7W0W2uHhYTNP7ZbTn52dRX9/P2ZnZ3Hs2LGWXsvq9DsZ7/B7t+tSt1gsmgdZb29vZESfnX4ikcDo6Ggg0TcMA7Ozs8jlcujr6wMQnujbrZrFrESnr4u+n3gHaF/Ew6JPRMhmsyL6USKfzwPAsoiHRT+XywGoDbebnJzs+PKCSinMzMzgpptuwvj4OH71q1+15E5YCHt7ezvu9Nst+qVSCT09PQBqny8qBxo7fQAYGxsLJPrz8/MwDAO5XA6JRAK9vb0dc/orUfQzmQz6+/vNfdvuWNFFv10RD4s+UNsXJdOPEF6iz0Mbx8fHUalUHEf6tIulpSWUy2XkcjncdddduHTpUkt12RcXF9HT04NkMtlxp88nnHY7faAWX0VB9JVSptMHaqJ/5coV34JqNR/9/f0dy/RXYryTz+dBRObVi93+3WnRF6cfMYaGhpBIJGxFn3NvAGZnbqdzff3ks3PnToyMjODgwYNNu3297k7cnL5V9KMQ73D/he70lVK+J2lZRX9gYKBp0TcMo0HI3eKdler02cTxcWsn6vqJoN3xDiBOP3IkEgmMjIwsE33OUZmxsTEkEglfoh+me9A7lBOJBO6880688847jrOIvSgUCmY23OnctpOiH5VMnz+z7vQB/yN4wnT6Bw8exCOPPIKjR4+apQiAeHTkGoaB6elpc+RWlJx+lKLGZvEl+kR0HxGdIKKTRPQVm/u/QETHiOgoER0kop317R8mohfq971ARB8K+wNYGR0dXSb6V65caRD9VCqFsbEx185cwzDw5JNP4hvf+EZoVTmtB/2uXbswMDCAgwcPNvV6VqffqXiHYw6gs06/20XX+LPydz46OgogmOhns9mGz9Ws6F++fBmlUgk/+9nP8MQTT5iGopUhm+VyORJzPWZmZmAYhvn9ujn9TnbkAqsk3iGiJIBHAXwUwE4AD7KoazymlLpZKbULwCMAvlnffgnA/6KUuhnAZwD8t9Ba7gA7fRYI7jzVRR+A6wiearWKv/3bv8WLL76ISqXStBO3MjMzg0QiYQ7XS6VSuP3223Hq1CnHSWVudMvpF4tFsxO8naLPDq+3txfVarXrgqSPlgJqYpTL5QLFO/p+2N/fj6WlpaZceKlUwtjYGO655x689tpr2L9/PwDneMfPe/z0pz/FE088EbgtYaOP3AFad/rVarWlvhO7eKfbBqQV/Dj9vQBOKqVOKaVKAB4H8An9AUqpWe1mPwBV335EKfVOfftxAL1E1NN6s53J5/Mol8uYn58H0Nh5qjM+Po65ubllO0O5XMaPfvQjHD9+HPfccw+y2SzefvvtUNo2Oztr9jswmzdvBrC889kPutPvZEeu7nQ65fSt79sNrPEOEGwEj53oA80N2yyXy+jp6cFdd92Fz372s8jn80gmkw1tY1KpFJRSrqPVlFI4ffp0JIoROom+V6bvJPq/+tWv8K1vfQuvv/56U+2xir6+qttKxI/obwRwRrt9tr6tASJ6iIjeQM3p/6nN6/wBgBeVUm39tqwjeKyRCmM3M7dYLOKxxx7D66+/jo9//OO46667sGXLltBEn6fg63C7gq7CxMW6utGRG5boLy4u2h481WoVlUplmeh3O9e3xjvAVdH34/ysv38roq/Xzt+wYQM+//nP4wtf+IKt6PtZHJ3rUc3NzXV8KLOVqakppFIpDAwMALga7zg5fd4/nEzP7OwsKpUKHn/8cRw/fjxQW5RSqFarDfEO0H0D0gqhdeQqpR5VSl0L4M8B/KV+HxHdCOC/APi83XOJ6HNEdJiIDrey4DTgX/S5Bs/ExAQKhQKeeeYZfPvb38bp06fx+7//+9i9ezeAmhO/fPlyKEPrrB3KQK1YGhEFFn3e6fR4p1NOXxffVkT/sccew89//vNl29mx6R251vftBtZ4B6iJfrlcNjN1J4rFIpaWlsxaQgBMUWtV9IGaMHLHshU/i6PzoAbDMEKv8x8Ufbgm4O70S6WS+T26TeAaHBzEpk2b8NOf/jRQCRQ+UepOH1jZ9XdSPh5zDsBm7fam+jYnHgfw13yDiDYB+B8A/lgp9YbdE5RS3wXwXQDYvXt3S2HZ8PBww7BNJ9Hv7+/HwMAADh06hGeeeQblchk33HAD3v/+92PjxqsXMlu3bgUAvP3223j3u9/ddLt4NqbV6SeTSQwODgYWfX1iFlA76KvVKpRS5sHSLlj8OJNuhnK5jHPnztk6ZHb/UYx3eE4Eo4/gse5jOvrILSYsp++FH6d//vx58++ZmRmzIFw3mJqaajiBeXXk8vfoZHqKxSL6+/vx6U9/Gj/60Y+wb98+FItFvO997/Nsi5Pod3tfbAU/Tv8QgB1EtI2IMgAeALBPfwAR7dBufhzA6/XtwwB+DuArSqlfhdNkdxKJBIaHh03Rv3LlCpLJpLlj6GzevBkzMzN497vfjT/5kz/BAw880CD4QK3ueTKZbDniWVhYMGdjWsnlcqE4faAz5ZX5vfP5fNOizwvUc9+LjpPod9vpLy0tIZvNNpxU/Q7btDMfnRJ9v06fH9fsgu9hwMM1+Yod8O7I5bIYTk6f+4fS6TQeeOAB3HDDDThw4MCyci12rErRV0pVADwM4ACAVwH8WCl1nIi+RkT31x/2MBEdJ6KjAL6M2kgd1J93HYCv1odzHiWiteF/jEb0apscqdi53/vvvx9f/vKX8clPfhJr19o3K5VKYePGjS2LvtMVB28Lw+kDnVlIhd97eHi46Q6td96p9e/brR5lFf2oxDt6HwrT39+PbDbblOhnMhkkk0lb0b9y5Yrrd9sOp799+3YA8Iyq2snc3Byq1WqD6Hs5/Uwm41mfh/elVCqFXbt2AYCt4bBiFf1Vk+krpfYrpa5XSl2rlPp6fdtXlVL76n9/USl1o1Jql1LqbqXU8fr2/00p1V/fzv8m2/dxaujVNu2GazLZbNbMA93YsmULzp8/39LkDz6QrPEOb5udnQ00DKzbTr+npwd9fX1NO30W/Uqlsux7tYp+IpFANpvtuujro6UYIsKaNWs8J/pduXIFRNSwvxGR7QQtpRT+5m/+Br/85S9tX8swDJTL5dCc/vz8PObn53HNNdcgnU531elbR+4AtZMWETk6fRZ9t3hH/65YuP3su3HM9GM1I5fJ5/MolUooFAquou+XLVu2QCmFc+fcujLc8XL6QccSd9Pps/hls9mGMftB0DNk6+e2ij4QjZmQet0dna1bt+LcuXOuztxuuC5gPyv38uXLKBQKjk6Uf+Ogou9kCPi3WL9+PXK5XFedPk+E5Nm4AMz6O05OP51Ou05O1If/Alf3q2ZEP51OI5lMdn1fbIXYij4AXLx4EXNzcy2LPo+lP336dNOvMTMzg3Q6bSsazQzbXFxcRCKRMA/8sJ3+9PS0Y+a5uLiIvr4+87MEjXhKpRIuXrxo9p/4Ef0oFF2zi3cAYPv27eY4dyeczIdd/R2+CnL6Xt3q7NjB8Y6T0+erlHXr1jUVNYaJ3bBYwH7GebVaRbVa9Yx3rKIfZL+1ij63rdv7YivEWvR5Jm2rop/NZjE+Po4zZ854P9gBt74Fbl+Qcg88G5dfL2yn/5Of/AT/8A//4Pje7PSB4Je6ExMTUEphx45a/7/V0TqJfhTiHbuT9ubNm5FKpXDq1CnH5zqJvp3TD1v0vQzBxMQERkZGkM1mMTQ0FAnRt342u/hGv+JxcvqGYSxbRrKVeAdY+UXXYin6vFjKm2++CaB10QdqEc+ZM2c8o4yJiQnbg8ZuYpbeXn6MX6z5cthOf25uzrE0hB7vAMFFn+MEFn0np68fqN0uumYYBorFoq3TT6VS2LJli7m/2T3Xbo4GcLX+jt6fw9+P18IhYTn98+fPY/369QBqx8rCwkLXqnLqi+fo2C2ZqH8PTk7fzkDw3+L0Y0QymcTw8DDOnj0LIDzRL5fLrh12s7Oz+P73v2874cjpoAdqO2Emkwks+tyJC4Tr9JVSKBQKjtluq6L/zjvvYGBgwJwgZyf6mUym4cDvdrzDAmHn9AFg27ZtmJyctM3h9cVTrPT396NarZqvbxiGKfpeTt+uuJodboZgaWkJ09PT5gx1NiZ+hjO2A2sUw9g5fV30ndbRtRP9RCKBdDrdtNNf6UXXYin6QC3iYVfu5LCDsGXLFgBwHbp54MABlMtlvPXWWw2uqlKpYH5+3rEdRBS4A40jFiZMp18qlcyZmVZ3yOUf9Ey/GdHfsGGDWSvGLt6xHvi9vb0olUpdc6B2s3F1eLijXcTj1olvHat/+fJllMtlpFIpR6cftCPXzenreb7exm5FPG6i7+X03QqyWV8zm822Nd6pVCo4efKk5+t3g9iKPvf+83JrrTI0NITh4WFH0X/jjTfwyiuvYPPmzSiXy+ZVBnDVNbldcQTtQLPGO2E6fd3FWAVZF79mRL9UKuHSpUtmnDAwMLAsttHHVTPdnpVrV2xNZ926dejt7bWNeIKIPuf5W7Zs6Uimr4/c0dtoty+Wy+Wmi5b5xUn07TL7IE7f+l3xyDMvmnX6r732Gn7wgx+Yv2eUiK3oc2duGNEOw8XXrOPpK5UK9u/fj3w+jz/8wz8EETU4PreDngki+hy/6PFOmE5f36GtVx+tij6LzIYNGwDURM+P0+/2rFynUSVMIpHAtm3bcOrUqWX7x8mTJ83I0Yq1/s4777yDdDqN9evXo1Qqua4LG5bTHxgYMNvBV6N2V52HDx/GY4895mtSU7OE7fTt4h2+3UqmXyqVXGc4s9FrZcRfuxDRD8DmzZuxsLCwLNd/7rnnMDU1hY997GMYGBjAhg0bGhyfdY1eO3K5HAqFgi+nzvFLu5y+LqzWbFefFBZkvDPDzkcXfbtM3y7esbatk3jFO0At15+dnW0okz05OYmXXnoJe/futRVpq9PnTlUu4esWWYTh9CcmJkyXD9T2o97eXlsDwifsdo5c4VIXVuycvHX0TqVSWTbQwkn0W413uK1O8H7ayoi/dhFb0edVd8LI85nrrrsO6XQa3/ve9/B3f/d3uHjxIqanp/HLX/4SO3fuxLXXXgugdvCfPXvW3OHsim1ZCZKlWmfjAp1z+vqkMCJCT09PYKc/ODhoOku/ou833nn99ddx/Pjx0DsiveIdwD7X/+d//mez7r0d/Lm4NhOLsFdlSSC407fuG+VyGRcvXjTzfMapf8mrgzkMrLNnGa94x6k+TztF321f5PtOnz4duQVX/FTZXJGMjIxgdHTU7IANg+HhYTz00EP49a9/jRdffBEvvfSSWRr5d3/3d83Hbd++HQcPHsTp06dx/fXXY2ZmBr29va59C7roO5XIZVh47UQ/7EzfLd4B/GejDHfiMlypU1+oopV458c//rF5oI6MjGDLli247bbbzAl2zeIV7/D75XI5nDp1Cnv27MHZs2dx4sQJ3H333Q2/lQ53Zi8sLODSpUsol8sN30+xWFxWKqRUKoGIGoTIDacyDJOTk1BKNTh9oLYvWofrcl8M/90uisWi7YmV4xu9iqw+ikk/Ser7jlNHbtB4h0+cgL/6O3xfoVDA1NSUaUKjQGydfjKZxMMPP4ydO60rO7ZGLpfDfffdhy996Uv4wAc+AMMwcO+99zZcUVgn67gN19RfFwjm9HUBSiQSSCQSoTh9FtahoSHXeAfw75iA2gF9+fLlBlFjQdPFvNl4p1KpoFKpYNeuXfjIRz6C8fFxvPrqqzhw4ICv9rmxuLiIVCrlKrREhO3bt+PNN9+EYRj4x3/8R/T393uW8OWrHT36YhGzEyauN+O3hDaXfrDuG9ZOXMZugpZ1saF2wDWFnDpyAfuVsjjesd6vt9V69eD3CpXNiP5d+413OLoLaxGmsIit6Lebvr4+3H333fizP/sz7N27t+E+62Qdt4lZTJDFVOycPhDe4uhcUG14eHiZ6BcKBTPWAYKJvp3I8IHBnYO8FJ31wE+lUshkMq7uikVg3bp1uOOOO/BHf/RH2LNnD86fP9/yydCpBIOV7du3o1gsmld6H/jABzxjGF30M5kMRkdHzc/vVlnSL0Rku07u+fPnkc1mlxmSXC6HYrHYIO56P1a7nL5TFAPYl1culUpIJBJIJpOOcVixWEQ6nV5W8yibzZortLmhX4EyfuKdQqGATZs2obe3V0R/tbB9+3Zzso4fpx9kMRWnTsWwFkfn4aBc/dPuPnY+QUTf2okLLO/IdLocB7xn5dqJxqZNm2AYRstD55xKMFjZtm0bAODpp5/G8PAwbrvtNs/ncP0d7sTVT6puTj8IdvvGxMQE1q1bt+yKwW4Ez/nz503h7Ibo25VX1q94nMovO/UR+B151oro9/X1hbrcaliI6LcJ7tR77bXXsLS05KtD2e+wTWuFTSasdXJZ2AcHBzE3N9fQEWWdHxDU6edyuYYFbaxDFt0OfK9ZuXaX8pzl6/MmmsGv0+/v7zdnGn/wgx9syIKd6Ovrw9zcXMNIGj/xThCsTl8phYsXL5pt1bGLGicmJsyTdbviHbdZz05On7c7deTazfkA/JdisBN9r0xfKWXOmN+yZQumpqbaOsw1KCL6bYIn6xwLhOhWAAAgAElEQVQ9ehSAv6GjfkWfXaf1kjWsdXLZpQwODqJSqTSIurX8Q1Cnr7t8YHm84+b0vYqu2T23v78fIyMjLQ+d8+v0AeDWW2/Ftddei5tvvtnX4/v7+1EsFlGpVMzvJ8x4B6jtG7roLywsoFwuN9StZ6yiX61WMTk5aZ5A2+X0eT/y6/TL5bK53c3p271eK04/kUi49gmUy2VUq1X09vb6msnfaUT02wQRYdu2bWYNfr+i72cxFbvFPIDwnb7dZb61/IPfmvpKqWVrnwJXO+H8OP1m4h2gFvGcPXu2paFzfp0+ANx+++349Kc/veyk7IR+5WMV/TCdvr5v8FwCvW49w/1L/LtfvHgR1WrV7GBut9MPkuk36/RbEX1+vpPT1/vc1q9fj1QqJaK/WuB8F/A3X4AXU/G6FLTOxmWcnH6hUMAzzzzj26FZRV/vzLWLdwDvy2R+bzu3rI/VDzveAWqiPz8/31I9mSBOPygs+j09Pabz9loiMGhpEavT5yGZdk4/kUg09C9xB/y6detc69a3ih/Rt8v0ne7n1ww73gHc6+/oop9MJrFp0yYR/dUC5/pAzT154XfYZlCn/8Ybb+Bf/uVf8NRTT3m2wTCMhkwfaHT6TqLv5ZjcLt3tRN/OyfKB5nRV4RQNcSzRbMRTrVZRKpV8O/2gsOhzJy5wdbUoO1EKslQiY+f0ici2NASAhk78iYkJpNNpc1RRu0XfaUYu4Oz03YZsht2RC7iXV7b2uW3ZsgUTExNtndQWBBH9NjIyMoLh4WEMDg766tDzK/rz8/MNkQDj5PR553zhhRdw4sQJ19fmg4AzfeCq6FcqFZTL5WWZvv48J9wOaH31KC+nr38ep/ewPnd8fBzpdLrpzlz+bO1y+tyZbR0v7zSBKIxMf3p6GkNDQ477pd6/pI/yaafTdzMG7XL6zYo+r4Ngh3UuSxjLrYaJiH4bISK8733vw3ve8x5fj/cj+oZhYG5uzjYucnL6vGOPj49j3759rvGRPhw0mUyiv7/fjHfsRg2F4fT7+vrMNvkRfadc3+kqIZFIYMOGDS2Lfruc/vDwMHbs2IGbbrqpYbudq1ZKhZLpT09P20Y7DDt9Lg3BpRr8zmR14sSJE3jllVds7ysWi0gkErYi6+T0eTuP1/c7ZLPVeGdgYMDxOLLOo9m0aROIKDLF10T028ztt9+Oe+65x9dj/Symwqss2Ym+m9NPp9P4gz/4A5RKJfzsZz9z7NS0zgHgYZt293GbgdadfqFQMCdm6a+r4zU+2mkiDlCLeCYmJpoa3eSn2ForJJNJfOpTn1o2ssnOVVerVRiG0bLTn5qasu3EZXK5HCqVCs6dO4dSqdQwlLQVp//ss8/i2Weftb3PadUsfl/A2enzY/Tfl9fQtduX/NaNchL9wcFBlEol25MGiz7v6z09PVi3bl1kiq+J6EcIP4up8H12fQROk7N45MmaNWvw4Q9/GCdPnsShQ4dsX9/qUvRs167Qm9+OXK9Mn8tFF4tFpFIp29jBy525OeBWJmn5KbbWDuxcddBia4y+bxSLRRQKBVfRZ1Px2muvAUBooj89Pe160rbbP4DaiZGIzPfmCqT692Cdke5mIAB/daPcnD6wfL0J4Gq/l24+Nm/ejLNnz7qWY+4UIvoRY3h42HWBdBZgp3jHzsnq5Wr37NmDHTt24Be/+EVDCWDGzem3Eu+4OX19Vq7bge/1Xk7D84Ca6APNTdJqd7zjhF1HbrOir0/Ochu5w3DUeOLECSQSCaxZs8axTX5ZWlrC4uKieVVnxe2351m3vH/bXfFYT0hucz4Af3NM3Jw+YL+spN1ACz/LrXYKEf2IYVfsSsdN9NnNWQ8oXfSJCPfddx8qlYrtKk9W0R8aGkKhUEClUgkl3rE7APVZua2Ivttz+/v7kc/nmxL9dsc7Tthl+mE4fbcx+gyL/uXLl7F27VrzyquV0Tv8vjwgwIrb7wc0irrd92Ctue82Egzw1z/RjNO3G1K9ceNGABDRF5aTy+WwuLjoeGDNzs4imUzajtNPp9NQSi0b0mh1HsPDw47F3bigGgus7mjs4h2v2YnM0tJSQ40UHX1WbrtEH6i5/TNnzgSepNWteKebTp/HmAONo4o4N/eajGeHXq7ZrjPeaQEVRnf6dt+DNdP3E++0w+nbiX4ul0MqlcLly5dd368TiOhHDHZYTrk+j9yx6+xyqqlvPZgSiYTjFYW1oJo+K7dQKCCVSi0Tbj8Hj1snnR7vuEU0nPU7uTOnkRrMpk2bsLCw4Bqf2bG0tIR0Ou1r2G2YhOn09dE7U1NTDSuf2cH9SwAaFllxmvnqBz1OtBN9P07fTfSdnL7Ta3qZFcMwYBiG44zcZDLpKPrWq0IiQj6fF9EXlsMHmpMwzc7OOk70YjG2dubaOSinOj/WqwKr07eLOPyKvpOL404vr3jH673cThhA85O0gpRgCJNMJrOs/G8r8Y7u9N2iHYZP+LrT9zvU0Q4vp+/12+uiri+VyDTj9N0+B39fdqJPRBgcHHTsyLW7Eh8bGzMXoukmIvoRgw9GN9F3Kulg5/QNw7AVXL+irzv9VkR/aWnJtZOOJ7t4HfhuOayX01+7dm1Tk7ScPne7sSu6FobT9xqjz7AB0Stxui3j6MX09LTrXItWnX7Qjlx2+k5xn91SiTp2Y/XL5TLK5bLt/pLP5zE9Pd31ETwi+hFjcHAQqVTKdmSNUspV9O2cPgukdSfkoaHWbNaaR/b09CCdTptO387BtOr0gauzcr2E2+29vEQjkUhg48aNTTn9Tuf5gL2rbsXpG4aBSqWCmZkZX07/1ltvxYc+9KGG92rF6U9NTZmjqKyiX6lUHMfUM2F35GazWcfF57lNgLPo6yPbGLt+L2ZsbAxKqWVLUXYaX6JPRPcR0QkiOklEX7G5/wtEdIyIjhLRQSLaqd33F/XnnSCi37U+V2iEiDAyMuI4nLJarQZy+k6dkLlcDoZhLHMqVldLROayiXZZJb+230zfif7+fs+OXLf3cpuIo3P99ddjYmIi0CiKbjl9O1dtF2v4gfsjLl++DKWUL9HfunUr3v/+93u2yQ+VSgWzs7NmfSGr6HtFMYB9R67ev+Q0Tt9N9AHngQF+nL7dynKAvejzOrndzvU9RZ+IkgAeBfBRADsBPKiLep3HlFI3K6V2AXgEwDfrz90J4AEANwK4D8D/WX89wYXR0VFb0XcbrgnYO32nujFOJR/shH1wcNA13vE7esfNLff392N2dtZTuJ1E3+sAZ3bt2oVUKoXnn3/e9XE67ayw6Yab02+myiZQK5MMuI/ccaNZ0ee4Mp/P266L4Ke+ke7kneIdHr8PuM/QBryvWvw4/WKx2HCicVrgCLgq+t3O9f04/b0ATiqlTimlSgAeB/AJ/QFKKX2oST8ADsk+AeBxpVRRKfUmgJP11xNcYKdvzRq9RN/O6TtNLLITfR4/bX0sO323TL9YLLoOhfSKbdjpA+5uz60Imddzgdr3cMstt+DYsWOuZZp1utmRCywX/VQq5btWP8NOn0Xfj9O3o9l4R58bYCf6fpy+n0xfv8+rYz8Mpw80jtV3i3d6e3vR19cXfacPYCMAPQQ9W9/WABE9RERvoOb0/zTIc4VG8vk8qtXqsmGbbiUYAHun7xbvAI2i77TDDg4O4sqVKzAMwzHTB5yFQCnly+kzrTh9L9EHgL1796JSqeDIkSOej+UTYTedvlu9Gb+wcF26dAnpdNoUrKA06/T1uQHNij47fS46p7eH7weumh4/gwKA5kXfbqy+W7wD1HL9lSD6vlBKPaqUuhbAnwP4yyDPJaLPEdFhIjrMTmQ1w5fe1ohndnYWROR4wPLO6Sfe6enpQTabtRV9u3iHcXL6+ntZKZfLUEq5HoD6Z/I6UHk5Oh2/8Q5QG42ydetWHDp0yHOS0alTpwB0fjYu4BzvNCP6utMfGRmxnS/hB7e1e92YmppCJpNBX19fS04fqO3fdlc81hOSn/4ht8/Siug77S+jo6MrQvTPAdis3d5U3+bE4wB+L8hzlVLfVUrtVkrt5hofqxkn0Z+bm8Pg4KDjpb1d+Vm3ujHWYZtOoq/HSc2IvlvdHSaI09dfk/Eb7zB79uzBlStX8Prrr9veX6lUcODAAfzwhz/E6Ogo3vWud/l63TBxindacfqXL19uOtrR29SM0+eTjd2yl34zfX5vu9XDrKuNdSve6enpcZzINzo6ioWFBd/rSrcDP6J/CMAOItpGRBnUOmb36Q8goh3azY8D4CNpH4AHiKiHiLYB2AHAfw/aKoUXt7Bz+m7LLto5/cXFRSSTSdsd1yr6Tpem+nu6xTtOO7JbhU0mqOhb3ytIvAMA73rXuzA4OGjboTs5OYnvfe97+M1vfoPdu3fj85//vK+Vz8ImzHiHRcgwjJZEv9mFVPS5Aez09T6gIE6fx8JbvwfrbGGvfiSv/gl+HSfR7+vrQyKRcF1O1EoUOnPtP42GUqpCRA8DOAAgCeD7SqnjRPQ1AIeVUvsAPExE9wIoA5gG8Jn6c48T0Y8BvAKgAuAhpVT3a4tGnEQiYTtsc3Z2FmvXrnV8npPTz2aztpfzuVyuYe3OdsU7fpy+33jHS/T9CmIymcTu3bvx9NNP49KlSxgbG8PMzAyee+45vPDCC8hms3jwwQdx/fXX+3q9dsALioTp9IHmR+4wQStt8th0/i77+vrMfh7en/xm+sBVp2/9HqxO388M30Qi0bTT56hVd/pO61czY2NjAGpXXDxnodN4ij4AKKX2A9hv2fZV7e8vujz36wC+3mwDVyv5fL5B9JVSmJmZwbXXXuv4HKdM30lsc7kclpaWzIPDKY8cGBgAEUEp1ZLoux2A1glhTjh1vgWNdwDgtttuw7PPPotnnnkG6XQaL7/8MgDglltuwT333NN0Z2eYWOvvlEqlptqlxw2tOH0geE19HorL78tXdfrwYLd1FPT3BWqmxk70gzp9r4VUvEQfWD5By0v0OeLqZq7vS/SFzpPP53Hq1CkopUBE5nhgt3iHDxir03e63NSLu61Zs8aMgqxZaSKRMCeiNCP6fvLaZDJpLjbdTKYfNN4BauJz44034uWXX0YqlcLu3btx5513mt9LFLC66ig4/aDlla1VPfVSDBx3eLlywNvpBx2yCbjX3/Ej+gMDA8tqCvFnsiOZTGJkZEREX1hOPp9HpVLB/Px8g5twE30uXWzN9J2chz5sk0W/r6/PNgri5eHcVrRqxekDNRFeWlpyPcjc4p1mxq/fe++92LBhA2666Sbbxea7jZ3TDzoxC7hqCPTqmc0S1Olb6/fb1d/xI/pWp2/dr/WTgp+yDoD7xEK/om+NSN2cPlDL9buZ6UvtnYhiHcHjNTGLsa6T6xXvAFdnS7p1QuVyOced2aumvp+OXKAm+k7llxkn0ffj6uwYHBzE7bffHknBB5ZPRmvV6Q8PD7dcIjro4ujT09NIJBLm/taq6JdKJc+OXL9Go1WnPzg4iMXFRfMkUywWPYf38oz7oOs6hIU4/Yiii/7WrVt9i77V6buJPmf1PILHTfTvvvtu28qIjFchNMD7AORyD27wgW3n9JsRw6iTyWTMjkIumNbK6J1W83xuU9B4Z3h42LwKsxN9P0Xt9IEKXkM2/fbxZLNZ25InwFXRd7t65EEO8/Pz5snBy+mPjY2Zhe+Gh4ddH9sORPQjSi6XQyKRMLM/r9m4jL4snlLKtW4ML6bCr10oFMzRBVa85k+4iT6XVfaaEPTBD34QCwsLro/hqwq7cfrNOP2o09PTY+4DzRZbA6661bBEP4jTn5qaanhfXpDG6vS9rrZ0p293xZNIJJBMJhucvtd35XbVwqtmue23+lh9fi8/8Q5QG8HTDdGXeCei8LBN7iSanZ1Ff3+/56W5tRKh04gbRh+r30phMbd4x8+lO1A7GLZs2dLUe/l9j5WGLrDNllXWn+PWyRjktYI6fV30ef2EoPGOV0eu3rYg8Y5bpu8W7QD2y4n6iXeA7o3VF9GPMPqwTV4m0Qvd6fsZNTM8PIyZmRkopTyHm7nhFe+EWbvGLoeNa7yjd+S2Ivq9vb341Kc+hfe+972htMnvOrmLi4tYWlpaNmKor6+voeCdH9FnAeaThZPoB8n02enb5et+RF93+l51d/TnZDKZro3gEdGPMHq1Ta/ZuIzu9N1KMDAc7xSLRRiG0XSNmVYWNwnjveIa7+gLkbci+gCwY8eOUL6jIOvkWkfuMLxSGlCLIf3sIzwbmJ9n9z1wUbYgmT5gPyu3Wq16in5/fz+IyFxvgj+b1+foZuE1Ef0Ik8/nUSqVsLCw4Lo2ro7u9J0qbOrwYipc6K4Vp++UjYa98pSd6MfZ6QONnZPd/pxBiq5Zx+gzerzjpyAfk06nzY5tP07f67tym2Pix+knEgn09/c3iL4f49TNwmsi+hGGs78LFy5gcXGxaafvJfoAcP78eQDNV5NkIba7TA7b6a+2TB+ofb6oiL5dTSAn3Jw+i6Sf/ZTx6/SDxDuA/QnMj+gDMBdIX1xcRDqd9jWPYnR0FDMzM76ulsJGRD/CsDt66623AHgP1wTsM32vjlwA5vKBrYg+YH/wuC2K3ux76e/DQxnjKPpRdvp+RH96ehoDAwPLhLCvrw9LS0swDCPQbOpOO30/As71d4Isq9nNpRNF9CNMLpcDEeH06dMA/Is+uwe/8Q5wVfRbiXcA+4OnXU6fryqCFltbSehONCqiH2T1LL26pg7vZ4uLi4FEP5PJeHbkstPPZDKew4TDcvoc7/g9fvTCa51GRD/CJJNJDA8P49y52hIEzTp9rwJm2WwWk5OTAFp3+tYlCHmmYtiZvlJq2dJ5cXT6UYx3/Dr9arWKyclJV9EvFAqBnb7d3/o2viry83qtZvpAzekvLCxgfn7et+jzdyKiLywjn8+bQ+P8dORaM32nsso6uVzOXImqWdHntlln1DZTCM0L64HajveIClGOd7yc/iuvvILFxUXceOONy+7TRT9opm/3N8P7v9+ryzBEn/f9S5cu+T5+MpkMhoaGRPSF5bAjyGazvg72VCqFarXqa11ahiOeTCbTdF0WvWKnTpAD2i+rSfStTp+IWq6d0yp+O3Kff/555PN523LgzTp9L9HneMev0w8j3uGx+nZF4NzoVuE1Ef2Iw6LvJ9oBGhdH99uxZC2E1QwDAwNIJBINK3EB7RFka1XP1ZDp67NQm13fNiz8xDvvvPMOzp49iz179ti2N4x4x8npG4aBQqHg2ySlUilHp+/nBOu1yJATPGyz04XXRPQjTlDRZ2dSLpcDO/1WFv8mooY6Pkw7nb61PEEcnb7uRO0qS3YDP/HOoUOHkE6nsWvXLtv7WfQXFhYCnbS9Mn1+jfn5ed/7g1MJkaDxDhDMOI2NjaFYLHrWmwobEf2Iw6Lvd41W3el3UvT5dTrh9FdTvJNMJpFIJMx4Jwqiz+s2ODn9QqGAY8eO4T3veY/j/pdKpcyROEtLS8hkMr7WQuDPn06nba8gmhF9p4mFfkVfLxQXNN4BOt+ZK6IfcUZGRtDT0+NZ5ZLRnb7fAmphxDsAXJ1+J0Q/CoIYNvpC5FERfcC9OuWLL76IarWKvXv3ur4G198JMqSXTY3T98D3G4bh+7ty+ix+RT+ZTJrHTtB4B+h84TUprRxxkskkHn74Yd87k9XpB8n0W41gWPQNwzBdm59F0YNizfTjHO8AV4uuRUn0eRKUFcMwcPjwYWzbts3TqPCs3HQ67fu348/v9D3o24M4fWu8YxgGDMPwJfpA7Uo8aMHCXC6HZDIpTl9YzsDAgO8RG7yTLi0toVKp+BLbgYEB9PX1tVx2l+v46BllO5w+L6DNJ5RisYhkMtn1US3tgp1olETfyR2fOHECMzMzni4fuCr6Qaqw+nX63EY/2Im+n1WzdDh+DSL6iUSiKzV4xOnHDN7peU1dPwdTIpHAww8/3LIw62vu8kFQLBaRTqcDr13rBhE1dL7Fte4OwzX1oyT6TjX1Dx06hFwuh+uvv97zNfr6+nDp0iUQkW/Rb4fTtzuBBRV9HrYZtF9sdHTUnBjZKcTpxwzeSVn0/e6Evb29LQszjzDSO3PDrqXP6J1vcS2rzOjxTjOLorcDu9WzCoUC3nzzTdx6662+9iXd6YeV6XuN47cjDKe/du1aDA4OBj4pj46OYnp62pwc2QnE6ceMZpx+WNhN0GqXC9cP1LiWVWZ6enowMzMTKaevL+PI8O3169f7eo2+vj6zdHhYmX4z8Q4vClOtVs2IMKjo33777bjtttsCz6EYHR2FYRiYnp52XKo0bMTpxwzeSbkSYSdFP5vNIp1ONzj9sGvp6++ld+TG2emvlHiHyyj77RvSi661w+kHyfSBxnkHQUU/kUg09dt0o/CaiH7MsDr9VsfeB4GIkMvlOuL0V1umXygUoJSKlOhb452pqSkQke/FvvVOz6BO3ynmatbpA62JfrN0Y6y+iH7MsGb6nXT6wPIJWmHX0mf0TH81xDuc+Ublc/KQTb2EwNTUlDkM0Q+66IfVketVpsEOu6JrnRL93t5es0O7U4joxwze6bsR7wDLJ2h1wunHPd7RP1tURN+u6NrU1JRtGWUnmnH6XvFOIpEwhTpovNMN0Qc6v3SiiH7M4J2Uh0p2euz60NAQ5ufnzYOmnaN3uPMt7k6/mREp7cau6FonRD+bzSKXy7lO/OK2+X1NbgcbJUBEX1hBJBIJc7hcJ/N8hkfwzM3NoVqtolwuty3eAWrurF3vERWi6PStRdcWFxextLQUSPT1/dPv75dMJvGlL30JO3fudHwM1+XxO7x1dHQUiUSiYbx8J0V/bGwMCwsLtkXf2oGIfgzhnb3T0Q7QOEGrHSUYGH5NjpLiLPpRdPrWeIedahDRTyQSpvCHuY9kMplAJaiTyeSySVKddvpA5zpzfYk+Ed1HRCeI6CQRfcXm/i8T0StE9DIR/RMRbdXue4SIjhPRq0T0f1C3i4GvAnhH7Ybo6xO02ln9kl+TO42jIobtYCU4fR6uGUT0gavRSpj7SJBaPsz4+DguXLhg3u6G6HeqM9dT9IkoCeBRAB8FsBPAg0RkvbY6AmC3UuoWAE8AeKT+3DsB/BsAtwC4CcAeAL8TWusFW9jpdzPemZ2dbUstfWY1Of0oir7V6bPoj4yMBHqddoh+JpMJ/Hpr165tMCqdFP18Pg8iipTT3wvgpFLqlFKqBOBxAJ/QH6CUelopVajf/A2ATXwXgCyADIAeAGkAFyC0lW46/XQ6jd7e3rY7ff5s7PTjLPpRjHesHbnT09PI5XKBRbIdor9lyxZs3brV+4Ea4+PjAGBGPJ0U/WQyiZGRkY6Jvp9PtBHAGe32WQC3uzz+swD+JwAopX5NRE8DOA+AAHxHKfWq9QlE9DkAnwNqP5jQGt3M9AGYE7TaUWGTEaffXezinaDRDlC7Gg3S6eqHu+++O/Bz1q5dCwC4cOECNm/e3FHRBzo7gifUjlwi+jSA3QC+Ub99HYB3o+b8NwL4EBG93/o8pdR3lVK7lVK7/S4WIjjTTacPXJ2g1c6OXBZCFv2oiGE70D9bVAqu2cU7zYj+tm3bcP3113d93d9cLoeenh4z169UKiCiUKvDutHJ9XL9fKJzADZrtzfVtzVARPcC+I8A7ldK8XzmTwL4jVJqXik1j9oVwB2tNVnwopuZPnB1glYnO3Lj7PRZ9FOpVMdEyAvd6S8tLaFQKDQl+rfccgseeOCBsJsXGCLC2rVrG+KdVCrVsZPR6OgoKpXKspXn2oGfPegQgB1EtI2IMgAeALBPfwAR3Qrg/0JN8PXi0G8D+B0iShFRGrVO3GXxjhAu3Xb6Q0NDWFpaMnfgdrSDC1xxuYk4iz4vmRilqxl9ndxmR+5EDRZ9pZTvpRLDgguvdWIEj6foK6UqAB4GcAA1wf6xUuo4EX2NiO6vP+wbAAYA/ISIjhIRnxSeAPAGgGMAXgLwklLqybA/hNBIFDJ9ALh48aK5ylU7yGazMAwDQLzjHaB2UovaZ+RKm3ER/fHxcdOsdFr0OzlW39enUkrtB7Dfsu2r2t/3OjyvCuDzrTRQCA7vrN2Kd1j0Jycn2+rAs9ksZmdnkUwmO3qAdoNMJhO55SB5cZdmh2tGDe7MnZyc7LjoDwwMIJPJRMPpCyuPKMQ7AHDlypW2toFfO2oOuB1E1ekXi0VMTU1haGgoMp3MzcLDNi9cuNBx0ScibNy4sSN9CPG2R6uUbsc7g4ODICIopdrq9Pm145znMzfeeGPXR7hYYacftOZOVMlmsxgaGuqK0weAP/7jP+7I+4jox5BuxzvJZBIDAwOYm5tre7wDrA7Rv/POO7vdhGVkMhnMz89jdnbW10LoKwEux9Df3x/byDCen2qVc91112FhYaGrO20ul8Pc3FxbrzZY7KMWe6wWePTUwsJCLJw+UMv133jjDaTT6djuV5Lpx5CtW7fi/vvv72ocwJ254vTjCzt9YOWP3GHGx8dhGAYmJydj6/RF9IW2wJ25IvrxRf/e/S6GHnV4BE+5XBbRF4QgsOjL6J34on/vK324JjM2NmbOehbRF4QAdCLeWU2jd6IIiz6PMY8DyWTSnB0roi8IAWDRF6cfX/hkG5c8n+Hx+iL6ghCA8fFx7NmzB9dee23b3kMy/e7CJ9u4iT7n+nEV/Xh+KqHrJJNJfOxjH2vre4jT7y5xdfpxF31x+sKKJZ/P484778QNN9zQ7aasSuLq9DneiVqto7CI56lMWBUkEgl8+MMf7nYzVi2bNm3CHXfcgeuuu67bTQmVoaEh3H333di507oUeDwQ0RcEoSnS6TQ+8pGPdLsZoUNE+MAHPtDtZrQNiXcEQRBWESL6giAIqwgRfUEQhFWEiL4gCMIqQkRfEARhFSGiLwiCsIoQ0RcEQVhFiOgLgiCsIkgp1e02NEBEFwGcbuElxgBcCqk57WIltBGQdobJSmgjIO0Mk063cQIOsWEAAARmSURBVKtSao3XgyIn+q1CRIeVUru73Q43VkIbAWlnmKyENgLSzjCJahsl3hEEQVhFiOgLgiCsIuIo+t/tdgN8sBLaCEg7w2QltBGQdoZJJNsYu0xfEARBcCaOTl8QBEFwIDaiT0T3EdEJIjpJRF/pdnsYIvo+EU0S0W+1bXki+gURvV7/f6TLbdxMRE8T0StEdJyIvhjRdmaJ6Hkieqnezv9c376NiP61/tv/iIgisX4iESWJ6AgR/X39duTaSURvEdExIjpKRIfr26L2uw8T0RNE9BoRvUpEd0SwjTfUv0P+N0tEX4paO4GYiD4RJQE8CuCjAHYCeJCIorLszf8L4D7Ltq8A+Cel1A4A/1S/3U0qAP5XpdROAO8D8FD9+4taO4sAPqSUeg+AXQDuI6L3AfgvAP6rUuo6ANMAPtvFNup8EcCr2u2otvNupdQubXhh1H73bwP4B6XUuwC8B7XvNFJtVEqdqH+HuwDcBqAA4H8gYu0EACilVvw/AHcAOKDd/gsAf9HtdmntuQbAb7XbJwCsr/+9HsCJbrfR0t6fAfhwlNsJoA/AiwBuR20CTMpuX+hi+zahdpB/CMDfA6CItvMtAGOWbZH53QHkALyJev9jFNto0+aPAPhVVNsZC6cPYCOAM9rts/VtUWVcKXW+/vcEgPFuNkaHiK4BcCuAf0UE21mPTI4CmATwCwBvALiilKrUHxKV3/5bAP4DAKN+exTRbKcC8BQRvUBEn6tvi9Lvvg3ARQD/Tz0q+x4R9SNabbTyAIAf1v+OXDvjIvorFlWzAJEYQkVEAwB+CuBLSqlZ/b6otFMpVVW1S+hNAPYCeFeXm7QMIvq3ACaVUi90uy0+uEsp9V7UotGHiKhhcdgI/O4pAO8F8NdKqVsBLMASkUSgjSb1fpr7AfzEel9U2hkX0T8HYLN2e1N9W1S5QETrAaD+/2SX2wMiSqMm+D9QSv1tfXPk2skopa4AeBq1mGSYiFL1u6Lw2/8bAPcT0VsAHkct4vk2otdOKKXO1f+fRC2D3oto/e5nAZxVSv1r/fYTqJ0EotRGnY8CeFEpdaF+O3LtjIvoHwKwoz46IoPa5dW+LrfJjX0APlP/+zOoZehdg4gIwP8N4FWl1De1u6LWzjVENFz/uxe1fodXURP/f1d/WNfbqZT6C6XUJqXUNajti/+slPr3iFg7iaifiAb5b9Sy6N8iQr+7UmoCwBkiuqG+6R4AryBCbbTwIK5GO0AU29ntToUQO08+BuD/Qy3j/Y/dbo/Wrh8COA+gjJpr+Sxq+e4/AXgdwD8CyHe5jXehdtn5MoCj9X8fi2A7bwFwpN7O3wL4an37dgDPAziJ2mV1T7d/d63NHwTw91FsZ709L9X/HefjJoK/+y4Ah+u/+98BGIlaG+vt7AdwGUBO2xa5dsqMXEEQhFVEXOIdQRAEwQci+oIgCKsIEX1BEIRVhIi+IAjCKkJEXxAEYRUhoi8IgrCKENEXBEFYRYjoC4IgrCL+f5ELtM5CtKYUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.Series(results).plot.line(color=\"grey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Low-level model classes\n",
    "\n",
    "The `cophi.corpus()` function wraps three classes:\n",
    "\n",
    "1. `cophi.model.Textfile` models a document on the file level.\n",
    "2. `cophi.model.Document` models a document on the text level.\n",
    "3. `cophi.model.Corpus` models a collection of documents on the corpus level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Textfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = \"/home/severin/kafka_derprozeß.xml\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = cophi.model.Textfile(filepath=file,\n",
    "                         treat_as=\".xml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kafka_derprozeß'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'.xml'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.suffix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/severin'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.parent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Der Prozeß'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.content.strip()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Context-manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 10 characters of 'kafka_derprozeß.xml', treated as XML: \n",
      "Der Prozeß\n"
     ]
    }
   ],
   "source": [
    "with cophi.model.Textfile(file, treat_as=\".xml\") as document:\n",
    "    print(f\"First 10 characters of '{document.title + document.suffix}', treated as XML: \\n{document.content.strip()[:10]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 39 characters of 'kafka_derprozeß.xml', treated as plain text: \n",
      "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with cophi.model.Textfile(file, treat_as=\".txt\") as document:\n",
    "    print(f\"First 39 characters of '{document.title + document.suffix}', treated as plain text: \\n{document.content.strip()[:39]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Processing XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<lxml.etree._ElementTree at 0x7fedb490ad08>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lxml import etree\n",
    "\n",
    "tree = f.parse_xml(parser=etree.XMLParser())\n",
    "tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Jemand mußte Josef K. verleumdet haben, denn ohne daß er etwas Böses getan hätte, wurde er eines Morgens verhaftet. Die Köchin der Frau Grubach, seiner Zimmervermieterin, die ihm jeden Tag gegen acht Uhr früh das Frühstück brachte, kam diesmal nicht. Das war noch niemals geschehen. K. wartete noch ein Weilchen, sah von seinem Kopfkissen aus die alte Frau, die ihm gegenüber wohnte und die ihn mit einer an ihr ganz ungewöhnlichen Neugierde beobachtete, dann aber, gleichzeitig befremdet und hungrig, läutete er. Sofort klopfte es und ein Mann, den er in dieser Wohnung noch niemals gesehen hatte, trat ein. Er war schlank und doch fest gebaut, er trug ein anliegendes schwarzes Kleid, das, ähnlich den Reiseanzügen, mit verschiedenen Falten, Taschen, Schnallen, Knöpfen und einem Gürtel versehen war und infolgedessen, ohne daß man sich darüber klar wurde, wozu es dienen sollte, besonders praktisch erschien. »Wer sind Sie?« fragte K. und saß gleich halb aufrecht im Bett. Der Mann aber ging über die Frage hinweg, als müsse man seine Erscheinung hinnehmen, und sagte bloß seinerseits: »Sie haben geläutet?« »Anna soll mir das Frühstück bringen«, sagte K. und versuchte, zunächst stillschweigend, durch Aufmerksamkeit und Überlegung festzustellen, wer der Mann eigentlich war. Aber dieser setzte sich nicht allzulange seinen '"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree.xpath(\"tei:text//text()\", namespaces={\"tei\": \"http://www.tei-c.org/ns/1.0\"})[31]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use case: Segmenting and tokenizing a textfile\n",
    "\n",
    "Construct a `Document` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = cophi.model.Document(text=f.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Segmenting the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71 segments in 'kafka_derprozeß'.\n"
     ]
    }
   ],
   "source": [
    "segments = list(d.segments(size=1000))\n",
    "print(\"{} segments in '{}'.\".format(len(segments), f.title))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating labels for the segment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = [\"{}_{}\".format(f.title, n) for n in range(len(segments))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "kafka_derprozeß_0    [der, prozeß, hdl, de, textgrid:qmx, der, anno...\n",
       "kafka_derprozeß_1    [vielleicht, noch, besaß, schätzte, er, nicht,...\n",
       "kafka_derprozeß_2    [worte, die, ich, mit, einem, mir, ebenbürtige...\n",
       "kafka_derprozeß_3    [gehen, pflegte, spät, nach, hause, kam, und, ...\n",
       "kafka_derprozeß_4    [er, dann, hinüber, die, drei, wichen, auch, s...\n",
       "kafka_derprozeß_5    [mann, aufmerksam, machte, den, er, selbst, sc...\n",
       "kafka_derprozeß_6    [ja, bloß, die, vermieterin, nun, ich, habe, a...\n",
       "kafka_derprozeß_7    [fenster, lag, und, die, müden, augen, drückte...\n",
       "kafka_derprozeß_8    [den, leib, schickt, da, sie, aber, doch, frei...\n",
       "kafka_derprozeß_9    [hat, eine, größere, summe, von, mir, geliehen...\n",
       "dtype: object"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(segments, index=labels).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Low-level helper functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The return value is a <class 'generator'>.\n",
      "Use it e.g. as a list: list(tokens) -> ['Jemand', 'mußte', 'Josef', 'K']\n",
      "Note that it stopped tokenizing after 4 words (because you set maximum=4).\n"
     ]
    }
   ],
   "source": [
    "tokens = cophi.utils.find_tokens(document=\"Jemand mußte Josef K. verleumdet haben.\",\n",
    "                                 token_pattern=\"\\w+\",\n",
    "                                 maximum=4)\n",
    "\n",
    "print(\"The return value is a {}.\".format(type(tokens)))\n",
    "print(\"Use it e.g. as a list: list(tokens) -> {}\".format(list(tokens)))\n",
    "print(\"Note that it stopped tokenizing after 4 words (because you set maximum=4).\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Constructing ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The return value is also a <class 'generator'>.\n",
      "You can use it as a list, too: list(bigrams) -> ['Jemand mußte', 'mußte Josef', 'Josef K']\n"
     ]
    }
   ],
   "source": [
    "bigrams = cophi.utils.construct_ngrams(tokens=['Jemand', 'mußte', 'Josef', 'K'],\n",
    "                                       n=2,\n",
    "                                       sep=\" \")\n",
    "print(\"The return value is also a {}.\".format(type(bigrams)))\n",
    "print(\"You can use it as a list, too: list(bigrams) -> {}\".format(list(bigrams)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}