wodehouse/index.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "8257da14-b6c6-4f5b-8381-9fc086d68d7d",
   "metadata": {},
   "source": [
    "---\n",
    "categories:\n",
    "  - gutenberg\n",
    "date: '2024-02-07T18:03:00+11:00'\n",
    "image: ./my_man_jeeves.jpg\n",
    "title: A dataset of P. G. Wodehouse books\n",
    "format:\n",
    "  html:\n",
    "    toc: true\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe81ce90-cba0-46ea-8839-2c4c8f304d79",
   "metadata": {},
   "source": [
    "We are going to create a text dataset from the books of [P. G. Wodehouse](https://en.wikipedia.org/wiki/P._G._Wodehouse) for language modelling.\n",
    "I enjoy his style of writing, and his books are in the public domain in many countries, so it's a fun dataset to train language models on.\n",
    "\n",
    "Here we will download his books from Project Gutenberg, do some light exploratory analysis and data cleaning, and save the dataset for future language modelling.\n",
    "\n",
    "::: {.callout-note}\n",
    "This is a Jupyter notebook you can [download](https://github.com/EdwardJRoss/skeptric/blob/master/wodehouse/index.ipynb) \n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "be912e81-16d0-4136-a14a-0b4e63f1e18a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| echo: false\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe5ab7fc-cd93-4978-b137-3458d747ed17",
   "metadata": {},
   "source": [
    "# Load the Data\n",
    "\n",
    "We'll download all the books written by P. G. Wodehouse from Project Gutenberg following my guide to [downloading from Project Gutenberg](https://skeptric.com/gutenberg/).\n",
    "Firstly we only want to download things once when iterating with this code, so I'll create a Python [shelve](https://docs.python.org/3/library/shelve.html) to store the downloaded data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a87d9106-7eb4-451b-a200-a8fdd44f8772",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "import shelve\n",
    "\n",
    "data_path = Path('./data/')\n",
    "data_path.mkdir(exist_ok=True)\n",
    "\n",
    "wodehouse_books = shelve.open(data_path / 'wodehouse.shelve')\n",
    "\n",
    "len(wodehouse_books)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30da7cdd-66e8-4920-8098-f5019db2c112",
   "metadata": {},
   "source": [
    "Then we search the index to find all P.G Wodehouse texts (excluding indexes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bdd18c2a-1867-48c9-98ae-e9afb34060a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "from io import StringIO\n",
    "import requests\n",
    "\n",
    "if not wodehouse_books:\n",
    "    GUTENBERG_CSV_URL = \"https://www.gutenberg.org/cache/epub/feeds/pg_catalog.csv.gz\"\n",
    "    \n",
    "    r = requests.get(GUTENBERG_CSV_URL)\n",
    "    csv_text = r.content.decode(\"utf-8\")\n",
    "    \n",
    "    for b in csv.DictReader(StringIO(csv_text)):\n",
    "        if \"Wodehouse, P. G.\" in b[\"Authors\"] \\\n",
    "            and \"Indexes\" not in b[\"Subjects\"] \\\n",
    "            and b[\"Type\"] == \"Text\":\n",
    "            wodehouse_books[b[\"Text#\"]] = b\n",
    "    \n",
    "    len(wodehouse_books)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c660e7a-2e41-4ece-ab92-7e4c473a6d29",
   "metadata": {},
   "source": [
    "And we download the text; we make a slight deviation here in using the txt files rather than the zip files.\n",
    "(Text# 70222 (Meet Mr Mulliner) is [only available as a txt](http://aleph.gutenberg.org/7/0/2/2/70222/))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "beb016c9-b18d-4507-b750-622f6b895312",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gutenberg_text_urls(id: str,\n",
    "                        mirror=\"http://aleph.gutenberg.org\",\n",
    "                        suffixes=(\"\", \"-8\", \"-0\")) -> list[str]:\n",
    "    path = \"/\".join(id[:-1]) or \"0\"\n",
    "    return [f\"{mirror}/{path}/{id}/{id}{suffix}.txt\" for suffix in suffixes]\n",
    "\n",
    "def download_gutenberg(id: str) -> str:\n",
    "    for url in gutenberg_text_urls(id):\n",
    "        r = requests.get(url)\n",
    "        if r.status_code == 404:\n",
    "            # Try the next possible URL\n",
    "            continue\n",
    "        r.raise_for_status()\n",
    "        break\n",
    "    else:\n",
    "        raise Exception(\"Unable to download %s\" % id)\n",
    "\n",
    "    return r.content.decode('utf-8')\n",
    "\n",
    "for book_id, book in wodehouse_books.items():\n",
    "    if \"raw_text\" not in book:\n",
    "        # Note: Assign the object for shelve\n",
    "        wodehouse_books[book_id] = book | dict(raw_text=download_gutenberg(book_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0ef5276c-149e-4c26-8fd6-9067bd1dcd76",
   "metadata": {},
   "outputs": [],
   "source": [
    "wodehouse_books.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ad8730d-4c1e-46e9-b9ef-f902fade123c",
   "metadata": {},
   "source": [
    "# Looking at the Data\n",
    "\n",
    "Let's have a look at our corpus of P. G. Wodehouse books and clean them up a little.\n",
    "\n",
    "First let's load the data from our shelf into a dictionary, and store the individual book ids and raw text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8e0ae697-bf01-4799-80ed-9337cd7b84d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shelf = shelve.open(data_path / 'wodehouse.shelve')\n",
    "wodehouse_books = {k:v for k,v in shelf.items()}\n",
    "shelf.close()\n",
    "\n",
    "book_ids = list(wodehouse_books.keys())\n",
    "raw_texts = [book['raw_text'] for book in wodehouse_books.values()]\n",
    "\n",
    "len(wodehouse_books)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6186e79-4505-47e9-9ad4-ee32cdf9d129",
   "metadata": {},
   "source": [
    "We can look at a small section of text to get an idea of the writing style."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "826b6d92-6f76-4af5-b273-017d652d6822",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "he is, I'll bet he doesn't\n",
      "show it. He's not like Blackburn. I wish he was. Here he comes, so\n",
      "perhaps we'd better talk about something else.\"\n",
      "\n",
      "The vanguard of the boys returning from preparation had passed them,\n",
      "and they were now standing at the gate of the house. As Fenn spoke, a\n",
      "little, restless-looking man in cap and gown came up. His clean-shaven\n",
      "face wore an expression of extreme alertness--the sort of look a ferret\n",
      "wears as he slips in at the mouth of a rabbit-hole. A doctor, calle\n"
     ]
    }
   ],
   "source": [
    "print(raw_texts[13][14_500:15_000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f64bbe2d-4f4e-41a8-805a-46ce68f9f49f",
   "metadata": {},
   "source": [
    "The Project Gutenberg metadata contains the titles for each of the books.\n",
    "\n",
    "If we compare with [P. G. Wodehouse's Bibliography](https://en.wikipedia.org/wiki/P._G._Wodehouse_bibliography) we can see a lot of texts are missing from Project Gutenberg, and that we have a mixture of Short Stories and Novels.\n",
    "Considerably more are available [on archive.org](https://archive.org/search?query=wodehouse&and%5B%5D=lending%3A%22is_readable%22&and%5B%5D=mediatype%3A%22texts%22) but their text is low-quality OCR and so requires better transcription (a fun project for another day)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d2be3327-bc7f-43bc-bc5d-cbd20f3be977",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Adventures of Sally       Carry On, Jeeves              The Clicking of Cuthbert      \n",
      "The Coming of Bill            A Damsel in Distress          Death at the Excelsior, ...   \n",
      "Divots                        The Gem Collector             A Gentleman of Leisure        \n",
      "The Girl on the Boat          The Gold Bat                  The Head of Kay's             \n",
      "Indiscretions of Archie       The Inimitable Jeeves         The Intrusion of Jimmy        \n",
      "Jill the Reckless             Leave it to Psmith            The Little Nugget             \n",
      "The Little Warrior            Love Among the Chickens       Love Among the Chickens ...   \n",
      "The Man Upstairs and Oth...   A Man of Means                The Man with Two Left Fe...   \n",
      "Meet Mr Mulliner              Mike                          Mike and Psmith               \n",
      "My Man Jeeves                 Not George Washington — ...   Piccadilly Jim                \n",
      "The Politeness of Prince...   The Pothunters                A Prefect's Uncle             \n",
      "The Prince and Betty          Psmith in the City            Psmith, Journalist            \n",
      "Right Ho, Jeeves              Sam in the Suburbs            Something New                 \n",
      "The Swoop! or, How Clare...   Tales of St. Austin's         Three Men and a Maid          \n",
      "Ukridge                       Uneasy Money                  The White Feather             \n",
      "William Tell Told Again       A Wodehouse Miscellany: ...   The small bachelor            \n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "def fmtcols(items: list[str], n_cols: int, max_len: int=40, buffer: int=5):\n",
    "    \"\"\"Formats a list of strings into a table with n_cols columns\n",
    "\n",
    "      items:   The strings to format\n",
    "      ncols:   The number of columns per line\n",
    "      max_len: The length of each field\n",
    "      buffer:  Space from end of text to the end of each field\n",
    "    \"\"\"\n",
    "    items_truncated = [item if len(item) < (max_len - buffer) \n",
    "                       else item[:max_len-buffer-3] + \"...\"\n",
    "                       for item in items]\n",
    "    format_str = \"{:<\" + str(max_len) + \"}\"\n",
    "    lines = (\n",
    "        (format_str * len(items[i:i+n_cols]))\n",
    "        .format(*items_truncated[i:i+n_cols])\n",
    "             for i in range(0,len(items),n_cols))\n",
    "    return '\\n'.join(lines)\n",
    "\n",
    "book_titles = sorted([re.sub('\\s+', ' ', book[\"Title\"])\n",
    "                      for book in wodehouse_books.values()],\n",
    "                     key=lambda x: re.sub(\"^(A |The )\", \"\", x))\n",
    "\n",
    "print(fmtcols(book_titles, 3, max_len=30, buffer=3))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4b3387f9-26ab-4e83-8b0f-e72adbb02def",
   "metadata": {},
   "source": [
    "# Removing footers and headers\n",
    "\n",
    "\n",
    "If we have a look at the start of any of the books there is a header inserted by Project Gutenberg before the main text starts.\n",
    "We can copy the [method of gutenbergr](https://github.com/ropensci/gutenbergr/blob/96cf4f5d06b3106d32f6720ac32172540f4cd65f/R/gutenberg_download.R#L186-L228) for cleaning the text.\n",
    "\n",
    "Here's my non-vectorised regular expression based version for finding the header:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b267b911-232f-4073-9f01-7a4a630b8f0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** START OF THIS PROJECT GUTENBERG EBOOK A DAMSEL IN DISTRESS ***\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Produced by Jim Tinsley\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "[Transcriber's Note for edition 11: in para. 4 of Chapter 19, the\n",
      "word \"leafy\" has been changed to \"leaky\". \"leafy\" was the word used\n",
      "in the printed edition, but was an obvious misprint. Some readers\n",
      "have noted that other editions have slightly different punctuation,\n",
      "notably some extra commas, and semi-colons where there are colons in\n",
      "this edition; but the punctuation herein does follow at least one\n",
      "printed text.--jt]\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "HEADER_RE = re.compile(\"\"\"\n",
    "(?:                                  # Find the end of the header\n",
    "  ^\\\\*\\\\*\\\\*.*PROJECT\\ GUTENBERG|     \n",
    "  END.*SMALL\\ PRINT                   \n",
    ")\n",
    ".*                                  # Until the end of the line\n",
    "[\\r\\n]+                             # And some number of blank lines\n",
    "(?:                                   # Search for informational paragraphs\n",
    "  (?i:                                # Their first line contains:\n",
    "    .*Produced\\ by\n",
    "    |.*prepared\\ by\n",
    "    |.*transcribed\\ from\n",
    "    |.*project\\ gutenberg\n",
    "    |special\\ thanks\n",
    "    |note:\\ \n",
    "    |\\\\[?transcriber[’']s\\ note           # Not from Gutenbergr\n",
    "    |for\\ this\\ text\\ version             # but useful here\n",
    "  )\n",
    "  (?:[^\\r\\n]*[\\r]?[\\n])               # Until the end of the line\n",
    "  (?:[^\\r\\n]+[\\r]?[\\n])*              # Until the end of the paragraph\n",
    "  [\\r\\n]+                             # And all newlines beneath it  \n",
    ")*                                  # Repeatedly \n",
    "\"\"\", \n",
    "flags=re.MULTILINE | re.VERBOSE)\n",
    "\n",
    "print(HEADER_RE.search(raw_texts[2]).group())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7c96349-09fb-4b88-aada-c9436aab96f7",
   "metadata": {},
   "source": [
    "As a sanity check all all the headers are below about 1000 characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dfd26c6a-e0b2-4438-a0a9-3ecaeb6619c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj8AAAGfCAYAAACwWLNEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8C0lEQVR4nO3dfXRU1aH+8WdCmASQTAiYTFIDRLRg5E2xxFShtaQEpLS2tlUExYqgNPgCXpumPytgbw2FLlv1Il57FXqXKNa7FAWVGt6rRlAwQohNBaOhmkmsmBlACZDs3x+unDIkIZlkZjIz5/tZ6yyYc/bM7Dnzcp7svc8+DmOMEQAAgE3EdXcFAAAAwonwAwAAbIXwAwAAbIXwAwAAbIXwAwAAbIXwAwAAbIXwAwAAbIXwAwAAbIXwAwAAbIXwAwAAbCU+kMLFxcV67rnn9Pe//129evXSN7/5Tf3ud7/T0KFDrTLHjh3TXXfdpTVr1qihoUH5+fl65JFHlJaWZpWprq7W3LlztWXLFp111lmaOXOmiouLFR//7+ps3bpVCxYs0L59+5SZmal77rlHN954Y4fr2tTUpE8++UR9+/aVw+EI5GUCAIBuYozR4cOHlZGRobi4ELXRmADk5+eblStXmvLyclNWVmauvPJKM3DgQHPkyBGrzK233moyMzPNpk2bzNtvv20uvfRS881vftPafvLkSTN8+HCTl5dn3nnnHfPyyy+bAQMGmKKiIqvMBx98YHr37m0WLFhgKioqzMMPP2x69OhhNmzY0OG6Hjx40EhiYWFhYWFhicLl4MGDgUSUgDiM6fyFTT/99FOlpqZq27ZtGj9+vLxer84++2w99dRT+vGPfyxJ+vvf/64LLrhApaWluvTSS/XKK6/oe9/7nj755BOrNejRRx9VYWGhPv30UzmdThUWFuqll15SeXm59VzXXnut6uvrtWHDhg7Vzev1Kjk5WQcPHlRSUlJnXyIAAAgjn8+nzMxM1dfXy+VyheQ5Aur2Op3X65UkpaSkSJJ27dqlEydOKC8vzyozbNgwDRw40Ao/paWlGjFihF83WH5+vubOnat9+/bpoosuUmlpqd9jNJe5884726xLQ0ODGhoarNuHDx+WJCUlJRF+AACIMqEcstLpzrSmpibdeeeduuyyyzR8+HBJksfjkdPpVHJysl/ZtLQ0eTweq8ypwad5e/O2M5Xx+Xz68ssvW61PcXGxXC6XtWRmZnb2pQEAgBjW6fBTUFCg8vJyrVmzJpj16bSioiJ5vV5rOXjwYHdXCQAARKBOdXvNmzdP69ev1/bt23XOOedY691ut44fP676+nq/1p/a2lq53W6rzM6dO/0er7a21trW/G/zulPLJCUlqVevXq3WKSEhQQkJCZ15OQAAwEYCavkxxmjevHl6/vnntXnzZmVlZfltHzNmjHr27KlNmzZZ6yorK1VdXa3c3FxJUm5urvbu3au6ujqrTElJiZKSkpSdnW2VOfUxmss0PwYAAEBnBXS2189//nM99dRTeuGFF/zm9nG5XFaLzNy5c/Xyyy9r1apVSkpK0m233SZJeuONNyRJjY2NGj16tDIyMrR06VJ5PB5df/31uvnmm3X//fdLkqqqqjR8+HAVFBTopptu0ubNm3X77bfrpZdeUn5+fofq6vP55HK55PV6GfAMAECUCMfxO6Dw09bI65UrV1oTEDZPcvj000/7TXLY3KUlSR999JHmzp2rrVu3qk+fPpo5c6aWLFnSYpLD+fPnq6KiQuecc45+/etfBzTJIeEHAIDoE3HhJ5oQfgAAiD7hOH5zbS8AAGArXZrk0G4am4x2Vh1S3eFjSu2bqLFZKeoRx3XDAACIJoSfDtpQXqPF6ypU4z1mrUt3JWrh1GxNGp7ejTUDAACBoNurAzaU12juk7v9go8kebzHNPfJ3dpQXtNNNQMAAIEi/LSjsclo8boKtTYqvHnd4nUVamyKyXHjAADEHMJPO3ZWHWrR4nMqI6nGe0w7qw6Fr1IAAKDTCD/tqDvcdvDpTDkAANC9CD/tSO2bGNRyAACgexF+2jE2K0XprkS1dUK7Q1+d9TU2KyWc1QIAAJ1E+GlHjziHFk796oKrpweg5tsLp2Yz3w8AAFGC8NMBk4ana8WMi+V2+XdtuV2JWjHjYub5AQAgijDJYQdNGp6u72a7meEZAIAoR/gJQI84h3KH9O/uagAAgC6g2wsAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANhKwOFn+/btmjp1qjIyMuRwOLR27Vq/7Q6Ho9Vl2bJlVpnBgwe32L5kyRK/x9mzZ4/GjRunxMREZWZmaunSpZ17hQAAAKcIOPwcPXpUo0aN0vLly1vdXlNT47c88cQTcjgcuvrqq/3K3XfffX7lbrvtNmubz+fTxIkTNWjQIO3atUvLli3TokWL9NhjjwVaXQAAAD/xgd5h8uTJmjx5cpvb3W633+0XXnhBV1xxhc4991y/9X379m1Rttnq1at1/PhxPfHEE3I6nbrwwgtVVlamBx54QHPmzAm0ygAAAJaQjvmpra3VSy+9pFmzZrXYtmTJEvXv318XXXSRli1bppMnT1rbSktLNX78eDmdTmtdfn6+Kisr9fnnn7f6XA0NDfL5fH4LAADA6QJu+QnEn//8Z/Xt21c/+tGP/Nbffvvtuvjii5WSkqI33nhDRUVFqqmp0QMPPCBJ8ng8ysrK8rtPWlqata1fv34tnqu4uFiLFy8O0SsBAACxIqTh54knntD06dOVmJjot37BggXW/0eOHCmn06lbbrlFxcXFSkhI6NRzFRUV+T2uz+dTZmZm5yoOAABiVsjCz9/+9jdVVlbqmWeeabdsTk6OTp48qQ8//FBDhw6V2+1WbW2tX5nm222NE0pISOh0cAIAAPYRsjE/jz/+uMaMGaNRo0a1W7asrExxcXFKTU2VJOXm5mr79u06ceKEVaakpERDhw5ttcsLAACgowIOP0eOHFFZWZnKysokSVVVVSorK1N1dbVVxufz6dlnn9XNN9/c4v6lpaX64x//qHfffVcffPCBVq9erfnz52vGjBlWsLnuuuvkdDo1a9Ys7du3T88884wefPBBv24tAACAzgi42+vtt9/WFVdcYd1uDiQzZ87UqlWrJElr1qyRMUbTpk1rcf+EhAStWbNGixYtUkNDg7KysjR//ny/YONyufTqq6+qoKBAY8aM0YABA3TvvfdymjsAAOgyhzHGdHclQsHn88nlcsnr9SopKam7qwMAADogHMdvru0FAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABshfADAABsJeDws337dk2dOlUZGRlyOBxau3at3/Ybb7xRDofDb5k0aZJfmUOHDmn69OlKSkpScnKyZs2apSNHjviV2bNnj8aNG6fExERlZmZq6dKlgb86AACA0wQcfo4ePapRo0Zp+fLlbZaZNGmSampqrOXpp5/22z59+nTt27dPJSUlWr9+vbZv3645c+ZY230+nyZOnKhBgwZp165dWrZsmRYtWqTHHnss0OoCAAD4iQ/0DpMnT9bkyZPPWCYhIUFut7vVbe+99542bNigt956S5dccokk6eGHH9aVV16p3//+98rIyNDq1at1/PhxPfHEE3I6nbrwwgtVVlamBx54wC8kAQAABCokY362bt2q1NRUDR06VHPnztVnn31mbSstLVVycrIVfCQpLy9PcXFx2rFjh1Vm/PjxcjqdVpn8/HxVVlbq888/b/U5Gxoa5PP5/BYAAIDTBT38TJo0Sf/7v/+rTZs26Xe/+522bdumyZMnq7GxUZLk8XiUmprqd5/4+HilpKTI4/FYZdLS0vzKNN9uLnO64uJiuVwua8nMzAz2SwMAADEg4G6v9lx77bXW/0eMGKGRI0dqyJAh2rp1qyZMmBDsp7MUFRVpwYIF1m2fz0cAAgAALYT8VPdzzz1XAwYM0P79+yVJbrdbdXV1fmVOnjypQ4cOWeOE3G63amtr/co0325rLFFCQoKSkpL8FgAAgNOFPPz885//1Geffab09HRJUm5ururr67Vr1y6rzObNm9XU1KScnByrzPbt23XixAmrTElJiYYOHap+/fqFusoAACCGBRx+jhw5orKyMpWVlUmSqqqqVFZWpurqah05ckR333233nzzTX344YfatGmTfvCDH+i8885Tfn6+JOmCCy7QpEmTNHv2bO3cuVOvv/665s2bp2uvvVYZGRmSpOuuu05Op1OzZs3Svn379Mwzz+jBBx/069YCAADoDIcxxgRyh61bt+qKK65osX7mzJlasWKFrrrqKr3zzjuqr69XRkaGJk6cqN/85jd+A5gPHTqkefPmad26dYqLi9PVV1+thx56SGeddZZVZs+ePSooKNBbb72lAQMG6LbbblNhYWGH6+nz+eRyueT1eukCAwAgSoTj+B1w+IkWhB8AAKJPOI7fXNsLAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYCuEHAADYSsDhZ/v27Zo6daoyMjLkcDi0du1aa9uJEydUWFioESNGqE+fPsrIyNANN9ygTz75xO8xBg8eLIfD4bcsWbLEr8yePXs0btw4JSYmKjMzU0uXLu3cKwQAADhFwOHn6NGjGjVqlJYvX95i2xdffKHdu3fr17/+tXbv3q3nnntOlZWV+v73v9+i7H333aeamhprue2226xtPp9PEydO1KBBg7Rr1y4tW7ZMixYt0mOPPRZodQEAAPzEB3qHyZMna/Lkya1uc7lcKikp8Vv3X//1Xxo7dqyqq6s1cOBAa33fvn3ldrtbfZzVq1fr+PHjeuKJJ+R0OnXhhReqrKxMDzzwgObMmdPqfRoaGtTQ0GDd9vl8gb40AABgAyEf8+P1euVwOJScnOy3fsmSJerfv78uuugiLVu2TCdPnrS2lZaWavz48XI6nda6/Px8VVZW6vPPP2/1eYqLi+VyuawlMzMzJK8HAABEt5CGn2PHjqmwsFDTpk1TUlKStf7222/XmjVrtGXLFt1yyy26//779Ytf/MLa7vF4lJaW5vdYzbc9Hk+rz1VUVCSv12stBw8eDMErAgAA0S7gbq+OOnHihH7605/KGKMVK1b4bVuwYIH1/5EjR8rpdOqWW25RcXGxEhISOvV8CQkJnb4vAACwj5C0/DQHn48++kglJSV+rT6tycnJ0cmTJ/Xhhx9Kktxut2pra/3KNN9ua5wQAABARwQ9/DQHn/fff18bN25U//79271PWVmZ4uLilJqaKknKzc3V9u3bdeLECatMSUmJhg4dqn79+gW7ygAAwEYC7vY6cuSI9u/fb92uqqpSWVmZUlJSlJ6erh//+MfavXu31q9fr8bGRmuMTkpKipxOp0pLS7Vjxw5dccUV6tu3r0pLSzV//nzNmDHDCjbXXXedFi9erFmzZqmwsFDl5eV68MEH9Yc//CFILxsAANiVwxhjArnD1q1bdcUVV7RYP3PmTC1atEhZWVmt3m/Lli369re/rd27d+vnP/+5/v73v6uhoUFZWVm6/vrrtWDBAr8xO3v27FFBQYHeeustDRgwQLfddpsKCws7XE+fzyeXyyWv19tutxsAAIgM4Th+Bxx+ogXhBwCA6BOO4zfX9gIAALZC+AEAALZC+AEAALZC+AEAALZC+AEAALZC+AEAALYSsmt7AQBa19hktLPqkOoOH1Nq30SNzUpRjzhHd1cLsA3CDwCE0YbyGi1eV6Ea7zFrXborUQunZmvS8PRurBlgH3R7AUCYbCiv0dwnd/sFH0nyeI9p7pO7taG8pptqBtgL4QcAwqCxyWjxugq1NqV+87rF6yrU2BSTk+4DEYXwAwBhsLPqUIsWn1MZSTXeY9pZdSh8lQJsivADAGFQd7jt4NOZcgA6j/ADAGGQ2jcxqOUAdB7hBwDCYGxWitJdiWrrhHaHvjrra2xWSjirBdgS4Qch1dhkVHrgM71Q9rFKD3zGYE7YVo84hxZOzZakFgGo+fbCqdnM9wOEAfP8IGSYzwTwN2l4ulbMuLjF98LN9wIIK4cxJib/FPf5fHK5XPJ6vUpKSuru6thO83wmp3+4mv+mXTHjYn7oYVvM8Ay0LRzHb1p+EHTtzWfi0FfzmXw3280PPmypR5xDuUP6d3c1ANtizA+CjvlMAACRjPCDoGM+EwBAJCP8IOiYzwQAEMkIPwg65jMBAEQywg+CjvlMAACRjPCDkGiez8Tt8u/acrsSOc0dANCtONUdITNpeLq+m+1mPhMAQEQh/CCkmM8EABBp6PYCAAC2QvgBAAC2QvgBAAC2QvgBAAC2QvgBAAC2QvgBAAC2QvgBAAC2QvgBAAC2EnD42b59u6ZOnaqMjAw5HA6tXbvWb7sxRvfee6/S09PVq1cv5eXl6f333/crc+jQIU2fPl1JSUlKTk7WrFmzdOTIEb8ye/bs0bhx45SYmKjMzEwtXbo08FcHAABwmoDDz9GjRzVq1CgtX7681e1Lly7VQw89pEcffVQ7duxQnz59lJ+fr2PHjlllpk+frn379qmkpETr16/X9u3bNWfOHGu7z+fTxIkTNWjQIO3atUvLli3TokWL9Nhjj3XiJQIAAJzCdIEk8/zzz1u3m5qajNvtNsuWLbPW1dfXm4SEBPP0008bY4ypqKgwksxbb71llXnllVeMw+EwH3/8sTHGmEceecT069fPNDQ0WGUKCwvN0KFDO1w3r9drJBmv19vZlwcAAMIsHMfvoI75qaqqksfjUV5enrXO5XIpJydHpaWlkqTS0lIlJyfrkksuscrk5eUpLi5OO3bssMqMHz9eTqfTKpOfn6/Kykp9/vnnrT53Q0ODfD6f3wIAAHC6oIYfj8cjSUpLS/Nbn5aWZm3zeDxKTU312x4fH6+UlBS/Mq09xqnPcbri4mK5XC5ryczM7PoLAoAI1NhkVHrgM71Q9rFKD3ymxibT3VUCokrMXNW9qKhICxYssG77fD4CEICYs6G8RovXVajG++9xlOmuRC2cmq1Jw9O7sWZA9Ahqy4/b7ZYk1dbW+q2vra21trndbtXV1fltP3nypA4dOuRXprXHOPU5TpeQkKCkpCS/BQBiyYbyGs19crdf8JEkj/eY5j65WxvKa7qpZkB0CWr4ycrKktvt1qZNm6x1Pp9PO3bsUG5uriQpNzdX9fX12rVrl1Vm8+bNampqUk5OjlVm+/btOnHihFWmpKREQ4cOVb9+/YJZZQCICo1NRovXVai1Dq7mdYvXVdAFBnRAwOHnyJEjKisrU1lZmaSvBjmXlZWpurpaDodDd955p/7zP/9TL774ovbu3asbbrhBGRkZuuqqqyRJF1xwgSZNmqTZs2dr586dev311zVv3jxde+21ysjIkCRdd911cjqdmjVrlvbt26dnnnlGDz74oF+3FgDYyc6qQy1afE5lJNV4j2ln1aHwVQqIUgGP+Xn77bd1xRVXWLebA8nMmTO1atUq/eIXv9DRo0c1Z84c1dfX6/LLL9eGDRuUmJho3Wf16tWaN2+eJkyYoLi4OF199dV66KGHrO0ul0uvvvqqCgoKNGbMGA0YMED33nuv31xAgN01NhntrDqkusPHlNo3UWOzUtQjztHd1UKI1B1uO/h0phxgZw5jTEy2kfp8PrlcLnm9Xsb/IOYw6NV+Sg98pml/erPdck/PvlS5Q/qHoUZAaITj+M21vYAow6BXexqblaJ0V6Laattz6KsAPDYrJZzVAqIS4QeIIgx6ta8ecQ4tnJotSS0CUPPthVOz6foEOoDwA0QRBr12r7YmFwzXpIOThqdrxYyL5XYl+q13uxK1YsbFdHkCHRQzkxwCdsCg1+7T1jir749K14vv1oRt/NWk4en6brabwe4ICrueOEH4AaJIat/E9gsFUA4d0zzO6vT2nBrvMf339qoW5ZvHX4WqNaZHnINBzegyO584QbcXEEUY9Bp+Zxpn1RbGXyHS2f3ECcIPbCsaLw7JoNfO6cp73d44q7Yw/ir6ReNvREdw4gTdXrCpaG7ubR70enr93VFS/1Bpa+xCV9/rro6fiqTxV3Yd39EZ0fwb0Z5ATpyI1e5Vwg9sp63xG6EepxFMDHr1d6bByI9tr+rSe93V8VORMv4qlg/mwRYLvxFnwokTdHvBZmKpubd50OsPRn9NuUP62zr4tDZ2oXkwclff6/bGWbUlksZf2X18RyBi6TeiLZw4QfiBzTBPTmzpzGDkZh19r880zqotkTT+yg4H82Cyw28EJ04QfmAzkdrcG6sDK0Ots4ORT9WR97qtyQVT+vTU7HGDlR7Bkw5258E8Gj/XkfobEUycOMGYH9hMJDb3Mhaj84JxAOroez1peLqamqR7XijXoaPHJUmHjp7Q+j0e/XpKtvr1cUbk+KvuOphH6+c6En8jQqErJ07EwsB5wg9spbm51+M91mo3gENfffnD1dwb6wMrQ60rB6BA3+sN5TUqeKr196rgqa/eqx+M/lqn6xMq3XEwj+bPdaT9RoRSZ06ciNZQezq6vWArkdTcy1iMruvsYORmHX2vo/m9Cvf4jmjeV1Jk/UaEQyAnTsTSwHnCD2wnUi4OaYeBlaHWmcHIktS/jzOg97qj79Wq16sibnxLuA/msfC5jpTfiEgS7aH2dHR7wZYiYZ4cOwysDIe2xi60JaVPT5UWTZAzvuN/+3X0PfjNS+9Z/4+kroBwTowZK5/rSPiNiCSxNjEi4Qe21d0XhwznWIxYGKB4JqceqDZWePT46x+2KNP8au//4YiAgo/Uufcg0sa3hOtgHksDhrv7NyKSxEqobUb4CaNYPwAhMOEaWBkrAxTb03ygyh3SX9/ISglqK0d771VrjL56Dxevq9B3hqVp10efd/t3PxwHczsNGLaTWAq1kuQwxkRHB12AfD6fXC6XvF6vkpKSurs6tjkAITBtnRUjfXWQ6GqrQVuP33zYjZRWiVAI9h8bzftSUsCTKqb06alDR09Yt2P9u9/WvrLD564rIvkP5MYmo8t/t7ndUPta4Xe6XOdwHL8JP2Fg5wMQ2lf8coX+9LcqnTpOMM4hzR6XpaIrszv9uM0/Vm310wfzx8ouWvsjpjPs8N3nD77ARMP+CleoJfx0QaSEHw5AOJNQBuPSA59p2p/ebLfc07MvZVxDAE796/xfhxv8BjkHwg7f/UhuyYgk0fQHcjhCWjiO34z5CbFYGyGP4Gnv1NHm8SLfzXZ36oARawMUuyKYB+FTx800Nhn9z2tVAY0FamaH7z4DhtsX6t+BYIuVs+AIPyHGAQhtCXUwjrUBip0Vyr9Um+fQmfvkbjkU+Fggie++3UXjH8ixEGqZ5DDEYu0AFI0XKoxUoQ7GXLk5PDPStjUhXv8+zg7dP1q++wgN/kDuHrT8hFgsnfYZDQPyokmog/GZWiVicZr+04WzO6G1roAxg/rpW8u2xMR3H6ETa38gRwtafkIsVq4TE0vXdIkU4WiZsfM0/eG+zMLp10hyxsfFxHcfoUULbfcg/IRBOA5AoeyOirVrukSKcAXjScPT9Vrhd/T07Ev14LWj9fTsS/Va4XdiOvhIkdGdYOfwiY6JlT+Qow2nuodRqE77DHV3FKdMh1ZH3j9OGQ5cRz+3864YosvOOzuk+5T3D+1hWMG/Mc9PF0Ri+AmFcMwP8ULZx7pjTVm75R68drR+MPprXXouuzr94DhmUD/rcggf/usLPb2zWh5f8H4U7XAwbm9G2tPZ9UCDyGGH72VHMM8PzihcAzrtNiCvO36ATj11dEN5jb61bMsZx6t05aKZdvkLM9DT0CPtQqSwn1g4hTxaMOYnioVrQKedBuRtKK/R5b/brGl/elN3rCnTtD+9qct/t9ka0B3qU/3bGlh+us6OtYrmgeud2fdtjblpDePXAPug5SeKhWtAp11OmW6rC7E5GMwZn6UX360JWYvJmVryWhPo5GfdNZNsMFrSutJadepp6K/v/5f+a8v+NstG4oRyAIKPlp8oFs7uqFg/a6W9YGAk/ff2qpC2mLTXkteWjobbcJ/6LbXfktbRx+hqa1Vzd8L5aWd16DmZUA6IbUEPP4MHD5bD4WixFBQUSJK+/e1vt9h26623+j1GdXW1pkyZot69eys1NVV33323Tp48GeyqRr1wd0fF8inTnQ0ewewq6ewBt6PhNtynfgcjtHRlmoXWusnsNn4NQOuC3u311ltvqbGx0bpdXl6u7373u/rJT35irZs9e7buu+8+63bv3r2t/zc2NmrKlClyu9164403VFNToxtuuEE9e/bU/fffH+zqRrXu6I6K1QF5XTngB6urJNADbmszBJ+piymcB/5gdbF19rpHbXWT/XpKdszMuA6g84Iefs4++2y/20uWLNGQIUP0rW99y1rXu3dvud3uVu//6quvqqKiQhs3blRaWppGjx6t3/zmNyosLNSiRYvkdHbsejl20dwddfoPvTsGz94JpWAc8LvaYtLepVBO1Vq4bW9cTDgvtRKsizV2prXqTGO3Cp76auzWY9urYnr8GoAzC+mYn+PHj+vJJ5/UTTfdJIfj3z8mq1ev1oABAzR8+HAVFRXpiy++sLaVlpZqxIgRSktLs9bl5+fL5/Np3759bT5XQ0ODfD6f32IXsdwdFS7tdSF2RFcD1Jlmej3d6WOtOtLFFM6ZZIPVxRZoa1VHuslefLdGy6+7KGbHrwFoX0jP9lq7dq3q6+t14403Wuuuu+46DRo0SBkZGdqzZ48KCwtVWVmp5557TpLk8Xj8go8k67bH42nzuYqLi7V48eLgv4goEavdUeES6Jwwpwpmi0mbLXlJCZo2dqAGD+jTojsrkC6mcLUUBquLLdDWqo62OPXrk6DXCr/DhHKATYU0/Dz++OOaPHmyMjIyrHVz5syx/j9ixAilp6drwoQJOnDggIYMGdLp5yoqKtKCBQus2z6fT5mZmZ1+PNhPW8HgTELRVdLaFcLPdGAOtIsp0MfvjGB1sQU6rq2jLU4bKzzKHdKfPxgAmwpZt9dHH32kjRs36uabbz5juZycHEnS/v1fzb3hdrtVW1vrV6b5dlvjhCQpISFBSUlJfgsQqOYuxPl553eofKi6Sk6/QviZgklnupgCefzOCGYXWyDTLHS0xenx1z+M6AkdAYRWyFp+Vq5cqdTUVE2ZMuWM5crKyiRJ6elf/YDl5ubqt7/9rerq6pSamipJKikpUVJSkrKzs0NVXcDPmrcOnnF7cu+eWj7tYl0aguAQqEg9fTuYXWwdba0KZNB4KCZ0BBAdQhJ+mpqatHLlSs2cOVPx8f9+igMHDuipp57SlVdeqf79+2vPnj2aP3++xo8fr5EjR0qSJk6cqOzsbF1//fVaunSpPB6P7rnnHhUUFCghISEU1QX8dGTOn/ovTiguzhERB85wnsUVqGB2sXVkXFtzi9OtT+5u9/GYyRmwr5B0e23cuFHV1dW66aab/NY7nU5t3LhREydO1LBhw3TXXXfp6quv1rp166wyPXr00Pr169WjRw/l5uZqxowZuuGGG/zmBQJCqaPdSB7vlyG9zldHhfMsrs4IdRfb6SYNT9dNlw3uUFlmcgbsyWGMickr+Pl8PrlcLnm9Xsb/xKBQXnm99MBnmvanN9stl9LHqUNHj1u3u/vK6Ha5WntHdPQ9fHr2pbT8ABEmHMdvLmyKqBPqg3xHx42cGnykf8+p011zxZypiymUYTESRXJXIIDuR8sPokpbs/c2H8aDFTyan0fq3Jw/rxV+J2LChV1bhNp6D4P9WQEQXOE4fnNVd1hauxBkJOnKRS4D1dbp1Sl9ep7xfqG4MnpXBOPiotEqkFPku1ukf/eAWEO3FyRFR+tAsK4X1VGtdSN5fMc0/5mydu8bCQNpg3Vx0WjuMgvHhI5dFQ3fPSDWEH5wxgtBducYltMF63pRgTj99OrSA5916H7hnlOnNcEIi7FwYI7kS79Ey3cPiDV0e9lcOLuSuioSJvNr7wKoDn0VDiJhIG1Xw6Kdu8zCIZq+e0CsIfzYXCCtA90tEoJHpM+pc6quhEUOzKEXTd89INYQfmyuO7qSOitSgke0DKTtSljkwBx60fTdA2INY35sLhK6kgIRzOtFdbUekT6QNtArop+KA3PoRdt3D4glhB+bi8bJ4CIleETyQNpmnQ2LHJhDLxq/e0CsIPzYXFdaB7pTNASPSNGZsMiBOfS6+t2L5ikIgO7GDM+QFBunNCO4mCE5PDrz3eP7ilgWjuM34QcW/pLE6TjIdk6g36VAyofrEi9AdyH8dAHhBwgOQnFgQhkYG5uMLv/d5jbPxIvEa8sBgeLaXgC6XfP4qh+M/ppyh/TnoHoGoZ4YkikIgOAg/ABAEIRjYkimIACCg/ADAEEQjlYZpiAAgoPwAwBBEI5WmUi4xAsQCwg/ABAE4WiViZRLvADRjvADAEEQrlaZaLm2HBDJmOEZAIIgnLOlR8olXoBoxTw/ABBETAwJdE04jt+0/ABAENEqA0Q+wg8ABBkX3gUiGwOeAQCArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArQQ9/CxatEgOh8NvGTZsmLX92LFjKigoUP/+/XXWWWfp6quvVm1trd9jVFdXa8qUKerdu7dSU1N199136+TJk8GuKgAAsKGQXNvrwgsv1MaNG//9JPH/fpr58+frpZde0rPPPiuXy6V58+bpRz/6kV5//XVJUmNjo6ZMmSK326033nhDNTU1uuGGG9SzZ0/df//9oaguAACwkZCEn/j4eLnd7hbrvV6vHn/8cT311FP6zne+I0lauXKlLrjgAr355pu69NJL9eqrr6qiokIbN25UWlqaRo8erd/85jcqLCzUokWL5HQ6Q1FlAABgEyEZ8/P+++8rIyND5557rqZPn67q6mpJ0q5du3TixAnl5eVZZYcNG6aBAweqtLRUklRaWqoRI0YoLS3NKpOfny+fz6d9+/a1+ZwNDQ3y+Xx+CwAAwOmCHn5ycnK0atUqbdiwQStWrFBVVZXGjRunw4cPy+PxyOl0Kjk52e8+aWlp8ng8kiSPx+MXfJq3N29rS3FxsVwul7VkZmYG94UBAICYEPRur8mTJ1v/HzlypHJycjRo0CD95S9/Ua9evYL9dJaioiItWLDAuu3z+QhAAACghZCf6p6cnKyvf/3r2r9/v9xut44fP676+nq/MrW1tdYYIbfb3eLsr+bbrY0japaQkKCkpCS/BQAA4HQhDz9HjhzRgQMHlJ6erjFjxqhnz57atGmTtb2yslLV1dXKzc2VJOXm5mrv3r2qq6uzypSUlCgpKUnZ2dmhri4AAIhxQe/2+o//+A9NnTpVgwYN0ieffKKFCxeqR48emjZtmlwul2bNmqUFCxYoJSVFSUlJuu2225Sbm6tLL71UkjRx4kRlZ2fr+uuv19KlS+XxeHTPPfeooKBACQkJwa4uAACwmaCHn3/+85+aNm2aPvvsM5199tm6/PLL9eabb+rss8+WJP3hD39QXFycrr76ajU0NCg/P1+PPPKIdf8ePXpo/fr1mjt3rnJzc9WnTx/NnDlT9913X7CrCgAAbMhhjDHdXYlQ8Pl8crlc8nq9jP8BACBKhOP4zbW9AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArRB+AACArQQ9/BQXF+sb3/iG+vbtq9TUVF111VWqrKz0K/Ptb39bDofDb7n11lv9ylRXV2vKlCnq3bu3UlNTdffdd+vkyZPBri4AALCZ+GA/4LZt21RQUKBvfOMbOnnypH71q19p4sSJqqioUJ8+faxys2fP1n333Wfd7t27t/X/xsZGTZkyRW63W2+88YZqamp0ww03qGfPnrr//vuDXWUAAGAjDmOMCeUTfPrpp0pNTdW2bds0fvx4SV+1/IwePVp//OMfW73PK6+8ou9973v65JNPlJaWJkl69NFHVVhYqE8//VROp7Pd5/X5fHK5XPJ6vUpKSgra6wEAAKETjuN3yMf8eL1eSVJKSorf+tWrV2vAgAEaPny4ioqK9MUXX1jbSktLNWLECCv4SFJ+fr58Pp/27dvX6vM0NDTI5/P5LQAAAKcLerfXqZqamnTnnXfqsssu0/Dhw6311113nQYNGqSMjAzt2bNHhYWFqqys1HPPPSdJ8ng8fsFHknXb4/G0+lzFxcVavHhxiF4JAACIFSENPwUFBSovL9drr73mt37OnDnW/0eMGKH09HRNmDBBBw4c0JAhQzr1XEVFRVqwYIF12+fzKTMzs3MVBwAAMStk3V7z5s3T+vXrtWXLFp1zzjlnLJuTkyNJ2r9/vyTJ7XartrbWr0zzbbfb3epjJCQkKCkpyW8BAAA4XdDDjzFG8+bN0/PPP6/NmzcrKyur3fuUlZVJktLT0yVJubm52rt3r+rq6qwyJSUlSkpKUnZ2drCrDAAAbCTo3V4FBQV66qmn9MILL6hv377WGB2Xy6VevXrpwIEDeuqpp3TllVeqf//+2rNnj+bPn6/x48dr5MiRkqSJEycqOztb119/vZYuXSqPx6N77rlHBQUFSkhICHaVAQCAjQT9VHeHw9Hq+pUrV+rGG2/UwYMHNWPGDJWXl+vo0aPKzMzUD3/4Q91zzz1+XVUfffSR5s6dq61bt6pPnz6aOXOmlixZovj4juU1TnUHACD6hOP4HfJ5froL4QcAgOgTE/P8AAAARBLCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsBXCDwAAsJX47q4AAJyuscloZ9Uh1R0+ptS+iRqblaIecY7urhaAGEH4ARBRNpTXaPG6CtV4j1nr0l2JWjg1W5OGp3djzQDECrq9AESMDeU1mvvkbr/gI0ke7zHNfXK3NpTXdFPNAMQSwg+AiNDYZLR4XYVMK9ua1y1eV6HGptZKAEDHEX4ARISdVYdatPicykiq8R7TzqpD4asUgJhE+AEQEeoOtx18OlMOANpC+AEQEVL7Jga1HAC0hfADICKMzUpRuitRbZ3Q7tBXZ32NzUoJZ7UAxCDCD4CI0CPOoYVTsyWpRQBqvr1wajbz/QDoMsIPgIgxaXi6Vsy4WG6Xf9eW25WoFTMuZp4fAEHBJIcAIsqk4en6brabGZ4BhAzhB0DE6RHnUO6Q/t1dDQAxim4vAABgKxEdfpYvX67BgwcrMTFROTk52rlzZ3dXCQAARLmIDT/PPPOMFixYoIULF2r37t0aNWqU8vPzVVdX191VAwAAUSxiw88DDzyg2bNn62c/+5mys7P16KOPqnfv3nriiSe6u2oAACCKRWT4OX78uHbt2qW8vDxrXVxcnPLy8lRaWtrqfRoaGuTz+fwWAACA00Vk+PnXv/6lxsZGpaWl+a1PS0uTx+Np9T7FxcVyuVzWkpmZGY6qAgCAKBOR4aczioqK5PV6reXgwYPdXSUAABCBInKenwEDBqhHjx6qra31W19bWyu3293qfRISEpSQkBCO6gEAgCgWkS0/TqdTY8aM0aZNm6x1TU1N2rRpk3Jzc7uxZgAAINpFZMuPJC1YsEAzZ87UJZdcorFjx+qPf/yjjh49qp/97Gcdur8xRpIY+AwAQBRpPm43H8dDIWLDzzXXXKNPP/1U9957rzwej0aPHq0NGza0GATdlsOHD0sSA58BAIhChw8flsvlCsljO0woo1U3ampq0ieffKK+ffvK4ej8BRF9Pp8yMzN18OBBJSUlBbGGsY391jnst8CxzzqH/dY57LfABbrPjDE6fPiwMjIyFBcXmtE5Edvy01VxcXE655xzgvZ4SUlJfNA7gf3WOey3wLHPOof91jnst8AFss9C1eLTLCIHPAMAAIQK4QcAANgK4acdCQkJWrhwIXMIBYj91jnst8CxzzqH/dY57LfAReI+i9kBzwAAAK2h5QcAANgK4QcAANgK4QcAANgK4QcAANgK4QcAANhKTISf7du3a+rUqcrIyJDD4dDatWv9thtjdO+99yo9PV29evVSXl6e3n//fb8yhw4d0vTp05WUlKTk5GTNmjVLR44c8SuzZ88ejRs3TomJicrMzNTSpUtb1OXZZ5/VsGHDlJiYqBEjRujll18OuC7hUFxcrG984xvq27evUlNTddVVV6mystKvzLFjx1RQUKD+/fvrrLPO0tVXX63a2lq/MtXV1ZoyZYp69+6t1NRU3X333Tp58qRfma1bt+riiy9WQkKCzjvvPK1atapFfZYvX67BgwcrMTFROTk52rlzZ8B1CbUVK1Zo5MiR1iylubm5euWVVwKqo532V1uWLFkih8OhO++801rHvmtp0aJFcjgcfsuwYcMCqqfd9pkkffzxx5oxY4b69++vXr16acSIEXr77bet7RwPWho8eHCLz5rD4VBBQYGkGP2smRjw8ssvm//3//6fee6554wk8/zzz/ttX7JkiXG5XGbt2rXm3XffNd///vdNVlaW+fLLL60ykyZNMqNGjTJvvvmm+dvf/mbOO+88M23aNGu71+s1aWlpZvr06aa8vNw8/fTTplevXua///u/rTKvv/666dGjh1m6dKmpqKgw99xzj+nZs6fZu3dvQHUJh/z8fLNy5UpTXl5uysrKzJVXXmkGDhxojhw5YpW59dZbTWZmptm0aZN5++23zaWXXmq++c1vWttPnjxphg8fbvLy8sw777xjXn75ZTNgwABTVFRklfnggw9M7969zYIFC0xFRYV5+OGHTY8ePcyGDRusMmvWrDFOp9M88cQTZt++fWb27NkmOTnZ1NbWdrgu4fDiiy+al156yfzjH/8wlZWV5le/+pXp2bOnKS8v71Ad7ba/WrNz504zePBgM3LkSHPHHXdY69l3LS1cuNBceOGFpqamxlo+/fTTDtfTjvvs0KFDZtCgQebGG280O3bsMB988IH561//avbv32+V4XjQUl1dnd/nrKSkxEgyW7ZsMcbE5mctJsLPqU4PP01NTcbtdptly5ZZ6+rr601CQoJ5+umnjTHGVFRUGEnmrbfessq88sorxuFwmI8//tgYY8wjjzxi+vXrZxoaGqwyhYWFZujQodbtn/70p2bKlCl+9cnJyTG33HJLh+vSXerq6owks23bNqtePXv2NM8++6xV5r333jOSTGlpqTHmq9AZFxdnPB6PVWbFihUmKSnJ2k+/+MUvzIUXXuj3XNdcc43Jz8+3bo8dO9YUFBRYtxsbG01GRoYpLi7ucF26S79+/cz//M//sL864PDhw+b88883JSUl5lvf+pYVfth3rVu4cKEZNWpUq9vYZ60rLCw0l19+eZvbOR50zB133GGGDBlimpqaYvazFhPdXmdSVVUlj8ejvLw8a53L5VJOTo5KS0slSaWlpUpOTtYll1xilcnLy1NcXJx27NhhlRk/frycTqdVJj8/X5WVlfr888+tMqc+T3OZ5ufpSF26i9frlSSlpKRIknbt2qUTJ0741XXYsGEaOHCg334bMWKE0tLSrDL5+fny+Xzat2+fVeZM++T48ePatWuXX5m4uDjl5eVZZTpSl3BrbGzUmjVrdPToUeXm5rK/OqCgoEBTpkxp8frYd217//33lZGRoXPPPVfTp09XdXV1h+tpx3324osv6pJLLtFPfvITpaam6qKLLtKf/vQnazvHg/YdP35cTz75pG666SY5HI6Y/azFfPjxeDyS5PemNN9u3ubxeJSamuq3PT4+XikpKX5lWnuMU5+jrTKnbm+vLt2hqalJd955py677DINHz5c0ld1dTqdSk5O9it7+uvp7D7x+Xz68ssv9a9//UuNjY3t7rf26hIue/fu1VlnnaWEhATdeuutev7555Wdnc3+aseaNWu0e/duFRcXt9jGvmtdTk6OVq1apQ0bNmjFihWqqqrSuHHjdPjwYfZZGz744AOtWLFC559/vv76179q7ty5uv322/XnP//Zqmtz3dqqq92PB2vXrlV9fb1uvPFGSbH7/YwPqDRiUkFBgcrLy/Xaa691d1Ui3tChQ1VWViav16v/+7//08yZM7Vt27burlZEO3jwoO644w6VlJQoMTGxu6sTNSZPnmz9f+TIkcrJydGgQYP0l7/8Rb169erGmkWupqYmXXLJJbr//vslSRdddJHKy8v16KOPaubMmd1cu+jw+OOPa/LkycrIyOjuqoRUzLf8uN1uSWoxGry2ttba5na7VVdX57f95MmTOnTokF+Z1h7j1Odoq8yp29urS7jNmzdP69ev15YtW3TOOedY691ut44fP676+nq/8qe/ns7uk6SkJPXq1UsDBgxQjx492t1v7dUlXJxOp8477zyNGTNGxcXFGjVqlB588EH21xns2rVLdXV1uvjiixUfH6/4+Hht27ZNDz30kOLj45WWlsa+64Dk5GR9/etf1/79+/m8tSE9PV3Z2dl+6y644AKru5DjwZl99NFH2rhxo26++WZrXax+1mI+/GRlZcntdmvTpk3WOp/Ppx07dig3N1eSlJubq/r6eu3atcsqs3nzZjU1NSknJ8cqs337dp04ccIqU1JSoqFDh6pfv35WmVOfp7lM8/N0pC7hYozRvHnz9Pzzz2vz5s3Kysry2z5mzBj17NnTr66VlZWqrq7222979+71+6EoKSlRUlKS9QPU3j5xOp0aM2aMX5mmpiZt2rTJKtORunSXpqYmNTQ0sL/OYMKECdq7d6/Kysqs5ZJLLtH06dOt/7Pv2nfkyBEdOHBA6enpfN7acNlll7WYsuMf//iHBg0aJInjQXtWrlyp1NRUTZkyxVoXs5+1gIZHR6jDhw+bd955x7zzzjtGknnggQfMO++8Yz766CNjzFenEyYnJ5sXXnjB7Nmzx/zgBz9o9dTGiy66yOzYscO89tpr5vzzz/c7tbG+vt6kpaWZ66+/3pSXl5s1a9aY3r17tzi1MT4+3vz+97837733nlm4cGGrpza2V5dwmDt3rnG5XGbr1q1+pzh+8cUXVplbb73VDBw40GzevNm8/fbbJjc31+Tm5lrbm09vnDhxoikrKzMbNmwwZ599dqunN959993mvffeM8uXL2/19MaEhASzatUqU1FRYebMmWOSk5P9zhxory7h8Mtf/tJs27bNVFVVmT179phf/vKXxuFwmFdffbVDdbTb/jqTU8/2MoZ915q77rrLbN261VRVVZnXX3/d5OXlmQEDBpi6uroO1dOO+2znzp0mPj7e/Pa3vzXvv/++Wb16tendu7d58sknrTIcD1rX2NhoBg4caAoLC1tsi8XPWkyEny1bthhJLZaZM2caY746pfDXv/61SUtLMwkJCWbChAmmsrLS7zE+++wzM23aNHPWWWeZpKQk87Of/cwcPnzYr8y7775rLr/8cpOQkGC+9rWvmSVLlrSoy1/+8hfz9a9/3TidTnPhhReal156yW97R+oSDq3tL0lm5cqVVpkvv/zS/PznPzf9+vUzvXv3Nj/84Q9NTU2N3+N8+OGHZvLkyaZXr15mwIAB5q677jInTpzwK7NlyxYzevRo43Q6zbnnnuv3HM0efvhhM3DgQON0Os3YsWPNm2++6be9I3UJtZtuuskMGjTIOJ1Oc/bZZ5sJEyZYwaejdbTT/jqT08MP+66la665xqSnpxun02m+9rWvmWuuucZvvhr2WevWrVtnhg8fbhISEsywYcPMY4895red40Hr/vrXvxpJrT5/LH7WHMYYE1hbEQAAQPSK+TE/AAAApyL8AAAAWyH8AAAAWyH8AAAAWyH8AAAAWyH8AAAAWyH8AAAAWyH8AAAAWyH8AAAAWyH8AAAAWyH8AAAAW/n/Qo5PY4Tq024AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lengths = [HEADER_RE.search(text).end() for text in raw_texts]\n",
    "text_lengths = [len(text) for text in raw_texts]\n",
    "plt.scatter(text_lengths, lengths);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "472b3292-23c8-4a82-b082-3ce228d02ede",
   "metadata": {},
   "source": [
    "The shortest one just doesn't have much information before the text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f8e958ec-bdec-4df8-90a0-be0ef4fc8bfe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "﻿\n",
      "*** START OF THE PROJECT GUTENBERG EBOOK DIVOTS ***\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "                                DIVOTS\n",
      "\n",
      "                            P. G. WODEHOUSE\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "                          By P. G. WODE\n"
     ]
    }
   ],
   "source": [
    "min_idx = lengths.index(min(lengths))\n",
    "\n",
    "print(raw_texts[min_idx][:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1ceb7ca-3256-4c85-892d-84f8c81bfa59",
   "metadata": {},
   "source": [
    "We can build a much simpler expression for the footer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ae72ec56-02a8-4e30-b2cf-d4acdde2a709",
   "metadata": {},
   "outputs": [],
   "source": [
    "FOOTER_RE = \"(?:End of .*Project Gutenberg.*|\\\\*\\\\*\\\\*.*END OF.*PROJECT GUTENBERG)\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf018916-0cb7-481d-958f-14da968538db",
   "metadata": {},
   "source": [
    "But we can also extend it to remove transcriber's notes and THE END."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "baf2d2a7-f4f2-4cc9-ba88-cad8018bdc71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "THE END\n",
      "\n",
      "\n",
      "\n",
      "Transcriber’s Note: While I left several variant spellings such as\n",
      "vodevil and bethrothed, I did correct the following:\n",
      "\n",
      "Fixed: course/coarse in\n",
      "Yet somehow this course, rough person in front of him never seemed to\n",
      "allow him a word\n",
      "\n",
      "Fixed: awfuly/awfully in:\n",
      "He’s awfuly good to girls who’ve worked in shows for him before.\n",
      "\n",
      "Fixed: Pullfan/Pullman\n",
      "Those Pullfan porters on parade!”\n",
      "\n",
      "Fixed: a large typo in the print edition, which originally read:\n",
      "“Yes. I’ve got the most damned attack of indigestion.” Derek should\n",
      "recline in the arm-chair which he had vacated; dinner!”\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "*** END OF THE PROJECT GUTENBERG\n"
     ]
    }
   ],
   "source": [
    "FOOTER_RE = re.compile(\"\"\"\n",
    "(\n",
    "[\\r\\n]+                                 # Newlines\n",
    "(?:\n",
    "  \\s*\n",
    "  (?:\n",
    "      THE\\ END\n",
    "    | [+|]?-{5,}[|+]?\n",
    "   )\n",
    "   \\s*\n",
    "  [\\r\\n]+                               # Newlines\n",
    ")?                                  # Optional\n",
    "(?:                                   # Additional data before footer\n",
    "  (?i:                                # Lines starting with:\n",
    "    \\s*[\\\\[|]?transcriber[’']s\\ note       \n",
    "   )\n",
    "  (?:[^\\r\\n]*)[\\r]?[\\n]{,1}           # Until the end of the line\n",
    "  ((?:[\\r]?[\\n]){1,2}[^\\r\\n]+)*       # Until the end of the paragraph\n",
    "                                      # (allowing 1 space)\n",
    "  [\\r\\n]+                             # And all newlines beneath it  \n",
    ")*\n",
    "(?:\n",
    "  End\\ of\\ .*Project\\ Gutenberg.*\n",
    "  |\\\\*\\\\*\\\\*.*END\\ OF.*PROJECT\\ GUTENBERG\n",
    ")\n",
    ")\n",
    "\"\"\",\n",
    "flags=re.MULTILINE | re.VERBOSE)\n",
    "\n",
    "print(FOOTER_RE.search(raw_texts[12]).group())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "235f1e2d-02e9-4de8-a036-063e3aad30a7",
   "metadata": {},
   "source": [
    "The footers all tend to be around 20000 characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a1419335-8e12-4b26-a8ee-22ed835eb383",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAGdCAYAAADpBYyuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9i0lEQVR4nO3dfXgU9b3//9cG2A0om3Bj7iTcCBUMBBCQkCPSekwJNqVNa1tEVFBE4YSWG49GjjeAPcdQOFpti1BbFc9Xq8L5CVXAcGK4qxJBAwECkiqEm5ZssGB2ASFA8vn94ZUpO0kgm+wmJHk+rmsusjPvnfnMZybMK7Mzsw5jjBEAAAAsYU3dAAAAgCsNAQkAAMCGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAAJu2Td2AplRZWamjR4+qY8eOcjgcTd0cAABQB8YYnTx5UnFxcQoLC825nlYdkI4ePar4+PimbgYAAKiHI0eOqFu3biGZd6sOSB07dpT0TQe73e4mbg0AAKgLn8+n+Ph46zgeCq06IFV9rOZ2uwlIAAA0M6G8PIaLtAEAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2LTqB0UCQEtWUWm0rfiEjp08q6iO4Rreq7PahPG9k0BdEJAAoAXKLizR/Pf2qsR71hoXGxGuuWMTNGZAbBO2DGge+IgNAFqY7MISTXt9u184kiSP96ymvb5d2YUlTdQyoPkgIAFAC1JRaTT/vb0yNUyrGjf/vb2qqKypAkAVAhIASd8cWPP2H9efC/6uvP3HOYA2U9uKT1Q7c3QxI6nEe1bbik80XqOAZohrkABwvUoLcuxk7eGoPnVAa8UZJKCV43qVliWqY3hQ64DWioAEtGJcr9LyDO/VWbER4artZn6Hvjk7OLxX58ZsFtDsEJCAVozrVVqeNmEOzR2bIEnVQlLV67ljE3geEnAZBCSgFeN6lZZpzIBYLbl7iGIi/D9Gi4kI15K7h3BdGVAHXKQNtGJcr9JyjRkQq+8mxPAkbaCeCEhAK1Z1vYrHe7bG65Ac+uasA9erNE9twhxK7t2lqZsBNEt8xAa0YlyvAgA1IyABrRzXqwBAdXzEBoDrVQDAhoAEQBLXqwDAxQL6iC0rK0s33XSTOnbsqKioKKWnp6uoqMiv5uzZs8rIyFCXLl109dVX64477lBpaalfzeHDh5WWlqYOHTooKipKjzzyiC5cuOBXs3HjRg0ZMkQul0t9+vTRsmXLqrVn8eLF6tmzp8LDw5WUlKRt27YFsjoAAAA1Ciggbdq0SRkZGfr444+Vk5Oj8+fPa/To0Tp9+rRVM2vWLL333ntasWKFNm3apKNHj+rHP/6xNb2iokJpaWk6d+6ctmzZotdee03Lli3TU089ZdUUFxcrLS1Nt956qwoKCjRz5kw98MADWrdunVXz9ttva/bs2Zo7d662b9+uQYMGKTU1VceOHWtIfwAAAEimAY4dO2YkmU2bNhljjCkrKzPt2rUzK1assGo+++wzI8nk5eUZY4xZu3atCQsLMx6Px6pZsmSJcbvdpry83BhjzKOPPmr69+/vt6xx48aZ1NRU6/Xw4cNNRkaG9bqiosLExcWZrKysOrff6/UaScbr9Qaw1gAAoCk1xvG7QXexeb1eSVLnzt88IyU/P1/nz59XSkqKVdOvXz91795deXl5kqS8vDwlJiYqOjraqklNTZXP59OePXusmovnUVVTNY9z584pPz/fryYsLEwpKSlWDQAAQH3V+yLtyspKzZw5UzfffLMGDBggSfJ4PHI6nYqMjPSrjY6OlsfjsWouDkdV06umXarG5/PpzJkz+uqrr1RRUVFjzb59+2ptc3l5ucrLy63XPp8vgDUGAACtRb3PIGVkZKiwsFBvvfVWMNsTUllZWYqIiLCG+Pj4pm4SAAC4AtUrIE2fPl2rV6/Whg0b1K1bN2t8TEyMzp07p7KyMr/60tJSxcTEWDX2u9qqXl+uxu12q3379uratavatGlTY03VPGoyZ84ceb1eazhy5EhgKw4AAFqFgAKSMUbTp0/XypUrtX79evXq1ctv+tChQ9WuXTvl5uZa44qKinT48GElJydLkpKTk7V7926/u81ycnLkdruVkJBg1Vw8j6qaqnk4nU4NHTrUr6ayslK5ublWTU1cLpfcbrffAAAAgqui0ihv/3H9ueDvytt/XBWVNX3b45UtoGuQMjIy9Kc//Ul//vOf1bFjR+uaoYiICLVv314RERGaPHmyZs+erc6dO8vtduvnP/+5kpOTNWLECEnS6NGjlZCQoHvuuUcLFy6Ux+PRE088oYyMDLlcLknS1KlT9bvf/U6PPvqo7r//fq1fv17Lly/XmjVrrLbMnj1bEydO1LBhwzR8+HA9//zzOn36tO67775g9Q0AAAhQdmGJ5r+3VyXes9a42IhwzR2b0Ky+ushhjKlzrHM4av7agVdffVWTJk2S9M2DIh9++GG9+eabKi8vV2pqql588UW/j74OHTqkadOmaePGjbrqqqs0ceJELViwQG3b/jOvbdy4UbNmzdLevXvVrVs3Pfnkk9Yyqvzud7/TokWL5PF4NHjwYP3mN79RUlJSnVfe5/MpIiJCXq+Xs0kAADRQdmGJpr2+XfZgUZUegvX9jo1x/A4oILU0BCQAAIKjotJo5K/W+505uphD33wJ9oeZ/9rg73lsjON3g56DBABouJZwvQawrfhEreFIkoykEu9ZbSs+0XiNagC+rDbIKioN34iOoGO/Cr2m6uOWcr0GcOxk7eGoPnVNjYAURA39j645HASbQxuvJMHor9r2qyfTEtTpKifbIgjW7irRE38u1InT56xxjRFSartew+M9q2mvbw/a9RpAY4jqGB7UuqbGNUhB+gyzoRemNcZfkQ09WNeljVdSgApVW+zzHdqjk/IPfVXtdc5ej1YVHG3QQbe2/aomMW6Xxg/vrp5dr6p1fevaJ425HYO5rEDnVVFpNOOtHVq9q6TWmvtv7qnvJsQEvQ8CvV7jSvrdAmpStU97vGdr/D+ruV2DREAKQgc39MK0xrjqv6EBrC5tlFSvZYTiP/5QBc6a5hvmkC6+ZMT++mKBbNPL7VeXY1/fuvZJffquvtswmNsp0AB/8B+n9fKHB+Q7W1Gn+Qf7D5aPvviHJvxx62Xrpt/aW+3atNGb2w7L4+NjOFzZqo4VkvyOF9zF1swEq4Pz9h/X+D98fNm6N6eMUHLvLn7jGuOq/4YGsLq0MbJDO3319fkap11qGaEIMqEKnIGczbmUum7Tuu5Xl1qO9M/wWpc+qU/f1XcbBnM71TfAByLYf7A89v/tVtmZ6r8zTdEeIJga4xMR7mJrJhpyYVqor/qvqDSa/97eGg/qVePmv7f3knfN1KWNNYWjyy2j6qBmn3fV9RfZhbV/7FGbYKxvoPMNVF23aUMvZLx4fee9u+eyfXLuQmXAfVffbRjM7VSXeT32zu4a2xmIhuw/F6vqs4aEo2C2Bw3DHYjVjRkQqw8z/1VvThmhF+4crDenjNCHmf/a7II8ASkIGnJhWqiv+g9GAAvGgdq+jFAFmVAFzsvNtz4u16/BuJCxan09vvLL1vy/vIMB9V1DtmEwt1Nd5lX29flGDbe1CWbQDkZ70DDZhSUa+av1Gv+HjzXjrQKN/8PHGvmr9fX6466laRPmUHLvLvrh4GuV3LtLs7xejoAUBMN7dVZsRLhq2/wOfXN6cXivztWmhfqq/2AEsGDdcXDxMkIVZEIVOENxW+rl+vVy+1WwHTrxdZ3qqvqiIdswmNupKW4ZDtUfLPXVXG6bbklCcQYcVxYCUhC0CXNo7thvvmjXfjCrej13bEKNCboh4aoughHAgnWgvngZoQoywQycF586/8fJ2s/ABKqu2/RS+1Uo9OjcoU51VX3XkG0YzO3UFLcMX2qZ5y5U6uW/HNBTfy7Uy385oHMXKq196f0QHTRrak99Pvpp7I+LmuvHU6E6A36laq7bqaF4DlKQjBkQqyV3D6l2YVrMZS5MqzoITnt9uxyq+ar/2sJVXVSFm8vddnmpg3Vd2hjRoZ28tXyMUdMyQnXmLBjrK9XtbrX6CHSbjhkQqwdH9dIf/lKs+txOUbW+xhiV+sov2Sf3JPfUHz8srnPfNWQbBms71WVewXS5dmWt3as//KXYbz/5z7WfqX27Nvr6XN3ulgtGe+pzkWxjP7CyOT8gM5Czp/Ybc5qb5rydGoozSEFU3wvTqsJVTIT/gSQmIrzBd6g05OxWIG1c8OPEgJYRqjNnwVjf2k6dB+OPpkC3aXZhiV7aXFyvZVeF2TtvildaYmytQUT6Z5/ceVN8neqkhm3DYO2XdZ1XZId2DT4Ld7l2Za3dq9/XsK2MUUDhKMxRtzOGtbWnPh/9NPbHRc3946mmfGJ0Y57Nae7bqaG4zf8K+rLaUD4ILlh/BVyqjYEuI5TPy6jv+tbl2UOXe+6R/XXnq9rpR4OvVUqADxusy+MVOl/l1BNpN+jwiTPVnpMT2aGdpG8uUK6tbVV9Il36Fvja+q6h27CxnoMkqcZ2BuJS7Tp3oVL9nny/QSG6qs8eHNVLL20ulnTpttbUnvo8NqQxv2C0KZYXCg15tEtDNObZnCt9O/EcpBC70gJSqDXGk3gDXUYof+Hrs751/Y/vybQb1LWj65JP0m5oPwf6n7D/QxC/1vMf/LXG5wIZ+T8dOmev55LPd5qVcr2m/2ufWtfhSvqKnUADfG3CHNILd96orle76tSul/9yQL9c81m92lzl4j6rqa11eVJ6fQ7cjX2wb6pwEUyN+cToKo3xQOGLXenbqTGO31yD1IpU3XZ5JS1jzIBYfTchJiTBrT7rW9dT4l07uvTDwddar+3LCUY/B3oav2p9q/7zru0CUoek9ws9ejztn2eOagtHDklvfXJY0/+1T63Lb+g2DOZ+eal5XdzOD/Z69PJHB2udz+/G36jvDYyr83LregdgTTo42+gP9wzTiItuha5vn9bno5/G/rioJXyhaaivHbW73EXhDn3ze/zdhJigLbMlbKeGIiChyTVGcKurK+nLFuvblkBvvw/GxaZX0ja8lKp2Jvfuopt6dQ7a2cu63gFYk6/PVSgszFHtwFafPq3PPtPY+/yV9DvWEPW9Mac+muKi8JaynRqCgARcJJh3VzVVW0Lxl19L/CsxmGcv70nuqf9a+1m9r0EKVv/WZ59p7H3+Svoda6hQngG/WFOczWlJ26m+uIsNuEgw765qqrYE8pdfa/8rMVhP+3W2DdOUW3rVux3B6t/67DONvc9fSb9jwdAYT4xuit/Tlrad6oOABNiE8rELjdGWQG6/D/WDSluTOd9L0EOjeimQ40Uo+rc++0xj7/NX0u9Yc9BUv6etfTtxF1sruosNgWmMu/5C1ZZAbr8P5eMWWqNzFyr1//IO6tCJr9WjcwdFucP1izd3NNrdR1Xqs/829j5/Jf2OXema8vf0StxO3OYfYgQktGSB3H7fmp+W2xjoXwQD+9E/EZBCjICEli6Qv/yuxL8SWxL6F8HAfvQNAlKIEZAAAGh+GuP4zUXaAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsAg5Imzdv1tixYxUXFyeHw6FVq1b5TXc4HDUOixYtsmp69uxZbfqCBQv85rNr1y7dcsstCg8PV3x8vBYuXFitLStWrFC/fv0UHh6uxMRErV27NtDVAQAAqCbggHT69GkNGjRIixcvrnF6SUmJ3/DKK6/I4XDojjvu8Kt7+umn/ep+/vOfW9N8Pp9Gjx6tHj16KD8/X4sWLdK8efP00ksvWTVbtmzR+PHjNXnyZO3YsUPp6elKT09XYWFhoKsEAADgp0FfNeJwOLRy5Uqlp6fXWpOenq6TJ08qNzfXGtezZ0/NnDlTM2fOrPE9S5Ys0eOPPy6PxyOn0ylJeuyxx7Rq1Srt27dPkjRu3DidPn1aq1evtt43YsQIDR48WEuXLq1T+/mqEQAAmp9m/1UjpaWlWrNmjSZPnlxt2oIFC9SlSxfdeOONWrRokS5cuGBNy8vL06hRo6xwJEmpqakqKirSV199ZdWkpKT4zTM1NVV5eXm1tqe8vFw+n89vAAAAsGsbypm/9tpr6tixo3784x/7jf/FL36hIUOGqHPnztqyZYvmzJmjkpISPffcc5Ikj8ejXr16+b0nOjramtapUyd5PB5r3MU1Ho+n1vZkZWVp/vz5wVg1AADQgoU0IL3yyiuaMGGCwsPD/cbPnj3b+nngwIFyOp166KGHlJWVJZfLFbL2zJkzx2/ZPp9P8fHxIVseAABonkIWkP7yl7+oqKhIb7/99mVrk5KSdOHCBR08eFB9+/ZVTEyMSktL/WqqXsfExFj/1lRTNb0mLpcrpAEMAAC0DCG7Bunll1/W0KFDNWjQoMvWFhQUKCwsTFFRUZKk5ORkbd68WefPn7dqcnJy1LdvX3Xq1MmqufjC76qa5OTkIK4FAABojQIOSKdOnVJBQYEKCgokScXFxSooKNDhw4etGp/PpxUrVuiBBx6o9v68vDw9//zz2rlzpw4cOKA33nhDs2bN0t13322Fn7vuuktOp1OTJ0/Wnj179Pbbb+uFF17w+3hsxowZys7O1rPPPqt9+/Zp3rx5+vTTTzV9+vRAVwkAAMCfCdCGDRuMpGrDxIkTrZrf//73pn379qasrKza+/Pz801SUpKJiIgw4eHh5oYbbjDPPPOMOXv2rF/dzp07zciRI43L5TLXXnutWbBgQbV5LV++3Fx//fXG6XSa/v37mzVr1gS0Ll6v10gyXq83oPcBAICm0xjH7wY9B6m54zlIAAA0P83+OUgAAADNEQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsCEgAAAA2BCQAAAAbAhIAAIANAQkAAMCGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsAg5Imzdv1tixYxUXFyeHw6FVq1b5TZ80aZIcDoffMGbMGL+aEydOaMKECXK73YqMjNTkyZN16tQpv5pdu3bplltuUXh4uOLj47Vw4cJqbVmxYoX69eun8PBwJSYmau3atYGuDgAAQDUBB6TTp09r0KBBWrx4ca01Y8aMUUlJiTW8+eabftMnTJigPXv2KCcnR6tXr9bmzZv14IMPWtN9Pp9Gjx6tHj16KD8/X4sWLdK8efP00ksvWTVbtmzR+PHjNXnyZO3YsUPp6elKT09XYWFhoKsEAADgx2GMMfV+s8OhlStXKj093Ro3adIklZWVVTuzVOWzzz5TQkKCPvnkEw0bNkySlJ2dre9973v629/+pri4OC1ZskSPP/64PB6PnE6nJOmxxx7TqlWrtG/fPknSuHHjdPr0aa1evdqa94gRIzR48GAtXbq0Tu33+XyKiIiQ1+uV2+2uRw8AAIDG1hjH75Bcg7Rx40ZFRUWpb9++mjZtmo4fP25Ny8vLU2RkpBWOJCklJUVhYWHaunWrVTNq1CgrHElSamqqioqK9NVXX1k1KSkpfstNTU1VXl5ere0qLy+Xz+fzGwAAAOyCHpDGjBmj//mf/1Fubq5+9atfadOmTbr99ttVUVEhSfJ4PIqKivJ7T9u2bdW5c2d5PB6rJjo62q+m6vXlaqqm1yQrK0sRERHWEB8f37CVBQAALVLbYM/wzjvvtH5OTEzUwIED1bt3b23cuFG33XZbsBcXkDlz5mj27NnWa5/PR0gCAADVhPw2/+uuu05du3bVF198IUmKiYnRsWPH/GouXLigEydOKCYmxqopLS31q6l6fbmaquk1cblccrvdfgMAAIBdyAPS3/72Nx0/flyxsbGSpOTkZJWVlSk/P9+qWb9+vSorK5WUlGTVbN68WefPn7dqcnJy1LdvX3Xq1Mmqyc3N9VtWTk6OkpOTQ71KAACghQs4IJ06dUoFBQUqKCiQJBUXF6ugoECHDx/WqVOn9Mgjj+jjjz/WwYMHlZubqx/+8Ifq06ePUlNTJUk33HCDxowZoylTpmjbtm366KOPNH36dN15552Ki4uTJN11111yOp2aPHmy9uzZo7ffflsvvPCC38djM2bMUHZ2tp599lnt27dP8+bN06effqrp06cHoVsAAECrZgK0YcMGI6naMHHiRPP111+b0aNHm2uuuca0a9fO9OjRw0yZMsV4PB6/eRw/ftyMHz/eXH311cbtdpv77rvPnDx50q9m586dZuTIkcblcplrr73WLFiwoFpbli9fbq6//nrjdDpN//79zZo1awJaF6/XayQZr9cbaDcAAIAm0hjH7wY9B6m54zlIAAA0P832OUgAAADNGQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsCEgAAAA2BCQAAAAbAhIAAIANAQkAAMCGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsAg5Imzdv1tixYxUXFyeHw6FVq1ZZ086fP6/MzEwlJibqqquuUlxcnO69914dPXrUbx49e/aUw+HwGxYsWOBXs2vXLt1yyy0KDw9XfHy8Fi5cWK0tK1asUL9+/RQeHq7ExEStXbs20NUBAACoJuCAdPr0aQ0aNEiLFy+uNu3rr7/W9u3b9eSTT2r79u165513VFRUpB/84AfVap9++mmVlJRYw89//nNrms/n0+jRo9WjRw/l5+dr0aJFmjdvnl566SWrZsuWLRo/frwmT56sHTt2KD09Xenp6SosLAx0lQAAAPw4jDGm3m92OLRy5Uqlp6fXWvPJJ59o+PDhOnTokLp37y7pmzNIM2fO1MyZM2t8z5IlS/T444/L4/HI6XRKkh577DGtWrVK+/btkySNGzdOp0+f1urVq633jRgxQoMHD9bSpUvr1H6fz6eIiAh5vV653e46vQcAADStxjh+h/waJK/XK4fDocjISL/xCxYsUJcuXXTjjTdq0aJFunDhgjUtLy9Po0aNssKRJKWmpqqoqEhfffWVVZOSkuI3z9TUVOXl5YVuZQAAQKvQNpQzP3v2rDIzMzV+/Hi/hPeLX/xCQ4YMUefOnbVlyxbNmTNHJSUleu655yRJHo9HvXr18ptXdHS0Na1Tp07yeDzWuItrPB5Pre0pLy9XeXm59drn8zV4HQEAQMsTsoB0/vx5/exnP5MxRkuWLPGbNnv2bOvngQMHyul06qGHHlJWVpZcLleomqSsrCzNnz8/ZPMHAAAtQ0g+YqsKR4cOHVJOTs5lPx9MSkrShQsXdPDgQUlSTEyMSktL/WqqXsfExFyypmp6TebMmSOv12sNR44cCXTVAABAKxD0gFQVjj7//HN98MEH6tKly2XfU1BQoLCwMEVFRUmSkpOTtXnzZp0/f96qycnJUd++fdWpUyerJjc3128+OTk5Sk5OrnU5LpdLbrfbbwAAALAL+CO2U6dO6YsvvrBeFxcXq6CgQJ07d1ZsbKx+8pOfaPv27Vq9erUqKiqsa4I6d+4sp9OpvLw8bd26Vbfeeqs6duyovLw8zZo1S3fffbcVfu666y7Nnz9fkydPVmZmpgoLC/XCCy/o17/+tbXcGTNm6Nvf/raeffZZpaWl6a233tKnn37q9ygAAACAejEB2rBhg5FUbZg4caIpLi6ucZoks2HDBmOMMfn5+SYpKclERESY8PBwc8MNN5hnnnnGnD171m85O3fuNCNHjjQul8tce+21ZsGCBdXasnz5cnP99dcbp9Np+vfvb9asWRPQuni9XiPJeL3eQLsBAAA0kcY4fjfoOUjNHc9BAgCg+WkRz0ECAABobghIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsCEgAAAA2BCQAAAAbAhIAAIANAQkAAMCGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwCTggbd68WWPHjlVcXJwcDodWrVrlN90Yo6eeekqxsbFq3769UlJS9Pnnn/vVnDhxQhMmTJDb7VZkZKQmT56sU6dO+dXs2rVLt9xyi8LDwxUfH6+FCxdWa8uKFSvUr18/hYeHKzExUWvXrg10dQAAAKoJOCCdPn1agwYN0uLFi2ucvnDhQv3mN7/R0qVLtXXrVl111VVKTU3V2bNnrZoJEyZoz549ysnJ0erVq7V582Y9+OCD1nSfz6fRo0erR48eys/P16JFizRv3jy99NJLVs2WLVs0fvx4TZ48WTt27FB6errS09NVWFgY6CoBAAD4Mw0gyaxcudJ6XVlZaWJiYsyiRYuscWVlZcblcpk333zTGGPM3r17jSTzySefWDXvv/++cTgc5u9//7sxxpgXX3zRdOrUyZSXl1s1mZmZpm/fvtbrn/3sZyYtLc2vPUlJSeahhx6qc/u9Xq+RZLxeb53fAwAAmlZjHL+Deg1ScXGxPB6PUlJSrHERERFKSkpSXl6eJCkvL0+RkZEaNmyYVZOSkqKwsDBt3brVqhk1apScTqdVk5qaqqKiIn311VdWzcXLqaqpWk5NysvL5fP5/AYAAAC7oAYkj8cjSYqOjvYbHx0dbU3zeDyKiorym962bVt17tzZr6ameVy8jNpqqqbXJCsrSxEREdYQHx8f6CoCAIBWoFXdxTZnzhx5vV5rOHLkSFM3CQAAXIGCGpBiYmIkSaWlpX7jS0tLrWkxMTE6duyY3/QLFy7oxIkTfjU1zePiZdRWUzW9Ji6XS263228AAACwC2pA6tWrl2JiYpSbm2uN8/l82rp1q5KTkyVJycnJKisrU35+vlWzfv16VVZWKikpyarZvHmzzp8/b9Xk5OSob9++6tSpk1Vz8XKqaqqWAwAAUF8BB6RTp06poKBABQUFkr65MLugoECHDx+Ww+HQzJkz9Z//+Z969913tXv3bt17772Ki4tTenq6JOmGG27QmDFjNGXKFG3btk0fffSRpk+frjvvvFNxcXGSpLvuuktOp1OTJ0/Wnj179Pbbb+uFF17Q7NmzrXbMmDFD2dnZevbZZ7Vv3z7NmzdPn376qaZPn97wXgEAAK1boLe9bdiwwUiqNkycONEY882t/k8++aSJjo42LpfL3HbbbaaoqMhvHsePHzfjx483V199tXG73ea+++4zJ0+e9KvZuXOnGTlypHG5XObaa681CxYsqNaW5cuXm+uvv944nU7Tv39/s2bNmoDWhdv8AQBofhrj+O0wxpgmzGdNyufzKSIiQl6vl+uRAABoJhrj+N2q7mIDAACoCwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsCEgAAAA2BCQAAAAbAhIAAIANAQkAAMCGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYBD0g9ezZUw6Ho9qQkZEhSfrOd75TbdrUqVP95nH48GGlpaWpQ4cOioqK0iOPPKILFy741WzcuFFDhgyRy+VSnz59tGzZsmCvCgAAaKXaBnuGn3zyiSoqKqzXhYWF+u53v6uf/vSn1rgpU6bo6aeftl536NDB+rmiokJpaWmKiYnRli1bVFJSonvvvVft2rXTM888I0kqLi5WWlqapk6dqjfeeEO5ubl64IEHFBsbq9TU1GCvEgAAaGUcxhgTygXMnDlTq1ev1ueffy6Hw6HvfOc7Gjx4sJ5//vka699//319//vf19GjRxUdHS1JWrp0qTIzM/Xll1/K6XQqMzNTa9asUWFhofW+O++8U2VlZcrOzq5z23w+nyIiIuT1euV2uxu0ngAAoHE0xvE7pNcgnTt3Tq+//rruv/9+ORwOa/wbb7yhrl27asCAAZozZ46+/vpra1peXp4SExOtcCRJqamp8vl82rNnj1WTkpLit6zU1FTl5eVdsj3l5eXy+Xx+AwAAgF3QP2K72KpVq1RWVqZJkyZZ4+666y716NFDcXFx2rVrlzIzM1VUVKR33nlHkuTxePzCkSTrtcfjuWSNz+fTmTNn1L59+xrbk5WVpfnz5wdr9QAAQAsV0oD08ssv6/bbb1dcXJw17sEHH7R+TkxMVGxsrG677Tbt379fvXv3DmVzNGfOHM2ePdt67fP5FB8fH9JlAgCA5idkAenQoUP64IMPrDNDtUlKSpIkffHFF+rdu7diYmK0bds2v5rS0lJJUkxMjPVv1biLa9xud61njyTJ5XLJ5XIFvC4AAKB1Cdk1SK+++qqioqKUlpZ2ybqCggJJUmxsrCQpOTlZu3fv1rFjx6yanJwcud1uJSQkWDW5ubl+88nJyVFycnIQ1wAAALRWIQlIlZWVevXVVzVx4kS1bfvPk1T79+/XL3/5S+Xn5+vgwYN69913de+992rUqFEaOHCgJGn06NFKSEjQPffco507d2rdunV64oknlJGRYZ39mTp1qg4cOKBHH31U+/bt04svvqjly5dr1qxZoVgdAADQyoQkIH3wwQc6fPiw7r//fr/xTqdTH3zwgUaPHq1+/frp4Ycf1h133KH33nvPqmnTpo1Wr16tNm3aKDk5WXfffbfuvfdev+cm9erVS2vWrFFOTo4GDRqkZ599Vn/84x95BhIAAAiKkD8H6UrGc5AAAGh+mv1zkAAAAJojAhIAAIANAQkAAMCGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsCEgAAAA2BCQAAAAbAhIAAIANAQkAAMCGgAQAAGBDQAIAALAhIAEAANgEPSDNmzdPDofDb+jXr581/ezZs8rIyFCXLl109dVX64477lBpaanfPA4fPqy0tDR16NBBUVFReuSRR3ThwgW/mo0bN2rIkCFyuVzq06ePli1bFuxVAQAArVRIziD1799fJSUl1vDhhx9a02bNmqX33ntPK1as0KZNm3T06FH9+Mc/tqZXVFQoLS1N586d05YtW/Taa69p2bJleuqpp6ya4uJipaWl6dZbb1VBQYFmzpypBx54QOvWrQvF6gAAgFbGYYwxwZzhvHnztGrVKhUUFFSb5vV6dc011+hPf/qTfvKTn0iS9u3bpxtuuEF5eXkaMWKE3n//fX3/+9/X0aNHFR0dLUlaunSpMjMz9eWXX8rpdCozM1Nr1qxRYWGhNe8777xTZWVlys7OrnNbfT6fIiIi5PV65Xa7G7biAACgUTTG8TskZ5A+//xzxcXF6brrrtOECRN0+PBhSVJ+fr7Onz+vlJQUq7Zfv37q3r278vLyJEl5eXlKTEy0wpEkpaamyufzac+ePVbNxfOoqqmaR23Ky8vl8/n8BgAAALugB6SkpCQtW7ZM2dnZWrJkiYqLi3XLLbfo5MmT8ng8cjqdioyM9HtPdHS0PB6PJMnj8fiFo6rpVdMuVePz+XTmzJla25aVlaWIiAhriI+Pb+jqAgCAFqhtsGd4++23Wz8PHDhQSUlJ6tGjh5YvX6727dsHe3EBmTNnjmbPnm299vl8hCQAAFBNyG/zj4yM1PXXX68vvvhCMTExOnfunMrKyvxqSktLFRMTI0mKiYmpdldb1evL1bjd7kuGMJfLJbfb7TcAAADYhTwgnTp1Svv371dsbKyGDh2qdu3aKTc315peVFSkw4cPKzk5WZKUnJys3bt369ixY1ZNTk6O3G63EhISrJqL51FVUzUPAACAhgh6QPr3f/93bdq0SQcPHtSWLVv0ox/9SG3atNH48eMVERGhyZMna/bs2dqwYYPy8/N13333KTk5WSNGjJAkjR49WgkJCbrnnnu0c+dOrVu3Tk888YQyMjLkcrkkSVOnTtWBAwf06KOPat++fXrxxRe1fPlyzZo1K9irAwAAWqGgX4P0t7/9TePHj9fx48d1zTXXaOTIkfr44491zTXXSJJ+/etfKywsTHfccYfKy8uVmpqqF1980Xp/mzZttHr1ak2bNk3Jycm66qqrNHHiRD399NNWTa9evbRmzRrNmjVLL7zwgrp166Y//vGPSk1NDfbqAACAVijoz0FqTngOEgAAzU+zfQ4SAABAc0ZAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwISABAADYEJAAAABsCEgAAAA2BCQAAAAbAhIAAIANAQkAAMCGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACATdADUlZWlm666SZ17NhRUVFRSk9PV1FRkV/Nd77zHTkcDr9h6tSpfjWHDx9WWlqaOnTooKioKD3yyCO6cOGCX83GjRs1ZMgQuVwu9enTR8uWLQv26gAAgFYo6AFp06ZNysjI0Mcff6ycnBydP39eo0eP1unTp/3qpkyZopKSEmtYuHChNa2iokJpaWk6d+6ctmzZotdee03Lli3TU089ZdUUFxcrLS1Nt956qwoKCjRz5kw98MADWrduXbBXCQAAtDIOY4wJ5QK+/PJLRUVFadOmTRo1apSkb84gDR48WM8//3yN73n//ff1/e9/X0ePHlV0dLQkaenSpcrMzNSXX34pp9OpzMxMrVmzRoWFhdb77rzzTpWVlSk7O7tObfP5fIqIiJDX65Xb7W7YigIAgEbRGMfvkF+D5PV6JUmdO3f2G//GG2+oa9euGjBggObMmaOvv/7ampaXl6fExEQrHElSamqqfD6f9uzZY9WkpKT4zTM1NVV5eXm1tqW8vFw+n89vAAAAsGsbyplXVlZq5syZuvnmmzVgwABr/F133aUePXooLi5Ou3btUmZmpoqKivTOO+9Ikjwej184kmS99ng8l6zx+Xw6c+aM2rdvX609WVlZmj9/flDXEQAAtDwhDUgZGRkqLCzUhx9+6Df+wQcftH5OTExUbGysbrvtNu3fv1+9e/cOWXvmzJmj2bNnW699Pp/i4+NDtjwAANA8hewjtunTp2v16tXasGGDunXrdsnapKQkSdIXX3whSYqJiVFpaalfTdXrmJiYS9a43e4azx5Jksvlktvt9hsAAADsgh6QjDGaPn26Vq5cqfXr16tXr16XfU9BQYEkKTY2VpKUnJys3bt369ixY1ZNTk6O3G63EhISrJrc3Fy/+eTk5Cg5OTlIawIAAFqroAekjIwMvf766/rTn/6kjh07yuPxyOPx6MyZM5Kk/fv365e//KXy8/N18OBBvfvuu7r33ns1atQoDRw4UJI0evRoJSQk6J577tHOnTu1bt06PfHEE8rIyJDL5ZIkTZ06VQcOHNCjjz6qffv26cUXX9Ty5cs1a9asYK8SAABoZYJ+m7/D4ahx/KuvvqpJkybpyJEjuvvuu1VYWKjTp08rPj5eP/rRj/TEE0/4feR16NAhTZs2TRs3btRVV12liRMnasGCBWrb9p+XTW3cuFGzZs3S3r171a1bNz355JOaNGlSndvKbf4AADQ/jXH8DvlzkK5kBCQAAJqfFvEcJAAAgOaGgAQAAGBDQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbAhIAAAANgQkAAAAGwISAACADQEJAADAhoAEAABgQ0ACAACwadvUDQCA+qioNNpWfELHTp5VVMdwDe/VWW3CHE3dLAAtBAEJQLOTXVii+e/tVYn3rDUuNiJcc8cmaMyA2CZsGYCWgo/YADQr2YUlmvb6dr9wJEke71lNe327sgtLmqhlAFoSAhKAZqOi0mj+e3tlaphWNW7+e3tVUVlTBQDUHQEJQLOxrfhEtTNHFzOSSrxnta34ROM1CkCLREAC0GwcO1l7OKpPHQDUhoAEoNmI6hge1DoAqA0BCUCzMbxXZ8VGhKu2m/kd+uZutuG9OjdmswC0QAQkAM1GmzCH5o5NkKRqIanq9dyxCTwPCUCDEZAANCtjBsRqyd1DFBPh/zFaTES4ltw9hOcgAQgKHhQJoNkZMyBW302I4UnaAEKm2Z9BWrx4sXr27Knw8HAlJSVp27ZtTd0kAI2gTZhDyb276IeDr1Vy7y6EIwBB1awD0ttvv63Zs2dr7ty52r59uwYNGqTU1FQdO3asqZsGAACasWYdkJ577jlNmTJF9913nxISErR06VJ16NBBr7zySlM3DQAANGPNNiCdO3dO+fn5SklJscaFhYUpJSVFeXl5Nb6nvLxcPp/PbwAAALBrtgHpH//4hyoqKhQdHe03Pjo6Wh6Pp8b3ZGVlKSIiwhri4+Mbo6kAAKCZabYBqT7mzJkjr9drDUeOHGnqJgEAgCtQs73Nv2vXrmrTpo1KS0v9xpeWliomJqbG97hcLrlcrsZoHgAAaMaa7Rkkp9OpoUOHKjc31xpXWVmp3NxcJScnN2HLAABAc9dszyBJ0uzZszVx4kQNGzZMw4cP1/PPP6/Tp0/rvvvua+qmAQCAZqxZB6Rx48bpyy+/1FNPPSWPx6PBgwcrOzu72oXbAAAAgXAYY0xTN6KpeL1eRUZG6siRI3K73U3dHAAAUAc+n0/x8fEqKytTRERESJbRrM8gNdTJkyclidv9AQBohk6ePBmygNSqzyBVVlbq6NGj6tixoxyO+n+PU1WS5UxUYOi3wNFn9UO/1Q/9Fjj6rH4C7TdjjE6ePKm4uDiFhYXmfrNWfQYpLCxM3bp1C9r83G43vxD1QL8Fjj6rH/qtfui3wNFn9RNIv4XqzFGVZnubPwAAQKgQkAAAAGwISEHgcrk0d+5cntIdIPotcPRZ/dBv9UO/BY4+q58rsd9a9UXaAAAANeEMEgAAgA0BCQAAwIaABAAAYENAAgAAsGkVAWnz5s0aO3as4uLi5HA4tGrVKr/pxhg99dRTio2NVfv27ZWSkqLPP//cr+bEiROaMGGC3G63IiMjNXnyZJ06dcqvZteuXbrlllsUHh6u+Ph4LVy4sFpbVqxYoX79+ik8PFyJiYlau3ZtwG1pDFlZWbrpppvUsWNHRUVFKT09XUVFRX41Z8+eVUZGhrp06aKrr75ad9xxh0pLS/1qDh8+rLS0NHXo0EFRUVF65JFHdOHCBb+ajRs3asiQIXK5XOrTp4+WLVtWrT2LFy9Wz549FR4erqSkJG3bti3gtjSGJUuWaODAgdbDzpKTk/X+++8H1M7W1md2CxYskMPh0MyZM61x9FvN5s2bJ4fD4Tf069cvoLa2xn77+9//rrvvvltdunRR+/btlZiYqE8//dSazjGhup49e1bb1xwOhzIyMiS10H3NtAJr1641jz/+uHnnnXeMJLNy5Uq/6QsWLDARERFm1apVZufOneYHP/iB6dWrlzlz5oxVM2bMGDNo0CDz8ccfm7/85S+mT58+Zvz48dZ0r9droqOjzYQJE0xhYaF58803Tfv27c3vf/97q+ajjz4ybdq0MQsXLjR79+41TzzxhGnXrp3ZvXt3QG1pDKmpqebVV181hYWFpqCgwHzve98z3bt3N6dOnbJqpk6dauLj401ubq759NNPzYgRI8y//Mu/WNMvXLhgBgwYYFJSUsyOHTvM2rVrTdeuXc2cOXOsmgMHDpgOHTqY2bNnm71795rf/va3pk2bNiY7O9uqeeutt4zT6TSvvPKK2bNnj5kyZYqJjIw0paWldW5LY3n33XfNmjVrzF//+ldTVFRk/uM//sO0a9fOFBYW1qmdrbHPLrZt2zbTs2dPM3DgQDNjxgxrPP1Ws7lz55r+/fubkpISa/jyyy/r3NbW2G8nTpwwPXr0MJMmTTJbt241Bw4cMOvWrTNffPGFVcMxobpjx4757Wc5OTlGktmwYYMxpmXua60iIF3MHpAqKytNTEyMWbRokTWurKzMuFwu8+abbxpjjNm7d6+RZD755BOr5v333zcOh8P8/e9/N8YY8+KLL5pOnTqZ8vJyqyYzM9P07dvXev2zn/3MpKWl+bUnKSnJPPTQQ3VuS1M5duyYkWQ2bdpktatdu3ZmxYoVVs1nn31mJJm8vDxjzDfBNCwszHg8HqtmyZIlxu12W/306KOPmv79+/sta9y4cSY1NdV6PXz4cJORkWG9rqioMHFxcSYrK6vObWlKnTp1Mn/84x/ps8s4efKk+da3vmVycnLMt7/9bSsg0W+1mzt3rhk0aFCN0+i3mmVmZpqRI0fWOp1jQt3MmDHD9O7d21RWVrbYfa1VfMR2KcXFxfJ4PEpJSbHGRUREKCkpSXl5eZKkvLw8RUZGatiwYVZNSkqKwsLCtHXrVqtm1KhRcjqdVk1qaqqKior01VdfWTUXL6eqpmo5dWlLU/F6vZKkzp07S5Ly8/N1/vx5v7b269dP3bt39+u3xMRERUdHWzWpqany+Xzas2ePVXOpPjl37pzy8/P9asLCwpSSkmLV1KUtTaGiokJvvfWWTp8+reTkZPrsMjIyMpSWllZt3ei3S/v8888VFxen6667ThMmTNDhw4fr3NbW2G/vvvuuhg0bpp/+9KeKiorSjTfeqD/84Q/WdI4Jl3fu3Dm9/vrruv/+++VwOFrsvtbqA5LH45Ekv41W9bpqmsfjUVRUlN/0tm3bqnPnzn41Nc3j4mXUVnPx9Mu1pSlUVlZq5syZuvnmmzVgwABJ37TV6XQqMjLSr9a+PvXtE5/PpzNnzugf//iHKioqLttvl2tLY9q9e7euvvpquVwuTZ06VStXrlRCQgJ9dglvvfWWtm/frqysrGrT6LfaJSUladmyZcrOztaSJUtUXFysW265RSdPnqTfanHgwAEtWbJE3/rWt7Ru3TpNmzZNv/jFL/Taa69Zba1qW21tbe3HhFWrVqmsrEyTJk2S1HJ/R9sGVI1WKSMjQ4WFhfrwww+buinNQt++fVVQUCCv16v//d//1cSJE7Vp06ambtYV68iRI5oxY4ZycnIUHh7e1M1pVm6//Xbr54EDByopKUk9evTQ8uXL1b59+yZs2ZWrsrJSw4YN0zPPPCNJuvHGG1VYWKilS5dq4sSJTdy65uHll1/W7bffrri4uKZuSki1+jNIMTExklTtCvfS0lJrWkxMjI4dO+Y3/cKFCzpx4oRfTU3zuHgZtdVcPP1ybWls06dP1+rVq7VhwwZ169bNGh8TE6Nz586prKzMr96+PvXtE7fbrfbt26tr165q06bNZfvtcm1pTE6nU3369NHQoUOVlZWlQYMG6YUXXqDPapGfn69jx45pyJAhatu2rdq2batNmzbpN7/5jdq2bavo6Gj6rY4iIyN1/fXX64svvmB/q0VsbKwSEhL8xt1www3WR5McEy7t0KFD+uCDD/TAAw9Y41rqvtbqA1KvXr0UExOj3Nxca5zP59PWrVuVnJwsSUpOTlZZWZny8/OtmvXr16uyslJJSUlWzebNm3X+/HmrJicnR3379lWnTp2smouXU1VTtZy6tKWxGGM0ffp0rVy5UuvXr1evXr38pg8dOlTt2rXza2tRUZEOHz7s12+7d+/2+48kJydHbrfb+g/qcn3idDo1dOhQv5rKykrl5uZaNXVpS1OqrKxUeXk5fVaL2267Tbt371ZBQYE1DBs2TBMmTLB+pt/q5tSpU9q/f79iY2PZ32px8803V3tkyV//+lf16NFDEseEy3n11VcVFRWltLQ0a1yL3dcCuqS7mTp58qTZsWOH2bFjh5FknnvuObNjxw5z6NAhY8w3t1FGRkaaP//5z2bXrl3mhz/8YY23dN54441m69at5sMPPzTf+ta3/G7pLCsrM9HR0eaee+4xhYWF5q233jIdOnSodktn27ZtzX//93+bzz77zMydO7fGWzov15bGMG3aNBMREWE2btzod2vn119/bdVMnTrVdO/e3axfv958+umnJjk52SQnJ1vTq27rHD16tCkoKDDZ2dnmmmuuqfG2zkceecR89tlnZvHixTXe1ulyucyyZcvM3r17zYMPPmgiIyP97oa4XFsay2OPPWY2bdpkiouLza5du8xjjz1mHA6H+b//+786tbM19llNLr6LzRj6rTYPP/yw2bhxoykuLjYfffSRSUlJMV27djXHjh2rU1tbY79t27bNtG3b1vzXf/2X+fzzz80bb7xhOnToYF5//XWrhmNCzSoqKkz37t1NZmZmtWktcV9rFQFpw4YNRlK1YeLEicaYb26lfPLJJ010dLRxuVzmtttuM0VFRX7zOH78uBk/fry5+uqrjdvtNvfdd585efKkX83OnTvNyJEjjcvlMtdee61ZsGBBtbYsX77cXH/99cbpdJr+/fubNWvW+E2vS1saQ039Jcm8+uqrVs2ZM2fMv/3bv5lOnTqZDh06mB/96EempKTEbz4HDx40t99+u2nfvr3p2rWrefjhh8358+f9ajZs2GAGDx5snE6nue666/yWUeW3v/2t6d69u3E6nWb48OHm448/9ptel7Y0hvvvv9/06NHDOJ1Oc80115jbbrvNCkd1bWdr67Oa2AMS/VazcePGmdjYWON0Os21115rxo0b5/c8H/qtZu+9954ZMGCAcblcpl+/fuall17ym84xoWbr1q0zkmpcfkvc1xzGGBPYOScAAICWrdVfgwQAAGBHQAIAALAhIAEAANgQkAAAAGwISAAAADYEJAAAABsCEgAAgA0BCQAAwIaABAAAYENAAgAAsCEgAQAA2BCQAAAAbP5/i75+k5eREGMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lengths = [len(text) - FOOTER_RE.search(text).start() for text in raw_texts]\n",
    "text_lengths = [len(text) for text in raw_texts]\n",
    "plt.scatter(text_lengths, lengths);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e396c73-efdd-40c5-8a21-5f7a88773d34",
   "metadata": {},
   "source": [
    "Here's the start of a typical footer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4af18589-e89d-4a33-9ab6-e1ffe23a9b48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "--------------------------------\n",
      "\n",
      "Transcriber's Notes for edition 11:\n",
      "\n",
      "I am greatly indebted to the Wodehouse readers from the BLANDINGS\n",
      "e-mail group who did such detailed research on this text, not only\n",
      "on simple typos but on the differences between the 1916 Saturday\n",
      "Evening Post serialization and the US and UK early printings.\n",
      "\n",
      "I have made use, in this new PG edition, of the 1918 UK first edition\n",
      "references provided by these helpful savants, to correct misprints or\n",
      "o\n"
     ]
    }
   ],
   "source": [
    "max_idx = lengths.index(max(lengths))\n",
    "\n",
    "print(raw_texts[max_idx][-lengths[max_idx]:-lengths[max_idx]+500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1473b53-149c-43b2-a205-28f758f3d6bd",
   "metadata": {},
   "source": [
    "The shortest one doesn't have the usual notices at the end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "99479f4e-35c4-4b4b-8fe0-5af931c7d39f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to far over” changed to “much too far over”\n",
      "\n",
      "\n",
      "\n",
      "*** END OF THE PROJECT GUTENBERG EBOOK DIVOTS ***\n"
     ]
    }
   ],
   "source": [
    "min_idx = lengths.index(min(lengths))\n",
    "\n",
    "print(raw_texts[min_idx][-100:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f558c85-8683-4aa0-b846-61c9e6e36784",
   "metadata": {},
   "source": [
    "We can put this together to make a general strip function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e57fc33c-ae0b-4056-a579-b87d3bf995eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gutenberg_strip(text: str) -> str:\n",
    "    start = HEADER_RE.search(text).end()\n",
    "    end = FOOTER_RE.search(text).start()\n",
    "    return text[start:end]\n",
    "\n",
    "texts = [gutenberg_strip(text) for text in raw_texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1468ac5-ab51-4309-85c5-bb8a500668a5",
   "metadata": {},
   "source": [
    "Note that we still haven't completely cleaned these; there still contains a few transcriber's notes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c06ee61b-e134-4015-9bd0-4b1e8235d8e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29     11/  4999 [Transcriber's note: This selection of early Wodehouse stories was\n",
      "30     11/  3405 [Transcriber's note: This selection of early Wodehouse stories was\n",
      "31     11/  4095 [Transcriber's note: This collection of early Wodehouse writings was\n"
     ]
    }
   ],
   "source": [
    "for idx, text in enumerate(texts):\n",
    "    for lno, line in enumerate(text.splitlines()):\n",
    "        if re.search(\"transcriber\", line, flags=re.IGNORECASE):\n",
    "            print(f\"{idx:2}\", f\"{lno:6}/{len(text.splitlines()):6}\", line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4af16e7e-0fb1-4ddb-a6fe-0d68e72a224c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24      0/  2257 [Illustration: Frontispiece]\n",
      "25     17/ 15152 [Illustration (Frontispiece): “ARE YOU THE M. JACKSON THEN WHO HAD AN\n",
      "28      0/  6865 [Illustration]\n",
      "36      0/  6769    [Illustration: \"Never mind the ink, old horse. It'll soak in.\"]\n",
      "39     56/  9530 [Illustration: A HERBERT JENKINS’ BOOK]\n",
      "40     64/ 11665 [Illustration: A HERBERT JENKINS’ BOOK]\n",
      "44     40/ 11765                             [Illustration]\n",
      "47     32/  9109                         [Illustration: NEW YORK]\n"
     ]
    }
   ],
   "source": [
    "for idx, text in enumerate(texts):\n",
    "    for lno, line in enumerate(text.splitlines()):\n",
    "        if re.search(\"\\\\[.*illustration\", line, flags=re.IGNORECASE):\n",
    "            print(f\"{idx:2}\", f\"{lno:6}/{len(text.splitlines()):6}\", line)\n",
    "            break"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e9bb5c42-3505-4a96-88ab-a24f0c5374d9",
   "metadata": {},
   "source": [
    "# Chapter numbers\n",
    "\n",
    "Ideally we would break each book into chapters; in this section I attempt to do that, find it too complex, and give up.\n",
    "This section can be skipped over, but is a good start as to how one might break a book into chapters (and about building rule-based systems with regex).\n",
    "\n",
    "Chapters come in a variety of formats that we could handle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "04608f58-8a6a-4ff6-804d-3f647aff9e39",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 11 CHAPTER I\n",
      "1 8 CHAPTER I\n",
      "2 8 CHAPTER 1.\n",
      "3 36 CHAPTER I\n",
      "4 7  CHAPTER I. DISTRESSING SCENE\n",
      "5 92 CHAPTER I\n",
      "6 11 CHAPTER\n",
      "7 1179 Chapter 1\n",
      "9 5991 perhaps it is as well to close the chapter. How it happened it is hard\n",
      "11 13 CHAPTER ONE\n",
      "12 7  CHAPTER ONE\n",
      "13 13 Chapter\n",
      "14 24 Chapter\n",
      "15 10  CHAPTER I. A Pawn of Fate\n",
      "16 31 Chapter\n",
      "17 17 CHAPTER\n",
      "18 2248 _Chapter 1_\n",
      "19 23 CHAPTER I Patient Perseverance Produces Pugilistic Prodigies\n",
      "20 326 which break off abruptly at the third chapter, owing to the premature\n",
      "21 62 POSTSCRIPT.--In the second chapter I allude to Stout Cortez staring at\n",
      "22 48 Chapter 1\n",
      "23 14 CHAPTER 1 JAMES ARRIVES\n",
      "24 91 CHAPTER I\n",
      "25 31 CHAPTER\n",
      "26 9 CHAPTER I. SALLY GIVES A PARTY\n",
      "27 4972 And he told me this chapter from the unwritten history of the London\n",
      "28 1785 novels who calls off the fight with his wife in the last chapter and\n",
      "29 3466 heavier monthly reviews; but Bingo had got chapter one of \"All for\n",
      "31 2410 \"Types of Ethical Theory\" and took a stab at a chapter headed\n",
      "33 12 CHAPTER I.\n",
      "35 13 CHAPTER\n",
      "36 47 CHAPTER\n",
      "37 38 CHAPTER\n",
      "38 103 CHAPTER I.\n",
      "39 67 CHAPTER\n",
      "40 76   CHAPTER                                           PAGE\n",
      "41 95   CHAPTER\n",
      "42 17   CHAPTER                                                           PAGE\n",
      "43 315 on _Types of Ethical Theory_ and took a stab at a chapter headed\n",
      "44 54 CHAPTER                                                             PAGE\n",
      "45 21                               CHAPTER ONE\n",
      "46 6541 he was moving on all six cylinders to the end of a chapter.\n",
      "47 165 CHAPTER                                                             PAGE\n"
     ]
    }
   ],
   "source": [
    "for idx, text in enumerate(texts):\n",
    "    lines = text.splitlines()\n",
    "    for lno, line in enumerate(lines):\n",
    "        if re.search(\"chapter\", line, flags=re.IGNORECASE):\n",
    "            print(idx, lno, line)\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fbb952a-32a4-4360-89f7-ed2a669e58af",
   "metadata": {},
   "source": [
    "## Roman Numerals\n",
    "\n",
    "Roman numerals do occur in chapter numbers (and some other places) as we can see looking for the unlikely string \"viii\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "93092a3a-063d-4c32-bfab-7f3590b22433",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 84  VIII. SIR MALLABY OFFERS A SUGGESTION\n",
      "0 3670 CHAPTER VIII.\n",
      "1 82  VIII. SIR RODERICK COMES TO LUNCH\n",
      "1 102 XVIII. ALL’S WELL\n",
      "1 2735 CHAPTER VIII\n",
      "1 8959 CHAPTER XVIII\n",
      "2 92   VIII CONFIDENCES ON THE LAKE                       135\n",
      "2 4685 CHAPTER VIII\n",
      "3 112   VIII. The Long Arm of Looney Coote\n",
      "3 6986   CHAPTER VIII\n",
      "4 25   VIII. AT DREEVER                                                    48\n",
      "4 35   XVIII. THE LOCHINVAR METHOD                                        118\n",
      "4 45   XXVIII. SPENNIE’S HOUR OF CLEAR VISION                             185\n",
      "4 6406 probably belonged to this class, and even Henry VIII must have become a\n",
      "6 70 VIII. SAM AT MON REPOS                                                78\n",
      "6 90 XVIII. DISCUSSION AT A LUNCHEON TABLE                                196\n",
      "6 110 XXVIII. THE MISSING MILLIONS                                         329\n",
      "8 2324     'Blessed shall be thy basket and thy store.' Deuteronomy xxviii, 5.\n",
      "9 181 VIII JANE GETS OFF THE FAIRWAY                                       246\n",
      "9 6968                              CHAPTER VIII\n"
     ]
    }
   ],
   "source": [
    "for idx, text in enumerate(texts[-10:]):\n",
    "    lines = text.splitlines()\n",
    "    for lno, line in enumerate(lines):\n",
    "        if re.search(\"viii\", line, flags=re.IGNORECASE):\n",
    "            print(idx, lno, line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e2f1c0b-c32f-4174-aaf7-0711eddbac76",
   "metadata": {},
   "source": [
    "It's unlikely there's more than 99 chapters so here's a simple expression to detect the roman numerals from 1-99:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "cd1c3d05-aab0-4f19-8657-d97758e0a161",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "ROMAN_NUMERAL_RE = \"(?:(?:(?:XC|XL|L?X{0,3})(?:IX|I?V|V?I{1,3}))|(?:XC|X?L|L?X{1,3}))\"\n",
    "\n",
    "def is_roman_numeral(s: str) -> bool:\n",
    "    return re.match(ROMAN_NUMERAL_RE + \"$\", s)\n",
    "\n",
    "assert is_roman_numeral(\"I\")\n",
    "assert is_roman_numeral(\"II\")\n",
    "assert is_roman_numeral(\"XLIV\")\n",
    "assert not is_roman_numeral(\"IL\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27026515-ca9b-48ab-a466-77d7213529f4",
   "metadata": {},
   "source": [
    "Note that some are ambiguous, like `I` occurs frequently in English sentences and `X` is sometimes used as a mark for a kiss.\n",
    "\n",
    "If we have a Roman Numeral we can then convert it back to an integer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2fac4fd3-f3c7-40ec-8437-5c91ce2bf775",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "ROMAN_NUMERALS = dict(\n",
    "    M=1000,\n",
    "    CM=900,\n",
    "    D=500,\n",
    "    CD=400,\n",
    "    C=100,\n",
    "    XC=90,\n",
    "    L=50,\n",
    "    XL=40,\n",
    "    X=10,\n",
    "    IX=9,\n",
    "    V=5,\n",
    "    IV=4,\n",
    "    I=1,\n",
    ")\n",
    "\n",
    "def int_to_roman_numeral(n: int) -> str:\n",
    "    if not (0 <= n < 4000 and int(n) == n):\n",
    "        raise ValueError(\"Expected an integer between 0 and 3999\")\n",
    "    ans = []\n",
    "    for numeral, base in ROMAN_NUMERALS.items():\n",
    "        count, n = divmod(n, base)\n",
    "        ans += count * numeral\n",
    "    assert n == 0\n",
    "    return ''.join(ans)\n",
    "\n",
    "def roman_numeral_to_int(numeral: str) -> int:\n",
    "    \"\"\"Expand roman numerals\"\"\"\n",
    "    numeral_expanded = (\n",
    "        numeral\n",
    "        .replace('CM', 'C' * 9)\n",
    "        .replace('CD', 'C' * 4)\n",
    "        .replace('XC', 'X' * 9)\n",
    "        .replace('XL', 'X' * 4)\n",
    "        .replace('IX', 'I' * 9)\n",
    "        .replace('IV', 'I' * 4)\n",
    "    )\n",
    "    return sum([count * ROMAN_NUMERALS[n]\n",
    "                for n, count in Counter(numeral_expanded).items()])\n",
    "\n",
    "assert roman_numeral_to_int('MCMXLVIII') == 1948"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1336b8b-fe3d-4dbd-b34d-71fda86d921a",
   "metadata": {},
   "source": [
    "## Words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7a23789-2a12-4b2d-a485-4a895d2609b4",
   "metadata": {},
   "source": [
    "Occasionally words are used for chapter numbers too (looking near the end of this passage):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "62cd1f8e-530d-43ed-bc4b-d8abcd171341",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 603 Eugenia that a boy of twenty-one needed regular work.\"\n",
      "0 1636 a young man of twenty-one. For forty-five years he had lived a\n",
      "1 626 over the age of twenty-one whose means happened to be a trifle\n",
      "2 1125 sixty-five or so, he became a sprightly lad of twenty-one in a\n",
      "2 1133 attired young man, aged about twenty-one, who during George's\n",
      "2 5357        \"I'm twenty-one today!\n",
      "2 5358         Twenty-one today!\n",
      "2 5360         Never been twenty-one before!\n",
      "2 5364           Twenty-one today.\"\n",
      "4 3213 gentlemen, I don’t want to give this superb figure away. Twenty-one\n",
      "4 3214 hundred. Twenty-one-one-one-one. This is more the sort of thing I have\n",
      "5 47 behind, I may mention that on page twenty-one I was able to make\n",
      "7 1184 I am strongly of the opinion that, after the age of twenty-one, a\n",
      "7 1186 The hour breeds thought. At twenty-one, life being all future, it\n",
      "8 1452 Elizabeth Boyd was twenty-one, but standing there with her hair\n",
      "11 3954 in verse, and twenty-one specifically asking her to marry him, and he\n",
      "11 3977 twenty-one any longer, and at your age I was working like a beaver.\n",
      "12 27  CHAPTER TWENTY-ONE\n",
      "12 4101 feel in the neighborhood of twenty-one, and Ambition is tapping me on\n",
      "12 14707 CHAPTER TWENTY-ONE\n"
     ]
    }
   ],
   "source": [
    "for idx, text in enumerate(texts[:20]):\n",
    "    lines = text.splitlines()\n",
    "    for lno, line in enumerate(lines):\n",
    "        if re.search(\"twenty.one\", line, flags=re.IGNORECASE):\n",
    "            print(idx, lno, line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1026ad33-4f24-4a1e-9c0a-f688a0045b8d",
   "metadata": {},
   "source": [
    "We can generate the number words and look them up:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "78416b93-7746-48bf-8744-0adbb7d418b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUMBER_WORDS = [\n",
    "    'ONE', 'TWO', 'THREE', 'FOUR', 'FIVE',\n",
    "    'SIX', 'SEVEN', 'EIGHT', 'NINE', 'TEN',\n",
    "    'ELEVEN', 'TWELVE', 'THIRTEEN', 'FOURTEEN', 'FIFTEEN',\n",
    "    'SIXTEEN', 'SEVENTEEN', 'EIGHTEEN', 'NINETEEN']\n",
    "\n",
    "NUMBER_WORDS = NUMBER_WORDS + [\n",
    "    (prefix + '-' + n).rstrip('-')\n",
    "    for prefix in (\"TWENTY\", \"THIRTY\", \"FORTY\", \"FIFTY\")\n",
    "    for n in ['-'] + NUMBER_WORDS[:9]\n",
    "]\n",
    "\n",
    "NUMBER_WORD_RE = '(?:' + '|'.join(NUMBER_WORDS) + ')'\n",
    "\n",
    "def number_word_to_int(s: str) -> int:\n",
    "    return NUMBER_WORDS.index(s) + 1\n",
    "\n",
    "assert number_word_to_int(\"TWO\") == 2\n",
    "assert number_word_to_int(\"TWENTY-NINE\") == 29\n",
    "assert number_word_to_int(\"THIRTY\") == 30"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6deb01a-d9ae-4e48-8748-2960a15f15fb",
   "metadata": {},
   "source": [
    "## Digits\n",
    "\n",
    "Digits are also used sometimes as chapter numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "176b7ea5-9c19-41ab-949d-cc5b8e683858",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 1466     Total (for four wickets) ....... 139\n",
      "2 4622 CHAPTER 13.\n",
      "7 7319 Chapter 13\n",
      "8 4098 13\n",
      "9 45 13. Mike is Moved On\n",
      "9 2709 13. Mike is Moved On\n",
      "12 12105 him. “Shoes” ($213.50) he could understand, but what on earth was\n",
      "18 60 13 Work\n",
      "18 5250 [13]\n",
      "20 46 13 Leicester's House Team Goes Into a Second Edition\n",
      "20 2676 'Rather. By a wicket. Two wickets, I mean. We made 213. Your bowling\n",
      "20 3549 [13]\n",
      "23 31 CHAPTER 13 THE SECOND GHOST\n",
      "23 434 Gerrard to 137 Glasgow, there is always that awful, that disintegrating\n",
      "23 3587 CHAPTER 13\n",
      "25 7895 Bishop Geoffrey, 1133-40----”\n",
      "27 8847 '\"Milicz, or Militsch (d. 1374), Bohemian divine, was the most\n",
      "30 1737     your favour of the 13th _prox_., but cannot see where it quite\n",
      "34 5231 -13-\n",
      "35 27  13.  JELLICOE GOES ON THE SICK LIST\n",
      "35 472 1133-40--\"\n",
      "35 3184 13\n",
      "37 12024 \"Shoes\" ($213.50) he could understand, but what on earth was \"Academy.\n",
      "37 14970 'em,\" which is page 313.\n",
      "40 92   VIII CONFIDENCES ON THE LAKE                       135\n",
      "40 104    XIV PSMITH ACCEPTS EMPLOYMENT                     313\n",
      "42 37   XX. A LESSON IN PIQUET                                             131\n",
      "42 38   XXI. LOATHSOME GIFTS                                               138\n",
      "42 3684                                  ★ 13 ★\n",
      "43 54                       ISBN-13: 978-0-140-28408-9\n",
      "44 106 XXVI. SAM HEARS GOOD NEWS                                            313\n"
     ]
    }
   ],
   "source": [
    "for idx, text in enumerate(texts):\n",
    "    lines = text.splitlines()\n",
    "    for lno, line in enumerate(lines):\n",
    "        if re.search(\"13\", line, flags=re.IGNORECASE):\n",
    "            print(idx, lno, line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae21e051-0740-4f99-a03e-3a7a9a409b6b",
   "metadata": {},
   "source": [
    "We can write an expression to get the numbers from 1 to 99:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a97abc20-08ed-494e-a74f-e1e6741fdf9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUMBER_DIGIT_RE = '(?:[1-9][0-9]?)'\n",
    "\n",
    "for i in range(1, 100):\n",
    "    assert re.match(NUMBER_DIGIT_RE + \"$\", str(i))\n",
    "\n",
    "def digit_to_int(s: str) -> int:\n",
    "    return int(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf920c1-0e87-44c3-8c9b-e84c852aac2c",
   "metadata": {},
   "source": [
    "## Putting them together\n",
    "\n",
    "We can combine these to turn any of these number strings into an integer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d0edc578-38d9-4183-9804-b72c8d697b08",
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_to_int(s: str) -> int:\n",
    "    if s in NUMBER_WORDS:\n",
    "        return number_word_to_int(s)\n",
    "    elif re.match(ROMAN_NUMERAL_RE + \"$\", s):\n",
    "        return roman_numeral_to_int(s)\n",
    "    elif re.match(NUMBER_DIGIT_RE + \"$\", s):\n",
    "        return int(s)\n",
    "    else:\n",
    "        raise ValueError(\"Can't convert %s to int\" % s)\n",
    "\n",
    "assert text_to_int(\"TWENTY-NINE\") == 29\n",
    "assert text_to_int(\"29\") == 29\n",
    "assert text_to_int(\"XXIX\") == 29"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "01f519a9-e0fa-45cd-a4a5-05c7065762c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUMBER_RE = \"(?:\" + \"|\".join([NUMBER_DIGIT_RE, ROMAN_NUMERAL_RE, NUMBER_WORD_RE]) + \")\"\n",
    "\n",
    "assert re.match(NUMBER_RE + \"$\", \"TWENTY-NINE\")\n",
    "assert re.match(NUMBER_RE + \"$\", \"29\")\n",
    "assert re.match(NUMBER_RE + \"$\", \"XXIX\")\n",
    "assert not re.match(NUMBER_RE + \"$\", \"\")\n",
    "assert not re.match(NUMBER_RE + \"$\", \"I am\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c8dc872-0bda-4233-927e-c0ecfe06acdf",
   "metadata": {},
   "source": [
    "## Detecting chapters\n",
    "\n",
    "Let's have a go at using what we've built to detect chapters using a rule based system.\n",
    "Because there's so much variability in how chapters are written between books we'll need some complex rules (or a heuristic to decide which rule to use).\n",
    "The only way to stay sane while doing this is to build a collection of test-cases which looks remarkably similar to a very small validation set.\n",
    "\n",
    "I've pulled these over time out of the texts by running the rules across them looking for counter examples (e.g. missed chapters)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "fd64cffd-d045-4ae4-a5f1-5f98f792cce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = [\n",
    "    \"\\n\\nCHAPTER I. DISTRESSING SCENE\",\n",
    "    \"\\n\\nCHAPTER I. A Pawn of Fate\",\n",
    "    \"\\n\\n_Chapter 1_\",\n",
    "    \"\\n\\n                              CHAPTER ONE\",\n",
    "    \"\\n\\nXVIII. ALL’S WELL\", \n",
    "    \"\\n\\n  XXVIII. SPENNIE’S HOUR OF CLEAR VISION \",\n",
    "    \"\\n\\nCHAPTER VIII\",\n",
    "    \"\\n\\nVIII. The Long Arm of Looney Coote\",\n",
    "    \"\\n\\nVIII   A NIGHT ADVENTURE--THE DETHRONEMENT OF FENN\",\n",
    "    \"\\n\\nXVIII\",\n",
    "    \"\\n\\nVIII.   --A LITTLE DINNER AT UKRIDGE'S\",\n",
    "    \"\\n\\nVIII CONFIDENCES ON THE LAKE                       135\",\n",
    "    \"\\n\\nVIII JANE GETS OFF THE FAIRWAY\",\n",
    "    \"\\n\\n                             CHAPTER VIII\",\n",
    "    \"\\n\\nCHAPTER TWENTY-ONE\",\n",
    "    \"\\n\\nCHAPTER 13.\",\n",
    "    \"\\n\\n13\",\n",
    "    \"\\n\\nChapter 13\",\n",
    "    \"\\n\\n-13-\",\n",
    "    \"\\n\\n13.  JELLICOE GOES ON THE SICK LIST\",\n",
    "    \"\\n\\n                                 ★ 13 ★\",\n",
    "    \"\\n\\n13. Mike is Moved On\\n\",\n",
    "    \"\\n\\n[13]\\n\",\n",
    "    \"\\n\\n  I. Ukridge’s Dog College\\n\",\n",
    "    \"\\n\\nX\\n\",\n",
    "]\n",
    "\n",
    "examples = [ex + \"\\n\" for ex in examples]\n",
    "\n",
    "non_examples = [\n",
    "\"\"\"TO MY\n",
    "BEST FRIEND\n",
    "X\n",
    "\n",
    "\"\"\",\n",
    "\"\"\"with business. You want Mr. Peters' five thousand dollars. So do\n",
    "I.\"\n",
    "\"\"\",\n",
    "    \n",
    "    \"\\nAnd he told me this chapter from the unwritten history of the London\",\n",
    "    \"\\nskip impatiently to Volume XXVIII (VET-ZYM) to see how it all comes out\",\n",
    "    \"\\n[Illustration: PLATE VIII]\",\n",
    "    \"\\nhe was moving on all six cylinders to the end of a chapter.\",\n",
    "    \"\\nprobably belonged to this class, and even Henry VIII must have become a\",\n",
    "    \"\\n'Blessed shall be thy basket and thy store.' Deuteronomy xxviii,\",\n",
    "    \"\\n        Twenty-one today!\",\n",
    "    \"\\ntwenty-one.\",\n",
    "    \"\\n3 YORK STREET LONDON S.W.1\",\n",
    "    \"\\n20. I shall inform her of my decision on the long-distance telephone.\",\n",
    "    \"\"\"Dear Dad:\\n\\nI have been thinking over what we talked about this\\n\"\"\"\n",
    "]\n",
    "\n",
    "non_examples = [ex + \"\\n\" for ex in non_examples]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dbf6be3-c6a0-4cce-8098-4e836c9b0ac0",
   "metadata": {},
   "source": [
    "We extend our number regular expression to:\n",
    "\n",
    "* capture the value with groups\n",
    "* exclude sentences starting with I and contianing a lower case letter as a Roman Numeral (if they're upper case they can be a chapter title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2107e621-c0fc-4b01-8c9d-522a457098f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'(?P<number>(?P<digit>(?:[1-9][0-9]?))|(?P<roman>(?:(?:(?:XC|XL|L?X{0,3})(?:IX|I?V|V?I{1,3}))|(?:XC|X?L|L?X{1,3}))(?=\\\\.|\\\\s*\\\\r?\\\\n|\\\\s+[^a-z]+$))|(?P<word>(?:ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE|TEN|ELEVEN|TWELVE|THIRTEEN|FOURTEEN|FIFTEEN|SIXTEEN|SEVENTEEN|EIGHTEEN|NINETEEN|TWENTY|TWENTY-ONE|TWENTY-TWO|TWENTY-THREE|TWENTY-FOUR|TWENTY-FIVE|TWENTY-SIX|TWENTY-SEVEN|TWENTY-EIGHT|TWENTY-NINE|THIRTY|THIRTY-ONE|THIRTY-TWO|THIRTY-THREE|THIRTY-FOUR|THIRTY-FIVE|THIRTY-SIX|THIRTY-SEVEN|THIRTY-EIGHT|THIRTY-NINE|FORTY|FORTY-ONE|FORTY-TWO|FORTY-THREE|FORTY-FOUR|FORTY-FIVE|FORTY-SIX|FORTY-SEVEN|FORTY-EIGHT|FORTY-NINE|FIFTY|FIFTY-ONE|FIFTY-TWO|FIFTY-THREE|FIFTY-FOUR|FIFTY-FIVE|FIFTY-SIX|FIFTY-SEVEN|FIFTY-EIGHT|FIFTY-NINE)))'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NUMBER_RE = \"(?P<number>\" + \"|\".join(\n",
    "    [\n",
    "      \"(?P<digit>\" + NUMBER_DIGIT_RE + \")\",\n",
    "      # We have to be more careful with roman numerals, especially I\n",
    "      # We look ahead to check there are no lowercase letters\n",
    "      \"(?P<roman>\" + ROMAN_NUMERAL_RE + r\"(?=\\.|\\s*\\r?\\n|\\s+[^a-z]+$))\",  \n",
    "      \"(?P<word>\" + NUMBER_WORD_RE + \")\"\n",
    "    ]) + \")\"\n",
    "\n",
    "NUMBER_RE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfde7852-34c1-4e3b-ac91-a4a87a7bf92c",
   "metadata": {},
   "source": [
    "We can then check all of our test expressions match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "202a6776-b653-4721-9dd7-249c41da548c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'25 / 25; 13 / 13 '"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CHAPTER_RE = re.compile(\"|\".join([\n",
    "    r\"(?:\\n\\r?\\n[\\s_★[-]*(?P<chapter>CHAPTER |Chapter )?\" + NUMBER_RE + r\"(_|\\b))\",\n",
    "]), flags=re.MULTILINE)\n",
    "\n",
    "f\"{sum([1 for example in examples if CHAPTER_RE.search(example)])} / {len(examples)}; \" + \\\n",
    "f\"{sum([1 for example in non_examples if not CHAPTER_RE.search(example)])} / {len(non_examples)} \""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ae60c87-2c8a-4957-973a-24e3c124771e",
   "metadata": {},
   "source": [
    "When we have a match we want to pull out the number and some details about the type of match (e.g. was it in Roman numerals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "237552d5-f49a-4152-9bad-054ce7b28968",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'number': 1, 'type': ['chapter', 'roman']}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def extract_match_data(match):\n",
    "    match_type = [k for k,v in match.groupdict().items() if v and k!= \"number\"]\n",
    "    return dict(number=text_to_int(match.group(\"number\")), type=match_type)\n",
    "\n",
    "extract_match_data(CHAPTER_RE.search(examples[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eda615b-de1a-4977-b7c5-b871a5f98835",
   "metadata": {},
   "source": [
    "Next we want to look into the chapters of books.\n",
    "We expect them to be in order so let's turn sequences of consecutive numbers into ranges:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f63f8d69-bf21-45e3-9403-7dbc2cc7b6ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import reduce\n",
    "\n",
    "def to_ranges_1(xs: list[tuple[int, int]], y: int) -> list[tuple[int, int]]:\n",
    "    if not xs:\n",
    "        return [(y,y)]\n",
    "    start, end = xs[-1]\n",
    "    if y == end + 1:\n",
    "        return xs[:-1] + [(start, y)]\n",
    "    else:\n",
    "        return xs + [(y,y)]\n",
    "\n",
    "def to_ranges(xs: list[int]) -> list[tuple[int, int]]:\n",
    "    return list(reduce(to_ranges_1, xs, []))\n",
    "\n",
    "assert to_ranges([1,2,3,4,5,\n",
    "           1, 2, 3,\n",
    "           5,\n",
    "           1, \n",
    "           5]) == [\n",
    "           (1,5),\n",
    "           (1,3),\n",
    "           (5,5),\n",
    "           (1,1),\n",
    "           (5,5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3090e317-2845-4bc8-89fb-74102859159d",
   "metadata": {},
   "source": [
    "We can then summarise the matches by getting the ranges for each type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "ad867b88-90a6-493d-9993-3d1c89f50b92",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(['roman'], (1, 3)), (['digit'], (4, 4)), (['digit'], (1, 1))]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from itertools import groupby\n",
    "from operator import itemgetter\n",
    "\n",
    "def summarise_matches(match_data):\n",
    "    for key, values in groupby(match_data, key=itemgetter(\"type\")):\n",
    "        for range in to_ranges(v[\"number\"] for v in values):\n",
    "            yield (key, range)\n",
    "\n",
    "list(\n",
    "    summarise_matches([\n",
    "    dict(number=1, type=[\"roman\"]),\n",
    "    dict(number=2, type=[\"roman\"]),\n",
    "    dict(number=3, type=[\"roman\"]),\n",
    "    # Break because we're changing type\n",
    "    dict(number=4, type=[\"digit\"]),\n",
    "    # Break because non-sequential\n",
    "    dict(number=1, type=[\"digit\"])\n",
    "])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc6d9390-8c17-407a-883c-f183dcfb9df3",
   "metadata": {},
   "source": [
    "We can then apply it to all the texts and learn a bit about them:\n",
    "\n",
    "* For some texts this works really well (e.g. the first 4)\n",
    "* Sometimes the chapters occur twice (e.g. idx=9): the first time generally corresponds to a table of contents. This can also surface as 1 chapter followed by 1-N as in idx 2 (when there are no lines between subsequent chapters in the contents).\n",
    "* Some texts have a heirarchy with books contianing multiple parts each with their own chapters (e.g. idx=23), or with chapters broken into sections (e.g. idx=26)\n",
    "* There are occasional errors here (like a chapter name starting in TWO as in idx=17)\n",
    "* Some texts aren't broken into chapters but a different unit such as books (e.g. for idx 28)\n",
    "* Some are a mixture of the above (e.g. idx 10 is mainly books, one which starts with the word \"THREE\", but the last one is split into numbered chapters)\n",
    "\n",
    "With some more rules we could remove the contents and break these texts reliably into chapters, but we won't for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "8e5d1d41-1277-40e4-b5ec-ff3f6e5d3f5d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 0 2005 [(['chapter', 'roman'], (1, 26))]\n",
      " 1 2042 [(['chapter', 'roman'], (1, 12))]\n",
      " 2 2233 [(['chapter', 'digit'], (1, 27))]\n",
      " 3 2607 [(['chapter', 'roman'], (1, 29))]\n",
      " 4 3756 [(['chapter', 'roman'], (1, 1)), (['chapter', 'roman'], (1, 26))]\n",
      " 5 3829 [(['roman'], (1, 1)), (['chapter', 'roman'], (1, 10)), (['roman'], (1, 1)), (['chapter', 'roman'], (11, 23))]\n",
      " 6 4075 [(['chapter', 'roman'], (1, 22)), (['word'], (2, 2)), (['chapter', 'roman'], (23, 30))]\n",
      " 7 6683 [(['roman'], (1, 2)), (['chapter', 'digit'], (1, 1)), (['roman'], (1, 3)), (['chapter', 'digit'], (2, 4)), (['roman'], (1, 2)), (['chapter', 'digit'], (5, 6)), (['roman'], (1, 2)), (['chapter', 'digit'], (7, 8)), (['roman'], (1, 3)), (['chapter', 'digit'], (9, 9)), (['roman'], (1, 2)), (['chapter', 'digit'], (10, 12)), (['roman'], (1, 2)), (['chapter', 'digit'], (13, 14)), (['roman'], (1, 2)), (['chapter', 'digit'], (15, 15)), (['roman'], (1, 2)), (['chapter', 'digit'], (16, 18))]\n",
      " 8 6684 [(['digit'], (1, 25))]\n",
      " 9 6753 [(['digit'], (1, 30)), (['digit'], (1, 30))]\n",
      "10 6768 [(['word'], (3, 3)), (['word'], (3, 3)), (['digit'], (2, 6))]\n",
      "11 6836 [(['chapter', 'word'], (1, 6)), (['digit'], (7, 7)), (['chapter', 'word'], (7, 16)), (['digit'], (1, 6))]\n",
      "12 6837 [(['chapter', 'word'], (1, 1)), (['chapter', 'word'], (1, 20)), (['chapter', 'word'], (20, 20))]\n",
      "13 6877 [(['roman'], (1, 24)), (['roman'], (1, 24))]\n",
      "14 6879 [(['roman'], (1, 24)), (['roman'], (1, 24))]\n",
      "15 6880 [(['chapter', 'roman'], (1, 12)), (['chapter', 'roman'], (1, 16))]\n",
      "16 6927 [(['roman'], (1, 24)), (['roman'], (1, 24))]\n",
      "17 6955 [(['roman'], (1, 30)), (['chapter', 'roman'], (1, 16)), (['word'], (2, 2)), (['chapter', 'roman'], (17, 30))]\n",
      "18 6980 [(['digit'], (1, 16)), (['digit'], (1, 3)), (['roman'], (1, 10)), (['digit'], (4, 8)), (['chapter', 'digit'], (1, 6)), (['digit'], (9, 9)), (['roman'], (2, 2)), (['digit'], (10, 16))]\n",
      "19 6984 [(['chapter', 'roman'], (1, 18))]\n",
      "20 6985 [(['digit'], (1, 18)), (['digit'], (1, 18))]\n",
      "21 7028 [(['roman'], (1, 10)), (['digit'], (1, 10))]\n",
      "22 7050 [(['chapter', 'digit'], (1, 7)), (['chapter', 'digit'], (1, 11))]\n",
      "23 7230 [(['chapter', 'digit'], (1, 3)), (['chapter', 'digit'], (1, 26))]\n",
      "24 7298 [(['chapter', 'roman'], (1, 15))]\n",
      "25 7423 [(['roman'], (2, 59)), (['chapter', 'roman'], (1, 59))]\n",
      "26 7464 [(['chapter', 'roman'], (1, 1)), (['digit'], (1, 3)), (['chapter', 'roman'], (2, 2)), (['digit'], (1, 6)), (['chapter', 'roman'], (3, 3)), (['digit'], (1, 3)), (['chapter', 'roman'], (4, 6)), (['digit'], (1, 4)), (['chapter', 'roman'], (7, 8)), (['digit'], (1, 3)), (['chapter', 'roman'], (9, 10)), (['digit'], (1, 3)), (['chapter', 'roman'], (11, 13)), (['digit'], (1, 4)), (['chapter', 'roman'], (14, 14)), (['digit'], (1, 3)), (['chapter', 'roman'], (15, 16)), (['digit'], (1, 5)), (['chapter', 'roman'], (17, 17)), (['digit'], (1, 3)), (['chapter', 'roman'], (18, 18))]\n",
      "27 7471 [(['word'], (1, 1)), (['roman'], (1, 2)), (['word'], (1, 1))]\n",
      "28 8164 []\n",
      "29 8176 [(['roman'], (1, 7))]\n",
      "30 8178 [(['digit'], (5, 5))]\n",
      "31 8190 []\n",
      "32 8713 []\n",
      "33 8931 [(['chapter', 'roman'], (1, 20))]\n",
      "34 10554 [(['digit'], (1, 23))]\n",
      "35 10586 [(['digit'], (1, 1)), (['digit'], (1, 30))]\n",
      "36 20532 [(['roman'], (1, 23)), (['roman'], (1, 9)), (['roman'], (1, 1)), (['roman'], (10, 18)), (['roman'], (1, 1)), (['roman'], (19, 23))]\n",
      "37 20533 [(['digit'], (3, 3)), (['roman'], (1, 1)), (['chapter', 'roman'], (1, 1)), (['roman'], (1, 4)), (['chapter', 'roman'], (2, 2)), (['roman'], (1, 2)), (['chapter', 'roman'], (3, 3)), (['roman'], (1, 2)), (['chapter', 'roman'], (4, 4)), (['roman'], (1, 4)), (['chapter', 'roman'], (5, 5)), (['roman'], (1, 3)), (['chapter', 'roman'], (6, 6)), (['roman'], (1, 2)), (['chapter', 'roman'], (7, 7)), (['roman'], (1, 3)), (['chapter', 'roman'], (8, 8)), (['roman'], (1, 2)), (['chapter', 'roman'], (9, 9)), (['roman'], (1, 3)), (['chapter', 'roman'], (10, 10)), (['roman'], (1, 2)), (['chapter', 'roman'], (11, 11)), (['roman'], (1, 2)), (['chapter', 'roman'], (12, 12)), (['roman'], (1, 2)), (['chapter', 'roman'], (13, 13)), (['roman'], (1, 2)), (['chapter', 'roman'], (14, 14)), (['roman'], (1, 3)), (['chapter', 'roman'], (15, 15)), (['roman'], (1, 3)), (['chapter', 'roman'], (16, 16)), (['roman'], (1, 4)), (['chapter', 'roman'], (17, 17)), (['roman'], (1, 2)), (['chapter', 'roman'], (18, 18)), (['roman'], (1, 4)), (['chapter', 'roman'], (19, 19)), (['roman'], (2, 2)), (['chapter', 'roman'], (20, 20)), (['roman'], (2, 4)), (['chapter', 'roman'], (21, 21)), (['roman'], (1, 2))]\n",
      "38 20717 [(['word'], (1, 1)), (['roman'], (1, 1)), (['chapter', 'roman'], (1, 6)), (['digit'], (7, 7)), (['chapter', 'roman'], (7, 17))]\n",
      "39 59254 [(['roman'], (2, 18)), (['chapter', 'roman'], (1, 18))]\n",
      "40 60067 [(['roman'], (1, 14)), (['chapter', 'roman'], (1, 14))]\n",
      "41 61507 [(['roman'], (1, 10)), (['chapter', 'roman'], (1, 10))]\n",
      "42 65172 [(['digit'], (1, 30))]\n",
      "43 65974 [(['digit'], (1, 10)), (['digit'], (1, 10))]\n",
      "44 67368 [(['roman'], (1, 29)), (['chapter', 'word'], (1, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20)), (['chapter', 'word'], (20, 20))]\n",
      "45 70041 [(['chapter', 'word'], (1, 1)), (['digit'], (1, 5)), (['chapter', 'word'], (2, 2)), (['digit'], (1, 4)), (['chapter', 'word'], (3, 3)), (['digit'], (1, 4)), (['chapter', 'word'], (4, 4)), (['digit'], (1, 5)), (['chapter', 'word'], (5, 5)), (['digit'], (1, 2)), (['chapter', 'word'], (6, 6)), (['digit'], (1, 5)), (['chapter', 'word'], (7, 14)), (['digit'], (1, 5)), (['chapter', 'word'], (15, 16)), (['digit'], (1, 5)), (['chapter', 'word'], (17, 17)), (['digit'], (1, 3)), (['chapter', 'word'], (18, 18))]\n",
      "46 70222 [(['digit'], (1, 9)), (['digit'], (1, 9))]\n",
      "47 72227 [(['roman'], (1, 9)), (['chapter', 'roman'], (1, 9))]\n"
     ]
    }
   ],
   "source": [
    "for idx, text in enumerate(texts):\n",
    "    matches = [extract_match_data(match) for match in CHAPTER_RE.finditer(text)]\n",
    "    print(\"%2s\" % idx, book_ids[idx], list(summarise_matches(matches)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4d9f214-d22f-4e6f-9fbe-57b2740ba5e3",
   "metadata": {},
   "source": [
    "As an example of one without chapter numbers, here is one that is a series of stories, not chapters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "1bd7ec60-468c-4caf-9fc6-0127dabd11bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A MAN OF MEANS\n",
      "\n",
      "A SERIES OF SIX STORIES\n",
      "\n",
      "\n",
      "By Pelham Grenville Wodehouse and C. H. Bovill\n",
      "\n",
      "From the _Pictorial Review_, May-October 1916\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "CONTENTS\n",
      "\n",
      "\n",
      "THE EPISODE OF THE LANDLADY'S DAUGHTER\n",
      "\n",
      "THE EPISODE OF THE FINANCIAL NAPOLEON\n",
      "\n",
      "THE EPISODE OF THE THEATRICAL VENTURE\n",
      "\n",
      "THE EPISODE OF THE LIVE WEEKLY\n",
      "\n",
      "THE DIVERTING EPISODE OF THE EXILED MONARCH\n",
      "\n",
      "THE EPISODE OF THE HIRED PAST\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "THE EPISODE OF THE LANDLADY'S DAUGHTER\n",
      "\n",
      "First of a Series of Six Stories [First publ\n"
     ]
    }
   ],
   "source": [
    "print(texts[32][:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e52bd47f-87b9-412b-9d96-936b4929d109",
   "metadata": {},
   "source": [
    "# Characters\n",
    "\n",
    "Let's have a look at the actual characters used in the texts, by the number of texts they occur in.\n",
    "There are some obscure characters near the end that occur rarely and will make language modelling very difficult."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a57291eb-366d-4c41-8f1d-04246cdb53b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "117\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dict_items([('\\n', 48), ('\\r', 48), (' ', 48), ('!', 48), ('\"', 33), ('$', 5), (\"'\", 34), ('(', 46), (')', 47), ('*', 37), (',', 48), ('-', 48), ('.', 48), ('0', 40), ('1', 48), ('2', 43), ('3', 34), ('5', 38), ('6', 39), ('7', 35), ('9', 44), (':', 48), (';', 48), ('?', 48), ('A', 48), ('B', 48), ('C', 48), ('D', 48), ('E', 48), ('F', 48), ('G', 48), ('H', 48), ('I', 48), ('J', 48), ('K', 48), ('L', 48), ('M', 48), ('N', 48), ('O', 48), ('P', 48), ('Q', 48), ('R', 48), ('S', 48), ('T', 48), ('U', 48), ('V', 48), ('W', 48), ('X', 37), ('Y', 48), ('Z', 35), ('_', 46), ('a', 48), ('b', 48), ('c', 48), ('d', 48), ('e', 48), ('f', 48), ('g', 48), ('h', 48), ('i', 48), ('j', 48), ('k', 48), ('l', 48), ('m', 48), ('n', 48), ('o', 48), ('p', 48), ('q', 48), ('r', 48), ('s', 48), ('t', 48), ('u', 48), ('v', 48), ('w', 48), ('x', 48), ('y', 48), ('z', 48), ('&', 19), ('8', 41), ('4', 37), ('à', 13), ('è', 12), ('é', 20), ('ê', 12), ('œ', 4), ('—', 8), ('‘', 14), ('’', 14), ('“', 15), ('”', 15), ('/', 6), ('[', 18), (']', 18), ('§', 4), ('ô', 8), ('…', 1), ('Æ', 2), ('æ', 8), ('#', 1), ('+', 2), ('=', 2), ('â', 5), ('ï', 3), ('<', 1), ('>', 1), ('£', 3), ('ö', 1), ('á', 2), ('ç', 4), ('·', 1), ('î', 1), ('É', 1), ('★', 1), ('Œ', 1), ('{', 1), ('}', 1), ('ë', 1)])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_counts_by_text = Counter(c for text in texts for c in sorted(set(text)))\n",
    "\n",
    "print(len(char_counts_by_text))\n",
    "char_counts_by_text.items()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6849dcdb-9f3a-4bfe-8f18-db67dbc7c1c9",
   "metadata": {},
   "source": [
    "We can have a look at how the characters are used; the `<` is used (incorrectly) as HTML markup and `{` in a chemical (?) formula."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "82b0d5ec-93e5-4255-bf7d-2c4d80c93494",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<      Some say he freed his country <i>so</i>,\n",
      "{ proving that H_{2}O+b3g4z7-m9z8=g6f5p3x. Before leaving London on what\n"
     ]
    }
   ],
   "source": [
    "for text in texts:\n",
    "    for line in text.splitlines():\n",
    "        for char in [\"{\", \"<\"]:\n",
    "            if char in line:\n",
    "                print(char, line)\n",
    "                break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59ab610a-3900-450b-8b4b-a4724d1d7071",
   "metadata": {},
   "source": [
    "We can look at a few more types of characters to see how they are used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "05f6e8f5-ffdb-488a-8b93-296704f653eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$ Stifien, $1.35 net, all rights of translation reserved, including\n",
      "$ FOR A PURSE OF $50 AND SIDE-BET\n",
      "/ goods supplied to the value of 25 pounds 1s. 8 1/2 d. was mere\n",
      "/ gentlemen, that our profits for the past year are 3,000,006-2-2 1/2\n",
      "$ him. “Shoes” ($213.50) he could understand, but what on earth was\n",
      "$ “Academy. Rehl. $105.50”? What was “Cuts … $15”? And what in the name\n",
      "$ “Friedmann, Samuel … Scenery … $3711” and “Unitt and Wickes … Scenery …\n",
      "$ $2120”). He was suffering the torments of the ruined gamester at the\n",
      "# amounted to #9 10s. I notched double figures in the next with #ll 1s.\n",
      "# 6d. Then I dropped to #7 0s. 6d. It was not starvation, but it was\n",
      "+ HATRED, denoted by x + Eva.\n",
      "+ REVERSE OF HATRED, “ “ y + Eva\n",
      "+ x + Eva = (y + Eva)z.\n",
      ">      Some say he freed his country <i>so</i>,\n",
      "£    “DEAR MR. BARLEY,” it ran.--“I send the £5, which I could not get\n",
      "£       HITHERTO UNKNOWN FUTURIST RECEIVED £2,000\n",
      "/ MEREDITH PRESS / NEW YORK\n",
      "$ \"Shoes\" ($213.50) he could understand, but what on earth was \"Academy.\n",
      "$ Rehl. $105.50\"? What was \"Cuts ... $15\"? And what in the name of\n",
      "$ \"Friedmann, Samuel ... Scenery ... $3711\" and \"Unitt and Wickes ...\n",
      "$ Scenery ... $2120\"). He was suffering the torments of the ruined\n",
      "· 3 York Street · St. James’s\n",
      "£ posts in his uncle’s business and £100 in bank-notes, coupled with the\n",
      "$ $3,000 which were the price of his promotion to detective-sergeant. He\n",
      "$ did not like paying $3,000 for promotion, but there must be sinking of\n",
      "$ $15,000 to spare for any small flutter that might take his fancy.\n",
      "£ £200,000 of good Dreever money, and the remainder of the family fortune\n",
      "£ He placed a £5 note on the table.\n",
      "/                      Set in 9/11pt Monotype Trump\n",
      "/ 73-7/8, caused shakings of the head. However, one rather liked the look\n",
      "/ of Royal Dutch Oil Ordinaries at 54-3/4.\n",
      "+ proving that H_{2}O+b3g4z7-m9z8=g6f5p3x. Before leaving London on what\n"
     ]
    }
   ],
   "source": [
    "for text in texts:\n",
    "    for line in text.splitlines():\n",
    "        for char in [\"+\", \"=\", \">\", \"<\", \"#\", \"£\", \"$\", \"/\", '·']:\n",
    "            if char in line:\n",
    "                print(char, line)\n",
    "                break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae3c45bb-d070-40ac-ae8a-1e5ad02fb069",
   "metadata": {},
   "source": [
    "We can clean these all up by mapping the characters to close equivalents in ASCII:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "91b7c45d-3bb8-4b9e-8784-8e1a9f0fe4c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "char_map = {\n",
    "    \"“\" : '\"',\n",
    "    \"”\": '\"',\n",
    "    \"’\": \"'\",\n",
    "    \"—\": \"--\",\n",
    "    \"…\": \"...\",\n",
    "    \"§\": \"Section \",\n",
    "    \"é\": \"e\",\n",
    "    \"ë\": \"e\",\n",
    "    \"★\": \"*\",\n",
    "    \"&\": \"and\",\n",
    "    \"à\": \"a\",\n",
    "    \"ô\": \"o\",\n",
    "    'è': \"e\",\n",
    "    'æ': \"ae\",\n",
    "    'ç': \"c\",\n",
    "    \"+\": \"plus\",\n",
    "    \"=\": \"equals\",\n",
    "    \"#\": \"number \",\n",
    "    \"/\": \" of \",\n",
    "    \"{\": \"\",\n",
    "    \"}\": \"\",\n",
    "    \"\\r\": \"\",\n",
    "    \"Œ\": \"Oe\",\n",
    "    \"É\": \"E\",\n",
    "    \"î\": \"i\",\n",
    "    '·': \"_\",\n",
    "    '‘': \"'\",\n",
    "    'ê': \"e\",\n",
    "    'â': \"a\",\n",
    "    'œ': \"oe\",\n",
    "    'ï': \"i\",\n",
    "    'Æ': \"Ae\",\n",
    "    'á': \"a\",\n",
    "    'ö': \"o\",\n",
    "    \"]\": \"\",\n",
    "    \"*\": \"\",\n",
    "}\n",
    "\n",
    "def normalise_text(text):\n",
    "    # Replace italics HTML tags with underscores\n",
    "    text = re.sub(\"</?i>\", \"_\", text)\n",
    "    # Remove square paranthetical comments [...]\n",
    "    text = re.sub(r\"\\[(.|\\n)*?\\]\", \"\", text, flags=re.MULTILINE)\n",
    "    # Dollar symbol to words (not perfect, but good enough)\n",
    "    # E.g. $5 note -> 5 dollars note\n",
    "    text = re.sub(r\"\\$([0-9.,]+)\\b\", r\"\\1 dollars\", text)\n",
    "    # Pound symbol to words\n",
    "    text = re.sub(r\"£([0-9.,]+)\\b\", r\"\\1 pounds\", text)\n",
    "    # Make more asii-ish\n",
    "    for chr_old, chr_new in char_map.items():\n",
    "        text = text.replace(chr_old, chr_new)\n",
    "    # Clean up superfluous whitespace\n",
    "    text = re.sub(\"^ *\", r\"\", text, flags=re.MULTILINE)\n",
    "    text = re.sub(\" *$\", r\"\", text, flags=re.MULTILINE)\n",
    "    #text = re.sub(\" +\", r\" \", text, flags=re.MULTILINE)\n",
    "    text = re.sub(\"\\n\\n+\", r\"\\n\\n\", text)\n",
    "    text = text.strip()\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6acb2f46-78a1-4aae-852d-9a0415db6f42",
   "metadata": {},
   "source": [
    "Now we've reduced to a list of 76 characters, all of which are used in many of the texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7c39d770-d774-43aa-8770-fe5c23c23448",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "76\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dict_items([('\\n', 48), (' ', 48), ('!', 48), ('\"', 48), (\"'\", 48), ('(', 46), (')', 47), (',', 48), ('-', 48), ('.', 48), ('0', 40), ('1', 48), ('2', 43), ('3', 34), ('5', 37), ('6', 39), ('7', 35), ('9', 44), (':', 48), (';', 48), ('?', 48), ('A', 48), ('B', 48), ('C', 48), ('D', 48), ('E', 48), ('F', 48), ('G', 48), ('H', 48), ('I', 48), ('J', 48), ('K', 48), ('L', 48), ('M', 48), ('N', 48), ('O', 48), ('P', 48), ('Q', 48), ('R', 48), ('S', 48), ('T', 48), ('U', 48), ('V', 48), ('W', 48), ('X', 37), ('Y', 48), ('Z', 35), ('_', 46), ('a', 48), ('b', 48), ('c', 48), ('d', 48), ('e', 48), ('f', 48), ('g', 48), ('h', 48), ('i', 48), ('j', 48), ('k', 48), ('l', 48), ('m', 48), ('n', 48), ('o', 48), ('p', 48), ('q', 48), ('r', 48), ('s', 48), ('t', 48), ('u', 48), ('v', 48), ('w', 48), ('x', 48), ('y', 48), ('z', 48), ('8', 40), ('4', 36)])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalised_texts = [normalise_text(t) for t in texts]\n",
    "char_counts_by_normalised_text = Counter(c for text in normalised_texts\n",
    "                                         for c in sorted(set(text)))\n",
    "\n",
    "print(len(char_counts_by_normalised_text))\n",
    "char_counts_by_normalised_text.items()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12291883-e177-40b7-a637-a43a310ff661",
   "metadata": {},
   "source": [
    "# Lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63b65a87-16b3-42a6-923a-0eec12c0944d",
   "metadata": {},
   "source": [
    "Most of the normalised texts are around 20k to 50k characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "3c9cd44f-7d27-40eb-9eb3-77b51aa8d748",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(79001, 343458.3958333333, 651807)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAduklEQVR4nO3de5CV5X3A8d8C7mFRdrnJwsblkmokAuIFpQST6rgTSonRTptYh7SUdJKYrFVKJgHSIGESs8RmHBpLMbETsROVmE4g1gspg1xqgshVIRqEispogKSGXUBdkX36R8YzriC3nn2Whc9n5sx43vfhvD+ewTnfOZfdspRSCgCATDq19wAAwOlFfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFZd2nuA92ppaYlXX301unfvHmVlZe09DgBwDFJKsXfv3qipqYlOnY782sZJFx+vvvpq1NbWtvcYAMAJ2LFjR5xzzjlHXHPSxUf37t0j4g/DV1ZWtvM0AMCxaGpqitra2uLz+JGcdPHxzlstlZWV4gMAOphj+ciED5wCAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AIKvjjo+VK1fGNddcEzU1NVFWVhaLFi0qnjtw4EBMnTo1hg8fHmeeeWbU1NTE3/zN38Srr75aypkBgA7suONj//79MWLEiJg7d+4h515//fVYv359zJgxI9avXx8//elPY8uWLfHJT36yJMMCAB1fWUopnfAfLiuLhQsXxnXXXfe+a9asWROXX355vPTSSzFgwICjPmZTU1NUVVVFY2OjXywHAB3E8Tx/t/lvtW1sbIyysrLo0aPHYc83NzdHc3Nz8X5TU1NbjwQAtKM2jY8333wzpk6dGjfccMP7VlBDQ0PMmjWrLceAVgZNe6S9R6CdvTh7fHuPAKe1Nvu2y4EDB+LTn/50pJRi3rx577tu+vTp0djYWLzt2LGjrUYCAE4CbfLKxzvh8dJLL8Xjjz9+xPd+CoVCFAqFthgDADgJlTw+3gmPrVu3xrJly6J3796lvgQA0IEdd3zs27cvtm3bVry/ffv22LhxY/Tq1Sv69+8ff/mXfxnr16+Phx9+OA4ePBg7d+6MiIhevXpFeXl56SYHADqk446PtWvXxlVXXVW8P2XKlIiImDhxYnzjG9+Ihx56KCIiLrroolZ/btmyZXHllVee+KQAwCnhuOPjyiuvjCP9aJD/x48NAQBOA363CwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKyOOz5WrlwZ11xzTdTU1ERZWVksWrSo1fmUUtx6663Rv3//qKioiLq6uti6dWup5gUAOrjjjo/9+/fHiBEjYu7cuYc9f/vtt8f3vve9uOuuu2L16tVx5plnxtixY+PNN9/8fw8LAHR8XY73D4wbNy7GjRt32HMppZgzZ058/etfj2uvvTYiIv793/89qqurY9GiRfFXf/VX/79pAYAOr6Sf+di+fXvs3Lkz6urqiseqqqpi1KhRsWrVqsP+mebm5mhqamp1AwBOXSWNj507d0ZERHV1davj1dXVxXPv1dDQEFVVVcVbbW1tKUcCAE4y7f5tl+nTp0djY2PxtmPHjvYeCQBoQyWNj379+kVExK5du1od37VrV/HcexUKhaisrGx1AwBOXSWNj8GDB0e/fv1i6dKlxWNNTU2xevXqGD16dCkvBQB0UMf9bZd9+/bFtm3bive3b98eGzdujF69esWAAQNi8uTJ8a1vfSvOO++8GDx4cMyYMSNqamriuuuuK+XcAEAHddzxsXbt2rjqqquK96dMmRIRERMnToz58+fHV7/61di/f398/vOfjz179sQVV1wRixcvjq5du5ZuagCgwypLKaX2HuLdmpqaoqqqKhobG33+gzYxaNoj7T0C7ezF2ePbewQ45RzP83e7f9sFADi9iA8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAsurS3gMA5DZo2iPtPUK7enH2+PYegdOcVz4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkVfL4OHjwYMyYMSMGDx4cFRUV8Ud/9EfxzW9+M1JKpb4UANABdSn1A37nO9+JefPmxb333htDhw6NtWvXxqRJk6KqqipuvvnmUl8OAOhgSh4fv/zlL+Paa6+N8ePHR0TEoEGD4oEHHoinnnqq1JcCADqgkr/t8pGPfCSWLl0azz//fEREPP300/HEE0/EuHHjSn0pAKADKvkrH9OmTYumpqYYMmRIdO7cOQ4ePBi33XZbTJgw4bDrm5ubo7m5uXi/qamp1CMBACeRkr/y8eCDD8Z9990X999/f6xfvz7uvffe+O53vxv33nvvYdc3NDREVVVV8VZbW1vqkQCAk0hZKvHXUGpra2PatGlRX19fPPatb30rfvSjH8Wvf/3rQ9Yf7pWP2traaGxsjMrKylKOBhERMWjaI+09ArSrF2ePb+8ROAU1NTVFVVXVMT1/l/xtl9dffz06dWr9gkrnzp2jpaXlsOsLhUIUCoVSjwEAnKRKHh/XXHNN3HbbbTFgwIAYOnRobNiwIe6444747Gc/W+pLAQAdUMnj484774wZM2bEl770pdi9e3fU1NTEF77whbj11ltLfSkAoAMqeXx079495syZE3PmzCn1QwMApwC/2wUAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBWbRIfr7zySnzmM5+J3r17R0VFRQwfPjzWrl3bFpcCADqYLqV+wN///vcxZsyYuOqqq+Kxxx6Ls88+O7Zu3Ro9e/Ys9aUAgA6o5PHxne98J2pra+Oee+4pHhs8eHCpLwMAdFAlf9vloYceipEjR8anPvWp6Nu3b1x88cVx9913v+/65ubmaGpqanUDAE5dJX/l44UXXoh58+bFlClT4mtf+1qsWbMmbr755igvL4+JEycesr6hoSFmzZpV6jF4H4OmPdLeIwBwmitLKaVSPmB5eXmMHDkyfvnLXxaP3XzzzbFmzZpYtWrVIeubm5ujubm5eL+pqSlqa2ujsbExKisrSzkaIT6AiBdnj2/vETgFNTU1RVVV1TE9f5f8bZf+/fvHBRdc0OrYhz/84Xj55ZcPu75QKERlZWWrGwBw6ip5fIwZMya2bNnS6tjzzz8fAwcOLPWlAIAOqOTx8Q//8A/x5JNPxre//e3Ytm1b3H///fGDH/wg6uvrS30pAKADKnl8XHbZZbFw4cJ44IEHYtiwYfHNb34z5syZExMmTCj1pQCADqjk33aJiPjEJz4Rn/jEJ9rioQGADs7vdgEAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJBVm8fH7Nmzo6ysLCZPntzWlwIAOoA2jY81a9bE97///bjwwgvb8jIAQAfSZvGxb9++mDBhQtx9993Rs2fPtroMANDBtFl81NfXx/jx46Ouru6I65qbm6OpqanVDQA4dXVpiwddsGBBrF+/PtasWXPUtQ0NDTFr1qy2GOOwBk17JNu1AIBDlfyVjx07dsQtt9wS9913X3Tt2vWo66dPnx6NjY3F244dO0o9EgBwEin5Kx/r1q2L3bt3xyWXXFI8dvDgwVi5cmX8y7/8SzQ3N0fnzp2L5wqFQhQKhVKPAQCcpEoeH1dffXVs2rSp1bFJkybFkCFDYurUqa3CAwA4/ZQ8Prp37x7Dhg1rdezMM8+M3r17H3IcADj9+AmnAEBWbfJtl/davnx5jssAAB2AVz4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZdWnvAQAgp0HTHmnvEdrdi7PHt+v1vfIBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgq5LHR0NDQ1x22WXRvXv36Nu3b1x33XWxZcuWUl8GAOigSh4fK1asiPr6+njyySdjyZIlceDAgfj4xz8e+/fvL/WlAIAOqEupH3Dx4sWt7s+fPz/69u0b69ati4997GOlvhwA0MG0+Wc+GhsbIyKiV69ebX0pAKADKPkrH+/W0tISkydPjjFjxsSwYcMOu6a5uTmam5uL95uamtpyJACgnbVpfNTX18fmzZvjiSeeeN81DQ0NMWvWrLYcA4B3GTTtkfYegdNcm73tctNNN8XDDz8cy5Yti3POOed9102fPj0aGxuLtx07drTVSADASaDkr3yklOLv//7vY+HChbF8+fIYPHjwEdcXCoUoFAqlHgMAOEmVPD7q6+vj/vvvj5/97GfRvXv32LlzZ0REVFVVRUVFRakvBwB0MCV/22XevHnR2NgYV155ZfTv3794+/GPf1zqSwEAHVCbvO0CAPB+/G4XACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWbVZfMydOzcGDRoUXbt2jVGjRsVTTz3VVpcCADqQNomPH//4xzFlypSYOXNmrF+/PkaMGBFjx46N3bt3t8XlAIAOpE3i44477ojPfe5zMWnSpLjgggvirrvuim7dusUPf/jDtrgcANCBdCn1A7711luxbt26mD59evFYp06doq6uLlatWnXI+ubm5mhubi7eb2xsjIiIpqamUo8WEREtza+3yeMCQEfRFs+x7zxmSumoa0seH7/73e/i4MGDUV1d3ep4dXV1/PrXvz5kfUNDQ8yaNeuQ47W1taUeDQCIiKo5bffYe/fujaqqqiOuKXl8HK/p06fHlClTivdbWlritddei969e0dZWVk7TnaopqamqK2tjR07dkRlZWV7j9Oh2LsTZ+9OnL07MfbtxJ3Oe5dSir1790ZNTc1R15Y8Pvr06ROdO3eOXbt2tTq+a9eu6Nev3yHrC4VCFAqFVsd69OhR6rFKqrKy8rT7R1Uq9u7E2bsTZ+9OjH07cafr3h3tFY93lPwDp+Xl5XHppZfG0qVLi8daWlpi6dKlMXr06FJfDgDoYNrkbZcpU6bExIkTY+TIkXH55ZfHnDlzYv/+/TFp0qS2uBwA0IG0SXxcf/318dvf/jZuvfXW2LlzZ1x00UWxePHiQz6E2tEUCoWYOXPmIW8TcXT27sTZuxNn706MfTtx9u7YlKVj+U4MAECJ+N0uAEBW4gMAyEp8AABZiQ8AIKtTLj5WrlwZ11xzTdTU1ERZWVksWrSo1fmUUtx6663Rv3//qKioiLq6uti6dWurNa+99lpMmDAhKisro0ePHvF3f/d3sW/fvlZrnnnmmfjoRz8aXbt2jdra2rj99tsPmeUnP/lJDBkyJLp27RrDhw+PRx999LhnyaWhoSEuu+yy6N69e/Tt2zeuu+662LJlS6s1b775ZtTX10fv3r3jrLPOir/4i7845IfJvfzyyzF+/Pjo1q1b9O3bN77yla/E22+/3WrN8uXL45JLLolCoRDnnntuzJ8//5B55s6dG4MGDYquXbvGqFGj4qmnnjruWXKZN29eXHjhhcUfKjR69Oh47LHHjmvW03Hf3mv27NlRVlYWkydPLh6zd4f3jW98I8rKylrdhgwZclyzno779o5XXnklPvOZz0Tv3r2joqIihg8fHmvXri2e9zyRQTrFPProo+kf//Ef009/+tMUEWnhwoWtzs+ePTtVVVWlRYsWpaeffjp98pOfTIMHD05vvPFGcc2f/umfphEjRqQnn3wy/fd//3c699xz0w033FA839jYmKqrq9OECRPS5s2b0wMPPJAqKirS97///eKaX/ziF6lz587p9ttvT88++2z6+te/ns4444y0adOm45oll7Fjx6Z77rknbd68OW3cuDH92Z/9WRowYEDat29fcc2NN96Yamtr09KlS9PatWvTH//xH6ePfOQjxfNvv/12GjZsWKqrq0sbNmxIjz76aOrTp0+aPn16cc0LL7yQunXrlqZMmZKeffbZdOedd6bOnTunxYsXF9csWLAglZeXpx/+8IfpV7/6Vfrc5z6XevTokXbt2nXMs+T00EMPpUceeSQ9//zzacuWLelrX/taOuOMM9LmzZuPadbTdd/e7amnnkqDBg1KF154YbrllluKx+3d4c2cOTMNHTo0/eY3vynefvvb3x7zrKfrvqWU0muvvZYGDhyY/vZv/zatXr06vfDCC+nnP/952rZtW3GN54m2d8rFx7u9Nz5aWlpSv3790j/90z8Vj+3ZsycVCoX0wAMPpJRSevbZZ1NEpDVr1hTXPPbYY6msrCy98sorKaWU/vVf/zX17NkzNTc3F9dMnTo1nX/++cX7n/70p9P48eNbzTNq1Kj0hS984ZhnaU+7d+9OEZFWrFiRUvrDbGeccUb6yU9+Ulzz3HPPpYhIq1atSin9Ifw6deqUdu7cWVwzb968VFlZWdyrr371q2no0KGtrnX99densWPHFu9ffvnlqb6+vnj/4MGDqaamJjU0NBzzLO2tZ8+e6d/+7d/s2zHYu3dvOu+889KSJUvSn/zJnxTjw969v5kzZ6YRI0Yc9px9O7KpU6emK6644n3Pe57I45R72+VItm/fHjt37oy6urrisaqqqhg1alSsWrUqIiJWrVoVPXr0iJEjRxbX1NXVRadOnWL16tXFNR/72MeivLy8uGbs2LGxZcuW+P3vf19c8+7rvLPmnescyyztqbGxMSIievXqFRER69atiwMHDrSad8iQITFgwIBWezd8+PBWP0xu7Nix0dTUFL/61a+Ka460L2+99VasW7eu1ZpOnTpFXV1dcc2xzNJeDh48GAsWLIj9+/fH6NGj7dsxqK+vj/Hjxx/y97N3R7Z169aoqamJD37wgzFhwoR4+eWXj3nW03nfHnrooRg5cmR86lOfir59+8bFF18cd999d/G854k8Tqv42LlzZ0TEIT9ptbq6unhu586d0bdv31bnu3TpEr169Wq15nCP8e5rvN+ad58/2iztpaWlJSZPnhxjxoyJYcOGRcQf5i0vLz/kl/699+90ovvS1NQUb7zxRvzud7+LgwcPHnXvjjZLbps2bYqzzjorCoVC3HjjjbFw4cK44IIL7NtRLFiwINavXx8NDQ2HnLN372/UqFExf/78WLx4ccybNy+2b98eH/3oR2Pv3r327SheeOGFmDdvXpx33nnx85//PL74xS/GzTffHPfee29EeJ7IpU1+vDodW319fWzevDmeeOKJ9h6lwzj//PNj48aN0djYGP/xH/8REydOjBUrVrT3WCe1HTt2xC233BJLliyJrl27tvc4Hcq4ceOK/33hhRfGqFGjYuDAgfHggw9GRUVFO0528mtpaYmRI0fGt7/97YiIuPjii2Pz5s1x1113xcSJE9t5utPHafXKR79+/SIiDvmk9a5du4rn+vXrF7t37251/u23347XXnut1ZrDPca7r/F+a959/miztIebbropHn744Vi2bFmcc845xeP9+vWLt956K/bs2dNq/Xv/Tie6L5WVlVFRURF9+vSJzp07H3XvjjZLbuXl5XHuuefGpZdeGg0NDTFixIj453/+Z/t2BOvWrYvdu3fHJZdcEl26dIkuXbrEihUr4nvf+1506dIlqqur7d0x6tGjR3zoQx+Kbdu2+Td3FP37948LLrig1bEPf/jDxbetPE/kcVrFx+DBg6Nfv36xdOnS4rGmpqZYvXp1jB49OiIiRo8eHXv27Il169YV1zz++OPR0tISo0aNKq5ZuXJlHDhwoLhmyZIlcf7550fPnj2La959nXfWvHOdY5klp5RS3HTTTbFw4cJ4/PHHY/Dgwa3OX3rppXHGGWe0mnfLli3x8ssvt9q7TZs2tfqfcsmSJVFZWVn8n/1o+1JeXh6XXnppqzUtLS2xdOnS4ppjmaW9tbS0RHNzs307gquvvjo2bdoUGzduLN5GjhwZEyZMKP63vTs2+/bti//5n/+J/v37+zd3FGPGjDnkxwg8//zzMXDgwIjwPJFNe3/itdT27t2bNmzYkDZs2JAiIt1xxx1pw4YN6aWXXkop/eFrSz169Eg/+9nP0jPPPJOuvfbaw36F6uKLL06rV69OTzzxRDrvvPNafYVqz549qbq6Ov31X/912rx5c1qwYEHq1q3bIV+h6tKlS/rud7+bnnvuuTRz5szDfoXqaLPk8sUvfjFVVVWl5cuXt/r63uuvv15cc+ONN6YBAwakxx9/PK1duzaNHj06jR49unj+na/vffzjH08bN25MixcvTmefffZhv773la98JT333HNp7ty5h/36XqFQSPPnz0/PPvts+vznP5969OjR6pP5R5slp2nTpqUVK1ak7du3p2eeeSZNmzYtlZWVpf/6r/86pllP1307nHd/2yUle/d+vvzlL6fly5en7du3p1/84heprq4u9enTJ+3evfuYZj1d9y2lP3ytu0uXLum2225LW7duTffdd1/q1q1b+tGPflRc43mi7Z1y8bFs2bIUEYfcJk6cmFL6w1eXZsyYkaqrq1OhUEhXX3112rJlS6vH+N///d90ww03pLPOOitVVlamSZMmpb1797Za8/TTT6crrrgiFQqF9IEPfCDNnj37kFkefPDB9KEPfSiVl5enoUOHpkceeaTV+WOZJZfD7VlEpHvuuae45o033khf+tKXUs+ePVO3bt3Sn//5n6ff/OY3rR7nxRdfTOPGjUsVFRWpT58+6ctf/nI6cOBAqzXLli1LF110USovL08f/OAHW13jHXfeeWcaMGBAKi8vT5dffnl68sknW50/llly+exnP5sGDhyYysvL09lnn52uvvrqYngc66yn474dznvjw94d3vXXX5/69++fysvL0wc+8IF0/fXXt/o5FfbtyP7zP/8zDRs2LBUKhTRkyJD0gx/8oNV5zxNtryyllNrnNRcA4HR0Wn3mAwBof+IDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgq/8D6hSfwygY9K4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "text_num_chars = [len(text) for text in normalised_texts]\n",
    "plt.hist(text_num_chars, bins=7);\n",
    "min(text_num_chars), sum(text_num_chars) / len(text_num_chars), max(text_num_chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bab51698-7af1-4ed3-b5c7-067fa4d75544",
   "metadata": {},
   "source": [
    "Defining a word using space separation we get close to 5.5 characters per word for all of the texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "c7df0005-8b4a-4fc7-b684-9ccc8e8160cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5.370687955844668, 5.565977554542365, 5.841355854038162)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAhoklEQVR4nO3de3BU9f3/8ddCYGOd7KIScsGVCAJRLsGixCBUGKIhZRBoi5CiBAWcKlg1Yk0sctM2tFbHOjDYWjF2LIJ0MDiCUYgK0nApYEZgaiQxIaBsECS7JI4JJuf3hz+33S9JYHE3+8nm+Zg5M549l7z3mDFPd0+yNsuyLAEAABisS7gHAAAAOB+CBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxosI9QDA0Nzfriy++UExMjGw2W7jHAQAAF8CyLJ05c0aJiYnq0qXt11AiIli++OILuVyucI8BAAAuwtGjR3XllVe2uU9EBEtMTIyk756ww+EI8zQAAOBCeL1euVwu38/xtkREsHz/NpDD4SBYAADoYC7kdg5uugUAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPGiwj0AAPMl5W4K9whhV7V8QrhHADo1XmEBAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYLOFi2b9+uiRMnKjExUTabTYWFhX7bbTZbi8vTTz/d6jmXLFlyzv7JyckBPxkAABCZAg6W+vp6paSkaOXKlS1uP378uN+yevVq2Ww2/fznP2/zvIMGDfI7bseOHYGOBgAAIlTAf4clMzNTmZmZrW6Pj4/3W9+4caPGjh2rvn37tj1IVNQ5xwIAAEghvoelpqZGmzZt0uzZs8+77+HDh5WYmKi+fftqxowZqq6ubnXfhoYGeb1evwUAAESukAbLK6+8opiYGP3sZz9rc7/U1FQVFBSoqKhIq1atUmVlpUaPHq0zZ860uH9+fr6cTqdvcblcoRgfAAAYIqTBsnr1as2YMUPR0dFt7peZmampU6dq6NChysjI0ObNm1VbW6vXX3+9xf3z8vLk8Xh8y9GjR0MxPgAAMETIPkvoww8/VFlZmdatWxfwsT169NCAAQNUXl7e4na73S673f5DRwQAAB1EyF5heemllzR8+HClpKQEfGxdXZ0qKiqUkJAQgskAAEBHE3Cw1NXVqbS0VKWlpZKkyspKlZaW+t0k6/V6tX79es2ZM6fFc4wbN04rVqzwrS9YsEDbtm1TVVWVSkpKNGXKFHXt2lVZWVmBjgcAACJQwG8J7d27V2PHjvWt5+TkSJKys7NVUFAgSVq7dq0sy2o1OCoqKnTy5Enf+rFjx5SVlaVTp04pNjZWo0aN0q5duxQbGxvoeAAAIALZLMuywj3ED+X1euV0OuXxeORwOMI9DhBxknI3hXuEsKtaPiHcIwARJ5Cf33yWEAAAMB7BAgAAjEewAAAA4xEsAADAeAQLAAAwHsECAACMR7AAAADjESwAAMB4BAsAADAewQIAAIxHsAAAAOMRLAAAwHgECwAAMB7BAgAAjEewAAAA4xEsAADAeAQLAAAwHsECAACMR7AAAADjESwAAMB4BAsAADAewQIAAIxHsAAAAOMRLAAAwHgECwAAMB7BAgAAjEewAAAA4xEsAADAeAQLAAAwHsECAACMR7AAAADjESwAAMB4BAsAADAewQIAAIxHsAAAAOMRLAAAwHgECwAAMB7BAgAAjBdwsGzfvl0TJ05UYmKibDabCgsL/bbPmjVLNpvNbxk/fvx5z7ty5UolJSUpOjpaqamp2rNnT6CjAQCACBVwsNTX1yslJUUrV65sdZ/x48fr+PHjvuW1115r85zr1q1TTk6OFi9erP379yslJUUZGRk6ceJEoOMBAIAIFBXoAZmZmcrMzGxzH7vdrvj4+As+57PPPqu5c+fq7rvvliS98MIL2rRpk1avXq3c3NxARwQAABEmJPewfPDBB+rVq5cGDhyo++67T6dOnWp138bGRu3bt0/p6en/HapLF6Wnp2vnzp0tHtPQ0CCv1+u3AACAyBX0YBk/frz+/ve/q7i4WH/4wx+0bds2ZWZmqqmpqcX9T548qaamJsXFxfk9HhcXJ7fb3eIx+fn5cjqdvsXlcgX7aQAAAIME/JbQ+UyfPt33z0OGDNHQoUPVr18/ffDBBxo3blxQvkZeXp5ycnJ8616vl2gBACCChfzXmvv27auePXuqvLy8xe09e/ZU165dVVNT4/d4TU1Nq/fB2O12ORwOvwUAAESukAfLsWPHdOrUKSUkJLS4vXv37ho+fLiKi4t9jzU3N6u4uFhpaWmhHg8AAHQAAQdLXV2dSktLVVpaKkmqrKxUaWmpqqurVVdXp0cffVS7du1SVVWViouLNWnSJF1zzTXKyMjwnWPcuHFasWKFbz0nJ0cvvviiXnnlFf3nP//Rfffdp/r6et9vDQEAgM4t4HtY9u7dq7Fjx/rWv7+XJDs7W6tWrdLHH3+sV155RbW1tUpMTNRtt92mJ598Una73XdMRUWFTp486VufNm2avvzySy1atEhut1vDhg1TUVHROTfiAgCAzslmWZYV7iF+KK/XK6fTKY/Hw/0sQAgk5W4K9whhV7V8QrhHACJOID+/+SwhAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYL+Bg2b59uyZOnKjExETZbDYVFhb6tp09e1aPPfaYhgwZoksvvVSJiYmaOXOmvvjiizbPuWTJEtlsNr8lOTk54CcDAAAiU8DBUl9fr5SUFK1cufKcbV9//bX279+vJ554Qvv379eGDRtUVlam22+//bznHTRokI4fP+5bduzYEehoAAAgQkUFekBmZqYyMzNb3OZ0OrVlyxa/x1asWKERI0aourpaV111VeuDREUpPj4+0HEAAEAnEPJ7WDwej2w2m3r06NHmfocPH1ZiYqL69u2rGTNmqLq6utV9Gxoa5PV6/RYAABC5Qhos33zzjR577DFlZWXJ4XC0ul9qaqoKCgpUVFSkVatWqbKyUqNHj9aZM2da3D8/P19Op9O3uFyuUD0FAABggJAFy9mzZ3XHHXfIsiytWrWqzX0zMzM1depUDR06VBkZGdq8ebNqa2v1+uuvt7h/Xl6ePB6Pbzl69GgongIAADBEwPewXIjvY+XIkSN677332nx1pSU9evTQgAEDVF5e3uJ2u90uu90ejFEBAEAHEPRXWL6PlcOHD2vr1q264oorAj5HXV2dKioqlJCQEOzxAABABxRwsNTV1am0tFSlpaWSpMrKSpWWlqq6ulpnz57VL37xC+3du1f/+Mc/1NTUJLfbLbfbrcbGRt85xo0bpxUrVvjWFyxYoG3btqmqqkolJSWaMmWKunbtqqysrB/+DAEAQIcX8FtCe/fu1dixY33rOTk5kqTs7GwtWbJEb775piRp2LBhfse9//77GjNmjCSpoqJCJ0+e9G07duyYsrKydOrUKcXGxmrUqFHatWuXYmNjAx0PAABEoICDZcyYMbIsq9XtbW37XlVVld/62rVrAx0DAAB0InyWEAAAMB7BAgAAjEewAAAA4xEsAADAeAQLAAAwHsECAACMR7AAAADjESwAAMB4BAsAADAewQIAAIxHsAAAAOMRLAAAwHgECwAAMB7BAgAAjEewAAAA4xEsAADAeAQLAAAwHsECAACMR7AAAADjESwAAMB4BAsAADAewQIAAIwXFe4BgI4gKXdTuEcAgE6NV1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABgv4GDZvn27Jk6cqMTERNlsNhUWFvpttyxLixYtUkJCgi655BKlp6fr8OHD5z3vypUrlZSUpOjoaKWmpmrPnj2BjgYAACJUwMFSX1+vlJQUrVy5ssXtf/zjH/X888/rhRde0O7du3XppZcqIyND33zzTavnXLdunXJycrR48WLt379fKSkpysjI0IkTJwIdDwAARCCbZVnWRR9ss+mNN97Q5MmTJX336kpiYqIeeeQRLViwQJLk8XgUFxengoICTZ8+vcXzpKam6sYbb9SKFSskSc3NzXK5XHrggQeUm5t73jm8Xq+cTqc8Ho8cDsfFPh2gVUm5m8I9AsKsavmEcI8ARJxAfn4H9R6WyspKud1upaen+x5zOp1KTU3Vzp07WzymsbFR+/bt8zumS5cuSk9Pb/WYhoYGeb1evwUAAESuoAaL2+2WJMXFxfk9HhcX59v2f508eVJNTU0BHZOfny+n0+lbXC5XEKYHAACm6pC/JZSXlyePx+Nbjh49Gu6RAABACAU1WOLj4yVJNTU1fo/X1NT4tv1fPXv2VNeuXQM6xm63y+Fw+C0AACByBTVYrr76asXHx6u4uNj3mNfr1e7du5WWltbiMd27d9fw4cP9jmlublZxcXGrxwAAgM4lKtAD6urqVF5e7luvrKxUaWmpLr/8cl111VV66KGH9NRTT6l///66+uqr9cQTTygxMdH3m0SSNG7cOE2ZMkXz58+XJOXk5Cg7O1s33HCDRowYoeeee0719fW6++67f/gzBAAAHV7AwbJ3716NHTvWt56TkyNJys7OVkFBgX7zm9+ovr5e9957r2prazVq1CgVFRUpOjrad0xFRYVOnjzpW582bZq+/PJLLVq0SG63W8OGDVNRUdE5N+ICAIDO6Qf9HRZT8HdYEGr8HRbwd1iA4Avb32EBAAAIBYIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QL+tGZ0PnzwHwAg3HiFBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYLygB0tSUpJsNts5y7x581rcv6Cg4Jx9o6Ojgz0WAADowKKCfcJ///vfampq8q0fPHhQt956q6ZOndrqMQ6HQ2VlZb51m80W7LEAAEAHFvRgiY2N9Vtfvny5+vXrp1tuuaXVY2w2m+Lj44M9CgAAiBAhvYelsbFRr776qu655542XzWpq6tTnz595HK5NGnSJB06dCiUYwEAgA4mpMFSWFio2tpazZo1q9V9Bg4cqNWrV2vjxo169dVX1dzcrJEjR+rYsWOtHtPQ0CCv1+u3AACAyBXSYHnppZeUmZmpxMTEVvdJS0vTzJkzNWzYMN1yyy3asGGDYmNj9Ze//KXVY/Lz8+V0On2Ly+UKxfgAAMAQIQuWI0eOaOvWrZozZ05Ax3Xr1k3XX3+9ysvLW90nLy9PHo/Htxw9evSHjgsAAAwWsmB5+eWX1atXL02YMCGg45qamnTgwAElJCS0uo/dbpfD4fBbAABA5ApJsDQ3N+vll19Wdna2oqL8fxFp5syZysvL860vW7ZM7777rj777DPt379fd955p44cORLwKzMAACByBf3XmiVp69atqq6u1j333HPOturqanXp8t9OOn36tObOnSu3263LLrtMw4cPV0lJia677rpQjAYAADogm2VZVriH+KG8Xq+cTqc8Hg9vD4VAUu6mcI8AhF3V8sDe3gZwfoH8/OazhAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYLygB8uSJUtks9n8luTk5DaPWb9+vZKTkxUdHa0hQ4Zo8+bNwR4LAAB0YCF5hWXQoEE6fvy4b9mxY0er+5aUlCgrK0uzZ8/WRx99pMmTJ2vy5Mk6ePBgKEYDAAAdUEiCJSoqSvHx8b6lZ8+ere775z//WePHj9ejjz6qa6+9Vk8++aR+/OMfa8WKFaEYDQAAdEAhCZbDhw8rMTFRffv21YwZM1RdXd3qvjt37lR6errfYxkZGdq5c2coRgMAAB1QVLBPmJqaqoKCAg0cOFDHjx/X0qVLNXr0aB08eFAxMTHn7O92uxUXF+f3WFxcnNxud6tfo6GhQQ0NDb51r9cbvCcAAACME/RgyczM9P3z0KFDlZqaqj59+uj111/X7Nmzg/I18vPztXTp0qCcCwAuRFLupnCPEFZVyyeEewR0ciH/teYePXpowIABKi8vb3F7fHy8ampq/B6rqalRfHx8q+fMy8uTx+PxLUePHg3qzAAAwCwhD5a6ujpVVFQoISGhxe1paWkqLi72e2zLli1KS0tr9Zx2u10Oh8NvAQAAkSvowbJgwQJt27ZNVVVVKikp0ZQpU9S1a1dlZWVJkmbOnKm8vDzf/g8++KCKior0zDPP6JNPPtGSJUu0d+9ezZ8/P9ijAQCADiro97AcO3ZMWVlZOnXqlGJjYzVq1Cjt2rVLsbGxkqTq6mp16fLfTho5cqTWrFmjhQsX6vHHH1f//v1VWFiowYMHB3s0AADQQdksy7LCPcQP5fV65XQ65fF4eHsoBDr7zYYAuOkWoRHIz28+SwgAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYLyrcA3QESbmbwj0CAACdGq+wAAAA4xEsAADAeAQLAAAwHsECAACMF/Rgyc/P14033qiYmBj16tVLkydPVllZWZvHFBQUyGaz+S3R0dHBHg0AAHRQQQ+Wbdu2ad68edq1a5e2bNmis2fP6rbbblN9fX2bxzkcDh0/fty3HDlyJNijAQCADirov9ZcVFTkt15QUKBevXpp3759+slPftLqcTabTfHx8cEeBwAARICQ38Pi8XgkSZdffnmb+9XV1alPnz5yuVyaNGmSDh061Oq+DQ0N8nq9fgsAAIhcIQ2W5uZmPfTQQ7r55ps1ePDgVvcbOHCgVq9erY0bN+rVV19Vc3OzRo4cqWPHjrW4f35+vpxOp29xuVyhegoAAMAANsuyrFCd/L777tPbb7+tHTt26Morr7zg486ePatrr71WWVlZevLJJ8/Z3tDQoIaGBt+61+uVy+WSx+ORw+EIyuz/i790C6Czq1o+IdwjIAJ5vV45nc4L+vkdsj/NP3/+fL311lvavn17QLEiSd26ddP111+v8vLyFrfb7XbZ7fZgjAkAADqAoL8lZFmW5s+frzfeeEPvvfeerr766oDP0dTUpAMHDighISHY4wEAgA4o6K+wzJs3T2vWrNHGjRsVExMjt9stSXI6nbrkkkskSTNnzlTv3r2Vn58vSVq2bJluuukmXXPNNaqtrdXTTz+tI0eOaM6cOcEeDwAAdEBBD5ZVq1ZJksaMGeP3+Msvv6xZs2ZJkqqrq9Wly39f3Dl9+rTmzp0rt9utyy67TMOHD1dJSYmuu+66YI8HAAA6oJDedNteArlp52Jw0y2Azo6bbhEKgfz85rOEAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGC8q3AMAAMyXlLsp3CMgzKqWTwjr1+cVFgAAYDyCBQAAGI9gAQAAxiNYAACA8QgWAABgPIIFAAAYj2ABAADGI1gAAIDxCBYAAGA8ggUAABiPYAEAAMYjWAAAgPEIFgAAYDyCBQAAGC9kwbJy5UolJSUpOjpaqamp2rNnT5v7r1+/XsnJyYqOjtaQIUO0efPmUI0GAAA6mJAEy7p165STk6PFixdr//79SklJUUZGhk6cONHi/iUlJcrKytLs2bP10UcfafLkyZo8ebIOHjwYivEAAEAHY7Msywr2SVNTU3XjjTdqxYoVkqTm5ma5XC498MADys3NPWf/adOmqb6+Xm+99ZbvsZtuuknDhg3TCy+8cN6v5/V65XQ65fF45HA4gvdE/r+k3E1BPycAAB1J1fIJQT9nID+/o4L9xRsbG7Vv3z7l5eX5HuvSpYvS09O1c+fOFo/ZuXOncnJy/B7LyMhQYWFhi/s3NDSooaHBt+7xeCR998RDobnh65CcFwCAjiIUP2O/P+eFvHYS9GA5efKkmpqaFBcX5/d4XFycPvnkkxaPcbvdLe7vdrtb3D8/P19Lly4953GXy3WRUwMAgLY4nwvduc+cOSOn09nmPkEPlvaQl5fn94pMc3OzvvrqK11xxRWy2WxhnOyH83q9crlcOnr0aEje3kLbuP7hxfUPL65/+HTWa29Zls6cOaPExMTz7hv0YOnZs6e6du2qmpoav8dramoUHx/f4jHx8fEB7W+322W32/0e69Gjx8UPbSCHw9GpvmlNw/UPL65/eHH9w6czXvvzvbLyvaD/llD37t01fPhwFRcX+x5rbm5WcXGx0tLSWjwmLS3Nb39J2rJlS6v7AwCAziUkbwnl5OQoOztbN9xwg0aMGKHnnntO9fX1uvvuuyVJM2fOVO/evZWfny9JevDBB3XLLbfomWee0YQJE7R27Vrt3btXf/3rX0MxHgAA6GBCEizTpk3Tl19+qUWLFsntdmvYsGEqKiry3VhbXV2tLl3+++LOyJEjtWbNGi1cuFCPP/64+vfvr8LCQg0ePDgU4xnNbrdr8eLF57zlhfbB9Q8vrn94cf3Dh2t/fiH5OywAAADBxGcJAQAA4xEsAADAeAQLAAAwHsECAACMR7C0oyVLlshms/ktycnJF3Ts2rVrZbPZNHny5NAOGcECvf4FBQXn7B8dHd2OE0eWi/n+r62t1bx585SQkCC73a4BAwZo8+bN7TRxZAn0+o8ZM+ac/W02myZMCP4H4EW6i/nef+655zRw4EBdcsklcrlcevjhh/XNN9+008Rm6pB/mr8jGzRokLZu3epbj4o6/7+CqqoqLViwQKNHjw7laJ1CoNff4XCorKzMt97RP/oh3AK5/o2Njbr11lvVq1cv/fOf/1Tv3r115MiRiPur1u0pkOu/YcMGNTY2+tZPnTqllJQUTZ06NaQzRqpArv2aNWuUm5ur1atXa+TIkfr00081a9Ys2Ww2Pfvss+0xrpEIlnYWFRXV6kcOtKSpqUkzZszQ0qVL9eGHH6q2tjZ0w3UCgV5/m80W0P5oWyDXf/Xq1frqq69UUlKibt26SZKSkpJCOF3kC+T6X3755X7ra9eu1Y9+9COC5SIFcu1LSkp0880365e//KWk777vs7KytHv37lCOaDzeEmpnhw8fVmJiovr27asZM2aourq6zf2XLVumXr16afbs2e00YWQL9PrX1dWpT58+crlcmjRpkg4dOtROk0amQK7/m2++qbS0NM2bN09xcXEaPHiwfv/736upqakdJ44sgX7//6+XXnpJ06dP16WXXhrCCSNXINd+5MiR2rdvn/bs2SNJ+uyzz7R582b99Kc/ba9xjcQfjmtHb7/9turq6jRw4EAdP35cS5cu1eeff66DBw8qJibmnP137Nih6dOnq7S0VD179tSsWbNUW1urwsLC9h8+AgR6/Xfu3KnDhw9r6NCh8ng8+tOf/qTt27fr0KFDuvLKK8PwDDq2QK9/cnKyqqqqNGPGDN1///0qLy/X/fffr1//+tdavHhxGJ5Bxxbo9f9fe/bsUWpqqnbv3q0RI0a008SR42Ku/fPPP68FCxbIsix9++23+tWvfqVVq1a18+SGsRA2p0+fthwOh/W3v/3tnG1er9dKSkqyNm/e7HssOzvbmjRpUjtOGNnauv4taWxstPr162ctXLgwxJN1Due7/v3797dcLpf17bff+h575plnrPj4+PYaMaIF8v1/7733WkOGDGmHqTqH8137999/34qLi7NefPFF6+OPP7Y2bNhguVwua9myZe08qVm4hyWMevTooQEDBqi8vPycbRUVFaqqqtLEiRN9jzU3N0v67r3QsrIy9evXr91mjURtXf+WdOvWTddff/0F74+2ne/6JyQkqFu3buratavvsWuvvVZut1uNjY3q3r17e40akS70+7++vl5r167VsmXL2mmyyHe+a//EE0/orrvu0pw5cyRJQ4YMUX19ve6991799re/9fssvs6kcz5rQ9TV1amiokIJCQnnbEtOTtaBAwdUWlrqW26//XaNHTtWpaWlcrlcYZg4srR1/VvS1NSkAwcOXPD+aNv5rv/NN9+s8vJyX6hL0qeffqqEhARiJQgu9Pt//fr1amho0J133tlOk0W+8137r7/++pwo+T7crc58F0e4X+LpTB555BHrgw8+sCorK61//etfVnp6utWzZ0/rxIkTlmVZ1l133WXl5ua2ejxvCf0wgV7/pUuXWu+8845VUVFh7du3z5o+fboVHR1tHTp0KFxPoUML9PpXV1dbMTEx1vz5862ysjLrrbfesnr16mU99dRT4XoKHdrF/vdn1KhR1rRp09p73IgS6LVfvHixFRMTY7322mvWZ599Zr377rtWv379rDvuuCNcT8EIvCXUjo4dO6asrCydOnVKsbGxGjVqlHbt2qXY2FhJUnV1dad9qa89BHr9T58+rblz58rtduuyyy7T8OHDVVJSouuuuy5cT6FDC/T6u1wuvfPOO3r44Yc1dOhQ9e7dWw8++KAee+yxcD2FDu1i/vtTVlamHTt26N133w3HyBEj0Gu/cOFC2Ww2LVy4UJ9//rliY2M1ceJE/e53vwvXUzACvyUEAACMx//OAwAA4xEsAADAeAQLAAAwHsECAACMR7AAAADjESwAAMB4BAsAADAewQIAAIxHsAAAAOMRLAAAwHgECwAAMB7BAgAAjPf/AFmUBQia0oQ6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "text_chars_per_word = [len(text)/len(text.split()) for text in normalised_texts]\n",
    "plt.hist(text_chars_per_word, bins=7);\n",
    "min(text_chars_per_word), sum(text_chars_per_word) / len(text_chars_per_word), max(text_chars_per_word)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dfd04e6-e7fb-4a81-bb51-1b8fd9856be3",
   "metadata": {},
   "source": [
    "# Finding Important Words per Document with TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "286b0e7b-db4d-4238-8fb8-70f7a08a292d",
   "metadata": {},
   "source": [
    "Inspired by Silge and Robinson's [Text Mining with R](https://www.tidytextmining.com/tfidf), we can run a TF-IDF to find the most characteristic words for each text.\n",
    "\n",
    "First we need to count the words for each text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "6177dc9a-ca72-42ed-8635-d3b6476bff12",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_counts = [Counter(term\n",
    "                       for term in re.split(\"[-_(),.?;!:' \\\"\\n]+\", text)\n",
    "                       if term)\n",
    "               for text in normalised_texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50423d4-97b7-420d-8b5b-c2aa5ffba136",
   "metadata": {},
   "source": [
    "Then get the set of all words from the texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "fcd2a053-143d-4328-aa10-96d0abbbc5df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44424"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = set(word for text_words in word_counts\n",
    "                 for word in text_words)\n",
    "len(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "856f5c49-dde4-4fc9-b4fa-336f7819990b",
   "metadata": {},
   "source": [
    "For each word we calculate the document frequency: the number of documents (books) that contain the word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "bb91c91f-2219-4f5f-8227-08b041538a90",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = {word: sum([word in text_words\n",
    "                 for text_words in word_counts])\n",
    "      for word in words}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d4fecbf-36b3-400a-bdc7-4dc5f7f0be34",
   "metadata": {},
   "source": [
    "From this we can calculate the Inverse Document Frequncy \n",
    "\n",
    "$$ {\\rm idf} = \\ln\\left(\\frac{\\rm \\# documents\\ with\\ word}{\\rm \\# documents\\ in\\ corpus}\\right) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "d0c71d25-8dc5-41f4-b490-81aefccf0f3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "idf = {word: np.log(len(word_counts)/f) for word, f in df.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57de27fa-ff04-4d3d-8ade-a41c4723a4fc",
   "metadata": {},
   "source": [
    "And multiply this by the counts to get the TF-IDF for each text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "8752245b-7e61-42d1-8be3-4486e04483e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidfs = [{word: tf * idf[word] for word, tf in word_count.items()}\n",
    "          for word_count in word_counts]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81114997-1dbf-43b6-88d8-2403e83389be",
   "metadata": {},
   "source": [
    "We can then look at the top 10 words by TF-IDF for each text: this tends to pull out the main charactersplaces from the texts (since they are different between texts).\n",
    "\n",
    "A few common characters stand out between series such as Ukridge, Psmith, and Jeeves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "0e7d172b-8a2f-4616-a805-f9ab49bc3596",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th>Title</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th></tr>\n",
       "<tr><th>Piccadilly Jim</th><td>Pett</td><td>Crocker</td><td>Ann</td><td>Wisbeach</td><td>Ogden</td><td>Jimmy</td><td>Bayliss</td><td>Trimble</td><td>Jerry</td><td>Skinner</td></tr><tr><th>Something New</th><td>Ashe</td><td>Peters</td><td>Aline</td><td>Baxter</td><td>Joan</td><td>scarab</td><td>Emsworth</td><td>Freddie</td><td>Marson</td><td>Honorable</td></tr><tr><th>A Damsel in Distress</th><td>Marshmoreton</td><td>Belpher</td><td>Maud</td><td>Caroline</td><td>Reggie</td><td>Plummer</td><td>Keggs</td><td>Albert</td><td>Byng</td><td>Billie</td></tr><tr><th>Psmith, Journalist</th><td>Psmith</td><td>Comrade</td><td>Windsor</td><td>Billy</td><td>Wilberfloss</td><td>Maloney</td><td>Jarvis</td><td>Cosy</td><td>Moments</td><td>Kid</td></tr><tr><th>Indiscretions of Archie</th><td>Archie</td><td>Lucille</td><td>Brewster</td><td>McCall</td><td>Cosmopolis</td><td>Tuyl</td><td>Silverton</td><td>Connolly</td><td>Moffam</td><td>Wheeler</td></tr><tr><th>Love Among the Chickens</th><td>Ukridge</td><td>Beale</td><td>Garnet</td><td>professor</td><td>Hawk</td><td>Phyllis</td><td>Derrick</td><td>Garny</td><td>Millie</td><td>fowls</td></tr><tr><th>The Intrusion of Jimmy</th><td>Dreever</td><td>Jimmy</td><td>Spike</td><td>McEachern</td><td>Molly</td><td>Hargate</td><td>Pitt</td><td>Mifflin</td><td>Spennie</td><td>lordship</td></tr><tr><th>The Little Nugget</th><td>Abney</td><td>Ogden</td><td>Ford</td><td>Audrey</td><td>Cynthia</td><td>MacGinnis</td><td>Nugget</td><td>Sanstead</td><td>Sam</td><td>Burns</td></tr><tr><th>Uneasy Money</th><td>Pickering</td><td>Nutty</td><td>Claire</td><td>Wetherby</td><td>Dudley</td><td>Dawlish</td><td>Elizabeth</td><td>Bill</td><td>Nichols</td><td>Nutcombe</td></tr><tr><th>Psmith in the City</th><td>Psmith</td><td>Bickersdyke</td><td>Mike</td><td>Comrade</td><td>Waller</td><td>Rossiter</td><td>Jackson</td><td>Gregory</td><td>Prebble</td><td>Bristow</td></tr><tr><th>The Man Upstairs and Other Stories</th><td>Owen</td><td>Agravaine</td><td>Rutherford</td><td>Peggy</td><td>Eddy</td><td>Annette</td><td>Beverley</td><td>Ruth</td><td>Shute</td><td>Jerry</td></tr><tr><th>Three Men and a Maid</th><td>Hignett</td><td>Sam</td><td>Billie</td><td>Bennett</td><td>Bream</td><td>Marlowe</td><td>Eustace</td><td>Peters</td><td>Mortimer</td><td>Webster</td></tr><tr><th>The Little Warrior</th><td>Jill</td><td>Derek</td><td>Chris</td><td>Wally</td><td>Pilkington</td><td>Goble</td><td>Freddie</td><td>Underhill</td><td>Nelly</td><td>Peagrim</td></tr><tr><th>The Head of Kay's</th><td>Fenn</td><td>Kennedy</td><td>Kay</td><td>Silver</td><td>Blackburn</td><td>Walton</td><td>Dencroft</td><td>Wren</td><td>Eckleton</td><td>Mulholland</td></tr><tr><th>The Gold Bat</th><td>Hara</td><td>Trevor</td><td>Clowes</td><td>Barry</td><td>Rand</td><td>Milton</td><td>Moriarty</td><td>Ruthven</td><td>Renford</td><td>Seymour</td></tr><tr><th>The Coming of Bill</th><td>Kirk</td><td>Ruth</td><td>Steve</td><td>Mamie</td><td>Porter</td><td>Bailey</td><td>Bannister</td><td>Winfield</td><td>Lora</td><td>Penway</td></tr><tr><th>The White Feather</th><td>Sheen</td><td>Stanning</td><td>Linton</td><td>Drummond</td><td>Bevan</td><td>Spence</td><td>Dunstable</td><td>Allardyce</td><td>Peteiro</td><td>Attell</td></tr><tr><th>The Prince and Betty</th><td>Betty</td><td>Scobell</td><td>Mervo</td><td>Crump</td><td>Jarvis</td><td>Pugsy</td><td>Maloney</td><td>Comrade</td><td>Peaceful</td><td>Moments</td></tr><tr><th>Tales of St. Austin's</th><td>Charteris</td><td>Babe</td><td>Bradshaw</td><td>Harrison</td><td>Merevale</td><td>Venables</td><td>Mellish</td><td>Pillingshot</td><td>Tony</td><td>Babington</td></tr><tr><th>The Pothunters</th><td>Tony</td><td>Barrett</td><td>Welch</td><td>Thomson</td><td>Jim</td><td>Charteris</td><td>Merevale</td><td>Plunkett</td><td>Vaughan</td><td>Thompson</td></tr><tr><th>A Prefect's Uncle</th><td>Gethryn</td><td>Norris</td><td>Farnie</td><td>Marriott</td><td>Pringle</td><td>Monk</td><td>Lorimer</td><td>Reece</td><td>Bishop</td><td>Beckford</td></tr><tr><th>The Clicking of Cuthbert</th><td>Ramsden</td><td>Jopp</td><td>Mortimer</td><td>Eunice</td><td>Mitchell</td><td>Merolchazzar</td><td>Waters</td><td>tee</td><td>Sturgis</td><td>Oldest</td></tr><tr><th>The Swoop! or, How Clarence Saved England: A Tale of the Great Invasion</th><td>Chugwater</td><td>Clarence</td><td>Prince</td><td>Otto</td><td>Vodkakoff</td><td>Russians</td><td>Scouts</td><td>Pfennig</td><td>Poppenheim</td><td>Quhayne</td></tr><tr><th>Not George Washington — an Autobiographical Novel</th><td>Cloyster</td><td>Julian</td><td>Hatton</td><td>Malim</td><td>Margaret</td><td>Eva</td><td>Orlebar</td><td>Gunton</td><td>Blake</td><td>Goodwin</td></tr><tr><th>William Tell Told Again</th><td>Gessler</td><td>Friesshardt</td><td>Excellency</td><td>Governor</td><td>Arnold</td><td>Walter</td><td>Leuthold</td><td>Hedwig</td><td>Sewa</td><td>PLATE</td></tr><tr><th>Mike</th><td>Mike</td><td>Psmith</td><td>Downing</td><td>Wyatt</td><td>Adair</td><td>Burgess</td><td>Wain</td><td>Jackson</td><td>Outwood</td><td>Wrykyn</td></tr><tr><th>The Adventures of Sally</th><td>Sally</td><td>Fillmore</td><td>Ginger</td><td>Carmyle</td><td>Gerald</td><td>Faucitt</td><td>Meecher</td><td>Cracknell</td><td>Hobson</td><td>Kemp</td></tr><tr><th>The Man with Two Left Feet, and Other Stories</th><td>Katie</td><td>Birdsey</td><td>Meggs</td><td>Wilton</td><td>Plimmer</td><td>Pillenger</td><td>Andy</td><td>Fred</td><td>Minnie</td><td>Gussie</td></tr><tr><th>My Man Jeeves</th><td>Jeeves</td><td>Bicky</td><td>Bobbie</td><td>Rocky</td><td>Corky</td><td>Motty</td><td>Voules</td><td>Bickersteth</td><td>Rockmetteller</td><td>Wooster</td></tr><tr><th>Death at the Excelsior, and Other Stories</th><td>Snyder</td><td>Pickett</td><td>Buffin</td><td>Jeeves</td><td>Rastall</td><td>Retford</td><td>Bassington</td><td>Eve</td><td>Oakes</td><td>Archie</td></tr><tr><th>The Politeness of Princes, and Other School Stories</th><td>Dunstable</td><td>Pillingshot</td><td>Linton</td><td>Chapple</td><td>Spencer</td><td>Shields</td><td>Clephane</td><td>Henfrey</td><td>Shearne</td><td>Phipps</td></tr><tr><th>A Wodehouse Miscellany: Articles & Stories</th><td>Florence</td><td>Sylvia</td><td>Duggie</td><td>Bodfish</td><td>Craye</td><td>Edwin</td><td>Dick</td><td>Darrell</td><td>Jeeves</td><td>lyricist</td></tr><tr><th>A Man of Means</th><td>Roland</td><td>Windlebird</td><td>Verepoint</td><td>Bleke</td><td>Petheram</td><td>Maraquita</td><td>Teal</td><td>Squibs</td><td>Coppin</td><td>Paranoya</td></tr><tr><th>The Gem Collector</th><td>Spennie</td><td>McEachern</td><td>Chames</td><td>Molly</td><td>Jimmy</td><td>Spike</td><td>Wesson</td><td>Blunt</td><td>Galer</td><td>Pitt</td></tr><tr><th>Right Ho, Jeeves</th><td>Jeeves</td><td>Nottle</td><td>Gussie</td><td>Fink</td><td>Dahlia</td><td>Tuppy</td><td>Angela</td><td>Bassett</td><td>Anatole</td><td>Bertie</td></tr><tr><th>Mike and Psmith</th><td>Psmith</td><td>Downing</td><td>Mike</td><td>Adair</td><td>Outwood</td><td>Jellicoe</td><td>Sedleigh</td><td>Jackson</td><td>Wrykyn</td><td>Comrade</td></tr><tr><th>Love Among the Chickens\n",
       "A Story of the Haps and Mishaps on an English Chicken Farm</th><td>Ukridge</td><td>Garnet</td><td>Beale</td><td>professor</td><td>Phyllis</td><td>Hawk</td><td>Derrick</td><td>fowls</td><td>Millie</td><td>Chase</td></tr><tr><th>Jill the Reckless</th><td>Jill</td><td>Derek</td><td>Chris</td><td>Wally</td><td>Pilkington</td><td>Goble</td><td>Freddie</td><td>Nelly</td><td>Underhill</td><td>Peagrim</td></tr><tr><th>The Girl on the Boat</th><td>Bennett</td><td>Billie</td><td>Hignett</td><td>Sam</td><td>Bream</td><td>Marlowe</td><td>Eustace</td><td>Mortimer</td><td>Hubbard</td><td>Mallaby</td></tr><tr><th>The Inimitable Jeeves</th><td>Bingo</td><td>Jeeves</td><td>Bertie</td><td>Steggles</td><td>Agatha</td><td>Wooster</td><td>Claude</td><td>Eustace</td><td>Heppenstall</td><td>Bassington</td></tr><tr><th>Leave it to Psmith</th><td>Psmith</td><td>Peavey</td><td>Baxter</td><td>McTodd</td><td>Eve</td><td>Constance</td><td>Emsworth</td><td>Cootes</td><td>Keeble</td><td>Freddie</td></tr><tr><th>Ukridge</th><td>Ukridge</td><td>Billson</td><td>Bowles</td><td>Tupper</td><td>laddie</td><td>Corky</td><td>Weeks</td><td>Tod</td><td>Flossie</td><td>Teddy</td></tr><tr><th>A Gentleman of Leisure</th><td>Dreever</td><td>Jimmy</td><td>Spike</td><td>McEachern</td><td>Molly</td><td>Hargate</td><td>Pitt</td><td>Mifflin</td><td>lordship</td><td>Spennie</td></tr><tr><th>Carry On, Jeeves</th><td>Jeeves</td><td>Biffy</td><td>Wooster</td><td>Bicky</td><td>Bertie</td><td>Corky</td><td>Rocky</td><td>Motty</td><td>Roderick</td><td>Bingo</td></tr><tr><th>Sam in the Suburbs</th><td>Braddock</td><td>Sam</td><td>Hash</td><td>Kay</td><td>Tilbury</td><td>Chimp</td><td>Shotter</td><td>Molloy</td><td>Wrenn</td><td>Claire</td></tr><tr><th>The small bachelor</th><td>Waddington</td><td>Beamish</td><td>Hamilton</td><td>Mullett</td><td>Finch</td><td>Garroway</td><td>Sigsbee</td><td>Molly</td><td>Hunstanton</td><td>Fanny</td></tr><tr><th>Meet Mr Mulliner</th><td>Mulliner</td><td>Augustine</td><td>Wilfred</td><td>bishop</td><td>Frederick</td><td>Rodman</td><td>Lancelot</td><td>McKinnon</td><td>Wilks</td><td>Clarence</td></tr><tr><th>Divots</th><td>Bradbury</td><td>Spelvin</td><td>Rodney</td><td>Fisher</td><td>Ferdinand</td><td>Bott</td><td>Chester</td><td>Jane</td><td>William</td><td>Anastatia</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import display, HTML\n",
    "\n",
    "n_top_words = 10\n",
    "\n",
    "html = \"<table><tr><th>Title</th>\" + \\\n",
    "       \"\".join([f\"<th>{i+1}</th>\" for i in range(n_top_words)]) + \\\n",
    "       \"</tr>\\n\"\n",
    "for idx, tfidf in enumerate(tfidfs):\n",
    "    title = wodehouse_books[book_ids[idx]][\"Title\"]\n",
    "    html += f\"<tr><th>{title}</th>\"\n",
    "    for item, score in sorted(tfidf.items(),\n",
    "                              key=lambda x: x[1],\n",
    "                              reverse=True)[:n_top_words]:\n",
    "        html += f\"<td>{item}</td>\"\n",
    "    html += \"</tr>\"\n",
    "html += \"</table>\"\n",
    "HTML(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7ebf7a4-10f7-4358-a7e9-c2363ea85d7f",
   "metadata": {},
   "source": [
    "# Split the data\n",
    "\n",
    "If we want to do some further language modelling on this data we can split it into train, valid and test sets by text.\n",
    "Note that we're likely to have some characters (like Jeeves, Psmith, and Ukridge) across splits.\n",
    "\n",
    "We really should split collections of books (like \"A Wodehouse Miscellany\") separately, but for now we will keep the implementation simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "1a7c5cbf-13fe-4d66-bbb3-7c9ae9466383",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'train': 30, 'valid': 10, 'test': 8})"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "from collections import Counter\n",
    "\n",
    "random.seed(\"pgwodehouse\")\n",
    "\n",
    "splits = dict(\n",
    "    train=0.6,\n",
    "    valid=0.2,\n",
    "    test=0.2,\n",
    ")\n",
    "\n",
    "assert sum(splits.values()) == 1.0\n",
    "\n",
    "split = random.choices(population=list(splits.keys()),\n",
    "                       weights=list(splits.values()),\n",
    "                       k=len(normalised_texts))\n",
    "\n",
    "Counter(split)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "501c9a80-76cd-4689-b5b2-221a6e7b7020",
   "metadata": {},
   "source": [
    "To make it easier to work with let's collect the mapping from a split to the list of books it contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "3c48f9ff-05c5-4602-a3e6-6bf8dbf3c06a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': 30, 'valid': 10, 'test': 8}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "split2idx = {s: [] for s in splits}\n",
    "for idx, s in enumerate(split):\n",
    "    split2idx[s].append(idx)\n",
    "\n",
    "for s, idxs in split2idx.items():\n",
    "    for idx in idxs:\n",
    "        assert split[idx] == s\n",
    "        \n",
    "assert sorted(sum([idxs for idxs in split2idx.values()],\n",
    "                  start=[])) == list(range(len(split)))\n",
    "\n",
    "{s: len(idxs) for s, idxs in split2idx.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4faec949-b5ef-4a6e-81f6-a49a728afeaf",
   "metadata": {},
   "source": [
    "We have over 10 million characters of training data (which is around 2 million words); enough for an interesting character level language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "dd09f59e-485e-4100-bb8a-3edf6be84172",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': '11_179_271', 'valid': '2_569_961', 'test': '2_736_771'}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{s: '{:_}'.format(sum([len(normalised_texts[idx]) for idx in idxs]))\n",
    " for s, idxs in split2idx.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "173c8285-298c-47fd-a0ea-55e5c1efe401",
   "metadata": {},
   "source": [
    "# Save the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83544604-e5da-476f-a091-6d6f3e7b914f",
   "metadata": {},
   "source": [
    "## Locally\n",
    "\n",
    "We can save each normalised text as a text file in a folder corresponding to its split, using the Project Gutenberg id as the filename.\n",
    "This is saved in a [downloadable zip file](https://github.com/EdwardJRoss/skeptric/raw/master/wodehouse/wodehouse_text.zip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "ab67c8d5-b95d-4d89-8ad6-7d5f7db53a62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "valid/6985.txt      valid/6955.txt      valid/8178.txt      valid/8931.txt      \n",
      "valid/6984.txt      valid/6753.txt      valid/65974.txt     valid/3829.txt      \n",
      "valid/7298.txt      valid/6684.txt      test/6927.txt       test/7230.txt       \n",
      "test/6980.txt       test/7028.txt       test/2005.txt       test/6768.txt       \n",
      "test/2042.txt       test/8713.txt       train/8164.txt      train/6836.txt      \n",
      "train/7050.txt      train/6880.txt      train/6837.txt      train/59254.txt     \n",
      "train/70041.txt     train/7471.txt      train/72227.txt     train/70222.txt     \n",
      "train/7464.txt      train/65172.txt     train/8176.txt      train/10554.txt     \n",
      "train/8190.txt      train/2233.txt      train/60067.txt     train/10586.txt     \n",
      "train/61507.txt     train/20717.txt     train/6877.txt      train/3756.txt      \n",
      "train/20532.txt     train/2607.txt      train/4075.txt      train/7423.txt      \n",
      "train/20533.txt     train/67368.txt     train/6683.txt      train/6879.txt      \n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "from zipfile import ZipFile\n",
    "\n",
    "output_dir = Path('output')\n",
    "\n",
    "\n",
    "if output_dir.exists():\n",
    "    shutil.rmtree(output_dir)\n",
    "output_dir.mkdir()\n",
    "\n",
    "for split, idxs in split2idx.items():\n",
    "    path = output_dir / split\n",
    "    path.mkdir()\n",
    "    for idx in idxs:\n",
    "        id = book_ids[idx]\n",
    "        with open(path / (id + \".txt\"), \"wt\") as f:\n",
    "            f.write(normalised_texts[idx])\n",
    "\n",
    "shutil.make_archive(\"wodehouse_text\", 'zip', output_dir)\n",
    "\n",
    "\n",
    "filenames = [f.filename for f in ZipFile(\"wodehouse_text.zip\").filelist\n",
    "             if not f.is_dir()]\n",
    "print(fmtcols(filenames, 4, max_len=20, buffer=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41956f5e-8a65-4dbc-914a-118def7aa57b",
   "metadata": {},
   "source": [
    "## On Huggingface Hub\n",
    "\n",
    "We can also save the texts, along with their metadata [to Huggingface Hub](https://huggingface.co/datasets/edwardjross/wodehouse)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "c5350e13-6d11-468e-8c58-d6b2759842f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['Text#', 'Type', 'Issued', 'Title', 'Language', 'Authors', 'Subjects', 'LoCC', 'Bookshelves', 'raw_text', 'content'],\n",
       "        num_rows: 30\n",
       "    })\n",
       "    valid: Dataset({\n",
       "        features: ['Text#', 'Type', 'Issued', 'Title', 'Language', 'Authors', 'Subjects', 'LoCC', 'Bookshelves', 'raw_text', 'content'],\n",
       "        num_rows: 10\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['Text#', 'Type', 'Issued', 'Title', 'Language', 'Authors', 'Subjects', 'LoCC', 'Bookshelves', 'raw_text', 'content'],\n",
       "        num_rows: 8\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import Dataset, DatasetDict\n",
    "\n",
    "ds = DatasetDict({s: Dataset.from_list([wodehouse_books[book_ids[idx]] | dict(content=normalised_texts[idx]) for idx in split2idx[s]]) for s in split2idx})\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "8ba0baf0-f3ae-4d7f-93aa-24c524fa11d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eeb899e79be4464e94ca8c421bf64c6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Uploading the dataset shards:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "801061869d254fdb8b0806866108357a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "511614a016844da5b346ba6d057bab4d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Uploading the dataset shards:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5315766ae16941eaacca3963bf36313d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "225482bf324a495398c4a7c7fc73d3b0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Uploading the dataset shards:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "759f2564a029412fae03c5aeb9407eef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "938cbe2f3a8340d8bccb619b1c67b2bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/830 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds.push_to_hub(\"edwardjross/wodehouse\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13fb0f2a-6041-45e6-8389-5482bc05fa80",
   "metadata": {},
   "source": [
    "# What next?\n",
    "\n",
    "We've downloaded the P. G. Wodehouse texts from Project Gutenberg and done a rough job of analysing and cleaning them.\n",
    "While we could do a better job of cleaning them, or obtaining more of them from archive.org, we have done enough to train and test language models on."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}