Model Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a Phoneme Language Model #\n",
    "In this notebook I train a language model over English sounds (also known as [phonemes](https://en.wikipedia.org/wiki/Phoneme)). Each phoneme is represented by a symbol from the [International Phonetic Alphabet](https://en.wikipedia.org/wiki/International_Phonetic_Alphabet), or IPA. So \"fish\" is represented by /ˈfɪʃ/. Pronunciations in IPA are enclosed in slashes, \"ˈ\" indicates primary stress and \"ˌ\" indicates secondary stress. The rest of the symbols represent individual sounds.\n",
    "\n",
    "By training on over a hundred thousand pronunciations I want the language model to be able to distinguish good sounding English words from less English sounding words. For example, /ˈfʌn/ (\"fun\") sounds good, but /ntɔˈ/ (maybe represented as \"ntaw\") does not.\n",
    "\n",
    "Check out the notebook `Model Exploration and Usage.ipynb` if you want to see what English words the model thinks are most standard and to see some generated words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "from torch.nn import functional as F\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "from sonorant.analysis_utils import plot_pronunciation_probability\n",
    "from sonorant.languagemodel import LanguageModel, ModelParams, Vocabulary\n",
    "from sonorant.pronunciationdata import load_pronunciations\n",
    "from sonorant.utils import split_data\n",
    "\n",
    "plt.rcParams['figure.figsize'] = [15, 8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Data ##\n",
    "The data for this model comes from the [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict), which contains over one hundred thousand pronunciations. The CMU Dict pronunciations were originally in [ARPABET](https://en.wikipedia.org/wiki/ARPABET), but I found a version translated into the more standard [IPA](https://en.wikipedia.org/wiki/International_Phonetic_Alphabet)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll use the `load_pronunciations` function to load the Pronouncing Dictionary into a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 131,964 pronunciations.\n",
      "\n",
      "Sample of 5 pronunciations:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pronunciation</th>\n",
       "      <th>as_string</th>\n",
       "      <th>num_phonemes</th>\n",
       "      <th>num_syllables</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>word</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>witts</th>\n",
       "      <td>(ˈ, w, ɪ, t, s)</td>\n",
       "      <td>ˈwɪts</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>motyka</th>\n",
       "      <td>(ˈ, m, ɑː, t, aɪ, k, ə)</td>\n",
       "      <td>ˈmɑːtaɪkə</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>kneed</th>\n",
       "      <td>(ˈ, n, iː, d)</td>\n",
       "      <td>ˈniːd</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jesuit</th>\n",
       "      <td>(ˈ, dʒ, ɛ, ʒ, uː, ɪ, t)</td>\n",
       "      <td>ˈdʒɛʒuːɪt</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>boehme</th>\n",
       "      <td>(ˈ, b, oʊ, m)</td>\n",
       "      <td>ˈboʊm</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  pronunciation  as_string  num_phonemes  num_syllables\n",
       "word                                                                   \n",
       "witts           (ˈ, w, ɪ, t, s)      ˈwɪts             4              1\n",
       "motyka  (ˈ, m, ɑː, t, aɪ, k, ə)  ˈmɑːtaɪkə             6              3\n",
       "kneed             (ˈ, n, iː, d)      ˈniːd             3              1\n",
       "jesuit  (ˈ, dʒ, ɛ, ʒ, uː, ɪ, t)  ˈdʒɛʒuːɪt             6              3\n",
       "boehme            (ˈ, b, oʊ, m)      ˈboʊm             3              1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pronunciations = load_pronunciations()\n",
    "print(f\"There are {len(pronunciations):,} pronunciations.\")\n",
    "print()\n",
    "print(\"Sample of 5 pronunciations:\")\n",
    "pronunciations.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the pronunciation for \"fish\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pronunciation    (ˈ, f, ɪ, ʃ)\n",
       "as_string                ˈfɪʃ\n",
       "num_phonemes                3\n",
       "num_syllables               1\n",
       "Name: fish, dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pronunciations.loc['fish']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here are all of the pronunciations for the word \"tomato\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pronunciation</th>\n",
       "      <th>as_string</th>\n",
       "      <th>num_phonemes</th>\n",
       "      <th>num_syllables</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>word</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tomato</th>\n",
       "      <td>(t, ə, ˈ, m, eɪ, ˌ, t, oʊ)</td>\n",
       "      <td>təˈmeɪˌtoʊ</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tomato</th>\n",
       "      <td>(t, ə, ˈ, m, ɑː, ˌ, t, oʊ)</td>\n",
       "      <td>təˈmɑːˌtoʊ</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     pronunciation   as_string  num_phonemes  num_syllables\n",
       "word                                                                       \n",
       "tomato  (t, ə, ˈ, m, eɪ, ˌ, t, oʊ)  təˈmeɪˌtoʊ             6              3\n",
       "tomato  (t, ə, ˈ, m, ɑː, ˌ, t, oʊ)  təˈmɑːˌtoʊ             6              3"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pronunciations.loc['tomato']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Model ##\n",
    "The module `languagemodel.py` contains a class `LanguageModel` that implements a simple neural language model. It's a PyTorch neural network comprised of the following layers:\n",
    "1. **Embedding layer** to translate each phoneme into a dense vector. Note that in the code this is called the _encoder since it encodes input phonemes into a representation the model can work with.\n",
    "2. An recurrent neural network (**RNN**) layer that processes each input phoneme sequentially and for each step generates (a) a hidden representation to pass on to the next step and (b) an output.\n",
    "3. A **linear layer** that decodes the outputes (2b) into distributions over each phoneme. Note that in the code this is called the _docoder since it decodes the model's internal representations back into phonemes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go through a simple example of what happens when we pass the pronunciation /F IH1 SH/ through the model. Ultimately what I want ouf of the model is a prediction at each position of what the next phoneme should be. For example, when a well trained model is sees /F IH1/ it should know that /SH/ is likely, or at least not unlikely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First I'll create a `Vocabulary` instance by passing in all the pronunciations. The `vocab` is used to convert phonemes into integer indices that the neural network handle. It does a few other things too, which you can see below. The `Vocabulary` class's code is in `sonorant/languagemodel.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ˈfɪʃ\n"
     ]
    }
   ],
   "source": [
    "print(''.join(pronunciations.loc['fish'].pronunciation))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 47 distinct phonemes.\n",
      "\n",
      "Looking up the int index for /ʃ/: 31\n",
      "\n",
      "Checking whether /ʃ/ is in the vocabulary: True\n",
      "\n",
      "Looking up the phoneme for a specific int index: ʃ\n",
      "\n",
      "Encoding /ˈfɪʃ/: [ 1  5 13 22 31  2]\n"
     ]
    }
   ],
   "source": [
    "vocab = Vocabulary.from_texts(pronunciations.pronunciation.values)\n",
    "\n",
    "print(f\"There are {len(vocab)} distinct phonemes.\")\n",
    "print()\n",
    "print(\"Looking up the int index for /ʃ/:\", vocab['ʃ'])\n",
    "print()\n",
    "print(\"Checking whether /ʃ/ is in the vocabulary:\", 'ʃ' in vocab)\n",
    "print()\n",
    "print(\"Looking up the phoneme for a specific int index:\", vocab.token_from_idx(vocab['ʃ']))\n",
    "print()\n",
    "print(\"Encoding /ˈfɪʃ/:\", vocab.encode_text(tuple(\"ˈfɪʃ\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I'll define the model. Note that I'm not actually fitting the model to any data so the output will be random. The hyperparameters aren't optimal, but again that doesn't matter here since I just want to show the flow of data through the network.\n",
    "\n",
    "The `ModelParams` class (from `sonorant/languagemodel.py` encapsulates hyperparameters and options for the model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_params = ModelParams(\n",
    "    rnn_type='rnn', embedding_dimension=10, hidden_dimension=3, num_layers=1,\n",
    "    max_epochs=3, early_stopping_rounds=3\n",
    ")\n",
    "\n",
    "language_model = LanguageModel(vocab, model_params, 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I'll walk through what happens when we pass the word \"fish\" /F IH1 SH/ through the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('ˈ', 'f', 'ɪ', 'ʃ')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fish_pronunciation = tuple(\"ˈfɪʃ\")\n",
    "fish_pronunciation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Vocabulary.encode_text` function we saw earlier does a few things. First, it adds dummy `<START>` and `<END>` tokens to the pronunciation indicating its start and end. This allows the model to learn transition probabilities from the start of the word to the first phoneme, and from the last phoneme to the end of the word.\n",
    "\n",
    "It then converts every phoneme to its ingeter index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1  5 13 22 31  2]\n",
      "\n",
      "1 => <START>\n",
      "5 => ˈ\n",
      "13 => f\n",
      "22 => ɪ\n",
      "31 => ʃ\n",
      "2 => <END>\n"
     ]
    }
   ],
   "source": [
    "fish_input = vocab.encode_text(fish_pronunciation)\n",
    "print(fish_input)\n",
    "print()\n",
    "for idx in fish_input:\n",
    "    phoneme = vocab.token_from_idx(idx)\n",
    "    print(f'{idx} => {phoneme}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we're ready to pass the input into the model's `forward` function, which takes in inputs and outputs predictions. This model's `forward` function expects a Tensor of dimension `(batch_size, NUMBER OF STEPS)`. A step here refers to a step forward in the sequence, so /<START> F IH1 SH <END>/ has 5 steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input's shape: (6,)\n",
      "Batch input's shape: torch.Size([1, 6])\n"
     ]
    }
   ],
   "source": [
    "print(\"Input's shape:\", fish_input.shape)\n",
    "fish_batch_input = torch.LongTensor(fish_input).unsqueeze(0)\n",
    "print(\"Batch input's shape:\", fish_batch_input.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing the `forward` function does is embed each phoneme using an [nn.Embedding](https://pytorch.org/docs/stable/nn.html#embedding). Each phoneme has a dedicated embedding vector of length `embedding_dimension`, so the shape of `embedded` is `(batch size, number of steps, embedding_dimension)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 6, 10])\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.2358,  0.0812,  0.4921,  2.0121,  1.1394, -0.0168, -0.1808,\n",
       "           0.5696,  0.2853, -1.1750],\n",
       "         [ 0.0177, -0.0499, -0.5503,  0.6401, -0.0039,  0.4457, -0.2539,\n",
       "          -0.4336,  1.4395, -0.3192],\n",
       "         [-1.0877, -0.0758, -1.1741, -0.9751, -2.1680, -1.1938,  0.6698,\n",
       "           0.4226,  0.7646,  0.3113],\n",
       "         [-1.1634,  0.9116, -1.1395,  0.8588, -0.3173,  1.5198, -1.0894,\n",
       "          -0.5033,  0.6822, -0.7204],\n",
       "         [ 0.2023, -1.0492, -0.4600, -0.6626,  1.3397,  2.5528,  0.0322,\n",
       "          -0.7443,  0.2199,  1.7352],\n",
       "         [ 3.0174, -0.3162, -2.0198, -0.4923,  0.2001, -1.0426, -0.0698,\n",
       "          -0.2581, -0.1434, -0.5738]]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedded = language_model._encoder(fish_batch_input)\n",
    "print(embedded.shape)\n",
    "print()\n",
    "embedded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll pass `embedded` to the [RNN layer](https://pytorch.org/docs/stable/nn.html#recurrent-layers), resulting in `rnn_output` and `hidden_state`. I won't go into detail on how RNNs work since there are many detailed posts on the web you can read, but the basic idea is a cell is applied sequentially to every token (i.e. step) in the input. At each step an output and a hidden state are produced. The hidden state can be passed on to the next step, and the output can be used to make a prediction.\n",
    "\n",
    "The `rnn` layer below operates on the full sequence, so the results are for the entire sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 6, 3])\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.9798,  0.4632, -0.9725],\n",
       "         [-0.7061,  0.7552, -0.7428],\n",
       "         [ 0.9519,  0.7755,  0.0025],\n",
       "         [-0.9005,  0.9203, -0.9741],\n",
       "         [ 0.7683, -0.0068,  0.6018],\n",
       "         [ 0.5360, -0.9496,  0.9670]]], grad_fn=<TransposeBackward1>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnn_output, hidden_state = language_model._rnn(embedded)\n",
    "print(rnn_output.shape)\n",
    "print()\n",
    "rnn_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our application we can ignore the `hidden_state`-- the `rnn_output` is the interesting part. The first dimension is for the batch, and we only have a single input in our batch. The second dimension is for each of the input phonemes. The third dimension corresponds to `hidden_dimension`: you can think of this as the state of the RNN at each step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As I said at the beginning of this section, I want the output of the RNN at each position to be predictions for the *next* position. So I'll apply a [linear layer](https://pytorch.org/docs/stable/nn.html#linear) to the `rnn_output`, resulting in a vector the size of the vocabularly at each position. The [softmax](https://pytorch.org/docs/stable/nn.functional.html#softmax) function normalizes the outputs into probability distributions for each prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6, 47])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs = language_model._decoder(rnn_output)\n",
    "probabilities = F.softmax(outputs, dim=-1).squeeze()\n",
    "probabilities.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape of `probabilities` is (6, 47) because each of the six tokens in /ˈfɪʃ/ gets a a probability distribution over each of the 47 phonemes in the vocabulary.\n",
    "\n",
    "The first phoneme in the input is the `<START>` token; let's see what the model thinks should come next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'tʃ'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probabilities_for_first_phoneme = probabilities[0]\n",
    "most_likely_first_phoneme_idx = probabilities_for_first_phoneme.argmax().item()\n",
    "most_likely_first_phoneme = vocab.token_from_idx(most_likely_first_phoneme_idx)\n",
    "most_likely_first_phoneme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the model isn't fit yet this is just a random guess. In order to get the model to make good predictions I'll need to first train a good model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the Best Model ##\n",
    "In this section I'll train a number of models on the train set and select the one that has the lowest error on the dev set. I'll split the DataFrame of pronunciations into three DataFrames, with 79% for training, 20% for dev/validation, and 1% for testing of the final model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(104251, 26393, 1320)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_pronunciations, dev_pronunciations, test_pronunciations = split_data(pronunciations, dev_proportion=.2, test_proportion=.01)\n",
    "len(train_pronunciations), len(dev_pronunciations), len(test_pronunciations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I split the corpus up into train/dev/test so that I could use the standard approach of training models with different hyperparameters and selecting the model that performs best against the dev set. The standard metric for language models is perplexity, which is a measure of how well the model predicts the pronunciations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that I ran a larger parameter search before and saw that GRUs were consistently outperforming LSTMs and vanilla RNNs. Below I'm trying a large network with three layers and embedding dimension and hidden dimension of 500. This is too large of a network architecture for the relatively simple problem of word structure, but my goal here is to overfit the training set and get a sense of what kind of performance on the dev set is reasonable. I can also identify at which epoch the dev loss was lowest and later train a model using early stopping.\n",
    "\n",
    "The GRU will train for a maximum of 50 epochs. Every five epochs I'm printing out the train and dev loss five generated pronunciations.\n",
    "\n",
    "I'm also printing out the percentage of generated pronunciations that occurred in the train or dev sets. This isn't a rigorous way to measure whether the model is overfitting the training set, but it I've found it does seem to indicate some amount of overfitting: I've seen it get up to nearly 100% when training a large enough model for long enough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5: train loss: 0.4883\tdev loss: 0.7207                                                                                               \n",
      "\tGenerated: in train: 17%, assess: 4%, novel: 79%\n",
      "\t ˈ æ m b ɝ z\n",
      "\t ˌ m ɛ n s ə b ˈ j uː z\n",
      "\t ˈ k l ɔ r i\n",
      "\t ˈ k oʊ z oʊ s\n",
      "\t ˈ ɛ n d ɪ ŋ\n",
      "Epoch 10: train loss: 0.4492\tdev loss: 0.6908                                                                                               \n",
      "\tGenerated: in train: 20%, assess: 1%, novel: 79%\n",
      "\t ˈ g r æ n s\n",
      "\t ˈ k oʊ l z\n",
      "\t ˈ g ʌ d ˌ f i\n",
      "\t d ɪ ˈ s ɝː p t ɪ v l i\n",
      "\t ˈ r uː b ə\n",
      "Epoch 15: train loss: 0.4124\tdev loss: 0.6800                                                                                               \n",
      "\tGenerated: in train: 28%, assess: 1%, novel: 70%\n",
      "\t ˈ m ʌ n iː ˌ s iː m\n",
      "\t m ɑː n ˈ k ɑː l ə\n",
      "\t ˈ b l oʊ k ɔ f\n",
      "\t ˈ h j uː n ɝ\n",
      "\t ˈ b ɛ r i\n",
      "Epoch 20: train loss: 0.3829\tdev loss: 0.6867                                                                                               \n",
      "\tGenerated: in train: 42%, assess: 1%, novel: 56%\n",
      "\t ˈ l iː l ə s\n",
      "\t n oʊ ˈ oʊ ʃ iː ə\n",
      "\t ˈ s k ɪ p ɪ ŋ\n",
      "\t ˈ p iː l ɪ ŋ\n",
      "\t ˈ k r ɑː k ə l t\n",
      "Epoch 25: train loss: 0.3645\tdev loss: 0.6981                                                                                               \n",
      "\tGenerated: in train: 53%, assess: 3%, novel: 44%\n",
      "\t ˈ z ɔ r k ɪ n\n",
      "\t ˈ k oʊ t s\n",
      "\t ˈ t ɛ ˌ l ɑː k\n",
      "\t ˈ t ɪ n d ə l\n",
      "\t ˈ p r ɑː f ə t ˌ s p ɔ r t\n",
      "CPU times: user 26min 50s, sys: 8min 8s, total: 34min 59s\n",
      "Wall time: 34min 47s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "large_model_parameters = ModelParams(\n",
    "    rnn_type='gru', num_layers=3, embedding_dimension=500, hidden_dimension=500,\n",
    "    max_epochs=25, early_stopping_rounds=25\n",
    ")\n",
    "\n",
    "large_language_model = LanguageModel(vocab, large_model_parameters, device_name='cuda')\n",
    "\n",
    "large_train_losses, large_dev_losses = large_language_model.fit(\n",
    "    train_pronunciations.pronunciation.values.tolist(),\n",
    "    dev_pronunciations.pronunciation.values.tolist(),\n",
    "    print_every=5,\n",
    ")\n",
    "\n",
    "large_results = pd.DataFrame({\n",
    "    'epoch': range(1, len(large_train_losses) + 1),\n",
    "    'train_loss': large_train_losses,\n",
    "    'dev_loss': large_dev_losses,\n",
    "}).set_index('epoch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "large_results = pd.DataFrame({\n",
    "    'epoch': range(1, len(large_train_losses) + 1),\n",
    "    'train_loss': large_train_losses,\n",
    "    'dev_loss': large_dev_losses,\n",
    "}).set_index('epoch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3AAAAHgCAYAAADkE4+9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdeXyddZ33/9eVfTtJmj3dU7q30EI3oKAiigU74gKi81OoorgA4z2/0bmZ3z2PmZFx7nHu25/zu2cGQVGKIoKCCggCCoosBUorW/e9dF/SNkmXNNv1++M6OUnTtE1L0pOTvJ6Px/U451zne53re+pDkne+3+/nG4RhiCRJkiSp/0tLdgckSZIkST1jgJMkSZKkFGGAkyRJkqQUYYCTJEmSpBRhgJMkSZKkFGGAkyRJkqQUkZGsG5eVlYWjR49O1u0lSZIkKamWLl26NwzD8tO5JmkBbvTo0SxZsiRZt5ckSZKkpAqCYPPpXuMUSkmSJElKEQY4SZIkSUoRBjhJkiRJShFJWwMnSZIkqf9qbm5m69atNDY2JrsrKS8nJ4fhw4eTmZn5rj/LACdJkiTpOFu3biUWizF69GiCIEh2d1JWGIbU1taydetWampq3vXnOYVSkiRJ0nEaGxspLS01vL1LQRBQWlraayOZBjhJkiRJ3TK89Y7e/Hc0wEmSJElSijDASZIkSeqXDhw4wPe+973Tvu6qq67iwIEDp33dggULePjhh0/7urPJACdJkiSpXzpRgGtpaTnpdb/97W8pLi7uq24llVUoJUmSJJ3UN3+znBXb63v1MycPLeQf/2LKSdvcdtttrF+/nunTp5OZmUlOTg5Dhgxh1apVrFmzho9+9KNs2bKFxsZGvva1r3HTTTcBMHr0aJYsWcLBgwe58sorueSSS1i0aBHDhg3j0UcfJTc395T9e/bZZ/n6179OS0sLs2bN4s477yQ7O5vbbruNxx57jIyMDK644gq+853v8NBDD/HNb36T9PR0ioqKeP7553vl36g7BjhJkiRJ/dK3v/1tli1bxhtvvMFzzz3Hhz/8YZYtW5Yox3/PPfdQUlLCkSNHmDVrFp/4xCcoLS095jPWrl3LAw88wN13380nP/lJfvnLX/KZz3zmpPdtbGxkwYIFPPvss4wfP57rr7+eO++8k89+9rP8+te/ZtWqVQRBkJimefvtt/P0008zbNiwM5q6eToMcJIkSZJO6lQjZWfL7Nmzj9lL7T/+4z/49a9/DcCWLVtYu3btcQGupqaG6dOnAzBjxgw2bdp0yvusXr2ampoaxo8fD8ANN9zAHXfcwS233EJOTg433ngj8+fPZ/78+QDMnTuXBQsW8MlPfpKPf/zjvfFVT8g1cJIkSZJSQn5+fuL5c889xzPPPMPLL7/Mm2++yfnnn9/tXmvZ2dmJ5+np6adcP3cyGRkZLF68mGuuuYbHH3+cefPmAXDXXXfxrW99iy1btjBjxgxqa2vP+B6n7EOffbIkSZIkvQuxWIyGhoZu36urq2PIkCHk5eWxatUqXnnllV6774QJE9i0aRPr1q1j7Nix3Hfffbz3ve/l4MGDHD58mKuuuoq5c+cyZswYANavX8+cOXOYM2cOTz75JFu2bDluJLC3GOAkSZIk9UulpaXMnTuXqVOnkpubS2VlZeK9efPmcddddzFp0iQmTJjAhRde2Gv3zcnJYeHChVx77bWJIiZf/vKX2bdvH1dffTWNjY2EYch3v/tdAL7xjW+wdu1awjDk8ssvZ9q0ab3Wl66CMAz77MNPZub554VLXn8rKfeWJEmSdHIrV65k0qRJye7GgNHdv2cQBEvDMJx5Op+TvDVwtRug+UjSbi9JkiRJqSZ5Aa7lCPz+H5N2e0mSJEmD080338z06dOPORYuXJjsbvVI8tbAFVTA4u/D2A/A+CuS1g1JkiRJg8sdd9yR7C6cseSNwMWGQsUUePSrcHBP0rohSZIkSakieQEuCOATP4TGenj0ZkhSMRVJkiRJShXJ3ci7cjJc8c+w9ml47YdJ7YokSZIk9XfJDXAAs2+K1sH97u9h96pk90aSJEmS+q3kB7gggKu/B1kF8MsvQMvRZPdIkiRJUj/0T//0T3znO9/plc9asGABDz/8cK981tmU/AAHEKuEq++AXW/Ds7cnuzeSJEmS1C8lbxuBribMg1lfgJf/K5pSec5lye6RJEmSJIAnb4Odb/fuZ1adC1d++5TN/uVf/oUf//jHVFRUMGLECGbMmMH69eu5+eab2bNnD3l5edx9991UV1dz3nnnsXHjRtLS0jh06BATJ05kw4YNZGZmnvQezz77LF//+tdpaWlh1qxZ3HnnnWRnZ3Pbbbfx2GOPkZGRwRVXXMF3vvMdHnroIb75zW+Snp5OUVERzz//fG/9i/RI/wlwAB/8Z9j4AjzyFfjKIsgrSXaPJEmSJCXJ0qVLefDBB3njjTdoaWnhggsuYMaMGdx0003cddddjBs3jldffZWvfvWr/OEPf2D69On86U9/4rLLLuPxxx/nQx/60CnDW2NjIwsWLODZZ59l/PjxXH/99dx555189rOf5de//jWrVq0iCAIOHDgAwO23387TTz/NsGHDEufOpv4V4LLyoq0F7n4/PHYrXPfTaI2cJEmSpOTpwUhZX3jhhRf42Mc+Rl5eHgAf+chHaGxsZNGiRVx77bWJdkePRnU0rrvuOn7+859z2WWX8eCDD/LVr371lPdYvXo1NTU1jB8/HoAbbriBO+64g1tuuYWcnBxuvPFG5s+fz/z58wGYO3cuCxYs4JOf/CQf//jHe/srn1L/WAPXWfV5cPk/wKrH4fX7kt0bSZIkSf1IW1sbxcXFvPHGG4lj5cqVQBTwnnrqKfbt28fSpUt5//vff8b3ycjIYPHixVxzzTU8/vjjzJs3D4C77rqLb33rW2zZsoUZM2ZQW1vbK9+rp/pfgAO46BaoeQ88+d9h77pk90aSJElSErznPe/hkUce4ciRIzQ0NPCb3/yGvLw8ampqeOihhwAIw5A333wTgIKCAmbNmsXXvvY15s+fT3p6+invMWHCBDZt2sS6dVHuuO+++3jve9/LwYMHqaur46qrruLf//3fE/dYv349c+bM4fbbb6e8vJwtW7b00bfvXv8McGlp8LHvQ3oW/OoL0Nqc7B5JkiRJOssuuOACrrvuOqZNm8aVV17JrFmzALj//vv50Y9+xLRp05gyZQqPPvpo4prrrruOn/70p1x33XU9ukdOTg4LFy7k2muv5dxzzyUtLY0vf/nLNDQ0MH/+fM477zwuueQSvvvd7wLwjW98g3PPPZepU6dy8cUXM23atN7/4icRhGF4Vm/YbubMmeGSJUtO3mjFo/CL6+HSv4mmVUqSJEk6K1auXMmkSZOS3Y0Bo7t/zyAIloZhOPN0Pqd/jsC1m3w1nP9ZeOG7sOmlZPdGkiRJkpKqfwc4gHnfhpIa+NVNcOTsl+mUJEmSlLpuvvlmpk+ffsyxcOHCZHfrjPWvbQS6k10QbS3woyvg8b+Ga+5xawFJkiRJPXLHHXckuwu9qkcjcEEQzAuCYHUQBOuCILitm/dHBkHwxyAIXg+C4K0gCK7q1V4OmwHv+ztY/it46+e9+tGSJEmSupesehkDTW/+O54ywAVBkA7cAVwJTAY+HQTB5C7N/h74RRiG5wOfAr7Xaz1sd8lfw8iL4Ymvw76Nvf7xkiRJkjrk5ORQW1triHuXwjCktraWnJycXvm8nkyhnA2sC8NwA0AQBA8CVwMrOvcLKIw/LwK290rvOktLh49/H+68JFoP97knIb3/zwCVJEmSUtHw4cPZunUre/bsSXZXUl5OTg7Dhw/vlc/qSQIaBnTenW4rMKdLm38CfhcEwa1APvCBXuldV8UjYf534Zc3wgvfgfcdN5tTkiRJUi/IzMykpqYm2d1QF71VhfLTwL1hGA4HrgLuC4LguM8OguCmIAiWBEGw5IyT/LnXwHmfgj/9G7zz6rvqtCRJkiSlkp4EuG3AiE6vh8fPdXYj8AuAMAxfBnKAsq4fFIbhD8IwnBmG4czy8vIz6zHAVf8bikbAr74IjfVn/jmSJEmSlEJ6EuBeA8YFQVATBEEWUZGSx7q0eQe4HCAIgklEAa7vJsvmFMLH74a6LfDk3/bZbSRJkiSpPzllgAvDsAW4BXgaWElUbXJ5EAS3B0HwkXizvwG+GATBm8ADwIKwr8vVjJwD7/lbePMBePvhPr2VJEmSJPUHQbLKgs6cOTNcsmTJu/uQ1hZYOA/2rIGvvATFI059jSRJkiT1A0EQLA3DcObpXNNbRUySIz0DPv4DCFvh11+CttZk90iSJEmS+kxqBziAkjFRUZPNL8FL/1+yeyNJkiRJfSb1AxzAtE/DlI/BH/8nbPtzsnsjSZIkSX1iYAS4IID5/w4FlfDLL0DToWT3SJIkSZJ63cAIcAC5Q+Bj34d9G+Cpv0t2byRJkiSp1w2cAAdQcylc8t/gzz+Glb9Jdm8kSZIkqVcNrAAH8L7/B6qnw2O3Qv2OZPdGkiRJknrNwAtwGVnwiR9Cy1F45MvQ1pbsHkmSJElSrxh4AQ6gbBzM+1fY8By88r1k90aSJEmSesXADHAAF9wAE+fDs9+EHW8luzeSJEmS9K4N3AAXBPAX/wG5JfGtBQ4nu0eSJEmS9K4M3AAHkF8KH7sT9q6G3/9DsnsjSZIkSe/KwA5wAOe8Hy66BV67G1Y/lezeSJIkSdIZG/gBDuDyf4DKqfDozXBwd7J7I0mSJElnZHAEuIzsaGuBpoPwyFchDJPdI0mSJEk6bYMjwAFUTIIP/jOs+z0svjvZvZEkSZKk0zZ4AhzA7C/CuCvgd38Pu1cmuzeSJEmSdFoGV4ALArj6DsiORVsLNDcmu0eSJEmS1GODK8ABFFTAR78Hu5bBs7cnuzeSJEmS1GODL8ABjP8QzPoivHIHrHs22b2RJEmSpB4ZnAEO4Ip/hrIJ8MhX4FBtsnsjSZIkSac0eANcZm60tcCR/XDfR2HrkmT3SJIkSZJOavAGOIDq8+CahdHm3j+8HH71JajfkexeSZIkSVK3BneAA5g0H25dApf837D8V/CfM+CF/9cKlZIkSZL6HQMcRNsKfOAf4eZX4ZzLouqU35sDKx+HMEx27yRJkiQJMMAdq2QMfOp++OwjkJEDP/+/ovVxbvotSZIkqR8wwHXnnMvgyy/Blf8Ltr8Od86F3/5tVPBEkiRJkpLEAHci6Rkw50tw6+swYwG8djf8xwXw2g+htSXZvZMkSZI0CBngTiW/FOZ/F770AlROgSf+Bn7wXtj4QrJ7JkmSJGmQMcD1VNVUuOE38MmfQGM9/Hg+/OJ62L852T2TJEmSNEgY4E5HEMDkq+GWxXDZ/4A1v4M7ZsMf/gWaDiW7d5IkSZIGOAPcmcjMhff+bbR/3KS/gOf/F/zXLHj7YbcdkCRJktRnDHDvRtFw+MQP4fNPQ34Z/PJGuGdeVLlSkiRJknqZAa43jLwQvvhH+Mh/wr718IPL4NFb4OCeZPdMkiRJ0gBigOstaelwwfVw61K46GZ48wH4zwtg0X9CS1OyeydJkiRpADDA9bacIvjQv8BXX4ERc+B3fw93XhQVPJEkSZKkd8EA11fKxsFnHoa/fCh6/bNr4f5rYe/a5PZLkiRJUsoywPW18VfAV16GK74F77wC37sQnv4f0FiX7J5JkiRJSjEGuLMhIwsuvjVaHzft0/DyHfCfM+DPP4G2tmT3TpIkSVKKMMCdTQUVcPV/wU1/hJIx8NitcPdl0cicJEmSJJ2CAS4Zhp4f7R33iR/Bwd1wz4fg4RuhbluyeyZJkiSpHzPAJUsQwLnXwK1L4D1/Cyt/A/81Ex7/a1j3rFsPSJIkSTpOEIZhUm48c+bMcMmSJUm5d7+0fzP84Vuw6nFoPgzZRVEBlIkfhrEfgOxYsnsoSZIkqRcFQbA0DMOZp3NNRl91RqdpyCj4xN3QfAQ2PBcFudVPwtsPQXoWjHlfFOYmXBWtpZMkSZI06DgC15+1tcKWV2HVE9EUywObgSDaIHzih6Oj9Jxk91KSJEnSGTiTETgDXKoIQ9i1PApzqx6HnW9F58sndYS5oedHa+skSZIk9XsGuMHkwDuw6rdRmNu8CMJWKBwWTbGc+GEYfQmkZya7l5IkSZJOwAA3WB3eB2uejsLcumeh5QjkFMH4eVGYO+dyyC5Idi8lSZIkdWIRk8EqrwSmfzo6mg7Dhj9GUy1XPwlv/RzSs+Gcy6IwN/5KKChPdo8lSZIknQED3ECTldexJq61Bba8Ei+C8jiseQoIYOSFMHE+TLwKSsYku8eSJEmSesgplINFGMLOt+NFUJ6AXW9H5yumdAS+6mkWQZEkSZLOEtfAqef2b4oXQXkC3lkEYRsUDu8Ic6MutgiKJEmS1IcMcDozh2phzZNRmFv/B2hphJxiGPuB+HG5m4dLkiRJvcwiJjoz+aVw/meio+lQFOJWPQHrnoFlD0dtqs6DcR+MAt3wWY7OSZIkSUngCJxOrK0tWiu37hlY+wxseTXaby67EMa8t2OErmh4snsqSZIkpRxH4NS70tKiwibV0+DSv4HGOtjwpyjQrXsGVv4malc+KZpmOfYD0dq5jOzk9luSJEkaoByB05kJQ9izGtb9PgpzmxdBaxNk5sHoS+PTLS93mwJJkiTpBByB09kTBFAxMTouvjVaO7fpxfh0y9/D2qejdiVjOqZajr4EsvKT229JkiQphTkCp75Rux7WPRsFuo3PQ8sRSM+Opli2B7ryCe47J0mSpEHLbQTUPzU3wjsvd6yd27MqOl84vGPt3Jj3QU5hMnspSZIknVUGOKWGA1tgfXx0bsOf4Gg9pGXAiDkdga7y3KiIiiRJkjRAGeCUelqbYcvijtG5nW9F5/MrOjYRP+f9kFeS3H5KkiRJvcwAp9TXsCvaSHzdM9Eo3ZH90fkhNVA1FSrjR9VUKBrpKJ0kSZJSllUolfpilTD909HR1grbX4cNf4Sdb8POZbDycSD+R4esGFROiY72cFcxGbILkvoVJEmSpL5igFP/lZYOw2dGR7umQ7B7JexaFgW6Xcvg7YdgyY862jhaJ0mSpAHKAKfUkpV/fKgLQzjwDuxaHgW69nDnaJ0kSZIGGAOcUl8QwJBR0THxqo7zjtZJkiRpgDHAaeBytE6SJEkDjAFOg8u7Ha2rnAJl46FsHJSOg7KxkDvk7H8PSZIkDUoGOAl6Plq3azmseQraWjra5ZVB6dgozJWO6wh3Q0ZDRtZZ/yqSJEkauAxw0omcaLSutRn2b4LadbB3LdSuhb3rYM3TcOinna5Pj0Jc6dh4qBvbEe4KKqLPlyRJkk6DAU46XemZURArGwcTrjz2vSMHomDXNdxt/BO0NHa0yy6MAl3XcFdyDmTlnd3vI0mSpJRhgJN6U27x8VMxAdraoG5LR6CrXRc937wI3v7FsW2LRnQf7gqHWx1TkiRpkDPASWdDWlrHdMyxHzj2vaZDULu+U7hbG43evfkANB3saJeRC6XndAp346FiYvSYmXt2v48kSZKSwgAnJVtWPlSfFx2dhSEc3HXsVMzatbDjTVj5GIRtUbsgDUrGQPnEaLuDivhj6dhouqckSZIGDAOc1F8FAcSqoqPm0mPfa2mCfeujrQ92r4Q98cfVv+0IdmmZUYirmNTpmBwVVklLP+tfR5IkSe+eAU5KRRlZHaGss+ZG2Lvm2FC3bSks/1Wna3Pi0y87hbryidHaO9fYSZIk9Ws9CnBBEMwD/g+QDvwwDMNvd3n/34HL4i/zgIowDIt7s6OSeiAzp/vpmEcPwp7VHaFu9wrY+AK89fOONlkF8WmYEztCXcXkaATQLQ8kSZL6hVMGuCAI0oE7gA8CW4HXgiB4LAzDFe1twjD8607tbwXO74O+SjpT2QUwfEZ0dHZkfxTsdq+A3auix9VPweud9rPLKe4YrSvvNGqXX3p2v4MkSZJ6NAI3G1gXhuEGgCAIHgSuBlacoP2ngX/sne5J6lO5Q2DkhdHR2cE98dG6eKjbvRKW/RIa6zra5Jd3hLqycVA4FGLV0WN+uevsJEmS+kBPAtwwYEun11uBOd01DIJgFFAD/OEE798E3AQwcuTI0+qopLOooDw6at7TcS4MoWFHR+GU9nV2r/8Umg8de31aBhRUQWF1R6jr7tFNyyVJkk5Lbxcx+RTwcBiGrd29GYbhD4AfAMycOTPs5XtL6ktBEAWvwqEw9vKO821t0XYHDduhfkcU8uq3dzzuWQXr/whNDcd/Zk4RxIbGg97Q7gNfXpnFVSRJUv/W1hb9QfvowWgf36MN8ceTvT506s/tRk8C3DZgRKfXw+PnuvMp4OYz6omk1JSWFgWvwupovP5EjjbEA972Lo/xoLd7ZRQE27dBSHx+Znw7heouQa9L4HMzc0mS1FNhCC2N8UDV0ClY9eR1N4Gs6RDQk/GpICocl10QPZ6BngS414BxQRDUEAW3TwF/eVxXgmAiMAR4+Yx6Imlgy45BeQzKx5+4TWsLHNrdKeBtP3Y0b9dyWPvM8VM2ISq2UjgUCofBkFFQPLLTMQrySq2mKUnSQBOG0Rr9I/ui4myH90fPD+/rdG7fseca66Lg1f2kweNl5HYEruwCyIpBQQVkjel43fX9E73OzDt2ZtFfnf7vJqcMcGEYtgRBcAvwNNE2AveEYbg8CILbgSVhGD4Wb/op4MEwDJ0aKenMpGd0TNNkRvdtwhCO1p94NK9uK2x9DRoPHHtdZn6XUNc14JUY8CRJSqbmI90HrsS5/d2fO2EQC6LlGnklkFsSFVkrnxD90TcRsGLHjoh1fZ1VEP1+0o8EycpbM2fODJcsWZKUe0saBBrr4MAWOLAZDrzT6Yi/7lxRE6L/QJ8o3BWPjCp2GvAkSeqZMIwCVsMOOLgTDu4+QTDrNGLWcuTEn5eZF4Ww3CGQNyR63h7Mcod0PO98Lre431fFDoJgaRiGM0/nmv4VJyWpt+QUQVURVE3t/v0jB7oEu07H5kXRKF9nWbHjw13nqZo5xQY8SdLAF4ZR8GqIz3xp2BV/3BkFtYb4cXAXtDYdf32QHg9Z8RBWPAKqp8VDWZdg1rldZs7Z/679lAFO0uCUWxwd1ed1//6R/ScOeJtePL6qZnbh8SN3scrjfwBl5Rv0JEn9zzHBrD2I7YhXmm4PavGQ1l0wyymOFx2rgtGXQEFlVGis/VxBRbQePbvQn4PvkgFOkrqTG/9LYPW0499r/yHXXbjbvxk2Ph9VpOpOelaX6R5Djg143f3VMXcIZGT17feVJA1MiWDWNZDtPPY4YTArioJYQSWMurgjkLVXiC6ojJ5bDfqsMcBJ0ukKgihg5ZXA0OnHv9/+w/Lgrm4WY+8/9nnteti6JDrX3Q/Odlmxbub9n2gNQPwxu8g99CSpt7W1RiXjmw93ejwcVUhuOhwV4mhrhraW+NEaP1qiYhtdz7UfYdsprunmuuOuaTn2upaj0dqz1qPHf4/soo4gNuqiLoGs08iZwazfMcBJUm/rHPB6KgyjXwS6LXm8//hzB96Jtz3ACfedCdKiKS3djerlDeGEC78zc53eIim1tTR1ClQnClqnev8E57sLQ2cqSI+KbKRlREeQ1vE8LSP6I1zivfZ2Xa5Jy4CMnE6vO72fnhVVXuwcyGJVUFAFWXm99z10VhngJKk/CIKoZHF2vBpmT7W1xve/OVnZ5fi59r30Du/rfi+9dunZ3YS+E1T46vy8n5VZlpSiWpqi/641HujyWBf90ar9edf3mjoFs7aW07tnRk5U5TArP/6YF20/U1DR/fmsvJOfT8/sFLi6CV3twc0/lukM+NNWklJZWnrHaF/pOT2/ruVoN1M6u4a++Ht713ScO9kvRdmF3QS9bqZ55hRF++y0H1kFTvWUBpLWlqiSb0+DV9c2JyslD9GoUk5x9N+SnKLoefGoKEx1G6h6ELj6eal5qTMDnCQNRhnZUFgdHT0VhnC04fh1fCca/du3IXredc+97rRvnnrcUdhpU9Uu545rVxB9L0nvTktTPIB1CViJc/XHvu4azrpW6e0qSO8UvoqiisCxqmMD2TGP8TbtzzNyHLnSoGaAkyT1TBBATmF0DBnV8+taW+K/5MUD3tGG6Be/ow1djk7nmg5GC+87nw/bTn2v9KzuA+AxIbD9XEHU/rhpTRnHry9J6+58lzbHrV3JcGRRZ1/7etpuQ9eBLsGsvvtgdqoRMOL/Lcgugtx40CqpOT50dRe+coqi/y8awKQzZoCTJPWt9AzIL42OMxWGUSGBY8LewRMHwM5Hw45oGmj765bG3vtupxSceA3MMSEwM5rKlVUQD5oF8elg8dDZ/rzre4nX8XNOA+t/2tqiohetTdHIVmtT/HVzNJW5tfkU7zd1HO3vNx06yQhZfVR98GTSszsFqsLosXDYsa+zi45vk13YEcD844SUNAY4SVL/FwQd61tiVe/us1qaohG+o/XHl/Hutrx3N22OKd3dXZtuSoSfsCR4S/yX8sNRv+q2Ro9NBzsq4PVU+xqf7sJd4nn7607BsPN7mTmd+tV87HdIvG7tKJPe2nyC1y2dzrV0et3a6XOaT36v9lHXxGhN0M3zoFObnjw/1eed4Hlba/dhqttznQLYqcLU6QrSo//d2sNVdmEUviomHRuy2kPXMWGsMHo/M6d3+yTprDLASZIGl4wsyDjNbR6SqX3PqaaD0ahj53B39GC03ijxvOt7B+HwXti/qeMzmg72bDpqbwvSopHGtIxoVLZ95PFkr4N0EttkhOHxz8P2LTRO9pyTtDnJZ3d9ThCtsUzPio6MbMgsjEazMuLn0rOj6oMZ8cf0ePvE+52u7fb9HlzvKKs06BngJEnqz9LSO9Ye9oYwjDYabg9zRw92CogN0ahR+/TO9Mzj1/V1PdeT10G6U+4kqZcY4CRJGkyCIL7eLg+oSHZvJEmnyT+HSZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJMDs6bAAACAASURBVElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoowwEmSJElSijDASZIkSVKKMMBJkiRJUoroUYALgmBeEASrgyBYFwTBbSdo88kgCFYEQbA8CIKf9W43JUmSJEkZp2oQBEE6cAfwQWAr8FoQBI+FYbiiU5txwN8Bc8Mw3B8EQUVfdViSJEmSBquejMDNBtaFYbghDMMm4EHg6i5tvgjcEYbhfoAwDHf3bjclSZIkST0JcMOALZ1eb42f62w8MD4IgpeCIHglCIJ5vdVBSZIkSVLklFMoT+NzxgHvA4YDzwdBcG4Yhgc6NwqC4CbgJoCRI0f20q0lSZIkaXDoyQjcNmBEp9fD4+c62wo8FoZhcxiGG4E1RIHuGGEY/iAMw5lhGM4sLy8/0z5LkiRJ0qDUkwD3GjAuCIKaIAiygE8Bj3Vp8wjR6BtBEJQRTanc0Iv9lCRJkqRB75QBLgzDFuAW4GlgJfCLMAyXB0FwexAEH4k3exqoDYJgBfBH4BthGNb2VaclSZIkaTAKwjBMyo1nzpwZLlmyJCn3liRJkqRkC4JgaRiGM0/nmh5t5C1JkiRJSj4DnCRJkiSlCAOcJEmSJKUIA5wkSZIkpQgDnCRJkiSlCAOcJEmSJKUIA5wkSZIkpQgDnCRJkiSlCAOcJEmSJKUIA5wkSZIkpQgDnCRJkiSlCAOcJEmSJKUIA5wkSZIkpQgDnCRJkiSlCAOcJEmSJKUIA5wkSZIkpQgDnCRJkiSlCAOcJEmSJKUIA5wkSZIkpQgDnCRJkiSlCAOcJEmSJKUIA5wkSZIkpYikBbjDTa3JurUkSZIkpaSkBbiNew+xdPP+ZN1ekiRJklJO0gJcRlrADfcsZunmfcnqgiRJkiSllKQFuDHlBZTHsrn+R4Y4SZIkSeqJpAW4zPSAB2+6kMrCHK7/0WKWbDLESZIkSdLJJLUKZWVhDg+0h7h7FvOaIU6SJEmSTijp2whUFubw4E0XUlWUww33LGbxRkOcJEmSJHUn6QEOoCIe4qqLcliwcDGvbqhNdpckSZIkqd/pFwEOoCIWTaccWpzLgoWv8YohTpIkSZKO0W8CHMRD3BcvZPiQXD638DVeXm+IkyRJkqR2/SrAAZTHsvlZe4i7dzGL1u9NdpckSZIkqV/odwEOohD3wE0XMrIkj8/f+xqL1hniJEmSJKlfBjiAsoJoJG5UST6f//FrvGSIkyRJkjTI9dsAB+0hbg6jS/P5/L2v8eJaQ5wkSZKkwatfBziA0oJs7v/CHGrK8rnxx6/xwto9ye6SJEmSJCVFvw9wEIW4n33xQmrK8vnCj5fw/BpDnCRJkqTBJyUCHEBJfhYPfPFCxpQX8IWfLOFPhjhJkiRJg0zKBDiAIflZ/OwLcxhbXsAXf7KE51bvTnaXJEmSJOmsSakAB1GIu/8LcxhXUcBNP1nKHw1xkiRJkgaJlAtw0BHixlcV8KWfLOWPqwxxkiRJkga+lAxwAMV5Wfz0xniIu28pf1i1K9ldkiRJkqQ+lbIBDqIQd/+NFzKhKsaX7lvKsysNcZIkSZIGrpQOcABFeZn89AtzmFRdyJd/upRnVhjiJEmSJA1MKR/gAIpyM7nvxjlMri7kK/cv5feGOEmSJEkD0IAIcBCFuJ/cOIfJQ4v46v1L+d3yncnukiRJkiT1qgET4KB9JG42U4YW8dX7/8zThjhJkiRJA8iACnAAhTmZ/OTG2Zw7vIib7/8zTy0zxEmSJEkaGAZcgIN4iPv8bM4bXsQtP/szT769I9ldkiRJkqR3bUAGOIBYTiY//vxspo0o5pYHXue3hjhJkiRJKW7ABjjoCHHnjyjm1gde54m3DHGSJEmSUteADnAABdkZ3Pv52Vwwspi/evB1Hn9re7K7JEmSJElnZMAHOIhC3MLPRSHuaw++wW/eNMRJkiRJSj2DIsBBfCTuc7OZMXIIX3vwdR4zxEmSJElKMYMmwAHkZ2ew8HOzmDm6hP/24Os8+sa2ZHdJkiRJknpsUAU4iELcvZ+bxazRJfz1z98wxEmSJElKGYMuwAHkZUUjcbNrohD3/T+t59DRlmR3S5IkSZJOalAGOIiHuAWzec/4cv71yVVc+D+f5Z8fX8Hm2kPJ7pokSZIkdSsIwzApN545c2a4ZMmSpNy7szAMeX3LAe59aRO/fXsHrWHI5RMrWHBxDXPHlhIEQbK7KEmSJGkACoJgaRiGM0/rmsEe4DrbVd/I/a9s5v5X36H2UBPjKgq44eLRfPyCYeRlZSS7e5IkSZIGEANcL2lsbuWJt3awcNFGlm2rpzAng+tmjeD6i0YzoiQv2d2TJEmSNAAY4HpZGIb8+Z39LHxpE08u20lbGPKBSZV87uLRXHSO0yslSZIknbkzCXDOCzyJIAiYMaqEGaNK2FF3hPtfeYefLX6H36/YxfjKAhZcXMNHzx/q9EpJkiRJZ4UjcKepsbmV37y5nXsXbWL59nqKcjP51KwRfObCUU6vlCRJktRjTqE8i8IwZMnm/dz70iaeWr6TMAz54ORKbrh4NBeNcXqlJEmSpJNzCuVZFAQBs0aXMGt0CdsPHOGnr2zmgcXv8PTyXUyojLFg7mg+On0YuVnpye6qJEmSpAHCEbhe1NjcymNvbGfhok2s3BGfXjl7BJ+9cBTDhzi9UpIkSVIHp1D2E2EYsnjjPu5dtImnl+8E4IrJVSyYO5o5NSVOr5QkSZLkFMr+IggC5owpZc6YUrYdOMJ9L2/mwdfe4anlO5lYFeNzc0dz9fRh5GQ6vVKSJElSzzkCd5YcaWrl0Te2ce+iTaza2UBxXiafnj2Sz144iqHFucnuniRJkqSzzCmUKSAMQ17ZsI97F23k9yt2EQQBH5pSyYKLa5g1eojTKyVJkqRBwimUKSAIAi46p5SLzilly77DieqVv317J5OrC/nY+cO4ZFwZE6tihjlJkiRJx3AErh843NTCI69v5ycvR9MrAcoKspg7toxLxpZx6bhyqopykttJSZIkSb3KKZQDwPYDR3hx3V5eih97DzYBMLaigEvige7Cc0opyHbwVJIkSUplBrgBpq0tZNXOBl5ct4cX19WyeGMtjc1tZKQFnD+ymLljy7h0XBnThheTkZ6W7O5KkiRJOg0GuAGusbmVP2/ezwvr9vLi2r0s215HGEIsO4MLzynl0nFlzB1bxpiyfNfPSZIkSf1cnwW4IAjmAf8HSAd+GIbht7u8vwD438C2+Kn/CsPwhyf7TAPcu7f/UBOL1tfy4ro9vLB2L1v3HwFgaFEOl4wr45Jx5cw9p5TSguwk91SSJElSV30S4IIgSAfWAB8EtgKvAZ8Ow3BFpzYLgJlhGN7S0xsb4HpXGIa8s+8wL6yNRucWrd9LfWMLAJOrC7l0XBmXjCtj1ugSNxCXJEmS+oG+2kZgNrAuDMMN8Zs8CFwNrDjpVTqrgiBgVGk+o0rz+cyFo2htC3l7Wx0vro1G5+55aSPff34DWRlpzBo9hEvGlnPpuDImVxeSluZ0S0mSJCkV9CTADQO2dHq9FZjTTbtPBEHwHqLRur8Ow3BLN210lqSnBUwfUcz0EcXc8v5xHDrawuJN+3gxPkL3b0+t4t+egpL8LC7utH5u+JC8ZHddkiRJ0gn0Vi363wAPhGF4NAiCLwE/Bt7ftVEQBDcBNwGMHDmyl26tnsjPzuCyCRVcNqECgN31jby4bm90rN3L42/tAKCmLD/armBcGXNqSijOy0pmtyVJkiR10pM1cBcB/xSG4Yfir/8OIAzDfz1B+3RgXxiGRSf7XNfA9R9hGLJ290FeWBvtPffKhloON7UCMLEqxuyakugYXUJFoRuKS5IkSb2hr4qYZBBNi7ycqMrka8BfhmG4vFOb6jAMd8Sffwz472EYXniyzzXA9V9NLW28seUAizfW8urGfSzdvD8R6EaX5jG7poRZo0uYU1PKiJJctyyQJEmSzkCfFDEJw7AlCIJbgKeJthG4JwzD5UEQ3A4sCcPwMeCvgiD4CNAC7AMWnHbv1W9kZaQlRt1uAVpa21i+vZ7XNu3j1Y37+N2KXfxiyVYAqgpzokBXU8KcmhLGlhdYFEWSJEnqI27krdPW1hZNuVy8aR+LN+5j8cZadtUfBWBIXiazRpckAuDk6kIy0tOS3GNJkiSp/+mrbQSkY6SlBUyoijGhKsZnLxxFGIZs2XeEVzfWRoFuUzRKB5Cflc4Fo4Ywp6aE2TWlnDe8yH3oJEmSpDPkCJz6xK76xvjoXHSs3tUARNMzpw8vTozQXTBqCAXZ/h1BkiRJg0+fFDHpKwa4wWX/oSaWbN7P4o21LN60n2Xb6mhtC0lPC5gytJDZ8WmXs0aXMCTfrQskSZI08BnglDIOHW3hz+/sT4zQvb7lAE0tbQCMryyIj9CVMnt0CVVFbl0gSZKkgccAp5R1tKWVt7bWJQLd0s37OXi0BYARJblMG17M1GFFTB1axJShhY7SSZIkKeUZ4DRgtLS2sXJHA4s37eO1jftYtr2OrfuPJN4fVpzL1GGFTB1axNRhRUwZVkhFzJE6SZIkpQ4DnAa0A4ebWL69nmXb6li2vZ7l2+rYsPdQ4v2KWHZ8lK6QKcOiYDe0KMeNxiVJktQvuY2ABrTivCzmji1j7tiyxLmGxmZW7miIh7o6lm+r57nVu2mL/11iSF5mNEI3tCgxYjeyJM/NxiVJkpSSDHBKabGczMSWBO2ONLWyamd9YpRu2fY6fvTiBppbo1QXy85g8tDCaLQuHurGlBeQbqiTJElSP2eA04CTm5XO+SOHcP7IIYlzTS1trNnVwPLtdSzbVs+y7XXc/+pmGpujype5melMqo51FEoZVsi4ihhZGWnJ+hqSJEnScVwDp0GrpbWNDXsPRdMv46Fuxfb6RPXLrPQ0JlTFmDqsMD4Fs4iJVTFyMtOT3HNJkiQNBBYxkd6ltraQzfsOH7Ombtn2Og4cbgYgPS2gpiyfiVUxJlUXMrEqxsTqQoulSJIk6bRZxER6l9LiAa2mLJ+/mDYUgDAM2XbgCMu21bN8ex2rdjbw5tYDPP7WjsR1sZyMKMxVFTKxOnqcUBWjINv/i0mSJKn3+NuldApBEDB8SB7Dh+Qxb2pV4nxDYzNrdjWwamcDq3Y0sGpnPY+8vo2GV1oSbUaU5DKxqpBJ8ZG6CVUxRpfmWzBFkiRJZ8QAJ52hWE4mM0aVMGNURwXM9tG6VTsaWL2rgZU76lm1s4FnV+5KbG2Qk5nG+MpYx4hdPNyV5Gcl6ZtIkiQpVbgGTjoLGptbWbf7ICt31LN6ZzRqt3JHPbWHmhJtKmLZTKxuH62LMaGykHMq8snOsGiKJEnSQOQaOKmfyslMj+87V3TM+T0NR1m1sz4+BTOahrnwpVqaWqPtDTLSAs4pL0isq4seY1QVWjRFkiRpMDLASUlUHsumPFbOpePKE+eaW9vYtPcQK3c2sDoe7pZs2s+jb2xPtCnKzWRSdYxpw4uZNiI6rIQpSZI08BngpH4mMz2NcZUxxlXGIF4JE6DuSHN8+mW0rm75tjoWvrQpMVpXVpDN9BFFHaFueDFFeZnJ+hqSJEnqAwY4KUUU5WYyu6aE2TUdRVOOtrSyake0rcEbWw7w5pYDPLNyd+L9mrJ8pg0vSozSTa4udCNySZKkFGYRE2mAqW9s5u2tdYlA98aWA+xuOApEa+omVRcyLT5SN31EMeeUF5DmtgaSJEln3ZkUMTHASYPAzrrGKNBtjULdW1vrOHg02q+uIDuDc4dFo3TTR0SPFkmRJEnqe1ahlNStqqIc5hVVJTYib2sL2bD3IG9sqePNeLD70YsbaG6N/qBTEcuOB7poLd25w4soynU9nSRJUrIZ4KRBKC0tYGxFjLEVMa6ZMRyI9qpbuaM+HujqeHPrAX6/YlfimjHl+UzvVPVyUnXMPeokSZLOMgOcJCDaq+78kUM4f+SQxLm6I828tbV9LV0dz6/dy69e3wZAZnrA5OpCpo0o5txhRUyqLmRsRYFFUiRJkvqQa+Ak9VgYhuyoa4wCXTzYvb21jkNNrQCkBTC6LJ9JVYVMqIoxoSrGpKpChg/JtVCKJElSF66Bk9SngiBgaHEuQ4tzufLcagBa20I21R6K9qjbEe1Rt2x7HU+8vSNxXV5WOuMrY0yqjjGhMsaEqkImVsUYkp+VrK8iSZKUkhyBk9QnDh1tYc2uhvjm49EG5Kt3NrD/cHOiTWVhNhOqCpkUH62bUBVjbEWBa+skSdKg4AicpH4jPzvjuDV1YRiyu+Eoq3Y2sHpnNFq3akcDC9fX0tTaBkB6WsCYsvxo+mV1IRMqY0ysjjGsONetDSRJ0qBngJN01gRBQGVhDpWFObx3fHnifEtrG5tqD7FyR/uIXT1vbDnA4291TMOMZWcwvirGxPgxIb7Ozu0NJEnSYOIUSkn9VkNjM2t2NcRH7KLRulU766lvbEm0GVqUE59+Wcik6mjUbkxZPhnpaUnsuSRJ0qk5hVLSgBLLyWTGqBJmjCpJnAvDkJ31jYnpl+1TMV9ctzexEXlOZhqTqws5d1gRU4cVce7wIsaWFxjqJElSynMETtKA0NTSxsa9h1i+vY5l2+pZtq2O5ds7tjjIzkhjUjzUtQe7cZUFZBrqJElSkpzJCJwBTtKA1doWsnHvIZZtq+Pt+LFiez0Hj0ZTMLMy0phUFYtG6eKhbnxljKwMQ50kSep7BjhJOoW2+L51b2+rSwS75dvqaWgPdelpTOgU6s4dVsT4Krc2kCRJvc8AJ0lnoK0t5J19h48Jdcu21SWKpWSmB4yvjHWsqRtWxISqGDmZhjpJknTmDHCS1EvCMGTLviOJqZftwa7uSLQReUZaFOqmDusoljKputBQJ0mSeswAJ0l9KAxDtu4/ctxI3f7DUahLTwsYV1GQGKWbMjTaqy6W4151kiTpeAY4STrLwjBk24EjnQJdVAGz9lBTos3wIblMrCqMNiGvjjGxqpDRpXluayBJ0iDnPnCSdJYFQcDwIXkMH5LHvKnVQBTqdtQ1snJHtEfdyh31rN7ZwB9X76a1LfqjWVZGGuMrCzqCXVUhE6tjlBVkJ/PrSJKkfs4AJ0m9LAgChhbnMrQ4l8snVSbONza3sm73QVbvbGBVfAPy51bv4eGlWxNtygqymVQdY0JljInVUbgbW1Hg2jpJkgQY4CTprMnJTGdqvOBJZ3sPHmV1fKRu1c4GVu9s4L5XNnO0pQ2I1tbVlOUzsSrGpOrCeLiLMaw4lyAIkvFVJElSkhjgJCnJygqyKRubzdyxZYlzLa1tbKo9zKqd9fFw18AbWw7w+Fs7Em1i2RlMrI4xIT4Fc1J1jPGVFk2RJGkgs4iJJKWQhsZm1uyKAl17uFu1oyGxETl0FE2ZFC+YMqEqRk1ZPulpjtZJktSfWMREkga4WE4mM0aVMGNUSeJceyXMVTsaWL2rYyrmH1btIl4zhZzMNCZWFTJ5aCGTq6PHiVUx8rL8MSBJUipxBE6SBqj2oikrd9SzckcDK3bUsWJ7PfWN0WhdEEBNWT6TqwuZMrQoEe7KY1bClCTpbHAETpKU0F3RlPbRuhXb61mxo54V2+uPW1tXHstOjNJF4a6Q0aX5pDkFU5KkpDPASdIg0nnfuiumVCXO1x1ujgJdPNSt2FHPS89voCU+BzMvK52JVbF4qItG6yZUxsjNcnsDSZLOJqdQSpK6dbQlmoK5Yns9y+OhbuX2+kTBlLQAxpQXMKXTurrJ1YWUuhm5JEk94hRKSVKvyc5IZ8rQIqYMLeLa+LkwDNm6/0gi0K3YXs+STft59I3tiesqCzumYE4ZWsTk6kJGluQ5BVOSpF5ggJMk9VgQBIwoyWNESR7zpnZMwdx/qImVXaZgPr92L63xKZj5WelMSoS6KNiNr4yRlZGWrK8iSVJKcgqlJKlPtFfBjKZg1kVTMHc0cDA+BTMzPWBCVYypQ4uYMqyIqUMLmVhV6Lo6SdKg4RRKSVK/cWwVzBEAtLWFbN53mOXb61i2LQp2Ty/fyYOvbQGidXVjKwqOCXWThxYSy8lM4jeRJKn/MMBJks6atLSAmrJ8asrymX/eUCBaV7e9rpFl2+pYvr2e5dvqeGn9Xn71+rbEdaNL8+KBroipw6IpmCX5Wcn6GpIkJY0BTpKUVEEQMKw4l2HFuXyo09YGuxsao2Ip2+tZtq2Ot7Ye4IlO+9UNK85l8tDCRKibOqyIilg2QWCxFEnSwGWAkyT1SxWxHCom5HDZhIrEubrDzdH0y+3RaN2ybXU8s3IX7cu5ywqy4yN07cGuiOFDcg11kqQBwwAnSUoZRXmZXDy2jIvHliXOHTrawsodUZhbFt+z7sW1HZuQF+ZkJNbitVfArCnLJ91tDSRJKcgAJ0lKafnZGcwcXcLM0SWJc43NrazZ1cCybfWJ0bp7F22iqaUNgLz4tgYTqmJMqIwxrrKACZUxNyGXJPV7BjhJ0oCTk5nOecOLOW94ceJcc2sb6/ccjELdtjpWbK/nibd28LMj7yTalOZnJcLcuMoY4ytjjK8soDjPgimSpP7BACdJGhQy09OYWBXtNXfNjOFAVAFzT8NRVu9qYM2ug6zd1cDqXQ388s/bEvvVAVTEsuNhLgp046tijKsocHsDSdJZZ4CTJA1aQRBQUZhDRWEOl44rT5xv39pgza6GKNTtPMja3Q08sPgdjjS3JtoNLcphfFUU7MZVFDChKsbYigLysvzxKknqG/6EkSSpi85bG3SugtnWFrJ1/xHWxEfqohG7gyxaX5tYXxcEMGJIHuMrCxhX2bHG7pzyAnIy05P1lSRJA4QBTpKkHkpLCxhZmsfI0jw+MLkycb6ltY139h1mTXwqZnu4e271nkQ1zLQARpfmM66yoNN0zBg1ZflkZaQl6ytJklKMAU6SpHcpIz2NMeUFjCkvYN7UjvNNLW1sqj0UBbudUbhbs7uB36/YRTzXkZEWUFOWz/h4Rcz2ypgjSvLc6kCSdBwDnCRJfSQrIy0x0sZ5Hecbm1vZsCce7OLHW1sP8MRbOxJtcjI7rp1QGWN8VYyJVTEqYtluTC5Jg5gBTpKksywnM53JQwuZPLTwmPOHjrawdvdBVu+sZ/XOg6zZ1cCf1uzh4aVbE22KcjMTI3WJUbvKGEV5VsSUpMHAACdJUj+Rn53B9BHFTB9RfMz52oNHo+mXuxpYtTMasXvk9W00dNrqoKowJx7oojV2E6sKGVtRQG6WhVMkaSAxwEmS1M+VFmRzUUE2F51TmjgXhiE76hpZvTOqiLlmZxTuXtlwbEXMUSV5/3979xokV1rXcfz37/tlerp7MvfJbZPNbtAF9xKwClChLCnxhWCJqCgFvoEXUAUvtFBLCwrLKrQUyxeUgkq5FOgqyMqWIHdZLrUsm8V1s5sNm2z2kpnMLZnpufRluqfn8cU53dM9l+xMkkn3Sb6fqqnT/ZzT3acrJyfzy/+5NMfVNSp2h/vTioaZOAUAgogABwBAAJmZRnNJjeaSeuPx9aUOVutrenGu1Ax0jSUPWidOiYVDOjKQ9rphDnlj6+4Yymgsl1SIiVMAoKsR4AAAuIlEwiEdHfDWnXvzK0ea7ZVaXedmlpuB7idTS3rs+Tl96YmLzWPSsXCzSnd8OKM7h3t1fDijfDrWia8CANgCAQ4AgFtAIhrWXWNZ3TWWbWtfrNR0tjG2zq/affXpKT3w2IXmMYOZuO4czugVI73NCVRuH2RhcgDoBAIcAAC3sN5EVPcd6tN9h/qabc45zSyt6MzUkn4ytagzk16w++cfvKBq3RtfFw6ZDu9L6bhfpbtz2Js4ZX+ebpgAsJcIcAAAoI2Zaag3oaHehH7hjoFm+2rdW5jcC3ZeqHtyoqAvn1pfvy4dC+uY3wWTbpgAcP2Zc64jH3zixAl38uTJjnw2AAC4fpZXVr2xdc1gt6gzU0sqlGrNYxrdMI/7lTq6YQKAZGaPO+dO7OY1VOAAAMA16YlHdO/BvO49mG+2beqG6Ye7+x95sbnMQWs3zNZwRzdMANgeAQ4AAFx3V+6GWdKZqcVmN8xTEwtt3TBTsbDuGPJmwzwykNaRgR4dGUjrYF+K9esA3PLoQgkAADqu6HfDPNPSDfPczLIuLVebx0RCpoN9Kd3Wn14Pdv3etr8nJjOqdgCCZc+6UJrZL0v6W0lhSf/onPvYNsf9uqQvSHq1c450BgAAdiQdj+ieg3nd09INU5IWSjWdv7Ss87PF9e1sUd87d6nZFVOSMomIjgz06Kgf7m7r7/G3acbZAbipvGyAM7OwpE9I+iVJ45IeM7OHnHOnNxyXkfQBSY/uxYkCAIBbTzYV3TLY1decLhbKOn+pqPOz6wHvkfOX9cX/nWgeZyaNZpM6MpDWUb8r5pH+Ht02kNZIb4KxdgACZycVuNdIOuecOy9JZvaApLdIOr3huD+T9BeS/uC6niEAAMAG4ZDpQF9KB/pSbWPsJKlUXdX52aKev1Rsq9x9/uQFFav15nHJaFiH/Yrd0f71sXa39aeVSURv9FcCgB3ZSYAbk3Sh5fm4pJ9tNhoWGgAADlhJREFUPcDM7pV0wDn3ZTMjwAEAgI5JxSK6ayyru8aybe2NmTGfm13vivn8pWU9NbGg/z41qbWWaQEGM3F/rF2PjvrVu9sHezSWY4ZMAJ11zbNQmllI0sclvXsHx75H0nsk6eDBg9f60QAAADvWOjPma4/2t+1bWa3rpcslPddSsXv+UlFffWpS8y3r2SWjYR0dTOvYoLeO3bFBL9gd7EspwgyZAG6AnQS4CUkHWp7v99saMpLukvQdf/anYUkPmdmvbpzIxDn3KUmfkrxZKK/hvAEAAK6beCSsY0MZHRvKbNo3V6zq/Oyyzs4s6+z0ss7OLOnR85f1YMtYu1g45HXF9EPdscGMjg316PC+tGIRgh2A62cnAe4xScfM7DZ5we23JL2jsdM5tyCp+d9YZvYdSb/PLJQAAOBm0JeOqS/dpxOH+9ralyo1PTdb1NnpJZ2bXda56WWdGl/QV05NqrFKUzhkOrQv1azUNSp3Rwd6lIwxOyaA3XvZAOecWzWz90v6mrxlBD7tnHvazD4q6aRz7qG9PkkAAIBuk0lEdfeBnO4+kGtrr9Tqem52Wef8it25Ga9q981nZlT3B9qZSfvzSa9SN9jTrNzdPtjDBCoAroiFvAEAAG6A6uqaXrhcbAa7szNLOjfjjber1tfXtBvJJnT7hordscEe5dOxDp49gL2wZwt5AwAA4NrEIiHdMZTRHUMZ6ZXr7av1NV2YL7d1xTw7s6wHfnRB5dr6sgf9PTEdHfDG1Q1nE82fkWxCI71J9SYj8ucjAHATI8ABAAB0UCQc0m393vpzb2ppX1tzurhQ1tkZL9Sdm1nWszNL+vZPZnRpeUUbO1EloiGNZJMa7vVC3ZAf7oZ718NefzrOMghAwBHgAAAAulAoZNqfT2l/PqU33jnYtq9WX9PM0oqmFsqaXKhoyv+ZXPS2jz4/p+nFilbX2lNeJOQtpdCs3vW2V/KGs0kNZuKKsiQC0LUIcAAAAAETDYc0lktqLJfc9pi1NadLxZVmuJtarLSFvdMXF/WtZ6ZVqa21vc5M6u+JN6t37dW8pBf4ehPMogl0CAEOAADgJhQKmQYzCQ1mEnrV/q2Pcc5poVxrC3eTCxVN+9W8Fy4X9cj5y1qqrG56bS4V1Wg2qQN9Sb9S6G0bz3vi/JoJ7AX+ZgEAANyizEy5VEy5VEzHh3u3Pa64sqqpxZaAt1jRxUJZFwtlnZ8t6uFnZzdV8nKpqPbnkzrQEu7255M60JfSWC6pNAEPuCr8zQEAAMAVpeMRHR3wFiDfinNOl4tVjc+XNT5f0oU5bzs+X9az00v69pkZray2B7y+dGxDwEtqf19KB/JJjeVSdNEEtkGAAwAAwDUxM/X3xNXfE9+0sLnkBbzZ5RU/4JV1Ya7UDHvPTC7qG6en29bCk7xlE/ZvUb3bn/fG/iWiBDzcmghwAAAA2FNm6+Px7j2Y37R/ba0R8EqbQt5TEwv62tNTqtXbZ9QcyMR1oCXcjeaSGs15E62M5hLKJqOsi4ebEgEOAAAAHRXylzcY6k3ovkOb99fXnGaWKs2q3ficH/LmS3riQkFfOTW5acmEZDSskVxCo9mkt9h5LqnRDVsmWkEQcdUCAACgq4VDppFsUiPZpF59uG/T/vqa0+zSii4ulDVZqGjSXx9vcqGsi4WKvnt2VjNLmxc/z8QjGsklNJL1qnYjftgbzfmhL5tkLB66DgEOAAAAgRYOWXNBch3c+phafU3T/nIJFwvrC6A3Hj99cUGXlqubXpdPRZvBbmPYG80mNZSNKx4h5OHGIcABAADgphcNh/zxcqltj6nU6v4SCetVvIuFshf0Fip6/KV5FUq1Ta/r74n7oc4LdmO59TF5Y7mk+nviCoUYj4frgwAHAAAASEpEwzq0L61D+9LbHlOqrnrdMwuVTV02n79U1A/OXdbySvvC59GwNSt3ozkv4K2HPK89FePXcuwMVwoAAACwQ6nYldfEk6TFSq250PlEoaKJ+XLz+Q+fu6ypxYo2zLmifCraDHRjufWwN5pLaj9VPLQgwAEAAADXUW8iqt7hqI4P9265f7W+pumlFS/gzZc1UVgPeC9dLumR53ZWxVsPfF4bVbxbA3/KAAAAwA0UCYea3ShffXjrY9qqePNeJe/lqni5VLQZ7MZySQ1nE9qXjmlfT0z5VEz70nHl01H1xCOskRdgBDgAAACgy+y0itfonrmTKl5DLBxSPh1VXzquvsY21f48n442A19fKqZIOLSXXxe7QIADAAAAAqa1iredpUpNc8Xq5p9SVXPLVc2XqrpcrGpivqDLxaqWKlsHPknKJqPqS8fUl25U82LKp7ffpmNhqnx7hAAHAAAA3IQyiagyiegVZ9VsVV1dU6G0HvDmSluEv2JV4/MlPTle0HypqlrdbflesUhIfanYeuhLx5RPRZVLRpVNxZRLRpVLeT/ZZMzfRhWl0veyCHAAAAAAFIuENNib0GBvYkfHO+e0tLKq+aJXyWvdNsJeo8o3Pl9SoVzTQrkmt3XmkySlY2HlUjFltwh4jdDXfJ6KKuc/TkRvncXUCXAAAAAAds3MvLF6u6jyra05LVVWVShXVSjVVCjXVChVtVCuaaH5vKYFf/+z08vecaWqVjfO2tIiHgk1A122Jey1hcGWSl9jXxC7ehLgAAAAANwQoZApm4oqm4rq0L6dv845p1K1vh74WsJeoew/9x8XSjW9NFfSk+Pe80ptbdv3jYatWdHLt1T38i3hL59qD375VEypDgY/AhwAAACArmZmSscjSscjV5y4ZSuVWl0LjbBXqmq+VNNiuab5UrWt4jdfrGmiUNbpiwuaL9VUrtW3fc9G8Mu3dPNsPN4q+OXT3ri/6xH8CHAAAAAAblqJaFiJaFhDOxzb11Cp1bVY9ip980Uv7C2Uri34xcKhti6eV4MABwAAAAAbNILfTid1adgu+BXKXvWvNfhdDQIcAAAAAFwnuwl+D7x39+/PQgsAAAAAEBAEOAAAAAAICAIcAAAAAAQEAQ4AAAAAAoIABwAAAAABQYADAAAAgIAgwAEAAABAQBDgAAAAACAgCHAAAAAAEBAEOAAAAAAICAIcAAAAAAQEAQ4AAAAAAoIABwAAAAABQYADAAAAgIAgwAEAAABAQBDgAAAAACAgCHAAAAAAEBAEOAAAAAAICHPOdeaDzWYlvdiRD0cQ9Uu61OmTAHaJ6xZBxHWLoOLaRRDd6ZzL7OYFkb06k5fjnBvo1GcjeMzspHPuRKfPA9gNrlsEEdctgoprF0FkZid3+xq6UAIAAABAQBDgAAAAACAgCHAIik91+gSAq8B1iyDiukVQce0iiHZ93XZsEhMAAAAAwO5QgQMAAACAgCDAoauZ2QtmdsrMnriaWXqAG8XMPm1mM2b2VEtbn5l9w8zO+tt8J88R2Gib6/YjZjbh33efMLNf6eQ5AhuZ2QEz+x8zO21mT5vZB/x27rnoWle4bnd9z6ULJbqamb0g6YRzjnVd0NXM7OclLUv6jHPuLr/tLyXNOec+ZmZ/KCnvnPtQJ88TaLXNdfsRScvOub/q5LkB2zGzEUkjzrkfm1lG0uOS3irp3eKeiy51hev27drlPZcKHABcB86570qa29D8Fkn3+4/vl3ejBrrGNtct0NWcc5POuR/7j5ckPSNpTNxz0cWucN3uGgEO3c5J+rqZPW5m7+n0yQC7NOScm/QfT0ka6uTJALvwfjN70u9iSTc0dC0zOyzpHkmPinsuAmLDdSvt8p5LgEO3e71z7l5Jb5b0Pr+7DxA4zuuvTp91BMHfSToq6W5Jk5L+urOnA2zNzHok/YekDzrnFlv3cc9Ft9riut31PZcAh67mnJvwtzOSHpT0ms6eEbAr036f90bf95kOnw/wspxz0865unNuTdI/iPsuupCZReX9Evw559wX/WbuuehqW123V3PPJcCha5lZ2h/kKTNLS3qTpKeu/Cqgqzwk6V3+43dJ+lIHzwXYkcYvwL5fE/dddBkzM0n/JOkZ59zHW3Zxz0XX2u66vZp7LrNQomuZ2RF5VTdJikj6F+fcn3fwlIBtmdm/SnqDpH5J05I+LOk/Jf27pIOSXpT0duccE0aga2xz3b5BXlceJ+kFSe9tGVcEdJyZvV7S9ySdkrTmN/+xvPFE3HPRla5w3f62dnnPJcABAAAAQEDQhRIAAAAAAoIABwAAAAABQYADAAAAgIAgwAEAAABAQBDgAAAAACAgCHAAAGzDzN5gZv/V6fMAAKCBAAcAAAAAAUGAAwAEnpn9rpn9yMyeMLNPmlnYzJbN7G/M7Gkz+5aZDfjH3m1mPzSzJ83sQTPL++23m9k3zez/zOzHZnbUf/seM/uCmZ0xs8+ZmXXsiwIAbnkEOABAoJnZKyT9pqTXOefullSX9DuS0pJOOud+WtLDkj7sv+Qzkj7knHuVpFMt7Z+T9Ann3M9Ieq2kSb/9HkkflPRTko5Iet2efykAALYR6fQJAABwjX5R0n2SHvOLY0lJM5LWJP2bf8xnJX3RzLKScs65h/32+yV93swyksaccw9KknOuIkn++/3IOTfuP39C0mFJ39/7rwUAwGYEOABA0Jmk+51zf9TWaPanG45zV/n+Ky2P6+LfTgBAB9GFEgAQdN+S9DYzG5QkM+szs0Py/o17m3/MOyR93zm3IGnezH7Ob3+npIedc0uSxs3srf57xM0sdUO/BQAAO8D/IgIAAs05d9rM/kTS180sJKkm6X2SipJe4++bkTdOTpLeJenv/YB2XtLv+e3vlPRJM/uo/x6/cQO/BgAAO2LOXW2PEgAAupeZLTvnejp9HgAAXE90oQQAAACAgKACBwAAAAABQQUOAAAAAAKCAAcAAAAAAUGAAwAAAICAIMABAAAAQEAQ4AAAAAAgIAhwAAAAABAQ/w+rvuVdphYUJQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = large_results.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train_loss</th>\n",
       "      <th>dev_loss</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>epoch</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.412386</td>\n",
       "      <td>0.679980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.399027</td>\n",
       "      <td>0.680581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.405207</td>\n",
       "      <td>0.680687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.392395</td>\n",
       "      <td>0.681612</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.419424</td>\n",
       "      <td>0.681979</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       train_loss  dev_loss\n",
       "epoch                      \n",
       "15       0.412386  0.679980\n",
       "17       0.399027  0.680581\n",
       "16       0.405207  0.680687\n",
       "18       0.392395  0.681612\n",
       "14       0.419424  0.681979"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "large_results.sort_values('dev_loss').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "train_loss    0.364454\n",
       "dev_loss      0.698131\n",
       "Name: 25, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "large_results.iloc[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dev loss is important because it tells me how well the model is doing at predicting pronunciations it's never seen before. If the model just memorizes the exact sequence of phonemes in the training set then performance on the dev set will suffer since it won't generalize.\n",
    "\n",
    "By epoch 25 the model has overfit the training set, with dev loss climbing back up to .698 from its low of .679. Also, at this point 53% of generated words were present in the training set, again indicating oerfitting.\n",
    "\n",
    "Another illustration of how much the model has overfit is the chart below, which shows the model's assigned probability for each phoneme in /ˌænəsˌθiːziːˈɑːlədʒɪst/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAH0CAYAAACJqkSAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3debgsZ1kv7N9DQpjnBJEMbJTAZ0BAz2ZQDhgFNYAGPSDCpQKKxInp4BHj8AVEwYCCyCeokSioKCSIGEgwqIdRRRIUIYmiIQSSABIgTDIGnu+PqgWdxd577WH1Xmu9676vq6/dXfVW9dO1anfXr9+3qqu7AwAAwJiutdEFAAAAsDxCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMYRFU9tar+dD+XfVRVvXkP819TVY/cVduq+lRVfd3+PO8+1ni9qnpVVX28qs5c9vMtPG9X1e3WYT173MZbUVX9YlW9cKPr2BtVdXxVXb7RdQBsBKEPYANV1aVV9Zk5OP1XVb2oqm640XWt1t337+4X72beDbv7kiSZ6/+1JZXxkCRfk+QW3f0Dq2fOofcL87ZcuX1sSbXs0qoaPlZV/1BV33IwaziYuvsZ3f3j+7pcVX1LVX2yqg5ZmPYHu5n2e+tVL8B2JfQBbLzv7e4bJvnmJDuT/PLqBjXZ7u/Zt0nyH9199R7avGwOoSu3mx6s4lbXkOSIJG9O8oqqqg2oYzM7P9MxyDcvTLt3kstXTbtPkjfu68qr6tADqg5gMNv9AAJg0+juK5K8JsmdkqSqXl9VT6+qv0/y6SRfV1W3rqqzquqjVXVxVT1m1WquW1Uvm3tM/rmq7rIyo6pOrqp3z/MuqqrvX7VsVdXvzMMn/72q7rsw4/VVtcsenZXhj1V1UpIfSvLkuafrVVX1c1X1F6vaP6+qfns36/qG+bk+VlUXVtWJ8/RfSXJKkh+c1/3otbfoV637t6vqsqr6RFW9raruvTDvkHmo4sr2eVtVHb2w+P2q6j/nup6/NyGuu7+Q5MVJbpXkFgvP9ZtVdVVVvaeq7r8wfbd/27kH8Yyq+uO5vguraueqZf+iqq6c1/v4VcueWVV/Oi/7zqq6fVX9QlV9aN4m37XQ/iZVdXpVfaCqrqiqX1vsfVu1Tb88pLiqdsz7wiOr6n1V9eGq+qU9bJu3ZAp1qapbJjksyRmrpt0+c+jbi+3z8vk1fiLJo2oaDvyieVtflORua/3NAEYl9AFsEnPIeECSf1mY/CNJTkpyoyTvTfLSTL0ht8403PEZVfUdC+0flOTMJDdP8mdJXllV157nvTtTb8pNkvxKkj+tqq9dWPYec5vDkzwlUw/Vzfe2/u4+LclLkjxr7mX73iR/muSEqrrp/BoPTfKwJH+8i9d/7SSvSvLaJLdM8rgkL6mqO3T3U5I8I1/pyTt9b+tacF6Su+Yr2+bMqrruPO9JSR6eafvfOMmPZQraK74nU2i4c5KHJvnutZ6sqq6T5FFJLuvuD8+T75HkXZm28bOSnL4QINf62544t7lpkrOS/M78PNfKtN3+NcmRSe6b5IlVtVjj9yb5kyQ3y7R/nZvpGODIJE9L8vsLbV+U5Ookt0vyTUm+K8m+DOH8n0nuMNdxSlV9w27avTFzwJv/ffN8W5z2nu5eOQ9vb/b9l2faPi/JtA9//Xz77iSP3IfXADAUoQ9g472ypnPP3pzkDZnCzYoXdfeF85DGWyW5V5Kf7+7Pdvfbk7wwySMW2r+tu18+96Q8J8l1k9wzSbr7zO5+f3d/qbtfluQ/k9x9YdkPJXlud39hnv+uJA88kBfW3R/IdHC/cg7eCUk+3N1v20Xzeya5YZJTu/vz3f1/k7w6UxjbWw+de+NWbq9bqOVPu/sj3X11dz87yXUyhZNkCjW/3N3v6sm/dvdHFtZ7and/rLvfl+R1mcLjHmtIclmS/5FksUf1vd39B939xUy9gF+b5GvmwL/W3/bN3X3OvOyfJFnpxb1bkiO6+2nzdrskyR9kCtcr3tTd58770ZmZhp6eOu8nL02yo6puWlVfkyn4PrG7/7u7P5Tkt1atay2/0t2f6e5/zRRE77Kbdm9I8j/n0HvvJG9K8o9J7rkw7Q3Jl78QWWv7/GN3v3Levz+TKZw/vbs/2t2XJXnePrwGgKEY8w6w8b6vu/92N/MuW7h/6yQf7e5PLkx7b6bzAL+qfXd/qaarFd46SarqEZl6tHbMTW6YqcdpxRXd3avWfet9eB278+IkP5UpiPxwpsCyK7fO1Cv2pVU1HLkPz3VGd//wrmZU1f9J8uj5eTpTj97K6z86Uy/n7nxw4f6nM227fa5hcT3d/em5k++GmYZ/rvW3XV3Ddeee09skuXVd86I1h2QKUSv+a+H+ZzIF7y8uPF6p49ZJrp3kAwsjWK+Va+6Ha9nbbfWWed6dMvXq/W53f6qqLluYthLU9mnfX1hmcdp79+E1AAxFTx/A5rYYwt6f5OZVdaOFacckuWLh8ZfPQ5uH/R2V5P1VdZtMoeuxma5+edMkFyRZPDftyFXnqh0zP+f+1rvilUnuXFV3yjRM8iW7Wfb9SY6ua16wZvXr2y/z+XtPztT7c7P59X88X3n9l2UaBrhR9uZvuzuXZRoGedOF2426+wH7UcdlST6X5PCFdd24u++4H+vao+7+bKYht9+b5Gu7+9/nWW+ap905X7mIy95sn9X73gey8P9hbg+wLQl9AFvEPETtH5L8elVdt6runKnnavG3+f5HVf2vuQfoiZkO4N+S5AaZDoqvTJKq+tHMF4xZcMskj6+qa1fVDyT5hiTn7GOZ/5XkGr/ZNx/cvzzTeXRvnYdI7so/ZeoZevJcw/GZDv5fuo817MqNMp2ndmWSQ6vqlEw9fStemORXq+rYmty5qm6xqxUtw17+bXfnrUk+WVU/P1+85JCqulNV7fOFS+bhuK9N8uyqunFVXauqvr6qvm1f17WX3pjkCZle+4o3z9M+0N3vnuvan+1zRpJfqKqbVdVRmc4RBdiWhD6AreXhmYZnvj/JXyZ5yqqhoX+V5AeTXJXpIjD/az5H76Ikz850ztR/JfnGJH+/at3/lOTYJB9O8vQkD1l1XtveOD3JcfP5dK9cmP7i+Tl3N7Qz3f35TCHv/nMNL0jyiIUeoL2xcnXPxdstM1245K+T/EemYX6fzTWH/j0nU0h4bZJPzK/jevvwvOthrb/tLs3DNL8n03mG78m07V6Y6YI9++MRma6keVGm/ejlmc49XIY3ZPqyYfFH6988T3vTqrb7un1+JdPf+j2Z/q673fcARlfXPH0DANZfVR2T5N+T3Kq7P7HR9QDAdqKnD4Clms/Re1KSlwp8AHDwuXonAEtTVTfINJz0vZl+rgEAOMgM7wQAABiY4Z0AAAADE/oAAAAGtiXP6Tv88MN7x44dG10GAADAhnjb29724e4+Ym/absnQt2PHjpx//vkbXQYAAMCGqKr37m1bwzsBAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEtNfRV1R9W1Yeq6oLdzK+qel5VXVxV76iqb15mPQAAANvNsnv6XpTkhD3Mv3+SY+fbSUl+d8n1AAAAbCtLDX3d/cYkH91Dkwcl+eOevCXJTavqa5dZEwAAwHay0ef0HZnksoXHl8/TvkpVnVRV51fV+VdeeeVBKQ4AAGCrO3SjC9hb3X1aktOSZOfOnb3B5QAwmB0nn72U9V566gOXsl6Wy/6wXLbvctm+rLbRoe+KJEcvPD5qngYA7IGDOhbZH4A92ejhnWclecR8Fc97Jvl4d39gg2sCAAAYxlJ7+qrqz5Mcn+Twqro8yVOSXDtJuvv3kpyT5AFJLk7y6SQ/usx6AAAAtpulhr7ufvga8zvJzyyzBgAAgO1so8/pAwC2AeecAWycjT6nDwAAgCUS+gAAAAYm9AEAAAzMOX3AtuUcIwBgO9DTBwAAMDChDwAAYGBCHwAAwMCc0wcAAGyYZZ1jnzjPfoWePgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwP9kAsEUs65LWLmcNAGPT0wcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAzt0owsAAIDNbMfJZy9lvZee+sClrBdW09MHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgR260QUAMKYdJ5+9lPVeeuoDl7JeABiVnj4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMLClhr6qOqGq3lVVF1fVybuYf0xVva6q/qWq3lFVD1hmPQAAANvN0kJfVR2S5PlJ7p/kuCQPr6rjVjX75SRndPc3JXlYkhcsqx4AAIDtaJk9fXdPcnF3X9Ldn0/y0iQPWtWmk9x4vn+TJO9fYj0AAADbzqFLXPeRSS5beHx5knusavPUJK+tqscluUGS+y2xHgAAgG1noy/k8vAkL+ruo5I8IMmfVNUua6qqk6rq/Ko6/8orrzyoRQIAAGxVywx9VyQ5euHxUfO0RY9OckaSdPc/JrluksN3tbLuPq27d3b3ziOOOGIJ5QIAAIxnmaHvvCTHVtVtq+qwTBdqOWtVm/cluW+SVNU3ZAp9uvEAAADWydJCX3dfneSxSc5N8m+ZrtJ5YVU9rapOnJv9bJLHVNW/JvnzJI/q7l5WTQAAANvNMi/kku4+J8k5q6adsnD/oiT3WmYNAAAA29lGX8gFAACAJRL6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADO3SjCwAAANhKdpx89lLWe+mpD1zKevX0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAa21NBXVSdU1buq6uKqOnk3bR5aVRdV1YVV9WfLrAcAAGC7OXRZK66qQ5I8P8l3Jrk8yXlVdVZ3X7TQ5tgkv5DkXt19VVXdcln1AAAAbEfL7Om7e5KLu/uS7v58kpcmedCqNo9J8vzuvipJuvtDS6wHAABg21mzp6+qjkrysCT3TnLrJJ9JckGSs5O8pru/tJtFj0xy2cLjy5PcY1Wb28/P8fdJDkny1O7+6315AQAAAOzeHkNfVf1RpvD26iTPTPKhJNfNFNZOSPJLVXVyd7/xAJ7/2CTHJzkqyRur6hu7+2O7qOWkJCclyTHHHLOfTwcAALC9rNXT9+zuvmAX0y9I8oqqOizJ7hLYFUmOXnh81Dxt0eVJ/qm7v5DkPVX1H5lC4HmrV9bdpyU5LUl27tzZa9QNbIAdJ5+9lPVeeuoDl7JeAIDtYI/n9O0m8C3O/3x3X7yb2eclObaqbjuHw4clOWtVm1dm6uVLVR2eqQfxkr2oGwAAgL2wV1fvrKp7JXlqktvMy1SS7u6v290y3X11VT02ybmZztf7w+6+sKqeluT87j5rnvddVXVRki8m+bnu/siBvCAAAAC+Ym9/suH0JP87ydsyhbO90t3nJDln1bRTFu53kifNNwAAANbZ3oa+j3f3a5ZaCQAAAOtub0Pf66rqN5K8IsnnViZ29z8vpSoAAADWxd6GvpXf19u5MK2TfMf6lgMAAMB62qvQ193fvuxCAAAAWH97/MkGAAAAtjahDwAAYGBCHwAAwMD2OfRV1WnLKAQAAID1tz89fTvXbgIAAMBmsD+h70PrXgUAAABLsc+hr7tPWEYhAAAArD8XcgEAABiY0AcAADAwoQ8AAGBgh+5pZlU9ubufVVX/X5JePb+7H7+0ygAAADhgewx9Sf5t/vf8ZRcCAADA+ttj6OvuV83/vvjglAMAAMB6ck4fAADAwIQ+AACAgQl9AAAAA9vv0FdV37OehQAAALD+DqSn727rVgUAAABLsd+hr7ufsp6FAAAAsP4OZHjnrdazEAAAANbfgQzvPH3dqgAAAGApDmR45wPXsxAAAADWn59sAAAAGNiBnNP36vUsBAAAgPV3ID19j1m3KgAAAFiKAzmn7wPrWQgAAADrzzl9AAAAAxP6AAAABib0AQAADGyPoa+qblJVp1bVv1fVR6vqI1X1b/O0mx6sIgEAANg/a/X0nZHkqiTHd/fNu/sWSb59nnbGsosDAADgwKwV+nZ09zO7+4MrE7r7g939zCS3WW5pAAAAHKi1Qt97q+rJVfU1KxOq6muq6ueTXLbc0gAAADhQa4W+H0xyiyRvmM/p+2iS1ye5eZKHLrk2AAAADtChe5rZ3Vcl+fn5BgAAwBbjJxsAAAAGJvQBAAAMTOgDAAAYmNAHAAAwsD1eyKWq3pOk99Rknv/c7n7eehYGAADAgVvr6p23PViFAAAAsP4M7wQAABiY0AcAADAwoQ8AAGBgQh8AAMDA1rp65xnd/dCqemeueRXPStLdfeelVgcAAMAB2WPoS/KE+d/vWXYhAAAArL+1Qt8Hk6S737u7BlVV3b2n3/IDAABgg6x1Tt/rqupxVXXM4sSqOqyqvqOqXpzkkcsrDwAAgAOxVk/fCUl+LMmfV9XXJbkqyXWTHJLktUme293/stwSAQAA2F97DH3d/dkkL0jygqq6dpLDk3ymuz92MIoDAADgwKx19c7rJvnJJLdL8o4kf9jdVx+MwgAAADhwa53T9+IkO5O8M8kDkjx76RUBAACwbtY6p++47v7GJKmq05O8dfklAQAAsF7W6un7wsodwzoBAAC2nrV6+u5SVZ+Y71eS682PK0l3942XWh0AAAAHZK2rdx5ysAoBAABg/a01vBMAAIAtTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCWGvqq6oSqeldVXVxVJ++h3YOrqqtq5zLrAQAA2G6WFvqq6pAkz09y/yTHJXl4VR23i3Y3SvKEJP+0rFoAAAC2q2X29N09ycXdfUl3fz7JS5M8aBftfjXJM5N8dom1AAAAbEvLDH1HJrls4fHl87Qvq6pvTnJ0d5+9xDoAAAC2rQ27kEtVXSvJc5L87F62P6mqzq+q86+88srlFgcAADCIZYa+K5IcvfD4qHnaihsluVOS11fVpUnumeSs3V3MpbtP6+6d3b3ziCOOWFLJAAAAY1lm6DsvybFVdduqOizJw5KctTKzuz/e3Yd3947u3pHkLUlO7O7zl1gTAADAtrK00NfdVyd5bJJzk/xbkjO6+8KqelpVnbis5wUAAOArDl3myrv7nCTnrJp2ym7aHr/MWgAAALajDbuQCwAAAMu31J4+2Gx2nLycXwe59NQHLmW9AABwoPT0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAzs0I0ugK1tx8lnL2W9l576wKWsFwAAths9fQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEsNfVV1QlW9q6ourqqTdzH/SVV1UVW9o6r+rqpus8x6AAAAtpulhb6qOiTJ85PcP8lxSR5eVcetavYvSXZ2952TvDzJs5ZVDwAAwHa0zJ6+uye5uLsv6e7PJ3lpkgctNuju13X3p+eHb0ly1BLrAQAA2HaWGfqOTHLZwuPL52m78+gkr1liPQAAANvOoRtdQJJU1Q8n2Znk2/bQ5qQkJyXJMcccc5AqAwAA2NqW2dN3RZKjFx4fNU+7hqq6X5JfSnJid39udyvr7tO6e2d37zziiCPWvVgAAIARLTP0nZfk2Kq6bVUdluRhSc5abFBV35Tk9zMFvg8tsRYAAIBtaWmhr7uvTvLYJOcm+bckZ3T3hVX1tKo6cW72G0lumOTMqnp7VZ21m9UBAACwH5Z6Tl93n5PknFXTTlm4f79lPj8AAMB2t9QfZwcAAGBjCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgh250Acu24+Szl7LeS0994FLWCwAAsJ709AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGNvyPs8NWtuPks5ey3ktPfeBS1gsAwOajpw8AAGBgQh8AAMDADO/cZAznAwAA1pOePgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwsKWGvqo6oareVVUXV9XJu5h/nap62Tz/n6pqxzLrAQAA2G6WFvqq6pAkz09y/yTHJXl4VR23qtmjk1zV3bdL8ltJnrmsegAAALajZfb03T3Jxd19SXd/PslLkzxoVZsHJXnxfP/lSe5bVbXEmgAAALaVZYa+I5NctvD48nnaLtt099VJPp7kFkusCQAAYFup7l7OiqsekuSE7v7x+fGPJLlHdz92oc0Fc5vL58fvntt8eBfrOynJSfPDOyR51xLKPjzJVz33Jqbe5VLvcql3udS7XOpdLvUul3qXS73LtdXqTZZX8226+4i9aXjoEp58xRVJjl54fNQ8bVdtLq+qQ5PcJMlHdrWy7j4tyWlLqPPLqur87t65zOdYT+pdLvUul3qXS73Lpd7lUu9yqXe51LtcW63eZHPUvMzhneclObaqbltVhyV5WJKzVrU5K8kj5/sPSfJ/e1ldjwAAANvQ0nr6uvvqqnpsknOTHJLkD7v7wqp6WpLzu/usJKcn+ZOqujjJRzMFQwAAANbJMod3prvPSXLOqmmnLNz/bJIfWGYN+2ipw0eXQL3Lpd7lUu9yqXe51Ltc6l0u9S6Xepdrq9WbbIKal3YhFwAAADbeMs/pAwAAYIMJfQCDqKoXVdWrN7qOUW3l7VtVT62qR210HaPayvvGZldVr66qF210HbDVCX0A43hCkh/e6CIGZvsCbKCqen1V/c4GPfelVdW7uJ06z98xP/5IVd1kT3XPXxStLP+FqvpQVb2uqn6mqq69jPqFPoDdqKrzquo3Fx7fvKo+u5E17Ul3f7y7P7bRdYzK9gXYXqrqxlV104VJT0vytatuv7ZqsesnOXkvVv+38/I7knxXklcl+ZUkb6qqGyzUcFRV1f6+hhVC3xZQVSdU1Zuq6qqq+u+q+tuquktV3b+qzq+qz1TVu6vqlPlH7jeN+ZuNF1TVM6rqw/M3Gb9ZVZty36uqp1XVBVX1qar6eFWdXVV33ui69qSq7lNVb1mo+a1VdaeNrmurq6qbJdmZ5OyVad390SRv27Ci1rDVhphV1fG7+db09Rtd265ste27VVTV9edt+6mq+q+q+kVD+tbf4ufbRteyO7vaFxbmPWo37xcv2sCSv8ou3tcetdE17clWOoaY/9bfluRnFrbvjiU8zyFV9d1V9WdJPpjkLguzP9ndH1x1W/1/6nlJnlBVR67xVJ+bl7+iu9/e3c9JcnySb9x/5OcAAAn7SURBVE7y5IV2v5rkkvn/8O3293VtygNvvsoNMu1A90hy9ySXJXlDkjOT/H6SOyX5iSQ/muSXN6jGPfmhJFcn+dYkj03yxCQ/uKEV7d7XZ9qG35Tkfkn+O8mbq+rYDa1qN+aQ/1dJ3pzpTekeSZ6b5IsbWdcgbjv/e/Gq6f95sAvZV1V1o6q6sqreX1XX3+h69uAfcs1vS3cm+ViS129gTWvaQtv3y7r7qd39oo2uYzd+M8l3Jnlwkvtmev+9T5JU1W2q6nNV9S/r8U33Nrf4+bZZ7XZfSPKyXPP94luTfCbJ+Qe/zD36hyR3nO8/OFPdm9IWPIZ4QpJ/TPJH+cp+cNl6rbyq7lhVz5rX+bJMx4AnJHnjPq7qzCTvzNQruE+6+4Ikf51p31nx+Hld907yrqr6+6o6aVUP5Jr8ZMMWVFWHJLkqyQu6++SF6T+c5DndfcsNK26V+Rv763T3tyxM+5sk7+3uH9+wwvbSvK3fkeTN3f0TG13PalV18yQfSXJ8d79ho+sZSVXdO9Mb/X+vmnVYdx+2ASWtaf4W9PDu/p6qelKSz3T3725wWXulqq6X5E1J3pfkwb0JP5y28vbdrKrqhpnew36su1+yMO3yJK/s7kdV1fOSvKa7X7OBpe7R4r6x0bVsVXuzLyy0PSzJ32fqhTlxs71fVNXhSa5M8u3d/foNLme3tuIxxHxceUF3P3ad1neLTJ0Tj0zyjZkC158kedX8e+KLbS/NFDS/sGo1D+vuV8+9ju9JcrdMHTZ/l+Qu3X3h6rr39J5R0zmCj+/ur/pSsaqOznRu+Y9k+nL6rCQvTvLX3f2lPb3WTTUUkF2rqttmSvjfkuSWmXpor5/kiVW1uNMfkuS6VXXj7v7Ewa90t96x6vH7M72OTa+7v1hVr8v07dem090fnd84zq2qv8v0BvPy7n7fxlY2hJWwd99MH4ornrEBteyP5yfZ4wfAZjH34Lwo03vYj2y2A7jd2DLbd5P7+iSHZfr2PknS3Z+qqncutPml7v7kQa+Mg21v9oUVv5HkVklO2CLvF5uSY4gkyeOSPCVTD+3tu/vSNdo/J8npq6Z9YHWj7n5DVZ2b5NeTnLiPNVWSXe7X3X3ZvM5fr6qHJ/ndJA/NFAAv3dNKhb6t4dWZvs36qSRXJPl8pjfF38uU7lfbbOP1V38j0tmkQ4t3c67DYUn++WDXsre6+0er6rmZhiCcmOTpVfV93X3uBpe21V0y//vJ7v7yEM+qus4G1bNPuvtzG13DPjgl0xCuu3X36p7VTWmLbd8tTeBbH4ufb919w42s5UBU1fcn+elMvVMfWas9e+YYIqdlOk59RJILquovM/X0/V1372qY60cWjwnWcHKSt88jh/bFcfnKMcg1zD2TP5ipp29nkr/JlAUuX2ulm/LAm69yXJJf6+6/6e6LMu2chyf5f7r74l3cfPu8/26QaRz1XRduZ2cTX7wjSbr7X7v7md19fKbzoR65sRVtffNVGs/LNIY+yZcD3312u9AmUlXXqU12YaddqaqHZDph/UHdveaH1maxVbbvFvDuTJ9p91yZUNNV6+606jEHbvHzbTNac1+Y/XSm49c/qqpfrarrHrwSx7TFjiE+n2lUyLro7vd399O7+w6ZruXwqSQvTXJ5VT27qvb7/0t3vzPJHyd51t4uM19E54QkL1+Ydp2qekhVvTJTr+JPzPOP7u4HdPfLuvvqtdYt9M1qc/9w7YeT/ERVHVtV35rkjEw70YNquirmnarqDvMO8fSNLXXL+5skP5vkRvPjB2S6ktJzN6qgPamq21bVqVX1rfMFD749yZ2TXLTRte3JJv//tuhXk/xaTVfKvV2SP8gWuJDL7KeTbOrzZucPtxcn+cUk76uqW823m29waXtj02/frWC+6t3pSZ5ZVd9ZVXdM8oe55kHdM6rqhA0pcCyrP982lb3cF5IpkNwx05dFD0nyZwe10DXM5xuunMJy8/l85U1pix5DXJrk7jX9Jt7htY5Xg+/ut3T3T2U6b+9xSW6f5LxVPXU3WvisWrndZJcrnJyS6YuWXZ0mdJ15+VvXdFX+J2UK3W/LdFGjFS/IdErBe5Lcvbvv0t3P7u4P7svrE/q2hodk2vHekeQlSV7a3Y/M9I3EvZK8Jclbk/xc1hjPy5oenmkI7Wsybe/vS/Kd3b1ZD/Q/nWnfODPJf2Q6gH5JkmduZFGj6O5XZQokv5dpf7hZpn1iU6uqG2Wq+//dzAccmYamXD/TlyofWLi9YiOLWssW2r5bxf9J8rokfzn/e0Hmq+VV1W2S/GSm4OfqnQdm8fNts9rtvrBi7pn59+5+ZZLHJPn+eT/ZLL41yYXz/b/I5r1aebI1jyF+M1Nv30WZLpZzzHo/QXd/rrtf3t3fm+TIXDMEn5Jrfl59IFMg2926Lst0Bf5d9Ujfb17+fZnOpzwxyVOT3GfVqQ6/nuTI7v7f3f32/X1drt45q6qnJrl0E1/SGobh/xsAB6qq7pPpJ6xu193v3uh6YDMT+gAA2NSq6geSXC/ThdX+O9Pl9Z+V5MPd/T83sjbYCpyEDgDAZve5TKex3CHJtTP9gPYrsrmHIsKmoacPAABgYC7kAgAAMDChDwAAYGBCHwAAwMCEPgCGVVVfrKq3V9UFVXVmVV1//lHfCza6NgA4WIQ+AEb2me6+a3ffKdMP+v7kRhcEAAeb0AfAdvGmJLeb7x9SVX9QVRdW1Wur6npJUlV3raq3VNU7quovq+pm8/TXV9Uzq+qtVfUfVXXvefohVfUbVXXevMxPzNOPr6o3VNVfVdUlVXVqVf3QvPw7q+rr53ZHVNVfzMufV1X3OvibBYDRCX0ADK+qDk1y/yTvnCcdm+T53X3HJB9L8uB5+h8n+fnuvvPc9ikLqzm0u++e5IkL0x+d5OPdfbckd0vymKq67TzvLpl6Fr8hyY8kuf28/AuTPG5u89tJfmte/sHzPABYV36cHYCRXa+q3j7ff1OS05PcOsl7untl+tuS7KiqmyS5aXe/YZ7+4iRnLqzrFYvt5/vfleTOVfWQ+fFNMgXKzyc5r7s/kCRV9e4kr53bvDPJt8/375fkuKpaeY4bV9UNu/tT+/+SAeCahD4ARvaZ7r7r4oQ5YH1uYdIXk1xvL9a1sswX85XPz0ryuO4+d9VzHL/qOb608PhLC8tfK8k9u/uze/H8ALBfDO8EgCTd/fEkV62cr5dpSOYb9rBIkpyb5Keq6tpJUlW3r6ob7MPTvjZfGeqZqrrrHtoCwH7R0wcAX/HIJL9XVddPckmSH12j/QszDfX855q6EK9M8n378HyPT/L8qnpHps/kN8YVRgFYZ9XdG10DAAAAS2J4JwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBg/z9HoabkAo+rewAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_pronunciation_probability(\n",
    "    large_language_model,\n",
    "    ('ˌ', 'æ', 'n', 'ə', 's', 'ˌ', 'θ', 'iː', 'z', 'iː', 'ˈ', 'ɑː', 'l', 'ə', 'dʒ', 'ɪ', 's', 't')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(train_pronunciations.as_string=='ˌænəsˌθiːziːˈɑːlədʒɪst').sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After seeing /ˌænəsˌ/ the model is almost entirely certain of each of the next phonemes that follows. This pronunciation appears twice in the training set, so the model appears to have memorized it.\n",
    "\n",
    "If I view the full CMU Dict as encompassing *all* of English then it's arguably ok to fit it very well since there's no need to generalize to new words. But I'm hoping to capture English pronunciaiton rules at a more general level than exact words, so that the model will generate words that aren't exactly like the training corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dev loss was at its minimum at epoch 16 so I'll train another model with the same parameters for 16 epochs. Note that I could also try smaller model architectures or regularizing while still training until convergence, but this is faster since I don't need to train a large number of models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5: train loss: 0.4895\tdev loss: 0.7222                                                                                               \n",
      "\tGenerated: in train: 11%, assess: 0%, novel: 89%\n",
      "\t ˈ v æ ŋ g ɝ d\n",
      "\t ˈ t uː t\n",
      "\t ˈ l ɛ b ɪ s t\n",
      "\t tʃ iː θ ɑː ˈ r ɑː s iː oʊ\n",
      "\t v ɪ s ɛ ˈ n ɑː θ ə\n",
      "Epoch 10: train loss: 0.4498\tdev loss: 0.6914                                                                                               \n",
      "\tGenerated: in train: 18%, assess: 3%, novel: 79%\n",
      "\t ˈ p ɛ l ˌ b r ʊ k\n",
      "\t ˈ k ɑː ŋ k ə n\n",
      "\t ˈ tʃ ɪ p\n",
      "\t ˈ w iː z l i\n",
      "\t ˈ w aɪ l ɝ\n",
      "Epoch 15: train loss: 0.4117\tdev loss: 0.6804                                                                                               \n",
      "\tGenerated: in train: 30%, assess: 0%, novel: 70%\n",
      "\t ˈ f l ɔ r ˌ m ɛ n t\n",
      "\t ˈ s m ɑː f i\n",
      "\t ˈ æ l f r ə n\n",
      "\t ˈ ɛ r d\n",
      "\t ˈ k ɑː l m ə n\n",
      "Epoch 16: train loss: 0.4056\tdev loss: 0.6816                                                                                               \n",
      "\tGenerated: in train: 21%, assess: 3%, novel: 76%\n",
      "\t t ɔ r ˈ t eɪ l\n",
      "\t ˈ m oʊ s t r ə s\n",
      "\t ˈ k æ n s ə l ɝ\n",
      "\t ˌ m æ g ə ˈ v iː n oʊ\n",
      "\t ˈ b ɔ r d ˌ s t iː n\n",
      "CPU times: user 17min 38s, sys: 5min 42s, total: 23min 21s\n",
      "Wall time: 23min 12s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "early_stopped_model_params = ModelParams(\n",
    "    rnn_type='gru', num_layers=3, embedding_dimension=500, hidden_dimension=500,\n",
    "    max_epochs=16, early_stopping_rounds=16\n",
    ")\n",
    "\n",
    "early_stopped_language_model = LanguageModel(vocab, early_stopped_model_params, device_name='cuda')\n",
    "\n",
    "_ = early_stopped_language_model.fit(\n",
    "    train_pronunciations.pronunciation.values.tolist(),\n",
    "    dev_pronunciations.pronunciation.values.tolist(),\n",
    "    print_every=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dev loss here does in fact match dev loss in the model above at epoch 16. I'll now look again at /ˌænəsˌθiːziːˈɑːlədʒɪst/:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAH0CAYAAACJqkSAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3debhkZ1kv7N9DQpjnBJEMNErgMyCgpxmUA0ZBDURBD4hwqYAicWI6eMQ4fAFRMKAg8glqBE1UFBJEDCQY1MOoIgmKkETREAJJAAkzyBh4vj/Waqhsunv3sKtr77fv+7rq6qq13nrrqbVXV9Wv3netqu4OAAAAY7rWqgsAAABgeYQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBzCIqnpqVf3pPt73UVX1pt2sf3VVPXJnbavqU1X1dfvyuHtZ4/Wq6pVV9fGqOmvZj7fwuF1Vt9uAfna7jbeiqvrFqnrhquvYE1V1fFVdseo6AFZB6ANYoaq6rKo+Mwen/6qq06vqhquua63uvn93n7GLdTfs7kuTZK7/15ZUxkOSfE2SW3T3D6xdOYfeL8zbcsflY0uqZafW1PCxqvqHqvqWA1nDgdTdz+juH9/b+1XVt1TVJ6vqkIVlf7CLZb+3UfUCHKyEPoDV+97uvmGSb06yPckvr21Qk4P9Nfs2Sf6ju6/eTZuXziF0x+WmB6q4tTUkOSLJm5K8vKpqBXVsZhdk+gzyzQvL7p3kijXL7pPkDXvbeVUdul/VAQzmYP8AAbBpdPeVSV6d5E5JUlWvq6qnV9XfJ/l0kq+rqltX1dlV9ZGquqSqHrOmm+tW1UvnEZN/rqq77FhRVSdX1bvmdRdX1fevuW9V1e/M0yf/varuu7DidVW10xGdHdMfq+qkJD+U5MnzSNcrq+rnquov1rR/XlX99i76+ob5sT5WVRdV1QPn5b+S5JQkPzj3/ej1t+hX9f3bVXV5VX2iqt5aVfdeWHfIPFVxx/Z5a1UdvXD3+1XVf851PX9PQlx3fyHJGUluleQWC4/1m1X10ap6d1Xdf2H5Lv+28wjimVX1x3N9F1XV9jX3/Yuqumru9/Fr7ntWVf3pfN93VNXtq+oXquqD8zb5roX2N6mqF1XV+6vqyqr6tcXRtzXb9MtTiqtq27wvPLKq3ltVH6qqX9rNtnlzplCXqrplksOSnLlm2e0zh7492D4vm5/jJ5I8qqbpwKfP2/riJHdb728GMCqhD2CTmEPGA5L8y8LiH0lyUpIbJXlPkpdkGg25dabpjs+oqu9YaP+gJGcluXmSP0vyiqq69rzuXZlGU26S5FeS/GlVfe3Cfe8xtzk8yVMyjVDdfE/r7+7Tkrw4ybPmUbbvTfKnSU6oqpvOz/HQJA9L8sc7ef7XTvLKJK9Jcsskj0vy4qq6Q3c/Jckz8pWRvBftaV0Lzk9y13xl25xVVded1z0pycMzbf8bJ/mxTEF7h+/JFBrunOShSb57vQerquskeVSSy7v7Q/PieyR5Z6Zt/KwkL1oIkOv9bR84t7lpkrOT/M78ONfKtN3+NcmRSe6b5IlVtVjj9yb5kyQ3y7R/nZfpM8CRSZ6W5PcX2p6e5Ookt0vyTUm+K8neTOH8n0nuMNdxSlV9wy7avSFzwJv/fdN8WVz27u7ecRzenuz7L8u0fV6caR/++vny3UkeuRfPAWAoQh/A6r2ipmPP3pTk9ZnCzQ6nd/dF85TGWyW5V5Kf7+7PdvfbkrwwySMW2r+1u182j6Q8J8l1k9wzSbr7rO5+X3d/qbtfmuQ/k9x94b4fTPLc7v7CvP6dSU7cnyfW3e/P9OF+xzF4JyT5UHe/dSfN75nkhklO7e7Pd/f/TfKqTGFsTz10Ho3bcXntQi1/2t0f7u6ru/vZSa6TKZwkU6j55e5+Z0/+tbs/vNDvqd39se5+b5LXZgqPu60hyeVJ/keSxRHV93T3H3T3FzONAn5tkq+ZA/96f9s3dfe5833/JMmOUdy7JTmiu582b7dLk/xBpnC9wxu7+7x5Pzor09TTU+f95CVJtlXVTavqazIF3yd293939weT/NaavtbzK939me7+10xB9C67aPf6JP9zDr33TvLGJP+Y5J4Ly16ffPkLkfW2zz929yvm/fszmcL507v7I919eZLn7cVzABiKOe8Aq/d93f23u1h3+cL1Wyf5SHd/cmHZezIdB/hV7bv7SzWdrfDWSVJVj8g0orVtbnLDTCNOO1zZ3b2m71vvxfPYlTOS/FSmIPLDmQLLztw606jYl9bUcORePNaZ3f3DO1tRVf8nyaPnx+lMI3o7nv/RmUY5d+UDC9c/nWnb7XUNi/1096fnQb4bZpr+ud7fdm0N151HTm+T5NZ1zZPWHJIpRO3wXwvXP5MpeH9x4faOOm6d5NpJ3r8wg/VaueZ+uJ493VZvntfdKdOo3u9296eq6vKFZTuC2l7t+wv3WVz2nr14DgBDMdIHsLkthrD3Jbl5Vd1oYdkxSa5cuP3l49DmaX9HJXlfVd0mU+h6bKazX940yYVJFo9NO3LNsWrHzI+5r/Xu8Iokd66qO2WaJvniXdz3fUmOrmuesGbt89sn8/F7T840+nOz+fl/PF95/pdnmga4Knvyt92VyzNNg7zpwuVG3f2Afajj8iSfS3L4Ql837u477kNfu9Xdn8005fZ7k3xtd//7vOqN87I75ysncdmT7bN233t/Fv4/zO0BDkpCH8AWMU9R+4ckv15V162qO2cauVr8bb7/UVX/ax4BemKmD/BvTnKDTB+Kr0qSqvrRzCeMWXDLJI+vqmtX1Q8k+YYk5+5lmf+V5Bq/2Td/uH9ZpuPo3jJPkdyZf8o0MvTkuYbjM334f8le1rAzN8p0nNpVSQ6tqlMyjfTt8MIkv1pVx9bkzlV1i511tAx7+Lfdlbck+WRV/fx88pJDqupOVbXXJy6Zp+O+Jsmzq+rGVXWtqvr6qvq2ve1rD70hyRMyPfcd3jQve393v2uua1+2z5lJfqGqblZVR2U6RhTgoCT0AWwtD880PfN9Sf4yyVPWTA39qyQ/mOSjmU4C87/mY/QuTvLsTMdM/VeSb0zy92v6/qckxyb5UJKnJ3nImuPa9sSLkhw3H0/3ioXlZ8yPuaupnenuz2cKefefa3hBkkcsjADtiR1n91y83DLTiUv+Osl/ZJrm99lcc+rfczKFhNck+cT8PK63F4+7Edb72+7UPE3zezIdZ/juTNvuhZlO2LMvHpHpTJoXZ9qPXpbp2MNleH2mLxsWf7T+TfOyN65pu7fb51cy/a3fnenvust9D2B0dc3DNwBg41XVMUn+PcmtuvsTq64HAA4mRvoAWKr5GL0nJXmJwAcAB56zdwKwNFV1g0zTSd+T6ecaAIADzPROAACAgZneCQAAMDChDwAAYGBb8pi+ww8/vLdt27bqMgAAAFbirW9964e6+4g9abslQ9+2bdtywQUXrLoMAACAlaiq9+xpW9M7AQAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBLTX0VdUfVtUHq+rCXayvqnpeVV1SVW+vqm9eZj0AAAAHm2WP9J2e5ITdrL9/kmPny0lJfnfJ9QAAABxUlhr6uvsNST6ymyYPSvLHPXlzkptW1dcusyYAAICDyaqP6TsyyeULt6+Yl32Vqjqpqi6oqguuuuqqA1IcAADAVnfoqgvYU919WpLTkmT79u294nIAAGBT2nbyOUvp97JTT1xKvyzfqkf6rkxy9MLto+ZlAAAAbIBVh76zkzxiPovnPZN8vLvfv+KaAAAAhrHU6Z1V9edJjk9yeFVdkeQpSa6dJN39e0nOTfKAJJck+XSSH11mPQAArJ7ph3BgLTX0dffD11nfSX5mmTUAwJ7Yah9Ct1q9AKzOqqd3AgAAsERCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwsENXXQAAAPtn28nnLKXfy049cSn9AgeWkT4AAICBCX0AAAADE/oAAAAG5pg+AJbCMUYAsDkY6QMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAAD85MNAFuEn0AAAPaFkT4AAICBGekDDlpGzgCAg4GRPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADO3TVBQAAbDbbTj5nKf1eduqJS+kXYHeM9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGttTQV1UnVNU7q+qSqjp5J+uPqarXVtW/VNXbq+oBy6wHAADgYLO00FdVhyR5fpL7JzkuycOr6rg1zX45yZnd/U1JHpbkBcuqBwAA4GC0zJG+uye5pLsv7e7PJ3lJkgetadNJbjxfv0mS9y2xHgAAgIPOoUvs+8gkly/cviLJPda0eWqS11TV45LcIMn9llgPAADAQWfVJ3J5eJLTu/uoJA9I8idVtdOaquqkqrqgqi646qqrDmiRAAAAW9UyQ9+VSY5euH3UvGzRo5OcmSTd/Y9Jrpvk8J111t2ndff27t5+xBFHLKFcAACA8Swz9J2f5Niqum1VHZbpRC1nr2nz3iT3TZKq+oZMoc8wHgAAwAZZWujr7quTPDbJeUn+LdNZOi+qqqdV1QPnZj+b5DFV9a9J/jzJo7q7l1UTAADAwWaZJ3JJd5+b5Nw1y05ZuH5xknstswYAAICD2apP5AIAAMASCX0AAAADE/oAAAAGttRj+gAAAHZn28nnLK3vy049cWl9byVG+gAAAAYm9AEAAAxM6AMAABiYY/oAgKVb1jE7jtcBWJ+RPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGduiqCwDGse3kc5bS72WnnriUfgEADgZG+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAlhr6quqEqnpnVV1SVSfvos1Dq+riqrqoqv5smfUAAAAcbJb2O31VdUiS5yf5ziRXJDm/qs7u7osX2hyb5BeS3Ku7P1pVt1xWPQAAAAejZY703T3JJd19aXd/PslLkjxoTZvHJHl+d380Sbr7g0usBwAA4KCz7khfVR2V5GFJ7p3k1kk+k+TCJOckeXV3f2kXdz0yyeULt69Ico81bW4/P8bfJzkkyVO7+6/35gkAAACwa7sNfVX1R5nC26uSPDPJB5NcN1NYOyHJL1XVyd39hv14/GOTHJ/kqCRvqKpv7O6P7aSWk5KclCTHHHPMPj4cAADAwWW9kb5nd/eFO1l+YZKXV9VhSXaVwK5McvTC7aPmZYuuSPJP3f2FJO+uqv/IFALPX9tZd5+W5LQk2b59e69TNwAAAFnnmL5dBL7F9Z/v7kt2sfr8JMdW1W3ncPiwJGevafOKTKN8qarDM40gXroHdQMAALAH9ujsnVV1ryRPTXKb+T6VpLv763Z1n+6+uqoem+S8TMfr/WF3X1RVT0tyQXefPa/7rqq6OMkXk/xcd394f54QAAAAX7GnP9nwoiT/O8lbM4WzPdLd5yY5d82yUxaud5InzRcAAAA22J6Gvo9396uXWgkAAAAbbk9D32ur6jeSvDzJ53Ys7O5/XkpVAAAAbIg9DX07fl9v+8KyTvIdG1sOAAAAG2mPQl93f/uyCwEAAGDj7fYnGwAAANjahD4AAICBCX0AAAAD2+vQV1WnLaMQAAAANt6+jPRtX78JAAAAm8G+hL4PbngVAAAALMVeh77uPmEZhQAAALDxnMgFAABgYEIfAADAwIQ+AACAgR26u5VV9eTuflZV/X9Jeu367n780ioDAABgv+029CX5t/nfC5ZdCAAAABtvt6Gvu185/3vGgSkHAACAjeSYPgAAgIEJfQAAAAMT+gAAAAa2z6Gvqr5nIwsBAABg4+3PSN/dNqwKAAAAlmKfQ193P2UjCwEAAGDj7c/0zlttZCEAAABsvP2Z3vmiDasCAACApdif6Z0nbmQhAAAAbDw/2QAAADCw/Tmm71UbWQgAAAAbb39G+h6zYVUAAACwFPtzTN/7N7IQAAAANp5j+gAAAAYm9AEAAAxM6AMAABjYbkNfVd2kqk6tqn+vqo9U1Yer6t/mZTc9UEUCAACwb9Yb6TszyUeTHN/dN+/uWyT59nnZmcsuDgAAgP2zXujb1t3P7O4P7FjQ3R/o7mcmuc1ySwMAAGB/rRf63lNVT66qr9mxoKq+pqp+Psnlyy0NAACA/bVe6PvBJLdI8vr5mL6PJHldkpsneeiSawMAAGA/Hbq7ld390SQ/P18AAADYYvxkAwAAwMCEPgAAgIEJfQAAAAMT+gAAAAa22xO5VNW7k/Tumszrn9vdz9vIwgAAANh/652987YHqhAAAAA2numdAAAAAxP6AAAABib0AQAADEzoAwAAGNh6Z+88s7sfWlXvyDXP4llJurvvvNTqAAAA2C+7DX1JnjD/+z3LLgQAAICNt17o+0CSdPd7dtWgqqq7d/dbfgAAAKzIesf0vbaqHldVxywurKrDquo7quqMJI9cXnkAAADsj/VG+k5I8mNJ/ryqvi7JR5NcN8khSV6T5Lnd/S/LLREAAIB9tdvQ192fTfKCJC+oqmsnOTzJZ7r7YweiOAAAAPbPemfvvG6Sn0xyuyRvT/KH3X31gSgMAACA/bfeMX1nJNme5B1JHpDk2UuvCAAAgA2z3jF9x3X3NyZJVb0oyVuWXxIAAAAbZb2Rvi/suGJaJwAAwNaz3kjfXarqE/P1SnK9+XYl6e6+8VKrAwAAYL+sd/bOQw5UIQAAAGy89aZ3AgAAsIUJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAY2FJDX1WdUFXvrKpLqurk3bR7cFV1VW1fZj0AAAAHm6WFvqo6JMnzk9w/yXFJHl5Vx+2k3Y2SPCHJPy2rFgAAgIPVMkf67p7kku6+tLs/n+QlSR60k3a/muSZST67xFoAAAAOSssMfUcmuXzh9hXzsi+rqm9OcnR3n7PEOgAAAA5aKzuRS1VdK8lzkvzsHrY/qaouqKoLrrrqquUWBwAAMIhlhr4rkxy9cPuoedkON0pypySvq6rLktwzydm7OplLd5/W3du7e/sRRxyxpJIBAADGsszQd36SY6vqtlV1WJKHJTl7x8ru/nh3H97d27p7W5I3J3lgd1+wxJoAAAAOKksLfd19dZLHJjkvyb8lObO7L6qqp1XVA5f1uAAAAHzFocvsvLvPTXLummWn7KLt8cusBQAA9sW2k5dzzsHLTj1xKf3CWis7kQsAAADLt9SRPgAAgNFstdFfI30AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMLBDV10AHEjbTj5nKf1eduqJS+kXAAD2l5E+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgh666ALa2bSefs5R+Lzv1xKX0CwAABxsjfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgR266gKAXdt28jlL6feyU09cSr8AAGw+RvoAAAAGttTQV1UnVNU7q+qSqjp5J+ufVFUXV9Xbq+rvquo2y6wHAADgYLO00FdVhyR5fpL7JzkuycOr6rg1zf4lyfbuvnOSlyV51rLqAQAAOBgtc6Tv7kku6e5Lu/vzSV6S5EGLDbr7td396fnmm5MctcR6AAAADjrLDH1HJrl84fYV87JdeXSSVy+xHgAAgIPOpjh7Z1X9cJLtSb5tN21OSnJSkhxzzDEHqDIAAICtbZkjfVcmOXrh9lHzsmuoqvsl+aUkD+zuz+2qs+4+rbu3d/f2I444YsOLBQAAGNEyQ9/5SY6tqttW1WFJHpbk7MUGVfVNSX4/U+D74BJrAQAAOCgtLfR199VJHpvkvCT/luTM7r6oqp5WVQ+cm/1GkhsmOauq3lZVZ++iOwAAAPbBUo/p6+5zk5y7ZtkpC9fvt8zHBwAAONgt9cfZAQAAWC2hDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADOzQVRewbNtOPmcp/V526olL6RcAAGAjGekDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGNihqy6Aa9p28jlL6feyU09cSr8AAMDmZqQPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADCwpYa+qjqhqt5ZVZdU1ck7WX+dqnrpvP6fqmrbMusBAAA42Cwt9FXVIUmen+T+SY5L8vCqOm5Ns0cn+Wh33y7JbyV55rLqAQAAOBgtc6Tv7kku6e5Lu/vzSV6S5EFr2jwoyRnz9ZcluW9V1RJrAgAAOKgsM/QdmeTyhdtXzMt22qa7r07y8SS3WGJNAAAAB5Xq7uV0XPWQJCd094/Pt38kyT26+7ELbS6c21wx337X3OZDO+nvpCQnzTfvkOSdSyj78CRf9dibmHqXS73Lpd7lUu9yqXe51Ltc6l0u9S7XVqs3WV7Nt+nuI/ak4aFLePAdrkxy9MLto+ZlO2tzRVUdmuQmST68s866+7Qkpy2hzi+rqgu6e/syH2MjqXe51Ltc6l0u9S6XepdLvcul3uVS73JttXqTzVHzMqd3np/k2Kq6bVUdluRhSc5e0+bsJI+crz8kyf/tZQ09AgAAHISWNtLX3VdX1WOTnJfkkCR/2N0XVdXTklzQ3WcneVGSP6mqS5J8JFMwBAAAYIMsc3pnuvvcJOeuWXbKwvXPJvmBZdawl5Y6fXQJ1Ltc6l0u9S6XepdLvcul3uVS73Kpd7m2Wr3JJqh5aSdyAQAAYPWWeUwfAAAAKyb0AQyiqk6vqletuo5RbeXtW1VPrapHrbqOUW3lfWOzq6pXVdXpq64DtjqhD2AcT0jyw6suYmC2L8AKVdXrqup3VvTYl1VV7+Ry6rx+23z7w1V1k93VPX9RtOP+X6iqD1bVa6vqZ6rq2suoX+gD2IWqOr+qfnPh9s2r6rOrrGl3uvvj3f2xVdcxKtsX4OBSVTeuqpsuLHpakq9dc/m1NXe7fpKT96D7v53vvy3JdyV5ZZJfSfLGqrrBQg1HVVXt63PYQejbAqrqhKp6Y1V9tKr+u6r+tqruUlX3r6oLquozVfWuqjpl/pH7TWP+ZuMFVfWMqvrQ/E3Gb1bVptz3quppVXVhVX2qqj5eVedU1Z1XXdfuVNV9qurNCzW/parutOq6trqqulmS7UnO2bGsuz+S5K0rK2odW22KWVUdv4tvTV+36tp2Zqtt362iqq4/b9tPVdV/VdUvmtK38Rbf31Zdy67sbF9YWPeoXbxenL7Ckr/KTl7XHrXqmnZnK32GmP/W35bkZxa277YlPM4hVfXdVfVnST6Q5C4Lqz/Z3R9Yc1n7f44QtPEAAAoUSURBVOp5SZ5QVUeu81Cfm+9/ZXe/rbufk+T4JN+c5MkL7X41yaXz/+Hb7evz2pQfvPkqN8i0A90jyd2TXJ7k9UnOSvL7Se6U5CeS/GiSX15RjbvzQ0muTvKtSR6b5IlJfnClFe3a12faht+U5H5J/jvJm6rq2JVWtQtzyP+rJG/K9KJ0jyTPTfLFVdY1iNvO/16yZvl/HuhC9lZV3aiqrqqq91XV9Vddz278Q675ben2JB9L8roV1rSuLbR9v6y7n9rdp6+6jl34zSTfmeTBSe6b6fX3PklSVbepqs9V1b9sxDfdB7nF97fNapf7QpKX5pqvF9+a5DNJLjjwZe7WPyS543z9wZnq3pS24GeIJyT5xyR/lK/sB5dvVOdVdceqetbc50szfQY8Ickb9rKrs5K8I9Oo4F7p7guT/HWmfWeHx8993TvJO6vq76vqpDUjkOvykw1bUFUdkuSjSV7Q3ScvLP/hJM/p7luurLg15m/sr9Pd37Kw7G+SvKe7f3xlhe2heVu/PcmbuvsnVl3PWlV18yQfTnJ8d79+1fWMpKrunemF/r/XrDqsuw9bQUnrmr8FPby7v6eqnpTkM939uysua49U1fWSvDHJe5M8uDfhm9NW3r6bVVXdMNNr2I9194sXll2R5BXd/aiqel6SV3f3q1dY6m4t7hurrmWr2pN9YaHtYUn+PtMozAM32+tFVR2e5Kok397dr1txObu0FT9DzJ8rL+zux25Qf7fINDjxyCTfmClw/UmSV86/J77Y9rJMQfMLa7p5WHe/ah51fHeSu2UasPm7JHfp7ovW1r2714yajhF8fHd/1ZeKVXV0pmPLfyTTl9NnJzkjyV9395d291w31VRAdq6qbpsp4X9LkltmGqG9fpInVtXiTn9IkutW1Y27+xMHvtJdevua2+/L9Dw2ve7+YlW9NtO3X5tOd39kfuE4r6r+LtMLzMu6+72rrWwIO8LefTO9Ke7wjBXUsi+en2S3bwCbxTyCc3qm17Af2Wwf4HZhy2zfTe7rkxyW6dv7JEl3f6qq3rHQ5pe6+5MHvDIOtD3ZF3b4jSS3SnLCFnm92JR8hkiSPC7JUzKN0N6+uy9bp/1zkrxozbL3r23U3a+vqvOS/HqSB+5lTZVkp/t1d18+9/nrVfXwJL+b5KGZAuBlu+tU6NsaXpXp26yfSnJlks9nelH8vUzpfq3NNl9/7TcinU06tXgXxzocluSfD3Qte6q7f7SqnptpCsIDkzy9qr6vu89bcWlb3aXzv5/s7i9P8ayq66yonr3S3Z9bdQ174ZRMU7ju1t1rR1Y3pS22fbc0gW9jLL6/dfcNV1nL/qiq70/y05lGpz68Xnt2z2eInJbpc+ojklxYVX+ZaaTv77p7Z9NcP7z4mWAdJyd52zxzaG8cl698BrmGeWTyBzON9G1P8jeZssAV63W6KT9481WOS/Jr3f033X1xpp3z8CT/T3dfspOLb5/33Q0yzaO+68LlnGzik3ckSXf/a3c/s7uPz3Q81CNXW9HWN5+l8fxMc+iTfDnw3WeXd9pEquo6tclO7LQzVfWQTAesP6i7133T2iy2yvbdAt6V6T3tnjsW1HTWujutuc3+W3x/24zW3RdmP53p8+sfVdWvVtV1D1yJY9pinyE+n2lWyIbo7vd199O7+w6ZzuXwqSQvSXJFVT27qvb5/0t3vyPJHyd51p7eZz6JzglJXraw7DpV9ZCqekWmUcWfmNcf3d0P6O6XdvfV6/Ut9M1qc/9w7YeS/ERVHVtV35rkzEw70YNqOivmnarqDvMO8fTVlrrl/U2Sn01yo/n2AzKdSem5qypod6rqtlV1alV963zCg29PcuckF6+6tt3Z5P/fFv1qkl+r6Uy5t0vyB9kCJ3KZ/XSSTX3c7PzmdkaSX0zy3qq61Xy5+YpL2xObfvtuBfNZ716U5JlV9Z1Vdcckf5hrfqh7RlWdsJICx7L2/W1T2cN9IZkCyR0zfVn0kCR/dkALXcd8vOGOQ1huPh+vvClt0c8QlyW5e02/iXd4beDZ4Lv7zd39U5mO23tcktsnOX/NSN2NFt6rdlxustMOJ6dk+qJlZ4cJXWe+/61rOiv/kzKF7rdmOqnRDi/IdEjBu5Pcvbvv0t3P7u4P7M3zE/q2hodk2vHenuTFSV7S3Y/M9I3EvZK8Oclbkvxc1pnPy7oenmkK7aszbe/vS/Kd3b1ZP+h/OtO+cVaS/8j0AfrFSZ65yqJG0d2vzBRIfi/T/nCzTPvEplZVN8pU9/+7mT9wZJqacv1MX6q8f+Hy8lUWtZ4ttH23iv+T5LVJ/nL+98LMZ8urqtsk+clMwc/ZO/fP4vvbZrXLfWGHeWTm37v7FUkek+T75/1ks/jWJBfN1/8im/ds5cnW/Azxm5lG+y7OdLKcYzb6Abr7c939su7+3iRH5poh+JRc8/3q/ZkC2a76ujzTGfh3NiJ9v/n+7810POUDkzw1yX3WHOrw60mO7O7/3d1v29fn5eyds6p6apLLNvEprWEY/r8BsL+q6j6ZfsLqdt39rlXXA5uZ0AcAwKZWVT+Q5HqZTqz235lOr/+sJB/q7v+5ytpgK3AQOgAAm93nMh3Gcock1870A9ovz+aeigibhpE+AACAgTmRCwAAwMCEPgAAgIEJfQAAAAMT+gAYVlV9sareVlUXVtVZVXX9+Ud9L1x1bQBwoAh9AIzsM9191+6+U6Yf9P3JVRcEAAea0AfAweKNSW43Xz+kqv6gqi6qqtdU1fWSpKruWlVvrqq3V9VfVtXN5uWvq6pnVtVbquo/qure8/JDquo3qur8+T4/MS8/vqpeX1V/VVWXVtWpVfVD8/3fUVVfP7c7oqr+Yr7/+VV1rwO/WQAYndAHwPCq6tAk90/yjnnRsUme3913TPKxJA+el/9xkp/v7jvPbZ+y0M2h3X33JE9cWP7oJB/v7rsluVuSx1TVbed1d8k0svgNSX4kye3n+78wyePmNr+d5Lfm+z94XgcAG8qPswMwsutV1dvm629M8qIkt07y7u7esfytSbZV1U2S3LS7Xz8vPyPJWQt9vXyx/Xz9u5LcuaoeMt++SaZA+fkk53f3+5Okqt6V5DVzm3ck+fb5+v2SHFdVOx7jxlV1w+7+1L4/ZQC4JqEPgJF9prvvurhgDlifW1j0xSTX24O+dtzni/nK+2cleVx3n7fmMY5f8xhfWrj9pYX7XyvJPbv7s3vw+ACwT0zvBIAk3f3xJB/dcbxepimZr9/NXZLkvCQ/VVXXTpKqun1V3WAvHvY1+cpUz1TVXXfTFgD2iZE+APiKRyb5vaq6fpJLk/zoOu1fmGmq5z/XNIR4VZLv24vHe3yS51fV2zO9J78hzjAKwAar7l51DQAAACyJ6Z0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgf3/vqCm5rywrG8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_pronunciation_probability(\n",
    "    early_stopped_language_model,\n",
    "    ('ˌ', 'æ', 'n', 'ə', 's', 'ˌ', 'θ', 'iː', 'z', 'iː', 'ˈ', 'ɑː', 'l', 'ə', 'dʒ', 'ɪ', 's', 't')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though this model didn't technically overfit the training set, the model is still way more confident in its predictions than I would have hoped.\n",
    "\n",
    "So ultimately I'll abandon my dev set perplexity as my metric. Language models are often evaluated on downstream tasks, but unfortunately I don't have a clear metric for my downstream task of generating \"good\" English pronunciations. I instead generated about 10 models and manually inspected them, looking for models that have the following characteristics:\n",
    "* Doesn't horribly overfit long words in the training set\n",
    "* Both low and high perplexity pronunciations from the training set make sense\n",
    "* The generated pronunciations feel pronouncable to me\n",
    "\n",
    "The model I ended up liking most is a GRU with an embedding dimension of 20, hidden dimension of 20, and a single layer. I train it until train loss stops dropping, so there's no early stopping here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 50: train loss: 0.5415\tdev loss: 0.7894                                                                                               \n",
      "\tGenerated: in train: 3%, assess: 0%, novel: 97%\n",
      "\t ˈ s k ɪ ŋ b ə l\n",
      "\t ˈ m aɪ t ˌ r ɑː s t ə ˌ k uː t\n",
      "\t ɑː ˈ r ɛ n d iː ɝ\n",
      "\t ˈ uː d ə ˌ g eɪ b ɝ\n",
      "\t ˈ r ɔ r n ə\n",
      "Epoch 100: train loss: 0.5311\tdev loss: 0.7751                                                                                               \n",
      "\tGenerated: in train: 13%, assess: 1%, novel: 86%\n",
      "\t ˈ ɛ n ˌ r ə s t\n",
      "\t ˈ dʒ ʌ k f l oʊ\n",
      "\t p ɪ ˈ k ɑː tʃ\n",
      "\t ˈ l aɪ t ə n\n",
      "\t ˈ l æ k ə l iː z\n",
      "Epoch 150: train loss: 0.5279\tdev loss: 0.7706                                                                                               \n",
      "\tGenerated: in train: 10%, assess: 3%, novel: 87%\n",
      "\t ˈ s t ɛ l oʊ s\n",
      "\t ˈ t æ θ s t\n",
      "\t ˈ æ m p ə ˌ s t r iː z\n",
      "\t ˈ s ɪ l ə s ə\n",
      "\t b ɪ ˈ z ɛ r\n",
      "Epoch 200: train loss: 0.5261\tdev loss: 0.7682                                                                                               \n",
      "\tGenerated: in train: 10%, assess: 2%, novel: 88%\n",
      "\t ˈ s oʊ m ə m\n",
      "\t r ə g n iː ˈ ɑː n d ɪ k\n",
      "\t ˈ m ɑː r m ə n\n",
      "\t s ə ˈ b ɔ r b ə n\n",
      "\t ˈ s l ɔ l ˌ k eɪ ˌ n ɪ z\n",
      "Epoch 250: train loss: 0.5251\tdev loss: 0.7669                                                                                               \n",
      "\tGenerated: in train: 23%, assess: 1%, novel: 76%\n",
      "\t ˈ ɪ l oʊ m ə n\n",
      "\t ˈ k w ɔ r g ˌ h ɑː n d\n",
      "\t ˈ p aɪ ɝ ˌ t eɪ t ɪ d\n",
      "\t ˈ r æ n d ə n\n",
      "\t ˈ ʃ r ɑː f ə v ə n\n",
      "Epoch 300: train loss: 0.5243\tdev loss: 0.7658                                                                                               \n",
      "\tGenerated: in train: 14%, assess: 0%, novel: 86%\n",
      "\t ˈ t r oʊ dʒ ə n s ə n z\n",
      "\t ˈ p l ɛ g ə s\n",
      "\t ˈ tʃ oʊ z\n",
      "\t ˈ p ɝː t ɝ oʊ z\n",
      "\t ˈ f ɪ ʒ ɝ ɪ ŋ\n",
      "Epoch 350: train loss: 0.5238\tdev loss: 0.7652                                                                                               \n",
      "\tGenerated: in train: 7%, assess: 0%, novel: 93%\n",
      "\t ˈ b oʊ l k n\n",
      "\t ˌ j uː t ˈ s eɪ dʒ\n",
      "\t r iː ˈ l iː b ə n\n",
      "\t r uː d ə ˈ g ɔ r z\n",
      "\t ˌ ɪ n ˈ s t ɝː d s ɪ n\n",
      "Epoch 400: train loss: 0.5232\tdev loss: 0.7644                                                                                               \n",
      "\tGenerated: in train: 19%, assess: 3%, novel: 78%\n",
      "\t ˈ k r uː dʒ k ɝ\n",
      "\t ˈ k æ k w ə l ə l\n",
      "\t ˈ f l æ ŋ\n",
      "\t ˌ t ɛ n j ə ˈ m iː s ə n i\n",
      "\t ˈ v iː d ə n\n",
      "Epoch 450: train loss: 0.5230\tdev loss: 0.7643                                                                                               \n",
      "\tGenerated: in train: 17%, assess: 2%, novel: 81%\n",
      "\t n ə ˈ t iː m\n",
      "\t ˈ z iː l ə p s\n",
      "\t ˈ s k oʊ s t ɪ n\n",
      "\t ˈ dʒ uː l ə\n",
      "\t ˈ p uː l ə n\n",
      "CPU times: user 34min 24s, sys: 2min 45s, total: 37min 9s                                                                                    \n",
      "Wall time: 34min 16s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Early stopping because of no decrease in 3 epochs.\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "smallish_model_parameters = ModelParams(\n",
    "    rnn_type='gru', num_layers=1, embedding_dimension=20, hidden_dimension=20,\n",
    "    max_epochs=1000, early_stopping_rounds=3\n",
    ")\n",
    "\n",
    "smallish_language_model = LanguageModel(vocab, smallish_model_parameters, device_name='cuda')\n",
    "\n",
    "_ = smallish_language_model.fit(\n",
    "    train_pronunciations.pronunciation.values.tolist(),\n",
    "    dev_pronunciations.pronunciation.values.tolist(),\n",
    "    print_every=50,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dev loss of this model is considerably higher than the more optimal model I trained above. But in practice I found that its predictions made sense to me. So this means that a loss of about .76 gets us pretty close to representing the basic structure of allowable English pronunciations, whereas the much lower loss occurs when a model has learned the probabilities of long sequences of phonemes. \n",
    "\n",
    "I'll see what this new, *smallish*, model predicts for /ˌænəsˌθiːziːˈɑːlədʒɪst/:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAH0CAYAAACJqkSAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3debxkV1kv/N9DQgjzlCBmolECr4EbUAOoXDAKaiAa9IIMrygogsMF5KJiHN6AKBhQEH0FlUFBxRsSRAwkGNQLAVQ0QRGSIBpCIAkgAcKkTIHn/rF3Q+XQ3aeHU13nrP5+P5/6dNWuVaue2mf3OfWrtfaq6u4AAAAwpuutugAAAACWR+gDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AIOoqqdW1Z/s5WMfVVVv3sX9r62qR+6obVV9qqq+Zm+edw9rvGFVvbqqPl5VZy37+Raet6vqDhvQzy738VZUVb9QVS9adR27o6pOrKorV10HwCoIfQArVFWXV9Wn5+D0H1X1kqq6yarrWqu779/dL93JfTfp7suSZK7/V5dUxoOTfFWSW3f396+9cw69n5/35fbLx5ZUyw6tqeFjVfV3VfXN+7OG/am7n9HdP7qnj6uqb66qT1bVQQvbXriTbb+3UfUCHKiEPoDV+57uvkmSb0hyQpJfWtugJgf67+zbJfm37r52F21ePofQ7Zdb7K/i1taQ5PAkb07yyqqqFdSxmV2Y6T3INyxsu3eSK9dsu0+SN+5p51V18D5VBzCYA/0NBMCm0d1XJXltkrskSVW9oaqeXlV/m+S/knxNVR1RVWdX1Uer6tKqesyabg6tqpfPIyb/VFV33X5HVZ1aVe+e77ukqr5vzWOrqn5nnj75r1V134U73lBVOxzR2T79saoem+QHkjx5Hul6dVX9bFX92Zr2v11Vv7WTvr5ufq6PVdXFVXXKvP2Xk5yW5KFz349ef49+Rd+/VVVXVNUnquqtVXXvhfsOmqcqbt8/b62qoxcefr+q+ve5ruftTojr7s8neWmS2ya59cJz/UZVXVNV76mq+y9s3+nPdh5BPLOq/miu7+KqOmHNY/+sqq6e+33CmseeVVV/Mj/2HVV1x6r6+ar60LxPvnOh/c2r6sVV9YGquqqqfnVx9G3NPv3SlOKq2jYfC4+sqvdV1Yer6hd3sW/ekinUpapuk+SQJGeu2XbHzKFvN/bPK+bX+Ikkj6ppOvBL5n19SZK7r/czAxiV0AewScwh4wFJ/nlh8w8meWySmyZ5b5IzMo2GHJFpuuMzqurbF9o/MMlZSW6V5E+TvKqqrj/f9+5Moyk3T/LLSf6kqr564bH3nNscluQpmUaobrW79Xf3C5K8LMmz5lG270nyJ0lOqqpbzK/x4CQPS/JHO3j910/y6iSvS3KbJI9P8rKqulN3PyXJM/LlkbwX725dCy5Icrd8ed+cVVWHzvc9KcnDM+3/myX5kUxBe7vvzhQajk/ykCTftd6TVdUNkjwqyRXd/eF58z2TvCvTPn5WkhcvBMj1franzG1ukeTsJL8zP8/1Mu23f0lyZJL7JnliVS3W+D1J/jjJLTMdX+dleg9wZJKnJfn9hbYvSXJtkjsk+fok35lkT6Zw/vckd5rrOK2qvm4n7d6YOeDN/755vixue093bz8Pb3eO/Vdk2j8vy3QMf+18+a4kj9yD1wAwFKEPYPVeVdO5Z29Ocn6mcLPdS7r74nlK422T3CvJz3X3Z7r7bUlelOSHFtq/tbtfMY+kPCfJoUm+KUm6+6zufn93f7G7X57k35PcY+GxH0ry3O7+/Hz/u5KcvC8vrLs/kOnN/fZz8E5K8uHufusOmn9TkpskOb27P9fd/yfJazKFsd31kHk0bvvl9Qu1/El3f6S7r+3uZye5QaZwkkyh5pe6+109+Zfu/shCv6d398e6+31JXp8pPO6yhiRXJPnGJIsjqu/t7hd29xcyjQJ+dZKvmgP/ej/bN3f3ufNj/zjJ9lHcuyc5vLufNu+3y5K8MFO43u5N3X3efBydlWnq6enzcXJGkm1VdYuq+qpMwfeJ3f2f3f2hJL+5pq/1/HJ3f7q7/yVTEL3rTtqdn+S/z6H33knelOTvk3zTwrbzky99ILLe/vn77n7VfHx/OlM4f3p3f7S7r0jy23vwGgCGYs47wOp9b3f/9U7uu2Lh+hFJPtrdn1zY9t5M5wF+Rfvu/mJNqxUekSRV9UOZRrS2zU1ukmnEabururvX9H3EHryOnXlpkp/IFEQekSmw7MgRmUbFvrimhiP34LnO7O5H7OiOqvqZJI+en6czjehtf/1HZxrl3JkPLlz/r0z7bo9rWOynu/9rHuS7Sabpn+v9bNfWcOg8cnq7JEfUdRetOShTiNruPxaufzpT8P7Cwu3tdRyR5PpJPrAwg/V6ue5xuJ7d3Vdvme+7S6ZRvd/t7k9V1RUL27YHtT069hces7jtvXvwGgCGYqQPYHNbDGHvT3KrqrrpwrZjkly1cPtL56HN0/6OSvL+qrpdptD1uEyrX94iyUVJFs9NO3LNuWrHzM+5t/Vu96okx1fVXTJNk3zZTh77/iRH13UXrFn7+vbKfP7ekzON/txyfv0fz5df/xWZpgGuyu78bHfmikzTIG+xcLlpdz9gL+q4Islnkxy20NfNuvvOe9HXLnX3ZzJNuf2eJF/d3f863/Wmedvx+fIiLruzf9Yeex/Iwv+HuT3AAUnoA9gi5ilqf5fk16rq0Ko6PtPI1eJ3831jVf2PeQToiZnewL8lyY0zvSm+Okmq6oczLxiz4DZJnlBV16+q70/ydUnO3cMy/yPJdb6zb35z/4pM59H94zxFckf+IdPI0JPnGk7M9Ob/jD2sYUdumuk8tauTHFxVp2Ua6dvuRUl+paqOrcnxVXXrHXW0DLv5s92Zf0zyyar6uXnxkoOq6i5VtccLl8zTcV+X5NlVdbOqul5VfW1Vfeue9rWb3pjkpzK99u3ePG/7QHe/e65rb/bPmUl+vqpuWVVHZTpHFOCAJPQBbC0PzzQ98/1J/jzJU9ZMDf2LJA9Nck2mRWD+x3yO3iVJnp3pnKn/SPLfkvztmr7/IcmxST6c5OlJHrzmvLbd8eIkx83n071qYftL5+fc2dTOdPfnMoW8+881PD/JDy2MAO2O7at7Ll5uk2nhkr9M8m+Zpvl9Jted+vecTCHhdUk+Mb+OG+7B826E9X62OzRP0/zuTOcZvifTvntRpgV79sYPZVpJ85JMx9ErMp17uAznZ/qwYfFL6988b3vTmrZ7un9+OdPP+j2Zfq47PfYARlfXPX0DADZeVR2T5F+T3La7P7HqegDgQGKkD4Clms/Re1KSMwQ+ANj/rN4JwNJU1Y0zTSd9b6avawAA9jPTOwEAAAZmeicAAMDAhD4AAICBbclz+g477LDetm3bqssAAABYibe+9a0f7u7Dd6ftlgx927Zty4UXXrjqMgAAAFaiqt67u21N7wQAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGAHr7oAAGDPbTv1nKX0e/npJy+lXwBWx0gfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAa21NBXVSdV1buq6tKqOnUH9x9TVa+vqn+uqrdX1QOWWQ8AAMCBZmmhr6oOSvK8JPdPclySh1fVcWua/VKSM7v765M8LMnzl1UPAADAgWiZI333SHJpd1/W3Z9LckaSB65p00luNl+/eZL3L7EeAACAA87BS+z7yCRXLNy+Msk917R5apLXVdXjk9w4yf2WWA8AAMABZ9ULuTw8yUu6+6gkD0jyx1W1w5qq6rFVdWFVXXj11Vfv1yIBAAC2qmWGvquSHL1w+6h526JHJzkzSbr775McmuSwHXXW3S/o7hO6+4TDDz98CeUCAACMZ5mh74Ikx1bV7avqkEwLtZy9ps37ktw3Sarq6zKFPsN4AAAAG2Rpoa+7r03yuCTnJXlnplU6L66qp1XVKXOzn07ymKr6lyT/O8mjuruXVRMAAMCBZpkLuaS7z01y7pptpy1cvyTJvZZZAwAAwIFs1Qu5AAAAsERCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGttTQV1UnVdW7qurSqjp1J20eUlWXVNXFVfWny6wHAADgQHPwsjquqoOSPC/JdyS5MskFVXV2d1+y0ObYJD+f5F7dfU1V3WZZ9QAAAByIljnSd48kl3b3Zd39uSRnJHngmjaPSfK87r4mSbr7Q0usBwAA4ICzzNB3ZJIrFm5fOW9bdMckd6yqv62qt1TVSTvrrKoeW1UXVtWFV1999RLKBQAAGM+qF3I5OMmxSU5M8vAkL6yqW+yoYXe/oLtP6O4TDj/88P1YIgAAwNa1zNB3VZKjF24fNW9bdGWSs7v78939niT/likEAgAAsAGWGfouSHJsVd2+qg5J8rAkZ69p86pMo3ypqsMyTfe8bIk1AQAAHFCWFvq6+9okj0tyXpJ3Jjmzuy+uqqdV1Slzs/OSfKSqLkny+iQ/290fWVZNAAAAB5qlfWVDknT3uUnOXbPttIXrneRJ8wUAAIANtuqFXAAAAFgioQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADW+rqnQAAALuy7dRzltb35aefvLS+txIjfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwA5edQEAAABbybZTz1lKv5effvJS+jXSBwAAMDChDwAAYGDrTu+sqqOSPCzJvZMckeTTSS5Kck6S13b3F5daIQAAAHttl6Gvqv4wyZFJXpPkmUk+lOTQJHdMclKSX6yqU7v7jcsuFAAAgD233kjfs7v7oh1svyjJK6vqkCTHbHxZAAAAbIRdhr6dBL7F+z+X5NINrQgAAIANs1tf2VBV90ry1CS3mx9TSbq7v2Z5pQEAALCvdvd7+l6c5H8leWuSLyyvHAAAADbS7oa+j3f3a5daCQAAABtud0Pf66vq15O8Mslnt2/s7n9aSlUAAABsiN0Nffec/z1hYVsn+faNLQcAAICNtFuhr7u/bdmFAAAAsPGut+oCAAAAWB6hDwAAYGBCHwAAwMD2OPRV1QuWUQgAAAAbb29G+k5YvwkAAACbwd6Evg9teBUAAAAsxR6Hvu4+aRmFAAAAsPEs5AIAADAwoQ8AAGBgQh8AAMDADt7VnVX15O5+VlX9/0l67f3d/YSlVQYAAMA+22XoS/LO+d8Ll10IAAAAG2+Xoa+7Xz3/+9L9Uw4AAAAbyTl9AAAAAxP6AAAABib0AQAADGyvQ19VffdGFgIAAMDG25eRvrtvWBUAAAAsxV6Hvu5+ykYWAgAAwMbbl+mdt93IQgAAANh4+zK988UbVgUAAABLsS/TO0/eyEIAAADYeL6yAQAAYGD7ck7fazayEAAAADbevoz0PWbDqgAAAGAp9uWcvg9sZCEAAABsPOf0AQAADEzoAwAAGJjQBwAAMLBdhr6qunlVnV5V/1pVH62qj1TVO+dtt9hfRQIAALB31hvpOzPJNUlO7O5bdfetk3zbvO3MZRcHAADAvlkv9G3r7md29we3b+juD3b3M5PcbrmlAQAAsK/WC33vraonV9VXbd9QVV9VVT+X5IrllgYAAMC+Wi/0PTTJrZOcP5/T99Ekb0hyqyQPWXJtAAAA7KODd3Vnd1+T5OfmCwAAAFuMr2wAAAAYmNAHAAAwMKEPAABgYEIfAADAwHa5kEtVvSdJ76rJfP9zu/u3N7IwAAAA9t16q3fefn8VAgAAwMYzvRMAAGBgQh8AAMDAhD4AAICBCX0AAAADW2/1zjO7+yFV9Y5cdxXPStLdffxSqwMAAGCf7DL0Jfmp+d/vXnYhAAAAbLz1Qt8Hk6S737uzBlVV3b2r7/IDAABgRdY7p+/1VfX4qjpmcWNVHVJV315VL03yyOWVBwAAwL5Yb6TvpCQ/kuR/V9XXJLkmyaFJDkryuiTP7e5/Xm6JAAAA7K1djvR192e6+/ndfa8kxyS5b5Jv6O7bdfdj1gt8VXVSVb2rqi6tqlN30e5BVdVVdcJevQoAAAB2aL3VOw9N8uNJ7pDk7Un+oLuv3Z2Oq+qgJM9L8h1JrkxyQVWd3d2XrGl300wLxvzDnpcPAADArqx3Tt9Lk5yQ5B1JHpDk2XvQ9z2SXNrdl3X355KckeSBO2j3K0memeQze9A3AAAAu2G90Hdcdz+iu38/yYOT3HsP+j4yyRULt6+ct31JVX1DkqO7+5z1Oquqx1bVhVV14dVXX70HZQAAABy41gt9n99+ZXende6uqrpekuck+endad/dL+juE7r7hMMPP3wjSwEAABjWeqt33rWqPjFfryQ3nG9Xku7um+3isVclOXrh9lHztu1umuQuSd5QVUly2yRnV9Up3X3hHrwGAAAAdmKXoa+7D9qHvi9IcmxV3T5T2HtYkv93oe+PJzls++2qekOSnxH4AAAANs560zv32jwd9HFJzkvyziRndvfFVfW0qjplWc8LAADAl603vXOfdPe5Sc5ds+20nbQ9cZm1AAAAHIiWNtIHAADA6gl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCW+uXsjG/bqecspd/LTz95Kf0CAMCBxkgfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAY2MGrLgAAgH2z7dRzltLv5aefvJR+gf3LSB8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBHbzqAgBgM9h26jlL6ffy009eSr8AsLuM9AEAAAxsqaGvqk6qqndV1aVVdeoO7n9SVV1SVW+vqr+pqtstsx4AAIADzdJCX1UdlOR5Se6f5LgkD6+q49Y0++ckJ3T38UlekeRZy6oHAADgQLTMkb57JLm0uy/r7s8lOSPJAxcbdPfru/u/5ptvSXLUEusBAAA44Cwz9B2Z5IqF21fO23bm0Uleu8R6AAAADjibYvXOqnpEkhOSfOsu2jw2yWOT5JhjjtlPlQEAAGxtyxzpuyrJ0Qu3j5q3XUdV3S/JLyY5pbs/u7POuvsF3X1Cd59w+OGHb3ixAAAAI1pm6LsgybFVdfuqOiTJw5Kcvdigqr4+ye9nCnwfWmItAAAAB6Slhb7uvjbJ45Kcl+SdSc7s7our6mlVdcrc7NeT3CTJWVX1tqo6eyfdAQAAsBeWek5fd5+b5Nw1205buH6/ZT4/AADAgW5TLOQCAACb1bZTz1lKv5effvJS+oW1lnlOHwAAACsm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgQh8AAMDAhD4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGNjBqy4AABjftlPPWUq/l59+8lL6BRiJkT4AAICBCX0AAAADM70TYIswPQ4A2BtG+gAAAAYm9AEAAAxM6AMAABiYc/oAAGAgzgFnLSN9AAAAAxP6AAAABib0AQAADMw5fQAAazgnChiJkT4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNU7OaBYjQ0AgAONkT4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGNjBqy5g2bades5S+r389JOX0i8AAMBGMtIHAAAwMKEPAABgYEIfAADAwIQ+AACAgQl9AAAAAxP6AAAABib0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAM7eNUFcF3bTj1nKf1efvrJS+kXAADY3Iz0AQAADEzoAwAAGJjQBwAAMDChDwAAYGBCHwAAwMCEPgAAgIEJfQAAAAMT+gAAAAYm9AEAAAxM6AMAABiY0AcAADAwoQ8AAGBgB6+6AAAADizbTj1nKf1efvrJS+kXtjojfQAAAAMz0gdsGJ/cAgBsPkIfAEvhQwAA2BxM7wQAABiY0AcAADCwpU7vrKqTkvxWkoOSvKi7T19z/w2S/FGSb0zykSQP7e7Ll1kTbCWmxwEAsK+WNtJXVQcleV6S+yc5LsnDq+q4Nc0eneSa7r5Dkt9M8sxl1QMAAHAgWuZI3z2SXNrdlyVJVZ2R5IFJLllo88AkT52vvyLJ71RVdXcvsS6AJEZSAYADwzLP6TsyyRULt6+ct+2wTXdfm+TjSW69xJoAAAAOKLWsQbWqenCSk7r7R+fbP5jknt39uIU2F81trpxvv3tu8+Ed9PfYJI+db94pybuWUPZhSb7iuTcx9S6XepdLvcul3uVS73Kpd7nUu1zqXa6tVm+yvJpv192H707DZU7vvCrJ0Qu3j5q37ajNlVV1cJKbZ1rQ5St09wuSvGAJdX5JVV3Y3Scs8zk2knqXS73Lpd7lUu9yqXe51Ltc6l0u9S7XVqs32Rw1L3N65wVJjq2q21fVIUkeluTsNW3OTvLI+fqDk/wf5/MBAABsnKWN9HX3tVX1uCTnZfrKhj/o7our6mlJLuzus5O8OMkfV9WlST6aKRgCAACwQZb6PX3dfW6Sc9dsO23h+meSfP8ya9hDS50+ugTqXS71Lpd6l0u9y6Xe5VLvcql3udS7XFut3mQT1Ly0hVwAAABYvWWe0wcAAMCKCX0Ag6iql1TVa1Zdx6i28v6tqqdW1aNWXceotvKxsdlV1Wuq6iWrrgO2OqEPYBw/leQRqy5iYPYvwApV1Ruq6ndW9NyXV1Xv4HL6fP+2+fZHqurmu6p7/qBo++M/X1UfqqrXV9X/rKrrL6N+oQ9gJ6rqgqr6jYXbt6qqz6yypl3p7o9398dWXceo7F+AA0tV3ayqbrGw6WlJvnrN5VfXPOxGSU7dje7/en78tiTfmeTVSX45yZuq6sYLNRxVVbW3r2E7oW8LqKqTqupNVXVNVf1nVf11Vd21qu5fVRdW1aer6t1Vddr8JfebxvzJxvOr6hlV9eH5k4zfqKpNeexV1dOq6qKq+lRVfbyqzqmq41dd165U1X2q6i0LNf9jVd1l1XVtdVV1yyQnJDln+7bu/miSt66sqHVstSlmVXXiTj41fcOqa9uRrbZ/t4qqutG8bz9VVf9RVb9gSt/GW/z7tupadmZHx8LCfY/aye+Ll6yw5K+wg99rj1p1Tbuyld5DzD/rb03yPxf277YlPM9BVfVdVfWnST6Y5K4Ld3+yuz+45rL2/9RvJ/mpqjpynaf67Pz4q7r7bd39nCQnJvmGJE9eaPcrSS6b/w/fYW9f16Z8481XuHGmA+ieSe6R5Iok5yc5K8nvJ7lLkh9L8sNJfmlFNe7KDyS5Nsm3JHlckicmeehKK9q5r820D78+yf2S/GeSN1fVsSutaifmkP8XSd6c6ZfSPZM8N8kXVlnXIG4//3vpmu3/vr8L2VNVddOqurqq3l9VN1p1Pbvwd7nup6UnJPlYkjessIx7OwkAAAlgSURBVKZ1baH9+yXd/dTufsmq69iJ30jyHUkelOS+mX7/3idJqup2VfXZqvrnjfik+wC3+Pdts9rpsZDk5bnu74tvSfLpJBfu/zJ36e+S3Hm+/qBMdW9KW/A9xE8l+fskf5gvHwdXbFTnVXXnqnrW3OfLM70HPCnJG/ewq7OSvCPTqOAe6e6LkvxlpmNnuyfMfd07ybuq6m+r6rFrRiDX5SsbtqCqOijJNUme392nLmx/RJLndPdtVlbcGvMn9jfo7m9e2PZXSd7b3T+6ssJ207yv357kzd39Y6uuZ62qulWSjyQ5sbvPX3U9I6mqe2f6Rf+fa+46pLsPWUFJ65o/BT2su7+7qp6U5NPd/bsrLmu3VNUNk7wpyfuSPKg34R+nrbx/N6uqukmm32E/0t0vW9h2ZZJXdfejquq3k7y2u1+7wlJ3afHYWHUtW9XuHAsLbQ9J8reZRmFO2Wy/L6rqsCRXJ/m27n7DisvZqa34HmJ+X3lRdz9ug/q7dabBiUcm+W+ZAtcfJ3n1/H3ii20vzxQ0P7+mm4d192vmUcf3JLl7pgGbv0ly1+6+eG3du/qdUdM5gk/o7q/4ULGqjs50bvkPZvpw+uwkL03yl939xV291k01FZAdq6rbZ0r435zkNplGaG+U5IlVtXjQH5Tk0Kq6WXd/Yv9XulNvX3P7/Zlex6bX3V+oqtdn+vRr0+nuj86/OM6rqr/J9AvmFd39vtVWNoTtYe++mf4obveMFdSyN56XZJd/ADaLeQTnJZl+h/3gZnsDtxNbZv9ucl+b5JBMn94nSbr7U1X1joU2v9jdn9zvlbG/7c6xsN2vJ7ltkpO2yO+LTcl7iCTJ45M8JdMI7R27+/J12j8nyYvXbPvA2kbdfX5VnZfk15Kcsoc1VZIdHtfdfcXc569V1cOT/G6Sh2QKgJfvqlOhb2t4TaZPs34iyVVJPpfpl+LvZUr3a222+fprPxHpbNKpxTs51+GQJP+0v2vZXd39w1X13ExTEE5J8vSq+t7uPm/FpW11l83/frK7vzTFs6pusKJ69kh3f3bVNeyB0zJN4bp7d68dWd2Uttj+3dIEvo2x+Petu2+yylr2RVV9X5KfzDQ69ZH12rNr3kPkBZnep/5Qkouq6s8zjfT9TXfvaJrrRxbfE6zj1CRvm2cO7Ynj8uX3INcxj0w+NNNI3wlJ/ipTFrhyvU435RtvvsJxSX61u/+quy/JdHAeluT/6e5Ld3Dx6fPeu3GmedR3W7ick028eEeSdPe/dPczu/vETOdDPXK1FW198yqNF2SaQ5/kS4HvPjt90CZSVTeoTbaw045U1YMznbD+wO5e94/WZrFV9u8W8O5Mf9O+afuGmlatu8ua2+y7xb9vm9G6x8LsJzO9f/3DqvqVqjp0/5U4pi32HuJzmWaFbIjufn93P72775RpLYdPJTkjyZVV9eyq2uv/L939jiR/lORZu/uYeRGdk5K8YmHbDarqwVX1qkyjij823390dz+gu1/e3deu17fQN6vN/cW1H07yY1V1bFV9S5IzMx1ED6xpVcy7VNWd5gPi6astdcv7qyQ/neSm8+0HZFpJ6bmrKmhXqur2VXV6VX3LvODBtyU5Psklq65tVzb5/7dFv5LkV2taKfcOSV6YLbCQy+wnk2zq82bnP24vTfILSd5XVbedL7dacWm7Y9Pv361gXvXuxUmeWVXfUVV3TvIHue6bumdU1UkrKXAsa/++bSq7eSwkUyC5c6YPix6c5E/3a6HrmM833H4Ky63m85U3pS36HuLyJPeo6TvxDqsNXA2+u9/S3T+R6by9xye5Y5IL1ozU3XThb9X2y8132OHktEwftOzoNKEbzI8/oqZV+Z+UKXS/NdOiRts9P9MpBe9Jco/uvmt3P7u7P7gnr0/o2xoenOnAe3uSlyU5o7sfmekTiXsleUuSf0zys1lnPi/renimKbSvzbS/vzfJd3T3Zn2j/1+Zjo2zkvxbpjfQL0vyzFUWNYrufnWmQPJ7mY6HW2Y6Jja1qrppprr/v838hiPT1JQbZfpQ5QMLl1eusqj1bKH9u1X8TJLXJ/nz+d+LMq+WV1W3S/LjmYKf1Tv3zeLft81qp8fCdvPIzL9296uSPCbJ983HyWbxLUkunq//WTbvauXJ1nwP8RuZRvsuybRYzjEb/QTd/dnufkV3f0+SI3PdEHxarvv36gOZAtnO+roi0wr8OxqRvt/8+PdlOp/ylCRPTXKfNac6/FqSI7v7f3X32/b2dVm9c1ZVT01y+SZe0hqG4f8bAPuqqu6T6Sus7tDd7151PbCZCX0AAGxqVfX9SW6YaWG1/8y0vP6zkny4u//7KmuDrcBJ6AAAbHafzXQay52SXD/TF2i/Mpt7KiJsGkb6AAAABmYhFwAAgIEJfQAAAAMT+gAAAAYm9AEwrKr6QlW9raouqqqzqupG85f6XrTq2gBgfxH6ABjZp7v7bt19l0xf6Pvjqy4IAPY3oQ+AA8Wbktxhvn5QVb2wqi6uqtdV1Q2TpKruVlVvqaq3V9WfV9Ut5+1vqKpnVtU/VtW/VdW95+0HVdWvV9UF82N+bN5+YlWdX1V/UVWXVdXpVfUD8+PfUVVfO7c7vKr+bH78BVV1r/2/WwAYndAHwPCq6uAk90/yjnnTsUme1913TvKxJA+at/9Rkp/r7uPntk9Z6Obg7r5HkicubH90ko93992T3D3JY6rq9vN9d800svh1SX4wyR3nx78oyePnNr+V5Dfnxz9ovg8ANpQvZwdgZDesqrfN19+U5MVJjkjynu7evv2tSbZV1c2T3KK7z5+3vzTJWQt9vXKx/Xz9O5McX1UPnm/fPFOg/FySC7r7A0lSVe9O8rq5zTuSfNt8/X5Jjquq7c9xs6q6SXd/au9fMgBcl9AHwMg+3d13W9wwB6zPLmz6QpIb7kZf2x/zhXz572cleXx3n7fmOU5c8xxfXLj9xYXHXy/JN3X3Z3bj+QFgr5jeCQBJuvvjSa7Zfr5epimZ5+/iIUlyXpKfqKrrJ0lV3bGqbrwHT/u6fHmqZ6rqbrtoCwB7xUgfAHzZI5P8XlXdKMllSX54nfYvyjTV859qGkK8Osn37sHzPSHJ86rq7Zn+Jr8xVhgFYINVd6+6BgAAAJbE9E4AAICBCX0AAAADE/oAAAAGJvQBAAAMTOgDAAAYmNAHAAAwMKEPAABgYEIfAADAwP4vN7Z+1SBzCaIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_pronunciation_probability(\n",
    "    smallish_language_model,\n",
    "    ('ˌ', 'æ', 'n', 'ə', 's', 'ˌ', 'θ', 'iː', 'z', 'iː', 'ˈ', 'ɑː', 'l', 'ə', 'dʒ', 'ɪ', 's', 't')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While many of the probabilities the model assigns are still suspiciously high, this looks better than the previous models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Final Model ##\n",
    "Now that I've found the best parameters for the model according to the dev set I'll train a final model using all of the data. This should increase model performance overall since more data is better, but is also necessary since I'll be using the model to predict probabilities of all English words. If some of those words weren't in the training set they would artificially get lower probabilities. (Another approach here could be to train a model on e.g. 4/5 folds of the data and make predictions about the remaining 1/5, doing that 5 times to get unbiased predictions for all data, but this would have taken much longer to run.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 50: train loss: 0.5409loss: 0.5461                                                                                                    \n",
      "\tGenerated: in train: 16%, assess: 0%, novel: 84%\n",
      "\t ˈ m ɔɪ m ə n t\n",
      "\t r uː ˈ n iː dʒ ɪ v\n",
      "\t s ə ˈ t r t eɪ n z\n",
      "\t ə ʃ ˈ tʃ oʊ p ɝ k\n",
      "\t ˈ h ɔɪ n ɝ\n",
      "Epoch 100: train loss: 0.5325loss: 0.5357                                                                                                    \n",
      "\tGenerated: in train: 12%, assess: 0%, novel: 88%\n",
      "\t k ə t ˈ ɝː d ə k\n",
      "\t ə n ˈ t r iː s l ɝ\n",
      "\t ˈ b ʊ k\n",
      "\t z ə ˈ k ɔ n t iː z\n",
      "\t ˈ h ɝː t ə s ɪ ŋ\n",
      "Epoch 150: train loss: 0.5299loss: 0.5286                                                                                                    \n",
      "\tGenerated: in train: 5%, assess: 0%, novel: 95%\n",
      "\t v ɑː ˈ k ɝː g ə n t\n",
      "\t ˌ iː v ɛ r t ə ˈ l æ m p i\n",
      "\t ˈ t æ t r ə\n",
      "\t ˌ t æ n ə n ˈ t r ɔ f k ə\n",
      "\t ˈ w eɪ m ɝ z\n",
      "Epoch 200: train loss: 0.5282loss: 0.5163                                                                                                    \n",
      "\tGenerated: in train: 12%, assess: 0%, novel: 88%\n",
      "\t ɪ ˈ n ɛ f ɪ s ə n\n",
      "\t ˈ k r ɪ t s ə m\n",
      "\t d p iː ˈ h eɪ n\n",
      "\t ˈ k ɑː n t s\n",
      "\t ˈ b j uː t ə n ɪ d z\n",
      "Epoch 250: train loss: 0.5271loss: 0.5260                                                                                                    \n",
      "\tGenerated: in train: 15%, assess: 0%, novel: 85%\n",
      "\t ˈ j ʊ r i\n",
      "\t ɑː n ˈ s ɛ d ɪ ŋ\n",
      "\t ˈ p r æ p\n",
      "\t ˈ m ɪ n d l i\n",
      "\t ˈ k r ɛ m n ɪ k s\n",
      "CPU times: user 23min 25s, sys: 1min 56s, total: 25min 21s                                                                                   \n",
      "Wall time: 23min 14s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Early stopping because of no decrease in 3 epochs.\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "\n",
    "final_language_model = LanguageModel(vocab, smallish_model_parameters, device_name='cuda')\n",
    "\n",
    "_ = final_language_model.fit(\n",
    "    pronunciations.pronunciation.values.tolist(),\n",
    "    print_every=50,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, I'll save the model so I can use it in the next notebook, `Model Exploration and Usage.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/gru_20_20_1.pt', 'wb') as fh:\n",
    "    final_language_model.save(fh)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}