posts/2023-06-10_transformers/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "a00b8d88-fc87-4561-a9f6-e8cd32a5e3aa",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"Transformers From Scratch\"\n",
    "author: \"Mat Miller\"\n",
    "date: \"01/06/2024\"\n",
    "categories: [python, transformers]\n",
    "image: transformer_coding_1.png\n",
    "card-style: summary\n",
    "format:\n",
    "  html:\n",
    "    css: '../../styles.css'\n",
    "    number-sections: true\n",
    "    toc: true\n",
    "    theme: cosmo\n",
    "    highlight-style: tango\n",
    "    code-fold: false\n",
    "    self-contained: true\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b965d4ab",
   "metadata": {},
   "source": [
    "[![Open in Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/matdmiller/blog/blob/main/posts/2023-06-10_transformers/notebook.ipynb)<br><br>\n",
    "In this blog we're going to walk through creating and training a transformer from scratch. We'll go through each foundational element step by step and explain what is happening along the way. This blog is written in a Jupyter notebook which you can [download](https://github.com/matdmiller/blog/blob/main/posts/2023-06-10_transformers/notebook.ipynb) and use to run the code yourself as you follow along. Running the code as you follow along and changing it to see how the output changes will help you learn the concepts better than reading alone. While this is a lengthy topic, please don't be too alarmed with the length of the notebook or the amount of code. Most of it is copied from previous cells as we build up the transformer. Rather than just showing the code that was changed which would have shortened things up considerably, I chose to copy all required code down to the next cell to allow this entire notebook to be run from top to bottom. This should make it easier to run as well as allow you to experiment with each new concept as we go.\n",
    "\n",
    "I'll be closely following [Andrej Karpathy's YouTube video 'Let's build GPT: from scratch, in code, spelled out.'](https://www.youtube.com/watch?v=kCc8FmEb1nY). We'll be heavily referencing the [Attention Is All You Need](https://arxiv.org/abs/1706.03762) paper as we go. If you would rather you can download the final version of the code from [Andrej's repo](https://github.com/karpathy/ng-video-lecture). The dataset we'll be using for this can downloaded [here](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt). You may also find referencing Andrej's [nanoGPT](https://github.com/karpathy/nanoGPT) repo helpful as well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bff786fa-66b6-46ff-ab62-690ce1528ac3",
   "metadata": {},
   "source": [
    "# Getting started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38ce4605-8a98-4d48-9075-867759ee2e61",
   "metadata": {},
   "source": [
    "To get started we'll first download the mini shakespeare dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cf3512b-78fc-44ae-b109-67a869bfa38e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-01-06 22:00:26--  https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1115394 (1.1M) [text/plain]\n",
      "Saving to: ‘input.txt.2’\n",
      "\n",
      "input.txt.2         100%[===================>]   1.06M  --.-KB/s    in 0.01s   \n",
      "\n",
      "2024-01-06 22:00:26 (108 MB/s) - ‘input.txt.2’ saved [1115394/1115394]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "!wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f50de3d5-1c6f-4bea-b276-3d89f12820c0",
   "metadata": {},
   "source": [
    "Let's open it up and take a peek at what's inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20c0dbc5-a40e-4697-9ab1-12f90d1d83a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of input.txt (characters): 1115394\n",
      "First 500 characters: First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n",
      "All:\n",
      "We know't, we know't.\n",
      "\n",
      "First Citizen:\n",
      "Let us kill him, and we'll have corn at our own price.\n",
      "Is't a verdict?\n",
      "\n",
      "All:\n",
      "No more talking on't; let it be done: away, away!\n",
      "\n",
      "Second Citizen:\n",
      "One word, good citizens.\n",
      "\n",
      "First Citizen:\n",
      "We are accounted poor\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "with open('input.txt') as f:\n",
    "    text = f.read()\n",
    "print('Length of input.txt (characters):',len(text))\n",
    "print('First 500 characters:',text[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd72b83-e1bf-47b1-a461-795c0ac11809",
   "metadata": {},
   "source": [
    "# Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "015da39c-5b7b-4e09-b239-177576158b73",
   "metadata": {},
   "source": [
    "Now that we have our dataset the first thing we need to do is break it down into tokens which will be fed into the model. This is known as tokenization. There are various techniques for tokenization, `sub-word` being used in most of the more modern LLM's. Because our dataset is small for this toy problem, we are instead going to tokenize by each individual character. This makes tokenzation much simpler to implement and will also significantly reduce the number of unique tokens. The first thing we need to do is to go through the dataset and get a list of all unique characters or tokens which is known as the vocab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7591cb0b-3181-475a-918d-0d02f431194f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab Size: 65\n",
      "Vocab: ['\\n', ' ', '!', '$', '&', \"'\", ',', '-', '.', '3', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "vocab = sorted(list(set(text)))\n",
    "vocab_size = len(vocab)\n",
    "print('Vocab Size:',vocab_size)\n",
    "print('Vocab:',vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a71e1ed4-ae39-4af8-947c-0fee8f102baf",
   "metadata": {},
   "source": [
    "The neural networks are unable to take in characters directly. We must first convert the tokens to integers, which in our case means converting each individual character into integers. Those integers will then be used to index into the set of token embeddings. Token embeddings are learned vectors that represent each token that will be passed into the model. Let's create our character to index (integer) mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c04083bc-1102-48d5-8438-b9e5c63ee88a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Character to index: {'\\n': 0, ' ': 1, '!': 2, '$': 3, '&': 4, \"'\": 5, ',': 6, '-': 7, '.': 8, '3': 9, ':': 10, ';': 11, '?': 12, 'A': 13, 'B': 14, 'C': 15, 'D': 16, 'E': 17, 'F': 18, 'G': 19, 'H': 20, 'I': 21, 'J': 22, 'K': 23, 'L': 24, 'M': 25, 'N': 26, 'O': 27, 'P': 28, 'Q': 29, 'R': 30, 'S': 31, 'T': 32, 'U': 33, 'V': 34, 'W': 35, 'X': 36, 'Y': 37, 'Z': 38, 'a': 39, 'b': 40, 'c': 41, 'd': 42, 'e': 43, 'f': 44, 'g': 45, 'h': 46, 'i': 47, 'j': 48, 'k': 49, 'l': 50, 'm': 51, 'n': 52, 'o': 53, 'p': 54, 'q': 55, 'r': 56, 's': 57, 't': 58, 'u': 59, 'v': 60, 'w': 61, 'x': 62, 'y': 63, 'z': 64}\n",
      "Index to character:, {0: '\\n', 1: ' ', 2: '!', 3: '$', 4: '&', 5: \"'\", 6: ',', 7: '-', 8: '.', 9: '3', 10: ':', 11: ';', 12: '?', 13: 'A', 14: 'B', 15: 'C', 16: 'D', 17: 'E', 18: 'F', 19: 'G', 20: 'H', 21: 'I', 22: 'J', 23: 'K', 24: 'L', 25: 'M', 26: 'N', 27: 'O', 28: 'P', 29: 'Q', 30: 'R', 31: 'S', 32: 'T', 33: 'U', 34: 'V', 35: 'W', 36: 'X', 37: 'Y', 38: 'Z', 39: 'a', 40: 'b', 41: 'c', 42: 'd', 43: 'e', 44: 'f', 45: 'g', 46: 'h', 47: 'i', 48: 'j', 49: 'k', 50: 'l', 51: 'm', 52: 'n', 53: 'o', 54: 'p', 55: 'q', 56: 'r', 57: 's', 58: 't', 59: 'u', 60: 'v', 61: 'w', 62: 'x', 63: 'y', 64: 'z'}\n",
      "Tokenization of `Hello World!`: [20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]\n",
      "String for token sequence `[20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]`: Hello World!\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "char2idx = {char:idx for idx,char in enumerate(vocab)}\n",
    "idx2char = {idx:char for char,idx in char2idx.items()}\n",
    "encode = lambda x: [char2idx[char] for char in x]\n",
    "decode = lambda idxs: ''.join([idx2char[idx] for idx in idxs])\n",
    "print('Character to index:',char2idx)\n",
    "print('Index to character:,',idx2char)\n",
    "print('Tokenization of `Hello World!`:',encode('Hello World!'))\n",
    "print('String for token sequence `[20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]`:',decode([20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e17f19f-2ac1-47c1-ac2a-bee45f8cd39f",
   "metadata": {},
   "source": [
    "Now that we have our tokenizer we can go through and tokenize the dataset. This step may sometimes be done on the fly during training and inference instead of ahead of time with large datasets, but for our small dataset we can easily and quickly do this without significant memory utilization."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f85b2f28-7341-406f-88cd-7309e8ae7218",
   "metadata": {},
   "source": [
    "We are going to import pytorch and create a tensor for our encoded dataset. Pytorch is the library we'll be using to create and train the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "722b8a0e-8cad-47c9-ba4b-efdfa6258e1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded Text shape: torch.Size([1115394]) Encoded Text Dtype: torch.int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([18, 47, 56,  ..., 45,  8,  0])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "import torch\n",
    "encoded_text = torch.tensor(encode(text))\n",
    "print('Encoded Text shape:',encoded_text.shape, 'Encoded Text Dtype:', encoded_text.dtype)\n",
    "encoded_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45936865-b8e5-4a58-8c92-9a9fd32e0d75",
   "metadata": {},
   "source": [
    "Now that we have the encoded text, we need to split it into a training and validation set. Creating a validation set will allow us to test the performance of our model after each epoch, or at an another interval of our choosing, to ensure that it's training correctly or improving over time. A typical split between training and validation is 90/10 which is what we'll use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c956234d-eef8-417c-9afb-d204197eadf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1003854"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "train_split_pct = 0.9\n",
    "train_split_idx = int(len(encoded_text)*train_split_pct)\n",
    "train_split_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf42d843-2aca-4781-a979-7c9e418fc0cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data length: 1003854 Valid data length: 111540 Train percentage: 0.8999994620734916\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "train_data = encoded_text[:train_split_idx]\n",
    "valid_data = encoded_text[train_split_idx:]\n",
    "print('Train data length:',len(train_data),'Valid data length:',len(valid_data),\n",
    "      'Train percentage:',len(train_data)/len(encoded_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1855521-e120-4ff8-9b1f-dc1003674a23",
   "metadata": {},
   "source": [
    "Next we need to choose our context length. The context lenth is the maximum length of the sequence used when training the transformer. This is sometimes also referred to as the block size, which is how Andrej refers to it. When the transformer is trained, it is trained on each combination of tokens up to the maximum context length. For example if the context length was 5 the transformer would be trained on (0,1), (0,1,2), (0,1,2,3), and (0,1,2,3,4). Let's look at a more concrete example in code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46c0216b-4cfd-46c0-99a0-a8d28ca51d69",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "idx: 0 x: tensor([18]) y: tensor(47)  | decoded version: x: F y: i\n",
      "idx: 1 x: tensor([18, 47]) y: tensor(56)  | decoded version: x: Fi y: r\n",
      "idx: 2 x: tensor([18, 47, 56]) y: tensor(57)  | decoded version: x: Fir y: s\n",
      "idx: 3 x: tensor([18, 47, 56, 57]) y: tensor(58)  | decoded version: x: Firs y: t\n",
      "idx: 4 x: tensor([18, 47, 56, 57, 58]) y: tensor(1)  | decoded version: x: First y:  \n",
      "idx: 5 x: tensor([18, 47, 56, 57, 58,  1]) y: tensor(15)  | decoded version: x: First  y: C\n",
      "idx: 6 x: tensor([18, 47, 56, 57, 58,  1, 15]) y: tensor(47)  | decoded version: x: First C y: i\n",
      "idx: 7 x: tensor([18, 47, 56, 57, 58,  1, 15, 47]) y: tensor(58)  | decoded version: x: First Ci y: t\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "context_length = 8\n",
    "for i in range(context_length):\n",
    "    x,y = train_data[:i+1], train_data[i+1]\n",
    "    print('idx:',i,'x:',x,'y:',y,' | decoded version: x:',decode(x.tolist()),'y:',decode(y[None].tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "895e5af0-d98c-499a-b58e-dc16af63348d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "TORCH_SEED = 1337 #Setting a manual torch seed for reproducable results\n",
    "torch.manual_seed(TORCH_SEED) #Used to compare against @karpathy's lecture\n",
    "context_length = 8 #Maximum number of tokens used in each training sequence\n",
    "batch_size = 4 #number of batches that will be trained in parallel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f3f574c-46f1-4473-ab68-d0c80691a57f",
   "metadata": {},
   "source": [
    "# Data Loader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7735beda-a6c1-4a5c-8b3a-f062a20a1a78",
   "metadata": {},
   "source": [
    "Now we'll implement a function to get a batch of data from our training and validation datasets. We'll specify which dataset to pull from as a parameter and return the inputs and targets which we're naming x and y respectively. We'll go ahead and print out everything through the process so you can see what's going on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a308609d-8d38-4d21-bc90-03f55f60a5cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs:\n",
      "shape: torch.Size([4, 8])\n",
      "tensor([[24, 43, 58,  5, 57,  1, 46, 43],\n",
      "        [44, 53, 56,  1, 58, 46, 39, 58],\n",
      "        [52, 58,  1, 58, 46, 39, 58,  1],\n",
      "        [25, 17, 27, 10,  0, 21,  1, 54]])\n",
      "targets:\n",
      "shape: torch.Size([4, 8])\n",
      "tensor([[43, 58,  5, 57,  1, 46, 43, 39],\n",
      "        [53, 56,  1, 58, 46, 39, 58,  1],\n",
      "        [58,  1, 58, 46, 39, 58,  1, 46],\n",
      "        [17, 27, 10,  0, 21,  1, 54, 39]]) \n",
      "-----------------------------------------------\n",
      "Given input context ([24]) the target is: 43\n",
      "Given input context ([24, 43]) the target is: 58\n",
      "Given input context ([24, 43, 58]) the target is: 5\n",
      "Given input context ([24, 43, 58, 5]) the target is: 57\n",
      "Given input context ([24, 43, 58, 5, 57]) the target is: 1\n",
      "Given input context ([24, 43, 58, 5, 57, 1]) the target is: 46\n",
      "Given input context ([24, 43, 58, 5, 57, 1, 46]) the target is: 43\n",
      "Given input context ([24, 43, 58, 5, 57, 1, 46, 43]) the target is: 39\n",
      "Given input context ([44]) the target is: 53\n",
      "Given input context ([44, 53]) the target is: 56\n",
      "Given input context ([44, 53, 56]) the target is: 1\n",
      "Given input context ([44, 53, 56, 1]) the target is: 58\n",
      "Given input context ([44, 53, 56, 1, 58]) the target is: 46\n",
      "Given input context ([44, 53, 56, 1, 58, 46]) the target is: 39\n",
      "Given input context ([44, 53, 56, 1, 58, 46, 39]) the target is: 58\n",
      "Given input context ([44, 53, 56, 1, 58, 46, 39, 58]) the target is: 1\n",
      "Given input context ([52]) the target is: 58\n",
      "Given input context ([52, 58]) the target is: 1\n",
      "Given input context ([52, 58, 1]) the target is: 58\n",
      "Given input context ([52, 58, 1, 58]) the target is: 46\n",
      "Given input context ([52, 58, 1, 58, 46]) the target is: 39\n",
      "Given input context ([52, 58, 1, 58, 46, 39]) the target is: 58\n",
      "Given input context ([52, 58, 1, 58, 46, 39, 58]) the target is: 1\n",
      "Given input context ([52, 58, 1, 58, 46, 39, 58, 1]) the target is: 46\n",
      "Given input context ([25]) the target is: 17\n",
      "Given input context ([25, 17]) the target is: 27\n",
      "Given input context ([25, 17, 27]) the target is: 10\n",
      "Given input context ([25, 17, 27, 10]) the target is: 0\n",
      "Given input context ([25, 17, 27, 10, 0]) the target is: 21\n",
      "Given input context ([25, 17, 27, 10, 0, 21]) the target is: 1\n",
      "Given input context ([25, 17, 27, 10, 0, 21, 1]) the target is: 54\n",
      "Given input context ([25, 17, 27, 10, 0, 21, 1, 54]) the target is: 39\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "def get_batch(train_valid):\n",
    "    data = train_data if train_valid == 'train' else valid_data\n",
    "    data_len = len(data)\n",
    "    start_idxs = torch.randint(high=len(data) - context_length, size=(batch_size,)) #tensor([ 76049, 234249, 934904, 560986])\n",
    "    # print('start_idxs:',start_idxs)\n",
    "    x = torch.stack([data[i:i+context_length] for i in start_idxs])\n",
    "    y = torch.stack([data[i+1:i+context_length+1] for i in start_idxs])\n",
    "    return x,y\n",
    "xb, yb = get_batch('train')\n",
    "print('inputs:')\n",
    "print('shape:',xb.shape)\n",
    "print(xb)\n",
    "print('targets:')\n",
    "print('shape:',yb.shape)\n",
    "print(yb,'\\n-----------------------------------------------')\n",
    "for batch_idx in range(batch_size):\n",
    "    for sequence_idx in range(context_length):\n",
    "        context = xb[batch_idx,:sequence_idx+1]\n",
    "        target = yb[batch_idx,sequence_idx]\n",
    "        print(f\"Given input context ({context.tolist()}) the target is: {target}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d28096cf-0b6b-41df-899d-ff4af4436b03",
   "metadata": {},
   "source": [
    "You may have noticed that the inputs and targets appear to be the same, just shifted by one, which is correct. To help give you a better understanding on how the inputs and targets are combined, we printed out every combination of input and target. As you can see each input sequence is really multiple sequences starting with the first token in the sequence as the input and the second token in the sequence being the target all the way to the full input sequence being the input and the subsequent character being the target."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67c06d58-9c86-4b05-8031-37f94c5582a4",
   "metadata": {},
   "source": [
    "# Bigram Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f0fade0-4897-454a-bc8b-9a1d2e759d74",
   "metadata": {},
   "source": [
    "Before jumping into using a transformer we'll start with a bigram model. A bigram model predicts the probability of one token following another. For example given the token for the letter 'a' what is the probability of each token in the vocab will be the next token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbafe199-ddd0-4e65-bf80-fe12d7c64c01",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c44c073d-3d9c-4787-b1d8-c4c9b3200348",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bigram Model Output Shapes out: torch.Size([4, 8, 65]) xb: torch.Size([4, 8]) yb: torch.Size([4, 8]) embeddings: Embedding(65, 65)\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class BigramLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        #Each token reads off the logits (~probabilities) from the subsequent token from the lookup table\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        logits = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        return logits\n",
    "\n",
    "bigram_model = BigramLanguageModel(vocab_size=vocab_size)\n",
    "out = bigram_model(xb, yb)\n",
    "print('Bigram Model Output Shapes out:',out.shape,'xb:',xb.shape,'yb:',yb.shape,'embeddings:',bigram_model.token_embedding_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e18eb5-86d9-4772-93b3-4cc467f4b4b6",
   "metadata": {},
   "source": [
    "Now we'll add a loss function to the forward method. We will use the negative log likelihood, which is known in pytorch as the [cross entropy loss](https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html). To be able to use the cross entropy loss we'll need to reshape the output and targets to match the format that it expects. The model output should be a 2D tensor (B\\*T x C) and targets should be a 1D tensor (B\\*T). We need to squash the batch and time dimensions on the model output and the batch and time dimensions of the targets using the `.view` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a255f18a-7df5-480f-bc2b-623a9a4705d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bigram Model Output Shapes out: torch.Size([4, 8, 65]) xb: torch.Size([4, 8]) yb: torch.Size([4, 8])\n",
      "The calculated loss is: tensor(4.8786, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class BigramLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        #Each token reads off the logits from the subsequent token from the lookup table\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        logits = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        B,T,C = logits.shape\n",
    "        logits_reshaped = logits.view(B*T,C)\n",
    "        targets_reshaped = targets.view(B*T)\n",
    "        loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        return logits, loss\n",
    "\n",
    "bigram_model = BigramLanguageModel(vocab_size=vocab_size)\n",
    "out,loss = bigram_model(xb, yb)\n",
    "print('Bigram Model Output Shapes out:',out.shape,'xb:',xb.shape,'yb:',yb.shape)\n",
    "print('The calculated loss is:',loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e405d6a1-21f4-4937-b326-5d828c2b4e7d",
   "metadata": {},
   "source": [
    "Now we're going to add a generate method to perform character generation from our model. Instead of explaining here what each generation step is doing, detailed code comments have been added before each step. There are also commented out `print` statements that you can uncomment and run to help you understand what's going on as well.<br>\n",
    "\n",
    "One pytorch function you may not have seen before is [`torch.multinomial`](https://pytorch.org/docs/stable/generated/torch.multinomial.html). It returns `num_samples`, in our case 1, based on the weighted probability distribution of the predictions for each token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a004c675-f14c-4a56-b091-1cf60ff865cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(4.8786, grad_fn=<NllLossBackward0>)\n",
      "100 Generated Tokens: \n",
      "Sr?qP-QWktXoL&jLDJgOLVz'RIoDqHdhsV&vLLxatjscMpwLERSPyao.qfzs$Ys$zF-w,;eEkzxjgCKFChs!iWW.ObzDnxA Ms$3\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class BigramLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        #Each token reads off the logits from the subsequent token from the lookup table\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        logits = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Get predictions - calling `forward`\n",
    "            logits, loss = self(idx)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "bigram_model = BigramLanguageModel(vocab_size=vocab_size)\n",
    "logits, loss = bigram_model(xb,yb)\n",
    "print('Loss:',loss)\n",
    "idx=torch.zeros((1,1), dtype=torch.long)#Creating a single batch with single time step with index of 0 \n",
    "#which is the `\\n` new line char to be our starting point.\n",
    "#Now let's generate a character sequence to see what it looks like.\n",
    "print('100 Generated Tokens:',decode(bigram_model.generate(idx,max_new_tokens=100)[0].tolist()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb10addf-b189-420c-9dfd-6874d6834de9",
   "metadata": {},
   "source": [
    "The output of this is totally nonsensical noise because the model is not trained.  Next we'll get set up to train the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be670c7a-1c5b-4d9d-b782-68231bb901db",
   "metadata": {},
   "source": [
    "The first thing we'll do is create an optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bec6e4a-ba02-4f27-90de-378bf8ccaaeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "optimizer = torch.optim.Adam(params=bigram_model.parameters(), lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "003d2d7d-447e-45ef-acb2-449d747dd023",
   "metadata": {},
   "source": [
    "For more info on the `Adam` optimizer or optimizers in general, check out this great [video](https://youtu.be/nlVOG2Nzc3k?si=Lj7UH14PG7u48F4M&t=995) from fast.ai. <br>\n",
    "<br>\n",
    "Next, we'll create a basic training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "088eb32d-b874-4bd7-9723-f0342c526940",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 4.704 Loss: 4.721 Loss: 4.653 Loss: 4.706 Loss: 4.781 Loss: 4.751 Loss: 4.84 Loss: 4.668 Loss: 4.744 Loss: 4.774 Loss: 4.691 Loss: 4.789 Loss: 4.618 Loss: 4.651 Loss: 4.887 Loss: 4.704 Loss: 4.758 Loss: 4.655 Loss: 4.709 Loss: 4.675 Loss: 4.761 Loss: 4.79 Loss: 4.654 Loss: 4.662 Loss: 4.673 Loss: 4.666 Loss: 4.73 Loss: 4.756 Loss: 4.712 Loss: 4.746 Loss: 4.727 Loss: 4.735 Loss: 4.778 Loss: 4.644 Loss: 4.666 Loss: 4.798 Loss: 4.718 Loss: 4.684 Loss: 4.819 Loss: 4.614 Loss: 4.574 Loss: 4.561 Loss: 4.816 Loss: 4.607 Loss: 4.62 Loss: 4.726 Loss: 4.651 Loss: 4.595 Loss: 4.721 Loss: 4.7 Loss: 4.673 Loss: 4.729 Loss: 4.662 Loss: 4.617 Loss: 4.6 Loss: 4.654 Loss: 4.717 Loss: 4.693 Loss: 4.668 Loss: 4.656 Loss: 4.656 Loss: 4.755 Loss: 4.724 Loss: 4.618 Loss: 4.705 Loss: 4.753 Loss: 4.57 Loss: 4.549 Loss: 4.572 Loss: 4.57 Loss: 4.599 Loss: 4.549 Loss: 4.592 Loss: 4.661 Loss: 4.624 Loss: 4.743 Loss: 4.645 Loss: 4.644 Loss: 4.581 Loss: 4.584 Loss: 4.723 Loss: 4.67 Loss: 4.631 Loss: 4.556 Loss: 4.682 Loss: 4.676 Loss: 4.657 Loss: 4.67 Loss: 4.65 Loss: 4.676 Loss: 4.654 Loss: 4.787 Loss: 4.732 Loss: 4.598 Loss: 4.588 Loss: 4.764 Loss: 4.742 Loss: 4.604 Loss: 4.663 Loss: 4.589 "
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "batch_size = 32\n",
    "for steps in range(100):\n",
    "    #Sample a batch of data\n",
    "    xb, yb = get_batch('train')\n",
    "    \n",
    "    #Evaluate the loss\n",
    "    logits, loss = bigram_model(xb, yb)\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    print('Loss:',round(loss.item(),3), end=' ')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cd434ff-d468-4823-87d0-96c6301f1a7f",
   "metadata": {},
   "source": [
    "As you can see the loss is improving, but only a little bit.  Next we'll train for longer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e8c638b-fec9-4c77-bd6a-4d105cd8d177",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Loss: 4.66 Step: 100 Loss: 4.472 Step: 200 Loss: 4.323 Step: 300 Loss: 4.256 Step: 400 Loss: 4.245 Step: 500 Loss: 4.166 Step: 600 Loss: 4.049 Step: 700 Loss: 4.097 Step: 800 Loss: 3.75 Step: 900 Loss: 3.707 "
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "batch_size = 32\n",
    "for steps in range(1000):\n",
    "    #Sample a batch of data\n",
    "    xb, yb = get_batch('train')\n",
    "    \n",
    "    #Evaluate the loss\n",
    "    logits, loss = bigram_model(xb, yb)\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if steps % 100 == 0: print('Step:',steps,'Loss:',round(loss.item(),3), end=' ')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c427db7b-7718-4fce-a38f-a205c372c308",
   "metadata": {},
   "source": [
    "The loss seems to be steadily decreasing. Let's train for a lot longer and then see what the results look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3040ae25-8c06-41dd-80e9-858cc2877168",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Loss: 3.716 Step: 100 Loss: 3.638 Step: 200 Loss: 3.425 Step: 300 Loss: 3.433 Step: 400 Loss: 3.427 Step: 500 Loss: 3.304 Step: 600 Loss: 3.286 Step: 700 Loss: 3.19 Step: 800 Loss: 3.203 Step: 900 Loss: 3.139 Step: 1000 Loss: 3.003 Step: 1100 Loss: 3.06 Step: 1200 Loss: 2.959 Step: 1300 Loss: 2.982 Step: 1400 Loss: 2.92 Step: 1500 Loss: 2.841 Step: 1600 Loss: 2.89 Step: 1700 Loss: 2.975 Step: 1800 Loss: 2.808 Step: 1900 Loss: 2.777 Step: 2000 Loss: 2.747 Step: 2100 Loss: 2.685 Step: 2200 Loss: 2.68 Step: 2300 Loss: 2.687 Step: 2400 Loss: 2.81 Step: 2500 Loss: 2.692 Step: 2600 Loss: 2.665 Step: 2700 Loss: 2.631 Step: 2800 Loss: 2.752 Step: 2900 Loss: 2.581 Step: 3000 Loss: 2.629 Step: 3100 Loss: 2.625 Step: 3200 Loss: 2.548 Step: 3300 Loss: 2.582 Step: 3400 Loss: 2.603 Step: 3500 Loss: 2.618 Step: 3600 Loss: 2.572 Step: 3700 Loss: 2.511 Step: 3800 Loss: 2.607 Step: 3900 Loss: 2.508 Step: 4000 Loss: 2.572 Step: 4100 Loss: 2.494 Step: 4200 Loss: 2.523 Step: 4300 Loss: 2.483 Step: 4400 Loss: 2.545 Step: 4500 Loss: 2.576 Step: 4600 Loss: 2.6 Step: 4700 Loss: 2.353 Step: 4800 Loss: 2.445 Step: 4900 Loss: 2.528 Step: 5000 Loss: 2.495 Step: 5100 Loss: 2.443 Step: 5200 Loss: 2.444 Step: 5300 Loss: 2.529 Step: 5400 Loss: 2.469 Step: 5500 Loss: 2.554 Step: 5600 Loss: 2.466 Step: 5700 Loss: 2.604 Step: 5800 Loss: 2.569 Step: 5900 Loss: 2.502 Step: 6000 Loss: 2.441 Step: 6100 Loss: 2.521 Step: 6200 Loss: 2.448 Step: 6300 Loss: 2.407 Step: 6400 Loss: 2.396 Step: 6500 Loss: 2.426 Step: 6600 Loss: 2.421 Step: 6700 Loss: 2.492 Step: 6800 Loss: 2.436 Step: 6900 Loss: 2.467 Step: 7000 Loss: 2.492 Step: 7100 Loss: 2.405 Step: 7200 Loss: 2.61 Step: 7300 Loss: 2.465 Step: 7400 Loss: 2.418 Step: 7500 Loss: 2.461 Step: 7600 Loss: 2.441 Step: 7700 Loss: 2.636 Step: 7800 Loss: 2.334 Step: 7900 Loss: 2.482 Step: 8000 Loss: 2.533 Step: 8100 Loss: 2.585 Step: 8200 Loss: 2.371 Step: 8300 Loss: 2.443 Step: 8400 Loss: 2.406 Step: 8500 Loss: 2.525 Step: 8600 Loss: 2.521 Step: 8700 Loss: 2.435 Step: 8800 Loss: 2.428 Step: 8900 Loss: 2.417 Step: 9000 Loss: 2.422 Step: 9100 Loss: 2.416 Step: 9200 Loss: 2.522 Step: 9300 Loss: 2.509 Step: 9400 Loss: 2.456 Step: 9500 Loss: 2.437 Step: 9600 Loss: 2.495 Step: 9700 Loss: 2.601 Step: 9800 Loss: 2.485 Step: 9900 Loss: 2.531 "
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "batch_size = 32\n",
    "for steps in range(10000):\n",
    "    #Sample a batch of data\n",
    "    xb, yb = get_batch('train')\n",
    "    \n",
    "    #Evaluate the loss\n",
    "    logits, loss = bigram_model(xb, yb)\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if steps % 100 == 0: print('Step:',steps,'Loss:',round(loss.item(),3), end=' ')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecf157df-2e57-45f2-901f-33f296190263",
   "metadata": {},
   "source": [
    "Ok, it looks like the loss has stopped improving. Now we'll see what the output looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83bba9b9-8258-4aec-9ffb-4d587d7f1a96",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "weangond\n",
      "OMave wap\n",
      "\n",
      "I RO:\n",
      "Banleenoalit-blt\n",
      "INRon\n",
      "\n",
      "UM: nd kngonesll;\n",
      "O: pa heore 'ga llis?-sur inidind;\n",
      "t me rthay n thavee\n",
      "Sw s serer Fofow.\n",
      "Houspathe t:\n",
      "Mind fit.\n",
      "DUKINoceamy hun.\n",
      "CKIUShorst onre t ache bar, simed?\n",
      "And me theluse BHENurind hesto f w m CK:\n",
      "YCESI fatass mbre lious ave\n",
      "Wer'dor' wod y:\n",
      "\n",
      "Henkns ges wise we me y to elil'doug p in t her spalisusin t wndalu?Y ber lishms vekeang-lumod n odas ine a! thayayor hannd t; frat.\n",
      "OLArZAUSum,\n",
      "s I f pin hondecharvyouke helldid t we ke,\n",
      "HOShe lll\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "print(decode(bigram_model.generate(torch.zeros((1,1), dtype=torch.long),max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da111902-b282-4576-a3b7-a678e39c1639",
   "metadata": {},
   "source": [
    "The results are still not great, but they're looking more reasonable than they were at first. There are now things that look like words and sentences. Keep in mind this model is predicting the next token or character solely based on the previous token so it doesn't really have a lot to go by.  Learning about the dataset, training loop and getting a basline with a simplistic model is always a good first step in any AI project. This allows you to see if further improvements are helping as well as gives you a baseline to compare your model with."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2124fbe8-c48e-4fba-92c3-1a4bd8b434db",
   "metadata": {},
   "source": [
    "Before moving on, let's do a quick loop through the validation set to see what its loss is so we can use it for comparison in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dd96e0f-7012-4d2c-a1b2-e2013a7caaac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of Validation Set:  111540 Shape of Batch: torch.Size([32, 8]) ~Steps needed to cover validation set: 435\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "print('Length of Validation Set: ',len(valid_data),'Shape of Batch:',xb.shape,'~Steps needed to cover validation set:',\n",
    "      len(valid_data)//batch_size//8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bb2d832-7152-4db7-bfa0-9f3b75ef260e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Loss: 2.441 Step: 10 Loss: 2.524 Step: 20 Loss: 2.545 Step: 30 Loss: 2.53 Step: 40 Loss: 2.397 Step: 50 Loss: 2.469 Step: 60 Loss: 2.563 Step: 70 Loss: 2.475 Step: 80 Loss: 2.621 Step: 90 Loss: 2.517 Step: 100 Loss: 2.536 Step: 110 Loss: 2.609 Step: 120 Loss: 2.492 Step: 130 Loss: 2.385 Step: 140 Loss: 2.473 Step: 150 Loss: 2.54 Step: 160 Loss: 2.53 Step: 170 Loss: 2.611 Step: 180 Loss: 2.504 Step: 190 Loss: 2.425 Step: 200 Loss: 2.454 Step: 210 Loss: 2.459 Step: 220 Loss: 2.539 Step: 230 Loss: 2.496 Step: 240 Loss: 2.392 Step: 250 Loss: 2.394 Step: 260 Loss: 2.439 Step: 270 Loss: 2.433 Step: 280 Loss: 2.509 Step: 290 Loss: 2.458 Step: 300 Loss: 2.452 Step: 310 Loss: 2.439 Step: 320 Loss: 2.593 Step: 330 Loss: 2.461 Step: 340 Loss: 2.348 Step: 350 Loss: 2.421 Step: 360 Loss: 2.455 Step: 370 Loss: 2.574 Step: 380 Loss: 2.47 Step: 390 Loss: 2.507 Step: 400 Loss: 2.404 Step: 410 Loss: 2.491 Step: 420 Loss: 2.365 Step: 430 Loss: 2.494 \n",
      "\n",
      "Final Validation Loss: tensor(2.4817)\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "batch_size = 32\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "losses = []\n",
    "for steps in range(len(valid_data)//batch_size//8):\n",
    "    #Sample a batch of data\n",
    "    xb, yb = get_batch('valid')\n",
    "    with torch.no_grad():\n",
    "    #Evaluate the loss\n",
    "        logits, loss = bigram_model(xb, yb)\n",
    "        losses.append(loss)\n",
    "    \n",
    "    if steps % 10 == 0: print('Step:',steps,'Loss:',round(loss.item(),3), end=' ')\n",
    "print('\\n\\nFinal Validation Loss:',torch.stack(losses,dim=0).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7794d914-6247-42e2-8572-3186ca44b740",
   "metadata": {},
   "source": [
    "The final validation loss appears similar to the training loss which makes sense given how basic this model is. Now we'll move on to creating a transformer and see how it does compared to the our basic bigram model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb19d2d7-8bfb-4582-b281-41d414bb7ac9",
   "metadata": {},
   "source": [
    "Clearing out variables from above to start fresh."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9d1c12c-32ab-4811-9322-5d6ab46d2b69",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reset -f"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4461437-360a-4f0b-9603-465db079baeb",
   "metadata": {},
   "source": [
    "# Code re-write in preparation for Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3111b892-9a5b-444e-93e3-e4007d0498a0",
   "metadata": {},
   "source": [
    "We're going to re-write some of our code from before to clean things up before jumping into building the transformer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80896116-66d8-40d4-85b1-ac3272bc1378",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01f5ee8a-7e65-4a23-b651-8f3c579f4293",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "#Hyperparameters\n",
    "batch_size = 32 #Number of token chunks per batch\n",
    "context_length = 8 #Length of the token chunks. Andrej called this block size\n",
    "learning_rate = 1e-2\n",
    "max_iters = 3000 #Number of training iterations or steps. Typically we specify the number of epochs but since we're randomly sampling...\n",
    "#...from the training set we don't necessarily know exactly when we've seen all the text from the training set so we'll use this instead.\n",
    "eval_interval = 300 #Number of steps between evaluating the validation set to see how our validation loss is doing.\n",
    "eval_iters = 200 #Number of steps to do on the validation set per each interval. We do more than 1 to get a more accurate overall valid loss\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu' #Instead of using the cpu, we'll use the GPU if it's availble.\n",
    "\n",
    "TORCH_SEED = 1337\n",
    "torch.manual_seed(TORCH_SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1410fe2b-8d30-4a34-a7e7-a47289ffa607",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of text: 1115394\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "with open('input.txt','r',encoding='utf-8') as f:\n",
    "    text = f.read()\n",
    "print('Length of text:',len(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7310f1f0-2018-47ee-ab0b-b4fd31b51ee6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size: 65 ,Vocab: ['\\n', ' ', '!', '$', '&', \"'\", ',', '-', '.', '3', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "vocab = sorted(list(set(text))) #Called chars in the video, but vocab is a more generic term. Both are correct.\n",
    "vocab_size = len(vocab)\n",
    "print('Vocab size:',vocab_size,',Vocab:',vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f933e0-5d29-4154-b392-44e526433b56",
   "metadata": {},
   "source": [
    "Create the token mapping functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3c8774b-9d8d-45f4-a8b2-34fc341a9082",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Character to index: {'\\n': 0, ' ': 1, '!': 2, '$': 3, '&': 4, \"'\": 5, ',': 6, '-': 7, '.': 8, '3': 9, ':': 10, ';': 11, '?': 12, 'A': 13, 'B': 14, 'C': 15, 'D': 16, 'E': 17, 'F': 18, 'G': 19, 'H': 20, 'I': 21, 'J': 22, 'K': 23, 'L': 24, 'M': 25, 'N': 26, 'O': 27, 'P': 28, 'Q': 29, 'R': 30, 'S': 31, 'T': 32, 'U': 33, 'V': 34, 'W': 35, 'X': 36, 'Y': 37, 'Z': 38, 'a': 39, 'b': 40, 'c': 41, 'd': 42, 'e': 43, 'f': 44, 'g': 45, 'h': 46, 'i': 47, 'j': 48, 'k': 49, 'l': 50, 'm': 51, 'n': 52, 'o': 53, 'p': 54, 'q': 55, 'r': 56, 's': 57, 't': 58, 'u': 59, 'v': 60, 'w': 61, 'x': 62, 'y': 63, 'z': 64}\n",
      "Index to character:, {0: '\\n', 1: ' ', 2: '!', 3: '$', 4: '&', 5: \"'\", 6: ',', 7: '-', 8: '.', 9: '3', 10: ':', 11: ';', 12: '?', 13: 'A', 14: 'B', 15: 'C', 16: 'D', 17: 'E', 18: 'F', 19: 'G', 20: 'H', 21: 'I', 22: 'J', 23: 'K', 24: 'L', 25: 'M', 26: 'N', 27: 'O', 28: 'P', 29: 'Q', 30: 'R', 31: 'S', 32: 'T', 33: 'U', 34: 'V', 35: 'W', 36: 'X', 37: 'Y', 38: 'Z', 39: 'a', 40: 'b', 41: 'c', 42: 'd', 43: 'e', 44: 'f', 45: 'g', 46: 'h', 47: 'i', 48: 'j', 49: 'k', 50: 'l', 51: 'm', 52: 'n', 53: 'o', 54: 'p', 55: 'q', 56: 'r', 57: 's', 58: 't', 59: 'u', 60: 'v', 61: 'w', 62: 'x', 63: 'y', 64: 'z'}\n",
      "Tokenization of `Hello World!`: [20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]\n",
      "String for token sequence `[20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]`: Hello World!\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "char2idx = {char:idx for idx,char in enumerate(vocab)}\n",
    "idx2char = {idx:char for char,idx in char2idx.items()}\n",
    "encode = lambda x: [char2idx[char] for char in x]\n",
    "decode = lambda idxs: ''.join([idx2char[idx] for idx in idxs])\n",
    "print('Character to index:',char2idx)\n",
    "print('Index to character:,',idx2char)\n",
    "print('Tokenization of `Hello World!`:',encode('Hello World!'))\n",
    "print('String for token sequence `[20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]`:',decode([20, 43, 50, 50, 53, 1, 35, 53, 56, 50, 42, 2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e204fe48-f29b-45bc-bf22-45adad86505d",
   "metadata": {},
   "source": [
    "Next we'll tokenize our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35df2589-af68-48d8-8947-c848d8e461ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([18, 47, 56, 57, 58,  1, 15, 47, 58, 47, 64, 43, 52, 10,  0, 14, 43, 44,\n",
       "        53, 56, 43,  1, 61, 43,  1, 54, 56, 53, 41, 43, 43, 42,  1, 39, 52, 63,\n",
       "         1, 44, 59, 56, 58, 46, 43, 56,  6,  1, 46, 43, 39, 56,  1, 51, 43,  1,\n",
       "        57, 54, 43, 39, 49,  8,  0,  0, 13, 50, 50, 10,  0, 31, 54, 43, 39, 49,\n",
       "         6,  1, 57, 54, 43, 39, 49,  8,  0,  0, 18, 47, 56, 57, 58,  1, 15, 47,\n",
       "        58, 47, 64, 43, 52, 10,  0, 37, 53, 59])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "tokenized_text = torch.tensor(encode(text),dtype=torch.long)\n",
    "tokenized_text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751aab0a-7bfb-4077-a96e-e1270342fbe2",
   "metadata": {},
   "source": [
    "And split it into training and validation sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b1a1d0e-7ff3-4b0f-b9d7-3d795b7fa170",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train test split index: 1003854\n",
      "Lenth of training data: 1003854 Length of validation data: 111540\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "#The first 90% of data will be the training set, the rest will be validation set.\n",
    "train_test_split_idx = int(len(tokenized_text) * 0.9)\n",
    "print('Train test split index:',train_test_split_idx)\n",
    "train_data = tokenized_text[:train_test_split_idx]\n",
    "valid_data = tokenized_text[train_test_split_idx:]\n",
    "print('Lenth of training data:',len(train_data),'Length of validation data:',len(valid_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bb5cd18-823c-4fa6-bfc7-56337e7214f4",
   "metadata": {},
   "source": [
    "Next we'll set up a basic data loader to get data in batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6ab5eff-95aa-47d2-b72f-f7c208cc8f23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch of x shape: torch.Size([32, 8]) Batch of y shape: torch.Size([32, 8])  dimension are [batch x context_length]\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "def get_batch(split:str, batch_size:int=batch_size, context_length:int=context_length):\n",
    "    #Function to get a batch of data from the train or valid dataset\n",
    "    data = train_data if split == 'train' else valid_data\n",
    "    idxs = torch.randint(low=0, high=len(data)-context_length, size=(batch_size,))\n",
    "    x = torch.stack([data[idx:idx+context_length] for idx in idxs])\n",
    "    y = torch.stack([data[idx+1:idx+context_length+1] for idx in idxs])\n",
    "    x,y = x.to(device), y.to(device) #Send data to the GPU if available\n",
    "    return x,y\n",
    "bx,by = get_batch('train')\n",
    "print('Batch of x shape:',bx.shape,'Batch of y shape:',by.shape,' dimension are [batch x context_length]')\n",
    "del bx, by"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5799b937-0b29-4115-aa9a-6609dbe0ba36",
   "metadata": {},
   "source": [
    "Next we'll create a function to estimate the loss for our model. Typically this is calculated against the training set for each training step and at the end of each epoch for the validation set but to keep things simple we'll just calculate it when called based on the number of steps specified as `eval_iters` and take the mean for the training and validation sets respectively. This also helps smooth out the loss values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2816659f-33d8-4b34-8810-b3af43a21e94",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "@torch.no_grad()\n",
    "def estimate_loss():\n",
    "    out = {}\n",
    "    model.eval()\n",
    "    for split in ['train','valid']:\n",
    "        losses = torch.zeros(eval_iters)\n",
    "        for k in range(eval_iters):\n",
    "            x_batch, y_batch = get_batch(split)\n",
    "            logits, loss = model(x_batch, y_batch)\n",
    "            losses[k] = loss.item()\n",
    "        out[split] = losses.mean()\n",
    "    model.train()\n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad1388ac-c82b-407b-9c9f-76576e88e35c",
   "metadata": {},
   "source": [
    "We're going to keep the model framework from the Bigram model and add in the transfomer parts shortly. For now, we just need to make sure that our updated code still works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "388001e9-8aad-44fd-9cb5-f4033b416744",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class BigramLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        #Each token reads off the logits from the subsequent token from the lookup table\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        logits = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = BigramLanguageModel(vocab_size=vocab_size)\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8c20e26-947e-4ddc-b9d7-53398d40f2d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "573f3099-b071-4182-a362-a0df4d46c4db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.7305) Validation Loss: tensor(4.7241)\n",
      "Step: 200 Training Loss: tensor(3.1193) Validation Loss: tensor(3.1313)\n",
      "Step: 400 Training Loss: tensor(2.6480) Validation Loss: tensor(2.6667)\n",
      "Step: 600 Training Loss: tensor(2.5434) Validation Loss: tensor(2.5689)\n",
      "Step: 800 Training Loss: tensor(2.5155) Validation Loss: tensor(2.5248)\n",
      "Step: 1000 Training Loss: tensor(2.5008) Validation Loss: tensor(2.5115)\n",
      "Step: 1200 Training Loss: tensor(2.4787) Validation Loss: tensor(2.4992)\n",
      "Step: 1400 Training Loss: tensor(2.4747) Validation Loss: tensor(2.4983)\n",
      "Step: 1600 Training Loss: tensor(2.4739) Validation Loss: tensor(2.4940)\n",
      "Step: 1800 Training Loss: tensor(2.4734) Validation Loss: tensor(2.4947)\n",
      "Step: 2000 Training Loss: tensor(2.4608) Validation Loss: tensor(2.4934)\n",
      "Step: 2200 Training Loss: tensor(2.4664) Validation Loss: tensor(2.4994)\n",
      "Step: 2400 Training Loss: tensor(2.4763) Validation Loss: tensor(2.4762)\n",
      "Step: 2600 Training Loss: tensor(2.4708) Validation Loss: tensor(2.4858)\n",
      "Step: 2800 Training Loss: tensor(2.4578) Validation Loss: tensor(2.4950)\n",
      "Step: 2999 Training Loss: tensor(2.4621) Validation Loss: tensor(2.4919)\n",
      "\n",
      "\n",
      "\n",
      "CEThik brid owindakis by ble\n",
      "\n",
      "HAPORDurayou e.\n",
      "S:\n",
      "O:3 my d?\n",
      "LUCous:\n",
      "Wanthar u qur, vet?\n",
      "F dXENDoate awice my.\n",
      "\n",
      "Hastacom oroup\n",
      "Yowhthetof isth ble mil ndill, ath ireeesengmin lat Heriliovets, and Win nghirileranousel lind me l.\n",
      "MAshe ce hiry:\n",
      "Supr aisspllw y.\n",
      "Hentoul n Boopetelaves\n",
      "MPOLI s, d mothakleo Windo whth eisbyo wie m dourive we higend t so mower; te\n",
      "\n",
      "AN ad nterupirf s ar iris! m:\n",
      "\n",
      "Thiny aleronth,\n",
      "Mad\n",
      "RD:\n",
      "\n",
      "WISo myrangoube!\n",
      "KENob&y arardsal thes ghesthidin cour ay aney RDUSts I&fr t ce.\n",
      "J\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc70ca4-d1e2-434d-8976-fcea9139dcb4",
   "metadata": {},
   "source": [
    "# Previous Token Averages - Building Intuition for Self Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8f2a7ff-ea2a-4d59-aab3-ddd1e10b63de",
   "metadata": {},
   "source": [
    "Attention was the key discovery that enabled the transformer architecture. The idea is that each token should be able to communicate with or look at each previous token in the sequence but not future tokens. For example given token number 4 in a sequence of 8 tokens, token 4 should be able to access token 1, 2 and 3, but not tokens 5 through 8. We will demonstrate this with a for loop implementation to cement the concept and then show the equivalent calculation using matrix multiplication which is how transformers are implemented in real life because the matrix multiplication is orders of magnitudes faster than basic nested `for` loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "389b4051-82d0-4175-a005-2b4eee58f6ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 8, 2])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "B,T,C = 4, 8, 2 #Batch, Time, Channel - Time is each token (char) in the sequence and channel is the embedding dimension\n",
    "x = torch.randn((B,T,C))\n",
    "x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18b9b8f6-46b5-41c4-947a-145a1fc6448a",
   "metadata": {},
   "source": [
    "In this example we're going to take the average of the previous tokens, just for illustration purposes, not because the average is a good way to represent the data from previous tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3f9fa43-f01b-4527-8959-f9b21f0b5649",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing if mean is being calculated correctly. Should be True, True, False: True True False\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "x_bag_of_words = torch.zeros((B,T,C))\n",
    "for batch_idx in range(B):\n",
    "    for token_idx in range(T):\n",
    "        x_previous = x[batch_idx,:token_idx+1] # (T,C)\n",
    "        x_bag_of_words[batch_idx,token_idx] = torch.mean(x_previous, dim=0)\n",
    "print('Testing if mean is being calculated correctly. Should be True, True, False:',torch.allclose(x[0,0],x_bag_of_words[0,0]), \n",
    "      torch.allclose(x[0,:2].mean(dim=0),x_bag_of_words[0,1]), torch.allclose(x[0,0],x_bag_of_words[0,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a71d71d5-ad0f-4753-8d71-7d687391ddb6",
   "metadata": {},
   "source": [
    "Each item in x_bag_of_words should be the cumulative mean of all values in x up to that index. For index 0 you can see the results are the same and for index 1 you can quickly recognize that in fact `x_bag_of_words` is the cumulative mean of x at index 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "182f798b-3e4d-4f69-9d90-5f4bb80a48be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 0.1808, -0.0700],\n",
       "         [-0.3596, -0.9152],\n",
       "         [ 0.6258,  0.0255],\n",
       "         [ 0.9545,  0.0643],\n",
       "         [ 0.3612,  1.1679],\n",
       "         [-1.3499, -0.5102],\n",
       "         [ 0.2360, -0.2398],\n",
       "         [-0.9211,  1.5433]]),\n",
       " tensor([[ 0.1808, -0.0700],\n",
       "         [-0.0894, -0.4926],\n",
       "         [ 0.1490, -0.3199],\n",
       "         [ 0.3504, -0.2238],\n",
       "         [ 0.3525,  0.0545],\n",
       "         [ 0.0688, -0.0396],\n",
       "         [ 0.0927, -0.0682],\n",
       "         [-0.0341,  0.1332]]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "x[0],x_bag_of_words[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adb2935a-fc90-4e00-86df-aa0ddfe535a2",
   "metadata": {},
   "source": [
    "Next, we'll delve into the basic operations of matrix multiplication. Specifically, we're focusing on the multiplication of two matrices, denoted as `a` and `b`. In Python, matrix multiplication is symbolized using the `@` operator. The process of matrix multiplication entails multiplying the rows of the first matrix (in this case, matrix `a`) by the columns of the second matrix (here, matrix `b`) when dealing with 2-dimensional matrices. After multiplying, the results are summed up to give the final outcome.\n",
    "\n",
    "Let's consider a practical example to illustrate this: calculating the value of `c[0,0]`. To achieve this, we need to multiply the first row of matrix `a` with the first column of matrix `b`, and then sum the results. \n",
    "\n",
    "In mathematical terms, it looks like this: `(a[0,0] * b[0,0]) + (a[0,1] * b[1,0]) + (a[0,2] * b[2,0])`. Using the example values below, the equation becomes `(1 * 1) + (2 * 3) + (3 * 5) = 22`. \n",
    "\n",
    "To enhance your understanding and intuition of matrix multiplication, I recommend the website http://matrixmultiplication.xyz/. It provides interactive visualizations of the matrix multiplication process, which can help you to understand the concept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "727e3e14-4c56-4726-a823-dbd6893f5f77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a =\n",
      " tensor([[1, 2, 3],\n",
      "        [4, 5, 6],\n",
      "        [7, 8, 9]]) \n",
      "-----\n",
      "b =\n",
      " tensor([[1, 2],\n",
      "        [3, 4],\n",
      "        [5, 6]]) \n",
      "-----\n",
      "c = a @ b =\n",
      " tensor([[ 22,  28],\n",
      "        [ 49,  64],\n",
      "        [ 76, 100]]) \n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "a = torch.tensor([[1,2,3],[4,5,6],[7,8,9]])\n",
    "b = torch.tensor([[1,2],[3,4],[5,6]])\n",
    "c = a @ b\n",
    "print('a =\\n',a,'\\n-----')\n",
    "print('b =\\n',b,'\\n-----')\n",
    "print('c = a @ b =\\n',c,'\\n-----')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8d62b96-5630-473e-acdf-d16ead1b5d29",
   "metadata": {},
   "source": [
    "The next function we need to learn about is the pytorch [`tril`](https://pytorch.org/docs/stable/generated/torch.tril.html) function. It zeros out the upper right portion of a matrix.  Let's look at a few quick examples to help visualize the concept.<br>\n",
    "\n",
    "First let's apply it to a 3x3 matrix of ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac801491-c248-469c-ba20-ae5ac913c347",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 0.],\n",
       "        [1., 1., 0.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.tril(torch.ones(3,3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e26338-f371-44c5-be2a-0633a2680ad5",
   "metadata": {},
   "source": [
    "Next we'll apply it to a 4x4 matrix of random numbers just to show this can be used with numbers other than ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbd786ca-8821-4501-af82-d608924d0f7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.8345,  0.0000,  0.0000,  0.0000],\n",
       "        [-0.4970,  0.4658,  0.0000,  0.0000],\n",
       "        [ 2.0089, -0.5370,  0.2228,  0.0000],\n",
       "        [-1.4267,  0.9059,  0.1446,  0.2280]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.tril(torch.randn((4,4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8863ae0d-bd32-4c37-b46a-57acb6cb6d66",
   "metadata": {},
   "source": [
    "Now let's see what happens when we use the tril function with our matrix multiply from before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76a4a749-6809-4c67-8fe6-2f8e0442138e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a =\n",
      " tensor([[1, 0, 0],\n",
      "        [4, 5, 0],\n",
      "        [7, 8, 9]]) \n",
      "-----\n",
      "b =\n",
      " tensor([[1, 2],\n",
      "        [3, 4],\n",
      "        [5, 6]]) \n",
      "-----\n",
      "c = a @ b =\n",
      " tensor([[  1,   2],\n",
      "        [ 19,  28],\n",
      "        [ 76, 100]]) \n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "a = torch.tril(torch.tensor([[1,2,3],[4,5,6],[7,8,9]]))\n",
    "b = torch.tensor([[1,2],[3,4],[5,6]])\n",
    "c = a @ b\n",
    "print('a =\\n',a,'\\n-----')\n",
    "print('b =\\n',b,'\\n-----')\n",
    "print('c = a @ b =\\n',c,'\\n-----')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "522a38a9-7bee-4131-a90b-c9279ccd2af4",
   "metadata": {},
   "source": [
    "Next we'll switch out our first matrix with a matrix of ones instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e90406e3-9f64-482a-8f0a-a3d8b82a0cbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a =\n",
      " tensor([[1, 0, 0],\n",
      "        [1, 1, 0],\n",
      "        [1, 1, 1]]) \n",
      "-----\n",
      "b =\n",
      " tensor([[1, 2],\n",
      "        [3, 4],\n",
      "        [5, 6]]) \n",
      "-----\n",
      "c = a @ b =\n",
      " tensor([[ 1,  2],\n",
      "        [ 4,  6],\n",
      "        [ 9, 12]]) \n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "a = torch.tril(torch.ones((3,3),dtype=torch.long))\n",
    "b = torch.tensor([[1,2],[3,4],[5,6]])\n",
    "c = a @ b\n",
    "print('a =\\n',a,'\\n-----')\n",
    "print('b =\\n',b,'\\n-----')\n",
    "print('c = a @ b =\\n',c,'\\n-----')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3b81cd9-0c42-4c74-92fe-3a2c7cde969a",
   "metadata": {},
   "source": [
    "Let's examine the first row in `c` which contains the values `[1, 2]`. Notice that it is the cumulative sum of the first row of `b` and rows 2 and 3 in `b` are effectively masked out. Row 2 in `c` is the cumulative sum of rows 1 and 2 from `b` and so on. Hopefully you can see that, like in our `for` loop, we are effectively cumulatively summing up `b`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0650bdb2-1024-4ed0-ae8e-d7b06e28fc64",
   "metadata": {},
   "source": [
    "Now we're going to reproduce our `for` loop results using matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d91f3033-9e4e-40f1-aec2-e5e56df32740",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4, 8, 2]), torch.Size([4, 8, 2]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "x.shape, x_bag_of_words.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e2b2c59-d46d-4987-b9e4-9e46b5a51c7c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of c: torch.Size([4, 8, 2])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.1808, -0.0700],\n",
       "         [-0.1789, -0.9852],\n",
       "         [ 0.4469, -0.9597],\n",
       "         [ 1.4014, -0.8953],\n",
       "         [ 1.7626,  0.2725],\n",
       "         [ 0.4127, -0.2376],\n",
       "         [ 0.6486, -0.4774],\n",
       "         [-0.2725,  1.0659]],\n",
       "\n",
       "        [[ 1.3488, -0.1396],\n",
       "         [ 1.6346,  0.8255],\n",
       "         [-0.4025,  1.3186],\n",
       "         [ 1.0845,  1.9096],\n",
       "         [ 1.2105,  0.3470],\n",
       "         [ 0.0504,  0.0121],\n",
       "         [ 0.4982, -0.7895],\n",
       "         [ 2.0218,  1.7191]],\n",
       "\n",
       "        [[-0.6631, -0.2513],\n",
       "         [ 0.3470, -0.1297],\n",
       "         [ 0.5054,  1.0043],\n",
       "         [-0.6484,  0.7059],\n",
       "         [-1.1560, -0.2180],\n",
       "         [-0.6093, -1.7128],\n",
       "         [-1.8150, -1.1410],\n",
       "         [-2.4123, -1.8347]],\n",
       "\n",
       "        [[ 1.6455, -0.8030],\n",
       "         [ 2.9969, -1.0789],\n",
       "         [ 1.4861,  1.0259],\n",
       "         [ 4.2491, -0.7207],\n",
       "         [ 5.7007, -2.2310],\n",
       "         [ 6.5218, -2.4425],\n",
       "         [ 7.3007, -0.9092],\n",
       "         [ 8.9105, -1.3124]]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "mask = torch.tril(torch.ones((1,8,8),dtype=torch.float))\n",
    "a = mask\n",
    "b = x\n",
    "c = a @ b\n",
    "print('Shape of c:',c.shape)\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf4ba50-074d-422c-b2ff-da59f017610c",
   "metadata": {},
   "source": [
    "Comparing to `x_bag_of_words`, we can see the values are different.  This is because we have calculated the cumulative sum, but not the mean. This is why at each batch's idx 0 the answer is correct as it's only summing 1 item and the sum divided by 1 is both the mean and sum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "676cf2e0-f15a-4c25-8514-b07be3166440",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.1808, -0.0700],\n",
       "         [-0.0894, -0.4926],\n",
       "         [ 0.1490, -0.3199],\n",
       "         [ 0.3504, -0.2238],\n",
       "         [ 0.3525,  0.0545],\n",
       "         [ 0.0688, -0.0396],\n",
       "         [ 0.0927, -0.0682],\n",
       "         [-0.0341,  0.1332]],\n",
       "\n",
       "        [[ 1.3488, -0.1396],\n",
       "         [ 0.8173,  0.4127],\n",
       "         [-0.1342,  0.4395],\n",
       "         [ 0.2711,  0.4774],\n",
       "         [ 0.2421,  0.0694],\n",
       "         [ 0.0084,  0.0020],\n",
       "         [ 0.0712, -0.1128],\n",
       "         [ 0.2527,  0.2149]],\n",
       "\n",
       "        [[-0.6631, -0.2513],\n",
       "         [ 0.1735, -0.0649],\n",
       "         [ 0.1685,  0.3348],\n",
       "         [-0.1621,  0.1765],\n",
       "         [-0.2312, -0.0436],\n",
       "         [-0.1015, -0.2855],\n",
       "         [-0.2593, -0.1630],\n",
       "         [-0.3015, -0.2293]],\n",
       "\n",
       "        [[ 1.6455, -0.8030],\n",
       "         [ 1.4985, -0.5395],\n",
       "         [ 0.4954,  0.3420],\n",
       "         [ 1.0623, -0.1802],\n",
       "         [ 1.1401, -0.4462],\n",
       "         [ 1.0870, -0.4071],\n",
       "         [ 1.0430, -0.1299],\n",
       "         [ 1.1138, -0.1641]]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "x_bag_of_words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e727fca-74e2-4129-b3d8-023ab9a3d4c4",
   "metadata": {},
   "source": [
    "Now we'll divide by the number of items that were summed to get the mean. Just like before we're going to use a matrix operation to keep things fast. First we'll create the matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b4384a5-9b15-4bbd-a49f-b3529c5a5691",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 1],\n",
       "        [2, 2],\n",
       "        [3, 3],\n",
       "        [4, 4],\n",
       "        [5, 5],\n",
       "        [6, 6],\n",
       "        [7, 7],\n",
       "        [8, 8]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "num_toks_summed = torch.tensor(list(zip(list(range(1,9)),list(range(1,9)))))\n",
    "num_toks_summed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bda8e91-a062-4302-9ede-f6c1345904c4",
   "metadata": {},
   "source": [
    "And now we'll divide our cumulative sum by the number of items summed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d996719-80a8-454f-83aa-e6853801d1d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.1808, -0.0700],\n",
       "         [-0.0894, -0.4926],\n",
       "         [ 0.1490, -0.3199],\n",
       "         [ 0.3504, -0.2238],\n",
       "         [ 0.3525,  0.0545],\n",
       "         [ 0.0688, -0.0396],\n",
       "         [ 0.0927, -0.0682],\n",
       "         [-0.0341,  0.1332]],\n",
       "\n",
       "        [[ 1.3488, -0.1396],\n",
       "         [ 0.8173,  0.4127],\n",
       "         [-0.1342,  0.4395],\n",
       "         [ 0.2711,  0.4774],\n",
       "         [ 0.2421,  0.0694],\n",
       "         [ 0.0084,  0.0020],\n",
       "         [ 0.0712, -0.1128],\n",
       "         [ 0.2527,  0.2149]],\n",
       "\n",
       "        [[-0.6631, -0.2513],\n",
       "         [ 0.1735, -0.0649],\n",
       "         [ 0.1685,  0.3348],\n",
       "         [-0.1621,  0.1765],\n",
       "         [-0.2312, -0.0436],\n",
       "         [-0.1015, -0.2855],\n",
       "         [-0.2593, -0.1630],\n",
       "         [-0.3015, -0.2293]],\n",
       "\n",
       "        [[ 1.6455, -0.8030],\n",
       "         [ 1.4985, -0.5395],\n",
       "         [ 0.4954,  0.3420],\n",
       "         [ 1.0623, -0.1802],\n",
       "         [ 1.1401, -0.4462],\n",
       "         [ 1.0870, -0.4071],\n",
       "         [ 1.0430, -0.1299],\n",
       "         [ 1.1138, -0.1641]]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "c2 = c/num_toks_summed\n",
    "c2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1b458b0-1445-4e04-a4f0-fd7e3ad927c3",
   "metadata": {},
   "source": [
    "And check whether it matches our previously calculated bow, which it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a674c9d-62d2-4f57-b5d9-a39177cfb111",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.allclose(x_bag_of_words, c2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0a0cea6-fbca-46c3-8a03-7f11e6d0e60e",
   "metadata": {},
   "source": [
    "As an aside, let's look at how long the x_bag_of_words took to calculate via a for loop vs our matrix version. As the number of calculations is tiny in this case, we're going to increase the complexity so you can more easily see the time differences.<br>\n",
    "First we'll do the for loop version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b063f856-b8fd-4281-b289-70a3d158f8fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "# B,T,C = 4, 8, 2 #Batch, Time, Channel - Time is each token (char) in the sequence and channel is the embedding dimension\n",
    "B,T,C = 256, 512, 64\n",
    "x = torch.randn((B,T,C))\n",
    "x.shape\n",
    "x_bag_of_words = torch.zeros((B,T,C))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adedaa34-fd1a-482d-9b6b-7189337b3c45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing if mean is being calculated correctly. Should be True, True, False: True True False\n",
      "Testing if mean is being calculated correctly. Should be True, True, False: True True False\n",
      "Testing if mean is being calculated correctly. Should be True, True, False: True True False\n",
      "Testing if mean is being calculated correctly. Should be True, True, False: True True False\n",
      "4.84 s ± 22.3 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 2 -r 2\n",
    "#| output: true\n",
    "for batch_idx in range(B):\n",
    "    for token_idx in range(T):\n",
    "        x_previous = x[batch_idx,:token_idx+1] # (T,C)\n",
    "        x_bag_of_words[batch_idx,token_idx] = torch.mean(x_previous, dim=0)\n",
    "print('Testing if mean is being calculated correctly. Should be True, True, False:',torch.allclose(x[0,0],x_bag_of_words[0,0]), \n",
    "      torch.allclose(x[0,:2].mean(dim=0),x_bag_of_words[0,1]), torch.allclose(x[0,0],x_bag_of_words[0,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9533c21a-4c72-46e9-85ad-ee2beb36238b",
   "metadata": {},
   "source": [
    "The for loop version on my PC took 4.3 seconds: `4.33 s ± 24.2 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)`. <br>\n",
    "Now we'll test the matrix version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "450fc0ac-8ff8-44c4-909a-3b55b7d01a80",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "mask = torch.tril(torch.ones((1,T,T),dtype=torch.float))\n",
    "num_toks_summed = torch.Tensor([[i+1]*C for i in range(T)])\n",
    "a = mask\n",
    "b = x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "912c7bd9-4925-4d68-9796-e2a780ee115f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23.6 ms ± 1.17 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 2 -r 2 \n",
    "#| output: true\n",
    "c = a @ b / num_toks_summed\n",
    "# print('Shape of c:',c.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a52d2317-e955-4c5c-9276-0ed623a5540c",
   "metadata": {},
   "source": [
    "The matrix multiplication version on the other hand took 20 ms. `19.8 ms ± 872 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)`<br>\n",
    "Now we'll confirm the results are the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6703f9fc-e318-4d3f-a297-5c8b90dc143e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "c = a @ b / num_toks_summed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aafe1fb-052b-4b0c-8d33-1af529c7cf73",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.allclose(x_bag_of_words, c, atol=1e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "280dbcf6-d242-41d3-995c-c4a5390fa9c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "218.68686868686868"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "4.33/(19.8/1000) # 4.33 seconds / 19.8 ms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "418632d4-ed30-43ae-b314-6bb13c1f644a",
   "metadata": {},
   "source": [
    "As you can see the results were the same but the matrix multiplication version finished in ~20ms vs ~4.3s (numbers may vary between runs and machines) for the for loop version which is ~220x faster and that is running the matmul on the cpu instead of gpu which would speed it up even further."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e99510a-a7e7-48ea-a4a9-ec28b0cd47ea",
   "metadata": {},
   "source": [
    "Alternatively we can use Andrej's method which should be even more efficient because we're dividing the mask elements by the number of elements to be summed which is effectively moving the division by `num_toks_summed` out of our final timed calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47d4fb64-73a4-439f-93bf-024150270bf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "mask = torch.tril(torch.ones((1,T,T),dtype=torch.float))\n",
    "mask = mask / mask.sum(dim=2,keepdim=True)\n",
    "a = mask\n",
    "b = x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f2dd1f2-390c-41aa-8049-7a477c9aacbd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18.3 ms ± 289 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 2 -r 2 \n",
    "#| output: true\n",
    "c = a @ b\n",
    "# print('Shape of c:',c.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "733151f6-1c87-4687-9cb7-b944bcf1aaa7",
   "metadata": {},
   "source": [
    "`20.7 ms ± 1.21 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)` In this case the numbers are very close. This could be influenced by a number of factors, but in general it's best to try and compute things once vs for each iteration if possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6308566-1e99-4384-8375-44c3e0258150",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "c = a @ b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd434847-dfd1-468a-8fff-189e3cc63033",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.allclose(x_bag_of_words, c, atol=1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c652d9f-e151-4b90-bbcb-ed2c0939ff91",
   "metadata": {},
   "source": [
    "Now let's look at the 3rd version of calculating this using softmax. It should produce an identical result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ee265e6-3ef5-4a99-b38a-836cdaa4fbb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1., 0., 0.,  ..., 0., 0., 0.],\n",
       "         [1., 1., 0.,  ..., 0., 0., 0.],\n",
       "         [1., 1., 1.,  ..., 0., 0., 0.],\n",
       "         ...,\n",
       "         [1., 1., 1.,  ..., 1., 0., 0.],\n",
       "         [1., 1., 1.,  ..., 1., 1., 0.],\n",
       "         [1., 1., 1.,  ..., 1., 1., 1.]]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "mask = torch.tril(torch.ones((1,T,T),dtype=torch.float))\n",
    "mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b317fc73-d3ad-40cd-bab7-fa91aa0c9df9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[0., -inf, -inf,  ..., -inf, -inf, -inf],\n",
       "         [0., 0., -inf,  ..., -inf, -inf, -inf],\n",
       "         [0., 0., 0.,  ..., -inf, -inf, -inf],\n",
       "         ...,\n",
       "         [0., 0., 0.,  ..., 0., -inf, -inf],\n",
       "         [0., 0., 0.,  ..., 0., 0., -inf],\n",
       "         [0., 0., 0.,  ..., 0., 0., 0.]]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "mask2 = torch.zeros_like(mask).masked_fill(mask==0,float('-inf'))\n",
    "mask2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ea84802-1b15-4e0e-ab20-3e36b86323e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "         [0.5000, 0.5000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "         [0.3333, 0.3333, 0.3333,  ..., 0.0000, 0.0000, 0.0000],\n",
       "         ...,\n",
       "         [0.0020, 0.0020, 0.0020,  ..., 0.0020, 0.0000, 0.0000],\n",
       "         [0.0020, 0.0020, 0.0020,  ..., 0.0020, 0.0020, 0.0000],\n",
       "         [0.0020, 0.0020, 0.0020,  ..., 0.0020, 0.0020, 0.0020]]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "mask3 = F.softmax(mask2,dim=2)\n",
    "mask3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fc219bc-7cb1-4e1e-94ce-345221fbddfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "a = mask3\n",
    "b = x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0781a644-6444-4246-ab5f-db220112f1ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18.8 ms ± 798 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 2 -r 2\n",
    "#| output: true\n",
    "c = a @ b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70a5757f-4656-4ee4-9b62-fc778fc9c876",
   "metadata": {},
   "source": [
    "`20.5 ms ± 1.97 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "990995f8-9a46-47a9-8c8d-14a4784073db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.allclose(x_bag_of_words, c, atol=1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e426f6-1e01-405f-a43b-79e01c5ffaf6",
   "metadata": {},
   "source": [
    "Softmax is used to create probabilities that previous tokens will interact with the current token. Intuitvely you can think of this as not all previous tokens in a sequence carry equal weight or importance so this allows the model to assign a weigting to all previous tokens. All probabilities output from softmax add up to 1. `-inf` is used to mask out future tokens which should not be accesible to the current token. Running `-inf` through softmax yields a 0 probability. In the transformer instead of the `mask` values being either identical or 0 (ex: `[0.5, 0.5, 0, 0]`) the weighting of past tokens will be learned. i.e. certain tokens can communicate with certain other tokens with a stronger weight based on learned values from the data. Andrej calls this `wei` for 'weights', which is the weighting used to determine which tokens communicate with eachother. This is different that what we're doing now which is forcing all non-0 weights to be the same. What we're doing now is just building up the math and intuition on how attention works."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a3aa647-b5a6-4942-ab24-f5fae0571261",
   "metadata": {},
   "source": [
    "# Self attention"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc403cd4-a125-41a9-8426-96c12fffc705",
   "metadata": {},
   "source": [
    "## Initial Code Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fca4c134-1a0b-4270-bddc-091aff4510e7",
   "metadata": {},
   "source": [
    "To start with we're going to modify our `BigramLanguageModel` to be a `TransformerLanguageModel` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c47c79e-3bf8-490a-9d1a-1ea16808bec7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "embedding_dim = 32 #The vector size of the token embeddings. Andrej used n_embed as the variable name."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b58fe169-42b3-4fb2-8c96-584f06d3a30c",
   "metadata": {},
   "source": [
    "We're going to add an embedding dimension, change up the token embedding table and modify the token embedding lookup and logits calculation as we work towards modifying this class into a true transformer. We'll iteratively test as we go to make sure it is still able to train correctly. Please read through the below code taking note of the comments explaining the changes being made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d937b9b3-1f00-42bc-8e4f-463c5887fedf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.3278) Validation Loss: tensor(4.3231)\n",
      "Step: 200 Training Loss: tensor(2.5421) Validation Loss: tensor(2.5626)\n",
      "Step: 400 Training Loss: tensor(2.4982) Validation Loss: tensor(2.5163)\n",
      "Step: 600 Training Loss: tensor(2.4936) Validation Loss: tensor(2.5354)\n",
      "Step: 800 Training Loss: tensor(2.4983) Validation Loss: tensor(2.5067)\n",
      "Step: 1000 Training Loss: tensor(2.5025) Validation Loss: tensor(2.5045)\n",
      "Step: 1200 Training Loss: tensor(2.4831) Validation Loss: tensor(2.5028)\n",
      "Step: 1400 Training Loss: tensor(2.4866) Validation Loss: tensor(2.5157)\n",
      "Step: 1600 Training Loss: tensor(2.4927) Validation Loss: tensor(2.5120)\n",
      "Step: 1800 Training Loss: tensor(2.4899) Validation Loss: tensor(2.5120)\n",
      "Step: 2000 Training Loss: tensor(2.4804) Validation Loss: tensor(2.5071)\n",
      "Step: 2200 Training Loss: tensor(2.4841) Validation Loss: tensor(2.5178)\n",
      "Step: 2400 Training Loss: tensor(2.4940) Validation Loss: tensor(2.4883)\n",
      "Step: 2600 Training Loss: tensor(2.4956) Validation Loss: tensor(2.5065)\n",
      "Step: 2800 Training Loss: tensor(2.4799) Validation Loss: tensor(2.5138)\n",
      "Step: 2999 Training Loss: tensor(2.4870) Validation Loss: tensor(2.5165)\n",
      "\n",
      "\n",
      "\n",
      "CExthy bridcowindakis s, bth\n",
      "\n",
      "HAPORDurayoule.\n",
      "S:\n",
      "O:\n",
      "IS:\n",
      "Thachangs ar bthar usqur, vethar dilasoate arche my.\n",
      "\n",
      "HD:\n",
      "\n",
      "Yom o mur\n",
      "Yowhthetof isth bot mil ndill, bes ireeesenghin lat Heridrovets, and Win nghir.\n",
      "Thabousel lind me l.\n",
      "HAser ce wiry ptupr aisspllwhy.\n",
      "HAntoul noroopetelaves\n",
      "MPOLI swod mothakleo Windo whth eiiby we ath dourive wee, ired t so mo she te\n",
      "\n",
      "AN ad nterurt f sor; irist m:\n",
      "\n",
      "Thiny aleronth, af Pre?\n",
      "\n",
      "WISo myay INouk!\n",
      "KENoby sarardsal thes ghesthinin cour ay aney RDUES:\n",
      "I fr t ce.\n",
      "J\n",
      "CPU times: user 17 s, sys: 438 ms, total: 17.4 s\n",
      "Wall time: 17.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size=vocab_size, embedding_dim=embedding_dim):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        #A basic linear layer to pass our token embeddings through. This is a preliminary step, not the final network.\n",
    "        #Note the input size in the embedding_dim and the output size is the number of tokens or vocab size.\n",
    "        #This is because we are going to be predicting the probability for every token in the vocab that it is the next token.\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.vocab_size)\n",
    "\n",
    "        #This will be our lookup table for all of the token embeddings. We'll have an entry for each token (aka vocab size)... \n",
    "        #...and each embedding will be a vector of dimension embedding_dim.\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel - Lookup token embeddings\n",
    "        logits = self.language_model_head_linear_layer(token_embeddings) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx) #This is calling `forward`\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are the token indicies (int).\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx #TODO: Stopped Here        \n",
    "\n",
    "model = TransformerLanguageModel(vocab_size=vocab_size, embedding_dim=embedding_dim)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a09eb01f-7d44-46b6-88d5-e095d066ac12",
   "metadata": {},
   "source": [
    "We need to also encode the token position so we'll need to add another embedding table for that which will be learned as well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc94a5ee-b442-47e3-bcc4-a0df530abfdf",
   "metadata": {},
   "source": [
    "## Building Up To Self Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59ff4905-8685-4799-bfdd-8a6684d6d85b",
   "metadata": {},
   "source": [
    "We'll go through the simple cumulative token average again using matrix multiplication and modify it over time to be self attention. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab7a5450-34af-4e4b-b73e-ad72c87eefdb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch Size (B): 32 Context Length (T): 8 Embedding Dimension (C): 32\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[1, 0, 0, 0, 0, 0, 0, 0],\n",
       "        [1, 1, 0, 0, 0, 0, 0, 0],\n",
       "        [1, 1, 1, 0, 0, 0, 0, 0],\n",
       "        [1, 1, 1, 1, 0, 0, 0, 0],\n",
       "        [1, 1, 1, 1, 1, 0, 0, 0],\n",
       "        [1, 1, 1, 1, 1, 1, 0, 0],\n",
       "        [1, 1, 1, 1, 1, 1, 1, 0],\n",
       "        [1, 1, 1, 1, 1, 1, 1, 1]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "B,T,C = batch_size, context_length, embedding_dim\n",
    "x = torch.randn((B,T,C))\n",
    "print('Batch Size (B):',B,'Context Length (T):',T,'Embedding Dimension (C):',C)\n",
    "tril = torch.tril(torch.ones(T,T,dtype=torch.long))\n",
    "tril"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e80f9a44-fc6a-450a-8120-66ca7f921588",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.5000, 0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.3333, 0.3333, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.2000, 0.2000, 0.2000, 0.2000, 0.2000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.0000, 0.0000],\n",
       "        [0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.0000],\n",
       "        [0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "weights = torch.zeros(T,T,dtype=torch.float)\n",
    "weights = weights.masked_fill(tril == 0, float('-inf'))\n",
    "weights = torch.softmax(weights, dim=-1)\n",
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "731ec026-5276-4300-98c6-c54080402572",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 8, 32])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "out = weights @ x\n",
    "out.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec40774a-d603-4310-bd3f-411f4a2f17e7",
   "metadata": {},
   "source": [
    "When using the cumulative mean the weights are fixed, but ideally we want the weights to variable and learned so each token can interact with other tokens a varying amount based on learned paramters of what is most important. Some tokens will find other tokens more interesting than others and we want that to be data dependent and learned through training.<br>\n",
    "\n",
    "[The example Andrej gives](https://youtu.be/kCc8FmEb1nY?t=3820) is \"maybe if I'm a vowel token I am more interested in past consonant tokens and I want the [consonant] data to flow to me, this is the problem that self attention solves\". The way that self attention solves this is that every single node or token will emit 2 vectors, a query and key vector. The query vector roughly represents \"what am I looking for\" and the key vector roughly represents \"what do I contain\". The way we then get the affinities between each token is by performing a dot product (matrix multiplication) of all the query vectors against all of the key vectors. So for a given query vector, the dot product is calculated against all of the key vectors and the results of that become the weights that are used for each token. This is the `weight` variable we used above except now instead of being a fixed average, it varies per token and is learned. If the key and query are aligned they will produce a larger value when the dot product is taken between them which leads to a larger value in the `weight` matrix. \n",
    "\n",
    "<br>Let's take the above example and modify it to implement self attention.\n",
    "\n",
    "<br>First we need to define our head size. We will use 16. This will be the side dimension of a matrix where each query and key vector are matrix multiplied together. To get the query and key vectors from the token embeddings we first need to run the token embedding through a linear layer for the query and key which will generate a vector of size `head_size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e97af1f9-48f6-4b3d-aecc-10639e031f4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch Size (B): 32 Context Length (T): 8 Embedding Dimension (C): 32\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "#version 4: self attention\n",
    "#setup\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "B,T,C = batch_size, context_length, embedding_dim\n",
    "x = torch.randn((B,T,C))\n",
    "print('Batch Size (B):',B,'Context Length (T):',T,'Embedding Dimension (C):',C)\n",
    "\n",
    "head_size = 16 #Self attention head size\n",
    "\n",
    "#Learned vector to extract key vector from token embedding vector\n",
    "key_layer = nn.Linear(in_features=C, out_features=head_size, bias=False) \n",
    "\n",
    "#Learned vector to extract query vector from token embedding vector\n",
    "query_layer = nn.Linear(in_features=C, out_features=head_size, bias=False) \n",
    "\n",
    "#Extract query and key values for every token in the batch in parallel\n",
    "key = key_layer(x) # (B,T,head_size)\n",
    "query = query_layer(x) # (B,T,head_size) #TODO:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69fac63c-c2d7-4222-ab34-2ab6ff625805",
   "metadata": {},
   "source": [
    "Now we will calculate the affinities (weights) between each token in each sequence by matrix multiplying all of the queries and keys. If we simply try to calculate `query @ key` it will fail because the shapes are not correct to be able to do matrix multiplication. In our case both `key` and `query` are of shape (B,T,head_size) which are incompatible shapes to be matrix multiplied together. We need to transpose, or rotate, the `key` in the T and head_size dimension so they can be matrix multiplied. We cannot simply use the `.T` transpose because it would transpose in the batch dimension as well which we do not want so instead we'll specify which dimensions to transpose which we can do by calling `key.transpose(-2, -1)` which will transpose the last 2 dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8f04c8b-6e18-4a16-85f4-a46647d0ebe8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 8, 8])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "#Calculate affinity (weights)\n",
    "#(B,T,head_size) @ (B, head_size, T) which is (32,8,16) @ (32,16,8) -> (B, T, T) which is (32,8,8)\n",
    "weights = query @ key.transpose(-2, -1) \n",
    "weights.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf8e62c9-7cc6-4a28-8d3c-dbb065c535d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.2746, -1.2489,  0.5325,  0.2690,  0.1749, -0.7806,  0.1727,  1.3460],\n",
       "        [ 1.1833,  1.3634,  0.8982, -0.2749, -0.6459, -0.9106, -0.1111, -1.6019],\n",
       "        [-0.5018, -0.0807, -1.0500, -0.7615,  0.3193, -0.1342,  0.3969, -1.8405],\n",
       "        [ 0.5840,  0.6687, -0.5924,  0.4017,  0.3058, -0.6051, -0.0642, -0.6784],\n",
       "        [-0.2627, -2.5447,  1.5607, -1.5866, -0.6412, -1.3504, -0.0761, -2.3587],\n",
       "        [-0.8941, -2.0453, -1.5232,  1.5689, -0.5483,  1.9307, -0.6665, -1.7059],\n",
       "        [-0.2565,  0.5645, -0.7255, -0.2768, -0.3469,  0.7811, -0.5242, -0.9621],\n",
       "        [ 0.7061,  1.1336,  0.4749,  0.7493,  0.5897,  1.3509,  0.7673, -0.4065]],\n",
       "       grad_fn=<SelectBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "weights[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec8bc727-74cb-4975-aee9-3ae8cf91f229",
   "metadata": {},
   "source": [
    "Now we have weights that are calculated based on each token's affinity to every other token. We then apply the same filtering that we did previously with our cumulative mean so we simply remove the line where the `weights` were set to zero. This will allow us to finally apply a learned weighting to each previous token embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b59b54b9-bc45-4742-b215-589cf87dbe4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 8, 32])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "tril = torch.tril(torch.ones(T,T,dtype=torch.long))\n",
    "\n",
    "# weights = torch.zeros(T,T,dtype=torch.float) #Removed now that weights are calculated\n",
    "weights = weights.masked_fill(tril == 0, float('-inf')) #Masks future tokens\n",
    "weights = torch.softmax(weights, dim=-1) #Provides even distribution (weights that add up to 1)\n",
    "\n",
    "out = weights @ x\n",
    "out.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4257aed3-6426-4275-a049-44a92285f5b4",
   "metadata": {},
   "source": [
    "You can see the weights below. Notice they are no longer uniform. They can now be indivual and learned from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d1514d9-f798-4559-8df6-7d850d34dcc9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.4551, 0.5449, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.3224, 0.4912, 0.1864, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.3096, 0.3370, 0.0955, 0.2580, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.1213, 0.0124, 0.7510, 0.0323, 0.0831, 0.0000, 0.0000, 0.0000],\n",
       "        [0.0314, 0.0099, 0.0167, 0.3685, 0.0444, 0.5291, 0.0000, 0.0000],\n",
       "        [0.1066, 0.2423, 0.0667, 0.1045, 0.0974, 0.3009, 0.0816, 0.0000],\n",
       "        [0.1168, 0.1792, 0.0927, 0.1220, 0.1040, 0.2227, 0.1242, 0.0384]],\n",
       "       grad_fn=<SelectBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "weights[0] #TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e607a5-a590-43a0-937e-6ec047921b6e",
   "metadata": {},
   "source": [
    "Now each token will be able to calculate its affinity to all other tokens. You can see in the example by looking at the bottom row, that the 8th token has a high affinity for the 6th token because it has the highest value: <br>\n",
    "\n",
    "`[0.1168, 0.1792, 0.0927, 0.1220, 0.1040, `**`0.2227`**`, 0.1242, 0.0384]`<br>\n",
    "\n",
    "There is [one more part](https://youtu.be/kCc8FmEb1nY?t=4207) of self attention we need to look at.  That is that when we aggregate the tokens `out = weights @ x` we don't aggregate the tokens exactly, we aggregate the `value`, so in the same way that we calculate `key` and `query` via passing the token embedding through a linear layer, we will do the same to get the `value`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb638b63-5188-45ac-a139-1d94d4365e24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch Size (B): 32 Context Length (T): 8 Embedding Dimension (C): 32\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 8, 8])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "#version 4: self attention\n",
    "#setup\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "B,T,C = batch_size, context_length, embedding_dim\n",
    "x = torch.randn((B,T,C))\n",
    "print('Batch Size (B):',B,'Context Length (T):',T,'Embedding Dimension (C):',C)\n",
    "\n",
    "head_size = 16 #Self attention head size\n",
    "\n",
    "#Learned vector to extract key vector from token embedding vector\n",
    "key_layer = nn.Linear(in_features=C, out_features=head_size, bias=False) \n",
    "\n",
    "#Learned vector to extract query vector from token embedding vector\n",
    "query_layer = nn.Linear(in_features=C, out_features=head_size, bias=False) \n",
    "\n",
    "#Learned vector to extract value vector from token embedding vector\n",
    "value_layer = nn.Linear(in_features=C, out_features=head_size, bias=False) #NEW\n",
    "\n",
    "#Extract query, key and value for every token in the batch in parallel\n",
    "key = key_layer(x) # (B,T,head_size)\n",
    "query = query_layer(x) # (B,T,head_size)\n",
    "value = value_layer(x) # (B,T,head_size) #NEW\n",
    "\n",
    "#Calculate affinity (weights)\n",
    "#(B,T,head_size) @ (B, head_size, T) which is (32,8,16) @ (32,16,8) -> (B, T, T) which is (32,8,8)\n",
    "weights = query @ key.transpose(-2, -1) \n",
    "weights.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f66591c5-2947-4a74-bda3-78f0156aec69",
   "metadata": {},
   "source": [
    "And now instead of calculating the output by matrix multiplying the `weights` by `x` we multiply the weights by `value`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b262d729-d4c9-4817-880e-cd0dce75f1ba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([32, 8, 16])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "tril = torch.tril(torch.ones(T,T,dtype=torch.long))\n",
    "\n",
    "weights = weights.masked_fill(tril == 0, float('-inf')) #Masks future tokens\n",
    "weights = torch.softmax(weights, dim=-1) #Provides even distribution (weights that add up to 1)\n",
    "\n",
    "# out = weights @ x\n",
    "out = weights @ value #NEW (B, T, T) @ (B, T, head_size) = (B, T, head_size)\n",
    "out.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29543c03-11fa-4762-9ecc-6d15a7f31814",
   "metadata": {},
   "source": [
    "Notice how the shape of `out` has changed from `torch.Size([32, 8, 32])` to `torch.Size([32, 8, 16])` now that the we are using `value` which is of length 16 instead of the token embedding `x` which was of length 32.<br>\n",
    "\n",
    "You can think of the token embedding `x` as private information of the token and it must be passed through the linear layers to get the `query`, `key` and `value`. You can think of it as the token embedding `x` has all the information about the token and:<br>\n",
    "\n",
    "`query`: represents the things that the token is interested in or wants.<br>\n",
    "\n",
    "`key`: represents the things the token has.<br>\n",
    "\n",
    "`value`: represents, if you find the token interesting, the information the token wants to communicate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10cabb4-1c02-4d0b-9499-13a9265ee2f6",
   "metadata": {},
   "source": [
    "**Additional Notes on Attention:** [link](https://youtu.be/kCc8FmEb1nY?t=4298)<br>\n",
    "1) Attention is a communication mechanism. You can think of it as if you had nodes in a directed graph:<br>\n",
    "\n",
    "![Image of a Directed Graph](./directed_graph.png)<br>\n",
    "\n",
    "Each node has a vector of information (token embedding) and it gets a weighted sum of all of the nodes that point to it. This is done in a data dependent manner, so it depends on what data is stored in each node at any point in time. Our graph does not look quite like the example.  Instead, our graph has 8 nodes. The first node is pointed to by only itself, our second node is pointed to by the first node and itself, our third node is pointed to by our first and second nodes as well as itself and so on. This structure is common in auto-regressive scenarious. <br>\n",
    "\n",
    "**Auto-regressive** in this context refers to a type of model that generates sequences by modeling the probability of each item in the sequence given its preceding items. In other words, autoregressive language models generate predictions step by step, where each prediction is dependent on the ones that came before it.<br>\n",
    "\n",
    "In principal attention can be applied to any arbitrary directed graph as it is just a communication mechanism between nodes.<br>\n",
    "\n",
    "2) There is no notion of space or position. Attention simply acts over a set of vectors in this graph. The nodes have no idea of where they are positioned within the space which is why we need to encode them positionally which gives them information that anchors them to a specific position. i.e. inherently the nodes, representing characters in our example, don't know what position they occur in relative to the other nodes which is why we need to positionally encode them. You can contrast this with convolutional neural networks where the data and network inherently are modeled spatially. For example CNN's are regularly used for computer vision applications. In these applications adjacent pixels are fed into the CNN where convolutional filters act in space preserving the spatial information about the data.<br>\n",
    "\n",
    "Attention, in contrast with CNN's, has no notion of space, so space or position or location need to be encoded into the nodes through some other mechanism, which in our case is a positional encoding vector. This position vector will be added to the token prior to it being processed through the linear layers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21083239-0f46-4a00-8cd7-9c8d097652e4",
   "metadata": {},
   "source": [
    "**Additional Notes:** [link](https://youtu.be/kCc8FmEb1nY?t=4420)<br>\n",
    "* Each example across `batch` dimensions is processed completely independently. Information from an item in a batch does not affect information in another item within the batch. Different items within a batch never talk to eachother.\n",
    "* In an encoder network (block), you do not filter out future tokens, only in a decoder network. This means that in an encoder network, these lines from our previous example would be removed:\n",
    "```python\n",
    "tril = torch.tril(torch.ones(T,T,dtype=torch.long)) \n",
    "weights = weights.masked_fill(tril == 0, float('-inf'))\n",
    "```\n",
    "> There are many instances where you want all of the nodes to talk to each other, such as in sentiment analysis for example, because later on in the network you are making a simple prediction on whether the text is positive or negative. Another example would be vision transformers where you want all image patches to talk to each other. In these instances you use an encoder block which does not have masking in contrast to the decoder block which is what we have been focusing on here.\n",
    "* There are different types of attention. What we're looking at now is self attention. The reason this is self-attention is because the data comes from the same source (`x`). Attention can be much more general than self attention, in that the source of the data can be from a different source. For example in encoder decoder networks, the queries could be produced from `x` but the the keys and values could come from a completely different source, for example from different encoder blocks that we would want to condition on. A real world example of this could be translating from one language to another, where the original or input language comes from an separate encoder network. The encoder network provides the `keys` and `values` and the decoder network provides the `queries`. This is called cross attention and is where there is a separate set of nodes we would like to pull information from into our node. Self attention, again, is where we are pulling `keys`, `queries` and `values` from one set of nodes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bed003b-b49f-4afd-a887-6b7381988752",
   "metadata": {},
   "source": [
    "So far we've implemented most of the attenion equation from the original [Attention is all you need paper](https://arxiv.org/abs/1706.03762).\n",
    "![Attention Equation from Attention is all you need paper](./attention_equation1.png)<br>\n",
    "\n",
    "`Attention(Q,K,V) = softmax((Q*K^T)/(sqrt(dk))*V`<br>\n",
    "Where: Q = Query, K = Key, V = Value, dk = dimension of the Q and K or 'head'.<br>\n",
    "\n",
    "The piece we are missing is dividing by `sqrt(dk)` which makes this **'scaled self attention'**. To do this we need to divide `weights` by `sqrt(dk)` or the dimension of the Q,K head. This makes it so when Q,K are unit variance, `weights` will be unit variance too which is important so `softmax` will remain diffused and not be saturated too much, i.e. the dot products betweek Q and K can become very large which pushes the gradients through the softmax to become very small which negatively impact training. This is why we want to scale them first before taking the `softmax`.<br>\n",
    "\n",
    "Let's look at a real example of this:<br>\n",
    "\n",
    "Where `q` and `k` are a gausian or normal distributions so the mean of the values is 0 and the standard deviation is 1. When you compute the matrix multiplication between them you will notice that the variance of `weights` is quite high."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6804aeec-eec9-4633-ac4f-62c0ee990e0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean of q: tensor(0.0021) Variance of q: tensor(0.9985) Mean of k: tensor(-0.0129) Variance of k: tensor(1.0255)\n",
      "Shape of weights: torch.Size([32, 8, 8]) Mean of weights: tensor(-0.0302) Variance of weights: tensor(17.3386) \n",
      "Min of weights: tensor(-16.3490) Max of weights: tensor(13.1295)\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "q = torch.randn(B,T,head_size)\n",
    "k = torch.randn(B,T,head_size)\n",
    "print('Mean of q:',q.mean(),'Variance of q:',q.var(),'Mean of k:',k.mean(),'Variance of k:',k.var())\n",
    "weights = q @ k.transpose(-2,-1)\n",
    "print('Shape of weights:',weights.shape,'Mean of weights:',weights.mean(),'Variance of weights:',weights.var(),\n",
    "      '\\nMin of weights:',weights.min(),'Max of weights:',weights.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e997d4a-0fae-4a15-b840-4d7077771108",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-3.9763,  1.5713, -1.4034, -2.8632,  4.2487,  1.1146, -6.5130, -2.7662])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "weights[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b34ded1-7b91-4e09-8769-54506be7f82e",
   "metadata": {},
   "source": [
    "Now if you divide the dot product of `q` and `k` by the square root of the head_size you can see that it returns the variance of `weights` back to 1 instead of approximately 17 prior to scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e487fe8-7319-4c92-b88b-d27bf2b443d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of weights: torch.Size([32, 8, 8]) Mean of weights: tensor(-0.0075) Variance of weights: tensor(1.0837) \n",
      "Min of weights: tensor(-4.0872) Max of weights: tensor(3.2824)\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "import math\n",
    "weights = (q @ k.transpose(-2,-1)) / math.sqrt(head_size) #TODO Output size is (B,T,T) (32,8,8)\n",
    "print('Shape of weights:',weights.shape,'Mean of weights:',weights.mean(),'Variance of weights:',weights.var(),\n",
    "      '\\nMin of weights:',weights.min(),'Max of weights:',weights.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c965f2d8-7462-4a67-9549-2baca77903d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.9941,  0.3928, -0.3508, -0.7158,  1.0622,  0.2786, -1.6282, -0.6915])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#| output: true\n",
    "weights[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef9ece9-1966-4897-ba5b-8996b44706d7",
   "metadata": {},
   "source": [
    "We'll create a very basic function to plot the tensors to help visualize the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87de239d-1890-4264-8901-512a9d1421b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "import matplotlib.pyplot as plt\n",
    "def plot_1d_tensor(x):\n",
    "    print(x)\n",
    "    plt.bar(range(len(x)), x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bf0f04c-a5c8-4f1c-9e07-7e136117183d",
   "metadata": {},
   "source": [
    "Again, the reason that scaling `weights` is important is because of the subsequent `softmax` that is applied. When large values are input into `softmax` it causes the gradients to be small and the output of the softmax to converge toward one-hot vectors. First we'll start out with one of the example weights that has already been divided by `math.sqrt(head_size)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7642dc8-b540-41f2-b239-ac13adc2d8f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0465, 0.1862, 0.0885, 0.0614, 0.3636, 0.1661, 0.0247, 0.0630])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmf0lEQVR4nO3df0xVd57/8dcV5F7XlbutKGhAZBxbZLAOvaj8GNrMtl6HsY3d+SE7s0WbwloibUUyyUqxO2o2xSatoh3BMtMp625EurGONqWL1+ysYGC6IwPdptN03VQH4lyGwV256jeFEc73D9Obvb1gvbfQ++H6fCQn8X54n899f6oJr37OuefaLMuyBAAAYLAZkW4AAADg8xBYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGi410A5NlbGxMv//97zVnzhzZbLZItwMAAG6DZVm6evWqFi5cqBkzJt5HiZrA8vvf/14pKSmRbgMAAIShr69PycnJE/48agLLnDlzJN1ccHx8fIS7AQAAt8Pn8yklJcX/e3wiURNYPr0MFB8fT2ABAGCa+bzbObjpFgAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4sZFuAAC+iMXb3450C2G5uGddpFsAphV2WAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA44UVWOrq6pSWliaHwyGXy6X29vYJa8+ePav8/HzNnTtXs2bNUnp6uvbt2xdQ09jYKJvNFnR88skn4bQHAACiTMhPum1ublZFRYXq6uqUn5+vV199VYWFhfrtb3+rRYsWBdXPnj1bTz/9tO677z7Nnj1bZ8+e1VNPPaXZs2dr8+bN/rr4+Hh99NFHAec6HI4wlgQAAKJNyIFl7969KikpUWlpqSSptrZWra2tqq+vV01NTVB9VlaWsrKy/K8XL16sN998U+3t7QGBxWazKSkpKZw1AACAKBfSJaGRkRF1dXXJ7XYHjLvdbnV0dNzWHN3d3ero6NCDDz4YMH7t2jWlpqYqOTlZjzzyiLq7u285z/DwsHw+X8ABAACiU0iBZXBwUKOjo0pMTAwYT0xMVH9//y3PTU5Olt1uV3Z2tsrLy/07NJKUnp6uxsZGnTx5Uk1NTXI4HMrPz9f58+cnnK+mpkZOp9N/pKSkhLIUAAAwjYT1bc02my3gtWVZQWOf1d7ermvXrulXv/qVtm/frq9+9av6wQ9+IEnKyclRTk6OvzY/P1/333+/XnnlFR04cGDc+aqqqlRZWel/7fP5CC0AAESpkAJLQkKCYmJignZTBgYGgnZdPistLU2StHz5cv3hD3/Qzp07/YHls2bMmKGVK1fecofFbrfLbreH0j4AAJimQrokFBcXJ5fLJY/HEzDu8XiUl5d32/NYlqXh4eFb/rynp0cLFiwIpT0AABClQr4kVFlZqeLiYmVnZys3N1cNDQ3q7e1VWVmZpJuXai5duqTDhw9Lkg4ePKhFixYpPT1d0s3nsrz00kt65pln/HPu2rVLOTk5Wrp0qXw+nw4cOKCenh4dPHhwMtYIAACmuZADS1FRkS5fvqzdu3fL6/UqMzNTLS0tSk1NlSR5vV719vb668fGxlRVVaULFy4oNjZWS5Ys0Z49e/TUU0/5a65cuaLNmzerv79fTqdTWVlZamtr06pVqyZhiQAAYLqzWZZlRbqJyeDz+eR0OjU0NKT4+PhItwPgS7J4+9uRbiEsF/esi3QLgBFu9/c33yUEAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYLywAktdXZ3S0tLkcDjkcrnU3t4+Ye3Zs2eVn5+vuXPnatasWUpPT9e+ffuC6o4dO6aMjAzZ7XZlZGTo+PHj4bQGAACiUMiBpbm5WRUVFaqurlZ3d7cKCgpUWFio3t7ecetnz56tp59+Wm1tbfrwww+1Y8cO7dixQw0NDf6azs5OFRUVqbi4WO+9956Ki4u1YcMGvfvuu+GvDAAARA2bZVlWKCesXr1a999/v+rr6/1jy5Yt02OPPaaamprbmuM73/mOZs+erX/6p3+SJBUVFcnn8+mdd97x13zrW9/SXXfdpaamptua0+fzyel0amhoSPHx8SGsCMB0tnj725FuISwX96yLdAuAEW7393dIOywjIyPq6uqS2+0OGHe73ero6LitObq7u9XR0aEHH3zQP9bZ2Rk059q1a2855/DwsHw+X8ABAACiU0iBZXBwUKOjo0pMTAwYT0xMVH9//y3PTU5Olt1uV3Z2tsrLy1VaWur/WX9/f8hz1tTUyOl0+o+UlJRQlgIAAKaRsG66tdlsAa8tywoa+6z29nadO3dOhw4dUm1tbdClnlDnrKqq0tDQkP/o6+sLcRUAAGC6iA2lOCEhQTExMUE7HwMDA0E7JJ+VlpYmSVq+fLn+8Ic/aOfOnfrBD34gSUpKSgp5TrvdLrvdHkr7AABgmgpphyUuLk4ul0sejydg3OPxKC8v77bnsSxLw8PD/te5ublBc546dSqkOQEAQPQKaYdFkiorK1VcXKzs7Gzl5uaqoaFBvb29Kisrk3TzUs2lS5d0+PBhSdLBgwe1aNEipaenS7r5XJaXXnpJzzzzjH/OrVu36oEHHtCLL76o9evX68SJEzp9+rTOnj07GWsEAADTXMiBpaioSJcvX9bu3bvl9XqVmZmplpYWpaamSpK8Xm/AM1nGxsZUVVWlCxcuKDY2VkuWLNGePXv01FNP+Wvy8vJ09OhR7dixQ88//7yWLFmi5uZmrV69ehKWCAAApruQn8NiKp7DAtyZeA4LML1NyXNYAAAAIoHAAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxwgosdXV1SktLk8PhkMvlUnt7+4S1b775ptasWaN58+YpPj5eubm5am1tDahpbGyUzWYLOj755JNw2gMAAFEm5MDS3NysiooKVVdXq7u7WwUFBSosLFRvb++49W1tbVqzZo1aWlrU1dWlb37zm3r00UfV3d0dUBcfHy+v1xtwOByO8FYFAACiSmyoJ+zdu1clJSUqLS2VJNXW1qq1tVX19fWqqakJqq+trQ14/cILL+jEiRN66623lJWV5R+32WxKSkoKtR0AAHAHCGmHZWRkRF1dXXK73QHjbrdbHR0dtzXH2NiYrl69qrvvvjtg/Nq1a0pNTVVycrIeeeSRoB2YzxoeHpbP5ws4AABAdAopsAwODmp0dFSJiYkB44mJierv77+tOV5++WVdv35dGzZs8I+lp6ersbFRJ0+eVFNTkxwOh/Lz83X+/PkJ56mpqZHT6fQfKSkpoSwFAABMI2HddGuz2QJeW5YVNDaepqYm7dy5U83NzZo/f75/PCcnR48//rhWrFihgoICvfHGG7rnnnv0yiuvTDhXVVWVhoaG/EdfX184SwEAANNASPewJCQkKCYmJmg3ZWBgIGjX5bOam5tVUlKif/mXf9HDDz98y9oZM2Zo5cqVt9xhsdvtstvtt988AACYtkLaYYmLi5PL5ZLH4wkY93g8ysvLm/C8pqYmPfHEEzpy5IjWrVv3ue9jWZZ6enq0YMGCUNoDAABRKuRPCVVWVqq4uFjZ2dnKzc1VQ0ODent7VVZWJunmpZpLly7p8OHDkm6GlY0bN2r//v3Kycnx787MmjVLTqdTkrRr1y7l5ORo6dKl8vl8OnDggHp6enTw4MHJWicAAJjGQg4sRUVFunz5snbv3i2v16vMzEy1tLQoNTVVkuT1egOeyfLqq6/qxo0bKi8vV3l5uX9806ZNamxslCRduXJFmzdvVn9/v5xOp7KystTW1qZVq1Z9weUBAIBoYLMsy4p0E5PB5/PJ6XRqaGhI8fHxkW4HwJdk8fa3I91CWC7u+fzL48Cd4HZ/f/NdQgAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMF1ZgqaurU1pamhwOh1wul9rb2yesffPNN7VmzRrNmzdP8fHxys3NVWtra1DdsWPHlJGRIbvdroyMDB0/fjyc1gAAQBQKObA0NzeroqJC1dXV6u7uVkFBgQoLC9Xb2ztufVtbm9asWaOWlhZ1dXXpm9/8ph599FF1d3f7azo7O1VUVKTi4mK99957Ki4u1oYNG/Tuu++GvzIAABA1bJZlWaGcsHr1at1///2qr6/3jy1btkyPPfaYampqbmuOr33tayoqKtLf//3fS5KKiork8/n0zjvv+Gu+9a1v6a677lJTU9Ntzenz+eR0OjU0NKT4+PgQVgRgOlu8/e1ItxCWi3vWRboFwAi3+/s7pB2WkZERdXV1ye12B4y73W51dHTc1hxjY2O6evWq7r77bv9YZ2dn0Jxr16697TkBAEB0iw2leHBwUKOjo0pMTAwYT0xMVH9//23N8fLLL+v69evasGGDf6y/vz/kOYeHhzU8POx/7fP5buv9AQDA9BPWTbc2my3gtWVZQWPjaWpq0s6dO9Xc3Kz58+d/oTlramrkdDr9R0pKSggrAAAA00lIgSUhIUExMTFBOx8DAwNBOySf1dzcrJKSEr3xxht6+OGHA36WlJQU8pxVVVUaGhryH319faEsBQAATCMhBZa4uDi5XC55PJ6AcY/Ho7y8vAnPa2pq0hNPPKEjR45o3brgG81yc3OD5jx16tQt57Tb7YqPjw84AABAdArpHhZJqqysVHFxsbKzs5Wbm6uGhgb19vaqrKxM0s2dj0uXLunw4cOSboaVjRs3av/+/crJyfHvpMyaNUtOp1OStHXrVj3wwAN68cUXtX79ep04cUKnT5/W2bNnJ2udAABgGgv5HpaioiLV1tZq9+7d+vrXv662tja1tLQoNTVVkuT1egOeyfLqq6/qxo0bKi8v14IFC/zH1q1b/TV5eXk6evSoXn/9dd13331qbGxUc3OzVq9ePQlLBAAA013Iz2ExFc9hAe5MPIcFmN6m5DksAAAAkUBgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwXshPukX04nkWAABTscMCAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxouNdAMAgFtbvP3tSLcQtot71kW6BUQJdlgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYLywAktdXZ3S0tLkcDjkcrnU3t4+Ya3X69UPf/hD3XvvvZoxY4YqKiqCahobG2Wz2YKOTz75JJz2AABAlAk5sDQ3N6uiokLV1dXq7u5WQUGBCgsL1dvbO2798PCw5s2bp+rqaq1YsWLCeePj4+X1egMOh8MRansAACAKhRxY9u7dq5KSEpWWlmrZsmWqra1VSkqK6uvrx61fvHix9u/fr40bN8rpdE44r81mU1JSUsABAAAghRhYRkZG1NXVJbfbHTDudrvV0dHxhRq5du2aUlNTlZycrEceeUTd3d23rB8eHpbP5ws4AABAdAopsAwODmp0dFSJiYkB44mJierv7w+7ifT0dDU2NurkyZNqamqSw+FQfn6+zp8/P+E5NTU1cjqd/iMlJSXs9wcAAGYL66Zbm80W8NqyrKCxUOTk5Ojxxx/XihUrVFBQoDfeeEP33HOPXnnllQnPqaqq0tDQkP/o6+sL+/0BAIDZYkMpTkhIUExMTNBuysDAQNCuyxcxY8YMrVy58pY7LHa7XXa7fdLeEwAAmCukHZa4uDi5XC55PJ6AcY/Ho7y8vElryrIs9fT0aMGCBZM2JwAAmL5C2mGRpMrKShUXFys7O1u5ublqaGhQb2+vysrKJN28VHPp0iUdPnzYf05PT4+kmzfW/vGPf1RPT4/i4uKUkZEhSdq1a5dycnK0dOlS+Xw+HThwQD09PTp48OAkLBEAAEx3IQeWoqIiXb58Wbt375bX61VmZqZaWlqUmpoq6eaD4j77TJasrCz/n7u6unTkyBGlpqbq4sWLkqQrV65o8+bN6u/vl9PpVFZWltra2rRq1aovsDQAABAtQg4skrRlyxZt2bJl3J81NjYGjVmWdcv59u3bp3379oXTCgAAuAPwXUIAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjBdWYKmrq1NaWpocDodcLpfa29snrPV6vfrhD3+oe++9VzNmzFBFRcW4dceOHVNGRobsdrsyMjJ0/PjxcFoDAABRKOTA0tzcrIqKClVXV6u7u1sFBQUqLCxUb2/vuPXDw8OaN2+eqqurtWLFinFrOjs7VVRUpOLiYr333nsqLi7Whg0b9O6774baHgAAiEIhB5a9e/eqpKREpaWlWrZsmWpra5WSkqL6+vpx6xcvXqz9+/dr48aNcjqd49bU1tZqzZo1qqqqUnp6uqqqqvTQQw+ptrY21PYAAEAUCimwjIyMqKurS263O2Dc7Xaro6Mj7CY6OzuD5ly7du0t5xweHpbP5ws4AABAdAopsAwODmp0dFSJiYkB44mJierv7w+7if7+/pDnrKmpkdPp9B8pKSlhvz8AADBbWDfd2my2gNeWZQWNTfWcVVVVGhoa8h99fX1f6P0BAIC5YkMpTkhIUExMTNDOx8DAQNAOSSiSkpJCntNut8tut4f9ngAAYPoIaYclLi5OLpdLHo8nYNzj8SgvLy/sJnJzc4PmPHXq1BeaEwAARI+QdlgkqbKyUsXFxcrOzlZubq4aGhrU29ursrIySTcv1Vy6dEmHDx/2n9PT0yNJunbtmv74xz+qp6dHcXFxysjIkCRt3bpVDzzwgF588UWtX79eJ06c0OnTp3X27NlJWCIAAJjuQg4sRUVFunz5snbv3i2v16vMzEy1tLQoNTVV0s0HxX32mSxZWVn+P3d1denIkSNKTU3VxYsXJUl5eXk6evSoduzYoeeff15LlixRc3OzVq9e/QWWBgAAokXIgUWStmzZoi1btoz7s8bGxqAxy7I+d87vfe97+t73vhdOOwAAIMrxXUIAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIwX1qeEgOls8fa3I91CWC7uWRfpFgAgYthhAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGC820g0AAHAnWbz97Ui3EJaLe9ZF9P3ZYQEAAMZjhwWIUvxfHIBowg4LAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxwgosdXV1SktLk8PhkMvlUnt7+y3rz5w5I5fLJYfDoa985Ss6dOhQwM8bGxtls9mCjk8++SSc9gAAQJQJObA0NzeroqJC1dXV6u7uVkFBgQoLC9Xb2ztu/YULF/Ttb39bBQUF6u7u1nPPPadnn31Wx44dC6iLj4+X1+sNOBwOR3irAgAAUSXkB8ft3btXJSUlKi0tlSTV1taqtbVV9fX1qqmpCao/dOiQFi1apNraWknSsmXLdO7cOb300kv67ne/66+z2WxKSkoKcxkAACCahbTDMjIyoq6uLrnd7oBxt9utjo6Occ/p7OwMql+7dq3OnTunP/3pT/6xa9euKTU1VcnJyXrkkUfU3d19y16Gh4fl8/kCDgAAEJ1CCiyDg4MaHR1VYmJiwHhiYqL6+/vHPae/v3/c+hs3bmhwcFCSlJ6ersbGRp08eVJNTU1yOBzKz8/X+fPnJ+ylpqZGTqfTf6SkpISyFAAAMI2EddOtzWYLeG1ZVtDY59X/3/GcnBw9/vjjWrFihQoKCvTGG2/onnvu0SuvvDLhnFVVVRoaGvIffX194SwFAABMAyHdw5KQkKCYmJig3ZSBgYGgXZRPJSUljVsfGxuruXPnjnvOjBkztHLlylvusNjtdtnt9lDaBwAA01RIgSUuLk4ul0sej0d/9Vd/5R/3eDxav379uOfk5ubqrbfeChg7deqUsrOzNXPmzHHPsSxLPT09Wr58eSjtTZnp+q23Et98CwCIDiFfEqqsrNTPfvYz/fznP9eHH36obdu2qbe3V2VlZZJuXqrZuHGjv76srEy/+93vVFlZqQ8//FA///nP9dprr+lHP/qRv2bXrl1qbW3Vxx9/rJ6eHpWUlKinp8c/JwAAuLOF/LHmoqIiXb58Wbt375bX61VmZqZaWlqUmpoqSfJ6vQHPZElLS1NLS4u2bdumgwcPauHChTpw4EDAR5qvXLmizZs3q7+/X06nU1lZWWpra9OqVasmYYkAAGC6CzmwSNKWLVu0ZcuWcX/W2NgYNPbggw/qN7/5zYTz7du3T/v27QunFQAAcAfgu4QAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8WIj3QAAAJK0ePvbkW4hLBf3rIt0C3cEdlgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgvLACS11dndLS0uRwOORyudTe3n7L+jNnzsjlcsnhcOgrX/mKDh06FFRz7NgxZWRkyG63KyMjQ8ePHw+nNQAAEIVCDizNzc2qqKhQdXW1uru7VVBQoMLCQvX29o5bf+HCBX37299WQUGBuru79dxzz+nZZ5/VsWPH/DWdnZ0qKipScXGx3nvvPRUXF2vDhg169913w18ZAACIGiEHlr1796qkpESlpaVatmyZamtrlZKSovr6+nHrDx06pEWLFqm2tlbLli1TaWmpnnzySb300kv+mtraWq1Zs0ZVVVVKT09XVVWVHnroIdXW1oa9MAAAED1iQykeGRlRV1eXtm/fHjDudrvV0dEx7jmdnZ1yu90BY2vXrtVrr72mP/3pT5o5c6Y6Ozu1bdu2oJpbBZbh4WENDw/7Xw8NDUmSfD5fKEu6LWPD/2/S5/yyhPLfY7quM9S/c9ZpNtYZbLquUboz1sm/2cmZ17KsW9aFFFgGBwc1OjqqxMTEgPHExET19/ePe05/f/+49Tdu3NDg4KAWLFgwYc1Ec0pSTU2Ndu3aFTSekpJyu8u5IzhrI93B1LsT1iixzmjDOqPHnbBGaerXefXqVTmdzgl/HlJg+ZTNZgt4bVlW0Njn1X92PNQ5q6qqVFlZ6X89Njam//mf/9HcuXNveZ5JfD6fUlJS1NfXp/j4+Ei3M2VYZ3RhndHjTlijxDpNZ1mWrl69qoULF96yLqTAkpCQoJiYmKCdj4GBgaAdkk8lJSWNWx8bG6u5c+fesmaiOSXJbrfLbrcHjP3FX/zF7S7FKPHx8dPqH1e4WGd0YZ3R405Yo8Q6TXarnZVPhXTTbVxcnFwulzweT8C4x+NRXl7euOfk5uYG1Z86dUrZ2dmaOXPmLWsmmhMAANxZQr4kVFlZqeLiYmVnZys3N1cNDQ3q7e1VWVmZpJuXai5duqTDhw9LksrKyvSTn/xElZWV+tu//Vt1dnbqtddeU1NTk3/OrVu36oEHHtCLL76o9evX68SJEzp9+rTOnj07ScsEAADTWciBpaioSJcvX9bu3bvl9XqVmZmplpYWpaamSpK8Xm/AM1nS0tLU0tKibdu26eDBg1q4cKEOHDig7373u/6avLw8HT16VDt27NDzzz+vJUuWqLm5WatXr56EJZrLbrfrxz/+cdClrWjDOqML64wed8IaJdYZLWzW532OCAAAIML4LiEAAGA8AgsAADAegQUAABiPwAIAAIxHYImguro6paWlyeFwyOVyqb29PdItTaq2tjY9+uijWrhwoWw2m37xi19EuqUpUVNTo5UrV2rOnDmaP3++HnvsMX300UeRbmtS1dfX67777vM/kCo3N1fvvPNOpNuacjU1NbLZbKqoqIh0K5Nq586dstlsAUdSUlKk25oSly5d0uOPP665c+fqz/7sz/T1r39dXV1dkW5rUi1evDjo79Nms6m8vDzSrU0qAkuENDc3q6KiQtXV1eru7lZBQYEKCwsDPhI+3V2/fl0rVqzQT37yk0i3MqXOnDmj8vJy/epXv5LH49GNGzfkdrt1/fr1SLc2aZKTk7Vnzx6dO3dO586d01/+5V9q/fr1+uCDDyLd2pT59a9/rYaGBt13332RbmVKfO1rX5PX6/Uf77//fqRbmnT/+7//q/z8fM2cOVPvvPOOfvvb3+rll1+etk9Fn8ivf/3rgL/LTx/E+v3vfz/CnU0yCxGxatUqq6ysLGAsPT3d2r59e4Q6mlqSrOPHj0e6jS/FwMCAJck6c+ZMpFuZUnfddZf1s5/9LNJtTImrV69aS5cutTwej/Xggw9aW7dujXRLk+rHP/6xtWLFiki3MeX+7u/+zvrGN74R6Ta+dFu3brWWLFlijY2NRbqVScUOSwSMjIyoq6tLbrc7YNztdqujoyNCXWGyDA0NSZLuvvvuCHcyNUZHR3X06FFdv35dubm5kW5nSpSXl2vdunV6+OGHI93KlDl//rwWLlyotLQ0/fVf/7U+/vjjSLc06U6ePKns7Gx9//vf1/z585WVlaWf/vSnkW5rSo2MjOif//mf9eSTT06bLwK+XQSWCBgcHNTo6GjQlzsmJiYGfQkkphfLslRZWalvfOMbyszMjHQ7k+r999/Xn//5n8tut6usrEzHjx9XRkZGpNuadEePHtVvfvMb1dTURLqVKbN69WodPnxYra2t+ulPf6r+/n7l5eXp8uXLkW5tUn388ceqr6/X0qVL1draqrKyMj377LP+r46JRr/4xS905coVPfHEE5FuZdKF/Gh+TJ7Ppl/LsqIuEd9pnn76af3nf/5nVH4P1r333quenh5duXJFx44d06ZNm3TmzJmoCi19fX3aunWrTp06JYfDEel2pkxhYaH/z8uXL1dubq6WLFmif/zHf1RlZWUEO5tcY2Njys7O1gsvvCBJysrK0gcffKD6+npt3Lgxwt1Njddee02FhYVauHBhpFuZdOywREBCQoJiYmKCdlMGBgaCdl0wfTzzzDM6efKkfvnLXyo5OTnS7Uy6uLg4ffWrX1V2drZqamq0YsUK7d+/P9JtTaquri4NDAzI5XIpNjZWsbGxOnPmjA4cOKDY2FiNjo5GusUpMXv2bC1fvlznz5+PdCuTasGCBUGBetmyZVH14Yb/63e/+51Onz6t0tLSSLcyJQgsERAXFyeXy+W/k/tTHo9HeXl5EeoK4bIsS08//bTefPNN/du//ZvS0tIi3dKXwrIsDQ8PR7qNSfXQQw/p/fffV09Pj//Izs7W3/zN36inp0cxMTGRbnFKDA8P68MPP9SCBQsi3cqkys/PD3rEwH/913/5v6w32rz++uuaP3++1q1bF+lWpgSXhCKksrJSxcXFys7OVm5urhoaGtTb26uysrJItzZprl27pv/+7//2v75w4YJ6enp09913a9GiRRHsbHKVl5fryJEjOnHihObMmePfOXM6nZo1a1aEu5sczz33nAoLC5WSkqKrV6/q6NGj+vd//3f967/+a6Rbm1Rz5swJuvdo9uzZmjt3blTdk/SjH/1Ijz76qBYtWqSBgQH9wz/8g3w+nzZt2hTp1ibVtm3blJeXpxdeeEEbNmzQf/zHf6ihoUENDQ2Rbm3SjY2N6fXXX9emTZsUGxulv9oj+yGlO9vBgwet1NRUKy4uzrr//vuj7mOwv/zlLy1JQcemTZsi3dqkGm+NkqzXX3890q1NmieffNL/b3XevHnWQw89ZJ06dSrSbX0povFjzUVFRdaCBQusmTNnWgsXLrS+853vWB988EGk25oSb731lpWZmWnZ7XYrPT3damhoiHRLU6K1tdWSZH300UeRbmXK2CzLsiITlQAAAG4P97AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYLz/DzssoOzio3XgAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_1d_tensor(F.softmax(torch.tensor([-0.9941,  0.3928, -0.3508, -0.7158,  1.0622,  0.2786, -1.6282, -0.6915]), dim=-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8007df63-e553-4549-a2a7-dd7273ad370e",
   "metadata": {},
   "source": [
    "You can see the the output of softmax here is diffuse. None of the output values are overly large or small. If you multiply these same values by `math.sqrt(head_size)`, effectively undoing scaling we applied, you will see that the results after softmax are less evenly distributed or diffuse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4d57bbf-da0a-4415-8302-85862e21b619",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2.3960e-04, 6.1487e-02, 3.1406e-03, 7.2937e-04, 8.9464e-01, 3.8940e-02,\n",
      "        1.8965e-05, 8.0382e-04])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAbKElEQVR4nO3dcXDfdX348VdoSILahNHa0EooGaJ2RlCSwZLS2w00XOxxcttJPTZapL0zt2JpM7y19k6k5xm2m1x12miF4jhRuk1w7KyU7OZKsfMGWXtj0FO2oqmQmmudScVdOpLP7w9+zS1LCv2GxBdJH4+7zx3fdz+f5PW55MjzPt/v9/MtK4qiCACAJGdlDwAAnNnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQqjx7gNMxMjISL774YsydOzfKysqyxwEATkNRFHH8+PFYtGhRnHXWqa9/zIgYefHFF6Ouri57DABgEg4fPhwXXHDBKf99RsTI3LlzI+KVk6murk6eBgA4HYODg1FXVzf6d/xUZkSMnHxqprq6WowAwAzzWi+x8AJWACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUpVnDwBwKhdt/E72CJPy47uWZ48AM4orIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKSaVIxs27Yt6uvro6qqKhobG2Pv3r2vuv8DDzwQl112WbzpTW+KhQsXxkc/+tE4duzYpAYGAGaXkmNk586dsX79+ti8eXPs378/li1bFm1tbdHb2zvh/k888USsXLkyVq9eHc8880z8zd/8TTz55JOxZs2a1z08ADDzlRwjd999d6xevTrWrFkTS5Ysia1bt0ZdXV10dXVNuP8PfvCDuOiii2LdunVRX18fV111VXzsYx+Lp5566nUPDwDMfCXFyIkTJ6KnpydaW1vHrLe2tsa+ffsmPKalpSV++tOfxq5du6IoivjZz34Wf/u3fxvLly+f/NQAwKxRUowcPXo0hoeHo7a2dsx6bW1tHDlyZMJjWlpa4oEHHogVK1ZERUVFnH/++XHuuefGX/7lX57y+wwNDcXg4OCYDQCYnSb1AtaysrIxj4uiGLd20rPPPhvr1q2LT33qU9HT0xOPPvpoPP/889He3n7Kr9/Z2Rk1NTWjW11d3WTGBABmgJJiZP78+TFnzpxxV0H6+/vHXS05qbOzM5YuXRqf+MQn4tJLL41rr702tm3bFjt27Ii+vr4Jj9m0aVMMDAyMbocPHy5lTABgBikpRioqKqKxsTG6u7vHrHd3d0dLS8uEx/zqV7+Ks84a+23mzJkTEa9cUZlIZWVlVFdXj9kAgNmp5KdpOjo64p577okdO3bEwYMHY8OGDdHb2zv6tMumTZti5cqVo/tfd9118dBDD0VXV1ccOnQovv/978e6deviiiuuiEWLFk3dmQAAM1J5qQesWLEijh07Flu2bIm+vr5oaGiIXbt2xeLFiyMioq+vb8w9R26++eY4fvx4fPGLX4w/+ZM/iXPPPTeuvvrq+LM/+7OpOwsAYMYqK071XMkbyODgYNTU1MTAwICnbOAMctHG72SPMCk/vsutCyDi9P9++2waACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUk0qRrZt2xb19fVRVVUVjY2NsXfv3lfdf2hoKDZv3hyLFy+OysrKuPjii2PHjh2TGhgAmF3KSz1g586dsX79+ti2bVssXbo0vvKVr0RbW1s8++yzceGFF054zA033BA/+9nP4t577423v/3t0d/fHy+//PLrHh4AmPnKiqIoSjngyiuvjMsvvzy6urpG15YsWRLXX399dHZ2jtv/0UcfjY985CNx6NChOO+88yY15ODgYNTU1MTAwEBUV1dP6msAM89FG7+TPcKk/Piu5dkjwBvC6f79LulpmhMnTkRPT0+0traOWW9tbY19+/ZNeMwjjzwSTU1N8ed//ufxtre9Ld7xjnfE7bffHv/93/99yu8zNDQUg4ODYzYAYHYq6Wmao0ePxvDwcNTW1o5Zr62tjSNHjkx4zKFDh+KJJ56IqqqqePjhh+Po0aPxx3/8x/Hzn//8lK8b6ezsjDvvvLOU0QCAGWpSL2AtKysb87goinFrJ42MjERZWVk88MADccUVV8QHP/jBuPvuu+NrX/vaKa+ObNq0KQYGBka3w4cPT2ZMAGAGKOnKyPz582POnDnjroL09/ePu1py0sKFC+Ntb3tb1NTUjK4tWbIkiqKIn/70p3HJJZeMO6aysjIqKytLGQ0AmKFKujJSUVERjY2N0d3dPWa9u7s7WlpaJjxm6dKl8eKLL8Yvf/nL0bUf/ehHcdZZZ8UFF1wwiZEBgNmk5KdpOjo64p577okdO3bEwYMHY8OGDdHb2xvt7e0R8cpTLCtXrhzd/8Ybb4x58+bFRz/60Xj22Wfj8ccfj0984hNxyy23xDnnnDN1ZwIAzEgl32dkxYoVcezYsdiyZUv09fVFQ0ND7Nq1KxYvXhwREX19fdHb2zu6/1ve8pbo7u6Oj3/849HU1BTz5s2LG264IT7zmc9M3VkAADNWyfcZyeA+I3Bmcp8RmNmm5T4jAABTTYwAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKkmFSPbtm2L+vr6qKqqisbGxti7d+9pHff9738/ysvL473vfe9kvi0AMAuVHCM7d+6M9evXx+bNm2P//v2xbNmyaGtri97e3lc9bmBgIFauXBnXXHPNpIcFAGafkmPk7rvvjtWrV8eaNWtiyZIlsXXr1qirq4uurq5XPe5jH/tY3HjjjdHc3DzpYQGA2aekGDlx4kT09PREa2vrmPXW1tbYt2/fKY+777774j//8z/jjjvuOK3vMzQ0FIODg2M2AGB2KilGjh49GsPDw1FbWztmvba2No4cOTLhMc8991xs3LgxHnjggSgvLz+t79PZ2Rk1NTWjW11dXSljAgAzyKRewFpWVjbmcVEU49YiIoaHh+PGG2+MO++8M97xjnec9tfftGlTDAwMjG6HDx+ezJgAwAxwepcq/r/58+fHnDlzxl0F6e/vH3e1JCLi+PHj8dRTT8X+/fvj1ltvjYiIkZGRKIoiysvL47HHHourr7563HGVlZVRWVlZymgAwAxV0pWRioqKaGxsjO7u7jHr3d3d0dLSMm7/6urqePrpp+PAgQOjW3t7e7zzne+MAwcOxJVXXvn6pgcAZrySroxERHR0dMRNN90UTU1N0dzcHNu3b4/e3t5ob2+PiFeeYnnhhRfi/vvvj7POOisaGhrGHL9gwYKoqqoatw4AnJlKjpEVK1bEsWPHYsuWLdHX1xcNDQ2xa9euWLx4cURE9PX1veY9RwAATioriqLIHuK1DA4ORk1NTQwMDER1dXX2OMCvyUUbv5M9wqT8+K7l2SPAG8Lp/v322TQAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQKpJxci2bduivr4+qqqqorGxMfbu3XvKfR966KH4wAc+EG9961ujuro6mpubY/fu3ZMeGACYXUqOkZ07d8b69etj8+bNsX///li2bFm0tbVFb2/vhPs//vjj8YEPfCB27doVPT098Xu/93tx3XXXxf79+1/38ADAzFdWFEVRygFXXnllXH755dHV1TW6tmTJkrj++uujs7PztL7Gu9/97lixYkV86lOfOq39BwcHo6amJgYGBqK6urqUcYEZ7KKN38keYVJ+fNfy7BHgDeF0/36XdGXkxIkT0dPTE62trWPWW1tbY9++faf1NUZGRuL48eNx3nnnlfKtAYBZqryUnY8ePRrDw8NRW1s7Zr22tjaOHDlyWl/jc5/7XLz00ktxww03nHKfoaGhGBoaGn08ODhYypgAwAwyqRewlpWVjXlcFMW4tYl885vfjE9/+tOxc+fOWLBgwSn36+zsjJqamtGtrq5uMmMCADNASTEyf/78mDNnzrirIP39/eOulvxfO3fujNWrV8df//Vfx/vf//5X3XfTpk0xMDAwuh0+fLiUMQGAGaSkGKmoqIjGxsbo7u4es97d3R0tLS2nPO6b3/xm3HzzzfGNb3wjli9/7Rd2VVZWRnV19ZgNAJidSnrNSERER0dH3HTTTdHU1BTNzc2xffv26O3tjfb29oh45arGCy+8EPfff39EvBIiK1eujM9//vPxO7/zO6NXVc4555yoqamZwlMBAGaikmNkxYoVcezYsdiyZUv09fVFQ0ND7Nq1KxYvXhwREX19fWPuOfKVr3wlXn755Vi7dm2sXbt2dH3VqlXxta997fWfAQAwo5V8n5EM7jMCZyb3GYGZbVruMwIAMNXECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQqjx7AH49Ltr4newRJuXHdy3PHgGAaebKCACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnc9Awg0Uy9IWGEmxIydVwZAQBSiREAIJUYAQBSiREAIJUYAQBSiREAIJUYAQBSiREAIJUYAQBSiREAINWkYmTbtm1RX18fVVVV0djYGHv37n3V/ffs2RONjY1RVVUVv/mbvxlf/vKXJzUsADD7lBwjO3fujPXr18fmzZtj//79sWzZsmhra4ve3t4J93/++efjgx/8YCxbtiz2798fn/zkJ2PdunXxrW9963UPDwDMfCXHyN133x2rV6+ONWvWxJIlS2Lr1q1RV1cXXV1dE+7/5S9/OS688MLYunVrLFmyJNasWRO33HJL/MVf/MXrHh4AmPlK+tTeEydORE9PT2zcuHHMemtra+zbt2/CY/75n/85Wltbx6xde+21ce+998b//M//xNlnnz3umKGhoRgaGhp9PDAwEBERg4ODpYzL/zIy9KvsESal1J95wx27p2mS6fXvd16bPcIb0pnweztTzzHC/5N5bSd/R4qieNX9SoqRo0ePxvDwcNTW1o5Zr62tjSNHjkx4zJEjRybc/+WXX46jR4/GwoULxx3T2dkZd95557j1urq6UsZlFqjZmj3Br8eZcp5nijPl53mmnCev3/Hjx6OmpuaU/15SjJxUVlY25nFRFOPWXmv/idZP2rRpU3R0dIw+HhkZiZ///Ocxb968V/0+bzSDg4NRV1cXhw8fjurq6uxxpsWZcI4RznO2cZ6zx5lwjhEz9zyLoojjx4/HokWLXnW/kmJk/vz5MWfOnHFXQfr7+8dd/Tjp/PPPn3D/8vLymDdv3oTHVFZWRmVl5Zi1c889t5RR31Cqq6tn1C/PZJwJ5xjhPGcb5zl7nAnnGDEzz/PVroicVNILWCsqKqKxsTG6u7vHrHd3d0dLS8uExzQ3N4/b/7HHHoumpqYJXy8CAJxZSn43TUdHR9xzzz2xY8eOOHjwYGzYsCF6e3ujvb09Il55imXlypWj+7e3t8dPfvKT6OjoiIMHD8aOHTvi3nvvjdtvv33qzgIAmLFKfs3IihUr4tixY7Fly5bo6+uLhoaG2LVrVyxevDgiIvr6+sbcc6S+vj527doVGzZsiC996UuxaNGi+MIXvhB/8Ad/MHVn8QZVWVkZd9xxx7innGaTM+EcI5znbOM8Z48z4RwjZv95lhWv9X4bAIBp5LNpAIBUYgQASCVGAIBUYgQASCVGpsm2bduivr4+qqqqorGxMfbu3Zs90pR6/PHH47rrrotFixZFWVlZfPvb384eaVp0dnbGb//2b8fcuXNjwYIFcf3118cPf/jD7LGmXFdXV1x66aWjN1Rqbm6O7373u9ljTavOzs4oKyuL9evXZ48ypT796U9HWVnZmO3888/PHmtavPDCC/FHf/RHMW/evHjTm94U733ve6Onpyd7rCl10UUXjft5lpWVxdq1a7NHm1JiZBrs3Lkz1q9fH5s3b479+/fHsmXLoq2tbcxbnme6l156KS677LL44he/mD3KtNqzZ0+sXbs2fvCDH0R3d3e8/PLL0draGi+99FL2aFPqggsuiLvuuiueeuqpeOqpp+Lqq6+OD33oQ/HMM89kjzYtnnzyydi+fXtceuml2aNMi3e/+93R19c3uj399NPZI025//qv/4qlS5fG2WefHd/97nfj2Wefjc997nMz+m7dE3nyySfH/CxP3kT0wx/+cPJkU6xgyl1xxRVFe3v7mLV3vetdxcaNG5Mmml4RUTz88MPZY/xa9Pf3FxFR7NmzJ3uUafcbv/EbxT333JM9xpQ7fvx4cckllxTd3d3F7/7u7xa33XZb9khT6o477iguu+yy7DGm3Z/+6Z8WV111VfYYv3a33XZbcfHFFxcjIyPZo0wpV0am2IkTJ6KnpydaW1vHrLe2tsa+ffuSpmKqDAwMRETEeeedlzzJ9BkeHo4HH3wwXnrppWhubs4eZ8qtXbs2li9fHu9///uzR5k2zz33XCxatCjq6+vjIx/5SBw6dCh7pCn3yCOPRFNTU3z4wx+OBQsWxPve97746le/mj3WtDpx4kR8/etfj1tuuWVGfWjs6RAjU+zo0aMxPDw87oMDa2trx31gIDNLURTR0dERV111VTQ0NGSPM+WefvrpeMtb3hKVlZXR3t4eDz/8cPzWb/1W9lhT6sEHH4x//dd/jc7OzuxRps2VV14Z999/f+zevTu++tWvxpEjR6KlpSWOHTuWPdqUOnToUHR1dcUll1wSu3fvjvb29li3bl3cf//92aNNm29/+9vxi1/8Im6++ebsUaZcybeD5/T832otimLWleyZ5tZbb41/+7d/iyeeeCJ7lGnxzne+Mw4cOBC/+MUv4lvf+lasWrUq9uzZM2uC5PDhw3HbbbfFY489FlVVVdnjTJu2trbR/37Pe94Tzc3NcfHFF8df/dVfRUdHR+JkU2tkZCSampris5/9bEREvO9974tnnnkmurq6xnw+2mxy7733RltbWyxatCh7lCnnysgUmz9/fsyZM2fcVZD+/v5xV0uYOT7+8Y/HI488Et/73vfiggsuyB5nWlRUVMTb3/72aGpqis7Ozrjsssvi85//fPZYU6anpyf6+/ujsbExysvLo7y8PPbs2RNf+MIXory8PIaHh7NHnBZvfvOb4z3veU8899xz2aNMqYULF44L5SVLlsyqNwr8bz/5yU/iH/7hH2LNmjXZo0wLMTLFKioqorGxcfQVzyd1d3dHS0tL0lRMVlEUceutt8ZDDz0U//iP/xj19fXZI/3aFEURQ0ND2WNMmWuuuSaefvrpOHDgwOjW1NQUf/iHfxgHDhyIOXPmZI84LYaGhuLgwYOxcOHC7FGm1NKlS8e9zf5HP/rR6Ie2zjb33XdfLFiwIJYvX549yrTwNM006OjoiJtuuimampqiubk5tm/fHr29vdHe3p492pT55S9/Gf/xH/8x+vj555+PAwcOxHnnnRcXXnhh4mRTa+3atfGNb3wj/u7v/i7mzp07esWrpqYmzjnnnOTpps4nP/nJaGtri7q6ujh+/Hg8+OCD8U//9E/x6KOPZo82ZebOnTvutT5vfvObY968ebPqNUC33357XHfddXHhhRdGf39/fOYzn4nBwcFYtWpV9mhTasOGDdHS0hKf/exn44Ybboh/+Zd/ie3bt8f27duzR5tyIyMjcd9998WqVauivHyW/tnOfTPP7PWlL32pWLx4cVFRUVFcfvnls+6toN/73veKiBi3rVq1Knu0KTXROUZEcd9992WPNqVuueWW0d/Xt771rcU111xTPPbYY9ljTbvZ+NbeFStWFAsXLizOPvvsYtGiRcXv//7vF88880z2WNPi7//+74uGhoaisrKyeNe73lVs3749e6RpsXv37iIiih/+8IfZo0ybsqIoipwMAgDwmhEAIJkYAQBSiREAIJUYAQBSiREAIJUYAQBSiREAIJUYAQBSiREAIJUYAQBSiREAIJUYAQBS/T+L7xUTyr5PpQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| output: true\n",
    "plot_1d_tensor(F.softmax(torch.tensor([-0.9941,  0.3928, -0.3508, -0.7158,  1.0622,  0.2786, -1.6282, -0.6915])*math.sqrt(head_size), dim=-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1b59a7a-714e-4d37-ab5c-443b3ea87085",
   "metadata": {},
   "source": [
    "If you push it even further you can see that the second item in the vector continues to grow even though the value of each element, relative to eachother has not changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2bff877-9c0e-4bc4-859b-2da01c19d6b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([5.1446e-15, 2.2311e-05, 1.5187e-10, 4.4175e-13, 9.9997e-01, 3.5890e-06,\n",
      "        2.0192e-19, 6.5168e-13])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAcvUlEQVR4nO3df2xd9X3w8Y+JsU3b2IykMXExwaW0zZqSFntkdoimQmtkomhoVUnFRkIBqdZCQ+JRNSFSKVFVs2lFlEFMUxIYGi1Rx48xNYV4WhdCQzXiORuCqGULxSk4tZKudkgnp0nu8wdP/DyeHZrr2v1g5/WSjsT95hzfz1EAv3XuveeWFAqFQgAAJDkjewAA4PQmRgCAVGIEAEglRgCAVGIEAEglRgCAVGIEAEglRgCAVKXZA5yK48ePxxtvvBHTp0+PkpKS7HEAgFNQKBTi0KFDUVNTE2eccfLrH5MiRt54442ora3NHgMAGIN9+/bFeeedd9I/nxQxMn369Ih462QqKyuTpwEATsXAwEDU1tYO/R4/mUkRIydemqmsrBQjADDJ/Ka3WHgDKwCQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKmKjpFnn302lixZEjU1NVFSUhJPPvnkbzxm+/btUV9fHxUVFfH+978/7r///rHMCgBMQUXHyOHDh2P+/Plx7733ntL+r776alx11VWxaNGi6O7ujttuuy1WrlwZjz32WNHDAgBTT9FflNfS0hItLS2nvP/9998f559/ftx9990RETF37tzYtWtX/PVf/3V8+tOfLvbpAYApZsLfM/L8889Hc3PzsLUrr7wydu3aFb/+9a9HPWZwcDAGBgaGbQDA1FT0lZFi7d+/P6qrq4etVVdXx9GjR+PAgQMxe/bsEce0t7fHHXfcMdGjAe9wF6z5XvYIY/LTOxdnjwCTyu/k0zQlJSXDHhcKhVHXT1i7dm309/cPbfv27ZvwGQGAHBN+ZeTcc8+N/fv3D1vr6+uL0tLSmDFjxqjHlJeXR3l5+USPBgC8A0z4lZHGxsbo7OwctrZt27ZoaGiIM888c6KfHgB4hys6Rt58883YvXt37N69OyLe+uju7t27o6enJyLeeoll2bJlQ/u3trbGa6+9Fm1tbbFnz57YvHlzbNq0KW699dbxOQMAYFIr+mWaXbt2xSc+8Ymhx21tbRERsXz58njooYeit7d3KEwiIurq6mLr1q2xevXquO+++6KmpibuueceH+sFACIioqRw4t2k72ADAwNRVVUV/f39UVlZmT0O8Dvi0zQwuZ3q72/fTQMApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBIjAEAqMQIApBpTjGzYsCHq6uqioqIi6uvrY8eOHW+7/yOPPBLz58+Pd73rXTF79uz43Oc+FwcPHhzTwADA1FJ0jGzZsiVWrVoV69ati+7u7li0aFG0tLRET0/PqPs/99xzsWzZsrjxxhvjpZdeiu9+97vxwgsvxE033fRbDw8ATH5Fx8hdd90VN954Y9x0000xd+7cuPvuu6O2tjY6OjpG3f9HP/pRXHDBBbFy5cqoq6uLyy67LD7/+c/Hrl27fuvhAYDJr6gYOXLkSHR1dUVzc/Ow9ebm5ti5c+eoxzQ1NcXPfvaz2Lp1axQKhfj5z38ef//3fx+LFy8e+9QAwJRRVIwcOHAgjh07FtXV1cPWq6urY//+/aMe09TUFI888kgsXbo0ysrK4txzz42zzz47/uZv/uakzzM4OBgDAwPDNgBgahrTG1hLSkqGPS4UCiPWTnj55Zdj5cqV8eUvfzm6urri6aefjldffTVaW1tP+vPb29ujqqpqaKutrR3LmADAJFBUjMycOTOmTZs24ipIX1/fiKslJ7S3t8fChQvji1/8Ylx88cVx5ZVXxoYNG2Lz5s3R29s76jFr166N/v7+oW3fvn3FjAkATCJFxUhZWVnU19dHZ2fnsPXOzs5oamoa9Zhf/epXccYZw59m2rRpEfHWFZXRlJeXR2Vl5bANAJiain6Zpq2tLR544IHYvHlz7NmzJ1avXh09PT1DL7usXbs2li1bNrT/kiVL4vHHH4+Ojo7Yu3dv/PCHP4yVK1fGpZdeGjU1NeN3JgDApFRa7AFLly6NgwcPxvr166O3tzfmzZsXW7dujTlz5kRERG9v77B7jlx//fVx6NChuPfee+Mv/uIv4uyzz47LL788/vIv/3L8zgIAmLRKCid7reQdZGBgIKqqqqK/v99LNnAauWDN97JHGJOf3unWBRBx6r+/fTcNAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcQIAJBKjAAAqcYUIxs2bIi6urqoqKiI+vr62LFjx9vuPzg4GOvWrYs5c+ZEeXl5XHjhhbF58+YxDQwATC2lxR6wZcuWWLVqVWzYsCEWLlwY3/zmN6OlpSVefvnlOP/880c95pprromf//znsWnTpvjABz4QfX19cfTo0d96eABg8ispFAqFYg5YsGBBXHLJJdHR0TG0Nnfu3Lj66qujvb19xP5PP/10fPazn429e/fGOeecM6YhBwYGoqqqKvr7+6OysnJMPwOYfC5Y873sEcbkp3cuzh4B3hFO9fd3US/THDlyJLq6uqK5uXnYenNzc+zcuXPUY5566qloaGiIv/qrv4r3ve998cEPfjBuvfXW+J//+Z+TPs/g4GAMDAwM2wCAqamol2kOHDgQx44di+rq6mHr1dXVsX///lGP2bt3bzz33HNRUVERTzzxRBw4cCD+/M//PH7xi1+c9H0j7e3tcccddxQzGgAwSY3pDawlJSXDHhcKhRFrJxw/fjxKSkrikUceiUsvvTSuuuqquOuuu+Khhx466dWRtWvXRn9//9C2b9++sYwJAEwCRV0ZmTlzZkybNm3EVZC+vr4RV0tOmD17drzvfe+LqqqqobW5c+dGoVCIn/3sZ3HRRReNOKa8vDzKy8uLGQ0AmKSKujJSVlYW9fX10dnZOWy9s7MzmpqaRj1m4cKF8cYbb8Sbb745tPaTn/wkzjjjjDjvvPPGMDIAMJUU/TJNW1tbPPDAA7F58+bYs2dPrF69Onp6eqK1tTUi3nqJZdmyZUP7X3vttTFjxoz43Oc+Fy+//HI8++yz8cUvfjFuuOGGOOuss8bvTACASano+4wsXbo0Dh48GOvXr4/e3t6YN29ebN26NebMmRMREb29vdHT0zO0/3ve857o7OyML3zhC9HQ0BAzZsyIa665Jr761a+O31kAAJNW0fcZyeA+I3B6cp8RmNwm5D4jAADjTYwAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQSowAAKnECACQakwxsmHDhqirq4uKioqor6+PHTt2nNJxP/zhD6O0tDQ+9rGPjeVpAYApqOgY2bJlS6xatSrWrVsX3d3dsWjRomhpaYmenp63Pa6/vz+WLVsWV1xxxZiHBQCmnqJj5K677oobb7wxbrrpppg7d27cfffdUVtbGx0dHW973Oc///m49tpro7GxcczDAgBTT1ExcuTIkejq6orm5uZh683NzbFz586THvfggw/Gf/3Xf8Xtt99+Ss8zODgYAwMDwzYAYGoqKkYOHDgQx44di+rq6mHr1dXVsX///lGPeeWVV2LNmjXxyCOPRGlp6Sk9T3t7e1RVVQ1ttbW1xYwJAEwiY3oDa0lJybDHhUJhxFpExLFjx+Laa6+NO+64Iz74wQ+e8s9fu3Zt9Pf3D2379u0by5gAwCRwapcq/q+ZM2fGtGnTRlwF6evrG3G1JCLi0KFDsWvXruju7o6bb745IiKOHz8ehUIhSktLY9u2bXH55ZePOK68vDzKy8uLGQ0AmKSKujJSVlYW9fX10dnZOWy9s7MzmpqaRuxfWVkZL774YuzevXtoa21tjQ996EOxe/fuWLBgwW83PQAw6RV1ZSQioq2tLa677rpoaGiIxsbG2LhxY/T09ERra2tEvPUSy+uvvx4PP/xwnHHGGTFv3rxhx8+aNSsqKipGrAMAp6eiY2Tp0qVx8ODBWL9+ffT29sa8efNi69atMWfOnIiI6O3t/Y33HAEAOKGkUCgUsof4TQYGBqKqqir6+/ujsrIyexzgd+SCNd/LHmFMfnrn4uwR4B3hVH9/+24aACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUokRACCVGAEAUo0pRjZs2BB1dXVRUVER9fX1sWPHjpPu+/jjj8enPvWpeO973xuVlZXR2NgYzzzzzJgHBgCmlqJjZMuWLbFq1apYt25ddHd3x6JFi6KlpSV6enpG3f/ZZ5+NT33qU7F169bo6uqKT3ziE7FkyZLo7u7+rYcHACa/kkKhUCjmgAULFsQll1wSHR0dQ2tz586Nq6++Otrb20/pZ3zkIx+JpUuXxpe//OVT2n9gYCCqqqqiv78/KisrixkXmMQuWPO97BHG5Kd3Ls4eAd4RTvX3d1FXRo4cORJdXV3R3Nw8bL25uTl27tx5Sj/j+PHjcejQoTjnnHNOus/g4GAMDAwM2wCAqamoGDlw4EAcO3Ysqqurh61XV1fH/v37T+lnfP3rX4/Dhw/HNddcc9J92tvbo6qqamirra0tZkwAYBIZ0xtYS0pKhj0uFAoj1kbzne98J77yla/Eli1bYtasWSfdb+3atdHf3z+07du3byxjAgCTQGkxO8+cOTOmTZs24ipIX1/fiKsl/9uWLVvixhtvjO9+97vxyU9+8m33LS8vj/Ly8mJGAwAmqaKujJSVlUV9fX10dnYOW+/s7IympqaTHved73wnrr/++vj2t78dixd7YxcA8P8UdWUkIqKtrS2uu+66aGhoiMbGxti4cWP09PREa2trRLz1Esvrr78eDz/8cES8FSLLli2Lb3zjG/GHf/iHQ1dVzjrrrKiqqhrHUwEAJqOiY2Tp0qVx8ODBWL9+ffT29sa8efNi69atMWfOnIiI6O3tHXbPkW9+85tx9OjRWLFiRaxYsWJoffny5fHQQw/99mcAAExqRd9nJIP7jMDpyX1GYHKbkPuMAACMNzECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAKjECAKQSIwBAqjHFyIYNG6Kuri4qKiqivr4+duzY8bb7b9++Perr66OioiLe//73x/333z+mYQGAqafoGNmyZUusWrUq1q1bF93d3bFo0aJoaWmJnp6eUfd/9dVX46qrropFixZFd3d33HbbbbFy5cp47LHHfuvhAYDJr6RQKBSKOWDBggVxySWXREdHx9Da3Llz4+qrr4729vYR+3/pS1+Kp556Kvbs2TO01traGv/+7/8ezz///Ck958DAQFRVVUV/f39UVlYWMy4wiV2w5nvZI4zJT+9cnD0CvCOc6u/v0mJ+6JEjR6KrqyvWrFkzbL25uTl27tw56jHPP/98NDc3D1u78sorY9OmTfHrX/86zjzzzBHHDA4OxuDg4NDj/v7+iHjrpIDTx/HBX2WPMCb+XwVvOfHfwm+67lFUjBw4cCCOHTsW1dXVw9arq6tj//79ox6zf//+Ufc/evRoHDhwIGbPnj3imPb29rjjjjtGrNfW1hYzLkCKqruzJ4B3lkOHDkVVVdVJ/7yoGDmhpKRk2ONCoTBi7TftP9r6CWvXro22trahx8ePH49f/OIXMWPGjLd9nneagYGBqK2tjX379k3Zl5dOh3OMcJ5TjfOcOk6Hc4yYvOdZKBTi0KFDUVNT87b7FRUjM2fOjGnTpo24CtLX1zfi6scJ55577qj7l5aWxowZM0Y9pry8PMrLy4etnX322cWM+o5SWVk5qf7lGYvT4RwjnOdU4zynjtPhHCMm53m+3RWRE4r6NE1ZWVnU19dHZ2fnsPXOzs5oamoa9ZjGxsYR+2/bti0aGhpGfb8IAHB6KfqjvW1tbfHAAw/E5s2bY8+ePbF69ero6emJ1tbWiHjrJZZly5YN7d/a2hqvvfZatLW1xZ49e2Lz5s2xadOmuPXWW8fvLACASavo94wsXbo0Dh48GOvXr4/e3t6YN29ebN26NebMmRMREb29vcPuOVJXVxdbt26N1atXx3333Rc1NTVxzz33xKc//enxO4t3qPLy8rj99ttHvOQ0lZwO5xjhPKca5zl1nA7nGDH1z7Po+4wAAIwn300DAKQSIwBAKjECAKQSIwBAKjEyQTZs2BB1dXVRUVER9fX1sWPHjuyRxtWzzz4bS5YsiZqamigpKYknn3wye6QJ0d7eHn/wB38Q06dPj1mzZsXVV18dP/7xj7PHGncdHR1x8cUXD91QqbGxMb7//e9njzWh2tvbo6SkJFatWpU9yrj6yle+EiUlJcO2c889N3usCfH666/Hn/3Zn8WMGTPiXe96V3zsYx+Lrq6u7LHG1QUXXDDi77OkpCRWrFiRPdq4EiMTYMuWLbFq1apYt25ddHd3x6JFi6KlpWXYR54nu8OHD8f8+fPj3nvvzR5lQm3fvj1WrFgRP/rRj6KzszOOHj0azc3Ncfjw4ezRxtV5550Xd955Z+zatSt27doVl19+efzxH/9xvPTSS9mjTYgXXnghNm7cGBdffHH2KBPiIx/5SPT29g5tL774YvZI4+6///u/Y+HChXHmmWfG97///Xj55Zfj61//+qS+W/doXnjhhWF/lyduIvqZz3wmebJxVmDcXXrppYXW1tZhax/+8IcLa9asSZpoYkVE4Yknnsge43eir6+vEBGF7du3Z48y4X7v936v8MADD2SPMe4OHTpUuOiiiwqdnZ2FP/qjPyrccsst2SONq9tvv70wf/787DEm3Je+9KXCZZddlj3G79wtt9xSuPDCCwvHjx/PHmVcuTIyzo4cORJdXV3R3Nw8bL25uTl27tyZNBXjpb+/PyIizjnnnORJJs6xY8fi0UcfjcOHD0djY2P2OONuxYoVsXjx4vjkJz+ZPcqEeeWVV6Kmpibq6uris5/9bOzduzd7pHH31FNPRUNDQ3zmM5+JWbNmxcc//vH41re+lT3WhDpy5Ej83d/9Xdxwww2T6ktjT4UYGWcHDhyIY8eOjfjiwOrq6hFfGMjkUigUoq2tLS677LKYN29e9jjj7sUXX4z3vOc9UV5eHq2trfHEE0/E7//+72ePNa4effTR+Ld/+7dob2/PHmXCLFiwIB5++OF45pln4lvf+lbs378/mpqa4uDBg9mjjau9e/dGR0dHXHTRRfHMM89Ea2trrFy5Mh5++OHs0SbMk08+Gb/85S/j+uuvzx5l3BV9O3hOzf+u1kKhMOVK9nRz8803x3/8x3/Ec889lz3KhPjQhz4Uu3fvjl/+8pfx2GOPxfLly2P79u1TJkj27dsXt9xyS2zbti0qKiqyx5kwLS0tQ//80Y9+NBobG+PCCy+Mv/3bv422trbEycbX8ePHo6GhIb72ta9FRMTHP/7xeOmll6Kjo2PY96NNJZs2bYqWlpaoqanJHmXcuTIyzmbOnBnTpk0bcRWkr69vxNUSJo8vfOEL8dRTT8UPfvCDOO+887LHmRBlZWXxgQ98IBoaGqK9vT3mz58f3/jGN7LHGjddXV3R19cX9fX1UVpaGqWlpbF9+/a45557orS0NI4dO5Y94oR497vfHR/96EfjlVdeyR5lXM2ePXtEKM+dO3dKfVDg//faa6/FP/3TP8VNN92UPcqEECPjrKysLOrr64fe8XxCZ2dnNDU1JU3FWBUKhbj55pvj8ccfj3/+53+Ourq67JF+ZwqFQgwODmaPMW6uuOKKePHFF2P37t1DW0NDQ/zpn/5p7N69O6ZNm5Y94oQYHByMPXv2xOzZs7NHGVcLFy4c8TH7n/zkJ0Nf2jrVPPjggzFr1qxYvHhx9igTwss0E6CtrS2uu+66aGhoiMbGxti4cWP09PREa2tr9mjj5s0334z//M//HHr86quvxu7du+Occ86J888/P3Gy8bVixYr49re/Hf/wD/8Q06dPH7riVVVVFWeddVbydOPntttui5aWlqitrY1Dhw7Fo48+Gv/yL/8STz/9dPZo42b69Okj3uvz7ne/O2bMmDGl3gN06623xpIlS+L888+Pvr6++OpXvxoDAwOxfPny7NHG1erVq6OpqSm+9rWvxTXXXBP/+q//Ghs3boyNGzdmjzbujh8/Hg8++GAsX748Skun6K/t3A/zTF333XdfYc6cOYWysrLCJZdcMuU+CvqDH/ygEBEjtuXLl2ePNq5GO8eIKDz44IPZo42rG264Yejf1/e+972FK664orBt27bssSbcVPxo79KlSwuzZ88unHnmmYWamprCn/zJnxReeuml7LEmxD/+4z8W5s2bVygvLy98+MMfLmzcuDF7pAnxzDPPFCKi8OMf/zh7lAlTUigUCjkZBADgPSMAQDIxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCkEiMAQCoxAgCk+j9ZN8TjyolWtgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#| output: true\n",
    "plot_1d_tensor(F.softmax(torch.tensor([-0.9941,  0.3928, -0.3508, -0.7158,  1.0622,  0.2786, -1.6282, -0.6915])*head_size, dim=-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3407243-2d42-40ce-8be1-ad47a31a7c16",
   "metadata": {},
   "source": [
    "As the input values to the softmax continue to grow the result of the `softmax` continues to converge to a one-hot encoded vector, which is where one of the values in the vector is 1 and all the rest are 0's. In effect this would make it so 1 node will only draw information from one other node, which is generally not what we want. This is especially a problem during initialization of the network before training, as it can be difficult for the network to recover from this during training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a48505f4-f5da-4269-95bb-20036aca4e19",
   "metadata": {},
   "source": [
    "## Continuing model definition"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b93206e-6a69-455e-8d5b-8cc348fa4924",
   "metadata": {},
   "source": [
    "Now we're going to create a `Head` module where we'll implement a single self attention head which we'll use in our transformer, replacing the bigram model. You can reference the video [link](https://youtu.be/kCc8FmEb1nY?t=4752) here to follow along if you would like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b0d58b2-86ef-47f5-9ecb-c22af4852cdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class Head(nn.Module):\n",
    "    \"\"\" one self attention head \"\"\"\n",
    "    def __init__(self, head_size:int=head_size, embedding_dim:int=embedding_dim, context_length:int=context_length):\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.head_size = head_size\n",
    "        self.context_length = context_length\n",
    "        self.key_layer   = nn.Linear(in_features=self.embedding_dim, out_features=self.head_size, bias=False)\n",
    "        self.query_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.head_size, bias=False)\n",
    "        self.value_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.head_size, bias=False)\n",
    "        self.register_buffer('tril', torch.tril(torch.ones((self.context_length, self.context_length))))\n",
    "    \n",
    "    def forward(self, x):\n",
    "        B,T,C = x.shape\n",
    "        assert T <= self.context_length #check that x.shape matches pre-defined dims\n",
    "        assert C == self.embedding_dim\n",
    "        q = self.query_layer(x) #(B,T,C) (batch size, context length, head_size\n",
    "        k = self.key_layer(x) #(B,T,C)\n",
    "        v = self.value_layer(x) #(B,T,C)\n",
    "        \n",
    "        #compute scores based on affinities\n",
    "        weights = (q @ k.transpose(-2,-1)) * self.head_size**-0.5 # (B,T,C) @ (B,C,T) -> (B,T,T)\n",
    "        weights = weights.masked_fill(self.tril[:T,:T] == 0, float('-inf')) #(B,T,T)\n",
    "        weights = F.softmax(input=weights, dim=-1) #(B,T,T)\n",
    "        \n",
    "        #perform weighted aggragation of the values\n",
    "        out = weights @ v # (B,T,T) @ (B,T,C) -> (B,T,C)\n",
    "        return out\n",
    "\n",
    "# Head()(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "961631c3-afbc-40d8-8571-36c3535a62d0",
   "metadata": {},
   "source": [
    "The `register_buffer` method is utilized to incorporate the tril matrix as a part of the model's state. This integration ensures that tril is consistently saved and loaded with the model, maintaining uniform behavior across various runs and settings. Crucially, being a buffer, tril is excluded from gradient calculations and is not included as a `parameter` during model optimization, thereby rendering it a non-trainable component of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c6eb829-4b01-4e51-8292-0c1d2831b19d",
   "metadata": {},
   "source": [
    "To make visualizing the training loss easier we'll create a simple function to plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91c5532c-b773-4f3c-b21b-0e0b96a6906b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "def plot_losses(losses):\n",
    "    train_losses = [o['train'] for o in losses if o.get('train') is not None]\n",
    "    valid_losses = [o['valid'] for o in losses if o.get('valid') is not None]\n",
    "    plt.plot(train_losses, label='Training Loss')\n",
    "    plt.plot(valid_losses, label='Validation Loss')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.title('Losses')\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bd7bbae-61a6-47c5-8bcb-3fbfa301115b",
   "metadata": {},
   "source": [
    "Now we'll add our new `Head` implementation to the `TransformerLanguageModel` class and train a model to ensure everything is working as well as to get a baseline of the results. Note we are also adding a `token_position_embedding_table` to encode the token positions. This learned looked up value will be added to the `token_embeddings`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "475fe08e-cc10-4fbc-ba63-46e4804916dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "learning_rate = 1e-3 # decreate the learning rate because self attention cannot tolerate very high learning rates.\n",
    "max_iters = 5000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faac7cfc-45a2-42a7-b4c1-e85bcd119c2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.1743) Validation Loss: tensor(4.1712)\n",
      "Step: 200 Training Loss: tensor(3.1199) Validation Loss: tensor(3.1343)\n",
      "Step: 400 Training Loss: tensor(2.8712) Validation Loss: tensor(2.8892)\n",
      "Step: 600 Training Loss: tensor(2.7071) Validation Loss: tensor(2.7260)\n",
      "Step: 800 Training Loss: tensor(2.6324) Validation Loss: tensor(2.6392)\n",
      "Step: 1000 Training Loss: tensor(2.5896) Validation Loss: tensor(2.5849)\n",
      "Step: 1200 Training Loss: tensor(2.5460) Validation Loss: tensor(2.5497)\n",
      "Step: 1400 Training Loss: tensor(2.5158) Validation Loss: tensor(2.5259)\n",
      "Step: 1600 Training Loss: tensor(2.5000) Validation Loss: tensor(2.5051)\n",
      "Step: 1800 Training Loss: tensor(2.4885) Validation Loss: tensor(2.4980)\n",
      "Step: 2000 Training Loss: tensor(2.4632) Validation Loss: tensor(2.4858)\n",
      "Step: 2200 Training Loss: tensor(2.4572) Validation Loss: tensor(2.4797)\n",
      "Step: 2400 Training Loss: tensor(2.4632) Validation Loss: tensor(2.4467)\n",
      "Step: 2600 Training Loss: tensor(2.4587) Validation Loss: tensor(2.4553)\n",
      "Step: 2800 Training Loss: tensor(2.4338) Validation Loss: tensor(2.4533)\n",
      "Step: 3000 Training Loss: tensor(2.4402) Validation Loss: tensor(2.4562)\n",
      "Step: 3200 Training Loss: tensor(2.4409) Validation Loss: tensor(2.4492)\n",
      "Step: 3400 Training Loss: tensor(2.4249) Validation Loss: tensor(2.4487)\n",
      "Step: 3600 Training Loss: tensor(2.4376) Validation Loss: tensor(2.4395)\n",
      "Step: 3800 Training Loss: tensor(2.4166) Validation Loss: tensor(2.4278)\n",
      "Step: 4000 Training Loss: tensor(2.4102) Validation Loss: tensor(2.4275)\n",
      "Step: 4200 Training Loss: tensor(2.4191) Validation Loss: tensor(2.4384)\n",
      "Step: 4400 Training Loss: tensor(2.4178) Validation Loss: tensor(2.4217)\n",
      "Step: 4600 Training Loss: tensor(2.4077) Validation Loss: tensor(2.4109)\n",
      "Step: 4800 Training Loss: tensor(2.4062) Validation Loss: tensor(2.4189)\n",
      "Step: 4999 Training Loss: tensor(2.4043) Validation Loss: tensor(2.4176)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGxCAYAAACKvAkXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABcvklEQVR4nO3deXRU9f3/8eedLJN1Jglkw4RNEATZwxJQRFldKIoK9mtBFOsPi/vXqri0UhfUlhb5utVWwRVQEbVV2YqAKCpUQFREkCUICRAgmayTZOb+/phkIBAgZLuT5PU4556ZuXPvnfedTs2Lu3zehmmaJiIiIiLNiM3qAkREREQamgKQiIiINDsKQCIiItLsKACJiIhIs6MAJCIiIs2OApCIiIg0OwpAIiIi0uwoAImIiEizowAkIiIizY4CkIgEjLlz52IYBuvXr7e6FBFp4hSAREREpNlRABIREZFmRwFIRBqVNWvWMHToUKKjo4mIiGDgwIF89NFHlZYpLCzknnvuoV27doSFhREXF0daWhrz5s3zL7Njxw6uvfZaWrVqhd1uJzExkaFDh7Jx48ZK21qwYAHp6elERkYSFRXFyJEj2bBhQ6VlqrstEQkcwVYXICJSXatWrWL48OF0796dl19+GbvdzvPPP8/o0aOZN28e48ePB+Duu+/m9ddf57HHHqNXr14UFBTw3XffcejQIf+2Lr30UjweD08//TStW7cmOzubL774gpycHP8yTzzxBA899BA33HADDz30ECUlJfz5z3/mggsu4Ouvv6ZLly7V3paIBBbDNE3T6iJERMB3EfQNN9zAunXrSEtLO+H99PR0duzYwc8//0xUVBQAHo+Hnj17kpOTQ0ZGBoZh0K1bNzp06MCiRYuq/JxDhw7RsmVLZs2axR133FHlMnv27KF9+/bccsstzJ492z8/Pz+fjh07MnjwYBYsWFCtbYlI4NEpMBFpFAoKCvjqq6+4+uqr/eEHICgoiAkTJvDLL7+wdetWAPr168cnn3zC/fffz8qVKykqKqq0rbi4OM4++2z+/Oc/89e//pUNGzbg9XorLbNkyRLKysqYOHEiZWVl/iksLIwLL7yQlStXVntbIhJ4FIBEpFE4cuQIpmmSnJx8wnutWrUC8J/imj17Nvfddx/vv/8+F110EXFxcVxxxRVs27YNAMMw+M9//sPIkSN5+umn6d27N/Hx8dx+++3k5eUBsH//fgD69u1LSEhIpWnBggVkZ2dXe1siEnh0DZCINAqxsbHYbDYyMzNPeG/fvn0AtGzZEoDIyEimT5/O9OnT2b9/v/9o0OjRo/nxxx8BaNOmDS+//DIAP/30E2+//TaPPPIIJSUlvPjii/5tvfvuu7Rp0+aUtZ1uWyISeHQNkIgEjNNdAzRw4EB27tzJjh07CA8PB8Dr9dKzZ0+OHDnivwaoKnfddRezZs2ioKCAiIiIKpfp1asXISEhfP311+zatYuOHTvy+OOPc++9957xvhy7LREJPDoCJCIBZ8WKFezateuE+TNmzGD48OFcdNFF3HPPPYSGhvL888/z3XffMW/ePH/46d+/P5dffjndu3cnNjaWLVu28Prrr5Oenk5ERATffvstt956K9dccw0dO3YkNDSUFStW8O2333L//fcD0LZtW/70pz/x4IMPsmPHDkaNGkVsbCz79+/n66+/9h9lqs62RCQAmSIiAWLOnDkmcNJp586d5meffWZefPHFZmRkpBkeHm4OGDDA/Ne//lVpO/fff7+ZlpZmxsbGmna73Wzfvr151113mdnZ2aZpmub+/fvNSZMmmZ07dzYjIyPNqKgos3v37ubf/vY3s6ysrNK23n//ffOiiy4yHQ6HabfbzTZt2phXX321uXz58jPelogEDp0CExERkWZHd4GJiIhIs6MAJCIiIs2OApCIiIg0OwpAIiIi0uwoAImIiEizEzABaMaMGRiGwZ133nnSZd577z2GDx9OfHw8DoeD9PR0lixZUmmZuXPnYhjGCVNxcXE974GIiIg0FgExEOK6det46aWX6N69+ymXW716NcOHD+eJJ54gJiaGOXPmMHr0aL766it69erlX87hcPibIlYICwurdj1er5d9+/YRHR190lFlRUREJLCYpkleXh6tWrXCZjvNMR6LxyEy8/LyzI4dO5rLli0zL7zwQvOOO+44o/W7dOliTp8+3f96zpw5ptPprFVNe/bsOeVgbJo0adKkSZOmwJ327Nlz2r/1lh8Bmjp1KpdddhnDhg3jscceO6N1vV4veXl5xMXFVZqfn59PmzZt8Hg89OzZk0cffbTSEaLjud1u3G63/7VZPjbknj17cDgcZ1STiIiIWMPlcpGamkp0dPRpl7U0AM2fP59vvvmGdevW1Wj9mTNnUlBQwLhx4/zzOnfuzNy5c+nWrRsul4tnnnmGQYMGsWnTJjp27FjldmbMmMH06dNPmO9wOBSAREREGpnqXL5iWSuMPXv2kJaWxtKlS+nRowcAQ4YMoWfPnsyaNeu068+bN4+bbrqJDz74gGHDhp10Oa/XS+/evRk8eDCzZ8+ucpnjjwBVJMjc3FwFIBERkUbC5XLhdDqr9ffbsiNA//3vfzlw4AB9+vTxz/N4PKxevZpnn30Wt9tNUFBQlesuWLCAyZMn884775wy/ADYbDb69u3Ltm3bTrqM3W7HbrfXbEdERESk0bEsAA0dOpTNmzdXmnfDDTfQuXNn7rvvvpOGn3nz5nHjjTcyb948LrvsstN+jmmabNy4kW7dutVJ3SIiItL4WRaAoqOjOe+88yrNi4yMpEWLFv7506ZNY+/evbz22muAL/xMnDiRZ555hgEDBpCVlQVAeHg4TqcTgOnTpzNgwAA6duyIy+Vi9uzZbNy4keeee64B905EpHkzTZOysjI8Ho/VpUgTExISctKDJGfC8rvATiUzM5OMjAz/67///e+UlZUxdepUpk6d6p9//fXXM3fuXABycnK4+eabycrKwul00qtXL1avXk2/fv0aunwRkWappKSEzMxMCgsLrS5FmiDDMEhJSSEqKqp227HqIuhAdiYXUYmIyFFer5dt27YRFBREfHw8oaGhGlBW6oxpmhw8eJDCwkI6dux4wpGgRnERtIiIND0lJSV4vV5SU1OJiIiwuhxpguLj49m1axelpaW1OhUWML3ARESk6ThtGwKRGqqrI4r6hYqIiEizowAkIiIizY4CkIiISD0YMmQId955Z7WX37VrF4ZhsHHjxnqrSY5SABIRkWbNMIxTTpMmTarRdt977z0effTRai+fmppKZmbmCWPk1TUFLR/dBdaAdmUX8NbXGUQFe7l9RFeryxEREXxjzlVYsGABf/jDH9i6dat/Xnh4eKXlS0tLCQkJOe124+LizqiOoKAgkpKSzmgdqTkdAWpA7p9Xc/tXQ7h07f9YXYqISIMwTZPCkjJLpuoOc5eUlOSfnE4nhmH4XxcXFxMTE8Pbb7/NkCFDCAsL44033uDQoUP8+te/JiUlhYiICLp168a8efMqbff4U2Bt27bliSee4MYbbyQ6OprWrVvz0ksv+d8//sjMypUrMQyD//znP6SlpREREcHAgQMrhTOAxx57jISEBKKjo7npppu4//776dmzZ43+9wJfg/Dbb7+dhIQEwsLCOP/881m3bp3//SNHjnDdddcRHx9PeHg4HTt2ZM6cOYBvGIRbb72V5ORkwsLCaNu2LTNmzKhxLfVJR4AaULQzjiijGKf3CKZpanAwEWnyiko9dPnDEks++4c/jSQitG7+zN13333MnDmTOXPmYLfbKS4upk+fPtx33304HA4++ugjJkyYQPv27enfv/9JtzNz5kweffRRHnjgAd59911uueUWBg8eTOfOnU+6zoMPPsjMmTOJj49nypQp3HjjjXz++ecAvPnmmzz++OM8//zzDBo0iPnz5zNz5kzatWtX43299957WbhwIa+++ipt2rTh6aefZuTIkWzfvp24uDgefvhhfvjhBz755BNatmzJ9u3bKSoqAmD27Nl8+OGHvP3227Ru3Zo9e/awZ8+eGtdSnxSAGlBMfAoAcbgoKC4hKlwd6EVEGoM777yTsWPHVpp3zz33+J/fdtttLF68mHfeeeeUAejSSy/ld7/7HeALVX/7299YuXLlKQPQ448/zoUXXgjA/fffz2WXXUZxcTFhYWH83//9H5MnT+aGG24A4A9/+ANLly4lPz+/RvtZUFDACy+8wNy5c7nkkksA+Mc//sGyZct4+eWX+f3vf09GRga9evUiLS0N8B3ZqpCRkUHHjh05//zzMQyDNm3a1KiOhqAA1IAiYhLxmgZBhsnhg5lEtW5rdUkiIvUqPCSIH/400rLPrisVf+wreDwennzySRYsWMDevXtxu9243W4iIyNPuZ3u3bv7n1ecajtw4EC110lOTgbgwIEDtG7dmq1bt/oDVYV+/fqxYsWKau3X8X7++WdKS0sZNGiQf15ISAj9+vVjy5YtANxyyy1cddVVfPPNN4wYMYIrrriCgQMHAjBp0iSGDx9Op06dGDVqFJdffjkjRoyoUS31TdcANaSgYHIMX2+SvOxfLC5GRKT+GYZBRGiwJVNdXmZwfLCZOXMmf/vb37j33ntZsWIFGzduZOTIkZSUlJxyO8dfPG0YBl6vt9rrVOzTsescv5+1afFZsW5V26yYd8kll7B7927uvPNO9u3bx9ChQ/1Hw3r37s3OnTt59NFHKSoqYty4cVx99dU1rqc+KQA1MFdQLACFhzNPs6SIiASqzz77jDFjxvCb3/yGHj160L59e7Zt29bgdXTq1Imvv/660rz169fXeHsdOnQgNDSUNWvW+OeVlpayfv16zj33XP+8+Ph4Jk2axBtvvMGsWbMqXcztcDgYP348//jHP1iwYAELFy7k8OHDNa6pvugUWAMrDG0BRbsoyc2yuhQREamhDh06sHDhQr744gtiY2P561//SlZWVqWQ0BBuu+02fvvb35KWlsbAgQNZsGAB3377Le3btz/tusffTQbQpUsXbrnlFn7/+98TFxdH69atefrppyksLGTy5MmA7zqjPn360LVrV9xuN//+97/9+/23v/2N5ORkevbsic1m45133iEpKYmYmJg63e+6oADUwNxhLaEIPK5Tn/MVEZHA9fDDD7Nz505GjhxJREQEN998M1dccQW5ubkNWsd1113Hjh07uOeeeyguLmbcuHFMmjTphKNCVbn22mtPmLdz506efPJJvF4vEyZMIC8vj7S0NJYsWUJsrO8MRmhoKNOmTWPXrl2Eh4dzwQUXMH/+fACioqJ46qmn2LZtG0FBQfTt25ePP/44IJvjGmZtThY2US6XC6fTSW5uLg6Ho063vf6lqaTte4M18eM5f+pLp19BRKQRKS4uZufOnbRr146wsDCry2mWhg8fTlJSEq+//rrVpdSLU/3GzuTvt44ANTAjKgGAkKJsiysREZHGrrCwkBdffJGRI0cSFBTEvHnzWL58OcuWLbO6tICnANTAgp2+Yc7DSw5ZXImIiDR2hmHw8ccf89hjj+F2u+nUqRMLFy5k2LBhVpcW8BSAGlhYrC8ARZUG3hXxIiLSuISHh7N8+XKry2iUAu+qpCYuMq4VAE5vjrWFiIiINGMKQA0sJv4sAGLJo7C42OJqREREmicFoAYWGZOAxzSwGSaHD2gwRBERESsoADUwIyiYI0YMAC61wxAREbGEApAFKtphFB3RESARERErKABZoDA0DoCSHLXDEBERsYICkAVKwloC4Mnbb3ElIiJSV4YMGcKdd97pf922bVtmzZp1ynUMw+D999+v9WfX1XaaEwUgC5SFxwNgK1A/MBERq40ePfqkAweuXbsWwzD45ptvzni769at4+abb65teZU88sgj9OzZ84T5mZmZXHLJJXX6WcebO3duQDY1rSkFIAuoHYaISOCYPHkyK1asYPfu3Se898orr9CzZ0969+59xtuNj48nIiKiLko8raSkJOx2e4N8VlOhAGQBtcMQkWbDNKGkwJqpmr2+L7/8chISEpg7d26l+YWFhSxYsIDJkydz6NAhfv3rX5OSkkJERATdunVj3rx5p9zu8afAtm3bxuDBgwkLC6NLly5V9uu67777OOecc4iIiKB9+/Y8/PDDlJaWAr4jMNOnT2fTpk0YhoFhGP6ajz8FtnnzZi6++GLCw8Np0aIFN998M/n5+f73J02axBVXXMFf/vIXkpOTadGiBVOnTvV/Vk1kZGQwZswYoqKicDgcjBs3jv37j17qsWnTJi666CKio6NxOBz06dOH9evXA7B7925Gjx5NbGwskZGRdO3alY8//rjGtVSHWmFYICymvB1GmdphiEgTV1oIT7Sy5rMf2AehkaddLDg4mIkTJzJ37lz+8Ic/YBgGAO+88w4lJSVcd911FBYW0qdPH+677z4cDgcfffQREyZMoH379vTv3/+0n+H1ehk7diwtW7bkyy+/xOVyVbpeqEJ0dDRz586lVatWbN68md/+9rdER0dz7733Mn78eL777jsWL17sb3/hdDpP2EZhYSGjRo1iwIABrFu3jgMHDnDTTTdx6623Vgp5n376KcnJyXz66ads376d8ePH07NnT37729+edn+OZ5omV1xxBZGRkaxatYqysjJ+97vfMX78eFauXAnAddddR69evXjhhRcICgpi48aNhISEADB16lRKSkpYvXo1kZGR/PDDD0RFRZ1xHWdCAcgCkS18o0HHeI5YXImIiADceOON/PnPf2blypVcdNFFgO/019ixY4mNjSU2NpZ77rnHv/xtt93G4sWLeeedd6oVgJYvX86WLVvYtWsXKSkpADzxxBMnXLfz0EMP+Z+3bduW//3f/2XBggXce++9hIeHExUVRXBwMElJSSf9rDfffJOioiJee+01IiN9AfDZZ59l9OjRPPXUUyQmJgIQGxvLs88+S1BQEJ07d+ayyy7jP//5T40C0PLly/n222/ZuXMnqampALz++ut07dqVdevW0bdvXzIyMvj9739P586dAejYsaN//YyMDK666iq6desGQPv27c+4hjOlAGSBinYYMUY+xcXFhIWFWVyRiEg9CYnwHYmx6rOrqXPnzgwcOJBXXnmFiy66iJ9//pnPPvuMpUuXAuDxeHjyySdZsGABe/fuxe1243a7/QHjdLZs2ULr1q394QcgPT39hOXeffddZs2axfbt28nPz6esrAyHw1Ht/aj4rB49elSqbdCgQXi9XrZu3eoPQF27diUoKMi/THJyMps3bz6jzzr2M1NTU/3hB6BLly7ExMSwZcsW+vbty913381NN93E66+/zrBhw7jmmms4++yzAbj99tu55ZZbWLp0KcOGDeOqq66ie/fuNaqlunQNkAWiY+MpM31f/eGDey2uRkSkHhmG7zSUFVP5qazqmjx5MgsXLsTlcjFnzhzatGnD0KFDAZg5cyZ/+9vfuPfee1mxYgUbN25k5MiRlJSUVGvbZhXXIxnH1ffll19y7bXXcskll/Dvf/+bDRs28OCDD1b7M479rOO3XdVnVpx+OvY9r9d7Rp91us88dv4jjzzC999/z2WXXcaKFSvo0qULixYtAuCmm25ix44dTJgwgc2bN5OWlsb//d//1aiW6lIAsoBhC/K3w8jLtuhfRiIiUsm4ceMICgrirbfe4tVXX+WGG27w//H+7LPPGDNmDL/5zW/o0aMH7du3Z9u2bdXedpcuXcjIyGDfvqP/zV+7dm2lZT7//HPatGnDgw8+SFpaGh07djzhzrTQ0FA8Hs9pP2vjxo0UFBRU2rbNZuOcc86pds1nomL/9uzZ45/3ww8/kJuby7nnnuufd84553DXXXexdOlSxo4dy5w5c/zvpaamMmXKFN577z3+93//l3/84x/1UmsFBSCLVLTDKDysACQiEgiioqIYP348DzzwAPv27WPSpEn+9zp06MCyZcv44osv2LJlC//v//0/srKqP5r/sGHD6NSpExMnTmTTpk189tlnPPjgg5WW6dChAxkZGcyfP5+ff/6Z2bNn+4+QVGjbti07d+5k48aNZGdn43a7T/is6667jrCwMK6//nq+++47Pv30U2677TYmTJjgP/1VUx6Ph40bN1aafvjhB4YNG0b37t257rrr+Oabb/j666+ZOHEiF154IWlpaRQVFXHrrbeycuVKdu/ezeeff866dev84ejOO+9kyZIl7Ny5k2+++YYVK1ZUCk71QQHIIgWhLQC1wxARCSSTJ0/myJEjDBs2jNatW/vnP/zww/Tu3ZuRI0cyZMgQkpKSuOKKK6q9XZvNxqJFi3C73fTr14+bbrqJxx9/vNIyY8aM4a677uLWW2+lZ8+efPHFFzz88MOVlrnqqqsYNWoUF110EfHx8VXeih8REcGSJUs4fPgwffv25eqrr2bo0KE8++yzZ/ZlVCE/P59evXpVmi699FL/bfixsbEMHjyYYcOG0b59exYsWABAUFAQhw4dYuLEiZxzzjmMGzeOSy65hOnTpwO+YDV16lTOPfdcRo0aRadOnXj++edrXe+pGGZVJyabOZfLhdPpJDc394wvPquudbOupW/OJ3zRdioDJz1RL58hItLQiouL2blzJ+3atdMNHlIvTvUbO5O/3wFzBGjGjBkYhlHluAjHWrVqFX369CEsLIz27dvz4osvnrDMwoUL6dKlC3a7vdJFVoHEE6F2GCIiIlYJiAC0bt06XnrppdPe8rZz504uvfRSLrjgAjZs2MADDzzA7bffzsKFC/3LrF27lvHjxzNhwgQ2bdrEhAkTGDduHF999VV978aZKW+HEax2GCIiIg3O8gCUn5/Pddddxz/+8Q9iY2NPueyLL75I69atmTVrFueeey433XQTN954I3/5y1/8y8yaNYvhw4czbdo0OnfuzLRp0xg6dOhpO/I2tKPtMBSAREREGprlAWjq1KlcdtllJ+3Ee6y1a9cyYsSISvNGjhzJ+vXr/f1LTrbMF198cdLtut1uXC5Xpam+hcckAxBVptGgRUREGpqlAWj+/Pl88803zJgxo1rLZ2VlnXALX2JiImVlZWRnZ59ymVPdrjhjxgycTqd/OnYky/oS1cLXGyfWqwAkIk2P7q+R+lJXvy3LAtCePXu44447eOONN87oToHjR5qs+CKOnV/VMicbFRNg2rRp5Obm+qdjB3KqLxXtMBwU4C4urPfPExFpCBWjCxcW6r9rUj8qRsY+to1HTVjWC+y///0vBw4coE+fPv55Ho+H1atX8+yzz+J2u0/YuaSkpBOO5Bw4cIDg4GBatGhxymVONfiT3W7HbrfXdpfOiCM2nhIziFDDw5GD+0hK7dCgny8iUh+CgoKIiYnhwAHfHa4RERGn/AeoyJnwer0cPHiQiIgIgoNrF2EsC0BDhw49oenaDTfcQOfOnbnvvvuqTHbp6en861//qjRv6dKlpKWl+f/VkZ6ezrJly7jrrrsqLTNw4MB62IuaM2w2jhgxJHKIvOy9CkAi0mRUdCqvCEEidclms9G6detaB2vLAlB0dDTnnXdepXmRkZG0aNHCP3/atGns3buX1157DYApU6bw7LPPcvfdd/Pb3/6WtWvX8vLLL1caCfOOO+5g8ODBPPXUU4wZM4YPPviA5cuXs2bNmobbuWpyBcWS6DlE4eFMq0sREakzhmGQnJxMQkKC/wYVkboSGhqKzVb7K3gsC0DVkZmZSUZGhv91u3bt+Pjjj7nrrrt47rnnaNWqFbNnz+aqq67yLzNw4EDmz5/PQw89xMMPP8zZZ5/NggUL6N+/vxW7cEoFoS2gaDslOQpAItL0BAUF1fo6DZH6olYYVWiIVhgAX836Nf1zPubLtr9jwKTq3QknIiIiVWuUrTCaI0+EbzRotcMQERFpWApAFjKifP3Agos1GrSIiEhDUgCykL8dhlsBSEREpCEpAFkoTO0wRERELKEAZKHIFr7RoGO8hy2uREREpHlRALJQTEtfP7Boiih1a9h4ERGRhqIAZKGY2Ja4Td8I1kcO7LW4GhERkeZDAchCtiAbhw0nAHnZCkAiIiINRQHIYq6gOAAK1A5DRESkwSgAWawg1BeA1A5DRESk4SgAWazE3gIAb55GgxYREWkoCkAWq2iHYRTst7gSERGR5kMByGpRvgAUonYYIiIiDUYByGLBzkQAwt2HLK5ERESk+VAAstjRdhgaDVpERKShKABZ7Gg7DPUDExERaSgKQBaLaekLQJEUU1aUZ3E1IiIizYMCkMViY+MoMkMByDmo0aBFREQaggKQxYKCbBw2YgBwHdpnbTEiIiLNhAJQAHAFxQJQeDjL4kpERESaBwWgAFAQonYYIiIiDUkBKACUhMUD4M3TaNAiIiINQQEoAHgiWgJgFKgfmIiISENQAAoEaochIiLSoBSAAkCwIwlQOwwREZGGogAUAMJife0wItUOQ0REpEEoAAWAyBa+ABSrdhgiIiINQgEoAMS2TAEgHDeeYrXDEBERqW8KQAEgNjaWAtMOQO7BXyyuRkREpOlTAAoAwce0w8jLVjsMERGR+qYAFCDUDkNERKThKAAFiIKQFgC4c3UESEREpL4pAAUId5hvNGiPS6NBi4iI1DcFoADhCff1A7OpHYaIiEi9UwAKFNG+dhjBaochIiJS7xSAAkSwIxGAcLcCkIiISH2zNAC98MILdO/eHYfDgcPhID09nU8++eSky0+aNAnDME6Yunbt6l9m7ty5VS5TXFzcELtUY+Hl7TCiyjQatIiISH0LtvLDU1JSePLJJ+nQoQMAr776KmPGjGHDhg2VQk2FZ555hieffNL/uqysjB49enDNNddUWs7hcLB169ZK88LCwuphD+pOVFwrAGK8R8A0wTAsrkhERKTpsjQAjR49utLrxx9/nBdeeIEvv/yyygDkdDpxOp3+1++//z5HjhzhhhtuqLScYRgkJSXVT9H1xBl/FgBhlOAtdmELd55mDREREampgLkGyOPxMH/+fAoKCkhPT6/WOi+//DLDhg2jTZs2lebn5+fTpk0bUlJSuPzyy9mwYcMpt+N2u3G5XJWmhtYiNoY8MxwAl0aDFhERqVeWB6DNmzcTFRWF3W5nypQpLFq0iC5dupx2vczMTD755BNuuummSvM7d+7M3Llz+fDDD5k3bx5hYWEMGjSIbdu2nXRbM2bM8B9dcjqdpKam1nq/zlRIkI0jhu+oT1723gb/fBERkebEME3TtLKAkpISMjIyyMnJYeHChfzzn/9k1apVpw1BM2bMYObMmezbt4/Q0NCTLuf1eunduzeDBw9m9uzZVS7jdrtxu93+1y6Xi9TUVHJzc3E4HDXbsRrY/Gg63Tw/sPWC/6PT0IkN9rkiIiJNgcvlwul0Vuvvt6XXAAGEhob6L4JOS0tj3bp1PPPMM/z9738/6TqmafLKK68wYcKEU4YfAJvNRt++fU95BMhut2O322u2A3WoMCQOPODOVT8wERGR+mT5KbDjmaZZ6WhMVVatWsX27duZPHlytba3ceNGkpOT66rEelNsL2+HkafRoEVEROqTpUeAHnjgAS655BJSU1PJy8tj/vz5rFy5ksWLFwMwbdo09u7dy2uvvVZpvZdffpn+/ftz3nnnnbDN6dOnM2DAADp27IjL5WL27Nls3LiR5557rkH2qTY8EfGQq3YYIiIi9c3SALR//34mTJhAZmYmTqeT7t27s3jxYoYPHw74LnTOyMiotE5ubi4LFy7kmWeeqXKbOTk53HzzzWRlZeF0OunVqxerV6+mX79+9b4/tWVEJ0AmBBcdtLoUERGRJs3yi6AD0ZlcRFWXPvv3q1yw/nZ2hHai/QNfN9jnioiINAVn8vc74K4Bas7CYiraYRy2uBIREZGmTQEogEQe3w5DRERE6oUCUABxJvjaYYRShlmcY20xIiIiTZgCUABp4XTgMiMAyFM7DBERkXqjABRAwkKCOFzeDsOldhgiIiL1RgEowOQGxQFQeDjT4kpERESaLgWgAFMY4gtAJWqHISIiUm8UgAKM298OY7/FlYiIiDRdCkABxhMRD4CRr3YYIiIi9UUBKNBEJQAQUpxtcSEiIiJNlwJQgAl2JAEQ7j5kcSUiIiJNlwJQgAmLVTsMERGR+qYAFGAi43wByOnNUTsMERGReqIAFGCcLX39wEIowyw6YnE1IiIiTZMCUICJj3WQY0YCkH9I7TBERETqgwJQgAkLCeIwMYDaYYiIiNQXBaAAlBsUC0DRYR0BEhERqQ8KQAGooh2GO1ejQYuIiNQHBaAA5A7ztcPwutQPTEREpD4oAAUgfzuMgoMWVyIiItI0KQAFoqhEQO0wRERE6osCUAAKdvgCUJhbAUhERKQ+KAAFILu/HYYGQhQREakPCkABKCrONxq003sEvF6LqxEREWl6FIACUEx5O4xgvJhFaooqIiJS1xSAAlDLmEgOm1EAFB3JtLgaERGRpkcBKABFhAZzqKIdxkG1wxAREalrCkABylXeDqNA7TBERETqnAJQgCoob4dRkqPRoEVEROqaAlCA8rfDyFM/MBERkbqmABSgKtphoHYYIiIidU4BKFD522EoAImIiNQ1BaAAFRLtC0Dh7kMWVyIiItL0KAAFqLC4inYYGghRRESkrikABaiKdhgOby54PRZXIyIi0rQoAAUoZ8tkvKZBEF4o1FEgERGRuqQAFKBaOiM5TDQARTkaDFFERKQuWRqAXnjhBbp3747D4cDhcJCens4nn3xy0uVXrlyJYRgnTD/++GOl5RYuXEiXLl2w2+106dKFRYsW1feu1LnI0CB/O4y8gwpAIiIidcnSAJSSksKTTz7J+vXrWb9+PRdffDFjxozh+++/P+V6W7duJTMz0z917NjR/97atWsZP348EyZMYNOmTUyYMIFx48bx1Vdf1ffu1CnDMNQOQ0REpJ4YpmmaVhdxrLi4OP785z8zefLkE95buXIlF110EUeOHCEmJqbK9cePH4/L5ap0JGnUqFHExsYyb968Ktdxu9243W7/a5fLRWpqKrm5uTgcjtrtUC2snHEFQ9yfsrX7vXQa+6BldYiIiDQGLpcLp9NZrb/fAXMNkMfjYf78+RQUFJCenn7KZXv16kVycjJDhw7l008/rfTe2rVrGTFiRKV5I0eO5Isvvjjp9mbMmIHT6fRPqampNd+ROuS2+9phePLUD0xERKQuWR6ANm/eTFRUFHa7nSlTprBo0SK6dOlS5bLJycm89NJLLFy4kPfee49OnToxdOhQVq9e7V8mKyuLxMTESuslJiaSlXXyEDFt2jRyc3P90549e+pm52rJE+ELQEa+RoMWERGpS8FWF9CpUyc2btxITk4OCxcu5Prrr2fVqlVVhqBOnTrRqVMn/+v09HT27NnDX/7yFwYPHuyfbxhGpfVM0zxh3rHsdjt2u70O9qaOVbTDKFIAEhERqUuWHwEKDQ2lQ4cOpKWlMWPGDHr06MEzzzxT7fUHDBjAtm3b/K+TkpJOONpz4MCBE44KNQYhDl/NYSUaB0hERKQuWR6AjmeaZqULkk9nw4YNJCcn+1+np6ezbNmySsssXbqUgQMH1lmNDcUe6xsNWu0wRERE6palp8AeeOABLrnkElJTU8nLy2P+/PmsXLmSxYsXA75rc/bu3ctrr70GwKxZs2jbti1du3alpKSEN954g4ULF7Jw4UL/Nu+44w4GDx7MU089xZgxY/jggw9Yvnw5a9assWQfayMyLgkob4fhKYMgy89YioiINAmW/kXdv38/EyZMIDMzE6fTSffu3Vm8eDHDhw8HIDMzk4yMDP/yJSUl3HPPPezdu5fw8HC6du3KRx99xKWXXupfZuDAgcyfP5+HHnqIhx9+mLPPPpsFCxbQv3//Bt+/2oppmYzHNAgyTCg8BNGN7zSeiIhIIAq4cYACwZmMI1CfcotKKXnybOKNXNyTV2FP7WlZLSIiIoGuUY4DJCdyhAWTXd4Ow5W919piREREmhAFoAB2bDuMQrXDEBERqTMKQAGuIKQFAO7c/RZXIiIi0nQoAAU4t90XgLwutcMQERGpKwpAAc4TEQ+AUaDRoEVEROqKAlCgi0oA1A5DRESkLikABbhgfzuMQxZXIiIi0nQoAAW4sFhfm4+osiMWVyIiItJ0KAAFuMg4Xz8wXzuMUourERERaRoUgAJcbMtEyszy/5kKsq0tRkREpIlQAApwLaPDOYRvOG93TqbF1YiIiDQNCkABzhke4m+HkXdIo0GLiIjUBQWgAGcYBi6b2mGIiIjUJQWgRiA/1DcadEmuRoMWERGpCwpAjUCJvx2G+oGJiIjUBQWgRqCsvB0GaochIiJSJxSAGgFD7TBERETqlAJQIxDsSALUDkNERKSuKAA1AmExvgAUXaoAJCIiUhcUgBqBqBa+dhhRZj6UlVhcjYiISOOnANQIxLRIoMQM8r3QhdAiIiK1VqMAtGfPHn755Rf/66+//po777yTl156qc4Kk6N87TCcAJS5NBaQiIhIbdUoAP3P//wPn376KQBZWVkMHz6cr7/+mgceeIA//elPdVqgQEx4CNmmLwDlZWs0aBERkdqqUQD67rvv6NevHwBvv/025513Hl988QVvvfUWc+fOrcv6BLDZDHKDfO0wCtQOQ0REpNZqFIBKS0ux2+0ALF++nF/96lcAdO7cmcxMdSyvDwUh5e0wcnQKTEREpLZqFIC6du3Kiy++yGeffcayZcsYNWoUAPv27aNFixZ1WqD4uMN836snTwFIRESktmoUgJ566in+/ve/M2TIEH7961/To0cPAD788EP/qTGpW55wXzsMQ3eBiYiI1FpwTVYaMmQI2dnZuFwuYmNj/fNvvvlmIiIi6qw4OcqISoD9aochIiJSF2p0BKioqAi32+0PP7t372bWrFls3bqVhISEOi1QfILUDkNERKTO1CgAjRkzhtdeew2AnJwc+vfvz8yZM7niiit44YUX6rRA8QmPTQYgqvSwxZWIiIg0fjUKQN988w0XXHABAO+++y6JiYns3r2b1157jdmzZ9dpgeITGedrhxFpFkBpscXViIiING41CkCFhYVER0cDsHTpUsaOHYvNZmPAgAHs3r27TgsUn9gWLXGb5Zds6UJoERGRWqlRAOrQoQPvv/8+e/bsYcmSJYwYMQKAAwcO4HA46rRA8WkZHUa22mGIiIjUiRoFoD/84Q/cc889tG3bln79+pGeng74jgb16tWrTgsUn9iIUH87jPxDGg1aRESkNmp0G/zVV1/N+eefT2Zmpn8MIIChQ4dy5ZVX1llxclRQRTsMEwoP7yPG6oJEREQasRoFIICkpCSSkpL45ZdfMAyDs846S4Mg1rOCkBZQAu7c/VaXIiIi0qjV6BSY1+vlT3/6E06nkzZt2tC6dWtiYmJ49NFH8Xq91d7OCy+8QPfu3XE4HDgcDtLT0/nkk09Ouvx7773H8OHDiY+P9y+/ZMmSSsvMnTsXwzBOmIqLG/+dUyXl7TC8ugZIRESkVmp0BOjBBx/k5Zdf5sknn2TQoEGYpsnnn3/OI488QnFxMY8//ni1tpOSksKTTz5Jhw4dAHj11VcZM2YMGzZsoGvXricsv3r1aoYPH84TTzxBTEwMc+bMYfTo0Xz11VeVrj1yOBxs3bq10rphYWE12dWAUhaeAC50F5iIiEgt1SgAvfrqq/zzn//0d4EH6NGjB2eddRa/+93vqh2ARo8eXen1448/zgsvvMCXX35ZZQCaNWtWpddPPPEEH3zwAf/6178qBSDDMEhKSjqDPWokouLVDkNERKQO1OgU2OHDh+ncufMJ8zt37szhwzUbqdjj8TB//nwKCgr8d5WdjtfrJS8vj7i4uErz8/PzadOmDSkpKVx++eVs2LDhlNtxu924XK5KUyAKrmiH4VY7DBERkdqoUQDq0aMHzz777Anzn332Wbp3735G29q8eTNRUVHY7XamTJnCokWL6NKlS7XWnTlzJgUFBYwbN84/r3PnzsydO5cPP/yQefPmERYWxqBBg9i2bdtJtzNjxgycTqd/Sk1NPaN9aChhMb52GNFlaochIiJSG4ZpmuaZrrRq1Souu+wyWrduTXp6OoZh8MUXX7Bnzx4+/vhjf5uM6igpKSEjI4OcnBwWLlzIP//5T1atWnXaEDRv3jxuuukmPvjgA4YNG3bS5bxeL71792bw4MEnbdPhdrtxu93+1y6Xi9TUVHJzcwNqYMc13+/g/HfKT/U9mAUh4dYWJCIiEkBcLhdOp7Naf79rdATowgsv5KeffuLKK68kJyeHw4cPM3bsWL7//nvmzJlzRtsKDQ2lQ4cOpKWlMWPGDHr06MEzzzxzynUWLFjA5MmTefvtt08ZfgBsNht9+/Y95REgu93uvxOtYgpEsTFxFJshvhf5B6wtRkREpBGr8ThArVq1OuFi502bNvHqq6/yyiuv1Lgg0zQrHY053rx587jxxhuZN28el112WbW2t3HjRrp161bjmgJFfHk7jBSy8eTtJyi2jdUliYiINEo1DkB14YEHHuCSSy4hNTWVvLw85s+fz8qVK1m8eDEA06ZNY+/evbz22muAL/xMnDiRZ555hgEDBpCV5RsPJzw8HKfT1yZi+vTpDBgwgI4dO+JyuZg9ezYbN27kueees2Yn61BcZCibzRhSjGzyD+3D2drqikRERBonSwPQ/v37mTBhApmZmTidTrp3787ixYsZPnw4AJmZmWRkZPiX//vf/05ZWRlTp05l6tSp/vnXX389c+fOBSAnJ4ebb76ZrKwsnE4nvXr1YvXq1U1ilOrgIBu5thjA1w7DaW05IiIijVaNLoI+mU2bNtG7d288Hk9dbdISZ3IRVUP76IlxXFayhN3d76DN2D9ZXY6IiEjAOJO/32d0BGjs2LGnfD8nJ+dMNic14Lb7+oF51A5DRESkxs4oAFVcZ3Oq9ydOnFirguTUPBEJkAeG2mGIiIjU2BkFoDO9xV3qQVSC2mGIiIjUUo3GARLrHG2HodGgRUREakoBqJGxx/gCUFSZ+oGJiIjUlAJQIxPVohUAYWYxuPMtrkZERKRxUgBqZOJi4yg07b4XBWqHISIiUhMKQI1Myyg7B03f3XjePAUgERGRmlAAamRaRIWSXT4GdMHhfRZXIyIi0jgpADUyIZXaYWRaW4yIiEgjpQDUCBWEtADAnaPRoEVERGpCAagRctt9AciTpwAkIiJSEwpAjVBZZAKgdhgiIiI1pQDUCBnlAUjtMERERGpGAagRCipvhxFZvB9M0+JqREREGh8FoEYoKKEzxWYIMWUHYe83VpcjIiLS6CgANULOuBZ84u3ne7HhNWuLERERaYQUgBqhhOgw3vYMAcDcvBBKCqwtSEREpJFRAGqEOiVFsz28B7u8iRglefD9+1aXJCIi0qgoADVCIUE2ruzTmrc9F/pmbHjd2oJEREQaGQWgRuqaPiks9AzGYxqQsRayt1ldkoiISKOhANRIdUyMplXr9qz09vTN0FEgERGRalMAasSu6ZPKgoqLoTfOA0+ptQWJiIg0EgpAjdjlPZL5IqgPB00nRsEB+GmJ1SWJiIg0CgpAjZgjLIQR56Wy0HOBb4ZOg4mIiFSLAlAjd01aKu+U3w1mblsKrkyLKxIREQl8CkCNXP92cZTGduRrbycM0wub3rK6JBERkYCnANTI2WwGV/dJ8Y8MzTevg9draU0iIiKBTgGoCbiqTwofe/uTZ4bDkZ2w+3OrSxIREQloCkBNwFkx4fTpkMK/PAN8M3QxtIiIyCkpADUR49JSWeC5CADzhw+gKMfagkRERAKYAlATMbxLIjvtnfjRm4pRVgzfvWt1SSIiIgFLAaiJCAsJ4opex14M/Zql9YiIiAQyBaAmZFxaKos8gygxgyBzE2R+a3VJIiIiAUkBqAnp2spBUnIKS71pvhm6GFpERKRKCkBNiGEYjEtL8V8MzbdvQ2mxtUWJiIgEIAWgJmZMz7P4yujGL2ZLKM6BH/9tdUkiIiIBx9IA9MILL9C9e3ccDgcOh4P09HQ++eSTU66zatUq+vTpQ1hYGO3bt+fFF188YZmFCxfSpUsX7HY7Xbp0YdGiRfW1CwEnLjKUYV2Sedcz2Dfjm1etLUhERCQAWRqAUlJSePLJJ1m/fj3r16/n4osvZsyYMXz//fdVLr9z504uvfRSLrjgAjZs2MADDzzA7bffzsKFC/3LrF27lvHjxzNhwgQ2bdrEhAkTGDduHF999VVD7ZblrklL5Z2yC/FiwM7VcHin1SWJiIgEFMM0TdPqIo4VFxfHn//8ZyZPnnzCe/fddx8ffvghW7Zs8c+bMmUKmzZtYu3atQCMHz8el8tV6UjSqFGjiI2NZd68eVV+ptvtxu12+1+7XC5SU1PJzc3F4XDU1a41GI/XZNCTK3i66I8MDtoMg38PFz9kdVkiIiL1yuVy4XQ6q/X3O2CuAfJ4PMyfP5+CggLS09OrXGbt2rWMGDGi0ryRI0eyfv16SktLT7nMF198cdLPnjFjBk6n0z+lpqbWcm+sFWQzGNv7rKMXQ294E7wea4sSEREJIJYHoM2bNxMVFYXdbmfKlCksWrSILl26VLlsVlYWiYmJleYlJiZSVlZGdnb2KZfJyso6aQ3Tpk0jNzfXP+3Zs6eWe2W9a9JSWebtw2EzCvL2wfb/WF2SiIhIwLA8AHXq1ImNGzfy5Zdfcsstt3D99dfzww8/nHR5wzAqva44g3fs/KqWOX7esex2u/9C7IqpsWvXMpKebRN533O+b8YGjQwtIiJSwfIAFBoaSocOHUhLS2PGjBn06NGDZ555psplk5KSTjiSc+DAAYKDg2nRosUplzn+qFBzcE1aCgvKW2OYWz+B/IPWFiQiIhIgLA9AxzNNs9IFycdKT09n2bJlleYtXbqUtLQ0QkJCTrnMwIED66fgAHZpt2R+CWnHRu/ZGN4y+Ha+1SWJiIgEBEsD0AMPPMBnn33Grl272Lx5Mw8++CArV67kuuuuA3zX5kycONG//JQpU9i9ezd33303W7Zs4ZVXXuHll1/mnnvu8S9zxx13sHTpUp566il+/PFHnnrqKZYvX86dd97Z0LtnuUh7MJd1T/YfBeKb1yGwbvoTERGxhKUBaP/+/UyYMIFOnToxdOhQvvrqKxYvXszw4cMByMzMJCMjw798u3bt+Pjjj1m5ciU9e/bk0UcfZfbs2Vx11VX+ZQYOHMj8+fOZM2cO3bt3Z+7cuSxYsID+/fs3+P4FgnFpqfzLk06haYfsrbDna6tLEhERsVzAjQMUCM5kHIFAZ5omQ2eu4pacmVwTvBp6/QbGPGd1WSIiInWuUY4DJPXDMAyuSUs9ehrsu0XgzrO0JhEREaspADUDY3ufxTd04mdvMpQWwPfNpzeaiIhIVRSAmoFERxhDOiXy9rEXQ4uIiDRjCkDNxLi0FN7zXEAZQfDL13DgR6tLEhERsYwCUDNxcedEPJEJ/MfTyzdjg44CiYhI86UA1EyEBtu4stdZRy+G3jQPykosrUlERMQqCkDNyDVpKazy9mC/GQOFh+CnT6wuSURExBIKQM1I5yQHXVPieNcz2DfjGzVIFRGR5kkBqJm5Ji3VfzeYuf0/kPuLtQWJiIhYQAGomflVj1ZkBbViracLBiZsfMvqkkRERBqcAlAz4wwPYWTXpKMXQ294HbxeS2sSERFpaApAzdC4tFQ+8fbDRQTkZMCu1VaXJCIi0qAUgJqhgWe3oGWMk/fLBvlm6GJoERFpZhSAmiGbzeDqPiks8Fzkm/HDB3Bwq7VFiYiINCAFoGbq6j4pfG+2ZamnD3jLYPH9YJpWlyUiItIgFICaqdS4CAae3YLHyn5DmRECP6+ArRoYUUREmgcFoGZsXFoqGWYib9pG+2YsmQalxdYWJSIi0gAUgJqxUeclkeQI46mCy8kPjYcju+DL56wuS0REpN4pADVjYSFBTLu0M4WE8WjxeN/M1TPBtc/awkREROqZAlAz96serejTJpYFJensDO8KpQWw/BGryxIREalXCkDNnGEYPDK6K4ZhcHvOrzEx4NsFsOdrq0sTERGpNwpAQrcUJ9f0SWGz2Z7l9mG+mZ/cqxYZIiLSZCkACQD3jOxElD2YabljKQ2Ogn0bYOObVpclIiJSLxSABICE6DBuu7gD2Th51nuVb+Z/pkNxrrWFiYiI1AMFIPG7YVA72rWM5PnCoRwKawMFB2HV01aXJSIiUucUgMQvNNjGQ5edSynB3Jv/a9/Mr16Egz9ZW5iIiEgdUwCSSi7unMDgc+L5T1l3NkUM8PUJWzJNfcJERKRJUQCSSgzD4A+Xn0uQzeCOI9fgtYXA9uXw0xKrSxMREakzCkBygg4J0UxMb8MuM5l3go/pE1bmtrYwERGROqIAJFW6c+g5xEWG8ifXZRSGtoTDO+DLF6wuS0REpE4oAEmVnBEh/O+IcyggnCdKxvlmrv4z5GVZW5iIiEgdUACSk7q2b2s6J0XzZvFA9kR0gZJ8WD7d6rJERERqTQFITirIZvDH0V0xsXF7zrW+mZvegl/WW1uYiIhILSkAySmln92CS7slscHbgZXhw30zP/69+oSJiEijpgAkpzXtknOxB9v4/ZErKQuOhH3fwKZ5VpclIiJSYwpAclqpcRH8v8HtOUgMf6e8T9jyR6DYZWldIiIiNWVpAJoxYwZ9+/YlOjqahIQErrjiCrZu3XrKdSZNmoRhGCdMXbt29S8zd+7cKpcpLi6u711qsqYMOZskRxiz8oeRE94aCg747goTERFphCwNQKtWrWLq1Kl8+eWXLFu2jLKyMkaMGEFBQcFJ13nmmWfIzMz0T3v27CEuLo5rrrmm0nIOh6PScpmZmYSFhdX3LjVZEaHBTLu0M6UEc39BeZ+wL1+A7O3WFiYiIlIDwVZ++OLFiyu9njNnDgkJCfz3v/9l8ODBVa7jdDpxOp3+1++//z5HjhzhhhtuqLScYRgkJSXVfdHN2K96tOK1tbtZvLsHP7ToT5eCr2DJA3Dd21aXJiIickYC6hqg3NxcAOLi4qq9zssvv8ywYcNo06ZNpfn5+fm0adOGlJQULr/8cjZs2HDSbbjdblwuV6VJTmQYBo+M7ophwK2Hr8E0gmHbEvhpqdWliYiInJGACUCmaXL33Xdz/vnnc95551VrnczMTD755BNuuummSvM7d+7M3Llz+fDDD5k3bx5hYWEMGjSIbdu2VbmdGTNm+I8sOZ1OUlNTa70/TVW3FCfX9Elhh9mK98OO7RNWYm1hIiIiZ8AwTdO0ugiAqVOn8tFHH7FmzRpSUlKqtc6MGTOYOXMm+/btIzQ09KTLeb1eevfuzeDBg5k9e/YJ77vdbtzuo40+XS4Xqamp5Obm4nA4znxnmrgDecVc/JdVGG4X6xz3EVZyCIY/CoNut7o0ERFpxlwuF06ns1p/vwPiCNBtt93Ghx9+yKefflrt8GOaJq+88goTJkw4ZfgBsNls9O3b96RHgOx2Ow6Ho9IkJ5cQHcZtF3cgjwie9pSPEL3qacjbb21hIiIi1WRpADJNk1tvvZX33nuPFStW0K5du2qvu2rVKrZv387kyZOr9TkbN24kOTm5NuXKMSYNakvbFhHMKUhnX2QXKMmDZQ9DYBxQFBEROSVLA9DUqVN54403eOutt4iOjiYrK4usrCyKior8y0ybNo2JEyeesO7LL79M//79q7xeaPr06SxZsoQdO3awceNGJk+ezMaNG5kyZUq97k9zYg8O4uHLu2Bi447c8tviv10AH98DXo+1xYmIiJyGpQHohRdeIDc3lyFDhpCcnOyfFixY4F8mMzOTjIyMSuvl5uaycOHCkx79ycnJ4eabb+bcc89lxIgR7N27l9WrV9OvX7963Z/m5uLOCQw+J551ZWfzRss7AAPW/RMWToYy92nXFxERsUrAXAQdSM7kIqrmbvuBPEbO+gyP1+STYQc594t7wFsK7YfA+DfAHm11iSIi0kw0uougpfHqkBDNxHTfGEw3f9OafZe/BiGRsGMlvDoaCrKtLVBERKQKCkBSa3cOPYezYsLZc7iIER8E8cUFcyGiBezbAK+MhJyM025DRESkISkASa05I0JYNHUg/drFke8u438+LuWfHV/AdKbAoe3w8gg4sMXqMkVERPwUgKROJESH8eZN/Zl8vm8og8e+KuOOiKcpa9kZ8jLhlVGQ8ZXFVYqIiPgoAEmdCQmy8fDlXZj9616EhwTx4U64PO8BChL6QHEOvDZGfcNERCQgKABJnftVj1a8P3UQbVtE8GNuMOn7bmdf/AVQVgTzroVN860uUUREmjkFIKkXnZKi+eDW8xl2bgKushAG7/ktG2JGgumBRf8P1j5ndYkiItKMKQBJvXGGh/DShDT+d/g5eIxgxmZN4IPwK3xvLnkAlj+i1hkiImIJBSCpVzabwW1DOzJnUl8c4XbuOHINs43rfG+u+Rt8eCt4yqwtUkREmh0FIGkQQzol8O/bzqdLspO/Fl3G/WU348UGG96AtydCadHpNyIiIlJHFICkwaTGRfDe7wYytvdZzC8bwpSSOyg1QmDrR/DGVVCca3WJIiLSTCgASYMKCwli5jU9eHRMVz41+vGb4vspIAJ2fw5zLoO8/VaXKCIizYACkDQ4wzCYkN6W+Tensyu6J9e4HyLbdML+zfDKCDi8w+oSRUSkiVMAEsv0aRPLv247n6i2vRlb8gi7vQlwZBfm3y/03SZfVmJ1iSIi0kQpAImlKlpoDB80gKtLHmGj92wMtwuWPID5QjpsXaxb5UVEpM4pAInlKlpoPPzri7jOfIx7S3/LQdOBcWg7zBuP97Ur1ExVRETqlAKQBIxf9WjF0v+9iOj0G7nMfIYXykbjNoOx7VyJ9/mBFL9/JxQcsrpMERFpAgzT1PmF47lcLpxOJ7m5uTgcDqvLaZZyi0qZ/3UGS9Z8yU3Fc7k06GsAimxRFA28h7ghUyE41OIqRUQkkJzJ328FoCooAAWOUo+Xj77NZO1/3mei6+90te0GICskhbzBj9Bh0FUYNh3IFBERBaBaUwAKPKZpsnbbAX5c/CKjD/2TeMMFwMaQXrgunM6ggYMJshkWVykiIlZSAKolBaDA9vOefWR88BiDDi4g1CjDYxp8GDKSokH3MWZgdyLtwVaXKCIiFlAAqiUFoMYhe8+PHF40jXMOrwDAZUbwonE1Rr+bmXh+RxIdYRZXKCIiDUkBqJYUgBoX97ZVFHx4L3F5PwKww5vEU97fENVtNDdfeDadkqItrlBERBqCAlAtKQA1Ql4P3g1vULp0Ona371b5r7ydebtsCAVnX8r1Q85jQPs4DEPXCYmINFUKQLWkANSIFbvgs5l41z6PzetrpVFg2lns7cfG2FGkD7uSkee10gXTIiJNkAJQLSkANQG5e2HTW5T+9y1Cco82V91rtmBF6EU4+k9g5IUXEBYSZGGRIiJSlxSAakkBqAkxTfhlPUXrX8f47j3CPHn+tzbTkeyzx9LrksnEtEy0sEgREakLCkC1pADURJUW4/7hIw58Npfk7DUE4wWgxAxmW+z5JJx/A/G9LoOgEIsLFRGRmlAAqiUFoKavLDeLH5e/QvgP73C25+gpsrygGNydx9Jy0PWQ3AN00bSISKOhAFRLCkDNh2mabFz/OVmrXyHNtZx4I9f/Xr7zHCL7TcA47ypwnmVhlSIiUh0KQLWkANQ8/fDLYT5bvICU3e8zzPZf7Eap/z1PbHuC2g+GthdAm0HgSLawUhERqYoCUC0pADVve3OKeGvlJvK/eZfLWUVvYxtBxnH/N2nRwReG2p7vm6KTrClWRET8FIBqSQFIAHILS3nz693855ufiMlezwDbFgbYfqCrsRvbCYGooy8ItbsA2pwP0bqrTESkoSkA1ZICkBxv96EClnyfxZLv97M9Yw99ja0MsP3AANsWuth2Y+O4/xu1PKf86FD5UaKoBGsKFxFpRhSAakkBSE7lgKuYpT/sZ8n3Waz9+RAR3jz6235kgG0LF4T8SEdzF8YJgagTtBkIqf0gpS/EnQ02mzU7ICLSRCkA1ZICkFRXbmEpK7buZ8l3+1n100GKSj04yae/bQtD7D8xxL6VVsXbT1wxLAZS0nxhKCUNzuoD4bENXr+ISFPSaALQjBkzeO+99/jxxx8JDw9n4MCBPPXUU3Tq1Omk66xcuZKLLrrohPlbtmyhc+fO/tcLFy7k4Ycf5ueff+bss8/m8ccf58orr6xWXQpAUhNFJR5WbzvIku+z+M+WA+QW+e4iiyGPC+3buDQmg67erSQX/EiQ133iBlqeczQQpfSF+HMhKLiB90JEpPFqNAFo1KhRXHvttfTt25eysjIefPBBNm/ezA8//EBkZGSV61QEoK1bt1baufj4eIKCfH2d1q5dywUXXMCjjz7KlVdeyaJFi/jDH/7AmjVr6N+//2nrUgCS2ir1ePlqx2GWfJ/F0h+y2O86GniCKaOzkUEf23YGhe+iB9tILNt74kZCIuGs3kcD0VlpurhaROQUGk0AOt7BgwdJSEhg1apVDB48uMplKgLQkSNHiImJqXKZ8ePH43K5+OSTT/zzRo0aRWxsLPPmzTttHQpAUpe8XpONv+SwMSOHbQfy2bY/j5/25+EqLvMvE4eLHraf6WXbRm/bdnrZfiaSohM3FtMaEs+DiDgIj/M9RrQ4+jy84nWsjh6JSLNzJn+/A+q/kLm5vlF44+LiTrtsr169KC4upkuXLjz00EOVToutXbuWu+66q9LyI0eOZNasWVVuy+1243Yf/Re6y+WqQfUiVbPZDHq3jqV366PX+JimycE899FAdCCf7fvb8sqBfvy1sBQbXs429tHLto1exnZ62bZzjvELtpwMyMmo3gfbnRARezQUVQpNcRCZ4AtUsW10/ZGINDsBE4BM0+Tuu+/m/PPP57zzzjvpcsnJybz00kv06dMHt9vN66+/ztChQ1m5cqX/qFFWVhaJiZVPFSQmJpKVlVXlNmfMmMH06dPrbmdETsMwDBIcYSQ4whjUoaV/vmmaZOeXsO1AHtv257PtwEAW7c/n6f15lBbm0t22g3ZGFjHkE2vkEWvkE0v5o5FPnJGPg3zfxty5vunIrtMXZHceDUMxbcofWx99Hlr1KWkRkcYqYE6BTZ06lY8++og1a9aQkpJyRuuOHj0awzD48MMPAQgNDeXVV1/l17/+tX+ZN998k8mTJ1NcXHzC+lUdAUpNTdUpMAko2fluftqfx+5Dhew9UsS+nCJ+ySli75EislzFeLy+/ysH4cFJAbFGXnlQ8oWlOCOPVqFFtAotIiG4kASO0LIsk5DiQ6f/8IiWJ4aimPLJeRaEhNfz3ouInF6jOwV222238eGHH7J69eozDj8AAwYM4I033vC/TkpKOuFoz4EDB044KlTBbrdjt9vP+HNFGlLLKDsto+wMPPvE98o8Xvbnudl7pIi9OYXlj0X8cqSIHeUhyV3mhTKgsPK6caFljDqrhAsTCukRnUuiZz9Gzm44shtydkNxLhRm+6a9/626uCA7hMdAmNN3i/+xz8OcJ3ld/tzu0JhIItLgLA1Apmly2223sWjRIlauXEm7du1qtJ0NGzaQnHy0OWV6ejrLli2rdB3Q0qVLGThwYK1rFglEwUE2zooJ56yYcODEa+hM0+RQQYk/GO09UsTmvbl8vj2bQwXw1s5g3toZAbSkZVQXBnVowaC0lpzfoSWt7G7fdUf+UHTs891QWggeN+Tv901nzIAwBzhS4NzL4byrIf6c2n4lIiKnZGkAmjp1Km+99RYffPAB0dHR/qM2TqeT8HDfIfVp06axd+9eXnvtNQBmzZpF27Zt6dq1KyUlJbzxxhssXLiQhQsX+rd7xx13MHjwYJ566inGjBnDBx98wPLly1mzZk3D76RIADAMw38EqUdqjH++12uydX8en2/PZs32bL7acZjsfDcfbNzHBxv3AdC+ZSSDOrTk/I59GdBrFM7wkKMbNk1wu3xHiYpyfI/FOce8zjn1e2XFgFn+Xi4c+B5WPQVJ3XxB6LyrICa1Yb4kEWlWLL0GyDCMKufPmTOHSZMmATBp0iR27drFypUrAXj66ad56aWX2Lt3L+Hh4XTt2pVp06Zx6aWXVtrGu+++y0MPPcSOHTv8AyGOHTu2WnXpNnhprtxlHjZk5PgD0aY9OXiP+S+EzYDuKTGc36Elgzq0pHebGOzBQTX/wNLio+Fn3wb47l34eQV4jw4RQOoA6HY1dLkCouJr/lki0uQ12nGAAoUCkIhPblEpX+04xJryQLTjYEGl98NCbHQ7y0lsRCiO8BAcYSE4w0NwhAfjCAvBEX7i68jQoJP+4weAgkOw5QPYvBB2fw7lfdVMI4j8VoPYl3oZu+Iv5rAnDFdRKa7iUlxFZeWPpbjLvESEBhERGkykvfwxNIgIe/njsfP97wcTYQ8iMjSYsBDbqesTkYClAFRLCkAiVduXU8Tn27PLjxAdIju/ipYepxFkM3CEBfsDkyM8GGd4CCFBNvKKyyqFmrDi/Qz1rOFXQWvpYdvh34bbDGaltycfegbyH28viqmbmxiC8BBn5NE6NJ9WYSWckxzLeakt6N4mnhaOSLCFQFD5VNXz6gQnr9d36s/jhjK373lZSfmj++TvmR7fheMRcb5xmyrGdAqNqt7nijQDCkC1pAAkcnqm6bt+6Kf9+SccicktKi2fV0Ze+Xu5RaWUemr+n5su9oNcGfIlI7xraOPd45/vtkWwo8VgMlpdiuusCwgNtVNc6iHf7aHQXUah241RmE1wwUFCirOxuw8RXnKIiNLDOMoO4/AeIdabQxy5xJGHzajFfxJtwUfDkC0YgkLBsJUHmvIg4y2t+far/MwQXyCqGOgyPPaYATCPC0vhsb4772zBgOGrzSh/hGOeH//eMa8rvafgJYFFAaiWFIBE6p5pmhSXev2nqo4GJV9oKinz+o8IVZwuq3gdZQ8mOMhWsSHY/x18t9B3miz3mJGxw2OhzSDfhdn5B6HgABQepuI0WrXqxMAb3gJ3cDTuklJKS92YnlJCKCMYDyF4fI+Gp/ZfimGD4DAItvuGEgi2l78OLX8M84Wo4DBf2CjO9e1P0WHfo+fMj8DVGcPmGx8qOhGiEiEq6ZjniRCdBFEJvvmhEdbVKc2KAlAtKQCJNBKmCb+sg83vwveLfIGnKhV/rKMSIDL+mMfE4+Yl+NqGHNdHLaewhC/Lr4X6YvshdmQXAGZ5GCojItgkLSWK9LYO+rWOplN8OEFmGXhKwPSWB5vycHNs0KmiX5vXa5LnLiO30BcQK6Z8dykeL3hNE7N8MsqKCC3JJaQkB3tJLqGlOYSW5GIvzSGsLBd7SS720lzCysqn0lzsnnwMTAzTi3EGwbBW7I5jglF5WIpKKA9Jib4BNp0pvu+lqSgr8QVZaVAKQLWkACTSCHnKYNdncOCH8rAT7ws0UeWhxlaLu9WOk5lbxOfbD/HF9mw+/zmb/a7KR2Kiw4IZ0L4Fg85uQY/UGIpLveQWlfjDTM5x4ebY+XnFpZXuvKtPNgPaxkVwTmIk5yRG0jEhinMSImjbIgJ7kM0X3kzT94h59DX4rlEqOOgb+ykv6+g4UJWe74eyKpr6nkRJeAJBcW0I8o86nnp09HFnSuCMOO4pBdc+cO2F3F+OTse+Ls7x/f4Su0BC1/LHLhDfWUfE6pECUC0pAIlIdZmmyc8H8/l8+yE+357N2h2HyCsuO/2KpxEWYiMmPBRn+Z100WHB2GwGNgNshoHNMDDKnx//aPO/rph3dJ3iUg/bD+Tz0/48XCepM8hm0LZFBOckRtMxMZpzEqM4JzGati0iCQ2uetTuAncZe44UknGokIzDhew5XMjuQwUcOpxNyZEsYs3DJJBDvOGbEowcEjhCopHDWUY2EUY1TudVNPD1B6OKcJTqa8kSFIrvGqWKa5OMY65bquY1S6bpC3bHB5pjX+dlcSanVSsxbBDX3heGErsefYxtW6chvblSAKolBSARqSmP1+S7vbm+02U/Z7Ntfz5RYcHElAcZZ3gIMeXDBlSeV/68/LFW4ytVg2maHMjz9Zf7aX8+2/bn8dN+XxPePHfVwSjYZtCuZSTnJEaTEhvOflcxGYd9gSc7v+SUnxdsMzgrNpzWcRGkxkXQOi6CNnEROCNC2LLPxdZdu8nO+Al7wV7OMrJJMQ6WT9mk2g4SyYl9HGvnuHBUMc/0+u64O50gOzha+Y5MVUyOs46Gsch4XyPi/d/7jkpWPBaepPdecDgkdC4/WtT16JEjjX11RhSAakkBSESaK9M0yXIVVwpFP+3PZ/uBfPJPEowqxESEVAo4x07JzrCjF7KfQlZuMRsyjvBNxhE2ZOTw7d5cSsp8DX4rAlGK7SDdInLpGJbDWRwgujiToBJXXX0FgOG7PskfaqoIOZEtKx1R8nhNDhW4OZjnmw4XlGCaEBzkO/IWZDMIMiDcfQhH3k9E5/5EVO5PROZsJSJ3G7aTXNDuCfc1Ig6KSfG1i3Ge5avBcZbveVRi/R458nqg6IjvqFhBti/Amd7yOx6DfI9GUPnzY18H+3r8VXpdvkzF62C77+7EOqQAVEsKQCIilZmmyb7c4vKjRHnsyykmyRnmDzipcRGV26TUkZIyLz9kuspDUQ7f7D7C3pwTrytKCCsjKSrYP8aUs/yoW3RYEI6wYJxhFXcXBpU/DybKHkSY/5Se6Tv9ZRi+ozdBIZimiau4zB9qDuYfDTjHvz5c4K7xtVs2vLQ1suhk7KGzbQ+djD10MjJoYxw47bAMXiOYsohEcJ5FcGwqNudZR4NaxRGqiJZHGw57vUcDTWF25WBT8bygvPlxxfz6ulg+pS/ctLxON6kAVEsKQCIigeuAq5hvMnL8R4q+/SUXd5m3RtuyB9uOnn4MDyHSHkxOYak/4JScwXYNA1pEhvr77hmG7649j/eYyQSP14vHW/Fo4jWhzOvF6/UdSSrzmnhNk1BPESneX2hRtp9WxiGSjMO0Mg6RbBwm2ThEIkcINk5fn8cWQml4IkGeYoLdhzHMGnxXYTG+YBjRwnf0xlvmO1XoLSufvMfNO/61xzcd+zqlL0xecua1nIICUC0pAImINB6lHi87DhZwuKDiTrvT33HnKqr+3XbRYcHER9uJj7L7Hium417HRYRW6zTfmSpwl5HlKmZ/bjFZrmL/8wO5BbhzsrC59hJWlEkSR8ORLygdIp7cKo8i5ZiRHDIdHCaaQ6aTw2Y02Tg4bPqmiueHTAd5tmhMWwjBNt+pvJAgG8FBBsE2G6HBNoJtBsFBNkKCjOOe2wipeB5kI8Rm+NYrf946LoLJF7Sv0+/qTP5+W9oNXkREpLZCgmx0Soo+o3UqxltyHReUCtxlOCNCKgWcsBBr786KtAdzdnwUZ8dHnXSZMo+X7PwSX0DKLWaTq5glrmKyc/JwH9mLmZdJnieUbNPBQU80JaaNsvKjUmVe03cE6mSB0At4vdT1sJu9W8fUeQA6EwpAIiLS7Nhshv+0V6rVxdSB4CAbSc4wkpxhnLhDfau1Da/XxFN+yq7U4/WHI39I8ngp9Rx9v6z8sdTjpczjC1GlHtP/vKTMW2m9499PdITV+fdwJhSARERExDfOFAYhQVh+1Ksh1P3JShEREZEApwAkIiIizY4CkIiIiDQ7CkAiIiLS7CgAiYiISLOjACQiIiLNjgKQiIiINDsKQCIiItLsKACJiIhIs6MAJCIiIs2OApCIiIg0OwpAIiIi0uwoAImIiEizo27wVTBNEwCXy2VxJSIiIlJdFX+3K/6On4oCUBXy8vIASE1NtbgSEREROVN5eXk4nc5TLmOY1YlJzYzX62Xfvn1ER0djGEadbtvlcpGamsqePXtwOBx1um05St9zw9D33DD0PTccfdcNo76+Z9M0ycvLo1WrVthsp77KR0eAqmCz2UhJSanXz3A4HPo/VwPQ99ww9D03DH3PDUffdcOoj+/5dEd+KugiaBEREWl2FIBERESk2VEAamB2u50//vGP2O12q0tp0vQ9Nwx9zw1D33PD0XfdMALhe9ZF0CIiItLs6AiQiIiINDsKQCIiItLsKACJiIhIs6MAJCIiIs2OApCIiIg0OwpADej555+nXbt2hIWF0adPHz777DOrS2pyHnnkEQzDqDQlJSVZXVajt3r1akaPHk2rVq0wDIP333+/0vumafLII4/QqlUrwsPDGTJkCN9//701xTZip/ueJ02adMLve8CAAdYU24jNmDGDvn37Eh0dTUJCAldccQVbt26ttIx+07VXne/Zyt+0AlADWbBgAXfeeScPPvggGzZs4IILLuCSSy4hIyPD6tKanK5du5KZmemfNm/ebHVJjV5BQQE9evTg2WefrfL9p59+mr/+9a88++yzrFu3jqSkJIYPH+5vLCzVc7rvGWDUqFGVft8ff/xxA1bYNKxatYqpU6fy5ZdfsmzZMsrKyhgxYgQFBQX+ZfSbrr3qfM9g4W/alAbRr18/c8qUKZXmde7c2bz//vstqqhp+uMf/2j26NHD6jKaNMBctGiR/7XX6zWTkpLMJ5980j+vuLjYdDqd5osvvmhBhU3D8d+zaZrm9ddfb44ZM8aSepqyAwcOmIC5atUq0zT1m64vx3/Ppmntb1pHgBpASUkJ//3vfxkxYkSl+SNGjOCLL76wqKqma9u2bbRq1Yp27dpx7bXXsmPHDqtLatJ27txJVlZWpd+33W7nwgsv1O+7HqxcuZKEhATOOeccfvvb33LgwAGrS2r0cnNzAYiLiwP0m64vx3/PFaz6TSsANYDs7Gw8Hg+JiYmV5icmJpKVlWVRVU1T//79ee2111iyZAn/+Mc/yMrKYuDAgRw6dMjq0pqsit+wft/175JLLuHNN99kxYoVzJw5k3Xr1nHxxRfjdrutLq3RMk2Tu+++m/PPP5/zzjsP0G+6PlT1PYO1v+ngev8E8TMMo9Jr0zRPmCe1c8kll/ifd+vWjfT0dM4++2xeffVV7r77bgsra/r0+65/48eP9z8/77zzSEtLo02bNnz00UeMHTvWwsoar1tvvZVvv/2WNWvWnPCeftN152Tfs5W/aR0BagAtW7YkKCjohH85HDhw4IR/YUjdioyMpFu3bmzbts3qUpqsirvs9PtueMnJybRp00a/7xq67bbb+PDDD/n0009JSUnxz9dvum6d7HuuSkP+phWAGkBoaCh9+vRh2bJlleYvW7aMgQMHWlRV8+B2u9myZQvJyclWl9JktWvXjqSkpEq/75KSElatWqXfdz07dOgQe/bs0e/7DJmmya233sp7773HihUraNeuXaX39ZuuG6f7nqvSkL9pnQJrIHfffTcTJkwgLS2N9PR0XnrpJTIyMpgyZYrVpTUp99xzD6NHj6Z169YcOHCAxx57DJfLxfXXX291aY1afn4+27dv97/euXMnGzduJC4ujtatW3PnnXfyxBNP0LFjRzp27MgTTzxBREQE//M//2Nh1Y3Pqb7nuLg4HnnkEa666iqSk5PZtWsXDzzwAC1btuTKK6+0sOrGZ+rUqbz11lt88MEHREdH+4/0OJ1OwsPDMQxDv+k6cLrvOT8/39rftCX3njVTzz33nNmmTRszNDTU7N27d6VbAaVujB8/3kxOTjZDQkLMVq1amWPHjjW///57q8tq9D799FMTOGG6/vrrTdP03Tb8xz/+0UxKSjLtdrs5ePBgc/PmzdYW3Qid6nsuLCw0R4wYYcbHx5shISFm69atzeuvv97MyMiwuuxGp6rvGDDnzJnjX0a/6do73fds9W/aKC9SREREpNnQNUAiIiLS7CgAiYiISLOjACQiIiLNjgKQiIiINDsKQCIiItLsKACJiIhIs6MAJCIiIs2OApCIiIg0OwpAIiIi0uwoAImIiEizowAkIiIizc7/B3GCim+eovGbAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "And thef tridcowind tis n, ber\n",
      "\n",
      "Hiset bobe toe.\n",
      "S:\n",
      "O-' my dalatanss:\n",
      "Want he uw hathe.\n",
      "War dthas ate awice my.\n",
      "\n",
      "Haldaru zorou wabuts, tof is hy me mil ndill, aes iree sen cin lat Het drovets, and Win ng:\n",
      "Wilerabous lplind peallllishe onchiry:\n",
      "Augr aiss hawty.\n",
      "\n",
      "'Thake norodpeeelaves\n",
      "Momy.\n",
      "Whod mothake onWindo whe Ceiiby, wout, fourive wees ired thoous\n",
      "Ar-x's uhe kad nterthirf so;\n",
      "Angis! m:\n",
      "E nge male ont ffaf Pre?\n",
      "\n",
      "WISo myat houre!\n",
      "\n",
      "Widby ak\n",
      "Sadsal thes ghe thidin cour ay aney Iry ts chan th voul\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size:int=vocab_size, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 head_size:int=head_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = head_size\n",
    "\n",
    "        #This will be our lookup table for embeddings. We'll have an entry for each token (aka vocab size) and each embedding will... \n",
    "        #...be a vector of dimension embedding_dim.\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        self.token_position_embedding_table = nn.Embedding(num_embeddings=self.context_length, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        self.self_attention_head_linear_layer = Head(head_size=head_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.head_size, out_features=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        B,T = idx.shape\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        token_position_embeddings = self.token_position_embedding_table(torch.arange(T, device=device)) #(T,C)\n",
    "        x = token_embeddings + token_position_embeddings\n",
    "        x = self.self_attention_head_linear_layer(x) #apply one head of self attention\n",
    "        logits = self.language_model_head_linear_layer(x) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Crop idx to the max size of our positional embeddings table \n",
    "            idx_crop = idx[:,-self.context_length:]\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx_crop)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = TransformerLanguageModel(vocab_size=vocab_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "tracked_losses = list()\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        tracked_losses.append(losses)\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "        \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "plot_losses(tracked_losses)\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be681498-ebd6-46d2-b82b-98a6a0adf41c",
   "metadata": {},
   "source": [
    "Next we'll add multi-head attention which is just computing multiple attention heads together in parallel and then concatenating the results. \n",
    "![Multi head attention formula](./multi_head_attention1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19a73c44-22c9-4191-acb6-02f0475e45b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, num_heads:int, head_size:int=head_size, embedding_dim:int=embedding_dim, context_length:int=context_length):\n",
    "        super().__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.head_size = head_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        \n",
    "        self.heads = nn.ModuleList([\n",
    "            Head(head_size=self.head_size, embedding_dim=self.embedding_dim, context_length=self.context_length) \n",
    "            for _ in range(self.num_heads)\n",
    "        ])\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return torch.cat([h(x) for h in self.heads], dim=-1) #Note the concat is in the last 'C' dimension => (B,T,C*num_heads)\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e38bbbbf-64cd-46ed-a8f6-97c7b247b742",
   "metadata": {},
   "source": [
    "Now let's add our newly created multi-head attention back into our Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "182499e4-4eb9-48b8-98f5-f59ca1f49ddc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.2248) Validation Loss: tensor(4.2250)\n",
      "Step: 200 Training Loss: tensor(3.0112) Validation Loss: tensor(3.0132)\n",
      "Step: 400 Training Loss: tensor(2.7330) Validation Loss: tensor(2.7487)\n",
      "Step: 600 Training Loss: tensor(2.6190) Validation Loss: tensor(2.6244)\n",
      "Step: 800 Training Loss: tensor(2.5537) Validation Loss: tensor(2.5700)\n",
      "Step: 1000 Training Loss: tensor(2.5222) Validation Loss: tensor(2.5220)\n",
      "Step: 1200 Training Loss: tensor(2.4785) Validation Loss: tensor(2.4870)\n",
      "Step: 1400 Training Loss: tensor(2.4509) Validation Loss: tensor(2.4563)\n",
      "Step: 1600 Training Loss: tensor(2.4205) Validation Loss: tensor(2.4278)\n",
      "Step: 1800 Training Loss: tensor(2.3966) Validation Loss: tensor(2.4144)\n",
      "Step: 2000 Training Loss: tensor(2.3658) Validation Loss: tensor(2.3828)\n",
      "Step: 2200 Training Loss: tensor(2.3729) Validation Loss: tensor(2.3910)\n",
      "Step: 2400 Training Loss: tensor(2.3579) Validation Loss: tensor(2.3466)\n",
      "Step: 2600 Training Loss: tensor(2.3544) Validation Loss: tensor(2.3499)\n",
      "Step: 2800 Training Loss: tensor(2.3267) Validation Loss: tensor(2.3427)\n",
      "Step: 3000 Training Loss: tensor(2.3259) Validation Loss: tensor(2.3410)\n",
      "Step: 3200 Training Loss: tensor(2.3180) Validation Loss: tensor(2.3313)\n",
      "Step: 3400 Training Loss: tensor(2.3070) Validation Loss: tensor(2.3142)\n",
      "Step: 3600 Training Loss: tensor(2.3024) Validation Loss: tensor(2.3078)\n",
      "Step: 3800 Training Loss: tensor(2.2728) Validation Loss: tensor(2.3038)\n",
      "Step: 4000 Training Loss: tensor(2.2630) Validation Loss: tensor(2.2855)\n",
      "Step: 4200 Training Loss: tensor(2.2825) Validation Loss: tensor(2.2850)\n",
      "Step: 4400 Training Loss: tensor(2.2734) Validation Loss: tensor(2.2868)\n",
      "Step: 4600 Training Loss: tensor(2.2629) Validation Loss: tensor(2.2753)\n",
      "Step: 4800 Training Loss: tensor(2.2425) Validation Loss: tensor(2.2706)\n",
      "Step: 4999 Training Loss: tensor(2.2440) Validation Loss: tensor(2.2609)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGxCAYAAACKvAkXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABf+0lEQVR4nO3deXhU9d3+8feZLJN1JgkQEiTsCIKy7yqC7CoPbpX2sSAW9Yei1VqrxaXFuuDyUBWttLYILhVQEaVVURACIogoBFEWQZYgJISEJJN1ksyc3x8DI4EA2c8kuV/Xda6ZOfM9Zz5nOjY3Z/sYpmmaiIiIiDQhNqsLEBEREalvCkAiIiLS5CgAiYiISJOjACQiIiJNjgKQiIiINDkKQCIiItLkKACJiIhIk6MAJCIiIk2OApCIiIg0OQpAIhIwFixYgGEYfP3111aXIiKNnAKQiIiINDkKQCIiItLkKACJSIOybt06RowYQXR0NBEREQwZMoQPP/yw3JjCwkLuu+8+2rdvT1hYGHFxcfTr14+FCxf6x+zdu5df/vKXtGrVCrvdTsuWLRkxYgQpKSnl1rV48WIGDx5MZGQkUVFRjBkzhi1btpQbU9l1iUjgCLa6ABGRylqzZg2jRo2iR48ezJs3D7vdzssvv8z48eNZuHAhEydOBODee+/ljTfe4PHHH6d3794UFBTw3XffkZWV5V/XFVdcgcfj4ZlnnqFNmzZkZmayfv16cnJy/GOefPJJHn74YW6++WYefvhhSkpKePbZZ7n00kv56quv6NatW6XXJSKBxTBN07S6CBER8J0EffPNN7Np0yb69et32vuDBw9m7969/Pjjj0RFRQHg8Xjo1asXOTk5pKamYhgGF110EZ06dWLp0qUVfk5WVhbNmzfn+eef5+67765wzMGDB+nQoQO33347c+bM8c/Pz8+nc+fODB06lMWLF1dqXSISeHQITEQahIKCAjZu3Mj111/vDz8AQUFBTJo0iZ9++oldu3YBMGDAAD7++GP++Mc/kpycTFFRUbl1xcXF0bFjR5599ln++te/smXLFrxeb7kxn3zyCWVlZUyePJmysjL/FBYWxmWXXUZycnKl1yUigUcBSEQahOzsbEzTJDEx8bT3WrVqBeA/xDVnzhweeOAB3n//fYYPH05cXBxXX301u3fvBsAwDD777DPGjBnDM888Q58+fWjRogW//e1vycvLA+DIkSMA9O/fn5CQkHLT4sWLyczMrPS6RCTw6BwgEWkQYmNjsdlspKWlnfbe4cOHAWjevDkAkZGRPProozz66KMcOXLEvzdo/Pjx7Ny5E4C2bdsyb948AH744QfefvttZs6cSUlJCX//+9/963r33Xdp27btWWs717pEJPDoHCARCRjnOgdoyJAh7Nu3j7179xIeHg6A1+ulV69eZGdn+88Bqsjvfvc7nn/+eQoKCoiIiKhwTO/evQkJCeGrr75i//79dO7cmSeeeIL777+/ytty8rpEJPBoD5CIBJxVq1axf//+0+bPmjWLUaNGMXz4cO677z5CQ0N5+eWX+e6771i4cKE//AwcOJCrrrqKHj16EBsby44dO3jjjTcYPHgwERERfPvtt9x555384he/oHPnzoSGhrJq1Sq+/fZb/vjHPwLQrl07/vKXv/DQQw+xd+9exo4dS2xsLEeOHOGrr77y72WqzLpEJACZIiIBYv78+SZwxmnfvn3m559/bl5++eVmZGSkGR4ebg4aNMj8z3/+U249f/zjH81+/fqZsbGxpt1uNzt06GD+7ne/MzMzM03TNM0jR46YU6ZMMbt27WpGRkaaUVFRZo8ePcznnnvOLCsrK7eu999/3xw+fLjpcDhMu91utm3b1rz++uvNlStXVnldIhI4dAhMREREmhxdBSYiIiJNjgKQiIiINDkKQCIiItLkKACJiIhIk6MAJCIiIk2OApCIiIg0OboRYgW8Xi+HDx8mOjr6jHeVFRERkcBimiZ5eXm0atUKm+3s+3gUgCpw+PBhkpKSrC5DREREquHgwYO0bt36rGMUgCoQHR0N+L5Ah8NhcTUiIiJSGS6Xi6SkJP/f8bNRAKrAicNeDodDAUhERKSBqczpKzoJWkRERJocBSARERFpchSAREREpMnROUAiIlLrTNOkrKwMj8djdSnSyISEhBAUFFTj9SgAiYhIrSopKSEtLY3CwkKrS5FGyDAMWrduTVRUVI3WEzABaNasWTz44IPcfffdPP/88xWOee+995g7dy4pKSm43W66d+/OzJkzGTNmjH/MggULuPnmm09btqioiLCwsLoqX0RE8N1Idt++fQQFBdGqVStCQ0N1Q1mpNaZpcvToUX766Sc6d+5coz1BARGANm3axCuvvEKPHj3OOm7t2rWMGjWKJ598kpiYGObPn8/48ePZuHEjvXv39o9zOBzs2rWr3LIKPyIida+kpASv10tSUhIRERFWlyONUIsWLdi/fz+lpaUNOwDl5+dz44038s9//pPHH3/8rGNP3TP05JNP8sEHH/Cf//ynXAAyDIOEhIS6KFdERCrhXG0IRKqrtvYoWv4LnT59OldeeSUjR46s8rJer5e8vDzi4uLKzc/Pz6dt27a0bt2aq666ii1btpx1PW63G5fLVW4SERGRxsvSALRo0SI2b97MrFmzqrX87NmzKSgo4IYbbvDP69q1KwsWLGDZsmUsXLiQsLAwLr74Ynbv3n3G9cyaNQun0+mf1AdMRESkcbMsAB08eJC7776bN998s1rn5yxcuJCZM2eyePFi4uPj/fMHDRrEr3/9a3r27Mmll17K22+/zfnnn8+LL754xnXNmDGD3Nxc/3Tw4MFqbZOIiMgJw4YN45577qn0+P3792MYBikpKXVWk/zMsnOAvvnmGzIyMujbt69/nsfjYe3atbz00ku43e4znty0ePFipk6dyjvvvHPOQ2c2m43+/fufdQ+Q3W7HbrdXb0NERKRBO9c5JTfddBMLFiyo8nrfe+89QkJCKj0+KSmJtLQ0mjdvXuXPqor9+/fTvn17tmzZQq9ever0swKZZQFoxIgRbNu2rdy8m2++ma5du/LAAw+cMfwsXLiQ3/zmNyxcuJArr7zynJ9jmiYpKSlcdNFFtVJ3TRz+8XsOffYyZmgkA6Y8Y3U5IiICpKWl+Z8vXryYP/3pT+WuJA4PDy83vrS0tFLB5tTzU88lKChIF/DUI8sOgUVHR3PhhReWmyIjI2nWrBkXXngh4Ds0NXnyZP8yCxcuZPLkycyePZtBgwaRnp5Oeno6ubm5/jGPPvoon3zyCXv37iUlJYWpU6eSkpLCtGnT6n0bT+U6lkb/w2/S+sBSq0sREakXpmlSWFJmyWSaZqVqTEhI8E9Op9N/JXFCQgLFxcXExMTw9ttvM2zYMMLCwnjzzTfJysriV7/6Fa1btyYiIoKLLrqIhQsXllvvqYfA2rVrx5NPPslvfvMboqOjadOmDa+88or//VMPgSUnJ2MYBp999hn9+vUjIiKCIUOGnHabl8cff5z4+Hiio6O55ZZb+OMf/1ijPTtut5vf/va3xMfHExYWxiWXXMKmTZv872dnZ3PjjTfSokULwsPD6dy5M/Pnzwd8t0G48847SUxMJCwsjHbt2lX7PN+6Zvll8GeTlpZGamqq//U//vEPysrKmD59OtOnT/fPP3n3ZE5ODrfddhvp6ek4nU569+7N2rVrGTBgQH2Xf5roZq0AiPHmnmOkiEjjUFTqodufPrHks7f/ZQwRobXzZ+6BBx5g9uzZzJ8/H7vdTnFxMX379uWBBx7A4XDw4YcfMmnSJDp06MDAgQPPuJ7Zs2fz2GOP8eCDD/Luu+9y++23M3ToULp27XrGZR566CFmz55NixYtmDZtGr/5zW/44osvAPj3v//NE088wcsvv8zFF1/MokWLmD17Nu3bt6/2tt5///0sWbKE1157jbZt2/LMM88wZswY9uzZQ1xcHI888gjbt2/n448/pnnz5uzZs4eioiIA5syZw7Jly3j77bdp06YNBw8eDNjzagMqACUnJ5d7feox11Pfr8hzzz3Hc889V3tF1SJnC18AijDcFObnEBEVY21BIiJSKffccw/XXnttuXn33Xef//ldd93F8uXLeeedd84agK644gruuOMOwBeqnnvuOZKTk88agJ544gkuu+wyAP74xz9y5ZVXUlxcTFhYGC+++CJTp071d0D405/+xKeffkp+fn61trOgoIC5c+eyYMECxo0bB8A///lPVqxYwbx58/jDH/5AamoqvXv3pl+/foBvz9YJqampdO7cmUsuuQTDMGjbtm216qgPARWAGrvISAdFZijhRgk5R9MUgESk0QsPCWL7X8ace2AdfXZtOfHH/gSPx8NTTz3F4sWLOXToEG63G7fbTWRk5FnXc3LHgxOH2jIyMiq9TGJiIgAZGRm0adOGXbt2+QPVCQMGDGDVqlWV2q5T/fjjj5SWlnLxxRf754WEhDBgwAB27NgBwO233851113H5s2bGT16NFdffTVDhgwBYMqUKYwaNYouXbowduxYrrrqKkaPHl2tWuqa5TdCbEoMm41sWwwAeVmHrC1GRKQeGIZBRGiwJVNt9iA7NdjMnj2b5557jvvvv59Vq1aRkpLCmDFjKCkpOet6Tj152jAMvF5vpZc5sU0nL3Pqdlb23KeKnFi2onWemDdu3DgOHDjAPffcw+HDhxkxYoR/b1ifPn3Yt28fjz32GEVFRdxwww1cf/311a6nLikA1bP8oBgAirKPWFuIiIhU2+eff86ECRP8953r0KHDWW+3Ule6dOnCV199VW7e119/Xe31derUidDQUNatW+efV1paytdff80FF1zgn9eiRQumTJnCm2++yfPPP1/uZG6Hw8HEiRP55z//yeLFi1myZAnHjh2rdk11RYfA6llhSDMog5LcdKtLERGRaurUqRNLlixh/fr1xMbG8te//pX09PRyIaE+3HXXXdx6663069ePIUOGsHjxYr799ls6dOhwzmVPvZoMoFu3btx+++384Q9/IC4ujjZt2vDMM89QWFjI1KlTAd95Rn379qV79+643W7++9//+rf7ueeeIzExkV69emGz2XjnnXdISEggJiamVre7NigA1bOSsDgoAm/e2Y/5iohI4HrkkUfYt28fY8aMISIigttuu42rr7663G1Z6sONN97I3r17ue+++yguLuaGG25gypQpp+0Vqsgvf/nL0+bt27ePp556Cq/Xy6RJk8jLy6Nfv3588sknxMbGAhAaGsqMGTPYv38/4eHhXHrppSxatAiAqKgonn76aXbv3k1QUBD9+/fno48+CsjmuIZZk4OFjZTL5cLpdJKbm4vD4ajVdW945W4GH17AxhbXM3D6vFpdt4iI1YqLi9m3bx/t27evVpsjqblRo0aRkJDAG2+8YXUpdeJsv7Gq/P3WHqB6ZkS1ACC4KNPiSkREpKErLCzk73//O2PGjCEoKIiFCxeycuVKVqxYYXVpAU8BqJ4FO1oCEFYSeCeEiYhIw2IYBh999BGPP/44brebLl26sGTJknP2yRQFoHoXFuMLQFFl2RZXIiIiDV14eDgrV660uowGKfDOSmrkIuN8d4N2eHOsLURERKQJUwCqZ47mvrt4xpJHWYnb4mpERESaJgWgehbTLAGP6bubZk6W7gUkIiJiBQWgehYUFES24QQg9+hhi6sRERFpmhSALOA63g+sMCfN2kJERESaKAUgCxSE+O6mWZKjfmAiIiJWUACyQHFoMwDKXApAIiKNxbBhw7jnnnv8r9u1a8fzzz9/1mUMw+D999+v8WfX1nqaEgUgC3jCmwNgFhy1uBIRERk/fvwZbxy4YcMGDMNg8+bNVV7vpk2buO2222paXjkzZ86kV69ep81PS0tj3LhxtfpZp1qwYEFANjWtLgUgC3gjfe0wgtQOQ0TEclOnTmXVqlUcOHDgtPdeffVVevXqRZ8+faq83hYtWhAREVEbJZ5TQkICdru9Xj6rsVAAskBwdDwA9uIsiysREaljpgklBdZMlez1fdVVVxEfH8+CBQvKzS8sLGTx4sVMnTqVrKwsfvWrX9G6dWsiIiK46KKLWLhw4VnXe+ohsN27dzN06FDCwsLo1q1bhf26HnjgAc4//3wiIiLo0KEDjzzyCKWlpYBvD8yjjz7K1q1bMQwDwzD8NZ96CGzbtm1cfvnlhIeH06xZM2677Tby8/P970+ZMoWrr76a//u//yMxMZFmzZoxffp0/2dVR2pqKhMmTCAqKgqHw8ENN9zAkSM/n+qxdetWhg8fTnR0NA6Hg759+/L1118DcODAAcaPH09sbCyRkZF0796djz76qNq1VIZaYVgg1OlrhxFRqn5gItLIlRbCk62s+ewHD0No5DmHBQcHM3nyZBYsWMCf/vQnDMN3r7Z33nmHkpISbrzxRgoLC+nbty8PPPAADoeDDz/8kEmTJtGhQwcGDhx4zs/wer1ce+21NG/enC+//BKXy1XufKEToqOjWbBgAa1atWLbtm3ceuutREdHc//99zNx4kS+++47li9f7m9/4XQ6T1tHYWEhY8eOZdCgQWzatImMjAxuueUW7rzzznIhb/Xq1SQmJrJ69Wr27NnDxIkT6dWrF7feeus5t+dUpmly9dVXExkZyZo1aygrK+OOO+5g4sSJJCcnA3DjjTfSu3dv5s6dS1BQECkpKYSEhAAwffp0SkpKWLt2LZGRkWzfvp2oqKgq11EVCkAWiIj13Q3a4VE/MBGRQPCb3/yGZ599luTkZIYPHw74Dn9de+21xMbGEhsby3333ecff9ddd7F8+XLeeeedSgWglStXsmPHDvbv30/r1q0BePLJJ087b+fhhx/2P2/Xrh2///3vWbx4Mffffz/h4eFERUURHBxMQkLCGT/r3//+N0VFRbz++utERvoC4EsvvcT48eN5+umnadnS94/w2NhYXnrpJYKCgujatStXXnkln332WbUC0MqVK/n222/Zt28fSUlJALzxxht0796dTZs20b9/f1JTU/nDH/5A165dAejcubN/+dTUVK677jouuugiADp06FDlGqpKAcgC0cfbYcSYuZheL4ZNRyJFpJEKifDtibHqsyupa9euDBkyhFdffZXhw4fz448/8vnnn/Ppp58C4PF4eOqpp1i8eDGHDh3C7Xbjdrv9AeNcduzYQZs2bfzhB2Dw4MGnjXv33Xd5/vnn2bNnD/n5+ZSVleFwOCq9HSc+q2fPnuVqu/jii/F6vezatcsfgLp3705QUJB/TGJiItu2bavSZ538mUlJSf7wA9CtWzdiYmLYsWMH/fv359577+WWW27hjTfeYOTIkfziF7+gY8eOAPz2t7/l9ttv59NPP2XkyJFcd9119OjRo1q1VJb+8logtoVvd3Co4SHPpfOARKQRMwzfYSgrpuOHsipr6tSpLFmyBJfLxfz582nbti0jRowAYPbs2Tz33HPcf//9rFq1ipSUFMaMGUNJSUml1m1WcD6ScUp9X375Jb/85S8ZN24c//3vf9myZQsPPfRQpT/j5M86dd0VfeaJw08nv+f1eqv0Wef6zJPnz5w5k++//54rr7ySVatW0a1bN5YuXQrALbfcwt69e5k0aRLbtm2jX79+vPjii9WqpbIUgCwQFh6Jy/T9y8SldhgiIgHhhhtuICgoiLfeeovXXnuNm2++2f/H+/PPP2fChAn8+te/pmfPnnTo0IHdu3dXet3dunUjNTWVw4d//v/8DRs2lBvzxRdf0LZtWx566CH69etH586dT7syLTQ0FI/Hc87PSklJoaCgoNy6bTYb559/fqVrrooT23fw4EH/vO3bt5Obm8sFF1zgn3f++efzu9/9jk8//ZRrr72W+fPn+99LSkpi2rRpvPfee/z+97/nn//8Z53UeoICkEVyj7fDyMtSABIRCQRRUVFMnDiRBx98kMOHDzNlyhT/e506dWLFihWsX7+eHTt28P/+3/8jPb3yDa1HjhxJly5dmDx5Mlu3buXzzz/noYceKjemU6dOpKamsmjRIn788UfmzJnj30NyQrt27di3bx8pKSlkZmbidrtP+6wbb7yRsLAwbrrpJr777jtWr17NXXfdxaRJk/yHv6rL4/GQkpJSbtq+fTsjR46kR48e3HjjjWzevJmvvvqKyZMnc9lll9GvXz+Kioq48847SU5O5sCBA3zxxRds2rTJH47uuecePvnkE/bt28fmzZtZtWpVueBUFxSALJIfHANAcY46wouIBIqpU6eSnZ3NyJEjadOmjX/+I488Qp8+fRgzZgzDhg0jISGBq6++utLrtdlsLF26FLfbzYABA7jlllt44oknyo2ZMGECv/vd77jzzjvp1asX69ev55FHHik35rrrrmPs2LEMHz6cFi1aVHgpfkREBJ988gnHjh2jf//+XH/99YwYMYKXXnqpal9GBfLz8+ndu3e56YorrvBfhh8bG8vQoUMZOXIkHTp0YPHixYCvEXhWVhaTJ0/m/PPP54YbbmDcuHE8+uijgC9YTZ8+nQsuuICxY8fSpUsXXn755RrXezaGWdGBySbO5XLhdDrJzc2t8slnlbX5mavoU/g5Gy94kIETH6iTzxARqW/FxcXs27eP9u3bExYWZnU50gid7TdWlb/f2gNkkdJwXz8wMz/D4kpERESaHgUgi3iP9wOzFaofmIiISH1TALKI7Xg7jBC1wxAREal3CkAWCXb4zsQPL1E7DBERkfqmAGSRsBhfAIoqUzsMEWl8dH2N1JXa+m0FTACaNWsWhmFU2BzuZGvWrKFv376EhYXRoUMH/v73v582ZsmSJXTr1g273V7uTpOBJKqZ727QMWaOtYWIiNSiE3cXLiwstLgSaaxO3Bn75DYe1REQvcA2bdrEK6+8cs6+H/v27eOKK67g1ltv5c033+SLL77gjjvuoEWLFlx33XWA786aEydO5LHHHuOaa65h6dKl3HDDDaxbt65SDevqS0xzXwCKogh3UT728LrteisiUh+CgoKIiYkhI8N3hWtERMQZ2zKIVJXX6+Xo0aNEREQQHFyzCGP5fYDy8/Pp06cPL7/8Mo8//ji9evXi+eefr3DsAw88wLJly9ixY4d/3rRp09i6dav/luITJ07E5XLx8ccf+8eMHTuW2NjYCm8YVZH6uA+Q6fVS+mgLQo0yjkz9mpZJnc+9kIhIA2CaJunp6eTk5FhdijRCNpuN9u3bExoaetp7Vfn7bfkeoOnTp3PllVcycuRIHn/88bOO3bBhA6NHjy43b8yYMcybN4/S0lJCQkLYsGEDv/vd704bc6ZQBfi7+p7gcrmqviFVZNhsZBtOWpJFXmaaApCINBqGYZCYmEh8fDylpaVWlyONTGhoKDZbzc/gsTQALVq0iM2bN7Np06ZKjU9PTz+tj0nLli0pKysjMzOTxMTEM445W8+WWbNm+W/HXZ9cQbG09GRRmJ1W758tIlLXgoKCanyehkhdsewk6IMHD3L33Xfz5ptvVul26aceSz5xBO/k+RWNOdsx6BkzZpCbm+ufTu5mW5cKQ2IBcKsfmIiISL2ybA/QN998Q0ZGBn379vXP83g8rF27lpdeegm3233avxwSEhJO25OTkZFBcHAwzZo1O+uYs3XAtdvt2O32mm5SlbntzaAYPPm6G7SIiEh9smwP0IgRI9i2bRspKSn+qV+/ftx4442kpKRUuNt08ODBrFixoty8Tz/9lH79+vkvvTzTmCFDhtTdxlSTJ8LXDsNQPzAREZF6ZdkeoOjoaC688MJy8yIjI2nWrJl//owZMzh06BCvv/464Lvi66WXXuLee+/l1ltvZcOGDcybN6/c1V133303Q4cO5emnn2bChAl88MEHrFy5knXr1tXfxlWSGeFrhxGsdhgiIiL1KmBuhFiRtLQ0UlNT/a/bt2/PRx99RHJyMr169eKxxx5jzpw5/nsAAQwZMoRFixYxf/58evTowYIFC1i8eHFA3QPohGCHLwDZ3QpAIiIi9cny+wAFovq4DxDA1uT36Jl8M/uC2tH+ka119jkiIiJNQVX+fgf0HqDGLjIuEQCHR/3ARERE6pMCkIUczc8DIMZ04S0rs7gaERGRpkMByEIxzRPwmgZBhonr2BGryxEREWkyFIAsFBoaSo7ha4Kam3nY4mpERESaDgUgi7lsMQDkH1M7DBERkfqiAGSx/GBfO4ziHB0CExERqS8KQBYrDvW18PC41A9MRESkvigAWaws3BeAvAWZFlciIiLSdCgAWcwb0QKAoEI1RBUREakvCkAWs0X7AlCo2mGIiIjUGwUgi4U4EgCIKDlmcSUiIiJNhwKQxSJjfQEoSu0wRERE6o0CkMWimrUCINabA+pLKyIiUi8UgCwW08IXgMKMUgrzc6wtRkREpIlQALJYZJSDAtMOQO5RtcMQERGpDwpAFjMMg5zj7TDystQOQ0REpD4oAAWAvCBfO4zCbAUgERGR+qAAFACKQnwBqNSlfmAiIiL1QQEoAJSENQfAm6+7QYuIiNQHBaAA4InwBSCjQAFIRESkPigABQAjMh6A4GK1wxAREakPCkABINjhC0Dh6gcmIiJSLxSAAkBYjK8dRmSZ2mGIiIjUBwWgABAZ5wtATm+OtYWIiIg0EQpAAcDR/DwAnORTVlJscTUiIiKNnwJQAIhtFk+Z6fufIidTN0MUERGpawpAASAoKIhswwmAS+0wRERE6pwCUIBwHe8HVnhMAUhERKSuKQAFiIKQOADcOWqHISIiUtcUgAKE2+4LQGV5CkAiIiJ1TQEoQJSFNfM9UTsMERGROmdpAJo7dy49evTA4XDgcDgYPHgwH3/88RnHT5kyBcMwTpu6d+/uH7NgwYIKxxQXB/bl5eaJdhiFmRZXIiIi0vgFW/nhrVu35qmnnqJTp04AvPbaa0yYMIEtW7aUCzUnvPDCCzz11FP+12VlZfTs2ZNf/OIX5cY5HA527dpVbl5YWFgdbEHtCYpuAUCo2mGIiIjUOUsD0Pjx48u9fuKJJ5g7dy5ffvllhQHI6XTidDr9r99//32ys7O5+eaby40zDIOEhIS6KbqOhDqPt8MoPWZxJSIiIo1fwJwD5PF4WLRoEQUFBQwePLhSy8ybN4+RI0fStm3bcvPz8/Np27YtrVu35qqrrmLLli1nXY/b7cblcpWb6ltEXCIA0Z6cev9sERGRpsbyALRt2zaioqKw2+1MmzaNpUuX0q1bt3Mul5aWxscff8wtt9xSbn7Xrl1ZsGABy5YtY+HChYSFhXHxxReze/fuM65r1qxZ/r1LTqeTpKSkGm9XVUUfD0AxZi6m11Pvny8iItKUGKZpmlYWUFJSQmpqKjk5OSxZsoR//etfrFmz5pwhaNasWcyePZvDhw8TGhp6xnFer5c+ffowdOhQ5syZU+EYt9uN2+32v3a5XCQlJZGbm4vD4ajehlVRUWEh4c/4QlDe3buJjo2vl88VERFpLFwuF06ns1J/vy09BwggNDTUfxJ0v3792LRpEy+88AL/+Mc/zriMaZq8+uqrTJo06azhB8Bms9G/f/+z7gGy2+3Y7fbqbUAtCY+IINeMxGkUkHv0sAKQiIhIHbL8ENipTNMstzemImvWrGHPnj1MnTq1UutLSUkhMTGxtkqsM7k23wne+ccOW1yJiIhI42bpHqAHH3yQcePGkZSURF5eHosWLSI5OZnly5cDMGPGDA4dOsTrr79ebrl58+YxcOBALrzwwtPW+eijjzJo0CA6d+6My+Vizpw5pKSk8Le//a1etqkm8oJjofQwxdnpVpciIiLSqFkagI4cOcKkSZNIS0vD6XTSo0cPli9fzqhRowDfic6pqanllsnNzWXJkiW88MILFa4zJyeH2267jfT0dJxOJ71792bt2rUMGDCgzrenpopD46AUSlwZVpciIiLSqFl+EnQgqspJVLVpw4tTGJy1lI1Jv2Hg1Ofq7XNFREQag6r8/Q64c4CaMjOiOQC2Qt0NWkREpC4pAAUQI8p35VdIsfqBiYiI1CUFoAAS4vAFoPAStcMQERGpSwpAASQs1nepflRZtsWViIiING4KQAEkOs7XEDXGzLG2EBERkUZOASiAOFu0BiCSYkqK8i2uRkREpPFSAAogDkcsbjMEgJyjhyyuRkREpPFSAAogtiAbxwxfO4y8LN0NWkREpK4oAAWYvKBYAAqy1Q9MRESkrigABZiCkDgASnLVDkNERKSuKAAFmBK7LwB5XEcsrkRERKTxUgAKMGXhvnYYRuFRiysRERFpvBSAAowR1QKA4CK1wxAREakrCkABJii6JQB2t9phiIiI1BUFoAATFuMLQJGlCkAiIiJ1RQEowEQc7wfm8OZYW4iIiEgjpgAUYJzNWwEQY+bhLSu1uBoREZHGSQEowMQ0T8BrGtgME9cxXQovIiJSFxSAAkxoaCg5RjQArkz1AxMREakLCkABKNcWA0D+MfUDExERqQsKQAEoP9h3N2h3jgKQiIhIXVAACkDFob6GqKVqhyEiIlInFIACUGmYrx0GBWqHISIiUhcUgAJRpK8dhq1Q7TBERETqggJQADKi4wEILc6yuBIREZHGSQEoAIU6fO0wItQOQ0REpE4oAAWgiNgEAKI92RZXIiIi0jgpAAWg6GbH22F4c8E0La5GRESk8VEACkDOFr6GqHajlKK8HGuLERERaYQUgAJQVJSDfDMMgBy1wxAREal1CkAByDAMco63w8jLSrO2GBERkUZIAShA5QXFAFCUrQAkIiJS2ywNQHPnzqVHjx44HA4cDgeDBw/m448/PuP45ORkDMM4bdq5c2e5cUuWLKFbt27Y7Xa6devG0qVL63pTal1RaDMASl0ZFlciIiLS+FgagFq3bs1TTz3F119/zddff83ll1/OhAkT+P7778+63K5du0hLS/NPnTt39r+3YcMGJk6cyKRJk9i6dSuTJk3ihhtuYOPGjXW9ObWqxO5riOrNUwASERGpbYZpBtZ11nFxcTz77LNMnTr1tPeSk5MZPnw42dnZxMTEVLj8xIkTcblc5fYkjR07ltjYWBYuXFjhMm63G7fb7X/tcrlISkoiNzcXh8NRsw2qpi/++TsuPvQqm1pcS//p8y2pQUREpCFxuVw4nc5K/f0OmHOAPB4PixYtoqCggMGDB591bO/evUlMTGTEiBGsXr263HsbNmxg9OjR5eaNGTOG9evXn3F9s2bNwul0+qekpKTqb0gtMY73AwsuUj8wERGR2mZ5ANq2bRtRUVHY7XamTZvG0qVL6datW4VjExMTeeWVV1iyZAnvvfceXbp0YcSIEaxdu9Y/Jj09nZYtW5ZbrmXLlqSnp5+xhhkzZpCbm+ufDh48WDsbVwMhTl8/sLAStcMQERGpbcFWF9ClSxdSUlLIyclhyZIl3HTTTaxZs6bCENSlSxe6dOnifz148GAOHjzI//3f/zF06FD/fMMwyi1nmuZp805mt9ux2+21sDW1x+703QwxqkztMERERGqb5XuAQkND6dSpE/369WPWrFn07NmTF154odLLDxo0iN27d/tfJyQknLa3JyMj47S9QoEuMs7XD8zpzbG2EBERkUbI8gB0KtM0y52QfC5btmwhMTHR/3rw4MGsWLGi3JhPP/2UIUOG1FqN9cHZ/DwAHBRQ5i6yuBoREZHGxdJDYA8++CDjxo0jKSmJvLw8Fi1aRHJyMsuXLwd85+YcOnSI119/HYDnn3+edu3a0b17d0pKSnjzzTdZsmQJS5Ys8a/z7rvvZujQoTz99NNMmDCBDz74gJUrV7Ju3TpLtrG6YuKaU2oGEWJ4yM1Kp1mr9laXJCIi0mhYGoCOHDnCpEmTSEtLw+l00qNHD5YvX86oUaMASEtLIzU11T++pKSE++67j0OHDhEeHk737t358MMPueKKK/xjhgwZwqJFi3j44Yd55JFH6NixI4sXL2bgwIH1vn01ERwcTIbhJJ5juLIOKQCJiIjUooC7D1AgqMp9BOrSnr/0opN3H98Pm0f3YddbVoeIiEhD0CDvAySnKwjx3Q3anXvmS/hFRESk6hSAApjb7usHVqZ2GCIiIrVKASiAlYU39z0pOGptISIiIo2MAlAgO94OI6hQ7TBERERqkwJQALNF+dph2N1ZFlciIiLSuCgABTC703f36ohS9QMTERGpTQpAASz8eDsMhyfH2kJEREQaGQWgAOZo1gqAGDMX0+uxuBoREZHGQwEogMW28AWgYMNLQY6uBBMREaktCkABLDw8nFwzEoDczDSLqxEREWk8FIACXI4tFoD8rMMWVyIiItJ4KAAFuPzgGACKco5YW4iIiEgjogAU4IpCfe0wStUPTEREpNYoAAW40jBfADLVDkNERKTWKAAFOG+Erx2GoXYYIiIitUYBKMAZUb4AFFqsACQiIlJbFIACXMjxdhjhJWqHISIiUlsUgAJcmNPXDiOqLNviSkRERBoPBaAA52juC0AxZo61hYiIiDQiCkABztG8NQARuCkpzLO4GhERkcZBASjAOR0xFJshAORm6m7QIiIitUEBKMDZgmwcM2IAcCkAiYiI1AoFoAYgLygGgMJs3Q1aRESkNigANQCFIXEAlKgdhoiISK1QAGoA3HZfOwxPXobFlYiIiDQOCkANgCeiOQCG+oGJiIjUimoFoIMHD/LTTz/5X3/11Vfcc889vPLKK7VWmJwk0tcOI7g4y+JCREREGodqBaD//d//ZfXq1QCkp6czatQovvrqKx588EH+8pe/1GqBAsEOXzsMu1sBSEREpDZUKwB99913DBgwAIC3336bCy+8kPXr1/PWW2+xYMGC2qxPAPvxfmCRpWqHISIiUhuqFYBKS0ux2+0ArFy5kv/5n/8BoGvXrqSlpdVedQJAZFwrAJxeBSAREZHaUK0A1L17d/7+97/z+eefs2LFCsaOHQvA4cOHadasWa0WKCf1AyMPb1mpxdWIiIg0fNUKQE8//TT/+Mc/GDZsGL/61a/o2bMnAMuWLfMfGquMuXPn0qNHDxwOBw6Hg8GDB/Pxxx+fcfx7773HqFGjaNGihX/8J598Um7MggULMAzjtKm4uLg6mxoQYpol4DENAPKOaQ+biIhITQVXZ6Fhw4aRmZmJy+UiNjbWP/+2224jIiKi0utp3bo1Tz31FJ06dQLgtddeY8KECWzZsoXu3bufNn7t2rWMGjWKJ598kpiYGObPn8/48ePZuHEjvXv39o9zOBzs2rWr3LJhYWFV3cyAYQ8NJctw0IxccjPTcMa3sbokERGRBq1aAaioqAjTNP3h58CBAyxdupQLLriAMWPGVHo948ePL/f6iSeeYO7cuXz55ZcVBqDnn3++3Osnn3ySDz74gP/85z/lApBhGCQkJFRhiwJfji2GZt5cCrQHSEREpMaqdQhswoQJvP766wDk5OQwcOBAZs+ezdVXX83cuXOrVYjH42HRokUUFBQwePDgSi3j9XrJy8sjLi6u3Pz8/Hzatm1L69atueqqq9iyZctZ1+N2u3G5XOWmQFMQ7AubxTlqhyEiIlJT1QpAmzdv5tJLLwXg3XffpWXLlhw4cIDXX3+dOXPmVGld27ZtIyoqCrvdzrRp01i6dCndunWr1LKzZ8+moKCAG264wT+va9euLFiwgGXLlrFw4ULCwsK4+OKL2b179xnXM2vWLJxOp39KSkqq0jbUh+JQ38nlZa4jFlciIiLS8FUrABUWFhIdHQ3Ap59+yrXXXovNZmPQoEEcOHCgSuvq0qULKSkpfPnll9x+++3cdNNNbN++/ZzLLVy4kJkzZ7J48WLi4+P98wcNGsSvf/1revbsyaWXXsrbb7/N+eefz4svvnjGdc2YMYPc3Fz/dPDgwSptQ30oCz9+dV2+2mGIiIjUVLUCUKdOnXj//fc5ePAgn3zyCaNHjwYgIyMDh8NRpXWFhobSqVMn+vXrx6xZs+jZsycvvPDCWZdZvHgxU6dO5e2332bkyJFnHWuz2ejfv/9Z9wDZ7Xb/lWgnpkDjjfC1w7AVZVpciYiISMNXrQD0pz/9ifvuu4927doxYMAA/zk7n376abmTkavDNE3cbvcZ31+4cCFTpkzhrbfe4sorr6zU+lJSUkhMTKxRXVYLivbt5QpVPzAREZEaq9ZVYNdffz2XXHIJaWlp/nsAAYwYMYJrrrmm0ut58MEHGTduHElJSeTl5bFo0SKSk5NZvnw54Ds0dejQIf8J1wsXLmTy5Mm88MILDBo0iPR03wnB4eHhOJ1OAB599FEGDRpE586dcblczJkzh5SUFP72t79VZ1MDRujxdhgRpccsrkRERKThq1YAAkhISCAhIYGffvoJwzA477zzqnQTRIAjR44wadIk0tLScDqd9OjRg+XLlzNq1CgA0tLSSE1N9Y//xz/+QVlZGdOnT2f69On++TfddJO/B1lOTg633XYb6enpOJ1Oevfuzdq1a6tcW6AJj/Vd1h/tUTsMERGRmjJM0zSrupDX6+Xxxx9n9uzZ5OfnAxAdHc3vf/97HnroIWy2ah1ZCxgulwun00lubm7AnA+Uum8XbV4bQIkZTOjMTDAMq0sSEREJKFX5+12tPUAPPfQQ8+bN46mnnuLiiy/GNE2++OILZs6cSXFxMU888US1Cpczi2nuO4cp1CijKC+bcEfcOZYQERGRM6lWAHrttdf417/+5e8CD9CzZ0/OO+887rjjDgWgOhAdFU2+GU6UUUTO0UMKQCIiIjVQrWNVx44do2vXrqfN79q1K8eO6STdumAYBtm2GADy1Q5DRESkRqoVgHr27MlLL7102vyXXnqJHj161LgoqVh+UAwARdlqhyEiIlIT1ToE9swzz3DllVeycuVKBg8ejGEYrF+/noMHD/LRRx/Vdo1yXGFIHJRBaa4CkIiISE1Uaw/QZZddxg8//MA111xDTk4Ox44d49prr+X7779n/vz5tV2jHFcS5muH4VE7DBERkRqp9n2AWrVqddrJzlu3buW1117j1VdfrXFhcjpPRAvIBluBApCIiEhNNOwb9jQxRqSvH1iw2mGIiIjUiAJQAxJyvB1GeIkCkIiISE0oADUg9uMBKLJM7TBERERqokrnAF177bVnfT8nJ6cmtcg5RDVrBUCMN8faQkRERBq4KgWgEx3Xz/b+5MmTa1SQnJnjeDuMKArxlBQRFBpucUUiIiINU5UCkC5xt1ZsbAtKzCBCDQ+5mYeJa9XR6pJEREQaJJ0D1IAEBweRbfj2wrkyD1tcjYiISMOlANTAuGyxABQe092gRUREqksBqIEpCPEFILfaYYiIiFSbAlADU2z3tcMoy8uwuBIREZGGSwGogfGENfc9UT8wERGRalMAamDMqOPtMIoUgERERKpLAaiBCY6OByDUfcziSkRERBouBaAGJsSZAEBkqQKQiIhIdSkANTCRsb5+YNGeHGsLERERacAUgBqY6ObnARBj5mJ6yiyuRkREpGFSAGpgYpv7DoEFGSYFrkyLqxEREWmYFIAamIjwcLLNaAByMw5ZXI2IiEjDpADUAOXafP3ACo6lWVyJiIhIw6QA1ADlBx/vB5ZzxOJKREREGiYFoAaoKPR4OwyX+oGJiIhUhwJQA1QaFgeAN093gxYREakOBaAGyBvha4cRpHYYIiIi1aIA1ADZjvcDCynW3aBFRESqw9IANHfuXHr06IHD4cDhcDB48GA+/vjjsy6zZs0a+vbtS1hYGB06dODvf//7aWOWLFlCt27dsNvtdOvWjaVLl9bVJlgi2OG7F1B4SZbFlYiIiDRMlgag1q1b89RTT/H111/z9ddfc/nllzNhwgS+//77Csfv27ePK664gksvvZQtW7bw4IMP8tvf/pYlS5b4x2zYsIGJEycyadIktm7dyqRJk7jhhhvYuHFjfW1WnQuP9QWg2LIMME2LqxEREWl4DNMMrL+gcXFxPPvss0ydOvW09x544AGWLVvGjh07/POmTZvG1q1b2bBhAwATJ07E5XKV25M0duxYYmNjWbhwYaVqcLlcOJ1OcnNzcTgcNdyi2nc44yjOv3Un0nBz7Ib3ies23OqSRERELFeVv98Bcw6Qx+Nh0aJFFBQUMHjw4ArHbNiwgdGjR5ebN2bMGL7++mtKS0vPOmb9+vVn/Gy3243L5So3BbJW8S3YEHE5AJnJpx8CFBERkbOzPABt27aNqKgo7HY706ZNY+nSpXTr1q3Csenp6bRs2bLcvJYtW1JWVkZmZuZZx6Snn/meObNmzcLpdPqnpKSkGm5V3TP7TgGgXcZKzAL1BBMREakKywNQly5dSElJ4csvv+T222/npptuYvv27WccbxhGudcnjuCdPL+iMafOO9mMGTPIzc31TwcPHqzOptSrIZeO5HuzPaGUcXD1PKvLERERaVAsD0ChoaF06tSJfv36MWvWLHr27MkLL7xQ4diEhITT9uRkZGQQHBxMs2bNzjrm1L1CJ7Pb7f4r0U5MgS7SHszO864DIOzbN3QytIiISBVYHoBOZZombre7wvcGDx7MihUrys379NNP6devHyEhIWcdM2TIkLop2ELth99EvhlGfMlBinYnW12OiIhIg2FpAHrwwQf5/PPP2b9/P9u2beOhhx4iOTmZG2+8EfAdmpo8ebJ//LRp0zhw4AD33nsvO3bs4NVXX2XevHncd999/jF33303n376KU8//TQ7d+7k6aefZuXKldxzzz31vXl1rnenJFaFXAbA0dU6GVpERKSyLA1AR44cYdKkSXTp0oURI0awceNGli9fzqhRowBIS0sjNTXVP759+/Z89NFHJCcn06tXLx577DHmzJnDdddd5x8zZMgQFi1axPz58+nRowcLFixg8eLFDBw4sN63r64ZhoG7500AJKathHy1xhAREamMgLsPUCAI9PsAnSzDVUza/w2hp+1HsgY/RLMx91tdkoiIiCUa5H2ApHriHWFsbnE1ALbNr4HXa21BIiIiDYACUCPQ+tJf4zLDiXX/hOfHZKvLERERCXgKQI3AZRe2Y7ltKACZa/5hcTUiIiKBTwGoEQgNtpFzwa8BaP7TCsg7YnFFIiIigU0BqJG4bOhwNns7EYSHgq9es7ocERGRgKYA1Eh0SYjmc+d4ADybFuhkaBERkbNQAGpE4gf9EpcZgaP4EOaPq6wuR0REJGApADUiV/TpyPum72To3HWvWFyNiIhI4FIAakSc4SEc7jgRgOgDK8GVZnFFIiIigUkBqJEZevFQNnnPJwgPpd+8YXU5IiIiAUkBqJEZ1KEZy+3jACj9aj54PRZXJCIiEngUgBoZm80gtt8vyDEjiSg6DDoZWkRE5DQKQI3Q1QM6ssTjOxm6cMM/La5GREQk8CgANUKtYyP4ofV1AITtXQmuwxZXJCIiElgUgBqpiwdfzEZvV2x48H7zutXliIiIBBQFoEZqdLeWLLWNBqBk0wKdDC0iInISBaBGKiwkiPCe13DMjCKsMA12r7C6JBERkYChANSIXTegI+96LgOg9Kt5FlcjIiISOBSAGrHurRx8Ffc/AAT9uBJyf7K4IhERkcCgANSIGYbBkAEDWe/phg0vbNbJ0CIiIqAA1Ohd3fs8FpsjASjdtAA8ZdYWJCIiEgAUgBq5uMhQvF2vJNN0EFJ4BHZ/YnVJIiIillMAagKu7d/BfzK0Z9OrFlcjIiJiPQWgJmBo5xasDB8LgO3HzyD7gMUViYiIWEsBqAkIshkM7NePzz0XYmDqZGgREWnyFICaiF/0TeItzwgAPN+8Dp5SiysSERGxjgJQE9GueSQ5bUZx1HQSVJgBuz62uiQRERHLKAA1Idf1b8/bx0+GNr+Zb3E1IiIi1lEAakKuuCiBD2yj8JoGxo+r4Ng+q0sSERGxhAJQExIRGkyfnr343HuRb8bm16wtSERExCIKQE3ML/r9fDK0d/ObUFZicUUiIiL1TwGoienTJoZ9cZdwxIzBVngUdn1odUkiIiL1ztIANGvWLPr37090dDTx8fFcffXV7Nq166zLTJkyBcMwTpu6d+/uH7NgwYIKxxQXF9f1JgU8wzC4tn97FnuG+WZ8rZOhRUSk6bE0AK1Zs4bp06fz5ZdfsmLFCsrKyhg9ejQFBQVnXOaFF14gLS3NPx08eJC4uDh+8YtflBvncDjKjUtLSyMsLKyuN6lBuLb3ebzjvRyvacC+NZD1o9UliYiI1KtgKz98+fLl5V7Pnz+f+Ph4vvnmG4YOHVrhMk6nE6fT6X/9/vvvk52dzc0331xunGEYJCQkVKoOt9uN2+32v3a5XJXdhAYp3hFGly7dSN7Tk8uDUuCbBTD6MavLEhERqTcBdQ5Qbm4uAHFxcZVeZt68eYwcOZK2bduWm5+fn0/btm1p3bo1V111FVu2bDnjOmbNmuUPVk6nk6SkpOptQAPyi35JLPRcDoD59XzI2GlxRSIiIvUnYAKQaZrce++9XHLJJVx44YWVWiYtLY2PP/6YW265pdz8rl27smDBApYtW8bChQsJCwvj4osvZvfu3RWuZ8aMGeTm5vqngwcP1nh7At3lXePZGj6Qjd6uGCV58NYvID/D6rJERETqhWGapml1EQDTp0/nww8/ZN26dbRu3bpSy8yaNYvZs2dz+PBhQkNDzzjO6/XSp08fhg4dypw5c865XpfLhdPpJDc3F4fDUeltaGie+HA773z+LZ9E/YWWZYfgvH4w5b8QEm51aSIiIlVWlb/fAbEH6K677mLZsmWsXr260uHHNE1effVVJk2adNbwA2Cz2ejfv/8Z9wA1VTf0SyKHaCYW3EtxsBMOfQ1L/x94vVaXJiIiUqcsDUCmaXLnnXfy3nvvsWrVKtq3b1/pZdesWcOePXuYOnVqpT4nJSWFxMTEmpTb6HRuGc2DV3Rlv5nI5ILf4jGCYfsHsOovVpcmIiJSpywNQNOnT+fNN9/krbfeIjo6mvT0dNLT0ykqKvKPmTFjBpMnTz5t2Xnz5jFw4MAKzxd69NFH+eSTT9i7dy8pKSlMnTqVlJQUpk2bVqfb0xDdNrQjdwzryFfmBdxXcptv5rrnYPMb1hYmIiJShyy9DH7u3LkADBs2rNz8+fPnM2XKFMB3onNqamq593Nzc1myZAkvvPBChevNycnhtttuIz09HafTSe/evVm7di0DBgyo9W1oDP4wpguu4lLe/PISOtjSuSvoPfjvPRCTBB2GWV2eiIhIrQuYk6ADSVM5CfpkXq/JPYtTWLb1EC+Gvsx42xdgd8LUTyG+q9XliYiInFODOwlarGezGcy+oSeXd23JfSW38o3ZFdy5xy+PP2p1eSIiIrVKAUj8QoJsvHxjH3q2T+AW9z0cJAFyUmHRr6C06NwrEBERaSAUgKScsJAg5t3Uj/POa81k9x9wEQU/bYKl03R5vIiINBoKQHKa6LAQXrt5ALbmnbjV/TtKCYbt78Mq9QsTEZHGQQFIKtQsys4bUwfyk7MPD5QcbzWy7q+w5U1rCxMREakFCkByRq1iwnnzloGsjRjJC2XXAGD+527Yu8biykRERGpGAUjOqn3zSF7/zUD+FfxLPvAMwfCWYS7+NRzdZXVpIiIi1aYAJOfUrZWD+VMG8AjT2OQ9H8Ptwvy3Lo8XEZGGSwFIKqVfuzhenDSEOz2/54A3HiPnAOai/4XSYqtLExERqTIFIKm0y85vwZ8mXsbUsj+Qa0Zg/PQVvH+7Lo8XEZEGRwFIquTKHonccvVY/l/pvZSYQfD9e7D6cavLEhERqRIFIKmyXw5ow+Vjr2VG6a2+GZ/P1uXxIiLSoCgASbXcNrQjCZfdzItlVwPgXabL40VEpOFQAJJqu290F472/T3/8QzCZpbhfeMaeP8OyN5vdWkiIiJnpQAk1WYYBjMnXETyBY/ykWcANtMDKf/GfLEv/OduyP3J6hJFREQqpAAkNWKzGTw1cQBre83mGvejrPVchOEtg28WYM7pDR/9AfLSrS5TRESkHMM0TdPqIgKNy+XC6XSSm5uLw+GwupwGY+PeLB5+/ztijn7N70PeYZBth++N4DDofwtcfA9EtbC0RhERabyq8vdbAagCCkDVV1Lm5V/r9jLnsx/o7fmO+4Lfoa/tB9+bIZEw8DYY8luIiLO2UBERaXQUgGpIAajmDh4r5NH/fM/KHUe4zPYtf7Qv4QJzj+/N0GgYfAcMugPCYyytU0REGg8FoBpSAKo9n36fzsxl33M4t4iRts38Oep9kkp+9L0Z5oQhd8HAaWCPtrZQERFp8BSAakgBqHYVlpTxwme7mff5PjxeD/8T8g1/jv6AuMK9vgHhcXDJPdD/VgiNsLRWERFpuBSAakgBqG7sSs/jkfe/46v9x7Dh5bbYLdwTsoQw137fgMh4uPRe6HszhIRZWquIiDQ8CkA1pABUd0zT5N1vfmLWxzs5VlBCEB6e7Lid6/PfIij3gG9QVEsYcBv0+41OlhYRkUpTAKohBaC6l11QwtPLd7Jo00EAmocbzL1wJ/0OzMNwHb+BYkgE9P41DLod4jpYWK2IiDQECkA1pABUf745cIyHln7HzvQ8AAa2ieLprj/S9odXMdK3HR9lwAVX+S6fTxpgXbEiIhLQFIBqSAGofpV5vCxYv5+/rviBwhIPAC2jQ7m19UGuLlpK8/S1Pw9uPcB35VjXK8EWZFHFIiISiBSAakgByBppuUU8s3wXy79Lp6jU45/fLfgQDzg/45LCzwgyS30zY9vD4OnQ638hNNKiikVEJJAoANWQApC1iks9bNx3jFU7jvDZzgx+yi4CoAU5TA7+lJtCPsNh+g6ZmeGxGP2m+k6ajm5pZdkiImIxBaAaUgAKHKZpsicjn892ZrBqZwbfHMgm1FvE9UFrmRr0Me1sRwDw2EIo63499kvvhvgLLK5aRESsoABUQwpAgSunsIQ1Pxxl9c4M1u5Kp797I7cGf0i/E/3GgNS4Idguvovzeo/FsNksrFZEROqTAlANKQA1DGUeL1sO5rBqZwZp29Yy2vUOY2ybCDJ8P+n9ttYca3slnYb/GkfSRWAYFlcsIiJ1qSp/vy395/GsWbPo378/0dHRxMfHc/XVV7Nr166zLpOcnIxhGKdNO3fuLDduyZIldOvWDbvdTrdu3Vi6dGldbopYIDjIRv92cTwwtivP/+E2etz7AcuG/oeV0RMoNO208/5En33/wPHqpRyZ1YP09x/GTPsWlPlFRJo8SwPQmjVrmD59Ol9++SUrVqygrKyM0aNHU1BQcM5ld+3aRVpamn/q3Lmz/70NGzYwceJEJk2axNatW5k0aRI33HADGzdurMvNEYu1jo3gmhGXMvL3r+O9dwcbLnqcjSH9cZvBtCxJJSHlRYx/XEresz0o/eRPcHiLwpCISBMVUIfAjh49Snx8PGvWrGHo0KEVjklOTmb48OFkZ2cTExNT4ZiJEyficrn4+OOP/fPGjh1LbGwsCxcuPGcdOgTWeJimScqeVL5f/TYtf/qEoUYKdqPU/35pdBIhF10N3a6B8/roMJmISAPWYA6BnSo3NxeAuLhz93/q3bs3iYmJjBgxgtWrV5d7b8OGDYwePbrcvDFjxrB+/foK1+V2u3G5XOUmaRwMw6B357b8+rY/0PeBj3jrslXMtN/Hh54BFJmhhOQdhPUvwr8ux3yuOyx/EFI3gtdrdekiIlKHgq0u4ATTNLn33nu55JJLuPDCC884LjExkVdeeYW+ffvidrt54403GDFiBMnJyf69Runp6bRsWf6eMC1btiQ9Pb3Cdc6aNYtHH3209jZGAlJcZCg3X94D77CL+HxPJvet34m5eyVjbRsZYdtMpOsQfPk33xSdCBf8D3SbAG0G6a7TIiKNTMAcAps+fToffvgh69ato3Xr1lVadvz48RiGwbJlywAIDQ3ltdde41e/+pV/zL///W+mTp1KcXHxacu73W7cbrf/tcvlIikpSYfAmoDDOUUs+iqV977aQ7fCTYwL+oqRts1EG0U/D4poDp1GQudR0PFydagXEQlQVTkEFhB7gO666y6WLVvG2rVrqxx+AAYNGsSbb77pf52QkHDa3p6MjIzT9gqdYLfbsdvtVf5cafhaxYRz7+gu3DWiMyu29+HNL8fywI/pXGLbxhVBXzEm6BuiCzPh20W+ybD5+pF1HgWdR0OCLq8XEWmILN0DZJomd911F0uXLiU5ObnclVxVcf3113Ps2DFWrVoF+E6CzsvL46OPPvKPGTduHDExMToJWs5pT0Y+/954gHe/+Yni4mL62n5gmC2FEUFb6WwcLD84KgE6j/SFoQ7DIMxpSc0iItKAboR4xx138NZbb/HBBx/QpUsX/3yn00l4eDgAM2bM4NChQ7z++usAPP/887Rr147u3btTUlLCm2++yVNPPcWSJUu49tprAVi/fj1Dhw7liSeeYMKECXzwwQc8/PDDrFu3joEDB56zLgUgASgq8fCfbw+zemcGG/ZmkVNYSisyGRa0leG2FC4J+o5wfj50atqCMdoMPn64bLSvJYf2DomI1JsGE4CMM/xxmD9/PlOmTAFgypQp7N+/n+TkZACeeeYZXnnlFQ4dOkR4eDjdu3dnxowZXHHFFeXW8e677/Lwww+zd+9eOnbsyBNPPOEPSOeiACSn8npNtqe52PBjFl/8mMlX+45RVlLMANtOhttSGGZLoaMtrfxCjtY/HyprPxTsUdYULyLSRDSYABSoFIDkXEo9Xr79KYcv9mSx/sdMNh/IIcGbxjBbCsNtKQy2bSfspPsNmUGhvr1DHYf7DpUl9AT1KRMRqVUKQDWkACRVVVzq4ZsD2XyxJ5P1P2ax66cMBhrbGWZL4XLbFtrYjpYbb4bHYXS4DDocD0Sxba0pXESkEVEAqiEFIKkpV3EpX+09xvofs1i/5yglGT9wqW0bl9i2Mdi2nSjjlNsxxHXwhaGOw6HdpRAeY0ndIiINmQJQDSkASW3LzHezbncmybsyWP9DOklFO7g0aBuX2L6jl7GHYOPnO0+bhg2jVZ+fD5e1HgDBodYVLyLSQCgA1ZACkNQlj9fk259ySN51lOQfjrLvp8MMMHZwiW0bl9q2nX4ydUgktLv458NlurpMRKRCCkA1pAAk9Skr383a3UdJ3nWUNT8cJbwwnUuO7x262PYdzY1TetOFOSGmre+8oZi2ENOm/GSPtmZDREQspgBUQwpAYhWP12Trib1DuzLY9lM2XY2D/r1DA4J2Ekbp2VcSHndKKDoRltqAM0mX44tIo6UAVEMKQBIoMvPdrP3Bt3do7e6jFBUW0NY4QmvjKEnGUdraMrkgIpt2QVk0K0sntCT33CuNaOYLQ827+O5T1GkEhMfW/caIiNQxBaAaUgCSQOTxmqQczGHjviw2H8hhc2o2xwpKyo2JppDejjwublFIz6hcOgb7gpEtNxVyUqG4goBkBEGbwXD+GDh/LDTvrHOMRKRBUgCqIQUgaQhM0+RAViGbU7P55kA2m1Nz2JXuwnvKf9FhITZ6nBdD77YxDEwIorczj1h3Gvy0CX74BI7uKL9AbHtfEDp/DLS9WFegiUiDoQBUQwpA0lDlu8vYejCHzQey2ZzqC0W5RaefM9QmLoJ+7WIZ1iWeYS0KcBxcBT8sh/3rwHPSXqXQaN/l+OeP9bX0iGpRj1sjIlI1CkA1pAAkjYXXa7I3s8AXho6Hot0Z+Zz8X73NgL5tfWFoRPsIuhR+jbH7E/jhUyjIOGltBrTu9/OhspYXVu5QmWn6Dr0VZkHhMSg6dvz58dcnngfboeevoOMItQkRkWpRAKohBSBpzHKLSkk5mMP6PZms3pXBD0fyy72f4AhjWJcWDO/SnEsjfyJi/0rf3qG0reVX5DjPF4Za9YZi189BpuhY+WBTlA3essoXGNsO+k2F3r+GiLiab7CINBkKQDWkACRNyU/ZhSTvOsrqnRl88WMmxaU/35U6JMhgQPs4hneJZ8R5HtodW4fxwyewNxnKiqr2QSGRvivQIuJOeWzmuwrt2F5I+ffPJ2oHh8GF10P/qXBen9rbYBFptBSAakgBSJqq4lIPG/cdY/XODFbvyuBAVmG599vERTC8Swsu7+RgsG07oT9+Ctn7fWEm/NRg43sstceSa0TjKg0it6jUP7lOep5bVIrXhA5Og8GFq+mauojwY9t//uDz+kL/W6H7NRASVr9fiog0GApANaQAJOKzL7PAH4Y27j1GiefnvUP2YBtDOjbjotYxFLjLzhhuCks81fhkkz7Gbm6xr2QUXxKC7xCaOzSG7C4TCRt8G87Ejhi6XF9ETqIAVEMKQCKnK3CXsf7HLFbvyiB5ZwaHc4vPvdBJou3BOMJDcJ46RfgevV6T1GOFHMgqZH9WARl5bgCakcvEoNXcGPwZ5xlZAHhNg7VGb1ZHT+BYwiW0bR5N22YRtGseSauYcMJDgrAH2wgLCSLIppAk0lQoANWQApDI2ZmmyQ9H8lm1M4PUY4U4woNPDzbHJ0dYCNFhwQQHVe3KrgJ32fFAVMD+rEIOZuYS91Myl+a8zwDz5xOy93tb8qZnJO94LiOX09t8BNsMwo4HohOhKPT4oz3Yhj0kiLByjzbswUHEhIfQr10cfdrGYA8OqvF3JiJ1TwGohhSARAKbO30nhV+8QtTOtwkpzQOgxAjlU9ulvOseRLHXwE4pdkp8j0bp8dfHJ6PklNe+saGU+efZDC8uM5I8I5JwRzPimrfkvMREWiUkYguP9Z24HR4DYTG+x6AQK78SEUEBqMYUgEQaiJIC2PYOfPUvOLLN2lpCo34OQycew2PA7oSgYLCdmELAFnTS6+CfXweFlH998hRsh4jmEBXvC186/0nkNApANaQAJNLAmCYc3Aib/gU/fQ1Bob7AEBxWwWNF80JPn4+BWZxLZuYR0tPTyM46SnFeFhGePGKMApwU4DQKcBiF5yyv1tlCILKF787cUS0hMv6k5y18IenEc4UlaUKq8vc7uJ5qEhGpO4YBbQb5ptpcLdDi+AS+hrTfHcrl8x+zWP9jJpv2H6OktAzH8TDkpICO0WX0j4fuzUw6RpcSZRb5bgR52uQBT2n51xWOOz6VFkHBUd99krylkHfYN52LPyzF/xyMmneG5l2gxfkQ09a3x0mkidEeoApoD5CIVIa7zMOWVN9dtdf/mEXKwRzKTulGmxQXTlhwEDbDwDDAZhjYbBx/bWA7Ps+As44JttlwhofQPMykVUg+8bZcWhi5xHhzcHiOEVl6jDB3FkGFRyE/w9fG5MRNJc8mOAyadfaFoeZdoMXxKa6jGuFKg6NDYDWkACQi1ZHvLmPT/mP+QLQ9zUV9/z9seEgQsREhxESEEh9h0jq0gPNC8kiwuWhu5NLcm0mL4gNE5+0lNHcvhsdd8YqMIIhrDy26QvPzfaGo+fm+yX761XYigUABqIYUgESkNmQXlPDDkTw8polpgreCR6//9cnPTx9T6vGSU1hKTmEJ2YUlZPuf//zo8Vbt/85teOliP0afiCN0D0mno3GIpLKDNC/eT6in4MwLOpOgWUffHb/tDghzQpjj+PMY3/MwZ/n3QqN0LpLUOQWgGlIAEpGGxjRN8txl5BSUHg9IJeQUlnKsoMQfkLILSzhWUEJGnpsjucXkuc/UpNakJdl0sh2ik3GYTrZDdAtKo4NxiFgzp3oFGrbjgehEODopNIVGQHA4hJw6RfgO0YVEnP5e8EljgkIUrgTQSdAiIk2OYRg4wnw3nmzTLKJSyxS4yzjiKibdVUyGy026q5j03GIy8opJz41lv+s8vsorprTUhFLfMk7y6WQcop1xBIdRgINCoo1CoinCYRQQTSEOo9D/6KCQEMMDpheKc3xTrW+8zbeHqVlHiO8G8Rccn7pBdKLCkVRIe4AqoD1AIiI+Xq9JdmEJ6a5ijriKOeJyk55bzNF8N0UlHgpLyigs8Rx/7qGotPy8Mq+XMErKBaITASn6+GM4JYQbJcSElNHWYdA6CuLDTeym23f1W7mpEMqKfY+m99wbEOY8KRSd9BgRV/dfntQ7HQKrIQUgEZHaUVLmpejUYFR6PCyVlHHE5WbdnkzW78mk4JTGud1bORh6fgsuO78FfdrEEhp8UjsV0wRPyc/BqDgXMndBxo6fp6w9YJ6hGW9Uy9NDUYsuYI+uw2+jEkzTd9uDE7dIML2+w4S2qrWSaaoUgGpIAUhEpH6VlHnZnJrN2h+Osnb3Ub475Cr3fmRoEEM6NfcFos4tKneYr8wNmbuPB6LtPz/mHDjzMo7zfOcWYfgOnRm2n5+f+ljBPBMbxWVeisu8BOMl2PASjIdgPBgV3d/Jc/Lr0or3atlCfIfyHK3Akeir0dHq+LzzfPOiE9WOBQWgGlMAEhGx1tE8N+v2HGXtD5ms/eEoWQUl5d5v3zySoZ19gWhQh2ZE2qtwSqs7H47u+jkUHT2+xygvrZa3oj4ZvhtdnhyKHK18z0+EpzDn8RPKIxrtHiUFoBpSABIRCRxer8n2NBdrfjjKmh+OsvlAdrkbToYEGfRpE0u7ZpE0jw6lRZSd5tF2mkfZaXH80REWjHGuk6ELj8Gxvb5Da6YJmGB68Xi9HMktIvVYAalZBRw8VshPx/LJLigBTHz7f8zjE4QHGyQ67bi9No4VmeQUeygliDLTRhnBlGHDQxBl/smGxwyilKDj820YtmAckeHERobRMaKQjvZckkJySDSyae7NJKYsk0j3EUIKj2DLS/PVXBXB4b6r70IjISTS9zzk+OvQyJ+fVzQvIu7nFiwRcQF1J/EGE4BmzZrFe++9x86dOwkPD2fIkCE8/fTTdOnS5YzLvPfee8ydO5eUlBTcbjfdu3dn5syZjBkzxj9mwYIF3HzzzactW1RURFhY2DnrUgASEQlcecWlrP8xy3+47OCxonMuExps8wWjqFB/KDo5IJ2YHx0Wwt6j+exMz2NHmosdaS52HcmjuLTiE65bx4ZzQaKDCxKifY+JDtrERWCz/Ry2PF6T3KJSsvLdZOaXkFXgJiu/xPe6wPeYlV9CVkEJmflu8orPdHuCisWEBdE5yk3niDw6hObSOjiHBI7RzJuJo/Qo4cW+oGS486q03koxbL5WK6f1o2vp2yN18vPwuDrf89RgLoNfs2YN06dPp3///pSVlfHQQw8xevRotm/fTmRkZIXLrF27llGjRvHkk08SExPD/PnzGT9+PBs3bqR3797+cQ6Hg127dpVbtjLhR0REAlt0WAhjuicwpnsCpmmyP6uQTfuOccTluzotM99NZl6J73memzx3GSVlXg7lFHEo59xhqSJhITa6JDjolhhN1wRf0OmaGI0j7Nzn3QTZDOIiQ4mLDKVzy3N/lrvMw7GCErLyfdtwNM9NhquYjDw3GS43GXk/Py/xeMkp9rCpOJhNxAKxZ1zvec4weiXa6REfTLfmwXSJs9HC7sEoLYSSAt+VdSc/njavEEryoTAL8o/4Hk2v73n+EThyjg0zgn5u4hsZDwkXwahHz/2F1JGAOgR29OhR4uPjWbNmDUOHDq30ct27d2fixIn86U9/Anx7gO655x5ycnKqVYf2AImINB7FpR6O5vmCke+x5KTnPz9m5peQ7y6jlTPMvzena6Jvz067ZpEE2QLrfkKm6duzdHIwOnJSQDp60ryi0oqvhouNCKF7KyfdWzno1spB91ZO2jev5LZ6SqEg09d3Lv+oLwQVZPh60Z3oR3fiedGx05dPGgRTP6nht1Beg9kDdKrcXF/jvri4yt+fwev1kpeXd9oy+fn5tG3bFo/HQ69evXjsscfK7SE6mdvtxu3+uR+Oy+WqcJyIiDQ8YSFBJMVFkBR37ivHSj1eQoIaxgnChmEQExFKTEQo57c88+X7pmniKipjZ7qL7w67+P5wLtsPu9idkU92YSnr9mSybk+mf3x4SBAXJEb7g1H3Vk7OT4jCHnzKuT5BIcdPtk48d7GeUig4Wj4c2a3dwRAwe4BM02TChAlkZ2fz+eefV3q5Z599lqeeeoodO3YQHx8PwJdffsmePXu46KKLcLlcvPDCC3z00Uds3bqVzp07n7aOmTNn8uijp++G0x4gERFprIpLPfxwJI/vj4ei7w+72JmWV+HeomCbQaf4KLq1chBlD6bMa+LxmHhME4/XpMxr4vWalHm9eLw/zyv/nul/z+M1uSAxmud/WfGOiepqMCdBn2z69Ol8+OGHrFu3jtatW1dqmYULF3LLLbfwwQcfMHLkyDOO83q99OnTh6FDhzJnzpzT3q9oD1BSUpICkIiINCker8m+zPzjoejnYJRTWFrrn9W7TQxL77i4VtfZ4A6B3XXXXSxbtoy1a9dWOvwsXryYqVOn8s4775w1/ADYbDb69+/P7t27K3zfbrdjt9urXLeIiEhjEmQz6BQfTaf4aCb0Og/wHaE5nFvM94dy2ZWeR6nHS5DNRnCQgc0wCLYZBNmMs7y2EXRi3kmPjnBrb9xoaQAyTZO77rqLpUuXkpycTPv27Su13MKFC/nNb37DwoULufLKKyv1OSkpKVx00UU1LVlERKRJMQyD82LCOS8mnNHdE6wup9ZYGoCmT5/OW2+9xQcffEB0dDTp6ekAOJ1OwsPDAZgxYwaHDh3i9ddfB3zhZ/LkybzwwgsMGjTIv0x4eDhOpxOARx99lEGDBtG5c2dcLhdz5swhJSWFv/3tbxZspYiIiAQaS091nzt3Lrm5uQwbNozExET/tHjxYv+YtLQ0UlNT/a//8Y9/UFZWxvTp08stc/fdd/vH5OTkcNttt3HBBRcwevRoDh06xNq1axkwYEC9bp+IiIgEpoA5CTqQ6D5AIiIiDU9V/n43jJsdiIiIiNQiBSARERFpchSAREREpMlRABIREZEmRwFIREREmhwFIBEREWlyFIBERESkyVEAEhERkSZHAUhERESaHAUgERERaXIsbYYaqE50B3G5XBZXIiIiIpV14u92Zbp8KQBVIC8vD4CkpCSLKxEREZGqysvLw+l0nnWMmqFWwOv1cvjwYaKjozEMo1bX7XK5SEpK4uDBg2q0Wof0PdcPfc/1Q99z/dF3XT/q6ns2TZO8vDxatWqFzXb2s3y0B6gCNpuN1q1b1+lnOBwO/cdVD/Q91w99z/VD33P90XddP+riez7Xnp8TdBK0iIiINDkKQCIiItLkKADVM7vdzp///GfsdrvVpTRq+p7rh77n+qHvuf7ou64fgfA96yRoERERaXK0B0hERESaHAUgERERaXIUgERERKTJUQASERGRJkcBSERERJocBaB69PLLL9O+fXvCwsLo27cvn3/+udUlNTozZ87EMIxyU0JCgtVlNXhr165l/PjxtGrVCsMweP/998u9b5omM2fOpFWrVoSHhzNs2DC+//57a4ptwM71PU+ZMuW03/egQYOsKbYBmzVrFv379yc6Opr4+Hiuvvpqdu3aVW6MftM1V5nv2crftAJQPVm8eDH33HMPDz30EFu2bOHSSy9l3LhxpKamWl1ao9O9e3fS0tL807Zt26wuqcErKCigZ8+evPTSSxW+/8wzz/DXv/6Vl156iU2bNpGQkMCoUaP8jYWlcs71PQOMHTu23O/7o48+qscKG4c1a9Ywffp0vvzyS1asWEFZWRmjR4+moKDAP0a/6ZqrzPcMFv6mTakXAwYMMKdNm1ZuXteuXc0//vGPFlXUOP35z382e/bsaXUZjRpgLl261P/a6/WaCQkJ5lNPPeWfV1xcbDqdTvPvf/+7BRU2Dqd+z6ZpmjfddJM5YcIES+ppzDIyMkzAXLNmjWma+k3XlVO/Z9O09jetPUD1oKSkhG+++YbRo0eXmz969GjWr19vUVWN1+7du2nVqhXt27fnl7/8JXv37rW6pEZt3759pKenl/t92+12LrvsMv2+60BycjLx8fGcf/753HrrrWRkZFhdUoOXm5sLQFxcHKDfdF059Xs+warftAJQPcjMzMTj8dCyZcty81u2bEl6erpFVTVOAwcO5PXXX+eTTz7hn//8J+np6QwZMoSsrCyrS2u0TvyG9fuue+PGjePf//43q1atYvbs2WzatInLL78ct9ttdWkNlmma3HvvvVxyySVceOGFgH7TdaGi7xms/U0H1/kniJ9hGOVem6Z52jypmXHjxvmfX3TRRQwePJiOHTvy2muvce+991pYWeOn33fdmzhxov/5hRdeSL9+/Wjbti0ffvgh1157rYWVNVx33nkn3377LevWrTvtPf2ma8+Zvmcrf9PaA1QPmjdvTlBQ0Gn/csjIyDjtXxhSuyIjI7nooovYvXu31aU0WieustPvu/4lJibStm1b/b6r6a677mLZsmWsXr2a1q1b++frN127zvQ9V6Q+f9MKQPUgNDSUvn37smLFinLzV6xYwZAhQyyqqmlwu93s2LGDxMREq0tptNq3b09CQkK533dJSQlr1qzR77uOZWVlcfDgQf2+q8g0Te68807ee+89Vq1aRfv27cu9r9907TjX91yR+vxN6xBYPbn33nuZNGkS/fr1Y/DgwbzyyiukpqYybdo0q0trVO677z7Gjx9PmzZtyMjI4PHHH8flcnHTTTdZXVqDlp+fz549e/yv9+3bR0pKCnFxcbRp04Z77rmHJ598ks6dO9O5c2eefPJJIiIi+N///V8Lq254zvY9x8XFMXPmTK677joSExPZv38/Dz74IM2bN+eaa66xsOqGZ/r06bz11lt88MEHREdH+/f0OJ1OwsPDMQxDv+lacK7vOT8/39rftCXXnjVRf/vb38y2bduaoaGhZp8+fcpdCii1Y+LEiWZiYqIZEhJitmrVyrz22mvN77//3uqyGrzVq1ebwGnTTTfdZJqm77LhP//5z2ZCQoJpt9vNoUOHmtu2bbO26AbobN9zYWGhOXr0aLNFixZmSEiI2aZNG/Omm24yU1NTrS67wanoOwbM+fPn+8foN11z5/qerf5NG8eLFBEREWkydA6QiIiINDkKQCIiItLkKACJiIhIk6MAJCIiIk2OApCIiIg0OQpAIiIi0uQoAImIiEiTowAkIiIiTY4CkIiIiDQ5CkAiIiLS5CgAiYiISJPz/wEklcqCqpQ5twAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "And they tridcowd,\n",
      "This so be madises bube to tavegr-'t theall ands:\n",
      "Want he us hat tot?\n",
      "Wedtlas anes wice my.\n",
      "\n",
      "HDER:\n",
      "At onoth\n",
      "Youts, tof is hy me mil nowlit,\n",
      "Wheirwe sen cin lat Het drov the and the nown iserans!\n",
      " lolind teall thus, cocrivy prugh aiss hewty.\n",
      "Hllings kne\n",
      "To thig I whom.\n",
      "\n",
      "Whoul to ake onWinso whre piiby we atit,\n",
      "Crive winghience poo mo the thu the danterupt fis are;\n",
      "De! muf thre male of,\n",
      "To fis.\n",
      "Fe I So myakny, be!\n",
      "Whied is:\n",
      "Sadsal the E'd st huin couk ay andy Iry to cof my carey\n"
     ]
    }
   ],
   "source": [
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size:int=vocab_size, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 head_size:int=head_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = head_size\n",
    "\n",
    "        #This will be our lookup table for embeddings. We'll have an entry for each token (aka vocab size) and each embedding will... \n",
    "        #...be a vector of dimension embedding_dim.\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        self.token_position_embedding_table = nn.Embedding(num_embeddings=self.context_length, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        # self.self_attention_head_linear_layer = Head(head_size=head_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "        #4 heads of 8 dimensional self attention.\n",
    "        self.multi_self_attention_heads_layer = MultiHeadAttention(num_heads=4, head_size=self.embedding_dim//4) #NEW\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        B,T = idx.shape\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        token_position_embeddings = self.token_position_embedding_table(torch.arange(T, device=device)) #(T,C)\n",
    "        x = token_embeddings + token_position_embeddings\n",
    "        # x = self.self_attention_head_linear_layer(x) #apply one head of self attention\n",
    "        x = self.multi_self_attention_heads_layer(x)\n",
    "        logits = self.language_model_head_linear_layer(x) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Crop idx to the max size of our positional embeddings table \n",
    "            idx_crop = idx[:,-self.context_length:]\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx_crop)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = TransformerLanguageModel(vocab_size=vocab_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "tracked_losses = list()\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        tracked_losses.append(losses)\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "plot_losses(tracked_losses)\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25a713d5-2948-4953-a938-6f635391e884",
   "metadata": {},
   "source": [
    "As you can see there is quite an improvement in the loss, going from `Validation Loss: tensor(2.4176)` with a single attention head to `Validation Loss: tensor(2.2609)` with our multi-attention head that has 4 heads. Note, these losses may vary somewhat between training runs.  The results are still nonsense, but are looking closer to the training text than previous attempts. The reason that multi-headed attention works better than a single self attention block is that it is helpful to have multiple communication channels between tokens so they can each be looking for different things over different communication channels. As an example one communication channel make be looking back at vowels or consonants while another might be looking for the previous space.<br>\n",
    "\n",
    "If you look at this transformer block diagram, you can see that we've implemented quite a bit of it so far.<br>\n",
    "![transformer architecture diagram](./transformer_architecture_diagram2.png)<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8480d9dd-aa69-4958-82e3-d18e63667c5a",
   "metadata": {},
   "source": [
    "We've implemented the output embeddings, positional embeddings, (the lower) masked multi-head attention, and the final linear and softmax layers. We are going to skip the multi-head attention block in the middle as that is only needed if your model has an encoder block, which ours does not. This leaves the feed forward network to implement which is just a simple multi layer perceptron. In addition the entire block between the positional encodings and final linear layer can be stacked on top of itself multiple times signified by Nx.<br>\n",
    "\n",
    "Here is the equation for the feed forward network, which is a simple multi layer perceptron: <br>\n",
    "![feed_forward_network_mlp1.png](./feed_forward_network_mlp1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06227b5a-4fc2-4eff-9f89-6a1832120b32",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class FeedForwardNetwork(nn.Module):\n",
    "    \"\"\"A simple linear network followed by a non-linearity\"\"\"\n",
    "    def __init__(self, embedding_dim:int=embedding_dim):\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.ffn = nn.Sequential(\n",
    "            nn.Linear(in_features=self.embedding_dim, out_features=self.embedding_dim),\n",
    "            nn.ReLU()\n",
    "        )\n",
    "    def forward(self, x):\n",
    "        return self.ffn(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92175c34-9af7-42f3-ade9-ee069c8d2044",
   "metadata": {},
   "source": [
    "Note: In the equation it defines a (linear layer), (relu), and (linear layer). We'll add the final linear layer later.<br>\n",
    "Now let's add our FFN to our Transformer Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5c8f034-c4dd-47cf-92dd-090996c53782",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.2022) Validation Loss: tensor(4.2019)\n",
      "Step: 200 Training Loss: tensor(2.9494) Validation Loss: tensor(2.9685)\n",
      "Step: 400 Training Loss: tensor(2.6759) Validation Loss: tensor(2.6864)\n",
      "Step: 600 Training Loss: tensor(2.5779) Validation Loss: tensor(2.5799)\n",
      "Step: 800 Training Loss: tensor(2.5171) Validation Loss: tensor(2.5197)\n",
      "Step: 1000 Training Loss: tensor(2.4739) Validation Loss: tensor(2.4704)\n",
      "Step: 1200 Training Loss: tensor(2.4210) Validation Loss: tensor(2.4257)\n",
      "Step: 1400 Training Loss: tensor(2.4079) Validation Loss: tensor(2.4105)\n",
      "Step: 1600 Training Loss: tensor(2.3843) Validation Loss: tensor(2.3845)\n",
      "Step: 1800 Training Loss: tensor(2.3682) Validation Loss: tensor(2.3731)\n",
      "Step: 2000 Training Loss: tensor(2.3387) Validation Loss: tensor(2.3475)\n",
      "Step: 2200 Training Loss: tensor(2.3342) Validation Loss: tensor(2.3500)\n",
      "Step: 2400 Training Loss: tensor(2.3180) Validation Loss: tensor(2.3127)\n",
      "Step: 2600 Training Loss: tensor(2.3176) Validation Loss: tensor(2.3160)\n",
      "Step: 2800 Training Loss: tensor(2.2881) Validation Loss: tensor(2.3087)\n",
      "Step: 3000 Training Loss: tensor(2.2834) Validation Loss: tensor(2.3059)\n",
      "Step: 3200 Training Loss: tensor(2.2796) Validation Loss: tensor(2.2901)\n",
      "Step: 3400 Training Loss: tensor(2.2719) Validation Loss: tensor(2.2743)\n",
      "Step: 3600 Training Loss: tensor(2.2675) Validation Loss: tensor(2.2681)\n",
      "Step: 3800 Training Loss: tensor(2.2428) Validation Loss: tensor(2.2751)\n",
      "Step: 4000 Training Loss: tensor(2.2294) Validation Loss: tensor(2.2524)\n",
      "Step: 4200 Training Loss: tensor(2.2468) Validation Loss: tensor(2.2545)\n",
      "Step: 4400 Training Loss: tensor(2.2373) Validation Loss: tensor(2.2437)\n",
      "Step: 4600 Training Loss: tensor(2.2310) Validation Loss: tensor(2.2448)\n",
      "Step: 4800 Training Loss: tensor(2.2182) Validation Loss: tensor(2.2522)\n",
      "Step: 4999 Training Loss: tensor(2.2135) Validation Loss: tensor(2.2291)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGxCAYAAACKvAkXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABeEklEQVR4nO3deXhU5d3/8feZLJN1JglbgoQAgiDIHpag4sImKg9ULdj6oCjoD4tW5bEqWlutVVxKi9TWrQouFVADolURKLIJKihBREWQJSwJYUkyWSfJzPn9MWEkJEDIdibJ53Vd58rMmfuc+Z65Rubjfc65b8M0TRMRERGRZsRmdQEiIiIiDU0BSERERJodBSARERFpdhSAREREpNlRABIREZFmRwFIREREmh0FIBEREWl2FIBERESk2VEAEhERkWZHAUhEAsa8efMwDINNmzZZXYqINHEKQCIiItLsKACJiIhIs6MAJCKNyrp16xg2bBjR0dFEREQwZMgQPvzwwwptCgsLuffee+nYsSNhYWHExcWRnJzM/Pnz/W127drF9ddfT9u2bbHb7bRp04Zhw4aRlpZWYV8LFy4kJSWFyMhIoqKiGDVqFJs3b67Qprr7EpHAEWx1ASIi1bV69WpGjBhBr169eOWVV7Db7fzzn/9kzJgxzJ8/nwkTJgAwffp03njjDf785z/Tt29fCgoK+Pbbbzl69Kh/X1deeSUej4enn36a9u3bc+TIEdavX09OTo6/zRNPPMHvf/97br75Zn7/+99TUlLCM888w8UXX8yXX35J9+7dq70vEQkshmmaptVFiIiA7yLom2++mY0bN5KcnFzp9ZSUFHbt2sVPP/1EVFQUAB6Phz59+pCTk0N6ejqGYdCzZ086d+7M4sWLq3yfo0eP0rJlS2bPns1dd91VZZt9+/bRqVMnbr/9dubMmeNfn5+fT5cuXRg6dCgLFy6s1r5EJPDoFJiINAoFBQV88cUXXHfddf7wAxAUFMTEiRPZv38/27dvB2DgwIF8/PHHPPDAA6xatYqioqIK+4qLi+Pcc8/lmWee4a9//SubN2/G6/VWaPPJJ59QVlbGjTfeSFlZmX8JCwvjkksuYdWqVdXel4gEHgUgEWkUsrOzMU2ThISESq+1bdsWwH+Ka86cOdx///289957XHbZZcTFxTFu3Dh27NgBgGEY/Pe//2XUqFE8/fTT9OvXj1atWvHb3/6WvLw8AA4dOgTAgAEDCAkJqbAsXLiQI0eOVHtfIhJ4dA2QiDQKsbGx2Gw2MjIyKr128OBBAFq2bAlAZGQkjz76KI8++iiHDh3y9waNGTOGH374AYCkpCReeeUVAH788UfefvttHnnkEUpKSnjhhRf8+3r33XdJSko6bW1n2peIBB5dAyQiAeNM1wANGTKE3bt3s2vXLsLDwwHwer306dOH7Oxs/zVAVbnnnnuYPXs2BQUFREREVNmmb9++hISE8OWXX7Jnzx66dOnC448/zn333XfWx3LivkQk8KgHSEQCzsqVK9mzZ0+l9TNnzmTEiBFcdtll3HvvvYSGhvLPf/6Tb7/9lvnz5/vDz6BBg7j66qvp1asXsbGxfP/997zxxhukpKQQERHBN998wx133MEvf/lLunTpQmhoKCtXruSbb77hgQceAKBDhw786U9/4qGHHmLXrl1cccUVxMbGcujQIb788kt/L1N19iUiAcgUEQkQc+fONYFTLrt37zbXrl1rXn755WZkZKQZHh5uDh482Pzggw8q7OeBBx4wk5OTzdjYWNNut5udOnUy77nnHvPIkSOmaZrmoUOHzEmTJpndunUzIyMjzaioKLNXr17m3/72N7OsrKzCvt577z3zsssuMx0Oh2m3282kpCTzuuuuM1esWHHW+xKRwKFTYCIiItLs6C4wERERaXYUgERERKTZUQASERGRZkcBSERERJodBSARERFpdhSAREREpNnRQIhV8Hq9HDx4kOjo6FOOKisiIiKBxTRN8vLyaNu2LTbb6ft4FICqcPDgQRITE60uQ0RERGpg3759tGvX7rRtFICqEB0dDfg+QIfDYXE1IiIiUh0ul4vExET/7/jpKABV4fhpL4fDoQAkIiLSyFTn8hVdBC0iIiLNjgKQiIiINDsBE4BmzpyJYRjcfffdp2yzaNEiRowYQatWrXA4HKSkpPDJJ59UaDNv3jwMw6i0FBcX1/MRiIiISGMRENcAbdy4kZdeeolevXqdtt2aNWsYMWIETzzxBDExMcydO5cxY8bwxRdf0LdvX387h8PB9u3bK2wbFhZWL7WLiEhlpmlSVlaGx+OxuhRpYkJCQggKCqr1fiwPQPn5+dxwww28/PLL/PnPfz5t29mzZ1d4/sQTT7BkyRI++OCDCgHIMAzi4+OrXYPb7cbtdvufu1yuam8rIiIVlZSUkJGRQWFhodWlSBNkGAbt2rUjKiqqVvuxPABNmzaNq666iuHDh58xAJ3M6/WSl5dHXFxchfX5+fkkJSXh8Xjo06cPjz32WIWAdLKZM2fy6KOP1qh+ERH5mdfrZffu3QQFBdG2bVtCQ0M1oKzUGdM0OXz4MPv376dLly616gmyNAAtWLCAr7/+mo0bN9Zo+1mzZlFQUMD48eP967p168a8efPo2bMnLpeLZ599lgsvvJAtW7bQpUuXKvczY8YMpk+f7n9+fBwBERE5OyUlJXi9XhITE4mIiLC6HGmCWrVqxZ49eygtLW2cAWjfvn3cddddLFu2rEbX58yfP59HHnmEJUuW0Lp1a//6wYMHM3jwYP/zCy+8kH79+vH3v/+dOXPmVLkvu92O3W4/+4MQEZEqnWkaApGaqqseRcsC0FdffUVWVhb9+/f3r/N4PKxZs4bnnnsOt9t9ymS3cOFCJk+ezDvvvMPw4cNP+z42m40BAwawY8eOOq1fREREGi/LAtCwYcPYunVrhXU333wz3bp14/777z9l+Jk/fz633HIL8+fP56qrrjrj+5imSVpaGj179qyTukVERKTxsywARUdHc8EFF1RYFxkZSYsWLfzrZ8yYwYEDB3j99dcBX/i58cYbefbZZxk8eDCZmZkAhIeH43Q6AXj00UcZPHgwXbp0weVyMWfOHNLS0vjHP/7RgEcnIiLN3aWXXkqfPn0q3cF8Knv27KFjx45s3ryZPn361GttEkADIVYlIyOD9PR0//MXX3yRsrIypk2bRkJCgn+56667/G1ycnK47bbbOP/88xk5ciQHDhxgzZo1DBw40IpDEBGRAFfV4LknLpMmTarRfhctWsRjjz1W7faJiYlkZGRU6hyoa3v27MEwDNLS0ur1fQKdYZqmaXURgcblcuF0OsnNza3TyVD37/qeXf/9F0ZIBBdPqv5/FCIijUVxcTG7d++mY8eOjWYA2uNnE8B3jekf/vCHCoPpnniWAaC0tJSQkJAGrbEuNfaeptN9x87m9zuge4CamvwjBxh64F902rvQ6lJERBqEaZoUlpRZslT3/+/j4+P9i9Pp9A+mGx8fT3FxMTExMbz99ttceumlhIWF8eabb3L06FF+9atf0a5dOyIiIujZsyfz58+vsN9LL720wvROHTp04IknnuCWW24hOjqa9u3b89JLL/lfP7lnZtWqVRiGwX//+1+Sk5OJiIhgyJAhlWY6+POf/0zr1q2Jjo5mypQpPPDAA7UKNm63m9/+9re0bt2asLAwLrroogrD1WRnZ3PDDTfQqlUrwsPD6dKlC3PnzgV8wyDccccdJCQkEBYWRocOHZg5c2aNa6lPlg+E2JxExflGp47x5lpciYhIwygq9dD9D5+cuWE9+O5Po4gIrZufufvvv59Zs2Yxd+5c7HY7xcXF9O/fn/vvvx+Hw8GHH37IxIkT6dSpE4MGDTrlfmbNmsVjjz3Ggw8+yLvvvsvtt9/O0KFD6dat2ym3eeihh5g1axatWrVi6tSp3HLLLXz22WcA/Pvf/+bxxx/nn//8JxdeeCELFixg1qxZdOzYscbHet9995Gamsprr71GUlISTz/9NKNGjWLnzp3ExcXx8MMP89133/Hxxx/TsmVLdu7cSVFREQBz5szh/fff5+2336Z9+/bs27ePffv21biW+qQA1ICcrdoCEGkUU1jgIiKy7k6viYhI/bn77ru55pprKqy79957/Y/vvPNOli5dyjvvvHPaAHTllVfym9/8BvCFqr/97W+sWrXqtAHo8ccf55JLLgHggQce4KqrrqK4uJiwsDD+/ve/M3nyZG6++WYA/vCHP7Bs2TLy8/NrdJwFBQU8//zzzJs3j9GjRwPw8ssvs3z5cl555RV+97vfkZ6eTt++fUlOTgZ8PVvHpaen06VLFy666CIMwyApKalGdTQEBaAGFBUdg9sMwW6UknM4QwFIRJq88JAgvvvTKMveu64c/7E/zuPx8OSTT7Jw4UIOHDjgn1MyMjLytPs5cdLv46fasrKyqr1NQkICAFlZWbRv357t27f7A9VxAwcOZOXKldU6rpP99NNPlJaWcuGFF/rXhYSEMHDgQL7//nsAbr/9dq699lq+/vprRo4cybhx4xgyZAgAkyZNYsSIEXTt2pUrrriCq6++mpEjR9aolvqma4AakGGzkW34LqTLP5ZhcTUiIvXPMAwiQoMtWepyDrKTg82sWbP429/+xn333cfKlStJS0tj1KhRlJSUnHY/J188bRgGXq+32tscP6YTtzn5OGtzb9Pxbava5/F1o0ePZu/evdx9990cPHiQYcOG+XvD+vXrx+7du3nssccoKipi/PjxXHfddTWupz4pADWwvKBYAAqzM8/QUkREAtXatWsZO3Ys//u//0vv3r3p1KmTJTMOdO3alS+//LLCuk2bNtV4f507dyY0NJR169b515WWlrJp0ybOP/98/7pWrVoxadIk3nzzTWbPnl3hYm6Hw8GECRN4+eWXWbhwIampqRw7dqzGNdUXnQJrYIWhsVAEpbmHrC5FRERqqHPnzqSmprJ+/XpiY2P561//SmZmZoWQ0BDuvPNObr31VpKTkxkyZAgLFy7km2++oVOnTmfc9uS7yQC6d+/O7bffzu9+9zvi4uJo3749Tz/9NIWFhUyePBnwXWfUv39/evTogdvt5j//+Y//uP/2t7+RkJBAnz59sNlsvPPOO8THxxMTE1Onx10XFIAamDs0DorAk3fY6lJERKSGHn74YXbv3s2oUaOIiIjgtttuY9y4ceTmNuxdvjfccAO7du3i3nvvpbi4mPHjxzNp0qRKvUJVuf766yut2717N08++SRer5eJEyeSl5dHcnIyn3zyCbGxvjMYoaGhzJgxgz179hAeHs7FF1/MggULAIiKiuKpp55ix44dBAUFMWDAAD766KOAnBxXAyFWob4GQgTY8OI0UjLe5PM21zP49hfrdN8iIlZrjAMhNjUjRowgPj6eN954w+pS6kVdDYSoHqCGFtEKgJCiIxYXIiIijV1hYSEvvPACo0aNIigoiPnz57NixQqWL19udWkBTwGogQVFtwbA7j5qcSUiItLYGYbBRx99xJ///Gfcbjddu3YlNTWV4cOHW11awFMAamChMW0AiCjLtrgSERFp7MLDw1mxYoXVZTRKgXdVUhMXEeubDsPhybG2EBERkWZMAaiBRbfwTYcRY7owvR6LqxEREWmeFIAaWGxLXw9QsOElL0cXQouIiFhBAaiBhYWFk2v6hlTPPXzQ4mpERESaJwUgC+TYYgAo0HxgIiIillAAskB+sG80zeIczQcmIiJiBQUgCxSFxAFQmpdlcSUiIlJXLr30Uu6++27/8w4dOjB79uzTbmMYBu+9916t37uu9tOcKABZoDSsBQBezQcmImK5MWPGnHLgwA0bNmAYBl9//fVZ73fjxo3cdttttS2vgkceeYQ+ffpUWp+RkcHo0aPr9L1ONm/evICc1LSmFIAs4I1oCYCtUAFIRMRqkydPZuXKlezdu7fSa6+++ip9+vShX79+Z73fVq1aERERURclnlF8fDx2u71B3qupUACygBFVPh+Y+5jFlYiI1DPThJICa5ZqzvV99dVX07p1a+bNm1dhfWFhIQsXLmTy5MkcPXqUX/3qV7Rr146IiAh69uzJ/PnzT7vfk0+B7dixg6FDhxIWFkb37t2rnK/r/vvv57zzziMiIoJOnTrx8MMPU1paCvh6YB599FG2bNmCYRgYhuGv+eRTYFu3buXyyy8nPDycFi1acNttt5Gfn+9/fdKkSYwbN46//OUvJCQk0KJFC6ZNm+Z/r5pIT09n7NixREVF4XA4GD9+PIcOHfK/vmXLFi677DKio6NxOBz079+fTZs2AbB3717GjBlDbGwskZGR9OjRg48++qjGtVSHpsKwQHD5fGDhmg9MRJq60kJ4oq017/3gQQiNPGOz4OBgbrzxRubNm8cf/vAHDMMA4J133qGkpIQbbriBwsJC+vfvz/3334/D4eDDDz9k4sSJdOrUiUGDBp3xPbxeL9dccw0tW7bk888/x+VyVbhe6Ljo6GjmzZtH27Zt2bp1K7feeivR0dHcd999TJgwgW+//ZalS5f6p79wOp2V9lFYWMgVV1zB4MGD2bhxI1lZWUyZMoU77rijQsj79NNPSUhI4NNPP2Xnzp1MmDCBPn36cOutt57xeE5mmibjxo0jMjKS1atXU1ZWxm9+8xsmTJjAqlWrALjhhhvo27cvzz//PEFBQaSlpRESEgLAtGnTKCkpYc2aNURGRvLdd98RFRV11nWcDQUgC4TF+AZDjPJoPjARkUBwyy238Mwzz7Bq1Souu+wywHf665prriE2NpbY2Fjuvfdef/s777yTpUuX8s4771QrAK1YsYLvv/+ePXv20K5dOwCeeOKJStft/P73v/c/7tChA//3f//HwoULue+++wgPDycqKorg4GDi4+NP+V7//ve/KSoq4vXXXycy0hcAn3vuOcaMGcNTTz1Fmza+OSljY2N57rnnCAoKolu3blx11VX897//rVEAWrFiBd988w27d+8mMTERgDfeeIMePXqwceNGBgwYQHp6Or/73e/o1q0bAF26dPFvn56ezrXXXkvPnj0B6NSp01nXcLYUgCwQGZcAgNOba3ElIiL1LCTC1xNj1XtXU7du3RgyZAivvvoql112GT/99BNr165l2bJlAHg8Hp588kkWLlzIgQMHcLvduN1uf8A4k++//5727dv7ww9ASkpKpXbvvvsus2fPZufOneTn51NWVobD4aj2cRx/r969e1eo7cILL8Tr9bJ9+3Z/AOrRowdBQUH+NgkJCWzduvWs3uvE90xMTPSHH4Du3bsTExPD999/z4ABA5g+fTpTpkzhjTfeYPjw4fzyl7/k3HPPBeC3v/0tt99+O8uWLWP48OFce+219OrVq0a1VJeuAbKAo6UvAEVRiKekyOJqRETqkWH4TkNZsZSfyqquyZMnk5qaisvlYu7cuSQlJTFs2DAAZs2axd/+9jfuu+8+Vq5cSVpaGqNGjaKkpKRa+zaruB7JOKm+zz//nOuvv57Ro0fzn//8h82bN/PQQw9V+z1OfK+T913Vex4//XTia16v96ze60zveeL6Rx55hG3btnHVVVexcuVKunfvzuLFiwGYMmUKu3btYuLEiWzdupXk5GT+/ve/16iW6lIAskBsbEtKTF/qzj2i0aBFRALB+PHjCQoK4q233uK1117j5ptv9v94r127lrFjx/K///u/9O7dm06dOrFjx45q77t79+6kp6dz8ODPvWEbNmyo0Oazzz4jKSmJhx56iOTkZLp06VLpzrTQ0FA8ntNPpN29e3fS0tIoKCiosG+bzcZ5551X7ZrPxvHj27dvn3/dd999R25uLueff75/3Xnnncc999zDsmXLuOaaa5g7d67/tcTERKZOncqiRYv4v//7P15++eV6qfU4BSALBAcHkW34LlxzHVUAEhEJBFFRUUyYMIEHH3yQgwcPMmnSJP9rnTt3Zvny5axfv57vv/+e//f//h+ZmdUfzX/48OF07dqVG2+8kS1btrB27VoeeuihCm06d+5Meno6CxYs4KeffmLOnDn+HpLjOnTowO7du0lLS+PIkSO43e5K73XDDTcQFhbGTTfdxLfffsunn37KnXfeycSJE/2nv2rK4/GQlpZWYfnuu+8YPnw4vXr14oYbbuDrr7/myy+/5MYbb+SSSy4hOTmZoqIi7rjjDlatWsXevXv57LPP2Lhxoz8c3X333XzyySfs3r2br7/+mpUrV1YITvVBAcgirvL5wAqzNSGqiEigmDx5MtnZ2QwfPpz27dv71z/88MP069ePUaNGcemllxIfH8+4ceOqvV+bzcbixYtxu90MHDiQKVOm8Pjjj1doM3bsWO655x7uuOMO+vTpw/r163n44YcrtLn22mu54ooruOyyy2jVqlWVt+JHRETwySefcOzYMQYMGMB1113HsGHDeO65587uw6hCfn4+ffv2rbBceeWV/tvwY2NjGTp0KMOHD6dTp04sXLgQgKCgII4ePcqNN97Ieeedx/jx4xk9ejSPPvoo4AtW06ZN4/zzz+eKK66ga9eu/POf/6x1vadjmFWdmGzmXC4XTqeT3Nzcs774rLrSZl5OH/dXbO73BH3/Z1q9vIeISEMrLi5m9+7ddOzYkbCwMKvLkSbodN+xs/n9Vg+QRYpDfdNhePIOnaGliIiI1LWACUAzZ87EMIwqB4Y60erVq+nfvz9hYWF06tSJF154oVKb1NRUunfvjt1ur3CVeSApK58PjIIj1hYiIiLSDAVEANq4cSMvvfTSGe/53717N1deeSUXX3wxmzdv5sEHH+S3v/0tqamp/jYbNmxgwoQJTJw4kS1btjBx4kTGjx/PF198Ud+HcVbMSN90GLZCBSAREZGGZnkAys/P54YbbuDll18mNjb2tG1feOEF2rdvz+zZszn//POZMmUKt9xyC3/5y1/8bWbPns2IESOYMWMG3bp1Y8aMGQwbNqzCfCyBwBbtC0B2zQcmIiLS4CwPQNOmTeOqq65i+PDhZ2y7YcMGRo4cWWHdqFGj2LRpk38Ct1O1Wb9+/Sn363a7cblcFZb6Furw3YoYUar5wESk6dH9NVJf6uq7ZWkAWrBgAV9//TUzZ86sVvvMzMxKYxi0adOGsrIyjhw5cto2pxuvYebMmTidTv9y4lDe9SUsxldjlCen3t9LRKShHB9duLCw0OJKpKk6PjL2idN41IRlc4Ht27ePu+66i2XLlp3VrZInD7V9PAmeuL6qNqcaFhxgxowZTJ8+3f/c5XLVewiKLp8PLMbMBdM86yHbRUQCUVBQEDExMWRlZQG+MWlO9++vyNnwer0cPnyYiIgIgoNrF2EsC0BfffUVWVlZ9O/f37/O4/GwZs0annvuOdxud6V0Fx8fX6knJysri+DgYFq0aHHaNqcb/dJut2O322t7SGclplVbAELw4M4/hj26RYO+v4hIfTk+U/nxECRSl2w2G+3bt691sLYsAA0bNqzSrLM333wz3bp14/7776+yayslJYUPPvigwrply5aRnJzs73ZNSUlh+fLl3HPPPRXaDBkypB6OouYcUVG4zAgcRiGuIxm0UgASkSbCMAwSEhJo3bq1//pMkboSGhqKzVb7K3gsC0DR0dFccMEFFdZFRkbSokUL//oZM2Zw4MABXn/9dQCmTp3Kc889x/Tp07n11lvZsGEDr7zySoWhwO+66y6GDh3KU089xdixY1myZAkrVqxg3bp1DXdw1WCzGWQbThwUknf0IK06XnDmjUREGpGgoKBaX6chUl8svwvsdDIyMkhPT/c/79ixIx999BGrVq2iT58+PPbYY8yZM4drr73W32bIkCEsWLCAuXPn0qtXL+bNm8fChQsZNGiQFYdwWvlBMQAU5Wg0aBERkYakucCq0BBzgQFsfPJKBhR/RlrP39Pn2t/V2/uIiIg0B5oLrJEoCYsDwJN32OJKREREmhcFIAuVhftGgzYKFYBEREQakgKQlcrnAwsu1mjQIiIiDUkByELBmg9MRETEEgpAFrI7fYOFRZZlW1yJiIhI86IAZKGIOF8AcngUgERERBqSApCFHOXzgTkogLISi6sRERFpPhSALBTbsjWlpm+U1KLcU89WLyIiInVLAchCEfYQsvEN1JR7OMPiakRERJoPBSALGYZBrs0JQP4xBSAREZGGogBksfzgWACKNR+YiIhIg1EAslhxqG86jDKXApCIiEhDUQCyWElYCwDMAk2HISIi0lAUgCzmDW8JgK3wiMWViIiINB8KQBYzonzTYYRoPjAREZEGowBksRBHGwDCSjQfmIiISENRALJYWIwvAEWV5VhbiIiISDOiAGSxqPLpMGLMbDBNi6sRERFpHhSALOZo6ZsQNZQyzOJci6sRERFpHhSALBYXE0O+GQZA/lHNByYiItIQFIAsZg8O4hi+6TBcmg5DRESkQSgABQBXUAwAhcfUAyQiItIQFIACQGGIbzoMd64CkIiISENQAAoAx+cD8+RpOgwREZGGoAAUAMrCffOBUZBlbSEiIiLNhAJQADAjfNNhBBVpOgwREZGGoAAUAIKifQEo1K0AJCIi0hAUgAJAiNM3GGJEabbFlYiIiDQPCkABIKJ8PrBojwKQiIhIQ1AACgBRLXzzgTnNPPCUWVyNiIhI06cAFABi4trgMQ0AyvJ0J5iIiEh9szQAPf/88/Tq1QuHw4HD4SAlJYWPP/74lO0nTZqEYRiVlh49evjbzJs3r8o2xcXFDXFINRIbFUY20QC4jmo6DBERkfpmaQBq164dTz75JJs2bWLTpk1cfvnljB07lm3btlXZ/tlnnyUjI8O/7Nu3j7i4OH75y19WaOdwOCq0y8jIICwsrCEOqUaCg2xkGzEA5Gs+MBERkXoXbOWbjxkzpsLzxx9/nOeff57PP/+8Qq/OcU6nE6fT6X/+3nvvkZ2dzc0331yhnWEYxMfH10/R9SQvKBY86RTnHLK6FBERkSYvYK4B8ng8LFiwgIKCAlJSUqq1zSuvvMLw4cNJSkqqsD4/P5+kpCTatWvH1VdfzebNm0+7H7fbjcvlqrA0tKLQWABKXApAIiIi9c3yALR161aioqKw2+1MnTqVxYsX07179zNul5GRwccff8yUKVMqrO/WrRvz5s3j/fffZ/78+YSFhXHhhReyY8eOU+5r5syZ/t4lp9NJYmJirY/rbLntvukwvJoPTEREpN4ZpmmaVhZQUlJCeno6OTk5pKam8q9//YvVq1efMQTNnDmTWbNmcfDgQUJDQ0/Zzuv10q9fP4YOHcqcOXOqbON2u3G73f7nLpeLxMREcnNzcTgcNTuws7Tixd8xPOMltrYaQ89pbzbIe4qIiDQlLpcLp9NZrd9vS68BAggNDaVz584AJCcns3HjRp599llefPHFU25jmiavvvoqEydOPG34AbDZbAwYMOC0PUB2ux273V6zA6grka0BCCk+Ym0dIiIizYDlp8BOZppmhd6YqqxevZqdO3cyefLkau0vLS2NhISEuiqxXgQ7fPOB2UuOWVyJiIhI02dpD9CDDz7I6NGjSUxMJC8vjwULFrBq1SqWLl0KwIwZMzhw4ACvv/56he1eeeUVBg0axAUXXFBpn48++iiDBw+mS5cuuFwu5syZQ1paGv/4xz8a5Jhqyu70TYcRqfnARERE6p2lAejQoUNMnDiRjIwMnE4nvXr1YunSpYwYMQLwXeicnp5eYZvc3FxSU1N59tlnq9xnTk4Ot912G5mZmTidTvr27cuaNWsYOHBgvR9PbYTHtgXA4c0B0wTDsLYgERGRJszyi6AD0dlcRFVXdh3MotNLXXxPZhwAe1SDvK+IiEhTcTa/3wF3DVBz1SImlkLTdyG2OzfT4mpERESaNgWgAOEID+YovrTqOqLpMEREROqTAlCAMAyDXFsMAIXZ6gESERGpTwpAAaQgOAaA4hwFIBERkfqkABRAikJ902GUurIsrkRERKRpUwAKIKVhvgBkFmg+MBERkfqkABRAvBEtAQgqVAASERGpTwpAAcQW5ZsOI6T4qMWViIiING0KQAEkxOGbDiNc02GIiIjUKwWgABJWPh9YVJkCkIiISH1SAAogkXHxADhMF3g9FlcjIiLSdCkABZCYlgl4TQMbJmahrgMSERGpLwpAAaSFI4JsfJOgFuUcsrgaERGRpksBKIBEhAaTjROAvCMHLa5GRESk6VIACjCuoBgACrM1IaqIiEh9UQAKMIXBsQC4czUdhoiISH1RAAowxfY4ADx5CkAiIiL1RQEowJSF+6bDQPOBiYiI1BsFoABjRvqmwwgu0m3wIiIi9UUBKMAER7cGwO4+YnElIiIiTZcCUIAJcfgCUITmAxMREak3CkABJjzWNx1GtCfH2kJERESaMAWgABMdlwBAOMVQUmBxNSIiIk2TAlCAiYttgdsMAcCbrzvBRERE6oMCUICJi7JzuHw6jILsTIurERERaZoUgAJMaLCNHKN8PrCjmg9MRESkPigABaD88vnAitUDJCIiUi8UgAJQYYhvOoxSl6bDEBERqQ8KQAGoNMwXgHQRtIiISP1QAApAnvL5wIxCBSAREZH6oAAUgIwo32jQmg9MRESkflgagJ5//nl69eqFw+HA4XCQkpLCxx9/fMr2q1atwjCMSssPP/xQoV1qairdu3fHbrfTvXt3Fi9eXN+HUqeCon0TooaXHLO4EhERkabJ0gDUrl07nnzySTZt2sSmTZu4/PLLGTt2LNu2bTvtdtu3bycjI8O/dOnSxf/ahg0bmDBhAhMnTmTLli1MnDiR8ePH88UXX9T34dQZu9M3HUZkmeYDExERqQ+GaZqm1UWcKC4ujmeeeYbJkydXem3VqlVcdtllZGdnExMTU+X2EyZMwOVyVehJuuKKK4iNjWX+/PnVqsHlcuF0OsnNzcXhcNToOGpj49ZtDEgdggcbQX84CjadqRQRETmTs/n9DphfVo/Hw4IFCygoKCAlJeW0bfv27UtCQgLDhg3j008/rfDahg0bGDlyZIV1o0aNYv369afcn9vtxuVyVVis5Gzh6wEKwgtF6gUSERGpa5YHoK1btxIVFYXdbmfq1KksXryY7t27V9k2ISGBl156idTUVBYtWkTXrl0ZNmwYa9as8bfJzMykTZs2FbZr06YNmZmnHlRw5syZOJ1O/5KYmFg3B1dDLRxR5JiRAJTlHbK0FhERkaYo2OoCunbtSlpaGjk5OaSmpnLTTTexevXqKkNQ165d6dq1q/95SkoK+/bt4y9/+QtDhw71rzcMo8J2pmlWWneiGTNmMH36dP9zl8tlaQiKiQhlj+kgxigg71gGsfFVB0IRERGpGct7gEJDQ+ncuTPJycnMnDmT3r178+yzz1Z7+8GDB7Njxw7/8/j4+Eq9PVlZWZV6hU5kt9v9d6IdX6wUZDPIscUCUHBM02GIiIjUNcsD0MlM08Ttdle7/ebNm0lISPA/T0lJYfny5RXaLFu2jCFDhtRZjQ2hIDgGAHeOApCIiEhds/QU2IMPPsjo0aNJTEwkLy+PBQsWsGrVKpYuXQr4Tk0dOHCA119/HYDZs2fToUMHevToQUlJCW+++Sapqamkpqb693nXXXcxdOhQnnrqKcaOHcuSJUtYsWIF69ats+QYa6o4tAWUQZnmAxMREalzlgagQ4cOMXHiRDIyMnA6nfTq1YulS5cyYsQIADIyMkhPT/e3Lykp4d577+XAgQOEh4fTo0cPPvzwQ6688kp/myFDhrBgwQJ+//vf8/DDD3PuueeycOFCBg0a1ODHVxul4S2gEMwCTYchIiJS1wJuHKBAYPU4QAD/+ddjXL3/L+yIvYQud71vSQ0iIiKNSaMcB0gqMsqnwwhxaz4wERGRuqYAFKBCHL671sJLNBCiiIhIXVMAClBhMb7RoKM9mhBVRESkrikABajoON+t/RFmEZQWWVyNiIhI06IAFKBi4lpSYgb5nhQcsbYYERGRJkYBKEDFRdk5ihMAd67mAxMREalLCkAByhEWzDF8t/DlHcuwuBoREZGmRQEoQBmGgcsWA0BRtqbDEBERqUsKQAGsICQOgJJcBSAREZG6pAAUwErsLQDw5Gk6DBERkbqkABTAysJ9AQjNByYiIlKnFIACWURLAIKLNB2GiIhIXVIACmBB0b7pMEJLFIBERETqkgJQAAt1tgYgslTzgYmIiNQlBaAAFlE+HUa0Jwe8XmuLERERaUIUgAJYdJxvQtRgPFCcY20xIiIiTYgCUACLdUTjMiMAMHUnmIiISJ1RAApgLaJCOWL6psPQaNAiIiJ1RwEogEWEBpNt+CZEzT+mACQiIlJXFIACXH5wDADFOZoRXkREpK4oAAW4whDfaNClLgUgERGRuqIAFOBKy+cD8+ZnWVyJiIhI06EAFOA8Eb4AZCvUXWAiIiJ1RQEowBlRvtGgg4uOWVyJiIhI06EAFOCCo1sBEFaiACQiIlJXFIACXFiMbzToqDLNByYiIlJXFIACXESsLwBFmvlQVmJxNSIiIk2DAlCAc8a1otQM8j3RdBgiIiJ1QgEowLWMDucY0YBuhRcREakrCkABLjYilKOmbzqMAs0HJiIiUicUgAJcaLCNXJsvABVqPjAREZE6UaMAtG/fPvbv3+9//uWXX3L33Xfz0ksv1Vlh8rP84FgAinM1HYaIiEhdqFEA+vWvf82nn34KQGZmJiNGjODLL7/kwQcf5E9/+lO19/P888/Tq1cvHA4HDoeDlJQUPv7441O2X7RoESNGjKBVq1b+9p988kmFNvPmzcMwjEpLcXFxTQ41ILhD4wAoc+kaIBERkbpQowD07bffMnDgQADefvttLrjgAtavX89bb73FvHnzqr2fdu3a8eSTT7Jp0yY2bdrE5ZdfztixY9m2bVuV7desWcOIESP46KOP+Oqrr7jssssYM2YMmzdvrtDO4XCQkZFRYQkLC6vJoQaEkrCWvgcFCkAiIiJ1IbgmG5WWlmK32wFYsWIF//M//wNAt27dyMjIqPZ+xowZU+H5448/zvPPP8/nn39Ojx49KrWfPXt2hedPPPEES5Ys4YMPPqBv377+9YZhEB8fX+063G43brfb/9zlclV724ZgRraEY2ArPGJ1KSIiIk1CjXqAevTowQsvvMDatWtZvnw5V1xxBQAHDx6kRYsWNSrE4/GwYMECCgoKSElJqdY2Xq+XvLw84uLiKqzPz88nKSmJdu3acfXVV1fqITrZzJkzcTqd/iUxMbFGx1BfbOXzgYW6j1pciYiISNNQowD01FNP8eKLL3LppZfyq1/9it69ewPw/vvv+0+NVdfWrVuJiorCbrczdepUFi9eTPfu3au17axZsygoKGD8+PH+dd26dWPevHm8//77zJ8/n7CwMC688EJ27Nhxyv3MmDGD3Nxc/7Jv376zOob6Fur0BaDwUk2HISIiUhcM0zTNmmzo8XhwuVzExsb61+3Zs4eIiAhat25d7f2UlJSQnp5OTk4Oqamp/Otf/2L16tVnDEHz589nypQpLFmyhOHDh5+yndfrpV+/fgwdOpQ5c+ZUqyaXy4XT6SQ3NxeHw1HtY6kv//18I8OWDqeUEEL+eBgMw+qSREREAs7Z/H7XqAeoqKgIt9vtDz979+5l9uzZbN++/azCD0BoaCidO3cmOTmZmTNn0rt3b5599tnTbrNw4UImT57M22+/fdrwA2Cz2RgwYMBpe4ACXVRsAgAhlII7sK5PEhERaYxqFIDGjh3L66+/DkBOTg6DBg1i1qxZjBs3jueff75WBZmmWeGC5JPNnz+fSZMm8dZbb3HVVVdVa39paWkkJCTUqi4rxcU4yTfL72LL13xgIiIitVWjAPT1119z8cUXA/Duu+/Spk0b9u7dy+uvv17t00wADz74IGvXrmXPnj1s3bqVhx56iFWrVnHDDTcAvmtzbrzxRn/7+fPnc+ONNzJr1iwGDx5MZmYmmZmZ5Obm+ts8+uijfPLJJ+zatYu0tDQmT55MWloaU6dOrcmhBoQWUXaOlE+HUZanwRBFRERqq0YBqLCwkOho3wSdy5Yt45prrsFmszF48GD27t1b7f0cOnSIiRMn0rVrV4YNG8YXX3zB0qVLGTFiBAAZGRmkp6f727/44ouUlZUxbdo0EhIS/Mtdd93lb5OTk8Ntt93G+eefz8iRIzlw4ABr1qw564uzA0lMeAhH8Z3LLNB0GCIiIrVWo4uge/XqxZQpU/jFL37BBRdcwNKlS0lJSeGrr77iqquuIjOzcf9IB9pF0ACrHh3OpeZGMi56goTh06wuR0REJODU+0XQf/jDH7j33nvp0KEDAwcO9I/bs2zZsgoDEkrdKQz2jXVU4tIpMBERkdqq0UjQ1113HRdddBEZGRn+MYAAhg0bxi9+8Ys6K05+5g6Lg1Lw5OkiaBERkdqqUQACiI+PJz4+nv3792MYBuecc06jvs4m0HnCW0IeUKAAJCIiUls1OgXm9Xr505/+hNPpJCkpifbt2xMTE8Njjz2G1+ut6xoFMCNaARBSpPnAREREaqtGPUAPPfQQr7zyCk8++SQXXnghpmny2Wef8cgjj1BcXMzjjz9e13U2e8EO3wCT9pJjFlciIiLS+NUoAL322mv861//8s8CD9C7d2/OOeccfvOb3ygA1YNQRxsAIkoVgERERGqrRqfAjh07Rrdu3Sqt79atG8eO6Qe6PoTHxQMQ5c0DT6nF1YiIiDRuNQpAvXv35rnnnqu0/rnnnqNXr161Lkoqc8a2wmOWT4JaeNTaYkRERBq5Gp0Ce/rpp7nqqqtYsWIFKSkpGIbB+vXr2bdvHx999FFd1yhAi+gIjhFNK1y+O8Gi460uSUREpNGqUQ/QJZdcwo8//sgvfvELcnJyOHbsGNdccw3btm1j7ty5dV2jAC2iQjlaPh+YO6dxj7QtIiJitRqPA9S2bdtKFztv2bKF1157jVdffbXWhUlFUfZgjuEE9lGQnYHd6oJEREQasRr1AEnDMwyD/OBYAIqyNR2GiIhIbSgANSJFob4AVJanACQiIlIbCkCNSIm9BQDefE2HISIiUhtndQ3QNddcc9rXc3JyalOLnIE3oiXkgq1Q02GIiIjUxlkFIKfTecbXb7zxxloVJKdmizo+H5jGARIREamNswpAusXdWkHRvukwwjQfmIiISK3oGqBGxB7jC0BRnmwwTYurERERabwUgBqRqLgEAELNEijJt7gaERGRxksBqBGJjYmh0CwfArFAd4KJiIjUlAJQI9Iiys5R0wGAqVvhRUREakwBqBFpERnKUXwBqEjzgYmIiNSYAlAjEhYSRI7hG4qg8JgCkIiISE0pADUy+cFxALhzFYBERERqSgGokSm2+wJQWZ6uARIREakpBaBGpiyspe+B7gITERGpMQWgRsaM9AUgzQcmIiJScwpAjczx+cDsbs0HJiIiUlMKQI1MiMM3HUZ4qeYDExERqSkFoEYmIjYegEiPCzxlFlcjIiLSOCkANTJRsW3wmgY2TChSL5CIiEhNWBqAnn/+eXr16oXD4cDhcJCSksLHH3982m1Wr15N//79CQsLo1OnTrzwwguV2qSmptK9e3fsdjvdu3dn8eLF9XUIDa6FI4JsonxPdCeYiIhIjVgagNq1a8eTTz7Jpk2b2LRpE5dffjljx45l27ZtVbbfvXs3V155JRdffDGbN2/mwQcf5Le//S2pqan+Nhs2bGDChAlMnDiRLVu2MHHiRMaPH88XX3zRUIdVr1pEhvrnA/NoLCAREZEaMUzTNK0u4kRxcXE888wzTJ48udJr999/P++//z7ff/+9f93UqVPZsmULGzZsAGDChAm4XK4KPUlXXHEFsbGxzJ8/v1o1uFwunE4nubm5OByOWh5R3Sr1eNn0yIWkBH1H3lUvED3gV1aXJCIiEhDO5vc7YK4B8ng8LFiwgIKCAlJSUqpss2HDBkaOHFlh3ahRo9i0aROlpaWnbbN+/fpTvrfb7cblclVYAlVIkA1XkG8+sOJsTYchIiJSE5YHoK1btxIVFYXdbmfq1KksXryY7t27V9k2MzOTNm3aVFjXpk0bysrKOHLkyGnbZGaeOizMnDkTp9PpXxITE2t5VPWrMKR8PjDXIYsrERERaZwsD0Bdu3YlLS2Nzz//nNtvv52bbrqJ77777pTtDcOo8Pz4GbwT11fV5uR1J5oxYwa5ubn+Zd++fTU5lAbjtrcAwJuXZXElIiIijVOw1QWEhobSuXNnAJKTk9m4cSPPPvssL774YqW28fHxlXpysrKyCA4OpkWLFqdtc3Kv0Insdjt2u722h9JgPOEtIR8MTYchIiJSI5b3AJ3MNE3cbneVr6WkpLB8+fIK65YtW0ZycjIhISGnbTNkyJD6KdgK5fOBBRdpOgwREZGasLQH6MEHH2T06NEkJiaSl5fHggULWLVqFUuXLgV8p6YOHDjA66+/Dvju+HruueeYPn06t956Kxs2bOCVV16pcHfXXXfdxdChQ3nqqacYO3YsS5YsYcWKFaxbt86SY6wPQdGtAQh3Z4FpwmlO74mIiEhllvYAHTp0iIkTJ9K1a1eGDRvGF198wdKlSxkxYgQAGRkZpKen+9t37NiRjz76iFWrVtGnTx8ee+wx5syZw7XXXutvM2TIEBYsWMDcuXPp1asX8+bNY+HChQwaNKjBj6/etOqK2wwmpjQLMrZYXY2IiEijE3DjAAWCQB4HCOCjrRl4357E1UGfw8Db4MpnrC5JRETEco1yHCCpvhaRobzjucT3ZOs7UFb1NVMiIiJSNQWgRqhDy0jWmT3JMOOgKBu2n37+NBEREalIAagRauMIY0jn1izyXORbkfZvawsSERFpZBSAGqkJAxJ5t/w0mLlzBbgyLK5IRESk8VAAaqRGdG9DTnh7NnrPwzC98M0Cq0sSERFpNBSAGil7cBDX9Gv388XQm//tGxNIREREzkgBqBGbMCCRDz2DKTTtcHQH7N9odUkiIiKNggJQI3Zem2jOa5/Ax94BvhWb37S2IBERkUZCAaiRu/7Ei6G3LYaSQosrEhERCXwKQI3c1b3asjX4AvZ5W2G4XfDDf6wuSUREJOApADVykfZgru7djnc9Q30rdBpMRETkjBSAmoAJAxJJ9foCkLl7DeSkn2ELERGR5k0BqAnokxhDZOtOfObpgYEJafOtLklERCSgKQA1AYZhMGFA4s9jAqX9G7xea4sSEREJYApATcQv+p7DSmMQeWY45OyFvZ9ZXZKIiEjAUgBqImIjQ7nkgiT+4xnsW6EJUkVERE5JAagJuf6E02Dmd0vAnWdxRSIiIoFJAagJSenUgsMxvfjJm4BRWgjb3rO6JBERkYCkANSE2GwG45Pb+0eG1mkwERGRqikANTHXJbdjsfdiPKYB6Rvg6E9WlyQiIhJwFICamARnON27dmWNt5dvhXqBREREKlEAaoJOHBPITJsPXo/FFYmIiAQWBaAm6PJurdkcPoRsMwoj7yDs+tTqkkRERAKKAlATFBJkY2xyR5Z4hvhWbNZpMBERkRMpADVREwYk+meIN3/4EIqyLa5IREQkcCgANVEdW0YSmdSf772JGB43fJtqdUkiIiIBQwGoCbt+0M9jApk6DSYiIuKnANSEjb4ggeUhl1JqBmEc/Bqyvre6JBERkYCgANSEhYUEcWnf81np7etbsflNawsSEREJEApATdz45J/HBPJuWQieUosrEhERsZ4CUBN3wTlODsdfzGHTga3wMOxYbnVJIiIilrM0AM2cOZMBAwYQHR1N69atGTduHNu3bz/tNpMmTcIwjEpLjx49/G3mzZtXZZvi4uL6PqSAdN3ATiz2XAyAmabTYCIiIpYGoNWrVzNt2jQ+//xzli9fTllZGSNHjqSgoOCU2zz77LNkZGT4l3379hEXF8cvf/nLCu0cDkeFdhkZGYSFhdX3IQWk/+ndlg+MS31Ptn8CBUesLEdERMRywVa++dKlSys8nzt3Lq1bt+arr75i6NChVW7jdDpxOp3+5++99x7Z2dncfPPNFdoZhkF8fHzdF90IOcND6NJzIFu+7URv2y745m1I+Y3VZYmIiFgmoK4Bys3NBSAuLq7a27zyyisMHz6cpKSkCuvz8/NJSkqiXbt2XH311WzevPmU+3C73bhcrgpLU3P9gPb+i6E9m98E07S4IhEREesETAAyTZPp06dz0UUXccEFF1Rrm4yMDD7++GOmTJlSYX23bt2YN28e77//PvPnzycsLIwLL7yQHTt2VLmfmTNn+nuWnE4niYmJtT6eQDOgQyzfxAzHbYYQlLUNMrZYXZKIiIhlDNMMjK6AadOm8eGHH7Ju3TratWtXrW1mzpzJrFmzOHjwIKGhoads5/V66devH0OHDmXOnDmVXne73bjdbv9zl8tFYmIiubm5OByOsz+YAPXi6p9ou+I3jAn6HAbeBlc+Y3VJIiIidcblcuF0Oqv1+x0QPUB33nkn77//Pp9++mm1w49pmrz66qtMnDjxtOEHwGazMWDAgFP2ANntdhwOR4WlKbqmXzsWeS8FwLPlbShzn34DERGRJsrSAGSaJnfccQeLFi1i5cqVdOzYsdrbrl69mp07dzJ58uRqvU9aWhoJCQm1KbfRaxVtJ6zrMA6acQS5c2D7R1aXJCIiYglLA9C0adN48803eeutt4iOjiYzM5PMzEyKior8bWbMmMGNN95YadtXXnmFQYMGVXm90KOPPsonn3zCrl27SEtLY/LkyaSlpTF16tR6PZ7GYPygDiwqHxPI87XGBBIRkebJ0gD0/PPPk5uby6WXXkpCQoJ/Wbhwob9NRkYG6enpFbbLzc0lNTX1lL0/OTk53HbbbZx//vmMHDmSAwcOsGbNGgYOHFivx9MYDO3SitXhIwCw/bQSXBkWVyQiItLwAuYi6EByNhdRNUZ/Xbadi9f9LwNsP8LwR+Cie6wuSUREpNYa3UXQ0rB+mZzIu+VjApV+9YbGBBIRkWZHAagZSoyL4FiHqyg07YRk/wT7N1pdkoiISINSAGqmxg7qysde3zVR3s26GFpERJoXBaBmakT3NnwcPMz3ZPO/4fsPrC1IRESkASkANVP24CCS+o1kkecibGYZ5juTFIJERKTZUABqxq4f2J7flU3lPc8QDG95CPrufavLEhERqXcKQM1YlzbRPDauN/9X9hsWey70haB3b1YIEhGRJk8BqJn79aD2/P3XyczwTjspBC2xujQREZF6E2x1AWK9K3sm4AgbxNQ3DEyPwTWsw3z3FozrXoXuY60uT0REpM6pB0gAuKhLS/596xCeCLmTRZ6Lyq8JUk+QiIg0TQpA4tc7MYYFUy/ir+F3+UKQ6VEIEhGRJkkBSCro3DqKt39zMS/E/B+pJ4agbe9ZXZqIiEid0TVAUknbmHAWTr2IW+YGwyG4Nqj8miCAHuMsrk5ERKT21AMkVYqNDOXNW4ewJOkhUj0XY5gevO/eop4gERFpEhSA5JQi7cH8a9JgVnX7I6mei7H5Q9Biq0sTERGpFQUgOa3QYBuzf5VMWr8/865nqC8EvTMZ81uFIBERabwUgOSMgmwGfxrXm/0XP+0LQXjwvjsZ79ZFVpcmIiJSIwpAUi2GYXD3yPMpvGI275QNJQgPZupkyr5Jtbo0ERGRs6YAJGflxgvPJfTaf/Cu5xKC8GIsmoI77R2ryxIRETkrCkBy1sb2bU/LX7/IIq8vBAW/dxsFXy20uiwREZFqUwCSGrn0/ASSbn6V97iMILyEfTCVnA2vg2laXZqIiMgZKQBJjfXv2JLuU1/jA9vlBOEl5pM7yXxuFJ6D31hdmoiIyGkpAEmtnBfvpO8db/Bv+3jcZjDxR7/AeGkoh968FfIOWV2eiIhIlRSApNbaxUXxy9+9yHsXLWEpKdgwabPzbYr/2ptjS5+A0iKrSxQREanAME1dtHEyl8uF0+kkNzcXh8NhdTmNSnZBCanvpZK8/Rn62H4CIDe0DSGjHiWi7wSwKXOLiEj9OJvfb/0aSZ2KjQxlyg2/IvI3n/J8ixkcMFvgLDlExAdTOfLsxZTtWW91iSIiIuoBqop6gOrO6m3p7Hz/KSYUv0uUUQxAVvsraf2LmRDbwdriRESkSTmb328FoCooANWtUo+XRWu+JmT1E4wzV2IzTEoJIb/vrcSOegDCnFaXKCIiTYACUC0pANWPnMISFnzwET23PcOFtm8BKAiOwbjsISIG3wJBwRZXKCIijZkCUC0pANWvn7Ly+M+787g68x+ca8sAIDuyE9H/8xTBXUdaXJ2IiDRWCkC1pADUMNb+cJAt7/2NG4reItbIB+Bo/MW0+MVT0KaHxdWJiEhj02juAps5cyYDBgwgOjqa1q1bM27cOLZv337abVatWoVhGJWWH374oUK71NRUunfvjt1up3v37ixevLg+D0Vq4OJubZn6u6dYPnwpbxhjKDGDaJG5Fp4fQtYLYynb/Zmm1hARkXphaQBavXo106ZN4/PPP2f58uWUlZUxcuRICgoKzrjt9u3bycjI8C9dunTxv7ZhwwYmTJjAxIkT2bJlCxMnTmT8+PF88cUX9Xk4UgPBQTbGX9yT/7lvLv/qNZ+PvIPwmgatM1cR/NqVHPzrxRzZlAper9WliohIExJQp8AOHz5M69atWb16NUOHDq2yzapVq7jsssvIzs4mJiamyjYTJkzA5XLx8ccf+9ddccUVxMbGMn/+/DPWoVNg1tmfXcjS1euI2/IiV3lXYTfKAMgITiS771S6jphCUGiYxVWKiEggajSnwE6Wm5sLQFxc3Bnb9u3bl4SEBIYNG8ann35a4bUNGzYwcmTFi2lHjRrF+vVVD8LndrtxuVwVFrFGu9gIpowbydUPvcPaqz5lSdQEXGYECWX76L7xIY490Y11835P1uEsq0sVEZFGLGACkGmaTJ8+nYsuuogLLrjglO0SEhJ46aWXSE1NZdGiRXTt2pVhw4axZs0af5vMzEzatGlTYbs2bdqQmZlZ5T5nzpyJ0+n0L4mJiXVzUFJjocE2hg/sxdh7X+LobZtZkXgnh4ijFdlctOfvhD/Xi09m/z8+T/sWrzdgOjFFRKSRCJhTYNOmTePDDz9k3bp1tGvX7qy2HTNmDIZh8P777wMQGhrKa6+9xq9+9St/m3//+99MnjyZ4uLiStu73W7cbrf/ucvlIjExUafAAkxxcRHbPnmF1ltfJLEsHYASM4jlwZeS1/92RgwdSosou8VVioiIVRrdKbA777yT999/n08//fSsww/A4MGD2bFjh/95fHx8pd6erKysSr1Cx9ntdhwOR4VFAk9YWDj9x95B4oNb2D96HnsjexFqeLjK81+u//I6vn76Sv766ht8vusoAZLrRUQkQFkagEzT5I477mDRokWsXLmSjh071mg/mzdvJiEhwf88JSWF5cuXV2izbNkyhgwZUqt6JUDYbLQb9AuSfreW4hs/Zn+bywEYYdvE9PQ7sM0bzcNPP8Nrn+2iuNRjcbEiIhKILJ17YNq0abz11lssWbKE6Ohof6+N0+kkPDwcgBkzZnDgwAFef/11AGbPnk2HDh3o0aMHJSUlvPnmm6SmppKamurf71133cXQoUN56qmnGDt2LEuWLGHFihWsW7eu4Q9S6lVYpyG0u30xHP6RYytm4fgxlYG27Qwsepydn8zl7/8dTeuLJzH+wh6EhwZZXa6IiAQIS68BMgyjyvVz585l0qRJAEyaNIk9e/awatUqAJ5++mleeuklDhw4QHh4OD169GDGjBlceeWVFfbx7rvv8vvf/55du3Zx7rnn8vjjj3PNNddUqy7dBt+IuTJwr/8nxqZXCS3zjS5dYNr52HYJDLiVK4dfTkSo5hwTEWmKNBVGLSkANQHFLsrS5lOw7gWc+bv8q7+iO8d63MSFV99ERHkvo4iINA0KQLWkANSEmCZlu9aQsfzvJGT+l2B8I0pnEcfeDr+kx5jfEtHi7C+8FxGRwKMAVEsKQE1TafY+dnz0d9rsXEgLM8e3jiD2tLqcc0bcSUSXoXCK07IiIhL4FIBqSQGoaSsrKearpa8RkTaXnt7v/euPRHQm8uL/R3i/X4M9ysIKRUSkJhSAakkBqHko83hZu3YlBZ+9yOUlq4kwfINhuoMioc+vsQ++DVqdZ3GVIiJSXQpAtaQA1Lx4vCZLN/3A3v++zOii/9DRdsj/WlnSUIL7TICW50HcuRARp9NkIiIBSgGolhSAmieP1+Sjbw6wftk7XJ73PsNsm7EZFf/zKAt1QFwnglp2xojrBC3O9QWjuE4KRyIiFlMAqiUFoObN6zX5+NtMFi5fx6Ds9+ln7KSDLZME49hptysJcVDq7ICtxbmEte6M0bKzwpGISANSAKolBSABXxD67w9ZfLU3m/3ZhRzOzsaWvQdH4T46GJkkGZl0NA6RZMuk7RnCUXFwNIWOLgR3upDorpdgJA6CMH23RETqkgJQLSkAyekUl3rIyC3mQHYR+7MLOZBTRNbRbMqO7CbUtRtHUTpJZNLBOESHU4QjLzaORnfFkziEuO6XEtrpIl8vkYiI1JgCUC0pAEltlHq8ZOYWcyCniP3ZRWQeyaY4ayfBmWmc49rMQON7kmxZlbY7FNaJwoRBRJ93CS16XIrhSKhi7yIicioKQLWkACT1pajEw9YDufz44w8U/7SOmCMb6VW2jfNsByq1zQw+h2MtkwnpdDHt+gwjvFVHXUckInIaCkC1pAAkDcU0TQ7kFLFtxy5yflhDeMbnnFu4hfPZW+kOtCyjFQdj+mIm9CW6TQdate2Is00SRLaGIE3wKiKiAFRLCkBipeJSDz/s3kfmttUY6etpm/M13bw/EWJ4qmzvwUZ+cBzF4W0wo9tijzuHqFZJhMScA44EcJwD0QkQGtHARyIi0rAUgGpJAUgCTeaRI+zdsgb3zrWE5fxIRHEWsd6jtCGbYMNbrX2UhTowo9sSHHOO7/qiuE6QOBjO6Q8hYfV8BCIi9U8BqJYUgKQxKCwpY09WHgcO7OVoxl4KjqRTln0AW34Gcd4jxJNNvHGMeOMYkeXTfFTFDArFOCcZklIgaQgkDgJ7dAMeiYhI3VAAqiUFIGnMTNPkcL6b3YcL2HWkgF1ZeWQezqIgax9e10FacZR4jtHVto9Bth9oZeRW3N6wYcT38oWhpCHQPgUiW1p0NCIi1acAVEsKQNJUlXq87DtWyK7DBXydns2aH7PIz/iRgbYfGGjbzkDje9rbDlfesGXXnwNR0hBwtmv44kVEzkABqJYUgKQ5OZLv5rOdR1j942HW7jhCUN5BBtq2M8D2AwNtP9DVtr/yRs725WEoBVr3ANMDnpLypfSkv8cfn2p9+WN7NHS72ncKzmZr+A9CRBo9BaBaUgCS5so0Tb7PyGPtjsOs2XGYjbuzifDkMsC2nQG27Qy0/UBP226CqN6F1zXiaAc9r4ULroP4nhr7SESqTQGolhSARHwKS8r4Ytcx1uw4zJofD/PT4QIiKKavbQcDbT8wJHg7HYOO4LWF4LWFYNpCMG2hmEEhEBQKQSEYwaHYgkJ9f4PtBIWEEhTi+xscYic41E5QsB3j2C744T/gdv1cQMuu0PM6uOBaaHGudR+EiDQKCkC1pAAkUrX92YWs3XGENT8eZt3OI+QVl9XJfoNsBs7wELrEBTM6bCsXFa2i47G1BHlLfm7Utp8vDPW4xje+kYjISRSAakkBSOTMyjxetuzPZWdWHoUlHopKPRSVeCgsX4pLPRSWlPleq7DOt76o1EOp59T//ERTyEjbJv4neD0X2r4luPy0m4mBq80g6PlLHP2uxYiIbahDFpEApwBUSwpAIg2j1OP1B6cj+W52HS5g95ECdh3O993Cf7iAfHcZLcjlyqAvGBu0nmTbjz9vbwbxVWgyP7QcSX6HESS2aUm72HBshoFhGBj4LiEyMCpcSnTiugqPT3g90h5MgjO8QT8PEakdBaBaUgASCQymaXI4z81PhwvYdSSfXYcLcGX8RJesT7jYvZrzben+tgWmneXe/qzy9MEE7EYpdk5YjJKTnpdip+Sk56WElj+3YVIcHEVwRCxRMS1p0SqekMhYCIuB8FgIj6n82B6ti7ZFLKQAVEsKQCKBr6TMS+bOrynd8g6tdn+Ao/iA1SWBEVQ5GNkdEBwGQSEQbPddHH78b7Ufh0JoFMS0hxD1SomcigJQLSkAiTQypgkHvoKt70DGN77AEBzmCxGn/Vv+OMhe6bXCUg/b9+znp/QDHMw8iKcgG6dRgNPIx0kBLYKKSAgtwmkUEFbmwvCUnLnOWjN8g1C2OBdadD5hOdc3NlNQcAPUIBK4FIBqSQFIRE6292gBa8rvgNvw01Hy3SfeAWfSu00owzraueicYHrEebGXuqAox3dbf5n75wEfjz8uc5cPBOk+6fUS37oTH5eV+PZz4hABJ7OFQFzHnwPRiQEpqo1OzUmzoABUSwpAInI6pR4vX+/NZs0O3+jZWw/kcuK/pGEhNgZ3asHQLq0Y1CmOmIhQIkKCCA8Nwh5sw6hJGDFNKDwKR3eetPzkWzynnvCW0CiI63RSr1F5UAqPOftaRAKUAlAtKQCJyNk4mu9m3c4jrPnxCGt3HCYr79RhJMhm+MNQRGgQ4aHBRBx/HOL7G2EPJiKk4uvhoUGEhQQRFmzz/Q3xtQ8LsREWbBBenEF43l7sObsIyv4J4+hPvoCUsxfM04zcHdHyhF6jE3qO4jrpeiNpdBSAakkBSERqyjRNth/KY82Ph1nz4xG2HcyloMRDSVk9Th9yEptBeTgKIirYS4egw3QyMulgHKS9eZBzvAeJL92Po+zo6XfkTPw5FMWd+3NQiknS9UYSkBSAakkBSETqWpnHS2GFwSLLKjw+PoCkf12p7/UCt4ei0jL/QJLFpV6KSz24y7zlz32DUBaXnn3AiqSIDsYhOhoZvsWWSafyx06j8JTbeYwgCsLaUmqPwRMajRnq8N3tFubAFu4kKNxJcISTkIgYQiNjCIpwlr9e/jc4tOodm6bvOqfiXN/1U8U5P/8907rSImjVFc5JhnP6Q7tkXy+Wrn1qVhpNAJo5cyaLFi3ihx9+IDw8nCFDhvDUU0/RtWvXU26zaNEinn/+edLS0nC73fTo0YNHHnmEUaNG+dvMmzePm2++udK2RUVFhIWFnbEuBSARaWxM08Rd5sVd6qW4rGIwOh6Uiku9FJaUkVtUSnZhKbmFJWQXlpJdWFK+roScwlLyikuJJY+ORiadbOXhyMiko5FJByOTcKN2d7y5CaXQFkmxLYIyWzgOWzER3nyCS10Ypztdd7bCYnxh6HggOqc/RLasu/1LwDmb329L+zBXr17NtGnTGDBgAGVlZTz00EOMHDmS7777jsjIyCq3WbNmDSNGjOCJJ54gJiaGuXPnMmbMGL744gv69u3rb+dwONi+fXuFbasTfkREGiPDMPzXBjkJqdW+yjxef0jKKQ9F2YUlfFZUyn8KijFzDxKSt4+gEhdBJXkEl+VjL80j1JNPmKeAMG8+EWYh0ZQvRhHRFBJlFAP4BqD0loA3u+r3N0LwhDoJiowlOOKEgSbDnCeMs3TSuqBQyNzqGw5h/ybI2OLrGfrpv77luJikEwJRMiT00rVOzVRAnQI7fPgwrVu3ZvXq1QwdOrTa2/Xo0YMJEybwhz/8AfD1AN19993k5OTUqA71AImI1M7xHqnj878VlXooKi6hpDCXkoIcPIW5eIpyKMx38d1Rky8zPezKDyWXSNyEcHxikg4tIkjuEEdyUizJHWI5t1VU9e6iKyuBrG2+MHTgaziwCY78WLmdLRja9CjvKUqGhN6+8aA8JeAtBU9Z+d/Sn//6Hx9/raR8XdnPr3m9EBEH0QkQHe/7G9mqcV87VVYC7jzfIJ82m9XVVKnR9ACdLDc3F4C4uLhqb+P1esnLy6u0TX5+PklJSXg8Hvr06cNjjz1WoYfoRG63G7f757s2XK7TjLUhIiJndGKPVEzEia9U/vf9CnyBaX92EZv2HmPTnmy+2pvN9kN57DlayJ6jhbz71X4AYiNC6J8US/+kOAZ0iOWCc5yEhQRVLiA4FNr29S3HFeeWh6Gvfu4pKsjy9RZlbIFNr9bpZ1CJYYPI1j8Hokp/2/j+RrSsv4Bhmr4QU5TtW4pzyh/nnGJdzs/rSvJ9+7A7fJ9ru2RoN8AXHKNa1U+99ShgeoBM02Ts2LFkZ2ezdu3aam/3zDPP8OSTT/L999/TunVrAD7//HN27txJz549cblcPPvss3z00Uds2bKFLl26VNrHI488wqOPPlppvXqARESsk1tUytfp2Xy1J5uNe46xZX9OpYu9Q4Ns9GznpH9SLJ1aRtIuNoJ2seG0jQknNPgMIcI0IXd/eSAq7yk6tA0wfQNLBoWU/w0u/xt6wuMQyowgCssM8ksNXCWQ64ZsN+SW+HYdZ+TR2sgmwZZDS3IIoprXN9mCfYNXRsf7/tqCfD1Kpsc3pIHX43vs9fjeyP/Ye9LjE9uWlQefHN/zuhaTVDEQJfTyjarewBrNRdAnmjZtGh9++CHr1q2jXbt21dpm/vz5TJkyhSVLljB8+PBTtvN6vfTr14+hQ4cyZ86cSq9X1QOUmJioACQiEkBKyrx8l+Fi0x5fL9Gmvdkcya96zCXDgHhHGO1iw/2h6MTHCc5qBKRyOYUl7MzKZ0dWPjsO5bMjK4+dWflk5BafcptoezDFZR5KPb6fWBteWuCitZFNmxOW1mTTxsjxPTZyaEkuNqMBfpaDw3ynsk6cty489oR1J6wPO+H10Eg4vN0XGPd/Bfs3wpHtlfdvC/GFoHPKQ1G7/hDbsd7vymt0AejOO+/kvffeY82aNXTs2LFa2yxcuJCbb76Zd955h6uuuuqM7W+99Vb279/Pxx9/fMa2ugZIRCTwmabJ3qOFbNqbzTf7c9h3rJB92UXszy4847AApwpIraLtHMguOiHs5J8yZAG0jrbTuXUUXVpH0blNNF3KH7eIsmOaJrlFpRzJd3Mkv8T3N++Ex/6/vqW41EsQHlqS6w9IrYxcbHjxYsODDS8GHtOGF5t/nQcbpmEQGhyCPSQYe2goYfZgwkJDCQsNIcweSrg9lGhHLLEt29CqVRvatorDERZcs1HJT3b81OL+TeXBaKNv1PKTRbQoD0TlS9t+dT4SeaMJQKZpcuedd7J48WJWrVpV5empqsyfP59bbrmF+fPnM27cuGq9z8CBA+nZsyevvnrmc7wKQCIijZdpmhwtKGF/eRiq+Ld6Aelk58SE/xx0WkfRpU0UnVtF44yo3R13J9ZcUOLhSJ6bowVuDuf5wtHR/BJcxaW4ikrJKy4jz13+t7iMvOJSXEVllHhqNnRAlD2Yc2LCaRsTxjmx4ZwTE0HbmDD/KcTW0WEE2Wo4bUv2nvLrrDb6glHmN76LxU8U0x7u3lqj2k+l0VwEPW3aNN566y2WLFlCdHQ0mZmZADidTsLDfbclzpgxgwMHDvD6668DvvBz44038uyzzzJ48GD/NuHh4TidTgAeffRRBg8eTJcuXXC5XMyZM4e0tDT+8Y9/WHCUIiLSkAzDoGWUnZZRdvokxlR6/XhA2nesYijan11EVp6bc2LC6Nw62h92zm0dRZS9fn8uDcMgyh5MlD2YDi2rHgbmVIpLPf5AdGI4yisu84Wn4jJcRaVk5hZzIKeIAzlFHCsoId9dxvZDeWw/lFflfoNtBvHOMM6JCfct5cHIGR5SPn3Lz9O0RIYG+6d3CQmy+SbmjesIPa/z7azM7Rum4MReorb9avux1YqlPUCn6nqbO3cukyZNAmDSpEns2bOHVatWAXDppZeyevXqStvcdNNNzJs3D4B77rmHRYsWkZmZidPppG/fvjzyyCOkpKRUqy71AImISFNWVOLhQE4RB8sD0YFs3+P95esyc4sp89YsHoQG2fxh6HhQOv78xKDUMTaUm4eeV6fH1WhOgQUqBSAREWnOPF6TQ67inwNSeUjKyC0mr7jUP21LQfk0LkUlnrMOTP3ax7DoNxfWad2N5hSYiIiIBJ4gm0HbGN8pr+RqblNS5q00r13BKee8K6O1w9rZGRSAREREpNZCg22EBoeeNPBl4ArMsaxFRERE6pECkIiIiDQ7CkAiIiLS7CgAiYiISLOjACQiIiLNjgKQiIiINDsKQCIiItLsKACJiIhIs6MAJCIiIs2OApCIiIg0OwpAIiIi0uwoAImIiEizowAkIiIizY5mg6+CaZoAuFwuiysRERGR6jr+u338d/x0FICqkJeXB0BiYqLFlYiIiMjZysvLw+l0nraNYVYnJjUzXq+XgwcPEh0djWEYdbpvl8tFYmIi+/btw+Fw1Om+5Wf6nBuGPueGoc+54eizbhj19TmbpkleXh5t27bFZjv9VT7qAaqCzWajXbt29foeDodD/3E1AH3ODUOfc8PQ59xw9Fk3jPr4nM/U83OcLoIWERGRZkcBSERERJodBaAGZrfb+eMf/4jdbre6lCZNn3PD0OfcMPQ5Nxx91g0jED5nXQQtIiIizY56gERERKTZUQASERGRZkcBSERERJodBSARERFpdhSAREREpNlRAGpA//znP+nYsSNhYWH079+ftWvXWl1Sk/PII49gGEaFJT4+3uqyGr01a9YwZswY2rZti2EYvPfeexVeN02TRx55hLZt2xIeHs6ll17Ktm3brCm2ETvT5zxp0qRK3+/BgwdbU2wjNnPmTAYMGEB0dDStW7dm3LhxbN++vUIbfadrrzqfs5XfaQWgBrJw4ULuvvtuHnroITZv3szFF1/M6NGjSU9Pt7q0JqdHjx5kZGT4l61bt1pdUqNXUFBA7969ee6556p8/emnn+avf/0rzz33HBs3biQ+Pp4RI0b4JxaW6jnT5wxwxRVXVPh+f/TRRw1YYdOwevVqpk2bxueff87y5cspKytj5MiRFBQU+NvoO1171fmcwcLvtCkNYuDAgebUqVMrrOvWrZv5wAMPWFRR0/THP/7R7N27t9VlNGmAuXjxYv9zr9drxsfHm08++aR/XXFxsel0Os0XXnjBggqbhpM/Z9M0zZtuuskcO3asJfU0ZVlZWSZgrl692jRNfafry8mfs2la+51WD1ADKCkp4auvvmLkyJEV1o8cOZL169dbVFXTtWPHDtq2bUvHjh25/vrr2bVrl9UlNWm7d+8mMzOzwvfbbrdzySWX6PtdD1atWkXr1q0577zzuPXWW8nKyrK6pEYvNzcXgLi4OEDf6fpy8ud8nFXfaQWgBnDkyBE8Hg9t2rSpsL5NmzZkZmZaVFXTNGjQIF5//XU++eQTXn75ZTIzMxkyZAhHjx61urQm6/h3WN/v+jd69Gj+/e9/s3LlSmbNmsXGjRu5/PLLcbvdVpfWaJmmyfTp07nooou44IILAH2n60NVnzNY+50Orvd3ED/DMCo8N02z0jqpndGjR/sf9+zZk5SUFM4991xee+01pk+fbmFlTZ++3/VvwoQJ/scXXHABycnJJCUl8eGHH3LNNddYWFnjdccdd/DNN9+wbt26Sq/pO113TvU5W/mdVg9QA2jZsiVBQUGV/s8hKyur0v9hSN2KjIykZ8+e7Nixw+pSmqzjd9np+93wEhISSEpK0ve7hu68807ef/99Pv30U9q1a+dfr+903TrV51yVhvxOKwA1gNDQUPr378/y5csrrF++fDlDhgyxqKrmwe128/3335OQkGB1KU1Wx44diY+Pr/D9LikpYfXq1fp+17OjR4+yb98+fb/Pkmma3HHHHSxatIiVK1fSsWPHCq/rO103zvQ5V6Uhv9M6BdZApk+fzsSJE0lOTiYlJYWXXnqJ9PR0pk6danVpTcq9997LmDFjaN++PVlZWfz5z3/G5XJx0003WV1ao5afn8/OnTv9z3fv3k1aWhpxcXG0b9+eu+++myeeeIIuXbrQpUsXnnjiCSIiIvj1r39tYdWNz+k+57i4OB555BGuvfZaEhIS2LNnDw8++CAtW7bkF7/4hYVVNz7Tpk3jrbfeYsmSJURHR/t7epxOJ+Hh4RiGoe90HTjT55yfn2/td9qSe8+aqX/84x9mUlKSGRoaavbr16/CrYBSNyZMmGAmJCSYISEhZtu2bc1rrrnG3LZtm9VlNXqffvqpCVRabrrpJtM0fbcN//GPfzTj4+NNu91uDh061Ny6dau1RTdCp/ucCwsLzZEjR5qtWrUyQ0JCzPbt25s33XSTmZ6ebnXZjU5VnzFgzp07199G3+naO9PnbPV32igvUkRERKTZ0DVAIiIi0uwoAImIiEizowAkIiIizY4CkIiIiDQ7CkAiIiLS7CgAiYiISLOjACQiIiLNjgKQiIiINDsKQCIiItLsKACJiIhIs6MAJCIiIs3O/wcc4xXNj+wInwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Wher bef bridcowf,\n",
      "The lay ble\n",
      "bairet bube to tave O-' my dalllauss:\n",
      "Want he us he hertbar dilth anes with my thand a wizorm he offs, to fit her! Varl nowlit,\n",
      "Wheiree sen cin lat Heacliov the and the nown!\n",
      "Ferablesel lind teall thull cechir speave aiss hewty.\n",
      "HETBHUSIRCBETI:\n",
      "Alave whom\n",
      "Ill, demet aklecal-'so wher piichs withe dour warce hidend thoouse the the the danderthirf son; igis! muf thre ifled at tise Pried my of.\n",
      "\n",
      "HKINGLER:\n",
      "Widby and adsal ther grest hoin cour ay aney Iry thel fronf veay\n",
      "CPU times: user 1min 15s, sys: 445 ms, total: 1min 15s\n",
      "Wall time: 1min 15s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size:int=vocab_size, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 head_size:int=head_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = head_size\n",
    "\n",
    "        #This will be our lookup table for embeddings. We'll have an entry for each token (aka vocab size) and each embedding will... \n",
    "        #...be a vector of dimension embedding_dim.\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        self.token_position_embedding_table = nn.Embedding(num_embeddings=self.context_length, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        #4 heads of 8 dimensional self attention.\n",
    "        self.multi_self_attention_heads_layer = MultiHeadAttention(num_heads=4, head_size=self.embedding_dim//4)\n",
    "        self.feed_forward_network = FeedForwardNetwork(embedding_dim=self.embedding_dim) #NEW\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        B,T = idx.shape\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        token_position_embeddings = self.token_position_embedding_table(torch.arange(T, device=device)) #(T,C)\n",
    "        x = token_embeddings + token_position_embeddings\n",
    "        x = self.multi_self_attention_heads_layer(x) # (B,T,C)\n",
    "        x = self.feed_forward_network(x) # (B,T,C) NEW\n",
    "        logits = self.language_model_head_linear_layer(x) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Crop idx to the max size of our positional embeddings table \n",
    "            idx_crop = idx[:,-self.context_length:]\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx_crop)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = TransformerLanguageModel(vocab_size=vocab_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "tracked_losses = list()\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        tracked_losses.append(losses)\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "plot_losses(tracked_losses)\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8e8707f-5724-4c2c-a87e-8af84a28ece0",
   "metadata": {},
   "source": [
    "Our loss has improved again from `Validation Loss: tensor(2.2854)` now to `Validation Loss: tensor(2.2720)` now that we've added the feed forward network.<br>\n",
    "\n",
    "Next we need to create a Block module that incorporates everthing within the block on the transformer architecture diagram (grey box) which will then allow us to stack them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4008a385-bff3-476b-8b50-6512d2284860",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class TransformerBlock(nn.Module):\n",
    "    \"\"\"Transformer Block: Communication folled by computation.\"\"\"\n",
    "    def __init__(self, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 num_heads:int=4):\n",
    "        #embedding_dim: embedding dimension, num_heads: the number of heads that we want\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = embedding_dim // num_heads\n",
    "        self.num_heads = num_heads\n",
    "        \n",
    "        self.multi_self_attention_heads_layer = MultiHeadAttention(num_heads=self.num_heads, head_size=self.head_size, \n",
    "                                                                   embedding_dim=embedding_dim, context_length=context_length)\n",
    "        self.feed_forward_network = FeedForwardNetwork(embedding_dim=self.embedding_dim)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.feed_forward_network(self.multi_self_attention_heads_layer(x))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d7cfe8-b966-4415-bea1-ce3e97eef36e",
   "metadata": {},
   "source": [
    "Now we can add our new Transformer Block to our model and start stacking it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8056b81e-c4b9-48c9-8955-70b1fc4d6e59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.2116) Validation Loss: tensor(4.2078)\n",
      "Step: 200 Training Loss: tensor(3.2643) Validation Loss: tensor(3.2907)\n",
      "Step: 400 Training Loss: tensor(3.1541) Validation Loss: tensor(3.1676)\n",
      "Step: 600 Training Loss: tensor(3.0360) Validation Loss: tensor(3.0239)\n",
      "Step: 800 Training Loss: tensor(2.8569) Validation Loss: tensor(2.8526)\n",
      "Step: 1000 Training Loss: tensor(2.7738) Validation Loss: tensor(2.7607)\n",
      "Step: 1200 Training Loss: tensor(2.6645) Validation Loss: tensor(2.6827)\n",
      "Step: 1400 Training Loss: tensor(2.6202) Validation Loss: tensor(2.6159)\n",
      "Step: 1600 Training Loss: tensor(2.5581) Validation Loss: tensor(2.5613)\n",
      "Step: 1800 Training Loss: tensor(2.5231) Validation Loss: tensor(2.5388)\n",
      "Step: 2000 Training Loss: tensor(2.5020) Validation Loss: tensor(2.5028)\n",
      "Step: 2200 Training Loss: tensor(2.4899) Validation Loss: tensor(2.4974)\n",
      "Step: 2400 Training Loss: tensor(2.4812) Validation Loss: tensor(2.4668)\n",
      "Step: 2600 Training Loss: tensor(2.4656) Validation Loss: tensor(2.4641)\n",
      "Step: 2800 Training Loss: tensor(2.4574) Validation Loss: tensor(2.4511)\n",
      "Step: 3000 Training Loss: tensor(2.4306) Validation Loss: tensor(2.4413)\n",
      "Step: 3200 Training Loss: tensor(2.4016) Validation Loss: tensor(2.4273)\n",
      "Step: 3400 Training Loss: tensor(2.3946) Validation Loss: tensor(2.3999)\n",
      "Step: 3600 Training Loss: tensor(2.3738) Validation Loss: tensor(2.3823)\n",
      "Step: 3800 Training Loss: tensor(2.3844) Validation Loss: tensor(2.3700)\n",
      "Step: 4000 Training Loss: tensor(2.3571) Validation Loss: tensor(2.3570)\n",
      "Step: 4200 Training Loss: tensor(2.3426) Validation Loss: tensor(2.3729)\n",
      "Step: 4400 Training Loss: tensor(2.3394) Validation Loss: tensor(2.3696)\n",
      "Step: 4600 Training Loss: tensor(2.3300) Validation Loss: tensor(2.3343)\n",
      "Step: 4800 Training Loss: tensor(2.3263) Validation Loss: tensor(2.3400)\n",
      "Step: 4999 Training Loss: tensor(2.3301) Validation Loss: tensor(2.3403)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGxCAYAAACKvAkXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABiGUlEQVR4nO3deXwU9f3H8ddujs29uUkwIdwgl3IfCoLcKoWKgi0FUdQfFq1IrQoelXrgUSpSvIvEowZURGkFEYsBERRQgqiIKEcCJAQCuZPNsfP7Y2ElnCHXbJL38/GYR7Kz35n5zHZr3nznO9+xGIZhICIiItKIWM0uQERERKSuKQCJiIhIo6MAJCIiIo2OApCIiIg0OgpAIiIi0ugoAImIiEijowAkIiIijY4CkIiIiDQ6CkAiIiLS6CgAiYjHSExMxGKxsGXLFrNLEZEGTgFIREREGh0FIBEREWl0FIBEpF5Zv349gwcPJjg4mICAAPr168dHH31UoU1hYSH33HMPLVq0wM/Pj/DwcHr06EFSUpK7ze7du7nhhhto2rQpNpuNJk2aMHjwYFJSUirsa8mSJfTt25fAwECCgoIYPnw4W7durdCmsvsSEc/hbXYBIiKVtXbtWoYOHUqXLl1YuHAhNpuNF154gVGjRpGUlMT48eMBmDFjBm+++SaPPfYYXbt2paCggO+++46srCz3vq666irKy8t5+umnadasGUeOHGHDhg1kZ2e72zzxxBM8+OCD3HTTTTz44IOUlJTwzDPP0L9/fzZt2kSHDh0qvS8R8SwWwzAMs4sQEQHXIOibbrqJzZs306NHj9Pe79u3L7t37+aXX34hKCgIgPLyci699FKys7NJTU3FYrHQuXNnWrduzbJly854nKysLCIjI5k3bx533XXXGdukpaXRsmVLbr/9dubPn+9en5+fT5s2bRgwYABLliyp1L5ExPPoEpiI1AsFBQV89dVXXHfdde7wA+Dl5cXEiRPZv38/O3fuBKBXr16sXLmS+++/n+TkZIqKiirsKzw8nFatWvHMM8/wj3/8g61bt+J0Oiu0WbVqFWVlZUyaNImysjL34ufnxxVXXEFycnKl9yUinkcBSETqhWPHjmEYBrGxsae917RpUwD3Ja758+dz33338cEHHzBo0CDCw8MZM2YMu3btAsBisfC///2P4cOH8/TTT9OtWzeioqL405/+RF5eHgCHDh0CoGfPnvj4+FRYlixZwpEjRyq9LxHxPBoDJCL1QlhYGFarlfT09NPeO3jwIACRkZEABAYGMnv2bGbPns2hQ4fcvUGjRo3ixx9/BCAhIYGFCxcC8NNPP/HOO+/wyCOPUFJSwksvveTe13vvvUdCQsI5azvfvkTE82gMkIh4jPONAerXrx979uxh9+7d+Pv7A+B0Orn00ks5duyYewzQmdx9993MmzePgoICAgICztima9eu+Pj4sGnTJvbu3UubNm14/PHHuffeey/4XE7el4h4HvUAiYjHWbNmDXv37j1t/Zw5cxg6dCiDBg3innvuwdfXlxdeeIHvvvuOpKQkd/jp3bs311xzDV26dCEsLIwdO3bw5ptv0rdvXwICAvj222+54447uP7662nTpg2+vr6sWbOGb7/9lvvvvx+A5s2b87e//Y0HHniA3bt3M2LECMLCwjh06BCbNm1y9zJVZl8i4oEMEREPsWjRIgM467Jnzx7j888/N6688kojMDDQ8Pf3N/r06WP85z//qbCf+++/3+jRo4cRFhZm2Gw2o2XLlsbdd99tHDlyxDAMwzh06JAxefJko3379kZgYKARFBRkdOnSxXj22WeNsrKyCvv64IMPjEGDBhkhISGGzWYzEhISjOuuu8749NNPL3hfIuI5dAlMREREGh3dBSYiIiKNjgKQiIiINDoKQCIiItLoKACJiIhIo6MAJCIiIo2OApCIiIg0OpoI8QycTicHDx4kODj4rLPKioiIiGcxDIO8vDyaNm2K1XruPh4FoDM4ePAg8fHxZpchIiIiVZCWlkZcXNw52ygAnUFwcDDg+gBDQkJMrkZEREQqIzc3l/j4ePff8XNRADqDE5e9QkJCFIBERETqmcoMX9EgaBEREWl0FIBERESk0VEAEhERkUbHY8YAzZkzh1mzZnHXXXcxb968M7Z5//33efHFF0lJScHhcNCxY0ceeeQRhg8f7m6TmJjITTfddNq2RUVF+Pn51Vb5IiJyEsMwKCsro7y83OxSpIHx8fHBy8ur2vvxiAC0efNmXnnlFbp06XLOduvWrWPo0KE88cQThIaGsmjRIkaNGsVXX31F165d3e1CQkLYuXNnhW0VfkRE6kZJSQnp6ekUFhaaXYo0QBaLhbi4OIKCgqq1H9MDUH5+PhMmTODVV1/lscceO2fbU3uGnnjiCT788EP+85//VAhAFouFmJiY2ihXRETOwel0smfPHry8vGjatCm+vr6aUFZqjGEYHD58mP3799OmTZtq9QSZHoCmTZvG1VdfzZAhQ84bgE7ldDrJy8sjPDy8wvr8/HwSEhIoLy/n0ksv5dFHH60QkE7lcDhwOBzu17m5uRd2EiIiArh6f5xOJ/Hx8QQEBJhdjjRAUVFR7N27l9LS0moFIFMHQS9evJhvvvmGOXPmVGn7uXPnUlBQwLhx49zr2rdvT2JiIsuXLycpKQk/Pz8uu+wydu3addb9zJkzB7vd7l40C7SISPWc7zEEIlVVUz2Kpn1D09LSuOuuu3jrrbeqND4nKSmJRx55hCVLlhAdHe1e36dPH/7whz9wySWX0L9/f9555x3atm3LP//5z7Pua+bMmeTk5LiXtLS0Kp2TiIiI1A+mXQL7+uuvyczMpHv37u515eXlrFu3jgULFuBwOM7atbVkyRKmTJnCu+++y5AhQ855HKvVSs+ePc/ZA2Sz2bDZbFU7EREREal3TOsBGjx4MNu3byclJcW99OjRgwkTJpCSknLW8JOUlMTkyZN5++23ufrqq897HMMwSElJITY2tqZPQURE5KwGDhzI9OnTK91+7969WCwWUlJSaq0m+ZVpPUDBwcF06tSpwrrAwEAiIiLc62fOnMmBAwd44403AFf4mTRpEs899xx9+vQhIyMDAH9/f+x2OwCzZ8+mT58+tGnThtzcXObPn09KSgrPP/98HZ6diIjUF+cbU3LjjTeSmJh4wft9//338fHxqXT7+Ph40tPTiYyMvOBjXYi9e/fSokULtm7dyqWXXlqrx/Jkpt8Fdi7p6emkpqa6X7/88suUlZUxbdo0pk2b5l5/8pczOzub2267jYyMDOx2O127dmXdunX06tWrrss/zZ4jBSzelIq/rxfTh7Q1uxwREcH1t+aEJUuW8PDDD1eYS87f379C+9LS0koFm1PvUD4fLy8vTeFShzxqmH5ycnKFuX4SExNJTk6u8L5hGKctJyfzZ599ln379uFwOMjMzGTVqlX07du37k7iHBy7NzD+yzEM+mKS2aWIiNQJwzAoLCkzZTEMo1I1xsTEuBe73e6eSy4mJobi4mJCQ0N55513GDhwIH5+frz11ltkZWXxu9/9jri4OAICAujcuTNJSUkV9nvqJbDmzZvzxBNPcPPNNxMcHEyzZs145ZVX3O+fegksOTkZi8XC//73P3r06EFAQAD9+vU7baLfxx57jOjoaIKDg7nlllu4//77q9Wz43A4+NOf/kR0dDR+fn5cfvnlbN682f3+sWPHmDBhAlFRUfj7+9OmTRsWLVoEuKZBuOOOO4iNjcXPz4/mzZtX+U7v2ubRPUANTZg9hCbWDI44C3E6DaxWTQ4mIg1bUWk5HR5eZcqxf/jbcAJ8a+bP3H333cfcuXNZtGgRNpuN4uJiunfvzn333UdISAgfffQREydOpGXLlvTu3fus+5k7dy6PPvoos2bN4r333uP2229nwIABtG/f/qzbPPDAA8ydO5eoqCimTp3KzTffzBdffAHAv//9bx5//HFeeOEFLrvsMhYvXszcuXNp0aJFlc/13nvvZenSpbz++uskJCTw9NNPM3z4cH7++WfCw8N56KGH+OGHH1i5ciWRkZH8/PPPFBUVATB//nyWL1/OO++8Q7NmzUhLS/PYO6sVgOpQWIzrCxlpyeVIbh6RoSEmVyQiIpUxffp0rr322grr7rnnHvfvd955Jx9//DHvvvvuOQPQVVddxR//+EfAFaqeffZZkpOTzxmAHn/8ca644goA7r//fq6++mqKi4vx8/Pjn//8J1OmTHE/A/Phhx/mk08+IT8/v0rnWVBQwIsvvkhiYiIjR44E4NVXX2X16tUsXLiQv/zlL6SmptK1a1d69OgBuHq2TkhNTaVNmzZcfvnlWCwWEhISqlRHXVAAqkO+IVE48MFGKVnp+4gM7Wx2SSIitcrfx4sf/jb8/A1r6dg15cQf+xPKy8t58sknWbJkCQcOHHA/USAwMPCc+zn5mZcnLrVlZmZWepsTdzRnZmbSrFkzdu7c6Q5UJ/Tq1Ys1a9ZU6rxO9csvv1BaWspll13mXufj40OvXr3YsWMHALfffjtjx47lm2++YdiwYYwZM4Z+/foBMHnyZIYOHUq7du0YMWIE11xzDcOGDatSLbXNo8YANXgWC1lW1+j+vMx9JhcjIlL7LBYLAb7epiw1+QyyU4PN3LlzefbZZ7n33ntZs2YNKSkpDB8+nJKSknPu59TB0xaLBafTWeltTpzTyducep6VHft0Jie2PdM+T6wbOXIk+/btY/r06Rw8eJDBgwe7e8O6devGnj17ePTRRykqKmLcuHFcd911Va6nNikA1bE8X9es1UVZqedpKSIinurzzz9n9OjR7icPtGzZ8pwT7taWdu3asWnTpgrrtmzZUuX9tW7dGl9fX9avX+9eV1paypYtW7j44ovd66Kiopg8eTJvvfUW8+bNqzCYOyQkhPHjx/Pqq6+yZMkSli5dytGjR6tcU23RJbA6VuTfBIrBmX3A7FJERKSKWrduzdKlS9mwYQNhYWH84x//ICMjo0JIqAt33nknt956Kz169KBfv34sWbKEb7/9lpYtW55321PvJgPo0KEDt99+O3/5y18IDw+nWbNmPP300xQWFjJlyhTANc6oe/fudOzYEYfDwX//+1/3eT/77LPExsZy6aWXYrVaeffdd4mJiSE0NLRGz7smKADVsfKgWDgG1vz08zcWERGP9NBDD7Fnzx6GDx9OQEAAt912G2PGjCEnJ6dO65gwYQK7d+/mnnvuobi4mHHjxjF58uTTeoXO5IYbbjht3Z49e3jyySdxOp1MnDiRvLw8evTowapVqwgLCwPA19eXmTNnsnfvXvz9/enfvz+LFy8GICgoiKeeeopdu3bh5eVFz549WbFihUc+HNdiVOdiYQOVm5uL3W4nJyeHkJCavVNr63tP0/W7x9nsdxk9719Ro/sWETFbcXExe/bsoUWLFlV60LVU39ChQ4mJieHNN980u5Raca7v2IX8/VYPUB2zhccDEFJyyORKRESkvissLOSll15i+PDheHl5kZSUxKeffsrq1avNLs3jKQDVseBo15wI4eVHTK5ERETqO4vFwooVK3jsscdwOBy0a9eOpUuXMmTIELNL83gKQHUsvGlzACLIIb+wkKCAAHMLEhGResvf359PP/3U7DLqJc8bldTABYbGUGJ4Y7UYHEnXrfAiIiJmUACqa1YrWdYIAHIP7TW3FhERkUZKAcgEOT5RABQeUQ+QiIiIGRSATFDoFwNAefZ+kysRERFpnBSATFAW5HqYnSX3oMmViIiINE4KQGawXwSArTDD5EJEREQaJwUgE/iGxQEQpMkQRUQajIEDBzJ9+nT36+bNmzNv3rxzbmOxWPjggw+qfeya2k9jogBkghOTIYaVaTJEERGzjRo16qwTB27cuBGLxcI333xzwfvdvHkzt912W3XLq+CRRx7h0ksvPW19eno6I0eOrNFjnSoxMdEjH2paVQpAJrDHNAcg0jhGaYnD3GJERBq5KVOmsGbNGvbt23fae6+99hqXXnop3bp1u+D9RkVFEVBHk93GxMRgs9nq5FgNhQKQCcKjLqLU8MJqMcjK0K3wItKAGQaUFJizVPJZ39dccw3R0dEkJiZWWF9YWMiSJUuYMmUKWVlZ/O53vyMuLo6AgAA6d+5MUlLSOfd76iWwXbt2MWDAAPz8/OjQocMZn9d133330bZtWwICAmjZsiUPPfQQpaWlgKsHZvbs2Wzbtg2LxYLFYnHXfOolsO3bt3PllVfi7+9PREQEt912G/n5+e73J0+ezJgxY/j73/9ObGwsERERTJs2zX2sqkhNTWX06NEEBQUREhLCuHHjOHTo16Ee27ZtY9CgQQQHBxMSEkL37t3ZsmULAPv27WPUqFGEhYURGBhIx44dWbGidh8YrkdhmMDq5cUhSzixHCY7Yx8xzdqYXZKISO0oLYQnmppz7FkHwTfwvM28vb2ZNGkSiYmJPPzww1gsFgDeffddSkpKmDBhAoWFhXTv3p377ruPkJAQPvroIyZOnEjLli3p3bv3eY/hdDq59tpriYyM5MsvvyQ3N7fCeKETgoODSUxMpGnTpmzfvp1bb72V4OBg7r33XsaPH893333Hxx9/7H78hd1uP20fhYWFjBgxgj59+rB582YyMzO55ZZbuOOOOyqEvM8++4zY2Fg+++wzfv75Z8aPH8+ll17Krbfeet7zOZVhGIwZM4bAwEDWrl1LWVkZf/zjHxk/fjzJyckATJgwga5du/Liiy/i5eVFSkoKPj4+AEybNo2SkhLWrVtHYGAgP/zwA0FBQRdcx4VQADJJtk8UsaWHNRmiiIgHuPnmm3nmmWdITk5m0KBBgOvy17XXXktYWBhhYWHcc8897vZ33nknH3/8Me+++26lAtCnn37Kjh072Lt3L3FxrhthnnjiidPG7Tz44IPu35s3b86f//xnlixZwr333ou/vz9BQUF4e3sTExNz1mP9+9//pqioiDfeeIPAQFcAXLBgAaNGjeKpp56iSZMmAISFhbFgwQK8vLxo3749V199Nf/73/+qFIA+/fRTvv32W/bs2UN8fDwAb775Jh07dmTz5s307NmT1NRU/vKXv9C+fXsA2rT59R//qampjB07ls6dOwPQsmXLC67hQikAmaTALwZKf6D0mCZDFJEGzCfA1RNj1rErqX379vTr14/XXnuNQYMG8csvv/D555/zySefAFBeXs6TTz7JkiVLOHDgAA6HA4fD4Q4Y57Njxw6aNWvmDj8Affv2Pa3de++9x7x58/j555/Jz8+nrKyMkJCQSp/HiWNdcsklFWq77LLLcDqd7Ny50x2AOnbsiJeXl7tNbGws27dvv6BjnXzM+Ph4d/gB6NChA6GhoezYsYOePXsyY8YMbrnlFt58802GDBnC9ddfT6tWrQD405/+xO23384nn3zCkCFDGDt2LF26dKlSLZWlMUAmKQlwpXdL7gGTKxERqUUWi+sylBnL8UtZlTVlyhSWLl1Kbm4uixYtIiEhgcGDBwMwd+5cnn32We69917WrFlDSkoKw4cPp6SkpFL7Ns4wHslySn1ffvklN9xwAyNHjuS///0vW7du5YEHHqj0MU4+1qn7PtMxT1x+Ovk9p9N5Qcc63zFPXv/II4/w/fffc/XVV7NmzRo6dOjAsmXLALjlllvYvXs3EydOZPv27fTo0YN//vOfVaqlshSAzBLimgzRpyDd5EJERARg3LhxeHl58fbbb/P6669z0003uf94f/7554wePZo//OEPXHLJJbRs2ZJdu3ZVet8dOnQgNTWVgwd/7Q3buHFjhTZffPEFCQkJPPDAA/To0YM2bdqcdmear68v5eXl5z1WSkoKBQUFFfZttVpp27ZtpWu+ECfOLy0tzb3uhx9+ICcnh4svvti9rm3bttx999188sknXHvttSxatMj9Xnx8PFOnTuX999/nz3/+M6+++mqt1HqCApBJTkyGGOjQZIgiIp4gKCiI8ePHM2vWLA4ePMjkyZPd77Vu3ZrVq1ezYcMGduzYwf/93/+RkVH52fyHDBlCu3btmDRpEtu2bePzzz/ngQceqNCmdevWpKamsnjxYn755Rfmz5/v7iE5oXnz5uzZs4eUlBSOHDmCw3H6VCoTJkzAz8+PG2+8ke+++47PPvuMO++8k4kTJ7ovf1VVeXk5KSkpFZYffviBIUOG0KVLFyZMmMA333zDpk2bmDRpEldccQU9evSgqKiIO+64g+TkZPbt28cXX3zB5s2b3eFo+vTprFq1ij179vDNN9+wZs2aCsGpNigAmSQgqhkA9rLDJlciIiInTJkyhWPHjjFkyBCaNWvmXv/QQw/RrVs3hg8fzsCBA4mJiWHMmDGV3q/VamXZsmU4HA569erFLbfcwuOPP16hzejRo7n77ru54447uPTSS9mwYQMPPfRQhTZjx45lxIgRDBo0iKioqDPeih8QEMCqVas4evQoPXv25LrrrmPw4MEsWLDgwj6MM8jPz6dr164Vlquuusp9G35YWBgDBgxgyJAhtGzZkiVLlgDg5eVFVlYWkyZNom3btowbN46RI0cye/ZswBWspk2bxsUXX8yIESNo164dL7zwQrXrPReLcaYLk41cbm4udrudnJycCx58VlkHUndz0WtdKTcsWB8+jMXL5/wbiYh4uOLiYvbs2UOLFi3w8/MzuxxpgM71HbuQv9/qATJJZEwcZYYVL4tB9mENhBYREalLCkAmsfn6csQSDsCx9D0mVyMiItK4KACZ6KhXFAAFmgxRRESkTikAmSjfFg1ASZYmQxQREalLHhOA5syZg8ViOeOzUU62du1aunfvjp+fHy1btuSll146rc3SpUvp0KEDNputwkRLnsYR6JoM0chRABKRhkX310htqanvlkcEoM2bN/PKK6+cd9rrPXv2cNVVV9G/f3+2bt3KrFmz+NOf/sTSpUvdbTZu3Mj48eOZOHEi27ZtY+LEiYwbN46vvvqqtk/jghnBrgcEajJEEWkoTswuXFhYaHIl0lCdmBn75Md4VIXpzwLLz89nwoQJvPrqqzz22GPnbPvSSy/RrFkz5s2bB8DFF1/Mli1b+Pvf/87YsWMBmDdvHkOHDmXmzJkAzJw5k7Vr1zJv3rwzzpdgJu9Q1zNT/Is1GaKINAxeXl6EhoaSmZkJuOakOdtjGUQulNPp5PDhwwQEBODtXb0IY3oAmjZtGldffTVDhgw5bwDauHEjw4YNq7Bu+PDhLFy4kNLSUnx8fNi4cSN33333aW1OhKYzOfFQuxNyc3Mv/ESqwD/SFYDspZoMUUQajhNPKj8RgkRqktVqpVmzZtUO1qYGoMWLF/PNN9+wefPmSrXPyMg4bRrvJk2aUFZWxpEjR4iNjT1rm3NNWT5nzhz3bJR1KbRJAgARRhY4y8Fave48ERFPYLFYiI2NJTo6mtLSUrPLkQbG19cXq7X6I3hMC0BpaWncddddfPLJJxc0W+ipie/EYKiT15+pzbmS4syZM5kxY4b7dW5uLvHx8ZWuqaoiYhMoM6x4W5wUHTuIf0TtH1NEpK54eXlVe5yGSG0xLQB9/fXXZGZm0r17d/e68vJy1q1bx4IFC3A4HKf9HycmJua0npzMzEy8vb2JiIg4Z5tzPQDOZrNhs9mqe0oXLNjfRgZhxJLF0fS9XKQAJCIiUidMuwts8ODBbN++vcITZXv06MGECRNISUk5478a+vbty+rVqyus++STT+jRo4f7zoOztenXr1/tnUwVWSwWjnpFApB3WJMhioiI1BXTeoCCg4Pp1KlThXWBgYFERES418+cOZMDBw7wxhtvADB16lQWLFjAjBkzuPXWW9m4cSMLFy6scHfXXXfdxYABA3jqqacYPXo0H374IZ9++inr16+vu5O7APm+0VC8k5IsBSAREZG64hHzAJ1Neno6qam/BoMWLVqwYsUKkpOTufTSS3n00UeZP3+++xZ4gH79+rF48WIWLVpEly5dSExMZMmSJfTu3duMUzivogDX3RLlOXogqoiISF2xGJqu8zS5ubnY7XZycnIICQmp1WOtWfQwV+57ju2hQ+g8fen5NxAREZEzupC/3x7dA9QYeIVeBIBf8dlv0xcREZGapQBkMr9w151fISWaMExERKSuKACZLKRJcwAinFngdJpbjIiISCOhAGSyiCbxlBsWvCmnLE/PBBMREakLCkAmi7AHcZhQALIz9ppai4iISGOhAGQyL6uFLKtrMsTcQ3vNLUZERKSRUADyADm+0QAUZ6WZXImIiEjjoADkAYr9XM8p02SIIiIidUMByAOUBTUFwJp30ORKREREGgcFIA9gtbsmQ7QVajJEERGRuqAA5AFsEa7JEIM1GaKIiEidUADyAEFRCQCElR/RZIgiIiJ1QAHIA4THxOM0LPhShlF4xOxyREREGjwFIA/QJCyEI9gByM/cZ3I1IiIiDZ8CkAfw8/Ei0xIBQI4mQxQREal1CkAeIsfHNRli4RFNhigiIlLbFIA8ROHxyRDLju03uRIREZGGTwHIQ5QFxgBgydNs0CIiIrVNAchT2OMATYYoIiJSFxSAPIRvuCsABTo0GaKIiEhtUwDyEIGRzQAIKzsMhmFyNSIiIg2bApCHCItxBSBfSqEwy+RqREREGjYFIA8RExbKYcM1GaLjqG6FFxERqU0KQB4ixN+bQ2gyRBERkbqgAOQhLBYL2d6RABQcTjW5GhERkYZNAciDFNhckyGWHtMlMBERkdqkAORBSgJjATByDppciYiISMOmAORBjJCmAPgWpJtciYiISMOmAORBfEJdkyEGaDJEERGRWqUA5EEColxzAYWWZmoyRBERkVqkAORBQpskAGCjBIqOmVyNiIhIw6UA5EGahIdyxAgBoDx7v8nViIiINFwKQB4kMsiXDCMcgNzMfSZXIyIi0nApAHkQby8rx7yOT4aoACQiIlJrTA1AL774Il26dCEkJISQkBD69u3LypUrz9p+8uTJWCyW05aOHTu62yQmJp6xTXFxcV2cUrXlHZ8M0aHJEEVERGqNqQEoLi6OJ598ki1btrBlyxauvPJKRo8ezffff3/G9s899xzp6enuJS0tjfDwcK6//voK7UJCQiq0S09Px8/Pry5OqdocATGAJkMUERGpTd5mHnzUqFEVXj/++OO8+OKLfPnllxV6dU6w2+3Y7Xb36w8++IBjx45x0003VWhnsViIiYmpnaJrmRHcFI6Cd74mQxQREaktHjMGqLy8nMWLF1NQUEDfvn0rtc3ChQsZMmQICQkJFdbn5+eTkJBAXFwc11xzDVu3bj3nfhwOB7m5uRUWs3gdnwzRv/iQaTWIiIg0dKYHoO3btxMUFITNZmPq1KksW7aMDh06nHe79PR0Vq5cyS233FJhffv27UlMTGT58uUkJSXh5+fHZZddxq5du866rzlz5rh7l+x2O/Hx8dU+r6ryj3Qd2156SJMhioiI1BKLYZj7V7akpITU1FSys7NZunQp//rXv1i7du15Q9CcOXOYO3cuBw8exNfX96ztnE4n3bp1Y8CAAcyfP/+MbRwOBw6Hw/06NzeX+Ph4cnJyCAkJqdqJVdFXOw/QO+n4ud+3F/zD6vT4IiIi9VVubi52u71Sf79NHQME4OvrS+vWrQHo0aMHmzdv5rnnnuPll18+6zaGYfDaa68xceLEc4YfAKvVSs+ePc/ZA2Sz2bDZbFU7gRoWHRHKUSOIcEs+Rs4BLApAIiIiNc70S2CnMgyjQm/Mmaxdu5aff/6ZKVOmVGp/KSkpxMbG1lSJtSomxI8MIwKAwiO6FV5ERKQ2mNoDNGvWLEaOHEl8fDx5eXksXryY5ORkPv74YwBmzpzJgQMHeOONNypst3DhQnr37k2nTp1O2+fs2bPp06cPbdq0ITc3l/nz55OSksLzzz9fJ+dUXf6+Xhy2RgL7yD+8l0CzCxIREWmATA1Ahw4dYuLEiaSnp2O32+nSpQsff/wxQ4cOBVwDnVNTUytsk5OTw9KlS3nuuefOuM/s7Gxuu+02MjIysNvtdO3alXXr1tGrV69aP5+akucbDSVQnKXngYmIiNQG0wdBe6ILGURVG9599i6uz0lkT9wYWtzyep0fX0REpD66kL/fHjcGSMAZ1BQAL02GKCIiUisUgDyQ9fhkiH5FGSZXIiIi0jApAHkg/wjXZIghJZoMUUREpDYoAHmg4OhmAPgZxeAw77EcIiIiDZUCkAeKCg8n2zh+A3yungovIiJS0xSAPFCs3Y/045MhlhxLPU9rERERuVAKQB4oNMCHQ7gCUP4hBSAREZGapgDkgSwWC7m+UQAUZelxGCIiIjVNAchDFfk1AaAsW7NBi4iI1DQFIA9VemIyxDwNghYREalpCkAeymq/CADfQk2GKCIiUtMUgDyULdw1GWJwSabJlYiIiDQ8CkAeKuj4ZIj+zgIo1mSIIiIiNUkByENFhkeQYwS4XmgyRBERkRqlAOShTp4M0ZlzwORqREREGhYFIA8VFWwjg3AACg5rMkQREZGapADkoXy8rGR7uSZDLDyiACQiIlKTFIA8WIG/azLE0mOaDVpERKQmKQB5sNKAWAAsmgxRRESkRikAeTCL3TUbtG+BJkMUERGpSQpAHswnzDUZYqDjkMmViIiINCwKQB4sMCoBgABnPjjyTa5GRESk4VAA8mBRERHkGv6uF5oMUUREpMYoAHmwJnY/MgzXXEDkajJEERGRmqIA5MFiQn4NQEVZuhVeRESkpigAebBAmzeH3ZMh7jO5GhERkYZDAcjDFfhGA1BydL/JlYiIiDQcCkAeriTQNRmixgCJiIjUHAUgTxfsmgzRR5MhioiI1BgFIA/nHR4HQECxApCIiEhNUQDycP4RzQAIKM+DkgKTqxEREWkYFIA8XGREJPmGn+tFbrq5xYiIiDQQCkAerondX5MhioiI1DBTA9CLL75Ily5dCAkJISQkhL59+7Jy5cqztk9OTsZisZy2/PjjjxXaLV26lA4dOmCz2ejQoQPLli2r7VOpNTF2Pw4aEQCUHdOt8CIiIjXB1AAUFxfHk08+yZYtW9iyZQtXXnklo0eP5vvvvz/ndjt37iQ9Pd29tGnTxv3exo0bGT9+PBMnTmTbtm1MnDiRcePG8dVXX9X26dSK8ABfMi2uAFSgyRBFRERqhMUwDMPsIk4WHh7OM888w5QpU057Lzk5mUGDBnHs2DFCQ0PPuP348ePJzc2t0JM0YsQIwsLCSEpKqlQNubm52O12cnJyCAkJqdJ51KRFj93CTWXvcrjdBKJ+94LZ5YiIiHikC/n77TFjgMrLy1m8eDEFBQX07dv3nG27du1KbGwsgwcP5rPPPqvw3saNGxk2bFiFdcOHD2fDhg1n3Z/D4SA3N7fC4kkc/jEAOHM0BkhERKQmmB6Atm/fTlBQEDabjalTp7Js2TI6dOhwxraxsbG88sorLF26lPfff5927doxePBg1q1b526TkZFBkyZNKmzXpEkTMjLOPo/OnDlzsNvt7iU+Pr5mTq6GOEMuAsA7/6DJlYiIiDQM3mYX0K5dO1JSUsjOzmbp0qXceOONrF279owhqF27drRr1879um/fvqSlpfH3v/+dAQMGuNdbLJYK2xmGcdq6k82cOZMZM2a4X+fm5npUCPKyXwQHwL/4kNmliIiINAimByBfX19at24NQI8ePdi8eTPPPfccL7/8cqW279OnD2+99Zb7dUxMzGm9PZmZmaf1Cp3MZrNhs9mqUH3d8I88PhliWQ6UFoGPv8kViYiI1G+mXwI7lWEYOByOSrffunUrsbGx7td9+/Zl9erVFdp88skn9OvXr8ZqrGvhEVEUGscDWq4ug4mIiFSXqT1As2bNYuTIkcTHx5OXl8fixYtJTk7m448/BlyXpg4cOMAbb7wBwLx582jevDkdO3akpKSEt956i6VLl7J06VL3Pu+66y4GDBjAU089xejRo/nwww/59NNPWb9+vSnnWBNi7P6kG+G0sqS7AlBEK7NLEhERqddMDUCHDh1i4sSJpKenY7fb6dKlCx9//DFDhw4FID09ndTUVHf7kpIS7rnnHg4cOIC/vz8dO3bko48+4qqrrnK36devH4sXL+bBBx/koYceolWrVixZsoTevXvX+fnVlBi7H3uNcFqRjpGzn7OPZhIREZHK8Lh5gDyBp80DVFLmZPns0VzntY6C/g8QOPhes0sSERHxOPVyHiA5O19vK9neUQA4stJMrkZERKT+UwCqJ4oDXJMhlmdrMkQREZHqUgCqJ8qDXHe6eeUrAImIiFSXAlA9YbW7ZoP2K9RkiCIiItWlAFRP+EWcmAzxGJQWm1yNiIhI/aYAVE+EhkdTZPi6XuSlm1uMiIhIPacAVE/EhLomQwQgV+OAREREqkMBqJ6ItfuR4Q5AehyGiIhIdSgA1RNNQvxIJwKAkqOaC0hERKQ6FIDqiWA/H45YIwEo0mSIIiIi1aIAVI8U+TUBoOyYApCIiEh1KADVI2XHJ0O06i4wERGRalEAqkcs9jgAbIUZJlciIiJSvykA1SO2cFcACijNgrISk6sRERGpvxSA6hF7eBMcho/rRZ5uhRcREakqBaB6JCY04KTJEBWAREREqkoBqB6JCfEj3XDNBaQAJCIiUnUKQPVIE7uNdFw9QOXZ+02uRkREpP5SAKpHIgNtZB4PQEVZqSZXIyIiUn8pANUjVquFAtuJyRDVAyQiIlJVCkD1TEmgazJEjQESERGpOgWgesYSchEAvgWaDFFERKSqFIDqGe8w12SI/iVHNBmiiIhIFVUpAKWlpbF//69jUDZt2sT06dN55ZVXaqwwObOQiCY4DG8sGJCvXiAREZGqqFIA+v3vf89nn30GQEZGBkOHDmXTpk3MmjWLv/3tbzVaoFQUExpIhiZDFBERqZYqBaDvvvuOXr16AfDOO+/QqVMnNmzYwNtvv01iYmJN1ieniAnxI4MTAeiAucWIiIjUU1UKQKWlpdhsNgA+/fRTfvOb3wDQvn170tPTa646OY1rNmhXADJyFIBERESqokoBqGPHjrz00kt8/vnnrF69mhEjRgBw8OBBIiIiarRAqSg6xEbG8cdhOI6mmVyNiIhI/VSlAPTUU0/x8ssvM3DgQH73u99xySWXALB8+XL3pTGpHX4+XuT4RANQclSTIYqIiFSFd1U2GjhwIEeOHCE3N5ewsDD3+ttuu42AgIAaK07OrCQgBgrA0BggERGRKqlSD1BRUREOh8Mdfvbt28e8efPYuXMn0dHRNVqgnK48xDUXUOCxnXDoB5OrERERqX+qFIBGjx7NG2+8AUB2dja9e/dm7ty5jBkzhhdffLFGC5TTlUR25Ivyjng7iyHpBijIMrskERGReqVKAeibb76hf//+ALz33ns0adKEffv28cYbbzB//vwaLVBOFxMawLTSP5Hl0xSy98G7N0J5qdlliYiI1BtVCkCFhYUEBwcD8Mknn3DttdditVrp06cP+/btq9EC5XQxIX5kE8zciNngGwR7P4eV95ldloiISL1RpQDUunVrPvjgA9LS0li1ahXDhg0DIDMzk5CQkErv58UXX6RLly6EhIQQEhJC3759Wbly5Vnbv//++wwdOpSoqCh3+1WrVlVok5iYiMViOW0pLi6uyql6pCZ2PwA2FzaBsf8CLLBlIWz+l7mFiYiI1BNVCkAPP/ww99xzD82bN6dXr1707dsXcPUGde3atdL7iYuL48knn2TLli1s2bKFK6+8ktGjR/P999+fsf26desYOnQoK1as4Ouvv2bQoEGMGjWKrVu3VmgXEhJCenp6hcXPz68qp+qRWkcHYbHArsx8Xs5oC4Mfdr2x8j7Ys87c4kREROoBi2EYRlU2zMjIID09nUsuuQSr1ZWjNm3aREhICO3bt69yQeHh4TzzzDNMmTKlUu07duzI+PHjefhhVwhITExk+vTpZGdnV7mG3Nxc7HY7OTk5F9SjVZdeWfcLT6z4EYCnx3ZmXOrfYPu74B8Gt34G4S1MrlBERKRuXcjf7yr1AAHExMTQtWtXDh48yIEDrvloevXqVeXwU15ezuLFiykoKHD3KJ2P0+kkLy+P8PDwCuvz8/NJSEggLi6Oa6655rQeolM5HA5yc3MrLJ7utgGt+L8BLQG4//3tfNrmQWjaDYqOQdLvwJFncoUiIiKeq0oByOl08re//Q273U5CQgLNmjUjNDSURx99FKfTeUH72r59O0FBQdhsNqZOncqyZcvo0KFDpbadO3cuBQUFjBs3zr2uffv2JCYmsnz5cpKSkvDz8+Oyyy5j165dZ93PnDlzsNvt7iU+Pv6CzsEs949sz/Xd43Aa8Md3dvB1vwUQFAOHd8D7t8EF/m8hIiLSWFTpEtjMmTNZuHAhs2fP5rLLLsMwDL744gseeeQRbr31Vh5//PFK76ukpITU1FSys7NZunQp//rXv1i7du15Q1BSUhK33HILH374IUOGDDlrO6fTSbdu3RgwYMBZb9F3OBw4HA7369zcXOLj4z36EtgJZeVObv/3N6z+4RDBNm8+GONHq/9eD+UO6P/nX8cHiYiINHAXcgmsSgGoadOmvPTSS+6nwJ/w4Ycf8sc//tF9SawqhgwZQqtWrXj55ZfP2mbJkiXcdNNNvPvuu1x99dXn3eett97K/v37z3mH2cnqwxigkxWXljPptU1s2nOUyCBfVg5KJ2r1na43xy6EzteZW6CIiEgdqPUxQEePHj3jWJ/27dtz9OjRquzSzTCMCr0xp0pKSmLy5Mm8/fbblQo/hmGQkpJCbGxsteryZH4+Xvzrxh50iA3hSH4J134RR0GPO1xvfjgNDnxtboEiIiIepkoB6JJLLmHBggWnrV+wYAFdunSp9H5mzZrF559/zt69e9m+fTsPPPAAycnJTJgwAXBdaps0aZK7fVJSEpMmTWLu3Ln06dOHjIwMMjIyyMnJcbeZPXs2q1atYvfu3aSkpDBlyhRSUlKYOnVqVU613gjx8+H1m3uREBFA2tEirt81hNJWQ6GsGBZPgLwMs0sUERHxGFV6GvzTTz/N1Vdfzaeffkrfvn2xWCxs2LCBtLQ0VqxYUen9HDp0iIkTJ5Keno7dbqdLly58/PHHDB06FID09HRSU1Pd7V9++WXKysqYNm0a06ZNc6+/8cYbSUxMBFzPJrvtttvIyMjAbrfTtWtX1q1bR69evapyqvVKVLCNN2/uzdiXNvDDoUKm2P6PxIh9WLN+gsW/h8krwKfhzIckIiJSVVWeB+jgwYM8//zz/PjjjxiGQYcOHbjtttt45JFHeO2112q6zjpV38YAnWpHei7jXt5IXnEZv2tVyhNZd2EpzoYuN8BvXwKLxewSRUREalytD4I+m23bttGtWzfKy8trapemqO8BCGDTnqNMXPgVjjIn97XNYGraX7AY5TD0b3DZXWaXJyIiUuPqZCJE8Wy9WoTzwoRueFktPPVTDKvij4ee1X+Fnz4xtzgRERGTKQA1YIMvbsLTY12D0qf+1J3vY68FDFg6BQ7vNLc4EREREykANXBju8fx4NUXAxbG7BlDZnh3cOTC2+OhsHpTFoiIiNRXF3QX2LXXXnvO96vzAFKpPbf0b0lWQQkvJv/CVem38nlYOv7H9sC7k+EP74NXlW4GFBERqbcuqAfo5OdlnWlJSEioMG+PeI57h7fjhp7xHDFCGJdzF+XeAbBnLayaZXZpIiIida5G7wJrKBrCXWBnUlbuZNrb37Dq+0OMtn3Dc5a/u94Y9Rx0n2xqbSIiItWlu8DkjLy9rDx3Q1f6tAznQ0c3XrDc4Hpjxb1w6HtzixMREalDCkCNjJ+PF69O6kHHpiE8XTSKL6zdXU+OX3orlBabXZ6IiEidUABqhIL9fEi8qRfNIwK5q/AWsi12yPwe1jxqdmkiIiJ1QgGokYoKtvHmlN5Yg6OZ4bjVtXLjAtidbGpdIiIidUEBqBGLDw9g4Y092ejVk7fKBgNgLLtd8wOJiEiDpwDUyHWOs/PcDZfyRPkEfnHGYsk7CP+dDro5UEREGjAFIGFYxxhmXNWV6aXTKDW84IcPYdtis8sSERGpNQpAAsCUy1twSe+BPFs2FoDyj+6BY3vNLUpERKSWKAAJABaLhUdGdeSHljez2dkWr9J8HO/eCs5ys0sTERGpcQpA4ubtZeWfE3qwwP4X8gx/bAc34Uj+u9lliYiI1DgFIKkg2M+HJ6aM4hnvWwDwWvcUZWlbTK5KRESkZikAyWkuCvXn+sn3sNLZF2/KOfbWZAxHvtlliYiI1BgFIDmjzvGh2H47j3QjnChHGjveuMvskkRERGqMApCc1ZVd27O12xMAdDjwHt+sftvkikRERGqGApCc08jf3MD6KNdT4xPW38cPP+0yuSIREZHqUwCSc7JYLPS5ZR5pPi2IsOSSlfR/HDhWaHZZIiIi1aIAJOflbfMnfNIblOBDf+Nrlr78KHnFpWaXJSIiUmUKQFIpgfFdKLziYQBuLfoXj73+IWXlTpOrEhERqRoFIKm00CvuIO+i/vhbSvjDgUeZ/UEKhh6aKiIi9ZACkFSe1Urw+Fcp8Q2ls3UvTbbO41+f7zG7KhERkQumACQXJiQW3zHzAfij13JWf7yMVd9nmFyUiIjIhVEAkgvXYTTGpROwWgz+4fMCDyz+gm/3Z5tdlYiISKUpAEmVWEY+hRHWnDjLEWbyGlNe38KB7CKzyxIREakUBSCpGlswlt++gmGxMtZrPb0Lkrn19S26M0xEROoFBSCpuma9sfS/B4AnfF/jaPoekjanmVyUiIjI+SkASfVccS9c1J0QCvi7z0s8+8lOcoo0SaKIiHg2BSCpHi8fuPZVDG9/Lvf6nsuL1/L8Zz+bXZWIiMg5mRqAXnzxRbp06UJISAghISH07duXlStXnnObtWvX0r17d/z8/GjZsiUvvfTSaW2WLl1Khw4dsNlsdOjQgWXLltXWKQhARCss/f8MwCyff/POFz+yL6vA5KJERETOztQAFBcXx5NPPsmWLVvYsmULV155JaNHj+b7778/Y/s9e/Zw1VVX0b9/f7Zu3cqsWbP405/+xNKlS91tNm7cyPjx45k4cSLbtm1j4sSJjBs3jq+++qquTqtx6ncnRlhzYizHuM3yPk+u/NHsikRERM7KYnjYswzCw8N55plnmDJlymnv3XfffSxfvpwdO3a4102dOpVt27axceNGAMaPH09ubm6FnqQRI0YQFhZGUlLSGY/pcDhwOBzu17m5ucTHx5OTk0NISEhNnVrDt3MlJN1AieHF8JKnefLW39K7ZYTZVYmISCORm5uL3W6v1N9vjxkDVF5ezuLFiykoKKBv375nbLNx40aGDRtWYd3w4cPZsmULpaWl52yzYcOGsx57zpw52O129xIfH1/Ns2mk2o6A1kPxtZTzsPcbPPbfH3A6PSpfi4iIAB4QgLZv305QUBA2m42pU6eybNkyOnTocMa2GRkZNGnSpMK6Jk2aUFZWxpEjR87ZJiPj7I9rmDlzJjk5Oe4lLU23cleJxQIjnsSw+jDIaxvRGZ+xbOsBs6sSERE5jekBqF27dqSkpPDll19y++23c+ONN/LDDz+ctb3FYqnw+sQVvJPXn6nNqetOZrPZ3AOxTyxSRZGtsfSdBsDD3m8y7+NvKSwpM7koERGRikwPQL6+vrRu3ZoePXowZ84cLrnkEp577rkzto2JiTmtJyczMxNvb28iIiLO2ebUXiGpRQP+ghEUS4I1k98ULuPltbvNrkhERKQC0wPQqQzDqDAg+WR9+/Zl9erVFdZ98skn9OjRAx8fn3O26devX+0ULKezBWEZ/hgAd3h/wPJ1X5GRU2xyUSIiIr8yNQDNmjWLzz//nL1797J9+3YeeOABkpOTmTBhAuAamzNp0iR3+6lTp7Jv3z5mzJjBjh07eO2111i4cCH33HOPu81dd93FJ598wlNPPcWPP/7IU089xaeffsr06dPr+vQat05jMRL64W8p4c+8ydOrdFu8iIh4DlMD0KFDh5g4cSLt2rVj8ODBfPXVV3z88ccMHToUgPT0dFJTU93tW7RowYoVK0hOTubSSy/l0UcfZf78+YwdO9bdpl+/fixevJhFixbRpUsXEhMTWbJkCb17967z82vULBYsI5/GsFi5xusr0lM+4dv92WZXJSIiAnjgPECe4ELmEZDzWPEX2PQKPzkv4q+xL/P21MvPOSBdRESkqurlPEDSQA2aRbl/OG2tB7h4/xI+/u7s0xGIiIjUFQUgqV3+YXgN+SsA073f46UVX+IoKze5KBERaewUgKT2dZ1IecwlhFiKmJC3iNc37DW7IhERaeQUgKT2Wb3wunouAOO817L2fyvIyj/zVAciIiJ1QQFI6kZ8T4xLfg/AvcZrzFut2+JFRMQ8CkBSZyxDZ1PmE8Ql1t2UbnmTnw7lmV2SiIg0UgpAUneCovG+chYAf/FezLP/2WRyQSIi0lgpAEnd6nUbJeFtibDk0XvvSyTvzDS7IhERaYQUgKRuefnge83fAZjotZq3l6+grNxpclEiItLYKABJ3Wt5BaXtfoOXxWBK3oskbUo9/zYiIiI1SAFITOEz8gnKrH70tv7ID58sIqeo1OySRESkEVEAEnOExmMZ8GcA7nK+ziuffmtyQSIi0pgoAIlpvC77E0WB8cRYjhGy6Tn2ZRWYXZKIiDQSCkBiHh8//EY9BcBN1v/y2oerTS5IREQaCwUgMZWl3VXkxw/E11LOwD3/4KtfjphdkoiINAIKQGIui4Wg0XMps3gzyGsbqz54HafTMLsqERFp4BSAxHyRrSnpeTsAN+a8xIdbdptckIiINHQKQOIRAgbfT4FvFAnWTA6ufIaUtGyzSxIRkQZMAUg8gy0In5GPA3Cr8x1ef+XvLNu63+SiRESkoVIAEo/he+k4Sjteh6+lnGe9/sm37z3JnBU7KNeYIBERqWEKQOI5LBZ8xr6K0es2AP7q8yYhG57glsRN5BZrpmgREak5CkDiWaxWLCOfhisfAmCa93JG7H6CsQvWseeIJkoUEZGaoQAknsdigQH3wKj5GBYr472TuTfnMcYv+B+f7zpsdnUiItIAKACJ5+p+I5bxb2F4+zHU6xuedz7Kna+t4bX1ezAMjQsSEZGqUwASz9b+aiwTl2HYQuhp/YklPn/j5f+u576l3+IoKze7OhERqacUgMTzJfTDcvPHGEExtLPu533bI3z99VdMePUrDuc5zK5ORETqIQUgqR+adMQy5ROIaM1FliO8Z/sbZambGb1gPd8dyDG7OhERqWcUgKT+CEuAm1dB026EkUeS7XHa5H3FdS9t4KNv082uTkRE6hEFIKlfAiPhxv9Aqyvxx8FC378zvHwd097+hn98slMPUhURkUpRAJL6xxYEv1sCna7Dm3Ke832BKV4rmL/mZ6a+9TUFjjKzKxQREQ+nACT1k7cvXPsq9HY9Rf4hn7eY5ZPEJz9kMPbFDaQdLTS5QBER8WQKQFJ/Wa0wYg4M/isAt3n9hwX+r/JzxjFGP/8FX+7OMrlAERHxVApAUr9ZLNB/Box+HixeXGMkkxT8TwoL8pi0cBMbfjlidoUiIuKBTA1Ac+bMoWfPngQHBxMdHc2YMWPYuXPnObeZPHkyFovltKVjx47uNomJiWdsU1xcXNunJGbp+ge44d/g7UfP0s381/4M/uW53Pr6Fr7dn212dSIi4mFMDUBr165l2rRpfPnll6xevZqysjKGDRtGQcHZH3r53HPPkZ6e7l7S0tIIDw/n+uuvr9AuJCSkQrv09HT8/Pxq+5TETO1GwsQPwM9Oa8cPrAh6lJjSVCYv2szPmflmVyciIh7E28yDf/zxxxVeL1q0iOjoaL7++msGDBhwxm3sdjt2u939+oMPPuDYsWPcdNNNFdpZLBZiYmJqvmjxbAl94aaP4a2xXJSXxn9tD/GXoluZtNDKu7f346JQf7MrFBERD+BRY4Byclwz+oaHh1d6m4ULFzJkyBASEhIqrM/PzychIYG4uDiuueYatm7detZ9OBwOcnNzKyxSjzXpALclQ/P++FPMAt9/cmvBy9z0r/Vk5evRGSIi4kEByDAMZsyYweWXX06nTp0qtU16ejorV67klltuqbC+ffv2JCYmsnz5cpKSkvDz8+Oyyy5j165dZ9zPnDlz3D1Ldrud+Pj4ap+PmCy4iety2OV3A3CT9yqezL2fP/9rJXnFpebWJiIiprMYhuERU+dOmzaNjz76iPXr1xMXF1epbebMmcPcuXM5ePAgvr6+Z23ndDrp1q0bAwYMYP78+ae973A4cDh+7RnIzc0lPj6enJwcQkJCLvxkxLP8uILy9/8Pr5JcsoxgXoqcxZ+n/h9+Pl5mVyYiIjUoNzcXu91eqb/fHtEDdOedd7J8+XI+++yzSocfwzB47bXXmDhx4jnDD4DVaqVnz55n7QGy2WyEhIRUWKQBaX8VXlPXUhTRkQhLHjOPzOLjF/5MWZlmjBYRaaxMDUCGYXDHHXfw/vvvs2bNGlq0aFHpbdeuXcvPP//MlClTKnWclJQUYmNjq1Ou1GfhLfGf+j8yW4/DajEYc2wRu+ZdjbPgqNmViYiICUwNQNOmTeOtt97i7bffJjg4mIyMDDIyMigqKnK3mTlzJpMmTTpt24ULF9K7d+8zjheaPXs2q1atYvfu3aSkpDBlyhRSUlKYOnVqrZ6PeDgff6L/8Crf95xDseHDxflfkvtcX4wD35hdmYiI1DFTA9CLL75ITk4OAwcOJDY21r0sWbLE3SY9PZ3U1NQK2+Xk5LB06dKz9v5kZ2dz2223cfHFFzNs2DAOHDjAunXr6NWrV62ej9QPHa/+I+uvSGKvswmhJRk4/zUMtrwGnjEcTkRE6oDHDIL2JBcyiErqrzc/20aTNXczzOtr14ouN8A1z4JvgLmFiYhIldS7QdAiZpg46BK2X/4Cc0p/R5lhhW8Xw78Gw5GfzS5NRERqmQKQNGozhrWjoOc0JpQ8wGHDDpk/wCsD4YcPzS5NRERqkQKQNGoWi4XZv+lEVOcrucrxBJuNi6EkD96ZBKsegHJNmigi0hApAEmj52W18I9xl9KhbVtucMxiEb9xvbFxAbw+CnLTzS1QRERqnAKQCODrbeXFP3Tj0oRIZhffwL1e9+L0DYbUjfDSZfD9MrNLFBGRGqQAJHJcgK83r93Yk/YxwbxTcCl/8HqK0qiOUJgF706GJRMhP9PsMkVEpAYoAImcxB7gwxs396JZeAAbjoXy25JHKe53D1i9YcdyeL4XfPuO5gwSEannFIBEThEd4sdbU3oTFWzju0PFjP1xIJuGvocR0xmKjsH7t8Li30NehtmliohIFSkAiZxBs4gA3pzSC7u/D98fzGXch4UMzXuEra2nYVh9YOcKV29QytvqDRIRqYcUgETOon1MCCvv6s+t/VsQ7OfNz1kOfvvdZYx1zuFg4MVQnAMf3A7/vh5yDphdroiIXAA9CuMM9CgMOVWBo4yl3+xn0Rd72XOkAC/Kuc37I2b4LMXHKMWwhWAZ9hh0mwQWi9nliog0Shfy91sB6AwUgORsnE6D5J8yeW39Xtb/fIRWlgM84/My3ayux2c4WwzCOno+hDYzuVIRkcZHAaiaFICkMn7MyCXxi718sDWNPxgfcY/3O/hZSinxCqBk0F8J6ncbWHWVWUSkrigAVZMCkFyIrHwHSZtSSd6wkXtLFtDLuhOAXwK7wuh/0qptZ5MrFBFpHBSAqkkBSKqipMzJim8PkPm/f/KH/EQCLA4KDRvvhd7MRcPvYlD7GKxWjQ8SEaktCkDVpAAk1WEYBt9tT8G2cjpti1IA2ORsx/PB0/m/3w6jX+tIcwsUEWmgFICqSQFIaoTTybF1LxG47m/4OotwGN68WT6UPe3/j7tG9yU62M/sCkVEGhQFoGpSAJIadWwfZcun471nDQC5hj+LLKOJHDKdG/q1x0uXxUREaoQCUDUpAEmNMwz45X8UrXgI/6M/AHDICOXd4IlcMe5uOjeLMLlAEZH670L+fuseXZG6YLFA6yH43/EFzt++Qp5/U5pYsrkj/58E/Osykl5/ntyiErOrFBFpNNQDdAbqAZJaV+Yg74uXsax7hqDyXAC+tbQj57IHuHzwb7BoNmkRkQumHiART+dtI/iKPxH0l+9I6/RHivGli7GT/usn8c1TI0j78WuzKxQRadAUgETM5Gcn/ro5WO5KYXvMtZQZVroXf0nTpMF898IfKM7aZ3aFIiINkgKQiAewhV1E56mLODQxmc3+/fGyGHTK/A+Wf/Ygdck9UHTM7BJFRBoUBSARD3JR60voce9/2DhwMd9YOmCjhGY7XqXwmU7k/m8ulBaZXaKISIOgQdBnoEHQ4gnyi0v5z3uJdPvpOdpZ01zrbE3wHzQDr5YDIbKtHrYqInISzQNUTQpA4kl+2H+MT5fMZ2zu61xkyXKvd9rsWON6QHwviOsJcT3Az25ipSIi5lIAqiYFIPE0TqfBu1/9TOqqfzLA+RVdLLvxt5w6b5AFotpDfE+I6+UKRhFt1EskIo2GAlA1KQCJp8opLOWj7emsSEklZ99Wulp20c3qWppZDp++gZ/9eO9QL1cwuqi7eolEpMFSAKomBSCpDw7lFvPRt+ks33aQlLRsIsmhq3UXPbx+ZmDAXlqV/YR3efEpW53US5RwObQbCX76jotIw6AAVE0KQFLfpB0tZPm2g/xn20F+zMgDwJsyLvHZz/iYDC73201M3nas2afMK+TtB21HQJdx0HooePuaUL2ISM1QAKomBSCpz3YdyuM/2w6yfNtB9mYVutcH27y5tp0v18ekc3HpDrx2fQxHfvp1Q79Q6DgGOl8Pzfpp7JCI1DsKQNWkACQNgWEYfHcgl/986+oZSs/59XJYeKAvQ9tHc2VoOj3yPiV893+w5Gf8unFIHHQeC53HQUwnE6oXEblwCkDVpAAkDY3TafB16jGWpxxkxfZ0sgoq3kEW6GNhXORefuP1BZ2yk/Epy//1zegOrl6hztdBaLM6rlxEpPLqzcNQ58yZQ8+ePQkODiY6OpoxY8awc+fOc26TnJyMxWI5bfnxxx8rtFu6dCkdOnTAZrPRoUMHli1bVpunIuLRrFYLPZuH8+iYTnw1azBv3NyL/7uiJX1bRhBk86ag1GBRegK/3f97OuUvYGrJdP5HL0rxgcwf4H+zYV5neG0kbF4IhUfNPiURkWoxtQdoxIgR3HDDDfTs2ZOysjIeeOABtm/fzg8//EBgYOAZt0lOTmbQoEHs3LmzQrqLiorCy8sLgI0bN9K/f38effRRfvvb37Js2TIefvhh1q9fT+/evc9bl3qApDFxOg12H8lnW1oO2/Zns21/DjsO5lJS7iSEAkZ4bWKM9Qv6WHdgtbj+c1Fu8San6RX49biBgI7XgG+AyWchIlKPL4EdPnyY6Oho1q5dy4ABA87Y5kQAOnbsGKGhoWdsM378eHJzc1m5cqV73YgRIwgLCyMpKem8dSgASWNXUubkx4xctu3PYVtaNt/uzyYvcx/XWDcyxusLOlor3k3msNgo8QqkzDsIwxaEl18wPgF2bIGhePmHgC3YtfgG//q7ezn+vn8oeNvMOWERaRAu5O+3dx3VVCk5OTkAhIeHn7dt165dKS4upkOHDjz44IMMGjTI/d7GjRu5++67K7QfPnw48+bNO+O+HA4HDofD/To3N7cK1Ys0HL7eVrrEhdIlLpSJfRIAyHeU8d2Bq/lifzbv/7yNZgc/YlDJWppZD2MzHNjKHFB2FIqBnCoc1MsXWgxw3ZbfbiTY42r0nERETuYxAcgwDGbMmMHll19Op05nv+skNjaWV155he7du+NwOHjzzTcZPHgwycnJ7l6jjIwMmjRpUmG7Jk2akJGRcaZdMmfOHGbPnl1zJyPSAAXZvOnTMoI+LSNgQCvgWrLyitm8/wBZWUc4ejSLnOyj5Oceoyg/m5KCHPychQRbigji+GIpJJgiAi3Fx18Xud+3lpfAz5+6lhX3kG1vT078EMpaDyeoRQ/CAv3w9dat+SJSMzzmEti0adP46KOPWL9+PXFxF/Yvv1GjRmGxWFi+fDkAvr6+vP766/zud79zt/n3v//NlClTKC4+dWbcM/cAxcfH6xKYSDUYhkFWQQkHs4s4cKyIA9lF7D9W5Hp9fMkuLD3RmtaWAwyxfsNgr2/oZtmFl+XX/zQdMkL5X3lXvvTuxa6g7gQHhRAe6EtEkC8RQTYig3yJCLQREeRLdLCN+PAAfLwUlkQam3p3CezOO+9k+fLlrFu37oLDD0CfPn1466233K9jYmJO6+3JzMw8rVfoBJvNhs2msQciNclisRAZZCMyyEaXuNAztilwlHEwu4j92UVk5BRztGAwK/IdLMvOpNnR9XTM30C30m9oYsnm996f8Xs+oyjPl/U5nfifsxuflHflMGGn7dfbaqFZRAAtI4NoFR1Iq+M/W0YGERao2a5FxOQeIMMwuPPOO1m2bBnJycm0adOmSvu57rrrOHr0KGvWrAFcg6Dz8vJYsWKFu83IkSMJDQ3VIGiR+qbMgXP355Ts+AivXavwyT9Q4e39/hfzjV8fPrd25+viONJzHRSVlp91d+GBvrSMDKRVVBAto3792Sw8AG/1GonUa/WmB2jatGm8/fbbfPjhhwQHB7t7bex2O/7+/gDMnDmTAwcO8MYbbwAwb948mjdvTseOHSkpKeGtt95i6dKlLF261L3fu+66iwEDBvDUU08xevRoPvzwQz799FPWr19f9ycpItXjbcPadgh+bYeAYcCh72HnSvhpJRz4mriiHcQV7eA3LIKQOIyul1No8SfbYSXLAVlFkFlokFFgcLgYSoq9KU3zJj/Nhy14swEfSvCm3OpLeEgQMeF2YsPtNIkMJyjiIsKC/IgI9CU8yJdgmzcWi8XsT0REaoCpPUBn+w/JokWLmDx5MgCTJ09m7969JCcnA/D000/zyiuvcODAAfz9/enYsSMzZ87kqquuqrCP9957jwcffJDdu3fTqlUrHn/8ca699tpK1aUeIJF6Ii8DfloFP30Mv3wGZUU1unuH4c0BI5L9RhRpRjQHLdEc840l3/8iHEFx+ARHERFkIyzQ1xWSAm2EBfoQEWgjPNCXsAAf9SqJ1KF6Ow+Qp1AAEqmHSotg91o4tB3KHK6lvBTKHVBWcvynA8pLTvnpwCgvpbykmPLSYpylDizlJfg6i7DiPOch8w2/4+Eoyh2S9huRpBnRpBlR5BNAZJCN3i3Cuax1JJe3jqRZhCaNFKktCkDVpAAkIpSXQe4ByN4Hx/ZRlrWHkqy9GMf24Z2bhq3o0Hl3ccwIYr8RSaYRRqYRSiahlPpHEx3bjBYtWtGpXVvCouM0AaRIDVEAqiYFIBE5r9JiyEmDY/sgey9kpx7/3RWYKKr889IKvUIwgprgF9YUr5BYCG4CQTG//gxqAsExYAuqvfMRaQDqzSBoEZF6y8cPItu4ljNx5LlCUXYa5GdA3iFKcw6SnbmfkuyD+BRmEuo8hq+lnIDyXMjJhZxd5z5mYBSEtYDwFhDe8tffw1pAYCRogLZIpSkAiYjUBlswNOnoWo7zAaJOanI4t5jNP/7Cjp9+Im3fHqwFh4i2ZLuXWGsOcT65hBvH8CkvhILDrmX/ptOP5xsM4c2Ph6KWvwaj8BYQchFYvWr7jEXqFV0COwNdAhORumYYBvuyCln/8xG++PkIG37JIqeo1P1+CAV0D8lhWGwhPUKyaW7NxCdnHxzd7RqrdC5evhCa4ApDEa2h+eXQvD/46b9v0rBoDFA1KQCJiNnKnQbfH8xh/c9H2PBzFpv2HqWk7Ne70ny9rPRqEc7AdlEMahVMS+8sLMf2ugLRsT1wdI/r57F94Cw9/QAWL4jrAa2uhJaD4KLu4KWLAlK/KQBVkwKQiHiawpIyvtydRfLOw3y2M5O0oxXnPIoL82dguygGto2mX+sIAnyPhxlnOeTs/zUUHfoOdidD1s8VD2ALgRYDoOVAVygKb6kxRVLvKABVkwKQiHgywzDYfaSA5J2HSd6ZyVe7j1JSfubeoYHtomkVFXj6xLPZqa7JI39ZA3vWQtGxiu+HNnP1DLUaBC2ugIDwOjgzkepRAKomBSARqU8KS8rY+MuvvUP7j525d2hQu2j6tjqpd+gEZzmkb3OFod3JkPrlKZfNLNC0q6tnqNUgiOsF3nqorHgeBaBqUgASkfrKMAx+OVxA8s5MknceZtOeir1DVgvY/X1cS4Avof4+hAa4XoceXxfhU0JCfgpNs74kNH09tmM/VTyITyAk9IPYS6BJB4ju4Bpc7eVTx2crUpECUDUpAIlIQ1HgON479FMmn/14mAPZF/68tCYc5XLrdwzy2U4/y3eEk3NaG8PqgyWyLURf/Gsoir4Y7M3AquehSd1QAKomBSARaYgMw+BwvoOcwlKyi0rJLiwlp6iU7MKS4z+Pvy4qJefEuiLXuhN/KSw4udiSSi/rj7SzpNHOmkZby36CLMVnPqhvEES1Px6MOrp+RneAoOi6O3FpNDQTtIiInMZisRAd7Ed0sN8Fbed0GuQ5yo4HJ1cwSs8p5vsDOSzZn8OO9Gyiyg/T1pJGO8t+2llTaWfZT2vrQXxK8uHAFtdysoDIX8NQWALY410Dr0ObgX9YhTvQDMPgaEEJGbnFHMot5lCug4wc1+8ZucVk5BSTW1RKh6Z2+reJ5PI2kbSMPMPAb5GTqAfoDNQDJCJSeSVlTn46lMe2/dl8m5bDtv3Z7MrMx+Ispbklg3aW/bS1ptHekkYH7wNcZGRg5ex/ekqs/mR5R5NOFPvKI/ilJJx95REcMCLZb0RxGDsG576s1tTux+VtIrm8TRSXtYogIkgPnG0MdAmsmhSARESqp6iknO8P5vDt/hy+3Z/Nt/tz2H2kAAA/HLSxHKCdNY3WlgPEWQ4TZznCRZYjRFlOH190qlJ8yPWNptC/KaXBF2EJbYZhb8a2/BDWZPizer8XxeUVt+kQG+LuHerZPBw/nxp6NIhhuB5PcuSn48su18+je8BiBW8/8LaBj7/rp7ffScup623gfdJrHz/wCXBNUqlpCCpFAaiaFIBERGpeTlEp3x3IcfcUfbs/m4M5xfj7eBFj9yM62EZcsIW2thwSvLNoajlCVHkmIY50/AsOYM3dD7kHwSg/53EMqw/FAbGkW5vwY3E43xXY2W9Ek2ZEkWZEk+cdSs/mEa4eotaRdIgNwWo9z+Wy8jI4tvekoHPSUnz+0FYtVh9oPQQ6XwftRoJvYO0erx5TAKomBSARkbpRXFqOzdta+fE65WWQdxCy0yAn7fjPVNfEjsf2uWa9PtOjP05SaNjYb0SSZkSz34gkyyeW4JhWNGt1MZ3btyOWLCxZu+Dwzl97dY7uPsd+La5xTJFtjy9tILyVqweorBjKHMd/Fv/6urTopPUOKDvl9Yn3Cw5D1q5fD+UTAO2ucoWhVoM1H9MpFICqSQFIRKSecpZDXrorDGXvO+2nkXsQyznGH52L4R0Aka1dt/tHtoWo4z/DW7ouZdWWzB/hu/dg+3uuR5qc4BcKHUa7wlDCZWCtoct69ZgCUDUpAImINFBlDlcv0fFQVH50L9kHf6bkyB4CCvZjN3LINEL5xdmUX4xYfjYu4hejKb84m5JOOEF+vrSIDCQhIpAWEQE0P/F7ZCBhAT61e+eZYcCBb1xh6Lv3IT/j1/eCY6Hjta4w1LRro32OmwJQNSkAiYg0TsUOB2nZJew5UsDerAL2ZhWy90gBe48UcDDnLHMdHRfi503zyECaRwSSEBFAsJ83Nm8vbN5W/HxcP20+Vvc6m7cXfide+1jd62ze1vOPSXKWw971rjD0w4cVxyGFt4RO17nCUFS7GvhUTnIiMhgGYPz62v37+d4/6XerV42PZ1IAqiYFIBEROVVxaTmpRwtd4ejISeEoq4D084SjC+XrdTwQ+XgRHuhDVLCNqCCb6+eJJcjP9dMfQg+uw/r9Uti5EkoLf91RTGdXGOo01nUnmSPPtRTngiP3+Ovck9bn/Pp7hfUnta3iJcTTxPWCW1bXzL6OUwCqJgUgERG5EMWl5ezLcoWjfVkFpB4tpKikHEeZk+JS109H2fGfpa7fi0t/XVdcWo6zGn+Nva0WIoJ8iQ9yMtT6Df0da2mX/xVe57ljzkyF0d0J+OOaGt2nAlA1KQCJiEhdKyt3nhKYnBSVlHO0oITD+cUcznP8uuQ7OJJXwuF8B0cLSs64v1DyGOm1id9YN9LbugOrxaDcsJCPP3kEkGf4k48/hQRQaA2gyBqIwxpAiVcgDu9gSr0CKfMJwukTRLlvCIYtCGxBOJxeFJc6KS5zUnTSz5JSJ0Vl5RSVnlhvUFLmxDg+7Nz189dLe90Swkm6/Yoa/Qz1KAwREZF6xtvLireXlUDbhf1pLi13kpVfcjwYnRqU2vBs3vXk5x2joMTJsVJvdzCpmsLzvG89vpzO19uKv49r3JO/jxcR9uAq1lAzFIBERETqMR8vKzF2P2LsfoC9Uts4nYarh6m0nOLji+t3J47ScorLyikqcfVGFR+/XHeinY+X1R1ibD5e+B9f/Hy88Pd1DeT29z3++njg8fP2Ov/A7jqmACQiItLIWK0W/H1dQaWxOvfT5EREREQaIAUgERERaXQUgERERKTRUQASERGRRkcBSERERBodBSARERFpdBSAREREpNExNQDNmTOHnj17EhwcTHR0NGPGjGHnzp3n3Ob9999n6NChREVFERISQt++fVm1alWFNomJiVgsltOW4uKafVidiIiI1E+mBqC1a9cybdo0vvzyS1avXk1ZWRnDhg2joKDgrNusW7eOoUOHsmLFCr7++msGDRrEqFGj2Lp1a4V2ISEhpKenV1j8/Pxq+5RERESkHvCoh6EePnyY6Oho1q5dy4ABAyq9XceOHRk/fjwPP/ww4OoBmj59OtnZ2VWqQw9DFRERqX8u5O+3R40BysnJASA8PLzS2zidTvLy8k7bJj8/n4SEBOLi4rjmmmtO6yE6mcPhIDc3t8IiIiIiDZfHBCDDMJgxYwaXX345nTp1qvR2c+fOpaCggHHjxrnXtW/fnsTERJYvX05SUhJ+fn5cdtll7Nq164z7mDNnDna73b3Ex8dX+3xERETEc3nMJbBp06bx0UcfsX79euLi4iq1TVJSErfccgsffvghQ4YMOWs7p9NJt27dGDBgAPPnzz/tfYfDgcPhcL/Ozc0lPj5el8BERETqkQu5BOYRT4O/8847Wb58OevWrat0+FmyZAlTpkzh3XffPWf4AbBarfTs2fOsPUA2mw2bzeZ+fSIT6lKYiIhI/XHi73Zl+nZMDUCGYXDnnXeybNkykpOTadGiRaW2S0pK4uabbyYpKYmrr766UsdJSUmhc+fOldp/Xl4egC6FiYiI1EN5eXnY7fZztjE1AE2bNo23336bDz/8kODgYDIyMgCw2+34+/sDMHPmTA4cOMAbb7wBuMLPpEmTeO655+jTp497G39/f/fJzp49mz59+tCmTRtyc3OZP38+KSkpPP/885Wqq2nTpqSlpREcHIzFYqnRcz5xeS0tLU2X12qRPue6oc+5buhzrjv6rOtGbX3OhmGQl5dH06ZNz9vW1AD04osvAjBw4MAK6xctWsTkyZMBSE9PJzU11f3eyy+/TFlZGdOmTWPatGnu9TfeeCOJiYkAZGdnc9ttt5GRkYHdbqdr166sW7eOXr16Vaouq9Va6UtxVRUSEqL/c9UBfc51Q59z3dDnXHf0WdeN2vicz9fzc4LHDIJuLDTHUN3Q51w39DnXDX3OdUefdd3whM/ZY26DFxEREakrCkB1zGaz8de//rXCXWdS8/Q51w19znVDn3Pd0WddNzzhc9YlMBEREWl01AMkIiIijY4CkIiIiDQ6CkAiIiLS6CgAiYiISKOjACQiIiKNjgJQHXrhhRdo0aIFfn5+dO/enc8//9zskhqcRx55BIvFUmGJiYkxu6x6b926dYwaNYqmTZtisVj44IMPKrxvGAaPPPIITZs2xd/fn4EDB/L999+bU2w9dr7PefLkyad9v/v06WNOsfXYnDlz6NmzJ8HBwURHRzNmzBh27txZoY2+09VXmc/ZzO+0AlAdWbJkCdOnT+eBBx5g69at9O/fn5EjR1Z4zIfUjI4dO5Kenu5etm/fbnZJ9V5BQQGXXHIJCxYsOOP7Tz/9NP/4xz9YsGABmzdvJiYmhqFDh7ofLCyVc77PGWDEiBEVvt8rVqyowwobhrVr1zJt2jS+/PJLVq9eTVlZGcOGDaOgoMDdRt/p6qvM5wwmfqcNqRO9evUypk6dWmFd+/btjfvvv9+kihqmv/71r8Yll1xidhkNGmAsW7bM/drpdBoxMTHGk08+6V5XXFxs2O1246WXXjKhwobh1M/ZMAzjxhtvNEaPHm1KPQ1ZZmamARhr1641DEPf6dpy6udsGOZ+p9UDVAdKSkr4+uuvGTZsWIX1w4YNY8OGDSZV1XDt2rWLpk2b0qJFC2644QZ2795tdkkN2p49e8jIyKjw/bbZbFxxxRX6fteC5ORkoqOjadu2LbfeeiuZmZlml1Tv5eTkABAeHg7oO11bTv2cTzDrO60AVAeOHDlCeXk5TZo0qbC+SZMmZGRkmFRVw9S7d2/eeOMNVq1axauvvkpGRgb9+vUjKyvL7NIarBPfYX2/a9/IkSP597//zZo1a5g7dy6bN2/myiuvxOFwmF1avWUYBjNmzODyyy+nU6dOgL7TteFMnzOY+532rvUjiJvFYqnw2jCM09ZJ9YwcOdL9e+fOnenbty+tWrXi9ddfZ8aMGSZW1vDp+137xo8f7/69U6dO9OjRg4SEBD766COuvfZaEyurv+644w6+/fZb1q9ff9p7+k7XnLN9zmZ+p9UDVAciIyPx8vI67V8OmZmZp/0LQ2pWYGAgnTt3ZteuXWaX0mCduMtO3++6FxsbS0JCgr7fVXTnnXeyfPlyPvvsM+Li4tzr9Z2uWWf7nM+kLr/TCkB1wNfXl+7du7N69eoK61evXk2/fv1MqqpxcDgc7Nixg9jYWLNLabBatGhBTExMhe93SUkJa9eu1fe7lmVlZZGWlqbv9wUyDIM77riD999/nzVr1tCiRYsK7+s7XTPO9zmfSV1+p3UJrI7MmDGDiRMn0qNHD/r27csrr7xCamoqU6dONbu0BuWee+5h1KhRNGvWjMzMTB577DFyc3O58cYbzS6tXsvPz+fnn392v96zZw8pKSmEh4fTrFkzpk+fzhNPPEGbNm1o06YNTzzxBAEBAfz+9783ser651yfc3h4OI888ghjx44lNjaWvXv3MmvWLCIjI/ntb39rYtX1z7Rp03j77bf58MMPCQ4Odvf02O12/P39sVgs+k7XgPN9zvn5+eZ+p02596yRev75542EhATD19fX6NatW4VbAaVmjB8/3oiNjTV8fHyMpk2bGtdee63x/fffm11WvffZZ58ZwGnLjTfeaBiG67bhv/71r0ZMTIxhs9mMAQMGGNu3bze36HroXJ9zYWGhMWzYMCMqKsrw8fExmjVrZtx4441Gamqq2WXXO2f6jAFj0aJF7jb6Tlff+T5ns7/TluNFioiIiDQaGgMkIiIijY4CkIiIiDQ6CkAiIiLS6CgAiYiISKOjACQiIiKNjgKQiIiINDoKQCIiItLoKACJiIhIo6MAJCIiIo2OApCIiIg0OgpAIiIi0uj8PzGOVKz5v65zAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "And thik bry cowd,\n",
      "This bor thibe sou bobe to:\n",
      "ave rud my thichanss:\n",
      "Warth fou qor, ve bar dilth afe aw cramy.\n",
      "\n",
      "Hhy ar mereou waow somtof is he ce mil nowlincaes ireees, hein latiser lilv the and the non ond wans!\n",
      "\n",
      "Aplind pealltliser cechiry: tur hais's, why hou to u nor\n",
      "To thigh sond:\n",
      "Il wo to thake o Windo wher eiibk we ati dourive we hidend thoo mowr-x'd und kad nonrtf he sor; iris! mef thin inled,\n",
      "The af Pre?\n",
      "\n",
      "KIS\n",
      "INUSH:\n",
      "Nube!\n",
      "\n",
      "Giyd is:\n",
      "ards beace Eghes bidin cou afar tey ir-ltome fronf ve y\n",
      "CPU times: user 2min 56s, sys: 421 ms, total: 2min 57s\n",
      "Wall time: 2min 56s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size:int=vocab_size, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 head_size:int=head_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = head_size\n",
    "\n",
    "        #This will be our lookup table for embeddings. We'll have an entry for each token (aka vocab size) and each embedding will... \n",
    "        #...be a vector of dimension embedding_dim.\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        self.token_position_embedding_table = nn.Embedding(num_embeddings=self.context_length, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        # self.multi_self_attention_heads_layer = MultiHeadAttention(num_heads=4, head_size=self.embedding_dim//4)\n",
    "        # self.feed_forward_network = FeedForwardNetwork(embedding_dim=self.embedding_dim)\n",
    "        self.transformer_blocks = nn.Sequential(\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "        ) #NEW\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        B,T = idx.shape\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        token_position_embeddings = self.token_position_embedding_table(torch.arange(T, device=device)) #(T,C)\n",
    "        x = token_embeddings + token_position_embeddings\n",
    "        # x = self.multi_self_attention_heads_layer(x) # (B,T,C)\n",
    "        # x = self.feed_forward_network(x) # (B,T,C)\n",
    "        x = self.transformer_blocks(x) #NEW\n",
    "        logits = self.language_model_head_linear_layer(x) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Crop idx to the max size of our positional embeddings table \n",
    "            idx_crop = idx[:,-self.context_length:]\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx_crop)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = TransformerLanguageModel(vocab_size=vocab_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "tracked_losses = list()\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        tracked_losses.append(losses)\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "plot_losses(tracked_losses)\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c64e49d7-4215-4494-8009-18f1c6b8c42a",
   "metadata": {},
   "source": [
    "As you can see the accuracy actually got worse. Given our new much more powerful model, this is not something that we want. As the depth of models increase they can become harder to train. Fortunately there are a few things that we can do about that. [link](https://youtu.be/kCc8FmEb1nY?t=5208)<br>\n",
    "\n",
    "First we can implement skip connections, also known as residual connections, which are depicted on the transformer architecture diagram as black lines that bypass the masked multi-head attention block and feed into the add and norm block. You can also see one bypassing the FFN. The idea for these originally came from deep residual networks paper. In this case we are going to add the input data back to the output of the blocks that are being skipped. When you use addition, the gradients are evenly distributed between both the skip branch and the block branch. An alternative that is sometimes used is a simple concatenation of the input and output of the skipped block. <br>\n",
    "\n",
    "When we initialize the network before training we typically want to start off with very low weights for the branches that go through the blocks so the blocks contribute very little to the overall loss. This way the gradient signal makes its way through the entire network. Then during training the network will slowly increase the weights and participation of the blocks.\n",
    "\n",
    "Now let's implement the skip connections in our TransfomerBlock module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b48cd6ed-a8b3-4c72-a9da-215310063399",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class TransformerBlock(nn.Module):\n",
    "    \"\"\"Transformer Block: Communication folled by computation.\"\"\"\n",
    "    def __init__(self, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 num_heads:int=4):\n",
    "        #embedding_dim: embedding dimension, num_heads: the number of heads that we want\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = embedding_dim // num_heads\n",
    "        self.num_heads = num_heads\n",
    "        \n",
    "        self.multi_self_attention_heads_layer = MultiHeadAttention(num_heads=self.num_heads, head_size=self.head_size, \n",
    "                                                                   embedding_dim=embedding_dim, context_length=context_length)\n",
    "        self.feed_forward_network = FeedForwardNetwork(embedding_dim=self.embedding_dim)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # return self.feed_forward_network(self.multi_self_attention_heads_layer(x))\n",
    "        x = x + self.multi_self_attention_heads_layer(x) # adding input back to the output of each block for skip connection. NEW\n",
    "        x = x + self.feed_forward_network(x) # adding input back to the output of each block for skip connection. NEW\n",
    "        return x\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c85468-f605-4dbc-b9eb-e5308f056ce3",
   "metadata": {},
   "source": [
    "We also need to add a projection layer to our MultiHeadAttention module as well as the feed forward network. This is a simple linear layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1279c00e-1699-4149-a52d-0f086183409b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, num_heads:int, head_size:int=head_size, embedding_dim:int=embedding_dim, context_length:int=context_length):\n",
    "        super().__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.head_size = head_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        \n",
    "        self.heads = nn.ModuleList([\n",
    "            Head(head_size=self.head_size, embedding_dim=self.embedding_dim, \n",
    "                 context_length=self.context_length) \n",
    "            for _ in range(self.num_heads)])\n",
    "        \n",
    "        self.projection_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.embedding_dim, bias=True) #NEW\n",
    "        \n",
    "    def forward(self, x):\n",
    "        out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
    "        out = self.projection_layer(out) #NEW\n",
    "        return out\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76785a9-a6f8-4a57-a357-d7b8add0244f",
   "metadata": {},
   "source": [
    "In the FFN rather than adding the same projection layer parameter we'll simply just add an additional linear layer to the existing sequential module. Also we are going to fan out and then back in by a factor of 4 between the linear layers in the FFN to add additional computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "802eff3a-a4d0-41be-a8a7-aa55dc0f20ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class FeedForwardNetwork(nn.Module):\n",
    "    \"\"\"A simple linear network followed by a non-linearity\"\"\"\n",
    "    def __init__(self, embedding_dim:int=embedding_dim):\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.ffn = nn.Sequential(\n",
    "            nn.Linear(in_features=self.embedding_dim, out_features=self.embedding_dim*4),#Updated\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(in_features=self.embedding_dim*4, out_features=self.embedding_dim) #NEW\n",
    "        )\n",
    "    def forward(self, x):\n",
    "        return self.ffn(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a235bc5-ec2e-4965-8212-291a81659b7f",
   "metadata": {},
   "source": [
    "Now let's train the network again to see how we end up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9a538b0-c165-4864-8b13-9a1c41243feb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.6328) Validation Loss: tensor(4.6313)\n",
      "Step: 200 Training Loss: tensor(2.5782) Validation Loss: tensor(2.5969)\n",
      "Step: 400 Training Loss: tensor(2.4491) Validation Loss: tensor(2.4365)\n",
      "Step: 600 Training Loss: tensor(2.3560) Validation Loss: tensor(2.3455)\n",
      "Step: 800 Training Loss: tensor(2.2816) Validation Loss: tensor(2.2922)\n",
      "Step: 1000 Training Loss: tensor(2.2414) Validation Loss: tensor(2.2609)\n",
      "Step: 1200 Training Loss: tensor(2.2245) Validation Loss: tensor(2.2473)\n",
      "Step: 1400 Training Loss: tensor(2.1878) Validation Loss: tensor(2.2126)\n",
      "Step: 1600 Training Loss: tensor(2.1557) Validation Loss: tensor(2.1949)\n",
      "Step: 1800 Training Loss: tensor(2.1444) Validation Loss: tensor(2.1952)\n",
      "Step: 2000 Training Loss: tensor(2.1448) Validation Loss: tensor(2.1569)\n",
      "Step: 2200 Training Loss: tensor(2.1297) Validation Loss: tensor(2.1741)\n",
      "Step: 2400 Training Loss: tensor(2.0952) Validation Loss: tensor(2.1558)\n",
      "Step: 2600 Training Loss: tensor(2.0832) Validation Loss: tensor(2.1392)\n",
      "Step: 2800 Training Loss: tensor(2.0740) Validation Loss: tensor(2.1216)\n",
      "Step: 3000 Training Loss: tensor(2.0602) Validation Loss: tensor(2.1131)\n",
      "Step: 3200 Training Loss: tensor(2.0669) Validation Loss: tensor(2.1428)\n",
      "Step: 3400 Training Loss: tensor(2.0427) Validation Loss: tensor(2.0881)\n",
      "Step: 3600 Training Loss: tensor(2.0371) Validation Loss: tensor(2.1069)\n",
      "Step: 3800 Training Loss: tensor(2.0253) Validation Loss: tensor(2.1075)\n",
      "Step: 4000 Training Loss: tensor(2.0300) Validation Loss: tensor(2.1037)\n",
      "Step: 4200 Training Loss: tensor(2.0191) Validation Loss: tensor(2.0958)\n",
      "Step: 4400 Training Loss: tensor(2.0207) Validation Loss: tensor(2.0896)\n",
      "Step: 4600 Training Loss: tensor(1.9983) Validation Loss: tensor(2.0888)\n",
      "Step: 4800 Training Loss: tensor(1.9998) Validation Loss: tensor(2.0826)\n",
      "Step: 4999 Training Loss: tensor(1.9828) Validation Loss: tensor(2.0681)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGxCAYAAACeKZf2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABSjUlEQVR4nO3deXwU9d0H8M/sZu8rCZALkgByCcgZEPACuQRLpWDFloJ49cECFilFgdpqPVCrlfJQ4dFavIEqam3xAAoJKFBAQBEQQQKJISEQkt3sbrLnPH/MZsmSEHLs7mw2n/frNa+dnZ2d/e6wNp/O/Ga+giiKIoiIiIjihELuAoiIiIjCieGGiIiI4grDDREREcUVhhsiIiKKKww3REREFFcYboiIiCiuMNwQERFRXGG4ISIiorjCcENERERxheGGiKLitddegyAI2Ldvn9ylEFGcY7ghIiKiuMJwQ0RERHGF4YaIYsbnn3+O0aNHw2QyQa/XY8SIEdi4cWPIOk6nEwsXLkSXLl2g1WqRnJyMnJwcrF27NrjOyZMnceeddyIjIwMajQapqakYPXo0Dh48GLKt9evXY/jw4TAYDDAajRg/fjwOHDgQsk5jt0VEsSNB7gKIiAAgLy8PY8eORb9+/fDqq69Co9HgpZdewqRJk7B27VpMmzYNALBgwQK8+eabePLJJzFw4EA4HA588803KCsrC25r4sSJ8Pl8eO6555CVlYXz589j586dqKioCK7z9NNP43e/+x3uvvtu/O53v4Pb7caf/vQn3HDDDdizZw969+7d6G0RUWwRRFEU5S6CiOLfa6+9hrvvvht79+5FTk5OndeHDx+OkydP4vvvv4fRaAQA+Hw+DBgwABUVFSgoKIAgCLjmmmvQrVs3fPDBB/V+TllZGdq3b4/ly5fj17/+db3rFBYWomvXrnjggQewYsWK4HK73Y7u3bvjxhtvxPr16xu1LSKKPTwtRUSyczgc+O9//4vbb789GGwAQKlUYsaMGfjhhx9w7NgxAMDQoUPxySef4JFHHkFubi6qqqpCtpWcnIyrrroKf/rTn/DnP/8ZBw4cgN/vD1nns88+g9frxcyZM+H1eoOTVqvFTTfdhNzc3EZvi4hiD8MNEcmuvLwcoigiPT29zmsZGRkAEDzttGLFCjz88MP48MMPMWrUKCQnJ2Py5Mk4fvw4AEAQBPznP//B+PHj8dxzz2HQoEHo0KEDHnzwQVRWVgIAzp49CwAYMmQIVCpVyLR+/XqcP3++0dsiotjDMTdEJLukpCQoFAoUFxfXee3MmTMAgPbt2wMADAYDHn/8cTz++OM4e/Zs8CjOpEmT8O233wIAsrOz8eqrrwIAvvvuO/zjH//AY489BrfbjdWrVwe39d577yE7O7vB2q60LSKKPRxzQ0RRcaUxNyNGjEB+fj5OnjwJnU4HAPD7/RgwYADKy8uDY27q89BDD2H58uVwOBzQ6/X1rjNw4ECoVCrs2bMHp06dQvfu3fHUU09h0aJFTf4utbdFRLGHR26IKKq2bt2KU6dO1Vm+bNkyjB07FqNGjcLChQuhVqvx0ksv4ZtvvsHatWuDwebaa6/Fj370I/Tr1w9JSUk4evQo3nzzTQwfPhx6vR5ff/015s6di5/+9Kfo3r071Go1tm7diq+//hqPPPIIAKBz58744x//iKVLl+LkyZO45ZZbkJSUhLNnz2LPnj3Bo0ON2RYRxSCRiCgK1qxZIwK47JSfny/u2LFDvPnmm0WDwSDqdDpx2LBh4r/+9a+Q7TzyyCNiTk6OmJSUJGo0GrFr167iQw89JJ4/f14URVE8e/asOGvWLLFXr16iwWAQjUaj2K9fP/HFF18UvV5vyLY+/PBDcdSoUaLZbBY1Go2YnZ0t3n777eKWLVuavC0iih08LUVERERxhVdLERERUVxhuCEiIqK4wnBDREREcYXhhoiIiOIKww0RERHFFYYbIiIiiitt7iZ+fr8fZ86cgclkuuzdTomIiCi2iKKIyspKZGRkQKFo+NhMmws3Z86cQWZmptxlEBERUTMUFhaiU6dODa7T5sKNyWQCIO0cs9ksczVERETUGDabDZmZmcG/4w1pc+Gm5lSU2WxmuCEiImplGjOkhAOKiYiIKK4w3BAREVFcYbghIiKiuNLmxtwQEVHLiKIIr9cLn88ndykUZ1QqFZRKZYu3w3BDRESN5na7UVxcDKfTKXcpFIcEQUCnTp1gNBpbtB2GGyIiahS/34/8/HwolUpkZGRArVbzZqgUNqIo4ty5c/jhhx/QvXv3Fh3BYbghIqJGcbvd8Pv9yMzMhF6vl7scikMdOnTAqVOn4PF4WhRuOKCYiIia5Eq3vidqrnAdCeQvlIiIiOIKww0RERHFFYYbIiKiJho5ciTmz5/f6PVPnToFQRBw8ODBiNVEFzHcEBFR3BIEocFp1qxZzdru+++/jyeeeKLR62dmZqK4uBh9+/Zt1uc1FkOUhFdLhUl5+QXkbnwbgqcKk+9eJHc5REQEoLi4ODi/fv16/P73v8exY8eCy3Q6Xcj6Ho8HKpXqittNTk5uUh1KpRJpaWlNeg81H4/chInPWYGfnPgdJp56Bn6fX+5yiIiiQhRFON3eqE+iKDaqvrS0tOBksVggCELweXV1NRITE/GPf/wDI0eOhFarxVtvvYWysjL87Gc/Q6dOnaDX63HNNddg7dq1Idu99LRU586d8fTTT+Oee+6ByWRCVlYWXn755eDrlx5Ryc3NhSAI+M9//oOcnBzo9XqMGDEiJHgBwJNPPomUlBSYTCbcd999eOSRRzBgwIBm/VsBgMvlwoMPPoiUlBRotVpcf/312Lt3b/D18vJyTJ8+HR06dIBOp0P37t2xZs0aANKtAObOnYv09HRotVp07twZy5Yta3YtkcQjN2FiTGwPAFALPtgcVpjNSTJXREQUeVUeH3r//rOof+6RP46HXh2eP2EPP/wwXnjhBaxZswYajQbV1dUYPHgwHn74YZjNZmzcuBEzZsxA165dce211152Oy+88AKeeOIJLFmyBO+99x4eeOAB3HjjjejVq9dl37N06VK88MIL6NChA2bPno177rkHX3zxBQDg7bffxlNPPYWXXnoJ1113HdatW4cXXngBXbp0afZ3XbRoETZs2IDXX38d2dnZeO655zB+/HicOHECycnJePTRR3HkyBF88sknaN++PU6cOIGqqioAwIoVK/DRRx/hH//4B7KyslBYWIjCwsJm1xJJDDdhotWb4BYToBa8sJefY7ghImol5s+fjylTpoQsW7hwYXB+3rx5+PTTT/Huu+82GG4mTpyIX/3qVwCkwPTiiy8iNze3wXDz1FNP4aabbgIAPPLII7j11ltRXV0NrVaL//3f/8W9996Lu+++GwDw+9//Hps2bYLdbm/W93Q4HFi1ahVee+01TJgwAQDwyiuvYPPmzXj11Vfx29/+FgUFBRg4cCBycnIASEekahQUFKB79+64/vrrIQgCsrOzm1VHNDDchIsgwCYY0R4VcFScB7J7yF0REVHE6VRKHPnjeFk+N1xq/pDX8Pl8eOaZZ7B+/XoUFRXB5XLB5XLBYDA0uJ1+/foF52tOf5WWljb6Penp6QCA0tJSZGVl4dixY8GwVGPo0KHYunVro77Xpb7//nt4PB5cd911wWUqlQpDhw7F0aNHAQAPPPAApk6div3792PcuHGYPHkyRowYAQCYNWsWxo4di549e+KWW27Bj370I4wbN65ZtUQaw00YORQmtPdXoMp2Xu5SiIiiQhCEsJ0eksuloeWFF17Aiy++iOXLl+Oaa66BwWDA/Pnz4Xa7G9zOpQORBUGA39/wGMza76m5O2/t91x6x97GjjWqT81769tmzbIJEybg9OnT2LhxI7Zs2YLRo0djzpw5eP755zFo0CDk5+fjk08+wZYtW3DHHXdgzJgxeO+995pdU6RwQHEYOZVmAIDHXiZzJURE1Fw7duzAbbfdhl/84hfo378/unbtiuPHj0e9jp49e2LPnj0hy/bt29fs7XXr1g1qtRqff/55cJnH48G+fftw9dVXB5d16NABs2bNwltvvYXly5eHDIw2m82YNm0aXnnlFaxfvx4bNmzAhQsXml1TpLTuuB1jXCoL4AE89tj7hyYiosbp1q0bNmzYgJ07dyIpKQl//vOfUVJSEhIAomHevHm4//77kZOTgxEjRmD9+vX4+uuv0bVr1yu+99KrrgCgd+/eeOCBB/Db3/4WycnJyMrKwnPPPQen04l7770XgDSuZ/DgwejTpw9cLhf+/e9/B7/3iy++iPT0dAwYMAAKhQLvvvsu0tLSkJiYGNbvHQ4MN2HkUVsAJyA6GW6IiFqrRx99FPn5+Rg/fjz0ej1++ctfYvLkybBarVGtY/r06Th58iQWLlyI6upq3HHHHZg1a1adozn1ufPOO+ssy8/PxzPPPAO/348ZM2agsrISOTk5+Oyzz5CUJF0Eo1arsXjxYpw6dQo6nQ433HAD1q1bBwAwGo149tlncfz4cSiVSgwZMgQff/xxTDZSFcSWnMBrhWw2GywWC6xWK8xmc1i3/d/Vs3FtyVrsTPsFRsz+a1i3TUQkt+rqauTn56NLly7QarVyl9MmjR07FmlpaXjzzTflLiUiGvqNNeXvN4/chJGolZKv0lUhbyFERNTqOZ1OrF69GuPHj4dSqcTatWuxZcsWbN68We7SYh7DTRgJeincqFzRPXRJRETxRxAEfPzxx3jyySfhcrnQs2dPbNiwAWPGjJG7tJjHcBNGKkPgnKWX4YaIiFpGp9Nhy5YtcpfRKsXeKKBWTG2SWjDovTaZKyEiImq7YibcLFu2DIIghDQiu1RNo7FLp2+//TZ6hTZAa2oHADD4K2WuhIiIqO2KidNSe/fuxcsvvxxyG+qGHDt2LGSkdIcOHSJVWpPoE6U6TGLz+n4QERFRy8l+5MZut2P69Ol45ZVXgtfZX0lKSkpIG3ulMnw9RlrCmCSFG73gQnWVQ+ZqiIiI2ibZw82cOXNw6623Nmn098CBA5Geno7Ro0dj27ZtDa7rcrlgs9lCpkgxmpLgE6X+HPZy9pciIiKSg6zhZt26ddi/fz+WLVvWqPXT09Px8ssvY8OGDXj//ffRs2dPjB49Gtu3b7/se5YtWwaLxRKcMjMzw1V+HQqlEjbBCACwW89F7HOIiIjo8mQLN4WFhfj1r3+Nt956q9F3uuzZsyfuv/9+DBo0CMOHD8dLL72EW2+9Fc8///xl37N48WJYrdbgVFhYGK6vUC+7YAIAOK08ckNEFC9GjhwZcsFL586dsXz58gbfIwgCPvzwwxZ/dri205bIFm6+/PJLlJaWYvDgwUhISEBCQgLy8vKwYsUKJCQkwOfzNWo7w4YNa7Bbq0ajgdlsDpkiyamUwo2rkp3BiYjkNmnSpMsOe9i1axcEQcD+/fubvN29e/fil7/8ZUvLC/HYY49hwIABdZYXFxdjwoQJYf2sS7322msx2QCzuWS7Wmr06NE4dOhQyLK7774bvXr1wsMPP9zoQcIHDhxAenp6JEpsluoEM+BlZ3Aiolhw7733YsqUKTh9+jSys7NDXvv73/+OAQMGYNCgQU3ebjSv0k1LS4vaZ8UL2Y7cmEwm9O3bN2QyGAxo164d+vbtC0A6pTRz5szge5YvX44PP/wQx48fx+HDh7F48WJs2LABc+fOletr1OFRWwAAfnYGJ6K2QBQBtyP6UyN7Pv/oRz9CSkoKXnvttZDlTqcT69evx7333ouysjL87Gc/Q6dOnaDX63HNNddg7dq1DW730tNSx48fx4033gitVovevXvX2//p4YcfRo8ePaDX69G1a1c8+uij8Hg8AKQjJ48//ji++uqr4D3camq+9LTUoUOHcPPNN0On06Fdu3b45S9/Cbv94i1IZs2ahcmTJ+P5559Heno62rVrhzlz5gQ/qzkKCgpw2223wWg0wmw244477sDZs2eDr3/11VcYNWoUTCYTzGYzBg8ejH379gEATp8+jUmTJiEpKQkGgwF9+vTBxx9/3OxaGiMm7nNzOcXFxSgoKAg+d7vdWLhwIYqKiqDT6dCnTx9s3LgREydOlLHKUL5AuBGd5TJXQkQUBR4n8HRG9D93yRlAbbjiagkJCZg5cyZee+01/P73v4cgSFe0vvvuu3C73Zg+fTqcTicGDx6Mhx9+GGazGRs3bsSMGTPQtWtXXHvttVf8DL/fjylTpqB9+/bYvXs3bDZbvTekNZlMeO2115CRkYFDhw7h/vvvh8lkwqJFizBt2jR88803+PTTT4MtFywWS51tOJ1O3HLLLRg2bBj27t2L0tJS3HfffZg7d25IgNu2bRvS09Oxbds2nDhxAtOmTcOAAQNw//33X/H7XEoURUyePBkGgwF5eXnwer341a9+hWnTpiE3NxcAMH36dAwcOBCrVq2CUqnEwYMHoVKpAEhXRbvdbmzfvh0GgwFHjhyB0Whsch1NEVPhpmYn1bg0aS9atAiLFi2KXkHN4NdJ9+pRVFfIWwgREQEA7rnnHvzpT39Cbm4uRo0aBUA6JTVlyhQkJSUhKSkJCxcuDK4/b948fPrpp3j33XcbFW62bNmCo0eP4tSpU+jUqRMA4Omnn64zTuZ3v/tdcL5z5874zW9+g/Xr12PRokXQ6XQwGo1ISEho8DTU22+/jaqqKrzxxhswGKRwt3LlSkyaNAnPPvssUlNTAQBJSUlYuXIllEolevXqhVtvvRX/+c9/mhVutmzZgq+//hr5+fnBK47ffPNN9OnTB3v37sWQIUNQUFCA3/72t+jVqxcAoHv37sH3FxQUYOrUqbjmmmsAAF27dm1yDU0VU+EmHgiBcJPgrpC3ECKiaFDppaMocnxuI/Xq1QsjRozA3//+d4waNQrff/89duzYgU2bNgEAfD4fnnnmGaxfvx5FRUVwuVxwuVzB8HAlR48eRVZWVjDYAMDw4cPrrPfee+9h+fLlOHHiBOx2O7xeb5Mvcjl69Cj69+8fUtt1110Hv9+PY8eOBcNNnz59Qsaupqen1xnn2pTPzMzMDLmVSu/evZGYmIijR49iyJAhWLBgAe677z68+eabGDNmDH7605/iqquuAgA8+OCDeOCBB7Bp0yaMGTMGU6dObXRHguaS/SZ+8UapTwYAqN3sDE5EbYAgSKeHoj0FTi811r333osNGzbAZrNhzZo1yM7OxujRowEAL7zwAl588UUsWrQIW7duxcGDBzF+/Hi43e5GbVusZ/yPcEl9u3fvxp133okJEybg3//+Nw4cOIClS5c2+jNqf9al267vM2tOCdV+ze/3N+mzrvSZtZc/9thjOHz4MG699VZs3boVvXv3xgcffAAAuO+++3Dy5EnMmDEDhw4dQk5ODv73f/+3WbU0FsNNmKlMUrjR+tgZnIgoVtxxxx1QKpV455138Prrr+Puu+8O/mHesWMHbrvtNvziF79A//790bVr1wZvMXKp3r17o6CgAGfOXDyCtWvXrpB1vvjiC2RnZ2Pp0qXIyclB9+7dcfr06ZB11Gr1FW+D0rt3bxw8eBAOx8UWP1988QUUCgV69OjR6Jqboub71b5P3JEjR2C1WnH11VcHl/Xo0QMPPfQQNm3ahClTpmDNmjXB1zIzMzF79my8//77+M1vfoNXXnklIrXWYLgJM42pPQDA4GNncCKiWGE0GjFt2jQsWbIEZ86cwaxZs4KvdevWDZs3b8bOnTtx9OhR/M///A9KSkoave0xY8agZ8+emDlzJr766ivs2LEDS5cuDVmnW7duKCgowLp16/D9999jxYoVwSMbNTp37oz8/HwcPHgQ58+fh8vlqvNZ06dPh1arxV133YVvvvkG27Ztw7x58zBjxozgKanm8vl8OHjwYMh05MgRjBkzBv369cP06dOxf/9+7NmzBzNnzsRNN92EnJwcVFVVYe7cucjNzcXp06fxxRdfYO/evcHgM3/+fHz22WfIz8/H/v37sXXr1pBQFAkMN2Gmt0jhxigy3BARxZJ7770X5eXlGDNmDLKysoLLH330UQwaNAjjx4/HyJEjkZaWhsmTJzd6uwqFAh988AFcLheGDh2K++67D0899VTIOrfddhseeughzJ07FwMGDMDOnTvx6KOPhqwzdepU3HLLLRg1ahQ6dOhQ7+Xoer0en332GS5cuIAhQ4bg9ttvx+jRo7Fy5cqm7Yx62O12DBw4MGSaOHFi8FL0pKQk3HjjjRgzZgy6du2K9evXAwCUSiXKysowc+ZM9OjRA3fccQcmTJiAxx9/HIAUmubMmYOrr74at9xyC3r27ImXXnqpxfU2RBDrO1kYx2w2GywWC6xWa0TuVnz+bBHar+oNAPD9rgzKBI7ZJqL4UF1djfz8fHTp0qXRbXOImqKh31hT/n7zyE2YmZMu3rXSzv5SREREUcdwE2ZqtRqVog4AYK9gZ3AiIqJoY7iJALsg3XnRWcEjN0RERNHGcBMBDqV0LrCancGJiIiijuEmAqqUJgCA285wQ0Txp41dh0JRFK7fFsNNBLhVUrMzH8MNEcWRmrveOp1OmSuheFVzx+barSOag9cpR4BHkwg42BmciOKLUqlEYmIiSktLAUj3XLlcKwCipvL7/Th37hz0ej0SWngbFYabCPBrE6UZdgYnojhT07G6JuAQhZNCoUBWVlaLQzPDTSQEwk2Cq0LWMoiIwk0QBKSnpyMlJQUej0fucijOqNVqKBQtHzHDcBMBCn07AEACO4MTUZxSKpUtHhdBFCkcUBwBKmOgM7iXncGJiIiijeEmAjQm6ciNzsdwQ0REFG0MNxGgMUvhxuhnZ3AiIqJoY7iJAENiewCAWbRD9PtlroaIiKhtYbiJAHNSCgAgQfCj2sFBxURERNHEcBMBBoMR1aJ0J097OTuDExERRRPDTQQIggCbIPWXslsZboiIiKKJ4SZC7Aop3FRZz8tcCRERUdvCcBMhwc7glWyeSUREFE0MNxFSnSB1Bvc62DyTiIgomhhuIsSjlsKN33lB5kqIiIjaFoabCPFpEqWZKh65ISIiiiaGm0jRJQIAFOwMTkREFFUMNxEi6JIAAAku3sSPiIgomhhuIiQh0Blc42G4ISIiiiaGmwhRG9kZnIiISA4MNxGiNknNMw3sDE5ERBRVDDcRUtMZ3CQy3BAREUUTw02EmBI7AAC08MBb7ZC5GiIioraD4SZCzJYkeEVp91ZWsL8UERFRtDDcREhCghJWGAEADjbPJCIiihqGmwiq6QzutJ6TuRIiIqK2g+EmgpwK6ciNi53BiYiIoobhJoKqVYHO4HaGGyIiomhhuIkgt8oMAPA52DyTiIgoWhhuIsirTgQAiE6GGyIiomhhuIkgMdA8U6hmuCEiIooWhptIqukM7q6Qtw4iIqI2hOEmgpR6Kdyo3ewMTkREFC0MNxGkCnQG13rZGZyIiChaGG4iSBPoDK7zsXkmERFRtDDcRJDOUtMZ3C5zJURERG0Hw00EGROl01JGOCF63TJXQ0RE1DYw3ESQOal9cN5ZeUHGSoiIiNoOhpsI0mk0sIl6AIC9nM0ziYiIooHhJoIEQYBNkJpnOq3nZa6GiIiobWC4iTCHQuovVW1juCEiIoqGmAk3y5YtgyAImD9/foPr5eXlYfDgwdBqtejatStWr14dnQKbqTrBBABwV7IzOBERUTTERLjZu3cvXn75ZfTr16/B9fLz8zFx4kTccMMNOHDgAJYsWYIHH3wQGzZsiFKlTedKsAAAfA4OKCYiIooG2cON3W7H9OnT8corryApKanBdVevXo2srCwsX74cV199Ne677z7cc889eP7556NUbdN5NFK48Vcx3BAREUWD7OFmzpw5uPXWWzFmzJgrrrtr1y6MGzcuZNn48eOxb98+eDyeet/jcrlgs9lCpmjyawKBrYr9pYiIiKJB1nCzbt067N+/H8uWLWvU+iUlJUhNTQ1ZlpqaCq/Xi/Pn6x+wu2zZMlgsluCUmZnZ4rqbRJcIAFC6yqP7uURERG2UbOGmsLAQv/71r/HWW29Bq9U2+n2CIIQ8F0Wx3uU1Fi9eDKvVGpwKCwubX3QzKAKdwVXsDE5ERBQVCXJ98JdffonS0lIMHjw4uMzn82H79u1YuXIlXC4XlEplyHvS0tJQUlISsqy0tBQJCQlo165dvZ+j0Wig0WjC/wUaKcGYLNXhYWdwIiKiaJAt3IwePRqHDh0KWXb33XejV69eePjhh+sEGwAYPnw4/vWvf4Us27RpE3JycqBSqSJab3OpjVLo0vsYboiIiKJBtnBjMpnQt2/fkGUGgwHt2rULLl+8eDGKiorwxhtvAABmz56NlStXYsGCBbj//vuxa9cuvPrqq1i7dm3U628snaUDAMDgr5S5EiIiorZB9qulGlJcXIyCgoLg8y5duuDjjz9Gbm4uBgwYgCeeeAIrVqzA1KlTZayyYXqL1DzTJNoBv1/maoiIiOKfINaMyG0jbDYbLBYLrFYrzGZzxD+v3GpD0ovSFVqe356CytDwvXyIiIiorqb8/Y7pIzfxwGwywSlKA5ory0tlroaIiCj+MdxEmFLBzuBERETRxHATBXZBap7ptLJ5JhERUaQx3ESBUxnoDG7nkRsiIqJIY7iJApdKGvjkqWTzTCIiokhjuIkCtyrQGdzJcENERBRpDDdR4NMkSjNVbJ5JREQUaQw3USAGOoMrqitkrYOIiKgtYLiJAoVOap6ZwM7gREREEcdwEwVKQ01ncIYbIiKiSGO4iQKVSQo3Wi87gxMREUUaw00UaE1S80x2BiciIoo8hpso0AU6gxtFO9C2+pQSERFFHcNNFJgSUwAAGnggepwyV0NERBTfGG6iwGKxwCMqAQCOCrZgICIiiiSGmyjQqhNghdQZ3F5xTuZqiIiI4hvDTZRUClK4qbLyyA0REVEkMdxESU1n8OrKMpkrISIiim8MN1FSnSA1z/Qw3BAREUUUw02UuFRmAICPncGJiIgiiuEmSrzqRACAyHBDREQUUQw3UeLXJgIABHYGJyIiiiiGmygR9EkAgARXhbyFEBERxTmGmyhR6KXmmSoPm2cSERFFEsNNlCQYpXCjYbghIiKKKIabKNEa2wEA9D6GGyIiokhiuImSi53BK2WuhIiIKL4x3ESJIVEKNwZUA163zNUQERHFL4abKDFb2sMvCgAAl513KSYiIooUhpsoMek1sEEPgJ3BiYiIIonhJkoUCgE2dgYnIiKKOIabKHIopP5SVTaGGyIiokhhuImiKqUJAOBmZ3AiIqKIYbiJIpfKAgDwOMplroSIiCh+MdxEkUcthRvRyXBDREQUKQw3UeTXSOFGqGK4ISIiihSGmygSdVJncAU7gxMREUUMw00UKfRSuFEx3BAREUUMw00UKQ1S80yNl80ziYiIIoXhJorUJinc6BhuiIiIIobhJop05kDzTD87gxMREUUKw00U6S1SuDHCAfh9MldDREQUnxhuosiUKIUbBUT4q6wyV0NERBSfGG6iyGw0wC5qAQAONs8kIiKKCIabKNKqlLBB6gzuqGC4ISIiigSGmyizK6Rww87gREREkcFwE2VOpRkA4KpkuCEiIooEhpsoq06Qwo3HXiZzJURERPGJ4SbKPCqpeabPweaZREREkcBwE2XeQGdwsDM4ERFRRDDcRFmwM3g1ww0REVEkMNxEWyDcJLh4Ez8iIqJIkDXcrFq1Cv369YPZbIbZbMbw4cPxySefXHb93NxcCIJQZ/r222+jWHXLKPXJAAC1h+GGiIgoEhLk/PBOnTrhmWeeQbdu3QAAr7/+Om677TYcOHAAffr0uez7jh07BrPZHHzeoUOHiNcaLjWdwbU+Ns8kIiKKBFnDzaRJk0KeP/XUU1i1ahV2797dYLhJSUlBYmJihKuLDI1J6i+l99lkroSIiCg+xcyYG5/Ph3Xr1sHhcGD48OENrjtw4ECkp6dj9OjR2LZtW4Prulwu2Gy2kElOBot05MYk2gFRlLUWIiKieCR7uDl06BCMRiM0Gg1mz56NDz74AL1796533fT0dLz88svYsGED3n//ffTs2ROjR4/G9u3bL7v9ZcuWwWKxBKfMzMxIfZVGMSRKp9BU8AJuh6y1EBERxSNBFOU9fOB2u1FQUICKigps2LABf/vb35CXl3fZgHOpSZMmQRAEfPTRR/W+7nK54HK5gs9tNhsyMzNhtVpDxu1Ei63KDc0z6dAIXlTP/Rra9tlRr4GIiKi1sdlssFgsjfr7LfuRG7VajW7duiEnJwfLli1D//798Ze//KXR7x82bBiOHz9+2dc1Gk3waqyaSU4mrQrWYGfwc7LWQkREFI9kDzeXEkUx5EjLlRw4cADp6ekRrCi8BEFApSCFG6eVzTOJiIjCTdarpZYsWYIJEyYgMzMTlZWVWLduHXJzc/Hpp58CABYvXoyioiK88cYbAIDly5ejc+fO6NOnD9xuN9566y1s2LABGzZskPNrNJlDYQb8QLWN4YaIiCjcZA03Z8+exYwZM1BcXAyLxYJ+/frh008/xdixYwEAxcXFKCgoCK7vdruxcOFCFBUVQafToU+fPti4cSMmTpwo11dolqoEM+AG3OwMTkREFHayDyiOtqYMSIqUHX/6KW5wbMLXveaj352Py1IDERFRa9KqBhS3RV51ojTjZPNMIiKicGO4kYFfmyjNVFfIWQYREVFcYriRgy4RAKB0VchaBhERUTxiuJGBItAZXOVmZ3AiIqJwY7iRQYJBCjdaL5tnEhERhRvDjQw0Zql5po6dwYmIiMKuWeGmsLAQP/zwQ/D5nj17MH/+fLz88sthKyye6SztAQBGf6XMlRAREcWfZoWbn//859i2bRsAoKSkBGPHjsWePXuwZMkS/PGPfwxrgfHIEAg3OrgAT7XM1RAREcWXZoWbb775BkOHDgUA/OMf/0Dfvn2xc+dOvPPOO3jttdfCWV9cMlnawScKAAAf73VDREQUVs0KNx6PBxqNBgCwZcsW/PjHPwYA9OrVC8XFxeGrLk5Z9BpYYQAAOKzsDE5ERBROzQo3ffr0werVq7Fjxw5s3rwZt9xyCwDgzJkzaNeuXVgLjEfqBAVsMAEAHBVsnklERBROzQo3zz77LP7v//4PI0eOxM9+9jP0798fAPDRRx8FT1dRw+wKIwCg2sbmmUREROHUrK7gI0eOxPnz52Gz2ZCUlBRc/stf/hJ6vT5sxcWzKqUZ8AIudgYnIiIKq2YduamqqoLL5QoGm9OnT2P58uU4duwYUlJSwlpgvHKppI6mXoYbIiKisGpWuLntttvwxhtvAAAqKipw7bXX4oUXXsDkyZOxatWqsBYYrzxqCwDAz6uliIiIwqpZ4Wb//v244YYbAADvvfceUlNTcfr0abzxxhtYsWJFWAuMVz5NojRTxXBDREQUTs0KN06nEyaTdLXPpk2bMGXKFCgUCgwbNgynT58Oa4HxStRJp/SU1RXyFkJERBRnmhVuunXrhg8//BCFhYX47LPPMG7cOABAaWkpzGZzWAuMV4pAuElwV8hbCBERUZxpVrj5/e9/j4ULF6Jz584YOnQohg8fDkA6ijNw4MCwFhivlAbpfkAaD5tnEhERhVOzLgW//fbbcf3116O4uDh4jxsAGD16NH7yk5+Erbh4pjYlA2BncCIionBrVrgBgLS0NKSlpeGHH36AIAjo2LEjb+DXBFqT1DzT4GNncCIionBq1mkpv9+PP/7xj7BYLMjOzkZWVhYSExPxxBNPwO/3h7vGuKRPlMKNEQ7A75O5GiIiovjRrCM3S5cuxauvvopnnnkG1113HURRxBdffIHHHnsM1dXVeOqpp8JdZ9wxWi724BKrKiAY2JOLiIgoHJoVbl5//XX87W9/C3YDB4D+/fujY8eO+NWvfsVw0wiJJgMqRR1MQhVclWXQMtwQERGFRbNOS124cAG9evWqs7xXr164cOFCi4tqCwxqJawwAADsFedkroaIiCh+NCvc9O/fHytXrqyzfOXKlejXr1+Li2oLBEGAXZBuhOi0npe5GiIiovjRrNNSzz33HG699VZs2bIFw4cPhyAI2LlzJwoLC/Hxxx+Hu8a45VCaAB/gqmS4ISIiCpdmHbm56aab8N133+EnP/kJKioqcOHCBUyZMgWHDx/GmjVrwl1j3KpOkO7m7LHzVB4REVG4NPs+NxkZGXUGDn/11Vd4/fXX8fe//73FhbUFblUi4AL8DoYbIiKicGnWkRsKD6/GAgAQ2RmciIgobBhuZCRqpeaZimqGGyIionBhuJFToDO40sX+UkREROHSpDE3U6ZMafD1ioqKltTS5iQYpHCj9lhlroSIiCh+NCncWCyWK74+c+bMFhXUliQYpc7gWi/DDRERUbg0KdzwMu/wqukMrmdncCIiorDhmBsZ6SyBzuBiJSCKMldDREQUHxhuZGRMlMJNAvyAi0dviIiIwoHhRkZmkxnVogoA4OWN/IiIiMKC4UZGFp0KFTACABxWdgYnIiIKB4YbGSUoFagMhBtnBcMNERFRODDcyMyuMAEAqmxlMldCREQUHxhuZFYV7AzOcENERBQODDcyc6ukcON1sL8UERFRODDcyMyrTgQAiE5eLUVERBQODDcy82kTAQBCdYWsdRAREcULhhu5BTqDK1wV8tZBREQUJxhuZKbQBzqDu9k8k4iIKBwYbmSWYJA6g2s8DDdEREThwHAjM42pHQBAx87gREREYcFwIzOtuQMAwOhnuCEiIgoHhhuZGQKdwTVwA54qmashIiJq/RhuZGYyJ8ErSv8MvNcNERFRyzHcyCzRoIYVBgBAVSXDDRERUUvJGm5WrVqFfv36wWw2w2w2Y/jw4fjkk08afE9eXh4GDx4MrVaLrl27YvXq1VGqNjJ0KiWsgc7gDnYGJyIiajFZw02nTp3wzDPPYN++fdi3bx9uvvlm3HbbbTh8+HC96+fn52PixIm44YYbcODAASxZsgQPPvggNmzYEOXKw0cQhFqdwc/LXA0REVHrlyDnh0+aNCnk+VNPPYVVq1Zh9+7d6NOnT531V69ejaysLCxfvhwAcPXVV2Pfvn14/vnnMXXq1GiUHBFVSjPgBdyV7AxORETUUjEz5sbn82HdunVwOBwYPnx4vevs2rUL48aNC1k2fvx47Nu3Dx6Pp973uFwu2Gy2kCnWVCcEOoPbGW6IiIhaSvZwc+jQIRiNRmg0GsyePRsffPABevfuXe+6JSUlSE1NDVmWmpoKr9eL8+frP6WzbNkyWCyW4JSZmRn279BSHrUFAOBzlstcCRERUesne7jp2bMnDh48iN27d+OBBx7AXXfdhSNHjlx2fUEQQp6Loljv8hqLFy+G1WoNToWFheErPky8mkRphuGGiIioxWQdcwMAarUa3bp1AwDk5ORg7969+Mtf/oL/+7//q7NuWloaSkpKQpaVlpYiISEB7dq1q3f7Go0GGo0m/IWHkahlZ3AiIqJwkf3IzaVEUYTL5ar3teHDh2Pz5s0hyzZt2oScnByoVKpolBcRNZ3BVe4KeQshIiKKA7KGmyVLlmDHjh04deoUDh06hKVLlyI3NxfTp08HIJ1SmjlzZnD92bNn4/Tp01iwYAGOHj2Kv//973j11VexcOFCub5CWCgDncHV7AxORETUYrKeljp79ixmzJiB4uJiWCwW9OvXD59++inGjh0LACguLkZBQUFw/S5duuDjjz/GQw89hL/+9a/IyMjAihUrWvVl4ACgNkrhRutl80wiIqKWEsSaEblthM1mg8VigdVqhdlslrscAMB/9/4X124cBwf0MDxWLHc5REREMacpf79jbsxNW6RP7AAAMMAJ+Oq/Xw8RERE1DsNNDDBZal3pVc1xN0RERC3BcBMDLAYdbKIeAODhXYqJiIhahOEmBph1KlSIBgCAw8rO4ERERC3BcBMDlAoBlYHO4M4KdgYnIiJqCYabGOEIhBtXJcMNERFRSzDcxIjqBKl5psd+QeZKiIiIWjeGmxjhVknX7PscDDdEREQtwXATI3wa6ciNv6pC3kKIiIhaOYabGOHTJAIAFNXl8hZCRETUyjHcxAgh0Bk8wcWb+BEREbUEw02MUOql5pkqd4W8hRAREbVyDDcxIsEotWDQem0yV0JERNS6MdzECLVJCjd6X6XMlRAREbVuDDcxQm9uDwAwiHbA75e5GiIiotaL4SZGGBOlcKOEH3Dx1BQREVFzMdzECLPJBKeoAQCIVbwcnIiIqLkYbmJEol6FCkidwZ3sDE5ERNRsDDcxQqtSwgYjAMDBzuBERETNxnATQ2o6g1dXlslcCRERUevFcBNDqpRSuGFncCIiouZjuIkhLpXUPNPLcENERNRsDDcxxKMOdAZ3MtwQERE1F8NNDPEHOoODl4ITERE1G8NNDBF1UmdwpatC3kKIiIhaMYabGKLQS+FG5bbKXAkREVHrxXATQ2o6g2s8DDdERETNxXATQ1SBcKPzsbcUERFRczHcxBBdTWdwfyUgijJXQ0RE1Dox3MQQg0U6cqOGF/BUyVwNERFR68RwE0PM5iR4RKX0hJeDExERNQvDTQyx6NXBzuBuO/tLERERNQfDTQwxaRNgFQOdwa3nZK6GiIiodWK4iSEKhQB7oDN4lfW8zNUQERG1Tgw3McYZ6AzuquRpKSIiouZguIkx1QnsDE5ERNQSDDcxxh3oDO5jZ3AiIqJmYbiJMT51ojTj5KXgREREzcFwE2P82kQAgKK6QtY6iIiIWiuGmxij0CcCABLYGZyIiKhZGG5ijNKQDABQszM4ERFRszDcxJiazuBaLzuDExERNQfDTYzRmGp1BiciIqImY7iJMfrEFACATqwCvtskczVEREStD8NNjDEltsNa7yjpybt3AUVfylsQERFRK8NwE2MsOhUe9d6N7f5rAI8TePsO4MJJucsiIiJqNRhuYkx7owbd0pLwgHs+vkUXwHkeeGsq4GAjTSIiosZguIkxCoWAN+4diuz0VMyoXogisYN05OadaYDbKXd5REREMY/hJgalmLRY/z/D0K1rN8x0L0K5aASK9gHv3QP4vHKXR0REFNMYbmKUSavCmruHoEefwbjP/RtUiyrgu0+AjxcCoih3eURERDGL4SaGaVVKrPz5IPQaOha/9syFXxSAL9dA3P683KURERHFLIabGKdUCHhycl9cffPP8Zh3JgBA2PYk/AfelrkyIiKi2CRruFm2bBmGDBkCk8mElJQUTJ48GceOHWvwPbm5uRAEoc707bffRqnq6BMEAfPH9ECPSb/Bau8kAID4z3lwH9ssc2VERESxR9Zwk5eXhzlz5mD37t3YvHkzvF4vxo0bB4fDccX3Hjt2DMXFxcGpe/fuUahYXr8Ylo3sO57FP/3XQwkffOtmwH5qn9xlERERxRRBFGNndOq5c+eQkpKCvLw83HjjjfWuk5ubi1GjRqG8vByJiYlN/gybzQaLxQKr1Qqz2dzCiuWx+7ti4O3bMUz4BheEJPjv2YT2mT3kLouIiChimvL3O6bG3FitVgBAcnLyFdcdOHAg0tPTMXr0aGzbtu2y67lcLthstpCptRvWIx2Wu9fhO2QjWSyH4++TcfqHH+Qui4iIKCbETLgRRRELFizA9ddfj759+152vfT0dLz88svYsGED3n//ffTs2ROjR4/G9u3b611/2bJlsFgswSkzMzNSXyGqru6cCcPdH+Cs0B7ZYhHK/zYV35w6K3dZREREsouZ01Jz5szBxo0b8fnnn6NTp05Neu+kSZMgCAI++uijOq+5XC64XK7gc5vNhszMzFZ9Wqq2C/lfQf3GBBhFBzaLQ6H7+Vu4vmeq3GURERGFVas7LTVv3jx89NFH2LZtW5ODDQAMGzYMx48fr/c1jUYDs9kcMsWT5C79ofjZO/BAhbHCHpx8ax7+dbBI7rKIiIhkI2u4EUURc+fOxfvvv4+tW7eiS5cuzdrOgQMHkJ6eHubqWg99j5HAlNUAgJnKz/DNe0/gtS/y5S2KiIhIJglyfvicOXPwzjvv4J///CdMJhNKSkoAABaLBTqdDgCwePFiFBUV4Y033gAALF++HJ07d0afPn3gdrvx1ltvYcOGDdiwYYNs3yMWqPrdDr+9BIpNS7E4YS0e3JiM8/a78JtxPSAIgtzlERERRY2s4WbVqlUAgJEjR4YsX7NmDWbNmgUAKC4uRkFBQfA1t9uNhQsXoqioCDqdDn369MHGjRsxceLEaJUdsxQj5kK0FUHY/RKeV63GXXmJ+K1tPB7/cR8YNLL+UxMREUVNzAwojpZ4uM9Ng/x+YMM9wOEPUCnq8FP3H2BP7Ilnp/bDdd3ay10dERFRs7S6AcUURgoFMHk1kH0dTEIV1mqeRj/rNkz/2248suFr2Ko9cldIREQUUQw38UilBe58G0jrhyTY8JJ6BV5RvYDcvQcx9s95+M9R3g+HiIjiF8NNvNIlAfdtAW56BFCoMFa5H//RLsJYx79x3+t7MH/dAVxwuOWukoiIKOwYbuJZggYYtRiYvQPoNBQGVOFJ1Rq8q/4jvvlqD8b+OQ///voM2tiwKyIiinMMN21BytXAPZ8BE58H1EbkKL7DJ5ol+EX1Wix4Zw9mv/UlSm3VcldJREQUFgw3bYVCAQy9H5jzX6DHLVDBi4dUG/CxZgnOHdmBMX/Ow3tf/sCjOERE1OrxUvC2SBSBwx8AnywCHOfgh4A3vWPwJ+80DO6RjaenXIOOiTq5qyQiIgripeDUMEEA+k4B5uwBBv4CCoi4K2EztmgWQXXiU4z7cx7e3H0afn+byr1ERBQneOSGgJO5wL/mA+VSP6p/+67F45670LVLVzw7tR86tzfIWh4RERGP3FDTdB0JPLATuO7XEAUlfqT8L7Zofousgvdxy1/y8Mr2k/DxKA4REbUSPHJDoYq/Aj6aJz0C2OnrjSXee2Hu2AuP3NILI9jCgYiIZNCUv98MN1SXzwv8dxXErU9B8FahWlRhtW8S3vGORvdu3bBwXE8MzEqSu0oiImpDGG4awHDTBBfygX/Pl8bkAPCJArb7++E9303w95iAX9/SF73SuA+JiCjyGG4awHDTRKIIfLMB2PMKULg7uNgq6vGRfwTOdZ2KKT/6MTp3MMpYJBERxTuGmwYw3LTA+RPAV2vh2f82VI7i4OITYkccT/8xBk36H6R27CJjgUREFK8YbhrAcBMGfh+Qvx0Vu16H/vuNUItSA06fKCDfMgypN94DU/8fS93JiYiIwoDhpgEMN2FWbUX+9rfh2vsmenmOXFysNEHR73aoB88AOg6SbhxIRETUTAw3DWC4iQxRFLH3y704+Z+/4UbnZmQIF4Kv+dv1gGLgdKDfNMCcLmOVRETUWjHcNIDhJrJEUcSnh4qw9ZP3MMK+CRMUe6AVPNJrggJC9nVA1nCg0xCgUw6gT5a5YiIiag0YbhrAcBMdPr+IDw4U4ZXNBzCgMg+3K/MwRPFd3RXbdQsEncCU0htQJkS/YCIiimkMNw1guIkut9eP9XsLsGLrCRjsp3Cd4jAGKY7jWtVJdPIX1X2DSg9kDAIyawUeY0r0CyciopjCcNMAhht5VLl9eHP3Kfz762J8/YMVAJCISgxQnMAw1UncqD+Fbp5jUHvtdd+cmH0x6GQOAVKvARLUUf4GREQkJ4abBjDcyO+83YUdx88h79g5bD9+Hhcc0qXkAvzoJpzBhMRC3Gw8he7ub6G3noCAS36iSg2Q0gswpUtHdYxpgCkVMKZK88YUaZ6XohMRxQ2GmwYw3MQWn1/EN0VW5B47h7zvSnGwsAK1G5CnqF24M6MUNxsL0NP7LXRn9wNV5Y3buDZRCjnB4JMKmNIuzhtTAUsnQMO7KxMRxTqGmwYw3MS2cocbn584Hwg753De7gp5vWs7PaZ0rsJwSznSlVYk+SugrT4HwX4WqD353I37QEEBpPUDsq8DOgeu5OIVXEREMYfhpgEMN62H3y/iaIktGHS+PF0On7/uz1WvViIjUYeOiTrp0aJBtsGNbLUdaUorksVyJDhKLwafyhLAXgrYS4Bqa90PTukthZ3sEdKjKTUK35aIiBrCcNMAhpvWy1btwc7AUZ0jxTacqajCefuVj9AIApBq0qJjkhR+MhK16BgIQ1dpbciuPAihYCdw6gvg/LG6G2jX7WLQyR4BJGZF4NsREVFDGG4awHATX6o9PhRbq3GmogpF5VUoqqjCmYoqnLFKz89UVMPt8ze4jQ4mDUZc1Q4jrmqH69OBjraDwOkvpKnkG+DSAc2WrEDYGQF0vh5I7sr2EkREEcZw0wCGm7bF7xdR5nBLgadCCj9FteaPn7XD5Q0NP52SdBhxVTtc1609RmQkoEP5gUDY2QmcOQCIvtAPMaZKl6kndZYuW0/MCkyZgMYUvS9LRBTHGG4awHBDtVV7fDhQUIFd35/Hzu/LcLCwAt5LxvV0SzEGj+wM66RBYtlBKeic+gIo2tfw4GVdcq2wU8/E8ENE1CgMNw1guKGG2F1e7D11Abu+L8PO78/j8Bkbav8XIghAnwwzRlzVHsOvaoehnfQwnDsonb6yFgIVp4GKAmlqzCXruqRaYSc79G7MoojgKbHLzqP+5WqDNEYovT+gUDZzbxARxQ6GmwYw3FBTVDjd2H2yDDu/l6YTpaF3UE5QCBiQmYhhXdthcOckDMpMgkWvkl6stgUCT0GtqSb8FAJVF+r5xDDTJQFdbgKuGgV0HQUkZUf+M4mIIoDhpgEMN9QSpbbqQNA5jy9OlKGooqrOOt1SjBiclYTB2UkYlJ2Iru2NUCjqGXDsqpRCTu3g4zgfGJwcWD84UFkILgp97TLzlSXAqc8Bly30M5O7SiHnqlFA5xsAXWJzdwURUVQx3DSA4YbCqfCCE1+cOI89py5g/+lynCpz1lnHolNhUFaiFHayktA/MxEGTRQ6n/u8QNGXwMltwPfbgB/2hg6GFhRAx8EXw06nIYBS1bLPqzwjBTZrYeAxENyqrYC5o3TqLSm71mOWdAqNiOgKGG4awHBDkXTe7sKBggp8eboc+wvK8VVhRZ2rsRQCcHW6GYNqju5kJSEzWQch0peTV9ukozk1YafseOjraqN0aXtN2GnfI/QSd08VYP1BCivB8FLr0Xam7pVkjWHoEBp6ErMuzlsy2SSViAAw3DSI4YaiyePz42ixDV+eLseXp8txoKCi3lNZ7Y0aDM5OxDUdLWhv1CDJoEY7gxpJBjWS9WpYdKr6T221hPUHKeSc3AaczAWcZaGvmzKkAcn2s1J4cZy78jYVKqlfV2KmdD+gxEwpoOgSAWuRdOqt/FTgsQBw1XOH6NoEhVRH7aM9lk6BKVM6GsQGqURtAsNNAxhuSG4l1mrsLygPHt35psgKj6/h/wwVApCkD4SdQOBJNkqPlwahZKMaJu3F0141F1GJgSupRPHibQlFUVoq+v1IKD0M1elcqE/nQVX0Xwg+V506oDZKoaImtAQfs6RHYyqgUDR+Z1RVBILO6XoeCwBv3SBYh6FDaOCpmTcHHg0dmlYTEcUkhpsGMNxQrKn2+PBNkRX7C8rx3Vk7yh1uXHC6ccEhTZXV3qjXpIEbQxTH0EUoxjkko9rQEX5zJxgT2yPVokO6RYtUsxZpZi3SAvNaVZgvORdF6WhRMPCckh6tRdJRJ+sPgMdx5e0o1dIRnkvDT80l+JZOQIImvLVHi8sOFH8FnNkvja+6cBLIvh4YNANIuVru6ojCiuGmAQw31Nq4vX5UOAOBxy49ljvcKHO4A0HIgwsOFy44PNJzh/uKLSfqUzO8RgAgCAIEoM4NDRuSpFdJgcdyMfSkmbVItWjRKVGHqzpc5qqx5hJF6V5Ctlphx1pYa/4HoLIYEBuxL0zpF49A1dxdOjHr4qk1lS58dTeXzwOcPSyFmDP7gaL9wLlvL//9Og4GBs4A+k4BtJbo1koUAQw3DWC4oXgniiJcXj8EARACl4cLwsXQAtTM44qDmD0+P85VulBiq8ZZazVKbIHJKk1nA8+rPVcOEGZtAoZ2SQ5M7dAnwwyVMsKni3weKeBcGn6Cg6ELAE/dK9zqMKRcDDw1p+ASA+N/dEmA1gyo9OHrMeb3S0dhgkHmS6D4a6C+U4XmjkDGQCnMWDoBR/4JfPcp4A8c8UvQAX0mAwN/Id3YkX3QqJViuGkAww1ReImiCFuVF8W2qouBx+oKhKAqlNhcOF3mgNMdeiWVXq3E4OwkDO0sBZ7+mYnhP7V15eIB54WLN1esc9PFAsBtv/J2AECRAGjMUtDRWgLzlkvma79Wa16RAJQcuhhkig7UP9haa5FCTMYg6bHjIMCUVnc9eynw9Xpg/5uhne6TukghZ8DPAXNG8/YZkUwYbhrAcEMUfV6fH0eKbdiTfwH/zb+AvacuoMLpCVlHrVRgQGZi8OjOoOwkGKNxP6CG1Jz6qjf4FAK2H6R7+DTm1FdTJWilq9VqB5mmdqAXReCHfcCBN4Fv3gfcldJyQQF0GyMFnR4Twn+5vdsZGB/1g9R7LTiqvfYj6l92uXWVKmn8lFIt1avUSGOllOrAo6qeZWoeqYojDDcNYLghkp/fL+J4qR178suwO/8C9uRfwLnK0FMuSoWAvhnm4GmsIZ2TkKiPwXveiKJ0dKfaJt0Rutpaa77iMssvmfdWAx16XTy91HGwNCC4JTdVvJTbIZ2yOvCW1OW+hr4d0O9OKeik9m7ctvx+6XRf+amLU82g7/JT0u0DYoVCFRp41AbpaJnGFJgC89p6loWsF3iulDlwt2EMNw1guCGKPaIo4lSZE3vyy/DfQNj5obz+1hbpFi3aGdRINmjQzqhGe2OteYMGyUY1DGpl5G+KGE6iGN0jDGXfSyHn4DuAveTi8o6DpZDTdyoAITSwlJ8ODTI+d8OfobFI45NUWmlbNS1C6msbItT3/JJ1/R7A65bGHfk8gNclzdcs87qlmvweRJRKLwUdlV4KSmrDxXmVHlDrAVVgeXBef/n1FQnSkTRBkB5rvnfI80tfv8z6cY7hpgEMN0StQ1FFFfYGTmPtyS/D9+cacdl3gCZBgfZGDZINarQzqtEuEH7aGdRoZ9SgnUENs04Fiy4h8KiCJqENdk/3eYHv/yOdtjr2ycVByILyynebViRIA6uTOgem7FrznaWB1nLw+6WQUzv4+NzSvLdaOoLlqgxM1lrzganmiFrIcpv03lim1NQTpoyhwaomfKmNlwSxWuvr20njuDRGub9RHQw3DWC4IWqdzttd+KbIijK7dLn7eYcrOF9md+G83Y0yh6tRV27VR5OggCUQdGoCj1mbELLs4nLp0aJXob1RHR/ByH5OGoR84E3pEnNA+kNXO7AkdQ7cKbqzdJVWWzpF43VLpx9dNikAeZxSUHI7Ls57nNJ4I7f94rzHEVjmuDhfe32/9+L4oliiNkkhx5QmDT43pUm3TKj9aEyL6h3CGW4awHBDFN+cbi/K7NJ9gMrsrtD5wP2BLjhcsFZ5YHV6UOnyoqX/K5ioV6GDUYMUswYpJi06mDRIMWnQITClmLRIMWtg0iSE7XRZzSX/DpcXTrcPDrcXbq8fmUl6JBlaMDZJFKWBwLpE6fQLRY8oSoPTax5rBlQH5+t7LfDcWx0ITfa6AepygavO+nbAUXZx4Hlj6JLqhh5TujRd/aOw7h6GmwYw3BBRbX6/iEqXF7YqD6xVHtiqPRfnq7zBZdbgspr1vLA6PU26YaJWpbgYdoLBR4NkgwZurw8Otw9OtxcOV+DR7YPT5Q0ud7qkEFPzeLl7LHYwadAj1YgeqaZakxEmbRgHKFP8clUClWeByjNAZYk0ePzSR1tx/fddqqFLBh7OD2tZTfn73YaOKRIR1aVQCMFTT5lNfK8oirBWeVBa6cK5ShdKK6tRanOFPq904ZzNhUqXF9UePwovVKHwQiN6ZjWBTqWEQaOEQhCCn32u0oUvToQ2Q82waNEjLTTwdEsxQq/mnwKqpebqsPbdLr+OKEpXA1aWALZ6QpDMd/XmkRsioiiocvuCgUd6vBiGyp1uaFRKGNRK6NUJMGgCj2ol9JoEGNQJ0GuU0qNaCYPm4ms6lRLKWm0t7C4vTpTa8V1JJb47W4ljZytx/KwdJbb6B8QKApCZpA+GnZ5pJnRLMSJRr4ZOpYROpYQmQRH+rvRETcTTUg1guCGitsjq9OB46cWwc6ykEsdLK3HefoVLugO0KkUw7GjVyuC8Tq2EVnWZ52oFEvXqwKX76uAVbPrWdqk+xQSeliIiohAWvQo5nZOR0zk5ZHmZ3YXvztprHeWpxMlzDthdXri8F8cTVXv8qPb4UY6W30em9qX6yYaaS/QD9ysKzkuX8Ne+b5HPL6La44PL66/z6PL4UH2Zx5rXXT4/jOoEJBnUSNKrkWRQIUkvfVaivo3eDiBOyRpuli1bhvfffx/ffvstdDodRowYgWeffRY9e/Zs8H15eXlYsGABDh8+jIyMDCxatAizZ8+OUtVERPGjnVGD4UYNhl/Vrs5rPr8Il9eHKrcPVR4fqj0+VLn9qPJIz6vcgWWeS9eR5p1uH8qd7ouX79tdUtDw+lFUUYWiisaNPVIrFfCLYpO61DeHQa1EkqEm7KiRrFdJjwZ1IBCpkKyXXlMqpLDlF0X4/CJ8ogi/v/Y86iwTRRG+Wsv9oogUkxbdUoxINWt4NCuMZA03eXl5mDNnDoYMGQKv14ulS5di3LhxOHLkCAwGQ73vyc/Px8SJE3H//ffjrbfewhdffIFf/epX6NChA6ZOnRrlb0BEFL+UCgF6dULYBhyLogin2yfdm6jW5fkX6swHHgP3LarvijSVUoA2QQmNSgFNrUetSgFNggLawFih2o8qpQIOlxcXHG5UOD244HSj3OFGRZUHPr8Ih9sHh7uq3rtjR5pBrcRVKUZ062DEVSlGXNXBiG4pBmQlG6BOUES9ntYupsbcnDt3DikpKcjLy8ONN95Y7zoPP/wwPvroIxw9ejS4bPbs2fjqq6+wa9euK34Gx9wQEbUeTrcURpSK0DCjDOMAZ79fRGW1F+VOdzDwlDs9gcfA5PCEvCaKIhQKAUpBgFIhQKEAlIIQskwQBCgvWV77dQA4U1GF0xec8F3mqFSCQkBWO30g7Eih56oOBlyVYoS5jV3a32rH3FitVgBAcnLyZdfZtWsXxo0bF7Js/PjxePXVV+HxeKBShf5ju1wuuFwXr8W32WxhrJiIiCIpnEeOLkehEGDRS3ec7oz6zxpEksvrQ0GZE9+fs+NEqR3fn3MEHu1wun04ec6Bk+cc2HwktCFpikkjhZ0UAwyaBIiidCrR5w+cAgucBhPF2qfGcMlpsovr+EURFp0KHZN0yEjUoWNgykjUwaCJqbhwRTFTrSiKWLBgAa6//nr07dv3suuVlJQgNTU1ZFlqaiq8Xi/Onz+P9PT0kNeWLVuGxx9/PCI1ExERtZQmQYnuqSZ0Tw29I7QoiiixVUtBp9SOE+fs+L7Uge/P2QO3EpCmXSfLLrPl8EnUq5Bh0aFjUmjokYKQFu0Nmpi6XUDMhJu5c+fi66+/xueff37FdS8ddFVzZq2+wViLFy/GggULgs9tNhsyM5t6qy4iIqLoEgQB6RYd0i063NC9Q8hr1ioPTp6TjvKcPGeHy+uXTo8JAhQCas1Lp8aCp8WCp8ZQ69SZ9BoEoNzhRlFFFc5USGOPzlRUwVbtRYXTgwqnB0eK6z/7oU5QIMOilcKORYdOSXo8OLqbbIOkYyLczJs3Dx999BG2b9+OTp06NbhuWloaSkpKQpaVlpYiISEB7drVHe2v0Wig0WjCWi8REZGcLDoVBmYlYWBW5LuvV1Z7cKaiGkUVThRVVKMoEHqKKqpQVF6Fs5XVcHv9OFXmxKkyJwCpBcivx3SPeG2XI2u4EUUR8+bNwwcffIDc3Fx06dLliu8ZPnw4/vWvf4Us27RpE3JycuqMtyEiIqKWMWlV6JmmQs+0+hupenx+lFirg2HnTEUV5L6qXdZwM2fOHLzzzjv45z//CZPJFDwiY7FYoNNJfSkWL16MoqIivPHGGwCkK6NWrlyJBQsW4P7778euXbvw6quvYu3atbJ9DyIiorZKpVQgM1mPzGS93KUEyXrx/KpVq2C1WjFy5Eikp6cHp/Xr1wfXKS4uRkFBQfB5ly5d8PHHHyM3NxcDBgzAE088gRUrVvAeN0RERAQgxu5zEw28zw0REVHr05S/37ztIREREcUVhhsiIiKKKww3REREFFcYboiIiCiuMNwQERFRXGG4ISIiorjCcENERERxheGGiIiI4grDDREREcUVhhsiIiKKKww3REREFFdk7Qouh5pWWjabTeZKiIiIqLFq/m43piVmmws3lZWVAIDMzEyZKyEiIqKmqqyshMViaXCdNtcV3O/348yZMzCZTBAEIazbttlsyMzMRGFhITuORxD3c3RwP0cH93P0cF9HR6T2syiKqKysREZGBhSKhkfVtLkjNwqFAp06dYroZ5jNZv6HEwXcz9HB/Rwd3M/Rw30dHZHYz1c6YlODA4qJiIgorjDcEBERUVxhuAkjjUaDP/zhD9BoNHKXEte4n6OD+zk6uJ+jh/s6OmJhP7e5AcVEREQU33jkhoiIiOIKww0RERHFFYYbIiIiiisMN0RERBRXGG6IiIgorjDchMlLL72ELl26QKvVYvDgwdixY4fcJcWdxx57DIIghExpaWlyl9Xqbd++HZMmTUJGRgYEQcCHH34Y8rooinjssceQkZEBnU6HkSNH4vDhw/IU24pdaT/PmjWrzu972LBh8hTbii1btgxDhgyByWRCSkoKJk+ejGPHjoWsw990yzVmP8v5m2a4CYP169dj/vz5WLp0KQ4cOIAbbrgBEyZMQEFBgdylxZ0+ffqguLg4OB06dEjuklo9h8OB/v37Y+XKlfW+/txzz+HPf/4zVq5cib179yItLQ1jx44NNqGlxrnSfgaAW265JeT3/fHHH0exwviQl5eHOXPmYPfu3di8eTO8Xi/GjRsHh8MRXIe/6ZZrzH4GZPxNi9RiQ4cOFWfPnh2yrFevXuIjjzwiU0Xx6Q9/+IPYv39/ucuIawDEDz74IPjc7/eLaWlp4jPPPBNcVl1dLVosFnH16tUyVBgfLt3PoiiKd911l3jbbbfJUk88Ky0tFQGIeXl5oijyNx0pl+5nUZT3N80jNy3kdrvx5ZdfYty4cSHLx40bh507d8pUVfw6fvw4MjIy0KVLF9x55504efKk3CXFtfz8fJSUlIT8vjUaDW666Sb+viMgNzcXKSkp6NGjB+6//36UlpbKXVKrZ7VaAQDJyckA+JuOlEv3cw25ftMMNy10/vx5+Hw+pKamhixPTU1FSUmJTFXFp2uvvRZvvPEGPvvsM7zyyisoKSnBiBEjUFZWJndpcavmN8zfd+RNmDABb7/9NrZu3YoXXngBe/fuxc033wyXyyV3aa2WKIpYsGABrr/+evTt2xcAf9ORUN9+BuT9TSdE/BPaCEEQQp6LolhnGbXMhAkTgvPXXHMNhg8fjquuugqvv/46FixYIGNl8Y+/78ibNm1acL5v377IyclBdnY2Nm7ciClTpshYWes1d+5cfP311/j888/rvMbfdPhcbj/L+ZvmkZsWat++PZRKZZ3EX1paWuf/GVB4GQwGXHPNNTh+/LjcpcStmqvR+PuOvvT0dGRnZ/P33Uzz5s3DRx99hG3btqFTp07B5fxNh9fl9nN9ovmbZrhpIbVajcGDB2Pz5s0hyzdv3owRI0bIVFXb4HK5cPToUaSnp8tdStzq0qUL0tLSQn7fbrcbeXl5/H1HWFlZGQoLC/n7biJRFDF37ly8//772Lp1K7p06RLyOn/T4XGl/VyfaP6meVoqDBYsWIAZM2YgJycHw4cPx8svv4yCggLMnj1b7tLiysKFCzFp0iRkZWWhtLQUTz75JGw2G+666y65S2vV7HY7Tpw4EXyen5+PgwcPIjk5GVlZWZg/fz6efvppdO/eHd27d8fTTz8NvV6Pn//85zJW3fo0tJ+Tk5Px2GOPYerUqUhPT8epU6ewZMkStG/fHj/5yU9krLr1mTNnDt555x3885//hMlkCh6hsVgs0Ol0EASBv+kwuNJ+ttvt8v6mZblGKw799a9/FbOzs0W1Wi0OGjQo5HI4Co9p06aJ6enpokqlEjMyMsQpU6aIhw8flrusVm/btm0igDrTXXfdJYqidOnsH/7wBzEtLU3UaDTijTfeKB46dEjeoluhhvaz0+kUx40bJ3bo0EFUqVRiVlaWeNddd4kFBQVyl93q1LePAYhr1qwJrsPfdMtdaT/L/ZsWAkUSERERxQWOuSEiIqK4wnBDREREcYXhhoiIiOIKww0RERHFFYYbIiIiiisMN0RERBRXGG6IiIgorjDcEBERUVxhuCEiIqK4wnBDREREcYXhhoiIiOLK/wNkLTWQDoGgYQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "KING RIVAR:\n",
      "I will to lay ble\n",
      "\n",
      "HAPOMENBELA:\n",
      "And thruans that hands:\n",
      "Waither us his vet?\n",
      "\n",
      "MEXENDEL:\n",
      "Warch, my feans' to zokn he oursertef it her than welll butes is eesen cin latistlivilv the do kine nown is wace!\n",
      " lill dise littius, on him speage aissell, yet lord.\n",
      "I mame, this down'st you, thee killo Wicho dhat evings to thed suis Then, it he poorter,-; the day danter firf sorre;\n",
      "I therf threy fleront than Pried by of.\n",
      "\n",
      "HENNG ERLANCE:\n",
      "YO:\n",
      "Ard all his a for huin cour ay and your to-chan the!\n",
      "\n",
      "J\n",
      "CPU times: user 3min 17s, sys: 490 ms, total: 3min 18s\n",
      "Wall time: 3min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size:int=vocab_size, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 head_size:int=head_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = head_size\n",
    "\n",
    "        #This will be our lookup table for embeddings. We'll have an entry for each token (aka vocab size) and each embedding will... \n",
    "        #...be a vector of dimension embedding_dim.\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        self.token_position_embedding_table = nn.Embedding(num_embeddings=self.context_length, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        self.transformer_blocks = nn.Sequential(\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "        )\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        B,T = idx.shape\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        token_position_embeddings = self.token_position_embedding_table(torch.arange(T, device=device)) #(T,C)\n",
    "        x = token_embeddings + token_position_embeddings\n",
    "        x = self.transformer_blocks(x)\n",
    "        logits = self.language_model_head_linear_layer(x) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Crop idx to the max size of our positional embeddings table \n",
    "            idx_crop = idx[:,-self.context_length:]\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx_crop)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = TransformerLanguageModel(vocab_size=vocab_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "tracked_losses = list()\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        tracked_losses.append(losses)\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "plot_losses(tracked_losses)\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf765ced-8e45-48b4-81b0-a7741b4ff941",
   "metadata": {},
   "source": [
    "This looks much better than our last run without the residual layers which had a loss of `Validation Loss: tensor(2.4430)` and it also beats the previous run before that had a los of `Validation Loss: tensor(2.2720)` with a final loss of `Validation Loss: tensor(2.0940)`. Also, as you can see the text output, while still gibberish, is much better than in all previous runs.<br>\n",
    "\n",
    "The second trick that helps with training deep neural nets, in addition to residual blocks, is the Norm as depicted on the block which in our case is layer norm. Let's implement and add that. [link](https://youtu.be/kCc8FmEb1nY?t=5709)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d94cdf4-f4ee-4ad3-8403-ab3420cbac3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class LayerNorm:\n",
    "    def __init__(self, dim:int, eps:float=1e-5):\n",
    "        self.dim = dim\n",
    "        self.eps = eps\n",
    "        self.gamma = torch.ones(dim)\n",
    "        self.beta = torch.zeros(dim)\n",
    "    \n",
    "    def __call__(self, x):\n",
    "        x_mean = x.mean(dim=1, keepdim=True) # layer mean\n",
    "        x_variance = x.var(dim=1, keepdim=True) # layer variance\n",
    "        x_hat = (x - x_mean) / torch.sqrt(x_variance + self.eps) # normalize to the unit variance\n",
    "        self.out = self.gamma * x_hat + self.beta\n",
    "        return self.out\n",
    "    \n",
    "    def parameters(self):\n",
    "        return [self.gamma, self.beta]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a43f3a7-c591-4011-8c70-540f347a5e88",
   "metadata": {},
   "source": [
    "Since the original attention is all you need paper came out, it has become more common to apply the norm prior to the blocks instead of after them with the add as is depicted on the transformer architecture diagram. We will follow what common practice is today. Also instead of using the layer norm we developed, we will use the Pytorch version instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2f526ce-643b-4ae6-8c16-506792dc4213",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class TransformerBlock(nn.Module):\n",
    "    \"\"\"Transformer Block: Communication folled by computation.\"\"\"\n",
    "    def __init__(self, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 num_heads:int=4):\n",
    "        #embedding_dim: embedding dimension, num_heads: the number of heads that we want\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = embedding_dim // num_heads\n",
    "        self.num_heads = num_heads\n",
    "        \n",
    "        self.multi_self_attention_heads_layer = MultiHeadAttention(num_heads=self.num_heads, head_size=self.head_size, \n",
    "                                                                   embedding_dim=embedding_dim, context_length=context_length)\n",
    "        self.feed_forward_network = FeedForwardNetwork(embedding_dim=self.embedding_dim)\n",
    "        self.layer_norm_1 = nn.LayerNorm(normalized_shape=self.embedding_dim) #NEW\n",
    "        self.layer_norm_2 = nn.LayerNorm(normalized_shape=self.embedding_dim) #NEW\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # return self.feed_forward_network(self.multi_self_attention_heads_layer(x))\n",
    "        x = x + self.multi_self_attention_heads_layer(self.layer_norm_1(x)) # added layer norm. UPDATED\n",
    "        x = x + self.feed_forward_network(self.layer_norm_2(x)) # added layer norm. UPDATED\n",
    "        return x\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a91dc3fd-9e55-4c2f-ba2f-3614a431c2a7",
   "metadata": {},
   "source": [
    "These layer norms are applied to each token embedding to ensure they start off having a unit gausian at initialization, but because of the trainable parameters, this may change during training. <br>\n",
    "\n",
    "We also need to add a layer norm after the last transformer block and before the last linear layer. Now let's train the model again and see how it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4175214-a991-421c-8599-f949afa77fc3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: tensor(4.3103) Validation Loss: tensor(4.3100)\n",
      "Step: 200 Training Loss: tensor(2.6644) Validation Loss: tensor(2.6888)\n",
      "Step: 400 Training Loss: tensor(2.4590) Validation Loss: tensor(2.4470)\n",
      "Step: 600 Training Loss: tensor(2.3602) Validation Loss: tensor(2.3479)\n",
      "Step: 800 Training Loss: tensor(2.2801) Validation Loss: tensor(2.2854)\n",
      "Step: 1000 Training Loss: tensor(2.2313) Validation Loss: tensor(2.2563)\n",
      "Step: 1200 Training Loss: tensor(2.2185) Validation Loss: tensor(2.2377)\n",
      "Step: 1400 Training Loss: tensor(2.1741) Validation Loss: tensor(2.2103)\n",
      "Step: 1600 Training Loss: tensor(2.1425) Validation Loss: tensor(2.1853)\n",
      "Step: 1800 Training Loss: tensor(2.1290) Validation Loss: tensor(2.1792)\n",
      "Step: 2000 Training Loss: tensor(2.1295) Validation Loss: tensor(2.1381)\n",
      "Step: 2200 Training Loss: tensor(2.1140) Validation Loss: tensor(2.1594)\n",
      "Step: 2400 Training Loss: tensor(2.0825) Validation Loss: tensor(2.1407)\n",
      "Step: 2600 Training Loss: tensor(2.0727) Validation Loss: tensor(2.1325)\n",
      "Step: 2800 Training Loss: tensor(2.0618) Validation Loss: tensor(2.1148)\n",
      "Step: 3000 Training Loss: tensor(2.0459) Validation Loss: tensor(2.1033)\n",
      "Step: 3200 Training Loss: tensor(2.0515) Validation Loss: tensor(2.1216)\n",
      "Step: 3400 Training Loss: tensor(2.0321) Validation Loss: tensor(2.0743)\n",
      "Step: 3600 Training Loss: tensor(2.0179) Validation Loss: tensor(2.0913)\n",
      "Step: 3800 Training Loss: tensor(2.0171) Validation Loss: tensor(2.0952)\n",
      "Step: 4000 Training Loss: tensor(2.0151) Validation Loss: tensor(2.0876)\n",
      "Step: 4200 Training Loss: tensor(1.9998) Validation Loss: tensor(2.0803)\n",
      "Step: 4400 Training Loss: tensor(2.0134) Validation Loss: tensor(2.0872)\n",
      "Step: 4600 Training Loss: tensor(1.9862) Validation Loss: tensor(2.0807)\n",
      "Step: 4800 Training Loss: tensor(1.9923) Validation Loss: tensor(2.0776)\n",
      "Step: 4999 Training Loss: tensor(1.9644) Validation Loss: tensor(2.0590)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGxCAYAAACeKZf2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABSQUlEQVR4nO3deXgTdf4H8Pckac42aQo9aQut3PdRlOKJ3LgsCCq6LIrnoqCL/FgVWF1dRdSVlVVWWF0ET0TFg11UjoUiKCjIqSByFFqgpfRK2rRJmmR+f0waKC2lR5JJ0/freeZJMplMPsmG7duZ73w/giiKIoiIiIjChELuAoiIiIj8ieGGiIiIwgrDDREREYUVhhsiIiIKKww3REREFFYYboiIiCisMNwQERFRWGG4ISIiorDCcENERERhheGGiIJixYoVEAQBu3btkrsUIgpzDDdEREQUVhhuiIiIKKww3BBRyNi2bRuGDh2KqKgo6PV6DB48GGvXrq2xTUVFBWbPno20tDRotVrExMQgIyMDK1eu9G1z/Phx3H777UhKSoJGo0F8fDyGDh2KvXv31tjXqlWrkJmZCYPBgMjISIwcORJ79uypsU1D90VEoUMldwFERACwZcsWDB8+HL1798ayZcug0Wjw+uuvY+zYsVi5ciUmTZoEAJg1axbeffddPPfcc+jXrx9sNht++uknFBUV+fY1ZswYuN1uvPTSS0hNTUVhYSG+++47lJaW+rZ5/vnn8ec//xl33303/vznP8PpdOJvf/sbrr32Wvzwww/o3r17g/dFRKFFEEVRlLsIIgp/K1aswN13342dO3ciIyOj1vOZmZk4fvw4jh07hsjISACA2+1G3759UVpaipycHAiCgF69eqFjx4747LPP6nyfoqIitG3bFosWLcIf//jHOrfJzc1Feno6HnzwQbz66qu+9eXl5ejUqROuu+46rFq1qkH7IqLQw9NSRCQ7m82G77//Hrfccosv2ACAUqnElClTcOrUKRw+fBgAcOWVV+Krr77CE088gaysLFRWVtbYV0xMDK644gr87W9/w9///nfs2bMHHo+nxjbr1q2Dy+XCnXfeCZfL5Vu0Wi2uv/56ZGVlNXhfRBR6GG6ISHYlJSUQRRGJiYm1nktKSgIA32mnV199FY8//jg+//xzDBkyBDExMRg/fjyOHDkCABAEAf/73/8wcuRIvPTSS+jfvz9iY2PxyCOPoKysDABw9uxZAMDAgQMRERFRY1m1ahUKCwsbvC8iCj0cc0NEsjObzVAoFMjLy6v13JkzZwAAbdu2BQAYDAY888wzeOaZZ3D27FnfUZyxY8fil19+AQC0b98ey5YtAwD8+uuv+Oijj/D000/D6XRi6dKlvn198sknaN++fb21XW5fRBR6OOaGiILicmNuBg8ejOzsbBw/fhw6nQ4A4PF40LdvX5SUlPjG3NTl0UcfxaJFi2Cz2aDX6+vcpl+/foiIiMAPP/yAEydOoFOnTpg/fz4ee+yxRn+WC/dFRKGHR26IKKg2bdqEEydO1Fq/YMECDB8+HEOGDMHs2bOhVqvx+uuv46effsLKlSt9weaqq67Cb37zG/Tu3RtmsxmHDh3Cu+++i8zMTOj1euzfvx8zZszArbfeik6dOkGtVmPTpk3Yv38/nnjiCQBAhw4d8Ne//hXz5s3D8ePHMWrUKJjNZpw9exY//PCD7+hQQ/ZFRCFIJCIKguXLl4sALrlkZ2eLW7duFW+88UbRYDCIOp1OHDRokPif//ynxn6eeOIJMSMjQzSbzaJGoxHT09PFRx99VCwsLBRFURTPnj0rTp06VezatatoMBjEyMhIsXfv3uIrr7wiulyuGvv6/PPPxSFDhohGo1HUaDRi+/btxVtuuUXcuHFjo/dFRKGDp6WIiIgorPBqKSIiIgorDDdEREQUVhhuiIiIKKww3BAREVFYYbghIiKisMJwQ0RERGGl1U3i5/F4cObMGURFRV1ytlMiIiIKLaIooqysDElJSVAo6j820+rCzZkzZ5CSkiJ3GURERNQEubm5SE5OrnebVhduoqKiAEhfjtFolLkaIiIiagir1YqUlBTf3/H6tLpwU30qymg0MtwQERG1MA0ZUsIBxURERBRWGG6IiIgorDDcEBERUVhpdWNuiIioeURRhMvlgtvtlrsUCjMRERFQKpXN3g/DDRERNZjT6UReXh4qKirkLoXCkCAISE5ORmRkZLP2w3BDREQN4vF4kJ2dDaVSiaSkJKjVak6GSn4jiiLOnTuHU6dOoVOnTs06gsNwQ0REDeJ0OuHxeJCSkgK9Xi93ORSGYmNjceLECVRVVTUr3HBAMRERNcrlpr4naip/HQnkL5SIiIjCCsMNERERhRWGGyIioka64YYbMHPmzAZvf+LECQiCgL179wasJjqP4YaIiMKWIAj1LlOnTm3Sfj/99FM8++yzDd4+JSUFeXl56NmzZ5Per6EYoiS8WspPSgvPYv/nL0OosuPaB1+TuxwiIgKQl5fnu79q1So89dRTOHz4sG+dTqersX1VVRUiIiIuu9+YmJhG1aFUKpGQkNCo11DT8ciNn7iclbju1BvIzH8PHs7aSUSthCiKqHC6gr6Iotig+hISEnyLyWSCIAi+x3a7HdHR0fjoo49www03QKvV4r333kNRURHuuOMOJCcnQ6/Xo1evXli5cmWN/V58WqpDhw54/vnncc899yAqKgqpqal44403fM9ffEQlKysLgiDgf//7HzIyMqDX6zF48OAawQsAnnvuOcTFxSEqKgr33XcfnnjiCfTt27dJ/1sBgMPhwCOPPIK4uDhotVpcc8012Llzp+/5kpISTJ48GbGxsdDpdOjUqROWL18OQJoKYMaMGUhMTIRWq0WHDh2wYMGCJtcSSDxy4yfGmHgAgErwwGIpgikmTuaKiIgCr7LKje5PrQv6+x7860jo1f75E/b4449j4cKFWL58OTQaDex2OwYMGIDHH38cRqMRa9euxZQpU5Ceno6rrrrqkvtZuHAhnn32WcydOxeffPIJHnzwQVx33XXo2rXrJV8zb948LFy4ELGxsZg2bRruuecefPvttwCA999/H/Pnz8frr7+Oq6++Gh9++CEWLlyItLS0Jn/Wxx57DKtXr8bbb7+N9u3b46WXXsLIkSNx9OhRxMTE4Mknn8TBgwfx1VdfoW3btjh69CgqKysBAK+++irWrFmDjz76CKmpqcjNzUVubm6Tawkkhhs/UWt1KBd1iBQqYS0+y3BDRNRCzJw5ExMmTKixbvbs2b77Dz/8ML7++mt8/PHH9YabMWPG4KGHHgIgBaZXXnkFWVlZ9Yab+fPn4/rrrwcAPPHEE7jppptgt9uh1Wrx2muv4d5778Xdd98NAHjqqaewfv16lJeXN+lz2mw2LFmyBCtWrMDo0aMBAG+++SY2bNiAZcuW4U9/+hNycnLQr18/ZGRkAJCOSFXLyclBp06dcM0110AQBLRv375JdQQDw40fWRVGRIqVsJWcBdBL7nKIiAJOF6HEwb+OlOV9/aX6D3k1t9uNF154AatWrcLp06fhcDjgcDhgMBjq3U/v3r1996tPfxUUFDT4NYmJiQCAgoICpKam4vDhw76wVO3KK6/Epk2bGvS5Lnbs2DFUVVXh6quv9q2LiIjAlVdeiUOHDgEAHnzwQUycOBG7d+/GiBEjMH78eAwePBgAMHXqVAwfPhxdunTBqFGj8Jvf/AYjRoxoUi2BxnDjR+VKE+A6C4flnNylEBEFhSAIfjs9JJeLQ8vChQvxyiuvYNGiRejVqxcMBgNmzpwJp9NZ734uHogsCAI8Hk+DX1M9O++Fr7l4xt6GjjWqS/Vr69pn9brRo0fj5MmTWLt2LTZu3IihQ4di+vTpePnll9G/f39kZ2fjq6++wsaNG3Hbbbdh2LBh+OSTT5pcU6BwQLEf2VUmAICzjOGGiKil2rp1K8aNG4ff//736NOnD9LT03HkyJGg19GlSxf88MMPNdbt2rWryfvr2LEj1Go1tm3b5ltXVVWFXbt2oVu3br51sbGxmDp1Kt577z0sWrSoxsBoo9GISZMm4c0338SqVauwevVqFBcXN7mmQGnZcTvEONXRgB1wlxfKXQoRETVRx44dsXr1anz33Xcwm834+9//jvz8/BoBIBgefvhh3H///cjIyMDgwYOxatUq7N+/H+np6Zd97cVXXQFA9+7d8eCDD+JPf/oTYmJikJqaipdeegkVFRW49957AUjjegYMGIAePXrA4XDgv//9r+9zv/LKK0hMTETfvn2hUCjw8ccfIyEhAdHR0X793P7AcONHLm0MYAVQEXoploiIGubJJ59EdnY2Ro4cCb1ejwceeADjx4+HxWIJah2TJ0/G8ePHMXv2bNjtdtx2222YOnVqraM5dbn99ttrrcvOzsYLL7wAj8eDKVOmoKysDBkZGVi3bh3MZjMAQK1WY86cOThx4gR0Oh2uvfZafPjhhwCAyMhIvPjiizhy5AiUSiUGDhyIL7/8MiQbqQpic07gtUBWqxUmkwkWiwVGo9Gv+96+/AlknlyCneabMPCPH/h130REcrPb7cjOzkZaWhq0Wq3c5bRKw4cPR0JCAt599125SwmI+n5jjfn7zSM3fqQwtAUARDhKZK6EiIhauoqKCixduhQjR46EUqnEypUrsXHjRmzYsEHu0kIew40fqaKkcKOtKpW3ECIiavEEQcCXX36J5557Dg6HA126dMHq1asxbNgwuUsLeQw3fqQxxQIA9O7gnpclIqLwo9PpsHHjRrnLaJFCbxRQC6YzSbMSGz1WmSshIiJqvRhu/Ki6v5RRLIfodslcDRERUevEcONHRm8/KYUgoqyUc90QERHJgeHGjzQaLayiHgBQVpwvczVEREStE8ONn1kV0rX3tlK2YCAiIpIDw42flSul/lJ2S/2dYImIiCgwGG78zK6KBgA4yzjmhogoXNxwww2YOXOm73GHDh2waNGiel8jCAI+//zzZr+3v/bTmjDc+JlTHQ0A8LB5JhGR7MaOHXvJSe+2b98OQRCwe/fuRu93586deOCBB5pbXg1PP/00+vbtW2t9Xl4eRo8e7df3utiKFStCsgFmUzHc+JlLGyPdqSiStxAiIsK9996LTZs24eTJk7Wee+utt9C3b1/079+/0fuNjY2FXq/3R4mXlZCQAI1GE5T3ChcMN34m6tsAABR29pciolZAFAGnLfhLA3s+/+Y3v0FcXBxWrFhRY31FRQVWrVqFe++9F0VFRbjjjjuQnJwMvV6PXr16YeXKlfXu9+LTUkeOHMF1110HrVaL7t2719n/6fHHH0fnzp2h1+uRnp6OJ598ElVVVQCkIyfPPPMM9u3bB0EQIAiCr+aLT0sdOHAAN954I3Q6Hdq0aYMHHngA5eXlvuenTp2K8ePH4+WXX0ZiYiLatGmD6dOn+96rKXJycjBu3DhERkbCaDTitttuw9mzZ33P79u3D0OGDEFUVBSMRiMGDBiAXbt2AQBOnjyJsWPHwmw2w2AwoEePHvjyyy+bXEtDsP2Cnyn00pEbtaNY5kqIiIKgqgJ4Pin47zv3DKA2XHYzlUqFO++8EytWrMBTTz0FQRAAAB9//DGcTicmT56MiooKDBgwAI8//jiMRiPWrl2LKVOmID09HVddddVl38Pj8WDChAlo27YtduzYAavVWmN8TrWoqCisWLECSUlJOHDgAO6//35ERUXhsccew6RJk/DTTz/h66+/9rVcMJlMtfZRUVGBUaNGYdCgQdi5cycKCgpw3333YcaMGTUC3ObNm5GYmIjNmzfj6NGjmDRpEvr27Yv777//sp/nYqIoYvz48TAYDNiyZQtcLhceeughTJo0CVlZWQCAyZMno1+/fliyZAmUSiX27t2LiIgIAMD06dPhdDrxzTffwGAw4ODBg4iMjGx0HY3BcONnEWyeSUQUUu655x787W9/Q1ZWFoYMGQJAOiU1YcIEmM1mmM1mzJ4927f9ww8/jK+//hoff/xxg8LNxo0bcejQIZw4cQLJyckAgOeff77WOJk///nPvvsdOnTA//3f/2HVqlV47LHHoNPpEBkZCZVKhYSEhEu+1/vvv4/Kykq88847MBikcLd48WKMHTsWL774IuLjpZnyzWYzFi9eDKVSia5du+Kmm27C//73vyaFm40bN2L//v3Izs5GSkoKAODdd99Fjx49sHPnTgwcOBA5OTn405/+hK5duwIAOnXq5Ht9Tk4OJk6ciF69egEA0tPTG11DYzHc+JnG21/KwOaZRNQaROiloyhyvG8Dde3aFYMHD8Zbb72FIUOG4NixY9i6dSvWr18PAHC73XjhhRewatUqnD59Gg6HAw6HwxceLufQoUNITU31BRsAyMzMrLXdJ598gkWLFuHo0aMoLy+Hy+WC0Whs8Oeofq8+ffrUqO3qq6+Gx+PB4cOHfeGmR48eUCqVvm0SExNx4MCBRr3Xhe+ZkpLiCzYA0L17d0RHR+PQoUMYOHAgZs2ahfvuuw/vvvsuhg0bhltvvRVXXHEFAOCRRx7Bgw8+iPXr12PYsGGYOHEievfu3aRaGopjbvysunlmpKdM5kqIiIJAEKTTQ8FevKeXGuree+/F6tWrYbVasXz5crRv3x5Dhw4FACxcuBCvvPIKHnvsMWzatAl79+7FyJEj4XQ6G7RvsY7xP8JF9e3YsQO33347Ro8ejf/+97/Ys2cP5s2b1+D3uPC9Lt53Xe9ZfUrowuc8Hk+j3uty73nh+qeffho///wzbrrpJmzatAndu3fHZ599BgC47777cPz4cUyZMgUHDhxARkYGXnvttSbV0lAMN34W5W2eaUI5RHfTB28REZH/3HbbbVAqlfjggw/w9ttv4+677/b9Yd66dSvGjRuH3//+9+jTpw/S09Nx5MiRBu+7e/fuyMnJwZkz549gbd++vcY23377Ldq3b4958+YhIyMDnTp1qnUFl1qthtvtvux77d27Fzabrca+FQoFOnfu3OCaG6P68+Xm5vrWHTx4EBaLBd26dfOt69y5Mx599FGsX78eEyZMwPLly33PpaSkYNq0afj000/xf//3f3jzzTcDUms1hhs/M8XEwiNK/2DK2YKBiCgkREZGYtKkSZg7dy7OnDmDqVOn+p7r2LEjNmzYgO+++w6HDh3CH/7wB+TnN7w/4LBhw9ClSxfceeed2LdvH7Zu3Yp58+bV2KZjx47IycnBhx9+iGPHjuHVV1/1Hdmo1qFDB2RnZ2Pv3r0oLCyEw+Go9V6TJ0+GVqvFXXfdhZ9++gmbN2/Gww8/jClTpvhOSTWV2+3G3r17aywHDx7EsGHD0Lt3b0yePBm7d+/GDz/8gDvvvBPXX389MjIyUFlZiRkzZiArKwsnT57Et99+i507d/qCz8yZM7Fu3TpkZ2dj9+7d2LRpU41QFAgMN36m1WhghXQutKz47GW2JiKiYLn33ntRUlKCYcOGITU11bf+ySefRP/+/TFy5EjccMMNSEhIwPjx4xu8X4VCgc8++wwOhwNXXnkl7rvvPsyfP7/GNuPGjcOjjz6KGTNmoG/fvvjuu+/w5JNP1thm4sSJGDVqFIYMGYLY2Ng6L0fX6/VYt24diouLMXDgQNxyyy0YOnQoFi9e3Lgvow7l5eXo169fjWXMmDG+S9HNZjOuu+46DBs2DOnp6Vi1ahUAQKlUoqioCHfeeSc6d+6M2267DaNHj8YzzzwDQApN06dPR7du3TBq1Ch06dIFr7/+erPrrY8g1nWyMIxZrVaYTCZYLJZGD+RqqJxnuiFVPIMjYz5CpytHBuQ9iIiCzW63Izs7G2lpadBqtXKXQ2Govt9YY/5+88hNALB5JhERkXwYbgLArpLCTVUZx9wQEREFG8NNADjVZgCA28b+UkRERMHGcBMAbm/zTIHNM4mIiIKO4SYAPN7+UmyeSUThqJVdh0JB5K/fFsNNACgNUmfwCAfDDRGFj+pZbysqKmSuhMJV9YzNF7aOaAr2lgoAZVQsAEDH5plEFEaUSiWio6NRUCBdCarX6y/ZCoCosTweD86dOwe9Xg+VqnnxhOEmALRGKdzo2TyTiMJMdcfq6oBD5E8KhQKpqanNDs0MNwGgj5aaZxo9VpkrISLyL0EQkJiYiLi4OFRVsX8e+ZdarYZC0fwRMww3ARDpbZ4ZiQqILgcElUbmioiI/EupVDZ7XARRoHBAcQBEm2Ph9jbPtJXy0C0REVEwhUy4WbBgAQRBwMyZM+vdbsuWLRgwYAC0Wi3S09OxdOnS4BTYCDpNBCyIBACUFzPcEBERBVNIhJudO3fijTfeQO/evevdLjs7G2PGjMG1116LPXv2YO7cuXjkkUewevXqIFXacFaF1NSrvISdwYmIiIJJ9nBTXl6OyZMn480334TZbK5326VLlyI1NRWLFi1Ct27dcN999+Gee+7Byy+/HKRqG666eabDyv5SREREwSR7uJk+fTpuuukmDBs27LLbbt++HSNGjKixbuTIkdi1a9clR+07HA5YrdYaSzDYVdEAAGdZYVDej4iIiCSyhpsPP/wQu3fvxoIFCxq0fX5+PuLj42usi4+Ph8vlQmFh3SFiwYIFMJlMviUlJaXZdTeEUyMdhfKUs78UERFRMMkWbnJzc/HHP/4R7733HrRabYNfd/HEPtV9KC414c+cOXNgsVh8S25ubtOLbgSX1nuKrZLhhoiIKJhkm+fmxx9/REFBAQYMGOBb53a78c0332Dx4sVwOBy15lBISEhAfn5+jXUFBQVQqVRo06ZNne+j0Wig0QR/nhlRJ9WjrCwO+nsTERG1ZrKFm6FDh+LAgQM11t19993o2rUrHn/88Tonh8rMzMR//vOfGuvWr1+PjIwMX0O3UKE0SJ3B1U42zyQiIgom2cJNVFQUevbsWWOdwWBAmzZtfOvnzJmD06dP45133gEATJs2DYsXL8asWbNw//33Y/v27Vi2bBlWrlwZ9PovR+Vtnqll80wiIqKgkv1qqfrk5eUhJyfH9zgtLQ1ffvklsrKy0LdvXzz77LN49dVXMXHiRBmrrJvWJIUbg4vNM4mIiIJJEKtH5LYSVqsVJpMJFosFRqMxYO/z68F96PzRdbBBC8PTnMiPiIioORrz9zukj9y0ZNXNMw2wQ6yqlLkaIiKi1oPhJkCizW3gEqWvt9LCWYqJiIiCheEmQHRqFUoQBQCwFvG0FBERUbAw3ASIIAi+5pkVpewMTkREFCwMNwFk8zbPtLN5JhERUdAw3ASQPUIKN1UMN0REREHDcBNATrW3eaaNncGJiIiCheEmgNxaqQUDKtg8k4iIKFgYbgJI1EnhRmkvlbcQIiKiVoThJoAUhrYAgAg2zyQiIgoahpsAijBK4YbNM4mIiIKH4SaANMY4AGyeSUREFEwMNwGkj5bCjVFkuCEiIgoWhpsAijRLzTO1cEJ02mSuhoiIqHVguAkgc7QZTlEJALCzeSYREVFQMNwEkF6jQgmk/lJlJWyeSUREFAwMNwEkNc+UOoNXlLB5JhERUTAw3ASYTRkNAKjkaSkiIqKgYLgJMLvK2zyzjOGGiIgoGBhuAsypYfNMIiKiYGK4CTCXr3lmsbyFEBERtRIMN4GmbwMAUNoZboiIiIKB4SbAFHrpyI3aWSpvIURERK0Ew02AsXkmERFRcDHcBNj55pml8hZCRETUSjDcBJjeVN080wqIoszVEBERhT+GmwAztkkAAKjhAtg8k4iIKOAYbgIs2mSCXYwAwOaZREREwcBwE2AGjQolkPpLlZXky1wNERFR+GO4CTBBEGAVpBYMNjbPJCIiCjiGmyCweftLOaw8LUVERBRoDDdBwOaZREREwcNwEwROTTQAwGMrkrcQIiKiVoDhJgjcvuaZDDdERESBxnATDLrq5pklMhdCREQU/hhugkAwSOFG7WC4ISIiCjSGmyCIMMYCALTsL0VERBRwDDdBoPWGG4PLInMlRERE4Y/hJgj05ngAbJ5JREQUDAw3QWA0S53BVXADDqvM1RAREYU3hpsgMJmMqBA1ADhLMRERUaAx3ARB1IXNM4vPylwNERFReGO4CQKpeaYRAFBRyuaZREREgcRwEyTVzTPtPC1FREQUUAw3QVKpigYAVJUVylsIERFRmGO4CZIqjRkA4ClnuCEiIgokhpsgOd88s1jeQoiIiMIcw02QiDop3KjsDDdERESBxHATJMpIqXlmhJPNM4mIiAKJ4SZIVFFtAQC6qlJ5CyEiIgpzDDdBojV5m2e62TyTiIgokBhugkQfLTXPjBLLAI9H5mqIiIjCF8NNkER5m2cq4QEcPHpDREQUKAw3QWI2RqFM1AEAnGWcpZiIiChQGG6CxKg93zyznM0ziYiIAobhJkgEQUCZt3mmrYTNM4mIiAKF4SaIyr3NMx1snklERBQwDDdB5PCGGzbPJCIiChyGmyByeptnum0MN0RERIHCcBNE1c0zhYoimSshIiIKXww3QSTqpXCjZPNMIiKigJE13CxZsgS9e/eG0WiE0WhEZmYmvvrqq0tun5WVBUEQai2//PJLEKtuOqVB6i+ldnISPyIiokBRyfnmycnJeOGFF9CxY0cAwNtvv41x48Zhz5496NGjxyVfd/jwYRiNRt/j2NjYgNfqD9XNM7VsnklERBQwsoabsWPH1ng8f/58LFmyBDt27Kg33MTFxSE6OjrA1fmfziS1YIhk80wiIqKACZkxN263Gx9++CFsNhsyMzPr3bZfv35ITEzE0KFDsXnz5nq3dTgcsFqtNRa56KK94UYsAzxu2eogIiIKZ7KHmwMHDiAyMhIajQbTpk3DZ599hu7du9e5bWJiIt544w2sXr0an376Kbp06YKhQ4fim2++ueT+FyxYAJPJ5FtSUlIC9VEuy+htnqmACFSWylYHERFROBNEURTlLMDpdCInJwelpaVYvXo1/v3vf2PLli2XDDgXGzt2LARBwJo1a+p83uFwwOFw+B5brVakpKTAYrHUGLcTDKUVTihebA+jUAHntO+hTuga1PcnIiJqqaxWK0wmU4P+fst+5EatVqNjx47IyMjAggUL0KdPH/zjH/9o8OsHDRqEI0eOXPJ5jUbjuxqrepGLURuBYm/zTFsJm2cSEREFguzh5mKiKNY40nI5e/bsQWJiYgAr8h+F4nzzzIpSNs8kIiIKBFmvlpo7dy5Gjx6NlJQUlJWV4cMPP0RWVha+/vprAMCcOXNw+vRpvPPOOwCARYsWoUOHDujRowecTifee+89rF69GqtXr5bzYzSKTWkC3IDdwuaZREREgSBruDl79iymTJmCvLw8mEwm9O7dG19//TWGDx8OAMjLy0NOTo5ve6fTidmzZ+P06dPQ6XTo0aMH1q5dizFjxsj1ERrNHhENuIGqcvaXIiIiCgTZBxQHW2MGJAXCulfuw0jLxziYdhe63/Vq0N+fiIioJWpRA4pbG4+veSb7SxEREQUCw02QsXkmERFRYDHcBJnS0AYAoHaWyFwJERFReGK4CbKIKKnJJ5tnEhERBQbDTZBpTVK4iXTL1+OKiIgonDHcBJnBXN08sxxwu2SuhoiIKPww3ARZZHQsPKIgPajkuBsiIiJ/Y7gJsphIPSwwAACqyjlLMRERkb8x3ASZUReBElFqnlnO5plERER+x3ATZEqFAKtCmlmxks0ziYiI/I7hRgY2lQkAm2cSEREFAsONDOzecOMqY7ghIiLyN4YbGVRpzAAAj61I5kqIiIjCD8ONDNze5pmoZH8pIiIif2O4kYNe6i+lYrghIiLyO4YbGSgi2wIAIpyl8hZCREQUhhhuZBDhDTc6V6m8hRAREYUhhhsZVDfPNLgtMldCREQUfhhuZKA3xwMADGIF4HLKXA0REVF4YbiRgTG6Ldy+5pkcVExERORPDDcyMBs0KEUkAE7kR0RE5G8MNzIwXdA808b+UkRERH7FcCMDlVLha55ZwXBDRETkVww3MrEppf5SDjbPJCIi8iuGG5nYI6IBAM7yQnkLISIiCjMMNzJxeptnimyeSURE5FcMNzLxeJtnChUMN0RERP7EcCMXb/NMpZ3z3BAREfkTw41MFAbpyI2azTOJiIj8iuFGJuooqb+UrqpU3kKIiIjCDMONTDSmOACAwcPmmURERP7EcCMTg7d5pk60A1V2mashIiIKHww3MjGaYuASvV8/m2cSERH5TZPCTW5uLk6dOuV7/MMPP2DmzJl44403/FZYuDNHalACqb8Um2cSERH5T5PCze9+9zts3rwZAJCfn4/hw4fjhx9+wNy5c/HXv/7VrwWGq2hdBIrZPJOIiMjvmhRufvrpJ1x55ZUAgI8++gg9e/bEd999hw8++AArVqzwZ31h68LmmZUMN0RERH7TpHBTVVUFjUYDANi4cSN++9vfAgC6du2KvLw8/1UX5iqqm2daeVqKiIjIX5oUbnr06IGlS5di69at2LBhA0aNGgUAOHPmDNq0aePXAsNZdfPMqjI2zyQiIvKXJoWbF198Ef/6179www034I477kCfPn0AAGvWrPGdrqLLq/I1z2S4ISIi8hdVU150ww03oLCwEFarFWaz2bf+gQcegF6v91tx4c6tjQEs4KXgREREftSkIzeVlZVwOBy+YHPy5EksWrQIhw8fRlxcnF8LDGt6qb+Uis0ziYiI/KZJ4WbcuHF45513AAClpaW46qqrsHDhQowfPx5Llizxa4HhTGFoCwCIYPNMIiIiv2lSuNm9ezeuvfZaAMAnn3yC+Ph4nDx5Eu+88w5effVVvxYYziJ8zTPZX4qIiMhfmhRuKioqEBUlTUC3fv16TJgwAQqFAoMGDcLJkyf9WmA400VL4SbSzXBDRETkL00KNx07dsTnn3+O3NxcrFu3DiNGjAAAFBQUwGg0+rXAcKaLlsYnaeAAnBUyV0NERBQemhRunnrqKcyePRsdOnTAlVdeiczMTADSUZx+/fr5tcBwFm0ywykqpQcVRfIWQ0REFCaadCn4LbfcgmuuuQZ5eXm+OW4AYOjQobj55pv9Vly4Mxuk5pnxKIXbVghldIrcJREREbV4TQo3AJCQkICEhAScOnUKgiCgXbt2nMCvkaL1ETgqRiFeKEVFSQGi2sldERERUcvXpNNSHo8Hf/3rX2EymdC+fXukpqYiOjoazz77LDwej79rDFsRFzTPrLCweSYREZE/NOnIzbx587Bs2TK88MILuPrqqyGKIr799ls8/fTTsNvtmD9/vr/rDFs2pQlwAw6GGyIiIr9oUrh5++238e9//9vXDRwA+vTpg3bt2uGhhx5iuGkEe4QZcANV5RxQTERE5A9NOi1VXFyMrl271lrftWtXFBezlUBjVGmiAQCijeGGiIjIH5oUbvr06YPFixfXWr948WL07t272UW1Jh6t1F9KqGS4ISIi8ocmnZZ66aWXcNNNN2Hjxo3IzMyEIAj47rvvkJubiy+//NLfNYY3fRsAgNJeInMhRERE4aFJR26uv/56/Prrr7j55ptRWlqK4uJiTJgwAT///DOWL1/u7xrDmtIghRu1k+GGiIjIH5o8z01SUlKtgcP79u3D22+/jbfeeqvZhbUWEcbq5pml8hZCREQUJpp05Ib8R2O8oHmmKMpcDRERUcvHcCMzQ3Q8ACACLsBZLnM1RERELR/DjcyiTSbYxQjpAZtnEhERNVujxtxMmDCh3udLS0ubU0urZDaoUYwoJKEYHlsxFOYOcpdERETUojXqyI3JZKp3ad++Pe68884G72/JkiXo3bs3jEYjjEYjMjMz8dVXX9X7mi1btmDAgAHQarVIT0/H0qVLG/MRQk60Xo0SMQoAUFF6VuZqiIiIWr5GHbnx92XeycnJeOGFF9CxY0cAUluHcePGYc+ePejRo0et7bOzszFmzBjcf//9eO+99/Dtt9/ioYceQmxsLCZOnOjX2oJFrVLAKnibZ5YWIFLmeoiIiFq6Jl8K7g9jx46t8Xj+/PlYsmQJduzYUWe4Wbp0KVJTU7Fo0SIAQLdu3bBr1y68/PLLLTbcAIBN5W2eaT0ndylEREQtXsgMKHa73fjwww9hs9mQmZlZ5zbbt2/HiBEjaqwbOXIkdu3ahaqqqjpf43A4YLVaayyhxhERDQBwlRXKWwgREVEYkD3cHDhwAJGRkdBoNJg2bRo+++wzdO/evc5t8/PzER8fX2NdfHw8XC4XCgvrDgYLFiyoMS4oJSXF75+huZxqMwDAY2O4ISIiai7Zw02XLl2wd+9e7NixAw8++CDuuusuHDx48JLbC4JQ47Honfju4vXV5syZA4vF4ltyc3P9V7yfeHTVzTPZUZ2IiKi5ZB1zAwBqtdo3oDgjIwM7d+7EP/7xD/zrX/+qtW1CQgLy8/NrrCsoKIBKpUKbNm3q3L9Go4FGo/F/4X4k6qVwo2LzTCIiomaT/cjNxURRhMPhqPO5zMxMbNiwoca69evXIyMjAxEREcEoLyBUhrYAALWzVN5CiIiIwoCs4Wbu3LnYunUrTpw4gQMHDmDevHnIysrC5MmTAUinlC6cN2fatGk4efIkZs2ahUOHDuGtt97CsmXLMHv2bLk+gl+oqptnukrlLYSIiCgMyHpa6uzZs5gyZQry8vJgMpnQu3dvfP311xg+fDgAIC8vDzk5Ob7t09LS8OWXX+LRRx/FP//5TyQlJeHVV19t0ZeBA4DOGAcAMFQ3z7zE+CEiIiK6PEEUW1craqvVCpPJBIvFAqPRKHc5AIAdh09h0ErvvD5P5ABak7wFERERhZjG/P0OuTE3rZHJaESF6B30zOaZREREzcJwEwJivM0zAcBTznBDRETUHAw3ISBaH4ESUeoqVWkpkLkaIiKilo3hJgRoVMoazTOJiIio6RhuQoRNFQ2AzTOJiIiai+EmRNirm2eWs78UERFRczDchIgqTXXzTA4oJiIiag6GmxDh1rJ5JhERkT8w3IQKvdT4U2VnuCEiImoOhpsQoTRI4UbtZGdwIiKi5mC4CRFqo9QZXFdlkbkSIiKilo3hJkRoTVLzzEiPFfB4ZK6GiIio5WK4CRGG6FgAgAIewF4qbzFEREQtGMNNiIiOikKZqJMeVHBQMRERUVMx3IQIs+F8fymRncGJiIiajOEmRJj15zuDs3kmERFR0zHchAhtBJtnEhER+QPDTQixKaMBsHkmERFRczDchBCH2gQAcJWxeSYREVFTMdyEEDbPJCIiaj6GmxBS3TxTUclwQ0RE1FQMN6FEL4UblYP9pYiIiJqK4SaEKCOl/lJsnklERNR0DDchRG2UWjCweSYREVHTMdyEEK1Rap6p95QBHrfM1RAREbVMDDchJNLXPFMEKkvlLYaIiKiFYrgJIaZIPayiXnrA/lJERERNwnATQmIMahSLUn8psYIT+RERETUFw00IMevVKPE2z7Rb2IKBiIioKRhuQohOrYRFkMINm2cSERE1DcNNiKlunulk80wiIqImYbgJMfaIaABAVTnH3BARETUFw02IcXmbZ4psnklERNQkDDchxqWT+ksJlcUyV0JERNQyMdyEGEHfBgCgsjPcEBERNQXDTYhRGaRwo3aWylsIERFRC8VwE2IioqTO4HpXqbyFEBERtVAMNyFGGx0PANB7yoFyznVDRETUWAw3IcZgjsdezxXSg22vyFsMERFRC8RwE2JiDGq87LpNerDz34DllLwFERERtTAMNyHGbFBjm6cnvvd0A9xOYMtLcpdERETUojDchJi2kWoY1Cq8VOU9erPnPaDomLxFERERtSAMNyFGo1LikaGd8KPYBVvRHxDdQNYCucsiIiJqMRhuQtA916ShS3wUXnDcIq048Alw9qC8RREREbUQDDchKEKpwHM398TPYgf8130VABHYPF/usoiIiFoEhpsQNbBDDG4dkIxXXLfADQXwy3+BUz/KXRYREVHIY7gJYXPGdEORrgM+c18jrdj0rLwFERERtQAMNyEsxqDGE6O6YpFrAqpEJXB8M5C9Ve6yiIiIQhrDTYi7LSMFcSmdsdJ9o7Ri07OAKMpbFBERUQhjuAlxCoWA+Tf3whLPeNjFCCD3e+DIBrnLIiIiClkMNy1At0QjbhrcHyvcIwEAnv89C3g8MldFREQUmhhuWoiZwzvjU92tKBN1UJzdDxz6Qu6SiIiIQhLDTQsRqVHh0d9ehWXu0QAA58bnAI9b5qqIiIhCD8NNCzKqZwJ+SbsTJWIk1CVHIe77UO6SiIiIQg7DTQsiCALmjr8Kb3rGAQAqNswHXE6ZqyIiIgotDDctTGobPaKufRAFYjQMFadR+cNyuUsiIiIKKQw3LdA9N3bHB9pJAICqTS8CzgqZKyIiIgodDDctkEalxJUTZiLXEwujqwh5GxfLXRIREVHIYLhpoQZ3ScKWpHsAAIadr8JdaZG5IiIiotDAcNOCjbjjERwX28EoluGnTxbIXQ4REVFIYLhpweJMkcjpMxMAcMWxFThXcEbegoiIiEKArOFmwYIFGDhwIKKiohAXF4fx48fj8OHD9b4mKysLgiDUWn755ZcgVR1arh13L44p0xGJSuz/8Bm5yyEiIpKdrOFmy5YtmD59Onbs2IENGzbA5XJhxIgRsNlsl33t4cOHkZeX51s6deoUhIpDj1KphGLYkwCAwUWfYuf+gzJXREREJC+VnG/+9ddf13i8fPlyxMXF4ccff8R1111X72vj4uIQHR0dwOpajrRBN+Pktr+jve0ATv/nWfTu/h40KqXcZREREckipMbcWCzSFT8xMTGX3bZfv35ITEzE0KFDsXnz5ktu53A4YLVaayxhRxDQ5rfPAgDGONfhw/XbZC6IiIhIPiETbkRRxKxZs3DNNdegZ8+el9wuMTERb7zxBlavXo1PP/0UXbp0wdChQ/HNN9/Uuf2CBQtgMpl8S0pKSqA+gqwiuwxBQexgqAU3IncsRE4RJ/YjIqLWSRBFUZS7CACYPn061q5di23btiE5OblRrx07diwEQcCaNWtqPedwOOBwOHyPrVYrUlJSYLFYYDQam113KBFP7YLw76FwiwL+3O7feP7+iRAEQe6yiIiIms1qtcJkMjXo73dIHLl5+OGHsWbNGmzevLnRwQYABg0ahCNHjtT5nEajgdForLGEKyE5A+VpI6EURFyd+wbW/Zwvd0lERERBJ2u4EUURM2bMwKeffopNmzYhLS2tSfvZs2cPEhMT/VxdyxQ56i8QIeA3yu/xwef/RbnDJXdJREREQSVruJk+fTree+89fPDBB4iKikJ+fj7y8/NRWVnp22bOnDm48847fY8XLVqEzz//HEeOHMHPP/+MOXPmYPXq1ZgxY4YcHyH0xPeAp8dEAMBdjvexaMOvMhdEREQUXLKGmyVLlsBiseCGG25AYmKib1m1apVvm7y8POTk5PgeO51OzJ49G71798a1116Lbdu2Ye3atZgwYYIcHyEkKW+cC4+gxFDlHuz+dh1e+OoXuNweucsiIiIKipAZUBwsjRmQ1KKteQTY/TZ+8HTB751zkXFFAl67ox/aRGrkroyIiKjRWtyAYgqA6x8DlBpcqTiMjzTP4dixIxj72jbsyy2VuzIiIqKAYrgJV6Zk4I4PAK0JfYUj+Eo3DynWPbh16Xas/CHn8q8nIiJqoRhuwlnHYcADW4D4XogRLfhAMx9T8F/M+XQ/Hv9kP+xVbrkrJCIi8juGm3AXkwbcux7oPQlKePBkxHt4LeI1/GfXEdz2r+04VcKZjImIKLww3LQGaj1w87+A0X8DFCqMVe7AGu1fYD39C8a+tg3bjhTKXSEREZHfMNy0FoIAXPUAMHUtEJmAjsjFWu2T6G/fgTvf+h6vZx1FK7twjoiIwhTDTWuTOgj4wxYgNRMGsQLL1AsxU/kRXv76EP7w7o8os1fJXSEREVGzMNy0RlEJwF3/Aa6aBgB4RPU5Vqj/hu8PHsO4xd/iyNkymQskIiJqOoab1koZAYx+EZjwJqDS4TrFPnypfRK6op8x7p/fYu3+PLkrJCIiahKGm9au923AfRsAcwe0w1l8pnkao1ybMf2D3Zi/9iDbNhARUYvDcENAQi/ggSyg0wio4cTf1UvxV9VyrNh6BJP//T3OlTnkrpCIiKjBGG5IojMDd6wCrn8CAHCnagM+0jyH7OxjGPvaNuzOKZG5QCIiooZhuKHzFApgyBwp5GhM6Cf8iq9185BaJrVtWPDVIVQ6OasxERGFNoYbqq3LKOCBzUBcD8SIpVipmY97hTV4a8uvGPWPbzjpHxERhTSGG6pbmyukgca9boUSHsyNWImt2lm4pnQN7lm2Df/30T6U2JxyV0lERFQLww1dmtogXSo+9h9AZAISUIj5EW8hS/ModPuWY/TCjfhi72nObExERCFFEFvZXyar1QqTyQSLxQKj0Sh3OS1HlR3Y/Taw7RWgTJoDJ0+MwRLXWJxJvw1/ubk/UmL0MhdJREThqjF/vxluqHGq7MCedyFu/TuEsjMAgHzRjGXiOCTd+AfceV03KBWCzEUSEVG4YbipB8ONn7gcwJ53UbVlISLKpZBTIEZjTeStuHrSn9AtNV7mAomIKJww3NSD4cbPXA549ryPyv+9BINdOl11TjRhf/upuPr2P0Grj5K5QCIiCgcMN/VguAkQlxPWHStQlfUy2rjOAgCKYYKl/4NIG/WINDiZiIioiRhu6sFwE2AuJw5+/S9E73oVSSgAAJQpo6G8+hHor/4DoImUuUAiImqJGG7qwXATHGW2Cmxc9Sr6n1yG9oIUchxqM9SD7oOQdj3QbgCg5tVVRETUMAw39WC4Ca4fswuw6aPFuNX2IToozvrWiwoVhMQ+QMogIPUq6TaKg5CJiKhuDDf1YLgJPqfLg39lHcbJrHdxg7ALGYpfkSDU0YjTnAakDgJSrgJSM4G2naV+V0RE1Oox3NSD4UY+J4ts+OD7HKz+MRfaitPIEH5FhuIwrtUeQ6rrJARc9FPURnuDjjfsJPUDInSy1E5ERPJiuKkHw438qtwe/O9QAT7alYuswwXwiIARNgzWHMdt8acxUPkrIs/tg+CqrPlCRQSQ1FcKPMkZQJtOQEw6x+4QEbUCDDf1YLgJLfkWO1bvPoWPduXiZFGFb333OB3+0KUcwyJPwJC/E8j9Hig/W/dOjO2kRp9tOp5fYq4AzO0BZUSQPgkREQUSw009GG5Ck8cjYkd2ET7amYuvfsqHw+UBAKiVCgzvHo/bMpJxTZtyKE99D+TuAPJ/AoqOAvbSS+9UUALmDt7Ac0XNABSVxPE8REQtCMNNPRhuQp+lsgpr9p7Gql25+Om01be+XbQOEwck49YByeebdFYUSyHHtxzzLkeBi09rXUilk05pxXaRLktPHggk9gEitAH+dERE1BQMN/VguGlZfj5jwUc7c/HZntOw2l0AAEEAeidHI9msQ3yUFvFGDeKNWsR5b+ONWkRGKKTu5cXHLgg93tuSbMDjqv1migggoZcUdJIzpMWcJr0hERHJiuGmHgw3LZO9yo11P+dj1c5cfHes6LLbG9TKWoEnLsobggxKtMM5xDpPQVP0M3DqR+DUD4DtXO0d6dtIYaedN+y06w9oTQH4hEREVB+Gm3ow3LR8ucUV2HeqFAVWB86W2aVbqx1nrdL9MkcdR2UuIdGkxZheiRjXJxG9Ii0QTu0CTv8InNoJ5O0D3M6LXiEAsV2BZO+prOSB0mOF0r8fkoiIamC4qQfDTfizOVwoKKsZeM5a7TjrXVdgtSPfaoe9ylPjdWltDfhtnySM65uE9NhIwOUA8g8Ap3ZJYefUTqD0ZO03VEdKc/Ak9vHe9pXG83DAMhGR3zDc1IPhhgBAFEVY7S7szC7G53tPY+OhszXCTu9kE37bJwm/7ZOEOOMFg4zLzwGnLwg7p3cDzvLab6COAhJ7S0Enqa902+YKHuEhImoihpt6MNxQXcodLmw4mI/P95zBtqOFcHukfxaCAAy+og3G9WmHkT0TYNJdNG+Oxw2c+0UKOXl7pVNZ+QcAl732m0QYLgo8fbwtJhh4iIguh+GmHgw3dDmF5Q58eSAPX+w9gx9Pnu+BpVYpcGOXOIzrm4QhXeOgjbhEKHG7gMLDwJm9UtjJ2ysFnqqK2ttG6KUrtBL7SKEnriugMQGaSEBtkAIRT28RETHc1Ifhhhojt7gCa/adwed7TuNIwfnTT1EaFUb2TMD4vu2QeUUbKBWXuVzc4wYKf5XCzpm93qM8+4Eq2+WLiDCcDzvqSGm53OPIWKD91ezFRURhg+GmHgw31BSiKOKX/DJ8sfcM1uw9jTOW86edYqM0GNMzAYM7tsVVaTGI1qsbtlOPW5p758LAU3wccNqkcTyi53J7qF+EAeg8Aug+Dug4XApAREQtFMNNPRhuqLk8HhG7Tpbgi72nsfZAHkorqnzPCQLQLcGIQeltkHlFG1zZIQYmfRP6W4kiUFXpDTpl0q2jvI7H1cuFj23AucOA9dT5/am0QMdhUtDpPJJz9RBRi8NwUw+GG/Inp8uDrUfOYfPhAuw4XoyjBTWvnBIEoHuiEZnpbTAovQ0GpsXUHpQcCKIoDXI++DlwaA1QcuL8c0o1kD5ECjpdRgP6mMDUUB2yCg5JTU/N7c83NeVRJCJqJIabejDcUCAVlNnx/fFi7DhehO3Hi3D8XM0xNYIA9EiqGXaM2gCHHVEE8vcDB9cAB78Aio6cf06hAtKuA7r9Fuj6G2msTmNV2aXxRAWHgHOHgIJfgIKDQGkOgEv830tU4gUNTS/o5h7dHlA18LQeEbUqDDf1YLihYCqw2rEjuxjbjxXh++NFOF5YM+woBKBnOxMGpbfBoPQYDEht4mmshhJF6dL1g19IYafg5/PPCQppEHL3cVLQMSbWfK3LKY0ROndICjLVS0n2pccHGWKlGZyjEqWwU3QUqCi8dH2C8vwRnhrhp5O0D145RtRqMdzUg+GG5HTWaseO40XepRjZhbWvlorUqBBv1CDRpEO8UYsEkwYJJh0SjFokmqQ+WW0Maigud4VWQxQeBQ59IYWdvH0XPCEAKVcBqYOkU1rnfpGCSV0NRwFAGw3EdZcuZY/rLgWauG6AoW3tbStLgKLjF3Vz9zY1re/qsQg9YO4A6GIAXbT0nlrT+fuXWqfSNPprIaLQw3BTD4YbCiV5lsoap7FOFtUxF04dIpQC4qK0SDB5lwuCT6JJixiDdGpHhHSwBhAhitJjj+i9LwKidz0AqKw5MGZ/CVP2VzCc21P3G6ujpNAS1xWI7ea93w2IjG9+93RRlDq5Xxh2qu+XnLh0sLocla524NHFSEeIYtLPLzozO8AThTCGm3ow3FAoq3C6kG+xS4vVjjyL1B/rwtvCcgcC/a82AUUYpdyJTsJpnBDjcULRHjZTR6hjUpAco0e7aD2SzTq0M+uQbNYhNlIDIZDBwF0lndYqOSEd+bFbAHspUFl6we1F6+xWXHLMT120ppph58LFEBv6wUcUpYki1Qa5KyEKCIabejDcUEtX5fbgXJmjzuBz1mJHnrUSpTbv5ekCIAAQBAEKQbqVHktPCr7nAaHGY+kPucPlQWG547I1qVUKJEefDzvtonVINut9j+OitJef6NDfPB7AYb0o8Fik+7ZzUlAqzpbmFio7U/++1JFATFrt0GPuIB21UgbhCjhACjAVxUDxMenIVvGx80e5io9LUwEk9gX6/R7oOTFwV8IRyYDhph4MN0SNY69y40xpJU6XVuJUSSVOl1TiVEmF73G+1X7ZI0kalQL9U82+gdN9U6OhUYVQTy1nhTfsHL9oyQYsubjsESBdDBAZJx3hiYy/4H6c9Lj6viG2YUHIbvGeljtWO8jYLQ37TEq1dKl/38nAFUMBpaphryMKUQw39WC4IfIvp8uDfIsdp0orLgg/lTjtfZxnsfsakVYL+bBzIZcDKDlZO/iUZEvrRXfj9lcjCHnDj8YohajqozD1XVEGAMZ20pGj6ivKYq6QbjVR0uDwPe8DZw+c3z4yHug9SQo6cV0b/x0QhQCGm3ow3BAFl8vtQXahDTuypYHT3x8vQmG5s8Y2LSrsXMjjASqLgfICwFYg3frun5MmL6y+bzvXuCAUGe8NLeneW2+QMacBav3lX5+3H9j7AXDgI6Ci6Pz6pP5Av8nSaSudufGf2d/cLsBll0Kky37Rfe+tu0r6zBqjNDZKa5TuK1rAb4T8huGmHgw3RPISRRHHzpVj+/EwDDv1qS8I2S2AMfmCIOM9CuMPLidwZL0UdI6sO3/VmVINdL1JOpqTPqT5p60qS71Hs05Ip/NKTkiDwKsq6g4sLqd029gjXxdSR50PO1pTzfBT43F1ILogGGmN0vQCwR4oLorS53aUSYsywtv8Nip4Y7daKIabejDcEIUWKezYasz/c/Eg5uqwk9HB7Jvnx2xQI8aghlmvRrQ+AhFKTvB3WeXnpCM5e96vOYFjVKL3tNXvgNgudb/W4wasp2uGlwvDjL20+fUpIqQ+aCpNzVulSuq1ZrdIV8G5Kpv/XoA0aaQm6nzg0RgvenzRc1rvOk2UdDTJYT0fUnxLXesuWn+paQ1UWm/QiZSCmybyfPCptS5Sqqn6vkonhSOlWprlW6kGlJoL1mla/JEuhpt6MNwQhbaGhJ26GLUqKewY1IjRS8HnwsdSGIqA2fucSRcR2MvXQ1l1S469HwD7P5KOKFVrlwH0ulX6A1wdYIqzpaMwnqpL7hIAYIiTriCLSZNuo9tLgeDiwKLU1A4wjfnj63J6r4SznF98j62XeGw5/9hR1rwjRn4hSJftu6sA9+V/3/55S0XN0KO6IPxUr4tKqN0WxR/zWPkBw009GG6IWpYLw85Ppy0osjlRYnOiuEK6La2satK8PyqFgDaRarQxaNAmUo22kRq0MajRJrL6cc3ntBEt+796L8nlAH5d5z1ttb7+P/qKCGnyQ3MHaeyPucP5MBPdvuU0RK2eE6g66FQHId/9C9b7HnvXVT+n0pw/uuNbqo+yXLy+jnURhvPtRNxV0j6d5d73LQec1bfl0q2jrPY6Z9n57V0OwO2svfiDOvJ84Im5omZ7FF20f96jARhu6sFwQxRe3B4RlsoqFNucvqWkwnt7QQgq9t2vQrmj8bMdR2pU3jAkBaC23tCTFK1Dilma1DApWge1qgWfHisvkI7kHN3gncW5w/mjMOY0wJjU4k9ttCqi6D0ydFHgcVXfd5x/3uWQFuvpmjOEl568dO84ANC3vSDspJ+/H5MOROj8+nEYburBcENEDpcbxTYnisqdKCx3oLDciaJyB4ps0uOicieKbN7bciec7nr+z/0CggDER2mREiNNYpjsncRQCj96JEZrAzY2yOMRUeXxtPyB1xRaXE7p9GR1K5TqeZeKjkrtUi5FpQPmnvFrs9vG/P3mrE5E1OpoVEokmnRINF3+vyxFUUSZw+ULQkW+MOREQZndN5nhqZIK2Ks8yLdKrTN2niiptS+FACQYtVLwuSAAJRi1cHk8qHC6UeFwo8LpQkVV9X03KqtcsF10v9Lphs3pQqWzer10OqltpAbpsQZcERuJK7y36bEGJJv1wZ8lmlo+lRqI7SwtF3OUSXM+FR29qBnuEWkuJj8Gm8bikRsiIj8QRRFFNidyiyu8YUcKPBfeOlwNOwIUCGqVAmltDL7gc+FtlJaXIJMfiSLgtPl9DBaP3BARBZkgCGgbqUHbSA36pdaeHM/jEVFoc9QIPrnF0m2B1QG1SgGdWgmDWgm9WgW9Wgm9WgmdWgWDWgmdd71Bo4QuQgmDRuVdp4RBLd1XKQTkFlfi2LlyHDtXjuPnbNJtoQ1OlweHz5bh8NmyWrXFRWlqBZ5ks3Rky6DhnwlqJEGQfXA5j9wQEYU5t0fEmdJKHL0w8Jwrx7FzNpwrq/8yZKNWhaRoHRJMWiSadEgyaZEYff420aQN3yvJKKRwQHE9GG6IiM6z2qukwFNQjuOF5ThWYEN2oQ1nSitR1sCrysz6CCn4REsBKDFaiySTDvFGLTQRCggAFILUdb76VoAAhcL7GN6O9d7nFRd2qfeuUykEKBUCVAoFlErpcfV6BccStQotJtwsWLAAn376KX755RfodDoMHjwYL774Irp0ucQMmV5btmzBrFmz8PPPPyMpKQmPPfYYpk2b1qD3ZLghImqYMnsV8i12nLHYkVda6bvNt9pxplRqilrhlHsyPCkA1Qg/igvDkOANQwqoFNKpw3beq9jaReuk+9GBvZINkK7QK7A6kGexI89SibNWOwqsDiSYtOiXGo0eSSYeAbuMFjPmZsuWLZg+fToGDhwIl8uFefPmYcSIETh48CAMBkOdr8nOzsaYMWNw//3347333sO3336Lhx56CLGxsZg4cWKQPwERUfiK0kYgShuBTvF197kSRRHWShfOWCq9IagSeaXnb8+W2eFyi/CIIkRR2t4jAiK8t751IkRI45JESOurX1N96xbFWt3lz9cBVLlFVLlFAPUP2j5SUF7neoUAxBu15wOPWYd20Xq084agZLPukuHD5nAhz2JHvkW6Ui7fIgW/s1a7b32Rrf4J9VQKAd0SjeibEo1+qdHomxKNtLaG1juLdjOF1Gmpc+fOIS4uDlu2bMF1111X5zaPP/441qxZg0OHDvnWTZs2Dfv27cP27dtrbe9wOOBwnD+nbLVakZKSwiM3REQtjOgNOC7Pxbce6dYbpHzr3TWfd7o8OFtmx2nvoO7TpZXS/dJKOBtwJVvbSDXaRUsDrW1OlxRmLPYGn75TqxRINGmRYNQiwaRF20gNThZVYG9uaZ0tRky6CPRNkYJO39Ro9E2OhtmgbvT3Fi5azJGbi1ksFgBATEzMJbfZvn07RowYUWPdyJEjsWzZMlRVVSEiouYljQsWLMAzzzzj/2KJiCioBEGASinA3/MUVl/JdtobeE6VVF5wvwKnSyphc7pRWO5EYbkT+05Zau0jSqNCgkkKLQlGLRJNWsSbtN4wIw28jtbX3c9MFEWcLq3E3txS7Mkpxd7cUvx02gJLZRW2/HoOW34959s2ra3hfOBJiUa3RGO9s2K73B7YnG7YHC5UOF0od7hR4XD51tmcLunWO79SgkmH/qnR6J5kbNETQobMkRtRFDFu3DiUlJRg69atl9yuc+fOmDp1KubOnetb99133+Hqq6/GmTNnkJiYWGN7HrkhIqLmEEWpxUf1Zfz5lkoYNCokmnS+QBPp50vmq9we/JJXhr25Jb7Ac7zQVms7tUqBHklGRGpU3gDjRvkFtw05IlUXtUqBnklG9E81o397M/qlRjdo0stAapFHbmbMmIH9+/dj27Ztl9324uRbnc/qSsQajQYajcY/RRIRUasjCAKi9WpE69Xo2c4UlPeMUCrQK9mEXskmTMmU1pVWSEeN9uaUYk9uCfbmlqK0ogp7ckobsD8BBo0KBu8cSgaNNGeSXq1CpEZap41Q4kShDbtzSlBSUYXdOaXYnVMKbMsGACSatOifKgWdfqlm9GwXukd3QiLcPPzww1izZg2++eYbJCcn17ttQkIC8vPza6wrKCiASqVCmzZtAlkmERGRbKL1alzfORbXd44FIP2H/cmiCuw/bYHb45EmefRO9GjwBhYpuKga1dBVFEWcKKrA7pMl2JNbgt0nS/FLvhV5FjvWHsjD2gNSTym1UoEe7Yy+wNM/1YykaHmP7lSTNdyIooiHH34Yn332GbKyspCWlnbZ12RmZuI///lPjXXr169HRkZGrfE2RERE4UoQBHRoa0CHtnVfXdyc/aa1NSCtrQETB0gHHGwOF/afsmB3jnSabE9OCYpsTu/9Ut9rE4xa9G8fjX4pZtw1uEOjQpVfP4OcY24eeughfPDBB/jiiy9qzG1jMpmg00npb86cOTh9+jTeeecdANKl4D179sQf/vAH3H///di+fTumTZuGlStXNuhScM5zQ0RE1DyiKCKnuMIXdnbnlOBQXpnvcv0Ygxo//nmYXy9lbzGT+F3qQy9fvhxTp04FAEydOhUnTpxAVlaW7/ktW7bg0Ucf9U3i9/jjj3MSPyIiIhlVOF04cMqC3Tml8Igipg/p6Nf9t5hwIweGGyIiopanMX+/5TkZRkRERBQgDDdEREQUVhhuiIiIKKww3BAREVFYYbghIiKisMJwQ0RERGGF4YaIiIjCCsMNERERhRWGGyIiIgorDDdEREQUVhhuiIiIKKww3BAREVFYYbghIiKisKKSu4Bgq26CbrVaZa6EiIiIGqr673b13/H6tLpwU1ZWBgBISUmRuRIiIiJqrLKyMphMpnq3EcSGRKAw4vF4cObMGURFRUEQBL/u22q1IiUlBbm5uTAajX7dN53H7zk4+D0HB7/n4OF3HRyB+p5FUURZWRmSkpKgUNQ/qqbVHblRKBRITk4O6HsYjUb+wwkCfs/Bwe85OPg9Bw+/6+AIxPd8uSM21TigmIiIiMIKww0RERGFFYYbP9JoNPjLX/4CjUYjdylhjd9zcPB7Dg5+z8HD7zo4QuF7bnUDiomIiCi88cgNERERhRWGGyIiIgorDDdEREQUVhhuiIiIKKww3BAREVFYYbjxk9dffx1paWnQarUYMGAAtm7dKndJYefpp5+GIAg1loSEBLnLavG++eYbjB07FklJSRAEAZ9//nmN50VRxNNPP42kpCTodDrccMMN+Pnnn+UptgW73Pc8derUWr/vQYMGyVNsC7ZgwQIMHDgQUVFRiIuLw/jx43H48OEa2/A33XwN+Z7l/E0z3PjBqlWrMHPmTMybNw979uzBtddei9GjRyMnJ0fu0sJOjx49kJeX51sOHDggd0ktns1mQ58+fbB48eI6n3/ppZfw97//HYsXL8bOnTuRkJCA4cOH+5rQUsNc7nsGgFGjRtX4fX/55ZdBrDA8bNmyBdOnT8eOHTuwYcMGuFwujBgxAjabzbcNf9PN15DvGZDxNy1Ss1155ZXitGnTaqzr2rWr+MQTT8hUUXj6y1/+Ivbp00fuMsIaAPGzzz7zPfZ4PGJCQoL4wgsv+NbZ7XbRZDKJS5culaHC8HDx9yyKonjXXXeJ48aNk6WecFZQUCACELds2SKKIn/TgXLx9yyK8v6meeSmmZxOJ3788UeMGDGixvoRI0bgu+++k6mq8HXkyBEkJSUhLS0Nt99+O44fPy53SWEtOzsb+fn5NX7fGo0G119/PX/fAZCVlYW4uDh07twZ999/PwoKCuQuqcWzWCwAgJiYGAD8TQfKxd9zNbl+0ww3zVRYWAi32434+Pga6+Pj45Gfny9TVeHpqquuwjvvvIN169bhzTffRH5+PgYPHoyioiK5Swtb1b9h/r4Db/To0Xj//fexadMmLFy4EDt37sSNN94Ih8Mhd2ktliiKmDVrFq655hr07NkTAH/TgVDX9wzI+5tWBfwdWglBEGo8FkWx1jpqntGjR/vu9+rVC5mZmbjiiivw9ttvY9asWTJWFv74+w68SZMm+e737NkTGRkZaN++PdauXYsJEybIWFnLNWPGDOzfvx/btm2r9Rx/0/5zqe9Zzt80j9w0U9u2baFUKmsl/oKCglr/ZUD+ZTAY0KtXLxw5ckTuUsJW9dVo/H0HX2JiItq3b8/fdxM9/PDDWLNmDTZv3ozk5GTfev6m/etS33NdgvmbZrhpJrVajQEDBmDDhg011m/YsAGDBw+WqarWweFw4NChQ0hMTJS7lLCVlpaGhISEGr9vp9OJLVu28PcdYEVFRcjNzeXvu5FEUcSMGTPw6aefYtOmTUhLS6vxPH/T/nG577kuwfxN87SUH8yaNQtTpkxBRkYGMjMz8cYbbyAnJwfTpk2Tu7SwMnv2bIwdOxapqakoKCjAc889B6vVirvuukvu0lq08vJyHD161Pc4Ozsbe/fuRUxMDFJTUzFz5kw8//zz6NSpEzp16oTnn38eer0ev/vd72SsuuWp73uOiYnB008/jYkTJyIxMREnTpzA3Llz0bZtW9x8880yVt3yTJ8+HR988AG++OILREVF+Y7QmEwm6HQ6CILA37QfXO57Li8vl/c3Lcs1WmHon//8p9i+fXtRrVaL/fv3r3E5HPnHpEmTxMTERDEiIkJMSkoSJ0yYIP78889yl9Xibd68WQRQa7nrrrtEUZQunf3LX/4iJiQkiBqNRrzuuuvEAwcOyFt0C1Tf91xRUSGOGDFCjI2NFSMiIsTU1FTxrrvuEnNycuQuu8Wp6zsGIC5fvty3DX/TzXe571nu37TgLZKIiIgoLHDMDREREYUVhhsiIiIKKww3REREFFYYboiIiCisMNwQERFRWGG4ISIiorDCcENERERhheGGiIiIwgrDDREREYUVhhsiIiIKKww3REREFFb+HztMXoC8Xm7VAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Will be Roridce.\n",
      "\n",
      "STAOLOLIO:\n",
      "KI a set bube to takegry.\n",
      "\n",
      "MBROKING\n",
      "My LANGANGENV KINCE:\n",
      "that dight ane away, my feans' to zormuse off Lroof is here vail; dight,\n",
      "Whiiree,\n",
      "You, will is therev the do;\n",
      "Whe now oir wans!\n",
      "Al lind teal.\n",
      "-huch courly speap; airse, why.\n",
      "Herents norfore elguls;\n",
      "Protle, demees kneoul-wou what eiich o' maits, rive ceessience poor gier; thume known,\n",
      "refter so;\n",
      "Angatt must wity ale of whith Pried by of.\n",
      "\n",
      "HKING ESTEL:\n",
      "Prisar adaid the Edwart hiin courchard ny ity to chan the whi\n",
      "CPU times: user 3min 38s, sys: 526 ms, total: 3min 38s\n",
      "Wall time: 3min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size:int=vocab_size, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 head_size:int=head_size):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = head_size\n",
    "\n",
    "        #This will be our lookup table for embeddings. We'll have an entry for each token (aka vocab size) and each embedding will... \n",
    "        #...be a vector of dimension embedding_dim.\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        self.token_position_embedding_table = nn.Embedding(num_embeddings=self.context_length, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        self.transformer_blocks = nn.Sequential(\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=4, context_length=context_length),\n",
    "            nn.LayerNorm(embedding_dim), #NEW\n",
    "        )\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        B,T = idx.shape\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        token_position_embeddings = self.token_position_embedding_table(torch.arange(T, device=device)) #(T,C)\n",
    "        x = token_embeddings + token_position_embeddings\n",
    "        x = self.transformer_blocks(x)\n",
    "        logits = self.language_model_head_linear_layer(x) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Crop idx to the max size of our positional embeddings table \n",
    "            idx_crop = idx[:,-self.context_length:]\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx_crop)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            # print('Shape of logits_last_timestep:',logits_last_timestep.shape) #confirming shape\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            # print('Shape of probs:', probs.shape) #confirming shape\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            # print('Shape of idx_next:',idx_next.shape,'and contents:',idx_next) #look at the shape and contents of idx_next\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = TransformerLanguageModel(vocab_size=vocab_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "tracked_losses = list()\n",
    "for step in range(max_iters):\n",
    "    \n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        tracked_losses.append(losses)\n",
    "        print('Step:',step,'Training Loss:',losses['train'],'Validation Loss:',losses['valid'])\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "plot_losses(tracked_losses)\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81231411-944b-4b2c-a27d-f806f6d5e492",
   "metadata": {},
   "source": [
    "The loss is now down to `Validation Loss: tensor(2.0630)` from `Validation Loss: tensor(2.0940)` during the last run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "599d910a-e582-4447-830e-81320b3ddfa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reset -f\n",
    "#| output: true"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4511334-669e-4da0-8b35-9939a81cb29c",
   "metadata": {},
   "source": [
    "# Scaling Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cf82e3f-1c99-420e-91fa-1ee56baa8b0f",
   "metadata": {},
   "source": [
    "Now that we have a fully functioning transformer network, to achieve better performance, we need to scale up. We'll be doing a bit of code cleanup and refactoring as we scale up the architecture. To make things easier to follow, I've reset the kernel so we'll be re-declaring everything again from scratch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1598a75-67f9-4ec1-a826-98f0a8e2acac",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "#Hyperparameters\n",
    "batch_size = 64 #Number of token chunks per batch #UPDATED\n",
    "context_length = 256 #Length of the token chunks. Andrej called this block size #UPDATED\n",
    "embedding_dim = 384 #The vector size of the token embeddings. Andrej used n_embed as the variable name. #UPDATED\n",
    "head_size1 = 16 #Self attention head size\n",
    "num_layers = 6 #Number of transformer block layers # NEW\n",
    "num_heads = 6  # NEW\n",
    "learning_rate = 3e-4 #UPDATED\n",
    "dropout = 0.2 #NEW\n",
    "max_iters = 5000 #Number of training iterations or steps. \n",
    "eval_interval = 500 #Number of steps between evaluating the validation set to see how our validation loss is doing. #UPDATED\n",
    "eval_iters = 200 #Number of steps to do on the validation set per each interval. We do more than 1 to get a more accurate overall valid loss\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu' #Instead of using the cpu, we'll use the GPU if it's availble.\n",
    "\n",
    "TORCH_SEED = 1337\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "\n",
    "#Dataset\n",
    "with open('input.txt','r',encoding='utf-8') as f:\n",
    "    text = f.read()\n",
    "vocab = sorted(list(set(text))) #Called chars in the video, but vocab is a more generic term. Both are correct.\n",
    "vocab_size = len(vocab)\n",
    "\n",
    "char2idx = {char:idx for idx,char in enumerate(vocab)}\n",
    "idx2char = {idx:char for char,idx in char2idx.items()}\n",
    "encode = lambda x: [char2idx[char] for char in x]\n",
    "decode = lambda idxs: ''.join([idx2char[idx] for idx in idxs])\n",
    "\n",
    "tokenized_text = torch.tensor(encode(text),dtype=torch.long)\n",
    "\n",
    "#Train / Valid split.\n",
    "train_test_split_idx = int(len(tokenized_text) * 0.9)\n",
    "train_data = tokenized_text[:train_test_split_idx]\n",
    "valid_data = tokenized_text[train_test_split_idx:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84bd3989-5fa8-4bb1-83e5-d03afbea1194",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "def plot_losses(losses):\n",
    "    train_losses = [o['train'] for o in losses if o.get('train') is not None]\n",
    "    valid_losses = [o['valid'] for o in losses if o.get('valid') is not None]\n",
    "    plt.plot(train_losses, label='Training Loss')\n",
    "    plt.plot(valid_losses, label='Validation Loss')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.title('Losses')\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aff7d057-78ff-42f1-ae34-9b3d15d67ad8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "def get_batch(split:str, batch_size:int=batch_size, context_length:int=context_length):\n",
    "    #Function to get a batch of data from the train or valid dataset\n",
    "    data = train_data if split == 'train' else valid_data\n",
    "    idxs = torch.randint(low=0, high=len(data)-context_length, size=(batch_size,))\n",
    "    x = torch.stack([data[idx:idx+context_length] for idx in idxs])\n",
    "    y = torch.stack([data[idx+1:idx+context_length+1] for idx in idxs])\n",
    "    x,y = x.to(device), y.to(device) #Send data to the GPU if available\n",
    "    return x,y\n",
    "\n",
    "@torch.no_grad()\n",
    "def estimate_loss():\n",
    "    out = {}\n",
    "    model.eval()\n",
    "    for split in ['train','valid']:\n",
    "        losses = torch.zeros(eval_iters)\n",
    "        for k in range(eval_iters):\n",
    "            x_batch, y_batch = get_batch(split)\n",
    "            logits, loss = model(x_batch, y_batch)\n",
    "            losses[k] = loss.item()\n",
    "        out[split] = losses.mean()\n",
    "    model.train()\n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1faf65c2-9646-4ef8-8db0-d5a29caa496b",
   "metadata": {},
   "source": [
    "Adding dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d3d8a56-8f8e-4ec2-9d50-21cf3d3434ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class FeedForwardNetwork(nn.Module):\n",
    "    \"\"\"A simple linear network followed by a non-linearity\"\"\"\n",
    "    def __init__(self, embedding_dim:int=embedding_dim, dropout:float=dropout):\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.dropout = dropout\n",
    "        self.ffn = nn.Sequential(\n",
    "            nn.Linear(in_features=self.embedding_dim, out_features=self.embedding_dim*4),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(in_features=self.embedding_dim*4, out_features=self.embedding_dim),\n",
    "            nn.Dropout(dropout), #NEW\n",
    "        )\n",
    "    def forward(self, x):\n",
    "        return self.ffn(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2ca6a8f-61a2-458a-8bd8-0139b8295bdf",
   "metadata": {},
   "source": [
    "Adding dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dc1c95a-3ad8-4601-bc06-ac0e45c20ec3",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class Head(nn.Module):\n",
    "    \"\"\" one head of self attention \"\"\"\n",
    "    def __init__(self, head_size:int, embedding_dim:int=embedding_dim, \n",
    "                 context_length:int=context_length, dropout:float=dropout):\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.head_size = head_size\n",
    "        self.context_length = context_length\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.key_layer   = nn.Linear(in_features=self.embedding_dim, out_features=self.head_size, bias=False)\n",
    "        self.query_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.head_size, bias=False)\n",
    "        self.value_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.head_size, bias=False)\n",
    "        self.register_buffer('tril', torch.tril(torch.ones((self.context_length, self.context_length))))\n",
    "        self.dropout_layer = nn.Dropout(dropout) # NEW\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # input of size (batch, time-step, channels)\n",
    "        # output of size (batch, time-step, head size)\n",
    "        B,T,C = x.shape\n",
    "        assert T <= self.context_length\n",
    "        assert C == self.embedding_dim\n",
    "        q = self.query_layer(x) #(B,T,C) (batch size, context length, head_size)\n",
    "        k = self.key_layer(x) #(B,T,C) (batch size, context length, head_size)\n",
    "        v = self.value_layer(x) #(B,T,C) (batch size, context length, head_size)\n",
    "        \n",
    "        #compute scores based on affinities\n",
    "        weights = (q @ k.transpose(-2,-1)) * self.head_size**-0.5 # (B,T,C) @ (B,C,T) -> (B,T,T) #FIXED ^-.5 is 1/sqrt() so need to mult, not div\n",
    "        weights = weights.masked_fill(self.tril[:T,:T] == 0, float('-inf')) #(B,T,T)\n",
    "        weights = F.softmax(input=weights, dim=-1) #(B,T,T)\n",
    "        weights = self.dropout_layer(weights) # NEW\n",
    "        \n",
    "        #perform weighted aggragation of the values\n",
    "        out = weights @ v # (B,T,T) @ (B,T,C) -> (B,T,C)\n",
    "        return out\n",
    "\n",
    "# Head()(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ec82411-5755-4aee-9319-8b82f48f327e",
   "metadata": {},
   "source": [
    "Adding dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12c64b22-5967-447f-ad0f-ecaacf635345",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, num_heads:int, head_size:int, embedding_dim:int=embedding_dim, \n",
    "                 context_length:int=context_length, dropout:float=dropout):\n",
    "        super().__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.head_size = head_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.heads = nn.ModuleList([\n",
    "            Head(head_size=self.head_size, embedding_dim=self.embedding_dim, \n",
    "                 context_length=self.context_length) \n",
    "            for _ in range(self.num_heads)])\n",
    "        \n",
    "        self.projection_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.embedding_dim, bias=True)\n",
    "        self.dropout_layer = nn.Dropout(dropout) # NEW\n",
    "        \n",
    "    def forward(self, x):\n",
    "        out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
    "        out = self.projection_layer(out)\n",
    "        out = self.dropout_layer(out) # NEW\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77c5e96c-f44c-42b2-8539-bf8288a71658",
   "metadata": {},
   "outputs": [],
   "source": [
    "#| output: true\n",
    "class TransformerBlock(nn.Module):\n",
    "    \"\"\"Transformer Block: Communication folled by computation.\"\"\"\n",
    "    def __init__(self, num_heads:int, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 dropout:float=dropout): #UPDATED\n",
    "        #embedding_dim: embedding dimension, num_heads: the number of heads that we want\n",
    "        super().__init__()\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = embedding_dim // num_heads\n",
    "        self.num_heads = num_heads\n",
    "        self.dropout = dropout # NEW\n",
    "        \n",
    "        self.multi_self_attention_heads_layer = MultiHeadAttention(num_heads=num_heads, head_size=self.head_size,\n",
    "                                                                   embedding_dim=embedding_dim, context_length=context_length, \n",
    "                                                                   dropout=dropout) #UPDATED\n",
    "        self.feed_forward_network = FeedForwardNetwork(embedding_dim=self.embedding_dim, dropout=dropout)\n",
    "        self.layer_norm_1 = nn.LayerNorm(normalized_shape=self.embedding_dim)\n",
    "        self.layer_norm_2 = nn.LayerNorm(normalized_shape=self.embedding_dim)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = x + self.multi_self_attention_heads_layer(self.layer_norm_1(x))\n",
    "        x = x + self.feed_forward_network(self.layer_norm_2(x))\n",
    "        return x\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbd3eb26-041f-4cbb-b609-14e8d72103ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step: 0 Training Loss: 4.285 Validation Loss: 4.282\n",
      "Step: 200 Training Loss: 2.393 Validation Loss: 2.414\n",
      "Step: 400 Training Loss: 2.021 Validation Loss: 2.097\n",
      "Step: 600 Training Loss: 1.774 Validation Loss: 1.912\n",
      "Step: 800 Training Loss: 1.632 Validation Loss: 1.808\n",
      "Step: 1000 Training Loss: 1.533 Validation Loss: 1.719\n",
      "Step: 1200 Training Loss: 1.465 Validation Loss: 1.671\n",
      "Step: 1400 Training Loss: 1.409 Validation Loss: 1.611\n",
      "Step: 1600 Training Loss: 1.368 Validation Loss: 1.596\n",
      "Step: 1800 Training Loss: 1.337 Validation Loss: 1.567\n",
      "Step: 2000 Training Loss: 1.309 Validation Loss: 1.55\n",
      "Step: 2200 Training Loss: 1.281 Validation Loss: 1.523\n",
      "Step: 2400 Training Loss: 1.26 Validation Loss: 1.514\n",
      "Step: 2600 Training Loss: 1.239 Validation Loss: 1.503\n",
      "Step: 2800 Training Loss: 1.227 Validation Loss: 1.509\n",
      "Step: 3000 Training Loss: 1.203 Validation Loss: 1.497\n",
      "Step: 3200 Training Loss: 1.185 Validation Loss: 1.492\n",
      "Step: 3400 Training Loss: 1.169 Validation Loss: 1.487\n",
      "Step: 3600 Training Loss: 1.151 Validation Loss: 1.484\n",
      "Step: 3800 Training Loss: 1.138 Validation Loss: 1.485\n",
      "Step: 4000 Training Loss: 1.122 Validation Loss: 1.477\n",
      "Step: 4200 Training Loss: 1.105 Validation Loss: 1.479\n",
      "Step: 4400 Training Loss: 1.09 Validation Loss: 1.487\n",
      "Step: 4600 Training Loss: 1.077 Validation Loss: 1.487\n",
      "Step: 4800 Training Loss: 1.066 Validation Loss: 1.492\n",
      "Step: 4999 Training Loss: 1.048 Validation Loss: 1.494\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGxCAYAAACeKZf2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZDElEQVR4nO3deXwU9f3H8ddmk92cmzskQLgEQW4kqHggiILQUim0HrXe2mJBq5Sq4FGtB9bWSq1Vq1URqYoVD1q8QA5R8ScoqCgiRyAcCSH3vZtk5/fHJAuBEELY7CSb9/PxmMfuzszOfLKu7tvvfOf7tRmGYSAiIiISJEKsLkBERETEnxRuREREJKgo3IiIiEhQUbgRERGRoKJwIyIiIkFF4UZERESCisKNiIiIBBWFGxEREQkqCjciIiISVBRuRCQg5s+fj81mY/369VaXIiJBTuFGREREgorCjYiIiAQVhRsRaTM+/vhjxo4dS0xMDJGRkZx55pksXbq0wT4VFRXMmjWLnj17Eh4eTkJCAhkZGbzyyiu+fXbs2MGll15K586dcTqddOrUibFjx7Jx48YGx1q0aBEjR44kKiqK6Ohoxo8fz4YNGxrs09xjiUjbEWp1ASIiAKtXr+aCCy5g8ODBPPfcczidTp588kkmTZrEK6+8wiWXXALAzJkzeemll3jggQcYNmwY5eXlbNq0ifz8fN+xJk6cSG1tLY888gjdunUjLy+PTz/9lKKiIt8+Dz30EHfddRfXXHMNd911Fx6Phz//+c+cc845fP755/Tv37/ZxxKRtsVmGIZhdREiEvzmz5/PNddcw7p168jIyDhi+8iRI9mxYwfbt28nOjoagNraWoYOHUpRURFZWVnYbDYGDRpE7969efPNNxs9T35+PklJScybN4/f/va3je6ze/duevXqxY033sjjjz/uW19WVkafPn0YNWoUixYtataxRKTt0WUpEbFceXk5//d//8fPfvYzX7ABsNvtXHHFFezZs4ctW7YAcNppp/Huu+9yxx13sGrVKiorKxscKyEhgZNOOok///nP/PWvf2XDhg14vd4G+7z//vvU1NRw5ZVXUlNT41vCw8M599xzWbVqVbOPJSJtj8KNiFiusLAQwzBIS0s7Ylvnzp0BfJedHn/8cW6//XbeeustxowZQ0JCApMnT2br1q0A2Gw2PvzwQ8aPH88jjzzCqaeeSnJyMjfffDOlpaUA7N+/H4ARI0YQFhbWYFm0aBF5eXnNPpaItD3qcyMilouPjyckJITs7Owjtu3btw+ApKQkAKKiorjvvvu477772L9/v68VZ9KkSXz//fcAdO/eneeeew6AH374gddee417770Xj8fD008/7TvW66+/Tvfu3Zus7VjHEpG2R31uRCQgjtXn5swzzyQzM5MdO3YQEREBgNfrZejQoRQWFvr63DTm1ltvZd68eZSXlxMZGdnoPsOGDSMsLIzPP/+cnTt30qdPHx588EFuu+224/5bDj2WiLQ9arkRkYBasWIFO3fuPGL93LlzueCCCxgzZgyzZs3C4XDw5JNPsmnTJl555RVfsDn99NP58Y9/zODBg4mPj2fz5s289NJLjBw5ksjISL7++mtmzJjBz3/+c/r06YPD4WDFihV8/fXX3HHHHQD06NGDP/7xj9x5553s2LGDCy+8kPj4ePbv38/nn3/uax1qzrFEpA0yREQC4IUXXjCAoy6ZmZnGmjVrjPPOO8+IiooyIiIijDPOOMP473//2+A4d9xxh5GRkWHEx8cbTqfT6NWrl3HrrbcaeXl5hmEYxv79+42rr77a6NevnxEVFWVER0cbgwcPNh577DGjpqamwbHeeustY8yYMYbL5TKcTqfRvXt342c/+5mxfPny4z6WiLQduiwlIiIiQUV3S4mIiEhQUbgRERGRoKJwIyIiIkFF4UZERESCisKNiIiIBBWFGxEREQkqHW4QP6/Xy759+4iJiTnqaKciIiLSthiGQWlpKZ07dyYkpOm2mQ4Xbvbt20d6errVZYiIiEgL7N69m65duza5T4cLNzExMYD54bhcLourERERkeYoKSkhPT3d9zvelA4XbuovRblcLoUbERGRdqY5XUrUoVhERESCisKNiIiIBBWFGxEREQkqHa7PjYiInBjDMKipqaG2ttbqUiTIhIWFYbfbT/g4CjciItJsHo+H7OxsKioqrC5FgpDNZqNr165ER0ef0HEUbkREpFm8Xi+ZmZnY7XY6d+6Mw+HQYKjiN4ZhcODAAfbs2UOfPn1OqAVH4UZERJrF4/Hg9XpJT08nMjLS6nIkCCUnJ7Nz506qq6tPKNyoQ7GIiByXYw19L9JS/moJ1DdUREREgorCjYiIiAQVhRsREZHjNHr0aG655ZZm779z505sNhsbN25stZrkIIUbEREJWjabrcnl6quvbtFx33jjDe6///5m75+enk52djYDBw5s0fmaSyHKpLul/KSgqIj33/svNk8pl175G6vLERERIDs72/d80aJF3HPPPWzZssW3LiIiosH+1dXVhIWFHfO4CQkJx1WH3W4nNTX1uN4jLaeWGz/xlh3gsu9n8NPtd1Nb67W6HBGRgDAMgwpPTcAXwzCaVV9qaqpviY2NxWaz+V5XVVURFxfHa6+9xujRowkPD2fhwoXk5+dz2WWX0bVrVyIjIxk0aBCvvPJKg+MeflmqR48ePPTQQ1x77bXExMTQrVs3nnnmGd/2w1tUVq1ahc1m48MPPyQjI4PIyEjOPPPMBsEL4IEHHiAlJYWYmBiuv/567rjjDoYOHdqif1YAbrebm2++mZSUFMLDwzn77LNZt26db3thYSGXX345ycnJRERE0KdPH1544QXAHApgxowZpKWlER4eTo8ePZg7d26La2lNarnxk7jkLgA4bTXkFx4gMamTxRWJiLS+yupa+t/zfsDP+90fxxPp8M9P2O23386jjz7KCy+8gNPppKqqiuHDh3P77bfjcrlYunQpV1xxBb169eL0008/6nEeffRR7r//fubMmcPrr7/OjTfeyKhRo+jXr99R33PnnXfy6KOPkpyczLRp07j22mv55JNPAPj3v//Ngw8+yJNPPslZZ53Fq6++yqOPPkrPnj1b/LfedtttLF68mBdffJHu3bvzyCOPMH78eLZt20ZCQgJ333033333He+++y5JSUls27aNyspKAB5//HGWLFnCa6+9Rrdu3di9eze7d+9ucS2tSeHGT0KdkZQQhYtyinL3KtyIiLQTt9xyC1OmTGmwbtasWb7nN910E++99x7/+c9/mgw3EydO5De/Mbsl3H777Tz22GOsWrWqyXDz4IMPcu655wJwxx138KMf/YiqqirCw8P5+9//znXXXcc111wDwD333MMHH3xAWVlZi/7O8vJynnrqKebPn8+ECRMAePbZZ1m2bBnPPfccv//978nKymLYsGFkZGQAZotUvaysLPr06cPZZ5+NzWaje/fuLaojEBRu/Kg4JA6Xt5zygr3AqVaXIyLS6iLC7Hz3x/GWnNdf6n/I69XW1vLwww+zaNEi9u7di9vtxu12ExUV1eRxBg8e7Htef/krNze32e9JS0sDIDc3l27durFlyxZfWKp32mmnsWLFimb9XYfbvn071dXVnHXWWb51YWFhnHbaaWzevBmAG2+8kalTp/Lll18ybtw4Jk+ezJlnngnA1VdfzQUXXEDfvn258MIL+fGPf8y4ceNaVEtrU7jxo7LQRPDspaow+9g7i4gEAZvN5rfLQ1Y5PLQ8+uijPPbYY8ybN49BgwYRFRXFLbfcgsfjafI4h3dEttlseL1N98E89D31o/Me+p7DR+xtbl+jxtS/t7Fj1q+bMGECu3btYunSpSxfvpyxY8cyffp0/vKXv3DqqaeSmZnJu+++y/Lly7n44os5//zzef3111tcU2tRh2I/qgxPAqCmJMfiSkREpKXWrFnDRRddxC9/+UuGDBlCr1692Lp1a8Dr6Nu3L59//nmDdevXr2/x8Xr37o3D4eDjjz/2rauurmb9+vWccsopvnXJyclcffXVLFy4kHnz5jXoGO1yubjkkkt49tlnWbRoEYsXL6agoKDFNbWW9h2325iaiCQoAUqbboYUEZG2q3fv3ixevJhPP/2U+Ph4/vrXv5KTk9MgAATCTTfdxA033EBGRgZnnnkmixYt4uuvv6ZXr17HfO/hd10B9O/fnxtvvJHf//73JCQk0K1bNx555BEqKiq47rrrALNfz/DhwxkwYABut5v//e9/vr/7scceIy0tjaFDhxISEsJ//vMfUlNTiYuL8+vf7Q8KN35kRJmdiO2VByyuREREWuruu+8mMzOT8ePHExkZya9+9SsmT55McXFxQOu4/PLL2bFjB7NmzaKqqoqLL76Yq6+++ojWnMZceumlR6zLzMzk4Ycfxuv1csUVV1BaWkpGRgbvv/8+8fHxADgcDmbPns3OnTuJiIjgnHPO4dVXXwUgOjqaP/3pT2zduhW73c6IESN455132uREqjbjRC7gtUMlJSXExsZSXFyMy+Xy67G/eOvvDN94F185Mxgy+0O/HltExGpVVVVkZmbSs2dPwsPDrS6nQ7rgggtITU3lpZdesrqUVtHUd+x4fr/VcuNHzjizp3tUddu7/igiIu1LRUUFTz/9NOPHj8dut/PKK6+wfPlyli1bZnVpbZ7CjR9FJnYGIM6rcCMiIifGZrPxzjvv8MADD+B2u+nbty+LFy/m/PPPt7q0Nk/hxo9ik7sCEG8UU1NdTWgz5icRERFpTEREBMuXL7e6jHap7fUCasfiktLwGjbsNoPCfN0OLiIiYgWFGz+yh4ZRaDM7ORXl7rW4GhERkY5J4cbPSuzm7XQVBQo3IiIiVlC48bOysAQA3EWagkFERMQKCjd+VuU0p2CoLdlvcSUiIiIdk8KNn9VGJptPyjRKsYiIiBUUbvwt2pyCIUxTMIiIBI3Ro0dzyy23+F736NGDefPmNfkem83GW2+9dcLn9tdxOhKFGz8Lc5nhJtydZ3ElIiIyadKkow56t3btWmw2G19++eVxH3fdunX86le/OtHyGrj33nsZOnToEeuzs7OZMGGCX891uPnz57fJCTBbSuHGz+qnYIiuybe4EhERue6661ixYgW7du06Ytvzzz/P0KFDOfXUU4/7uMnJyURGRvqjxGNKTU3F6XQG5FzBQuHGz6KTugAQ5y20uBIRkQAwDPCUB35p5pzPP/7xj0lJSWH+/PkN1ldUVLBo0SKuu+468vPzueyyy+jatSuRkZEMGjSIV155pcnjHn5ZauvWrYwaNYrw8HD69+/f6PxPt99+OyeffDKRkZH06tWLu+++m+rqasBsObnvvvv46quvsNls2Gw2X82HX5b65ptvOO+884iIiCAxMZFf/epXlJWV+bZfffXVTJ48mb/85S+kpaWRmJjI9OnTfedqiaysLC666CKio6NxuVxcfPHF7N9/8MaZr776ijFjxhATE4PL5WL48OGsX78egF27djFp0iTi4+OJiopiwIABvPPOOy2upTk0/YKf1U/BEEcZHncVDqdmzhWRIFZdAQ91Dvx55+wDR9QxdwsNDeXKK69k/vz53HPPPdhsNgD+85//4PF4uPzyy6moqGD48OHcfvvtuFwuli5dyhVXXEGvXr04/fTTj3kOr9fLlClTSEpK4rPPPqOkpKRB/5x6MTExzJ8/n86dO/PNN99www03EBMTw2233cYll1zCpk2beO+993xTLsTGxh5xjIqKCi688ELOOOMM1q1bR25uLtdffz0zZsxoEOBWrlxJWloaK1euZNu2bVxyySUMHTqUG2644Zh/z+EMw2Dy5MlERUWxevVqampq+M1vfsMll1zCqlWrALj88ssZNmwYTz31FHa7nY0bNxJWNwXR9OnT8Xg8fPTRR0RFRfHdd98RHR193HUcD4UbP4uNT8Zj2HHYaik8sJdOXU+yuiQRkQ7t2muv5c9//jOrVq1izJgxgHlJasqUKcTHxxMfH8+sWbN8+99000289957/Oc//2lWuFm+fDmbN29m586ddO1q/g/uQw89dEQ/mbvuusv3vEePHvzud79j0aJF3HbbbURERBAdHU1oaCipqalHPde///1vKisrWbBgAVFRZrh74oknmDRpEn/605/o1Mns9xkfH88TTzyB3W6nX79+/OhHP+LDDz9sUbhZvnw5X3/9NZmZmaSnpwPw0ksvMWDAANatW8eIESPIysri97//Pf369QOgT58+vvdnZWUxdepUBg0aBECvXr2Ou4bjpXDjZyF2O4W2ODqRT6nCjYgEu7BIsxXFivM2U79+/TjzzDN5/vnnGTNmDNu3b2fNmjV88MEHANTW1vLwww+zaNEi9u7di9vtxu12+8LDsWzevJlu3br5gg3AyJEjj9jv9ddfZ968eWzbto2ysjJqampwuVzN/jvqzzVkyJAGtZ111ll4vV62bNniCzcDBgzAbrf79klLS+Obb745rnMdes709HRfsAHo378/cXFxbN68mREjRjBz5kyuv/56XnrpJc4//3x+/vOfc9JJ5u/fzTffzI033sgHH3zA+eefz9SpUxk8eHCLamku9blpBfVTMJQXWPAvvIhIINls5uWhQC91l5ea67rrrmPx4sWUlJTwwgsv0L17d8aOHQvAo48+ymOPPcZtt93GihUr2LhxI+PHj8fj8TTr2EYj/X9sh9X32WefcemllzJhwgT+97//sWHDBu68885mn+PQcx1+7MbOWX9J6NBtXq/3uM51rHMeuv7ee+/l22+/5Uc/+hErVqygf//+vPnmmwBcf/317NixgyuuuIJvvvmGjIwM/v73v7eoluZqM+Fm7ty52Gy2Rq9THmr16tUMHz6c8PBwevXqxdNPPx2YAo9DuSMRAI+mYBARaRMuvvhi7HY7L7/8Mi+++CLXXHON74d5zZo1XHTRRfzyl79kyJAh9OrVi61btzb72P379ycrK4t9+w7+D+3atWsb7PPJJ5/QvXt37rzzTjIyMujTp88Rd3A5HA5qa2uPea6NGzdSXl7e4NghISGcfPLJza75eNT/fbt37/at++677yguLuaUU07xrTv55JO59dZb+eCDD5gyZQovvPCCb1t6ejrTpk3jjTfe4He/+x3PPvtsq9Rar02Em3Xr1vHMM88cs5kqMzOTiRMncs4557BhwwbmzJnDzTffzOLFiwNUafO466dgKNUUDCIibUF0dDSXXHIJc+bMYd++fVx99dW+bb1792bZsmV8+umnbN68mV//+tfk5OQ0+9jnn38+ffv25corr+Srr75izZo13HnnnQ326d27N1lZWbz66qts376dxx9/3NeyUa9Hjx5kZmayceNG8vLycLvdR5zr8ssvJzw8nKuuuopNmzaxcuVKbrrpJq644grfJamWqq2tZePGjQ2W7777jvPPP5/Bgwdz+eWX8+WXX/L5559z5ZVXcu6555KRkUFlZSUzZsxg1apV7Nq1i08++YR169b5gs8tt9zC+++/T2ZmJl9++SUrVqxoEIpag+XhpqysjMsvv5xnn32W+Pj4Jvd9+umn6datG/PmzeOUU07h+uuv59prr+Uvf/lLgKptntqoFABs5RqlWESkrbjuuusoLCzk/PPPp1u3br71d999N6eeeirjx49n9OjRpKamMnny5GYfNyQkhDfffBO3281pp53G9ddfz4MPPthgn4suuohbb72VGTNmMHToUD799FPuvvvuBvtMnTqVCy+8kDFjxpCcnNzo7eiRkZG8//77FBQUMGLECH72s58xduxYnnjiieP7MBpRVlbGsGHDGiwTJ0703YoeHx/PqFGjOP/88+nVqxeLFi0CwG63k5+fz5VXXsnJJ5/MxRdfzIQJE7jvvvsAMzRNnz6dU045hQsvvJC+ffvy5JNPnnC9TbEZjV0sDKCrrrqKhIQEHnvsMUaPHs3QoUOPOqT1qFGjGDZsGH/729986958800uvvhiKioqjrjGCPg6htUrKSkhPT2d4uLi4+7I1VxrX5nLyC0P82X0uZw6a0mrnENEJNCqqqrIzMykZ8+ehIdrmAvxv6a+YyUlJcTGxjbr99vSlptXX32VL7/8krlz5zZr/5ycnCOa3Tp16kRNTQ15eY1PdzB37lxiY2N9y6G9vVtLaKxZY4SmYBAREQk4y8LN7t27+e1vf8vChQuP6/8ADu+xXd/wdLTe47Nnz6a4uNi3HNohqrVE1E3BEFNT0OrnEhERkYYsG+fmiy++IDc3l+HDh/vW1dbW8tFHH/HEE0/gdrsb3KMP5vwah3fyys3NJTQ0lMTExEbP43Q6Az4nR0ySOVpnvKEpGERERALNsnAzduzYIwYUuuaaa+jXrx+33377EcEGzEGR/vvf/zZY98EHH5CRkdFofxurxCWbl76iqKKqvITwqNbp2yMiIiJHsizcxMTEMHDgwAbroqKiSExM9K2fPXs2e/fuZcGCBQBMmzaNJ554gpkzZ3LDDTewdu1annvuuWNOcBZortg4KgwnkTY3RXn7SFW4EZEgYvF9KBLE/PXdsvxW8KZkZ2eTlZXle92zZ0/eeecdVq1axdChQ7n//vt5/PHHmTp1qoVVHskWEkKhLQ6AkgN7rC1GRMRP6lvIKyoqLK5EglX9iM2NXb05Hm1qbqn62UXrHT5FPcC5557Ll19+GZiCTkBJaDxdavZTqSkYRCRI2O124uLiyM3NBcwxV452M4fI8fJ6vRw4cIDIyEhCQ08snrSpcBNMKhxJUAOe4uaPciki0tbVz1hdH3BE/CkkJIRu3bqdcGhWuGklnvAkqACjRFMwiEjwsNlspKWlkZKSQnV1tdXlSJBxOByEhJx4jxmFm1bijUyBArBVaAoGEQk+drv9hPtFiLSWNt2huD2zucxRih1VCjciIiKBpHDTShwu87p0pCff4kpEREQ6FoWbVhKRoCkYRERErKBw00piErsAkOAtBA14JSIiEjAKN60kvpMZbhy2GipLNceUiIhIoCjctJLoqGiKjSgAinJbfyZyERERMSnctBKbzUZRSBwApfl7rS1GRESkA1G4aUWloQkAVBZkW1yJiIhIx6Fw04oqHIkAVBcr3IiIiASKwk0rqo5IBsAo0xwsIiIigaJw04q8USkA2DUFg4iISMAo3LSikJi6KRgqFW5EREQCReGmFTlj66ZgqNYUDCIiIoGicNOKIhI6AxBbo0H8REREAkXhphXFJpujFMcaxeCttbgaERGRjkHhphXFJ3fGa9gItXkpL9IdUyIiIoGgcNOKoiLCKcAFQPGBPRZXIyIi0jEo3LSy4ropGMryNAWDiIhIICjctLLSsLopGAo1SrGIiEggKNy0skpHEgA1JTkWVyIiItIxKNy0sprIuikYStWhWEREJBAUblpZ/RQMoRqlWEREJCAUblpZqMucgsFZlWdxJSIiIh2Dwk0rc8SmARClKRhEREQCQuGmlUUl1k3BUFtgcSUiIiIdg8JNK4tNqpuCgTKMGrfF1YiIiAQ/hZtWlpDUCY9hB6CsQLeDi4iItDaFm1YW4QyjgDhAUzCIiIgEgsJNABTb4wAoz9cUDCIiIq1N4SYAysISAXAX6rKUiIhIa1O4CYAqp6ZgEBERCRSFmwCoiTCnYKBcUzCIiIi0NoWbQIium4KhQlMwiIiItDaFmwCw103BEO7WFAwiIiKtTeEmAMLjzCkYoqs1SrGIiEhrU7gJgMjEulGKvQo3IiIirU3hJgDiUroCEEUVhrvM4mpERESCm8JNACTGx1NhOAEozc+2uBoREZHgZmm4eeqppxg8eDAulwuXy8XIkSN59913j7r/qlWrsNlsRyzff/99AKs+fs6wUPJtcQCUaAoGERGRVhVq5cm7du3Kww8/TO/evQF48cUXueiii9iwYQMDBgw46vu2bNmCy+XyvU5OTm71Wk9USUgcePdTXrDP6lJERESCmqXhZtKkSQ1eP/jggzz11FN89tlnTYablJQU4uLiWrk6/yoPSwQ3uIt0WUpERKQ1tZk+N7W1tbz66quUl5czcuTIJvcdNmwYaWlpjB07lpUrVza5r9vtpqSkpMFihapwcwqG2pL9lpxfRESko7A83HzzzTdER0fjdDqZNm0ab775Jv37929037S0NJ555hkWL17MG2+8Qd++fRk7diwfffTRUY8/d+5cYmNjfUt6enpr/SlNqo00Rym2aQoGERGRVmUzDMOwsgCPx0NWVhZFRUUsXryYf/3rX6xevfqoAedwkyZNwmazsWTJkka3u91u3G6373VJSQnp6ekUFxc36LfT2lb++xHGbH2Qb2POYsDv3gnYeUVERIJBSUkJsbGxzfr9trTPDYDD4fB1KM7IyGDdunX87W9/45///Gez3n/GGWewcOHCo253Op04nU6/1HoiwnxTMORbXImIiEhws/yy1OEMw2jQ0nIsGzZsIC0trRUr8o+IBLPGmBqFGxERkdZkacvNnDlzmDBhAunp6ZSWlvLqq6+yatUq3nvvPQBmz57N3r17WbBgAQDz5s2jR48eDBgwAI/Hw8KFC1m8eDGLFy+28s9olqiE+ikYisAwwGaztiAREZEgZWm42b9/P1dccQXZ2dnExsYyePBg3nvvPS644AIAsrOzycrK8u3v8XiYNWsWe/fuJSIiggEDBrB06VImTpxo1Z/QbHEpZrhxUo23soiQyHiLKxIREQlOlncoDrTj6ZDkT9W1Xir+2IVYWwVF135KXLejj+MjIiIiDR3P73eb63MTrMLsIeTbzNaa0jxNwSAiItJaFG4CqNRuhpsKTcEgIiLSahRuAqjCkQiAp1ijFIuIiLQWhZsAcvumYMixuBIREZHgpXATQPVTMIRoCgYREZFWo3ATQLZoc5TisMoDFlciIiISvBRuAsgRmwpApEejFIuIiLQWhZsACo83w01MTYHFlYiIiAQvhZsAikk2Ryl2GcXgrbW4GhERkeCkcBNA8Umd8Ro2QvFSW65LUyIiIq1B4SaAEmIiKSAGgBKNUiwiItIqFG4CKNQeQqEtDoDSA3utLUZERCRIKdwEWEloAgCVhdkWVyIiIhKcFG4CrMJhjlLsKdYoxSIiIq1B4SbAquumYDBKNb+UiIhIa1C4CbDaKE3BICIi0poUbgIsJMacgsFRlWdxJSIiIsFJ4SbANAWDiIhI61K4CbDIhDQAYmo1BYOIiEhrULgJsJikrgDEGqVQ47G4GhERkeCjcBNgCUkpeAw7ADW6Y0pERMTvFG4CLD4qnDxiASjRKMUiIiJ+p3ATYPYQG0Uh8QCU5inciIiI+JvCjQXK6qdgKNIoxSIiIv6mcGOBCkciANXFml9KRETE3xRuLFAdkWw+KdMoxSIiIv6mcGMBo24KBrumYBAREfE7hRsL1E/B4HRrCgYRERF/U7ixgDNOUzCIiIi0FoUbC0QldAbAVVtocSUiIiLBR+HGAjHJ5hQMUVSCp9ziakRERIKLwo0FkuITqDCcAHiKNQWDiIiIPyncWCA20nFwCoa8PRZXIyIiElwUbiwQcsgUDOV5+yyuRkREJLgo3FikLMycgqGqSKMUi4iI+JPCjUWqHEkA1JRofikRERF/UrixiKZgEBERaR0KN1aJrpuCoULhRkRExJ8Ubixir5uCIVxTMIiIiPiVwo1FnPFpAERWF1hciYiISHBRuLFIZEIXAOJqC8EwLK5GREQkeFgabp566ikGDx6My+XC5XIxcuRI3n333Sbfs3r1aoYPH054eDi9evXi6aefDlC1/hWXbM4v5aAaqootrkZERCR4WBpuunbtysMPP8z69etZv3495513HhdddBHffvtto/tnZmYyceJEzjnnHDZs2MCcOXO4+eabWbx4cYArP3FJcbEUG5EAuDXWjYiIiN/YDKNtXRNJSEjgz3/+M9ddd90R226//XaWLFnC5s2bfeumTZvGV199xdq1a5t1/JKSEmJjYykuLsblcvmt7uNlGAY77u3PSbZ9HJi6mORB51tWi4iISFt3PL/fbabPTW1tLa+++irl5eWMHDmy0X3Wrl3LuHHjGqwbP34869evp7q6utH3uN1uSkpKGixtgc1mo7h+CoZ8TcEgIiLiL5aHm2+++Ybo6GicTifTpk3jzTffpH///o3um5OTQ6dOnRqs69SpEzU1NeTlNX5L9dy5c4mNjfUt6enpfv8bWqo8LBEAd5FGKRYREfEXy8NN37592bhxI5999hk33ngjV111Fd99991R97fZbA1e119VO3x9vdmzZ1NcXOxbdu/e7b/iT1CV05yCoVZTMIiIiPhNqNUFOBwOevfuDUBGRgbr1q3jb3/7G//85z+P2Dc1NZWcnIZBIDc3l9DQUBITExs9vtPpxOl0+r9wP6iJTIZSoFyjFIuIiPiL5S03hzMMA7fb3ei2kSNHsmzZsgbrPvjgAzIyMggLCwtEef4VbV5iC6s4YHEhIiIiwcPScDNnzhzWrFnDzp07+eabb7jzzjtZtWoVl19+OWBeUrryyit9+0+bNo1du3Yxc+ZMNm/ezPPPP89zzz3HrFmzrPoTTkiYq24KBo+mYBAREfEXSy9L7d+/nyuuuILs7GxiY2MZPHgw7733HhdccAEA2dnZZGVl+fbv2bMn77zzDrfeeiv/+Mc/6Ny5M48//jhTp0616k84IfVTMERpCgYRERG/aXPj3LS2tjLODcCGb79n2H9Op5YQ7PfkQYjd0npERETaqnY5zk1HFJuUitewYccLFWq9ERER8QeFGwslx0aRTwwAVZqCQURExC8UbiwU7QwlnzgASg7ssbYYERGRIKFwYyGbzUaJ3ZyCoaJQLTciIiL+oHBjsYNTMCjciIiI+IPCjcXc4eYUDN6S/RZXIiIiEhwUbixWG5UCgK1c4UZERMQfFG4sZqufgqFSoxSLiIj4g8KNxcJi66ZgcOdbXImIiEhwULixWEScOQVDTI0G8RMREfEHhRuLRSd1AcBllECNx+JqRERE2j+FG4vFJ6bgMermlCo/YG0xIiIiQUDhxmJJMRHkEQtAZeE+i6sRERFp/xRuLBblDKWgbgqG0ry91hYjIiISBBRu2oCS0AQAKgo0SrGIiMiJUrhpAyrqpmDwFOdYXImIiEj716Jws3v3bvbsOTiL9eeff84tt9zCM88847fCOhJP/RQMpQo3IiIiJ6pF4eYXv/gFK1euBCAnJ4cLLriAzz//nDlz5vDHP/7RrwV2BPVTMISU51pciYiISPvXonCzadMmTjvtNABee+01Bg4cyKeffsrLL7/M/Pnz/VlfhxASoykYRERE/KVF4aa6uhqn0wnA8uXL+clPfgJAv379yM5Wp9jjVT8FQ4RHUzCIiIicqBaFmwEDBvD000+zZs0ali1bxoUXXgjAvn37SExM9GuBHUFkfGcAXLWagkFEROREtSjc/OlPf+Kf//wno0eP5rLLLmPIkCEALFmyxHe5SpovOtmcgiHSqARPucXViIiItG+hLXnT6NGjycvLo6SkhPj4eN/6X/3qV0RGRvqtuI4iMS6BCsNJpM2NUZaLLaGn1SWJiIi0Wy1quamsrMTtdvuCza5du5g3bx5btmwhJSXFrwV2BMmucA4Y5hQMFQWagkFEROREtCjcXHTRRSxYsACAoqIiTj/9dB599FEmT57MU0895dcCO4LwMDuFtjgASvMVbkRERE5Ei8LNl19+yTnnnAPA66+/TqdOndi1axcLFizg8ccf92uBHUX9FAyVarkRERE5IS0KNxUVFcTExADwwQcfMGXKFEJCQjjjjDPYtWuXXwvsKCod5ijF1ZqCQURE5IS0KNz07t2bt956i927d/P+++8zbtw4AHJzc3G5XH4tsKPwRJjhxijdb3ElIiIi7VuLws0999zDrFmz6NGjB6eddhojR44EzFacYcOG+bXAjsKom4LBXnHA4kpERETatxbdCv6zn/2Ms88+m+zsbN8YNwBjx47lpz/9qd+K60jqp2BwVCnciIiInIgWhRuA1NRUUlNT2bNnDzabjS5dumgAvxPgiEsFINKjUYpFRERORIsuS3m9Xv74xz8SGxtL9+7d6datG3Fxcdx///14vV5/19ghNJiCwTAsrkZERKT9alHLzZ133slzzz3Hww8/zFlnnYVhGHzyySfce++9VFVV8eCDD/q7zqDnSjLDjYNqqCqGiDhrCxIREWmnWhRuXnzxRf71r3/5ZgMHGDJkCF26dOE3v/mNwk0LJMbHUmxEEmurwCjbj03hRkREpEVadFmqoKCAfv36HbG+X79+FBSoz0hLJEU7OGDEAVCuUYpFRERarEXhZsiQITzxxBNHrH/iiScYPHjwCRfVETlD7RSGxAFQpnAjIiLSYi26LPXII4/wox/9iOXLlzNy5EhsNhuffvopu3fv5p133vF3jR1GWWgC1EBlYbbVpYiIiLRbLWq5Offcc/nhhx/46U9/SlFREQUFBUyZMoVvv/2WF154wd81dhj1UzDUlGgKBhERkZZq8Tg3nTt3PqLj8FdffcWLL77I888/f8KFdUTVkclQoSkYRERETkSLWm6kddRPwRCqKRhERERaTOGmDbG7zFGKHe48iysRERFpvywNN3PnzmXEiBHExMSQkpLC5MmT2bJlS5PvWbVqFTab7Yjl+++/D1DVrccZa4abqOp8iysRERFpv46rz82UKVOa3F5UVHRcJ1+9ejXTp09nxIgR1NTUcOeddzJu3Di+++47oqKimnzvli1bcLlcvtfJycnHde62KCqxfgqGIvDWQojd2oJERETaoeMKN7GxscfcfuWVVzb7eO+9916D1y+88AIpKSl88cUXjBo1qsn3pqSkEBcX1+xztQexSWl4DRt2mxcqCiC6/Qc2ERGRQDuucNPat3kXFxcDkJCQcMx9hw0bRlVVFf379+euu+5izJgxje7ndrtxu92+1yUlJf4pthUkuaLIJ4ZkSvCW5hCicCMiInLc2kyHYsMwmDlzJmeffTYDBw486n5paWk888wzLF68mDfeeIO+ffsyduxYPvroo0b3nzt3LrGxsb4lPT29tf6EE5YY7SCvbgqGsgIN5CciItISNsMwDKuLAJg+fTpLly7l448/pmvXrsf13kmTJmGz2ViyZMkR2xpruUlPT6e4uLhBn522Yu295zCSr8k+bx5po66xuhwREZE2oaSkhNjY2Gb9freJlpubbrqJJUuWsHLlyuMONgBnnHEGW7dubXSb0+nE5XI1WNqysrBEANyagkFERKRFWjxCsT8YhsFNN93Em2++yapVq+jZs2eLjrNhwwbS0tL8XJ01qpxJUK0pGERERFrK0nAzffp0Xn75Zd5++21iYmLIyTF/0GNjY4mIiABg9uzZ7N27lwULFgAwb948evTowYABA/B4PCxcuJDFixezePFiy/4Of6qJTIYyoCzX6lJERETaJUvDzVNPPQXA6NGjG6x/4YUXuPrqqwHIzs4mKyvLt83j8TBr1iz27t1LREQEAwYMYOnSpUycODFQZbeu+ikYKjUFg4iISEtYflnqWObPn9/g9W233cZtt93WShVZL9TVCQCnW6MUi4iItESb6FAsBznjzVGKozUFg4iISIso3LQxUUnm3WIx3hIo3GltMSIiIu2Qwk0bE5eQzEe1g8wXKx6wthgREZF2SOGmjUmOcfKnmsvwGjb45j+w90urSxIREWlXFG7amORoJ+UJ/XnTe5a5Ytk90DYGkRYREWkXFG7aGJvNxg2jevHX6p/jJgx2roGty6wuS0REpN1QuGmDpp7aFXd0V16oGW+uWHYPeGutLUpERKSdULhpg8LD7Fx7dg+erPkJJbZoOLAZNr5sdVkiIiLtgsJNG/XLM7pjOOP4m+cic8XKB8FTYW1RIiIi7YDCTRvlCg/jF2d046XacewP6QSl2fDZP6wuS0REpM1TuGnDrjurJ9idPFj1c3PFx3+DMs05JSIi0hSFmzYsxRXO1OFd+K/3DHY6TgZPKXz0iNVliYiItGkKN23cr0adBLYQZpfVtd6sfx7yt1tblIiISBumcNPG9UyKYsLAVNZ6B/Bd1BngrYEP77O6LBERkTZL4aYdmHbuSQD8rmgKhi0Evnsbdq+zuCoREZG2SeGmHRjcNY6zeieyubYrGxImmis/uEvTMoiIiDRC4aaduPHc3gDckjsRIzQCdn8G3y+1uCoREZG2R+GmnTirdyIDu7jIqo7j89TLzJXL/wC11dYWJiIi0sYo3LQTNpvN13ozc++5GBGJkL8NvlxgcWUiIiJti8JNO3LhwFR6JEaytzKMz7rdYK5cNRfcpdYWJiIi0oYo3LQj9hCbOe4NcHvmMIyEXlB+AD59wuLKRERE2g6Fm3ZmyqldSIp2klVSy//1nGGu/PRxKM2xtjAREZE2QuGmnQkPs3Pd2T0BuPuHkzC6ZEB1hXl5SkRERBRu2qPLz+hGjDOUrQfKWX/yTHPllwvgwBZrCxMREWkDFG7aIVd4GJef0R2Ah7+Lh34/BsMLy++1tjAREZE2QOGmnbr2rB447CF8sauQb/rdAjY7bHkHdn5idWkiIiKWUrhpp1Jc4Uwd3hWAxzYCw68yNyy7W9MyiIhIh6Zw0479alQvbDZY8X0uW/tPh7Ao2PsFfPum1aWJiIhYRuGmHeuZFMXEgWkAPLmuDM662dzw4X1Q47GwMhEREeso3LRz0841B/Vb8tU+9p5yHUSlQOFOWP+8tYWJiIhYROGmnRvUNZazeydR6zV49v9yYcxsc8PqP0FVsbXFiYiIWEDhJgjUt968ui6Lgr6XQtLJUFkAH8+ztjARERELKNwEgbN6JzKoSyxV1V7mf7YHzr/P3PDZk1C819riREREAkzhJgjYbDZf682CtTsp73EBdDsTaqpg5YMWVyciIhJYCjdB4sKBqfRIjKSooppX1++BcfebGzb+25yaQUREpINQuAkS9hAbvxpltt48t2YHntRT4azfmhv/+1uNfSMiIh2Gwk0QmXJqF5JjnOwrrmLJV/vMvjenXmXOO7X4Bti6zOoSRUREWp3CTRAJD7Nz7Vk9Afjn6u14DeDHj8GAKeCthkVXwK5PrS1SRESklSncBJnLz+hGjDOUrbllfPh9LoTY4af/hD7joKYS/n0x7NtgdZkiIiKtRuEmyLjCw7j8jO4APLVqG4ZhQKgDLl4A3c8CTyksnAoHtlhcqYiISOtQuAlC157VA0doCF9mFbFuZ6G5MiwCLnsVOg+DinxYcJE5TYOIiEiQsTTczJ07lxEjRhATE0NKSgqTJ09my5ZjtyisXr2a4cOHEx4eTq9evXj66acDUG37keIKZ+qpXQGz9cYn3AWXL4bkflCaDQsmQ2mONUWKiIi0EkvDzerVq5k+fTqfffYZy5Yto6amhnHjxlFeXn7U92RmZjJx4kTOOeccNmzYwJw5c7j55ptZvHhxACtv+349qhchNli55QDLv9t/cENUIlzxFsR1h8JMM+BUFFhVpoiIiN/ZDMMwrC6i3oEDB0hJSWH16tWMGjWq0X1uv/12lixZwubNm33rpk2bxldffcXatWuPeY6SkhJiY2MpLi7G5XL5rfa26IH/fce/Ps4kMcrBe7eMIjnGeXBjQSY8fyGU5UCX4XDl2+CMsa5YERGRJhzP73eb6nNTXGzOYp2QkHDUfdauXcu4ceMarBs/fjzr16+nurr6iP3dbjclJSUNlo5i1vi+9EuNIb/cw22vf0WDHJvQE658CyISYO8X8MplUF1pWa0iIiL+0mbCjWEYzJw5k7PPPpuBAwcedb+cnBw6derUYF2nTp2oqakhLy/viP3nzp1LbGysb0lPT/d77W1VeJidv106DEdoCCu3HGDhZ7sa7pByCvxyMThiYOca+M81UHtkQBQREWlP2ky4mTFjBl9//TWvvPLKMfe12WwNXte3SBy+HmD27NkUFxf7lt27d/un4Haib2oMd1zYD4AHlm5mW25pwx26nAq/eBVCw+GHd+HNaeCttaBSERER/2gT4eamm25iyZIlrFy5kq5duza5b2pqKjk5De/wyc3NJTQ0lMTExCP2dzqduFyuBktHc/WZPTinTxLuGi+3LNqIp8bbcIceZ5vj4ISEwqbX4Z1Z0Ha6YomIiBwXS8ONYRjMmDGDN954gxUrVtCzZ89jvmfkyJEsW9ZwjqQPPviAjIwMwsLCWqvUdi0kxMZffj6EuMgwNu0t4bHlPxy508njYcozgA3WPw/L7w10mSIiIn5habiZPn06Cxcu5OWXXyYmJoacnBxycnKorDzYsXX27NlceeWVvtfTpk1j165dzJw5k82bN/P888/z3HPPMWvWLCv+hHajkyuch6cMAuDp1dv5bEf+kTsNnAqT5pnPP5kHa/4asPpERET8xdJw89RTT1FcXMzo0aNJS0vzLYsWLfLtk52dTVZWlu91z549eeedd1i1ahVDhw7l/vvv5/HHH2fq1KlW/AntyoUD07g4oyuGAb977SuKKxvpPDz8arjgfvP5h/fB588GtEYREZET1abGuQmEjjTOTWPK3DX86PE17Mqv4KKhnfnbpcMa33HFA/DRn83nP30GhlwSuCJFREQO027HuZHWF+0M5bFLhmIPsfH2xn28vXFv4zuOuRNO+7X5/K0b4fulgStSRETkBCjcdECndotnxpjeANz11ib2FFYcuZPNBhc+DEMuA6MWFv0Sls6CysIAVysiInJ8FG46qJvO683Q9DhKq2r43WtfUett5OpkSAj85AkY+kswvLDuWfh7BmxYCF7vkfuLiIi0AQo3HVSoPYR5lwwl0mHn/zILeOajHY3vaA+Fyf+AK5dAUl+oyIO3p8Pz42DfxoDWLCIi0hwKNx1Yj6Qo7p00AIC/LtvCpr3FR9+517lw4ycw7gFwRMOedfDMaFj6O12qEhGRNkXhpoP7eUZXxg/oRHWtwW9f3UClp4mpF+xhcOZNMGMdDPwZYMC6f8Hfh8OXC3SpSkRE2gSFmw7OZrPx8JTBpMQ42X6gnLnvbj72m1yd4WfPwVX/g+R+UJEPS26C5y6AfRtav2gREZEmKNwI8VEO/vLzIQAsWLuLld/nNu+NPc+BaR/DuAfNmcX3rodnxsD/boWKglasWERE5OgUbgSAUScnc81ZPQD4/etfkVfmbt4b7WFw5gzzUtWgnwOGOTfV34fDF/N1qUpERAJO4UZ8br+wHyd3iiavzMMdi7/muAavdqXB1H/B1UshpT9UFsB/fwv/Ggt7v2i9okVERA6jcCM+4WF2/nbpMBz2EJZvzuXlz7OO/abD9Tgbfv0RjJ9rXqra9yU8O9YMOrpUJSIiAaBwIw2ckubitgv7AnD//75j+4Gy4z+IPQxG/gZuWg+DLwEM8xLV30+Fdc9BbSMTdoqIiPiJwo0c4dqzenJW70Sqqr3cumgj1bUt7DcTkwpTnoGr34GUAeZ4OEtnwhMZsPEV8DZx27mIiEgLKdzIEUJCbPzl50OIjQjj6z3F/G351hM7YI+zzEtVEx6BqGQo3AlvTYMnz4BNi9XpWERE/ErhRhqVFhvBQz8dBMCTq7axbucJ9pexh8Lpv4bffgXn3wsR8ZD3A7x+LfzzHNj8PzieDswiIiJHoXAjR/WjwWlMPbUrXgN++8qGlvW/OZwjCs6+FX77NYyeA04X7N8Eiy43p3PYukwhR0RETojCjTTp3p/0p2dSFPuKq5j8j09YuaWZA/wdS7gLRt9utuSc8zsIi4LsjfDvn8Fz42DHav+cR0REOhybcVyDmbR/JSUlxMbGUlxcjMvlsrqcdiGvzM20l75g/a5CQmwwe8IpXH9OT2w2m/9OUp4HHz9mzlVVU2Wu63EOjLkTuo/033lERKRdOp7fb4UbaRZPjZd73t7Eq+t2AzDl1C489NNBhIfZ/Xui0hxY86h563itx1x30lg4707oMty/5xIRkXZD4aYJCjctZxgGC9bu4o//+45ar8HQ9Dj+ecVwOrnC/X+yot2w5i+wYSF4a8x1fSfCmDmQOsj/5xMRkTZN4aYJCjcn7pNtefzm319SXFlNJ5eTZ67IYEh6XOucrCATVj8CX78KRt0t4/0vglG/h04DwZ+XxkREpM1SuGmCwo1/7Mov5/oX17M1twxHaAiPTB3M5GFdWu+EeVth1VzY9AZQ95WN7QYnjYHeY6HnKPP2chERCUoKN01QuPGf0qpqbnl1Ix9+b95B9etze3Hb+H7YQ1qxNWX/t7DqYfjhvYN9cgBsIdAlA046zww7nU81x9YREZGgoHDTBIUb/6r1Gjz6wRaeXLUdgDF9k/nbZcNwhYe17ok9FbDrE9i+ArZ9CHlbGm53xkKvUWZn5JPOg/jurVuPiIi0KoWbJijctI63N+7ltte/xl3j5aTkKP511Qh6JkUFroDiPWbQ2b4Ctq+EqqKG2xN7myHnpLHmzOXO6MDVJiIiJ0zhpgkKN63n6z1F/GrBF+SUVOEKD+Ufl5/KOX2SA1+Itxb2bYTtH5phZ/fnYBwySWdIGKSfDr3Pg94XmHdfqWOyiEibpnDTBIWb1pVbUsWvF37BhqwiQmxw14/6c81ZPfw74N/xqiqGzDVm2Nn2IRTtarg94SQYOAUGToWUU6ypUUREmqRw0wSFm9ZXVV3LnW9uYvGXewC4OKMr908eiDPUzwP+tVT+9kMuYa04OCIyQEp/M+gMmAKJJ1lXo4iINKBw0wSFm8AwDIPnPs7koXc24zVgePd4nv7lcJJjnFaX1pC7FLa8B5sWw7bl4K0+uC1tqNmaM+CnEJduWYkiIqJw0ySFm8Ba/cMBZrz8JaVVNaTFhvPslRkM7BJrdVmNqyyE75eaQWfH6ob9dNJPN4NO/8kQ08myEkVEOiqFmyYo3ATejgNlXL9gPTsOlOOwh3DNWT2YcV5vYlr7dvETUXYANr8Nm940bzmvHzjQFmLebTVgijlScmSCpWWKiHQUCjdNULixRnFlNb97bSPLN5sD/iVFO/jduL5cnJHeuoP++UPJPvj2LbNFZ+/6g+tDQqHXGLOPzskXmiMk664rEZFWoXDTBIUb6xiGwcotuTzwv83syCsHoF9qDPf8uD9n9k6yuLpmKtxpTgGx6Q3Y/03DbaEREJ0MUUdZDt0WmQghbaSDtYhIO6Bw0wSFG+t5arws/GwX85b/QEmVOeP3Bf07MWfiKYEd+O9EHfgBvn3DbNHJ++E432wzA05UMkQlQXSK+TwmFZL7mUtcdwgJaZXSRUTaG4WbJijctB2F5R7mLf+Bhf+XRa3XIMxu4+ozezDjvD7ERrTh/jiNcZdCeR6UH2i4lB36Og/Kc6GiAF8fnqaERULSyebt6Sn9IPkU8zE2XZe/RKTDUbhpgsJN27N1fyn3L93MRz8cACAhysHMC07m0hHphNqDsOWitgYqCxoPQMW7Ifd7syWo1t34+x3RkNzXHHCwPvAknwKuzgo9IhK0FG6aoHDTdpn9cb5j+wGzP07fTjHc/eP+nN2nnfTH8afaGijMhNzNcOD7g495WxuOxXMoZ+zB0JN0snmJKyYVojuZizNG4UdE2i2FmyYo3LRt1bVe/v3ZLh5bvpXiSvNH/PxTUpgz8RR6JWuyS2qrzRGWD2w2W3gObDaDT/72huPyNCYs0uzbE51qjtVTH3piUs110Snmc3V2FpE2SOGmCQo37UNRhYd5y7fy0me7qPUahIbYuOrMHtw8th32xwmEGjfkbzvYwpO/HcpyoSwHSveDp7T5x7LZ6zo3d4KYzuY0FEl9ILGP2SIUlaQWIBEJOIWbJijctC/bckt5cOlmVm4x++PER4Yxc1xfLgvW/jitxVMOZfvNoFNWt5TmNAxAZfvNfj/H6uwcHnsw6CT1Pvg8oSeEtrHpNUQkaCjcNEHhpn1a/cMBHvjfd2zNLQOgV1IUvzi9Gz8d1oXEaP2g+k1tTV0n57qwU7wb8rZB/lazv09RFkcNP7YQ8/b1pLqwk9j74POoZLX2iMgJaTfh5qOPPuLPf/4zX3zxBdnZ2bz55ptMnjz5qPuvWrWKMWPGHLF+8+bN9OvXr1nnVLhpv2pqvbzyeRZ/XfYDhRVmf5wwu43zT+nExSPSGdUnue2PdtzeVVdCwQ4z6ORtPRh68rY2fekrLAoi4sxOzY5o89EZA04XOA99XbfO0cg6Z7RahkQ6sOP5/Q4NUE2NKi8vZ8iQIVxzzTVMnTq12e/bsmVLgz8sOTm5NcqTNibUHsIVI3tw0bAuvL1xH6+t2803e4t5d1MO727KIS02nJ8N78rFGemkJ0RaXW5wCouATgPM5VCGYbb05G01b2PP33Yw/BTugupyczlRdod5WSw8ru7xkCWikXXhcQ33DXUc/di11WZ4q6kyH6sroaYSqquO8lgJ3hozeEUmQERC3WO8+dyh76CIVdrMZSmbzdbslpvCwkLi4uJadB613ASX7/aV8Nr63by5Ya/v7iqAM09K5JIR6YwfkEp4mO78sVR1FZTsNQc6bLCUmI+essbX+5Yy/wQjMKfICI8FRxTUehqGmWPdbXbc5wo3Q05E/MHQc2j4qQ9EEfHmEuo0F7sT7GF1zx26nCdSp9203LTUsGHDqKqqon///tx1112NXqqq53a7cbsPDoZWUlISiBIlQPp3dnHvTwZwx4R+fPDdfv6zfjcfb8vj0+35fLo9H1d4KJOHdeHijHQGdom1utyOKSzcvOPqRNTW1IWgEqgqPvpSWdT4enexeZyaSiirPPb5QiPMuo/6GG7eWh8SWnfeAqgsNEefriwwW3RqqqB0n7mciJD6oBNmBp9Qhxl6Dg9BdsfB5w0enYcEp8O2hYYfZX+HeV57mDksQEio+TokFOyhda/r19kVwJrLMMwgXR/oPeV1odpW9xk289FWdzPFoetqq83vXI277vGQ59WHvW6wz2HrvTVgeA9bjEbWHWOfyCS49l2LPuh2Fm7S0tJ45plnGD58OG63m5deeomxY8eyatUqRo0a1eh75s6dy3333RfgSiXQwsPs/GRIZ34ypDO7Cyp4/Ys9vP7FHvYWVbJg7S4WrN3FgM4uLhmRzkVDuhAbqdvJ2xV7qHnZKSKuZe/31jYMRp5y80c8rJHwEuo8sR9rwzB/uBoEnkMeKwsOhqD69VVF5g9Lrcf8cWlQezV4jjJwY1sREnr0BQ7+4GE08mNoNLGt/n2Y/6wc0Wbfq/o+WS15bbOZ3wej9pBH72Gvm1rvNR+ry81WRU9Z3WNdWPGtKz1k2yH71P89wS7a2oaEdnVZqjGTJk3CZrOxZMmSRrc31nKTnp6uy1IdQK3X4JNteSxav5tl3+7HU2v+R8URGsKEgalckpHOGb0SCVEnZGlLvLVmyKn1QE3dY6277v/M3Ydsq1tX6z7suefguhr3YeuOY5u3puFSW+3/S3cdWViUGbxsdsxgZzTzkcbX28PMlrhQ58GQ7nt9yGN9y+MR2yIOttjZQg5ZbIe9bmxpZJ9QJ3TN8OtHFvSXpQ51xhlnsHDhwqNudzqdOJ26w6IjsofYGHVyMqNOTqag3MNbG/by2vrdfJ9Tytsb9/H2xn10iYtg4qBULhyYxrD0OAUdsV6IHUIizJaKtsYwjgw83lqzdanB65qD67Ad/UfQt+0Y+4B5SbG+D1ajLSPHel33iGEGihD7YY8h5qMt5Mh1Dfatq8sR1fKWpLAo89jSatp9uNmwYQNpaWlWlyFtXEKUg2vP7sk1Z/Xg6z3FLFq/m/9u3MfeokqeXZPJs2sySXWFM35AJy4cmMZpPRN0W7nI4Wy2ur4/7fSyrmGof1AHYWm4KSsrY9u2bb7XmZmZbNy4kYSEBLp168bs2bPZu3cvCxYsAGDevHn06NGDAQMG4PF4WLhwIYsXL2bx4sVW/QnSzthsNoakxzEkPY67f9SfVVtyeXdTDiu+zyWnpIoX1+7ixbW7SIxyMK4u6Jx5UiJhGg1ZpP1TsOkwLA0369evb3Cn08yZMwG46qqrmD9/PtnZ2WRlZfm2ezweZs2axd69e4mIiGDAgAEsXbqUiRMnBrx2af8iHHYmDEpjwqA0qqpr+WRbHu9uymHZd/vJL/fwyue7eeXz3bjCQzm/fycmDEzjnD5JurVcRKSNazMdigNF49zIsVTXevlsRz7vbsrhg29zyCvz+LZFOeyM6ZfChIFpjOmXTKSj3V/ZFRFpF9rN9AtWULiR41HrNVi/s4B3N+Xw/rc5ZBdX+bY5Q0M49+RkJgxKZewpnXCFt9N+CCIi7YDCTRMUbqSlvF6Dr/YU8V7ddA9ZBRW+bfYQG/3TXGT0iGdEjwQyesSTEhNuYbUiIsFF4aYJCjfiD4Zh8F12iS/obKubrfxQPRIjyeiRwGl1YadnUhQ2dWgUEWkRhZsmKNxIa8gurmTdzkLW7yzg88wCtuwv5fB/s5KiHWR0T/C17gzo7CJUd2GJiDSLwk0TFG4kEIorq/lyVyHrdhawfmchG/cU4alpOOx6pMPOsG5xZHRP4LSeCQxNjyPKqQ7KIiKNUbhpgsKNWKGqupZNe4tZt7M+8BRQUtVwDiF7iI1T0mLon+aiX6qLfmkxnJLqIj7KYVHVIiJth8JNExRupC3weg225pbxeV3QWZdZwL5D7sQ6VKor3Aw6aS76pZqPvZKidElLRDoUhZsmKNxIW7W3qJKvdhfxfXYJm3NK+T6nhN0FlY3u67CH0KdTNP1SXZxySPBJjNY8aiISnBRumqBwI+1JSVU1P+SUmmEnu4TN2SVsySml3NP47MzJMU5f607v5GhOSonipORo4iJ1aUtE2jeFmyYo3Eh75/Ua7CmsZHOOGXa+zzZbeXYVVBxxh1a9pGgHvZKjOSk5mpOSo+idYj7vEhehmdBFpF1QuGmCwo0Eq3J3DVv2l/J9dik/7C9l+4EytueWHbUvD0B4WAg9k6Lrwo7ZytM7JZqeSVGaQ0tE2hSFmyYo3EhHU+6uYceBcjPsHChjW675mJlXTnVt4//622zQNT7CDDvJ0fTpZIae3skxxEZqmgkRCTyFmyYo3IiYamq97C6sZHtd2KkPPttyy464Tf1QSdFO+qTUhZ1DlpQYp0ZgFpFWo3DTBIUbkaYZhkF+uccXdOpberblljWYOPRwMeGhda07Ztjp08ls6ekar349InLiFG6aoHAj0nKlVdVsP1B+RPDZlV+O9yj/JXGGhtAzKYpuCZF0iY+ga3wkXeMj6pZIYiN0mUtEjk3hpgkKNyL+V1Vdy878hqFnW24ZO/LKj5h24nAx4aFHBB6FHxE5nMJNExRuRAKn1muwu6CCHXll7C2sZI9vqWBPYSX55Z5jHiPGGXpEi0+XuAjfuvjIMPX1EekAjuf3W7P0iUirsYfY6JEURY+kqEa3V3hqDgk9FY2Gn1J3Dd/nlPJ9Tmmjx4gIs9OlQeAxn5uPkaTEONXnR6SDUbgREctEOkLp0ymGPp1iGt1e4alhX1Elu+tDT0EFe4oq2VtYyd6iSg6UuqmsrvVdBmtMmN1G57i68FMXgDrHRRAXEUZsRBixkWG4ws3nkQ67WoFEgoDCjYi0WZGOUHqnxNA7pfHwU1Vdy74iM+jUB576lqC9RZVkF1dSXWuwK7+CXfkVxzxfaIgNV0QYrvBQYiPCzOd1Iag+ALkizG2xEWHERThIT4jQ9BYibYzCjYi0W+FhdnolR9MrObrR7TW1XnJKqsywc0j4yS6poriympK6pbiymhqvQY3XoKDcQ0Ez+gIdKjYijB6JkXRPjDr4mGQ+JkY51BokEmAKNyIStELtIXUdkSOb3M8wDCqra+sCT40v+BTXLSVV1Q23VZnb88s9HCh1U1xZzVd7ivlqT/ERx452htI9MZIeiVENHrsnRqk/kEgrUbgRkQ7PZrMR6Qgl0hFKWuzxvbfCU0NWQQU78yrYlV/OznzzcVd+BfuKKylz1/DtvhK+3VdyxHvDw0LonhBFt8RIOseGk+IKp5MrnE4up/kYE44rIlQtPyLHSeFGROQERDpC6Zfqol/qkbemVlXXsqfQDD476wJP/ePeokqqqr1s2V/Klv2N3wkGZgCqDzop9aGn7jElJpzUWPN1pEP/ORepp38bRERaSXiY/agdoqtrvewtrGRnfjlZBRXsL6lif4m77tF8XlxZTVW1t1kdomOcoaS4nCTHOEmOCSc52mm+jq5fZy4JkQ5dCpOgp3AjImKBMHtIk2MAgdnyk1viZn/pwcCTW1JFTl0Ayi1xk1NSRYWnllJ3DaUHath+oLzJ89pDbCRGOQ4GnsPCT/3rTq5wopz6iZD2Sd9cEZE2KjzMTrfESLolNt0husxdY4af4ioOlLk5UOo2H0vcB1+Xuimo8FDrNcgtdZNb6j7m+etbg1JjzctinWLD6RRjvq7vH5QS4yTMHuKvP1nELxRuRETauWhnKNHJ0Zx0lFvi61XXeimou8PLtxwSfupf7z+O1iCbDRKjHHV9gQ52iE6te54Q5SA+0kFcVBgxTnWOlsBQuBER6SDC7CG+AHIsZe4acoqryC2pYn9pFTnFh/YHqrtEVlpFda1BXpmHvDJPo3eEHSo0xEZcpIP4yDDio+oeIx3ERTpIiAqr29bweWxEGHb1EZLjpHAjIiJHiHaG0jslmt4pR28N8noNCis85BzS/+fQ8LO/pIrCcg+FFdVUVtdS4zXIK3OTV3bsS2L1bDZzkMTEKAdJ0U6S6voFJUXXva5bV/86PMzujz9f2jmFGxERaZGQEBuJ0U4So50M6Nz0vlXVtRRWeCgsr6aowkNBhRl6isrN50UV1eb2iuq6QOShtKoGw4CiimqKKqqP2VkazH5Ch4adgwHIfJ0S4yTFZd5N5ghVX6FgpXAjIiKtLjzMTlpsBGmxEc1+T3Wt1xd6zBYfD3mlbl/rj3k5zF23zoOn1mv2E3LXkJl37CCUEOXwhZ2UGDP41HeS9q1zOXGGqjWovVG4ERGRNinMHuK7Rf3ko8wcX88wDEqqahqEnYMhyM2BUg8H6rbV9xWqn0fs+5yjD6IIEBcZ5gs+yTFOUmLMx/rWocRoB4lRThKiHOof1EYo3IiISLtns9l8s7Uf664xr9egqLKa3NKDYwfllh583O977cZT13pUVFHND/vLjlEDJEQ6SPSFHqdvTKHEKAeJh/UVinCoRai1KNyIiEiHEhJiIyHKQUKUg36pR9/PMAyKK6sPBp66ARVzS8zWoPwyD/nlZitRYYUHw4D8cg/55Z5jBiGASIed+EgH8VHmXWPxDe4kcxAXGXbwVvq6O8siHXbdTt8MCjciIiKNsNnMW9fjIh3HvCxWU+ulsKLaF3rqL4fll5v9hPLLPb5tB8rceGq8VHhqqfBUsreostk1OUJDfLfQ1wejuEgHcRFhuCLCcIWbrVeuiNBDnocREx7aoQZbVLgRERE5QaGH9A86FsMwKHPX+Fp8iio8FNTdRXb4HWNFFdUUlJuPnlovnhpv3W32zb+dvl6kw26GnXAz/Bx8HoYrPBRXRJh5R9khgzC21yk42mfVIiIi7ZTNZiMmPIyY8DB6cvS5xQ5lGAYVnoO30xfWB6G6cYSKK6spqaqmpLKm7rFuqaqhzF0DUNdSVEt2cVWza42un4KjfrqNQ57Xz06fHNP27ihTuBEREWnjbDYbUc5QopyhdI0/vvfW1HoprTJDT3HlwQBU7AtA5vPiyhrySg/2Kypzm8Go7EANO44xxlD9bfX185ClxYVzy/knn8BffGIUbkRERIJYqD3E7KQc5Tiu9/kmZD1s1OlDR6Ouv6Ps8Nvqk2OcCjciIiLStjRnQlbDMCisqPYFoPrgY/VwP5Z2nf7oo4+YNGkSnTt3xmaz8dZbbx3zPatXr2b48OGEh4fTq1cvnn766dYvVERERI5gs5m31Z+S5mJ03xQuHpHOzWP7MOO8PpbWZWm4KS8vZ8iQITzxxBPN2j8zM5OJEydyzjnnsGHDBubMmcPNN9/M4sWLW7lSERERaS8svSw1YcIEJkyY0Oz9n376abp168a8efMAOOWUU1i/fj1/+ctfmDp1aqPvcbvduN0Hb5krKSk5oZpFRESkbWtXI/qsXbuWcePGNVg3fvx41q9fT3V1daPvmTt3LrGxsb4lPT09EKWKiIiIRdpVuMnJyaFTp04N1nXq1Imamhry8vIafc/s2bMpLi72Lbt37w5EqSIiImKRdne31OFzahiG0ej6ek6nE6fz2CNGioiISHBoVy03qamp5OTkNFiXm5tLaGgoiYmJFlUlIiIibUm7CjcjR45k2bJlDdZ98MEHZGRkEBYWZlFVIiIi0pZYGm7KysrYuHEjGzduBMxbvTdu3EhWVhZg9pe58sorfftPmzaNXbt2MXPmTDZv3szzzz/Pc889x6xZs6woX0RERNogS/vcrF+/njFjxvhez5w5E4CrrrqK+fPnk52d7Qs6AD179uSdd97h1ltv5R//+AedO3fm8ccfP+pt4CIiItLx2Iz6HrkdRElJCbGxsRQXF+NyuawuR0RERJrheH6/21WfGxEREZFjUbgRERGRoKJwIyIiIkGl3Q3id6LquxhpjikREZH2o/53uzldhTtcuCktLQXQHFMiIiLtUGlpKbGxsU3u0+HulvJ6vezbt4+YmJijTtnQUiUlJaSnp7N7927didWK9DkHhj7nwNDnHDj6rAOjtT5nwzAoLS2lc+fOhIQ03aumw7XchISE0LVr11Y9h8vl0r84AaDPOTD0OQeGPufA0WcdGK3xOR+rxaaeOhSLiIhIUFG4ERERkaCicONHTqeTP/zhDzidTqtLCWr6nANDn3Ng6HMOHH3WgdEWPucO16FYREREgptabkRERCSoKNyIiIhIUFG4ERERkaCicCMiIiJBReFGREREgorCjZ88+eST9OzZk/DwcIYPH86aNWusLino3HvvvdhstgZLamqq1WW1ex999BGTJk2ic+fO2Gw23nrrrQbbDcPg3nvvpXPnzkRERDB69Gi+/fZba4ptx471OV999dVHfL/POOMMa4ptx+bOncuIESOIiYkhJSWFyZMns2XLlgb76Dt94przOVv5nVa48YNFixZxyy23cOedd7JhwwbOOeccJkyYQFZWltWlBZ0BAwaQnZ3tW7755hurS2r3ysvLGTJkCE888USj2x955BH++te/8sQTT7Bu3TpSU1O54IILfJPQSvMc63MGuPDCCxt8v995550AVhgcVq9ezfTp0/nss89YtmwZNTU1jBs3jvLyct8++k6fuOZ8zmDhd9qQE3baaacZ06ZNa7CuX79+xh133GFRRcHpD3/4gzFkyBCrywhqgPHmm2/6Xnu9XiM1NdV4+OGHfeuqqqqM2NhY4+mnn7agwuBw+OdsGIZx1VVXGRdddJEl9QSz3NxcAzBWr15tGIa+063l8M/ZMKz9Tqvl5gR5PB6++OILxo0b12D9uHHj+PTTTy2qKnht3bqVzp0707NnTy699FJ27NhhdUlBLTMzk5ycnAbfb6fTybnnnqvvdytYtWoVKSkpnHzyydxwww3k5uZaXVK7V1xcDEBCQgKg73RrOfxzrmfVd1rh5gTl5eVRW1tLp06dGqzv1KkTOTk5FlUVnE4//XQWLFjA+++/z7PPPktOTg5nnnkm+fn5VpcWtOq/w/p+t74JEybw73//mxUrVvDoo4+ybt06zjvvPNxut9WltVuGYTBz5kzOPvtsBg4cCOg73Roa+5zB2u90aKufoYOw2WwNXhuGccQ6OTETJkzwPR80aBAjR47kpJNO4sUXX2TmzJkWVhb89P1ufZdcconv+cCBA8nIyKB79+4sXbqUKVOmWFhZ+zVjxgy+/vprPv744yO26TvtP0f7nK38Tqvl5gQlJSVht9uPSPy5ublH/J+B+FdUVBSDBg1i69atVpcStOrvRtP3O/DS0tLo3r27vt8tdNNNN7FkyRJWrlxJ165dfev1nfavo33OjQnkd1rh5gQ5HA6GDx/OsmXLGqxftmwZZ555pkVVdQxut5vNmzeTlpZmdSlBq2fPnqSmpjb4fns8HlavXq3vdyvLz89n9+7d+n4fJ8MwmDFjBm+88QYrVqygZ8+eDbbrO+0fx/qcGxPI77QuS/nBzJkzueKKK8jIyGDkyJE888wzZGVlMW3aNKtLCyqzZs1i0qRJdOvWjdzcXB544AFKSkq46qqrrC6tXSsrK2Pbtm2+15mZmWzcuJGEhAS6devGLbfcwkMPPUSfPn3o06cPDz30EJGRkfziF7+wsOr2p6nPOSEhgXvvvZepU6eSlpbGzp07mTNnDklJSfz0pz+1sOr2Z/r06bz88su8/fbbxMTE+FpoYmNjiYiIwGaz6TvtB8f6nMvKyqz9Tltyj1YQ+sc//mF0797dcDgcxqmnntrgdjjxj0suucRIS0szwsLCjM6dOxtTpkwxvv32W6vLavdWrlxpAEcsV111lWEY5q2zf/jDH4zU1FTD6XQao0aNMr755htri26HmvqcKyoqjHHjxhnJyclGWFiY0a1bN+Oqq64ysrKyrC673WnsMwaMF154wbePvtMn7lifs9XfaVtdkSIiIiJBQX1uREREJKgo3IiIiEhQUbgRERGRoKJwIyIiIkFF4UZERESCisKNiIiIBBWFGxEREQkqCjciIiISVBRuREREJKgo3IiIiEhQUbgRERGRoPL/RvX+nRlal80AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Thy women divorcuse and me, whereof, if you live,\n",
      "Here overthrives be gentle climber, thy ball;\n",
      "My cripation, Tybalt of face and my hand,\n",
      "That is our hately made for requends:\n",
      "Your conquers, my suffice shive, to my service doubt\n",
      "To whom I am life and tafe malice thus\n",
      "Ere you all not with this householy persive of true:\n",
      "Your mirrous arm'd, when, they to say it at\n",
      "The world's love not takes me drid. Coriolanus: all\n",
      "they not to the remedier's small flap\n",
      "To liese as drively in answer'd: any whom\n",
      "Hav\n",
      "CPU times: user 13min 56s, sys: 2min 50s, total: 16min 47s\n",
      "Wall time: 16min 45s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#| output: true\n",
    "torch.manual_seed(TORCH_SEED)\n",
    "class TransformerLanguageModel(nn.Module):\n",
    "    def __init__(self, head_size:int, vocab_size:int=vocab_size, embedding_dim:int=embedding_dim, context_length:int=context_length, \n",
    "                 num_layers:int=num_layers, dropout:float=dropout, num_heads:int=num_heads): #UPDATED\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.context_length = context_length\n",
    "        self.head_size = head_size\n",
    "        self.num_layers = num_layers #NEW\n",
    "        self.dropout = dropout\n",
    "\n",
    "        self.token_embedding_table = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.embedding_dim)\n",
    "        self.token_position_embedding_table = nn.Embedding(num_embeddings=self.context_length, embedding_dim=self.embedding_dim)\n",
    "        \n",
    "        self.transformer_blocks = nn.Sequential(*([\n",
    "            TransformerBlock(embedding_dim=embedding_dim, num_heads=num_heads, context_length=context_length, \n",
    "                             dropout=self.dropout) for _ in range(self.num_layers)]+[\n",
    "            nn.LayerNorm(embedding_dim)\n",
    "        ])) #UPDATED\n",
    "        self.language_model_head_linear_layer = nn.Linear(in_features=self.embedding_dim, out_features=self.vocab_size)\n",
    "        \n",
    "    def forward(self, idx, targets=None):\n",
    "        #Both idx and targets are (B,T) Batch x Time array of integers\n",
    "        B,T = idx.shape\n",
    "        token_embeddings = self.token_embedding_table(idx) #(B,T,C) Batch, Time, Channel\n",
    "        token_position_embeddings = self.token_position_embedding_table(torch.arange(T, device=device)) #(T,C)\n",
    "        x = token_embeddings + token_position_embeddings\n",
    "        x = self.transformer_blocks(x)\n",
    "        logits = self.language_model_head_linear_layer(x) #(B,T,C) Where C is now token logits of size vocab_size\n",
    "        \n",
    "        if targets is not None:\n",
    "            B,T,C = logits.shape\n",
    "            logits_reshaped = logits.view(B*T,C)\n",
    "            targets_reshaped = targets.view(B*T)\n",
    "            loss = F.cross_entropy(input=logits_reshaped, target=targets_reshaped)\n",
    "        else:\n",
    "            loss=None\n",
    "        return logits, loss\n",
    "    \n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        #idx is (B,T) array of indicies in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            #Crop idx to the max size of our positional embeddings table \n",
    "            idx_crop = idx[:,-self.context_length:]\n",
    "            #Get predictions\n",
    "            logits, loss = self(idx_crop)\n",
    "            #Get the last time step from logits where the dimensions of the logits are (B,T,C)\n",
    "            logits_last_timestep = logits[:,-1,:] #Becomes (B,C)\n",
    "            #Apply softmax to get probabilities\n",
    "            probs = F.softmax(input=logits_last_timestep, dim=-1) #(B,C)\n",
    "            #Sample from the probs distribution.\n",
    "            idx_next = torch.multinomial(input=probs, num_samples=1) #(B,1) Returns (B,idxs) where idxs are random integer indicies.\n",
    "            #Append the sampled indexes idx_next to idx\n",
    "            idx = torch.cat((idx, idx_next), dim=1) #(B, T+1)\n",
    "        return idx\n",
    "\n",
    "model = TransformerLanguageModel(head_size=head_size1, vocab_size=vocab_size, embedding_dim=embedding_dim, context_length=context_length)\n",
    "model = model.to(device)\n",
    "optimizer = torch.optim.AdamW(params=model.parameters(), lr=learning_rate)\n",
    "tracked_losses = list()\n",
    "for step in range(max_iters):\n",
    "    if step % eval_iters == 0 or step == max_iters-1:\n",
    "        losses = estimate_loss()\n",
    "        tracked_losses.append(losses)\n",
    "        print('Step:',step,'Training Loss:',round(losses['train'].item(),3),'Validation Loss:',round(losses['valid'].item(),3))\n",
    "    \n",
    "    xb,yb = get_batch('train')\n",
    "    logits, loss = model(xb,yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "plot_losses(tracked_losses)\n",
    "context = torch.zeros((1,1), dtype=torch.long, device=device)\n",
    "print(decode(model.generate(context,max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85eb2504-d5d9-45d0-99e0-f9310e88a0fe",
   "metadata": {},
   "source": [
    "The results are starting to look pretty decent. The loss has dramatically improved. Scaling up the network has made a big difference. <br><br>\n",
    "**Debugging Models Aside**: While trying to train the model I realized I had made a mistake in the code. I tried to train this several times but loss would not drop below about 2.4. I went back through my code and nothing obvious was standing out to me that was wrong. After an hour of scouring my code I finally found the issue. It was a subtle change, but made all the difference. This was my code before the fix: <br>\n",
    "```python\n",
    "weights = (q @ k.transpose(-2,-1)) / self.head_size**-0.5\n",
    "```\n",
    "and this was my code after:<br>\n",
    "```python\n",
    "weights = (q @ k.transpose(-2,-1)) * self.head_size**-0.5\n",
    "```\n",
    "When we first implemented dividing the `weights` by the head size to fix the issue with large numbers passing through softmax, I had been using the `.../math.sqrt(head_size)`. To try and make things more consistent with Andrej's code and to remove the requirement to `import math` I switched the implementation over to the way he was doing it which was taking the head_size to the power of 0.5 which is equivalent to the sqrt, but I had missed that it was a -0.5 which is equivalent to 1/sqrt(0.5) so instead of dividing by `self.head_size**-0.5` I should have been multiplying by it. This can be one of the big challenges in deep learning. Often times when you make a mistake, no error is thrown, it just doesn't work. Sometimes it affects the results by a little and other times it affects it by a lot which is what happened in my case. Thankfully I had benchmarks to compare my result against so it was clear that I had an implementation issue, not that the model I was using was incapable of getting better results. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc1afc3-9160-4658-af13-b511df24e1f2",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d626f6c-72a5-4804-9c7d-cc0b0fbc54cb",
   "metadata": {},
   "source": [
    "In this notebook we have built a transformer model based on the [Attention Is All You Need](https://arxiv.org/abs/1706.03762) paper following along with Andrej Karpathy's fantastic YouTube video: [Let's build GPT: from scratch, in code, spelled out](https://www.youtube.com/watch?v=kCc8FmEb1nY). While building out the transformer we tried to build an intuition on what makes the transformer work. I hope you found the format of this notebook useful, adding and modifying the code as we went, enabling you to follow along, run the code and see the output with each step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3bfd8c3-c52a-4a7e-821d-59e0e1ebc949",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}