nanoGPT/part6_nano_gpt.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1- Preparing the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1- Opening and Exploring the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of dataset in characters: 1115394\n"
     ]
    }
   ],
   "source": [
    "with open(\"../data/input.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "    text = f.read()\n",
    "\n",
    "print(f\"length of dataset in characters: {len(text)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first 1000 characters of dataset:\n",
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n",
      "All:\n",
      "We know't, we know't.\n",
      "\n",
      "First Citizen:\n",
      "Let us kill him, and we'll have corn at our own price.\n",
      "Is't a verdict?\n",
      "\n",
      "All:\n",
      "No more talking on't; let it be done: away, away!\n",
      "\n",
      "Second Citizen:\n",
      "One word, good citizens.\n",
      "\n",
      "First Citizen:\n",
      "We are accounted poor citizens, the patricians good.\n",
      "What authority surfeits on would relieve us: if they\n",
      "would yield us but the superfluity, while it were\n",
      "wholesome, we might guess they relieved us humanely;\n",
      "but they think we are too dear: the leanness that\n",
      "afflicts us, the object of our misery, is as an\n",
      "inventory to particularise their abundance; our\n",
      "sufferance is a gain to them Let us revenge this with\n",
      "our pikes, ere we become rakes: for the gods know I\n",
      "speak this in hunger for bread, not in thirst for revenge.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(f\"first 1000 characters of dataset:\\n{text[:1000]}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2- Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all chars are:\n",
      " !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\n",
      "vocab size: 65\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(list(set(text)))\n",
    "vocab_size = len(chars)\n",
    "\n",
    "print(f\"all chars are:{''.join(chars)}\")\n",
    "print(f\"vocab size: {vocab_size}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20, 43, 50, 50, 53, 6, 1, 61, 53, 56, 50, 42, 2]\n",
      "Hello, world!\n"
     ]
    }
   ],
   "source": [
    "stoi = {ch:i for i, ch in enumerate(chars)}\n",
    "itos = {val:key for key, val in stoi.items()}\n",
    "\n",
    "encode = lambda s: [stoi[c] for c in s]\n",
    "decode = lambda l: ''.join([itos[i] for i in l])\n",
    "\n",
    "print(encode(\"Hello, world!\"))\n",
    "print(decode(encode(\"Hello, world!\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data shape: torch.Size([1115394])\n",
      "data type: torch.int64\n",
      "--------------------------------------------------\n",
      "first 100 characters of dataset:\n",
      "tensor([18, 47, 56, 57, 58,  1, 15, 47, 58, 47, 64, 43, 52, 10,  0, 14, 43, 44,\n",
      "        53, 56, 43,  1, 61, 43,  1, 54, 56, 53, 41, 43, 43, 42,  1, 39, 52, 63,\n",
      "         1, 44, 59, 56, 58, 46, 43, 56,  6,  1, 46, 43, 39, 56,  1, 51, 43,  1,\n",
      "        57, 54, 43, 39, 49,  8,  0,  0, 13, 50, 50, 10,  0, 31, 54, 43, 39, 49,\n",
      "         6,  1, 57, 54, 43, 39, 49,  8,  0,  0, 18, 47, 56, 57, 58,  1, 15, 47,\n",
      "        58, 47, 64, 43, 52, 10,  0, 37, 53, 59])\n",
      "first 100 characters of dataset:\n",
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "data = torch.tensor(encode(text), dtype = torch.long)\n",
    "\n",
    "print(f\"data shape: {data.shape}\")\n",
    "print(f\"data type: {data.dtype}\")\n",
    "\n",
    "print(\"-\"*50)\n",
    "\n",
    "print(f\"first 100 characters of dataset:\\n{data[:100]}\")\n",
    "print(f\"first 100 characters of dataset:\\n{decode(data[:100].tolist())}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3- Train Validation Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = int(0.9 * len(data))\n",
    "\n",
    "train_data = data[:n]\n",
    "val_data = data[n:]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Create DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([18, 47, 56, 57, 58,  1, 15, 47, 58])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "block_size = 8\n",
    "train_data[:block_size + 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "when input tensor is [18], target is 47\n",
      "when input tensor is [18, 47], target is 56\n",
      "when input tensor is [18, 47, 56], target is 57\n",
      "when input tensor is [18, 47, 56, 57], target is 58\n",
      "when input tensor is [18, 47, 56, 57, 58], target is 1\n",
      "when input tensor is [18, 47, 56, 57, 58, 1], target is 15\n",
      "when input tensor is [18, 47, 56, 57, 58, 1, 15], target is 47\n",
      "when input tensor is [18, 47, 56, 57, 58, 1, 15, 47], target is 58\n"
     ]
    }
   ],
   "source": [
    "x = train_data[:block_size]\n",
    "y = train_data[1:block_size + 1]\n",
    "\n",
    "for t in range(block_size):\n",
    "    context = x[:t + 1]\n",
    "    target = y[t]\n",
    "    print(f\"when input tensor is {context.tolist()}, target is {target}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs:\n",
      "shapetorch.Size([4, 8])\n",
      "data: tensor([[24, 43, 58,  5, 57,  1, 46, 43],\n",
      "        [44, 53, 56,  1, 58, 46, 39, 58],\n",
      "        [52, 58,  1, 58, 46, 39, 58,  1],\n",
      "        [25, 17, 27, 10,  0, 21,  1, 54]])\n",
      "targets:\n",
      "shapetorch.Size([4, 8])\n",
      "data: tensor([[43, 58,  5, 57,  1, 46, 43, 39],\n",
      "        [53, 56,  1, 58, 46, 39, 58,  1],\n",
      "        [58,  1, 58, 46, 39, 58,  1, 46],\n",
      "        [17, 27, 10,  0, 21,  1, 54, 39]])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1337)\n",
    "batch_size = 4 # how many independent sequences will we process in parallel\n",
    "block_size = 8 # what is the maximum context length for predictions?\n",
    "\n",
    "# number of input_examples = batch_size * block_size (4 * 8 = 32)\n",
    "\n",
    "def get_batch(split):\n",
    "    # Select the appropriate dataset based on the split parameter\n",
    "    data = train_data if split == \"train\" else val_data\n",
    "\n",
    "    # Generate a batch of random starting indices within the dataset\n",
    "    ix = torch.randint(len(data) - block_size, (batch_size,))\n",
    "\n",
    "    # Select a block of text of size block_size starting from each random index\n",
    "    x = torch.stack([data[i:i+block_size] for i in ix])\n",
    "\n",
    "    # Shift the selected block of text by one character to the right to create the target sequence\n",
    "    y = torch.stack([data[i+1:i+block_size+1] for i in ix])\n",
    "    return x, y\n",
    "\n",
    "xb, yb = get_batch(\"train\")\n",
    "print(f\"inputs:\\nshape{xb.shape}\\ndata: {xb}\")\n",
    "print(f\"targets:\\nshape{yb.shape}\\ndata: {yb}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2- Bigram Language Model\n",
    "simpleset language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7ff1588ebab0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "torch.manual_seed(1337)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Creating the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logits shape: torch.Size([32, 65])\n",
      "loss: 4.878634929656982 | we are expecting a loss of around 4.174387454986572\n"
     ]
    }
   ],
   "source": [
    "class BigramLanguageModel(nn.Module):\n",
    "    def __init__(self, vocab_size):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a loockup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets = None):\n",
    "        # idx and targets are both (B, T) tensor of ints\n",
    "        logits = self.token_embedding_table(idx) # (B, T, C) = (4, 8 , vocab_size)\n",
    "        \n",
    "        if targets is None:\n",
    "            loss = None\n",
    "        else:\n",
    "            # note that F.cross_entropy accepts inputs in shape (B, C, T)\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B * T, C)\n",
    "            targets = targets.view(B * T) # can be as targets = targets.view(-1)\n",
    "            \n",
    "            loss = F.cross_entropy(logits, targets)\n",
    "\n",
    "        return logits, loss\n",
    "\n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        # idx is (B, T) array of indices in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            # get the logits for the next token\n",
    "            logits, loss = self(idx)\n",
    "            # focus only on the last time step\n",
    "            # (note that we are feeding the whole context each time, however we only care about the last prediction)\n",
    "            # (this make doesn't make sense now, but the function will be modified later)\n",
    "            logits = logits[:, -1, :] # Becomes (B, C) (get the last time step for each sequence)\n",
    "            # apply softmax to convert to probabilities\n",
    "            probs = F.softmax(logits, dim = -1) # (B, C)\n",
    "            # sample from the distribution\n",
    "            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)\n",
    "            # append sampled token to the context\n",
    "            idx = torch.cat((idx, idx_next), dim = 1) # (B, T + 1)\n",
    "        return idx\n",
    "    \n",
    "\n",
    "m = BigramLanguageModel(vocab_size)\n",
    "logits, loss = m(xb, yb)\n",
    "\n",
    "print(f\"logits shape: {logits.shape}\")\n",
    "print(f\"loss: {loss} | we are expecting a loss of around {torch.log(torch.tensor(vocab_size))}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Sr?qP-QWktXoL&jLDJgOLVz'RIoDqHdhsV&vLLxatjscMpwLERSPyao.qfzs$Ys$zF-w,;eEkzxjgCKFChs!iWW.ObzDnxA Ms$3\n"
     ]
    }
   ],
   "source": [
    "idx = torch.zeros((1,1), dtype = torch.long)\n",
    "generated = m.generate(idx, 100) # shape (1, 101)\n",
    "print(decode(generated[0].tolist()))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2- Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 2.5727508068084717\n"
     ]
    }
   ],
   "source": [
    "optimizer = torch.optim.AdamW(m.parameters(), lr = 1e-3)\n",
    "batch_size = 32\n",
    "lossi = []\n",
    "\n",
    "for i in range(10000):\n",
    "    # sample a batch of training data\n",
    "    xb, yb = get_batch(\"train\")\n",
    "\n",
    "    # evaluate the loss\n",
    "    logits, loss = m(xb, yb)\n",
    "    optimizer.zero_grad(set_to_none = True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    lossi.append(loss.item())\n",
    "\n",
    "print(f\"loss: {loss.item()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGgCAYAAAB45mdaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOjUlEQVR4nO3deVhU9f4H8PewDSA7CIgCLqCogOIKuJVSLmRa3TLTtK76y+2mLVaUlbbhbb2WZbaolZptLrmkkfuCigsKoriioAIqsu/M+f2BjDMwM8wwyxmY9+t55nmYc77nez5zXObDd5UIgiCAiIiISCRWYgdARERElo3JCBEREYmKyQgRERGJiskIERERiYrJCBEREYmKyQgRERGJiskIERERiYrJCBEREYmKyQgRERGJiskIERERiUqnZGTBggWQSCRKr5CQEI3X/PbbbwgJCYG9vT3CwsKwdetWvQImIiKilsVG1wu6d++Of/75514FNuqrOHjwIMaPH4/4+Hg89NBDWLNmDcaOHYvjx48jNDRU63vKZDJcv34dzs7OkEgkuoZMREREIhAEAUVFRfDz84OVlfr2D4kuG+UtWLAAGzZsQHJyslblx40bh5KSEmzevFl+LDIyEj179sTXX3+t7W2RlZUFf39/rcsTERGR+cjMzES7du3Unte5ZeT8+fPw8/ODvb09oqKiEB8fj4CAAJVlExMT8eKLLyodGz58ODZs2KDxHhUVFaioqJC/r8uXMjMz4eLiomvIREREJILCwkL4+/vD2dlZYzmdkpH+/ftj5cqV6NKlC27cuIGFCxdi0KBBSE1NVXmj7Oxs+Pj4KB3z8fFBdna2xvvEx8dj4cKFDY67uLgwGSEiImpmGhtiodMA1pEjR+Lxxx9HeHg4hg8fjq1btyI/Px+//vqrXkHWFxcXh4KCAvkrMzPToPUTERGR+dC5m0aRm5sbOnfujAsXLqg87+vri5ycHKVjOTk58PX11VivVCqFVCrVJzQiIiJqJvRaZ6S4uBgXL15EmzZtVJ6PiorCjh07lI4lJCQgKipKn9sSERFRC6JTMvLyyy9jz549yMjIwMGDB/HII4/A2toa48ePBwBMmjQJcXFx8vJz5szBtm3b8Mknn+Ds2bNYsGABjh49itmzZxv2UxAREVGzpVM3TVZWFsaPH4/bt2+jdevWGDhwIA4dOoTWrVsDAK5evao0jzg6Ohpr1qzB/Pnz8frrryM4OBgbNmzQaY0RIiIiatl0WmdELIWFhXB1dUVBQQFn0xARETUT2n5/c28aIiIiEhWTESIiIhIVkxEiIiISFZMRIiIiEhWTESIiIhIVkxEiIiISlUUnIzKZgBUHLuNUVr7YoRAREVksvfamae42JF/Dwk1pAICMRbEiR0NERGSZLLpl5Gx2kdghEBERWTyLTkaIiIhIfExGiIiISFRMRoiIiEhUFp2MXLtTJnYIREREFs+ik5EtKTfEDoGIiMjiWXQyQkREROJjMnLXQ1/sEzsEIiIii8Rk5K7Ua4W4XVwhdhhEREQWh8mIgvyyKrFDICIisjhMRhQIgtgREBERWR4mIwoOX76N/247i8pqmdihEBERWQyL3iivvjfWpwIAvJykmDKwg8jREBERWQa2jKhw5XaJ2CEQERFZDCYjKtwoKBc7BCIiIovBZESFhLQcsUMgIiKyGExGiIiISFRMRoiIiEhUTEaIiIhIVExGiIiISFRMRtQQBAHbUrORmVcqdihEREQtGhc9U2P76WxMX3UcAJCxKFbkaIiIiFoutoyoceTyHbFDICIisghMRtSQcdc8IiIik2AyokZRebXYIRAREVkEJiNq/HE8S+wQiIiILAKTESIiIhIVkxEiIiISFZMRIiIiEhWTESIiIhKVRScjEQFuYodARERk8Sw6GfFwtBM7BCIiIotn0clIDRc2IyIiEp1FJyNOUm7NQ0REJDaLTkbmx3aDt7O00XLcuZeIiMh4LDoZ8XW1x7qZ0Y2Wu1FQboJoiIiILJNFJyPa2paaLXYIRERELZbFJyMSiaTRMssPXDZBJERERJaJyYiW5facu2nUOIiIiCyVxScj2pq8/Ajav7YF20+zy4aIiMiQLD4Z0aKXRslzPx0zTiBEREQWyuKTES+nxqf21pd6rcAIkRAREVkmi09GbK2tcHrhcJ2uKSirMlI0RERElsfikxEAaCW1wYAgT63L69izQ0RERBowGblr9dRIvPxgZ7HDICIisjh6JSOLFi2CRCLB3Llz1ZZZuXIlJBKJ0sve3l6f2xqN1vvmSYC8kkqjxkJERGQpmrxTXFJSEpYtW4bw8PBGy7q4uCA9PV3+XpuFxsSg7S6+qw5dwdaUbLz8YGfMHhps5KiIiIhatia1jBQXF2PChAn49ttv4e7u3mh5iUQCX19f+cvHx6cptzU6mUy7ZGRrSu1aIx//fc6Y4RAREVmEJiUjs2bNQmxsLGJiYrQqX1xcjMDAQPj7+2PMmDE4ffq0xvIVFRUoLCxUepnC5Oj2JrkPERER3aNzMrJ27VocP34c8fHxWpXv0qULli9fjo0bN2LVqlWQyWSIjo5GVlaW2mvi4+Ph6uoqf/n7++saZpN4Oknx3JCOJrkXERER1dIpGcnMzMScOXOwevVqrQehRkVFYdKkSejZsyeGDBmCdevWoXXr1li2bJnaa+Li4lBQUCB/ZWZm6hKmXuYO6ww7a04yIiIiMhWdvnWPHTuG3Nxc9OrVCzY2NrCxscGePXvw+eefw8bGBjU1NY3WYWtri4iICFy4cEFtGalUChcXF6WXqTjYWePl4ZziS0REZCo6zaYZNmwYUlJSlI49++yzCAkJwauvvgpra+tG66ipqUFKSgpGjRqlW6QmpOU4ViIiIjIAnZIRZ2dnhIaGKh1r1aoVPD095ccnTZqEtm3byseUvPPOO4iMjERQUBDy8/Px0Ucf4cqVK5g6daqBPoLh1TAbISIiMpkmrzOiztWrV2Flda/3586dO5g2bRqys7Ph7u6O3r174+DBg+jWrZuhb20wVTUysUMgIiKyGBJB0HrdUdEUFhbC1dUVBQUFJhk/8vH2dCzZpX5Mi6KMRbFGjoaIiKh50vb7m9NGVKiSsWWEiIjIVJiMqDA5qr3WZfNLuUcNERGRPpiMqODn5qB12WdWJBkxEiIiopaPyYiekjPzld4nZeQht7BcnGCIiIiaIYPPprFkRy7n4YlliQA4sJWIiEhbbBkxoAMXbokdAhERUbPDZMQACsqqAABmP0eaiIjIDDEZMYAeC/+unVVj/ku2EBERmR0mIwby321n2TJCRETUBExGDCQzr0zsEIiIiJolJiMGUlhexV4aIiKiJmAyogV728Yf09W8UsgUshFutkdERKQdJiNaCPFtfHO+/NIqpQRk/vpUY4ZERETUYjAZ0cJXE3ppVe7bfZflP/9yNNNY4RAREbUoTEYa8caorjrtVUNERES64XLwaqyfGY09525icnR7sUMhIiJq0ZiMqBER4I6IAHexwyAiImrx2E1jRKsPX8Hu9FwInPNLRESkFpMRLc24r5PO17yxPhXPrEjCWxtPGyEiIiKiloHJiJZeHRGCmK4+Tbr2p0NXDBwNERFRy8FkRAdfjI8QOwQiIqIWh8mIDhzsrMUOgYiIqMVhMkJERESiYjKio9+nR4kdAhERUYvCZERHfdp74Mjrw8QOg4iIqMVgMtIE3i72YodARETUYjAZaaI3H+qmU/mK6hojRUJERNS8MRlpovaejjqVn/fbKew6y9VYiYiI6mMy0kQ21ro9uj9PXsezK5Pwd1qOkSIiIiJqnpiMNNHAIK8mXZd48baBIyEiImremIw0kbWVROwQiIiIWgQmI0RERCQqJiN6+GJ8BNwdbXW6ZuXBDBSUVRkpIiIiouaHyYgeRvfww+dN2DzvnU1pRoiGiIioeWIyoqeubVx0viblWr7hAyEiImqmmIzoyctJqvM1EnDwKxERUR0mIyIQwIXPiIiI6jAZEUlyZj4W/HkaBaUczEpERJbNRuwALNXYLw8AqJ1dc+mDUbDiuiVERGSh2DJiBrLulIkdAhERkWiYjBiAro0a53KKld5L2ChCREQWjMmIAaybOQD9Onjgy6d6iR0KERFRs8NkxAB6+rvh1+eiEBvepknXC5xcQ0REFozJCBEREYmKyYiRhLdz1bps3ZiR7/ZdwmNLD6K4otpIUREREZkfJiNG8khEW52veW/LGRy7cgcr9l82QkRERETmicmIkUR29NS67KrDV5Tel1fXGDocIiIis8VkxMAOvjYUf8yI1mkDvWV7LhkxIiIiIvPGFVgNzM/NAX5uDmKHQURE1GywZYSIiIhExWSEiIiIRMVkxAxJwPXhiYjIcjAZISIiIlHplYwsWrQIEokEc+fO1Vjut99+Q0hICOzt7REWFoatW7fqc9tmY9nTvcUOgYiIyOw1ORlJSkrCsmXLEB4errHcwYMHMX78eEyZMgUnTpzA2LFjMXbsWKSmpjb11s2GLquwtn9tixEjISIiMl9NSkaKi4sxYcIEfPvtt3B3d9dYdvHixRgxYgTmzZuHrl274t1330WvXr2wZMmSJgVMRERELUuTkpFZs2YhNjYWMTExjZZNTExsUG748OFITExUe01FRQUKCwuVXpbkj+NZ+CvlBs5mW9bnJiIiy6Tzomdr167F8ePHkZSUpFX57Oxs+Pj4KB3z8fFBdna22mvi4+OxcOFCXUNrMW4UlGPG6uMAgIxFsSJHQ0REZFw6tYxkZmZizpw5WL16Nezt7Y0VE+Li4lBQUCB/ZWZmGu1eREREJC6dWkaOHTuG3Nxc9OrVS36spqYGe/fuxZIlS1BRUQFra2ula3x9fZGTk6N0LCcnB76+vmrvI5VKIZVKdQnNLDnYWjdeiIiIyMLp1DIybNgwpKSkIDk5Wf7q06cPJkyYgOTk5AaJCABERUVhx44dSscSEhIQFRWlX+TNgJujHd4bGwoJ1zAjIiJSS6eWEWdnZ4SGhioda9WqFTw9PeXHJ02ahLZt2yI+Ph4AMGfOHAwZMgSffPIJYmNjsXbtWhw9ehTffPONgT6CeZsYGYjT1wvx85GrTbpeEARImM0QEVELZvAVWK9evYobN27I30dHR2PNmjX45ptv0KNHD/z+++/YsGFDg6SGVAt5c5vYIRARERmVzrNp6tu9e7fG9wDw+OOP4/HHH9f3Vs2WPg0bFdUywwVCRERkhrg3DREREYmKyQgRERGJiskIERERiYrJiIn997EwsUMgIiIyK0xGTMzX1UHsEIiIiMwKkxETCPZ2kv8sCIKIkRAREZkfvaf2UuMmRgaisKwaA4O9UFReJXY4REREZoUtIyZga22FOTHB6B3orrSaag9/N62uf/r7w/hu3yUjRUdERCQuJiMi+uTxHlqV23f+Ft7bcgblVTVGjoiIiMj0mIyIyMZKt6VZQ97chuv5ZUaKhoiISBxMRkxMcQBrU4ayPvTFfsMFQ0REZAaYjIjIz81e52vySiqNEAkREZF4mIyYmOIAVqmNdZPqOHjhlqHCISIiEh2TERPzc1VuDXk6MlDnOp767rChwiEiIhIdkxETC/Zxxv/G9cSaaf0BAK+P6tqkepbsPI+UrAJDhkZERCQKJiMiGBvRFtGdvAAADnZN66r5+O9zGL2Eg1mJiKj5YzJiBgYFe4kdAhERkWiYjBAREZGomIy0IDUyAW9vTMWWUzfEDoWIiEhrTEbMgOJ0X31sTL6GHxKvYNaa4wapj4iIyBSYjJiZAA9HncrPXXsCFdW1e9bkFlUYIyQiIiKjYjJiZt4Z012n8huSr2PN4asAartpiIiImhsmI2ZgZKgvAMDHRYpBwa11vj6vpBIV1TX4aHu6oUMjIiIyOhuxAyBgXB9/tHVzQFhbV1jruJMvAAgCcDTjjhEiIyIiMj62jJgBKysJBnduDfdWdgCAecO76FyHYYbAEhERmR5bRszQjCGdcPzKHew4m6tV+SW7LiApI8/IURERERkHW0bMkJWVBK+NDNHpmsOXmYwQEVHzxGTETBlo6REiIiKzx2TEQuw4k4O1R66KHQYREVEDHDNipvzcHAxa35QfjgIA+rT3QJC3k0HrJiIi0gdbRsyUo51x8sRbxVyllYiIzAuTESIiIhIVkxEz1qOda5OvzS+tNGAkRERExsNkxIxFdfJq8rXvbE4zYCRERETGw2TEjAlo+sZ3J67mo7yqpsFxzhgmIiJzw2TEjEmtm/7Hc/lWCXq9m9DguEwAyiobJilERERiYTJixqYM6ojWztImX196N+n4Jy1Hfmz8t4fQ9a1tnFVDRERmg8mIGXN1sEXSGzHIWBSLNq72TapDJhMw9cejDY4nKCQoREREYmIy0kw0dRG0apnqcSccO0JEROaCyUgLV6MmGSEiIjIXTEZauGdXHhE7BCIiIo2YjLRwhy7liR0CERGRRkxGLJSEg0aIiMhMMBlpJvzdDbuLLxERkbkwztawZHBvje4OiUQCe1sr/HwkU+/6isqrDRAVERGR/tgy0kx4tLLDZ+N6IrKjp/xYdz+XJtf33pYzhgiLiIhIb0xGmpk2rve6a7r4OIsYCRERkWGwm6aZ6dveHW+M6opO3q2wMfm62OEQERHpjS0jzYxEIsG0wR0xNMQH+q5n9uaGVLy1MdUwgRERETURk5FmTKZnNvLToSv4MfEKCkqrDBQRERGR7piMNGNTB3UwSD0ygUvGExGReJiMNGMRAe44+daDuBw/Sq965v1+Eocv3TZQVERERLphMtLMuTraQqLncqr/nMnFuG8OGSgiIiIi3eiUjCxduhTh4eFwcXGBi4sLoqKi8Ndff6ktv3LlSkgkEqWXvb293kGTccxcfQwCu2yIiMjEdJra265dOyxatAjBwcEQBAE//PADxowZgxMnTqB79+4qr3FxcUF6err8vb6/xZNqdtZWqKyR6VXH1pRspN0oRHc/VwNFRURE1DidWkZGjx6NUaNGITg4GJ07d8b7778PJycnHDqkvolfIpHA19dX/vLx8dE7aGpo/6v3o38HD73ree6nY6hWSGp2nc3FqMX7cOZGod51ExERqdLkMSM1NTVYu3YtSkpKEBUVpbZccXExAgMD4e/vjzFjxuD06dON1l1RUYHCwkKlF2nm7WKPX55T/+egraw7ZVj011n5+2dXJiHtRiGe++mY3nUTERGponMykpKSAicnJ0ilUkyfPh3r169Ht27dVJbt0qULli9fjo0bN2LVqlWQyWSIjo5GVlaWxnvEx8fD1dVV/vL399c1TNLDd/svNzhWWM61SIiIyDh0Tka6dOmC5ORkHD58GDNmzMDkyZORlpamsmxUVBQmTZqEnj17YsiQIVi3bh1at26NZcuWabxHXFwcCgoK5K/MTP13qSUiIiLzpPPeNHZ2dggKCgIA9O7dG0lJSVi8eHGjCQYA2NraIiIiAhcuXNBYTiqVQiqV6hoaGdD1/DL4uTk0XpCIiEhPem+UJ5PJUFFRoVXZmpoapKSkYNQo/RbpIuOLXrQTAR6OYodBREQWQKdkJC4uDiNHjkRAQACKioqwZs0a7N69G9u3bwcATJo0CW3btkV8fDwA4J133kFkZCSCgoKQn5+Pjz76CFeuXMHUqVMN/0nI4K7mlYodAhERWQCdkpHc3FxMmjQJN27cgKurK8LDw7F9+3Y88MADAICrV6/CyureMJQ7d+5g2rRpyM7Ohru7O3r37o2DBw+qHfBKRERElkciNIMlNwsLC+Hq6oqCggK4uLiIHY5Za//aFgBAv/YeOJKRZ7B63RxtkfzWgwarj4iIWj5tv7+5N00LFezjZND6isqrcfFmsUHrJCIiApiMtDjrZ0bjjVFd8XAPP4PWWyMTMOyTPfjz5HWD1ktERMRkpIWJCHDHtMEdYWVlnD2Anv/5hFHqJSIiy8VkpIUy/5FAREREtZiMtFDujrZih0BERKQVJiMtVLCPM94Y1dVo9QuCgN+PZSE9u8ho9yAiIsvAZKQFmza4o/zn10eFGLTubanZePm3kxj+v70oKOUmekRE1HRMRizE8O6+6OHvZrD6Tl0rkP/c452/DVYvERFZHiYjLVzyWw9g50tDEOjZCh88Eip2OERERA3ovVEemTc3Rzu4OdoBAIK9nQ1Wr4zTdYiIyEDYMmJB7Gys8NOUfhgU7KVXPW+sT8GyPZcMFBUREVk6JiMWZlBwa3zyRA+96lh9+GqDYzeLKvSqk4iILBeTEQvk7Wxv8Dr7vv+PweskIiLLwGSEiIiIRMVkhIiIiETFZISIiIhExWTEQv00pZ/YIRAREQFgMmKxBgW3xtH5MTi9cLjYoRARkYVjMmLBvJyksLGWGKy+rDulBquLiIgsB5MRCyeB4ZKRgf/dhWNX7hisPiIisgxMRsig1h3PQl5JJQQuF09ERFpiMkIGtfrwVfR6NwHvbj4jdihERNRMMBkho1h+4DIAQCYT8M3eizh2JU/kiIiIyFxx114LZ+zddzeduo4Ptp4FAGQsijXqvYiIqHliy4iFs7W+91fg0V5tDVp3SlYBLt4sMWidRETU8rBlxMJZW0mQ8MJgVNUICPR0RJC3E0Z098XQT/boXffoJfsxKSqwwXGZrLY1xsrKcDN5iIio+WIyQgj2cZb/PPO+IIPW/WPiFaX3MpmAUZ/vg621Ff6cPQASCRMSIiJLx2SETCq7sBxns4sAACWVNXCS8q8gEZGl45gRIiIiEhWTEVIp0NPRKPUqzt3hwmhERAQwGSE1/pozCF9P7GWSe9UNaCUiIsvEZIRUcrSzwYjQNlgztT+8naUGqXPm6mN44utE+XuJRIKkjDyELdiOn49cNcg9iIio+WEyQhpFB3mhs8JsG31sTcnGtfwy+XtBEDBj1XGUVNYgbl2KQe5BRETND5MRapRxZ9/e66LZcSZHfSmOLyEiarGYjFCjegW4G6Xe+muMTPnhKD7eno4fEzOUjh++dBs930nAhhPXjBIHERGJi8kINeqxXu2U3vdtb5jkRBAE1G/wWLLrAt7aeFrp2NQfjqKgrApzf0k2yH2JiMi8MBmhRlXWyJTer5kWaZB6SypqtCrHDhoiopaNyQg1yt/DATZ395H5Y0aU0uZ6+oiM36FVosEF44mIWjauxU2NktpYI3XhcFhbSeSJyMAgL+y/cEvvuvNKKlUeP5dTZLBZPEREZN7YMkJasbe1VmoRmXlfJ6Pe78HP9uJOXaLCphEiohaNyQg1iY2Bumo0+enQFbR/bQuKyqvlx87nFBn9vkREZFpMRqhJege6Y2CQFyZGBuD36VFGucenCecaHHvgs71GuRcREYmHY0aoSaytJFg1tT8AoKrebBsxpV4rQGFZFaKDvMQOhYiItMRkhPRmToujPvTFfgDAwdeGws/NQeRoiIhIG+ymIb3VTfs1lZOZ+fg1KVPjEvHXFfbAISIi88aWEdKblZUEQ0O8sfNsrknuN+bLAwAAX1d7DO7cWmUZM2qsISKiRrBlhAzi+8l9MDTE26T3vJBbDADYd/4mlu+/bNJ7ExGR4bBlhAxCIpHAs5WdSe/5zuY0PBPdHk9/fwQA0LWNy714TBoJERHpgy0jZDASETKAuHUp8p+vcZwIEVGzxGSEDEai0B5hqJ19G/PL0UyT3IeIiIyHyQgZTKCXo/znNdMi8dOUfia9/wGFvXI4gJWIqPngmBEymCkDOyCvuBJDu3rD1toKEQGmaR2ps/7ENZPej4iIDIPJCBmM1MYa8x/qJn/PQaRERKQNnbppli5divDwcLi4uMDFxQVRUVH466+/NF7z22+/ISQkBPb29ggLC8PWrVv1CphIG9U1AnIKy/HY0oPYmMwWEyIic6ZTMtKuXTssWrQIx44dw9GjRzF06FCMGTMGp0+fVln+4MGDGD9+PKZMmYITJ05g7NixGDt2LFJTUw0SPJk3MWbX1Bn/7SH0/2AHjl25gzlrk/Wuz5z23yEiamkkgqY1tbXg4eGBjz76CFOmTGlwbty4cSgpKcHmzZvlxyIjI9GzZ098/fXXWt+jsLAQrq6uKCgogIuLS+MXkFkoq6xB17e2iR0GACBjUazK42WVNbiQW4zQti6QqMme3tqYip8OXcGOF4egY2snY4ZJRNSiaPv93eTZNDU1NVi7di1KSkoQFaV6C/nExETExMQoHRs+fDgSExM11l1RUYHCwkKlFzU/NtbmNWrk4s1i3CquUDr21HeHMHrJfvx+LEvtdT8mXoEgAF/uumjsEImILJLOyUhKSgqcnJwglUoxffp0rF+/Ht26dVNZNjs7Gz4+PkrHfHx8kJ2drfEe8fHxcHV1lb/8/f11DZPMgK21Fb55urf8fbC3eK0K1/LLMOyTPejz3j/yY1U1Mpy4mg8A+CWp8fVKTmblGyk6IiLLpnMy0qVLFyQnJ+Pw4cOYMWMGJk+ejLS0NIMGFRcXh4KCAvkrM5MLWzVXD3b3lf8c3clTtDgOnL+l9L6ssgb93v9HTWnVSiuqDRkSERHdpfPUXjs7OwQFBQEAevfujaSkJCxevBjLli1rUNbX1xc5OTlKx3JycuDr69ugrCKpVAqpVKpraGTm1I3JMIVX/jgl/znx4m2M//aQaLEQEZEyvVdglclkqKioUHkuKioKO3bsUDqWkJCgdowJkSmoSkS4YisRkXh0Skbi4uKwd+9eZGRkICUlBXFxcdi9ezcmTJgAAJg0aRLi4uLk5efMmYNt27bhk08+wdmzZ7FgwQIcPXoUs2fPNuynILPWxtUeADAiVHOLWHNQWF6Fq7dLxQ6DiKhF0ambJjc3F5MmTcKNGzfg6uqK8PBwbN++HQ888AAA4OrVq7CyupffREdHY82aNZg/fz5ef/11BAcHY8OGDQgNDTXspyCz9s+LQ3AtvwydfZyVjrd1czCbnXYFQUBuUTm2nLqBR3u1g6uDbYMy1wvKEb7gbwDArpfvQwevVqYOk4ioRdJ7nRFT4DojLUf717bIf054YTAe+GyviNGoN2dYMB6JaIv2Xq2UYq7zwSNheKp/gAiRERE1H9p+f3NvGjIpZ3sbFJXXzkoRc0BrYxbvOI/FO85j5n2dVJ6XmX8OT0TUbOg9gJVIF8rph/l/oX+1W/VCZ6ZoUGwGjZZERAbBZIRMypxbQ3Tx1e6LGLcsEeVVNUapf+Gm0xj6yR4Uc20TIrIATEbIpMLbuQIArK0kaOfuKD8e09UH30/ugyARV2nVxY2Cchy+nIdfjxpnQb4VBzJw+VYJfjNS/URE5oRjRsikPnmiB77adRFP9Q+Ava01Ti8cDmsrCextrQEA3+y9hAsix6iLiirj7uZbI2NXDRG1fGwZIZPydrbHgoe7y6f5tpLayBMRALDUYRLHr97ByMX7kHjxttihEBGZHJMRMitCMxjUquhmvV2ABUFAQloOMvMaLoy2bM9FfLfvksp6nlx2CGduFDZYHdZSkzMisizspiGz0tx6Jb7Zewk+LvZYdzwLq6b0R1JGHv7vp2MAgNdHhWBcnwC4OtriTkkl4v86CwB4sl8AnKTK//Qqa4zb3UNEZM6YjJBZqT+d1aOVHfJKKpWOdWvjgrQbhaYMS6N3N9fuWv3lLuXRLh9sPYtdZ2+id6A7QtveW+ynqloGaLkPZHNrKSIiagp205BZqf/Vu+k/A2Fnfe+vaWRHD2ydM8i0QWmpTMU038RLt7Fk1wVMX3VcfkyXBdPYTUNEloDJCJmV+t0Xbd0ccPLtBxuU6xPobqqQtCYAKKlsfN2R6/nlOtVJRNTSMRkhs/LBI2ENjjnYWTc4NrSrtynC0UlFlQw/H7naaLnRS/bjTr2uJ3UEATh48Rae/v4wCkqr9A2RiMgsMRkhs+Lv4YgdLw1pcNzbuXaQxQPdfAEA0wZ1NGlc2vjjeJbWZS/eLNaq3KWbxXjq28PYd/4Whn26p6mhERGZNQ5gJbPTqbUT1s+MhpfTvVGeW+cMwvErdzA0pLZFxNbaCu09HZFxu+EU2pbkt2P3Epxb9aYRq7LrbC6CvJ3g7+HYaFkiInPBZITMUkSA8pgQLycpHuzuK1I0xlFQWoUVBy/DWof9enKLyuHtbK/y3L7zN/HsyiQAQMaiWIPESERkCkxGqNlq7oM7X1+fgi0pN3S65r3NZ/D5+AiV545duWOIsIiITI5jRohE8K+vE3VORADgdknjXTUAMHP1MZRpMbNHV/XXgSEiMgQmI9RstdTvxWNX8vSuY2tKttql528UlOFGQZnOdeaVVGLAop34YOsZfcMjIlLCZISaLXdHW7FDMIrHlibqfE1JRTX+9895pWO3VUwfrqiuQVT8TkTF70SVjkvQL99/GdcLyvHNXtVJDhFRUzEZoWbrs3E9ERHghu8n9xE7FFElZeRh0vIjDY6vPJiB8nqrwuYrrFVSWqFbN05NE5uiUrIK8OWuCzonP0RkOZiMULPVsbUT1s8cgGFdfcQOxWTq5wMFZVV4/OtEtYNXVx7MUF+Z9pN4Gtx74abTWl83esl+fLQ9HT9oiqUFO329AJ/vON8gMSSie5iMUIvTqXUrsUMwmuv5ZSiuqAYArDp0BT0W/q2x/O27a5MUlVfhp0NXcLNIuwGwAPD2xlTMWn1c5aDVFQcytA/6rrPZRTpf0xLEfr4fnyacw1e7L4odCpHZYjJCLcKLD3SGv4cDvprQCxtnDxQ7HKPJuF0qT0Dmb0httPyt4tpxI3HrUvDmhlRM+O6w/Fxjy5v8kHgFW1Ju4Hxu7WqxmnYQvllUga92X0Bukfp9d4w14PhsdiGW7r6IimrzbnlIu24+O00TmRuuM0ItwvPDgvH8sGD5+5+nRaKkohpTfzwqYlTGUSMTcPDCLa3Krj9xDW+P7oa/T+cAqO3W0YZia0iN7O7PGpKJp749hPO5xdiWmo3vJvXBK3+cwtORgSbpQhvxv30AgKoamdLfASJqPtgyQi1SVCdPxHRruWNJdp7N1bps2o1Cja0aigRBwOrDV3Ayq0B+zOpuE0r9GupaIq7ll8lbT05lFWDBptPYnX4TU34wbSJ4KivfpPfTx6FLtzHif3txNEP/adxELQGTEbIIT0cGih2CQX23/7JR6t1+OhtvrE/F2C8PyI/VdefUHzvy0q8ncS6nCEM/3q10PLdQu3EpReVVWLjpNE5cbTj4tqK6Bv+k5aCoXPudipvTujNPfnMIZ7OL8K+vdZ/GTdSY4opqs++2rI/JCLVoX4yPQExXH8wb0aXRso/3bmeCiESg5ks69vN9uP/j3aiorsGmk9fx4i/JSM4saFBuW2o2MvNKUX9m7uZTN/DgZ3tRUd20KbvvbErDigMZeOSrgw3OfbgtHVN/PIqpJm5dMRfVNTKkZxdxxVvSWXFFNULf3o7o+J1ih6ITjhmhFm10Dz+M7uGnVdnJ0e2Vdslt6TLzaldhfWdTGlYfvgoAaOPacBO+TxPO4dOEc+jh76b3PQVBwO5zN9GtjYvGZ/1rUiYA4PBl7bsxWtLX9rzfT2H9iWuIGxmC54Z00unab/deQlFFNV58oLORotPO2exCbDp5HdOHdIKzfctcoNAcpV6r/YVC1aKH5owtI2SRnKWWk4c/9d1hVNWo/6quS0QA4EaB+tkwVVq2gBxVWPPk4+3pSuf+Ss3GsyuSEBW/Q6u6dLHzbC4S0nIMXq8Y1p+4BgD4ctcFna6TyQS8v/UMPt9xHtfydV/y35BG/G8fvtx1ER9sPatXPWK0Dl26WYy4dSnIzCs1+b0tFZMRshh21vf+unf1c5H/HBvWBs8N7ihGSM1KWRMW7Vqi8GWaW1SOfedrZwHJGvt+0TDtOL+0EsmZ+SrPTbs7e2rRX2cxY9UxFFdU4+CFW/dmBGnp4MVbDfb2uXyrBJ/8nY47Zvwbp+KnNJdF1k5fb9j1p61Xfj+JmE/3mPyzPLEsET8fuYpnVjRc2djcNdeePcv59ZAs3sG4oVj8z3k82c8fCzelyY9/OaEXAKC0slqs0JqFy7dK9Lq+LhHR1+APd6GwvBo//rsfBndurbLM13tqFxjbfjobMgFoZWeNT57ogRGhbeRlvt17CQ521pioYnDzU9/Wrsfy9+kc/Do9CgBw/92BuudyirDsafPZgiA9uwhujrbwcVHuYmuuX0qKfj1a25X3V+oNPBJhujFddevzXLx57+98cUU17pRUwt/D0WRxWBK2jJDF8HKS4t2xoeju56ryF29HOxtY6bhEOhlfblE5rt4uxZe7LqCgrAqF5bVJ46TlRzBrzXGN19Y1iJRU1mD6quPYd/4mamQCsgvK8f7WM5i/IRXVGvbMOXJ36q1iS4y6pfdNrbSyGltTbmD4//ai/wf6dXsJgoDX/jiFxXc3W1x/IkvnLiJjeuGXk9hy6oaoMUR+sAODPtyFizeLRY2jpWLLCJGCJ/r4Y+3dwZMA8NWEXth08jo8Wtkpja0gwyquqIaTwjieovJ7rVT93r/3RXs+R3lJeV2/oJ7+/gjmx3bF/SHe8mPaNCAkpGXrdB9TeGZFEo5oGODb2Aq7ik5fL5T/vZ8TE4wXfjkJABjW1Rshvi6aLm2ULvl9dkE5XB1s4WBn3eDcrDXHERseq1cs+qjbhmH/+Vvo1NpJtDhaKraMECl4a3Q3fPhYuPy9BMDSib3x/iNh4gVlAaZpOYVXm9k17V/bovH8huRrSu9l9foz4tadanCNlS7f7BrIZAI2nLjWeEEF+8/fwqAPdyqtupucma8xEQF066ZRNyZDcZdnY8vMK0Vk/A70/+CfRsuezS7Enyevm824GHX+PHkde8/dNNn9KqtlWi9waG6YjJBFcnVQPdXQ0c4GT/T1l7830HcQNSLx0m1k5pUiJatAYzKhabaPtgRB+bf1N9anKpwT8PORzAbXGOqvwe/HszD3l2SlY3V71mw/nY32r23BuGXKC6FN/P4wMvPK8JTCvkKKi9Ipqj/z5EJuMfbU+zIsqajGnyevq11Q7oFP9yi9T71WgF3p2q/421T77yZbheWax25tOnkdI/63D8//fAIL/lTePfrizWLMWHVMr0Gz+sq7O8A5M68Uz/98ApOWm2YQbG5ROULf3o45a5NNcj9DYzJCFmnhmO7oFeCGz8dHNFLy3tfQHO57YlSvr0/B6CX7TXIvxa/s349lIb+09gvkbx2mBheVVyEpIw8yHWbqqGrNqBs4/dxPxwDotrZKY2I+3YPJy4/I154AgFd+P4Xnfz6B//x8AqWV1RAE5d+l65b2B2r/9j/0xX48uyIJF3KbOFbibkZfUlGNr3ZfwCU9x1x8sfO8/Od1x2tbmTJulSA5Mx/PrDiCv1KzMWaJ6mRNG439eWqaarz2yFX0ejcBnyWcQ64OO2Qbwq9Jmaiskem0M7c5YTJCFqmNqwPWzRyAh7VcEA0AXhB5EamWzlCzbZqi5zsJSM7Mx44zDZORMV8ewKlrDX/TfvzrRDz+dSKeXn4Yj3x1AL8mZeLNDamo0jAgVlULi6bWt8ZaCbSluGPwlpTacTa702+i21vb8UK9lhpFil1aqmZTnc8pwqTlR7D+RBa+2HFevhFjpYo1af677Sw+3JaOoZ/saXAO0K5b6VRWPs7lNExm7vt4N8Z+eUC+kF+1jlO5Fa050vSxYXHrUwAAi3ecb3CuoKwKm05eR1ll07qWKqtl8kRIEARcyy9TmrKu6fkVlFbh4+3pKhPKssoaDP5wFz5NONekuAyFA1iJNGA3Tctz+VYJslV096jr+jipZk2Ts9m1g2kPXLgNADhxtbZcZ19nTOwfAImKvzz/qEh2KqplWneDVFbLYGej/ndITV/Bz644AhvrhtduSL6OCWr2blLsslK1VsszK5JwLb9MPi7i0q0SfDaup9L6MnWSMrSfhVRdI1MZ68P1Wjwqa2R4WENr2o+JGcgvrdJqN2dBEPDr0Ux88ne6xnJpNwrxyu8nMev+IAR4OKr8c66vqkaGaT8exZHLeXiiTzt8+K8ejV6jqKi8CpEf7EAPfzesmRaJmE/34OLNEvi4SHH49ZhGr3/rz1RsTL6OL3dfwOX42kHAZZU1KK6oxtCPd6Ooohqf7zgv6qq9TEaINND0Hz81T6WVNZigMP7C0I5czsP/Es5hyqAOeDa6g/y4qkQEuLemiTYeXXoA7o52as+XKvzWrdidkHajELvS9RtIeeZGIUaE+iodq7/Ka1JGHgRBwN+n780+EgQB8zek4MyNQmii2Fn0acI5vDIiRKu4TmWpHh+y59xNvLWxdkzJmJ5+CPRspbGe7/dfxntbzjR6v7q1T349moWHwttgyVO9UFpZjU/+PqfUOqGYo4S+vV2+h9O649c0JiM/JWagRibgmQEdkF9aCSepDXan30RJZQ0OXqxNfOvWP8nRclPKukRZEIBbxRXwcpKiz3sJKGliK40x8H9aIhX+MzQIw0K8MThY9aJaZMkkGncT3nTyOm6XVOLDbenILzPsaq2p1wo1dme9o7CY3wOf7ZX/vPJghsZ6r2uxdPziHedx5HKexjETWXfK0CFuq7zVCKhNFlYdUu76KKmoRm5hOTadvC7v1qpW2LLgq90XG42nMZMVBo6WavGlq00iUt/mu1PLl+y8gO817KSt7WaSpZXVeHPjaSzYlIaUrAL0fCcBj3x10GCzuQCgz3v/oLii2qwSEYAtI0QqvfRg47v8kmW6VVyB3u81Pv0UAH4/atqNF5u6Hoq2MzCeWJYIP1d7HIwb1qT71HlnUxr+TsvGndIqvDYyBJ6t7PB2vZkxdeNPDMGY3a3TfzqmcZyQOqWV1bCxslJqfa2qvpeQ/XG89u9OyrUCjdN127+2BZ+Pj0CVim60+K1n4KaiJe31dSk6x2tsTEaIiHSkaoCmKuuTdVtTRF+GGvCqyfWCcmTdKUU796Yvi777XC7u3F3DZOfZXJWzjJIMOKtod/pNfL37ImbcF4S27g549Y9TOH7lDvzcHLB6an/Y2zZcZE1b206rTgA15T8lFdXo/vZ2pTEfr/5+CmkKXVmKLVCz15zQGMPzP6s+v2xv7f5KAfWWsP/z5HWN9YmByQiRHva9cj/Ss4vw/NoTKK2sQXtPR2Tc5k6fVOvSTf3289HFXymmWy595urjWPlsvyZfrzjWQd3ibbdLDDdFddFftTsHJ6TlYHJ0e/nKvTcKatfmGBXWRu21x6/ewdImdBv9R02CANS2dgDKz+GXo8rr25RX6d7a0pxxzAiRDhY+3F3pvb+HI2K6+WDdzGiM6emn13/QRPqYsVrzPj2GlJ5dhD7vJRj1Hq/+YfiuhJLKmgbrf1TLBI0tBY9+dRAJOqw/UyfrjvpxOIqbck794SimrExqUKZ+cqLq2paELSNEOpgc3R5ujrYN+thDfF2w+MnGFlAjajn0WMpDVGLvZlwtE/Dvlfe2P1A3y0qdbm9tN3RIZoEtI0QmMETNVvdEZFqHL98WOwSTyy7UfxsFY2MyQmQC7o6q98Ihao60napqjjR1n7RU2g64rtuZWAxMRohMoJm2aBORBWnqUvWGwGSESEfd/Vw1nv95WiTu79Iab4zqaqKIiIiaNw5gJdJRkLcT1s2MRmsnqcrzUZ08EdXJ06BTLZc93Vu+qysRkTGIuRcXW0aImqBXgDv8PZq+6JM6Lz3QGfteub/B8WEh3rCScOM+ImqZmIwQmUC/Dh7yn9fNjEagpyP++1iYUpmdLw3Bf4YFq0xybKytcHrhCKQtHGH0WImITI3dNERGorgnxJN9A2BvY40+7d0R6NkKe+bVtn7ULewU1tYVHVs7NahjVJgvpgzsCABwsGv6ktVERI0Rs+FVp5aR+Ph49O3bF87OzvD29sbYsWORnp6u8ZqVK1dCIpEovezt7fUKmqg5iOzogecGd8RH/wqHtZUEj/Vup3Yb8/obYfm61P4bWfhwKHoHuhs9ViIiMenUMrJnzx7MmjULffv2RXV1NV5//XU8+OCDSEtLQ6tWqv+TBQAXFxelpEXCjm+yABKJBHFNnFGz55X7UFZZo3LHTS8nKW4VV2DGfZ2w7/xNhPi6YHw/fzy2NFHfkInIgon53axTMrJt2zal9ytXroS3tzeOHTuGwYMHq71OIpHA19e3aREStWAz7+uEr3ZfxPzYbkrHpTbWkNqo7pbZ+fIQZNwqQXg7N7w6IsQUYRIRGZVeA1gLCmp3HvTw8NBYrri4GIGBgfD398eYMWNw+vRpjeUrKipQWFio9CJqiV4ZEYKz745AZEdPra9xsbdFeDu3Bsc/G9fDgJEREZlOk5MRmUyGuXPnYsCAAQgNDVVbrkuXLli+fDk2btyIVatWQSaTITo6GllZWWqviY+Ph6urq/zl7+/f1DCJzJ69rWEGpj4S0a7BscFN2BNnwehujRciohan2QxgVTRr1iykpqZi7dq1GstFRUVh0qRJ6NmzJ4YMGYJ169ahdevWWLZsmdpr4uLiUFBQIH9lZqreSpmINPv4X+GYeV8nted3vjRE/vOoMF8sfrInJkYGYlwf/gJARKbTpGRk9uzZ2Lx5M3bt2oV27Rr+NqaJra0tIiIicOHCBbVlpFIpXFxclF5EpDuJRKJ27AkApenEPdq5YUzPtrCxtsKCh7ubIjwiMiPNZgVWQRAwe/ZsrF+/Hjt37kSHDh10vmFNTQ1SUlLQpk0bna8lIt1Iba0aTBuu78uneuHRiLaYHN1efkxxTZMuPs5qrw1rq3mfHiIibeiUjMyaNQurVq3CmjVr4OzsjOzsbGRnZ6Os7N6WzJMmTUJcXJz8/TvvvIO///4bly5dwvHjxzFx4kRcuXIFU6dONdynICIAwPqZ0UrvXextITSyZXBseBt8Oq6n2rEr3f00t0z++O9+OsVIRFSfTlN7ly5dCgC47777lI6vWLECzzzzDADg6tWrsLK6l+PcuXMH06ZNQ3Z2Ntzd3dG7d28cPHgQ3bpxkByRoUUE3Fsgzd/DAQAaaRdpnEQiQcfWrXDpZonK800ZJEtE5kci4hBWnZIRobFfsQDs3r1b6f1nn32Gzz77TKegiKjpEl4YjG/3XcJ/hgYDACI7eOBzPeoTIOCbp/vg7T9TMWVgB6w/cR2bTl4HwI37iMgwuDcNUQsT7OOMD/91b82R6CAvvesM8nbC6qmRAIChIT7yZISIWpDmMoCViJo3O2vt/8m3davt5hkVarrB5h6tGi5/T0QtH5MRIgvwzN2ZMu+O1X7K7ra5g/Dn7AEY1tVbbRnPu8nDe2NDEejpiJce6Ax3R1ut7/HHjGjMGRYsfz86vI081sZ8+VQvjAw1v20mYsM5U5BIV0xGiCzA26O74VDcMIzrG6D1Nc53l51XtXnW8mf6YGCQF95/JAwAMDEyEHvm3Y//DAvGgdeGan2P3oHueOGBzvL3EokECx7ujnfHql7Vua7lZGJkAGLD2+D/BndsUGZgkBcWP9lT6xgMba5CckVE2mEyQmQBJBIJfF3tDVbf0BAfrJraH353u3IU2aroCvp6Ym8cnR+Dtf8XqbFeqa3m/5I2/WcgFj7cHa/f3Q1ZcfZQHRcHG4zp2Vbl9T9N6Yez747A+H7aJ2W60nb2kpeTHe7vwplIRACTESIyMFtrK6x4ti+cpTaYGBmAC++PxIhQX3g5SZU2BFz5bF/5z/NjuyK0rQtmDFG/dD1QO45lcnR7ONqpH3uvaXpidCcvg+0FpE6Ah6NW5aYP6YQVz/bDCoXnQCQqfdcB0ANn0xCRwd3fxRspC4erPPfXnEFIzy7CEIX1SaYO6oipgxp2uejDzdEW+aVVBq1T0eIne2LO2mSlY6+PClHZMqTJ/V3Uj8khMqXGVms2JraMEJFJdW3jgrERbVWORaljiBmGnz8ZobZeZ3vl38M+H9+wbGPG9GyLQcHK06Z7+rtrHXtPfzf5z9MG6b61hr6+fKoXYrr6qD3/RB/d9h2j5k+LpcSMhskIkYWxuvtt2crOuN0VYhvcuTX2zLtP6Vhd/jPrviD07+CB/z4WhuS3HsDDPfxwOX5Ug+SiznN3B8oGejoipqs3vpvUR2W5fh08Gl0IbuOsAfjh3/3Qp72HQlz3LvppimmW148Nb4NvJ/XGQDXr0LwzRvUg4qb49IkejRci0YmYizAZIbI062YOQGRHD/zyXJTYoajlpjA9uLG9bxLjhmJ8P3+V5wI9W8l//veADvIvfVdHW/zyXBTG9Q2Am2PtDJ36OxwrdiP5uTngcvwo7H75Pnw3uS9iuvnIr6ljay1pcAxouFx+D383pbrrGxRsukGtEokEP03phwe6NWwhMdTYmj9mROHRXu3Qt/29wcauDtpP/zYHwd5OjRdqAbRZZd1YmIwQWZie/m5Y+39RCDXjHXdHhrbB473bYdGjYQj20fxF0MbVAfGPhsvft3NvOMMHAP7Vu/FuB3uF2TyKCVHvQHdIJJIGiYbiu7/mDFZZ5w/P9sX7j+jWyuDYhFarps4QkkjUD/l9UWHadVN8PbEXegfWtgB9N+neQF3FL73ufi6YOlC/bipNyV10J09kLIrVq/6hXb1x4s0HlBIqXfi6GG4mmzGxZYSISIG1lQQfPd4DT/YLQBtXB2z+z0Dse+V+jdf8NKUfnuzrj//UW+dj5n2d8K/e7dC1jXOj962bMgzUrlZ78LWh+GOG+sRNMTcJUvHbc7/2HpBIJAj0aNXgnFI99d7/8+KQRmOtL/7RMJ2vaczzw4JxasGD8vcvPtAZo8K0W2juvi6tMUJh9V5XheTO00kq/3njrAGY/5B+G6fOG95F7bnHejWehD43pCN8XKQqz/Xwd8MLMZ3h3soOX0/sjVn3a57xpcrDPf10vkYMYm41xWSEiMxeaFtX+DcyZXZQcGsseiwcTlLlwamvjAjBx4/30Dhgto6fmwMWPtwdnX2c8PLwLvBzc5D/Zq+Kv7vqmJLeiMGC0d3w7eTasSVujaxK61rvvJ+bA/49QPvWgmEhtTNyJkcFan2Ntlzs78VmbSWBg612kzA1JQER/m7Y8vxA/P3CYNhoOfuoXwf1fw6a/mgf7aV6zRlFcSO7ws5GdRyz7w+Sd1l5Okkxb3iI2nE26shk99ocvprQC7PvD1I67yxV/0xjunqrHcukizcf6gZrK/UPKqarj1KSaGpMRoiIFEyObo+/XxgCHy2a1l8e3gWP9mrbYNBpa2cpnhnQQT42orufC6YN6oA31bQAPBPdHsNCvPHhY/e6m8LbadeNtv/V+/HN3QG1Cx5WXu7/2QHt5T/7e6juvlJFXVIzMMgLMi3GFayZ1h8PqVgW/8/ZAzApKhBvje6G7n6u6Oxzr7XqjxnRSmW9nOywemp/+fu5w4LRO9C9XhkpVk3pD020SUIBYOmE3iqPR3ZsmAS9NjKk0foWKvxZKOQiGNHdFy/Xa8lJmh+jtp7RPfzwk4rPuPDh7g0Sb02mDOyAM++MaPAM63w6TtxBxlxnhIioiVwdbPHpEz0bLSeRSPBGrPquCEc7G3z/jPLiZ1p+h6KdQuuMRCLBimf74osd5/Hhv3ogyNsJKw5kAABa2dlgZKgv/krN1ljfnnn3NVi47ej8GFy7U4Ye/m4NkhEbKwmqZcrHojup/k0+vJ0bwtu5qTzXO9Ad6e+NwMoDGRjSpTVCfF0AAP99LAzp2cWI6uSJ3ztFoUPcVgCAj4sUh+KGQSKRIO16ocbPpI3Qtq7494AOWH7gsvzY6B5+cLZv2KrVsfW9bjcrCTDr/iDUyAR8tfsiACA2rA0mR7fH23+eBnBvcDPQ8M/V2d5G42Dh7n6u8nJF5dXy45Oj22NcX3+EvLlNqfyvz0XhiWWJKuuys7HCHzOisfnUdcxec0LpnLW2f+GMhMkIEZEZ6u7novH866NCVI5Tub+Lt14LqSnOQKrj5SSF190m/Hp5B6YP6YQluy40+X6KpDbWeK7eKrzq9lMKa+vaaKvH4xoGLUd29MChS3lKxxSr69/BA2+paclytLPB0fkxsLW2krd+VVbLEN7ODZEdPeQztDp4tcLlWyV4ZkB7FJRVwb2VXYOYVa1x087dAVl3ymBnbSX/M178ZE/8e+VRpXKqkph+HTwQNzIEX+2+iDau9jibXdSgTGxYG6wLuYadZ3PlxzR14ZgCkxEiIjMU5O2M36dHYXf6TZVf9qN7+KGNq/ZdL4BhFrVSHP/w+/QohLVzxZW8Umw6eV3/yrXw9wuD8UtSJmbe1/hA0kBP7ZbmrzMsxBvf77+MVnbWjU5996o3vsLOxgoj6u0i/fcLg1FSUQ03RzssUuiCa8y84V3Qv4MnvJzs5MdU7QOlznNDOmHaoI74MTEDCzalNTgvkUjw/LBgs0pGOGaEiMhM9WnvodRlsuvl+8QL5i7FLoc+7T0gtbHGwz3uzRZpbAyHvjr7OOPNh7ppNdhScd2Y6E6eSufqJxMAEB3khQ2zBmD/q9rvPK2JrbWVvJVEHa9WDeN4uIcffF3ttRrcOzHyXsvRlucHyn+2spJgYmQg3n8kFDteajg7S3HtlMd7t9N5GwNDY8sIEVEz4dHIF5sqY3v6YUPydfxnaDCCfZywPS0b0/TYB+iVESE4da0Ak6Paqzw/0AAzP3SlrrdmgsIX9ZKneuHZlUk4mZkPAHhrdDcUlVdjYqTyYF3FZfqNaemEXsi6U4YwFQOVVXU/qWsZWTC6O/p38ETf9h4Ndua2sbbChP6Nz7CaExPcaBljYzJCRGTGvNWsf6GtT5/oiVdHhsi7dM69N1Kv34L93Byw86X7lI6J28Cv2ojuvkq7O3u0ssO0QR3kAze9ne3xQyOr+xrTyLCGs400cbG3xdbnB2HFgct4VmHat421FUb3aB7rmGjCZISIyIwN6dwazw8NQrd6A1rVr5uqzMpKojS2ROzmeGNQt0ZIS9PNzwUfPd4y9/lhMkJEZMYkEglefLB2XYqSintTO6UG/AJubEE5c9fRqxUejWgL91Z2OHrlDk5m5uOJvg1n0oi5K21jJkcF4ofEK1ot0tYSMRkhImomWklt8NZD3VAjE+DeSvfxI+rMjQlGSUV1s23ul0gk+HRcTwBARXUNrt0pQ8fWDac9R90dxNrG1fz2ipn/UDeMCmuDngFuJrmfg8K0YG9n8Z+HRBBzmz4tFRYWwtXVFQUFBXBx0Tz3noiITCs9uwjD/7cXAPTelM7YCsqq4GBrbTFdO5qUVlZDJkCnlVx1pe33N1tGiIhIL118nbHkqQizbHGor26RMoLSAF+xmU8kRETUbD0U3jy7eMg8sJ2KiIiIRMVkhIiIiETFZISIiIhExWSEiIiIRMVkhIiIiETFZISIiIhExWSEiIiIRMVkhIiIiETFZISIiIhExWSEiIiIRMVkhIiIiETFZISIiIhExWSEiIiIRNUsdu0VBAEAUFhYKHIkREREpK267+2673F1mkUyUlRUBADw9/cXORIiIiLSVVFREVxdXdWelwiNpStmQCaT4fr163B2doZEIjFYvYWFhfD390dmZiZcXFwMVi8p43M2HT5r0+BzNg0+Z9Mw5nMWBAFFRUXw8/ODlZX6kSHNomXEysoK7dq1M1r9Li4u/ItuAnzOpsNnbRp8zqbB52waxnrOmlpE6nAAKxEREYmKyQgRERGJyqKTEalUirfffhtSqVTsUFo0PmfT4bM2DT5n0+BzNg1zeM7NYgArERERtVwW3TJCRERE4mMyQkRERKJiMkJERESiYjJCREREomIyQkRERKKy6GTkyy+/RPv27WFvb4/+/fvjyJEjYodktuLj49G3b184OzvD29sbY8eORXp6ulKZ8vJyzJo1C56ennBycsJjjz2GnJwcpTJXr15FbGwsHB0d4e3tjXnz5qG6ulqpzO7du9GrVy9IpVIEBQVh5cqVxv54ZmvRokWQSCSYO3eu/Bifs2Fcu3YNEydOhKenJxwcHBAWFoajR4/KzwuCgLfeegtt2rSBg4MDYmJicP78eaU68vLyMGHCBLi4uMDNzQ1TpkxBcXGxUplTp05h0KBBsLe3h7+/Pz788EOTfD5zUFNTgzfffBMdOnSAg4MDOnXqhHfffVdp0zQ+56bZu3cvRo8eDT8/P0gkEmzYsEHpvCmf62+//YaQkBDY29sjLCwMW7du1f0DCRZq7dq1gp2dnbB8+XLh9OnTwrRp0wQ3NzchJydH7NDM0vDhw4UVK1YIqampQnJysjBq1CghICBAKC4ulpeZPn264O/vL+zYsUM4evSoEBkZKURHR8vPV1dXC6GhoUJMTIxw4sQJYevWrYKXl5cQFxcnL3Pp0iXB0dFRePHFF4W0tDThiy++EKytrYVt27aZ9POagyNHjgjt27cXwsPDhTlz5siP8znrLy8vTwgMDBSeeeYZ4fDhw8KlS5eE7du3CxcuXJCXWbRokeDq6ips2LBBOHnypPDwww8LHTp0EMrKyuRlRowYIfTo0UM4dOiQsG/fPiEoKEgYP368/HxBQYHg4+MjTJgwQUhNTRV+/vlnwcHBQVi2bJlJP69Y3n//fcHT01PYvHmzcPnyZeG3334TnJychMWLF8vL8Dk3zdatW4U33nhDWLdunQBAWL9+vdJ5Uz3XAwcOCNbW1sKHH34opKWlCfPnzxdsbW2FlJQUnT6PxSYj/fr1E2bNmiV/X1NTI/j5+Qnx8fEiRtV85ObmCgCEPXv2CIIgCPn5+YKtra3w22+/ycucOXNGACAkJiYKglD7j8fKykrIzs6Wl1m6dKng4uIiVFRUCIIgCK+88orQvXt3pXuNGzdOGD58uLE/klkpKioSgoODhYSEBGHIkCHyZITP2TBeffVVYeDAgWrPy2QywdfXV/joo4/kx/Lz8wWpVCr8/PPPgiAIQlpamgBASEpKkpf566+/BIlEIly7dk0QBEH46quvBHd3d/lzr7t3ly5dDP2RzFJsbKzw73//W+nYo48+KkyYMEEQBD5nQ6mfjJjyuT7xxBNCbGysUjz9+/cXnnvuOZ0+g0V201RWVuLYsWOIiYmRH7OyskJMTAwSExNFjKz5KCgoAAB4eHgAAI4dO4aqqiqlZxoSEoKAgAD5M01MTERYWBh8fHzkZYYPH47CwkKcPn1aXkaxjroylvbnMmvWLMTGxjZ4FnzOhvHnn3+iT58+ePzxx+Ht7Y2IiAh8++238vOXL19Gdna20jNydXVF//79lZ6zm5sb+vTpIy8TExMDKysrHD58WF5m8ODBsLOzk5cZPnw40tPTcefOHWN/TNFFR0djx44dOHfuHADg5MmT2L9/P0aOHAmAz9lYTPlcDfV/iUUmI7du3UJNTY3Sf9YA4OPjg+zsbJGiaj5kMhnmzp2LAQMGIDQ0FACQnZ0NOzs7uLm5KZVVfKbZ2dkqn3ndOU1lCgsLUVZWZoyPY3bWrl2L48ePIz4+vsE5PmfDuHTpEpYuXYrg4GBs374dM2bMwPPPP48ffvgBwL3npOn/iOzsbHh7eyudt7GxgYeHh05/Fi3Za6+9hieffBIhISGwtbVFREQE5s6diwkTJgDgczYWUz5XdWV0fe42OpUmQu1v7ampqdi/f7/YobQ4mZmZmDNnDhISEmBvby92OC2WTCZDnz598MEHHwAAIiIikJqaiq+//hqTJ08WObqW49dff8Xq1auxZs0adO/eHcnJyZg7dy78/Pz4nEmJRbaMeHl5wdrausEMhJycHPj6+ooUVfMwe/ZsbN68Gbt27UK7du3kx319fVFZWYn8/Hyl8orP1NfXV+UzrzunqYyLiwscHBwM/XHMzrFjx5Cbm4tevXrBxsYGNjY22LNnDz7//HPY2NjAx8eHz9kA2rRpg27duikd69q1K65evQrg3nPS9H+Er68vcnNzlc5XV1cjLy9Ppz+LlmzevHny1pGwsDA8/fTTeOGFF+StfnzOxmHK56qujK7P3SKTETs7O/Tu3Rs7duyQH5PJZNixYweioqJEjMx8CYKA2bNnY/369di5cyc6dOigdL53796wtbVVeqbp6em4evWq/JlGRUUhJSVF6R9AQkICXFxc5F8MUVFRSnXUlbGUP5dhw4YhJSUFycnJ8lefPn0wYcIE+c98zvobMGBAg6np586dQ2BgIACgQ4cO8PX1VXpGhYWFOHz4sNJzzs/Px7Fjx+Rldu7cCZlMhv79+8vL7N27F1VVVfIyCQkJ6NKlC9zd3Y32+cxFaWkprKyUv2asra0hk8kA8Dkbiymfq8H+L9FpuGsLsnbtWkEqlQorV64U0tLShP/7v/8T3NzclGYg0D0zZswQXF1dhd27dws3btyQv0pLS+Vlpk+fLgQEBAg7d+4Ujh49KkRFRQlRUVHy83VTTh988EEhOTlZ2LZtm9C6dWuVU07nzZsnnDlzRvjyyy8tasqpKoqzaQSBz9kQjhw5ItjY2Ajvv/++cP78eWH16tWCo6OjsGrVKnmZRYsWCW5ubsLGjRuFU6dOCWPGjFE5NTIiIkI4fPiwsH//fiE4OFhpamR+fr7g4+MjPP3000Jqaqqwdu1awdHRsUVPOVU0efJkoW3btvKpvevWrRO8vLyEV155RV6Gz7lpioqKhBMnTggnTpwQAAiffvqpcOLECeHKlSuCIJjuuR44cECwsbERPv74Y+HMmTPC22+/zam9uvriiy+EgIAAwc7OTujXr59w6NAhsUMyWwBUvlasWCEvU1ZWJsycOVNwd3cXHB0dhUceeUS4ceOGUj0ZGRnCyJEjBQcHB8HLy0t46aWXhKqqKqUyu3btEnr27CnY2dkJHTt2VLqHJaqfjPA5G8amTZuE0NBQQSqVCiEhIcI333yjdF4mkwlvvvmm4OPjI0ilUmHYsGFCenq6Upnbt28L48ePF5ycnAQXFxfh2WefFYqKipTKnDx5Uhg4cKAglUqFtm3bCosWLTL6ZzMXhYWFwpw5c4SAgADB3t5e6Nixo/DGG28oTRXlc26aXbt2qfw/efLkyYIgmPa5/vrrr0Lnzp0FOzs7oXv37sKWLVt0/jwSQVBYCo+IiIjIxCxyzAgRERGZDyYjREREJComI0RERCQqJiNEREQkKiYjREREJComI0RERCQqJiNEREQkKiYjREREJComI0RERCQqJiNEREQkKiYjREREJKr/B9trBnQJq2cCAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(lossi);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Wengerofo'dsssit ey\n",
      "KIN d pe wither vouprrouthercc.\n",
      "hathe; d!\n",
      "My hind tt hinig t ouchos tes; st yo hind wotte grotonear 'so it t jod weancotha:\n",
      "h hay.JUCle n prids, r loncave w hollular s O:\n",
      "HIs; ht anjx?\n",
      "\n",
      "DUThinqunt.\n",
      "\n",
      "LaZAnde.\n",
      "athave l.\n",
      "KEONH:\n",
      "ARThanco be y,-hedarwnoddy scace, tridesar, wnl'shenous s ls, theresseys\n",
      "PlorseelapinghiybHen yof GLUCEN t l-t E:\n",
      "I hisgothers je are!-e!\n",
      "QLYotouciullle'z\n"
     ]
    }
   ],
   "source": [
    "# sampling from the model\n",
    "idx = torch.zeros((1,1), dtype = torch.long)\n",
    "generated = m.generate(idx, 400) # shape (1, 101)\n",
    "print(decode(generated[0].tolist()))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3- Porting our code to a script\n",
    "check `bigram.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: bigram.py [-h] [--batch_size BATCH_SIZE] [--block_size BLOCK_SIZE]\n",
      "                 [--max_iters MAX_ITERS] [--eval_interval EVAL_INTERVAL]\n",
      "                 [--learning_rate LEARNING_RATE] [--eval_iters EVAL_ITERS]\n",
      "\n",
      "Process some integers.\n",
      "\n",
      "options:\n",
      "  -h, --help            show this help message and exit\n",
      "  --batch_size BATCH_SIZE\n",
      "                        how many independent sequences will we process in\n",
      "                        parallel (default: 4)\n",
      "  --block_size BLOCK_SIZE\n",
      "                        what is the maximum context length for predictions?\n",
      "                        (default: 8)\n",
      "  --max_iters MAX_ITERS\n",
      "                        how many training iterations do we want? (default:\n",
      "                        3000)\n",
      "  --eval_interval EVAL_INTERVAL\n",
      "                        how often do we evaluate the loss on train and val?\n",
      "                        (default: 300)\n",
      "  --learning_rate LEARNING_RATE\n",
      "                        what is the learning rate for the optimizer? (default:\n",
      "                        0.01)\n",
      "  --eval_iters EVAL_ITERS\n",
      "                        how many batches we average the loss over for train\n",
      "                        and val (default: 200)\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " step 0 | train loss: 4.7303 | val loss: 4.7320\n",
      " step 300 | train loss: 3.2123 | val loss: 3.2111\n",
      " step 600 | train loss: 2.7370 | val loss: 2.7890\n",
      " step 900 | train loss: 2.5998 | val loss: 2.5870\n",
      " step 1200 | train loss: 2.5524 | val loss: 2.5206\n",
      " step 1500 | train loss: 2.5420 | val loss: 2.5416\n",
      " step 1800 | train loss: 2.5044 | val loss: 2.5503\n",
      " step 2100 | train loss: 2.4710 | val loss: 2.5091\n",
      " step 2400 | train loss: 2.4881 | val loss: 2.5257\n",
      " step 2700 | train loss: 2.4807 | val loss: 2.5065\n",
      "\n",
      "\n",
      "\n",
      "CEThik brid owindakis s, bth\n",
      "\n",
      "HAPONThobe d e.\n",
      "S:\n",
      "O:3 my d?\n",
      "LUCous:\n",
      "Wanthainusqur, vet?\n",
      "F dXENDoate awice my.\n",
      "\n",
      "HDYoncom oroup\n",
      "Yowh$Frtof isth be V! Whedilin,\n",
      "\n",
      "W:\n",
      "\n",
      "Ye sengmin lat Hetidrovets, and Win nghirileranousel lind pe l.\n",
      "HAule cechiry:\n",
      "Supr aisspll, y.\n",
      "Hentous n Boopetelaves\n",
      "MPOLI s, d mothakeeo Windo wh t pisbys the m dourive ce shiend t so mower; te\n",
      "\n",
      "AN ad nthrupief sor; iris! m:\n",
      "\n",
      "Thiny aleronth,\n",
      "MadPre?\n",
      "\n",
      "WISo myr fursuk!\n",
      "KENob&y, wardsal this ghesthidin cour ay aney Iry ts I&fr t ce.\n",
      "J\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3- Self Attention Trick (Averaging the previous tokens embeddings)\n",
    "\n",
    "- we need each token commuinicate with all previous tokens, for example: the 5th token communicate with 1st, 2nd, 3rd, 4th tokens\n",
    "\n",
    "- since we are predicting the next token, we need to consider the previous tokens only\n",
    "\n",
    "- The easiset way to make them communicate is averaging the previous tokens embeddings (it's kinda lossy since we are losing the spatial information)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1- Using Explicit loops "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x shape: torch.Size([4, 8, 2])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1337)\n",
    "B, T, C = 4, 8, 2 # batch size, time, channels\n",
    "x = torch.randn(B, T, C)\n",
    "print(f\"x shape: {x.shape}\")\n",
    "\n",
    "# We want x[b, t] = mean_(i<=t) x[b, i]\n",
    "xbow = torch.zeros((B, T, C))\n",
    "for b in range(B):\n",
    "    for t in range(T):\n",
    "        xprev = x[b, :t+1] # (t, C)\n",
    "        xbow[b, t] = torch.mean(xprev, dim = 0) # average over time dimension (t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x[0]: tensor([[ 0.1808, -0.0700],\n",
      "        [-0.3596, -0.9152],\n",
      "        [ 0.6258,  0.0255],\n",
      "        [ 0.9545,  0.0643],\n",
      "        [ 0.3612,  1.1679],\n",
      "        [-1.3499, -0.5102],\n",
      "        [ 0.2360, -0.2398],\n",
      "        [-0.9211,  1.5433]])\n",
      "xbow[0]: tensor([[ 0.1808, -0.0700],\n",
      "        [-0.0894, -0.4926],\n",
      "        [ 0.1490, -0.3199],\n",
      "        [ 0.3504, -0.2238],\n",
      "        [ 0.3525,  0.0545],\n",
      "        [ 0.0688, -0.0396],\n",
      "        [ 0.0927, -0.0682],\n",
      "        [-0.0341,  0.1332]])\n",
      "tensor([True, True])\n",
      "tensor([True, True])\n"
     ]
    }
   ],
   "source": [
    "# Let's Check the first Batch\n",
    "print(f\"x[0]: {x[0]}\")\n",
    "print(f\"xbow[0]: {xbow[0]}\")\n",
    "\n",
    "# the first row is the same \n",
    "print(x[0, 0] == xbow[0, 0])\n",
    "# the second row is the average of the first two rows\n",
    "print((x[0, 0] + x[0, 1]) / 2 == xbow[0, 1])\n",
    "# etc ..."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2- Using Matrix Multiplication\n",
    "- instead of nested loops, we can make it using matrix multiplication\n",
    "- This can be done by multiplying the matrix with lower triangular matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a (shape = torch.Size([3, 3])) =\n",
      "tensor([[1.0000, 0.0000, 0.0000],\n",
      "        [0.5000, 0.5000, 0.0000],\n",
      "        [0.3333, 0.3333, 0.3333]])\n",
      "b (shape = torch.Size([3, 2])) =\n",
      "tensor([[2., 7.],\n",
      "        [6., 4.],\n",
      "        [6., 5.]])\n",
      "c (shape = torch.Size([3, 2])) =\n",
      "tensor([[2.0000, 7.0000],\n",
      "        [4.0000, 5.5000],\n",
      "        [4.6667, 5.3333]])\n"
     ]
    }
   ],
   "source": [
    "\n",
    "torch.manual_seed(42)\n",
    "# lower triangular matrix of ones\n",
    "a = torch.tril(torch.ones(3,3)) \n",
    "# make all rows sum to 1\n",
    "a = a / torch.sum(a, 1, keepdim = True)\n",
    "# create a random matrix\n",
    "b = torch.randint(0, 10, (3, 2)).float() \n",
    "\n",
    "c = a @ b\n",
    "print(f\"a (shape = {a.shape}) =\\n{a}\")\n",
    "print(f\"b (shape = {b.shape}) =\\n{b}\")\n",
    "print(f\"c (shape = {c.shape}) =\\n{c}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "# We want x[b, t] = mean_(i<=t) x[b, i]\n",
    "wei = torch.tril(torch.ones(T, T)) # (T, T)\n",
    "# make all rows sum to 1\n",
    "wei = wei / torch.sum(wei, 1, keepdim = True) # (T, T)\n",
    "xbow2 = wei @ x # (T, T) @ (B, T, C) ----broadcasting----> (B, T, T) @ (B, T, C) ➡️ (B, T, C)\n",
    "\n",
    "# check if xbow2 is the same as xbow\n",
    "print(torch.allclose(xbow, xbow2, atol = 1e-7))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3- Using Softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wei:\n",
      "tensor([[0., -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
      "        [0., 0., -inf, -inf, -inf, -inf, -inf, -inf],\n",
      "        [0., 0., 0., -inf, -inf, -inf, -inf, -inf],\n",
      "        [0., 0., 0., 0., -inf, -inf, -inf, -inf],\n",
      "        [0., 0., 0., 0., 0., -inf, -inf, -inf],\n",
      "        [0., 0., 0., 0., 0., 0., -inf, -inf],\n",
      "        [0., 0., 0., 0., 0., 0., 0., -inf],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0.]])\n",
      "wei:\n",
      "tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.5000, 0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.3333, 0.3333, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.2000, 0.2000, 0.2000, 0.2000, 0.2000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.0000, 0.0000],\n",
      "        [0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.0000],\n",
      "        [0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250]])\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "tril = torch.tril(torch.ones(T, T))\n",
    "# we start with zeros, but later these will be replaced with data dependent values (affinities)\n",
    "wei = torch.zeros((T, T))\n",
    "# masked_fill: for all elements where tril == 0, replace with float(\"-inf\")\n",
    "wei = wei.masked_fill(tril == 0, float(\"-inf\"))\n",
    "print(f\"wei:\\n{wei}\")\n",
    "wei = F.softmax(wei, dim = -1)\n",
    "print(f\"wei:\\n{wei}\")\n",
    "xbow3 = wei @ x\n",
    "\n",
    "# check if xbow3 is the same as xbow\n",
    "print(torch.allclose(xbow, xbow3, atol = 1e-7))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4- Minor Code Cleanup\n",
    "(these modifications are done in `bigram.py`)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1- Adding variable embedding size\n",
    "1. removing vocab_size from the constructor of `BigramLanguageModel` class, since it's already defiend above\n",
    "2. Modifying the embedding layer to has an output size of `n_embed` instead of `vocab_size`\n",
    "3. Adding a linear layer with `vocab_size` outputs after the embedding layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BigramLanguageModel(nn.Module):\n",
    "    # no need to pass vocab_size as an argument, since it is a global variable in this file\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a loockup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embed)\n",
    "        # the output layer is a linear layer with vocab_size outputs\n",
    "        self.lm_head = nn.Linear(n_embed, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets = None):\n",
    "        # idx and targets are both (B, T) tensor of ints\n",
    "        token_emb = self.token_embedding_table(idx) # (B, T, C) = (4, 8 , vocab_size)\n",
    "        logits = self.lm_head(token_emb) # (B, T, vocab_size) = (4, 8, vocab_size)\n",
    "    \n",
    "    # rest of the code .."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2- Positional Encoding\n",
    "1. Adding a positional encoding layer to the model `self.position_embedding_table`\n",
    "2. Adding the positional encoding to the input embeddings `x = token_emb + pos_emb`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BigramLanguageModel(nn.Module):\n",
    "    # no need to pass vocab_size as an argument, since it is a global variable in this file\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a loockup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embed)\n",
    "        # each position is also associated with an embedding vector\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embed)\n",
    "        # the output layer is a linear layer with vocab_size outputs\n",
    "        self.lm_head = nn.Linear(n_embed, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets = None):\n",
    "        B, T = idx.shape\n",
    "        # idx and targets are both (B, T) tensor of ints\n",
    "        token_emb = self.token_embedding_table(idx) # (B, T, C) = (4, 8 , vocab_size)\n",
    "        pos_emb = self.position_embedding_table(torch.arange(T, device = idx.device)) # (T, C) = (8, vocab_size)\n",
    "        # x has the token identities + the position embeddings\n",
    "        x = token_emb + pos_emb # (B, T, C) = (4, 8, vocab_size)\n",
    "        logits = self.lm_head(x) # (B, T, vocab_size) = (4, 8, vocab_size)\n",
    "    \n",
    "    # rest of the code .."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5- Self Attention\n",
    "Consider his as `version 4 ` of part 3"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1- Previous code from part 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tril:\n",
      "tensor([[1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [1., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [1., 1., 1., 0., 0., 0., 0., 0.],\n",
      "        [1., 1., 1., 1., 0., 0., 0., 0.],\n",
      "        [1., 1., 1., 1., 1., 0., 0., 0.],\n",
      "        [1., 1., 1., 1., 1., 1., 0., 0.],\n",
      "        [1., 1., 1., 1., 1., 1., 1., 0.],\n",
      "        [1., 1., 1., 1., 1., 1., 1., 1.]])\n",
      "wei:\n",
      "tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.5000, 0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.3333, 0.3333, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.2000, 0.2000, 0.2000, 0.2000, 0.2000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.0000, 0.0000],\n",
      "        [0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.0000],\n",
      "        [0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250]])\n",
      "out.shape:\n",
      "torch.Size([4, 8, 32])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1337)\n",
    "B, T, C = 4, 8, 32 # batch, time, channels\n",
    "x = torch.randn(B, T, C)\n",
    "\n",
    "tril = torch.tril(torch.ones(T, T))\n",
    "wei = torch.zeros((T, T))\n",
    "wei = wei.masked_fill(tril == 0, float(\"-inf\"))\n",
    "wei = F.softmax(wei, dim = -1)\n",
    "\n",
    "out = wei @ x\n",
    "\n",
    "print(f\"tril:\\n{tril}\")\n",
    "print(f\"wei:\\n{wei}\")\n",
    "print(f\"out.shape:\\n{out.shape}\")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2- Building the Self-Attention\n",
    "each token (can be called node too) at each position emmits 2 vectors:\n",
    "1. Query: What I'm looking for?\n",
    "2. Key: What do I contain?\n",
    "3. Value: What I will tell you or the information I have `in this head`?\n",
    "    \n",
    "- affinities between tokens `wei` = my Query @ all Keys\n",
    "- if key and query are aligned ➡️ high value ➡️ learn more about this sequence\n",
    "- instead of multiplying wei with tokens directly, we multiply it with values (which is the information we want to learn about)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wei[0]: tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.1574, 0.8426, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.2088, 0.1646, 0.6266, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.5792, 0.1187, 0.1889, 0.1131, 0.0000, 0.0000, 0.0000, 0.0000],\n",
      "        [0.0294, 0.1052, 0.0469, 0.0276, 0.7909, 0.0000, 0.0000, 0.0000],\n",
      "        [0.0176, 0.2689, 0.0215, 0.0089, 0.6812, 0.0019, 0.0000, 0.0000],\n",
      "        [0.1691, 0.4066, 0.0438, 0.0416, 0.1048, 0.2012, 0.0329, 0.0000],\n",
      "        [0.0210, 0.0843, 0.0555, 0.2297, 0.0573, 0.0709, 0.2423, 0.2391]],\n",
      "       grad_fn=<SelectBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1337)\n",
    "B, T, C = 4, 8, 32 # batch, time, channels\n",
    "\n",
    "# x is private information of each token\n",
    "x = torch.randn(B, T, C)\n",
    "\n",
    "# single Head perform self-attention\n",
    "head_size = 16\n",
    "key = nn.Linear(C, head_size, bias = False)\n",
    "query = nn.Linear(C, head_size, bias = False)\n",
    "value = nn.Linear(C, head_size, bias = False)\n",
    "\n",
    "\n",
    "k = key(x) # (B, T, head_size) = (4, 8, 16)\n",
    "q = query(x) # (B, T, head_size) = (4, 8, 16)\n",
    "\n",
    "# now every token in every batch is associated with a key and a query (in parallel), no communication between tokens has happened yet\n",
    "\n",
    "wei = q @ k.transpose(-2, -1) # (B, T, head_size) @ (B, head_size, T) = (B, T, T)\n",
    "tril = torch.tril(torch.ones(T, T))\n",
    "# wei are no longer zeros, but data dependent values (affinities)\n",
    "# wei = torch.zeros((T, T))\n",
    "wei = wei.masked_fill(tril == 0, float(\"-inf\"))\n",
    "wei = F.softmax(wei, dim = -1)\n",
    "\n",
    "print(f\"wei[0]: {wei[0]}\")\n",
    "\n",
    "# multiply with value instead of x\n",
    "v = value(x) # (B, T, head_size) = (4, 8, 16)\n",
    "out = wei @ v # (B, T, T) @ (B, T, head_size) = (B, T, head_size)\n",
    "# out = wei @ x\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3- Notes About Self Attention\n",
    "\n",
    "1. Attention is a **communication mechanism**. Can be seen as nodes in a directed graph looking at each other and aggregating information with a weighted sum from all nodes that point to them, with data-dependent weights.\n",
    "\n",
    "2. There is no notion of space. Attention simply acts over a set of vectors. **This is why we need to positionally encode tokens.**\n",
    "\n",
    "3. Each example across batch dimension is of course processed completely **independently** and **never \"talk\" to each other**\n",
    "\n",
    "4. In an `encoder` attention block just delete the single line that does masking with `tril`, **allowing all tokens to communicate**, it can be used for some applications like translation and sentiment analysis. This block here is called a \"decoder\" attention block because it has triangular masking, and is usually used in autoregressive settings, like **language modeling**.\n",
    "\n",
    "5. `self-attention` just means that the **keys** and **values** are **produced from the same source** as **queries**. In \"cross-attention\", the **queries** still get produced from **x**, but the **keys** and **values** come from some other, external source (e.g. an **encoder** module)\n",
    "\n",
    "6. `Scaled Dot-Product Attention`: $\\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)V$\n",
    "\n",
    "    `Scaled` attention additional divides `wei` by $\\frac{1}{\\sqrt{\\text{head\\_size}}}$. This makes it so when input `Q`, `K` are unit variance, `wei` will be **unit variance** too and Softmax will stay diffuse and **not saturate** too much. it's important especially in initialization.\n",
    "\n",
    "    if the **variance** is **very high**, **softmax** will **converge** to **one-hot vector**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unscaled Attention\n",
      "var(k) = 1.0104080438613892\n",
      "var(q) = 1.0203500986099243\n",
      "var(wei) = 17.684080123901367\n",
      "\n",
      "Scaled Attention\n",
      "var(k) = 1.0104080438613892\n",
      "var(q) = 1.0203500986099243\n",
      "var(wei) = 1.1052550077438354\n"
     ]
    }
   ],
   "source": [
    "# Scaled Attention\n",
    "k = torch.randn(B, T, head_size)\n",
    "q = torch.randn(B, T, head_size)\n",
    "\n",
    "print(\"Unscaled Attention\")\n",
    "wei = q @ k.transpose(-2, -1)\n",
    "print(f\"var(k) = {torch.var(k)}\")\n",
    "print(f\"var(q) = {torch.var(q)}\")\n",
    "print(f\"var(wei) = {torch.var(wei)}\")\n",
    "\n",
    "print(\"\\nScaled Attention\")\n",
    "wei = q @ k.transpose(-2, -1) * (head_size ** -0.5)\n",
    "print(f\"var(k) = {torch.var(k)}\")\n",
    "print(f\"var(q) = {torch.var(q)}\")\n",
    "print(f\"var(wei) = {torch.var(wei)}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4- Adding single Self Attention Head to the Bigram Language Model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.1- Making new `Head` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making the Head Class\n",
    "class Head(nn.Module):\n",
    "    \"\"\" one head of self attention \"\"\"\n",
    "    \n",
    "    def __init__(self, head_size):\n",
    "        super().__init__()\n",
    "        self.key = nn.Linear(n_embed, head_size, bias = False)\n",
    "        self.query = nn.Linear(n_embed, head_size, bias = False)\n",
    "        self.value = nn.Linear(n_embed, head_size, bias = False)\n",
    "        # since tril isn't a parameter, we register it as a buffer\n",
    "        self.register_buffer(\"tril\", torch.tril(torch.ones(block_size, block_size)))\n",
    "    \n",
    "    def forward(self, x):\n",
    "        B, T, C = x.shape\n",
    "        k = self.key(x) # (B, T, C)\n",
    "        q = self.query(x) # (B, T, C)\n",
    "\n",
    "        # compute attention scores (affinities)\n",
    "        wei = q @ k.transpose(-2, -1) * (C ** -0.5) # (B, T, C) @ (B, C, T) = (B, T, T)\n",
    "        wei = wei.masked_fill(self.tril[:T, :T] == 0 , float(\"-inf\")) # (B, T, T)\n",
    "        wei = F.softmax(wei, dim = -1) # (B, T, T)\n",
    "\n",
    "        # perform weighted aggregation of the values\n",
    "        v = self.value(x) # (B, T, C)\n",
    "        out = wei @ v # (B, T, T) @ (B, T, C) = (B, T, C)\n",
    "        return out"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.2- Modifying `BigramLanguageModel` class\n",
    "1. Adding `Head` to the `BigramLanguageModel` class\n",
    "2. Adding `Head` to the `BigramLanguageModel` forward pass\n",
    "3. for `generate` function, we need crop idx to keep `idx.shape <= block_size`, since we are using `positional embedding`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding Head to the BigramLanguageModel\n",
    "\n",
    "# super simple bigram model\n",
    "class BigramLanguageModel(nn.Module):\n",
    "    # no need to pass vocab_size as an argument, since it is a global variable in this file\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a loockup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embed)\n",
    "        # each position is also associated with an embedding vector\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embed)\n",
    "        # a single head of self attention\n",
    "        self.sa_head = Head(n_embed)\n",
    "        # the output layer is a linear layer with vocab_size outputs\n",
    "        self.lm_head = nn.Linear(n_embed, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets = None):\n",
    "        B, T = idx.shape\n",
    "        # idx and targets are both (B, T) tensor of ints\n",
    "        token_emb = self.token_embedding_table(idx) # (B, T, C) = (4, 8 , vocab_size)\n",
    "        pos_emb = self.position_embedding_table(torch.arange(T, device = idx.device)) # (T, C) = (8, vocab_size)\n",
    "        # x has the token identities + the position embeddings\n",
    "        x = token_emb + pos_emb # (B, T, C) = (4, 8, vocab_size)\n",
    "        # feed the input to the self attention head\n",
    "        x = self.sa_head(x) # (B, T, C) = (4, 8, vocab_size)\n",
    "        logits = self.lm_head(x) # (B, T, vocab_size) = (4, 8, vocab_size)\n",
    "\n",
    "        if targets is None:\n",
    "            loss = None\n",
    "        else:\n",
    "            # note that F.cross_entropy accepts inputs in shape (B, C, T)\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B * T, C)\n",
    "            targets = targets.view(B * T) # can be as targets = targets.view(-1)\n",
    "            \n",
    "            loss = F.cross_entropy(logits, targets)\n",
    "\n",
    "        return logits, loss\n",
    "\n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        # idx is (B, T) array of indices in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            # crop idx to the last block_size tokens\n",
    "            idx_cond = idx[:, -block_size:] # (B, T)\n",
    "            # get the logits for the next token\n",
    "            logits, loss = self(idx_cond)\n",
    "            # focus only on the last time step\n",
    "            # (note that we are feeding the whole context each time, however we only care about the last prediction)\n",
    "            # (this make doesn't make sense now, but the function will be modified later)\n",
    "            logits = logits[:, -1, :] # Becomes (B, C) (get the last time step for each sequence)\n",
    "            # apply softmax to convert to probabilities\n",
    "            probs = F.softmax(logits, dim = -1) # (B, C)\n",
    "            # sample from the distribution\n",
    "            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)\n",
    "            # append sampled token to the context\n",
    "            idx = torch.cat((idx, idx_next), dim = 1) # (B, T + 1)\n",
    "        return idx"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.3- Testing\n",
    "(loss is `2.54` instead of `2.57`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " step 0 | train loss: 4.1985 | val loss: 4.2013\n",
      " step 2999 | train loss: 2.5560 | val loss: 2.5439\n",
      "\n",
      "Whent ak tridcowf, h is t, bte\n",
      "\n",
      "Hiset bube tantavegr-ans thalitanss ar hthaf uwqhe he.\n",
      "War dthas ate awice my.\n",
      "\n",
      "HDEUncom orour\n",
      "Yowts\n",
      "Mtof itichice mil ndilincaes ireeesen cin latistt drov ts, anen k pngan ileranses lhin dite lltenser cechiry ptur haiss hawty.\n",
      "Hllin.\n",
      "I yradpetelaves\n",
      "Mk:\n",
      "I I the tthakleo W-SNAn brt eiibys the m dourive cens ireds pso mowrif te\n",
      "Thak dmnterurt f so;; irist m:\n",
      "EE:\n",
      "I inlerontatise Pre?\n",
      "\n",
      "Wy om.\n",
      "\n",
      "HK-NLI\n",
      "!\n",
      "Ktiybyis:\n",
      "Sadsal this ghest huin cst amaraney Iry ts chan tf ve y\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py --eval_interval 2999"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.5- Multi-Head Attention\n",
    "- make the new `MultiHeadAttention` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    \"\"\" multiple heads of self-attention in parallel \"\"\"\n",
    "\n",
    "    def __init__(self, num_head, head_size):\n",
    "        super().__init__()\n",
    "        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_head)])\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # concatenate them into the channel dimension\n",
    "        return torch.cat([h(x) for h in self.heads], dim = -1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "then add it to `BigramLanguageModel` class \n",
    "\n",
    "previously:\n",
    "```python\n",
    "self.sa_head = Head(n_embed)\n",
    "```\n",
    "now:\n",
    "```python\n",
    "self.sa_heads = MultiHeadAttention(num_head = 4, head_size = n_embed // 4)\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5.1- Testing\n",
    "(loss is `2.51` instead of `2.54`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " step 0 | train loss: 4.2230 | val loss: 4.2180\n",
      " step 2999 | train loss: 2.5103 | val loss: 2.5133\n",
      "\n",
      "Whent ak tridcowd,\n",
      "T, son, bte\n",
      "\n",
      "Hiret bobe toe.\n",
      "S\n",
      "BO-EIS\n",
      "BF\n",
      "YLTanss arththat uwqhe he.\n",
      "War dilthoate awice my.\n",
      "\n",
      "HDEE:\n",
      "D zorou thowts\n",
      "Mtof is hy ce mil; dilincaes iree sen cin;\n",
      "SWiHmaclrev te anden k ther tilerans!\n",
      "el lindite let-hulr cechiry ptur; niss hiwty.\n",
      "H'n nis kormopetelavle\n",
      "Mom\n",
      "Dy wod methakleo W-ndo whde eiibas the m dourive cen\n",
      "DKen\n",
      "CUC-LOK\n",
      "ANNNTur\n",
      "Te kadmn,\n",
      "ruft f sor; igist m:\n",
      "EE\n",
      "CI I ledont tis derred my om.\n",
      "\n",
      "HKINLI\n",
      "S\n",
      "K,\n",
      "Sb&is:\n",
      "Sadsad thes and thiuin cst asaratny Iry tome fr th vouy\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py --eval_interval 2999"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6- Adding FeedForward Layer \n",
    "the feedforward is applied to each token independently"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeedForward(nn.Module):\n",
    "    \"\"\" a simple linear layer followed by a non-linearity \"\"\"\n",
    "    def __init__(self, n_embed):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(n_embed, n_embed),\n",
    "            nn.ReLU(),\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return self.net(x)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# then add it to forward of the BigramLanguageModel\n",
    "``` python\n",
    "# previous code ..\n",
    "x = self.sa_heads(x) # (B, T, C) = (4, 8, vocab_size)\n",
    "# feed the output of the self attention head to the feed forward layer\n",
    "x = self.ff(x) # (B, T, C) = (4, 8, vocab_size)\n",
    "logits = self.lm_head(x) # (B, T, vocab_size) = (4, 8, vocab_size)\n",
    "# rest of the code ..\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1- Test\n",
    "(loss is `2.46` instead of `2.51`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " step 0 | train loss: 4.2016 | val loss: 4.2023\n",
      " step 2999 | train loss: 2.4432 | val loss: 2.4681\n",
      "\n",
      "And;\n",
      "BRIR\n",
      "INwcow afn is sorst mad set bobe uweravegr-ans meagatanss:\n",
      "Wanthae uwqhe hent?\n",
      "F dilthoate awice my thanstacum onou,\n",
      "Yowts\n",
      "Mtof ist cice mil nowlincaeg ireees, hain lat Heaclrev tea ane the nean!\n",
      "Ferabs!\n",
      "elerind peal.\n",
      "Mns;\n",
      "Sonchiry:\n",
      "Sup; aiss hawty.\n",
      "The nis kormopeeelg Is\n",
      "Mom\n",
      "Ill, demet akleo Windo wher eiiby wey thad, rive cee, irecs poous\n",
      "Alixs bume kadmntertptef son; iris! mef thiny aled ath, af Pri?\n",
      "\n",
      "WISom.\n",
      "\n",
      "HeyNLIE!\n",
      "Ktiybyis:\n",
      "Sads.\n",
      "Wic:\n",
      "Eow she uin cour ayranny Iry thachan yf veay\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py --eval_interval 2999"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7- Residual Connections"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.1 Stacking the Blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Block(nn.Module):\n",
    "    \"\"\" Transformer Block: Communication followed by Computation \"\"\"\n",
    "\n",
    "    def __init__(self, n_embed, n_head):\n",
    "        \"\"\" n_embed: embedding dimension\n",
    "            n_head: number of heads in the multi-head attention\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        head_size = n_embed // n_head\n",
    "        self.sa = MultiHeadAttention(n_head, head_size)\n",
    "        self.ffwd = FeedForward(n_embed)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.sa(x)\n",
    "        x = self.ffwd(x)\n",
    "        return x"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``` python\n",
    "# previous code ..\n",
    "# each token directly reads off the logits for the next token from a loockup table\n",
    "self.token_embedding_table = nn.Embedding(vocab_size, n_embed)\n",
    "# each position is also associated with an embedding vector\n",
    "self.position_embedding_table = nn.Embedding(block_size, n_embed)\n",
    "# transformer blocks\n",
    "self.blocks = nn.Sequential(\n",
    "        Block(n_embed, n_head = 4),\n",
    "        Block(n_embed, n_head = 4),\n",
    "        Block(n_embed, n_head = 4),\n",
    ")\n",
    "self.lm_head = nn.Linear(n_embed, vocab_size)\n",
    "# rest of the code ..\n",
    "```\n",
    "\n",
    "add them in the forward pass\n",
    "``` python\n",
    "# previous code ..\n",
    "x = token_emb + pos_emb # (B, T, C) = (4, 8, vocab_size)\n",
    "# feed the input to the self attention head\n",
    "x = self.blocks(x) # (B, T, C) = (4, 8, vocab_size)\n",
    "logits = self.lm_head(x) # (B, T, vocab_size) = (4, 8, vocab_size)\n",
    "# rest of the code ..\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1.1 Test\n",
    "(loss is `2.81` instead of `2.46`) ➡️ **WORSE**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " step 0 | train loss: 4.2117 | val loss: 4.2063\n",
      " step 2999 | train loss: 2.8333 | val loss: 2.8197\n",
      "\n",
      "AYCCWADRR\n",
      "INOc'wd,en i on,ert maairet bobe u ehanvgrna mime?\n",
      "Ihauhs:\n",
      "Wa thheuu qhrt tetb cedtlrh a euuwccr moan,fnsdn o zoeou:\n",
      "Yowns\n",
      "Mtof ia hyi!t mil ndil bhaeg ireeus,n cin lat Hetidrov te annen me nean ilera mosel li daee lhhenuer ce wihy:\n",
      "Uuge nisslllwty. nlein'u nor\n",
      "Toetelgvls\n",
      "Mo:\n",
      "Iylwod motenkleo Winso ohdn piibas eoe m dosri Ehaee\n",
      ":Ieeus poo mo ehhore\n",
      "\n",
      "Apkadmntnruft f son; ioy !am:\n",
      "\n",
      "thre maleooats,fsf Pre?\n",
      "\n",
      "W Som.\n",
      "ran-,su\n",
      "Har,dybdi aa adaa Wahes ghesn hoih cotk aaranny Iry tamchan yfca hy\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py --eval_interval 2999"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.2 - Adding Residual Connections (Skip Connections)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2.1- Transformer Block"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Block(nn.Module):\n",
    "    \"\"\" Transformer Block: Communication followed by Computation \"\"\"\n",
    "\n",
    "    def __init__(self, n_embed, n_head):\n",
    "        \"\"\" n_embed: embedding dimension\n",
    "            n_head: number of heads in the multi-head attention\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        head_size = n_embed // n_head\n",
    "        self.sa = MultiHeadAttention(n_head, head_size)\n",
    "        self.ffwd = FeedForward(n_embed)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # residual connection (add the input to the output)\n",
    "        x = x + self.sa(x)\n",
    "        x = x + self.ffwd(x)\n",
    "        return x"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2.2- Multi-Head Attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multi Head Attention Class\n",
    "class MultiHeadAttention(nn.Module):\n",
    "    \"\"\" multiple heads of self-attention in parallel \"\"\"\n",
    "\n",
    "    def __init__(self, num_head, head_size):\n",
    "        super().__init__()\n",
    "        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_head)])\n",
    "        # linear transformation to the output of the multi-head attention as projection back to the residual pathway\n",
    "        self.proj = nn.Linear(n_embed, n_embed)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # out is the outptu of the multi-head attention\n",
    "        out =  torch.cat([h(x) for h in self.heads], dim = -1)\n",
    "        # apply a linear layer to the concatenated output\n",
    "        out = self.proj(out)\n",
    "        return out"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2.3- FeedForward Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feed Forward Class\n",
    "class FeedForward(nn.Module):\n",
    "    \"\"\" a simple linear layer followed by a non-linearity \"\"\"\n",
    "    def __init__(self, n_embed):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            # multiply by 4 to follow the original implementation\n",
    "            nn.Linear(n_embed, 4 * n_embed),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(n_embed * 4, n_embed),\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return self.net(x)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2.4 Test\n",
    "(loss is `2.33` instead of `2.81` and `2.46` before it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " step 0 | train loss: 4.6178 | val loss: 4.6449\n",
      " step 2999 | train loss: 2.2907 | val loss: 2.3392\n",
      "\n",
      "\n",
      "Afnt ak bridce?\n",
      ", This s, but\n",
      "\n",
      "Hirenk Bbe die.\n",
      "Sagrth my dagatanss:\n",
      "Watthy, ulqor, vet?\n",
      "redthas ane arche my.\n",
      "\n",
      "ODEZOY F I mun\n",
      "Youns, tof in hy HEvarl noill!,\n",
      "\n",
      "WHENYe:\n",
      "Youcin latistt drov the deen knongh pirerjbm!\n",
      " lelind peal.\n",
      "-hull cechiry prupe aiss hew ye nuren.\n",
      "I yroupe elives\n",
      "Akinty wod moth?\n",
      "\n",
      "KNRWLONARLOLICO:\n",
      "The the m dour sEThe shire sth-LOK:\n",
      "The thume kid nunrtpirf sor; ir he muf thie knled at ffife\n",
      "rred my om.\n",
      "rand, be!\n",
      "re you sar adspe the Eurest huin cour ay, tey Iry thechen you! hy\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py --eval_interval 2999 "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8- LayerNorm"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.1- BatchNorm1d from makemore part 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean of first column: 0.0000 | std of first column: 1.0000\n",
      "mean of first row: 0.0411 | std of first row: 1.0431\n"
     ]
    }
   ],
   "source": [
    "class BatchNorm1d:\n",
    "    def __init__(self, dim, eps=1e-5, momentum = 0.1):\n",
    "        self.eps = eps\n",
    "        self.momentum = momentum\n",
    "        self.training = True\n",
    "\n",
    "        # parameters (trained with backprop)\n",
    "        self.gamma = torch.ones(dim)\n",
    "        self.beta = torch.zeros(dim)\n",
    "\n",
    "        # buffers (trained while running `momentum update`)\n",
    "        self.running_mean = torch.zeros(dim)\n",
    "        self.running_var = torch.ones(dim)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        if self.training:\n",
    "            # batch mean\n",
    "            xmean = x.mean(0, keepdim= True)\n",
    "            # batch variance\n",
    "            xvar = x.var(0, keepdim= True)\n",
    "        else:\n",
    "            xmean = self.running_mean\n",
    "            xvar = self.running_var\n",
    "        \n",
    "        xhat = (x - xmean) / torch.sqrt(xvar + self.eps)\n",
    "        self.out = self.gamma * xhat + self.beta\n",
    "\n",
    "        # update the buffers in training\n",
    "        if self.training:\n",
    "            with torch.no_grad():\n",
    "                self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * xmean\n",
    "                self.running_var = (1 - self.momentum) * self.running_var + self.momentum * xvar\n",
    "        \n",
    "        return self.out\n",
    "    \n",
    "    def parameters(self):\n",
    "        return [self.gamma, self.beta]\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "module = BatchNorm1d(100)\n",
    "x = torch.randn(32, 100)\n",
    "x = module(x)\n",
    "x.shape\n",
    "\n",
    "# columns are normalized\n",
    "print(f\"mean of first column: {x[:, 0].mean():.4f} | std of first column: {x[:, 0].std():.4f}\")\n",
    "# rows are not normalized ➡️ we need to normalize the rows instead\n",
    "print(f\"mean of first row: {x[0, :].mean():.4f} | std of first row: {x[0, :].std():.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean of first column: 0.1469 | std of first column: 0.8803\n",
      "mean of first row: -0.0000 | std of first row: 1.0000\n"
     ]
    }
   ],
   "source": [
    "# after normalizing the rows (and removing the buffers too)\n",
    "class BatchNorm1d:\n",
    "    def __init__(self, dim, eps=1e-5, momentum = 0.1):\n",
    "        self.eps = eps\n",
    "        self.gamma = torch.ones(dim)\n",
    "        self.beta = torch.zeros(dim)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        xmean = x.mean(1, keepdim= True)\n",
    "        xvar = x.var(1, keepdim= True)\n",
    "        xhat = (x - xmean) / torch.sqrt(xvar + self.eps)\n",
    "        self.out = self.gamma * xhat + self.beta\n",
    "        return self.out\n",
    "    \n",
    "    def parameters(self):\n",
    "        return [self.gamma, self.beta]\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "module = BatchNorm1d(100)\n",
    "x = torch.randn(32, 100)\n",
    "x = module(x)\n",
    "x.shape\n",
    "\n",
    "# columns are not normalized now\n",
    "print(f\"mean of first column: {x[:, 0].mean():.4f} | std of first column: {x[:, 0].std():.4f}\")\n",
    "# rows are normalized now\n",
    "print(f\"mean of first row: {x[0, :].mean():.4f} | std of first row: {x[0, :].std():.4f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.2- Adding LayerNorm"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.2.1- in the transformer blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Block(nn.Module):\n",
    "    \"\"\" Transformer Block: Communication followed by Computation \"\"\"\n",
    "\n",
    "    def __init__(self, n_embed, n_head):\n",
    "        \"\"\" n_embed: embedding dimension\n",
    "            n_head: number of heads in the multi-head attention\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        head_size = n_embed // n_head\n",
    "        self.sa = MultiHeadAttention(n_head, head_size)\n",
    "        self.ffwd = FeedForward(n_embed)\n",
    "        # ln1 is applied directly on input before the multi-head attention\n",
    "        self.ln1 = nn.LayerNorm(n_embed)\n",
    "        # ln2 is applied directly on the output of the multi-head attention before the feed-forward layer\n",
    "        self.ln2 = nn.LayerNorm(n_embed)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # residual connection (add the input to the output)\n",
    "        x = x + self.sa(self.ln1(x))\n",
    "        x = x + self.ffwd(self.ln2(x))\n",
    "        return x"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.2.1- after all blocks (before last linear layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BigramLanguageModel(nn.Module):\n",
    "    # no need to pass vocab_size as an argument, since it is a global variable in this file\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a loockup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embed)\n",
    "        # each position is also associated with an embedding vector\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embed)\n",
    "        # transformer blocks\n",
    "        self.blocks = nn.Sequential(\n",
    "                Block(n_embed, n_head = 4),\n",
    "                Block(n_embed, n_head = 4),\n",
    "                Block(n_embed, n_head = 4),\n",
    "                # add layernorm here\n",
    "                nn.LayerNorm(n_embed),\n",
    "        )\n",
    "        self.lm_head = nn.Linear(n_embed, vocab_size)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9- Scaling up the model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.1- Adding `n_layer` variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# super simple bigram model\n",
    "class BigramLanguageModel(nn.Module):\n",
    "    # no need to pass vocab_size as an argument, since it is a global variable in this file\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a loockup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embed)\n",
    "        # each position is also associated with an embedding vector\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embed)\n",
    "        # transformer blocks\n",
    "        self.blocks = nn.Sequential(*[Block(n_embed, n_head = 4) for _ in range(n_layer)])\n",
    "        # Remember to add it in forward too\n",
    "        self.ln_f = nn.LayerNorm(n_embed)\n",
    "        self.lm_head = nn.Linear(n_embed, vocab_size)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.2- Adding Dropouts\n",
    "- in `Head` after calculating `wei`\n",
    "- in `MultiHeadAttention` after `self.proj`\n",
    "- in `FeedForward` after last linear"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.3- Try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " step 0 | train loss: 4.4116 | val loss: 4.4048\n",
      " step 100 | train loss: 2.6539 | val loss: 2.6605\n",
      " step 200 | train loss: 2.5139 | val loss: 2.5159\n",
      " step 300 | train loss: 2.4038 | val loss: 2.4126\n",
      " step 400 | train loss: 2.3433 | val loss: 2.3510\n",
      " step 500 | train loss: 2.2921 | val loss: 2.3006\n",
      " step 600 | train loss: 2.2344 | val loss: 2.2512\n",
      " step 700 | train loss: 2.1932 | val loss: 2.2093\n",
      " step 800 | train loss: 2.1559 | val loss: 2.1801\n",
      " step 900 | train loss: 2.1208 | val loss: 2.1500\n",
      " step 1000 | train loss: 2.0774 | val loss: 2.1180\n",
      " step 1100 | train loss: 2.0619 | val loss: 2.1036\n",
      " step 1200 | train loss: 2.0294 | val loss: 2.0747\n",
      " step 1300 | train loss: 2.0156 | val loss: 2.0591\n",
      " step 1400 | train loss: 1.9943 | val loss: 2.0548\n",
      " step 1500 | train loss: 1.9765 | val loss: 2.0379\n",
      " step 1600 | train loss: 1.9503 | val loss: 2.0301\n",
      " step 1700 | train loss: 1.9322 | val loss: 2.0212\n",
      " step 1800 | train loss: 1.9143 | val loss: 2.0021\n",
      " step 1900 | train loss: 1.9076 | val loss: 2.0012\n",
      " step 2000 | train loss: 1.8849 | val loss: 1.9906\n",
      " step 2100 | train loss: 1.8666 | val loss: 1.9728\n",
      " step 2200 | train loss: 1.8726 | val loss: 1.9664\n",
      " step 2300 | train loss: 1.8506 | val loss: 1.9482\n",
      " step 2400 | train loss: 1.8424 | val loss: 1.9487\n",
      " step 2500 | train loss: 1.8296 | val loss: 1.9448\n",
      " step 2600 | train loss: 1.8122 | val loss: 1.9379\n",
      " step 2700 | train loss: 1.8075 | val loss: 1.9298\n",
      " step 2800 | train loss: 1.8044 | val loss: 1.9225\n",
      " step 2900 | train loss: 1.7831 | val loss: 1.9009\n",
      " step 3000 | train loss: 1.7788 | val loss: 1.8989\n",
      " step 3100 | train loss: 1.7812 | val loss: 1.9153\n",
      " step 3200 | train loss: 1.7774 | val loss: 1.9140\n",
      " step 3300 | train loss: 1.7530 | val loss: 1.8908\n",
      " step 3400 | train loss: 1.7552 | val loss: 1.8924\n",
      " step 3500 | train loss: 1.7409 | val loss: 1.8869\n",
      " step 3600 | train loss: 1.7349 | val loss: 1.8675\n",
      " step 3700 | train loss: 1.7289 | val loss: 1.8794\n",
      " step 3800 | train loss: 1.7138 | val loss: 1.8651\n",
      " step 3900 | train loss: 1.7235 | val loss: 1.8700\n",
      " step 4000 | train loss: 1.7144 | val loss: 1.8609\n",
      " step 4100 | train loss: 1.7039 | val loss: 1.8623\n",
      " step 4200 | train loss: 1.7089 | val loss: 1.8444\n",
      " step 4300 | train loss: 1.7015 | val loss: 1.8448\n",
      " step 4400 | train loss: 1.6836 | val loss: 1.8415\n",
      " step 4500 | train loss: 1.6991 | val loss: 1.8510\n",
      " step 4600 | train loss: 1.6972 | val loss: 1.8480\n",
      " step 4700 | train loss: 1.6860 | val loss: 1.8427\n",
      " step 4800 | train loss: 1.6801 | val loss: 1.8325\n",
      " step 4900 | train loss: 1.6725 | val loss: 1.8227\n",
      "\n",
      "\n",
      "Clown:\n",
      "Romiolow and is so bleman and boay tongurage-and me?\n",
      "Than your baphed wearth. that dilay ane away, my feance,\n",
      "And I make what tofficion!\n",
      "\n",
      "VOLUMNIA:\n",
      "What misters, hoin latise, lieve to done me now on trans!\n",
      "\n",
      "AUlingion upolish wouch by atchminess havey.\n",
      "\n",
      "Slings norforething sonder.\n",
      "Who like again Wilike have iinstreng more rive weephing strong\n",
      "of his burder of thrust for armiging must with allook,\n",
      "To figrand my of trangious! stild is wards.\n",
      "Wich Edward huil couragain.\n",
      "\n",
      "Clry tomenan you!\n",
      "My\n"
     ]
    }
   ],
   "source": [
    "!python3 bigram.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "main",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}