notebooks/Interpreting Transformer Layers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5c4e9a63",
   "metadata": {},
   "source": [
    "Functional Discourse Grammar is normally used to describe the production of spoken language in a way that is comparable across widely differing languages. Can we define a notation informed by FDG to describe the structure of a transformer model?\n",
    "\n",
    "The features of such a notation include:\n",
    "* Describe a discourse, not just a sentence\n",
    "* Utterances are described in parallel at a number of levels, which are completed in parallel as the sentence is completed.  There are links between the levels, FDG describes 4 levels, including the phonological layer which is less relevant to text proessing. Lower levels follow the actual word ordering more closely, upper levels may not be completed out of order. \n",
    "* Each level has a hierarchically ordered layered organization. \n",
    "* Layers always have a head, and usually a modifier (both of which may themselves be layers or lexical content), and may also be specified by a linguistic-operator and carry a linguistic-function.\n",
    "\n",
    "Challenges include similar terminology with quiet distinct meanings.In particular the use of layer and function in FDG may lead to confusion, I will attempt to be specific in referring to linguistic-layer and transformer-layers.\n",
    "\n",
    "A key difference between transformer language models and typical FDG tasks is that FDG is focussed primarily on the formulation of speech rather than language interpretation, while generative language models both intepret and formulate language, with an indistinct boundary between the two. I expect the first and last layers to remain close to morphosyntactic representation, and internal transformer-layers to be reflect the interpersonal and representational linguistic-levels."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045ace02",
   "metadata": {},
   "source": [
    "I do not propose trying to train a network to interpret an utterance using FDG - I am sure this would be possible since analysis using more widely used linguistic representations are already possible. I am more interested in what parallels there may be between the structure of the transformer latent space and the structure of FDG\n",
    "\n",
    "My understanding of the transformer architecture is that the token generated depends on a transformation of the same position in a lower transformer-level, and through attention heads on other positions in the same level. (describe this more rigorously...)\n",
    "\n",
    "The idea is to identify elements that span multiple consecutive positions within a transformer-layer based on \n",
    "a) identifying sets of neurons that have been  significant to later positions\n",
    "b) identifying how long those neurons remain active \n",
    "c) where they came from (previous history or lower layers)\n",
    "\n",
    "step down through levels to identify where certain activation patterns were introduced, then search across to identify which positions in history at that level contributed. These are then treated as starting points on that layer.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b39f12e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\stephenprior\\.virtualenvs\\transformer-utils-kjd8b5rh\\lib\\site-packages\\torch\\nn\\modules\\lazy.py:175: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.\n",
      "  warnings.warn('Lazy modules are a new feature under heavy development '\n"
     ]
    }
   ],
   "source": [
    "from transformer_utils.low_memory import enable_low_memory_load\n",
    "import transformers\n",
    "\n",
    "enable_low_memory_load()\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(\"gpt2\")\n",
    "model = transformers.AutoModelForCausalLM.from_pretrained('EleutherAI/gpt-neo-125M')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2dc9f13",
   "metadata": {},
   "source": [
    "Embeds 50,257 BPE vocab tokens to 768-dimensional latent space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2214439e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(50257, 768)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.transformer.wte"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f4a3e61",
   "metadata": {},
   "source": [
    "Embeds 2048 token positions to 768-dimensional latent space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6a00e2cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(2048, 768)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.transformer.wpe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "be6937f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPTNeoBlock(\n",
       "  (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "  (attn): GPTNeoAttention(\n",
       "    (attention): GPTNeoSelfAttention(\n",
       "      (attn_dropout): Dropout(p=0, inplace=False)\n",
       "      (resid_dropout): Dropout(p=0, inplace=False)\n",
       "      (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "      (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "      (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "      (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "    )\n",
       "  )\n",
       "  (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "  (mlp): GPTNeoMLP(\n",
       "    (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "    (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "    (dropout): Dropout(p=0, inplace=False)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.transformer.h[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7868ee44",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "def text_to_input_ids(text):\n",
    "    toks = tokenizer.encode(text)\n",
    "    return torch.as_tensor(toks).view(1, -1).cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3206bb39",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformer_utils.partial_forward import partial_forward\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "10bc0708",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "input_ids = text_to_input_ids(\"Watch out, because there is a bull in the field. Don't\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0123b5be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10723,   503,    11,   780,   612,   318,   257,  6473,   287,   262,\n",
       "          2214,    13,  2094,   470]], device='cuda:0')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "03852787",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Watch out, because there is a bull in the field. Don't\"]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.batch_decode(input_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f7309524",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Watch'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(10723)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bbe6b8ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 14])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a3ead924",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 14, 768])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    output_names=['h.0', 'h.1','h.2','h.3', 'h.4','h.5','h.6', 'h.7','h.8','h.9','h.10','h.11']\n",
    "    with torch.no_grad():\n",
    "        hidden_states = partial_forward(\n",
    "            model.base_model,\n",
    "            output_names,\n",
    "            input_ids,\n",
    "        )\n",
    "    hidden_states['h.11'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9037dd22",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([768])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    hidden_states['h.11'][0][0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5eac94ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(200.0554, dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "    \n",
    "torch.linalg.norm(hidden_states['h.11'][0][0]).cpu().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "908ed5e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0e97a07e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x22895a9d790>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAn3klEQVR4nO3de3xcdZ3/8ddnJrfmNkmapGkyKU3v10SgFljuoAisCLjIgv4EFdfLAq7uxcs+dlcf6+q6D3d/LoKo/EAFfy7iDxCqoi4rBVGubWnJhLY0tLTNpGnSJp3Jpc1l5vP7Y05CKC1Jm0zOzDmf5+PRR2fOnMn3c9rkPSff7/d8j6gqxhhj/CHgdgHGGGNmjoW+Mcb4iIW+Mcb4iIW+Mcb4iIW+Mcb4SI7bBbydyspKnT9/vttlGGNMVtm4ceMBVa061msZHfrz589nw4YNbpdhjDFZRUR2H+81694xxhgfsdA3xhgfsdA3xhgfsdA3xhgfsdA3xhgfsdA3xhgfsdA3xhgfyeh5+sZks5FEksGRJEMjb/w9lEgwklQSSSWZhJFkkqQqiSSpbaqMJJWks09Cnb+d10Yfj742ut/I2OuM7TeSVHICQnlhLuVFeVQU5lFWmEdFUR5lhbkU5Abd/icyLrDQN762dV+cSDT2lnAeHEkcFdZv3jZ4jH2Pfn8yw29VUZgXpHzch0BFUR7lhXnOttyxD4jywjzKi3IpL8yzDwoPsNA3vnbTj16kPXbkLduDASE/J0BeTmDc30HyggHycwPkBQOEZuWOPc8ft31s36Pf6zzOCQgBEXICQjAgBAJCUIRAAHICAYIBCEjqtbE/8sZ+47ePfp3Am/ZLfZ2AwEhSOTQwTM/AEN39QxwaGKK7P/W8p3+IbufvnoFh9nQP0N0/RO+RkeP+e41+UIx+CJQf9cFQX1HIyrmlVJcWpPO/zUyBhb7xrc7eI7THjvDZdy3mg2vnvSmcgwFxu7xpkRsUqkryqSrJn/R7hhPJsQ+K1AfCWz8oDg0M090/xJ7uAXr6h4gf9UFRWZzPytpSVtSWsrK2lJW1IU6pKCTgkX/XbGahb3yrJRoH4KwFs+3MdJzcYOCkPih6BobY2dVPS3ucV9rjtLTH+GPrAUacfq6ivCDL577xIbCitpTFc4rJz7Euo5lkoW98KxKNAbCyLuRyJdkvNxiguqSA6pICzlwwe2z74EiCHfv7aGmPOR8EcR7c2Ma9z6bWA8sJCIvnlLBi7MOglOW1pZQW5Lp1KJ5noW98K9IeY0FlEcX59mOQLvk5QVbVhVg17oM1mVR2dw/Q0h4b+63gqVe7eGhT29g+8yoKU91Dc0tZWZf6zaC6JB8R6x6aKvtuN74VicY57ZRyt8vwnUBAaKgsoqGyiPc21o5t74wfoWXfG11Dr7TH+XWkY+z12UV5zhhBqmuosS7E/MoiNw4hq1noG1/q7h8ieugwN5x1itulGEd1aQHVpQVcuLR6bFvvkWG27uvlFee3gpb2OPf8YSfDidQ4wf+96QzOWVzpVslZyULf+FJLe6o/f7X152e0koJc1jZUsLahYmzb0EiS7R29XHXnH3l+10EL/RNkyzAYX4o4M3dW1lroZ5u8nACrwyEWVxezpS3mdjlZx0Lf+FIkGqO+YhahQpslkq3eUV/Gy22HUM3wS58zjIW+8aVIe8y6drJcY7iMQ86VxGbyLPSN78QOD7P74IB17WS5xnDq/8+6eE6Mhb7xndFB3FV2pp/VltaUkJ8T4OW9h9wuJatY6BvfGV1+YVVtqcuVmKnIDQZYUVvKy3amf0Is9I3vRNpj1IYKmF08+bVlTGZqCpcRaY+RyPR1rDOIhb7xnUg0ZuvteERjOMTAUILWzj63S8kaFvrGV/oGR9h5oJ9VNojrCU31ZQBssX79SbPQN76ydV8cVVhVZ/35XtAwu4iS/By2tB1yu5SsYaFvfGV0OWWbo+8NgYCwOhyywdwTYKFvfKU5GqOqJN9umuIhjeEytnXEGRxJuF1KVrDQN77SEo3bVE2PaQqHGE4oW/f1ul1KVrDQN75xeCjBjs5e69rxmEZnMPdl69efFAt94xvbOuIk1W6P6DW1oQIqi/PYstf69SfDQt/4xuggri2/4C0iQmO4zM70J8lC3/hGJBqnvDCX2pAN4npNU7iM1q4++gZH3C4l41noG9+ItMdYVReym2t7UGN9CFVotqmbE7LQN74wOJLg1f291rXjUU3hMsAGcyfDQt/4wqsdfQwn1JZf8KiKojzC5bPsIq1JmFToi8jnRKRFRCIicr+IFIhIg4g8LyKtIvKAiOQ5++Y7z1ud1+eP+zpfcrZvF5H3pOmYjHmLyNga+jZH36uawmW2HMMkTBj6IlIHfAZYo6qrgCBwHfBvwLdUdRHQA9zkvOUmoMfZ/i1nP0RkhfO+lcClwJ0iEpzewzHm2CLRGCUFOcyrKHS7FJMmjeEQbT2HOdg36HYpGW2y3Ts5wCwRyQEKgX3ARcCDzuv3Alc5j690nuO8frGkRs6uBH6qqoOqugtoBdZO+QiMmYRINMaqWhvE9bLG0X79qHXxvJ0JQ19Vo8C/A3tIhX0M2AgcUtXR+VFtQJ3zuA7Y67x3xNl/9vjtx3jPGBH5hIhsEJENXV1dJ3NMxrzJcCLJ1o5e69rxuNXhECK2zPJEJtO9U07qLL0BqAWKSHXPpIWq3qWqa1R1TVVVVbqaMT7S2tnH0EjSZu54XHF+Douqim0wdwKT6d55F7BLVbtUdRh4GDgbKHO6ewDCQNR5HAXqAZzXQ8DB8duP8R5j0sauxPWP0StzVe32icczmdDfA5wpIoVO3/zFwCvAeuAaZ58bgUedx+uc5zivP6Gp/4F1wHXO7J4GYDHwwvQchjHHF4nGKMoL0jC7yO1STJo11Yc40DdEe+yI26VkrJyJdlDV50XkQWATMAK8BNwF/Ar4qYj8i7PtHuct9wA/FpFWoJvUjB1UtUVEfkbqA2MEuFlVbQFsk3aR9jgraksJBGwQ1+vGBnP3HqKubJa7xWSoCUMfQFW/DHz5qM07OcbsG1U9AnzgOF/na8DXTrBGY05aIqm80h7nurX1E+9sst7yuSXkBoUtbTEuWz3X7XIykl2RazxtZ1cfh4cTdiWuT+TnBFlWU2rLMbwNC33jaW9ciWuh7xeN4RDNbTGSSRvMPRYLfeNpkWicgtwAC6tsENcvmurL6B0cYeeBfrdLyUgW+sbTItEYy+eWkhO0b3W/sBU33579JBjPSiaVlva49ef7zKLqYgrzgnaR1nFY6BvP2t09QN/giC2/4DPBgLCqNmQrbh6Hhb7xLLsS178awyFeaY8znEi6XUrGsdA3nhVpj5EXDLC4usTtUswMa6wvY3AkyfaOXrdLyTgW+sazItEYS2tKyMuxb3O/aQqnfruzfv23sp8G40mqSiQat/58n5pXUUhZYa4ts3wMFvrGk9p6DhM7PGz9+T4lIjTa7ROPyULfeNLYIK5N1/StpnCIHZ19HB6ydR3Hs9A3nhRpj5ETEJbW2CCuXzWGy0gklZZ269cfz0LfeFIkGmfxnBIKcoNul2JcMjqYu8UGc9/EQt94TmoQN8aqWhvE9bPq0gJqSgtsOYajWOgbz+mIH+Fg/5AN4hoawyGbtnkUC33jOZFoHMCmaxqa6svYdaCf2MCw26VkDAt94zmRaIyAwPK5Fvp+N7biZvSQq3VkEgt94zmRaIyFVcUU5k3qbqDGw1bblblvYaFvPCfSHrP+fANAaFYuDZVFdmXuOBb6xlM6e4+wPz7ISpu5Yxw2mPtmFvrGU1raU4O4q+1M3zgaw2V0xI/QGT/idikZwULfeErEOaNbYWf6xmEXab2Zhb7xlEh7jIbKIkoKct0uxWSIlbUhggGxi7QcFvrGU1LLKVvXjnnDrLwgS+aUsNkGcwELfeMhPf1DRA8dtuUXzFs0hUM0R2OoqtuluM5C33hGpN3uiWuOrTFcxqGBYfZ0D7hdiuss9I1njC6/YNM1zdEabTB3jIW+8YxIe4z6ilmUFea5XYrJMEtrSsjPCfCy9etb6BvvSC2nbF075q1ygwFW1JbaRVpY6BuPiB0eZvfBAevPN8fVFC4j0h4jkfT3YK6FvvGEV9pHl1O20DfH1lQfYmAoQWtnn9uluMpC33jC6H1QbRDXHE+js8yy3xdfm1Toi0iZiDwoIttEZKuInCUiFSLyuIjscP4ud/YVEfm2iLSKyMsictq4r3Ojs/8OEbkxXQdl/Kc5GmNuqIDK4ny3SzEZqmF2ESX5OWzx+ZW5kz3Tvw34jaouA5qArcAXgd+p6mLgd85zgMuAxc6fTwDfBRCRCuDLwBnAWuDLox8UxkxVJBpjpQ3imrcRCAirbcXNiUNfRELAecA9AKo6pKqHgCuBe53d7gWuch5fCdynKc8BZSIyF3gP8LiqdqtqD/A4cOk0Hovxqf7BEXYe6LeVNc2EGsNlbOuIMziScLsU10zmTL8B6AJ+KCIvicjdIlIEzFHVfc4+HcAc53EdsHfc+9ucbcfbbsyUvLIvjqrdE9dMrCkcYjihbN3X63YprplM6OcApwHfVdVTgX7e6MoBQFMLWkzLPCgR+YSIbBCRDV1dXdPxJY3HRaK2/IKZnMb6MgBfr7g5mdBvA9pU9Xnn+YOkPgT2O902OH93Oq9Hgfpx7w872463/U1U9S5VXaOqa6qqqk7kWIxPRaJxKovzqS6xQVzz9mpDBVQW57Flr3/79ScMfVXtAPaKyFJn08XAK8A6YHQGzo3Ao87jdcANziyeM4GY0w30W+ASESl3BnAvcbYZMyUt7TFW15UiIm6XYjKciNAULvP1DJ6cSe53K/ATEckDdgIfJfWB8TMRuQnYDVzr7PsYcDnQCgw4+6Kq3SLyVeBFZ79/VtXuaTkK41tHhhPs6Ozj3SvmTLyzMaQGc5/Y3knf4AjF+ZONQO+Y1BGr6mZgzTFeuvgY+ypw83G+zg+AH5xAfca8ra374iSSatM1zaQ11odQhea2GGctnO12OTPOrsg1WS0yeiP0sIW+mZwm58pcvw7mWuibrBZpi1FemEttqMDtUkyWqCjKI1w+y7cXaVnom6wWaY+xqi5kg7jmhPh5MNdC32StwZEEr+7vtf58c8IawyHaeg5zsG/Q7VJmnIW+yVo79vcxnFBbfsGcsKbRi7Si/uvisdA3Wat57EpcW37BnJhUl6A/l1m20DdZKxKNUVKQw7yKQrdLMVmmOD+HRVXFvhzMtdA3WSvSHmdVrQ3impPTGC7j5bZDpC4t8g8LfZOVhhNJtu6LW9eOOWlN9SEO9A3RHjvidikzykLfZKXWzj6GRpK2sqY5aaO3T3zZZ/36FvomK40up2zTNc3JWj63hNygsMVn/foW+iYrtbTHKcoLsqCyyO1STJbKzwmyrKbUd8sxWOibrNQcjbGitpRAwAZxzclrqg/R3BYjmfTPYK6Fvsk6iaTySnvcunbMlDWGy+h17rHsFxb6JuvsOtDH4eGEDeKaKfPjipsW+ibrRKLOcsoW+maKFlUXU5gX9NVFWhb6Jus0R2Pk5wRYWGWDuGZqggFhVW3IVytuWuibrBOJxlg+t5ScoH37mqlrDId4pT3OcCLpdikzwn5qTFZJOoO41rVjpktjfRmDI0m2d/S6XcqMsNA3WWV39wC9gyO2/IKZNu8YG8z1R7++hb7JKnYlrplu9RWzKC/M9c0yyxb6JqtE2mPkBQMsmVPidinGI0SE1T66faKFvskqLdE4S2tKyMuxb10zfZrCIXZ09nF4KOF2KWlnPzkma6gqzdGY9eebadcYLiORVFravd+vb6FvskZbz2Fih4etP99Mu6Zw6nvKDytuWuibrDF6FmbLL5jpVl1aQE1pgS+WY7DQN1kjEo0TDAjLamwQ10y/pvqQL6ZtWuibrNEcjbG4upiC3KDbpRgPagyXsetAP7GBYbdLSSsLfZMVVJVINGZdOyZtxlbcjB5ytY50s9A3WWF/fJCD/UO2/IJJm9XOYK7Xu3gs9E1WaI6ODuLadE2THqFZuTRUFnn+ylwLfZMVItEYIrB8roW+SZ/GsPcHcy30TVZoaY+xsKqYwrwct0sxHtYYLqMjfoTO+BG3S0kbC32TFSJRW07ZpJ8fLtKadOiLSFBEXhKRXzrPG0TkeRFpFZEHRCTP2Z7vPG91Xp8/7mt8ydm+XUTeM+1HYzypq3eQjvgRVtZa145Jr5W1IYIB8fRFWidypv9XwNZxz/8N+JaqLgJ6gJuc7TcBPc72bzn7ISIrgOuAlcClwJ0iYhOuzYQidiWumSGz8oIsmVPCZg8P5k4q9EUkDPwpcLfzXICLgAedXe4FrnIeX+k8x3n9Ymf/K4Gfquqgqu4CWoG103AMxuNaxtbQtzN9k35N4RDN0Riq6nYpaTHZM/3/BD4PjN5EcjZwSFVHnOdtQJ3zuA7YC+C8HnP2H9t+jPeMEZFPiMgGEdnQ1dU1+SMxntUcjdFQWURJQa7bpRgfaAyXcWhgmD3dA26XkhYThr6IvBfoVNWNM1APqnqXqq5R1TVVVVUz0aTJcJFo3M7yzYxp9Phg7mTO9M8G3icirwM/JdWtcxtQJiKj8+fCQNR5HAXqAZzXQ8DB8duP8R5jjqmnf4joocPWn29mzNKaEvJzArzs0X79CUNfVb+kqmFVnU9qIPYJVf0QsB64xtntRuBR5/E65znO609oqnNsHXCdM7unAVgMvDBtR2I8qaU9DmDTNc2MyQ0GWFFb6tmLtKYyT/8LwF+LSCupPvt7nO33ALOd7X8NfBFAVVuAnwGvAL8BblZV79+bzExJsw3iGhc0hctojsYYSSQn3jnLnNDljar6JPCk83gnx5h9o6pHgA8c5/1fA752okUa/4q0xwiXz6KsMM/tUoyPNNWH+NEzr9Pa1ceyGm+dcNgVuSajtURjrLLbI5oZ1ji6zPJe73XxWOibjBU/MszrBwfGlrw1ZqY0zC6iJD+HLR68MtdC32SslmhqENf6881MCwSE1R5dcdNC32QsuxG6cVNjuIxtHXEGR7w138RC32SsSDTG3FABlcX5bpdifKgpHGI4oWzd1+t2KdPKQt9krOZojJU2iGtc0lhfBuC5FTct9E1G6h8cYeeBfrs9onFNrfNb5haPzeCx0DcZaeu+OKrYdE3jGhGhKRzy3AweC32TkSLOlbg2XdO4qTFcxmtdffQNjky8c5aw0DcZqTkap7I4n+oSG8Q17mmsD6EKzR6aummhbzJSS3uMVXWlpO6/Y4w7mkavzPVQF4+Fvsk4R4YT7Ojss/5847qKojzC5bM8dZGWhb7JOFv3xUkk1S7KMhmhKVzmqcFcC32TcSLOGvo2XdNkgqb6EG09hznYN+h2KdPCQt9knJZojLLCXOrKZrldijFvrLjpkS4eC32TcSLtMVbXhWwQ12SEVXUhRPBMF4+FvskogyMJtnf02vILJmMU5+ewqKrYzvSNSYcd+/sYTqj155uM0hguY8veQySS6nYpU2ahbzJGMqk8ub0TsOUXTGZ51/JqDvYP8evIPrdLmbITukeuMenQ2tnHz19q45GX2okeOsyi6mLmVRS6XZYxYy5ZWcPCqiLueKKVy1fNJRDI3vEmC33jiu7+IX6xpZ2HN7WxpS1GQODcxVV8/tKlXLKiJqt/qIz3BAPCLRct4nMPbOF/tu7nkpU1bpd00iz0zYwZHEnwxNZOHtoU5cntnYwkleVzS/mHP13O+5pqqS4tcLtEY47risZavvX4Du5Y38q7V8zJ2tllFvomrVSVTXt6eGhTlF9uaSd+ZITqknw+dk4DV59ax/K5NmBrskNOMMBfXrCQLz7czFOvdnHB0mq3SzopFvomLfYcHODhl9r4+UtRdh8coCA3wKUra3j/aWHOXlRJ0LpvTBZ6/2lhvv27Hdz+RCvnL6nKyrN9C30zbWKHh/nVy/v4+UttvPh6DyJw1oLZ3HrRYi5dVUNxvn27meyWlxPgUxcs5J8ebeG5nd2ctXC22yWdMPspNFMynEjy1PYufv5SlMe37mdoJMmi6mI+f+lSrnpHHbW2lILxmGvX1HP7E63c/sQOC33jD6pKczTGw5ui/GJLOwf7h6goyuODa+fxZ6eFbR1842kFuUE+ed4C/uVXW9m4u5vTT6lwu6QTYqFvJq390GEe2Rzl4U1RWjv7yMsJ8O7lc3j/aXWct6SK3KBd62f84YNnzOM761u544lWfvjRtW6Xc0Is9M1xDSeS7DrQz+Y9h3hkc5Rndx5EFd45v5x/ff9qLl89l9CsXLfLNGbGFebl8PFzF/DN326nuS2WVfdyttA3qCr744Ns64izraOX7R29bOvo5bXOPoYSSQBOmV3IZy9ewtWn1jFvtl0ta8wNZ53C9596jTvW7+D7H17jdjmTZqHvM/2DI2zf38u2fb1sd0J+W0cvscPDY/vUlBawbG4J5y2pZFlNCcvnlrJ0Ton10xszTklBLh85u4Fv/24H2zt6WVpT4nZJk2Kh71EjiSSvHxxgW0d87Mx9W0ecvd2Hx/YpyguypKaEy1fPZVlNCctqSlhaU0JZYZ6LlRuTPT529nzueXond6xv5fbrT3W7nEmx0M9yqkpX32Aq2Pelwn37/jiv7u9jaCTVNRMQaKgsojFcxrWn17PUOXuvK5tla9wYMwVlhXl8+Kz5fP/3r/HZdy1mYVWx2yVNaMLQF5F64D5gDqDAXap6m4hUAA8A84HXgWtVtUdSfQC3AZcDA8BHVHWT87VuBP7B+dL/oqr3Tu/heN/BvkGe3nGALW2Hxs7gu/uHxl6vKslnWU0JN551CktrSllWU8Ki6mIKcoMuVm2Md3383AZ+9Mwu7lz/Gv9xbZPb5UxoMmf6I8DfqOomESkBNorI48BHgN+p6jdE5IvAF4EvAJcBi50/ZwDfBc5wPiS+DKwh9eGxUUTWqWrPdB+UlySTSqQ9xvptXazf3smWtkOowqzcIEvmFPOu5dUsc8J9aU0Js4vz3S7ZGF+pLM7n+rXzuO/Z3Xz2XYupz/BlwScMfVXdB+xzHveKyFagDrgSuMDZ7V7gSVKhfyVwn6oq8JyIlInIXGffx1W1G8D54LgUuH8aj8cTYgPDPN3axfptXTz1aicH+oYQgaZwGZ+9eAkXLqtiZW3I1q8xJkN88ryF/OS5PXz3qdf4+tWr3S7nbZ1Qn76IzAdOBZ4H5jgfCAAdpLp/IPWBsHfc29qcbcfb7nuqyraOXtZv7+TJbV1s3NNDIqmEZuVy/pIqLlxWxXmLq+ws3pgMVRMq4ANrwvy/DW3cetEi5oYyd/mRSYe+iBQDDwGfVdX4+Ol7qqoiMi03jxSRTwCfAJg3b950fMmM1Dc4wh9bD/Dk9k7Wb+uiI34EgJW1pXz6/IVcuKyKpnAZOXaVqzFZ4VPnL+SBF/fy/ad28pX3rXS7nOOaVOiLSC6pwP+Jqj7sbN4vInNVdZ/TfdPpbI8C9ePeHna2RXmjO2h0+5NHt6WqdwF3AaxZsyb770LsUFVe6+pPhfz2Tl7Y1c1wQinOz+HcxZVcuLSa85dWMcduJGJMVqqvKOTqU+u4/4U93HzhIqpKMvM388nM3hHgHmCrqv7vcS+tA24EvuH8/ei47beIyE9JDeTGnA+G3wJfF5FyZ79LgC9Nz2FkpsNDCZ7beZD1TtCPzpFfMqeYj53dwAVLqzn9lHLycuxs3hgv+PQFC3loUxt3P72TL12+3O1yjmkyZ/pnAx8GmkVks7Pt70mF/c9E5CZgN3Ct89pjpKZrtpKasvlRAFXtFpGvAi86+/3z6KCul+w5ODAW8s++dpDBkSSzcoOcvWg2nzxvIRcsrSJcntmj+8aYk7Ogqpj3Ntby4+d286nzF1JelHkXOkpqkk1mWrNmjW7YsMHtMib0+oF+fvzcbtZv72RnVz+QuhjqgqVVXLi0mrUNFTZP3hifeHV/L5d86/d85qJF/PUlS12pQUQ2quoxFwSyK3Kn6NHNUf7+4WaGk8qZC2bz4TNP4YKl1TRUFrldmjHGBUvmlHDpyhp++MzrfPy8BZQWZNZKtBb6J+nwUIKvrGvhgQ17eef8cr59/akZPU3LGDNzbrloEb9p6eC+Z17nlosWu13Om9gI4kl4dX8v77vjD/xs415uuXAR9//FmRb4xpgxq+pCXLSsmnv+sIv+wRG3y3kTC/0ToKr87MW9vO+OP9AzMMR9H1vL375nqc2lN8a8xS0XLaJnYJifPL/b7VLexNJqkvoGR/jcA5v5/EMvc9q8ch77q3M5d3GV22UZYzLUafPKOWdRJXf9fhdHhhNulzPGQn8SWtpjXHH7H1i3pZ2/efcSfnzTGVSX2EVUxpi3d8tFizjQN8gDL+6deOcZYqH/NlSVHz/7Olff+QwDQyP811+cya0XL7aFzowxk3JGQwXvnF/O9556jcGRzDjbt9A/jtjhYW7+r03846Mt/MnC2Tz2mXM5c8Fst8syxmQREeHWixazL3aEhzdF3S4HsCmbx7Rl7yFuuX8T+w4d4UuXLeMvzl1gd5gyxpyUcxdX0hQOceeTrVxzephclyd+2Jn+OKrK3U/v5JrvPUMyCQ988iw+ef5CC3xjzEkTEW65aDF7uw+zbnO72+XYmf6onv4h/u7BLfzP1k4uWTGHb17TRKgws66kM8Zkp3ctr2b53FK+82QrV51a5+q4oJ3pAxte7+bybz/N7189wFeuWMH3P3y6Bb4xZtqICLdcuIidXf081rxv4jekka9DP5lUvrO+lT+/6znycgI89Ok/4SNnNzD+BjHGGDMdLl1Vw8KqIu54opVk0r2FLn0b+gf6Brnxhy/wzd9u57JVNfzy1nNYHQ65XZYxxqOCAeGWixaxfX8v/7N1v2t1+DL0n3ntAJfd9jQv7Orm61ev5vbrT6Ukw1bCM8Z4zxWNtcyrKOT2J1pxa1l7X4V+Iql86/FX+dDdz1NakMMjN5/NB8+YZ905xpgZkRMM8JcXLKQ5GuOpV7tcqcE3ob8/foQP3f0ct/1uB+8/Ncy6W85h+dxSt8syxvjM+08LUxsqcO1s3xeh/+T2Ti677Wm27I3x7x9o4j+ubaIo32arGmNmXl5OgE9dsJCNu3t4dufBGW/f06E/nEjyjV9v4yM/fJHqknx+cevZXHN62O2yjDE+d+2aeqpK8rnjidYZb9uzoR89dJg///6zfO+p1/jgGfN45OazWVRd4nZZxhhDQW6QT563gGdeO8jG3d0z2rYnQ/+lPT1cftvTvLq/j9uvP5WvX73abkxujMkoHzxjHuWFudw+w2f7ngz9hdXF/MnC2fzy1nO4oqnW7XKMMeYtCvNy+Pi5C3hyexfNbbEZa9eToV9akMt3/9fpzK8scrsUY4w5rhvOOoXSghzuWL9jxtr0ZOgbY0w2KCnI5SNnN/Dblv1s64jPSJsW+sYY46KPnT2forwg31n/2oy0Z6FvjDEuKivM48NnzeeXL7fzWldf2tuz0DfGGJd9/NwG8nMC3DkDZ/sW+sYY47LK4nyuXzuPRzZH2ds9kNa2LPSNMSYDfPK8hQRFuPPJ9J7tW+gbY0wGqAkV8IE1YR7cuJd9scNpa8dC3xhjMsSnzl+IKnz/qZ1pa8NC3xhjMkR9RSFXn1rH/S/sobP3SFrasNA3xpgM8ukLFjKcSHLP07vS8vUt9I0xJoMsqCrmhrPmMzdUkJavP+N3EhGRS4HbgCBwt6p+Y6ZrMMaYTPaV961M29ee0TN9EQkC3wEuA1YA14vIipmswRhj/Gymu3fWAq2qulNVh4CfAlfOcA3GGONbMx36dcDecc/bnG1jROQTIrJBRDZ0dblzt3hjjPGqjBvIVdW7VHWNqq6pqqpyuxxjjPGUmQ79KFA/7nnY2WaMMWYGzHTovwgsFpEGEckDrgPWzXANxhjjWzM6ZVNVR0TkFuC3pKZs/kBVW2ayBmOM8bMZn6evqo8Bj810u8YYY0BU1e0ajktEuoDdU/gSlcCBaSonk9ryentePraZbs/LxzbT7WXTsZ2iqsecCZPRoT9VIrJBVdd4rS2vt+flY5vp9rx8bDPdnleOLeOmbBpjjEkfC31jjPERr4f+XR5ty+vtefnYZro9Lx/bTLfniWPzdJ++McaYN/P6mb4xxphxLPSNMcZHsjr0RWS+iEQm2EdE5Nsi0ioiL4vIaWlub5mIPCsigyLytyfb1gm09yHnuJpF5BkRaUpjW1c6bW12VkI952Tammx74/Z9p4iMiMg16WxPRC4QkZhzfJtF5J/S1da49jaLSIuIPHUybU22PRH5u3HHFRGRhIhUpLG9kIj8QkS2OMf30TS2VS4iP3e+N18QkVVpaOO4P9cicqmIbHcy5osz0N4PRKRzsj8/R5vxK3JdcBmw2PlzBvBd5+906QY+A1yVxjbG2wWcr6o9InIZqcGfdB3f74B1qqoi0gj8DFiWpraAsRvv/Bvw3+lsZ5ynVfW96W5ERMqAO4FLVXWPiFSnsz1V/SbwTaftK4DPqWp3Gpu8GXhFVa8QkSpgu4j8xLmPxnT7e2Czql4tIstI3ajp4mlu45g/1+NuDPVuUkvFvygi61T1lXS05/gRcAdw38l84aw+03cEReT/OGcT/y0is456/UrgPk15DigTkbnpak9VO1X1RWB4Cm2cSHvPqGqP8/Q5UiuXpqutPn1j5L8ImOosgIn+7wBuBR4COqfY1mTbmy4TtfVB4GFV3QOp75s0tzfe9cD9aW5PgRIREaCYVIiNpKmtFcATAKq6DZgvInOms423+bk+2RtDnXSOqOrvSf17nhQvhP5i4DuquhI4BPzZUa9PeOOWaW5vup1IezcBv05nWyJytYhsA34FfGwKbU3YnojUAVeT+u1sOkzm3/Isp0vi1yIylRuVTtTWEqBcRJ4UkY0icsMU2ppMewCISCFwKakP0nS2dwewHGgHmoG/UtVkmtraArwfQETWAqdw4ic/J/tzfbL5MtM5MsYLob9LVTc7jzcC8/3YnohcSCr0v5DOtlT156q6jNSvnV+dQluTae8/gS9MISxOtL1NpNYsaQJuBx5JY1s5wOnAnwLvAf5RRJaksb1RVwB/nIaunYnaew+wGagF3gHcISKlaWrrG6R+g99M6jfDl4DENLcx3Wa6vTFeCP3BcY8TvHWcYrpv3DJRe9Ntwvac/vW7gStV9WA62xrl/Iq5QEQq09jeGuCnIvI6cA1wp4hcla72VDWuqn3O48eA3Ckc30TH1gb8VlX7VfUA8HvgpAbhJ9neqOuYetfOZNr7KKnuK1XVVlJjTyc7/jOZ/7ePquo7gBuAKmDndLbxNk42X2Y6R8Z4IfQnsg64QVLOBGKqus/toqaLiMwDHgY+rKqvprmtRU4fLZKaBZUPTOVD5m2paoOqzlfV+cCDwF+q6iPpak9EasYd31pSPx/pOr5HgXNEJMfpcjkD2JqmtoDUjBrgfKftdNuDM5jq9K8v5cSDeFJEpExSN2UC+Djwe1WNp6OtY8i6G0N5cvaOiHwKQFW/R2rt/suBVmCA1BlI2toTkRpgA1AKJEXks8CK6fwmPOr4/gmYTeosGGBkOlfmO6qtPyP1AToMHAb+fNzAbjraS7uj2rsG+LSIjJA6vuum8/jGt6WqW0XkN8DLQBK4W1VPagreZNpzNl0N/Leq9k9nO8dp76vAj0SkGRBS3XTTtizxUW0tB+4VEQVaSHVzTmsbb/dzLdN0Y6gTaO9+4AKgUkTagC+r6j2Tbmeaf2aNMcZkMD907xhjjHFY6BtjjI9Y6BtjjI9Y6BtjjI9Y6BtjjI9Y6BtjjI9Y6BtjjI/8fyQWstBjBs4UAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot the norm of hidden states across layers for first word \n",
    "def norm(layer):\n",
    "    return torch.linalg.norm(hidden_states[layer][0][0]).cpu().numpy()\n",
    "\n",
    "norms = list(map(norm,output_names))\n",
    "plt.plot(output_names,norms)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bcb93733",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228988f38e0>]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkVUlEQVR4nO3deXiV5Z3/8fc3C0lIAgkkYQmBsMkiqGBYBFvt2FqxWsVRxKWlasWq7cy08/t1bDu/OlfbmXY6c03HTq27o7ZeIO5O3WunogGUoMhuOWFLwpINyEb2+/fHeYIRgSznnJzt87quXJw85znnvh8ln3NzP3furznnEBGR+JAQ7g6IiMjAUeiLiMQRhb6ISBxR6IuIxBGFvohIHEkKdwdOJycnxxUWFoa7GyIiUWXDhg3Vzrnckz0X0aFfWFhISUlJuLshIhJVzGzvqZ7T9I6ISBxR6IuIxBGFvohIHFHoi4jEEYW+iEgcUeiLiMQRhb6ISBxR6ItIwJrbOnhxYwXv7KwKd1ekBxH9y1kiEtm27a/jqfX7eP7DCuqa20lKMJ64eS4LJuWEu2tyCgp9EemThpZ2Xtq4n6fW7+Oj8qMMSkpg0YyRXDkrn395eTvf+v0GnrtjAZPyMsPdVTkJhb6I9Mg5x4dlR1j5/j7+sOkATa0dTBmRyd2XT2fxrHyyBg8CYFJuBot/W8xNj63n+TsWkpOREuaey4kU+iJySocbW3n+wwpWrt/HXw41MHhQIpefNZqlcws4pyALM/vU+QXDBvPwsjlc+8Babn2ihBW3zic1OTFMvZeTUeiLyKd0djrW7aphxfoyXt9ykNaOTs4uyOLnV83k8rNHk5Fy+tg4pyCL/7z2HG5/8gP+/umP+K+ls0hIsNO+RgaOQl9kANU3t5GanEhyYuQtnKusa+bpDeWsKiljb00TQ1KTuH7eWK6dU8C0UUP69F6LZo7iB4um8vNXdzBu2GC+f8nUEPVa+kqhLzIAtlQc5cHVu3h58wEMKMxJZ3JeBpO6fU3MzRjwqZD2jk7e/ksVK9eX8acdlXR0OuZPGMZ3v3gGl8wYGVB/ln9+Antqmvjtn0spHJ7OkjkFQey59JdCXyREnHMU+2p4YHUp7+ysJiMliWXnFZKanMDOygY+PljPG9sO0dHpADCDMdlpTM7L9H8Q5GYwaYT/A2FIanJQ+1ZW28SqkjKeLinnYF0zORkp3Pq5CVw7p4DxOelBacPM+MkVZ1J+uIkfPr+Z/Ow0FmopZ9iZcy7cfTiloqIipyIqEm3aOzp5efMBHly9i63768jLTOHm88dz/byxnwnvlvYO9lQ34atswFfZwM7KenyVDeyqbqS1vfP4eXmZKUwe4X0Q5GUwyftgyMkY9JmbqafS0t7Bm9sO8dT6Mt71VQNwwRm5LJ0zloum5YVsyqmuuY2r71vDgaPNPHf7AiaP0FLOUDOzDc65opM+p9AXCY6m1nZWrS/j4Xd3U374GBNz07nt8xO5YtZoUpL6Nk3S0ekoq23yPgj8Hwi+qgZ8h+ppbO04ft7QtORPTRNNzMtgcl4Go4emHb956qusZ+X7ZTz3YQW1ja3kZ6VxTdEYlhQVMDorLaj/DU6l/HATV967htTkBJ6/YyG5mVrKGUoKfZEQqmlo4fG1e3li7R6ONLVRNC6b2y6YyEVT84K+asU5x8G6Zv+HwSHvg8D7UKhtbD1+XlpyIhPz0klMSOCjsiMkJRhfmj6Ca+cU8LnJuSSGYTXNR2VHuPbBtUwdOYSVy7WUM5QU+iIhsLemkYff2c2qkjJa2ju5ePoIbrtgAueOGxaW/tQ0tHzyLwLvg6DuWBuXzhzFVbPHRMTo+rUtB7n9yQ0smjGS31w3W0s5Q+R0oa8buSJ9tKn8CA+s3sWrmw+QlJDAVbPz+ebnJjApLyOs/RqekcLwjBTmTRge1n6cziUzRvLDRdP451e288thH3PXIi3lHGgKfZFecM6xemc1D7xdyprSGjJTk7jtgonctKCQvCGp4e5eVPnm58azu6aR+98upXD4YJbOHRvuLsUVhb7IabR1dPKHTft54O1d7DhYz8ghqfzo0mksnVtAZpCXUcYLM+MnXz2T8sPH+NELW8jPTuNzk3PD3a24oTl9kZNobGln5foyHnlnF/uPNnPGiAyWf34iXz17NIOSIu+3aaNRfXMb19y/lorDx3j2jgWcoaWcQaMbuSK9VFXfwuNr9vC7dXs5eqyNeeOHcdsFE7jwjOCvxBGoOHKMK+8tZlBiAi/cqaWcwaIbuSI92F3dyEPv7OKZDeW0dXTy5ekjue2CCcwamx3ursW0/Kw0HllWxJIH1vLNJ0pYeet80gZpKWcoKfQlrrS2d3KkqZWaxlYON7ZS3djKq5sP8NrWgyQnJnD1uWP45vnjmZAb3pU48eSsMVncs3QW3/r9Br63aiP3Xq+lnKGk0Jeo1dnpqGtuOx7gtY2tHO4W6LWNbZ/6/nBjK/Ut7Z95nyGpSdx54SSWLSjU9EKYfPnMkfzo0mn87OXt/OvrO/jBomnh7lLMUuhLRGloaWd3VSM1jS3+wG7wB3ltY9vxYK9t8kK8qZXOU9ySSk1OYHh6CtnpyWQPHkTh8MEMSx/EsMGDyE4f5H/sfRVkD9aUQgS45fzx7Klp5IG3dzFuWDrXz9NSzlDoMfTN7FHgMqDSOTfjhOf+Hvh3INc5V23+nZ/uAS4FmoBvOOc+8M5dBvyj99KfOeceD95lSDSqbWxl6/6jbKmoY+v+o2zdX8eemkZOXFuQmGBkDx7EMC/AJ+dlHA9s//FPvrK9YFeIRx8z458uP5Oy2mP8vxe3MCY7jc+foaWcwdbj6h0z+zzQADzRPfTNrAB4GJgKnOuF/qXAd/CH/jzgHufcPDMbBpQARYADNnivOXy6trV6JzZ07RfTFe5bKurYtv8o+482Hz8nPyuNM0cP4czRQ5kyMpPczJTjI/PM1CTN8caR7ks5n7l9AVNGailnXwW0esc5t9rMCk/y1K+A7wMvdjt2Bf4PBwesM7MsMxsFXAi86Zyr9Tr0JnAJsKIvFyKRr7PTsbe26TMj+K7NwMxgfE46RYXDOHP0EGbkD2X6qCFkpw8Kc88lUmSmJvPoN+Zw5b3F3PzYep6/cwF5mfqt52Dp15y+mV0BVDjnPjphL+98oKzb9+XesVMdP9l7LweWA4wdqzm9SNbW0UlpVcMn4V5Rx7YDdTR4N0uTE43JeZl8cVoeZ44eypmjhzBt1BDSe6ixKjI6K41Hls1hyQNrufXxElYuP09TdkHS558+MxsM/BC4OPjdAefcg8CD4J/eCUUb0nfNbR3sOFjPlgr/yH3b/qNsP1h/vNBHWnIi00ZlsnhWPjPy/dM0k0dk9HkfeZEuM8cM5dfXzWL570r4u6c+5L4bztU0XxD0Z8g1ERgPdI3yxwAfmNlcoALoXghzjHesAv8UT/fjf+5H2zKAqhtaeOHDCl7YWMH2A/XHy/oNSU1iRv5Qlp03jhn5/hH8+JyMsOzRLrHtS9NH8I9fmc5P/7CNX7y2gx9eqqWcgepz6DvnNgN5Xd+b2R6gyLuR+xLwbTNbif9G7lHn3AEzex34FzPr+vXGi4EfBNx7Cbr2jk5W76xi1fpy/rj9EO2djnMKsrjjwonHb7SOyU7rdYk+kUDdvLCQvTWNPLh6F+OGD+aGeePC3aWo1pslmyvwj9JzzKwcuNs598gpTn8F/8odH/4lmzcBOOdqzeynwHrvvJ903dSVyLC7upGnS8p49oNyDtW1MDx9EDctLGRJUYFqmkpYmRk/vmw6ZbVN/PjFrYzJHswFWsrZb9pwLY41tbbzyuaDrFpfxvt7aklMML4wJZdrigr4q6mhK5Qt0h8NLe1cc/9aymqbeOb285g6cki4uxSxtMumHOec44N9R3i6pIz/+Wg/ja0dTMhJ55qiAq6anc8IFQSRCHbgqH9XzkQzXrhzoQrYnIJ22RSq6lt4/sNyVpWU46tsYPCgRL4ycxRL5hRQNC5bc/QSFUYN/WQp5y2Pl/DUbfMZPEgx1hca6cew9o5O/vxxFU+VlPG/Oypp73ScOy6bJUVj+MpZo8nQenmJUm9tP8StT5Qwa2w2D3+9SL/cdwJN78QZX2UDT28o47kPKqiqbyEnI4W/Pjefa84tCHvxbpFgeXnTAb67aiNjstN4/Ka5FAwbHO4uRQxN78SBhpZ2Xtl0gFUlZZTsPezdlM3j2jkFXDglVzdlJeZ85axR5GamcOsTJSz+7Rr++xtzmDlmaLi7FfE00o9izjk27D3MU+vLeHnzAZpaO5iQm861RQUsnp2v/UokLvgq61n26HoON7Vy7w2z+cKUvJ5fFOM0vRNjDje2snJ9GU+XlLGrupH0QYlcdtZolswZw+yxuikr8edQXTM3/fd6Pj5Uz88Xz2TJnIKeXxTDNL0TQzo7HV979D22VNQxpzCb2y+cyKUzR2kTM4lrI4ak8tRt87njyQ/4/rOb2H/0GH970WQNgE5CSRFl3th2iC0Vdfzy6rNYUhTfoxmR7rq2ZL7r2c385x93cuBIMz9bPEP3s06g0I8inZ2Oe97ayYScdK6addKdqUXiWnJiAv9+zVmMzkrlv/7k41B9M/deP1v/Eu5GH4FR5I1th9h+oI7vXDSJJI1eRE7KzPj7i6fwL4tnsvovVSx9cB1V9S3h7lbEUHJEie6j/MvPGh3u7ohEvOvnjeWhrxfhq2zgqvuKKa1qCHeXIoJCP0polC/SdxdNG8GK5fNpaung6vvWsGGvNvdVekQBjfJF+u+cgiyevX0BQ9OSuf6h93hty8FwdymsFPpRQKN8kcAU5qTz7O0LmDZqCLc/uYEn1u4Jd5fCRgkS4TTKFwmO4RkprLh1PhdNzePHL27lF6/uoLMzcn85NVQU+hFOo3yR4EkblMj9N57LDfPGcv/bpXx31UZa2jvC3a0BpcWrEUyjfJHgS0pM4GdXziA/O41fvvYxlXUt3P+1cxmalhzurg0IDR0jmEb5IqFhZtxx4ST+Y8nZrN9Ty5L713Lg6LFwd2tAKEkilEb5IqF31ewxPHbTXCqOHGPxvWvYcbAu3F0KOYV+hNIoX2RgnD85h1W3nYfDcc19a1lTWh3uLoWU0iQCaZQvMrCmjx7Cc3csZOTQVJY9+j4vbqwId5dCRqEfgTTKFxl4+VlpPPOtBcwem83frtzIA2+XEsn1RvpLiRJhNMoXCZ+hg5N54pa5XHbWKH7+6g7+6aWtdMTYWn4t2YwwXaP8X117tkb5ImGQkpTIr5fOYtTQVB56ZzcH65q5Z+ksUpMTw921oFCqRBCN8kUiQ0KC8aOvTOfHl03njW2HuP6hdRxubA13t4JCoR9Bukb53/4rzeWLRIKbzx/PvdfPZsv+Oq6+fw1Nre3h7lLAlCwRomuUPz4nna+erVG+SKS4dOYoHrjxXEqrGnlx4/5wdydgCv0IcXzFjkb5IhHnwim5TB2Zye/X7Y36FT1KlwigUb5IZDMzbpw/jq3769hYdiTc3QmIQj8CaJQvEvmunJVP+qBEfrdub7i7EhAlTJhplC8SHTJSkrhq9hj+sOlAVK/kUeiHmUb5ItHjxvnjaG3v5OkNZeHuSr8pZcJIo3yR6DJlZCZzCrN58r19UVt1S6EfRhrli0SfG+ePY29NE+/6onM3TiVNmDjn+LVG+SJR55IZIxmePihqb+j2GPpm9qiZVZrZlm7H/s3MdpjZJjN73syyuj33AzPzmdnHZvblbscv8Y75zOyuoF9JlHlj2yG2aZQvEnVSkhK5dk4Bb20/xP4j0Vdtqzdp8xhwyQnH3gRmOOfOAv4C/ADAzKYDS4Ezvdf81swSzSwRuBdYBEwHrvPOjUvOOe75o0b5ItHqurljccCK9/eFuyt91mPoO+dWA7UnHHvDOde1CcU6YIz3+ApgpXOuxTm3G/ABc70vn3Nul3OuFVjpnRuXNMoXiW4FwwbzhSl5rFxfRltHZ7i70yfBSJybgVe9x/lA97VM5d6xUx3/DDNbbmYlZlZSVVUVhO5FFo3yRWLD1+aPo6q+hTe2Hgp3V/okoNA3sx8B7cCTwekOOOcedM4VOeeKcnNzg/W2EUOjfJHY8PkzchmTncbv1u0Jd1f6pN+pY2bfAC4DbnCf7EBUARR0O22Md+xUx+OKRvkisSMxwbhh3jjW7arFV1kf7u70Wr9C38wuAb4PfNU519TtqZeApWaWYmbjgcnA+8B6YLKZjTezQfhv9r4UWNejj0b5IrFlSdEYBiUm8Pt10XNDtzdLNlcAa4EpZlZuZrcAvwEygTfNbKOZ3Q/gnNsKrAK2Aa8BdzrnOrybvt8GXge2A6u8c+OGRvkisWd4RgqLZo7k2Q3lUVNgpccauc65605y+JHTnP/PwD+f5PgrwCt96l0M6Rrl/8cS1b4ViSVfmz+OFzfu56WN+1k6d2y4u9Mjpc8A0ChfJHadOy6bqSMz+V2UFFhR6A8AzeWLxK5oK7CiBAoxjfJFYl80FVhR6IeYRvkisS8jJYnFs/OjosCKUiiENMoXiR9dBVae2VAe7q6clkI/hDTKF4kfU0cOYU5hNr9/b29EF1hREoWIRvki8ScaCqwo9ENEo3yR+BMNBVaURiGgUb5IfEpJSmRJhBdYUeiHgEb5IvHreq/AysoILbCiRAoyjfJF4ltXgZUVEVpgRaEfZBrli8iN88dGbIEVpVIQaZQvIgAXnJEXsQVWFPpBpFG+iIC/wMr188ZGZIEVJVOQaJQvIt0tKSqIyAIrCv0g0ShfRLrLidACK0qnINAoX0RO5sb546hvaeeljfvD3ZXjFPpB8L8fV2qULyKfURSBBVaUUEHw5rZDZKYmaZQvIp9iZtwQYQVWFPpB8K6vmvkThmuULyKfsdgrsBIpN3SVUgHaV9NEWe0xzp+UE+6uiEgE6iqw8j+b9kdEgRWFfoCKS/1bqC6cNDzMPRGRSBVJBVYU+gEq9lUzYkgKE3Mzwt0VEYlQXQVWnoyAAisK/QB0djrWlNawcGIOZhbu7ohIBLtx/jj2RECBFYV+AHYcrKe2sZWFms8XkR50FVj5fZgLrCj0A7Dm+Hy+Ql9ETq+rwMofw1xgRaEfgHd91UzITWfk0NRwd0VEokAkFFhR6PdTa3sn7++u1VJNEem1SCiwotDvp41lR2hq7WDBRIW+iPReuAusKPT7qdhXTYLBeRO0Pl9Eeq+rwEq4bugq9Pup2FfNzPyhDB2cHO6uiEgU6SqwsnZXTVgKrCj0+6GhpZ2NZUe0akdE+mVJUQHJiRaW/XgU+v3w/u4a2judQl9E+iUnI4VLZ47i2Q8GvsCKQr8fin01DEpK4Nxx2eHuiohEqRvnj6O+eeALrCj0+6HYV82cwmxSkxPD3RURiVLhKrDSY+ib2aNmVmlmW7odG2Zmb5rZTu/PbO+4mdmvzcxnZpvMbHa31yzzzt9pZstCczmhV93Qwo6D9VqqKSIBCVeBld6M9B8DLjnh2F3AW865ycBb3vcAi4DJ3tdy4D7wf0gAdwPzgLnA3V0fFNFmTWkNgH4pS0QCFo4CKz2GvnNuNVB7wuErgMe9x48DV3Y7/oTzWwdkmdko4MvAm865WufcYeBNPvtBEhWKd1YzJDWJGflDw90VEYlyXQVW/jCABVb6O6c/wjl3wHt8EBjhPc4HyrqdV+4dO9XxzzCz5WZWYmYlVVVV/exeaDjneNdXzXkTh5OYoK2URSRwN84fR8sAFlgJ+Eau89+BCNpdCOfcg865IudcUW5ubrDeNij21TZRceSYlmqKSNAMdIGV/ob+IW/aBu/PSu94BVDQ7bwx3rFTHY8qxT7/fL5CX0SCaSALrPQ39F8CulbgLANe7Hb8694qnvnAUW8a6HXgYjPL9m7gXuwdiyrFvmpGDkllQk56uLsiIjFkIAus9GbJ5gpgLTDFzMrN7BbgF8CXzGwn8EXve4BXgF2AD3gIuAPAOVcL/BRY7339xDsWNfylEatZMGm4SiOKSFB1L7By4GhoC6wk9XSCc+66Uzx10UnOdcCdp3ifR4FH+9S7CLL9YB2Hm9q0VFNEQuL6uWO5/+1SVry3j+9dPCVk7eg3cnup2KfSiCISOgXDBnPhGbkhL7Ci0O+lYl8Nk/IyGDFEpRFFJDS+dt64kBdYUej3QldpxIUTVTBFRELngjPyyM8KbYEVhX4vfLjvMMfaOjS1IyIhlZhg3DA/tAVWFPq90FUacZ5KI4pIiIW6wIpCvxeKS2uYOSaLoWkqjSgioZWTkcKiGaPwVTaEZMvlHpdsxrv65jY2lh3hWxdMCHdXRCRO/PLqs0JWr0Mj/R68v7uWjk7HQu2fLyIDJJQFmhT6PSj21ZCSlMBslUYUkRig0O+BvzTiMJVGFJGYoNA/jcr6Zj4+VK+lmiISMxT6p7G2tGsrZS3VFJHYoNA/jWJfNUPTkjlztEojikhsUOifgnOOYl8N501QaUQRiR0K/VPYW9NVGlFTOyISOxT6p/CutlIWkRik0D+FNaXVjBqayniVRhSRGKLQPwl/acQaFk7KUWlEEYkpCv2T2HagjiNNbZrPF5GYo9A/ieOlEbXfjojEGIX+Sbzrq2ZyXgZ5Ko0oIjFGoX+ClvYO1u+p1aodEYlJCv0TfLD3CM1tnQp9EYlJCv0TrCntKo04LNxdEREJOoX+Cd71VXN2QRZDUlUaUURij0K/m7rmNjaVH9WqHRGJWQr9bt7b5ZVG1Hy+iMQohX43xb5qUpMTmD0uK9xdEREJCYV+N2tK/aURU5JUGlFEYpNC31NZ18xfDjVoakdEYppC37OmqzSibuKKSAxT6Hve9VWTNTiZ6aOHhLsrIiIho9DHXxpxja9apRFFJOYp9IHd1Y3sP9qs+XwRiXkKfaC4az5foS8iMU6hDxTvrCY/K43C4YPD3RURkZAKKPTN7LtmttXMtpjZCjNLNbPxZvaemfnM7CkzG+Sdm+J97/OeLwzKFQSoo9OxdlcNCyYOV2lEEYl5/Q59M8sH/gYocs7NABKBpcC/Ar9yzk0CDgO3eC+5BTjsHf+Vd17Ybdtfx9FjbZraEZG4EOj0ThKQZmZJwGDgAPBXwDPe848DV3qPr/C+x3v+IouAofW7XmnEBaqHKyJxoN+h75yrAP4d2Ic/7I8CG4Ajzrl277RyIN97nA+Uea9t987/TNKa2XIzKzGzkqqqqv52r9fWlFZzxogM8jJVGlFEYl8g0zvZ+Efv44HRQDpwSaAdcs496Jwrcs4V5ebmBvp2p9Xc1sH7u1UaUUTiRyDTO18EdjvnqpxzbcBzwEIgy5vuARgDVHiPK4ACAO/5oUBNAO0H7IN9h2lp79TWCyISNwIJ/X3AfDMb7M3NXwRsA/4XuNo7Zxnwovf4Je97vOf/5JxzAbQfsGJfNYkJptKIIhI3ApnTfw//DdkPgM3eez0I/APwPTPz4Z+zf8R7ySPAcO/494C7Auh3UBT7ajh7zFAyVRpRROJEUs+nnJpz7m7g7hMO7wLmnuTcZuCaQNoLpqPH2thUfoRvf2FSuLsiIjJg4vY3ct/bVUOngwW6iSsicSRuQ39NaQ2pyQnMGpsV7q6IiAyYuA39d33VzB0/XKURRSSuxGXoH6prxlfZwMKJ+i1cEYkvcRn6xd7WC/qlLBGJN3Ea+jVkD05m+iiVRhSR+BJ3oe+co9hXzYKJOSSoNKKIxJm4C/1d1Y0crGvWrpoiEpfiLvSPz+drvx0RiUNxGfr5WWmMU2lEEYlDcRX6HZ2OtaU1LJyk0ogiEp/iKvS3VBylrrldSzVFJG7FVegXl3qlETWfLyJxKr5C31fN1JGZ5GamhLsrIiJhETeh39zWwfo9hzXKF5G4Fjehv2HvYVrbOzl/stbni0j8ipvQL/ZVk5RgzB2v0BeR+BVXoX92QRYZKQEVCxMRiWpxEfpHm9rYXHFUSzVFJO7FReiv9Uojav98EYl3cRH6a0qrSUtOZNbY7HB3RUQkrOIi9It91cwdP4xBSXFxuSIipxTzKXjwaDOlVY2cr/l8EZHYD/2urZS1f76ISJyE/rD0QUwbqdKIIiIxHfrOOYpLqzlv4nCVRhQRIcZDv7SqgUN1LaqSJSLiienQL/bVAOgmroiIJ6ZD/11fNWOy0xir0ogiIkAMh357RyfrdtVolC8i0k3Mhv6W/XXUN7ezQKEvInJczIb+8fX52m9HROS4mA79qSMzyclQaUQRkS4xGfrNbR2U7D2srZRFRE4Qk6Ffd6yNRTNGctHUvHB3RUQkosRkGam8Iancs3RWuLshIhJxAhrpm1mWmT1jZjvMbLuZnWdmw8zsTTPb6f2Z7Z1rZvZrM/OZ2SYzmx2cSxARkd4KdHrnHuA159xU4GxgO3AX8JZzbjLwlvc9wCJgsve1HLgvwLZFRKSP+h36ZjYU+DzwCIBzrtU5dwS4AnjcO+1x4Erv8RXAE85vHZBlZqP6276IiPRdICP98UAV8N9m9qGZPWxm6cAI59wB75yDwAjvcT5Q1u315d6xTzGz5WZWYmYlVVVVAXRPREROFEjoJwGzgfucc7OARj6ZygHAOecA15c3dc496Jwrcs4V5ebmBtA9ERE5USChXw6UO+fe875/Bv+HwKGuaRvvz0rv+QqgoNvrx3jHRERkgPQ79J1zB4EyM5viHboI2Aa8BCzzji0DXvQevwR83VvFMx842m0aSEREBkCg6/S/AzxpZoOAXcBN+D9IVpnZLcBeYIl37ivApYAPaPLOFRGRAWT+affIZGZV+D84+isHqA5SdyKprVhvL5avbaDbi+VrG+j2ounaxjnnTnpTNKJDP1BmVuKcK4q1tmK9vVi+toFuL5avbaDbi5Vri8m9d0RE5OQU+iIicSTWQ//BGG0r1tuL5Wsb6PZi+doGur2YuLaYntMXEZFPi/WRvoiIdKPQFxGJI1Ef+mZWaGZbejgnKHv597KtqWa21sxazOz/9KedPrZ3g3dNm81sjZmdHeL2rvDa2+htjHd+qNrqdu4cM2s3s6v701Zv2zOzC83sqHdtG83sx6Fsr1ubG81sq5m9Haq2zOz/druuLWbWYWbDQtjeUDP7HzP7yLu2fv8yZi/byzaz572/m++b2Ywgv/8pf67N7BIz+9jLl7tO9R5BbO9RM6vs7c/PiWKyctZJdN/Lfx7+vfznhaitWuBv+GRL6VDbDVzgnDtsZovw3/wJ1bWBv0bCS845Z2ZnAauAqaFqzMwSgX8F3ghVGyd4xzl32UA0ZGZZwG+BS5xz+8wsZPU9nXP/Bvyb1+7lwHedc7Whag+4E9jmnLvczHKBj83sSedca4ja+yGw0Tm32MymAvfi3xomWE76c+39/bwX+BL+/cjWm9lLzrltoWjP8xjwG+CJ/rxx1I/0PYlm9pA3onjDzNJOeD6Ye/mfti3nXKVzbj3Q1s/372t7a5xzh71v1+HfyC6U7TW4T+7+p9PHXVT70pbnO8CzfLJxXyB6014w9dTe9cBzzrl94P+7E8K2ursOWBFAW71pzwGZZmZABv4Qaw9he9OBPwE453YAhWY24jPv0s/3P83P9VzA55zb5X2grcSfN6FqD+fcavz/PfslVkJ/MnCvc+5M4Ajw1yc836u9/IPUVrD1pb1bgFdD3Z6ZLTazHcDLwM2hasvM8oHFBK/KWm/+W57nTUm8amZnhri9M4BsM/uzmW0ws6+HsC0AzGwwcAn+D9JA9NTeb4BpwH5gM/C3zrnOELb3EXAVgJnNBcbRtwFQf3+u+5stA50jx8VK6O92zm30Hm8ACmOkrV63Z2ZfwB/6/xDq9pxzz3slMq8EfhrCtv4T+IcAw6Iv7X2Af8+Ss4H/Al4IcXtJwLnAV4AvA//PzM4IUVtdLgeKgzC101N7XwY2AqOBc4DfmNmQELb3C/z/gt+I/1+HHwIdQXz/YBvo9o6LldBv6fa4g8/eqwjmXv49tRVsPbbnza0/DFzhnKsJdXtdvH9mTjCznBC1VQSsNLM9wNXAb83syn621WN7zrk651yD9/gVIDmAa+uxPfyjwtedc43OuWpgNf5a06Foq8tSAp/a6U17N+GfunLOOR/+e0+B3Pvpzf+7m5xz5wBfB3Lx7/wblPc/jf5my0DnyHGxEvo9idm9/M1sLPAc8DXn3F8GoL1J3jwt5l8FlQIE+kFzUs658c65QudcIf4iPXc4514IRVsAZjay27XNxf/zEZJr87wInG9mSd60yzxge6gaM39d6wv4pMZFKO3Du5Hqza1PoW8h3CdmlmX+Ld4Bvgmsds7Vhaq9btYDk81svNf+Uvx5E7FidvWOmX0LwDl3PyHey797W2Y2EigBhgCdZvZ3wPRg/gU84dp+DAzHPwoGaA/2znwntPfX+D9A24BjwLXdbuwGu62QO6G9q4Hbzawd/7UtDea1ndiec267mb0GbAI6gYedc/1ahtdTW96hxcAbzrnGYLVxmvZ+CjxmZpsBwz9NF9RtiU9obxrwuJk5YCv+qc6gvf/pfq7N7NvA60Ai8KhzbmuI21sBXAjkmFk5cLdz7pFetxPkv9MiIhLB4mV6R0REUOiLiMQVhb6ISBxR6IuIxBGFvohIHFHoi4jEEYW+iEgc+f+RO9oHKd3LkgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot the norm of hidden states across layers for second word (later ones are fairly similar)\n",
    "\n",
    "def norm(layer):\n",
    "    return torch.linalg.norm(hidden_states[layer][0][1]).cpu().numpy()\n",
    "\n",
    "norms = list(map(norm,output_names))\n",
    "plt.plot(output_names,norms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "66aa3c4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x22898966970>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjZ0lEQVR4nO3deXzUd53H8dcnJwFSIISjHIHU0lLoCZEetrUHpbRri1q7S3fValXUpeu6W7elVq1XXdfd1fWoVVRW7drS2guW4iL0sHc5KlcolHCVhCsJEMhBzs/+Mb/EGcg1mRlmEt7Px2Memfn+fjO/T+b3nXnP9/f7zfzM3REREWmVluwCREQktSgYREQkgoJBREQiKBhERCSCgkFERCJkJLuA7sjPz/fx48cnuwwRkV5lzZo1Fe4+LNr79YpgGD9+PKtXr052GSIivYqZ7erJ/bQpSUREIigYREQkgoJBREQiKBhERCSCgkFERCIoGEREJIKCQUREIigYRKTbjh5rZNHasmSXIQnWK77gJiKp4ctPb+R/1+1hwvBcJo06LdnlSIJoxCAi3VZ2qBaA2oamJFciiaRgEJGo6byPfZuC4STYfbAWnUJV+pIDR+r5/erdyS5DEkTBkGDv7D/KFd97gV+9sqPHj/HEmlKKvr2c5haFiySXmQFw56Nv8S9PrGfLvqNJrkgSQcGQYDsragB4bVtljx9j3pPrqahu4HBtQ7zKEolJ6wC4RvsaOvXw6zuZ+8hbyS4jakkLBjObaWZbzKzEzOYlajnNLU5L2Cftiur6uGzWWbPrEHc9vi7isdtTUl4NgMWwrKZgGYdqG6KqfdXOg/zmtZ0xLDmxXt5azj8/tjYu6+NQTQMrNu2PeKzmFqepuSXmx+6JhqbQchubW2gMq6Gyup7fvr6zz2xajMeHlZbjXqMAR441tn2oSoQ9h+v4/vJ32tbNoZoGnnqr9IT5Dtc2UNfQ3O3HbTyuv311UTHPrt/b9v9tLKti/LxnWbf7cLv3b+0X7T0nJ1NSDlc1s3TgQeA6oBRYZWaL3X1TvJf15ac2sHLnQX5820XsrKzhzkf+zJdmnIU7XHvOCPYcruO9hXkMyslke3k1T6wppTB/AD98biuZ6WmMPK0fl0/I56KCwZw7ehB3Pb6OcXn9eXTlu9Q0NHPm8IEsWb+HuVefyY3nnQ5AXUMzz23ez4bSKn7+0nYg1NE/9qs3+dKMszl39CD+4dG3qG1o5v6bJvP23iNcN2kEAP+6dDMLXt3BC1+6isL8AbxWUtH2v3x98SZeKamg5IEbyEhP44UtB3hidSn/est5uMOitWXcOnUsP/vTNipr6vmfN94FoCCvP1dPHA6E3ix/sPwdLjtzKJcUDuXosSYcZ1BOJu6w9UA160oPM/2cEeQNyKKlxXl2w17MYFphHq+WVLBlXzUfvaSAMUP6A3/pzGbGc2/v5+yRuew5fIzsjDSe/nMZX/vAJNLSQtH4ytYKDtU2cON5p/PxBStxh3tumMj28hrOOT2XssN1lByo5soJw6iubyJ/YDb9MkOfX97ccZCJI3MZ3D8LgKraRo4ca+RYYzM/fr6Exev2APDRSwpYtHYPR4+FPs2uu38Gg3IyKTlQzWvbKnh+8wHyBmTxmSvOYFt5NdMK8xie2489h+vIzkijpr6Zbz27iZzMdO6acRY5WekU7znC8NxszsgfSL/MNJpbnIrqBqrrmzhz+ECq6hqpb2pmYHYGD724jR8/X8L0c0awckcl/bMymD1tLP+1Ymvbupw8ahCjBvdjza5D5PXP4rIz8wF4ccsB1u4+zOa9Rxl+Wja3Th1LRU09udkZTB41iMaWFk7rl4m7s3b3YSaNOo1fvryD26YVkGYwKCeTHz1XwqCcDK49J9SnRg3OYW9VHa+VVDJ1/BA2llXxgfNHkWawaO0eCob2Z8LwgSwr3s/idXv49Sfe27a+ILSPLG9AFjmZ6Se8vu749WqumTic5hbnN3dM4+ixRhyorW+mqaWFmvpmCvL6k5OVzu/e3EVNfRMfv3Q82RlpLCvex5VnDeOWh17nSF0jP//YVMbm9eeN7ZV89uE1ALx899WMzevf9tz8zxvv8tBHp1BT38Sy4n18eMoY1u4+zIVjB1NZ3cD3l2/BMHYfqmXeDRM5f8xgADbtOcKw3GyG5WZTfrSeT/9mNZv2HmH04H7cMmUMd/1+Hc9vPsCFYwczanAO/TLTaW5xLvzmcgBmTBrB69sqWfqPV7Bq50GyMtKYMWkkr5ZUcNXZwzAznvlzGV98bC0vfukqXi6p4DvPvt32PH1zySY2llVRG4TMrAdf5cnPXwY4NfXN5PbL4K7H17Gnqo61X5vBnY+8RVVdI/92y/mMHzogYn2cDJaMTy5mdinwdXe/Prh9L4C7/2t78xcVFXlPTtRTXd/Eufcv69a8Y/Ny2H2wLuplhLtlyhhumTqa+xcVs/VAddT3HzMkh9JDoRr+6rzTmTz6NL73f1vandfsL8P5eLj90nHsqKzlpXfK29pGD85hT1Vdu8vpn5XOqME5HDhyjIbmFo41tnT6HOYNyOK0fhnsrKxte+yyw917vs8fM4j1pVVttz9zRSG7D9bxf8X7uv3/DcvNJis9rcNlphl09wPadZNG8GpJRduLfOLIXDbHuK39t3dM48m3Slm0dk+X814/eQSHahpZufMgGWlGU4szZkjo+QxfV9E8x+HSDP7hmgk8svJdyo/Wd7uvDc/N5sDR+hPaczLTqWuM/NTd2lc+eOEonunif/7EZeN58q3StqAfeVo/9h051q3/5f6bJpGTmc68pzZ0OM+Q/pkcqm0E4Iz8AeyorKFw6AC2d3PE8pGpYygaN6TTZUQjPc0i9ifefuk4vjHr3B49lpmtcfeiqO+XpGD4CDDT3T8d3P4YcLG73xk2zxxgDkBBQcHUXbuiPxFRRXU9P1yxlYff6NFJjDqVm53B0fqOt6+2fjI5Xv7AbCqqT2zvzIjTstl/pPP7DMrJpKquMarHnVaYR1VtI1v2h97U2nsDeN+ZQ3m1JHL/yFc/MIlvLel4cDetMI/Sg7Xsqerei7e78gdmUVHd8aaLy8/M5wvXTuC3r+/kQxeNZlt5Nd9Zujlinsx0o7H55Pb5my4Yxf+u6/oNvzO3Th3D79ecuKnjeD3pB/GWnZFGfVPkJpWJI3N5Z//RLsN3QFY6Dc0t3H/TZL7yzMZuLW9AVjpDB2ZTdriO8UP7s638xDf07Iw0Jo7MZV3YB4xPX17IL1/ZwcDsDKo7eS0nQ/7ALCaNGsRL75Rz+Zn5PPypaW07/qPR54IhXE9HDK3W7DrIz/60ndqGJr5wzQSmFeaxq7KWX76yndumFfDSOxUM7p9J+dF6PvG+8RyqaSB/YDaV1Q1sr6hm6/5qrjlnOC9sPsC3n32bL1xzJv84/SyaW5yDNQ3k9stgwSs7+O0bu5g4Mpd/u+V8Rg3OAaCqrpGf/2kbq3Ye5DNXnMF1k0awq7KWlTsPcvcT6wH481evA+DZDXu5YMxgbvrJK221jx/anz/+0/v5yQslDBuYRWVNA1MKhnBRwWCyM9LZUVFDXWMzF44dDMAfNuwlMz2NVbsOUlXbyIenjGFYbjbjh/ansdnZc7iOypoGDtU0MD3YfFVZXc/SDXu58qxhvLD5AD95YRsXF+Zx5Vn5/M17C9i05wj3PLmeX91eBAbDc/tRU9/Eyh0H+eSvVwFw8wWjyO2XwfRJI7j67NBmK3fntl+8weHaRp79whXsP3KM+qYWvvrMRl4JNpENz81mxuQRfOWvJvHG9krue3pjxKfcq84exotbyrltWgH33zSJY43NbcP7b33wXH64Yiu/+/TFZGekkdsvg6EDsyPW/dt7j/Dy1nIWr9vDpj1HWDjnUvYfOcaBo/X83cUFrNt9mOc2HyArPY3rJo3gU79ZRU19M3WNzXzu/e9hSP9MbjzvdNaVHgZg/kvbWV9axT0zJ3LzhaN433ef5yNTx/DvHzmfRWv38J2lb3PH5YVcXJjHh376GgAbvj6DpmYnzYwLvvlHAJ76+8vIyUynur6Jo8cauePXof79s49O4ZGVu/nV7UWUH62nX2Y6uw/WckGwfltanOI9R2hobqZfZjoLV+7mk+8bz46KGsqP1nNr0VgWryvj2fX7yEw3vnvL+VRU1zPrJ69y/02TmDF5JD//0zZW7zxEfVMzmFFV28Djn72Udw/Wsquylrt+vy7iOfz+X1/A9ZNH4oQ251w7cQT9MtMwM4r3VPG1RcWs2XWIsXk5LJp7OXkDsvjvV3ewr+oYr5RU8PWbJ/Pe8Xmh/jzvWQAe+czF7D18rG1ZU8cN4Td3TGNg9l+2bm8sq+I7S9/mtW2VXDh2MHfPPJsvLlxLVkYan3v/e/jKMxu5uDCPxz57adt93J0vPraWP2zYx9NzL+PDP32N+qYWPnV5IV/5q3PYvO8oL24pZ3huNrdMHdN2v7W7D/PBB18F4K2vXsfBmgaWFe/jUE0Dn7nyDP7njV3kZKXz8Ou72Bv2gednH53KfU9vILdfBg9/6mK2V9QwMDudbeU13P3Eev7z1gva/sf/uPUCfv3aDr79wfOYODKXzPQ0fvJ8CT9Y8Q4AP77tIqrrmyg/Ws8Xrp0AQH1TM1npaT0KBeh5MODuJ/0CXAosC7t9L3BvR/NPnTrVU0F9Y7P/4qVtfqyxKS6P925lje89XHdC+4bSw/5uZU1clpFoi9aW+bwn13U4vbm5xRubmk9oP1LX0OnjvltZ46WHat3dfV9VXdtz3tTc4uPuWeLj7lnir2wt73adR+oafMu+I92e/1hjk7e0tJzQfs8T63zcPUv8ly9vd/fQumpo5/9zd/9j8T7/8lPrI9qeXb/HX37nxLp3lFf7vqoT+0IyHD3W6D99oaTtee5KS0uLP795f4fPQ7hVOyr920uK227vPljjNfWNHc7/tWc2+Lh7lvgXHn3rhGl1DU1dLvNvf/G6j7tniX/zf4s7nc/dvfzoMd+6/2in89Q1NPmuihp/9M1dvmbXwba2uob23xMampo7fR7rG5v9kTd3tfsaiQdgtffgPTpZv5W0CphgZoVAGTAb+Nsk1dJtWRlpfPqKM+L2eK071Y537uhBcVtGot18wShuvmBUh9PT0oy0do7Jyu2X2enjhj83I07r13Y9/JGi+QyV2y+zy2WGy844cUcrwNyrz2TL/qPcdEHoQIPO1tV1k0a0HVTQqvUAheONzx/Q7doSbWB2BueO7v7vIJlZ2yixK0Xj8ygKRg9A2wEMHclIDx14kJF24gGU/drZGX5CbUEv6U5fyR+YTf5xI872llkwtD8FQwu6VUdmeucHfmZlpHHbtIJO50mGpASDuzeZ2Z3AMiAdWODuxcmoRXqX8BF1T4fXsRib15+n//59J325J1t7b8TJ0PrGmpVx8td1vJjB2C4CMNUk7ddV3X0psDRZy5feKTwMkpALp4zM9NR4clvr6GlQtfaRZPaVTd+Y2ev6qn52W3qttN72autFutoEcrK0BkKsh/EnY3TZKier601eqSY11r5IDygXEicjVUYMvXgTUm+mYJBeS28ZiZMqI4bMOO3rUF+JTmqsfZEeSObmgb4u4yT/BENH0mOsQ32kZxQM0mvpNZ84qTJiiNs6Vl+JSmqsfZEe0M7nxEmVYIiVekjP9I21L6ckvegTJ1V2PseLqbdERcEgvZYGDIkTr52+yaY+0jN9Y+3LKUmbkhKnrx0mqq4SHQWDiJwgVX4SI1bKg57pG2tfTkn6FJg4qfKTGLFqPVy1b/w3J4+CQXotbUpKnFQ5/j81qjj1KBik10qR9y5JYa1dRH0lOgoG6bU0Yki8mzo518bJcNbIXIC2M9j1lA5XjY5+XVV6Lb3UE6vkgRuSHr6XvSefP/3LVRR0cFKrruizQ88oGKTX0os+sTJS5NvP44bGfnY79ZXopMaaF+mBVNlBKqlMfaQnFAzSa+klL92lvhKdmILBzG41s2IzazGzouOm3WtmJWa2xcyuD2ufGbSVmNm8WJYvpzaNGKQr6iI9E+uIYSPwYeCl8EYzmwTMBiYDM4Gfmlm6maUDDwI3AJOA24J5RaKWIqcMkN5ACRGVmHY+u/vb0O4nt1nAQnevB3aYWQkwLZhW4u7bg/stDObdFEsdcmrSIYjSFfWQnknUPobRwO6w26VBW0ftJzCzOWa22sxWl5eXJ6hM6c30IVC60tpH1FWi0+WIwcxWACPbmXSfuy+Kf0kh7j4fmA9QVFTkiVqO9F4KBpHE6DIY3H16Dx63DBgbdntM0EYn7SJR0c5n6S51legkalPSYmC2mWWbWSEwAVgJrAImmFmhmWUR2kG9OEE1SB+nnc8iiRHTzmcz+xDwY2AY8KyZrXX369292MweJ7RTuQmY6+7NwX3uBJYB6cACdy+O6T+QU5Z2Pkt3qa9EJ9ajkp4Gnu5g2gPAA+20LwWWxrJcEdDmAZFE0TefpddSMEhXPDhsRX0lOgoG6bW0eUC6Sz0lOgoG6bW081kkMRQM0mvpcFXpLnWV6CgYpNfSa126okDoGQWD9FrJPruY9B4aXUZHwSC9l17rIgmhYJBeSzufRRJDwSC9ljYPiCSGgkF6LcWCdJc+Q0RHwSC9lnY+iySGgkF6LeWCdJe+JR8dBYOIiERQMEivpU1J0hX9iF7PKBik19KLXbpLXSU6CgbptTRikK6oi/SMgkF6Lb3mpbsUENFRMEivpRe7SGLEFAxm9u9mttnM1pvZ02Y2OGzavWZWYmZbzOz6sPaZQVuJmc2LZflyatM3n6W7dLhqdGIdMSwHznX384F3gHsBzGwSMBuYDMwEfmpm6WaWDjwI3ABMAm4L5hURkRQRUzC4+x/dvSm4+QYwJrg+C1jo7vXuvgMoAaYFlxJ33+7uDcDCYF4RkYTR4DI68dzHcAfwh+D6aGB32LTSoK2j9hOY2RwzW21mq8vLy+NYpoiIdCajqxnMbAUwsp1J97n7omCe+4Am4HfxKszd5wPzAYqKijxejysiIp3rMhjcfXpn083sE8AHgGvdW79nSBkwNmy2MUEbnbSLiEgKiPWopJnA3cDN7l4bNmkxMNvMss2sEJgArARWARPMrNDMsgjtoF4cSw0iIl3REWzR6XLE0IWfANnA8uCJf8PdP+fuxWb2OLCJ0Camue7eDGBmdwLLgHRggbsXx1iDiIjEUUzB4O5ndjLtAeCBdtqXAktjWa6ISDQ0XoiOvvksIiIRFAwi0udpF0N0FAwi0ucpF6KjYBARkQgKBhHp83S4anQUDCIiEkHBICJ9ngYM0VEwiIhIBAWDiPR5GjBER8EgIiIRFAwi0vdpJ0NUFAwiIhJBwSAifZ7GC9FRMEiv0y9T3VYkkWI9H4PISffKPddQVdeY7DKkF9EuhugoGKTXyR+YTf7A7GSXIdJnaUwuIn2eaS9DVBQMItLnaVNSdGIKBjP7lpmtN7O1ZvZHMxsVtJuZ/cjMSoLpU8Luc7uZbQ0ut8f6D4iISHzFOmL4d3c/390vBJYAXwvabwAmBJc5wEMAZpYH3A9cDEwD7jezITHWICLSKQ0YohNTMLj7kbCbAwAPrs8CfushbwCDzex04HpgubsfdPdDwHJgZiw1iIhIfMV8VJKZPQB8HKgCrg6aRwO7w2YrDdo6am/vcecQGm1QUFAQa5kicgrTPobodDliMLMVZraxncssAHe/z93HAr8D7oxXYe4+392L3L1o2LBh8XpYERHpQpcjBnef3s3H+h2wlNA+hDJgbNi0MUFbGXDVce0vdvPxRUR6RIerRifWo5ImhN2cBWwOri8GPh4cnXQJUOXue4FlwAwzGxLsdJ4RtImISIqIdR/Dd83sbKAF2AV8LmhfCtwIlAC1wCcB3P2gmX0LWBXM9013PxhjDSIindOAISoxBYO739JBuwNzO5i2AFgQy3JFRCRx9M1nEenzNGCIjoJBREQiKBhEpM8zfZEhKgoGEenzFAvRUTCIiEgEBYOI9HnakhQdBYOIiERQMIhIn6cRQ3QUDCIiEkHBICJ9nn5ELzoKBhERiaBgEJE+T/sYoqNgEBGRCAoGERGJoGAQEZEICgYR6fP0I3rRUTCIiEiEuASDmd1lZm5m+cFtM7MfmVmJma03sylh895uZluDy+3xWL6ISGc0XohOrOd8xszGAjOAd8OabwAmBJeLgYeAi80sD7gfKAIcWGNmi939UKx1iIh0RFuSohOPEcMPgLsJvdG3mgX81kPeAAab2enA9cBydz8YhMFyYGYcahARkTiJKRjMbBZQ5u7rjps0Gtgddrs0aOuoXUQkYfSTGNHpclOSma0ARrYz6T7gy4Q2I8Wdmc0B5gAUFBQkYhEiItKOLoPB3ae3125m5wGFwLrgULAxwFtmNg0oA8aGzT4maCsDrjqu/cUOljsfmA9QVFTk7c0jItId2scQnR5vSnL3De4+3N3Hu/t4QpuFprj7PmAx8PHg6KRLgCp33wssA2aY2RAzG0JotLEs9n9DRETiJeajkjqwFLgRKAFqgU8CuPtBM/sWsCqY75vufjBBNYiIADpcNVpxC4Zg1NB63YG5Hcy3AFgQr+WKiEh86ZvPIiISQcEgIiIRFAwiIhJBwSAiIhEUDCLS5+l7DNFRMIhIn+f6imxUFAwiIhJBwSAiIhEUDCIiEkHBICIiERQMIiISQcEgIiIRFAwiIhJBwSAiIhEUDCIiEkHBICIiERQMIiISQcEgIiIRYgoGM/u6mZWZ2drgcmPYtHvNrMTMtpjZ9WHtM4O2EjObF8vyRUQ6Yzrbc4/E45zPP3D3/whvMLNJwGxgMjAKWGFmZwWTHwSuA0qBVWa22N03xaEOEZEIjn5WtSfiEQztmQUsdPd6YIeZlQDTgmkl7r4dwMwWBvMqGEREUkQ89jHcaWbrzWyBmQ0J2kYDu8PmKQ3aOmo/gZnNMbPVZra6vLw8DmWKiEh3dBkMZrbCzDa2c5kFPAS8B7gQ2Av8Z7wKc/f57l7k7kXDhg2L18OKiEgXutyU5O7Tu/NAZvYLYElwswwYGzZ5TNBGJ+0iIpICYj0q6fSwmx8CNgbXFwOzzSzbzAqBCcBKYBUwwcwKzSyL0A7qxbHUICIi8RXrzufvmdmFgAM7gc8CuHuxmT1OaKdyEzDX3ZsBzOxOYBmQDixw9+IYaxARkTiKKRjc/WOdTHsAeKCd9qXA0liWKyIiiaNvPouISAQFg4iIRFAwiEifpZ/E6BkFg4iIRFAwiIhIBAWDiPRZ+hG9nlEwiIhIBAWDiIhEUDCIiEgEBYOIiERQMIiISAQFg4iIRFAwiIhIBAWDiPRZ+kmMnlEwiIhIBAWDiIhEUDCIiEgEBYOIiESIORjM7B/MbLOZFZvZ98La7zWzEjPbYmbXh7XPDNpKzGxerMsXEZH4iumcz2Z2NTALuMDd681seNA+CZgNTAZGASvM7Kzgbg8C1wGlwCozW+zum2KpQ0RE4iemYAA+D3zX3esB3P1A0D4LWBi07zCzEmBaMK3E3bcDmNnCYF4Fg4jEnX52u2di3ZR0FnCFmb1pZn8ys/cG7aOB3WHzlQZtHbWfwMzmmNlqM1tdXl4eY5kiItJdXY4YzGwFMLKdSfcF988DLgHeCzxuZmfEozB3nw/MBygqKlLsi4icJF0Gg7tP72iamX0eeMrdHVhpZi1APlAGjA2bdUzQRiftIiKSAmLdlPQMcDVAsHM5C6gAFgOzzSzbzAqBCcBKYBUwwcwKzSyL0A7qxTHWICLSLv0kRs/EuvN5AbDAzDYCDcDtweih2MweJ7RTuQmY6+7NAGZ2J7AMSAcWuHtxjDWIiEgcxRQM7t4AfLSDaQ8AD7TTvhRYGstyRUQkcfTNZxERiaBgEBGRCAoGERGJoGAQEZEICgYREYmgYBARkQgKBhHps/Qjej2jYBARkQgKBhERiaBgEBGRCAoGEemz9CN6PaNgEBGRCAoGERGJoGAQEZEICgYREYmgYBARkQgKBhERiaBgEBGRCDEFg5k9ZmZrg8tOM1sbNu1eMysxsy1mdn1Y+8ygrcTM5sWyfBERib9Yz/n8N63Xzew/garg+iRgNjAZGAWsMLOzglkfBK4DSoFVZrbY3TfFUoeIiMRPTMHQyswM+GvgmqBpFrDQ3euBHWZWAkwLppW4+/bgfguDeRUMIiIpIl77GK4A9rv71uD2aGB32PTSoK2j9hOY2RwzW21mq8vLy+NUpoicSrIzQ29xaWn6aYxodDliMLMVwMh2Jt3n7ouC67cBj8azMHefD8wHKCoq0o+qi0jUvn7TZEYNzmH6OSOSXUqv0mUwuPv0zqabWQbwYWBqWHMZMDbs9pigjU7aRUTiasiALO6ZOTHZZfQ68diUNB3Y7O6lYW2Lgdlmlm1mhcAEYCWwCphgZoVmlkVoB/XiONQgIiJxEo+dz7M5bjOSuxeb2eOEdio3AXPdvRnAzO4ElgHpwAJ3L45DDSIiEifmnvqb74uKinz16tXJLkNEpFcxszXuXhTt/fTNZxERiaBgEBGRCAoGERGJoGAQEZEICgYREYnQK45KMrNyYFcMD5EPVMSpnHhL5dpA9cUilWsD1ReLVK4N/lLfOHcfFu2de0UwxMrMVvfkkK2TIZVrA9UXi1SuDVRfLFK5Noi9Pm1KEhGRCAoGERGJcKoEw/xkF9CJVK4NVF8sUrk2UH2xSOXaIMb6Tol9DCIi0n2nyohBRES6ScEgIiIR+nQwmNlMM9tiZiVmNi9JNSwwswNmtjGsLc/MlpvZ1uDvkKDdzOxHQb3rzWxKgmsba2YvmNkmMys2s39Msfr6mdlKM1sX1PeNoL3QzN4M6ngsOLcHwfk/Hgva3zSz8YmsL1hmupn92cyWpGBtO81sg5mtNbPVQVtKrNtgmYPN7Akz22xmb5vZpalSn5mdHTxvrZcjZvbFFKrvn4LXxEYzezR4rcSv77l7n7wQOt/DNuAMIAtYB0xKQh1XAlOAjWFt3wPmBdfnAf8WXL8R+ANgwCXAmwmu7XRgSnA9F3gHmJRC9RkwMLieCbwZLPdxYHbQ/jPg88H1vwd+FlyfDTx2EtbvPwOPAEuC26lU204g/7i2lFi3wTJ/A3w6uJ4FDE6l+sLqTAf2AeNSoT5gNLADyAnrc5+IZ987KU9sMi7ApcCysNv3AvcmqZbxRAbDFuD04PrpwJbg+s+B29qb7yTVuQi4LhXrA/oDbwEXE/pGZ8bx65nQCaAuDa5nBPNZAmsaAzwHXAMsCd4UUqK2YDk7OTEYUmLdAoOCNzdLxfqOq2kG8Gqq1EcoGHYDeUFfWgJcH8++15c3JbU+ea1Kg7ZUMMLd9wbX9wGtZypPWs3B8PIiQp/KU6a+YFPNWuAAsJzQKPCwuze1U0NbfcH0KmBoAsv7L+BuoCW4PTSFagNw4I9mtsbM5gRtqbJuC4Fy4L+DTXG/NLMBKVRfuPCzVCa9PncvA/4DeBfYS6gvrSGOfa8vB0Ov4KEYT+oxw2Y2EHgS+KK7Hwmfluz63L3Z3S8k9Ol8GpASZ3Y3sw8AB9x9TbJr6cTl7j4FuAGYa2ZXhk9M8rrNILSJ9SF3vwioIbRppk2y+x5AsJ3+ZuD3x09LVn3Bfo1ZhMJ1FDAAmBnPZfTlYCgDxobdHhO0pYL9ZnY6QPD3QNB+0ms2s0xCofA7d38q1epr5e6HgRcIDZEHm1nr+crDa2irL5g+CKhMUEnvA242s53AQkKbk36YIrUBbZ8scfcDwNOEgjVV1m0pUOrubwa3nyAUFKlSX6sbgLfcfX9wOxXqmw7scPdyd28EniLUH+PW9/pyMKwCJgR76rMIDQcXJ7mmVouB24PrtxPatt/a/vHgCIdLgKqwYWvcmZkBvwLedvfvp2B9w8xscHA9h9D+j7cJBcRHOqivte6PAM8Hn+rizt3vdfcx7j6eUN963t3/LhVqAzCzAWaW23qd0HbyjaTIunX3fcBuMzs7aLoW2JQq9YW5jb9sRmqtI9n1vQtcYmb9g9dw63MXv753MnbeJOtC6EiBdwhtl74vSTU8Smg7YCOhT0mfIrR97zlgK7ACyAvmNeDBoN4NQFGCa7uc0FB4PbA2uNyYQvWdD/w5qG8j8LWg/QxgJVBCaIifHbT3C26XBNPPOEnr+Cr+clRSStQW1LEuuBS39v9UWbfBMi8EVgfr9xlgSIrVN4DQJ+tBYW0pUR/wDWBz8Lp4GMiOZ9/TT2KIiEiEvrwpSUREekDBICIiERQMIiISQcEgIiIRFAwiIhJBwSAiIhEUDCIiEuH/AW0WpAIBZkgBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hidden state waveform for first word\n",
    "plt.plot(hidden_states['h.0'][0][0].cpu().numpy())\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6239b8b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Watch'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(10723)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d692340d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[10723]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.encode('Watch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0e17f5a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2d89baf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "last_layer_normed = model.base_model.ln_f(hidden_states['h.11'][0])\n",
    "all_logits = model.lm_head(last_layer_normed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d76fabf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "logits=all_logits[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b9e71727",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([50257]),\n",
       " tensor([ -3.3300,  -2.4929,  -7.4483,  ..., -14.6817, -10.4255,  -5.0853],\n",
       "        device='cuda:0', grad_fn=<SelectBackward>))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits.shape, logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c38f08dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(278, device='cuda:0')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.argmax(logits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e3027bb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(486, device='cuda:0')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.argmax(last_layer_normed[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9f3338b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ide'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(485)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "8b1de8db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPTNeoForCausalLM(\n",
       "  (transformer): GPTNeoModel(\n",
       "    (wte): Embedding(50257, 768)\n",
       "    (wpe): Embedding(2048, 768)\n",
       "    (drop): Dropout(p=0, inplace=False)\n",
       "    (h): ModuleList(\n",
       "      (0): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (1): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoLocalSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (2): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (3): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoLocalSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (4): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (5): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoLocalSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (6): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (7): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoLocalSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (8): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (9): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoLocalSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (10): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (11): GPTNeoBlock(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPTNeoAttention(\n",
       "          (attention): GPTNeoLocalSelfAttention(\n",
       "            (attn_dropout): Dropout(p=0, inplace=False)\n",
       "            (resid_dropout): Dropout(p=0, inplace=False)\n",
       "            (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "            (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          )\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPTNeoMLP(\n",
       "          (c_fc): Linear(in_features=0, out_features=3072, bias=True)\n",
       "          (c_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       "          (dropout): Dropout(p=0, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "  )\n",
       "  (lm_head): Linear(in_features=0, out_features=50257, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "27a3d402",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CausalLMOutputWithPast(loss=None, logits=tensor([[[ -3.3300,  -2.4929,  -7.4483,  ..., -14.6817, -10.4255,  -5.0853],\n",
       "         [ -0.0507,  -5.2973, -10.6778,  ..., -15.0821, -11.9509,  -6.5869],\n",
       "         [ -5.7390,  -6.8029,  -9.7417,  ..., -13.3648, -11.1025,  -7.3311],\n",
       "         ...,\n",
       "         [ -7.7671,  -5.1437,  -7.8700,  ..., -21.8658, -13.1095,  -1.5508],\n",
       "         [ -8.2493,  -6.0710, -12.2007,  ..., -23.3041, -18.3014,  -9.1534],\n",
       "         [ -5.3456,  -8.2189, -13.6079,  ..., -13.1011, -16.0331,  -7.2978]]],\n",
       "       device='cuda:0'), past_key_values=((tensor([[[[ 3.0395e-01,  2.8227e-01,  2.5899e-01,  ...,  3.6717e-02,\n",
       "            1.5334e-01, -7.2069e-02],\n",
       "          [ 4.2654e-01,  4.2869e-01,  5.2565e-01,  ..., -1.5126e-01,\n",
       "           -1.5136e-01,  1.9724e-01],\n",
       "          [ 1.2494e-01,  1.3457e-01,  2.5722e-01,  ..., -6.1420e-02,\n",
       "            1.6631e-01, -4.5170e-02],\n",
       "          ...,\n",
       "          [ 3.6231e-01,  1.1602e-02,  2.3648e-01,  ..., -3.5882e-01,\n",
       "            2.7190e-01, -3.7830e-02],\n",
       "          [ 3.4541e-01, -2.3671e-02,  3.6047e-01,  ...,  1.3792e-01,\n",
       "           -1.7394e-01,  3.2376e-01],\n",
       "          [ 3.0108e-02, -4.2100e-02,  1.2416e-01,  ..., -6.1611e-01,\n",
       "           -3.1975e-01, -1.8367e-01]],\n",
       "\n",
       "         [[-2.1133e-01,  4.4042e-01, -2.1468e-01,  ..., -4.7462e-01,\n",
       "            5.8422e-01,  1.7098e-01],\n",
       "          [-2.0270e-01,  1.2119e+00, -1.8230e-01,  ..., -3.0898e-01,\n",
       "            5.5387e-01,  1.1504e-02],\n",
       "          [-3.1592e-01,  6.3633e-01, -6.0788e-01,  ..., -6.9493e-02,\n",
       "           -6.8099e-01,  2.5172e-01],\n",
       "          ...,\n",
       "          [-2.9317e-01,  8.2625e-01, -5.6060e-01,  ...,  5.5693e-02,\n",
       "           -9.0270e-01,  8.2348e-02],\n",
       "          [-4.8034e-01,  3.1867e-01, -3.4136e-01,  ..., -5.1586e-01,\n",
       "            3.9195e-01,  1.0581e-02],\n",
       "          [-5.7552e-01,  7.8861e-01, -6.8420e-01,  ..., -8.0105e-01,\n",
       "            8.8289e-02,  3.7978e-01]],\n",
       "\n",
       "         [[-4.3179e-01,  3.8149e-01, -8.0136e-01,  ..., -1.2256e-01,\n",
       "            1.8879e-02,  6.0345e-01],\n",
       "          [-1.4544e-01,  4.9492e-01, -3.3063e+00,  ..., -6.7634e-01,\n",
       "           -3.8061e-02,  3.3137e-01],\n",
       "          [-5.8597e-02,  2.3447e-01, -2.6575e+00,  ...,  2.2741e-01,\n",
       "            2.0389e-01,  6.2222e-01],\n",
       "          ...,\n",
       "          [-3.6043e-01, -3.3232e-02, -1.4091e+00,  ..., -1.0074e-01,\n",
       "           -4.6132e-03,  3.1508e-01],\n",
       "          [-1.3175e+00, -2.8772e-01, -1.6953e+00,  ..., -3.3263e-01,\n",
       "           -8.2175e-01,  1.7171e+00],\n",
       "          [-8.8114e-01, -1.9794e-01, -1.0974e+00,  ...,  1.4673e+00,\n",
       "            6.8920e-02,  1.0323e+00]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[ 4.2534e-01, -4.6308e-01, -9.8706e-02,  ...,  3.0366e-01,\n",
       "           -4.3416e-01,  8.4232e-01],\n",
       "          [ 7.6061e-01, -4.7492e-01,  3.6697e-01,  ...,  6.6607e-01,\n",
       "           -8.7833e-01,  5.4042e-01],\n",
       "          [ 8.9407e-01, -2.5731e-01, -2.0688e-02,  ...,  4.4893e-01,\n",
       "           -6.3414e-01,  1.2606e+00],\n",
       "          ...,\n",
       "          [ 7.5779e-01, -1.9296e-01,  3.6932e-01,  ...,  6.1792e-01,\n",
       "           -4.8362e-01,  1.1022e+00],\n",
       "          [ 6.8172e-02, -6.3310e-01, -6.7165e-02,  ...,  2.9424e-01,\n",
       "           -4.4943e-01,  1.2725e+00],\n",
       "          [ 8.6277e-01,  5.5685e-02,  1.9734e-01,  ...,  1.0141e+00,\n",
       "            5.2686e-02,  7.4258e-01]],\n",
       "\n",
       "         [[ 2.3379e+00,  3.1765e-01,  1.9218e-01,  ..., -2.0912e+00,\n",
       "            9.6007e-01, -2.5755e-01],\n",
       "          [ 3.1787e+00,  1.1593e+00,  5.8974e-01,  ..., -4.6689e+00,\n",
       "            1.1615e+00, -2.5838e-01],\n",
       "          [ 3.1228e+00,  9.3002e-01,  3.9349e-01,  ..., -4.0545e+00,\n",
       "            1.5674e+00,  9.1774e-02],\n",
       "          ...,\n",
       "          [ 2.8569e+00,  8.0205e-01,  4.4299e-01,  ..., -4.0135e+00,\n",
       "            1.6384e+00, -1.0205e-01],\n",
       "          [ 2.8386e+00,  1.5660e+00,  1.2966e+00,  ..., -4.1143e+00,\n",
       "            1.4596e+00, -1.2421e+00],\n",
       "          [ 2.5728e+00,  2.3802e-01,  2.3203e-02,  ..., -3.1519e+00,\n",
       "            1.6117e+00, -1.3738e-01]],\n",
       "\n",
       "         [[-1.7210e-01,  2.4302e-01, -9.5712e-01,  ..., -6.9195e-01,\n",
       "            4.6665e-01,  2.0374e-02],\n",
       "          [ 2.8738e-01, -1.0488e-01,  1.2425e-01,  ..., -1.2032e+00,\n",
       "            8.8951e-01, -1.3886e-01],\n",
       "          [ 1.8556e-01, -5.3653e-01, -9.8860e-01,  ..., -4.9613e-01,\n",
       "            5.7635e-01,  6.2495e-02],\n",
       "          ...,\n",
       "          [ 1.4466e-01, -1.6333e-01, -6.9551e-01,  ..., -4.9223e-01,\n",
       "            2.4501e-01,  7.7760e-03],\n",
       "          [ 1.3392e+00, -1.0288e-02,  2.3406e-01,  ..., -5.5815e-01,\n",
       "            5.3906e-01,  3.3512e-01],\n",
       "          [-3.1307e-01, -2.5849e-01, -1.2989e-01,  ..., -8.2089e-01,\n",
       "            4.5803e-01,  1.3301e-01]]]], device='cuda:0'), tensor([[[[-1.6424e-01, -1.0477e-01,  8.5032e-03,  ...,  8.7105e-03,\n",
       "           -6.6897e-02, -1.0200e-02],\n",
       "          [-5.7992e-01,  5.4037e-02, -1.1535e-01,  ..., -1.0931e-01,\n",
       "            8.1896e-01, -5.8565e-01],\n",
       "          [-1.9132e-01,  4.0676e-02,  4.8580e-01,  ..., -5.4541e-02,\n",
       "           -8.3277e-02,  3.5673e-01],\n",
       "          ...,\n",
       "          [-1.2033e-01, -4.0296e-02,  1.4799e-03,  ..., -3.3388e-02,\n",
       "           -1.5530e-01, -7.0780e-02],\n",
       "          [-2.0511e-01,  9.3126e-02,  7.7801e-02,  ..., -2.4031e-01,\n",
       "            6.2481e-02, -4.8098e-03],\n",
       "          [-6.2963e-02,  9.6018e-02, -5.1099e-03,  ...,  2.0165e-01,\n",
       "           -3.6895e-01, -1.1672e-01]],\n",
       "\n",
       "         [[-5.7404e-01,  9.5781e-02,  1.1555e-01,  ..., -1.8617e-01,\n",
       "            1.8476e-01, -1.1018e-01],\n",
       "          [-4.6191e-01, -4.7656e-02, -3.4643e-01,  ..., -2.0478e-01,\n",
       "           -4.0240e-01, -3.5355e-01],\n",
       "          [-7.5687e-01, -1.0166e-01, -2.5106e-01,  ...,  5.2498e-02,\n",
       "            3.3886e-01, -1.4988e-01],\n",
       "          ...,\n",
       "          [-3.7471e-01,  1.8691e-01, -1.0496e-01,  ...,  2.0967e-02,\n",
       "            4.1864e-01, -1.5591e-01],\n",
       "          [-4.2606e-01,  2.4490e-01, -2.6078e-01,  ...,  1.2062e-02,\n",
       "            7.8830e-02, -2.8669e-01],\n",
       "          [-1.7082e-01,  4.2581e-01, -4.1859e-01,  ...,  3.2414e-01,\n",
       "            2.2517e-01,  9.0652e-02]],\n",
       "\n",
       "         [[ 8.8461e-02,  3.5288e-01,  2.4923e-01,  ..., -4.7269e-01,\n",
       "           -6.9376e-02, -5.0862e-01],\n",
       "          [ 9.5580e-01,  1.2355e-01, -6.8514e-02,  ...,  7.8013e-02,\n",
       "           -5.2906e-01, -1.1463e+00],\n",
       "          [ 4.9946e-02, -5.5702e-01,  2.0475e-01,  ...,  7.1157e-02,\n",
       "            1.2124e-01,  2.2570e-01],\n",
       "          ...,\n",
       "          [ 2.8392e-01, -8.2554e-01,  1.6905e-01,  ..., -3.5985e-01,\n",
       "           -6.0036e-02, -5.7777e-01],\n",
       "          [ 4.6253e-01, -4.9404e-01, -1.0559e-01,  ..., -1.2547e-01,\n",
       "            3.0485e-01, -6.1806e-01],\n",
       "          [-1.1830e-02, -5.5361e-01, -1.1608e+00,  ...,  6.6731e-01,\n",
       "           -3.5124e-01, -4.2003e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[ 2.2565e-02,  2.2603e-01, -1.4563e-01,  ..., -9.2835e-01,\n",
       "           -4.6352e-01, -1.2466e-01],\n",
       "          [ 1.8500e-01,  4.0490e-01, -1.1068e-01,  ..., -1.0377e+00,\n",
       "            4.2851e-01, -2.8466e-01],\n",
       "          [ 1.3198e-01, -2.1008e-02, -2.5333e-02,  ..., -1.0628e+00,\n",
       "           -7.4744e-02, -1.1501e-01],\n",
       "          ...,\n",
       "          [ 1.2696e-01,  1.2544e-02, -2.2467e-02,  ..., -1.1486e+00,\n",
       "           -3.5611e-02, -3.8101e-02],\n",
       "          [-1.1888e-02,  3.0057e-01,  2.4725e-01,  ..., -1.1989e+00,\n",
       "           -1.1287e-01,  2.4556e-02],\n",
       "          [ 8.1091e-02, -1.7828e-02,  4.0837e-02,  ..., -1.5558e+00,\n",
       "           -2.6143e-02, -2.3121e-01]],\n",
       "\n",
       "         [[-2.8092e-01, -3.3220e-02,  5.7778e-02,  ..., -2.0559e-02,\n",
       "           -2.1366e-01, -1.6024e-01],\n",
       "          [-4.8697e-01,  3.0550e-01, -3.2256e-02,  ...,  1.3647e-01,\n",
       "            5.2383e-01, -3.7829e-01],\n",
       "          [-7.8907e-02, -4.2935e-02, -1.1175e-01,  ..., -6.5678e-02,\n",
       "           -9.4778e-02, -5.4903e-02],\n",
       "          ...,\n",
       "          [-4.6421e-02, -4.7720e-02, -1.3289e-01,  ..., -9.1533e-02,\n",
       "           -1.1136e-01, -6.8671e-02],\n",
       "          [-2.2296e-02,  3.2290e-01, -8.3383e-02,  ..., -4.2084e-01,\n",
       "           -4.2035e-01,  2.2440e-04],\n",
       "          [-5.9727e-02,  1.1212e-01,  4.5843e-01,  ..., -2.0580e-01,\n",
       "           -2.6470e-01, -9.1115e-02]],\n",
       "\n",
       "         [[ 1.9066e-01, -4.2320e-01,  1.5106e-01,  ..., -1.8992e-01,\n",
       "           -1.7584e-02, -8.6754e-02],\n",
       "          [-9.4662e-01, -1.8857e-01,  3.1178e-01,  ..., -2.5394e-01,\n",
       "            6.9686e-01, -4.0636e-01],\n",
       "          [-1.2118e-01, -2.3733e-01,  2.9617e-01,  ..., -1.3735e-02,\n",
       "            1.7544e-01, -1.1676e-01],\n",
       "          ...,\n",
       "          [-1.0241e-01, -1.8526e-01,  2.8457e-01,  ..., -3.8843e-02,\n",
       "            1.4548e-01, -5.4005e-02],\n",
       "          [-1.7351e-01,  1.5653e-01,  5.8282e-01,  ..., -6.6232e-01,\n",
       "            9.0271e-04, -4.0292e-01],\n",
       "          [-8.1873e-02, -1.8059e-01,  6.6787e-01,  ..., -1.0304e-01,\n",
       "           -1.5561e-01, -4.9592e-02]]]], device='cuda:0')), (tensor([[[-0.0018,  0.0179,  0.0248,  ..., -0.0100, -0.0017,  0.0130],\n",
       "         [ 0.0101,  0.0196,  0.0216,  ...,  0.0325,  0.0088,  0.0258],\n",
       "         [-0.0283,  0.0100, -0.0336,  ..., -0.0117,  0.0379,  0.0131],\n",
       "         ...,\n",
       "         [ 0.0137, -0.0120, -0.0325,  ..., -0.0123, -0.0250, -0.0097],\n",
       "         [ 0.0672,  0.0151,  0.0093,  ...,  0.0201, -0.0249, -0.0276],\n",
       "         [ 0.0295,  0.0149,  0.0193,  ..., -0.0535,  0.0573,  0.0623]]],\n",
       "       device='cuda:0'),), (tensor([[[[-1.5379e+01, -4.9857e+00,  1.6425e+01,  ..., -3.8692e+00,\n",
       "            8.7120e-01,  3.4729e+00],\n",
       "          [-1.6125e+01, -5.9857e+00,  1.7528e+01,  ..., -5.7031e+00,\n",
       "            1.6103e+00,  3.7699e+00],\n",
       "          [-1.5298e+01, -6.7175e+00,  1.7559e+01,  ..., -5.4372e+00,\n",
       "            1.4191e+00,  4.0791e+00],\n",
       "          ...,\n",
       "          [-1.4809e+01, -9.1951e+00,  1.8459e+01,  ..., -5.8652e+00,\n",
       "           -2.7671e-01,  6.6570e+00],\n",
       "          [-1.4816e+01, -9.8457e+00,  1.8210e+01,  ..., -6.0432e+00,\n",
       "           -3.2335e-01,  6.5057e+00],\n",
       "          [-1.4858e+01, -8.8641e+00,  1.8803e+01,  ..., -6.1482e+00,\n",
       "           -7.6743e-01,  8.0445e+00]],\n",
       "\n",
       "         [[-4.3501e-01, -3.8999e-01, -3.6070e-02,  ..., -2.1861e-02,\n",
       "           -9.3934e-01, -3.7987e-01],\n",
       "          [-3.5131e-01, -5.7931e-02,  1.3842e-01,  ..., -3.9683e-01,\n",
       "           -1.2368e+00, -6.7767e-02],\n",
       "          [-3.8376e-01, -1.2665e-01, -1.5842e-01,  ..., -9.8414e-02,\n",
       "           -1.0883e+00, -4.2642e-01],\n",
       "          ...,\n",
       "          [-2.9076e-01, -8.4747e-02, -1.2351e-01,  ..., -1.6304e-01,\n",
       "           -8.7720e-01, -2.1029e-01],\n",
       "          [-4.2909e-01,  4.8063e-01, -2.4251e-01,  ..., -5.5111e-01,\n",
       "           -9.2009e-01, -2.0863e-01],\n",
       "          [-6.4793e-02,  3.3021e-01, -1.9639e-01,  ..., -1.4480e-01,\n",
       "           -1.0446e+00,  2.3491e-02]],\n",
       "\n",
       "         [[ 1.7048e-01, -1.9818e-01,  1.7859e-01,  ...,  9.2665e-02,\n",
       "           -4.2431e-01,  1.7111e-01],\n",
       "          [ 1.7708e-01, -2.6468e-01,  5.4103e-02,  ...,  1.1226e-01,\n",
       "           -5.6038e-01,  1.8000e-01],\n",
       "          [ 2.5322e-01, -2.6132e-01,  4.3178e-02,  ...,  1.1028e-01,\n",
       "           -4.5988e-01,  4.8856e-02],\n",
       "          ...,\n",
       "          [ 1.2754e-01, -2.9898e-01,  5.0244e-02,  ...,  1.1776e-01,\n",
       "           -4.6927e-01,  1.2414e-01],\n",
       "          [ 1.3496e-01, -2.0252e-01,  6.5051e-02,  ...,  5.0747e-02,\n",
       "           -5.1411e-01,  2.3478e-01],\n",
       "          [ 1.8269e-01, -1.0486e-01,  1.6545e-01,  ...,  4.5167e-02,\n",
       "           -4.2646e-01,  1.3045e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[-5.6199e+00,  4.0715e+00,  2.1879e+00,  ...,  2.2803e+00,\n",
       "            3.0027e-03, -3.1937e+00],\n",
       "          [-6.1984e+00,  4.1590e+00,  2.2554e+00,  ...,  1.8099e+00,\n",
       "           -5.4188e-03, -3.1157e+00],\n",
       "          [-6.0774e+00,  4.0430e+00,  2.1589e+00,  ...,  1.5615e+00,\n",
       "            9.5879e-03, -2.9634e+00],\n",
       "          ...,\n",
       "          [-6.1781e+00,  3.9824e+00,  2.1130e+00,  ...,  1.4015e+00,\n",
       "           -1.6641e-02, -2.8082e+00],\n",
       "          [-5.9113e+00,  4.6343e+00,  2.9115e+00,  ...,  1.3500e+00,\n",
       "           -4.0647e-01, -3.1913e+00],\n",
       "          [-6.8768e+00,  4.1761e+00,  2.1443e+00,  ...,  1.6097e+00,\n",
       "           -2.1860e-01, -2.7857e+00]],\n",
       "\n",
       "         [[ 9.2586e+00, -3.1597e+00, -4.9621e+00,  ..., -7.0831e+00,\n",
       "            4.1828e+00,  1.6188e+00],\n",
       "          [ 9.1849e+00, -3.4455e+00, -4.7900e+00,  ..., -6.9490e+00,\n",
       "            4.0225e+00,  1.6347e+00],\n",
       "          [ 9.5219e+00, -3.2550e+00, -5.2132e+00,  ..., -6.5683e+00,\n",
       "            4.0868e+00,  1.6037e+00],\n",
       "          ...,\n",
       "          [ 1.0064e+01, -2.7464e+00, -4.9753e+00,  ..., -5.3912e+00,\n",
       "            4.2613e+00,  1.4306e+00],\n",
       "          [ 1.0433e+01, -2.6502e+00, -5.2231e+00,  ..., -5.6491e+00,\n",
       "            3.6375e+00,  1.0886e+00],\n",
       "          [ 1.0868e+01, -2.4545e+00, -5.4986e+00,  ..., -5.7537e+00,\n",
       "            4.2181e+00,  1.7296e+00]],\n",
       "\n",
       "         [[-1.7359e-01, -8.0909e-01,  2.4853e-01,  ...,  2.8366e+00,\n",
       "           -8.6573e-01, -7.7450e-01],\n",
       "          [ 4.8138e-01, -8.4873e-01,  7.5321e-01,  ...,  2.4534e+00,\n",
       "           -4.5350e-01, -9.2082e-01],\n",
       "          [-1.0517e-01, -9.6678e-01,  2.3920e-01,  ...,  2.4359e+00,\n",
       "           -6.9394e-01, -6.4563e-01],\n",
       "          ...,\n",
       "          [-3.8334e-01, -1.1360e+00, -5.3457e-02,  ...,  2.3387e+00,\n",
       "           -1.0940e+00, -8.5545e-01],\n",
       "          [ 1.9599e-01, -4.6298e-01,  5.3705e-01,  ...,  2.1044e+00,\n",
       "           -8.9752e-01, -1.1437e+00],\n",
       "          [ 2.3130e-01, -1.0356e+00,  5.2666e-01,  ...,  2.2458e+00,\n",
       "           -4.5117e-01, -1.1194e+00]]]], device='cuda:0'), tensor([[[[-2.4480e-02,  6.5264e-02,  7.0064e-02,  ...,  4.5403e-02,\n",
       "           -5.8553e-02,  9.7354e-03],\n",
       "          [-1.9236e-01,  2.4851e-01, -1.0805e-01,  ...,  3.5835e-01,\n",
       "            3.3786e-01,  8.9261e-01],\n",
       "          [ 7.5301e-02, -7.5948e-02, -6.8468e-02,  ..., -1.1226e-01,\n",
       "           -2.5371e-01, -4.0043e-02],\n",
       "          ...,\n",
       "          [-7.3806e-03,  1.6430e-01, -1.4736e-01,  ...,  5.7992e-02,\n",
       "           -5.5419e-02,  2.0224e-01],\n",
       "          [ 6.7687e-01,  4.2322e-01, -2.2775e-01,  ...,  1.7924e-01,\n",
       "            4.6269e-01,  1.9709e-02],\n",
       "          [-1.4721e+00,  3.9771e-01, -1.0052e-01,  ..., -1.7463e-01,\n",
       "           -4.7389e-02, -6.3627e-01]],\n",
       "\n",
       "         [[-1.0062e-01, -1.0802e-01, -9.0673e-02,  ...,  2.5056e-01,\n",
       "           -2.6105e-01, -3.4776e-01],\n",
       "          [ 2.0952e-01, -9.2080e-02,  2.4260e-01,  ...,  4.1610e-01,\n",
       "            2.9454e-01, -1.6161e-01],\n",
       "          [ 5.2592e-02, -1.1859e-01,  2.0282e-01,  ..., -2.3177e-01,\n",
       "           -2.2391e-01, -3.9979e-01],\n",
       "          ...,\n",
       "          [-2.6134e-01, -1.5615e-01, -8.3920e-02,  ...,  6.5984e-02,\n",
       "           -2.4046e-01, -6.2956e-01],\n",
       "          [ 4.8232e-01,  4.0230e-01,  1.9390e-02,  ...,  2.5495e-02,\n",
       "           -4.4311e-01,  1.5963e-01],\n",
       "          [ 7.4237e-02, -1.4066e-01, -7.2824e-01,  ...,  3.0418e-03,\n",
       "           -3.5786e-01,  7.6894e-02]],\n",
       "\n",
       "         [[-2.0455e-01, -3.1252e-02,  4.4353e+00,  ...,  2.0858e-01,\n",
       "           -6.7225e-04,  1.1755e-01],\n",
       "          [ 1.6837e-01, -1.8336e-01,  3.9354e+00,  ...,  6.9954e-01,\n",
       "            5.2651e-02, -1.4823e-01],\n",
       "          [ 1.9598e-01, -2.5079e-01,  3.5821e+00,  ...,  1.7641e-01,\n",
       "           -5.6702e-01, -2.3414e-01],\n",
       "          ...,\n",
       "          [-1.1909e-01, -1.5540e-02,  3.5592e+00,  ...,  2.5423e-01,\n",
       "           -8.6674e-01, -2.6393e-01],\n",
       "          [-1.1263e+00,  7.8819e-01,  3.8424e+00,  ...,  2.7988e-01,\n",
       "            3.0636e-01,  5.3609e-01],\n",
       "          [-5.3686e-03,  3.5644e-01,  3.9766e+00,  ...,  2.3440e-01,\n",
       "           -9.5346e-02, -6.9701e-02]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[ 3.4311e-01,  2.9446e-02, -4.6686e-01,  ...,  2.8393e-01,\n",
       "           -1.4607e-02,  2.8009e-01],\n",
       "          [-2.6734e-01,  1.6236e-01, -4.1703e-02,  ..., -4.7654e-01,\n",
       "            7.3852e-02,  5.4486e-01],\n",
       "          [ 2.5072e-02, -1.1880e-02,  8.4936e-02,  ...,  5.4005e-02,\n",
       "           -9.7116e-02,  2.8075e-02],\n",
       "          ...,\n",
       "          [ 1.2125e-01,  1.3438e-01, -1.1388e-01,  ...,  2.1311e-02,\n",
       "           -6.9858e-03,  6.6653e-02],\n",
       "          [-6.1714e-01, -5.9528e-01, -2.2743e-02,  ..., -3.7028e-01,\n",
       "           -9.6964e-01,  1.4416e-01],\n",
       "          [-6.9427e-01, -5.0998e-03, -5.5245e-02,  ..., -4.7648e-01,\n",
       "           -2.4420e-01,  1.2905e-02]],\n",
       "\n",
       "         [[-2.0039e-01,  2.2447e-01, -2.2289e-01,  ..., -4.3938e-02,\n",
       "            1.1465e-01,  1.5386e-02],\n",
       "          [ 2.1507e-01, -3.2809e-02,  3.4851e-02,  ...,  2.7515e-02,\n",
       "            3.5478e-01, -2.7487e-01],\n",
       "          [-2.2122e-02,  1.0979e-01,  7.1842e-02,  ..., -2.5639e-01,\n",
       "            2.3466e-02,  5.3190e-02],\n",
       "          ...,\n",
       "          [-1.6219e-01,  1.0223e-01,  8.5561e-02,  ...,  8.7810e-04,\n",
       "            1.9495e-02,  2.1659e-01],\n",
       "          [-2.1090e-01, -1.0839e+00, -5.1710e-01,  ...,  6.5671e-02,\n",
       "           -5.0413e-01, -2.0723e-01],\n",
       "          [-9.3533e-02, -6.1086e-01, -3.3991e-01,  ...,  1.1869e-02,\n",
       "           -2.0452e-01,  8.1457e-02]],\n",
       "\n",
       "         [[-3.7902e-02,  1.5490e-01, -4.0446e-02,  ..., -1.4351e-01,\n",
       "            1.9619e-01, -4.4310e-02],\n",
       "          [-4.8240e-01, -5.1755e-02, -5.4813e-02,  ...,  2.6421e-03,\n",
       "            4.9965e-03, -2.1244e-01],\n",
       "          [-9.8991e-03,  8.5366e-02, -1.7962e-01,  ...,  6.3954e-02,\n",
       "            9.7281e-02,  1.4292e-01],\n",
       "          ...,\n",
       "          [ 3.8254e-03,  2.0716e-01, -3.8970e-02,  ..., -3.9245e-02,\n",
       "            2.7920e-01, -1.4152e-02],\n",
       "          [-1.6364e-01,  1.5852e-01,  2.3818e-01,  ..., -8.2184e-01,\n",
       "            1.0894e-01, -2.1762e-01],\n",
       "          [-3.9912e-01,  8.7055e-02, -5.6529e-01,  ..., -4.6991e-01,\n",
       "            5.7569e-01,  4.8171e-01]]]], device='cuda:0')), (tensor([[[ 0.0010,  0.0182,  0.0484,  ..., -0.0096, -0.0249,  0.0377],\n",
       "         [ 0.0160, -0.0178,  0.0339,  ...,  0.0009,  0.0566,  0.0019],\n",
       "         [-0.0292, -0.0421, -0.0142,  ...,  0.0033,  0.0485,  0.0084],\n",
       "         ...,\n",
       "         [ 0.0006, -0.0200, -0.0061,  ...,  0.0019, -0.0094,  0.0093],\n",
       "         [ 0.0684,  0.0085,  0.0313,  ...,  0.0559, -0.0560, -0.0377],\n",
       "         [-0.0264, -0.0198,  0.0696,  ...,  0.0130,  0.0275,  0.0589]]],\n",
       "       device='cuda:0'),), (tensor([[[[-1.4464, -6.9313,  5.7620,  ...,  2.6086,  6.4921,  2.9518],\n",
       "          [-2.0082, -6.8863,  5.2136,  ...,  2.7577,  6.2812,  3.4438],\n",
       "          [-1.5954, -6.7530,  4.7862,  ...,  2.6419,  6.1445,  3.4360],\n",
       "          ...,\n",
       "          [-1.5562, -6.1768,  4.2715,  ...,  2.2253,  5.9302,  3.0892],\n",
       "          [-1.7717, -6.3630,  4.7777,  ...,  1.5447,  6.0699,  3.4136],\n",
       "          [-1.4951, -5.7212,  4.6933,  ...,  1.7595,  6.1193,  2.9560]],\n",
       "\n",
       "         [[ 4.3868, -1.9409, -0.7837,  ..., -0.4558,  1.5511, -7.3759],\n",
       "          [ 4.2356, -2.3057, -1.4254,  ..., -0.6070,  1.9636, -7.4726],\n",
       "          [ 3.7347, -2.0562, -1.6317,  ..., -0.6478,  2.0631, -7.1359],\n",
       "          ...,\n",
       "          [ 3.3834, -1.8800, -2.2788,  ..., -0.8848,  2.3340, -6.7912],\n",
       "          [ 3.3973, -1.5316, -1.8909,  ..., -1.0405,  1.6635, -6.6111],\n",
       "          [ 3.2491, -1.4991, -2.6122,  ..., -1.2365,  2.4943, -6.1459]],\n",
       "\n",
       "         [[ 2.1535,  3.3689, -3.3367,  ...,  4.0357, -2.1962,  2.7387],\n",
       "          [ 2.1004,  4.9866, -2.9924,  ...,  4.3702, -3.1884,  2.0954],\n",
       "          [ 2.2703,  4.6331, -2.6272,  ...,  4.4062, -2.7500,  2.1957],\n",
       "          ...,\n",
       "          [ 2.1084,  5.0292, -2.1826,  ...,  3.9547, -2.7142,  2.6480],\n",
       "          [ 2.1295,  4.9431, -2.4411,  ...,  4.2018, -3.0274,  2.8623],\n",
       "          [ 2.1402,  5.2578, -2.4982,  ...,  4.5405, -2.6428,  2.7426]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[ 2.3173,  0.2854, -3.2025,  ...,  3.2963,  1.9044, -5.3129],\n",
       "          [ 1.5387, -0.4764, -1.9041,  ...,  4.4445,  1.8948, -5.1511],\n",
       "          [ 1.4567, -0.2104, -2.5661,  ...,  4.1128,  2.3333, -4.4198],\n",
       "          ...,\n",
       "          [ 1.3285, -0.6894, -2.9696,  ...,  4.4202,  2.0572, -3.8965],\n",
       "          [ 1.3609, -1.2091, -3.3455,  ...,  4.4635,  2.3643, -3.2787],\n",
       "          [ 0.5923, -1.2502, -3.6974,  ...,  4.7872,  1.8063, -3.4169]],\n",
       "\n",
       "         [[-3.8217, -5.3960,  2.4895,  ...,  1.4962,  0.2507,  1.6748],\n",
       "          [-4.3528, -5.4673,  3.0976,  ...,  1.4963, -0.0642,  1.2709],\n",
       "          [-4.4019, -5.2840,  2.8588,  ...,  1.0474, -0.2872,  1.0552],\n",
       "          ...,\n",
       "          [-4.5089, -4.4783,  2.5969,  ...,  1.6758,  0.2021,  1.5981],\n",
       "          [-4.6770, -4.3556,  2.6978,  ...,  0.8457,  0.1711,  1.3726],\n",
       "          [-4.0669, -4.4298,  2.6106,  ...,  1.6419,  0.2184,  1.1465]],\n",
       "\n",
       "         [[ 3.3102, -6.8785,  4.1371,  ...,  1.0459, -1.3202,  4.2903],\n",
       "          [ 3.2260, -8.1804,  3.2330,  ...,  0.3342, -1.6673,  4.3714],\n",
       "          [ 3.3244, -8.1076,  3.7836,  ...,  0.1021, -2.3266,  4.4098],\n",
       "          ...,\n",
       "          [ 3.6118, -7.4811,  3.5379,  ...,  0.1428, -1.9336,  4.4219],\n",
       "          [ 3.2553, -7.7544,  3.4808,  ..., -0.0693, -1.8752,  5.0942],\n",
       "          [ 3.1959, -7.8780,  3.3211,  ..., -0.5073, -1.1010,  5.3891]]]],\n",
       "       device='cuda:0'), tensor([[[[ 0.0180, -0.0791,  0.0680,  ..., -0.1324, -0.1787,  0.0665],\n",
       "          [-0.0477,  0.2909, -0.0535,  ...,  0.1256, -0.3977,  0.3077],\n",
       "          [ 0.1115,  0.0718,  0.0757,  ..., -0.0032,  0.2988, -0.1337],\n",
       "          ...,\n",
       "          [ 0.1570,  0.0049,  0.0490,  ..., -0.1513, -0.1280, -0.0865],\n",
       "          [-0.0022,  0.1993, -0.2078,  ..., -0.3125,  0.0975,  0.1913],\n",
       "          [ 0.0153, -0.2037,  0.2866,  ...,  0.1024,  0.2244,  0.0086]],\n",
       "\n",
       "         [[-0.0785, -0.0832, -0.0393,  ..., -0.3545,  0.0045,  0.1667],\n",
       "          [-0.1982,  0.3053, -0.0510,  ..., -0.1972, -0.0360,  0.2900],\n",
       "          [-0.1928,  0.3192, -0.0586,  ...,  0.1022,  0.0256, -0.0507],\n",
       "          ...,\n",
       "          [-0.0808, -0.0684, -0.0346,  ..., -0.0229,  0.0908,  0.0075],\n",
       "          [-0.4640,  0.2440, -0.5582,  ..., -0.0872,  0.0806,  0.6575],\n",
       "          [-0.7389,  0.1555, -0.1747,  ...,  0.3401,  0.0822, -0.5905]],\n",
       "\n",
       "         [[-0.0952,  0.0246, -0.0682,  ..., -0.2064,  0.2500,  0.2097],\n",
       "          [-0.0593,  0.0873,  0.1220,  ...,  0.0081,  0.3921, -0.0576],\n",
       "          [-0.2367, -0.2395,  0.1005,  ...,  0.1090, -0.1308,  0.2212],\n",
       "          ...,\n",
       "          [-0.0307, -0.1602, -0.1869,  ..., -0.0304, -0.0543, -0.0464],\n",
       "          [ 0.2579,  0.2236,  0.0168,  ..., -0.2869,  0.5022,  0.1085],\n",
       "          [-0.7335, -0.1931, -0.1835,  ..., -0.2150, -0.1729,  0.2422]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[ 0.0822,  0.0883, -0.1892,  ..., -0.0803, -0.1196, -0.1247],\n",
       "          [-0.0019,  0.0925,  0.5349,  ..., -0.1048,  0.0430, -0.1769],\n",
       "          [ 0.0414,  0.1539,  0.1878,  ...,  0.4185,  0.0226,  0.7664],\n",
       "          ...,\n",
       "          [-0.0950, -0.0805,  0.0767,  ...,  0.0627,  0.0131,  0.8393],\n",
       "          [ 0.8355, -0.1785, -0.4202,  ...,  0.0708,  0.0117, -0.2371],\n",
       "          [-0.0935,  0.2239, -0.1560,  ..., -0.3002,  0.5801, -0.1853]],\n",
       "\n",
       "         [[-0.0974, -0.0111,  0.1771,  ...,  0.0236,  0.2023, -0.1102],\n",
       "          [ 0.2792,  0.1232,  0.2185,  ...,  0.2927, -0.2308,  0.1198],\n",
       "          [ 0.3259, -0.3685,  0.2571,  ...,  0.3364,  0.3374,  0.3068],\n",
       "          ...,\n",
       "          [ 0.3439, -0.5257,  0.1560,  ...,  0.0258,  0.0589,  0.0248],\n",
       "          [-0.4056, -0.2525,  0.0538,  ..., -0.0415,  0.3454, -0.1459],\n",
       "          [-0.0407,  0.0771,  0.4445,  ..., -0.2927, -0.1238, -0.2522]],\n",
       "\n",
       "         [[-0.0072,  0.1428, -0.0914,  ...,  0.0253, -0.0692,  0.0276],\n",
       "          [-0.0265, -0.0943, -0.1380,  ...,  0.3464, -0.1087,  0.4098],\n",
       "          [ 0.0268,  0.0057,  0.1857,  ...,  0.0034, -0.1817, -0.1866],\n",
       "          ...,\n",
       "          [-0.0617,  0.2155,  0.0509,  ...,  0.1937, -0.1533,  0.1381],\n",
       "          [-0.0153, -0.2136,  0.1170,  ..., -0.4246, -0.3333, -0.2014],\n",
       "          [-0.1753, -0.3797, -0.0557,  ..., -0.1464, -0.5318,  0.1656]]]],\n",
       "       device='cuda:0')), (tensor([[[ 3.3202e-02,  1.2717e-01,  3.0822e-02,  ..., -1.3018e-02,\n",
       "          -1.5016e-02,  2.2380e-02],\n",
       "         [ 1.8108e-02,  1.4861e-02,  1.5918e-02,  ...,  3.8849e-02,\n",
       "           6.4601e-02, -8.9165e-02],\n",
       "         [-6.2239e-02, -1.6075e-02, -4.7945e-02,  ...,  4.8619e-03,\n",
       "           6.3219e-02,  2.0639e-02],\n",
       "         ...,\n",
       "         [-1.2148e-02, -2.4159e-02,  3.0005e-03,  ...,  1.5810e-02,\n",
       "          -9.6206e-05, -1.1974e-02],\n",
       "         [ 4.2364e-02,  5.2858e-02, -2.3920e-02,  ...,  1.7971e-02,\n",
       "          -5.7916e-02, -3.2759e-02],\n",
       "         [-4.4267e-02, -2.1778e-02,  9.4944e-02,  ...,  1.4636e-02,\n",
       "           5.2593e-02,  5.2435e-02]]], device='cuda:0'),), (tensor([[[[ 2.1245e-02,  6.6854e-02,  4.0477e-03,  ...,  4.3762e-02,\n",
       "            1.7137e-01,  3.9124e-02],\n",
       "          [-5.1840e-01,  3.6467e-01,  3.2180e-01,  ...,  5.7570e-01,\n",
       "            1.1782e+00,  3.4271e-01],\n",
       "          [-8.7985e-01,  6.1949e-01, -1.6877e-01,  ...,  9.3638e-01,\n",
       "            2.8356e-01, -1.2038e-01],\n",
       "          ...,\n",
       "          [-8.2975e-01, -6.2245e-02, -6.2467e-01,  ...,  4.1222e-01,\n",
       "           -2.9419e-01,  6.8926e-01],\n",
       "          [-1.2371e-01, -5.3002e-01, -9.2437e-04,  ...,  5.9049e-01,\n",
       "           -5.9429e-01,  5.5371e-01],\n",
       "          [-3.8796e-01,  6.8069e-02,  8.6333e-02,  ...,  3.5184e-01,\n",
       "            6.1079e-01,  1.1608e-01]],\n",
       "\n",
       "         [[ 1.8555e-01, -1.1569e-01,  1.3442e-02,  ..., -7.9217e-03,\n",
       "           -9.2894e-02, -1.8134e-02],\n",
       "          [ 1.2113e-01,  2.6811e-01, -3.1072e-01,  ...,  6.4567e-02,\n",
       "            7.3284e-01,  3.5966e-01],\n",
       "          [-6.6304e-02, -1.3766e+00, -7.8123e-01,  ...,  7.0384e-01,\n",
       "            5.9627e-01, -4.1522e-02],\n",
       "          ...,\n",
       "          [-1.4305e-01, -8.2174e-01, -2.4817e-01,  ...,  6.0076e-01,\n",
       "            4.0984e-01,  3.3728e-02],\n",
       "          [ 2.6113e-01, -3.0392e-01,  5.5068e-02,  ...,  5.2385e-01,\n",
       "           -2.5830e-02,  5.5548e-01],\n",
       "          [ 6.4080e-01, -8.5569e-01, -6.3489e-01,  ..., -4.4292e-01,\n",
       "            1.5611e-01, -1.7702e-01]],\n",
       "\n",
       "         [[ 4.3614e-01, -3.0717e+00, -3.7609e-01,  ..., -1.0120e-01,\n",
       "            2.4463e-01, -2.3435e-01],\n",
       "          [ 8.3394e-01, -3.4042e+00, -3.4800e-01,  ..., -7.1425e-01,\n",
       "           -2.0946e-01,  2.7614e-01],\n",
       "          [ 4.2166e-01, -2.8489e+00, -5.7373e-01,  ..., -6.3021e-02,\n",
       "            3.6752e-01, -1.7130e-01],\n",
       "          ...,\n",
       "          [ 2.3724e-01, -2.6061e+00, -5.3108e-01,  ..., -8.2833e-03,\n",
       "            3.4661e-01, -4.4218e-01],\n",
       "          [ 3.9453e-01, -3.2233e+00, -3.5420e-01,  ..., -1.2814e-01,\n",
       "            1.7694e-01, -2.4447e-01],\n",
       "          [ 4.1669e-01, -3.1696e+00, -5.9939e-01,  ..., -4.2337e-02,\n",
       "            4.7765e-01, -1.3887e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[-1.2465e+00, -1.7467e+00, -7.7617e-01,  ...,  2.5981e+00,\n",
       "           -1.8762e-01,  1.6511e+00],\n",
       "          [-5.6366e-01, -1.7873e+00, -9.2502e-01,  ...,  2.2492e+00,\n",
       "           -2.6920e-01,  6.0171e-01],\n",
       "          [-1.3962e+00, -1.7783e+00, -6.0459e-01,  ...,  2.9599e+00,\n",
       "           -3.8224e-01,  1.0963e+00],\n",
       "          ...,\n",
       "          [-9.8888e-01, -1.7186e+00, -1.6792e-01,  ...,  3.0576e+00,\n",
       "           -2.8774e-01,  1.6662e+00],\n",
       "          [-1.0950e+00, -1.4654e+00, -3.3410e-01,  ...,  3.1002e+00,\n",
       "           -3.8538e-01,  1.4172e+00],\n",
       "          [-1.7189e+00, -1.7514e+00, -5.0978e-01,  ...,  2.8260e+00,\n",
       "           -2.7317e-01,  1.1435e+00]],\n",
       "\n",
       "         [[ 4.2619e-01,  1.4368e+00, -1.3391e+00,  ..., -1.9925e+00,\n",
       "           -1.7636e+00,  1.2326e+00],\n",
       "          [ 1.6973e-01,  8.0739e-01, -7.4913e-01,  ..., -2.7465e+00,\n",
       "           -1.2212e+00,  5.1510e-01],\n",
       "          [ 1.2102e-01,  9.9830e-01, -1.0357e+00,  ..., -2.0992e+00,\n",
       "           -1.5807e+00,  8.8355e-01],\n",
       "          ...,\n",
       "          [-9.0382e-02,  1.3419e+00, -1.3144e+00,  ..., -2.2657e+00,\n",
       "           -1.4336e+00,  1.1153e+00],\n",
       "          [-3.8411e-01,  1.3750e+00, -6.5379e-01,  ..., -2.3489e+00,\n",
       "           -1.3272e+00,  9.5272e-01],\n",
       "          [-2.8862e-01,  7.3897e-01, -1.6382e+00,  ..., -2.2834e+00,\n",
       "           -9.9276e-01,  7.1461e-01]],\n",
       "\n",
       "         [[ 6.6464e-02, -2.0699e-01, -6.8900e-02,  ..., -3.5194e-03,\n",
       "            6.7859e-02,  9.2806e-02],\n",
       "          [-2.8943e-03,  1.9107e-02,  9.3343e-01,  ...,  3.8547e-01,\n",
       "            7.8971e-01, -1.8745e-01],\n",
       "          [ 1.5736e-01,  1.2078e-02,  1.3496e+00,  ..., -2.1986e-01,\n",
       "            3.5577e-01, -8.8202e-02],\n",
       "          ...,\n",
       "          [-3.4443e-01, -5.4729e-01,  6.2018e-02,  ...,  2.7668e-02,\n",
       "           -3.6494e-01, -6.0495e-01],\n",
       "          [-6.9436e-01, -1.6102e-01, -2.0395e-01,  ...,  3.0462e-01,\n",
       "            3.6303e-01, -5.9849e-02],\n",
       "          [ 1.5010e-01, -2.0024e-01,  5.4574e-01,  ...,  2.9375e-03,\n",
       "            2.2180e-01, -6.0754e-02]]]], device='cuda:0'), tensor([[[[ 8.4076e-02,  2.4204e-01,  2.9747e-02,  ..., -2.4208e-02,\n",
       "            6.5167e-02,  9.4342e-03],\n",
       "          [ 3.2774e-01,  3.5955e-01,  6.6106e-01,  ...,  4.2227e-03,\n",
       "           -6.5504e-02,  3.6911e-01],\n",
       "          [-1.9394e-01, -1.6233e-01,  1.0467e+00,  ..., -1.5271e-01,\n",
       "           -1.1125e-01,  1.7089e-01],\n",
       "          ...,\n",
       "          [-3.3713e-01, -1.3673e-01,  3.2539e-02,  ...,  1.1400e-02,\n",
       "           -2.6266e-01, -1.9086e-01],\n",
       "          [ 2.1570e-01, -2.5306e-01,  3.7109e-01,  ...,  8.1261e-02,\n",
       "           -1.0154e-01, -2.9139e-01],\n",
       "          [-2.1340e-01, -7.4305e-01,  6.0321e-01,  ...,  9.6948e-02,\n",
       "            6.8159e-01, -2.6470e-01]],\n",
       "\n",
       "         [[ 9.5824e-02,  2.0116e-02,  4.2335e-03,  ...,  9.3527e-05,\n",
       "           -9.1461e-02, -2.1626e-02],\n",
       "          [ 2.2253e-02,  2.2826e-01, -1.4482e-01,  ..., -1.2622e+00,\n",
       "            5.1073e-01,  2.4239e-01],\n",
       "          [ 4.5390e-01,  2.4275e-01, -4.8538e-01,  ..., -1.1321e-01,\n",
       "            3.8996e-01, -8.1815e-01],\n",
       "          ...,\n",
       "          [ 1.5077e-01,  3.2511e-01, -3.1599e-01,  ...,  1.7936e-01,\n",
       "            2.5879e-01, -7.0305e-01],\n",
       "          [-5.7313e-01,  7.2995e-01,  8.5016e-02,  ..., -2.1039e-01,\n",
       "            2.8451e-01, -5.7122e-01],\n",
       "          [-5.7026e-01,  3.2866e-01, -7.0725e-02,  ..., -1.1977e+00,\n",
       "            4.6734e-02, -9.2009e-01]],\n",
       "\n",
       "         [[ 1.9275e-01,  2.5829e-02, -2.6279e-01,  ..., -4.0987e-01,\n",
       "            4.7813e-01,  1.1147e-01],\n",
       "          [ 4.1168e-01,  1.6777e+00, -2.1649e-01,  ..., -4.5905e-01,\n",
       "            1.7784e+00,  4.0413e-01],\n",
       "          [ 7.0690e-02,  7.1818e-01,  1.9246e-01,  ..., -6.1435e-02,\n",
       "            2.7139e-01,  5.4367e-01],\n",
       "          ...,\n",
       "          [-9.0044e-04,  3.1236e-01, -1.8502e-01,  ...,  2.2463e-01,\n",
       "            2.3746e-01, -1.4481e-01],\n",
       "          [ 7.1790e-02,  1.1034e+00, -1.3034e-01,  ...,  3.9186e-01,\n",
       "            1.6009e+00,  3.8643e-02],\n",
       "          [ 4.5241e-01,  7.1521e-01,  1.1446e+00,  ..., -1.3114e-01,\n",
       "            1.7352e+00,  3.5001e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[-1.0348e-01,  1.9300e-03, -5.1191e-03,  ...,  1.3941e-02,\n",
       "            5.4299e-02, -3.5010e-02],\n",
       "          [-9.5899e-01, -6.5005e-01,  2.1692e-01,  ..., -1.1189e-01,\n",
       "            4.3127e-01,  1.1434e-03],\n",
       "          [-7.0769e-01, -8.8733e-01,  1.8767e-01,  ..., -3.9215e-01,\n",
       "            3.3940e-01,  1.2224e-01],\n",
       "          ...,\n",
       "          [-1.8137e-01, -1.3884e-01, -1.5814e-01,  ...,  3.2023e-01,\n",
       "            7.2713e-02, -8.9957e-03],\n",
       "          [-8.1669e-01, -4.4941e-01,  1.8717e-01,  ..., -4.7526e-02,\n",
       "            3.4823e-01,  1.4331e-01],\n",
       "          [-1.0781e+00, -8.8235e-01, -8.4386e-02,  ..., -2.6449e-01,\n",
       "            3.4161e-02,  2.7565e-03]],\n",
       "\n",
       "         [[-9.9202e-02, -5.1084e-02, -2.4650e-02,  ..., -1.4567e-01,\n",
       "            8.9208e-02,  2.4645e-01],\n",
       "          [-1.2442e-01,  5.2830e-01, -8.9052e-01,  ..., -1.0303e+00,\n",
       "            4.0229e-01,  9.0639e-02],\n",
       "          [-6.6516e-01, -3.3472e-01, -2.2335e-01,  ..., -2.3862e-02,\n",
       "            1.9635e-01,  2.3876e-01],\n",
       "          ...,\n",
       "          [-1.0904e-01, -5.8656e-02, -2.3870e-01,  ..., -1.0126e-01,\n",
       "           -6.1289e-02,  3.0260e-01],\n",
       "          [-3.9854e-01,  8.3184e-02, -7.4304e-01,  ..., -2.7541e-01,\n",
       "            1.1290e-01, -3.2458e-01],\n",
       "          [-2.7298e-01,  8.2570e-02, -2.1188e+00,  ...,  6.8451e-01,\n",
       "            7.0552e-01,  2.9701e-01]],\n",
       "\n",
       "         [[-5.8261e-02,  4.5192e-02,  2.5332e-02,  ...,  3.5912e-03,\n",
       "            5.2413e-02,  3.8676e-02],\n",
       "          [ 8.2037e-01, -1.0306e+00, -1.6746e-01,  ..., -5.6719e-01,\n",
       "           -2.6918e-01,  4.6069e-01],\n",
       "          [ 8.1410e-02, -3.5033e-01, -1.1436e-01,  ..., -1.0012e+00,\n",
       "           -8.5087e-01, -2.3717e-01],\n",
       "          ...,\n",
       "          [ 2.2154e-05,  2.1677e-02, -1.6031e-02,  ...,  3.5208e-01,\n",
       "           -4.6171e-01, -1.1799e-01],\n",
       "          [ 2.0675e-01, -7.6029e-01,  2.1156e-01,  ..., -2.4089e-01,\n",
       "           -5.2691e-01,  4.5755e-02],\n",
       "          [ 3.0891e-01,  1.5471e-01, -2.0039e-01,  ..., -2.8716e-01,\n",
       "           -2.9062e-01, -2.7642e-02]]]], device='cuda:0')), (tensor([[[ 0.0117,  0.0225,  0.0007,  ..., -0.0099,  0.0065,  0.0236],\n",
       "         [ 0.0516,  0.0179,  0.0197,  ...,  0.0106,  0.0946, -0.0555],\n",
       "         [-0.0678, -0.0233, -0.0303,  ...,  0.0058,  0.0638,  0.0477],\n",
       "         ...,\n",
       "         [-0.0102, -0.0192,  0.0189,  ...,  0.0241, -0.0011, -0.0017],\n",
       "         [ 0.0259,  0.0659,  0.0524,  ...,  0.0194, -0.0235,  0.0400],\n",
       "         [-0.1186, -0.0450,  0.0699,  ...,  0.0093,  0.0289,  0.0568]]],\n",
       "       device='cuda:0'),), (tensor([[[[ 6.6912e-02,  1.8678e-02,  1.1983e-01,  ..., -1.0013e-01,\n",
       "           -6.4139e-01,  1.0147e-01],\n",
       "          [-4.9320e-02, -3.1370e-01,  2.5705e-01,  ...,  4.6800e-01,\n",
       "           -9.1009e-01,  4.8227e-01],\n",
       "          [-3.0239e-01, -1.0501e-01,  2.4345e-01,  ...,  4.5678e-01,\n",
       "           -8.7622e-01,  1.0732e-01],\n",
       "          ...,\n",
       "          [-1.5472e-01, -1.4611e-01,  1.6105e-01,  ...,  4.4358e-01,\n",
       "           -1.2613e+00,  1.8700e-01],\n",
       "          [-4.8088e-01, -4.4382e-02,  2.8202e-02,  ...,  2.8944e-02,\n",
       "           -1.0547e+00,  8.5394e-02],\n",
       "          [-6.9817e-02,  2.5668e-01,  2.9524e-01,  ...,  1.4021e-01,\n",
       "           -1.0121e+00, -4.4703e-01]],\n",
       "\n",
       "         [[ 1.2038e-02,  9.6941e-01, -2.4846e-01,  ...,  5.5256e-04,\n",
       "            1.4103e-01, -1.9617e-01],\n",
       "          [ 4.4481e-02,  3.7331e-01, -8.9199e-02,  ..., -9.9004e-01,\n",
       "           -5.4003e-01,  8.4918e-01],\n",
       "          [ 2.1118e-01,  8.0485e-01, -4.4721e-01,  ..., -7.1502e-01,\n",
       "           -1.7943e-01,  6.7990e-01],\n",
       "          ...,\n",
       "          [ 8.7860e-01,  5.3572e-01, -6.0862e-01,  ..., -2.2092e-02,\n",
       "            4.0687e-01,  7.2725e-01],\n",
       "          [ 7.2158e-01,  9.4048e-01, -3.6890e-01,  ..., -2.4406e-01,\n",
       "            1.9345e-01,  8.7870e-01],\n",
       "          [-1.5009e-01,  3.2003e-01, -6.6814e-01,  ..., -3.7387e-01,\n",
       "            2.5131e-01,  1.1808e+00]],\n",
       "\n",
       "         [[ 3.5102e-03,  3.4815e-02, -2.3275e-01,  ..., -1.6786e-02,\n",
       "           -2.7218e-01,  4.3955e-01],\n",
       "          [-1.8379e-01, -2.8552e-01, -5.0252e-01,  ..., -6.7251e-02,\n",
       "           -3.8808e-01, -9.4865e-01],\n",
       "          [-1.8053e-01, -7.6535e-03, -4.1512e-01,  ...,  4.9973e-02,\n",
       "            1.1606e-01, -1.0134e+00],\n",
       "          ...,\n",
       "          [ 2.1216e-01,  3.7599e-01, -5.3512e-01,  ...,  1.5257e-01,\n",
       "           -2.9907e-01, -5.6733e-01],\n",
       "          [-2.7383e-01, -3.3453e-01, -2.7389e-01,  ...,  1.1150e-01,\n",
       "           -2.1878e-01, -5.3126e-01],\n",
       "          [-2.9046e-01, -1.9608e-01, -9.2212e-02,  ..., -2.8833e-01,\n",
       "           -3.9130e-01, -5.0313e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[-8.1481e-01,  5.8699e-01,  2.5895e-01,  ...,  4.0250e-01,\n",
       "           -4.0785e-01, -3.4588e-01],\n",
       "          [-2.4992e+00,  3.7659e-01, -1.6371e-01,  ...,  7.1676e-02,\n",
       "           -7.8361e-02, -9.7794e-01],\n",
       "          [-1.8133e+00,  3.4295e-01, -2.0807e-01,  ...,  4.7453e-01,\n",
       "           -3.8963e-01, -1.0373e+00],\n",
       "          ...,\n",
       "          [-1.1749e+00,  2.1459e-01,  1.1766e-01,  ...,  3.6449e-01,\n",
       "           -3.6181e-01, -7.8866e-01],\n",
       "          [-1.9610e+00, -3.6631e-01, -2.6170e-01,  ...,  7.4128e-01,\n",
       "           -2.8952e-01, -1.3567e+00],\n",
       "          [-1.9389e+00, -9.3336e-03,  6.2780e-01,  ...,  6.1034e-02,\n",
       "           -4.7948e-01, -9.1084e-01]],\n",
       "\n",
       "         [[-5.4746e-02,  1.0031e-01, -1.0004e+00,  ...,  4.3766e-02,\n",
       "           -8.7479e-02,  4.4583e-02],\n",
       "          [-1.3463e-01,  5.5247e-01, -1.5678e+00,  ..., -3.9536e-01,\n",
       "            9.1137e-01,  1.3254e+00],\n",
       "          [-1.4705e-01,  1.9070e-01, -1.7596e+00,  ..., -4.6403e-02,\n",
       "            4.1111e-01,  8.7828e-01],\n",
       "          ...,\n",
       "          [ 3.0237e-02,  2.2013e-01, -1.3271e+00,  ..., -1.8548e-01,\n",
       "           -5.6203e-02,  5.3755e-01],\n",
       "          [-5.2645e-01,  3.8282e-01, -1.5772e+00,  ..., -3.5530e-01,\n",
       "            6.8436e-02,  4.3540e-01],\n",
       "          [-3.0408e-01,  7.6776e-01, -2.1230e+00,  ..., -4.8059e-02,\n",
       "           -2.2510e-01,  6.9366e-01]],\n",
       "\n",
       "         [[ 1.7917e-01, -2.3554e-01,  7.3636e-02,  ..., -6.2843e-02,\n",
       "            2.5099e-02,  2.2295e-01],\n",
       "          [ 3.5950e-01,  6.6560e-01, -2.1914e-01,  ..., -5.0328e-01,\n",
       "           -2.7185e-01,  1.0397e+00],\n",
       "          [ 3.9451e-01,  8.0570e-01,  5.9549e-02,  ...,  2.6083e-01,\n",
       "           -2.2350e-01,  8.1296e-01],\n",
       "          ...,\n",
       "          [-2.7963e-01,  7.1867e-01,  2.3586e-01,  ...,  3.1400e-01,\n",
       "           -1.1555e-01,  4.1611e-01],\n",
       "          [ 2.3006e-01,  1.2976e+00,  1.9992e-01,  ..., -4.6416e-01,\n",
       "           -5.4419e-01,  3.2521e-01],\n",
       "          [-1.3374e-01,  1.4189e+00, -1.1707e-01,  ..., -7.4213e-01,\n",
       "           -4.2050e-01, -3.6848e-02]]]], device='cuda:0'), tensor([[[[ 3.8640e-02,  4.1499e-02,  1.7774e-03,  ...,  3.6233e-02,\n",
       "           -3.8853e-03, -1.8698e-02],\n",
       "          [-4.6641e-01,  1.9700e+00, -1.6498e-01,  ...,  4.0639e-01,\n",
       "           -1.2879e-02,  5.7257e-01],\n",
       "          [-9.8676e-01,  7.9826e-01, -1.0159e+00,  ...,  3.3163e-01,\n",
       "            9.6259e-01, -2.2676e-01],\n",
       "          ...,\n",
       "          [ 4.5372e-02,  6.4206e-02, -6.2535e-01,  ...,  1.8555e-01,\n",
       "            1.8218e-01,  5.6103e-01],\n",
       "          [ 8.0914e-01,  5.9463e-01, -4.0072e-01,  ..., -1.3953e+00,\n",
       "           -1.3105e+00,  1.3823e-01],\n",
       "          [ 1.3052e-01,  1.2590e+00, -1.5742e-02,  ...,  7.0803e-01,\n",
       "            6.2727e-01,  8.7814e-02]],\n",
       "\n",
       "         [[ 1.8324e-02, -9.9568e-03, -1.7759e-02,  ..., -6.1373e-02,\n",
       "           -5.2801e-02, -5.3220e-02],\n",
       "          [ 9.4992e-02,  3.1663e-01, -6.4161e-01,  ...,  5.8802e-01,\n",
       "            8.0260e-01, -1.0098e+00],\n",
       "          [-3.7657e-01, -2.8342e-01,  1.2742e-01,  ..., -5.9438e-02,\n",
       "            2.2984e-01, -2.1542e-01],\n",
       "          ...,\n",
       "          [-1.5934e-01, -6.0541e-01,  9.8913e-02,  ...,  5.1086e-02,\n",
       "            4.2927e-01, -2.9398e-01],\n",
       "          [-9.0791e-01,  8.2021e-02,  1.5138e+00,  ...,  5.1346e-01,\n",
       "           -1.7961e+00, -2.6973e-01],\n",
       "          [-4.5642e-01,  1.3924e-02, -7.9589e-01,  ...,  5.1587e-01,\n",
       "            8.8427e-01, -9.2447e-01]],\n",
       "\n",
       "         [[-9.6934e-02, -3.7226e-02,  3.7373e-02,  ...,  6.5083e-02,\n",
       "           -2.4876e-02, -7.7265e-03],\n",
       "          [-1.3682e+00,  6.3283e-01,  5.6410e-01,  ..., -7.2499e-01,\n",
       "            2.3939e-01,  3.8996e-01],\n",
       "          [-4.2905e-01, -5.3603e-02, -2.8793e-01,  ..., -4.6943e-01,\n",
       "            2.4438e-01, -6.2943e-01],\n",
       "          ...,\n",
       "          [ 7.1488e-01,  3.5830e-01, -2.2285e-01,  ..., -3.4576e-01,\n",
       "            2.4367e-02, -5.6942e-01],\n",
       "          [-6.1372e-01, -5.4127e-01,  9.8599e-01,  ..., -5.8351e-02,\n",
       "           -1.8792e-02, -6.8105e-02],\n",
       "          [-5.1922e-01, -3.9865e-01, -1.5248e+00,  ..., -9.5950e-01,\n",
       "            1.3553e-01, -2.7247e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[-1.7828e-01,  1.6390e-01, -4.5900e-02,  ...,  1.0924e-01,\n",
       "           -9.2568e-02, -1.9293e-01],\n",
       "          [-2.9823e-01, -3.3128e-01, -6.3130e-01,  ...,  7.4001e-01,\n",
       "           -3.6490e-01,  5.5527e-01],\n",
       "          [-2.2251e-01, -2.3257e-01, -8.4422e-01,  ...,  2.2192e-01,\n",
       "           -3.3432e-01,  3.1562e-01],\n",
       "          ...,\n",
       "          [ 1.3536e-01,  9.7704e-02, -3.2880e-01,  ...,  2.9114e-01,\n",
       "           -8.4098e-02,  1.2170e-01],\n",
       "          [ 1.3831e-01,  1.4059e-01, -1.1275e-01,  ...,  2.5925e-01,\n",
       "           -4.1493e-01, -1.2172e+00],\n",
       "          [ 7.6437e-01, -2.5605e-01, -4.1359e-01,  ...,  6.6229e-01,\n",
       "           -1.5498e+00, -3.9660e-01]],\n",
       "\n",
       "         [[-5.5392e-02, -4.3086e-03, -9.1094e-03,  ...,  4.3051e-01,\n",
       "            8.6261e-02, -8.3499e-02],\n",
       "          [-5.2238e-01, -8.8725e-01,  8.5549e-01,  ...,  7.1615e-01,\n",
       "           -1.0289e+00, -3.3991e-01],\n",
       "          [-5.5622e-01, -4.3849e-01,  6.8555e-01,  ...,  1.8072e-01,\n",
       "           -8.7945e-01, -3.9787e-01],\n",
       "          ...,\n",
       "          [ 3.4339e-01, -5.6310e-02,  2.7026e-01,  ..., -4.4938e-02,\n",
       "           -3.5516e-01,  2.0646e-01],\n",
       "          [-1.4100e+00, -5.9548e-01, -2.5623e-01,  ...,  8.1230e-01,\n",
       "           -2.7566e-03, -1.6530e+00],\n",
       "          [-8.8145e-03,  1.9497e-01,  1.1846e+00,  ...,  9.3861e-01,\n",
       "           -3.6563e-01, -7.8422e-01]],\n",
       "\n",
       "         [[-1.4017e-02,  8.1241e-03, -3.5474e-02,  ...,  3.8430e-02,\n",
       "            7.0385e-02, -1.8458e-02],\n",
       "          [ 3.8140e-02,  2.9037e-01, -4.2346e-01,  ..., -3.5464e-01,\n",
       "           -1.9030e-02, -2.0585e-01],\n",
       "          [ 4.8256e-01,  1.1128e-01, -4.8169e-01,  ..., -5.6226e-02,\n",
       "           -9.9684e-02,  7.5518e-02],\n",
       "          ...,\n",
       "          [-4.1740e-01,  2.0259e-01,  3.0404e-02,  ...,  2.2888e-01,\n",
       "            1.2087e-01,  2.6661e-01],\n",
       "          [-1.9698e-01,  1.0093e+00, -4.9897e-01,  ..., -2.8917e-02,\n",
       "           -1.1685e+00,  2.3343e+00],\n",
       "          [ 7.3050e-01,  1.5055e+00, -1.5105e-01,  ...,  5.4897e-01,\n",
       "           -7.3964e-02,  3.9993e-01]]]], device='cuda:0')), (tensor([[[ 0.0194,  0.0311,  0.0045,  ...,  0.0068,  0.0168,  0.0149],\n",
       "         [ 0.1106,  0.0218,  0.1254,  ...,  0.0171,  0.1262, -0.0152],\n",
       "         [-0.0229,  0.0058, -0.0369,  ..., -0.0136,  0.0578, -0.0029],\n",
       "         ...,\n",
       "         [-0.0189, -0.0289, -0.0251,  ..., -0.0429,  0.0059,  0.0167],\n",
       "         [ 0.0953,  0.1120,  0.0615,  ...,  0.0084, -0.0237, -0.0436],\n",
       "         [-0.1237,  0.0323,  0.0108,  ...,  0.1071,  0.1484,  0.0624]]],\n",
       "       device='cuda:0'),), (tensor([[[[-9.9503e-02, -1.0941e-01, -1.2493e-01,  ...,  6.9461e-02,\n",
       "           -1.4443e+00,  5.8131e-02],\n",
       "          [ 2.5232e-01,  1.3268e+00, -2.6228e-01,  ..., -6.9209e-01,\n",
       "           -3.1619e-01,  1.0294e-01],\n",
       "          [-1.3871e-01,  8.9446e-01,  2.7005e-02,  ..., -9.1149e-01,\n",
       "           -3.2509e-01,  1.5318e-01],\n",
       "          ...,\n",
       "          [-4.8446e-01,  9.0121e-01,  4.8570e-01,  ..., -9.6656e-01,\n",
       "           -1.7058e-01,  1.1186e-01],\n",
       "          [ 2.2674e-01,  1.2390e+00, -4.4467e-01,  ..., -2.2927e-01,\n",
       "           -1.1737e+00,  3.1876e-01],\n",
       "          [ 1.3389e-01,  1.1995e+00, -1.3162e-01,  ..., -5.4417e-01,\n",
       "           -1.9222e-01,  5.4781e-02]],\n",
       "\n",
       "         [[-6.0269e-02,  1.7839e-01,  4.9633e-02,  ...,  9.6622e-02,\n",
       "           -7.2786e-02,  4.4308e-01],\n",
       "          [-6.1443e-01, -4.8270e-02,  5.5257e-01,  ..., -4.7801e-01,\n",
       "           -4.4177e-01, -3.9386e-01],\n",
       "          [-2.3251e-01, -1.7973e-01,  8.0518e-01,  ..., -3.5163e-01,\n",
       "           -1.8128e-01, -3.5660e-01],\n",
       "          ...,\n",
       "          [-1.2170e-01,  2.6446e-01,  4.4720e-01,  ..., -6.6663e-01,\n",
       "           -2.1567e-01, -2.0933e-01],\n",
       "          [-1.2622e-01,  1.7988e-01, -2.1241e-01,  ..., -8.4419e-01,\n",
       "           -1.3025e-01,  2.3183e-01],\n",
       "          [-5.2051e-01,  3.1944e-01,  7.3798e-01,  ..., -1.0008e+00,\n",
       "           -7.1963e-01, -5.1221e-01]],\n",
       "\n",
       "         [[ 3.6197e-01,  1.0285e-01,  8.8707e-02,  ..., -4.3507e-01,\n",
       "           -1.2941e-01, -1.9722e-01],\n",
       "          [ 2.2167e-01,  4.2366e-01,  3.4769e-01,  ...,  4.7998e-01,\n",
       "           -8.7950e-01, -5.7239e-01],\n",
       "          [ 2.6319e-01,  7.5568e-01,  3.4698e-01,  ...,  3.3529e-01,\n",
       "           -8.8796e-01, -5.9306e-01],\n",
       "          ...,\n",
       "          [ 4.0965e-01,  2.1049e-01,  3.0427e-01,  ...,  4.4440e-01,\n",
       "           -5.0714e-01,  2.4218e-02],\n",
       "          [ 3.4984e-01,  4.2540e-01,  1.5019e-01,  ...,  2.1668e-01,\n",
       "           -1.0948e+00,  1.7342e-03],\n",
       "          [ 4.4798e-01,  6.3181e-01,  8.8025e-02,  ..., -7.5657e-02,\n",
       "           -8.2679e-01, -4.5987e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[-1.9673e-02,  2.7804e-01, -2.0209e-01,  ..., -3.9083e-01,\n",
       "            2.7603e-02, -2.7144e-01],\n",
       "          [ 8.6998e-01,  3.4792e-02,  1.3503e-01,  ..., -2.7635e-01,\n",
       "            4.6269e-01,  1.4736e-01],\n",
       "          [ 5.8344e-01,  7.8152e-02, -2.1776e-01,  ..., -8.4728e-01,\n",
       "            3.3128e-01,  2.1783e-01],\n",
       "          ...,\n",
       "          [ 2.1910e-01,  4.3623e-01, -3.7988e-01,  ..., -5.9450e-01,\n",
       "           -1.0870e-01,  2.6582e-01],\n",
       "          [ 1.9035e-01,  3.9064e-01, -1.4868e-01,  ..., -3.1325e-02,\n",
       "           -2.2819e-01, -4.4303e-01],\n",
       "          [ 5.4088e-01,  6.2347e-02, -3.4166e-01,  ..., -5.9220e-01,\n",
       "           -1.6674e-01,  1.2307e-01]],\n",
       "\n",
       "         [[ 4.0982e-01,  7.4052e-02, -3.9260e-01,  ...,  8.3756e-02,\n",
       "            8.6598e-02,  2.6989e-01],\n",
       "          [ 5.0209e-01,  9.9314e-01,  1.3348e+00,  ..., -3.2925e-03,\n",
       "            1.8337e-01,  2.5361e-01],\n",
       "          [ 2.5107e-02,  9.3321e-01,  9.5178e-01,  ...,  1.3819e-01,\n",
       "            1.6887e-01,  4.8406e-01],\n",
       "          ...,\n",
       "          [-1.2163e-01,  5.8975e-01,  4.0128e-01,  ...,  1.0242e-01,\n",
       "            2.2571e-01,  4.0731e-01],\n",
       "          [ 1.3903e-01,  4.2460e-01,  1.6580e+00,  ...,  3.5101e-01,\n",
       "           -4.8291e-01, -4.2781e-02],\n",
       "          [ 4.6551e-01,  3.8210e-01,  8.4868e-01,  ..., -4.2305e-01,\n",
       "            4.6520e-02,  5.5188e-01]],\n",
       "\n",
       "         [[ 1.3172e-01, -7.2190e-02, -3.1498e-02,  ...,  2.2706e-01,\n",
       "           -2.0132e+00,  1.4472e-02],\n",
       "          [ 2.1295e-01, -1.1367e-01,  6.0877e-01,  ...,  5.4989e-02,\n",
       "            1.5464e+00, -1.5856e-01],\n",
       "          [ 3.3894e-01,  9.2392e-02,  8.2777e-01,  ..., -2.1799e-01,\n",
       "            1.2663e+00, -9.8409e-02],\n",
       "          ...,\n",
       "          [-3.3002e-01,  6.0096e-01,  5.2361e-01,  ..., -1.2007e-01,\n",
       "            9.3533e-01, -2.0856e-01],\n",
       "          [ 1.1491e-01, -1.4095e-01,  4.8496e-01,  ...,  1.7024e-01,\n",
       "            9.9615e-01, -1.2846e-01],\n",
       "          [-3.8801e-02,  1.0435e-01,  1.0465e+00,  ..., -1.1660e-02,\n",
       "            1.2719e+00,  1.0103e-01]]]], device='cuda:0'), tensor([[[[-3.9988e-02, -1.5192e-01, -1.9843e-01,  ...,  1.0713e-01,\n",
       "            1.0680e-01,  7.9271e-02],\n",
       "          [ 8.4547e-01, -8.0354e-01,  5.3797e-01,  ...,  2.1284e+00,\n",
       "            6.9641e-01,  7.6630e-01],\n",
       "          [ 1.1122e-01, -8.8837e-02, -2.4365e-01,  ...,  6.6594e-01,\n",
       "            4.3836e-01,  6.8941e-02],\n",
       "          ...,\n",
       "          [-3.2718e-01, -3.0774e-01,  2.5755e-02,  ..., -1.4676e-01,\n",
       "            8.7327e-02, -2.3000e-01],\n",
       "          [ 2.5445e+00,  2.2314e-01,  4.8335e-01,  ..., -1.8474e+00,\n",
       "            5.1843e-01,  2.5897e-01],\n",
       "          [-4.7556e-01, -2.8663e-01, -1.0394e+00,  ...,  6.5229e-03,\n",
       "            1.1653e+00,  1.7845e-01]],\n",
       "\n",
       "         [[-5.5104e-02, -6.3610e-02, -2.3272e-02,  ...,  2.2168e-02,\n",
       "            7.4063e-02,  2.3681e-02],\n",
       "          [ 7.9212e-01,  1.0141e+00,  1.1486e+00,  ..., -4.3966e-01,\n",
       "            1.1789e+00,  8.8923e-01],\n",
       "          [ 5.6390e-01,  1.0968e-01, -3.9272e-01,  ..., -5.9348e-01,\n",
       "            2.1916e-01,  4.7706e-01],\n",
       "          ...,\n",
       "          [ 3.0687e-01, -4.6856e-01, -3.7134e-01,  ..., -4.2570e-01,\n",
       "            2.1797e-01,  5.2081e-02],\n",
       "          [-7.3656e-01, -1.1488e-01, -1.6209e+00,  ...,  2.2181e-01,\n",
       "           -6.3537e-01,  3.0102e+00],\n",
       "          [-1.1063e+00, -5.7405e-01, -8.4438e-02,  ..., -3.3915e-01,\n",
       "            7.4157e-01,  1.1868e+00]],\n",
       "\n",
       "         [[-1.9223e-01,  5.2655e-01,  2.7766e-02,  ..., -2.5413e-01,\n",
       "            4.6034e-01,  1.6731e-01],\n",
       "          [-1.5470e-01,  2.3907e-01, -8.1784e-01,  ..., -6.2048e-01,\n",
       "            8.5878e-01,  6.3670e-01],\n",
       "          [ 8.1721e-01,  3.0962e-01, -7.6124e-01,  ..., -3.4324e-01,\n",
       "           -1.1044e+00,  8.7362e-02],\n",
       "          ...,\n",
       "          [ 1.9877e-01,  7.1181e-01, -3.7408e-01,  ..., -4.6885e-02,\n",
       "           -8.5857e-01, -7.5940e-01],\n",
       "          [-1.8875e+00, -6.3333e-01, -7.2320e-01,  ...,  7.6235e-02,\n",
       "           -3.2964e-01,  2.1201e+00],\n",
       "          [-8.5390e-02,  2.3423e-01,  5.3891e-03,  ...,  8.7868e-01,\n",
       "           -5.8641e-01, -2.6218e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[-1.8881e-02,  6.3002e-03,  4.7043e-02,  ..., -1.0803e-02,\n",
       "            1.2333e-01,  1.5647e-01],\n",
       "          [ 5.4204e-01, -1.2925e+00,  9.1246e-02,  ..., -5.6456e-01,\n",
       "           -8.1692e-01, -1.4630e-01],\n",
       "          [ 4.7071e-01, -3.2337e-01,  5.0571e-01,  ..., -5.2063e-01,\n",
       "           -9.4655e-02, -7.7723e-01],\n",
       "          ...,\n",
       "          [ 5.7353e-01,  2.5388e-01,  6.0767e-02,  ...,  6.1173e-01,\n",
       "           -7.4065e-02, -5.7095e-01],\n",
       "          [-2.1503e+00, -2.3381e-01,  1.0526e+00,  ..., -1.6587e+00,\n",
       "            8.9131e-01, -1.8252e+00],\n",
       "          [ 6.3448e-01, -6.5096e-01,  9.6012e-01,  ...,  4.1268e-01,\n",
       "            8.8941e-01, -4.9082e-01]],\n",
       "\n",
       "         [[ 1.5659e-01, -3.4999e-01, -3.7550e-01,  ..., -7.8419e-02,\n",
       "           -2.3998e-01, -1.6990e-01],\n",
       "          [ 2.8390e-01,  3.6365e-01,  7.6142e-02,  ..., -3.5094e-01,\n",
       "            6.3586e-01, -4.6756e-01],\n",
       "          [ 6.3373e-01,  8.4895e-01,  3.7593e-01,  ..., -1.8617e-01,\n",
       "            1.1869e+00,  4.5486e-02],\n",
       "          ...,\n",
       "          [ 2.1846e-01,  7.3363e-01,  1.5845e-01,  ..., -4.2689e-01,\n",
       "            1.4651e+00,  6.2296e-01],\n",
       "          [ 1.4220e+00,  1.0232e+00,  5.4447e-01,  ..., -4.3134e-02,\n",
       "           -1.1664e+00, -6.8485e-02],\n",
       "          [-7.2099e-01, -2.7404e-01, -1.9226e-01,  ...,  2.6615e-01,\n",
       "            5.3018e-01,  2.5738e-01]],\n",
       "\n",
       "         [[ 4.4379e-03, -2.0531e-01,  9.6022e-02,  ...,  3.5602e-02,\n",
       "           -4.6871e-02,  1.3290e-03],\n",
       "          [-1.3840e+00, -1.4063e+00,  1.7414e+00,  ...,  4.0924e-01,\n",
       "           -4.6569e-01, -1.0408e-01],\n",
       "          [ 2.1374e-01,  1.6508e-01,  9.9087e-01,  ...,  8.9546e-01,\n",
       "            3.1209e-01,  5.0820e-01],\n",
       "          ...,\n",
       "          [ 3.1736e-02,  1.8359e-01,  1.6010e-01,  ...,  1.6065e-01,\n",
       "           -2.9444e-01,  1.8630e-01],\n",
       "          [ 1.3867e+00, -1.6805e+00, -1.3660e+00,  ...,  1.9890e+00,\n",
       "            4.2527e-01,  6.7836e-02],\n",
       "          [ 1.5785e-01, -1.9585e-01, -1.6755e-01,  ...,  1.0886e+00,\n",
       "            5.9894e-02,  8.1503e-02]]]], device='cuda:0')), (tensor([[[-0.0267,  0.0071, -0.0113,  ..., -0.0189,  0.0248, -0.0020],\n",
       "         [ 0.1521, -0.0602,  0.0511,  ...,  0.0530,  0.0156,  0.0225],\n",
       "         [ 0.1351, -0.0691,  0.0067,  ...,  0.0173,  0.0375, -0.0073],\n",
       "         ...,\n",
       "         [ 0.1273, -0.0979, -0.0177,  ..., -0.0733,  0.0802, -0.0547],\n",
       "         [ 0.0709,  0.0021,  0.0843,  ..., -0.0344,  0.0983, -0.1258],\n",
       "         [-0.0227, -0.0392, -0.0330,  ...,  0.1579,  0.0430,  0.0536]]],\n",
       "       device='cuda:0'),)), hidden_states=(tensor([[[ 0.5396, -1.5234,  0.2064,  ...,  0.5176,  0.9336, -0.4326],\n",
       "         [ 0.2168, -0.4077,  0.1301,  ..., -0.0173,  0.4839, -0.2134],\n",
       "         [ 0.3650, -0.6060,  0.0972,  ...,  0.0474,  0.2548, -0.1841],\n",
       "         ...,\n",
       "         [ 0.3481, -0.5387,  0.2075,  ...,  0.0229,  0.2872, -0.3275],\n",
       "         [ 0.2274, -0.6158,  0.1001,  ...,  0.1313,  0.4292, -0.3456],\n",
       "         [ 0.4707, -0.8146,  0.3867,  ...,  0.1060,  0.4831, -0.2590]]],\n",
       "       device='cuda:0'), tensor([[[ 2.1671,  6.5915,  4.6633,  ...,  0.7975,  3.5714,  1.0726],\n",
       "         [ 2.6676,  5.4243,  3.4460,  ...,  3.1898,  3.5653,  1.6934],\n",
       "         [-0.3280,  4.8443, -1.7205,  ...,  0.5641,  5.5163,  0.8936],\n",
       "         ...,\n",
       "         [ 2.6794,  3.1122, -1.4317,  ...,  0.4800,  1.2238, -0.5464],\n",
       "         [ 6.3941,  4.5781,  2.0426,  ...,  2.1816,  1.1962, -1.6221],\n",
       "         [ 3.8091,  4.6427,  2.9401,  ..., -1.7746,  6.1267,  3.7297]]],\n",
       "       device='cuda:0'), tensor([[[ 6.4327, 12.6107,  7.0611,  ...,  2.0543, -0.4663,  5.3838],\n",
       "         [ 5.2611,  6.8011,  4.0905,  ...,  4.8521,  5.2878,  1.0928],\n",
       "         [ 1.5440,  6.9271, -0.3628,  ...,  3.5003,  6.4521,  1.7133],\n",
       "         ...,\n",
       "         [ 4.1481,  6.0130, -0.4304,  ...,  2.1381,  0.1845,  0.6374],\n",
       "         [ 8.8967,  8.1003,  4.7676,  ...,  8.0415,  1.0959, -0.2427],\n",
       "         [ 3.5905,  6.1343,  6.3515,  ...,  2.8706,  3.4722,  3.2673]]],\n",
       "       device='cuda:0'), tensor([[[ 8.7479, 13.0530,  7.1601,  ...,  3.5091,  1.0321,  6.0123],\n",
       "         [ 7.1524,  6.0789,  3.6485,  ...,  3.2737,  6.9207,  1.8868],\n",
       "         [ 3.5108,  4.4157, -0.5404,  ...,  3.4918,  6.4428,  2.3463],\n",
       "         ...,\n",
       "         [ 5.8112,  5.8511, -0.0249,  ...,  3.2118,  1.9942,  2.2161],\n",
       "         [11.6811,  8.0389,  3.2863,  ...,  6.8412, -1.5771, -0.9807],\n",
       "         [ 3.5544,  5.9089,  6.7603,  ...,  3.9449,  4.7422,  5.4951]]],\n",
       "       device='cuda:0'), tensor([[[12.3870, 22.0382,  6.2712,  ...,  7.4582,  0.2220,  6.8650],\n",
       "         [ 6.5555,  9.0063,  2.6356,  ...,  5.9402,  6.4323,  1.6827],\n",
       "         [ 3.4740,  8.6576,  0.2632,  ...,  2.3182,  5.8760,  3.9429],\n",
       "         ...,\n",
       "         [ 6.5486,  8.4889,  1.1864,  ...,  4.9439,  0.6813,  1.4995],\n",
       "         [12.1249, 10.3075,  1.3719,  ...,  9.8197, -1.7440, -0.4329],\n",
       "         [ 3.2115,  7.4644,  7.7458,  ...,  4.4191,  6.4407,  6.9991]]],\n",
       "       device='cuda:0'), tensor([[[14.0289, 34.4606,  7.3013,  ...,  8.8094,  0.3596,  7.5945],\n",
       "         [ 6.4207,  9.3664,  3.0374,  ...,  8.5157,  6.1482, -2.8993],\n",
       "         [ 0.2736,  7.2274, -1.9864,  ...,  6.3751,  6.2304,  4.4032],\n",
       "         ...,\n",
       "         [ 4.0727,  6.5232,  1.9248,  ...,  7.0610,  1.6606,  2.0489],\n",
       "         [ 8.6951, 12.9741, -0.4920,  ...,  7.3756, -2.7142,  0.4931],\n",
       "         [ 1.3453,  6.7334,  9.6397,  ...,  7.0091,  5.3677,  6.4332]]],\n",
       "       device='cuda:0'), tensor([[[14.0888, 45.5000, 21.4961,  ..., 19.9406, 17.5324, 18.4835],\n",
       "         [ 6.0196,  8.4225,  6.6700,  ...,  7.5985,  8.9735, -2.7372],\n",
       "         [-2.4455,  6.2836,  1.5956,  ...,  4.5266,  7.6018,  5.6986],\n",
       "         ...,\n",
       "         [ 1.2514,  6.9569,  3.3467,  ...,  5.6753,  5.0969,  1.8822],\n",
       "         [ 6.4588, 11.3947,  5.4744,  ...,  8.8488, -1.2376,  5.1721],\n",
       "         [-0.3598, -1.7840, 11.4283,  ...,  4.7839,  6.2650,  8.3886]]],\n",
       "       device='cuda:0'), tensor([[[13.8645, 46.0734, 22.4950,  ..., 20.3990, 16.6036, 18.8319],\n",
       "         [ 7.5107,  8.6850,  7.0655,  ...,  6.6837, 11.3517, -1.1347],\n",
       "         [-2.5823,  5.3060,  2.8320,  ...,  6.3938,  9.0072,  6.7351],\n",
       "         ...,\n",
       "         [ 2.0754,  5.3679,  6.7799,  ...,  7.5056,  3.7549,  2.8106],\n",
       "         [ 5.1830, 13.0020, 10.0368,  ...,  7.3590,  1.8857,  5.9954],\n",
       "         [-8.0003,  3.0216, 11.4780,  ...,  6.3852,  6.0233,  7.1745]]],\n",
       "       device='cuda:0'), tensor([[[12.3852, 41.8946, 23.5551,  ..., 18.1826, 17.2671, 20.6961],\n",
       "         [ 9.2745,  8.5529, 11.0395,  ...,  7.3363, 14.8269,  0.8702],\n",
       "         [ 0.6516,  5.3902,  4.7858,  ...,  3.8904,  7.6492,  7.0601],\n",
       "         ...,\n",
       "         [ 0.8676,  4.1978,  1.6056,  ...,  0.9392,  4.0476,  8.0731],\n",
       "         [ 9.5971, 13.2630, 12.6970,  ...,  2.1828,  7.9414, 10.3481],\n",
       "         [-6.2657,  1.1827, 10.2295,  ...,  9.1476, 15.2805, 11.0183]]],\n",
       "       device='cuda:0'), tensor([[[ 11.6553,  42.2936,  18.7467,  ...,  13.3546,  16.7597,  19.7606],\n",
       "         [ 12.8290,   8.2423,  17.1560,  ...,   5.5814,  15.7834,   2.5463],\n",
       "         [  0.4898,   6.9478,   1.5838,  ...,   2.6995,   9.4287,   4.0313],\n",
       "         ...,\n",
       "         [  0.5294,   3.1004,   2.4088,  ...,  -0.6527,   3.9194,   5.6671],\n",
       "         [ 11.8052,  17.9829,  11.3276,  ...,   4.6886,   0.8695,  -0.6170],\n",
       "         [-10.6630,   8.9368,   5.4568,  ...,  14.7065,  18.1493,  10.0275]]],\n",
       "       device='cuda:0'), tensor([[[13.9353, 41.9025, 19.8025,  ..., 12.2086, 13.3643, 20.7499],\n",
       "         [15.1510,  8.0495, 13.3704,  ...,  7.0392, 10.1510,  8.8078],\n",
       "         [ 8.3588,  3.4794,  3.6181,  ...,  3.0068,  7.7263, 10.8579],\n",
       "         ...,\n",
       "         [ 4.9508, -1.2802,  1.3425,  ..., -1.7933,  4.2339,  3.5765],\n",
       "         [14.7574, 10.9489, 11.5594,  ..., 13.2082,  7.7419,  0.6783],\n",
       "         [-2.9347,  0.5357,  3.6029,  ..., 10.7352,  8.2632, 13.1301]]],\n",
       "       device='cuda:0'), tensor([[[-1.7715e-01,  9.7574e+00,  1.0555e+01,  ...,  8.4558e+00,\n",
       "           1.3449e+01,  9.5487e+00],\n",
       "         [ 2.4070e+01, -1.5305e+00,  1.4880e+01,  ...,  1.5699e+01,\n",
       "           7.2382e+00,  9.4403e+00],\n",
       "         [ 2.3200e+01, -2.0147e+00,  1.0175e+01,  ...,  1.1869e+01,\n",
       "           1.0631e+01,  6.7759e+00],\n",
       "         ...,\n",
       "         [ 2.1340e+01, -6.7829e+00,  5.9921e+00,  ..., -1.6096e+00,\n",
       "           1.4673e+01,  2.8739e-02],\n",
       "         [ 1.4853e+01,  6.0461e+00,  2.0083e+01,  ...,  3.8670e+00,\n",
       "           1.7500e+01, -8.9564e+00],\n",
       "         [ 2.9108e-01, -1.6990e+00,  1.8403e+00,  ...,  2.9479e+01,\n",
       "           8.4537e+00,  1.1393e+01]]], device='cuda:0'), tensor([[[ 8.7518e-01, -2.9193e-01, -2.0787e+00,  ...,  9.5691e-01,\n",
       "           1.2586e+00, -1.1766e+00],\n",
       "         [ 2.8245e-01,  7.7959e-04, -4.8437e-01,  ...,  1.8132e+00,\n",
       "          -6.6867e-02, -2.0718e-01],\n",
       "         [ 3.7990e-01,  4.1733e-01, -4.6942e-01,  ...,  6.8315e-01,\n",
       "          -8.2745e-01,  3.0548e-01],\n",
       "         ...,\n",
       "         [-1.6876e-01, -9.4678e-02, -2.7121e-01,  ...,  9.6991e-01,\n",
       "          -5.4538e-01, -7.8037e-01],\n",
       "         [ 3.9214e-01, -4.5037e-01, -2.8424e-01,  ...,  7.0173e-01,\n",
       "           7.8281e-01, -2.2244e-01],\n",
       "         [-1.1355e-01,  9.1667e-01, -1.4762e+00,  ...,  1.8230e+00,\n",
       "           2.9018e-01,  7.3630e-01]]], device='cuda:0')), attentions=(tensor([[[[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.9433e-01, 4.0567e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.6221e-01, 6.5035e-02, 5.7276e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [6.3374e-02, 1.2088e-02, 1.2817e-01,  ..., 4.6295e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.0678e-01, 6.3396e-03, 9.8437e-02,  ..., 3.1356e-01,\n",
       "           3.0954e-01, 0.0000e+00],\n",
       "          [8.8000e-02, 3.3642e-02, 7.8870e-02,  ..., 1.8150e-01,\n",
       "           5.5232e-02, 2.8930e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.4466e-01, 1.5534e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.2721e-01, 1.4188e-01, 3.3091e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.0108e-02, 9.6906e-04, 4.4864e-01,  ..., 5.1113e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.0014e-01, 5.0204e-03, 3.6707e-03,  ..., 2.2797e-03,\n",
       "           6.5507e-02, 0.0000e+00],\n",
       "          [4.6103e-01, 2.4745e-02, 1.6216e-03,  ..., 1.5707e-03,\n",
       "           5.4887e-02, 1.0902e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.7251e-14, 1.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.0400e-10, 6.0289e-04, 9.9940e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [6.7761e-07, 1.9693e-06, 8.1165e-04,  ..., 9.9856e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.3468e-09, 1.3542e-06, 7.4653e-07,  ..., 2.4815e-07,\n",
       "           9.9995e-01, 0.0000e+00],\n",
       "          [1.2108e-06, 1.2348e-07, 1.2196e-06,  ..., 3.4481e-05,\n",
       "           1.3145e-06, 9.9988e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [4.9608e-02, 9.5039e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.8893e-01, 6.7015e-03, 8.0437e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [2.0253e-02, 3.0622e-03, 1.6889e-01,  ..., 4.2259e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.4340e-03, 4.8114e-03, 1.7167e-01,  ..., 2.5169e-01,\n",
       "           1.6936e-01, 0.0000e+00],\n",
       "          [5.6580e-02, 7.9343e-04, 1.4558e-01,  ..., 3.6116e-01,\n",
       "           9.7167e-03, 1.1134e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.9659e-01, 1.0341e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [2.8572e-02, 2.6242e-03, 9.6880e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.6626e-03, 8.2992e-04, 1.0719e-01,  ..., 2.2734e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.0277e-02, 1.7263e-04, 1.0777e-02,  ..., 1.4432e-02,\n",
       "           9.4700e-01, 0.0000e+00],\n",
       "          [1.9025e-02, 2.1527e-03, 1.8574e-01,  ..., 2.2925e-01,\n",
       "           1.0926e-03, 1.7767e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [7.3116e-01, 2.6884e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.6815e-02, 1.6632e-04, 9.8302e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [3.1617e-03, 2.1977e-04, 2.5928e-01,  ..., 6.3532e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.0404e-06, 1.2217e-07, 6.6833e-06,  ..., 1.0011e-05,\n",
       "           9.9996e-01, 0.0000e+00],\n",
       "          [8.3391e-03, 3.4520e-06, 5.9995e-01,  ..., 3.1342e-01,\n",
       "           1.7001e-04, 2.2553e-02]]]], device='cuda:0'), tensor([[[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0637, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0576, 0.0580, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0569, 0.0550, 0.0454]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.4053, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0913, 0.0806, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0937, 0.0805, 0.1025]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0422, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0289, 0.0883, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0390, 0.1134, 0.0665]],\n",
       "\n",
       "          ...,\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2038, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2107, 0.0046, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2145, 0.0130, 0.0366]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0503, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1016, 0.0412, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1677, 0.0384, 0.0766]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1750, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1511, 0.2042, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1599, 0.0083, 0.1119]]]]],\n",
       "       device='cuda:0'), tensor([[[[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.1477e-01, 8.5227e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.1242e-01, 4.0537e-02, 6.4705e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.1252e-03, 1.3450e-04, 1.0127e-03,  ..., 6.4182e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.8138e-05, 2.0565e-06, 3.4201e-04,  ..., 6.6598e-01,\n",
       "           2.8476e-01, 0.0000e+00],\n",
       "          [1.1487e-04, 7.5121e-07, 1.4535e-04,  ..., 4.4421e-01,\n",
       "           4.2135e-01, 6.9489e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.6880e-01, 3.1205e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.3322e-01, 3.2244e-02, 4.3453e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [3.8936e-01, 1.7834e-02, 1.1623e-01,  ..., 2.2754e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.6177e-01, 4.5808e-02, 1.2133e-01,  ..., 1.1855e-01,\n",
       "           1.2872e-02, 0.0000e+00],\n",
       "          [2.4225e-01, 3.7209e-02, 7.9135e-02,  ..., 9.5379e-02,\n",
       "           3.2952e-02, 2.3019e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.8877e-01, 1.1123e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.0019e-01, 1.0595e-01, 9.3855e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [3.8979e-01, 5.1661e-02, 4.8939e-02,  ..., 7.8177e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.7972e-01, 4.9039e-02, 4.2341e-02,  ..., 7.1738e-02,\n",
       "           5.6071e-02, 0.0000e+00],\n",
       "          [3.5455e-01, 4.4101e-02, 4.3535e-02,  ..., 7.1547e-02,\n",
       "           5.7714e-02, 4.2420e-02]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.2104e-01, 7.8956e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.4993e-01, 2.3148e-03, 4.4776e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [3.4149e-01, 9.2462e-04, 6.3472e-02,  ..., 4.0321e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.3866e-01, 1.2780e-03, 1.2083e-01,  ..., 3.7874e-01,\n",
       "           3.0530e-02, 0.0000e+00],\n",
       "          [5.0081e-02, 4.1344e-03, 8.8130e-02,  ..., 2.1397e-01,\n",
       "           2.1289e-02, 1.4722e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.3011e-01, 6.9891e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.3070e-01, 3.4056e-02, 1.3524e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.8804e-01, 1.2038e-02, 1.3011e-02,  ..., 9.8061e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.1289e-01, 3.9873e-03, 1.8250e-02,  ..., 3.7782e-01,\n",
       "           3.0154e-01, 0.0000e+00],\n",
       "          [2.6774e-02, 9.6691e-04, 1.9921e-03,  ..., 3.8269e-02,\n",
       "           8.3740e-01, 2.0365e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.7040e-01, 2.9600e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.4940e-01, 4.8406e-02, 1.0219e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [4.0335e-01, 4.2623e-02, 3.4647e-02,  ..., 5.2481e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.4184e-01, 2.4957e-02, 9.4577e-02,  ..., 1.2171e-01,\n",
       "           7.3293e-02, 0.0000e+00],\n",
       "          [4.4363e-01, 2.2049e-02, 5.7328e-02,  ..., 9.8247e-02,\n",
       "           4.6135e-02, 3.6170e-02]]]], device='cuda:0'), tensor([[[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.3200, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.4507, 0.0154, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.3901, 0.0583, 0.1664]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.3872, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2482, 0.4267, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2007, 0.1078, 0.5363]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.6134, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.7999, 0.1280, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.3218, 0.3597, 0.2842]],\n",
       "\n",
       "          ...,\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.9491, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.7622, 0.1981, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.6932, 0.0943, 0.1910]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.4369, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.3547, 0.1581, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.4933, 0.0435, 0.0449]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.7545, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.7286, 0.2523, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1019, 0.8064, 0.0645]]]]],\n",
       "       device='cuda:0'), tensor([[[[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [6.5184e-01, 3.4816e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.6519e-01, 5.0853e-01, 1.2628e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.3098e-01, 8.2093e-02, 2.5183e-02,  ..., 7.3346e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.1596e-01, 3.2186e-02, 6.7870e-02,  ..., 4.2684e-01,\n",
       "           1.0508e-01, 0.0000e+00],\n",
       "          [1.2596e-02, 6.9367e-03, 9.9311e-03,  ..., 5.8975e-02,\n",
       "           1.7296e-01, 6.6740e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.3277e-01, 1.6723e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.2125e-01, 1.1913e-01, 3.5962e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.7813e-02, 1.1373e-03, 9.3607e-03,  ..., 8.2869e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.2240e-02, 2.5231e-03, 9.9832e-03,  ..., 8.3603e-01,\n",
       "           6.4814e-02, 0.0000e+00],\n",
       "          [3.1664e-03, 1.8779e-03, 1.2064e-02,  ..., 4.5320e-01,\n",
       "           2.5431e-01, 1.6379e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [7.9426e-01, 2.0574e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.6717e-01, 2.8607e-01, 3.4676e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.0378e-02, 3.9542e-03, 9.6196e-03,  ..., 6.2063e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [4.8632e-03, 2.6346e-03, 4.2243e-03,  ..., 1.4524e-01,\n",
       "           7.4055e-01, 0.0000e+00],\n",
       "          [1.5332e-03, 1.7550e-04, 2.4239e-03,  ..., 1.4641e-01,\n",
       "           7.1241e-02, 6.8575e-01]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9680e-01, 3.2030e-03, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.3812e-02, 9.8541e-01, 7.7895e-04,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [3.6507e-03, 5.3835e-04, 2.8881e-04,  ..., 2.6448e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [1.5424e-05, 1.0478e-06, 2.1550e-06,  ..., 9.9456e-01,\n",
       "           4.4349e-04, 0.0000e+00],\n",
       "          [7.9781e-04, 7.4549e-06, 2.5080e-04,  ..., 6.0525e-01,\n",
       "           2.5360e-01, 8.4637e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.9202e-01, 4.0798e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.7989e-01, 1.7232e-01, 2.4779e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [6.2252e-02, 3.1522e-02, 3.5347e-02,  ..., 3.3386e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [2.2390e-02, 3.7974e-03, 1.4629e-02,  ..., 4.3263e-01,\n",
       "           3.4685e-01, 0.0000e+00],\n",
       "          [1.2548e-02, 1.1111e-03, 6.0797e-03,  ..., 4.6476e-01,\n",
       "           1.1177e-01, 3.2716e-01]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.7308e-01, 2.6924e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [7.6751e-01, 1.9056e-01, 4.1932e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.0668e-01, 1.9004e-02, 9.9518e-03,  ..., 1.8365e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [4.4577e-02, 3.0153e-03, 3.2701e-03,  ..., 6.5169e-02,\n",
       "           7.7326e-01, 0.0000e+00],\n",
       "          [8.1693e-03, 1.0947e-03, 2.4020e-03,  ..., 2.3344e-02,\n",
       "           2.8898e-02, 8.7956e-01]]]], device='cuda:0'), tensor([[[[[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           ...,\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 6.2302e-02,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 9.9293e-01,\n",
       "            1.3664e-04, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 1.9342e-02,\n",
       "            9.7169e-01, 8.8753e-03]],\n",
       "\n",
       "          [[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           ...,\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 3.9325e-01,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 7.7897e-01,\n",
       "            1.2329e-01, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 2.1260e-01,\n",
       "            4.6989e-01, 2.5701e-01]],\n",
       "\n",
       "          [[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           ...,\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 9.3218e-02,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 1.4272e-01,\n",
       "            3.3192e-02, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 1.0822e-01,\n",
       "            2.9701e-02, 6.2916e-02]],\n",
       "\n",
       "          ...,\n",
       "\n",
       "          [[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           ...,\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 3.2045e-01,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 8.6991e-01,\n",
       "            5.9262e-02, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 5.0367e-01,\n",
       "            2.3200e-01, 1.1044e-01]],\n",
       "\n",
       "          [[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           ...,\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 3.8765e-01,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 1.1021e-01,\n",
       "            8.0758e-01, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 1.0987e-01,\n",
       "            2.1250e-01, 5.7878e-01]],\n",
       "\n",
       "          [[0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           ...,\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 2.9223e-01,\n",
       "            0.0000e+00, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 4.6289e-01,\n",
       "            6.3131e-02, 0.0000e+00],\n",
       "           [0.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 5.5686e-01,\n",
       "            1.7554e-01, 7.4762e-02]]]]], device='cuda:0'), tensor([[[[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8210e-01, 1.7904e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.7975e-01, 4.5816e-04, 1.9792e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.7763e-01, 9.3546e-04, 9.7001e-04,  ..., 6.4377e-03,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8252e-01, 5.1733e-03, 1.2454e-03,  ..., 6.8883e-04,\n",
       "           1.6404e-03, 0.0000e+00],\n",
       "          [8.8477e-01, 6.7224e-03, 3.6538e-03,  ..., 2.4474e-04,\n",
       "           1.8365e-03, 3.5263e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9457e-01, 5.4254e-03, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9153e-01, 6.1952e-04, 7.8523e-03,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.8740e-01, 3.3210e-06, 5.6479e-06,  ..., 8.5580e-03,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9977e-01, 2.0857e-08, 1.6643e-07,  ..., 7.5456e-06,\n",
       "           1.7713e-04, 0.0000e+00],\n",
       "          [9.9851e-01, 3.5545e-06, 1.8185e-05,  ..., 5.3726e-06,\n",
       "           1.0540e-04, 5.4127e-04]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.5044e-01, 4.9559e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [6.0475e-01, 9.4735e-02, 3.0051e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [3.9606e-01, 2.3935e-02, 7.2493e-02,  ..., 2.4958e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [4.7920e-01, 2.3430e-02, 4.2185e-02,  ..., 1.0899e-01,\n",
       "           5.9912e-02, 0.0000e+00],\n",
       "          [5.5881e-01, 2.0210e-02, 4.3704e-02,  ..., 1.2199e-01,\n",
       "           3.7250e-02, 3.2944e-02]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [7.2505e-01, 2.7495e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [4.3125e-01, 4.1897e-01, 1.4978e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.1677e-02, 3.5574e-02, 1.0703e-01,  ..., 1.0927e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.3018e-01, 4.2536e-02, 5.9461e-02,  ..., 1.4459e-01,\n",
       "           2.6023e-01, 0.0000e+00],\n",
       "          [4.1816e-02, 1.6204e-02, 6.4292e-02,  ..., 2.5367e-02,\n",
       "           6.8881e-01, 8.8385e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8007e-01, 1.9925e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [6.4111e-01, 2.4331e-01, 1.1558e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [3.9959e-01, 1.3489e-01, 7.3203e-02,  ..., 1.2445e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [3.4823e-01, 1.8753e-01, 1.2360e-01,  ..., 1.0824e-01,\n",
       "           3.1912e-02, 0.0000e+00],\n",
       "          [3.5513e-01, 8.2514e-02, 8.4168e-02,  ..., 6.9727e-02,\n",
       "           1.9211e-02, 3.8553e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9601e-01, 3.9916e-03, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8012e-01, 2.6357e-03, 1.7246e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.2502e-01, 2.4934e-04, 3.4550e-04,  ..., 4.2324e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8753e-01, 3.0122e-05, 9.4748e-04,  ..., 1.0488e-03,\n",
       "           2.7376e-03, 0.0000e+00],\n",
       "          [9.5201e-01, 5.6990e-04, 1.5978e-03,  ..., 2.5284e-03,\n",
       "           2.5647e-03, 3.4641e-03]]]], device='cuda:0'), tensor([[[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1161, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0688, 0.0178, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0269, 0.0161, 0.0215]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1574, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.7118, 0.0450, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.3850, 0.4613, 0.0440]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2805, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1966, 0.0050, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2186, 0.0095, 0.0399]],\n",
       "\n",
       "          ...,\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0396, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0271, 0.4430, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0291, 0.0583, 0.4138]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1409, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1391, 0.0180, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0906, 0.0222, 0.0121]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.5113, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0026, 0.9770, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0089, 0.0321, 0.8517]]]]],\n",
       "       device='cuda:0'), tensor([[[[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.7641e-01, 2.3591e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9576e-01, 1.3844e-03, 2.8544e-03,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.7946e-01, 2.6613e-03, 2.0001e-03,  ..., 2.7149e-03,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9397e-01, 2.4806e-04, 1.0859e-03,  ..., 1.2379e-03,\n",
       "           7.7361e-04, 0.0000e+00],\n",
       "          [9.9262e-01, 3.0779e-04, 1.6345e-04,  ..., 3.9772e-04,\n",
       "           1.3106e-03, 6.0623e-04]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8432e-01, 1.5683e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8708e-01, 5.4047e-03, 7.5169e-03,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.6174e-01, 4.9232e-03, 2.9103e-03,  ..., 8.9919e-03,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9248e-01, 1.9867e-04, 7.2997e-04,  ..., 1.0746e-03,\n",
       "           7.3668e-04, 0.0000e+00],\n",
       "          [9.8247e-01, 6.8648e-04, 7.2608e-04,  ..., 1.9106e-03,\n",
       "           1.8317e-03, 3.1735e-03]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8258e-01, 1.7418e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8792e-01, 4.8826e-03, 7.1937e-03,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.5079e-01, 5.7359e-03, 4.0271e-03,  ..., 1.1636e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.3172e-01, 1.1664e-02, 1.0888e-02,  ..., 7.3703e-03,\n",
       "           8.1443e-03, 0.0000e+00],\n",
       "          [8.9907e-01, 3.8474e-03, 7.8422e-03,  ..., 9.0608e-03,\n",
       "           1.1594e-02, 3.5373e-02]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8891e-01, 1.1085e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [7.9727e-01, 3.4752e-02, 1.6798e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [6.1456e-01, 2.2353e-02, 8.3103e-02,  ..., 1.5659e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [6.8606e-01, 4.3661e-02, 1.2398e-01,  ..., 5.0090e-02,\n",
       "           3.5633e-02, 0.0000e+00],\n",
       "          [5.3422e-01, 7.4958e-02, 4.1980e-02,  ..., 1.5232e-01,\n",
       "           7.4536e-03, 3.3917e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9720e-01, 2.7989e-03, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.2669e-01, 7.8194e-03, 6.5495e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.0544e-01, 2.0479e-03, 1.0854e-02,  ..., 3.1631e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.3174e-01, 1.2424e-03, 9.0356e-03,  ..., 1.7442e-02,\n",
       "           6.6914e-03, 0.0000e+00],\n",
       "          [9.0185e-01, 3.0590e-03, 1.1506e-02,  ..., 2.9142e-02,\n",
       "           4.9388e-03, 8.0483e-03]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.5792e-01, 4.2079e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8957e-01, 3.5895e-03, 6.8365e-03,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.2041e-01, 9.8687e-03, 2.1577e-03,  ..., 2.4121e-03,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8868e-01, 1.4206e-03, 8.6551e-04,  ..., 1.0161e-03,\n",
       "           2.0379e-03, 0.0000e+00],\n",
       "          [9.5118e-01, 2.1014e-03, 6.4063e-04,  ..., 1.4289e-03,\n",
       "           9.2268e-03, 1.1254e-02]]]], device='cuda:0'), tensor([[[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2385, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1615, 0.0493, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1789, 0.0177, 0.0393]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0382, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1179, 0.0040, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0994, 0.0142, 0.0123]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1405, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2220, 0.0082, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1919, 0.0061, 0.0205]],\n",
       "\n",
       "          ...,\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0820, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0756, 0.0127, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0796, 0.0062, 0.0100]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1975, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1601, 0.0461, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0868, 0.0513, 0.5478]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2036, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2329, 0.0511, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1138, 0.0076, 0.0227]]]]],\n",
       "       device='cuda:0'), tensor([[[[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8823e-01, 1.1773e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9295e-01, 3.4912e-03, 3.5553e-03,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [9.0201e-01, 5.8266e-03, 6.2012e-03,  ..., 1.1629e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8062e-01, 1.7926e-03, 1.2440e-03,  ..., 3.3050e-03,\n",
       "           5.5118e-03, 0.0000e+00],\n",
       "          [9.6136e-01, 4.8013e-03, 2.6793e-03,  ..., 2.9327e-03,\n",
       "           1.8893e-03, 1.3631e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.6477e-01, 3.5234e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.3722e-01, 3.7516e-02, 2.5269e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [8.3275e-01, 4.3931e-02, 1.8565e-02,  ..., 1.2375e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8225e-01, 2.3540e-03, 1.3744e-03,  ..., 1.3466e-03,\n",
       "           3.9835e-03, 0.0000e+00],\n",
       "          [8.4728e-01, 3.4680e-02, 1.1453e-02,  ..., 1.0055e-02,\n",
       "           1.0100e-03, 1.3874e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.9124e-01, 8.7608e-03, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8167e-01, 1.3913e-03, 1.6943e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [8.5633e-01, 2.5241e-03, 1.2079e-02,  ..., 5.7199e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [7.7715e-01, 6.1796e-03, 1.4247e-02,  ..., 3.9748e-02,\n",
       "           2.3269e-02, 0.0000e+00],\n",
       "          [8.6100e-01, 1.2063e-02, 1.6055e-02,  ..., 1.4901e-02,\n",
       "           1.9934e-02, 1.8932e-02]],\n",
       "\n",
       "         ...,\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.7346e-01, 2.6537e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.7002e-01, 9.3727e-03, 2.0612e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [8.9866e-01, 8.4518e-03, 1.3520e-02,  ..., 3.9152e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.8693e-01, 2.2107e-03, 1.0570e-03,  ..., 1.4663e-03,\n",
       "           9.9648e-04, 0.0000e+00],\n",
       "          [9.5667e-01, 7.0803e-03, 3.9780e-03,  ..., 4.4366e-03,\n",
       "           2.4186e-03, 8.8355e-03]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [4.9055e-01, 5.0945e-01, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [5.1461e-01, 4.0796e-02, 4.4459e-01,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [1.6073e-01, 1.8041e-02, 1.6256e-01,  ..., 2.4228e-01,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [7.5270e-01, 1.4220e-03, 2.0179e-02,  ..., 9.0046e-02,\n",
       "           1.0528e-01, 0.0000e+00],\n",
       "          [7.4130e-01, 3.5979e-03, 1.2891e-02,  ..., 7.6218e-02,\n",
       "           1.5860e-02, 7.8043e-02]],\n",
       "\n",
       "         [[1.0000e+00, 0.0000e+00, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.7050e-01, 2.9500e-02, 0.0000e+00,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [8.9770e-01, 2.0796e-02, 8.1501e-02,  ..., 0.0000e+00,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          ...,\n",
       "          [7.1614e-01, 1.1350e-02, 3.9824e-02,  ..., 2.9973e-02,\n",
       "           0.0000e+00, 0.0000e+00],\n",
       "          [9.0816e-01, 3.1289e-03, 1.2781e-02,  ..., 1.7169e-02,\n",
       "           1.2273e-02, 0.0000e+00],\n",
       "          [8.7584e-01, 7.9552e-03, 1.0025e-02,  ..., 1.1654e-02,\n",
       "           9.6966e-04, 5.9688e-03]]]], device='cuda:0'), tensor([[[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1437, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0638, 0.0131, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1065, 0.0124, 0.2067]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.3185, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0883, 0.0135, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1503, 0.0134, 0.2896]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.8751, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.5313, 0.1115, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0268, 0.0056, 0.9479]],\n",
       "\n",
       "          ...,\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1344, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0974, 0.0479, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0670, 0.0095, 0.0372]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.5323, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2996, 0.3266, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0604, 0.1085, 0.7294]],\n",
       "\n",
       "          [[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],\n",
       "           ...,\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.2658, 0.0000, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.1935, 0.1604, 0.0000],\n",
       "           [0.0000, 0.0000, 0.0000,  ..., 0.0506, 0.0050, 0.5207]]]]],\n",
       "       device='cuda:0')))"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():    \n",
    "    y = model.forward(input_ids,output_attentions=True,\n",
    "        output_hidden_states=True,\n",
    "        return_dict=True)\n",
    "\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "328ab70f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['logits', 'past_key_values', 'hidden_states', 'attentions'])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "4953d6fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([50257])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.logits[0][0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7da88a5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(278, device='cuda:0')"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.argmax(y.logits[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "60d033a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ing'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(278)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6057e9d9",
   "metadata": {},
   "source": [
    "Note that most logits are negative, a few early (therefore more common?) ones are higher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8036b5c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2286fe27220>]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAt70lEQVR4nO3dd3wUZf4H8M83BUKvAUINvUuLEaQIIt2Kcoo/sZ6c3nnn2QE9O4Lt7Cfgnb03FEWkN6kGpPcSmkDoIAFCkuf3x84mW2Z3Z3dndjaTz/v1yiu7M7Mzz2z5zjNPFaUUiIjImRLsTgAREVmHQZ6IyMEY5ImIHIxBnojIwRjkiYgcLMnuBHiqWbOmSk9PtzsZREQlyooVKw4rpVL11sVVkE9PT0dWVpbdySAiKlFEZFegdSyuISJyMAZ5IiIHY5AnInIwBnkiIgdjkCcicjAGeSIiB2OQJyJyMEcF+f0nzmDOpoN2J4OIKG44Kshf8cYi3P4+O1MREbk5Ksgf/uOc3UkgIoorjgryRETkjUGeiMjBGOSJiByMQZ6IyMEY5ImIHIxBnojIwSwP8iIyUEQ2i8g2ERll9fGIiKiYpUFeRBIBvAVgEIA2AIaLSBsrj0lERMWszslnAtimlNqhlMoD8DmAqyw+JhERaawO8vUA7PF4vldbVkRERopIlohkHTp0yOLkEBGVLrZXvCqlJimlMpRSGampupONG92PiakiInIGq4P8PgANPJ7X15aZLr+QQZ6IyJfVQf5XAM1FpLGIlAFwA4ApFh+TiIg0lgZ5pVQ+gHsATAewEcCXSqn1Vh4TAC54cjreWbDD6sMQEcU9y8vklVI/KaVaKKWaKqXGWn08ADh5Nh9jf9oYi0MREcU12ytezVBQqPD8tE12J4OIKO44Isj/su0w/vvLTr/l5/ILTDvGjkN/IOfUWdP2R0QUC44I8vkFhbrLn/lxg2nHuPTl+cgcO9u0/RERxYIjgnygJvIfL90d24QQEcUZZwR5uxNARBSnnBHkg/R2jUU5+ru/7MSWg6csPw4RUbicEeSDrNMrR+8+fg56vjDHtOM//eMGXP76L6btj4jILEl2J8AMh/84F3KbnJNncct7v2JE10bYd/xMyO1PnDmPo6fz0LhmBUNpyAtQ+UtEZCdHBPlz50MH2D9NXILsI7kYM3mtoX1e8cYv2H00F/3b1MZve45HmUIiIns4orjGiF1Hc/2WjZm8Fu/qtK8HgN3a9jM2HMShU6HvFIiI4pEjgryR1jV6dbOfLtuNp01sS09EsXW+oBCZY2dh6pr9diclbjkjyJfgseQPnDiL8yzPJ4rIsdw85Jw6hyemWD7uYYnliCAfrfRRU/HB4mycyD2Ps+cLkD5qquHXrv/9hN+yFbuO4peth0O+9vtV+9B13Gxc8QZb5hCRNRxR8VpoQk7+iSnr8cSU9binTzND22cfPo3eL83zWnYmrwAbD5zEtW8vcW0zfkjQfdz7+SoAwKYD9rSxV0rhk2W7cWXHuqickmxLGojIWg4J8sHXvzlnq+F9/XEu39B2/9OpsG39+M+GjxMPVu4+hse+W4clO47grRs7252cEi3n1FlUTklGSnKi3UkpXUpuSW3MOKK4JlRG/qUZW2KTkCjlnDqL+VtiN5n5Wa3p6bHTeTE7plNljp2N29//1e5klFoidqcgfjkjyJt4OQ/1ZTmXX4A9R3NNn1P2wImzGDZhCW55d7mp+w3G8+KolDLUSSxaC7YcwverLJnmF8dO5+Hk2fOW7NuIxduPGNqu/RPT8cT36yxODZGLI4J87UopMTvWBU/OQM8X5uKz5aFHuCwsVLj05Xn4esVev3W+wyN3HTcbu474t+U3w5aDpwIOxwy4LmwfL9uN7uPn4Mkp6y1trXTzu8tx7+erLAnGnZ6ZiU5PzzR9v2Y7dS4fHyzZZXcyHIGlNaE5IsinVYldkD+Xb7y5Y+75Auw4dBoPfrUaALD14Cms3H0MAII2+Tp06hyyso8aOsbvx8/gq6w9yM3Lx7hpG3H2vPdEKf/7ZSf6v7IA46dtwuhv12JbTnElr+cd0K87Xcd7f3E2flp7wO84u4/k4rmfNkZ1Afjn578VPb7gyRkR7yeYApPvsKhkYGlNYI4I8mZ6b1G2Zfvu98oCDP3PYgDAJ8sC3wlc/dYiXDdhie66L37djfRRU4uC+Q2TluKhr9fglZlbMHH+Dny81DuH6J445b+/7MRny3fj7o9X+u1TfH4iernsuz5egUkLdmDLwT+CnGFw3636PeLXxovPlu/G/hPWF2sF8s6CHWE18SVyRJB3Wt4tUNn4+YJCPPKNa+ydo1plqXvIhaJK1Nw8XDh2lqE7Ac9MuWddRKFSyMo+iqvfWlQ0hWJ+Yew6bJ04cx4ncu0rWw/k2Ok8jP52LUb8L3b1Jr7snKD+9Ll8bLapuW8gJbgfZMw4IsiXBH/5KKvo8VdZewy9ZvF27w5Vb8zZ5vV8W84pnNFy9O6il7fmbsehU+dw/aSlhnN8vpXNK7KPYczktVi15zg+0+44Yvlj6vDUDHR42prinGi43wIjo576uuTFubjqrUXmJijGRn6UhQGvLghavxNr7u89W9cExiBvoZ2HThc9nr7+YNHjh75eY+j1N76zDICr5c34aZvwu08Ov98rC4oe+wbhcMumPX8jBR47e/IH77F97PoxfbtyL2ZuOBh6QwAjP8wKvVEEErWTj6Tcf9eRXKyOYDTTvPzCmLR6MmLZDtfd4d5j8ZEeT75FjlTMEUE+Xj/eK96MfriC9FFT0XXcbEyYvx0rdx0rWr77aK5XYA9Wxu9J6TzeozNCp2/Zu3vbV2Z69zkoKFQ4FaClzPHcPIz8MAvHc423w1+3r3iYiG05p4pyzfd/uRp3GgzeMzYcxG+7j4Xe0ICPl+7CzsOui3WC9mspjGHl7qhv1qD7+DnIzTPWSS8WfHt624nFNaE5IsiXls/Z8zy35kRWAbot5w+/FjLZIZpujpu2sSj3Om2dd8ubR75Zg/ZPzkBBoUKP5+eg/yvzi9a9uygbMzYcxPuLsw1Pw3i5xzg+l/17AXq9MNfQ63xd85/FeH/RTq+7nw8WZ2Pj/pNFz/MLCoMWPSil8Nh363C1VsySoOXkze4jEcyczTkAjM2ZYLV4/p2xuCYwy4K8iDwpIvtEZJX2N9iqY5UWZn2POz0zE+mjpuL12cXDPQRr+TJx/o6i3Kwvdx+AQqWw99gZbDn4B2asP4Apq73396ZPfQIAPPVD6JEDc/MK8NL0zQHXn8krwPhpm/yajgKuoqZ+/56PP01cghNnzuOJKesx6LWFReszn5uNDk/pl/2v2Xsc/bXisBNnvO9UzBgryQ6FhQofL92FL3/dE7Lo6MMl2djw+8mg25hhw+8nMXH+dlP3+cvWw5i21nvoYaUUPl222/YKfaUUflj9e0yb+lqdk39FKdVR+/vJ4mOVKgujGP7guPZFX7FLv0gj52T4FYueQXbkRyvwj89+87qX/lCn849vc9XHA/QCfXOu/wXC7e352zFh/nZ8FKBz0em8AizfeRQ/r/Mfb/zo6TyczitO9/mCQvx34Q4cOnUOT/2wIeDdUkGhglLKq2ipJJi6dj8e+24dHv5mTchK4Me/X4/Bry8Muo0ZBr++EOOmbYr49Xqh8qb/LcPdn3g3FV677wTGTF6Lh75eHfGxzPDNyn34+2e/4b1F+pMVWcERxTWl0QyDlZCRWLIjdPf80+fyvcry2+t0bnpdy71vP6R/F+BL70Kgp6BQYfshVwBerwXa8yGaeL4y03uQujmbit+/jGddvWQ/WJyNZ6duxIVjZ3ldAH2LAgoV8N+FO3H5G7/g+Z83YfS3xRXpe47m4i8fZRVd9DYdOIkmo61v137gxFlkZR/FuGkbkRegw57e4HufL9+NTQdC59gvf2Nh0NynUipkR7mVu48h56SxYrtgck6eRfNHf/K6G3F/RG8FyBC4OzEe1Rmn6eDJsxjxv2UxyeUf0eqYcnxmm9uWc0q3bswMVo9CeY+I3AwgC8ADSim/rKOIjAQwEgAaNmwY0UFW7z0eRRJLjngqJmj7xHTD27p70wZzMIwff9MxrpvCCTd1xuxNrjLrUK0rDvjs//b3iytxD//h+uGfPudf5AO4bkh8y+7d7dXfnudd1DB26kZMX38QczflYFD7NAx81Ts3/GQYk1vsPHy66K7LP00KIoJlO45g9qYcTFqwo2hd/WrlMaJrI0PHGPWtq99FqGGx1+0LfCE4kXseHZ6egceGtMafezYJuN3Q/yxGlXLJWP1Ef0NpC2Th1sM4X6DwweJsPDCgZdHyQ6fO4cUgRXuBvD1vOxZuPYxvf9uL27o39lufX1AIBSA50bo88cgPV6BtvSp4Y3gn0/cdVapFZJaIrNP5uwrA2wCaAugIYD+Al/X2oZSapJTKUEplpKamRpSOs3n6P06nCVVBGivh5nh8A6yvK974BRc9NzvsdKzaU1xcYrRiN5hglXdmTRP5/uLsosfdxs3GufwCfJm1B0q5ysu/+NXVSiovvxB9DLRiuX7SUq8ADwAFBtuxrzSpBZL78/3SQP8Pd/3Gku1HijraRcPz7iFQcZ/X9jrLQrVcuuTFeWj+6LRwk2b4+FaLKsgrpS5TSrXT+fteKXVQKVWglCoE8A6ATHOSTHabuta8+TT7vjwPayMs2/b8cYYzHIVvfwMjjBYlhWP/ibNo+djPePjrNfhu1T489t26oh7NeVF0ODr8Rx6un7jEr9OW743g8ElLQ+5rVYAK2s7PzNQt+th7LDdksc3mA6cw/J2luHjcHN31uXn5GDN5LU6dPY9Crf5j8fbDuvUqbiLi1+rpySnri6bWDHaf92WW/wCCnszsp+Au8oplYyArW9ekeTy9BoB1Y6uy/VRMjZm81rR9GS2vd5urNSkEIg+8A19doLs8IcTXyEhxmTuoBAqOgezz6GCk99oCpXDt24uLnu85GjjwvL84G8t2HsXYqRsNd8BKHzUV437aiIMnz+JZj7sW317XbkdP52HRNu91Ww7+gR7Pz0Xj0T8FzVUf0/pNHAkwj8FHS3bh02W78eqsrWgy5ie8NGMzbnxnGe76eCXmbzmEB74qrjwN9T5MX+8/2J6bbz3C58uN9UQPRSnldwHcuP8kPl22u6g4aW8MO7hZWfH6goisFZE1APoAuM+qAyWH+nWSY9z2XvQTc5w8q397LiEyC3qVy77cdQQTF+zAmjDqijwntrn6rUV+lafHc/O8KoN7vRi4/4C7gnXyb/uKWtHc8f6vIS/OExfswMNfr8F/dWY9C1ewC3CoX6s7Q+6+U3trbnG9h+98C8Pfcd2NnC8o1O0R7Xtd9nwP+748H+086pY2H/Qfl2eLzjK3nJNnddePmbwOnZ+ZiV1HijMwg15b6PX+Hzp1DgdOnI1JJzfLKl6VUiOs2rffsWJ1IHKseZtzvDpKmeHWKC5ID/sMfeEZ6MJ1+lx+0cXHk96w2b4D0b3w82a88LN+ZWakvzu9i+m0tfvRtm4VNKxRvmjZ1hAjnnrW9ZwKcOEuPmbx45yTZ1Grcgp2BOj74Xby7PmivhIAsOvIaeTmFaB1WmUAwEXjZkMp/0pr91wTe4+dQaMaFXT3vXznUXQdNxtNUyvgvVszQ7YOiwabUBLBFZB/XGNeXQOg31zPqAVbvftBTP4t8tm0wmkJtWibsdmtPE1ZrZ+2cOo+7v5kJXq9OBe5eflFg45lBejH4eY5E1c4Jbaf/7on4F3WiP8tK3p8xqdBxyUvzvPqTOe+S/BtTux25HReyMHcth86jV4vzg1a7BQtBnmiOBSorbunUd8YG+jOKu65gQPdZVw8fo5uoA8WkNs8Ph27Doffiiw3jBZ2/565BVe+qd8ZbOHWw1i95zjy8gsNN1ke/s5S9NQZfuMfn/2Gv3260lA/BCs5IsjHUfNxopj5/FdzKgoj9dQP60MOu3zx+Dl+rXjmbfYvOvL0hcGhuI345xertCG3jWf17/tiFVo8Ns1wn4Y1e12tw07rdDabvv6gX1+JWHNGkGepPFHMFSog49lZIbfz7UEdTf1CuCIZI8adg/ccHtyTUgrrf/dv9tv2iem2j42jxxFBnojimxUTt1vFd8gBX1NW/44hr+sPI379JP1pO+3kiCDP4hqi+Hbkj8groc0QzgT0ocr3g02BuCnOpkcEnBLk7U4AEQVlZIgGKz071by5cY+FMQlOPHBEkCciCibcHsjBfGZSz9hYcUaQZ3kNEZEuZwR5IiLS5Yggz3w8EZE+ZwR5RnkiIl2OCPJERKSPQZ6IyMEcEeQ5rAERlXSzdMbDN4MzgjxjPBGVcGfOWzNXtSOCPBER6XNEkGdGnohInyOCPKd4JSLS54gg36NZqt1JICKKS44I8m20iXWJiMibI4J8lfLJdieBiCguOSLIExGRPgZ5IiIHiyrIi8gwEVkvIoUikuGzbrSIbBORzSIyILpkEhFRJJKifP06AEMBTPRcKCJtANwAoC2AugBmiUgLpZQ1XbqIiEhXVDl5pdRGpdRmnVVXAfhcKXVOKbUTwDYAmdEci4iIwmdVmXw9AJ4TIe7VlhERUQyFLK4RkVkA6uiselQp9X20CRCRkQBGAkDDhg2j3R0REXkIGeSVUpdFsN99ABp4PK+vLdPb/yQAkwAgIyODw9AQEZnIquKaKQBuEJGyItIYQHMAyy06FhERBRBtE8prRGQvgG4AporIdABQSq0H8CWADQB+BvA3tqwhIoq9qJpQKqUmA5gcYN1YAGOj2T8REUWHPV6JiByMQZ6IyMEY5ImIHIxBnojIwRjkiYgcjEGeiMjBGOSJiByMQZ6IyMEY5ImIHIxBnojIwRjkiYgcjEGeiMjBGOSJiByMQZ6IyMEcGeTv7dvc7iQQEcUFRwb5+/q1sDsJRERxwZFBnoiIXBjkiYgcjEGeiMjBGOSJiByMQZ6IyMEY5ImIHIxBnojIwRjkiYgcjEGeiMjBogryIjJMRNaLSKGIZHgsTxeRMyKySvubEH1SzVG1fLLdSSAiipmkKF+/DsBQABN11m1XSnWMcv+mq16hDI7nnrc7GUREMRFVkFdKbQQAETEnNUREZCory+Qbi8hvIjJfRHoG2khERopIlohkHTp0yMLkEBGVPiGDvIjMEpF1On9XBXnZfgANlVKdANwP4FMRqay3oVJqklIqQymVkZqaGtlZ6Hjl+g4AgCHt07yWJyewrpmISo+QEU8pdZlSqp3O3/dBXnNOKXVEe7wCwHYAMR3/95pO9ZE9fgga1ijvtfydmzMCvIKIyHksydaKSKqIJGqPmwBoDmCHFccKpXvTml7PfYM+EZGTRduE8hoR2QugG4CpIjJdW9ULwBoRWQXgawB3KaWORpXSCPVoXhObnx1ox6GJiGwXVZBXSk1WStVXSpVVStVWSg3Qln+jlGqrlOqolOqslPrBnORGpmxSIiqnJOGOHo0jev2wLvVNTlH8ykyvbncSiMhE0baTLzHWPDkg4te+OKwDvlqx18TUxC8FZXcSiMhEbGpCXgoZ44kchUGeiMjBGOTjTL2q5aJ6feeGVaN6vVLMyhM5SakO8p/8+SK7k2C6auXLBF1fpZyzB2gbPaiV3UkgiiulOsh3b1Yz9EYOs/zRvnYnwVJNUisie/wQu5NBFDdKdZAPx0d3ZOLvlzaz/DiPX9HG0v2XTUoMur6kF9ZwKGkibwzyBvVsnorbu/u3s39jeCdTjzOgbR2ULxM8EFvJt0i+Q4OqtqTD19NXtbU7CQFVSik1LZGpBGKQj1JyYuC3MCU5sre3bhSVr2bnxOOlDP/mbul2JyGgmhXL2p0EooAY5MPgHjY/MaF4/PxgQXDhw5danSTbdahfxe4keOHMBkTeGOQjULFs8e15pwBNFiuUSURqpchyeKEC1bu3ZuD+fi2w4enIe/GapU6VFLuT4JWT1ruTifRzIHICBvkwuItmWtauZGs6Lm1VG//o2xzly5hfFty3Va2wtr/xokamp8FsDw1oaXcSiGzDIB+GCmWT8OmdFxkakz6aKRFH6bT1TohROcTgC9JCb+TBaLpevO6CCFJTMrADGcWzUhvka1Z0dRrSa8mS9dhl+OD2TN3XXdy0JqpE0EyvZ3PjbfL7tq6N7PFDMLRzPb91E0d0CfpaswOOWdeWYRkNou6NS6XX3b2b2p2EEqtUBvmP7sjEj38POO0salYsi6omtypp5DNZSVqVFDx/bfugr2lSs4Lfssta1zY1XWSdaIeoIJdXru+ARwayJ3OkSmWQ79k8NWSFYdkImz+6hcoBX5heHSnJxtvDh5s/b6xzgYhHreqEX7/hWfEdyMvDOqBNmu60whRCvLWYuqZT6ZnPwQqlMsh7ClS60aqOuQFCTCr4MLqXf13eOqL9x7p4eeo/iu+oJtwUvCjKzch7cG2X+mhTV/sMLT4nx5XIR1GfRPGn1Af5aEX6exBx5eY91ajgPbiYZ9M/o8G3htacUO8uoWMEvVevM3FWrIbV/efXTfSqubW/PiGSO4tgjNx1BNO9WQ2TUmJcvHSAK22sarXHIG8VnwijdzGoW7UcZt3fCwDQNLUCVvyrn9f6YV0a+O82ROR66sq2eO6a9ujWpIZfxe1Lw8Jv4XJFh7phvyaQ54Z610FMClGJHGvf/vVifHZnV9P2JwI82L9F0fMujaoF3LZccqJfQG9bt7JfRsBqVcsnx6wlV6yFGi7EqXUoDPIeRnRthKGd/Fu0mCGS302CwV+b52YVyibhxosaQkTw7NXtipa3SauMZrX8cwoLHupT9Hh4pv9FJRKBcsOe7fqHdq6H/m3rGH5tLHRuWA3VKgQfqjkc1SuUQZ0q+oGjjM9wGEM719MdHmHe5kOmpceIQe3SHNtrOLNx8Atm5yAX4Vj4R9/mluyXQd7DM1e3w7+v72jKvkI1+brigvByyJe11u+k9MsjfZD1WD/ddZ5BNdAdQMMa5TGgravFTtPUirrbRDIA1/IxfYvuUjy5R/LUC2gdGlT1KqO3QtPUyCqkh2c29Fs2+4FLsPDhPkGL0gK973rjGt3U1b9j2aFT5wynMZBwcuYjezUpevzQgJZY82T/qI8PAFd1NO+OMFLdmwZvxtyurn0V9U9f1RZDwuyjYhSDfIR8c2Ke5j7YG3/tHXhY4uzxQ3BZm/CaQr55Y2f88kgfv05W9auVR/Uoc5/1qhaXldet6t/qyN0567ou9fHpncEnWqlXtRyeurItalVO0b1zqKCVUevFnbQq5XzK6M0z54FLsO6pAZj9QG/T9tk0tSIaeNQzfHZnV7x7q3dHufQawS8qV3oUh/kWzdzUtRGu7hRdcEyrkoIZ911iePskj/e/dVolVE4xp3z+tRvMHa01En/u2RhLRweeT+HOnk3Q1qZAH6woL1oM8hbr3TLVlP2kJCeifjX/ikujjDapLF8mKeCkG8FG3HSb9s+euKiJtZWFLxjsPesOwDUqlkGT1IpRVYL6Vhrrld/WrZqCS1t5X7xbhih+alFb/+4JcN09hNsvol097yA1tHM9NKsV+BiAf+Xuo0NaIzO9Orpa/DkGsjbE3cPz17bH4lHFg/8ZvdMUEdSpkoJKOt+Dx4a0RkKCOLLZrWOCfFqEA2VFeoukDLYEuaSFK8hHM8xBNNxBJNidRTiibQpqpJXQBSHaaQ9qb+wz+2vvpvjvzRm4NMzxePR4FmMAwI0X+RffhCPQ98G3Z3SnhtVCznSlN89BIOV0Wl3VruT922lWqxK+vKtbRGMjDTOhNVallOSgv+cG1cujskcLoB4mzPB2y8XpusvrWjgA323d09GguiuzYFYTaz2OCfIVIsyljRsavNdpKHofTvUQ86x6cs/UFE0u3Qi9YhCj0+R5Bmbf3UTc3C7Id3rKPT1wa4AfXTiSEhNwWZvaEV9gHxtS3NcgWDGSO0DUiGJcefdbbCTnfkcP76DeOi3w3YLnZ1cmKQEbnxmITc8M9Prsn/GooI9Wt6bR5f4XPtwn4Lo6lV0BV3x+dYE+3nA+9kCbLg5SvPP+bReGXbzoeRF84oq2qGDBIIO+ogryIvKiiGwSkTUiMllEqnqsGy0i20Rks4hYPiau3hAARhgpggjHhqcHhDW2TYPq5THhps543eQZpqwgAmSkV8dt3dMx78HeeO+2C3FB/apR7/eD2zPxqc+k6uH0BvZKY9SpKfbnnk1CbwRX0M0eP8RQcVDZpARc16V+wAB0Q4AWTjUqlEHvlqm4skPdiFphfPmXbpj3YG8Axe/tV3d1w6QRXSLOIFmhhjamlF4jAN+hQSKldzPpXhbOhaFB9fL4Zxifxaz7e2H8tbEfqC/aCDcTQDul1AUAtgAYDQAi0gbADQDaAhgI4D8iYumcdq+Y1ComEs21Ms9HBrbyu8V1V7y1CNLRYWC7tJh2QKldOfIcZ2KC4Ikr2iK9ZgX0aRl+McjwzAbo2bymV270khapuNjnlvsffZvhXoM/oFphjhc/qJ1/081wRdpJbPOzg/DSsA4B1weag3fFv/rh/dsy8frwTn7flTJJoX/GmY2r+804dmF6dd1mrNEwq1Tyrf/r7LcsWElfqHGgzGBG0U1igmBo53r4Sy9jGQgzRHUJV0rN8Hi6FMB12uOrAHyulDoHYKeIbAOQCWBJNMcLxs7cyNd3X4xDp87pVnD1aVULP9zTA+3qVcaYyWttSJ2/eQ/2QV5BoWn769+mNmZsOGho26rly+CjO4K30AFcFcD39WuB12ZvLVr2zd0XF40e6vbaDR0BAPd+vspwei9pkYpp6w4Y3t7tszu7YmvOqYinIkwMEgEjHU7iqSvbBmz/fUmLVNzew3h5fSDNa1XE1pw/ip5/dVc3DJtg2U8ZQPBiQBHvi4lAYjIX8c/39cLpc/noNm5ORK+vphXj/vtPHU1MVWhmllXcDmCa9rgegD0e6/Zqy/yIyEgRyRKRrEOHYtvxwyxVyiUHbcHQvn4VyyteFzzUBz/+vYffcr3gUa5Moql3DhNu6qJboWe2Lo2qoZFPk8SrOtZDUkJsqpa6Na1hKMC/daMrF+r7nryqXZA86X0vfC9kwdxycXrASrsPbs8Me/5Zve/L5L919+qnEarFUMyJ/1hTTWpWQP8wmilL0X//99Ld56VySjLSqpTzas0TzrU5WJ2NleEh5K9DRGaJyDqdv6s8tnkUQD6AT8JNgFJqklIqQymVkZpqTnPDePWXXk3w/m0XWrLvhjXKo129wK1SLP0SJYhX++rSbnD7OnhoQEu84VPPYnSC9vkP9cHqx413QvJt6RVoSspIVSybhF4tin+bkX7SRooofIs7vTJPYUTUOQ/2RssAgwyGO+fCIwNbeVVUp5awidtDlnEopS4Ltl5EbgVwOYC+qvjd2wfAswapvrasVBs9OLKRIe3mDuDJpTCQ39KtEQ6cPBvWa0QEf+sTbpPV4sATTdFjgggm/7V7xK8PZETXRnj8+/VR7WP04NY4e74AHyzZZWj7Lc8O0u2tG+hb+Pb/dcbdn6wsen5Pn2ZoUK0cHvp6jeE0/l/Xhvgia0/oDcNQtXwybtbpzRwr0bauGQjgYQBXKqVyPVZNAXCDiJQVkcYAmgNYHs2xjHp4YGzm8xzRNR1A8KZ1TjG0c338uUdjPBDhXKmZFgyy9dEdmX69S43Sy8cNDFAB+dRV7TBxRGTHiRXPjKm7qMhsgYobrZy3oExSApICtH7zLFZxP/LtP1EmKQHDMgKPxzRqUCu/i8gF9asWNdU0Q9PUClj1eH/c31//txOLob2jra18E0BZADO1L8FSpdRdSqn1IvIlgA1wFeP8TSlVEOWxQjLa7tsM/7q8NcYMblUqgnyZpAQ8dnmbiF674KE+qFnJvEG/3Ho2N7doz4wOU+Gw4luTViXFa5gFq3gG/O7NamDn4dNe66/sUBenzp7HXJ3B1epVKy6yev7a9ujWxNWqasrqfdh/Ivgdk9EOiEbd1LURXvh5EwDvc1o6pi/SR0019VihWFmcGm3rmoD3pEqpsQDGRrP/eCYiSEqM/wA/YUQXvL8oG80CDD4GuIZeeHfRTkuGtW1oUtvmWIvVGCZOnAT89eGd8Gv2Ud0gf0ePJmhcsyIua13LK7Dec6nx9ubBGjE8f2175OUHbzk2uH0avlqxF8kx+P3a1dPdU/z0grBRn5apul24X7zuAr/WHCVN09SKIXs09mqRim1jBwW8NS4NPHOJ39zdLeCInGYx88dv5mXCyF1XhRDjsgeTmCDoF+bgfHq8mlB6PLn+wtDDTYwb2h6jBrUK2CchlGoVygCHTyMpQUKOUR8PGOQBvHdbpu7yYOV5TlOaA7yvLo1iO1FHNDzvBKK9bJRLTjQ0Xo1nUI3ljciYwa3x8Ndr/AZhC+XazvW9RldNSkwoas7Yv00d/Lz+QFjv3ds3dcaM9QfRqEYF3HJxOgoKFbbl/IGvVuz12zbUftvWrYzNB09FPYNYMAzyce7xy9sgv9C8jkslzYSbupjePDPWJSSJCYKCQusO6n53jPR8DeSHe3pE1RPalxXvcaeG1TDzftewyWfPG6/ie/lPgXsYvza8I46ezjM8QQ8A1KqUUjT2f3JiAv5ySdOAHR1D3bA9N7Q9/q9rQ0vHrmKQj3Nm9FgsyQaaMARBMG3SKmPD/pOWHmPRI5eGnPwjmqCYViUF9/ZtjmuimNWsfYiRP52qbFIi0gLM3hULKcmJlt858h6dSq3hmQ1w/YWuIrmG1a2re6lTJcXSICoiuK9fC6Rb2Jwxntlftelv2r3WznAWDubkqcQLZxgAXzd3a4TMxtXROsaTRZgVmJSytvldKH+/tDl2H81FmcQEzN6UY19CbOZ7J+b+TKwcJ94o5uSpxLuoSQ3c1j0dANBcZ8pBX54taUQk5gHeDO4gYncDzDpVUvDRHRdFNA+wWcIdhbS0YZAnU5TVmZg6lp64oi2yxw8Jayz/+LzRNybeU2710Nmedy8PxaiXezjiqfsDgzyZ4vOR3fBAvxamTfxc2sRTUIiEb/Jb1qmEjw0MKR2t5ESJuL17LMRBXygGeTJHs1oV8fcIZiwq7dxtziPtsh9vPWY9y6B7+MxXa/XxSB+DPJU67o4nlcvZV47snuYuIcKsXjx0l48HZo9nEznvdLiHlQ5nknWrsHUNlTpXXFAXx07n4YbM0F3grfLh7Rdh3uacuLidN1Osgm685eB9b6iqlEuO6YCJwTAnT6VOQoLg1u6NI54s3Ax1qqREdZH5U4Zrjtl4ydHHRypID4M8UQn07NXtsf6pAaViqGuKDoM8UQmUmCC2Tl7vK15Kxu0SZ/XfXuLnW0JEJV68lZXHWu+WqXhkYCu7k+GFOXmiOBDPOUEybmDbOnHXg5pBnshGpT3nGy1eHENjkCci08RPu3VyY5AnoqjZdT8SJy1I0by2a7pIz4nK4wUrXokoanbl3+OluOb27o3RqWE1dGlUze6k+GFOnohME6s6hnjJwbslJEhcBniAOXkiW7lbYnRrWsPmlFhj0oguqF4h8kldKHoM8kQ2al+/Clb+q59jA2H/ttbO0UuhRVVcIyIvisgmEVkjIpNFpKq2PF1EzojIKu1vgimpJXIgpwZ4KyUlCGpWLIPnrmlvd1LiXrQ5+ZkARiul8kXkeQCjATyirduulOoY5f6JiPyICLIe62d3MkqEqHLySqkZSql87elSAPWjTxIRlTR9WtYCALRKCz3HLsWWmWXytwP4wuN5YxH5DcBJAI8ppRbqvUhERgIYCQANG9o3vjcRRe7qTvXQr03tuBo0LRrLxvRFkkNG+Az5iYjILAB6tSePKqW+17Z5FEA+gE+0dfsBNFRKHRGRLgC+E5G2SqmTvjtRSk0CMAkAMjIy4qTVKxGFyykBHgBqV06xOwmmCfmpKKUuC7ZeRG4FcDmAvkqbcFIpdQ7AOe3xChHZDqAFgKxoE0xERMZFdekVkYEAHgZwiVIq12N5KoCjSqkCEWkCoDmAHVGllGJq5b/6cegsIgeI9v7qTQBlAczUpiFbqpS6C0AvAE+LyHkAhQDuUkodjfJYFENs1kfkDFEFeaVUswDLvwHwTTT7JiKi6HHsGiIiB3NOdTgRxcyqx/shv5CN4UoCBnkiClvV8qyzKSlYXENE5GAM8kREDsYgT0TkYAzyREQOxiBPRORgDPJERA7GIE9E5GAM8kREDsYgT0TkYAzyREQOxmENiMiQl4Z1QMPq5e1OBoWJQZ6IDLmuS327k0ARYHENEZGDMcgTETkYgzwRkYMxyBMRORiDPBGRgzHIExE5GIM8EZGDMcgTETmYKBU/M66LyCEAu6LYRU0Ah01KTklQ2s4XKH3nXNrOFyh952zG+TZSSqXqrYirIB8tEclSSmXYnY5YKW3nC5S+cy5t5wuUvnO2+nxZXENE5GAM8kREDua0ID/J7gTEWGk7X6D0nXNpO1+g9J2zpefrqDJ5IiLy5rScPBEReWCQJyJyMEcEeREZKCKbRWSbiIyyOz3hEpF3RSRHRNZ5LKsuIjNFZKv2v5q2XETkde1c14hIZ4/X3KJtv1VEbvFY3kVE1mqveV1EJLZn6E1EGojIXBHZICLrReRebbkjz1lEUkRkuYis1s73KW15YxFZpqXxCxEpoy0vqz3fpq1P99jXaG35ZhEZ4LE8Ln8DIpIoIr+JyI/ac8ees4hka9+5VSKSpS2z/zutlCrRfwASAWwH0ARAGQCrAbSxO11hnkMvAJ0BrPNY9gKAUdrjUQCe1x4PBjANgADoCmCZtrw6gB3a/2ra42rauuXatqK9dpDN55sGoLP2uBKALQDaOPWctTRU1B4nA1impe1LADdoyycAuFt7/FcAE7THNwD4QnvcRvt+lwXQWPveJ8bzbwDA/QA+BfCj9tyx5wwgG0BNn2W2f6dt/xKY8MZ2AzDd4/loAKPtTlcE55EO7yC/GUCa9jgNwGbt8UQAw323AzAcwESP5RO1ZWkANnks99ouHv4AfA+gX2k4ZwDlAawEcBFcvRyTtOVF32MA0wF00x4naduJ73fbvV28/gYA1AcwG8ClAH7UzsGx5wz9IG/7d9oJxTX1AOzxeL5XW1bS1VZK7dceHwBQW3sc6HyDLd+rszwuaLflneDK3Tr2nLVii1UAcgDMhCsXelwpla9t4pnGovPS1p8AUAPhvw92exXAwwAKtec14OxzVgBmiMgKERmpLbP9O82JvEsApZQSEce1dRWRigC+AfBPpdRJzyJGp52zUqoAQEcRqQpgMoBW9qbIWiJyOYAcpdQKEeltc3JipYdSap+I1AIwU0Q2ea606zvthJz8PgANPJ7X15aVdAdFJA0AtP852vJA5xtseX2d5bYSkWS4AvwnSqlvtcWOPmcAUEodBzAXruKGqiLizmh5prHovLT1VQAcQfjvg526A7hSRLIBfA5Xkc1rcPA5K6X2af9z4LqQZyIevtN2l9uZUA6WBFflRGMUV8C0tTtdEZxHOrzL5F+Ed4XNC9rjIfCusFmuLa8OYCdclTXVtMfVtXW+FTaDbT5XAfAhgFd9ljvynAGkAqiqPS4HYCGAywF8Be9KyL9qj/8G70rIL7XHbeFdCbkDrgrIuP4NAOiN4opXR54zgAoAKnk8XgxgYDx8p23/Apj0Bg+Gq4XGdgCP2p2eCNL/GYD9AM7DVdZ2B1zlkbMBbAUwy+ODFgBvaee6FkCGx35uB7BN+7vNY3kGgHXaa96E1tPZxvPtAVf55RoAq7S/wU49ZwAXAPhNO991AB7XljfRfrjb4Ap+ZbXlKdrzbdr6Jh77elQ7p83waF0Rz78BeAd5R56zdl6rtb/17vTEw3eawxoQETmYE8rkiYgoAAZ5IiIHY5AnInIwBnkiIgdjkCcicjAGeSIiB2OQJyJysP8HvNm2EgrrMM0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(y.logits[0][0].cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "15d3c8b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2286fe85bb0>]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABZYElEQVR4nO2debgcVZn/v6eqern7TXKzkX1jCVuEsArINqwqgqj8RMVxZhgd9x1HHXAURRkdddRxXHBfUBRBWYJIkLCGAElICIFskD03ublr316q6vz+qDqnTlVXdVd1V3ff2/d8nidPbldXV52urvrWW9/znvcQSikkEolE0pwojW6ARCKRSGqHFHmJRCJpYqTISyQSSRMjRV4ikUiaGCnyEolE0sRojW6ASE9PD50/f36jmyGRSCTjimeeeeYgpXSq33tjSuTnz5+PNWvWNLoZEolEMq4ghLwS9J60ayQSiaSJkSIvkUgkTYwUeYlEImlipMhLJBJJEyNFXiKRSJoYKfISiUTSxEiRl0gkkiZGirxEInFBKcUfn92FTF5vdFMkMSBFXiKRuHh6x2F87Hfr8J9/fqHRTZHEgBR5iUTiYjhXAADsG8w2uCWSOJAiL5FIXBAQAICcNK45kCIvkUjckEY3QBInUuQlEokvMpBvDqTISyQSSRMjRV4ikbiQbk1zIUVeIpH4QmXPa1MgRV4ikbggRMbyzYQUeYlE4kJKfHMhRV4iaXIopRgYLVTwuRo0RlJ3pMhLJE3Ob5/eiRO/8AC2HBhudFMkDaDmIk8IuYQQspkQsoUQckOt9yeRSNyserkXALB531Co9Q0ZwjcVNRV5QogK4LsALgWwFMD/I4QsreU+JRKJm7SmAgBGC0ao9U3TEnkqh0M1BbWO5E8FsIVSuo1SmgfwWwBX1HifEolEIJWwRD4bVuRtbZcBfXNQa5GfBWCn8HqXvYxDCLmeELKGELKmt7e3xs2RSCYeLRFF3jClujcTDe94pZT+gFK6nFK6fOrUqY1ujkTiYiSn4123rcarhzKNbkrFpBPWZR4+krftGqn1TUGtRX43gDnC69n2MolkXPC3Fw/gkZd68dUVLza6KRXjRPJmqPVlJN9c1FrknwawhBCygBCSBHANgLtrvE+JJDbY0H5lHI8CTWrWZR6645XKjtdmQqvlximlOiHkAwBWAFAB3EYp3VjLfUokccIsC2X8ajyXaunJT0xq7slTSu+llB5JKV1EKb251vuTSOLEHIeRvGlS/PCRbRjKWqNcmWiHtWtkdk1z0fCOV4lkLMMEb/xIPLBy8wHcfO8m3HzPJgCCyOtR8+QbwxXffQz/tWJzg/bu8Oyrh/HdlVsa3YyqkSIvkZSAl9sdRyrPInZWr4aLfD6kXdPgEH7dzn58ZwyI61Xfexy3joGbTbVMGJH/4l9ewP8+vLXRzZCMM5jcjSe7hjWVaTUT+eGcHurzjfTkC0Y4SyksumHi1JsfxN3r9sS63fHEhBH5Hz+6HV+9f/ymwUkag5Nd0+CGRMDbVBaZj+TDiTzrh2iEX3N4JF/y/cFsAR/8zXPoz5RejzGU1XFgKIfP/2lDHM0bl0wYkZdIKsHk2TXjSOVtWAoki8xHcvFm1/zs8R3YciBc0bOwHCoj8ht3D+LP6/Zgw+7BwHU+8OtnuZfuPNVUfsca7zNkSZGXSErAotrxpPFx2TWl8uQppbjx7o14/f88WnlDfeizRb4tqfq+r5uWnVMwg22dv6zfG6uXPt4zSqXISyQlcIK4caTyHpho5/VwfjcNkULJhC9sWmZYDg7nAACdLQnf93WDuv4vhxFDppBe4obCME2Km+7eOCbLX0w4kZcDPSRRcDpeG9qMqmDnfNhOzTDZNWGErxJYJN8VIPLsO+hRv0sVl32Yr/rC3kH89PEd+OBvnq18RzViwok8GyAikYRhPJY1KBju6FUPIfKD2QJ/P0wgVKtgiYl8R9p/MD7brx5y/3Hci6KklIZtVz2ZcCJfyVyXkokLGxg0jjSeR9lMm0wu8pT/7eWEmx7A+3/1rGv9UnJVKzFjHa9Bmy9wkY8WyVfT2ig3tLHYRzvhRL4/U1rkTZPiGw9s5t6gZGLDrtlxpPE8kmetFwW5VIflAy/sBxAucg26WVTLoB2EBd1EmE1TCPDkvYLMb1hVqG+Y78o7u+3Xv3t6J/YPZiveZ5xMPJEvE8k/vaMP335oCz51x/o6tUgyluEDXsdRKO/tlDQFgfMTR6+/zUTN9BHGp7Ydwvwb7sELe4NTGKth1B6VawTcjNh3C4quvZ3LcdhKYZ5aiB0GUEpxaDiHT/1hPd79k6er3nccTDiRL2fXJOyyrIdkJC/B+CxQ5vXeXZG8T4aNKPy/euoVfPshK8fcTyD/tNYaOfrktr5Y2uolY4t8UPYM+y5BHa9FIh+DXeN3s/Mippuy49Y7NDY0ZMKJfLmRcmyChZGQdT4kzc14LDXMRN7ryYvviYjC+Nk7nZGhfiKfsUfNpuxgKO57X6bAIvkgkQ+2azJ5HXetc89J5Ng1lbcpzNMAe/CgVLD4xsg5U9N68mOJdEJBtmDixX2lR+ixu3Ym5MARSXMzHgdD6Z6OU7GTMm+LPKstn06oyBn+AY2fuLFRs1zkY2mxw6h9EwkS1kIJu+bGuzbi98/sci2Lo9haqGwjYaKVsTZ9YtNE8odH8rjv+b1l13ty66GS77PrIRNygoVG0zuUwyuHRhrdjKbFicrGj8p7rQzxJRPJoz9/P069+UEAwYOkDJPiuyu34J71znU1WnAHP3EfF27XlOt49fHs9wyMFi3zG727sy+DH63aFrpNYewaQ8hoCjtQq140jci/+yer8b5fPRvouZsUSKgE2w6OYN9AcK+3wSP58SHyZ3zlb3jdrQ83uhlNi9Px2th2RCFvuDtOxU5MUdAHs3rRMhGDUty6YjPe/2tngA+L5MPOMhUVp+O1nCdf/L6qFMuZaKMwrvvJanzpnk2h+93CdLzqwtiEuCtpVkvTiPy6XQMAgtOdTJPi6BmdAIBdh4OHHrMLIh/TD3X/hn0YKJO2WQ1jcfBFMzEeO151PirUFnnhFPH15APOdb9riXnyrJxB3EfFieSLs2RGcrpQ1qC4zZpPx4mfXTM4an2HoDRML2FSKPkTA6XCDWpsXJtNI/KMoDxgg1K02kWPciVqeJTT9rDDqQHLSnnvL5/Be3/5TOjPVMp4r5Q3VhmPpYa9I1wNH09eJCiS9wsgWCTPJgWP895nmpRv1xvJf+5PG3DsjSuQN4LtHNVP5H0GdrHVwk5sHq7MgxjJj61rselE3u8xj1IKSoG2lNXPXOpRs9RIuie2HsLiz96HtTv7Q7WFXWQb9gyEWr8aSt24JJXjTP83flS+wAcMFZcp8EuhDBJ5vwiWCSO7hggI/vDMLvzkse3VNRru6Qm9Iv6b1a8CAF6xC4D5iXxCLf6NKKWu/wHnZpAJWV8/UpkHWn407lC2gJUvHgi17zhoOpH38+rYD8Ai+VKV80r9Po+83AsAeNT+vxxMeMOWeI2KeBEOynINNWE8evIFz6hQw6RcAP2izFKevBevXQMCfPz36/CFP79QcXt39mXw4r5B7sdrCoHhaefktiQA8Ow4P9vJz5P3E2hmvY3mDZgmLZvPHkXkKcpbqB/+7Vr840+fxl6fjuJa0BQiL4qdeIAPDucw/4Z7cI+dddOWjBbJey2QpGodrrAlW9l+auWkDGWdm8dgVqZ81gKWlTG+smu8dg1F2h7/4SeOuQAL0k/cmLizqDuOo3L211bikm+u4n58R1orEkom8lsODAe2LeGxa0yTOqmNwursXpDJG/j2Qy/jlJsfxIESJQjCiDy3aygtm12z2b5R1asiblOIfJ8wwEn0H5/cZqVL3mHnzrawSL7ErPViupTXAknaucFhrZFaWyiHRpwIZFBW16wJzsxQjW1HFFi0njdMnPdfD+PZV/u5yFfjyYvBVC6En712Zz9+v2Zn6HYzK6gjnQisQcPweyLxevIF03Sya4TlLJLP5A3cv2EfAGBviYy7cCmUQiRfpt+O6YLm8+RRC5pC5MVCQOKPz/y7OZNbAQBtqfJ2jXgXHswWcGAoi0/fsR7ZgsFPIj/xNkyK3f3ux69apZkxDgs3N2nX1Ab2NDeePHlnVKiJ7QetMRQtJSL5IJH3+vdiRyXPrilxWN703cfwyQg1oNyRvHvffZ6R6n5C6m1LwaC+lpPK7JqCzgW/r8S0g2FyLcTKn05U779u3g4yS828FSdNIfJT2lI4ed4kAO5UJnaCd6atCQhaQ9g14l07kzNwy70v4vY1O7Fi4z7uGfqJ/K0rNuO1tzzkysEX1wtbtc/qJA637qFh58QcknZNTRjPnrwYsKQTwVZjkMiL5++rhzJ4cNN+/lrseI0L5vd3pDWY1LlmdMMsGv/iZ3V4o3vdMH2rUCp2sDaSM/jvemCoOrtGHHRVruM179MhXkuaQuRndKXxgfMWA3BO8K8/8BK3adgJ2ZJQQUjpR03xEVV8tNUNyk/CgdHiu/7fX7I6Y8USxeLN5HDI2eWPu3EF/vGn4arXuSJ5adfUBBZtjUe7RuzwFyN5r7gE5cmL18I5t67Eh3+7lr/mnnyMx2U079g1gNPx2z9aKIqKCz4C6b1Z5YXv6ptCmRdEfjC48zWaJ188aQtg3cBe3GdV7mTv12hyrSKaQuQBQLOzB9gP8pCQosTEVlUIUpqCbMk8eUHkdZPf9Q1KedGywyPFgmr3yQZ6+geHw4n8SN7Aw5vLZ+/87PEdeGq7UwmQDfCQxAtPoRxHoTyzMsSnO8eTp0WWTdhEAhEmyHEeFdGuAZxr8bCPleJn13hvVnqAXSN68uy6OVAiwyZMnrwhiLzfTeEjv11rdy7r/P0wXn8cNI3IM7+c3SUHRgu48jWz0JZUuZeoKAQpTS1p14g/UE43uX9nmpQXLfOLytmJI0Y/4n6Gc9Ei7VKWDaUUN969EX981qm4JyP52mD65FmPdfxS+FjSQUE3i0S+kmH4USbwDnvsRuzri9mr7Huw2aKO6ErzdcOkghZcdo17OQBkCjp/8i6VRhllxKu4ffEG+MwrhwEAw8KNN47iaWFoGpFP2KG0YVLkdRN7B0YxZ1ILkprCow6VEKQTCnIlTlC3yBs8kjepU37Yb3YpFullhRLFYiSfyRvY2ZdBLiCzJ1swXCdpqZPOO1IvnVBkx2uN4PNAjx+N943MRbtGFEhKaUVZYOwzYvRcrnJkObYdHIGqEMywxZzlyrNze1qnI/J+k4oU37z8I3l2gzo0nOdPD7F58kIKpfgp9iQlpjrXK3BoGpHnkbxpYk//KExqZdUkBXtGIdbBDptCmddNbsMY1PHk/SN56//RgoHRvIE/Pbfb5f0Pjuo4+2sr8bHfrSv67NbeYRz9+ftxu5BuxvKB/fBaMx3pBEYLBh7bchA/WrUNBwazdevUaXbYhTieDqdfJJ92ibwjhgWDVmTXsHNbFPCgJ4Kg5dmC4TpPX9gziLmTW/mgRdaByYR4ip0rb70XLpL3uw7YDerVPiv7jhArEy9IdEMVKBO8f799stLM4hN3veqYNY3Is+JEhkGx87CTOpnUFB5dKwpBuoxdo3s8eWbX5HWT1+3I+TzyqoLPd8/ze/GR29fi5f2OUO+z0zxX2Hm5Io++fBAA8MdnnVrYOw4FF1HzZhq0JlXkdBPX/ugpfOmeTTj1y3/DrSs2B35eEh4eyY+RYlNh8POrmcjkPaKum2ZFxfj86r4EPRF4r5Wv3LsJ82+4B0d//n588g4n6Nm0dxCLprbxgI2J5YgdXE0SRN7vxlE0I5ZBfX1vdoPaZYv8JcfOwKGRPF4WAit2TQJW4KcbJk+z7B3K4c7ndrluCmKpYb/6WSn7Jiv2k8jsmoiwgQW6SbG33xLUWd0tSKoKPyGZXVO6rIHbk2d2TbZguGpdZDwzR/Gh0gWDWy2iGO+zhzCzrW/aO8irUz6/26ptMypsc/O+QTy17RDe/ZPVeHyLc8J5twtYj+JeC+pvQrqbpHJ4SZLxo/GBA4WSqlIcyeuVRfJ+0W1QxO69ifzfI04td7Ff6dBIHoumtvOAje2DXReTBZEvFaGL+/VrEnuS32OnO192/EwA4NfZ7v5RvOPHT7n29fHfr8NJX/wrTJPipj9vxEdvX4dNe50JiJxIXrBrhJOGR/LCtSs7XiPCsmt00+Si3pbSkNScjldVIUglokXy7HfI5HUeyQNuQQacVLLRvIE+eyTqiHBT2GenaLEf/tJvrcKV//sYNuwewLN2p4w4a9XPnngFb/vBk3h4c2/RhMCiyCvE6lTzev1+Ffkk0eEdr1VsY0//KL5y36bQYyWqxU9sNYUgoRLkdbcnXzBN5HWTj+auBnazyOmG62kiSlXGhVPbeA2ap7ZbI9bZdTepVYzky99kxDx5hmHSos8eN6sL0ztT2LDHSnH0zjdhUoq77LltDWEcC2sf4PQfiIOhRNg4BTGSlyIfEU14xGOCl9QUV8erohDbkw+ZQmmYfFujeROZvM7Tu7wV7MRIng1SGs7p6LArX+61R8OaQorVtt4RvP5/HsW2g6VndsobpuuiEUVetS0obyQ/nlL+xjJGDKH8R29fi//7+zas3dUfT6PK4CcyikKQ0HwiecMSeXaeVgPb7lGfux/X/siJhKOU5+5pT/Fr+aO3r8Pq7X3IFHSkNIWPWAf8Kz3mDRNXvmYWfvevZ9jrFHe8+iU+TGpNoKslwTNfvPNAi5pgmJTXwBItHXHCcH+7jHW8ip68FPlIMLtG7EhKaQpSquLkyROCtKaUHAzlzZNnj4CjBQMjeQNT21MAiu0apqmZvMFTvkZyuvU0oSqu2hjl6lizQmgiYpTvFflUQik6eWUgHw9mDB2vTPzqGcnPmdziWqYQy64ZGC24bA3doMgbJloFAa0U0fYRx3BESdFsT2mup9DN+4eQyRloTapIa4LI25Hzc68e5se1oFvVNtlTvTgYiuENhhRipWy2pTT+5O3NbBNvmrpJeeLFlt5hFAwTz+8a4PsxTFqy1v1QVrRrnPd39mVKZvhUQ81EnhByEyFkNyFkrf3vslrtCxAHQ1nCTIgV3Sc1R+R5dk0pkXcNZjL4BTGULSCvm+ixRT5bMJAtGNi0l41iM/lyVjhsJGcglVDQklR5xyvgX8d6ybR2/vfsSc4F2tVi5QyLXp5L5Ik1wGvIU85Y2jXx4Ng1lQt0vZ+qdINi+bzJrmWGSXFoJI+71u7BTx/fwZfn7UjeL7CISlAHbl4Pf+za05prhqe+4TxG8jpakxpSCaeNuknx1LZDuPJ7j+OH9nytecOyndh38Xa83rV2N/fjZ9ppmi0JFYpC0JbUkMkbeGn/EH7+xCuuNonlIQyT4rDdl9Y3kseX/vIC3vCdR7Gt13oaLximbxVKdgMUM+PEtl132+qqyjWXotaR/H9TSpfZ/+6t5Y40YTBUXjeR0hQQWwDdg6FKd7yKdazzusl/HFZ8bJYtwJm8gX+/83lc+q1VODSc49vM5HX02XbNSF5HWlPRmlRdEYXXzweAY4/o5H+zfVx72lz88p9Os7clpmN6InlNxX6Pjziepqsby4gjGaulXn23BcPkaYgM3XSi2lXCfAi6YeXJJ7XgSP6dp8/DjW9Yyl8HnVp53fS1KqJE8h2phCtA6RvJYTRvR/IJMZI3+Sjy517tt/ajm0ioCg/4vCmUH/7tWn7tLehpAwBu3balVIzkdLz9h0/iBTtwY4gFEHuHcjziHsrqeHyr5cv326VOdMOpXeN1BQB35di3fP8JfO3+FwFYncapGG60fjSNXaO6PHknMklqCn8ssrJrSufJG9TfrmEVLefaFS1HcjoesevVrN89wO2STN7AQcGuYZG8iF+Jg7lT2vjfzBKa0pbknxWjf1HkNVVBOqG4bgLi8ZBUB9OnagSa/RL1ytDRTQpNIVjxkXNw2fEzrGVC8NIqiGXBMHkEHMRbl8/BFPucBOCyTUQKBvW1IiPZNWmNizRgBVcjeQOtKc0l8gXDmc6T2Sw5+3uwgZF+efJsMBITefZ+W5LZNcXXDcunB4ALv/F37Owb5Tc6dgNgv61uOh3bom3DavZ7a+R87+GtAFD2N6iGWov8Bwgh6wkhtxFCJtVyR5rqpFDmdJPnpYoHTlVYCmVpT54Qy8/P6SYvC8pyZOf3WCL/vl89y8V6/c4BfjM4OJznd+2CQXkkL+LtvQeApM/UZZPbkryzSewD2C94dwohvFNHRGp8PDhlDSrfBhOEeo1wNEwKTVVw1IwOnHvUNADWufjop8/D5LYk9guec8EwMZrX0ZYMjuQ1lbgm5PAGLYy8bvo+pUbJw29Lqa466zv7RjGa19GaUHkaIvuOLL05kzdAqVWTJ6kqSAj9c94MFtapykSe0ZpSMZIzcPysTnhZ5zPd56Kplr3Kbhrs+iwYtGgidcCJ5PcH+O5xZTj5UdVWCSEPEkI2+Py7AsD/AlgEYBmAvQC+HrCN6wkhawgha3p7w02r5wfPrfV4jKLXyGrXlErpMkwKlVhefk6I5BnzprQVfWbtzsO8Q2fXYfcgplRCQWvCnbngN+2Xpiq44OhpSGkKjwA6WxL8s+wk2j+YxZPbnE4tzbagvMjsmniIo3YNK8dbL7tGN01+PSSE1OLZk1qxeFq7K7rVTYrhnMHLcPthdWY651hLIiiSN4sSEqzllvCNhJgGM6WpLk9+S+8wNu8bQlvKY9eYjpU6ktOhm9Y8zklVQUJztMB7f2H9WQunuq/jtpSGkZzu+xv5Zb/N9+iA+KTNbmrioCgWLB4c8i9UGFe/iB9V5U1RSi8Msx4h5IcA/hKwjR8A+AEALF++vOLrwMmTt1IoWSeNK5InVs+7YVKYQiQgYlDKq1WyTimRebZdI/LygWFuAe067BbwlKYW+ePeyUUAS6x/dN1yAMAHfv2c1V6FOHaNfYHc+/xeGCbF9M4U9g/meHZNEeNo8M5Yxq9UbWR4JF91c0JhmJTbdXyQoB3YTBZyzQHg+p+vgWFSHDW9HUGIPjfg5Hx7yen+Iq8bJj7++3W4a+0ebP9K+fwL0WpUCcFg1up4FfcrZtGNFgxuCSU0Rci0M4sieVbRsqslgZ72JN5x+jwAll2T001kcgZOmT8Je/qzvtcpwxtDid+bPc1QCq4zTPiDMuvGpV1DCJkpvLwSwIZa7Qtwn8x5jyfPUIhTyMxv6DFgdbyyzsy8J5Kf3JZEp53tAgDnHTUV15+zEHv6R3mOrfemkNIUbtccM9N6FNzZV1yyIKFaHcWEEP4oOa0jzbMFMgVWSCmHpKpgYY91UbI8eS9BhdAk0YgnkreoV2kE3SXyTickAExud4v84UzBEtESefKaqrii63SJSH60UBytFwyTDyY6VGIGJmd/zr7OWDQFAIpSKA2TIm9Y5/hIzinul1QV16TlXk++347kU5qKNZ/7B3zkwiP59q325ZBOqEV9Wos8kf+7zpjnei2KvDgOh+lMqVHFhmm1s1YiX/0IiGC+RghZBisI2gHgX2u4L6gKASFOCiWzMFwibw/tBqwTwO+8ZpF8knvyzo+TEjp1AODjFx2FTXsHrVlsAkQgoRIcsB/RLj1uBjbtHeSduCLiif2hC5bg5HmT+AneklTRn8nj4v9+BAeHc+gQOqe0gEg+WzBxz/q9SCcUXHDMdP+DJilLHNk19XTOTNu24CIvVGcF3EW+RNpTGhTiPx4goRCXT17KrhnN+w1Scjbqd+57UYV9HTOzE39/qRcJVfF0vDrXZiav80g5UabjlVWQ9T6NtNti0DeSx4Kedtf1CDipzADw75cdjdMXTnG9L1pRYp+fbutMKZF/ab81BmbcRfKU0ndSSo+nlJ5AKX0jpXRvrfbF0BSCgslSKK0TQkxLUhVnoIR3/koGe9RNqgryuoGcbvDIeoZQzxqwIns/j14koSrYaA+XPu+oaUgnFFdvPV9POLGTmoLzjp7GX7clVfxm9U5s3j+EQyN5dLY4aWasn8FLVjfw5Xs34XN/2lC3QTjNCDt0sRzCOvwMLDtM4yLPqrNayye1+ot8a7LYVmRoQnQMlO549RsDckiYLe3VvtKju8W2J1UFx8zsAGBNtJ325Mk7Iu9E8inBWtLN4o7XASGSF2FPMoczBaQT7icXwPLsGapi3UjY6HfAXTdHFPm3//BJAKVF/tJvreLftxY0TQolYFk2rKxB0i+SJ6S8XWOnnyU1hefJn72kB/d9+Gx8823LXOtObkvylEoA6GkvvoASmsILIB0zswNTO1K+1fq8kYOI96LqSGu86qXY8arZTzOAlaq1u38UeweyrtGHkmjwSSeqUGgmnvW41bLIlUXDCW5jWufcFJ9zFCgeaSri7XgtbdcU24QsyAGA7b3lRZ61I6U5tuSBoaxrvyxVmsGy39rTGv/OIzm9aIJull3jffptF0b8pjTV9TRhve8IOrsBeDtfGWKG0bpdAxjNW4MqvTcOL34JFHHQZCJPeN6vn12j2kWagOCiSYZJ7bREJ7smpSk4ZmZnUdSeTqiY1uHkDx85vaNoewmF4ItXHIvnb7oImqpgWke6aB3WtiDaPL5SZ1qI5O3cf9Ye9igtXmx3rd0NSWUYQgrl41sP8snho+CkUMbZMn9YZhYTFHaesI7X2ZOKEwcAa5L7YJEv7ckzPzuo4/Xl/U5JDrE8RxBsX6mEiqVHdOJty+fgy1cezyNd1k6xYOBjW6xBScfP6oKiEKgKwfce3opfPfWqa9v9QZG8kF1ULpJnAZk4Sl3EOw7n4HAOuknRHfAUxRh3dk0jUO3MmVzB6akWH4FUMZIvYdd4I/lSB19RCO+UOWqGj8irCjRV4ZMTTxUGlXjXC8J7Ubk8eZW4bmjenPyzl/Tg3uf3yo7YChE93bf/8Cmc918PV7ytenS8shHbTAgXTbPOzTefPAuAJYJ+tKVK2DWCzQkALZ4omAlgwaBc5DsFK2P7wREQAlx4zDQ88EJwCWwWMClCJK8qBF+9+gQcZ4v3jlsux0326NveYSfn/Ilth9DTnuQlQRIBT8YDAZ58m0vkizte/SJ5sfyIiHeswB47S2dSa8JvdY4U+RBoipVj7o7kHdEjQnaNXxU7wBlkkbTLIegmRVItXbzpnCOnWvvyEeqE54eb2uEv8qUe5bwnTWc6wS9IhTgdr0nVPbp2UmsC1542D4NZHRt2u4dqS8IRZ558PWDnNROpaR1p7LjlcrztlLkAgoWkLakFDqBTy3S8ssDi8a0H8cW/WPVXxCy0wayO6R1p/PPZCwPb/ZELl+C+D58NwLHIglI12eQh+wYcr3/19kNYNqebjw9JKP6f7R8tgJDia7XNZdcUR/LtHk8eAGZ0hRN5VpxwUkCnN6OczlRKk4k8gW6Y7kg+wK4JKppkUDuSVxUM2z3mXv/urve/Fne9/7X89b+cvRBzJrfgzSfPBgAcLUT0Cc/JMrPbsmtakyru/dDZznolInlWnpRNZNwhFHHShI7XVELBP565ACfOtqK1GV0t6GzR7O9bp7nGmgwWC1TT8VpPu8bx5INvLJ+7/BhXXxJgReMsgv6Xsxfgzn87k79HCHFFxmnP02JSVaAQYJVQetd7I5g9qQWnLXAXTRM57oguXjqBee1+CQWAk+sv1pTJFkwc0e2Irje4YvRn8ryulUh72h3Je/vI2nwi+QuXTvMdKewtFrhnQEbysaGpRIjkbeErEnknvcoP3Y7kUwmV14X33vVPnNONE+d089dHdLdg1afOx5HTO7DuPy7C7defwd/zivcZdupVJm+4HwFLdLyyjIAltuffltJ4NMEGbrHv+p6zFuANJx4BwHpqYBFYvSYoOOdrK/Gu21bXZV/1wKkTXv3xq0vHqye7xo9/Pnsh7vnQWa5lbSmVd+a/4/R5eM1cdxUS8abhHVClKqToJqh5zvvZk1pACMGsbv/oVzz/F01tx/wprfgPoSiaCIuI93rKg4jXetD3N6l/x3F3i/OdLE8+uGNWfEra+J+XFAm9ODEIAD5TnbgPP8ZjnnzdsSJ5ilzBya4Rf3gxuybIrjHtsgYpzYrkFQIcP9vfx/SjqzXhqsTnjShOmN3ttFc4sb0nlQgTedEDZNeQlRbKvqu7Xs/U9pQzEXmd0ihf7cv4poiOV3h2TRxVKOtwo9WN8pE8UBx8iJG8V6C963tTiVWF4OgZVuYYi+a9njjr8D16RkfAiG/BDkqqePiT5wW2neX6Hxx2F/sSxbvUk7FfFososFZ2TXCk772BpBKqVUQtqfp2PLMyJt1tZSJ5mUJZHk1V7JFwjl0j/vBinnx/poD/WrG5aL5UNlrwLSfPxtuWz8HP33MaTpkf/Jjph3iCeE82VSF437mL8OELlrhEPqijCACuPsmygVgte5NSVyTPnkrYd2adS1M7Uty7r5fINxssMq7m8DUihbLUkyFQLCjtKSct1++TorBN7ywW+Xs/dDZ+YZfFZstEWIDy9tPm+ranXHtFgrJURPEudT0FpYAyB8c3uyYpPnW7jx3br5g3Lz6l77Ej+aAxCgxp14RAUwhy9hyWKT+RJ86I17vW7sF3Vm7B91ZucW3DtEX+tIVT8NWrT8BZS3oit0P0+/xOtk9fcjQ++g9HujqH/KInxleuOh7rb7pIKDZFXSlyLGPnXLsDuNeOcES7Rop8ZfBZhyJUUvTCT4c6plB687y9KArB0591Sk+1Jp3o1e9cEc/PGT4iz54CHr/hfNz/kbOLgpv59oDCC46Zjg1fuBjphOLq6C0lyl6SmuI7XWE1kTzg9CP4Zdf4efLe7bEMOgA4bcFk7Ljlcpw4p5tH8uU8eZknHwJVIbx2hhPJO19RzK5ZsXEfAOBnT+xwTbvFOl7jotTJ5rZrgvepqQo60wm8+eTZWNjThrefOtdVm2R+TxtWfeo8vP+8xQCAa06Zi860hsuOnwF2rXvnuqwF9SqlW0+Y3vkNYItKXVIomciHqKUwtSOFf7/saABWnviHL1gCwHli/N2/noGvvvl4AO7zs8iuEfZ1RHcLjp7RWZRFJpb2bU9pSKjuEiGl7Eo/vDV4AI8nHyElmcFEPqWpRdetO7vGK/LW58RInulPd0uCzyQl8+RjQFMV7ok5Ha/+dk1ON3HW4h4UDIrvP7yNryPWqY6DUiLvOslDRDIzu1rw0CfOxZzJrfxEY//PmdzK2730iE6sv+lizOxqqWskX27u2vEIO275KsYZsF82oBsoVrwplOW4/pxF2HHL5VAVgreeMgc7brmcp+GeumAyT710V6F0i6Tf9fLlK4/H5y4/hr+e5hH9lKa4RC2KXQM41oeYxSNe637zM4j79oN9r1RCKTp+4viTYk++OJJntW7Emjc9AWNknDZLkS+LphBk7FFwfpE8q0nDeO3iHrzxxCPw+zU7uTdvmHFH8iUidEWM5KP9FN4RjUHUs+NVHIHYLLCspGpSUJl9V4+nKXYjifMcBoLzzoP21dWScOXFe1MWE6riuhZLBUN+sM5XsUNUTHWuKJK3hTztqWkPeMofB9g1rQmVW3NsnEC3YNEwu+YCoS6ViIzkQ6AphE8F5ufJi9k1gDVy7+qTZ2Mop+PELzyAlZsPWCmUMZYNLHXyujtoo+3TieTLeK/2d6lHCmWYSSHGGzySr8aTt/+vS3YNi+Qjnk/l8Ebrn7z4KBw53Sl3XYrpncURrNeuiTpdJbODRItEjOS919MNlx6N5fMm2esFRfLO2BpvRp2mBD91sP1qKuE3B79IflJrEi996VL89zXLfPcvRT4Emkr4pB28nkuAXQNYPuTpC6fwH/8nj+2AadLIj46lKCXy4iCTUpGHH1zkyzTVO2lELRnxqUA43okjkne2VfUmysKza2KO5L28/7zFPFIvJdDrb7oIKz9xbtHyhEqQ0BxRLPWk4AcTedErF5/avdfde1+3iNeeSpXx5AuGiXefOd81lkC8aXoDK7FAIKuJ1ekj8p0tCSR9RtMypMhH4IiuNM5ebGXFiI9wYnYNYI3cUxWCO953Jj54/mKserkXfZl8rJF8uRuGxqsF1iiSr2PH61iwa/62aT8e3nwgtu0xYa7OrmHbqkckHy5PPg5Yh2upTt7OdMJ3asGkpiIplAWOGlgxkRcLDYqRvJ+QLrDnZw56omKp0l0tCRBCcOwRTjQvfkfvtlnRuuOEukCsdo9Y3kEsKuiH9ORDwCrRfeGK4/ioOPHRjCjuO7w4a/0xMztBqTVrU5xRULkfTqswknc8+dLrlUqLi5uxEMn/08/W4N0/eTq27bHjVl12Tf1+gyjZNdXi7fyPQlK1rFMWwUcWebsTU6zdXiqSB5w0zn0D/pNpf/yio3D3B17LZ3ATEb+jt63s3LjaLmsC+Ns1ftsSGZNzvI41zl7Sg1UvH3R1bIgdPiohIMKTmljMi6WFFQwaaxRUrkOJvR91n0rISL6uIi89eV8aUmo4Zk/eD6UKkU+oCiicdkZ9emaR/LBwzrk9+eLrgqVxesshMFSFuEaki7iTJNxt/dl7TsGe/qwrRdLPrmEEfdc4s/pEmkrkf3TdchSM4BRIVSFQhfF8osjPFHJ/4xX5cnYNCbVe0OfKPXWodeh4zekGrvjOY1g01eqIq9Wgjkbg9eQrecrjKZR1ya4JNxgqTiq5XrpbE8jpJuYtmYo/r9sTObuGibw4Yj3liuSL28Qm+QjKrimFqCneY7t4WgcWT3OXGS8VydfBSXPRVCKf0lTfeVsZCiGuE1LMsZ3anuJzXDYiko+aQimWNSi9nvV+LTteDw7n8eK+IT4hRCUX0ViFCTN7JK8kQmaBWz2epryThtQSI2JOvsiX3nQ8TEoxuS2JD56/2FcMS8FyzsW+EvG887M/21IavvHWE3GynWgRhm9dswwpT0plmGPbmQ4WeW86aa1pKpEvh6oEi7ymKpjakcL+wVzdUihZm6z1Ina8Evfny22/llGktwlBdcDHI8ylcSL56N+N1ZOvT6nhyoU3+r5Q8b7EUbN+M6qVg5UZeNOyI/CntXsAeCrOBlzDV50023d5EFcssyZbEb3/MN+3VCRfbyaUyHt/G+8sSjO6WrB/MFe3wVCAFRmqCol8d1dDevn18OS9m65Hp1+9YDdHVrumGq+7ntk1dY3kG/R7v/CfFyOpKlzkxUg+7hISrkg+xDnA0iFbkypOWzAZ7z5zfuC6l58wE19/y4lVtzGICSXyXiH1Tn4wqzuNdTujdwKVoqxdowTnzZYi7IhX9l30Woq8Z9u13Fe9ccoaVBHJ8xTK2JoVSJhJQ+LbF+q2Lz+8qZliJB/3/dSVXVPiHLj9+tPx7Kv9/DUhBLf/6xmB61998mzc+IalNbU4m+e5ugK8s9dctHQGAOCp7X2x7aOcyGsqidzpBDjRU7kbBHvfK8Rxwi6oD12wBG9dPrtuE5TUA/ZdqomQuSc/hurJV8Ks7hacuWgKf80i+VplhURF/G3iPtKuLL0S3/e0hVPwvnMXhd7utafNddW8qQUTKpL34hXXS46bAdwOnDgn/CQh5fdRzq5RKrIAyg2s8K5XS4FhQrigpxX9mXxTlTX2HrZKxNPx5OuQJ09rJ/KP3XC+e191zMkPgyjEtTzUcVphlTwZRt5HzfcwjkgnVKz53IWuodLVEjTXJH9fIZXZNWq4SJ4QAkJq7clb21YIgUJIU9k13qeSijx5ZtfUcTBUPcSjnqNro1LLss5xft96ZLpKkfdQrhxoVMrV5LCKGkX/pVkEH+ZRWVNIXTpeiZ2iWg8xqxdFIl/FBV6Pw1JP4WWHph6dvJXyqUuOwjWn+M9GVSlxft96/E5S5GtMObsmUaFdE3YwFGDdEGop8pRH8vYNpak8effram7IcfZV3HLfi+hqSRT5v4ZR+aCtqLz9tLl4cd8QPnD+4prvKzL2oZ7WkcbkttKTdUQlTmGux+80IUQ+qSpVDUuvhnInhKZU2PEaYUi5WqdIXiHWNHD1mByjXnifSioaDMW2FaPI//2lXkxpSxaJPI/k61DWoC2l4etvrV3qX1i+eMWxeOWQe/J4dqRrcRTiHMwUZyZfEBNC5Fd+8lzsPlw8Q3w9KHdCJNTKUigji3wdOl4VYnXC6U2k8mPVrikYpu9MXGOtM7QevPOM+UXL2NPlWD8MsuM1JmZ1t2BWdwt/ndIUV92aRvKuM+bj0Egu8ufUCHZNnJG8YVIMjhZ4lU/AEULCInlqXWT1Hr5dC7yHrZLvVIuyBgXDxGjeR+RrmF0znuCR/Bg/DLLjtUasv+miRjeBc9aSnoo+xyKAene83rpiM77/961Y9x8XocuezowKdg3Py6flJzQZ6/ilPFZyFNlhiDOFUjeoa6g9wzDqN+J1LMMONamJYRMf9bgZT8jBUFYhs9pG8q+Z213T7bMIIGzHa1x+8IqN+wAAfZk8X8YjedS3tLGXuPPQfb9CBfsgxLnxxUU+wK4Zy2mN9WS8RPIyu2Yc85t/OR25Qu28aR7JhziLVYXEVoXSEXHnu/GOV8VpTyNEPu59+t0Yq9lFnB2vBcOEj8bDMGlFtZCajX86awFWbNyHM4QRumORevSdSJGvEemEWtN6FNyTD+GJxNnxyp4cxAFPoifP3m9EGmXcg7D8RLmaQTZxjh8o6Kbv99XNeCe9Ga8sm9ONl750aaObURZp10gCCTvHK1s3LoHxq08vevJKGbtmZ18GX7v/xZpE+nFv0+8+VUniELOR4s2uocjpJjJ5Ha8cGuHLDdOcUJk14x0p8pJAeBXKMHZNjKUG/CJ56kqhtJYFCe4t972I7z28Fate7o2lPSJxPz34R/KVbCd4e5VAKeXjPr754Mt43a0P48lt1vzGhik7XccTY17kCSFvIYRsJISYhJDlnvc+QwjZQgjZTAi5uLpmSrxM60whpSmYM7ml7LqqEl/HKzspxRl5xMFQ5TpeZ02y2iuWY40LI+bZr/y+QyWdu+wTcUXy4g1212FrENC3//YyADuSH+9pTROI8dDxugHAVQD+T1xICFkK4BoAxwI4AsCDhJAjKaU+XUWSSpjWkcbmkJ5jnHnyrMM3pzs/pSkMPGH2UdBNZYqdX792Z38s7RGJ35MvXlbJvdLkdk087SsIo7fZ79qfseY61U0qI/lxRD2staoieUrpJkrpZp+3rgDwW0ppjlK6HcAWAKdWsy9J5cRZu8Y/kneqULIKDUH7Y0K89cBwLO0RibuOvV/UXtE+mF0T029Q0J3tDI7qAICRvPW/ITtexxVj3q4pwSwAO4XXu+xlRRBCrieErCGErOntjd+nlVgZOLFF8rYVkBNE3tXxWiaFspaplXWJ5CvaTrwdr2IdpsGsFcGP5KwnKyuSl11t44V6pLqWPRsIIQ8SQjb4/LsijgZQSn9AKV1OKV0+derUODYp8RBnjXcm4v6RvHMTKBfJ14K4PXn/PPkKPPkYO14f23IQA6POQLSBUSbyTiQvNb42HNGVjnWuiXpRtsWU0gsr2O5uAHOE17PtZZIGEGfHK/N73Z689T8RI/mA/Rk1LF5Wj+yaSkJ5lltf7W+wp38U1/7oKRw3q5MvG7RFfrRgwDCpjORryKpPn19+pTFIrc6GuwFcQwhJEUIWAFgCYHWN9iUpQ5wdr6U9+fJlDVh+fW3y5OO9gfjmyVcg1HGlUGbsgmQbdg/yZUN2BA9YQm9KT75mqAoZl8e22hTKKwkhuwCcAeAeQsgKAKCUbgTwOwAvALgfwPtlZk3jUGPsePX35J2OV62cyJvRo9pNewfx+T9t4Nv8xO/X4YSbVgRuOy7iypN37Jrq2lNq2wBw3/N7cc/zezFk+/QSCVB9ds2dlNLZlNIUpXQ6pfRi4b2bKaWLKKVHUUrvq76pkkqJN5JnKZRCJG//6dfxumZHHw6POB6ywUU+/D5vunsjfvHkK1i78zAA4I5ndmEwq+ORl3pdlRjjr13jt6wST97+zhHbt+XAMJ55pY+/Fp+e/Hj4JStx4crXzI7YQkkzI827CYCqELywdxAv7Bksv3IZ2NNqzseuIYJdY1IKw6S4+vtP4F23OU4dm1AkiljOmdwKAFi7c8C1/F23rcZn/vg8fx27yPsOhoq+HWcwVLQPX/iNv+PN//sEthwYhmFSZPXSD8O7Do9iclsSN1x6dPRGSpoWKfITAFUhyBZMXPbtVVVvi+le0IhXRSh7wDoFX9jr3FyMCuyarharbv3T2/uK3rvn+b3877rYNZWIfJUplBd+4+/49t9eDqxqyuYR3tWXiX0+U8n4R4r8BCDOziLWuZn38+QVJ/vGNCn6bZFPa85pVqig45XVTd9+cKTovbxuYsAe7RlnlUcgaMRrFR2vEdrn3c+aV/pcGU0i3a2WsB8ayWNyqxR5iRsp8hOAOIdOM3H2S6FUCOH7MkzKc7jFksvs81G0kk1zV7BvMB1pd+bvngFr/l6/omnVEFc9+UrsGlamgG+DAtmASL7bftIBgEltCd91JBMXKfITgHgjeUuoglIoxVLD/fbsUSkhkq8ku4aLvD3S0/tRFumLTwdxBPX+0/9V0fFa5qM7+zL4+RM7rL/twmMiQZH8JCF6l3aNxIsU+QlALUR+5+EMFv37vXh6R1/gpCH+kbzp2k4YMraI6waFaVIM53QsmdaOz152DAAgm/cT+Tgi+XDLyhF2xOt1t63Gf9y1EQOjBew6PFq0jVxAdk13qxDJS7tG4mH8jdGVRCbO8hgsEn9ym9UJ+vs1O3HmImsycu+kIUzkDUrx/K4BHD+7iw+GiqLBWSGSZ4L/1uVzcPpCa2q3TM1EPp6O17BVKAez1sCmbMHAvoGse7+gyHnm+0uqCvKG6RJ5GcmPD57+7IX8CbXWyEh+ApD1mwy0QrwR+PyeNveIV+KkUDJf+ZVDGbzhO49iKFvgn49SgiBTsMQvr5sYtoWwLaWhJWmdvn52TRwVDvwG0FZUT553vAavM5LTkbSzZIZzOoaFkaxsG95IvrPFitFahCelzhbpyY8HpnakMHdKa132JUV+AhDUYVcJXpFXCfGdNEQ3nEie0Z8poFCFJ6+bFMM5a5vtaY3bQOL7jKiR/EhOL8p+iW3Ea5naNYZJceyNK7DHjt5HcjovOCZSJPJpS9CTmoJbrjoeADBnUn2EQzJ+kCI/AahlJJ8tmIGDobwZIoczee7JUxo+KhY7XoftkrodKQ2tSSuSrbbjNVswcOyNK3DzvZtcy9n3Evs0qqtd4//+4Uze9Xo4p7tq0ojtFOmwo/aEquCaU+fiic+cj9MXTo7cPklzI0V+AlCNyG/cM4D5N9yDZ1+1Sgp4bZbRguGqXaMKg6Fe2j/kWrdvJO+aAJyJ3nBOx/wb7sEvn3zFtw3Mhy8YlNdlaUtp3KbgnjytLJJnN5Hfr9npWs7aJ4p8RTZQiY7X/YNZ/OGZXa5lw9niSJ7CL5K3bnIJe6aWmV0tdalPLhlfSJGfAIxWIfJPbLUmiL577R4A7mhZUwiyBcM94tUWmdse3Y7nd7vLEDy8uRcHhnL8NRO9A4OWTfGjVdv82y90ULGng/aUhnTC68kLA7QiOFRBusgjeVJtJB9s11x322p85b4XXctG8j52DUVRxyvz35OavIwlwcizYwJQjSfPsjWYpWCYFNM7U7jt3csxuS1pi7zT8cpSKLfY0/u9dblTLOunj+9wjVr12iF+nbGGSZHTTR61s3a0pzQQQtCSUDFqT30nPiVE6djlA7Q8y6mPXVNZPXkLP7tmtydVEgCGcwaGsn52jft3bLftqqQqL2NJMPLsmACIkXzU7BBmBfSNOCK/bE43zj96OtIJ1RXJE8GuGczqmDelFZPbUoHbFqtXiq9FmNXEMkn221E/G9nZklT59zMrtGv404nnI84Tirisikje3uDPHt+BK7/3GAAry8LLSE7nc7YyKGjRYKiWpHXjk5G8pBTy7JgAZAW7oxBxijw2slWM5JmQtyRUjyfvjHgFrMJizFLxwyuYfgLK/HZWpGxPfxYtCZVPw2ZF8lYbK82u0QMieSbKLk8+9FYdvIOhbrx7I557tR+6YaLHR+QtT97ghcfYNryePBP5hIzkJSWQZ8cEQCxRqwvhsl+anhcmLIdH7IFNJuU15dMJxcquMYWOV+IVeRVBMEulVGVKHsnb6YK7D49iWmeKdzBakbzu2g4QrYPUqafjvelY/6vCdHoV1ZMP+GzvcM6V484YzukYyuq88BgjWzCQVBUs6GnD999xEv+seDOQSLxIkZ8AvHW5M93u67/9KHb2ZbD94AiOvXEF7vBkdnhhFgGP5CkF05S0Hcn75ckDVnXEdAkrgXWOskja8LFrWCTPOhl3Hc5gekeav9+aVDGaN7By8wFuKQHxRPKOJy8uC73Zou14Pfl9A1nfUY8sT16sKGlSq2/ixDldWPmJc3HJcTO5yEu7RlIKeXZMAL54xXF8IoltB0fwf49sxQ67A/SutaXnV2eRfCZvzR+qG2IkryIndLwSxW1tdLVoSJWI5L1ZJ379BSySZ5Un9wxkMbXTsTjSCRWrt/fhH3/yNL754MvCtkt+LRdGwEQmPJIn1aVQOiNerT9YR+m+gSwyBR1tSfcxGhgtYLRguMoVFAxL5FOas26aefLSrpGUQJ4dEwBFIa7yvHnd5NFfRogkn97Rh1/YVRAZ4kQVowXDtmus18yuoWIkLwhid0uSv3fNKXNw2gL3QB1m1/DJvX0UlFWebBWEcJrgY7ckVIz4RMNRarezJwjv7ll7FJcnX30KZVvK+i77BrPI5A2cd/Q017FhaaZiHZq8biJbMFx9HI5dIy9jSTCyQNkEISH4yjnd5OLOfHlKKd7y/ScAAO88Y76wriOgI3ndsmvsbbVwu8bpeIXq9uRZlKwqpCiq94qfnzAzK6Ul4Zyq0wS7JhVgVUSJuFk/RVHHq70RzTXiNfx2vW1hn9VYJD+YRSZnoDWpuiwXlkEkevJ5w0SmYGJ+TxtfJu0aSRjk2TFB0ATxzesmMnaKHkvV27Dbf/5XMaNjNO+N5NWiwVCi6Ha1Jng2T0JV+HtMM02PJ+8nzCzKZ8XIAHfa4WbPqFpGHCmUfnnylc0M5baj2I1130AWmbyO1qTmslwODVt9C5MEu2b7wRHs7Bt1RfzsmMhIXlIKeXZMEDRBCPK6iRG7BkwmZ5W1fWLbQf6+LvSAipF8xhZ5TfDkxUieEEtwptueuRXJO0LJRJ5FnuxzpSpTstmgXJUWBeuJieOyOd2uz1XW8erx5O3DoFYZyTMMSqEbzlPU4GgBmbwVyYtCnbePv7eiZEpT8KbXzOKvl87swklzu7FkenvljZI0PVLkJwgJQahyusmjyeGcjtO/8jd8+V5naH1GGDwlevIZHslb27I6Xk1X7RoAuGjpDACWzfGW5bNx9pIeXH/OQt5pyITZK/J+wmzYkbyYitmWckT+x9edgp+/51TM85Rtjdbx6v8kYXq+FyNqNC+WGmY3V8AaMKabtMiuYXg7ZBf0tPFUUgCY0ZXGH//ttehpDx5wJpFIT36CUBTJ2zaN32xDo3mDi4m/XeMMhsobJrdkmBh+9vJjsHhaO1535FRoqoJf/NNpAIBUwhPJ25vmIu+TQsn88hZB8MRO2LlTWjF3SiseevGA63NRhJhPZOJZ7legzNp2tIlYxL6HwaxTmfPQsNXB2pLUfC0X8WYGQIq5pCKkyE8QRE8+J1gGfoi526Jds2LjPowWDCGSdxcIU4T8+evOnF+0XW7XeCL5UgW8mJUiCrtX/IDiGZEqi+TdH6I+Ha9A9FGv4mAocTIQ5r23JVUkteK7RrHIy1mfJNGRds0EwZVdUzBKjnbNuETe5CUEfmGXAlaF0aaA05FYrswti+AT9v88hbKEJ887XhP+kTyDCSK78USZQzY4u8b6n6VQsq8XddSrcxMDLzw2tSPFa8a3JFXfXPd2j8iXqgMkkQQhRX6CIEbyQz71ykVGCzr29I9i5eYDyBXcc4gCjn2R1pyZmcLMFc6EjEXGfCRoiewalifv8uSTxZE8GzR1zMxOa5uVVKEM8OTZTc1pd+hNWwi1a9jMVmKuf2uAXeO9mXXJqf0kFSDtmglCwiXyBd8BRAmVoGBQZPIGPvDrZ/Hsq/1Y0NOGSa1J7BJK4jKRZzeOvGEWdU764fW2WaSsl4i6DdMnkk8VR/LvOH0eBkYLOH5WFz746nMR8+T9VxazhgCr/QWDRo7k2drUE8kz2gI6Xr2RPKvEKZFEQUbyEwRNsGuGcjqGszpmT2rxXSeTN3jt8u0HRwIjefZ/IaTIeyNhnjpZQuTZnLBix6uftdHVksC/X3YMH01aUZ68By7yYJF8ZZeLmEXERN41ajepujrGGd7ibjKSl1SCFPkJguYpW9s7nMOsbrfInzinC4BlfSwQRlaKJREAIZK3Ra9gUCCEXcNGyoodkUBpkdd9yhqU8v7Ze5XkyXvh2T5CJB9124D7puaIvDNqt7MlwQ+f2Mnrje7F9EmJJCxS5CcIXs/34HDOlZHylauOx1euOgGAFcmLk1aIRbEAR4jckXz5Nni9eC5+JUST3QBKlSwW4ROQRNDhoDo3TiRvUaknz7aTLRgYzhWgKcT1dNTVkuCWkPg7iU8sbzl5Ns45cmq0HUskkJ78hMGbBtifKaBV6MC8/ISZvPRvJu/OvvHWK1c8HZF5PZxdoyhuAXbsmuDpCQs+2TUl92E3I1KefIDIs02QmCL5oaxVJ749rbmOfUdacywhlQB2Kn1CiORvfcuJkfYpkTCkyE8Q/AfbOMLZntS42I3mddcco2KddsCxflS1Qk8e7rx4vzryDMNnMFQpKonkg24yXjF32h8NdsPJGyYODefRkdZctXhSmspvJEmfSH5Km8yPl1SOFPkJguYze5AYTSoKQYJY0epowcBwTsfMrjT2DmSxdyDr+pw3ki8YNNQIUNVjd5SqPslgkXxaCyfyleSyB2fX2NuE+6ZGS9yU/BC3vrt/FO2pROCTifg7JVSChz7+uqIZoiSSKEhPfoLglxnSnlJdNg4hBK0JFZm8JfLHHmF1xHpFPq7smjAplLppQlWIq6Z7KZQKOl7LZtfYu9Z4x3F0u4aN9t3TP4qOtFbUx8C+HXviSqoKCCFYOLW9aDSvRBIFKfITBL95QFuTGh6/4Xz89aPn8GUt9nR6Izkdi6e14zVzu3Hr1SdgqT3ICHAm92ailw/Z8ap6Ol5Nn5GuXi9dF2rlhIGJfKQ8+YDJzb1tUZXoVpC1PuXpjweGcuhIaYGRPLNo5LytkrioSuQJIW8hhGwkhJiEkOXC8vmEkFFCyFr73/erb6qkGvzysNtSKqZ1prFkegdf1ppUcTiTR8Gg6EhruPPfXosLjpmOez98Nv75rAUAnNGlTPR0g4aK5FnpXGY/8Im8BVPeG9XrBnVV0CyHUoFdExzJW/87kbz7JhUWCneOu+XJeyJ5tg9b3OVEIJK4qNaT3wDgKgD/5/PeVkrpsiq3L4kJb3YN4PbkGe1pjdsz3vx4ZjGwgVKaYNeUq1sDABctnY4vvuk4zJ3ciutuWy2kUDrrFAzT1UlsmJTfoC47fgZeVyaNkFTQ8SreWCilfBtM/LknX2EkT4VIHrCOsTN1H6uL4x57ICcCkcRFVSJPKd0ElC9MJWk8fqLhHTYPAFPaUli7s9/3fVb8K6u7I/mwnjwhBO88fR7W7OgD4J9CWdApIFjQBcPkN5PvXXty2X04s05FieSF/RuUV4Qs9uTd2UFhodQ9AUhHOsFvmN56PizClyIviYtankkLCCHPEUL+Tgg5O2glQsj1hJA1hJA1vb29NWzOxMbP1/ar5jilPYmBUStR21vq1onkLZHntWtC5skzvKNSxRTKvCefUjeob2ZQEJV0vIqRvC4IvncT3uygsJiUumazak9pvLY+y4V/1xnz8e4z5+MD5y8GEDx3rUQSlbJnEiHkQULIBp9/V5T42F4AcymlrwHwMQC/JoR0+q1IKf0BpXQ5pXT51KlyRF8tufa0ufj5e07lr/3qsk8VJqbo8LzPLAfmF4splBFs8yKxdEfSHpEXphuMsu1K6slb+3f+diJ5t5USVeQpLIuG0ZnWeCXNK+3p/FqSKm5647GYbPdXyEheEhdl7RpK6YVRN0opzQHI2X8/QwjZCuBIAGsit1ASGzdfeTwAS6TzuhkYyTO8c4xeddJsHBzO47oz5wFwatGE9eQZ7Ibg2DXOez9+dDs+//ql/LVumpEi+Wrz5MX5bb03CrUKu0a8UZ26YAraUhqe+/w/FB1jnkIpI3lJTNTkTCKETCWEqPbfCwEsAbCtFvuSRId1+vlF8uIUc2KRMsASufedu4h32LJIXjcpohRo9FoqYiT/40e3u7JXdIP6dhqX23aUDBjTZdf4RPL2a3azqSSFUuSoGVY206S2ZJGNxvoDZAqlJC6qTaG8khCyC8AZAO4hhKyw3zoHwHpCyFoAdwB4L6W0r6qWSmKjlMhPEUTe730RUaCiePJFIu8RwSGhbo5umpHsmkrKGuguu0b05IvryYvLgzgwmMXyL/0VL+0fsjdktes3/3I6HvzYOSU/KyN5SdxUdSZRSu+klM6mlKYopdMppRfby/9AKT2WUrqMUnoSpfTP8TRXEgcsg8NvQE6UeUS1SkXePuucQmXW/+zJoX/Emew6escr23ZlefK6y5O3/o9ahXLFC/txcDiPnz6+g7eFEOCMRVOweFpHyc/yKRKlJy+JCXkmTUDSCRUtCdU344YNoV84ta3oPS/i56Nk0ao+dk1LQsXnLj8GANCXcQqi6UKefBgqy5M3ff/23ijCZtd4DwX1WRYEE3eZXSOJC1mgbALSmlRdFShFZnSm8cmLj8IbTzyi7HZE8a0khVLseNUUgkn2DeawS+TNiJ689X8UTz4wu8Zzp2C2USWlhsPW3kmoMpKXxIsU+QlIS0IN9NsJIXj/eYtDbcdt14Tfv7cuu2GaUBSCSXb64GGhtHGhwo7XSNk1grAbptuuURXCnwqc7JpwiNU2w36DpBR5ScxIkZ+ATOtI8SJj1VB5x6v1/0dvX4fhrA6DWkLOcsQPZxxP3jApH2kbbtu2yEf4eqKwP/DCfhw3y6q+aVIr/5+9W/GkIQg/Kpx58rLjVRIX8kyagNz4xmPxvXecVPV2vGWKwyLeED5/10YUdApFIehIa1AV4orkdSNadk2lefItCRVJTcF3V27hNo1Jre/FrJ9KKlxa64ertw9YNxKFyEheEh/yTJqAdLUkXPnwlaJWaNd4/enth0ag2TXju1sSLk++YNBIOeNKyM5REcOkmDWpBR+98EgYJkXOfsqh1D2Sl+lueL+f8rZEyXpPqIrseJXEhjyTJBVDCOFCX4ldwxgcLfDPT2pL4tW+jFNrPnI9eev/UpODe9FNEyohfARwxp7E3LJrCH8qCOvJew8F205YLlw6HSfPmxR6fYmkFFLkJVXhiHyEz3gEbyir8+2ctbgHq14+iDue2QUAKJhmpBTKSmeGUhXCxw1k8oa9DWt7bFOVbBtgnnz49b/79pPwhhDZTRJJGKTIS6qC+fJRPHnvukPZAt/OjW9YiqSmYGvvMABLgKNNGlLZiFdNJXyQGKuyyQYxUW92TWRPXpbjljQOKfKSqqgkkveuO5TTuZdOCEFXS4KXO9YNyguhRdl21Dx5VRHtGsPeBlx2TdhIXszW4aURQrdGIokXKfKSqtAq8OS9HrtVpdFZJoq8NVNUBZF8pElDrBROZteM2pG8YfqnUPYLKZ5+sAFVlKLI6pFI6o0UeUlVsCi7khGvIuLnuwWRj97xWpldoyqOXTOad+waVXFSKFk7rv3RU9hm20m+2zOKSyNIjZc0CinykqpwPPnwn/HTbLEIWVdLgkfL3jlfy0F48bOokbziiHzB6XglQser+B1fPlBC5EW7hrUrdGskkniRIi+pikpSKP0ic/HzXa2CJ29WWk8+9Ed4JN+asAaAO568x64R2uidwUqEvUepc7MJW7tGIokbKfKSqmAReCWThri24/HkB8WO1xqXGi7olu/v2DU6Ht96EIPZgp1C6bZrgNIiz2rh6CaNnIkjkcSNrF0jqYrKBkM567anNAwL2TWAJfJDOR0Fw0TeMJHW/Ctmltp2FE8+k9fRmtS4yD+1vQ+fv2sjAKvOj+nTeVrQg3dQsAvniGWLZcerpFHISF5SFZXkyYvOBZscXPV0vALAgaEcAKv+fVgqqV0zkjfQllJ5ds2rfRn+Xt9Ivii7BgByYSJ5g8qOV0nDkSIvqQonuyb8Z7yRPODpeG21RH7fQBYA0FJBFcooefIjOR1tSas4WlJTsNfeL2B3ovI8eeczpap4suyavGEKKZShmyORxIoUeUlVVJInL1ozzCJxdbzakfz+QUtso0TyUe0a06TI5A202jeb1qSKXvsJgsE2JbY7Y89Du+XAEC/BwCiYLJI3hcnApcpLGoMUeUlV8I7XCjWMzVAldrx2pC2RPzhsiS27EYRBIda2WGmCcmTs9dpTwfPe8gJlwo1oxM7A+fkTr+DTf1jvmSeWefLUSaGUGi9pEFLkJVVRiScv0mKnLYpRMisvcGjYKjmcitDxSghBe1rDUFYPtT6LyNlMWc6ThbOOt3YNYFk8ANA7lINhUveUhbYnn9dNUNNpl0TSCKTIS6qikto1ImzWJzGSb0tagttnTx4SZWYoAOhIaxjKFrC1dxib9g6WXHeYiXzSsWsAYNakFr6OMxhKjOStz7GnDfY/INg1JgWL5aXESxqFFHlJVWgVlDUQmdmVBgAsmd7Bl7Xa1smhEduuieDJA0BHKoGhrI4Lvv53XPqtVSXXHclZtguL5NmAqO6WJF9nSrv1t3izESN5ADg45J7Niv0vO14ljUbmyUuqopI8eZGPXHgk3nH6PMyb0saXsaia2TVROl4BFsmHs2tYRN5mR/Dze1qxekcf7/wFgG++bRnu37gPRwo3IjYq9qDdRlckz+waVwqlVHlJY5CRvKQqKqldI9KW0lwCDziR+yHbronS8QpYHbeD2dKVIhkjHk+eTeLNZocCgCntKVx72jxXND6c0zGaN7jdI4o8GwSlG6bseJU0HBnJS6qi2kjeD8Wu7c49+QgdrwDQmdZ4+mU5WJYMy/JhIr/z8ChWfOQcXqzMwvmOz73aj+dePcxfi2mXYlkDGclLGo0UeUlVMJFPxjzxdGtS49FxOhm94/WwUPO9VCVLbyS/dGYn//+oGR2udb2++nU/Wc3/7nXZNfZgKN3kSfZS4iWNQoq8pCr22KNDT5zdFet2WwWLJronn3C9Hs7qODCUw5pX+rB+5wDed+4izOxO485nd+P2p3cCcEQ+nVBx74fOxuzJLUXb9UbjzHtPagr29I/y5TrPrnHsGlm7RtIopMhLqmLdzn4AwKkLpsS6XVHkI2fXpN2n9Q1/XI8VG/fz16/0jWC0YPK2A0CrsI+lR3T6bleM5M89aioe3twLwJp8/PGtB7G7fxSzuluE7BpZu0bSeGTHq6QqjpzeDgBYMq091u2yyFpVSKRJQ4DiSJ4J/D8snY6zl/TgyW19WL+rH9+6ZhlfRwuxD7E0Aau5AwCXHDsD2YKJ197yENbv6keeZ9fIFEpJ45GRvKQqfvMvp2NgtBB5UozPXHo0jp7pHzEDTiSfrsDr90byAPCGE4/AN9+2DCN5HXc+uxvHzOzEqQsmY0pbCk/v6Au1XRaNE+K+kVy4dDrwB+vvHzyyjQ/AckXy0pWXNAgp8pKqmNKewpT2VOTP/evrFpV8n+XKR02fBJxaM+KE4F++8jioCkFnOoHrzpzP1z1rSQ/OWtITartM5DWF8BtJW1LF5LYkHvzYObjwG4/gL+v38vV104nkpcZLGoW0ayRjEjbqNUrdGsZrF/fgjIVTXHaM18KpBNZ5qikKOnitG+v/xdM6MKvb3VlbMKhg10iVlzQGGclLxiTVRPI97Sn85vrTcSBkrnxYeCSvWkXQxGWAUwdHJG93wkqJlzQKGclLxiStPiWIo9Lu481XA4vGE6rCnwzE5i316WNgk4tEmQNXIomTqk49QsithJAXCSHrCSF3EkK6hfc+QwjZQgjZTAi5uOqWSiYUR9l1YmZPKs5XD0vU1MtyMD1XFcKza8QO1e9eexI+cdGRrs+w8gqy41XSKKoNdf4K4DOUUp0Q8lUAnwHwaULIUgDXADgWwBEAHiSEHEkpDTeTg2TCc9VJs3HOkVO5bVMJhBD8v1Pn4ryjpsbSJjYYKqEQdPrYNZPbknjr8jn4rwde4stWvWzl0ne2SGdU0hiqiuQppQ9QSpkR+SSA2fbfVwD4LaU0RyndDmALgFOr2Zdk4tHTnqrIkxf5ylXH46JjZ8TSHseTVxxP3rNOj51pdIxt3Xx35VakEwpOXxjvYDGJJCxxOoXvAXCf/fcsADuF93bZy4oghFxPCFlDCFnT29sbY3MkknhhmTKaKtg1nqwZRSFY9anzcMd7z0DCnhrxnCVT0VrFE4lEUg1lzzxCyIMA/EKhz1JK77LX+SwAHcCvojaAUvoDAD8AgOXLl4ecflkiqT+shLCmEKRsv/+I7nTRenMmtwJwatuUGxMgkdSSsiJPKb2w1PuEkHcDeD2ACyjlQz92A5gjrDbbXiaRjFtYCWFNUTCruwVfu/oEnHfUtMD1f3zdcuzpH8XJ8ybVq4kSSRFVPUMSQi4B8CkAr6OUZoS37gbwa0LIN2B1vC4BsNpnExLJuIGVEGY2zFuXzym1Oi44ZnrN2ySRlKNao/A7AFIA/mp7k09SSt9LKd1ICPkdgBdg2Tjvl5k1kvEOe0xtS0l/XTJ+qOpspZQuLvHezQBurmb7EslYYtnsbnzw/MV45+nzGt0UiSQ0MiSRSEKiKAQfv+ioRjdDIomEHGwtkUgkTYwUeYlEImlipMhLJBJJEyNFXiKRSJoYKfISiUTSxEiRl0gkkiZGirxEIpE0MVLkJRKJpIkhTk2xxkMI6QXwShWb6AFwMKbmxIlsVzRku6Ih2xWdsdq2Sts1j1LqOzvOmBL5aiGErKGULm90O7zIdkVDtisasl3RGattq0W7pF0jkUgkTYwUeYlEImlimk3kf9DoBgQg2xUN2a5oyHZFZ6y2LfZ2NZUnL5FIJBI3zRbJSyQSiURAirxEIpE0MU0h8oSQSwghmwkhWwghNzS4LTsIIc8TQtYSQtbYyyYTQv5KCHnZ/r8uMzsTQm4jhBwghGwQlvm2hVh82z6G6wkhJ9W5XTcRQnbbx20tIeQy4b3P2O3aTAi5uEZtmkMIWUkIeYEQspEQ8mF7eUOPV4l2NfR42ftJE0JWE0LW2W37gr18ASHkKbsNtxNCkvbylP16i/3+/Dq366eEkO3CMVtmL6/buW/vTyWEPEcI+Yv9urbHi1I6rv8BUAFsBbAQQBLAOgBLG9ieHQB6PMu+BuAG++8bAHy1Tm05B8BJADaUawuAywDcB4AAOB3AU3Vu100APuGz7lL7N00BWGD/1moN2jQTwEn23x0AXrL33dDjVaJdDT1e9r4IgHb77wSAp+xj8TsA19jLvw/gffbf/wbg+/bf1wC4vc7t+imAq33Wr9u5b+/vYwB+DeAv9uuaHq9miORPBbCFUrqNUpoH8FsAVzS4TV6uAPAz+++fAXhTPXZKKX0EQF/ItlwB4OfU4kkA3YSQmXVsVxBXAPgtpTRHKd0OYAus3zzuNu2llD5r/z0EYBOAWWjw8SrRriDqcrzs9lBK6bD9MmH/owDOB3CHvdx7zNixvAPABYQQUsd2BVG3c58QMhvA5QB+ZL8mqPHxagaRnwVgp/B6F0pfBLWGAniAEPIMIeR6e9l0Sule++99AKY3pmkl2zIWjuMH7Mfl2wRLq+7tsh+LXwMrAhwzx8vTLmAMHC/belgL4ACAv8J6cuinlOo+++dts98fADClHu2ilLJjdrN9zP6bEJLytsunzXHzTQCfAmDar6egxserGUR+rHEWpfQkAJcCeD8h5BzxTWo9e42JvNWx1BYA/wtgEYBlAPYC+HojGkEIaQfwBwAfoZQOiu818nj5tGtMHC9KqUEpXQZgNqwnhqMb0Q4v3nYRQo4D8BlY7TsFwGQAn65nmwghrwdwgFL6TD332wwivxvAHOH1bHtZQ6CU7rb/PwDgTlgn/n72+Gf/f6BR7SvRloYeR0rpfvvCNAH8EI7FULd2EUISsIT0V5TSP9qLG368/No1Fo6XCKW0H8BKAGfAsjs0n/3zttnvdwE4VKd2XWJbX5RSmgPwE9T/mL0WwBsJITtg2crnA/gWany8mkHknwawxO6hTsLqoLi7EQ0hhLQRQjrY3wAuArDBbs919mrXAbirEe2zCWrL3QDeZWcanA5gQLApao7HA70S1nFj7brGzjRYAGAJgNU12D8B8GMAmyil3xDeaujxCmpXo4+X3YaphJBu++8WAP8Aq89gJYCr7dW8x4wdy6sBPGQ/HdWjXS8KN2sCy/cWj1nNf0tK6WcopbMppfNh6dRDlNJrUevjFWevcaP+weodfwmWH/jZBrZjIazMhnUANrK2wPLR/gbgZQAPAphcp/b8BtajfAGW1/dPQW2BlVnwXfsYPg9geZ3b9Qt7v+vtk3umsP5n7XZtBnBpjdp0FiwrZj2Atfa/yxp9vEq0q6HHy97PCQCes9uwAcB/CNfBalidvr8HkLKXp+3XW+z3F9a5XQ/Zx2wDgF/CycCp27kvtPFcONk1NT1esqyBRCKRNDHNYNdIJBKJJAAp8hKJRNLESJGXSCSSJkaKvEQikTQxUuQlEomkiZEiL5FIJE2MFHmJRCJpYv4/XdldzWG3WJoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(y.logits[0][0][range(0,400)].cpu().numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1092a21e",
   "metadata": {},
   "source": [
    "Try to compare the pattern for 'ing' to the embedding matrix for 'ing'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "68f80db9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 14, 768])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.hidden_states[11].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "977409d5",
   "metadata": {},
   "source": [
    "y.hidden_states is a tuple - 1 element per layer. \n",
    "Each has 1 batch dim, 14 tokens, 768 weights. Look at the first token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "bb5c0f3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2286fefc0a0>]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiLElEQVR4nO3deXxcdbnH8c/TdKP7ShuabtCwdGFpYwHZlEWKVRAELXoVxQtyEa/rtUUUV1TQK14FwaooqFC4LKXGQi2lLHJpS0r3PV0gaZulW9K02fPcP+ZkmLRJejqTZCbJ9/16zStnfuecOc/MOTnfOb9zZsbcHRERkTC6JLsAERFpPxQaIiISmkJDRERCU2iIiEhoCg0REQmta7ILSNSQIUN8zJgxyS5DRKRdWb58+R53H3q887X70BgzZgw5OTnJLkNEpF0xs3fimU/dUyIiEppCQ0REQlNoiIhIaAoNEREJTaEhIiKhKTRERCQ0hYaIiISm0BARSYIDh6v4+6pdyS7juLX7D/eJiLRHX35iBa9v2cM5owaQMbBXsssJTUcaIiJJsH3PIQAqquuSXMnxUWiIiCRBXV3kV1MPVdYkuZLjo9DogPaUVZJ513yWbtub7FJEWtSeskpunL2EnQfKk11KwoLMoEyhIYly94TefeTs2E91rTP7tW0tWNV7slfv4vmVO485XW2dU1haEeoxa2rrqKltX4fp9Sqqa6luodrX7iwhb9/hFnms1jZ3xU7uePztNl3mE0vf5c1te/nzG9upq3MOVlQDUFha0S62n9o65y9v7qC8qpZaj6RG/XNozO6Scm55LIcDh6vaqsRjSrnQMLNpZrbJzHLNbFZbLHNvWSVVNQ03uJLyap5Y9i4erNj6lVZeVUttnVNQUsFf3twRHQ+RHV9FdS3uzu6Shu+EKqpryS0qo7bOKTpYQfHByibr+f3r25jwvQUsXF/I5sKDQORQtqK6NvpYtXV+1Hx5+w5z3W/f4La/Lgdg0cYifpy9nudW5PODv69j/prd/PaVXP7w+jYWbypqMO+RO/c3t+5lR9Dn+sqmIg5X1USXfcfjK/jKnJUs2lAYrSn2daiv5d4XN3LuTxY1eK7uzsaCUsoqa/j1oi3RcR9/+E0+/OvXWbuzhDX5Jew7VEVVTR0/yl4fra20opq6Omdl3gFu/9tylm7by/5DVVzws5f535w8/m/rHhauL6S2znF3nl6eT25RGe7O3rLIcgpKKqI1A5Qcro7u8B97cwd/eXMH7+49zK4D5SzdtperH/hX9LnH2ltWyfY9hygpr+b0777IRfcujq7/WLtLyqOvTd6+w7y6uTg6rryqlj1llVTX1lFeFZnvI7/5FxfdtxiADbtLGzxe8cFK8vcfJreojO89v5YV7+4HYP6a3WSvbvoqHHfn/3L3UBrsnKpr69ix59BR66zei2sL2FhQ2mB+gIXrC5n1zGrcI28GvvrkSrJX7yZv3+EmHwuI/k/EKiipIH//Yb719Kqj3mlvLjzIvz+a0+CNU2VNLSWHq6PT9urelf9ZtIVJ3/8n63aVcO5PFvHQK1spOljBu3sPR5/nht2lvB28TrUxIbPvUNVRIbMy7wAPLs5lw+5SFm0ojP6/xK7/PWWVHK6qobKm4Xo+0tvv7mdPWcP/8U0FBznjuy/y3efXccbdL0a3/SffymPD7tLoc6uqqYu+Xn9d8g4L1xfyyBs7ml1eW7LmVnZbM7M0YDNwBZAPvAXc6O7rm5onKyvL4/lqdHfnvgWbeOiVrQB88LSh/O4zWXRLM/aUVfGzFzbyzNv5fGf6Gfz+9W0Ulja+k582YTg3ZGUwf00Bz7ydf9T4my8Yy6SMfsxdsYtXNxdz2rC+bAqC4P2nDGZQ7+7sO1RFQUkFfXp25ZJTh/Kbl3MbPMZjN0/lP/66nMxhfblx6khmPrOGU4b2ZmvxIT597iimT0qnT8+uXP3AG8f1Gpx/8mC6de3Ca8GO7DvTz2DSiP5kDuvL5B8tBODMjP6szi8h88Q+fOrcUTz25jvRE3j10roYtXXOGen9+OrlmQzq3Z0bHn4zOv6Ll5zM/kNVTMoYwHfnrm0w74XjhjCwd/ejLj08sW8Pqmvr2H+4mosyh/CJrJF86+nV9Oqext5Dzb/rSu/fk3PHDmLuyshjfvnScfzm5VwevXkqNz2yjFOG9ubvX76Qx5e+y4//sYHPXzCGj551Etf99v8afTwzuOTUoZSUV1NQUsH49H4s2hgJ3a5djJpGAvy+j5/JhBH9mP7rf3FmRn/OP2Uwv3s1cuT3mxvP4XevbWXtzsiO+bRhfdm2p4x/fu0SPviLVwD4ymWZ/M+iLVx7zgi+/9EJ9O/VjfN+soiCZo7cvjP9DKpq61iwtoBxJ/ZlUO9uTDipP927duH2v73NdZNHcOWE4Tz7dj4L1hVGX5sxg3vz0Ktbyd9/mLFD+rBhd6Su6ZPSue2SU/ivp1dxqKqGvH3l0fq//MSKBsu++yPjuXHqKApKK/jgL17hRx+byEXjhrB2Vwl3PL6CL186jgvGDWHG7CVH1f3JrJH89LpJ/GrRFooPVvDEsjwARg3qRd7+w3zrytOZt2oXmwpKOXvkAN5+9wDXTR7BlsIy1uwsiT7OuWMHsTLvAJU1dWSe2IctRWXRcc/e/n7+9MYO/r5qF5NG9GfdrhJOH96Pb3/4DH7+z02cMbwvc97Ka/R17ZZmvDHzUj77yDI2FhyMts+66nSWbtvLFy48mQG9uvHi2gK2FB3kzIwB/HzBJgAuPf1EThnam7U7S3nzGN3FQ/r0YPE3L+Gjv/kXQ/v24K7p41m4voAHF2+lW5px3/Vncunpw5gxewmDe3dnxtSRTJ+Ujpk1+7hNMbPl7p513POlWGicD3zf3a8M7t8J4O4/bWqeeEPjsTd3cPfz6xq0nTqsD5sLy5qYI7X17dGVg+2sb1TCO2vkAFblHUh2GZJinr39/UweNTCueeMNjVTrnhoBxMZ9ftDWgJndamY5ZpZTXFx85OhQXtu8h4kj+jHrqtNZ9I1LuChzyFGBcc6oAdHhnt26MGpQL6afmc7cL10Qbf/lJ86KDp+V0Z97rp3IT66dxLofXEnPbg1f3qF9ezRay7B+PTh5aO+j2i/KHNLo9N/76HgAhvfrySeyMgA4WFnDqEG9WPzND/Dnz7+Pgb26cd/Hz2TxNz/QYLl/+tz7Gn3MeoN7d2fiiH4N2k4f3pefXDuJ/7ryNN6YdSkr774i2g4wqHf3ox7ntGF9GXdiH0YNavr68+9MP4NRg3px8alD+eNNWXzhwrGNTtezWxdmTjudVXd/iN99Zgor776Ch/9tChA5WhozuBc3TMlodN5pE4YDcEZ6P943ZiDTJ6Vz1cTh0fGxb9L69ezKmMGRei8cN4QPTxrOp88dFR1/7thBvPLND/DZ80cD8KNrJnDiEes0Y+AJXHdOZJMd0qcHn8waGR133eT3NuVe3dMY3q9nk6/NkeoD46kvns/XrziVqWMH8aHxw/jFDWfx+rc+yKq7P8SIASc0+3pnntiHK8YPi94f2KsbA3p1496PT2LJnZdF28cOabgtfuTMdJ645TzS+/fkwU9NbjDuq5dnNlt3v54NPwr2xUtObrA9Pn7LuZw9ckD0/vj0fjx+y7k8fsu5fOOKU/nq5ZmcNqwvV00czkOfnswNUzL4/Web3s/94OoJ5N5zVbM1XTd5BPddf2aT41/6+sU8/G9TmDpmEPdcO5GrzzqpwfhzRg1gwkn9OP/kwdzxwXGMDraZBz81maljBkWne+729/PYzVO5YUoGn79gDAA3Th3FD6+ZEJ1m8Tc/wH9elsn9n3xvP3LtOe9tJycP6c1/Xtb4a/yrT54dd2AkxN1T5gZcD/wh5v5ngAeam2fKlCkej7q6Oj9wqCp6v6i0wh9+Jdf3llX6jbPf9NEzs72mts7X5B/wLYUHj5p//upd/tb2vX64ssYvvu9lf2HN7kaXc+BQlVfX1Lq7+7biMr/1sbd84+5S33+o0nN27PNrH/yXH6yodnf33KKD/kZusS9/Z5/vOnDY3d0//6dlPnpmttfW1vntf13uo2dm+4FDVV5S/l7tT+fk+eiZ2X7NA/9q8rm+vKHQtxeXubt7zo59/trmIp+7It/X5B9wd/fVeQf8rudWe01tnbu7H6qs9g27S3zHnrJofbG2Fh30gxXVPm/lTt9eXOar8vb71qKDfv/CTZ6371CDaQ9VVvuy7Xuj99//00X+jadWNlprZXWtl1fVeE1tnb+2ucjvX7gpWtORyqtqvK4uMu5gRbWPnpnto2dm+9fmrPB7X9jgz76d5+7ur28u9kOVDZ/D4coa/3H2Ot+5/7BX1dRGay6vqvGfv7jR9x+qjE67uaC0wXOqqK7x3KKD0ccpKa/yt7bv9dEzs6Ovcb0Dh6uide0rq/Szf7DAz//JS/7a5iJ3d99eXOY/+vs631J40H//2lbfVFAaXc+3/SXHb/tLjv84e50v2brHX91U1OjrEPs6H6qs9uKDFT56ZraP+/Y/fPTMbH9jS7EXlpZHX6cf/n2dF5aUHzX/Fb98xUfPzPYthaU+ema2379wkz/w8pajpp23cqd/46mV0e161jOrfPTM7Oi6KCwp97kr8v21zUVeUV3jhaXlPuuZVdH/t7KKan9hza4Gr9XuA+VeWFoefYxjqX+9R8/M9n/7wxJ/OifP73txQ3T8t59d7e/78UJ/Yc0u31J40O+eu8a/O3dNdPzcFfnR+d/Zc8ira2p9zrJ3fO6K/KOWVVcX2Q+8uqnIvzt3zVH/D7W1dV5ZHXktyqtqfP2uEv/fnLwmn0ttbZ0/sfSdBttUXV2dj56Z7Zfc97K7uz+57F3/zaLNDeapra3zP7+x3QtKyn1fWeVRj3u8gByPYz/dabunmlNRXUtJeTXDjuOdYGuprKmlqqaOvj27UVlTS/7+ck4Z2qfBNG/k7uHTf1jKOaMG8NztFzTxSB3fTY8s49XNxXx8cgb/HXMEmGxff2olHxo/nGkxRzjNcXeKD1ZyYgLb3+6Scgb26k6Prl1C93kXlVbw9rv7mTYxnb1llQzq3T3UvO5OTZ3TLa3tOi5yduzj+uC82aq7P0T/Xt0araup+p9fuZOvzFkJwI6fTW+1Oo9HQUkFJ3RPo/8JRz+X1hBv91SqfY3IW0CmmY0FdgIzgE+1dRE9u6XRs1taWy+2UT26ptGja1p0+MjAAJg8aiCXnDqUWVed3tblpZT6rqI4zwu2ml9+4uzjmt7MEgoMgPT+Jxz3PCf268m0iekADO7TeFdqY8yMbmlt+6LHruO0JpbdXODFe/K4NQ3vn/w3qWGkVGi4e42Z3QEsANKAR9x93TFm6/RO6J7GozdPTXYZSVe/H+iSevsDaXHWyFB42kbil1KhAeDu84H5ya5D2p8uQWpYXLsR6Uy0jcQv1a6eEolb9EhDW3WHF9u7FE9PUwr2TrUb+veSDuO9fmrtEaR56p6Kn0JDOoxoZGiH0OFZg+F4Vrg2kngpNKTDqD+noXeRnYu6p9qWQkM6jPodgU5ydnyJXjLbRakRN4WGdBj1uwEdacixaBOJn0JDOoz6d5+p+MEtaVmJrmFtIvFTaEiHEe2e0g6hU4lnfat7Kn4KDekw9OG+ziPhfb42kbgpNKTD0DmNzimeNwnaROKn0JAOo0uX+nMaSS5EWl2iR5PqnoqfQkM6jPc+3KcdQmeiz2m0LYWGdBjvXT2V5EKk1SW6jnXeK34KDekw9OG+zklfjd62FBrSYehEuISmbSRuCg3pMLqoe6rTaPjV6PFcPaWNJF4KDekw3vvlPu0QpHk6Go2fQkM6jOiJ8CTXIa0v9kghri9G1xuLuCk0pMOI7ga0Q+hUdMlt21JoiEi7k+hOX91T8VNoSIehd4+dU3xdTdpY4qXQEJF2J+EP9ykz4qbQEJFOR1fYxU+hIR2Ge7IrkLaS6OcsFBnxU2iISKejA434KTREpN1J/OoppUa8FBoiIhKaQkM6DL157DwS/rVXbStxU2iISLuj7qnkabXQMLPvm9lOM1sZ3D4cM+5OM8s1s01mdmVM+7SgLdfMZrVWbSLSuSkz4te1lR//fnf/RWyDmY0HZgATgJOAl8zs1GD0g8AVQD7wlpnNc/f1rVyjiLQ7iV5yq9SIV2uHRmOuAea4eyWw3cxyganBuFx33wZgZnOCaRUaItKi9N1T8Wvtcxp3mNlqM3vEzAYGbSOAvJhp8oO2ptqPYma3mlmOmeUUFxe3Rt0iksL0NSLJk1BomNlLZra2kds1wEPAKcDZwG7gvxMvN8LdZ7t7lrtnDR06tKUeVkQ6DaVGvBLqnnL3y8NMZ2a/B7KDuzuBkTGjM4I2mmkXOSZ9jUjnkeguX91T8WvNq6fSY+5eC6wNhucBM8ysh5mNBTKBZcBbQKaZjTWz7kROls9rrfpEpPPSL/fFrzVPhN9nZmcDDuwAvgjg7uvM7CkiJ7hrgC+5ey2Amd0BLADSgEfcfV0r1icdjPYDnUeiO31tKvFrtdBw9880M+4e4J5G2ucD81urJhER0If7EqFPhItIu6OvEUkehYaIiISm0BCRdifh757S5VNxU2iISKejyIifQkNE2p2Ef+5VqRE3hYZ0GPpwn4Slq6fip9AQkXYn4e+eapkyOiWFhnQYevMooWlbiZtCQ0Q6HXVPxU+hISLtjrqnkkehISKdjr6wMH4KDRFpdxLd6euzffFTaIhIp6PfCI+fQkNE2p2Ed/nKjLgpNKTD0If7JCx1T8VPoSEi7U7CV0/pRHjcFBrSYWg/IGFpU4mfQkNE2p1ET2Trw33xU2iISKejzIifQkNE2h3t9JNHoSEinY66p+Kn0BCRdifRXb4yI34KDRFpf/SFhUmj0BCRTkfdU/FTaEiHMbRvj8jfPt2TXIm0Nv1GePJ0TXYBIi3lxveNol/PbkyflJ7sUiTF6RPh8VNoSIfRpYvx0bNOSnYZ0ga0z08edU+JiEhoCg0RaXd0oJE8CYWGmd1gZuvMrM7Mso4Yd6eZ5ZrZJjO7MqZ9WtCWa2azYtrHmtnSoP1JM9PZTBGRFJPokcZa4DrgtdhGMxsPzAAmANOA35pZmpmlAQ8CVwHjgRuDaQHuBe5393HAfuALCdYmIh2UTmQnT0Kh4e4b3H1TI6OuAea4e6W7bwdyganBLdfdt7l7FTAHuMYiW8ClwNPB/I8CH0ukNhERaXmtdU5jBJAXcz8/aGuqfTBwwN1rjmhvlJndamY5ZpZTXFzcooWLSOprieOMIX26c+dVp7fAI3Uux7zk1sxeAoY3Muoud3++5Us6NnefDcwGyMrK0o98ishxy/nOFckuoV06Zmi4++VxPO5OYGTM/YygjSba9wIDzKxrcLQRO72ISAM6pZE8rdU9NQ+YYWY9zGwskAksA94CMoMrpboTOVk+z90dWAxcH8x/E5CUoxgREWlaopfcXmtm+cD5wD/MbAGAu68DngLWAy8CX3L32uAo4g5gAbABeCqYFmAm8HUzyyVyjuOPidQmIh1Xot89JfFL6GtE3P054Lkmxt0D3NNI+3xgfiPt24hcXSUi0jxlRtLoE+EiIhKaQkNE2h2dCE8ehYaIiISm0BCRdkcHGsmj0BARkdAUGiLS7ugLC5NHoSEiIqEpNESk3dFxRvIoNEREJDSFhoi0OzqlkTwKDRERCU2hISLtjr6wMHkUGiIiEppCQ0TaHZ3TSB6FhoiIhKbQEBGR0BQaItLuqHsqeRQaIiISmkJDRNodXXKbPAoNEREJTaEhIu2Ozmkkj0JDRERCU2iISLujA43kUWiIiEhoCg0RaXf0c6/Jo9AQEZHQFBoi0u7oOCN5FBoiIhJaQqFhZjeY2TozqzOzrJj2MWZWbmYrg9vDMeOmmNkaM8s1s19b0DlpZoPMbKGZbQn+DkykNhHpuHRKI3kSPdJYC1wHvNbIuK3ufnZwuy2m/SHgFiAzuE0L2mcBi9w9E1gU3BcRkRSSUGi4+wZ33xR2ejNLB/q5+xJ3d+Ax4GPB6GuAR4PhR2PaRUQa0NVTydOa5zTGmtkKM3vVzC4K2kYA+THT5AdtAMPcfXcwXAAMa8XaREQkDl2PNYGZvQQMb2TUXe7+fBOz7QZGufteM5sCzDWzCWGLcnc3M2+mpluBWwFGjRoV9mFFRCRBxwwNd7/8eB/U3SuBymB4uZltBU4FdgIZMZNmBG0AhWaW7u67g26somYefzYwGyArK6vJcBERkZbVKt1TZjbUzNKC4ZOJnPDeFnQ/lZrZecFVU58F6o9W5gE3BcM3xbSLiEiKSPSS22vNLB84H/iHmS0IRl0MrDazlcDTwG3uvi8YdzvwByAX2Aq8ELT/DLjCzLYAlwf3RUQkhRyze6o57v4c8Fwj7c8AzzQxTw4wsZH2vcBlidQjIiKtS58IFxGR0BQaIiISmkJDRERCU2iIiEhoCg0REQlNoSEiIqEpNEREJDSFhoiIhKbQEBGR0BQaIiISmkJDRERCU2iIiEhoCg0REQlNoSEiIqEpNEREJDSFhoiIhKbQEBGR0BQaIiISmkJDRERCU2iIiEhoCg0REQlNoSEiIqEpNEREJDSFhoiIhKbQEBGR0BQaIiISmkJDRERCU2iIiEhoCg0REQktodAws5+b2UYzW21mz5nZgJhxd5pZrpltMrMrY9qnBW25ZjYrpn2smS0N2p80s+6J1CYiIi0v0SONhcBEdz8T2AzcCWBm44EZwARgGvBbM0szszTgQeAqYDxwYzAtwL3A/e4+DtgPfCHB2kREpIUlFBru/k93rwnuLgEyguFrgDnuXunu24FcYGpwy3X3be5eBcwBrjEzAy4Fng7mfxT4WCK1iYhIy2vJcxo3Ay8EwyOAvJhx+UFbU+2DgQMxAVTf3igzu9XMcswsp7i4uIXKFxGRY+l6rAnM7CVgeCOj7nL354Np7gJqgL+1bHmNc/fZwGyArKwsb4tliohIiNBw98ubG29mnwM+Alzm7vU78J3AyJjJMoI2mmjfCwwws67B0Ubs9CIikiISvXpqGvAt4Gp3Pxwzah4ww8x6mNlYIBNYBrwFZAZXSnUncrJ8XhA2i4Hrg/lvAp5PpDYREWl5xzzSOIYHgB7Awsi5bJa4+23uvs7MngLWE+m2+pK71wKY2R3AAiANeMTd1wWPNROYY2Y/BlYAf0ywNhERaWEJhUZweWxT4+4B7mmkfT4wv5H2bUSurhIRkRSlT4SLiEhoCg0REQlNoSEiIqEpNEREJDSFhoiIhKbQEBGR0BQaIiISmkJDRERCU2iIiEhoCg0REQlNoSEiIqEpNEREJDSFhoiIhKbQEBGR0BQaIiISmkJDRERCU2iIiEhoCg0REQlNoSEiIqEpNEREJDSFhoiIhKbQEBGR0BQaIiISmkJDRERCU2iIiEhoCg0REQlNoSEiIqEpNEREJDSFhoiIhJZQaJjZz81so5mtNrPnzGxA0D7GzMrNbGVwezhmnilmtsbMcs3s12ZmQfsgM1toZluCvwMTemYiItLiEj3SWAhMdPczgc3AnTHjtrr72cHttpj2h4BbgMzgNi1onwUscvdMYFFwX0REUkhCoeHu/3T3muDuEiCjuenNLB3o5+5L3N2Bx4CPBaOvAR4Nhh+NaRcRkRTRkuc0bgZeiLk/1sxWmNmrZnZR0DYCyI+ZJj9oAxjm7ruD4QJgWFMLMrNbzSzHzHKKi4tbqHwRETmWrseawMxeAoY3Muoud38+mOYuoAb4WzBuNzDK3fea2RRgrplNCFuUu7uZeTPjZwOzAbKyspqcTkQ6rl998mxO7Nsj2WV0OscMDXe/vLnxZvY54CPAZUGXE+5eCVQGw8vNbCtwKrCThl1YGUEbQKGZpbv77qAbq+g4n4uIdCIfO2fEsSeSFpfo1VPTgG8BV7v74Zj2oWaWFgyfTOSE97ag+6nUzM4Lrpr6LPB8MNs84KZg+KaYdhERSRHHPNI4hgeAHsDC4MrZJcGVUhcDPzSzaqAOuM3d9wXz3A78GTiByDmQ+vMgPwOeMrMvAO8An0iwNhERaWEJhYa7j2ui/RngmSbG5QATG2nfC1yWSD0iItK69IlwEREJTaEhIiKhKTRERCQ0hYaIiISm0BARkdAs+Dxeu2VmxUQu0Y3HEGBPC5bT0lK5vlSuDVRfIlK5NlB9iaqvb7S7Dz3emdt9aCTCzHLcPSvZdTQlletL5dpA9SUilWsD1ZeoROtT95SIiISm0BARkdA6e2jMTnYBx5DK9aVybaD6EpHKtYHqS1RC9XXqcxoiInJ8OvuRhoiIHAeFhoiIhNZpQ8PMppnZJjPLNbNZSVj+I2ZWZGZrY9oGmdlCM9sS/B0YtJuZ/TqodbWZTW6D+kaa2WIzW29m68zsK6lSo5n1NLNlZrYqqO0HQftYM1sa1PCkmXUP2nsE93OD8WNaq7Yj6kwLfvI4O9XqM7MdZrbGzFaaWU7QlvR1GyxvgJk9bWYbzWyDmZ2fQrWdFrxm9bdSM/tqqtQXLPNrwf/FWjN7Ivh/abltz9073Q1IA7YCJwPdgVXA+Dau4WJgMrA2pu0+YFYwPAu4Nxj+MJHfHTHgPGBpG9SXDkwOhvsCm4HxqVBjsIw+wXA3YGmwzKeAGUH7w8B/BMO3Aw8HwzOAJ9toHX8deBzIDu6nTH3ADmDIEW1JX7fB8h4F/j0Y7g4MSJXajqgzDSgARqdKfcAIYDtwQsw297mW3Pba5MVNtRtwPrAg5v6dwJ1JqGMMDUNjE5AeDKcDm4Lh3wE3NjZdG9b6PHBFqtUI9ALeBs4l8inXrkeuY2ABcH4w3DWYzlq5rgxgEXApkB3sNFKpvh0cHRpJX7dA/2CnZ6lWWyO1fgh4I5XqIxIaecCgYFvKBq5syW2vs3ZP1b+w9fKDtmQb5pGfxIXIO5hhwXBS6w0OWc8h8o4+JWoMun5WEvkt+YVEjhwPuHtNI8uP1haMLwEGt1ZtgV8R+SnkuuD+4BSrz4F/mtlyM7s1aEuFdTsWKAb+FHTt/cHMeqdIbUeaATwRDKdEfe6+E/gF8C6wm8i2tJwW3PY6a2ikPI9Ef9KvhzazPkR+hfGr7l4aOy6ZNbp7rbufTeQd/VTg9GTU0Rgz+whQ5O7Lk11LMy5098nAVcCXzOzi2JFJXLddiXTbPuTu5wCHiHT3pEJtUcE5gauB/z1yXDLrC86lXEMkfE8CegPTWnIZnTU0dgIjY+5nBG3JVmhm6QDB36KgPSn1mlk3IoHxN3d/NhVrdPcDwGIih9wDzKz+J4xjlx+tLRjfH9jbimVdAFxtZjuAOUS6qP4nheqrf0eKuxcBzxEJ3lRYt/lAvrsvDe4/TSREUqG2WFcBb7t7YXA/Veq7HNju7sXuXg08S2R7bLFtr7OGxltAZnBFQXcih5nzklwTRGq4KRi+ich5hPr2zwZXYpwHlMQcCrcKMzPgj8AGd/9lKtVoZkPNbEAwfAKRcy0biITH9U3UVl/z9cDLwbvBVuHud7p7hruPIbJtvezun06V+syst5n1rR8m0je/lhRYt+5eAOSZ2WlB02XA+lSo7Qg38l7XVH0dqVDfu8B5ZtYr+B+uf/1abttrixNGqXgjclXDZiJ94XclYflPEOlzrCby7uoLRPoSFwFbgJeAQcG0BjwY1LoGyGqD+i4kcoi9GlgZ3D6cCjUCZwIrgtrWAncH7ScDy4BcIt0GPYL2nsH93GD8yW24nj/Ae1dPpUR9QR2rgtu6+u0/FdZtsLyzgZxg/c4FBqZKbcEyexN5N94/pi2V6vsBsDH43/gL0KMltz19jYiIiITWWbunREQkDgoNEREJTaEhIiKhKTRERCQ0hYaIiISm0BARkdAUGiIiEtr/AzQClIP0xbL1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(y['hidden_states'][11][0][0].cpu().numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "395780c1",
   "metadata": {},
   "source": [
    "And look at the embedding row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8884fdef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2286ff62970>]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3yklEQVR4nO3dd3xUVfo/8M+TMqmQAIHQO6IgghCagkhREFdxbYuu3RV7+7rrT3TtYmFdV911XVlx7QV1XQsozYYKaFB6r9IJaBIgpE3O74+5d3LvnXunT2YyfN6vFy8mdyYzJzNzz3POc8oVpRSIiIh0KfEuABERJRYGBiIiMmFgICIiEwYGIiIyYWAgIiKTtHgXIFgFBQWqc+fO8S4GEVGjsWTJkv1KqZah/l6jCQydO3dGcXFxvItBRNRoiMi2cH6PqSQiIjJhYCAiIhMGBiIiMmFgICIiEwYGIiIyYWAgIiITBgYiIjJhYKBGZ195Jeau3hvvYhAlLQYGanQufGEhrnm1GHV1vJYIUSwwMFCjs/VABQCgjheZClqtuw5/encZtu4/HO+iUCPAwECNlpuBIWg//lyKd5fswJ/eWxbvolAjwMBAjY6I5/+6uviWgyhZMTBQo6PFBaaSwsC3jILBwECNFlNJwdN7WUTBYGCgRouzkohiI6aBQUQ6iMgXIrJaRFaJyK3a8eYiMldENmj/N4tlOSi5iNb8dTMwEMVErHsMtQDuUEr1AjAEwI0i0gvAXQDmK6V6AJiv/UwUEsaF0PEto2DENDAopXYrpX7Ubh8EsAZAOwATALyiPewVAOfEshyUnDj4HDwOMVAoGmyMQUQ6AzgRwGIAhUqp3dpdewAUOvzOJBEpFpHikpKShikoJTy9kmMqicKxZf9hPD1vPRQbFo4aJDCISC6A9wHcppQqN96nPJ+O7SeklJqmlCpSShW1bBny9awpybHHQOG4dPpiPD1vA0oOVcW7KAkr5oFBRNLhCQpvKKX+qx3eKyJttPvbANgX63JQ8uECt9CxlQxU1rgBAMIEm6NYz0oSANMBrFFKPWW46yMAl2u3LwfwYSzLQclFn5PPdQzB4zqGenoGku+Js7QYP//JAC4FsEJElmrH7gbwOIAZInI1gG0ALoxxOSiJeFp6imMMIWjMMVTv5UiUanL2mgKLaWBQSn0D5wkRo2P52pT8eIIHr7HG0NKKavR7aC7u+00vXDWsS1SeU38rOEbljCufqdFiKil4jbUS3Fl6BAAwo3h71J5Tfys4RuWMgYEahbKKGuw7WOn5QR9jaKzN4DjQA0Nje8f0Sjw1JXoDAnpPs7EGy4bAwECNwpDH5mPQlPmmY43pvP7dCwvR+a6ZcXv9aL9Xc1btweuLtkX3SW3olXdKFEeK9feCDQtnsR58JoqKI9oUQ6PGdGIv3vJLXF8/2u/VpNeWAAAuGdIpqs9rpZc7ih0Gb6+pMTUsGhp7DNToeFc+88wOWmNNm3h7DFGMDPpz8vvjjIEhymav2oOv13P7joYQybbb+8or8ebin6NYmsSm14GNrS50awPETCU1LKaSouxarYu99fEz41yS5BfJeX3Na0uwbHspTu3ZEm3zs6JXqATVWHsMtdrUodRoBgYtmRTJdOfVu8rhShN0b9UkWsVKKOwxkKMZxduxdf/heBfDh0RhVtL+g1URP0djYvw7Rz35JU56bL6fRyeOGreeSorec3p7DBEEhvHPLsCYp76OUokSDwMD4dn5G9Dn/tmmY0op3Pneckx47ts4lcqZvsdNJK1gFYPcdTy46xR+OVzt9zGVNW7Te7V5/2HsKquMddGioqbW02OIaipJ+5/rGJwxMCSZsiM12BPiSf/U3PU4WFVrOlartTDLjtRErWzRFklgqPPm3Bt3j+GxWWvQ/+G5jp/TvoOVOPbez/DSN1sBNJ51DCUHqzBz+W5Ua4MMXMfQsBgYGsCbi3/G9l8qGuS1Rj75JYZEIU1Q607Mk8ZYkUeSBtLzzNFqNda46zD1s7V4+dstfj/raAeimSs8lzU5ZAnsum0HPGX5fmt8p8uG6tLpi3Hjmz/i1wpPbyha+yQBhpXPDAyOGBhirLLGjbs/WIGJ0xY1yOsFSiv4Y5zlU5Og/ewat/KOMUSWStKeL0p/54dLd+GfX27CAx+vxoUvLHR8XLTHNPTncxqcrapJzM/RibtOoexIDTaXeMa2qmv1wefovYZ3uupRMr4UDgaGGNO/fI3hoiDGSjJReww17voyuiOo8/S/zqlyOFRVi7V7ym3vs1NrKEzJQefPujZGgcEpSFbV+i4MtNq472DCpNQem7UGfR+c400hVdbEIJWk/c8egzMGhhjzVrAJ+h1ctavMe9sYDGojqXX9+HjZLmyJYKaTsYzhnNh1dQplFTXeHoNTALzmlWKMe3pBWGsl/A1oRzsw6M9nF+CUUtiw75Df31+xowxjnvoa077eHNbrRzugvP/jDtPP+or3cFJJ+w9V4eVvt/iUsT6VFF4ZjwYMDDGmt8JVgkaGZ+Zt8N6uNaWSQi+vUipgRXrzWz9h/DMLQn5uXbW7zrvyOZxK++n5G9D3oTn45bD/6aoLNx/wvp6TsiM1KK/0HfS1pnWWbPvVezvaAVcvv13A+d/SnXj807Xmg5ZKcmepZwyi2FDGUNQp4MUFm/Htxv1h/X7RI3NxkSHNav07jlR7xk4CdRj2lVdiyTbzOMrt7yzFAx+vxrq9B21/h6kkZwwMDiqqazH1s7VBdcX90VukwTas/vXVJvT886cRvSZQX2kqpXxOGNPjDOUyVlrBVGDTv9mCez5Y4f35nOe+xXH3feb4eD1fbLfvUbBqDemucOahf7J8F4D6v7vWZozBOJDrLzD0fXAO+j44B4D5amBHatzYuK++MtIXPQL18/IBT2CJ+Pulld9t83cs/bnUe9spFZOqLRAIdxV5bV0dHpm5Br9/cTEAz/ctlIkW+w9Ve4Mw4FtZ69+VQKmkM55ZgPOeN4/tlFZ4gnalwzhLOD3OA4eq8NwXG0P+vcaGgcHB819uwj+/3IS3Itg2Yev+w94ZQrV1Cq8t3Brwdx7/dC2qausiblm+/cN2PDl7Hd75YTvOe34hPlu5x3tfdW2dd5DaeHKYegx+Xr+0ohpXv/wDHv5kNd4wvD/LdpShqtb5945UR1YJAkBNrfKmFezqsuraOny+di9ufusn717+/rjrFCpr3DhsCAZPzVlveD3/n4P+9lmvH2xMzSjTe1z/fH0fnIOz//4tnpq7Hl+s22f7+GDK73le398xpl+cKtY07Xi4KS5rRf7W99sxfOoXWLa9NODv2qUUfXsMwa1jOGCZdPHs/A1YsbPMtow6f/MOlmz71duQMbrzveX4y+x1fssSrF8OV0Mphb3llSFPMY81BgYH+gXD/VV0utcWbsXqXeaBSqWUqSUEAPd+uAo/HwiuNVVR48YDH63C7rLAlZuduz9YgX98sRGbSjw55m0H6k/CO95dhv4Pz0VVrRv7DYPixmCgt2ztzsfXF23D/LX7fO8IwDieEazyyhpTr8Q4QG7Xyp362Vpc9XIxPl62C4/OWuM9friqFnvLK72zXXRbD1Rg6GPz0duwwK/0SH0l46/H4I+x12EspXVMY93eg3h2/gZc+Z8fAHh6NF0mzwq61a1XpHZjJcbKVK/k9pZbBsaDWEXuL1BZK/JibVqsdWyjxl3n03Md+eSX5udy1/lUxvp5GGysrHXXYVPJITw1tz64O/UMnI6v33sQ5z3/nW8aDvBZGBjuGMueskr0f3gunv9qEwY/Oh9DHpuPbQcO48//W2EbkBpa3AKDiIwTkXUislFE7opXOQKx+9grqmu9LXqlFO79cBXGP7vA9CWpcStku1J9frfsSA0OBDFDaeGmA3j5u63447vLUFZR47OA6am563HaU18FfB691Wj8Oz5e5kmnXP1yMZbvcBp81gKD9vOhqlp8pW0OmBrG/gSrd5XjYi3doNuw9yBufusnx97Jwk0H8NistaZeSY1hjMGuMjNWSOlaa3jlzjL0vn82Bj/qu77jj+8uw68V5vfW2BgI9yQ9WOkJDN9u3G+aQlxj+N7YebfYM/j62qJtqKiuDVjx+NsQLtXmY9pTbq7Y9L/PKTA8/MlqHH//bG859pVXehsbgG9A0nsm1tTW45+uxXnPe65JsWa3/Wyvd5fs8Dmmp5KsKb8Zxdtx6fTFUEqZzqfK2jqM/qv5vHD6DJ1Ske9qV4vbsM93bEIf8/A+d5gNB71B9s4P9Vemu/CFhXh90c9Yrb0/Simc//x3mLNqj+1zxFJcAoOIpAJ4DsAZAHoBuEhEesX6dWf8sB2d75ppaiXrHvlktWlQVG9t2X13et0325s3Nq4YdltSMVnpvoHhnv+twIBH5plalNt/qcAlLy62Xb1aWlGDvg/V57J1z87fEHDGiaf8njLpC4UAwJXm+di/sQwY1tYpLN9RCnedQskhTwWiB5Y/zliGy1/6Hjt+rfCmH5zUuuuw/1AVatx1mLNqD5RStmmDm9/6CR8v24V1e+wHBy/69yK89b05lWesiOxObGPR0rWaMdieSmWNG3+bux7fGd4Xfyk1E8tbon++v7cEQ70X6dQT1VvI077ejF73zcZ9H65yfMm9hkrebqwkmG0kvIHBoZKc/s0WHK52Y+VOT2U19umvTRWvdbwoLdWcmlq7pxxr95Tjp5/rB7fPeGaBqQerq7BJNervhzVw3fnecizYsB87S49gwCPzfB5v5JTCdBpX+feCLQCAjDTf6tFaxnAbDno5DxyqPy/13tzOX49AKYVdZZUo3vYr/m/GsrBeIxLx2l11EICNSqnNACAibwOYAGB1LF/05e+2AgB2l1YiPSUF5ZU16NA8GwDw4jdbzA/WzinrbCL9C6qnUsoMrU1jnrOqts4n7wnA20I/VFmL3Iw0VFTXYvjULwDA1DLQKw67k8Wo5GAVyo7UoHurXNvW5aEqz++/8NVmuFJTcOPI7shIS7H9Qi/cfAD3/m8lzjyhDWYu96yo1asWfU7/xGmLfHL37jplymE/8dla/HvBFpzdty0+0nonFw3q6PN6eiDMsulZOTEO1tr9vca8epoWGILt4Xy/5Rc8M3+D6VgwqUQ7P/1civdtWsD3fLASzbJdmDJzjc1v+VZs7xRvx8PnHG/72GFPfO69bZdKCmaKZ6Aeg25veSX6IM+nd1VhaOCUHanxfg/0Ad9xT3saW33a5Zl+b8RfvgxYNqD++28sn3EmmB6wdHZBwGmyw9WvFOOLP56KLgU53mPG71RGWioe+ng19h2sxE2juuOu91f4LCCtrq1DZY0brtQU22nKO0uPYMH6EvQobAJXagpeW7QVrZtmor1W79itWN/+awVueXupt2ffvlnD7/4br8DQDoDx6t47AAyO9osopTBwynxcNKgD7ji9Z32+Egpn/n0Bdvx6xGd7bL2S0wcTjXXP6l3lGP+seaplqeFEMaYqnp2/wRuI7BypceMSS2sy21X/cZRrleZhmy+OsaUz+NF5qFPAsvtPR8lB3wEs4xfv759vxPRvtjgGm1XaYJ0eFIz0V9zxq++YR1Wt21R2vcVlnMJobfkD9YFh9F+/wvw7RqBtXpY3SDilG0orarzjHsb89r7yStQpc8NdT6XYpVTs2FUgTi1CY6Xv1Ku44137lt6UmWtsB8YPHKrymUHTpUUOtv9SgZZNMpBp6YEaZzgt3V6KwV1b4HBVLXIyPJ9FoDRUXZ3yBj7rWEFpRbX3Km0AUOkwe8r4Xer74BwM6NQMQP33t76sgQOsPoXYqNKbSlJYtPkADlXWonVepvf+615fYnr8yp2+vUP9czW+N7o5q/bg2hHdAADr9hz0VsYAkJGegpe+9XyXyytrsdRmQL30SA0GPDIPV5zUGX8c2xMPfbwK94zvhbzsdADAne8tw7cbD/j8nj8lB6tM5TD+vQ0loa/HICKTAEwCgI4dfVucQfw+lKrffVL/gizcdMBUwRlb6l+u24fRxxXaDrp+ZPiwAM9YwwUvfGf72p+u9K1cjXaXHfFJ5Rgrdj1Hba3EK2vc6PdQfVpJP58fnbkG7xRvh9Vey2CZvx7I/kO+PRz9ffBXx1TW1CHb5Xvcbo6/U1lG//UrDO9RgNeuHoxDVbU4w2Gtw4FD1d4K8Ui1G7XuOizcfADXvrbE52/bU+ZJZwW7M6ddGkJ/rbo6hee+2IiLB3dEi9wMU6VfUeUOaRCyfbMs28BgTInoSg5VYfjULzCyZ0v858pBAIAv1u7D8ZYW+GOfrkWf9nm4+N+L8dY1QzC0Wwvbv8fIrRSqtQrfOvtq5ord+N5wOVK9JZ4i5tlg1vdcTxntKTNfCCm4wOD7/TtiSCXp28oM6tLc8Tmuf+NHn2NVNW4s2fYLznt+If51SX/TfemGVsP5z39nSg1npNUH4uU7Sm1fT3+PXlm4FW3yMjGjeAcKcjNw57hjAQTuraaliE9Qtr6nadHcczxI8QoMOwF0MPzcXjtmopSaBmAaABQVFYU1/J+XnY5SrfWif8keM8w2WLT5gKlldPUrxbh2RFd8vmafXgbvfdYT7fFP1zrOkQ5UGekVv9HucmNg8JTZ2OJfsu0XPD1vg+1rznIIRKFsnnbApsXm7Tn5WaDn1FWv8bOtxrn/9N3Oe8EGT6CsqLbfEA4A3v7hZ+/rHal247kvNuFv89bbPnbemr245a2fMPLYVo7PZ2Rt5QLAjl8rcOELC3Hx4I54c/HPWLWrHMe3a2p6zKHqWr9/q9Vqh96QHb2y/GKdZ+C/qtaNK1/+Ace29r1AzKLNns/6y3X7UFpRHXC9yDcb9+OBj1d7y/Tigs24bGhnuNJSkGtpWVfqexalCOoMf+uRGvNnpddx7xRvNzVUAg3STnjuWyzbXooWOS5TClbfBNCYSvo+xOtn7yg9gnu1sRp9l1mdyzCOYH2/XIYNmkor7Bs5eg9FKXOv61BVLVypKeikpYyceOoJ+0V99Y/x+xQxEa/A8AOAHiLSBZ6AMBHAxbF4ofysdO8Jbzcn3W5zuxe+qp+DvmJnGapr6+BKS/FJ67y6cJvj6wYKDHaV0Kvf1T+fXWvbuoDHKBp7G5XZfPmr3XX447vL/PYYPl+zF5cO7Yx2+VnIdqUGNSj+o2HxlZW/fLfx9ypq3Pg5wLTOT1fuQZ/2eX4fo7vXZqD3h62eFrDe+p2/di8+s8wSqaxxh7SK1q5RECw9/bbWZsBe/069YFhDUZDrsu0JAvBOkdU9MnMNvlpfgmcnnugz8Lrj1wpcOn2xTwD8cKm5F+1k+y/+p13r6x5O7JiPeWt8p0JHskrZeD5bG0quVM957UpL8Wm5B7P63xgw9AkW7jqF4++fjTNPaIOcAONndps4WnsM8VigHZdZSUqpWgA3AZgNYA2AGUop5+kXEcjLSvd+eOHstzJ71V5MmelpVVU4tMAmDuzgc8xulojRa4t8g4qxxRJq5RHJamLdZoc9jN5bssNvYLj3w1X4bpNnhkj/js3w1wv6RlSOYIPcrBW7TY2tHq1ybR839bPwFyTNXb0XANBEa0Hb9Qwqa9wR7WobLHedsm1Q6Iwzz3TNc1y4c1zPoF9jwYb9mPRasc+g+5uLf/b26IyCDQzBOuWYlrbH3UqheY5NvtJBZnpwVdue8kr0vn+27W4DwexMa/zc39PGnfTAPHP57oCTR+zOqyM1btPMv0hXx4cjbusYlFKzlFLHKKW6KaWmxOp18rNdWLGzDGf9/ZuwK89XFm5DXZ1CRVUtuhTkoElmfUcrLUXQItf3C+uzkMjCuH7ATiStyliwy4nfMqq79/bF/64fSD9vQHu/zzX39lP83h/sFNFtBypMYzl/Ght8BejPhzee7L2tT222XsgIAEZoldgdM5b5zGYK5Ks/nYqHJvQO6Xd6/vlTvLhgi+P9dr2nJpnpuOHU7jaPdrZ2z0HT7J7UFGmw3XYvHtQRY47zTf256+zXBTlZ8cBYx/tevWqQ9/aCDZ4UnV2r3Lqhnx27YKzLz04POM5jle1KxbLtpabeSzy2Tk/6lc95WZ7ZASt2ljl2R11BTFt54evNqKh2o2WTDDxwVv0JnZ+dHtaCr0A+D2NlcUO77KTOPseCWfDTo7AJurbMsb3v4U9WY5RlgZI/+kc6vk9r2wAdjt5tmwZ+EICRPT2BwS6tE0iWKxUdmvnPP1vV1inbRWC6TTYpvGba7JhQNM1M9zaiFtw5Eu46FVajqk1eJiad0tX7840ju/l9/G1jeiAtNQXPTDwRY44rNN134FA1KqrdttOe7aT7OaeN01Pt1tcU5GYE9RpNM9MCfvahNvCaZbtQbvkdpxlhsZT0gcE6R75Fjssn5dA2P/B0sE9XerqFOa5UU48hRcTxIimhePzcPhE/R7iGdHWe5eFPQW4Gjik0v5fBLvhxWiQ33bqeJAjHt2uKv1/U39sIMArnfU1LTcF71w0N+Li+HfJDfm5dtivNuxhM17ppJlIEuHxoJ8y4tv71Lx/ayXvbX67dWqEAnh5zqPKy0r0THIKtJI308Ym7xx/nTcE1yUhDt5b2qT5djjblOScjDbeN6WG6b2fpEfxyuNq0aNTps33Xz2eXkWZeb2A3/pLlCq5aLGzqv94orajBYoeB8ptGdscDZ/mu6W2W4/sdZo8hBqyrjy8d2glt880LRtoFsYBk+Y4yrNhZhpJDVRjarQXaaHOLh/do6Z0nP6hzc5wfII3i5PTerb23p55/QljPoQt1FoM+tS6Qlk18KwnrgJ2em371qkE4f0B73HWG/XNHcnH3M45vbfo525WG1BQxraXQdWyR7Xd6o5MBnZqhX4d8jHKY0VT85zE+6wpCkZWe6tOqLczLxObHzsSDE443ldku4AXL2mOwC8gzbxlm+rmgSYZ3eqbd6t+2DvPqRYDVD43FrFuHY+p5J+Csvm2RoeX6Myw5/56FTTC+j/lzNM4QsgZNXWZ6Csb3aY1h3QswcVBH9O+Y7/N9ty6mM1o0eXTAhpzdjgV2QumhDuteYPp50oiutusTmtkEcvYYYsD6IWe7Un1y2G3zfAPDE+f1wX+uGOhzfOXOcjTJTMfCyaPx7V2j8PA5vb0tkIz0FDx5QV9Mu3QAJjtUiAAwvEeBzzHjYNmFRebB7HSbk+TTW4c7Pr/dTpr+0iMndsjHn888Di9f6fv36trlZ6GFNvg3vk9rnNgxH4BvC1ZPJZ1yTEs8eUFfXDeimzflYhTO3mODOjfHJUM6+gQy/UR3ykG/8YfBWP3QWMy9/RQsvnu07WOuOrmL6WcRwX+vPwnTLy/yeezJ3VugINd3wVkoUlPE+7l2KcjB4C7NMcVhhXOen1Z/oPhq7THYbRzXq43nuzGka3N0bZmDr9eX4FNtN1671bzWBoK+MtszK83TM7hQm5ChrwWwpmsL8zLRv2Mz0zFjusqpR5mVnop//n4AXv+DZz3sf284Gef1r2+MDe9R4P1cZt4yzNsbLmyagWcvOhHNclwBG05XWr4LS/48xvZxh6t8K2y7iSiZ6Sm4z9I7yEpPtW3ItLAZYGePIQYyLZVFlivNp5Vr1xL+3cCOGHlsK59c+NO/6+e9rZ8IeutXn/V0eu/WuHZEN+8qUKuWuRneE/r/jTsWV5zU2bGVsuWx8Zhz+wjTsdyMNBzXxrmit5t99f71J5l+1luhAzo1g4jgD8O74tSerdDRZt71A2f1wqtXD/LObX/qwn744AbPAK11UNJuSrBdeVo1DT1F0TQrDY+c0wddCnIw7dIB3q0C9EBoV1ELBOmpKch2paFHYRMUNs3E/512jM/jjmvjuy4gJUVMZf/ghpOw+dHxeOMPQ7TXcz59rL0aO3qPISMtBe9cO9Rn0ZpOf50uBTm4sKi9aXA2UM9Ln8nzp7E9kZ4q3jEZYwUmIvj8jhGYfvlAn91nAWD2bebJAtZA1VJLN9mN1elld1l6Hq1szrkK04C3/Xtr9xnrAeWZif3w2tX1Gyj0bpuHrlr6ql+HfJzdty0A/1fYAzzbt7zxh/rnaWFJpz3/+/544rw+tpvsPX6eb2//45uGmQLddSO6ad9J37/Fev6N7NnStiEZa0kfGKwVbnqKeHdGLdQqJ3+5wk9vHY41D43z/nzOie18HqO3WK1ft7euGYKTu7fwefxxbZp6HzugUzM8cHZvn8pTH3wTEZ+/IdBAoF1X2XhCPXFeH8y+7RR8f89ovG44kQBgym+PN7Va3p40BFec3AXdWuYiNzMNrtQU03M59RiM9NI8NKE3frr3NADAo7/t4zOt9aUrfFvnRsY546f3bo0LBngqN/3PtaY9rjq5CwZ29g3Og21SS02DSNe0zc8yVSqZac49lDNPaOP9uSDXhZ6FTXDP+ONMj7M2KJzo73HPwiaYen5fnNarfmA2UEJO/47fOLI7NkwZ7z3e2xKEurbM9dkuQndMYa5pEDnf8l411cbc7AZ8czM8j1UARh9bX+4WOS7v35+vpbv6dagvk/E7bJymarevlj4ekuHweQDm62UEk8Y82ZL6MY75DOzSHL8b2BE9WtU3JlxpKabP3KhVk0xv7+ACQ3rV7m851ZK6/M+Vg/CXCKd/h+OoCwwpUr8E/a8X9MOLlxX5DEYbF6VkpKUiy5WKx8/t45gecmqBuNJS8K9LBuAKw+ydZyb2w1XDungrA2Pax7ja9J+/7++tRK0tU72i+PtFJ+LlKwfimYn9TGmgQFe7GtCpOVrnZaJVk0yfL+fwHi2xRHtdABjStT6w5WSkmQbeAd8xBruppvp5WNg0E820k7xD82yM6WWeedIuPxuf3eacIrO2OvV6yHuxHMsJf99Zvbwb6RkN7toCPxr+RgA+f5edlpaWozVvDgArHjgdJ3cvwDGF9ZVGighm334KTu9t/nvrL/LjXz9tkFufBmyswPX3v6lD+Vs1sW/0FDisCbB7HhHB3eOPQzttbC7fMm5RqOXKB9tMYtB73PvKq5CXnY6rh3nSNM1zXN7vxTn92uGHe8ZglCFwpBrSp8bvhF1P7OZR3dE2LxNDu/o2wgK9t8YUa7YrFXcYepO/K+qAf1x8IgDzNhx6S//lKwfid1rad9Hk0XjuYvN2G7osVypa52Xi45uG4ZHf1qcLc2xSSf07NsOCO0cGKHXsJfReSdFgnGHwylWDcFK3FnArheU7VqBX26ZonuPCD5bVkBNsegUT/UyTcxgnA+CZR35W3zbeDfUm9PM8t/4rxvGDRXeP9lb6rrQUuNI8J6+xhd4uPwvXjfC03s7SusZW1rjw/vXmWRrBTM+1M/rYVii0VDTWffftxmucTk/r2IkI0LOwKVyW3V8LcjNw6+jupgF6ILzrQuisi6WaZqbj3euG+l2oZm0A2PUYmmR6Ks1jCptgzu2n4PS/fe1N31hb1Po2I4EasMcUNjFt9mhs7GS7UnF82zzcOKo7Jr+/HLvKKjH39lNw2t++BmCfsgF80yO6mbcMx0fLdtlepUzfHkYftzi+XVPcPKoHurXMxSc3DzMFQ50+NVRPHervb7Mcl3cnAaWUTzrXmHoxDsjalbtvh3x8N9l+7Eh/b43vsd7QumVUd/zf6T3R98E5KDtSg3n/N8I0MeUJh0kg+ufeIjcDj5/XBw+d09tvb0Vv0FhX4BtTSaOObeWdoq7v+Gw3XtFQkj4w6JWqSP2CpIsGdTTNh9ZP2PRUwaxbhpvmOQdDrzCcTnC7TbD0xxrvs+5PozOmSL69a1RI5VEKPimC9LTwZgSd2789zrU0ii4d2hnPaou7bhzZzbtTpZHTRnzWilIfr7BuC54intex8vYYbPZx+u8NJ/kc8yfLleqY43/zmsG2A4D6+3zx4I4Yf3wbrLRc98EagPXKTh/f0d+PQKkN6/MY988a3qMAL1zqScHNu2MEamoV8rLT8ZfzT8DDn6x2DABOM/E6NM/GiGNa2gYGPcDp5c/PcmGsFqydx0dSMf3yIvTU9nbSA0OLHBeKtDE4u4aY/p6kp4rfNQnBMr7F2a40rH/kDG/DRO9hB5NOBMwNBBHxGxT8MfbWX7h0gGmMZd0j45Aeh83zdEdNYGjtZxxBP/EKcjPQw6bVE0igE9tu6p1om2fZzTiyfyxwTj/7HoJV5xY5WFpRirQUQY1b+XRZo7lb4+1jeuDW0T2QIs658pE9W2Hu6r3oZhnIN7YKrzy5s3djuMz0VNPCIKdtEvz1GKwzXuy8fvVgXDLds2LbrluvO6mb8+DfpkfHe//2YZZBwnRvQPfUqAW5GbhoUAdcrqUWe7Zugj7t8vDA2f6vUWXtqZzWqxAXFrXH+QM6mKZmZrvSAK1xfUFRB1xQ5NzitKbFjJyuj6EHYD3dFOwK9dGGxWpn9mmDr9aXoHfbPLTOy/TZ9l6nt6ZvHBnaqm0r67W4dca05OtXD8asFbsD7mv0zqQhQW3gN/+OEbjm1WLbgXwj46yk9NQU5GXVlyncYBMtSR8Y9O6vv7nsLq0F7XQN2ED0FodTFW/X4tEfG2g8QLf24XFBt5z+fVkRvtu0Hw9+vBq/HK72mckRbirJjoj4TaUBwEWDOmBs70Kf1quI4D9XDkTvNk3RyhC4bxrZHfd/5Nk6680/DMYAmwFkwH8KLxjDehRg4eRRmL9mX9h73vv7/PSg7+0ZpAgeO7c+PZGZnoqPbx5m96t+ZbvSMPX8yAYk/TVInKb96n+H/n0KZ2O7Cwd2wHkD2gf83udkpGHtw+O8veUrT+6Mvu3zQ349byrJz2hDr7ZN0SuI1e6Du7bAYJtxDKtuLXPx2a2n4Bib/ZeMgj334yHpA0P/js3wyDnH49z+vt1VnV7hhruLYaAFM7aBQZzvsxPKnPmWTTIwoV877Cw9gqmfrUPTLEuPIYga9dSeLdE0M/yFVUYi4pjSGNnTdwHZ5Sd19gaGbq1yHVtPqdp7Z4znx7ZuEtIJ1yYvC5cM6RT4gWHQvxfh7jL0/vVDUbz118APDMHd44/Fc19s8jsTymnq9JTf9sFjs9Z4Z/EFs/uonWA/H+N3/v6zQttXykcD18GutBQsu/90VAWaQZgiuGxobL5/kUj6wCAiAU/8dJsKJrTXqH8tO3aLdfQWTDCVdLDuOuNY74psALjh1O62G6gFE4xevnJQwMc0BH9pOruA/Jllzn08hbObr9GATs0xoFN425U4mXRKN0w6xTMO5DT7RU8lWce8TutViNN6FWKFtgFkbbDXw46jeLbJ87LSgQDjFpseHe/3/nhJ+sAQjDRvlz82LSC7ilj/lWh2J6+zGfg1+u2J7fDBTzsdV5UmIn/vTxQzYjHRLDsdV57cOextUmKtg8NFZDLSUvGnsT19NrLT6edLQ+24Gg2N5xufGBgYEHmXP9AYg/PgMxp05sHU80/Afb/pFXDlZyLxl6aLxa620SQikadA4sTfoK8+PmF3kZlEo59nkfbejjaJfWY1EH0RVJswByADzUqyq/z132jISjo9NcW7wKyx8Ff3J3qPIVm1b5aN3Iw03Dk2uM0XqfFhjwGeXOCzF50Y9vbT9Vsb2N9v12MY2KU5Pl+7L6jpqkczf1Nr9R5DuGNDFJ7M9FSsfND5QjiJiGdZaBgYNGc7rCIORqCWq11g+MfFJ2JzyWHbHRapnt8eA9MDFIDdymcKjJ3xKKjPXzps/WBTu2W70hxXi1I9/2MMnv/tVj4TAf7XL5CzmAUGEfmLiKwVkeUi8oGI5BvumywiG0VknYg0rj5pGBrTYG+i8T8rie0aCg7PwNDE8syaC+B4pdQJANYDmAwAItILwEQAvQGMA/BPEYnv+u8oYXc1+vzNJuHgMwUSaI0R2YvZqaWUmqOU0je8WQRAn8w9AcDbSqkqpdQWABsBJMZqqjBx8DM+OPhMgTAchKeh2lxXAdA3DmkHYLvhvh3aMR8iMklEikWkuKSkJMZFpMaGg88ULH5TQhPRlBgRmQfA7hqG9yilPtQecw+AWgBvhPr8SqlpAKYBQFFRUQK3C7V99eNciqNNinfwmchegHkh5CCiwKCUsr9KtkZErgDwGwCjVf1+EzsBGPcDbq8dIyKKKo4thCeWs5LGAbgTwNlKqQrDXR8BmCgiGSLSBUAPAN/HqhwNid/BhqVPReTbToFw2mpoYrm66h8AMgDM1aL2IqXUdUqpVSIyA8BqeFJMNyql/O9Nm+A4+BlffPvJiTeTxLgQkpgFBqWU4y5cSqkpAKbE6rUbml4x+WuVjD62Fbq1ym2YAh0leLJTQPyOhIX7MTSQ6VcMjHcRkhe7DBQA40NouEQoitiCJUos3nEonpshYWCIAo4xxAfPdQqEASE8DAxRUNjUcz3jHoVN4lyS5PGbE9qgWXZw15zmJnoUCGclhYZjDFFQ1Lk5Zlw7FAM6NYt3UZLGPy7uH/AxnKNOgXBWUngYGKJkUJfoXrSdAuuoXbN4fJ82cS4JUXJhYKBGq3VeJtY8NA6Z6cyIkj1eqCc8DAzUqGW5kmLHdoqR+rEFRoZQsKlFREQmDAxElLSYSgoPAwMRJS0mksLDwEBERCYMDESUvIRbYoSDgYGIklZ9KomRIRQMDEREZMLAQERJiymk8DAwEFHSYgopPAwMRERkwsBAREmLqaTwxDwwiMgdIqJEpED7WUTkWRHZKCLLRSTw/spERGFgXAhPTAODiHQAcDqAnw2HzwDQQ/s3CcDzsSwDERGFJtY9hr8BuBPmy7VPAPCq8lgEIF9EuKE+EUUdU0nhiVlgEJEJAHYqpZZZ7moHYLvh5x3aMbvnmCQixSJSXFJSEqOSEhGRUUTXYxCReQBa29x1D4C74UkjhU0pNQ3ANAAoKirihX2JKCS8/Gt4IgoMSqkxdsdFpA+ALgCWaR9MewA/isggADsBdDA8vL12jIiIEkBMUklKqRVKqVZKqc5Kqc7wpIv6K6X2APgIwGXa7KQhAMqUUrtjUQ4iIgpdPC7tOQvAeAAbAVQAuDIOZSCiowAzSeFpkMCg9Rr02wrAjQ3xukR0dOOWGOHhymciIjJhYCCipMVUUngYGIgoaTEuhIeBgYiITBgYiChpMZUUHgYGIkpanJUUHgYGIiIyYWAgoqTFVFJ4GBiIiMiEgYGIiEwYGIgoaXHb7fAwMBBR0mJYCA8DAxERmTAwEFHSYiYpPAwMRJT0GCBCw8BARElLjweKV4wPCQMDESUtzkoKDwMDESU9xofQxDQwiMjNIrJWRFaJyFTD8ckislFE1onI2FiWgYiIqaTQxOyazyIyEsAEAH2VUlUi0ko73gvARAC9AbQFME9EjlFKuWNVFiIiCl4sewzXA3hcKVUFAEqpfdrxCQDeVkpVKaW2ANgIYFAMy0FERzmmkkITy8BwDIDhIrJYRL4SkYHa8XYAthset0M75kNEJolIsYgUl5SUxLCoRJTMmEoKTUSpJBGZB6C1zV33aM/dHMAQAAMBzBCRrqE8v1JqGoBpAFBUVMSPloioAUQUGJRSY5zuE5HrAfxXKaUAfC8idQAKAOwE0MHw0PbaMSKimGAqKTSxTCX9D8BIABCRYwC4AOwH8BGAiSKSISJdAPQA8H0My0FERzmmkkITs1lJAF4C8JKIrARQDeByrfewSkRmAFgNoBbAjZyRRESUOGIWGJRS1QAucbhvCoApsXptIiIjppJCw5XPRJT0mEoKDQMDERGZMDAQUdJjKik0DAxERGTCwEBERCYMDEREZMLAQEREJgwMRERkwsBAREQmDAxERGTCwEBERCYMDEREZMLAQEREJgwMRERkwsBAREQmDAxERGTCwEBERCYMDEREZMLAQEREJjELDCLST0QWichSESkWkUHacRGRZ0Vko4gsF5H+sSoDERGFLpY9hqkAHlRK9QNwn/YzAJwBoIf2bxKA52NYBiI6iile7DkssQwMCkBT7XYegF3a7QkAXlUeiwDki0ibGJaDiIhCkBbD574NwGwReRKeAHSSdrwdgO2Gx+3Qju22PoGITIKnV4GOHTvGsKhElIyEF3sOS0SBQUTmAWhtc9c9AEYDuF0p9b6IXAhgOoAxoTy/UmoagGkAUFRUxD4hEYWEqaTwRBQYlFKOFb2IvArgVu3HdwG8qN3eCaCD4aHttWNERJQAYjnGsAvACO32KAAbtNsfAbhMm500BECZUsonjUREFCmmksITyzGGawA8IyJpACqhjRUAmAVgPICNACoAXBnDMhDRUYyppPDELDAopb4BMMDmuAJwY6xel4iIIsOVz0SUtJhKCg8DAxElLaaSwsPAQEREJgwMRJS0mEoKDwMDESUtppLCw8BAREQmDAxElLSYSgoPAwMRJS2mksLDwEBERCYMDESUtJhKCg8DAxElLaaSwsPAQEREJgwMRERkwsBAREmPIw2hYWAgoqTHkYbQMDAQEZEJAwMRJT2mkkLDwEBESY+ppNBEFBhE5AIRWSUidSJSZLlvsohsFJF1IjLWcHycdmyjiNwVyesTEVH0RdpjWAngXABfGw+KSC8AEwH0BjAOwD9FJFVEUgE8B+AMAL0AXKQ9logoZphKCk1aJL+slFoD2C47nwDgbaVUFYAtIrIRwCDtvo1Kqc3a772tPXZ1JOUgIvKHqaTQxGqMoR2A7Yafd2jHnI7bEpFJIlIsIsUlJSUxKSgREZkF7DGIyDwArW3uukcp9WH0i1RPKTUNwDQAKCoqYtAnorAwlRSagIFBKTUmjOfdCaCD4ef22jH4OU5EFBNsVYYmVqmkjwBMFJEMEekCoAeA7wH8AKCHiHQRERc8A9QfxagMREQUhogGn0XktwD+DqAlgJkislQpNVYptUpEZsAzqFwL4EallFv7nZsAzAaQCuAlpdSqiP4CIqIAmEoKTaSzkj4A8IHDfVMATLE5PgvArEhel4goFEwlhYYrn4mIyISBgYiSHlNJoWFgICIiEwYGIiIyYWAgIiITBgYiIjJhYCAiIhMGBiIiMmFgICIiEwYGIiIyYWAgIiITBgYiIjJhYCAiIhMGBiIiMmFgICIiEwYGIiIyYWAgoqSVluqp4tJTWdWFIqIruBERJbILizpg+y8VuHl0j3gXpVFhYCCipOVKS8Hk8cfFuxiNTkT9KxG5QERWiUidiBQZjp8mIktEZIX2/yjDfQO04xtF5FkR4cWViIgSSKSJt5UAzgXwteX4fgBnKaX6ALgcwGuG+54HcA2AHtq/cRGWgYiIoiiiVJJSag0AWBv9SqmfDD+uApAlIhkAmgNoqpRapP3eqwDOAfBpJOUgIqLoaYih+vMA/KiUqgLQDsAOw307tGO2RGSSiBSLSHFJSUmMi0lEREAQPQYRmQegtc1d9yilPgzwu70BPAHg9HAKp5SaBmAaABQVFalwnoOIiEITMDAopcaE88Qi0h7ABwAuU0pt0g7vBNDe8LD22jEiIkoQMUkliUg+gJkA7lJKfasfV0rtBlAuIkO02UiXAfDb6yAiooYV6XTV34rIDgBDAcwUkdnaXTcB6A7gPhFZqv1rpd13A4AXAWwEsAkceCYiSiiiVONI3YtICYBtYf56ATxTaBNRIpcNYPkilcjlS+SyASxfJPSydVJKtQz1lxtNYIiEiBQrpYoCP7LhJXLZAJYvUolcvkQuG8DyRSLSsnFnKSIiMmFgICIik6MlMEyLdwH8SOSyASxfpBK5fIlcNoDli0REZTsqxhiIiCh4R0uPgYiIgsTAQEREJkkdGERknIis0679cFecyvCSiOwTkZWGY81FZK6IbND+b6YdF+0aFRtFZLmI9I9x2TqIyBcislq7rsatCVa+TBH5XkSWaeV7UDveRUQWa+V4R0Rc2vEM7eeN2v2dY1k+QzlTReQnEfkk0conIlu1658sFZFi7ViifL75IvKeiKwVkTUiMjSBytbTsDh3qYiUi8htiVI+7TVv186LlSLylna+ROe7p5RKyn8AUuFZWd0VgAvAMgC94lCOUwD0B7DScGwqPNuFAMBdAJ7Qbo+HZyW4ABgCYHGMy9YGQH/tdhMA6wH0SqDyCYBc7XY6gMXa684AMFE7/i8A12u3bwDwL+32RADvNNBn/H8A3gTwifZzwpQPwFYABZZjifL5vgLgD9ptF4D8RCmbpZypAPYA6JQo5YNnV+otALIM37krovXda5A3Nh7/4NmmY7bh58kAJsepLJ1hDgzrALTRbrcBsE67/QKAi+we10Dl/BDAaYlYPgDZAH4EMBieFZ1p1s8ZwGwAQ7XbadrjJMblag9gPoBRAD7RKoZEKt9W+AaGuH++APK0ik0SrWw2ZT0dwLeJVD54AsN2eK5xk6Z998ZG67uXzKkk/Y3T+b32QwMrVJ4NBQFPS6RQux23MmtdyxPhaZUnTPm0NM1SAPsAzIWnF1iqlKq1KYO3fNr9ZQBaxLJ8AJ4GcCeAOu3nFglWPgVgjngusTtJO5YIn28XACUA/qOl4V4UkZwEKZvVRABvabcTonxKqZ0AngTwM4Dd8HyXliBK371kDgyNgvKE8LjOGRaRXADvA7hNKVVuvC/e5VNKuZVS/eBpmQ8CcGy8ymIlIr8BsE8ptSTeZfFjmFKqP4AzANwoIqcY74zj55sGT4r1eaXUiQAOw5OaSYSyeWk5+rMBvGu9L57l08Y2JsATYNsCyEEUL5OczIFhJ4AOhp8T6doPe0WkDQBo/+/Tjjd4mUUkHZ6g8IZS6r+JVj6dUqoUwBfwdI/zRUS/loixDN7yaffnATgQw2KdDOBsEdkK4G140knPJFD59JYllFL74Lk+yiAkxue7A8AOpdRi7ef34AkUiVA2ozPguQLlXu3nRCnfGABblFIlSqkaAP+F5/sYle9eMgeGHwD00EbpXfB0Bz+Kc5l0HwG4XLt9OeqvSfERgMu0GQ5DAJQZuq1RJyICYDqANUqppxKwfC3Fc20PiEgWPOMfa+AJEOc7lE8v9/kAPtdadTGhlJqslGqvlOoMz/frc6XU7xOlfCKSIyJN9Nvw5MpXIgE+X6XUHgDbRaSndmg0gNWJUDaLi1CfRtLLkQjl+xnAEBHJ1s5j/f2LznevIQZv4vUPnpkC6+HJS98TpzK8BU8OsAaeVtLV8OT25gPYAGAegObaYwXAc1p5VwAoinHZhsHTFV4OYKn2b3wCle8EAD9p5VsJ4D7teFcA38NzTY93AWRoxzO1nzdq93dtwM/5VNTPSkqI8mnlWKb9W6WfAwn0+fYDUKx9vv8D0CxRyqa9Zg48reo8w7FEKt+DANZq58ZrADKi9d3jlhhERGSSzKkkIiIKAwMDERGZMDAQEZEJAwMREZkwMBARkQkDAxERmTAwEBGRyf8HPM0V0BN843YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(hidden_states['h.11'][0][0].cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b49e1a5e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2289dac3dc0>]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABXAklEQVR4nO2dd7wdRdnHf885t6UH0khIIBVCkB5CCSU0aQIWVPBVUSmCIIo1wCtYXhUrovCC6IugUhUQFDDSBKQkhBJCQiAhhCSkkl7uzb33nHn/2J3d2dmZ3dlyyr13vp9Pcs/ZNnN2d+aZp8wzxBiDxWKxWHo2hVpXwGKxWCy1xwoDi8VisVhhYLFYLBYrDCwWi8UCKwwsFovFAisMLBaLxYKchAER3UJEa4jodc3+aUS0iYhedf9dlUe5FovFYsmHhpyucyuA6wH8MeKYZxhjH8qpPIvFYrHkSC6aAWPsaQDr87iWxWKxWKpPXpqBCYcR0RwAKwB8gzE2L+rgwYMHs9GjR1elYhaLxdJdeOmll95njA1Jel61hMHLAHZnjG0lolMA/A3ABPkgIroAwAUAsNtuu2H27NlVqp7FYrF0D4jo3TTnVSWaiDG2mTG21f38MIBGIhqsOO5mxthkxtjkIUMSCzaLxWKxpKQqwoCIdiEicj9PcctdV42yLRaLxRJPLmYiIroTwDQAg4loOYCrATQCAGPsJgBnAriIiDoBtAI4i9l0qRaLxVI35CIMGGNnx+y/Hk7oqcVisVjqEDsD2WKxWCxWGFgsFovFCgOLxWKxwAoDSwXoLJVxz4vLUCrbGAGLpatghYEld257/l18697XcMespbWuisViMcQKA0vurN+2AwCwaXt7jWtisVhMscLAYrFYLFYYWCqHnVZosXQdrDCw5A6Bal0Fi8WSECsMLLnDYFUCi6WrYYWBpWKQVRAsli6DFQaWimF9BhZL18EKA0vuWJ+BxdL1sMLAYrFYLFYYWCwWi8UKA4vFYrHACgNLBbH+467HH59fgrnLN9W6GpYakMtKZxaLpXtw1QPzAABLrjm1xjWxVBurGVgqho0psli6DlYYWCwWi8UKA0vlsD4Di6XrYIWBJXdsGgqLpethhYHFYrFYrDCwWCwWixUGFovFYoEVBhaLxWKBFQYWi8VigRUGFovFYoEVBhaLxWKBFQaWCmJXOrNYug5WGFhyx845s1i6HlYYWHLHKgQWS9cjF2FARLcQ0Roiel2zn4jo10S0iIheI6ID8yjXYgGAd9dtw23PLal1Nbo8zNr1ejR5aQa3AjgpYv/JACa4/y4AcGNO5Vos+PhNz+PqB+ehraNU66p0aaws6NnkIgwYY08DWB9xyBkA/sgcXgAwkIiG51G2pf6ots9gU2tHlUvsnlhZ0LOpls9gVwDLhO/L3W0WS27YkW02rJmoZ1NXDmQiuoCIZhPR7LVr19a6OpYugk2ZnQ9WFPRsqiUM3gMwSvg+0t0WgDF2M2NsMmNs8pAhQ6pUNUulYFXqXviAVizvH6+twNYdnVUpv7tgFYOeTbWEwYMAPutGFR0KYBNjbGWVyrZUmxoN1ctuZzZ/xWZccscruPy+uTWpR1elWsLbUp805HERIroTwDQAg4loOYCrATQCAGPsJgAPAzgFwCIA2wF8Po9yLWEWrt6Cxe9vw4l771K7SlR5iMllD7d5b293NIIVG1urWo+ujtUMeja5CAPG2Nkx+xmAi/MoyxLNCdc+DQBYcs2pNa4JQFWOK+oKfdmy9dsxpF8zWhqLta6KxRKgrhzIFksWWLnWNYimvbOMI3/6JC67+9VaV0WJ1Qx6NlYYWCpGtW3Q9W7zLru97eML1tS4Jmrq/f5ZKosVBpb8qbIDmZujyl2kLyvXaUWtZtCzscLA0m2QJ03V6ySqcp3Wqz5rZakWVhhYug18wF2vk9C4DKhTxaBuhaelOlhhYOk21LvNu141Ak59185SaawwsHQfWOTXmlNv9ZGpc1llqTBWGFi6DfVqfuHUvRmmzqvX1Zm7fBMenb+61tXQksukM4ulHvDNRPXpNKj3vrbezWxdndOu/w+A+pgQqsJqBt2UeghfrNZAmDuMs/zkFRtb8ZvHF1Z09F73ikGd189SWaww6KbU0llZq3F5lo78wj+/hF88+hbeXrs1xxpJ2M7WUsdYYSBx2d2vYvT0h2pdjcxkVQwenLMC9760PJ/KVAlfFiT/8dVId13vZpj6rp2l0lifgcT9r4SWWTCmvbOMpob6kK9ZNYNL73wFAPCxg0YmPrdWnYq3rkGaCnjnVE6vqXczTN07uC0VpT56rm7AQ6+txB7//QgWrt5S66oAqP+Y9krAR95ptCLP9VxBG1e9P5F6r19adnSWsKOzVOtq1D1WGOTEjHmrAADzV26ucU0cSnXgQK42/CenEYR8VFxJf0e9j7zrvHqpmXTVDBz8P4/Vuhp1jxUGOVFv7aiWsiBLh1ouM9z67DveAjVJ4J0tFwZJOjd+vwoVVA3qXT7Xu08jLaUyw+a2+lkCtV4HBVYY5ES9PeB6CC0FnMVc7pq11Pj4R99Yje/+fT5+8siCxGWVM/gMuABJIgxef28T9rl6BtZu2WF0fN13tnVevWoxY94qvLZ8Y8Wu31knbVPGCoOcIakzmb9ic0065nrxGXzyt89j+n1z0dZhZrPd5kb1pBvJBTWDRGemSHJ389OLsWVHJ55d9L5hIYmrVVXqvHpV44t/egmnX/9sxa7fWarPO22FQU6oHu9ryzfilF8/gxufervq9amHwQcD8P7WdgDmnayXeTRBOfxY32eQ4OQMJHU618EjiSRKhi5bvx2jpz+E2UvWV69C3ZSOcn0uyWeFQc6I/cLKTW0AgFeXbax6PWo66Uy4CUnrwY+XNSwT/BTRGRzICYpNWte0j6RUZvj+3+djlfs+VYooM9bzb68DANz94rKK1qEnULKaQTdHMjMwxjz7c15mIsaYsW8ibYeYh+9DvARTbIurAwAUUvhxfcdx8t/gr4WQoGDP6Wx6eLp7O/Oddbjl2Xfwzb/OSXW+KVG3raHo/Mh6tXd3Jaxm0M3hDZ0vwfj1e+bgW27jLeXQwa7Z0oYxlz+MOwydsWlCS7/74DyMufzhxOfpICTvmNPY7uVz07S1NILE0wwMjVppXwN+XqVtzVFXbyg6XUVHKb+ObNWmNsxcvC6361WajlIZX77zlcwpS6zPoJsjd2LLN7Riw/YOAPnE/C9dtx0AtCkiGGO47+XlwvfkZdz2/Lup6qaDIYVm4P5NE+LJsjiQ+d8Ep25qdZ6vuWaQjmrleooShA3uj8yzI/vgtU/hkze/kNv1Ks2ryzbi73NW4Ft/fS3TdawwqDGPv7Ea77y/rWLXl9uRaBLIw3zvmZ80+598cw2+do9vRqgXn4GXIsKwK/Tt8MnLzbKsZNLbtXJTK55z7ejGDuQEhSxeuxUfvPYprN/W7p9fYRd0pJmIC4McTRz1FPtvAr8/aUyYItZMVGPOvW02jvn5vyt2fd9M5H4XGlY+HXP0G7i5NdiwsmgjlZgzYXrJVLZ7qYx09U82UW3FRtGZm7+Z6Kan3sZbq7fi0fmr6mJ5hkbXTNSTfQZZghtErGbQQ1CN4PMwE/mOafV+edSYpchKKBWml0zjQOaNkzfWNL/dm7BmWFOxfuaaQcJK5Xx+lusX3R/87zfXYnt7J5at347xVzxcN7m4qoHvI8pGnn6XPLHCICfkhiRqA3loBp7GkaL8pFSizzEdrfuqeLjJLd+wHTc8uUh7Lb7VcwZXoH4csX6mnUNaMw93UFd6PBlVv8/eMsv7/O667Xho7kp0lhn+0oXSnGdOU54huEHkhicXYdn67dkuUgF6vDB4Y+VmzxGYBb8ZuQ1XaFe1SBqXRRhUwt9gesWo0dc5t8zCz2a8iTWa9A++ZuDZi4zrlzSVhSgMTJ3daW9rJTOpipjWr1lI014HFiwlm7Z3YMGqYNLIT/9+ZqZrZgluEHnk9VU45w+z4g+sMj1eGJx83TM4uwIRDWK7ykMWUIydKKQZZNBEK2ImMp5n4PxV2WX5pKsGjQ1JLmPD9g7MMZzwxxJqE4WC+nNkGYbXDpwjnqS5wPNvr8Pvnl6c4uo+naUyrnpwntGxxQLVfYbTM296Dif96pnAtqyTP7MEN8i0tddfSu0eLwwAddrptPHxTDEqrYqZSPqezUxUu5Ye1eC2uQ1IX7ugZrB0/XaccYNZjhk/tNTUZyCaiZzPD85ZgdHTH8J7G1vVZVSoBz37dy/ghw+/kekas95Zj6ffWqvcJ9c7oOnWqWqwcE3+y5dGmTBNEE8rZA1JqgA9QhikmQGcvN06J5QUI8xqOJBlMgmDNA7YMkNr1GgnoWYQ1eB0v83LTZRCK/JDYIO89O56LFGEJAfq5378q2s/f0vjVE3zRKpmIoraJ+0slVlokmUWHp2/Gt/7u5lWUkuyDurEO6XTbmtJjxAGaWYAJ33wcoy7eH4uwsBzIppdK1toafJzfvX4Qux11T+D/hfhQqb15sdFtRVtRJXi3puiG7V/7MbnMe3n/0Z7Zxn/9fsX8MrSDdpr8EGHqRlLRVtHycvcyql1tyHfzxJjmWaKy5z/x9n4w7NLsl+owmT1GYimz26rGRDRSUT0JhEtIqLpiv2fI6K1RPSq+++8PMo1RdcxRqntSbsT2cwQyM/DHIfWHTOXVsxUIF83U2hpijHs39y1ozdsa1eOFtPOM3hwzgqskMwuvHPqKJWxZkubkLU0fO9NiXMgL35/K55dtA7fvteZfRq4R+5H/p4VtZ1FfMWO+8VT2PvqGf4ZgWIYfvjQfPwoo0koKXKtxfZUf11a5UiTzFBEPK2SiyilpSHrBYioCOAGACcAWA7gRSJ6kDE2Xzr0bsbYJVnLS4NupBjVaaRVCXlDCUQTMYZv3/sa/jlvFT6wa3/sO3Jg4uvGzzMIUm0zUdwMadNLiiPOcpnh0jtfwYgBLXju8uO8Y3hfdOX9c3HP7OVoLAYjuNKlo/DEuXI/Nz35yQfD53INVDfqMxHQ3N8gClRxRPm7Z94BAFxxyl7xF8sJpZmo3j3IFSCrz0BEP2CoHXloBlMALGKMLWaMtQO4C8AZOVw3N7SaQcQ5Sd91f8nF8LXLZYb3tzrhkG0d2SacGI+wM6gGv3rsrcTn+DOvmVKzSJpttUDkXWXl5mDqZv7bHnndWXe6o8QFcPAZqHhywRqlozQuGlVeCS0wj8R9pJ5mkMFM5B2rvIfm5+eJXJeAZlDDPo0xhjdi1hzPU2jp1trYuqPTaN5AT3Ag7wpATHK+3N0m8zEieo2I/kpEo3Io1xidMIgaQSYWBtI1xZewxFggtXUakk6kymIm4qPPJMRN0TfXDLhjMvmMT/kZqPj8rS8GJlD55UbXU+7oxSJ4ebHCIIH5TdSQ4rSuSqPWDGpTF5G7XlyGk697RhsFBUS3g3KZJZpjpDMTnXnjczjyp0/Gni9qe8U69NZWq0p/BzCaMbYvgEcB3KY6iIguIKLZRDR77Vr9A05KGmGQ1sxSVpiJyoxlnkXq27TN7ES1SlTHoI4wSTrPoECk7aDLmh2h8F6ovyvLjclN5JmAKHi8Ux/3GMFnsHjtVjw2f7Wyfia0dZa9c4yFYZWeedBnUNkR7uX3zcVNmpUCuVbAE1AyxrBG1iIj7smvHl+I/b73r0AywCj4leSBz4JVhik5hNPaOso1WfQqijyEwXsAxJH+SHebB2NsHWOMTxv9PYCDVBdijN3MGJvMGJs8ZMiQHKrmIEcTPf3WWrR3liMbZ9JmJUcTBTqLMrwXIW17jWvo4dxEyQpat9VsUXcdqgR9IuZZS93rEfDEgjWRx+iEhCz7TW5F3DE8uZifBylwNgC/kywQ4dhfPIXz/jg7cT04W9vCI9a4d6BSM92V0UTuZ5VCOO6Kh3Hx7S/nUvads5bimkcWKPfJgQN3vbgMU370OF5/b5Nf14h78tBrKwCYv/tZFl4CgkJ90Zqt+PANzxqvDV4N8hAGLwKYQERjiKgJwFkAHhQPIKLhwtfTAVQ1HEJ09r26bCM+e8ss/PiRN3J1IIfNRMFrZR0/JW3nSTuGH2sanPFoM+4HmmoG7oEPzV2JC//8kluH4DHagADNfpOifTOc+uh2d6TOTUDB3FPR9fLrYf5MtgjpnU3PymMRJRVRZiLVYy+VGR6au7IidRGRR+g8pbi4+ExUO+DnJzW95jHpzL9mHdjbXDILA8ZYJ4BLAMyA08nfwxibR0TfJ6LT3cMuJaJ5RDQHwKUAPpe13CSIOdg3bHdUwsVrt0U2zrTPSGXbL5UFn0FqQ1G0GUPenrT+GzSqcvLRJlN80je41vYSPnvLLLy9dis2t3V4ZrbFa/VrT/jhu2Ft6Ct3vYLv/X1+6PjX39sUGDHKxIWW8kyTnpkoIAyCmkGa6DUZMamaaTBApTQD+aripLNq8uaqLcqRtM48CESn3I7TZnXliJ16knuexXxaDTKHlgIAY+xhAA9L264SPl8O4PI8ykrCEwtW49pHF+KsKb4VS1Qto55j8nQUbkdQDndUos8gbRuKe+e0dvUIpvzwMRw7cSi+/sE98XiMSSaKu2Ytjey8Af1L/+yi9/H0W2txzi2zsHxDK3Yd2Ct0jDyi0pmJwIAHXl0RLhvAh37zn8j6xdVzRycXBioHsvPXCys2Ksm8HsaaQZlhS1sHGgoF9Goq5lALXhfJTBRIR1GdqJhNrR048VdP40P7Dsf1nzowUDSvDf8rjtw7I9JF88NmvrMO//23ubjj/EO9dRsAYOHqLZgxbxUuOXaCe33XVCh06teliLwTqSNZkI8wqFe+cKtjs517vz8iFF+UyElnGTttJm3LGhHiCZkcZyCv2bIDd724DJ+bOlpfrsGNmH7fXO8zY5qwSE29+cht+QYnvl6V18fUTJTHiFxrJioFhYF4e/l7xM00Yj0YY745ImU9TCPJymVgv+//C4P6NOHrH9wTQ/s14/hJw8wL1dYlSC2iiXiqk1nvrPe2eUEZkqZoOnLn51/1gJMKY+XGNuw2qLe3/2M3PofNbZ0478ixaGksBvxZnLkR2maovO5uJupqiHHi4nsiLziRPh1F2JwjmonSPnxv9GN4ehKLQWNEnFsa34nqFG2UTgrTBrf6mZrG8jBpRPkM+EfVhEPRTJEmtNQ9MbxNARdG67a144r754Yc2MblyfukwXVZdCBHXHPRmvwWvlENpuTOVYxE43QkeL9kn4s8J8gXNuoJgXGojqwjWdDzhIHXKZeDmsGEKx/BaiEsLWkfJS/GHjATlYXQ0tQaR/To0HT0rKIxIgfzqk1t+PmMN407bUczUGzXHJ/G6emFgRpGUOmK+OZf5uA/C983OpYLg4LSgaw3EYr3TaVN6AgIm8gjfbKsTxzpP5P2iQIuqi88/pdPh+7vth2dqUJgVcXINn/VvBQTMxFHfn9CgQhcM4ipq748xZlWGNQO0ZErv5PLN/izCJOOJv3RIT/fJzDpLNFVxQLidpt1jCqiGvRnb5mF659chDc1sdSykFiybhvWuovPmFShJHVgvTW2brED0WsA5pTdVbo+/X/qBU/aOkoYPf0h73t7p2Om8MIKhcL4LeCdpE7jDPqRousXNDOam4nSEiXsVdFE8Dre6K7xTSGD6/IN27H31TPwx+ff1ZSTrHXI2rbKwZtkzWZVQEJgv/sEAkueGl9dfaw1E9UQ/vKWmepBiP6EdNfXhZb610134dhJZxJJzC9RLyS35bd1quOhZfPaF//0Eu6ctRRAUEDp6i0vDq4TBsERdvgeq75HbecOYZmFa7ZgxcZWrJOiq7jPoKjwGcgrrIm/Vfx9qg5eS0CLiD6UkyW0NNJMJH0PmIliekNR2L+7zhlszZi3SnmsySsr1lMODRX0Fe9T1OLz8kg9ND/F2+584q+6LrQ0tm0aKgYrNrbWZHXEHicMxNBA+XaLEt+kXW1v78TBP3wMzyxcK9iL1SYMedH2pCRNR5GkGJP3bvsOtTCI7EQMOjT5pdfVpVNhbpE7dF1jVN1zMUTx+0Io6mV3z8Hh1zyBktSJdHiTzhTXZM66B+9vbQ/9hoDPQDNAUKEUHHGmpQwdSLRmIJmJSuYO5I6IzlhVzvb2zsj0Euoun5/v/C0ENANzdUlnFvLMULrkRNLxOnSaAWMMi925Ec+/vQ6HX/ME7nu5+mtL9yhhUCDf5ssUmoE4UjDptN9e45hEfvLPBV7HxEcP8jvIr9xZYli2fjteXLIeSYirjVzdV5dtxP2vmL1QJr9Vt5h4ZH4ng7JlNV5n4/3Ha/4kpjJjeFgxqUlXnmq7qOnc8uw7inoF6yGPMGWfwcdufF4oT9AMAtcRzjHso5hwVtzIP4lJROTNVVtCGp6IfFnxt8eZSUoaYagr54r75uKzt8zyUkx456qu4RZ+zSMLcPl9c4UZwoaagVx+WT2gkDW/YEbZ8HFJYAy47+X3cOwvnsJ/Fr7vtdk81mVPSrcOLZUpkP8YHYkc3C++HEkfKx9p/v4/i/GV4yeE9osjSp7Uask1pxpfX5ePhyNvvvW5JQCAjxwwMvbaJqan7e0phIGBZhBKdaDp0L7xlznCtRjmrQiH9OkdyOHtkauyKa4lp7oIjtzl8vzPATNRAs1AZVqM6tiA5JFZm1o78O2/OqnVdxdCKmVUDmS+7e7Zy7BuWzu+e/reynMT2ezB8LY7X2VrW/B9U90vsVO+c9ZSHDtxqLPd0GegciCrHossjIPpKMQBpLYotzzFpDMwzFvh5FhasGqzN5js3VT9rrlHaQZOBkjfzig/eHFEYaJyi42k1RUGPI2A3AF5mkFaVV7qhB549T1cfIeQ/yWDvTiqSvzFf+qttXhZWuXr9pnv4ksROWgCPgPh86d+9wJOvs5ZrFzu4Ezs3mWWbDananOc+UJ+Trwz6pTMgeI+sR78vt0+cyleenc9TrnumUCooqkwIPi/K+7dMdXwRk9/CH+ZvQw/m7EA/3Tt99yer0TWDIR6LN/Q6g08VERF88j85vFFXvK5cEBE+PhwaCkLbU9SvspawLcD/u/W+Unigk6UwUQMaGxwdrSXyplD0LPQw4QBCTdbZSbyP5s8Cy96AYTtwkhT5Y/gQshk9Hb7zHdjsy9+5a5X8ZBgOsny6kS9eC2NjkP3gVdX4KP/+1xg35X3v45npNBBHWIRz729zmv08v0wuT86QZ0ktDTKLALozUK8cwksbqPQIvh8hJueehsfu/F5zF+5OZB7PzaaSKFVxXVsJvdupTup76an3vbCZeOQrxo16SxytrILzyEkc/2TizyB9/DcVYHn7E+69BH71rGD+/iTwkQHcsQ9kR3BZUW7BZz2CPi/ZVt7CVN++BhmS6beND4DxoAmd55PR2dYG6wmPUoYFEh4IJoH7+1O0L0SBc0OnVJjYcwvN67BvrexFVfe/zou+NNLge26aCJv9qvmuovWbFVuF7lQKktEl5vfCKb8GEBurEbCgCXMgqC4ZFw5YSHl/PXDR0WtJ1ycKuKkXRNmGgUTyorTZkzuXarZ0NKxjpnI7FhdnV9YrBYInJueejugJTLF+y/e4qaGgjLCycRMJI7GVfflp/98EwDAf8rc5ZuwZssO/PLRtxINIFVmojJj3qRPcYBSg2CiniYMKGAmkkeYQWeQ+XUZ881EgBttEYomcv7GNVg++pNzrPtRStLxith2keN/+RTWbGlT73RZEmEi0NmpV22KviYQrKuu85PnGRiFF0KdBVZvJgrviIsykTsRWeiKe+X36JxbZilDV5M0djGShR8aV+ek4cSmaxGo/CfmPiB1nd83SBvN56sA/jMMagbB+qtmCMva1LWPvqU14ZZZ9CBQTmEt34M4047ynQU0wsBqBhVFfBiqm22at8g7xv3bUSoHGmJHuRycbSqUbvqQdTZT+fR2bxEU/XXjnKVR6DqYL98Zn68+MFFMsX/NlrZEoYf+daFUDZKZicyEsnxtvj3JBDK/TP+axjO64a83HOtANni3eB0SDXak76IDmTN6+kO4Y+bSUB106ThMyheTH8b5DEQBFZiBLJ143eMLvUVlQtFEEUIOENar4BGJ0qAkVhiochOVmbeGd3upHBBM1aZHCQNn9SznLr+9ZhsenBPOcMlJ8jBapbS6qjhs/iLEOQF1ozWeflluhLxxR3UuWSawdGhGdlva1NFFInHRRFN++Dj+7z/vJK6Tbn2IJA7kuI5VFha8H+fbVZPO4ugIRBZFnyNOrOOHimamK+6fi2Xrt+Py+17zzzF4znzwkGTkKdeVm01kbnhyUdikpLnPJnVdt83XDNTRRGIdNRqgonzvWlLvHDd/ggs6XYbetK2suSGsGdTCZ9CjQkuJ/AfW2lHCz2YEX+rg/Y9+GGfc8CzmuCOMbVIMvjN61KiiKSNCrn9ykXJ7u8FITzfb1gS9o9Dg3IhvHN38hSh0v3WtxvSgali/fmJhZBmyEJRjzVW5ieJIZybyEyqKHdsdM5fijplLA+eYCP0OT7MxqLBUF5G/K1KFNxYpVAfd4MfkngXnKPjC0UPU5OE79cVDVKY1lQbB6xRlJpLbrvwT1m9tx8qNbdhzl37K81U+g45S2dsuOpBrYSbqUcJgc1snrnbT1apIosbOEdYv5TNPOR1ytAUz1wz46GPlxjbc9/JyfPTA4DwBnYMu6uURo0ZWbWrDoL5NkZlKTTB5WeM0gyxlq1Ru3fKIqqLFVMgqOjrVwsC7pvDVNHpR7MxNV0UT7dh5+AzSaAYiw/o3Y/XmHVih8Bk1Fguh6+rqnERwAcEAinVbd6CtsxyI9xc7crEOcRqgeG058EN3HIchqFycfN0zaO0oaecPqbTZY3/xlPe5o1QWsuLGVjt3uqWZiDGmneEblX88a+c12p2401kqh0NLYRZayvd3lhm+ds+c0MpOyze0BlYlM/EZrHDDCbe0deDQHz/u5W/PQtI8OHm+20lV6DQdn26egUoz0DlJZZI4CPluMdV6bDSRwe9sF8yKphFZ4mWPnKBfm7yhWAh1Yrp0HCaP5Nv3BtfJ4Bz0P49h6jVPBE2qTOhAhWOjBGho0llElBSgbrtiHWRz8XcfnIfJ//NYxBWDiOHp1oGcE398/l18/Kbn4w+UEO9/0odRIHgzjztKTDuHIT6aKDwClWPixVW7fJ+B/poX3f4yZsxb5b1sj72xOrIOJpiZiYIdwcpNrZny53DK5fhsmVJFEiPfcy8bLQv+BcwnEgZDS/3tbR0lxXoa/nF5Jif0NQOjS7rH+gdzZ6eKN1ZuDi0tKmpYYpFJBxOq3FzBSY3+zoBmEBVaKn13NIMIM5G8L+Yn3PrcEqOoKc4/563CX15y0lHUQBZ0T2EgLoidBPHlWrp+O+6YuRSjpz9kFI3T0lhEc4MzQauzXA6OgoTrxjUCuUEzsJDNX1wNrL2zjE/89nlcG7P83sNzV3qTxvJwTpl0PGIxa7a04bAfP4Gf/0vtfExCmgV3kiKPwr1lTVU+A8OeVbymeP8mfuefoQl93rUT/FbVsWfe+FwgtPjJN53lTZO8A+KRDRFrXwDAk9LyqYuFHEOBSWQphYEYuCA78b3wU1FQK7QpXclvr90aKSSVmkGFxiR5DJqS0i2FQVrE+//FP72EK+531NS4OH3AiQhocO19TlRC8GF6K6zFaQaSWssYlIuAc9pL5Vj7N+DMIOZCJI/3LGlj5o1YlWAuKbKtNvb4XDSDYEcT8BmYOpA79WaisPnSL8/0Xqu0w9nvbsDds5Z53+90P8ebqZjyc9JJiIvWbFVOjEza2amqW5ZMUKJpLcpHx5+t7NC95pEFkT122GegP1jVZpMI4Fr4DLqlAzntnFndwzIxSbQ0FgOTR0JXUryc4vq4nHBKZxYpDGRHpwmMRavDZtcwK0c+fqsmFbYJuw/qjXfXbXfuW4Lz0ix7Kc8z8NR397upKUJEdoi+8/42jBncR3ms2LGZPiqd76Jvi9PMxfcoTqAy5uxfsbE14ORsaogeP6quyTXbu2f7QilpZ6cSXqK50/GtOMfMWb4xMoUH1xbUk8ASmIk0jJ7+EKbt6ftWeDtP0uRsNFFOJFmXVER3+00u19xQQINrT+10QkCUF//VY35II29wInLHsu/3/oWDd99ZW26aSVtllm60LGJkJhI+86R6W3ekT83Lhe2Ff46f8CZy9M/+nbisdl18vCLE8e016rjz8DX9julf81fhp/98E//7XwdGllNm5h2nTij1aXaauThrXLdAC4dfaeY7wbQRDTGagarttbaX8L//XoRH5/udd9KEjar3dYGw+h4T7tMNT74dWQ7XvnWJ43SETLiK9sv595v+mgw7OstoaSwmGpLY3EQ1JuoB3DFzKX708BsA1B1hc0PRs6d2lMqKafnhc1TSX/XCzYpY+6C9lHykLY6i0pI0tJQjLzKehLThsKYJ2UR0SeHEETvH1CEvJvWb76Yt5n9D5Xh/zbU4XRhlLzfZ4Coh+WGcuUeVux8wEQbhbW2dpUBqCSDa9BlVnyhUR6juia8ZqFJK64mLJtLR7s3tqG8zUbcUBikVA+2ogMiZ8Xnz04sBhEPIAKClseBFWji5iYLXVb3MZRZOFZF0xHTrc++GtsU1dKZQXJJSCzW2KSKSJW90ocmqqJY0eL6HQMRV2Kwm2sLjaNcIMF5nsQMuxjSS1a7gkA9riBHIqs5RFYCRNEVKXLMoa26UKrTUa2NKzUBfkLxPXN85iqwRXNWiewqDlF4D3cOSVV/VQi/NDUWvocjRRM61wxd/ccl67HXVP/HsIn/EaBqzzlEtEWgy6suuGcQfk8ZWH0VcRySy68BeOGP/EanLeuyNNcrtKs0gDXFaIe94HntjNR5+3czprkvLfd3jC7F03fbA04h7R474yZNYtGZLwKxpcp6Kto5yqA1t0yyWpCNuVK0b4KjMqH963hlAqX5JZDSRVIctbZ2R85Y4OwzmAiWpR6XolsIgLQ+8+p7RcapRTXOjH03Uochaqnq4T7hheGI0kGgHTUvcqC+JUzJ0rpclNZ2ZKAtRMe4yDUXCEeMH51sBiCGm2a6j0xS9cty/ryzdGLA/R6EzEy1euw3n/GGWd9FBfZrcEX/0/Vy2vjW0/GScMFDtbu0ohUqSVzKLIzZ9B9QDHJW57/mI9Nm6AczyDdsD6SI4qzbHRxp6E0NjjxTqYTWDfEhrJrpdyvXCkcPgVGai5oaCZ9OWE14xqO2NG7c7ztR+Lb4fX5cELAlxg7ckpgeZf8xdifbOspEDOe/RTRKfASF9IEEUYlqELMyYt9q9jr8t8F6luHzUgj1tHSWvoxvUt8kotFN1++IGGspoIkV7yVMzOHvKKG1QhM7sur29M5Qm3ilIXcYRP3kyEA3FifOhAGKWgNhDPayZqE4JNNj2EtZsDs8qbG4sCtFE4dBS1cPd1Oq8jFmyiqooGPgM0r5sl975Cq5/YqHRix23mlhSkjqQs6zLo0M1sSnb9Xz2+96/vM9pnk9UZFlBCG0sFgpGcyNUwjTu3VL6DDpKoWuZZL0ViW4ipB3g6NJRnH79s1ioWPgp6V03MZuZpIyRsfMMciLvTkBsmNN+/iRWq4RBQwGNXjRRMAKEacwyPGPnjx9ZgPOOHJttVTGBuLBB3fJ+pqzc1JYo9UFemIzCRCqgGHhO/9//Z3Eu19P1D2nWyo4SvkR+WQ0FMkqup7p9qaKJOsqh7Umz1UZ1pE6VGOavDEdm6UxnuhUAk8pgkwHKjk5HM0pyaasZ5ABjLFXsfRTig1EJAoA7kHk0UdiBrOo8RTV1w/b23OyEcX1mVgdygyI7pQqVMOjbnH780Rgz4UmEiFIHEkTBGMOvn1iIZetb4w/OgOnz4WGjQPQayQUirzMqFgjvb92BO2epzaJRxAlY1W6VWTVPnwFROHMwJ6lQTdoukmkG5tetgSzofsJg3bZ23PrcklyvafI+9W7yhUGHIvuh6iUThUE5hRA7/8gxyu3x0UTZXrbGIhmtkcBHRCIDezemLrcpoZmoEpoBY8lj5COvpxkvmvZhor9JN1EOcAYIfLBhqmHl1R+1KRzI8hogcUR10lGacJSAlBHXUTbF5F5yLShrnqlK0+2EQZyJJB3xD6Z3U9EzEzmagX9OmQEzFfmDRGFw1s0v4Lt/T5Zaelj/FuV2bp89YdIw7blZtBDdRCkZlXDr15JeGCSJJqrUjIR129pxz4thR2JadI/B1I/U0ljEQbvvBCCZZmBWt+TviKranaWwmWhLUjNRxL6oX9ORQDMgJP/NJv3NK+7aJz3CTEREJxHRm0S0iIimK/Y3E9Hd7v6ZRDQ6j3JVVMJpaPI+9WoSzURmNvkyc8L8ACf8T165Ko6mhoKyg+QRH1H3Isu7ts1wwtB/hPkTnP4t6c1ESeYZAJWJJgLMf78JJouvRNFQJNx+3iEAgN9HLCG6+P1t2ORGrzUYClVVzeLeG3X6h/C2PDWDqOdcSnB/00TZxS04BAC3PrsEtz23JJE06JLzDIioCOAGACcDmATgbCKaJB12LoANjLHxAK4F8JOs5WrrU4ExockorbeYqE4x6UxHFrMJEXlps0W2tDmNPmrUkmXksbk1fX6hPhl8BkmfbDXmK39i8sj4gyJIOslQprFQMB7pc/NpMSYNNSdNGmXV7+kosVC7THrpqBF7lMw36aw5SddYAMz6htaOEq5+cF6iNtdV5xlMAbCIMbaYMdYO4C4AZ0jHnAHgNvfzXwEcRxUatlEFDF8mUTG9mxsCKaxNyeJQJahVfj5ylScMiWR51TalFAbf+dAktDSmf0B9E2oVFVIMAgwf0CvT+WmihkQaihQb+8/h0Uam1jaVmS+uKJWlqkNhJkqK3KdPGe0nb4waACa5v2kCK5L4+RKZifINxDMij65zVwCiEXW5u015DGOsE8AmAINyKDtEJXwGJrMMezcVvY45SXx976b0wqBAFGlHj5K3f34hnNPIlLQO1CIBLQpNxpTRg9TpnnVUQkuUSfO+iU7HB6SF5fccpl5MXXutYiE29p/jCQNDzSBqMXkdKs1A5TNISmgNaqFrjdQMpM46yuGbykyUoK0nm2fQNTWD3CCiC4hoNhHNXrvWbAp+6Bo51wlwFrqJo3dTEUSEhgJpE4ap6NMc7hzjcsZzChS98lRUYjc550wS0o5mi8UCmhvTC4NJw/ubH0zV0QzSJFIV6yXH219x6l6JrtWYwEnGR7Gm0URp/Bnyu9FYJHSUWaaJlUSKjPDChqifIwu0lpj3L6mpKImDmgG4aNo4DO3XHHtsl/QZAHgPwCjh+0h3m/IYImoAMABAKEEIY+xmxthkxtjkIUP0C29HoRqpXfPRfTByp2zqfBy9Gp0RfkORlDlMtOcpNIOxmgVPgKBZqEAUaS8uMZZ4pFlJikRoTjBXQOQHZ+yNsUMSaAasOj4D01G5KaYmH46pMxgQNAPDc9LMIJc7/YZCAZ2lciZHeUOBQqNq8VuUBiyXGycMkvpJkmkGziQ1Ex9PV/UZvAhgAhGNIaImAGcBeFA65kEA57ifzwTwBKvQr1W9F0SVX1O0d5PzkjUWConWGOjTFH45+0eEXwbeI4oOtyyVgQcumZq6A86bhgLFNkYd+40amMgks6Mzu2nChKSdNxBtvjK04HgkSdGxxl1TwFgzkNrM9Z86IPYcWRg0FgkdJZZoBK1Czhsmdh9Rj0AuNy4N+lfuehWAeVRi0p9l+rZ0STOR6wO4BMAMAG8AuIcxNo+Ivk9Ep7uH/R+AQUS0CMDXAITCT/NC92KkiRRIAjftJNUMVD4DU0dpnGZQLjO0NBbRv1f6iKU8KRQo9ZoEBSKtMNhJEZG1cXt7xUJLRVL5qCJOSawZpNBMdKbFzxy6O8YJ2pc86t2pd1PstdduDc7QbywW0FEqJxpB7yLNn+koscDCQICjGXANOkq4yj6MOK2Ip7RIGsZsipgjKoquaiYCY+xhxtgejLFxjLEfutuuYow96H5uY4x9nDE2njE2hTGWT2IXBbrGmXPOtBB8hN7gvvymqHwGURFGsq1UHhmKM1K5AKyGucSEhgKlbmTFQrSzXGZbe3jWq45eGfwYecsb2ezDJ5Tpj09+P3UCpF9LA84/cqz3XY6UMZmUJafbbigSOkvJZtebmFHKDJhx2VFY+MOTEzmQGw1VrzRCFgB+9JF9Ivc7/o/4e9ElNYN6QycMom7uLz+xX+ZyeYRGY4Fw3ytm6yIAzmQ1GVPNgCjYcH7x8f3w4pXHe9/zeqFMHdpxFAoU6Oz2GzXQ+NxihCDRaQCmmsG3T9oz8N003PemT6vXL86C+P7uO3IApo6LDrpLIiA5utGxrH2FookiivrsYbsrtzcUCugolxPNpzDqrxlDY9FJGx/tQJZ8GIb3K23SyJ16N0bWZ+2WHUaagc1NlAOq58BY9OSQPLKF8pFE0pFaH4WZqF+UZiB8LlCwg+zTXAzY5PPyk/BZ0knYdWAvzLriuICgbShQYGQ2pK/5daPMMbo9aZ/qviMHGB131B5DUpmios4Q38V+LQ2xDuqoaDL9OTphEKycPJrXPYOnvjkN5x0xVrmvqaGAjhJDW0fZWPib/KaAAzlqnoGkpZuG1abVDIoFiqz/yk1mCQ6tZpADurYpd4yiUzWPuQncXJNUsPSWNIOmhkLkSLwccJxR4KXlv+Peiw4HIJiJMv68nVMIAwAY2r8FI3fqHaifeH+inJ+XHjs+8D3JfR09qDf+ddlRRr97+skTsa/USZlGvjgjaeNqGV+TM6BXY2ynlCSaiKO7l06mVx/5PuhKamksau91Q4HQWSpjU2sHBhu+Ryb3tGzqQJbNRMaaQbqusaFIkc+EL2gVhxUGOaAaqRGFHci3fWGK91lsHEMMYoBVeBlLEzon5PQMvRqLkR2f7DMQOwt+Hn/huWae9b0a3DfdPQGCDVW2+0dpUV/74J4YPcgXJEkcqyfvMxx7DOtnJAymjhuMA3fbCX+58DBvm2mu/WIh/2lt4rPv26zWDP79jWnYdaATKm1qAxfRCRiioDCS32WdFhQpDIqOZrCptQMDejXibxdPxYShfWPqZ6AZCO90ZG4iaRBoOqhIohmI2XSLhULkueu2tRvnLas23U4Y6FCFvHHEBpA2DJM3yqSTa2TNoJeQ40hG9AcAjnrcoPgdDSnrouPDB4zAmQcly8HDb6nYLGQHctwoTWzkUf2D3BeQ9ze+QfNzxXBe0yUZi0SpzERRwlnsrLa3lwJC8Osn7IG7LzgUowf3wYl77wIgrWagvpkFosC97CjLwgA4c/Io7DEs2Jm3NBa096GxSOgsl7G5tQP9ezVi/1EDccjYnZXHct7bGG9KCQiDiONkv4dpJ59EEw0McAoUqfF++lC1b0Wmq84z6BKoJsNwxAef1s7uawbJzpc1g5ZG/aQUeXNoBrK7n7+cqnDacUkmbrkM69eC//nwBxKd4wkDoc6FQjCPTtyoVvy5SRpnQSWJYo4V62lsJipQKhNcVJizODDZuL0j8Lu/fNwEHDLWcSjz/ibpUqBAtM9A/D1yxk+Co63c+OmDAtubigXtrW4sFtDWUcKWHZ0Y4IY4x9U5SjO77qz9AciTzvTXkh3IlUhX01AM9iU6Ad2vpQHnHjHG6JpWM6gQx0wcGmqA4gMT380Vm+LzEDnnBB+4uP5xEmTNwHFAmUXHyD4Dvy7OD2IKn8Hxe+nXONAxqG9zqk7HIThqEs0ejQ3mDTPaTCTfl/DWqz4kJ9J1UMmjsw4ehS8epXaIhkpO0blEaWzi89xn5AD9wIAHLKRwWuiqTFI0kZwSm/9W+VnI54m8/t4mvLDYWcuDZ+hN8i5dfMy4wPcm6d0G4ha3SWcmSvJYxd/TEOFA5lYHs3kGVjPInX9ddhSG9msJPQDxAaYZLcgvlWcmSqgZyDNymeLaHHkrUbAz4p94B6HqdNKYFQb1bUrsGFeZaGSH66ThftTOh/YdHnm9qKiakJnIG+07f8cP7YsvHDEG+yuiWTzNQNjW3FjA5aeY5QiqpM/gayfsob3vvENOM89Ap73GtQO+W1Un3aniini+ZpBCy3NRCZIkZiLTth7XF//8436UXFNgYKmfD+PX3WSeQewhudPthYHu4ascrwCwu+C0jEJOEMY7K9W8gejrSI+A6Ru4/Fuc6BzFNYt6n0GaUEQ+8/Q7H5pk7GBXmonIv9dHThjshXASAdd/KjpmP4kDmTR/7zz/ULxw+XGBYwuKeiZpiHlbHQKaU0QeG/4upPFx6YIcChSt6fA9KsEsb/nf/zoQP5BMi1wYJHkH5evywYw4co4aKMiagWkuqbiR+RHjB3ufxbW5GwoFbftNMhCzPoMKENeYgGBH8+DFR2S6btIwTNUEIK3qr/AZiL+DvNEibzD8NLGDMX8hjxg/GF+YOsb7receMQYTdwknvuunmCQnd8QAsGF7u1ff/i2N3nW1773kbzBF9gPwv72aithlQDDVgd/5+ddP0g7Fe/v5qaONzvn0obtp98lCTycEuS08zYRAWRjwTjouVJbfK+Ux0rZT9hmOzxy6u7cKm1hOkjrLwokLEtNnJPsMTF//uMy8ojxrlH0GmpuYJPLLTjqrAFFhdByxYzd9UXV2z6TCQK5flJmoQMAIoTOTZ4zyT1GRTUnMCoePH4SrTlPb2gHg/CPH4IGLp2LOVR/UHiM25lE79/Z+W6nMEgmmKDOVynwGmJkEPFEQ0AzMW6J4XpypC3Du2Q/O0Dvj5Z/Jf7e8XCifTMXv4a/PPsCkugDCwmCfXX0NLSoCi+9RCSjdvRbTjvuaQVCrOX2/EfoyNffD1IEsY2rujAskEX+v2Iaj5hnEDn7E8q1mkD+6EaX40oumHVPhrXup/vtUfeepQm5EjOk7SSLC/RdPFTaoG0JDRDRRMnNL+FjxklecspeTTVRxL/hh4p69RwzwOrsyY7ETe8Rzk9TbM/0YHKtqdElUdPGnmziTp+05NPI4+V7y92zEwGAKdj5y5fcwqkOVifIZRP0ET8gamIk4otY4QHIgj9q5N5ZccypOmKQPapDbh9+hmjmQZUyFQVxiS7FMcQAZNQM5yUDMCoMKwDuRH380mEBKfH9EJ65pp6N7qSaN6I9vnrincp8KtWagycEDYFj/Fm8SmDyjl1edCwNVNFESc0vcrYjq1Pi7LB/CG1GZsUSRMPyWHLDbQEU9NPXy/ujLae8MJ/NLayYy6ZTijtHdE3EmN+A7RtPkJgpPJuN1i579y+uuaiO6d0HsAGUHMn8/Ixe7l74rNQP37xemjtFeRz4/jrg5OoEQadFMRHoHcoOi7jqsA7kC8Id/9pTdcI4mmZa4FKPpKCPqqCSRN0l8Bl5jLPDv6vpyM9Fp7mhRbGspM0h7mGRcBMTwTvXIrlRm2vukyoXEG989XzwM134ymFhQLkMuO+qR8o5R7Mx0DVFpQhQFrcG9jXMiysL6vQ3OBKxxQ4PzQzq9VcuSN2FunvnArsGBi1O2gUBT/FBx02ekiVWfO3w0PnLArmh22xnvPI2Epxyowb8qnpHJwNu0fceZicS11sX22thQiNAMzBufdSBXALHD2XMX334Z1AyEMFPh+AuPDsY4i0Qt1pFkxKs6Vp87hp/jNybxfN75FQqEV686AT/92L7G11aWZ3xkGNVkLl43ACgx/X164JKp7rn+fj/VRgE794mOaDIRAhyeQlw8VDdSbVEIg+B5iPSfAOr7/5OP+VqrPOre2U3md/i4wYHtpktYyr4GALjkmAn45ol74m9fmoqLj/FzQBGi75kqtPQGNwpMFMhH7xFcpfC7p++Naz+5v/edawv8OvL9VgUpcFSagWeSNPERGQqDODOR+JzEQUJTseB1+p+fOhqHjPFnW0c9q99/dnJA67WaQQUQH9rZU/zVOQPx+ZoXJCoFQ3unfnJZkg5XzqXvLI0XN3p0/hKCYW0iA3s3eY0uTzOR6YBFVwwXEozpNQPZJOLUxT9WXh0ubIpytyvOFfnTuVMwfmi/0DV0ozLV+s3itTtKZc8urkNlYjlwt538/dI9Ofvg3fD3S44IdbDcTBQ12vzmiXsqAx16NRVx8THjhfeDvL/R2WGDZqLmhgJO5U7zwDumvQQA//3mZYm/HwAe+cqR/rWk+nDtQhQgOpOkDpP2GWcmEusl3uMmN602P0asp8rfwRnUtykQrmp9BhVANMOoJmgBUR2X/rpRwiCJZkBE+NK0cRjW3xntMugdq3LuISIKJMkyIU8HchQ6zYCXXyqzWBOHrqaq1eEC53FhEPNbxcYn/lbdTzxqQnhdbvFRR70THFVHpPL7eNcvEPZRpNTmDuQ4p6RKWOhuS4GisznJ5wUz5orHRd93OcPv7oP6YMk1pyrPl28X1+TEqD3ecRrlooJZG4gTBuIlxMFbU4OfqK5YoECIqhcWq7hesRD0/x03cWhsHfOm+wsDzYMng5c3cpp7RNoJ0/S3J7nJxr510kRcd9YBAJzONi4ctiD8FScd7Tcq3GnIZNYMYnwG/31qcOau3ED5rSmVmfHC7DKq1eFUSH5kxX71HpXAe+qb07C/4v6Kl2h3/Q/yKF4kVhgYGue80NKY5/nncw8JbZPfa/K2R4+u+XktjQVcNG0c7hEyvQY7cDNhYPIuypcauVNv/Pij++C3n/HzI/GRtumrbdI848w04jMTBydNDb6ZiCgoVPhvUb1fosn3tP1G4JJjJ8RXMmfMlnTqwugevNjodO9Q1EsdlZAubrAujoI4cn6TqPoENANXGFx2/B7o16I2UYhXM9Fajp04FE8sWBN5jC7Xj2zT1WkGSaOJROQUHvJVWtvLge0mylDcPAMn7FIxyhZK73A1gx99dB9MveYJZTlxPiJTxc1zIMe8bBOGhe3vUe97pJlI0Li+fdJE7TXjRt68s5THArsO7BXKWNrWER50nT0lOGmPPy5T53DSdaZFBvRqxKbWjkBZ4uBEDC0tEgVmQEfVz4kMdM7La1GqpHR7YaAzRQTs6Abaw5kHjcRfX1oeOubSY8fjomnBhVjS5DryE3Dp7cDeCK7Aba6+EDFdR8HEXsr9GFHqvs7JJ3dO8iVGu4uYn7rP8Ni66Iof2NtZ9OXLx07AtY+9FarndjcFtd95RRYTQqfGq64jbuO5j6KKU/1m8R01fXc8M1HEPdT5PsKhuMLfiOKjTIRiveNesSbJTMR56NIjsEFa/OXlpRuiLwZhpr2xZpBeGMy8wklnIl5CzjzcIPhERM0gMmy34D/LWvgLgB5gJtI9ANJ+Ec41mJnct6UhlI8oTTI4MYOndp4B8f38u587vd1QGCQJ51Md6b2nmst4sdQaO+6w/i1Y8IOTcM7ho1OP0Jobilj0o1Pw0QN3Ve7f3l4KVDLJugaA3xhFu7Ru1MwF0R7D+mJo/xbvWB0qYaDIQh4Ln8wlZ70V0fUpuvsR5zOIMhGamF3FcsS/nIG9mzBmcDCE9vwjx2K4lEJEVy/Te2cyIBo7WJ3qvaWx6C7m419DXqa20Yv28zO1Ot+DbUOkQH5GXysMKoT2xQxoBupDxO2yo5ZflidxE0mzZJ6fcyVoPmlUOMCLnpnIF1JRzktViKbIV46bgN99drJQF9/mqb2mpunJE2tU1+CNKW6ElnYdsW2SZmCCeI/4QjdPfXOat61QUHc2ym0R5cZpBqZ1/uGH98F3PjQJU4TQRRltlyJrBu7fQsHvsFRzPUyzs8f1tb6NP/7HTh0/GM9ffhyuPm0Sbv7MQcpj/Gii7GaixiLh39+YhmMSOHDlgAbuCysUCL/51AHYe0T/2PqJPoMayYLubybSEfAZ6MxEwjFydsiLjh6HYf1b8NEDw+GnaWzhorARNYtRO/fG4rXblNcvl5kvDDJoBl88emygkypGaQYx1/fqLh2YQTNPTKurGegimuI4/0hnLYN+LY3YuU8T1m9rj9AM3L+B90l/bZXZUuycTDu0Ab0bjRdKkQmbifyROt83pF8zhvRrxoJVW7zjokashZgBhwiP4U+iQX8+YnaxaqZ9FFGDkJaGomfKNEUOBS8K93Novxacc9hofOve12Jmd+vnXVSLbq8Z6CBy8sV/+6SJRpqB/OI2NxQdU4dyNmbyno+PJhiCwuS2z08JH+tev7PM0FyM1wyYItZZrq/YmfmageJ3xNhnZa3IG3WmuCdp/XzbPDNRgrKEz6JJsCyOYhX1UU1wi9JoVEpjc2NlmqHeTKT+TsJ7QBqHuY4kZiJuR89r1TH+M/NwIE9LEdIpv/O8iRUlDZv/bVO0VTHbaY38x91bGDzzrWO0+wjApcdNwEXTxmkbr/hSH6mIMddhGvoo1wdwGrD4co3aOTwBi+8OaAYGMe5AONLpgN0GhhY0j1qJKs4+26gxE0U1VNWiMyI/OGNvdV00gmm7u2yi77eIR1e9Tx/ipFbo01xUxp6rzosaAcqawcOXHhmKjsoLnY0/vGKe89dJb+JuQ/hdiRqwilc0dSDvFDNBz5RywtBSneYy84rj8MtP7KfcF4VsCShIQoCXx9uAqq2KecYqsTSnCd3aTKTqSDnBGcjqY/gzbmks4FB37dm4cwBnMtOVp+yFme+sw2Nv+CGa9150eOzM3Kh86BxvwXvGvGyWUYvy6GLAv33SRFx4tGMSEUs0iVDSjf7CsyyjTTXPX36sl8BMx0G76+3i/Np9mxu8tXP5zHEmHhCDbkDw9Q/ugcvcFcdUOe5VjypqZCx3RJNG9NccmR1TzYAjaj9E4RG0qZkorjM7bNwgXHHKRHzyYP26DkkoJ/QZ6Fx6w/pHO6p1yM9UdpCbmCsLBX+2clx7qBTdWhhEIT4XvTBwdiRNBkZEOP+osXhr9ZbA9hEDWzB8QC/lOcP6N+PCo8fhzIN2NU5m1llmOHLMzrj9vEMiHYki4m85ce9hAXsxh0c2mfohRGStIi68U3c/TBBHvi9ccRxKJaZMB5FFMyAiLx6+JNwPv/2rzG76cpIuH5oFXdcd/q3k/S92XHLnGpldVDg0PmSYcMFR+rxfSYlLR9FULHjvMpHZyFt1xB8+f7DyWDklTDgdCv8bYT4kYHOrM5iplTDo1maiKExsvKr1cf1z4pEbReTCIUSYfvJEjB/aL7YxeWscuxNapo4fbLzIeK8mdUy7eD+4Gt/RGW78cb6t0KQzRVlJiZv1TCD0bW4ICQJdXX/1yf3x4f2D+f9NatcZiBnXj/ainnPaiXZ5EmUWJeEYuapRT8FE064UYhjzkmtOxecOHx3Yf+G0cYGZ8SYhzafvH14fYvyQvspj5dBSP0Q0eBx/Z/587iG44pTgpL0iETa1OnMsrDCoMsE89NqDMqFNvxtDnCbCX6q4zIoyf/jcwYEXLTjz1f/c5M1d0Dtide3JX0sheN10DuToc0bu1Bsnf2AX/Ea7ypc6yuTDB+yKX52lO0ePat1df/Qn3Evp8X3jg3v451Wzp0wx6Uxc2lIelCTNS1UtfAey81cuvrmhgEF9hTkjBg1x35ED8ehlRwW26c7r19KA4/ca6q10J67ZAYgzpJ2/R0wYjLOkWdRE5Jk5B+bkS0lKjxUGYkev7XSk4e3U8b7fwOR9l0eBpjbNODPR4eOceoxSZPeMYtTOvdFfEAZatbqB+wzCrX/CMGd0pBu9+Mm4gs7bSvQPxQLhxk8fhP00DmhPIJlczOCgTsVsUi/jZ8SlxDwz1dAMLjjK8QMlDUphTBiwCKG03B+lSoetotrCgKdv0AUrNBULgcFf2smOurP6tTTi9+ccjOvddN5iMkZAikhzkd+DYoFw8THj8ZEDdsXHJ+uzJVeSnuszCJiJ1PAOjT/E2887FD+bsQA3PPm2URm6iWpxxHUYn586GidMGhbpIFdB5E+oAvSNtjEiXPXq0/bGafuOwB6KnDdAWJB5o85ENQ2SNuw6UZ57gxqWFHlmVI8qMgdNFYSBP9NVvT+kGXifWOB57T1iAGa+sx7XnXUAtu/oVOY5UpeftMbZ8Mds6mciZw9I+wx0z7V/L8lM5B7GhYAqXUbY6Qzs3K85sO5Dtem5wkD8LD3jX35iP+zUu8nrFMUFKpLMdvzMYbvj1ueWeCNK4zhoAwdcEkEgjpgCS3xqymmMmMjW0ljE4eMHh7Zz5FmU4oI7ScmrTzG5jsmjMfYZ1NgtEHerTQRfgYDLT5mIU/bZJTb0N3RulaWB7EAOaQYNBW9gF5UVOA7daXKCSM9nEKEZyNpJte+ZikzCgIh2BnA3gNEAlgD4BGMslFmKiEoA5rpflzLGTs9Sbh5ELW4jziqe8dWjsJvQ8SYZpO4+qA+m7TkUj72x2inH8Lw0SxmaIL9v+ugLZ0eH4dwFkaIsDDQNtBokW8s4HlU6YtWks1rFiXOSJgDk3xkLDnYaiwVMHm0WpSZSdTOR3NnKmoGkoYv3Z/zQvli0ZqtSQIReH83Pkhdb8n0G7nUU8yB04ai1JKtmMB3A44yxa4houvv924rjWhlj+2csK1fkWz9iQAsuPnZ86Lg9I5bgM0Gc/ZuXZpCW0NoCmvr4PoPsoaV+2cnJ2j6SpimIo6R6lhqnZS3xzES6SWeh76KhSH1MsvIznJwC+TmrNIOS4Ffguy89bgK+ctwEvL91RyjdjAr5unecfwieenNtaDCpMxMVIgagdaAYZBYGZwCY5n6+DcC/oRYGdYfceJ+7/LiKlBOI+KnxA5d/s86R5k86S26sD4XTep1lHbztEZjUT5WOWCVQaz3Kk7UzGe2iPgg7Y9NQu2gi/eCmVZGipFdjEcUCaSeb9ZVDRqXrHz5ucGhtaiAcWuqtxBZxX9ImZcyTrPaIYYyxle7nVQCGaY5rIaLZRPQCEX04Y5m5kPbmJ11v1TSfeS2IcyDvSGEmahSyrwLBUMW0ZHYgm6SwNrieaqES1Xkm74YuJXoexPsMpO/ChiT3TFt+lV90OR2F/L7I95rvj1trfMTAXrj3osO976a/Sg4t9TUD/Tn1MFaK1QyI6DEAuyh2XSl+YYwxItI1290ZY+8R0VgATxDRXMZYKCSHiC4AcAEA7LZbPlPVdaS9+X7IpNkFAopBjZ+43EjkeHhOU8IFc0TEhHsiaUaLB4/eGfNWbE4ddx239oKISfVKQg5nfxF5XoR/AZPf+tz0Y7G1rTO+0BTwupnOQFb5DLIMVKs96PHGWxrzWLPGdGniSD5o9528z0kT4fGBYFzK7t9+5qCK5adKQuzwhDF2PGPsA4p/DwBYTUTDAcD9q1wrkTH2nvt3MRxT0gGa425mjE1mjE0eMsQ8MVw9c7DggKuVZqB7h3VmIh6pJDYE47Kk70nXpxW58tS98OhlRyUOoY2rk/qY+KM+cfAo77O3wJAya2mY0VLuqMF9mxOnSk6KqZmI/wYGlul5capuJpJG3nGaASduyVAZ3eBJRl6kplyOvqcn7q0aa1efrLrqgwDOcT+fA+AB+QAi2omImt3PgwFMBTA/Y7mZqdb7+mXBKV0PdkEg3kQxfmhf/Psb03DxMWGHehwkNcikC4+INBYLxrHtKuLSWAQwqN7eIwZgxledWalJQ0v/cemRmHWF3i+lWkwmLd4zSDjtjLF8zETVV4CD2rq8klhjsaC8F3KUURzmZiLnrx9a6p5fD7agCLI6kK8BcA8RnQvgXQCfAAAimgzgQsbYeQD2AvBbIirDET7XMMZqLwzSvuwJ7dei/bTW7wJvEDd9+kBc/8SiSNU07ahVHGUCwqitFnPdE/p3TNAmIROOUTX6vs0NIYekyNPfOibgk8hC4nfbEx7JfWIqsiw4nwa5znJOoJDPwP2bdHnapNGAsgO51oEFcWQSBoyxdQBCwx3G2GwA57mfnwOwT5ZyKkF6n0H685O+DGnMNCrkzuHYicNw7ESdrz8bQ/o148gJgz2topYNIZED2bB6cnRUXtqevKh6LqSQLfKs+zRU+1kftccQ3PXiMuyz6wAABmYib6W1ZCMU09/F3w0eSSibseoVOwO5iucnaSMLfnBSXWS4TEqxQPjTuYd4372wuhrUhS+urspAKSOP9LXHuQfIq1jp4GalauKbiQyPd/8yxnLRDKotDE7ZZzjmfe9ET6DKqbblrKKcxoTty/RnyWaiIf2aAQDDB6ZP114Neq4wqMFINUkjyTO6oJo59GXiYsAryYiBvbDwhycbCVXT94EfJZuLdD1F1kmLaZjqxr4fa7iEo/jb8zBUmTpao+jTVEzkL4rSrPrrkiom9RkYvsIfPWAkZi/ZgG+cuCcA4Iz9R6ClsYgTJlVGG8+LnisMUp4nO6eqUWZWfn/OZNw1a2kgrUa1yHsWcFJM13kg6a8OWbipspYCzvraR+1Rm4i4fUYOwJJrTk11bh5mvTwE/7zvn5T6XFkzEAdWBP8Zxs0zkDH9Xb2aioGEc0SEkz5QHxFDUfRcYZDWZ5BBja5VhzhuSF9ceeqkipbx84/v56nDIjyJ1+SYpSvrhbgGH06XrD7u0uMmqHfUIYGf0AUdyDKm4zXTgQKn6xltk9GDhUG6R9vbTUqVxoxT76FlWeDrDssM69+CR75yJMYOqWxMfVZ8x3D0cTxCxNRn0BXo665T0FgsYPhAJzXDgbulD16opVkSiF6eE/CFRVKfXL1HA2WlxwqDtHzpmPFoKBZw9pTKzpDuTuw1vHKLvueFaVSQ7xCXJm7l2E/8+uwDMGZQ9YTn5SdPxIgBLThx711QLBAe+9pRGDtYvcSjCZVMtWGCqSHX1Gfw04/ti188+ma3EPxRWGGQkJbGYpcyAVjMUKWVUFGWfCCJVlMz5PT94qOfVFx5yl54ZtH7ic/r19IYWI1t/NDqO73zhPupTttvBD6lGLTx8FlTzeATB48KzD7vrlhhYLGIxPQPfsx4/Q0Tzz9qLM53l7zsyfBndNjYQTjMXSJWZTmqx2dYS6wwsFhgPsLfY1g/HLXHEHzzg3tWvE6WdJhO8rKyIIgVBhaPC48el9us566Gl402poNoaijgj1+YIpwH9zzbs9QLsinPYka3FAZPf/MYbO+oTHrg7sz0kyfWugo1gyctu/DocanOt/1O/WAFdDq6pTDYbVD1J1dZujYNxUKqiVoZ5iBaKkRcCpR9Rw7E6+9txgDNzOSeSrcUBhZLtbGD0DoiwslPRLj6tEk46+BRmdfJ6G5YYWCxZCBLepLuxnVn7Y+RO9U+GVucz6C5oYh9Rw6sXoW6CD1OGDQWKdVC7xZLFPWycFEtOWP/XWtdBQC1TY6YhHsvOhwbtrXXuhoePU4YPHrZ0Zi3YnOtq2HpJthhRf1RziG/UjWot8i9HicMRg/uU/G1Z2Wu/eR+ePwN5fLQFoF7Lzq8y63hkMcC8pZ88TPl+g+FO4uH9g8nU7Q49DhhUAs+csBIfOQAdSI3E2Z89Si0dZRyrFF9Um8jJUvXRDWB8NiJQ3HdWft3iVTStcIKgy5ALRZIsVi6KqoJhERUNz6NeqW26QUtli6O1/HUuB4Wn2LB6da6msmx1ljNwGLJQhdxVvYkrj5tEob0bcZxe9X3MpP1hhUGFoulWzG4bzOuOq2yK/t1R6yZyGLp5px7xBhMGdM1lh211A6rGVgsGfAjS+vXTvSdDzmj5NHTH6pxTSz1jNUMLJYM7Obmtzlur6E1ronFkg2rGVgsGRi1c2+8etUJNgOmpctjhYHFkpGBvZtqXQWLJTPWTGSxWCwWKwwsFovFYoWBxWKxWGCFgcVisVhghYHFYrFYYIWBxWKxWJBRGBDRx4loHhGViWhyxHEnEdGbRLSIiKZnKdNisVgs+ZNVM3gdwEcBPK07gIiKAG4AcDKASQDOJiKbRcpisVjqiEyTzhhjbwDB5eUUTAGwiDG22D32LgBnAJifpWyLxWKx5Ec1fAa7AlgmfF/ubgtBRBcQ0Wwimr127doqVM1isVgsgIFmQESPAVAtHHolY+yBPCvDGLsZwM0AMHnyZBZzuMVisVhyIlYYMMaOz1jGewBGCd9HutssFksVuf28Q/D+1h21roalTqlGoroXAUwgojFwhMBZAD5VhXItFovA1PGDa10FSx2TNbT0I0S0HMBhAB4iohnu9hFE9DAAMMY6AVwCYAaANwDcwxibl63aFovFYsmTrNFE9wO4X7F9BYBThO8PA3g4S1kWi8ViqRx2BrLFYrFYrDCwWCwWixUGFovFYoEVBhaLxWKBFQYWi8VigRUGFovFYgFAjNVn1gciWgvg3QyXGAzg/Zyqkzf1XDfA1i8L9Vw3wNYvC/VcN8Cv3+6MsSFJT65bYZAVIprNGNOusVBL6rlugK1fFuq5boCtXxbquW5A9vpZM5HFYrFYrDCwWCwWS/cWBjfXugIR1HPdAFu/LNRz3QBbvyzUc92AjPXrtj4Di8VisZjTnTUDi8VisRjS7YQBEZ1ERG8S0SIiml6jOtxCRGuI6HVh285E9CgRLXT/7uRuJyL6tVvf14jowArXbRQRPUlE84loHhF9pc7q10JEs4hojlu/77nbxxDRTLcedxNRk7u92f2+yN0/upL1c8ssEtErRPSPOqzbEiKaS0SvEtFsd1tdPFu3zIFE9FciWkBEbxDRYfVSPyLa071v/N9mIvpqHdXvMrdNvE5Ed7ptJb93jzHWbf4BKAJ4G8BYAE0A5gCYVIN6HAXgQACvC9t+CmC6+3k6gJ+4n08B8AgAAnAogJkVrttwAAe6n/sBeAvApDqqHwHo635uBDDTLfceAGe5228CcJH7+UsAbnI/nwXg7io8368BuAPAP9zv9VS3JQAGS9vq4tm6Zd4G4Dz3cxOAgfVUP6GeRQCrAOxeD/WDs278OwB6Ce/c5/J896pyY6v4AA8DMEP4fjmAy2tUl9EICoM3AQx3Pw8H8Kb7+bcAzlYdV6V6PgDghHqsH4DeAF4GcAicyTQN8nOGs2jSYe7nBvc4qmCdRgJ4HMCxAP7hdgR1UTe3nCUIC4O6eLYABrgdGtVj/aQ6fRDAs/VSPzjCYBmAnd136R8ATszz3etuZiJ+wzjL3W31wDDG2Er38yoAw9zPNauzqzoeAGf0XTf1c80wrwJYA+BRONreRuasmifXwaufu38TgEEVrN6vAHwLQNn9PqiO6gYADMC/iOglIrrA3VYvz3YMgLUA/uCa2X5PRH3qqH4iZwG40/1c8/oxxt4D8HMASwGshPMuvYQc373uJgy6BMwR1zUN4yKivgDuBfBVxthmcV+t68cYKzHG9oczCp8CYGKt6iJCRB8CsIYx9lKt6xLBEYyxAwGcDOBiIjpK3FnjZ9sAx3x6I2PsAADb4JhdPGr97gGAa3c/HcBf5H21qp/rpzgDjkAdAaAPgJPyLKO7CYP3AIwSvo90t9UDq4loOAC4f9e426teZyJqhCMIbmeM3Vdv9eMwxjYCeBKO+juQiPgyrWIdvPq5+wcAWFehKk0FcDoRLQFwFxxT0XV1UjcA3ggSjLE1cJaknYL6ebbLASxnjM10v/8VjnCol/pxTgbwMmNstfu9Hup3PIB3GGNrGWMdAO6D8z7m9u51N2HwIoAJroe9CY6q92CN68R5EMA57udz4Njq+fbPupEJhwLYJKikuUNEBOD/ALzBGPtlHdZvCBENdD/3guPPeAOOUDhTUz9e7zMBPOGO3nKHMXY5Y2wkY2w0nHfrCcbYf9VD3QCAiPoQUT/+GY7d+3XUybNljK0CsIyI9nQ3HQdgfr3UT+Bs+CYiXo9a128pgEOJqLfbhvm9y+/dq4Yzppr/4Hj434JjZ76yRnW4E45drwPOaOhcOPa6xwEsBPAYgJ3dYwnADW595wKYXOG6HQFHzX0NwKvuv1PqqH77AnjFrd/rAK5yt48FMAvAIjjqe7O7vcX9vsjdP7ZKz3ga/GiiuqibW4857r95/P2vl2frlrk/gNnu8/0bgJ3qrH594IygBwjb6qJ+AL4HYIHbLv4EoDnPd8/OQLZYLBZLtzMTWSwWiyUFVhhYLBaLxQoDi8VisVhhYLFYLBZYYWCxWCwWWGFgsVgsFlhhYLFYLBZYYWCxWCwWAP8PXcxjHFF8VNIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "predicted_token_id = 278\n",
    "plt.plot(model.lm_head.weight[predicted_token_id].cpu().detach().numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe6bfdf",
   "metadata": {},
   "source": [
    "No obvious similarity. Can we replicate how logits were generated from hidden layer 11?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b8bd4f72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2289e3276a0>]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3yklEQVR4nO3dd3xUVfo/8M+TMqmQAIHQO6IgghCagkhREFdxbYuu3RV7+7rrT3TtYmFdV911XVlx7QV1XQsozYYKaFB6r9IJaBIgpE3O74+5d3LvnXunT2YyfN6vFy8mdyYzJzNzz3POc8oVpRSIiIh0KfEuABERJRYGBiIiMmFgICIiEwYGIiIyYWAgIiKTtHgXIFgFBQWqc+fO8S4GEVGjsWTJkv1KqZah/l6jCQydO3dGcXFxvItBRNRoiMi2cH6PqSQiIjJhYCAiIhMGBiIiMmFgICIiEwYGIiIyYWAgIiITBgYiIjJhYKBGZ195Jeau3hvvYhAlLQYGanQufGEhrnm1GHV1vJYIUSwwMFCjs/VABQCgjheZClqtuw5/encZtu4/HO+iUCPAwECNlpuBIWg//lyKd5fswJ/eWxbvolAjwMBAjY6I5/+6uviWgyhZMTBQo6PFBaaSwsC3jILBwECNFlNJwdN7WUTBYGCgRouzkohiI6aBQUQ6iMgXIrJaRFaJyK3a8eYiMldENmj/N4tlOSi5iNb8dTMwEMVErHsMtQDuUEr1AjAEwI0i0gvAXQDmK6V6AJiv/UwUEsaF0PEto2DENDAopXYrpX7Ubh8EsAZAOwATALyiPewVAOfEshyUnDj4HDwOMVAoGmyMQUQ6AzgRwGIAhUqp3dpdewAUOvzOJBEpFpHikpKShikoJTy9kmMqicKxZf9hPD1vPRQbFo4aJDCISC6A9wHcppQqN96nPJ+O7SeklJqmlCpSShW1bBny9awpybHHQOG4dPpiPD1vA0oOVcW7KAkr5oFBRNLhCQpvKKX+qx3eKyJttPvbANgX63JQ8uECt9CxlQxU1rgBAMIEm6NYz0oSANMBrFFKPWW46yMAl2u3LwfwYSzLQclFn5PPdQzB4zqGenoGku+Js7QYP//JAC4FsEJElmrH7gbwOIAZInI1gG0ALoxxOSiJeFp6imMMIWjMMVTv5UiUanL2mgKLaWBQSn0D5wkRo2P52pT8eIIHr7HG0NKKavR7aC7u+00vXDWsS1SeU38rOEbljCufqdFiKil4jbUS3Fl6BAAwo3h71J5Tfys4RuWMgYEahbKKGuw7WOn5QR9jaKzN4DjQA0Nje8f0Sjw1JXoDAnpPs7EGy4bAwECNwpDH5mPQlPmmY43pvP7dCwvR+a6ZcXv9aL9Xc1btweuLtkX3SW3olXdKFEeK9feCDQtnsR58JoqKI9oUQ6PGdGIv3vJLXF8/2u/VpNeWAAAuGdIpqs9rpZc7ih0Gb6+pMTUsGhp7DNToeFc+88wOWmNNm3h7DFGMDPpz8vvjjIEhymav2oOv13P7joYQybbb+8or8ebin6NYmsSm14GNrS50awPETCU1LKaSouxarYu99fEz41yS5BfJeX3Na0uwbHspTu3ZEm3zs6JXqATVWHsMtdrUodRoBgYtmRTJdOfVu8rhShN0b9UkWsVKKOwxkKMZxduxdf/heBfDh0RhVtL+g1URP0djYvw7Rz35JU56bL6fRyeOGreeSorec3p7DBEEhvHPLsCYp76OUokSDwMD4dn5G9Dn/tmmY0op3Pneckx47ts4lcqZvsdNJK1gFYPcdTy46xR+OVzt9zGVNW7Te7V5/2HsKquMddGioqbW02OIaipJ+5/rGJwxMCSZsiM12BPiSf/U3PU4WFVrOlartTDLjtRErWzRFklgqPPm3Bt3j+GxWWvQ/+G5jp/TvoOVOPbez/DSN1sBNJ51DCUHqzBz+W5Ua4MMXMfQsBgYGsCbi3/G9l8qGuS1Rj75JYZEIU1Q607Mk8ZYkUeSBtLzzNFqNda46zD1s7V4+dstfj/raAeimSs8lzU5ZAnsum0HPGX5fmt8p8uG6tLpi3Hjmz/i1wpPbyha+yQBhpXPDAyOGBhirLLGjbs/WIGJ0xY1yOsFSiv4Y5zlU5Og/ewat/KOMUSWStKeL0p/54dLd+GfX27CAx+vxoUvLHR8XLTHNPTncxqcrapJzM/RibtOoexIDTaXeMa2qmv1wefovYZ3uupRMr4UDgaGGNO/fI3hoiDGSjJReww17voyuiOo8/S/zqlyOFRVi7V7ym3vs1NrKEzJQefPujZGgcEpSFbV+i4MtNq472DCpNQem7UGfR+c400hVdbEIJWk/c8egzMGhhjzVrAJ+h1ctavMe9sYDGojqXX9+HjZLmyJYKaTsYzhnNh1dQplFTXeHoNTALzmlWKMe3pBWGsl/A1oRzsw6M9nF+CUUtiw75Df31+xowxjnvoa077eHNbrRzugvP/jDtPP+or3cFJJ+w9V4eVvt/iUsT6VFF4ZjwYMDDGmt8JVgkaGZ+Zt8N6uNaWSQi+vUipgRXrzWz9h/DMLQn5uXbW7zrvyOZxK++n5G9D3oTn45bD/6aoLNx/wvp6TsiM1KK/0HfS1pnWWbPvVezvaAVcvv13A+d/SnXj807Xmg5ZKcmepZwyi2FDGUNQp4MUFm/Htxv1h/X7RI3NxkSHNav07jlR7xk4CdRj2lVdiyTbzOMrt7yzFAx+vxrq9B21/h6kkZwwMDiqqazH1s7VBdcX90VukwTas/vXVJvT886cRvSZQX2kqpXxOGNPjDOUyVlrBVGDTv9mCez5Y4f35nOe+xXH3feb4eD1fbLfvUbBqDemucOahf7J8F4D6v7vWZozBOJDrLzD0fXAO+j44B4D5amBHatzYuK++MtIXPQL18/IBT2CJ+Pulld9t83cs/bnUe9spFZOqLRAIdxV5bV0dHpm5Br9/cTEAz/ctlIkW+w9Ve4Mw4FtZ69+VQKmkM55ZgPOeN4/tlFZ4gnalwzhLOD3OA4eq8NwXG0P+vcaGgcHB819uwj+/3IS3Itg2Yev+w94ZQrV1Cq8t3Brwdx7/dC2qausiblm+/cN2PDl7Hd75YTvOe34hPlu5x3tfdW2dd5DaeHKYegx+Xr+0ohpXv/wDHv5kNd4wvD/LdpShqtb5945UR1YJAkBNrfKmFezqsuraOny+di9ufusn717+/rjrFCpr3DhsCAZPzVlveD3/n4P+9lmvH2xMzSjTe1z/fH0fnIOz//4tnpq7Hl+s22f7+GDK73le398xpl+cKtY07Xi4KS5rRf7W99sxfOoXWLa9NODv2qUUfXsMwa1jOGCZdPHs/A1YsbPMtow6f/MOlmz71duQMbrzveX4y+x1fssSrF8OV0Mphb3llSFPMY81BgYH+gXD/VV0utcWbsXqXeaBSqWUqSUEAPd+uAo/HwiuNVVR48YDH63C7rLAlZuduz9YgX98sRGbSjw55m0H6k/CO95dhv4Pz0VVrRv7DYPixmCgt2ztzsfXF23D/LX7fO8IwDieEazyyhpTr8Q4QG7Xyp362Vpc9XIxPl62C4/OWuM9friqFnvLK72zXXRbD1Rg6GPz0duwwK/0SH0l46/H4I+x12EspXVMY93eg3h2/gZc+Z8fAHh6NF0mzwq61a1XpHZjJcbKVK/k9pZbBsaDWEXuL1BZK/JibVqsdWyjxl3n03Md+eSX5udy1/lUxvp5GGysrHXXYVPJITw1tz64O/UMnI6v33sQ5z3/nW8aDvBZGBjuGMueskr0f3gunv9qEwY/Oh9DHpuPbQcO48//W2EbkBpa3AKDiIwTkXUislFE7opXOQKx+9grqmu9LXqlFO79cBXGP7vA9CWpcStku1J9frfsSA0OBDFDaeGmA3j5u63447vLUFZR47OA6am563HaU18FfB691Wj8Oz5e5kmnXP1yMZbvcBp81gKD9vOhqlp8pW0OmBrG/gSrd5XjYi3doNuw9yBufusnx97Jwk0H8NistaZeSY1hjMGuMjNWSOlaa3jlzjL0vn82Bj/qu77jj+8uw68V5vfW2BgI9yQ9WOkJDN9u3G+aQlxj+N7YebfYM/j62qJtqKiuDVjx+NsQLtXmY9pTbq7Y9L/PKTA8/MlqHH//bG859pVXehsbgG9A0nsm1tTW45+uxXnPe65JsWa3/Wyvd5fs8Dmmp5KsKb8Zxdtx6fTFUEqZzqfK2jqM/qv5vHD6DJ1Ske9qV4vbsM93bEIf8/A+d5gNB71B9s4P9Vemu/CFhXh90c9Yrb0/Simc//x3mLNqj+1zxFJcAoOIpAJ4DsAZAHoBuEhEesX6dWf8sB2d75ppaiXrHvlktWlQVG9t2X13et0325s3Nq4YdltSMVnpvoHhnv+twIBH5plalNt/qcAlLy62Xb1aWlGDvg/V57J1z87fEHDGiaf8njLpC4UAwJXm+di/sQwY1tYpLN9RCnedQskhTwWiB5Y/zliGy1/6Hjt+rfCmH5zUuuuw/1AVatx1mLNqD5RStmmDm9/6CR8v24V1e+wHBy/69yK89b05lWesiOxObGPR0rWaMdieSmWNG3+bux7fGd4Xfyk1E8tbon++v7cEQ70X6dQT1VvI077ejF73zcZ9H65yfMm9hkrebqwkmG0kvIHBoZKc/s0WHK52Y+VOT2U19umvTRWvdbwoLdWcmlq7pxxr95Tjp5/rB7fPeGaBqQerq7BJNervhzVw3fnecizYsB87S49gwCPzfB5v5JTCdBpX+feCLQCAjDTf6tFaxnAbDno5DxyqPy/13tzOX49AKYVdZZUo3vYr/m/GsrBeIxLx2l11EICNSqnNACAibwOYAGB1LF/05e+2AgB2l1YiPSUF5ZU16NA8GwDw4jdbzA/WzinrbCL9C6qnUsoMrU1jnrOqts4n7wnA20I/VFmL3Iw0VFTXYvjULwDA1DLQKw67k8Wo5GAVyo7UoHurXNvW5aEqz++/8NVmuFJTcOPI7shIS7H9Qi/cfAD3/m8lzjyhDWYu96yo1asWfU7/xGmLfHL37jplymE/8dla/HvBFpzdty0+0nonFw3q6PN6eiDMsulZOTEO1tr9vca8epoWGILt4Xy/5Rc8M3+D6VgwqUQ7P/1civdtWsD3fLASzbJdmDJzjc1v+VZs7xRvx8PnHG/72GFPfO69bZdKCmaKZ6Aeg25veSX6IM+nd1VhaOCUHanxfg/0Ad9xT3saW33a5Zl+b8RfvgxYNqD++28sn3EmmB6wdHZBwGmyw9WvFOOLP56KLgU53mPG71RGWioe+ng19h2sxE2juuOu91f4LCCtrq1DZY0brtQU22nKO0uPYMH6EvQobAJXagpeW7QVrZtmor1W79itWN/+awVueXupt2ffvlnD7/4br8DQDoDx6t47AAyO9osopTBwynxcNKgD7ji9Z32+Egpn/n0Bdvx6xGd7bL2S0wcTjXXP6l3lGP+seaplqeFEMaYqnp2/wRuI7BypceMSS2sy21X/cZRrleZhmy+OsaUz+NF5qFPAsvtPR8lB3wEs4xfv759vxPRvtjgGm1XaYJ0eFIz0V9zxq++YR1Wt21R2vcVlnMJobfkD9YFh9F+/wvw7RqBtXpY3SDilG0orarzjHsb89r7yStQpc8NdT6XYpVTs2FUgTi1CY6Xv1Ku44137lt6UmWtsB8YPHKrymUHTpUUOtv9SgZZNMpBp6YEaZzgt3V6KwV1b4HBVLXIyPJ9FoDRUXZ3yBj7rWEFpRbX3Km0AUOkwe8r4Xer74BwM6NQMQP33t76sgQOsPoXYqNKbSlJYtPkADlXWonVepvf+615fYnr8yp2+vUP9czW+N7o5q/bg2hHdAADr9hz0VsYAkJGegpe+9XyXyytrsdRmQL30SA0GPDIPV5zUGX8c2xMPfbwK94zvhbzsdADAne8tw7cbD/j8nj8lB6tM5TD+vQ0loa/HICKTAEwCgI4dfVucQfw+lKrffVL/gizcdMBUwRlb6l+u24fRxxXaDrp+ZPiwAM9YwwUvfGf72p+u9K1cjXaXHfFJ5Rgrdj1Hba3EK2vc6PdQfVpJP58fnbkG7xRvh9Vey2CZvx7I/kO+PRz9ffBXx1TW1CHb5Xvcbo6/U1lG//UrDO9RgNeuHoxDVbU4w2Gtw4FD1d4K8Ui1G7XuOizcfADXvrbE52/bU+ZJZwW7M6ddGkJ/rbo6hee+2IiLB3dEi9wMU6VfUeUOaRCyfbMs28BgTInoSg5VYfjULzCyZ0v858pBAIAv1u7D8ZYW+GOfrkWf9nm4+N+L8dY1QzC0Wwvbv8fIrRSqtQrfOvtq5ord+N5wOVK9JZ4i5tlg1vdcTxntKTNfCCm4wOD7/TtiSCXp28oM6tLc8Tmuf+NHn2NVNW4s2fYLznt+If51SX/TfemGVsP5z39nSg1npNUH4uU7Sm1fT3+PXlm4FW3yMjGjeAcKcjNw57hjAQTuraaliE9Qtr6nadHcczxI8QoMOwF0MPzcXjtmopSaBmAaABQVFYU1/J+XnY5SrfWif8keM8w2WLT5gKlldPUrxbh2RFd8vmafXgbvfdYT7fFP1zrOkQ5UGekVv9HucmNg8JTZ2OJfsu0XPD1vg+1rznIIRKFsnnbApsXm7Tn5WaDn1FWv8bOtxrn/9N3Oe8EGT6CsqLbfEA4A3v7hZ+/rHal247kvNuFv89bbPnbemr245a2fMPLYVo7PZ2Rt5QLAjl8rcOELC3Hx4I54c/HPWLWrHMe3a2p6zKHqWr9/q9Vqh96QHb2y/GKdZ+C/qtaNK1/+Ace29r1AzKLNns/6y3X7UFpRHXC9yDcb9+OBj1d7y/Tigs24bGhnuNJSkGtpWVfqexalCOoMf+uRGvNnpddx7xRvNzVUAg3STnjuWyzbXooWOS5TClbfBNCYSvo+xOtn7yg9gnu1sRp9l1mdyzCOYH2/XIYNmkor7Bs5eg9FKXOv61BVLVypKeikpYyceOoJ+0V99Y/x+xQxEa/A8AOAHiLSBZ6AMBHAxbF4ofysdO8Jbzcn3W5zuxe+qp+DvmJnGapr6+BKS/FJ67y6cJvj6wYKDHaV0Kvf1T+fXWvbuoDHKBp7G5XZfPmr3XX447vL/PYYPl+zF5cO7Yx2+VnIdqUGNSj+o2HxlZW/fLfx9ypq3Pg5wLTOT1fuQZ/2eX4fo7vXZqD3h62eFrDe+p2/di8+s8wSqaxxh7SK1q5RECw9/bbWZsBe/069YFhDUZDrsu0JAvBOkdU9MnMNvlpfgmcnnugz8Lrj1wpcOn2xTwD8cKm5F+1k+y/+p13r6x5O7JiPeWt8p0JHskrZeD5bG0quVM957UpL8Wm5B7P63xgw9AkW7jqF4++fjTNPaIOcAONndps4WnsM8VigHZdZSUqpWgA3AZgNYA2AGUop5+kXEcjLSvd+eOHstzJ71V5MmelpVVU4tMAmDuzgc8xulojRa4t8g4qxxRJq5RHJamLdZoc9jN5bssNvYLj3w1X4bpNnhkj/js3w1wv6RlSOYIPcrBW7TY2tHq1ybR839bPwFyTNXb0XANBEa0Hb9Qwqa9wR7WobLHedsm1Q6Iwzz3TNc1y4c1zPoF9jwYb9mPRasc+g+5uLf/b26IyCDQzBOuWYlrbH3UqheY5NvtJBZnpwVdue8kr0vn+27W4DwexMa/zc39PGnfTAPHP57oCTR+zOqyM1btPMv0hXx4cjbusYlFKzlFLHKKW6KaWmxOp18rNdWLGzDGf9/ZuwK89XFm5DXZ1CRVUtuhTkoElmfUcrLUXQItf3C+uzkMjCuH7ATiStyliwy4nfMqq79/bF/64fSD9vQHu/zzX39lP83h/sFNFtBypMYzl/Ght8BejPhzee7L2tT222XsgIAEZoldgdM5b5zGYK5Ks/nYqHJvQO6Xd6/vlTvLhgi+P9dr2nJpnpuOHU7jaPdrZ2z0HT7J7UFGmw3XYvHtQRY47zTf256+zXBTlZ8cBYx/tevWqQ9/aCDZ4UnV2r3Lqhnx27YKzLz04POM5jle1KxbLtpabeSzy2Tk/6lc95WZ7ZASt2ljl2R11BTFt54evNqKh2o2WTDDxwVv0JnZ+dHtaCr0A+D2NlcUO77KTOPseCWfDTo7AJurbMsb3v4U9WY5RlgZI/+kc6vk9r2wAdjt5tmwZ+EICRPT2BwS6tE0iWKxUdmvnPP1vV1inbRWC6TTYpvGba7JhQNM1M9zaiFtw5Eu46FVajqk1eJiad0tX7840ju/l9/G1jeiAtNQXPTDwRY44rNN134FA1KqrdttOe7aT7OaeN01Pt1tcU5GYE9RpNM9MCfvahNvCaZbtQbvkdpxlhsZT0gcE6R75Fjssn5dA2P/B0sE9XerqFOa5UU48hRcTxIimhePzcPhE/R7iGdHWe5eFPQW4Gjik0v5fBLvhxWiQ33bqeJAjHt2uKv1/U39sIMArnfU1LTcF71w0N+Li+HfJDfm5dtivNuxhM17ppJlIEuHxoJ8y4tv71Lx/ayXvbX67dWqEAnh5zqPKy0r0THIKtJI308Ym7xx/nTcE1yUhDt5b2qT5djjblOScjDbeN6WG6b2fpEfxyuNq0aNTps33Xz2eXkWZeb2A3/pLlCq5aLGzqv94orajBYoeB8ptGdscDZ/mu6W2W4/sdZo8hBqyrjy8d2glt880LRtoFsYBk+Y4yrNhZhpJDVRjarQXaaHOLh/do6Z0nP6hzc5wfII3i5PTerb23p55/QljPoQt1FoM+tS6Qlk18KwnrgJ2em371qkE4f0B73HWG/XNHcnH3M45vbfo525WG1BQxraXQdWyR7Xd6o5MBnZqhX4d8jHKY0VT85zE+6wpCkZWe6tOqLczLxObHzsSDE443ldku4AXL2mOwC8gzbxlm+rmgSYZ3eqbd6t+2DvPqRYDVD43FrFuHY+p5J+Csvm2RoeX6Myw5/56FTTC+j/lzNM4QsgZNXWZ6Csb3aY1h3QswcVBH9O+Y7/N9ty6mM1o0eXTAhpzdjgV2QumhDuteYPp50oiutusTmtkEcvYYYsD6IWe7Un1y2G3zfAPDE+f1wX+uGOhzfOXOcjTJTMfCyaPx7V2j8PA5vb0tkIz0FDx5QV9Mu3QAJjtUiAAwvEeBzzHjYNmFRebB7HSbk+TTW4c7Pr/dTpr+0iMndsjHn888Di9f6fv36trlZ6GFNvg3vk9rnNgxH4BvC1ZPJZ1yTEs8eUFfXDeimzflYhTO3mODOjfHJUM6+gQy/UR3ykG/8YfBWP3QWMy9/RQsvnu07WOuOrmL6WcRwX+vPwnTLy/yeezJ3VugINd3wVkoUlPE+7l2KcjB4C7NMcVhhXOen1Z/oPhq7THYbRzXq43nuzGka3N0bZmDr9eX4FNtN1671bzWBoK+MtszK83TM7hQm5ChrwWwpmsL8zLRv2Mz0zFjusqpR5mVnop//n4AXv+DZz3sf284Gef1r2+MDe9R4P1cZt4yzNsbLmyagWcvOhHNclwBG05XWr4LS/48xvZxh6t8K2y7iSiZ6Sm4z9I7yEpPtW3ItLAZYGePIQYyLZVFlivNp5Vr1xL+3cCOGHlsK59c+NO/6+e9rZ8IeutXn/V0eu/WuHZEN+8qUKuWuRneE/r/jTsWV5zU2bGVsuWx8Zhz+wjTsdyMNBzXxrmit5t99f71J5l+1luhAzo1g4jgD8O74tSerdDRZt71A2f1wqtXD/LObX/qwn744AbPAK11UNJuSrBdeVo1DT1F0TQrDY+c0wddCnIw7dIB3q0C9EBoV1ELBOmpKch2paFHYRMUNs3E/512jM/jjmvjuy4gJUVMZf/ghpOw+dHxeOMPQ7TXcz59rL0aO3qPISMtBe9cO9Rn0ZpOf50uBTm4sKi9aXA2UM9Ln8nzp7E9kZ4q3jEZYwUmIvj8jhGYfvlAn91nAWD2bebJAtZA1VJLN9mN1elld1l6Hq1szrkK04C3/Xtr9xnrAeWZif3w2tX1Gyj0bpuHrlr6ql+HfJzdty0A/1fYAzzbt7zxh/rnaWFJpz3/+/544rw+tpvsPX6eb2//45uGmQLddSO6ad9J37/Fev6N7NnStiEZa0kfGKwVbnqKeHdGLdQqJ3+5wk9vHY41D43z/nzOie18HqO3WK1ft7euGYKTu7fwefxxbZp6HzugUzM8cHZvn8pTH3wTEZ+/IdBAoF1X2XhCPXFeH8y+7RR8f89ovG44kQBgym+PN7Va3p40BFec3AXdWuYiNzMNrtQU03M59RiM9NI8NKE3frr3NADAo7/t4zOt9aUrfFvnRsY546f3bo0LBngqN/3PtaY9rjq5CwZ29g3Og21SS02DSNe0zc8yVSqZac49lDNPaOP9uSDXhZ6FTXDP+ONMj7M2KJzo73HPwiaYen5fnNarfmA2UEJO/47fOLI7NkwZ7z3e2xKEurbM9dkuQndMYa5pEDnf8l411cbc7AZ8czM8j1UARh9bX+4WOS7v35+vpbv6dagvk/E7bJymarevlj4ekuHweQDm62UEk8Y82ZL6MY75DOzSHL8b2BE9WtU3JlxpKabP3KhVk0xv7+ACQ3rV7m851ZK6/M+Vg/CXCKd/h+OoCwwpUr8E/a8X9MOLlxX5DEYbF6VkpKUiy5WKx8/t45gecmqBuNJS8K9LBuAKw+ydZyb2w1XDungrA2Pax7ja9J+/7++tRK0tU72i+PtFJ+LlKwfimYn9TGmgQFe7GtCpOVrnZaJVk0yfL+fwHi2xRHtdABjStT6w5WSkmQbeAd8xBruppvp5WNg0E820k7xD82yM6WWeedIuPxuf3eacIrO2OvV6yHuxHMsJf99Zvbwb6RkN7toCPxr+RgA+f5edlpaWozVvDgArHjgdJ3cvwDGF9ZVGighm334KTu9t/nvrL/LjXz9tkFufBmyswPX3v6lD+Vs1sW/0FDisCbB7HhHB3eOPQzttbC7fMm5RqOXKB9tMYtB73PvKq5CXnY6rh3nSNM1zXN7vxTn92uGHe8ZglCFwpBrSp8bvhF1P7OZR3dE2LxNDu/o2wgK9t8YUa7YrFXcYepO/K+qAf1x8IgDzNhx6S//lKwfid1rad9Hk0XjuYvN2G7osVypa52Xi45uG4ZHf1qcLc2xSSf07NsOCO0cGKHXsJfReSdFgnGHwylWDcFK3FnArheU7VqBX26ZonuPCD5bVkBNsegUT/UyTcxgnA+CZR35W3zbeDfUm9PM8t/4rxvGDRXeP9lb6rrQUuNI8J6+xhd4uPwvXjfC03s7SusZW1rjw/vXmWRrBTM+1M/rYVii0VDTWffftxmucTk/r2IkI0LOwKVyW3V8LcjNw6+jupgF6ILzrQuisi6WaZqbj3euG+l2oZm0A2PUYmmR6Ks1jCptgzu2n4PS/fe1N31hb1Po2I4EasMcUNjFt9mhs7GS7UnF82zzcOKo7Jr+/HLvKKjH39lNw2t++BmCfsgF80yO6mbcMx0fLdtlepUzfHkYftzi+XVPcPKoHurXMxSc3DzMFQ50+NVRPHervb7Mcl3cnAaWUTzrXmHoxDsjalbtvh3x8N9l+7Eh/b43vsd7QumVUd/zf6T3R98E5KDtSg3n/N8I0MeUJh0kg+ufeIjcDj5/XBw+d09tvb0Vv0FhX4BtTSaOObeWdoq7v+Gw3XtFQkj4w6JWqSP2CpIsGdTTNh9ZP2PRUwaxbhpvmOQdDrzCcTnC7TbD0xxrvs+5PozOmSL69a1RI5VEKPimC9LTwZgSd2789zrU0ii4d2hnPaou7bhzZzbtTpZHTRnzWilIfr7BuC54intex8vYYbPZx+u8NJ/kc8yfLleqY43/zmsG2A4D6+3zx4I4Yf3wbrLRc98EagPXKTh/f0d+PQKkN6/MY988a3qMAL1zqScHNu2MEamoV8rLT8ZfzT8DDn6x2DABOM/E6NM/GiGNa2gYGPcDp5c/PcmGsFqydx0dSMf3yIvTU9nbSA0OLHBeKtDE4u4aY/p6kp4rfNQnBMr7F2a40rH/kDG/DRO9hB5NOBMwNBBHxGxT8MfbWX7h0gGmMZd0j45Aeh83zdEdNYGjtZxxBP/EKcjPQw6bVE0igE9tu6p1om2fZzTiyfyxwTj/7HoJV5xY5WFpRirQUQY1b+XRZo7lb4+1jeuDW0T2QIs658pE9W2Hu6r3oZhnIN7YKrzy5s3djuMz0VNPCIKdtEvz1GKwzXuy8fvVgXDLds2LbrluvO6mb8+DfpkfHe//2YZZBwnRvQPfUqAW5GbhoUAdcrqUWe7Zugj7t8vDA2f6vUWXtqZzWqxAXFrXH+QM6mKZmZrvSAK1xfUFRB1xQ5NzitKbFjJyuj6EHYD3dFOwK9dGGxWpn9mmDr9aXoHfbPLTOy/TZ9l6nt6ZvHBnaqm0r67W4dca05OtXD8asFbsD7mv0zqQhQW3gN/+OEbjm1WLbgXwj46yk9NQU5GXVlyncYBMtSR8Y9O6vv7nsLq0F7XQN2ED0FodTFW/X4tEfG2g8QLf24XFBt5z+fVkRvtu0Hw9+vBq/HK72mckRbirJjoj4TaUBwEWDOmBs70Kf1quI4D9XDkTvNk3RyhC4bxrZHfd/5Nk6680/DMYAmwFkwH8KLxjDehRg4eRRmL9mX9h73vv7/PSg7+0ZpAgeO7c+PZGZnoqPbx5m96t+ZbvSMPX8yAYk/TVInKb96n+H/n0KZ2O7Cwd2wHkD2gf83udkpGHtw+O8veUrT+6Mvu3zQ349byrJz2hDr7ZN0SuI1e6Du7bAYJtxDKtuLXPx2a2n4Bib/ZeMgj334yHpA0P/js3wyDnH49z+vt1VnV7hhruLYaAFM7aBQZzvsxPKnPmWTTIwoV877Cw9gqmfrUPTLEuPIYga9dSeLdE0M/yFVUYi4pjSGNnTdwHZ5Sd19gaGbq1yHVtPqdp7Z4znx7ZuEtIJ1yYvC5cM6RT4gWHQvxfh7jL0/vVDUbz118APDMHd44/Fc19s8jsTymnq9JTf9sFjs9Z4Z/EFs/uonWA/H+N3/v6zQttXykcD18GutBQsu/90VAWaQZgiuGxobL5/kUj6wCAiAU/8dJsKJrTXqH8tO3aLdfQWTDCVdLDuOuNY74psALjh1O62G6gFE4xevnJQwMc0BH9pOruA/Jllzn08hbObr9GATs0xoFN425U4mXRKN0w6xTMO5DT7RU8lWce8TutViNN6FWKFtgFkbbDXw46jeLbJ87LSgQDjFpseHe/3/nhJ+sAQjDRvlz82LSC7ilj/lWh2J6+zGfg1+u2J7fDBTzsdV5UmIn/vTxQzYjHRLDsdV57cOextUmKtg8NFZDLSUvGnsT19NrLT6edLQ+24Gg2N5xufGBgYEHmXP9AYg/PgMxp05sHU80/Afb/pFXDlZyLxl6aLxa620SQikadA4sTfoK8+PmF3kZlEo59nkfbejjaJfWY1EH0RVJswByADzUqyq/z132jISjo9NcW7wKyx8Ff3J3qPIVm1b5aN3Iw03Dk2uM0XqfFhjwGeXOCzF50Y9vbT9Vsb2N9v12MY2KU5Pl+7L6jpqkczf1Nr9R5DuGNDFJ7M9FSsfND5QjiJiGdZaBgYNGc7rCIORqCWq11g+MfFJ2JzyWHbHRapnt8eA9MDFIDdymcKjJ3xKKjPXzps/WBTu2W70hxXi1I9/2MMnv/tVj4TAf7XL5CzmAUGEfmLiKwVkeUi8oGI5BvumywiG0VknYg0rj5pGBrTYG+i8T8rie0aCg7PwNDE8syaC+B4pdQJANYDmAwAItILwEQAvQGMA/BPEYnv+u8oYXc1+vzNJuHgMwUSaI0R2YvZqaWUmqOU0je8WQRAn8w9AcDbSqkqpdQWABsBJMZqqjBx8DM+OPhMgTAchKeh2lxXAdA3DmkHYLvhvh3aMR8iMklEikWkuKSkJMZFpMaGg88ULH5TQhPRlBgRmQfA7hqG9yilPtQecw+AWgBvhPr8SqlpAKYBQFFRUQK3C7V99eNciqNNinfwmchegHkh5CCiwKCUsr9KtkZErgDwGwCjVf1+EzsBGPcDbq8dIyKKKo4thCeWs5LGAbgTwNlKqQrDXR8BmCgiGSLSBUAPAN/HqhwNid/BhqVPReTbToFw2mpoYrm66h8AMgDM1aL2IqXUdUqpVSIyA8BqeFJMNyql/O9Nm+A4+BlffPvJiTeTxLgQkpgFBqWU4y5cSqkpAKbE6rUbml4x+WuVjD62Fbq1ym2YAh0leLJTQPyOhIX7MTSQ6VcMjHcRkhe7DBQA40NouEQoitiCJUos3nEonpshYWCIAo4xxAfPdQqEASE8DAxRUNjUcz3jHoVN4lyS5PGbE9qgWXZw15zmJnoUCGclhYZjDFFQ1Lk5Zlw7FAM6NYt3UZLGPy7uH/AxnKNOgXBWUngYGKJkUJfoXrSdAuuoXbN4fJ82cS4JUXJhYKBGq3VeJtY8NA6Z6cyIkj1eqCc8DAzUqGW5kmLHdoqR+rEFRoZQsKlFREQmDAxElLSYSgoPAwMRJS0mksLDwEBERCYMDESUvIRbYoSDgYGIklZ9KomRIRQMDEREZMLAQERJiymk8DAwEFHSYgopPAwMRERkwsBAREmLqaTwxDwwiMgdIqJEpED7WUTkWRHZKCLLRSTw/spERGFgXAhPTAODiHQAcDqAnw2HzwDQQ/s3CcDzsSwDERGFJtY9hr8BuBPmy7VPAPCq8lgEIF9EuKE+EUUdU0nhiVlgEJEJAHYqpZZZ7moHYLvh5x3aMbvnmCQixSJSXFJSEqOSEhGRUUTXYxCReQBa29x1D4C74UkjhU0pNQ3ANAAoKirihX2JKCS8/Gt4IgoMSqkxdsdFpA+ALgCWaR9MewA/isggADsBdDA8vL12jIiIEkBMUklKqRVKqVZKqc5Kqc7wpIv6K6X2APgIwGXa7KQhAMqUUrtjUQ4iIgpdPC7tOQvAeAAbAVQAuDIOZSCiowAzSeFpkMCg9Rr02wrAjQ3xukR0dOOWGOHhymciIjJhYCCipMVUUngYGIgoaTEuhIeBgYiITBgYiChpMZUUHgYGIkpanJUUHgYGIiIyYWAgoqTFVFJ4GBiIiMiEgYGIiEwYGIgoaXHb7fAwMBBR0mJYCA8DAxERmTAwEFHSYiYpPAwMRJT0GCBCw8BARElLjweKV4wPCQMDESUtzkoKDwMDESU9xofQxDQwiMjNIrJWRFaJyFTD8ckislFE1onI2FiWgYiIqaTQxOyazyIyEsAEAH2VUlUi0ko73gvARAC9AbQFME9EjlFKuWNVFiIiCl4sewzXA3hcKVUFAEqpfdrxCQDeVkpVKaW2ANgIYFAMy0FERzmmkkITy8BwDIDhIrJYRL4SkYHa8XYAthset0M75kNEJolIsYgUl5SUxLCoRJTMmEoKTUSpJBGZB6C1zV33aM/dHMAQAAMBzBCRrqE8v1JqGoBpAFBUVMSPloioAUQUGJRSY5zuE5HrAfxXKaUAfC8idQAKAOwE0MHw0PbaMSKimGAqKTSxTCX9D8BIABCRYwC4AOwH8BGAiSKSISJdAPQA8H0My0FERzmmkkITs1lJAF4C8JKIrARQDeByrfewSkRmAFgNoBbAjZyRRESUOGIWGJRS1QAucbhvCoApsXptIiIjppJCw5XPRJT0mEoKDQMDERGZMDAQUdJjKik0DAxERGTCwEBERCYMDEREZMLAQEREJgwMRERkwsBAREQmDAxERGTCwEBERCYMDEREZMLAQEREJgwMRERkwsBAREQmDAxERGTCwEBERCYMDEREZMLAQEREJjELDCLST0QWichSESkWkUHacRGRZ0Vko4gsF5H+sSoDERGFLpY9hqkAHlRK9QNwn/YzAJwBoIf2bxKA52NYBiI6iile7DkssQwMCkBT7XYegF3a7QkAXlUeiwDki0ibGJaDiIhCkBbD574NwGwReRKeAHSSdrwdgO2Gx+3Qju22PoGITIKnV4GOHTvGsKhElIyEF3sOS0SBQUTmAWhtc9c9AEYDuF0p9b6IXAhgOoAxoTy/UmoagGkAUFRUxD4hEYWEqaTwRBQYlFKOFb2IvArgVu3HdwG8qN3eCaCD4aHttWNERJQAYjnGsAvACO32KAAbtNsfAbhMm500BECZUsonjUREFCmmksITyzGGawA8IyJpACqhjRUAmAVgPICNACoAXBnDMhDRUYyppPDELDAopb4BMMDmuAJwY6xel4iIIsOVz0SUtJhKCg8DAxElLaaSwsPAQEREJgwMRJS0mEoKDwMDESUtppLCw8BAREQmDAxElLSYSgoPAwMRJS2mksLDwEBERCYMDESUtJhKCg8DAxElLaaSwsPAQEREJgwMRERkwsBAREmPIw2hYWAgoqTHkYbQMDAQEZEJAwMRJT2mkkLDwEBESY+ppNBEFBhE5AIRWSUidSJSZLlvsohsFJF1IjLWcHycdmyjiNwVyesTEVH0RdpjWAngXABfGw+KSC8AEwH0BjAOwD9FJFVEUgE8B+AMAL0AXKQ9logoZphKCk1aJL+slFoD2C47nwDgbaVUFYAtIrIRwCDtvo1Kqc3a772tPXZ1JOUgIvKHqaTQxGqMoR2A7Yafd2jHnI7bEpFJIlIsIsUlJSUxKSgREZkF7DGIyDwArW3uukcp9WH0i1RPKTUNwDQAKCoqYtAnorAwlRSagIFBKTUmjOfdCaCD4ef22jH4OU5EFBNsVYYmVqmkjwBMFJEMEekCoAeA7wH8AKCHiHQRERc8A9QfxagMREQUhogGn0XktwD+DqAlgJkislQpNVYptUpEZsAzqFwL4EallFv7nZsAzAaQCuAlpdSqiP4CIqIAmEoKTaSzkj4A8IHDfVMATLE5PgvArEhel4goFEwlhYYrn4mIyISBgYiSHlNJoWFgICIiEwYGIiIyYWAgIiITBgYiIjJhYCAiIhMGBiIiMmFgICIiEwYGIiIyYWAgIiITBgYiIjJhYCAiIhMGBiIiMmFgICIiEwYGIiIyYWAgoqSVluqp4tJTWdWFIqIruBERJbILizpg+y8VuHl0j3gXpVFhYCCipOVKS8Hk8cfFuxiNTkT9KxG5QERWiUidiBQZjp8mIktEZIX2/yjDfQO04xtF5FkR4cWViIgSSKSJt5UAzgXwteX4fgBnKaX6ALgcwGuG+54HcA2AHtq/cRGWgYiIoiiiVJJSag0AWBv9SqmfDD+uApAlIhkAmgNoqpRapP3eqwDOAfBpJOUgIqLoaYih+vMA/KiUqgLQDsAOw307tGO2RGSSiBSLSHFJSUmMi0lEREAQPQYRmQegtc1d9yilPgzwu70BPAHg9HAKp5SaBmAaABQVFalwnoOIiEITMDAopcaE88Qi0h7ABwAuU0pt0g7vBNDe8LD22jEiIkoQMUkliUg+gJkA7lJKfasfV0rtBlAuIkO02UiXAfDb6yAiooYV6XTV34rIDgBDAcwUkdnaXTcB6A7gPhFZqv1rpd13A4AXAWwEsAkceCYiSiiiVONI3YtICYBtYf56ATxTaBNRIpcNYPkilcjlS+SyASxfJPSydVJKtQz1lxtNYIiEiBQrpYoCP7LhJXLZAJYvUolcvkQuG8DyRSLSsnFnKSIiMmFgICIik6MlMEyLdwH8SOSyASxfpBK5fIlcNoDli0REZTsqxhiIiCh4R0uPgYiIgsTAQEREJkkdGERknIis0679cFecyvCSiOwTkZWGY81FZK6IbND+b6YdF+0aFRtFZLmI9I9x2TqIyBcislq7rsatCVa+TBH5XkSWaeV7UDveRUQWa+V4R0Rc2vEM7eeN2v2dY1k+QzlTReQnEfkk0conIlu1658sFZFi7ViifL75IvKeiKwVkTUiMjSBytbTsDh3qYiUi8htiVI+7TVv186LlSLylna+ROe7p5RKyn8AUuFZWd0VgAvAMgC94lCOUwD0B7DScGwqPNuFAMBdAJ7Qbo+HZyW4ABgCYHGMy9YGQH/tdhMA6wH0SqDyCYBc7XY6gMXa684AMFE7/i8A12u3bwDwL+32RADvNNBn/H8A3gTwifZzwpQPwFYABZZjifL5vgLgD9ptF4D8RCmbpZypAPYA6JQo5YNnV+otALIM37krovXda5A3Nh7/4NmmY7bh58kAJsepLJ1hDgzrALTRbrcBsE67/QKAi+we10Dl/BDAaYlYPgDZAH4EMBieFZ1p1s8ZwGwAQ7XbadrjJMblag9gPoBRAD7RKoZEKt9W+AaGuH++APK0ik0SrWw2ZT0dwLeJVD54AsN2eK5xk6Z998ZG67uXzKkk/Y3T+b32QwMrVJ4NBQFPS6RQux23MmtdyxPhaZUnTPm0NM1SAPsAzIWnF1iqlKq1KYO3fNr9ZQBaxLJ8AJ4GcCeAOu3nFglWPgVgjngusTtJO5YIn28XACUA/qOl4V4UkZwEKZvVRABvabcTonxKqZ0AngTwM4Dd8HyXliBK371kDgyNgvKE8LjOGRaRXADvA7hNKVVuvC/e5VNKuZVS/eBpmQ8CcGy8ymIlIr8BsE8ptSTeZfFjmFKqP4AzANwoIqcY74zj55sGT4r1eaXUiQAOw5OaSYSyeWk5+rMBvGu9L57l08Y2JsATYNsCyEEUL5OczIFhJ4AOhp8T6doPe0WkDQBo/+/Tjjd4mUUkHZ6g8IZS6r+JVj6dUqoUwBfwdI/zRUS/loixDN7yaffnATgQw2KdDOBsEdkK4G140knPJFD59JYllFL74Lk+yiAkxue7A8AOpdRi7ef34AkUiVA2ozPguQLlXu3nRCnfGABblFIlSqkaAP+F5/sYle9eMgeGHwD00EbpXfB0Bz+Kc5l0HwG4XLt9OeqvSfERgMu0GQ5DAJQZuq1RJyICYDqANUqppxKwfC3Fc20PiEgWPOMfa+AJEOc7lE8v9/kAPtdadTGhlJqslGqvlOoMz/frc6XU7xOlfCKSIyJN9Nvw5MpXIgE+X6XUHgDbRaSndmg0gNWJUDaLi1CfRtLLkQjl+xnAEBHJ1s5j/f2LznevIQZv4vUPnpkC6+HJS98TpzK8BU8OsAaeVtLV8OT25gPYAGAegObaYwXAc1p5VwAoinHZhsHTFV4OYKn2b3wCle8EAD9p5VsJ4D7teFcA38NzTY93AWRoxzO1nzdq93dtwM/5VNTPSkqI8mnlWKb9W6WfAwn0+fYDUKx9vv8D0CxRyqa9Zg48reo8w7FEKt+DANZq58ZrADKi9d3jlhhERGSSzKkkIiIKAwMDERGZMDAQEZEJAwMREZkwMBARkQkDAxERmTAwEBGRyf8HPM0V0BN843YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#tf_utils \n",
    "plt.plot(hidden_states['h.11'][0][0].cpu().detach().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "69000c63",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2289e395b50>]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9vUlEQVR4nO2dd5wU5f3HP9/dvcIdHO2OXo6OIIh4IiooAipilNh+UWOsCdFo1BgLamKPGoklGhv2mMTeQVFQxAp4KL33IhxHPeC4u73d5/fHzjM7OzuzOzM7W273+3697nW7s7Mz35155vk+3/J8HxJCgGEYhsk9POkWgGEYhkkPrAAYhmFyFFYADMMwOQorAIZhmByFFQDDMEyO4ku3AFpKS0tFeXl5usVgGIZpUsyfP3+nEKLM7vcySgGUl5ejsrIy3WIwDMM0KYhoo5PvsQuIYRgmR2EFwDAMk6OwAmAYhslRWAEwDMPkKK4oACJ6kYh2ENESzba7iGgrES1Q/sa7cS6GYRjGHdyyAF4GMM5g+6NCiCHK38cunYthGIZxAVcUgBDiKwC73TgWwzAMkxqSHQO4hogWKS6i1kk+F8PkLIcaAnj3xy3g8u6MHZKpAJ4G0AvAEADbADxstBMRTSSiSiKqrK6uTqI4DJO93P3RUtzw5kI8OnM1yidNw97ahnSLxDQBkqYAhBBVQoiAECII4DkAw0z2myKEqBBCVJSV2Z7JzDAMgO01dQCAxz9fDQBY+nNNOsVhmghJUwBE1FHz9iwAS8z2ZRiGYVKPK7WAiOg1AKMAlBLRFgB3AhhFREMACAAbAPzejXMxDMMw7uCKAhBCXGCw+QU3js0wjH04FsxYgWcCMwzD5CisABiGYXIUVgAMkwXoXT5E6ZGDaVqwAmAYhslRWAEwTBbCQWDGCqwAGIZhchRWAAyTBfCAn3ECKwCGYZgchRUAwzBMjsIKgGEYJkdhBcAwDJOjsAJgmCyAF4JhnMAKgGEYJkdhBcAwDJOjsAJgmCxE8MwAxgKsABiGYXIUVgAMwzA5CisAhslCCFwPmokPKwCGYZgcxRUFQEQvEtEOIlqi2daGiGYQ0Wrlf2s3zsUwTDT6aQAcBGas4JYF8DKAcbptkwB8LoToA+Bz5T3DMAyTIbiiAIQQXwHYrds8AcAryutXAPzSjXMxDBMNLwHJOCGZMYD2QohtyuvtANob7UREE4mokogqq6urkygOw2QvXAmCcUJKgsAiVKjEsIkKIaYIISqEEBVlZWWpEIdhGIZBchVAFRF1BADl/44knothGIaxSTIVwIcALlFeXwLggySei2EYDewSYqzgVhroawC+B9CPiLYQ0RUAHgRwMhGtBjBWec8wTBLQp30KAG/P34INOw+mRyCmSeBz4yBCiAtMPhrjxvEZhrGHEAI3vrUQLQp8WHz3qekWh8lQeCYww2QB0RPBQuyvb0zouD9u2oOqmrqEjsFkLqwAGCYLCQbdCQKc/dR3GPPwbFeOxWQerAAYJgtpdEkBAMCBBK0IJnNhBcAwWYhbFgCT3bACYJgsJMB5oIwFWAE0UQ7763Tc9NZCW98Z99hXuOTFeUmSiMkkAmwBMBZgBdBEOeQP4K35W2x9Z8X2/Zi9iust5QJBtgAYC7ACYJgsQN/fNwZYATDxYQXAZDWPf74ak95ZlG4xUg5bAIwVWAEwWc0jM1bh9R82p1uMlBMIpluC9LPvkB+7DzakW4yMxpVSEAzDZBacBQQcec9nCApgw4Onp1uUjIUtAIbJAvTF4AJsAoAToeLDCoBhshA3ZwIz2QsrAIbJQjgIzFiBFQDDZCHsAWKswAqAYbIA/YCfLQDGCqwAmiCCH24mDlwKgrECK4AmCD/c6eO1eZtQPmka6vyBdIsSEw4CM1ZgBdAE4Rzv9PHYzFUAgL21/jRLEhs3ykE3BUtz54F67K/L7HuRySR9IhgRbQCwH0AAQKMQoiLZ58x2msBzyaQZNwYJTaGdVdw3E22K8/HjX09OtyhNklTNBD5JCLEzRefKetgFxOjRtwhXLICEj5AauNyDc9gF1ARhF1D60c+8zTTciAE0BRcQkxipUAACwGdENJ+IJuo/JKKJRFRJRJXV1Vyr3goiRTneK7fvx5Oz1qTmZE0EAqVbBEu4YSWmovsPBgUvX5lGUqEARgghhgI4DcDVRHSC9kMhxBQhRIUQoqKsrCwF4iSPPSkyRVNlAZz91LeY/OlKNPKsIpVMH/lLXFEAKfipYx6ZjSPu+Sz5J3JAQ2MQ5ZOm4fmv16VblKSRdAUghNiq/N8B4D0Aw5J9znQwY1kVjrx3Br5fuwsA4A8EUd+YnFRBtyf5/Lz3kOH2usZQx88up2js9K8/bNgNf7KVqE4eN+5ZKiaTrd95EPvrGpN+HiccrA/J9cQX2WsFJ1UBEFExEbWQrwGcAmBJMs+ZLio37AYALNyyFwAwavKX6PeX6Uk5lxOT2Sxv/cuVO3Dcg1/g06XbTb/LQecw0gVk9R4s3rIP5z3zPf7x6cpkihVFMtwqc9btSr4iyyBI8fZlcywk2RZAewDfENFCAPMATBNCJKdXTDdqYwn932oyqnYDJ6O7Xz37veH2xVv2AQAWKYpLi/R2N8VJRXd8sATnTzH+zW5gdgvmrtuFLXtq1fe7DtYDAJZtq0nofJ8vr8JbldYXtnHbBbRoy16cP2UOJqdYkaWTptju7ZLUNFAhxDoARyTzHJlGKnzETtrlQqWj1yMPZRTclCOgQBNcX/bf329M6vHf/WkLrh3dBx5P5HX71ZQ5AMKLkHiVzxN1p1zxSiUA4LyKrpb2dycIHD7GrgOh+NbK7fsTPm5TQV7Dptf6rcNpoC4hO9BUWIupzpqwa3HUNjTihjcWYNeB+iRJ5B67DzY4yiN/bOZqvPbDprj7eRQtWucP4sa3FmLbvuRZhlrcnggmBwOZVGQu2c+BagFkzk92HVYALkEpzA7Uju5em7cJP23a4/hY8nk2kl8qNbujybcqt+Ddn7bisZmr4+4bDAr8ffqKCLdJKhl67wwMvXeGo+9ayfqS13X+xj14e/4W3PXhUkfnsou8Z4m0S+1dpwQb+IH6Rtc77GS7aHIh+y3nFcCK7TV46dv1CR1j7rpdePrLtQCsB4x++eS3uPC5OY7Opx2F3fruYpz11HeOjqPF8PFWNtp90OQ1sNJnLNtWg6e/XIs/vvaTrXNkAlY6Ra9un1QNoN1wAWnbmfwVTuTfd8iPw+/8FI8qdZSc8uI367FmxwH1fbKTExrZBZT9jHvsa9z90bKEjnHlf+arr60+IAs278V3SsqoXdw0w2PFLORDbzcGIPf2WOgg5W+R2SUV983Eg5+ssHU+SzIloee18vtS1Xno76PsvBIZt2svmfytTtre3tqQpfT+gq2OZQkEBe6ZugxnPfWtuq0xmNwRuhoDyCC3l9vkvAKQJHKTvZpAYCqaSlIs0xidmd0HLZGB2c4D9Xhm9lrnBzAhGc+wJ07vuudgA+7/eHmkHO6Lger99ajcGOkGdMXdYhADSFdfKAcItQ3hdObGJCcnyHNmb/fPCkDFn0Bj0o4EUxIEdtMCsHAovan98rfrsWDz3ohti7fsw1lPfYs6f8CWCyhVpRXcmsym/U3xLIC7P1qKRbrsq2S0j7Oe+jbquPrfu++QH9OXbLN1XK1VIX+pk7bnRoKE7Iy1Slfrmrz4xXnOD25C2AJw/dAZAysAhUTMyQgFkILxgpu+z3AaaDRqGqjuCbjro2X45ZPf6rYtxU+b9mLJ1nCHJx/8xhizop1er237DhnOXTDDLaWpPYz+mPpR94F685ngd324FIPv+tQVmbbsic4sCgeBQ/fgxrcW4sr//IgNOw9aPm7kb43eZhU3EiTkaN8fEPhgwVYIIXDftLDr9qtV7tcRC8cAslcDsAJQSMQC8MbzBehI1KfoaipejNG67MDnrN1lOUtHhA+pHvPcZ743nRXtVJmN/PssnPmvb+PvKOUyOM20RfZGxHoe+GRFxAxrvaIMGA4qQvu8/N0G1NQ1Yl+SFpb5enVk9XWpJA7UWy+7oP018rfJzvDWdxejfNI0WzIlZAForuV1ry/A3lo/Pljws/MDWkAqHbYAcoBERtUezVW00lgaEnTiG4m6cPNePPF5/LRLO8gO/K6PluHEyV/G3lf5L0S4k5B6Ue8u0iJHWUu21uCOD6xXCbGbmRQUAuWTpuGa//2obrta8zoWM5ZVoXp/aE6DXlHuqQ2ngurbkNGYQt8+jrjnM+zYX2dJjkTweeyn9GoHKlKZyU2vzTOfA/HIZyvxhmaOhP6ci7fsw0PTV0Qc/4cNu9U6WkboB2jxniE3ArfJDjJnAjmnAN6Zv8Uwbz5ezu/m3bUonzQtwsUh8UbEAMwb3ocLf8aVr85HbQzXgBWMHuIJT36Lh2dEptk9O3stnvvKeSVDbV8Xr+PQdoxB1QKIbxlpa8s4mb27aVe0ZfLO/C1R26RMU22O+usbA/jdvyvxmxfmGn5e5w/LH88lZEYqFjTxeRV3nMVO7ZvVO7FBc23lbdL/Rn17P9QQwONfrMEt7yzGjGVVoe/q9jnnme/w1JdrIwK65z3zPS6IkRatfz613zVCe+nNnsl563ej4r4ZqDFZUtLpTOBAUGT8mtGSnFMAf35roWHevD/Owzpr5Q4AxiMffTkAM6597SdMX7odtZrGsWlXLSa9swiNgSAe+GQ5FsYYLUtiuYC0nc4Dn6zA3z5eHrMxhmMAod9w7tPfoc/tH4e2GXTg8UZWQgjUK52ilauSaCbHCZNnRY2g//zWwqj9nJb0lR38ak3+uZZv1oRdLXqrxKizNfq1Pk/yH0NpAdT7jRXAw5+txOMaC/KiF+binKfDz4lqAei+V98Yebwz//WN+vqmt0P3QV8Ir0H5zt5D1t1fegugtiHalaVVEtoBi9mjPfnTFdh5oAHLf46u07S6aj++WBF65u1qgD+9sQD9/zpdkTuIif+uxOfLq9TPF27ei/d/cp4S6yapWhIy44lnAZCaBx39WYQFoPtMCBHVkR7U+GFvensh5q7fjfGDOuLZ2evw/Nfrsfb+8TFliTWy9AeDKPB4I7bJxqiVSYiQ4tL76/XphFHnNjm1VCAvfLMenykjPysawA0zu+ZQI9q1iL2PflayVVeIDF6b7f/X95egbXE+npy1Bm2bF0R8ZtXbcu1rP+HVK4ZFfd+urEbIWyCVjNnIWZY8/v2JPVHg80Z9rgZEdaLU+QMozAvvb6QoP1myXflu5Jf31sa2fAJBgSlfrcMFw7pGVSE9ZPA76hqDaO4N/U7tICkoBLwGjVEqr4K86N978qNfxZQtFh8uDMUmhBD4fHkVPltWhZo6P654pRJXnthLTXOeMKRTwjOsEyXnLAAz4gWB5SDfaASszQJ64os1uOylcEqa0bO7X2Ny5vtCt0CaoVYe9lgpjVZqq7/wzXr0vO1j7Kv1q/56wywgg23xOmy184fFeEhj4r5au+subNh5EL1u+9jSvlau54rt+7H055qoTBQjRW3UfpZtq8FzX683PX4iJZhl05QuoEMG1qC2zZk9B2aToupMLAogvv6PFwBftGUv/j59BUb+fZZqNUiMFJnW0tUrAD07aurUFN08r362ts7NZcEE2LjrINbsiCyU1xAIYqdSRK+kMA8AIua47MqAtYxZASgYdWzakbpsE0aNSe8CmrUy3BEYdeg1h8LHzVdGLPsUc1g7IJi3fjeenb0W+w75sXXvIXXEFKtjrbhvpvmHCq/OCfnaZaliOxj1/1+tqsY8ZT0ELVY6LjOFYieId1CJqazZsR8P6CZeGfHRQuvZI2Menh3x3qhTqzXJrDEKVDcEgoYdX6uiPFMZ9G4WJ0gXkNHIuUbjigkEhOG1N/OHH1LmfZi5L/8zJxzX0X93bZyU1L3Kddpf34gNuyL3jacAtM/dnR9E11/SJibom6BeqVlpiidO/hJjH4m0Gur8QfV6G/UDG3fV4i/vL8azSZj4aJWsdAGt2XEA1fvrcWyvtpa/o/dFr6raj1Me/Qr/PH8IJgzprDYu/X18dc5GLI9R613e+M27wwE1bdCpIC+kAGRj13Yw/6fU8H/2q3XYfbABrYvy8NMdp1iyEmJ1oHI0tae2Aa/GCrwa9HZa6+Ob1TvRu11z00k4lhSAyYizIRA0dEUYcVDxB1/+ciU27Y6frlqdQJVSI2kPGnRGr8/bZDhY+HbNLsMlENsU55ueUz/6dYJXcQHpLYBnZ6/FA5rSG/5g0FBxyW2LtuzDYRqX4qGGAOobg3h29jq88t2GiO/sqfXjL++bZ3b91eSzHzftQbc2RdipuU/f6NJaD/mjlW6dP4jv1u7Ev7/biAfPGaRuf/2HzXjg7EGqu2V11f6IduLXaYCDBvEFq9z23uLw79i4R73eRllLuw7U4z9zNim/J4ArT+wV4U5LBVlpAYx9ZHbMjAIj9J2VzPaZsawKS7buw33TQiPLqpo6jP/n12r2iVkjltTU+XHlq/Nx79Rlmm3hBiY7uX0xAmIyS2RPrR+rq/bjXwZL1Gkth5nLqqL8/lqkMrvwubmqLPpH/uIX5xm6P7R1gS56YS5OeXR21D4SI6VaPmkaVlWFTWUzJVHXEMR3a3aifNI0NQXTDGmpWfWVW6nhY4dDBh3GQ5+utJUJ0hgQuO71nyIGChLtNXKa3ijdHPqRs9711BgQhvdE687SKpG6xoA6oMmLE8y2KvrZT32HM5/4JiI76i1dZpeZBXD5yz9g+tLtUfMdtErt5Ee/Up9nILLdTPlqbUTwG4i2+ldX7TdNuf7f3HCSyGUv/4C560OprUaWl1YpPDZztRo7SCVZYQFs31eHt+dvxhlHdEL3tsXq9i9WVOGHDXvQu6w5fF7CnTFK8TbqGsH9H4dGRUIAT3wRvtlygs0Jk2fhjCM6xZVt9qpqTNctt6g1uVUXkLQA4nROZsEpD5E6Ov/tvytjHkOauFrXgr7zNJtZqY8/1MTwketHPR8vDqVgTl34M244pR8Acx97rb8RLykjyvkb96Ck0IcfNuzBtWN6o6omUiFIBWB1Qp6RoggGBdbvOojS4gJMXfwzLhzWzfBeGJ3BaPU3D5HqmrLCt2t2YtribdiwqxYfXH18xGfauQZ1/iCIQhZjh5aFlo8v3ZT7DvlR3xjAs7PXYeIJPZGv83/7A0FDi8Ns3kVdQ0C9h4X5XuyPM9HMShYZAPy8rw5rq40zrwDjDrW+MaAqGb3y9QeCyPMaKyh/IIhr/vcjTh/UUX3utQRFSK46fxDN8r341ZQ52H2wAVeM7IGi/NhdqFxxz6id669zq2bmbsBkkRUKYOeBevzjs1Xo275FhAK4/OVwR9i5VTPVzQIAHyzYiglDOqvvtaNVbSMQECgvDR9TixVfspH/VtsY8nzhB1MybdE2VJS3jntsLXYyReoMgqZWZxfbydpZsnUfZmqCwjITp0G51jOXVeGeqcaVWGsbAihQAuT1jQFc+Hyo4qpRSWHZ0VpVAEYj3MdmrsLjX6xB/w4tsGL7fgzu3AoDOpVE7Wd0lX7YEJ05Vdo839ayoNInXaB0Uo2BID5Y8DN+eWRnnP54OLXynKe/Q7c2RZi+dDtW3XeaYUeoRVuOAwi5Hf793UY8MmMV8rweNQlB0hgUhu4Ks/Z1yB/AAaU9N7PgvoinILSK5s3K6PkcEiMLYF31QfV5+2Rx5KDL3ygAEy9bnT+AqYu2xZwjMn3Jdlz13x/xxsTh6u+NdUyJHCDtN5hroO8b0rEEZVYogBaFoZ+xv67RtKF2alUY8UBe9/qCSAWg6djalxSoo8yPF2/HsB5tHMvmN1AA2hiATM+To7xAUODq//2Irm2aOT5nPIz6+ie+WIOBBh2eHnmZTh/cMW4phVVVBwytETnykXMrjPhwwc/qAxkvCCotAKsVOYw6uJnLQ7LIevN1jQHDXHOrNDQGLWUQSWTblDGh/83bhDs+WBrls1+2rUZ1oV31n/n4fMUOlLUoiOsmk9k9WldKYyAYrQACQTQ0Rl9Is+fqnqnLsFFxhxb44nuUq/aF5mwM6FhiuE6y1YD3IzOiBwI3v7NIfa2fFKn382vRJmWYIa/bnHW71cFSKPvM2qjdigXgRqzHLkmPARDROCJaSURriGhSMs4hU6xq6vymftcNBjNGtbVMtBZA84JIvThv/W50ad3MUgepx2i0qXUBSS+DPiVs8+7ULB2oResXNWLqop9VF9CJfcrw6K+OcHQeeU1iTQL7p8bHGq9TeHjGKlTV1MWMo2h598foSTjyu7IT27SrFpPeXRy1nxUdk+clrLNRdE2LvDbS/71ie3Qn2Sw/NNL+XJmodLC+ERcM62Z4vIZAELe/tzg8qUkrp88T5RZZvn0/npwVHWN69yfj0fhGzXMVr87QwYZG1YXZu11zY3kT6ARjGbH+QBBVNXURgWWJlXYj9wkKoSqAxVv34ekv10IIgdYxsrgAY8tH/1vtpjO7QVItACLyAngSwMkAtgD4gYg+FEIktgKLDq0FYBZUjDdCenv+FozqVwYiMhwhlhTmOapqaGTWaUcDshGkohxAPMx8pJI3K7dgcOdWAEI+5ZYOfZaqArA6GUun1L0eihqRHnP/545kkcgUWymT0WxiqxT6vPAHnFkPsj3ItE2ZJaKlWZ43og1p3WVG/Hdu9DGAUPxJbwFca7Iy25Kt5plukm37Ytc00so8uEtLw6BnsjrBl7/dgGdNyqJYUQCyXWzde0jNBLzilZB1e+rA9o4SC/S/NRstgGEA1ggh1gkhGgC8DmCC2yfxeT0oyvei5pDfcVXPaYu34ft1oYi90XT5ls3yHNWun6ybBg+EAsOS+gxSAPpcaz1frarG3R+FAum+BBSAbOhW4wlz1kUWCZtgIfhuhdMHd1Rfy1ROM6VUlO/Fpl21+DlOJweEJ101L/BZ8otrkYMPXwxlbCRhUb799MF8n0dNQkg1o/u3M9xutxN86bKjLe03U1OKQY8lBaDcl7cN6kxVbtzjqPPWW7ZuzPewS7LvfmcAmzXvtyjbVIhoIhFVElFldbXzmt4lhXnYvKcW6x2a3gDU79Y3BjF+UIfI4zfzuVuGWcFOqmC7FsalAtzC6Oe10LnDpNshZAHEjoA9ccGRhtsbbFoA0j8PAOMGdkDzQncMVyNFYuTrPr53W3iIcMLkWZaOK0eD+T4PZt04Kq57QEtDYxA1df6Y7gyjsiXNC32Yd9sYy+eRcsaz+pJFz7LmqpWjxW4naCXuAAD7DPz8E4Z0Uj6LrwBiJVnU+wO2Kvx6PYR8nycqkJ2NCiAuQogpQogKIURFWVmZ4+O0KPTh06VV6uQpJ9z+3hIs31aD+sYAurQuQv8O4QIzJYV5SakLHmsqvR47tdwl+mnudinI8+LIbq2itnuJUBZHIcXz89pdaxgARvQpjYrROMWq2V7ettjWtZcpl/leDzq0LMRxvUstfa9zq2ZYVXUAg+/6TM0fN8IodtKiwIdim9fFHwiqVtwfRvWy9V0tZx/ZWX1WOpTET019/uIKAMDnfz4RvcoiM+zkbOI7fjHA0rljTZxqoRko7DsUbWG3LQ61X6NRvZ5Ykxr9AePMKTN8HkKBzxO1nkE2KoCtALpq3ndRtrlOkUudwvyNe1DfGESBzxORVtiyWV5S1gWKZZrqiVcCF0CE0gIiH5CeZcbprLHYW9uAly6NNrO9HkJJnJG42eiy3qYLSDJhSCf8+phurlkAg7q0tLRfi0J7ri7ZbFT/usWGM7xneOZ6LJ97oYG7p0Vhnm03kD8QRFAI9GnXHGMOM3bJWKH6QL3q9urWpgjd2xbF3F9mOnVvW4zzKrpGfHbT26FMHm1pjFhutEKT2eLtSwrw19PDSsTINdy8wPr1iuVaDpXDMP+ufqAkRMhy0Qek0xEETrYC+AFAHyLqQUT5AM4H8GEyTlSkayTnDO0StU+8DgsAduyvV2+Q1kQtScMkDSfo5dSOcfc6WH2qMSjQqija1eP1UNxJa2b+5b2H/LjprYURrh0r9CgtBhEZWgAPnTPY1rEAoH1JoaXv2fWSeDUuICC6mNjN4/phTP92WHHvOHx980mG5zHKVpEcMEgpbF7gs1xZ8vDOoWy2+sYg/AGBPK/HdknqozXzVK45qbf6/dIWcRLjgYgSH2apu1oF8MDZgwz3KfBFB7ElPo8nbtKGHYspVokRoziflorukXN6zMptZF0QWAjRCOAaAJ8CWA7gTSGE+XTcBGimG/2cdniHqH2sdOJzlYBjgc8bUeRN21guGNbVll9Xy1UJmNpW0D9Q2mZ20IIbo3Mra/MPjPy3euQkNz0LN++NmtpvBWlRGM2+HO1wBBsvrjP9+pER5b6tIDtiqQD1pzisYwleuPRoFOZ5I0aHhyy6Aw/5A1GuPStWkZxbclyvkEvKHwiiMRhEnpfUEbxVHjr3CLwxcTjeueo4HNOzrSpPvJmxQKTf3swNp00wMPO/f3/rGFMXZ543/gBF+0xXdG+NXx9jnEqbCGP6t1OfKSmOEMZxh2x0AUEI8bEQoq8QopcQ4m/JOo/eTDRq0GZZK/3ah90mc9fvVr9/y7j+6nZtGmm3NsV49w+R0/WN+PPJfaO2nRkjg6XYwISPVSXSiJF9IuMo2s7n/avjy3xSf2txGKkcv775JHxzy0mG+7gdYJRKx8gloL23zyk+ZivEu779O5SohdT0AXEz5GBadk7yHnRpHeoI2rcI+8m1HaBRTSEz9C6JeHGRtsX56lwBQuha+gNB+ANB+LzR8wGMuPS4cgDAtaN7o0dpMY7p2RZHKaNbaQEU53sj2twvh0S3d+kCAswVQIHPiwEdS/D3cwZFDe4kbYrzo2Z/j1DiLWce0SluMUJtjKBZvhe/P8H9wdkLlx6tWiklGlei0bgj6yyAVKIPBhm5H0p0vtwLj+mGO88YgP/+7piofXcfbMDwnm3x5Y2jAIQsCm1T61FajCV3nxpTpjwD81Q7+tEH3q4e3Tvi/RlHdMKU31jvzHqWFuOqEyOPKTvGa07qjcM6lkS4HIwo1SxK8trvhmP2TaMM95Oj4q5titCldRGW3zMuah+nCkB2lHpkamSz/Ojjai2Skwe0N81A+vT6EyLenzygA2bfNAoXakZ/suOWaaLyZ7S0qIzltZFKUrqAbhnXH+/94biIEhPaDsxKjMcMM0U29rD2AEJWh+xsgyLk9nly1lp8u2ZXyAKwYNHJOj1GC6jIAZc+FteiMC9KYVtxARX4PPj4upH41dHdMG5gB9x1hnFQWAZyJWMPa4cFd5yM68f2jZo/okdeG3m+wrzEusPnL64wLEciFYBWSbuRAeUGWaMA9J2CUedb0iyycZYU5uGy43ug2MBs3a7ke5eXFmPDg6fjiK6t1M/koKV5gQ8r7o3u+FQZNB3g7eMPw32/PDzCZ/knjYWw9O5To5RWv/bNY5YJ1tO5dbOotQmG92yLp389FNeO6QMg1GGv+dtpUd+Vv0/bSI/t1TaitpIWfUM3GqU5zTG/9bTDMPncsG9eWkaycJlR5gcRYeofR6gK7owjOuE65Tdr0QdKvR5C97bFuPqksPJd/bfx2PDg6XjywqEAwh251XkPQ5VRsexw5Wgvz0s4slukP1h7GWXZB+38BKvIdvLMRUPVLBt5TiDUQUtf9LG92ka0wzyLFoD0W8dKvdTH4grzon3xES4gEw2gjTt5PIRLj++hvh/ZpxT/uSI0aGuW78WGB09HtzZF6r6tivLh8VDMDvW0wztEtKMCnzdKsf3vt8dg2rUjor5b2rwA14+NbltjB7THOMX1fPO4fmpblNdaq2AmntAz4rtEwFlHujO/xQ7ZowD0LiCDhqWvLy/bvFGDvtLAVy+tNu2RY6WhaYPO5w/riouGd4+QQfvQFRf4oh7ChsagrVHJ6qro6okNgSBOG9Qx4oH3eT2Yfv3IiADoMCWoZ72kcvzPY6WgxnKn5HkpInVSjvx9MWIAAHB455bo2iacgfKnk/ti+vUjI/YpMLmeeTF+kBzRx3MXPXj2IKy9fzwGdgplF6mryKl7GFQXjXABhRTAH0b1wop7x+HUge2j9jdDtv9xh3fE2AHh76nXzuNBRXkbLL7rFIzu3z6ireV5jQOmeiUv4yVGbV66L4oKfPidpnMr8EUvxmglBtA2xsDnuYsrMKJPZGqttLK0x4ulAIJCRLThW8b1j+oHjutdqt5LLR4CLj623Pi4yvPTrU2R2hblQEh73W46tZ8aczj3qC5Yee9pGN3f+v12i+xRALpOwWhEo29r0rdrNArpVWacw250nKl/jB4lAJEjRtkwzTogIDpuUR8ImmZnGHWu22siZ6nmez2q31ZP/w4l6KtJGZUdRWNQ4NLjyiNG4EbE0hNnH9kZH1w9wrQ650PnDMbjFxq7aIDQtdJ+Ux4nVgzAjFLdOrsFPi/uOmMA3rnquIjtsUbA3hgWwF91+epeD6n3xu56rzJ9slVRPgrzvBHKOJ6LxuxcUrFJmWRKqzbLyOw+6bfL+QdGbU/624vyvfjN8O6q8irweaJk0460zRSAmWUgj6lHWllamWOlVQZF+Jp1bFmIbm2LLE8q095jPUMVC697m7DlLI+rjTkQETopweHS5gWm2UzJJnsUgD4GYHBB9bdMO+rr0rqZocvACH1JiMM7G+eTGymAWG4R/YIaI3uXqW6rE/tGBmetuIZW/e00NUhnhDa75YKju6FDSSEmDOmEu84cGJWfrSeWpXDb6YdhUJeWpp3S/x3dFb1KzRVsKMU0/F47sxaIvNfjB3XAVzeZxzVa61JYC3weXHp8j6jrEisLRnYqelfhyD6luOTY7jipX+jeyCsilbY8ouyc4umDyecdgZcvO1rNGtFWqXRaKlgqtlilsv2BILq0LooKJOutInnPjTrtBo0C0O5bmOdVf/dIZdReGGEBRMsTz7qMpVi13zUq6SKR8Yz//vYYdf0FIsKcW81nU8uJjbFmUF8xogdm3nBixBwTqcw6toyMbY0fFHL1xUoMSTZZowD0vl3tiOmbW07Cp9efENVwvF7tPqNVn7y+DIQeqwM7bdqp/E7MUYbmuNOvH4kRfUpRlB+KM9yhC4J5iPDdpNGYpQSpnaDtFLq1LcKc28agS+vYk3gkscrrWhnNtC42d6cQRSpZ+azJjrVQE+/pXdYc3WJMPNJ3fGbX34oFoO345v9lLF694hj4vB60181+lcokvH+os4nXbEoK8zCqXzidVZsqGCvL55qTept+JmXR/74nLjhSTQuVo/dzhkZUaYm6drISrJGyrFXWZJCjXHm+gjyP+rsfP/9IrLxvXESdIyNlksicGyMX0N1nDsSCO06O2E/q0+N7l6Kd5v6Zjez7tW+Bj64JWfrXjult2l48HoqaAS/bsr7t9VDii0brTqSKrFEAx+nW/9XeoC6ti9BPN0MWMDarF991Cv55vrl7wowbT4lO+dQ+tPJhijV60a6WpO2IC/O8UZaDRzEhe5gsVmOFsEzx933moqMi3scq42Al+Btrdq3eApCWiux4tBbANaPjW23aOICpqySGzLJT0X61bfPoMhjaYC8QTgcNWwD2XELadSOuGtULfzn9sCh347lHdcGNp/YzPUaeGgOIPPcZR3TCtcq1kwHXqAGSiQXgNXBLXj6iB4Z2a6XOMWimBu7D+xJFx+GMLolZsP3co7rEtQ60MsvEhoGdSiKCyl3bNDMM4gLGySMr7h2Hj/44Qg04/+robpYXHwLCv1HGUOKVUEklWbEgDAD0ad8C/758mLpAuWEMQPfeqCFbmfZv9CBfM7oPPB7CQ9PDswK1aafakclNp/ZTp/2P7FOqNnjZUfxfRZeoEV/HloU4dWB7nNSvnWGdeifIS2SlJs64wzvgzd8fq9ZaiuWSMFMAlx1fjhsM5kYAodzyuet3Y+763SCdk82j82NLBdCnXXNL1kb/DvFHWLEeaHl5iEKZIa117jd1go8y0pdzRjopJr9R8oAV7vjFAFz13x8BhEaPvx3ZM6J44IYHT497DNnxG1UXlRVozYoMyufjzjMGYFS/dnhoemilPKOJcRcN746LhndX38uApz8QxJBurfHVqmpDGeoMArX6dG3JP847Av84z3gNCvnsaNvyOUM7Y3jPNlFW7dc3jzY8BmDcdhNdqF1NBhDAwjtPsZRymyqyxgIAgBP6lqn+U8M6M7rr7vRGmH1LHxsoLtAGu8Lbrz6pt+qDfvWKY/AvJd1QqPtGn8Hn9eDZ31SoS0XanLlviFfnq47HsB5tMOvGUTi2Z1vVn2uEWQCvQ0lhhIL97Yge6usbTumnduYBISKE8qoKIJwN9OKlFYbzNxLFaDZo2DALZScd1jG2QpGBwMuV3yctO7sl408b1BFXqMcIbbMaqOzSuhkO71yijmiNXBvjB3VEh5JCXGKS0SK/071tEXqUFqtK38roVyrpQ/4Anvr1ULz7h+MM3VhGs9NjufTioW17RGTZpSlJRnVUOWAMipB1Y7doXzLJHElc4qXLjsbL321QR19a9B203env8ZAP+OGdS/D4+UdGjHismP9BCx2FVA5OFqDQ4zVwbcSjR2kxXps43Pa57pkwMGrVqjOHdMLz36wPy6M8vMGgiPh9qgtIo/WSkTK3/oHxhtvVEbzpdQp9IDvpivI2WP/AePWeD+rcErNWVkfFCqwQjiLItmHtZn1zS2iU+8hnIYvUqNPu2iYU9zFDdREqUkgXkJWBk1QAdf4gmhf4VKWoR68A2pcUmNb+sYLdsh1R30/C6Fx/DzOJrFMAfdu3wP1nhRrQR9eMQPWB6AU8/nxyX/y4aQ/OcBh9N2tjsu0U+rzoGSON1IygZqRpfm6Ks4d1ZBDcyUI3dvB5yDBvWn9e+fAGgsLQBeS2wo6Sx+zGithBXKOvaY913di+GDugvWm2WGyZbH8lAjkIcTJgUDt65d+dZwyAz0NROfhGnDaoA/41a43pwi8S/Vq5vx3R09QFZIUM8q6oeMI+wowj6xSAllAqVvihk/ehfUkhXrpsmOPjWnUB2UW6CmI14qC6j/lOz1w01NLaBU4sgFjcM2EgZunWnp1z6xhTt4X+vLKjDwgR0YEmOqpLlPgWQOR+erwewuAurRydu0+7UPKCdkb2LeP6Y7DFUtaJKE2pPOQRurctxhSLdZYGdmppKUahtwCc3mr12Ynx8Dx83hG26vbH49bT+uOBT0JxkXeuOtZ0P30QOJPIagWgx61uxGykmGg/JXP9zz0qupS1xEpO+bjDrZUSsJMFZIWLjy2PGul3aGnd7SE7+qCJBRB0mAsPAO9cdZylaqhGqNfcpAWpJn4SHvDzKrqgb4cWGKIpRWKnomwirkKf2j6Sp4D/dHJf7Kn1o8DnwbTF2xJ2bcb6/jkxnisn/P7EXqoCOKp7G9P9RvQuhc9DuMRkUmY6ySkFIEnUF5es56F72+K4oyZZjuCkftGmtb6+SDyM8ttTSbQFEPofmqUZ3i47ogT6/5gT4uLhNIjrBkQU0fnbZaOyzrNZgb1YhONNjk8fl06tmuH5Sypw89sLEzqXbBppWuEyJu1KCrHmfuP4UrrJKQWgrced0HFMtqvFvxI7fExKmxfg+1tHo8wgD/228YfZOlY4yJce9CNqef0CQkR8plbVTJMJHS+NU1/0LZOQKYxmQdhY+FIUIwI0aZwJapt0DWaaKhmoL5OHbMhOn1MRx/+SqrbXsWUzw5xqu0h50/XQ6FNZ5Si9S+tmEb3tDSf3ReuiPBxu0e+dLJLl+ksmN57SD69cPiyimq1V3HYRWiFRdxMrAHvkpAWQ8HFsbs9UpM+9r8Es6VSgH1leelw5RvYpRe92LSJmwR7Tow1+uuOUVIunYnVkny4LJRbFBb6oOlJW8aXBQnTsAjIoBsfEJ6cUgCRhF5BJG0tmsCwZFBf48OoVwzDIQXqiG+gvFxGht5L1kklXMt78DFnV0ag8hBXemDjctYXuE0GvwFLZmcaaBGkHNyyAWTeOSnhxmKZC+ltdCtFP2U/W8ZsS+iUkU0msy6V9kDPlupr5wn83sie6tylSFwOxyzE928bfKQ2EVxBL5TmdfS+8HkDiMiRSX6upkTQ1R0R3EdFWIlqg/GVAGDyxYF14EpaZLzi9wcqmRqyOXftZKoKQVjCT1+shnDaoY5OzAPXo5U/2gEmL04J5etgFZI9kWwCPCiH+keRzWMa1GICZC8idw+cQ5ldM2+mnu1+VI8J+7dMTK0kV+oFLOrKbEr3ViWYR5Ro55QKSJJoFZBoE5rZnC+sWQHoZc1h7TP3jCAxMY932dBAuYpYCCyBBK8OoGmgquOMXA9Sqqk2RZCuAa4joYgCVAP4shNij34GIJgKYCADdukVXYnQTtWkk2KDNLYB0d1VNC6tXKxNcK07q+DR1wjOcU3hOh/danQiW4rZyuaaibVMkoRgAEc0koiUGfxMAPA2gF4AhALYBeNjoGEKIKUKICiFERVlZcgOS4ZocCR7HpOuKXgSciUWshz2TLIBcJdyeU9CiXTpFBowVmhQJWQBCiLFW9iOi5wBMTeRcGYVpGmhqxWjqxLpcmRQDyFXULCD36qeZ4nTRHD0cBLZHMrOAtBXJzgKwJFnnsorHpSwd84lg3PjsEMtfG2EBsAZICfrV8FJZxdKtekusAOyRzBjAQ0Q0BCHlvgHA75N4LkvIkrodDBaLsYNph8RtzxZWFr5hUsc1o3ujpJkP93+8QtmS/NpWepyXgw795/7fHklTAEKI3yTr2E657Lhy9G3fHCN6x1/QwgnpanuTzx3suNRxpsLPceopzPNi4gm9VAWgXcs22SR+ivjrZDDR5FQaqMdDrsx8zbSqkOdVdE3tCV3Cahookx7C1XNT16ATdaOyC8geuVHwwmXMawGF/nMWkDViZwHxg5xuUlHe3G3YArAHKwAHxFMAjDX4cmU24VpAqSsFkej3eSawPVgBOMB8aUBufHZghZnhuDRvxtYpE80C4kZlC1YADohrAXAxOEuwwsxsws05FaUg3Pk+GwD2yKkgcLIx8lsP6FiCnQfq0yBN5sODtcxGzgvIz8SFdnVIJcUuIHuwAnCA6dKABts+vm5kcoVpwvCjmtncNr4/OrcqxCkDna1zYAe3rAxuU/ZgBeAArgbqDpzpk9m0KMzDNaP7pPSc3CZSS+bbdhkIVwN1B37WGYlbUQZWIPZgBeAiXA3UHvyoMnqctgl+5pzBLiAHmC8JmWJBmjg8WstM3v3DcaitD6T2pAn24Pf98nDcN3U5Sgq5S7MDXy0HmPdb3KHZga9WZjK0W+u0ndvpmOAXgzvhF4M7uStMDsAuIAdwENgd+HoxkpQsOsNEwQrAAWYdV7qKwTVVOGjO6OE2kVpYATjC+jwAJgZ8wRgmrbACcAAXg3MHvl6MhK3m9MAKwAEcA3AHvlyMHn6GUgsrAAeYl4Lg1msHrt3OSNgCSA+sAFwkvCAMt2YrcP/PSIryvQCAvCZQeC6bSOhqE9F5RLSUiIJEVKH77FYiWkNEK4no1MTEzCzMXUDco9mBLSZGcucZA3H92D4Y3b9dukXJKRKdCLYEwNkAntVuJKIBAM4HMBBAJwAziaivECLF0wuTg3ktIMYOrC8ZScuiPFw/tm+6xcg5ErIAhBDLhRArDT6aAOB1IUS9EGI9gDUAhiVyrkyCs4AYhskGkuVw6wxgs+b9FmVbFEQ0kYgqiaiyuro6SeK4i5nrgieC2YMVJsOkl7guICKaCcBoRYjbhRAfJCqAEGIKgCkAUFFR0TS6TnYBuQLHABgmvcRVAEKIsQ6OuxVAV837Lsq2Jo0c2XMtOHdgC4Bh0kuyXEAfAjifiAqIqAeAPgDmJelcKYfnAbgDXy2GSS+JpoGeRURbABwLYBoRfQoAQoilAN4EsAzAdABXZ0sGEMAzgd2C02YZJr0klAYqhHgPwHsmn/0NwN8SOX6mEa+/4iCwPTzc/zNMWuFpdw7gNFB3YAuAYdILKwAHmC4JmWI5GIZhEoEVgAPYAmAYJhtgBWCDeGmg0qXBIQCGYZoCrACcwBPBGIbJAlgBOMA0BsA+IIZhmhCsABzA1UAZhskGWAE4gCeCMQyTDbACsIH1iWAcBmYYJvNhBWADNQuIh/oMw2QBrAAcwPMAGIbJBlgBOMA0BsBhYIZhmhCsABwQzwLgEADDME0BVgCOMJsHkGIxGIZhEoAVgAO4o2cYJhtgBWADoVT54f6fYZhsgBWAAzgNlGGYbIAVQBIQXA+UYZgmACsAG8g0T04DZRgmG0h0UfjziGgpEQWJqEKzvZyIDhHRAuXvmcRFzRzYA8QwTDaQ0KLwAJYAOBvAswafrRVCDEnw+BkJj/QZhskGElIAQojlQO4FReP9XJ4IxjBMUyCZMYAeRPQTEc0mopFmOxHRRCKqJKLK6urqJIqTOPGCuzmmBxmGaeLEtQCIaCaADgYf3S6E+MDka9sAdBNC7CKiowC8T0QDhRA1+h2FEFMATAGAioqKJjF25o6eYZhsIK4CEEKMtXtQIUQ9gHrl9XwiWgugL4BK2xJmIBwDYBgmG0iKC4iIyojIq7zuCaAPgHXJOFc6YAvAXU7oW5ZuERgmJ0koCExEZwF4AkAZgGlEtEAIcSqAEwDcQ0R+AEEAVwohdicsbYYQNwicGjGygrX3j2d7imHSRKJZQO8BeM9g+zsA3knk2JlIPNcPd2T28Xr4qjFMuuCZwDbgEg8Mw2QTrAAcwEFghmGyAVYASUDwTDCGYZoAiZaCYDRwdpC7TD53MDvdGCaJsAJwkZJmeQCAY3q2TbMk2cF5FV3TLQLDZDWsAFykXYtCzLzhRHRrU5RuURiGYeLCCsAGVlz7vds1T74gDMMwLsBBYIZhmByFFYANOMjLMEw2wQqAYRgmR2EFwDAMk6OwAmAYhslRWAHYgCf4MgyTTbACcAAHgxmGyQZYATAMw+QorAAYhmFyFFYADuBYAMMw2QArABuw759hmGwiIQVARJOJaAURLSKi94ioleazW4loDRGtJKJTE5aUYRiGcZVELYAZAA4XQgwGsArArQBARAMAnA9gIIBxAJ4iIm+C50o77PphGCabSEgBCCE+E0I0Km/nAOiivJ4A4HUhRL0QYj2ANQCGJXKuTIJdQQzDZANuxgAuB/CJ8rozgM2az7Yo2xiGYZgMIe56AEQ0E0AHg49uF0J8oOxzO4BGAP+1KwARTQQwEQC6detm9+sMwzCMQ+IqACHE2FifE9GlAH4BYIwIr4a+FYB2Pb8uyjaj408BMAUAKioq2MvOMAyTIhLNAhoH4GYAZwohajUffQjgfCIqIKIeAPoAmJfIuTKBZvmhODaHABiGyQYSXRLyXwAKAMygUGR0jhDiSiHEUiJ6E8AyhFxDVwshAgmeK+08fdFReKtyMy/7yDBMVkAig3IbKyoqRGVlZbrFYBiGaVIQ0XwhRIXd7/FMYIZhmByFFQDDMEyOwgqAYRgmR2EFwDAMk6OwAmAYhslRWAEwDMPkKKwAGIZhchRWAAzDMDlKRk0EI6JqABsTOEQpgJ0uieM2mSwbwPIlQibLBrB8iZDJsgFh+boLIcrsfjmjFECiEFGlk9lwqSCTZQNYvkTIZNkAli8RMlk2IHH52AXEMAyTo7ACYBiGyVGyTQFMSbcAMchk2QCWLxEyWTaA5UuETJYNSFC+rIoBMAzDMNbJNguAYRiGsQgrAIZhmBwlKxQAEY0jopVEtIaIJqVJhheJaAcRLdFsa0NEM4hotfK/tbKdiOhxRd5FRDQ0ybJ1JaJZRLSMiJYS0XUZJl8hEc0jooWKfHcr23sQ0VxFjjeIKF/ZXqC8X6N8Xp5M+ZRzeonoJyKamoGybSCixUS0gIgqlW0ZcW+Vc7YioreJaAURLSeiYzNFPiLqp1w3+VdDRNdnkHx/Up6JJUT0mvKsuNf2hBBN+g+AF8BaAD0B5ANYCGBAGuQ4AcBQAEs02x4CMEl5PQnA35XX4wF8gtDywsMBzE2ybB0BDFVetwCwCsCADJKPADRXXucBmKuc900A5yvbnwFwlfL6DwCeUV6fD+CNFNzfGwD8D8BU5X0mybYBQKluW0bcW+WcrwD4rfI6H0CrTJJPI6cXwHYA3TNBPgCdAawH0EzT5i51s+2l5MIm+aYdC+BTzftbAdyaJlnKEakAVgLoqLzuCGCl8vpZABcY7ZciOT8AcHImygegCMCPAI5BaIajT3+fAXwK4FjltU/Zj5IoUxcAnwMYDWCq8vBnhGzKeTYgWgFkxL0F0FLpxCgT5dPJdAqAbzNFPoQUwGYAbZS2NBXAqW62vWxwAcmLJNmibMsE2gshtimvtwNor7xOm8yKWXgkQqPsjJFPcbEsALADwAyErLq9QohGAxlU+ZTP9wFom0TxHgNwM4Cg8r5tBskGAALAZ0Q0n4gmKtsy5d72AFAN4CXFhfY8ERVnkHxazgfwmvI67fIJIbYC+AeATQC2IdSW5sPFtpcNCqBJIEJqOa05t0TUHMA7AK4XQtRoP0u3fEKIgBBiCEKj7WEA+qdLFi1E9AsAO4QQ89MtSwxGCCGGAjgNwNVEdIL2wzTfWx9CrtGnhRBHAjiIkEtFJd1tDwAUP/qZAN7Sf5Yu+ZS4wwSElGgnAMUAxrl5jmxQAFsBdNW876JsywSqiKgjACj/dyjbUy4zEeUh1Pn/VwjxbqbJJxFC7AUwCyHTthUR+QxkUOVTPm8JYFeSRDoewJlEtAHA6wi5gf6ZIbIBUEeKEELsAPAeQgo0U+7tFgBbhBBzlfdvI6QQMkU+yWkAfhRCVCnvM0G+sQDWCyGqhRB+AO8i1B5da3vZoAB+ANBHiYznI2TGfZhmmSQfArhEeX0JQr53uf1iJaNgOIB9GnPTdYiIALwAYLkQ4pEMlK+MiFopr5shFJ9YjpAiONdEPin3uQC+UEZpriOEuFUI0UUIUY5Q2/pCCPHrTJANAIiomIhayNcI+bGXIEPurRBiO4DNRNRP2TQGwLJMkU/DBQi7f6Qc6ZZvE4DhRFSkPMPy2rnX9lIRXEn2H0KR+VUI+Y1vT5MMryHkp/MjNOq5AiH/2+cAVgOYCaCNsi8BeFKRdzGAiiTLNgIhE3YRgAXK3/gMkm8wgJ8U+ZYAuEPZ3hPAPABrEDLNC5Tthcr7NcrnPVN0j0chnAWUEbIpcixU/pbK9p8p91Y55xAAlcr9fR9A6wyTrxihkXJLzbaMkA/A3QBWKM/FqwAK3Gx7XAqCYRgmR8kGFxDDMAzjAFYADMMwOQorAIZhmByFFQDDMEyOwgqAYRgmR2EFwDAMk6OwAmAYhslR/h+MN8bO5nTI3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "normed =model.base_model.ln_f(hidden_states['h.11'])\n",
    "plt.plot(normed[0][0].cpu().detach().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b2eca726",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228a380bee0>]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAt70lEQVR4nO3dd3wUZf4H8M83BUKvAUINvUuLEaQIIt2Kcoo/sZ6c3nnn2QE9O4Lt7Cfgnb03FEWkN6kGpPcSmkDoIAFCkuf3x84mW2Z3Z3dndjaTz/v1yiu7M7Mzz2z5zjNPFaUUiIjImRLsTgAREVmHQZ6IyMEY5ImIHIxBnojIwRjkiYgcLMnuBHiqWbOmSk9PtzsZREQlyooVKw4rpVL11sVVkE9PT0dWVpbdySAiKlFEZFegdSyuISJyMAZ5IiIHY5AnInIwBnkiIgdjkCcicjAGeSIiB2OQJyJyMEcF+f0nzmDOpoN2J4OIKG44Kshf8cYi3P4+O1MREbk5Ksgf/uOc3UkgIoorjgryRETkjUGeiMjBGOSJiByMQZ6IyMEY5ImIHIxBnojIwSwP8iIyUEQ2i8g2ERll9fGIiKiYpUFeRBIBvAVgEIA2AIaLSBsrj0lERMWszslnAtimlNqhlMoD8DmAqyw+JhERaawO8vUA7PF4vldbVkRERopIlohkHTp0yOLkEBGVLrZXvCqlJimlMpRSGampupONG92PiakiInIGq4P8PgANPJ7X15aZLr+QQZ6IyJfVQf5XAM1FpLGIlAFwA4ApFh+TiIg0lgZ5pVQ+gHsATAewEcCXSqn1Vh4TAC54cjreWbDD6sMQEcU9y8vklVI/KaVaKKWaKqXGWn08ADh5Nh9jf9oYi0MREcU12ytezVBQqPD8tE12J4OIKO44Isj/su0w/vvLTr/l5/ILTDvGjkN/IOfUWdP2R0QUC44I8vkFhbrLn/lxg2nHuPTl+cgcO9u0/RERxYIjgnygJvIfL90d24QQEcUZZwR5uxNARBSnnBHkg/R2jUU5+ru/7MSWg6csPw4RUbicEeSDrNMrR+8+fg56vjDHtOM//eMGXP76L6btj4jILEl2J8AMh/84F3KbnJNncct7v2JE10bYd/xMyO1PnDmPo6fz0LhmBUNpyAtQ+UtEZCdHBPlz50MH2D9NXILsI7kYM3mtoX1e8cYv2H00F/3b1MZve45HmUIiIns4orjGiF1Hc/2WjZm8Fu/qtK8HgN3a9jM2HMShU6HvFIiI4pEjgryR1jV6dbOfLtuNp01sS09EsXW+oBCZY2dh6pr9diclbjkjyJfgseQPnDiL8yzPJ4rIsdw85Jw6hyemWD7uYYnliCAfrfRRU/HB4mycyD2Ps+cLkD5qquHXrv/9hN+yFbuO4peth0O+9vtV+9B13Gxc8QZb5hCRNRxR8VpoQk7+iSnr8cSU9binTzND22cfPo3eL83zWnYmrwAbD5zEtW8vcW0zfkjQfdz7+SoAwKYD9rSxV0rhk2W7cWXHuqickmxLGojIWg4J8sHXvzlnq+F9/XEu39B2/9OpsG39+M+GjxMPVu4+hse+W4clO47grRs7252cEi3n1FlUTklGSnKi3UkpXUpuSW3MOKK4JlRG/qUZW2KTkCjlnDqL+VtiN5n5Wa3p6bHTeTE7plNljp2N29//1e5klFoidqcgfjkjyJt4OQ/1ZTmXX4A9R3NNn1P2wImzGDZhCW55d7mp+w3G8+KolDLUSSxaC7YcwverLJnmF8dO5+Hk2fOW7NuIxduPGNqu/RPT8cT36yxODZGLI4J87UopMTvWBU/OQM8X5uKz5aFHuCwsVLj05Xn4esVev3W+wyN3HTcbu474t+U3w5aDpwIOxwy4LmwfL9uN7uPn4Mkp6y1trXTzu8tx7+erLAnGnZ6ZiU5PzzR9v2Y7dS4fHyzZZXcyHIGlNaE5IsinVYldkD+Xb7y5Y+75Auw4dBoPfrUaALD14Cms3H0MAII2+Tp06hyyso8aOsbvx8/gq6w9yM3Lx7hpG3H2vPdEKf/7ZSf6v7IA46dtwuhv12JbTnElr+cd0K87Xcd7f3E2flp7wO84u4/k4rmfNkZ1Afjn578VPb7gyRkR7yeYApPvsKhkYGlNYI4I8mZ6b1G2Zfvu98oCDP3PYgDAJ8sC3wlc/dYiXDdhie66L37djfRRU4uC+Q2TluKhr9fglZlbMHH+Dny81DuH6J445b+/7MRny3fj7o9X+u1TfH4iernsuz5egUkLdmDLwT+CnGFw3636PeLXxovPlu/G/hPWF2sF8s6CHWE18SVyRJB3Wt4tUNn4+YJCPPKNa+ydo1plqXvIhaJK1Nw8XDh2lqE7Ac9MuWddRKFSyMo+iqvfWlQ0hWJ+Yew6bJ04cx4ncu0rWw/k2Ok8jP52LUb8L3b1Jr7snKD+9Ll8bLapuW8gJbgfZMw4IsiXBH/5KKvo8VdZewy9ZvF27w5Vb8zZ5vV8W84pnNFy9O6il7fmbsehU+dw/aSlhnN8vpXNK7KPYczktVi15zg+0+44Yvlj6vDUDHR42prinGi43wIjo576uuTFubjqrUXmJijGRn6UhQGvLghavxNr7u89W9cExiBvoZ2HThc9nr7+YNHjh75eY+j1N76zDICr5c34aZvwu08Ov98rC4oe+wbhcMumPX8jBR47e/IH77F97PoxfbtyL2ZuOBh6QwAjP8wKvVEEErWTj6Tcf9eRXKyOYDTTvPzCmLR6MmLZDtfd4d5j8ZEeT75FjlTMEUE+Xj/eK96MfriC9FFT0XXcbEyYvx0rdx0rWr77aK5XYA9Wxu9J6TzeozNCp2/Zu3vbV2Z69zkoKFQ4FaClzPHcPIz8MAvHc423w1+3r3iYiG05p4pyzfd/uRp3GgzeMzYcxG+7j4Xe0ICPl+7CzsOui3WC9mspjGHl7qhv1qD7+DnIzTPWSS8WfHt624nFNaE5IsiXls/Z8zy35kRWAbot5w+/FjLZIZpujpu2sSj3Om2dd8ubR75Zg/ZPzkBBoUKP5+eg/yvzi9a9uygbMzYcxPuLsw1Pw3i5xzg+l/17AXq9MNfQ63xd85/FeH/RTq+7nw8WZ2Pj/pNFz/MLCoMWPSil8Nh363C1VsySoOXkze4jEcyczTkAjM2ZYLV4/p2xuCYwy4K8iDwpIvtEZJX2N9iqY5UWZn2POz0zE+mjpuL12cXDPQRr+TJx/o6i3Kwvdx+AQqWw99gZbDn4B2asP4Apq73396ZPfQIAPPVD6JEDc/MK8NL0zQHXn8krwPhpm/yajgKuoqZ+/56PP01cghNnzuOJKesx6LWFReszn5uNDk/pl/2v2Xsc/bXisBNnvO9UzBgryQ6FhQofL92FL3/dE7Lo6MMl2djw+8mg25hhw+8nMXH+dlP3+cvWw5i21nvoYaUUPl222/YKfaUUflj9e0yb+lqdk39FKdVR+/vJ4mOVKgujGP7guPZFX7FLv0gj52T4FYueQXbkRyvwj89+87qX/lCn849vc9XHA/QCfXOu/wXC7e352zFh/nZ8FKBz0em8AizfeRQ/r/Mfb/zo6TyczitO9/mCQvx34Q4cOnUOT/2wIeDdUkGhglLKq2ipJJi6dj8e+24dHv5mTchK4Me/X4/Bry8Muo0ZBr++EOOmbYr49Xqh8qb/LcPdn3g3FV677wTGTF6Lh75eHfGxzPDNyn34+2e/4b1F+pMVWcERxTWl0QyDlZCRWLIjdPf80+fyvcry2+t0bnpdy71vP6R/F+BL70Kgp6BQYfshVwBerwXa8yGaeL4y03uQujmbit+/jGddvWQ/WJyNZ6duxIVjZ3ldAH2LAgoV8N+FO3H5G7/g+Z83YfS3xRXpe47m4i8fZRVd9DYdOIkmo61v137gxFlkZR/FuGkbkRegw57e4HufL9+NTQdC59gvf2Nh0NynUipkR7mVu48h56SxYrtgck6eRfNHf/K6G3F/RG8FyBC4OzEe1Rmn6eDJsxjxv2UxyeUf0eqYcnxmm9uWc0q3bswMVo9CeY+I3AwgC8ADSim/rKOIjAQwEgAaNmwY0UFW7z0eRRJLjngqJmj7xHTD27p70wZzMIwff9MxrpvCCTd1xuxNrjLrUK0rDvjs//b3iytxD//h+uGfPudf5AO4bkh8y+7d7dXfnudd1DB26kZMX38QczflYFD7NAx81Ts3/GQYk1vsPHy66K7LP00KIoJlO45g9qYcTFqwo2hd/WrlMaJrI0PHGPWtq99FqGGx1+0LfCE4kXseHZ6egceGtMafezYJuN3Q/yxGlXLJWP1Ef0NpC2Th1sM4X6DwweJsPDCgZdHyQ6fO4cUgRXuBvD1vOxZuPYxvf9uL27o39lufX1AIBSA50bo88cgPV6BtvSp4Y3gn0/cdVapFZJaIrNP5uwrA2wCaAugIYD+Al/X2oZSapJTKUEplpKamRpSOs3n6P06nCVVBGivh5nh8A6yvK974BRc9NzvsdKzaU1xcYrRiN5hglXdmTRP5/uLsosfdxs3GufwCfJm1B0q5ysu/+NXVSiovvxB9DLRiuX7SUq8ADwAFBtuxrzSpBZL78/3SQP8Pd/3Gku1HijraRcPz7iFQcZ/X9jrLQrVcuuTFeWj+6LRwk2b4+FaLKsgrpS5TSrXT+fteKXVQKVWglCoE8A6ATHOSTHabuta8+TT7vjwPayMs2/b8cYYzHIVvfwMjjBYlhWP/ibNo+djPePjrNfhu1T489t26oh7NeVF0ODr8Rx6un7jEr9OW743g8ElLQ+5rVYAK2s7PzNQt+th7LDdksc3mA6cw/J2luHjcHN31uXn5GDN5LU6dPY9Crf5j8fbDuvUqbiLi1+rpySnri6bWDHaf92WW/wCCnszsp+Au8oplYyArW9ekeTy9BoB1Y6uy/VRMjZm81rR9GS2vd5urNSkEIg+8A19doLs8IcTXyEhxmTuoBAqOgezz6GCk99oCpXDt24uLnu85GjjwvL84G8t2HsXYqRsNd8BKHzUV437aiIMnz+JZj7sW317XbkdP52HRNu91Ww7+gR7Pz0Xj0T8FzVUf0/pNHAkwj8FHS3bh02W78eqsrWgy5ie8NGMzbnxnGe76eCXmbzmEB74qrjwN9T5MX+8/2J6bbz3C58uN9UQPRSnldwHcuP8kPl22u6g4aW8MO7hZWfH6goisFZE1APoAuM+qAyWH+nWSY9z2XvQTc5w8q397LiEyC3qVy77cdQQTF+zAmjDqijwntrn6rUV+lafHc/O8KoN7vRi4/4C7gnXyb/uKWtHc8f6vIS/OExfswMNfr8F/dWY9C1ewC3CoX6s7Q+6+U3trbnG9h+98C8Pfcd2NnC8o1O0R7Xtd9nwP+748H+086pY2H/Qfl2eLzjK3nJNnddePmbwOnZ+ZiV1HijMwg15b6PX+Hzp1DgdOnI1JJzfLKl6VUiOs2rffsWJ1IHKseZtzvDpKmeHWKC5ID/sMfeEZ6MJ1+lx+0cXHk96w2b4D0b3w82a88LN+ZWakvzu9i+m0tfvRtm4VNKxRvmjZ1hAjnnrW9ZwKcOEuPmbx45yTZ1Grcgp2BOj74Xby7PmivhIAsOvIaeTmFaB1WmUAwEXjZkMp/0pr91wTe4+dQaMaFXT3vXznUXQdNxtNUyvgvVszQ7YOiwabUBLBFZB/XGNeXQOg31zPqAVbvftBTP4t8tm0wmkJtWibsdmtPE1ZrZ+2cOo+7v5kJXq9OBe5eflFg45lBejH4eY5E1c4Jbaf/7on4F3WiP8tK3p8xqdBxyUvzvPqTOe+S/BtTux25HReyMHcth86jV4vzg1a7BQtBnmiOBSorbunUd8YG+jOKu65gQPdZVw8fo5uoA8WkNs8Ph27Doffiiw3jBZ2/565BVe+qd8ZbOHWw1i95zjy8gsNN1ke/s5S9NQZfuMfn/2Gv3260lA/BCs5IsjHUfNxopj5/FdzKgoj9dQP60MOu3zx+Dl+rXjmbfYvOvL0hcGhuI345xertCG3jWf17/tiFVo8Ns1wn4Y1e12tw07rdDabvv6gX1+JWHNGkGepPFHMFSog49lZIbfz7UEdTf1CuCIZI8adg/ccHtyTUgrrf/dv9tv2iem2j42jxxFBnojimxUTt1vFd8gBX1NW/44hr+sPI379JP1pO+3kiCDP4hqi+Hbkj8groc0QzgT0ocr3g02BuCnOpkcEnBLk7U4AEQVlZIgGKz071by5cY+FMQlOPHBEkCciCibcHsjBfGZSz9hYcUaQZ3kNEZEuZwR5IiLS5Yggz3w8EZE+ZwR5RnkiIl2OCPJERKSPQZ6IyMEcEeQ5rAERlXSzdMbDN4MzgjxjPBGVcGfOWzNXtSOCPBER6XNEkGdGnohInyOCPKd4JSLS54gg36NZqt1JICKKS44I8m20iXWJiMibI4J8lfLJdieBiCguOSLIExGRPgZ5IiIHiyrIi8gwEVkvIoUikuGzbrSIbBORzSIyILpkEhFRJJKifP06AEMBTPRcKCJtANwAoC2AugBmiUgLpZQ1XbqIiEhXVDl5pdRGpdRmnVVXAfhcKXVOKbUTwDYAmdEci4iIwmdVmXw9AJ4TIe7VlhERUQyFLK4RkVkA6uiselQp9X20CRCRkQBGAkDDhg2j3R0REXkIGeSVUpdFsN99ABp4PK+vLdPb/yQAkwAgIyODw9AQEZnIquKaKQBuEJGyItIYQHMAyy06FhERBRBtE8prRGQvgG4AporIdABQSq0H8CWADQB+BvA3tqwhIoq9qJpQKqUmA5gcYN1YAGOj2T8REUWHPV6JiByMQZ6IyMEY5ImIHIxBnojIwRjkiYgcjEGeiMjBGOSJiByMQZ6IyMEY5ImIHIxBnojIwRjkiYgcjEGeiMjBGOSJiByMQZ6IyMEcGeTv7dvc7iQQEcUFRwb5+/q1sDsJRERxwZFBnoiIXBjkiYgcjEGeiMjBGOSJiByMQZ6IyMEY5ImIHIxBnojIwRjkiYgcjEGeiMjBogryIjJMRNaLSKGIZHgsTxeRMyKySvubEH1SzVG1fLLdSSAiipmkKF+/DsBQABN11m1XSnWMcv+mq16hDI7nnrc7GUREMRFVkFdKbQQAETEnNUREZCory+Qbi8hvIjJfRHoG2khERopIlohkHTp0yMLkEBGVPiGDvIjMEpF1On9XBXnZfgANlVKdANwP4FMRqay3oVJqklIqQymVkZqaGtlZ6Hjl+g4AgCHt07yWJyewrpmISo+QEU8pdZlSqp3O3/dBXnNOKXVEe7wCwHYAMR3/95pO9ZE9fgga1ijvtfydmzMCvIKIyHksydaKSKqIJGqPmwBoDmCHFccKpXvTml7PfYM+EZGTRduE8hoR2QugG4CpIjJdW9ULwBoRWQXgawB3KaWORpXSCPVoXhObnx1ox6GJiGwXVZBXSk1WStVXSpVVStVWSg3Qln+jlGqrlOqolOqslPrBnORGpmxSIiqnJOGOHo0jev2wLvVNTlH8ykyvbncSiMhE0baTLzHWPDkg4te+OKwDvlqx18TUxC8FZXcSiMhEbGpCXgoZ44kchUGeiMjBGOTjTL2q5aJ6feeGVaN6vVLMyhM5SakO8p/8+SK7k2C6auXLBF1fpZyzB2gbPaiV3UkgiiulOsh3b1Yz9EYOs/zRvnYnwVJNUisie/wQu5NBFDdKdZAPx0d3ZOLvlzaz/DiPX9HG0v2XTUoMur6kF9ZwKGkibwzyBvVsnorbu/u3s39jeCdTjzOgbR2ULxM8EFvJt0i+Q4OqtqTD19NXtbU7CQFVSik1LZGpBGKQj1JyYuC3MCU5sre3bhSVr2bnxOOlDP/mbul2JyGgmhXL2p0EooAY5MPgHjY/MaF4/PxgQXDhw5danSTbdahfxe4keOHMBkTeGOQjULFs8e15pwBNFiuUSURqpchyeKEC1bu3ZuD+fi2w4enIe/GapU6VFLuT4JWT1ruTifRzIHICBvkwuItmWtauZGs6Lm1VG//o2xzly5hfFty3Va2wtr/xokamp8FsDw1oaXcSiGzDIB+GCmWT8OmdFxkakz6aKRFH6bT1TohROcTgC9JCb+TBaLpevO6CCFJTMrADGcWzUhvka1Z0dRrSa8mS9dhl+OD2TN3XXdy0JqpE0EyvZ3PjbfL7tq6N7PFDMLRzPb91E0d0CfpaswOOWdeWYRkNou6NS6XX3b2b2p2EEqtUBvmP7sjEj38POO0salYsi6omtypp5DNZSVqVFDx/bfugr2lSs4Lfssta1zY1XWSdaIeoIJdXru+ARwayJ3OkSmWQ79k8NWSFYdkImz+6hcoBX5heHSnJxtvDh5s/b6xzgYhHreqEX7/hWfEdyMvDOqBNmu60whRCvLWYuqZT6ZnPwQqlMsh7ClS60aqOuQFCTCr4MLqXf13eOqL9x7p4eeo/iu+oJtwUvCjKzch7cG2X+mhTV/sMLT4nx5XIR1GfRPGn1Af5aEX6exBx5eY91ajgPbiYZ9M/o8G3htacUO8uoWMEvVevM3FWrIbV/efXTfSqubW/PiGSO4tgjNx1BNO9WQ2TUmJcvHSAK22sarXHIG8VnwijdzGoW7UcZt3fCwDQNLUCVvyrn9f6YV0a+O82ROR66sq2eO6a9ujWpIZfxe1Lw8Jv4XJFh7phvyaQ54Z610FMClGJHGvf/vVifHZnV9P2JwI82L9F0fMujaoF3LZccqJfQG9bt7JfRsBqVcsnx6wlV6yFGi7EqXUoDPIeRnRthKGd/Fu0mCGS302CwV+b52YVyibhxosaQkTw7NXtipa3SauMZrX8cwoLHupT9Hh4pv9FJRKBcsOe7fqHdq6H/m3rGH5tLHRuWA3VKgQfqjkc1SuUQZ0q+oGjjM9wGEM719MdHmHe5kOmpceIQe3SHNtrOLNx8Atm5yAX4Vj4R9/mluyXQd7DM1e3w7+v72jKvkI1+brigvByyJe11u+k9MsjfZD1WD/ddZ5BNdAdQMMa5TGgravFTtPUirrbRDIA1/IxfYvuUjy5R/LUC2gdGlT1KqO3QtPUyCqkh2c29Fs2+4FLsPDhPkGL0gK973rjGt3U1b9j2aFT5wynMZBwcuYjezUpevzQgJZY82T/qI8PAFd1NO+OMFLdmwZvxtyurn0V9U9f1RZDwuyjYhSDfIR8c2Ke5j7YG3/tHXhY4uzxQ3BZm/CaQr55Y2f88kgfv05W9auVR/Uoc5/1qhaXldet6t/qyN0567ou9fHpncEnWqlXtRyeurItalVO0b1zqKCVUevFnbQq5XzK6M0z54FLsO6pAZj9QG/T9tk0tSIaeNQzfHZnV7x7q3dHufQawS8qV3oUh/kWzdzUtRGu7hRdcEyrkoIZ911iePskj/e/dVolVE4xp3z+tRvMHa01En/u2RhLRweeT+HOnk3Q1qZAH6woL1oM8hbr3TLVlP2kJCeifjX/ikujjDapLF8mKeCkG8FG3HSb9s+euKiJtZWFLxjsPesOwDUqlkGT1IpRVYL6Vhrrld/WrZqCS1t5X7xbhih+alFb/+4JcN09hNsvol097yA1tHM9NKsV+BiAf+Xuo0NaIzO9Orpa/DkGsjbE3cPz17bH4lHFg/8ZvdMUEdSpkoJKOt+Dx4a0RkKCOLLZrWOCfFqEA2VFeoukDLYEuaSFK8hHM8xBNNxBJNidRTiibQpqpJXQBSHaaQ9qb+wz+2vvpvjvzRm4NMzxePR4FmMAwI0X+RffhCPQ98G3Z3SnhtVCznSlN89BIOV0Wl3VruT922lWqxK+vKtbRGMjDTOhNVallOSgv+cG1cujskcLoB4mzPB2y8XpusvrWjgA323d09GguiuzYFYTaz2OCfIVIsyljRsavNdpKHofTvUQ86x6cs/UFE0u3Qi9YhCj0+R5Bmbf3UTc3C7Id3rKPT1wa4AfXTiSEhNwWZvaEV9gHxtS3NcgWDGSO0DUiGJcefdbbCTnfkcP76DeOi3w3YLnZ1cmKQEbnxmITc8M9Prsn/GooI9Wt6bR5f4XPtwn4Lo6lV0BV3x+dYE+3nA+9kCbLg5SvPP+bReGXbzoeRF84oq2qGDBIIO+ogryIvKiiGwSkTUiMllEqnqsGy0i20Rks4hYPiau3hAARhgpggjHhqcHhDW2TYPq5THhps543eQZpqwgAmSkV8dt3dMx78HeeO+2C3FB/apR7/eD2zPxqc+k6uH0BvZKY9SpKfbnnk1CbwRX0M0eP8RQcVDZpARc16V+wAB0Q4AWTjUqlEHvlqm4skPdiFphfPmXbpj3YG8Axe/tV3d1w6QRXSLOIFmhhjamlF4jAN+hQSKldzPpXhbOhaFB9fL4Zxifxaz7e2H8tbEfqC/aCDcTQDul1AUAtgAYDQAi0gbADQDaAhgI4D8iYumcdq+Y1ComEs21Ms9HBrbyu8V1V7y1CNLRYWC7tJh2QKldOfIcZ2KC4Ikr2iK9ZgX0aRl+McjwzAbo2bymV270khapuNjnlvsffZvhXoM/oFphjhc/qJ1/081wRdpJbPOzg/DSsA4B1weag3fFv/rh/dsy8frwTn7flTJJoX/GmY2r+804dmF6dd1mrNEwq1Tyrf/r7LcsWElfqHGgzGBG0U1igmBo53r4Sy9jGQgzRHUJV0rN8Hi6FMB12uOrAHyulDoHYKeIbAOQCWBJNMcLxs7cyNd3X4xDp87pVnD1aVULP9zTA+3qVcaYyWttSJ2/eQ/2QV5BoWn769+mNmZsOGho26rly+CjO4K30AFcFcD39WuB12ZvLVr2zd0XF40e6vbaDR0BAPd+vspwei9pkYpp6w4Y3t7tszu7YmvOqYinIkwMEgEjHU7iqSvbBmz/fUmLVNzew3h5fSDNa1XE1pw/ip5/dVc3DJtg2U8ZQPBiQBHvi4lAYjIX8c/39cLpc/noNm5ORK+vphXj/vtPHU1MVWhmllXcDmCa9rgegD0e6/Zqy/yIyEgRyRKRrEOHYtvxwyxVyiUHbcHQvn4VyyteFzzUBz/+vYffcr3gUa5Moql3DhNu6qJboWe2Lo2qoZFPk8SrOtZDUkJsqpa6Na1hKMC/daMrF+r7nryqXZA86X0vfC9kwdxycXrASrsPbs8Me/5Zve/L5L919+qnEarFUMyJ/1hTTWpWQP8wmilL0X//99Ld56VySjLSqpTzas0TzrU5WJ2NleEh5K9DRGaJyDqdv6s8tnkUQD6AT8JNgFJqklIqQymVkZpqTnPDePWXXk3w/m0XWrLvhjXKo129wK1SLP0SJYhX++rSbnD7OnhoQEu84VPPYnSC9vkP9cHqx413QvJt6RVoSspIVSybhF4tin+bkX7SRooofIs7vTJPYUTUOQ/2RssAgwyGO+fCIwNbeVVUp5awidtDlnEopS4Ltl5EbgVwOYC+qvjd2wfAswapvrasVBs9OLKRIe3mDuDJpTCQ39KtEQ6cPBvWa0QEf+sTbpPV4sATTdFjgggm/7V7xK8PZETXRnj8+/VR7WP04NY4e74AHyzZZWj7Lc8O0u2tG+hb+Pb/dcbdn6wsen5Pn2ZoUK0cHvp6jeE0/l/Xhvgia0/oDcNQtXwybtbpzRwr0bauGQjgYQBXKqVyPVZNAXCDiJQVkcYAmgNYHs2xjHp4YGzm8xzRNR1A8KZ1TjG0c338uUdjPBDhXKmZFgyy9dEdmX69S43Sy8cNDFAB+dRV7TBxRGTHiRXPjKm7qMhsgYobrZy3oExSApICtH7zLFZxP/LtP1EmKQHDMgKPxzRqUCu/i8gF9asWNdU0Q9PUClj1eH/c31//txOLob2jra18E0BZADO1L8FSpdRdSqn1IvIlgA1wFeP8TSlVEOWxQjLa7tsM/7q8NcYMblUqgnyZpAQ8dnmbiF674KE+qFnJvEG/3Ho2N7doz4wOU+Gw4luTViXFa5gFq3gG/O7NamDn4dNe66/sUBenzp7HXJ3B1epVKy6yev7a9ujWxNWqasrqfdh/Ivgdk9EOiEbd1LURXvh5EwDvc1o6pi/SR0019VihWFmcGm3rmoD3pEqpsQDGRrP/eCYiSEqM/wA/YUQXvL8oG80CDD4GuIZeeHfRTkuGtW1oUtvmWIvVGCZOnAT89eGd8Gv2Ud0gf0ePJmhcsyIua13LK7Dec6nx9ubBGjE8f2175OUHbzk2uH0avlqxF8kx+P3a1dPdU/z0grBRn5apul24X7zuAr/WHCVN09SKIXs09mqRim1jBwW8NS4NPHOJ39zdLeCInGYx88dv5mXCyF1XhRDjsgeTmCDoF+bgfHq8mlB6PLn+wtDDTYwb2h6jBrUK2CchlGoVygCHTyMpQUKOUR8PGOQBvHdbpu7yYOV5TlOaA7yvLo1iO1FHNDzvBKK9bJRLTjQ0Xo1nUI3ljciYwa3x8Ndr/AZhC+XazvW9RldNSkwoas7Yv00d/Lz+QFjv3ds3dcaM9QfRqEYF3HJxOgoKFbbl/IGvVuz12zbUftvWrYzNB09FPYNYMAzyce7xy9sgv9C8jkslzYSbupjePDPWJSSJCYKCQusO6n53jPR8DeSHe3pE1RPalxXvcaeG1TDzftewyWfPG6/ie/lPgXsYvza8I46ezjM8QQ8A1KqUUjT2f3JiAv5ySdOAHR1D3bA9N7Q9/q9rQ0vHrmKQj3Nm9FgsyQaaMARBMG3SKmPD/pOWHmPRI5eGnPwjmqCYViUF9/ZtjmuimNWsfYiRP52qbFIi0gLM3hULKcmJlt858h6dSq3hmQ1w/YWuIrmG1a2re6lTJcXSICoiuK9fC6Rb2Jwxntlftelv2r3WznAWDubkqcQLZxgAXzd3a4TMxtXROsaTRZgVmJSytvldKH+/tDl2H81FmcQEzN6UY19CbOZ7J+b+TKwcJ94o5uSpxLuoSQ3c1j0dANBcZ8pBX54taUQk5gHeDO4gYncDzDpVUvDRHRdFNA+wWcIdhbS0YZAnU5TVmZg6lp64oi2yxw8Jayz/+LzRNybeU2710Nmedy8PxaiXezjiqfsDgzyZ4vOR3fBAvxamTfxc2sRTUIiEb/Jb1qmEjw0MKR2t5ESJuL17LMRBXygGeTJHs1oV8fcIZiwq7dxtziPtsh9vPWY9y6B7+MxXa/XxSB+DPJU67o4nlcvZV47snuYuIcKsXjx0l48HZo9nEznvdLiHlQ5nknWrsHUNlTpXXFAXx07n4YbM0F3grfLh7Rdh3uacuLidN1Osgm685eB9b6iqlEuO6YCJwTAnT6VOQoLg1u6NI54s3Ax1qqREdZH5U4Zrjtl4ydHHRypID4M8UQn07NXtsf6pAaViqGuKDoM8UQmUmCC2Tl7vK15Kxu0SZ/XfXuLnW0JEJV68lZXHWu+WqXhkYCu7k+GFOXmiOBDPOUEybmDbOnHXg5pBnshGpT3nGy1eHENjkCci08RPu3VyY5AnoqjZdT8SJy1I0by2a7pIz4nK4wUrXokoanbl3+OluOb27o3RqWE1dGlUze6k+GFOnohME6s6hnjJwbslJEhcBniAOXkiW7lbYnRrWsPmlFhj0oguqF4h8kldKHoM8kQ2al+/Clb+q59jA2H/ttbO0UuhRVVcIyIvisgmEVkjIpNFpKq2PF1EzojIKu1vgimpJXIgpwZ4KyUlCGpWLIPnrmlvd1LiXrQ5+ZkARiul8kXkeQCjATyirduulOoY5f6JiPyICLIe62d3MkqEqHLySqkZSql87elSAPWjTxIRlTR9WtYCALRKCz3HLsWWmWXytwP4wuN5YxH5DcBJAI8ppRbqvUhERgIYCQANG9o3vjcRRe7qTvXQr03tuBo0LRrLxvRFkkNG+Az5iYjILAB6tSePKqW+17Z5FEA+gE+0dfsBNFRKHRGRLgC+E5G2SqmTvjtRSk0CMAkAMjIy4qTVKxGFyykBHgBqV06xOwmmCfmpKKUuC7ZeRG4FcDmAvkqbcFIpdQ7AOe3xChHZDqAFgKxoE0xERMZFdekVkYEAHgZwiVIq12N5KoCjSqkCEWkCoDmAHVGllGJq5b/6cegsIgeI9v7qTQBlAczUpiFbqpS6C0AvAE+LyHkAhQDuUkodjfJYFENs1kfkDFEFeaVUswDLvwHwTTT7JiKi6HHsGiIiB3NOdTgRxcyqx/shv5CN4UoCBnkiClvV8qyzKSlYXENE5GAM8kREDsYgT0TkYAzyREQOxiBPRORgDPJERA7GIE9E5GAM8kREDsYgT0TkYAzyREQOxmENiMiQl4Z1QMPq5e1OBoWJQZ6IDLmuS327k0ARYHENEZGDMcgTETkYgzwRkYMxyBMRORiDPBGRgzHIExE5GIM8EZGDMcgTETmYKBU/M66LyCEAu6LYRU0Ah01KTklQ2s4XKH3nXNrOFyh952zG+TZSSqXqrYirIB8tEclSSmXYnY5YKW3nC5S+cy5t5wuUvnO2+nxZXENE5GAM8kREDua0ID/J7gTEWGk7X6D0nXNpO1+g9J2zpefrqDJ5IiLy5rScPBEReWCQJyJyMEcEeREZKCKbRWSbiIyyOz3hEpF3RSRHRNZ5LKsuIjNFZKv2v5q2XETkde1c14hIZ4/X3KJtv1VEbvFY3kVE1mqveV1EJLZn6E1EGojIXBHZICLrReRebbkjz1lEUkRkuYis1s73KW15YxFZpqXxCxEpoy0vqz3fpq1P99jXaG35ZhEZ4LE8Ln8DIpIoIr+JyI/ac8ees4hka9+5VSKSpS2z/zutlCrRfwASAWwH0ARAGQCrAbSxO11hnkMvAJ0BrPNY9gKAUdrjUQCe1x4PBjANgADoCmCZtrw6gB3a/2ra42rauuXatqK9dpDN55sGoLP2uBKALQDaOPWctTRU1B4nA1impe1LADdoyycAuFt7/FcAE7THNwD4QnvcRvt+lwXQWPveJ8bzbwDA/QA+BfCj9tyx5wwgG0BNn2W2f6dt/xKY8MZ2AzDd4/loAKPtTlcE55EO7yC/GUCa9jgNwGbt8UQAw323AzAcwESP5RO1ZWkANnks99ouHv4AfA+gX2k4ZwDlAawEcBFcvRyTtOVF32MA0wF00x4naduJ73fbvV28/gYA1AcwG8ClAH7UzsGx5wz9IG/7d9oJxTX1AOzxeL5XW1bS1VZK7dceHwBQW3sc6HyDLd+rszwuaLflneDK3Tr2nLVii1UAcgDMhCsXelwpla9t4pnGovPS1p8AUAPhvw92exXAwwAKtec14OxzVgBmiMgKERmpLbP9O82JvEsApZQSEce1dRWRigC+AfBPpdRJzyJGp52zUqoAQEcRqQpgMoBW9qbIWiJyOYAcpdQKEeltc3JipYdSap+I1AIwU0Q2ea606zvthJz8PgANPJ7X15aVdAdFJA0AtP852vJA5xtseX2d5bYSkWS4AvwnSqlvtcWOPmcAUEodBzAXruKGqiLizmh5prHovLT1VQAcQfjvg526A7hSRLIBfA5Xkc1rcPA5K6X2af9z4LqQZyIevtN2l9uZUA6WBFflRGMUV8C0tTtdEZxHOrzL5F+Ed4XNC9rjIfCusFmuLa8OYCdclTXVtMfVtXW+FTaDbT5XAfAhgFd9ljvynAGkAqiqPS4HYCGAywF8Be9KyL9qj/8G70rIL7XHbeFdCbkDrgrIuP4NAOiN4opXR54zgAoAKnk8XgxgYDx8p23/Apj0Bg+Gq4XGdgCP2p2eCNL/GYD9AM7DVdZ2B1zlkbMBbAUwy+ODFgBvaee6FkCGx35uB7BN+7vNY3kGgHXaa96E1tPZxvPtAVf55RoAq7S/wU49ZwAXAPhNO991AB7XljfRfrjb4Ap+ZbXlKdrzbdr6Jh77elQ7p83waF0Rz78BeAd5R56zdl6rtb/17vTEw3eawxoQETmYE8rkiYgoAAZ5IiIHY5AnInIwBnkiIgdjkCcicjAGeSIiB2OQJyJysP8HvNm2EgrrMM0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "logits2 = model.lm_head(normed)\n",
    "plt.plot(logits2[0][0].cpu().detach().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "42ee2c68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(278, device='cuda:0'),\n",
       " tensor(1.6583, device='cuda:0', grad_fn=<SelectBackward>))"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.argmax(logits2[0][0]), logits2[0][0][278]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a425397f",
   "metadata": {},
   "source": [
    "Above results (based on torch_utils) are same as those from huggingface. Can we do the same using vector alone?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ee99679b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([768]),\n",
       " torch.Size([768]),\n",
       " tensor(71.0902, device='cuda:0', grad_fn=<DotBackward>))"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v1 = model.base_model.wte.weight[278]\n",
    "v2 = hidden_states['h.11'][0][0]\n",
    "v1.shape, v2.shape, torch.dot(v1,v2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a968084b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([768]),\n",
       " torch.Size([768]),\n",
       " tensor(1.6583, device='cuda:0', grad_fn=<DotBackward>))"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v1 = model.base_model.wte.weight[278]\n",
    "v2 = normed[0][0]\n",
    "v1.shape, v2.shape, torch.dot(v1,v2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca147c7",
   "metadata": {},
   "source": [
    "My plan is to measure other hidden state vectors against the projection of v1 onto v2. v1 is the full embedding for the token 'ing'. v2 is a point in the final layer latent space which is closest to v1 so the projection from v1 ont v2 should represent information directly related to the prediction. I am interested in how much of v1 came from lower layers, and how much from attention (probably none in this case because it is the first token), and if anything carried through to later layers. \n",
    "\n",
    "First I want to understand norm better, then confirm whether attention made a contribution for this token, if so attention to what?\n",
    "\n",
    "---\n",
    "\n",
    "The norm function is layer norm, and earlier we applied to the entire layer. Are results altered if we apply it to a single vector?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "02d5723f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LayerNorm((768,), eps=1e-05, elementwise_affine=True)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.base_model.ln_f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "06ce92c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228a8c7f340>]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9vUlEQVR4nO2dd5wU5f3HP9/dvcIdHO2OXo6OIIh4IiooAipilNh+UWOsCdFo1BgLamKPGoklGhv2mMTeQVFQxAp4KL33IhxHPeC4u73d5/fHzjM7OzuzOzM7W273+3697nW7s7Mz35155vk+3/J8HxJCgGEYhsk9POkWgGEYhkkPrAAYhmFyFFYADMMwOQorAIZhmByFFQDDMEyO4ku3AFpKS0tFeXl5usVgGIZpUsyfP3+nEKLM7vcySgGUl5ejsrIy3WIwDMM0KYhoo5PvsQuIYRgmR2EFwDAMk6OwAmAYhslRWAEwDMPkKK4oACJ6kYh2ENESzba7iGgrES1Q/sa7cS6GYRjGHdyyAF4GMM5g+6NCiCHK38cunYthGIZxAVcUgBDiKwC73TgWwzAMkxqSHQO4hogWKS6i1kk+F8PkLIcaAnj3xy3g8u6MHZKpAJ4G0AvAEADbADxstBMRTSSiSiKqrK6uTqI4DJO93P3RUtzw5kI8OnM1yidNw97ahnSLxDQBkqYAhBBVQoiAECII4DkAw0z2myKEqBBCVJSV2Z7JzDAMgO01dQCAxz9fDQBY+nNNOsVhmghJUwBE1FHz9iwAS8z2ZRiGYVKPK7WAiOg1AKMAlBLRFgB3AhhFREMACAAbAPzejXMxDMMw7uCKAhBCXGCw+QU3js0wjH04FsxYgWcCMwzD5CisABiGYXIUVgAMkwXoXT5E6ZGDaVqwAmAYhslRWAEwTBbCQWDGCqwAGIZhchRWAAyTBfCAn3ECKwCGYZgchRUAwzBMjsIKgGEYJkdhBcAwDJOjsAJgmCyAF4JhnMAKgGEYJkdhBcAwDJOjsAJgmCxE8MwAxgKsABiGYXIUVgAMwzA5CisAhslCCFwPmokPKwCGYZgcxRUFQEQvEtEOIlqi2daGiGYQ0Wrlf2s3zsUwTDT6aQAcBGas4JYF8DKAcbptkwB8LoToA+Bz5T3DMAyTIbiiAIQQXwHYrds8AcAryutXAPzSjXMxDBMNLwHJOCGZMYD2QohtyuvtANob7UREE4mokogqq6urkygOw2QvXAmCcUJKgsAiVKjEsIkKIaYIISqEEBVlZWWpEIdhGIZBchVAFRF1BADl/44knothGIaxSTIVwIcALlFeXwLggySei2EYDewSYqzgVhroawC+B9CPiLYQ0RUAHgRwMhGtBjBWec8wTBLQp30KAG/P34INOw+mRyCmSeBz4yBCiAtMPhrjxvEZhrGHEAI3vrUQLQp8WHz3qekWh8lQeCYww2QB0RPBQuyvb0zouD9u2oOqmrqEjsFkLqwAGCYLCQbdCQKc/dR3GPPwbFeOxWQerAAYJgtpdEkBAMCBBK0IJnNhBcAwWYhbFgCT3bACYJgsJMB5oIwFWAE0UQ7763Tc9NZCW98Z99hXuOTFeUmSiMkkAmwBMBZgBdBEOeQP4K35W2x9Z8X2/Zi9iust5QJBtgAYC7ACYJgsQN/fNwZYATDxYQXAZDWPf74ak95ZlG4xUg5bAIwVWAEwWc0jM1bh9R82p1uMlBMIpluC9LPvkB+7DzakW4yMxpVSEAzDZBacBQQcec9nCApgw4Onp1uUjIUtAIbJAvTF4AJsAoAToeLDCoBhshA3ZwIz2QsrAIbJQjgIzFiBFQDDZCHsAWKswAqAYbIA/YCfLQDGCqwAmiCCH24mDlwKgrECK4AmCD/c6eO1eZtQPmka6vyBdIsSEw4CM1ZgBdAE4Rzv9PHYzFUAgL21/jRLEhs3ykE3BUtz54F67K/L7HuRySR9IhgRbQCwH0AAQKMQoiLZ58x2msBzyaQZNwYJTaGdVdw3E22K8/HjX09OtyhNklTNBD5JCLEzRefKetgFxOjRtwhXLICEj5AauNyDc9gF1ARhF1D60c+8zTTciAE0BRcQkxipUAACwGdENJ+IJuo/JKKJRFRJRJXV1Vyr3goiRTneK7fvx5Oz1qTmZE0EAqVbBEu4YSWmovsPBgUvX5lGUqEARgghhgI4DcDVRHSC9kMhxBQhRIUQoqKsrCwF4iSPPSkyRVNlAZz91LeY/OlKNPKsIpVMH/lLXFEAKfipYx6ZjSPu+Sz5J3JAQ2MQ5ZOm4fmv16VblKSRdAUghNiq/N8B4D0Aw5J9znQwY1kVjrx3Br5fuwsA4A8EUd+YnFRBtyf5/Lz3kOH2usZQx88up2js9K8/bNgNf7KVqE4eN+5ZKiaTrd95EPvrGpN+HiccrA/J9cQX2WsFJ1UBEFExEbWQrwGcAmBJMs+ZLio37AYALNyyFwAwavKX6PeX6Uk5lxOT2Sxv/cuVO3Dcg1/g06XbTb/LQecw0gVk9R4s3rIP5z3zPf7x6cpkihVFMtwqc9btSr4iyyBI8fZlcywk2RZAewDfENFCAPMATBNCJKdXTDdqYwn932oyqnYDJ6O7Xz37veH2xVv2AQAWKYpLi/R2N8VJRXd8sATnTzH+zW5gdgvmrtuFLXtq1fe7DtYDAJZtq0nofJ8vr8JbldYXtnHbBbRoy16cP2UOJqdYkaWTptju7ZLUNFAhxDoARyTzHJlGKnzETtrlQqWj1yMPZRTclCOgQBNcX/bf329M6vHf/WkLrh3dBx5P5HX71ZQ5AMKLkHiVzxN1p1zxSiUA4LyKrpb2dycIHD7GrgOh+NbK7fsTPm5TQV7Dptf6rcNpoC4hO9BUWIupzpqwa3HUNjTihjcWYNeB+iRJ5B67DzY4yiN/bOZqvPbDprj7eRQtWucP4sa3FmLbvuRZhlrcnggmBwOZVGQu2c+BagFkzk92HVYALkEpzA7Uju5em7cJP23a4/hY8nk2kl8qNbujybcqt+Ddn7bisZmr4+4bDAr8ffqKCLdJKhl67wwMvXeGo+9ayfqS13X+xj14e/4W3PXhUkfnsou8Z4m0S+1dpwQb+IH6Rtc77GS7aHIh+y3nFcCK7TV46dv1CR1j7rpdePrLtQCsB4x++eS3uPC5OY7Opx2F3fruYpz11HeOjqPF8PFWNtp90OQ1sNJnLNtWg6e/XIs/vvaTrXNkAlY6Ra9un1QNoN1wAWnbmfwVTuTfd8iPw+/8FI8qdZSc8uI367FmxwH1fbKTExrZBZT9jHvsa9z90bKEjnHlf+arr60+IAs278V3SsqoXdw0w2PFLORDbzcGIPf2WOgg5W+R2SUV983Eg5+ssHU+SzIloee18vtS1Xno76PsvBIZt2svmfytTtre3tqQpfT+gq2OZQkEBe6ZugxnPfWtuq0xmNwRuhoDyCC3l9vkvAKQJHKTvZpAYCqaSlIs0xidmd0HLZGB2c4D9Xhm9lrnBzAhGc+wJ07vuudgA+7/eHmkHO6Lger99ajcGOkGdMXdYhADSFdfKAcItQ3hdObGJCcnyHNmb/fPCkDFn0Bj0o4EUxIEdtMCsHAovan98rfrsWDz3ohti7fsw1lPfYs6f8CWCyhVpRXcmsym/U3xLIC7P1qKRbrsq2S0j7Oe+jbquPrfu++QH9OXbLN1XK1VIX+pk7bnRoKE7Iy1Slfrmrz4xXnOD25C2AJw/dAZAysAhUTMyQgFkILxgpu+z3AaaDRqGqjuCbjro2X45ZPf6rYtxU+b9mLJ1nCHJx/8xhizop1er237DhnOXTDDLaWpPYz+mPpR94F685ngd324FIPv+tQVmbbsic4sCgeBQ/fgxrcW4sr//IgNOw9aPm7kb43eZhU3EiTkaN8fEPhgwVYIIXDftLDr9qtV7tcRC8cAslcDsAJQSMQC8MbzBehI1KfoaipejNG67MDnrN1lOUtHhA+pHvPcZ743nRXtVJmN/PssnPmvb+PvKOUyOM20RfZGxHoe+GRFxAxrvaIMGA4qQvu8/N0G1NQ1Yl+SFpb5enVk9XWpJA7UWy+7oP018rfJzvDWdxejfNI0WzIlZAForuV1ry/A3lo/Pljws/MDWkAqHbYAcoBERtUezVW00lgaEnTiG4m6cPNePPF5/LRLO8gO/K6PluHEyV/G3lf5L0S4k5B6Ue8u0iJHWUu21uCOD6xXCbGbmRQUAuWTpuGa//2obrta8zoWM5ZVoXp/aE6DXlHuqQ2ngurbkNGYQt8+jrjnM+zYX2dJjkTweeyn9GoHKlKZyU2vzTOfA/HIZyvxhmaOhP6ci7fsw0PTV0Qc/4cNu9U6WkboB2jxniE3ArfJDjJnAjmnAN6Zv8Uwbz5ezu/m3bUonzQtwsUh8UbEAMwb3ocLf8aVr85HbQzXgBWMHuIJT36Lh2dEptk9O3stnvvKeSVDbV8Xr+PQdoxB1QKIbxlpa8s4mb27aVe0ZfLO/C1R26RMU22O+usbA/jdvyvxmxfmGn5e5w/LH88lZEYqFjTxeRV3nMVO7ZvVO7FBc23lbdL/Rn17P9QQwONfrMEt7yzGjGVVoe/q9jnnme/w1JdrIwK65z3zPS6IkRatfz613zVCe+nNnsl563ej4r4ZqDFZUtLpTOBAUGT8mtGSnFMAf35roWHevD/Owzpr5Q4AxiMffTkAM6597SdMX7odtZrGsWlXLSa9swiNgSAe+GQ5FsYYLUtiuYC0nc4Dn6zA3z5eHrMxhmMAod9w7tPfoc/tH4e2GXTg8UZWQgjUK52ilauSaCbHCZNnRY2g//zWwqj9nJb0lR38ak3+uZZv1oRdLXqrxKizNfq1Pk/yH0NpAdT7jRXAw5+txOMaC/KiF+binKfDz4lqAei+V98Yebwz//WN+vqmt0P3QV8Ir0H5zt5D1t1fegugtiHalaVVEtoBi9mjPfnTFdh5oAHLf46u07S6aj++WBF65u1qgD+9sQD9/zpdkTuIif+uxOfLq9TPF27ei/d/cp4S6yapWhIy44lnAZCaBx39WYQFoPtMCBHVkR7U+GFvensh5q7fjfGDOuLZ2evw/Nfrsfb+8TFliTWy9AeDKPB4I7bJxqiVSYiQ4tL76/XphFHnNjm1VCAvfLMenykjPysawA0zu+ZQI9q1iL2PflayVVeIDF6b7f/X95egbXE+npy1Bm2bF0R8ZtXbcu1rP+HVK4ZFfd+urEbIWyCVjNnIWZY8/v2JPVHg80Z9rgZEdaLU+QMozAvvb6QoP1myXflu5Jf31sa2fAJBgSlfrcMFw7pGVSE9ZPA76hqDaO4N/U7tICkoBLwGjVEqr4K86N978qNfxZQtFh8uDMUmhBD4fHkVPltWhZo6P654pRJXnthLTXOeMKRTwjOsEyXnLAAz4gWB5SDfaASszQJ64os1uOylcEqa0bO7X2Ny5vtCt0CaoVYe9lgpjVZqq7/wzXr0vO1j7Kv1q/56wywgg23xOmy184fFeEhj4r5au+subNh5EL1u+9jSvlau54rt+7H055qoTBQjRW3UfpZtq8FzX683PX4iJZhl05QuoEMG1qC2zZk9B2aToupMLAogvv6PFwBftGUv/j59BUb+fZZqNUiMFJnW0tUrAD07aurUFN08r362ts7NZcEE2LjrINbsiCyU1xAIYqdSRK+kMA8AIua47MqAtYxZASgYdWzakbpsE0aNSe8CmrUy3BEYdeg1h8LHzVdGLPsUc1g7IJi3fjeenb0W+w75sXXvIXXEFKtjrbhvpvmHCq/OCfnaZaliOxj1/1+tqsY8ZT0ELVY6LjOFYieId1CJqazZsR8P6CZeGfHRQuvZI2Menh3x3qhTqzXJrDEKVDcEgoYdX6uiPFMZ9G4WJ0gXkNHIuUbjigkEhOG1N/OHH1LmfZi5L/8zJxzX0X93bZyU1L3Kddpf34gNuyL3jacAtM/dnR9E11/SJibom6BeqVlpiidO/hJjH4m0Gur8QfV6G/UDG3fV4i/vL8azSZj4aJWsdAGt2XEA1fvrcWyvtpa/o/dFr6raj1Me/Qr/PH8IJgzprDYu/X18dc5GLI9R613e+M27wwE1bdCpIC+kAGRj13Yw/6fU8H/2q3XYfbABrYvy8NMdp1iyEmJ1oHI0tae2Aa/GCrwa9HZa6+Ob1TvRu11z00k4lhSAyYizIRA0dEUYcVDxB1/+ciU27Y6frlqdQJVSI2kPGnRGr8/bZDhY+HbNLsMlENsU55ueUz/6dYJXcQHpLYBnZ6/FA5rSG/5g0FBxyW2LtuzDYRqX4qGGAOobg3h29jq88t2GiO/sqfXjL++bZ3b91eSzHzftQbc2RdipuU/f6NJaD/mjlW6dP4jv1u7Ev7/biAfPGaRuf/2HzXjg7EGqu2V11f6IduLXaYCDBvEFq9z23uLw79i4R73eRllLuw7U4z9zNim/J4ArT+wV4U5LBVlpAYx9ZHbMjAIj9J2VzPaZsawKS7buw33TQiPLqpo6jP/n12r2iVkjltTU+XHlq/Nx79Rlmm3hBiY7uX0xAmIyS2RPrR+rq/bjXwZL1Gkth5nLqqL8/lqkMrvwubmqLPpH/uIX5xm6P7R1gS56YS5OeXR21D4SI6VaPmkaVlWFTWUzJVHXEMR3a3aifNI0NQXTDGmpWfWVW6nhY4dDBh3GQ5+utJUJ0hgQuO71nyIGChLtNXKa3ijdHPqRs9711BgQhvdE687SKpG6xoA6oMmLE8y2KvrZT32HM5/4JiI76i1dZpeZBXD5yz9g+tLtUfMdtErt5Ee/Up9nILLdTPlqbUTwG4i2+ldX7TdNuf7f3HCSyGUv/4C560OprUaWl1YpPDZztRo7SCVZYQFs31eHt+dvxhlHdEL3tsXq9i9WVOGHDXvQu6w5fF7CnTFK8TbqGsH9H4dGRUIAT3wRvtlygs0Jk2fhjCM6xZVt9qpqTNctt6g1uVUXkLQA4nROZsEpD5E6Ov/tvytjHkOauFrXgr7zNJtZqY8/1MTwketHPR8vDqVgTl34M244pR8Acx97rb8RLykjyvkb96Ck0IcfNuzBtWN6o6omUiFIBWB1Qp6RoggGBdbvOojS4gJMXfwzLhzWzfBeGJ3BaPU3D5HqmrLCt2t2YtribdiwqxYfXH18xGfauQZ1/iCIQhZjh5aFlo8v3ZT7DvlR3xjAs7PXYeIJPZGv83/7A0FDi8Ns3kVdQ0C9h4X5XuyPM9HMShYZAPy8rw5rq40zrwDjDrW+MaAqGb3y9QeCyPMaKyh/IIhr/vcjTh/UUX3utQRFSK46fxDN8r341ZQ52H2wAVeM7IGi/NhdqFxxz6id669zq2bmbsBkkRUKYOeBevzjs1Xo275FhAK4/OVwR9i5VTPVzQIAHyzYiglDOqvvtaNVbSMQECgvDR9TixVfspH/VtsY8nzhB1MybdE2VJS3jntsLXYyReoMgqZWZxfbydpZsnUfZmqCwjITp0G51jOXVeGeqcaVWGsbAihQAuT1jQFc+Hyo4qpRSWHZ0VpVAEYj3MdmrsLjX6xB/w4tsGL7fgzu3AoDOpVE7Wd0lX7YEJ05Vdo839ayoNInXaB0Uo2BID5Y8DN+eWRnnP54OLXynKe/Q7c2RZi+dDtW3XeaYUeoRVuOAwi5Hf793UY8MmMV8rweNQlB0hgUhu4Ks/Z1yB/AAaU9N7PgvoinILSK5s3K6PkcEiMLYF31QfV5+2Rx5KDL3ygAEy9bnT+AqYu2xZwjMn3Jdlz13x/xxsTh6u+NdUyJHCDtN5hroO8b0rEEZVYogBaFoZ+xv67RtKF2alUY8UBe9/qCSAWg6djalxSoo8yPF2/HsB5tHMvmN1AA2hiATM+To7xAUODq//2Irm2aOT5nPIz6+ie+WIOBBh2eHnmZTh/cMW4phVVVBwytETnykXMrjPhwwc/qAxkvCCotAKsVOYw6uJnLQ7LIevN1jQHDXHOrNDQGLWUQSWTblDGh/83bhDs+WBrls1+2rUZ1oV31n/n4fMUOlLUoiOsmk9k9WldKYyAYrQACQTQ0Rl9Is+fqnqnLsFFxhxb44nuUq/aF5mwM6FhiuE6y1YD3IzOiBwI3v7NIfa2fFKn382vRJmWYIa/bnHW71cFSKPvM2qjdigXgRqzHLkmPARDROCJaSURriGhSMs4hU6xq6vymftcNBjNGtbVMtBZA84JIvThv/W50ad3MUgepx2i0qXUBSS+DPiVs8+7ULB2oResXNWLqop9VF9CJfcrw6K+OcHQeeU1iTQL7p8bHGq9TeHjGKlTV1MWMo2h598foSTjyu7IT27SrFpPeXRy1nxUdk+clrLNRdE2LvDbS/71ie3Qn2Sw/NNL+XJmodLC+ERcM62Z4vIZAELe/tzg8qUkrp88T5RZZvn0/npwVHWN69yfj0fhGzXMVr87QwYZG1YXZu11zY3kT6ARjGbH+QBBVNXURgWWJlXYj9wkKoSqAxVv34ekv10IIgdYxsrgAY8tH/1vtpjO7QVItACLyAngSwMkAtgD4gYg+FEIktgKLDq0FYBZUjDdCenv+FozqVwYiMhwhlhTmOapqaGTWaUcDshGkohxAPMx8pJI3K7dgcOdWAEI+5ZYOfZaqArA6GUun1L0eihqRHnP/545kkcgUWymT0WxiqxT6vPAHnFkPsj3ItE2ZJaKlWZ43og1p3WVG/Hdu9DGAUPxJbwFca7Iy25Kt5plukm37Ytc00so8uEtLw6BnsjrBl7/dgGdNyqJYUQCyXWzde0jNBLzilZB1e+rA9o4SC/S/NRstgGEA1ggh1gkhGgC8DmCC2yfxeT0oyvei5pDfcVXPaYu34ft1oYi90XT5ls3yHNWun6ybBg+EAsOS+gxSAPpcaz1frarG3R+FAum+BBSAbOhW4wlz1kUWCZtgIfhuhdMHd1Rfy1ROM6VUlO/Fpl21+DlOJweEJ101L/BZ8otrkYMPXwxlbCRhUb799MF8n0dNQkg1o/u3M9xutxN86bKjLe03U1OKQY8lBaDcl7cN6kxVbtzjqPPWW7ZuzPewS7LvfmcAmzXvtyjbVIhoIhFVElFldbXzmt4lhXnYvKcW6x2a3gDU79Y3BjF+UIfI4zfzuVuGWcFOqmC7FsalAtzC6Oe10LnDpNshZAHEjoA9ccGRhtsbbFoA0j8PAOMGdkDzQncMVyNFYuTrPr53W3iIcMLkWZaOK0eD+T4PZt04Kq57QEtDYxA1df6Y7gyjsiXNC32Yd9sYy+eRcsaz+pJFz7LmqpWjxW4naCXuAAD7DPz8E4Z0Uj6LrwBiJVnU+wO2Kvx6PYR8nycqkJ2NCiAuQogpQogKIURFWVmZ4+O0KPTh06VV6uQpJ9z+3hIs31aD+sYAurQuQv8O4QIzJYV5SakLHmsqvR47tdwl+mnudinI8+LIbq2itnuJUBZHIcXz89pdaxgARvQpjYrROMWq2V7ettjWtZcpl/leDzq0LMRxvUstfa9zq2ZYVXUAg+/6TM0fN8IodtKiwIdim9fFHwiqVtwfRvWy9V0tZx/ZWX1WOpTET019/uIKAMDnfz4RvcoiM+zkbOI7fjHA0rljTZxqoRko7DsUbWG3LQ61X6NRvZ5Ykxr9AePMKTN8HkKBzxO1nkE2KoCtALpq3ndRtrlOkUudwvyNe1DfGESBzxORVtiyWV5S1gWKZZrqiVcCF0CE0gIiH5CeZcbprLHYW9uAly6NNrO9HkJJnJG42eiy3qYLSDJhSCf8+phurlkAg7q0tLRfi0J7ri7ZbFT/usWGM7xneOZ6LJ97oYG7p0Vhnm03kD8QRFAI9GnXHGMOM3bJWKH6QL3q9urWpgjd2xbF3F9mOnVvW4zzKrpGfHbT26FMHm1pjFhutEKT2eLtSwrw19PDSsTINdy8wPr1iuVaDpXDMP+ufqAkRMhy0Qek0xEETrYC+AFAHyLqQUT5AM4H8GEyTlSkayTnDO0StU+8DgsAduyvV2+Q1kQtScMkDSfo5dSOcfc6WH2qMSjQqija1eP1UNxJa2b+5b2H/LjprYURrh0r9CgtBhEZWgAPnTPY1rEAoH1JoaXv2fWSeDUuICC6mNjN4/phTP92WHHvOHx980mG5zHKVpEcMEgpbF7gs1xZ8vDOoWy2+sYg/AGBPK/HdknqozXzVK45qbf6/dIWcRLjgYgSH2apu1oF8MDZgwz3KfBFB7ElPo8nbtKGHYspVokRoziflorukXN6zMptZF0QWAjRCOAaAJ8CWA7gTSGE+XTcBGimG/2cdniHqH2sdOJzlYBjgc8bUeRN21guGNbVll9Xy1UJmNpW0D9Q2mZ20IIbo3Mra/MPjPy3euQkNz0LN++NmtpvBWlRGM2+HO1wBBsvrjP9+pER5b6tIDtiqQD1pzisYwleuPRoFOZ5I0aHhyy6Aw/5A1GuPStWkZxbclyvkEvKHwiiMRhEnpfUEbxVHjr3CLwxcTjeueo4HNOzrSpPvJmxQKTf3swNp00wMPO/f3/rGFMXZ543/gBF+0xXdG+NXx9jnEqbCGP6t1OfKSmOEMZxh2x0AUEI8bEQoq8QopcQ4m/JOo/eTDRq0GZZK/3ah90mc9fvVr9/y7j+6nZtGmm3NsV49w+R0/WN+PPJfaO2nRkjg6XYwISPVSXSiJF9IuMo2s7n/avjy3xSf2txGKkcv775JHxzy0mG+7gdYJRKx8gloL23zyk+ZivEu779O5SohdT0AXEz5GBadk7yHnRpHeoI2rcI+8m1HaBRTSEz9C6JeHGRtsX56lwBQuha+gNB+ANB+LzR8wGMuPS4cgDAtaN7o0dpMY7p2RZHKaNbaQEU53sj2twvh0S3d+kCAswVQIHPiwEdS/D3cwZFDe4kbYrzo2Z/j1DiLWce0SluMUJtjKBZvhe/P8H9wdkLlx6tWiklGlei0bgj6yyAVKIPBhm5H0p0vtwLj+mGO88YgP/+7piofXcfbMDwnm3x5Y2jAIQsCm1T61FajCV3nxpTpjwD81Q7+tEH3q4e3Tvi/RlHdMKU31jvzHqWFuOqEyOPKTvGa07qjcM6lkS4HIwo1SxK8trvhmP2TaMM95Oj4q5titCldRGW3zMuah+nCkB2lHpkamSz/Ojjai2Skwe0N81A+vT6EyLenzygA2bfNAoXakZ/suOWaaLyZ7S0qIzltZFKUrqAbhnXH+/94biIEhPaDsxKjMcMM0U29rD2AEJWh+xsgyLk9nly1lp8u2ZXyAKwYNHJOj1GC6jIAZc+FteiMC9KYVtxARX4PPj4upH41dHdMG5gB9x1hnFQWAZyJWMPa4cFd5yM68f2jZo/okdeG3m+wrzEusPnL64wLEciFYBWSbuRAeUGWaMA9J2CUedb0iyycZYU5uGy43ug2MBs3a7ke5eXFmPDg6fjiK6t1M/koKV5gQ8r7o3u+FQZNB3g7eMPw32/PDzCZ/knjYWw9O5To5RWv/bNY5YJ1tO5dbOotQmG92yLp389FNeO6QMg1GGv+dtpUd+Vv0/bSI/t1TaitpIWfUM3GqU5zTG/9bTDMPncsG9eWkaycJlR5gcRYeofR6gK7owjOuE65Tdr0QdKvR5C97bFuPqksPJd/bfx2PDg6XjywqEAwh251XkPQ5VRsexw5Wgvz0s4slukP1h7GWXZB+38BKvIdvLMRUPVLBt5TiDUQUtf9LG92ka0wzyLFoD0W8dKvdTH4grzon3xES4gEw2gjTt5PIRLj++hvh/ZpxT/uSI0aGuW78WGB09HtzZF6r6tivLh8VDMDvW0wztEtKMCnzdKsf3vt8dg2rUjor5b2rwA14+NbltjB7THOMX1fPO4fmpblNdaq2AmntAz4rtEwFlHujO/xQ7ZowD0LiCDhqWvLy/bvFGDvtLAVy+tNu2RY6WhaYPO5w/riouGd4+QQfvQFRf4oh7ChsagrVHJ6qro6okNgSBOG9Qx4oH3eT2Yfv3IiADoMCWoZ72kcvzPY6WgxnKn5HkpInVSjvx9MWIAAHB455bo2iacgfKnk/ti+vUjI/YpMLmeeTF+kBzRx3MXPXj2IKy9fzwGdgplF6mryKl7GFQXjXABhRTAH0b1wop7x+HUge2j9jdDtv9xh3fE2AHh76nXzuNBRXkbLL7rFIzu3z6ireV5jQOmeiUv4yVGbV66L4oKfPidpnMr8EUvxmglBtA2xsDnuYsrMKJPZGqttLK0x4ulAIJCRLThW8b1j+oHjutdqt5LLR4CLj623Pi4yvPTrU2R2hblQEh73W46tZ8aczj3qC5Yee9pGN3f+v12i+xRALpOwWhEo29r0rdrNArpVWacw250nKl/jB4lAJEjRtkwzTogIDpuUR8ImmZnGHWu22siZ6nmez2q31ZP/w4l6KtJGZUdRWNQ4NLjyiNG4EbE0hNnH9kZH1w9wrQ650PnDMbjFxq7aIDQtdJ+Ux4nVgzAjFLdOrsFPi/uOmMA3rnquIjtsUbA3hgWwF91+epeD6n3xu56rzJ9slVRPgrzvBHKOJ6LxuxcUrFJmWRKqzbLyOw+6bfL+QdGbU/624vyvfjN8O6q8irweaJk0460zRSAmWUgj6lHWllamWOlVQZF+Jp1bFmIbm2LLE8q095jPUMVC697m7DlLI+rjTkQETopweHS5gWm2UzJJnsUgD4GYHBB9bdMO+rr0rqZocvACH1JiMM7G+eTGymAWG4R/YIaI3uXqW6rE/tGBmetuIZW/e00NUhnhDa75YKju6FDSSEmDOmEu84cGJWfrSeWpXDb6YdhUJeWpp3S/x3dFb1KzRVsKMU0/F47sxaIvNfjB3XAVzeZxzVa61JYC3weXHp8j6jrEisLRnYqelfhyD6luOTY7jipX+jeyCsilbY8ouyc4umDyecdgZcvO1rNGtFWqXRaKlgqtlilsv2BILq0LooKJOutInnPjTrtBo0C0O5bmOdVf/dIZdReGGEBRMsTz7qMpVi13zUq6SKR8Yz//vYYdf0FIsKcW81nU8uJjbFmUF8xogdm3nBixBwTqcw6toyMbY0fFHL1xUoMSTZZowD0vl3tiOmbW07Cp9efENVwvF7tPqNVn7y+DIQeqwM7bdqp/E7MUYbmuNOvH4kRfUpRlB+KM9yhC4J5iPDdpNGYpQSpnaDtFLq1LcKc28agS+vYk3gkscrrWhnNtC42d6cQRSpZ+azJjrVQE+/pXdYc3WJMPNJ3fGbX34oFoO345v9lLF694hj4vB60181+lcokvH+os4nXbEoK8zCqXzidVZsqGCvL55qTept+JmXR/74nLjhSTQuVo/dzhkZUaYm6drISrJGyrFXWZJCjXHm+gjyP+rsfP/9IrLxvXESdIyNlksicGyMX0N1nDsSCO06O2E/q0+N7l6Kd5v6Zjez7tW+Bj64JWfrXjult2l48HoqaAS/bsr7t9VDii0brTqSKrFEAx+nW/9XeoC6ti9BPN0MWMDarF991Cv55vrl7wowbT4lO+dQ+tPJhijV60a6WpO2IC/O8UZaDRzEhe5gsVmOFsEzx933moqMi3scq42Al+Btrdq3eApCWiux4tBbANaPjW23aOICpqySGzLJT0X61bfPoMhjaYC8QTgcNWwD2XELadSOuGtULfzn9sCh347lHdcGNp/YzPUaeGgOIPPcZR3TCtcq1kwHXqAGSiQXgNXBLXj6iB4Z2a6XOMWimBu7D+xJFx+GMLolZsP3co7rEtQ60MsvEhoGdSiKCyl3bNDMM4gLGySMr7h2Hj/44Qg04/+robpYXHwLCv1HGUOKVUEklWbEgDAD0ad8C/758mLpAuWEMQPfeqCFbmfZv9CBfM7oPPB7CQ9PDswK1aafakclNp/ZTp/2P7FOqNnjZUfxfRZeoEV/HloU4dWB7nNSvnWGdeifIS2SlJs64wzvgzd8fq9ZaiuWSMFMAlx1fjhsM5kYAodzyuet3Y+763SCdk82j82NLBdCnXXNL1kb/DvFHWLEeaHl5iEKZIa117jd1go8y0pdzRjopJr9R8oAV7vjFAFz13x8BhEaPvx3ZM6J44IYHT497DNnxG1UXlRVozYoMyufjzjMGYFS/dnhoemilPKOJcRcN746LhndX38uApz8QxJBurfHVqmpDGeoMArX6dG3JP847Av84z3gNCvnsaNvyOUM7Y3jPNlFW7dc3jzY8BmDcdhNdqF1NBhDAwjtPsZRymyqyxgIAgBP6lqn+U8M6M7rr7vRGmH1LHxsoLtAGu8Lbrz6pt+qDfvWKY/AvJd1QqPtGn8Hn9eDZ31SoS0XanLlviFfnq47HsB5tMOvGUTi2Z1vVn2uEWQCvQ0lhhIL97Yge6usbTumnduYBISKE8qoKIJwN9OKlFYbzNxLFaDZo2DALZScd1jG2QpGBwMuV3yctO7sl408b1BFXqMcIbbMaqOzSuhkO71yijmiNXBvjB3VEh5JCXGKS0SK/071tEXqUFqtK38roVyrpQ/4Anvr1ULz7h+MM3VhGs9NjufTioW17RGTZpSlJRnVUOWAMipB1Y7doXzLJHElc4qXLjsbL321QR19a9B203env8ZAP+OGdS/D4+UdGjHismP9BCx2FVA5OFqDQ4zVwbcSjR2kxXps43Pa57pkwMGrVqjOHdMLz36wPy6M8vMGgiPh9qgtIo/WSkTK3/oHxhtvVEbzpdQp9IDvpivI2WP/AePWeD+rcErNWVkfFCqwQjiLItmHtZn1zS2iU+8hnIYvUqNPu2iYU9zFDdREqUkgXkJWBk1QAdf4gmhf4VKWoR68A2pcUmNb+sYLdsh1R30/C6Fx/DzOJrFMAfdu3wP1nhRrQR9eMQPWB6AU8/nxyX/y4aQ/OcBh9N2tjsu0U+rzoGSON1IygZqRpfm6Ks4d1ZBDcyUI3dvB5yDBvWn9e+fAGgsLQBeS2wo6Sx+zGithBXKOvaY913di+GDugvWm2WGyZbH8lAjkIcTJgUDt65d+dZwyAz0NROfhGnDaoA/41a43pwi8S/Vq5vx3R09QFZIUM8q6oeMI+wowj6xSAllAqVvihk/ehfUkhXrpsmOPjWnUB2UW6CmI14qC6j/lOz1w01NLaBU4sgFjcM2EgZunWnp1z6xhTt4X+vLKjDwgR0YEmOqpLlPgWQOR+erwewuAurRydu0+7UPKCdkb2LeP6Y7DFUtaJKE2pPOQRurctxhSLdZYGdmppKUahtwCc3mr12Ynx8Dx83hG26vbH49bT+uOBT0JxkXeuOtZ0P30QOJPIagWgx61uxGykmGg/JXP9zz0qupS1xEpO+bjDrZUSsJMFZIWLjy2PGul3aGnd7SE7+qCJBRB0mAsPAO9cdZylaqhGqNfcpAWpJn4SHvDzKrqgb4cWGKIpRWKnomwirkKf2j6Sp4D/dHJf7Kn1o8DnwbTF2xJ2bcb6/jkxnisn/P7EXqoCOKp7G9P9RvQuhc9DuMRkUmY6ySkFIEnUF5es56F72+K4oyZZjuCkftGmtb6+SDyM8ttTSbQFEPofmqUZ3i47ogT6/5gT4uLhNIjrBkQU0fnbZaOyzrNZgb1YhONNjk8fl06tmuH5Sypw89sLEzqXbBppWuEyJu1KCrHmfuP4UrrJKQWgrced0HFMtqvFvxI7fExKmxfg+1tHo8wgD/228YfZOlY4yJce9CNqef0CQkR8plbVTJMJHS+NU1/0LZOQKYxmQdhY+FIUIwI0aZwJapt0DWaaKhmoL5OHbMhOn1MRx/+SqrbXsWUzw5xqu0h50/XQ6FNZ5Si9S+tmEb3tDSf3ReuiPBxu0e+dLJLl+ksmN57SD69cPiyimq1V3HYRWiFRdxMrAHvkpAWQ8HFsbs9UpM+9r8Es6VSgH1leelw5RvYpRe92LSJmwR7Tow1+uuOUVIunYnVkny4LJRbFBb6oOlJW8aXBQnTsAjIoBsfEJ6cUgCRhF5BJG0tmsCwZFBf48OoVwzDIQXqiG+gvFxGht5L1kklXMt78DFnV0ag8hBXemDjctYXuE0GvwFLZmcaaBGkHNyyAWTeOSnhxmKZC+ltdCtFP2U/W8ZsS+iUkU0msy6V9kDPlupr5wn83sie6tylSFwOxyzE928bfKQ2EVxBL5TmdfS+8HkDiMiRSX6upkTQ1R0R3EdFWIlqg/GVAGDyxYF14EpaZLzi9wcqmRqyOXftZKoKQVjCT1+shnDaoY5OzAPXo5U/2gEmL04J5etgFZI9kWwCPCiH+keRzWMa1GICZC8idw+cQ5ldM2+mnu1+VI8J+7dMTK0kV+oFLOrKbEr3ViWYR5Ro55QKSJJoFZBoE5rZnC+sWQHoZc1h7TP3jCAxMY932dBAuYpYCCyBBK8OoGmgquOMXA9Sqqk2RZCuAa4joYgCVAP4shNij34GIJgKYCADdukVXYnQTtWkk2KDNLYB0d1VNC6tXKxNcK07q+DR1wjOcU3hOh/danQiW4rZyuaaibVMkoRgAEc0koiUGfxMAPA2gF4AhALYBeNjoGEKIKUKICiFERVlZcgOS4ZocCR7HpOuKXgSciUWshz2TLIBcJdyeU9CiXTpFBowVmhQJWQBCiLFW9iOi5wBMTeRcGYVpGmhqxWjqxLpcmRQDyFXULCD36qeZ4nTRHD0cBLZHMrOAtBXJzgKwJFnnsorHpSwd84lg3PjsEMtfG2EBsAZICfrV8FJZxdKtekusAOyRzBjAQ0Q0BCHlvgHA75N4LkvIkrodDBaLsYNph8RtzxZWFr5hUsc1o3ujpJkP93+8QtmS/NpWepyXgw795/7fHklTAEKI3yTr2E657Lhy9G3fHCN6x1/QwgnpanuTzx3suNRxpsLPceopzPNi4gm9VAWgXcs22SR+ivjrZDDR5FQaqMdDrsx8zbSqkOdVdE3tCV3Cahookx7C1XNT16ATdaOyC8geuVHwwmXMawGF/nMWkDViZwHxg5xuUlHe3G3YArAHKwAHxFMAjDX4cmU24VpAqSsFkej3eSawPVgBOMB8aUBufHZghZnhuDRvxtYpE80C4kZlC1YADohrAXAxOEuwwsxsws05FaUg3Pk+GwD2yKkgcLIx8lsP6FiCnQfq0yBN5sODtcxGzgvIz8SFdnVIJcUuIHuwAnCA6dKABts+vm5kcoVpwvCjmtncNr4/OrcqxCkDna1zYAe3rAxuU/ZgBeAArgbqDpzpk9m0KMzDNaP7pPSc3CZSS+bbdhkIVwN1B37WGYlbUQZWIPZgBeAiXA3UHvyoMnqctgl+5pzBLiAHmC8JmWJBmjg8WstM3v3DcaitD6T2pAn24Pf98nDcN3U5Sgq5S7MDXy0HmPdb3KHZga9WZjK0W+u0ndvpmOAXgzvhF4M7uStMDsAuIAdwENgd+HoxkpQsOsNEwQrAAWYdV7qKwTVVOGjO6OE2kVpYATjC+jwAJgZ8wRgmrbACcAAXg3MHvl6MhK3m9MAKwAEcA3AHvlyMHn6GUgsrAAeYl4Lg1msHrt3OSNgCSA+sAFwkvCAMt2YrcP/PSIryvQCAvCZQeC6bSOhqE9F5RLSUiIJEVKH77FYiWkNEK4no1MTEzCzMXUDco9mBLSZGcucZA3H92D4Y3b9dukXJKRKdCLYEwNkAntVuJKIBAM4HMBBAJwAziaivECLF0wuTg3ktIMYOrC8ZScuiPFw/tm+6xcg5ErIAhBDLhRArDT6aAOB1IUS9EGI9gDUAhiVyrkyCs4AYhskGkuVw6wxgs+b9FmVbFEQ0kYgqiaiyuro6SeK4i5nrgieC2YMVJsOkl7guICKaCcBoRYjbhRAfJCqAEGIKgCkAUFFR0TS6TnYBuQLHABgmvcRVAEKIsQ6OuxVAV837Lsq2Jo0c2XMtOHdgC4Bh0kuyXEAfAjifiAqIqAeAPgDmJelcKYfnAbgDXy2GSS+JpoGeRURbABwLYBoRfQoAQoilAN4EsAzAdABXZ0sGEMAzgd2C02YZJr0klAYqhHgPwHsmn/0NwN8SOX6mEa+/4iCwPTzc/zNMWuFpdw7gNFB3YAuAYdILKwAHmC4JmWI5GIZhEoEVgAPYAmAYJhtgBWCDeGmg0qXBIQCGYZoCrACcwBPBGIbJAlgBOMA0BsA+IIZhmhCsABzA1UAZhskGWAE4gCeCMQyTDbACsIH1iWAcBmYYJvNhBWADNQuIh/oMw2QBrAAcwPMAGIbJBlgBOMA0BsBhYIZhmhCsABwQzwLgEADDME0BVgCOMJsHkGIxGIZhEoAVgAO4o2cYJhtgBWADoVT54f6fYZhsgBWAAzgNlGGYbIAVQBIQXA+UYZgmACsAG8g0T04DZRgmG0h0UfjziGgpEQWJqEKzvZyIDhHRAuXvmcRFzRzYA8QwTDaQ0KLwAJYAOBvAswafrRVCDEnw+BkJj/QZhskGElIAQojlQO4FReP9XJ4IxjBMUyCZMYAeRPQTEc0mopFmOxHRRCKqJKLK6urqJIqTOPGCuzmmBxmGaeLEtQCIaCaADgYf3S6E+MDka9sAdBNC7CKiowC8T0QDhRA1+h2FEFMATAGAioqKJjF25o6eYZhsIK4CEEKMtXtQIUQ9gHrl9XwiWgugL4BK2xJmIBwDYBgmG0iKC4iIyojIq7zuCaAPgHXJOFc6YAvAXU7oW5ZuERgmJ0koCExEZwF4AkAZgGlEtEAIcSqAEwDcQ0R+AEEAVwohdicsbYYQNwicGjGygrX3j2d7imHSRKJZQO8BeM9g+zsA3knk2JlIPNcPd2T28Xr4qjFMuuCZwDbgEg8Mw2QTrAAcwEFghmGyAVYASUDwTDCGYZoAiZaCYDRwdpC7TD53MDvdGCaJsAJwkZJmeQCAY3q2TbMk2cF5FV3TLQLDZDWsAFykXYtCzLzhRHRrU5RuURiGYeLCCsAGVlz7vds1T74gDMMwLsBBYIZhmByFFYANOMjLMEw2wQqAYRgmR2EFwDAMk6OwAmAYhslRWAHYgCf4MgyTTbACcAAHgxmGyQZYATAMw+QorAAYhmFyFFYADuBYAMMw2QArABuw759hmGwiIQVARJOJaAURLSKi94ioleazW4loDRGtJKJTE5aUYRiGcZVELYAZAA4XQgwGsArArQBARAMAnA9gIIBxAJ4iIm+C50o77PphGCabSEgBCCE+E0I0Km/nAOiivJ4A4HUhRL0QYj2ANQCGJXKuTIJdQQzDZANuxgAuB/CJ8rozgM2az7Yo2xiGYZgMIe56AEQ0E0AHg49uF0J8oOxzO4BGAP+1KwARTQQwEQC6detm9+sMwzCMQ+IqACHE2FifE9GlAH4BYIwIr4a+FYB2Pb8uyjaj408BMAUAKioq2MvOMAyTIhLNAhoH4GYAZwohajUffQjgfCIqIKIeAPoAmJfIuTKBZvmhODaHABiGyQYSXRLyXwAKAMygUGR0jhDiSiHEUiJ6E8AyhFxDVwshAgmeK+08fdFReKtyMy/7yDBMVkAig3IbKyoqRGVlZbrFYBiGaVIQ0XwhRIXd7/FMYIZhmByFFQDDMEyOwgqAYRgmR2EFwDAMk6OwAmAYhslRWAEwDMPkKKwAGIZhchRWAAzDMDlKRk0EI6JqABsTOEQpgJ0uieM2mSwbwPIlQibLBrB8iZDJsgFh+boLIcrsfjmjFECiEFGlk9lwqSCTZQNYvkTIZNkAli8RMlk2IHH52AXEMAyTo7ACYBiGyVGyTQFMSbcAMchk2QCWLxEyWTaA5UuETJYNSFC+rIoBMAzDMNbJNguAYRiGsQgrAIZhmBwlKxQAEY0jopVEtIaIJqVJhheJaAcRLdFsa0NEM4hotfK/tbKdiOhxRd5FRDQ0ybJ1JaJZRLSMiJYS0XUZJl8hEc0jooWKfHcr23sQ0VxFjjeIKF/ZXqC8X6N8Xp5M+ZRzeonoJyKamoGybSCixUS0gIgqlW0ZcW+Vc7YioreJaAURLSeiYzNFPiLqp1w3+VdDRNdnkHx/Up6JJUT0mvKsuNf2hBBN+g+AF8BaAD0B5ANYCGBAGuQ4AcBQAEs02x4CMEl5PQnA35XX4wF8gtDywsMBzE2ybB0BDFVetwCwCsCADJKPADRXXucBmKuc900A5yvbnwFwlfL6DwCeUV6fD+CNFNzfGwD8D8BU5X0mybYBQKluW0bcW+WcrwD4rfI6H0CrTJJPI6cXwHYA3TNBPgCdAawH0EzT5i51s+2l5MIm+aYdC+BTzftbAdyaJlnKEakAVgLoqLzuCGCl8vpZABcY7ZciOT8AcHImygegCMCPAI5BaIajT3+fAXwK4FjltU/Zj5IoUxcAnwMYDWCq8vBnhGzKeTYgWgFkxL0F0FLpxCgT5dPJdAqAbzNFPoQUwGYAbZS2NBXAqW62vWxwAcmLJNmibMsE2gshtimvtwNor7xOm8yKWXgkQqPsjJFPcbEsALADwAyErLq9QohGAxlU+ZTP9wFom0TxHgNwM4Cg8r5tBskGAALAZ0Q0n4gmKtsy5d72AFAN4CXFhfY8ERVnkHxazgfwmvI67fIJIbYC+AeATQC2IdSW5sPFtpcNCqBJIEJqOa05t0TUHMA7AK4XQtRoP0u3fEKIgBBiCEKj7WEA+qdLFi1E9AsAO4QQ89MtSwxGCCGGAjgNwNVEdIL2wzTfWx9CrtGnhRBHAjiIkEtFJd1tDwAUP/qZAN7Sf5Yu+ZS4wwSElGgnAMUAxrl5jmxQAFsBdNW876JsywSqiKgjACj/dyjbUy4zEeUh1Pn/VwjxbqbJJxFC7AUwCyHTthUR+QxkUOVTPm8JYFeSRDoewJlEtAHA6wi5gf6ZIbIBUEeKEELsAPAeQgo0U+7tFgBbhBBzlfdvI6QQMkU+yWkAfhRCVCnvM0G+sQDWCyGqhRB+AO8i1B5da3vZoAB+ANBHiYznI2TGfZhmmSQfArhEeX0JQr53uf1iJaNgOIB9GnPTdYiIALwAYLkQ4pEMlK+MiFopr5shFJ9YjpAiONdEPin3uQC+UEZpriOEuFUI0UUIUY5Q2/pCCPHrTJANAIiomIhayNcI+bGXIEPurRBiO4DNRNRP2TQGwLJMkU/DBQi7f6Qc6ZZvE4DhRFSkPMPy2rnX9lIRXEn2H0KR+VUI+Y1vT5MMryHkp/MjNOq5AiH/2+cAVgOYCaCNsi8BeFKRdzGAiiTLNgIhE3YRgAXK3/gMkm8wgJ8U+ZYAuEPZ3hPAPABrEDLNC5Tthcr7NcrnPVN0j0chnAWUEbIpcixU/pbK9p8p91Y55xAAlcr9fR9A6wyTrxihkXJLzbaMkA/A3QBWKM/FqwAK3Gx7XAqCYRgmR8kGFxDDMAzjAFYADMMwOQorAIZhmByFFQDDMEyOwgqAYRgmR2EFwDAMk6OwAmAYhslR/h+MN8bO5nTI3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "normv =model.base_model.ln_f(hidden_states['h.11'][0][0])\n",
    "plt.plot(normed[0][0].cpu().detach().numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9db172e",
   "metadata": {},
   "source": [
    "Looks the same both in pattern and magnitude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "ee8f6c44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.6583, device='cuda:0', grad_fn=<DotBackward>)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " torch.dot(v1,normv)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "499a0682",
   "metadata": {},
   "source": [
    "Same resulting logit element.\n",
    "\n",
    "LayerNorm is described in https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html, implemented in https://github.com/pytorch/pytorch/blob/c3d40fdf5658d8f4d387a39633c11bbb67b17224/aten/src/ATen/native/cuda/layer_norm_kernel.cu#L68 and based on https://arxiv.org/abs/1607.06450\n",
    "\n",
    "Layer Normalisation re-centers and re-scales the activations of a layer. Given $a^t$ as the summed inputs to layer t, the hidden state of layer t $h^t$ is defined as\n",
    "\n",
    "$$ h^t = f [ \\frac{g}{\\sigma^t} \\odot ( a^t - \\mu^t) + b]$$\n",
    "where the mean activation is \n",
    "$$ \\mu^t = \\frac{1}{H} \\sum_{i=1}^{H} a^t_i $$\n",
    "and the variance is\n",
    "$$ \\sigma^t = \\sqrt{\\frac{1}{H} \\sum_{i=1}^H (a^t_i - \\mu^t)^2}$$\n",
    "\n",
    "Weight $g$ and bias $b$ are learnable parameters, and $\\odot$ is elementwise multiplication. All the hidden units in a layer share the same normalization terms $\\mu$ and $\\sigma$, but unlike batch normalisation different training cases have different normalization terms.\n",
    "\n",
    "Based on the experiment above a layer here means a single embedding vector, rather than the layer of vectors across which attention is applied. \n",
    "\n",
    "---\n",
    "\n",
    "Next to investigate if attention has much impact on the first prediction.\n",
    "\n",
    "https://jalammar.github.io/illustrated-gpt2/ has a good high-level description of attention in GPT-style models.\n",
    "\n",
    "From the huggingface version of modeling_gpt_neo GPTNeoBlock\n",
    "\n",
    "hidden_states = (attn_output + residual) + mlp(ln_2(attn_output + residual)) where attn_output is GPTNeoAttention block applied to ln_2(input_hidden_state).\n",
    "GPTNeoLocalSelfAttention\n",
    "GPTNeoAttention implemented in either GPTNeoSelfAttention or GPTNeoLocalSelfAttention\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "88a90053",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPTNeoLocalSelfAttention(\n",
       "  (attn_dropout): Dropout(p=0, inplace=False)\n",
       "  (resid_dropout): Dropout(p=0, inplace=False)\n",
       "  (k_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "  (v_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "  (q_proj): Linear(in_features=0, out_features=768, bias=False)\n",
       "  (out_proj): Linear(in_features=0, out_features=768, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.base_model.h[11].attn.attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "d637f8c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.base_model.h[11].attn.attention.num_heads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7ff5929",
   "metadata": {},
   "source": [
    "From GPTNeoAttentionMixin.\\_attn and GPTNeoLocalSelfAttention - the values added for output_attentions=True are attn_weights before multiplying by value matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "26e9e83c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12, torch.Size([1, 1, 12, 14, 270]))"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(y.attentions), y.attentions[11].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "497292c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10723]], device='cuda:0')"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs2=text_to_input_ids(\"Watch\")\n",
    "inputs2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "47a7dae3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['logits', 'past_key_values', 'hidden_states'])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():    \n",
    "    y2 = model.forward(inputs2,output_attentions=False,\n",
    "        output_hidden_states=True,\n",
    "        return_dict=True)\n",
    "\n",
    "y2.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9d91b8c",
   "metadata": {},
   "source": [
    "output_attentions does result in the attentions value in output dictionary. Reducing the number of input tokens..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "296638aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['logits', 'past_key_values', 'hidden_states', 'attentions'])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():    \n",
    "    y2 = model.forward(inputs2,output_attentions=True,\n",
    "        output_hidden_states=True,\n",
    "        return_dict=True)\n",
    "\n",
    "y2.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8224fc21",
   "metadata": {},
   "source": [
    "The predicted token is 278 'ing' as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "34ce82b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12, torch.Size([1, 1, 12, 1, 257]), tensor(278, device='cuda:0'))"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(y2.attentions), y2.attentions[11].shape, torch.argmax(y2.logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d349438",
   "metadata": {},
   "source": [
    "Lots of dimensions. Is the first 12 for the layers attention is applied for, second 12 the num heads?\n",
    "From \\_split\\_heads the tensor dimensions could be (batch, blocks, head, block_length, head_features) - but what is blocks and block_length?\n",
    "\n",
    "\n",
    "Looking at a couple of attention vectors, they have last element=1, rest=0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "b97f3814",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAP5klEQVR4nO3df4xlZ13H8fenXYpRKhV2hKa7ZRdZEhZjpJnUJiVIIuJu/+hq/JE2IaBp2H+owYAmSzC1qX8BERNiRZfY8CNKrfhrE5cUhRqMsXWn0pbuNgtDAXeXSocfVk1DS7lf/7h3Zi/DTHe6e+89c555v5LNnHvO03O/T8/222ee83zPSVUhSeq/i7oOQJI0GSZ0SWqECV2SGmFCl6RGmNAlqRHbuvri7du3165du7r6eknqpfvvv/8bVTW31rHOEvquXbtYWFjo6uslqZeSfHW9Y065SFIjTOiS1AgTuiQ1woQuSY0woUtSI86Z0JPckeTxJA+vczxJPpBkMclDSa6afJiSpHPZyAj9w8C+Zzm+H9gz+nMQ+OCFhyVJeq7OmdCr6rPAt56lyQHgozV0L3BZkssnFaAkteL/nnqG93/qJA+e+u+pnH8Sc+hXAKfGPp8e7fsBSQ4mWUiysLS0NIGvlqT+ePKpZ/jAZxZ5+GtPTOX8M70pWlWHq2q+qubn5tasXJWkZg1G7xMKmcr5J5HQzwA7xz7vGO2TJI0phhn9ounk84kk9CPAm0erXa4BnqiqxyZwXklqysoIfUoJ/ZwP50ryceD1wPYkp4HfA54HUFV/AhwFrgMWgSeB35hOqJLUb8vvcM6UMvo5E3pV3XiO4wW8bWIRSVKjamUOfTqsFJWkGRnU8hz65r0pKknagOUR+kVTyrwmdEmakeUR+mZetihJ2oDRAH1qq1xM6JI0I9Ne5WJCl6QZWZlDd4QuSf3Wh9J/SdIG9KH0X5K0AYPB8Kc3RSWp55ZH6N4UlaSes/RfkhpxdpWLI3RJ6rWVSlHn0CWp33w4lyQ1wtJ/SWqEpf+S1AhL/yWpEZb+S1Ijqiz9l6QmDFbuik7n/CZ0SZqRsw/ncspFknrN0n9JasTZl0Q7QpekXjv7kujpMKFL0oycrRR1hC5JvebDuSSpFT4+V5LaMLCwSJLaYOm/JDWinEOXpDasjNC7TOhJ9iU5mWQxyaE1jl+Z5J4kn0vyUJLrJh+qJPVdx6X/SS4Gbgf2A3uBG5PsXdXsd4G7quo1wA3AH086UEnqu80wQr8aWKyqR6vqaeBO4MCqNgX86Gj7hcDXJheiJLWhNsGyxSuAU2OfT4/2jbsVeFOS08BR4DfXOlGSg0kWkiwsLS2dR7iS1F99Kf2/EfhwVe0ArgM+luQHzl1Vh6tqvqrm5+bmJvTVktQPm6H0/wywc+zzjtG+cTcBdwFU1b8BPwRsn0SAktSKzbBs8RiwJ8nuJJcwvOl5ZFWb/wR+DiDJqxgmdOdUJGlM53PoVfUMcDNwN/AIw9Usx5PcluT6UbN3Am9N8iDwceDXa/l/RZIkYPpz6Ns20qiqjjK82Tm+75ax7RPAtZMNTZLa0vkIXZI0GT4+V5IaUZugsEiSNAHVdem/JGkyNkPpvyRpArwpKkmN6EvpvyTpHDZD6b8kaQI2Q+m/JGkCnEOXpEY4hy5JjXCELkmNGKyUik7n/CZ0SZqxi0zoktRvZx/O5ZSLJPXaYGUOfTrnN6FL0ox4U1SSGjGY8ovcTOiSNGOO0CWp5wYDS/8lqQnLEy6O0CWp5yz9l6RG+E5RSWpEWVgkSW0opldUBCZ0SZqZQdXURudgQpekmalyhC5JTRgUZGprXEzokjQzVTW1FS5gQpekmRneFHWELkm9NxhsghF6kn1JTiZZTHJonTa/luREkuNJ/mKyYUpS/017hL7tXA2SXAzcDvw8cBo4luRIVZ0Ya7MHeBdwbVV9O8mPTytgSeqrQdUUb4lubIR+NbBYVY9W1dPAncCBVW3eCtxeVd8GqKrHJxumJPVf1fTK/mFjCf0K4NTY59OjfeNeCbwyyb8muTfJvrVOlORgkoUkC0tLS+cXsST1VPWksGgbsAd4PXAj8KEkl61uVFWHq2q+qubn5uYm9NWS1A+bofT/DLBz7POO0b5xp4EjVfXdqvoy8AWGCV6SNLIZSv+PAXuS7E5yCXADcGRVm79jODonyXaGUzCPTi5MSeq/zkv/q+oZ4GbgbuAR4K6qOp7ktiTXj5rdDXwzyQngHuB3quqb0wpakvpo+Aa6DpctAlTVUeDoqn23jG0X8I7RH0nSmqrzOXRJ0gQMBt0vW5QkTcCgyme5SFILfDiXJDVisPyW6CkxoUvSrBRcNMWsa0KXpBkZPpzLKRdJ6r3NUPovSZqAQdF56b8kaQJ8p6gkNaJqmoX/JnRJmpnCwiJJaoKl/5LUCEfoktSIwXQLRU3okjQr5cO5JKkNZem/JLXB0n9JaoSl/5LUiEEx1XWLJnRJmpHhTdHpnd+ELkkzYum/JDXCwiJJaoSl/5LUiKJ8HroktWDgHLokNaJwDl2SWjDwjUWS1IaBD+eSpDZMuVDUhC5JszIoXOUiSU3YDKX/SfYlOZlkMcmhZ2n3y0kqyfzkQpSkNnS+bDHJxcDtwH5gL3Bjkr1rtLsUeDtw36SDlKQWbIbS/6uBxap6tKqeBu4EDqzR7veB9wDfmWB8ktSMzVD6fwVwauzz6dG+FUmuAnZW1T8824mSHEyykGRhaWnpOQcrSX02XOWyiW+KJrkIeD/wznO1rarDVTVfVfNzc3MX+tWS1CtV1Xnp/xlg59jnHaN9yy4FfhL45yRfAa4BjnhjVJK+X22C0v9jwJ4ku5NcAtwAHFk+WFVPVNX2qtpVVbuAe4Hrq2phKhFLUk91XvpfVc8ANwN3A48Ad1XV8SS3Jbl+eqFJUluGL4meXkbftqEgqo4CR1ftu2Wdtq+/8LAkqT2DKb+DzkpRSZqRzTCHLkmagM2wykWSNAGDovtnuUiSLtxmKP2XJE3AYIA3RSWpFY7QJakBA2+KSlIbXLYoSY3ovPRfkjQZm/7xuZKkjSlH6JLUhrKwSJLaMFzl4pSLJPWepf+S1IjhHLojdEnqvSq8KSpJLZj2G4tM6JI0I5b+S1IjquCiKd4VNaFL0ow4QpekRlj6L0mNsPRfkhph6b8kNcLSf0lqxHAd+vTOb0KXpBmoKmp4V3Rq32FCl6QZqBr+dIQuST03yufOoUtS3w1GQ3RH6JLUcytTLl2X/ifZl+RkksUkh9Y4/o4kJ5I8lOTTSV42+VAlqb+WR+jTdM6EnuRi4HZgP7AXuDHJ3lXNPgfMV9VPAZ8A3jvpQCWpBV0/PvdqYLGqHq2qp4E7gQPjDarqnqp6cvTxXmDHZMOUpH5bHqF3Xfp/BXBq7PPp0b713AR8cq0DSQ4mWUiysLS0tPEoJannerdsMcmbgHngfWsdr6rDVTVfVfNzc3OT/GpJ2tRWRuhTXLa4bQNtzgA7xz7vGO37PkneALwb+Nmqemoy4UlSG1bWoXc8Qj8G7EmyO8klwA3AkfEGSV4D/ClwfVU9PvkwJanfajD82enz0KvqGeBm4G7gEeCuqjqe5LYk14+avQ94AfBXSR5IcmSd00nSllRMv7BoI1MuVNVR4OiqfbeMbb9hwnFJUlMGozkXX0EnST1Xy6X/XVeKSpIujCN0SWpErRQWOUKXpF5bXrbYdem/JOkCbZbSf0nSBepd6b8kaW2zKP03oUvSDCyP0J1ykaSeO5vQHaFLUq/NovTfhC5JMzBwykWS2rBS+u+UiyT122D674g2oUvSbDhCl6QmOIcuSY0YOIcuSW0oH58rSW0Y+PhcSWqDD+eSpEZY+i9JjbD0X5Ia4bJFSWqE7xSVpEYMXLYoSa2wsEiSmuAcuiQ14uw6dEfoktRrZ18SPT0mdEmaAUv/JakVzqFLUhsGm2UOPcm+JCeTLCY5tMbx5yf5y9Hx+5LsmnikktRjm6L0P8nFwO3AfmAvcGOSvaua3QR8u6peAfwh8J5JBypJfTaLZYvbNtDmamCxqh4dBpM7gQPAibE2B4BbR9ufAP4oSWq51nWC7jp2ig/9y6OTPq0kTdWTT39vtDW9jL6RhH4FcGrs82ngZ9ZrU1XPJHkCeDHwjfFGSQ4CBwGuvPLK8wr4sh9+Hnte8oLz+mclqUvXvuLFvOryS6d2/o0k9ImpqsPAYYD5+fnzGr2/8dUv5Y2vfulE45KkFmzkpugZYOfY5x2jfWu2SbINeCHwzUkEKEnamI0k9GPAniS7k1wC3AAcWdXmCPCW0favAJ+Zxvy5JGl955xyGc2J3wzcDVwM3FFVx5PcBixU1RHgz4CPJVkEvsUw6UuSZmhDc+hVdRQ4umrfLWPb3wF+dbKhSZKeCytFJakRJnRJaoQJXZIaYUKXpEakq9WFSZaAr57nP76dVVWojdtK/d1KfYWt1d+t1FeYXn9fVlVzax3oLKFfiCQLVTXfdRyzspX6u5X6Clurv1upr9BNf51ykaRGmNAlqRF9TeiHuw5gxrZSf7dSX2Fr9Xcr9RU66G8v59AlST+oryN0SdIqJnRJakTvEvq5Xljdd0m+kuTzSR5IsjDa96Ik/5jki6OfP9Z1nOcryR1JHk/y8Ni+NfuXoQ+MrvVDSa7qLvLnbp2+3prkzOj6PpDkurFj7xr19WSSX+gm6vOTZGeSe5KcSHI8ydtH+1u9tuv1t9vrW1W9+cPw8b1fAl4OXAI8COztOq4J9/ErwPZV+94LHBptHwLe03WcF9C/1wFXAQ+fq3/AdcAnGb6E8Rrgvq7jn0BfbwV+e422e0d/n58P7B79Pb+46z48h75eDlw12r4U+MKoT61e2/X62+n17dsIfeWF1VX1NLD8wurWHQA+Mtr+CPCL3YVyYarqswyfmT9uvf4dAD5aQ/cClyW5fCaBTsA6fV3PAeDOqnqqqr4MLDL8+94LVfVYVf3HaPt/gUcYvmu41Wu7Xn/XM5Pr27eEvtYLq5/tX2IfFfCpJPePXqoN8JKqemy0/V/AS7oJbWrW61+r1/vm0TTDHWPTZ830Ncku4DXAfWyBa7uqv9Dh9e1bQt8KXltVVwH7gbcled34wRr+/tbsWtPW+wd8EPgJ4KeBx4A/6DSaCUvyAuCvgd+qqv8ZP9bitV2jv51e374l9I28sLrXqurM6OfjwN8y/LXs68u/jo5+Pt5dhFOxXv+au95V9fWq+l5VDYAPcfbX7t73NcnzGCa3P6+qvxntbvbartXfrq9v3xL6Rl5Y3VtJfiTJpcvbwBuBh/n+l3C/Bfj7biKcmvX6dwR482hFxDXAE2O/vvfSqnniX2J4fWHY1xuSPD/JbmAP8O+zju98JQnDdws/UlXvHzvU5LVdr7+dX9+u7xafx93l6xjeUf4S8O6u45lw317O8E74g8Dx5f4BLwY+DXwR+CfgRV3HegF9/DjDX0W/y3Ae8ab1+sdwBcTto2v9eWC+6/gn0NePjfry0Og/8svH2r971NeTwP6u43+OfX0tw+mUh4AHRn+ua/jartffTq+vpf+S1Ii+TblIktZhQpekRpjQJakRJnRJaoQJXZIaYUKXpEaY0CWpEf8PIBmQxuYkBkwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAP5klEQVR4nO3df4xlZ13H8fenXYpRKhV2hKa7ZRdZEhZjpJnUJiVIIuJu/+hq/JE2IaBp2H+owYAmSzC1qX8BERNiRZfY8CNKrfhrE5cUhRqMsXWn0pbuNgtDAXeXSocfVk1DS7lf/7h3Zi/DTHe6e+89c555v5LNnHvO03O/T8/222ee83zPSVUhSeq/i7oOQJI0GSZ0SWqECV2SGmFCl6RGmNAlqRHbuvri7du3165du7r6eknqpfvvv/8bVTW31rHOEvquXbtYWFjo6uslqZeSfHW9Y065SFIjTOiS1AgTuiQ1woQuSY0woUtSI86Z0JPckeTxJA+vczxJPpBkMclDSa6afJiSpHPZyAj9w8C+Zzm+H9gz+nMQ+OCFhyVJeq7OmdCr6rPAt56lyQHgozV0L3BZkssnFaAkteL/nnqG93/qJA+e+u+pnH8Sc+hXAKfGPp8e7fsBSQ4mWUiysLS0NIGvlqT+ePKpZ/jAZxZ5+GtPTOX8M70pWlWHq2q+qubn5tasXJWkZg1G7xMKmcr5J5HQzwA7xz7vGO2TJI0phhn9ounk84kk9CPAm0erXa4BnqiqxyZwXklqysoIfUoJ/ZwP50ryceD1wPYkp4HfA54HUFV/AhwFrgMWgSeB35hOqJLUb8vvcM6UMvo5E3pV3XiO4wW8bWIRSVKjamUOfTqsFJWkGRnU8hz65r0pKknagOUR+kVTyrwmdEmakeUR+mZetihJ2oDRAH1qq1xM6JI0I9Ne5WJCl6QZWZlDd4QuSf3Wh9J/SdIG9KH0X5K0AYPB8Kc3RSWp55ZH6N4UlaSes/RfkhpxdpWLI3RJ6rWVSlHn0CWp33w4lyQ1wtJ/SWqEpf+S1AhL/yWpEZb+S1Ijqiz9l6QmDFbuik7n/CZ0SZqRsw/ncspFknrN0n9JasTZl0Q7QpekXjv7kujpMKFL0oycrRR1hC5JvebDuSSpFT4+V5LaMLCwSJLaYOm/JDWinEOXpDasjNC7TOhJ9iU5mWQxyaE1jl+Z5J4kn0vyUJLrJh+qJPVdx6X/SS4Gbgf2A3uBG5PsXdXsd4G7quo1wA3AH086UEnqu80wQr8aWKyqR6vqaeBO4MCqNgX86Gj7hcDXJheiJLWhNsGyxSuAU2OfT4/2jbsVeFOS08BR4DfXOlGSg0kWkiwsLS2dR7iS1F99Kf2/EfhwVe0ArgM+luQHzl1Vh6tqvqrm5+bmJvTVktQPm6H0/wywc+zzjtG+cTcBdwFU1b8BPwRsn0SAktSKzbBs8RiwJ8nuJJcwvOl5ZFWb/wR+DiDJqxgmdOdUJGlM53PoVfUMcDNwN/AIw9Usx5PcluT6UbN3Am9N8iDwceDXa/l/RZIkYPpz6Ns20qiqjjK82Tm+75ax7RPAtZMNTZLa0vkIXZI0GT4+V5IaUZugsEiSNAHVdem/JGkyNkPpvyRpArwpKkmN6EvpvyTpHDZD6b8kaQI2Q+m/JGkCnEOXpEY4hy5JjXCELkmNGKyUik7n/CZ0SZqxi0zoktRvZx/O5ZSLJPXaYGUOfTrnN6FL0ox4U1SSGjGY8ovcTOiSNGOO0CWp5wYDS/8lqQnLEy6O0CWp5yz9l6RG+E5RSWpEWVgkSW0opldUBCZ0SZqZQdXURudgQpekmalyhC5JTRgUZGprXEzokjQzVTW1FS5gQpekmRneFHWELkm9NxhsghF6kn1JTiZZTHJonTa/luREkuNJ/mKyYUpS/017hL7tXA2SXAzcDvw8cBo4luRIVZ0Ya7MHeBdwbVV9O8mPTytgSeqrQdUUb4lubIR+NbBYVY9W1dPAncCBVW3eCtxeVd8GqKrHJxumJPVf1fTK/mFjCf0K4NTY59OjfeNeCbwyyb8muTfJvrVOlORgkoUkC0tLS+cXsST1VPWksGgbsAd4PXAj8KEkl61uVFWHq2q+qubn5uYm9NWS1A+bofT/DLBz7POO0b5xp4EjVfXdqvoy8AWGCV6SNLIZSv+PAXuS7E5yCXADcGRVm79jODonyXaGUzCPTi5MSeq/zkv/q+oZ4GbgbuAR4K6qOp7ktiTXj5rdDXwzyQngHuB3quqb0wpakvpo+Aa6DpctAlTVUeDoqn23jG0X8I7RH0nSmqrzOXRJ0gQMBt0vW5QkTcCgyme5SFILfDiXJDVisPyW6CkxoUvSrBRcNMWsa0KXpBkZPpzLKRdJ6r3NUPovSZqAQdF56b8kaQJ8p6gkNaJqmoX/JnRJmpnCwiJJaoKl/5LUCEfoktSIwXQLRU3okjQr5cO5JKkNZem/JLXB0n9JaoSl/5LUiEEx1XWLJnRJmpHhTdHpnd+ELkkzYum/JDXCwiJJaoSl/5LUiKJ8HroktWDgHLokNaJwDl2SWjDwjUWS1IaBD+eSpDZMuVDUhC5JszIoXOUiSU3YDKX/SfYlOZlkMcmhZ2n3y0kqyfzkQpSkNnS+bDHJxcDtwH5gL3Bjkr1rtLsUeDtw36SDlKQWbIbS/6uBxap6tKqeBu4EDqzR7veB9wDfmWB8ktSMzVD6fwVwauzz6dG+FUmuAnZW1T8824mSHEyykGRhaWnpOQcrSX02XOWyiW+KJrkIeD/wznO1rarDVTVfVfNzc3MX+tWS1CtV1Xnp/xlg59jnHaN9yy4FfhL45yRfAa4BjnhjVJK+X22C0v9jwJ4ku5NcAtwAHFk+WFVPVNX2qtpVVbuAe4Hrq2phKhFLUk91XvpfVc8ANwN3A48Ad1XV8SS3Jbl+eqFJUluGL4meXkbftqEgqo4CR1ftu2Wdtq+/8LAkqT2DKb+DzkpRSZqRzTCHLkmagM2wykWSNAGDovtnuUiSLtxmKP2XJE3AYIA3RSWpFY7QJakBA2+KSlIbXLYoSY3ovPRfkjQZm/7xuZKkjSlH6JLUhrKwSJLaMFzl4pSLJPWepf+S1IjhHLojdEnqvSq8KSpJLZj2G4tM6JI0I5b+S1IjquCiKd4VNaFL0ow4QpekRlj6L0mNsPRfkhph6b8kNcLSf0lqxHAd+vTOb0KXpBmoKmp4V3Rq32FCl6QZqBr+dIQuST03yufOoUtS3w1GQ3RH6JLUcytTLl2X/ifZl+RkksUkh9Y4/o4kJ5I8lOTTSV42+VAlqb+WR+jTdM6EnuRi4HZgP7AXuDHJ3lXNPgfMV9VPAZ8A3jvpQCWpBV0/PvdqYLGqHq2qp4E7gQPjDarqnqp6cvTxXmDHZMOUpH5bHqF3Xfp/BXBq7PPp0b713AR8cq0DSQ4mWUiysLS0tPEoJannerdsMcmbgHngfWsdr6rDVTVfVfNzc3OT/GpJ2tRWRuhTXLa4bQNtzgA7xz7vGO37PkneALwb+Nmqemoy4UlSG1bWoXc8Qj8G7EmyO8klwA3AkfEGSV4D/ClwfVU9PvkwJanfajD82enz0KvqGeBm4G7gEeCuqjqe5LYk14+avQ94AfBXSR5IcmSd00nSllRMv7BoI1MuVNVR4OiqfbeMbb9hwnFJUlMGozkXX0EnST1Xy6X/XVeKSpIujCN0SWpErRQWOUKXpF5bXrbYdem/JOkCbZbSf0nSBepd6b8kaW2zKP03oUvSDCyP0J1ykaSeO5vQHaFLUq/NovTfhC5JMzBwykWS2rBS+u+UiyT122D674g2oUvSbDhCl6QmOIcuSY0YOIcuSW0oH58rSW0Y+PhcSWqDD+eSpEZY+i9JjbD0X5Ia4bJFSWqE7xSVpEYMXLYoSa2wsEiSmuAcuiQ14uw6dEfoktRrZ18SPT0mdEmaAUv/JakVzqFLUhsGm2UOPcm+JCeTLCY5tMbx5yf5y9Hx+5LsmnikktRjm6L0P8nFwO3AfmAvcGOSvaua3QR8u6peAfwh8J5JBypJfTaLZYvbNtDmamCxqh4dBpM7gQPAibE2B4BbR9ufAP4oSWq51nWC7jp2ig/9y6OTPq0kTdWTT39vtDW9jL6RhH4FcGrs82ngZ9ZrU1XPJHkCeDHwjfFGSQ4CBwGuvPLK8wr4sh9+Hnte8oLz+mclqUvXvuLFvOryS6d2/o0k9ImpqsPAYYD5+fnzGr2/8dUv5Y2vfulE45KkFmzkpugZYOfY5x2jfWu2SbINeCHwzUkEKEnamI0k9GPAniS7k1wC3AAcWdXmCPCW0favAJ+Zxvy5JGl955xyGc2J3wzcDVwM3FFVx5PcBixU1RHgz4CPJVkEvsUw6UuSZmhDc+hVdRQ4umrfLWPb3wF+dbKhSZKeCytFJakRJnRJaoQJXZIaYUKXpEakq9WFSZaAr57nP76dVVWojdtK/d1KfYWt1d+t1FeYXn9fVlVzax3oLKFfiCQLVTXfdRyzspX6u5X6Clurv1upr9BNf51ykaRGmNAlqRF9TeiHuw5gxrZSf7dSX2Fr9Xcr9RU66G8v59AlST+oryN0SdIqJnRJakTvEvq5Xljdd0m+kuTzSR5IsjDa96Ik/5jki6OfP9Z1nOcryR1JHk/y8Ni+NfuXoQ+MrvVDSa7qLvLnbp2+3prkzOj6PpDkurFj7xr19WSSX+gm6vOTZGeSe5KcSHI8ydtH+1u9tuv1t9vrW1W9+cPw8b1fAl4OXAI8COztOq4J9/ErwPZV+94LHBptHwLe03WcF9C/1wFXAQ+fq3/AdcAnGb6E8Rrgvq7jn0BfbwV+e422e0d/n58P7B79Pb+46z48h75eDlw12r4U+MKoT61e2/X62+n17dsIfeWF1VX1NLD8wurWHQA+Mtr+CPCL3YVyYarqswyfmT9uvf4dAD5aQ/cClyW5fCaBTsA6fV3PAeDOqnqqqr4MLDL8+94LVfVYVf3HaPt/gUcYvmu41Wu7Xn/XM5Pr27eEvtYLq5/tX2IfFfCpJPePXqoN8JKqemy0/V/AS7oJbWrW61+r1/vm0TTDHWPTZ830Ncku4DXAfWyBa7uqv9Dh9e1bQt8KXltVVwH7gbcled34wRr+/tbsWtPW+wd8EPgJ4KeBx4A/6DSaCUvyAuCvgd+qqv8ZP9bitV2jv51e374l9I28sLrXqurM6OfjwN8y/LXs68u/jo5+Pt5dhFOxXv+au95V9fWq+l5VDYAPcfbX7t73NcnzGCa3P6+qvxntbvbartXfrq9v3xL6Rl5Y3VtJfiTJpcvbwBuBh/n+l3C/Bfj7biKcmvX6dwR482hFxDXAE2O/vvfSqnniX2J4fWHY1xuSPD/JbmAP8O+zju98JQnDdws/UlXvHzvU5LVdr7+dX9+u7xafx93l6xjeUf4S8O6u45lw317O8E74g8Dx5f4BLwY+DXwR+CfgRV3HegF9/DjDX0W/y3Ae8ab1+sdwBcTto2v9eWC+6/gn0NePjfry0Og/8svH2r971NeTwP6u43+OfX0tw+mUh4AHRn+ua/jartffTq+vpf+S1Ii+TblIktZhQpekRpjQJakRJnRJaoQJXZIaYUKXpEaY0CWpEf8PIBmQxuYkBkwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(y2.attentions[11][0][0][11][0].cpu())\n",
    "plt.show()\n",
    "\n",
    "plt.plot(y2.attentions[3][0][0][8][0].cpu())\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "935930a7",
   "metadata": {},
   "source": [
    "Checking same again for a later token in the transformer output generated from a phrase earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "0ff8aa5a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12, torch.Size([1, 1, 12, 14, 270]))"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(y.attentions), y.attentions[11].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "4b7dede9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228e0c90a30>]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAATEklEQVR4nO3df6xfdX3H8ef7fr/9AbQWoRdlhdrq6rLGLQPvmE7jXKYOWEK3bHMlIdMF5Z+xaHQmGBdCcFmiRrNf7AdmZs44GXNu67Ia5hRjMgdrUUAKK9YCowWlQlPkZ3vv970/zvfbe+7tve098G3PD56P5OZ7vud7+N73J6d58bmf8/mcE5mJJKn9JuouQJI0Hga6JHWEgS5JHWGgS1JHGOiS1BH9un7x2rVrc8OGDXX9eklqpTvuuOOHmTm50Ge1BfqGDRvYuXNnXb9eklopIh5a7DOHXCSpIwx0SeoIA12SOsJAl6SOMNAlqSMMdEnqCANdkjrCQJekiu546CD3Pfpk3WUcw0CXpIqu/7ddfOor99ddxjEMdEmq6PBMMj0zqLuMYxjoklRRZtLEZ70Z6JJUUSYMGpjoBrokVZQkTXwes4EuSRVlFj9NY6BLUkWDTJo4im6gS1JFCQyaN8nFQJekqjJpbw89Ii6OiN0RsScirlng8/URcWtEfDsi7o6IS8dfqiQ1Q2a2c5ZLRPSAG4BLgM3A5RGxed5hfwDcnJkXAFuBvxh3oZLUFIOEBnbQl9RDvwjYk5l7M/MwcBOwZd4xCbxsuL0GeGR8JUpSsyTJoIHTXJYS6OuAh0vv9w33lV0HXBER+4DtwO8t9EURcVVE7IyInQcOHHgB5UpS/QYDWhvoS3E58LeZeR5wKfC5iDjmuzPzxsycysypycnJMf1qSTr1mhfnSwv0/cD5pffnDfeVXQncDJCZ/w2sBNaOo0BJappBWy+KAjuATRGxMSKWU1z03DbvmP8DfgkgIn6SItAdU5HUSZk0cqnoCQM9M6eBq4FbgPsoZrPsiojrI+Ky4WEfBN4bEXcBXwDenU280YEkjUFTe+j9pRyUmdspLnaW911b2r4XeNN4S5OkZipmLTYv0V0pKkkVZaZL/yWpC4ql/81joEtSRYP0fuiS1AkNneRioEtSVcUj6JqX6Aa6JFU08CHRktQR9tAlqRsGDZ3mYqBLUkWJPXRJ6oSmLv030CWpolY/U1SSNCsTl/5LUhc0sXcOBrokVTZw2qIkdUNmuvRfkrrAHrokdcDoLovNi3MDXZIqGXXMvX2uJLXcKMYbmOcGuiRVMeqZO4YuSS03WvLfvDg30CWpktGiokEDb+ZioEtSBWkPXZK6YXaWS711LMRAl6QKRhdDnbYoSS03ivEGDqEb6JJUxcBpi5LUDV4UlaSOSMfQJakbnOUiSR3hGLokdUTOe20SA12SKpidh15zIQsw0CWpilKQN+3CqIEuSRWUI7xpi4sMdEmqoHwx1B66JLVYOcPtoUtSi83poTdsrsuSAj0iLo6I3RGxJyKuWeSYd0bEvRGxKyL+frxlSlIzZC683QT9Ex0QET3gBuDtwD5gR0Rsy8x7S8dsAj4MvCkzD0bEOSerYEmqU5MDfSk99IuAPZm5NzMPAzcBW+Yd817ghsw8CJCZj423TElqhvIwS9NWiy4l0NcBD5fe7xvuK3st8NqI+K+IuC0iLl7oiyLiqojYGRE7Dxw48MIqlqQaDeZcFG1foC9FH9gEvBW4HPh0RJw5/6DMvDEzpzJzanJycky/WpJOnZxzUbRZlhLo+4HzS+/PG+4r2wdsy8wjmfkAcD9FwEtSp5R76Dmor46FLCXQdwCbImJjRCwHtgLb5h3zLxS9cyJiLcUQzN7xlSlJTdHiaYuZOQ1cDdwC3AfcnJm7IuL6iLhseNgtwOMRcS9wK/ChzHz8ZBUtSXUZNHhh0QmnLQJk5nZg+7x915a2E/jA8EeSOiu9OZckdUN5ZkvTeugGuiRVMKeH3rYxdEnSrHKIN2zExUCXpCravvRfkjSUL4GVopL0kjBo+UpRSdLQnEfQNWyai4EuSRU0bZilzECXpAocQ5ekjkgXFklSN5Qz3KX/ktRi5Quh9tAlqcXyOO/qZqBLUgXenEuSusKl/5LUDS+Fh0RL0kuCd1uUpI5wYZEkdUTTQrzMQJekCubcnKth4W6gS1IF5dWhDctzA12SqnAMXZI6ojxtsVlxbqBLUiVzh1yaFekGuiRVMHdhUX11LMRAl6RKvCgqSZ3g0n9J6oj05lyS1A0DL4pKUjfkIttNYKBLUgVzHxLdrEg30CWpAsfQJakjBvbQJakb0qX/ktQNcy6K2kOXpPYaePtcSeqItt/LJSIujojdEbEnIq45znG/HhEZEVPjK1GSmqPVC4siogfcAFwCbAYuj4jNCxy3GngfcPu4i5Skppj7CLrayljQUnroFwF7MnNvZh4GbgK2LHDcR4GPAc+NsT5JapS5UxWblehLCfR1wMOl9/uG+46KiAuB8zPz34/3RRFxVUTsjIidBw4cqFysJNUt2z6GfjwRMQF8CvjgiY7NzBszcyozpyYnJ1/sr5akU67tS//3A+eX3p833DeyGngd8PWIeBB4A7DNC6OSumjuPPTayljQUgJ9B7ApIjZGxHJgK7Bt9GFmHsrMtZm5ITM3ALcBl2XmzpNSsSTVaDBocQ89M6eBq4FbgPuAmzNzV0RcHxGXnewCJalJmhXhc/WXclBmbge2z9t37SLHvvXFlyVJzeQj6CSpI9Kl/5LUDZ2etihJLyVJi5f+S5Jm+cQiSeqI8jBLNmzOi4EuSRWUQ9wxdElqMYdcJKkj2n4vF0nS0Nwx9GYx0CWpgrlDLs2KdANdkiooD7MMGnZV1ECXpAqa+7wiA12SKpl7UbTGQhZgoEtSBY6hS1JHDLzboiR1w9wx9GYl+pIecCFJKgwy+Xj/rznAGgZ5Xd3lzGEPXZKqSPipiQfYHA855CJJbZZAjxn6zLj0X5LabDBI+jGgz6DuUo5hoEtSBQn0maEf064UlaQ2G2TSp+ihNyvODXRJqiQTejFDn2nH0CWpzTKTZcwUPfRm5bmBLklVlGe5uPRfklpskEmPwXDaYt3VzGWgS1IFmaNZLjONW/pvoEtSBYOEHgOW2UOXpLZLeszQY8aLopLUZjkzYIL0oqgktV3kEaAYR29WnBvoklRJ5AxQBLpL/yWpxWIwG+jNinMDXZIqiZwGYFnMMBg0646LBrokVTAacgEgS4H+zBPwR+vgoW+e+qKGDHRJqmBiMF3aPjL7wZOPwOGn4OCDp76oUT21/WZJaqNSD31Ob/3Is8XrzOFTXNCsJQV6RFwcEbsjYk9EXLPA5x+IiHsj4u6I+GpEvGr8pUpS/SbKIV7uoR95pnidOUJdThjoEdEDbgAuATYDl0fE5nmHfRuYysyfBr4IfHzchUpSE4wuihbbC/XQGxzowEXAnszcm5mHgZuALeUDMvPWzBz+74nbgPPGW6YkNcPEoDTkMigH+tPFa8OHXNYBD5fe7xvuW8yVwJdfTFGS1FRzeuilC6RHe+iD+nro/XF+WURcAUwBv7DI51cBVwGsX79+nL9akk6Jcq98YqFAb/iQy37g/NL784b75oiItwEfAS7LzOcX+qLMvDEzpzJzanJy8oXUK0m1KvfQNz79bfjaHxZv2nBRFNgBbIqIjRGxHNgKbCsfEBEXAH9NEeaPjb9MSWqGCWZ76O985GPwjU/A4afbMW0xM6eBq4FbgPuAmzNzV0RcHxGXDQ/7BLAK+MeIuDMiti3ydZLUanOGWUYOPjjbQ1/o81NkSWPombkd2D5v37Wl7beNuS5JaqQ5UxVHntjbjh66JGnWRGkM/emJ1cXGE3tbM4YuSRoqrxTNGEbonB66gS5JrVAO9BWDYa+8HOg1zkM30CWpgvIY+rLh4+h44oHSkItj6JLUChMLXRQ9tA+ePVhsz9Q3y8VAl6QKyhdFZyU8vrfYtIcuSe1wTA992RnF6+EfFa8GuiS1wzE99DXzbi5b48IiA12SKjimh37mvBsN2kOXpHY4JtDXrANi9r3z0CWpHco35wKKMfTTz5p9b6BLUjsccy+X/gpY9YrZ9y4skqR26B0T6CvhjNLzHRxDl6R2OLaHvny2hx49FxZJUlv05o+h91fCqnOK7ZUvs4cuSW3Ry2kG5ejsr5gdclm5xjF0SWqLiZzhSCxjMJqq2F85O+Sy4mXOcpGktpjIGWboMaBX7Ogth8nXAgEvf5WBLkltMcEMg+gxHcNA76+Eda+HD30PztlcDLlk1lSbJGnJelkE+tEeen9l8XrG2dBbVmzXdD8XA12SKpigGHKZiX6xo79i9sPe8uK1ppkuBrokVTDqoc8c7aGXAn1i2EOvaRy9X8tvlaSWmmC6GG6JQbFjTg/dQJek1ujlgEH0IUeBvrL04SjQ6xlyMdAlqYIJphnEBDkaQx+Nm5e3a1pcZKBLUgW9HDATfSKGUxPLPXTH0CWpPXrMDKcsjgLdMXRJaqXRwiIW6qE7hi5J7VFMW+yTsVAPfTSG7sIiSWq8HqOVon2m6cFEr/RhvT10A12SKji69D96HGb53A9rvihqoEtSBUd76NHn+ezzZ1/9LgCPP/U87/n8XcVBBrokNV8R6H1+7OzVDHor+ORX7ufRQ89y9/5DfP+p4dOMnIcuSc3XY4Zpeqw5/TQOrzoDnoFb//cATz8/zZFRpDrLRZKar5cz5EQPTjuTZavXsu7wady6+zFefvqy4iIpOA9dktqgx6BYWPT2jxLTz/KLXz/El761n41rz+Dw0R66Y+iS1Hg9ZhhM9GHVJJy5nktfdy7PHJ5h1yNPciSLQE+nLUpS8/WYIWN27vkbX3M2m85ZBcArz1oNwLPPPVdLbUsK9Ii4OCJ2R8SeiLhmgc9XRMQ/DD+/PSI2jL1SSWqA/mjp/1BE8O43bQDgjZteCcChp56uo7QTj6FHRA+4AXg7sA/YERHbMvPe0mFXAgcz88cjYivwMeC3TkbBklSn0bTFsq0/u551Z57G5PJpuAsOPfUs59ZQ21Iuil4E7MnMvQARcROwBSgH+hbguuH2F4E/j4jIHP+jr3d86U+YvOfT4/5aSVqS8/NHc4ZcAHoTwVt/4hwOPvkUAOd85694cNcXFv2Ox1//fl7/K+8Ze21LCfR1wMOl9/uAn1vsmMycjohDwNnAD8sHRcRVwFUA69evf2EFrzqbJ07f+IL+W0l6sZ7g1ay56PIFPztz9Rl8c93vsPLQ9477HctXnXUySju10xYz80bgRoCpqakX1Hu/4B1XwDuuGGtdkjQOEcHPv/ePa/v9S7kouh84v/T+vOG+BY+JiD6wBnh8HAVKkpZmKYG+A9gUERsjYjmwFdg275htwLuG278BfO1kjJ9LkhZ3wiGX4Zj41cAtQA/4TGbuiojrgZ2ZuQ34G+BzEbEHeIIi9CVJp9CSxtAzczuwfd6+a0vbzwG/Od7SJElVuFJUkjrCQJekjjDQJakjDHRJ6oioa3ZhRBwAHnqB//la5q1C7aCut7Hr7QPb2AVNbN+rMnNyoQ9qC/QXIyJ2ZuZU3XWcTF1vY9fbB7axC9rWPodcJKkjDHRJ6oi2BvqNdRdwCnS9jV1vH9jGLmhV+1o5hi5JOlZbe+iSpHkMdEnqiNYF+okeWN1GEfFgRHwnIu6MiJ3DfWdFxFci4rvD15fXXWcVEfGZiHgsIu4p7VuwTVH40+E5vTsiLqyv8qVbpI3XRcT+4bm8MyIuLX324WEbd0fEL9dT9dJFxPkRcWtE3BsRuyLifcP9nTiPx2lfe89hZrbmh+L2vd8DXg0sB+4CNtdd1xja9SCwdt6+jwPXDLevAT5Wd50V2/QW4ELgnhO1CbgU+DIQwBuA2+uu/0W08Trg9xc4dvPw3+sKYOPw33Gv7jacoH3nAhcOt1cD9w/b0YnzeJz2tfYctq2HfvSB1Zl5GBg9sLqLtgCfHW5/FvjV+kqpLjO/QXFv/LLF2rQF+Lss3AacGRF1PDS9kkXauJgtwE2Z+XxmPgDsofj33FiZ+Whmfmu4/SPgPornB3fiPB6nfYtp/DlsW6Av9MDq452AtkjgPyLijuGDtAFekZmPDre/D7yintLGarE2de28Xj0ccvhMaais1W2MiA3ABcDtdPA8zmsftPQcti3Qu+rNmXkhcAnwuxHxlvKHWfy916n5pV1s09BfAq8BfgZ4FPhkrdWMQUSsAv4JeH9mPln+rAvncYH2tfYcti3Ql/LA6tbJzP3D18eAf6b4M+4Hoz9Xh6+P1Vfh2CzWps6c18z8QWbOZOYA+DSzf5K3so0RsYwi7D6fmV8a7u7MeVyofW0+h20L9KU8sLpVIuKMiFg92gbeAdzD3Advvwv413oqHKvF2rQN+O3hLIk3AIdKf9K3yrwx41+jOJdQtHFrRKyIiI3AJuB/TnV9VUREUDwv+L7M/FTpo06cx8Xa1+pzWPdV2ao/FFfS76e4wvyRuusZQ3teTXHl/C5g16hNwNnAV4HvAv8JnFV3rRXb9QWKP1ePUIw1XrlYmyhmRdwwPKffAabqrv9FtPFzwzbcTREA55aO/8iwjbuBS+qufwntezPFcMrdwJ3Dn0u7ch6P077WnkOX/ktSR7RtyEWStAgDXZI6wkCXpI4w0CWpIwx0SeoIA12SOsJAl6SO+H/rA7fAV8UWRgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(y.attentions[11][0][0][11][5].cpu())\n",
    "plt.plot(y.attentions[3][0][0][8][5].cpu())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6c716ea",
   "metadata": {},
   "source": [
    "The vectors still look sparse, but have a few nonzero weights. Checking shape hasn't changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "08a04d52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(13, torch.Size([1, 1, 768]), 13, torch.Size([1, 14, 768]))"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(y2.hidden_states), y2.hidden_states[0].shape, len(y.hidden_states), y.hidden_states[0].shape,"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40797f0c",
   "metadata": {},
   "source": [
    "Look at the norm of activation weights in each head for first token position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "5022674f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228e0cf9550>]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAArJ0lEQVR4nO3deVxVdf7H8deHRRBFkEVQRMAd3BU3tGyxXCotm1JLxWqymnKymaaxml81ztTUTDXtTU6Za5pTWtakZnvugvsuLiC4IaCoiGzf3x9cijEUlHvvudz7eT4ePLyc5d73fURvLud8z/eIMQallFLuy8vqAEoppRxLi14ppdycFr1SSrk5LXqllHJzWvRKKeXmfKwOcL6wsDATGxtrdQyllKpTUlNTjxtjwqta53JFHxsbS0pKitUxlFKqThGR9Aut00M3Sinl5rTolVLKzWnRK6WUm9OiV0opN6dFr5RSbq7aoheRaSJyTES2XmC9iMhrIpImIptFpHuldckissf2lWzP4EoppWqmJp/opwODL7J+CNDG9jUBeBtAREKAp4HeQC/gaRFpXJuwSimlLl21RW+M+QHIvcgmw4GZptxqIFhEmgKDgGXGmFxjTB6wjIv/wqiVopIy/vbFDjLzChz1EkopVSfZ4xh9FHCw0veZtmUXWv4LIjJBRFJEJCU7O/uyQhzNL+SDNRk8OGc950pKL+s5lFLKHbnEyVhjzFRjTKIxJjE8vMoreKsVHRLAi7d3YVPmSf7y+XY7J1RKqbrLHkWfBURX+r65bdmFljvMoA6R3DegJbNXZ7BgfaYjX0oppeoMexT9ImCcbfRNH+CkMeYwsBS4XkQa207CXm9b5lB/uL4dfVqG8MTCLew4nO/ol1NKKZdXk+GVc4FVQDsRyRSRe0TkfhG537bJF8A+IA34N/AbAGNMLvAXYJ3ta4ptmUP5eHvx+ujuNPL35YHZqeQXFjv6JZVSyqWJq90cPDEx0dhj9sqUA7mMmrqaa9o34Z2xPRARO6RTSinXJCKpxpjEqta5xMlYR0iMDeHxofF8uf0oU3/YZ3UcpZSyjNsWPcDd/WK5oVNTXliyk1V7c6yOo5RSlnDrohcRXvhVZ2LDGjBx7gaO5hdaHUkppZzOrYseoKGfD++M6UFBUQkPzllPcWmZ1ZGUUsqp3L7oAdpEBPL8rZ1JSc/j+cU7rY6jlFJO5RFFDzCsSzPGJ8Xy3vL9/HfzYavjKKWU03hM0QM8MTSe7i2CeeyjTaQdO2V1HKWUcgqPKvp6Pl68eWd3/H29uX/2es6cK7E6klJKOZxHFT1A06D6vD66G/uyTzN5wRZc7YIxpZSyN48reoCk1mE8Oqgdn206xIyVB6yOo5RSDuWRRQ9w/5WtGBgfwV//u4PU9Dyr4yillMN4bNF7eQkv3d6FZsH1eXDOeo6fPmd1JKWUcgiPLXqAoPq+vD2mO3kFRUz8YAMlejGVUsoNeXTRA3RoFsRfb+7Iqn05vLxst9VxlFLK7jy+6AFuS4xmdK9o3vpuL8u2H7U6jlJK2ZUWvc3TN3WgU1QQv5u/kfScM1bHUUopu9Git/H39eatO7vjJcL9s9dztqjU6khKKWUXWvSVRIcE8Mqoruw8ks+fPtmqF1MppdyCFv15rm7XhInXtOHj9ZnMW3fQ6jhKKVVrWvRVePjaNlzRJoynP93G5swTVsdRSqla0aKvgreX8OqoboQH+vHA7PXknSmyOpJSSl02LfoLCGlQj7fu7E72qXNM+nAjZWV6vF4pVTdp0V9El+hgnropge93Z/P6N2lWx1FKqcuiRV+NO3u3YES3KF75ejff7862Oo5SSl0yLfpqiAjP3tKJdhGBPDxvA5l5BVZHUkqpS1KjoheRwSKyS0TSRGRyFetjRORrEdksIt+JSPNK60pFZKPta5E9wztL/XrevD2mB6WlhgfnrOdciV5MpZSqO6otehHxBt4EhgAJwGgRSThvsxeBmcaYzsAU4G+V1p01xnS1fQ2zU26niwtrwIu3d2FT5kmmfLbd6jhKKVVjNflE3wtIM8bsM8YUAfOA4edtkwB8Y3v8bRXr3cKgDpHcN6Alc9Zk8HFqptVxlFKqRmpS9FFA5UtEM23LKtsEjLA9vgUIFJFQ2/f+IpIiIqtF5ObahHUFf7i+Hb3jQnjyky3sOJxvdRyllKqWvU7GPgoMEJENwAAgC6g4kB1jjEkE7gBeEZFW5+8sIhNsvwxSsrNde2SLj7cXr9/RjUb+vjwwO5X8wmKrIyml1EXVpOizgOhK3ze3LfuJMeaQMWaEMaYb8KRt2Qnbv1m2f/cB3wHdzn8BY8xUY0yiMSYxPDz8Mt6GczUJ9OfNO7uTmXeWR+dv0snPlFIurSZFvw5oIyJxIlIPGAX8z+gZEQkTkYrnehyYZlveWET8KrYB+gFucSazZ2wIjw+N58vtR3nnh31Wx1FKqQuqtuiNMSXAQ8BSYAcw3xizTUSmiEjFKJqrgF0ishuIAJ61LY8HUkRkE+UnaZ83xrhF0QPc3S+WGzo15e9LdrJqb47VcZRSqkriaocdEhMTTUpKitUxauz0uRKGvbGcc8VlfP+Hq/Dx1mvQlFLOJyKptvOhv6CtVEsN/XyYPLg9WSfO8tUOvd+sUsr1aNHbwbXxETRvXJ/3VxywOopSSv2CFr0deHsJ4/rGsGZ/ro6tV0q5HC16OxmZ2IL6vt7MWHnA6ihKKfU/tOjtJCjAl5u7RbFwQ5bekUop5VK06O1ofFIs50rK9KbiSimXokVvR+0iA0lqFcrs1emUlJZZHUcppQAtertLTorVoZZKKZeiRW9nA+MjiArWoZZKKdehRW9n3l5CcpIOtVRKuQ4tege4PTEaf18vHWqplHIJWvQOEBxQj1u6Ndehlkopl6BF7yAVQy0/TNGhlkopa2nRO0i7yED6tgxl1iodaqmUspYWvQON76dDLZVS1tOid6CKoZbT9aSsUspCWvQOVDGr5ep9OtRSKWUdLXoHG9lTh1oqpaylRe9gFUMtP9moQy2VUtbQoneC5KQYCot1qKVSyhpa9E7QPrKRDrVUSllGi95Jfh5qeczqKEopD6NF7yQ/D7Xcb3UUpZSH0aJ3Eh1qqZSyiha9E1UMtZy56oDVUZRSHkSL3onKh1rqDcSVUs5Vo6IXkcEisktE0kRkchXrY0TkaxHZLCLfiUjzSuuSRWSP7SvZnuHrouSkWB1qqZRyqmqLXkS8gTeBIUACMFpEEs7b7EVgpjGmMzAF+Jtt3xDgaaA30At4WkQa2y9+3aNDLZVSzlaTT/S9gDRjzD5jTBEwDxh+3jYJwDe2x99WWj8IWGaMyTXG5AHLgMG1j123/XwDcR1qqZRyvJoUfRRQ+ThDpm1ZZZuAEbbHtwCBIhJaw30RkQkikiIiKdnZ2TXNXmcNjG+iQy2VUk5jr5OxjwIDRGQDMADIAkprurMxZqoxJtEYkxgeHm6nSK7Lx9vrp6GWO4/oUEullGPVpOizgOhK3ze3LfuJMeaQMWaEMaYb8KRt2Yma7OupdFZLpZSz1KTo1wFtRCROROoBo4BFlTcQkTARqXiux4FptsdLgetFpLHtJOz1tmUer/JQyxMFOtRSKeU41Ra9MaYEeIjygt4BzDfGbBORKSIyzLbZVcAuEdkNRADP2vbNBf5C+S+LdcAU2zJFpaGW63SopVLKccQYY3WG/5GYmGhSUlKsjuE0o6au4mDuWb7/w1X4eOv1a0qpyyMiqcaYxKrWabNYbHxSnA61VEo5lBa9xSqGWupJWaWUo2jRW8zH24uxfWNYtS9Hh1oqpRxCi94FjNKhlkopB9KidwE61FIp5Uha9C5Ch1oqpRxFi95FtI9sRJ+WIcxclU5pmWsNeVVK1W1a9C7k56GWR62OopRyI1r0LuSnWS1XHLA6ilLKjWjRuxAdaqmUcgQtehfz81DLdKujKKXchBa9iwkOqMfNXaNYuCFTh1oqpexCi94F6VBLpZQ9adG7oPimOtRSKWU/WvQuavxPNxDXoZZKqdrRondRA+MjdKilUsoutOhdVOWhlruOnLI6jlKqDtOid2EjE6Px8/Fius5qqTyAMYa0Y6eZteoAD8xO5fEFm62O5DZ8rA6gLqxxg4pZLTP54+B2BAfUszqSUnZ16MRZVqQdZ+XeHFbuPc7R/HMANPL3Ib+whJE9W9A1OtjakG5Ai97FJSfFMm/dQeanHGTCla2sjqNUreScPseqfTnlxZ52nAM5BQCENqhH31ah9GsdRlKrUEIb+tHnua+ZufIAXUd2tTa0G9Cid3HxTRvROy6EGSvTuad/S7y9xOpIStXYqcJi1u7PZeXeHFakHWen7XxToJ8PvVuGMLZvLP1ah9IuIhCR//3ZvrV7FHPXHuSJG+IJa+hnRXy3oUVfB9zVL5b7Z6/nqx1HGdQh0uo4Sl1QYXEp6zPyWJmWw4q9x9mceZLSMoOfjxeJsY35w6B2JLUKpVNUED7eFz9FOLZvLDNWpTNvbQYPXdPGSe/APWnR1wEVQy1nrDzgEkW/7dBJ5q09SNvIQMb2ibE6jrJQSWkZW7JO/nSMPeVAHudKyvD2Ero0D+KBAa1Iah1K9xaN8ff1vqTnbt2kIf1bhzF7dQb3D2hV7S8GdWFa9HWAj7cXY/rE8MKSnew6cop2kYFOz1BYXMrirYeZtSqd9RknAPD2Enq0aExCs0ZOz6OsUVZm2H3sFCvScli19zhr9uVy6lwJAO0jAxnTJ4akVqH0igsh0N+31q83rm8ME2alsmz7UYZ0alrr5/NUWvR1xKie0bzy1W5mrDrAc7d0ctrrHswtYM6aDOanHCT3TBFxYQ34vxsTGBjfhBFvreSJhVv4+IEkPXfgpowxZOQWsCKt/BP7qr055Jwpn2wvNjSAm7o2I6lVKH1blp9AtbdrK/6aXXVAi74WalT0IjIYeBXwBt41xjx/3voWwAwg2LbNZGPMFyISC+wAdtk2XW2Mud8+0T1LxVDLBesz+eOg9gQF1P7T0oWUlhl+2J3NrNXpfLvrGAJclxDB2D6xJLUKxctW6v93YwKTPtzInDXpjOsb67A8yhonCoq4499r2H64/N4IEY38GNA2nL6tQklqHUZUcH2HZ/D2Esv/mnUH1Ra9iHgDbwLXAZnAOhFZZIzZXmmzPwHzjTFvi0gC8AUQa1u31xjT1a6pPVTFUMsPUzIcMtQy90wR81MOMmdNOgdzzxIe6MfEq1szqlcLmlXxP/Xwrs34eH0mf1+yi0EdIolo5G/3TMoaxhieXLiV3UdP8dSNCVzZNpxW4Q1+MTLGGSr+mp256gDPOvGvWXdSk7MbvYA0Y8w+Y0wRMA8Yft42Bqg4UBsEHLJfRFWhYqilPWe1NMaQmp7HIx9upM9zX/P84p1EBdfnjTu6seKP1/C769tVWfIAIsJfb+5IcWkZf/5sm13yKNfw8fos/rvlML+7vi1394+jdZOGlpQ8lP81O6xLMxasz+Lk2WJLMtR1NSn6KKDyxOiZtmWVPQOMEZFMyj/NT6y0Lk5ENojI9yJyRVUvICITRCRFRFKys7Nrnt4D3dUvlsy8s3xdy1ktC4pKmLs2gxteW86tb69k2fajjO4VzZePXMm8CX25sXMz6vlU/+MRE9qA317bhi+2HKl1JuUaMnIKePrTrfSKC+E+F7lILzkplrPFpXycmml1lDrJXidjRwPTjTEviUhfYJaIdAQOAy2MMTki0gP4REQ6GGP+54aoxpipwFSAxMREnYD9IgbGR9AsyJ/pKw9w/WUMtUw7dprZq9P5ODWTU+dKaB8ZyLO3dOTmrlE08Lu8H4d7r2jJJxuyeOrTbfRpGXrZz6OsV1JaxqQPN+DlJfxzZFeXOcneMSqI7i2CmbU6nfFJsT+dJ1I1U5NP9FlAdKXvm9uWVXYPMB/AGLMK8AfCjDHnjDE5tuWpwF6gbW1De7LyWS1jWbk3h91HazarZXFpGYu3HOaOf69m4Mvf88GaDK6Jb8JH9/dl8cNXcGfvmFqVcz0fL54b0YmsE2d55avdl/08ynpvfruX9Rkn+OvNHZ1ysvVSJCfFsv/4GX5MO251lDqnJkW/DmgjInEiUg8YBSw6b5sM4FoAEYmnvOizRSTcdjIXEWkJtAH22Su8pxrVs2azWh45Wcg/l+2m/wvf8MCc9aTnFPDY4HasfPwaXh3VjcTYELsdd+0ZG8LoXi2YtuIAW7NO2uU5lXOtz8jjtW/2MLxrM4Z3Pf/orPWGdGxKWEM/Zupsrpes2o9xxpgSEXkIWEr50MlpxphtIjIFSDHGLAJ+D/xbRB6h/MTseGOMEZErgSkiUgyUAfcbY3Id9m48ROMGthuIr8/6xVBLYwyr9uYwa3U6X24/SpkxDGgbznO3xHBVuyYO/VN88uD2LNt+hCcXbmHBb/q5zJ/9qnqnz5XwyIcbiWzkz5ThHa2OU6V6Pl7c0Sua179NIyOngBahAVZHqjPEGNc6JJ6YmGhSUlKsjuHyth/KZ+hrP/Lk0HjuvbIlJ88Ws2B9JrNXp7M3+wzBAb6MTIzmjt4tiAlt4LRcn27M4uF5G/nzsA4kJ8U67XVV7fzxo83MTz3IvHv70LtlqNVxLujIyUL6vfANd/eL5ckbEqyO41JEJNUYk1jVOj1rVkclNCsfajl95QH2HT/NJxsOcba4lK7Rwbx0Wxdu6Nz0kucWsYdhXZrxUWom/1haPrY+MkjH1ru6JVuP8GHKQX5zVSuXLnmAyCB/BneI5MN1B/ndde2oX8/5P+N1kc4SVIfd1a/8BuILN2QxvGszPp/Yn08e7MetPZpbUvJQPrb+2Zs7UVxaxjOLdGy9qzuaX8jkBZvpFBXEpIF1Y5zEuL4x5BeW8OnG88eEqAvRT/R12KAOkcz5dW86Ngty6JQIl6pFaAAPD2zD35fsYtn2o1yXEGF1JFWFsjLDo//ZxLniMl4Z1bVG1024gl5xIbSPDGTGqnRG9oy27EKuuqRu/JdVVRIR+rUOc6mSr3DvFS1pFxHI059u5YxtdkPlWt5feYAf9xznTzfG0yq8odVxakxESE6KZcfhfFLS86yOUydo0SuH8PX24rkRHTl0spCXl+nYelez80g+LyzZycD4CO7o1cLqOJdseNdmNPL3YYYOtawRLXrlMD1iQrizdwveX7Ffx9a7kMLiUh6eu5FG/r68cGunOnnoI6CeD7cnRrNk6xGO5hdaHcfladErh3pscHtCGvjx+IItdpuITdXO35fsYtfRU/zjV50dMoe8s4zpE0OpMXywJsPqKC5Pi145VFB9X56+KYEtWSeZueqA1XE83g+7s5m2Yj/j+sZwdfsmVsepldiwBlzVNpwP1mZQVFJmdRyXpkWvHO7Gzk0Z0DacF5fu4tCJs1bH8Vi5Z4p49D+baN2kIU8Mjbc6jl2MS4ol+9Q5lmw7YnUUl6ZFrxyuYt76UmN0bL1FjDE8sWALeQVFvDqqq2XXWdjbgDbhxIQG6Pw31dCiV04RHRLApIFt+XL7UZbqpy+n+09KJku2HeHR69vRoVmQ1XHsxstLGNsnhpT0PD3hfxFa9Mpp7ukfR/vIQJ5ZtI3TOrbeaQ4cP8Mzn22jb8tQ7r2ipdVx7O62HtHU9/Vm1qp0q6O4LC165TS+3l48e0snjuQX8tKXu6rfQdVacWkZkz7ciI+X8NLtXdzyhh1BAb7c3C2KTzZmkXemyOo4LkmLXjlVj5jG3Nm7BTNWHmBz5gmr47i9179JY+PBEzw3otMF7/3rDpKTYjhXUsb8lIPVb+yBtOiV0z02uD2hDf14YuEWSkp1WJyjpKbn8sY3exjRLYobOzezOo5DtY8sn8111up0vV6jClr0yuka+fvyzE0d2JqVzww9ruoQpwqLmfThRpoF1+fPwztYHccpkpNiycw7y7c7j1kdxeVo0StLDO0UydXtwnnpy11k6dh6u3tm0Xay8s7yysiuBPq73qR3jnBdQgSRjfyZoRfm/YIWvbKEiDBleEeMgac/3Yqr3emsLvvv5sN8vD6TB69uTWJsiNVxnMbX24s7e7fgxz3H2Zt92uo4LkWLXlkmOiSAR65rw1c7jrF021Gr47iFwyfP8sTCLXSJDua317axOo7TjerVAl9v0aGW59GiV5a6q18c8U0b8cyibZwqLLY6Tp1WVmb4/fxNFJeW8crIrvh6e97/3uGBftzQqSkfp2bqtRqVeN5PgnIpvt5ePHdLR46eKuSlL3Xe+tp4b/l+Vu7N4akbE4gLc94N4V3NuKRYTp0rYeEGvdVgBS16ZbluLRoztk8MM1YdYNPBE1bHqZO2H8rnH0t3cX1CBCN7Rlsdx1LdooPpFBXEzJUH9NyPjRa9cgmPDmpHeMPyeet1bP2lKSwu5eF5GwgK8OX5WzvXyRuJ2JOIMK5vDHuOnWbVvhyr47gELXrlEhr5+/LnYR3Yfjif6ToT4SV5fvFO9hw7zYu3dSGkQT2r47iEm7o0o3GALzNX6klZ0KJXLmRwx0iubd+El77cTWZegdVx6oTvdh1j+soDjE+KZUDbcKvjuAx/X29G9mzBl9uP6HUa1LDoRWSwiOwSkTQRmVzF+hYi8q2IbBCRzSIytNK6x2377RKRQfYMr9yLiPx0FefTn27T46vVyDl9jkf/s5m2EQ2ZPKS91XFczpg+5Tc9/2CNfqqvtuhFxBt4ExgCJACjRSThvM3+BMw3xnQDRgFv2fZNsH3fARgMvGV7PqWq1LxxAL+7ri1f7zzGkq06b/2FGGP448dbyD9bzKujurnNjUTsqXnjAAbGRzB37UEKi0utjmOpmnyi7wWkGWP2GWOKgHnA8PO2MUAj2+Mg4JDt8XBgnjHmnDFmP5Bmez6lLuiufrEkNG3E04u2ka9j66s0d+1BvtpxlMcGtyO+aaPqd/BQyUmx5J4p4r+bD1sdxVI1KfoooPLcn5m2ZZU9A4wRkUzgC2DiJeyLiEwQkRQRScnOzq5hdOWufLy9+NuITmSfPsdLS3Xe+vPtyz7NXz7fTv/WYdzdL87qOC4tqVUorcIbePyN6e11MnY0MN0Y0xwYCswSkRo/tzFmqjEm0RiTGB6uJ5QUdIkOJrlvLDNXp7MhI8/qOC6j4kYifr5evHibe95IxJ5EhOSkWDZlnmSjB1+jUZMyzgIqX4HR3LassnuA+QDGmFWAPxBWw32VqtLvr29LRKA/jy/YQrGOrQfg1a/2sDnzJM/d0onIIH+r49QJI7o3p6Gfj0ffQLwmRb8OaCMicSJSj/KTq4vO2yYDuBZAROIpL/ps23ajRMRPROKANsBae4VX7i3Q35dnhnVg55FTvL9iv9VxLLfuQC5vfZfGr3o0Z2inplbHqTMa+vlwa/coPt98mOOnz1kdxxLVFr0xpgR4CFgK7KB8dM02EZkiIsNsm/0euFdENgFzgfGm3DbKP+lvB5YADxpjPPv0t7okgzpEMDA+gn8u28PBXM8dW59fWMykeRtp3jiAZ4Z5xo1E7Gls31iKSsv4cJ1n3mpQXG2scmJioklJSbE6hnIhWSfOct3L39M7LoRp43t65CX+j3y4kUWbDjH/vr70iGlsdZw6acy7a9ibfZofH7saHzec2VNEUo0xiVWtc793q9xOVHB9fnddW77dlc0XWzxrbL0xhnd/3MfCDVk8dHVrLflaGNc3hsMnC/lqh+fd+0CLXtUJ45Ni6RjViGc+85yx9YXFpfx+/ib++t8dDIyPYOI1ra2OVKddGx9BVHB9Znjg/Dda9KpO8PH24m+3dCbn9Dn+scT9x9YfzC3g1rdXsmBDFpMGtmHq2B5uebjBmby9hLF9Y1i1L4fdR09ZHcep9CdH1RmdmgeRnBTL7DXpvLh0l9sOuVy+5zjD3lhORm4B7yUnMmlgWx0vbycjE6Px8/HyuAuotOhVnfLYoPb8qntz3vg2jVvfXsk+N7oJtDGGf32/l3HT1hAe6Meih/pzbXyE1bHcSuMG9RjWpRkL1md5zCFA0KJXdUz9et7847YuvHVndzJyC7jhteXMXp1e52e6PHOuhIc+2MDzi3cypGNTFv6mn0ffDtCRkpNiKSgq5aOUTKujOI0WvaqThnZqypKHryQxtjF/+mQr98xIIftU3bwYZv/xM9z85goWbz3M40Pa88Yd3Wjg52N1LLfVMSqI7i2CmbU6nbKyuv0Boaa06FWdFRnkz4y7evHUjQksTzvO4Fd+4KvtdWvo3Nc7jjLs9eUcP32OmXf35r4BrTzyOgFnS06KZf/xM/yYdtzqKE6hRa/qNC8v4e7+cXw+sT9NGvnz65kpPLFwCwVFJVZHu6iyMsMrX+3mnhkptAgNYNFD/enfJszqWB5jSMemhDX085j5b7TolVtoGxHIJw8mcd+VLZm7NoMbXlvOJhedrTC/sJgJs1J45as9jOgexccPJBEdEmB1LI9Sz8eLO3pF882uY2TkuP/UGlr0ym34+Xjz+NB45vy6N4XFpdz69kpe/3oPJS40DHP30VMMf2MF3+3K5s/DOvDSbV307lAWuaN3DF4izPaAWw1q0Su3k9QqjCUPX8mQTk15adluRk5d7RKf2r7Ycpib31zBqcISPri3D8lJsXo83kKRQf4M7hDJh+sOcrbIveda1KJXbikowJfXR3fj1VFd2X3kFENf+5GPUjMtGYZZWmZ4fvFOfjNnPe0iA/l8Yn96xYU4PYf6pXF9Yzh5tphFm9z7Nhla9MqtDe8axeJJV5DQrBGP/mcTD36wnrwzRU57/bwzRSRPW8u/vt/Lnb1bMG9CH71hiAvpFRdC+8hAZqys+9diXIwWvXJ7zRsHMPfePkwe0p5l248y+NUf+HGP4+9NvDXrJDe+vpy1+3P5+62defaWTvj56PF4V1Jxq8Hth/NJTXffW1Zq0SuP4O0l3D+gFQt/049Af1/GvreWKZ9tp7DYMcdmF6zP5Na3V1JmDP+5vy+394yufidlieFdm9HI34cZq9z3pKwWvfIoHaOC+Hxif8YnxTJtxX6GvbGc7Yfy7fb8xaVlPLNoG7+bv4mu0cF8NrE/XaKD7fb8yv4C6vlwe2I0i7cc5lh+odVxHEKLXnkcf19vnhnWgel39SSvoJib31zB1B/21vpy+GOnCrnz32uYvvIAd/eLY/avexPW0M9OqZUjjekTQ6kxfLA2w+ooDqFFrzzWVe2asHTSlVzVLpznvtjJne+u4dCJs5f1XOsz8rjp9eVszjrBq6O68tRNCfjq/PF1RmxYA65qG86cNRkUlbjOdRf2oj+JyqOFNKjHO2N78PdbO7Mp8wSDX/mBRZsOXdJzfLAmg5HvrKKejxcLHujH8K5RDkqrHGlcUizZp86xZJv73a5Si155PBHh9p7RLH74Clo1achv525g0rwNnDx78fnKC4tLmfzxZp5YuIW+rcL47KH+JDRr5KTUyt4GtAknJjTALee/0aJXyiYmtAH/ua8vjwxsy2ebDzP01R9ZvS+nym0PnTjLyKmrmbfuIA9e3Yr3x/ckOKCekxMre/LyEsb2iSElPY+UA7lWx7ErLXqlKvHx9uLhgW346P6++HoLo/+9mucX7/yf47ar9+Vw0+vLSTt6in+N6cEfBrXHW2/15xZu7xlNVHB97puV6lZ3L9OiV6oK3Vo05r+/vYJRPaP51/d7ueWtFaQdO8W05fu58901BAX48ulD/RjcMdLqqMqOGvn7MuueXgCMfW8th09e3sl5VyOudtlvYmKiSUlJsTqGUj/5ctsRJi/YwomCIsoMXJcQwcu3dyHQ39fqaMpBtmadZNTU1UQG+TP/vr6ENHD9w3IikmqMSaxqXY0+0YvIYBHZJSJpIjK5ivX/FJGNtq/dInKi0rrSSusWXfa7UMoi13eIZMmkKxjeNYo/Dm7PO2N6aMm7uY5RQbybnMjB3ALuen8tp8+59o1sqlPtJ3oR8QZ2A9cBmcA6YLQxZvsFtp8IdDPG3G37/rQxpmFNA+kneqWUq/hq+1Hum51Kn5YhTBvf06XnKqrtJ/peQJoxZp8xpgiYBwy/yPajgbmXHlMppVzLwIQI/vGrzqxIy2HSvI2U1tGbidek6KOAg5W+z7Qt+wURiQHigG8qLfYXkRQRWS0iN19uUKWUssKI7s156sYEFm89whMLttTJ6Yx97Px8o4CPjDGVpwSMMcZkiUhL4BsR2WKM2Vt5JxGZAEwAaNGihZ0jKaVU7dzdP44TBUW89k0awQ18eXxIvNWRLklNPtFnAZXnWG1uW1aVUZx32MYYk2X7dx/wHdDt/J2MMVONMYnGmMTw8PAaRFJKKed65Lq2jO0Twzvf7+Nf3++tfgcXUpOiXwe0EZE4EalHeZn/YvSMiLQHGgOrKi1rLCJ+tsdhQD+gypO4SinlykSEPw/rwE1dmvH84p3Mq0MzXVZ76MYYUyIiDwFLAW9gmjFmm4hMAVKMMRWlPwqYZ/73AFY88I6IlFH+S+X5C43WUUopV+flJbx0WxfyzxbzxMItBNX3ZUinplbHqpZeMKWUUpfobFEpY95bw5bMk0wb35P+bcKsjlT7C6aUUkr9rH49b6Yl96RleAMmzEph48ETVke6KC16pZS6DEEBvsy8uxdhDf0Y//5a9hw9ZXWkC9KiV0qpy9SkkT+z7+lNPW8vxr63lsy8AqsjVUmLXimlaqFFaAAz7+lFQVEJY99bS/apc1ZH+gUteqWUqqX2kY14/66eHD55luRpa8kvvPjdyZxNi14ppeygR0wI/xrTgz3HTvHr6SkUFpdWv5OTaNErpZSdXNWuCS/f3pV16bk8OGc9xaVl1e/kBFr0SillRzd1acZfhnfk653HeOyjzZS5wIyX9p7UTCmlPN6YPjGcKCjixS93Exzgy1M3JiBi3X2FteiVUsoBHry6NXkFxby3fD+NA+rx22vbWJZFi14ppRxARHhyaDwnCop5edluGgf4MrZvrCVZtOiVUspBvLyEF27txMmzxTy1aBuN6vsyvGuV921ybA6nv6JSSnkQH28v3rijG71iQ/j9/E18u/OY0zNo0SullIP5+3rzbnIi7ZsG8sCcVNYdyHXq62vRK6WUEwT6+zL9rl40C6rP3dPXseNwvtNeW4teKaWcJKyhH7N+3ZuGfj6MfW8t6TlnnPK6WvRKKeVEUcH1mXVPL0rLyhjz3hqO5hc6/DW16JVSyslaNwlkxt29yD1dxLj31nKioMihr6dFr5RSFujcPJh/j0tk//Ez3D19HQVFJQ57LS16pZSySFLrMF4b3Y2NB09w36xUikocMwmaFr1SSllocMdInh/RmR/3HOeR+RsdMgmaXhmrlFIWu71nNCfOFnH6XCmOmPtMi14ppVzAhCtbOey59dCNUkq5OS16pZRyc1r0Sinl5mpU9CIyWER2iUiaiEyuYv0/RWSj7Wu3iJyotC5ZRPbYvpLtmF0ppVQNVHsyVkS8gTeB64BMYJ2ILDLGbK/YxhjzSKXtJwLdbI9DgKeBRMAAqbZ98+z6LpRSSl1QTT7R9wLSjDH7jDFFwDxg+EW2Hw3MtT0eBCwzxuTayn0ZMLg2gZVSSl2amhR9FHCw0veZtmW/ICIxQBzwzaXsKyITRCRFRFKys7NrklsppVQN2ftk7CjgI2NM6aXsZIyZaoxJNMYkhoeH2zmSUkp5tppcMJUFRFf6vrltWVVGAQ+et+9V5+373cVeLDU19biIpNcg14WEAcdrsX9d5Gnv2dPeL+h79hS1ec8xF1ohxlx8XgUR8QF2A9dSXtzrgDuMMdvO2649sASIM7YntZ2MTQW62zZbD/QwxjjsPloikmKMSXTU87siT3vPnvZ+Qd+zp3DUe672E70xpkREHgKWAt7ANGPMNhGZAqQYYxbZNh0FzDOVfnMYY3JF5C+U/3IAmOLIkldKKfVLNZrrxhjzBfDFecueOu/7Zy6w7zRg2mXmU0opVUvueGXsVKsDWMDT3rOnvV/Q9+wpHPKeqz1Gr5RSqm5zx0/0SimlKtGiV0opN+c2RV/dxGvuRkSiReRbEdkuIttE5GGrMzmLiHiLyAYR+dzqLM4gIsEi8pGI7BSRHSLS1+pMjiYij9h+rreKyFwR8bc6k72JyDQROSYiWystCxGRZbZJIJeJSGN7vJZbFH2lideGAAnAaBFJsDaVw5UAvzfGJAB9gAc94D1XeBjYYXUIJ3oVWGKMaQ90wc3fu4hEAb8FEo0xHSkf1j3K2lQOMZ1fzv01GfjaGNMG+Nr2fa25RdFz6ROv1XnGmMPGmPW2x6co/5+/yjmI3ImINAduAN61OosziEgQcCXwHoAxpsgYc8LSUM7hA9S3XbAZAByyOI/dGWN+AM6/rmg4MMP2eAZwsz1ey12KvsYTr7kjEYmlfGroNRZHcYZXgMeAMotzOEsckA28bztc9a6INLA6lCMZY7KAF4EM4DBw0hjzpbWpnCbCGHPY9vgIEGGPJ3WXovdYItIQ+BiYZIzJtzqPI4nIjcAxY0yq1VmcyIfyKUTeNsZ0A85gpz/nXZXtuPRwyn/JNQMaiMgYa1M5n22WAbuMf3eXor+Uidfchoj4Ul7yc4wxC6zO4wT9gGEicoDyw3PXiMhsayM5XCaQaYyp+GvtI36eO8pdDQT2G2OyjTHFwAIgyeJMznJURJoC2P49Zo8ndZeiXwe0EZE4EalH+YmbRdXsU6eJiFB+3HaHMeZlq/M4gzHmcWNMc2NMLOX/jb8xxrj1Jz1jzBHgoIi0sy26Fth+kV3cQQbQR0QCbD/n1+LmJ6ArWQRU3HI1GfjUHk9ao7luXN2FJl6zOJaj9QPGAltEZKNt2RO2eYmUe5kIzLF9iNkH3GVxHocyxqwRkY8on+22BNiAG06HICJzKZ/GPUxEMim/7erzwHwRuQdIB263y2vpFAhKKeXe3OXQjVJKqQvQoldKKTenRa+UUm5Oi14ppdycFr1SSrk5LXqllHJzWvRKKeXm/h/7ZnwPvO1t5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "position0_last_layer_attention_vecs=y.attentions[11][0][0][0][range(0,11)].cpu().numpy()\n",
    "plt.plot([np.linalg.norm(v) for v in position0_last_layer_attention_vecs])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f8d693d",
   "metadata": {},
   "source": [
    "Comparing against the norm of activation weights in each head for the next token position, different but nothing stands out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "ec131d04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228e0dc9280>]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtGUlEQVR4nO3deVyVdd7/8dcHDosggsBBEVBRAcVdSE3No61qi2XRMktN0zJN20zTLNXMXVMzTbO1zV1Nd8001T3TopZmWVZaalkW4K6I4Aq4AKKIKPv39wfYj5tQjnLOuc7yeT4ePuRc5+Jcn6P49jrf6/v9XGKMQSmllO8LsroApZRSrqGBrpRSfkIDXSml/IQGulJK+QkNdKWU8hM2qw4cHx9vBg4caNXhlVLKJ+Xn51caY+ydPWdZoA8cOJC8vDyrDq+UUj5JRHaf7DkdclFKKT+hga6UUn5CA10ppfyEBrpSSvkJDXSllPITXQa6iLwkIuUisukkz4uI/E1EikVkg4iMc32ZSimluuLMGfrLwIxTPD8TSGv7dSvw9+6XpZRS6nR1GejGmJVA1Sl2mQ28alqtBmJEJNFVBXa07UANf1qyFW37q5RS/5crxtCTgJJ2j0vbtn2LiNwqInkikldRUXFGB/usqJK/L9/Oexv2ndH3K6WUv/LoRVFjzAvGmGxjTLbd3unK1S79YNJARiZF8/C7W6g+1ujiCpVSyne5ItDLgJR2j5PbtrlFcJDw2JyRVNXW86cPt7rrMEop5XNcEeiLgOvbZrtMBKqNMW4dDxmRFM2Nk1N57as95O061fC+UkoFDmemLb4OfAlkiEipiNwkIreJyG1tu7wP7ACKgReB291WbTs/uyCdpJgePLBgIw1NLZ44pFJKebUuuy0aY67r4nkD3OGyipwUGWbjkdnDuemVPF78bAd3TB/i6RKUUsqr+PRK0fOG9WHmiL48vayIXZW1VpejlFKW8ulAB/jtZcMJDQ7iNws36dx0pVRA8/lA79MrnF/OyODz4koWrnPb5BqllPJ6Ph/oAN+dMIAxKTH87r0CDtU2WF2OUkpZwi8C/cTc9OrjjTz2QYHV5SillCX8ItABhiX24uZzUpmbV8rqHQetLkcppTzObwId4KfnpZMS2zo3vb6p2epylFLKo/wq0HuEBvO72SPYUVHL35dvt7ocpZTyKL8KdIBpGQlcOrofz326neLyo1aXo5RSHuN3gQ7w4CWZhIcE8esFG3VuulIqYPhloNujwrh/1jC+2lnFvPxSq8tRSimP8MtAB7gmO4XsAb35w/sFHDxab3U5Sinldn4b6EFtc9Nr65t4dLHOTVdK+T+/DXSAtD5R3OYYzNtry/i8qNLqcpRSyq38OtAB7pg+hIFxEfx64UbqGnVuulLKf/l9oIeHBPPoFSPZffAYz3xSbHU5SinlNn4f6ACTh8QzZ1wSz6/YzrYDNVaXo5RSbhEQgQ7w61nDiAq38cDbG2lp0bnpSin/EzCBHtczjAdmDSNv9yHeyC2xuhyllHK5gAl0gKuykpk4KJbHPiigvKbO6nKUUsqlAirQRYRHrxhJfWMLv3tP56YrpfxLQAU6wGB7T+6YPoR31+9leWG51eUopZTLBFygA9w2bRCD7ZH8ZuEmjjU0WV2OUkq5REAGepgtmD9cMZLSQ8d5elmR1eUopZRLBGSgA0wYFMc12Sn847OdbNl7xOpylFKq2wI20AHunzWUmB4h3L9gI806N10p5eMCOtBjIkL5r0syWV9ymP98tdvqcpRSqlsCOtABZo/pxzlp8fx5SSH7q3VuulLKdwV8oIsIv798BI3NLTz87mary1FKqTMW8IEOMCAukrvPS+ODTftZuuWA1eUopdQZcSrQRWSGiBSKSLGI3NfJ8wNEZJmIbBCR5SKS7PpS3evWqYPI6BPFg+9sorZe56YrpXxPl4EuIsHAs8BMIBO4TkQyO+z2V+BVY8wo4BHgMVcX6m4hwUH8Yc4I9lbX8cTH26wuRymlTpszZ+jjgWJjzA5jTAPwBjC7wz6ZwCdtX3/ayfM+IWtALN+d0J9/rdrJxtJqq8tRSqnT4kygJwHt+82Wtm1rbz0wp+3rK4AoEYnr+EIicquI5IlIXkVFxZnU63a/nDGUuJ5h3L9gA03NLVaXo5RSTnPVRdGfAw4RWQs4gDLgWzfwNMa8YIzJNsZk2+12Fx3ataJ7hPDQpZlsKjvCK1/q3HSllO9wJtDLgJR2j5Pbtn3DGLPXGDPHGDMW+HXbtsOuKtLTLh6ZyPQMO49/VEjZ4eNWl6OUUk5xJtBzgTQRSRWRUOBaYFH7HUQkXkROvNb9wEuuLdOzRIRHZo/AGHjonU0Yo20BlFLer8tAN8Y0AXcCHwIFwFxjzGYReURELmvbbRpQKCLbgD7Ao26q12NSYiO454I0lhaU8+Hm/VaXo5RSXRKrzj6zs7NNXl6eJcd2VlNzC5c9s4qDtfV8/DMHvcJDrC5JKRXgRCTfGJPd2XO6UvQUbMFBPDZnJOU19Tz+YaHV5Sil1ClpoHdhdEoMN5w9kFdX72btnkNWl6OUUielge6Eey9Mp09UOPe/vZFGnZuulPJSGuhOiAoP4eHZw9m6v4bXv95jdTlKKdUpDXQnXTS8L6OSo3ntqz06jVEp5ZU00E9DTlYyW/fXsFnvQaqU8kIa6KfhstFJhNqCmJ9fanUpSin1LRropyE6IoQLM/uwcF0Z9U3falWjlFKW0kA/TTnZKRw+1siygnKrS1FKqf9DA/00TRkST99e4TrsopTyOhropyk4SJgzLonlheWUH6mzuhyllPqGBvoZuCormRYDb68t63pnpZTyEA30MzDI3pPsAb2Zl1eic9LdZMHaUt7QRVxKnRYN9DN0VVYy2ytqWVdy2OpS/E5J1TF+9dZGHvtgq94GUKnToIF+hi4elUh4SBDz9OKoyz26uICGphaqjzeyXm/WrZTTNNDPUFR4CLNGJPLu+r3UNeqcdFf5oriSJZv3c/OUVIIEVmzzzpuJK+WNNNC74arsZGrqmvSORi7S1NzCw+9uIbl3D35+UQZjUmJYUajz/ZVylgZ6N0xMjSO5dw/m5emwiyu89vUeCg/U8JuLhxEeEowjPYENZdUcPFpvdWlK+QQN9G4IChKuHJfMqu2VlB0+bnU5Pu1QbQOPf7SNSYPjuGh4XwCmZdgxBj4vrrS4OqV8gwZ6N12VlYwx8LZeHO2WJz7extH6Jh66dDgiAsDIpGhiI0NZUajj6Eo5QwO9m1JiIzh7UBzz15TqnPQzVLDvCP/5ajffm9CfjL5R32wPChLOSYtnZVEFLS36Z6tUVzTQXSAnO5ndB4/x9c4qq0vxOcYYHn53M716hHDPBenfet6RbqfyaANb9mkPeqW6ooHuAjNG9KVnmE0bdp2BDzbtZ/WOKu69IJ2YiNBvPX9Omh3Q6YtKOUMD3QUiQm1cPDKRxRv3UVvfZHU5PqOusZlHFxcwtG8U143v3+k+9qgwRiT10nF0pZygge4iOdnJHGto5v2N+6wuxWe8sHIHZYeP89Clw7EFn/xH0ZFuJ3/PIY7UNXqwOqV8jwa6i2QN6E1qfKS2AnDS3sPHeW55MbNG9uXswXGn3NeRnkBzi+ELnb6o1ClpoLuIiHBVVjJf76xi98Faq8vxeo99sBVj4IFZw7rcd2z/GKLCbDqOrlQXNNBdaM64JIIE3tKz9FP6emcV767fy48cg0nuHdHl/iHBQUweEs+KwgqdGqrUKWigu1BidA+mpNl5a02Zzps+ieYWw28XbaZfdDg/dgx2+vscGXb2VtdRVH7UjdUp5ds00F0sJyuZssPH+WL7QatL8Upv5pawZd8R7p81jB6hwU5/nyO9bfqiznZR6qScCnQRmSEihSJSLCL3dfJ8fxH5VETWisgGEZnl+lJ9wwWZfegVbmNefonVpXid6mON/PWjQsYPjOWSUYmn9b39YnqQ3qenjqMrdQpdBrqIBAPPAjOBTOA6EcnssNtvgLnGmLHAtcBzri7UV4SHBHPZmH4s2bRfp9l18NSybRw61sCDl2Z+06/ldDjS7Xy9s4pjDTrXX6nOOHOGPh4oNsbsMMY0AG8AszvsY4BebV9HA3tdV6LvyclKob6phffW65z0E4oO1PDql7u59qz+jEiKPqPXcKQn0NDcwuodOpylVGecCfQkoP34QWnbtvZ+C3xPREqB94G7OnshEblVRPJEJK+iwn8/Oo9Kjia9T08ddmljjOGR97YQERrMzy/8dr8WZ2UP7E2PkGAdR1fqJFx1UfQ64GVjTDIwC/hfEfnWaxtjXjDGZBtjsu12u4sO7X1OzElfu+cwxeU1VpdjuaUF5XxWVMk956cT1zPsjF8nPCSYswfH6Ti6UifhTKCXASntHie3bWvvJmAugDHmSyAciHdFgb7q8rFJBAcJ8/M7/lEFlvqmZn6/eAtpCT35/tkDuv16jnQ7uw4eY1elLt5SqiNnAj0XSBORVBEJpfWi56IO++wBzgMQkWG0BnpAn0YlRIUzPcPO22tKaWpusbocy/zz853sPniMBy/NJOQU/VqcdWL64sqigP7xUqpTXf4LM8Y0AXcCHwIFtM5m2Swij4jIZW273QvcIiLrgdeBHxhd0sdVWSmU19TzWVFg9iA5cKSOZz4p5oLMPt+0we2ugfGRDIiL0HF0pTphc2YnY8z7tF7sbL/twXZfbwEmu7Y033fu0ARiI0OZl1/C9KEJVpfjcX/6YCtNzYbfXNx1v5bT4Ui3My+vlPqmZsJszi9OUsrf6UpRNwq1BTF7TD+WbinnUG2D1eV41Jo9h3h7bRk3nZPKgLhIl762I93O8cZm8nYdcunrKuXrNNDdLCcrhYbmFhatD5yp+S0thocXbSYhKow7pg9x+eufPTiO0OAglheWu/y1lfJlGuhultmvF8P79QqoOelvrSllfWk1980cSs8wp0b1TktEqI3xqbE6fVGpDjTQPSAnK5lNZUcoCIAbHdfUNfKnJYWM7R/D5WM6rj9zHUe6nW0HjrL38HG3HUMpX6OB7gGXjUkiJFiYl+f/fdKf+aSYyqP1/PbS4QQFnX6/Fmc5MtqmL+pZulLf0ED3gNjIUM4f1oeF68poaPLfOek7Ko7y0qqd5GQlMzolxq3HSkvoSWJ0uA67KNWOBrqH5GQnU1XbwKd+fCHv94sLCLMF84sZGW4/lojgSLfzeVEljQG8cEup9jTQPWRqmh17VJjfDrt8WljOJ1vLuevcISREhXvkmI50OzX1TawrOeyR4ynl7TTQPcQWHMScsUl8WlhORU291eW4VENTC797dwup8ZHcODnVY8edNCSe4CDRVaNKtdFA96Cc7GSaWwwL1/pXw65Xv9zFjspa/uuSYYTaPPcjFd0jhHH9Y3QcXak2GugeNCQhijEpMczPL/Wbu9dX1NTz9NIipmXYOXdoH48f35FuZ2NZNZVH/etTj1JnQgPdw3Kykyk8UMPGsmqrS3GJv35YyPHGZv7rko53JfQMR3prj5zPtPuiUhronnbJqH6E2YL84uLoxtJq5uaXcOPkgQy297SkhuH9ehHfM1TH0ZVCA93jonuEcNHwvryzroy6xmaryzljxhh+++5m4iJDueu8NMvqCAoSpqbZWVlUSUuLfwxjKXWmNNAtkJOdzJG6JpYWHLC6lDP2zrq95O8+xC8uyqBXeIiltTgy7FTVNvjNMJZSZ0oD3QKTBsfTLzrcZ4ddauubeOyDAkYmRZOTldL1N7jZlCHxiKCzXVTA00C3QHCQcGVWMp8VVbC/us7qck7bc8uLOXCknt9elunWfi3OiusZxqikaA10FfA00C1y5bhkWkxrq1lfsufgMV78bCeXj+lH1oBYq8v5hiPdzto9h6g+1mh1KUqdVGNzCze9nMvnbrotpQa6RQbGRzJ+YCxv+dic9Eff34ItSLhvpmtvK9ddjgw7LQY+Lw7M+7cq37C8sIJlW8vdNiFCA91CV2Uns6OyljV7fONWaquKK/lw8wHumD6EvtGe6dfirNHJMfQKt7Fim/82P1O+783cEuxRYUzLcM1N0zvSQLfQxSMTiQgN9omLo03NLTz87mb6x0Zw0xTP9Wtxli04iHPS7KzYVuFTn3hU4CivqePTwnKuHJeMLdg90auBbqHIMBszRyTy3oZ9HGtosrqcU/r36t1sO3CUX188jPCQYKvL6ZQj3c6BI/UUHqixuhSlvmXBmjKaWww52cluO4YGusVyspM5Wt/Ekk37rS7lpKpqG3ji421MHhLHhZme79firKnprR9jddWo8jbGGN7MKyF7QG+3rqrWQLfYhNRY+sdGMD/fe4ddHv+okNqGZh66dDgi1k9TPJm+0eEM7Rul0xeV11mz5xA7Kmq5+iz3rtvQQLeYiHBVVjJfbD9ISdUxq8v5li17j/D613v4/sQBpPeJsrqcLjnS7eTuqqK23ruHsFRgeTO3hIjQYC4emejW42ige4E545IQ8b456ScuhEb3COGe89OtLscpjgw7jc2GL7YftLoUpYDWldXvbdjHJaMSiQyzufVYGuheILl3BJMGxzE/v9RrGkxtKD3M5c+t4qudVfzioqFER1jbr8VZ2QNiiQgN1umLymss3riPYw3NXOPm4RbQQPcaOVkplB46zlc7qyyt40hdIw+9s4nZz66i/Eg9z3xnLNeNt75fi7NCbUFMGhzP8kKdvqi8w9zcEgbZIxnXv7fbj6WB7iUuGt6XqDAb8/JLLDm+MYbFG/Zx/uMreHX1bq6fOICl9zq4ZFQ/r74Q2hlHhp3SQ8fZWVlrdSkqwG2vOEre7kNck53ikX9HGuheokdoMJeMTuSDjfs56uELeiVVx7jx5VzueG0N9qgwFt4+mYdnj7C8Le6ZcqS1TV/U2S7KYvPySgkOEq4Yl+SR4zkV6CIyQ0QKRaRYRO7r5PknRWRd269tInLY5ZUGgKuyUjje2MziDXs9cryGphaeW17MBU+uIHdnFQ9eksk7d0xmdEqMR47vLv3jIhgUH6mBrizV1NzCW2tKmZ6RQEKUZ1pldHnJVUSCgWeBC4BSIFdEFhljtpzYxxhzT7v97wLGuqFWvzeufwyD7JHMzy/lmrP6u/VYubuq+PWCjWw7cJQZw/vy0GWZJEb3cOsxPWlqup03cvdQ19jstStblX9bXlhBRU29Ry6GnuDMGfp4oNgYs8MY0wC8Acw+xf7XAa+7orhAIyLkZKWQu+uQ28Z/D9U28Kv5G8h5/ktq65v55w3ZPP/9LL8Kc2gdR69rbOFriy8yq8A1N6+E+J7ua8TVGWcCPQlof6WutG3bt4jIACAV+OQkz98qInkikldRoR+HOzNnXBJBAvNdfHHUGMNb+aWc98QK5q8p5UeOQXz8s6mcN8x7l/J3x8TUOEJtQTrsoixRUVPPJ1vLuXJcEiFuasTVGVcf6VpgvjGm02a/xpgXjDHZxphsu91z/2v5kj69wpmabuet/NZGPq5QXH6U615czb3z1jMwLoLFd0/h/pnDiAh17yIHK/UIDWZCaqwGurLEgrWlNLUYcrI9O+XXmUAvA9pXldy2rTPXosMt3ZaTlcL+I3XdvllDXWMzT3xUyMynV7Jl7xH+cMVI5t82iaF9e7moUu/mSLdTXH6U0kPe11JB+S9jDG/mlpA1oDdDEtzXiKszzgR6LpAmIqkiEkpraC/quJOIDAV6A1+6tsTAc35mAjERId1q2PVZUQUznlrJ3z4p5pJR/Vh27zS+M6G/V9wD1FNOjF2u3KZ3MVKes2bPYbZX1HK1G9vknkyXgW6MaQLuBD4ECoC5xpjNIvKIiFzWbtdrgTeMLs/rtjBbMLNH9+PDzftP+x6Z5TV13P36Wr7/z68REf5z8wSevGYM9qgwN1XrvQbbe5IU04PlhdoGQHnOvLy2Rlyj+nn82E4Nohpj3gfe77DtwQ6Pf+u6stRVWSm88uVuFm3Yy/cnDuhy/5YWw2tf7+FPS7ZS39jCT85L48fTBgf0lD0RwZFhZ9G6vTQ0tRBq03V0yr1q65t4d/1eLh6ZSE83N+LqjP6Ee6kRSb0Y2jeK+Xldz3bZsvcIVz7/Bb9ZuImRSdF88NNzuOeC9IAO8xMc6XaO1jf5zH1blW97f+M+ahua3d73/GQ00L3UiT7p60ur2XaSW6rV1jfx6OItXPrM5+w5eIwnrxnNf26e4NY7oviaSYPjsAWJznZRHjEvr5RB8ZFkD3B/I67OaKB7scvHJmELkk4vjn685QAXPLGCFz/bydXZySy718EVY5N9rpGWu0WFh5A1oLfelk653Y6Ko3y9q4ocDzXi6owGuheL7xnG9KEJvL2mjMbmFgD2Hj7Ora/mccureUSFhzD/trN5bM4oYiJCLa7Wezky7GzZd4TyI3VWl6L82Lz81kZcV3qoEVdnNNC9XE5WMpVH61lWUM4/PtvB+U+sYGVRBffNHMp7d08he2Cs1SV6PUfbzaNXFun0ReUeTc0tvJVfyvQMOwm9PNOIqzP+u1TQT0wfmkB8z1DufG0NTS2Gc4cm8PBlw0mJjbC6NJ+RmdgLe1QYK7ZVcFWW5+cGK/+3YlsF5TX1Hl8Z2pEGupcLCQ7ixsmpvJG7hwdmDmPGiL46Tn6aRISpaXaWbT1Ac4shOIAWVynPaG3EFcq5QxMsrUOHXHzAHdOH8Nkvz2XmyEQN8zPkyLBz+FgjG0oPW12K8jMVNa1DonPGJXu0EVdnNNBVQDhnSDxBoncxUq63cG0ZTS3GkqX+HWmgq4DQOzKU0SkxGujKpYwxzM0rYVz/GIYkRFldjga6ChyOdDvrSg5zqLbB6lKUn1hbcpii8qNcbfHF0BM00FXAcKTbMQY+62ZbYqVOmJdXQo+QYC4elWh1KYAGugogo5JjiIkI0VWjyiWONTTx7vp9XDwqkajwEKvLATTQVQAJDhLOSbOzYlsFLS66G5S/aGpbiayc9/7G/Rytb/Ka4RbQQFcBxpFup/JoPQX7j1hditeYm1vC2Ec+1js7naa5eSWkxkdy1kBrGnF1RgNdBZSpafGATl884UhdI39cspWa+iae/XS71eX4jJ2VtXy9s4qcbO9qiKeBrgJKQq9wMhN76Th6m+eXb6eqtoHJQ+KYl1dCSZWepTtjXl4JQQJXjrN+7nl7Gugq4Dgy7OTvPkRN3end3s/flB0+zj8/38kVY5N4PGcMQUHCM58UW12W12tqbuGtNaVMz0igj4WNuDqjga4CjiPdTlOL4YvtB60uxVJ//bAQA/z8ogz6RofznfH9mb+mlN0Ha60uzautLKrgwBHrG3F1RgNdBZxx/XvTM8wW0OPoG0urWbC2jJumpJIU0wOA26cNxhYk/LeepZ/S3NxS4iKtb8TVGQ10FXBCbUFMGhzHisIKjAm86YvGGB59fwuxkaH8eNrgb7Yn9ArnexMHsGBtGTsr9Sy9MweP1rO04ABzxiV55U3Hva8ipTxgWkYCZYePs70i8IJrWUE5q3dUcc/5afTqsCDmNsdgQoKF/15WZFF13m1BWyMubxxuAQ10FaCmprdOX1xeWG5xJZ7V2NzCHz4oYJA9kmvH9//W8/aoMK4/eyAL15WxveKoBRV6L2MMb+aWMCYlhvQ+1jfi6owGugpIyb0jGJLQM+DG0d/4eg87Kmq5f+awk/buvnXqIMJswXqW3sH60mqKyo9yzVneeXYOGugqgDnS7Xy1s4rjDc1Wl+IRR+oaeXJpERNSYzl/2Mkv6MX3DOP6SQN4Z/1eistrPFihd3szt7UR1yVe0oirMxroKmA50u00NLWwemdgTF88sYjo1xcP63J144+mDiYiJJinl+mMF4DjDc28u34vs0Z6TyOuzmigq4A1PjWW8JCggFg12n4R0ajkmC73j40M5YZJA3lvw162HdCz9A827WtrxOVdK0M70kBXASs8JJiJg+JYGQDj6O0XETnrlnMGERlq4+mlOpb+Zm4JA+MiGJ8aa3Upp6SBrgKaI93Ojspa9hz03x4mnS0ickbvyFBunDyQxRv3sTWAu1Puqqzlq51V5GSneFUjrs5ooKuA5ki3A7CiyD/P0k+2iMhZN08ZRFSYjac+Dtyz9Hn53tmIqzNOBbqIzBCRQhEpFpH7TrLP1SKyRUQ2i8hrri1TKfdIjY8kJbaH346jn2oRkTOiI0K4cUoqSzbvZ/PeajdU6N2aWwzz80txpNvpG+1djbg602Wgi0gw8CwwE8gErhORzA77pAH3A5ONMcOBn7q+VKVcT0RwpNv5YnslDU3+ddeerhYROeumKalEhQfmWPqJRlzePPe8PWfO0McDxcaYHcaYBuANYHaHfW4BnjXGHAIwxgTW8jvl0xzpCRxraCZvd5XVpbiUM4uInBHdI4Sbpwzioy0H2FQWWGfpc3NLiI0M5dyhfawuxSnO/C0nASXtHpe2bWsvHUgXkVUislpEZnT2QiJyq4jkiUheRYV/fsRVvmfS4DhCgsWvVo06u4jIWTdOGUivcBtPLd3mgup8w4lGXFeM9c5GXJ1xVZU2IA2YBlwHvCgiMR13Msa8YIzJNsZk2+12Fx1aqe6JDLNx1sBYvxpHP51FRM7oFR7CrVMHsbSgnA2lh7tfoA9YuG4vjc3Gq24C3RVnAr0MaP+Oktu2tVcKLDLGNBpjdgLbaA14pXyCI93O1v017K+us7qUbjvdRUTOumHSQGIiQnjyY/8/SzfGMDe3hNEpMWT09c5GXJ1xJtBzgTQRSRWRUOBaYFGHfRbSenaOiMTTOgSzw3VlKuVejozWT4z+sMjoTBYROSMqPIRbzhnEp4UVrN1zyKWv7W02lFZTeKDG61eGdtRloBtjmoA7gQ+BAmCuMWaziDwiIpe17fYhcFBEtgCfAr8wxgRGgwzlFzL6RNGnV5jPj6Of6SIiZ90waSCxkaE86eczXubmlRAeEsSlo/tZXcppsTmzkzHmfeD9DtsebPe1AX7W9kspn3Ni+uKSTftpam7B1o1ZIVbp7iIiZ/QMs3Hr1EH88YOt5O8+RNaA3m45jpWONzSzaN1eZo1IPKO5+1byvZ9apdzEkZ7Akbom1vvoRb/uLiJy1vVnDyAuMtRvZ7ws2byPmvomrvaRueftaaAr1WbKkHiCBJ+c7eKqRUTOiAi1cZtjMJ8VVZK7y7/m7kNrI64BcRFM8PJGXJ3RQFeqTXRECGP79/bJcXRXLSJy1vcmDiC+Z5jfzXjZfbCW1TuqyMlK9vpGXJ3RQFeqHUe6nQ1l1ew9fNzqUpzm6kVEzugRGsyPpw3mi+0HWb3Df+Y/zM8vbW3EleVbs1tO0EBXqp2LRyUSbgvm6v/50mduv+bqRUTO+u6E/iRE+c9Z+olGXFPT7SRGu36GkCdooCvVzmB7T9780UTqGluY89wXfLG90uqSTsldi4icER7Sepb+1c4qr/9zcsZnRRXsq67zqZWhHWmgK9XBqOQYFt4xib7R4dzw0te8lV9qdUkn9bibFhE567rx/enTK4ynPi6idfay75qXV0psZCjnD/ONRlyd0UBXqhPJvSOYd9skxqfGcu+89Tz58TavC6yNpdW87cZFRM4IDwnmjulD+HpXFauKfXcsvaq2gY+27OfyMb7TiKszvlu5Um4W3SOEf/1gPDlZyTy9rIh75673mp7pnlhE5KxrzkohMTqcJ5d63396zlq4tqy1EddZvnkx9AQNdKVOIdQWxJ+vGsXPL0zn7bVlXP/SV1Qfa7S6LI8tInJGmK31LD1/9yFWFvneWLoxhrl5JYxOjmZo315Wl9MtGuhKdUFEuPPcNJ6+dgxrdh9mzt9XUVJl3U2lPbmIyFlXZ6eQFNPDK4emurKxrJqt+2vI8eGLoSdooCvlpNljkvjfm8ZTebSBK55bZVnHQU8vInJGqC2IO6YPYV3JYZb72MKsuXklhNmCuGyMbzXi6ox3/DQo5SMmDIrj7dsnERFq49oXVrNk0z6PHt+KRUTOuiormeTevnWWXtfYzDvr9jJrpO814uqMBrpSp2mwvScLbp9EZr9e/Pg/a3hx5Q6PBZhVi4icEWoL4q5zh7ChtJpPtvrGbYWXbNpPTV0TOT7W9/xkNNCVOgNxPcN4/ZaJzBzRl0ffL+C/3tlEU7N7Z8BYuYjIWXPGJdM/NsJnZrzMzSuhf2wEE1PjrC7FJTTQlTpD4SHBPHPdOH7kGMS/V+/hllfzqK1vctvxrF5E5IyQ4Naz9E1lR/h4ywGryzmlPQeP8cX2g+RkJRMU5F2fds6UBrpS3RAUJNw/cxiPXjGClUWV5Dz/pVvuS+oNi4icdcXYJAbGRfDk0iJaWrz3LH1+fgniw424OqOBrpQLfHfCAP5xQza7D9ZyxXOrKNh3xGWv7U2LiJxhCw7irnPTKNh3hI+27Le6nE5904grzU4/L/8P8nRooCvlItMzEph32ySMgZznv3RZX3VvWkTkrNlj+jEoPpKnvPQsfWnBAfb6eCOuzmigK+VCmf16seCOSaTERvDDl3N57as93Xo9b1xE5AxbcBB3n5fG1v01LNnsPWfp+6qP84t567nt3/kkxfTg/EzvmvrZXRroSrlYYnQP5t12NlOGxPPAgo388YOtZ3yW6o2LiJx16eh+DLZH8tTSbZafpR+pa+RPS7Yy7S/LeWfdXm6eksriu6cQZgu2tC5X862fEKV8RM8wG/+8IZvvTujP8yu2c9cba6lrbD6t1/DmRUTOCA4SfnJ+OtsOHGXxRs8uwDqhvqmZlz7fiePPn/L35duZNTKRZfc6+PXFmcREhFpSkzvZrC5AKX9lCw7i95ePoH9sBI99sJX91XW8eH02sZHOBYk3LyJy1sUjE/nvZUU8vayIWSMTCfbQ9MCWFsPijfv484dbKak6zuQhcdw/cxgjkqI9cnyr6Bm6Um4kIvzIMZjnvjuOTWXVzHluFTsra7v8Pl9YROSM1rP0NIrLj/Lehr0eOeaX2w9y+XOruOv1tUSG2njlh+P5900T/D7MQQNdKY+YNTKR126ZyJG6Jq54bhW5u6pOub8vLCJy1qwRiWT0ieLpZUU0u3EsvXB/DT98OZfrXlxNZU09j+eMZvHd5+BIt/vsJ5zTpYGulIdkDejNgtsnERsRyndf/Ip31pV1up8vLSJyRlCQ8NPz09hRUcui9Z2/5+7YX13HL+evZ+bTK8ndVcV9M4fyyc+ncWVWsseGeLyFjqEr5UED4iJ5+/ZJ3PpqPj95Yx2lh45z+7TB35xB+toiImddNLwvwxJ78fTSIi4d1Q+bC2bsHKlr5H9WbOefn++kpQV+ODmVO6YPobeT1yj8kZ6hK+VhMRGh/O/N45k9ph9/+bCQ+97aSGNbYy9fXETkjBNn6bsOHmPhuu6NpTc0tfCvVa0zV579dDszhvdl2b0OfnNJZkCHOegZulKWCLMF89Q1YxgQG8HfPilmb/Vx/nbtWJ9cROSsCzP7MLxfL/62rIjZY/qd9rx6Y9pmriwpZE/VMSYNbp25MjLZ/y92OksDXSmLiAg/uzCD5NgIHnh7I9MfX87hY428eH22zy0icoaIcM/56dz8ah4L1pRx9VnOL7tfveMgj71fwPrSaob2jeLlG88KqIudznLqp0ZEZohIoYgUi8h9nTz/AxGpEJF1bb9udn2pSvmnq7NTeOWH42luNkwaHOeTi4icdd6wBEYlR/O3T4q+GWY6lW0Harjp5VyufWE15TX1/OWqUSy++xymZSRomHeiyzN0EQkGngUuAEqBXBFZZIzZ0mHXN40xd7qhRqX83uQh8Xz+q3MJtQX5dVCJtI6l//DlPObnl3LdSYaW9lfX8eTH25iXX0JkqI1fzRjKjZMHEh7iX0v1Xc2ZIZfxQLExZgeAiLwBzAY6BrpSqhuiI/znIuipTM9IYHRKDM98UsyV45IJtf3/gYKaukb+Z8UO/vH5DppbDD+YlMqd5w5xenVtoHNmyCUJKGn3uLRtW0dXisgGEZkvIp0OjonIrSKSJyJ5FRW+dWdwpZRrtI6lp1F2+Djz8lujpaGphZdX7cTxl+U882kxF2b2ZdnPpvHgpZka5qfBVRdF3wVeN8bUi8iPgFeAczvuZIx5AXgBIDs72/uaJCulPMKRbmds/9az9J5hNp74eBu7Dx7j7EFx3D9rqE+3O7CSM2foZUD7M+7ktm3fMMYcNMbUtz38B5DlmvKUUv5IRPjZBensq67jJ2+sI9wWzL9uPIvXbpmgYd4Nzpyh5wJpIpJKa5BfC3yn/Q4ikmiMOdEf8zKgwKVVKqX8zpQh8dx7QTp9o8OZMy7wlum7Q5eBboxpEpE7gQ+BYOAlY8xmEXkEyDPGLALuFpHLgCagCviBG2tWSvkBEeGu89KsLsOviDHWDGVnZ2ebvLw8S46tlFK+SkTyjTHZnT3nf8vRlFIqQGmgK6WUn9BAV0opP6GBrpRSfkIDXSml/IQGulJK+QkNdKWU8hOWzUMXkQpg9xl+ezxQ6cJyfIG+58Cg7zkwdOc9DzDG2Dt7wrJA7w4RyTvZxHp/pe85MOh7Dgzues865KKUUn5CA10ppfyErwb6C1YXYAF9z4FB33NgcMt79skxdKWUUt/mq2foSimlOtBAV0opP+FzgS4iM0SkUESKReQ+q+txNxFJEZFPRWSLiGwWkZ9YXZMniEiwiKwVkfesrsUTRCSm7QbrW0WkQETOtromdxORe9p+pjeJyOsiEm51Ta4mIi+JSLmIbGq3LVZEPhaRorbfe7vqeD4V6CISDDwLzAQygetEJNPaqtyuCbjXGJMJTATuCID3DPATAutWhk8DS4wxQ4HR+Pl7F5Ek4G4g2xgzgta7oV1rbVVu8TIwo8O2+4Blxpg0YFnbY5fwqUAHxgPFxpgdxpgG4A1gtsU1uZUxZp8xZk3b1zW0/kNPsrYq9xKRZOBiWm847vdEJBqYCvwTwBjTYIw5bGlRnmEDeoiIDYgA9lpcj8sZY1bSelvO9mYDr7R9/QpwuauO52uBngSUtHtcip+HW3siMhAYC3xlcSnu9hTwS6DF4jo8JRWoAP7VNsz0DxGJtLoodzLGlAF/BfYA+4BqY8xH1lblMX2MMfvavt4P9HHVC/taoAcsEekJvAX81BhzxOp63EVELgHKjTH5VtfiQTZgHPB3Y8xYoBYXfgz3Rm3jxrNp/c+sHxApIt+ztirPM63zxl02d9zXAr0MSGn3OLltm18TkRBaw/w/xpi3ra7HzSYDl4nILlqH1M4VkX9bW5LblQKlxpgTn7zm0xrw/ux8YKcxpsIY0wi8DUyyuCZPOSAiiQBtv5e76oV9LdBzgTQRSRWRUFovoiyyuCa3EhGhdWy1wBjzhNX1uJsx5n5jTLIxZiCtf7+fGGP8+szNGLMfKBGRjLZN5wFbLCzJE/YAE0Ukou1n/Dz8/EJwO4uAG9q+vgF4x1UvbHPVC3mCMaZJRO4EPqT1qvhLxpjNFpflbpOB7wMbRWRd27YHjDHvW1eScoO7gP+0najsAG60uB63MsZ8JSLzgTW0zuRaix+2ABCR14FpQLyIlAIPAX8E5orITbS2EL/aZcfTpf9KKeUffG3IRSml1ElooCullJ/QQFdKKT+hga6UUn5CA10ppfyEBrpSSvkJDXSllPIT/w+gAVN6+JklTwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "position1_last_layer_attention_vecs = y.attentions[11][0][0][1][range(0,11)]\n",
    "position1_last_layer_attention_vecs = position1_last_layer_attention_vecs.cpu()\n",
    "plt.plot([np.linalg.norm(v) for v in position1_last_layer_attention_vecs])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec3cc6a9",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "0001da2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228dad50cd0>]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAARoUlEQVR4nO3df4xlZ13H8fdnty3+oFJhl1K6W7boEl2MSDOpTSCI4YdtE7oaf6RNCGga+g9VDGhSgqlN/QuINSFWdIkNP6LUir/WuKQiVjHG1k6llG6blnUBu2ulAzSFiNCW+/WPe2b23unMzpzdu3vPOX2/munee+7pzPfJ2Xz6zHOe5zypKiRJ/bdl3gVIkmbDQJekgTDQJWkgDHRJGggDXZIG4ox5/eBt27bVrl275vXjJamX7rnnnq9W1fa1PptboO/atYvFxcV5/XhJ6qUkX17vM4dcJGkgDHRJGggDXZIGwkCXpIEw0CVpIDYM9CS3JHksyf3rfJ4kH0hyKMl9SS6afZmSpI1spof+YeDS43x+GbC7+boG+ODJlyVJamvDQK+qzwBfP84pe4GP1tidwDlJzptVgZLUNfd8+XEefPQb8y7jGWYxhn4+8MjE+yPNsWdIck2SxSSLS0tLM/jRknT63fi3B7npUw/Pu4xnOK03RatqX1UtVNXC9u1rrlyVpM578rvF098dzbuMZ5hFoB8Fdk6839Eck6RBqiq6uNfbLAJ9P/CWZrbLJcATVfXoDL6vJHVSFYw6mOgbPpwryceB1wLbkhwBfhs4E6Cq/hA4AFwOHAK+BfzKqSpWkrqgKLq4H/OGgV5VV23weQFvn1lFktRxHcxywJWiktTaqIpRB1PdQJeklopu9tINdElqaXxTtHuJbqBLUktVZQ9dkoZgVA65SNIgVPNP1xjoktTSaNTNhUUGuiSdgC4uLDLQJaml8Tz0eVfxTAa6JLVURQdH0A10SWptVN18louBLkktuVJUkgZi/Dz07iW6gS5JLVWNpy52jYEuSS2NBrxjkSQ9q4zH0LsX6Qa6JLVUPstFkobBDS4kaShcWCRJw2APXZIGolb+1S0GuiS1ZA9dkgbCh3NJ0kC4SbQkDUThJtGSNAhuEi1JA1E+D12ShmHkTVFJ6r/lnrk3RSWp55ZzvIN5bqBLUhvLOT7qa6AnuTTJQ0kOJblujc8vSHJHks8muS/J5bMvVZLm79jN0O4l+oaBnmQrcDNwGbAHuCrJnlWn/RZwW1W9ErgS+INZFypJXbDcM+9rD/1i4FBVHa6qJ4Fbgb2rzingB5rXzwP+e3YlSlJ3LG8+19dpi+cDj0y8P9Icm3QD8OYkR4ADwK+u9Y2SXJNkMcni0tLSCZQrSfNVPe+hb8ZVwIeragdwOfCxJM/43lW1r6oWqmph+/btM/rRknT6HJvl0r1E30ygHwV2Trzf0RybdDVwG0BV/RvwPcC2WRQoSV2yPP+8g3m+qUC/G9id5MIkZzG+6bl/1Tn/BbwOIMmPMg50x1QkDU5357hsItCr6mngWuB24EHGs1kOJrkxyRXNae8C3pbkc8DHgV+uLv4+Ikkn6VgPvXsRd8ZmTqqqA4xvdk4eu37i9QPAq2ZbmiR1z7PhpqgkPSss98yrg4MuBroktWAPXZIGYuUpiwa6JPXbsYdzdS/RDXRJamFllsuc61iLgS5JbayMoXcv0g10SWphZWFR9/LcQJekNiZ75l1bXGSgS1ILkxnesTw30CWpjckeetfG0Q10SWphqoc+vzLWZKBLUguTgW4PXZJ6bPIZLh3LcwNdktro4jNclhnoktRCeVNUkoZh5LRFSRoKe+iSNAgjpy1K0jBMzUMfza+OtRjoktTC1LNcOtZHN9AlqYXphUXzq2MtBroktTC9sKhbiW6gS1IL9tAlaSCmH87VrUQ30CWphekNLuZYyBoMdElqYTLDDXRJ6jGnLUrSQHhTVJIGotwkWpKGofdj6EkuTfJQkkNJrlvnnF9K8kCSg0n+dLZlSlI3jEbdneVyxkYnJNkK3Ay8ATgC3J1kf1U9MHHObuDdwKuq6vEkLzxVBUvSPE1meB8fn3sxcKiqDlfVk8CtwN5V57wNuLmqHgeoqsdmW6YkdcP0LJdu2Uygnw88MvH+SHNs0suAlyX51yR3Jrl0rW+U5Joki0kWl5aWTqxiSZqnqVku3Yr0Wd0UPQPYDbwWuAr4UJJzVp9UVfuqaqGqFrZv3z6jHy1Jp0/ft6A7CuyceL+jOTbpCLC/qp6qqi8CDzMOeEkalL4/bfFuYHeSC5OcBVwJ7F91zl8z7p2TZBvjIZjDsytTkrph+uFc3bJhoFfV08C1wO3Ag8BtVXUwyY1JrmhOux34WpIHgDuA36yqr52qoiVpXibHzbs2hr7htEWAqjoAHFh17PqJ1wW8s/mSpMHq/cIiSdJY+fhcSRqGehZMW5SkZ4WuPWFxkoEuSS1Uh2+KGuiS1ELfFxZJklbYQ5ekQRj1eWGRJOmYqZWi9tAlqb9GzkOXpGGY3uBibmWsyUCXpBbcJFqSBmJ6pej86liLgS5JLUxvQdetRDfQJamFqVGWbuW5gS5JbXhTVJIGwiEXSRoKb4pK0jCMnLYoScPgFnSSNBCOoUvSQEwtLBrNr461GOiS1MLU0v851rEWA12SWpieh96tSDfQJamF0cjH50rSIEzPculWohvoktSCW9BJ0kCUOxZJ0jBMPw+9W4luoEtSC5OLiboV5wa6JLUy2Sn3pqgk9djUTdFu5fnmAj3JpUkeSnIoyXXHOe/nk1SShdmVKEndMTnk0rsx9CRbgZuBy4A9wFVJ9qxx3tnAO4C7Zl2kJHVF9byHfjFwqKoOV9WTwK3A3jXO+x3gvcC3Z1ifJHXK5Lh573rowPnAIxPvjzTHViS5CNhZVX93vG+U5Joki0kWl5aWWhcrSfM26IVFSbYANwHv2ujcqtpXVQtVtbB9+/aT/dGSdNr1fZbLUWDnxPsdzbFlZwM/BvxTki8BlwD7vTEqaYhGPV8pejewO8mFSc4CrgT2L39YVU9U1baq2lVVu4A7gSuqavGUVCxJczT9+Ny5lbGmDQO9qp4GrgVuBx4Ebquqg0luTHLFqS5QkrqkOrwF3RmbOamqDgAHVh27fp1zX3vyZUlSN/V92qIkqTE9ht6tRDfQJamFWud1FxjoktTCZA991LG7oga6JLUx5IVFkvRs0utpi5KkYyaHWbwpKkk9NnVTtFt5bqBLUhujKrZuCdC9hUUGuiS1UAVNnjuGLkl9VlVsSdNDN9Alqb8KVoZc+rjBhSSpMapia9ND7xoDXZJaqIItyz30jg2iG+iS1MKomJjl0i0GuiS14k1RSRqE0Why2mK3Et1Al6QWarKHPudaVjPQJamFUbOwaEt8losk9VoVJCGJY+iS1GdFkUBwDF2Sem3cQ4ctiWPoktRnK89yiT10Seq1UY2HW7aEzk1zMdAlqYViPNwSYg9dkvpsVOMu+nja4ryrmWagS1Ib1fTQEze4kKQ+G1URxjNd3IJOknpsedpicMhFknpt1Exb3LIlLv2XpD5bjvAOzlrcXKAnuTTJQ0kOJblujc/fmeSBJPcl+XSSl8y+VEmav+WFReObot2K9A0DPclW4GbgMmAPcFWSPatO+yywUFU/DnwCeN+sC5WkLji29L+fY+gXA4eq6nBVPQncCuydPKGq7qiqbzVv7wR2zLZMSeqG5TF06Oe0xfOBRybeH2mOredq4JNrfZDkmiSLSRaXlpY2X6UkdURxrIfetVH0md4UTfJmYAF4/1qfV9W+qlqoqoXt27fP8kdL0mlx7Hno4+3ouuSMTZxzFNg58X5Hc2xKktcD7wF+qqq+M5vyJKlblhcWjR+f278e+t3A7iQXJjkLuBLYP3lCklcCfwRcUVWPzb5MSeqOLSsbXMy7kmkbBnpVPQ1cC9wOPAjcVlUHk9yY5IrmtPcDzwX+PMm9Sfav8+0kqddGVZ3dgm4zQy5U1QHgwKpj10+8fv2M65KkTqpmk+i4SbQk9dt4DD1uQSdJfbfycC576JLUb5NPW+zdTVFJ0jFF87RFh1wkqd9GTQ+d0L+Hc0mSjqmJm6Jd66Ib6JLUwmhqDL1biW6gS1IL44dzNWPo3cpzA12S2hhvcDHupdtDl6QeqxoPt8RZLpLUb8sbXAQXFklSr61sQbeln1vQSZIayzdFQw83iZYkHVMrG1x0bhq6gS5JbYwfnzue5tKxDrqBLkltjDe4cGGRJPVeQfNwrnlX8kwGuiS1MGomoifeFJWkfqtjPfSO5bmBLkltjJpZLk5blKSeG89DX96Cbt7VTDPQJamFlaX/Brok9dvyw7nGW9B1K9ENdElqYfwslzSPz513NdMMdElqoZqFReMNLrqV6Aa6JLUwKlYWFdlDl6QeK8abRLvBhST1XNX4WehbQuemuRjoktTCeJglzcO55lzMKga6JLUy3iTaaYuS1HOjOrZSdDSadzXTNhXoSS5N8lCSQ0muW+Pz5yT5s+bzu5LsmnmlktQBtbJStIc3RZNsBW4GLgP2AFcl2bPqtKuBx6vqh4HfA94760IlqQtGzUrR8T3RbkX6GZs452LgUFUdBkhyK7AXeGDinL3ADc3rTwC/nyR1Clp7292P8KF/OTzrbytJm/LNbz9FErYkHF76X95w0z+3/h6/9rrdvOkVL555bZsJ9POBRybeHwF+cr1zqurpJE8ALwC+OnlSkmuAawAuuOCCEyr4nO87k93nPveE/ltJOlkvO/ds3vSKF/ON/3uKLSd4F/J533vmbItqbCbQZ6aq9gH7ABYWFk6o9/7Gl7+IN778RTOtS5JOxE//yAvnXcKUzfz/5Siwc+L9jubYmuckOQN4HvC1WRQoSdqczQT63cDuJBcmOQu4Eti/6pz9wFub178A/OOpGD+XJK1vwyGXZkz8WuB2YCtwS1UdTHIjsFhV+4E/Bj6W5BDwdcahL0k6jTY1hl5VB4ADq45dP/H628AvzrY0SVIbrhSVpIEw0CVpIAx0SRoIA12SBiLzml2YZAn48gn+59tYtQp1gIbexqG3D2zjEHSxfS+pqu1rfTC3QD8ZSRaramHedZxKQ2/j0NsHtnEI+tY+h1wkaSAMdEkaiL4G+r55F3AaDL2NQ28f2MYh6FX7ejmGLkl6pr720CVJqxjokjQQvQv0jTas7qMkX0ry+ST3Jllsjj0/yaeSfKH58wfnXWcbSW5J8liS+yeOrdmmjH2guab3JblofpVv3jptvCHJ0eZa3pvk8onP3t208aEkPzOfqjcvyc4kdyR5IMnBJO9ojg/iOh6nff29hlXVmy/Gj+/9T+ClwFnA54A9865rBu36ErBt1bH3Adc1r68D3jvvOlu26TXARcD9G7UJuBz4JON9dy8B7pp3/SfRxhuA31jj3D3N39fnABc2f4+3zrsNG7TvPOCi5vXZwMNNOwZxHY/Tvt5ew7710Fc2rK6qJ4HlDauHaC/wkeb1R4CfnV8p7VXVZxg/G3/Sem3aC3y0xu4Ezkly3mkp9CSs08b17AVurarvVNUXgUOM/z53VlU9WlX/0bz+JvAg4/2DB3Edj9O+9XT+GvYt0NfasPp4F6AvCvj7JPc0G2kDnFtVjzav/wc4dz6lzdR6bRradb22GXK4ZWKorNdtTLILeCVwFwO8jqvaBz29hn0L9KF6dVVdBFwGvD3JayY/rPHve4OaXzrENjU+CPwQ8BPAo8DvzrWaGUjyXOAvgF+vqm9MfjaE67hG+3p7DfsW6JvZsLp3qupo8+djwF8x/jXuK8u/rjZ/Pja/CmdmvTYN5rpW1Veq6rtVNQI+xLFfyXvZxiRnMg67P6mqv2wOD+Y6rtW+Pl/DvgX6Zjas7pUk35/k7OXXwBuB+5neePutwN/Mp8KZWq9N+4G3NLMkLgGemPiVvldWjRn/HONrCeM2XpnkOUkuBHYD/36662sjSRjvF/xgVd008dEgruN67ev1NZz3Xdm2X4zvpD/M+A7ze+Zdzwza81LGd84/BxxcbhPwAuDTwBeAfwCeP+9aW7br44x/XX2K8Vjj1eu1ifGsiJuba/p5YGHe9Z9EGz/WtOE+xgFw3sT572na+BBw2bzr30T7Xs14OOU+4N7m6/KhXMfjtK+319Cl/5I0EH0bcpEkrcNAl6SBMNAlaSAMdEkaCANdkgbCQJekgTDQJWkg/h+hVyaxm9r8IQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(y.attentions[11][0][0][0][0].cpu())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae362d4",
   "metadata": {},
   "source": [
    "----\n",
    "Scratchpad Below\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30580053",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "58dace40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228a8dd2610>]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEDCAYAAAAcI05xAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgAklEQVR4nO3deXhc1X3/8fdX0kiakWRpRpJXyQsGTFiCkwhICGXJQiFJgSSQpaQtLU/J1qRJf0lKk5Y0a2lJoE3TJnGW8mtC0iy/AE8gZQkBQkISkNm8gVlk40XYkizJ1r6d3x/3jjy2ZWskzdXcufN5PY8e3Zm5M+c7M1cfnTn33DvmnENERMKrJN8FiIjIsSmoRURCTkEtIhJyCmoRkZBTUIuIhJyCWkQk5AILajP7jpntNbONWa7/DjPbbGabzOz7QdUlIlJoLKh51GZ2LtAH/Ldz7tRp1j0B+BHwOudct5ktdM7tDaQwEZECE1iP2jn3K2Bf5nVmttrM7jKz9Wb2kJmd5N/0l8B/OOe6/fsqpEVEfPM9Rr0O+JBz7lXAx4D/9K8/ETjRzH5jZr8zs4vmuS4RkdAqm6+GzKwaOBv4sZmlr67IqOME4HygCfiVmZ3mnOuZr/pERMJq3oIar/fe45xbO8VtO4HfO+dGgTYz24oX3I/OY30iIqE0b0Mfzrn9eCF8BYB5Tvdvvg2vN42ZNeANhbwwX7WJiIRZkNPzfgD8FlhjZjvN7GrgSuBqM3sS2ARc6q9+N9BlZpuB+4GPO+e6gqpNRKSQBDY9T0REckNHJoqIhFwgOxMbGhrcypUrg3hoEZFIWr9+fadzrnGq2wIJ6pUrV9La2hrEQ4uIRJKZbT/abRr6EBEJOQW1iEjIKahFREJOQS0iEnIKahGRkFNQi4iEnIJaRCTkQhPU4xOOr/7yWR7c2pHvUkREQiU0QV1aYqz71Qv8YvOefJciIhIqoQlqgOZUgh3dA/kuQ0QkVMIV1MkEO/YpqEVEMoUqqJuScXZ2D6JTr4qIHBSqoG5OJRgem6CjbzjfpYiIhEbIgjoOwI59g3muREQkPMIV1MkEADu1Q1FEZFKogrrJD2rtUBQROShUQR0vL6WhukJDHyIiGbIKajP7qJltMrONZvYDM6sMqqCmZFxzqUVEMkwb1Ga2DPgw0OKcOxUoBd4VVEHNqQQ7u9WjFhFJy3boowyIm1kZkAB2B1VQczLO7p5Bxic0l1pEBLIIaufcLuBLwItAO9DrnLvn8PXM7BozazWz1o6O2Z9YqTmVYGzC0d6rXrWICGQ39JEELgVWAUuBKjN7z+HrOefWOedanHMtjY1TfuN5VponZ34oqEVEILuhjzcAbc65DufcKPBT4OygCpo86EU7FEVEgOyC+kXg1WaWMDMDXg9sCaqgJbVxSgx2ai61iAiQ3Rj174GfAI8BG/z7rAuqoPKyEhYvqGSHZn6IiADebI5pOec+DXw64FomNaUSOoxcRMQXqiMT07zzUqtHLSICYQ3qVJw9B4YYHhvPdykiInkXzqBOJnAOdmmcWkQkpEGd8udSK6hFRMIa1OkvENAORRGRUAb1wppKYqWmkzOJiBDSoC4tMZbV6XSnIiIQ0qAG/3SnGvoQEQlvUDclE9qZKCJCiIO6ORVnX/8I/cNj+S5FRCSvwhvU6dOdapxaRIpcaIO6KelN0dupQ8lFpMiFNqgPHvSiHrWIFLfQBnV9VTnxWKlOziQiRS+0QW1mNKc0l1pEJLRBDenTnSqoRaS4hTuoUwl2dg/inMt3KSIieRPqoG5KxukbHqNnYDTfpYiI5E3Ig9qb+aGTM4lIMQt1UE+e7lQ7FEWkiIU8qP251NqhKCJFLNRBvaAyRm08ph61iBS1UAc1eMMfOuhFRIpZ+IM6mVCPWkSKWviDOpVgl+ZSi0gRC31QNyXjDI9N0HFgON+liIjkReiDWuelFpFiF/6gTs+l1g5FESlSoQ/q9NGJmkstIsUq9EFdGSulsaZCQx8iUrSmDWozW2NmT2T87Dezj8xDbZOak5pLLSLFq2y6FZxzzwBrAcysFNgF3BpsWYdqSiZ4fEf3fDYpIhIaMx36eD3wvHNuexDFHE1zKs7uniHGxifms1kRkVCYaVC/C/jBVDeY2TVm1mpmrR0dHXOvLENzMsH4hKO9dyinjysiUgiyDmozKwcuAX481e3OuXXOuRbnXEtjY2Ou6gP0jeQiUtxm0qO+GHjMObcnqGKOJn3Qy07tUBSRIjSToH43Rxn2CNqSukpKTD1qESlOWQW1mVUBbwR+Gmw5U4uVlrCkNq6v5BKRojTt9DwA51w/UB9wLcfUlIzr6EQRKUqhPzIxrTml81KLSHEqnKBOJtizf5ih0fF8lyIiMq8KJ6j9s+jt6tE4tYgUlwIKap1FT0SKU+EE9eQXCKhHLSLFpWCCemFNBeWlJezUDkURKTIFE9QlJcayZFxHJ4pI0SmYoAZ/LrV61CJSZAoqqJtTCe1MFJGiU1hBnUzQPTBK3/BYvksREZk3hRXUk99Irl61iBSPwgrq9OlONUVPRIpIQQV1U1I9ahEpPgUV1KmqchLlpZr5ISJFpaCC2sxoTibYobnUIlJECiqowduhqKMTRaSYFFxQNyW9udTOuXyXIiIyLwouqJtTCfpHxukeGM13KSIi86Lwgtqf+aHhDxEpFgUX1E3p051qh6KIFImCC+rJoxPVoxaRIlFwQV1TGaMuEdNBLyJSNAouqME7lFzf9CIixaIwgzoVZ6d61CJSJAozqJMJdvYMMjGhudQiEn0FGdRNqQQjYxN09A3nuxQRkcAVZlDrLHoiUkQKMqjT56XWFD0RKQYFGdQHe9Sa+SEi0VeQQV0ZK2VhTYWGPkSkKBRkUIP/jeQa+hCRIpBVUJtZnZn9xMyeNrMtZvaaoAubTnMyru9OFJGikG2P+t+Au5xzJwGnA1uCKyk7TckE7b1DjI1P5LsUEZFATRvUZlYLnAt8G8A5N+Kc6wm4rmk1p+KMTzjae4fyXYqISKCy6VGvAjqA/zKzx83sW2ZWdfhKZnaNmbWaWWtHR0fOCz3c5BQ97VAUkYjLJqjLgFcCX3POvQLoB649fCXn3DrnXItzrqWxsTHHZR6pOaW51CJSHLIJ6p3ATufc7/3LP8EL7rxaUltJaYlpLrWIRN60Qe2cewnYYWZr/KteD2wOtKoslJWWsKS2Ul/JJSKRV5bleh8CbjGzcuAF4M+DKyl7Oi+1iBSDrILaOfcE0BJsKTPXlIzz4Nbgd1yKiORTwR6ZCN4Oxb0HhhkaHc93KSIigSnwoPZOzqQjFEUkygo7qHW6UxEpAoUd1P5can1/oohEWUEHdWN1BeVlJRr6EJFIK+igLikxmpJxDX2ISKQVdFCDdxY9HZ0oIlFW8EHdrB61iERc4Qd1KkHPwCgHhkbzXYqISCAKP6gnT3eq4Q8RiabCD+rJg140/CEi0VT4QT150It61CISTQUf1HWJGFXlpfqmFxGJrIIPajOjOZXQ0IeIRFbBBzVoLrWIRFskgro55c2lds7luxQRkZyLRlAnEwyMjNM9oLnUIhI90Qjq9DeSa4eiiERQRILam0utQ8lFJIoiEdRNOjpRRCIsEkFdXVFGMhFTj1pEIikSQQ3eOLXGqEUkiqIT1MmEvulFRCIpMkHdlIqzq3uQiQnNpRaRaIlMUDcnE4yMT7D3wHC+SxERyanoBHV6LrV2KIpIxEQmqJuS/lxq7VAUkYiJTFAvq0sHtXYoiki0RCaoK2OlLFpQoaEPEYmcyAQ1pKfoKahFJFqiFdQpnZdaRKInq6A2s21mtsHMnjCz1qCLmq3mZJz23kFGxyfyXYqISM6UzWDdC5xznYFVkgNNyQQTDtp7hlhen8h3OSIiORGpoY8mne5URCIo26B2wD1mtt7MrplqBTO7xsxazay1o6MjdxXOQHNSXyAgItGTbVCf45x7JXAx8EEzO/fwFZxz65xzLc65lsbGxpwWma0ltZWUlph61CISKVkFtXNul/97L3ArcGaQRc1WWWkJS+sqdRY9EYmUaYPazKrMrCa9DFwIbAy6sNlqTuq81CISLdn0qBcBvzazJ4FHgDudc3cFW9bsNScT7FCPWkQiZNrpec65F4DT56GWnGhKxuk4MMzQ6DiVsdJ8lyMiMmeRmp4HB093qkPJRSQqIhjUOoueiERL9II6qR61iERL5IK6saaCirIS7VAUkciIXFCbGU3JuKboiUhkRC6owT/dqYY+RCQiIhnUXo9aQx8iEg2RDOrmZILewVH2D43muxQRkTmLZlCndBY9EYmOaAb15BQ9DX+ISOGLZlBPHvSiHrWIFL5IBnVtPEZNRZl61CISCZEMajNjmeZSi0hERDKoQXOpRSQ6ohvUyQQ79g3inMt3KSIicxLdoE7FGRwdZ1//SL5LERGZk+gGdfobybVDUUQKXHSDWge9iEhERDaom5L+XGrtUBSRAhfZoK6qKCNVVa6TM4lIwYtsUAM0J+P6phcRKXiRDuqmVEJj1CJS8CId1M3JBLt6BpmY0FxqESlc0Q7qVJzRcceeA0P5LkVEZNaiHdTpudTaoSgiBSzaQa251CISAZEO6qV1lZhpLrWIFLZIB3VFWSmLaio19CEiBS3SQQ3eDkXNpRaRQhb9oE4m9E0vIlLQsg5qMys1s8fN7I4gC8q1plSC9t5BRscn8l2KiMiszKRH/dfAlqAKCUpzMs6Eg9096lWLSGHKKqjNrAl4M/CtYMvJvSbNpRaRApdtj/pfgU8ABTd+0JzS6U5FpLBNG9Rm9hZgr3Nu/TTrXWNmrWbW2tHRkbMC52pJbZyyEtNBLyJSsLLpUb8WuMTMtgH/A7zOzL53+ErOuXXOuRbnXEtjY2OOy5y90hJjaV1cMz9EpGBNG9TOub9zzjU551YC7wJ+6Zx7T+CV5VBzKq6hDxEpWJGfRw3eXGrtTBSRQjWjoHbOPeCce0tQxQSlOZWgs2+YwZHxfJciIjJjRdGjTn/RrQ4lF5FCVCRB7c+lVlCLSAEqiqBOz6XWzA8RKURFEdSN1RVUxko0l1oCNTQ6zn/c/xx/cfOjPNK2L9/lSISU5buA+WBmNGnmhwRkYsJx+5O7uOGuZ9jdO0RtPMY7vvFb3n3mcq69+CRq47F8lygFriiCGryTM2mMWnLtkbZ9fP7OzTy1s5fTltVy4zvX8vKmWm66dyvf/nUb923Zw2cuOYWLTl2MmeW7XClQxRPUqQTrt3fnuwyJiLbOfq7/3y3cvWkPS2orufEdp3PZ2mWUlHhh/Kk3n8wlpy/j2p8+xftveYw3nryIz116KotrK/NcuRSiognqpmSc/UNj9A6O6qOozFrPwAhfue85vvu7bZSXlvCxC0/k6nOOI15eesS6pzXVcvsHX8u3f93GTb/YyhtufJC/vWgNV561YjLQRbJRNEHdnDz4jeS1y2rzXI0UmpGxCf77t9v4918+x4GhUd55RjMffeOJLKw5dg+5rLSE9563motOXcynbt3IP9y+idue2M31bzuNExbVzFP1UuiKJ6hTXlDv7B7kVAW1ZMk5x92bXuL6/32abV0D/MEJDXzqzS/jpMULZvQ4K+qr+O7VZ/LTx3bxuTs386avPMT7zz+eD16wmoqyI3vjIpmKJ6iT6aDWDkXJzpM7evjCnVt4ZNs+TlxUzc1/fgbnr1k468czM97+qibOX9PI5+7YzFfue5Y7n9rN9W9/OWesTOWwcomaognq2kSMmsoyzaWWae3qGeSGu57mtid201BdzhffehrvaGmirDQ3hx3UV1fwr+96BZe9YhmfunUjV3z9t/zxWd5UvgWV2n8iRyqaoAb/LHo6OlGOom94jK898BzfeqgNgA9esJr3nbeamoDC8/w1C7n3b87lxnu28p3ftPGLzXv47KWncNGpSwJpTwpXUQV1UzJOW2d/vssQvKP4uvpH2Nc3Qlf/MPuHxlhUU8Gqhioaayrmdc7x2PgEP2zdwU33bqWzb4TL1i7l4xedxLK6eOBtJ8rL+Pu3nMwla5dy7f/bwPu+9xgXnryIz2oqX6iNjk+wvWuA5zv6vJ+9/Tzf0cf4hONnHzon5+0VVVA3pxI89GwnzjkdfJBDzjkGRsbZ1z/ihW//MF19I+zrH8m47tDbBo5xytmq8lJWNVaxsr6K4xqqMparqU3ktnf7wDN7+eLPt7B1Tx9nrkzx7T97Gac31+W0jWy8vKmO2//Kn8p371beeOODfOLik7jyzOWRnso3MjbB0Ng41eVloXyevYOjvNDRx/Md/Ty3t28ymF/sGmBswk2ut2hBBasbqzlxUU0g+VJcQZ2MM+j35BqqK/JdTuj1D4+xvWuAXT2DXsD6PeB06Hb1D/s94hGGx6b+3uPyshLqq8pJ+T+r6hOkqiqorz54XX1VOQviMV7qHaKts3/y56mdvfx8QzsZfw8kEzFWNVSxqqGaVQ0J/3cVKxsSJMqz35yffmk/X7hzCw8928mK+gRff88r+cNT8nv0YKy0hPedt5qLT13MJ2/dwD/ctpHbH9/FP4V0Kt/EhKN/ZIwDQ2PsHxr1fg+OHnF5f8blA0Ojh6wzNOptN2UlRtLfFhqqve2j3t9OGg5Z9n7P5L3O5nns7h3k+Y5+ns8I4+c7+uk4MDy5XqzUWFFfxQkLq7n41MWsbqxmdWM1xzVWBTY8lmbOuenXmqGWlhbX2tqa88edq/u27OHq/9vKrR84m1csT+a7nFDoHRhl+75+tnUNsL3T+/2ifzlzI02Lx0q9cD0saFNVFQcDudq7rr66gqry0jmF38jYBC/uG2CbH94vdPZPLr+0f+iQdRcvqGSlH97HNVSxsqGKVQ1VLE8lKC/zdgTuPTDETfdu5YeP7qCmMsaHX38Cf/LqFZO3h4Vzjp+s38kXfr6F/uExPnD+8Xwg4Kl8Q6Pj7O4ZpL13iN09g7zUO0RX/8gRYesF7SgHhseYLj7Ky0pYUBljQWUZNXHv94JKb8f+gniMmooyKmOl9AyO0NU3Qqc/FNbVN0JX3zD9R/nkFY+VemFe7W136e2tobr8sJCvIJkop7yshKHRcdo6+w8Zqnhubx8vdPZN/sMAWFBZxvELvRBenf7dWEVzKkEsRzuUp2Jm651zLVPdVlw96lT6vNSDRRPUzjm6+kfY3tXP9q4BL5C7+id/9wyMHrL+4gWVLK9PcMGaRlbUe0MOTcn45MY/1RF4QSovK+H4hdUcv7D6iNsGRsbY1jlAW2c/27r6eaGjn7bOPu7e9BL7+kcm1ysx75zkK+oTPLa9m5HxCa46exUffv3x1CXK5/PpZM3MuKKlmQtOWshnf7aZf7vvWe7c0M4/ve20WU3lGx4bZ0/vMLt7B2nvHWR3zxDtvV4Yp5e7D9sWAC9QK2NeqFaWsawuzssW10xePiR0K8uoSYeyf31lbG7bS3pfRlffsB/kw4de7h9hz/4htrTvp6tvhJHxqT/Z1VSW0Zfxj8UMltXFOX5hNa9ZXT8ZxqsXVlNfVR66odGi6lEPjIxx8nV38/E/XMMHLzg+3+XkzMSEY++BYbZ19U+G8ItdA/7lAfqGxybXLTFYWhdnZX0VK+oT/o8XyMtTiXkP4qD0DozS1uUFd1tHP21dA7R19rGivoqPXbiGVQ1V+S5xRu5/Zi9/f+tGdvUMcuVZy/nbjKl8Y+MT7DkwTHvPILt7h2jP6BW39w7R3jtEZ9+Rn45q4zGW1FaytC7OktpK/yfOkrpKltbGWVxbOeegnU/OOQ4Mj032xjN75/v6R6hLxCaHK1Y1VIVuW1eP2pcoL6OhunxOB7045+gfGadnYISegVG6B0boHhilZ2CE7n7vcs/ACL2Do4eMrQJM9S8x23+UU63mcHQcGObFfQOHfHQrKzGWpxIsr09wxsoUK+oTXhDXJ2hKxoviSLjaRIy1iTrW5mHHYBAuWLOQez56Ll++Zys3P9zGPZv3sKwuTnvvIB0Hho/Y1qoryrzgrYtzytIFLK71wndJnR/GtZVUVUTrz9/M/GGWWMH9I55OtN6pLCzLOC/12PgEPYN+yA6M0t0/RfgesjxK78DoUT9eAdRUlFFXFaM2HqO05MjxrKk+UE31KWvq9Y68dkV9Feed2Mjy+ipW+oG8pLYyZwdnSHhUVZRx3R+dzKVrl/Kle55hwjn+4ITGI3rCS+oqdeBMxBTV0AfAX33/Me7a+BLx8lIODI0ddb1YqVGXKCeZiE3+TibKD1uOHbJOXSIW6M4GEYkuDX1k+JNXryBWWkJtPEZdRuAmE+UHl6vK5zxbQUQkV4ouqM86rp6zjqvPdxkiIlnT53QRkZBTUIuIhJyCWkQk5BTUIiIhp6AWEQk5BbWISMgpqEVEQk5BLSIScoEcQm5mHcD2Wd69AejMYTn5amO+2olKG/PVTlTamK92otLGfLUzlzZWOOcap7ohkKCeCzNrPdrx7oXUxny1E5U25qudqLQxX+1EpY35aieoNjT0ISIScgpqEZGQC2NQr4tIG/PVTlTamK92otLGfLUTlTbmq51A2gjdGLWIiBwqjD1qERHJoKAWEQm5QILazG4ys49kXL7bzL6VcfnLZvY3R7nvVWa2dJrHv8rMvpqzgg8+7mVmdnKuH3eKdvqCbiOLGj6ZsbzSzDbmsZafm1ldvtqfrZm+j2Z2vpnd4S/PehsO4vUKcnswsyvMbIuZ3W9mLWb2lWnWn3ydprhtm5k15Kq22fBr2OD/bDazz5tZZZBtBtWj/g1wNoCZleBNAj8l4/azgYePct+rgGMGdYAuAwIP6pD45PSrZMfM5vRNQc65NznnenJUTuQF9HrlbHuYwtXAXzrnLnDOtTrnPhxgW/PlAufcacCZwHHAN4JsLKigfhh4jb98CrAROGBmSTOrAF4GXGhmj5rZRjNbZ57LgRbgFjN7wsziZnaGmT1sZk+a2SNmVuM/7lIzu8vMnjWzf5mqCDN7t/9fb6OZ/XPG9X0Zy5eb2c1mdjZwCXCD3/bqnL8qh9Z2k5ltMrP7zKzRv261/5zWm9lDZnaSf/0iM7vVfw2e9GvFzG7z191kZtcc6/n5y1f4r8UeoMp/nt8H/g5YbWZdZrbbzO4xs/g0Nd1sZl83s98D/3K09bJ8LbaZWYOZVZnZnf5z3Ghm75zjy3x4O18zs1b/9fpMjh5zqvfxATNr8ZcbzGxbLtrKaDP9eq30e6rf9GuYfN9m+HjXA3F/e7jFv7p0qsed6ftsZtcB5wDfNrMb7NBPFVVm9h3/7/pxM7t0ivvX++1vMu9Teai+yNQ51we8D7jMzFJ+jt3gb78b0tuw/7wfMLOfmNnTZnaL2Qy+lNU5F8gP0AYsB97rP5HPAW8CXgs8BKQy1v0u8Ef+8gNAi79cDrwAnOFfXoD3PY9X+dfXApV4h6s3H9b+UuBFoNG/zy+By/zb+jLWuxy42V++Gbg8qNcko00HXOkvXwd81V++DzjBXz4L+KW//EPgI/5yKVDrL6f833G8f4b10zy/DcCyzHWAa4AvAWPAGUArcCfwnmlquhm4Ayg91npZvh7b8D51vR34Zsb1tTl+3dOvV6m/nb08oPcxcxtuALb5y+cDd/jLV6XXn0W76ddrpf++rfWv/1H6fZvFY2ZuM0d93Nm8z4e9HpmvwRczHrcO2ApUHbbOV4Dr/OU3+695Qy63i9m+/odd94T/erwduNffxhbhZdAS/zn1Ak14HeTfAudk22aQX277MN4Qx9nAjcAyf7kXb2jkAjP7BJAAUsAm4GeHPcYaoN059yiAc24/kP528Pucc73+5c3ACmBHxn3PAB5wznX469wCnAvcluPnORsTeOEL8D3gp2ZWjff6/DjjH22F//t1wJ8COOfG8V5DgA+b2Vv95WbgBKDrGO3+BrjZzH6Ucd2FwCuBceCbeP/8ngBWTlMTwI+dc+NZrJetDcCXzfv0c4dz7qFZPMaxvMP/5FGG98dzMvDUHB7viPdxbuXNSptz7gl/eT1eyAbyuDl8n9MuBC4xs4/5lyvxOneZzgXeBuCcu9PMuufQXpDSL8g5wA/8v9M9ZvYgXhbtBx5xzu0EMLMn8N6rX2fz4EEGdXqc+jS83t4O4P/gFfxfeKHQ4pzbYWb/iPcmzcRwxvI4M3sumZPHA90JkCWH91+2xzm3Nps7mNn5wBuA1zjnBszsAQ4+lymfn3PufWZ2Fl7PJGFm9Xgb2D8Cn0i37f/hVGdRU7//e0a1H41zbquZvRLvk9fnzew+59xn5/KYaWa2CvgY3qezbn84KNfvffp1H+PgsGLQ29fhfwczHvqYwePm5H3OYMDbnXPPHHKl2aIcPf68MG84diXeJ4JjmXVmBTk972HgLcA+59y4c24f3seb13BwR2Kn/1/68oz7HQDS49DPAEvM7AzwXhDLfsfVI8B5/lheKfBu4EH/tj1m9jLzdnS+NeM+mW3PmT9uuWyKm0o4+Jz/GPi1/2mhzcyu8O9rZna6v859wPv960vNrBav59vth/RJwKszHn/K52dmq51zv3fOXYfXG1wF3A28J2OdE/GGnJimpknZrjcd82b7DDjnvgfcgNfTz5UFeP9Yev0guDgHj3nE++gvbwNe5S9fTmEYNbPYsVbI1fuc4W7gQ+mxWjN7xRTr/ArvtcXMLgaSc2gv5/z8+k/gNudcN96w7jv9v9NGvE8Ej8y1nSCDegPeONrvDruu1znXidej3oj3Zj2asc7NwNf9jwalwDuBfzezJ/HGfrLqoTjn2oFrgfuBJ4H1zrnb/ZuvxRtffRhoz7jb/wAf93dszGlnoh+SxwP7pri5HzjTvClQrwPSvcYrgav957oJSO9c+Wu8oaINeB9DTwbuAsrMbAtwPYe+zkd7fjf4Ozg24g1vfBc4D3gWON6//hscul0crabDZbvesZwGPOK/958GPj+Lx5iSc+5J4HHgaeD7eJ/45upo7+OXgPeb2eN4fwOFYB3wlB3cmXg0uXif0z4HxPx2N/mXD/cZ4Fz/9rfhjfnOmHlTGnM5m+x+/31/xK/pvf71t+INpz2Jt1/sE865l+bamA4hD4iZnQr8hXNuyvniIiLZUlCLiIScDiEXEQk5BbWISMgpqEVEQk5BLSIScgpqEZGQU1CLiITc/wfRk23a2Ns3CwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def compare(position):\n",
    "    return torch.dot(y['hidden_states'][11][0][0], y['hidden_states'][11][0][position]).cpu().detach().numpy()\n",
    "\n",
    "plt.plot([tokenizer.decode(input_ids[0][i]) for i in range(0,13)], [compare(pos) for pos in range(0,13)])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "8327c13b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 14])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "08ad9f5d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 1, 12, 14, 270])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y['attentions'][11].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "e31b3b74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228a93bfd00>]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEDCAYAAAA7jc+ZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnUklEQVR4nO3de3hcV33u8e+ru8Z25BlbcRxrbOdOEmgScBIS0iSkIQRaCJRQklIop6EplLbQAi3lPA8UaM/TkgI9FGgIJSe0TYBQCKSUS0JIyA2cyMG52Lk2N9sxtnyRbVmyrMvv/LG35LEiWSNppJFm3s/z6Jk9a6+919ozo99es/baaxQRmJlZ5aopdwXMzGx6OdCbmVU4B3ozswrnQG9mVuEc6M3MKpwDvZlZhZu1gV7StZK2SnqkyPy/I2m9pHWSbpju+pmZzRWarePoJZ0LdAH/FhEvHSfvccCNwAURsVPS4RGxdSbqaWY2283aFn1E3AnsKEyTdIykH0laI+kuSS9JV/0h8MWI2Jlu6yBvZpaatYF+DNcAfxoRrwA+BHwpTT8eOF7SPZJ+IenistXQzGyWqSt3BYolaT5wNvAtSUPJjeljHXAccD7QBtwp6WUR0TnD1TQzm3XmTKAn+fbRGRGnjrJuI7A6IvqAZyQ9QRL475/B+pmZzUpzpusmInaTBPG3AihxSrr6uySteSQtJunKeboM1TQzm3VmbaCX9HXg58AJkjZKugJ4O3CFpAeBdcAlafYfA9slrQduBz4cEdvLUW8zs9lm1g6vNDOz0pi1LXozMyuNWXkxdvHixbFy5cpyV8PMbM5Ys2bNtohoHW3drAz0K1eupL29vdzVMDObMyQ9N9Y6d92YmVU4B3ozswrnQG9mVuEc6M3MKpwDvZlZhXOgNzOrcA70ZmYVrmIC/cBg8IWfPsnPnugod1XMzGaVign0tTXimjuf5ifrt5S7KmZms0rFBHqAfC7Dhp3d5a6GmdmsUlmBPpthww4HejOzQhUV6NuyzWzc2YOnXjYzO6CiAn0+l6G3f5COrt5yV8XMbNaosEDfDMCGHT1lromZ2exRWYE+mwFgoy/ImpkNq6hA35YGel+QNTM7oKICfXNDLYvnN7rrxsyswLiBXlKTpPskPShpnaRPjJLnLyStl/SQpNskrShYNyBpbfp3c6kPYKS2bLPH0puZFSjmpwR7gQsioktSPXC3pB9GxC8K8vwSWBUR3ZLeC3waeFu6riciTi1prQ8hn8vw4IbOmSrOzGzWG7dFH4mu9Gl9+hcj8tweEUPN6F8AbSWt5QTks8280NnDwKDH0puZQZF99JJqJa0FtgK3RsTqQ2S/AvhhwfMmSe2SfiHpTYco48o0X3tHx+QnJsvnMvQPBpt3uZ/ezAyKDPQRMZB2v7QBZ0h66Wj5JP0esAq4qiB5RUSsAn4X+CdJx4xRxjURsSoiVrW2tk7kGA6SHx5540BvZgYTHHUTEZ3A7cDFI9dJuhD438AbI6K3YJtN6ePTwB3AaZOv7viGb5ryBVkzM6C4UTetkhamy83Aa4DHRuQ5DfgySZDfWpCeldSYLi8GXgWsL1ntR7G0pZkawUaPpTczA4obdbMU+JqkWpITw40R8X1JnwTaI+Jmkq6a+cC3JAE8HxFvBE4EvixpMN327yNiWgN9Q10NRxzWxIad7roxM4MiAn1EPMQo3S0R8bGC5QvH2PZe4GVTqeBktOUyngbBzCxVUXfGDknmpXeL3swMKjXQ55rZsmcfvf0D5a6KmVnZVWagz2aIgE3upzczq9BAn0vH0jvQm5lVaqAf+gESX5A1M6vIQH/4gibqa8VGt+jNzCoz0NfWiGULPV2xmRlUaKCHpJ/ed8eamVVwoG/LZnwx1syMCg70+VwzO/buZ29vf7mrYmZWVpUb6IemK3Y/vZlVuYoN9G3ZZIjlRk+FYGZVrmID/YGbptyiN7PqVrGBftG8Bprraz25mZlVvYoN9JLI5zyW3sysYgM9DE1X7EBvZtWtsgN9LsPGnT1ERLmrYmZWNhUd6NuyzXT19tPZ3VfuqpiZlU2FB/pk5I0nNzOzajZuoJfUJOk+SQ9KWifpE6PkaZT0TUlPSVotaWXBur9O0x+X9NoS1/+Qhqcr9gVZM6tixbToe4ELIuIU4FTgYkmvHJHnCmBnRBwLfA74BwBJJwGXAScDFwNfklRborqPa3gsvS/ImlkVGzfQR6IrfVqf/o28unkJ8LV0+T+B35CkNP0bEdEbEc8ATwFnlKTmRTisqZ6W5nq36M2sqhXVRy+pVtJaYCtwa0SsHpFlGbABICL6gV3AosL01MY0bbQyrpTULqm9o6NjQgdxKPlcs2+aMrOqVlSgj4iBiDgVaAPOkPTSUlckIq6JiFURsaq1tbVk+81nM27Rm1lVm9Com4joBG4n6W8vtAnIA0iqA1qA7YXpqbY0bcbkcxk2eSy9mVWxYkbdtEpamC43A68BHhuR7Wbg99PlS4GfRhJZbwYuS0flHAUcB9xXoroXpS3bTG//IB17emeyWDOzWaOuiDxLga+lo2VqgBsj4vuSPgm0R8TNwFeBf5f0FLCDZKQNEbFO0o3AeqAfeF9EDEzHgYylcF76ww9rmsmizcxmhXEDfUQ8BJw2SvrHCpb3AW8dY/u/A/5uCnWckuGx9Dt6eMWKctXCzKx8KvrOWDhwd6zH0ptZtar4QN9UX0vrgkaPvDGzqlXxgR4gn/VYejOrXlUR6NuyGTZ2ukVvZtWpKgJ9PtfMC5376B8YLHdVzMxmXHUE+myGgcFg86595a6KmdmMq45Anzswlt7MrNpUR6Af+gESX5A1sypUFYF+6cImauQWvZlVp6oI9PW1NSxtafZPCppZVaqKQA/J5Ga+O9bMqlHVBPp8zvPSm1l1qp5An82wZXcv+/pmdPJMM7Oyq55An85iuanT/fRmVl2qKNB7Fkszq07VE+iHf4DELXozqy5VE+gPX9BIQ20NG31B1syqTNUE+poasSzb7LtjzazqVE2gh3QsvVv0ZlZlxg30kvKSbpe0XtI6Se8fJc+HJa1N/x6RNCApl657VtLD6br26TiIYuVzGV+MNbOqM+6PgwP9wAcj4gFJC4A1km6NiPVDGSLiKuAqAElvAP48InYU7OPVEbGtlBWfjHw2w87uPrp6+5nfWMyhm5nNfeO26CNic0Q8kC7vAR4Flh1ik8uBr5emeqU1NJberXozqyYT6qOXtBI4DVg9xvoMcDHw7YLkAG6RtEbSlYfY95WS2iW1d3R0TKRaRRuerthDLM2sihQd6CXNJwngH4iI3WNkewNwz4hum3Mi4uXA64D3STp3tA0j4pqIWBURq1pbW4ut1oS0Zd2iN7PqU1Sgl1RPEuSvj4jvHCLrZYzotomITenjVuAm4IzJVXXqcvMayDTUeuSNmVWVYkbdCPgq8GhEfPYQ+VqA84DvFaTNSy/gImkecBHwyFQrPVmSyGczbPBYejOrIsUMPXkV8A7gYUlr07SPAssBIuLqNO3NwC0Rsbdg2yXATcm5gjrghoj4UQnqPWn5XLPvjjWzqjJuoI+IuwEVke864LoRaU8Dp0yybtOiLZvh5/+znYggPQGZmVW0qrozFpKbpvbuH2Bnd1+5q2JmNiOqL9CnI2/cfWNm1aLqAn3b0HTFviBrZlWi6gL98N2xbtGbWZWoukC/oKmehZl63zRlZlWj6gI9JFMh+JemzKxaVGegzzWz0S16M6sS1Rnosxk2dvYwOBjlroqZ2bSrykDflsuwv3+Qjq7eclfFzGzaVWeg9yyWZlZFqjLQD81L7yGWZlYNqjLQH2jRe+SNmVW+qgz0TfW1HL6g0V03ZlYVqjLQQzK5mbtuzKwaVG+gzzb7t2PNrCpUbaBvy2bYvGsf/QOD5a6Kmdm0qtpAn881MzAYbN61r9xVMTObVtUb6IenK3Y/vZlVtuoN9DmPpTez6jBuoJeUl3S7pPWS1kl6/yh5zpe0S9La9O9jBesulvS4pKckfaTUBzBZS1uaqK2Rx9KbWcUb98fBgX7ggxHxgKQFwBpJt0bE+hH57oqI3ypMkFQLfBF4DbARuF/SzaNsO+PqamtY2tLknxQ0s4o3bos+IjZHxAPp8h7gUWBZkfs/A3gqIp6OiP3AN4BLJlvZUvO89GZWDSbURy9pJXAasHqU1WdJelDSDyWdnKYtAzYU5NnIGCcJSVdKapfU3tHRMZFqTVpbttkXY82s4hUd6CXNB74NfCAido9Y/QCwIiJOAf4Z+O5EKxIR10TEqohY1draOtHNJyWfy7B1Ty/7+gZmpDwzs3IoKtBLqicJ8tdHxHdGro+I3RHRlS7/AKiXtBjYBOQLsralabPC0A+F+w5ZM6tkxYy6EfBV4NGI+OwYeY5I8yHpjHS/24H7geMkHSWpAbgMuLlUlZ8qT1dsZtWgmFE3rwLeATwsaW2a9lFgOUBEXA1cCrxXUj/QA1wWEQH0S/oT4MdALXBtRKwr7SFM3tBYev9+rJlVsnEDfUTcDWicPF8AvjDGuh8AP5hU7aZZ6/xGGupq3HVjZhWtau+MBaipUTLyxl03ZlbBqjrQQzKLpe+ONbNKVvWBPu8WvZlVOAf6XIbO7j727Osrd1XMzKaFA/3wdMXuvjGzyuRAP3zTlLtvzKwyOdAP3zTlFr2ZVaaqD/QLM/XMa6j15GZmVrGqPtBLIp/LuOvGzCpW1Qd68Fh6M6tsDvQkF2Q37OwmmZ7HzKyyONCTXJDt3j/Ajr37y10VM7OSc6CnYBZLj7wxswrkQM+BsfSeCsHMKpEDPcnFWPDdsWZWmRzogfmNdWQz9W7Rm1lFcqBP5XMZ3zRlZhXJgT6Vz2Z8MdbMKlIxPw6el3S7pPWS1kl6/yh53i7pIUkPS7pX0ikF655N09dKai/1AZRKW66ZTTt7GBz0WHozqyzF/Dh4P/DBiHhA0gJgjaRbI2J9QZ5ngPMiYqek1wHXAGcWrH91RGwrXbVLL5/NsH9gkK17ejmipanc1TEzK5lxW/QRsTkiHkiX9wCPAstG5Lk3InamT38BtJW6otNtaCy9L8iaWaWZUB+9pJXAacDqQ2S7AvhhwfMAbpG0RtKVh9j3lZLaJbV3dHRMpFol0ZZNx9L7gqyZVZhium4AkDQf+DbwgYjYPUaeV5ME+nMKks+JiE2SDgdulfRYRNw5ctuIuIaky4dVq1bNeEf5soVDgd4XZM2sshTVopdUTxLkr4+I74yR59eAfwUuiYjtQ+kRsSl93ArcBJwx1UpPh6b6WpYc1uiuGzOrOMWMuhHwVeDRiPjsGHmWA98B3hERTxSkz0sv4CJpHnAR8EgpKj4d8lmPpTezylNM182rgHcAD0tam6Z9FFgOEBFXAx8DFgFfSs4L9EfEKmAJcFOaVgfcEBE/KuUBlFI+l+G+Z3aUuxpmZiU1bqCPiLsBjZPn3cC7R0l/GjjlxVvMTvlsM99b20PfwCD1tb6XzMwqg6NZgbZshsGAzZ37yl0VM7OScaAv0Obpis2sAjnQF8gPT1fsQG9mlcOBvsDSliZqa+QWvZlVFAf6AnW1NRy5sMmzWJpZRXGgH8Fj6c2s0jjQj5DPZtjgFr2ZVRAH+hHass107OllX99AuatiZlYSDvQjDE1XvNEXZM2sQjjQj5DPeRZLM6ssDvQjDI2ld4vezCqFA/0IrQsaaayr8QVZM6sYDvQjSKIt2+whlmZWMRzoR5HPZXx3rJlVDAf6USQtenfdmFllcKAfRT6bYVdPH7v39ZW7KmZmU+ZAP4qhsfTupzezSuBAP4oDQyzdfWNmc58D/SgO3DTlFr2ZzX3jBnpJeUm3S1ovaZ2k94+SR5I+L+kpSQ9JennBut+X9GT69/ulPoDp0NJcz4LGOrfozawijPvj4EA/8MGIeEDSAmCNpFsjYn1BntcBx6V/ZwL/ApwpKQd8HFgFRLrtzRGxs6RHUWKSWOax9GZWIcZt0UfE5oh4IF3eAzwKLBuR7RLg3yLxC2ChpKXAa4FbI2JHGtxvBS4u6RFME4+lN7NKMaE+ekkrgdOA1SNWLQM2FDzfmKaNlT7avq+U1C6pvaOjYyLVmhbJD5D0EBHlroqZ2ZQUHeglzQe+DXwgInaXuiIRcU1ErIqIVa2traXe/YTlc8309A2wY+/+clfFzGxKigr0kupJgvz1EfGdUbJsAvIFz9vStLHSZ72hIZae3MzM5rpiRt0I+CrwaER8doxsNwPvTEffvBLYFRGbgR8DF0nKSsoCF6Vps55vmjKzSlHMqJtXAe8AHpa0Nk37KLAcICKuBn4AvB54CugG/le6boekTwH3p9t9MiJ2lKz206gtm46l9wVZM5vjxg30EXE3oHHyBPC+MdZdC1w7qdqV0bzGOnLzGjy5mZnNeb4z9hDy2Wb/0pSZzXkO9IfQlsu4j97M5jwH+kPIZzNs6uxhcNBj6c1s7nKgP4R8rpm+gWDLnn3lroqZ2aQ50B/C8Fh6X5A1sznMgf4QPJbezCqBA/0hHLmwCclj6c1sbnOgP4TGulqWLGhy142ZzWkO9OPI5zyW3szmNgf6ceSzGf/SlJnNaQ7042jLZdi8q4e+gcFyV8XMbFIc6MeRzzYzGPBCp1v1ZjY3OdCPo81j6c1sjnOgH0c+5+mKzWxuc6Afx9KWZupq5JumzGzOcqAfR22NOHJhs0femNmc5UBfhHyu2V03ZjZnOdAXIZ/N+GKsmc1ZDvRFyOcybOvqpWf/QLmrYmY2YeMGeknXStoq6ZEx1n9Y0tr07xFJA5Jy6bpnJT2crmsvdeVnytAPhXsqBDObi4pp0V8HXDzWyoi4KiJOjYhTgb8GfhYROwqyvDpdv2pKNS2j4bH0DvRmNgeNG+gj4k5gx3j5UpcDX59SjWahobH0HnljZnNRyfroJWVIWv7fLkgO4BZJayRdOc72V0pql9Te0dFRqmqVROv8RprqazyW3szmpFJejH0DcM+IbptzIuLlwOuA90k6d6yNI+KaiFgVEataW1tLWK2pk0SbR96Y2RxVykB/GSO6bSJiU/q4FbgJOKOE5c2ofNZj6c1sbipJoJfUApwHfK8gbZ6kBUPLwEXAqCN35oJ8LuOuG5t2/QODbOvqLXc1rMLUjZdB0teB84HFkjYCHwfqASLi6jTbm4FbImJvwaZLgJskDZVzQ0T8qHRVn1lt2WZ27+tnV08fLc315a6OVZgXOnv4xv0buPH+Dfxq9z5e99Ij+IvXHM9xSxaUu2pWAcYN9BFxeRF5riMZhlmY9jRwymQrNtvkh6cr7qZlWUuZa2OVYGAwuOPxrdyw+nluf3wrAZx7XCtvPPVIblj9PD9e9yvedOoyPnDh8SxflCl3dW0OGzfQWyKfS/7RNu7s4aUO9DYFm3f18M37N/DN+zewedc+Whc08sfnH8vbTs8Pf87ec94xXP2z/+Fr9z7LzQ++wNtOz/OnFxzHES1NZa69zUUO9EUaatH77libjIHB4M4nOrh+9fP89LEtDAb8+nGL+fgbTuI3TlxCfe3Bl8ty8xr46OtP5IpzjuILP32Kb9z/PP+5ZiPvPGsF7z3/WHLzGsp0JDYXOdAXqSVTz4KmOl+QtQnZsnvfcOt9U2cPi+c38p7zjuGy05cX1R2z5LAmPvWml3LluUfzTz95kq/e/Qw3rH6eK845inefezSHNfl6kY3PgX4C8tkMG3x3rI1jYDC468kOblj9PLc9tpWBweCcYxfzv3/zRC48cQkNdRMf7JbPZfjM75zCe88/ms/d+iSf/+lTfO3nz/FH5x3Nu85eSabB/8o2Nn86JqAt28wz2/aOn3EKnu7o4pb1W1j99HYyjXUsntfAovmNLJrfwKJ5jbQuSB4XzW9gfmMd6aimqtY/MEhHVy/ZTANN9bVlq8fW3fu4sX0DX78vab0vmtfAu3/9KC4/fTkrF88rSRnHHr6AL7795bx30y4+e+sTfPpHj3Pt3c/yJ68+hsvPXE5jXfmO36ZuYDCorSn9/7QD/QTkcxnuenIbEVGyADs4GDy4sZNb1m/h1vVbeGprFwDHHT6fgQi27ell977+UbdtqKsZPhEsnn/ghLA4PREMpS+e30huXsOL+oHniohgZ3cfG3Z083z6t2FHNxt2JssvdO5jYDAAWDy/gWXZDG3ZZtoWNieP6fNl2eaSt3wHB4O7n9rGDauf5yePbqF/MDj7mEX89etfwkUnHTGp1nsxXrqshWvfdTprntvBVT9+nL/5r/V85a5n+LPfOJa3vLyNujn6XleyiGD73v280NnDC5370sceNu/ax6Z0uaGuhrv/6oKSl+1APwH5bDM9fQNs37ufxfMbJ72f3v4Bfv4/27k1De5b9/RSWyNeeXSOd7xyBReetIRlC5uH8+/vH2TH3v1s6+pl+979bO/qZXvXfrbt7WXbnv1s39vLtq79PP6rPWzr2s/+gcFRy21prj/oRLB4fiPZTD0tmQZamutZ2FzPwkw9Lc31tKSPM9VC3Nc3wMadPQcC+PY0oKdpXb0Hn+wWz28gn8twWj7LJadkOKKlic7u/Wzq7GHjzh7Wv7CbW9dtedFrkZvXkAT9UU4CbdkM8xuL+5fo2NPLje0b+Mb9z7NhRw+5eQ38wTlHcfkZyzmqRK33YrxiRY6v/+Erueep7Vx1y+P81bcf5uqfPc2fv+Z4futlS6mZhtahjW5vbz+bdx0cxF/YdfDy/v6DP4+NdTUsW9jMkQubOe/4VpbnpmcYrSJiWnY8FatWrYr29tk3ff1tj27hiq+1c9Mfn81py7MT2nb3vj7ueLyDW9b9ijse76Crt59MQy3nn9DKa05awgUnLKElM/ULaxFBV28/27uSE0BHeiLY3pWcILYVnii6euns6eNQH4Hm+toDwT89ESxsbhg+ERyUNnSiyNSzYES30uBg0NHVm7TIh4N493Arfcvug+8GbaqvIZ/NsDyXIZ/+JcvN5LMZ5hURkAcHg21dvWzY2cPGnd3DJ4GNQ8939tA74h9vYab+oJPAyOWHN+3ihvue45Z1Sev9lUfn+N0zV/Dak5eUvdskIvjJo1v5zC2P89iv9vCSIxbwwYtO4MITD6/YLr7O7v1s3rWPCKipASFqlMxPJUGNhEgfxYG0gsehbQ7kObC+Jl0vwfa9+9nc2ZO2vvelQb2HTelyZ3ffQXWTYMmCJo5c2MTShUnj4siWguWFzWQz9SV7byStGWs6eAf6CXhiyx4u+tydfP7y03jjKUeOm/9Xu/Zx66NbuGXdr/jF09vpGwgWz2/gwhOXcNHJSzj7mMVl7VOGJBju6e1nV3cfnT372dXTR2d3H509fezu6aOzez+d3X1Jek8fu4aX97Ovb/RvDgA1Ig38DUjJ/QeFrRkJlh7WRFsawIeC+FBgb53fOO3BKSLY1rV/xEkgOQEMnRB6+l78q2ILM/Vc+vI2Lj9zOce0zp/WOk7G4GDwXw+9wD/95Eme2baXU/ML+fBrT+BVxy4ud9UmbHAw2Lqnl+e27+W5Hd3JY9pQeG57N7t6+sbfyTQ5rKmOI9OgvXRh04HllmaOXNjEksOaZrS79FCB3l03EzD0S1NjDbGMCJ7amlxMvWX9Fh7c0AnAykUZ/uBVR/Gak5Zw2vLstFxsmayaGg23zJczsa+N+/oGkpNBT9+BE0R3crIYer6rp4/+wUEuPHHJgVZ52lVS7hawJFoXNNK6oHHUb2hD1wY27uwePgksOayJ1558RNlP0IdSUyMuOXUZv/mypXz7gY383588ydv/dTVnHb2ID732BF6xYmLfRqdb38Agm3b28NyObp7fvpdnt3enwXwvz+/oPqhBUVsj2rJJg+ANpyxl5aJ5HLmwmRqJiGAwIEgfI4iAwRGPB9YPpQVBclJJtj9426F9ZjMNLG1pSgN7c9HdfLPB3KnpLJBpqGPx/IaDbpoaGAzWbtjJLeuS4D40KueUtBV10UlLOPbw+RX51bmpvpam+loOP6wy79aURG5eA7l5Dfxa28JyV2fC6mpreNvpy3nTacu4YfXzfPH2p3jLv9zLBS85nA9edDwnHzlzd3h37+8fboU/v72bZ7fvHX6+qbNn+GI6JN12K3LzWLFoHuce18qKxfNYkcuwYlGGIxc2z9lBBeXkrpsJuuSL99BYV8N7zjuaW9Zt4SePbmVbVy91NeKsYxZx0clH8JoTl/hWdZt1uvf3c929z/Llnz3Nrp4+GmprqKmBWomaGlEjUTv8eCC9tkYHltP+69qawrxD63lR2u59fTy3vZutew6+BrMwU8+KXIbliw4E8RWL5rFiUYbDF0x/t10lctdNCeWzzXz/oc3c98wO5jfWcf4JrVx08hGcf0Kr71K0WS3TUMcfn38sbz9zBd9q38D2vfsZHAwGBoOBiGQ5goFBhpeT7oxgIBgjb7p+MBgcTLphBgvWZxrqOO/41oMC+YrcvJIMPLDiOdBP0DvPWknrgkbOO76Vs45ZVPZ+ZrOJammu592/fnS5q2EzyIF+gs44KscZR+XKXQ0zs6L5qoaZWYVzoDczq3AO9GZmFc6B3syswo0b6CVdK2mrpEfGWH++pF2S1qZ/HytYd7GkxyU9Jekjpay4mZkVp5gW/XXAxePkuSsiTk3/PgkgqRb4IvA64CTgckknTaWyZmY2ceMG+oi4E9gxiX2fATwVEU9HxH7gG8Alk9iPmZlNQan66M+S9KCkH0o6OU1bBmwoyLMxTRuVpCsltUtq7+joKFG1zMysFDdMPQCsiIguSa8HvgscN9GdRMQ1wDUAkjokPTfJ+iwGtk1y29lUxkyVUyllzFQ5lVLGTJVTKWXMVDlTKWPFWCumHOgjYnfB8g8kfUnSYmATkC/I2pamFbPP1snWR1L7WBP7lMpMlDFT5VRKGTNVTqWUMVPlVEoZM1XOdJUx5a4bSUconWpO0hnpPrcD9wPHSTpKUgNwGXDzVMszM7OJGbdFL+nrwPnAYkkbgY8D9QARcTVwKfBeSf1AD3BZJHMf90v6E+DHQC1wbUSsm5ajMDOzMY0b6CPi8nHWfwH4whjrfgD8YHJVm7RrKqSMmSqnUsqYqXIqpYyZKqdSypipcqaljFn5wyNmZlY6ngLBzKzCOdCbmVW4WRnoJX1O0gcKnv9Y0r8WPP+MpL8YY9t3STpynP2/S9Ko1xWmQtKbZmKaB0ld011GEXX4aMHyyrHmQpqhuvxA0sJylT9ZE30f03mlvp8uT/ozPB2v13R+HiS9VdKjkm6XtErS58fJP/w6jbLu2XT4d9mkdXg4/Vsv6W8lTeuPTM/KQA/cA5wNIKmG5CaCkwvWnw3cO8a27wIOGein0ZtI5vWpBh8dP0txJE3pfo6IeH1EdJaoOhVvml6vkn0eRnEF8IcR8eqIaI+IP5vGsmbKqyPiZSRTxRwNfHk6C5utgf5e4Kx0+WTgEWCPpKykRuBE4CJJ90t6RNI1SlwKrAKuT2fSbJZ0uqR70yka7pO0IN3vkZJ+JOlJSZ8erRKSLk/Puo9I+oeC9K6C5UslXSfpbOCNwFVp2ceU/FU5uG6fk7RO0m2SWtO0Y9JjWiPpLkkvSdOXSLopfQ0eTOuKpO+meddJuvJQx5cuvzV9LbYA89LjvAH4a+AYSdslvSDpFknN49TpOklXS1oNfHqsfEW+Fs9KWixpnqT/To/xEUlvm+LLPLKcf1EyTcc6SZ8o0T5Hex/vkLQqXV4s6dlSlFVQ5tDrtTJtKX8lrcPw+zbB/f090Jx+Hq5Pk2tH2+9E32cls+GeA3xV0lU6+FvNPCWz694n6ZeSXjSXlqRFafnrlPQKaKLHN50iogt4D/AmSbk0jl2Vfn4fHvoMp8d9h6T/lPSYpOslFX8sETEr/4BngOXAH6UvxKeA1wOvAu4CcgV5/x14Q7p8B7AqXW4AngZOT58fRjKk9F1pegvQBDwH5EeUfyTwPNCabvNT4E3puq6CfJcC16XL1wGXzsBrE8Db0+WPAV9Il28DjkuXzwR+mi5/E/hAulwLtKTLufSxmeRkumic43sYWFaYB7gS+EegHzgdaAf+G/i9cep0HfB9oPZQ+Yp8PZ4l+db3FuArBektJX7dh16v2vRz9mvT9D4WfoYXA8+my+cD30+X3zWUfxLlDr1eK9P37dQ0/cah920S+yz8zIy538m8zyNej8LX4P8U7Hch8AQwb0SezwMfS5d/M33NF5fyczHZ139E2tr09XgLcGv6GVtCEoOWpse0i2SGgRrg58A5xZY5m38c/F6SLpqzgc+STIh2NsnB3gO8WtJfAhkgB6wD/mvEPk4ANkfE/XBguob0RHhbROxKn68nmSeicBK204E7IqIjzXM9cC7JXD7lNkgSvAH+A/iOpPkkr8+3Ck70jenjBcA7ASJigOQ1BPgzSW9Ol/MkcxRtP0S59wDXSbqxIO0i4OXAAPAVkpPnWmDlOHUC+FZEDBSRr1gPA59R8u3r+xFx1yT2cSi/k37zqSP55zsJeGgK+3vR+zi16k3KMxGxNl1eQxKkp2W/JXyfh1wEvFHSh9LnTSSNw0LnAr8NEBH/LWnnFMqbTkMvyDnA19P/0y2SfkYSi3YD90XERgBJa0neq7uL2flsDvRD/fQvI2ltbgA+SHLA/48kqKyKiA2S/obkTZ6I3oLlASb2WhTefDCtF1GKFCRn+c6IOLWYDSSdD1wInBUR3ZLu4MCxjHp8EfEeSWeStIwykhaRfED/BvjLobLTf7z5RdRpb/o4obqPJSKekPRykm9+fyvptkh/H2GqJB0FfIjk2+HOtDur1O/90Ovez4Fu1en+fI38P5hw180E9luS97mAgLdExOMHJUpLSrT/GaGkO3klyTeSQ5l0zJqtffSQtOh/C9gREQMRsYPk69lZHLgQuy1tJVxasN0eYKgf/nFgqaTTIXlBVfyFv/uA89K+zFrgcuBn6botkk5UcqH4zQXbFJY9ZWm/7WhTO9dw4Jh/F7g7/bbyjKS3pttK0ilpntuA96bptZJaSFreO9Mg/xLglQX7H/X4JB0TEasj4mMkrdGjSKa4+L2CPMeTdJkxTp2GFZtvPEpGW3VHxH8AV5F80yiVw0hOTLvSQPK6EuzzRe9juvws8Ip0+VLmhj5J9YfKUKr3ucCPgT8d6quWdNooee4keW2R9DogO4XySi6NX18CvhsRO0m6pd+W/p+2knwjuW+q5czmQP8wST/iL0ak7YqIbSQt+kdI3uz7C/JcB1ydfrWpBd4G/LOkB0n6vopqIUXEZuAjwO3Ag8CaiPheuvojJP3L9wKbCzb7BvDh9MLQlC7GpkH2WEb/0Ze9wBlKhrBdAAy1Wt8OXJEe6zoO/NDL+0m6uh4m+Rp9EvAjoE7So8Dfc/DrPNbxXZVeIHqEpHvm34HzgCeBY9P0L3Pw52qsOo1UbL5DeRlwX/refxz420nsY1QR8SDwS+Ax4AaSb5xTNdb7+I8k80f9kuR/YC64BnhIBy7GjqUU7/OQT5HMu/WQpHXp85E+AZybrv9tkj7vCVMyJLWUo/luT9/3+9I6/VGafhNJd+CDJNcF/zIifjXVwjwFwiwl6aXAH0TEqPcLmJkVy4HezKzCzeauGzMzKwEHejOzCudAb2ZW4RzozcwqnAO9mVmFc6A3M6tw/x86aeFLkhfM1wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def compare(a,b):\n",
    "    return torch.dot(y['hidden_states'][11][0][a], y['hidden_states'][11][0][b]).cpu().detach().numpy()\n",
    "\n",
    "plt.plot([tokenizer.decode(input_ids[0][i]) for i in range(0,13)], [compare(2,pos) for pos in range(0,13)])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "b3f407ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x228a9430bb0>]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAApcElEQVR4nO3de5hkVX3u8e/b96q5dTXdDEN34cA4RAdEweEioqJGGIgRE00CJyioJxw8eCGHxKPkiXiJeTzi0SM5iiFhQkxQgkeiE4MhE+Qa5DIgtwFkhpvTw8D0MPeZ7unb7/yxd3cXPd3T1d3V091V7+d56ulda6+919pV1b9atfbaeykiMDOzylA13RUwM7ODx0HfzKyCOOibmVUQB30zswrioG9mVkFqprsCB9Lc3ByLFy+e7mqYmc0qDz744JaIaBlp3YwO+osXL2bNmjXTXQ0zs1lF0gujrXP3jplZBXHQNzOrIA76ZmYVxEHfzKyCOOibmVUQB30zswrioG9mVkHKMujv6Ozh//zH0zyyYft0V8XMbEYpy6Avwf/5j3Xc++wr010VM7MZpSyD/vyGWhZkatmwbe90V8XMbEYpy6APkG/KsGFr53RXw8xsRinboN/WmHVL38xsmLIN+vmmDBu3deI5gM3MhpRx0M+yr7efjl37prsqZmYzRvkG/VwWwF08ZmYFxgz6kvKSbpP0hKS1kj6dpr9J0r2SHpa0RtJJabokXSVpvaRHJZ1QsK8LJK1LHxdM3WEl3TuAT+aamRUoZhKVXuCyiHhI0jzgQUmrga8BX4yIn0k6O31+OnAWsDR9nAxcDZwsqQm4AlgORLqfVRGxrdQHBdA20NLf6pa+mdmAMVv6EbEpIh5Kl3cBTwKtJIF7fpptAfBiunwO8L1I3As0SloEnAmsjoitaaBfDawo6dEUaKitpnluvbt3zMwKjGu6REmLgeOB+4BLgVskfZ3ky+PUNFsrsKFgs/Y0bbT04WVcBFwEcMQRR4ynevvJN2Vo3+buHTOzAUWfyJU0F/gRcGlE7AQ+DvxxROSBPwauLUWFIuKaiFgeEctbWkac17do+ZzH6puZFSoq6EuqJQn410fETWnyBcDA8g+Bk9LljUC+YPO2NG209CmTb8rw4vYuevv6p7IYM7NZo5jROyJpxT8ZEd8oWPUi8I50+V3AunR5FfDhdBTPKcCOiNgE3AKcISknKQeckaZNmXwuS19/sGlH11QWY2Y2axTTp/9W4EPAY5IeTtMuB/4I+JakGqCLtB8euBk4G1gP7AU+AhARWyV9GXggzfeliNhaioMYTb5paKz+wLKZWSUbM+hHxN2ARln95hHyB3DJKPtaCawcTwUnY+ACrfatnbDkYJVqZjZzle0VuQCLGhuokq/KNTMbUNZBv7a6ikULPGzTzGxAWQd9gLZcxlflmpmlyj7o55s8Vt/MbED5B/1clpd37qOrp2+6q2JmNu3KP+ind9vcuN39+mZmFRD0fbdNM7MBZR/023LpffU9gsfMrPyD/sJ5DdRVV9Huk7lmZuUf9KuqRGsuk1yVa2ZW4co+6EM6Vt8tfTOzygj6+aasT+SamVEpQT+XZdveHnbv653uqpiZTauKCPqDI3jc2jezClcRQX9grL5vvGZmla6YmbPykm6T9ISktZI+XbDuk5KeStO/VpD+OUnrJf1K0pkF6SvStPWSPlv6wxlZ3i19MzOguJmzeoHLIuIhSfOAByWtBhYC5wBvjIh9kg4FkLQMOBc4Bjgc+A9JR6f7+jbwHqAdeEDSqoh4orSHtL+mOXVk66o9gsfMKl4xM2dtAjaly7skPQm0kkyX+NWI2Jeu25xucg5wQ5r+nKT1DE2avj4ingWQdEOad8qDviTyuSwbPFbfzCrcuPr0JS0GjgfuA44G3ibpPkl3SDoxzdYKbCjYrD1NGy19eBkXSVojaU1HR8d4qndA+aaMr8o1s4pXdNCXNBf4EXBpROwk+ZXQBJwC/Clwo6TR5tItWkRcExHLI2J5S0vLZHc3qC2XjNVPpvA1M6tMRQV9SbUkAf/6iLgpTW4HborE/UA/0AxsBPIFm7elaaOlHxRtuQx7uvvYtrfnYBVpZjbjFDN6R8C1wJMR8Y2CVT8G3pnmORqoA7YAq4BzJdVLOhJYCtwPPAAslXSkpDqSk72rSngsBzQ0bNNdPGZWuYoZvfNW4EPAY5IeTtMuB1YCKyU9DnQDF0TSd7JW0o0kJ2h7gUsiog9A0ieAW4BqYGVErC3lwRxIPjdwX/1OjmtrPFjFmpnNKMWM3rkbGK2v/vxRtvkK8JUR0m8Gbh5PBUtlYAYtD9s0s0pWEVfkAsxrqKUxW+sLtMysolVM0Ieki8czaJlZJausoN+Uod0tfTOrYBUV9NtyWdq3ddLf77H6ZlaZKiro53MZuvv66di9b7qrYmY2LSoq6Lc1DQzbdBePmVWmigr6g2P1PWzTzCpURQX9oRm0PILHzCpTRQX9htpqDp1X7+4dM6tYFRX0IWntu3vHzCpVxQX9fFPWc+WaWcWqvKCfy7JpRxe9ff3TXRUzs4Ou8oJ+U4a+/mDTjq7proqZ2UFXeUE/57H6Zla5Ki/oN3msvplVrmJmzspLuk3SE5LWSvr0sPWXSQpJzelzSbpK0npJj0o6oSDvBZLWpY8LSn84Y1u0oIHqKnmsvplVpGJmzuoFLouIhyTNAx6UtDoinpCUB84Afl2Q/yySKRKXAicDVwMnS2oCrgCWA5HuZ1VEbCvh8YypprqKw+Y3uKVvZhVpzJZ+RGyKiIfS5V3Ak0BruvqbwGdIgviAc4DvpROm3ws0SloEnAmsjoitaaBfDawo3aEUL9+U8bBNM6tI4+rTl7QYOB64T9I5wMaIeGRYtlZgQ8Hz9jRttPThZVwkaY2kNR0dHeOpXtHyuaxP5JpZRSo66EuaC/wIuJSky+dy4POlrlBEXBMRyyNieUtLS6l3DyQnczfv2kdXT9+U7N/MbKYqKuhLqiUJ+NdHxE3AEuBI4BFJzwNtwEOSDgM2AvmCzdvStNHSD7qBSdLdxWNmlaaY0TsCrgWejIhvAETEYxFxaEQsjojFJF01J0TES8Aq4MPpKJ5TgB0RsQm4BThDUk5SjuQE8C1Tc1gH5lssm1mlKmb0zluBDwGPSXo4Tbs8Im4eJf/NwNnAemAv8BGAiNgq6cvAA2m+L0XE1olWfDIGxup7vlwzqzRjBv2IuBvQGHkWFywHcMko+VYCK8dXxdJrmVtPXU0VG9y9Y2YVpuKuyAWoqhJtjRna3b1jZhWmIoM+JPPl+qpcM6s0FRv0855MxcwqUOUG/aYs2/f2sKurZ7qrYmZ20FRu0B+8xbK7eMysclRs0G/LJRdouYvHzCpJxQb9wbH6HrZpZhWkYoN+LlvLnLpq33jNzCpKxQZ9SeSbsh6rb2YVpWKDPkBbzmP1zayyVHTQzzclY/WTO0eYmZW/yg76uSx7u/vYuqd7uqtiZnZQVHTQHxq26S4eM6sMFR30h4Zt+mSumVUGB318Va6ZVY6KDvpz62vIZWt9Va6ZVYxipkvMS7pN0hOS1kr6dJp+paSnJD0q6Z8lNRZs8zlJ6yX9StKZBekr0rT1kj47JUc0TvmmrC/QMrOKUUxLvxe4LCKWAacAl0haBqwGjo2I44Cngc8BpOvOBY4BVgDfkVQtqRr4NnAWsAw4L807rfK5rG/FYGYVY8ygHxGbIuKhdHkX8CTQGhH/HhG9abZ7gbZ0+RzghojYFxHPkcyVe1L6WB8Rz0ZEN3BDmndatTVl2Litk/5+j9U3s/I3rj59SYuB44H7hq36KPCzdLkV2FCwrj1NGy19eBkXSVojaU1HR8d4qjchbbks3X39vLyra8rLMjObbkUHfUlzgR8Bl0bEzoL0PyPpArq+FBWKiGsiYnlELG9paSnFLg8on47VdxePmVWCooK+pFqSgH99RNxUkH4h8F7gD2PoXgYbgXzB5m1p2mjp02po2KZP5ppZ+Stm9I6Aa4EnI+IbBekrgM8A74uIwoi5CjhXUr2kI4GlwP3AA8BSSUdKqiM52buqdIcyMa2N6VW5HqtvZhWgpog8bwU+BDwm6eE07XLgKqAeWJ18L3BvRFwcEWsl3Qg8QdLtc0lE9AFI+gRwC1ANrIyItaU8mIloqK1m4fx6j9U3s4owZtCPiLsBjbDq5gNs8xXgKyOk33yg7aZLPuex+mZWGSr6itwBbbmMT+SaWUVw0Cc5mbtpRyc9ff3TXRUzsynloE/SvdMfsGm7x+qbWXlz0Ce5KhfwyVwzK3sO+iQtffBYfTMrfw76wKIFDVRXyS19Myt7DvpATXUVhzc2+AItMyt7DvqptsasW/pmVvYc9FP5Jo/VN7Py56CfyueydOzaR1dP33RXxcxsyjjopwbuttnuLh4zK2MO+ql8k++2aWblz0E/NThW3y19MytjDvqplnn11NdU+QItMytrDvopSbTmMu7eMbOyVszMWXlJt0l6QtJaSZ9O05skrZa0Lv2bS9Ml6SpJ6yU9KumEgn1dkOZfJ+mCqTusicnnsrRvd0vfzMpXMS39XuCyiFgGnAJcImkZ8Fng1ohYCtyaPgc4i2SKxKXARcDVkHxJAFcAJwMnAVcMfFHMFPkmt/TNrLyNGfQjYlNEPJQu7wKeBFqBc4C/T7P9PfD+dPkc4HuRuBdolLQIOBNYHRFbI2IbsBpYUcqDmax8LsuOzh52dvVMd1XMzKbEuPr0JS0GjgfuAxZGxKZ01UvAwnS5FdhQsFl7mjZa+vAyLpK0RtKajo6O8VRv0gbG6vtkrpmVq6KDvqS5wI+ASyNiZ+G6iAggSlGhiLgmIpZHxPKWlpZS7LJoQ7dYdhePmZWnooK+pFqSgH99RNyUJr+cdtuQ/t2cpm8E8gWbt6Vpo6XPGG255AItX5VrZuWqmNE7Aq4FnoyIbxSsWgUMjMC5APhJQfqH01E8pwA70m6gW4AzJOXSE7hnpGkzRmO2lrn1Nb7xmpmVrZoi8rwV+BDwmKSH07TLga8CN0r6GPAC8PvpupuBs4H1wF7gIwARsVXSl4EH0nxfioitpTiIUpFEWy7jPn0zK1tjBv2IuBvQKKvfPUL+AC4ZZV8rgZXjqeDBlm/K8sIre6a7GmZmU8JX5A6Tz2XZsLWT5LvLzKy8OOgPk2/K0NnTxyt7uqe7KmZmJeegP8zQsE3365tZ+XHQH6Zt4L76HsFjZmXIQX+YgZa+x+qbWTly0B9mTn0NTXPqfFWumZUlB/0R5HMZt/TNrCw56I+grSnrE7lmVpYc9EeQz2XZuL2Tvn6P1Tez8uKgP4J8U4aevuDlnV3TXRUzs5Jy0B9Bm8fqm1mZctAfQX7wFssewWNm5cVBfwStuQwSbPAIHjMrMw76I6ivqWbhvAaP1TezsuOgP4p8U8YtfTMrO8XMnLVS0mZJjxekvUnSvZIeTicxPylNl6SrJK2X9KikEwq2uUDSuvRxwUhlzST5XJZ2n8g1szJTTEv/OmDFsLSvAV+MiDcBn0+fA5wFLE0fFwFXA0hqAq4ATgZOAq5Ip0ycsdpyGTbt7KK7t3+6q2JmVjJjBv2IuBMYPq1hAPPT5QXAi+nyOcD3InEv0JhOmn4msDoitkbENmA1+3+RzChtTVkiYNMO9+ubWfkoZo7ckVwK3CLp6yRfHKem6a3AhoJ87WnaaOn7kXQRya8EjjjiiAlWb/KG7qvfyWsOmTNt9TAzK6WJnsj9OPDHEZEH/hi4tlQViohrImJ5RCxvaWkp1W7HLT94X33365tZ+Zho0L8AuCld/iFJPz3ARiBfkK8tTRstfcZatCBDTZV8Va6ZlZWJBv0XgXeky+8C1qXLq4APp6N4TgF2RMQm4BbgDEm59ATuGWnajFVdJQ5vzHgGLTMrK2P26Uv6AXA60CypnWQUzh8B35JUA3SR9sEDNwNnA+uBvcBHACJiq6QvAw+k+b4UEcNPDs84+aaMW/pmVlbGDPoRcd4oq948Qt4ALhllPyuBleOq3TRra8xy61MvT3c1zMxKxlfkHkC+KcOW3d10dvdNd1XMzErCQf8A8k2eJN3MyouD/gEM3lffQd/MyoSD/gEMjtX33TbNrEw46B9Ay9x6GmqrPILHzMqGg/4BSKItl3X3jpmVDQf9MbTlMu7eMbOy4aA/hnwu69E7ZlY2HPTHkG/KsLOrlx2dPdNdFTOzSXPQH8PQLZbd2jez2c9Bfwy+QMvMyomD/hgKJ1MxM5vtHPTHMD9Tw7z6Gg/bNLOy4KA/Bkm0NWVp9331zawMOOgXIZ/zffXNrDw46Bchn7b0k+kCzMxmrzGDvqSVkjZLenxY+iclPSVpraSvFaR/TtJ6Sb+SdGZB+oo0bb2kz5b2MKZWPpehs6ePLbu7p7sqZmaTUkxL/zpgRWGCpHcC5wBvjIhjgK+n6cuAc4Fj0m2+I6laUjXwbeAsYBlwXpp3VhgYtumTuWY2240Z9CPiTmD4fLYfB74aEfvSPJvT9HOAGyJiX0Q8RzJX7knpY31EPBsR3cANad5ZYTDou1/fzGa5ifbpHw28TdJ9ku6QdGKa3gpsKMjXnqaNlr4fSRdJWiNpTUdHxwSrV1qtjcl99T2Cx8xmu4kG/RqgCTgF+FPgRkkqRYUi4pqIWB4Ry1taWkqxy0mbU1/DIXPqfFWumc16NRPcrh24KZLhLPdL6geagY1AviBfW5rGAdJnhbamrK/KNbNZb6It/R8D7wSQdDRQB2wBVgHnSqqXdCSwFLgfeABYKulISXUkJ3tXTbLuB1U+l/GJXDOb9cZs6Uv6AXA60CypHbgCWAmsTIdxdgMXpK3+tZJuBJ4AeoFLIqIv3c8ngFuAamBlRKydguOZMvmmLLesfYm+/qC6qiQ9WWZmB92YQT8izhtl1fmj5P8K8JUR0m8Gbh5X7WaQfC5LT1/w0s6uwRO7Zmazja/ILVK+KQn0HrZpZrOZg36R2jyZipmVAQf9Ih3e2IDksfpmNrs56Bepvqaaw+Y3eASPmc1qDvrjkM9lafdYfTObxRz0x6GtyWP1zWx2c9Afh3wuy0s7u9jX2zfdVTEzmxAH/XFoy2WIgBe3d013VczMJsRBfxwGbrHsG6+Z2WzloD8OQ/fV98lcM5udHPTH4bD5DdRWyydzzWzWctAfh+oqcXhjxlflmtms5aA/Tvlclg2+KtfMZikH/XHKN2Vod0vfzGYpB/1xastleWVPN3v29U53VczMxm3MoC9ppaTN6YQpw9ddJikkNafPJekqSeslPSrphIK8F0halz4uKO1hHDxtueQWyxu3u4vHzGafYlr61wErhidKygNnAL8uSD6LZIrEpcBFwNVp3iaSGbdOBk4CrpCUm0zFp8vQsE138ZjZ7DNm0I+IO4GtI6z6JvAZIArSzgG+F4l7gUZJi4AzgdURsTUitgGrGeGLZDbI+776dhDc++wrnP+39/Gur9/OqkdeJJmN1GzyJtSnL+kcYGNEPDJsVSuwoeB5e5o2WvpI+75I0hpJazo6OiZSvSnVPLeOTG21R/BYyUUEdz7dwe9/9xece829PPXSLupqqvjUD37J+dfex/rNu6e7ilYGxpwjdzhJWeBykq6dkouIa4BrAJYvXz7jmjeSaMt5rL6VTkTw86c281c/X8/DG7Zz2PwGvvDbyzj3pCOora7i+/f/miv/7SnO+tad/Ne3HcUn3/VasnXj/tc1AyYQ9IElwJHAI5IA2oCHJJ0EbATyBXnb0rSNwOnD0m+fQNkzQr7JY/Vt8vr7g39/4iX+6ufrWfviTtpyGf7yd97AB97cSn1N9WC+D53yGs469jC++rOnuPr2Z1j18Iv8+XuXceYxC0n/B82KNu7unYh4LCIOjYjFEbGYpKvmhIh4CVgFfDgdxXMKsCMiNgG3AGdIyqUncM9I02altlwyVt/9rDYRff3BTx7eyIpv3cnF//gQe7v7uPKDx3Hbn5zOfzn5iFcF/AHNc+v5+u+9kR9e/BbmNdRw8T8+yEeue4AXXtkzDUdgs9mYLX1JPyBppTdLageuiIhrR8l+M3A2sB7YC3wEICK2Svoy8ECa70sRMdLJ4Vkhn8uya18vOzt7WZCtne7q2CzR09fPTx5+ke/ctp5nt+xh6aFz+da5b+K9xx1OdVVxLfYTFzfx00+ext//4gW+ufpp3vPNO7n4HUv476cvoaF2/y8Lm726evqm5D0dM+hHxHljrF9csBzAJaPkWwmsHGf9ZqR8UzJWf8O2vSzILpjm2thM193bz48eauc7t69nw9ZOXr9oPlf/4QmcecxhVBUZ7AvVVFfxsdOO5L3HLeIr//okV926jh//ciNfeN8y3vW6hVNwBHYw7O3u5b7ntnLX01u4c10HixY08A8fO7nk5fhs0AS0FQzbPLbVQd9G1tXTxz89sIHv3vEMm3Z08ca2BVzx3mN49+sPLUlf/ML5DVx13vGce2KeP//J43z0ujW8Z9lCrvjtZYOfUZu5+vuDJ1/ayV3rtnDn0x2seX4b3X391NdUcdKRTbxn2dR8gTvoT8DgBVq+xbKNYG93L9+/79f89Z3P0rFrH8tfk+OrHziOty9tnpITr6e+tpmfffrtXHv3c1x16zp+8xt38Ml3LeW/vu3IEc8P2PTZvKuLu9dt4a70sWX3PgBed9g8LnzrYt62tJkTFzdNaVedg/4ELMjUMr+hxpOpHET9/cGe7l52dQ08etjV1cvO9G9hWldPHy3z6mnNZTi8MUNr+phTP7Uf911dPXzvFy9w7d3PsXVPN6cuOYSrzj2eU45qmvJRNnU1VXz89CW8702H8xc/fYIrb/kVP3qwnS+dcyynLW2e0rJtdF09fax5fht3revgjqc7eOqlXQAcMqeO05Y28/alLZy2tJmF8xsOWp0c9CeoLZd1S38cIoJte3vYvKtrWNAeWt41QgAfCOy79/Uy1mCp6ioxv6GG+ppqtuzeR2//qzdYkKmltXHgi6Bh8Evh8MYMbY0ZmufWT6iPfcfeHv7unuf4u/98nh2dPbzj6BY+9e7X8ubXNI17X5PV2pjh6vPfzO2/2swXVq3l/Gvv47eOW8Sf/9YyDltw8AJLpYoI1m3ezZ1Pd3DXui3c99wrdPX0U1ddxfLFOf7nitfxtqXNLFs0f0KftVJw0J+gfFOGZzo8XG5Af3+wZfc+NmzrZOP2TjZu66R92142bu+kfVvyvLOnb9Tta6rEvIYa5jXUpn9ryDdlmddQw/yCtKH1tem6oeVMbfVgi7qvP9i8q4sXt3eycXsXG7d18uL25NG+bS/3PfcKu7pefafU2mqxaEFm1C+G1sbMq352b93Tzd/e9Szf+8UL7N7Xy3uWLeQT73wtb8w3TslrPB6n/8ah/Nulh3DNnc/y7dvWc/tTm7n0N4/mwrcuprZ6dt9ct6evn1d2d7Nl9z46du8bXN6yax97unuZ11A7+Gt8fqY2eQykZWpYkKktabfX1j3d3L1+C3elgf6lnV0ALGmZw3knHcHbl7Zw8lFNM+aCuplRi1kon8tyx9MdRERFXCDT1x+8tLNrKJhvS4P59qEg393X/6ptGrNJy3pJyxzevrSF1lyGhfPrC4J47WDQbqitKunrWF2VBPBFCzK8+TUj59nZ1TP4RVD4xbBxeyf3PLOFl3d2MezHAofMqePwxgyHzqvnnmdeoau3j7OPXcQl73wtyw6fX7L6l0JDbTWfevdS3v+mVr74L2v5ys1P8sMHN/Dlc47l5KMOme7qvUpXTx8du/axpTCI797Hlt3ddKQB/ZU9Sfr2vT0j7qOhtoq59bXs3tdDV0//iHkG1NdUpV8Cr/6CWDDCF8T8htpXrauvreLhDdu5a10S5B/buIOI5Jdk0mXTzGlLW2htzEzFSzVpmskXGC1fvjzWrFkz3dUY0d/f8zxXrFrL/X/2bg6dNzt/NkcEff1BX/q3Y9e+JJgXtM4HWusv7ejar7ukeW7Sb96WS7pH2nIZWnMZWhuztOYyzJ3iPvSp1tPXz0s7kl8LL+5IXo+N25Pnm3Z0cszhC/jvpy9h6cJ5013Voqx+4mW+sGotG7d38jvHt/K5s183JZ/d/v5gd3cvO/b2sKNz6PHKnm627No3GNCHgns3u0eZn2JeQw0tc+s5ZG4dzXPrhx7z6jhkTj0t84bSC8/Z7OvtY2dnLzs6e9jZ1cPOtA47u3rZ2Vn4PP3b2Vuw3LPfl/1IaqrECUfkeNvSZt52dAtvaF1Q9PUWU03SgxGxfKR1s/u/choNjtXf2lnSf5x9vX288Mpentm8m2e37OGZzbtp395Jb18/fZH8Q/WnQXroL0PLg0E8DerD0vvTffRFHLCPXEomgm9tzLD8Nbk0uGeTk6K5/bs6ylFtdRX5puzgaK3Z7j3LFnLaa5v59m3ruebOZ/mPJ17msjOO5vxTXkPNsC6fiGBPd18SsPf2sL2zezBQDjy2FwT1nZ09bO8cO2hKkMvW0ZwG8ePaGgcDektBMG+eV88hc+om/Bmrr6mmZV41LfPqx71tRLB7X3K+acfe/b80dnf18vpF83jLkkOY1zD7Ls500J+ggVsst2/by5tfM76pASKCrXu6eaZjD8907ObZjt2Dyxu27n3VP8xh8xs4oilLtq6GqipRLaiS0mVRXZUsVwmqC9IH06pEVZpvYHmkdAlaBlrujVkOW9BAXc3s7vu1/WXqqvmTM3+D3z2hlStWreUL//IE37//1yxakBkM2AOBfPgvu0LVVaJxoMsjU0tjto7FzXNYkKbt98jW0pSto2lO3X5fMDONpPQ8Ue2M7aKZDAf9CWor4r76PX39vPDK3lcF9YHlHZ1D/ZL1NVUc2TyHY1sXcM4bD2fJoXM5qnkuR7bMmfVdJDYzHdUyl+999CR+9vhLfPeOZ9i2t5sFmVracpnBQN2YHQra8zO1NGbqWJCmzamrrohzWeXIEWWCMnXVNM+tY8PWTrbv7eaZjt08s3kPz2xJ/j67ZTe/fmXvq1pLh86r56iWObz3uEUsaZnLUS1zWNIyl9bGzLQN37LKJYmz37CIs9+waLqrYgeRg/4ktOWy3PjgBv5pzdD8MHXVVSxuzvIbC+dx1rGHsaRlLktaklb7/FnY/2dm5cVBfxIufscS7ni6gyUtcwZb7W257Iw5g29mNpyD/iSsOPYwVhx72HRXw8ysaDP7NLqZmZXUmEFf0kpJmyU9XpB2paSnJD0q6Z8lNRas+5yk9ZJ+JenMgvQVadp6SZ8t+ZGYmdmYimnpXwesGJa2Gjg2Io4DngY+ByBpGXAucEy6zXckVUuqBr4NnAUsA85L85qZ2UE0ZtCPiDuBrcPS/j0iBq6bvpdkonOAc4AbImJfRDxHMm3iSeljfUQ8GxHdwA1pXjMzO4hK0af/UeBn6XIrsKFgXXuaNlr6fiRdJGmNpDUdHR0lqJ6ZmQ2YVNCX9GdAL3B9aaoDEXFNRCyPiOUtLS2l2q2ZmTGJIZuSLgTeC7w7hm7VuRHIF2RrS9M4QLqZmR0kE2rpS1oBfAZ4X0QU3nxmFXCupHpJRwJLgfuBB4Clko6UVEdysnfV5KpuZmbjNWZLX9IPgNOBZkntwBUko3XqgdXpTZfujYiLI2KtpBuBJ0i6fS6JiL50P58AbgGqgZURsXassh988MEtkl6Y0JElmoEtk9i+kso4WOWUSxkHqxwfy8wr42CVM5kyRpk6aIZPojJZktaMNpGAy5iecsqljINVjo9l5pVxsMqZqjJ8Ra6ZWQVx0DczqyDlHvSvcRkzrpxyKeNgleNjmXllHKxypqSMsu7TNzOzVyv3lr6ZmRVw0DczqyAzOuhL+qakSwue3yLpbwue/29J/2OUbS+UdPgY+79Q0v8tWYVfve/3H4w7iUraPdVlFFGHywuWFxfehnsa6nJz4a2+Z4PxvoeSTpf003R5wp/hqXqtpvLzIOn3JD0p6TZJyyVdNUb+wddqhHXPS2ouVd3GKy3/sfTxhKS/kNQw1eXO6KAP/CdwKoCkKpKLFY4pWH8qcM8o214IHDDoT7H3k9xGuhJcPnaW4kia1GxuEXF2RGwvUXXK2hS+ViX7PIzgY8AfRcQ7I2JNRHxqCss6GN4ZEW8guRPxUcBfT3WBMz3o3wO8JV0+Bngc2CUpJ6keeD1whqQHJD0u6RolPggsB66X9LCkjKQTJd0j6RFJ90ual+73cEn/JmmdpK+NVhFJ56XfyI9L+l8F6bsLlj8o6TpJpwLvA65My19S0ldl/7p9U9JaSbdKaknTlqTH9aCkuyS9Lk1fqGTim0fSx8CX6o/TvGslXXSg40uXfy99LV4G5qTH+X2Sq7WXSHpF0ouS/l1SZow6XSfpu5LuA742Wr4iX4vnJTVLmiPpX9NjfFzSH0zyZR5eztVK7ga7VtIXS7C/kd7D2yUtT5ebJT0/2XKGlTnwWi1OW89/k9Zh8D2bwD6/CmTSz8PAjRirR9r3eN9nSZ8HTgOuVTKRU+EvnjlKJny6X9IvJe1363ZJh6Tlr1XSYzBjJrOOiN3AxcD7JTWlcezK9LP72MDnNz3m2yX9PyUTWV0vaXzHEREz+gE8BxwB/Lf0RfkycDbwVuAuoKkg7z8Av50u3w4sT5frgGeBE9Pn80luQXFhmr4AaABeAPIj1OFw4NdAS7rdz4H3p+t2F+T7IHBdunwd8MGD8PoE8Ifp8ueB/5su3wosTZdPBn6eLv8TcGm6XA0sSJeb0r8Zki/XQ8Y4vseA1sI8wEXA10luwXEisAb4V+D8Mep0HfBToPpA+Yp8PZ4n+UX4AeBvCtIXlPh1H3i9qtPP2nFT8B4WfoabgefT5dOBn6bLFw7kn0C5A6/V4vQ9e1OafuPAezbB/RZ+Zkbd90Te52GvSeHr8JcF+20kmdxpzrA8VwGfT5d/K33dm0v5uZjI6z8s7eH0tfgAyWRV1cBCkvizKD2eHSQ3rawCfgGcNp5yZ8PE6PeQdOOcCnyD5D78p5Ic+H8C75T0GSALNAFrgX8Zto/fADZFxAMAEbETIP2CvDUidqTPnyC5Z8WGYdufCNweER1pvuuBtwM/LuFxTlQ/SSAH+EfgJklzSV6jHxY0AurTv+8CPgwQyX2RdqTpn5L0O+lynuRmea8coNz/BK5Tcq+lAWcAJwB9wN+QfJk+DCweo04AP4yIviLyFesx4H8r+VX204i4awL7OJDfT38R1ZD8My4DHp3gvvZ7DydfvXF7LiIeTpcfJAnWU7bvEr7PA84A3ifpT9LnDSSNxUJvB34XICL+VdK2SZQ3VQZejNOAH6T/oy9LuoMkDu0E7o+IdgBJD5O8V3cXW8BsCPoD/fpvIGmBbgAuIzn4vyMJLssjYoOkL5C82eOxr2C5j/G/JoUXOkz5SZgiBEkLYHtEvKmYDSSdDvwm8JaI2CvpdoaOZcTji4iLJZ1M0mLKSjqE5AP7BeAzA2Wn/4Rzi6jTnvTvuOo+moh4WtIJJL8K/0LSrRHxpcnsc4CSO8j+Cckvx21pl1cp3/uB17yXoS7Yqf5sDf8/mFD3zjj2XZL3uYCAD0TEr16VKC0s0f6nnJIu58Ukv1IOZFIxa6b36UPS0n8vsDUi+iJiK8nPt7cwdBJ3S9py+GDBdruAgX77XwGLJJ0IyYur8Z0wvB94R9r/WQ2cB9yRrntZ0uuVnGj+nYJtCsuftLSvd6TZxqoYOu7/Atyd/pJ5TtLvpdtK0hvTPLcCH0/TqyUtIGmRb0sD/uuAUwr2P+LxSVoSEfdFxOdJWqpHktxF9fyCPEeTdK0xRp0GFZtvLEpGbu2NiH8EriT5BVIq80m+pHakQeWsSe5vv/cwXX4eeHO6/EFmjx5JtQfKUKr3ucAtwCcH+rclHT9CnjtJXl8knQXkJlFeSaXx6zvAjyNiG0nX9R+k/6MtJL9S7i9FWbMh6D9G0u9477C0HRGxhaSl/zjJm/5AQZ7rgO+mP3+qgT8A/krSIyR9ZUW3nCJiE/BZ4DbgEeDBiPhJuvqzJP3R9wCbCja7AfjT9KTSpE7kpgH3tQybqzi1BzhJybC4dwEDrdk/BD6WHu9ahuYk/jRJl9hjJD+1lwH/BtRIehL4Kq9+rUc7vivTE0yPk3Th/APwDmAd8No0/a959WdstDoNV2y+A3kDcH/6/l8B/MUE9jGiiHgE+CXwFPB9kl+jkzHae/h14OOSfknyPzBbXAM8qqETuaMpxfs84MtAbVru2vT5cF8E3p6u/12SfvJxUzLUtVQjA29L3/f70/r8tzT9n0m6Cx8hOYf4mYh4qRQF+jYMs4CkY4GPRsSI1ySYmRXLQd/MrILMhu4dMzMrEQd9M7MK4qBvZlZBHPTNzCqIg76ZWQVx0DczqyD/H9bDgWSVA4LTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "words = [tokenizer.decode(input_ids[0][i]) for i in range(0,13)]\n",
    "norms = [torch.linalg.norm(y['hidden_states'][11][0][i]).cpu().numpy() for i in range(0,13)]\n",
    "plt.plot(words,norms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "c15955e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABQ60lEQVR4nO3deZwdVZ3w/8+36tZd+/a+ZE93QpOEJCQxYTEgW5RNVMYFVNBhRkHHcZxxFNR5FJUZfZgZ1GeUeUARf4zPg4wKbqMwogQUnrAkkSVAErInnXSS3re71XJ+f1T1ze2kO+nldjqdPu/Xq15VdepU1bl17/2eU6fq1hWlFJqmadrUYEx0ATRN07STRwd9TdO0KUQHfU3TtClEB31N07QpRAd9TdO0KSQ00QU4kerqalVfXz/RxdA0TZs0Nm7c2KqUqhls2Skf9Ovr69mwYcNEF0PTNG3SEJE9Qy3T3TuapmlTiA76mqZpU4gO+pqmaVPIKd+nPxjbtmlqaiKTyUx0USadaDTKrFmzsCxroouiadoEmJRBv6mpiWQySX19PSIy0cWZNJRStLW10dTURENDw0QXR9O0CTApu3cymQxVVVU64I+QiFBVVaXPkDRtCpuUQR/QAX+U9HHTtKlt0gb941FKcbg7QzrnTHRRNE3TTimnZdB3PUVbX45drSkytlv07X/605/mf/2v/5Wfv+KKK/joRz+an//MZz7DN7/5zUHXfeCBBzhw4MBxt//AAw/wyU9+sihl1TRNK3RaBv2QaTCvOgHArtY+ck5xA/8FF1zAunXrAPA8j9bWVl577bX88nXr1rF69epB1x1O0Nc0TRsvp2XQB4hYJg3VCTyl2Nnah+16Rdv26tWrefbZZwF47bXXWLJkCclkko6ODrLZLJs3b+bxxx/nnHPOYcmSJdxyyy0opXj44YfZsGEDN9xwA8uXLyedTrN+/XpWr17NsmXLOPfcc+np6QHgwIEDXHnllTQ2NnLbbbcVreyapk1tk/KWzUJf/a/XeP1A95DLPaVI2y6GCFHLZDiXMc+aUcqX37F4yOUzZswgFAqxd+9e1q1bx5vf/Gb279/Ps88+S1lZGUuXLuWTn/wkt99+OwAf+tCH+PWvf8173/te7r77bu666y5WrVpFLpfj+uuv58c//jHnnHMO3d3dxGIxAF566SVefPFFIpEICxYs4G/+5m+YPXv2iI6Npmna0U7bln4/Q4RoyMRTqqj9+6tXr2bdunX5oP/mN785P3/BBRfw5JNPct5557F06VLWrl07oPun39atW5k+fTrnnHMOAKWlpYRCfj28Zs0aysrKiEajnHXWWezZM+TzkzRN04Zt0rf0j9ciL9SdttnTliIeMWmoSmAYY7t1sb9ff9OmTSxZsoTZs2fzjW98g9LSUv7iL/6Cm2++mQ0bNjB79my+8pWvjPje+Egkkp82TRPH0XciaZo2dqd9S79facxidmWMvqzDnvYUnlJj2t7q1av59a9/TWVlJaZpUllZSWdnJ88++2z+Im51dTW9vb08/PDD+fWSyWS+337BggU0Nzezfv16AHp6enRw1zRtXE36lv5IlMfDuEqxvyPNvvYUcyrjo/6x0tKlS2ltbeWDH/zggLTe3l6qq6u5+eabWbJkCdOmTct33wDcdNNNfPzjHycWi/Hss8/y4x//mL/5m78hnU4Ti8X4/e9/P+bXqWmaNhRRY2zxjrdVq1apo/9EZfPmzSxatGjU22zpydLclaYiHmZWRWzK/Up1rMdP07RTm4hsVEqtGmzZlGrp96tJRvCU4lB3BtMQppdFp1zg1zRtapqSQR+gNhnB9RStvVkMEaaVRSe6SJqmaeNuygZ9Eb+F73mKwz0ZTANqkjrwa5p2ejvh3TsiMltEnhSR10XkNRH52yD9xyLyUjDsFpGXgvR6EUkXLLu3YFsrRWSTiGwXkW/LBPepiAgzK2KUxSyauzK09WYnsjiapmnjbjgtfQf4jFLqTyKSBDaKyO+UUtf3ZxCRbwBdBevsUEotH2Rb9wA3A88DjwJXAo+NtvDFICLMroyj2lLs70xjGkJ5PDyRRdI0TRs3J2zpK6WalVJ/CqZ7gM3AzP7lQWv9OuCh421HRKYDpUqp55R/y9APgWtHX/TiMUSYUxknEQmxrz1Nd9qe6CJpmqaNixH9OEtE6oEV+C31fm8BDimlthWkNYjIiyLyBxF5S5A2E2gqyNNEQeVx1H5uEZENIrKhpaVlJEUcNcMQ6qsSxMIme9pT9GbGL/D/4he/4PXXXx+37Wuapg1l2EFfREqAR4C/U0oVPuHsAwxs5TcDc5RSK4C/B34kIqUjKZRS6ntKqVVKqVU1NTUjWXVMTEOor4oTCRnsbkvRlx2fX8fqoK9p2kQZVtAXEQs/4D+olPpZQXoIeDfw4/40pVRWKdUWTG8EdgBnAvuBWQWbnRWknVJCpkFDdQLLFHa39Q3r37ceeughli5dypIlS/jc5z6XTy8pKclPP/zww9x0002sW7eOX/3qV9x6660sX76cHTt2jMvr0DRNG8wJL+QGffb3A5uVUkf/HdRbgS1KqaaC/DVAu1LKFZF5QCOwUynVLiLdInI+fvfQh4HvjPkVPPZ5OLhpzJspZE1bSsPbvsaOlj52taaYV5MgapmD5j1w4ACf+9zn2LhxIxUVFVx++eX84he/4Nprrx00/+rVq3nnO9/JNddcw3vf+96illvTNO1EhtPSvwD4EHBZwW2YVwfL3s+xF3AvAl4JbuF8GPi4Uqo9WPYJ4PvAdvwzgAm9c+d4wiFzWP++tX79ei655BJqamoIhULccMMN/PGPfzyZRdU0TRu2E7b0lVLPwOD/PaKUummQtEfwu4IGy78BWDKyIp7AVXcWdXOF+v99a2drLztb+5hfU4JlDv/ad+HPEEb6aGVN07TxMGUerTxasbBJfVUCx1Xsau3DOepvF88991z+8Ic/0Nraiuu6PPTQQ1x88cUA1NXVsXnzZjzP4+c//3l+ncLHK2vaYJRSbD3Yw29eaWZfe4pT/cGI2uQxZR/DMBKJSIj6qji72lLsbuujoboEM/gTlunTp3PnnXdy6aWXopTi7W9/O+9617sAuPPOO7nmmmuoqalh1apV9Pb2AvD+97+fm2++mW9/+9s8/PDDzJ8/f8Jem3bqSOdc1u1oZe2Wwzy1tYX9nen8smmlUc5pqOTc+grOaajkzNrkmP8ISJuapuSjlUer2P++NVH0o5VPHfvaU6zdcpgntx7m2R1tZB2PeNjkwjOquWxhLQunl7KpqZMXdnewflc7B7v9bsKymMWquRWc21DJOQ2VLJlRRjikT9w1n360cpH0//vW3vYUe9pTzK2KY+hHMmsjYLse63e38+SWwzy5tYXth/2zv4bqBDecN5fLFtZyTkMFkdCRu8WWzy7nQ2+uRylFU0eaF3a188KudtbvbueJLYcBiFoGK2ZXBGcDlbxpbjnx8MR+vW3X42BXhqaONPs706RyDlHLJGaZ+XEsbBAJmcTCA9MjIWPSNqpOdTroj1Ax/31LmxoO92R4amsLT209zNNvtNKTdQibBufNq+SD587h0oW1NAR3ih1P/3OiZlfGec9K/ycvLT1ZNuxu54XdfiVw99pteApChrB4ZpnfHVRfyTn1lVQkivtMqYzt0tyVoakjxf6OdD64+9MpDnZn8MbQkRAJGcTCJtGgUohaJlHL8CsLq3/erzj688TCJqVRi7LYsUNpzMp3y05lOuiPQlUigudBc1ea3W0pEmGTcMjwB9PANERXBOPM8xQ9GYeutE1nOkdnyqYzbdOVytGTdSiLWdQmo9QmI9SVRqkuCRMawZ1XYy3bK/u7gtb8YV5p8p9FWFca4Zpl07lkQS0XnlFNIjL2r19NMsJVS6dz1dLpAPRkbDbu6WD97nbW7+rgP57dw31P7wKgsbaEcxoqOa/BrwRmlMeOu+2+rHMkiHem88F9f6cf4Ft6Bj6V1jSEaaVRZlbEOH9eFTMrYsyqiDGzPM7MihjJaIiM7ZKxXdI5j4zjks4F80F6xvZI20G645LJ9S/zCvK4dGfsYF3vyDZt94SVTDISonSQCqEsfqRiGLTCiIZO2udnvOmgP0o1yQgKRWtPjp6jntNjGkLYNAZUBP3TlmnoLqECOcejK23T1R+4U3YQyP0A3pm2jwT0grSutM1ILkeJQFUi7FcEpZF8ZVCbjFATpNWVRqkpiYyqb7wrbfP0thae3NLCH944TGtvDkNgxZwKbr1iAZcsqOGs6aXj3hhIRi0uWVDLJQtqAb81vml/V75L6L9eOsCPnt8LwMzyGOc2VPKmuRU4rue31DvSNHX6wb0jNfBzHTYNZpT7Qf3SBTXMqogzszyWD+7TSqMTGhiVUmQdj+7+z8pxhv48O1t782kZ2zvu9ksioXwlMLsyxpl1Sc6sS7JgWpL6qsSkuaaig/4Y+C3JKK6nsF2PrOORczxyrj/O2B7dGWfA7XaCYIWGqBRMY9K2JpRS9OVcOvpydKRytOfHNp0F8x19AwN6Kjf4j97AD9SlUYvyuEV5zKIsHmZuZZyymJ/mj8P5+fKgxZaMWHSmcxzuznK4J8vhngyHurO09GQ43J3lUE+G1w9009qbHbRlWBG3/AogGaE2GaUuqCRqS4+cOdQkI+xtT/HklsOs3XKYDXs6cD1FWczi4jNruGxhLRefWVP0LpWRilpmvnvnry8F11Nsbu72zwR2t/P0tlZ+/qL/NJSYZTKzIsbM8hjLZpXnp2dVxJlVEaOmJHJK97OLSL7Lp7Z05H+IlHXcARVCfkjZdKWd/HxnKsf2w738fvNh3OADFDKEeTUJGuuSLKhLcmZdCWfWJZlblTjlupR00C8C0xBMwxz0UQ1KKWxX5SuCwkqhO+3geN4x2xqsQjBE/F/IiV9xiPi/mBPxP+yDLxvdh00pRSrn5oN0eyqXD+Ydfblg3j4SyIP5nDt4S8kQqIiHqUiEKY9ZzCyPsXhGqR+s+wN4f/DOB/AwyWho1EEmFo4xvez43Reup2jrPVIxHO7Ocqg7mO7Jcrg7w/bDvbT0ZHGO02+wcFqSj100j8sW1rJ8dvkpXXGbhrBkZhlLZpbxFxc0oJRif2eamGVSmQhP6W7JSMikNmlSO8x/0MvYLjtb+th2uIetB3t441Avm5q6eHRTc/4sNBIymF9TwoJpSRrrSoIKIcnM8tiEVaA66I8zESEcEv/UL3LsctdTAyqC/unBzhJGvO+CCqCwQjjYleETdz1FyBBCpoFlCiFDSNtePqjnnMEDuPQH8LhFZSLM7Mo4y2aVU5EIU5nwW96VQYCvTPj5SqPWKdlCNA3xW++lUaBsyHyep2hP9Z85HKkQKhMRLl1Yc8LK5VQmIsyqiE90MSalqGVy1oxSzpox8CHCqZzD9sO9bD3Yw7Zg/PzOtvwZFUA8bNJYl+TMWr9C6O8qqiuNjHvFq4P+BDMN8e86YOizBNv18JRCKVD+ApQ/CsaF837isenBvIJwyGDJzDIc18N2FY7n4biKyoTJ0pmlfsCOh/Ot88qEFQT68JS8A8IwhOqSCNUlEc5iRE8J16ageDjE2bPKOXtW+YD07ozNtkP+GYFfIfTw5NYWfrrxyN+MlEZDfgUwze8m+vCb5xa9EtBB/xQ24CyhiHoPhfnOB/SPszTtZCqNWqycW8nKuZUD0tv7crxxqOfIcLCX37zSzB8iLfz56vqil2PSB/1/fuGf2dK+pajbXFi5kM+d+7kTZ9Q0TRujykSY8+dVcf68qnyaUoruzPj8idOpe8VJ0zRtihIRymLWuGx70rf0dYtc0zRt+HRLf5TWrFnD/v2n3L89apqmHZcO+qPgeR7bt2+nsrLyxJk1TdNOITroj8Lrr7/Oe97zHmKxyXt/tqZpU5MO+qOwZMkSvvnNo/8jXtM07dR3wqAvIrNF5EkReV1EXhORvw3SvyIi+wf5s3RE5Asisl1EtorIFQXpVwZp20Xk8+PzkjRN07ShDOfuHQf4jFLqTyKSBDaKyO+CZd9SSt1VmFlEzgLeDywGZgC/F5Ezg8X/DrwNaALWi8ivlFKvF+OFaJqmaSd2wqCvlGoGmoPpHhHZDMw8zirvAv5TKZUFdonIduDcYNl2pdROABH5zyCvDvqapmknyYj69EWkHlgBPB8kfVJEXhGRH4hIRZA2E9hXsFpTkDZUuqZpmnaSDDvoi0gJ8Ajwd0qpbuAeYD6wHP9M4BvFKpSI3CIiG0RkQ0tLS7E2q2maNuUNK+iLiIUf8B9USv0MQCl1SCnlKqU84D6OdOHsB2YXrD4rSBsq/RhKqe8ppVYppVbV1NSM5PVMiJKSkokugqZp2rAM5+4dAe4HNiulvlmQPr0g258BrwbTvwLeLyIREWkAGoEXgPVAo4g0iEgY/2Lvr4rzMjRN07ThGE5L/wLgQ8BlR92e+S8isklEXgEuBT4NoJR6DfgJ/gXa/wb+OjgjcIBPAr8FNgM/CfKeFj796U+zePFi1qxZQ3+X1I4dO7jyyitZuXIlb3nLW9iyxX8a6KFDh/izP/szli1bxrJly1i3bh0A1157LStXrmTx4sV873vfy2+78Ezi4Ycf5qabbgLgpz/9KUuWLGHZsmVcdNFFALiuy6233so555zD2WefzXe/+92T8fI1TZskhnP3zjMEf750lEePs87XgK8Nkv7o8dYbjYNf/zrZzcV9tHJk0UKm/cM/DDt/X18fq1at4lvf+hZ33HEHX/3qV7n77ru55ZZbuPfee2lsbOT555/nE5/4BGvXruVTn/oUF198MT//+c9xXZfe3l4AfvCDH1BZWUk6neacc87hPe95D1VVVUPu94477uC3v/0tM2fOpLOzE4D777+fsrIy1q9fTzab5YILLuDyyy+noaFhTMdE07TTw6R/yuapwDAMrr/+egBuvPFG3v3ud9Pb28u6det43/vel8+XzWYBWLt2LT/84Q8BME2TsjL/r/q+/e1v8/Of/xyAffv2sW3btuMG/QsuuICbbrqJ6667jne/+90APP7447zyyis8/PDDAHR1dbFt2zYd9DVNA06DoD+SFvnJIiJ4nkd5eTkvvfTSsNZ56qmn+P3vf8+zzz5LPB7nkksuIZPJ5LfXrz8N4N577+X555/nN7/5DStXrmTjxo0opfjOd77DFVdcccw+NE3T9LN3isDzvHzL+kc/+hEXXnghpaWlNDQ08NOf/hTw/wnn5ZdfBvzHMt9zzz2A3wff1dVFV1cXFRUVxONxtmzZwnPPPZfffl1dHZs3b8bzvPyZAPjXDM477zzuuOMOampq2LdvH1dccQX33HMPtm0D8MYbb9DX13dSjoOmaac+HfSLIJFI8MILL7BkyRLWrl3L7bffDsCDDz7I/fffz7Jly1i8eDG//OUvAfi3f/s3nnzySZYuXcrKlSt5/fXXufLKK3Ech0WLFvH5z3+e888/P7/9O++8k2uuuYbVq1czffqRm6ZuvfVWli5dypIlS1i9ejXLli3jox/9KGeddRZvetObWLJkCR/72MdwnPH52zVN0yYfUUpNdBmOa9WqVWrDhg0D0jZv3syiRfqPvUdLHz9NO72JyEal1KrBlumWvqZp2hSig76madoUooO+pmnaFKKDvqZp2hSig76madoUooO+pmnaFKKDfhF8/etfz0/v3r2bJUuWTGBpNE3ThqaDfhEUBv2x0j+k0jRtPOmgP0af//znSafTLF++nBtuuAHwH61w8803s3jxYi6//HLS6TQw9KOWb7rpJj7+8Y9z3nnncdtttw2ZT9M0bawm/QPXnv7JG7Tu6y3qNqtnl/CW684cVt4777yTu+++O/9gtd27d7Nt2zYeeugh7rvvPq677joeeeQRbrzxxiEftQzQ1NTEunXrME2TNWvWDJlP0zRtLCZ90D8VNTQ0sHz5cgBWrlzJ7t27j/uoZYD3ve99mKZ5wnyapmljMemD/nBb5CdTJBLJT5umSTqdPuGjlhOJBMCIH8msaZo2ErpPvwgsy8o/yngox3vU8mjyaZqmjYYO+kVwyy23cPbZZ+cv5A5lqEctjzafpmnaSOlHK09B+vhp2ultTI9WFpHZIvKkiLwuIq+JyN8G6f8qIltE5BUR+bmIlAfp9SKSFpGXguHegm2tFJFNIrJdRL4thf8DqGmapo274XTvOMBnlFJnAecDfy0iZwG/A5Yopc4G3gC+ULDODqXU8mD4eEH6PcDNQGMwXFmMF6FpmqYNzwmDvlKqWSn1p2C6B9gMzFRKPa6U6v/56HPArONtR0SmA6VKqeeU36f0Q+DasRRe0zRNG5kRXcgVkXpgBfD8UYv+EnisYL5BRF4UkT+IyFuCtJlAU0GepiBtsP3cIiIbRGRDS0vLSIqoaZqmHcewg76IlACPAH+nlOouSP8f+F1ADwZJzcAcpdQK4O+BH4lI6UgKpZT6nlJqlVJqVU1NzUhW1TRN045jWD/OEhELP+A/qJT6WUH6TcA1wJqgywalVBbIBtMbRWQHcCawn4FdQLOCNE3TNO0kGc7dOwLcD2xWSn2zIP1K4DbgnUqpVEF6jYiYwfQ8/Au2O5VSzUC3iJwfbPPDwGlxA/rVV19NZ2fnRBdD0zTthIbT0r8A+BCwSUReCtL+Afg2EAF+F9x5+Vxwp85FwB0iYgMe8HGlVHuw3ieAB4AY/jWAwusAk9ajjz460UXQNE0blhMGfaXUM8Bg99MPGumUUo/gdwUNtmwDcNr9w0h9fT0bNmwgFotx3XXX0dTUhOu6fOlLX+L666+f6OJpmqblTfoHrj35wPc4vGdnUbdZO3cel950y4jX++///m9mzJjBb37zGwC6urqKWi5N07Sx0s/eKaKlS5fyu9/9js997nM8/fTTlJWVTXSRNE3TBpj0Lf3RtMjHy5lnnsmf/vQnHn30Ub74xS+yZs0abr/99okulqZpWt6kD/qnkgMHDlBZWcmNN95IeXk53//+9ye6SJqmaQPooF9EmzZt4tZbb8UwDCzL4p577pnoImmapg2gg34R7N69G4ArrriCK664YmILo2madhz6Qq6madoUooO+pmnaFKKDvqZp2hSig76madoUooO+pmnaFKKDvqZp2hSig76madoUooO+pmnaFKKDfhH81V/9FatWrWLx4sV8+ctfnujiaJqmDWnS/yK38792kDvQV9RthmckKH/H/GHn/9rXvkZlZSWu67JmzRpeeeUVzj777KKWSdM0rRh0S78IfvKTn/CmN72JFStW8Nprr/H6669PdJE0TdMGNelb+iNpkY+HXbt2cdddd7F+/XoqKiq46aabyGQyE1omTdO0oeiW/hh1d3eTSCQoKyvj0KFDPPbYafG3v5qmnaYmfUt/oi1btowVK1awcOFCZs+ezQUXXDDRRdI0TRvSCYO+iMwGfgjUAQr4nlLq30SkEvgxUA/sBq5TSnWIiAD/BlwNpICblFJ/Crb158AXg03/k1LqP4r7cibGAw88MNFF0DRNG5bhdO84wGeUUmcB5wN/LSJnAZ8HnlBKNQJPBPMAVwGNwXALcA9AUEl8GTgPOBf4sohUFPG1aJqmaSdwwqCvlGrub6krpXqAzcBM4F1Af0v9P4Brg+l3AT9UvueAchGZDlwB/E4p1a6U6gB+B1xZzBejaZqmHd+ILuSKSD2wAngeqFNKNQeLDuJ3/4BfIewrWK0pSBsqfbD93CIiG0RkQ0tLy0iKqGmaph3HsIO+iJQAjwB/p5TqLlymlFL4/f1FoZT6nlJqlVJqVU1NTbE2q2maNuUNK+iLiIUf8B9USv0sSD4UdNsQjA8H6fuB2QWrzwrShkrXNE3TTpITBv3gbpz7gc1KqW8WLPoV8OfB9J8DvyxI/7D4zge6gm6g3wKXi0hFcAH38iBN0zRNO0mG09K/APgQcJmIvBQMVwN3Am8TkW3AW4N5gEeBncB24D7gEwBKqXbgH4H1wXBHkDbplZSUjCj/U089xTXXXAP4t3t+8pOfHI9iaZqmHeOE9+krpZ4BZIjFawbJr4C/HmJbPwB+MJICapqmacWjH8NQJJ/+9KdZvHgxa9asof+Oo0suuYQNGzYA0NraSn19/QSWUNM07TR4DMNjjz3GwYMHi7rNadOmcdVVVw07f19fH6tWreJb3/oWd9xxB1/96le5++67i1omTdO0YtAt/SIwDIPrr78egBtvvJFnnnlmgkukaZo2uEnf0h9Ji/xk8W94glAohOd5APpxy5qmnRJ0S78IPM/j4YcfBuBHP/oRF154IQD19fVs3LgRIL9c0zRtIumgXwSJRIIXXniBJUuWsHbtWm6//XYAPvvZz3LPPfewYsUKWltbJ7iUmqZpIP4dlqeuVatWqf47YPpt3ryZRYsWTVCJJj99/DTt9CYiG5VSqwZbplv6mqZpU4gO+pqmaVOIDvqapmlTiA76mqZpU4gO+pqmaVOIDvqapmlTiA76RXD11VfT2dk50cXQNE07oUn/GIZTwaOPPjrRRdA0TRsW3dIvgvr6elpbW9m9ezeLFi3i5ptvZvHixVx++eWk0+mJLp6maVrepG/pv/HGP9LTu7mo20yWLOLMM780qnW3bdvGQw89xH333cd1113HI488wo033ljU8mmapo2WbukXWUNDA8uXLwdg5cqV7N69e0LLo2maVmjSt/RH2yIfL5FIJD9tmqbu3tE07ZRywpa+iPxARA6LyKsFaT8u+JP03SLyUpBeLyLpgmX3FqyzUkQ2ich2Efm29D90XtM0TTtphtPSfwC4G/hhf4JS6vr+aRH5BtBVkH+HUmr5INu5B7gZeB54FLgSeGzEJdY0TdNG7YRBXyn1RxGpH2xZ0Fq/DrjseNsQkelAqVLquWD+h8C1nCZBv7/fvrq6mldfzZ8Q8dnPfnaCSqRpmja4sV7IfQtwSCm1rSCtQUReFJE/iMhbgrSZQFNBnqYgbVAicouIbBCRDS0tLWMsoqZpmtZvrEH/A8BDBfPNwByl1Arg74EfiUjpSDeqlPqeUmqVUmpVTU3NGIuoaZqm9Rv13TsiEgLeDazsT1NKZYFsML1RRHYAZwL7gVkFq88K0jRN07STaCwt/bcCW5RS+W4bEakRETOYngc0AjuVUs1At4icH1wH+DDwyzHsW9M0TRuF4dyy+RDwLLBARJpE5CPBovczsGsH4CLgleAWzoeBjyul2oNlnwC+D2wHdnCaXMTVNE2bTIZz984Hhki/aZC0R4BHhsi/AVgywvJpmqZpRaQfw1AEX//61/PTu3fvZskSXbdpmnZq0kG/CAqDvqZp2qlMB/0x+vznP086nWb58uXccMMNALiuO+jjlXfs2MGVV17JypUrectb3sKWLVsmsuiapk1BopSa6DIc16pVq9SGDRsGpG3evJlFixYB8KVtTbzaW9yHmi0pifGPjbNOnDFQUlJCb28v4HfvnHHGGWzYsIHly5dz3XXX8c53vpMbb7yRNWvWcO+999LY2Mjzzz/PF77wBdauXVvUsg9H4fHTNO30IyIblVKrBls26Z+yeSoa7PHKvb29rFu3jve97335fNlsdoJKqGnaVDXpg/5IWuQny2CPV/Y8j/Lycl566aWJK5imaVOe7tMvAsuysG37uHlKS0tpaGjgpz/9KQBKKV5++eWTUTxN07Q8HfSL4JZbbuHss8/OX8gdyoMPPsj999/PsmXLWLx4Mb/8pf5RsqZpJ9ekv5CrjZw+fpp2ejvehVzd0tc0TZtCdNDXNE2bQnTQ1zRNm0J00Nc0TZtCdNDXNE2bQnTQ1zRNm0J00C+Cn/70pyxatIhLL72UDRs28KlPfeq4+Z966imuueaaQZfV19fT2to6HsXUNE2b/I9hOBXcf//93HfffVx44YUArFo16O2xmqZpE0639Mfojjvu4JlnnuEjH/kIt95664BWfF9fH3/5l3/Jueeey4oVKwb9BW5bWxuXX345ixcv5qMf/Sin+o/lNE2b3CZ9S/+r//Uarx/oLuo2z5pRypffsXhYeW+//XbWrl3LXXfdxapVq3jqqafyy772ta9x2WWX8YMf/IDOzk7OPfdc3vrWtw5Y/6tf/SoXXnght99+O7/5zW+4//77i/lSNE3TBpj0Qf9U9vjjj/OrX/2Ku+66C4BMJsPevXsH5PnjH//Iz372MwDe/va3U1FRcdLLqWna1HHCoC8iPwCuAQ4rpZYEaV8BbgZagmz/oJR6NFj2BeAjgAt8Sin12yD9SuDfABP4vlLqzmK8gOG2yCeCUopHHnmEBQsWDEg/dOjQBJVI07Spbjh9+g8AVw6S/i2l1PJg6A/4ZwHvBxYH6/xvETFFxAT+HbgKOAv4QJD3tHbFFVfwne98J99P/+KLLx6T56KLLuJHP/oRAI899hgdHR0ntYyapk0tJwz6Sqk/Au3D3N67gP9USmWVUruA7cC5wbBdKbVTKZUD/jPIe1r70pe+hG3bnH322SxevJgvfelLx+T58pe/zB//+EcWL17Mz372M+bMmTMBJdU0baoYS5/+J0Xkw8AG4DNKqQ5gJvBcQZ6mIA1g31Hp5w21YRG5BbgFmBRBsPDi7SWXXMIll1wCQCwW47vf/e4x+QvzVFVV8fjjj5+EUmqapo3+ls17gPnAcqAZ+EaxCgSglPqeUmqVUmpVTU1NMTetaZo2pY2qpa+Uyl+JFJH7gF8Hs/uB2QVZZwVpHCdd0zRNO0lG1dIXkekFs38GvBpM/wp4v4hERKQBaAReANYDjSLSICJh/Iu9vxp9sTVN07TRGM4tmw8BlwDVItIEfBm4RESWAwrYDXwMQCn1moj8BHgdcIC/Vkq5wXY+CfwW/5bNHyilXiv2i9E0TdOO74RBXyn1gUGSh/zZqFLqa8DXBkl/FHh0RKXTNE3Tiko/e0fTNG0K0UFf0zRtCtFBX9M0bQrRQV/TNG0KmfxP2Xzs83BwU3G3OW0pXFWU58Fp2qh0dXWxbds22tvbmTt3Lg0NDYTD4YkulnYamPxBX9NOA47jsG/fPrZt28b27ds5fPgwACLCunXrME2T+vp6zjzzTBobG6msrJzgEmuT1eQP+rpFrk1SXV1dbN++nW3btrFz505yuRyGYTB37lwuv/xyzjjjDCorK9mzZw/btm1j27ZtPPbYYzz22GNUVVXR2NjImWeeyZw5cwiFJv9XWTs59CdF006SoVrzZWVlLF26lMbGRhoaGohEIgPWmz9/PvPnz+fKK6+kra0tXwGsX7+e5557jnA4zLx582hsbKSxsZHS0tKJeHnaJKGD/ihdffXVfP/732fGjBkTXRTtFDac1nxNTQ0iMqztVVVVUVVVxfnnn08ul2PXrl288cYbbNu2jS1btgAwbdq0fAUwa9YsDEPfr6EdoYP+KD36qP5xsXas0bbmRyMcDrNgwQIWLFiAUorDhw/nzwKeeeYZnn76aWKxGGeccQaNjY2cccYZxOPxMe93Iiil8DwPx3FwHAfbtvPThYPneSSTScrLy4lGo8OuTE9FtmtjmVbRt6uDvqaN0fFa829729tobGwcUWt+NESEuro66urquPDCC0mn0+zYsSNfCWzatAkRYdasWfmzgGnTphWlTK7rksvlyOVyZLPZIaePDtRDBe6hlvX/A91whcNhysrKKC8vp6ysLD/0zyeTyVPuLOhA7wGe2PsET+x9gq5sFz9/18+Lvg8d9Megv/UhIvlBO3X0vz/HG4aTZ7B1XNfl4MGDA1rzpaWlRW/Nj1YsFmPJkiUsWbIEz/M4cOBAvgJYu3Yta9euJZlM5s8CIpHIcQP28aYdxxl2uUKh0KCDZVmEQiEikchxl59ofYDu7m66urryQ2dnJ01NTaTT6QFlMQyD0tLSYyqDwmG8b5NVSrG9cztP7H2CtXvXsrl9MwBnlJ/BmjlrsD0byyhua19GWnuebKtWrVIbNmwYkLZ582YWLVo07vvu/3IXDo7jDJgfzNGVwGDTI50vLNNIX8PRtm/fTjqdzi/rH4sIhmGM6wBg23a+JVc4Hkva0cv6g/N46m/N9wfO8W7NF0tPT0/+zGTHjh1ks9lB84kI4XCYcDhMJBIZMD5R2mDLLcua0OOTzWaPqQwK57u7u4/5zMTj8UErhHg8fszrtCxrWGcOnvJ4peWVfIt+X88+BGFZzTLWzFnDZXMuY07p2P4xUEQ2KqVWDbZsSrf0+1t5Rwf0wsB+NMMwME2TcDiMaZr5N1kplR9ONH+ivOMtnU7z29/+9qTsqxj6W3L9rbnCcSKRIBQK4dk2dqqXTE83mUwv4XCEWLKUWLKUeGkZibIyoomS/Hs2kgpuqLzJZHJCW/OjlUwmWbFiBStWrMB1XQ4cOIDneYMGsclQiQ1XJBKhtraW2traQZe7rktPT8+gFUJLSwvbt2/Htu3j7sOyrAEVX//xNC2TfcY+trpb2ZzbTLfXjYnJWcmzuLL+SlbXrqaupM7P70VIpVKEw+FxuRX3tG/p91/8Gaq1fvTr7w/qpmkSCoXy0/3DePcBHl0RFKYNZSRfTBFhy5YtNDQ0DDib6N/PeHSHeJ6Hk3PobE2RSzuU15RQWhk/JpAPNh0KhY55femebg5uf4Pm7Vtp3raVg9vfINPXC4AVjVEzp55Mbw9dhw/iFnQ9WJEo5dOmUzFtBuXTZwwYx8vKT6sApxWfUop0Ok1nZyeZTGbQLq/C+d5sL1vtrWz1trLb2I0tNiEvRF2qjpmpmUxLTcNSQ3fdxONxbrvttlGVdcq19JVStLa2DnrxR0TyATwSiRwT4Cf6ws7RgXi89hGLxcZt+wDZlE3zji6ad3ZyYFsXh/ek8FwFmOwlTSRuU1tfSl19lLr6JFUNpcSSx/afuo5Ny+5d+QDfvH0rnQeb+18I1bPn0njeaqadsYDpjQuomjUbwzAB8DyXntYWOpoP0HHwAJ3BuGXvLrZveA6v4EwuHItRPs2vACqmz/Cng3EsWaorhFOU43q83tzNczvbeH5nO9tbemmsTbJ8dhnLZ1ewdFYZZbHi9ImLCPF4/Lh3QLVn2nlq31M8ufdJnjvwHDnJURGv4B1z3sFlsy/j/BnnEzbC2LZ93Aojl8uN22futAz6IjKgG+DolnqxD2Z9fT3JZBLwTxHf/e5388UvfpFoNFrU/ZzK+jqzHNjeSfO2Tg5s76LtQC8oMAyhZm6SZZfNZnpjOSUVEVr29nBoVzeHdnez8bHd9NfLJZURKqe7WOFWnMwBug7vomXPTtzglDpRXsH0xgUsufRypp+xgGnzzyAcG/oLaBgmZbXTKKudRv2yNw1Y5rkuXS2H8hVBR/MBOg8e4ODObbzx3P9DKQ8Ay4hQmZxBbVU9FSXTKbUqiXpxzIyBEbOInVlJ5IxyIvPKMRPFv71OG8h2PTbt7+L5ne08v6uNDbs76M36Z3MN1QkWTkuy7XAvv9+c/xtv5tUkWD67nOWzy1k2q5xF00sJh0bRuFMKuvdD88tQNgumnQ0i7O/dzxN7nmDtvrW8ePhFPOUxIzGD6xZcx5o5a1hRuwIzaIj06+/6KSkpGdPxGI3TvntnvCmlaGhoYMOGDVRXV9Pb28stt9yCZVn8x3/8x5i37XgOjnIwxcQUE0NOXGnlu4byXUTKD6zB/NYtWyjp7sDp7MTu6sLu6sTu7sEqKyOxaBHR6TMIRSJYkQihcDCORAhZYUQEpRSdh1I0b+/yA/32TrpbMwCEIibTGkqZ0VjO9DPKqWsoxQqbg5Yzm+qjafMWdmx8hQNvbKXz4C5cuy9YamKEppGomEPN3EbmLj2LuWfXUzktgRjFrbS9lI3dmsZty+C0pcm1pMgd7MHtyCK5gXlTTjc9dge9TidRI05tbC6WEUahyMVyMC1EpLGc8qWzSVRXTtkzhD7XZXsqyxt9GX9IZTAQzitLcH55CUtKYoSG8T5mHZdXmrp4fmcbz+9qZ+OeDlI5/wxtfk2C8+dVcd68Ks5rqKSu9Egjqytt80pTJy/v6+SlfV28tK+T1l7/gnXYNDhrRumRimB2OfVV8WPfq1Q7HHgR9v8J9m/0h77DKGCbZfFEZR1rk2VscXsAaKxoZM2cNayZs4YFFQsm9L0/XveODvoj5Hou3alOejJdpFQGVxSXv+lyfvy7H1NZWQFAb08va5a/lcc3/Jay8jK+8dVv8McnnkZE+KtPf5yrr72KF/7fC9x9179TUVHBG1u3sfjss7jr3//F/6Dk35Ij741wJF0GphZmGzijjs7ta9rbzL2PfTGfpz9dlGB6guEJpgeGJxj5NDA8A1NCGG4IwwtheiFCEiYaiRKPJ0gmEyRLS4iF40QiMWLhOOFI7EjlEY1iZ9I0b/P749sPNNHfzK+cMYvpjQuYdsYCyqc14NjltOxLcWhXN4f3dGNn/C96OGoG3UKl1DWUUltfSqLs+BdTlVJ4KQenNY3TlsYJgrs/n0GlC245FDDLIoSqooSqY4SqYkemK6O4eHQdOuh3Fx08QGdzM05zikinRZlXQ1VkBqaYuJ5Dm91Mj9WBXeUSnlVK+bTplNdNp3zadJKVVchJ7EpUSpFLp0h3d5Pu6Sbd2+1P93aTSe1HSBCOlhGORrEiUaz+cf90NEo46r+XViSaL3u347KtL8PWVCYf4LelsuzLHKktLRHmxSPkPI9daT89YRqcW5bg/LIS3lyeYFlpnIhhkLFdXtrXmW/Jb9zTQdbxz7oW1CU5b14l5zVUcW5DJVUlFlk3S8bJkHWzpJ00WTdL1s2Sc3NHBi9H1slyuK+P3W3d7GnvYn9nD4d6erE9GzEcYiGb2fE+qsxuonQjThe2kyEnkBMhF4qSsyLkDIsUHl1OHwIsz9qs6e3lshzMrr8YFlwFjVdASc1Je28HM6Y+fRH5AXANcFgptSRI+1fgHUAO2AH8hVKqU0Tqgc3A1mD155RSHw/WWQk8AMTw/yv3b1URapx/fuGf2dK+ZaybGWBh5UI+d+7nAP/LknEydKU66M31kpUjV+8tZWERQRCiKk6UBKCIJmPMnjOLAzsPsPHARra8upX/evIXtLe1894rr2PV+StRSvH6ps381x9+Se20Wj74jhvZsH4jK89bmY/CKh+vFYXBfLQHrX97Tsjj9Xk9R7Yj/XsocgMgB5INKg+PIxWKaRBeHCGyIkY0EiceKyEcSmMZrxEytmK1WFimRagyhFVtETo3hJcRcr0u2W7Fxk6H7GsexismhjKJxyKUV5ZQURGnIh4jaVmEU0KoRwh1KUKdCitjElEWES9MGIt4aQnRqiTxZTV+UK+K+YG9IopYQwfjECZVs2ZTNWv2Mctcx6brwEG6Xz1AbmeG0pYaarOzoQMybSmaXtjBy/bv2G/voTfUS6SmglBlKVZFCUZpHCmJoGIhnLCQclKknBRpJ42nPFzl+hfFlYenXBzXwXZyOI6N49q4roPjOrieg+s6uJ6bX8dVLh6e/x6L8j8HpkIZCiV+rW8AoTSYSjBdMFwDwzEQ28CwBQjhhqLYoRh2OE42kiQTSZKz4iixQEIYyiTpeJR7Hhd6UKOE6WIwLRQiEY5jmCatns125bEbxdYOxXPK4dupNGY6TSiTxrOzIDYYNvFQjuq5WSIhG9PIYWPzZKfDb190sP9kY8vgt0wPiwVGJZQosJQiqjwyStHqKEQZOF4EmxoUMQwjQYlhUYVJRJlYYjG/rIGLZ7yFuTVzKEttxdq1Frb+N2z5NSAw6xxYcCUsuBpqFsIpdMY3nD79B4C7gR8WpP0O+IJSyhGRfwa+AHwuWLZDKbV8kO3cA9wMPI8f9K8EHhtdsceXpzw6U+10Z7pIuRlc8VsaphIiKg7EydoRsgosHJQysHNx7GziyDY8A9cuYcO613j7O96H4dZQW1bNuedewOb1uygtKWH5shXMq1yAkVOcvXAp7dvbKF+aQBQYKAQQpRAU/jdTUGKABOOCeUTwxM/pjxUe4AVdOh5Btw9gOj1U7/4grhIUBp4y8TBx3QQJt4SkHcFCYeEStTKU0UVZ9z4ShzYTszsJGQ6xWXXE580mPH8ORsMcvEQU27WxPZucm8P2gmknSyaXJmtnyDlZbOUg4RCu5+bzOJ7j5/VypJyUn+7aOMrJb7M/jyMOdpmNWzrIF743GAolgmEQpphE+6JEshGih6JEQhGiZpRoKErE9KePTouYEWKhGBEzgqc8Uk6KPruPPruPlJ2iz+kjZaTom9tHamaKvlwvfbk+0l5mkBI0HZnsDgYA1wIvDCqC8sKAgeFHbEQFnwkFhue3D4z8tGAqC1MihAyTiGlimSbhkEk4bBOy0oTMHgzpxTJcLMMkFplBPDYT28uSyrTTZffR4Xj0qjB9KkxGWWQJ4SqFKBuUg6HaMNVBwlmbcMbGU06+oeIArcHQ3/IjB6QGfw/6e7SVKRC3EImgJIwyw9ieYOY8zJRNJJMj4YDpmVheiJBKEPIMLM8kpI6MQ8ok5Amm6zcwDFeRlBx1Rg91oW6mm13MCHWTEAcLyHkhDmVLOZgp5VCmlKZsBTuNGRy0ajgYruFQpJaekH+9TpRHVa6duuwhuuxOtjm/pNTpIen0Uh63KKt+G7MrFXMjzdS2byPxxB3wxB14ZbORhW9HFlwFcy+AcXi0wkicMOgrpf4YtOAL0x4vmH0OeO/xtiEi04FSpdRzwfwPgWspQtDvb5GPRb41n+6kN9dDFpv9qWZEQcizsCSJ48XJOgYOilJS1NFNQlIY4mHh0GgcoNrwf/HX09tHc9Ne3jY/xrPSwzSjgzON/QAkJc00s5NS06YsophjBb/mtHIk6WJauH0UL4ARNf9tuvg9/7OgX8cfOWLQGirnQLiKA8ofDqpKDqgqdpTN4EDpUlopQ/X/4dphMA66xP+QJU4fiXCIZCJBacU0SsqTlERC+aE6EqIkZhKxTFxP4XgK1/Vw+qc9heMqHK8/zSGbsrH7bOyUg522sTMOdtbFVQoHD1s8XNPFDStcS2GbDjnXrzA8cRBxgByoHErZoHJI0IoUwwZxwPDncyEHO+TQG3LATKOMHpRp44mNKzYuWWxy2F4uqE6PCBtRLIkRkigGUfAiKC+C55Zg2zPJ2RZOzsJxwsGyMMqL5PMpL4LlhSk1YiQMi5jhEHFThLI9GNkUyoriWVHcUBjHCGMbFo6EyCmDrCdkPMg4ipw78jO1iOFRZfRiGwbdEiMjFkok3/q38AiLQ1RyWJImImksbAxRGAgmFiZhQliYmBhiYSCI4fnHVhwQB4VNu+OyJy24noWhLBqTSVbWVHLejGrOmV1JIhalUyk2ZrK8kM6wMZ3htVwWB78s85TJYjfEwqzQ2CfEMi52tmDI2Jh2J9XGdqqNN6hSW6lUW4ioTgBcwvSGG+mJvZVDicX0lSzFjs/BtEKELIM5IYOGkMElIQPTEkIhE0M8OtIZtrZ2s7mtj9faYmzpqOa1o36EHBGPcsmSbOshnm4nkV3EbHWIc8LbWNa7j3kd3yP0/L3YhGmLLaC35jzceZdSMn0+pTU1lJRXnrTuvmLcvfOXwI8L5htE5EX8dssXlVJPAzMZ0KyhKUibMI7n0JvtoTvTRZ+bCk5+wXSFCDEgQdaNkPH8VnclKWbRR0xSGKJQysAjga0swMBWFdiqkt6+Pj75hX/knVdcQUnZPN583sV8//8+yAev+wjtnV388fmX+fpX/idbt+9ASRQ7NAsFuEYJTqiSnDULgrZ9cO0Vj/64HLT+g9cgQZkLfq87cHqQawOg8IwMnSV/D3jgeQgeeC6G20WpfYhyr4WlshfT+BPGUVcyXWXSTTntlNFCGa1eGR2hctpyZbT0lHCgo4wDTaXsC8dIx0pIhyL0KQNvmPHIAMyCIYQE0woTRQgPU7mYysFwc4ScDFEnTdxNEfPSxFWWqGRxxCQrFlkJkzUsMkaEtJEga4ZJS4SsESYtFhkJ44ox4CgenwJcP6BhgGdR+K+jpvgXCiPBEA+ZVIYMoqZBPGoSD5nELJNEMJS5UJXzSKYcIqmc37UCqLIwqiyCWxqm1/XodV36PI8+T9GHRx+QMhR9BvQZit5wlu5wlj7x6MOEHFRnu6jNtlOXa6U2106t3U6d3UaN00Gd006N20G114VR8PlwMch4EbJEyaooGaJkVJR0/1j845aSECkjRFpMUobpz6soKSL0ESPtJUh5JaS9EtJuEldFqPBMPoDJSjyWYBNvzyLtezG2voFIhixZ4mS4SDJcrNIoMjhGlvZIlq5wjpSVxTUzRFWGuJuhzE5T6mSIu2kiKkOIrP82KlCO0Kdm0uIuo9OZT4czn87cHFwslDJQSlB4KNlH/tpYPzn20xAGlgPLMVHE6FDQrDxaxKXHcOgxoceK0mPGaS6poy+heBb4SbB+IpfmcuMl1sgGLvA2MS29CXfP/ezKVPNKbzVNveVI2SxKq2v9oaaGstppLL54zTA/l8M3pqAvIv8D/4zuwSCpGZijlGoL+vB/ISKLR7HdW4BbAObMGdvPkfsppUg7abqzXfRme8gqv29eFITcEGFJ4Ko4Gccgh9/CqaKPMuklShoRhVImHkmyIuSiLqmQiQO4Blz6/g/43SeexxXXXMGnbvsUB6OK8993BWtf+RPLLn8bIsJt//gF1NwK2vY6ZExFczSHAlKmR2fIoTniFPT/ST6MS+GcqAHzR4L/EOlH5c9GXF64pIkQHpa4mHiEcIlaSSLRswmH6giraiynEqvXxOpIY3S2Qvd+pHc/0fQBZmQPMsdrJmRsQkzH/1aU9B9rC0dV4XgVuHYSJ5cgnYvRF67Gjtdieh4hJYQMwRIbkywmWQy3B8PuALsT3G7E60VUH0IWw1RISKEs/P5YU2EaHpYxsNU9WjkJ4YiJIyEcDFwxsSXkjzGDcQgXw0/HJGOEyZgR+kIxUoWDGfUHI0rKjOWnu80YB/PLjuTzxPRfFEef9nvAkW4hUR6Vdjc1uTamZduYlW1hfvoQ0+1D1Nqt1OTaqc52U5HrIeYd+2gFVxmk3RLSTpy0E6PDmc5+NZ9UKIl4HmE3S9jLEVEOEbGxxCVm2iTNbkyjnZDp962HTJuQ5DCU8uvA470Fhj+4Qb1ojvDtUsqgygnj2VGUiuCpCH2hOB3hOIcj1bycjNNt+ccz5EUoycU5aM1nX6QRR+KI8hsPosBU/nfAVIXzQfeYCoqqVMH00cuC5UB5TnFBRlGVcYik0uRyDum0Q5/r0aegFZP9oQjtpkW3Cd1GiC3GhTxvXECP4TDf2Mma0J94a2QjV8U2Qw3s9KaxLr2Ex7fn2PtKN+Wyle+PQ9Af1t07QffOr/sv5AZpNwEfA9YopQbtsRORp4DPAvuBJ5VSC4P0DwCXKKU+dqJ9j+XuHdu16bV7/b5550hrPuQKphcBKSHrRXCDD2IMj3JSlEof4XygD2GToDcUojccImOGyRHGLepPHArv0hlsvr8Hp7CdP3qHdmzjw53Hvu+GcgmTwyIXjG0sbMJkCYsiaghR0yRqWsRCYWJmlBIzTFUuS22qm7Lug8S69pLsbSKZOkRZtpVyu40SrwuT4V10yxIiY1ikjQhpM0LajJLuD5xmnLQR9dONaJBn4DhlRnFFsLCJqBxRlSGicoRVjqiXJUyOiJcjrPzllrIJqxxh5WApOxgcQsr1x54/HQrSTOWfaYSUh+l5mC6YriLkKkzXI+S5jPQk3a9IwjhGFDcUw7USOOESHCuB5xkksq1EM4eJ5DowBomwDoJjGzg9Jl6vQSZr0olBu2nRFU/QV1pFtnYW5uxGKuoXUDtnATPL51AZrcSQwUubcTJ09bXRdXAvvQebSLU0k2s5jN3aimprRzo6sbp6ifX2kUhnSdgORkhhWAoj5PnTIYUb98hVKVS5BxHA8PvvlSV4luBEBScOblRwYoJrgmsKriG4pqCM/l+Lg8qAlxOcnP96c55J1ovSFD6D7ZGz2Bk9kwPhmXhi4gXXurygqaPEH/fP++e3xelOiSqHaidLbdpmWspgWp9FXUaoS7tU9XRR29pCrKsVJ9NNLttLt5GjPWHSUlFHZ/l0zKjHHON1FrjrWei9iiUOHaqEp70VvPMrvwBz5LGm6L/IFZErgduAiwsDvojUAO1KKVdE5gGNwE6lVLuIdIvI+fgXcj8MfGc0+x4O13PZ2bmDnOe35g3Pb80bxHFJkHX9E1oDIa4cykmRNPoIkUYEXCx6jDI6rThdVhyP/vvMFWFxKTGEeChE3DQJG4WnhgNPEwsv2A9IH5CnYE4p8Gxwg8ErGKv+EhQOctR4kDTBP5UVgg872F6af+l5mhwmDga2mOQwyDk5Mrk0adcmFQqTCkVIhyKkzTDpUJy0GaHD9S/s5TCxEXKATQmuVEDJXCg575j3w1AuNbkOZmQPU+70DAzc+XEYwxSi4hKTNDHVR0T1ElU9xEgTI3XUOEOJaVNpQakFZWGTsnCY8rBFLFyG9D+ZMN+oKbz76di04+crzKZwAAeF62VwnG4cuxvH6cFxurGdLtxcJ162G5XtxnBdTE9husFw1LSRhVBOEcoqTDuD6aQxU21EehRxTyEC2bjQUW6Qi0TIiUGu06Sn1aSlLUKrqiBXW4s1czolsxuorD+TaXVnMCM5k8boEL8TyHRD9wE48Ap0N0PvQbASkKgOhhqiiRqiJXXULZgJC47dxDHHxbZx2jvItDTT07yPvsP7yRw+RK71ME5bG+zshJyNVxJHJeNQkkCSJYgkMSJJTLOUcCyBkbQIJUKYCRMJexgqDV4K5fbi5Y9xJ7bdjeN0YttdzLBf4hw2nqCA/kgIYUgwEEIRQgiBhIEQiAUF6QoLwcATf6xUCI8QbUaYJiI0exaH3DDtoUraklXsTlbRQYl/gwXg30Uwg6jnUJt1qEsr6tIm0zKKmr4MszraqW45TM3hOOHORl63KolMP0xFTRcXxpvwxChS1XTEcG7ZfAi4BKgWkSbgy/h360SA3wUfqv5bMy8C7hARG/+k7+NKqf4rk5/gyC2bjzGOd+4YYiBZj4jyW/M5L0o6eNMthDLlUCZ9xI0+QmT8Lg8JczhUSaeVJGNEETzC5EjQS8wQElaMhFVyzC/rikoEzLA/DJWFsbf1D8U7+fA7bx06g2v7QaFzL3Ttg8590PkSdO0N0vbjKZts2CAbMciGDfqS5fQky+mJJem1LHpN6FUOOUxyRLDDFnbYwsMoCN6ZfBCPikM4VE7YLMUySwhLAosYYRXG8kKEXQPLEcI2hHNhQlmF5Pog2wq5Xsj2HhnbfSCmf5eEaR05pqYFhnVkOj8+Ks0YZL3B8oSTEJ0N0TIoK4NYuT8dKQPDCJ7S2udXDE4PttMdVBJdQfDqwbG76Mm10Zc5TDbdipPrQrl+l5YZclEepHvjZLwZGKVnUzrnLcyonM/CkhmcHznqeUGeC30t/nu3+1noafane5r9X5J2N/vTuaNvcRqKQLwSEjXBUD3EdA2SqMaqrcGqqyW5ZNnoP5xO1v9RVKoN0u3+dNqGlArGaUj3QaoL0h2QakdlOnENhR0SnFDwmxKlEA8ML+iu8fxrXONx46RnJcjWzCRdUU2mtISeSJhmK8o+QjQ7BofcCO1SRVu0ivZoFTsqqumkAk8iQBnQAEDMdanLKGrTUJd2qevu5J/G4eLupP1x1sKFC4f8xZvrKTY3d+Mp/w4DS0HUsEkaKeL0ElV+f2faiNAZStJjJlCiiBhZwmSJkCWGgRWpJhQuQ2QcA/1JppRiy5YtY/txm+dCz8GCCmFPML3Xn+/aB04GBeQs8SuGkiTZskqcaAwr5xLO5LAyGcKpFFaqj5BtD+8LaYYhXOIPkaPHyWBZHJQHrgNuLhj6z5xyR86k8tO5gjOs/vzOset5w31uvECk1K8AYmUQLfen8+OCCmKwdCvu35arXJRyMYww2OkjQby7GXoOBNP9QT0I6OqobjQjBMnp/lA6HZIz/HHpzCNpJdPAyfgVRn5oPWq+7ch0pnPwl21Yx68cjFBBID967Adw8r/IHoQVh1glxCuCceWx42iZXw4REOPYwTCD6cGWmwXTgyzvXxf849CxJ/jM7xk4ne0eUGw3WkamZgbpikoyJQnSMZNew+EAwgFHcdiN0U4VbVTRThXt1NJONZaCl9dcMszP3ECn3QPXotEobW1tVFVVDRr4BbAMIWQ4JI0+km4PMS8HClJGlDazipwKYUiWiNVLQjoBCLlgWeWE4rOOdA+cRpRStLW1jf2ZQIYJZTP9Yc75g+0I+lqQzn1EOvcQKawQ2jognIBoCZQOEbT75wdbFhrfP7U4Ls8bWDnk+iDT5QfBTJc/pAum+9PTndC+60jaiVrahgXRMiRahphhv/sl3XFsvnAyCOAzoOGiIKgHAb0/wCdqYDitxXDcD5o1w+jLcXJ+S3zQCqJgunUb9B32K5QBxK/c+gN1cjrULR4ioFcF0xVgje9DAkekpNYv89GU8t/jjj1+BdC5F7NjD4nOvSSa90Dni2D7PeIrglWcZBXpqumky8vJlMRJR4S0mcMLRfE7WYprUrb0bdumqamJTGawH7yA8jzcnoOElOPfYCdhPMJ4nosYNsoKLscHp32mMjHCpUjQwjqdRaNRZs2ahWWdfpXapOE6fmsw3XHiSsPJQnJaEMxnDAzqkeQEv5BhUMqvHPta/DPEfGv89Dl7HhGl/Iqxc8+xZwgdwRmzG9wmHauEz+0a1W5Ou5a+ZVk0NDQMuVwpRe///Bfs6Ay22fvIsRH7rBwqAcoxqO6wmXm4j8qy8zEv+DTMu/S0D/baKcQM+cEvXjnRJRl/Iv7ZWuTkP03ylCTiP5enpAZmDRKTPc8/s+vc619wHweTMuifSFf7Xl6ofJHQrGdQcVBZweotY+HuA9R0ZDHOeje891MwfQwXnDRN04rNMPwzutIZ47aL0zLoJ8tnIaUZcq2lzJFSFu5+CcPshTd9GN78Caion+giapqmTYjTMuibdh+X9c3C3L/RvxB08T/AOR+FRNVEF03TNG1CnZZBn2gpZuV8WPYBWH6Df2eCpmmadpoGfYD33DfRJdA0TTvlTOy/gGuapmknlQ76mqZpU4gO+pqmaVOIDvqapmlTiA76mqZpU4gO+pqmaVOIDvqapmlTiA76mqZpU8gp/2hlEWkB9oxy9WqgtYjFmah9nKz9nC77OFn7OV32cbL2c7rs42TtZyz7mKuUqhlswSkf9MdCRDYM9UzpybSPk7Wf02UfJ2s/p8s+TtZ+Tpd9nKz9jNc+dPeOpmnaFKKDvqZp2hRyugf9750m+zhZ+zld9nGy9nO67ONk7ed02cfJ2s+47OO07tPXNE3TBjrdW/qapmlaAR30NU3TppBTPuiLyLdE5O8K5n8rIt8vmP+GiPz9EOveJCLH/YfhIM/dRSvwke1eKyJnFXu7g+ynd7z3MYwy/EPBdL2IvDqBZXlURMonav+jNdL3UUQuEZFfB9Oj/gyPx/Eaz8+DiLxPRDaLyJMiskpEvn2C/PnjNMiy3SJSXayyjUZQhk3B8LqI/JOIRMdzn6d80Af+H7AaQEQM/B8sLC5YvhpYN8S6NwHj97fyx3ctMO5B/xTxDyfOMjwiMqZ/c1NKXa2U6ixScU5743S8ivZ5GMRHgJuVUpcqpTYopT41jvs6WS5VSi0FzgXmAd8dz51NhqC/DnhzML0YeBXoEZEKEYkAi4DLRWS9iLwqIt8T33uBVcCDIvKSiMRE5BwRWSciL4vICyKSDLY7Q0T+W0S2ici/DFYIEflAUBu/KiL/XJDeWzD9XhF5QERWA+8E/jXY9/yiH5WBZfuWiLwmIk+ISE2QNj94TRtF5GkRWRik14nIz4Nj8HJQVkTkF0He10TkluO9vmD6fcGxOAQkgtf5I+ALwHwRaRORAyLyuIjETlCmB0TkXhF5HviXofIN81jsFpFqEUmIyG+C1/iqiFw/xsN89H7uEZENwfH6apG2Odj7+JSIrAqmq0VkdzH2VbDP/uNVH7Sg7wvKkH/fRri9O4FY8Hl4MEg2B9vuSN9nEbkduBC4X0T+VQae7SRE5AfB9/pFEXnXIOtXBft/TfzeAhnp6xtPSqle4OPAtSJSGcSxfw0+v5v6P8PB635KRB4WkS0i8qCIDP+1KKVO+QHYBcwBPhYclH8ErgYuAJ4GKgvy/h/gHcH0U8CqYDoM7ATOCeZL8f8j+KYgvQyI4j/yYfZR+58B7AVqgnXWAtcGy3oL8r0XeCCYfgB470k4Ngq4IZi+Hbg7mH4CaAymzwPWBtM/Bv4umDaBsmC6MhjH8CvWqhO8vk3AzMI8wC3AXYADnANsAH4D3HiCMj0A/Bowj5dvmMdjN/7Z4HuA+wrSy4p83PuPlxl8zs4ep/ex8DNcDewOpi8Bfh1M39SffxT77T9e9cH7tjxI/0n/+zaKbRZ+Zobc7mje56OOR+Ex+HrBdsuBN4DEUXm+DdweTL89OObVxfxcjPb4H5X2UnA83gP8LviM1eHHoOnBa+oCZuE33J8FLhzuPifLH6Ovw+/GWQ18E5gZTHfhd/9cKiK3AXGgEngN+K+jtrEAaFZKrQdQSnUDBBXkE0qprmD+dWAusK9g3XOAp5RSLUGeB4GLgF8U+XWOhocfyAH+L/AzESnBPz4/LWgARILxZcCHAZRSLv4xBPiUiPxZMD0baATajrPf/wc8ICI/KUi7HHgT4AL34VekLwH1JygTwE+VUu4w8g3XJuAb4p+V/Vop9fQotnE81wVnRCH8L+JZwCtj2N4x7+PYijcqu5RSLwXTG/ED9rhst4jvc7/LgXeKyGeD+Sh+Q7HQRcC7AZRSvxGRjjHsbzz1H5ALgYeC7+khEfkDfizqBl5QSjUBiMhL+O/VM8PZ+GQJ+v39+kvxW6H7gM/gv/j/Dz/ArFJK7RORr+C/4SORLZh2GdlxKfyhw7hegBkmhV/7dyqllg9nBRG5BHgr8GalVEpEnuLIaxn09SmlPi4i5+G3mOIiUoX/Yf0KcFv/voMvYckwytQXjEdU9qEopd4QkTfhnxH+k4g8oZS6Yyzb7CciDcBn8c8aO4Iur2K/9/3H3eFIN+x4f76O/h6MuHtnBNstyvtcQID3KKW2DkgUqSvS9k8K8buc6/HPVI5n1DFrMvTpg9/SvwZoV0q5Sql2/FO4N3PkIm5r0Hp4b8F6PUB/v/1WYLqInAP+wZXhXzR8Abg46Ps0gQ8AfwiWHRKRReJfZP6zgnUK9z1mQT/vzEEWGRx5zR8EngnOYnaJyPuCdUVElgV5ngD+Kkg3RaQMv0XeEQT8hcD5Bdsf9PWJyHyl1PNKqdvxW6kNwG+BGwvynInfrcYJypQ33HwnIv5dWyml1P8F/hX/DKRYSvErqa4gqFxVhG0e8z4G07uBlcH0e5kcbBGxjpehWO9zgd8Cf9Pfty0iKwbJ80f8Y4uIXAVUjGF/RRfEr/8N/EIp1YHfdX198D2twT9TeWGs+5ksQX8Tfr/jc0eldSmlWvFb+q/iv/HrC/I8ANwbnP6YwPXAd0TkZfy+smG1nJRSzcDngSeBl4GNSqlfBos/j98fvQ5oLljtP4Fbg4tKY7qQGwTcM4D2QRb3AeeKf1vcZUB/a/YG4CPBa30N6L+w9bf43WGb8E+1zwL+GwiJyGbgTgYe56Fe378GF5dexe/C+T/AxcA24Iwg/bsM/IwNVaajDTff8SwFXgje+y8D/zSKbQxKKfUy8CKwBfgR/pnoWA31Pt4F/JWIvIj/HZgMvge8Ikcu5A6lGO9zv38ErGC/rwXzR/sqcFGw/N34feQjJv5trsW8K/DJ4H1/ISjTx4L0n+N3Gb6Mfx3xNqXUwbHuTD+GYRIQkSXAXyqlBv09gqZp2nDpoK9pmjaFTJbuHU3TNK0IdNDXNE2bQnTQ1zRNm0J00Nc0TZtCdNDXNE2bQnTQ1zRNm0L+f1hPkUbeYRJ4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def project(x,y):\n",
    "    return (torch.dot(x, y) / torch.linalg.norm(y)).cpu()\n",
    "words =  [tokenizer.decode(input_ids[0][i]) for i in range(0,13)]\n",
    "\n",
    "def against(posn):\n",
    "    ref_vec = y['hidden_states'][11][0][posn]\n",
    "    return plt.plot(words, [project(ref_vec, y['hidden_states'][11][0][i]) for i in range(0,13)], label=words[posn])\n",
    "\n",
    "lines = []\n",
    "for i in range(0,13):\n",
    "    lines.extend(against(i))\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "f69b164b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABUvUlEQVR4nO3deWAdZb3w8e9v5uwn+9I2bdOmdG/TjaYFyy7KohUQBUTw3nqFil716lUEvQqKF1/uKy5XeF9QhBf1Aiqr7IICApalLZTudKFbuiZp1rPOmXneP2aSnrRpkzQnbZM8n3I4c57Znplz8ptnnnnmGVFKoWmapg0NxvHOgKZpmnbs6KCvaZo2hOigr2maNoTooK9pmjaE6KCvaZo2hPiOdwa6U1ZWpqqqqo53NjRN0waM5cuX1yulyrsad8IH/aqqKpYtW3a8s6FpmjZgiMi2w43T1TuapmlDiA76mqZpQ4gO+pqmaUPICV+n3xXLsqitrSWZTB7vrAw4oVCI0aNH4/f7j3dWNE07DgZk0K+trSU/P5+qqipE5HhnZ8BQStHQ0EBtbS3jxo073tnRNO04GJDVO8lkktLSUh3we0lEKC0t1WdImjaEDcigD+iAf5T0ftO0oW3ABv0jUUqxryVJa9I63lnRNE07oQzKoC8i1LWlaElk+mX53/jGN/jFL37R8fn888/nmmuu6fj8zW9+k5/97Gddznv//feza9euIy7//vvv5ytf+UpO8qppmpZtUAZ9gIBpkLadfln2aaedxpIlSwBwHIf6+nrWrFnTMX7JkiUsWLCgy3l7EvQ1TdP6y+AN+j6DdKZ/gv6CBQt44403AFizZg3V1dXk5+fT2NhIKpVi3bp1vPDCC8ybN4/q6moWL16MUopHHnmEZcuWcdVVVzF79mwSiQRLly5lwYIFzJo1i/nz59Pa2grArl27uOCCC5g4cSLf/va3+2U7NE0begZkk81sP3xqDWt3tRySns44WI5DNND7TZw2soCbPzH9sONHjhyJz+dj+/btLFmyhA996EPs3LmTN954g8LCQmbMmMFXvvIVbrrpJgA+97nP8fTTT/PpT3+aO++8k9tvv52amhrS6TRXXHEFf/zjH5k3bx4tLS2Ew2EAVqxYwbvvvkswGGTy5Ml89atfpbKystfbommalm3QlvRFAAX99QTgBQsWsGTJko6g/6EPfajj82mnncbLL7/MKaecwowZM3jppZc6Vf+0e//996moqGDevHkAFBQU4PO5B6lzzz2XwsJCQqEQ06ZNY9u2w/afpGma1mMDvqR/uBJ5S8Jia0OM8eV5RIO538z2ev1Vq1ZRXV1NZWUlP/3pTykoKODzn/881157LcuWLaOyspIf/OAHvW4bHwwGO4ZN0yST6Z+L0pqmDS2DtqQf8LmbZvXTxdwFCxbw9NNPU1JSgmmalJSU0NTUxBtvvNFxEbesrIy2tjYeeeSRjvny8/M76u0nT57M7t27Wbp0KQCtra06uGua1q8GfEn/cAKmG/T762LujBkzqK+v57Of/WyntLa2NsrKyrj22muprq5mxIgRHdU3AIsWLeK6664jHA7zxhtv8Mc//pGvfvWrJBIJwuEwf/3rX/slv5qmaQCiVH/VeudGTU2NOvghKuvWrWPq1Kndzrt2VwsFIR+jSyL9lb0Bqaf7T9O0gUlEliularoaN2ird8BrttlP1TuapmkD0eAO+v14g5amadpANLiDvk+wMooTvQpL0zTtWOk26ItIpYi8LCJrRWSNiPybl/5HEVnhvbaKyAovvUpEElnj7s5a1lwRWSUim0Tkl9LPXT76fQYK1W8teDRN0waanrTeyQDfVEq9IyL5wHIReVEpdUX7BCLyU6A5a57NSqnZXSzrLuBa4C3gWeAC4LmjzXx3DrTgURzFjbmapmmDTrclfaXUbqXUO95wK7AOGNU+3iutXw48dKTliEgFUKCUelO59S2/Ay45+qx3r72tvq7X1zRNc/WqTl9EqoA5uCX1dmcAe5VSG7PSxonIuyLydxE5w0sbBdRmTVNL1sHjoPUsFpFlIrKsrq6uN1nsxG8aCP3XVv9oPfHEE6xdu/Z4Z0PTtCGox0FfRPKAR4GvK6Wyezi7ks6l/N3AGKXUHODfgQdFpKA3mVJK/VopVaOUqikvL+/NrJ0YIvhM44Sr09dBX9O046VHQV9E/LgB/wGl1GNZ6T7gUuCP7WlKqZRSqsEbXg5sBiYBO4HRWYsd7aX1q/7sYrndQw89xIwZM6iuruaGG27oSM/Ly+sYfuSRR1i0aBFLlizhySef5Prrr2f27Nls3ry5X/OmaZqWrdvLm16d/b3AOqXUwY+D+giwXilVmzV9ObBfKWWLyEnAROADpdR+EWkRkVNxq4f+Cbijz1vw3I2wZ9VhR4/K2NiOoldXckfMgAtv69Gku3bt4oYbbmD58uUUFxdz3nnn8cQTT3DJJZd0Of2CBQu46KKLWLhwIZ/+9Kd7nidN07Qc6ElJ/zTgc8CHs5phfswb9xkOvYB7JrDSa8L5CHCdUmq/N+7LwG+ATbhnAP3WcqedIYJSoPqpk+WlS5dy9tlnU15ejs/n46qrruLVV1/tl3Vpmqb1VbfFX6XU60CX7emVUou6SHsUtyqoq+mXAdW9y2I3uimRx2JpdjTGmTw8n6DfzOmqu5N9G0Jvu1bWNE3rD4PyjlylFPF4nHQ63e/NNufPn8/f//536uvrsW2bhx56iLPOOguA4cOHs27dOhzH4fHHH++YJ7t7ZU3TtGNpUAZ9gObmZhKJBP5+7mK5oqKC2267jXPOOYdZs2Yxd+5cLr74YgBuu+02Fi5cyIIFC6ioqOiY5zOf+Qw/+clPmDNnjr6Qq2naMTVou1auq6vDMAxKSkpYvauFsrwAFYXh/szqgKG7Vta0wW3Ida2slCLhDxB3FCJCwJQT7gYtTdO042FQ9kijgKaMgV+ZKKXwn4A3aGmaph0Pg7KkL0pBysZ2DDKZjHeD1oldjaVpmnYsDMqgj8B42U2J09wR9DOO496kpWmaNoQNzqCPEMAirFJYmcyBLpZ1FY+maUPcIA36YImPIBbxjN0R9C19MVfTtCFucAZ9pXDEJIhFIuPg1/3qa5qmAYO09Q4itIoCQ2E7Nj5DMEQ329Q0TRucQR9oMm0UBj6VQSl1TLpY1jRNO9EN+KD/X2//F+v3rz8kPWHFERSifAT8AVIZB6UgHOi+07UpJVO4Yf4N3U6naZo20AzOOn1AEBzAwEEphSGCc4J3OaFpmtbfBnxJ/3Al8t37d7DfaWF02sTOG4ktQXY3J5hWUYDPHLTHOk3TtCMatNEvEnIfVShiEbecnHexfO6557JzZ78/7VHTNC2nBm3QDwUiADjikLIzBEz3gSa5aKvvOA6bNm2ipKSkz8vSNE07lgZt0A+YAQBSIhhOJqdt9deuXcunPvUpwmHdVbOmaQPLoA36IoJfGaRE8KkMohSmkZu2+tXV1fzsZwc/I17TNO3E123QF5FKEXlZRNaKyBoR+Tcv/QcisrOLh6UjIt8RkU0i8r6InJ+VfoGXtklEbuyfTTrAp0xSIgSU5Xa8Zhqkbd2CR9O0oasnrXcywDeVUu+ISD6wXERe9Mb9XCl1e/bEIjIN+AwwHRgJ/FVEJnmj/w/wUaAWWCoiTyql1uZiQ7oSNIM0KYugstyO13wGSUvfoKVp2tDVbUlfKbVbKfWON9wKrANGHWGWi4E/KKVSSqktwCZgvvfapJT6QCmVBv7gTdtvwkH3Yq4hFnHLDfqW7bbb1zRNG4p6VacvIlXAHOAtL+krIrJSRO4TkWIvbRSwI2u2Wi/tcOn9JhyMAqAkQ9yy8ZsGjlJkdL/6mqYNUT0O+iKSBzwKfF0p1QLcBYwHZgO7gZ/mKlMislhElonIsrq6uqNeTsAMApA2BMexDvSrr/vg0TRtiOpR0BcRP27Af0Ap9RiAUmqvUspWSjnAPbjVNwA7gcqs2Ud7aYdLP4RS6tdKqRqlVE15eXlvtqcT0zAxlZAW8KkMAZ/bVj/XXSzn5eXldHmapmn9pSetdwS4F1inlPpZVnpF1mSfBFZ7w08CnxGRoIiMAyYCbwNLgYkiMk5EArgXe5/MzWYcnt9rweNXGQzcah1d0tc0bajqSUn/NOBzwIcPap75v0VklYisBM4BvgGglFoD/AlYCzwP/Kt3RpABvgL8Bfdi8J+8afuV3/CTEoOAsnBsG59p9MsTtL7xjW8wffp0zj33XNqrpDZv3swFF1zA3LlzOeOMM1i/3u0NdO/evXzyk59k1qxZzJo1iyVLlgBwySWXMHfuXKZPn86vf/3rjmVnn0k88sgjLFq0CICHH36Y6upqZs2axZlnngmAbdtcf/31zJs3j5kzZ/KrX/0q59uqadrA1W2TTaXU64B0MerZI8xzK3BrF+nPHmm+o7Hnxz8mte7QrpXbpe00yrFoUkKTGcRwoA3Y5j98F8vBqVMY8d3v9jgPsViMmpoafv7zn3PLLbfwwx/+kDvvvJPFixdz9913M3HiRN566y2+/OUv89JLL/G1r32Ns846i8cffxzbtmlrawPgvvvuo6SkhEQiwbx58/jUpz5FaWnpYdd7yy238Je//IVRo0bR1NQEwL333kthYSFLly4llUpx2mmncd555zFu3Lgeb4+maYPXgO9lszumYWI5FqCwlULEwMlx6x3DMLjiiisAuPrqq7n00ktpa2tjyZIlXHbZZR3TpVIpAF566SV+97vfufkzTQoLCwH45S9/yeOPPw7Ajh072Lhx4xGD/mmnncaiRYu4/PLLufTSSwF44YUXWLlyJY888ggAzc3NbNy4UQd9TdOAQRD0uyuRW7bFhsYNFGUytBgjCQfzqGtNM2ZUAe7litwTERzHoaioiBUrVvRonldeeYW//vWvvPHGG0QiEc4++2ySyWTH8tq1pwHcfffdvPXWWzzzzDPMnTuX5cuXo5Tijjvu4Pzzzz9kHZqmaYO27512PsOHAGmv47WAz0ChsHLYgsdxnI6S9YMPPsjpp59OQUEB48aN4+GHHwZAKcV7770HuN0y33XXXYBbB9/c3ExzczPFxcVEIhHWr1/Pm2++2bH84cOHs27dOhzH6TgTAPeawSmnnMItt9xCeXk5O3bs4Pzzz+euu+7CsiwANmzYQCwWy9m2apo2sA36oO92vGZ2dLzmN7xmmzm8mBuNRnn77beprq7mpZde4qabbgLggQce4N5772XWrFlMnz6dP//5zwD893//Ny+//DIzZsxg7ty5rF27lgsuuIBMJsPUqVO58cYbOfXUUzuWf9ttt7Fw4UIWLFhARcWBRlPXX389M2bMoLq6mgULFjBr1iyuueYapk2bxsknn0x1dTVf/OIXyWQyOdtWTdMGNjnRuySoqalRy5Yt65S2bt06pk6d2uNlbK3bTIoEJVaEaMkoNtUnGF0coSQayHV2B4Te7j9N0wYWEVmulKrpatygL+kDhPxBMiL4SYNju9U9uq2+pmlD0JAI+uGORydmiFsWftPIaZ2+pmnaQDEkgn7I5z7hSolNPO0+RUuX9DVNG4qGRND3m37AfXRie8drue5/R9M0bSAYEkHfEAO/EtIimOpAv/q5vklL0zTtRDckgj6AD19Hx2t+b6t1aV/TtKFmyAT9oBki7T0v1xS3hJ+ri7k//vGPO4a3bt1KdXV1TparaZqWa0Mm6IdDURRgkgbHvVkpVxdzs4N+X+kbqTRN609DJuiH/G4LHsQmnrYQkZxU79x4440kEglmz57NVVddBbhdK1x77bVMnz6d8847j0QiARy+q+VFixZx3XXXccopp/Dtb3/7sNNpmqb11YDvcO21P22gfkdbt9MpFHErzhoUNnEcZWIYEPId2sVyWWUeZ1w+qUfrv+2227jzzjs7OlbbunUrGzdu5KGHHuKee+7h8ssv59FHH+Xqq68+bFfLALW1tSxZsgTTNDn33HMPO52maVpfDPig31OCIIADiFIYAv3VA8W4ceOYPXs2AHPnzmXr1q1H7GoZ4LLLLsM0zW6n0zRN64sBH/R7WiIH2Fz3PkKacKYQO1RGS8Ji2sjCnOcpGAx2DJumSSKR6Lar5Wg0CtDrLpk1TdN6Y8jU6QP4JeA127QIGJBxFHYO2ur7/f6OrowP50hdLR/NdJqmaUdjSAX9cDCCg+AjjSnuRdxctOBZvHgxM2fO7LiQeziH62r5aKfTNE3rrSHRtXK7tnQb21q2MdpySAVHsycuVJVGKQj7c5XdAUF3raxpg1ufulYWkUoReVlE1orIGhH5Ny/9JyKyXkRWisjjIlLkpVeJSEJEVnivu7OWNVdEVonIJhH5pfTX8woPI2i6de22OKQyuW2rr2maNhD0pHonA3xTKTUNOBX4VxGZBrwIVCulZgIbgO9kzbNZKTXbe12XlX4XcC0w0XtdkIuN6Kn2RyemRFBOBiNHbfU1TdMGim6DvlJqt1LqHW+4FVgHjFJKvaCUar999E1g9JGWIyIVQIFS6k3l1in9DrikL5nvLffRiUanjtd0SV/TtKGkVxdyRaQKmAO8ddCofwGey/o8TkTeFZG/i8gZXtoooDZrmlovrav1LBaRZSKyrK6urjdZ7JafQEfHawFTl/Q1TRtaehz0RSQPeBT4ulKqJSv9P3CrgB7wknYDY5RSc4B/Bx4UkYLeZEop9WulVI1Sqqa8vLw3s3Yr5A9heY9ODBhunf6JfjFb0zQtV3p0c5aI+HED/gNKqcey0hcBC4FzvSoblFIpIOUNLxeRzcAkYCedq4BGe2nHVDiUB21NGGTwkcFRYDsKn3lMrylrmqYdFz1pvSPAvcA6pdTPstIvAL4NXKSUimell4uI6Q2fhHvB9gOl1G6gRURO9Zb5T8Axb4Ae9LkteJSRwbK9Fjx9rOL52Mc+RlNTU1+zpmma1u96UtI/DfgcsEpEVnhp3wV+CQSBF72Wl296LXXOBG4REQu3q5vrlFL7vfm+DNwPhHGvAWRfBzgmAmYAgLQIjpMBfKQzDpHA0S/z2WefzU3mNE3T+lm3QV8p9TrQVd1Hl5FOKfUoblVQV+OWAcf1CSOGGPiUkBJBnNyU9Kuqqli2bBnhcJjLL7+c2tpabNvm+9//PldccUUusq1pmpYTA77DtZfv/zX7tn3Qq3mS6QQKB1P5sPDhM4Sg/0BN17CxJ3HOosW9zsvzzz/PyJEjeeaZZwBobm7u9TI0TdP605Dqe6ediIESQXDcLpZztNwZM2bw4osvcsMNN/Daa69RWJj7Hjw1TdP6YsCX9I+mRL4/3sDu+B5GpYVW/0jitsnkEfl9zsukSZN45513ePbZZ/ne977Hueeey0033dTn5WqapuXKgA/6R+PAoxMzmGRI24JSir52BbRr1y5KSkq4+uqrKSoq4je/+U0OcqtpmpY7QzLot7fgyYjCdmyU8pFxFP4+ttVftWoV119/PYZh4Pf7ueuuu3KRXU3TtJwZkkHfZ/gwENKGgJMBgqQzDn7z6C5xbN26FYDzzz+f888/P3cZ1TRNy7EheSEXwK8MUiKYygb63mxT0zRtIBiyQT8gwY6O10D3q69p2tAwZIN+OBTF9h6dGDR00Nc0bWgYskE/FHBb8IhkiBiWrt7RNG1IGLJBv/3RiY7YgIOlS/qapg0BQzbo+w0/gtvxGmSwbKX71dc0bdAbskFfRPB5LXgM5aBQWLqKR9O0QW7IBn2AAH5SIvjQLXg0TRsahnTQDwUi7qMTleVW9RxlSf9LX/oSNTU1TJ8+nZtvvjm3mdQ0TcuhAX9HbtNTm0nvih3VvMrJELVTJBUUqBbiPh8Z0yAwMkrRJ8b3eDm33norJSUl2LbNueeey8qVK5k5c+ZR5UnTNK0/DemSvkj75iu3i+WjvJD7pz/9iZNPPpk5c+awZs0a1q5dm7tMapqm5dCAL+n3pkR+MEc5rGtYR8S2sZ0yUmYBlcPyerWMLVu2cPvtt7N06VKKi4tZtGgRyWTyqPOkaZrWn4Z0Sd8QA1MJaREM7KNqvdPS0kI0GqWwsJC9e/fy3HPH/LG/mqZpPTbgS/p95cdHShyiOFi2g+MoDKPnXSzPmjWLOXPmMGXKFCorKznttNP6Mbeapml9023QF5FK4HfAcNwnC/5aKfXfIlIC/BGoArYClyulGsV9Esl/Ax8D4sAipdQ73rL+Gfiet+j/VEr9Nreb03shM0yzY1GkLMBtwRMyzF4t4/777++HnGmapuVeT6p3MsA3lVLTgFOBfxWRacCNwN+UUhOBv3mfAS4EJnqvxcBdAN5B4mbgFGA+cLOIFOdwW45KOBRBAQYWfmzdB4+maYNat0FfKbW7vaSulGoF1gGjgIuB9pL6b4FLvOGLgd8p15tAkYhUAOcDLyql9iulGoEXgQtyuTFHI+gPAW7Ha0Es3QePpmmDWq8u5IpIFTAHeAsYrpTa7Y3ag1v9A+4BYUfWbLVe2uHSu1rPYhFZJiLL6urqepPFXmvveM0WB5+hS/qapg1uPQ76IpIHPAp8XSnVkj1OuQ3cc9ZbmVLq10qpGqVUTXl5ea4W2yVTTARx++ARR3fFoGnaoNajoC8iftyA/4BS6jEvea9XbYP3vs9L3wlUZs0+2ks7XPpxJSIHHp2IDvqapg1u3QZ9rzXOvcA6pdTPskY9CfyzN/zPwJ+z0v9JXKcCzV410F+A80Sk2LuAe56XdtwFvUcn+vSFXE3TBrmelPRPAz4HfFhEVnivjwG3AR8VkY3AR7zPAM8CHwCbgHuALwMopfYDPwKWeq9bvLTjLhyKYotgKgvl2NhO7wJ/Xl7v7uJ95ZVXWLhwIeA29/zKV77Sq/k1TdOOVrft9JVSrwOHu1vp3C6mV8C/HmZZ9wH39SaDx0IoEIYkGJIhhEU6owgHjneuNE3Tcm9Id8PQLmC6Ed4RG79kjqqK5xvf+AbTp0/n3HPPpb3F0dlnn82yZcsAqK+vp6qqKmd51jRNOxoDvhuG5557jj179vR5OXErjg+FUj6GVYzm0osW9njeWCxGTU0NP//5z7nlllv44Q9/yJ133tnnPGmapuWaLul7BLfNqQCO07vWp4ZhcMUVVwBw9dVX8/rrr+c8f5qmabkw4Ev6F154YU6Ws6V+ExmVpCATJeEf1qdluQ2ewOfz4XgXhXV3y5qmnQh0Sd8T8kdJi+BTGayM3at5HcfhkUceAeDBBx/k9NNPB6Cqqorly5cDdIzXNE07nnTQ94SDYQAMscBO9eopWtFolLfffpvq6mpeeuklbrrpJgC+9a1vcddddzFnzhzq6+v7Jd+apmm9IUf7iMBjpaamRrW3gGm3bt06pk6dmtP1JDIJPmj6gJEZh+ZMCaMrKvCZg/OY2B/7T9O0E4eILFdK1XQ1bnBGtaMQMNxmmxlR+MTRd+ZqmjYo6aDvMQ0To73jNUP3waNp2uCkg34Wv/KREjBQuqSvadqgpIN+lpAZdlvwYOuHqWiaNijpoJ8lHIrg4Ha8ZmfSxzs7mqZpOaeDfpaQz310IpJBMvpmKk3TBh8d9LMEfAc6XhPb6nFb/Y997GM0NTX1Y840TdNyY8B3w5BLPvEh4LXgscnYCr/vcL1KH/Dss8/2f+Y0TdNyQJf0s4gIPmWSFkGEHrfgqaqqor6+nq1btzJ16lSuvfZapk+fznnnnUcikejnXGuapvXcgC/pb9jwI1rb1uVseUkrgS80lpOGfY10xiEa7N38Gzdu5KGHHuKee+7h8ssv59FHH+Xqq6/OWf40TdP6Qpf0D2IaJo6AqTJkMlav5x83bhyzZ88GYO7cuWzdujW3GdQ0TeuDAV/SnzTp+zldXkuqhR2tOxDLQllJINqr+YPBA6cGpmnq6h1N004o3Zb0ReQ+EdknIquz0v6Y9ZD0rSKywkuvEpFE1ri7s+aZKyKrRGSTiPxS2judP8EETTdoK7FRutmmpmmDTE9K+vcDdwK/a09QSl3RPiwiPwWas6bfrJSa3cVy7gKuBd4CngUuAJ7rdY77md/0A2AJKLv31Tuapmknsm6DvlLqVRGp6mqcV1q/HPjwkZYhIhVAgVLqTe/z74BLOAGDviEGJgYpcfALOEphdHNS0l5vX1ZWxurVHSdEfOtb3+rPrGqapvVaXy/kngHsVUptzEobJyLvisjfReQML20UUJs1Ta2X1iURWSwiy0RkWV1dXR+z2Ht+/KREMFFYuuM1TdMGkb4G/SuBh7I+7wbGKKXmAP8OPCgiBb1dqFLq10qpGqVUTXl5eR+z2HthX4S0CKaysazePTpR0zTtRHbUQV9EfMClwB/b05RSKaVUgze8HNgMTAJ2AqOzZh/tpZ2Qsh+daFu69Y2maYNHX0r6HwHWK6U6qm1EpFxETG/4JGAi8IFSajfQIiKnetcB/gn4cx/W3a/aO15TZLxmm5qmaYNDT5psPgS8AUwWkVoR+YI36jN0rtoBOBNY6TXhfAS4Tim13xv3ZeA3wCbcM4AT7iJuu4DpdrxmG4pMWpf0NU0bPHrSeufKw6Qv6iLtUeDRw0y/DKjuZf6Oi/ZHJ6YFfEpfyNU0bfDQ3TAchqlMUiLQg3vIfvzjH3cMb926lerqAXFs0zRtCNJB/zBCZthrtungOEcu7WcHfU3TtBOZDvqHEQm6j040yGClD38x98YbbySRSDB79myuuuoqAGzb7rJ75c2bN3PBBRcwd+5czjjjDNavX39MtkXTNK2d9PTpUMdLTU2NWrZsWae0devWMXXqVAC+v7GW1W25vdhanRfmhrFFbGvZRoUFoegIIgWlh50+Ly+PtrY2wK3emTBhAsuWLWP27NlcfvnlXHTRRVx99dWce+653H333UycOJG33nqL73znO7z00ks5zXtPZO8/TdMGHxFZrpSq6WrcgO9ls79kd7yWSfXuoNJV98ptbW0sWbKEyy67rGO6VCqVs/xqmqb1xIAP+j+aOLr7iY6CUgoB0gLK6l1w7qp7ZcdxKCoqYsWKFbnNqKZpWi/oOv3DEJGOFjw2R27B4/f7sawj98hZUFDAuHHjePjhhwH3oPLee+/lLL+apmk9oYP+EfglQFoEQ4AjXPtYvHgxM2fO7LiQezgPPPAA9957L7NmzWL69On8+c8n7E3JmqYNUgP+Qm5/2tOyl4Z0PRVpPyXDq8C7U3eg0xdyNW1wO9KFXF3SP4JwwO2DB8lgH6HZpqZp2kChg/4RBH1ZLXh0Hzyapg0COugfQXvHaxlRJBKx45wbTdO0vhvwTTb7kyEGBgZpcTDsE/vah6ZpWk/okn43fPh63PGapmnaiU4H/W6EzLDbbBMHnMzxzo6maVqf6KDfjWgwggKEDGS6vjP34YcfZurUqZxzzjksW7aMr33ta0dc5iuvvMLChQu7HFdVVUV9fX1fs61pmtYlXaffjZDfa7ZJBmUlkED0kGnuvfde7rnnHk4//XQAamq6bB6raZp23OmSfjc6Hp0oDsnkoS14brnlFl5//XW+8IUvcP3113cqxcdiMf7lX/6F+fPnM2fOnC7vwG1oaOC8885j+vTpXHPNNZzoN8tpmjawDfiS/g+fWsPaXS05Xea0kQXc/InpAPgMH4KQNiCRShM+aNqbbrqJl156idtvv52amhpeeeWVjnG33norH/7wh7nvvvtoampi/vz5fOQjH+mc/x/+kNNPP52bbrqJZ555hnvvvTen26JpmpZtwAf9Y8HteM0m0MsToxdeeIEnn3yS22+/HYBkMsn27ds7TfPqq6/y2GOPAfDxj3+c4uLi3GRa0zStC90GfRG5D1gI7FNKVXtpPwCuBeq8yb6rlHrWG/cd4AuADXxNKfUXL/0C4L8BE/iNUuq2XGxAe4m8P/mNIGllua02HQeMngV/pRSPPvookydP7pS+d+/efsilpmla93oSve4HLugi/edKqdneqz3gTwM+A0z35vm/ImKKiAn8H+BCYBpwpTftgBANRLC9Rydi97xv/fPPP5877rijo57+3XffPWSaM888kwcffBCA5557jsbGxtxkWtM0rQvdBn2l1KvA/h4u72LgD0qplFJqC7AJmO+9NimlPlBKpYE/eNMOCJFge01+BjI973jt+9//PpZlMXPmTKZPn873v//9Q6a5+eabefXVV5k+fTqPPfYYY8aMyVGuNU3TDtWXOv2viMg/AcuAbyqlGoFRwJtZ09R6aQA7Dko/5XALFpHFwGLghAiC7Y9OdMQmk47jC3eud8++eHv22Wdz9tlnAxAOh/nVr351yPKypyktLeWFF17ol3xrmqYd7GibbN4FjAdmA7uBn+YqQwBKqV8rpWqUUjXl5eW5XPRR8Rt+BLAE4vH48c6OpmnaUTuqkr5SquNKpIjcAzztfdwJVGZNOtpL4wjpJzwRwVAmaXHwK90Hj6ZpA9dRlfRFpCLr4yeB1d7wk8BnRCQoIuOAicDbwFJgooiME5EA7sXeJ48+28ee2d7xGnLERydqmqadyHrSZPMh4GygTERqgZuBs0VkNqCArcAXAZRSa0TkT8BaIAP8q1LK9pbzFeAvuE0271NKrcn1xvSngBmmzUlhiAN2GrwHrGiapg0k3QZ9pdSVXSQf9rZRpdStwK1dpD8LPNur3J1A8sNR2mJNdLTg0UFf07QBSPe900MRr+M1JRkcSz86UdO0gUkH/R460PGaIpFoO8650TRNOzo66PdQ+6MTUyKkLOd4Z0fTNO2o6KDfC4YySYngiKFb8GiaNiAN/F42n7sR9qzK7TJHzIALD+0PzpQgadJex2sZMP25Xa+mZdm1axfvv/8+CxYsIBjUDQe03Bj4Qf8YCvnCpDJt3qMTkzroa/3CcRzeeOMN/va3v+E4DuvXr+fKK6+kqKjoeGdNGwQGftDvokTeX/JDYZrbQGGjrCQSzD9m69aGhtbWVp544gk2b97M1KlTqa6u5sknn+See+7hM5/5DJWVld0vRNOOYOAH/WMoGnB723TEJpVsI5R3/PsF0gaPDRs28MQTT5BOp1m4cCFz585FRBg2bBgPPfQQ999/PxdddBGzZs063lnVBjB9IbcX2h+daAlcdMUidu3adbyzpA0CmUyG559/ngcffJC8vDwWL15MTU0NIm4/T+Xl5VxzzTVUVlby+OOP89e//hXH0S3ItKOjS/q9ZCiTlGHz0O9+Q+nIkcc7O9oAV1dXx6OPPsqePXuYP38+H/3oR/H73WtFbakM2xpiTKsoIBKJ8LnPfY5nn32W119/nbq6Oi699FJ9gXcQi1txIv5Izperg34vGfhJiQUi4NhgmMc7S9oApJTi3Xff5bnnnsPv93PllVd2PFYzkbb53RtbufOVzbQmLOafVMJ/XlzNpOH5LFy4kGHDhvH8889z33336Qu8g1Btay2/eOcXbGnewp8W/gkzxzFGB/1e8kmQBAlEbLcFTyB6vLOkDTCJRIKnnnqKtWvXMm7cOD75yU9SUFBAKmPzP29u5xcvbaQ1bmGXBnFGF/D21ibO/8WrXHnqWG48fzKnnHIKpaWlPPzww/oC7yDSkm7hnpX38MC6BzDF5PPVn8dWNiY66B9XIX+YhNUE2KhMEtFBX+uF7du38+ijj9La2spHPvIRFsz/EIm9Me78yyruWr2TWMrGKQ5QMCXMJflPMNNawW9HfI73tkzgwTe28di7tXz7wiksmjeea665hgcffFBf4B3gLMfikQ2P8H9X/F+aU81cNP4ivjrnqwyPDu+X9emg30vRQJhGy3t0YiqOP1J6vLOkAZl0mp3vr2XbqhXs2fg+xRWjqJw+g8rpM4kWFXe/gH5mZ2z+/sJLvPb2EgpCeXxq1IcpejPIrX99lQftJAnLwSn0Uzm1ieuC93HO7pWMWJfCpxw+7t/EysqR3Dp/ESvfn8Qtj6/hjr+v4pbzC/nCF/6ZRx55gscff5y6ujo+/OEPYxi6fcZAoJTi1dpXuX3Z7Wxt2cr8EfP5Vs23mFo6tV/Xq4N+L4UDAYhBRhTJZBx9e1bPKKWor21j/64YRcMiFI+IEAgf/c9POQ5127eybeW7bFu1gp3r1pCx0himSfnYcaxf8ior//Y8ACUjR1M5fab3mkGkoDBXm3VovpTCbk5j7Y2R2RPH2hNj/646/tq0jD3SxAR7BDXNk7gvoHgi1kA6aaPyTU6ZspZv2fcxZXMdBZbFW4E8biybyMawxSVNFl/euonH7VvZXDiH/5hwEcu2j+UrD7VSMepprp+7ivnDheXLn6W+vo5PflJf4D3Rrd+/ntuX3s5be96iqqCKOz58B2eNPqujxVZ/EnWC9yFTU1Ojli1b1ilt3bp1TJ3av0fDw1FKsb7hfSKORX46QMnoycclH31xLPdf0944G5ftZePSvTTu6fx84WhRkOIREYoropSMiFA8IkpxRZRwvr/LH39rQ31HkN+2agWJlmYASkePYeyM2YydOYfRU6cTCEdwHJt9Wz5gx5qV7Fizktr1a7GSbpfYZZVjGT1tBmOmz2T0tGrC+QVHtW12zMLaEyOzN461N4a1x31XSbtjmq15+3nNXo0jitnj5/KHtI/XazOomIMZcfj8yKf457ZnGNkUw0DxtG889w8vYEOggZBRQNAeR4usJOwE+HS98K/xjYQck93G2dwYOY9X948GUzhpwk6uGX0fRek4yeR45p78OUaP/gg+3+C/gTBp2fx9Qx3PrdrNytpmZo4u5PSJ5ZwxsYzhBaHjnb1O9sX3cce7d/DnTX+mMFjIl2Z9icsmX4bfyG3xUUSWK6Vquhyng37vvV+3CYMkpVaQkorxVI07ifx894/Ltm0uvfRSvve97xEKnVg/uHb9vf/aGlNsWu4G+n3bWgEYObGIifOGUzG+kOa6BI17YjTujrvve+JYqQOBMhjxUVIRpaDUQKmdxBo3Ub99Lc173fsiIoVFHUF+zIxZ5JeUdcyrlMJuSaOSGZTloDIOynKwUxmad+2iftt2mmp30rKvDrHBFB95RaUUlgwnv7iMaF4RBkanebPfyf6c1duqhHz4R0Twj4jiHxGB0gB/W/ka7656j3hhCW8Gy9nWVIK0ZBgVruebRQ9wfvO7RJ00cSePPwTn8ERlhi1qJwX+MlL1Z1K/ezaFoSitznbGTXyVffa75Dt5XLonzJcT7xExMuxRJ/Mj40KeSUxD5fmZMb2BK/J+xUizFjApKjyZktIzKC05g/z8akQGR9VPPJ3h5fV1PLt6Ny+v30c8bVMU8TOnsoiVtc00xNIATBqex+kT3APAKSeVEAn0oXIjHYf9m6FhM6AgWOC+Qt57MN9t2NFFgSVuxfntmt/y/9b8PzJOhqumXsW1M6+lIHB0BY7u6KCfYxvqtmNJKxXpACXllVRNnMKyZcsoKyujra2NxYsX4/f7+e1vf9vndTnKIW2nERFMMTHF7PMp4Lq1a5k4fDiZ+gbshnoyDQ1kGhowQiH8I0fiHzUK/8iRGOFwj5eZjFlsfmcfG5fuZefGJlBQPiafiTXDmVAzjPySwx8AlVK0NaZo2NnMtpXr2Ln+Pfbvep9U2w7cJ3L6MHyj8YWqKK6YTHnVOEpHumcFRQUBQskMmZ1tpLbsx9oVR6V695t2cMg4aWwng4MNPsEXCuCPhgnm52EGfYjPQPwGeO/iNzDzAh1B3sgPICLEW5p597VX+PvyFWwoKeO9gpE07MrD35TgvNAy/tX3GNOtHYjAttR0Hh82g7+M2EFtegfDQqNwms5hy5YpzBhVyn98fCozRxfy8xc3cO/rWygu2cXok17hg7aVFDslXLyzhMWpt8k322hQY/iZcyGPWB8iPqKQiSObuNB6lvmjt5GxPwDA7y+hpOR0SkvOoKTkDILB/r2jPO04rG5NsLQlxrLmODHbZmI0xORIiInREBMjQYr8PQ/CrUmLl9bv49lVu/n7hjqSlkNpNMD51SO4sHoEp55Uit80cBzFuj0tvL6xntc31fPWlv2kMw5+U5g7tpgzJpZz+oQyqkcVYhoH/S05DrTUQv1GaNjkvW+E+k1uenfEdIO/dzBwgvk86Xe4Q9WzT1mcFxrF18tPpTJv1KEHjI7hAjB9YCXA3/O/wU7Z0EE/tz5o2EtC1VNhGZQUVVA1dXZH0AdoaWmhsrKSLVu2UFxczLe//W2ee+45RITvfe97XHHFFbzyyiv84Ac/oKysjNWrVzN37lz+53/+B4UimUkSs2LErBjxTJyDvyNDDEzD7DgImIaJiYlPCT4HTAdMGwxHIbbjvjI22DYqk2HDzp34//Ur3W6nWVLiHgSyDgT+UaPwj3LfbV+IrSvr2bh0L9vX7sexFUXDI0ysGcbEecMpHnHklk1KKRp37+yostmxZiXpRAJEGHHSBMbMmE3FxGqC0TE074qxf1Md6Z2tGG2KKD6KfULEEBQOVnA/rf5akuYWMuZ2fC0pAg0pjDYTVAgzXERoRAX5J00mOmkGgXGV+MpK3GBuCBnLYs/G99m+ZiU71q5k94b12JkMYhgMP2kCldPci8KjpkwjEAp32oaG2u1sXv42m99ZyobmFtbPm8U7wQkktgjjGrfz2cBfuZKXKZQ4cbuIVdY5vDypipeib7A7uYux+eMJtn2U5WvHUFEY4frzJ3PJ7FEYWQFpVW0zNz62kjW7mpk3tQGn6Bk2NK2j3K7g49tGsSizhFL/HlpVIb+xP8rv1EfZd9JIxkZaWDwMzptq0dj4Og37X8Wy9gOQlzfNPQCUnkE0MoFAoASRo28eWJe2WNYcY2lznGUtMd5rjZNy3N/u6JCfQp/J5niKpHPg9zws4GNSJMSkqHsgmBQJMikaoszvQ0Rojlu8uG4vz63azWsb60nbDsPyg1xYPYILqiuYP67k0MB9kKRls3Trfl7fWM9rG+tZu7uFfOLMCu/jo8NamZtXz0mym0jrVrcUn8l6Ml4gH8omQOlEKJsIpeOhdAIYPki1QrIFUt6rY9hNfyuxi9utnawXi5kZ+FZLgjmtTeBY3e5LyxFa7CClt+49mq+ib0FfRO4DFgL7lFLVXtpPgE8AaWAz8HmlVJOIVAHrgPe92d9USl3nzTMXuB8I4z4r999UD4443QX9/3r7v1i/f313i+mVKSVTuGH+DYcdv6u5mUarlhEZRWG4hAmzFnQK+gCzZ8/mV7/6FbW1tdx99908//zz1NfXM2/ePN566y3ef/99Lr74YlatXkXJsBLOPvNsbvjhDUyfN70jyAd9QfLMCBHlh4yNyliojA2ZDGI7kLExHIVhOxiHuStfCdhG+0twTGHLrr28+cxtqJJCpKQYX1kZofLh5NsBiposog1xInVtBOqbMfc0IHvrcXbvRaVSOGLSUDKNvcNqqC+biWMGCJGgMr+RcVXCsAnlBEa7BwizqOiQs5J4S7NbJ7/yXbaveo/WhjoACkrLGD1yJBUhHyXJVlK7a7GafBiZMvyB0ZgFo7CH+UlH6mkJ1dNg7KXB30yLL4mSNI4VQKX8+NIQtG2SAT+psCB5KfILGxkW3Eu0zcLf6OBrVJhNgtMmpDM+MmYEouX4Rownb9LJlEydS0lJJc1bd1C7dhU71qxk98YNOHYGwzQZPn4iY6bPJJ1I8ME7b9O8by+NY8tYueBDrKCawMZmLmz8B5/1/Y0PGetwlMH21BzeD5zHO7MDvGA9RX2ynqkl0ylJf4y/vVNCwPTxpbPH84XTTyIc6DrwZmyHe1/fws//ugGfIXzytP2sjP+RD5o/YKRVxUe3TOYK5zUqg+tJKz+P2Kdzn28ha6dVUx1u5b9Om8/s4nxa29ayv+E1Gva/RnPzcpTKACBiEgiUEwiUEwwOJxgcRjAwjID3HgwOJxAcRsBfgq2E9bEES1viLG+OsbQ5xrakW6USEGFmfpi5hVHmFUSpKYwyIujWWdtKUZtM834sycZ4ig2xJBvjSTbEkrTZ3o84bZNfnyZUl6RtbxyloCQ/yEenD+fTs0Yyd2xJpwMiSkHrbqjfAM214AtBIA/8IUi2QqzOHd9cC01bceo2YsTrDuxXZbBDlbPHX4lTMoGiyqlUTZ5FdORUyBveZXXNkWzZv5GfvHUrr+1bTqkR4RPpEUzenSS1vwm7OYa0OVhOGCtUgB2JkAkFSYfCpP1+lBgYSpFHEjEcrrnlj71ad7sjBf2enFvdD9wJ/C4r7UXgO0qpjIj8F/AdoD1KblZKze5iOXcB1wJv4Qb9C4DnerIBJ5qIP0SjBbY4pBKxLqdpD9yvv/46V155JaZpMnz4cM4880xeW/IqgYifGSfPoC3SRmtrK+OnjWfnB1v58Kw5hDKCz3JQaQuV2X/owkUQUxDDQHwGBHzeZ0GZgmOA0x7oBWwUtlLYKBwUVgOsmNZCi9pPs9pEc5NNoinrqCHAMO81DUQJFc3jqd47lzFNs/E5ERzVRtB6i5LG9yhq3IEvrtj+mrDJNLB8JpZpkAr4sEIBrKAPyzCxRLC8P2wDCNqKooQi2CaoJoMtTSbvlRfSVjAMe8QozIpmfLQQUJuJOmsotOOUtLRS1tLCRGlmgTRRJF3vf+JAkzvoKGEPxdTKMHaZZewOlLGvvIjGqjxi+UEyBQ55gTh5tJKfeJmCJU+R19KGP9aGmUrghKF4fiFpczh2upTGPY3s+fMjqJBi1ymjWPbxK9mQnkT1+yu5ufkOPmW+SlEgRrM9jDfbPkvd2AtZO2UzTzc8RlNrEzXD53FG8dd44o0IbckMV8yr5BsfncSw/CNfA/KZBl88azwXVI/gPx5fze9fsjl57PV89ZSdPPrBffx20l94JTWZMzdfwiXGP7g88Dqf5WVeXjWLe6IX8zEp4vRQE9+dM5c5VV+iqupLZDKtNDUtI5GsJZ3aSypdRyq1l2RiB83N73ScFcSIspFJbGQyG2UKm5lEEreFUImRZGaojUvLFScXhJhTWEpBuAS/v+SQawimCGPDQcaGg5yXlb6vJcmf3tvJs6v2sG5HE5YCifoxT8onVhZiV6Gf34rNo9t2MHbbBkYn9zOidTfD9m1leP0WCpMthIwMeUacYaqeUhopMOMYcqBcaWWEhGWQtPwknHySKkhCwmw2x/CefzKrMiexdkclqR1+jCVtTAi9wMy8JmaVxphQYuOPRJFwHioUJV5XS+vOD0js3YW1vxHVGCOZTPOPcYplEwS/LZy5sZDKxjHsiZSyMlTMvkgJdZES9meKcA664cofzCAhIRMOkA6HUWEfwUCaf3GcnDfB7TboK6Ve9Urw2WkvZH18E/j0kZYhIhVAgVLqTe/z74BLyEHQP1KJvL8EfSYGQlqEgJVxu2NorgWjDZRDa2sLW7d8wKRiUPEGUs07qdu3ipgILelmGmJ1FDlRwobJ6P02vgwUJw0i+1NEGpIAKFNhmAoJe+8GYLiFjo6CR6cSSHbJR8DGfXFQKUWEZsfh3oY4Dia2EjLKT8qBVhtaLUVbxiGegbZUAfFECVYyD6UUSr2LktdwSGE7JhkJsLfAz46iMaQNP2kJYHnvKcOHZfixCJDBR0YCpCUApkHUn6LAiFMmzZRLM0XSRDkbKJellEkTpbRi2IeeBLaZIeqMYurMIjaZY3jbrKbFyKdNosQljKUC2LYfn4QwrDYCxIk6bRQ5LZQ5TQxXDcy136civgRfwgHveOooYR9F1BrD2OEbTm1wGGvD09lRVMG2gpHsKSjHb1pEaSOPNvLsVsJWnA2hqcRbw1z23lP8PPYzTjY2kTF9bE7O5w11Icb8U1kx4u88vv3HxHbFOHP0mcyIXsofXjN4uSHOGROL+I+PT2XKiN5dzBtbGuX3X5jPY+/s5EfPrOUnj+az+KyfMXzqCu5bfQ+/n/Ygb8ZnMX/Tjzg/9DY15l85J3kL69+u5J6iS7jYyWe2tYfvz57FvFHjKCs7p9PyHaXYFE95VTUtLG1uY1PCvdBuopjgb+UC31YmGR8wwV5FobWBTKwRYsA+eK/jp+bzzhrcMwXTlwfKQSm3+FEXC/DG9mH8Y/sI1tWVohBG5jVz0fjNnFy6jpGhHZBOkk4aJHflgRUgYCnyUilKU62MTNVRZjV5f5TumyUme0IlbA6PpCGYT2MoSksgRMznx3Ay5CdbKYi1Eo3HCMXSBONpRlubqIxt4hMZyFgmG2UsK32TeS85mccS43ik3iBsJ5nZspHZjauZvX8DRjBDfVkJewqL2R0dw7rpDewu2YASC6tpPi11H+UZyYMSAEUgmMEIQSYcwApHcMJ+VNhEhX0QFEKZFOUJm1Fxh6q4wUmNwuhMqF/uuchFO/1/AbLPQcaJyLtAC/A9pdRrwCgg+ypIrZc2IPlNA1E+UuIQEROU455GWgFa4gm++O0fcsGF59JaVMDkmpP5w+8f5pKPX4TT0MKK15fz31/+Jhu2bMGXgYDtwwj6MYJ+fAVhAiPzgAzYFmJbiOMcHLYPOFztWHeVZk4GadqKCZhAAIgAnW5h8nmvfr7hOCl+6n3F7POXsC0wmuX+apr9ebRKPm0qSpsTJI0fwxchz++nJKAoDJoEQn78IT9FQZOSgIERMDD8BvjcC7NloTLKfeUUUEA6niYWi9HW1sbatjbebmkm3riLVONujGQ9EbuJImmlzG6ixlnHJ9Kv42vrXF+2TxWyyyhjl6+cncFyasPDuaLpZT6WfpM8SbJTjeTl1kW0ln+Cko+U847vGR7f/GXSm9OcX3U+Z5Rfwf+8anHb1v1MGp7H/Z+fx9mThx31fhMRPjV3NGdPLudHT6/l/7y0lfHlo/jPSx5gffw57lt9Hw/OXMm7bfOYs/E2Ti1YxTie5Kctd3DjP37H70sWssgsZMSbb/PdySMJlk9iaSzD0tYk77TEacq4Qb7YZzK3MMplI6LUFEaYXRAhah5a/eQ4KZLxfSTbdpOK7yGV2EsquZdUeh/peB0tLRvIOHHq4oUs3TuV5XXT2dxSSZQEs8Pv861hTzMjuIkRqolIq0W0wSKStgjadqf12CIkfSHS/jCxSCHvBCawPTSKDeFKVkfGsyY8lhYzQswIuo81PQyfcg/iUWIdB/MDn1uJEuM0lnGW9Q/q9xdT21DBxoYxvFU8A05qX4rCl7+K4LDnMAKNqORErORCUuEqrCnBjqCuQiZk0uQn4wxLxhkeb2F0G4xv8jM9kM+USBHhoiKMkgDmuCBmfgCzMIBZ0D/3WvQp6IvIfwAZ4AEvaTcwRinV4NXhPyEi049iuYuBxQBjxozpSxb7hc80EPykJQ0iKMPHmZddi+M4KNtm4UfO5bvXfpHIPpt/PvVcNix5j498/NOIafC/bv4uldVj+KC5FiMo+IssDCeJISkMuw0z2YgCMphYYpImRIoACnFjuXLf3bguB97VwbG+86Eie1yDSnJd+uudlqG86duXGBSboOHgFwfTUPgMhZhu/3KOYeD4wPaZZEzB8vmwfD7Spg/HFDKGRUaSWJIg4ySxSJJRcRwnToYUaeX+8doIImBIBgMbJIlDLRkyWI5FRmWwVec/etLeq7Vn35UgjIiOYEz+GCoLKqkcVcmYKWOozp9FZX4lEX8E27Y7DgpNjS1s2LaPPds+oK1xB0ZyH1GngSLZT5lqpNrezEfTS/G32SRUgCWZ+eyLf5xhs8+meL7i5f1/5KkPngIFC8cvZOHYq3jo9QRfe24XZXkBfvzJGVxeMxqfmZsSXGlekF98Zg6XzBnFfzy+mqvvWcFnTzmFP338kzy26QF+v/b3rJ+zjNXNpzFj048YU7yZMc6T/HvjA3z5zT/xRPFH+EHwM2zctwu/YzG6cS/z9tUyqWEvE5saGBFP4jMD+MwgPsPPRjOAT3yYhg/ET0YM4uIQwyEuGeKSJoFFSixS2FhAhhIUJYjKUCwtnCOr+bz8jVHBegok7v44W9ztsZWPpFNC2hlNs1NCyiklYZeRdMqI2+WkVDEOPoxAEH8khJkXoNCXYorTTEWmiQ8l1oAFkjKwLSFt+UjZQtoxSCsfKcNH3DRp9QmtfpNWv482fx61/hKS/hApf4i0L3BgBweAEd4LkHgGf30cjO1EeAS/s5mMv5J4wReIMp6iRBv5yUbyk3FKm1KMsX1MDkWZMKyCkaPHUlJRhq8whJHnR7q5AN1fetR6x6veebr9Qq6Xtgj4InCuUip+mPleAb4F7AReVkpN8dKvBM5WSn2xu3WfiK13AN7ft5OM0USFZRJuyHSkKz9I2MQMGIjhAA7i2BiOg5EVehVg4SONj6QKkMJPSvlJ48fCh8KtvVFuVDyw4q5+J1nVPHLQwKEVQMKerZv41qoYbo2RYIr77kRNUnkmbSGDmEDHOYaTxHBaMGz3ZdpNBKwm/FYTht2M4bSC0wq0YRMHur6qXOAvoDBYSNgfJmAECJgBgmaQoBkkYB743P7uN/xHHJ+9jOzxhhjsje9lR+sOdrTsYHvrdne4dQf7k52vkZSFy6jMr6Qyv9I9MORXMqbAfS8Munfu2rZD4+4Yddvb2LutiW1b91DXsI380DDmn1mNf3obv9t0P3/Z9hf8hp9LJ17K5RM/x+NLY/zmtS0AXHPGOK47azz5of67hzuezvCzFzZw3z+2UJYX5JaLp1Mz3s89K+/hTxv+hChhVv1ZTP3gTFpKNjPDeY1LzH8QlO5bk+RKWvloSYeJp8O02EW02OU0OxW02hW0OiNJqBFgRskryiOvNIBRmsYqaiMebaYt2ESz0cB+u569iT3sadtDXaIO1e2pbc8ZysCnfBiYoAzARDABH0r8CH6UCEnZis+JMDZ+MhObRlPsmAwrKaeyqpIxJ1VSUVFBYWHhMbnDtit9brJ5cNAXkQuAnwFnKaXqsqYrB/YrpWwROQl4DZihlNovIm8DX+PAhdw7lFLPdrfuEzXob66rIyn7GGU5BDAwcfApGzM7sCs3sKfwk8IN6Cnlw8JPRnwoMUDA8VrVuMVe3Au1grs8x8JQGej001ZZ/ycr1UuRI4wD9nywnW9uegYlyivlu/8MJ45pNyN2K6hWHOI4xHAkQ1dCRoiiQBGl4VLKImWURcooCZV0vIpDxZSESigNl1IULMJn9E+vH0nLpiGWZn9bmoZYipZkhuH5QcaVRSnPD3b6w2tLt7GjtfOBYHvLdra3bmdffF+n5RYECg6cIRx0UCgJlrC6YTX3rLyHV2pfIeKL8Jkpn+Gzk6/mr2vi/PzFDdS3pblk9kiuv2AKo4qOrr310VhZ28SNj65i7e4Wzps2nFsursY2G7hrxV089cFTBAgye++HGbF3AuuDCWaynnyJk1EmNiY2RseZn/su3n/uP0NMDAx8mJgYmMqH4fjx2X58tg+xA4jtQxw/pvJ7fx0maRUh4RQCQqTAT6BckSmJkypoIRFpotXfRJPU05CpY29iL3tie7AOat4YNINURCsYHh3OiMgIKvIq3PdoBSOiIygNl2Irm7Sd7nilnBSWbZGyUwfSnXSnzyk7RdpJd/psORapTIpYKkZrvJW2RIx4Kk4qk+Ik4yQuGX0J4ytPYuTIkeTn5x91gLdsh20NcTbXtbmvfTE217VhO4qnvnr6US2zr002HwLOBsqAvcDNuK11gkCDN9mbSqnrRORTwC2AhVvcu1kp9ZS3nBoONNl8DvhqLppsHi+1TW00Z7YxIpOhwJaOoJ7BRwY/GXEvYtqGgWMIGVOwTS+ot5e5lY3ppDGUhTgWoiyUdwXWESenJZhse7bs4etrv35IuolJgb+A4kAxJeESyqPllEfLKQ2Xdgrm7QE97Mt9IFNKEU/b7I+l3UAeS9HQlmZ/LJ2V1nlcPG0fdnnRgMm48ihVpVFOKotmDedRGOlc6k5mktS21h5yQNjRuoNdsV046sAZTNgXJpFJUBgs5KqpV/HZKZ/l3a0pfvzsOjbsbWN+VQn/8fGpzKosyvk+6gmrvXnnixsImAbfvnAKV80fw9aWLdy54k5e3PYieVLAjN0fIj9WjFImAcdHyDEJOAFMJ4hhBzDtIIYTQGw/DgaOYUPIQYIOEnIgYKMCdse78mdQPhvHl8Hx2dimhW1ksE2LhMRopI56q4698b20WW2d8myKSXmkvFMQz35VRCsoCh7aDLhdOuOQzNjkBXydm3SeIJoTFh/UtbG5LsamfW0dQX57Q5xM1r0LwwuCjC/PY9LwfG7+xLSjOpgMypuzpkyZctxOnQDqW1PsTX0ARIg6JVimYBmQMcUrHSnAQZSF4bgvUW4NJ8oN6ofWwgs+w4ff8LtVFz63ysJvHOiLRpBO02d96NE4pRQb3t/A2AljEWkvuRmICAEj0G/7NJbKsK0hzs6mhBusvZJ5ewBviKW8knqaVKbr6qGAz6A0GqDEe7nDQUrzOqcVhP3saU6ypT7W6VXbGCfrb4viiJ9xZVHGleUxrizivUepKosccru+ZVvsiu3qdCAYmTeST038FNsbMtz6zDpe21jP2NII37lwCudPH3Fcf5/ttjXE+O7jq/jHpgZqxhbzvy6dwcTh+aypX8Mv3/0lS3YtodgsZbg5Elss3H9pLJXGUhZpJ0VapQ8pcfeWT3zkB/I7BfDs4WHh4YTNIhJpaElatCYztCS896zPLVmfW5NWp2mSXrcYPkMo9n4LZXnu76PU+52UdRp23/vUNcNBHEexqznB5roYm7MC++a6GHWtqY7p/KYwtjTK+PIoE4blMb7cfZ1UHs1JFeCgC/pbtmwhPz+f0tLS4/aH1ZKw2BHbihJwzAIvqGcQlQEyXom9M0MMfOJzA7npd+uhDXfYb/hz0sXCkSilaGhooLW1lXHjxuV8+c1xi237Y2xtiLOt3n3f7n3O/sG3C/tNN1DndRHI24N7nptWmhckGujb/klnHLbvj7PVOwh8UB/rGN7Tkuw07YiCEFXegeCksihVZVHGlUUZUxIh4HMvwu5rTfLzFzfwx6U7yA/5+dq5E/ncqWM7xp8olFI8sryWW59dRyyV4ctnT+DL54wn6DNZumcp96+5n5gV67g2EjJDna63ZF9XCZpBgr5gt9Mq5aOxzWF/m6KuxWZfi0VDLH1I4HaDtkVrKnPYxmjtAj6DgpCfgpCP/LD7XhDykx/yURD2kx/0EfKbNCXSNLSlqfeq+xra0jS0pYgd5oww7DfdA0Oe+7tr/72V5QUOOmAEKY4ECPgMkpbNlvpYp+qYTfva+KC+rePgA1AQ8h0I6h3BPUplSQR/ji7md2XQBX3LsqitrSWZTB5mrv5n2Q71sf0oI92RZojR0S2C28Ihq5sEMTFOgM6uQqEQo0eP7ngOa28opWiIpdnWEGNbQ9wN7g2xjvemeOfS4IiCEGNKI1SVRhhb6larjC4Od/whHe7O0+Mhns6wtT7OlvoYWxtifFAXY0t9G1sb4uyPZX/HMLo4wtjSCO9sayRtO3zu1Cq+du4EiiKBI6zh+KtvS3HLU2t58r1dTBiWx/+6dAbzqkp6vZxUxmZvc4pdzQl2NyfY1ZRkd3OCPc3JjuHG+KFnBvlekC4I+w8MtwfsgwN4yEd+e4D30kP+vv1e2q/9NLSlvINCqvNnb3h/zD1opO2uzzjzQz7asg5SIjCqKNypxD6+PMr4YXmURvvv7PlIBl3QPxHE0xmm/eBxrjzD5IsLTmZEdAQh34nZq2ZvOI5iX2uKrQ2xjoC+vSHufY7TljpwUdcQGFkUpqo0ytjSiPdyg/uYksgJFdT7ojlusaXBPQhsqYuxpSHOlvo2xpZG+dZ5kxlXNrCenvby+/v43uOr2dmU4KpTxnDDhVMo8KoUMrbD3tYUu5sS7GpOsrspwe7mJLu8993NSerbDj1rKwz7qSgMMbIoTEVhyHuFqSgKMbIwzIjCUJ+D9rGklKI1lek4S8g+a9gfS1MU8XcE+HFl0RPut97Xbhi0LkQCPsoiRZAYTlVh1VEtQylFLG3TFE/TFLdojKdpjFs0xdM0xtzPTfE0zQmrU100dH3/VU8P4F1NplDUtabYvj/e6fTUZwhjSiKMKY0wr6qEsaURN6iXRhhdHCboO7F+7P2hMOJndqSI2cfpomyunTN5GC9840x++sIG7l+yhRfW7mVUUZjdzQnqWlOH/Nbygj43iBeFmT6ygBGFbiCvKPICe2GIaHBwhRIR8c5E/APuoN6dwfVNHWOjiiPs2O/2yJexHZoSXsCOWzTGugjknYYtmuPWYU8hAfKDPoqifgrDfswubsfupsl+N9Mdmjq2NMpZk8oZUxqlygvuFYWhnN1IpJ04okEfN31iGhfPHsntL7yPoxRnTCw/pIReURTqOAvQBgddvdMHX3nwHZ5fvYdwwKQ12XVbdnCv1BdFAhRH/B3vxZHAQcP+TtMURfz9eqFH07TBS1fv9JPPnToWv2lQGPZTlBW8iyOBA8PRQJ9bnWiapuWKDvp9cMpJpZxyUunxzoamaVqP6foDTdO0IUQHfU3TtCFEB31N07QhRAd9TdO0IUQHfU3TtCFEB31N07QhRAd9TdO0IUQHfU3TtCHkhO+GQUTqgG1HOXsZUJ/D7ByvdRyr9QyWdRyr9QyWdRyr9QyWdRyr9fRlHWOVUuVdjTjhg35fiMiyw/U/MZDWcazWM1jWcazWM1jWcazWM1jWcazW01/r0NU7mqZpQ4gO+pqmaUPIYA/6vx4k6zhW6xks6zhW6xks6zhW6xks6zhW6+mXdQzqOn1N0zSts8Fe0tc0TdOy6KCvaZo2hJzwQV9Efi4iX8/6/BcR+U3W55+KyL8fZt5FIjKym+UvEpE7c5bhA8u9RESm5Xq5Xaynrb/X0YM8fDdruEpEVh/HvDwrIkXHa/1Hq7ffo4icLSJPe8NH/Rvuj/3Vn78HEblMRNaJyMsiUiMiv+xm+o791MW4rSJSlqu8HQ0vD6u811oR+U8RCfXnOk/4oA/8A1gAICIG7g0L07PGLwCWHGbeRcARg34/ugTo96B/gvhu95P0jIj06WluSqmPKaWacpSdQa+f9lfOfg9d+AJwrVLqHKXUMqXU1/pxXcfKOUqpGcB84CTgV/25soEQ9JcAH/KGpwOrgVYRKRaRIDAVOE9ElorIahH5tbg+DdQAD4jIChEJi8g8EVkiIu+JyNsiku8td6SIPC8iG0Xkf3eVCRG50jsarxaR/8pKb8sa/rSI3C8iC4CLgJ946x6f873SOW8/F5E1IvI3ESn30sZ727RcRF4TkSle+nARedzbB+95eUVEnvCmXSMii4+0fd7wZd6+2AtEve18EPgOMF5EGkRkl4i8ICLhbvJ0v4jcLSJvAf/7cNP1cF9sFZEyEYmKyDPeNq4WkSv6uJsPXs9dIrLM218/zNEyu/oeXxGRGm+4TES25mJdWets319VXgn6Hi8PHd9bL5d3GxD2fg8PeMlmV8vt7fcsIjcBpwP3ishPpPPZTlRE7vP+rt8VkYu7mL/UW/8acWsLTqgHVyul2oDrgEtEpMSLYz/xfr+r2n/D3na/IiKPiMh6EXlApBcP4VZKnfAvYAswBviit1N+BHwMOA14DSjJmvb3wCe84VeAGm84AHwAzPM+F+A+I3iRl14IhHC7fKg8aP0jge1AuTfPS8Al3ri2rOk+DdzvDd8PfPoY7BsFXOUN3wTc6Q3/DZjoDZ8CvOQN/xH4ujdsAoXecIn3HsY9sJZ2s32rgFHZ0wCLgduBDDAPWAY8A1zdTZ7uB54GzCNN18P9sRX3bPBTwD1Z6YU53u/t+8v0fmcz++l7zP4NlwFbveGzgae94UXt0x/Fetv3V5X3vc320v/U/r0dxTKzfzOHXe7RfM8H7Y/sffDjrOUWARuA6EHT/BK4yRv+uLfPy3L5uzja/X9Q2gpvf3wKeNH7jQ3HjUEV3jY1A6NxC+5vAKf3dJ0D5cHoS3CrcRYAPwNGecPNuNU/54jIt4EIUAKsAZ46aBmTgd1KqaUASqkWAO8A+TelVLP3eS0wFtiRNe884BWlVJ03zQPAmcATOd7Oo+HgBnKA/wEeE5E83P3zcFYBIOi9fxj4JwCllI27DwG+JiKf9IYrgYlAwxHW+w/gfhH5U1baecDJgA3cg3sgXQFUdZMngIeVUnYPpuupVcBPxT0re1op9dpRLONILvfOiHy4f4jTgJV9WN4h32PfsndUtiilVnjDy3EDdr8sN4ffc7vzgItE5Fve5xBuQTHbmcClAEqpZ0SksQ/r60/tO+R04CHv73SviPwdNxa1AG8rpWoBRGQF7nf1ek8WPlCCfnu9/gzcUugO4Ju4G///cANMjVJqh4j8APcL741U1rBN7/ZL9o0O/XoBpocU7tG/SSk1uycziMjZwEeADyml4iLyCge2pcvtU0pdJyKn4JaYIiJSivtj/QHw7fZ1e3+EeT3IU8x771XeD0cptUFETsY9I/xPEfmbUuqWviyznYiMA76Fe9bY6FV55fq7b9/vGQ5Uw/b37+vgv4NeV+/0Yrk5+Z6zCPAppdT7nRJFhudo+ceEuFXOVbhnKkdy1DFrINTpg1vSXwjsV0rZSqn9uKdwH+LARdx6r/Tw6az5WoH2evv3gQoRmQfuzpWeXzR8GzjLq/s0gSuBv3vj9orIVHEvMn8ya57sdfeZV887qotRBge2+bPA695ZzBYRucybV0RkljfN34AveemmiBTilsgbvYA/BTg1a/ldbp+IjFdKvaWUugm3lDoO+AtwddY0k3Cr1egmTx16Ol13xG21FVdK/Q/wE9wzkFwpwD1INXtB5cIcLPOQ79Eb3grM9YY/zcBgiYj/SBPk6nvO8hfgq+112yIyp4tpXsXdt4jIhUBxH9aXc178+r/AE0qpRtyq6yu8v9Ny3DOVt/u6noES9Ffh1ju+eVBas1KqHrekvxr3i1+aNc39wN3e6Y8JXAHcISLv4daV9ajkpJTaDdwIvAy8ByxXSv3ZG30jbn30EmB31mx/AK73Lir16UKuF3AnAPu7GB0D5ovbLO7DQHtp9irgC962rgHaL2z9G2512CrcU+1pwPOAT0TWAbfReT8fbvt+4l1cWo1bhfN74CxgIzDBS/8VnX9jh8vTwXo63ZHMAN72vvubgf88imV0SSn1HvAusB54EPdMtK8O9z3eDnxJRN7F/RsYCH4NrJQDF3IPJxffc7sfAX5vvWu8zwf7IXCmN/5S3DryXhO3mWsuWwW+7H3vb3t5+qKX/jhuleF7uNcRv62U2tPXleluGAYAEakG/kUp1eX9CJqmaT2lg76madoQMlCqdzRN07Qc0EFf0zRtCNFBX9M0bQjRQV/TNG0I0UFf0zRtCNFBX9M0bQj5/0rCiXCycv5VAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def against(posn):\n",
    "    ref_vec = y['hidden_states'][11][0][posn]\n",
    "    return plt.plot(words, [project( y['hidden_states'][11][0][i], ref_vec) for i in range(0,13)], label=words[posn])\n",
    "\n",
    "lines = []\n",
    "for i in range(0,13):\n",
    "    lines.extend(against(i))\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f8c2bff",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  },
  "metadata": {
   "interpreter": {
    "hash": "ba85587d0f2f867a4e78210392c7bc7374047d8b061b2b933587dccc022acd03"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}