examples/ghn_all_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "22462f65",
   "metadata": {},
   "source": [
    "# Loading GHN-3 and Predicting Parameters for all PyTorch ImageNet models \n",
    "\n",
    "## Paper: Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?  https://arxiv.org/abs/2303.04143 \n",
    "\n",
    "**ICML 2023**\n",
    "\n",
    "Authors: Boris Knyazev, Doha Hwang, Simon Lacoste-Julien"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a1a75bb",
   "metadata": {},
   "source": [
    "## 1. Install ppuda (optional, if not installed yet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93835e26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright (c) 2023. Samsung Electronics Co., Ltd. All Rights Reserved.\n",
    "# All rights reserved.\n",
    "#\n",
    "# This source code is licensed under the license found in the\n",
    "# LICENSE file in the root directory of this source tree.\n",
    "\n",
    "!pip install git+https://github.com/facebookresearch/ppuda.git\n",
    "!pip show ppuda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "85c7ed79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch 2.0.1+cu117\n",
      "torchvision 0.15.2+cu117\n",
      "Make sure PyTorch: >= 1.12.1 and Torchvision >= 0.13.1 are installed to make our code work properly\n"
     ]
    }
   ],
   "source": [
    "# Import required modules\n",
    "import inspect\n",
    "import sys\n",
    "import os\n",
    "import types\n",
    "import time\n",
    "import json\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "import torchvision.models as models\n",
    "import joblib\n",
    "import matplotlib.pyplot as plt\n",
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "sys.path.append(os.path.abspath(os.path.join('..')))  # to import ghn3 from the root folder\n",
    "from ghn3 import from_pretrained, get_metadata, Graph\n",
    "\n",
    "\n",
    "print('torch', torch.__version__)\n",
    "print('torchvision', torchvision.__version__)\n",
    "print('Make sure PyTorch: >= 1.12.1 and Torchvision >= 0.13.1 are installed to make our code work properly')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69ee744c",
   "metadata": {},
   "source": [
    "## 2. Load GHN-3\n",
    "\n",
    "As the largest and best performing GHN-3 model (`ghn3xlm16`) takes about 2.5GB, it takes a while to download the model.\n",
    "\n",
    "The model is downloaded from HuggingFace: https://huggingface.co/SamsungSAILMontreal/ghn3/tree/main.\n",
    "\n",
    "We made four main GHN-3 variants from our paper available: `ghn3tm8, ghn3sm8, ghn3lm8, ghn3xlm16`.\n",
    "\n",
    "We also released other GHN-3 variants such as ablations at https://huggingface.co/SamsungSAILMontreal/ghn3/tree/main/other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ad84b1c9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading ghn3xlm16.pt with 654365184 parameters is done!\n"
     ]
    }
   ],
   "source": [
    "ghn3_name = 'ghn3xlm16.pt'\n",
    "ghn = from_pretrained(ghn3_name)  # one of 'ghn3tm8.pt', 'ghn3sm8.pt', 'ghn3lm8.pt', 'ghn3xlm16.pt' (default)\n",
    "print('loading %s with %d parameters is done!' % (ghn3_name, sum([p.numel() for p in ghn.parameters()])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a59e25f",
   "metadata": {},
   "source": [
    "## 3. Predict parameters for ResNet-50 and compare to the downloaded ResNet-50 for sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "80533f2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter prediction for ResNet-50 done in 3.385 seconds on cpu\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAF2CAYAAAC72fnJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABbzElEQVR4nO29eZhU1bX+/56aq6u6qrqbnptuGkTBAVRAbNHrRORn4oyJek3EIRoTJCrxJul7owiKjXgTp+CU+EWTK1dFg9HcqDGomFwBA4rBCRkauqHnoaq6umuu/fvDUHGftbGqi/L2aVyf56nn4aze59Q6++yz67Dfs9bShBACDMMwzIhiGmkHGIZhGJ6MGYZhDAFPxgzDMAaAJ2OGYRgDwJMxwzCMAeDJmGEYxgDwZMwwDGMAeDJmGIYxADwZMwzDGACejL/ivPnmm9A0Dc8999yX/l3jxo3DlVde+aV+x7PPPovi4mKEQiEAwO7du6FpWvrzf3GeX2V8Pl+6r2+44Ya0/ZFHHkFtbS2i0egIemdseDIeYd5++23cfvvt8Pv9I+1KzmzduhUXX3wx6urq4HA4UF1dja997Wt48MEH/0/9SCaTWLRoERYsWAC32y397brrrsNvf/tbnHDCCWS/d999F+eddx6Ki4tRUFCAo48+Gg888MABv8fv96OsrOygJ/df/epXOPXUU1FeXg673Y76+npcddVV2L17d87HPO2006Qfn89/rFZrTse88sorD3hMTdOwb9++dNvHHnsMv/3tb5XHiMViePTRR3M+t0Mdy0g78FXn7bffxuLFi3HllVfC5/ONtDvD5u2338bpp5+O2tpaXHvttaioqEBrays2bNiA+++/HwsWLEi33bZtG0ymL+/3/6WXXsK2bdtw3XXXkb81NDTg29/+NrH/6U9/wrnnnovjjjsOt956K9xuN3bu3Im9e/ce8Htuu+02DA0NHbS/7733Hurr63HeeeehqKgIzc3N+NWvfoU//OEPeP/991FVVTXsY/7Hf/wHvvvd70q2wcFBXH/99TjrrLNy8vN73/seZs+eLdmEELj++usxbtw4VFdXp+3f+ta3AADf+c53pPYOhwPz5s3DL37xCyxYsACapuXkyyGNYEaUe+65RwAQzc3NeTtmKpUSQ0NDWbV94403BACxevXqnL7r61//uigtLRX9/f3kb52dnTkdM1fOO+88cfLJJ0u25uZmAUCsXLmStA8EAqK8vFxceOGFIplMZvUdW7duFRaLRSxZsuSg+u1AbNq0SQAQTU1NeTvmb3/7WwFAPPXUU3k75l/+8hcBQCxdulT5dwBi/vz5km3/ua1duzZvfhxK8DLFCHL77bfj3/7t3wAA9fX16f/27f9vaiKRwB133IEJEybAbrdj3Lhx+Pd//3ey7jZu3Dicc845ePXVVzF9+nQ4nc70fwf9fj9uvvlmjBs3Dna7HTU1NbjiiivQ09MjHSOVSmHp0qWoqamBw+HAmWeeiR07dmQ8h507d+Koo45SPtWXlZURPz+/ZvxF//X9/H/VP/nkE1x88cUoLi6Gw+HA9OnT8eKLL0rHjkQieOWVV8gT3BexatUqdHZ2YunSpTCZTBgcHEQqlfrCfW688UZceOGFOOWUU7L+nuEwbtw4AMjrstWqVavgcrlw/vnn5/WYmqbhX//1X7PeZ9q0aSguLsbvf//7vPlxKMHLFCPIRRddhE8//RT//d//jXvvvRdjxowBAJSWlgIAvvvd7+LJJ5/ExRdfjB/96EfYuHEjmpqa8PHHH2PNmjXSsbZt24bLLrsM3/ve93DttdfiiCOOQCgUwimnnIKPP/4YV199NY4//nj09PTgxRdfxN69e9PfBwDLli2DyWTCLbfcgkAggOXLl+Pyyy/Hxo0bv/Ac6urqsH79enzwwQc4+uijh3X+qrXFn/3sZ+jq6kqv+X744YeYNWsWqqur8dOf/hQulwvPPvssLrjgAjz//PO48MILAQCbN29GLBbD8ccfn/X3//nPf4bH48G+fftwwQUX4NNPP4XL5cJ3vvMd3HvvvXA4HFL71atX4+2338bHH398UOu6enp7e5FMJtHS0oIlS5YAAM4888y8HLu7uxuvvfYaLrnkErhcrrwcMx6P49lnn8VJJ52U/vHIluOPPx7/+7//mxc/DjlG+tH8q86Blim2bNkiAIjvfve7kv2WW24RAMTrr7+ettXV1QkA4pVXXpHa3nbbbQKA+N3vfke+N5VKCSH+uUwxefJkEY1G03+///77BQCxdevWL/T/T3/6kzCbzcJsNouGhgbx4x//WLz66qsiFouRtnV1dWLevHkHPNby5csFAPGb3/wmbTvzzDPFMcccIyKRiOT7SSedJCZOnJi2/frXv1b6+0XLFFOmTBEFBQWioKBALFiwQDz//PNiwYIFAoC49NJLpbZDQ0OitrZWNDY2CiEOfnnn89jtdgFAABAlJSXigQceOOhj7ufBBx8UAMQf//jHvB3zpZdeEgDEQw89dMA2UCxTCCHEddddJ5xOZ958OZTgZQqD8sc//hEAsHDhQsn+ox/9CADwP//zP5K9vr4ec+bMkWzPP/88pk6dmn56/Dx6AeWqq66CzWZLb+//b/iuXbu+0M+vfe1rWL9+Pc477zy8//77WL58OebMmYPq6mqylPBFvPHGG2hsbMSCBQvS4k9fXx9ef/11fOtb38LAwAB6enrQ09OD3t5ezJkzB9u3b08r+b29vQCAoqKirL8zFAphaGgIV1xxBR544AFcdNFFeOCBB/C9730PTz/9NLZv355uu2zZMsTjcfz7v/971sfPlpdffhl//OMf8fOf/xy1tbUYHBzM27FXrVqF0tJSfO1rX8vrMa1Wa1qsGw5FRUUIh8N5EUAPNXgyNih79uyByWTCYYcdJtkrKirg8/mwZ88eyV5fX0+OsXPnzqyXDmpra6Xt/ZNaf39/xn1nzJiB3/3ud+jv78c777yDxsZGDAwM4OKLL8ZHH32Ucf+9e/fikksuwaxZs/CLX/wibd+xYweEELj11ltRWloqfRYtWgQA6Orqko4lhlG4xul0AgAuu+wyyb5/HXT9+vUAPntX+Z577sHSpUvJK3P54PTTT8fZZ5+NhQsXYvXq1Vi8eDF++ctfHvRxd+3ahfXr1+OSSy6BxZKfFclQKITf//73mDNnDkpKSoa9//7rw29TUHjN2OBkO2j3Tyy5YjablfbhTG42mw0zZszAjBkzcPjhh+Oqq67C6tWr0xOnilgshosvvhh2ux3PPvusNGnsF9NuueUW8tS/n/0/Vvsnhv7+ftTU1GTlb1VVFT788EOUl5dL9v3C4/4fottuuw3V1dU47bTT0mvFHR0dAD5bk929ezdqa2vz8trehAkTcNxxx+Gpp56SgiZyYdWqVQCAyy+//KD92s8LL7yAoaGhnI/Z39+PgoKCgx6vhyI8GY8wB5ps6+rqkEqlsH37dkyePDlt7+zshN/vR11dXcZjT5gwAR988EHefB0O06dPBwC0t7d/Ybsf/vCH2LJlC9566y0yKY4fPx4AYLVaM74lMWnSJABAc3MzjjnmmKx8nDZtGl577TXs27cPRxxxRNre1tYG4J9CaktLC3bs2JH25/P84Ac/APDZJJOv98TD4XBeItVWrVqFCRMm4MQTT8yDV5/x1FNPwe1247zzzstp/+bmZmk8M/+ElylGmP0Kt/5Vpq9//esAgPvuu0+y7/9v/De+8Y2Mx547dy7ef/998uYFMLwn3v309PTgk08+kdb73njjDeWx9q95f36S07Ny5Uo8+uijWLFihTIyrqysDKeddhoeffRR5aTe3d2d/ve0adNgs9mwadOmrM9n/5rn448/Ltl//etfw2Kx4LTTTgMA3HnnnVizZo30ueOOOwAAP/7xj7FmzZphv6mQSCSUS0DvvPMOtm7dmv4xy5X33nsPH3/88bBePctEd3c3/vznP+PCCy9EQUFBTsd49913cdJJJ+XNp0MJfjIeYaZNmwbgs8ipSy+9FFarFeeeey6mTp2KefPm4bHHHoPf78epp56Kd955B08++SQuuOACnH766RmP/W//9m947rnn8M1vfhNXX301pk2bhr6+Prz44ot45JFHMHXq1GH5+stf/hKLFy/GG2+8kZ6oFixYgKGhIVx44YWYNGkSYrEY3n77bTzzzDMYN24crrrqKuWxenp68IMf/ABHHnkk7HY7/uu//kv6+4UXXgiXy4UVK1bg5JNPxjHHHINrr70W48ePR2dnJ9avX4+9e/fi/fffB/BZhNdZZ52FP//5z+nXwzJx3HHH4eqrr8b/+3//D4lEAqeeeirefPNNrF69Go2NjekIuJNPPpnsu/8peMaMGbjgggukv2malj7WgQiFQhg7diwuueQSHHXUUXC5XNi6dStWrlwJr9eLW2+9VWp/2mmnYd26dVn/iD711FMAvniJ4sorr8STTz6J5ubmrF5Re+aZZ5BIJHJeoti8eTP6+vry+r7zIcUIvsnB/IM77rhDVFdXC5PJJL3mFo/HxeLFi0V9fb2wWq1i7NixorGxUXrNS4jPXhn7xje+oTx2b2+vuOGGG0R1dbWw2WyipqZGzJs3T/T09AghDvyKluqVsEWLFgkA4o033kjbXn75ZXH11VeLSZMmCbfbLWw2mzjssMPEggULSATe519t23/8A30+/6rfzp07xRVXXCEqKiqE1WoV1dXV4pxzzhHPPfecdPzf/e53QtM00dLS8oXn8XlisZi4/fbbRV1dnbBareKwww4T9957r7Lt5zlQvw0MDChfjdMTjUbFjTfeKKZMmSI8Ho+wWq2irq5OXHPNNcpozGnTpomKioqMfgkhRDKZFNXV1eL444//wnZz584VTqdTGT2p4sQTTxRlZWUikUhkbAvFq20/+clPRG1tbfq1SkaGJ2PmkCGRSIjDDz9c/OxnP0vb9k/GDz74oOju7pbepf4y+J//+R+haZr4+9//nrdjBoNBYbFYxC9/+cu8HVMIIcrKysQtt9yS12P29vaK7u5uMhlHIhFRUVEh7rvvvrx+36EErxkzhwxmsxlLlizBihUr0ik097NgwQKUlpYO693nXHjjjTdw6aWXZi0iZsNbb72F6upqXHvttXk75ocffohwOIyf/OQneTsm8Jnoul/4/DwrV66E1WrF9ddfn9fvO5TQhMhByWGYUUIkEsFf//rX9PaUKVNIzgwmf6xbtw7xeBwAMHbs2C8UcBkZnowZhmEMAC9TMAzDGACejBmGYQzAlzYZr1ixAuPGjYPD4cDMmTPxzjvvfFlfxTAMM+r5UtaMn3nmGVxxxRV45JFHMHPmTNx3331YvXo1tm3bllE8SaVSaGtrQ2FhIScTYRhmVCOEwMDAAKqqqjLnLvky3pc74YQTpHcMk8mkqKqqyqqUTGtr6xcGA/CHP/zhz2j7tLa2Zpz78r5MEYvFsHnzZimxi8lkwuzZs9MpCb+IwsLCfLvEMAwzomQzr+U9N0VPTw+SySTJwFVeXo5PPvmEtI9Go1KGqoGBgXy7xDAMM6Jks+Q64m9TNDU1wev1pj9jx44daZcYhmH+z8n7ZDxmzBiYzWZ0dnZK9s7OTlRUVJD2jY2NCAQC6U9ra2u+XWIYhjE8eV+msNlsmDZtGtauXZtOLZhKpbB27Vpl5QK73Q673Z7xuI2NjTn5k0gKYrOYM/+XIZ5IEtt/3rM8b35ZNdr1cZHIvKPifJqWL8ubX6KA1l/ThjLn6o3Y6O/6vYuX5sUnAEhp8vFNIpXdjgpZpGlZ/vyy6vyKZ+nXZwn6ZJYula9jrn5poONbgH6fCltKHoOL776HtMnVL2ecju+wNbspyJqUz2nJ8rvy5hdUL5Rl+SaXzW6Tthffvjg3H/Al5TNeuHAh5s2bh+nTp+OEE07Afffdh8HBwQPmtmUYhvmq86VMxpdccgm6u7tx2223oaOjA8ceeyxeeeUVIuoxDMMwn/GlVfq44YYbDrqgIsMwzFeFUVN2KaFbwzUlqOtRxVqwP0TX8BIReV3UaqNtKkpye9/ZW0B9sJfSemGpIPV/3z6/tO0w0XVr4cjukmkWeX0uNnknPVaEVuiNm2lfJKOyH6KT5qv1DmUuoKlahQu4rfRYAdrOgZh8rDjth5hFsVZqyWIdXuGYxUrXmpOgfeN1y9vd/XS/Agf9gnCYXttsHCuxdhPbvoR8PZyC9k0yRf0qBH2NdMhkIzY9ZkUkmS1Cz6coKeeULoqHSRutl+pFQTu9X1qK6DjRIxRrv7ub9xGbt1J+kaDC6yBt7AW0WrrVSu+Xvm5/Rr+yZcRfbWMYhmF4MmYYhjEEPBkzDMMYAJ6MGYZhDMCoEfAsFnlBvapaEYigUfGhu2+I2Ho6ZNUlkPIflG+fx1NaQmxDgvqwZxeNNIxEZAFiSKOCRFGFLys/hE7gdPfQvjGVdhJbf6CW2HpD8iuJjqRq2GQW8FRhB0UBKrA5wlS4KgzJ35nwBEmbxJCX2PyejG4pHbOIGLElElToiYZl/30KETEWVvVNZkFK5dgebRyxfWidKO9ljpM2R5k+JLaCGBWpQHclJFNUyNzrpqJbUifqpgQV8MaZu4jNGS4itpaiMRn9UuV/EAl6juFu2S97BR0kXo9CiLdSsbGP6qk5w0/GDMMwBoAnY4ZhGAPAkzHDMIwB4MmYYRjGAIwaAc+qEzP6+2mY1tgaejol9ZOJrfmDFmnb3EXFlGiWRVCsNnmhf6Cfii7vvEmjgN7+6y5iu/g7M6Tt6klUMIgFsoncAsyOiOyng/aXI0LFDdMHNH+IPzVB2hZ2Km5FCzMLeHEr7RtnhAolZbvp9XCm5HZ+zU3aDPioGJhSZEfTo6pN5nBRgXiwj55jWaUcqblvHxWp4KERi+ZIiLbToYGej9nWR21ueXztaT+CtElU1hDbKe1URE6aM4/7mCLjYJ+VimD9Y3zSdqlC7BLl24itrE0R6agUQWUGIzTjYDxGx1ewSx6/NXV0PI8/horB3bvptXUrhMtc4SdjhmEYA8CTMcMwjAHgyZhhGMYA8GTMMAxjAEaNgBfXpRMMdtD0f0WlVLiI76MVqU+YOk7a3rqDRnN1d/Rk51dMFoia2/eSNkETFRaWPHwWPZgufWW4i4p1gYhCIFKQjMjRYlWtVPDyvjeO2D7tH09sdnu1tC2OfY+2iVMRRI81TsUUZ4yeo9lHRapIUBYW7UHaDzZF2sOoPXMZpJQioqyzmwpGxYo0jv6g3M6coMeKJqlYl0xkTu0pFLensNMQueqQHOFZWfU2aeNM0fPRrPR8VP7rsSnKg8Xt9DpGLfKx+mpoCkqEaapaWyEVmwPOzKXZXA4quno8EWJz6kolJVL0fPraaNSsyUTFulCIzh25wk/GDMMwBoAnY4ZhGAPAkzHDMIwBGDVrxvrVxtZOumbcPUgzU3kL6O/Nju3yvlYbzWhmNmeTVQtI6WIKzDSxF2Y2TCC2YJCu4YX65fWtkJ+eo0mROUqFVbcOus8/kbTpSpTR/eyKMkjePdJ2eIiepLAqMoBlwYCTfl9KkWnLmZDXSgMl9JrFFcEbGjKvZasqtXt99NYI+ek1c5TI66CxBF0Xt7noOcayWDM2KbK2OSJ0P7eQx0m8j64rCzs9lkWj66lRE9UW9KQUZaqKrH5ii0fl67jXQ/tB9NHruMftI7YsYncg9DcjgJQicKakXB6/Dhv1a9BP18ALC+m41EyqYmK5wU/GDMMwBoAnY4ZhGAPAkzHDMIwB4MmYYRjGAIwaAS+qk/AmHF5N2kTCVKwxa/QldpNNFl1SinItyPzuOwBAJGXRoLCAvtgu4lRQCQXod0bDsmhgsSleks+SuC44ICnopY4dRgXChEJYMvn6pW2Xok+zQSjEjoSFCiUDPvqMEHEmdNvZZa/LBpUIEx2ix1f5P6QL+okrBk58iAaoZPMUJAT9PusQFbygK7NkT1CRNxWlfsVA22UjR9kU/VDRTsd4wiZ/Z0GMlvkqUohuKkE1GxKCnmNFNc2YB12/qqTnYJBe/74+WiIqn/CTMcMwjAHgyZhhGMYA8GTMMAxjAHgyZhiGMQCaELkul385BINBeL205AnDMMxoJRAIwOOhpak+Dz8ZMwzDGACejBmGYQwAT8YMwzAGgCdjhmEYAzBqIvAaGxtz2k/TVBE+meOMhCKSalnT3Xnzy2KmkW6JZObLIcz0fJbduSx/fplo5FEilTk9pkkRlbV06V158QkAXJrs16DILmWn3U6j326//b68+WW1yOkX4wkaiaZCU0R93nVXfvrL5qBjJBbJLtWjPgLznnvuIW1y9WsgRK9FoTu7KNNkUu7X5cv/M29+JZK0nJLFTMsuKYnK6TibfrEoJx8AfjJmGIYxBDwZMwzDGIBhT8ZvvfUWzj33XFRVVUHTNLzwwgvS34UQuO2221BZWQmn04nZs2dj+/bt+fKXYRjmkGTYa8aDg4OYOnUqrr76alx00UXk78uXL8cDDzyAJ598EvX19bj11lsxZ84cfPTRR3A4FDWJskS/CtbrpSWD3KE2Yhtjpmt44bi8/phSZSFLffEL2v/0S16LKyikpWz8MboultJo10d1mchsVkUZ+35F1i6VX7oO08z0WKlBum4d1ujxbbqS6yYbXX9MZpHIzZyk+5VodG3OKmh/RbQ+abvMOUjaDBZS37t6qjL6pVq/LfLQdUSnh14zzSxnPgsGQ6RNSpGZLOSn66d6zGb6rGSO0/Fl9sj3VUy13KnIaBYdoja7K/OUkFJc7N27O4gtEdZlR3PQY/db6HV0uKkeUDamKLNfGu0vU5weK6HLAGgp6CVtkn1jiM0Zp+MkVkivd64MezI+++yzcfbZZyv/JoTAfffdh5/97Gc4//zzAQC/+c1vUF5ejhdeeAGXXnrpwXnLMAxziJLXNePm5mZ0dHRg9uzZaZvX68XMmTOxfv165T7RaBTBYFD6MAzDfNXI62Tc0fHZf1XKy8sle3l5efpvepqamuD1etOfsWPH5tMlhmGYUcGIv03R2NiIQCCQ/rS2to60SwzDMP/n5DXoo6KiAgDQ2dmJysrKtL2zsxPHHnusch+73Q67nZZ/0aNfOnf3UgHE7nIRW2IgSmxjCuUyQugjTdBJD3UAv2Rxps9PRUrNTEWEsKBfkCqRl2gSCvGxMDu3oNekRJiKD6EovfwiTEU96IRFt42eT7LASmykjSJgZXcnFUoOE7QclCiR/7dl99MSPqF2KrqhOKNbUCUuTAgqlCVd9Pgel3yNrE46lpO0GlhWAl5SIZRZ7fT5KdAvH0soynUVFChKODkVt38ksxJrUgiLnkIqsIV14z6mOHbUTPvZa3Nn9EHpl0KkNHkDxOaGboxb6QUyFVMVtMdCz7EgmL/n2bw+GdfX16OiogJr165N24LBIDZu3IiGhoZ8fhXDMMwhxbCfjEOhEHbs2JHebm5uxpYtW1BcXIza2lrcdNNNuPPOOzFx4sT0q21VVVW44IIL8uk3wzDMIcWwJ+NNmzbh9NNPT28vXLgQADBv3jw88cQT+PGPf4zBwUFcd9118Pv9OPnkk/HKK68c1DvGDMMwhzrDnoxPO+005RrbfjRNw5IlS7BkyZKDcoxhGOarxKjJ2qbPClbgoKqb00/3K99Ho2s6amWBQxRSMcWMzAILAGi6DHBhUUnadA7RMlLudmpzRPZJ2wW+btImac0uK5jQK3ghKpQIQf+34i6gwmLUIgscCRsVSpLZZMJLUonC7qai2CfJamLb+6n8nf/fkV2kjaWYPiS4w5mvo1BEYLbuppFh3iBtV1AvR2r29PlJG7udClIF2WQrUwh4SRcVSguE3IefRum9cViMRqz2Dfrpd1oy/w82qRiCpeX0HEtc8jkOCXr9B0JUrHUXUBE0HlcIy3oUQ9Bion3YX6ebE+x03NQ105OMBek9mxT5m0JH/NU2hmEYhidjhmEYQ8CTMcMwjAHgyZhhGMYAjBoBT5+G0KooI+NURHhFxlAxwGeWxZmQiXZDOJFdORihK+sUc9HIHVeM+qDV0GNFAvKxXCX0HM1RmgpThaZ74yVZRM+nUCHqWSyqqDxZBEs56bHMivJWxCczFVOEg/bN+i3HEpu7fpO03RM6irSxDuwitpAn83XUFMLSUIgKOJ++10JsfxmS21VNLCdtqsbT83YqohgJiki35CDtZ79PHhPFQdqmu5AeK5Sk4YnjkDklpFkRbDlGcZ8JXWTbuFIq1vb1K8Q6qp2itzcLAU8xBE0avY6mlByVa4rRMd9RQffrt9F7r6Qji9yxWcJPxgzDMAaAJ2OGYRgDwJMxwzCMARg1a8b65SCLiWZaikXpOo8PNFl9i11e11NljlJUrFf7pQuucCXojgUWmjkqHKe+FhXL2eQSipIxJJjjQH7p1oztFkXJGDtd/EuA+m8pltddTQofvigqM33sJA0o2KPRoIyqyveIza4LRmnx9pA2xXYasKINZfYrHqPrkaUVdD3VWUDLOnV3ydfW56PPN1176RgcO86X0S+TmfZzS4ReH1uv3M5eTLOLiX20vxBX9E1p5jV2s4WOS5udXttYVL6vhvyKcROjfepQlPUC6D2qJ2Wh5zMQpAFEpk75+Akfvf6FfXR+sSr6XmhZThRZwE/GDMMwBoAnY4ZhGAPAkzHDMIwB4MmYYRjGAIwaAc+qW5uPWOkL5GGNLsTHTYoX51OySKElqPiQWfb5x76QX/p2p+hL8ykr9cGpEK5sFtlmyy5BmxKz7sqqsqqZFWV34ooSQWadkJRMKXonC13RaaUBMeNS9JqFq6lYEw/LopEvSvtZoxW2snLMoQg8UWVy8ynKGbl8sghqVgQQlZiyCyDSk4pRH2rtVDzTB0Q1D9H96rz0HMeYshODCYryRl2dVKTUj5tggA6uRJKOpWQyN1HMEqGCtGZXiPMDcvEyB00ch0hCEdmSyrG/soSfjBmGYQwAT8YMwzAGgCdjhmEYA8CTMcMwjAHQRDahU/+HBINBeL20vAnDMMxoJRAIwOPxfGEbfjJmGIYxADwZMwzDGACejBmGYQwAT8YMwzAGYNRE4DU2Nua0n9VKI2niivSVelIpGgV0993L8+aXxUJ/BxOJzCVcVHLrsmVNefPLZKI+pFJZ/GYrgpOa7pL9ytWnz3yQr4fJlEXZIgBQpPtsuuuuvPllc8p+xMLZRY8lFOPrHt34ytkvVaBYljJ9Shepefeyu0ibXP1SZX/N9vUBky6SdunSpaRNrn71K8Z3keI+UJHQPc/e00T9yhZ+MmYYhjEAPBkzDMMYAJ6MGYZhDMCoWTPWry11dtFUS9OPm0hsQ1GaTerDLfuk7epxtDRLrn4JRdn3aJRmaKsoK6TtEvL6Y29vP2njKqAZ5pRosmNhxRq1dYhm8vKGqa+JAnl9M6JPoQdAE5mHkllRet7tof2wo/lTYrNCfmG+uoqWRYKZrsMODijS0OkwKfwaGKClskwRmimwcKJc4qhQUeK9bScteWQ3Zb6OQpEdLx5xE1vYI2e+C8foeC529tEvCPuJyWbKvIirihMzx+n1EDZ5/KaStCyWqpyS2azKTJf5OmqKRel9EZoVMBKSvzNeQIPMuhVLxuVWOr68tuzWlrOBn4wZhmEMAE/GDMMwBoAnY4ZhGAPAkzHDMIwBGDUCnn5tPhygJXycpbS8Tbyftutql0W9icfUkzYhf3Y1j/R+2Ww0yGT3jn3ENu0EKrJs3SKLLGZkGdigQvfyvk0hbpiKqUgVC9PvdO6UhSsz1bEwVJpZYEkmqdjh76Hi1sTJ9Hrs/KBb/r4een1cJbn1V0rhV2UFvT4vP7GB2A4Py+Pr3GuOI22G+qkoGvZnDg7RFGWRzFZ6rBK3LOoJE70WOyKVxOawUAG3MtWZ2S/FWApqtAyWRcj3QkmC+rVPo8KfGfQcCxXnpEclLB5e7CM2R608gMe56H6ndm8jtm47Hfi/6aF9mCv8ZMwwDGMAeDJmGIYxAMOajJuamjBjxgwUFhairKwMF1xwAbZtkx/nI5EI5s+fj5KSErjdbsydOxednZn/68MwDPNVZliT8bp16zB//nxs2LABr732GuLxOM466ywMDg6m29x888146aWXsHr1aqxbtw5tbW246KKL8u44wzDMocSwBLxXXnlF2n7iiSdQVlaGzZs341/+5V8QCATw+OOPY9WqVTjjjDMAACtXrsTkyZOxYcMGnHjiiTk7mkjIokEyQV0fM4YKeANdNMIn0CZH5dhsVPixF1AhToVVt6/TQRf0O3bSKEBvuSL6zStHo+3eRaMMLfbs/IrBJm2HPLQfihWZwxLxdmKLHl4hGyIK8UlkjiiLKbLltexpIbYffPNfie3wI2WB6D+ueYi0OePrJxCb25PZL4uVClK+ItrPcxeeQmx3L1wjbceiVAycOutwYkuEaXQl8UtxfUK2McTWF5Cvdae5hLSZBCpImQQ9fkqZ8k3nlyJCrjDsIzahtUnbtSduJG1a/nwksVUVjSO2WCrzuLcI6rsjpLiHBmWhfE8/vT/vcdJowV5FdrfCLANis+Gg1owDgc/U+OLizxTRzZs3Ix6PY/bs2ek2kyZNQm1tLdavX688RjQaRTAYlD4MwzBfNXKejFOpFG666SbMmjULRx99NACgo6MDNpsNPp9PalteXo6Ojg7lcZqamuD1etOfsWPH5uoSwzDMqCXnyXj+/Pn44IMP8PTTTx+UA42NjQgEAulPa2vrQR2PYRhmNJJT0McNN9yAP/zhD3jrrbdQU1OTtldUVCAWi8Hv90tPx52dnaioqFAcCbDb7bDbM784bbHIL2bbHfR3JJWk66JdbYr12gp5bdleQLuhryu7oI94TF53sxfQF8gL3PT4vYq1bFjl70wpXvq3ZBnXYIP8krxQdHFMUf5Bs9D/mYiAfKyUuYi0MWWRVcumqLpy5BHHENt9P/4dsR01Uw5auPOx60mbN19+l9hUgRN6EnHaD8279xLblBnjie3OBy+Rtp9YQZfjxtTQ/xXabVn4pahmYje1EZvLLl8Ps6CBTjGHj9jifvqWkzeLkiCJJF1rtiiyI8IVlTaf300ftCrr6Rq7yU/HVyKb66hR34MROi73huX7bMhRStqICL0/U16aYbBUU9zHOTKsJ2MhBG644QasWbMGr7/+Ourr5UipadOmwWq1Yu3atWnbtm3b0NLSgoaGhvx4zDAMcwgyrCfj+fPnY9WqVfj973+PwsLC9Dqw1+uF0+mE1+vFNddcg4ULF6K4uBgejwcLFixAQ0PDQb1JwTAMc6gzrMn44YcfBgCcdtppkn3lypW48sorAQD33nsvTCYT5s6di2g0ijlz5uChh+hrSAzDMMw/GdZkrErEocfhcGDFihVYsWJFzk4xDMN81Rg1WdtSunV+TzFVpLr30cxRNkUQhqdEflPb30tLs2im7JbT9e3iijIv5XW0rEtPKxU8LCb55f0Cu+LyZFnaXOjK55hiVAwyCxuxBRUyQomupFLMTl+kt0eyyV5FRRhPEd2vZoKH2Lra5Axze7ZTUazQR1/UF7HMHabKJjc0SK/jzg9phrnqStnXaSfVkDbxoSixOew0QEmP2Uz7a8ikGBMxnUhlVZTY6u0iNju9/NASmYUyFW5FBriuuCzEHT5wKfVBkQEulqDXHxY65vSoPA+6aaY1n12+tnVuuqdI0uszqHhpQAvk1l8qOFEQwzCMAeDJmGEYxgDwZMwwDGMAeDJmGIYxAKNGwDPrfjbcinRJba00E1Y0QgWc2vFyxE144GCSE8niTyRMo59KKqiANxSifsWTcrkZd6FCYcmWpNxhRT1UpBIKMcij72gAqbjsh00himWjK1oVwlJnZy+xuYqoEOfSCT3dbX7qQxZv+6jQFM8khR4abRUO06jM9jZZNHa4FLeURm3Z+BpXRLq5UlSITcRlAa9IUwhNGr3+oBplVtfRJOj5+AvpPWTTSWo2QcW0SHSQ2JLW3O7HpCICz5mkJyl03RMP0zZJfSMA5gi9t3MbcWr4yZhhGMYA8GTMMAxjAHgyZhiGMQA8GTMMwxgATeSqenxJBINBeL1U8GIYhhmtBAIBeDyKyMLPwU/GDMMwBoAnY4ZhGAPAkzHDMIwBGDVBH42NjTntF0/QF+etWdQuslho19xxxx1588vtolmuQoM0uxeFLvE3NS0jtlz9clro73M4oQgY0BGL0hfnf/6Le/LiEwA4daW5wtFs+gowKSSRpcvk/joYv2w2eSzFYnS8qbDa6PhaslgeX7n6NaQIYihwZnerW6xyuzuW5G/Mx2PUL1U/qDDrAlnuvGtp3vwaitJgngI7LRGmQujux2WKezFb+MmYYRjGAPBkzDAMYwB4MmYYhjEAPBkzDMMYgFEj4OljUxIpKuCUVRQTW1eXn9j6OuTsaJVjx5A20XgkS8fkzQIz7VK7mWarEikq9DitslCmFKmyLAel7y93N804FSsrJbZAgJ53gUsWM6KxMGljU5WI0qMqsaMob1RW5KbH15XGqSgvI22aP20jtpQjs19mRaa64jE08EjfDwAQ02UFS6aoGBQPUwG0v4dmK9Oj6C5UltKMdtv3yMfq6aTXevxh9N5wOukXhAao/9n4VTaG+vXph926/Wg/WHz03tCLogAQF5lFZM1EHRtbQ8/7w537pO3YEN2vu7uP2IqKaSkmT0Hm8lnZwk/GDMMwBoAnY4ZhGAPAkzHDMIwB4MmYYRjGAIwaAU/TqQYO0IXzmmoq6vjG+IjtuXfXSdtJG13ALy2mgoTaMXnzg+27SZO27QPEJuJUkCifXCFtF1VQH0rdtNyU0i1df6VqS0ibRJiKSGOPqCK2tl17pe3KQiq6dEVjxEZQRMMVltBjhQQ9Vl21LDZ2NwdIm5ZOWnarto6KlHqSSXotPvloH7F5fXTMFRbLzzOuEh9p09tJfc3mKUiVT1FY6VgdUy6Piffe2kXazDipmtgiQ1lcsyz96minom6oVxaDHZW0lFV5Pb1nB8NURI73hoiN+JWijtkU0bblY+TsaV0Ben/2x+gV6uui5aA841jAYxiGOaTgyZhhGMYA8GTMMAxjAHgyZhiGMQCjRsAz6yLP+rvpgv6NlzxAbDcvvYzYnnhVTrW3Yslq0iYeyi4Vos0mR2XNvvRk0mb89Apis1qp2LDr3T3SdnAPjcDr6aBigwrNYpO2hyw0oqwnQm2/21ZDbAtOlLd7X/+EtCmssBGbHqciLaG7hO5XU1dJbG3b5PPe8g4V2OwO+mwxNJQ5knIwRNsEe6nt443dxLbjQ9kPm4kKRjPPHk9s5dWZBWJNWRGNio3JmBw1F+iiUY0ehVA6OKCIAtQyV2GLJ2ibYB/tL39KFvUuuXgGaRP20/3a19NISs2cOe2tz0MjN80KAa/QI49D8zgq8g7soKJrKNRBbL1+2i5X+MmYYRjGAPBkzDAMYwB4MmYYhjEAo2bNOJmS18rqJtN1xUUrfkBsf/7DW8S284oWaXvC4XWkzYAl80vmABDTrddtfWsbafPK49QHC+haVkGhvH5aVEbX+eyu7IJRREJ+od/e30PaVKTo+vMFtXRdrLO7S9o+6kS6nr5lJ11/1hNWlLepcpUT28fvdRHbxr/K/Vroof3gLaalrAoKMgfJuBSBNNNm0mxfp59D17d9XnlBvbeLBj/s/oT2vTmLMkhCkR7N5qDr7gNBee3X4aXryrEIXUdOJRXPYllkR7NaqF/2QrqOfOF5J8iH7qMayPa36PpwVFUGSaEt6PEH6T1rVQTJRGJyf5WVFJE27fvoGO9R6AglPjoP5Qo/GTMMwxgAnowZhmEMwLAm44cffhhTpkyBx+OBx+NBQ0MDXn755fTfI5EI5s+fj5KSErjdbsydOxednZ15d5phGOZQY1iTcU1NDZYtW4bNmzdj06ZNOOOMM3D++efjww8/BADcfPPNeOmll7B69WqsW7cObW1tuOiii74UxxmGYQ4lhiXgnXvuudL20qVL8fDDD2PDhg2oqanB448/jlWrVuGMM84AAKxcuRKTJ0/Ghg0bcOKJJ6oOmT26dXh/mIpPmpmKD0cdO4HY2ltlQWXIT8vUaJn1AgBAIiEv9Ds1KsxNOIxmzLJa6ReYdUERcUV5I305pQORTMh9URWh/dWpyDDmdLYTmxaS/V//9xbSxlGa2S+T4sX9XdupgLNDYSurkUUWn5cey63IqpaKKmoE6bArglGiiix0ne3U1tsrXyOHQpgbexgt69XRQTOA6VFV2BoaUAQj6YZ95Tgqpgb7qCjmsCvELXorZOVXoZf2ffeOXmm7vYUGSMTDVNSLxajYWAAqZutRlYPSUtTocshjKR6hferx0vuzPEyzzmV7P2ZDzmvGyWQSTz/9NAYHB9HQ0IDNmzcjHo9j9uzZ6TaTJk1CbW0t1q9fnxdnGYZhDlWG/Wrb1q1b0dDQgEgkArfbjTVr1uDII4/Eli1bYLPZ4PP5pPbl5eXo6KCvS+0nGo0i+rnCm8Fg5icGhmGYQ41hPxkfccQR2LJlCzZu3Ijvf//7mDdvHj766KOcHWhqaoLX601/xo4dm/OxGIZhRivDnoxtNhsOO+wwTJs2DU1NTZg6dSruv/9+VFRUIBaLwe/3S+07OztRUUET5eynsbERgUAg/WltbR32STAMw4x2DjoCL5VKIRqNYtq0abBarVi7di3mzp0LANi2bRtaWlrQ0NBwwP3tdjvsdho5pUdAXigXiqxqgSiNwFFFMVXUyoJKMouoowNh0YlSJjv9fbNZqEAUT1GRIpHQiRmKSKfMctQ/2pnklrsLPaSNSVFuqLCDlsEJJGRfnWW0NFM2qMSOVIJex+pqGv0mdH1hN9Ghm6R6J4DMAkssSn2ICGrTR4ECgMUqX9uBXipI5Yqm0bHU309P0qYbc2PraURZsJ/eGynF+WRDUhG5Z1b084Bf7ouU4voXFNNMa4XmHKUshVjX1U9FQ5vumhXYqViXUlz/sgoqxOaTYU3GjY2NOPvss1FbW4uBgQGsWrUKb775Jl599VV4vV5cc801WLhwIYqLi+HxeLBgwQI0NDQc/JsUDMMwhzjDmoy7urpwxRVXoL29HV6vF1OmTMGrr76Kr33tawCAe++9FyaTCXPnzkU0GsWcOXPw0EMPfSmOMwzDHEoMazJ+/PHHv/DvDocDK1aswIoVKw7KKYZhmK8anJuCYRjGAGginyEkeSAYDMLrzZyOkWEYZrQQCATg8VAR/fPwkzHDMIwB4MmYYRjGAPBkzDAMYwBGTdmlxsbGnPYTisAGLYuXylOKlfS7lzXlzS8MKYIDCjIHv1gc9AX1OxYtJrZc/VLEzcBO38sn2BR9uvjOpXnxCQDCFjkrmDOhjPAgRAUtqfSLZYvy5pewyNnEtEQWac8AJBQBF/fcfXde/IpbaECRNZHdre60yNfxtjuWkja5+hW20LHrTNBMeNnQ1JS/e9HlomNkcJCWWFKRSMp9fc/ye3LyAeAnY4ZhGEPAkzHDMIwB4MmYYRjGAPBkzDAMYwBGjYCnz0ImVOnLvFQM6C8bJDZbryyeWLppyRiHRm0qLBY5a5vDTZW/3l7ql9VBM2vF47rsaHZaKicRyU7w0GfkMrnoC+f97TTp/1CKljNy+WWxrKyKZm2LDWUWPITit7+9gGaJGwzRoB8tJg9VB2gZqVLQDF12E7XpSanU2gj1IRWloqvJKfdXKkZFWM2lyPZmptdWjyoey5JSZJOzyT4MKbLQ1VbSfvYH+ogtHM2cyc2sEGsHNDq+EpF+abvCQo/dGfQTm6+U+ppUlMHSo7qOXUNU6D1yYqW0PRCkoqtD0EyL3f1+YissdmX0K1v4yZhhGMYA8GTMMAxjAHgyZhiGMQA8GTMMwxiAUSPgCd3ivCNAF+vjJqrq2etpNFJ8jBxxY6qlkUHYnJ1fCV3ZoMEIFcBCfdQHnyIxnV2n6VmtVESI+rMT8Ewm+Xe2q7edtBlTRsvIiC4qjHUOyGJmMkxFkZIsCkJpUIhDSSqKRRVCjFO3q6uXCqxmG/UdJRndgkkxbsxmhahXQMeJQ5OvR8xCwxXFPrpfopZeDz2aomSYbwwtSdXvl4U4s6L/Cib6iG3oI8W9Ec1cnT2piGqNxHqILTAoj8HpU2tImx1vdlMfgtSv4szBqcrrGA7QkNKSsfKNZt1ORd6gQpBEqp/a8gg/GTMMwxgAnowZhmEMAE/GDMMwBoAnY4ZhGAMwagQ8c1wWxsy7fKSNvZ1GHpktdAG/dbwsnrgTFaRNfEwBsamw6VIO7munAltfB40CLB5bTmxuTRZBrCYapRcCFfVUmE1yf40vrSRtYhYqno1R+OXr7ZW2PU6qig3GFOKZjrCFipvBYhqJVmKn4pZTyNf2qDYaNRU3074JDGV+3hBxRZ8GqRiUdFJhKZGSIw+FXSGKldIUjeZg5tJiFiv1PaSIAosOyNfR1NdF2uzeWEhsWoAKVzZrNkoZ7ZtSh4/YfDZZ6K155S+kjctGx+B41zhii8QzC4uBABWW+3vpOEnpwnc1D73XUz30no32U19TWv6q1vGTMcMwjAHgyZhhGMYA8GTMMAxjAEbNmnHSKq8Zpgrp+qO3la6LmXEksVVG2qTtARdd73SF6PFVxBK67Ggx+kK8207X4XweulZqCsnrzUMDdA3MmuXvZ1KX3WsoRtfAHHGaac1VoKi75JOPFQrQF/yFYn1TjzNB1/QrTW3EVuHtJLYJvXJwwFgnXefbqrjWUUWghh7NSq+FcNOgEmuYXlthlYM8Ul3UL6uJ9k2imp4jaRNXBMk46PqzQ5O/0xSnbRItdO3UXU61kpg589qsqiaZ30R99ejWn2NVNOjj1AC991J9vcT2d0/mqcrrpdfM46NzQn+rX/ZLcc/WTKIay85dO4ltTMm4jH5lCz8ZMwzDGACejBmGYQwAT8YMwzAGgCdjhmEYAzBqBDxNyL8bKRcVn1qPoOJDQchHbMmg/JK/E3SR36Rl9zsVj8milNdDRYTKMirgeSxUzAra5GCKUK9CwHNmFqRU2GOKIAYLLRljjtGglYhJbmdVlBFSik06hELkKQ/RcyxXZL6DTiD8pKyUNHFEaRkhM2gQix6TIjtaWFG6yOumfR/VVa5yTKDnM9RLz1tLZg4WSCnKLoXD1C+LLvDA5KJj0GKl98ZQ3E9sVnNm4dpup0EykQF6P0Z017s5SvsmqkjHNghF6SrF2NGjz1QIACUeeqwBXSCI1UL7ORGn/ewsoPeLqjRWrvCTMcMwjAHgyZhhGMYA8GTMMAxjAHgyZhiGMQCjRsCDWRZZwhU045Q5ScWHaDHNYAWrLmpOsQafUpUIUmB1yV2YCNPMTkIRgdcXoALB4KCc8StXsQ4AzLoIvJBQnE+UClcOFy0bFPTLUXkqoSQbtJTi+gSpENeu0Yxm9pgcLRZWRPzFNRp5lg1JGoAHZzGNWBOKrHPJArlfIzRpGzSPaixlLlNl1hRCFqjAqi9nFUnRMejy0sjAZEoRUYjMglQoTE/SZqXHiunE7T4fFRaHovT7NBO9N7K5HaMK8bmsjGY+DA/qBDwXHTf79tFyUCXVdKzmE34yZhiGMQA8GTMMwxiAg5qMly1bBk3TcNNNN6VtkUgE8+fPR0lJCdxuN+bOnYvOzsxJURiGYb7K5DwZ/+1vf8Ojjz6KKVOmSPabb74ZL730ElavXo1169ahra0NF1100UE7yjAMc0gjcmBgYEBMnDhRvPbaa+LUU08VN954oxBCCL/fL6xWq1i9enW67ccffywAiPXr12d17EAgIADwhz/84c8h8wkEAhnnvpyejOfPn49vfOMbmD17tmTfvHkz4vG4ZJ80aRJqa2uxfv165bGi0SiCwaD0YRiG+aox7Ffbnn76abz77rv429/+Rv7W0dEBm80Gn88n2cvLy9HR0UHaA0BTUxMWL148XDcYhmEOKYb1ZNza2oobb7wRTz31FByKigO50NjYiEAgkP60trbm5bgMwzCjiWE9GW/evBldXV04/vjj07ZkMom33noLv/zlL/Hqq68iFovB7/dLT8ednZ2oqKAlXgDAbrfDrgiK0NPY2DgcV9Mk4vQFdYs182kLRWmZZXcvy5tfFkVWuIQqMENHKkazY93983vz5pdNkd0tZqN9oUez0Bfu77rj53nxCQCsmnzecZHdw0ASNIBgedPyvPmVisv9ZbJm7isAsCiCVu5YsjQ/fqlihejlUZLSXf6772oibXIe8zZ63yViiigZBUNDcmTO/ff/Z978SihcsGQ5M5ot8nW8846lB2iZmWFNxmeeeSa2bt0q2a666ipMmjQJP/nJTzB27FhYrVasXbsWc+fOBQBs27YNLS0taGhoyNlJhmGYQ51hTcaFhYU4+uijJZvL5UJJSUnafs0112DhwoUoLi6Gx+PBggUL0NDQgBNPPDF/XjMMwxxi5D03xb333guTyYS5c+ciGo1izpw5eOihh/L9NQzDMIcUBz0Zv/nmm9K2w+HAihUrsGLFioM9NMMwzFeG0ZO1TUeou5fYUnGafkuYqVBi170JYnFSMchiyy5jmllfpiasKC2ToqIbTDQ7mhmy+BMK08xhTi8tEaXCpFNiKj6pIW2GHPSdbnPJALH1uWXbYBEVqQo6s/BLkags6abHig/SPiwokK9jYh8VO80uRckgRaY14kOSinyOFFW8TIqsczGdGOz00JJEsQg9ViILQU1TZccroOMyEZWztNkV5xxJ0H62CnreqgyGxC9Fmap4hI7xwjF10rbZTPfrDdAx2Nu/j9iqqnwZ/Uom6PkUKN76KiySM9ht/tunpE1pOX3hYFydl9i6uvwZ/coWThTEMAxjAHgyZhiGMQA8GTMMwxgAnowZhmEMwKgV8CYcO4XYrB5a1iXlpMJSJCKLBrGeEGkT6souBzMRfxSii902htisdirExEz6aC6ViKgoSaMgpVNizBgkbQqC9PJ3ldDv7KuR1aZ4AVWfshLwFOJQxEnFOk/UR2zRsK700wQq1oYVIpI1RMsN6SEiLACHi46lZIiKhikh92EH1Z4w2EevWWVdZmFRKMoiORRRmtFAn7xdQM+5UHEfIEIFtTAyR8QJQS9kONxHbLEO+bynTp1G2pgtPsUXqKYlWmaNHov2aUgxJo4ZJ5dP+surtEyVaaxi3ohmGcaYI/xkzDAMYwB4MmYYhjEAPBkzDMMYgFG0Ziyvb3X10PzIroSL2AI9O4gtMSSvi5mTdB3OVkBf3lcRj8v7xlO0S/v62omt6qjDiO2oCZXSdnsrXYAMtGWXfN/TK2fC83YoyqSX0cAZk5uunw3Z5XO0KqI3IkWKwBYdMRfdz2ym/RXW6LEsTnnNW/jpmp4jQc8xYsq8BqqZ6FrjYJ8i/7arhJgCQ3IwzZ4920mbScd6iC0VpGvl1DE6LodC9JqNKS+Ttu2KteDmFj+xlYynfWgdzLyWbVWsK/vG0Oe6PTv2StufbqX35/hjphJbKECzOCaTmdf+TRrVO/y9tL+G4vL688Rj6L2449NuYqs+xaf41vytI/OTMcMwjAHgyZhhGMYA8GTMMAxjAHgyZhiGMQCjSMCTXzRv3UFr5QX29hCby0IX9b1jZOGisLyYtEkpa9dQrLryOU5FOSBPGc32FOqkAsEe3W9jbzsV/pyWzEIGAARLZCGur5yKLl01VFDpLKbCUsgn20qD9Bwd/ZnLINkGabBAVBWMEKNBEvGU/KJ+soa2Se2lNmsi8/OGSCn2K/YRW08nPVZRgTzmxtTSsRRppwE3ptIsnoMUxdsLPVXEFgzKxzfF6LUYWzee2IYG6fiK6+suKYgrpg2Xm2YhdPpk8Wzntg20TREdz1bFGE/GaQZDPSlBxbRkggrSrbvka3bsrONJm61b3iA2/wAViNU1rnKDn4wZhmEMAE/GDMMwBoAnY4ZhGAPAkzHDMIwBGD0Cni5T1OGHTyBN3NOOIbY4aBRTLCpn/BIxGvEViWSO3AIAm03uwkSUilQxRTkYe5RmHQu0ylFfCT/NJmcqV4kICnQlgnpKqA+d5TSaz0a7CxW7ZVEqVkyPpajEo4A2sljpFwpNUYqpTLbZWmiUlkWR0M6URYRUShGBmeijNpdTEUHYK/fhmCOpeDYwQEXRwXhmwdNqo7fnQJuf2KJDskjVvPFt0mby1xuITVjo+WTzdKYqu+QpLie28qh8jVymNtKmt4uKiL6SUmIzqUpQ6YjHFUKsIqDQBFkg7Gml0ZaVNVRENCl6RzVWc4WfjBmGYQwAT8YMwzAGgCdjhmEYA8CTMcMwjAEYNQKe2SL/boQjVBSJxalYY7HRCJlQcEBuY8m9G/SlfqwWmnrTYqJiUwq0nd0mqw1WRxlpo9DXlFiFLGaEq2kqQU+cCjGKqj4QYTkFpLedCiXZyBgWE/0+bZ/qWPRotj75+mtDVGBNOqgtm/6yKa5ZSiGeORSCVzwsf2dHHxVmgSxFVx2RsKIcUCF9fnK4ZbFp/PGHkzYpRbkuc47PYjbF/dK2k4rNVrs87jV7EWnjdtAIzHicnnc26KNhAaCknJY8i0bl+yw0QO+NsnIaURiPZifq5wo/GTMMwxgAnowZhmEMAE/GDMMwBkATqrrbI0gwGITXS7OcMQzDjFYCgQA8Hlp+6/PwkzHDMIwB4MmYYRjGAPBkzDAMYwB4MmYYhjEAoyboo7GxMaf9nEkaCBI2Zy6VYlLImkuXNeXNL5OJhiOkUlmUCFJkzFp211158ytipyfuiGZOySbMdCgtu/OOvPgEAIVOuezOQDi78lOqcJSmpmXS9sH4pWnyNROqqBnVfhbq11135Mcvu6DBCVEtu1tdn+Tszqb8jXmzjQbXJGOqIBmKySzvu/TOJXnzSygCYjTSE2ocKXkcLrr71px8APjJmGEYxhDwZMwwDGMAhjUZ33777dA0TfpMmjQp/fdIJIL58+ejpKQEbrcbc+fORWdnZ96dZhiGOdQY9pPxUUcdhfb29vTnr3/9a/pvN998M1566SWsXr0a69atQ1tbGy666KK8OswwDHMoMmwBz2KxoKKigtgDgQAef/xxrFq1CmeccQYAYOXKlZg8eTI2bNiAE0888aAcNekCBe1JKlKY4zQ7WsJOf2+cUVlkSdgU2biyqyNECgk5TbRLTYVUIIgO0CxXeiEhHKM+OGy+rPwSOs/8PlXGKeqrRdFM05WWiSv2c2dRpkpTlF0qstOsXcEYLQdlTclZtLw2eq0DQiEGxbPItKW41h/t3ElstZXFxOZ2yX6UlflIm8EA9WEwoqgRlQWqMeH1yMcqELRv9nYPEFt1aQmxxaI0G6IekaLiY3SICpcBnQZW4aTCeWU1zeSWUpQu6+zxZ/TLpMgKaE5QIS6qK8VlSrpIm0IzvWZBjWaTi5iGiC1Xhv1kvH37dlRVVWH8+PG4/PLL0dLSAgDYvHkz4vE4Zs+enW47adIk1NbWYv369XlzmGEY5lBkWE/GM2fOxBNPPIEjjjgC7e3tWLx4MU455RR88MEH6OjogM1mg8/nk/YpLy9HRwct+LefaDSKaPSfvzjBIH0qYhiGOdQZ1mR89tlnp/89ZcoUzJw5E3V1dXj22WfhdOaWQLupqQmLFy/OaV+GYZhDhYN6tc3n8+Hwww/Hjh07UFFRgVgsBr/fL7Xp7OxUrjHvp7GxEYFAIP1pbW09GJcYhmFGJQcVgRcKhbBz50585zvfwbRp02C1WrF27VrMnTsXALBt2za0tLSgoaHhgMew2+2w26ngoCelE1mSCqHMHQso9nMQ25BZ/g0yhRViXUF2mUX1rfyDVKSoLabRYp5K+ju4s6NZ2rY5Fb+VSV9Wfmk6z5wxKmTEFMKVh+qK6PfK4oywK8riRDIPJVU5pdaPm4mteRdd1jqqYZK0nfIqSiUl6fFpK5VjdL+yUlryqsBDx6lbN04sGj2WXRF5lquAl4xTkTKpiwIcMtFxY0vS629O5vYspimEsniYCn/JoDxugoqoNruJ3rM+T01OfqUUwqLJQvsrEZR91bx07PYH6Rh3eHz0WMnBYXj4xQxrMr7llltw7rnnoq6uDm1tbVi0aBHMZjMuu+wyeL1eXHPNNVi4cCGKi4vh8XiwYMECNDQ0HPSbFAzDMIc6w5qM9+7di8suuwy9vb0oLS3FySefjA0bNqC0tBQAcO+998JkMmHu3LmIRqOYM2cOHnrooS/FcYZhmEOJYU3GTz/99Bf+3eFwYMWKFVixYsVBOcUwDPNVY9RkbbPo1gOdiqCPmGLtOeKg61TlPfIL8L0+uqZryi6ZFPy6l93DHRHSJu6n68innV9PbBU18triBx/R9dRUiB5fRcQl989AAb3Uvj66X8JNj+8ekvsnmqTrnUOuLNbY49SHMkXfJ6voOr/bIb9cb6urJW3627qIzRTPnH3LolgDrRjjI7YSB80AWOqUB0rYTEvr7OmiHZ1SrC3rsQrqV6HiOhb65PXtzW9vIm1OO3MGsbV8sIfYEgWZs+EVKAKpfLV0vzffape294RoObUpPhp4Yo7T9VpbAR0TemIavf+dFtpfnko5yCM+SM9HVNBAkMQQDZwxW/OX3ocTBTEMwxgAnowZhmEMAE/GDMMwBoAnY4ZhGAMwagS8hFkWM8JCIUiFqainaVSJizlkMSAepy/lx5zZlc/xFci/ZwETFcA+3kmFn+5nqK/T/kUWQXyF40ibrgFFwIUCx6DcP3GNnk/cQsUta4yGtfcXyIKd3UoFPG9vFmVqrPScE/VVxFaoCK6IeuXj9+/bS9pE4vTZwqMIBCE+KIIFip1UkHIXjCG25n553x1+Oi7DXX5iG19LhT49cYXI53BRIat7+25pe9yYOtKmz0+DExS6MoqQedwPRWmbyUdRca6jY7u03bmP5jZv3UmFMtM4mh3Pmcg87m2CtomZ6f3v0uQMgHEL7RvNTMego5BmGIz4qaiXK/xkzDAMYwB4MmYYhjEAPBkzDMMYAJ6MGYZhDMCoEfDMOi3DIqiIMGRXlF1J0VOMa7Kg5lBkk0opSgQp/TLJ+x5xNM32VTqGLvL3+umxWj/skba95VSsUQRlKRG6jGxClb4sSftwsIAKHhUDstLT5ab9pSmuB8FE97OYqLipuGTQJznzKcTHXoXgZVKIesQHRZRWJETP59PuXmILhuV9Y937SJu+IXqO9SkqBhEUfRpTHCule6YqLFOUsmqnmfDKHHR8xbMIpFR0PYSZDrCJRxwtbRcV0PtgVw/N9maKKTKhmTNPVapKaaFuGv3ot8t++KM0VWFtLxWy91roWCpxuIktV/jJmGEYxgDwZMwwDGMAeDJmGIYxADwZMwzDGIBRI+AldD8bSYWKoClEKkeCiiB9Vjm6yqwou5MtZru8bzTkJ21cLvqbZzZR0cCui+aLh6lYk6V+R35mC/sUl1rxUywUKQFTOqFvjF+xXxaOaSrBsJ+KNRaFWJOI6koLCRoF6FRcx5Si1BNpo/BrSKiishQCIeR2EyZRAbcuTiMPsyFlpiFyIkkFVqtDHvjxFBXFzA4aUZiNWKdClf3z/Y9o+SSbVR4UMQu9QctKqU0vigOg9c1UfinGoMtDo/mSKd3YGaL3ovDSCEkaf5lf+MmYYRjGAPBkzDAMYwB4MmYYhjEAmhAHsWD6JRAMBuH10gxQDMMwo5VAIACP54sz9fGTMcMwjAHgyZhhGMYA8GTMMAxjAHgyZhiGMQCjJuijsbExp/2Ejf7eaLHMGca0FC27ctfdt+fNL7OgfiUVpZH0xGmsA/5zeVPe/EolaICCSZHVjLRR2JY2yX7l6hMA6CtjWWnsg5oUDSBouvtOaftg/IKm6xuRXYBHMkkv5PLly6Xt3K8hLT9kstDxrMJkkSMnlt5xF2mTq18mxTVLqbIJKtC/Z7Bs2TLSJne/6DVLWbObGlNW+Z69e8ndOfkA8JMxwzCMIeDJmGEYxgDwZMwwDGMAeDJmGIYxAKNGwNMTGAgSW5XXRWzhGBUubE45s5bLQoWz3gFaPicbrIVUkWjZ0UNsXjfNyDX95OOk7b/84S+kTUl1aVZ+6DNYaYosZ4UOWlpG2KnNpDtWeJAKHtF4ZkVNXwoKADSFIBmlLsAZlUUpTS+cAYiYaJ9akVlQM5vpM0llKRX+gr30OvZ09UvbAwH6fRV19aovzeiXCnuSln6Kh+TSQiWKANaIRrO2JVBCbNEELcWkx6QfEAAKKuj1D4Tley88QK/ZQActi1RTQgdATMvtubFl525iKy7ySdsuRTa5qCJjnjlFM/lZoRisOcJPxgzDMAaAJ2OGYRgDwJMxwzCMAeDJmGEYxgCMWgGvdgwV63rCirI7nXuIzTVuprRttxYqviE3Aa+sggpsf1+3jdgqZ9QSm88pX44UMkfkHQh9YtSh/ghp0xmitsEBarO7ZIHD4aZ+FZX4MvqkKbK1CgsVg+wJ6oPJKotBpgAVjIoV0W8DdJgQkoqyS/v2tBGbWV+uB0CsWxb1HDUTSJuSOmrraNmd2TEFJjcVA4NBub+0tr2kTeFY2jcOr0K46sss4KVSivtMo/t53G5puydIr5mmEOZSKSo2wkzHRDZU19URW8QilzPzltN7tq+1m9g0q5vYAIUCnSP8ZMwwDGMAeDJmGIYxAMOejPft24dvf/vbKCkpgdPpxDHHHINNmzal/y6EwG233YbKyko4nU7Mnj0b27dvz6vTDMMwhxrDWjPu7+/HrFmzcPrpp+Pll19GaWkptm/fjqKionSb5cuX44EHHsCTTz6J+vp63HrrrZgzZw4++ugjOByZ16MORFyT17y6B+nL4oVF9G33pKIUfE/Hn6XtQTctr+5y+LLyK5YYkrYjQwOkTbGzgti2f0h/oOb869elbX8vzb7lLc1ujUqDHFRgUgSZWBTl6AsUdditujW24jIaLGCGok663qcY/e2Pmug1C1hpoI6nXV7fTIyja3refrommUxkTguWUmXtCtOgolSCrhn2u+Q13Nryw+l+Fh+xWeyKdVEdJhsNKEgVlNNjjZU1D38zvYbxdsX5hALEJpKZn880Kx2DZkU6wU92hmWDlY4by6d0DFad4iO2kCtzcEUiTs876qfnWDaxUto2h8OkjQVDxGazU7+SMXqP5sqwJuO7774bY8eOxcqVK9O2+vp/DkYhBO677z787Gc/w/nnnw8A+M1vfoPy8nK88MILuPTSS/PkNsMwzKHFsJYpXnzxRUyfPh3f/OY3UVZWhuOOOw6/+tWv0n9vbm5GR0cHZs+enbZ5vV7MnDkT69evVx4zGo0iGAxKH4ZhmK8aw5qMd+3ahYcffhgTJ07Eq6++iu9///v44Q9/iCeffBIA0NHRAQAoL5f/K1VeXp7+m56mpiZ4vd70Z+zYsbmcB8MwzKhmWJNxKpXC8ccfj7vuugvHHXccrrvuOlx77bV45JFHcnagsbERgUAg/Wltbc35WAzDMKOVYa0ZV1ZW4sgjj5RskydPxvPPPw8AqKj4TKjq7OxEZeU/F8k7Oztx7LHHKo9pt9tht2cuCWMVsqsxRcYxq2KR3zblaGIztchCT9RMF/4Hk3ThX4XNIgsx4UH6cnr5OPpSeXAP/dGJtMvfWVLuo99nyy7bl9C9jD7l+CNJmwnHVxGbxUavRWRQFik+/dtu0mb3dhpcQ3yy0eAKa4KKbjZFyZuUV+5X2wC91skUfbYw26kQo8ekKrHjon3T0U9FyqqjjpG2vSVUYGvd3Uxs9mRmv1IxKiwNhagPrrIp0nbUPJkeLLSVmJKRXdQvZ+YoGRGnY9DuoAIxoiFp05ugS5BaqYfYHF56bVt6/Rn9slhp31iq6HXsbNFluSsfQ9rEElRYdpqoWBc7iMAsPcN6Mp41axa2bZOjyT799FPU/SPKpb6+HhUVFVi7dm3678FgEBs3bkRDQ0Me3GUYhjk0GdaT8c0334yTTjoJd911F771rW/hnXfewWOPPYbHHnsMAKBpGm666SbceeedmDhxYvrVtqqqKlxwwQVfhv8MwzCHBMOajGfMmIE1a9agsbERS5YsQX19Pe677z5cfvnl6TY//vGPMTg4iOuuuw5+vx8nn3wyXnnllYN6x5hhGOZQZ9iJgs455xycc845B/y7pmlYsmQJlixZclCOMQzDfJUYNVnbREpeKB9QRLqZeujCfyBMs2+VCzmSSjPTCCxFZZkDeSZt+YNUdLHZ6MHq68cR2ycf7ZS2o3EqGKgyZmXDhtf/RmwvPuMnNsVXQq+Vun00eqy6mkYZUmg/iKSiv5KK62GS5Q2rnQqlAwod2J1FwGI8FiM2u4MKS3UTiogtZpK/INS9k7TRolQMNivGBEGR5S7YS4Vft0kWxsIxei0KFFFzdihKeKUyZ0czK26OwTA9voAcXWcroZGusUEaiZqM00g9LYtxrynGV38HjdQsH1sjt+mjr91WlBQTW9dAiNiciqyDucKJghiGYQwAT8YMwzAGgCdjhmEYA8CTMcMwjAEYNQJeSieUldhoZI0qFaIHtKRS0pY5yihbnUzTRbrZUrRLhYlGmVmdVGzs6eyXtitrFAJLlqR04k9RqY+0cbgVKRoVJYgKXPJriZpGRQtV6SI9qthB4aDCT0GKHot8Y5JeoMLctE119JiCWKif2ExmWUhOJhURhRbqWBbdBSjSmRYV0si9VEQW3YrNVBRNRKjgjVyjxxTXP9xNr2NViRzFlkjRdJmuahr9tjOo8DULRT2lUR98JfReT0Tl4xeY6MgMRaiQ6VCIdTkOOSX8ZMwwDGMAeDJmGIYxADwZMwzDGABNCMWb5SNIMBiE10szJjEMw4xWAoEAPB4aSPR5+MmYYRjGAPBkzDAMYwB4MmYYhjEAhpuMDbaEzTAMc9BkM68ZbjIeGFC9nM4wDDN6yWZeM9zbFKlUCm1tbSgsLMTAwADGjh2L1tbWjEqk0QgGg+z7CDGa/R/NvgOj2/8vw3chBAYGBlBVVUXSwOoxXDi0yWRCTc1n+Ub3h916PJ5Rd2H3w76PHKPZ/9HsOzC6/c+379m+qmu4ZQqGYZivIjwZMwzDGABDT8Z2ux2LFi2C3a6op2Nw2PeRYzT7P5p9B0a3/yPtu+EEPIZhmK8ihn4yZhiG+arAkzHDMIwB4MmYYRjGAPBkzDAMYwAMOxmvWLEC48aNg8PhwMyZM/HOO++MtEtK3nrrLZx77rmoqqqCpml44YUXpL8LIXDbbbehsrISTqcTs2fPxvbt20fGWR1NTU2YMWMGCgsLUVZWhgsuuADbtm2T2kQiEcyfPx8lJSVwu92YO3cuOjs7R8jjf/Lwww9jypQp6Rf0Gxoa8PLLL6f/blS/VSxbtgyapuGmm25K24zs/+233w5N06TPpEmT0n83su8AsG/fPnz7299GSUkJnE4njjnmGGzatCn995G6Zw05GT/zzDNYuHAhFi1ahHfffRdTp07FnDlz0NXVNdKuEQYHBzF16lSsWLFC+ffly5fjgQcewCOPPIKNGzfC5XJhzpw5iCgKHv5fs27dOsyfPx8bNmzAa6+9hng8jrPOOguDg/8sHHnzzTfjpZdewurVq7Fu3Tq0tbXhoosuGkGvP6OmpgbLli3D5s2bsWnTJpxxxhk4//zz8eGHHwIwrt96/va3v+HRRx/FlClTJLvR/T/qqKPQ3t6e/vz1r39N/83Ivvf392PWrFmwWq14+eWX8dFHH+HnP/85ioqK0m1G7J4VBuSEE04Q8+fPT28nk0lRVVUlmpqaRtCrzAAQa9asSW+nUilRUVEh7rnnnrTN7/cLu90u/vu//3sEPPxiurq6BACxbt06IcRnvlqtVrF69ep0m48//lgAEOvXrx8pNw9IUVGR+PWvfz1q/B4YGBATJ04Ur732mjj11FPFjTfeKIQwfr8vWrRITJ06Vfk3o/v+k5/8RJx88skH/PtI3rOGezKOxWLYvHkzZs+enbaZTCbMnj0b69evH0HPhk9zczM6Ojqkc/F6vZg5c6YhzyUQCAAAiouLAQCbN29GPB6X/J80aRJqa2sN5X8ymcTTTz+NwcFBNDQ0jBq/58+fj2984xuSn8Do6Pft27ejqqoK48ePx+WXX46WlhYAxvf9xRdfxPTp0/HNb34TZWVlOO644/CrX/0q/feRvGcNNxn39PQgmUyivLxcspeXl6Ojo2OEvMqN/f6OhnNJpVK46aabMGvWLBx99NEAPvPfZrPB5/NJbY3i/9atW+F2u2G323H99ddjzZo1OPLIIw3vNwA8/fTTePfdd9HU1ET+ZnT/Z86ciSeeeAKvvPIKHn74YTQ3N+OUU07BwMCA4X3ftWsXHn74YUycOBGvvvoqvv/97+OHP/whnnzySQAje88aLmsbMzLMnz8fH3zwgbT2Z3SOOOIIbNmyBYFAAM899xzmzZuHdevWjbRbGWltbcWNN96I1157DQ6HY6TdGTZnn312+t9TpkzBzJkzUVdXh2effRZOp3MEPctMKpXC9OnTcddddwEAjjvuOHzwwQd45JFHMG/evBH1zXBPxmPGjIHZbCbqa2dnJyoqKkbIq9zY76/Rz+WGG27AH/7wB7zxxhvp9KXAZ/7HYjH4/X6pvVH8t9lsOOywwzBt2jQ0NTVh6tSpuP/++w3v9+bNm9HV1YXjjz8eFosFFosF69atwwMPPACLxYLy8nJD+6/H5/Ph8MMPx44dOwzf95WVlTjyyCMl2+TJk9PLLCN5zxpuMrbZbJg2bRrWrl2btqVSKaxduxYNDQ0j6Nnwqa+vR0VFhXQuwWAQGzduNMS5CCFwww03YM2aNXj99ddRX18v/X3atGmwWq2S/9u2bUNLS4sh/NeTSqUQjUYN7/eZZ56JrVu3YsuWLenP9OnTcfnll6f/bWT/9YRCIezcuROVlZWG7/tZs2aR1zc//fRT1NXVARjhe/ZLlQdz5OmnnxZ2u1088cQT4qOPPhLXXXed8Pl8oqOjY6RdIwwMDIj33ntPvPfeewKA+MUvfiHee+89sWfPHiGEEMuWLRM+n0/8/ve/F3//+9/F+eefL+rr60U4HB5hz4X4/ve/L7xer3jzzTdFe3t7+jM0NJRuc/3114va2lrx+uuvi02bNomGhgbR0NAwgl5/xk9/+lOxbt060dzcLP7+97+Ln/70p0LTNPGnP/1JCGFcvw/E59+mEMLY/v/oRz8Sb775pmhubhb/+7//K2bPni3GjBkjurq6hBDG9v2dd94RFotFLF26VGzfvl089dRToqCgQPzXf/1Xus1I3bOGnIyFEOLBBx8UtbW1wmaziRNOOEFs2LBhpF1S8sYbbwgA5DNv3jwhxGevytx6662ivLxc2O12ceaZZ4pt27aNrNP/QOU3ALFy5cp0m3A4LH7wgx+IoqIiUVBQIC688ELR3t4+ck7/g6uvvlrU1dUJm80mSktLxZlnnpmeiIUwrt8HQj8ZG9n/Sy65RFRWVgqbzSaqq6vFJZdcInbs2JH+u5F9F0KIl156SRx99NHCbreLSZMmiccee0z6+0jds5xCk2EYxgAYbs2YYRjmqwhPxgzDMAaAJ2OGYRgDwJMxwzCMAeDJmGEYxgDwZMwwDGMAeDJmGIYxADwZMwzDGACejBmGYQwAT8YMwzAGgCdjhmEYA8CTMcMwjAH4/wEhBw5HJZotYAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 400x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sanity check on ResNet-50 is done!\n"
     ]
    }
   ],
   "source": [
    "device = 'cpu'  # let's do everything on cpu because it's fast anyway\n",
    "\n",
    "fine_tuning = True  # true if predicted parameters will be fined or false for direct evaluation of predicted parameters\n",
    "\n",
    "model = models.resnet50()\n",
    "graph = Graph(model)\n",
    "# graph.visualize()  # to visualize ResNet-50 graph\n",
    "start = time.time()\n",
    "model = ghn(model, graph, bn_track_running_stats=fine_tuning, keep_grads=False)\n",
    "print('Parameter prediction for ResNet-50 done in %.3f seconds on %s' % (time.time() - start, device))\n",
    "\n",
    "weights = model.conv1.weight.data.cpu()\n",
    "grid = torchvision.utils.make_grid(weights, nrow=8, normalize=True, padding=1)\n",
    "plt.figure(figsize=(4, 4))\n",
    "plt.title(weights.shape)\n",
    "plt.imshow(grid.numpy().transpose((1, 2, 0)))\n",
    "plt.show()\n",
    "\n",
    "model_sanity_check = models.resnet50()\n",
    "model_sanity_check.load_state_dict(joblib.load(hf_hub_download(repo_id='SamsungSAILMontreal/ghn3',\n",
    "                                                               filename='resnet50_ghn3xlm16_init.pt'))['state_dict'])\n",
    "\n",
    "dict1 = dict(model.named_parameters())\n",
    "for n, p in model_sanity_check.named_parameters():\n",
    "    if not torch.allclose(p, dict1[n], atol=1e-6):\n",
    "        print('warning: big discrepancy {:.3e} for param: {}'.format(torch.norm(p - dict1[n]), n))        \n",
    "\n",
    "print('Sanity check on ResNet-50 is done!')        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca002e2c",
   "metadata": {},
   "source": [
    "## 4. Predict parameters for all PyTorch models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2c645a2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== MAXVIT_T was not in PyTorch at the moment of GHN-3 evaluation, so skipping it in this notebook ===\n",
      "=== REGNET_Y_128GF was not in PyTorch at the moment of GHN-3 evaluation, so skipping it in this notebook ===\n",
      "=== SWIN_V2_B was not in PyTorch at the moment of GHN-3 evaluation, so skipping it in this notebook ===\n",
      "=== SWIN_V2_S was not in PyTorch at the moment of GHN-3 evaluation, so skipping it in this notebook ===\n",
      "=== SWIN_V2_T was not in PyTorch at the moment of GHN-3 evaluation, so skipping it in this notebook ===\n",
      "=== VIT_H_14 was not in PyTorch at the moment of GHN-3 evaluation, so skipping it in this notebook ===\n",
      "\n",
      "74 PyTorch models found. Predicting parameters for all...\n",
      "1/74: ALEXNET with 61.10M params...done in 2.05 sec, total param norm=72.48 (norms matched)\n",
      "2/74: CONVNEXT_BASE with 88.59M params...done in 7.43 sec, total param norm=355.19 (norms matched)\n",
      "3/74: CONVNEXT_LARGE with 197.77M params...done in 6.67 sec, total param norm=399.83 (norms matched)\n",
      "4/74: CONVNEXT_SMALL with 50.22M params...done in 6.51 sec, total param norm=286.94 (norms matched)\n",
      "5/74: CONVNEXT_TINY with 28.59M params...done in 4.08 sec, total param norm=219.80 (norms matched)\n",
      "6/74: DENSENET121 with 7.98M params...done in 3.63 sec, total param norm=94.98 (norms matched)\n",
      "7/74: DENSENET161 with 28.68M params...done in 5.52 sec, total param norm=94.24 (norms matched)\n",
      "8/74: DENSENET169 with 14.15M params...done in 4.65 sec, total param norm=99.61 (norms matched)\n",
      "9/74: DENSENET201 with 20.01M params...done in 5.71 sec, total param norm=103.12 (norms matched)\n",
      "10/74: EFFICIENTNET_B0 with 5.29M params...done in 5.87 sec, total param norm=138.01 (norms matched)\n",
      "11/74: EFFICIENTNET_B1 with 7.79M params...done in 7.85 sec, total param norm=150.36 (norms matched)\n",
      "12/74: EFFICIENTNET_B2 with 9.11M params...done in 8.90 sec, total param norm=139.94 (norms matched)\n",
      "13/74: EFFICIENTNET_B3 with 12.23M params...done in 8.40 sec, total param norm=156.64 (norms matched)\n",
      "14/74: EFFICIENTNET_B4 with 19.34M params...done in 9.55 sec, total param norm=169.39 (norms matched)\n",
      "15/74: EFFICIENTNET_B5 with 30.39M params...done in 10.10 sec, total param norm=185.97 (norms matched)\n",
      "16/74: EFFICIENTNET_B6 with 43.04M params...done in 12.90 sec, total param norm=203.34 (norms matched)\n",
      "17/74: EFFICIENTNET_B7 with 66.35M params...done in 13.38 sec, total param norm=239.44 (norms matched)\n",
      "18/74: EFFICIENTNET_V2_L with 118.52M params...done in 12.78 sec, total param norm=278.36 (norms matched)\n",
      "19/74: EFFICIENTNET_V2_M with 54.14M params...done in 10.85 sec, total param norm=220.88 (norms matched)\n",
      "20/74: EFFICIENTNET_V2_S with 21.46M params...done in 7.98 sec, total param norm=182.78 (norms matched)\n",
      "21/74: GOOGLENET with 13.00M params...WARNING: node=56-AdaptiveAvgPool2DBackward0 does not have a path to node=153-fc.bias\n",
      "WARNING: node=105-AdaptiveAvgPool2DBackward0 does not have a path to node=153-fc.bias\n",
      "WARNING: n_glob_avg should be 1 in most architectures, but is 3 in this architecture.\n",
      "done in 5.41 sec, total param norm=98.24 (norms matched)\n",
      "22/74: INCEPTION_V3 with 27.16M params...WARNING: node=162-AvgPool2DBackward0 does not have a path to node=227-fc.bias\n",
      "WARNING: node=175-MeanBackward1 does not have a path to node=227-fc.bias\n",
      "WARNING: n_glob_avg should be 1 in most architectures, but is 2 in this architecture.\n",
      "done in 8.45 sec, total param norm=118.01 (norms matched)\n",
      "23/74: MNASNET0_5 with 2.22M params...done in 5.22 sec, total param norm=90.50 (norms matched)\n",
      "24/74: MNASNET0_75 with 3.17M params...done in 5.02 sec, total param norm=105.20 (norms matched)\n",
      "25/74: MNASNET1_0 with 4.38M params...done in 4.95 sec, total param norm=117.64 (norms matched)\n",
      "26/74: MNASNET1_3 with 6.28M params...done in 4.87 sec, total param norm=114.14 (norms matched)\n",
      "27/74: MOBILENET_V2 with 3.50M params...done in 3.62 sec, total param norm=110.26 (norms matched)\n",
      "28/74: MOBILENET_V3_LARGE with 5.48M params...done in 6.07 sec, total param norm=87.07 (norms matched)\n",
      "29/74: MOBILENET_V3_SMALL with 2.54M params...done in 6.23 sec, total param norm=78.99 (norms matched)\n",
      "30/74: REGNET_X_16GF with 54.28M params...done in 2.39 sec, total param norm=122.67 (norms matched)\n",
      "31/74: REGNET_X_1_6GF with 9.19M params...done in 2.36 sec, total param norm=115.36 (norms matched)\n",
      "32/74: REGNET_X_32GF with 107.81M params...done in 2.38 sec, total param norm=125.59 (norms matched)\n",
      "33/74: REGNET_X_3_2GF with 15.30M params...done in 2.54 sec, total param norm=114.02 (norms matched)\n",
      "34/74: REGNET_X_400MF with 5.50M params...done in 3.67 sec, total param norm=104.01 (norms matched)\n",
      "35/74: REGNET_X_800MF with 7.26M params...done in 3.14 sec, total param norm=99.12 (norms matched)\n",
      "36/74: REGNET_X_8GF with 39.57M params...done in 2.38 sec, total param norm=114.95 (norms matched)\n",
      "37/74: REGNET_Y_16GF with 83.59M params...done in 3.21 sec, total param norm=170.15 (norms matched)\n",
      "38/74: REGNET_Y_1_6GF with 11.20M params...done in 5.80 sec, total param norm=124.49 (norms matched)\n",
      "39/74: REGNET_Y_32GF with 145.05M params...done in 3.06 sec, total param norm=184.03 (norms matched)\n",
      "40/74: REGNET_Y_3_2GF with 19.44M params...done in 4.19 sec, total param norm=146.14 (norms matched)\n",
      "41/74: REGNET_Y_400MF with 4.34M params...done in 4.93 sec, total param norm=109.98 (norms matched)\n",
      "42/74: REGNET_Y_800MF with 6.43M params...done in 3.95 sec, total param norm=127.56 (norms matched)\n",
      "43/74: REGNET_Y_8GF with 39.38M params...done in 3.09 sec, total param norm=153.89 (norms matched)\n",
      "44/74: RESNET101 with 44.55M params...done in 3.88 sec, total param norm=117.25 (norms matched)\n",
      "45/74: RESNET152 with 60.19M params...done in 4.95 sec, total param norm=123.91 (norms matched)\n",
      "46/74: RESNET18 with 11.69M params...done in 2.67 sec, total param norm=73.91 (norms matched)\n",
      "47/74: RESNET34 with 21.80M params...done in 3.06 sec, total param norm=75.99 (norms matched)\n",
      "48/74: RESNET50 with 25.56M params...done in 3.30 sec, total param norm=108.45 (norms matched)\n",
      "49/74: RESNEXT101_32X8D with 88.79M params...done in 4.13 sec, total param norm=116.05 (norms matched)\n",
      "50/74: RESNEXT101_64X4D with 83.46M params...done in 3.87 sec, total param norm=115.20 (norms matched)\n",
      "51/74: RESNEXT50_32X4D with 25.03M params...done in 3.01 sec, total param norm=104.48 (norms matched)\n",
      "52/74: SHUFFLENET_V2_X0_5 with 1.37M params...done in 2.59 sec, total param norm=83.48 (norms matched)\n",
      "53/74: SHUFFLENET_V2_X1_0 with 2.28M params...done in 2.89 sec, total param norm=85.52 (norms matched)\n",
      "54/74: SHUFFLENET_V2_X1_5 with 3.50M params...done in 2.92 sec, total param norm=84.96 (norms matched)\n",
      "55/74: SHUFFLENET_V2_X2_0 with 7.39M params...done in 2.54 sec, total param norm=100.91 (norms matched)\n",
      "56/74: SQUEEZENET1_0 with 1.25M params...done in 2.52 sec, total param norm=72.14 (norms matched)\n",
      "57/74: SQUEEZENET1_1 with 1.24M params...done in 2.45 sec, total param norm=71.42 (norms matched)\n",
      "58/74: SWIN_B with 87.77M params...done in 3.16 sec, total param norm=232.28 (norms matched)\n",
      "59/74: SWIN_S with 49.61M params...done in 3.28 sec, total param norm=189.28 (norms matched)\n",
      "60/74: SWIN_T with 28.29M params...done in 1.78 sec, total param norm=184.69 (norms matched)\n",
      "61/74: VGG11 with 132.86M params...done in 1.54 sec, total param norm=67.99 (norms matched)\n",
      "62/74: VGG11_BN with 132.87M params...done in 1.54 sec, total param norm=64.61 (norms matched)\n",
      "63/74: VGG13 with 133.05M params...done in 1.97 sec, total param norm=59.84 (norms matched)\n",
      "64/74: VGG13_BN with 133.05M params...done in 2.00 sec, total param norm=66.59 (norms matched)\n",
      "65/74: VGG16 with 138.36M params...done in 2.20 sec, total param norm=62.99 (norms matched)\n",
      "66/74: VGG16_BN with 138.37M params...done in 2.26 sec, total param norm=75.76 (norms matched)\n",
      "67/74: VGG19 with 143.67M params...done in 2.22 sec, total param norm=64.59 (norms matched)\n",
      "68/74: VGG19_BN with 143.68M params...done in 2.74 sec, total param norm=73.97 (norms matched)\n",
      "69/74: VIT_B_16 with 86.57M params...WARNING: setting a 4d size (1, 768, 14, 14) instead of 3d (1, 197, 768)\n",
      "WARNING: n_glob_avg should be 1 in most architectures, but is 0 in this architecture.\n",
      "done in 2.61 sec, total param norm=366.08 (norms matched)\n",
      "70/74: VIT_B_32 with 88.22M params...WARNING: setting a 4d size (1, 768, 7, 7) instead of 3d (1, 50, 768)\n",
      "WARNING: n_glob_avg should be 1 in most architectures, but is 0 in this architecture.\n",
      "done in 2.36 sec, total param norm=358.91 (norms matched)\n",
      "71/74: VIT_L_16 with 304.33M params...WARNING: setting a 4d size (1, 1024, 14, 14) instead of 3d (1, 197, 1024)\n",
      "WARNING: n_glob_avg should be 1 in most architectures, but is 0 in this architecture.\n",
      "done in 3.82 sec, total param norm=511.73 (norms matched)\n",
      "72/74: VIT_L_32 with 306.54M params...WARNING: setting a 4d size (1, 1024, 7, 7) instead of 3d (1, 50, 1024)\n",
      "WARNING: n_glob_avg should be 1 in most architectures, but is 0 in this architecture.\n",
      "done in 3.30 sec, total param norm=512.32 (norms matched)\n",
      "73/74: WIDE_RESNET101_2 with 126.89M params...done in 3.42 sec, total param norm=118.16 (norms matched)\n",
      "74/74: WIDE_RESNET50_2 with 68.88M params...done in 2.62 sec, total param norm=108.55 (norms matched)\n",
      "\n",
      "done for 74 models in 381.50 seconds, norms matched for 74 models\n"
     ]
    }
   ],
   "source": [
    "norms = get_metadata(ghn3_name, attr='paramnorm')\n",
    "\n",
    "# Enumerate all PyTorch models of ImageNet classification\n",
    "# Should be >= 74 models in torchvision>=0.13.1\n",
    "all_torch_models = []\n",
    "for m in dir(models):\n",
    "    if m[0].isupper() or m.startswith('_') or m.startswith('get') or m == 'list_models' or \\\n",
    "      not inspect.isfunction(eval('models.%s' % m)):\n",
    "        continue\n",
    "    \n",
    "    if m not in norms:\n",
    "        print('=== %s was not in PyTorch at the moment of GHN-3 evaluation, so skipping it in this notebook ===' \n",
    "              % m.upper())\n",
    "        continue  # skip for consistency with the paper\n",
    "\n",
    "    all_torch_models.append(m)\n",
    "    \n",
    "print('\\n%d PyTorch models found. Predicting parameters for all...' % len(all_torch_models))    \n",
    "\n",
    "fine_tuning = True  # true if predicted parameters will be fined or false for direct evaluation of predicted parameters\n",
    "start_all = time.time()\n",
    "norms_matched = []\n",
    "for m_ind, m in enumerate(all_torch_models):\n",
    "    kw_args = {'init_weights': False} if m in ['googlenet', 'inception_v3'] else {}\n",
    "\n",
    "    # Predict parameters\n",
    "    model = eval(f'models.{m}(**kw_args)')\n",
    "    \n",
    "    if not isinstance(model, torch.nn.Module):\n",
    "        print('skipping %s, because it is not torch.nn.Module' % m)\n",
    "        continue\n",
    "        \n",
    "    n_params = sum([p.numel() for p in model.parameters()]) / 10**6\n",
    "    if m == 'inception_v3':\n",
    "        model.expected_input_sz = 299\n",
    "    print('{}/{}: {} with {:.2f}M params'.format(m_ind + 1, \n",
    "                                                 len(all_torch_models),\n",
    "                                                 m.upper(), \n",
    "                                                 n_params), end='...')\n",
    "    start = time.time()\n",
    "    try:\n",
    "        model = ghn(model, bn_track_running_stats=fine_tuning, keep_grads=False)\n",
    "    except Exception as e:\n",
    "        print('ERROR for model %s: %s' % (m, e))\n",
    "        continue\n",
    "    \n",
    "    total_norm = torch.norm(torch.stack([p.norm() for p in model.parameters()]), 2)\n",
    "    norms_matched.append(abs(norms[m] - total_norm.item()) < 1e-2)\n",
    "    print('done in {:.2f} sec, total param norm={:.2f} ({})'.\n",
    "          format(time.time() - start,\n",
    "                 total_norm.item(),\n",
    "                 'norms matched' if norms_matched[-1] else ('ERROR: norm not matched with %.2f' % norms[m])))\n",
    "        \n",
    "    # \"WARNING: number of predicted ...\" means that some layers in the model are not supported by the GHN\n",
    "    # unsupported modules are initialized using built-in PyTorch methods\n",
    "\n",
    "print('\\ndone for {} models in {:.2f} seconds, norms matched for {} models'.format(len(norms_matched), \n",
    "                                                                                   time.time() - start_all,\n",
    "                                                                                   sum(norms_matched)))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}