NIMA-pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5eff278b",
   "metadata": {},
   "source": [
    "# Using NIMA  as an Aesthetic Critic \n",
    "\n",
    "This notebook is divided into two sections:\n",
    "- <a href=\"#section1\">Section 1 - NIMA Pytorch</a> Reproduces the paper [NIMA: Neural Image Assessment](https://ieeexplore.ieee.org/document/8352823) which uses deep learning to aesthetically evaluate images as explained in this [blog post](https://ai.googleblog.com/2017/12/introducing-nima-neural-image-assessment.html). \n",
    "\n",
    "- <a href=\"#section2\">Section 2 - Deploy Model as an Aesthetic Critic</a> Adapts and evaluates the model for use as an aesthetic critic to evaluate designs generated by genetic algorithms. \n",
    "\n",
    "## Section 1) NIMA Pytorch\n",
    "<a name=\"section1\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "646cf536",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PYTORCH_JIT_LOG_LEVEL=GRAPH_DEBUG\n",
      "env: PYTORCH_JIT_TYPE_VERBOSITY=3 # the max\n",
      "num gpu: 1\n",
      "found CUDA\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "from torch.nn import functional as F\n",
    "from torch.utils.data import DataLoader, Dataset\n",
    "#%env CUDA_VISIBLE_DEVICES=\"\"\n",
    "#https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/OVERVIEW.md#jit-logging\n",
    "%env PYTORCH_JIT_LOG_LEVEL=GRAPH_DEBUG\n",
    "%env PYTORCH_JIT_TYPE_VERBOSITY=3 # the max\n",
    "n_gpu = torch.cuda.device_count()\n",
    "print(\"num gpu:\",n_gpu )\n",
    "import numpy as np\n",
    "import scipy\n",
    "from tqdm import tqdm\n",
    "import sys\n",
    "import os\n",
    "import re\n",
    "import inspect\n",
    "from glob import glob\n",
    "if torch.cuda.is_available():\n",
    "  torch.cuda.empty_cache()\n",
    "  device = torch.device('cuda') \n",
    "  print(\"found CUDA\")\n",
    "else:\n",
    "  device = torch.device('cpu')\n",
    "  \n",
    "def softmax(x):\n",
    "  return np.exp(x)/np.sum(np.exp(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39516576",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "The primary dataset is [Aesthetic Visual Analysis](https://github.com/imfing/ava_downloader) (AVA)  which contains 255510 images with aesthetic ratings of photos scraped from the photo competition website [dpchallenge.com](https://www.dpchallenge.com/)\n",
    "\n",
    "The paper used two other smaller datasets; Tampere Image Database 2013 (TID2013) and LIVE In the Wild Image Quality Challenge Database which will not be used. The paper itself found:\n",
    "\n",
    ">We believe this observation shows that NIMA models trained on AVA can generalize to other test examples more effectively, whereas training on TID2013 results in poor performance on LIVE and AVA test sets.\n",
    "\n",
    "The AVA dataset is not in the official collection of torch datasets so we must write our own data loader. Because we focus on aesthetics we have to be careful not to apply data augmentation that might reduce  the aesthetics, keeping only horizontal flip. We'll also crop landscape/portrait images to accommodate the model's preference for square images.\n",
    "\n",
    "#### Labels\n",
    "\n",
    "First we'll pre-process the labels. If you look ahead to the loss function, instead of labels of **one score** we want at a **histogram of scores**, which luckily, the data provides. Here is the first line from the file AVA.txt:\n",
    "\n",
    "`-          -----score histogram----\n",
    "DEX  IMGID  1 2 3 4  5  6  7  8 9 10  tag1 tag2 challenge\n",
    "1    953619 0 1 5 17 38 36 15 6 5 1   1    22   1396\n",
    "`\n",
    "\n",
    "This image is rated above average with one person even scoring it a 10. We will not be using category tags but they can be found in the tags.txt file, in this case the tags are: Abstract,Macro and the  challenge was: 100_Meters\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "318706b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (255530, 15)  first line: [     1 953619      0      1      5     17     38     36     15      6\n",
      "      5      1      1     22   1396]\n",
      "file: 953619.jpg  lbl: [0.         0.         0.         0.         0.88079708 0.11920292\n",
      " 0.         0.         0.         0.        ]\n"
     ]
    }
   ],
   "source": [
    "# DATASET Preprocess AVA Labels\n",
    "DATA_PATH = \"data/AVA_dataset/images/\" \n",
    "all_files = []\n",
    "  \n",
    "\n",
    "lblraw = np.loadtxt(\"data/AVA_dataset/AVA.txt\", dtype=int)\n",
    "print(\"shape:\",lblraw.shape,\" first line:\",lblraw[0])\n",
    "# make a hash by IMGID with a softmax of the scores\n",
    "lblO = {}\n",
    "for ii in range(lblraw.shape[0]):\n",
    "  filename = str(lblraw[ii][1])+\".jpg\"\n",
    "  lblO[filename] = softmax(lblraw[ii][2:12])\n",
    "  all_files.append(filename)\n",
    "  if ii==0:\n",
    "    with np.printoptions(precision = 8, suppress = True):\n",
    "      print(\"file:\",filename,\" lbl:\",lblO[filename])\n",
    "      \n",
    "# convert file list to full path\n",
    "mm=0\n",
    "id2del=[]\n",
    "all_files_save=all_files.copy()\n",
    "for ii in range(len(all_files)):\n",
    "  if os.path.isfile(DATA_PATH +all_files[ii]):\n",
    "    all_files[ii] = DATA_PATH +all_files[ii]\n",
    "  elif(True): # remove it\n",
    "    mm+=1\n",
    "    id2del.append(ii)\n",
    "    print(\"Missing file:\",DATA_PATH+all_files[ii],mm)\n",
    "all_files = np.array(all_files)\n",
    "all_files = np.delete(all_files, np.array(id2del) )\n",
    "print(\"num files:\",all_files.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1622bc54",
   "metadata": {},
   "source": [
    "<a name='loader'></a>\n",
    "## Torch Dataloader\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "af5e819b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 13353 total  images.  first one: data/cfcolor_dataset/images/4684.jpg\n",
      "WARNING partial batch: 9\n"
     ]
    }
   ],
   "source": [
    "# DATASET Image Loader \n",
    "# images are all 640x480 or 480x640 jpeg format\n",
    "import torchvision.transforms as T\n",
    "from PIL import Image\n",
    "\n",
    "W_H = 224 # MobileNet wants >= 224 and < 480\n",
    "BATCH_SIZE = 32\n",
    "BATCH_SIZE_VAL = BATCH_SIZE * 3 # if low on RAM decrease this\n",
    "NUM_EPOCHS = 4\n",
    "LEARNING_RATE = 0.00001\n",
    "\n",
    "nfiles = len(all_files)\n",
    "print('There are %d total  images.' % nfiles,\" first one:\",all_files[0])\n",
    "train_size = int(0.8 * nfiles)\n",
    "test_size = nfiles - train_size\n",
    "if nfiles % BATCH_SIZE !=0:\n",
    "  print(\"WARNING partial batch:\",nfiles % BATCH_SIZE)\n",
    "\n",
    "# DATASET CLASS\n",
    "class ImageSet(Dataset):\n",
    "    def __init__(self, imgA, size=None, lblO=lblO, device=None):\n",
    "        self.size = size\n",
    "        self.images = imgA\n",
    "        self.lblO = lblO\n",
    "        self.device = device\n",
    "        self._length = len(self.images)\n",
    "        # should work for: A batch of Tensor Images is a tensor of (B, C, H, W)\n",
    "        self.transform = T.Compose([\n",
    "                                T.RandomHorizontalFlip(.3),\n",
    "                                T.Resize(W_H),\n",
    "                                T.CenterCrop(W_H),\n",
    "                                T.ToTensor(),# gets (C * H * W). scale 0–255 to 0–1.\n",
    "                                T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),#https://pytorch.org/hub/pytorch_vision_mobilenet_v2/\n",
    "            \n",
    "                                    ])\n",
    "\n",
    "    def __len__(self):\n",
    "        return self._length\n",
    "    #https://pytorch.org/hub/pytorch_vision_mobilenet_v2/    color first, 0-1 float range\n",
    "    def preprocess_image(self, image_path):\n",
    "        #print(\"opening:\",image_path)\n",
    "        image = Image.open(image_path)\n",
    "        if not image.mode == \"RGB\":\n",
    "            image = image.convert(\"RGB\")\n",
    "        #image = np.array(image).astype(np.uint8)\n",
    "        #image = (image / 255.0).astype(np.float32)\n",
    "        #image = image.transpose(2, 1, 0)\n",
    "        return image\n",
    "\n",
    "    def __getitem__(self, i):\n",
    "        img_path = self.images[i]\n",
    "        img_dex  = re.sub(r'.*\\/','',img_path)\n",
    "        try:\n",
    "            zimg = self.preprocess_image( img_path)\n",
    "            image = self.transform( zimg )\n",
    "        except:# this hack works when using shuffle!\n",
    "            print(\" X X X corrupt image:\",img_path,\" SKIPPING \")\n",
    "            img_path = self.goodpath\n",
    "            img_dex  = re.sub(r'.*\\/','',img_path)\n",
    "            zimg = self.preprocess_image( img_path)\n",
    "            image = self.transform( zimg )\n",
    "         \n",
    "        label = torch.tensor(lblO[img_dex])\n",
    "        self.goodpath = img_path\n",
    "        if self.device is not None:\n",
    "          image = image.to(self.device)\n",
    "          label = label.to(self.device)\n",
    "        return image,label\n",
    "\n",
    "#train_set = all_files[:10000]\n",
    "full_dataset = ImageSet(all_files, size=nfiles, lblO=lblO, device=device)\n",
    "train_dataset, valid_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])\n",
    "\n",
    "train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)\n",
    "valid_loader = DataLoader(valid_dataset, batch_size=BATCH_SIZE_VAL, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "ed010095",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label: tensor([5.3983e-06, 1.9859e-06, 2.1778e-03, 1.9859e-06, 1.1891e-01, 1.9859e-06,\n",
      "        8.7861e-01, 1.9859e-06, 2.9474e-04, 1.9859e-06], device='cuda:0',\n",
      "       dtype=torch.float64)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD8CAYAAAB3lxGOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPnklEQVR4nO3df4wc9X3G8fdzZzBgE9kY7FjGqQ0ypFA1hli0EgWlpQk/VMVQidRWhdwW1SAZCdRUqg1SiipZStMA/1QQGYHqVgTj1FCsiqRYFgqKVH4YMAZjDGcwcPhkBxIDAQq+u0//mO+S4djjjp0dz56+z0ta7cx3ZnY/4+EeZnZX81FEYGb56mu6ADNrlkPALHMOAbPMOQTMMucQMMucQ8Asc7WFgKRLJO2VNCBpbV3vY2bVqI7fCUjqB14CvgkMAk8CKyPiha6/mZlVUteZwHnAQES8EhEfA5uA5TW9l5lVMK2m110AvFGaHwT+YLyVZ504K74858s1lfLFiGCUPgIQo4ggJII++iKYNvwR0458yLQYIegj0CfbQaR5NbkLnxiOUQ6PDPP+yEjTpRRa/yzx2YEQHHf8cZww8wT6pvURrZVUrK/Sv2mv/MZ1+rEj9Pc3XcXk7Xrmtbci4pSx43WFQLu/gk8dO0mrgdUA8+bMY8P3NtRUyhcQcMxo8EH/dELBMXyAGOaIjmOYEzj+yMfMPbyPk998ljkfvcMwJ/Bx3/EQQT//hzTKkb5+YBr9PfBf6qEjH/Pfv36Lx95/p+lSCIJQ8afdF30ogChOREf6RhnuG+XMr53Bued/nRmzZzDSd4RQgAQh+kb70SdR2xuhtmjRB3xp5jDqjcyf0IIZq19rN17X5cAgsLA0fypwoLxCRGyIiGURsWzWzFk1lWFmE6krBJ4ElkhaLOlYYAWwtab3MrMKarkciIhhSdcB/wP0A3dHxO463svMqqnrMwEi4iHgobpe38y6w78YNMucQ8Ascw4Bs8w5BMwy5xAwy5xDwCxzDgGzzDkEzDLnEDDLnEPALHMOAbPMOQTMMucQMMucQ8Ascw4Bs8x1HAKSFkp6RNIeSbslXZ/Gb5b0pqSd6XFZ98o1s26rclORYeC7EfG0pBOBpyRtS8tui4gfVi/PzOrWcQhExBAwlKbfk7SH4lbjZjaFdOUzAUmLgHOAx9PQdZJ2Sbpb0uxuvIeZ1aNyCEiaCWwBboiId4E7gNOBpRRnCreMs91qSTsk7Tj8m8NVyzCzDlUKAUnHUATAPRFxP0BEHIyIkYgYBe6kaEn2Ge47YNYbqnw7IOAuYE9E3Foan19a7Qrg+c7LM7O6Vfl24HzgKuA5STvT2I3ASklLKdqO7QeuqfAeZlazKt8O/IL2PQfda8BsCvEvBs0y5xAwy5xDwCxzDgGzzDkEzDLnEDDLnEPALHMOAbPMOQTMMucQMMucQ8Ascw4Bs8w5BMwy5xAwy5xDwCxzDgGzzFW5sxCS9gPvASPAcEQsk3QScB+wiOLOQt+JiF9XK9PM6tKNM4E/joilEbEsza8FtkfEEmB7mjezHlXH5cByYGOa3ghcXsN7mFmXVA2BAB6W9JSk1WlsXupO1OpSNLfdhu47YNYbKn0mAJwfEQckzQW2SXpxshtGxAZgA8BXF301KtZhZh2qdCYQEQfS8yHgAYpGIwdbvQfS86GqRZpZfao0H5mRuhEjaQbwLYpGI1uBVWm1VcCDVYs0s/pUuRyYBzxQNCJiGvDjiPiZpCeBzZKuBl4HrqxeppnVpUrzkVeAr7UZfxu4qEpRZnb0+BeDZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFglrmO7ycg6UyK/gItpwHfA2YBfwv8Mo3fGBEPdfo+ZlavKjcV2QssBZDUD7xJcZ/BvwZui4gfdqNAM6tXty4HLgL2RcRrXXo9MztKuhUCK4B7S/PXSdol6W5Js7v0HmZWg8ohIOlY4NvAT9LQHcDpFJcKQ8At42zn5iNmPaAbZwKXAk9HxEGAiDgYESMRMQrcSdGL4DMiYkNELIuIZbNmzupCGWbWiW6EwEpKlwKtxiPJFRS9CMysR1VtTX4C8E3gmtLwDyQtpehTuH/MMjPrMZVCICI+AOaMGbuqUkVmdlT5F4NmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWuQlDIN0s9JCk50tjJ0naJunl9Dy7tGydpAFJeyVdXFfhZtYdkzkT+DfgkjFja4HtEbEE2J7mkXQWxZ2Hz07b3J56EphZj5owBCLiUeBXY4aXAxvT9Ebg8tL4poj4KCJeBQYY50ajZtYbOv1MYF5EDAGk57lpfAHwRmm9wTRmZj2q2x8Mqs1YtF3RfQfMekKnIXCwdWvx9HwojQ8CC0vrnQocaPcC7jtg1hs6DYGtwKo0vQp4sDS+QtJ0SYuBJcAT1Uo0szpNeMtxSfcC3wBOljQI/CPwfWCzpKuB14ErASJit6TNwAvAMLAmIkZqqt3MumDCEIiIleMsumic9dcD66sUZWZHj38xaJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlrtPmI/8i6UVJuyQ9IGlWGl8k6UNJO9PjRzXWbmZd0GnzkW3A70XE7wMvAetKy/ZFxNL0uLY7ZZpZXTpqPhIRD0fEcJp9jOKuwmY2BXXjM4G/AX5aml8s6RlJP5d0wXgbue+AWW+oFAKSbqK4q/A9aWgI+EpEnAP8HfBjSV9qt637Dpj1ho5DQNIq4M+Av4yIAEg9CN9O008B+4AzulGomdWjoxCQdAnwD8C3I+KD0vgprS7Ekk6jaD7ySjcKNbN6dNp8ZB0wHdgmCeCx9E3AhcA/SRoGRoBrI2JsR2Mz6yGdNh+5a5x1twBbqhZlZkePfzFoljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmeu078DNkt4s9Re4rLRsnaQBSXslXVxX4WbWHZ32HQC4rdRf4CEASWcBK4Cz0za3t243Zma9qaO+A59jObAp3XD0VWAAOK9CfWZWsyqfCVyX2pDdLWl2GlsAvFFaZzCNfYb7Dpj1hk5D4A7gdGApRa+BW9K42qwb7V7AfQfMekNHIRARByNiJCJGgTv57Sn/ILCwtOqpwIFqJZpZnTrtOzC/NHsF0PrmYCuwQtJ0SYsp+g48Ua1EM6tTp30HviFpKcWp/n7gGoCI2C1pM/ACRXuyNRExUkvlZtYVXe07kNZfD6yvUpSZHT3+xaBZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlrtO+A/eVeg7sl7QzjS+S9GFp2Y9qrN3MumDCm4pQ9B34V+DfWwMR8RetaUm3AO+U1t8XEUu7VJ+Z1WwydxZ6VNKidsskCfgO8CddrsvMjpKqnwlcAByMiJdLY4slPSPp55IuqPj6ZlazyVwOfJ6VwL2l+SHgKxHxtqSvA/8l6eyIeHfshpJWA6sB5s2ZV7EMM+tUx2cCkqYBfw7c1xpL7cfeTtNPAfuAM9pt7+YjZr2hyuXAnwIvRsRga0DSKa0GpJJOo+g78Eq1Es2sTpP5ivBe4H+BMyUNSro6LVrBpy8FAC4Edkl6FvhP4NqImGwzUzNrQKd9B4iIv2oztgXYUr0sMzta/ItBs8w5BMwy5xAwy5xDwCxzDgGzzDkEzDLnEDDLnEPALHMOAbPMOQTMMucQMMucQ8Ascw4Bs8w5BMwy5xAwy9xkbiqyUNIjkvZI2i3p+jR+kqRtkl5Oz7NL26yTNCBpr6SL69wBM6tmMmcCw8B3I+J3gT8E1kg6C1gLbI+IJcD2NE9atgI4G7gEuL11yzEz6z0ThkBEDEXE02n6PWAPsABYDmxMq20ELk/Ty4FN6aajrwIDwHldrtvMuuQLfSaQmpCcAzwOzIuIISiCApibVlsAvFHabDCNmVkPmnQISJpJcf/AG9r1ESiv2mYs2rzeakk7JO04/JvDky3DzLpsUiEg6RiKALgnIu5PwwclzU/L5wOH0vggsLC0+anAgbGv6b4DZr1hMt8OCLgL2BMRt5YWbQVWpelVwIOl8RWSpktaTNF74InulWxm3TSZNmTnA1cBz7VakAM3At8HNqc+BK8DVwJExG5Jm4EXKL5ZWBMRI90u3My6YzJ9B35B++t8gIvG2WY9sL5CXWZ2lPgXg2aZcwiYZc4hYJY5h4BZ5hwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWOYeAWeYcAmaZcwiYZc4hYJY5h4BZ5hwCZplzCJhlThGfuRv40S9C+iXwPvBW07VUcDJTu36Y+vsw1euHevfhdyLilLGDPRECAJJ2RMSypuvo1FSvH6b+Pkz1+qGZffDlgFnmHAJmmeulENjQdAEVTfX6Yervw1SvHxrYh575TMDMmtFLZwJm1oDGQ0DSJZL2ShqQtLbpeiZL0n5Jz0naKWlHGjtJ0jZJL6fn2U3X2SLpbkmHJD1fGhu3Xknr0jHZK+niZqr+tHH24WZJb6bjsFPSZaVlPbUPkhZKekTSHkm7JV2fxps9DhHR2APoB/YBpwHHAs8CZzVZ0xeofT9w8pixHwBr0/Ra4J+brrNU24XAucDzE9ULnJWOxXRgcTpG/T26DzcDf99m3Z7bB2A+cG6aPhF4KdXZ6HFo+kzgPGAgIl6JiI+BTcDyhmuqYjmwMU1vBC5vrpRPi4hHgV+NGR6v3uXApoj4KCJeBQYojlWjxtmH8fTcPkTEUEQ8nabfA/YAC2j4ODQdAguAN0rzg2lsKgjgYUlPSVqdxuZFxBAUBxyY21h1kzNevVPtuFwnaVe6XGidSvf0PkhaBJwDPE7Dx6HpEGjX7XiqfF1xfkScC1wKrJF0YdMFddFUOi53AKcDS4Eh4JY03rP7IGkmsAW4ISLe/bxV24x1fR+aDoFBYGFp/lTgQEO1fCERcSA9HwIeoDhNOyhpPkB6PtRchZMyXr1T5rhExMGIGImIUeBOfnu63JP7IOkYigC4JyLuT8ONHoemQ+BJYImkxZKOBVYAWxuuaUKSZkg6sTUNfAt4nqL2VWm1VcCDzVQ4aePVuxVYIWm6pMXAEuCJBuqbUOuPJ7mC4jhAD+6DJAF3AXsi4tbSomaPQw984nsZxaek+4Cbmq5nkjWfRvGp7bPA7lbdwBxgO/Byej6p6VpLNd9Lcbp8hOL/MFd/Xr3ATemY7AUubbr+z9mH/wCeA3alP5r5vboPwB9RnM7vAnamx2VNHwf/YtAsc01fDphZwxwCZplzCJhlziFgljmHgFnmHAJmmXMImGXOIWCWuf8HMfaaoAq/+BUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# test an image\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import torchvision\n",
    "inv_normalize = torchvision.transforms.Normalize(\n",
    "    mean=[-0.485/0.229, -0.456/0.224, -0.406/0.225],\n",
    "    std=[1/0.229, 1/0.224, 1/0.225]\n",
    ")\n",
    "\n",
    "def imshow(img):\n",
    "    inv_tensor = inv_normalize(img)\n",
    "    npimg = inv_tensor.detach().cpu().numpy()\n",
    "    #print(\"min:\",np.min(npimg),\"max:\",np.max(npimg))\n",
    "    npimg = (((npimg)*255 )  ).astype(np.int32)\n",
    "    npimg = np.transpose(npimg, (1, 2, 0))\n",
    "    plt.imshow(npimg)#\n",
    "valid_iter = iter( valid_loader)\n",
    "img,lbl = next( valid_iter  )\n",
    "print(\"label:\",lbl[2])\n",
    "imshow(img[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f08060cd",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "The paper tested three pretrained image classification models with the final layer converted for score prediction. The models were: VGG16, Inception-v2, and MobileNet. It found Inception-v2 gave the slightly better results (accuracy of 81.51% vs 80.36%) but that MobileNet was twice as fast. We'll go with MobileNet as the huge speed gain trumps the slight accuracy deficit in production."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "1543430a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Dropout(p=0.2, inplace=False)\n",
      "  (1): Linear(in_features=1280, out_features=1000, bias=True)\n",
      ")\n",
      "Sequential(\n",
      "  (0): Dropout(p=0.2, inplace=False)\n",
      "  (1): Linear(in_features=1280, out_features=10, bias=True)\n",
      ")\n",
      " =============== RESTORE CHECKPOINT  ===============\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using cache found in /home/craig/.cache/torch/hub/pytorch_vision_v0.10.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MobileNetV2(\n",
       "  (features): Sequential(\n",
       "    (0): Conv2dNormActivation(\n",
       "      (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (2): ReLU6(inplace=True)\n",
       "    )\n",
       "    (1): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False)\n",
       "          (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (2): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(16, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False)\n",
       "          (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(96, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (3): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=144, bias=False)\n",
       "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(144, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (4): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(144, 144, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=144, bias=False)\n",
       "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(144, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (5): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
       "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (6): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
       "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (7): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=192, bias=False)\n",
       "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (8): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (9): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (10): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (11): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
       "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (12): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
       "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (13): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
       "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (14): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=576, bias=False)\n",
       "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(576, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (15): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
       "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (16): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
       "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (17): InvertedResidual(\n",
       "      (conv): Sequential(\n",
       "        (0): Conv2dNormActivation(\n",
       "          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (1): Conv2dNormActivation(\n",
       "          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
       "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "          (2): ReLU6(inplace=True)\n",
       "        )\n",
       "        (2): Conv2d(960, 320, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (3): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (18): Conv2dNormActivation(\n",
       "      (0): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (1): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (2): ReLU6(inplace=True)\n",
       "    )\n",
       "  )\n",
       "  (classifier): Sequential(\n",
       "    (0): Dropout(p=0.2, inplace=False)\n",
       "    (1): Linear(in_features=1280, out_features=10, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# GET PRETRAINED    = =  M O D E L  = =\n",
    "model = torch.hub.load('pytorch/vision:v0.10.0', 'mobilenet_v2', pretrained=True)\n",
    "print(model.classifier)\n",
    "# instead of classifing images we want an aesthetic scale of 1 to 10\n",
    "model.classifier[1] = torch.nn.Linear(in_features=model.classifier[1].in_features, out_features=10)\n",
    "print(model.classifier)\n",
    "if False: # RECOVER from checkpoint\n",
    "  print(\" =============== RESTORE CHECKPOINT  ===============\")\n",
    "  checkpoint = torch.load(\"./models/model-NIMA-chk.pt\")\n",
    "  model.load_state_dict(checkpoint['model_state_dict'])\n",
    "  #optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n",
    "  epoch = checkpoint['epoch']\n",
    "  loss = checkpoint['loss']\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d96442b5",
   "metadata": {},
   "source": [
    "## Loss Function\n",
    "\n",
    "One insight of the paper was to use an Earth Movers Distance(EMD) loss function to better approximate the complexities of images that have high standard deviations. For example, an image scoring low with a high standard deviation, is better than an image scoring low with a low deviation. The high deviation indicates it is a controversial image which some people like and some dislike.\n",
    "\n",
    "Pytorch does not have this loss built-in (and recommends not using it for [point clouds](https://github.com/facebookresearch/pytorch3d/issues/211) ) so you need to implement your own, with a little help from the [internet](https://discuss.pytorch.org/t/implementation-of-squared-earth-movers-distance-loss-function-for-ordinal-scale/107927)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "770f85cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def earth_mover_distance(y_true, y_pred):\n",
    "    return torch.sum(torch.mean(torch.square(torch.cumsum(y_true, dim=-1) - torch.cumsum(y_pred, dim=-1)), dim=-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0c4cc8a",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "The paper hints at using multiple rounds of 10 epochs, however as will be clear by the loss graph after training, 4 epochs is sufficient. For the first 2 epoch only the last classification layer is trained. Then all model layers are trained for 2 more epochs. The paper used stochastic gradient descent but we'll use Adam with a higher learning rate than the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8564bef2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MobileNetV2(\n",
      "  (features): Sequential(\n",
      "    (0): Conv2dNormActivation(\n",
      "      (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (2): ReLU6(inplace=True)\n",
      "    )\n",
      "    (1): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False)\n",
      "          (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (2): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(16, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False)\n",
      "          (1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(96, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (3): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=144, bias=False)\n",
      "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(144, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (4): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(144, 144, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=144, bias=False)\n",
      "          (1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(144, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (5): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
      "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (6): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)\n",
      "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (7): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(192, 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=192, bias=False)\n",
      "          (1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (8): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (9): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (10): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (11): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)\n",
      "          (1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (12): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
      "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (13): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)\n",
      "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (14): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(576, 576, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=576, bias=False)\n",
      "          (1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(576, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (15): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (16): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (17): InvertedResidual(\n",
      "      (conv): Sequential(\n",
      "        (0): Conv2dNormActivation(\n",
      "          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (1): Conv2dNormActivation(\n",
      "          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)\n",
      "          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (2): ReLU6(inplace=True)\n",
      "        )\n",
      "        (2): Conv2d(960, 320, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "        (3): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      )\n",
      "    )\n",
      "    (18): Conv2dNormActivation(\n",
      "      (0): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
      "      (1): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (2): ReLU6(inplace=True)\n",
      "    )\n",
      "  )\n",
      "  (classifier): Sequential(\n",
      "    (0): Dropout(p=0.2, inplace=False)\n",
      "    (1): Linear(in_features=1280, out_features=10, bias=True)\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "for param in model.parameters():\n",
    "  param.requires_grad = False\n",
    "for param in model.classifier.parameters():\n",
    "  param.requires_grad = True\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "275df878",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "optim = torch.optim.Adam(model.parameters(),  lr=LEARNING_RATE)\n",
    "decayRate = 0.9\n",
    "my_lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer=optim, gamma=decayRate)\n",
    "#NUM_EPOCHS=7 # customize for additional training epochs\n",
    "#LEARNING_RATE = 0.000005 # tweak lr, orig:0.00001\n",
    "\n",
    "historyt=[]\n",
    "historyv=[]\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    with tqdm(total=train_size) as pbar:\n",
    "        nn=0\n",
    "        for imgs,lbls in  train_loader:\n",
    "            preds = model(imgs)\n",
    "            loss = earth_mover_distance(lbls,preds)\n",
    "            optim.zero_grad()\n",
    "            loss.backward()\n",
    "            optim.step()\n",
    "            Loss=np.round(loss.cpu().detach().numpy().item(), 5)\n",
    "            pbar.set_postfix(                Loss=Loss,                )\n",
    "            pbar.update(BATCH_SIZE)\n",
    "            pbar.refresh()\n",
    "            if nn % 100 == 0 and nn>0:# validate\n",
    "              model.eval()\n",
    "              with torch.no_grad():\n",
    "                  img,lbl = next( iter( valid_loader)  )\n",
    "                  pred = model(img)\n",
    "                  historyt.append(Loss)\n",
    "                  loss = earth_mover_distance(lbl,pred)\n",
    "                  Loss=np.round(loss.cpu().detach().numpy().item(), 5)\n",
    "                  historyv.append(Loss)\n",
    "                  print(f'validation loss: {Loss:.5f} ')\n",
    "              model.train()\n",
    "  \n",
    "            nn +=  1\n",
    "    my_lr_scheduler.step()\n",
    "    if epoch ==1:\n",
    "        print(\"==== STARTING TRAINING ON FULL MODEL=====\")\n",
    "        for param in model.parameters():\n",
    "          param.requires_grad = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5364164",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "The fist phase, just training the final layer reached a loss of 4.5, in the second phase, training the full model, loss reached: 2.5\n",
    "The accuracy (below) matches the paper very well, right near 80%. Note that the paper only computed accuracy for binary classification; whether the picture was good or bad (score over 5 or below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "d94e1926",
   "metadata": {},
   "outputs": [],
   "source": [
    "# first SAVE!\n",
    "state_dict = model.state_dict()\n",
    "torch.save(state_dict, \"./models/model-NIMA-dict.pt\")\n",
    "torch.save({\n",
    "            'epoch': epoch,\n",
    "            'model_state_dict': model.state_dict(),\n",
    "            'optimizer_state_dict': optim.state_dict(),\n",
    "            'loss': loss,\n",
    "            }, \"./models/model-NIMA-chk.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4b252bff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3UAAAHcCAYAAABxgoMvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAACKU0lEQVR4nOzdd3jV5f3G8feTvfdgE2bYG0EQRQQVRYYbFXBrtVpHax21WqtWbatW7c9RF7hFQAVxMURZsvcegTATsvc4eX5/nJOYQIBAcnII3K/rypWc7/x8T6LkzrOMtRYRERERERFpmLw8XYCIiIiIiIicPIU6ERERERGRBkyhTkREREREpAFTqBMREREREWnAFOpEREREREQaMIU6ERERERGRBkyhTkTkDGSMSTDGWGOMTw2OvdEYM78+6joVGGMGGWM21/WxpzJjzPXGmB88XYeIiJwchToRkVOcMSbJGFNsjIk5bPsqVzBL8FBplcPhN4dt/9AY86Tr68HGmD2V9v3kOqf7Yed86do++LDtN7q2X32MOh41xuS6PgqNMY5Kr9efyDNZa3+x1ibW9bEn6kTfp6Nco0bh3Vr7kbX2wtpVLCIinqJQJyLSMOwExpa/MMZ0BQI9V84R+htjBp7A8VuA8eUvjDHRQH8gtZpjJwDprs/VstY+a60NsdaGAHcCi8pfW2s7V7qPMcY0pH/7TuR9Oik1aa0VEZFTW0P6h01E5Ez2AZV+uccZcCZVPsAYE26MmWSMSTXG7DLG/KU8wBhjvI0x/zLGHDLG7AAurebcd4wx+40xe40xTxtjvE+gvheAp0/g+I+AayrdYywwDSg+rK6WwHnA7cBFxpj4E7hH+TV+MsY8Y4xZAOQDrY0xNxljNhpjcowxO4wxd1Q6/vCWxSRjzB+NMWuMMVnGmM+MMQEneqxr/0Ou93ifMeZWVyta29q8T8YYL2PMw8aY7caYNGPM58aYKNfun12fM12tlme7Wj4XGGNeMsakA08e3sXWGNPZGPOjMSbdGHPQGPOoa/tZxphlxphs1/YXT+R7ISIi7qFQJyLSMCwGwowxHV2/4F8DfHjYMa8C4UBrnEFoPHCTa99twAigJ9AHuPKwcycCpUBb1zEXAreeQH3/BdobY4bW8Ph9wAbXfXDVOqma48YDy6y1U4CNwPUnUFNl43AGw1BgF5CC8/0Iw/kevWSM6XWM868GLgZaAd2AG0/0WGPMxcADwFCc7/N5Nai7Ju/TvcBo1/WaABk4vx8A57o+R7haLRe5XvcDdgBxwDOVL2aMCQVmAd+5rtcWmO3a/R/gP9baMKAN8HkNnkFERNxMoU5EpOEob60bBmwC9pbvqBT0HrHW5lhrk4B/4wwz4AwaL1trk6216cA/Kp0bDwwH7rPW5llrU4CXgGtPoLZCnOHgRFrrJgHjjTGJOEPHomqOGQ987Pr6Y47RBfM43rfWrrfWllprS6y131hrt1unecAPwKBjnP+KtXaf672bDvQ4iWOvBt5z1ZEP/K2GtR/vfboDeMxau8daWwQ8CVx5nG6V+6y1r7rej4LD9o0ADlhr/22tLXT9PP3q2lcCtDXGxFhrc621i2v4DCIi4kYKdSIiDccHwHU4W34Ob62JAfxwtkKV2wU0dX3dBEg+bF+5loAvsN8Yk2mMyQTexNmKcyL+B8QbYy6r4fFTgSHAPTifrQrXGL1WwKeuTR8DXY0xPU6wLqj67BhjhhtjFru6F2YCl+B8D4/mQKWv84GQkzj28O9BlZqO4ZjvE87v37RK37uNgAM4VlfVY927ObD9KPtuAdoDm4wxS40xI45Tu4iI1AMNjhYRaSCstbuMMTtxBpBbDtt9CGcrSkuc3fUAWvBba95+nL+sU2lfuWSgCIix1pbWor4SY8zfgL8Dx51x0lqbb4z5Fvgdzq58h5sAGGCVMaby9vHAqhMtr/wLY4w/MMV1na9cdX/pupc77QeaVXrd/GgHVlaD9ykZuNlau+DwHa4xidVe9hi3TKbSpDyH1bIVGOsaq3k58IUxJtpam3esZxAREfdSS52ISMNyCzDk8F+irbUOnOObnjHGhLp+mX+A38bdfQ7ca4xpZoyJBB6udO5+nN0P/22MCXNNvNHGGFOTMV+H+wDwxzmmrCYeBc5zdRet4Jpc5Gqc4+B6VPq4B7i+ljM2+rlqTAVKjTHD+W3Mmjt9DtzkGhcZBPz1BM6t9n1yeQPn970lgDEm1hgzyrUvFSjDOc6ypmYAjYwx9xlj/F0/T/1c177BGBNrrS0DMl3HO07g2iIi4gYKdSIiDYhrHNiyo+y+B8jDOQHGfJzdFd917fsf8D2wGliBs0tfZeNxhp0NOCfa+AJofBL1OYAngKjjHes6fp+1trqFzUcDBcAka+2B8g/gHcCbmofG6u6Zg3Nykc9xPut1wNcne70TuO+3wCvAXGAbUD42rqgG5x7tfQLn5CVfAz8YY3JwTqrTz3VePs6xjgtc3TP71+BeOTjHbV6GsyvpVuB81+6LgfXGmFzXfa+11hYe75oiIuJextpj9cAQERERdzDGdATWAf616fYqIiKiljoREZF6YowZY4zxc3WBfR6YrkAnIiK1pVAnIiJSf+7AOc5tO86xaL/zbDkiInI6UPdLERERERGRBkwtdSIiIiIiIg2YQp2IiIiIiEgDplAnIiIiIiLSgCnUiYiIiIiINGAKdSIiIiIiIg2YQp2IiIiIiEgDplAnIiIiIiLSgCnUiYiIiIiINGAKdSIiIiIiIg2YQp2IiIiIiEgDplAnIiIiIiLSgCnUiYiIiIiINGAKdSIiIiIiIg2YQp2IiIiIiEgDplAnIiIiIiLSgCnUiYiIiIiINGAKdSIiIiIiIg2YQp2IiIiIiEgDplAnIiIiIiLSgCnUiYiIiIiINGAKdSIiIiIiIg2YQp2IiHiUMeZCY8yXHrivvzFmkzEmrr7vXV+MMW8YYx6v62NFROTUolAnIiIYY5KMMUM9dPtngecq1WKNMXnGmNxKHw+59j3p2n9v5QsYY+5zbX/S9XqwMaas0vl7jDGfG2P6lp9jrS0C3gX+XB8PeaLq4ntirb3TWvv3uj5WREROLQp1IiLiMa6QFW6tXXzYru7W2pBKHy9U2rcFmHDY8eNd2yvbZ60NAUKB/sAm4BdjzAWVjvkYmGCM8a/1w5wk43TC/x4bY3zcUY+IiDQ8CnUiInJUri6KLxtj9rk+Xi4PQMaYGGPMDGNMpjEm3RjzS3k4Mcb82Riz1xiTY4zZfFiQqmw4MO8Ey1oKBBljOrvu1RkIdG0/gnXaY639K/A28HylfXuADJyh70Sff6MxZkSlY32MMYeMMb1cr/sbYxa63p/VxpjBlY79yRjzjDFmAZAPtD7svh8ALYDp5S2VxpgEV2vkLcaY3cAc17GTjTEHjDFZxpify98X1773jTFPu74e7GqxfNAYk2KM2W+Muekkj402xkw3xmQbY5YaY542xsyv/tslIiLuplAnIiLH8hjOwNMD6A6cBfzFte9BYA8QC8QDjwLWGJMI/B7oa60NBS4Cko5y/a7A5pOo6wOcrXPgbLWbVMPzpgK9jDHBlbZtxPls1TnW838CjK107EXAIWvtCmNMU+Ab4GkgCvgjMMUYE1vp+HHA7ThbEndVvqm1dhywG7ismpbK84COrvsBfAu0A+KAFcBHx3j+RkA40BS4BfivMSbyJI79L5DnOmYCR7aciohIPVKoExGRY7keeMpam2KtTQX+hjOMAJQAjYGW1toSa+0v1loLOAB/oJMxxtdam2St3X6U60cAOdVsX+Fq4Sr/uOiw/R8CY40xvsC1rtc1sQ8wrvuWyznsdWXHev6PgZHGmCDX6+tc2wBuAGZaa2daa8ustT8Cy4BLKl37fWvtemttqbW2pIb1Azxprc2z1hYAWGvftdbmuMYIPgl0N8aEH+XcEtfzlFhrZwK5QOKJHGuM8QauAJ6w1uZbazcAE0+gfhERqWMKdSIicixNqNqKtMu1DeCfwDbgB2PMDmPMwwDW2m3AfTgDRoox5lNjTBOql4Gzpepwvay1EZU+vq+801q723XvZ4Gt1trkGj5PU8ACmZW2hR72urKjPr/rOTcCl7mC3Uh+C3UtgasqB1PgHJwhuFxNaz5cxXnGGG9jzHPGmO3GmGx+axGNOcq5adba0kqv84GQEzw2FvChav0n+ywiIlIHFOpERORY9uEMKOVauLbhah160FrbGrgMeKB87Jy19mNr7Tmucy2VxrEdZg3Q/iRrm4SzC2hNu14CjAFWWGvzKm3rCKw+yvFHfX6X8i6Yo4ANrqAHzpDzwWHBNNha+1ylc+1xaj3a/srbr3PdeyjOrpIJru3mONeujVSgFGhWaVtzN95PRESOQ6FORETK+RpjAip9+OAMLX8xxsQaY2KAv+Lq6miMGWGMaWuMMUA2zm6XDmNMojFmiGtCkUKgwLWvOjNxjhE7GZ8BFwKfH+sg1+ySTY0xTwC34hz7V76vKc4xb4fPvlnuqM/v8qmrht/xWysdrmMuM8Zc5GpNC3BNPlI5CB3PQQ6bQKUaoUARkAYE4Wy5dCtrrQPn2MQnjTFBxpgO/Da+UUREPEChTkREys3EGcDKP57EOdHHMpwtamtxTsTxtOv4dsAsnGOtFgH/Z639Ced4uueAQ8ABnBN4VASpyqy1K4AsY0y/w3atNlXXqXu5mnMLrLWzyseWVaOJMSbXVd9SnJOyDLbW/lDpmOuAia7xaNU51vNjrd3vevYBOENm+fZknC1oj+Js2UoG/sSJ/bv7D5yBMtMY88ejHDMJZ5fQvcAGjh5O69rvcbYMHsA5ac0nOMOliIh4gHGOaRcREfEMY8yFwF3W2tH1fF9/nN0uz7XWptTnvU83xpjngUbWWs2CKSLiAQp1IiIickJcXS79cLZe9sXZynurtfZLT9YlInKm8vF0ASIiItLghOLsctkESAH+DXzl0YpERM5gaqkTERERERFpwDRRioiIiIiISAPWILpfxsTE2ISEBE+XISIiIiIi4hHLly8/ZK2NrW5fgwh1CQkJLFu2zNNliIiIiIiIeIQxZtfR9qn7pYiIiIiISAOmUCciIiIiItKAKdSJiIiIiIg0YA1iTJ2IiIiIiJy5SkpK2LNnD4WFhZ4uxe0CAgJo1qwZvr6+NT5HoU5ERERERE5pe/bsITQ0lISEBIwxni7Hbay1pKWlsWfPHlq1alXj89T9UkRERERETmmFhYVER0ef1oEOwBhDdHT0CbdIKtSJiIiIiMgp73QPdOVO5jkV6kRERERERBowhToREREREZFjSEtLo0ePHvTo0YNGjRrRtGnTitfFxcXHPHfZsmXce++9bq1PE6WIiIiIiIgcQ3R0NKtWrQLgySefJCQkhD/+8Y8V+0tLS/HxqT5a9enThz59+ri1PrXUiYiIiIiInKAbb7yRBx54gPPPP58///nPLFmyhAEDBtCzZ08GDBjA5s2bAfjpp58YMWIE4AyEN998M4MHD6Z169a88sordVKLWupERERERKTBuO++OaxalVKn1+zRI46XXx5ywudt2bKFWbNm4e3tTXZ2Nj///DM+Pj7MmjWLRx99lClTphxxzqZNm5g7dy45OTkkJibyu9/97oTWpKuOQp2IiIiIiMhJuOqqq/D29gYgKyuLCRMmsHXrVowxlJSUVHvOpZdeir+/P/7+/sTFxXHw4EGaNWtWqzoU6kREREREpME4mRY1dwkODq74+vHHH+f8889n2rRpJCUlMXjw4GrP8ff3r/ja29ub0tLSWtehMXUnKSspiYxt2zxdhoiIiIiInAKysrJo2rQpAO+//3693luh7iTNHDeOH26/3dNliIiIiIjIKeChhx7ikUceYeDAgTgcjnq9t7HW1usNT0afPn3ssmXLPF1GFVMuuYT81FTGLV3q6VJERERERE5rGzdupGPHjp4uo95U97zGmOXW2mrXRlBL3UnyCw2lJDfX02WIiIiIiMgZTqHuJPmGhFCck+PpMkRERERE5AynUHeS/EJDFepERERERMTjFOpOUnn3y4YwJlFERERERE5fCnUnyS80FFtWRmlBgadLERERERGRM5hC3UnyDQkBUBdMERERERHxKIW6k+QXGgoo1ImIiIiInO4GDx7M999/X2Xbyy+/zF133XXU4+tzSTaFupOkUCciIiIicmYYO3Ysn376aZVtn376KWPHjvVQRVUp1J2k8lCntepERERERE5vV155JTNmzKCoqAiApKQk9u3bx8cff0yfPn3o3LkzTzzxhMfq8/HYnRs4P42pExERERGpd3Puu4+UVavq9JpxPXow5OWXj7o/Ojqas846i++++45Ro0bx6aefcs011/DII48QFRWFw+HgggsuYM2aNXTr1q1Oa6sJtdSdJF91vxQREREROWNU7oJZ3vXy888/p1evXvTs2ZP169ezYcMGj9SmlrqTpDF1IiIiIiL171gtau40evRoHnjgAVasWEFBQQGRkZH861//YunSpURGRnLjjTdSWFjokdrUUneSyrtfakydiIiIiMjpLyQkhMGDB3PzzTczduxYsrOzCQ4OJjw8nIMHD/Ltt996rDa11J0ktdSJiIiIiJxZxo4dy+WXX86nn35Khw4d6NmzJ507d6Z169YMHDjQY3Up1J0kLx8ffAICFOpERERERM4QY8aMwVpb8fr999+v9riffvqpfgpyUffLWvANDaVY3S9FRERERMSDFOpqwS8khBK11ImIiIiIiAcp1NWCX2ioul+KiIiIiNSDyt0eT2cn85wKdbWgUCciIiIi4n4BAQGkpaWd9sHOWktaWhoBAQEndJ4mSqkF39BQCtPTPV2GiIiIiMhprVmzZuzZs4fU1FRPl+J2AQEBNGvW7ITOUairBb+QELJ37fJ0GSIiIiIipzVfX19atWrl6TJOWep+WQt+oaGaKEVERERERDxKoa4WNKZOREREREQ8TaGuFnxDQijOzT3tB2yKiIiIiMipS6GuFvxCQ7EOB6WFhZ4uRUREREREzlAKdbXgFxoKoC6YIiIiIiLiMQp1tVAe6jRZioiIiIiIeIpCXS34hoQAUJyb6+FKRERERETkTKVQVwvqfikiIiIiIp6mUFcLCnUiIiIiIuJpCnW1oFAnIiIiIiKeplBXC+Vj6ko0pk5ERERERDxEoa4W1FInIiIiIiKeplBXCwp1IiIiIiLiaQp1teDt64u3v7+6X4qIiIiIiMco1NWSX0iIWupERERERMRjFOpqyTc0VKFOREREREQ8RqGulvwU6kRERERExIMU6mrJLyREY+pERERERMRjFOpqSS11IiIiIiLiSQp1taQxdSIiIiIi4kkKdbWkljoREREREfEkhbpa0pg6ERERERHxJIW6WipvqbPWeroUERERERE5AynU1ZJfaChlpaU4ioo8XYqIiIiIiJyBFOpqyTc0FEDj6kRERERExCMU6mrJLyQEgGKNqxMREREREQ9QqKslP1dLXYla6kRERERExAPcHuqMMd7GmJXGmBmu11HGmB+NMVtdnyPdXYM7+an7pYiIiIiIeFB9tNT9AdhY6fXDwGxrbTtgtut1g+Vb3v1SoU5ERERERDzAraHOGNMMuBR4u9LmUcBE19cTgdHurMHdKlrqNKZOREREREQ8wN0tdS8DDwFllbbFW2v3A7g+x1V3ojHmdmPMMmPMstTUVDeXefLU/VJERERERDzJbaHOGDMCSLHWLj+Z8621b1lr+1hr+8TGxtZxdXVHE6WIiIiIiIgn+bjx2gOBkcaYS4AAIMwY8yFw0BjT2Fq73xjTGEhxYw1u56slDURERERExIPc1lJnrX3EWtvMWpsAXAvMsdbeAHwNTHAdNgH4yl011Acff3+8fH3V/VJERERERDzCE+vUPQcMM8ZsBYa5XjdofqGhCnUiIiIiIuIR7ux+WcFa+xPwk+vrNOCC+rhvfVGoExERERERT/FES91pxzckhBKNqRMREREREQ9QqKsDaqkTERERERFPUairA/7h4RSmp3u6DBEREREROQMp1NWBiLZtydiyBWutp0sREREREZEzjEJdHYju2JHinBxy9+71dCkiIiIiInKGUairA9GdOgGQtnGjhysREREREZEzjUJdHYju2BGAdIU6ERERERGpZwp1dSAoPh7/iAi11ImIiIiISL1TqKsDxhiiO3ZUqBMRERERkXqnUFdHojp2VPdLERERERGpdwp1dSS6Y0fyU1Io0Hp1IiIiIiJSjxTq6kiUJksREREREREPUKirI+UzYGpcnYiIiIiI1CeFujoS1rIlPgEBCnUiIiIiIlKvFOrqiJe3N5GJiep+KSIiIiIi9Uqhrg5pWQMREREREalvCnV1KKpjR7KTkijOy/N0KSIiIiIicoZQqKtD5ZOlZGze7OFKRERERETkTKFQV4eiNAOmiIiIiIjUM4W6OhTZrh3Gy0uTpYiIiIiISL1RqKtDPv7+RLRpo5Y6ERERERGpNwp1dSwyMZGMrVs9XYaIiIiIiJwhFOrqWEjjxuQfPOjpMkRERERE5AyhUFfHguLjKTh0iDKHw9OliIiIiIjIGUChro4FxcVhy8ooSEvzdCkiIiIiInIGUKirY0Hx8QDqgikiIiIiIvVCoa6OBZeHupQUD1ciIiIiIiJnAoW6OhYUFweopU5EREREROqHQl0dC1JLnYiIiIiI1COFujoWEBmJl48PeWqpExERERGReqBQV8eMMQTFxan7pYiIiIiI1AuFOjcIio9X90sREREREakXCnVuEBQXp+6XIiIiIiJSLxTq3EAtdSIiIiIiUl8U6twgOD6e/IMHsdZ6uhQRERERETnNKdS5QVBcHI6iIoqzsz1dioiIiIiInOYU6txAa9WJiIiIiEh9Uahzg2BXqNNkKSIiIiIi4m4KdW4QFBcHqKVORERERETcT6HODSq6X6qlTkRERERE3Eyhzg2CYmMBdb8UERERERH3U6hzAy8fHwKjo9X9UkRERERE3E6hzk2CXGvViYiIiIiIuJNCnZsExcWppU5ERERERNxOoc5N1FInIiIiIiL1QaHOTYLj4zVRioiIiIiIuJ1CnZsExcVRnJ1NaWGhp0sREREREZHTmEKdm1SsVadxdSIiIiIi4kYKdW4SrFAnIiIiIiL1QKHOTYLi4gAtQC4iIiIiIu6lUOcmFd0vFepERERERMSNFOrcpLylTt0vRURERETEnRTq3MQ3KAjfkBC11ImIiIiIiFsp1LlRcHy8WupERERERMStFOrcKCguThOliIiIiIiIWynUuVFwo0bk7d/v6TJEREREROQ0plDnRlEdO5KxZQulRUWeLkVERERERE5TCnVuFN+rF2WlpRxat87TpYiIiIiIyGlKoc6N4nr2BCBlxQoPVyIiIiIiIqcrhTo3Cm/VCv/wcA4q1ImIiIiIiJso1LmRMYa4Xr1IWbnS06WIiIiIiMhpSqHOzeJ69iR19WrKSks9XYqIiIiIiJyGFOrcLL5XL0oLC0nbtMnTpYiIiIiIyGlIoc7N4nv1AlAXTBERERERcQuFOjeLbN8en6AgzYApIiIiIiJu4bZQZ4wJMMYsMcasNsasN8b8zbU9yhjzozFmq+tzpLtqOBV4eXsT1727ZsAUERERERG3cGdLXREwxFrbHegBXGyM6Q88DMy21rYDZrten9bKZ8C0ZWWeLkVERERERE4zbgt11inX9dLX9WGBUcBE1/aJwGh31XCqiO/Zk+KcHDJ37PB0KSIiIiIicppx65g6Y4y3MWYVkAL8aK39FYi31u4HcH2Oc2cNp4I412Qp6oIpIiIiIiJ1za2hzlrrsNb2AJoBZxljutT0XGPM7caYZcaYZampqW6rsT7EdO6Ml6+vJksREREREZE6Vy+zX1prM4GfgIuBg8aYxgCuzylHOecta20fa22f2NjY+ijTbbz9/Ijp0kXLGoiIiIiISJ1z5+yXscaYCNfXgcBQYBPwNTDBddgE4Ct31XAqievRg5RVqzxdhoiIiIiInGbc2VLXGJhrjFkDLMU5pm4G8BwwzBizFRjmen3ai+7UifyUFArS0z1dioiIiIiInEZ83HVha+0aoGc129OAC9x131NVdKdOAKRv3EjTgQM9XI2IiIiIiJwu6mVMnUBUx44ApG3c6OFKRERERETkdKJQV0/CW7bEJzCQtA0bPF2KiIiIiIicRhTq6onx8iKqQwe11ImIiIiISJ1SqKtH0R07kq5QJyIiIiIidUihrh5FdexI9q5dFOfleboUERERERE5TSjU1aNo12Qp6Zs2ebgSERERERE5XSjU1aPKyxqIiIiIiIjUBYW6ehTRti1ePj6aLEVEREREROqMQl098vb1JaJtW7XUiYiIiIhInVGoq2fRHTtqrToREREREakzCnX1LLpTJzK2bcNRXOzpUkRERERE5DSgUFfPojp2xDocZGzb5ulSRERERETkNKBQV88qljXQuDoREREREakDCnX1LDIxEUDj6kREREREpE4o1NUzv+BgwhIStKyBiIiIiIjUCZ+aHGSM6QMMApoABcA6YJa1Nt2NtZ22ojt2VPdLERERERGpE8dsqTPG3GiMWQE8AgQCm4EU4BzgR2PMRGNMC/eXeXqJ6tiR9M2bsWVlni5FREREREQauOO11AUDA621BdXtNMb0ANoBu+u4rtNaVGIipQUFZCcnE96ypafLERERERGRBuyYoc5a+9/j7F9Vp9WcIaJck6VkbN6sUCciIiIiIrVy3IlSjDHnG2OmGmPWuz6+MMYMdn9pp6+oDh0ASN+0ycOViIiIiIhIQ3e8MXWXAu8C04HrgOuBmcC7xphL3F/e6SkoLg7/8HDSN2/2dCkiIiIiItLAHW9M3Z+A0dba1ZW2rTLGLANexRnw5AQZY4hMTFSoExERERGRWjte98tGhwU6AKy1a4B495R0Zojq0EHdL0VEREREpNaOF+ryTnKfHEdUYiK5e/dSnJvr6VJERERERKQBO173yzbGmK+r2W6A1m6o54xRMQPmli3E9+rl4WpERERERKShOl6oG3WMff+qy0LONJVnwFSoExERERGRk3W8derm1VchZ5qItm0xXl6aLEVERERERGrlmKHOGLPmWPuttd3qtpwzh4+/P2EJCQp1IiIiIiJSK8frflkGWOBjnGvVFbi9ojOIZsAUEREREZHaOubsl9baHsBYIARnsHsG6Azstdbucnt1p7moxEQytmzBlpV5uhQREREREWmgjrekAdbaTdbaJ6y1vXC21k0C7nd7ZWeAqMRESgsKyE5O9nQpIiIiIiLSQB2v+yXGmKbAtcAYIANnoJvm5rrOCOUzYGZs3kx4y5YerkZERERERBqiY7bUGWPm4Wyd8wVuBCYA3wB+xpgot1d3mitfq06TpYiIiIiIyMk6XktdS5wTpdwB3F5pu3Ft1wLktRAUH49/eLgmSxERERERkZN2vHXqEuqpjjOSMYbIxES11ImIiIiIyEk7XvfLGyp9PfCwfb93V1FnkqjERFJWrCBj61ZPlyIiIiIiIg3Q8Wa/fKDS168etu/mOq7ljNTznnvAGD7s25ftM2ZUbLfWaqkDERERERE5ruONqTNH+bq613ISGvfty7jly/n6iiuYdtlltL/ySvIPHuTQunX4hYVx6/bteHl7e7pMERERERE5RR2vpc4e5evqXstJCk9I4Nr58+l2220k//QTtqyMmG7dyN61i6wdOzxdnoiIiIiInMKO11LXwRizBmerXBvX17hea+bLOuQbGMiFb73FhW+9BcD+X3/lo/79ObR+PZHt2nm4OhEREREROVUdL9R1rJcq5AjRnToBkLZ+Pe1Gj/ZsMSIiIiIicso6Xqjbba09ZjdLY4w53jFy4vxCQwlt0YJD69d7uhQRERERETmFHW9M3VxjzD3GmBaVNxpj/IwxQ4wxE4EJ7ivvzBbTuTNpCnUiIiIiInIMxwt1FwMO4BNjzD5jzAZjzA5gKzAWeMla+76bazxjRXfuTPqmTZSVlnq6FBEREREROUUds/ultbYQ+D/g/4wxvkAMUGCtzayH2s54MZ074yguJnP7dqISEz1djoiIiIiInIKO11JXwVpbYq3dr0BXf6I7dwbQuDoRERERETmqGoc6qX/RHZ2Tj2pcnYiIiIiIHI1C3SnMLySEsIQEtdSJiIiIiMhR1SjUGWOCjTFerq/bG2NGusbYnbE2bkxjzZpUt99HM2CKiIiIiMix1LSl7mcgwBjTFJgN3AS8766iGoLbb/+BP/xhjtvvE925M+mbN+MoKXH7vUREREREpOGpaagz1tp84HLgVWvtGKCT+8o69cXFBZGSku/2+8R07kxZSQmZ27a5/V4iIiIiItLw1DjUGWPOBq4HvnFtO+ZyCKe72NggUlPdH+o0A6aIiIiIiBxLTUPdfcAjwDRr7XpjTGtgrtuqagDi4gI5dKgAh6PMrfeJ7tgRjKkYV1daVERWUpJb7ykiIiIiIg1HjUKdtXaetXaktfZ514Qph6y197q5tlNaXFwQ1kJaWoFb7+MbFER4q1YcWr+e9M2b+eiss3i3QwfyUlLcel8REREREWkYajr75cfGmDBjTDCwAdhsjPmTe0s7tcXFBQHU27i63bNn80Hv3mRs3YqjqIh9Cxe6/b4iIiIiInLqq2n3y07W2mxgNDATaAGMc1dRDcFvoc69LXUAMV27UpieTnyvXty4bh3efn7sXbDA7fcVEREREZFTX00nO/F1rUs3GnjNWltijLHuK+vUFxvrDHX1MVlK7/vuI7pTJzpccw1ePj7E9+nDPoU6ERERERGh5i11bwJJQDDwszGmJZDtrqIagvrsfhkUG0un66/Hy8eZwZsMGMDB5cspLSx0+71FREREROTUVtOJUl6x1ja11l5inXYB57u5tlNaVFQAXl6mxqGuqKiUrKyiOrl304EDcRQXc3D58jq5noiIiIiINFw1nSgl3BjzojFmmevj3zhb7c5Y3t5exMQE1jjU/fnPP9OnzweUldW+12qTAQMANK5ORERERERq3P3yXSAHuNr1kQ28566iGoq4uKAah7qdO7PYti2TJUv21/q+wXFxRLRtqxkwRURERESkxqGujbX2CWvtDtfH34DW7iysIYiLCyI1tWazX5Z3vZwyZUud3LvpwIHsXbgQa8/o+WpERERERM54NQ11BcaYc8pfGGMGAu6fy/8UFxtb8+6XmZnloW5rnQSxpgMHUpCaSsbWrbW+loiIiIiINFw1DXV3Av81xiQZY5KA14A73FZVA3Ei3S+zsooICPBh584sVq5MqfW9mwwcCKAumCIiIiIiZ7iazn652lrbHegGdLPW9gSGuLWyBiAuLoisrCKKikqPe2xmZhGjR7fF29vUSRfM6A4d8I+I0GQpIiIiIiJnuJq21AFgrc221pavT/fAsY41xjQ3xsw1xmw0xqw3xvzBtT3KGPOjMWar63PkSdbuceVr1R1vXJ21luzsYtq0iWDw4OZ88cWWWnfBNF5eNBkwQKFOREREROQMd0Kh7jDmOPtLgQettR2B/sDdxphOwMPAbGttO2C263WD9FuoO3YXzNzcEsrKLBER/lxxRXu2bMlg/fpDtb5/s3POIX3jRn5++GEKMzJqfT0REREREWl4ahPqjtnUZK3db61d4fo6B9gINAVGARNdh00ERteiBo8qD3XHG1eXmVkIQHi4P2PGtMMY54QptdXj7rvpNG4cS154gbfbtGH5f/6j2TBFRERERM4wxwx1xpgcY0x2NR85QJOa3sQYkwD0BH4F4q21+8EZ/IC4ky/fs2JjaxbqypcziIjwp1GjYM45pxnTptU+1PmHhXHJpElMWLWK+N69mXvffRxYsqTW1xURERERkYbjmKHOWhtqrQ2r5iPUWutTkxsYY0KAKcB9lcbj1eS8240xy4wxy1JTU2t6Wr2qeUudM9SFh/sD0Lt3PNu3Z9ZZHbHdunHZ55/j5ePDlqlT6+y6IiIiIiJy6qtN98vjMsb44gx0H1lry9PGQWNMY9f+xkC18/tba9+y1vax1vaJjY11Z5knLSzMDz8/7xq31JWHushIf3JzSygpcdRZLQGRkbQYMoStU6eqC6aIiIiIyBnEbaHOGGOAd4CN1toXK+36Gpjg+noC8JW7anA3YwxxcUHHnf0yK6sYcHa/BIiKCgB+a8GrK23HjCFz2zYOrV9fp9cVEREREZFTlztb6gYC44AhxphVro9LgOeAYcaYrcAw1+sGqyYLkFeeKAUgMtIZ6tLTC+u0lrajRoExbFUXTBERERGRM0aNxsWdDGvtfI6+7MEF7rpvfYuNDaxB98uqLXXloS4jo25DXUjjxjQdMIBt06Yx4K9/rdNri4iIiIjIqcmtY+rOBDVtqfPz8yYgwJmhy7tf1nVLHUC7yy8nZdUqMnfsqPNri4iIiIjIqUehrpbKQ92xJifJyiomPNyv4rW7WurAOa4OYOu0aXV+bREREREROfUo1NVSXFwQBQWl5OWVHPWYrKwiIiICKl6Xt9S5I9RFtGpFXI8ebFOoExERERE5IyjU1VL5WnWpqUfvgpmZWVilpa58bJ07ul+Cswvm3oULyTtwwC3XFxERERGRU4dCXS39tgD50Zc1yMoqrtJS5+vrTUiILxkZdbukQblWw4eDtST/9JNbri8iIiIiIqcOhbpaio0NBDjmZCmHt9SBswtmevqx17c7WXE9e+IXFqZQJyIiIiJyBlCoq6XfWuqOHuoOb6kD52Qp7mqp8/L2ptm557J77ly3XF9ERERERE4dCnW1FBtbk1BXVG1LnTsmSinXfPBgMrZsIXffPrfdQ0REREREPE+hrpaCgnwJCfE96kQpJSUO8vJKCA/3r7I9MjLAbROlgDPUASTPm+e2e4iIiIiIiOcp1NWBYy1Anp1dDHBE90t3t9TF9eiBf3i4xtWJiIiIiJzmFOrqQGxsELt2ZVe7AHlmpnPc3OHdL93dUlc+rk6hTkRERETk9KZQVweGDGnB/Pl7ufvuWTgcZVX2ZWU5Q111E6UUFTkoKDj6ouW1pXF1IiIiIiKnP4W6OvD00+fw0EN9ef311YwZ8xV5ecUV+8pDXXUTpQBumwETKo2rU2udiIiIiMhpS6GuDnh5GZ5//jxee+0CvvlmBzfe+F3Fvt+6Xx4+UYrztbvWqgOI7d4d/4gIhToRERERkdOYj6cLOJ3cfXdPli8/yIwZ2yu2/db9smqoi4pyLlruzpY6jasTERERETn9qaWujrVvH0lqagE5Oc4umMdvqXPfZCngGle3dSsHli93631ERERERMQzFOrqWJs2EQBs354J/NZSFxZ25Dp1gFuXNQDocM01hDZrxuShQ9m7cKFb7yUiIiIiIvVPoa6OVRfqQkJ88fGp+lb/NlGKe0NdSJMmXDt/PoExMUweNozt33zDwRUrWPveeyx+9llKCtw3pk9ERERERNxPY+rq2OGhLjOz6IjlDMDZcmeM+7tfAoS3bMnYX35h8oUXMm3EiCr7Qps1o/P48W6vQURERERE3EMtdXUsPNyfmJhAtm3LBJwtdYcvZwDOGTMjIwPc3lJXLrhRI66dN48h//kPl33+OTdt3Ehw48bs+Oaberm/iIiIiIi4h1rq3KBNm4gqLXWHT5JSLjIyoF5a6soFREbS6957K163vuQSNk+ejKOkBG9f33qrQ0RERERE6o5a6tygcqjLyio6YjmDclFR9ddSV53WI0ZQnJ3N3gULPFaDiIiIiIjUjkKdG7RpE05ycg7FxQ5X98ujt9S5c52642k5dCjefn7smDHDYzWIiIiIiEjtKNS5QZs2EZSVWZKSslwTpRwt1PnXa/fLw/mFhNB88GCNqxMRERERacAU6tygfAbMbdsyj9lS5+nulwCtL72U9E2byNy+3aN1iIiIiIjIyVGoc4O2bSMBWLfuECUlZcdoqXOGOmttfZZXRetLLwVQa52IiIiISAOlUOcG8fFBBAf7smLFQYBjttQ5HJacnOL6LK+KiDZtiOrQge0aVyciIiIi0iAp1LmBMYbWrcNZvvzYoS4y0rkouce7YI4YwZ558yjOyfFoHSIiIiIicuIU6tykTZuIigXIj9X9EvDoZCkAbUeOxFFczJLnn/doHSIiIiIicuIU6tykfLIUOHb3S8CjyxoAND3nHLrcfDOLn3mGzV984dFaRERERETkxCjUuUnlUHf8lrqC+ijpqIwxDP2//6Nx//58O2ECqWvWeLQeERERERGpOYU6N2nbNqLi61O9pQ7Ax9+fUVOnEhARwbRRo8jZs8fTJYmIiIiISA0o1LlJTbpfRkY6t3t6opRyIY0bM+rLLyk4dIiJ3buz7euvPV2SiIiIiIgch0Kdm7RoEYaPjxfe3obgYN9qjwkK8sXX18vjE6VU1rhvX8YtX05Yy5Z8OWoUs++5h4K0NE+XJSIiIiIiR6FQ5yY+Pl60bBlGeLg/xphqjzHGEBUVcMq01JWLat+e6xYtovf997Pytdd4vVEjpl52GRs/+YSy0lJPlyciIiIiIpUo1LlRmzYRR50kpVxkZMAp1VJXzsffn/NffJEJq1fT+/77SV21im+uu45vJ0zAlpV5ujwREREREXFRqHOjxx7rxz/+MeiYx5yKLXWVxXbrxnkvvMDtu3Yx8O9/Z+PHH/PTgw9irQUgc/t25tx3H+mbN3u4UhERERGRM5OPpws4nZ17bvPjHhMZGcD+/Xn1UM2xbdqURmJi1NG7inp50f+xxyg4dIjlL7+Mf0QEpQUFLH/pJRzFxZTm53PhW2/Vc9UiIiIiIqJQ52FRUQEsXXqA995bS0ZGES1bhnHFFe3rtYZVq1Lo2XMSc+ZczfnntzjqccYYzn/xRQpSU1n45JMAdJ4wgbz9+9k+fTq2rAzjpcZfEREREZH6pFDnYS1ahJGSks/NN38PgLe3ITv7XoKCqp8x0x02bUoHYPXq1GOGOnC22F383nvEdu9O8/PPp3Hfvmz46CNm3nADB5YupXG/fvVRsoiIiIiIuKhZxcMef7w/K1eOZ8eOW5k4cTgOh2X9+kP1WkNycjYAW7ak1+h4bz8/znroIRr37QtAq+HDMd7eWtdORERERMQDFOo8zN/fhx494mjVKoIBA5oAzhaz+pScnAPAli0ZJ3V+YFQUzc49l+0KdSIiIiIi9U6h7hTSunUEISG+9R7qdu+uXagDaDtyJIfWrSNzx466KktERERERGpAoe4U4uVl6NYt1mMtdcnJOeTnl5zUNdqMHAmg1joRERERkXqmUHeK6d49ltWrUyrWgasPycnZREYGALBtW+ZJXSOidWtiunTRuDoRERERkXqmUHeK6d49juzsYpKSsurlfoWFpaSmFjB4sHNNva1bT74LZpuRI9nz888UpNdswhUREREREak9hbpTTPfusUD9TZayZ4+z6+WQIc5QV6txdaNGYR0OVr/+er22NIqIiIiInMkU6k4xXbvGYEz9hbry8XQdO0bTpElIjZc1qE6jPn1IuPhi5v/lL3w7fjzFOTl1VaaIiIiIiByFQt0pJjjYj3btIust1O3e7VyjrkWLMNq3j6xVS53x8uLyGTMY+NRTbPz4Yz7o04c98+fXVakiIiIiIlINhbpTUPlkKeWSk7OZOHGdW7o0lrfUNWsWQrt2tQt1AF7e3pz9+ONcPWcOJXl5fDpoEFNHjCBl9eq6KFdERERERA6jUHcK6t49jh07ssjOLsJay7hxM7nxxu/4/vukOr9XcnIOMTGBBAb60r59JIcOFZCeXlDr6zY/7zxu3ryZQf/4B3sXLGBSjx78+o9/1EHFIiIiIiJSmULdKah8spQ1a1KZOXMH8+btwdfXi0cf/YWysrptrUtOzqF581AA2rePBGDr1sw6ubZfcDD9Hn6Y23bsIPGaa/jl0UfZ8NFHdXJtERERERFxUqg7BfXoEQfAihUpPPTQz7RrF8nrrw9j5coUpk7dUqf3qj7U1a4L5uECIiO5ZNIkmp13Ht/ffDN7Fyyo0+uLiIiIiJzJFOpOQU2bhhAVFcA//vErGzak8dxzg7jxxs507BjF448vwOEoq7N7VQ51rVtH4OVlajUD5tF4+/kxasoUwlq25MtRo0jbtKnO7yEiIiIiciZSqDsFGWPo3j2WAwfyGDCgCWPGtMPb24u///0cNm1K58MPN9TJfbKzi8jKKqJFizAA/Py8SUgIq/VkKUcTGB3N5d98A8AHPXuy+JlnKC1yjhtMnjePH+64g9VvvaU17kREREREToCPpwuQ6vXsGcfcucn885/nYYwB4PLL29G7dzwPP/wLW7dmctZZjTj77CbExgad1D3KZ74sb6kDar2swfFEtmvH+FWrmHv//cz/y19YP3EiGEPGli14+/mx5q23OLBkCUP/7//w9vNzWx0iIiIiIqcLtdSdoh58sC9ffjmaAQOaVmwzxvDGG8No0iSE5577lVGjvqRVq/+xcWPaSd2j+lAXxZYtGW5tLQtt1oyRkydzxXff4RscTFBcHMMnTuTu9HT6P/44a995h8lDh5KfWj9r9YmIiIiINGQKdaeoJk1CGDWq7RHb+/RpxPLl48jOvpd5867Bz8+bu+6adVIh7GgtdXl5Jezfn3fyxddQq4suYvzKlYz95Rc6jx+PX3Aw5zz1FCM++YQDS5fy8YABZO7Y4fY6REREREQaMoW6BiooyJdzz23OP/4xiJ9+Subjjzee8DWSk7Mxxhkgy7Vr55wB0x2TpdRUh2uv5eo5cyhMT+fjAQM4uHKlx2oRERERETnVKdQ1cLfd1o2zzmrEgw/+RGZm4Qmdm5ycQ+PGIfj6elds69gxCoANG06uS2ddaXL22YydPx9vPz8+O+88tn75pSZQERERERGphkJdA+flZXj99WGkphZwzz1zeOWVFVx99dcMHvwpOTnFxzw3OTmHFi1Cq2xr1iyU8HB/1q495M6yayS6Y0euW7iQsJYt+WrMGD486yy2ff01B5Yv5+dHHuGdxESmX3utwp6IiIiInNE0++VpoFeveO6+uwevvrqSDz/cQESEP5mZRaxdm1plopXD7d6dQ48esVW2GWPo0iXmlAh14JxUZdzy5ayfNIlfn32WL0eNAsB4exPdsSObP/uMDtdcQ7sxYzxcqYiIiIiIZ6il7jTxwgvnMX36GHbvvp1ffrkW+G0ilOpYa6ssPF5Z164xrFt36JRpAfP286Pbrbdy8+bNXPrxx1z07rvcdfAg41euJKZrV+bcdx8l+fmeLlNERERExCMU6k4TAQE+jBjRhubNw2jWzBnU9uw5eqhLSyugsLCU5s3DjtjXtWsMWVlFxzzfE7x9fek4dixdb7qJwOhovHx8uOC118jZvZtfn33W0+WJiIiIiHiEQt1pKDzcn+Bg32O21C1atB/gKC11zi6Zp0oXzGNpfu65dLrhBpb+859kbN3q6XJEREREROqdQt1pyBhD8+ah1ba0JSdnc8MN3zBy5DQaNw5m4MAjx9x16RIDwNq1DWPx7/P++U+8AwL44Y47KHM4PF2OiIiIiEi9cluoM8a8a4xJMcasq7QtyhjzozFmq+tzpLvuf6Zr1iz0iJa6rVsz6NDhXb74YguPPtqPzZtvoVGj4CPOjYwMoFmz0AbRUgcQ3KgR57/0Eslz57Lg8ceP2F+Sn09eSgqZO3ZQlJXlgQpFRERERNzHnS117wMXH7btYWC2tbYdMNv1WtzA2VKXW2XbvHnJ5OeXsmjRdTzzzCBCQ/2Oen6XLtGsW9cwQh1A15tvptttt/HrP/7B1mnTADi0fj2fnHMO/wkO5vX4eN5u04Y3mzdn/QcfnDKTwIiIiIiI1JbbljSw1v5sjEk4bPMoYLDr64nAT8Cf3VXDmaxZsxD278+lpMRRsbj4tm2Z+Pp60a1b7HHOdo6rmzMnucr5p7ohr75KyurVfDthAnt++YWVr72Gf1gYA558ksDYWHyDglj7zjt8O348O2bMYOjrrxMYFeXpskVEREREaqW+x9TFW2v3A7g+xx3tQGPM7caYZcaYZampDWNs16mkefMwrIX9+/Mqtm3dmkHr1hF4ex//2961awzFxQ62bs2ok3qmT9/O7t3ZdXKto/Hx92fkF1/gHRDA8pdeIvHqq7lp40YGPPEEPe+6iy433sg1P/3EoGefZevUqUzq0YODK1e6tSYREREREXc7ZSdKsda+Za3tY63tExt7/JYlqapZsxCg6rIG27Zl0q5dRI3Or8sZMPfty2XUqGm8+OKyWl/reMKaN+faefO49uefufTDDwk67GfHy9ubfo88wtiFC8FaPjnnHLZMmeL2ukRERERE3KW+Q91BY0xjANfnlHq+/xmjfKmC8slSrLVs25ZB27Y1m5umY8covL1NnYS6KVO2YC3s3Fk/k5REd+xIs0GDjnlM4759uWHpUmK7dePrK69k/uOPU1pUVC/1iYiIiIjUpfoOdV8DE1xfTwC+quf7nzEOX4B8//488vNLads2okbn+/v70L595FEnS8nJKa7xZCOTJ28BICnJvd0vT1Rwo0ZcM3cunSdMYPHTT/Nep05smTJFk6iIiIiISIPiziUNPgEWAYnGmD3GmFuA54BhxpitwDDXa3GD8HB/QkJ+W4B82zbn2Lh27Wq+ikTXrrHVrlWXkpJHo0b/x0cfbTzuNfbty2X+/D34+Xmza9epFeoAfAICGP7++1z144/4BgXx9ZVX8sk557B58mQcJSWeLk9ERERE5LjcFuqstWOttY2ttb7W2mbW2nestWnW2guste1cn9Pddf8znTGGZs1+W4B869ZMgBq31IFzspQdO7LIzS2usn3evD3k55fy5Zdbj3uN8q6X48Z1IiuriMzMwhrfvz61HDqU8StXMuyNN8jdt4/pV1/N/xIS+OXRR9k1axbFubnHv4iIiIiIiAecshOlSO01bx5apaXO19eLFi3Canx+ly4xAKxfn1Zl+/z5ewGYMycZh6PsmNeYPHkLXbrEcPHFCcCp1wWzMi8fH7rfcQe3btvGmBkziO3enSXPP8/kYcN4NSKCz84/n/TNmz1dpoiIiIhIFQp1pzFnS52zhWnr1gxatQrHx6fm3/JeveIBWLRoX5Xtv/yyB19fLzIyClmx4uBRzy/vennVVe1JSAgHOKIL5l/+Mp+vv95W45rqg5e3N20uvZQrZs7k9+npXPHdd/R7+GEOrV3LB717s+HDDwEozMhg9Vtv8cMdd7D+gw/IO3j090JERERExF3ctvi4eF7z5qEVC5Bv25Z5Ql0vAVq0CKNTp2imT9/Offf1BiA7u4jVq1O5/fZuvPHGambN2k3fvo2rPb+86+VVVyUSGxsIQFLSbzNgFhWV8txzvzJwYFNGjmx7cg/pZv7h4bS66CJaXXQR3e+8k2+uu46Z48ax4pVXSF29GkdxMT5BQax56y0AYrp0IbxVK0KaNCEyMZEed92Fj7+/h59CRERERE5naqk7jTVrFoq1zhYz5xp1NZ8kpdxll7Xh55/3kJXlnO5/0aJ9lJVZrriiPd26xTJr1q6jnlve9bJjx2iiowMJCvKp0v1yy5YMHA7LokX7yM+vOinJI4/8zLvvrj3het0ptFkzrp4zh/6PP05xbi497r6bccuX84ecHMatWME5zzxDaIsWZO/axZYpU/jpgQf4cvRoSvLzPV26iIiIiJzGFOpOY+Vr1S1bdpC8vJITbqkDZ6grLS3j++93AvDLL3vx9jb079+YYcNaMn/+3iMCGThnyJw/fw9XXtkecE7ckpAQXqX7ZflyCSUlZSxc+FsXz/T0Av75z6W8/fapFerAOe7unKee4uYNGzj/xReJ79UL4+VFfM+e9H/0Ua745hsmrF7N3ampXPT22yR9/z1TL72U4pyc419cREREROQkKNSdxpo1CwFg7tzdwIktZ1Cuf//GxMQEMn36DsA5nq5nzzhCQvwYOrQlxcWOiolTKvv2251Y6wyF5RISwqp0v1y37hDe3gYfHy/mzNldsf2bb3bgcFjWrk2lrOzoa8Y99tgvPPLIzyf8TPWl6y23cOmHH7Lnl1/4fOhQdnz7LY7i4uOfKCIiIiJyAjSm7jTWvLlzpsu5c5MBaNv2xEOdt7cXl1zSihkzdpCfX8Kvv+7nrrt6ADBoUFP8/LyZNWsXF16YUOW8mTN30qhRMD16xFVsa9kyjEWL9le8Xr8+jfbtI4mMDKgS6r7+ejsAubklJCVl0bp1xBF1lZVZ3nxzDV5e8OyzgzDGnPCz1YeO112HT2Ag3910E1MvucQ5Rm/4cCLatiW4cWNCmzWj5dCh+AYFebpUEREREWmg1FJ3GgsL8yMkxJcNG9Lw8fGiZcuaL2dQ2YgRbUhPL+TVV1dQVORg0KBmAAQH+zFgQBN+/LHquLqSEgfff5/EJZe0wsvrt7CVkBBORkYh2dnO8Xnr1h2ic+cYhgxpwbJlB8jOLqKoqJTvvttJz57OMLhmzZGLnwNs3pxOWloBqakF7NiRVe0xp4p2Y8bwu4MHGTNjBm3HjGHPzz+z+JlnmH333Xw5ahRvNGnC7Hvv5dD69Z4uVUREREQaIIW605gxpmJc3YkuZ1DZRRcl4OvrxXPPLQFg4MAmFfuGDm3JqlUppKb+NhnIwoX7yMoq4tJLW1e5TkKCM1Tu2pVNfn4JO3Zk0qWLM9Q5HJZfftnDnDm7yc0t4dFH+2HM0UNd5S6fixfvq/aYU4mPvz9tLr2U4e+9x5179/JAcTG/27+fq2fPpvWll7LmzTd5v0sXJvboweJnniFt40YKMzMpyc+nrLTU0+WLiIiIyClMoe4016yZM9SdzCQp5cLC/DnvvOZkZhaRmBhFXFxwxb5hw1oCzjF05b75Zge+vl4MHdqyynVatnSuVZeUlM3GjWlY61zg/Oyzm+Dv783cucl89dV2goN9GTGiDW3bRh4j1O0hJiaQkBBfFi/eX+0xpzIvHx+CGzWixZAhXPrRR9yxZw/nv/QSvkFBzP/LX3ivUydei4zkP8HBvOTvz5djxrB3wQKsPfoYQxERERE5M2lM3WmuvKXuZCZJqeyyy9owa9YuBg1qWmV7797xdOoUzeOPz+fyy9sREuLHzJk7GDSoGWFhVddn+62lLov09AIAOneOJiDAhwEDmjBr1i5SUwu4+OIEAgJ86NYthjVrDlVbz/z5exk0qBmZmYVHLI7eEAXFxtL7vvvofd99ZCcns+uHHyjOycFRXEzegQOsnziRbV9+SeN+/Ui8+mpaDh1KTJcuGC/9XUZERETkTKffCE9zddFSBzB6dFsCAny45JKqXSq9vb14660L2b07h7/+dQG7dmWxfn3aEV0vAeLigggIcK5Vt359Gn5+3hWTtwwZ0oLVq1PZty+3YiHybt1i2bYtg7y8qjNG7tuXy44dWZxzTlPOPrsJq1enVrusQkMV1rw5XW+5hd733cdZDz3E+S++yO27d3PBa69RnJPDTw8+yMTu3Xm9cWN++tOfyE5O9nTJIiIiIuJBaqk7zZW31NU21LVoEUZKyl2EhPgesW/gwKbceWd3/vOfFRWLlFcX6owxtGzpXNYgP7+Ujh2jKsb5nX9+C2AB3t6m4txu3WKx1jlL5llnNa64zoIFzvF055zTlIMH8yktLWP58oMVE7icjvyCg+l59930vPtucvbsYdfs2eyYMYPlL73Eipdfpv2VVxLStCn5qakUZWTQuF8/Ok+YQGiz0/c9EREREREnhbrT3ODBzRk0qBn9+jU+/sHHERrqd9R9zz13Ll99tY13311H69bhtG9ffXdP51p12aSm5nPOOb8Fjr59GxEc7EufPvFERwcCzlAHzslSKoe6+fP3EhjoQ8+ecWRmOkPk4sX7TutQV1los2Z0mTCBLhMmkLVrFyv+8x/Wvv02ZaWlBMbG4hcSwvbp01nw17/S8sILadSnDwFRUfhHRJCfkkLGli1kbt9Oo759OeuhhwiKjfX0I4mIiIhILSjUnebato3k55+vdft9wsP9ee21C7jiiq+59NLWR103LiEhnAUL9pKbW0KXLjEV2/38vPnkkxEVLYvlx4aE+B4xWcr8+Xvp378xvr7exMYG0aZNRIOcLKUuhLdsyfkvvsjgf/+7ynueuWMH695/n40ffsiuH37AlpVV7Atu1IjQ5s1Z/uKLrH7jDXrfdx8dxo4lrEUL/EJCPPEYIiIiIlILCnVSZy6/vD2ffjqCc889eotZy5Zh5OY6x79VDnXgnIylMi8vQ9eusVVCXU5OMatWpfDYY/0qtvXv35g5c3ZjrT1lFyF3t8OfO6J1a8556inOeeopbFkZRdnZFGZkEBgVhX+4cxbStE2bWPjEEyx++mkWP/00AAFRUUS0aUNUYiKRiYk4Cgs5tH496Rs30mTgQIb+3//h4+9/xP1FRERExHMU6qROXXNNh2PuL58BE5wzXx5Pt26xfP755orAtnjxPsrKbJWulmef3YSPPtpIcnIOLVqc3ALrpzPj5UVARAQBERFVtkd36MBln33GgL/9jZSVK8nevZvsXbvI3LaN5Hnz2PDhhxhvbyLbtSMsIYF1775LTnIyo6ZOxS8khOLcXFa/+Sa+QUF0v/POMzZQi4iIiHiaQp3Uq4QEZytRUJBPxdfH0q1bDG++uZq9e3Np1iyU+fP34uVl6N//twXQ+/d3jrdbtGjfEaFuxYqDHDiQd8SsnfKb6A4diO5wZBgvzs3Fy9e3omVu3fvv8/2tt/L5BRfQbvRolv373xSkpQGw89tvufj99wmMisKWlbF3wQKMtzdNBwyo12cRERERORNpSQOpVy1bOkNX584xeHkdv2Wn8mQpq1en8MEHG+jePbbKpC3dusUSGOhT7bi6P/xhDtdcM53CwtIq2199dQUXXjgZh6PsiHPEyS8kpEpXyy433sioKVNIXb2aXx59lPi+fblu0SKGvPIKO7/7jkk9ezL7nnt4s3lzPj33XD4ZOJDJF17IwZUrPfgUIiIiIqc/tdRJvWrUKJiAAJ8jxtMdTdeuzlD35JMLWbkyhejoAN5996Iqx/j6etOnTyN++WVPle1ZWUUsWrQPh8Mye/YuLr3UOWbPWstLLy1n584spkzZwtVXH7vLqPym7ahR3LBkCY7iYhr16QNAk/79ady/PzOuuYY1b71Fq+HD6XDtteQdOMDip5/mg169SLjwQqK7dCGyXTuCGzWirKSEspISAqKjaX7eefgEBFTco8zhoDg7G/+ICHXpFBEREakBhTqpV15ehsmTL6Njx+OPpwPnrJotW4axdOkBxo7twKuvXlCx5EFlw4e34tFHf2HPnpyKBdfnzt2Nw2EBmDp1a0WoW7r0ADt3ZuHj48Xf/76YK69MrFGroTjFdut2xLbGfftyy5YtlBYV4RccXLG9y003seSFF9g+fTp7fvmF0oKCI871DQmh1fDhRCUmsv/XX9m/eDHFOTn4hoQQnpBAeKtWhCUkEJ6QQGBsLCV5ec79wcF0uekmfAOP/HkQEREROZMYa62naziuPn362GXLlnm6DPGQOXN2U1hYesxxcZs2pdGx43u8+uoQfv/7XgD87nc/8uGHGxg+vBVz5iRz4MDv8PHx4sEH5/Laa6t48cXB/P73s5k6dRRjxrSrr8c5Y9myMnL27qXg0CG8fH3x8vEha+dOtn/1Fdu++oq8gweJ7daNJgMGENG6NTl79pCdlETWzp1k7dxJcU7OEdeMbNeOC99+m+bnnou1lvRNm0hds4bwhAQiExOPmBxGREREpKEyxiy31vapdp9CnZwuOnZ8lyZNQpg9+2qstbRp8zZdusRw442dueKKr5kz52rOO685LVq8Se/e8UyZMopOnd4jONiXFSvGqaufB9myMkoLC/ENCqp+v7UUZWaSn5qKX2gofqGh7F+8mB/uuIOsHTtIuPhi0tavJyc5ucp5QXFxRHXoQFRiIlEdOhCZmEh0hw6EJSTg5e1dH48mIiIiUieOFerU/VJOG2PGtOOFF5aQllZARkYhO3dm8eCDfbjoogQCA32YOnUrvr5e7N2bywsvnIePjxePPdaPG2/8jhkzdhyxTp7UH+PlddRAB851+AIiIwmIjKzY1nLoUCasWcPCJ55g8+ef0+iss+j/l7/QqG9fcpKTSd+8mfRNm8jYvJmt06ZRcOhQxbnefn5EtG1LVIcOhLdujaOoiKLMTBzFxSRefTXtRo/GeGkeKREREWkY1FInp42lS/dz1lkf8f77F5ObW8Lvfz+brVtvoW3bSC6//CuWLNnPqFFteffddaSk3EVoqB8lJQ4SE9+lqMhBv36NiY4OoGnTEPr0aUS/fo2JjAxg7dpUFi/eT3Gxg3vv7aUWvQaqIC3NGfQqhb30zZvJ2rkTn8BA/MPDcRQVkXfgAFGJifR96CHajBxJUIxzUp+SggK2TZvGli++wFrrDJlRUbQZOZJmgwbp50JERETcSt0v5YxgraVFi7fo3TuesjLL+vWH2L79NgA+/HAD48bNxM/Pm5Ej2zB58siK8+bO3c3f/raQQ4cKSE8v5ODBfMrKnP9d+Pl5U1zsqDh2zpyrOf/8FvX7YFJvyhwOtkyZwpLnniPFtRRDZPv2xHTpwu45cyjKzCS0eXP8w8Mpysyk4NAhSgsLaXTWWfT9059oOmAAAdHR+Pj7U5KfT+aOHeTs3k18nz4Ex8V5+OlERESkIVOokzPGPffM5u231+LjY7jhhk68/vowADIyComL+z9KS8uYPPkyrrwy8ajXyMsrZvnygyxZcoADB/Lo06cRPXvGce65n9K7dzwzZ15RX48jHmKtZf/ixST//DP7FiwgZfVqmp1zDl1vuYXmgwdXdM0syc9n3fvvs/zFF8ncvr3ifN/gYEry8n57HRJCv0ceoff99+MTEEDqmjXs/PZbsNY5s2erVsR06YJfSEjFOTl797Jh0iQCoqLofOONVdYMrKw4NxfrcOAfHu6md0NEREROBQp1csaYM2c3F1zwOQDTpo1i9OjfZrW86KIvWLBgLykpdxEU5HvC13766UU8/vgC1q6dQJcusXVWszR8ZQ4Hu378kaykJAoOHaIwPZ3A6Ggi2rQhKC6OFa++yrYvvySkaVOMl9cRE7oAePn40OTss2k+ZAiH1q1j25dfYh3OVuLQZs3o9+ijJFx8McVZWRRmZHBg6VJ2fvste+fPp8zhIL5XL5qffz4tzj+fZoMG4RcaWt9vg4iIiLiRQp2cMUpLy4iP/z+ys4tJS7ubsLDfWjd27Mhk//48Bg5selLXTksroEWLN7nqqkTef394jc/bvDmdP//5Z95556Jq19iTM0PyvHksfvppfENCaHPZZbS+9FJ8Q0LI3rWLrO3b2btwIbt+/JGDK1YQGBVFl1tuofsdd5C5fTsLn3iCfYsWHXHN2G7daDV8ON4BASTPncv+xYtxFBdjvL1p1KcPsd264e3vj5efH2HNm9N2zBjCW7YEnDOOpq5dS0FqKo379asSAnP27KEkL4+oxKO3aIuIiEj9UqiTM8oTTyxgz54c3nnn4jq/9r33zuaNN1azc+dtNG1as5aQSy+dwsyZO3nxxcHcf/9v/x2WlVl++CGJYcNa4u2tmRbFqSgrC++AgCrdLa21JP/0E1k7dxIQGYl/RASR7dsT2rTqHyhK8vPZt2gRu+fMIXnuXDJ37KCspARHUVFFd9BGZ51FeKtWJM+dS35KCgDG25u4nj0JbdqUA0uXkrtvHwAtLriAfg8/TIsLLjhiIhhrLaUFBcectVRERETqjkKdSB3ZuTOTtm3f4cEH+/DCC+cd9/i5c3czZMjn+Pp60alTNKtWTajYN3HiOm688TvefHMYt9/e3Z1li5CxbRtbpkxhy+TJ5B04QIvzz6fF0KGENG7Mnvnz2TNvHnkHDtCoTx8a9+tHaWEhy19+mbz9+4np0oXm559Pk/79CYyNZefMmWz76iuydu4ktEUL4nv2JLZ7d+dagO3bE9m2LX5hYRVBsLSoiPSNG8nYuhVHURG2rAyMIaxFCyLatiWkcWMtISEiInIcCnUidejaa6czY8YO/va3AdxxR3dCQvyqPa6szNKv34ccPJjPvff24k9/mseqVePp3j0Oay09ekxizZpUWrcOZ/PmW/Dx0S+1cmopLSpi/cSJbPz4Yw4sXUppfj4A3v7+tLjgAhqfdRbpmzeTsnIl6Zs3Q6V/T7z9/QmKjcUnMJDMHTsqxgdWxycggMC4OIJiYgiMiXF2GfX1xcvXl4DISAJjYgiKjSWmSxfievUiICICcHYhLcrKwtvfH5/AQC0rISIipzWFOpE6tGtXFrfc8j2zZ+8mMjKA66/vSHGxg4MH83E4yrjrrh5cfHErPv98M9deO4P337+YSy9tTZMmb3DPPT3597/PZ/bsXQwdOpnLL2/H1Klb+eijS7nuuo6efjSRoyorLeXQunXk7t/vnIil0kydAKWFhWRu307Gli1kbt9OfkoK+amplOTmEpWYSEzXrkR37OgMX97elJWWkr1rF5nbtpG5YwcFqankp6ZSmJaGo7jY2W20uJjCjAwK09KcrXsuYQkJlBUXk5+SQllpKeDsQuofHk54QoKzxTAxEf+wMGcLoDGUFhRQnJ1NUXb2b+eUh0DX58i2bWkxZAgxXbqo5VBERE45CnUibvDrr/v5xz9+ZebMHURGBhAfH0RGRhF79uTQr19jDhzIIzzcnxUrxuHt7cWYMV+yaNE+9uy5k1GjprF8+UF27ryNvn0/xBjD6tUT8PJy/nJZXOxg7tzdTJ26le+/TyIrq4jiYgdlZTByZBseeqgvvXs3OqImh6OMP/5xHmPHduCssxoftfbiYgc+Pl4V9xM5ldmyMvIPHSJ19WoOLltG6po1+AQGEhQfT1BsLGWlpRRnZ1OYkUHmjh2kb9pEdlLSEdcxXl74hYXh5ev7W6ui63OZw0FRZiYAgTExNB04kMb9+jk/zj4b30BNciQiIp51rFDnU9/FiJwu+vVrzJdfjsZaW/EX/+JiBxMnrufvf19EcnIO3313RcUkKBMmdObLL7fxyisrmDlzJ089NZDAQF8eeaQfN9wwkxkztjNiRBv+9781PPbYfNLSCggO9uXiixNo2jQUPz8v8vJK+OijjXz++WYuuKAF//vfhbRqFVFR0/vvr+fll5ezalUKc+deU23dCxbsZcyYL4mICOD++3szYULnk1riQaS+GC8vguPiCB42jIRhw2p0TmlhIaWFhdiyMmxZGT6BgfgGBR2zi2b27t3snjuX5Llz2bdwIdu++gqAoLg4+v7pT3T/3e/wCw6uk2cSERGpS2qpE3GDoqJStm7NqLKeXXGxgyZN3iAjoxA/P292776d2NggSkvLaN/+HUJD/QgN9WPBgr0MHtycBx7ow7BhLQkIqPq3l6ysIt58czXPPvsr7dpFsHDhdfj6epObW0z79u+Qnl5IUZGjYvxeZR9+uIFbbvmeFi1CiYwMYOnSA0RFBXDffb154IHeBAdXPz5Q5ExUkJbGvoULWfHKK+yaNYvA2Fi63nwzLYcOpcmAAZr5U0RE6tWxWuo0aEDEDfz9fY5YoNzPz5uxYztQVmYZP74TsbHOXwh9fLz485/PYs2aVDZuTOO99y5mzpyrueyyNkcEOoDwcH8eeugs3n33IpYtO8hTTznXL/v3v5exf38eU6eOIijIh//8Z0WV8/72t4WMGzeTAQOa8Ouv1/Prr9czf/5YzjmnKX/96wLatn2HN95YRUnJ0Se0EDmTBEZH0+ayy7jqxx8Zu2AB8b17s+zf/2bysGG8FhnJF8OHs3nyZEqLijxdqoiInOHUUidSjzZsOMTVV09n2rTRtGsXWbG9tLSMjz7awPDhrYiLq3n3rptu+pZJkzbw6acjuPHGb7n00tZ8/vlI7rrrR959dx27d99OXFww//vfGm6//QcmTOjMW29diJ+fd5XrLFq0j4cemsf8+Xv5/e978uqrF9TZM4ucTopzc9k7fz67Zs9m82efkZOcTGBMDC2HDSOkSROCGzWq+Ahq1IjwVq3UZVNEROqEJkoROU1lZxfRo8ckdu7MwtfXi02bbqZ16wg2bUqjY8f3eOqpgQwd2pLzzvuU889vwcyZlx91oXNrLVdfPZ2ffkrmwIHfVXtcYWEpDzwwlxYtwnj44X41qvFf/1rK0qUHeP/9iwkM/G3s3quvruDXX/fz7rsXHxEyy+tJScmnrMzSuHHIEftFPK3M4WDXjz+y9t13ObhsGXn791NaWFjlGG9/f1pfcgmJ11xD6xEjFPBEROSkKdSJnMYWLNjLeed9ygMPVF0QffjwL1i5MgUvL0NQkC9LllxPVNSxZ/D77LNNXHvtDObPH8vAgU2r7EtPL2D06K/45Zc9+Ph4sX37rbRoEXbM661Zk0qvXpNwOCwjRrRm6tRR+Pp689//ruT3v58NwH339eall86vOGfatK08//wSNm1KJyurCC8vw5/+1JcnnxxQbXdUkVOFtZbinBzyDhxwfuzfz94FCyoWfPePiKD3/ffT+w9/wD883NPliohIA6NQJ3Ka27cvl0aNgqssUfD99zu5+OIpBAf7snjxdUeM8atOVlYRsbH/5b77elcJiElJWQwfPoUdO7J47rlB/PnPP3P77d147bWhR71WWZll0KBP2LIlgwcf7MMjj/zCDTd0YujQFtx443eMHNmG5s1D+e9/VzF58mVceWUi77yzlttu+56OHaM577xmJCZGsXp1Ku+9t44OHaJ4992LOfvsJrV7s0TqWZnDwZ6ff2b5yy+z/euv8Q8Pp/OECUR17EhEmzZEJSYS2ry5Fk8XEZFjUqgTOQNZa3nwwZ+46KIELrqoVY3Pu/DCyezencOmTTcDzrXvunZ9n/378/jqq9Gce25zbrvtez74YANJSbfTqJGzO9mLLy5j9+5sHn/8bKKjA3nnnbXceuv3vPfexdx4YxeefXYxjz02H4ALLmjBjBmX4+VlOPfcT9mwIY3f/a47L7ywlIsvTmDKlFFVlln44Yckbrvte5KTc7jvvt48/fQ5WoZBGqSDK1ey6Kmn2PnttzgqTbASFB9P47POotFZZzk/9+1LQKRz3G35v9MKfSIiZzaFOhGpsfKukZs23UxiYhQffriBceNm8sUXI7niivYAbNuWQWLiuzzwQG/++c/BPP/8rzz88C8AREUF8PjjZ/P3vy+ic+cY5s27BmMM1lqeemoRK1em8OGHlxAS4lw+YffubHr1+oC0tAKuvLI9H310abVj7HJyivnzn+fx+uuradMmgnffvYhzz21ef2+MSB2yZWXk7N1L1o4dHFq3jv1LlnBgyRLSN22qOCYoLo7SwkJK8vLw9vOjcf/+NDv3XJoPHkyzc87By0fdkUVEziQKdSJSY7t3Z9Oy5Vu88MK53H9/Hzp2fJeQED+WLx9XpXvn9dd/w1dfbeORR/rxl7/MZ+zYDjz8cD/uvXc28+Y5x92tXDmuRt0+Fy/ex7x5yTz4YF98fI690spPP+3mllu+Z9eubH799Xp6925U62euD8nJ2cycuZMLLmhB27aRxz9BzkhFWVkcWLaMA0uWkLljB75BQfgGB1Ocm8u+BQtIWbUKW1ZGQFQUbUaOJOHCC/ENDsYYg5evL4GxsQTFxREcH4+3n9adFBE5nSjUicgJ6dlzEsHBvtx6a1duuuk7vvpqNCNHtq1yzPr1h+jS5X0ARo1qy+TJl+Hr6421lilTtmCMqWjZq2sZGYV06vQeTZuG8Ouv11fM1OlwlLFxYxpt20a6ZVKVxYv3MXnyZoYMacEll7SuUXe4t99ew1tvrWHp0gMA9OwZx9KlN1SZXXTt2lTi44NOaDkLOTMVZWWxa/Zstk2bxvbp0ynKyqr2OOPtTUyXLjTu14/G/frRYsgQwhMS6rdYERGpUwp1InJCnnxyAU89tYimTUOJjw9i6dIbqg0w99wzm5SUfCZOHF7vM1OWz9T5n/8M4d57e1FUVMq1187gyy+3ERDgwznnNGXkyDbcdVePKgFqw4ZDXH/9TIYMac4tt3SlU6cYiosdLF16gLVrUxk3rhPBwb+1cJSH1H/9axm//rofY8BaGDiwKc8+e84xu4DOmrWLYcMm0717LNde24GAAB/uv38ub7wxjDvu6A7AihUHOfvsj2nbNoJly26osuzDqSAjo5DMzEJatYrwdClyGEdxMWkbN2IdDmxZGY7iYgpSU8k7eJDsXbs4uGwZ+5csoSgzE4CItm1pecEFxPboQUznzkS2a0dRdjZ5Bw5QlJFBcKNGhCUkEBQXp/F7IiKnIIU6ETkhK1YcpHfvDwCYMWMMl17axsMVHclayyWXTGH+/L0sXz6Oe++dw/ffJ/Hoo/3Izy/lxx+TWL8+jbffvohbbulacd6YMV/y3XdJlJaWUVpaRqdO0SQlZZGfXwrAbbd14623Lqw4/oMP1jN+/Le0axfJvff25LrrOvL555t56qlF7N+fx5gx7XjppcG0bFl1ivriYgfdu0+kuNjB+vU3ERDgg7WW88//jHXr0tiy5Wa8vb3o1WsS2dnFHDpUwF139eC//z36jKJ1oaCghKSkbJo0CSEszK/aX9537Mhk0qT1fP99EkuWHMDb27Bs2Ti6dTt+V1o5tVhrSdu4kd2zZrFr1iyS582jODv7mOd4+/vjFxaGb3AwfiEhBMXHE9K0KcGNGuEoLKQwPZ3CzEx8g4MJjIkhKDaWwJgYAmNiCIiKojg7m9z9+8k/eBCMwTc4GN/gYEKbNiWibVsi2rTBNyioyj1LCgrI2LIF//Bwwlq2VKgUEamGQp2InBBrLa1b/4/4+GAWLbrulP0Fa+fOTDp3fh9roaiolP/977cAZ61l4MBPSErKYuvWWwgO9mP58gP06fMhTz01kDvu6MYHH2zg22930rlzDIMHN2fevGT+858VzJx5OcOHtyYpKYtu3SbSvXssP/10TZUWv/z8El5+eTnPPLMYa+Gxx/rzxz/2wd/f2WL5r38t5U9/msf06WMYMeK3ULxmTSo9e07izju7c+BAHl9/vZ15865h2rSt/Otfy6rt6lqXrrjiK6ZO3QpAcLAvHTpEMWZMO666qj0+Pl4888xiJk5cj7XQt28jLrywJW+8sZqWLcNYtOj6ijGPq1al8P776/jd73qQmBjltnpPxJIl+wkN9aNDh6hT9mfW06y15CQnc2jdOjJ37CAgIoKg+HgCIiPJO3CArKQkcnbvpjgnh5K8vIp193L37iXvwAF8g4IIiIrCPyKCktxcCg4dojAjo9p7GS8v58yd1fye4R8R4QyC0dHkp6aStXNnxXFB8fE07tePsJYt8fbzw9vPj9KiIooyMpyBMj2dQtfXZaWleAcE4BsYSGjz5rQYMoQWF1xATJcuGC8vMIb8lBRSV68mdfVqCjMzCWvenNDmzQmKj8cnIABvf3+8fHwocziwDgfGywu/0FD8QkPx8vOjJDeX4pwcykpLCYqLIyAyssrPl7WWQ+vXs3vWLNI2biS2e3eanH02sV27Yry9cRQV4SgqorSwsNrPjsJCSouKKCspISAysuJ9MT4+v93HGOfXrtfGGIyXF94BAXj7nlqt++BsRS4tLMQvNFT/LYrUIYU6ETlhSUlZBAf7EhsbdPyDPeill5bx0EM/M2nScMaO7Vhl34IFeznnnE/4+98H8pe/nM1ll01lwYJ9JCXdRliY/xHXKioqpXfvD0hPL2TNmglcfvnXrFqVwpo1E0hIqH6x6N27s3nggblMmbKVtm0jePXVC+jWLZbExHc477zmzJhx+RHn3HPPbF57bSUA//rXeTz4YF+Kix307/8Ru3fnsGbNBJo0Cak4fs2aVG688VsaNQrmiy9GnvRyDvPn72HQoE+5+eYudOoUzZ49ufz6634WLdoHOH9f9Pf34c47u/OnP/WtqKG8q2t5ratXpzBkyOekpxfi7W248cYu/PGPfXA4LAcO5AEwZEiL4/4yZ63lhx+S+PzzzbRtG8nAgU3o27fRMbugOhxlzJ2bzDnnNK3S5XfNmlR69JiItRAXF8Tgwc25//7e9O+vdQ3dray0lIL0dGfAS0vDLzSU4MaNCYyJwXh5UVpQQHFODjnJyWRu307mtm3kHThAQVoaBYcOERAZSXSnTkR16EBBWhr7f/2VA7/+Sn5KCqWuQOQTEEBAZKQzUEZGEuj67O3rS2lhIaUFBaRt3MihtWuPWau3nx+O4uJaPa+Xry+BMTF4+/nh5eNDUXY2BampAPiFhVW0hBovL2xZWa3uVdN6yifU8Q0OxicoCEdhIUXZ2RTn5BDSpAmx3boR07UrfiEhlBYUON8z1/tWWlBAYXo6+QcPkp+SgpePj3PCndhYjI8Ppfn5lBYUUOL6XFpQgF9ICOGtWztbXYODKcrKoigri9y9e0nfvJnspCRsWRnefn4ExsbiHx6O8fLCeHnhExRESJMmBDdujJePDxlbt5KxZQvF2dlEtGlDRLt2hLVsSUBEBP4REXj5+pKfklJRX97Bg+QfPEhZSQlNBw6kxQUXENu9O5nbt5O2fj05ycmENGtGeKtWhDVvjre/P8bbu6KW8qVCsnftYv+vv5K2cSOR7doR37s3Ue3bY7y8cBQXU5KXV3U5kUr/P3MUFVGSn4+jsJCA6GiCXD/rAMV5eeTt20dJfj5lpaWUlZT89lFa6qzF3x9vf3/CXH9cqE3wdZSUUJiRQUlenrNmhwMvX19n2DeGspISHMXF+AYFEZaQcEr+EaA+leTn4xMQUPH9amgU6kTktJabW1yxRMLhLr/8K378MYlPPhnBZZdN49lnB/HII/2Oeq0VKw7Sr99HNGkSzO7dOUyaNJxx4zoft4Yffkjinntms2VLBo0bB5OWVsiGDTfRpk3EEcdmZBTSpcv79O/fmC++GFnxD/rmzen06jWJwEBfbrmlC3fc0Z1p07by6KPzCQvzIy2tgMGDmzN9+hiCg/0oKirlnXfW8vPPe9i3L4/9+3PJzS3Bx8cLHx/DwIFNeeedi/D3d3b9HDDgY3bvzmHr1luqBMM9e3KYOnUraWkF3Hlndxo3DqlSr7WW0aO/5Mcfd/HZZ5dx883fERDgw+TJl/Hpp5t4/fXVFBc7qpwzYUJn3nrrwmqXp3A4ypgyZQvPPbeElStTCA31IyfH+Yt2YKAPU6aMZPjw1kect2DBXu69dw4rVhzkjju688Ybwyr2XXHFV8yatYvnnz+XBQv28f33O0lPL+Tpp8/hoYfOqpi5tbjYwZw5u/niiy3MnLmDs85qzJtvDiM+3jlJTUmJg08+2US7dpFa6L4Byjt4kN1z5pC9a1dFy59feDix3boR260bfqGhFBw6RPbu3RSkpjpbzIqKsK5ftsuDWHFODsU5OTiKi39rtfP2doaKAwcoOHSo4hd2b39/mg4aRMsLLiC0eXOyd+1i36JFpK1fj/H2rmgN9A4IwMf12dvfv2K7T0AA3gEBzoCYmUl+aiqFaWmUORzOZ3A9R3mrZ8XnsrKKJS9K8/Mrfqkv/6XVLywMv5AQsnfvJnXNGjK3b/+t1dQYfAID8Q0MxNvfn4CoKILi4giKi6PM4aAgNZX81FRsWRm+gYH4BAU5j3d9LsrKInP7drJ27qSspASfwED8w8MJio8nKjGRyMRE/MPCyE9NpSA1laLs7Iqai3NyyNu/n9x9+ygrKSGyXTsi27fHLzSUzB07yNy6lZy9e49o4fX28yMoPp6g+HiC4+OxZWXsnT+f4pycKsf5BAVRmp9/1J+R8hBc3SRDPgEBFeNTT4SXry/BjRpRnJNTMYa1pgJjYojp2pWwFi0IiI4mMCqK0qIi8g8eJO/AAUrz84/83lvr7Oa8dy/5qanVtoZXx3h7E56QQEiTJs6Q6gp9RZmZFGVm4igpITA6mqDYWPzCw/Hy9na2ErsCeeWP6rZ7+fg4A6W/f8UfYXxDQpx/NEhJIT8lxfmzlZJCYXo6xsen4ucqLCGBiDZtCG3enPwDB8jYtq3iv2MvX9+KoOpV6cPb1xfj4/PbHyVSUwmIjCQ8IYHQFi2cf3A6dIiC1FRy9uwhZ/duCtLS8AkKIrZrV2K7d8cvNNT5B4v8fHyCggiOjye4USNiunSh6cCBJ/S9rA8KdSJyxtq8OZ3Ond/Dy8sQHu7Pjh23ERp67Knen3pqIU88sZCrr07k009H1PivqEVFpbz00nL+/vdFrqUezj7qsfn5JQQG+hxx7SVL9vP880v46qttOBzO/z+PHt2Wt966kB9+SGL8+G8ZNKgp48Z15u9/X8SuXdkkJITRokUYTZqEEBrqh8NRRm5uCZ9/vpnRo9vy+eeX8fXX27nyyq95552LuPnmrtWVdEx79+bQqdN7ZGcX07hxMPPmXUu7ds6/eO/alcU33+wgKiqQxo2DmTNnN089tYihQ1vyxRcjCQ/3r3h/Jk3awAsvLGHbtkzat4/kz38+i+uv70hubgmLFu3jscfms2dPDqtWjad58zDAuUbh3XfP4oMPNtC0aQhdu8a4xvvdQJ8+jVi1KoWePSfx17+ezd/+5vxHODOzkNtv/4HJk7dw4YUJ9O4dz6JF+1i69AB5eSWEhvoxZEgLvv8+idBQX95552JKS8t4+OGf2bIlA39/b778cjQXX9zqhN+r6lhrKSkpqzbkitSH8pYjn4AAvHx966RbZHmX1bpePqPM4XAGpKwsyoqLCYqLwy8s7IiaHSUlHFy2jLSNG4lo25aYzp0JjI6mKCuLrJ07ydmzxxm+S0txFBVVTCRUlJVFTOfONO7fn+hOncjcto2Dy5dzaN06vHx98QsJwScoCC9v7ypdiMt/Z/b298cnMBAff38K0tLI3buX3P378QsNJbRpU4KbNHH+IaA85JQHEVc33/Jut5k7dnBo7VoOrVtH7r59FKSlOQOpMQTFxhIUH49fSEiV7rfln31DQght2pSQpk0JiI6uaKn18vGpaBm01lbcuzgnh8xt28jYupX8lJSK5/Ly8cE/IoKAyEi8fHwqWtCLsrKwZWUVH+Wh/KjbHI6KP3SUtwJXZry9nc/k+uNBQFQUZaWlFS352UlJ5OzZU3F8+cRN5c/jKC6ueC7HYa2f/hERBMfHExgbS2F6OllJSeTu2VPR6hwYE0NI06aEtWxJaLNm5B88SMrq1Rxas4bSoiLnHyuCgijNy6MgLQ2Ajtddx6UffVSnP9d1QaFORM5od931I6+/vpoXXjiXP/3prOMeX1Li4LPPNjNyZJtqu2nW5Hxf39r98r5nTw6TJq0nISGcsWM7VPwy88knG7nhhpmUlVl6947n2WcHMWxY9RNLvPrqCu69dw7XXdeRJUv24+/vzerVE6qMDTwRH320geefX8Lnn19Ghw7Rxzx24sR13HrrD7RoEUpCQjh5eSXs3JlFSko+vXrF88gjZzFmTLsjatm6NYNevSbRrZtzHOOhQwVceulU1qxJ5c9/PotHHumHw2Hp0OFdmjcPZfHi67niiq+YOzeZpKTbiIgIqLiWtZb//W8Nf/jDXEpLy+jRI5b+/Ztw4YUJDBvWkoAAH9avP8QNN8xk1aoUADp0iOKvfz2bF15YyoYNaUydOpKLLmrFZ59t4qWXluPtbbjrrh5cc02HGs/4unt3Nldd9TWHDhWwdOkNREUFnuA7LyJnitLCQmcY9KnfGaXrWmlREYXp6ZTk5hIQHU1ARMRxuzyWFBSQt2/fb2G2Fspc42NP9I8XjpIS8lOc/x6ENm1aqxrcQaFORM5o6ekFvP76ah54oPcpt2TAyZg9exe5uSWMHNnmuP9gPfvsYh57bD4A33xzOZdccmS3Rnf58ccknnxyIcYYgoN9iY4O4KabujB06LFnN/zkk41cd903TJjQmZ9+SubQoQImT76sSpfMjz7awA03zOTuu3vw3/+u4sknB/DEEwOqvV52dhE+Pl5HHYtYVFTKiy8uJz4+iPHjO+Pj40V6egEXXvgFa9ak0rixsytup07OILthQxqxsYE899y5x231nDVrF9deO4PiYgeFhaVcdFECX301pqI7aHUOHcrn3nvnsGzZQebPv1brF4qICKBQJyJyRnv++V/ZvTuH1167oMHMRHfHHT/w1ltraNQomG++uZxeveKr7LfWMnjwZ/z88x4iIvxJSrq9optnXcnMLOTyy7+iuLiMhx7qy4gRbTAG5szZzRNPLGTJkv2sXDmezp1jjjh3z54cXnppGS+/vIKOHaOYOnUU33+fxL33zjlmi/G0aVu5884fycgoxBjD0KEtmDHj8gbzfRMREfdRqBMRkQaloKCEV15ZwTXXdDjqzKPr1qVy1lkf8eSTA3jooeN3q61Lqan5dOz4Hu3bRzJ//tiKlreNG9N4+unFfP75ZsrKLBMmdOaVV4YQEuKHtZarr57OtGlbmT59DM2bh5KRUcSOHZksXLiPBQv2sn59Gj17xvH++8OZNy+Ze++dwyuvDOGee3rV6/OJiMipR6FOREROS1lZRXXeQldT5QvTv/baBdx9d0+mTNnC+PEz8fb24rbbunLPPb2OCKRZWUX06jWJHTuqzrwXEeHP2Wc34aKLErjrrh74+jonaBgxYiqzZ+9m6dIb6No1loyMQqy1R4zLmz59O089tZDXXx9Gnz6N3PrcaWkFPPPMYm65pWu1rZQiIuIeCnUiIiJ1zFrLRRd9weLF+7nttq68+OJy+vdvzNSpo45YFqKyPXty+O67nYSF+REZGUCzZqEkJkZVO84uJSWPbt0mUlxchrWWzMwifH29+Oyzyxgzph0Aa9emcvbZH5OXV0JwsC9TpjgndzkZX365leefX8K//jWYgQOPnCRg06Y0RoyYxvbtmbRtG8GKFeOPO5tsfdu+PZPc3GK6d4/zdCm1VlLi4KuvtnHxxa2OumyLiJw5FOpERETcYMeOTLp0eZ+CglJuvLEzb7wxDH//up21bsGCvbz44jIaNQqmdetwvvhiC8uWHeTTT0cweHBz+vb9kMLCUmbMuJxbbvmedesO8dprFzBwYBN8fLxwOCy7dmWzY0cmaWmFjBnTjm7dYqvcIyOjkHvvncOHH27Ay8sQFxfEypXjadTot0lafvwxiauumo6/vzePPtqPBx74iWuv7cCHH15S4zF/1lpWrkyhVatwIiMDjn/CCdq/P5eePSeRn1/K5s03HzNcn+pSU/O5+urp/PRTMqNHt2Xq1FEaW+kBBw7kMX/+Hvr2bUTLltV3BT9TfP75JqZP38Ezz5xDixZhni7njKRQJyIi4iYzZmwnLa2A8eM718sv3dnZRQwfPoVff91Pp07RbN6cwc8/X0u/fo3Jzi7i8su/Yvbs3ce8xsCBTRk3rhNpaQWsWZPK3LnJpKcX8thj/bjssjYMGvQpZ53ViFmzrsYYePbZX/nb3xbSqVM006ePoWXLcJ55ZjF/+cv8atc+LCws5amnFgEweHBz+vVrzA8/JPGvfy1lyZIDdOkSw88/X1unwa60tIwLLvicZcsOUFpqufbaRCZOvOSkr7d06X4++mgjF16YwJAhLWq8hMXh8vNL+OmnZC66KKHGy4ksX36AMWO+IjW1gNGj2/Lpp5v4978H88AD1f4ud8o6cCCPTz/dhJeX4c47u5/wOo2FhaXs3ZtLmzYRta5l69YMvvtuJ999t5O1aw/x2WeXcfbZTao9trS0jFdfXcHnn29m8eL9FdsHD27O+PGduP76Tqf1mpPz5+9h2rStjB7djoEDm5KXV8I998xm4sT1AERFBfD++8O57LI2Hq70zKNQJyIichrJySnmkkumMH/+Xt599yJuuum3UFVc7GD27F3k5ZVQUlKGMYYWLUJp0yYCHx8vJk5cz+uvr2LbtkwAEhLC6NEjjscfP7tiltHy8YJ33NGdDRvS+OWXPVx/fUdef31YRXdLh6OMiy+ewoIFe/n++ysZNKgZ4JzkZvTor/jhhyR8fLwoLS2rqK1t2wiuuaYD//znUvr0ieeHH64kOLhuuhU+8sjPPPfcEj744BLWrz/Ec88tYcGCsQwY4OxGOmXKFpKSsrj//j7HXFICnC2w/fp9xKFDzgWUg4N9GTCgCaGhfgQE+NCoUVC1YyYPl5qaz4gRU1my5ADPPTeIP/+533Gf44cfkhg16ktiYwP58svR9OwZxxVXfM306duZN+8aBgxoirWW5OQcoqMDqn3/rLXs25fLypUprFhxkBUrUli//hAXX9yKF14496hLu5SVWZxrW5/8HydSUvL47rskPvtsE99/n4TD4fw9s2fPOCZNGk6XLrHHuYLT7t3ZjBr1JWvWpPLmm8O49dZu1R536FA+Cxfu47zzmlc7vnb37mz++MefmDx5CwDt2kWSl1eCt7dh5crxREcfuW7kX/4yn2eeWUyvXvGMGdOWwYObM2/eHiZOXM/WrRkMG9aSadNGVfveL1q0j7FjZzBwYFNeemlwg1uS5Ntvd3D55V9TWFgKQPPmoXh7G3bvzuGxx/oxdmxHrr/+G1auTOHee3vxxBNn12rtzcLCUr77bic9e8ad8S2hNaFQJyIicpopKChh3bpD9O3b+ITPLSuzbNqURrNmoYSFVT/RzJ13/sibb64mJMSX118fxg03dDrimAMH8jj77I/YtSubO+/szuOPn82ECd8ya9Yu3nnnIq66KpFFi/axcOE+unWLZeTINnh7ezFlyhauvno6F17Ykq++GlOl1WPlyoO88MJSMjML6dAhmg4dorjggha0bRt51Of58sutjBnzFbff3o0337yQ3NxiOnR4l/j4YBYsGMtDD83j1VdXAjBhQmfefvsifHyqbzXLyipiwICP2b8/j59/voY9e3L5+uttLFlygMLCUgoLHSQn52Ct5dZbu/LYY/1p2jT0iOts25bB8OFT2LMnlx49Ylm+/CCLF19/xPIclc2evYsRI6aRmBjJDz9cWREIMjML6d37AwoLHfTv35j58/eSkpKPMdC6dQSdOkXj5+dFQUEpeXklbNyYTkpKPgDGQPv2USQkhPH990l07hzNp5+OqBKu9u7N4dVXV/LGG6uJigrgD3/oxc03d63xeMlNm9KYPHkL06dvZ9myA1gLzZqFMm5cJ8aP78SmTencfvsPZGUV8+ij/bj99m7H7Bq7cOFexoz5isLCUrp2jWXBgr08//y5VWa5LSuz/O9/a3jkkV/IyCgkMNCHK69sz9VXJ+Lv701hoYNlyw7wz38uBeChh/oyblxn2rSJYPnyAwwY8AlDh7Zg+vTLq4T8b7/dwSWXTOWWW7ry9tsXVanLWst7763jttt+oF+/xnzzzeVVWps/+GA9t976A7GxgaSk5BMa6seLLw6uthV/+/ZMIiP9TygQ5eYWM3/+Xn76KZlmzUK4447u+Pp6V7wfn322idTUAq6/vmO1YbWykhIH//nPCpYtO8CIEW0YObINP/64i7FjZ9ClSwxTp45i4cJ9fPzxRvbvz+Pll8+v+MNNUVEpf/zjPF57bSUhIb7ceWd3HnigT5XvaUmJgylTtjJ37m6efvocYmODjqhh+fIDjB//LRs2pAHQv39jrrmmA9dd18EjYbiszHLwYB5JSdkEBfmckuNyFepERETkhBQVlfLqqysZM6bdMbu/5eQU89e/LuCVV1ZgjPMXo/feu5gJE7oc8/r/+98abr/9B5o1C+W885px9tlNmDVrF19+uY3wcH9atw5n8+Z08vNL8fIyXH11Io8+2o+uXX8LI6WlZfz974t4+unF9OwZx/z5Yyu6SZYvYt+8eSjJyTncf39vwsP9efLJhYwe3ZZ3372Y777byaRJ69m1K5thwxK49NLWvPTSMmbN2s0PP1zJ+ee3qLb2vXtzeOaZxfzvf2spK7N07hxN//5N6NQpmtTUfJKTc/j2251YC9OnjyExMZJu3SYSGurH8uXj8PIyPPPMYl5/fTX9+zfmhhs6ERHhz+WXf0XbthHMmXM1MTFVfwleufIggwZ9SmxsIIMGNaNfv8akpxeybt0hNmxIo6zMEhjoQ2CgD+3aRdKzZxy9esXTvXtsxSQr33+/k/HjvyU7u5hzz22GMc738Oef9+BwWEaPbsvBg/ksWLCX8HB/+vZtRFmZPeLD19eL0FA/QkP9WLfuEGvXHgKcv5RfemlrLrmkNT16xFUJSykpedx11yymTNmKl5fhoosSGDiwKTt2ZLJlSwYpKfkEBDjrX7kyhRYtQvn66zG0aRPBhAnf8umnm5gwoTMJCWEUFJQyd24yS5ce4LzzmvHAA32YOXMnn3yykezs4irv21VXteef/zzviFag//53Jb///ewqYTE5OZuePT+gadMQFi++7qgtmlOnbmHs2G/o0CGK8eM74XBYtm7N4O231zJ4cHO++GIkBw/mcdttP7Bw4T5atAjlkktac+GFCWzalM4nn2xk7dpD+Pp6MWJEG8aP78Tw4a2OGI9bUFDCokX7mTNnN3Pn7mbJkgOUlpbh7W1wOCxdusTwxhvDCAjw5ve/n13RVTQgwIfrr+/IVVe1p1GjYGJiAomNDar448nKlQe55ZbvWbkyhaioANLTC/H396akpIx+/Rozc+blREQcv2v02rWpPPfcEj79dBMA3brFMnBgE2JiAnnnnXXs2ZNT8XMxe/bVBAU538+SEgf/+Mev/P3vi4mLC+Lf/x5MUlIWn322mVWrUvD19WLUqLaMG9eJnJxi/r+9+w+uqj7zOP5+CAEhBCQKBDCAKD91pqDIVh0tmlLRqYIVAXWUdjrtuqNYtK7bumPXGdsZRte6CqgjlhFbFbEsU5y2Kj9sw7ILRDALCT9MhAARCIEYE0hISPLsH/eQvYR7IVcIN+f6ec1k7jnfe865z708fOc89/s9527ZEsnxiopaDh8+xpEjDUyfPpJf/epaeve+gOZm5403CvnNb9YxZEgvfvSjK/nBD4bRvXs6x483cfBgLZ06GT17dqF793RqahrYu7eGPXuqKSw8REFBBQUFBykpqaKhoQmAadNG8O67t5/xMzjfVNSJiIhIu9q48QBPPbWW++8fzT33jGrTPn/84w6WLNlBXl4Z5eW19OrVlcceu5pHHrmKCy+MnKyVln7FggWbmTfvU44cOc611w5g/Phsxo7ty4IFW1i79gtmzryCuXNzTxpZcndyc5eQn3+AhQsncffdIwCYO3cTjzyymk6djOZmJycnk1GjLiIvr6xlytlrr32Pn/wk9nS/aLt2VbFoURHr1u1n/fr9VFXVk5ZmDBzYgxEjspg3L5fhw7MAWLlyNxMnvseddw6jsPAQxcVfMmnSEAoLD7ec/I4efREffzwt7ihFY2Nz3BHGtiovP8rPf/43SkqqcHfcIyfds2dfzdChFwKwfv1+XnppE6WlX9Gpk53y19DQRHV1A9XV9QwcmMnUqcO5665hMUcsW/vss0refHMrixYVUVZWQ3Z2BsOH9yY7O4P6+ibq6hoZMCCD55+f0DKK1dzszJ69umW0tWvXNAYM6MEzz1zPvfeOahkFq609Tn7+ATp37kTXrmlkZV3Q8p5ac3emT3+fpUuLufrqfowalcXmzYcoKfmSjRvvb/l3i2fFilKmTl3eUkSawYMPfosXX7z5pNGzt97aytKlxaxcGZkSDXDddQOYNm0Eu3dX8/bb2ygvr6Vnzy7cdttQpky5nJqaBt5//3NWrNhNXV0jaWnGNddkc9NNg7j55kFcd13kC5BZs1axZ08NZtC3b3eeffY7jBnTl/nzP+UPf9hKbW3jSTH37NmFPn26U1r6FX36dGf+/FymTBnGunX7ePfdHVRX1zN3bm7Cd1r9/PMqfv/7Itau3ce6dfs4cuQ4ubmDePTRqzl2rIm7717OHXdcztKld1Bc/CUPPPBX8vMPcO+9o5g3L/ek0c6tWw/x+utbWLSoiMrKYwCkp3dixIgsBgzoQVbWBTQ0NLFsWTFZWd14/PFxLFtWzIYNB7jmmmwOH65j586vyMzsQrdunamoqCW61DGD1qXPoEGZjBnTl5Ejsxg8uCeDB/dk5MiLzsm1nOeaijoRERHpsNydXbsiJ5rxpvxVVtbx8ssFfPBBKZs2lVNX10hmZhdeeeW73HffqVNDIXKSX1fXeMpUtCVLtrN69V5mzBjBjTfm0KmTtdzQ5OjR4y0FYCKam53Dh+vIyrog7g1RHnvsY154YSNDh/bi1VcnMnHiEJqbnby8vaxatYeHHhp70h1HU1lzs1NbezyhAqK+vpH09LQzXhPZVtXV9fz61+vYtKmcbdsqqao6xhtv3Nrmf//6+kbq65tISzPS09NOe/OU+vpGNmw4wKBBmSeNGjY2NrNiRSlLlxazfHkJFRWR6zgHD+7J7bdfxqRJQ7jhhktiTpM+erSBOXM20NDQxJNPfvukawqrqo6xeXMFFRV1HDpUR0VFLRUVkcfs7AyeeuradrkDbWNjM5WVdSd9MTFv3iZmzVpNbu4g1q7dR0ZGOq+++l2mTo3/OR871siaNWX075/B8OFZp3y2n35azuzZH5OXV0Z2dgbPPfcd7rtvFO6Ql7eXxYt34O7075/R8n+qurqBmpoGevXqSk5OJjk5kZ+TOdNU1Y5ERZ2IiIikjMbGZrZvP0zfvt1DdSOKhoYm/vznndxyy5CWqWjScTQ1Nbf5DqXt9fobNhygR490rrzy4pT6CYsnnvg7zz2Xz/e/P5QFC245J19euDsFBQe57LIL414bnGo6XFFnZpOAF4E04HV3n3O67VXUiYiIiIiEk7uzfXslI0dmpVSxer6drqg7719HmFkaMB+4FRgN3GNmsedNiIiIiIhIqJkZo0ZdpIKuHSVjjHk8UOLuO929AVgMTE5CHCIiIiIiIqGXjKJuILA3ar0saBMREREREZEEJaOoizXuesqFfWb2UzP7xMw+qaioOA9hiYiIiIiIhE8yiroyICdq/RJgX+uN3P01dx/n7uP69OnT+mkREREREREhOUVdPjDMzC41sy7ADGB5EuIQEREREREJvc7n+wXdvdHMHgY+JPKTBgvdveh8xyEiIiIiIpIKzntRB+DufwH+kozXFhERERERSSXJmH4pIiIiIiIi54iKOhERERERkRBTUSciIiIiIhJiKupERERERERCTEWdiIiIiIhIiKmoExERERERCTEVdSIiIiIiIiGmok5ERERERCTEzN2THcMZmVkFsDvZccRwMXAo2UFIylJ+SXtRbkl7Un5Je1FuSXsKQ34Ndvc+sZ4IRVHXUZnZJ+4+LtlxSGpSfkl7UW5Je1J+SXtRbkl7Cnt+afqliIiIiIhIiKmoExERERERCTEVdWfntWQHIClN+SXtRbkl7Un5Je1FuSXtKdT5pWvqREREREREQkwjdSIiIiIiIiGmou5rMLNJZrbDzErM7BfJjkfCz8xKzWyLmRWY2SdBW5aZrTCz4uCxd7LjlHAws4VmdtDMCqPa4uaTmf0y6M92mNktyYlawiBObj1tZl8E/VeBmd0W9ZxyS9rEzHLM7GMz22ZmRWb2s6BdfZectdPkV8r0X5p+mSAzSwM+AyYCZUA+cI+7b01qYBJqZlYKjHP3Q1FtzwKV7j4n+PKgt7v/S7JilPAwsxuBI8Cb7n5l0BYzn8xsNPAOMB4YAKwEhrt7U5LClw4sTm49DRxx939vta1yS9rMzPoD/d19k5llAhuBKcAPUd8lZ+k0+TWNFOm/NFKXuPFAibvvdPcGYDEwOckxSWqaDCwKlhcR6XxEzsjd84DKVs3x8mkysNjd6919F1BCpJ8TOUWc3IpHuSVt5u773X1TsFwDbAMGor5LzoHT5Fc8ocsvFXWJGwjsjVov4/RJIdIWDnxkZhvN7KdBWz933w+Rzgjom7ToJBXEyyf1aXIuPGxmm4PpmSemxym35GsxsyHAWGA96rvkHGuVX5Ai/ZeKusRZjDbNYZWzdb27XwXcCjwUTHESOR/Up8nZegW4DBgD7AeeD9qVW5IwM+sBLAVmu3v16TaN0ab8ktOKkV8p03+pqEtcGZATtX4JsC9JsUiKcPd9weNBYBmRIf7yYA74ibngB5MXoaSAePmkPk3OiruXu3uTuzcDC/j/KUrKLUmImaUTOeF+y93/M2hW3yXnRKz8SqX+S0Vd4vKBYWZ2qZl1AWYAy5Mck4SYmWUEF+1iZhnA94BCInk1M9hsJvCn5EQoKSJePi0HZphZVzO7FBgGbEhCfBJSJ064A3cS6b9AuSUJMDMDfgdsc/ffRj2lvkvOWrz8SqX+q3OyAwgbd280s4eBD4E0YKG7FyU5LAm3fsCySH9DZ+Btd//AzPKBJWb2Y2APcHcSY5QQMbN3gAnAxWZWBvwbMIcY+eTuRWa2BNgKNAIPdeS7e0lyxcmtCWY2hsjUpFLgH0G5JQm7Hrgf2GJmBUHbk6jvknMjXn7dkyr9l37SQEREREREJMQ0/VJERERERCTEVNSJiIiIiIiEmIo6ERERERGREFNRJyIiIiIiEmIq6kREREREREJMRZ2IiHRoZtZkZgVmVmhm75lZ97M41kIzO2hmha3as8xshZkVB4+9o577pZmVmNkOM7slznFLzeziBOKYYGbXnWGbIa3jFBERiUVFnYiIdHR17j7G3a8EGoAH27KTmcX6LdY3gEkx2n8BrHL3YcCqYB0zGw3MAK4I9nvZzNISfgenmgCctqgTERFpKxV1IiISJmuAy80sIxh1yzezT81sMoCZ/TAYzXsf+Kj1zu6eB1TGOO5kYFGwvAiYEtW+2N3r3X0XUAKMjxPbP5vZhuDv8iCe281sfRDjSjPrZ2ZDiBSmjwYjkDcE7cvM7H+DvxMFX5qZLTCzIjP7yMy6Jfh5iYjIN4CKOhERCYVg5O1WYAvwr8Bqd78GuAl4zswygk2vBWa6+80JHL6fu+8HCB77Bu0Dgb1R25UFbbFUu/t4YB7wH0HbfwHfdvexwGLgCXcvBV4FXghGINcALwF/d/dvAVcBRcH+w4D57n4FUAXclcB7EhGRb4hYU1NEREQ6km5mVhAsrwF+B/w3cIeZPR60XwAMCpZXuHus0bivw2K0eZxt34l6fCFYvgR418z6A12AXXH2vRl4AMDdm4Cvguv6drl7QbDNRmBIIsGLiMg3g4o6ERHp6OrcfUx0g5kZcJe772jV/g/A0a/xGuVm1t/d9wcF2MGgvQzIidruEmBfnGN4jOW5wG/dfbmZTQCeTjCu+qjlJkDTL0VE5BSafikiImH0ITArKO4ws7FnebzlwMxgeSbwp6j2GWbW1cwuJTIdckOcY0yPevyfYLkX8EXUcU+oATKj1lcB/wRgZmlm1vNrvg8REfkGUlEnIiJh9AyQDmwObvv/TFt2MrN3iBRcI8yszMx+HDw1B5hoZsXAxGAddy8ClgBbgQ+Ah4LpkbF0NbP1wM+AR4O2p4H3zGwNcChq2/eBO0/cKCXY5yYz20JkmuUVbXk/IiIiAOYe79IAERERERER6eg0UiciIiIiIhJiKupERERERERCTEWdiIiIiIhIiKmoExERERERCTEVdSIiIiIiIiGmok5ERERERCTEVNSJiIiIiIiEmIo6ERERERGREPs/l9ThG5v4I3gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot training losses\n",
    "fig,(ax1) = plt.subplots(1, 1, figsize=(15,7))\n",
    "fig.suptitle(f'Model NIMA Training Metrics')\n",
    "ax1.plot(historyt, color='darkblue', label=\"Train\")\n",
    "ax1.plot(historyv, color='darkred', label=\"Val\")\n",
    "ax1.set_title('Loss (EMD) over training')\n",
    "ax1.set_xlabel('Per 100 batch')\n",
    "ax1.set_ylabel('Loss (EMD)')\n",
    "ax1.legend()\n",
    "#plt.savefig(f'metrics_training')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ee05dc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TEST Model on Validation Dataset\n",
    "# label images with: ground truth scores (t) and predicted scores(p) , green=match (score diff <.5), orange=difference < 1, red=difference > 1\n",
    "torch.cuda.empty_cache()\n",
    "zimg,zlbl = next( valid_iter  )\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "  prediction = model(zimg)\n",
    "  prediction = prediction.cpu().detach().numpy()\n",
    "  zlbl=zlbl.cpu().detach().numpy()\n",
    "print(\"img shape:\",zimg.shape,\" lbl shape:\",zlbl.shape, \" pred shape:\",prediction.shape)\n",
    "fig = plt.figure(figsize=(36, 24)) #Width, height in inches.\n",
    "correct=0\n",
    "gbcorrect=0\n",
    "cats = [1,2,3,4,5,6,7,8,9,10]\n",
    "catsf = np.array(cats)\n",
    "for idx in np.arange(BATCH_SIZE_VAL):\n",
    "    if False:# int scores\n",
    "      mylbl = np.argmax(zlbl[idx])\n",
    "      mypred = np.argmax(prediction[idx])\n",
    "    else:    # float scores\n",
    "      mylbl = np.sum(zlbl[idx]*catsf)\n",
    "      mypred = np.sum(prediction[idx]*catsf)\n",
    "    #print(\"lbl:\",mylbl,\" pred:\",mypred )\n",
    "    if mylbl >= 5 and mypred >=5:\n",
    "      gbcorrect +=1\n",
    "    elif mylbl <5 and mypred <5:\n",
    "      gbcorrect +=1\n",
    "    if np.abs(mylbl-mypred)<0.5:\n",
    "      correct+=1;\n",
    "      mycolor = \"green\"\n",
    "    elif  np.abs(mylbl-mypred)<1:\n",
    "      mycolor = \"orange\"\n",
    "    else: \n",
    "      mycolor = \"red\"\n",
    "    ax = fig.add_subplot(8, BATCH_SIZE_VAL/8, idx+1, xticks=[], yticks=[]) #nrows, ncols, index\n",
    "    ax.set_title(\"t{:.2f} p{:.2f}\".format(mylbl,mypred), fontstyle='italic',color=mycolor)\n",
    "    #ax.set_title(label, fontfamily='serif', loc='left', fontsize='medium')\n",
    "    imshow(zimg[idx])\n",
    "print(\"Exact Score Accuracy:\",int(correct/BATCH_SIZE_VAL*100),\"%\")\n",
    "print(\"Good/Bad Accuracy:\",int(gbcorrect/BATCH_SIZE_VAL*100),\"%\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6d081a6",
   "metadata": {},
   "source": [
    "\n",
    "<a name=\"section2\"></a>\n",
    "# Section 2) Deploy Model as an Aesthetic Critic\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79bed0eb",
   "metadata": {},
   "source": [
    "In this section the trained model is tested as a design critic using shapes from [Genolve](https://www.genolve.com), a website where you can make web graphics, banners, slide shows, music videos and more. A key feature of Genolve is that whenever you click a shape or design it uses simulated evolution to generate new variations of that shape.  Sometimes these variations look crufty and the idea is to use the NIMA model as an AI critic that will filter out the cruft and only show the good ones to the user.  It worked fairly well except for some edge cases *where the model prefers the cruft*. \n",
    "\n",
    "Here the top scoring designs are distorted (std=standard deviation and p=predicted score):\n",
    "![Alt text](img/star_fail.png)![Alt text](img/horse_fail.png)\n",
    "\n",
    "Another problem was liking pure whitespace (tip to photographers, use a lot of whitespace and you'll get a high rating):\n",
    "![Alt text](img/NIMAwhitespace.png)\n",
    "\n",
    "And sometimes a preference for the more boring design:\n",
    "![Alt text](img/dino.png)\n",
    "\n",
    "**To help remedy these deficiencies we train with more data.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac73c2f8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b03ae8b2",
   "metadata": {},
   "source": [
    "##  SAC Dataset\n",
    "The [Simulacra Aesthetic Captions](https://github.com/JD-P/simulacra-aesthetic-captions) dataset contains images produced from Stable Diffusion runs where users rated the images produced.\n",
    "To process the data we can re-use the same image loader used for AVA but the ratings require extra processsing:\n",
    "- Extract from an SQL databasse\n",
    "- Tabulate into a histogram\n",
    "- Softmax the histogram\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abdd0799",
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# OPTIONAL Preprocess Labels for https://github.com/JD-P/simulacra-aesthetic-captions\n",
    "#\n",
    "import sqlite3\n",
    "DATA_PATH = \"data/sac_dataset/images/\"\n",
    "# Create a SQL connection to our SQLite database\n",
    "con = sqlite3.connect(\"data/sac_dataset/sac_labels.sqlite\")\n",
    "\n",
    "sql_query = \"\"\"SELECT name FROM sqlite_master\n",
    "WHERE type='table';\"\"\"\n",
    "\n",
    "# Creating cursor object using connection object\n",
    "cursor = con.cursor()\n",
    "\n",
    "# executing our sql query\n",
    "cursor.execute(sql_query)\n",
    "print(\"List of tables\\n\")\n",
    "\n",
    "# printing all tables list\n",
    "print(cursor.fetchall())\n",
    "\n",
    "# The result of a \"cursor.execute\" can be iterated over by row\n",
    "for row in cursor.execute('SELECT * FROM ratings LIMIT 15;'):\n",
    "    print(row)\n",
    "names = [description[0] for description in cursor.description]\n",
    "print(names)\n",
    "for row in cursor.execute('SELECT * FROM paths LIMIT 5;'):\n",
    "    print(row)\n",
    "names = [description[0] for description in cursor.description]\n",
    "print(names)\n",
    "\n",
    "sql_query = \"\"\"SELECT * FROM ratings\n",
    "INNER JOIN paths ON paths.iid = ratings.iid\n",
    "WHERE ratings.rating IS NOT NULL  LIMIT 10;\"\"\"\n",
    "for row in cursor.execute(sql_query):\n",
    "    print(row)\n",
    "    print(\"--score:\",row[2],\" -path:\",row[5])\n",
    "#PROCESS\n",
    "lblO = {}\n",
    "stemp = [0,0,0,0,0,0,0,0,0,0]\n",
    "all_files = []\n",
    "sql_query = \"\"\"SELECT * FROM ratings\n",
    "INNER JOIN paths ON paths.iid = ratings.iid\n",
    "WHERE ratings.rating IS NOT NULL;\"\"\"\n",
    "for row in cursor.execute(sql_query):\n",
    "    key = row[5]\n",
    "    if key in lblO:\n",
    "      score = lblO[key].copy()\n",
    "      score[row[2]-1]+=1\n",
    "      print(\"already have:\",key,\"old score:\",lblO[key],\" and \",row[2],\" -> \",score)\n",
    "      lblO[key]=score\n",
    "    else:\n",
    "      score = stemp.copy()\n",
    "      score[row[2]-1]=1\n",
    "      all_files.append( key)\n",
    "      lblO[key] = np.array(score)\n",
    "con.close()\n",
    "print(\"total labels:\",len(lblO),\"files:\",len(all_files),\" first one:\",lblO[all_files[0]],\" 2nd:\",lblO[all_files[1]])\n",
    "for ii in lblO:\n",
    "  lblO[ii] = softmax(lblO[ii])\n",
    "print(\"softmax first one:\",lblO[all_files[0]],\" 2nd:\",lblO[all_files[1]])\n",
    "for ii in range(len(all_files)):\n",
    "  all_files[ii] = DATA_PATH +all_files[ii]\n",
    "all_files = np.array(all_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b0899d",
   "metadata": {},
   "source": [
    "Recreate the <a href=\"#loader\">loader</a> with the new data and train.\n",
    "\n",
    "## SAC Results\n",
    "After training just 1 epoch using the same training loop from AVA at half the learning rate, the binary accuracy is up to 92%.\n",
    "![SAC results](img/NIMAsac.png)\n",
    "\n",
    "Most earlier deficiencies have resolved:\n",
    "- The dinosaur kept its spots\n",
    "- The horse silhouette is correct most of the time\n",
    "- Blank white squares now score low\n",
    "\n",
    "However, other concerns have surfaced; the model still likes the mangled star and has poor color taste; choosing a green rimmed heart:\n",
    "![Mangled star](img/SACstar.png)\n",
    "![Poor color sense](img/SACheart.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "570a2abb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "475f9fe5",
   "metadata": {},
   "source": [
    "## Color Dataset\n",
    "[Collaborative Filtering of Color Aesthetics](http://www.dgp.toronto.edu/~donovan/cfcolor/cfcolor.pdf) trys to train models to predict the color preferences of individual users. \n",
    "\n",
    "To get the dataset\n",
    "13,343 color\n",
    "themes were randomly sampled from the Adobe Kuler website. Each theme\n",
    "was then rated on a scale of 1-5 stars by 40 participants on Amazon’s Mechanical Turk, producing a final dataset of 528,106\n",
    "individual ratings. \n",
    "\n",
    "The ratings are in allMTurkRatings.mat (Mathimatica/Octave) variable testRatings of shape 528106x3 has columns: userid,dataid,rating\n",
    "\n",
    "The data is inthemeData.mat variable datapoints->rgb  of shape 13343x15  has 13343 color swatches of 5 color patches in 3 rgb values.  This data was converted to image files in a separate process. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "1d151117",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theme keys: dict_keys(['__header__', '__version__', '__globals__', 'datapoints'])\n",
      "header b'MATLAB 5.0 MAT-file, Platform: PCWIN, Created on: Sun Feb 20 16:34:07 2011'\n",
      "rgb len: (13343, 15)  first one: [0.34509804 0.54901961 0.54901961 0.28627451 0.41960784 0.45098039\n",
      " 0.74901961 0.81960784 0.85098039 0.57647059 0.68235294 0.74901961\n",
      " 0.45098039 0.21176471 0.25490196]\n",
      "dex len: (13343, 1)  first one: [429863]\n"
     ]
    }
   ],
   "source": [
    "# Color Data Swatches\n",
    "import scipy.io\n",
    "DATA_PATH = \"data/cfcolor_dataset/images/\"\n",
    "themat = scipy.io.loadmat('data/cfcolor_dataset/release/themeData.mat')\n",
    "print(\"theme keys:\",themat.keys())\n",
    "print(\"header\",themat['__header__'])\n",
    "#print(\"datapoints keys:\",themat['datapoints'][0][0].keys())\n",
    "rgbA=themat['datapoints']['rgb'][0][0]\n",
    "dexA=themat['datapoints']['ids'][0][0]\n",
    "print(\"rgb len:\",rgbA.shape,\" first one:\",rgbA[0])\n",
    "print(\"dex len:\",dexA.shape,\" first one:\",dexA[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1730fec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Color Rating Data\n",
    "themat = scipy.io.loadmat('data/cfcolor_dataset/release/allMTurkRatings.mat')\n",
    "print(\"theme keys:\",themat.keys())\n",
    "print(\"header\",themat['__header__'])\n",
    "\n",
    "lblraw = themat['testRatings']\n",
    "print(\"shape:\",lblraw.shape,\" first line:\",lblraw[0])\n",
    "# build up a histogram of the scores\n",
    "lblO = {}\n",
    "stemp = [0,0,0,0,0,0,0,0,0,0]\n",
    "all_files = []\n",
    "bestone=[0,0,0,0,0,0,0,0,0,0]\n",
    "bestdex=[0,0,0,0,0,0,0,0,0,0]\n",
    "for rr in range(lblraw.shape[0]):\n",
    "    sdex = lblraw[rr][2]*2 - 2 # OPTION1 high std: convert 1-5 scale to 1-9, zero based \n",
    "    #sdex = lblraw[rr][2]+1     # OPTION2 low std: convert 1-5 shift 2-7, zero based\n",
    "    key  = str(lblraw[rr][1])+\".jpg\"       # file id is hash key\n",
    "    if key in lblO:\n",
    "      score = lblO[key].copy()\n",
    "      score[sdex]+=1\n",
    "      print(\"already have:\",key,\"old score:\",lblO[key],\" and \",sdex,\" -> \",score)\n",
    "      lblO[key]=score\n",
    "      if score[sdex] > bestone[sdex]:# keep track of highest rated in each cat\n",
    "        bestone[sdex] = score[sdex]\n",
    "        bestdex[sdex] = key\n",
    "    else:\n",
    "      score = stemp.copy()\n",
    "      score[sdex]=1\n",
    "      all_files.append(key)\n",
    "      lblO[key] = np.array(score)\n",
    "\n",
    "print(\"total labels:\",len(lblO),\"files:\",len(all_files),\" first one:\",lblO[all_files[0]],\" 2nd:\",lblO[all_files[1]])\n",
    "for ii in lblO:\n",
    "  lblO[ii] = softmax(lblO[ii])\n",
    "print(\"softmax first one:\",lblO[all_files[0]],\" 2nd:\",lblO[all_files[1]])\n",
    "print(\"high votes by scores:\",bestone,\" swatch:\",bestdex)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97573c7e",
   "metadata": {},
   "source": [
    "### Add A Specific Training Nudge\n",
    "Sometimes the generated SVG is corrupted and produces an error which gets captured as an image like this:\n",
    "![Corrupted SVG](img/0.jpg)\n",
    "We will add 10 copies of this image (0.jpg) into the color dataset with a low score so the model can recognize it and score it low and also add the complete white image (1.jpg) as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "b071e7b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "adding image: cp img/0.jpg data/cfcolor_dataset/images/13353.jpg\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'numpy.ndarray' object has no attribute 'append'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_3020278/455102755.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      6\u001b[0m   \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"adding image:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcmd\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m   \u001b[0;31m#os.system(cmd)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m   \u001b[0mall_files\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m   \u001b[0mlblO\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msoftmax\u001b[0m\u001b[0;34m(\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'numpy.ndarray' object has no attribute 'append'"
     ]
    }
   ],
   "source": [
    "nextdex = len(all_files)\n",
    "for ii in range(20):\n",
    "  jj = int(ii/10)\n",
    "  key = str(nextdex+ii)+\".jpg\"\n",
    "  cmd = 'cp img/'+str(jj)+'.jpg '+DATA_PATH+key\n",
    "  print(\"adding image:\",cmd)\n",
    "  os.system(cmd)\n",
    "  all_files.append(key)\n",
    "  lblO[key] = softmax( np.array([10,0,0,0,0,0,0,0,0,0]) )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "779881f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Convert to full datapath\n",
    "mm=0\n",
    "all_files_save=all_files.copy()\n",
    "DATA_PATH = \"data/cfcolor_dataset/images/\" \n",
    "for ii in range(len(all_files)):\n",
    "  if os.path.isfile(DATA_PATH +all_files[ii]):\n",
    "    all_files[ii] = DATA_PATH +all_files[ii]\n",
    "  else:\n",
    "    mm+=1\n",
    "    print(\"Missing file:\",DATA_PATH+all_files[ii],mm)\n",
    "all_files = np.array(all_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "63da39f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "#high votes in each cat\n",
    "for ii in range(10):\n",
    "  if bestdex[ii] is not 0:\n",
    "      image = Image.open(DATA_PATH+bestdex[ii])\n",
    "      image.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e15e362",
   "metadata": {},
   "source": [
    "Recreate the <a href=\"#loader\">loader</a> with the new data and train for 7 epochs at half the AVA learning rate.\n",
    "\n",
    "## Color Results\n",
    "After training for 7 epochs, at half the AVA learning rate, the binary accuracy was 80% while the exact score accuracy was near 50%. This is likely because we chose the higher standard deviation method to convert the scale of 1-5 to a scale of 1-10.\n",
    "![SAC results](img/NIMAcolor.png)\n",
    "\n",
    "The poor color sense is now solved. Though color is very subjective, most people would agree that the color combinations of the heart are now ordered correctly from least attractive to most attractive:\n",
    "![hearts](img/hearts.png)\n",
    "Unfortunatly the star is proving a stubborn one to fix:\n",
    "![stars](img/starfinal.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97111927",
   "metadata": {},
   "source": [
    "## Evolving the Perfect Design or Shape\n",
    "With the model perfected the genetic algorithm can now run recursively: \n",
    "- Evolve a batch of 32 designs \n",
    "- The model selects the best one\n",
    "- Repeat \n",
    "\n",
    "Here is the spirelli heart at 10 rounds of evolution with the model automatically selecting the best design for each round:\n",
    "![10 rounds of evolution](img/heart10.png)\n",
    "\n",
    "It got stuck from 4-7, a lot of designs were generated but the critic didn't think they were better, but a human may spot a better one such as this one:\n",
    "![missed heart](img/heartsym.png)\n",
    "\n",
    "10 rounds of evolution on a stacked trapezoid:\n",
    "![10 rounds of evolution](img/purple.png)\n",
    "\n",
    "And a similar case here, it got stuck after the 3rd round passing up several neat designs including this one:\n",
    "![kite-like design](img/kite.png)\n",
    "\n",
    "10 rounds of evolution on a landscape:\n",
    "![landscape](img/hills.png)\n",
    "\n",
    "And again choosing a rather plain landscape while alternatives like these were passed by: \n",
    "![landscape](img/hillsbetter.png)\n",
    "\n",
    "## Conclusion\n",
    "Though the AI critic falls short for unsupervised recursive design, it is still very useful for filtering cruft, spotting duplicates and providing aesthetic feedback to users. To try it out on [Genolve](https://www.genolve.com), just start up the design editor and while you're there ... maybe it is time to change your Twitter or LinkedIn cover photo :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "260e1a60",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:torch] *",
   "language": "python",
   "name": "conda-env-torch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}