Week2/frozenlake_Qlearning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-learning applied to FrozenLake "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Remember**: Q-learning is a model free, off-policy algorithm that can be used to find an optimal action using a Q function. Q can be represented as a table that contains a value for each pair state-action\n",
    "    \n",
    "To review Q-learning watch [Q learning explained by Siraj](https://www.youtube.com/watch?v=aCEvtRtNO-M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q-learning pipeline is quite easy an can be summarised in 5 blocks:\n",
    "\n",
    "![as](img/short_diag.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WHAT'S THE ENVIRONMENT?\n",
    "\n",
    "#### We'll apply Q-learning on a [Gym](http://gym.openai.com/) game called [FrozenLake](https://gym.openai.com/envs/FrozenLake-v0/)\n",
    "\n",
    "![](img/frozenlake_v0.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LET'S START TO CODE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import random\n",
    "from collections import namedtuple\n",
    "import collections\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BASIC FUNCTION TO CHOOSE AN ACTION FOLLOWING DIFFERENT POLICIES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def select_eps_greedy_action(table, obs, n_actions):\n",
    "    '''\n",
    "    Select the action using a ε-greedy policy (add a randomness ε for the choice of the action)\n",
    "    '''\n",
    "    value, action = best_action_value(table, obs)\n",
    "\n",
    "    if random.random() < epsilon:\n",
    "        return random.randint(0,n_actions-1)\n",
    "    else:\n",
    "        return action\n",
    "\n",
    "def select_greedy_action(table, obs, n_actions):\n",
    "    '''\n",
    "    Select the action using a greedy policy (take the best action according to the policy)\n",
    "    '''\n",
    "    value, action = best_action_value(table, obs)\n",
    "    return action\n",
    "\n",
    "\n",
    "def best_action_value(table, state):\n",
    "    '''\n",
    "    Exploring the table, take the best action that maximize Q(s,a)\n",
    "    '''\n",
    "    best_action = 0\n",
    "    max_value = 0\n",
    "    for action in range(n_actions):\n",
    "        if table[(state, action)] > max_value:\n",
    "            best_action = action\n",
    "            max_value = table[(state, action)]\n",
    "\n",
    "    return max_value, best_action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/Q_function.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Q_learning(table, obs0, obs1, reward, action):\n",
    "    '''\n",
    "    Q-learning. Update Q(obs0,action) according to Q(obs1,*) and the reward just obtained\n",
    "    '''\n",
    "    \n",
    "    # Take the best value reachable from the state obs1\n",
    "    best_value, _ = best_action_value(table, obs1)\n",
    "\n",
    "    # Calculate Q-target value \n",
    "    Q_target = reward + GAMMA * best_value\n",
    "\n",
    "    # Calculate the Q-error between the target and the previous value\n",
    "    Q_error = Q_target - table[(obs0, action)]\n",
    "\n",
    "    # Update Q(obs0,action)\n",
    "    table[(obs0, action)] += LEARNING_RATE * Q_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TEST THE POLICY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_game(env, table):\n",
    "    '''\n",
    "    Test the new table playing TEST_EPISODES games\n",
    "    '''\n",
    "    \n",
    "    n_actions = env.action_space.n\n",
    "    \n",
    "    reward_games = []\n",
    "    for _ in range(TEST_EPISODES):\n",
    "        obs = env.reset()\n",
    "        rewards = 0\n",
    "        while True:\n",
    "            # Act greedly \n",
    "            next_obs, reward, done, _ = env.step(select_greedy_action(table, obs, n_actions))\n",
    "            obs = next_obs\n",
    "            rewards += reward\n",
    "\n",
    "            if done:\n",
    "                reward_games.append(rewards)\n",
    "                break\n",
    "\n",
    "    return np.mean(reward_games)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MAIN PROCEDURE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some hyperparameters..\n",
    "GAMMA = 0.95\n",
    "\n",
    "# NB: the decay rate allow to regulate the Exploration - Exploitation trade-off\n",
    "#     start with a EPSILON of 1 and decay until reach 0\n",
    "EPS_DECAY_RATE = 0.9993\n",
    "\n",
    "LEARNING_RATE = 0.8\n",
    "\n",
    "# .. and constants\n",
    "TEST_EPISODES = 100\n",
    "MAX_GAMES = 15000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tEp: 999 Test reward: 0.3 0.5\n",
      "\tEp: 1999 Test reward: 0.56 0.25\n",
      "\tEp: 2999 Test reward: 0.71 0.12\n",
      "\tEp: 3999 Test reward: 0.7 0.06\n",
      "\tEp: 4999 Test reward: 0.19 0.03\n",
      "\tEp: 5999 Test reward: 0.0 0.01\n",
      "\tEp: 6999 Test reward: 0.78 0.01\n",
      "\tEp: 7999 Test reward: 0.74 0.0\n",
      "\tEp: 8999 Test reward: 0.8 0.0\n",
      "\tEp: 9999 Test reward: 0.77 0.0\n",
      "\tEp: 10999 Test reward: 0.77 0.0\n",
      "\tEp: 11999 Test reward: 0.74 0.0\n",
      "\tEp: 12999 Test reward: 0.7 0.0\n",
      "\tEp: 13999 Test reward: 0.75 0.0\n",
      "\tEp: 14999 Test reward: 0.75 0.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABB8AAAIWCAYAAAAf5yVaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXzdZZ3+/+vOvicnW7OfLN2XlDYB2kLZlyIt6qgIgqKjuI77jKOzqOP8dGZ+LiPO4Ci4gIIi7rSyg0ihBZp0SRfa0p4sbZM0Odlzsp5z7u8fSWMpKU3bnHxOktfz8eiDLIfkKiRtcuW+329jrRUAAAAAAECoRDgdAAAAAAAAzGyUDwAAAAAAIKQoHwAAAAAAQEhRPgAAAAAAgJCifAAAAAAAACFF+QAAAAAAAEIqyukAZyszM9MWFxc7HQMAAAAAAJykurraa63NGu910658KC4uVlVVldMxAAAAAADASYwx9ad7HdcuAAAAAABASFE+AAAAAACAkKJ8AAAAAAAAIUX5AAAAAAAAQoryAQAAAAAAhBTlAwAAAAAACCnKBwAAAAAAEFKUDwAAAAAAIKQoHwAAAAAAQEhRPgAAAAAAgJCifAAAAAAAACFF+QAAAAAAAEKK8gEAAAAAAIQU5QMAAAAAAAgpygcAAAAAABBSlA8AAAAAACCkQlo+GGPWGWMOGGMOGWO+OM7ri4wxfzbG7DDG1Bhj3hLKPAAAAAAAYOqFrHwwxkRKulvSDZIWS7rVGLP4lIf9i6SHrbUrJN0i6fuhygMAAAAAAJwRypMPF0k6ZK31WGuHJD0k6a2nPMZKShl9OlVSYwjzAAAAAAAAB0SF8G3nSzpy0vNHJV18ymO+KulJY8wnJSVKuiaEeQAAAMJGW++gfvi8R4/sbNS8OUlaU5ap1WUZWpqXoqhIxnIBAGaWUJYPZpyX2VOev1XSfdbabxtjVkv6uTFmqbU2+Lo3ZMyHJX1YkoqKikISFgAAYCp09g3p3s0e3fdinfqGA7pyQbaOdvTpvx7fL0lKjo3SRSXpWl2WodVlGVqUk6KIiPG+rAIAYPoIZflwVFLhSc8X6I3XKj4oaZ0kWWu3GmPiJGVKajn5QdbaeyTdI0mVlZWnFhgAAABhr3tgWD95oVY/3lyrnkG/bizP1Wevmae52cmSpNaeQb3kadNWT5u2Hm7TM/tHvhxKS4jWqpKRImJNWYbmZifJGMoIAMD0EsryYZukecaYEknHNDJQ8j2nPKZB0tWS7jPGLJIUJ6k1hJkAAACmlG/Qr/u21Ome5z3q6h/WdYvn6LPXztei3JTXPS4rOVYbludpw/I8SVJTV7+2Hh4pIrYcbtPje5slSZlJsSOnIkpHygh3RgJlBAAg7BlrQ3eQYHR15nclRUr6ibX268aYr0mqstY+Mrr94l5JSRq5kvEFa+2Tb/Y2KysrbVVVVcgyAwAATIb+oYAeeKleP/jLYbX5hnTlgix97toFWlaQek5v70h732gR4dVWT5uOdw9KknJT48bKiNVlGSpwJUzmbwMAgAkzxlRbayvHfV0oy4dQoHwAAADhbGA4oIdeadDdzx1Wa8+g1s7L1Geuma8Kt2vS3oe1Vh6vb+xkxFZPm9p9Q5KkovSEkVMRc0cKieyUuEl7vwAAvBnKBwAAgBAb8gf16+oj+t9nD6mpa0AXlaTr89fO18WlGSF/38Gg1cGWnrErGi972tQ94JcklWUljs6LyNSq0gylJ8aEPA8AYHaifAAAAAgRfyCo3+04pu8985qOdvRrZVGaPn/dAq0py3BsFkMgaLWvsVtbPV5tOdymbbXt8g0FJEkLc5LHrmlcXJqh1PhoRzICAGYeygcAAIBJFghaPbLrmO56+jXVtfVpWX6qPnfdfF0xPyvsBkAOB4KqOdo1sk3jcJu21bVr0B9UhJGW5KVqTVmGVpVl6KLidCXGhnIeOQBgJqN8AAAAmCTBoNWje5r03adf06GWXi3MSdbnrp2vaxfPCbvS4XQG/QHtbOjUltF5ETsaOjQcsIqKMCovSNWaskytLstQhduluOhIp+MCAKYJygcAAIDzZK3Vk/uO67+fOqj9zT2am52kz14zXzcszVFExPQoHU6nfyig6vqOsU0aNUe7FAhaxURGaEVR2lgZcUFhmmKiIpyOCwAIU5QPAACEmZ9vrdOz+1u0bmmOrl+So7QEhgCGK2utnjvQqu88dVC7j3WpJDNRn756njYsz1PkNC8dTqdnYFhVdX8tI/Y2dstaKT46UpXFrrGZEcvyUxUVSRkBABhB+QAAQJjZ8D8vaE9jl6yVoiON1s7L0vryXF27eI6S4xgAGA6stXrxUJu+/dQB7WjoVIErXp+6ep7+ZkX+rPuGu7NvSC/Xto+t9jxwvEeSlBQbpYtK0rW6NEOryzK0ODdl2p8CAQCcuzcrH5goBADAFLPWqtbr03tXufXOigJtqmnSpl2NenZ/i2KiInTlgiytL8/T1YuylRDDX9VOeNnTpm8/dVCv1LYrNzVOX3/7Ur2ronDWXjlIS4jR9UtGTulIkrd3cGx45dbDbXp2f4skKTU+WqtKR8qINXMzNS87adrMwQAAhBZf0QAAMMVaewbVO+hXaWaiygvSVF6Qpi+uW6gdRzq0cVeTHt3dpCf2Hld8dKSuXpSt9eV5umJBFoP/psD2hg5958mDeuGQV1nJsfrqhsW65aIi/tufIjMpVuvL87S+PE+S1Nw1oK0er7YebtOWw216Yu/x0cfFaNXoqYg1ZZkqzkigjACAWYprFwAATLGXPG265Z6X9LO/vUiXzc96w+sDQatXatu1qaZRj+1pVrtvSEmxUbpu8RytX56rS+dmzdqfwIfK7qNd+s5TB/TnA63KSIzRx64o020XuxUfQ+lwLo60942civC0acthr453D0qSclLixtZ6rinLUIErweGkAIDJxMwHAADCyC9ebtA//X63Nn/hShWmv/k3X/5AUFsOt2lTTaMe39Os7gG/UuOjdf2SOdqwPE+rSzNm3fyByfRqU7e+89RBPbXvuFLjo/WRy0t1x+piJcZyOHSynLhmdGKt50uH29TmG5IkFabHj82LyEiMdTip8/LS4lWWlcjpEADTFuUDAABh5Ot/2qf7t9Zr/9fWndVwviF/UJtfa9WmmiY9te+4egf9ykiM0bqlOdqwPE8XFqfP2O0Lk+214z367tOv6U+7m5QcG6UPrS3V315azLDPKWCt1cHjvdp62Ksth9v0kqdN3QN+p2OFjezk2LFtImvKMlWYHk8ZAWDaoHwAACCMfPC+bTra0a8nPnvZOb+NgeGAnjvQqo01jXrm1eMaGA4qOzlWb1mWqw3L87SyKI1vWMZR6/XprqcP6o+7GpUQHakPXFKiO9eWKjWB0sEpgaDVay098g3O7gLCWum1lt6xuRne3pGrKvlp8WNlxOqyDOWlxTucFABOj/IBAIAwctW3ntOCnGT93+0Vk/L2+ob8eubVFm3c1ajnDrZqyB9Uflq8bizP1fryXC3LT531RcSR9j5975nX9LsdxxQdaXTH6mJ95PIypSfGOB0NeANrrQ639o5cVRm9rtLZNyxJKs5I0OqyzLFCIiuZ6yoAwgflAwAAYWI4ENSif31cH76sVF9Yt3DS337PwLCe2ndcG3c1avNrXvmDVu6MBK0vz9X68jwtzEmeVUVEY2e//vfPh/TwtiOKiDC67eIifeyKMmUnxzkdDZiwYNBqf3OPthz26iVPm172tKtn9KTIvOwkrSkbORVxcUmGXBRqABxE+QAAQJjwtPbqqm//Rd9613K9s6IgpO+rs29IT+xt1qaaJr14yKugleZmJ40VEXOzk0L6/p3U0j2g7z93WL94uUFWVu++sFCfuHKuclM5so7pzx8Iam9j9+g2kTZtq21X/3BAxkiLclLGyoiLStKZYwJgSlE+AAAQJp7ed1wf+lmVfvuxNapwu6bs/Xp7B/XYnmZt2tWoV+raZa20KDdF68tztaE8T0UZM2PlYVvvoH7wl8P62dZ6+YNW71xZoE9ePZeVjpjRhvxB1RztHJsXUd3QoSF/UJERRkvzU0eHV2aostilhBg2uQAIHcoHAADCxL3Pe/T1R1/Vzi9fq7QEZ45HH+8e0J9qmrSpplHbGzolSeUFqdpQnqcby3On5UC7zr4h3fO8R/dtqdPAcEBvW5GvT101T8WZiU5HA6bcwHBA2xs69NJoGbHzSKf8QavoSKMLCtNGh1dmakVRmuKiI52OC2AGoXwAACBMfOl3NXp8T7N2fPk6p6NIko529I0WEU3afaxLklThdmlDea7esixX2SnhPRuhe2BYP95cqx+/UCvfkF/ry/P06avnzegrJcDZ8g36VVXfMTK88rBXu491KWil2KgIVbhdIycj5maovCBN0ZERTscFMI1RPgAAECbe/cOt8getfvuxNU5HeYM6r0+bahq1qaZJ+5t7ZIx0cUm6NizP0w1Lc8NqM0TvoF/3vVire573qHvAr3VLcvSZa+dpYU6K09GAsNc9MKxtte1j2zT2NXVLkhJiInVhcbpWl41c01iSl6rIiNkzoBbA+aN8AAAgTFz49ad1+fwsfetdy52O8qYOtfRo464mbaxplKfVp8gIozVlGdpQnqfrl+QoNcGZIXb9QwH9bGudfvi8R+2+IV29MFufvXa+luanOpIHmAk6fEN6ubZtrIx4raVXkpQcF6WLSzLG1nouzElWBGUEgDdB+QAAQBjoGRjWsq8+qS+sW6CPXzHX6TgTYq3Vq0092ljTqE01jTrS3q/oSKPL5mVp/fJcXbNozpRM0x8YDugXLzfo+88dlrd3UGvnZepz187XiqKpG9oJzBYtPQN6ydOurYe92nq4TXVtfZIkV0K0Vo0Or1xdlqGyrKRZtboXU6epq199QwGnYzguJS5aWcmxTsc4K29WPjDuFgCAKVLr9UmSSqfREERjjBbnpWhxXoq+cP0C1RztGrua8cz+FsVEReiqBdlavzxXVy+co/iYyR1eN+QP6ldVR3T3s4fU3D2gVaXp+r/bV+rC4vRJfT8A/io7OU43Lc/TTcvzJEmNnf1jmzRe8rTpsT3NkqSs5NixTRqryzJUlJ5AGYFz5g8E9eS+47pvS51eqW13Ok5YeP+aYn31piVOx5g0lA8AAEyRsfIha3oOQzTGaHlhmpYXpulLNyzS9oYObapp0p92N+nxvc1KiInU1YvmaH15ri6fn3VeU/SHA0H9bvtRfe+ZQzrW2a8Kt0vfuXm51szNnMTfEYCJyEuL1zsqCvSOigJZa9XQ3jcyvNIzUkg8sqtRkpSfFv+6kxHTcXMOpl5b76Ae2nZED7xUr6auARW44vWFdQuUz8ePSjOn59cLp8O1CwAApsh3njqo/3n2Nb36tXUzar1dIGj1cm2bNtU06bHdTeroG1ZybJSuXTJHG8rzdMncTMVETWyCfiBo9cedx3TXM6+pvq1P5QWp+ty183X5/Cx+ogqEIWutDrf6Rq5oeEZmRnT0DUuSijMSRuZFlGVqVWm6spPDe3sOptbuo126b0udNtY0asgf1CVzM3TH6mJdvWgOg06nMWY+AAAQBj75yx3aeaRDm79wldNRQmY4ENSWw23atKtRT+xtVveAX6nx0Vq3JEcbludpVWm6osZZ5RcMWv1pd5O++/RBHW71aVFuij537Xxdsyib0gGYRoJBqwPHe8aGV75c26aeAb8kaW520sipiNIMrSrNkCuMNuhgagz5g3psT5Pu31Kn7Q2dSoiJ1N+szNcdq4s1b06y0/EwCSgfAAAIAzd+b7MykmL1s7+9yOkoU2LQH9Dmg15tqmnUU/uOyzcUUEZijG5YlqMN5Xm6sDhdxkhP7D2u7z59UPubezR/TpI+e818Xb8kh6n6wAwQCFrtbewaKyO21bWrbyggY6SFOSljZcRFpelKmYLhtXBGS8+AfvnyET34cr1aegblzkjQ+1YX650VBUqN5//7TEL5AACAw6y1WvKVJ3RzZeGMGh41UQPDAT13oEUbdzXpmf3HNTAc1JyUWLkSYrS/uUelmYn69DXztL48j+O2wAw2HAiq5mjn2ADL6voODfqDijDSsvxUrZ2XpRvLc7UwJ5lTTzPAjoYO3b+lTn/a3aThgNXl87P0/jXFunx+FgXzDEX5AACAw5q7BrTqP57R1966RO9bXex0HEf5Bv16Zn+LNu5qVFNXv96/pkRvuyBv3OsYAGa2geGAdjR0js6L8Gp7Q6cCQauyrEStL8/ThuW5mpvNcfzpZNAf0J9qRq5W7DrapaTYKL2zokDvXe1W2TQduIyJo3wAAMBhWw579Z57X9YDH7xYl85jYwMAjKetd1CP7WnWpppGvVzbLmulhTnJ2rA8T+vLc+XOmD6rimeb5q4BPfhyvX75SoO8vUMqzUrUHauL9Y6KAiXFsmRxtniz8oGPAgAApoCndWTNZkkWXzgDwOlkJMXq9lVu3b7KrePdA3p0d5M21TTpm08c0DefOKDyglStL8/VjeV5rGIMA9ZaVdd36L4tdXp8T7MC1uqqBdm6Y02xLp2bydUKvA7lAwAAU6DW61NcdIRyU1g1BwATMSclTh+4pEQfuKRExzr79WhNkzbWNOobj+7XNx7drwq3a6SIWJarbP5snVIDwwE9sqtR92+p097GbiXHRen9a4r13tVuTqfgtLh2AQDAFPjAT19RU9eAHv/MZU5HAYBprb7Np001IyciXm3qljHSxSXpWl+epxuW5igjKdbpiDPWsc5+PfBSvR56pUEdfcOaPydJd6wp1ttX5Cshhp9rg5kPAAA47opv/llL8lJ1920rnY4CADPGoZZebapp1MZdjTrc6lNkhNGasgxtKM/T9UtylJrAGsfzZa3VS5523b+lTk/ua5YkXbNojt6/pliryzLYSoLXoXwAAMBBQ/6gFn35cX3s8jL9/fULnI4DADOOtVb7m3u0qaZRm2qaVN/Wp+hIo7XzsrS+PFfXLp6j5DiKiLPRPxTQ73cc08+21ml/c4/SEqL17gsL9d5VbhW4EpyOhzDFwEkAABzU0N6nQNCqlGGTABASxhgtyk3RotwU/f11C7TnWLc21jTqTzVNenZ/i2KiInTlgiytL8/T1YuyuSLwJo609+nnL9XrV9uOqKt/WItyU/Rf71imm5bnKz4m0ul4mMb4rAMAIMQ8rb2SpJJMygcACDVjjJYVpGpZQaq+uG6hdhzp1MZdjXp0d5Oe2Htc8dGRunpRttaX5+mKBVmKi+YbamutXjzUpvu21OmZ/ccVYYzWLcnRHWuKdWGxi6sVmBSUDwAAhFitd2TNZmlmksNJAGB2iYgwqnC7VOF26V/XL9a2unZt3NWox/Y0a1NNk5Jio3Td4jlavzxXl87NUkxUhNORp5Rv0K/fbT+q+7fW61BLrzISY/SJK+bqtlVFyk1llSkmF+UDAAAh5mn1KSMxhsFnAOCgyAijVaUZWlWaoX+7aYm2etq0aVeTHtvTpN/tOKbU+Ghdv2SONizP0+rSDEVFztwios7r08+21uvXVUfUM+jXsvxUfetdy7W+PJeTIAgZygcAAEKs1utj3gMAhJGoyAitnZeltfOy9O9vW6oXDrVq064mPbq7WQ9XHVVGYozWLc3R+vI8XVSSrsiI6X/tIBi0ev61Vt2/pU7PHWxVpDF6y7Jc3bGmWCuL0rhagZCjfAAAIMQ83l5dtTDb6RgAgHHEREXoqoVzdNXCORoYDui5A63aVNOo320/pgdfblB2cqzesixXG5bnTctv0nsGhvWb6qP62dZ61Xp9ykyK1aeumqfbLi5Sdkqc0/Ewi1A+AAAQQl39w/L2Dqk0i3kPABDu4qIjtW5pjtYtzVHfkF/P7m/Rxl2N+sUrDbpvS53y0+J1Y3mu1pfnall+algXEYdaevWzrXX6bfVR+YYCWlGUprtuuUA3LM2ddbMtEB4oHwAACKETwybZdAEA00tCTJTWl+dpfXmeegaG9fSrx7VxV5N++mKt7nneI3dGgtaX52p9eZ4W5iSHRRERCFo9d6BF922p0+bXvIqJjND68pGrFcsL05yOh1mO8gEAgBCq9Y6s2Sxj5gMATFvJcdF6+4oCvX1Fgbr6hvXE3mZtrGnUD/7i0d1/PqyyrERtWD5SVMzNnvqTbl39w/p11RH9bGu9Gtr7NCclVp+/dr5uvbhImUmxU54HGA/lAzBFfviXw/rZ1nqlxkcrIylGmUmxykiMUXpSjDITY5WRFKOM0ZdlJMUoIYZPT2Am8LT6FGGkwvQEp6MAACZBakK0br6wUDdfWKi23sHRtZ2NuuuZ1/Tdp1/Twpzk0SIiV+6M0BbPB4/36L4tdfr99mPqHw7owmKXvrBuga5fkqPoGbytA9MT390AU+DJvc36j8f2q9LtUmp8tLy+IdV6fWrrHVL/cGDcfyc+OnKskMgcLSTSE2OVmTTydMaJwiIxVumJMdzdA8KUx+tTYXqCYqNYXQYAM01GUqxuX+XW7avcOt49oEd3N2lTTZO++cQBffOJAyovSNX68lzdWJ6n/LT4SXmfgaDVU/uO6/4tddrqaVNsVITeekGe3re6WEvzUyflfQChYKy1Tmc4K5WVlbaqqsrpGMCEeVp79db/fVElWYl6+COr37A7uW/Ir7beIbX5htTWO/j6p31D8o69bFDtviENB8b/nE2Jixo5TZEUo/TEk0uL2FPKihilJcTMiJVRwHRww12bNSclVvd94CKnowAApsixzn79qaZRm2qaVHO0S5JU4XaNFBHLcs9py0SHb0gPbTuiB16q17HOfuWlxum9q4t1y4WFciXGTPZvATgnxphqa23luK8LZflgjFkn6S5JkZJ+ZK39z1Ne/9+Srhx9NkFStrX2TSehUD5gOukb8uttd7+o1p5BbfzkpSpwnd+xa2utugf8Y8XEX/858rR39GXtoy9r7xvSeJ/iEUYjBcXoqYmTr4H8taz469PJsVFhMUQJmG6CQaslX3lCt15UpC9vWOx0HACAA+rbfNpU06SNuxq1v7lHxkgXl6RrfXmebliao4wzzGTY19it+7fU6Q87j2nQH9Sq0nS9f02xrlk0R1FcrUCYebPyIWTXLowxkZLulnStpKOSthljHrHW7jvxGGvtZ096/CclrQhVHmCqWWv1xd/uHllz9LcXn3fxIEnGGKXGRys1PlqlWWd+fCBo1dH313Li9Scq/lpU7G3slrd3UD0D/nHfTkxkxDgnKl4/o+LEyYrMpNg3nO4AZqvm7gH1DwdUwrBJAJi13BmJ+sSVc/WJK+fqUEuvNtU0auOuRv3LH/boK4/s1ZqyDG0oz9P1S3KUmhAtSRoOBPXk3pGrFa/UtSsuOkJ/s7JAd6xxa2FOisO/I+DchHLmw0WSDllrPZJkjHlI0lsl7TvN42+V9JUQ5gGm1E9frNMjuxr1D9cv0KXzMh3JEBlhlJkUOzrlOPmMjx/0B8ZOTZx8DcTrG/ln++jLDrf0yts7qEF/cNy3kxgTqfTRQiLz5CsfSbEqyUzQVQvnTPLvFAhPJ9ZslrFmEwAgaW52kj5zzXx9+up52t/cM1pENOkLv63RP/9ht9bOy9LCnGT9bvsxNXcPqDA9Xv/8lkW6ubJwrJgApqtQlg/5ko6c9PxRSReP90BjjFtSiaRnT/P6D0v6sCQVFRVNbkogBF6pbdc3Hn1V1y6eo49dXuZ0nAmLjYpUbmq8clPPPBDJWqu+ocDYPIoT/xw5UTGkdt/ICYtjnQOqOdqldt+Q/MGROyCPf2YtrT1mBU/ryJpNTj4AAE5mjNGi3BQtyk3R31+3QLuPdWlTTZM27WrUs/tbdOncTP3725bqqoXZzOnCjBHK8mG8z5LTDZi4RdJvrLXjjv231t4j6R5pZObD5MQDQqOle0Cf+MV2Fbji9e2blytihv6FYYxRYmyUEmOjVJRx5isl1lq91tKr6/77eW2rbad8wKzg8foUHx2pnHMYLAYAmB2MMSovSFN5QZq+uG6hugeGlZbAAEnMPKGcUHJUUuFJzxdIajzNY2+R9MsQZgGmxHAgqE/8Yrt6B/z64XsrlRLH8bgTjDGal52kOSmxqqrvcDoOMCU8rT6VZCYysBUAMCEREYbiATNWKMuHbZLmGWNKjDExGikYHjn1QcaYBZJckraGMAswJf7j0f3aVteh/3zHMi3IOfOMhdnGGKNKd7qq6igfMDvUen0q5coFAABA6MoHa61f0t9JekLSq5IettbuNcZ8zRhz00kPvVXSQzaUOz+BKfDHncf0kxdr9YFLivXWC/KdjhO2KtwuHevsV3PXgNNRgJAa9Ad0tKNPpQybBAAACOnMB1lrH5X06Ckv+/Ipz381lBmAqXCguUdf/O1uVbpd+qe3LHI6TlirLHZJkqrrO3Rjea7DaYDQaWjrU9BKpVlJTkcBAABwXCivXQCzQvfAsD76QLWS4qL0/dtWKjqST6s3syg3RfHRkaqqb3c6ChBSh1tH1myWcPIBAACA8gE4H8Gg1ecf3qUj7X36/m0rlc1E+zOKjozQ8sJUVTN0EjNcrXe0fGDmAwAAAOUDcD5+8PxhPbXvuP7pLYt0YXG603GmjUp3uvY2dqtvyO90FCBkPK29ykyKZesNAACAKB+Ac/bCa15964kD2rA8Tx+4pNjpONNKRbFLgaDVziOdTkcBQoZNFwAAAH9F+QCcg2Od/frkL7drbnaS/vNvlskY43SkaWVlkUvGSNWs3MQM5vH62HQBAAAwivIBOEsDwwF97IFq+QNWP7i9QomxIV0aMyOlxkdrfnayqpj7gBmqs29I7b4hTj4AAACMonwAztK/bdynmqNd+vbNy1mhdx4qil3a3tChYNA6HQWYdJ4TwyYz+TMCAABAonwAzsrD247ol6806ONXlOm6JTlOx5nWKt0u9Qz4dbClx+kowKSrHV2zyckHAACAEZQPwATtPtqlf/njHl06N1Ofv26B03GmvQq3S5JYuYkZyePtVWSEUaErwekoAAAAYYHyAZiADt+QPvpAtTITY3TXLRcoMoIBk+erKD1BmUmxDJ3EjFTr9akoPUExUfw1CwAAIElMygPOIBC0+vSvdqq1Z1C//uhqZSTFOh1pRjDGqNLtYugkZiuKvq4AACAASURBVCRPq08lbLoAAAAYw49kgDO46+mDev5gq/7trUu0vDDN6TgzSmWxSw3tfWrpGXA6CjBpgkGrWtZsAgAAvA7lA/Amnt53XN979pBurizQLRcWOh1nxhmb+8DVC8wgjV39GvQHVcKwSQAAgDGUD8Bp1Hl9+uzDO7U0P0Vfe+tSGcOch8m2JC9VsVERXL3AjFI7umazlDWbAAAAYygfgHH0DwX00QeqFRlh9H+3VSguOtLpSDNSTFSElhemUT5gRvGwZhMAAOANKB+AU1hr9U+/360Dx3t01y0rVJjOqrxQqnS7tPdYl/qHAk5HASZFrdenxJhIZScznBYAAOAEygfgFD9/qV6/33FMn7tmvi6fn+V0nBmvstglf9Bq19FOp6MAk+Jwa69KshK5qgUAAHASygfgJNX17fraxn26emG2PnHlXKfjzAori0aHTnL1AjPEyKYL5j0AAACcjPIBGNXSM6CPP7hd+a54fefdFygigp9aToW0hBjNzU6ifMCMMDAc0LHOfpWwZhMAAOB1KB8AScOBoP7uFzvU1T+sH9xeodT4aKcjzSqVbpeq6zsUDFqnowDnpb6tT9YybBIAAOBUlA+ApP//8f16pbZd//E3y7QoN8XpOLNOhdulrv5hHW7tdToKcF48ox/DXLsAAAB4PcoHzHqbahp17+Za3bHarbevKHA6zqxUWZwuSazcxLTn8Y6s2Szh5AMAAMDrUD5gVnvteI++8JsarSxK0z/fuNjpOLNWcUaCMhJjVFVH+YDpzdPqU3ZyrJJio5yOAgAAEFYoHzBr9QwM6yMPVCshJlLfv61CMVF8OjjFGKMKt0vV9e1ORwHOS623l3kPAAAA4+C7LcxK1lr9w69rVN/Wp/99z0rlpMY5HWnWqyx2qa6tT609g05HAc6Zx+tTCfMeAAAA3oDyAbPSPc979PjeZn3phoVaVZrhdBxIqnCPzH1g5Samqw7fkDr7hlXGyQcAAIA3oHzArLPlkFf/9fh+3bgsVx+8tMTpOBi1ND9FMVER2t5A+YDpyeMd3XRB+QAAAPAGlA+YVRo7+/XJX+5QaVaS/uud5TLGOB0Jo2KjIlWen6qqOuY+YHrytI5uuuDaBQAAwBtQPmDWGPQH9PEHt2vQH9QPbq9gGn0Yqih2ac+xbg0MB5yOApw1j9enqAijQle801EAAADCDuUDZo1/37RPO4906lvvKtfcbH4yGY4q3ekaCgS1+1iX01GAs1bb6lNRRoKiIvmrFQAA4FR8hYRZ4TfVR/XASw36yOWlWrc01+k4OI0Kt0uSVFXH3AdMPx5vr0q5cgEAADAuygfMeHuOdemff79bq0sz9A/XLXA6Dt5EemKMSrMSVV3P3AdML4GgVV1bH8MmAQAAToPyATNaZ9+QPvZgtVwJMfqf96zgOPQ0UOl2qbq+Q9Zap6MAE9bY2a8hf1ClmZQPAAAA4+E7McxYwaDVZ361U81dA/r+7SuVmRTrdCRMQKU7XR19wzo8ujkAmA483hObLigfAAAAxkP5gBnre8++pucOtOorG5ZoZZHL6TiYoIrikf9XXL3AdOJp7ZUklWYx8wEAAGA8lA+Ykf68v0V3PfOa3rGyQLddXOR0HJyF0sxEuRKiGTqJaaXW61NybJQyk2KcjgIAABCWKB8w4zS09enTD+3QopwUff3tS2WMcToSzoIxRhVul6obKB8wfXhafSrNSuTPGwAAgNOgfMCM0j8U0EceqJYxRj+4vUJx0ZFOR8I5qHCny9PqU7tvyOkowITUen3MewAAAHgTlA+YMay1+uc/7Nb+5m5995YLVJSR4HQknKPKsbkPnH5A+OsfCuhYZz/zHgAAAN4E5QNmjAdfbtDvth/Tp6+epysXZDsdB+dhWX6qYiIjVMXQSUwDdW1sugAAADgTygfMCNsbOvRvG/fqygVZ+tRV85yOg/MUFx2ppfkpqmboJKYBz+ha2NIsygcAAIDToXzAtOftHdTHH9iunNQ4/fe7L1BEBAPfZoLK4nTVHOvSoD/gdBTgTdV6R9ZscvIBAADg9CgfMK35A0F98hc71NE3pP+7rUJpCay5mykq3C4N+YPac6zL6SjAm/K0+pSbGqeEmCinowAAAIQtygdMa9988oC2etr0jbcv09L8VKfjYBJVuEeGTlZx9QJhzsOmCwAAgDOifMC09djuJv3wLx7dvqpI76gocDoOJllmUqyKMxJUxcYLhDFrrTytvcx7AAAAOAPKB0xLh1p69fe/3qULCtP0r+sXOx0HIVLhTtf2+g5Za52OAoyr3Tek7gG/SjJZswkAAPBmKB8w7fQO+vXRB6oVFx2p/7t9pWKjIp2OhBCpLHapzTekurY+p6MA4/J42XQBAAAwESEtH4wx64wxB4wxh4wxXzzNY242xuwzxuw1xvwilHkw/Vlr9Y+/qZGntVf/854Vyk2NdzoSQqhybO5Du8NJgPHVnlizycwHAACANxWy8sEYEynpbkk3SFos6VZjzOJTHjNP0pckXWKtXSLpM6HKg5nhxy/U6k+7m/SP6xZqTVmm03EQYmVZSUqNj1Y1cx8Qpg57exUdaVTgSnA6CgAAQFgL5cmHiyQdstZ6rLVDkh6S9NZTHnOnpLuttR2SZK1tCWEeTHNbD7fpPx7br3VLcvThy0qdjoMpEBFhVOF2MXQSYau21Sd3RqIiI4zTUQAAAMJaKMuHfElHTnr+6OjLTjZf0nxjzIvGmJeMMevGe0PGmA8bY6qMMVWtra0hiotw1tw1oE/+crvcGQn65rvKZQxf6M8WFW6XDrX0qrNvyOkowBt4vD6uXAAAAExAKMuH8b47PHVkfZSkeZKukHSrpB8ZY9Le8C9Ze4+1ttJaW5mVlTXpQRHehvxBffzBavUNBfTD2yuUHBftdCRMoRNzH7h6gXATCFrVt/lUwrBJAACAMwpl+XBUUuFJzxdIahznMX+01g5ba2slHdBIGQGM+fqf9ml7Q6e++c7lmjcn2ek4mGLlBWmKijBcvUDYOdrRp+GAVRlrNgEAAM4olOXDNknzjDElxpgYSbdIeuSUx/xB0pWSZIzJ1Mg1DE8IM2Ga+f2Oo7p/a73uXFuiG8tznY4DB8THRGpJfqqq6ygfEF5OrNnk5AMAAMCZhax8sNb6Jf2dpCckvSrpYWvtXmPM14wxN40+7AlJbcaYfZL+LOkfrLVtocqE6WVfY7e+9LvdurgkXf+4bqHTceCgSrdLu452asgfdDoKMMbDmk0AAIAJiwrlG7fWPirp0VNe9uWTnraSPjf6CxjT1Tesjz5QrdT4aP3ve1YqKjKUh3QQ7irdLv34hVrtbezSiiKX03EASVKtt1cpcVFKT4xxOgoAAEDY4zs6hJ1g0OpzD+9UU1e/vn/bSmUlxzodCQ6rKGboJMKPp9Wn0qwktu8AAABMAOUDws7dfz6kZ/a36F/XL1aFO93pOAgD2clxKkpPUBVzHxBGalmzCQAAMGGUDwgrzx1o0XeePqi3r8jXe1e5nY6DMFLpdqmqvkMjt7UAZ/UN+dXUNaBShk0CAABMCOUDwsaR9j59+qGdWjAnWd94+zKOMuN1Kopd8vYOqqG9z+kogGpPbLpgzSYAAMCEUD4gLAwMB/SxB6sVtFY/uL1C8TGRTkdCmKkcvYLD1QuEg7FNF5x8AAAAmBDKBzjOWqt//cMe7TnWre+++wIVc4ca45iXnaTkuChVMXQSYeDEyYfiDP68AgAAmAjKBzjuoW1H9Ovqo/rUVXN19aI5TsdBmIqIMFpZ5FJ1fbvTUQB5WnuVnxbPKS0AAIAJonyAo3Ye6dRX/rhXl83P0qevme90HIS5SrdLB4/3qqt/2OkomOVqvT6VcEoLAABgwigf4Ji23kF9/IFqZSXH6q53X6DICAZM4s1VFLskSdsbuHoB51hr5Wn1Me8BAADgLFA+wBGBoNWnHtohr29IP3xvhVyJMU5HwjRwQWGaIiOMqhk6CQd5e4fUM+jn5AMAAMBZoHyAI7795AG9eKhN/9/blmppfqrTcTBNJMREaUleiqqY+wAHeVp7JUmlWazZBAAAmCjKB0y5J/Y26/vPHdatFxXp5spCp+Ngmqlwu7TzSKeGA0Gno2CWOrHpopSTDwAAABNG+YAp5Wnt1ecf3qXlBan66k2LnY6DaajSna6B4aD2NXY7HQWzlMfrU0xUhPLS4p2OAgAAMG1QPmDK+Ab9+ugD1YqONPr+7RWKjWJFHc5e5ejQyap65j7AGZ5Wn4ozEhiSCwAAcBYoHzAlrLX6x9/W6FBLr/7n1pXK5yeGOEdzUuKUnxavauY+wCEeb69KM5n3AAAAcDYoHzAl7ttSp001Tfr76xfo0nmZTsfBNFdZ7FJVXYestU5HwSzjDwTV0NanEtZsAgAAnBXKB4Rc98Cwvv3kQV2xIEsfu7zM6TiYASrdLrX0DOpoR7/TUTDLHOnolz9oGTYJAABwligfEHK/euWIegf9+vy1C2QMd6Rx/irc6ZKkauY+YIrVek+s2aR8AAAAOBuUDwip4UBQP3mxVqtK07WsINXpOJghFuQkKzk2SlXMfcAU87SeWLPJzAcAAICzQfmAkHp0d5OaugZ059pSp6NgBomMMLqgKE1VdZx8wNTyeH1KS4iWKzHG6SgAAADTCuUDQsZaq3ue96gsK1FXLsh2Og5mmEp3ug4c71H3wLDTUTCLeFp7mfcAAABwDigfEDJbPW3a29itD60tVUQEsx4wuSqLXbJW2tHQ6XQUzCK1Xp9KuHIBAABw1igfEDL3Pu9RZlKM3r4i3+komIEuKExThJGq65j7gKnRO+jX8e5Bhk0CAACcA8oHhMRrx3v05wOteu+qYsVFRzodBzNQYmyUFuWmqIqNF5gidd4TwyYpHwAAAM4W5QNC4kebaxUbFaH3rnY7HQUzWKXbpZ1HOuUPBJ2OglngcOuJNZtcuwAAADhblA+YdC09A/r9jmN6Z0WB0pkIjxCqKE5X31BA+5t7nI6CWcDT6pMxkjsjwekoAAAA0w7lAybdz7fWazgY1AcvLXE6Cma4SrdLklTF3AdMgVqvT/lp8VwlAwAAOAeUD5hUfUN+/fylel2zaA5HkxFyeWnxykuNY+4DpoTH26sS5j0AAACcE8oHTKrfVh9VZ9+wPnxZqdNRMEtUFKermvIBIWatVW2rT2WUqgAAAOeE8gGTJhC0+tELtVpemDZ2HB4ItUq3S01dAzrW2e90FMxgLT2D8g0FOPkAAABwjigfMGme2ndc9W19+vDaUhljnI6DWaKCuQ+YAp7W0TWbWZQPAAAA54LyAZPm3s0eFbjidf2SOU5HwSyyMCdZiTGRXL1ASHm8I2s2OfkAAABwbigfMCmq6ztUXd+hD15aoqhIPqwwdaIiI3RBUZqq6igfEDq1rT7FRkUoLzXe6SgAAADTEt8lYlL8aLNHKXFRurmy0OkomIUq3Ona39yt3kG/01EwQ3m8PpVkJioigitlAAAA54LyAeetvs2nJ/Y267ZVbiXGRjkdB7NQpduloJV2NnQ6HQUzVK3Xx7wHAACA80D5gPP2kxdqFRlh9P41xU5HwSy1oihNEUaqqmfoJCbfkD+ohvY+5j0AAACcB8oHnJfOviE9XHVUNy3P15yUOKfjYJZKjovWgpwUhk4iJI509CkQtCrNTHI6CgAAwLRF+YDz8uDLDeofDuhDa0ucjoJZrtLt0o6GTgWC1ukomGFOrNks4doFAADAOaN8wDkb9Ad035Y6rZ2XqUW5KU7HwSxXWexS76Bf+5u7nY6CGaZ2dM1mKdcuAAAAzhnlA87ZH3c2qrVnUHeuLXU6CqAKt0uSuHqBSedp9Sk9MUZpCTFORwEAAJi2KB9wTqy1+tFmjxbmJGvtvEyn4wDKT4vXnJRYVdVRPmByebw+Tj0AAACcJ8oHnJO/HGzVweO9+tDaUhnD3ns4zxijSnc6Jx8w6TytPjZdAAAAnCfKB5yTH22u1ZyUWN20PM/pKMCYCrdLxzr71dTV73QUzBDdA8Py9g6qNItNFwAAAOeD8gFnbW9jl1445NUda4oVE8WHEMJHZTFzHzC5ak9suuDkAwAAwHnhO0ectR9vrlVCTKRuu8jtdBTgdRblpig+OpK5D5g0td6R8qGMNZsAAADnhfIBZ6Wpq1+P7GrUzZWFSk2IdjoO8DrRkRG6oDCNkw+YNJ7WXkUYqSgjwekoAAAA01pIywdjzDpjzAFjzCFjzBfHef37jTGtxpido78+FMo8OH/3balT0Fp98NISp6MA46osdmlfU7d8g36no2AG8Hh9KnAlKDYq0ukoAAAA01rIygdjTKSkuyXdIGmxpFuNMYvHeeivrLUXjP76Uajy4Pz1Dvr1i5cbdMOyXBWm81NAhKcKt0uBoNWuI51OR8EMwKYLAACAyRHKkw8XSTpkrfVYa4ckPSTprSF8fwixX207op4Bv+5cW+p0FOC0VhS5ZIxUxdULnCdrrWq9PpUy7wEAAOC8hbJ8yJd05KTnj46+7FTvMMbUGGN+Y4wpDGEenAd/IKifvFCri4rTdUFhmtNxgNNKjY/W/Oxkygect+buAfUPB1TKyQcAAIDzFsrywYzzMnvK8xslFVtryyU9Len+cd+QMR82xlQZY6paW1snOSYm4rE9zTrW2a8PrWXWA8JfRbFLO+o7FAie+kcOMHEn1myWZiU5nAQAAGD6C2X5cFTSyScZCiQ1nvwAa22btXZw9Nl7JVWM94astfdYayuttZVZWVkhCYvTs9bq3s0elWQm6ppFc5yOA5xRpdulnkG/Dh7vcToKprHDo2s2mfkAAABw/kJZPmyTNM8YU2KMiZF0i6RHTn6AMSb3pGdvkvRqCPPgHL1S266ao1364KUliogY70ALEF4q3emSxMpNnJfaVp/ioyOVkxLndBQAAIBpL2Tlg7XWL+nvJD2hkVLhYWvtXmPM14wxN40+7FPGmL3GmF2SPiXp/aHKg3N372aP0hNj9I6VBU5HASakMD1eWcmxlA84Lx5vr4ozEyldAQAAJkFUKN+4tfZRSY+e8rIvn/T0lyR9KZQZcH4Ot/bq6Vdb9Kmr5yk+hj33mB6MMap0u1RV3+50FExjtV6fluanOh0DAABgRgjltQvMAD/aXKuYqAi9b7Xb6SjAWalwu3SkvV8t3QNOR8E0NOgP6Eh7H5suAAAAJgnlA07L2zuo320/qneszFdmUqzTcYCzUlk8MveBlZs4F0fa+xS0UmkW5QMAAMBkoHzAaf18a70G/UF98NJSp6MAZ21JXorioiNUVUf5gLN3uPXEpgvWbAIAAEwGygeMa2A4oJ+/VK+rF2ZrbjZffGP6iY6MUHlBmqqZ+4BzUMuaTQAAgElF+YBx/Xb7UbX7hnTnZZx6wPRV6XZpb2O3+ocCTkfBNONp7VVmUoxS46OdjgIAADAjUD7gDYJBqx9vrtWy/FRdXJLudBzgnFUWu+QPWu080ul0FEwztV6fSrlyAQAAMGkoH/AGz+xvkcfr052XlcoY9ttj+lpZ5JIkbW9g7gPOjqfVx5ULAACASUT5gDe4d7NH+WnxesvSHKejAOclLSFG87KTVFXH3AdMXFffsNp8Q2y6AAAAmESUD3idXUc69Uptuz5wSbGiIvnwwPRXWexSdX2HgkHrdBRMEx5vrySGTQIAAEwmvrvE69y72aPk2Ci9+8JCp6MAk6LCna7uAb8OtfY6HQXTxIlNF6VZzHwAAACYLJQPGHOkvU+P7m7Sey4uUnIcE94xM1S6R+Y+VNUx9wET42n1KTLCqCg9wekoAAAAMwblA8b89MU6RRij919S7HQUYNK4MxKUmRSjqnrmPmBiar0+FbriFRPFX5EAAACTha+sIEnq6h/Wr7Y1aMPyPOWmxjsdB5g0xhitLBqZ+wBMxOHWXuY9AAAATDLKB0iSfvlKg3xDAX1obYnTUYBJV1nsUn1bn1p7Bp2OgjAXDFrVtfmY9wAAADDJKB+gIX9QP32xVpfMzdCSvFSn4wCTrsKdLkmq5uoFzqCpe0ADw0FOPgAAAEwyygdoU02jjncP6kNrS52OAoTE0vwUxURFcPUCZ1TbemLTBeUDAADAZKJ8mOWstbrneY/mZSfpivlZTscBQiI2KlLLC1JVRfmAM/B4R1aylmZy7QIAAGAyUT7Mci8eatP+5h7dubZUxhin4wAhU+FO155jXRoYDjgdBWHM0+pTQkyk5qTEOh0FAABgRqF8mOXu2exRZlKs3roiz+koQEhVul0aDljVHO1yOgrCmMfrU0lmImUsAADAJKN8mMUONPfo+YOtev8at2KjIp2OA4TUSrdLklTF0Em8iVpvL5suAAAAQoDyYRa7d7NH8dGRuu1it9NRgJBLT4xRaVaiquuY+4DxDQwHdLSjn00XAAAAIUD5MEu1dA/ojzuP6V2VBXIlxjgdB5gSlW6Xqhs6FAxap6MgDDW098laqYxNFwAAAJOO8mGWum9LnfxBqw9eWuJ0FGDKVLrT1dk3PLbRADiZp3Xk44KTDwAAAJOP8mEW8g369eDLDbp+cY7cGXyRjdmjonh07gNXLzAOj9cnifIBAAAgFCgfZqFfVx1RV/+w7rys1OkowJQqzUxUemKMquspH/BGnlafspJjlRwX7XQUAACAGYfyYZYJBK1+/GKtVhalqWJ0+j8wWxhjtLLIRfmAcdV6fSrl1AMAAEBIUD7MMk/sbdaR9n59mFMPmKUqi13yeH1q6x10OgrCjKeVNZsAAAChQvkwi1hrdc/zHrkzEnTt4hyn4wCOqBw98cPpB5yswzekjr5hTj4AAACEyBnLB2PMt4wxS6YiDEKrur5DO4906oOXligywjgdB3DE0vxUxURGUD7gdU4MmyxlzSYAAEBITOTkw35J9xhjXjbGfNQYkxrqUAiNezd7lBofrXdWFDgdBXBMXHSkluanqIryASepZdMFAABASJ2xfLDW/shae4mk90kqllRjjPmFMebKUIfD5Kn1+vTkvuN67yq3EmKinI4DOKqyOF27j3ZpYDjgdBSECU9rr6IijArTE5yOAgAAMCNNaOaDMSZS0sLRX15JuyR9zhjzUAizYRL95IVaRUdE6H1r3E5HARxX4XZpKBDUnmNdTkdBmKj1+lSUnqDoSEYhAQAAhMJEZj58R9IBSW+R9A1rbYW19r+stRskrQh1QJy/Dt+Qfl19RG9bkafs5Din4wCOO7FmlqsXOMHT6mPeAwAAQAhN5Pz9Hkn/Yq3tG+d1F01yHoTAAy/Va2A4qA+tZb0mIEmZSbEqyUxk6CQkScGgVW2bT5fNz3Q6CgAAwIw1kfOlHZKiTzxjjEkzxrxNkqy1nFkOcwPDAd2/tU5XLMjS/DnJTscBwkaF26Xt9R2y1jodBQ471tmvIX9QpVlJTkcBAACYsSZSPnzl5JLBWtsp6Suhi4TJ9Medx+TtHdKdnHoAXqfS7VKbb2hsywFmLzZdAAAAhN5EyofxHsO6hGkgGLS6d3OtFuemaE1ZhtNxgLBSWczcB4zwtPZKEjMfAAAAQmgi5UOVMeY7xpgyY0ypMea/JVWHOhjO318OtupQS6/uvKxExhin4wBhpTQzSWkJ0aquo3yY7Wq9PiXFRikrKdbpKAAAADPWRMqHT0oakvQrSb+WNCDpE6EMhclxz/Me5aTEaX15ntNRgLATEWG0ssilqvp2p6PAYR7vyKYLSloAAIDQOeP1CWutT9IXpyALJtGeY13a6mnTl25YyN564DQq3C49u79FHb4huRJjnI4Dh3hafWPXcAAAABAaZywfjDFZkr4gaYmkuBMvt9ZeFcJcOE/3bvYoKTZKt15c5HQUIGxVuke+4ayu79A1i+c4nAZOGBgOqLGrX6WZhU5HAQAAmNEm8iPxByXtl1Qi6d8k1UnaFsJMOE+Nnf3aVNOkd19YqJS46DP/C8AstbwwTdGRhqGTs1hdm0/WSiUMmwQAAAipiZQPGdbaH0sattb+xVr7t5JWhTgXzsNPX6yVJH3gkmJngwBhLi46UkvyUrWd8mHW+n/t3Xmc3HWd5/H3p+8rR3W6cyfV3SEcIcjRLQIaFwZEECS46gysq7iiPNyHzOg47soMyjgczqgzzsyOrivgqOM4ouOKRhZFBg+QSzrIkRCOpKo76RykKndXp8/67B9dwTYkoZPUr75V1a/n45FH1/Hrqnfg90h3v/v7+3wTqfFtNjvYZhMAACBSkykfRnIft5rZZWZ2pqSFEWbCcdg7OKLv/GaT3nbaPC2MNYSOAxS9rnhMT/ft1vBoNnQUBJBMj5cP7ZQPAAAAkZpM+XCrmc2Q9GeSPiHpTkl/GmkqHLPv/maT+odG9aEV7aGjACWhqy2modGs1mzZEzoKAtiQ6tfc6XVqrH3NEUgAAAA4Dkf8bsvMKiUtdfd7JO2RdEFBUuGYjIxl9fWHkzqno1mvWzgzdBygJHTGmyVJq3t26azF7Hgw1STTGVY9AAAAFMARVz64+5ikKwqUBcfp3me3asueQX1oRUfoKEDJaJ1Wq/isBnX37gwdBQXm7kqkMupg2CQAAEDkJnPZxSNm9iUzW2FmZx34M5kXN7NLzOwFM1tvZjcc4bh3mZmbWdekk+P3uLvueCihJa2NuuCk2aHjACWlc3FMq3t3yd1DR0EB7RoY0Z79I6x8AAAAKIDJXOR6Xu7jzRMec0l/cKRPyl2y8WVJb5HUJ+kJM1vl7s8ddNw0SX8i6fHJhsarPZrYoTWb9+qv//Npqqiw0HGAktLZFtMPfrtZvTsG1MYPolNGItUvSVrS2hQ4CQAAQPl7rZkPFZK+4u7fO4bXPlvSendP5F7rLkkrJT130HG3SPq8xodZ4hjd+VBSLU01eseZC0JHAUpOV27uQ3fvLsqHKSTBThcAAAAF81ozH7KSrj/G114gadOE+325x16R27ZzUW6g5WGZ2XVm1m1m3alU6hjjlK/12/fp589v13vPaVNddWXoRf4wCgAAIABJREFUOEDJWTq7SdPrqrSauQ9TSiKVUXWlaWGsPnQUAACAsjeZmQ/3m9knzGyRmTUf+DOJzzvU2v9XLqjOrar4e41v4XlE7n67u3e5e1dra+sk3npqufOhpGqrKvTec+OhowAlqaLCdFZ8fO4Dpo5kul+LmxtUVTmZL4UAAAA4HpOZ+fCB3MePTHjMJb3Wlgp9khZNuL9Q0pYJ96dJWi7pl2YmSXMlrTKzK9y9exK5ICm1b0g/eHKz3t21UM2NNaHjACWrKx7T376Q0p6BEc1oqA4dBwUwvtMF8x4AAAAK4TV/3ePu7Yf4M5m9HJ+QtNTM2s2sRtJVklZNeN097t7i7m3u3ibpMUkUD0fpW4/2aCSb1bVvag8dBShpnbm5D09uZPXDVDCWdfXuGFAH8x4AAAAK4jVXPpjZ+w71uLv/y5E+z91Hzex6SfdJqpT0z+6+1sxultTt7quO9Pl4bfuHx/Stx3p10Slz+O0dcJzOWDRTVRWm7t6duuBktqstd5t37dfwWFYdrZQPAAAAhTCZyy5eP+F2naQLJT0p6YjlgyS5+72S7j3osZsOc+z5k8iCCb7/ZJ92DYzoujdPZiEKgCOpr6nUqfOnq7uHlQ9TQSI9vs1mewvFLQAAQCG8Zvng7n888b6ZzZD0rcgSYVLGsq6vPZTQ6YtmqiseCx0HKAtnxWP6zm82amQsq2qGEJa1RGp8m01WPgAAABTGsXx3PSBpab6D4Oj8x7qX1bNjQNet6FBuYCeA49QVb9bgSFZrt+wNHQURS6YzmlZXpVkM6gUAACiIycx8+LF+t0VmhaRlkr4XZSi8tjseTGhhrF5vPXVO6ChA2ehqG19F1N2zU2csmhk4DaKUSPero7WJ8hYAAKBAJjPz4W8n3B6V1OvufRHlwSQ8uXGXunt36S/fvoz96YE8mjO9Tgtj9Vrdu0sfXBE6DaKUTGX0ho5ZoWMAAABMGZMpHzZK2urug5JkZvVm1ubuPZEmw2Hd+VBC0+uq9Iddi0JHAcpOVzymhzfskLvzW/EyNTA8qi17BtlmEwAAoIAm82vzf5eUnXB/LPcYAti4Y0A/XbNN7zknrsbayXRHAI5GZ1uzUvuG1Ldrf+goiEhPekCS1M6wSQAAgIKZTPlQ5e7DB+7kbjOhK5B/fjipygrT+89rCx0FKEsHdo/p7t0ZOAmicmCbzQ622QQAACiYyZQPKTO74sAdM1spKR1dJBzO7oFhfa97k644fYHmTK8LHQcoSyfOmaZptVXq7tkVOgoiksxts9nW0hA4CQAAwNQxmXX7H5b0bTP7Uu5+n6T3RRcJh/PtxzdqYHhMH1zRHjoKULYqK0xnLJ6p1b2UD+Uqkc5o/ow6NdRw6RoAAEChvObKB3ff4O7naHyLzVPd/Tx3Xx99NEw0NDqmbz7SoxVLW3TKvOmh4wBlrSverBde3qc9+0dCR0EEEukM8x4AAAAK7DXLBzP7rJnNdPd+d99nZjEzu7UQ4fA7q57aou37hvShFR2howBlr6stJnfptxtZ/VBu3F2JVD/zHgAAAApsMjMfLnX33QfuuPsuSW+LLhIO5u6686GkTp47TSuWtoSOA5S9MxbNVGWFcelFGdqRGda+wVG1s80mAABAQU2mfKg0s9oDd8ysXlLtEY5Hnj34UlovvLxPH1zRITMLHQcoe421VTpl3jSGTpahRG7YZAeXXQAAABTUZMqHf5X0gJlda2bXSrpf0jejjYWJ7nwooTnTa3XF6fNDRwGmjK54s57atFujY9nQUZBHSbbZBAAACGIyAyc/L+lWSadofOjkTyXFI86FnOe27NVDL6V1zXltqqmaTFcEIB864zHtHxnTuq37QkdBHiVSGdVUVmhBrD50FAAAgCllsj/NbpOUlfROSRdKWhdZIvyeO3+dUENNpd5zNn0PUEhdbTFJUnfvzsBJkE+JdEbxWQ2qrOASNgAAgEI6bPlgZiea2U1mtk7SlyRtkmTufoG7f6lgCaewbXsGteqpLfrDrkWa0VAdOg4wpcybUa8FM+vVzdDJspJI9TPvAQAAIIAjrXx4XuOrHN7u7m9y93+SNFaYWJCkbzzSo6y7rn1Te+gowJR0Vjym1T275O6hoyAPRsey2rhzQO3MewAAACi4I5UP79T45Ra/MLM7zOxCSaxTLZD+oVF9+/FeXbp8nhY1N4SOA0xJXfGYtu0d1Obd+0NHQR707dqvkTFn5QMAAEAAhy0f3P1ud/8jSSdL+qWkP5U0x8y+YmYXFyjflPW9JzZp3+CoPriCVQ9AKJ3x8bkPq7n0oiwk07ltNlsoHwAAAAptMrtdZNz92+5+uaSFkp6SdEPkyaaw0bGsvvbrpF7fFtOZi2Oh4wBT1slzp6mxplLdPZQP5WBDKrfNZiuXXQAAABTaUe3d6O473f2r7v4HUQWC9NO127R59359aEVH6CjAlFZVWaEzF8cYOlkmkumMZtRXK8YAXwAAgII7qvIB0XN33fFgQu0tjbrolDmh4wBTXmc8phe27dW+wZHQUXCcEqmMOlobZcb4IgAAgEKjfCgyT/Ts0tN9e3Ttm9pVwT70QHBdbTFlXXpq0+7QUXCckumM2pn3AAAAEATlQ5G5/cGEYg3VeudZC0NHASDpzMUxVZiY+1DiMkOj2rZ3UEuY9wAAABAE5UMR2ZDq1wPPv6z3ntum+prK0HEASGqqrdLJc6ez40WJO7DTBSsfAAAAwqB8KCJf+3VS1ZUVet+58dBRAEzQGY/ptxt3aXQsGzoKjlHiwDabrZQPAAAAIVA+FIkd/UP6v6v79M6zFqilqTZ0HAATdLXFlBke0/Pb9oWOgmOUTGVkJrXNonwAAAAIgfKhSHzrsV4NjWZ17ZvYXhMoNp3xmCRx6UUJS6T7NX9GveqquaQNAAAgBMqHIjA4MqZvPdqrC0+erRNmMwwNKDYLZtZr7vQ6dVM+lKxkOsMlFwAAAAFRPhSBHzy5WTsyw/rgClY9AMXIzNTZFtPqnp2ho+AYuLsSqYw6GDYJAAAQDOVDYNms685fJ3Taghk6p6M5dBwAh9EVj2nLnkFt2b0/dBQcpVT/kPqHRtnpAgAAICDKh8B+/vx2JVIZfXBFu8wsdBwAh9EVHy8HmftQehKpAztdcFkbAABAKJQPgd3xUEILZtbrbafNCx0FwBGcMm+aGmoqKR9KUDK3zSYrHwAAAMKhfAjomb7dejy5U//tjW2qruR/BVDMqiordMaimeruZe5DqUmk+lVTVaEFM+tDRwEAAJiy+Ik3oDseSmpabZX+6PWLQkcBMAmd8ZjWbd2nzNBo6Cg4Csl0Ru2zGlVRwaVtAAAAoVA+BNK3a0D3PrtVV79hsabVVYeOA2ASOuMxjWVdT23aHToKjkIixTabAAAAoVE+BPL1h3tkkt5/XlvoKAAm6ax4TGZSdw9zH0rFyFhWG3cOMO8BAAAgMMqHAPbsH9Fdv9moy183T/O5BhkoGdPrqnXSnGnMfSghm3YOaDTr7HQBAAAQGOVDAHf9ZqMyw2P64IqO0FEAHKXOeEy/3bhbY1kPHQWTwE4XAAAAxYHyocCGR7P6+sM9Om/JLC1fMCN0HABHqastpv6hUb348r7QUTAJidR4+bCEmQ8AAABBUT4U2P97dou27R3Uh97MqgegFHXFmyVJ3b3MfSgFiXRGsYZqzWyoCR0FAABgSqN8KCB31+0PJrV0dpPOP7E1dBwAx2BhrF6zp9VqdQ9zH0pBItXPvAcAAIAiQPlQQI9s2KF1W/fqQys6ZMZ+80ApMjN1xmOsfCgRyXSGeQ8AAABFgPKhgG5/MKGWplqtPHN+6CgAjkNnPKa+Xfv18t7B0FFwBPsGR7R935A6mPcAAAAQHOVDgbywbZ9+9WJK7z8vrtqqytBxAByHrrbc3IceVj8Us570gCSpg5UPAAAAwUVaPpjZJWb2gpmtN7MbDvH8h83sWTN7ysx+bWbLoswT0p0PJVRfXan3vCEeOgqA43Tq/Omqq65Qdy9zH4pZIt0vScx8AAAAKAKRlQ9mVinpy5IulbRM0tWHKBf+zd1Pc/czJH1e0hejyhPS9r2D+tFTW/TuroWKNTJxHSh11ZUVOn3hTK1m7kNRS6QyMpMWNzeEjgIAADDlRbny4WxJ69094e7Dku6StHLiAe6+d8LdRkkeYZ5gvvloj0ayWV37pvbQUQDkSVdbTGu37NXA8GjoKDiMRDqjhbF61VVzqRsAAEBoUZYPCyRtmnC/L/fY7zGzj5jZBo2vfPiTCPMEMTA8qn99bKPeumyu4rO47hgoF13xZo1lXU9v2hM6Cg4jkepXewuXXAAAABSDKMuHQ+0l+aqVDe7+ZXdfIumTkj51yBcyu87Mus2sO5VK5TlmtLIufeCN7frw+UtCRwGQR2ctjkmSVjP3oSi5u5LpDMMmAQAAikSU5UOfpEUT7i+UtOUIx98l6cpDPeHut7t7l7t3tba25jFi9Jpqq/TRi5bqjEUzQ0cBkEczGqp14pwmdTP3oSi9vHdIA8NjbLMJAABQJKIsH56QtNTM2s2sRtJVklZNPMDMlk64e5mklyLMAwB51RmP6cneXcpmy3JcTUl7ZacLLrsAAAAoCpGVD+4+Kul6SfdJWifpe+6+1sxuNrMrcoddb2ZrzewpSR+XdE1UeQAg3zrjzdo7OKqXtveHjoKDJFIZSVI7Kx8AAACKQlWUL+7u90q696DHbppw+6NRvj8ARKkrPj73obt3p06aOy1wGkyUTGdUV12hedPrQkcBAACAor3sAgDKWnxWg1qaarS6h7kPxSaR6lfbrEZVVBxq9jEAAAAKjfIBAI6RmakzHmPoZBFKpjNa0sq8BwAAgGJB+QAAx6Er3qyNOwe0fd9g6CjIGR7NatOu/Wpnm00AAICiQfkAAMehs2187sOTrH4oGht3Dmgs62yzCQAAUEQoHwDgOCyfP0O1VRXqZu5D0UikxncfYeUDAABA8aB8AIDjUFNVodMXzmTuQxFJpse32exoYeYDAABAsaB8AIDjdFY8prVb9mhwZCx0FEhKpDKa1VijGQ3VoaMAAAAgh/IBAI5TVzymkTHX05t2h44Cja98YN4DAABAcaF8AIDj1BkfHzrJpRfFIZHuZ94DAABAkaF8AIDjFGus0ZLWRq2mfAhuz/4RpfuH1dHKvAcAAIBiQvkAAHnQFW/W6t5dymY9dJQp7cCwSVY+AAAAFBfKBwDIg862mPbsH9GG3DaPCCOZHv/vv4SZDwAAAEWF8gEA8qArN/eBSy/CSqQyqjBpUXND6CgAAACYgPIBAPKgvaVRsxprGDoZWCKd0aLmBtVWVYaOAgAAgAkoHwAgD8xMZ8VjrHwILJHKMO8BAACgCFE+AECedMZjSqYzSvcPhY4yJWWzrp50Rh0t7HQBAABQbCgfACBPmPsQ1ra9g9o/MqZ2hk0CAAAUHcoHAMiT5QtmqKaygvIhkAPbbC7hsgsAAICiQ/kAAHlSV12p0xbOUHfPztBRpqREbptTVj4AAAAUH8oHAMijrnhMazbv1eDIWOgoU04inVF9daXmTq8LHQUAAAAHoXwAgDzqjMc0PJbVs5v3hI4y5RzY6cLMQkcBAADAQSgfACCPOhk6GUwynVEHl1wAAAAUJcoHAMijWU216mhpVHcP5UMhDY2OqW/XgDoYNgkAAFCUKB8AIM864zE9uXGX3D10lClj444BZV3qaG0KHQUAAACHQPkAAHnWGY9pZ2ZYidzWj4jehtT4f2suuwAAAChOlA8AkGddbbm5D1x6UTDJXNHTzmUXAAAARYnyAQDyrKOlSTMbqtXduzN0lCkjkepX67RaTaurDh0FAAAAh0D5AAB5VlFh6lwcUzc7XhRMMp1h1QMAAEARo3wAgAh0tsWUSGW0MzMcOsqUkEhntIR5DwAAAEWL8gEAItAVb5YkrWb1Q+R2DwxrZ2aYlQ8AAABFjPIBACLwuoUzVF1plA8FcGBXkY4WttkEAAAoVpQPABCBuupKLV8wQ6sZOhm5ZG6bzXYuuwAAAChalA8AEJHOxTE93bdHQ6NjoaOUtUS6X5UVpsXNDaGjAAAA4DAoHwAgIl1tMQ2PZrVm897QUcpaMp3R4uYGVVfyJQ0AAKBY8Z0aAESk85Whk1x6EaVEKqMOhk0CAAAUNcoHAIhI67RaxWc1qLuHoZNRyWZdyXSGnS4AAACKHOUDAESoMx7T6t5dcvfQUcrSlj37NTSaVUcrO10AAAAUM8oHAIhQV7xZOzLD6tkxEDpKWUrmttlk5QMAAEBxo3wAgAh1tcUkSd09zH2IQiK3zeYSttkEAAAoapQPABChE1qbNL2uSk9uZO5DFJLpjBprKtU6rTZ0FAAAABwB5QMARKiiwtQZjzF0MiIbUv3qaG2SmYWOAgAAgCOgfACAiHXGY3ppe792DwyHjlJ22OkCAACgNFA+AEDEOuPNksSlF3k2ODKmzbv3q4N5DwAAAEWP8gEAInbGopmqqjAuvciz3h0DcmenCwAAgFJA+QAAEauvqdSp86eru5fyIZ8SqX5J0pLWpsBJAAAA8FooHwCgADrjzXp6024Nj2ZDRykbifT4NpttrHwAAAAoepQPAFAAXW0xDY1mtXbLntBRykYildGc6bVqqq0KHQUAAACvIdLywcwuMbMXzGy9md1wiOc/bmbPmdkzZvaAmcWjzAMAoXTFY5Kk1Vx6kTfJdD/zHgAAAEpEZOWDmVVK+rKkSyUtk3S1mS076LDfSupy99dJ+r6kz0eVBwBCmj29Toua6ykf8iiRzqiDeQ8AAAAlIcqVD2dLWu/uCXcflnSXpJUTD3D3X7j7QO7uY5IWRpgHAILqijeru3eX3D10lJK3KzOs3QMj6mDlAwAAQEmIsnxYIGnThPt9uccO51pJPznUE2Z2nZl1m1l3KpXKY0QAKJyz4jGl9g1p0879oaOUvER6fKeLjlbKBwAAgFIQZflgh3jskL/uM7P/KqlL0hcO9by73+7uXe7e1dramseIAFA4B+Y+dPfuDJyk9CVS4ztdtLdw2QUAAEApiLJ86JO0aML9hZK2HHyQmV0k6UZJV7j7UIR5ACCoE+dM07TaKnUz9+G4JdIZVVWYFsXqQ0cBAADAJERZPjwhaamZtZtZjaSrJK2aeICZnSnpqxovHrZHmAUAgqusMJ0Zj2l1D+XD8UqmMlo8q0FVlewYDQAAUAoi+67N3UclXS/pPknrJH3P3dea2c1mdkXusC9IapL072b2lJmtOszLAUBZ6IrH9OL2fdqzfyR0lJKWSPerg0suAAAASkZVlC/u7vdKuvegx26acPuiKN8fAIpNVzwmd+nxxA5dfOrc0HFK0ljW1bNjQOefNDt0FAAAAEwS61UBoIDOise0YGa9/vZnL2hkLBs6Tknasnu/hkezbLMJAABQQigfAKCA6qor9ZkrTtWLL/fra79Oho5TkhLpAztdUD4AAACUCsoHACiwtyybo7csm6N/+I8XtWnnQOg4JSeR6pckdbQy8wEAAKBUUD4AQACfueJUVZjpM6vWyt1DxykpyXRG02qr1NJUEzoKAAAAJonyAQACWDCzXn960Yl64Pntum/ty6HjlJREKqOO1kaZWegoAAAAmCTKBwAI5P1vbNPJc6fpr368Vv1Do6HjlIxkOsO8BwAAgBJD+QAAgVRXVui2d5ymrXsG9Q/3vxg6TknYPzymzbv3M+8BAACgxFA+AEBAnfGYrj57sb7+SI/WbtkTOk7R69nBThcAAACliPIBAAL75CUnaWZ9tW68e42yWYZPHkkiNV4+dLRSPgAAAJQSygcACGxmQ41uvOwUPbVpt77zxMbQcYpaMj2+zSYrHwAAAEoL5QMAFIF3nLlA53Q063M/eV6pfUOh4xStRCqjeTPq1FBTFToKAAAAjgLlAwAUATPTrVeepv0jY/rsvetCxylaCXa6AAAAKEmUDwBQJE6Y3aQP/6cluvu3m/XI+nToOEXH3ZVI9TPvAQAAoARRPgBAEfnIBSdocXODPvXDNRoaHQsdp6jszAxr7+Co2lvYZhMAAKDUUD4AQBGpq67UzStPVSKd0Vd/lQgdp6gk0ux0AQAAUKooHwCgyJx/0mxd9rp5+tIv1qsn9wM3pOSBbTaZ+QAAAFByKB8AoAjddPky1VRW6NM/WiN3Dx2nKGxI96u60rQw1hA6CgAAAI4S5QMAFKE50+v0iYtP1EMvpXXPM1tDxykKyVRG8VmNqqyw0FEAAABwlCgfAKBIvffcNp22YIZuvuc57R0cCR0nuEQ6wyUXAAAAJYryAQCKVGWF6bPvOE07+of0d/e9EDpOUGNZV++OjNoZNgkAAFCSKB8AoIidtnCG3ndum/7lsV49vWl36DjB9O0a0MiYawnbbAIAAJQkygcAKHIfv/hEtTbV6i/uflajY9nQcYI4sM0mKx8AAABKE+UDABS56XXVuunty7R2y15967He0HGCSLDNJgAAQEmjfACAEnDZafP05hNb9Xc/e1Hb9gyGjlNwyXS/ptdVqbmxJnQUAAAAHAPKBwAoAWamW1aequGxrG6557nQcQoukcqoo7VJZmyzCQAAUIooHwCgRMRnNeqPLzhB/+/ZrfrFC9tDxymoJNtsAgAAlDTKBwAoIdf9pw51tDbqph+t0eDIWOg4BTEwPKqtewbVwbBJAACAkkX5AAAlpLaqUrdeuVybdu7Xl36+PnScgkge2OmCbTYBAABKFuUDAJSY85a06D+fuUBffXCD1m/fFzpO5F7Z6YKVDwAAACWL8gEAStBfXHaK6qsrdePda+TuoeNE6sDKh7ZZlA8AAAClivIBAEpQS1Otbrj0FD2e3KkfPLk5dJxIJVL9WjCzXvU1laGjAAAA4BhRPgBAibrq9Yt05uKZuu3eddo9MBw6TmSS6Yza2ekCAACgpFE+AECJqqgw3Xbladqzf0Sf++nzoeNEwt2VSGWY9wAAAFDiKB8AoIQtmz9dH3hjm77zm01a3bszdJy8S/cPa9/QKCsfAAAAShzlAwCUuI9ddKLmz6jTjXev0chYNnScvEqk+iVJHa1sswkAAFDKKB8AoMQ11lbpM1ecque37dPXH06GjpNXB3a66GDlAwAAQEmjfACAMnDxqXN10Slz9Pf3v6TNu/eHjpM3iXRGNVUVmj+zPnQUAAAAHAfKBwAoE5+5Ytn4x1VrAyfJn0Qqo7ZZDaqssNBRAAAAcBwoHwCgTCyMNehjFy3V/c+9rJ+t3RY6Tl4k0v3qaGHeAwAAQKmjfACAMvKBN7XrpDnT9JlVa5UZGg0d57iMjmW1cceA2tlmEwAAoORRPgBAGamurNBt71iuLXsG9Y8PvBQ6znHZtGu/RrPOsEkAAIAyQPkAAGWmq61ZV71+kb7266TWbd0bOs4xS6YPbLNJ+QAAAFDqKB8AoAx98pKTNaO+Wjfe/ayyWQ8d55gkUge22WTmAwAAQKmjfACAMhRrrNFfvO0UPblxt77bvSl0nGOSSGc0s6Fascaa0FEAAABwnCgfAKBMvfOsBXpDe7P+5ifPK90/FDrOUUuk+pn3AAAAUCYoHwCgTJmZbnvHcg0Mj+qz964LHeeoJdMZtXPJBQAAQFmgfACAMnbC7Gm67s0d+sGTm/XIhnToOJPWPzSql/cOMWwSAACgTERaPpjZJWb2gpmtN7MbDvH8m83sSTMbNbN3RZkFAKaq6y9YqkXN9frUD9doaHQsdJxJ6UkfGDZJ+QAAAFAOIisfzKxS0pclXSppmaSrzWzZQYdtlPR+Sf8WVQ4AmOrqayp188rlSqQyuuPBROg4k7IhdWCbTS67AAAAKAdRrnw4W9J6d0+4+7CkuyStnHiAu/e4+zOSshHmAIAp74KTZuuy0+bpn36+Xr07MqHjvKZkOiMzKT6rIXQUAAAA5EGU5cMCSRP3d+vLPXbUzOw6M+s2s+5UKpWXcAAw1Xz68mWqrqzQTT9aK3cPHeeIEqmMFsysV111ZegoAAAAyIMoywc7xGPH9N2uu9/u7l3u3tXa2nqcsQBgapo7o05/dvGJ+tWLKd377LbQcY5ofKcL5j0AAACUiyjLhz5JiybcXyhpS4TvBwB4De89J67lC6brr368VvsGR0LHOSR3VyLVryXMewAAACgbUZYPT0haambtZlYj6SpJqyJ8PwDAa6iqrNBtV56mVP+Q/u5nL4aOc0ipfUPKDI+x8gEAAKCMRFY+uPuopOsl3SdpnaTvuftaM7vZzK6QJDN7vZn1SXq3pK+a2dqo8gAAxp2+aKbee05c//Joj57t2xM6zqtsSOW22WylfAAAACgXUa58kLvf6+4nuvsSd78t99hN7r4qd/sJd1/o7o3uPsvdT40yDwBg3CfeepJmNdXqxh8+q7FscQ2fTKbHywdWPgAAAJSPSMsHAEBxml5XrU9fvkzP9O3Rvz7WGzrO70mk+lVbVaH5M+pDRwEAAECeUD4AwBT19tfN04qlLfrCfS/o5b2DoeO84sBOFxUVh9o0CQAAAKWI8gEApigz080rl2t4LKtb7nkudJxXJNIZ5j0AAACUGcoHAJjC2lsa9ZHzT9A9z2zVr15MhY6jkbGsNu4cYN4DAABAmaF8AIAp7sPnd6ijpVE3/WiNBkfGgmbZuHNAY1lXR0tT0BwAAADIL8oHAJjiaqsqdcuVy9W7Y0D/+xfrg2ZJ5rbZbOeyCwAAgLJC+QAA0BtPaNGVZ8zXV361Qeu39wfLkUiPv3cHl10AAACUFcoHAIAk6cbLlqm+ulKf/uEauXuQDMl0Rs2NNZrZUBPk/QEAABANygcAgCSpdVqtPnnpyXo0sUM/fGpzkAwbUhlWPQAAAJQhygcAwCuufv1inbl4pm69Z532DIwU/P2T6Qw7XQAAAJQhygcAwCsqKky3XXmadu8f0efue76g771vcESpfUPqaGWnCwAAgHJD+QAA+D3L5k/XfzuvTf/2+Eat7t1VsPdNpnM+JEDAAAAODUlEQVQ7XbDyAQAAoOxQPgAAXuVjbzlR82bU6ca7n9XoWLYg75nIbbO5hG02AQAAyg7lAwDgVZpqq/SXbz9Vz2/bp2880lOQ90ykM6owafGshoK8HwAAAAqH8gEAcEhvPXWOLjx5tr54/4vasnt/5O+XSPVrYaxBtVWVkb8XAAAACovyAQBwSGamz1xxqrLu+qsfr438/djpAgAAoHxRPgAADmtRc4M+euGJum/ty/qP516O7H3cXcl0Rh3MewAAAChLlA8AgCP64Ip2nTinSX+5aq0GhkcjeY+X9w5pYHhMHax8AAAAKEuUDwCAI6qurNCtV56mzbv36x8feCmS90ik+iVJHa1Nkbw+AAAAwqJ8AAC8prPbm/WHXQv1tYeSen7b3ry//ob0+DabzHwAAAAoT5QPAIBJueHSUzStrkqfunuNslnP62snUxnVV1dq7vS6vL4uAAAAigPlAwBgUpoba/TnbztF3b279O+rN+X1tRPpfrW3NKqiwvL6ugAAACgOlA8AgEl711kLdXZbs/76J89rR/9Q3l43mc6onZ0uAAAAyhblAwBg0ioqTLe+Y7n6B0f11z95Pi+vOTQ6pk07B7SEeQ8AAABli/IBAHBUTpwzTde9uUPfX92nxxI7jvv1Nu0cUNbFygcAAIAyRvkAADhqf/wHS7UwVq9P/XCNhkezx/VaG1LjO110tLDNJgAAQLmifAAAHLX6mkrdsnK51m/v1x0PJY7rtZIHttlk5QMAAEDZonwAAByTC06erUuXz9X/euAlbdwxcMyvk0j1q6WpVtPrqvOYDgAAAMWE8gEAcMxuevsyVVWYblq1Ru5+TK+RTGfUwbBJAACAskb5AAA4ZvNm1OvjF5+kX76Q0k/XbDum10ikMurgkgsAAICyRvkAADgu15wb17J50/WZH69V/9DoUX3unoER7cgMq52VDwAAAGWN8gEAcFyqKit02zuWa/u+IX3xZy8e1ecm0v2SpI5WdroAAAAoZ5QPAIDjdubimN7zhsX6xiNJrdm8Z9Kf98pOF6x8AAAAKGuUDwCAvPgfbz1ZzY01uvHuZzWWndzwyUQqo8oK0+LmhojTAQAAICTKBwBAXsyor9anL1+mp/v26N8e753U5yTTGS2K1aumii9HAAAA5Yzv9gAAeXPF6fP1xhNm6fM/fUHb9w2+5vEbUv3MewAAAJgCKB8AAHljZrpl5XINjWZ16z3rjnhsNuvq2ZFh3gMAAMAUQPkAAMirjtYm/ffzl2jV01v00Eupwx63de+gBkey6milfAAAACh3lA8AgLz77+cvUXtLoz79wzUaHBk75DHJFDtdAAAATBWUDwCAvKurrtQtK5erZ8eAvvLLDYc8JpHulyQtYeYDAABA2aN8AABE4k1LW7TyjPn6yi83KJHqf9XziVRGjTWVmj2tNkA6AAAAFBLlAwAgMjdedopqqyv06R+tkbv/3nOJdEbtrY0ys0DpAAAAUCiUDwCAyMyeVqf/ecnJenj9Dq16esvvPZdM96u9hUsuAAAApgLKBwBApP7L2Yt1+qKZuuWe57RnYESSNDgypr5d+9XBsEkAAIApgfIBABCpygrTbVcu187MsL7ws+clSRt3DshdbLMJAAAwRVA+AAAit3zBDL3/vHZ9+/GN+u3GXa8MoOzgsgsAAIApIdLywcwuMbMXzGy9md1wiOdrzey7uecfN7O2KPMAAML5+MUnas60Ot149xq99PJ4+dDW0hA4FQAAAAohsvLBzColfVnSpZKWSbrazJYddNi1kna5+wmS/l7S56LKAwAIq6m2Sn/59mV6bute3fFQQrOn1WpaXXXoWAAAACiAKFc+nC1pvbsn3H1Y0l2SVh50zEpJ38zd/r6kC4091wCgbF2yfK4uOKlVewdH1c6wSQAAgCkjyvJhgaRNE+735R475DHuPippj6RZEWYCAARkZrp55XLVVVfo5LnTQscBAABAgVRF+NqHWsHgx3CMzOw6SddJ0uLFi48/GQAgmEXNDbr3T1ZoVlNt6CgAAAAokChXPvRJWjTh/kJJWw53jJlVSZohaefBL+Tut7t7l7t3tba2RhQXAFAoHa1NmlHPvAcAAICpIsry4QlJS82s3cxqJF0ladVBx6ySdE3u9rsk/dzdX7XyAQAAAAAAlK7ILrtw91Ezu17SfZIqJf2zu681s5sldbv7Kklfk/QtM1uv8RUPV0WVBwAAAAAAhBHlzAe5+72S7j3osZsm3B6U9O4oMwAAAAAAgLCivOwCAAAAAACA8gEAAAAAAESL8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAESK8gEAAAAAAETK3D10hqNiZilJvaFzHIMWSenQIRAc5wEO4FyAxHmAcZwHOIBzARLnAX6nFM+FuLu3HuqJkisfSpWZdbt7V+gcCIvzAAdwLkDiPMA4zgMcwLkAifMAv1Nu5wKXXQAAAAAAgEhRPgAAAAAAgEhRPhTO7aEDoChwHuAAzgVInAcYx3mAAzgXIHEe4HfK6lxg5gMAAAAAAIgUKx8AAAAAAECkKB8iZmaXmNkLZrbezG4InQdhmNkiM/uFma0zs7Vm9tHQmRCOmVWa2W/N7J7QWRCOmc00s++b2fO5fxvODZ0JhWdmf5r7urDGzL5jZnWhM6EwzOyfzWy7ma2Z8Fizmd1vZi/lPsZCZkT0DnMefCH3teEZM7vbzGaGzIjCONS5MOG5T5iZm1lLiGz5QvkQITOrlPRlSZdKWibpajNbFjYVAhmV9GfufoqkcyR9hHNhSvuopHWhQyC4f5T0U3c/WdLp4pyYcsxsgaQ/kdTl7sslVUq6KmwqFNA3JF1y0GM3SHrA3ZdKeiB3H+XtG3r1eXC/pOXu/jpJL0r680KHQhDf0KvPBZnZIklvkbSx0IHyjfIhWmdLWu/uCXcflnSXpJWBMyEAd9/q7k/mbu/T+A8ZC8KmQghmtlDSZZLuDJ0F4ZjZdElvlvQ1SXL3YXffHTYVAqmSVG9mVZIaJG0JnAcF4u4PStp50MMrJX0zd/ubkq4saCgU3KHOA3f/mbuP5u4+JmlhwYOh4A7zb4Ik/b2k/ymp5Ic1Uj5Ea4GkTRPu94kfOKc8M2uTdKakx8MmQSD/oPEvINnQQRBUh6SUpK/nLsG508waQ4dCYbn7Zkl/q/HfZm2VtMfdfxY2FQKb4+5bpfFfXEiaHTgPwvuApJ+EDoEwzOwKSZvd/enQWfKB8iFadojHSr6xwrEzsyZJ/1fSx9x9b+g8KCwzu1zSdndfHToLgquSdJakr7j7mZIyYnn1lJO7nn+lpHZJ8yU1mtl/DZsKQLEwsxs1funut0NnQeGZWYOkGyXdFDpLvlA+RKtP0qIJ9xeK5ZRTlplVa7x4+La7/yB0HgTxRklXmFmPxi/D+gMz+9ewkRBIn6Q+dz+wAur7Gi8jMLVcJCnp7il3H5H0A0nnBc6EsF42s3mSlPu4PXAeBGJm10i6XNJ73J1fXk5NSzReTj+d+95xoaQnzWxu0FTHgfIhWk9IWmpm7WZWo/EhUqsCZ0IAZmYav7Z7nbt/MXQehOHuf+7uC929TeP/Hvzc3fkt5xTk7tskbTKzk3IPXSjpuYCREMZGSeeYWUPu68SFYvDoVLdK0jW529dI+lHALAjEzC6R9ElJV7j7QOg8CMPdn3X32e7elvvesU/SWbnvIUoS5UOEcoNirpd0n8a/mfieu68NmwqBvFHSezX+m+6ncn/eFjoUgKD+WNK3zewZSWdI+mzgPCiw3MqX70t6UtKzGv++7PagoVAwZvYdSY9KOsnM+szsWkl/I+ktZvaSxqfb/03IjIjeYc6DL0maJun+3PeM/ydoSBTEYc6FsmKs4gEAAAAAAFFi5QMAAAAAAIgU5QMAAAAAAIgU5QMAAAAAAIgU5QMAAAAAAIgU5QMAAAAAAIgU5QMAAMgbM7vRzNaa2TO5LeLeYGYfM7OG0NkAAEA4bLUJAADywszOlfRFSee7+5CZtUiqkfSIpC53TwcNCAAAgmHlAwAAyJd5ktLuPiRJubLhXZLmS/qFmf1CkszsYjN71MyeNLN/N7Om3OM9ZvY5M/tN7s8JucffbWZrzOxpM3swzF8NAAAcD1Y+AACAvMiVCL+W1CDpPyR9191/ZWY9yq18yK2G+IGkS909Y2aflFTr7jfnjrvD3W8zs/dJ+kN3v9zMnpV0ibtvNrOZ7r47yF8QAAAcM1Y+AACAvHD3fkmdkq6TlJL0XTN7/0GHnSNpmaSHzewpSddIik94/jsTPp6bu/2wpG+Y2YckVUaTHgAARKkqdAAAAFA+3H1M0i8l/TK3YuGagw4xSfe7+9WHe4mDb7v7h83sDZIuk/SUmZ3h7jvymxwAAESJlQ8AACAvzOwkM1s64aEzJPVK2idpWu6xxyS9ccI8hwYzO3HC5/zRhI+P5o5Z4u6Pu/tNktKSFkX41wAAABFg5QMAAMiXJkn/ZGYzJY1KWq/xSzCulvQTM9vq7hfkLsX4jpnV5j7vU5JezN2uNbPHNf4LkgOrI76QKzVM0gOSni7I3wYAAOQNAycBAEBRmDiYMnQWAACQX1x2AQAAAAAAIsXKBwAAAAAAEClWPgAAAAAAgEhRPgAAAAAAgEhRPgAAAAAAgEhRPgAAAAAAgEhRPgAAAAAAgEhRPgAAAAAAgEj9f17Dy2TntJF/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1296x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create the environment\n",
    "#env = gym.make('Taxi-v2')\n",
    "env = gym.make(\"FrozenLake-v0\")\n",
    "obs = env.reset()\n",
    "\n",
    "obs_length = env.observation_space.n\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "reward_count = 0\n",
    "games_count = 0\n",
    "\n",
    "# Create and initialize the table with 0.0\n",
    "table = collections.defaultdict(float)\n",
    "    \n",
    "test_rewards_list = []\n",
    "\n",
    "# Reinitialize epsilon after each session\n",
    "epsilon = 1.0\n",
    "\n",
    "while games_count < MAX_GAMES:\n",
    "\n",
    "    # Select the action following an ε-greedy policy\n",
    "    action = select_eps_greedy_action(table, obs, n_actions)\n",
    "    next_obs, reward, done, _ = env.step(action)\n",
    "\n",
    "    # Update the Q-table\n",
    "    Q_learning(table, obs, next_obs, reward, action)\n",
    "\n",
    "    reward_count += reward\n",
    "    obs = next_obs\n",
    "\n",
    "    if done:\n",
    "        epsilon *= EPS_DECAY_RATE\n",
    "\n",
    "        # Test the new table every 1k games\n",
    "        if (games_count + 1) % 1000 == 0:\n",
    "            test_reward = test_game(env, table)\n",
    "            print('\\tEp:', games_count, 'Test reward:', test_reward, np.round(epsilon,2))\n",
    "\n",
    "            test_rewards_list.append(test_reward)\n",
    "\n",
    "        obs = env.reset()\n",
    "        reward_count = 0\n",
    "        games_count += 1    \n",
    "\n",
    "# Plot the accuracy over the number of steps\n",
    "plt.figure(figsize=(18,9))\n",
    "plt.xlabel('Steps')\n",
    "plt.ylabel('Accurracy')\n",
    "plt.plot(test_rewards_list)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NB: in case you want to apply Q-learning to continuous state and actions games, you have to quantize the state and action spaces"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}