reading_assignments/8_Note-ReinforcementLearning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$\\newcommand{\\xv}{\\mathbf{x}}\n",
    " \\newcommand{\\wv}{\\mathbf{w}}\n",
    " \\newcommand{\\yv}{\\mathbf{y}}\n",
    " \\newcommand{\\zv}{\\mathbf{z}}\n",
    " \\newcommand{\\Chi}{\\mathcal{X}}\n",
    " \\newcommand{\\R}{\\rm I\\!R}\n",
    " \\newcommand{\\sign}{\\text{sign}}\n",
    " \\newcommand{\\Tm}{\\mathbf{T}}\n",
    " \\newcommand{\\Xm}{\\mathbf{X}}\n",
    " \\newcommand{\\Zm}{\\mathbf{Z}}\n",
    " \\newcommand{\\I}{\\mathbf{I}}\n",
    " \\newcommand{\\muv}{\\boldsymbol\\mu}\n",
    " \\newcommand{\\Sigmav}{\\boldsymbol\\Sigma}\n",
    "$\n",
    "### ITCS6155\n",
    "\n",
    "\n",
    "# Reinforcement Learning\n",
    "\n",
    "Along with supervised and unsupervised learning, reinforcement learning is one of the interesting fields of machine learining. \n",
    "As we shortly discussed in the first week of class, reinforcement learning is different from other machine learning paradigms as summarized below: \n",
    "\n",
    "- only reward signal as feedback,\n",
    "- the feedback can be delayed, \n",
    "- sequential data, \n",
    "- interaction based on the actions taken. \n",
    "\n",
    "The reinforcement learning resembles huuman learning or animal training that treats reward good behavior. \n",
    "When the series of actions end up with good results, we can **reinforce** those actions by giving some rewards.\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "<td>\n",
    "<img src=\"http://webpages.uncc.edu/mlee173/teach/itcs4156online/images/class/werber_ls.jpeg\" height=300/>\n",
    "</td>\n",
    "<td>\n",
    "<img src=\"http://webpages.uncc.edu/mlee173/teach/itcs4156online/images/class/pavlovs-dogs-mark-stivers.gif\" width=400/>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "## Applictions\n",
    "\n",
    "\n",
    "There are many possible applications: \n",
    "\n",
    "- walking robot, \n",
    "- playing boardgames, \n",
    "- controlling joysticks for video games,\n",
    "- smart thermostat,\n",
    "- stock trading,\n",
    "- music personalization,\n",
    "- recommendation system,\n",
    "- marketing,\n",
    "- product delivery,\n",
    "- and so on...\n",
    "\n",
    "<img src=\"http://webpages.uncc.edu/mlee173/teach/itcs4156online/images/class/RLexamples.png\" width=600/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Terminology\n",
    "\n",
    "- Reward: $R_t$ is a scalar feedback signal at time $t$. It indicates how well agent is doing at the time. \n",
    "\n",
    "- State: $S_t$ represents what happens currently and next. \n",
    "\n",
    "- Action: $A_t$ is how an agent affects to the environment that can change the state. \n",
    "\n",
    "- Observation: $O_t$ is what an agent recognizes the world for the state $S_t$. \n",
    "\n",
    "- Policy: A function that determines an agent's behavior or a function that selects its action, $\\pi(S_t)$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> <font color=\"red\">Q: Think about the rewards for the example applications above, and answer the possible rewards for them.</font></b>\n",
    "- walking robot: Positive reward for continuous successful walking; negative reward if it falls down.\n",
    "- playing boardgames: Positive reward for wining; negative reward for loosing the game.\n",
    "- controlling joysticks for video games: Positive reward for continuing the game; negative reward for loosing to oponent.\n",
    "- smart thermostat: Not sure!\n",
    "- stock trading: Positive reward for good prediction of profit; negative reward for loosing money.\n",
    "- music personalization: Positive reward when user doesn't change the recommended music or play a lot from the recommended music; negative reward for not playing from recommendation or giving low ratting to the recommended music.\n",
    "- recommendation system: Same as the `music personalization`.\n",
    "- marketing: Positive reward for sales growth; negative reward for the oposit.\n",
    "- product delivery: Positive reward for confirmation of product delivery; negative reward for not getting notification on-time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling \n",
    "\n",
    "How can we simplify the listed problems to solve? We have the sets of decision making problems or control problems.\n",
    "With the assumption of **Markov property**, we can model the problems as Makrov Decision Process (MDP).\n",
    "\n",
    "Here, the definition of Markov property is as follows,\n",
    "\n",
    "**Definition**: A state $S_t$ is Markov if and only if \n",
    "<br/><br/>\n",
    "$$P [ S_{t+1} | S_t ] = P[ S_{t+1} | S_1, ..., S_t ].$$\n",
    "\n",
    "That is, we can say the state transition model has Markov property when the future is dependent only on the current state not the past. \n",
    "\n",
    "\n",
    "### Markov Decision Processes\n",
    "So, the MDP can be defined as follows and the example state transition diagram has the transition probabilities and rewards along with discounting factor. The `discounting factor` ensures the mathematical simplicity, and it allows the modeling the uncertainty of the future as well. \n",
    "\n",
    " <font color='red' size=5>$ ( S, A, P, R, \\gamma ) $ </font>\n",
    "* $S$ : a finite set of states\n",
    "* $A$ : a finite set of actions\n",
    "* $P$ : a state transition probability\n",
    "* $P^a_{ss^\\prime} = P [ S_{t+1} = s^\\prime | S_t = s, A_t = a ]$\n",
    "* $R$ : a reward function\n",
    "* $\\gamma$ : a discount factor\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/21/Markov_Decision_Process_example.png\"\n",
    "width=500 />  \n",
    "<center> MDP with 3 states and 2 actions (wikipedia) </center>\n",
    "\n",
    "\n",
    "### Policy\n",
    "\n",
    "The policy $\\pi(S_t)$ is a function that maps from a state to an action. Thus, it can be easily represented by a probability: \n",
    "\n",
    "$$ \\pi(a | s) = P[A_t = a | S_t = s]. $$\n",
    "\n",
    "Or, it can be a machine learning model to generate such probability for a certain behavior of an agent. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal\n",
    "\n",
    "Now with the MDP model, what we want to do is find a policy $\\pi$ that `maximizes the long-term rewards`. The long-term rewards can be modeled as the sum of rewards. Here, considering the different importance on the actions (the current reward is not equivalent to future rewards), we can define the **Return**, $G_t$, \n",
    "\n",
    "$$ G_t = R_{t+1} + \\gamma R_{t+2} + \\cdots = \\sum_{k=0}^{\\infty} \\gamma^k R_{t+k+1}. $$\n",
    "\n",
    "From this, we can model the long-term value of a certain state $s$ with the return:\n",
    "\n",
    "$$ V(s) = E [ G_t | S_t = s ]. $$\n",
    "\n",
    "We, we set our objective to maximize the return. Developing the optimal policy can be made by defining the evaluation function of state and action pair, the `Q funciton`: \n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "    Q^\\pi(s, a) &= E_\\pi [ G_t | S_t = s, A_t = a ] \\\\\n",
    "                &= E_\\pi [ R_{t+1} + \\gamma Q^\\pi (S_{t+1}, A_{t+1}) | S_t = s, A_t = a ]\n",
    "\\end{align}    \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to reach the goal? / How to find the optimal policy?\n",
    "\n",
    "There are several ways to reach the goal such as `value iteration`, `policy iteration`, `linear programming`, and `temporal difference learning`. \n",
    "\n",
    "\n",
    "First, let us examine the bootstraping representation of the value function to solve the problem.\n",
    "This is called **Bellman equation**. \n",
    "\n",
    "$$ \n",
    "\\begin{aligned}\n",
    "  V(s) &= E [ G_t | S_t = s ] \\\\\n",
    "       &= E [ R_{t+1} + \\gamma R_{t+2} + \\cdots | S_t = s ] \\\\\n",
    "       &= E [ R_{t+1} + \\gamma ( R_{t+2} + \\gamma R_{t+3} + \\cdots ) | S_t = s ] \\\\\n",
    "       &= E [ R_{t+1} + \\gamma G_{t+1} | S_t = s ] \\\\\n",
    "       &= E [ R_{t+1} + \\gamma V(s_{t+1}) | S_t= s ]\\\\\n",
    "      \\\\\n",
    "  V(s) &= R(s) + \\gamma \\sum_{s^\\prime \\in S} P_{ss^\\prime} V(s^\\prime)      \n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "With matrix representation, we can rewrite this as follows:\n",
    "\n",
    "$$\n",
    " \\begin{align}\n",
    "  \\begin{pmatrix}\n",
    "  V_1 \\\\\n",
    "  V_2 \\\\\n",
    "  \\vdots \\\\\n",
    "  V_n  \n",
    " \\end{pmatrix} &= \n",
    "  \\begin{pmatrix}\n",
    "  R_1 \\\\\n",
    "  R_2 \\\\\n",
    "  \\vdots \\\\\n",
    "  R_n  \n",
    " \\end{pmatrix} +   \n",
    " \\gamma\n",
    " \\begin{pmatrix}\n",
    "  P_{11} & P_{12} & \\cdots & P_{1n} \\\\\n",
    "  P_{21} & P_{22} & \\cdots & P_{2n} \\\\\n",
    "  \\vdots  & \\vdots  & \\ddots & \\vdots  \\\\\n",
    "  P_{n1} & P_{n2} & \\cdots & P_{nn} \n",
    " \\end{pmatrix}\n",
    "  \\begin{pmatrix}\n",
    "  V_1 \\\\\n",
    "  V_2 \\\\\n",
    "  \\vdots \\\\\n",
    "  V_n  \n",
    " \\end{pmatrix}\n",
    " \\\\\n",
    " \\\\\n",
    " V &= R + \\gamma PV \n",
    " \\end{align}\n",
    "$$\n",
    "\n",
    "\n",
    "This can be directly solved with\n",
    "\n",
    "$$ V = (I - \\gamma P)^{-1} R. $$\n",
    "\n",
    "Considering the computational complexity with inverse matrix, however, we can consider iterative solutions such as `dynamic programming`, `Monte-Carlo evaluation`, or `temporal difference learning`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Value Iteration\n",
    "\n",
    "* For state transition from $i$ to $j$ with action $k$\n",
    "$$ V^{n+1} (s_i) = \\max_k \\Big[ R_i + \\gamma \\sum_{j=1}^N P^k_{ij} V^n (s_j) \\Big] $$\n",
    "\n",
    "* Value Iteration:\n",
    "<font size=3>\n",
    "$$ \n",
    "\\begin{align}\n",
    "n=0 \\quad \\forall i, \\quad &V^1 (s_i) = \\max_k \\Big[ R_i + \\gamma \\sum_{j=1}^N P^k_{ij} V^0 (s_j) \\Big] \\\\\n",
    "n=1 \\quad \\forall i, \\quad &V^2 (s_i) = \\max_k \\Big[ R_i + \\gamma \\sum_{j=1}^N P^k_{ij} V^1 (s_j) \\Big] \\\\\n",
    "n=2 \\quad \\forall i, \\quad &V^3 (s_i) = \\max_k \\Big[ R_i + \\gamma \\sum_{j=1}^N P^k_{ij} V^2 (s_j) \\Big] \\\\\n",
    "\\vdots\n",
    "\\end{align}\n",
    "$$\n",
    "</font>\n",
    "\n",
    "* Convergence can be tested as follows: \n",
    "\n",
    "$$ \\max_i \\Big| V^{n+1}(s_i) - V^n(s_i) \\Big| \\lt \\epsilon. $$ \n",
    "    \n",
    "* Computing values for all $i$'s with Dynamic Programming. \n",
    "\n",
    "\n",
    "Here follows an example from Wikipedia. \n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/21/Markov_Decision_Process_example.png\"\n",
    "width=350 />  \n",
    "\n",
    "Let $\\gamma = 1$. The values for each state can be iteratively updated as the following table. \n",
    "\n",
    "| t | $V^\\pi_{S_0}$ | $V^\\pi_{S_1}$ | $V^\\pi_{S_2}$ |\n",
    "|:--|:-------------|:-------------|:-------------|\n",
    "| 0 |      0       |       0.0      |      0       |\n",
    "| 1 |      0       |       3.5      |    0.75      |\n",
    "| 2 |      0       |       3.85     |    0.75      |\n",
    "| 3 |      0.75       |      4.035      |    1.155      |\n",
    "| 4 |      1.155       |      4.6595      |    1.5975      |\n",
    "| 5 |      1.5975     |      5.09395   |    2.08335      |\n",
    "\n",
    "In Python, we can write the codes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma = 1.0\n",
    "n_states = 3\n",
    "n_actions = 2\n",
    "\n",
    "# value vector\n",
    "V = np.zeros(n_states).reshape((-1, 1))\n",
    "# Reward dictionary\n",
    "R = {(1,2,0): -1, (0,1,0): 5}\n",
    "# transition probability: key (action, from, to)\n",
    "P = {(0,0,0): 0.5, (0,0,2): 0.5, (1,0,2): 1.0, \n",
    "     (0,1,0): 0.7, (0,1,1): 0.1, (0,1,2): 0.2,\n",
    "     (1,1,1): 0.95, (1,1,2): 0.05,\n",
    "     (0,2,0): 0.4, (0,2,2): 0.6, \n",
    "     (1,2,0): 0.3, (1,2,1): 0.3, (1,2,2): 0.4 }\n",
    "\n",
    "def Rf(s, sn, a):\n",
    "    if (a, s, sn) in R:\n",
    "        return R[(a,s,sn)]\n",
    "    return 0\n",
    "\n",
    "def Pf(s, sn, a):\n",
    "    if (a, s, sn) in P:\n",
    "        return P[(a,s,sn)]\n",
    "    return 0.\n",
    "\n",
    "\n",
    "\n",
    "\"\"\" with no matrix\"\"\"\n",
    "def Rs(s, a):\n",
    "    r = 0\n",
    "    for j in range(3):\n",
    "        if (a, s, j) in P:\n",
    "            r += P[(a, s, j)] * Rf(a,s, j)\n",
    "    return r\n",
    "\n",
    "\n",
    "def Pfv(s, a):\n",
    "    ret = np.zeros(3).reshape((-1, 1))\n",
    "    for j in range(3):\n",
    "        if (a, s, j) in P:\n",
    "            ret[j] = P[(a, s, j)]\n",
    "    return ret\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t= 0 \tV =  [[0. 0. 0.]]\n",
      "t= 1 \tV =  [[0.  3.5 0. ]]\n",
      "t= 2 \tV =  [[0.   3.85 0.75]]\n",
      "t= 3 \tV =  [[0.75  4.035 1.155]]\n",
      "t= 4 \tV =  [[1.155  4.6595 1.5975]]\n",
      "t= 5 \tV =  [[1.5975  5.09395 2.08335]]\n"
     ]
    }
   ],
   "source": [
    "Rmat = np.zeros((n_states, n_actions))\n",
    "Pmat = np.zeros((n_actions, n_states, n_states))\n",
    "\n",
    "for i in range(n_states):\n",
    "    for k in range(n_actions):\n",
    "        for j in range(n_states):\n",
    "            if (k, i, j) in P:\n",
    "                Rmat[i, k] += P[(k, i, j)] * Rf(i, j, k)\n",
    "\n",
    "for k, i, j in P.keys():\n",
    "    Pmat[k,i,j] = P[(k, i, j)]\n",
    "\n",
    "\n",
    "for t in range(5):\n",
    "    print(\"t=\", t, \"\\tV = \", V.T)\n",
    "    # Bellman\n",
    "    newV = Rmat + gamma * np.sum(Pmat @ V, axis=2).T\n",
    "    V[:] = np.max(newV, axis=1, keepdims=True)\n",
    "    \n",
    "print(\"t=\", t+1, \"\\tV = \", V.T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t= 0 \tV =  [[1.5975  5.09395 2.08335]]\n",
      "t= 1 \tV =  [[2.08335  4.94342  2.941371]]\n",
      "t= 2 \tV =  [[2.941371   4.84331755 3.51195497]]\n",
      "t= 3 \tV =  [[3.51195497 4.77674942 3.8913933 ]]\n",
      "t= 4 \tV =  [[3.8913933  4.73248161 4.1437198 ]]\n",
      "t= 5 \tV =  [[4.1437198  4.70304352 4.31151691]]\n",
      "t= 6 \tV =  [[4.31151691 4.68346719 4.423102  ]]\n",
      "t= 7 \tV =  [[4.423102   4.67044893 4.49730608]]\n",
      "t= 8 \tV =  [[4.49730608 4.66179179 4.54665179]]\n",
      "t= 9 \tV =  [[4.54665179 4.65603479 4.57946669]]\n",
      "t= 10 \tV =  [[4.57946669 4.65220639 4.6012886 ]]\n",
      "t= 11 \tV =  [[4.6012886  4.6496605  4.61580017]]\n",
      "t= 12 \tV =  [[4.61580017 4.64796748 4.62545036]]\n",
      "t= 13 \tV =  [[4.62545036 4.64684162 4.63186774]]\n",
      "t= 14 \tV =  [[4.63186774 4.64609293 4.6361353 ]]\n",
      "t= 15 \tV =  [[4.6361353  4.64559505 4.63897322]]\n",
      "t= 16 \tV =  [[4.63897322 4.64526396 4.64086044]]\n",
      "t= 17 \tV =  [[4.64086044 4.64504378 4.64211544]]\n",
      "t= 18 \tV =  [[4.64211544 4.64489736 4.64295002]]\n",
      "t= 19 \tV =  [[4.64295002 4.6448     4.64350501]]\n",
      "t= 20 \tV =  [[4.64350501 4.64473525 4.64387408]]\n",
      "t= 21 \tV =  [[4.64387408 4.64469219 4.64411952]]\n",
      "t= 22 \tV =  [[4.64411952 4.64466356 4.64428273]]\n",
      "t= 23 \tV =  [[4.64428273 4.64464452 4.64439126]]\n",
      "t= 24 \tV =  [[4.64439126 4.64463185 4.64446344]]\n",
      "t= 25 \tV =  [[4.64446344 4.64462343 4.64451144]]\n",
      "t= 26 \tV =  [[4.64451144 4.64461783 4.64454336]]\n",
      "t= 27 \tV =  [[4.64454336 4.64461411 4.64456458]]\n",
      "t= 28 \tV =  [[4.64456458 4.64461163 4.6445787 ]]\n",
      "t= 29 \tV =  [[4.6445787  4.64460999 4.64458808]]\n",
      "t= 30 \tV =  [[4.64458808 4.64460889 4.64459433]]\n",
      "t= 31 \tV =  [[4.64459433 4.64460816 4.64459848]]\n",
      "t= 32 \tV =  [[4.64459848 4.64460768 4.64460124]]\n",
      "t= 33 \tV =  [[4.64460124 4.64460736 4.64460307]]\n",
      "t= 34 \tV =  [[4.64460307 4.64460714 4.64460429]]\n",
      "t= 35 \tV =  [[4.64460429 4.644607   4.6446051 ]]\n",
      "t= 36 \tV =  [[4.6446051  4.6446069  4.64460564]]\n",
      "t= 37 \tV =  [[4.64460564 4.64460684 4.644606  ]]\n",
      "t= 38 \tV =  [[4.644606   4.6446068  4.64460624]]\n",
      "t= 39 \tV =  [[4.64460624 4.64460677 4.6446064 ]]\n",
      "t= 40 \tV =  [[4.6446064  4.64460675 4.64460651]]\n",
      "t= 41 \tV =  [[4.64460651 4.64460674 4.64460658]]\n",
      "t= 42 \tV =  [[4.64460658 4.64460673 4.64460662]]\n",
      "t= 43 \tV =  [[4.64460662 4.64460673 4.64460665]]\n",
      "t= 44 \tV =  [[4.64460665 4.64460672 4.64460668]]\n",
      "t= 45 \tV =  [[4.64460668 4.64460672 4.64460669]]\n",
      "t= 46 \tV =  [[4.64460669 4.64460672 4.6446067 ]]\n",
      "t= 47 \tV =  [[4.6446067  4.64460672 4.6446067 ]]\n",
      "t= 48 \tV =  [[4.6446067  4.64460672 4.64460671]]\n",
      "t= 49 \tV =  [[4.64460671 4.64460672 4.64460671]]\n",
      "t= 50 \tV =  [[4.64460671 4.64460672 4.64460671]]\n"
     ]
    }
   ],
   "source": [
    "gamma=1\n",
    "for t in range(50):\n",
    "    print(\"t=\", t, \"\\tV = \", V.T)\n",
    "    for i in range(3):\n",
    "        # Bellman\n",
    "        V[i] = max(Rs(i, 0) + gamma * np.sum(Pfv(i, 0) * V), \n",
    "                   Rs(i, 1) + gamma * np.sum(Pfv(i, 1) * V))\n",
    "print(\"t=\", t+1, \"\\tV = \", V.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy Iteration\n",
    "\n",
    "![](http://incompleteideas.net/book/4/img158.gif)\n",
    "\n",
    "* Start with an initial policy $\\pi^0$.\n",
    "* Iteratively,\n",
    "    * Evaluate policy ($V^n(x) = V^{\\pi^n}(x)$):\n",
    "    $$ V^n(s) = R(s, a = \\pi^n(s)) + \\gamma \\sum_{s^\\prime} P(s^\\prime | s, a = \\pi^n(s)) V^{n}(s^\\prime) $$\n",
    "    * Improve policy:\n",
    "    $$ \\pi_{n+1}(s) = \\arg \\max_a \\Big[ R(s, a) + \\gamma \\sum_{s^\\prime} P(s^\\prime | s, a) V^{n}(s^\\prime) \\Big] $$\n",
    "    \n",
    "* Stop condition:\n",
    "    * Policy does not change anymore (for about 10 iterations)\n",
    "    * The changes in evaluation of values are minor \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n =  0 \t [1 0 0]\t [0. 0. 0.]\n",
      "n =  1 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  2 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  3 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  4 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  5 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  6 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  7 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  8 \t [0 0 0]\t [0. 0. 0.]\n",
      "n =  9 \t [0 0 0]\t [0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "n_iter = 10\n",
    "gamma = 1.\n",
    "V = np.zeros(3)\n",
    "\n",
    "pi = np.random.randint(2, size=n_states)\n",
    "\n",
    "for n in range(n_iter):\n",
    "    print(\"n = \", n, \"\\t\", pi, end=\"\")\n",
    "    \n",
    "    # evaluate\n",
    "    for s in range(n_states):\n",
    "        V[s] = Rs(s, pi[s]) + gamma * np.sum([Pf(s, sn, pi[s]) * V[sn] for sn in range(n_states)])\n",
    "    print(\"\\t\", V)\n",
    "    \n",
    "    # improve\n",
    "    for s in range(n_states):\n",
    "        pi[s] = np.argmax([Rs(s, a) + gamma * np.sum([Pf(s, sn, a) * V[sn] for sn in range(n_states)])  for a in range(n_actions)] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Programming\n",
    "\n",
    "                                                                                    [Manne '60]\n",
    "\n",
    "\n",
    "\\begin{equation*}\n",
    "\\begin{aligned}\n",
    "& \\underset{x}{\\text{minimize}}\n",
    "& & \\sum_x V(x) \\\\\n",
    "& \\text{subject to}\n",
    "& & V(x) \\geq R(x, u) + \\gamma \\sum_{x^\\prime} P(x^\\prime | x, u) V^{n}(x^\\prime) &\\forall x, a.\n",
    "\\end{aligned}\n",
    "\\end{equation*}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Temporal Difference Learning\n",
    "\n",
    "When there are large number of states, the memory requirement grows exponentially. \n",
    "*Temporal difference (TD) learning* considers that the agent knows only the partial information of the MDP. \n",
    "With only current and next state transition and without any model transition probability, TD lets the agent explore the environment to examine the random policy. \n",
    "With an estiate of the value function $V(s)$, $\\hat{V}(s)$, \n",
    "\n",
    "   $$ \n",
    "        \\begin{align}\n",
    "        V(s_t) &= R_{t+1} + \\gamma V(s_{t+1}) \\\\\n",
    "        V(s_t) &\\sim R_{t+1} + \\gamma \\hat{V}(s_{t+1}) \\\\\n",
    "        \\Rightarrow \\quad \\delta_t &= R_{t+1} + \\gamma \\hat{V}(s_{t+1}) - V(s_t).\n",
    "        \\end{align}\n",
    "    $$\n",
    "    \n",
    "Here, $\\delta$ represents the *temporal diffrence error*. \n",
    "We can use this error as a gradient to update the value estimation.\n",
    "\n",
    "$$\n",
    "    \\begin{align}\n",
    "        V(s_t) &\\leftarrow R_{t+1} + \\alpha \\delta_t \\\\\n",
    "        V(s_t) &\\leftarrow R_{t+1} + \\alpha (R_{t+1} + \\gamma \\hat{V}(s_{t+1}) - V(s_t))\n",
    "    \\end{align}\n",
    "$$\n",
    "\n",
    "\n",
    "## Example Model\n",
    "\n",
    "<img src=\"http://webpages.uncc.edu/mlee173/teach/itcs4156online/images/class/rl_simple_model.png\" width=700 />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model (problem to solve)\n",
    "# a: 0 - loop  1 - to next state\n",
    "# r:       +1                 +0      +100 when reaches 4\n",
    "#\n",
    "n_states = 5\n",
    "n_actions = 2\n",
    "\n",
    "def transition(s, a):\n",
    "    s1 = s + a\n",
    "    if s == s1: \n",
    "        r = 1\n",
    "    elif s1 == n_states-1:\n",
    "        r = 100\n",
    "    else:\n",
    "        r = 0\n",
    "    return s1, r\n",
    "\n",
    "def pause_print():\n",
    "    sys.stdout.write('\\r')\n",
    "    sys.stdout.write(\"\\033[F\")\n",
    "    sys.stdout.flush()\n",
    "    time.sleep(0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Policy: [1 0 0 0 1]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [0.   6.13 0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [5.51 8.5  0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [7.65 9.42 0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [8.48 9.77 0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [8.8  9.91 0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [8.92 9.97 0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [8.97 9.99 0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [8.99 9.99 0.   0.   0.  ]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\r",
      "\tTraj:  0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 \t [ 9. 10.  0.  0.  0.]\n",
      "\n",
      "Policy: [1 1 1 1 1]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [  9.   0.   0. 100.   0.]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [  0.   0.  90. 100.   0.]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [  0.  81.  90. 100.   0.]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\r",
      "\tTraj:  0 1 1 1 2 1 3 1 4 \t [ 72.9  81.   90.  100.    0. ]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Example TD(0) Update\n",
    "alpha = 1.0\n",
    "gamma = 0.9\n",
    "V = np.zeros(n_states)  # 5 states\n",
    "pi = np.random.randint(n_actions, size=n_states)\n",
    "\n",
    "np.set_printoptions(precision=2)\n",
    "for k in range(2):\n",
    "    if k == 1:\n",
    "        pi = np.ones(n_states, dtype=np.int)\n",
    "    print(\"Policy:\", pi)\n",
    "    # Evaluation of the random policy\n",
    "    for e in range(20):\n",
    "        s = 0 \n",
    "\n",
    "        print(\"\\r\\tTraj: \", s, end=\" \")\n",
    "        for step in range(10):\n",
    "            a = pi[s]\n",
    "            s1, r1 = transition(s, a)\n",
    "            V[s] += alpha * (r1 + gamma * V[s1] - V[s])\n",
    "            s = s1\n",
    "\n",
    "            print(a, s, end=\" \")\n",
    "            if s == n_states-1:\n",
    "                break\n",
    "\n",
    "        print(\"\\t\", V, end=\"\\n\")\n",
    "        #pause_print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Control Problems and Q function\n",
    "\n",
    "For control problems, we have defined Q function above to evaluate the state and action altogether.\n",
    "Updating the Q values with TD learning is similar to previous update with two different considerations. \n",
    "First, we update the Q with assumption that we follow a certain behavior policy. Thus, we call this as *on-policy control*, or **SARSA**. \n",
    "\n",
    "$$\n",
    "    Q(s_t, a_t) \\leftarrow Q(s_t, a_t) + \\alpha ( R_{t+1} + \\gamma Q(s_{t+1}, a_{t+1}) - Q(s_t, a_t)) \n",
    "$$\n",
    "\n",
    "Next, without making assumption of behavior policy, we can explore other possible policies to update the Q. We call this as *off-policy control*, or **Q-learning**. \n",
    "\n",
    "\n",
    "$$\n",
    "    Q(s_t, a_t) \\leftarrow Q(s_t, a_t) + \\alpha ( R_{t+1} + \\gamma \\max_a Q(s_{t+1}, a) - Q(s_t, a_t)) \n",
    "$$\n",
    "\n",
    "Here describes the psedocode for each algorithm. \n",
    "\n",
    "\n",
    "**[Algorithm: TD Learning]**\n",
    "![](http://incompleteideas.net/book/ebook/pseudotmp7.png)\n",
    "\n",
    "**[Algorithm: SARSA]**\n",
    "![](http://incompleteideas.net/book/ebook/pseudotmp8.png)\n",
    "\n",
    "\n",
    "**[Algorithm: Q-learning]**\n",
    "![](http://incompleteideas.net/book/ebook/pseudotmp9.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Choosing an Action\n",
    "\n",
    "Picking an action can be simple by selecting one with maximum Q value, so *Greedy!*. \n",
    "\n",
    "$$\n",
    "a^* = \\arg \\max_a Q(S_t, a)\n",
    "$$\n",
    "\n",
    "However, this can cause limited experience to develop good Q estimation, and eventually a good policy. \n",
    "Without new data, greedy action selection will repeat the same actions, or repeatedly *exploit* your current knowledge. Thus, you need to *explore* other non-greedy actions to increase the experience to improve the Q estimation. \n",
    "\n",
    "This is called \"exploration-exploitation dilemma.\" \n",
    "\n",
    "\n",
    "One of the way for this dilemma is $\\epsilon$-greedy action selection. With a parameter $\\epsilon \\in [0, 1]$, we can control the exploration and exploitation level. When $\\epsilon = 0$, the actions are selected in greedy manner, but when $\\epsilon = 1$, the actions are selected randomly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# action selection\n",
    "def greedy(Q, s):\n",
    "    return np.argmax(Q[s])  # greedy action selection\n",
    "\n",
    "def e_greedy(Q, s, e):   \n",
    "    if np.random.rand() < e:\n",
    "        return np.random.randint(n_actions)\n",
    "    else:\n",
    "        return greedy(Q,s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Q:  [[  2.52  73.17]\n",
      " [ 74.32  81.47]\n",
      " [ 82.47  90.52]\n",
      " [ 91.52 100.57]\n",
      " [  0.42   0.64]]\n",
      "Policy: [1 1 1 1 1]\n"
     ]
    }
   ],
   "source": [
    "# SARSA example\n",
    "alpha = 1.  # learning rate\n",
    "gamma = 0.9 # discount factor\n",
    "epsilon = 0.1\n",
    "\n",
    "# tabular approximation\n",
    "Q = np.random.rand(n_states, n_actions)  # 5 states and 2 actions\n",
    "\n",
    "for e in range(20):\n",
    "    s = 0 \n",
    "    a = e_greedy(Q, s, epsilon)  # greedy action selection\n",
    "    \n",
    "    for step in range(100):\n",
    "        s1, r1 = transition(s, a)\n",
    "        a1 = e_greedy(Q, s, epsilon)\n",
    "        \n",
    "        # TODO: update Q table here!\n",
    "        Q[s, a] += alpha * (r1 + gamma * Q[s1, a1] - Q[s,a])\n",
    "        \n",
    "        #print(\"s: \", s, \"a: \", a, \"s1:\", s1, \"a1:\", a1, \"Q: \", Q[s,a])\n",
    "        s, a = s1, a1\n",
    "        if s == n_states-1:\n",
    "            break\n",
    "    \n",
    "print(\"Final Q: \", Q)\n",
    "print(\"Policy:\", np.argmax(Q, axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s:  0 a:  s1: 1 a1: 1 Q:  0.7561608580140299\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  1.75616085801403\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  2.580544772212627\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  3.3224902949913644\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  3.990241265492228\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  4.591217138943005\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  5.132095425048704\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  5.618885882543834\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  0.7505308201058045\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  0.42173196249345\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  1.4217319624934501\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  2.2795587662441053\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  3.0516028896196947\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  3.7464426006577254\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  4.371798340591953\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  4.9346185065327575\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  5.441156655879482\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  5.897040990291534\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  6.307336891262381\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  6.676603202136143\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  7.008942881922528\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  7.308048593730275\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  7.577243734357248\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  7.819519360921523\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  8.037567424829371\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  8.233810682346434\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  8.410429614111791\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  8.569386652700612\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  8.71244798743055\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  8.841203188687496\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  8.957082869818747\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  5.0569972942894506\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  6.0569972942894506\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  6.451297564860505\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  6.806167808374455\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  7.12555102753701\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  7.412995924783309\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  7.671696332304978\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  7.904526699074481\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  8.114074029167032\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  8.302666626250328\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  8.472399963625296\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  8.625159967262768\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  8.762643970536491\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  8.886379573482841\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  8.997741616134558\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  9.097967454521102\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  9.188170709068991\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  9.269353638162093\n",
      "s:  1 a:  s1: 1 a1: 1 Q:  9.342418274345883\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  0.379558766244105\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  8.408176446911295\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 0 a1: 1 Q:  66.98755171443602\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 3 a1: 1 Q:  91.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 2 a1: 1 Q:  82.4661132276988\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 0 a1: 1 Q:  66.98755171443602\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "s:  0 a:  s1: 1 a1: 1 Q:  73.31950190492891\n",
      "s:  1 a:  s1: 2 a1: 1 Q:  81.46611322769878\n",
      "s:  2 a:  s1: 3 a1: 1 Q:  90.51790358633198\n",
      "s:  3 a:  s1: 4 a1: 1 Q:  100.57544842925776\n",
      "Final Q:  [[ 66.99  73.32]\n",
      " [  9.34  81.47]\n",
      " [ 82.47  90.52]\n",
      " [ 91.52 100.58]\n",
      " [  0.64   0.38]]\n",
      "Policy: [1 1 1 1 0]\n"
     ]
    }
   ],
   "source": [
    "# Q-learning example\n",
    "alpha = 1.  # learning rate\n",
    "gamma = 0.9 # discount factor\n",
    "epsilon = 0.1\n",
    "\n",
    "# tabular approximation\n",
    "Q = np.random.rand(n_states, n_actions)  # 5 states and 2 actions\n",
    "\n",
    "for e in range(20):\n",
    "    s = 0 \n",
    "    \n",
    "    for step in range(100):\n",
    "        a = e_greedy(Q, s, epsilon)  # greedy action selection\n",
    "        s1, r1 = transition(s, a)\n",
    "        \n",
    "        # TODO: update Q table here!\n",
    "        Q[s, a] += alpha * (r1 + gamma * np.max(Q[s1, :]) - Q[s,a])\n",
    "        \n",
    "        print(\"s: \", s, \"a: \", \"s1:\", s1, \"a1:\", a1, \"Q: \", Q[s,a])\n",
    "        s = s1\n",
    "        if s == n_states-1:\n",
    "            break\n",
    "            \n",
    "print(\"Final Q: \", Q)\n",
    "print(\"Policy:\", np.argmax(Q, axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Maze Example\n",
    "\n",
    "![](https://image.jimcdn.com/app/cms/image/transf/none/path/se30cb2e6454ed16a/image/if80ea7358ccd7b51/version/1422202157/image.jpg)\n",
    "\n",
    "Now, let us solve some maze problems. To deal with various maze configuration, we define our class that can handle a file input of maze. Here the file is expected to have 3 letters: 'O' for open space, 'H' for obstacles (walls), and 'G' for a goal position. The example grid.txt file looks as follows. \n",
    "\n",
    "Note: this bash command runs only on Linux or Mac. The shell is to show the content of file only. You can skip running the cell and creat a file with the same content. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OOOHOOOOO\n",
      "OOOHOOHOO\n",
      "OOOOOOHOO\n",
      "OOOOHHHOO\n",
      "OOHOOOOOH\n",
      "OOHOOOGOO\n",
      "OOOOOOOOO"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "cat grid.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us define the environment class. The environment constructor gets only one input, the filename. Along with some utility functions, the main simulation functions are <code>init()</code> and <code>next()</code> for initiating and running one step simulation. Please take a look at the example use and try them in a separate cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# maze example\n",
    "\n",
    "class GridWorld:\n",
    "    \"\"\" Grid World environment\n",
    "            there are four actions (left, right, up, and down) to move an agent\n",
    "            In a grid, if it reaches a goal, it get 30 points of reward.\n",
    "            If it falls in a hole or moves out of the grid world, it gets -5.\n",
    "            Each step costs -1 point. \n",
    "\n",
    "        to test GridWorld, run the following sample codes:\n",
    "\n",
    "            env = GridWorld('grid.txt')\n",
    "\n",
    "            env.print_map()\n",
    "            print [2,3], env.check_state([2,3])\n",
    "            print [0,0], env.check_state([0,0])\n",
    "            print [3,4], env.check_state([3,4])\n",
    "            print [10,3], env.check_state([10,3])\n",
    "\n",
    "            env.init([0,0])\n",
    "            print env.next(1)  # right\n",
    "            print env.next(3)  # down\n",
    "            print env.next(0)  # left\n",
    "            print env.next(2)  # up\n",
    "            print env.next(2)  # up\n",
    "\n",
    "        Parameters\n",
    "        ==========\n",
    "        _map        ndarray\n",
    "                    string array read from a file input\n",
    "        _size       1d array\n",
    "                    the size of _map in ndarray\n",
    "        goal_pos    tuple\n",
    "                    the index for the goal location\n",
    "        _actions    list\n",
    "                    list of actions for 4 actions\n",
    "        _s          1d array\n",
    "                    current state\n",
    "    \"\"\"\n",
    "    def __init__(self, fn):\n",
    "        # read a map from a file\n",
    "        self._map = self.read_map(fn)\n",
    "        self._size = np.asarray(self._map.shape)\n",
    "        self.goal_pos = np.where(self._map == 'G')\n",
    "\n",
    "        # definition of actions (left, right, up, and down repectively)\n",
    "        self._actions = [[0, -1], [0, 1], [-1, 0], [1, 0]]\n",
    "        self._s = None\n",
    "\n",
    "    def get_cur_state(self):\n",
    "        return self._s\n",
    "\n",
    "    def get_size(self):\n",
    "        return self._size\n",
    "\n",
    "    def read_map(self, fn):\n",
    "        grid = []\n",
    "        with open(fn) as f:\n",
    "            for line in f:\n",
    "               grid.append(list(line.strip()))\n",
    "        return np.asarray(grid)\n",
    "\n",
    "    def print_map(self):\n",
    "        print( self._map )\n",
    "\n",
    "    def check_state(self, s):\n",
    "        if isinstance(s, collections.Iterable) and len(s) == 2:\n",
    "            if s[0] < 0 or s[1] < 0 or\\\n",
    "               s[0] >= self._size[0] or s[1] >= self._size[1]:\n",
    "               return 'N'\n",
    "            return self._map[tuple(s)].upper()\n",
    "        else:\n",
    "            return 'F'  # wrong input\n",
    "\n",
    "    def init(self, state=None):\n",
    "        if state is None:\n",
    "            s = [0, 0]\n",
    "        else:\n",
    "            s = state\n",
    "\n",
    "        if self.check_state(s) == 'O':\n",
    "            self._s = np.asarray(state)\n",
    "        else:\n",
    "            raise ValueError(\"Invalid state for init\")\n",
    "\n",
    "    def next(self, a):\n",
    "        s1 = self._s + self._actions[a]\n",
    "        # state transition\n",
    "        curr = self.check_state(s1)\n",
    "        \n",
    "        if curr == 'H' or curr == 'N':\n",
    "            return -5\n",
    "        elif curr == 'F':\n",
    "            warnings.warn(\"invalid state \" + str(s1))\n",
    "            return -5\n",
    "        elif curr == 'G':\n",
    "            self._s = s1\n",
    "            return 30\n",
    "        else:\n",
    "            self._s = s1\n",
    "            return -1\n",
    "        \n",
    "    def is_goal(self):\n",
    "        return self.check_state(self._s) == 'G'\n",
    "            \n",
    "    def get_actions(self):\n",
    "        return self._actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['O' 'O' 'O' 'H' 'O' 'O' 'O' 'O' 'O']\n",
      " ['O' 'O' 'O' 'H' 'O' 'O' 'H' 'O' 'O']\n",
      " ['O' 'O' 'O' 'O' 'O' 'O' 'H' 'O' 'O']\n",
      " ['O' 'O' 'O' 'O' 'H' 'H' 'H' 'O' 'O']\n",
      " ['O' 'O' 'H' 'O' 'O' 'O' 'O' 'O' 'H']\n",
      " ['O' 'O' 'H' 'O' 'O' 'O' 'G' 'O' 'O']\n",
      " ['O' 'O' 'O' 'O' 'O' 'O' 'O' 'O' 'O']]\n"
     ]
    }
   ],
   "source": [
    "env = GridWorld(\"grid.txt\")\n",
    "env.print_map()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agent\n",
    "\n",
    "Let us define an RL agent class. Two main methods are <code>train()</code> and <code>test()</code> as other ML classes. Unlike supervised learining, train function actively collect data while interacting with an environment and update its Q estimation with SARSA (or Q-learning). The next cell defines a utility to adjust coordianates to start from left-top. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def coord_convert(s, sz):\n",
    "    return [s[1], sz[0]-s[0]-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RLAgent: \n",
    "    \"\"\"\n",
    "        Reinforcement Learning Agent Model for training/testing\n",
    "        with Tabular function approximation\n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, evn):\n",
    "        self.env = env\n",
    "        self.size = env.get_size()\n",
    "        self.n_a = len(env.get_actions())\n",
    "        # self.Q table including the surrounding border\n",
    "        self.Q = np.zeros((self.size[0], self.size[1], self.n_a))\n",
    "    \n",
    "    def epsilon_greed(self, epsilon, s):\n",
    "        if np.random.uniform() < epsilon:\n",
    "            a = np.random.randint(self.n_a)\n",
    "        else:\n",
    "            i_max = np.where(self.Q[s[0], s[1], :] == np.max(self.Q[s[0], s[1], :]))[0]\n",
    "            a = int(np.random.choice(i_max))\n",
    "        return a\n",
    "    \n",
    "    def train(self, start, **params):\n",
    "        \n",
    "        # parameters\n",
    "        gamma = params.pop('gamma', 0.99)\n",
    "        alpha = params.pop('alpha', 0.1)\n",
    "        epsilon= params.pop('epsilon', 0.1)\n",
    "        maxiter= params.pop('maxiter', 1000) \n",
    "        maxstep= params.pop('maxstep', 1000)\n",
    "        \n",
    "        # init self.Q matrix\n",
    "        self.Q[...] = 0\n",
    "        self.Q[self.env._map == 'H'] = -np.inf\n",
    "        \n",
    "        # online train\n",
    "        # rewards and step trace\n",
    "        rtrace = []\n",
    "        steps = []\n",
    "        for j in range(maxiter):\n",
    "\n",
    "            env.init(start)\n",
    "            s = env.get_cur_state()\n",
    "            # selection an action\n",
    "            a = self.epsilon_greed(epsilon, s)\n",
    "\n",
    "            rewards = []\n",
    "            trace = np.array(coord_convert(s, self.size))\n",
    "            # run simulation for max number of steps \n",
    "            for step in range(maxstep):\n",
    "                # move\n",
    "                r = env.next(a)\n",
    "                s1 = env.get_cur_state()\n",
    "                a1 = self.epsilon_greed(epsilon, s1)\n",
    "\n",
    "                rewards.append(r)\n",
    "                trace = np.vstack((trace, coord_convert(s1, self.size)))\n",
    "\n",
    "                # ToDo: update self.Q table (SARSA)\n",
    "                self.Q[s[0], s[1], a] += alpha * (r + gamma * self.Q[s1[0], s1[1], a1] -\\\n",
    "                                             self.Q[s[0], s[1], a])\n",
    "                \n",
    "                if env.is_goal(): # reached the goal\n",
    "                    # ToDo: update Q table\n",
    "                    self.Q[s1[0], s1[1], a1] = 0\n",
    "                    break\n",
    "\n",
    "                s = s1\n",
    "                a = a1\n",
    "\n",
    "            rtrace.append(np.sum(rewards))\n",
    "            steps.append(step+1)\n",
    "        return rtrace, steps, trace # last trace of trajectory\n",
    "\n",
    "    def test(self, start, maxstep=1000):\n",
    "        \n",
    "        #### ToDo: Here is one episode of simulation for testing. \n",
    "        ####       Add one line to make this code test the trained agent\n",
    "        epsilon = 0.0\n",
    "        env.init(start)\n",
    "        s = env.get_cur_state()\n",
    "        # selection an action\n",
    "        a = self.epsilon_greed(epsilon, s)\n",
    "\n",
    "        trace = np.array(coord_convert(s, self.size))\n",
    "        # run simulation for max number of steps \n",
    "        for step in range(maxstep):\n",
    "            # move\n",
    "            r = env.next(a)\n",
    "            s1 = env.get_cur_state()\n",
    "            a1 = self.epsilon_greed(epsilon, s1)\n",
    "\n",
    "            trace = np.vstack((trace, coord_convert(s1, self.size)))\n",
    "\n",
    "            if env.is_goal(): # reached the goal\n",
    "                break\n",
    "\n",
    "            s = s1\n",
    "            a = a1 \n",
    "\n",
    "        return trace            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Plotting tools \n",
    "\n",
    "def plot_trace(agent, start, trace, title=\"test trajectory\"):\n",
    "    plt.plot(trace[:, 0], trace[:, 1], \"ko-\")\n",
    "    plt.text(env.goal_pos[1], agent.size[0]-env.goal_pos[0]-1, 'G')\n",
    "    plt.text(start[1], agent.size[0]-start[0]-1, 'S')\n",
    "    plt.xlim([0, agent.size[1]])\n",
    "    plt.ylim([0, agent.size[0]])\n",
    "    \n",
    "\n",
    "def plot_train(agent, rtrace, steps, trace, start):\n",
    "\n",
    "    fig = plt.figure(figsize=(10,8))\n",
    "    ax = fig.add_subplot(221)\n",
    "    plt.plot(rtrace, \"b-\")\n",
    "    plt.ylabel(\"sum of rewards\")\n",
    "\n",
    "    ax1 = fig.add_subplot(222)\n",
    "    plt.plot(steps)\n",
    "    plt.ylabel(\"# steps\")\n",
    "\n",
    "    # contour plot for agent.Q\n",
    "    ax2 = fig.add_subplot(223)\n",
    "    xs = range(agent.size[1])\n",
    "    ys = range(agent.size[0])\n",
    "    maxQ = np.max(agent.Q, axis=2)\n",
    "    h_b = (maxQ==-np.inf)\n",
    "    maxQ[h_b] = 0\n",
    "    maxQ[h_b] = np.min(maxQ) - 100\n",
    "    cs = plt.contourf(xs, ys[::-1], maxQ)\n",
    "    plt.colorbar(cs)\n",
    "    plt.text(env.goal_pos[1], agent.size[0]-env.goal_pos[0]-1, 'G')\n",
    "    plt.text(start[1], agent.size[0]-start[0]-1, 'S')\n",
    "    plt.ylabel(\"max agent.Q\")\n",
    "\n",
    "    # plot traces\n",
    "    ax3 = fig.add_subplot(224)\n",
    "    plot_trace(agent, start, trace, \"trace of the last episode\")\n",
    "\n",
    "    plt.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = RLAgent(env)\n",
    "start = [0,0]\n",
    "rtrace, steps, trace = agent.train(start, \n",
    "                                   amma=0.99, \n",
    "                                   alpha=0.1, \n",
    "                                   epsilon=0.1, \n",
    "                                   maxiter=100, \n",
    "                                   maxstep=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm0AAAHSCAYAAAC+f/yDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAACPvUlEQVR4nO3dd5hb5Zn38e890nSPu7GN24AxECDUoYVAqKGEhDQIJfUleBNIz24C8WY3zbtpm2waJCakkDghhLIhQCB0QoIBm9BMdcG9d08f6X7/eM4ZHWmkGUkj6Uia+3NduiQdHUnPyLbm5/tpoqoYY4wxxpjyVhN2A4wxxhhjzNAstBljjDHGVAALbcYYY4wxFcBCmzHGGGNMBbDQZowxxhhTASy0GWOMMcZUgGjYDSi2iRMnamtra9jNMMaUyJIlS7aq6qSw21EI9v1lzMgz2HdY1Ye21tZWFi9eHHYzjDElIiKrwm5Dodj3lzEjz2DfYdY9aowxxhhTASy0GWOMMcZUAAttxhiTBxH5hYhsFpEXUo5/UkReFpGlIvLtwPFrRGSZiLwiImeXvsXGmEpX9WPajDGmSH4F/Bi40T8gIqcBFwBHqGq3iOzjHT8EuBg4FNgXuF9EDlTVWMlbbYypWFZpM8aYPKjqo8D2lMMfB76pqt3eOZu94xcAN6lqt6quBJYBx5WsscaYqmChzRhjCudA4GQReUJEHhGRY73j04A1gfPWeseMMSZrFtpM3r7+dfjOd0r/vrfdBoccAgcfDEccAS++mP9rzZ0L116buL9qFZx2Grz6anbPv/FGuPRS6Ooa+NhPfwpveINr5zHHwNq1g7/WxRe7cw8+GD784eze/5//hDPPTDzPvxx7LKxbl9yWq64C1cyv9fzz7mf/85/d/W3b4IorBr72oYfCPfcM3q5vf3vg84KXT3968Lbcdx+cey5s3Jjd51BGosB44ATg34CbRURyeQERmSsii0Vk8ZYtW3JuwDW3Pc+DL2/K+XnGmPJnoc3k7be/ha9+Ffbsye15r7wCa9YMfs7u3fDkkwOPx+NwzTWwd68LbM8/D7femtv7+x5/HK6/Hr7wBdjsdWJ9+cvw8MPw3/898PxYDO6/H3p63P0VK+BjH4Pf/x4+9ankc//6V7jyShg1ygW3p5+Gu+/O3JbXXoM//AEmTIBJk+DXv4bnnst8fl+fa/exx8ILL8CRRyYuRxzhnvuVr7hz16+Hz37WhdPf/jbxGv/4B+zcmbj/la+4n/0d74C3vc2Fq1/9yrU/+Prr18MvfpF4Xk8PPPhg4n53t/v8amqSn+dfZs2CH/4wEfhV3ee1dau7v2wZXHihC4YXXgi9vZk/hzK0FrhNnSeBODARWAfMCJw33Ts2gKouUNU2VW2bNCm3NYJVlZsXr+HhV3IPe8aYCqCqVX055phj1BRHS4sqqP7sZ9k/Jx5X3X9/1fe8Z/DzvvAF1dpa1Y6O5ON/+pN7z9//3t0//HDVM8/Mrd2+97zH/QyRiOqnP636/POqIqrjxrn3XrduYJtA9bzzXLve+lbVUaNUr7jCHV+wwJ23YoV7jTe+UXXvXvczT5yo+uEPZ27LD37gXmP5ctWtW1Xr6lQ/8YnM5//sZ+78yy9X3b594OOf+pT7uV55RfVjH1ONRt1nNWGC6pYtqv/93+75b32ra99rr7mf/V//VXX+fNX6etXjj1d97rmBr/2hD7nXicWS2/7Xv7r7//d/7v7dd6dvezyuetFFqjU17jM75RR3/oQJqtdfr3rYYe72t7/tjn/yk5k/h3SAxVqi7xegFXghcP9jwNe82wfiukQFNwHhWaAe2A9YAUSGev1cv7+6e2M664t36mdv+mduH5oxpmwM9h0Weqgq9sVCW3Hs3ev+9oBqW1v2z1u2zD3npJMGP6+tzZ23dGny8VNOUZ05U7W3193/xCdUm5pUe3pya/+yZS6kXHONCz51dapvfrPq6NGqTz7pAsU11yTOv/lm1543vck974AD3P0f/Ui1r0/17LNdSJo61QW5sWPde/je8Q7VAw/M3J6zz1Y96KDE/UsvVR0zRrW9Pf35xxzjQlg8nv7xTZtUm5vd5xWNql55pQul0ajq7Nmu7Ycc4q5/+1v3eF2d6vr17vm7diVCWaobb3TPe/ppd98PXWef7e5fdJELqYP9mezZ48IZuM/qu991ny24z94PgJ/9rDt2442ZXytVqUIb8HtgA9CLq7BdDtQBvwVeAJ4GTg+cPw9YDrwCnJvNe+T6/bWzvUdnffFO/eivn8rpecaY8mGhzRScH76OOSb5F/hQfvrTRGDIZPdu94sbVO+4I3H8iSfcse9/P3Hsj390xxYtSn6NeFz1ttsyB4dPfMJV09avV1292lWWQPXrX3ePv/e9Lkzs3Kl6zz0uAJ14omp3tws5kYirRPX1ufO3bXNVqiuuUJ07V/Xxx5Pf75vfdK+/ZcvAtrS3u/f/zGcSxx56yJ3/618PPH/JkkRgHMy//7s7r6kpEcbmzXPHrrjCfTbHH+8CVmOj6kc+Mvjr+datc6/x7W+rbt7s/qz23dcd+/vfVRsaXAgcysqVqv/xH6obN7r7sZjqL3+p+oc/JM7p6VH9wAfcz5ytUlbain3J9ftr/c4OnfXFO/WSBY8PfbIxpixZaDMF99hj7m/PTTe5X9If/3h2z7vwQve8qVMzn3PPPdpfxQsGtPe9z1Wfdu9OHNu4MREggu6/XzN23W7c6IJMMKRcc42r4O3Z4+4vWuSe39DgrqdNS+4uff55F9Sy9cgjOiCE+u68U5O6F1Vd6Jwzx1X/Un3sY65d6bpFg3budGHqq19NHOvrc8HKr9A9+6yrvoHqCy9k//O84Q2usvbzn7vnPvCA+0z98PbYY9m/VqGN5NC2fPMenfXFO/XtP/pbTs8zxpSPwb7DbCKCycuGDe76DW9wg8UXLkweMP7JT7rZpUHxeGLA+vbU1a0CHnkEIhFobobly90xVbj3XvdeLS2JcydPhoMOcs8Juu8+d33bbcnH+/rgfe9zbfnCFxLH58937zVqlLt//PFutuW73w1//CO8/DLsu2/i/MMOg/HjM/8MqdraIBp1g//BzTb1Z0befTc0NcEppyTOF4GPfhQeeyzxHID2dvdZX3ghjBs3+HuOGeNmw375y4ljkQi86U3u9QEOP9xNUJg3z80KzdaZZ8Kjj7rJE7NmuVmnl1/uJinMmgUnnpj9a5nC6ehxa/Xu7eoLuSXGmGKw0GZ46qnk2X/Z8APHlCkuBOzenZj9B/CXv8BDDyU/59ln3TIShxziZhh2dqZ/7UcfdSHnoIPcTEJwy1fs3OlmH6Z6y1tcuIkF1pa//353/eCDsGtX4vgXv+gC3oIFbnakT8SFqqAf/9gFpPe+NxHm8tXUBEcd5WasqsJ73gMzZ8J//qcLbWecAfX1yc+ZOxf22w8uugg2eSs43Hijm617xRXZvW80mghomVxxBXzjG7n9PGec4f787rsP3vUu9x6f+Yx7v/e/380cNaXX2euFtm4LbcZUI/tqHeFUXSg44wx37VfQhrJhg6vaTJzoLuACmW/btuSwBIkg9Z73uOsdOwa+bmenW+rjLW+B2bMTlbbnn3fXb3zjwOeccop7L/+c7dvdEhtvfaur/t11lzt+yy3wve+5KuAHPpDdz1lIb3qT+9l+8xsX1I44Ar72NXj9dTjvvIHnjx0Lt9/ufp4LL4SPf9wtI9LWBm9+c6lbn+zUUxPB7F3vctf77++WH/n3fw+tWSNeZ4+FNmOqmYW2Ee6ZZ1xoOPtsuOMOtwhscKHYuXPdGl+pNmxwXZM1NW5tMUhU2np7XVUsNbQ98IDrTvW74dKFtieecM8/5RQX2l5/3VXQhgpt4Cp04NYaU3VdflOmuOCzbZsLPMceC9/97pAfS1G86U0ulM6dC8cdB4sWuS7fSy91oSydI46An/8c/vY3Vx383OdcBTO35VoLb8wY9zNMmgQnnZQ4ftBB0NAQXrtGOr97tKMnRiw+yOrFxpiKZBvGj3C33+6C129+41bCv/xyNw7qoIPc4w884MaWpdq4EaZOdbf9Spsf2vzxasHQ1t3tgsfllyfGgvmhTdV1Rc6eDX//uwskb36zC4a9vW4h3uefh2nT0o/jmjEDDjjAdR1edZWr6I0a5cZVXXCBW1D2U59y73f//VBXN7zPLF/+OK9YzC3qG4m4auBb3zr48y691HWd7r+/62ItF9dd57pqI5GwW2J8nb2JClt7Tx+jG2pDbI0xptAstI1wt9/uAtKkSW78FLjxYwcd5MLU+vUDx1qBC1TTvJ0TU0Obf71rl3sNEdfN2dEBJ5yQCF5+uHv++eQdBY46ylVyDjjA3V++3HW7pauy+b7+dbjkEvjJT1zQfMtboLbWdd397Gfwu9+5nRQOPzz3z6hQZsxw3YpnnZV7O/wu5XKSbnyhCVdnT7z/9t4uC23GVJuK6x4VkXNE5BURWSYiV4fdnnLw3HPp977s6HDB6s470z9v2TIXhvwxSX4I8/es3LHDve6uXQO7OoOVNr971B/T5l/39iba5VfVJk5MhDb/mD+O7r/+y43b8mc7zp7trl9+GV56afDQ9r73ub0qr77a7Rt6xhnu+GmnuQA4e3byLMqwPPQQfOlLYbfCVKuOnkClzca1GVN1Kiq0iUgE+AlwLnAIcImIHBJuq8K1fj0cfbTbIzLVhg3u8SVL0j/3//7PXb/zne46NbQFNxwP7hUai7m9OqdMcffr6twyHKmVNkiEPT+gjRs3MLT5MyMvvNAtPxEMkfX1btxXd/fgoU3EPdcf63XmmYm23X23m83a2Jj5+cZUA38iAsAeC23GVJ2KCm3AccAyVV2hqj3ATcAFIbcpVP/4hwtR6TZg97sf/VAEbqD/m9/sBun/9reuK7K11T3W3OyqUulC2+rVidtbtrh1zvxKG7gKWjahbexY9x7BY377Jk9Obn9Njeuy/etf3f3BQhu4n+MHP3CB7bDDEsff9CaYM2fw5xpTDfwlP8DWajOmGlVaaJuG24DZt9Y7NmL5C68Gl9vw+aHIX1MN3Lpaf/87fOtbbt20d787+TnTpg0d2vzuTL/SBtmHtnHj3MD1sWOT29fYmH4ttNmzXZUtEnEzT4fy0Y+6nzHs2ZXGhKEjUGmz7lFjqk9VTkQQkbnAXICZM2eG3JriGiy0pau0bdjgxqC98opbkPass5Kfky60RSLJoc0PgZkqbcG2pKu0gQtvwfZNnpw+aPnj2g48MP2ECGNMgnWPGlPdKq3Stg6YEbg/3TuWRFUXqGqbqrZNmjSpZI0rta4ut4gsJFe3fKndj+DGuE2d6oLbBRe4lfqDUkPbpElu1uNQlbYJE4autLW0JHYdGDcuuX2pXaM+fwZpsLvTGJNeZ2+MsU1uxqhV2oypPpUW2p4C5ojIfiJSB1wM3BFym0KzZImbodnYmFulLVghSzVtmqukxWIutE2b5rZbSldpG6x71F8GxA9tO3cmr7GWbWjzK21DjWczxrju0UmjXEnaxrQZU30qKrSpah/wCeBe4CXgZlVdGm6rwuN3jZ555uChbe9et/wHZBfaYjEXpDKFtg0b3GSC4GzMiRPdQqs9PS60+WErWGkLhrbx47MLbUcd5Sp0p5+euc3GGKezt4/RjbXURWsKupXV4te3s6O9p2CvZ4zJT0WFNgBVvVtVD1TV2ao6P+z2hOkf/3Ddhwcf7EKbpuxaE9wmatMm9/iGDbDvvplfM7jsRzC0rVuX2JA9XfAL7j+6bVtiRmqm0OZX2mIxF/KCVbugqVPdZvTBrZKMKQci8gsR2SwiL6R57PMioiIy0bsvIvJDb33J50Tk6GK0qbMnRlNdhJb6aMFCm6py6c+f4NePv16Q1zPG5K/iQptxVF1oe9Ob3Hiy7m5ob08+x6+0gevS3LbNdacOVWkDWLnSLe0xbZob09bXl+gW3bhxYMgK7j+6daurnLW0JIc2fxICJCYibN3qlg/JVGkzpoz9Cjgn9aCIzADeCgTq05wLzPEuc4HritGgjp4YDbURRjUULrT1xZWevjg7O3oL8nrGmPxZaKtQK1a4BW7f9KbkKlfQ9u0uOIGrtPkTCLIJbYsXJ+77E3D9LtLBKm0bNrigNmGC60IdrNLW2+t+DrDQZiqPqj4KbE/z0PeBLwDB2vcFwI3qLALGisgg/xLz09nrKm3NddGCjWnri7kfo5DdrcaY/Fhoq1BPPumuTzhh4DZSvh07XNcpuNC2fr27PVho22cfN8PTf/3U0KaavIWVzw9tr76auD9YaPM3jX/5ZXdtoc1UAxG5AFinqs+mPFSSNSY7e2I0FrjS1hNz+5nabFRjwmehrUL549WmTEnumgzavj05tPmVtsHGtNXUuECWqdK2Z4+b1JDaPeqHtldeSdz3Q1tPj3tOaqUNLLSZ6iEiTcCXgP8Y5uvMFZHFIrJ4y5YtOT23sydGY4HHtPV5oc0qbcaEz0Jbher1hpfU1mbuHt2xw4Wh8eOz7x4FF9T88XHTpsHo0S6ArV4Nt9/ujqeum5ZaOQuGtp073bF0oe2ll9y1hTZTBWYD+wHPisjruHUknxaRKWS5xiTkv86kqtLhd48WMrTFrXvUmHJRlTsijATB0Jaue7Sz013Gj3eBaNMmV0VLXaojHX9cW0NDIlzNnAmvvQZ//jMccwyckzL8uq7OhTs/tPlj2l57LXkLK1+w0lZf755rTCVT1eeBffz7XnBrU9WtInIH8AkRuQk4HtilqhsK+f69MSUW1/7u0UJ1Z/b0WfeoMeXCQluF6vGWTKqtTYSwYGgLBiU/tMHQVTZIhLZp0xJbS82cCXfd5W4vWJB+y6mJExMTC4KVtnShza/MrViR/D7GVAoR+T1wKjBRRNYC/6mqN2Q4/W7gPGAZ0AF8pNDt8bewaqyLMqo+yp5CTUTwKm3t3bEhzjTGFJuFtgoVrLRFIi4gBce0+UHJr7QtWeKW1hhsPJsvGNp8/ri2U08duF+pLxjagrNHB6u0xWLWNWoqk6peMsTjrYHbClxVzPZ09LqQ1lQXYVR9lO6+OL2xOLWR4Y2C8ce07emyJT+MCZuNaatQvb2uuzMScfcnTkyutPlrtAUrbf6+o0NJF9r8HQ7+678yV8X8btrmZte1OmaMqwj667sF12lraXHtBwttxhRCf6Wt1oU2KEyXZq+35Ed7TwxNXcHbGFNSFtoqVG+vq7L5JkxI3z3qV9r27IG1a/MPbXPnwmOPwYknZn6ePyHCvx4zxl2//rq7DlbaamoSIS7TbgjGmOx19HePJkJbIbpIe71KWyyudHvj24wx4bDQVqHShbZg96hfafNDG7hdDbIJbdOnu+tgaGtpGXorqVxCm982sEqbMYXQ2etCW1Odm4gA0N4z/NDWF08EtUKNkzPG5MdCW4Xq6Rm80pbaPerLZkzb7Nnw/e/DpZfm1qbBQltTk5thGuSHOAttxgxfuu7RQuyK4HePgs0gNSZsNhGhQvX2Joeg1DFtO3a4LsjRo5O7H7OptInAZz6Te5v8MW3+tR/aVq0aWGUDC23GFFKwe1S8gad7ChCy+gKhzdZqMyZcVmkLyZ49iXFn+UjXPbp3r9s4HlylbexYF9yCoSib0JavTJW2NWsstBlTbJ39s0ejtDQUciJConvUKm3GhMtCW0iuugrOPz//56cLbZCotm3fnhgzts8+ifPCCG3xePrQZmPajCmczh4XrgrfPZoIbVZpMyZcFtpCsn49PPVUYpHcXKWOaUvdyiq4QXt9vau6jRrlJhQUS6buUUhe7sNnlTZjCqfDm3TQ6G1jBQND1pJVO9i2t7v/fl8szoMvbxp0KQ9/cd10r2eMKS0LbSHp6HDVMn/vzVyljmkbrNIGLhgVs8oGcNBBcMUVcN557n5wa6p0lbZ3vxs+//n0gc4Ykxt/IkJTYMmP1JD1oV88yQ2Prey///ArW/h/v1rMc2t3ZXzd5O5R2xXBmDDZRISQdHS462efhSOOAFW4/noXYM49d+iKWKbuUX/Zjx074IADEo8fdJB7j2KqrXVbXPkiEVfd27s3fWg75hh3McYMX2dvjGiN9O+A0FQXSeoeVVX2dvexvT1R3vdvr9i6lyNmjE37ujZ71JjyYZW2kHR2uutnnnHXr70G//Iv8L73ua7Or3998OdnM6YtGJR++1tYuLAgTc+J30WaLrQZYwqnoydGY12k/35zfTRpnTa/m3N3YDsq//aqbR0ZX7cvUGkrxGxUY0z+LLSFxK+0+aHtgQfc9cKF8OY3wze/mZgJms5goS0ed5W2YPdoS0txx7NlYqHNmNLo7InRFAhtLSmbxvvdnLs7E8d2d7rQtnqQ0NYbt0qbMeXCQltIgt2jqnD//W5T9ksugc99zj3+6KOZn586EaGx0S1gu20b7N7tXrMcgpKFNmNKo7M3RmNtcqUtOKatt8+Frz1JlTb3+Krtg4Q2b+uqumjNsEPb7q5ebnz8ddvD1Jg8WWgLSWenq3xt3w6rV8NDD8EZZ7iFbU87zc34/MtfMj8/dSICJLayCm5hFTYLbcaUhuseTQxTHlUfTQpZPX6lLVB9y6p71NvGalxT7bBnj97/4ib+409LB30/Y0xmFtpCoOpC2/HHu/u/+pXrzjzzTHe/qQlOPRXuvjvza6R2j0JiV4RyDG02Q9SY4urs7UvqHh3VkNw92h/aOgOVNq+rdOve7oxVNH8iwtjGumGHtq7eeFJbjDG5sdAWgq4ud33CCe76xz9216efnjjnvPPglVdgxYr0r5EutB14oOtm9ScclEN1yyptxpRGZ09y9+ioAd2jfqWtt797MjgpYXWGLlJ/LNzYptphd49297klQ3r6LLQZkw8LbSHwx7Pts49blmPrVjjssOQ9Qs89111n6iJNHdMGLvwdeij87/+6++VUabPQZqqNiPxCRDaLyAuBY98RkZdF5DkRuV1ExgYeu0ZElonIKyJydqHbkzp7tD5akxSO/PDVG9P+itfuzl72HdMAZO4i7YspNQItDbXsHeY6bX57ggv2GmOyZ6EtBH5oa2pya7SBG88WNGeOC3SZQlu6MW0TJ8KDD8JJJ7mxccHtq8IybRo0NJRHgDSmwH4FnJNy7D7gMFU9HHgVuAZARA4BLgYO9Z5zrYhEKKDO3uTZo7WRmqRuyOBtv8K2p6uPQ6e5/1mt2tae9nV743GikRpaGqLDrrT5oa3XukeNyYuFthD4a7Q1NcGRR7rb/ni2oPPOcyHMPz8oXfcouMrWfffBk0/CpEkFa3Le5s6Fp592wc2YaqKqjwLbU479VVX9ZLMImO7dvgC4SVW7VXUlsAw4rpDtSe0erYvW9HeJQvIiuf64tt2dvUwf18jYptqMM0j7YkptjdBcHxn2mDY/OPZa96gxebHQVmDxOJx8Mnz4w4ndCVL5lbbGRrjwQrjoouTxbL53vMMFtnTVtkyhzX/dtra8ml9wjY3whjeE3QpjQvH/AP9f7zRgTeCxtd6xAURkrogsFpHFW7ZsyfrNOlO6R+uiyZW23qRKWx/xuLK3p4/RDbXMGt+Uca223pirtKUuIZKP/kqbdY8ak5eyC20i8hURWSciz3iX8wKPFXVMSCF0dMBjj8Gvfw0HH5x+Bmiwe/Sgg+APf3C3U516qtsz9He/G/jYYKHNGBMuEZkH9AE570OiqgtUtU1V2yZlWS5XVTrSdI/2xrR/0kGwurW7q5c93X2owujGWmZOaGbV9gzdozGlNlLDqLooPX3xYU0i6O6zSpsxw1F2oc3zfVU90rvcDaUZE1IIe/e66098wi1z8aUvDTzH7+5sbBz8tSIRuPhiuPNO2JWyn3O6iQjGmPCJyIeB84HLNLGK7DpgRuC06d6xguiNKbG4JnWP1kfd17tfbesOVto6e/u7SEc3RJk1von1O7vSjjXri8WpjQijGtwacMMZ19bfPWpj2ozJS7mGtnSKPiYkkx07YP367M5t9/6zeuyxcNxxiapaULDSNpRLL3XbWd1+e/LxdBMRjDHhEpFzgC8A71DV4L/+O4CLRaReRPYD5gBPFup9O3vcrM7g4rp13sbxPWmqW7u7+vonI7hKWxOxuLJux8ABtK57VGiud689nC5S6x41ZnjKNbR9wpsy/wsR8ReLKPqYkEz+7d/gne/M7ly/0jZqlNvVwF+TLSiX0HbssTB79sAuUuseNSZcIvJ74HHgIBFZKyKXAz8GWoD7vOEdPwVQ1aXAzcCLwD3AVao6vPUzAjp6XZAKVtpqIwIkJiCkTkTwF9ZtaYjSOqEZSL+dVW/c6x71QltwE/pcpQuQxpjsRYc+pfBE5H5gSpqH5gHXAV8H1Lv+H9yA3qyp6gJgAUBbW9uw/0u3YQNs3pzduX6lrbnZzZhMF9qy7R4Ft3THpZfC/PmwcaNbyy0eh1jMQpsxYVLVS9IcvmGQ8+cD84vRFr/S1pQ0EcHdTrfMxu6u3kSlraGWSS31AKze1g4kj6Pri8WprQmEtmFU2vzFda171Jj8DFlpE5HZIlLv3T5VRD4VXDAyH6p6pqoelubyJ1XdpKoxVY0D15PoAi3qmJDBtLenD1+ZzoXBQ1sulTZw49ricbjjDne/11vE3EKbMQbcwrrAgNmjkAhtSeu0dfb1j2kb01jLPi31RGuEDbsGfmH1xTSpezS4NVaurHvUmOHJpnv0ViAmIgfgqlczgDTzGQtDRKYG7r4L8FcbL+qYkMG0t6dfKy2dYPdoQ4Mbj5YquE5bNvbf311v2+au/dBmY9qMMeAW1oX03aOpg/9b6qPs6ertD1+jG2oRERrrIv2vE9TjLfmRqLTl36tr67QZMzzZdI/GVbVPRN4F/EhVfyQi/yxim74tIkfiukdfB/4F3JgQEfHHhPRR4DEhg8m30lZf72Z5xuNQE4jHwXXasuGHMz8AWqXNGBOUrnu0Ppp+IsKEUXVJExH8WaGNtRG60oS2vphSF3GL68IwZ4/ajgjGDEs2oa1XRC4BPgS83TtWtLigqh8Y5LGijQkZTHt7+vCVTmqlDVzYCga0jg4XxCJZLlhSU+MCmoU2Y0w608c18snTD2DauMQXTW0keckP/3rCqPr+iQij6qNEalxFrqE20r8naVBfPE60poaWeveFU4jZo7b3qDH5yaZ79CPAicB8VV3pdU3+prjNKi9+9SybalvqmDYY2EXa2Zl9lc0X7Gq10GaMCdp/0ig+/9aDmDom8cXij2kLbhQPMKG5rn8iwuiGxP/bG2sj/RW7oJ7+MW3uf5nDCW3+4rrDWaDXmJFsyEqbqr4IfCpwfyXwrWI2qtz43ZldXUOPQ0vtHvWfl/p62Y5n8wWXD+npcdcW2owxmaSu09YT6B59erWbiDC6MfEl0lBbk3ZMW18sTl2khmikhvpoTUEW1+2LW2gzJh8ZQ5uIPI8bV5aWqh5elBaVmXg8MXEgm8kIe/e6qlgkkqi0FSq0pVbabCKCMSaT2pQdEXq9nQ1GN9QGKm3B0JZ5TFvUm9TQ0jC8/UcTY9qse9SYfAxWaTvfu77Ku/a7RN/PIGGu2gR3NMi2e7TZrVNZ0O7RdKHNKm3GmEwG7IgQi1MbqWF0Yy09fXG27u2hdULif4+NdRF2tPcMeJ3euJs9CtBcHy3IRATrHjUmPxlDm6quAhCRs1T1qMBDXxSRp4Gri924cuB3d0L2lbZRo9ztYlfaLLQZYzIZMHs0ptRFa/rHsa3b0ckbp43pP78hmn7Jj95YnFpvskJz3fAqbd191j1qzHBkMxFBROSkwJ03Zfm8qhAMbblW2go9ps1CmzEmW/7s0d7A7FG/0gZubbekiQh1GWaPxrT/tUbVF6h7tG/EdNYYU1DZLPnx/4Bfioj/X7Kd5LitVCXLtdKWrns0NbR1dsLEibm1Ixja/IkINqbNGJPJgB0R+tyEguA4tuSJCJkqbdrfPTqqIcqWPWlWDM9S6kK/xpjcDBraRCQCvEVVj/BDm6ruKknLykSulbZ03aOpY9ryqbQFt8SySpsxZih1mSYiNCa+9pMnItTQlWbJD/954BbvzXfD+FhciXnrs9k2VsbkZ9BuTm/HgUu827tGWmCD4nSPdnZa96gxprhqM0xEaEmqtCWv09bVl37JD/+16qMRutN0oWYjOPnAtrEyJj/ZdI/+XUR+DPwB6I8wqvp00VpVRoo1EcFmjxpjiqk+pdLW0+fGpiV1jwZuN9ZG6I0pfbHEbFFwVTF/yY/62pr+yQS5Sgpt1j1qTF6yCW1HetdfCxxT4PSCt6YMDafSVsju0XRj2iy0GWMySVtpi9YkVddaUtZpA+jqizMqENr6YnFqvf37GqIRutNU47LRHUs8z7pHjclPNjsinFaKhpSr4UxEGKx7tBCVNpuIYEx4ROQXuPUsN6vqYd6x8bheiVbgdeAiVd0hIgL8ADgP6AA+XOzeikiNEKmRwDZWceojNTTWRojWCH1xTQpwDd5m8509MUbVu+OxuBJXkitt1j1qTGiyWrpDRN4mIl8Qkf/wL8VuWLnIpdKmOnT3aG+vu9iYNmMq3q+Ac1KOXQ08oKpzgAdIrGd5LjDHu8wFritFA+siNSmVNkFE+meNJk1E8LpTg7si+IEvMaathp5YnHgelbJu6x41ZtiGDG0i8lPgfcAnAQEuBGYVuV1lI5dKW08PxGKDL/nhv4bNHjWmsqnqo8D2lMMXAL/2bv8aeGfg+I3qLALGisjUYrexNiL9W0b19CUmFPjrswWX/Gj0Km3B0NbnhTN/9mh91J3Tk0foSqq0WfeoMXnJptL2JlX9ILBDVb8KnAgcWNxmlY/2dvCGcwxZadu7110PtuSHH9psIoIxVWmyqm7wbm8EJnu3pwFrAuet9Y4VVV000l/h6gkskuuHtZaG5NmjQNJabX1eOIvWJCptQF5dpH5oqxHrHjUmX9mENr++1CEi+wK9QNH/h1gu/DFq9fVDV9r8qpxfaYtGXeALhj1/L9NCTESwMW3GDJ+IXCgiLd7tfxeR20Tk6OG+rqoqeezTLCJzRWSxiCzesmXLsNpQH03uHvX3I21piNJYG+kPcZCYiNAZWKvNr6j5m8/75+QzGcF/reb6qHWPGpOnbELbnSIyFvgO8DRucO3vitimsuKHtsbG3CttkNytCcMLbb29EI9bpc2YAvuyqu4RkTcDZwI3kP+Ys01+t6d3vdk7vg6YEThvundsAFVdoKptqto2adKkPJvhuO7R5MV1wY1lC05CgOTZo74+r2vV33u0vn/cW/6Vtua6aH+3qzEmN0OGNlX9uqruVNVbcWPZDlbVETURobl5YPjKdC4kKm3gnleo7lFwVTYLbcYUlF82ehuwQFXvAvKtY98BfMi7/SHgT4HjHxTnBGBXoBu1aOqClbbAmLaLjp3Bx98yO+ncxjSVNj+0+eu21dd63aP5VNr80FYfSRrfZozJ3pBLfojIY8AjwN+Av4+0XRH80BaP5949Ci5sFarSBi4AWmgzpqDWicjPgLOAb4lIPdlN0vo9cCowUUTWAv8JfBO4WUQuB1YBF3mn341b7mMZbsmPjxT6h0inzpvtCW5Mm7+11WkH7QMHJZ/bkCaQ9XePpkxEyGeB3e6+RPfo3u4sFr00xgyQzeK6HwBOBt4DfEdEuoG/qepni9qyMuGHtt7ewnSP5ltpC85EtcV1jSmoi3BLd3xXVXd63Zr/NtSTVPWSDA+dkeZcBa4aVivzUBupSekezZxFG+vSVNriA5f8gGGOaauL9s9oNcbkJpvFdVeKSBfQ411OA95Q7IaVi/Z2F8K6u/OrtKV2jxaq0lZTA5FIbq9hjBlIVTtE5HXgXBE5B9ej8NeQm1UQdZHEtlM9ffH+Sls6DdF0s0e97tGUMW3DmT3aXB+12aPG5CmbLoDlwP/hpq7fABymqqkLSlatXMa0FXsiAiRCm1XZjCkMb7HwXwMTgInAL0Xk38NtVWHUpcwe9bs500ms0zZwEVy/0paYPZpP96gLg831EXrjFtqMyUc23aM/BN4MXAIcBTwiIo+q6vKitqxMdHS40LZnT2HGtA13IoKFNmMK7jLgCFXtAhCRbwLPAN8Is1GFUOd1j8bjSl9cB+0e9atonUk7IvgTERLbWEHyArzZSqq0WfeoMXnJZvboD1T1QtxU+CXAV4BXi9yuspHLkh+Zuket0mZMWVsPNATu15NhOY5K41faelPGpqUjIjTU1iTviDBgG6v8K21+aBtVH3V7mtqyH8bkLJvZo/+Dq7SNAv4B/AduJumIkGv3aCSSCFjgnrc9sNFNIUJbT48trGtMAe0ClorIfbjFcM8CnhSRHwKo6qfCbNxw+BMR/MpW3SChDdyyH0l7jw7YxqoAS37URb3XjlNfYwNzjclFNt2jjwPfVtVNxW5MuVFNrrRl0z3a3AwSGDaSrntUJDnYZcMqbcYUze3exfdwSO0ouP5KW1/y0h2ZNNZGkmaP+s8bsI1VPpW2WBwRaKxzr9EbU+qz+Q1kjOmXzT+Z24BLRWQ/Vf26iMwEpqjqk0VuW+i6u936bLlU2oKTECB992hjY3Kwy0ZwyQ8LbcYUjqr+WkQagZmq+krY7Skkf502f0JBXXTwylZDbSR59mg8w0SEPGeP1kVq+l+rty/uOqKNMVnLZhurn+A2ib/Uu7/HO1b1gmPUcqm0BaVb8iPXSQhglTZjikVE3o6beHCPd/9IEbkj1EYVSF3EVdq6s6y0NdRGUmaPpu8ezWciQre35Eh/aLMZpMbkLJvQdryqXgV0AajqDvLf4gXo36B5qYjERaQt5bFrRGSZiLwiImcHjp/jHVsmIlcP5/2zFQxt2W5jlS60pXaP5jqeDWxMmzFF9BXgOGAngKo+A+wfXnMKZ2ClbYgxbXXJY9r8Spu/jVU0UkOkRvLuHq2P1vQHQJtBakzusgltvSISwQ3QRUQmAcP9L9ILwLuBR4MHReQQ4GLgUNwK5deKSMR7/58A5wKHAJd45xZVamjr7HTj3DJJ1z2abhur4YY2q7QZU1C9abbnq4oyUG1E3Ji2/orZ4F/5qbNHe/uSF9cFV23LdyLCgO5RY0xOsgltP8QN0t1HROYDjwH/NZw3VdWXMowduQC4SVW7VXUlbp++47zLMlVdoao9wE3euUWV2j0KiS2kMp2fTaXNukeNKStLReRSICIic0TkR7iZ8hWvLhIhronuzKFCW2PKmDa/CzNYoXOhLb+9R4Pdo33WPWpMzgb9FywiNcBK4AvAfwMbgHeq6h+L1J5pwJrA/bXesUzH0xKRuSKyWEQWb9myJe/G+KGtqSkxEWCwcW2ZJiJ0dycqdFZpM6bsfBJX3e8GfodbAuTTobaoQPyw1d7dBww9pq0+dSJCbGClraE2kudEhJgX2sS7b92jxuRq0NmjqhoXkZ+o6lHAy7m8sIjcD0xJ89A8Vf1TLq+VK1VdACwAaGtry/ubIV2lbbBxbekqbX7Y6ulxt/MNbTZ71JiieZuqzgPm+QdE5EKgWP85LRk/IO31Qls267R1p9nGKhpJrbTl1z1aH41Ypc2YYchmyY8HROQ9wG2qg43oSqaqZ+bRnnXAjMD96SRWJs90vGhSx7RB7qEt+Lz6elepmzAh97akTkQYMyb31zDGpHUNAwNaumMVx5/t2eGtvTbkRITU7tE0i/LWR5NnmGarJ5YyezRmoc2YXGUT2v4F+BzQJyJdgACqqqOL0J47gN+JyPeAfYE5wJPee84Rkf1wYe1iEkuQFE26Sls+3aOQWPYj30qbX1mz7lFjCkNEzgXOA6b5ux94RgN94bSqsPyQtre/e3ToiQjBxXX7+ittgYkItcObiBC17lFj8jZkaFPVlkK/qYi8C/gRMAm4S0SeUdWzVXWpiNwMvIj70rxKVWPecz4B3AtEgF+o6tJCtytVLpW2WMw9lql71H9evqHN30XBQpsxBbMeWAy8A7evsm8P8NnhvLCIfBb4KG7W/fPAR4CpuElUE7z3+4A3sapo/JDWnmVoa6yN0NUXQ1URkf5trAbOHs1vcd2mpmh/1c66R43JXSibiKhq6rYxwcfmA/PTHL8buLvITUuSLrRlqrT552aqtPmhLd/Zo2ChzZhCUtVngWdF5Heq2gsgIuOAGd56lHkRkWnAp4BDVLXT+4/oxbiq3vdV9SYR+SlwOXDdsH+QQaRORKiLDrG4bl0EVTfTs6E2Qm8sTrRGEAmGtuQu1GwNWFzXukeNyVk2S36MWMHZo0NNRAgGvKBCdY9CcmizxXWNKZj7RGS0iIwHngauF5HvD/M1o0CjiESBJtzM+9OBW7zHfw28c5jvMSS/qrW3O7slPxqiydtU9cXiA57TkG/3qDemzbpHjclfxn/B3vixEa293YW1mpqBlbann4bdu5PPhcEnIqjmv42V/1pdXW4iglXajCmYMaq6G7fg942qejxwRr4vpqrrgO8Cq3FhbReuO3Snqvpj5QZdtqhQavsnImTZPVrnQptfSeuNadJ4NhjGRIS+OPWRGuseNWYYBvsXfAuAiDxQoraUneBs0GClrbcX3vQm+GFg6PLeve463Y4I/vM6Otzt1GCXLeseNaYooiIyFbgIuHO4L+Z1sV4A7IebUNWM2+El2+cXZJ1JgPpI7hMRIBjaBlba8l3yw+8ejVr3qDF5G2xMW42IfAk4UEQ+l/qgqn6veM0qD8HQFqyYbd7swtPq1cnnwuCVtm3b3O2JE/Nrj4U2Y4ria7hJTo+p6lMisj/w2jBe70xgpapuARCR24CTgLEiEvWqbRmXLSrUOpOQqLQlxrQNPREBEjso9MV0wIK89bU1eS6um7L3qHWPGpOzwf4FXwzEcMGuJc2l6qWrtHV2wqZN7rZ/DZkrbcExbX5oy2edNrDQZkwxqOofVfVwVb3Su79CVd8zjJdcDZwgIk3iRvCfgZsR/xDwXu+cDwFFXWQcEmPa2v112oastKV0j8bjRGtSK22RvGeP1kUT3aO91j1qTM4yVtq8vUG/JSLPqepfStimspGp0rZxo7vtX/vnwuBLfmzd6m4Pt9LW02MTEYwpV6r6hIjcgpvU0Af8E1c5uwu4SUS+4R27odhtyXUbKz+0dfUMUWkb1kQE2zDemHxls+THP7zFbk/x7j8CfE1VdxWvWeUhXWjLVGnzq2jjxiW/Rur2U2Ddo8ZUO1X9T+A/Uw6vAI4rZTtqA2PaRCBSM3ho6+8e7UuMaYsOGNPmJiL4a7llIxZXYnGlLhJJdI/GrHvUmFxls+THL3CLTV7kXXYDvyxmo8pFR0f6SlswtPkbe61b5xbAnTo1+TWC3aPDrbQ1NLjQGItZaDPGDK2+v9IWozZSM2TI6p892uOqYL0xTTsRAVzlLFs9XlUtaZ026x41JmfZhLbZqvqf3jiPFar6VWD/YjesHAQrbdGouwQrbV1dsGePu71uHeyzz8AwlToRQWRgNS5b9fWJblgLbcYUhoj8e+B2fZhtKbRg9+hQ49kgsU5b/0SEeHxg96j3mrmMa0sb2mwigjE5yya0dYrIm/07InISMMgOnNUjdQP4xsbkShskbq9bB9PSrLqUOqZt3DiIRPJrT319IiTamDZjhkdEvigiJ5KYHADweFjtKQY/IHX2xoYczwbQUJe85EdfTJO2sAKor01egDcb3bHEhvWRGqFGbJ02Y/KRzZi2jwE3isgY7/4O3Mynqpca2vzuydTQNmeOC22zZg18jdTu0Xy7RsGFNn+WqlXajBm2l4ELgf1F5G/e/QkicpA3EaviBZf4GGq5DwhMRPBCW0/aMW1+pS37yQh+wPPXjauN1OTUvWqMcYb8V6yqz6rqEcDhwOGqepSqPlf8poVvsErb7NnumD+DNFOlrbbWdYn6lbbhhja/0mahzZhh2wl8CVgGnAr8wDt+tYj8I6Q2FVSwS3SohXUh3Tpt8QHdqn6wy6l7NJboHvXbYt2jxuQu671HVXW3t9XLiBCLuaU1gvuE+ttIbdwIhx/ujm3alBivli60ibiw5Ye2fNdoA/c6Xi+DhTZjhu9s3DIcs4HvAccD7ar6EVV9U6gtK5Bgl2g2Y9pqIzVEayTRPRpPt42Ve52uHDaN98e01feHNrHuUWPyYBvGZ+Bv8O53b/q39+xxAe3QQ10g27QJ1q93j6cLbf7z/GA33Eqbz0KbMcOjql9S1TOA14HfABFgkog8JiJ/DrVxBSIi/WEtm0obuEqaP3u0py/d4rrDm4gAEI3U2DZWxuQhmzFtI1JXl7sOhrbGxsTWVdOmwaRJLrStW5c4lo4f2obbPRpsi01EMKZg7lXVxcBiEfm4qr5ZRIbxL7W81EaEnhjURrNbU62hNtK/TltfXKmLplbacp+IkNo9Whepoce6R43J2ZChTUQiwNuA1uD51b73aLrQ1tAAL73kbk+e7C7Zhrbt291rWqXNmPKiql8I3P2wd2xrOK0pvLpoDe09sawrbY11NYEdEdJU2mpzn4jQX2mLWPeoMcORTaXtz0AX8DwwYv6VdXqLmqRW2nZ7o/pyCW319YlzLLQZU75U9dmw21BowcH/2WiIRhJ7j8YGjmnz13LLpnu0o6ePhmjEukeNKZBsQtt0VT286C0pM5kqbT4/tC1f7gJZUxOMGUNaDQ2J0DbciQg+C23GmGz4YS2biQjgdkXo6k1sY1WbodI21ESE9u4+TvrWg3z+rAOZ1OK+PIMB0rpHjcldNv+K/yIiby16S8pMpjFtPj+0bdyYWO4j0w4xDQ2JyQqFqrTZmDZjTDb6x5FlsU4beBMRArNHU8fCZTsR4YmV29jZ0ctTr+/oH9NW3z+mzbpHjclHNv+KFwG3i0iniOwWkT0iUvVLfwxWaWtqglGjXGjr7IRXXsncNQrJS3VY96gxppSC48iy0VDrNoQHb8P4AbNHs+seffRVNyzwlY176PZCYJ23HYx1jxqTn2xC2/eAE4EmVR2tqi2qOrrI7QrdYKFt8uTk66VLBw9twdco1OxRC23GmGzkOqatsbYmuXs0dZ02fyLCEN2jf3ttCwDLt+yloyexjZVri9jiusbkIZt/xWuAF1R1RP0LG6x7NDW09fVlF9pEYOzY/NtklTZjTK7qchzT1tJQy46OHsDtPZoa9rLpHl23s5PlW9p547Qx9MWVlzfuSXpubaSGXuseNSZn2fwrXgE8LCLXiMjn/EuxGxY2P7QFx7H54WvKlORrGLp7FGD8+Pw3iw++DlhoM6acichYEblFRF4WkZdE5EQRGS8i94nIa971uFK0pTbHxXVnjGti0+5uunpj3o4Iyc+ri9QgMnhoe8yrsn305P0AeH7dTvfcYGiz7lFjcpbNv+KVwANAHdASuFS1XCptkF2lbThdo2ATEYypID8A7lHVg4EjgJeAq4EHVHUO7jv16lI0pD8oZbm47qwJbu++FVva3fNqkp8nItRHawbtHn30ta1MHl3POYdNIVojvOJV2oLdo32xEdV5Y0xBDLnkh6p+tRQNKTfZjGmbNMl1eaqWPrRZpc2Y8iQiY4BTSCzU2wP0iMgFuI3pAX4NPAx8sdjtyXVM20wvtC3fshdgQKUN3GSETJW2WFz5+7KtnPmGydRHI+w/qZlXN+1FBKJeAKyN1PTPKDXGZC+bHREeAgb8l0hVTy9Ki8pENpW2aNStu7Z1a3ahbThrtIGFNmMqxH7AFuCXInIEsAT4NDBZVTd452wEJmd4fkH1j2nLcsmP1gnNQCK0pZt1Wh+tybgjwgvrdrGzo5eT57j/pR44uYVXN+31ulUToc26R43JXTaL6/5r4HYD8B6grzjNKR/ZVNr829u2JY9vS+WHreFW2mz2qDEVIQocDXxSVZ8QkR+Q0hWqqioiafsHRWQuMBdg5syZw25McL/PbIxrqqWlPsqyzX5oS1Npq63JuPfon59dT7RGOHnOJAAOntLCnc9tSAqN1j1qTH6G/FesqksCl7+r6udIlPjzIiIXishSEYmLSFvgeKu3Htwz3uWngceOEZHnRWSZiPxQJNNStoWRbhurTKFt8uTBQ5SNaTNmRFkLrFXVJ7z7t+BC3CYRmQrgXW9O92RVXaCqbaraNmnSpGE3pi7HiQgiwswJTSz3xrSlbmMFbiurdN2jPX1xbv/nOs58w2TGN7svqYOmuBWi6gOhzdZpMyY/Q/4r9mY8+ZeJInI2kGHDpqy9ALwbeDTNY8tV9Ujv8rHA8euAK4A53uWcYbZhUF1dUFPjukB9xxwDJ5wAhx2WOPb2t8NFFw3+WjamzZiRQ1U3AmtE5CDv0BnAi8AdwIe8Yx8C/lSK9vgTELINbeAmI6zYMnilLd02Vg++vJlt7T2879gZ/ccOmuzmrQUrfXWRmv79SI0x2cume3QJbkyb4LpFVwKXD+dNVfUlgGyLZd7/Sker6iLv/o3AO4G/DKcdg+nqcmEr2MSDD4bHH08+7zOfGfq1/LBlY9qMGTE+CSwUkTrcskkfwf0n+WYRuRxYBQzx373C8HchyHZHBICZ45v7K2npx7Slr7T9cfEaJo+u7x/PBjB9XCNNdZGB3aPx9N2jXb0xLvjx3/nXsw/irENKMuwvo46ePs7/0WN8892Hc9x+40NtizGQ3ezR/UrRkID9ROSfwG7g31X1b8A0XJeDb613rGj80FYIVmkzZmRR1WeAtjQPnVHipuS89ygklv0ABmxjBeknImza3cVDr2zmY2+ZnTTjtKZGmDO5hc6exFDowbpHn1i5nVc27WHp+l2hh7aNu7pYsaWd59ftstBmykI2s0cvxK03tEdE/h03NuMbqvr0EM+7H0g3PH+eqmbqFtgAzFTVbSJyDPB/InLoUG1M897DHshroc0YUw3qInl0j45PhLZMs0f3difPR7v16bXEFS5smzHg/I+/ZX92dvQGXrOG3piiqgN6XP72qluYt707/Plu7d0umO7p6h3iTGNKI5vu0S+r6h9F5M3AmcB3cOPLjh/sSap6Zq6NUdVuoNu7vURElgMHAuuA6YFTp3vHMr3OAmABQFtbW15TlAoZ2vxu0cGWBcmG356amuHtrGCMGTlynT0KibXaIMOYtmgkafZoT1+c3zy+ihP2H89+E5sHnH/OYVOT2+QFwb64DgiFf3vNbTS/t3vwvU1LwQ+muzvDD5DGQHY7Ivj/ct4GLFDVu3C7IxSciEwSkYh3e3/chIMV3tpGu0XkBG/W6Acp8iDerq7kLayG4+1vhyVLYNas4b1ONOoCm1XZjDHZ6t/GKofu0aljGvtDXrrFdRtqa+gKdI/e9vRaNuzq4uOnHpDV6/uvmdpFuml3F69scrsnlEelzbXBKm2mXGTzr3idiPwMeB9wt4jUZ/m8jETkXSKyFjgRuEtE7vUeOgV4TkSewU2T/5iqbvceuxL4ObAMWE4RJyFAYStt0SgcfXRhXqu+3kKbMSZ7iUpb9hMRIjXC9PHuf62p21hBcqWtLxbnukeW88ZpYzhlTnZjQPwg2duX3BHiV9kaayPlEdp6/NAWfluMgey6Ry/CLa/xXVXd6c3k/LfhvKmq3g7cnub4rcCtGZ6zGDgs3WPFUMjQVkj19ckzWo0xZjC5bmPlmzW+iRVb2tNW6OprExMR7np+A6u2dfDT9x+T9YoAfoDsjSdX2v722hYmjqpnv4lNA8bMhaG/e9QqbaZMZDN7tAO4LXB/A27CQFUr59BmjDHZqs1xcV3frAnNwJb+/UKD3OzROKrKtQ8t58DJo3hrDjM903WPxuPK317bylsOnMTOjh627O3Oqb3FkOgeDT9AGgPD7OasZuUc2qx71BiTrfo8K20zvRmkGSci9MV56vUdvLJpD1ecvD81acJdJum6R1/csJvt7T2cPGcizfXR/pmb4MaU/emZjHPPisZmj5pyY6Etg85OC23GmMqX2DA+t3EVx+8/nmljG9l37MAZWQ21NcTiyu+fXM2o+ihvO3xqmlfIrDZN9+jz63YBcGzreEbVR5PGtN313AY+fdMzrNnekdP7DFd7f/eoVdpMebDQlkG5VtoaGiy0GWOyt8/oBkRg4qjcxlYcuu8Y/n716f17iAbVR92aQ3c9t4G3HzGVprpshkcn1KbpHt3e3gO4djanhLYd3hpvW0vcZZqYiNCLqm1wb8JnoS2Dcg1t9fW2WbwxJnvHzBrHomvO8MaoFUZ9rfvV0ROLp11Mdyh+aOuLJYLQrs5e6qM1NNZFXGjriRH3trryJwL4wa5YPveHZ7j61uf67/trxfXGdMC2Xbu7ejnqa3/lH8u2FrVNxgRZaMugnEObVdqMMbmYPLqwX2b+OLkD9hnFUTPG5vz8qNc92hOotO1o72Fck/sf6ah6V8nr8Dal391ZmtC2dP1uXtywu/9+sNrnt8G3cVcXOzp6+9eVM6YULLRlUK6hzbpHjTFh87tHL2qbnvUyH0F1/RMRAqGto5exTe7Lrbnedbemjikrdmjb1dnLrkA4Cy47kjquzZ9RarslmFLKbSDCCKFavqHt3/4NusOfCW+MGcHeOH0MJx0wgfcek3vXKAS6R+OJ7tGdHcFKm/vVtLe7j8kkZm9u7yh+aKvvS9Qy2rv7iNYIfXEdsFabreFmwmChLY2+PojHC7eNVSGdfXbYLTDGjHSzJ41i4UdPyPv5abtHO3o4aEoLAM11KZU2v3t0b/FCW09fnM7eGN19sf6N7Nu7+5g8uoF1OzsHrNW2tyu5bcaUgnWPptHV5a7LsdJmjDGVLl336M6OXsZ6lbbmQKUNEl2TO4pYafO7ReOaeN+93TH2Het+EaSu1ba3u9drm4U2UzoW2tKw0GaMMcWT2j2qquzs7GVsoxvTNqp/TFvyRIRtRRzTFgxffoBr7+5jyphGrw02ps2Ez0JbGhbajDHDJSIREfmniNzp3d9PRJ4QkWUi8gcRGbGL9/jdo/46bbu7+ojFtX9MW7M3e7Q9ZdzYjiKGtuAEhF2dvcTiSmdvjKljMlXavDXcuq3SZkrHQlsaFtqMMQXwaeClwP1vAd9X1QOAHcDlobSqDPjdoz1e9+gub/Hc1Nmje7v76OmL09XrzitmpS01tPkL604cVUekRgYZ02aVNlM6FtrS6Ox01xbajDH5EJHpwNuAn3v3BTgduMU75dfAO0NpXBnwK21+96g/Vm1cypi29u6+/grXxFF17OnqS9pFIR8rt7azalv7gOPBCQW7O3vp8LpmR9XXMqo+arNHTVmw0OZRhQ5vWzurtBljhul/gS8AfsKYAOxUVb8ssxaYlu6JIjJXRBaLyOItW7YUvaFhSN3Gqj+0NbtKW1NtonvUn4Tg7+gwnC7S59bu5O0/eox5t78w4LHUSpsfyprrI4xujA6otO0JzGy1La5MqVho81xyCRxzjLttoc0Yky8ROR/YrKpL8nm+qi5Q1TZVbZs0aVKBW1cealO6R3f2d4+6SltNjdBcF2Fvd6y/AjZrQhOQ/1ptL23YzQdueJK93X1p9zD1u2jB6x71Qtmo+igt9bUDx7R5IS6u0N4Ty6tNxuTK1mnzjBkDO3a42xbajDHDcBLwDhE5D2gARgM/AMaKSNSrtk0H1oXYxlD1j2lLrbQ1JeZmNNdH6ejp6+9+3M+rtOWzVltXb4wP/uJJGmsjHDZtNCu2DOwe3dXZS2NthJ5YPCm0NddHaWmIDhi7tjdliyt/xqsxxWSVNs/48bB9e2I3BLDQZozJnapeo6rTVbUVuBh4UFUvAx4C3uud9iHgTyE1MXQNtTU010XYvNtVvHZ09CICYxoTe/SNqo+yt7uvPyzNmuiFtiwqbX96Zh23Pb22//7r29rZsqeba847mDdMGZ3UFerb1em20RrdEE3qHh1VH2V0Y+3AMW1dffg7eKV2nRpTLBbaPOPGQW8vtLdbaDPGFMUXgc+JyDLcGLcbQm5PaESEmROa+ycE7OzoYXRDLZGaxD6mzfVRb0xbcqUtmzFtP31kBdc9vLz//prtbnbZrAnNjGmspaMnNmBCw67OXsY01jKmsZZdnX39s0f9StuA2aPdfUxucb8kbDKCKRWr53rGj3fXO3YkQls5bmNljKkcqvow8LB3ewVwXJjtKSezxjfx6uY9QPJm8b7m+gjt3bH+sWQzx7sxbUMt+6GqrN7WTl9ciceVmhphzXY3y2zGuEbGeO+zq7OXiaPq+5+3q7OX0Q211EdrvEpbrL8doxsGVtr2dPVywD6j2Li7y7ayMiVjlTbPuHHuevt2q7QZY0yxzZrQxNrtncTiys6Onv5JCL5g92ikRhjdGGV0Q3TIStu29h7ae2J098XZvMd1v67Z0UFjbYTxzXX9XbCpXaS7OnsZ3VjL6MbaARMRRje4tsQDOzjs7e5j37HebglWaTMlYqHNk67SZqHNGGOKY+aEJnpicTbu7mJnRy/jBlTaorR7ExFaGqKICBNG1Q9ZaVu1rSNw23W/rt3RyYzxjYgIozOEtt1e9+joxlp2e6GtRqCxNkJLQy2qsNfrMu3sjRFXmDY2/RZXxhSLhTaPH9qs0maMMcU3a7wbo7ZqWzs7OnqSZo5CYEyb120JMK6pdshN41dvT8wMXeV1i67Z3sGMca57dbBKW2JMm5uI0FznwmJLgxtJ5I9r85f78Le4su5RUyoW2jzpukfr6zOfb4wxJn/+umurt3WwM82Ytv7u0a4+Rje60DS+uZ5tQyz5sWpbByIQqRFWb+tAVb1KW3JoCwat3lic9p5Yf2jzK23+zgx+dc4fX+cvrDuuuY7G2sig3aN3P7+Bv71WnYskm9KziQieYPdoZyfU1kIkEm6bjDGmWu07tpHaiLB8y172dvcNrLTVRenqjbPDm1kKML65lufXudC2bPMe+uLKwVNGJz1v9bYOpo5uIBqpYdV2Fwj3dvcxfZzryvRfa1fKtlUAYxqj1PfV0BdXtuzp7t+43q+0+d2gfqWtpSH9zNKg7977CpNHN3DynOpcKNmUloU2T3MzRKOJSpt1jRpjTPFEaoTp45p4du0ugDRj2lxg2ririyOmjwVcpW17ew+qytwbl1AXreGez5yS9LxV2zuYOaGJ2kgNq7e1s3aHW+5jemr3aMoOCABjmmqp9zanX7+zq7/S1tKQXGlLrOFWm3YNt6BNu7uy/kyMGYp1j3pEXLXNn4hgoc0YY4pr5vgmnvdCW+rsUT8wbdrd1V/pGt9cS29MefjVLazY2s5rm/fS1Zu8hdSqbR3MGt/MzPFNrNrewZod3nIf412lrS5aQ2NtZMBeo0B/9yjA+l2dNNd53aMpY9r8a39maaaJCHu7+2jvibFxd5ftT2oKwkJbgL8rgoU2Y4wpvlkTmuj0Qle6iQjg9vb0x5T55/zUWzg3Flde2bin/znt3r6iMyc0MWtCEzs7elm63oVCf0wb0D/ZwJcutO3p6htQadudUmlraUi/W4LPr7J19MT6N743ZjhCCW0i8h0ReVlEnhOR20VkbOCxa0RkmYi8IiJnB46f4x1bJiJXF6Nd48ZZaDPGmFKZGQhSAyciJAYV++PQJoxyoe2Jlds5cf8JACxdv7v/PH+5j1kTmpjpzU79+7JtbimPhsTrZxPagm0YOHu013s86hbezTB7NNg1at2kphDCqrTdBxymqocDrwLXAIjIIbi9+g4FzgGuFZGIiESAnwDnAocAl3jnFlSwe9R2QzDGmOKa5W1NBQNDm981CfTPHg1W4/717ANpaYj2V9IgsdzHrPHN/bNTn1u7s38Sgi81tPmha3RDcmjzK20NtRHqojUDKm3N9VFGN0YzVtH8vVXBjc0zZrhCCW2q+ldV9f+WLwKme7cvAG5S1W5VXQksw237chywTFVXqGoPcJN3bkFZpc0YY0rHD1aQuXsUApW2ZrcO0+xJzRw9cxyHTB2dttI2c0JTfxUvrvSv0db/ehkqbf6OCL5RSW1IjF3b091HfbSGumgNLV6lLd2YtWB1baNV2kwBlMOYtv8H/MW7PQ1YE3hsrXcs0/GCsokIxhhTOn6wqovU0FSXvMZSUmDygtSklnpa6qN88MRWRIRD9x3Dyxt3E/O2l1q1vYOxTa5a1lwf7d9b1J+E4PPXYvPt6uylPlpDQ22Elvoo4u1bHwyOLQ21idmjXX39XaajG2rpiytdvckb0ANs2t1NfdT9mq2GSlssrgMmfpjSKlpoE5H7ReSFNJcLAufMA/qAhQV+77kislhEFm/Zkv2ihuPHw65dsHevhTZjjCm2htoIk0fXM7apFvGTkqc5pcoF0FgXYdGXzuCDJ84C4NB9R9PVG2fFlr2AW6NtVmCcnF/JC05CAC+0Bbo0/d0QAGpqpL+y15wSHHd2JLpHR/UvvOut4ZZmMsLmPV3sO7aRCc11VVFp+8lDyzjze4/YTNgQFS20qeqZqnpYmsufAETkw8D5wGWa+BuwDpgReJnp3rFMxzO99wJVbVPVtkmTsl/Q0N8VYeNGC23GGFMKsyY0M765bsDxUSlVLl9zfbQ/4B06zS2s63eRrtrenjROrj+0jRsY2vZ299EXc9Wx3Z19SWPZ/NvByRCzxjexcqsbM7e3q49RgUqbe400oW13N5Na6pk8uoFNVVBpe27tTtbu6OT1wP6uprTCmj16DvAF4B2qGvzTvwO4WETqRWQ/YA7wJPAUMEdE9hOROtxkhTsK3S5/VwQLbcYYUxpfOPsgvnTeGwYcb6itocYrvvnVrFSzJ42iLlrD0vW72N3Vy/qdXUnj5Pz9TQdORPCrY67aFqy0uccHVtoO2GcU63Z20tHTx56kSlvyciBBm/Z0MXl0A1PGNLChCkKbP2bw2TU7w23ICBbWmLYfAy3AfSLyjIj8FEBVlwI3Ay8C9wBXqWrMm7TwCeBe4CXgZu/cgvIrbX19FtqMMfkRkRki8pCIvCgiS0Xk097x8SJyn4i85l2PC7ut5aCtdTynHDiwR0REBuz9mao2UsMbprSweNUOLv/VUwhw6kH79D9+9mGTefdR02id2Jz0vDFNyVtZZRvaAFZsaXeVtnp3zujAFleb93Rxx7PrAVBVNu3uYnJLPVPGNFT8kh/xuLJ6uwttz1hoC00o21ip6gGDPDYfmJ/m+N3A3cVsl19pAwttxpi89QGfV9WnRaQFWCIi9wEfBh5Q1W96a01eDXwxxHaWPX/T+FF1mX9VHbLvGH7/5GpqBH50ydEcMyuRhQ+eMprvve/IAc9J3X90V2cvB09p6X880T06MLQt2+z2SvUnIgQX3v2fe1/lD4vXcGzrOJq8vVMnj26gszfGtvYeuvti1Eczb2odjysvbtjNIVNHU1MjGc8Lw+Y93XT3ue5kC23hKYfZo2VjXOD/vRbajDH5UNUNqvq0d3sPrndgGm6Zol97p/0aeGcoDawgzfVRWuqjgwaYY2aNQwS+894jeNvhU7N63f79R73QtruzN6ma598OrhXXOqGZSI30h7bUiQgbd3Vx53Ouyvbsmp1s9ipr+4yuZ8po9wsluG5bkKry4MubOP9Hj3H+jx7jl/94Paufo5RWbXPj+d4wdTQvrt9Nd5/NIg2DhbYAq7QZYwpJRFqBo4AngMmqusF7aCMwOax2VQq3eG36rlHfu46axt+/eDrvOWb6oOcFBUNbLK7s6c40ESER2uqiNcwa3+RCW5qJCH94ag3tPS7I/HPNTjbvcQFt8ugGJo9xv1AyzSD9w1Nr+H+/Wsze7j4OnDyKBY8uL7tQtMrrGr3gyH3picV5ecOeIZ5hisFCW4BV2owxhSIio4Bbgc+o6u7gY96M+bTrJuS7ZFE1GlUfSZo5mk6kRth3bG5b2ARDW3Bh3dTHm+qTuzJn7zOKFzfspicW7w90/m4JK7a2s/+kZg6fPoZn1+zsH8M2eXQDU/3QlmEywkOvbGbm+CYe+Pxb+PL5h7Bpdze3LnELJPz+ydVc9LPHuf/FTaEutbF6WweRGuG8w1w1M5su0n/947P8cfGaIc8z2QtlTFu5qq2FUaPcOm22jZUxJl8iUosLbAtV9Tbv8CYRmaqqG0RkKrA53XNVdQGwAKCtrW1EL4g195TZdPYUvuLUP+Ozs5cXveVC9p+UmKxw/uFTUZQJKUuRHLDPKO57cROQ2I8UXLVt695uLmqbwfqdndy6ZC0nz3GTK/ZpqafPW/w3U2h7Zs1OTtx/ArWRGt58wESOmD6Gnz6yHBG45rbnaayN8NEbF3PUzLH89P3HMHl06asKr29rZ9rYRmaMb2RSSz3PrtlJPK786h+vc/SscRw5Y2zS+Zv3dHHLkrU89PJmzj98XxrrMo/lM9mzSlsKv4vUKm3GmHyIW0TsBuAlVf1e4KE7gA95tz8E/KnUbas0bzlwEuccNqXgr9tQG6E+WsOuzl6efH07NQJtgQkMM8Y3ceWpBwxY8Hf2pFH9t5N3bIgSqRHeffQ0jpwxlvaeGH9ftpWW+qjr4m2I0lgbSds9unFXF5t2d/eHHhHhqtMOYPX2Dq657XlOnjORJV8+k2+++40sXb+b/73/tQGvoarc88JGevoG7soQiysPvbKZ3tjAx3KxensHsyY0ISIcOWMsz6zZydfufJGv3fki//PXVwacv/j1HQBsa+/h90+uzus912zvYMmqHcNqd7Wx0JbC7yK10GaMydNJwAeA070ljZ4RkfOAbwJnichrwJnefROSMY217Oro5amV2zlk39FDdsNCYgYpJIe2gya38I4j9mWflgaO8MLXEyu3M2m020ZLRJg6piFtaHtmjQslRwQqVWe+YTLHzBrHyXMmsuADbTTVRbn4uJlc1DadW5esHVCxe/S1rXzst0v4Q5quyBseW8FHfvkU1/9txZA/32BWbevo33bsyBljWbG1nV/943Umj67niZXbB2xv9eTK7TTU1tA2axwLHl2R1xi9//jTC3zwhidKtnVWe3cfO9p7SvJe+bLQlsIqbcaY4VDVx1RVVPVwVT3Su9ytqttU9QxVnePtGLM97LaOZGMaXZfm06t3cGzr+KGfgNuo3jcq0D167WVH8z8XHgHAfhOaGd0QJRZXJrckfpFk2hXhmTW7qI0Ih+w7uv9YTY1w87+cyG8uPz6pW/FfTplNTJUFjyYHsHtecPNb7n1hY9Lx1ds6+N59ryICN/xtZd5dzbs63Ng/f+Fivyp52fEz+dZ7DqenL86TK5P/Oj+5cjtHzxzHp86Yw8bdXdz2dMZNjNLa3dXLY8u20t4T47HXtubV7lzE4soHf/Ek777uH8TjhRuVsLOjp39v3EKw0JbCQpsxxlS/MY21PLlyO919cY7fL7vQ1tJQ2798R0t9ojInIv3LktTUSH/VbLJXaQMy7orwzJodHDJ19ID12yJpljmZMb6JC47cl989uYpte93s1Fhc+evSTdQIPL5iGzs7XKVIVfnS7c8Tranhhxcfxbb2Hm56Kn03ZTyuvPMnf6f16rtovfouzvreI0ldrau2u+U+Zno7TBy333ju/OSb+foFh3H8fhOoi9bw6KuJSTO7u3p5aeNujttvPCfPmcjh08fwwwdey2mB4Yde3kxvTKkRuHfpxqGfkKPHl2/jpG8+yN+XuUD420WrWLJqByu3trNo5bYhn7+nq5ezv/8o3/vrK/0TRH6zaBWnfuch1u3s7D/vS7c/z7uu/XvBJpFYaEth3aPGGFP9xjTWsqfbbWPVlmWlDRJdpMFKW6oj+0NbcqVt856upK6+WFx5fu2uAYP4B3PlqbPp7otz/d9WArBk1Q62tffw0ZP3JxZXHnjJzW+59el1PLZsK18892DefsS+HNc6ngWPrkg77u3epRt5Zs1OLmqbzgdPnMVrm/dy29Nr+x/3t6/yK20iwmHTxlBTIzTWRTiudTx/C1TDlry+A1U4rnU8IsI33nkYuzt7ueznT/SHzaHcu3Qjk1rqefsR+3LfS5v694kN6ovF+cNTq7nixsU5BUJV5bt/fYV1Ozv56K8Xc8ez6/n2PS9z0gETGN0Q5eanhp7x+ttFq3ll0x5++OAyfvTgMm5+ag1f/r8XeH1bBz97ZDngFmL+ywsbOXnOxAHjI/NloS2FVdqMMab6+ct6zJ7UzMRR9UOcndAf2uqHDm37BELbKQdOpDemXPvQsv5jyzbvpb0nljSebej3b+FdR07jhsdW8OqmPdzzwkbqojV88vQDmDqmgXuWbmTr3m6+cdeLtM0ax2XHzQTgqtMPYMOuLn72yPKk7j9V5ccPLWO/ic3897sP56vvOJQ3ThvDdY8s7w9K/vZV/pi2VCfPmcgrm/b0B6cnX99OtEY4aqarghw+fSy/+PCxrN3RwcULFvGVO5bylTuWZtzDtKs3xkMvb+HsQydz7mFT2dnRm9T9Go8rdzy7nrO+/yhfvPV57ntxE//xpxf6H39ixbak0JnqiZXbWbJqB5/yPrNP/f6fxBW++e7DueDIafzlhY39S8Fkat8Nj63g5DkTeffR0/jefa/yxduec/ePmsZNT61h854urnt4OQ3RCP/vpP0yvlauLLSlsEqbMcZUP3/Zj+Oy7Br1nfmGyRy333jGNWWeuNDWOp7Dp4/huEAF702z3S/06x5Zzisb3cK0/iSEXCptAPPe9gZG1Ue5+tbnuHfpRk4+YCItDbWcfegUHn11C1+67Xk6umN88z1v7O+2PWXORE6eM5H/ue9Vzvvh37jPW/ft4Ve3sHT9bj7+ltlEaqR/9uqqbR3c9bwbK7dqWzuTWuqT9mIN8pc38attT63czhunj0kaj3f8/hNY8IE2Onpi3P7Pdfz+ydVcev0inl49cHboo69uobM3xjmHTuUtB06iobaGe5duRFW578VNnPfDv/Gp3/+T+mgN13+wjavPPZh7l27inhc28LfXtvCBG57kczc/21/xSvWTh5YxcVQ9V552AAuvOJ6jZo7lqxccyozxTVzUNoPuvjh/9vaQTecPT61h694ePnHaAXz7PYdzUdt0zjh4Mgs+0MYnz5hDXyzO1+98if97Zh2XHDeTCTn8p2Aotk5bCqu0GWNM9fNDW7aTEHxvnjORN8+ZOOg5YxprueMTbx5w/N/PP4SHX93CF299ju+/70j+sXwboxuitE5oTvMqmU0YVc+Xzz+Ez938LACfPnMOAGcfOoVf/eN1/vriJj575oEcsE9iP1UR4dcfOY4/P7ee79/3KlfcuJgjZ4ylqzfGvmMaeOdR0/rPfeshk5mzzyiufWg5R0wfy2ub9zIrQ5UN4OApLUwcVc99L27kyBljeHbtzrTVpVMOnMTfrz4dgM27u7jwZ4/z4V88yU/ff0zSAsl/emY9YxprOX7/8dRGanjLgZO46/kNPLt2F8+s2UnrhCZ+cPGRvP3wfampEU6LTeLPz65n3u0v0N7Tx+x9RrHfxCb++y8vE6kRznxDYvORlVvb+dtrW7n63INpqI0wdUwjt195Uv/jh00bzcFTWrjpqdW8+YCBf85xVX72yHLaZo3juP1c9++333tE/+P7TWzmbYfvy5+fXU9tRLjilMJV2cBC2wD77uuug7sjGGOMqS6TR9cjknulbTjGN9fxH+cfwmf+8AynffdhwK1Fl8/m8O86ahq3/3Mdjy/f1h9Kjm0dx8RRdYxvruPjp84e8JyaGuGCI6dx3huncuuStfzwgddYv6uLr7z9EOqiNUnnXXnabD77h2c51WvnRW2ZtwmrqRFOOXAitz29jnuXusWHT9h/wqDt32d0Aws/ejzv+9kiLv35EwMef+8x06mNuDad98ap3Lt0E7WRGr757jfynsBjANFIDd96z+Fc8JO/s9/EZn5z+XGMaaylN/Y037jrJb5x10tJrz2msZb3nzArbbtEhPcdO4Ov/vnF/p89nfnvfmPGcWpXnTabPz+7nvceM52pYwq7Ur+EuS1GKbS1tenixYuzPj8eh8cfh5NOGvpcY0z5EZElqtoWdjsKIdfvL5O9rt4Yr2zck9N4skJQVR5fvo1Ne9z4r2NbxzN9XOYq1mB2dfaycmt7Uvfqq5v2MLqhliljhu4u6uqNsfj1HZw4e8KA2arxuHLfS5vo6HGTNU46YCL7tGR+za17u3nsta0oSmNtlLceMjmrMLp5Txf/WLYNDezqJggnz5nY360YjyuPr9jGMbPG0VCbeWeFF9btYsa4JsZ4Xdc9fXEefHkTnSnrvB00eXTSEiupevri3PfiJnpi6ZdIGdNYy2kH7TPo5IIlq3Zw0JSWQcc+ZjLYd5iFNmNMVbHQZoypZIN9h9lEBGOMMcaYCmChzRhjjDGmAlhoM8YYY4ypABbajDHGGGMqgIU2Y4wxxpgKYKHNGGOMMaYCWGgzxhhjjKkAVb9Om4hsAVZlefpEYGsRm1NMldr2Sm03VG7bK7XdkF3bZ6nqpFI0ptjs+6siVGrbK7XdULltz7bdGb/Dqj605UJEFlfqopyV2vZKbTdUbtsrtd1Q2W0vtkr+bKztpVep7YbKbXsh2m3do8YYY4wxFcBCmzHGGGNMBbDQlmxB2A0Yhkpte6W2Gyq37ZXabqjsthdbJX821vbSq9R2Q+W2fdjttjFtxhhjjDEVwCptxhhjjDEVwEKbR0TOEZFXRGSZiFwddnsyEZEZIvKQiLwoIktF5NPe8fEicp+IvOZdjwu7remISERE/ikid3r39xORJ7zP/Q8iUhd2G9MRkbEicouIvCwiL4nIiRX0mX/W+7vygoj8XkQayvVzF5FfiMhmEXkhcCzt5yzOD72f4TkROTq8lofLvr9Kx77DSsu+v5JZaMP9IwR+ApwLHAJcIiKHhNuqjPqAz6vqIcAJwFVeW68GHlDVOcAD3v1y9GngpcD9bwHfV9UDgB3A5aG0amg/AO5R1YOBI3A/Q9l/5iIyDfgU0KaqhwER4GLK93P/FXBOyrFMn/O5wBzvMhe4rkRtLCv2/VVy9h1WIvb9lYaqjvgLcCJwb+D+NcA1Ybcry7b/CTgLeAWY6h2bCrwSdtvStHW695f2dOBOQHALDUbT/TmUywUYA6zEGwMaOF4Jn/k0YA0wHoh6n/vZ5fy5A63AC0N9zsDPgEvSnTeSLvb9VdL22ndYadtt318pF6u0Of5fDN9a71hZE5FW4CjgCWCyqm7wHtoITA6rXYP4X+ALQNy7PwHYqap93v1y/dz3A7YAv/S6RX4uIs1UwGeuquuA7wKrgQ3ALmAJlfG5+zJ9zhX577YIKvJzqMDvL7DvsJKy76+BLLRVKBEZBdwKfEZVdwcfUxfby2pasIicD2xW1SVhtyUPUeBo4DpVPQpoJ6UboRw/cwBv/MQFuC/tfYFmBpbvK0a5fs4mN5X2/QX2HRYG+/4ayEKbsw6YEbg/3TtWlkSkFveFt1BVb/MObxKRqd7jU4HNYbUvg5OAd4jI68BNuO6FHwBjRSTqnVOun/taYK2qPuHdvwX3BVjunznAmcBKVd2iqr3Abbg/i0r43H2ZPueK+ndbRBX1OVTo9xfYd1gY7PsrhYU25ylgjjcjpQ430PGOkNuUlogIcAPwkqp+L/DQHcCHvNsfwo0VKRuqeo2qTlfVVtzn+6CqXgY8BLzXO63s2g2gqhuBNSJykHfoDOBFyvwz96wGThCRJu/vjt/2sv/cAzJ9zncAH/RmYZ0A7Ap0Q4wk9v1VAvYdFgr7/koV9qC9crkA5wGvAsuBeWG3Z5B2vhlXXn0OeMa7nIcbW/EA8BpwPzA+7LYO8jOcCtzp3d4feBJYBvwRqA+7fRnafCSw2Pvc/w8YVymfOfBV4GXgBeA3QH25fu7A73FjV3px1YHLM33OuEHgP/H+zT6Pm2EW+s8Q0udm31+l/TnsO6x07bbvr8DFdkQwxhhjjKkA1j1qjDHGGFMBLLQZY4wxxlQAC23GGGOMMRXAQpsxxhhjTAWw0GaMMcYYUwEstBljjDHGVAALbcYYY4wxFcBCmzHGGGNMBbDQZowxxhhTASy0GWOMMcZUAAttxhhjjDEVwEKbMcYYY0wFsNBmjDHGGFMBLLQZY4wxxlQAC23GGGOMMRVgRIQ2EZknIktF5DkReUZEjg+7TcYYY4wxuYiG3YBiE5ETgfOBo1W1W0QmAnUhN8sYY4wxJidVH9qAqcBWVe0GUNWtIbfHGGOMMSZnoqpht6GoRGQU8BjQBNwP/EFVHwm3VcYYY4wxuam4MW0iMlZEbhGRl0XkJa/7MyNV3QscA8wFtgB/EJEPl6CpxpgsiUhERP4pInd69/cTkSdEZJmI/EFEKn5Ig4gc5I2p9S+7ReQzYbfLGFM5Kq7SJiK/Bv6mqj/3vsibVHVnDs9/L/AhVX17sdpojMmNiHwOaANGq+r5InIzcJuq3iQiPwWeVdXrwm1l4YhIBFgHHK+qq8JujzGmMlRUpU1ExgCnADcAqGrPUIHN+9/tnMChIwH7kjSmTIjIdOBtwM+9+wKcDtzinfJr4J2hNK54zgCWW2AzxuSi0iYi7Ifr4vyliBwBLAE+rartgzxnFPAjERkL9AHLcF2lxpjy8L/AF4AW7/4EYKeq9nn31wLTQmhXMV0M/D7sRhhjKkulhbYocDTwSVV9QkR+AFwNfDl4kojMxQtmNQ21xzTNHB98+I3Au1oOmlKaFhdRT8/APz7pkQHHIj0DnxvpGtgtLl09aMPAoUOxhoGvGUszwkjrBr5mXV3fwBNN2dn76qatqjopl+ecfGqD7tgeH/K8pc/3LgW6AocWqOoCABE5H9isqktE5NRc3r9SecM63gFck+Hx/u+v5ubmYw4++OASts4YE7YlS5Zk/D6utNC2Flirqk9492/BhbYk3i+EBQAtB03RY659f+laWEKvrx34Z1q/emCaalk1MEyNWd494Fjdy2vpOXj6gOO7ZtcPOLZn1sAg1z1zYDpsnb5lwDFTfh45839y7qbbsT3OrXdNHPK8g2du6FLVtgwPnwS8Q0TOAxqA0cAPgLEiEvWqbdNx47+qxbnA06q6Kd2Dwe+vtrY2Xbx4cSnbZowJmYhk/D6uqDFtqroRWCMiB3mHzgBeDLFJxphhUNVrVHW6qrbiugwfVNXLgIeA93qnfQj4U0hNLIZLsK5RY0weKiq0eT4JLBSR53CTCv4r3OYYY4rgi8DnRGQZbozbDSG3pyBEpBk4C7gt7LYYYypPpXWPoqrP4JYGMMZUEVV9GHjYu70COC7M9hSDN2lqQtjtMMZUpkqstBljjDHGjDgW2owxxhhjKoCFNmOMMcaYCmChzZg8vb52UtplV4wxxphisNBmTB4srBljjCk1C23G5Cg1sFmAM8YYUwoW2ozJQaaAZsHNGGNMsVloM6ZALLgZY4wpJgttxmQpm1Bmwc0YY0yxWGgzJgvpwlj96roQWmKMMWakstBmzBAGC2zpgptV24wxxhSDhTZjhsmCmzHGmFKw0GbMILLtFrXgZowxptgstBmTgY1jM8YYU04stBmTRj6BzaptxhhjislCmzFZSBfIWlZpVudZcDPGGFMIFtqMSZFLyLLgNnwiMkNEHhKRF0VkqYh82js+XkTuE5HXvOtxYbc1LPPnz+fQQw/l8MMP58gjj+SJJ54Iu0nGmBBEw26AMeUk227RYFhrWaXsmSUDntM9s6fwDaxOfcDnVfVpEWkBlojIfcCHgQdU9ZsicjVwNfDFENsZiscff5w777yTp59+mvr6erZu3UpPj/3dMmYkskqbMZ58AttwX9+Aqm5Q1ae923uAl4BpwAXAr73Tfg28M5QGhmzDhg1MnDiR+vp6ACZOnMi+++4bcquMMWGw0GYMww9s1k1aGCLSChwFPAFMVtUN3kMbgclhtStMb33rW1mzZg0HHnggV155JY888kjYTTLGhMRCmzEFMtKC245YE7fsPnrICzBRRBYHLnPTvZ6IjAJuBT6jqruDj6mqAvmVOMuIiIwVkVtE5GUReUlEThzs/CVLlnDYYYfx+c9/ngULFjBp0iTe97738atf/aqg7Vq4cCGtra3U1NTQ2trKwoULC/r6xpjCsNBmRrxCdotmG9xGmK2q2ha4LEg9QURqcYFtoare5h3eJCJTvcenAptL1+Si+QFwj6oeDByB6woe1KpVq/jYxz7GunXr+OpXv8qPf/xjbr311oI1aOHChcydO5dVq1ahqqxatYq5c+dacDOmDNlEBDOiDSewjVneDcCu2fV5vW/r9C05P68aiYgANwAvqer3Ag/dAXwI+KZ3/acQmlcwIjIGOAU3wQJV7QGymlHQ0dHB5ZdfzvXXX8/KlSvp6+vj1FNPLUi7Fi1aRHd394D3mzdvHpdddllB3sMYUxhWaTMjViECWzojrZu0AE4CPgCcLiLPeJfzcGHtLBF5DTjTu1/J9gO2AL8UkX+KyM9FpDn1JBGZ63clB493d3ezePFiOjo6aG1tLVijUgObb/Xq1QV7D2NMYVRcpU1EXgf2ADGgT1Xbwm2RqWbZBLYxy7sHVNuyXQbEKm6gqo8BkuHhM0rZliKLAkcDn1TVJ0TkB7hlTL4cPMnrPl4AICL9fwFnzZrF66+/XvBGtba2smrVqgHHZ86cWfD3MsYMT6VW2k5T1SNHemAb6b/sh6PQ+4qmq7xZxc2kWAusVVV/ZdxbcCFuSE1NTcyfP78ojZo/fz5NTU0lez9jTP4qNbQZk7didYtmG9zMyKSqG4E1InKQd+gM4MWhnjdr1iwWLFhQtPFll112GQsWLOhfB67Y72eMyV/FdY/ipv3/1es2+Fm6mWjGZFKswJYL6yYd0T4JLBSROmAF8JHBTj7mmGNYvHjxYKcUxGWXXcb1118PwMMPP1z09zPG5KcSK21vVtWjgXOBq0TklNQTggN5e3d2lL6FIUq3dVLq2CpIP+Ox5+DpWZ2X7vXSvW8lhJDhBLa6l9dmdZ51kxqfqj7jLXtyuKq+U1V3hN0mY0zlqLjQpqrrvOvNwO3AcWnOWeCvCVU7tin14Yp01pSX0x5PF4yGE9wKpRwDm4UiY4wxlayiuke96fE1qrrHu/1W4GshN2tYMoWxTOfet/HgAcdbp28ZEEi6Z/YMqObsmSUDqj67Ztdn7PYbTpXNGGOMMYVVUaENt/fg7W4tTqLA71T1nnCblFkugWy4sg1u2ar2blFjjDGm0lRUaFPVFbitX8pGKYOZ/37pqm3ZyqbaZoHNGGOMKT8VN6bNFHd8mwU2Y4wxpjxZaBuGUlfZsnnvsCYmWGAzxhhjistCW57CDGxDtWE4wS2bc2zigTHGGFN6FtoqXC7BLVfWLWqMMcaUDwtteSiHKls+cqm2WWAzxhhjyouFtipQ6G5SC2zGGGNM+bHQlqNyrbKVYnzbUK9rjDHGmOKx0JaDcg1svuEGN7CJB8YYY0y5stBWZYYTLK1b1IRFRM4RkVdEZJmIXB12e4wxphxZaMtSuVfZgtK1NZdq22CPW2AzhSYiEeAnwLnAIcAlInJIuK0yxpjyY6FtBMkluFlgMyV0HLBMVVeoag9wE3BByG0yxpiyY6EtC5VUZfMVYnybMSUyDVgTuL/WO2aMMSagojaMD0MlBjZfps3lW6dv4fW1k5KOdc/soX51Xf/tdM8xJmh3X0Pav18D3TVRRBYHDixQ1QXFapcxxlQrC21VLlNwy8QCmymCraraNsjj64AZgfvTvWPGGGMCrHt0EJVcZQsazsQEC2ymBJ4C5ojIfiJSB1wM3BFym4wxpuxYaBshsg1uuTxuTCGoah/wCeBe4CXgZlVdGm6rikNEXheR50XkmZQuY2OMGZKFtgyqpco2FAtmphyo6t2qeqCqzlbV+WG3p8hOU9Ujh+gyLqmFCxeyaNEiHnnkEVpbW1m4cGHYTTLGpGGhLY1qDWy5zCi1MGfMyLBw4ULmzp1Ld3c3AKtWrWLu3LkW3IwpQxbaRphsAqkFNmOKRoG/isgSEZkbdmMA5s2bR0dHR9Kxjo4O5s2bF1KLjDGZWGhLUa1VtqDBxrdZYDOmqN6sqkfjdn+4SkROST1BROaKyGIRWbxlS/H/Pa5evTqn48aY8FhoG6HymZhgjBkeVV3nXW8GbsftBpF6zgJVbVPVtkmTJqU+XHAzZ87M6bgxJjwW2gJGQpUtaKT9vMaESUSaRaTFvw28FXgh3FbB/PnzaWpqSjrW1NTE/PnVPh/EmMpji+t6LMAYY4psMnC7iID77v2dqt4TbpPgsssuA+Dyyy+nu7ubWbNmMX/+/P7jxpjyYaFthMt1xwRjTH5UdQVwRNjtSOeyyy7j+uuvB+Dhhx8OtzHGmIwqsntURCIi8k8RubMQrzfSq2wj/ec3xhhjKkFFhjbg07iV002BWHAzxhhjylvFhTYRmQ68Dfh5IV7PwkqCfRbGGGNM+arEMW3/C3wBaBnuC1lIGSifz8TGxBljjDHFV1GVNhE5H9isqkuGOK9/ccrenR2DnWqMMcYYUxEqKrQBJwHvEJHXgZuA00Xkt6knBRenrB3blPqwMTkZs7x7wLG6l9eG0BJjjDEjWWihTURGi8joXJ6jqteo6nRVbQUuBh5U1ffn2wbr1hs5Xl87cGX5+tV1A461rNJSNMcYY4zJWclDm4h8RkTWASuB10XkVRG52HtsRqnbY4anEoLvcAKbVdmMMcaUi5JORBCR/wSOB072FppERPYHfiAis4ArgAOyeS1VfRh4uDgtNdXCApsxxphqUerZo+8H3qiqXf4BVV0hIhcBW4BLS9weYyywlTER+Q7wdqAHWA58RFV3eo9dA1wOxIBPqeq9YbXTGGNKodShLRYMbD5V7RSRdap6R4nbY4Yh367RdNWvUklXZTNl7T7gGlXtE5FvAdcAXxSRQ3DjWg8F9gXuF5EDVTUWYluNMaaoSj2mbZ2InJF6UEROB9aVuC1mhLFu0cqjqn9V1T7v7iJgunf7AuAmVe1W1ZXAMuC4MNpojDGlUupK26eAP4nIY4C/1lob3lIeJW6LCUFYVTYLbKGaKCKLA/cXqOqCPF7n/wF/8G5Pw4U431rvmDHGVK2ShjZVXSoih+HGrh3qHX4U+Jd03abGFIIFtuLo6YlmG8K3qmpbpgdF5H5gSpqH5qnqn7xz5gF9wMJ82mqMMdWg5NtYeeHsF6V+X1NY+YxnC3MsWz4ssJWGqp452OMi8mHgfOAMVfXT9joguETQdGyIhTGmypXNjggikk93iakQldgtasInIufg9hp+h6oG96S7A7hYROpFZD9gDvBkGG00xphSKZvQBvwsjDethMVhTX6sW7Qq/BhoAe4TkWdE5KfghloANwMvAvcAV9nMUWNMtSt59yiAiFyoqn9MObw/ickJpozlGnRTq2xhLbthga3yqGrGxbZVdT4wv4TNMcaYUIVVabsmy2OmwpVLYEvHApsxxphKUuptrM4FzgOmicgPAw+Nxs0MM6YobCN4Uy5EJAIsBtap6vlht8cYUzlKXWlbj/uy6sJ1hfqXO4CzS9wWU2TlUmWzblFTZj4NvBR2I4IWLlzIokWLeOSRR2htbWXhQltZxZhyVOp12p4FnhWR36lqbynf2xRGtuPZymV5DwtsppyIyHTgbbixeJ8LuTmAC2xz586lu9v9u1i1ahVz584F4LLLLguzacaYFGGNaTtORO4TkVdFZIWIrBSRFSG1xZRAGFU2C2ymDP0vbgmTeMjt6Ddv3jw6OjqSjnV0dDBv3ryQWmSMySSU2aPADcBncV2jNk2/ygzVLWrjy8xIJCLnA5tVdYmInDrIeXOBuQAzZ84sertWr16d03FjTHjCqrTtUtW/qOpmVd3mX0JqiymgcukWTceqbCZkJwHvEJHXgZuA00Xkt6knqeoCVW1T1bZJk4r/7ylTMCxFYDTG5Cas0PaQiHxHRE4UkaP9S0htMVnKZyHicqmyWWAzYVPVa1R1uqq2AhcDD6rq+0NuFvPnz6epqSnpWFNTE/Pn2xJ4xpSbsLpHj/eug5tIK3B6CG3hvo0Hc9aUl8N466pSrt2iFtiMycyfbHD55ZfT3d3NrFmzmD9/vk1CMKYMhRLaVPW0MN7XGLDAZsKnqg8DD4fcjH6XXXYZ119/PQAPP/xwuI0xxmQUSveoiEwWkRtE5C/e/UNE5PIw2mKyM1TXaCVV2YwxxphKFNaYtl8B9wL7evdfBT4TUlvMMFVSYLMqmzHGmEoV1pi2iap6s4hcA6CqfSJiS3+MEGFVvyywGWOMqWRhVdraRWQCbvIBInICsCuktphhyLXKZoHNGGOMyU9YlbbP4fYbnS0ifwcmAe8NqS1mCJnGs5XL3qLGGGPMSBBKpU1VnwbeArwJ+BfgUFV9bqjniUiDiDwpIs+KyFIR+Wqx22ryZ1U2Uygi8nkRURGZ6N0XEfmhiCwTkedsnUdjzEgQSqVNRN6dcuhAEdkFPK+qmwd5ajdwuqruFZFa4DER+YuqLipaY01a1i1qSkVEZgBvBYL7Kp0LzPEuxwPXkVj/0RhjqlJY3aOXAycCD3n3T8XtQ7qfiHxNVX+T7kmqqsBe726td7GNLIsoXddopXSLWmCrGt/HbbL+p8CxC4Abve+ERSIyVkSmquqGUFpojDElEFZoiwJvUNVN4NZtA27E/U/5USBtaPPOjeAC3gHAT1T1iUI0yHZFKJyhqmwWpky2ROQCYJ2qPisiwYemAWsC99d6xyy0GWOqVlihbYYf2DybvWPbRaR3sCeqagw4UkTGAreLyGGq+kLwHBGZC8wFqN+npbAtH2HOmvJyXnuOjgQ9B08fcGzX7PoBx/bMkgHHumf2DDjWOn1LYRpWItIj2VZZJ4rI4sD9Baq6oP91RO4HpqR53jzgS7iuUWOMGfHCCm0Pi8idwB+9++/xjjUDO7N5AVXdKSIPAecAL6Q8tgBYANBy0BTrPi2w1ulbkrpIu2f2JP3y3jNLkqptu2bXJ1Xb0oWdajCSAluOtqpqW6YHVfXMdMdF5I3AfoBfZZsOPC0ixwHrgBmB06d7x4wxpmqFtU7bVbhdEY70LjcCV6lq+2D7korIJK/Chog0AmcBBevTtIpSeoXoNk4XaKqdBbbhUdXnVXUfVW1V1VZcF+jRqroRt2TQB71ZpCcAu2w8mzGm2oW15Ieq6i2q+lnvcos3oHgoU4GHROQ54CngPlW9s5Bts+CWXmpwSw0aqWEkXWCpZtmEUgtsBXU3sAJYBlwPXBluc4wxpvjC2jD+BBF5SkT2ikiPiMREZPdQz1PV51T1KFU9XFUPU9WvFaN9Ftyyk2twq9ZqW7bdoqmyCWxnTXnZJsh4vIrbVu+2qupVqjpbVd+oqouHer4xxlS6sLpHfwxcArwGNAIfBX4SUlvK3n0bDy6LIJlPeKj24DaccWxDsbBmjDEmKKzQhqouAyKqGlPVX+ImFJSNcghJ5RLWgnLtJq0mu2bXD7ikKtQ4NgtsxhhjUoU1e7RDROqAZ0Tk27i1lUILkJmEtXZbpqBWLmvJpS4Dks9s0mpkgc0YY0wxhRWUPuC99yeAdtzU/feE1JZBlbrSNdT7lVvlLVvVPjHBApsxxphiC2v26CpV7VLV3ar6VVX9nNddOmLl0hVaDsFtJHeTpqr0wJa6LZkxxpjyVHZdkuWomCGpHMetZcuWAUnPAptJR0QaRORJEXlWRJaKyFfDbpMxprJYaMtSoYNVLmHt9bWTBvxyrZSgV+3BrRBLe1hgGzG6gdNV9QjcouLneAsDh27hwoUsWrSIRx55hNbWVhYuXFj092ttbaWmpqYq38+YYgllIoKINKhqV8qxif4aTOWqEBMBcg1bwV+sr6+dlBQAymFiwlCTEipZPgEz127hsP/8TOl4C4jv9e7WepfQt9lbuHAhc+fOpbvbbTW3atUq5s6dC8Bll11WtPfr6OioyvczppjCmj36lIhcoaqLAETkPcB/AweG1J6SKEZ1rBKCW7rZpNUo127RMP/cqiVYVxoRiQBLgAOAn6jqEyE3iXnz5vUHGl9HRweXX345119/fcHfb9GiRf0BMcz3mzdvnoU2U3HC6h69FPiRiHxHRBYCVwCnh9SWnOQTvPIdt5buF2ul/LIdaRMTLLCZbHjrUh6J2+D+OBE5LPUcEZkrIotFZPGWLcXf5mz16tVpj6cGnULJ9Lqlfr9MP7cx5SyUSpuqPi8i84HfAHuAU1R1bRhtyUe21a3hVNYG+8VaCd2kI0mlB7ZgFdSUhqruFJGHcIuKv5Dy2AJgAUBbW1vRu09nzpzJqlWrBhyfNWsWDz/8cMHfr7W1tSzeb+bMmQV/L2OKLay9R28APgMcDnwEuFNErgqjLfkaLKAMd0ZoNr9Yy3FiwkhcBqSSAls6FthKR0QmichY73YjcBYQ+l+I+fPn09TUlHSsqamJ+fPn2/sZU2bC6h59HjhNVVeq6r3A8cDRIbWloIYbnqqtElLNwa3SApt1i4ZuKvCQiDwHPAXcp6p3htwmLrvsMhYsWMCsWbMQEWbNmsWCBQuKNt4rrPerr3c7sRT7/YwpJnETmqpXy0FT9Jhr31+01/d/ERei0pVNYEsNCqkhIexgAAM/i9Sfq9xDaL7BspiBrdj/GXj1Pz63RFXbcnnNhmkzdNbHPjfkefm8tnHa2tp08eLFYTejKpx66qkARemCNaaQRCTjd2ZYS37Mwc0WPQRo8I+r6v5htGc4CtUtmW2FrX51XVKoqITxbelmkxZKNgEw7Ope2H8e1Va9NcaYkSqs7tFfAtcBfcBpwI3Ab0NqS+iG+0u1Ese3FUr3zJ4hL6WQ6ecLO7ClY4HNGGMqU1ihrVFVH8B1z65S1a8AbwupLaHKJ7DZL93yUs6BLZtxbC2rynuIhIh8UkRe9rZ++nbg+DUiskxEXhGRs8NsozHGlEJYoa1bRGqA10TkEyLyLmBUSG2pCKm/WKthNmk1qLTAVmmBX0ROAy4AjlDVQ4HvescPAS4GDsUtm3Gtt3CtMcZUrbBC26eBJuBTwDHAB4APhdSW0GT7S9UPbBbcykPr9C39l3QqKbCVe5UN+DjwTVXtBlDVzd7xC4CbVLVbVVcCy4DjQmqjMcaURFiL6z7l3dyLW6dtxCnVL9VymJhQSYYbKsv1s67QwAZua7uTvcW4u4B/9b4/pgGLAuet9Y4ZY0zVCmv2aBswD5gVbIOqHh5Ge0ptOIGtZZUm7d051GzScpBuNulw5LveWLE/l3IJbKUaxxbpyfp1JopIcN2KBd6q/wCIyP3AlDTPm4f7fhgPnAAcC9wsIhU3y9wYYwohrA3jFwL/hltkNx5SG0JRiApbrsGtHKpthdzmqtxCKZR3YCuDcWxbB1unTVXPzPSYiHwcuE3dgpJPikgcmAisA2YETp3uHTPGmKoV1pi2Lap6h7cjwir/ElJbSqaQXaKVOL6tWlVaYKuQblHf/+GWBUJEDgTqgK3AHcDFIlIvIvsBc4Anw2qkMcaUQliVtv8UkZ8DDwDd/kFVvS2k9hTdcH6hjlnuPqJds+uH1YawK27VuKl8uQS2dHL9+1WmfgH8QkReAHqAD3lVt6UicjPwIm69x6tUNRZiO40xpujCCm0fAQ4Gakl0jyowaGgTkRm4hXgne+cvUNUfFLGdBVGIwObfDga3ahjfVknKOaDlO46tzAMbqtoDpN2HTlXnA7brtzFmxAgrtB2rqgfl8bw+4POq+rSItABLROQ+VX2xwO0rmEIFtuCx4QS3sKttUN7BLezPJh/5jmMr98BmjDEmWVih7R8ickiuYUtVNwAbvNt7ROQl3DT/sg1tqUrRZVUpwS1XhQp6Yf/shVSl49iMMcakEVZoOwF4RkRW4sa0CaC5LPkhIq3AUcATRWlhAeS7NMVQhqq2VatqClvFUiXj2IwxxqQR1uzRc3Czvd4KvB0437vOioiMAm4FPqOqu9M8PldEFovI4t6dHQVqcm4K3S1a9/LaQc+x2aTGGGNMdQsltAWX+ch1yQ8RqcUFtoWZZpuq6gJVbVPVttqxTYVsujFlLdgt7ktXhR3uTGRjjDGlF1alLS8iIsANwEuq+r2w21Nt3jv6ad47+umwm2FykG52cLbBzRhjTGWpqNAGnITbXP50EXnGu5wXdqOqjYW3ypJtcEtl1TZjjKksFRXaVPUxVRVVPVxVj/Qud4fdrmplwa1yZLMen3WTGmNMZauo0GZKz4Jb5bLxbcYYU10stJkhWXdpZbDxbeVNRGaIyEMi8qKILBWRT4fdppHiyiuv5JFHHuGRRx4hGo1y5ZVXht0kY/Jioc1kzYJb+bPgVtb8HV0Owa1VeZWIHBJym6relVdeyXXXXdd/PxaLcd1111lwMxXJQpvJiQW38ldu+80aR1U3qOrT3u09gL+jiymiBQsW5HTcmHJmoc3kzIJb5bFqW3kZbEeX4OLgW7ZYAB+uWCyW03FjypmFNpMXG+dW3qybtHwNtaNLcHHwSZOKsxXeSBKJRHI6bkw5s9BmhsWCW/my4FZ+stnRxRTW3LlzczpuTDmz0GaGzYJb+cp34V1TeLajSziuvfZaPv7xj/ffj0QifPzjH+faa68NsVXG5MdCmykI6y4tX5U8MUFEjhSRRd7uJ4tF5DjvuIjID0VkmYg8JyJHh93WLNiOLiEJBrS+vj4LbKZiWWgzBWXBrTJUULXt28BXVfVI4D+8+wDnAnO8y1zgurTPLiO2o4sxZrgstJmCs+BWfiq4m1SB0d7tMcB67/YFwI3qLALGisjUMBpojDGlYqHNFIV1l5afEIPbRH8JC++SywjwzwDfEZE1wHeBa7zj04A1gfPWYmueGWOqXDTsBpjq9t7RT3PL7koYbjQytE7fwutrC7OMRKRLGbO8O5tTt6pqW6YHReR+YEqah+YBZwCfVdVbReQi3ED+M/NprzHGVDoLbaboLLiVt7C7SVU1YwgTkRsBf4/OPwI/926vA2YETp3uHTPGmKpl3aOmJKy7tHxU2GzS9cBbvNunA695t+8APujNIj0B2KWqG8JooDHGlIqFNlNSFtzKQwUFtyuA/xGRZ4H/ws0UBbgbWAEsA64HbPdvY0zVs+5RU3KlCm7WJTu4Qo5vKxZVfQw4Js1xBa4qfYuMMSY8VmkzVcuqesYYY6qJhTZT1Sy4Da6CukmNMWbEs9BmzAhnwc0YYyqDjWkzVa/YS47s3drN3d9+kTXP7aBxdC2R2hpO/shsDjkz3dJjxhhjTH4stJkRoVjBTVVZ+OnFHPWO6Vz07aMA2LG+g5cf2lTw9zLGGDOyWfeoGTGKMb5txRPbiNTWcNz7ZvUfG7dvEydetl/B38sYY8zIZqHNmGHYvGwP+75hTNjNMMYYMwJUVGgTkV+IyGYReSHstpjKVOzZpH/+xgv8+N2Pct37Hivq+xhjjBl5Kiq0Ab8Czgm7EaayFTK47XNAC+tf2tV//+3/fhgfueEE2neEu5+nMcaU0qZNm7j00kvZf//9OeaYYzjxxBO5/fbbw25W1amo0KaqjwLbw26HqXyFCm77Hz+Bvu44T9y0qv9Yb1esIK9tjDGVQFV55zvfySmnnMKKFStYsmQJN910E2vXrg27aVXHZo8aMwwiwmU/PIa7v/Uif/vlcprH1VHXGOHszx4cdtOMMaYkHnzwQerq6vjYxz7Wf2zWrFl88pOfDLFV1akqQ5uIzMXbWLp+n5ZQ2pBuX8fumT3Ur65LOrZnltCySpOO7Zpdz5jl3UnHeg6ePuCc1NdJfa/U9gSdNeXlIX4Ck61FsSMZ/69HMj5wbCOwcWNYLTLGmNJZunQpRx9tez2XQkV1j2ZLVReoapuqttWObQqtHcVaaX64gc0UlgVgky2bTGVGgquuuoojjjiCY489NuymVJ2qDG3lLDVQwcDQBQODWSFZyKDgC+3aZ2qy9CtsMlXJXXnllf23o9Fo0v1iWbhwIa2trdTU1NDa2srChQuL/p6lFPz55s+fzz333NP/2E9+8hMeeOABtmyxgkGhVVRoE5HfA48DB4nIWhG5POw2DSVdlWs4wc2qbMNXzC2tjBmMTaYqvSuvvJLrrruu/34sFuO6664ranBbuHAhc+fOZdWqVagqq1atYu7cuVUT3FJ/vs2bN/Pqq6/ykY98pP+cjo6OEFtYvSpqTJuqXhJ2G/JRqPFthQhsVhEqnrOmvMx9G20CQiGJyIXAV4A3AMep6uLAY9cAlwMx4FOqeq93/BzgB0AE+LmqfrPU7TblY8GCBWmPX3fddbz44otFec9FixbR3Z08Lrmjo4N58+Zx2WWXFeU9S2nevHkDQlk8Hufmm2/m4YcfZtKkSTQ3N/Otb30rpBZWr4oKbZWs0BMTiiFYgSr2IrRhKXaVzYJbwb0AvBv4WfCgiBwCXAwcCuwL3C8iB3oP/wQ4C1gLPCUid6hqcX47F0FwItXMmTNDbk3li8VKvwRPamDzrV69usQtKY5MP0dnZycrV64scWtGFgttFaLUVbZM4aaSw1ypukUtuBWOqr4EbmmVFBcAN6lqN7BSRJYBx3mPLVPVFd7zbvLOrZjQpqoLgAUAbW1tOsTpZgiRSCRtcItEIjz88MNFec/W1lZWrVo14Hi1hPCZM2dW9c9XzipqTFulG874tuEqVLfoLbuPTnsxyawbuuimAWsC99d6xzIdNyPU3LlzczpeCPPnz6epKXnlgqamJubPn1+09yylav/5yplV2kpsON2kwcdSn5/6HqVW7pU5C5ahmigiiwP3F3jVJABE5H5gSprnzVPVPxW9dSXkTaY6FfeZrAX+U1VvCLdV1e3aa68F3Ni2WCxGJBJh7ty5/ceLwR+3dvnll9Pd3c2sWbOYP39+VYxng+r/+cqZhbYQDCe4Vdrkg3IPc8VU7d2k0tVD3ctZbVOzVVXbMj2oqmfm8fbrgBmB+9O9YwxyPHSVOpmq0l177bVFDWnpXHbZZVx//fUAReuGDVO1/3zlykJbSNIFt5GklGEuzCpbtQe3EN0B/E5EvoebiDAHeBIQYI6I7IcLaxcDl4bWSmOMKSALbWVkqGpbpVXZ8pEuYA0nyJVDt6gFt/yJyLuAHwGTgLtE5BlVPVtVl4rIzbgJBn3AVaoa857zCeBe3JIfv1DVpSE13xhjCspCW4hy6SZNVQ7j2EplJHexjnSqejtwe4bH5gMDRj6r6t3A3UVumjHGlJyFtpBlG9zyUWlVtlwNVZUrhyqbz6ptxhhjhstCWxnINbiNpCpbrsopqKWy4GaMMWY4bJ22KlXtVbZKZX8uxhhj8mWhrUxku/CuVdmMMcaYkclCWxkZKrhlG9ismlPe7M/HGGNMPiy0lZlsK26msllwM8YYkysLbWUom+BmVbbKZ39WxhhjcmGhrQLZOLbqYcHNGGNMtiy0lal8gpkFAGOMMaZ6WWgrY+mCm1XZqo+FbWOMMdmw0Fbmsg1p9ou/stmfnzHGmKFYaKsgVmWrbhbcjDHGDMa2saoAQ4U1+2VvjDHGVD+rtBlTRiyAG2OMycRCW4WzX/LVx/5MjTHGpGOhzZgyZMHNGGNMqooLbSJyjoi8IiLLROTqsNsTJvvFbqqdiFwoIktFJC4ibYHjZ4nIEhF53rs+PfDYMd7xZSLyQxGRcFo/kH1/GWOGo6JCm4hEgJ8A5wKHAJeIyCHhtsqY4rBQDsALwLuBR1OObwXerqpvBD4E/Cbw2HXAFcAc73JOCdo5JPv+GlkWLlzIokWLeOSRR2htbWXhwoVFf7/W1lZqampK9n6l/PmMU1GhDTgOWKaqK1S1B7gJuCDkNoXCfqGPDCP9z1lVX1LVV9Ic/6eqrvfuLgUaRaReRKYCo1V1kaoqcCPwztK1eFD2/TVCLFy4kLlz59Ld3Q3AqlWrmDt3btGCjf9+q1atQlVL9n6l+vlMQqUt+TENWBO4vxY4PqS2GFMSZ015mfs2Hhx2M8rZe4CnVbVbRKbhvhd8a3HfG+XAvr9GiHnz5tHR0ZF0rKOjg8svv5zrr7++4O+3aNGi/gAV5vvNmzePyy67rODvZxIqLbRlRUTmAnO9u92PnPk/L4TZngKZiOsSAuCREBtSAEk/S4Ur0c9yV7Hf4KBcn7C7b8u992y8dmIWpzaIyOLA/QWqusC/IyL3A1PSPG+eqv5psBcWkUOBbwFvzabNlSD1+0tEquH7K5Nq+i4IOibdwe7ubh555JEl1fp+q1atQkSK8X5hCuPv6KxMD1RaaFsHzAjcn+4dS+L9QlgAICKLVbUt9ZxKUy0/B9jPUo5SQlVWVLUgY8VU9cx8nici04HbgQ+q6nLv8Drc94Iv7XdESEbs91cm1f7zQfX/jPbzlValjWl7CpgjIvuJSB1wMXBHyG0yxpSYiIzFlR+vVtW/+8dVdQOwW0RO8GaNfhAYtFpXQvb9ZYwZlooKbaraB3wCuBd4CbhZVZeG2ypjTLGIyLtEZC1wInCXiNzrPfQJ4ADgP0TkGe+yj/fYlcDPgWXAcuAvpW53Ovb9ZYwZrkrrHkVV7wbuzuEpC4Y+pSJUy88B9rOUo7L8OVT1dlwXaOrxbwDfyPCcxcBhRW5aXkbw91cm1f7zQfX/jPbzlZC4WfHGGGOMMaacVVT3qDHGGGPMSFW1oa1atosRkRki8pCIvOht5/PpsNs0HCISEZF/isidYbdlOERkrIjcIiIvi8hLInJi2G3Kl4h81vu79YKI/F5EGsJu00hXLd9fmVTb91om1fJ9l0k1fQ+mU47fjVUZ2qpsu5g+4POqeghwAnBVBf8sAJ/GDcKudD8A7lHVg4EjqNCfyVuM9lNAm6oeBkRwsxpNSKrs+yuTavtey6Ravu8yqYrvwXTK9buxKkMbVbRdjKpuUNWnvdt7cP8oymWF95x462q9DTezr2KJyBjgFOAGAFXtUdWdoTZqeKK4baCiQBOwfojzTXFVzfdXJtX0vZZJtXzfZVKF34PplN13Y7WGtnTbxVT8F4KItAJHAU+E3JR8/S/wBSAecjuGaz9gC/BLr+vj5yLSHHaj8qGq64DvAquBDcAuVf1ruK0a8ary+yuTKvhey+R/qY7vu0yq5nswnXL9bqzW0FZ1RGQUcCvwGVXdHXZ7ciUi5wObVbUatjiJAkcD16nqUUA7UJHjjkRkHK6Ksx+wL9AsIu8Pt1VmpKj077VMquz7LpOq+R5Mp1y/G6s1tGW1XUylEJFa3BfbQlW9Lez25Okk4B0i8jquu+d0EfltuE3K21pgrar6lYFbcF9elehMYKWqblHVXuA24E0ht2mkq6rvr0yq5Hstk2r6vsukmr4H0ynL78ZqDW1Vs12MtxXPDcBLqvq9sNuTL1W9RlWnq2or7s/jQVUN/X8t+VDVjcAaEfE3WT8DeDHEJg3HauAEEWny/q6dQRUNJq5QVfP9lUm1fK9lUk3fd5lU2fdgOmX53VhxOyJkQ1X7RMTfLiYC/KKCt4s5CfgA8LyIPOMd+5K3sroJzyeBhd4v1RXAR0JuT15U9QkRuQV4Gjej75+U2QrgI02VfX9lYt9r1aEqvgfTKdfvRtsRwRhjjDGmAlRr96gxxhhjTFWx0GaMMcYYUwEstBljjDHGVAALbcYYY4wxFcBCmzHGGGNMBbDQZowxxhhTASy0GWOMMcZUAAttxhhjjDEV4P8DluJ9RRPygoUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x576 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_train(agent, rtrace, steps, trace, start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAD8CAYAAAC8TPVwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAR+klEQVR4nO3df2zcd33H8ec7TWt6Aa0T8RCj+K5VBwyq/khMaQeLMrpWZSCG9g8/jklDmm4VXRW2SgzwH6h/3B9IE2ISCOlK+CHl2or1hzRViJWpjbdJuGCHAEnDJiixm1JaVxsDasml8N4fvlRJasdneufvfZznQzrl7uOP716ynNd9/Pl+v3ZkJpKkMmyrOoAkqX+WtiQVxNKWpIJY2pJUEEtbkgpiaUtSQdYt7Yh4fUQcPuX284j4yCZkkySdITZynnZEnAc8AbwlM+eHlkqStKqNbo9cD/zIwpakamzf4Pz3AXet9oGIaAEtgB07dux+wxve8BKjSdK5Y25u7pnMHF9vXt/bIxFxAfAT4E2Z+dTZ5k5OTubs7GxfzytJgoiYy8zJ9eZtZHvkHcCh9QpbkjQ8Gynt97PG1ogkaXP0VdoRsQO4AbhvuHEkSWfT14HIzHwWeOWQs0iS1uEVkZJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqiKUtSQWxtCWpIJa2JBXE0pakgljaklQQS1uSCmJpj4B2u82b3vQmrrjiCq666ioeeeSRqiNJGlF9/Y1IDc83v/lNHnjgAQ4dOsTY2BjPPPMMzz33XNWxJI0oS7tiTz75JDt37mRsbAyAnTt3VpxI0ihze6RiN954I48//jive93r+PCHP8z09HTVkSSNsL5KOyIuioh7IuIHEXEsIq4bdrCtrNvt0mg02LZtG5dffjm33XYbnU6H8fFx3vve9/LlL3+56oiSRlS/K+1/Ar6emW8ArgSODS/S1tbtdmm1WszPz5OZzM/Pc/PNN/PEE09w++2389nPfpZ777236piSRtS6pR0RvwPsAfYDZOZzmfmzIefasqamplhaWjptbGlpiampKQAOHz5MvV6vIpqkAvRzIPISYBH4UkRcCcwB+zLz2VMnRUQLaAFMTEwMOueWsbCwsOr4/Pw8V1xxBZdddhmdTmeTU0kqRT/bI9uBXcDnM/Nq4FngY2dOysxOZk5m5uT4+PiAY24da72h1et1vve973Hfffd5BomkNfVT2ieAE5l58oqPe1gpcf0W2u02tVrttLFarUa73a4okaSSrFvamflT4PGIeH1v6Hrg0aGm2sKazSadTueF87Lr9TqdTodms1lxMkkl6PfimluBbkRcADwGfGh4kba+ZrPJHXfcAcDBgwerDSOpKH2VdmYeBiaHG0WStB6viJSkgljaklQQS1uSCmJpS1JBLG1JKoilLUkFsbQlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqiKUtSQXpq7Qj4nhEfD8iDkfE7LBDbXXdbpeZmRmmp6dpNBp0u92qI0kqxPYNzP2TzHxmaEnOEd1ul1arxfLyMgDz8/O0Wi0Ams1mldEkFcDtkU02NTXF0tLSaWNLS0tMTU1VlEhSSfot7QQejIi5iGitNiEiWhExGxGzi4uLg0u4xSwsLGxoXJJO1W9pvy0zdwHvAG6JiD1nTsjMTmZOZubk+Pj4QENuJRMTExsal6RT9VXamflE79+ngfuBa4YZaitrt9vUarXTxmq1Gu12u6JEkkqybmlHxI6IeMXJ+8CNwJFhB9uqms0mnU6HsbExAOr1Op1Ox4OQkvrSz9kjrwLuj4iT8+/MzK8PNdUW12w2ueOOOwA4ePBgtWEkFWXd0s7Mx4ArNyGLJGkdnvInSQWxtCWpIJa2JBXE0pakgljaklQQS1uSCmJpS1JBLG1JKoilLUkFsbQlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IK0ndpR8R5EfGdiHhgmIEkSWvbyEp7H3BsWEHOJd1ul5mZGaanp2k0GnS73aojSSpEX6UdERcD7wS+MNw4W1+326XVarG8vAzA/Pw8rVbL4pbUl35X2p8BPgr8ZnhRzg1TU1MsLS2dNra0tMTU1FRFiSSVZN3Sjoh3AU9n5tw681oRMRsRs4uLiwMLuNUsLCxsaFySTtXPSvutwLsj4jhwN/D2iDhw5qTM7GTmZGZOjo+PDzjm1jExMbGhcUk61bqlnZkfz8yLM7MBvA94KDM/OPRkW1S73aZWq502VqvVaLfbFSWSVBLP095kzWaTTqfD2NgYAPV6nU6nQ7PZrDiZpBJEZg78SScnJ3N2dnbgz7uV7N27F4CDBw9WmkPSaIiIucycXG+eK21JKoilLUkFsbQlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqiKUtSQWxtCWpIJa2JBXE0pakgljaklSQdUs7Il4WEd+KiO9GxNGIuH0zgkmSXqyflfYy8PbMvBK4CrgpIq4daqotrtvtMjMzw/T0NI1Gg263W3Ukut0ujUaDbdu2jUwmSS+2fb0JmZnAL3sPz+/dcpihtrJut0ur1WJ5eRmA+fl5Wq0WAM1ms9JMS0tLI5NJ0upipZPXmRRxHjAHXAZ8LjP/4WzzJycnc3Z2djAJt5hGo8H8/PyLxsfGxrj22mp+gJmZmXnhTeRU9Xqd48ePb34g6RwUEXOZObnevL4ORGbmrzPzKuBi4JqIuHyVF2xFxGxEzC4uLm448LliYWFh1fHVSnOzrPXaa2WVVJ11t0dOlZk/i4iHgZuAI2d8rAN0YGWlPbCEW8zExMSqK+16vc7Bgwc3PxBrr/4nJiYqSCPpbPo5e2Q8Ii7q3b8QuAH4wZBzbVntdptarXbaWK1Wo91uV5RoNDNJWl0/2yOvBh6OiO8B3wa+kZkPDDfW1tVsNul0OtTrdSKCer1Op9Op9IDfyUxjY2MAI5FJ0ur6OhC5UR6ILNPevXsBKtumkc5lAz0QKUkaDZa2JBXE0pakgljaklQQS1uSCmJpS1JBLG1JKoilLUkFsbQlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqyLqlHRGvjYiHI+LRiDgaEfs2I5g2V7fbZWZmhunpaRqNBt1ut+pIklaxvY85zwO3ZeahiHgFMBcR38jMR4ecTZuk2+3SarVYXl4GYH5+nlarBUCz2awymqQzrLvSzswnM/NQ7/4vgGPAa4YdTJtnamqKpaWl08aWlpaYmpqqKJGktWxoTzsiGsDVwCOrfKwVEbMRMbu4uDigeNoMCwsLGxqXVJ2+SzsiXg7cC3wkM39+5sczs5OZk5k5OT4+PsiMGrKJiYkNjUuqTl+lHRHns1LY3cy8b7iRtNna7Ta1Wu20sVqtRrvdriiRpLX0c/ZIAPuBY5n56eFH0mZrNpt0Oh3GxsYAqNfrdDodD0JKIygy8+wTIt4G/AfwfeA3veFPZObX1vqcycnJnJ2dHVhIbY69e/cCcPDgwUpzSOeiiJjLzMn15q17yl9m/icQA0klSXpJvCJSkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqiKUtSQWxtCWpIJa2JBXE0pakgljaklQQS1uSCmJpS1JBLG1pi3nqqaf4wAc+wKWXXsru3bu57rrruP/++6uOpQGxtKUtJDN5z3vew549e3jssceYm5vj7rvv5sSJE1VH04BY2tIW8tBDD3HBBRdw8803vzBWr9e59dZbK0ylQbK0pS3k6NGj7Nq1q+oYGqJ1SzsivhgRT0fEkc0IpGp0u11mZmaYnp6m0WjQ7XarjkS326XRaLBt27aRyQSjmetkpn379rF///4XMt1yyy1ceeWVvPnNb644oQYmM896A/YAu4Aj6809edu9e3eqHAcOHMharZbAC7darZYHDhwwUwG51su0uLiY9Xq9snzqDzCbffRrrMw9u4hoAA9k5uX9vBFMTk7m7Ozsxt49VJlGo8H8/PyLxsfGxrj22msrSAQzMzMsLy+/aLzKTDCaudbKVK/XOX78OAsLC+zZs4fjx49vfjj1LSLmMnNyvXnbB/iCLaAFMDExMain1SZYWFhYdXy1Itgsa712lZnO9vqj+LWan5/nmmuuYceOHXzqU5/a5FQaFlfaWnOlfXKlVoVRzASjmWsUM2nj+l1pe/aIaLfb1Gq108ZqtRrtdruiRKOZCUYz1yhm0hD1s/ENNPBA5JZ24MCBrNfrGRFZr9crP+A3qpkyRzPXKGbSxjCoA5ERcRewF9gJPAV8MjP3n+1z3B6RpI0Z2IHIzHz/YCJJkl4q97QlqSCWtiQVxNKWpIJY2pJUEEtbkgpiaUtSQSxtSSqIpS1JBbG0JakglrYkFcTSlqSCWNqSVBBLW5IKYmlLUkEsbUkqiKUtSQWxtCWpIJa2JBXE0pakgljaklQQS1uSCmJpS1JB+irtiLgpIv4rIn4YER8bdihJ0urWLe2IOA/4HPAO4I3A+yPijcMOJkl6sX5W2tcAP8zMxzLzOeBu4M+HG0uStJrtfcx5DfD4KY9PAG85c1JEtIBW7+FyRBx56fEGaifwTNUhzmCm/oxiJhjNXGbqzyhmen0/k/op7b5kZgfoAETEbGZODuq5B8FM/TFT/0Yxl5n6M6qZ+pnXz/bIE8BrT3l8cW9MkrTJ+intbwN/EBGXRMQFwPuAfxluLEnSatbdHsnM5yPib4F/Bc4DvpiZR9f5tM4gwg2Ymfpjpv6NYi4z9afYTJGZww4iSRoQr4iUpIJY2pJUkIGW9ihe7h4RX4yIp0fpvPGIeG1EPBwRj0bE0YjYNwKZXhYR34qI7/Yy3V51ppMi4ryI+E5EPFB1FoCIOB4R34+Iw/2epjVsEXFRRNwTET+IiGMRcd0IZHp972t08vbziPjICOT6u973+JGIuCsiXjYCmfb18hxd92uUmQO5sXKQ8kfApcAFwHeBNw7q+V9Crj3ALuBI1VlOyfRqYFfv/iuA/676awUE8PLe/fOBR4Brq/5a9fL8PXAn8EDVWXp5jgM7q85xRqavAH/du38BcFHVmc7Idx7wU6BecY7XAD8GLuw9/irwVxVnuhw4AtRYOTnk34DL1po/yJX2SF7unpn/DvxP1TlOlZlPZuah3v1fAMdY+WaqMlNm5i97D8/v3So/Sh0RFwPvBL5QdZZRFRG/w8riZD9AZj6XmT+rNNSLXQ/8KDPnqw7CSjFeGBHbWSnKn1Sc5w+BRzJzKTOfB6aBv1hr8iBLe7XL3SstohJERAO4mpWVbaV62xCHgaeBb2Rm5ZmAzwAfBX5TcY5TJfBgRMz1fn1D1S4BFoEv9baRvhARO6oOdYb3AXdVHSIznwD+EVgAngT+LzMfrDYVR4A/johXRkQN+DNOv6DxNB6IrFBEvBy4F/hIZv686jyZ+evMvIqVq16viYjLq8wTEe8Cns7MuSpzrOJtmbmLld98eUtE7Kk4z3ZWtgA/n5lXA88CI3FMCaB3Ud67gX8egSy/y8oOwCXA7wM7IuKDVWbKzGPAp4AHga8Dh4FfrzV/kKXt5e4bEBHns1LY3cy8r+o8p+r9aP0wcFPFUd4KvDsijrOy3fb2iDhQbaQXVmtk5tPA/axsDVbpBHDilJ+M7mGlxEfFO4BDmflU1UGAPwV+nJmLmfkr4D7gjyrORGbuz8zdmbkH+F9WjnOtapCl7eXufYqIYGX/8VhmfrrqPAARMR4RF/XuXwjcAPygykyZ+fHMvDgzG6x8Pz2UmZWuiiJiR0S84uR94EZWfrytTGb+FHg8Ik7+lrjrgUcrjHSm9zMCWyM9C8C1EVHr/T+8npVjSpWKiN/r/TvByn72nWvNHeRv+fttLncfuoi4C9gL7IyIE8AnM3N/tal4K/CXwPd7e8gAn8jMr1UXiVcDX+n90YttwFczcyROsRsxrwLuX/n/znbgzsz8erWRALgV6PYWTI8BH6o4D/DCG9sNwN9UnQUgMx+JiHuAQ8DzwHcYjUva742IVwK/Am4524FkL2OXpIJ4IFKSCmJpS1JBLG1JKoilLUkFsbQlqSCWtiQVxNKWpIL8P7DZg47RsG/yAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_start = [0,2]\n",
    "test_trace = agent.test(test_start)\n",
    "plot_trace(agent, test_start, test_trace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAD8CAYAAAC8TPVwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPF0lEQVR4nO3df6zddX3H8efLliKtRBZ7XZzIBcLECeHntcpwhMEg+CPOLEsE0ajZ0hEZgY3EzJFl4Y/+QbIYTSRkN6CYWCCOH2ZpNgcTqDGButtSsaVs0UqhFeklzqE2UtH3/rgHLO1tz7f0nH7vB56P5IZzT78trzTts9/7Pd/TpqqQJLXhdX0PkCR1Z7QlqSFGW5IaYrQlqSFGW5IaYrQlqSFDo53k5CQb9/h4Lsk1h2GbJGkvOZj7tJMsAnYA766qbWNbJUma18FeHrkQ+IHBlqR+LD7I4y8Fbp/vG5KsBFYCLFu27Ox3vOMdhzhNkl471q9f/2xVTQw7rvPlkSRLgB8Bp1TVMwc6dmpqqmZmZjr9uJIkSLK+qqaGHXcwl0feB2wYFmxJ0vgcTLQvYz+XRiRJh0enaCdZBlwE3D3eOZKkA+n0QmRV/QJ405i3SJKG8B2RktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDTHaktQQoy1JDekU7STHJLkzyeNJtiQ5Z9zDJLUtyT4ffVuImw7W4o7HfQH4RlX9eZIlwNIxbpLUuP3FMAlVdZjX/Pb/vb/n+9r0SgyNdpI3AucBnwSoqt3A7vHOkvRqdf755/c9oWldLo+cAMwCX07ySJKbkyzb+6AkK5PMJJmZnZ0d+VBJEmTYlwVJpoCHgXOral2SLwDPVdU/7O/7TE1N1czMzGiXSmrGga4VL7TLI9Dfpj0lWV9VU8OO63KmvR3YXlXrBp/fCZx1KOMkSa/M0GhX1Y+Bp5KcPHjqQuCxsa6S1LT9nbn2eUa7EDe9El3vHrkKWD24c2Qr8KnxTZL0arAQY1hVL70Q+uCDD/a65ZXqFO2q2ggMvdYiSRov3xEpSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMt6TVr1apVnHLKKZx22mmcccYZrFu3ru9JQ3X9h30l6VXloYceYs2aNWzYsIEjjzySZ599lt27d/c9ayijLek16emnn2b58uUceeSRACxfvrznRd14eUTSa9LFF1/MU089xdvf/nY+/elPs3bt2r4nddIp2kmeSPK9JBuTzIx7lCSNw+rVq3n44YdZu3Ytp556Ktdeey3T09NMTEzwkY98hFtvvbXviUMdzJn2H1fVGVU1NbY1kjQmq1evZuXKlTz//PMAbNu2jSuuuIIdO3Zw/fXX88UvfpG77rqr55XDeXlE0mvCddddx65du1723K5du7juuusA2LhxI5OTk31MOyhdX4gs4N4kBfxzVU3vfUCSlcBKgOOOO250CyVpBJ588sl5n9+2bRunnXYaJ510EtPT+6Rtweka7fdW1Y4kbwbuS/J4VX1rzwMGIZ8GmJqaqhHvlKRDctxxx7Ft27Z9np+cnOTRRx/tYdEr0+nySFXtGPx3J3APsGKcoyRp1FatWsXSpUtf9tzSpUtZtWpVT4temaHRTrIsydEvPgYuBjaNe5gkjdLll1/O9PQ0k5OTJGFycpLp6Wkuv/zyvqcdlFQd+EpGkhOZO7uGucspt1XVAf9ompqaqpkZ7wyUpK6SrO9yd97Qa9pVtRU4fSSrJEmHxFv+JKkhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGmK0JakhRluSGtI52kkWJXkkyZpxDpIk7d/BnGlfDWwZ1xBJ0nCdop3kWOADwM3jnSNJOpCuZ9qfBz4D/GZ8UyRJwwyNdpIPAjurav2Q41YmmUkyMzs7O7KBkqTf6nKmfS7woSRPAHcAFyT56t4HVdV0VU1V1dTExMSIZ0qSoEO0q+qzVXVsVR0PXArcX1UfG/sySdI+vE9bkhqy+GAOrqoHgQfHskSSNJRn2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0x2pLUEKMtSQ0ZGu0kr0/ynSTfTbI5yfWHY5gkaV+LOxzzPHBBVf08yRHAt5P8e1U9POZtkqS9DI12VRXw88GnRww+apyjJEnz63RNO8miJBuBncB9VbVurKskSfPqFO2q+nVVnQEcC6xIcurexyRZmWQmyczs7OyIZ0qS4CDvHqmqnwIPAJfM823TVTVVVVMTExMjmidJ2lOXu0cmkhwzeHwUcBHw+Jh3SZLm0eXukbcAX0myiLnIf62q1ox3liRpPl3uHnkUOPMwbJEkDeE7IiWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIUOjneRtSR5I8liSzUmuPhzDJEn7WtzhmBeAa6tqQ5KjgfVJ7quqx8a8TZK0l6Fn2lX1dFVtGDz+GbAFeOu4h0mS9nVQ17STHA+cCayb59tWJplJMjM7OzuieZKkPXWOdpI3AHcB11TVc3t/e1VNV9VUVU1NTEyMcqMkaaBTtJMcwVywV1fV3eOdJEnany53jwS4BdhSVZ8b/yRJ0v50OdM+F/g4cEGSjYOP9495lyRpHkNv+auqbwM5DFskSUP4jkhJaojRlqSGGG1JaojRlqSGGG1JaojRlqSGGG1JaojRlqSGGG1JaojRlqSGGG1JaojRlqSGGG1JaojRlqSGGG1JaojRlqSGGG1JaojRlqSGGG3pVeaZZ57hox/9KCeeeCJnn30255xzDvfcc0/fszQiRlt6FakqPvzhD3PeeeexdetW1q9fzx133MH27dv7nqYRMdrSq8j999/PkiVLuOKKK156bnJykquuuqrHVRoloy29imzevJmzzjqr7xkao6HRTvKlJDuTbDocgySNzpVXXsnpp5/Ou971rr6naES6nGnfClwy5h2SRuCUU05hw4YNL31+44038s1vfpPZ2dkeV2mUhka7qr4F/OQwbJF0iC644AJ++ctfctNNN7303K5du3pcpFEb2TXtJCuTzCSZ8U91qR9J+PrXv87atWs54YQTWLFiBZ/4xCe44YYb+p6mEUlVDT8oOR5YU1WndvlBp6amamZm5hCnSdJrR5L1VTU17DjvHpGkhhhtSWpIl1v+bgceAk5Osj3JX4x/liRpPouHHVBVlx2OIZKk4bw8IkkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1BCjLUkNMdqS1JBO0U5ySZL/TvL9JH837lGSpPkNjXaSRcCNwPuAdwKXJXnnuIdJkvbV5Ux7BfD9qtpaVbuBO4A/He8sSdJ8Fnc45q3AU3t8vh14994HJVkJrBx8+nySTYc+b6SWA8/2PWIvbupmIW6ChbnLTd0sxE0ndzmoS7Q7qappYBogyUxVTY3qxx4FN3Xjpu4W4i43dbNQN3U5rsvlkR3A2/b4/NjBc5Kkw6xLtP8L+P0kJyRZAlwK/Ot4Z0mS5jP08khVvZDkr4H/ABYBX6qqzUO+2/Qoxo2Ym7pxU3cLcZebuml2U6pq3EMkSSPiOyIlqSFGW5IaMtJoL8S3uyf5UpKdC+m+8SRvS/JAkseSbE5y9QLY9Pok30ny3cGm6/ve9KIki5I8kmRN31sAkjyR5HtJNna9TWvckhyT5M4kjyfZkuScBbDp5MHP0YsfzyW5ZgHs+pvBr/FNSW5P8voFsOnqwZ7NQ3+OqmokH8y9SPkD4ERgCfBd4J2j+vEPYdd5wFnApr637LHpLcBZg8dHA//T988VEOANg8dHAOuA9/T9czXY87fAbcCavrcM9jwBLO97x16bvgL85eDxEuCYvjfttW8R8GNgsucdbwV+CBw1+PxrwCd73nQqsAlYytzNIf8JnLS/40d5pr0g3+5eVd8CftL3jj1V1dNVtWHw+GfAFuZ+MfW5qarq54NPjxh89P4qdZJjgQ8AN/e9ZaFK8kbmTk5uAaiq3VX1015H7etC4AdVta3vIcyF8agki5kL5Y963vMHwLqq2lVVLwBrgT/b38GjjPZ8b3fvNUQtSHI8cCZzZ7a9GlyG2AjsBO6rqt43AZ8HPgP8pucdeyrg3iTrB399Q99OAGaBLw8uI92cZFnfo/ZyKXB73yOqagfwT8CTwNPA/1XVvf2uYhPwR0nelGQp8H5e/obGl/GFyB4leQNwF3BNVT3X956q+nVVncHcu15XJDm1zz1JPgjsrKr1fe6Yx3ur6izm/ubLK5Oc1/OexcxdArypqs4EfgEsiNeUAAZvyvsQ8C8LYMvvMHcF4ATg94BlST7W56aq2gLcANwLfAPYCPx6f8ePMtq+3f0gJDmCuWCvrqq7+96zp8GX1g8Al/Q85VzgQ0meYO5y2wVJvtrvpJfO1qiqncA9zF0a7NN2YPseXxndyVzEF4r3ARuq6pm+hwB/Avywqmar6lfA3cAf9ryJqrqlqs6uqvOA/2Xuda55jTLavt29oyRh7vrjlqr6XN97AJJMJDlm8Pgo4CLg8T43VdVnq+rYqjqeuV9P91dVr2dFSZYlOfrFx8DFzH1525uq+jHwVJIX/5a4C4HHepy0t8tYAJdGBp4E3pNk6eD34YXMvabUqyRvHvz3OOauZ9+2v2NH+bf8vZK3u49dktuB84HlSbYD/1hVt/S7inOBjwPfG1xDBvj7qvq3/ibxFuArg3/04nXA16pqQdxit8D8LnDP3O93FgO3VdU3+p0EwFXA6sEJ01bgUz3vAV76g+0i4K/63gJQVeuS3AlsAF4AHmFhvKX9riRvAn4FXHmgF5J9G7skNcQXIiWpIUZbkhpitCWpIUZbkhpitCWpIUZbkhpitCWpIf8PJlSfFj9p6BUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_start = [1,8]\n",
    "test_trace = agent.test(test_start)\n",
    "plot_trace(agent, test_start, test_trace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q: What do you think about the test results above? Does the agent run fine? "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}