reinforcement/Q-learning-cart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-learning\n",
    "\n",
    "In this notebook, we'll build a neural network that can learn to play games through reinforcement learning. More specifically, we'll use Q-learning to train an agent to play a game called [Cart-Pole](https://gym.openai.com/envs/CartPole-v0). In this game, a freely swinging pole is attached to a cart. The cart can move to the left and right, and the goal is to keep the pole upright as long as possible.\n",
    "\n",
    "![Cart-Pole](assets/cart-pole.jpg)\n",
    "\n",
    "We can simulate this game using [OpenAI Gym](https://gym.openai.com/). First, let's check out how OpenAI Gym works. Then, we'll get into training an agent to play the Cart-Pole game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Note:** Make sure you have OpenAI Gym cloned into the same directory with this notebook. I've included `gym` as a submodule, so you can run `git submodule --init --recursive` to pull the contents into the `gym` repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-04-13 12:20:53,011] Making new env: CartPole-v0\n"
     ]
    }
   ],
   "source": [
    "# Create the Cart-Pole game environment\n",
    "env = gym.make('CartPole-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We interact with the simulation through `env`. To show the simulation running, you can use `env.render()` to render one frame. Passing in an action as an integer to `env.step` will generate the next step in the simulation.  You can see how many actions are possible from `env.action_space` and to get a random action you can use `env.action_space.sample()`. This is general to all Gym games. In the Cart-Pole game, there are two possible actions, moving the cart left or right. So there are two actions we can take, encoded as 0 and 1.\n",
    "\n",
    "Run the code below to watch the simulation run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.reset()\n",
    "rewards = []\n",
    "for _ in range(100):\n",
    "    env.render()\n",
    "    state, reward, done, info = env.step(env.action_space.sample()) # take a random action\n",
    "    rewards.append(reward)\n",
    "    if done:\n",
    "        rewards = []\n",
    "        env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To shut the window showing the simulation, use `env.close()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you ran the simulation above, we can look at the rewards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "print(rewards[-20:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The game resets after the pole has fallen past a certain angle. For each frame while the simulation is running, it returns a reward of 1.0. The longer the game runs, the more reward we get. Then, our network's goal is to maximize the reward by keeping the pole vertical. It will do this by moving the cart to the left and the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-Network\n",
    "\n",
    "We train our Q-learning agent using the Bellman Equation:\n",
    "\n",
    "$$\n",
    "Q(s, a) = r + \\gamma \\max{Q(s', a')}\n",
    "$$\n",
    "\n",
    "where $s$ is a state, $a$ is an action, and $s'$ is the next state from state $s$ and action $a$.\n",
    "\n",
    "Before we used this equation to learn values for a Q-_table_. However, for this game there are a huge number of states available. The state has four values: the position and velocity of the cart, and the position and velocity of the pole. These are all real-valued numbers, so ignoring floating point precisions, you practically have infinite states. Instead of using a table then, we'll replace it with a neural network that will approximate the Q-table lookup function.\n",
    "\n",
    "<img src=\"assets/deep-q-learning.png\" width=450px>\n",
    "\n",
    "Now, our Q value, $Q(s, a)$ is calculated by passing in a state to the network. The output will be Q-values for each available action, with fully connected hidden layers.\n",
    "\n",
    "<img src=\"assets/q-network.png\" width=550px>\n",
    "\n",
    "\n",
    "As I showed before, we can define our targets for training as $\\hat{Q}(s,a) = r + \\gamma \\max{Q(s', a')}$. Then we update the weights by minimizing $(\\hat{Q}(s,a) - Q(s,a))^2$. \n",
    "\n",
    "For this Cart-Pole game, we have four inputs, one for each value in the state, and two outputs, one for each action. To get $\\hat{Q}$, we'll first choose an action, then simulate the game using that action. This will get us the next state, $s'$, and the reward. With that, we can calculate $\\hat{Q}$ then pass it back into the $Q$ network to run the optimizer and update the weights.\n",
    "\n",
    "Below is my implementation of the Q-network. I used two fully connected layers with ReLU activations. Two seems to be good enough, three might be better. Feel free to try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "class QNetwork:\n",
    "    def __init__(self, learning_rate=0.01, state_size=4, \n",
    "                 action_size=2, hidden_size=10, \n",
    "                 name='QNetwork'):\n",
    "        # state inputs to the Q-network\n",
    "        with tf.variable_scope(name):\n",
    "            self.inputs_ = tf.placeholder(tf.float32, [None, state_size], name='inputs')\n",
    "            \n",
    "            # One hot encode the actions to later choose the Q-value for the action\n",
    "            self.actions_ = tf.placeholder(tf.int32, [None], name='actions')\n",
    "            one_hot_actions = tf.one_hot(self.actions_, action_size)\n",
    "            \n",
    "            # Target Q values for training\n",
    "            self.targetQs_ = tf.placeholder(tf.float32, [None], name='target')\n",
    "            \n",
    "            # ReLU hidden layers\n",
    "            self.fc1 = tf.contrib.layers.fully_connected(self.inputs_, hidden_size)\n",
    "            self.fc2 = tf.contrib.layers.fully_connected(self.fc1, hidden_size)\n",
    "\n",
    "            # Linear output layer\n",
    "            self.output = tf.contrib.layers.fully_connected(self.fc2, action_size, \n",
    "                                                            activation_fn=None)\n",
    "            \n",
    "            ### Train with loss (targetQ - Q)^2\n",
    "            # output has length 2, for two actions. This next line chooses\n",
    "            # one value from output (per row) according to the one-hot encoded actions.\n",
    "            self.Q = tf.reduce_sum(tf.multiply(self.output, one_hot_actions), axis=1)\n",
    "            \n",
    "            self.loss = tf.reduce_mean(tf.square(self.targetQs_ - self.Q))\n",
    "            self.opt = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experience replay\n",
    "\n",
    "Reinforcement learning algorithms can have stability issues due to correlations between states. To reduce correlations when training, we can store the agent's experiences and later draw a random mini-batch of those experiences to train on. \n",
    "\n",
    "Here, we'll create a `Memory` object that will store our experiences, our transitions $<s, a, r, s'>$. This memory will have a maxmium capacity, so we can keep newer experiences in memory while getting rid of older experiences. Then, we'll sample a random mini-batch of transitions $<s, a, r, s'>$ and train on those.\n",
    "\n",
    "Below, I've implemented a `Memory` object. If you're unfamiliar with `deque`, this is a double-ended queue. You can think of it like a tube open on both sides. You can put objects in either side of the tube. But if it's full, adding anything more will push an object out the other side. This is a great data structure to use for the memory buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "class Memory():\n",
    "    def __init__(self, max_size = 1000):\n",
    "        self.buffer = deque(maxlen=max_size)\n",
    "    \n",
    "    def add(self, experience):\n",
    "        self.buffer.append(experience)\n",
    "            \n",
    "    def sample(self, batch_size):\n",
    "        idx = np.random.choice(np.arange(len(self.buffer)), \n",
    "                               size=batch_size, \n",
    "                               replace=False)\n",
    "        return [self.buffer[ii] for ii in idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploration - Exploitation\n",
    "\n",
    "To learn about the environment and rules of the game, the agent needs to explore by taking random actions. We'll do this by choosing a random action with some probability $\\epsilon$ (epsilon).  That is, with some probability $\\epsilon$ the agent will make a random action and with probability $1 - \\epsilon$, the agent will choose an action from $Q(s,a)$. This is called an **$\\epsilon$-greedy policy**.\n",
    "\n",
    "\n",
    "At first, the agent needs to do a lot of exploring. Later when it has learned more, the agent can favor choosing actions based on what it has learned. This is called _exploitation_. We'll set it up so the agent is more likely to explore early in training, then more likely to exploit later in training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-Learning training algorithm\n",
    "\n",
    "Putting all this together, we can list out the algorithm we'll use to train the network. We'll train the network in _episodes_. One *episode* is one simulation of the game. For this game, the goal is to keep the pole upright for 195 frames. So we can start a new episode once meeting that goal. The game ends if the pole tilts over too far, or if the cart moves too far the left or right. When a game ends, we'll start a new episode. Now, to train the agent:\n",
    "\n",
    "* Initialize the memory $D$\n",
    "* Initialize the action-value network $Q$ with random weights\n",
    "* **For** episode = 1, $M$ **do**\n",
    "  * **For** $t$, $T$ **do**\n",
    "     * With probability $\\epsilon$ select a random action $a_t$, otherwise select $a_t = \\mathrm{argmax}_a Q(s,a)$\n",
    "     * Execute action $a_t$ in simulator and observe reward $r_{t+1}$ and new state $s_{t+1}$\n",
    "     * Store transition $<s_t, a_t, r_{t+1}, s_{t+1}>$ in memory $D$\n",
    "     * Sample random mini-batch from $D$: $<s_j, a_j, r_j, s'_j>$\n",
    "     * Set $\\hat{Q}_j = r_j$ if the episode ends at $j+1$, otherwise set $\\hat{Q}_j = r_j + \\gamma \\max_{a'}{Q(s'_j, a')}$\n",
    "     * Make a gradient descent step with loss $(\\hat{Q}_j - Q(s_j, a_j))^2$\n",
    "  * **endfor**\n",
    "* **endfor**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters\n",
    "\n",
    "One of the more difficult aspects of reinforcememt learning are the large number of hyperparameters. Not only are we tuning the network, but we're tuning the simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_episodes = 1000          # max number of episodes to learn from\n",
    "max_steps = 200                # max steps in an episode\n",
    "gamma = 0.99                   # future reward discount\n",
    "\n",
    "# Exploration parameters\n",
    "explore_start = 1.0            # exploration probability at start\n",
    "explore_stop = 0.01            # minimum exploration probability \n",
    "decay_rate = 0.0001            # exponential decay rate for exploration prob\n",
    "\n",
    "# Network parameters\n",
    "hidden_size = 64               # number of units in each Q-network hidden layer\n",
    "learning_rate = 0.0001         # Q-network learning rate\n",
    "\n",
    "# Memory parameters\n",
    "memory_size = 10000            # memory capacity\n",
    "batch_size = 20                # experience mini-batch size\n",
    "pretrain_length = batch_size   # number experiences to pretrain the memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "mainQN = QNetwork(name='main', hidden_size=hidden_size, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate the experience memory\n",
    "\n",
    "Here I'm re-initializing the simulation and pre-populating the memory. The agent is taking random actions and storing the transitions in memory. This will help the agent with exploring the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the simulation\n",
    "env.reset()\n",
    "# Take one random step to get the pole and cart moving\n",
    "state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "memory = Memory(max_size=memory_size)\n",
    "\n",
    "# Make a bunch of random actions and store the experiences\n",
    "for ii in range(pretrain_length):\n",
    "    # Uncomment the line below to watch the simulation\n",
    "    # env.render()\n",
    "\n",
    "    # Make a random action\n",
    "    action = env.action_space.sample()\n",
    "    next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "    if done:\n",
    "        # The simulation fails so no next state\n",
    "        next_state = np.zeros(state.shape)\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        \n",
    "        # Start new episode\n",
    "        env.reset()\n",
    "        # Take one random step to get the pole and cart moving\n",
    "        state, reward, done, _ = env.step(env.action_space.sample())\n",
    "    else:\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Below we'll train our agent. If you want to watch it train, uncomment the `env.render()` line. This is slow because it's rendering the frames slower than the network can train. But, it's cool to watch the agent get better at the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now train with experiences\n",
    "saver = tf.train.Saver()\n",
    "rewards_list = []\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    step = 0\n",
    "    for ep in range(1, train_episodes):\n",
    "        total_reward = 0\n",
    "        t = 0\n",
    "        while t < max_steps:\n",
    "            step += 1\n",
    "            # Uncomment this next line to watch the training\n",
    "            # env.render() \n",
    "            \n",
    "            # Explore or Exploit\n",
    "            explore_p = explore_stop + (explore_start - explore_stop)*np.exp(-decay_rate*step) \n",
    "            if explore_p > np.random.rand():\n",
    "                # Make a random action\n",
    "                action = env.action_space.sample()\n",
    "            else:\n",
    "                # Get action from Q-network\n",
    "                feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "                Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "                action = np.argmax(Qs)\n",
    "            \n",
    "            # Take action, get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "    \n",
    "            total_reward += reward\n",
    "            \n",
    "            if done:\n",
    "                # the episode ends so no next state\n",
    "                next_state = np.zeros(state.shape)\n",
    "                t = max_steps\n",
    "                \n",
    "                print('Episode: {}'.format(ep),\n",
    "                      'Total reward: {}'.format(total_reward),\n",
    "                      'Training loss: {:.4f}'.format(loss),\n",
    "                      'Explore P: {:.4f}'.format(explore_p))\n",
    "                rewards_list.append((ep, total_reward))\n",
    "                \n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                \n",
    "                # Start new episode\n",
    "                env.reset()\n",
    "                # Take one random step to get the pole and cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "            else:\n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                state = next_state\n",
    "                t += 1\n",
    "            \n",
    "            # Sample mini-batch from memory\n",
    "            batch = memory.sample(batch_size)\n",
    "            states = np.array([each[0] for each in batch])\n",
    "            actions = np.array([each[1] for each in batch])\n",
    "            rewards = np.array([each[2] for each in batch])\n",
    "            next_states = np.array([each[3] for each in batch])\n",
    "            \n",
    "            # Train network\n",
    "            target_Qs = sess.run(mainQN.output, feed_dict={mainQN.inputs_: next_states})\n",
    "            \n",
    "            # Set target_Qs to 0 for states where episode ends\n",
    "            episode_ends = (next_states == np.zeros(states[0].shape)).all(axis=1)\n",
    "            target_Qs[episode_ends] = (0, 0)\n",
    "            \n",
    "            targets = rewards + gamma * np.max(target_Qs, axis=1)\n",
    "\n",
    "            loss, _ = sess.run([mainQN.loss, mainQN.opt],\n",
    "                                feed_dict={mainQN.inputs_: states,\n",
    "                                           mainQN.targetQs_: targets,\n",
    "                                           mainQN.actions_: actions})\n",
    "        \n",
    "    saver.save(sess, \"checkpoints/cartpole.ckpt\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing training\n",
    "\n",
    "Below I'll plot the total rewards for each episode. I'm plotting the rolling average too, in blue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def running_mean(x, N):\n",
    "    cumsum = np.cumsum(np.insert(x, 0, 0)) \n",
    "    return (cumsum[N:] - cumsum[:-N]) / N "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x125c136d8>"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXmYHGd94P/5dvU5t+bQaDTS6LJkI2xjG2GccMRgCISQ\ndWAJmISEJGwMuyQ/SHiygYTN7maXzbEEkn2SsGsCC4TTwUDYBALGmwRY8CEb37ZsHZY0kkYazX30\n9FH9/v6oqu7qnj6qe7rn/H6eZx71vF311luj7vdb31uMMSiKoihKKaG1XoCiKIqyPlEBoSiKopRF\nBYSiKIpSFhUQiqIoSllUQCiKoihlUQGhKIqilEUFhKIoilIWFRCKoihKWVRAKIqiKGUJr/UCVkJ/\nf7/Zu3fvWi9DURRlQ/Hggw9eNsYM1DpuQwuIvXv3cvTo0bVehqIoyoZCRE4HOU5NTIqiKEpZVEAo\niqIoZVEBoSiKopRFBYSiKIpSlpYJCBHZLSL/JCJPisgTIvIed7xXRO4WkWfdf7f5zvmAiBwXkWMi\n8ppWrU1RFEWpTSs1iCzwPmPMYeAm4N0ichh4P3CPMeYgcI/7O+57twHPB14L/JWIWC1cn6IoilKF\nlgkIY8wFY8xD7us54ClgGLgV+LR72KeBn3Vf3wp80RiTMsacAo4DN7ZqfYqiKEp1ViUPQkT2AtcD\n9wGDxpgL7ltjwKD7ehi413faqDtWOtftwO0AIyMjrVmwEpiZmRk6OzsJhVb+rJHL5Zifn6erqys/\nZoxhenoa27aJxWKEw2ESiUTdc2ezWebm5ohGo7S3twOQTqfJZDJkMhlEhEQiQTabpa2tLX+ebdtM\nTEzQ3d1NJpMhmUwCICKk02mi0Wj+XxHBGMPJy/N899h40fUNcMOuTq4d6UNEaG9vJxQKEYvFAPLz\nJpNJEokEi4uL+XVbVrEi/dCZSbqiFlfs6CadTiMiRCIRLMsiHo9j2zbz8/PYto2I0NHRQTabJZvN\nEo1GyeVy5HI5zk0lufupi7DB2g5vS4S49YY9AHzjsQtcml1a4xWtDft39PDGFx9s6TVaLiBEpAO4\nC3ivMWZWRPLvGWOMiNT16TTG3AHcAXDkyJGN9cneZCwsLDA2NkYqlWL79u0rnm98fJzp6WnC4XB+\nk85kMly6dKnouCuvvLLuuc+ePUs6nS46/9SpU2WP9c9/7tw5kskkU1NTga/1t98/yb0nJ0F8gwaO\nne5gV8dVAExMTBRd68yZM4Hn/y9fdpJDf+vVhxjuSdDdFql6/NzcXNnxL/3gOb737OXida533G/8\ntdtjhK0QH7/7UWdgI91Dk3jhFTs3toAQkQiOcPicMeYr7vBFERkyxlwQkSHA+/afA3b7Tt/ljinr\nlFwuBzhPuc3Am8ebFxwNohl4wqHRNVXC0xo89u/fz9S9c3QOdvCN97wsP/7vPvYNZhdTDa3Bw7Is\nrrjiCsAREB+5+xkODnbwO6+9iu7ubmZmZgLPtXfvXp78zmV27e3mC7fftKJ1rSaf/95TfOybD4EV\nZcfIbs7mTvGHb7yGt96o1oRW0MooJgE+ATxljPmI762vA293X78d+Dvf+G0iEhORfcBB4P5WrU/Z\nuDRLaDQDv0bs/b6YztIWLTYLxcIhUlm7SdeEff3tXDfSw8XZVNl1BGF8LsVQT7wpa1ot4mHn75rK\n5kimnb9nIqKxLK2ilVFMLwF+EXiliDzs/rwO+CPg1SLyLPAq93eMMU8AdwJPAv8IvNsY05xvlNIS\n1mqjvnjx4qpdq5GNdzFtk1gmICxS2VyFM4KTyxmMgauHu9nT28ZsMsPj52d5bmKBP/rm0yymgn1l\nPEHWHt1Y5djiUWfLSmdtljLO3zMe0XSuVtGyT4cx5vtUtgzeUuGcDwEfatWalM1BKrUyU00zKSdA\nkmmbHV3FT+bxSGjFAsIYw5KrhcTCIfYPdADn+YdHz9PVneT4pXmOnp7k5YdqFul0BYS9TNNZ78Qs\nRxik7BzJjPO3iKsG0TJU9CoN08jT9UYjyD0upLJ89t7TZGxHACyUMTFFwxbpzMo1iIVUQUBcNdTF\nTfv7GJ9LOXYn4PFzs4HmsXOGVDa3TNNZ78Tc9aazOVIqIFqOCghlw7GefBAAX3/kPP98bJwfnHAi\nk5JlTEzxcIhszvD5+05zemKxoesYY/J291jEQkTY0RVjejHDPU87sR4TC8G0qyVXm9loGoQnDNLZ\nggahPojWsbEMkIqyytTSIC7Pp3jguUnA8Q9UMt3sG2inKxHm/z49zveOX+bP3nwdsQY2tnPTTr5E\ndyKCiPDqw4Ps7msj3rmNL3//SRZSwSLK8pvrBvNBxFwn9RPnZwnPOzkkqkG0DtUgFGUF/Nd/eIrZ\npLMpGxwhkczYtJVsvNfu6uEjb76OVz1vO5msyW/09fDf//Fp/s2nHwBgZ08CESEWsXjBrh5+4spB\nRvra8ht/LZZcTaRtg22uPYkIVkj4xuMX+OvvnyIk0NcRXetlbVo21uODolBsYvJet8ofUmveyYU0\n3hZrjMlv0JVMNy89PMJ3nrpEsk5/RMbO8fTYHId27eXmwQQ9iXDR2kSERMQimbYxxtRc96J7/fbY\nxhIQXYkIf/KvryUjYQaGhulOROjviK31sjYtKiCUDc0zzzxDLBZjrXqTh0TypSoMTogrVBYQ7Qkn\n63kp4JO+hyd43nD9MC/uXZ77ICIkohbZnCGbM0Qs571nL82TyeY4vLOrZD5HQGw0ExNAd1uERCLB\nyGDnWi9l06MmJmVdU84hXTq2lmGvllXYpHPGkEw75qZKG29HzBEQi+n6ss89wdMVL19WQ0Ty5iLv\nWIA//ubTfOTuZ8rM51x/ozmpldVFBYSyZWmGWSrsK1KYyuQKpptKGoRr0plezAS+xtMXZrn7SSc5\nsDMeJhJZLiQ8DQIo64fI2sVCNa9BbDAfhLK6qIBQNhzNCnNtRgVav4BIpm3+w9ceA6iYX9AWCdOV\nCPPI6HTga3z428/wL2512M54hD179iyrZGxZFu2xcH4d4OQ6eCyUaCzJGqaw9cpWyL1ZT6iAUBpm\nveUj1EszBIS/ltGpiQV+dNYpmHfljvL28VBIuGlfH6NTyaINPCjXDHdjWVbZkuddnR1AYfP3m7FK\nw18LvpKN54NQVg8VEMqGYz0JprAlXLG9g972CDOu2ehv3nEjQ93le1aICIPdcbK2YSYZ3MzkUUkz\nEZG8NuBpJ4vpQqTUfEmNpm89MVZ1PkUBFRDKFqYZgiaTNYQtIRoO5Tf8jlj1p3LP7h80kilsBTOr\nDHU5Qml8LrVs/oVUQRiJCNPuWjtrrFXZ2qiAUJQVkLYNkZAQjzjVWg3VBUQtZ3IpxjhhqwDvuaV6\nc5hYJMThnV3MLWXdtRU0iIUSDSKdzfGq5w0SCqlNX6mMCghlw2HbdsMNgPwE0SBqHZO1c4StEFGr\n8FXqiAfVIGony2VzBgy88YZhrtnVXfE4z3nbFQ8z52oLaV/1WL+TWkRIZ21iYf36K9XRT4iyIZmc\nnFyV69SKmsnYOSJWqGizba9htvH6FwQxMU0uOJt90M28IxZhzi39kfY1KJovcVKnsjkVEEpN1ACp\nNMxGDzkMokG0tbWRTCYJhUJFrVA9MraTtXzTVduJRyxev3tfVbu+VxID4L6Tk/x8jfX9xT89C9TW\nSry545EQKTuHMaamiSm6AQXERv/MbTRaJiBE5JPA64FLxpir3bEvAV5H+B5g2hhznYjsBZ4Cjrnv\n3WuMeVer1qZsHNZLxJJlWWUFRNo1MV27q4drd/Vw6NChmpvYtnanuNxSlRakJ8bn+eL9Z7kwvcTL\nDw1wZE9v1Tm9a0YsAQN2rmBiioVDy8Jc0/bGFBDK6tJKDeJTwF8An/EGjDFv8V6LyJ8C/i7rJ4wx\n17VwPcomYiVPkrOzsxhj6hI+lY7N2IaIL5+i1rpEhJAIVw11FvkISrnz6CinLi8A8KrnbccK6EwO\nu76QbC6Xn7+3I7pcQGRzRX4TRSlHK1uOftfVDJYhzrfozcArW3V9RanEhQsXgMqbeZBqqB5ZO0ck\nXL+wilohFtOV8yBmkgUnfD25ChF308/YOdJZR6hta4vy5PnZvL8ENq6JSVld1uoT8jLgojHmWd/Y\nPhF5WET+RURetkbrUpTAGGPI5ExRuY2gRMOhfIvScnibOxDoSd8TaGFX08jmDAvpLFZIGOh0ymFf\nmFkCnKKC2ZxRAaHUZK0+IW8FvuD7/QIw4pqYfgv4vIh0lTtRRG4XkaMicnR8fHwVlqpUYr34Bxpl\npSYmLz8hEjCRDfy+glBVE1PG9149G7lnYspkDfOpDJ3xMEf2bAMKJTi8wn1edzZFqcSqCwgRCQNv\nBL7kjRljUsaYCff1g8AJ4FC5840xdxhjjhhjjgwMDKzGkpUtwtxShi/cf6bqxu0n4260QTOd/Tga\nhHP+4mJxj2o7Z0j6HNjhOpLZPGGVzeWYXMgQautelpj3f93+1apBKLVYi0/Iq4CnjTGj3oCIDIiI\n5b7eDxwETq7B2pQtzJceGOWepy7x7SfHAh2fcTOnI1bwJ/G8BhEqaBBnz54tOmZuKet0Hyo5pxT/\nA1KpiWl2KcOT52fpjkdIRIqrvD55YRaAnzi08R6wNMx1dWmZgBCRLwA/BK4UkVEReYf71m0Um5cA\nXg48KiIPA18G3mWMWZ1MKEVxKfgElrc0LWdi+swPnwOCJ7H5iYalog9iPhWsiF97e/uyDTPimo1O\nTzhayY9f0Uci6qzP0yBS2Rw37uvliu0dda9b2Vq0MorprRXGf7nM2F3AXa1ai6JU4/J8ivff9Vg+\nGc0K8JT69UfO88joDB2xGNeP9NR9zagVws4Z0pnlneUyduO+HU+DODeVBODHr+gntORoDJ6AyGRz\nJDrU/6DURjOplS3PfScnAJh3i9yFggiIh88D8Lab9tTVlc174vfs/+fGLi47xnNQ/8JNexjuKV82\nvNK8njYzs+RVa42QSgtIYd6lrE2/lvlWAqBeKmXLU/rAXk48lJqYrh7uYqAzxi1XbW/oml4+wsLS\n8qKDGTc6aqQ3waHB+sxAnkN6xq3HFA2HEBFiVoiUKyDSWaOtRpVAqIBQtjylnd3SVfIT8sdkc/S0\nRRp2mkbcJ/1UmYJ9nm+ikfwKrxf2rJto52kUsUgof19p2yauGoQSABUQyrpmNXItMiVhrakAZbgz\ndmOJZnkTky/jefnczlgj+RVeC9HZpJMk5+VFRKwQ/3JsnMWUTSqTo001CCUAKiCUDUkzwx1LN+lU\nmSJ6pYLKK1vRsAbhbv7+jOnC3F4CXv1fTysk+bn9+RMT845G8aWjZ0jZuQ3balTDXFcXFRDKlidT\nUqU1XaXKauGYxordFZzJVsVrFTSI2vP7BZf32iutkSqT8JfK5jAG4qpBKAFQAaFsedJZU1Qt1b+x\nVq7imqvLBFRK3gdRZhOvx8TU1bW8Is21u5aH3bbFHIHgOafbNqgGoawuKiCUDcPTY7N88GuPMT6X\nauq8GTtXtGEupKtrEMYYphYzKyqX7Z2bKuODyNZhYtqxYwdXXHFFkenl0GAHSRNmNFdoUfq+Vztt\nWDyzk0YxKUFQAaFsGI5dmGNsJsUPTkw0dd6MnctH/wBMLVQXQM9ecvo0NGIOL2rsA2XrPuWjmAJo\nECKCVVLq49pdPRgE4wvYHel18ik84bdRfRDK6qKJcsqGQdynX3+vhGaQsXO0xcKAIxguz1Wf37v+\nyw42XsvIi4CqJCCskARK2KvEG28YZi7Umf9dRLBCwmLayY9QDUIJggoIZcPgbaYzi8FqFQWf1xQ9\nUU8uVtcgPvPD0wB0xBvPg8ibmMoKCNNQBJOfX33pfoaGhpiens6PhUOS702tGoQSBDUxKRuGJTep\nLEgiWz1kcsUb8nyqsg/CGJOvitoeterO08ibmGpoECtxgFciYkm+jelG1SA0zHV1UQGhbBjypSJc\nJ+7kZHMK/mazNlFLePORXfS1R0mXyW72mPf1dl7JJh7Nt/4sH+baLAHhF2B+IXhwe2e5wxWlCBUQ\nyrrGv8F5Gc5e5vPU1FTdc5QjbeeIhEP85PN3cNVQZ15TKceUa97qSoQbepr1zrFCQigkZaOYSk1M\nQa9T6z69rOqh7jjdbZGgS1a2MCoglA1DyvZMTLUT2erBvyFHrVBVAeGZl25/2f4VXzcSkgqZ1LkV\n+yDK4U0Z26DmJWX1UQGhbBgKGkRz6zP5s6KjEYulKi1HPeHRaLE7vzYQi4SWlfW4PJ/iR2emG4uh\nrcGObifUdXKhuVFgyuZFBYSyYfB8EJlc4wJibHaJD3/rWJGWkMkZImG3iF5IWMrYFTvJeU13mlGq\nwt921ONTP3gOgNHJxTJnrIxbX7ATKJQCV5RatLLl6CdF5JKIPO4b+08ick5EHnZ/Xud77wMiclxE\njonIa1q1LmXjkvJ1RGs0muWuB0d5emyOJ847XdbG51LkcoaoW1o7XKUEBhRMTM2ohhoJLzcxefN7\nAguaF7mjfgelXlqZB/Ep4C+Az5SMf9QY82H/gIgcxulV/XxgJ/AdETlkjGmusVnZ0BSimOoLc/Vr\nAVLSDugDX3kMKISdxrz8hEyurJawUg3Cv9lHLWtZFNPUYpqDgx386kv2NTR/6XX8994R2/hpTxrm\nurq0TIMwxnwXCBqHeCvwRWNMyhhzCjgO3NiqtSkbE09A2DnTcLKct78YU9woyPNBeIIiWcFRXSjF\nvfKNKhIOFUUxGWOYTWa5arAzX5G1mXiZ2dfvrr+HtrI1WQsfxG+IyKOuCWqbOzYMnPUdM+qOKQrg\nbJ6prM2evjYATjdqo3f39ZwxZH2bsxc15HVgqxTJlLFzhC1p+Em2yElthYo6ymVdgRUpaUTUzKfm\nj73tBn7zVQebNp+yuVltAfExYD9wHXAB+NN6JxCR20XkqIgcHR8fb/b6lHVK0u1jMNQdB2Am2aAG\n4f6bsXNFzu54wpk3YoUQYKlMAttj52b44YmJpoWglvogvOgs//x9fX3s3r070HxBsrojVgirBSG0\nyuZkVT8pxpiLxhjbGJMDPk7BjHQO8H8Ldrlj5ea4wxhzxBhzZGCg8WJpysZifNapjzTY5Wzkdi64\nH6LcxpnK5vJltQE6Op3S2J6pKVmm5Peff+dZZpKZpmU5R6xQUU6H17go7NvA+/v7icWab25SlCCs\nqoAQkSHfr28AvAinrwO3iUhMRPYBB4H7V3Ntyvrm8rwjIHZ0ewKisVBXzw4/n8qS9QmZeKTYB7FU\npS91JNT416bYSR0q1iAa6EWtKK2kZWENIvIF4GagX0RGgf8I3Cwi1wEGeA54J4Ax5gkRuRN4EsgC\n79YIJsXPx/75BACdcecjm7EbExA5V5u4NLtU1IvaaQGaIWoJYJb5IPxaSJA+DdXo7u5mZmbGcVL7\nTFnNdIB71FtMUFH8tExAGGPeWmb4E1WO/xDwoVatR9mYlG5wbVHnI9uoBuFtwuNzqSIh4/VniJY4\nqb3rZ33Xa5YPwtEgCkLKc5qvREMJgoaKKkFRb5WyofA6v6XTS2Sz9WcEe1qD44PwaRBuXkPUsso6\nqf0CYiWNfPybc8SSIjNXwcSkX0tlfaCfRGVD4SWopZeWAp/j10KKBURh3Eu+i7m+iNI8C7/GcqZJ\nZTDCVoisXTB7eetZqQlLUZqFCghl3VPO/t+oicnbhNPZQphrOCRcMdABQFc8TEc8zPHx+bLnldLR\n0dHQOsAxJYXFzs/trUc1CGW9oJ9EZd3jlwWWOB/ZesJc/Xib8Ewyw9SiU9X0A697Xl4zERF2b0tw\nbipZdJ7fFHTzlYXw6kgkwhVXXNHQWiKWECHnE1puHSYVEMo6YeMXZ1FWjYWFBeLxOJa1uv0Ecj4N\nwts7q1TkrkrWl3fgVU6NlmQut4fNsmJ9ng/itht386rnDTZ2cRdPI/LyHRyzl5WvstqVaKyonkYs\nKc1GH1WUQNi2zejoKOfPn1/9a7ub87++YVfeyXtxNrgPwk9RLwn3ZTQcKnIed+YWlwmInLuGnkS0\noeuWwwtn9RLkZhbTiDhmrmahQkNZCSoglEB4G006vfrNZjzrjj8/4P5TjfWjLtdLIlaiQUQsWdbI\nxwuJbab1J5LXIFyz11KGzngEK6ROamV9oAJCWffYxpEQzdg4s3aO60ecaqZhd75SARG2ljfyscuU\nwVgp3vX9kVWla1GUtUQ/jcqaYtt2zXwGT4MINUFAZOwcg11xrJCQzRlEChu1h6NBOBctJMo571lN\nSDLz5vTKengCIq0CQlln6KdRWVOOHz/OiRMnqh5juxtqqXwIal/3tw/N2IaIFcoX5YuFLUSkJIEt\nlO9/nV+Dp0GUEVKNZiY75T0KdZ/Sdm5VIpg0k1oJigoIZd3jRTFZbgmKq4Y6geLs5iB4x4ctybf0\nLPfEHi7jg/DOtZqYxNYZdwTEfMpJyktnc00v1BePx5eNqYBQgqICQlnXGGPwKmJ4D+/XDjuluesV\nEIVENMnXO2qPLQ/ZjVihZa1AvbIc4SbWSWp3W4AupOz8NUpDbv1Eo/VHUJVL5FMBoQSlYjydiPyI\nfCDgcowxN7RkRYpSQs51Uns1kDxHcdbOQR29obPZQjE8T4NoL9OnOWKF6LWni8Y8n4RXimMleCav\ndrfw4NxSNn+Nnrby8x84cIBQDeFU1HtbhYDSBKoFXL/J/fddgAX8jfv7LwBailtZNbw8CC+KyfMD\nNK5BhIi4yX5t0TIaREjAFBfz8/wEiXBjSYKJRIK5ubmijdsKCbGw03bUzhnOTy+xvWu5SQggHG5e\nboQKDyUoFT91xpgTACJyS4m28CMReQj4nVYvTlGAZSYmT4N4dHSGnzgUvKtgvlpqWNy+D4Xy4UVO\natfM40+W88p/x6ONaRA7duygr69vWRZ62BIyuRyPjjoay8Nnpsud3lRUQChBCfJpt0TkJu8XEXkx\njkahKCsiaBSS56QOleQN/M0PT9d1PX+/hYW0Y9bZvS2x7DhPQykVECKFlqR+gmy4oVCobOtQKxTC\nzhlG3dpP737FgQB3sjJUQChBCaK3vgP4lIh4um8S+NXWLUlRCpw/f74gINyNbTaZqXZKRbzs67Al\nzCw6AmJ4W9uy47xQU38kUzJjE49YTdlc/YIxEhIytuHB01Ps6Wvj+pFtK56/FioglKBUFRAiYgF7\njDFXi0gfgDFmIsjEIvJJ4PXAJWPM1e7Yfwd+BkgDJ4BfMcZMi8he4CngmHv6vcaYd9V/O8pmxC5J\nUtvb3w5Ad8Cidt6G/I3HxgBYTNv53Ip4GadzXkBkcnjP/Mm0TaIJDurl1xKydo65pSzX7urOjw8M\nDNDZ2dn064EKCCU4VT/xbl/o33VfTwQVDi6fAl5bMnY3cLUx5lrgGeADvvdOGGOuc39UOCh5cm6S\nmmdiuma4mx1dMXb3LjcPVeM6t8TGNcPdea3Ey4MoTZQDJ3HNGIMxhgdOT9IRa6zKajWmFjM88NwU\ns0sZEj6HeTweJxJp/vUUpR6CPBJ9W0TeKyJDItLl/dQ6yRjzXWCyZOzbxhivrsK9wK76l6xsNfz5\nCx497dF8ZFFQ2qIWfe1R2mPhfI+JeJkwWe86Xjb14+dnyWRN0zrJ+SmU9ICEby21QlpXgmoQSlCC\n+CDe5v77Pt+YAUZWeO1fBb7k+32fiDwMzAAfNMZ8b4XzK+ucoE7qcklqiYjFxWR9Jb/tnMmHynrl\nu8tlUhf5IKKQabT5RAUq3XciapFIJOju7i6bAd0sVrufh7JxqSkgjDG7m31REfk9IAt8zh26AIwY\nYyZE5IXA10Tk+caY2TLn3g7cDjAyslIZpWwEPB+Evw7SQGeMx8/NkDMm77y2bZvTp08zPDxcNmIo\nmzPL+j2X0yDCli/MNVrIFn3bTXuacDeVaYuGsSyL7u7u2geXIajA7evra2h+ZesRSI8VkatE5I0i\n8vPeT6MXFJFfxnFe/4JxP9HGmJTn3zDGPIjjwD5U7nxjzB3GmCPGmCMDA8Fj4JWNS7ZMqe3BrjgZ\n2zC9WIhomp+fJ5PJMDlZvldE1jZ5080r3LahntAp9kE4r73ch0U3JPbq4fKW1WaZbFrhBC9HK81X\nyuaipgYhIh8EfhK4CvgW8Brg+8Dn672YiLwW+PfATxhjFn3jA8CkMcYWkf3AQeBkvfMrmxOvZ7P/\n6d+z15d2fquGncvh7cE//+IR3vKikbKbe197lFBIOPrUKZ530wif/oGTb1GtTlI9VDYxaQdgZX0R\n5BP/FuAVwAVjzC8CLwDaa50kIl8AfghcKSKjIvIO4C+ATuBuEXlYRP6ne/jLgUddH8SXgXcZYxpr\nGaZsOjwntd/ElC+3YdcjIEy+IqyILDM3eXQlIvS1R7k8OVU0Hi8ps9Hb20t7e82vQmASddSVUpTV\nIMgjS9J9ss+KSCcwBtQ0xhpj3lpm+BMVjr0LuCvAWpRNRFCbue3LgPbwHMnV6jFNT09jWVa+Cmo2\nZ8r2cyhHLBxapp2UluJuhonzj990Lb/z5UcBJ8pKI4yU9UQQAfEjEekBPgkcBWaB+1u6KkXxUejF\nUBiz8mU3KguIixcvArB3715nHtsQjZVXmks35ogVIl2inVTbvHt7e8uW1q5FX3uUHd0xxmZStMfV\nxFSLeDxOT08P27a1PuNcCRbF9E735V+KyLeALmPMQ61dlqIUyJQJcy0q+V1CpY3crlODSGedRLl4\nxOKlV/RXPb4ebaJUc/oPP32Y05NJ2iJWPilQKY+IMDg4uNbL2DIEcVL/b+C7wPeMMcdbvyRFKcbO\nGZDilqOeuSdbx4aazeUCN/yJhkPMuDWfsnYu3z+iFcQiFocGHe1jYWGhZddRlHoJ8m35PLAP+LiI\nnBCRL4nIu1u8LkXJk80ZIqHivtEFJ/VyE1Ml34Y/Ua4W0XAoX2rD8V1oaKiy9aj5qTfG3A38R+C3\ngTuAHwN+s8XrUtYZQR3KrSBrL3/yD/vqJd13aiJQNFPWDm5iiloh0plcUR/r9c5a/h8pm5MgJqZv\nAd3AA8D3gJuMMedbvTBl8xN0Q8uUyYD2TEz/fGyc45fmkZ5h/tXzeqrOUy6TuhKeBpFvMuSe19HR\nwfz8fKD6CZMEAAAgAElEQVQ5KtHf38/o6OiK5qiFRkMpzSCI3vwMTlmMgzjZzVeISP3d0xWlQexc\nbtnG7mkU56adRjv+HhGVndS5+kxM2Vy+DpMXYjs8PFzf4svQ3t7O9u3bVzyPorSaIFFMvwEgIt3A\nL+H0pt4O1FdrWVHqxIvocUxDhQQ3YwqaQDLtlMMo1+mtFDsHlgTzJcSsECk7VzZJT1G2CkFMTO8C\nXga8CDgPfAbH1KQogVhcXCQWi9WsIlpqcvLyGMqZhko37Nmlyl3mvHmzZTSRSkTCITCwkHLqMIUD\nCKB6UBOQshEIkpnTA/wV8IAxJt3i9SibDGMMZ8+eJZFIBK6+623o6bTzccvYxSUyjFkeVeQv2leJ\napnUpRu2V3dpJumsoVxZ8I1Ce3s7i4uL6sRW6iZIFNMfATZwG4CI9IqI1tnehCSTyabP6W1KS0v1\n9W7wn+sU2SvewEsf6L0n/Urz5IzBGJb5ICo9ycfcC0y7vavLlQVfCaupQezatYtDh8oWR1aUqtQU\nEG411/8IfNAdStBAJVdlfTM9Pc2ZM2fqitBp9RNp3jRkLzcxlRbbm6siIMBNtoNlmkeljToaKdEg\nVqkUt6KsJ4J86t8EvA5YADDGnANqthxVNhZ5c06mtqmmWdQV5lrGNOQv3je/VF2DyJSp5wRVBERe\ng3D+LqWVXNcjakJSmk0QAZFyG/sYABFpa+2SFMXBb2IqF57q1yAW0jU0iDIVYavhmZQuz7kCoska\nRLmOdyuls7Oz6XMqW5sgn/qviMhfAt0i8ivAt4H/3dplKeuNtXg69ZuYImWiiOZ8WkM1DSKTyeBV\n7g7qg9jR7URxH780Bzj1kppJPB7PlyFvFgMDA9pOVGkqQfIg/lhEfgpI4zQL+pAx5pstX5mypSgn\ngIrCU8u0BvVTzQcxNjaGXaZtabX5ehJhouEQ52eSxAiWZ1EvzW79KSI1Q4kVpR4CFaB3BcI3AcTh\nLcaYL7V0ZcqWxxMQGbt2sbxqUUzg6ylRMk0lASEitMcs5lzVI2gGdj2oz0BZ71T81olIh4j8toj8\nmYi80hUM7wJO4GRUV0VEPikil0Tkcd9Yr4jcLSLPuv9u8733ARE5LiLHROQ1K70xZX3gbYLVNIRa\n59oBaigtpu18pFI5PAFSqa1nubW0R8MILCs13ixUQCjrnWqPZZ/FMSk9C7wb+A7wNuDNxpifDjD3\np4DXloy9H7jHGHMQuMf9HRE5jJNn8Xz3nL8SEdWVNwitiukv+CBygUpdLLplN8pxZmIRgN29xTEW\n1dbeHnM+guGSUuPNQgWEst6pZmI6YIy5BkBE/idOL+oRY0ygbCpjzHdFZG/J8K3Aze7rTwP/DPyO\nO/5FY0wKOCUix4EbgR8GugtlTWnGRldtjmwuWKmLxSqRTLOuE7snESka9zb+ctfviEWAZMt6QZS7\nptceVVHWA9U++fmAeGOMDZwNKhyqMGiMueC+HgO83oHDwFnfcaPumKKQtcuHub7zJ/bnXwuGf3x8\nrOIcGbcrXKkmUE1AXDXkpPssZSprJiuh3DXD4ZX1pVatRGkm1QTEC0Rk0v2ZAq71XovI5Eov7M+t\nqAcRuV1EjorI0fHx8ZUuQylhvW0wXke3cmGuIXdzP7yzCzD5FqHlSGXtspFIkYijUYgIIyMj9Pb2\n5t8b6o4h9X9EA7Pe/taKUkq1x5VW9Hy4KCJDxpgLIjIEXHLHzwG7fcftcseWYYy5A6ezHUeOHNFv\n2CbEv3F6jeLK1crzlIpoOMSu7hjTycq1JDN2Ll+Az8/g4CAdHR3E43EAEokEk5PO809PW5QIuZaJ\nCP99ekUIV+rr8MJcNdxVaQYVNQhjjF3tp8HrfR14u/v67cDf+cZvE5GYiOzDaU50f4PXUBpgLcpP\n+zfI0qdpr7hftkL+AhSynTtiFtvaw8xUqeiazpp8AT4/oVBoWQayl5/QXeKvaCXN0ia6urrYsWNH\nkSakKI3SsgpkIvIFHCfzlSIyKiLvAP4IeLWIPAu8yv0dY8wTwJ3Ak8A/Au9egRBSVokgm1qjG9/0\n9DRQyF8oF8V01Y5OfuHFI7zlRSP0JCLMJCs7qVPZHOE66ynFXY3D0NooLT8rFdQiQnd3t/abUJrC\nyjxiVTDGvLXCW7dUOP5DwIdatR5lZTS60U9NTa3oulnbFRBlnv5FhFdc5bTujIdDTGcrP1Nksjms\nSH1WU91kla2O1jBWWopnz6/F6Oho2fG8ialGHkQ07PgZKpG2c8QarMjaSke1oqxnKmoQbuRSuW+G\n4AQhqZFTaRpLS0u0tS0vFJw3MdXIpI6GQtUFRDZHW2cHnZ2dzM3NrWyxLaSVWsvQ0JA6r5W6qGZi\n6l+1VShbilQqxfT0NNls9R4O4ORAwPJGP6VErBDpbOUn/bRtsy0aZmBgoC4B8dG3vACRzaFod3Vp\nGxelPioKiFInsYj0AnHf0PlWLUrZWNT71Ds6OlpTOMzOzgLVndR+opbkhYk3h59M1jTU9KczHiEU\nCpHLVdZOmsGePXtYWFho6TUUpV5qOqlF5KeBj+LkJkzgZDg/A1zV2qUpa8nU1BTxeJxEItHUebPZ\nbFXhADAzM5N/nW8VWqPURiRc3cSUsnPEI8EEWS2H/MDAQNN7OcTj8XwuhqKsF4Lozh8CXgIcM8bs\nBl4DfK+lq1LWnEuXLnHmzJmmz3vx4sW6jvc2/VrltqOWkK7hg4hHwk2x8ff29tLR0bHieRRlvRNE\nQGSNMeNASETEGHM3TiE9RambIOGy/k0844a5Rmo4qSNWiGzOVCwrnrZzJMqlYwdAw12VrUqQPIgZ\nEekAvg98RkQuASst2qcoyyi3uedNTDWc1F6dpWyZnhDZnAEDsej6iuDZsWMHY2OVCwwqyloT5JHq\nZ3EEwntxynOfA17fwjUpWxz/E3s+Ua6GicnTMDK55Q2K0m5XuEbzIFpFd3f3Wi9BUaoSREB8wK2/\nlDHGfMIY8xHgt1q9MGX900h2dRBzTZGAqFKLyY8nILJl/BCebyKxzjQIRVnvBBEQpV3hAIJ0lFM2\nIJU2/WrCoJVlqwsaRPXj4u7m72kLfrz8iEbCXFtNd3e3Ri8p65ZqmdTvBN4FHBKRh3xvdQIPtnph\nytZhKWNzcTbFVbuckNpiDSJYmGuHKyCSZdqOpt0aTfFIaN05nHfs2LHWS1CUilRzUt+J0zf6D3F7\nR7vMGWMulT9FUerns/ee5t6Tk/zZL9xIR8knspBJXX1jb485pbk9AVHkg3DniEXWnwahKOuZav0g\npowxx40xP4eTQf1q92dgtRanbH6+9cQY9550Cvo9e3F5CYyUJyBqhLm2expEmfagGdfslFABoSh1\nUdMHISLvBv4WGHF/7hSRf9fqhSkbh5WYbf72aKGKa8rdyP1P/89cnGOwK1a2XaifjlhlAeEJmVjY\nyjcDCsLIyEjgYxVlMxIkD+KdwI3GmHkAEflvwA+Av2rlwpTVZT3Y5lO+CKRTlxcYm1lifinL9s5Y\nzfW1RZ2P8lKmnJPai2JyfBCHDh3imWeeqbke75rr4W+jKGtBEAEhgL/Zb8YdU5SqlCtwV22zTbtP\n/4uLi3zoH54CYGdPnIivRHWliCmvFIedW66FpEryIIJu+CoYlK1OtSimsDEmC/wNcJ+I3OW+9Qbg\n041eUESuBL7kG9oP/D7QA/waMO6O/64x5huNXkdpDfWEtNYb/upt5H7BkrUNkXCA3AlsBLDLXPPk\n+DzxiMX2rlhd61EBoWx1qmkQ9wM3GGP+RET+GXipO/4uY8wDjV7QGHMMuA5ARCyczOyvAr8CfNQY\n8+FG51bWF/WWyPbCUedThWqvmVyuZgQTwKzb2tQu03X04uwSu3sTgTOpOzo61nVTIUVZLaoJiPy3\n0hhzP47AaDa3ACeMMaf1aW1942kD9fw/BREQVkjy9ZZSaUcwvPeLD+ffz9iGSA0HtTcPFDKv/Syk\nbPo7g2sPQ0NDbN++vaUJgIqyEagmIAZEpGJJDbfkxkq5DfiC7/ffEJFfAo4C7zPGrKzjvdJ06tk0\ngwiI7kSYUEi4PJfmnicv8DPXDha9v5SxiwREW1sb8/Pz+d9FBGMMIYFtkiSX61l2jflUln397UXC\n7cCBAxWFnYgQDofJZDI1168om5lqj2YW0IGTOV3uZ0WISBT4VzghtAAfw/FHXAdcAP60wnm3i8hR\nETk6Pj5e7hBlnRBEmGRzhsNDhaJ1/3KsOAcza5uiHIihoaGi970+1iKCFRKyprhYX8bOMbeUpT1W\n/CwUDodr9mdWrVbZ6lTTIC4YY/6ghdf+KeAhY8xFAO9fABH5OPD35U4yxtwB3AFw5MgRtQG0gNKN\n/fz583mbvH/TrCUAgkQxZW1DxBJ+9rqdfO3h80wsOE/t1wx38dg5p+3ozGLhSd6fx9DT00N/fz/H\njx8His1VHk9fmMPOGQ4MtFddq6Ioy6mmQbT68emt+MxLIuJ/NHwD8HiLr68ExO+wbbaJKZPLEbZC\n3LivD4DvPuNohV2JSP6YiYVU2XPb2tqKBE45AeE5vId72gKvW1EUh2oaxC2tuqiItOOU7Xinb/hP\nROQ6wADPlbynbEACmZhsQzgkxKPFzyp+AZHKlX+O6ezsLLpGOQGx5OZWlM4fBHVSK1udigLCGDPZ\nqosaYxaAvpKxX2zV9ZTm0cwoJjtnMAYiIVlWJ6kzXvho/trLDwS6niUFAeFt7l7pjUbqMHn3GolE\nyGazNY5WlM1HY016lS1LM01MXmJcNBIqynVoj1n0tzthqeGQMNIbzDwUrqBBWCEJlEuxbL5wmJ07\ndzI8PFz3uYqyGVABobQMT0B858mLfOH+M8uEi78Ehl8z+S+3Xs229ijg2Bur4T8vFJJ8JrVfg1hJ\nH4jOzs6a0U6KslkJUotJ2WI0y/buzfPFB84C8GszS0VPJCk3czpW0i4ubIXobXMERKlGUA0rtPz4\nZNpuSqvRnTt3EovVV6pDUTY6qkEoq8bYzFLRk/zfHnUER6mAiFhCV8J5dnn14eLEuWqEQ6FlAmJ2\nKUun20xoJXkNnZ2dRKPRhs9XlI2IahBKEc2M3Cmd6+LsEsMDhU36kbMzAMtqJIVDgojwP9/2QqxQ\n8DVZISFjF1dznU9l6Y7rx1xRGkE1CKVleJt0xM2EPnvu/LJjUlj5UFQP70k/bEldT/0ZO8fj52b5\n3rOXAcgZw9RCms54pMaZiqKUQwWEArS2rETYraV0eXK67Pt7+pqTxDaTdDKu7/juCYwxnJ5YZG4p\ny4HtHU2ZX1G2Gqp7Kw1TzfRjjGFmZgZjTL6j28R8cUZ0RzzMzfu309fRHOdv1nbW49VuWkg7msmu\nbYmmzK8oWw3VIDYhExMTpFLly1MEOXclfohkMsn09DQLCwuAU4zPcxw/MjqD7WsrupSxSUSb94zS\n6Tq2w+J8rJdcAREP2AdCUZRiVEBsMowxXL58mTNnzjQ8x9LSUl3Hj80scWnWOefMmTNcvJivu5jP\ndRjojJHO5phzayPZOUPWNsTDzfsIvu/VVwKQNTmMMYUyGxH9mCtKI+g3Z5OyEi0gqD/CGMOpywu8\n49NH+bn/9cOyc3ibdH+HEyI66/oJPMER92kQ14300NZAzkJnp1N9fqAzxk8+f5DJ+Qy5nF9AqAah\nKI2gPghlGUEEhG3bTE1N8aF/eAqIcnpisewcqYwjCPpcATG3lKG7Q/JJcv6n+19/xRUNrXfnzp2c\nOnWKdDrNUFectD3F+emkCghFWSGqQWxR0ul0UWe2ehkbG8MuaQDtOaP9LLmCoM+trTS3lHH/dUxN\nzd68B7viAJyeXGApmyNiSb4dqVZnVZT6UAGxyQi6CT733HOcO3eu4Tls22bJFQjiVkyaWkzn3y/V\nIHrbPQ0iizGGP/g/TwKFXtLNorcjigCXZpfcOkyqPShKo6iAWKcYY4oa9TRy/kreD8Jc0tECXnHV\ndgAeOr28hbhXbttvYjp5eSH//lU76u9eG4lE2LZtW9GYJ5B6Es51Ls4ukVIBoSgrQgVEE1hcXFxm\nbqlELpfLh4BWY3x8nPPnz7O4uFjz2GYTRHiIFPwIN+7txQoJj5+fWXbcsxfniVjCvr52QiFhNpkh\n4zNF9bbXnwOxf/9+tm/fXva9sCVsa4tw14OjjM+liPkEhJqYFKU+VEA0gbNnz3L27NlAx46NjTE6\nOko6na56XCbj2OqDtO1cC0QKpbUjltDfEWV8bnnuxcRCiu2dcWIRi/aoxdxSNm+aCkqptlCLkJ1i\nYTHJifEFEhriqigNsybfHhF5TkQeE5GHReSoO9YrIneLyLPuv/XtCmtM0MQ0TzC06mm2dN5sNltT\nGNWaoxJeApwVErZ3xrlURkAsprO0xZyn+PaYxXxyKd8nentnMO2hp6en5jH+yKsrBtroFGct/jwL\n1SAUpT7W8vHqFcaY64wxR9zf3w/cY4w5CNzj/r7uWe+bzokTJzh16lRL5vaSosMhobc9ymwZn8lC\nOpfPbeiIRVhMpph3I5h+/2cOt2Rdv/hje7lhxHm+SCX6GBgYABzfhaIowVlP+vetwKfd158GfnYN\n17LqZDKZdWtO8phJZph2I5VEJL/eUEjoSkRILhU0lXxHt1SWNjcZrj1mMbOUYXoxQywcapkDua89\nyjXDXQBkc9Db28uhQ4cIhzXtR1HqYa2+MQb4jojYwP8yxtwBDBpjLrjvjwFlO8WIyO3A7QAjIyOr\nsdaWUKp5nDx5kmg0yr59+9ZoRQUqaUXvu/MRbIRvHjwIgFsbD8nZdMYsFlzTEZB32i+m7bwGMZvM\ncmF6CQG2tTX3ab40uS/uXtO7l1ZWq1WUzcpaaRAvNcZcB/wU8G4Rebn/TeN8q8vuUsaYO4wxR4wx\nRzzTwWahXl8BOBvxSjSPoOGwXsQSwH/++hOOk9q9rrEzdGRn86YjgPPnz5MzhmTGzmsQV+90nurP\nTy+x3U1oaxWFAn0qGBSlUdZEQBhjzrn/XgK+CtwIXBSRIQD330trsbZ6qdcH0ewn2ePHjxcV5mvU\nJ+IP0/XPMTY2BkDGLow9NTbnCghnLCRCPJQlk8vlO7qB0w86h9DuPs1fu7vgbL5+pLbjuR6SyWTx\ngPtnXt8eIkVZ36y6gBCRdhHp9F4DPwk8DnwdeLt72NuBv1vttW1UGi3t7ZFMJpmaWp7kBuST9Wyf\ngIi6kUFe+2crBAnXn5DyhbAuugKiLeZoEF5nOYBtbv5Dq0w/3rxGRYSiNMxa+CAGga+6X+Aw8Hlj\nzD+KyAPAnSLyDuA08OY1WNuWpFTAlNNCsj4z1rxbT8kbs0IhrJAgFEJfwREQQN4HEbUKTumuOvtE\ni0hd2tGVgx1cM9zFe17fmkgpRdkKrLqAMMacBF5QZnwCuGW117PVKbfpPnZuhvufvMBPXzOUH/Ms\nR1FLmE9l3SgmZywcEiz3id0vIDynteeDiPhyEtpj9X30Dhw4UJevJWKFeM+rDnFgoP5SHoqiOKyn\nMNcNyXrLg2hkPaVmnnf+zVG++lBxIT9PW+iIRfKVWAs+CAiFPLNTOQ3CeS/qMzHF6mwUZFlWQ3kM\nGr2kKI2jAmIVmJyc5NixYy0VJtlslmPHjq2owF8yY/PtJ8boFcfh69/ss64wGOqJs5i2uTSXypfa\nsEIhwpYgGHw+ahbTjiDxtIWoTyjUKyAURVl99Fu6CngO4KAF/RrB8yPMzBQK5tUrkI4+N8mdR0fz\nv/sjkjxt4fluqOqDZ2bI5RwXcCjkCAnnuGIntaHggwj7SnuHm1zme/fu3WXHPc1GUZT60W/PFscv\nRGZ9eQwApy8XKsl6AmJvXzs9kuST9zzGxILjrA5LKN/Xwa91zKWyWCJELedj5jf3NNv0k0gkyo6r\niUlRGkcFxApphtlovfgx5ksExJ9861j+teeDCFuSL4Q3Ou0IECvk+AiAIhPTE+dmODjYuSqbtIhs\n6Mx6RVmPqIBYI4wxJJNJjh8/3tQaTCsRNnNLWfrao/zKS/Yue89Lb7BCId7yIseck/Oc1CEhFMIN\nc3WS5f7wm08zOpXkwPbViyKqpEUoitIYWr1sFSndvCcmJrBte3kWcIuuV4vZpQxdiXDeIe3HdlWD\nSEjY19/uHp8lagkRK0Q4VEieG51KcuKS0+96x45BQqFQXgjeet1OLs+vLLFPUZTVQQXEGtJM01I+\nc7jOOY0xjI+PAzCbzNDXEStyTnt4PgghR9gNV51ZTOcjlCzxopgMYzNL+fP29LWzbVuciYkJAH7m\nBTvLrltRlPXHljcxTU5O5qOMjDGMjo7WVbqing3Zf6wxZt1UGvWiq2aSGXraImR9ZTU8wfCjM9OA\nY2KKuNrC3FKWDldAmFyWuGSxc4bRqYJG9KK9vVWv3dfX17wbAfbs2ZN/vXPnzipHKopSiy0vIMbH\nx7l0yakLuLS0xMLCQr5AXbNYKwHgF0gXLlyoWi3WzhnmUlm64hFefqhQJdfLZfD+3d4ZI2wVPjZe\nCGtIIEwO2xguzS2xu7eNj//SC+mIR6ref+l7/f39ddzhcuLxQpXYjo6OFc2lKFudLS8gPLLZbO2D\nVogxpmhDXKmJqdz5leacnZ3l4sWLFeeaW8qCge5EhLaoxb952b7CODCftnneUCdWSPImJoAd3c6G\nHPZlUs+5voxGBGOzNQpFURpHBYSLp0WAo0lUqm5aSqObfKm5aTUpd+2ZpJPT0J1wTEZ7XUf0B7/2\nODljWEhl8v4Gf1XW/QPOU7qX92bbhtlklq7Y8rIYq93yc61Nd4qy0VEB4cO/oVy6dIl0Op13rjaD\nUkHQCsEQRPDMz88vG5t1K7R2JZxNfIevoc/EfJrFdC5fdM9vYtrvChIvUS6ZsZlZytDpVmttZXKc\noiitRQWES7nN69y5c1y+fJlMJtOSa3obeC1BkcvluHz5ctMETLnzMm6ig79G0i//+F4AphbSLKSy\ndMQdf0PEVyZjoNPp6xByx/7+0QtkbcO1u7uXXUMFhKJsLFRAuFTbvFb6pF8rBLXa/MYYJiYmmJiY\nYHZ2dkXrqHbtTD5TuvCR2N3bBsDJywvYOcOOLicRzV9HKeIe75X7Hp9L0d8Z5aodXfljvPtvZkKg\noiitRwWED3+hOwiWW7CSp/h6NIig12p0PZmsc17EJyA8c9MT552/y/bOyl3g/JVaL88VoqVEJB82\n3CpNTFGU1qCJci7GmIoCotKTrzGG6enpuq7hf+1tmLU29SCaRzlhU49JykuO8/ds6HEd1k9dcEqI\ndycKTuaPvOUFRfkSQTvERaPRquG2AO3t7ViW1RKNSVGU4KxFT+rdIvJPIvKkiDwhIu9xx/+TiJwT\nkYfdn9et9trKrBWoLCDm5ubqEhB+/FFSQZ/667XhLy4u1j7IxRMQfg2i9HodPiHQFY/Q2x6te237\n9u2recyuXbvYvn17oPkURWkda2FiygLvM8YcBm4C3i0iXuPgjxpjrnN/vtHKRSwuLhaZPMpt0ivx\nHdQ63p930QoTFlCU8LeYKd+Lwps/nV0uIADeffOB/Ot4mSY//ryF33jlFQC89uodZa9RD+rQVpS1\nZy16Ul8ALriv50TkKWB4tddx9uzZot8XFhYqHutpEAsLCyQSiXwTmlqbmDGm6rz+4xp5P2guxdRi\nmt/+20f51zfu470jI0XHeveWyRlCIcmHq3oc3rnc2eynv78/Hwr8gt09/PXbjxS9LyLL1haPx1la\nWkJRlPXNmjqpRWQvcD1wnzv0GyLyqIh8UkS2VTjndhE5KiJHvSJzzSCTtfnwt4/x2OjyjmwLCwuk\n02lGR0eLspFrCYipqSnOnTuXt7lX29CfOD/L+FyhBlS5DTToU/VzEwt87t7T+TpKJ8cdIfX5+88U\n3R84JTgAnrk4ly/f7acZrUFL79VfL0lRlPXLmgkIEekA7gLea4yZBT4G7Aeuw9Ew/rTcecaYO4wx\nR4wxRwYGBsodUu86ADhxaZ6nL8zxo7MF34BXxG52djbva6hVyC+bzeYFQmnUTqVzjTF89O5n+MBX\nHiuapxrGmIpP4f/175/in46Nc3neWcdzEwUt5vh4cc/qTCbDt58c49mLy5PnYOWmnnIahKIoG4M1\nERAiEsERDp8zxnwFwBhz0RhjG2NywMeBG1t1/VITyz8fG+fBM45gODm+wOiU49z1R9uU6yftn8ez\n4Z88eZJTp06Vve7ly5fLjqezlXtVl/otMpkMtm1z+vRpzp07t+y4opafbnb06cuL9LkO5cmF5aGm\ndz7g9KG+erhr2XuN0NbWln/tFxDa8U1RNhar7oMQ55H0E8BTxpiP+MaHXP8EwBuAx1u1Bv+me+zi\nPJ+993T+99GpJP/p60/y0be8gM54IazT8zuk02ls2y7KYxifS/GBrzzG214T5eYdJn9czQS4hTTJ\ndJbFyYK2YOcMJW6AwnXGx6lkVvOE2WK6IGxmkxkeGZ3myQuzvOxgPxePTzO1UBxi+u0nCo7sUuey\nxwded1VRiGs1du7cSUdHBydOnFgWeluqjUSj0WXnK4qyfliLPIiXAL8IPCYiD7tjvwu8VUSuAwzw\nHPDOVi3Av3GPzxbMNPv62zh12dEefvNLj3DHL72QkLupeSYfYwzHjx8HHAetMYa//Cfn9z/85tO8\n7O0HsELCqVOnaG9vz8/92LkZHj07zc//+AF+dHqKF+7t43e/8ljeT+BxaW6Jwc4Yxhguz6fZUX7P\nXsaFi5f49A9O8+L9hf4LY7NLPHvRMSndsGcbR0fnmVhI54XTqfEF7jw6mj/eLwQGBgbywujAQPCy\n2aFQCBFh//79+bH29naWlpYIhwsft4MHDyIiZTUzRVHWB2sRxfR9nPbFpbQ0rLVkDfnXsYiVf/2r\nL91Pf0eUf/vZhwB45Ow0149sI2cMf/R/HuamfX0c2VvwnV++fJm0nWN0KklPW4TRecNTY3M8f6gT\nESlKhPvz7zwLwN3HJgiT400vTC0TDgC//Ikf8isv2cfSxCSf+P4pfv9NbRwerNxr2c4ZkhmbU5cX\nuPnPgSMAAA28SURBVPfkBPeeLBQX/MpDjglqoDPGNcPddMTHmJpzmvn87dGzHH2uuGJtd7zwRO/f\nzAEGBwfp6OhgbGysamSWpyV4Ghc4obA9PT1Fc3rvh8NhDhw4wIkTJ8rOoyjK2rElM6n9iW837uvl\nhpFtLKSz+Sfon3/xCJ+/7wyfu+8M52eWODOxyMNnpnn4zDQf3/PCos3LM+lct7uH6WPj/NndM3zw\n9c9jb1973uxz8nJhQ7Vwrv3lBwtP7n9w6/PpaY/y/33+RyxevsAffHWGV+5xNus/+PJ9xCMW/+0N\nV+dLX/j5L3//ZFEHN4+Bzlg+KmrCNSt1RQxPHHuWex5L4MnIHz/Qx/6BDr72o1ES0cpJcj09PQAM\nDw9j2/ayDb3Sed5YqcDxU+09RVHWji1Zi6nUNxC2pMi88sqrtvOGG4aZXszw1YfO8eDpwpP2n377\nmaKezZ/+wXMAXLWji46Yo41Mzhfb+b0IoQ//3Av4H7ddxw0jBS3kf/3iC9nZk6AtYvHvbj7Aq563\nnb7QIs/5hMpSxua37nyE+VSWmcUMSV/SWznh8OYju/jZ63fSGQ/zvKFO3nPLQcCJ1AL41Pece9jV\n28avvnQfN185wJ/ddn3R5u7XAPx9HGpt9oqibB625Dc9SNjl1Tu7+epD55aNPz02x7/97EP8xJUD\nvOH6YR4/59QLikVCfPD1h3n/XY/ltYpUxubP73mWZy7O0xaz6GlzNto3v2gXe/oSvPJ5g0WJaTfs\n2cbVw91856lLzCQzWCHhv//ctfzWlx4B4L1fdFw2/Z1R/uiN17LkCorOeJi5pSwi8N5XHeL5bnLb\ni/cVd2e79bqd/N3D54lZ8OjoDAe2L/cteFFHfgFRD2oaUpTNw5YUEJFIhMHBwaotOLe1O5u5FRJ+\n57VXsr0rTiJi8bn7zvDdZ8b5l2PjRVrHUHecNrfj2snL8xwfn+P7zxb8AYupwlN/f0eMn752Z9nr\n+quitscsuuIR3nxkV5Ez+fJcmm8/MZYfe/OR3fzYgdqtOn/mBTuJRUL5sFZPo/DT1tbGwsJCwxv9\nSgSEChdFWV9sSQFhWRbd3d1VBURXPMKvvXwfhwY72dZWcN7+wotHeOLcDBMLab7+8HkA/vhN19LX\nHs1rJt99pjjf4aqhTl56RX/g9fW0RZhezPArL3EK2x3e2UV7zOLf3nyA/3d8gh+emCgSGId2dAae\ne09vIbLqTS/cBcD+/fs5efIk4ISpplIpLKvgvC+3ce/fvz9/v34nc6Ob/N69e4uuuRIOHDigvScU\npQlsSQEBlZ2pfvNTqYkGHI3ij990Ld98bIy7HnI26V7XdCQieedwW8zi12++gq62SFH7ziC855aD\nnBhf4GrXVLRrWxt/ftv1gKN9/PCEo5m89uodXD/Sk0+CC8LOnsJaXnGVk4nu9zGEQiESiUTNTG7/\nOX6fRKMCIhaLNXReOdRHoijNQb9JOBm+lmVx5syZwHH5r7l6EMuCH9vfh4hgWRa2bfOHb7yGx0Zn\nGOqJ098RfNMLhUL5p97dvW35bm6l9HfElhXEq4fOeISfPDxINBwiFq78xK4+CEVRVEAAiYSTZxBk\nc+vt7WVycpKQCD95uJDF5tc8rtm1vB9zNYaGhhARzp8/X9d5jfLmF+2ueYz/bxHk79LsCq0qaBRl\n7dmSYa6VqLYp7dy5k5GRETo7g9v7/fT3V/ZBdHV15U02oVCIwcFBDhw4UHRMJbNJKBQqmtsTdpXo\n6OgI1IxHRIp6PdRi165dDA0NqXlHUTYRKiB8lNvcEokEe/bsobOzk0QiUXRMX18f3d2OtlDJht7d\n3c3g4CC9vb1l3/eEkuegNcbQ09NTZOLp7e1l165d+Wt1dXWxbdu2/Jr9G3npPZQKjKGhofy5fvbt\n28fwcHFbjnqEoWVZdHU1p9hf6byKoqwNW/pxb2RkpMjnsHPnTubm5giHw0xPT7O4uEh7ezvxeMGx\n621Y3pN7LpcjGo2ybds2pqam8vWL9u3bx/z8/LLN3mP37t1ks9n83N68pRuiZVl4Zc07OjqYmZlB\nROju7mZqaiovEMLhMNlslu3btyMi+X7Og4ODPPfcc/l5KvkWotFoxeJ5a7VJ79ixo6ZGpChK69jS\nAqJ08wmHw/mn63g8zuTk5LKnaBFhcHAwf24oFMprB729vViWRTweJxqNVtQahoeHi0pie/Ns3749\nX+Cv9Hdwit719/fT09ODZVn09/fntYqRkZG8cPN8GpZlEYvFGBgYoLOzsyjyyGPv3r0V/z7eua3Q\nDILg3ZuiKGuDbORmLkeOHDFHjx5d62UEZnp6mng8XqSRrBWzs7OEw+FlgkpRlM2PiDxojKkZDrml\nNYjVxit4tx5YK61AUZSNgzqpFUVRlLKogFAURVHKogJCURRFKcu6ExAi8loROSYix0Xk/Wu9HkVR\nlK3KuhIQImIBfwn8FHAYp0/14bVdlaIoytZkXQkI4EbguDHmpDEmDXwRuHWN16QoirIlWW8CYhg4\n6/t91B3LIyK3i8hRETnqZS0riqIozWe9CYiaGGPuMMYcMcYc8UpQKIqiKM1nvSXKnQP8tah3uWNl\nefDBBy+LyOkGr9UPXK551OZC73lroPe8NVjJPe8JctC6KrUhImHgGeAWHMHwAPDzxpgnWnCto0FS\nzTcTes9bA73nrcFq3PO60iCMMVkR+XXgW4AFfLIVwkFRFEWpzboSEADGmG8A31jrdSiKomx1NpyT\nuoncsdYLWAP0nrcGes9bg5bf87ryQSiKoijrh62sQSiKoihV2JICYjPWexKR3SLyTyLypIg8ISLv\nccd7ReRuEXnW/Xeb75wPuH+DYyLymrVb/coQEUtEfiQif+/+vqnvWUR6ROTLIvK0iDwlIj+2Be75\nN93P9eMi8gURiW/GexaRT4rIJRF53DdW932KyAtF5DH3vf8hItLQgowxW+oHJzrqBLAfiAKPAIfX\nel1NuK8h4Ab3dSdOuPBh4E+A97vj7wf+2H192L33GLDP/ZtYa30fDd77bwGfB/7e/X1T3zPwaeDf\nuK+jQM9mvmecagqngIT7+53AL2/GewZeDtwAPO4bq/s+gfuBmwABvgn8VCPr2YoaxKas92SMuWCM\nech9PQc8hfPFuhVnQ8H992fd17cCXzTGpIwxp4DjOH+bDYWI7AJ+Gvhr3/CmvWcR6cbZRD4BYIxJ\nG2Om2cT37BIGEm6uVBtwnk14z8aY7wKTJcN13aeIDAFdxph7jSMtPuM7py62ooCoWe9poyMie4Hr\ngfuAQWPMBfetMWDQfb1Z/g5/Bvx7IOcb28z3vA8YB/63a1b7axFpZxPfszHmHPBh4AxwAZgxxnyb\nTXzPJdR7n8Pu69LxutmKAmJTIyIdwF3Ae40xs/733KeJTRO2JiKvBy4ZYx6sdMxmu2ecJ+kbgI8Z\nY64HFnDMDnk22z27NvdbcYTjTqBdRN7mP2az3XMlVvs+t6KAqKve00ZCRCI4wuFzxpivuMMXXZUT\n999L7vhm+Du8BPhXIvIcjqnwlSLyWTb3PY8Co8aY+9zfv4wjMDbzPb8KOGWMGTfGZICvAD/O5r5n\nP/Xe5zn3del43WxFAfEAcFBE9olIFLgN+Poar2nFuFEKnwCeMsZ8xPfW14G3u6/fDvydb/w2EYmJ\nyD7gII5ja8NgjPmAMWaXMWYvzv/j/zXGvI3Nfc9jwFkRudIdugV4kk18zzimpZtEpM39nN+C42Pb\nzPfsp677dM1RsyJyk/v3+iXfOfWx1l77tfgBXocT5XMC+L21Xk+T7umlOKrno8DD7s/rgD7gHuBZ\n4DtAr++c33P/BsdoMMphvfwAN1OIYtrU9wxcBxx1/6+/BmzbAvf8n4GngceBv8GJ3Nl09wx8AcfP\nksHRFt/RyH0CR9y/1QngL3CTouv90UxqRVEUpSxb0cSkKIqiBEAFhKIoilIWFRCKoij/f3v3D9pk\nEMZx/PuzLoGiWMVNXVxEWhTdXAQnERcVgn9wLghOgn/BdnVR3AQRBKVudhGkg4KKig6GgM51L6gg\nikh5HO5CX8NFX+prI/j7QHjf3OVyuSVP7rg8Z0UOEGZmVuQAYWZmRQ4QZhWSFiV1Ko9fZvuVNCnp\nZAP9zkva8KfvY9Ykb3M1q5D0OSJGh9DvPLA7IhZWum+zQTyDMKsh/8K/knPsv5K0NZdPSTqT708r\nncfRlXQvl41Jms1lLyVN5PL1kubyGQc3SWmZe32dyH10JN2QNDKEIZs5QJj1afUtMbUrdZ8iYpz0\nz9RrhbbngJ0RMQFM5rJp4E0uu0BKvQxwGXgWEduB+8BmAEnbgDawJyJ2AIvA8WaHaFbP6mF/ALN/\nzNf8xVwyU7leLdR3gbuSZkkpMCClQDkMEBGP8sxhDelMh0O5/IGkD/n1+4BdwOt8CFiLpeRsZivK\nAcKsvhhw33OA9MV/ELgoaXwZfQi4HRHnl9HWrFFeYjKrr125vqhWSFoFbIqIx8BZYC0wCjwlLxFJ\n2gssRDqn4wlwLJfvJyXcg5SU7YikjbluTNKWvzgms4E8gzD7WUtSp/L8YUT0trquk9QFvgFH+9qN\nAHfykaACrkfER0lTwK3c7gtLaZungRlJb4HnpJTWRMQ7SZeAuRx0vgOngPdND9Tsd7zN1awGb0O1\n/5GXmMzMrMgzCDMzK/IMwszMihwgzMysyAHCzMyKHCDMzKzIAcLMzIocIMzMrOgHhuCfhIOFU0IA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x120004dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eps, rews = np.array(rewards_list).T\n",
    "smoothed_rews = running_mean(rews, 10)\n",
    "plt.plot(eps[-len(smoothed_rews):], smoothed_rews)\n",
    "plt.plot(eps, rews, color='grey', alpha=0.3)\n",
    "plt.xlabel('Episode')\n",
    "plt.ylabel('Total Reward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing\n",
    "\n",
    "Let's checkout how our trained agent plays the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_episodes = 10\n",
    "test_max_steps = 400\n",
    "env.reset()\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))\n",
    "    \n",
    "    for ep in range(1, test_episodes):\n",
    "        t = 0\n",
    "        while t < test_max_steps:\n",
    "            env.render() \n",
    "            \n",
    "            # Get action from Q-network\n",
    "            feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "            Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "            action = np.argmax(Qs)\n",
    "            \n",
    "            # Take action, get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            if done:\n",
    "                t = test_max_steps\n",
    "                env.reset()\n",
    "                # Take one random step to get the pole and cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "            else:\n",
    "                state = next_state\n",
    "                t += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extending this\n",
    "\n",
    "So, Cart-Pole is a pretty simple game. However, the same model can be used to train an agent to play something much more complicated like Pong or Space Invaders. Instead of a state like we're using here though, you'd want to use convolutional layers to get the state from the screen images.\n",
    "\n",
    "![Deep Q-Learning Atari](assets/atari-network.png)\n",
    "\n",
    "I'll leave it as a challenge for you to use deep Q-learning to train an agent to play Atari games. Here's the original paper which will get you started: http://www.davidqiu.com:8888/research/nature14236.pdf."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}