assignment4/Q_learning_Basic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Q-Learning Algorithms\n",
    "In this exercise we are going be exploring a family of RL algorithms called Q-Learning algorithms. You will begin by implementing a simple lookup-table version of the algorithm, and then a neural-network equivalent using Tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenAI Gym Environment\n",
    "For this exercise we will use the [FrozenLake](https://gym.openai.com/envs/FrozenLake-v0) environment from the [OpenAI gym](https://gym.openai.com) as a toy example. For those unfamiliar, the OpenAI gym provides an easy way for people to experiment with their learning agents in an array of provided toy games. The FrozenLake environment consists of a `4 x 4` grid of blocks, each one either being the start block `S`, the goal block `G`, a safe frozen block `F`, or a dangerous hole `H`. The objective is to have an agent learn to navigate from the start to the goal without moving onto a hole. At any given time the agent can choose to move either up, down, left, or right. The catch is that there is a wind which occasionally blows the agent onto a space they didn’t choose. As such, perfect performance every time is impossible, but learning to avoid the holes and reach the goal are certainly still doable. The reward at every step is 0, except for entering the goal, which provides a reward of 1. Thus, we will need an algorithm that learns long-term expected rewards. This is exactly what Q-Learning is designed to provide."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install OpenAI Gym\n",
    "To install the OpenAI gym, simply use  `pip install gym`  to grab it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "env = gym.make('FrozenLake-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information, please refer to [OpenAI documentation](https://gym.openai.com/docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Q-Table learning algorithm\n",
    "In it’s simplest implementation, Q-Learning is a table of values for every state (row) and action (column) possible in the environment. Within each cell of the table, we learn a value for how good it is to take a given action within a given state. In the case of the FrozenLake environment, we have 16 possible states (one for each block), and 4 possible actions (the four directions of movement), giving us a 16 x 4 table of Q-values. We start by initializing the table to be uniform (all zeros), and then as we observe the rewards we obtain for various actions, we update the table accordingly.\n",
    "\n",
    "We make updates to our Q-table using something called the [Bellman equation](https://en.wikipedia.org/wiki/Bellman_equation), which states that the expected long-term reward for a given action is equal to the immediate reward from the current action combined with the expected reward from the best future action taken at the following state. In equation form, the rule looks like this (Equation 1):\n",
    "$$ Q(s,a) = r + γ(\\max(Q(s’,a’)) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This says that the Q-value for a given state ($s$) and action ($a$) should represent the current reward ($r$) plus the maximum discounted ($\\lambda$) future reward expected according to our own table for the next state ($s'$) we would end up in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Initialize table, with states as rows and actions (up, down, left, or right) as columns \n",
    "Q = np.zeros([env.observation_space.n,env.action_space.n])\n",
    "#Set learning parameters\n",
    "lr = .8\n",
    "#Set discounted factor\n",
    "y = .95\n",
    "num_episodes = 2000\n",
    "#create lists to contain total rewards and steps per episode\n",
    "rList = []\n",
    "for i in range(num_episodes):\n",
    "    #Reset environment and get first new observation\n",
    "    s = env.reset()\n",
    "    #Total reward in one episode\n",
    "    rAll = 0\n",
    "    d = False\n",
    "    j = 0\n",
    "    while j < 99:\n",
    "        j+=1\n",
    "        ###############################################################################\n",
    "        # TODO: Implement the Q-Table learning algorithm.                             #\n",
    "        # You will need to do the following:                                          #\n",
    "        # (1) Choose an action by greedily (with noise) picking from Q table given s  #\n",
    "        #     as input.                                                               #\n",
    "        # (2) Get new state s1, reward r and done d from environment                  #\n",
    "        # (3) Update Q-Table with new knowledge.                                      #\n",
    "        # (4) Cumulate the total reward rAll                                          #\n",
    "        # (5) Update s                                                                #\n",
    "        # Note: You may use the gym interfaces env.action_space, env.step etc.        #\n",
    "        #       E.g. observation, reward, done, info = env.step(action)               #\n",
    "        #       Please refer to the docs for more information.                        #\n",
    "        #       For (1), consider adding noise as a mean of encouraging exploration.  #\n",
    "        #       For (3), calculate the new target Q-value using Bellman equation.     #\n",
    "        #       Instead of directly updating toward it, we take a small step in the   #\n",
    "        #       direction that will make the Q value closer to the target, i.e. use   #\n",
    "        #       learning rate that controls how much of the difference between        #\n",
    "        #       newly proposed Q-value and previous Q-value                           #\n",
    "        ###############################################################################\n",
    "        a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))\n",
    "        s_new, r, d, _ = env.step(a)\n",
    "        q_new = r + y * (np.max(Q[s_new]))\n",
    "        Q[s, a] = (1 - lr) * Q[s, a] + lr * q_new\n",
    "        rAll += r\n",
    "        s = s_new\n",
    "    ##############################################################################\n",
    "    #                             END OF YOUR CODE                               #\n",
    "    ##############################################################################\n",
    "\n",
    "    #end of one episode\n",
    "        if d == True:\n",
    "            break\n",
    "    rList.append(rAll)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score is around 0.5 after 2000 episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score over time: 0.637\n"
     ]
    }
   ],
   "source": [
    "print(\"Score over time: \" +  str(sum(rList)/num_episodes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Q-Table Values\n",
      "[[  1.60654593e-01   1.18017501e-02   1.10602005e-02   1.19587244e-02]\n",
      " [  1.27286322e-04   4.23297144e-03   2.01027687e-03   3.04590758e-01]\n",
      " [  3.52037386e-03   3.71655525e-03   4.93010814e-03   1.80302749e-01]\n",
      " [  2.42274775e-03   2.19359109e-04   6.62894653e-04   3.87048151e-02]\n",
      " [  2.93244060e-01   5.12191094e-04   5.02800667e-03   5.81994189e-03]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]\n",
      " [  1.02597504e-05   3.01450066e-06   1.21736211e-01   3.59685992e-04]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   6.29550673e-03   5.23065725e-01]\n",
      " [  0.00000000e+00   3.09358147e-01   0.00000000e+00   0.00000000e+00]\n",
      " [  6.30171534e-01   1.49781821e-03   1.81831980e-04   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   8.08850236e-01   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   9.47854755e-01   0.00000000e+00]\n",
      " [  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Final Q-Table Values\")\n",
    "print(Q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (Up)\n",
      "SFFF\n",
      "FHF\u001b[41mH\u001b[0m\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "# print out the 4 x 4 grid and the current position of the agent\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 1:\n",
    "In TODO(3), why not directly apply the Bellman equation for updating the Q value? (in this case lr = 1 and why?)\n",
    "\n",
    "**Your answer:** Having lr = 1 will make the agent <b>only</b> consider the recent information and old value is <b>completely forgotten</b>. We still need some of the old information in order to complete the task. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question 2:\n",
    "An optimal Q table will tell you the true expected discounted reward for any action given any state. If you find the maximum value of the learned table is not what you believe it should be, do you think it still make sense? Explain briefly.**\n",
    "\n",
    "**Your answer:** It may still make sense because the values have been generated in a stochastic environment in which there is <b>randomness</b> in it therefore the generated values may not be perfect. However the values are still good enough to finish the task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Part 2 - Q-Network Approach\n",
    "While it is easy to have a 16x4 table for a simple grid world, the number of possible states in any modern game or real-world environment is nearly infinitely larger. For most interesting problems, tables simply don’t work. We instead need some way to take a description of our state, and produce Q-values for actions without a table: that is where neural networks come in. By acting as a function approximator, we can take any number of possible states that can be represented as a vector and learn to map them to Q-values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of the FrozenLake example, we will be using a one-layer network which takes the state encoded in a one-hot vector `(1x16)`, and produces a vector of 4 Q-values, one for each action. Such a simple network acts kind of like a glorified table, with the network weights serving as the old cells. The key difference is that we can easily expand the Tensorflow network with added layers, activation functions, and different input types, whereas all that is impossible with a regular table. The method of updating is a little different as well. Instead of directly updating our table, with a network we will be using backpropagation and a loss function. Our loss function will be sum-of-squares loss, where the difference between the current predicted Q-values, and the “target” value is computed and the gradients passed through the network. **In this case, our Q-target value for the chosen action is the equivalent to the Q-value computed in equation 1 above.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing the network itself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "env = gym.make('FrozenLake-v0')\n",
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#These lines establish the feed-forward part of the network used to choose actions\n",
    "inputs1 = tf.placeholder(shape=[1,16],dtype=tf.float32)\n",
    "W = tf.Variable(tf.random_uniform([16,4],0,0.01))\n",
    "Qout = tf.matmul(inputs1,W)\n",
    "predict = tf.argmax(Qout,1)\n",
    "\n",
    "#Below we obtain the loss by taking the sum of squares difference between the target and prediction Q values.\n",
    "nextQ = tf.placeholder(shape=[1,4],dtype=tf.float32)\n",
    "loss = tf.reduce_sum(tf.square(nextQ - Qout))\n",
    "trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1)\n",
    "updateModel = trainer.minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 9 reward: 0.1\n",
      "Episode 19 reward: 0.0\n",
      "Episode 29 reward: 0.0\n",
      "Episode 39 reward: 0.0\n",
      "Episode 49 reward: 0.0\n",
      "Episode 59 reward: 0.0\n",
      "Episode 69 reward: 0.0\n",
      "Episode 79 reward: 0.0\n",
      "Episode 89 reward: 0.0\n",
      "Episode 99 reward: 0.0\n",
      "Episode 109 reward: 0.0\n",
      "Episode 119 reward: 0.1\n",
      "Episode 129 reward: 0.0\n",
      "Episode 139 reward: 0.0\n",
      "Episode 149 reward: 0.0\n",
      "Episode 159 reward: 0.1\n",
      "Episode 169 reward: 0.0\n",
      "Episode 179 reward: 0.0\n",
      "Episode 189 reward: 0.0\n",
      "Episode 199 reward: 0.2\n",
      "Episode 209 reward: 0.0\n",
      "Episode 219 reward: 0.0\n",
      "Episode 229 reward: 0.0\n",
      "Episode 239 reward: 0.0\n",
      "Episode 249 reward: 0.1\n",
      "Episode 259 reward: 0.1\n",
      "Episode 269 reward: 0.0\n",
      "Episode 279 reward: 0.2\n",
      "Episode 289 reward: 0.0\n",
      "Episode 299 reward: 0.1\n",
      "Episode 309 reward: 0.3\n",
      "Episode 319 reward: 0.1\n",
      "Episode 329 reward: 0.1\n",
      "Episode 339 reward: 0.0\n",
      "Episode 349 reward: 0.1\n",
      "Episode 359 reward: 0.2\n",
      "Episode 369 reward: 0.0\n",
      "Episode 379 reward: 0.1\n",
      "Episode 389 reward: 0.1\n",
      "Episode 399 reward: 0.1\n",
      "Episode 409 reward: 0.1\n",
      "Episode 419 reward: 0.0\n",
      "Episode 429 reward: 0.1\n",
      "Episode 439 reward: 0.0\n",
      "Episode 449 reward: 0.0\n",
      "Episode 459 reward: 0.0\n",
      "Episode 469 reward: 0.1\n",
      "Episode 479 reward: 0.0\n",
      "Episode 489 reward: 0.0\n",
      "Episode 499 reward: 0.1\n",
      "Episode 509 reward: 0.3\n",
      "Episode 519 reward: 0.1\n",
      "Episode 529 reward: 0.2\n",
      "Episode 539 reward: 0.1\n",
      "Episode 549 reward: 0.2\n",
      "Episode 559 reward: 0.3\n",
      "Episode 569 reward: 0.0\n",
      "Episode 579 reward: 0.0\n",
      "Episode 589 reward: 0.1\n",
      "Episode 599 reward: 0.2\n",
      "Episode 609 reward: 0.1\n",
      "Episode 619 reward: 0.3\n",
      "Episode 629 reward: 0.4\n",
      "Episode 639 reward: 0.1\n",
      "Episode 649 reward: 0.3\n",
      "Episode 659 reward: 0.3\n",
      "Episode 669 reward: 0.3\n",
      "Episode 679 reward: 0.3\n",
      "Episode 689 reward: 0.2\n",
      "Episode 699 reward: 0.2\n",
      "Episode 709 reward: 0.4\n",
      "Episode 719 reward: 0.4\n",
      "Episode 729 reward: 0.3\n",
      "Episode 739 reward: 0.3\n",
      "Episode 749 reward: 0.2\n",
      "Episode 759 reward: 0.2\n",
      "Episode 769 reward: 0.1\n",
      "Episode 779 reward: 0.0\n",
      "Episode 789 reward: 0.4\n",
      "Episode 799 reward: 0.4\n",
      "Episode 809 reward: 0.2\n",
      "Episode 819 reward: 0.3\n",
      "Episode 829 reward: 0.2\n",
      "Episode 839 reward: 0.6\n",
      "Episode 849 reward: 0.1\n",
      "Episode 859 reward: 0.2\n",
      "Episode 869 reward: 0.4\n",
      "Episode 879 reward: 0.0\n",
      "Episode 889 reward: 0.3\n",
      "Episode 899 reward: 0.5\n",
      "Episode 909 reward: 0.0\n",
      "Episode 919 reward: 0.4\n",
      "Episode 929 reward: 0.2\n",
      "Episode 939 reward: 0.2\n",
      "Episode 949 reward: 0.4\n",
      "Episode 959 reward: 0.5\n",
      "Episode 969 reward: 0.5\n",
      "Episode 979 reward: 0.3\n",
      "Episode 989 reward: 0.3\n",
      "Episode 999 reward: 0.4\n",
      "Episode 1009 reward: 0.5\n",
      "Episode 1019 reward: 0.2\n",
      "Episode 1029 reward: 0.7\n",
      "Episode 1039 reward: 0.4\n",
      "Episode 1049 reward: 0.6\n",
      "Episode 1059 reward: 0.4\n",
      "Episode 1069 reward: 0.5\n",
      "Episode 1079 reward: 0.7\n",
      "Episode 1089 reward: 0.4\n",
      "Episode 1099 reward: 0.4\n",
      "Episode 1109 reward: 0.7\n",
      "Episode 1119 reward: 0.7\n",
      "Episode 1129 reward: 0.7\n",
      "Episode 1139 reward: 0.4\n",
      "Episode 1149 reward: 0.6\n",
      "Episode 1159 reward: 0.8\n",
      "Episode 1169 reward: 0.9\n",
      "Episode 1179 reward: 0.6\n",
      "Episode 1189 reward: 0.7\n",
      "Episode 1199 reward: 0.6\n",
      "Episode 1209 reward: 0.7\n",
      "Episode 1219 reward: 0.9\n",
      "Episode 1229 reward: 0.6\n",
      "Episode 1239 reward: 0.4\n",
      "Episode 1249 reward: 0.4\n",
      "Episode 1259 reward: 0.7\n",
      "Episode 1269 reward: 0.5\n",
      "Episode 1279 reward: 0.3\n",
      "Episode 1289 reward: 0.3\n",
      "Episode 1299 reward: 0.4\n",
      "Episode 1309 reward: 0.3\n",
      "Episode 1319 reward: 0.6\n",
      "Episode 1329 reward: 0.8\n",
      "Episode 1339 reward: 0.6\n",
      "Episode 1349 reward: 0.6\n",
      "Episode 1359 reward: 0.8\n",
      "Episode 1369 reward: 0.5\n",
      "Episode 1379 reward: 0.5\n",
      "Episode 1389 reward: 0.6\n",
      "Episode 1399 reward: 0.6\n",
      "Episode 1409 reward: 0.5\n",
      "Episode 1419 reward: 0.7\n",
      "Episode 1429 reward: 0.5\n",
      "Episode 1439 reward: 0.6\n",
      "Episode 1449 reward: 0.4\n",
      "Episode 1459 reward: 0.4\n",
      "Episode 1469 reward: 0.6\n",
      "Episode 1479 reward: 0.6\n",
      "Episode 1489 reward: 0.7\n",
      "Episode 1499 reward: 0.7\n",
      "Episode 1509 reward: 0.5\n",
      "Episode 1519 reward: 0.7\n",
      "Episode 1529 reward: 0.7\n",
      "Episode 1539 reward: 0.6\n",
      "Episode 1549 reward: 0.8\n",
      "Episode 1559 reward: 0.4\n",
      "Episode 1569 reward: 0.7\n",
      "Episode 1579 reward: 0.6\n",
      "Episode 1589 reward: 0.5\n",
      "Episode 1599 reward: 0.4\n",
      "Episode 1609 reward: 0.7\n",
      "Episode 1619 reward: 0.6\n",
      "Episode 1629 reward: 0.5\n",
      "Episode 1639 reward: 0.6\n",
      "Episode 1649 reward: 0.6\n",
      "Episode 1659 reward: 0.5\n",
      "Episode 1669 reward: 0.8\n",
      "Episode 1679 reward: 0.8\n",
      "Episode 1689 reward: 0.5\n",
      "Episode 1699 reward: 0.4\n",
      "Episode 1709 reward: 0.4\n",
      "Episode 1719 reward: 0.6\n",
      "Episode 1729 reward: 0.9\n",
      "Episode 1739 reward: 0.5\n",
      "Episode 1749 reward: 0.9\n",
      "Episode 1759 reward: 0.6\n",
      "Episode 1769 reward: 0.7\n",
      "Episode 1779 reward: 0.5\n",
      "Episode 1789 reward: 0.7\n",
      "Episode 1799 reward: 0.8\n",
      "Episode 1809 reward: 0.7\n",
      "Episode 1819 reward: 0.5\n",
      "Episode 1829 reward: 0.9\n",
      "Episode 1839 reward: 0.7\n",
      "Episode 1849 reward: 0.7\n",
      "Episode 1859 reward: 0.5\n",
      "Episode 1869 reward: 0.7\n",
      "Episode 1879 reward: 0.7\n",
      "Episode 1889 reward: 0.8\n",
      "Episode 1899 reward: 0.5\n",
      "Episode 1909 reward: 0.4\n",
      "Episode 1919 reward: 0.5\n",
      "Episode 1929 reward: 0.6\n",
      "Episode 1939 reward: 0.6\n",
      "Episode 1949 reward: 0.4\n",
      "Episode 1959 reward: 0.5\n",
      "Episode 1969 reward: 0.7\n",
      "Episode 1979 reward: 0.6\n",
      "Episode 1989 reward: 0.6\n",
      "Episode 1999 reward: 0.7\n",
      "Percent of succesful episodes: 0.3715%\n"
     ]
    }
   ],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# Set learning parameters\n",
    "#discounted factor\n",
    "y = .99\n",
    "#chance of random action\n",
    "e = 0.1\n",
    "num_episodes = 2000\n",
    "#create lists to contain total rewards and steps per episode\n",
    "jList = []\n",
    "rList = []\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for i in range(num_episodes):\n",
    "        #Reset environment and get first new observation\n",
    "        s = env.reset()\n",
    "        #Total reward in one episode\n",
    "        rAll = 0\n",
    "        d = False\n",
    "        j = 0\n",
    "        #The Q-Network\n",
    "        while j < 99:\n",
    "            j+=1\n",
    "            \n",
    "            ###############################################################################\n",
    "            # TODO: Implement the Q-network approach.                                     #\n",
    "            # You will need to do the following:                                          #\n",
    "            # (1) Choose an action by greedily (with e chance of random action, e=0.1)    # \n",
    "            #     from the Q-network                                                      #\n",
    "            # (2) Get new state s1, reward r and done d from environment                  #\n",
    "            # (3) Obtain the Q' values by feeding the new state through our network       # \n",
    "            # (4) Obtain maxQ' and set our target value for chosen action.                #\n",
    "            # (5) Train our network using target and predicted Q values                   #\n",
    "            # (6) Cumulate the total reward rAll                                          #\n",
    "            # (7) Update observation s                                                    #\n",
    "            # Note: In (1) we need to feed a one-hot vector encoding the state space to   # \n",
    "            #       our network. The environment represents the position in the grid-     #\n",
    "            #       world as a number between 0 and 15, e.g. if s=11, the one-hot vector  #\n",
    "            #       (here is inputs1) should be                                           #\n",
    "            #       [[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.]]   #\n",
    "            ###############################################################################\n",
    "            state_encoded = np.eye(env.observation_space.n)[s].reshape((1, env.observation_space.n))\n",
    "            a, q_out = sess.run([predict, Qout], feed_dict={inputs1: state_encoded}) \n",
    "            a[0] = env.action_space.sample() if np.random.rand(1) < e else a[0]\n",
    "            s_new, r, d, _ = env.step(a[0])\n",
    "            \n",
    "            state_new_encoded = np.eye(env.observation_space.n)[s_new].reshape((1, env.observation_space.n))\n",
    "            q_out_new = sess.run(Qout, feed_dict={inputs1: state_new_encoded})\n",
    "            a_new = np.max(q_out_new)\n",
    "            target_q = r + y * a_new\n",
    "            q_out[0, a[0]] = target_q\n",
    "            sess.run([updateModel, W], feed_dict={nextQ: q_out, inputs1: state_encoded})\n",
    "            rAll += r\n",
    "            s = s_new\n",
    "            ##############################################################################\n",
    "            #                             END OF YOUR CODE                               #\n",
    "            ##############################################################################\n",
    "            \n",
    "            if d == True:\n",
    "                #Reduce chance of random action as we train the model.\n",
    "                e = 1./((i/50) + 10)\n",
    "                break\n",
    "        jList.append(j)\n",
    "        rList.append(rAll)\n",
    "        if len(rList) % 10 == 0:\n",
    "            print(\"Episode\",i,\"reward:\",np.mean(rList[-10:]))\n",
    "print(\"Percent of succesful episodes: \" + str(sum(rList)/num_episodes) + \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some statistics on network performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the network beings to consistly reach the goal around the 750 episode mark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f5bc809f588>]"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHmFJREFUeJzt3Xl4HHed5/H315LlS5ZlR/IRW7Gc\n2Dmcy0mUA0ggbAKxnSVhCDDxzE4CA5PZecg+sDDLBsJmMuHZZ8mwC8/ObBY2PGS4ExIgrIdxCJCw\nM4TgEDn35Vg5HMunfMq3fHz3jy6JltSt7paqqruqPq/n0aPuql9V/bqq+9PVv1//uszdERGRdBlX\n7QqIiEj4FO4iIimkcBcRSSGFu4hICincRURSSOEuIpJCCncRkRRSuIuIpJDCXUQkheqrteGWlhZv\nb2+v1uZFRBJpzZo12929tVS5qoV7e3s7nZ2d1dq8iEgimdn6csqpWUZEJIUU7iIiKaRwFxFJIYW7\niEgKKdxFRFKoZLib2T1mts3MXigy38zs782sy8yeM7Pzw6+miIhUopwz928BS0eYvwxYFPzdBHxt\n7NUSEZGxKPk9d3f/VzNrH6HItcB3PHe9vtVm1mxmc9x9c0h1HKb30BF+/co2rl0yd1TL7zl4hH95\ntYdrzj1x2LxfvLiFJW3NzGyaODDt+HHntpUv8MEL2ljS1jxsGXfnJ09t5IozZvKrl7fx/iUn8r3V\n61l+9pyB9Tz0/GZ+07Wd2U0T+cS7F7Jt7yFe2tTLu0+byY+e6ubiBTP41uNv8rFLF1A/bhy/WddD\n44R6lp41m588tZE5zRNpntTA4hObAHjw6W7eu3g2Uyb84RBu7T3E8917uHLxLACOHXd+vKabqRPr\neXXrPi6YP50jx4/z1PpdzGyayKTxdXTvOsDW3sM0Tqjj88vPwMzYuPsgj3dtZ/XrO/noO9p5/LXt\nbOs9zNMbdnNh+wzefsoJvLipl90H+3ht234mN9QxZUI9B/qO8uaOA/zZJfNpqB/HkaPHWb/zAD17\nD3Pv79/iry4/hVlTJ/DIK9u4dGELvYeOcF7bdJ7esIvPvOc0Xti0hy/+7CVOmz2Vn7+wlb+95kx+\n9twmduzr48a3t/NY13a6dx3gQx1tNNQZ9/5+A1Mn1nPoyDE+v/wMPv/g87zjlBY61+9ixUVtvPPU\nVr79+HraZkzin57dxDnzmtm4+yDPbtjNG9v3c868aWztPczBvmN8/LIFbN5ziN6DR3hgTTd/+c6T\nmdk0kbVbenls3XZ6Dx3lkpNP4Fcvb+X02VM5e+40HljTDcCF7dN5c8cBGifUM6F+HD17D/OuU1v5\n7Wvb2XvoKIvnNLF5zyE27j7Ivzl9Jmed2IQD//BoFwBL2pp5ZsPugeN42aIWxteN49FXtgFw6cIW\nLjl5Bk+9tZvnundTP24cM5sm0DZ9MnXjjNapE1i7ZS+PdW3ntFlTmTB+HNMnN7D7QB8vb9lL08Tx\nnDa7kd927WDFRSfx8xc2s+vAEQDGGZwxp4mz507j6bd2s3brXs48sYnZTRN55JVtNE6o55x505g5\ndQI/fWYTZ81tomnieB5/bQcNdeNomlTPufOaeeSVbdSPM5adPYdXt+xl7da9AJwzbxrPde8ZeFzr\ntu5jS+8hpk0az4Xt05nbPIn7O7u57oK5fG/1WwAsnNnIopmNPLNhN5v3HALghCkN7NjfR/sJk9lz\n8Aj7+45x+amtvLF9P+u27ePk1inMbZ7EjCkN7NjXN7Dd37+xk/aWyew/fIw50ybSuX4XS9qaWbtl\nLwePHOPUWY28unUfC1qmcF5bM08Hz42rz55D8+TxfP+JtwZt/08uPon1O/azfW8fm/Yc5Mix4yxp\na2acGXXjjN+s286MKQ0YcPU5c9i5vw8z45+e3QRAS2MDvYeO8sBfvo1zC2RJmKyca6gG4f4zdz+r\nwLyfAV9y98eC+48A/9ndh41QMrObyJ3dc9JJJ12wfn1Z38Uf5q++t4aHXtjCzz91GafPbqp4+Y9/\nu5NfvbyVX336XSyc2Tgw/eix4yy89SFObpnCo399+cD0+zs38NkfPQfAm1+6etj6Hn9tO3/yjScG\n7n/yikX8z0fW8YHz5/KVDy8BoP2Wfx6Y/8X3n8Vdj3axpfcQX/rA2dzyk+cH5k1pqGPapPFsCp7U\nX/3jc/mPP3x2YP6bX7qaNet3ct3XfscHL5jHf//QuQPzLr3zUbp3HRyo43dXr+e//LRga1pBP/j4\nxbx9YQvn3P4wvYeOlr1cGL76x+fyhQdfYH/fsdDW+Y8fvZCP/uOToa1PJCz/9Y/O4k8vnj+qZc1s\njbt3lCoXa4equ9/t7h3u3tHaWnL0bFH9wXdwlEGwafdBAA4dGbx8/9vcWzsPDJq+JzjLKWbfkCDc\nub8PgO6dBwuW33Ogjy29ucewa8i69/cdG3h8uXUN3/a+w7l6b+09NGh6967B29sV1KNc+w7nHkfc\nwQ65fRxmsEPuE1ct+sLVZzCloa7a1ZCUCyPcNwJteffnBdNERKRKwgj3lcANwbdmLgH2RNneLiIi\npZXsUDWze4HLgRYz6wb+BhgP4O5fB1YBy4Eu4ADw0agqKyIi5Snn2zIrSsx34BOh1UgkJGV8V0Ck\nKgyLfBsaoSoikkIKd5GYmRlm0Z+5SbYp3EVEUkjhLiKSQgr3FCpn1HEWaC9IrYqjVU7hLiKSQgp3\nkZhZ8CcSJYW7iEgKKdxFRFJI4Z4nLf2QaXkcY6WOZalVcTTLKdxFRFJI4S4SM1OPqsRA4S4ikkIK\ndxGRFFK45/EqjGmMYpvqRszRfpBapRGqIiIyKgr3KovjR/ultqg/VeKgcBcRSSGFu4hICinc81Rj\nQGMkHaoamQlopK7ULl1DNQMUQCISBYV7lelSmtmja6hKHBTuIiIppHAXEUkhhXuVRdHmrmb8ftoT\nkl0KdxGRuOnnB0TSR32pEgeFu0gVKOAlagp3EZEUUrjnScuAorQ8jrHSfpAsKyvczWypma01sy4z\nu6XA/JPM7Ndm9rSZPWdmy8OvajrppwJEsqcmLpBtZnXAXcAyYDGwwswWDyn2BeB+dz8PuB7432FX\nNK00UjF7dMQlDuWcuV8EdLn76+7eB9wHXDukjANNwe1pwKbwqiiSPgp4iVp9GWXmAhvy7ncDFw8p\nczvwCzP7D8AU4MpQaiciIqMSVofqCuBb7j4PWA5818yGrdvMbjKzTjPr7OnpCWnT4anKNVQjaHOv\nxuOoRdoLkmXlhPtGoC3v/rxgWr6PAfcDuPvvgIlAy9AVufvd7t7h7h2tra2jq3HKqD9VJHvi6Gsr\nJ9yfBBaZ2QIzayDXYbpySJm3gCsAzOwMcuFee6fmNUj9qRmkgy4xKBnu7n4UuBl4GHiZ3LdiXjSz\nO8zsmqDYZ4C/MLNngXuBj7i+4ydSlL4lJVErp0MVd18FrBoy7ba82y8B7wi3aiIiMloaoZqnKtdQ\njeInf/WZCdB+kGxTuFeZ8kcke2pihKqIhEut7RIHhXuV6YWeTTruEjWFu4hICinc81Sj/Vtt7tHR\nSF3JskyHey181bjUNzpGqqK+Ky2STHG8dDMd7kn4qtxIVdQ4sWTSe7LEIdPhXgv0Qs8mHXeJmsJd\nRCSFFO55qtHMoRGq0dF+kCzLdLjXwkfjUt/oUIeqSPqoQzViSTizU4dq+piGMEkMMh3utUAv9KzS\ncZdoKdxFRFIo0eE+2kaJYst5kfml2sWLlS+2XH5rSsl1F2h6KdUcM7D9CndQNRt5oti2Gq0kyxId\n7rWqklAtOUJVnaYiqRNHc2yiw320u6fUckPnlzoQw8rbyMuNNa/LDfxKt1PNt5EsvYXp/VrikOhw\nTwO90EUkCgp3kSrQm7pELdHhHnqHqheeX3GH6sB6yuhQLfEgRtWh6oP/lyt1HaoaByAZluhwr1WV\nREqpsupQFUkfjVAtoWY7VEsspw7V2tp23LL0WKV6Eh3uaaAXuohEQeEuUgV6U5eoJTrcw+5QJawO\n1RLLDR6hOrLRjVAdvp1ypK1DVSTLEh3utaqiUNU3OkQkAokO95rtUNUI1YplqZlCX4CSOCQ63EVE\npDCFe7XpNC6TdNglaokO9/B/8tcLzo9yhOpo2txLj1DVT/6CujMk28oKdzNbamZrzazLzG4pUubD\nZvaSmb1oZj8It5pJU36qKH9EJAr1pQqYWR1wF/AeoBt40sxWuvtLeWUWAZ8D3uHuu8xsZlQVHlS3\niJYba4dq/xR1qCZj23HTpRUlDuWcuV8EdLn76+7eB9wHXDukzF8Ad7n7LgB33xZuNUVEpBLlhPtc\nYEPe/e5gWr5TgVPN7LdmttrMlhZakZndZGadZtbZ09MzuhqnjM7hskln7xK1sDpU64FFwOXACuAb\nZtY8tJC73+3uHe7e0draGtKmRURkqHLCfSPQlnd/XjAtXzew0t2PuPsbwKvkwj5Rwvt2RXU7VNVJ\nm1PqW04iaVZOuD8JLDKzBWbWAFwPrBxS5qfkztoxsxZyzTSvh1jPSNTCd43H8oai33pPKB02iUHJ\ncHf3o8DNwMPAy8D97v6imd1hZtcExR4GdpjZS8Cvgf/k7juiqnRYkvA96JHqqCsNiUgxJb8KCeDu\nq4BVQ6bdlnfbgU8Hf1IBnXxnk467RC3RI1RFRKQwhXuesBo5KmktiaJlRa01OdoPkmWZDvda+Gg8\nlm90qEM1mXTUJA6ZDve4zuzG0vE5UvirQ1VEisl0uNcCjVQUkSgo3EWqQG/pEjWFe56wmjkq6lCN\nYhSlWmsAdahKtmU63GuhP1IjVLNHx03ikOlwj69DNZpl1aEqIsVkOtxrgU7iRCQKCneRKlDTjERN\n4Z4ntBGqlfzkbyT9qWquAfUrS7ZlOtxr4eRpLAGks79k0lGTOGQ63GPrUI1oWXWoikgxmQ53EZG0\nUrhXmT6ii0gUFO55wmrlqGyEavjUWpOjZivJskyHey30R2qEavbosEkcMh3uifjJX41QFZFRyHS4\ni4iklcK9yvQRPZt03CVqCvc8YY3srGQt0YxQFdB+kGzLdLjXwtmTrqEaIaW7ZFimw12kGvSeLHHI\ndLgn4+cHdIFsEalcpsO9FugC2dmks3eJmsI9X1VGqIZ/9q0z+hz99LFkWabDvSbOnjRCNTJ6j5Ms\ny3S4i1SDmuIkDpkOd10gW0TSqqxwN7OlZrbWzLrM7JYRyl1nZm5mHeFVMeV0EiciESgZ7mZWB9wF\nLAMWAyvMbHGBclOBTwJPhF3JuFTjGqpR9PnpfD6nlveDmmYkauWcuV8EdLn76+7eB9wHXFug3BeB\nO4FDIdYvUrXQH6lrqEZHrVaSZeWE+1xgQ9797mDaADM7H2hz938OsW4iqaT3ZInDmDtUzWwc8BXg\nM2WUvcnMOs2ss6enZ6ybHrP4RqhGsyF1qIpIMeWE+0agLe/+vGBav6nAWcD/M7M3gUuAlYU6Vd39\nbnfvcPeO1tbW0dc6RXQSJyJRKCfcnwQWmdkCM2sArgdW9s909z3u3uLu7e7eDqwGrnH3zkhqHKHQ\nToR1DdWaUMsjVNU0I1ErGe7ufhS4GXgYeBm4391fNLM7zOyaqCsYpVp4gY2laUUdqiPTm5xkWX05\nhdx9FbBqyLTbipS9fOzVEhGRsdAI1Rrfzkhn9upQFZFiMh3uIiJppXDPU41rqEahljsS41TLe0G9\nJRI1hXuVjaVlRR2qJajZSjJM4S4SM70pSxwU7jVOP/krIqOhcBcRSaFEh/toz1uLLdd/Ijx0fqkO\nymHlgxUVW66ya6wWWr7ECvofR4U7qJqfAyIZqRvBOsOiphmJWqLDvVZVEiq1HEBJp1YrybJEh/to\nz31KLTd0fqkLKwwvP/JylZy0FSraf9ZX6uyv0pPDap5LZuk8NkuPVaon0eE+VokYoTrienVqKiKF\nZTrcRUTSKtHhHnqHapH5FXeolliuog7VAmVLnbEPbD/rHao1/MlGTTMStUSHe60aa3hLOLRrJcsS\nHe4126FqIy9XUYdqgbLqUE02fQtS4pDocBcRkcIyHe5x/XriWLajnx8QkdFIdLiHP0LVC86vfITq\nyMtVNkJ1eOGSHaoaoZpbZy2/96lpRiKW6HCvVZWNUK3lBEo27VnJskyGe7JGqA4vrA7VZCv1fBIJ\nQybDXUQk7TId7sn4+QFdIFtEKpfocB9ttkX+k78llot+hKpXvJ3cctWTtRGqIlFLdLjXqkpCRfmT\nTWp1l6hlMtyjH6FqIy6nEaq1te24aYSqxCGT4S4iknaZDve4WkTGsh2NUBWR0Uh4uI8u3EotlfRr\nqGqEarBOvfdJhiU83GuTMqU21PLoX10gW6KWyXBP1gjVQsurQzXJsvRYpXoyGe4iImlXVrib2VIz\nW2tmXWZ2S4H5nzazl8zsOTN7xMzmh1/V8MXVITmW7egC2SIyGiXD3czqgLuAZcBiYIWZLR5S7Gmg\nw93PAX4E/F3YFS0kDSNUSwZ0gfm6hmqZ69R7n2RYOWfuFwFd7v66u/cB9wHX5hdw91+7+4Hg7mpg\nXrjVTBiFSk2o5cOgdneJWjnhPhfYkHe/O5hWzMeAhwrNMLObzKzTzDp7enrKr2XIou5Q7Z8QRodq\nocLqUE02fVFG4hBqh6qZ/TugA/hyofnufre7d7h7R2tra5ibFhGRPPVllNkItOXdnxdMG8TMrgRu\nBd7l7ofDqV60kjBCdaSGY3Woikgx5Zy5PwksMrMFZtYAXA+szC9gZucB/we4xt23hV/NwkK/hmr/\nT+UWmV72+iq4hmqlo2Vzy5caoaqf/AV1qEq2lQx3dz8K3Aw8DLwM3O/uL5rZHWZ2TVDsy0Aj8ICZ\nPWNmK4usLhNqeWRkltTycVC7u0StnGYZ3H0VsGrItNvybl8Zcr0iFf0I1RB/8rfg8upQTbZsPVqp\nDo1QFRFJIYW7iEgKJTrcxzpCdejy4Y1QHfyTv8W2U2heqXXnlvFB/4fPL2/d5WwrLupQFQlXosO9\nVoUZKgqodCrVjyMyVpkM9+ivoTrycrqGam1tO276pozEIZPhLiKSdgp3EZEUSnS4j3b4vRe4lX9v\n7NdQHbLcSB2qpdZdYHapDtWRlh2xfGXFQxVNh6o6LCS7Eh3utSrUDtUaHmVZ65TtkmWZDPdEdagW\nWIc6VJPNUKeqRC+T4S4iknYKdxGRFEp0uI/1J3+HjxwN5yd/h3aoDl2+shGqxa+hqhGq8a9TJCkS\nHe61qpJO0JK/566EGjXtO8myTIZ7Lf3kb8m6aIRq6pQ6biJhyGS4i4ikncJdRCSFMh3usV0gewwb\nGmlZjcAUkWISHe5hZ1tYqwuzQ7XgMqV+dkDfEwG0HyTbEh3uWaeOuZHV6gcbHTWJg8JdRCSFFO4i\nIimU6XCP62P7WDYz0rLqUBWRYhId7mF3mIWVlZWsZzTb1KjW8mg3SJYlOtyzTh2qJdTou5wOm8RB\n4S4ikkIKdxGRFMp0uMfVITmW7WiEqoiMRrLDPfRsC2eFla0l/B5VRX6O9oNkWbLDPePUoTqyWv1g\no8MmcSgr3M1sqZmtNbMuM7ulwPwJZvbDYP4TZtYedkVFRKR8JcPdzOqAu4BlwGJghZktHlLsY8Au\nd18IfBW4M+yKiohI+co5c78I6HL31929D7gPuHZImWuBbwe3fwRcYWozEBGpGiv1jQsz+yCw1N0/\nHtz/M+Bid785r8wLQZnu4P5rQZntxdbb0dHhnZ2dFVf4/ic38NkfPwfA3OZJTG6oq3gd67btK7h8\n37HjrN9xAIBFMxsHpq/feYC+o8eHTe+37/BRNu85NHB/Qv04DueVP3bceX37/oH54wyOl9ke3DSx\nnt5DRwfuL5rZyP7DR9kUbC+/Pv2Pa/4Jk2moG0dXz76K2p1nN01k6sT6gfXEadL4Og4eORbqOqdP\nHs+uA0dCXWcY7vlIB9987A1+27Wj2lWRKvmHFefxvnNPHNWyZrbG3TtKlYu1Q9XMbjKzTjPr7Onp\nGdU6mieP552ntgJwbts0Fs1qrPjvskUtBZc/88QmADrmTx80/YrTZwLQ0jih4PrOO6kZYGC9V5yR\nK/+exbNYNKuR0+dMHfQYrjpzNkvacsssO2s2AIvn5LZ99txpA+sDuDRY56TxdZw4bSKLZjWyJJj/\n7tNaB9Xj/GD6mSc2sWhWI1ctnj1ou20zJo3YmXf+/GYWzWrknHnTBqadNGNyiSMy3NQJ9QBMaajj\nhCkNA9MLvRH31+fy01oHbRdyb4KFnNvWPGzaRe0zBt1fPKeJt51yAgtnNhZ8Qx6qMahzv/knDH/c\nDfXlv1w+3DGv4PTTZ0/lvLbp/M37zhyY1j5kWze/eyE3vG3+oGkLZzYO2pelvP2UEwbtpzPmNNFQ\nP27Yfup345DtjWTqxPpB/+c2TxqYd8uy0wc9Zy45+Q/b63995Ot/7Ncu+UPQNU6oH3Y8xtflngwj\n7YPb37eYzy8/nemTx/ONGzp4z+JZAMybPmlQuSVtzQN173dh+3S+8+cXDWTAGXOaBtX95NYpAFx3\nfuHj2jx5PGcEr+F+V58zh+Vnzx50fFsaG5g+eTzvOq216OMISzln7m8Dbnf3q4L7nwNw9/+WV+bh\noMzvzKwe2AK0+ggrH+2Zu4hIloV55v4ksMjMFphZA3A9sHJImZXAjcHtDwKPjhTsIiISrfpSBdz9\nqJndDDwM1AH3uPuLZnYH0OnuK4FvAt81sy5gJ7k3ABERqZKS4Q7g7quAVUOm3ZZ3+xDwoXCrJiIi\no6URqiIiKaRwFxFJIYW7iEgKKdxFRFJI4S4ikkIlBzFFtmGzHmD9KBdvAYr+tEEVqV6VqdV6Qe3W\nTfWqTBrrNd/dSw5xrVq4j4WZdZYzQituqldlarVeULt1U70qk+V6qVlGRCSFFO4iIimU1HC/u9oV\nKEL1qkyt1gtqt26qV2UyW69EtrmLiMjIknrmLiIiI0hcuJe6WHfE224zs1+b2Utm9qKZfTKYfruZ\nbTSzZ4K/5XnLfC6o61ozuyrCur1pZs8H2+8Mps0ws1+a2brg//RgupnZ3wf1es7Mzo+oTqfl7ZNn\nzKzXzD5Vjf1lZveY2bbgqmH90yreP2Z2Y1B+nZndWGhbIdTry2b2SrDtB82sOZjebmYH8/bb1/OW\nuSA4/l1B3cd0mcsi9ar4uIX9ei1Srx/m1elNM3smmB7n/iqWDdV7jrl7Yv7I/eTwa8DJQAPwLLA4\nxu3PAc4Pbk8FXiV30fDbgb8uUH5xUMcJwIKg7nUR1e1NoGXItL8Dbglu3wLcGdxeDjwEGHAJ8ERM\nx24LML8a+wt4J3A+8MJo9w8wA3g9+D89uD09gnq9F6gPbt+ZV6/2/HJD1vP7oK4W1H1ZBPWq6LhF\n8XotVK8h8/8HcFsV9lexbKjacyxpZ+7lXKw7Mu6+2d2fCm7vBV4G5o6wyLXAfe5+2N3fALrIPYa4\n5F+4/NvA+/Omf8dzVgPNZjYn4rpcAbzm7iMNXItsf7n7v5K71sDQ7VWyf64CfunuO919F/BLYGnY\n9XL3X7h7/4VzVwOFr+0WCOrW5O6rPZcQ38l7LKHVawTFjlvor9eR6hWcfX8YuHekdUS0v4plQ9We\nY0kL97nAhrz73YwcrpExs3bgPOCJYNLNwcere/o/ehFvfR34hZmtMbObgmmz3H1zcHsLMKsK9ep3\nPYNfdNXeX1D5/qnGfvtzcmd4/RaY2dNm9i9mdlkwbW5QlzjqVclxi3t/XQZsdfd1edNi319DsqFq\nz7GkhXtNMLNG4MfAp9y9F/gacAqwBNhM7qNh3C519/OBZcAnzOyd+TODM5SqfDXKcpdnvAZ4IJhU\nC/trkGrun2LM7FbgKPD9YNJm4CR3Pw/4NPADM2sqtnwEau64DbGCwScQse+vAtkwIO7nWNLCfSPQ\nlnd/XjAtNmY2ntzB+767/wTA3be6+zF3Pw58gz80JcRWX3ffGPzfBjwY1GFrf3NL8H9b3PUKLAOe\ncvetQR2rvr8Cle6f2OpnZh8B/i3wp0EoEDR77AhuryHXnn1qUIf8pptI6jWK4xbn/qoHPgD8MK++\nse6vQtlAFZ9jSQv3ci7WHZmgTe+bwMvu/pW86fnt1X8E9PfkrwSuN7MJZrYAWESuIyfsek0xs6n9\nt8l1yL3A4AuX3wj837x63RD02F8C7Mn76BiFQWdU1d5feSrdPw8D7zWz6UGTxHuDaaEys6XAZ4Fr\n3P1A3vRWM6sLbp9Mbv+8HtSt18wuCZ6jN+Q9ljDrVelxi/P1eiXwirsPNLfEub+KZQPVfI6NpYe4\nGn/keplfJfcufGvM276U3Meq54Bngr/lwHeB54PpK4E5ecvcGtR1LWPskR+hXieT+ybCs8CL/fsF\nOAF4BFgH/AqYEUw34K6gXs8DHRHusynADmBa3rTY9xe5N5fNwBFy7ZgfG83+IdcG3hX8fTSienWR\na3ftf459PSh7XXB8nwGeAt6Xt54OcmH7GvC/CAYohlyvio9b2K/XQvUKpn8L+PdDysa5v4plQ9We\nYxqhKiKSQklrlhERkTIo3EVEUkjhLiKSQgp3EZEUUriLiKSQwl1EJIUU7iIiKaRwFxFJof8PZD/9\nqO95nD8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bc8087828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(rList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also begins to progress through the environment for longer than chance around the 750 mark as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f5bd050ca58>]"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXl8FFW2x38nhICA7AFZlACCiCig\nERAUXBBFR8XRcVdcUUcdZxzHwaeOzrx5b3BH38w4KqLoKO4OrqgggqAgYd8hECCBEBIICSF7ct8f\nXdWp7tS+d/f5fj75pLu66t7T1VW/e+rcc+8lIQQYhmGY5CUtaAMYhmEYb2GhZxiGSXJY6BmGYZIc\nFnqGYZgkh4WeYRgmyWGhZxiGSXJY6BmGYZIcFnqGYZgkh4WeYRgmyUkP2gAA6Nq1q8jKygraDIZh\nmIRixYoVJUKITKP9QiH0WVlZyMnJCdoMhmGYhIKIdpnZj0M3DMMwSQ4LPcMwTJLDQs8wDJPkGAo9\nEc0kov1EtF6xrTMRfUtE26T/naTtREQvElEuEa0lolO9NJ5hGIYxxoxH/waAC+O2TQUwXwgxAMB8\n6T0ATAQwQPqbAuAld8xkGIZh7GIo9EKIRQAOxm2+DMAs6fUsAJMU298UEZYC6EhEPdwylmEYhrGO\n3Rh9dyFEofR6H4Du0uteAPIV+xVI25pBRFOIKIeIcoqLi22awTAMwxjhuDNWRNYitLweoRDiFSFE\nthAiOzPTMN+f8ZmcnQexZd/hoM1whTmr96Cipj5oMwAAmwrL8cmqAuwrq8b8TUUAgJKKGsxdv89y\nWVv2HcYD763G/sPV+HrDPlTVNuDjlQVQLg86++fdyCs5AiEEPsjJR019g2G5P+cdxGuL85B/sFJz\nnzeW5OH9nHz8uL0E24srottX5x/CzMV5eOLTDQCAbUWH8XNefEBAncPVdZizeg9+2n4AP+aW4M63\ncjTt/TG3BDuKK/DS99vx2Zq9qvssyS3BzpIj0fezf96NnSVHUFRejXkbi2L2/WzNXhw8Uov3l+ej\noVFgy77DWL5T3e7vNhdh76EqAMDna/eirLJO93tV1zXgwQ/WYIfiPPmN3QFTRUTUQwhRKIVm9kvb\n9wA4VrFfb2kbk2Bc+a+fAAA7p10csCXOWFdQhvvfXY1Lh/bEi9cOD9oc3PrGchSWVeOY9q2xr7wa\nO6ddjJte+xkbC8ux4c8XoG0r87fkBdMXAQA+XhW5xa44tTc+WlmAHh2Owhn9u+DgkVo8/PE6DDu2\nI+4c2w9/+HAt8kqO4KELB+mWe9XLkd/+r19sRN7fmv/+9Q2NeOKzjTHb5Otk0j+WRLddN/I4THh+\nUcznekz9aB2+WFcYs236vG34o4q9181YFvP+kqE9m+1zvbTPzmkXo6SiBg9/vA6n9emEovJqFJRW\nRW3KKzmC+2avih5X19iIRz5Zr2n3rW/koGu7DHx89xjc+84qjBuYiVm3jtD8Xn/7chM+XFGAD1cU\nBHY/2fXoPwUwWXo9GcAcxfabpOybUQDKFCEehvGdI7URT35fWXXAlkQolOzYV95kj+w5NwjLD8Yx\n7CuPeJlHpKeXhsZIeev3lKG8OuJ1llTUmC5PyxyzVlbVGj89KNlbVtVsW8lh8/bq0Sidiw17y1BQ\nGltPvJ2HDDx0ACipqEW19LQhe/daFJW78x2cYOg+ENFsAGcD6EpEBQAeBzANwPtEdBuAXQCuknb/\nEsBFAHIBVAK4xQObGcY0FLQBJnAm7xbr8rOyJCeRzqWh0AshrtX46DyVfQWAe5waxTCpiFeNkgBA\nATR5ZLFKNeFU09KCUu2+A82ydT+L/dSs3fJxRvtbPQ9ewCNjmZQg/mYOE8Jl11BE/wf7nb3yeM98\ncoE3BScxLPQMwzBJDgs9wyQZ8ZEC5RODnz6+GyELP+LgTsNaQYTFrMJCzyQ1JKlNInScURiCuUmK\n3u9vN8Rl9poKw8/KQs8wAeN2GySavQiGRPB03SAMQm4ECz3DeEh1XQNy95sbYexUL3SFNQAxcqMz\n2MsOZbc7weOJ/PbBjYZVwkLPJDWytxWUc/uHD9di/HOLUFZlPAjHKdF0v2bbFa8TIIQVNG49ifxm\n9iqMf24hqusaXSnPCSz0TFIT9FP10h0HAAA1ddZGibpJUOJuVTC9NNOLJwOj8/rT9shvX9fAQs8w\nKY/bQpxUTnsAX8atmHuYfgcWeiYl8DoeaxelXU4F5kBFbWzZitdBP9mYwSsb9xjMRROP25dKGLKp\nWOiZpCYE95ghboUVNpuYVtqVDlKP2kyvmuIx075Tnco4pG2/J7DQM4xH1DU0+tIJG08CtG2+sya/\nzPS+Vue6SQRY6JmUIIhb8v53V6G2Xr8jzguvMizy48rIWOdFeIb822mFZuSwXBgaXhZ6Jsnx5jar\nqKk3jPt/uc76qlFuojQvDHHisOK0sU2EM8tCzzAW2VdWjSGPf41Xf9gRtCnWCbOL7DOpdCpY6BnG\nInIWx1c21nmNR0Dfo6xvaIyujhQW/IxN17qUg27FZrc99DA8TLHQMymBJ7Fwt/PfVco7/pGvcNe/\nV7hbUQLxxdpCrNpdGrQZCQ8LPZPUeOFN+e2hfbOxyNZxSi82BE6lbVbsCrfQG64w5Y8ZurDQM0yA\n+D2QS64ta+oX+OvnG/HgB2uQNfULX20IAq9Pc9bUL/DHD9d6W4kDWOiZlCBcUe5YgrJtxuI8fLii\nIKDaVfC70XO5vvdy8mPLl/6HIeOJhZ5Jary8xcLceHhJmEeUTnkzByc+NrfZditaa3rAlNmFR8xX\n7RnpQRvAMImGVzeul/oZAqfSGBeM1OrP8LJxSoRzyx494wt//XwjBj7yVXAGhDTt5sv1+wxHz9pF\nyzy/+gUsC6CHdkVHsao007fPyrFXZgI907FHz/jCjMV5gdQbhvioHjMDOi+pRtMCNM3FeV95tSd1\nNk2R4EnxlmCPnmHsEoY7WAU9jz2s0zUHgQDw/vL8ZtvdXus2DKechZ5JCTy511y4g/3SgJC2Sb6h\n9VNNm7vZYjlC8TryPxEWQWehZ5Ia+RZ006uyGw7aXqyyUHSMcLhjpGyfVmleeJj3v7vK/UI9wAtR\ntnM5PPP1Flz2jyWu26IFCz3D+AEB/1m1R2V7+L1BM8xZ3XxhjzCtGdtUh39xFL26/r4gF2vyD/lm\nCws9w/iBj+61bozehfIf+896F0oJnjDEzv2ChZ5JavSyLZxitUQ1YfEtRu9iyOIDj0bThuXZRu8h\nS/kbJlI7wULPMBYJiyAZoeXZh1WgtOzKKzmCc5/5HgcqahyULa/2FPvrufFbJsL14Ejoieh3RLSB\niNYT0Wwiak1EfYloGRHlEtF7RJThlrEMk7CQcTjeLQEO+9gBq7y9bDd2lBzB3A325/+XBd7NJ7tE\nSlW1LfRE1AvAbwBkCyGGAGgB4BoATwJ4XghxPIBSALe5YSjD2CE0qW+ao1Q9qEoqdMPecl/qSwQS\naRSrFzgN3aQDOIqI0gG0AVAI4FwAH0qfzwIwyWEdDBNKwi6aj3zS1GmaZE6+ZdRy3u2If8h/ck1s\nC70QYg+AZwDsRkTgywCsAHBICFEv7VYAoJdTIxnGKe7m0ce+r65rwJ1v5WDXgSPuVeIRDULgt++t\n9qWuIBuX+LCKp7ZoFJ4UUyAQUScAlwHoC6AngLYALrRw/BQiyiGinOLiYrtmMIwuXt5kske4eFsJ\nvt5QhL98tlHHEP0yvKF52XsPVeGzNc1z3pMNzQnd3IzRO7TFT5yEbsYDyBNCFAsh6gB8DGAMgI5S\nKAcAegNQGSUCCCFeEUJkCyGyMzMzHZjBMMmBl4Lw4AdrPK/DKV7aFubv7QdOhH43gFFE1IYi3fzn\nAdgIYAGAK6V9JgOY48xEhnGOHze6bhUBC01dgwiDGY6w8htq7RqaznmfcRKjX4ZIp+tKAOuksl4B\n8EcADxBRLoAuAF5zwU6GCQ1uioVZ8SqpqMHjc9ajrsGbueuNKCqvxp8/24B6l+v/Ym0hPpVCSW6G\n2cykPgqhXqdeeqr6pGYa+xpa4B+O5qMXQjwO4PG4zTsAjHBSLsMkAvFa4qWv+PinG/DF2kKM6tcF\nE0/uYeoYN59ipn60Fgu2FOOcE7qZPsaMcN/zzkoAwKVDexra60ZD4EWfSBg6W43gkbGMYz7IycdP\n2w8EbYYuytt7XUEZ3lji/oIfuhKi1RlrUndkT9qMqOgOmLKh/qvzD2HBlkjChJsy+WNuiaX93Qjd\nuEuYfHZ9eIUpxjF/+HAtAGDntIsDtqQ5app3yd8XAwBuHtPXszqaYUYTdPaRIyZmRr26PanZJMV0\num6OBr1uxjLXyoon3kz5LcfoGYYxRbzWmtU+JxopC2yahThB4vibwIvztxmGVayESP6+INfkntaE\nv7KuQb/e77ahRJ6TJ0Q/AAs9kxJ4MS+JG0WazsWW/qc5dEjDmmb43LdbsX5P8ykblFix/cX520zu\nae2E/P275g2I8id55put0VTWMMGhGyYpqa1vxKs/7MDo/l08r8uUp2liUjM9Ghqte/RBoxUmmbN6\nDw5X16t+ZsTPeQdtHedWA1et8Oi1yqys1ff6g4CFnklKXl+Sh6e/3oJLh/YM2pQIwpnYNEoHO9V5\np1knbujl/e/an4Lhqpd/csECJc1PaOI0pebh0A2TlByRvKpPXRruv2zHAeTstOdN6qEMKW0oLMOC\nLfs19ov8TySPPky4ddrCGvoygj16hjHB1a8sBeAss8hIbK57dZlmHbJH38JhkD5Rhcopqqt7aZwL\nq41CIsz/zx49kxIYCdzGveVYtsPDsQAOtSAaupHebyrU77gE1DugrQp9Ii2uoY/572E6i8qmJUHA\nHj3DALjoxR8AhHMsAAA0Rqe8jUj9xBd+8KXebzYWxW6woG7hcnS9Mya+ZPl9mBY7YY+eYWyy0YRX\nHUXjnt+877C5w10K3VjFbnYMoO4Za/VBmGFN/iHbx1ppoXQHFpsoJzzy3gQLPZMSuOldaQmBl2GO\nJo/eWTlBipAQAre8vtz28R+sKHDRGuckUoiHhZ5hHGI2j94JjdGRseaPURMYP2PudkcQ+4WAcCW8\nFF+GVignSFjomZRntaOQgD8CJnv0ZtIrZXM8WXg8FP5pODBqNMPUsLHQM8lJ3F2md9OVVdV5bAwc\nP7/bmevGjre6o7jC+kFa9ce9D5Hu6aJ32pTXUUFplee2uAULPZPyWA1nxA/t9yO7xMrI2GjWhw1l\nPffZhdYP0iC++iBTNb2o+vcm57QJQwPHQs8kJ+HK7XMtUGtmmt2i8mrklRxR/awxTPGEgHHrVCTC\n1MecR88kJ/Ghm4DMcBszMfLH5mwAAHQ4qqX79VvJo48/1lVLwk/THPjBw0LP2KauoRFbi8zlgaca\nZZXN4/5+O9O+9D1YIBkeJlS/QhiU3AAWesY2f/tyM2Z6sCSfK3gYujFTtDzSNkpIwgTJILZ20Pra\nxYdrmm1LhLlrrMIxesY2awqcpSV6SrOsG38Vbs+h5hkZYVjcOtABUx7X7tbT5YEjta6UEyZY6JmU\nx2350ZoczUlbkwgdfkZ42dauyT+ECc8vsnSMljnmV6dKHFjomeTEp8fv6roG7CuLffzfeUA948Us\nB4/Uory6DuXVTTF22RsuKK2KrjZlBz+fbPz0jLcb5P9X1tqfs8eI+PmAKmq8q8suHKNnkhMPBU3Z\nhtz51gos3FocqVLPHJhve07972/RumWaqhf/67dX4t5zjjdvbIBcI83h7wdVBot2L93hzqIxapdV\n/JTRG/ZamOzOJ9ijZ5g4DlXWoqbe3Lqfssgbsb+8eaefHtV1jZri9cM2c3W6QXWcDU7aTy8fJqrr\nGr0r3CZhmsufhZ5JCazccsP+8q2jWRbVuOTvi/HNhiLjHU1Q7yR0Y3H/R/+z3nZdfmK2YQ6CMMg9\nCz3DqPDjdvurTWl1nG5xKSvESYw+SNXxMuvmiI24eJg8bq9hoWdSA6MAugWMMmC8TiN04tEHiZe6\n+o8F270rPIbEPPcs9IwuNfUNaExQYUkmlI2Lo6wbnc/i4/FWj09mzJybeMI0BQILPaPLCY/OxaNz\nEiNOaxuLd6JR9owXOe/Kp4T6Rvsdj1rhisKyKgx6bK7tck3V7Wnp3jFn9R4MemwuttkMvYXhe7PQ\nM4a8s2x30CY4RvdmU3xoJm4bdGi3ocF9A3YdqHS9zHjCFhM3Y40QAvM2Rda5tbRGcMjgPHqGUdD3\n4S91P9+is5i3lzqmfEqo8zHrJtXp+/CXGNCtXfR9yNoq0zjy6ImoIxF9SESbiWgTEZ1BRJ2J6Fsi\n2ib97+SWsQwTNKt2lwYy1b1bHbxOhcqJV56gGolt++2tuiWfqmSI0b8AYK4QYhCAoQA2AZgKYL4Q\nYgCA+dJ7hgkUP8IGcgPw0EdrPa/LLrsPqodo/BCjMHjDThrp+99d7Z4hPmNb6ImoA4CxAF4DACFE\nrRDiEIDLAMySdpsFYJJTIxnGS8LiLeuRDJOahYEgGpsQtG+OPPq+AIoBvE5Eq4hoBhG1BdBdCFEo\n7bMPQHenRjKMU/YfrkHW1C8wd/0+R+Xo3bQLtxbjz59tcFS+GRJW8sOgeApOeeIby8eE4anEDk6E\nPh3AqQBeEkIMB3AEcWEaEXleVl+UhWgKEeUQUU5xsX9zdzCpSWVtJA969s/eZhC9vmSnp+V7hdnF\nNhJU51IeJ0JfAKBACLFMev8hIsJfREQ9AED6v1/tYCHEK0KIbCFEdmZmpgMzGMY8LFTB4fWIYTMk\n4eJRprAt9EKIfQDyiegEadN5ADYC+BTAZGnbZABzHFnIMCFCiMR9fA+aMJy3MNgQBE6zbu4D8DYR\nrQUwDMD/ApgG4Hwi2gZgvPSeSUFW7S5F1tQvUFjWfFm9RCVn50FcMN3aSkZus19lnVPGH8w+lYx/\nbqHHlljD0YApIcRqANkqH53npFwmOXhr6S4AwJJc+zNB+oEVL+/jVXu8MyTJSSVnOtdm7r1X8BQI\njG3shDvzNfK4rXDL6z/jqbmbHZdjxKdr9npeRyoRhikQwhajv/yfS3yph4WesY2d2/bztYXGOxmw\nYEsx/vm999PSPvLxOs/rCAtmBTAEWu2IsNm/avchX+phoWdiWL+nDGc99Z2tY5/+2nsv2ylh8CoT\nmdoGBzNnumhHIhGGhwgWeiaG6fO2If+gvc5T/xZ/MCZVRcVrCkrth97C0MY6Dd3Y+Q4h+Nos9Iy/\nhCGXWoa9+yYe/cTcmgMzfsjz2JLkIwzXGQs9k5RYcdy07sPgb0//MLue7cEjtbbrCEMjH4Tmmh11\n7CU8Hz2TMLy3fDf2HKo2tW/wksI0g38U/OWzjTitj/8zt7PQMwnDHz9yNwtGiPCl2zHe4jhG77D+\nmUvyMHOJ/+EvDt0wnhNEjJL1O3ywQx8cLPRMHMlxO7rxLbixcJcQ9EmGwoYgYKFnbHGoshYrdpWa\n2lfZGZVIN1oCmcowurDQM3GY82Of/nqLx3Y4Q+tb1Dc0yTcLub+EIesmiDz6MMBCz9ii0eYF73bn\npxAC0+dtxfbi2EmkrJgnT77GeEuiimQywELPeI6XnbGllXWYPm8brn91mfHOGizcyiucMckNC32K\ns/tAJV5eqJy6wFu3y23NlxuRmvoGc/srvl8YRiymCv/8PjcEgZtgCMN1xkKf4lz/2lL87avNjkY8\nGhGGkYFMsDw1dwv2Hkr8BWjC0M9gBxb6FOdITcQTDoPXITNzcR72lZkbAavFln3qQ/rtfM33lnu7\noHiq0Gi3YyfBCYOjw0LPhIqC0kr85fONuP3N5Y7K+WZjkeE+ZmXH7RG5DOM3LPRMFCEE5m3a70m5\neryfk4890mN9g+T1lVfVG5a7YtdBLNlubZnCZXkHLe0fFkLgFDomRR36UMBz3TBRNhaW+15nZW09\nHvpwLfp0aYOFfzjH0rFXvPSTR1aFjxBF1lIbO/PRh+DHY4+eiVJTb371ICseplqM8uOVBThQURP1\n8ooP15gv0CVCcP8xKQDH6JnAsettOBHJPYeq8MD7a3D32yvtF+IDYfDEZEKgFY5J1IyVZICFnvGd\nWunJoai8KbOmstZcHrwWYfCavCREbY4mYWoYtahP0Y4CFnrGFlZ0NV4AtA7ddeCIfYNsYORhJnvj\nkYrUWghPqmGnmQhDA8idsYwpcvcfRnVdI4b06uBZHRU19fh+C09HoEYitDmGeha83qUsLPSMKcY/\ntwgAsHPaxZaPNesZv7JoB+as3mu5fC8JgzeWLPCZDA4O3TC+UlLRlF0Tr6GFDkfDuo0QIjSx8bDY\noUcCmOiY77e4P87ED1joUxy/FwV5fcnO0IQhEkE8k4lkON+llXWWjwlDXw8LfYrjR2hCq4780sq4\nHT03xRJEFBqTKmqMRwoHDYe5wgsLPRMYQri3LqsXIhMm4cor8TcjyQ7GfbHhOZ9+EobriIWe8Ryz\nj65hFIIw3KTJgnIZR8ZfWOgZz2meR68u/MrdwiL64bAiMTBqE1N1sFJSxOiJqAURrSKiz6X3fYlo\nGRHlEtF7RJTh3EzGDRobBTYFMHGZHql566cm9Q3OBisx9nHDo78fwCbF+ycBPC+EOB5AKYDbXKiD\ncYFXftiBiS/8gJW7S4M2JRSYicpw5MY8Rk9hdSnq0YcBR0JPRL0BXAxghvSeAJwL4ENpl1kAJjmp\ng3GPdQVlAIA9pc6WdCupqHG0ApQX67aWVtahrMp66psRYQkhJQJFZfozkKaqRx+Gfh6nHv10AA8B\nkH/BLgAOCSHkXLACAL3UDiSiKUSUQ0Q5xcU87N1P1EKGVi7F7L/Ow3ebzQ8cMd8Z64zT/2eewxIY\nJ4x9eoHu56kaozfCj4bAttAT0S8A7BdCrLBzvBDiFSFEthAiOzMz064ZjKt4c8HpXcjC5H5msDph\nlRlvPQTOWNLQwEIfGE7muhkD4FIiughAawDtAbwAoCMRpUtefW8Ae5ybybiBnrDpCRqRu4Jnpiyt\nzBwmcUnV0I0RQng/aZ1tj14I8bAQorcQIgvANQC+E0JcD2ABgCul3SYDmOPYSsYVZIFVE1G7jYAt\nOzTKZn8vueHfNzi8yKP/I4AHiCgXkZj9ax7UwdggKvQJ6CzvPlBpvJNFOOvGX/hcBocr0xQLIb4H\n8L30egeAEW6Uy3iDqs57GLrRXYtW6dErXjc0ClTXNaB1yxYAgN0H3Rd6xl/CkH0SRvw4KzwylvGc\nR/+zPua9mRt+z6EqDHpsrlcmmYbTK92D+2KDgxceSSF04/A+2hFbryKnXu1zIdAogPpG9zvyBPQb\nHU4HdBc+m+pErkFv46ks9ClI0DF6Kzf8fbNX4fO1hab3X7BlP845oZt5W3SMqalv5Liyi3DoRp1l\neQcx5viuntbBoZsUQu8+C+oejKlXxQgrIg8A31sYyAWYmVqXcQvWeXV+2FbieR0s9CnIXf9eGX0t\nr5jz+dq9rtyI0+dtNdxHq54gdKD/f32p+/l976zU/ZwxT6r2dywwWPC+wYOwZDws9AwA4D+r3RnX\nNnNxnom9wnHDmwklGN2kjHnYo1fHj3FkLPQphN595slNqOW58yCplIT7ttVpDPNcN0zi4XeMvkGj\nUK2q3LAhDIs8MOr4IWiJiBcZZfGw0KcU3qdXllfX46t1kQ5ULcmN8egVb/yO4bLs+MvTX28J2oRQ\n4sdkbyz0DAB3U99mLonE6bVKZM+OYZrwYy1dFnomilshleU7SzHu6QXa2TUaMXqesZJJRbRCnG7C\nQp9C+Bmj33WgUrvhCEnQRPCsuUwI8CN0wyNjGQDehFM+W7NXdXtsjN7dOt/4cSfyTU6AxiEkJgxw\njJ5xFd30St+s8D6fer7J0bEs80wYGNj9aM/rYKFPAB77z3o8FjcDpNvk7q/AX7/Y5GkdMrETmQUn\nt+zRM2EgO6uT53Ww0CcAby3dhbeW7vK8njX5hzyvA9D26P1OgWehZ8KAH0kILPQB0NgocNXLP+G7\nzUW+1huW2QNveG1Z05sgTQrH6WAYz2GhD4Cqugb8nHcQ97y9KmhTAuFwdX3QJgDgIflM6sBCHyB+\nhyrCqGv3zg6usePQDRMG/NABFvoUIoy69nPewehrv4dLhfB0MIwnsNAzlqjVW+g7wQhLnwXDeA0L\nfQAEtz6rc1btLnWhlHDAOs+EAT+eZHlkbADInqT8A6/OP4T5m4rw+wkn+GbD377ahLziI5aPS0vz\n7rJcU1DmWdlqcIyeSRVY6ANEnjt90j+WAIDnQq8MVby8cIetMpJp2jHOumFShaQM3Qgh8NTczcgr\nse6x+kEi64vZ6QUSAY7RM6lCUgp9QWkV/vn9dtz6xvKgTVElkfXlpe+3B22Ca7BHz4QBP1ZFS0qh\nl2Ovskf/4YoCLNtxIEiTYmGBCQXjn1sYtAkM4wspEaN/8IM1AICd0y4O2JIIjXGdsQzDMF6SlB59\n2EMjUfPilN5MzFgIgZe+34795dXW6w35eWGYVIRHxtok7HqmJehmhHhT4WE8OXcz7rMxdUBYVnZi\nGMZfEl7otxdX4JNVBUGbYQknciuvRnOkNhwTgzEM4wweMGWC859biEYBXD68d3Rb2NPmNBfNDqhe\nhmGSG9sePREdS0QLiGgjEW0govul7Z2J6Fsi2ib993T5FLUUObf1bMHm/diy77DuPh+tKDCMm28t\nOoz5m4qajYyVCXsDxTBMYuIkdFMP4PdCiMEARgG4h4gGA5gKYL4QYgCA+dL7hOaWN5bjgumLND8/\nUFGD33+wBrcY5O1PeH4RbpuVE22I4vNn2aNnmNQj1J2xQohCIcRK6fVhAJsA9AJwGYBZ0m6zAExy\naqR12/ytr156rNh/uMbU/rJ9ZVV1qtvNsHFvufmdGYZJaVzpjCWiLADDASwD0F0IUSh9tA9Ad41j\nphBRDhHlFBcXO7YhNuwRjOtqtmHWyn6xkhXDozoZhjGLY6EnonYAPgLwWyFEjJspIuqrKklCiFeE\nENlCiOzMzEynZkSzUQAgd3+F4/KEEPhszV5U1zU4Lqt52U2vC0orVbcvyS1BYVmV7Tr2H67Gwq2x\nDSinVzJMauJI6ImoJSIi/7YQ4mNpcxER9ZA+7wHAl1mwGhQqede/Vzoub+HWYtw3exWe/3ar47IA\noFgR1lFOj3v209+r7n/9jGXoi0j2AAAaFklEQVS4cPoPzbabFeurX16KyTN/tmYkwzABEOK5bijS\nk/gagE1CiOcUH30KYLL0ejKAOfbNM0+jSwsfbSs6jNz9h1FYFsmgOVRZZ3BEc+ZtLEJNfeRJ4OCR\nWiyNm2dH6bnX68Rg4mP4VlCbuZM7YxkmNXHi0Y8BcCOAc4lotfR3EYBpAM4nom0AxkvvPafBJRU7\n//lFGP/cIlTWRoT6qIwWhscoq87ZeRC3v5mDaV9tBgBc9+pSXPPKUrQwsWCHma9AJlp/ZSqosu+C\ndZ5hUhPbA6aEEIuh/cxxnt1y7dLgcu+k/MW2F8fG+3/MLQEADDuuI8qr6nGoqhYdj8qIHENAqfQE\n8O+lu/CnXwzGZkl0lfZpD5hy5ztU1DQ9CTQKoAXPnsYwocWP9MqEHxkr0+iy0MsO+A/bSqLb1hWU\n4boZywAAE4ccg6/W7wMALH24qV2Tf7O6BoF3l+c32RfjWduf60ZJfUMj0luoPZQ1XTkRj57kihmG\nSUESfq4bGTuhmyM19Vi/R32dUrW1UQ9W1kZfb9YYKZumOKN7DzVlzShj8VptktVvoPWdlR5CQan9\nzB3GmK7tMizt3yrd/1vuzOO72jru3EHdXLaECYqkEXo7Cz3f885K/OL/FqNSZYIwtacpM09Yyhh6\nmkJxG2NCN1oevbXvoLW70s6zn/m+aX926V0nzeJzdxC/wBn9u9g7rp+945zSumXSyJIp/IisJs0Z\nbWwEdhRX4HC1+UyVFbtKAQB19Sq3n8ENrPxUFtCi8pqYw4iaQkANMR691oApa6iVU1hWhZKK2pht\nQgisK1B/cmGcYaTzN47q46h8N54Axmh49COyOuseN7Kf/udeMeOm0x2X8cyvhuLm0VnOjUkSkiZG\n3yAEzn12IQb3aG/6GPkeVfN0VT16E02vcpBSGlFkPhshYkI3ry3OUz1W1m09z15pq1oI6Iy/fdds\n28wlO/Hfn280Mp2xgdFDWHy2lVXvbdKwXngvJ994RxvYecI7PasTlu8s9cCaJtJdyB5o1yrdVKZb\nqpBEHn3kot1YaH4OGHlSMbWb1fAWUFxDyuOVMf+C0sqoJ6/06LXi+wBQXl0XzeEHgPyDlZr7ymXm\n7q/Q7YzesJe9+TDx4ISBAIAB3dqpfv6nXwwGAEwa1hP/+8uTPbPDTkbyv28f6ajOS4f2NNzHjDwf\nr3HuZKyIvNOnLjWevMK7380OSSP0dtIrSScZRc2r/m6z+iBf5Z5Kb+f9nKYFUd5aujP6OkM1UyZS\n0HnPLsToaU1e+VlPLYi+Lquqi+kDEEJga9FhjH9uIV78bpt6mQA+XrlH8zPGf+RMqTYaYzRaSh5t\nW4+9UqM7Rq0h0Lx2TXLhkGMM94mf1VWNo1rqj29JI/MNmZmxMlZROmtGmPm+TkkeobfhnsgdaWqx\n7vg4NxA7h87+8qYpDcx0oq7OPxR9naERdxUQMVMlxFNV24ADR5o+b2gU2CNl9qzcfUjrMMZDjO7R\n+GtDQCFADm/wNX+a4Oh4t9n0lwvxwV1naH4+ZWw/XHRyD8NyzJwWuTEc0Ve9HyGNyHRoSvkbPXHJ\nYFPHGJF/0Hy2G3fGWsBOHr18gtWOfXG+tocMABU1TZk6ZtqYBsUUDQO7H626j5lybn69ac57ZePG\n0chwovaTyo6F5m9mQukGdGuHDm1amrNB48IyylSzekcdldECR7fW7vY7+wRzkxeauZZP6d1Rvwyb\nN8SQXh0M9zHTQT7sOH37/CZ5hN7EVVlb36S2VbUNUbFuFEBNfQN2qswPo0RzRKspoW+q+9hOR6nu\nYzS3TbyHEj83qDy/DhMOXrhmWDMxJTQJr10x+vy+MzHn3jEOrfNm7iOtKTp+eOgcjO5vnM//8MRB\nqufly9+cFfP+/MHdpfrUSSMy/f2U+2VndYZetGzpw+fhr5OGRN9/8uvRqvtdld1bdXtQJLTQbytq\n6tSsNzGr2e1v5kRfn/XUAtRIwt8gBG6YsSwm51yNxbklqtvNPCIq+xCe+Ew9A8aofrUy5brTCLh9\nVo7BEYzb6M091CajuXc7bmBm1CnROrJ/17YAtL3Lzm0zomW3NJGhohUDHm7D69QqK7tPZMVQraeE\nYzu3MVV+21bpUDszg3uqZ9NpNZZWxzco6d1J29ZjOrSO6Tfp3r616n5O+zLcJlzWWGSHwgOvbzAW\n20WK1MeSCsW0wY3CUcrY4ermA67i0Yu9262nUYjorJ1EFDNdA2Of/pltTe+rpydCiGZe5YvXDo9u\n0xLN0cd3xbwHxuKa049V/VwppmaeZLVCN3eP62/ruHjOGtAVb942AgCizpMTzGi00S5W+rDjv6WR\n4yaf87NPyNRsUIgIOY+ON1V/qJcSDBt1DfYvMKeLi/zi/xYb7lNuojEwYsLzsevWKh9iOEbvHhOH\nGHcYykQ8UHWO6dA6RjK6tmuF1i1bGMfoARzf7WjNhkCpv3ZGhLeX4uhuZZv07HBU9AnDzL10muT9\nq2FV9LSeqNSmMDGL0SmVz3mXtq1ipjyJp2u7VpYaHC9JaKFX/iBVDsTaDS8kCKrqGqJCojevPWPM\n6P5dsOLR8fjmd2PRLq5DUW8QXuc2Gcjqov6of0rvjjHXaDSd12GMXlmmnTj7M78aim9+NxZHt47t\nzH07LkfebNHKxkYp9PN/P051/5euP1WzLAIZhl2UYREt7zsSo/fmnpDLTSPjENE3v1M/B2MHOl9V\nzwoJLfRKbnzN/GpK8zYWxbx38jQQJBdMXxSdxiF+2cBko9vRrTwtv19mW3Rp1woDux+NFnE3b5d2\nGejcVnvyspNUYuly+Ec5sKdpJLb03qbS2/HilWSkp6lmfvXqqJ4kYIVObZrOU/9M9UFNek9BkTL0\ns4lO7dPR8BHWyqnN6hobqht6rH7fhexTpZF6o3SiwjHQGtil7KyVpzn3kgQXeuMLvlkesxD4YVus\nKCayN7xi18GgTWjGu1NG4fmrh7pa5tzfjnW1vD4aXjhg3dNupdLx9vHdkayYW0Zn4f+uHR5TrpnQ\njR5Or1at481+76euOEWzPCORBIy94D5d2uKTX4/GmsfVxwnMmNw0F44QzTNygEi/m2zXr8/u3yzt\nU/kdbhh5XMxnz1ypf+3Kv19aGjVzCoDI9a/GO3eMRI8Okc7b4cc1ha+O07kW3SLBhd6YZ77ZEvO+\n78NfYtZPu2K2menIDStezztih1H9uuDy4b0x5nj3Zj/U86jtcNpx2nFivdGo/VQ6avvHeW0Z6WnR\nHPe0NIrGpOV4clNnrCWTo1gNSXQ4KtZDbquSDQQ0H22qVU2bVub200Ivrt29feTJbfhxnZrZLdMu\n7olALSNHmeXWvX1r9FV47WkEXKXo6I5/sjLqu2hU/H6k8l207FZOIud36D7phV5rAjElZlIzUw09\nj9cs/7rhtGbb1ITSKp/fd6bjMhqEwFf3n4V7zzkeQKxYxQt9uuL9zMmnY6RyNCYBd43rjzdvHYEv\nfnOmvCmG+IGwTemVdkM35ved98A49Mtsh9l3jMLyR8bjpetPjaZCxtO5bQaOM5EGGe+RW214WqW3\nwEd3j8YjF50Y3Tayb2e8ddsInHdid0tlaVHf2DzjSUZ9sR7zWInRa5bhyALrJLTQm7m+quuMRdxK\nfD9VuHx4L8dlxHf2AbEdaXZjwvExVTs0ikgsdUiviDfYU2FLvIfXQuGCHt06HRef0iPuc8LYgZno\n11U9HtskDJJH7/A2t5ItI8eIz+jfBZlHt8LEk3toZqQQUdyUAup2yk9XJ0hx/l4aAwD1OK1Pp5hO\n736Z7XDWAPMdlEaNZH1jY/Q3zYzr3xmikZNvlq7tIuX17tTGclM9QDpnfi9Ak9BCz3iH7OkC0Mzn\nVkOOR2shxzd/f/5ATB7dfNZAo1kJgVgP2y7ytBcXnHQMXr7xNNw5tl/0s/i4a7wDeMPIPrhlTBYA\nc4/g8Q6J/F4vhKHFrFtHGDaQ72nEiPWYfccotEgjU+mAo/p1wYybsvHZfWfi1Zuy8ZvzBliuL57H\nbc4xo9Vk1jcITBnbDy/feBomKiZSmzpxEF6/eYRhufLTmRoTh0SumTvO6tcs/GZ0/v5+3XD8+7aR\n0cbCLxJa6BM3sh5+lI+3FxjMOHh6VlMo4BKDaWhlkZtw0jGqXtkvTjHOYXdD6OURjUSEC046Jub7\nxgt7elpaNMabnpaGtDSKDsG3giwK8vKDndtav9nHmUjLG2ljZSh5FaqeJp+yxg/ujoz0NJw/uDta\n2gyFKLNrWhvMRikje8Jy52pvDXvbtY7M/HnBScfEPKGN6tdFc46gtoonpZN6dtD8TL5mWqRRNMwn\nh7z0QjlEhPatW+LMAfaWdnRCQi884lGabNLz1m0jLIWrzh6Yib/98mS8vWwX1u9pPt+/mSmibx6d\nhStP643fvrcagDSNrEpTfc85x2P6PP0J5dQ6S1+9KRt3vGluCoiWLQgPXXiC5ufNQzeEGTdlY1V+\naVQkjtRE8sWVHYPy94m/1xvj8uZvO7MfurdvjfatW+KzNXsN7f3o7tG44qUfDfdzA+X59/r+uuAk\n4ymLlcy+Y1Q0Q2VIrw546fpTMU6aKO2734/Duc8uBBB5qjSzDOK8B8ZF57t6546RyOoSGxJc9Idz\nsDi3BEN6tUfXdq2w80DzubBapbfA67ecjv5d22Hs0wssxew/unu060kGWiS00DP2OKWXtTlOiAjX\njjgOh6vrbAv9E5eeBEApeqTaqWjGO1TLP7fiYV8ytKeuB9kxLmsiPY3QqW0Gzh3UVIdsgZnOZRHX\n+doijXDZsF5YvtNcaqzeSFK3adkiDRef3ANfrCs07WUbEZ8lI2N1HEH82rcTFVMe91GItN5TpbJG\nZZhQbcK147q0wXVdmlIvtZ52zjmhW9Nstha+kp+/a4KHbtilt4XNyEe8t/KvGyIjHBuEwLe/G4vZ\ndxjHhpWphWY9RjPlAhGvTo8bRkVuWiOv6/zB3aMdjYD6E8R5J3bDtF+ejAcv0H4ykNGafj67Tyc8\n+6uhuOOsvoZlmOHRi0/EKzc2z3SyyrQrTsbzVw81NWWvES9eO7zZiFsvMVwfwKt6pf9hmfIgnoQW\nesYechZIe425w7Vm3ov3wORZ/hoaI9kE8R4X0FwkhWKwkNmGWp5lUSs/WaZfZjvVGRm7SI/HQ6S4\nq9ogFyVEhKsVHdCd2zV/vCYiXDPiOLRKb/J65QbkmLgZDeOzbpRlXHFab9sxbiD2t8rq0hYTLIZD\n1Di6dUtcPjx2mt0uNkMMlw7taWoQlVv07OB8dK8T1PqduqpcP36T0KEbjtHbIzqEO43w/p1noKKm\nDkXlNXj443UAgG9+pz4KNX5GXFnE9RZ9mf/AOGze1xTuURv+P2Fwd/TseJRqSuf0q4dFQwif33cm\n1hREVtJ6+cbTMKBbu2hcVkZp4jt3jESbjHR0aZuB1fmHUF5dF/3eRtwwqk/U1uvjRk5q0bplC0y/\nehhG9otd9Sj6nTWOs7MMpsw3vxuLlbtLUVFTj/NO7Ga7HCPm3DsGq0K8ilmLNMIL1wzD6VnqK07J\neOVwy7+g2qU18+bTsW5PWaCLlSe00DP2kC+3ru1axeRNy0KvlaceL5CyAOt52lld28aU1xSvbnrd\nL7Mdpk4cpHr8JIX4H9u5TXRec62OvI6KuVaUcddjO7fBWz/tjHwPE/dbRnoabjvTekhlkkpjJQyU\nvs7ByOz48+sVvTu10Z2nPQxcNkx77Id8jaabmL/fDsq+p3hO6d3RcEUsr0looecRrfbo1DYD/3v5\nyThnUGyq3ss3nhbz6PvO7SNjLlz59ZnHd8Wk4b3Qt2tb/PdlJxmmXyppWihFWa6trwEAmHPPGOxX\nzPX/7K+G4tZZy/Fk3HwsQNOTjN+elTJcpYZ8HZ87qBvuPlt9jvjP7j0ThWXm1yE1y/t3nuF4mu54\nXr0pOzqVgREvXX+qL3O9TL96GD5ds1d3JlIniOi6EJ4U75iEFvoaE6NeE4WB3dthT2kVjtTq33TD\nju0Ys9C4Xa5TCUfEe8mjj4/NRJBj28d2PgpXnhaJ4d54RpalehsVN4SRAJohPv7bqW0GPvm1+jJ7\ncojEyepDdpCfhNRGCgNNk+qdM6ibZujh5N4dcHJv552j8Wgtru0EKxlQE00sFu4GXdq1wi1j3On0\n1sOoHykoErozttaD6YXtdjo5pba+0VTs+KUb1OfyVi68LA/rB9Bs7hK9EX9GyOa58SClzLrxS3cb\nNTpFvaZf17Z4eOIg1bl/AKBeuo5bhjVlgzGkQ5uWePySwaYzxPwmoYVey6MfrZL9oaS3ztwceqvY\ne0mnthmm5n7poZFVMOOm7OjrS4f2jA77/vNlJ8XsFz/izwoZ0qhEJ3HOVi0jZaQRoaVUXkYLd/K1\njZBDNq1b+nvZExHuHNcfx3RQX19UnkvHrbx1JhhuGdPX9Nq4fuPJFU9EFxLRFiLKJaKpXtQBNM2w\nOGFwdww6JpL3PGVsP7xwzfDo+3hev/l0vH37SHRs0xKj+nWOGYb91JWn4M1bm3J+xw7MRMc2LfHW\nbU1zY8gLBshzcww9tiOuODWy7YHzB6J7+1Y4ulU6rsruja7tMjD5jD74r4siHY3yVAFDj+2IEVmd\no1kmHdu0xF8uHRITV37qilMwoFs7PHXlKfiVFCaRB4LIecn3S3OMLHjwbKS3SMOpUmrhDaP64K+T\nhuDXZ/fH2AGZ+J/Lh6Bv17bRPHK7XDK0J+4c1w8PXajecWqGN24egQcnDESPDq1x8+gs3Dm2H6Yo\n5pkBgI/uPgMj+3Z23Tu6dsRxmDK2H+5RzOMTBqZeOAh3jevfbLI0J3x09xl48oqTHZfz7pRRePZX\n7q4tYIW3bx+JF64ZFlj9yQK5vdwWEbUAsBXA+QAKACwHcK0QYqPWMdnZ2SInx9zwdYZhGCYCEa0Q\nQmQb7eeFRz8CQK4QYocQohbAuwAu86AehmEYxgReCH0vAPmK9wXSNoZhGCYAAuuMJaIpRJRDRDnF\nxcm9sDXDMEyQeCH0ewAoV6roLW2LQQjxihAiWwiRnZlpfmUZhmEYxhpeCP1yAAOIqC8RZQC4BsCn\nHtTDMAzDmMD1pHEhRD0R3QvgawAtAMwUQmxwux6GYRjGHJ6MDhJCfAngSy/KZhiGYayR0CNjGYZh\nGGNcHzBlywiiYgC7bB7eFUCJi+a4BdtljbDaBYTXNrbLGsloVx8hhGE2SyiE3glElGNmZJjfsF3W\nCKtdQHhtY7uskcp2ceiGYRgmyWGhZxiGSXKSQehfCdoADdgua4TVLiC8trFd1khZuxI+Rs8wDMPo\nkwwePcMwDKNDQgu9XwucaNR9LBEtIKKNRLSBiO6Xtj9BRHuIaLX0d5HimIclW7cQ0QUe2raTiNZJ\n9edI2zoT0bdEtE3630naTkT0omTXWiJSX6vQuU0nKM7JaiIqJ6LfBnG+iGgmEe0novWKbZbPDxFN\nlvbfRkSTPbLraSLaLNX9CRF1lLZnEVGV4rz9S3HMadLvnyvZ7miNQg27LP9ubt+vGna9p7BpJxGt\nlrb7eb60tCG4a0wIkZB/iEyvsB1APwAZANYAGOxj/T0AnCq9PhqRxVYGA3gCwIMq+w+WbGwFoK9k\newuPbNsJoGvctqcATJVeTwXwpPT6IgBfIbJG9ygAy3z67fYB6BPE+QIwFsCpANbbPT8AOgPYIf3v\nJL3u5IFdEwCkS6+fVNiVpdwvrpyfJVtJsn2iB3ZZ+t28uF/V7Ir7/FkAfwrgfGlpQ2DXWCJ79IEu\ncCKEKBRCrJReHwawCfrz7l8G4F0hRI0QIg9ALiLfwS8uAzBLej0LwCTF9jdFhKUAOhKRe2vaqXMe\ngO1CCL1Bcp6dLyHEIgAHVeqzcn4uAPCtEOKgEKIUwLcALnTbLiHEN0KIeuntUkRmg9VEsq29EGKp\niKjFm4rv4ppdOmj9bq7fr3p2SV75VQBm65Xh0fnS0obArrFEFvrQLHBCRFkAhgNYJm26V3oEmyk/\nnsFfewWAb4hoBRFNkbZ1F0IUSq/3AegegF0y1yD2Bgz6fAHWz08Q5+1WRDw/mb5EtIqIFhLRWdK2\nXpItfthl5Xfz+3ydBaBICLFNsc338xWnDYFdY4ks9KGAiNoB+AjAb4UQ5QBeAtAfwDAAhYg8PvrN\nmUKIUwFMBHAPEY1Vfih5LoGkW1Fk6upLAXwgbQrD+YohyPOjBRE9AqAewNvSpkIAxwkhhgN4AMA7\nRNTeR5NC97vFcS1inQnfz5eKNkTx+xpLZKE3tcCJlxBRS0R+yLeFEB8DgBCiSAjRIIRoBPAqmsIN\nvtkrhNgj/d8P4BPJhiI5JCP93++3XRITAawUQhRJNgZ+viSsnh/f7COimwH8AsD1kkBACo0ckF6v\nQCT+PVCyQRne8cQuG7+bn+crHcAvAbynsNfX86WmDQjwGktkoQ90gRMpBvgagE1CiOcU25Xx7csB\nyBkBnwK4hohaEVFfAAMQ6QRy2662RHS0/BqRzrz1Uv1yr/1kAHMUdt0k9fyPAlCmeLz0ghhPK+jz\npcDq+fkawAQi6iSFLSZI21yFiC4E8BCAS4UQlYrtmUTUQnrdD5Hzs0OyrZyIRknX6E2K7+KmXVZ/\nNz/v1/EANgshoiEZP8+XljYgyGvMSe9y0H+I9FZvRaR1fsTnus9E5NFrLYDV0t9FAN4CsE7a/imA\nHopjHpFs3QKHPfs6dvVDJKNhDYAN8nkB0AXAfADbAMwD0FnaTgD+Idm1DkC2h+esLYADADootvl+\nvhBpaAoB1CES97zNzvlBJGaeK/3d4pFduYjEaeVr7F/SvldIv+9qACsBXKIoJxsR4d0O4O+QBka6\nbJfl383t+1XNLmn7GwDuitvXz/OlpQ2BXWM8MpZhGCbJSeTQDcMwDGMCFnqGYZgkh4WeYRgmyWGh\nZxiGSXJY6BmGYZIcFnqGYZgkh4WeYRgmyWGhZxiGSXL+H7P0OhRijBnTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5bc80a5a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(jList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the network learns to solve the FrozenLake problem, it turns out it doesn’t do so quite as efficiently as the Q-Table. While neural networks allow for greater flexibility, they do so at the cost of stability when it comes to Q-Learning. There are a number of possible extensions to our simple Q-Network which allow for greater performance and more robust learning. we will be exploring those additions in Exercise 2."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}