assignment4/DQN_WorldNavigate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# World Navigation with DQN\n",
    "\n",
    "In this exercise you will play a world navigation game with Deep Q-Networks. The agent learn to solve a navigation task in a basic grid world. It will be built upon the simple one layer Q-network you created in Exercise 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to transform an ordinary Q-Network into a DQN you will be making the following improvements:\n",
    "1. Going from a single-layer network to a multi-layer convolutional network.\n",
    "2. Implementing Experience Replay, which will allow our network to train itself using stored memories from it's experience.\n",
    "3. Utilizing a second \"target\" network, which we will use to compute target Q-values during our updates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will also implement two simple additional improvements to the DQN architecture, Double DQN and Dueling DQN, that allow for improved performance, stability, and faster training time. In the end you will have a network that can tackle a number of challenging Atari games, and we will demonstrate how to train the DQN to learn a basic navigation task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "\n",
    "import gym\n",
    "import numpy as np\n",
    "import random\n",
    "import tensorflow as tf\n",
    "import tensorflow.contrib.slim as slim\n",
    "import matplotlib.pyplot as plt\n",
    "import scipy.misc\n",
    "import os\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the game environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADMhJREFUeJzt3X/oXfV9x/Hna4nW1m7VqAuZ0X0zKkoYGF1wimV0Wjdr\ni+6PIkoZZQj+0226Flrd/pDC/mhhtPWPUZDaTobzR62uIRQ7l1rG/kmNP9ZqojXaWBOMiZ3OzsG2\ntO/9cU/YtyHxe7753l/Hz/MBl3vPOfdyPieH1z3nnpzv+52qQlJbfmXWA5A0fQZfapDBlxpk8KUG\nGXypQQZfapDBlxq0ouAnuTLJc0l2J7llXIOSNFk53ht4kqwCfgRcAewFHgOur6qd4xuepElYvYLP\nXgTsrqoXAZLcC1wDHDP4p59+ei0sLKxglZLezp49e3jttdey1PtWEvwzgZcXTe8FfvftPrCwsMCO\nHTtWsEpJb2fz5s293jfxi3tJbkyyI8mOgwcPTnp1knpYSfD3AWctml7fzfslVXVHVW2uqs1nnHHG\nClYnaVxWEvzHgHOSbEhyInAdsGU8w5I0Scf9G7+qDiX5U+A7wCrga1X1zNhGJmliVnJxj6r6NvDt\nMY1F0pR4557UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMv\nNcjgSw0y+FKDDL7UoCWDn+RrSQ4keXrRvDVJHknyfPd86mSHKWmc+hzx/w648oh5twDbquocYFs3\nLWkglgx+Vf0L8O9HzL4GuKt7fRfwR2Mel6QJOt7f+Gur6pXu9X5g7ZjGI2kKVnxxr0ZdN4/ZedNO\nOtL8Od7gv5pkHUD3fOBYb7STjjR/jjf4W4BPdK8/AXxrPMORNA1LNtRIcg/wQeD0JHuB24DPA/cn\nuQF4Cbh2koMch2TJzsGagGP+BpyCzHDtNcsN72HJ4FfV9cdYdPmYxyJpSrxzT2qQwZcaZPClBhl8\nqUEGX2qQwZcaZPClBhl8qUEGX2qQwZcaZPClBhl8qUEGX2qQwZcaZPClBhl8qUEGX2pQn046ZyV5\nNMnOJM8kuambbzcdaaD6HPEPAZ+uqo3AxcAnk2zEbjrSYPXppPNKVT3Rvf4ZsAs4E7vpSIO1rN/4\nSRaAC4Dt9OymY0MNaf70Dn6S9wLfBG6uqjcXL3u7bjo21JDmT6/gJzmBUejvrqoHu9m9u+lImi99\nruoHuBPYVVVfXLTIbjrSQC3ZUAO4FPhj4IdJnurm/SUD7KYjaaRPJ51/BY7Vf8puOtIAeeee1CCD\nLzXI4EsN6nNxT0M243bNs+1OPsuVz3efbI/4UoMMvtQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4\nUoMMvtQggy81yOBLDTL4UoP61Nw7Kcn3k/xb10nnc938DUm2J9md5L4kJ05+uJLGoc8R/7+By6rq\nfGATcGWSi4EvAF+qqvcDrwM3TG6YksapTyedqqr/7CZP6B4FXAY80M23k440IH3r6q/qKuweAB4B\nXgDeqKpD3Vv2MmqrdbTP2klHmjO9gl9VP6+qTcB64CLgvL4rsJOONH+WdVW/qt4AHgUuAU5Jcrh0\n13pg35jHJmlC+lzVPyPJKd3rdwNXMOqY+yjwse5tdtKRBqRPsc11wF1JVjH6ori/qrYm2Qncm+Sv\ngScZtdmSNAB9Oun8gFFr7CPnv8jo976kgfHOPalBBl9qkMGXGmTwpQYZfKlBBl9qkMGXGmTwpQbZ\nJvudbqZtqjWvPOJLDTL4UoMMvtQggy81yOBLDTL4UoMMvtQggy81qHfwuxLbTybZ2k3bSUcaqOUc\n8W9iVGTzMDvpSAPVt6HGeuAjwFe76WAnHWmw+h7xvwx8BvhFN30adtKRBqtPXf2PAgeq6vHjWYGd\ndKT50+ev8y4Frk5yFXAS8GvA7XSddLqjvp10pAHp0y331qpaX1ULwHXAd6vq49hJRxqslfw//meB\nTyXZzeg3v510pIFYViGOqvoe8L3utZ10pIHyzj2pQQZfapDBlxpk8KUGGXypQQZfapDBlxpk8KUG\nGXypQQZfapDBlxpk8KUGGXypQQZfatCy/ixXx6dmuO7McuUAmfH6dVQe8aUG9TriJ9kD/Az4OXCo\nqjYnWQPcBywAe4Brq+r1yQxT0jgt54j/+1W1qao2d9O3ANuq6hxgWzctaQBWcqp/DaNGGmBDDWlQ\n+ga/gH9K8niSG7t5a6vqle71fmDt2EcnaSL6XtX/QFXtS/LrwCNJnl28sKoqOfr14+6L4kaAs88+\ne0WDlTQevY74VbWvez4APMSouu6rSdYBdM8HjvFZO+lIc6ZPC62Tk/zq4dfAHwBPA1sYNdIAG2pI\ng9LnVH8t8NCoQS6rgX+oqoeTPAbcn+QG4CXg2skNU9I4LRn8rnHG+UeZ/1Pg8kkMStJkeeee1CCD\nLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+\n1KBewU9ySpIHkjybZFeSS5KsSfJIkue751MnPVhJ49H3iH878HBVnceoDNcu7KQjDVafKrvvA34P\nuBOgqv6nqt7ATjrSYPWpsrsBOAh8Pcn5wOPATdhJpzc7Rc/IrFuEz7E+p/qrgQuBr1TVBcBbHHFa\nX1XFMf6Zk9yYZEeSHQcPHlzpeCWNQZ/g7wX2VtX2bvoBRl8EdtKRBmrJ4FfVfuDlJOd2sy4HdmIn\nHWmw+jbN/DPg7iQnAi8Cf8LoS8NOOtIA9Qp+VT0FbD7KIjvpSAPknXtSgwy+1CCDLzXI4EsNMvhS\ngwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSg/rU1T83yVOLHm8m\nudlOOtJw9Sm2+VxVbaqqTcDvAP8FPISddKTBWu6p/uXAC1X1EnbSkQZrucG/Drine20nHWmgege/\nK619NfCNI5fZSUcaluUc8T8MPFFVr3bTdtKRBmo5wb+e/z/NBzvpSIPVK/hJTgauAB5cNPvzwBVJ\nngc+1E1LGoC+nXTeAk47Yt5PGVAnndFlCE2d/+xzyTv3pAYZfKlBBl9qkMGXGmTwpQYZfKlBBl9q\nkMGXGmTwpQYZfKlBBl9qkMGXGmTwpQYZfKlBBl9qkMGXGmTwpQb1Lb31F0meSfJ0knuSnJRkQ5Lt\nSXYnua+rwitpAPq00DoT+HNgc1X9NrCKUX39LwBfqqr3A68DN0xyoJLGp++p/mrg3UlWA+8BXgEu\nAx7olttJRxqQPr3z9gF/A/yEUeD/A3gceKOqDnVv2wucOalBShqvPqf6pzLqk7cB+A3gZODKviuw\nk440f/qc6n8I+HFVHayq/2VUW/9S4JTu1B9gPbDvaB+2k440f/oE/yfAxUnekySMaunvBB4FPta9\nx0460oD0+Y2/ndFFvCeAH3afuQP4LPCpJLsZNdu4c4LjlDRGfTvp3AbcdsTsF4GLxj4iSRPnnXtS\ngwy+1CCDLzXI4EsNyjTbRyc5CLwFvDa1lU7e6bg98+qdtC3Qb3t+s6qWvGFmqsEHSLKjqjZPdaUT\n5PbMr3fStsB4t8dTfalBBl9q0CyCf8cM1jlJbs/8eidtC4xxe6b+G1/S7HmqLzVoqsFPcmWS57o6\nfbdMc90rleSsJI8m2dnVH7ypm78mySNJnu+eT531WJcjyaokTybZ2k0PtpZiklOSPJDk2SS7klwy\n5P0zyVqXUwt+klXA3wIfBjYC1yfZOK31j8Eh4NNVtRG4GPhkN/5bgG1VdQ6wrZsekpuAXYumh1xL\n8Xbg4ao6Dzif0XYNcv9MvNZlVU3lAVwCfGfR9K3ArdNa/wS251vAFcBzwLpu3jrguVmPbRnbsJ5R\nGC4DtgJhdIPI6qPts3l+AO8Dfkx33WrR/EHuH0al7F4G1jD6K9qtwB+Oa/9M81T/8IYcNtg6fUkW\ngAuA7cDaqnqlW7QfWDujYR2PLwOfAX7RTZ/GcGspbgAOAl/vfrp8NcnJDHT/1IRrXXpxb5mSvBf4\nJnBzVb25eFmNvoYH8d8kST4KHKiqx2c9ljFZDVwIfKWqLmB0a/gvndYPbP+sqNblUqYZ/H3AWYum\nj1mnb14lOYFR6O+uqge72a8mWdctXwccmNX4lulS4Ooke4B7GZ3u307PWopzaC+wt0YVo2BUNepC\nhrt/VlTrcinTDP5jwDndVckTGV2o2DLF9a9IV2/wTmBXVX1x0aItjGoOwoBqD1bVrVW1vqoWGO2L\n71bVxxloLcWq2g+8nOTcbtbh2pCD3D9MutbllC9YXAX8CHgB+KtZX0BZ5tg/wOg08QfAU93jKka/\ni7cBzwP/DKyZ9ViPY9s+CGztXv8W8H1gN/AN4F2zHt8ytmMTsKPbR/8InDrk/QN8DngWeBr4e+Bd\n49o/3rknNciLe1KDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw36P7nm6nS1BQprAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x261d61fd4a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from gridworld import gameEnv\n",
    "\n",
    "env = gameEnv(partial=False,size=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above is an example of a starting environment in our simple game. The game environment outputs 84x84x3 color images, and uses function calls as similar to the OpenAI gym as possible. The agent controls the blue square, and can move up, down, left, or right. The goal is to move to the green square (for +1 reward) and avoid the red square (for -1 reward). \n",
    "\n",
    "At the start of each episode all squares are randomly placed within a 5x5 grid-world. The agent has 50 steps to achieve as large a reward as possible. Because they are randomly positioned, the agent needs to do more than simply learn a fixed path, as was the case in the FrozenLake environment from Exercise 1. Instead the agent must learn a notion of spatial relationships between the blocks. \n",
    "\n",
    "*Feel free to adjust the size of the gridworld (default 5). Making it smaller provides an easier task for our DQN agent, while making the world larger increases the challenge.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Addition 1: Convolutional Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first major addition to make DQNs work is to use convolutional layers to set up the networks. We are now familiar with convolutional layers after assignment `1-3`. For more information, see the [Tensorflow documentation](https://www.tensorflow.org/api_docs/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Addition 2: Experience Replay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second major addition to make DQNs work is Experience Replay. The basic idea is that by storing an agent’s experiences, and then randomly drawing batches of them to train the network, we can more robustly learn to perform well in the task. By keeping the experiences we draw random, we prevent the network from only learning about what it is immediately doing in the environment, and allow it to learn from a more varied array of past experiences. \n",
    "\n",
    "Each of these experiences are stored as a tuple of *(state,action,reward,next state)*. The Experience Replay buffer stores a fixed number of recent memories, and as new ones come in, old ones are removed. When the time comes to train, we simply draw a uniform batch of random memories from the buffer, and train our network with them. \n",
    "\n",
    "For our DQN, we build a simple class that allows us to store experies and sample them randomly to train the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class experience_buffer():\n",
    "    def __init__(self, buffer_size = 50000):\n",
    "        self.buffer = []\n",
    "        self.buffer_size = buffer_size\n",
    "    \n",
    "    def add(self,experience):\n",
    "        if len(self.buffer) + len(experience) >= self.buffer_size:\n",
    "            self.buffer[0:(len(experience)+len(self.buffer))-self.buffer_size] = []\n",
    "        self.buffer.extend(experience)\n",
    "            \n",
    "    def sample(self,size):\n",
    "        return np.reshape(np.array(random.sample(self.buffer,size)),[size,5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a simple function to resize our game frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def processState(states):\n",
    "    return np.reshape(states,[21168]) # 84 x 84 x 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Addition 3: Separate Target Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third major addition to the DQN that makes it unique is the utilization of a second network during the training procedure. This second network is used to generate the target-Q values that will be used to compute the loss for every action during training. Why not use just use one network for both estimations? The issue is that at every step of training, the Q-network’s values shift, and if we are using a constantly shifting set of values to adjust our network values, then the value estimations can easily spiral out of control. The network can become destabilized by falling into feedback loops between the target and estimated Q-values. In order to mitigate that risk, the target network’s weights are fixed, and only periodically or slowly updated to the primary Q-networks values. In this way training can proceed in a more stable manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions allow us to update the parameters of our target network with those of the primary network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def updateTargetGraph(tfVars,tau):\n",
    "    total_vars = len(tfVars)\n",
    "    op_holder = []\n",
    "    for idx,var in enumerate(tfVars[0:total_vars//2]):\n",
    "        op_holder.append(tfVars[idx+total_vars//2].assign((var.value()*tau) + ((1-tau)*tfVars[idx+total_vars//2].value())))\n",
    "    return op_holder\n",
    "\n",
    "def updateTarget(op_holder,sess):\n",
    "    for op in op_holder:\n",
    "        sess.run(op)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the additions above, we have everything we need to replicate the DQN.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dueling DQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to explain the reasoning behind the architecture changes that Dueling DQN makes, we need to first explain some a few additional reinforcement learning terms. The Q-values that we have been discussing so far correspond to how good it is to take a certain action given a certain state. This can be written as `Q(s,a)`. This action given state can actually be decomposed into two more fundamental notions of value. The first is the value function `V(s)`, which says simple how good it is to be in any given state. The second is the advantage function `A(a)`, which tells how much better taking a certain action would be compared to the others. We can then think of `Q` as being the combination of `V` and `A`. More formally:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Q(s,a) =V(s) + A(a) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of Dueling DQN is to have a network that separately computes the advantage and value functions, and combines them back into a single Q-function only at the final layer. It may seem somewhat pointless to do this at first glance. Why decompose a function that we will just put back together? The key to realizing the benefit is to appreciate that our reinforcement learning agent may not need to care about both value and advantage at any given time. We can achieve more robust estimates of state value by decoupling it from the necessity of being attached to specific actions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing the network itself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Qnetwork():\n",
    "    def __init__(self,h_size):\n",
    "        #The network recieves a frame from the game, flattened into an array.\n",
    "        #It then resizes it and processes it through four convolutional layers.\n",
    "        #We use slim.conv2d to set up our network \n",
    "        self.scalarInput =  tf.placeholder(shape=[None,21168],dtype=tf.float32)\n",
    "        self.imageIn = tf.reshape(self.scalarInput,shape=[-1,84,84,3])\n",
    "        self.conv1 = slim.conv2d( \\\n",
    "            inputs=self.imageIn,num_outputs=32,kernel_size=[8,8],stride=[4,4],padding='VALID', biases_initializer=None)\n",
    "        self.conv2 = slim.conv2d( \\\n",
    "            inputs=self.conv1,num_outputs=64,kernel_size=[4,4],stride=[2,2],padding='VALID', biases_initializer=None)\n",
    "        self.conv3 = slim.conv2d( \\\n",
    "            inputs=self.conv2,num_outputs=64,kernel_size=[3,3],stride=[1,1],padding='VALID', biases_initializer=None)\n",
    "        self.conv4 = slim.conv2d( \\\n",
    "            inputs=self.conv3,num_outputs=h_size,kernel_size=[7,7],stride=[1,1],padding='VALID', biases_initializer=None)\n",
    "        \n",
    "        ################################################################################\n",
    "        # TODO: Implement Dueling DQN                                                  #\n",
    "        # We take the output from the final convolutional layer i.e. self.conv4 and    #\n",
    "        # split it into separate advantage and value streams.                          #\n",
    "        # Outout: self.Advantage, self.Value                                           #\n",
    "        # Hint: Refer to Fig.1 in [Dueling DQN](https://arxiv.org/pdf/1511.06581.pdf)  #\n",
    "        #       In implementation, use tf.split to split into two branches. You may    #\n",
    "        #       use xavier_initializer for initializing the two additional linear      #\n",
    "        #       layers.                                                                # \n",
    "        ################################################################################\n",
    "        self.advantage_stream, self.value_stream = tf.split(self.conv4, 2, 3)\n",
    "        self.advantage_stream = slim.flatten(self.advantage_stream)\n",
    "        self.value_stream = slim.flatten(self.value_stream)\n",
    "        \n",
    "        xavier = tf.contrib.layers.xavier_initializer()\n",
    "        self.W_value = tf.Variable(xavier([h_size // 2, 1]))\n",
    "        self.W_advantage = tf.Variable(xavier([h_size // 2, env.actions]))\n",
    "        \n",
    "        self.Advantage = tf.matmul(self.advantage_stream, self.W_advantage)\n",
    "        self.Value = tf.matmul(self.value_stream, self.W_value)\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        \n",
    "        #Then combine them together to get our final Q-values. \n",
    "        #Please refer to Equation (9) in [Dueling DQN](https://arxiv.org/pdf/1511.06581.pdf)\n",
    "        self.Qout = self.Value + tf.subtract(self.Advantage,tf.reduce_mean(self.Advantage,axis=1,keep_dims=True))\n",
    "        self.predict = tf.argmax(self.Qout,1)\n",
    "        \n",
    "        #Below we obtain the loss by taking the sum of squares difference between the target and prediction Q values.\n",
    "        self.targetQ = tf.placeholder(shape=[None],dtype=tf.float32)\n",
    "        self.actions = tf.placeholder(shape=[None],dtype=tf.int32)\n",
    "        self.actions_onehot = tf.one_hot(self.actions,env.actions,dtype=tf.float32)\n",
    "        \n",
    "        ################################################################################\n",
    "        # TODO:                                                                        #\n",
    "        # Obtain the loss (self.loss) by taking the sum of squares difference          #\n",
    "        # between the target and prediction Q values.                                  #\n",
    "        ################################################################################\n",
    "        self.Q = tf.reduce_sum(tf.multiply(self.Qout, self.actions_onehot), axis=1)\n",
    "        self.loss = tf.reduce_mean(tf.square(self.targetQ - self.Q))\n",
    "        ################################################################################\n",
    "        #                                 END OF YOUR CODE                             #\n",
    "        ################################################################################\n",
    "        \n",
    "        self.trainer = tf.train.AdamOptimizer(learning_rate=0.0001)\n",
    "        self.updateModel = self.trainer.minimize(self.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting all the training parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 32 #How many experiences to use for each training step.\n",
    "update_freq = 4 #How often to perform a training step.\n",
    "y = .99 #Discount factor on the target Q-values\n",
    "startE = 1 #Starting chance of random action\n",
    "endE = 0.1 #Final chance of random action\n",
    "annealing_steps = 10000. #How many steps of training to reduce startE to endE.\n",
    "num_episodes = 5000 #How many episodes of game environment to train network with.\n",
    "pre_train_steps = 10000 #How many steps of random actions before training begins.\n",
    "max_epLength = 50 #The max allowed length of our episode.\n",
    "# max_epLength = 1\n",
    "load_model = False #Whether to load a saved model.\n",
    "path = \"./dqn\" #The path to save our model to.\n",
    "h_size = 512 #The size of the final convolutional layer before splitting it into Advantage and Value streams.\n",
    "tau = 0.001 #Rate to update target network toward primary network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Double DQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main intuition behind Double DQN is that the regular DQN often overestimates the Q-values of the potential actions to take in a given state. While this would be fine if all actions were always overestimates equally, there was reason to believe this wasn’t the case. You can easily imagine that if certain suboptimal actions regularly were given higher Q-values than optimal actions, the agent would have a hard time ever learning the ideal policy. In order to correct for this, the authors of DDQN paper propose a simple trick: instead of taking the max over Q-values when computing the target-Q value for our training step, we use our primary network to chose an action, and our target network to generate the target Q-value for that action. By decoupling the action choice from the target Q-value generation, we are able to substantially reduce the overestimation, and train faster and more reliably. Below is the new DDQN equation for updating the target value.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ Q \\text{-}target = r + γQ(s',\\arg \\max(Q(s’,a,\\theta),\\theta')) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved Model\n",
      "Episode 9 reward: 1.8\n",
      "Episode 19 reward: 1.1\n",
      "Episode 29 reward: 1.8\n",
      "Episode 39 reward: 2.7\n",
      "Episode 49 reward: 2.6\n",
      "Episode 59 reward: 3.1\n",
      "Episode 69 reward: 1.0\n",
      "Episode 79 reward: 2.2\n",
      "Episode 89 reward: 2.0\n",
      "Episode 99 reward: 2.2\n",
      "Episode 109 reward: 1.6\n",
      "Episode 119 reward: 3.6\n",
      "Episode 129 reward: 2.8\n",
      "Episode 139 reward: 3.4\n",
      "Episode 149 reward: 2.6\n",
      "Episode 159 reward: 0.1\n",
      "Episode 169 reward: 1.5\n",
      "Episode 179 reward: 3.3\n",
      "Episode 189 reward: 2.0\n",
      "Episode 199 reward: 1.6\n",
      "Episode 209 reward: 2.1\n",
      "Episode 219 reward: 2.4\n",
      "Episode 229 reward: 2.0\n",
      "Episode 239 reward: 2.0\n",
      "Episode 249 reward: 3.3\n",
      "Episode 259 reward: 2.6\n",
      "Episode 269 reward: 2.1\n",
      "Episode 279 reward: 3.4\n",
      "Episode 289 reward: 2.1\n",
      "Episode 299 reward: 3.6\n",
      "Episode 309 reward: 2.2\n",
      "Episode 319 reward: 2.9\n",
      "Episode 329 reward: 2.4\n",
      "Episode 339 reward: 3.1\n",
      "Episode 349 reward: 2.4\n",
      "Episode 359 reward: 1.6\n",
      "Episode 369 reward: 3.8\n",
      "Episode 379 reward: 3.5\n",
      "Episode 389 reward: 4.0\n",
      "Episode 399 reward: 3.0\n",
      "Episode 409 reward: 3.1\n",
      "Episode 419 reward: 4.2\n",
      "Episode 429 reward: 4.3\n",
      "Episode 439 reward: 3.6\n",
      "Episode 449 reward: 4.4\n",
      "Episode 459 reward: 3.0\n",
      "Episode 469 reward: 4.1\n",
      "Episode 479 reward: 3.8\n",
      "Episode 489 reward: 6.5\n",
      "Episode 499 reward: 8.3\n",
      "Episode 509 reward: 9.5\n",
      "Episode 519 reward: 7.7\n",
      "Episode 529 reward: 6.8\n",
      "Episode 539 reward: 8.2\n",
      "Episode 549 reward: 12.5\n",
      "Episode 559 reward: 12.5\n",
      "Episode 569 reward: 10.4\n",
      "Episode 579 reward: 10.2\n",
      "Episode 589 reward: 13.0\n",
      "Episode 599 reward: 12.8\n",
      "Episode 609 reward: 10.4\n",
      "Episode 619 reward: 12.8\n",
      "Episode 629 reward: 13.3\n",
      "Episode 639 reward: 9.7\n",
      "Episode 649 reward: 11.2\n",
      "Episode 659 reward: 13.8\n",
      "Episode 669 reward: 16.9\n",
      "Episode 679 reward: 13.2\n",
      "Episode 689 reward: 10.8\n",
      "Episode 699 reward: 19.1\n",
      "Episode 709 reward: 17.3\n",
      "Episode 719 reward: 19.3\n",
      "Episode 729 reward: 17.1\n",
      "Episode 739 reward: 18.8\n",
      "Episode 749 reward: 16.8\n",
      "Episode 759 reward: 15.0\n",
      "Episode 769 reward: 18.2\n",
      "Episode 779 reward: 17.4\n",
      "Episode 789 reward: 16.8\n",
      "Episode 799 reward: 15.0\n",
      "Episode 809 reward: 16.1\n",
      "Episode 819 reward: 17.6\n",
      "Episode 829 reward: 17.7\n",
      "Episode 839 reward: 17.1\n",
      "Episode 849 reward: 22.9\n",
      "Episode 859 reward: 16.8\n",
      "Episode 869 reward: 20.4\n",
      "Episode 879 reward: 13.3\n",
      "Episode 889 reward: 16.8\n",
      "Episode 899 reward: 20.6\n",
      "Episode 909 reward: 19.5\n",
      "Episode 919 reward: 19.9\n",
      "Episode 929 reward: 21.2\n",
      "Episode 939 reward: 20.0\n",
      "Episode 949 reward: 20.3\n",
      "Episode 959 reward: 21.3\n",
      "Episode 969 reward: 22.4\n",
      "Episode 979 reward: 22.7\n",
      "Episode 989 reward: 22.3\n",
      "Episode 999 reward: 23.0\n",
      "Saved Model\n",
      "Episode 1009 reward: 18.5\n",
      "Episode 1019 reward: 21.1\n",
      "Episode 1029 reward: 19.4\n",
      "Episode 1039 reward: 20.2\n",
      "Episode 1049 reward: 19.4\n",
      "Episode 1059 reward: 22.9\n",
      "Episode 1069 reward: 18.5\n",
      "Episode 1079 reward: 20.0\n",
      "Episode 1089 reward: 20.2\n",
      "Episode 1099 reward: 21.9\n",
      "Episode 1109 reward: 22.5\n",
      "Episode 1119 reward: 21.6\n",
      "Episode 1129 reward: 23.2\n",
      "Episode 1139 reward: 21.3\n",
      "Episode 1149 reward: 21.8\n",
      "Episode 1159 reward: 21.5\n",
      "Episode 1169 reward: 22.8\n",
      "Episode 1179 reward: 23.1\n",
      "Episode 1189 reward: 20.0\n",
      "Episode 1199 reward: 22.3\n",
      "Episode 1209 reward: 23.2\n",
      "Episode 1219 reward: 18.7\n",
      "Episode 1229 reward: 21.9\n",
      "Episode 1239 reward: 21.9\n",
      "Episode 1249 reward: 21.9\n",
      "Episode 1259 reward: 21.5\n",
      "Episode 1269 reward: 22.0\n",
      "Episode 1279 reward: 20.1\n",
      "Episode 1289 reward: 23.3\n",
      "Episode 1299 reward: 22.5\n",
      "Episode 1309 reward: 19.4\n",
      "Episode 1319 reward: 17.5\n",
      "Episode 1329 reward: 21.7\n",
      "Episode 1339 reward: 22.7\n",
      "Episode 1349 reward: 21.5\n",
      "Episode 1359 reward: 22.6\n",
      "Episode 1369 reward: 21.6\n",
      "Episode 1379 reward: 21.7\n",
      "Episode 1389 reward: 21.3\n",
      "Episode 1399 reward: 21.5\n",
      "Episode 1409 reward: 24.1\n",
      "Episode 1419 reward: 21.3\n",
      "Episode 1429 reward: 24.6\n",
      "Episode 1439 reward: 22.2\n",
      "Episode 1449 reward: 21.9\n",
      "Episode 1459 reward: 20.4\n",
      "Episode 1469 reward: 19.3\n",
      "Episode 1479 reward: 23.5\n",
      "Episode 1489 reward: 21.9\n",
      "Episode 1499 reward: 21.7\n",
      "Episode 1509 reward: 20.3\n",
      "Episode 1519 reward: 23.1\n",
      "Episode 1529 reward: 21.5\n",
      "Episode 1539 reward: 23.3\n",
      "Episode 1549 reward: 21.3\n",
      "Episode 1559 reward: 22.3\n",
      "Episode 1569 reward: 21.9\n",
      "Episode 1579 reward: 21.1\n",
      "Episode 1589 reward: 21.4\n",
      "Episode 1599 reward: 23.1\n",
      "Episode 1609 reward: 21.9\n",
      "Episode 1619 reward: 23.1\n",
      "Episode 1629 reward: 21.5\n",
      "Episode 1639 reward: 23.4\n",
      "Episode 1649 reward: 21.3\n",
      "Episode 1659 reward: 22.0\n",
      "Episode 1669 reward: 23.2\n",
      "Episode 1679 reward: 22.5\n",
      "Episode 1689 reward: 22.0\n",
      "Episode 1699 reward: 21.7\n",
      "Episode 1709 reward: 22.8\n",
      "Episode 1719 reward: 20.6\n",
      "Episode 1729 reward: 20.6\n",
      "Episode 1739 reward: 20.4\n",
      "Episode 1749 reward: 18.5\n",
      "Episode 1759 reward: 24.0\n",
      "Episode 1769 reward: 21.5\n",
      "Episode 1779 reward: 20.7\n",
      "Episode 1789 reward: 22.7\n",
      "Episode 1799 reward: 22.3\n",
      "Episode 1809 reward: 23.4\n",
      "Episode 1819 reward: 22.2\n",
      "Episode 1829 reward: 22.9\n",
      "Episode 1839 reward: 22.5\n",
      "Episode 1849 reward: 19.7\n",
      "Episode 1859 reward: 22.1\n",
      "Episode 1869 reward: 22.8\n",
      "Episode 1879 reward: 23.8\n",
      "Episode 1889 reward: 20.8\n",
      "Episode 1899 reward: 22.6\n",
      "Episode 1909 reward: 21.2\n",
      "Episode 1919 reward: 21.5\n",
      "Episode 1929 reward: 22.3\n",
      "Episode 1939 reward: 22.5\n",
      "Episode 1949 reward: 24.1\n",
      "Episode 1959 reward: 20.9\n",
      "Episode 1969 reward: 21.2\n",
      "Episode 1979 reward: 23.0\n",
      "Episode 1989 reward: 20.0\n",
      "Episode 1999 reward: 22.3\n",
      "Saved Model\n",
      "Episode 2009 reward: 22.5\n",
      "Episode 2019 reward: 23.3\n",
      "Episode 2029 reward: 20.1\n",
      "Episode 2039 reward: 21.0\n",
      "Episode 2049 reward: 21.8\n",
      "Episode 2059 reward: 21.0\n",
      "Episode 2069 reward: 21.5\n",
      "Episode 2079 reward: 21.4\n",
      "Episode 2089 reward: 23.6\n",
      "Episode 2099 reward: 21.7\n",
      "Episode 2109 reward: 22.7\n",
      "Episode 2119 reward: 21.9\n",
      "Episode 2129 reward: 22.1\n",
      "Episode 2139 reward: 22.9\n",
      "Episode 2149 reward: 22.7\n",
      "Episode 2159 reward: 23.3\n",
      "Episode 2169 reward: 23.6\n",
      "Episode 2179 reward: 21.8\n",
      "Episode 2189 reward: 20.4\n",
      "Episode 2199 reward: 21.6\n",
      "Episode 2209 reward: 19.8\n",
      "Episode 2219 reward: 19.6\n",
      "Episode 2229 reward: 22.7\n",
      "Episode 2239 reward: 20.2\n",
      "Episode 2249 reward: 22.1\n",
      "Episode 2259 reward: 21.6\n",
      "Episode 2269 reward: 22.4\n",
      "Episode 2279 reward: 22.7\n",
      "Episode 2289 reward: 22.2\n",
      "Episode 2299 reward: 22.2\n",
      "Episode 2309 reward: 25.2\n",
      "Episode 2319 reward: 20.7\n",
      "Episode 2329 reward: 23.9\n",
      "Episode 2339 reward: 23.5\n",
      "Episode 2349 reward: 21.4\n",
      "Episode 2359 reward: 20.9\n",
      "Episode 2369 reward: 22.2\n",
      "Episode 2379 reward: 21.3\n",
      "Episode 2389 reward: 21.0\n",
      "Episode 2399 reward: 22.1\n",
      "Episode 2409 reward: 23.5\n",
      "Episode 2419 reward: 19.8\n",
      "Episode 2429 reward: 21.7\n",
      "Episode 2439 reward: 22.7\n",
      "Episode 2449 reward: 21.5\n",
      "Episode 2459 reward: 21.6\n",
      "Episode 2469 reward: 21.9\n",
      "Episode 2479 reward: 22.3\n",
      "Episode 2489 reward: 20.7\n",
      "Episode 2499 reward: 20.2\n",
      "Episode 2509 reward: 22.9\n",
      "Episode 2519 reward: 23.3\n",
      "Episode 2529 reward: 22.1\n",
      "Episode 2539 reward: 23.3\n",
      "Episode 2549 reward: 22.0\n",
      "Episode 2559 reward: 22.6\n",
      "Episode 2569 reward: 24.4\n",
      "Episode 2579 reward: 21.9\n",
      "Episode 2589 reward: 23.9\n",
      "Episode 2599 reward: 20.7\n",
      "Episode 2609 reward: 23.0\n",
      "Episode 2619 reward: 21.6\n",
      "Episode 2629 reward: 21.7\n",
      "Episode 2639 reward: 21.6\n",
      "Episode 2649 reward: 22.2\n",
      "Episode 2659 reward: 22.6\n",
      "Episode 2669 reward: 22.5\n",
      "Episode 2679 reward: 22.8\n",
      "Episode 2689 reward: 24.2\n",
      "Episode 2699 reward: 20.7\n",
      "Episode 2709 reward: 24.8\n",
      "Episode 2719 reward: 22.4\n",
      "Episode 2729 reward: 23.6\n",
      "Episode 2739 reward: 22.8\n",
      "Episode 2749 reward: 22.1\n",
      "Episode 2759 reward: 22.7\n",
      "Episode 2769 reward: 21.3\n",
      "Episode 2779 reward: 20.6\n",
      "Episode 2789 reward: 23.7\n",
      "Episode 2799 reward: 22.0\n",
      "Episode 2809 reward: 22.5\n",
      "Episode 2819 reward: 22.3\n",
      "Episode 2829 reward: 23.8\n",
      "Episode 2839 reward: 23.5\n",
      "Episode 2849 reward: 22.0\n",
      "Episode 2859 reward: 22.8\n",
      "Episode 2869 reward: 21.4\n",
      "Episode 2879 reward: 21.1\n",
      "Episode 2889 reward: 21.8\n",
      "Episode 2899 reward: 19.2\n",
      "Episode 2909 reward: 19.3\n",
      "Episode 2919 reward: 24.3\n",
      "Episode 2929 reward: 21.4\n",
      "Episode 2939 reward: 21.7\n",
      "Episode 2949 reward: 24.0\n",
      "Episode 2959 reward: 22.4\n",
      "Episode 2969 reward: 22.2\n",
      "Episode 2979 reward: 23.0\n",
      "Episode 2989 reward: 21.6\n",
      "Episode 2999 reward: 22.0\n",
      "Saved Model\n",
      "Episode 3009 reward: 20.0\n",
      "Episode 3019 reward: 22.0\n",
      "Episode 3029 reward: 21.6\n",
      "Episode 3039 reward: 20.5\n",
      "Episode 3049 reward: 22.0\n",
      "Episode 3059 reward: 19.7\n",
      "Episode 3069 reward: 22.9\n",
      "Episode 3079 reward: 21.9\n",
      "Episode 3089 reward: 19.4\n",
      "Episode 3099 reward: 24.1\n",
      "Episode 3109 reward: 21.7\n",
      "Episode 3119 reward: 24.5\n",
      "Episode 3129 reward: 22.5\n",
      "Episode 3139 reward: 21.7\n",
      "Episode 3149 reward: 19.6\n",
      "Episode 3159 reward: 22.1\n",
      "Episode 3169 reward: 22.2\n",
      "Episode 3179 reward: 24.5\n",
      "Episode 3189 reward: 19.2\n",
      "Episode 3199 reward: 22.8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 3209 reward: 20.7\n",
      "Episode 3219 reward: 24.0\n",
      "Episode 3229 reward: 19.7\n",
      "Episode 3239 reward: 23.2\n",
      "Episode 3249 reward: 21.5\n",
      "Episode 3259 reward: 21.7\n",
      "Episode 3269 reward: 22.9\n",
      "Episode 3279 reward: 21.0\n",
      "Episode 3289 reward: 21.3\n",
      "Episode 3299 reward: 22.9\n",
      "Episode 3309 reward: 20.1\n",
      "Episode 3319 reward: 23.8\n",
      "Episode 3329 reward: 21.4\n",
      "Episode 3339 reward: 22.7\n",
      "Episode 3349 reward: 21.4\n",
      "Episode 3359 reward: 21.4\n",
      "Episode 3369 reward: 22.6\n",
      "Episode 3379 reward: 24.0\n",
      "Episode 3389 reward: 22.7\n",
      "Episode 3399 reward: 22.5\n",
      "Episode 3409 reward: 20.5\n",
      "Episode 3419 reward: 22.3\n",
      "Episode 3429 reward: 21.2\n",
      "Episode 3439 reward: 20.2\n",
      "Episode 3449 reward: 22.0\n",
      "Episode 3459 reward: 22.8\n",
      "Episode 3469 reward: 19.6\n",
      "Episode 3479 reward: 23.5\n",
      "Episode 3489 reward: 21.8\n",
      "Episode 3499 reward: 22.6\n",
      "Episode 3509 reward: 19.8\n",
      "Episode 3519 reward: 20.4\n",
      "Episode 3529 reward: 20.9\n",
      "Episode 3539 reward: 22.4\n",
      "Episode 3549 reward: 20.0\n",
      "Episode 3559 reward: 21.7\n",
      "Episode 3569 reward: 22.7\n",
      "Episode 3579 reward: 21.8\n",
      "Episode 3589 reward: 20.1\n",
      "Episode 3599 reward: 23.7\n",
      "Episode 3609 reward: 22.9\n",
      "Episode 3619 reward: 21.5\n",
      "Episode 3629 reward: 21.7\n",
      "Episode 3639 reward: 20.6\n",
      "Episode 3649 reward: 21.7\n",
      "Episode 3659 reward: 21.4\n",
      "Episode 3669 reward: 18.8\n",
      "Episode 3679 reward: 19.4\n",
      "Episode 3689 reward: 19.6\n",
      "Episode 3699 reward: 22.6\n",
      "Episode 3709 reward: 22.1\n",
      "Episode 3719 reward: 22.6\n",
      "Episode 3729 reward: 23.6\n",
      "Episode 3739 reward: 20.0\n",
      "Episode 3749 reward: 20.4\n",
      "Episode 3759 reward: 21.8\n",
      "Episode 3769 reward: 22.6\n",
      "Episode 3779 reward: 21.4\n",
      "Episode 3789 reward: 23.0\n",
      "Episode 3799 reward: 22.7\n",
      "Episode 3809 reward: 23.7\n",
      "Episode 3819 reward: 18.4\n",
      "Episode 3829 reward: 22.5\n",
      "Episode 3839 reward: 22.3\n",
      "Episode 3849 reward: 19.1\n",
      "Episode 3859 reward: 22.5\n",
      "Episode 3869 reward: 23.8\n",
      "Episode 3879 reward: 23.9\n",
      "Episode 3889 reward: 22.1\n",
      "Episode 3899 reward: 23.9\n",
      "Episode 3909 reward: 23.5\n",
      "Episode 3919 reward: 21.5\n",
      "Episode 3929 reward: 21.9\n",
      "Episode 3939 reward: 21.9\n",
      "Episode 3949 reward: 21.9\n",
      "Episode 3959 reward: 21.9\n",
      "Episode 3969 reward: 21.3\n",
      "Episode 3979 reward: 23.2\n",
      "Episode 3989 reward: 21.7\n",
      "Episode 3999 reward: 19.4\n",
      "Saved Model\n",
      "Episode 4009 reward: 19.7\n",
      "Episode 4019 reward: 22.9\n",
      "Episode 4029 reward: 23.5\n",
      "Episode 4039 reward: 23.6\n",
      "Episode 4049 reward: 24.1\n",
      "Episode 4059 reward: 23.4\n",
      "Episode 4069 reward: 19.3\n",
      "Episode 4079 reward: 22.9\n",
      "Episode 4089 reward: 20.1\n",
      "Episode 4099 reward: 21.7\n",
      "Episode 4109 reward: 20.1\n",
      "Episode 4119 reward: 20.0\n",
      "Episode 4129 reward: 23.0\n",
      "Episode 4139 reward: 21.9\n",
      "Episode 4149 reward: 23.2\n",
      "Episode 4159 reward: 20.0\n",
      "Episode 4169 reward: 21.1\n",
      "Episode 4179 reward: 23.6\n",
      "Episode 4189 reward: 20.2\n",
      "Episode 4199 reward: 22.8\n",
      "Episode 4209 reward: 20.0\n",
      "Episode 4219 reward: 22.8\n",
      "Episode 4229 reward: 22.6\n",
      "Episode 4239 reward: 22.0\n",
      "Episode 4249 reward: 20.1\n",
      "Episode 4259 reward: 21.1\n",
      "Episode 4269 reward: 22.9\n",
      "Episode 4279 reward: 23.4\n",
      "Episode 4289 reward: 18.7\n",
      "Episode 4299 reward: 24.4\n",
      "Episode 4309 reward: 22.5\n",
      "Episode 4319 reward: 21.3\n",
      "Episode 4329 reward: 23.3\n",
      "Episode 4339 reward: 21.2\n",
      "Episode 4349 reward: 24.1\n",
      "Episode 4359 reward: 21.8\n",
      "Episode 4369 reward: 23.7\n",
      "Episode 4379 reward: 21.5\n",
      "Episode 4389 reward: 21.6\n",
      "Episode 4399 reward: 22.6\n",
      "Episode 4409 reward: 22.1\n",
      "Episode 4419 reward: 22.8\n",
      "Episode 4429 reward: 20.6\n",
      "Episode 4439 reward: 23.8\n",
      "Episode 4449 reward: 23.3\n",
      "Episode 4459 reward: 20.5\n",
      "Episode 4469 reward: 21.8\n",
      "Episode 4479 reward: 24.0\n",
      "Episode 4489 reward: 20.4\n",
      "Episode 4499 reward: 22.1\n",
      "Episode 4509 reward: 19.4\n",
      "Episode 4519 reward: 21.5\n",
      "Episode 4529 reward: 23.1\n",
      "Episode 4539 reward: 22.5\n",
      "Episode 4549 reward: 22.3\n",
      "Episode 4559 reward: 23.9\n",
      "Episode 4569 reward: 22.2\n",
      "Episode 4579 reward: 21.9\n",
      "Episode 4589 reward: 23.7\n",
      "Episode 4599 reward: 21.7\n",
      "Episode 4609 reward: 22.6\n",
      "Episode 4619 reward: 23.4\n",
      "Episode 4629 reward: 24.5\n",
      "Episode 4639 reward: 22.0\n",
      "Episode 4649 reward: 19.1\n",
      "Episode 4659 reward: 20.6\n",
      "Episode 4669 reward: 21.2\n",
      "Episode 4679 reward: 23.4\n",
      "Episode 4689 reward: 21.5\n",
      "Episode 4699 reward: 21.8\n",
      "Episode 4709 reward: 22.2\n",
      "Episode 4719 reward: 24.0\n",
      "Episode 4729 reward: 21.7\n",
      "Episode 4739 reward: 22.3\n",
      "Episode 4749 reward: 24.2\n",
      "Episode 4759 reward: 21.9\n",
      "Episode 4769 reward: 21.9\n",
      "Episode 4779 reward: 20.4\n",
      "Episode 4789 reward: 23.3\n",
      "Episode 4799 reward: 24.0\n",
      "Episode 4809 reward: 21.9\n",
      "Episode 4819 reward: 23.1\n",
      "Episode 4829 reward: 23.8\n",
      "Episode 4839 reward: 22.8\n",
      "Episode 4849 reward: 21.7\n",
      "Episode 4859 reward: 21.9\n",
      "Episode 4869 reward: 23.2\n",
      "Episode 4879 reward: 18.6\n",
      "Episode 4889 reward: 21.4\n",
      "Episode 4899 reward: 23.5\n",
      "Episode 4909 reward: 20.1\n",
      "Episode 4919 reward: 24.0\n",
      "Episode 4929 reward: 22.9\n",
      "Episode 4939 reward: 22.6\n",
      "Episode 4949 reward: 22.4\n",
      "Episode 4959 reward: 23.9\n",
      "Episode 4969 reward: 23.3\n",
      "Episode 4979 reward: 21.6\n",
      "Episode 4989 reward: 22.6\n",
      "Episode 4999 reward: 23.6\n",
      "Mean reward per episode: 19.4356\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "mainQN = Qnetwork(h_size)\n",
    "targetQN = Qnetwork(h_size)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "trainables = tf.trainable_variables()\n",
    "\n",
    "targetOps = updateTargetGraph(trainables,tau)\n",
    "\n",
    "myBuffer = experience_buffer()\n",
    "\n",
    "#Set the rate of random action decrease. \n",
    "e = startE\n",
    "stepDrop = (startE - endE)/annealing_steps\n",
    "\n",
    "#create lists to contain total rewards and steps per episode\n",
    "jList = []\n",
    "rList = []\n",
    "total_steps = 0\n",
    "\n",
    "#Make a path for our model to be saved in.\n",
    "if not os.path.exists(path):\n",
    "    os.makedirs(path)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    if load_model == True:\n",
    "        print('Loading Model...')\n",
    "        ckpt = tf.train.get_checkpoint_state(path)\n",
    "        saver.restore(sess,ckpt.model_checkpoint_path)\n",
    "    for i in range(num_episodes):\n",
    "        episodeBuffer = experience_buffer()\n",
    "        #Reset environment and get first new observation\n",
    "        s = env.reset()\n",
    "        s = processState(s)\n",
    "        d = False\n",
    "        rAll = 0\n",
    "        j = 0\n",
    "        #The Q-Network\n",
    "        while j < max_epLength: #If the agent takes longer than 50 moves to reach either of the blocks, end the trial.\n",
    "            j+=1\n",
    "            #Choose an action by greedily (with e chance of random action) from the Q-network\n",
    "            if np.random.rand(1) < e or total_steps < pre_train_steps:\n",
    "                a = np.random.randint(0,4)\n",
    "            else:\n",
    "                a = sess.run(mainQN.predict,feed_dict={mainQN.scalarInput:[s]})[0]\n",
    "            total_steps += 1\n",
    "            \n",
    "            ################################################################################\n",
    "            # TODO: Save the experience to our episode buffer.                             #\n",
    "            # You will need to do the following:                                           #\n",
    "            # (1) Get new state s1 (resized), reward r and done d from a                   #\n",
    "            # (2) Add experience to episode buffer. Hint: experience includes              #\n",
    "            #     s, a, r, s1 and d.                                                       #\n",
    "            ################################################################################\n",
    "            s1, r, d = env.step(a)\n",
    "            s1 = processState(s1)\n",
    "            episodeBuffer.add(np.reshape(np.array([s,a,r,s1,d]),[1,5]))\n",
    "            ################################################################################\n",
    "            #                                 END OF YOUR CODE                             #\n",
    "            ################################################################################\n",
    "            \n",
    "            if total_steps > pre_train_steps:\n",
    "                if e > endE:\n",
    "                    e -= stepDrop\n",
    "                \n",
    "                if total_steps % (update_freq) == 0:\n",
    "                    \n",
    "                    ################################################################################\n",
    "                    # TODO: Implement Double-DQN                                                   #\n",
    "                    # (1) Get a random batch of experiences via experience_buffer class            #\n",
    "                    #                                                                              #\n",
    "                    # (2) Perform the Double-DQN update to the target Q-values                     #\n",
    "                    #     Hint: Use mainQN and targetQN separately to chose an action and predict  #\n",
    "                    #     the Q-values for that action.                                            #\n",
    "                    #     Then compute targetQ based on Double-DQN equation                        #\n",
    "                    #                                                                              #\n",
    "                    # (3) Update the primary network with our target values                        #\n",
    "                    ################################################################################ \n",
    "                    # (1)\n",
    "                    batch = myBuffer.sample(batch_size)\n",
    "                    s_batch, a_batch, r_batch, s1_batch, d_batch = batch[:,0], batch[:,1], batch[:,2], batch[:,3], batch[:,4]\n",
    "                    \n",
    "                    # Run 2 different networks\n",
    "                    Q1 = sess.run(mainQN.predict, feed_dict={mainQN.scalarInput:np.vstack(s1_batch)})\n",
    "                    Q2 = sess.run(targetQN.Qout, feed_dict={targetQN.scalarInput:np.vstack(s1_batch)})\n",
    "                    doubleQ = Q2[range(batch_size), Q1]\n",
    "                    \n",
    "                    targetQ = r_batch + (y * doubleQ * (-(d_batch - 1)))\n",
    "                    \n",
    "                    _ = sess.run(mainQN.updateModel, feed_dict={mainQN.targetQ:targetQ, mainQN.scalarInput:np.vstack(s_batch), mainQN.actions:a_batch})\n",
    "                    \n",
    "                    ################################################################################\n",
    "                    #                                 END OF YOUR CODE                             #\n",
    "                    ################################################################################\n",
    "                           \n",
    "                    updateTarget(targetOps,sess) #Update the target network toward the primary network.\n",
    "            rAll += r\n",
    "            s = s1\n",
    "            \n",
    "            if d == True:\n",
    "\n",
    "                break\n",
    "        \n",
    "        myBuffer.add(episodeBuffer.buffer)\n",
    "        jList.append(j)\n",
    "        rList.append(rAll)\n",
    "        #Periodically save the model. \n",
    "        if i % 1000 == 0:\n",
    "            saver.save(sess,path+'/model-'+str(i)+'.ckpt')\n",
    "            print(\"Saved Model\")\n",
    "        if len(rList) % 10 == 0:\n",
    "            print(\"Episode\",i,\"reward:\",np.mean(rList[-10:]))\n",
    "    saver.save(sess,path+'/model-'+str(i)+'.ckpt')\n",
    "print(\"Mean reward per episode: \" + str(sum(rList)/num_episodes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes about 40 minutes to train 5000 episodes in Lab 4 machines. Mean reward per episode (50 steps) should be around 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking network learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mean reward over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fca3260a438>]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4lNXd//H3yUYWIAlkIRBI2CQs\nIktAxRVcUSvWfWux2p/WpdU+9qlb+7R1ae2m1YpW64ZWrahF3BUFxQU1YRFCWBIggewJIXsmy8z5\n/ZGBBkhISCaZzOTzuq5cmeXOzPdOJp85c+5zzm2stYiIiO8L8HYBIiLiGQp0ERE/oUAXEfETCnQR\nET+hQBcR8RMKdBERP6FAFxHxEwp0ERE/oUAXEfETQb35ZDExMTY5Obk3n1JExOetWbOmzFob29F2\nvRroycnJpKen9+ZTioj4PGNMbme2U5eLiIifUKCLiPgJBbqIiJ9QoIuI+AkFuoiIn1Cgi4j4CQW6\niIifUKCLtFLb0MyLX+dSUdfo7VLETzQ0O1m0MhtHk7PHn0uBLtLKXz7ayq/fzOD0h1bx3sZCOjrn\nbrPTxfsbC9lSVNVLFYq35VfUsyGvolPbFlbWc9mTX/PnD7eycktJD1fWyzNFRfqyrOJqXlidy5mT\n4imsdHDTS2s5a3I89y2YQtzg0AO2dTQ5eWNtHv/4bDu7y+sJDQ7g0cunc+bkYV6qXnrDmty9XLc4\njYq6Ji6cPoK7z51IzMABbW771fYyfvryOhxNTv5x9QzOnpLQ4/WphS4CWGu5951MwkMC+cOFR7P0\npjncNT+FT7eWctpDn/Fq2i6stdQ2NPP05zs4+U8ruWdpBkMiBvDI5dOYMGwwN/xrDc9+sdPbu9Kv\nWWupcjT1yGMvzyzmqqe/JjIsmOtPHsPbGwqY95dPeembXFyu/36Ss9by1Krt/OCZb4kKD2bZLSf2\nSpgDmI4+UnpSamqq1Vouvs1aS35FPYnR4Z3+mS1FVThdlsnDI7v13JX1TbzwVQ4XzUxkeFRYtx7r\nYB9nFvPjF9L5v/Mmce2Jo/ffvrOsljvf2MA3O8uZNjKKnD21VNQ1MWfsUG6eO445Y4dijKG+0clt\nr67jw03FXDMnmV+fN4nAAOPRGo9EfaOTusZmhrbTevRHVY4mbn5pLZ9nlTFtZBTnTU3g3KkJJER2\n/7Xyyre7uGfpRqaMiOTZa2YRM3AA2SXV/OrNDL7eUc70UVHcf8EUkoZGcMfrG3h3YyHzpwzjz5cc\nw8AB3e8IMcassdamdridAl0666vtZTy8fBtpOXu5a34KN5wytsOf2bWnjnMf/ZzqhmYunpnIL8+e\nQNyg0A5/7mCbC6v4yb/WkLunjqPiB/LaT+YQGRZ82J+x1vLnD7dS1+jkV+dOJCiw7Q+kDc1Oznp4\nFYEBhg9uO5ngg7ZzuSz/TtvN31dkMXn4YG6aO44Zo6IPeRyny/Lg+5v55+c7OS0ljkevmE5Eq39m\nay25e+pIz91LbUMzPzguiQAPh77LZXljbR5/+nAr9Y1Onl6YynFjhnr0OdpT09DMZ1tLWZ5ZRF2j\nk9vPnMCEYYN65bnzK+q59rk0tpfWcOWxo1iTu5dNBS3HNVKTojlvagLzUuKpbmgif289ee6v/Io6\niqoamJQwmPlThnH82KEH/P2ttfzt4ywe+SSLUyfEsujKGYf8TZeuy+eBdzezt66RYYNDKapy8Muz\nU7jh5DEY45m/rwJdOiUtp5yvsveQmhzNjFHRhIUEHrLNtzvLeWj5Vr7eUU784AGMjA5n3e4KXv7x\nsRx7mLBoaHZyyT9Wk1NWy0UzE/nX17kMCArk1tPGs3BOMiFBnevxW7Y+nzve2MDg0JaPun/8YAuz\nRw/huWtmt/sY1lp+93Ymz3+VA8D8KcN45PLpbW7/5Gfb+cP7W1h87WxOOarDFUo79OLXufxmWQYT\nEwZz9zkT2VxYRXrOXtJz91JW07B/u5/OG8ftZ07o8PH2hUZ0RAjHjxlKaPChfyOAb3bs4b53M8nI\nr2LayChqGprZXV7HE1fPYF5KfLf3qy3FVQ6WZxazPLOY1dv30Oh0ER0ejAVqHM1cd9Jobj1tPOEh\nbbdS6xudLEnfzX/W5fPD45K4aGbiEdewMa+Saxen4Wh08sTVMzlxfAzQ8unq3Q0FvLOhkC1F1Yf8\nXHhIICOiwhg6MIQNeZXUNTqJDAvm9Inx+8P93rczeTV9N5fMTOT3Fx59yJv9PhV1jfzxg618kV3K\ngxdO5YRxMUe8H4ejQPdjhZX13PnGRk6fFM8lMxPb/QfvSEm1g7MeXsXeupY+x+BAw9TEKI4dPYRj\nxwwlNCiAx1Zm83lWGTEDB3Dz3LFcMXsUTU4X5z/2JbUNzbz7s5OIHdT2x/rfvb2J577M4ckfzOSs\nycPYWVbLfe9ksmJLCWNiI/jN9yYfNkCbnC4eeHczz3+Vw6zkaBZdOYO4waG8sSaP21/7jktmJvKn\ni6ce0gqy1vLg+1t4ctUOrj1hNInRYdz7TianpcSx6KoZB/y+SqodzPvLZxw3ZghPL5zVpd9jW1Zu\nKeHml9dS19gyVG3kkDBSk4YwMyma1ORonv1iJ0vS83jk8mksmDai3cex1vL791pa/QChwQGcMDaG\nuSlxzE2JY0RUGLvL6/jD+5t5b2MRCZGh3Dk/hfOPGc7euiYWPvstmwureOiyaZx/zHCP7Z+1lrv+\ns5F/p+0GIGloOGdMjOeMSfHMTIqmytHMg+9vZkl6HiOiwvjt+ZM5Y9J/31TKaxt5YXUOi7/KYW9d\nEzEDQyiraeTHJ47mzvkp7X6aOtjyzGJ+9so6hkSE8NyPZnFUfNufCLJLqvl6RzlDI0IYER1GYnQ4\n0eHB+187jiYnq7aV8sGmIj7OLKbK0UxggMHpstwydxy3n3mUx1rbXaFA92MPfbSVR1dkAxA7aAA/\nPnE0Vx2XdER9ddZafrw4nc+zy3j1+uOoqG/imx3lfLNzDxvzKml2H+QZGhHCjaeO5apjkw5ovW8u\nrOKCRV8yMymaF6879pD+4g83FXHDi2u4Zk4yvz1/8gH3rdhSzH3vbGZnWS2zk4dwzMhIxscPYkL8\nIMbFDSRiQBAlVQ5ufnktaTl7+dEJydx9zsQDWkcPL9/GI59kcfsZR/HT08Yf+PtZvo1HP8ni6uNG\ncd+CKRhj+NfXufzqzQxOGh/DUz9I3b8v//vad7y5Pp+Pfn4Ko2MiOv3764ycslq2FFUzfVQU8QeN\nkmlsdnH109+wPq+CJTccz7SRUW0+xl8/2srfV2Tzw+OTOG1iPCu3lLBiSwm7yusAGB83kNw9dQQG\nGH5yyliuP3nMAX+nakcT1y1OJy2nnAcuOJorjx3lkX17b2MhN720litmj+LaE5IZFzewzcBLyynn\nnqUb2VZcwxmT4vnJKWN4+7tCXk3bTX2Tk9MnxnHDKWOZNjJq/5v3SeNjeOyKGUSGH75L7bkvd3Lv\nO5lMHRHJPxemdqkrry2NzS5W79jDis3FTB8VzQXT23/D7S0KdD9lreXkP68kaUgEN80dy+Mrt/NF\ndhmRYcFcMyeZa+YkEx0R0uHj/PvbXdz5n438+rxJXNfqICC0TK5Zu2svxVUNzJ8y7IA+w9ZeS9/N\n/76+4ZCug93lLf3mSUMjeP3G4xkQdOgniIZmJ89/mcOy9QVkl9bQ2Ozaf19idBh17oN6f7xoapst\nWGstty/5jv+sy+dvl03b/0+3aGU2f/5wK5emJvLghVMP6KNekr6bO97YwHGjh/L0wlSyS2pYsOhL\nbjhlDHfNn9jh78zT9tQ0sGDRlzQ0u3jrlhMOOXj390+y+OvybVwxeyQPXHD0/n2x1rK9tJaVW0pY\nlVVKQmQoPz/jqHYP/tU3OrnxpTV8urWUu89J4fqTOz72cTgVdY2c/tBnDIsM5c2bTuiwNd3kdPHM\nFzt55OMs6pucBAcaFkwbwQ0nj2H8QS3qV9N28as3M0iMDuefP5zJuLgD769paGbllhKWrS/g483F\nnDkpnkcun95mV6E/UaD7qTW55Vz0xGr+eskx+/sb1++u4PGV2XyUWUx4SCD3nDuRq45Navcxdu2p\nY/4jqzhmZBT/uu7Ybh2Y++Xr37EkPY/nfjSLuRPiaGx2cemTq9leUsO7PzuJUUM7Hg3jdFly99Sy\nrbiGrOJqtpXUUO1o4o6zU5iYMLjdn2tsdvHDZ79hbW4FL1w3m4z8Su5/dzMXTBvOXy+d1uYok2Xr\n8/mfJd8xfWQUzS5L3t56Vv7iFAaFHr412FO2FlVz0RNfkRwTzpIbjt/f1/zUqu38/r0tXDh9BH+5\n5JhuHzxtbHbx8yXreXdDITeeOpbbTh/f5httZ/zPkvW8tb6At245kUnD2//7HCxvbx0rt5Zy+sS4\nw448WZNbzg0vrsXR5OSRy6eRmjSE5ZuL+SCjiFVZpTQ2u4gZGMKVxyZx62njvTqaqLco0P3UPUs3\n8sbaPNJ/dcYhXSxbi6q5/91MPs8q49oTRnPPuRMPebE7XZbLnlzN1qJqPvj5yYzo5vA/R5OTCxZ9\nSVGVg3d/dhLPf7mTf36+k8evmsE5R/f82NvKuiYufOJLCisd1DU6OefoYTx6+fTDthrf21jIz15Z\nR7PL8qeLp3Jp6sger/NwVmwp5rrF6cyfMozHrpjBC6tz+O3bmZw3NYG/XTat0/3JHXG6LPcsben3\njh88gOtOHM0Vs0cd0ZvZp1tLuOa5NG6ZO45fnNXxAd2uKqio54YX15BRUEmgMTS7LMMjQzlryjDm\nT0lgZlJ0vwjyfRTofqix2cWsBz7m1AmxPHL59Da3cbos97+byXNf5nBaShyPXDH9gOB/4tPt/PGD\nLTx06TFcOOPIRxS0ZWdZLd/7+xcMiQhhV3kdPzw+iXsXTPHIY3fG7vI6Lv7HV0wbGcVjV85odyRC\na59tK+Wr7WXccVaKx4cOdsU/V+3ggfc2c9L4GD7PKuPMSfEsuqpz+3IkrLWsyirjyc+289X2PQwK\nDeLq45L40ZzkQ2bDHqymoZmzHl5FaHAA7916Updb+J1V3+jk7yuycFrL/CkJHJMY6dUDk96kQPdD\nH20q4voX1+zv3jicF7/O5bdvbeKo+EE8szCV4VFhZBZUsWDRF5w+MZ7Hr5rh0X+O9zcWcuNLa5k8\nfDBv3DinyyNvuqrJ6SIowPjsP7y1ll++voHX1uQxd0Is//jBzB4PzA15FTz52Q7ezygkKCCAC2eM\n4KZTx7XbTfabZRm88HUur//keGYmDenR2uRACnQfsmpbKcVVDi7p4KP/jf9aQ1pOOV/fdVqnPoZ/\ntq2UW15aS2hIII9fNYNfv5lBWU0jH/38ZIZ04sDpkfoiq4wJwwa1O4xRDq+x2cWKLcWcOiGuV98Q\nc8pqefqLHbyWnofTZbl4ZiK3zBt3wGzgtJxyLvnH6jZHLUnPU6D7kIue+Iq1u/ay7OYTmJrY9vC1\nyvomZt3/MVcdN4rffK/z/1Dbiqu59vk08vbWA/DsNak9NslEfFtJlYPHP93Oy9/swmK5NHUkN88d\nx5CIEM555HManS4+vO3kdkc9Sc/pbKDrL+NlTpcls6AKa+HupRtZdvOJbR7seX9jIY1OF98/wjGx\nR8UP4s2bT+AXr31HyrDBCnNpV9zgUH57/mRuOGUMi1Zm82rabl5Lz2PS8MHsKKvlxetmK8z7OK22\n6GU7y2qob3IyLyWOjPwqXlyd0+Z2S9flMyY2gqNHHPkCVzEDB/D8j2Zz5/yU7hUr/UJCZBj3X3A0\nK39xKhfNHEFGfiWXpY7kpPHdXxZBepbebr0sI79lAaH/PWsCTU4Xf/loG/OPTjhgZmHe3jq+2VnO\nL7w8/Vj6l8TocP5w4VTuODvFa+P05ciohe5lGfmVDAgKYHzcQO6/YAqNThf3vp15wDbL1hcAHHbN\nD5GeEhUe0q/GfPsyBbqXbSqoIiVhMEGBASQNjeCnc8fx7sZCVm5tOV3VvpX2ZiVHM3JI59cgF5H+\nR4HuRdZaMgoqmdJq+vT1p4xhTGwE/7csA0eTk00FVWSX1PD96Z6ZBCQi/kuB7kW7y+updjQzpdWB\nzgFBgdx/wRR2l9fz2Ipslq7LJyQwgHN7YRq9iPg2HRT1ooyCSgCmHHRqtjljY7hwxgieXLWdiAFB\nzE2J7XApURGRDlvoxpiRxpiVxphMY8wmY8yt7tuHGGOWG2Oy3N8PPSeXHFZGfiVBAYajhg085L67\nz5lIeEgQFXVN6m4RkU7pTJdLM3C7tXYScBxwszFmEnAn8Im1djzwifu6HIGMgiqOih/U5podMQMH\ncN8FU5iZFM3cFI3/FZGOddjlYq0tBArdl6uNMZuBEcAC4FT3ZouBT4E7eqRKP2StZVN+JfNS2l9k\n6/xjhnv0tGEi4t+O6KCoMSYZmA58A8S7wx6gCGhzTrkx5npjTLoxJr20tLQbpfqXoioHe2obDzgg\nKiLSHZ0OdGPMQOAN4DZrbVXr+2zLCl9trvJlrX3KWptqrU2NjVXXwT77ZohOGdH5M76IiBxOpwLd\nGBNMS5i/ZK39j/vmYmNMgvv+BKCkZ0r0Txn5lRjDYU+xJiJyJDozysUAzwCbrbUPtbrrLWCh+/JC\nYJnny/NfmwoqGRs7cP85JEVEuqszLfQTgB8A84wx691f5wAPAmcYY7KA093XpZMy8qsOmCEqItJd\nnRnl8gXQ3so8p3m2nP6htLqBoiqHDoiKiEdp6r8XbHLPEJ08XIEuIp6jQPeCTQUtI1wmqctFRDxI\nge4FmwoqSRoaTmSY1mcREc9RoHtBRn4Vk9U6FxEPU6D3ssq6JnaV16n/XEQ8ToHeyzYVupfM1QgX\nEfEwBXov2+Se8q8uFxHxNAV6L8soqCQhMpSYgQO8XYqI+BkFei/LyK9U/7mI9AgFei+qbWhmR1mt\nVlgUkR6hQO9FmwursPbQc4iKiHiCAr0XZeRrhIuI9BwFei/KKKgiZmAI8YN1QFREPE+B3kuanS7W\n5u5l0vBIWpaYFxHxLAV6L7DWcu87mewoq+X703XSZxHpGQr0XvD8Vzm8sDqX/3fSaL4/PdHb5YiI\nn1Kg97BPNhdz3zuZnDkpnjvnT/R2OSLixxToPSgjv5KfvrKOycMj+dvl0wgMUN+5iPQcBXoPKap0\n8OPF6USGBfP0wlSdDFpEepwCvQfUNjRz3eI0qh1NPHvNLOIHh3q7JBHpB9Rs9DBrLbe9up7NhVU8\ns3AWExM0zV9Eeoda6B62o6yW5ZnF/Pz0o5ibEuftckSkH1Gge1hWcQ0AJx8V6+VKRKS/UaB72PbS\nlkAfGzfQy5WISH+jQPew7JIaEiJDGThAhydEpHcp0D0su6SGcWqdi4gXKNA9yOWybC+tYWysAl1E\nep8C3YMKqxzUNTrVQhcRr1Cge1B2ScsBUQW6iHiDAt2DFOgi4k0KdA/KLqkhKjyYoREh3i5FRPoh\nBboHbS+pYVzsQJ2RSES8QoHuQdmlGrIoIt6jQPeQPTUNlNc2KtBFxGsU6B6y74CopvyLiLco0D0k\n272GyzhNKhIRL1Gge0h2SQ1hwYGMiArzdiki0k8p0D0ku6SGsXERBOi8oSLiJR0GujHmWWNMiTEm\no9VtvzXG5Btj1ru/zunZMvu+fUMWRUS8pTMt9OeBs9u4/WFr7TT313ueLcu31DY0U1Dp0AgXEfGq\nDgPdWrsKKO+FWnzWvpNaKNBFxJu604d+izFmg7tLJtpjFfkgreEiIn1BVwP9CWAsMA0oBP7a3obG\nmOuNMenGmPTS0tIuPl3fll1SQ1CAIWlohLdLEZF+rEuBbq0tttY6rbUu4J/A7MNs+5S1NtVamxob\n658nTs4uqSFpaDjBgRo0JCLe06UEMsYktLr6fSCjvW37A63hIiJ9QYdnMjbGvAKcCsQYY/KA3wCn\nGmOmARbIAW7owRr7tMZmF7l76pg/ZZi3SxGRfq7DQLfWXtHGzc/0QC0+KXdPLU6XVQtdRLxOnb7d\ntH+ES+wgL1ciIv2dAr2b/rvKoka4iIh3KdC7Kbu0hhFRYYSHdNh7JSLSoxTo3dSyKJf6z0XE+xTo\n3eByWbaXalEuEekbFOjdkF9Rj6PJpREuItInKNC7IVuLcolIH6JA74btWpRLRPoQBXo3ZJfUMCQi\nhCERId4uRUREgd4d2TpLkYj0IQr0LrLWkl2qIYsi0nco0LtoT20jFXVNjFegi0gfoUDvIp2lSET6\nGgV6F23IqwBgYsJgL1ciItJCgd5F3+7cy+iYCGIHDfB2KSIigAK9S1wuy5rcclKT+vW5sUWkj1Gg\nd8GOshr21jUxK3mIt0sREdlPgd4FaTl7AZg1WoEuIn2HAr0L0nLKiRkYQvLQcG+XIiKynwK9C9Jy\nyklNGoIxxtuliIjsp0A/QkWVDnaX15OarAOiItK3KNCPUHpuOYAOiIpIn6NAP0LpOXsJDwlk8nBN\nKBKRvkWBfoTScsqZPiqKoED96kSkb1EqHYFqRxObC6tITVJ3i4j0PQr0I7B2VwUuq/5zEembFOhH\nID2nnMAAw/RRUd4uRUTkEAr0I5CWU87k4YOJGBDk7VJERA6hQO+kxmYX63dXqP9cRPosBXonZRRU\n4mhyMUsTikSkj1Kgd1J6TsuEolQdEBWRPkqB3klpOTqhhYj0bQr0TrDWkp6jE1qISN+mQO+E7aW1\nOqGFiPR5CvROSNvff64Wuoj0XQr0Tth3QovRMRHeLkVEpF0K9E5Iz9mrE1qISJ+nQO9AcZWDXeV1\n6m4RkT5Pgd6B9H0nhNYBURHp4zoMdGPMs8aYEmNMRqvbhhhjlhtjstzf/bb5uiG/gpCgACbphBYi\n0sd1poX+PHD2QbfdCXxirR0PfOK+7pcKKhwMjwwlWCe0EJE+rsOUstauAsoPunkBsNh9eTFwgYfr\n6jMKK+oZFhnq7TJERDrU1WZnvLW20H25CIj3UD19TmGlg+GRYd4uQ0SkQ93uR7DWWsC2d78x5npj\nTLoxJr20tLS7T9ernC5LcZVDLXQR8QldDfRiY0wCgPt7SXsbWmufstamWmtTY2Nju/h03rGnpoFm\nlyUhSi10Een7uhrobwEL3ZcXAss8U07fUlDpACBhsFroItL3dWbY4ivAamCCMSbPGHMd8CBwhjEm\nCzjdfd3vFFXWA5AQpUAXkb6vw5NjWmuvaOeu0zxcS59TUOFuoeugqIj4AA2uPoyiKgcDggKIDg/2\ndikiIh1SoB9GQUU9CZGhWpRLRHyCAv0wiiod6m4REZ+hQD+MwkoHCRqDLiI+QoHejn2TijTCRUR8\nhQK9HWXuSUXD1OUiIj5Cgd6OQvekouHqchERH6FAb0dhRcukIq3jIiK+QoHejv+20NXlIiK+QYHe\njsLKegYEBRClSUUi4iMU6O0orHQwPCpMk4pExGco0NtRWOlgmFZZFBEfokBvR1GlxqCLiG9RoLfB\n6bIUVWmWqIj4FgV6G8pqGnC6rNZxERGfokBvQ4F7DLpa6CLiSxTobSiq1IktRMT3KNDbsO9cosN1\nUFREfIgCvQ1FlfWEBgcQGaZJRSLiOxTobSiodDA8UpOKRMS3KNDbUFTp0KJcIuJzFOhtKKyo1wFR\nEfE5CvSDOF2W4uoGDVkUEZ+jQD9IabV7UpFGuIiIj1GgH6SwUpOKRMQ3KdAPUqhJRSLioxToB/lv\noKuFLiK+RYF+kMKKesKCAzWpSER8jgL9IIXuZXM1qUhEfI0C/SCFFfUa4SIiPkmBfpCiSgfDBuuA\nqIj4HgV6K/smFWmVRRHxRQr0VvZNKtI6LiLiixTorRS4JxUN1xh0EfFBCvRW9p2pSC10EfFFCvRW\n9p1LVC10EfFFCvRWiiodhAUHMjgsyNuliIgcMQV6K4WVDhKiNKlIRHyTAr2Vwsp6reEiIj6rW30L\nxpgcoBpwAs3W2lRPFOUthZUOThgX4+0yRES6xBOdxXOttWUeeByvana6KNGZikTEh6nLxa20xn2m\nIo1wEREf1d1At8BHxpg1xpjrPVGQt2gddBHxdd3tcjnRWptvjIkDlhtjtlhrV7XewB301wOMGjWq\nm0/Xcwor3IGudVxExEd1q4Vurc13fy8BlgKz29jmKWttqrU2NTY2tjtP16P2n0tUKy2KiI/qcqAb\nYyKMMYP2XQbOBDI8VVhvK6x0EB6iSUUi4ru6k17xwFL3JJwg4GVr7QceqcoLiiodDNOZikTEh3U5\n0K21O4BjPFiL1zQ5XXybU87MUdHeLkVEpMs0bBFYsaWE0uoGLp6Z6O1SRES6TIEOvPLtLuIHD+DU\nCX33oK2ISEf6faDnV9Tz2bZSLksdSVBgv/91iIgP6/cJtiRtNwCXzhrp5UpERLqnXwe602VZkr6b\nk8bHkhgd7u1yRES6pV8H+mfbSiisdHDlbLXORcT39etAf+Xb3cQMHMBpE+O9XYqISLf120AvrnKw\nYksJF89MJFgHQ0XED/TbJHstfTdOl+VyHQwVET/RLwPd5bL8O203c8YOJTkmwtvliIh4RL8M9C+y\ny8jbW8/ls/vucr4iIkeqXwb6v9N2ER0ezFmTdTBURPxHvwv00uoGPtpUzEUzEhkQFOjtckREPKbf\nBfoba/Nodlku19hzEfEz/eZsDtZaPtlcwrNf7GR28hDGxQ3ydkkiIh7l94FureXTbaU8vHwbG/Iq\nSRoazl3npHi7LBERj/PbQLfW8kV2GQ8t38a6XRUkRofxp4uncuH0EVpVUUT8kt8FurWWVVllPLYi\ni7ScvQyPDOUPFx7NRTMSCQlSkIuI//KbQHe5LB9uKmLRp9lk5FcxbHAo9y2YzKWzRmo0i4j0Cz4f\n6E1OF2+tL+DxT7PZXlpL8tBw/njR0Xx/ulrkItK/+GSgu1yWjfmVLM8sZum6fPIr6kkZNohHr5jO\nuUcnEBhgvF2iiEiv85lAb2h2snr7HpZnFvPx5mKKqxoIMHDs6KHcu2Ay81LiMEZBLiL9l08E+qOf\nZPHUqh3UNDQTHhLIyeNjOWNSPPNS4oiOCPF2eSIifYJPBPqwyFC+d0wCZ0yKZ87YGEKDdZBTRORg\nPhHol6aO5NJUTdUXETkcDQMREfETCnQRET+hQBcR8RMKdBERP6FAFxHxEwp0ERE/oUAXEfETCnQR\nET9hrLW992TGlAK5XfzxGKCW+uAeAAADtklEQVTMg+X4Cu13/9Nf91373b4ka21sRw/Uq4HeHcaY\ndGttqrfr6G3a7/6nv+679rv71OUiIuInFOgiIn7ClwL9KW8X4CXa7/6nv+679rubfKYPXUREDs+X\nWugiInIYPhHoxpizjTFbjTHZxpg7vV1PTzHGPGuMKTHGZLS6bYgxZrkxJsv9PdqbNfYEY8xIY8xK\nY0ymMWaTMeZW9+1+ve/GmFBjzLfGmO/c+/079+2jjTHfuF/vrxpj/PK0XMaYQGPMOmPMO+7rfr/f\nxpgcY8xGY8x6Y0y6+zaPvc77fKAbYwKBRcB8YBJwhTFmkner6jHPA2cfdNudwCfW2vHAJ+7r/qYZ\nuN1aOwk4DrjZ/Tf2931vAOZZa48BpgFnG2OOA/4IPGytHQfsBa7zYo096VZgc6vr/WW/51prp7Ua\nquix13mfD3RgNpBtrd1hrW0E/g0s8HJNPcJauwooP+jmBcBi9+XFwAW9WlQvsNYWWmvXui9X0/JP\nPgI/33fbosZ9Ndj9ZYF5wOvu2/1uvwGMMYnAucDT7uuGfrDf7fDY69wXAn0EsLvV9Tz3bf1FvLW2\n0H25CIj3ZjE9zRiTDEwHvqEf7Lu722E9UAIsB7YDFdbaZvcm/vp6/xvwS8Dlvj6U/rHfFvjIGLPG\nGHO9+zaPvc594pyi0sJaa40xfjssyRgzEHgDuM1aW9XSaGvhr/turXUC04wxUcBSIMXLJfU4Y8x5\nQIm1do0x5lRv19PLTrTW5htj4oDlxpgtre/s7uvcF1ro+UDrM0Qnum/rL4qNMQkA7u8lXq6nRxhj\ngmkJ85estf9x39wv9h3AWlsBrASOB6KMMfsaW/74ej8BON8Yk0NLF+o84BH8f7+x1ua7v5fQ8gY+\nGw++zn0h0NOA8e4j4CHA5cBbXq6pN70FLHRfXggs82ItPcLdf/oMsNla+1Cru/x6340xse6WOcaY\nMOAMWo4frAQudm/md/ttrb3LWptorU2m5f95hbX2Kvx8v40xEcaYQfsuA2cCGXjwde4TE4uMMefQ\n0ucWCDxrrX3AyyX1CGPMK8CptKy+Vgz8BngTWAKMomWlykuttQcfOPVpxpgTgc+Bjfy3T/VuWvrR\n/XbfjTFTaTkIFkhL42qJtfZeY8wYWlquQ4B1wNXW2gbvVdpz3F0uv7DWnufv++3ev6Xuq0HAy9ba\nB4wxQ/HQ69wnAl1ERDrmC10uIiLSCQp0ERE/oUAXEfETCnQRET+hQBcR8RMKdBERP6FAFxHxEwp0\nERE/8f8BzD6pBY+EhWUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fca31fe4358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rMat = np.resize(np.array(rList),[len(rList)//100,100])\n",
    "rMean = np.average(rMat,1)\n",
    "plt.plot(rMean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inline Question:\n",
    "Try a basic DQN without Dueling DQN and Double DQN (i.e. only one single network, no decomposition of the Q-function). You don't need to provide detailed source, just some quantitative comparison is OK (e.g. by comparing the mean reward). **\n",
    "\n",
    "**Your answer:** Basic DQN converges slower than Dueling / Double DQN and has lower overall mean reward compared to DDQN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}