France_David_ProjectTwo .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Treasure Hunt Game Notebook\n",
    "\n",
    "## Read and Review Your Starter Code\n",
    "The theme of this project is a popular treasure hunt game in which the player needs to find the treasure before the pirate does. While you will not be developing the entire game, you will write the part of the game that represents the intelligent agent, which is a pirate in this case. The pirate will try to find the optimal path to the treasure using deep Q-learning. \n",
    "\n",
    "You have been provided with two Python classes and this notebook to help you with this assignment. The first class, TreasureMaze.py, represents the environment, which includes a maze object defined as a matrix. The second class, GameExperience.py, stores the episodes – that is, all the states that come in between the initial state and the terminal state. This is later used by the agent for learning by experience, called \"exploration\". This notebook shows how to play a game. Your task is to complete the deep Q-learning implementation for which a skeleton implementation has been provided. The code blocks you will need to complete has #TODO as a header.\n",
    "\n",
    "First, read and review the next few code and instruction blocks to understand the code that you have been given."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Treasure Hunt Game Notebook\n",
    "\n",
    "## Read and Review Your Starter Code\n",
    "The theme of this project is a popular treasure hunt game in which the player needs to find the treasure before the pirate does. While you will not be developing the entire game, you will write the part of the game that represents the intelligent agent, which is a pirate in this case. The pirate will try to find the optimal path to the treasure using deep Q-learning. \n",
    "\n",
    "You have been provided with two Python classes and this notebook to help you with this assignment. The first class, TreasureMaze.py, represents the environment, which includes a maze object defined as a matrix. The second class, GameExperience.py, stores the episodes – that is, all the states that come in between the initial state and the terminal state. This is later used by the agent for learning by experience, called \"exploration\". This notebook shows how to play a game. Your task is to complete the deep Q-learning implementation for which a skeleton implementation has been provided. The code blocks you will need to complete has #TODO as a header.\n",
    "\n",
    "First, read and review the next few code and instruction blocks to understand the code that you have been given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import os, sys, time, datetime, json, random\n",
    "import numpy as np\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "from keras.optimizers import SGD , Adam, RMSprop\n",
    "from keras.layers.advanced_activations import PReLU\n",
    "import matplotlib.pyplot as plt\n",
    "from TreasureMaze import TreasureMaze\n",
    "from GameExperience import GameExperience\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block contains an 8x8 matrix that will be used as a maze object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "maze = np.array([\n",
    "    [ 1.,  0.,  1.,  1.,  1.,  1.,  1.,  1.],\n",
    "    [ 1.,  0.,  1.,  1.,  1.,  0.,  1.,  1.],\n",
    "    [ 1.,  1.,  1.,  1.,  0.,  1.,  0.,  1.],\n",
    "    [ 1.,  1.,  1.,  0.,  1.,  1.,  1.,  1.],\n",
    "    [ 1.,  1.,  0.,  1.,  1.,  1.,  1.,  1.],\n",
    "    [ 1.,  1.,  1.,  0.,  1.,  0.,  0.,  0.],\n",
    "    [ 1.,  1.,  1.,  0.,  1.,  1.,  1.,  1.],\n",
    "    [ 1.,  1.,  1.,  1.,  0.,  1.,  1.,  1.]\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This helper function allows a visual representation of the maze object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show(qmaze):\n",
    "    plt.grid('on')\n",
    "    nrows, ncols = qmaze.maze.shape\n",
    "    ax = plt.gca()\n",
    "    ax.set_xticks(np.arange(0.5, nrows, 1))\n",
    "    ax.set_yticks(np.arange(0.5, ncols, 1))\n",
    "    ax.set_xticklabels([])\n",
    "    ax.set_yticklabels([])\n",
    "    canvas = np.copy(qmaze.maze)\n",
    "    for row,col in qmaze.visited:\n",
    "        canvas[row,col] = 0.6\n",
    "    pirate_row, pirate_col, _ = qmaze.state\n",
    "    canvas[pirate_row, pirate_col] = 0.3   # pirate cell\n",
    "    canvas[nrows-1, ncols-1] = 0.9 # treasure cell\n",
    "    img = plt.imshow(canvas, interpolation='none', cmap='gray')\n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pirate agent can move in four directions: left, right, up, and down. \n",
    "\n",
    "While the agent primarily learns by experience through exploitation, often, the agent can choose to explore the environment to find previously undiscovered paths. This is called \"exploration\" and is defined by epsilon. This value is typically a lower value such as 0.1, which means for every ten attempts, the agent will attempt to learn by experience nine times and will randomly explore a new path one time. You are encouraged to try various values for the exploration factor and see how the algorithm performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEFT = 0\n",
    "UP = 1\n",
    "RIGHT = 2\n",
    "DOWN = 3\n",
    "\n",
    "\n",
    "# Exploration factor\n",
    "epsilon = 0.8\n",
    "\n",
    "# Actions dictionary\n",
    "actions_dict = {\n",
    "    LEFT: 'left',\n",
    "    UP: 'up',\n",
    "    RIGHT: 'right',\n",
    "    DOWN: 'down',\n",
    "}\n",
    "\n",
    "num_actions = len(actions_dict)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample code block and output below show creating a maze object and performing one action (DOWN), which returns the reward. The resulting updated environment is visualized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reward= -0.04\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1b907792508>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAFtElEQVR4nO3dMWpUexjG4W8ugoUJKLmQxlIY+5kFTDpX4gpO5w5kUguuwFZcwJkFzBSW6SwCEkgjamVxbnEVFBJz5yb5Z97j88BUEd6TGX6YNPkmwzAUsPv+uusHAP4bsUIIsUIIsUIIsUIIsUKIe9v84729veHg4OC2nuUX3759q48fPzbZevr0aT148KDJ1tevX0e51XpvrFsfPnyo8/PzyUVf2yrWg4ODevHixc081RU+f/5cXdc12Xr16lUtFosmW6vVapRbrffGujWfzy/9mh+DIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIcRWf+T706dP9e7du9t6ll+0/OPU3IzNZlNHR0dNtvq+b7KzSyZXXT6fTCbPq+p5VdWjR49mL1++bPFctb+/X6enp022ptNp7e3tNdn68uXLKLeqqs7Oznxm19R1Xa3X6/93PmMYhtdV9bqq6uHDh8Pbt29v+PEutlgsmp3P6Pt+lKcYWp/POD4+9pndIr+zQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQgixQoitzmc8efKk2fmM1WpVV10LuMmtsZpMLvzj7rei7/tmn9nx8XGzUx3L5XIn/sj3VuczDg8PZ2/evGnxXKM9M9F66+TkpMlWVduTFi1PdTx+/LgODw+bbP3ufEYNw/CfX7PZbGil73tbN7BVVc1eLb+35XLZ7PtaLpfNvq/vjV3Yn99ZIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYTzGXew1eqkRcuzD1Xj/sxabTmfsWNbNcKzDz++N1vX43wGjIBYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYq2qz2dRkMmny2mw2W11BuM5rNpvd9VvLDXLrpqrOzs7q9PS0yVbL+zMt38PWe2PdcuvmCsvlcpT3Z1q+h633xrrl1g2MgFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFghhFirajabNT1p0fJUR0utz5CMdesyzmfcwdbJyUmTrZanOqranyEZ41bXdTUMg/MZu7JVIzzVMQztz5CMcevfJJ3PgGhihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRBihRD37voBGI8fZ0haWK1Wo9yaz+eXfs35jDvYGuv5jDF/Zq22uq6r9XrtfMaubNVIz2eM+TNr5XtjzmdAMrFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCOczRr7V6lRHVdV0Oh3t+3j//v0mW13X1fv37y88n3FlrD+bz+fDer2+sQf7ndVqVYvFwtY1t46OjppsVVX1fT/a93E6nTbZevbs2aWx+jEYQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQmx1PqOqplXV6h7D31V1bitmq/XeWLemwzDsX/SFrc5ntDSZTNbDMMxtZWy13vsTt/wYDCHECiF2OdbXtqK2Wu/9cVs7+zsr8Ktd/p8V+IlYIYRYIYRYIYRYIcQ/8eViVeWzLxQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "qmaze = TreasureMaze(maze)\n",
    "canvas, reward, game_over = qmaze.act(DOWN)\n",
    "print(\"reward=\", reward)\n",
    "show(qmaze)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function simulates a full game based on the provided trained model. The other parameters include the TreasureMaze object and the starting position of the pirate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_game(model, qmaze, pirate_cell):\n",
    "    qmaze.reset(pirate_cell)\n",
    "    envstate = qmaze.observe()\n",
    "    while True:\n",
    "        prev_envstate = envstate\n",
    "        # get next action\n",
    "        q = model.predict(prev_envstate)\n",
    "        action = np.argmax(q[0])\n",
    "\n",
    "        # apply action, get rewards and new state\n",
    "        envstate, reward, game_status = qmaze.act(action)\n",
    "        if game_status == 'win':\n",
    "            return True\n",
    "        elif game_status == 'lose':\n",
    "            return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function helps you to determine whether the pirate can win any game at all. If your maze is not well designed, the pirate may not win any game at all. In this case, your training would not yield any result. The provided maze in this notebook ensures that there is a path to win and you can run this method to check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def completion_check(model, qmaze):\n",
    "    for cell in qmaze.free_cells:\n",
    "        if not qmaze.valid_actions(cell):\n",
    "            return False\n",
    "        if not play_game(model, qmaze, cell):\n",
    "            return False\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code you have been given in this block will build the neural network model. Review the code and note the number of layers, as well as the activation, optimizer, and loss functions that are used to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(maze):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(maze.size, input_shape=(maze.size,)))\n",
    "    model.add(PReLU())\n",
    "    model.add(Dense(maze.size))\n",
    "    model.add(PReLU())\n",
    "    model.add(Dense(num_actions))\n",
    "    model.compile(optimizer='adam', loss='mse')\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# #TODO: Complete the Q-Training Algorithm Code Block\n",
    "\n",
    "This is your deep Q-learning implementation. The goal of your deep Q-learning implementation is to find the best possible navigation sequence that results in reaching the treasure cell while maximizing the reward. In your implementation, you need to determine the optimal number of epochs to achieve a 100% win rate.\n",
    "\n",
    "You will need to complete the section starting with #pseudocode. The pseudocode has been included for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def qtrain(model, maze, **opt):\n",
    "\n",
    "    # exploration factor\n",
    "    global epsilon \n",
    "\n",
    "    # Learning rate decreases epsilon by 5% for each epoch\n",
    "    learning_rate = 0.95\n",
    "\n",
    "    # number of epochs\n",
    "    n_epoch = opt.get('n_epoch', 15000)\n",
    "\n",
    "    # maximum memory to store episodes\n",
    "    max_memory = opt.get('max_memory', 1000)\n",
    "\n",
    "    # maximum data size for training\n",
    "    data_size = opt.get('data_size', 50)\n",
    "\n",
    "    # start time\n",
    "    start_time = datetime.datetime.now()\n",
    "\n",
    "    # Construct environment/game from numpy array: maze (see above)\n",
    "    qmaze = TreasureMaze(maze)\n",
    "\n",
    "    # Initialize experience replay object\n",
    "    experience = GameExperience(model, max_memory=max_memory)\n",
    "    \n",
    "    win_history = []   # history of win/lose game\n",
    "    hsize = qmaze.maze.size//2   # history window size\n",
    "    win_rate = 0.0\n",
    "    \n",
    "    \n",
    "    # For each epoch:\n",
    "    for epoch in range(n_epoch):\n",
    "        n_episodes = 0\n",
    "        loss = 0\n",
    "        \n",
    "        agent_cell = random.choice(qmaze.free_cells)\n",
    "\n",
    "    #    Reset the maze with agent set to above position\n",
    "        qmaze.reset(agent_cell)\n",
    "        envstate = qmaze.observe()\n",
    "        \n",
    "    #    While state is not game over:\n",
    "        while qmaze.game_status() == 'not_over':\n",
    "            previous_envstate = envstate\n",
    "            \n",
    "    #       Use epsilon value to determine exploration or exploitation. As epsilon decrease, \n",
    "    #       chance of exploitation increases\n",
    "            if np.random.rand() < epsilon:\n",
    "                action = np.random.randint(0, num_actions)\n",
    "            else:\n",
    "                q_values = model.predict(previous_envstate)\n",
    "                action = np.argmax(q_values[0])\n",
    "                                          \n",
    "            envstate, reward, game_status = qmaze.act(action)\n",
    "            episode = [previous_envstate, action, reward, envstate, game_status]\n",
    "            experience.remember(episode)\n",
    "        \n",
    "    #       Train neural network model and evaluate loss\n",
    "            exp_input, exp_target = experience.get_data(data_size=data_size)\n",
    "            loss += model.train_on_batch(exp_input, exp_target)\n",
    "            \n",
    "            n_episodes += 1\n",
    "            \n",
    "    #       If game result is a win or loss, move to next epoch\n",
    "            if game_status == 'win':\n",
    "                win_history.append(1)\n",
    "                epsilon *= learning_rate\n",
    "                break\n",
    "                \n",
    "            if game_status == 'lose':\n",
    "                win_history.append(0)\n",
    "                epsilon *= learning_rate\n",
    "                break\n",
    "                \n",
    "    #   Calculate win rate    \n",
    "        if len(win_history) > hsize:\n",
    "            win_rate = sum(win_history[-hsize:]) / hsize\n",
    "    \n",
    "    #   Print the epoch, loss, episodes, win count, and win rate for each epoch\n",
    "        dt = datetime.datetime.now() - start_time\n",
    "        t = format_time(dt.total_seconds())\n",
    "        template = \"Epoch: {:03d}/{:d} | Loss: {:.4f} | Episodes: {:d} | Win count: {:d} | Win rate: {:.3f} | time: {}\"\n",
    "        print(template.format(epoch, n_epoch-1, loss, n_episodes, sum(win_history), win_rate, t))\n",
    "        # Check if training has exhausted all free cells and if in all\n",
    "        # cases the agent won.\n",
    "        if win_rate > 0.9: \n",
    "            epsilon = 0.05\n",
    "            learning_rate = 1.0\n",
    "            \n",
    "        if sum(win_history[-hsize:]) == hsize and completion_check(model, qmaze):\n",
    "            print(\"Reached 100%% win rate at epoch: %d\" % (epoch,))\n",
    "            break\n",
    "    \n",
    "    \n",
    "    # Determine the total time for training\n",
    "    dt = datetime.datetime.now() - start_time\n",
    "    seconds = dt.total_seconds()\n",
    "    t = format_time(seconds)\n",
    "\n",
    "    print(\"n_epoch: %d, max_mem: %d, data: %d, time: %s\" % (epoch, max_memory, data_size, t))\n",
    "    return seconds\n",
    "\n",
    "# This is a small utility for printing readable time strings:\n",
    "def format_time(seconds):\n",
    "    if seconds < 400:\n",
    "        s = float(seconds)\n",
    "        return \"%.1f seconds\" % (s,)\n",
    "    elif seconds < 4000:\n",
    "        m = seconds / 60.0\n",
    "        return \"%.2f minutes\" % (m,)\n",
    "    else:\n",
    "        h = seconds / 3600.0\n",
    "        return \"%.2f hours\" % (h,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Your Model\n",
    "\n",
    "Now we will start testing the deep Q-learning implementation. To begin, select **Cell**, then **Run All** from the menu bar. This will run your notebook. As it runs, you should see output begin to appear beneath the next few cells. The code below creates an instance of TreasureMaze."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1b907804bc8>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAFeklEQVR4nO3dv2qUaRjG4edbRGF0u4U0lsLYz7TCpPNIPILvMMZa2COw9wBmDmC+wjKdRUACKbX+tlgFhWRjSPbN3K/XBVONcM8ffpg0eYZ5ngs4fn889AsAfo1YIYRYIYRYIYRYIYRYIcSj2/zjx48fz4vF4v96LT9ZLBb1+fPnJlsvX76sp0+fNtn6+vVrl1ut93rd+vTpU11eXg5XPXerWBeLRb169ep+XtUNNptNjePYZOvdu3e12WyabO33+y63Wu/1urVer699zo/BEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEOJWf+T7xYsX9eHDh//rtfzk7du3TXa4P9M01enpaZOt3W7XZOeYDDddPh+G4U1VvamqOjk5Wb1//77F66qLi4s6Pz9vsrVcLuvZs2dNtr58+dLlVpXv7D6M41iHw+HK8xk1z/MvP1ar1dzKdrudq6rJY7fbNXtfvW7Ns+/sPnxr7Mr+/M4KIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIY421tVqdas/QH6XR8+GYWj2aPmdTdPU7H1N0/TQX2NVHfH5jF7PTLTeOjs7a7JV1fakRctTHc+fP6+Tk5MmW5HnM3o9j9B6qxqds6jGJy1anurYbrfN3pfzGdABsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUII5zMeYKvVSYuWZx+q+v7OWm05n3FkW9Xh2Yfv783W3TifAR0QK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQa1VN01TDMDR5TNN0qysId3msVquH/mi5R27dVNXFxUWdn5832Wp5f6blZ9h6r9ctt25usN1uu7w/0/IzbL3X65ZbN9ABsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsUIIsVbVarVqetKi5amOllqfIel16zrOZzzA1tnZWZOtlqc6qtqfIelxaxzHmufZ+Yxj2aoOT3XMc/szJD1u/Zuk8xkQTawQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQQqwQ4tFDvwD68f0MSQv7/b7LrfV6fe1zzmc8wFav5zN6/s5abY3jWIfDwfmMY9mqTs9n9PydtfKtMeczIJlYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYRYIYTzGZ1vtTrVUVW1XC67/RyfPHnSZGscx/r48eOV5zNujPVH6/V6PhwO9/bC/st+v6/NZmPrjlunp6dNtqqqdrtdt5/jcrlssvX69etrY/VjMIQQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4S41fmMqlpWVat7DH9V1aWtmK3We71uLed5/vOqJ251PqOlYRgO8zyvbWVstd77Hbf8GAwhxAohjjnWv21FbbXe++22jvZ3VuBnx/w/K/ADsUIIsUIIsUIIsUKIfwCZS8E/wRnKUQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "qmaze = TreasureMaze(maze)\n",
    "show(qmaze)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next code block, you will build your model and train it using deep Q-learning. Note: This step takes several minutes to fully run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 000/14999 | Loss: 1.1482 | Episodes: 150 | Win count: 0 | Win rate: 0.000 | time: 13.3 seconds\n",
      "Epoch: 001/14999 | Loss: 0.0039 | Episodes: 3 | Win count: 1 | Win rate: 0.000 | time: 13.6 seconds\n",
      "Epoch: 002/14999 | Loss: 0.5387 | Episodes: 141 | Win count: 1 | Win rate: 0.000 | time: 25.8 seconds\n",
      "Epoch: 003/14999 | Loss: 0.1304 | Episodes: 47 | Win count: 2 | Win rate: 0.000 | time: 30.4 seconds\n",
      "Epoch: 004/14999 | Loss: 0.4845 | Episodes: 139 | Win count: 2 | Win rate: 0.000 | time: 42.5 seconds\n",
      "Epoch: 005/14999 | Loss: 0.0613 | Episodes: 12 | Win count: 3 | Win rate: 0.000 | time: 43.5 seconds\n",
      "Epoch: 006/14999 | Loss: 0.5077 | Episodes: 139 | Win count: 3 | Win rate: 0.000 | time: 55.9 seconds\n",
      "Epoch: 007/14999 | Loss: 0.4540 | Episodes: 148 | Win count: 3 | Win rate: 0.000 | time: 69.3 seconds\n",
      "Epoch: 008/14999 | Loss: 0.0426 | Episodes: 19 | Win count: 4 | Win rate: 0.000 | time: 71.2 seconds\n",
      "Epoch: 009/14999 | Loss: 0.3960 | Episodes: 140 | Win count: 4 | Win rate: 0.000 | time: 83.8 seconds\n",
      "Epoch: 010/14999 | Loss: 0.2577 | Episodes: 122 | Win count: 5 | Win rate: 0.000 | time: 94.0 seconds\n",
      "Epoch: 011/14999 | Loss: 0.4026 | Episodes: 150 | Win count: 6 | Win rate: 0.000 | time: 107.1 seconds\n",
      "Epoch: 012/14999 | Loss: 0.4299 | Episodes: 148 | Win count: 6 | Win rate: 0.000 | time: 121.2 seconds\n",
      "Epoch: 013/14999 | Loss: 0.1797 | Episodes: 55 | Win count: 7 | Win rate: 0.000 | time: 126.1 seconds\n",
      "Epoch: 014/14999 | Loss: 0.4308 | Episodes: 141 | Win count: 7 | Win rate: 0.000 | time: 137.8 seconds\n",
      "Epoch: 015/14999 | Loss: 0.2961 | Episodes: 142 | Win count: 7 | Win rate: 0.000 | time: 149.9 seconds\n",
      "Epoch: 016/14999 | Loss: 0.1435 | Episodes: 49 | Win count: 8 | Win rate: 0.000 | time: 154.0 seconds\n",
      "Epoch: 017/14999 | Loss: 0.0483 | Episodes: 16 | Win count: 9 | Win rate: 0.000 | time: 155.3 seconds\n",
      "Epoch: 018/14999 | Loss: 0.0232 | Episodes: 1 | Win count: 10 | Win rate: 0.000 | time: 155.4 seconds\n",
      "Epoch: 019/14999 | Loss: 0.0236 | Episodes: 8 | Win count: 11 | Win rate: 0.000 | time: 156.2 seconds\n",
      "Epoch: 020/14999 | Loss: 0.1902 | Episodes: 36 | Win count: 12 | Win rate: 0.000 | time: 159.4 seconds\n",
      "Epoch: 021/14999 | Loss: 0.6377 | Episodes: 141 | Win count: 12 | Win rate: 0.000 | time: 172.7 seconds\n",
      "Epoch: 022/14999 | Loss: 0.4073 | Episodes: 141 | Win count: 12 | Win rate: 0.000 | time: 185.5 seconds\n",
      "Epoch: 023/14999 | Loss: 0.3475 | Episodes: 139 | Win count: 12 | Win rate: 0.000 | time: 197.6 seconds\n",
      "Epoch: 024/14999 | Loss: 0.0098 | Episodes: 6 | Win count: 13 | Win rate: 0.000 | time: 198.2 seconds\n",
      "Epoch: 025/14999 | Loss: 0.0032 | Episodes: 2 | Win count: 14 | Win rate: 0.000 | time: 198.4 seconds\n",
      "Epoch: 026/14999 | Loss: 0.0109 | Episodes: 5 | Win count: 15 | Win rate: 0.000 | time: 198.8 seconds\n",
      "Epoch: 027/14999 | Loss: 0.0416 | Episodes: 19 | Win count: 16 | Win rate: 0.000 | time: 200.8 seconds\n",
      "Epoch: 028/14999 | Loss: 0.0594 | Episodes: 36 | Win count: 17 | Win rate: 0.000 | time: 204.4 seconds\n",
      "Epoch: 029/14999 | Loss: 0.1696 | Episodes: 141 | Win count: 17 | Win rate: 0.000 | time: 216.8 seconds\n",
      "Epoch: 030/14999 | Loss: 0.0190 | Episodes: 17 | Win count: 18 | Win rate: 0.000 | time: 218.2 seconds\n",
      "Epoch: 031/14999 | Loss: 0.0443 | Episodes: 32 | Win count: 19 | Win rate: 0.000 | time: 221.0 seconds\n",
      "Epoch: 032/14999 | Loss: 0.0248 | Episodes: 12 | Win count: 20 | Win rate: 0.625 | time: 222.0 seconds\n",
      "Epoch: 033/14999 | Loss: 0.0391 | Episodes: 25 | Win count: 21 | Win rate: 0.625 | time: 224.1 seconds\n",
      "Epoch: 034/14999 | Loss: 0.0092 | Episodes: 8 | Win count: 22 | Win rate: 0.656 | time: 224.8 seconds\n",
      "Epoch: 035/14999 | Loss: 0.0053 | Episodes: 1 | Win count: 23 | Win rate: 0.656 | time: 224.9 seconds\n",
      "Epoch: 036/14999 | Loss: 0.0074 | Episodes: 4 | Win count: 24 | Win rate: 0.688 | time: 225.3 seconds\n",
      "Epoch: 037/14999 | Loss: 0.0132 | Episodes: 9 | Win count: 25 | Win rate: 0.688 | time: 226.1 seconds\n",
      "Epoch: 038/14999 | Loss: 0.0052 | Episodes: 1 | Win count: 26 | Win rate: 0.719 | time: 226.2 seconds\n",
      "Epoch: 039/14999 | Loss: 0.3224 | Episodes: 140 | Win count: 26 | Win rate: 0.719 | time: 238.5 seconds\n",
      "Epoch: 040/14999 | Loss: 0.1938 | Episodes: 138 | Win count: 26 | Win rate: 0.688 | time: 250.3 seconds\n",
      "Epoch: 041/14999 | Loss: 0.1865 | Episodes: 136 | Win count: 26 | Win rate: 0.688 | time: 261.5 seconds\n",
      "Epoch: 042/14999 | Loss: 0.0132 | Episodes: 13 | Win count: 27 | Win rate: 0.688 | time: 262.6 seconds\n",
      "Epoch: 043/14999 | Loss: 0.0027 | Episodes: 3 | Win count: 28 | Win rate: 0.688 | time: 262.9 seconds\n",
      "Epoch: 044/14999 | Loss: 0.0093 | Episodes: 7 | Win count: 29 | Win rate: 0.719 | time: 263.5 seconds\n",
      "Epoch: 045/14999 | Loss: 0.1278 | Episodes: 138 | Win count: 29 | Win rate: 0.688 | time: 275.6 seconds\n",
      "Epoch: 046/14999 | Loss: 0.1160 | Episodes: 139 | Win count: 29 | Win rate: 0.688 | time: 287.8 seconds\n",
      "Epoch: 047/14999 | Loss: 0.1390 | Episodes: 144 | Win count: 29 | Win rate: 0.688 | time: 299.9 seconds\n",
      "Epoch: 048/14999 | Loss: 0.1531 | Episodes: 142 | Win count: 29 | Win rate: 0.656 | time: 312.0 seconds\n",
      "Epoch: 049/14999 | Loss: 0.0054 | Episodes: 6 | Win count: 30 | Win rate: 0.656 | time: 312.5 seconds\n",
      "Epoch: 050/14999 | Loss: 0.0165 | Episodes: 15 | Win count: 31 | Win rate: 0.656 | time: 313.8 seconds\n",
      "Epoch: 051/14999 | Loss: 0.0260 | Episodes: 23 | Win count: 32 | Win rate: 0.656 | time: 315.7 seconds\n",
      "Epoch: 052/14999 | Loss: 0.0072 | Episodes: 6 | Win count: 33 | Win rate: 0.656 | time: 316.1 seconds\n",
      "Epoch: 053/14999 | Loss: 0.1562 | Episodes: 138 | Win count: 33 | Win rate: 0.656 | time: 328.3 seconds\n",
      "Epoch: 054/14999 | Loss: 0.0180 | Episodes: 17 | Win count: 34 | Win rate: 0.688 | time: 329.7 seconds\n",
      "Epoch: 055/14999 | Loss: 0.0065 | Episodes: 5 | Win count: 35 | Win rate: 0.719 | time: 330.1 seconds\n",
      "Epoch: 056/14999 | Loss: 0.0247 | Episodes: 21 | Win count: 36 | Win rate: 0.719 | time: 332.0 seconds\n",
      "Epoch: 057/14999 | Loss: 0.0416 | Episodes: 30 | Win count: 37 | Win rate: 0.719 | time: 334.6 seconds\n",
      "Epoch: 058/14999 | Loss: 0.0032 | Episodes: 3 | Win count: 38 | Win rate: 0.719 | time: 335.0 seconds\n",
      "Epoch: 059/14999 | Loss: 0.0221 | Episodes: 21 | Win count: 39 | Win rate: 0.719 | time: 336.7 seconds\n",
      "Epoch: 060/14999 | Loss: 0.1420 | Episodes: 140 | Win count: 39 | Win rate: 0.688 | time: 348.8 seconds\n",
      "Epoch: 061/14999 | Loss: 0.1441 | Episodes: 139 | Win count: 39 | Win rate: 0.688 | time: 361.6 seconds\n",
      "Epoch: 062/14999 | Loss: 0.0337 | Episodes: 31 | Win count: 40 | Win rate: 0.688 | time: 364.3 seconds\n",
      "Epoch: 063/14999 | Loss: 0.0083 | Episodes: 7 | Win count: 41 | Win rate: 0.688 | time: 364.9 seconds\n",
      "Epoch: 064/14999 | Loss: 0.0255 | Episodes: 25 | Win count: 42 | Win rate: 0.688 | time: 367.3 seconds\n",
      "Epoch: 065/14999 | Loss: 0.0078 | Episodes: 7 | Win count: 43 | Win rate: 0.688 | time: 368.0 seconds\n",
      "Epoch: 066/14999 | Loss: 0.0007 | Episodes: 1 | Win count: 44 | Win rate: 0.688 | time: 368.0 seconds\n",
      "Epoch: 067/14999 | Loss: 0.0211 | Episodes: 20 | Win count: 45 | Win rate: 0.688 | time: 369.8 seconds\n",
      "Epoch: 068/14999 | Loss: 0.0155 | Episodes: 11 | Win count: 46 | Win rate: 0.688 | time: 370.8 seconds\n",
      "Epoch: 069/14999 | Loss: 0.0215 | Episodes: 20 | Win count: 47 | Win rate: 0.688 | time: 372.6 seconds\n",
      "Epoch: 070/14999 | Loss: 0.1717 | Episodes: 144 | Win count: 47 | Win rate: 0.656 | time: 385.7 seconds\n",
      "Epoch: 071/14999 | Loss: 0.1855 | Episodes: 144 | Win count: 47 | Win rate: 0.656 | time: 397.9 seconds\n",
      "Epoch: 072/14999 | Loss: 0.0136 | Episodes: 15 | Win count: 48 | Win rate: 0.688 | time: 399.1 seconds\n",
      "Epoch: 073/14999 | Loss: 0.0165 | Episodes: 17 | Win count: 49 | Win rate: 0.719 | time: 6.68 minutes\n",
      "Epoch: 074/14999 | Loss: 0.0293 | Episodes: 25 | Win count: 50 | Win rate: 0.719 | time: 6.71 minutes\n",
      "Epoch: 075/14999 | Loss: 0.1652 | Episodes: 137 | Win count: 50 | Win rate: 0.688 | time: 6.90 minutes\n",
      "Epoch: 076/14999 | Loss: 0.1348 | Episodes: 139 | Win count: 50 | Win rate: 0.656 | time: 7.14 minutes\n",
      "Epoch: 077/14999 | Loss: 0.0184 | Episodes: 16 | Win count: 51 | Win rate: 0.688 | time: 7.17 minutes\n",
      "Epoch: 078/14999 | Loss: 0.0014 | Episodes: 1 | Win count: 52 | Win rate: 0.719 | time: 7.17 minutes\n",
      "Epoch: 079/14999 | Loss: 0.1231 | Episodes: 142 | Win count: 52 | Win rate: 0.719 | time: 7.39 minutes\n",
      "Epoch: 080/14999 | Loss: 0.1312 | Episodes: 142 | Win count: 52 | Win rate: 0.719 | time: 7.60 minutes\n",
      "Epoch: 081/14999 | Loss: 0.0236 | Episodes: 23 | Win count: 53 | Win rate: 0.719 | time: 7.63 minutes\n",
      "Epoch: 082/14999 | Loss: 0.1466 | Episodes: 143 | Win count: 53 | Win rate: 0.688 | time: 7.84 minutes\n",
      "Epoch: 083/14999 | Loss: 0.0092 | Episodes: 7 | Win count: 54 | Win rate: 0.688 | time: 7.86 minutes\n",
      "Epoch: 084/14999 | Loss: 0.1509 | Episodes: 133 | Win count: 55 | Win rate: 0.688 | time: 8.05 minutes\n",
      "Epoch: 085/14999 | Loss: 0.0570 | Episodes: 45 | Win count: 56 | Win rate: 0.719 | time: 8.12 minutes\n",
      "Epoch: 086/14999 | Loss: 0.2044 | Episodes: 144 | Win count: 56 | Win rate: 0.688 | time: 8.33 minutes\n",
      "Epoch: 087/14999 | Loss: 0.0295 | Episodes: 20 | Win count: 57 | Win rate: 0.688 | time: 8.36 minutes\n",
      "Epoch: 088/14999 | Loss: 0.0028 | Episodes: 2 | Win count: 58 | Win rate: 0.688 | time: 8.36 minutes\n",
      "Epoch: 089/14999 | Loss: 0.0049 | Episodes: 3 | Win count: 59 | Win rate: 0.688 | time: 8.37 minutes\n",
      "Epoch: 090/14999 | Loss: 0.0117 | Episodes: 8 | Win count: 60 | Win rate: 0.688 | time: 8.38 minutes\n",
      "Epoch: 091/14999 | Loss: 0.1821 | Episodes: 133 | Win count: 61 | Win rate: 0.688 | time: 8.58 minutes\n",
      "Epoch: 092/14999 | Loss: 0.0827 | Episodes: 62 | Win count: 62 | Win rate: 0.719 | time: 8.66 minutes\n",
      "Epoch: 093/14999 | Loss: 0.0170 | Episodes: 10 | Win count: 63 | Win rate: 0.750 | time: 8.68 minutes\n",
      "Epoch: 094/14999 | Loss: 0.0203 | Episodes: 15 | Win count: 64 | Win rate: 0.750 | time: 8.70 minutes\n",
      "Epoch: 095/14999 | Loss: 0.0108 | Episodes: 7 | Win count: 65 | Win rate: 0.750 | time: 8.71 minutes\n",
      "Epoch: 096/14999 | Loss: 0.0088 | Episodes: 7 | Win count: 66 | Win rate: 0.750 | time: 8.72 minutes\n",
      "Epoch: 097/14999 | Loss: 0.0085 | Episodes: 7 | Win count: 67 | Win rate: 0.750 | time: 8.73 minutes\n",
      "Epoch: 098/14999 | Loss: 0.0649 | Episodes: 38 | Win count: 68 | Win rate: 0.750 | time: 8.78 minutes\n",
      "Epoch: 099/14999 | Loss: 0.0356 | Episodes: 22 | Win count: 69 | Win rate: 0.750 | time: 8.81 minutes\n",
      "Epoch: 100/14999 | Loss: 0.1181 | Episodes: 92 | Win count: 70 | Win rate: 0.750 | time: 8.93 minutes\n",
      "Epoch: 101/14999 | Loss: 0.0011 | Episodes: 1 | Win count: 71 | Win rate: 0.750 | time: 8.93 minutes\n",
      "Epoch: 102/14999 | Loss: 0.0093 | Episodes: 9 | Win count: 72 | Win rate: 0.781 | time: 8.95 minutes\n",
      "Epoch: 103/14999 | Loss: 0.1343 | Episodes: 115 | Win count: 73 | Win rate: 0.812 | time: 9.11 minutes\n",
      "Epoch: 104/14999 | Loss: 0.0226 | Episodes: 22 | Win count: 74 | Win rate: 0.812 | time: 9.14 minutes\n",
      "Epoch: 105/14999 | Loss: 0.1320 | Episodes: 118 | Win count: 75 | Win rate: 0.812 | time: 9.31 minutes\n",
      "Epoch: 106/14999 | Loss: 0.0071 | Episodes: 8 | Win count: 76 | Win rate: 0.812 | time: 9.32 minutes\n",
      "Epoch: 107/14999 | Loss: 0.0118 | Episodes: 16 | Win count: 77 | Win rate: 0.844 | time: 9.35 minutes\n",
      "Epoch: 108/14999 | Loss: 0.0242 | Episodes: 25 | Win count: 78 | Win rate: 0.875 | time: 9.38 minutes\n",
      "Epoch: 109/14999 | Loss: 0.0061 | Episodes: 6 | Win count: 79 | Win rate: 0.875 | time: 9.39 minutes\n",
      "Epoch: 110/14999 | Loss: 0.0282 | Episodes: 31 | Win count: 80 | Win rate: 0.875 | time: 9.43 minutes\n",
      "Epoch: 111/14999 | Loss: 0.0154 | Episodes: 18 | Win count: 81 | Win rate: 0.906 | time: 9.46 minutes\n",
      "Epoch: 112/14999 | Loss: 0.0068 | Episodes: 7 | Win count: 82 | Win rate: 0.938 | time: 9.47 minutes\n",
      "Epoch: 113/14999 | Loss: 0.0067 | Episodes: 6 | Win count: 83 | Win rate: 0.938 | time: 9.48 minutes\n",
      "Epoch: 114/14999 | Loss: 0.0075 | Episodes: 8 | Win count: 84 | Win rate: 0.969 | time: 9.50 minutes\n",
      "Epoch: 115/14999 | Loss: 0.0015 | Episodes: 1 | Win count: 85 | Win rate: 0.969 | time: 9.50 minutes\n",
      "Epoch: 116/14999 | Loss: 0.0018 | Episodes: 2 | Win count: 86 | Win rate: 0.969 | time: 9.50 minutes\n",
      "Epoch: 117/14999 | Loss: 0.0439 | Episodes: 48 | Win count: 87 | Win rate: 0.969 | time: 9.57 minutes\n",
      "Epoch: 118/14999 | Loss: 0.0422 | Episodes: 46 | Win count: 88 | Win rate: 1.000 | time: 9.63 minutes\n",
      "Epoch: 119/14999 | Loss: 0.0288 | Episodes: 35 | Win count: 89 | Win rate: 1.000 | time: 9.69 minutes\n",
      "Epoch: 120/14999 | Loss: 0.0204 | Episodes: 26 | Win count: 90 | Win rate: 1.000 | time: 9.73 minutes\n",
      "Epoch: 121/14999 | Loss: 0.0196 | Episodes: 22 | Win count: 91 | Win rate: 1.000 | time: 9.77 minutes\n",
      "Epoch: 122/14999 | Loss: 0.0172 | Episodes: 19 | Win count: 92 | Win rate: 1.000 | time: 9.80 minutes\n",
      "Epoch: 123/14999 | Loss: 0.1250 | Episodes: 154 | Win count: 93 | Win rate: 1.000 | time: 10.05 minutes\n",
      "Epoch: 124/14999 | Loss: 0.0168 | Episodes: 26 | Win count: 94 | Win rate: 1.000 | time: 10.08 minutes\n",
      "Epoch: 125/14999 | Loss: 0.0529 | Episodes: 67 | Win count: 95 | Win rate: 1.000 | time: 10.18 minutes\n",
      "Epoch: 126/14999 | Loss: 0.0104 | Episodes: 15 | Win count: 96 | Win rate: 1.000 | time: 10.21 minutes\n",
      "Epoch: 127/14999 | Loss: 0.0105 | Episodes: 13 | Win count: 97 | Win rate: 1.000 | time: 10.23 minutes\n",
      "Epoch: 128/14999 | Loss: 0.0032 | Episodes: 5 | Win count: 98 | Win rate: 1.000 | time: 10.24 minutes\n",
      "Epoch: 129/14999 | Loss: 0.0383 | Episodes: 41 | Win count: 99 | Win rate: 1.000 | time: 10.30 minutes\n",
      "Epoch: 130/14999 | Loss: 0.0184 | Episodes: 24 | Win count: 100 | Win rate: 1.000 | time: 10.33 minutes\n",
      "Epoch: 131/14999 | Loss: 0.0095 | Episodes: 11 | Win count: 101 | Win rate: 1.000 | time: 10.35 minutes\n",
      "Epoch: 132/14999 | Loss: 0.0236 | Episodes: 28 | Win count: 102 | Win rate: 1.000 | time: 10.39 minutes\n",
      "Epoch: 133/14999 | Loss: 0.0315 | Episodes: 33 | Win count: 103 | Win rate: 1.000 | time: 10.45 minutes\n",
      "Epoch: 134/14999 | Loss: 0.0218 | Episodes: 27 | Win count: 104 | Win rate: 1.000 | time: 10.49 minutes\n",
      "Epoch: 135/14999 | Loss: 0.0324 | Episodes: 31 | Win count: 105 | Win rate: 1.000 | time: 10.53 minutes\n",
      "Epoch: 136/14999 | Loss: 0.0085 | Episodes: 8 | Win count: 106 | Win rate: 1.000 | time: 10.55 minutes\n",
      "Epoch: 137/14999 | Loss: 0.0294 | Episodes: 32 | Win count: 107 | Win rate: 1.000 | time: 10.60 minutes\n",
      "Epoch: 138/14999 | Loss: 0.0103 | Episodes: 13 | Win count: 108 | Win rate: 1.000 | time: 10.62 minutes\n",
      "Epoch: 139/14999 | Loss: 0.0073 | Episodes: 9 | Win count: 109 | Win rate: 1.000 | time: 10.64 minutes\n",
      "Epoch: 140/14999 | Loss: 0.0093 | Episodes: 7 | Win count: 110 | Win rate: 1.000 | time: 10.65 minutes\n",
      "Epoch: 141/14999 | Loss: 0.0497 | Episodes: 44 | Win count: 111 | Win rate: 1.000 | time: 10.72 minutes\n",
      "Epoch: 142/14999 | Loss: 0.0174 | Episodes: 18 | Win count: 112 | Win rate: 1.000 | time: 10.74 minutes\n",
      "Epoch: 143/14999 | Loss: 0.0263 | Episodes: 27 | Win count: 113 | Win rate: 1.000 | time: 10.78 minutes\n",
      "Epoch: 144/14999 | Loss: 0.0187 | Episodes: 28 | Win count: 114 | Win rate: 1.000 | time: 10.82 minutes\n",
      "Epoch: 145/14999 | Loss: 0.0013 | Episodes: 3 | Win count: 115 | Win rate: 1.000 | time: 10.83 minutes\n",
      "Epoch: 146/14999 | Loss: 0.0018 | Episodes: 2 | Win count: 116 | Win rate: 1.000 | time: 10.83 minutes\n",
      "Epoch: 147/14999 | Loss: 0.0131 | Episodes: 19 | Win count: 117 | Win rate: 1.000 | time: 10.86 minutes\n",
      "Epoch: 148/14999 | Loss: 0.0019 | Episodes: 3 | Win count: 118 | Win rate: 1.000 | time: 10.87 minutes\n",
      "Epoch: 149/14999 | Loss: 0.0229 | Episodes: 36 | Win count: 119 | Win rate: 1.000 | time: 10.93 minutes\n",
      "Epoch: 150/14999 | Loss: 0.0130 | Episodes: 26 | Win count: 120 | Win rate: 1.000 | time: 10.97 minutes\n",
      "Epoch: 151/14999 | Loss: 0.0162 | Episodes: 32 | Win count: 121 | Win rate: 1.000 | time: 11.02 minutes\n",
      "Epoch: 152/14999 | Loss: 0.0498 | Episodes: 83 | Win count: 122 | Win rate: 1.000 | time: 11.14 minutes\n",
      "Epoch: 153/14999 | Loss: 0.0056 | Episodes: 10 | Win count: 123 | Win rate: 1.000 | time: 11.16 minutes\n",
      "Epoch: 154/14999 | Loss: 0.0031 | Episodes: 6 | Win count: 124 | Win rate: 1.000 | time: 11.17 minutes\n",
      "Reached 100% win rate at epoch: 154\n",
      "n_epoch: 154, max_mem: 512, data: 32, time: 11.19 minutes\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "671.241313"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = build_model(maze)\n",
    "qtrain(model, maze, epochs=1000, max_memory=8*maze.size, data_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell will check to see if the model passes the completion check. Note: This could take several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1b90f2c9d08>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAFTklEQVR4nO3dMW5TaRSG4XMHOqBBkdJQ0JmCzl4Aq2EFXoZXwAoo2EOyALugTEcRgSJF0ECJ7hQD0iAlE6Jkjv39PI/kKkif7atXhIYzzfNcwOH7a99vAPg9YoUQYoUQYoUQYoUQYoUQD2/zh4+Ojubnz5//T2/lVx8/fqxPnz61bL148aIePXrUsvXt27cht7r3Rt368OFDXV5eTlf+cJ7n334tl8u5y2azmauq5XVyctL2uUbd6t4bdetHY1f259dgCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFWCCFW7s1ut6tpmlpef6JpvuHy+TRNr6vqdVXV8fHx8u3btx3vqy4uLur8/Lxla7FY1OPHj1u2vn79OuRWlWd2H9brdW23W+czrnuNeoqh+3yGZ3Z3zmfAAMQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIQ421uVyeav/gPwur5F1nbOYpqn1mXWe6tjtdvt+jFV1wOczRj0z0b11dnbWslXVe9Ki81THs2fP6vj4uGUr8nzGqOcRureq6ZxFNZ+06DzVsdls2j6X8xkwALFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCLFCCOcz9rDVddKi8+xD1djPrGvL+YwD26oBzz78/Gy27sb5DBiAWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWCGEWKtqt9vVNE0tr91ud6srCHd5LZfLfX+13CO3bqrq4uKizs/PW7Y67890fofde6NuuXVzg81mM+T9mc7vsHtv1C23bmAAYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYq2q5XLZetKi81RHp+4zJKNuXcf5jD1snZ2dtWx1nuqo6j9DMuLWer2ueZ6dzziUrRrwVMc8958hGXHrnySdz4BoYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQD/f9BhjHzzMkHU5PT4fcWq1W1/7M+Yw9bI16PmPkZ9a1tV6va7vdOp9xKFs16PmMkZ9Zlx+NOZ8BycQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIcQKIZzPGHyr61RHVdVisWj7bF++fKnv37+3bD148KD1fMb79++vPJ9x42GqeZ7fVNWbqqrVajW/evXqft/dNU5PT8vW3bfW63XLVlXVyclJ22d79+5dff78uWXr6dOn9fLly5at/+LXYAghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVgghVghxq/MZVbWoqq57DEdVdWkrZqt7b9StxTzPT676wY2x7ss0Tdt5nle2Mra69/7ELb8GQwixQohDjvWNrait7r0/butg/80K/OqQ/2YF/kWsEEKsEEKsEEKsEOJvbZzkO73fDXcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "completion_check(model, qmaze)\n",
    "show(qmaze)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell will test your model for one game. It will start the pirate at the top-left corner and run play_game. The agent should find a path from the starting position to the target (treasure). The treasure is located in the bottom-right corner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1b90f38a4c8>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAGuklEQVR4nO3dT07TexvG4advHZgciLGYMGlCwgAcOLML0D2YuAM3QGduoYwJrMCERcACcOBQR5IQDJEwAQ2JNr8zMCc554Q/L++LX7nrdU1rvEvsx9JJn17XdQXcff/51U8A+O+IFUKIFUKIFUKIFUKIFULcu8kfnpub6xYWFn7Wc/mHb9++1adPn5psPX78uP74448mW1++fKmvX7822bp37159//69yVbrvdZbDx8+bLL18ePHOj4+7l34PG7yFy0sLNTr169v51ld4/T0tMbjcZOtjY2NevbsWZOt3d3d+vDhQ5OtwWBQJycnTbZa77XeevHiRZOt0Wh06WN+DYYQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQN/qSb/K8evWq2db6+nqzL2bf3t5usvOXra2tJjufP3++9LHedZfPe73eq6p6VVX16NGjpxsbG7f65C4znU7r4OCgydbq6mrNzc012To7O6vz8/MmW/1+v9nZh6qqo6OjZv9my8vLNZ1Om2z1+/1mW2tra7W/v/+/nc/oum6rqraqqpaWlrpWJwtans/Y2dlpej7j8PCwydZgMGj2c1W1f2edxVMdV/GZFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFUKIFULc2fMZS0tLtbm52WTr+Pi42XmEwWDQZOcvvd6FX+7+U+zs7NR1Fx5uS8svFJ9MJjU/P99k6yrXxvqv8xnNXmz9ft/WLWxNJpMmW1U/ToPs7u422RoOh81+tuFwWP1+v8nWVe7s+YyWJwtmeavVu09V2zMkv+M7q8+sEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEEKsEOJG5zMWFxdrZWXlpz+pqh+nGGZ16/T0tMnWgwcPamdnp8lWVdvzGaurq81+trOzszo/P2+ydZUbnc8YjUZdq/MIu7u7zU4xtN5qefbh5cuXTbaqZvvf7PDwsMnWVfwaDCHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiHECiGu/Ub+38Hbt2/r+fPnTbYmk0l1Xddka3d3t7a2tppsVVUNBoNme6237oLedS+cf926efrmzZsWz6vOzs5qbm6uydbR0VEdHBw02RoOh7W4uNhkq/WNln6/X9Pp1Nb/YW1trfb393sXPebWTVWtr6/P5P2Z1jdaBoNBnZyc2PpJfGaFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEGKFEM5nVNXTp0+bnrTo9S78wvVbN5lMan5+vslWVdX+/n7TL0ufxa2rOJ/xC7bev3/fZGs4HFa/32+yVVU1nU6bniGZxa3xeFxd1134v/m1sf7daDTq9vb2bu2JXaXl+YzWWy2PYLV8Zz09PZ3Jd7vW76yXxeozK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4QQK4RwPoNbs7S0VJubm022VlZWmp48abU1Go0ufcz5jF+wNavnM/r9fk2n0yZb9+/fn8nXx3g8rr29Pecz7srWrJ7PGAwGdXJy0mRrZWVlJl8fo9Ho0lh9ZoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQYoUQzmfM+FarUx1VVcvLy83OZ7Q81dHv95uez3j37t2F38h/7WGqruu2qmqr6sf5jFk8WTDLW+PxuMlWVdX29naz8xktT3UMBoN68uRJk62r+DUYQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQogVQtzofEZVrVZVq3sMj6rq2FbMVuu9Wd1a7bpu/qIHro31V+n1entd141sZWy13vsdt/waDCHECiHucqxbtqK2Wu/9dlt39jMr8E93+Z0V+BuxQgixQgixQgixQog/ARdRUhvM/uZJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pirate_start = (0, 0)\n",
    "play_game(model, qmaze, pirate_start)\n",
    "show(qmaze)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save and Submit Your Work\n",
    "After you have finished creating the code for your notebook, save your work. Make sure that your notebook contains your name in the filename (e.g. Doe_Jane_ProjectTwo.ipynb). This will help your instructor access and grade your work easily. Download a copy of your IPYNB file and submit it to Brightspace. Refer to the Jupyter Notebook in Apporto Tutorial if you need help with these tasks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}