cgames/03_doom_defend_center/doom_defend_center_dqn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "colab_type": "text",
    "id": "LN0nZwyMGadB"
   },
   "outputs": [],
   "source": [
    "# Doom Defend the Center with Dqn\n",
    "\n",
    "The purpose of this scenario is to teach the agent that killing the monsters is GOOD and when monsters kill you is BAD. In addition, wasting amunition is not very good either. Agent is rewarded only for killing monsters so he has to figure out the rest for himself.\n",
    "\n",
    "### Enviroment:\n",
    "\n",
    "Map is a large circle. Player is spawned in the exact center. 5 melee-only, monsters are spawned along the wall. Monsters are killed after a single shot. After dying each monster is respawned after some time.Episode ends when the player dies.\n",
    "\n",
    "### Actions:\n",
    " - turn left\n",
    " - turn right\n",
    " - shoot (attack)\n",
    "\n",
    "### REWARDS:\n",
    " - +1 for killing a monster\n",
    " - -1 for death penalty\n",
    "\n",
    "## Step 1: Import the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random                # Handling random number generation\n",
    "import time                  # Handling time calculation\n",
    "import cv2\n",
    "\n",
    "import torch\n",
    "from vizdoom import *        # Doom Environment\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import clear_output\n",
    "from collections import namedtuple, deque\n",
    "import math\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../../')\n",
    "from algos.agents import DQNAgent\n",
    "from algos.models import DQNCnn\n",
    "from algos.preprocessing.stack_frame import preprocess_frame, stack_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "colab_type": "text",
    "id": "tfo8jleHGadK"
   },
   "outputs": [],
   "source": [
    "## Step 2: Create our environment\n",
    "\n",
    "Initialize the environment in the code cell below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_environment():\n",
    "    game = DoomGame()\n",
    "    \n",
    "    # Load the correct configuration\n",
    "    game.load_config(\"doom_files/defend_the_center.cfg\")\n",
    "    \n",
    "    # Load the correct scenario (in our case defend_the_center scenario)\n",
    "    game.set_doom_scenario_path(\"doom_files/defend_the_center.wad\")\n",
    "    \n",
    "    possible_actions  = np.identity(3,dtype=int).tolist()\n",
    "    \n",
    "    return game, possible_actions\n",
    "game, possible_actions = create_environment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if gpu is to be used\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "colab_type": "text",
    "id": "nS221MgXGadP"
   },
   "outputs": [],
   "source": [
    "## Step 3: Viewing our Enviroment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The size of frame is: (\", game.get_screen_height(), \", \", game.get_screen_width(), \")\")\n",
    "print(\"No. of Actions: \", possible_actions)\n",
    "game.init()\n",
    "plt.figure()\n",
    "plt.imshow(game.get_state().screen_buffer.transpose(1, 2, 0))\n",
    "plt.title('Original Frame')\n",
    "plt.show()\n",
    "game.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Execute the code cell below to play Pong with a random policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_play():\n",
    "    game.init()\n",
    "    game.new_episode()\n",
    "    score = 0\n",
    "    while True:\n",
    "        reward = game.make_action(possible_actions[np.random.randint(3)])\n",
    "        done = game.is_episode_finished()\n",
    "        score += reward\n",
    "        time.sleep(0.01)\n",
    "        if done:\n",
    "            print(\"Your total score is: \", score)\n",
    "            game.close()\n",
    "            break\n",
    "random_play()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "colab_type": "text",
    "id": "Sr52nmcpGada"
   },
   "outputs": [],
   "source": [
    "## Step 4:Preprocessing Frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "game.init()\n",
    "plt.figure()\n",
    "plt.imshow(preprocess_frame(game.get_state().screen_buffer.transpose(1, 2, 0),(100, -12, -80, 4), 84), cmap=\"gray\")\n",
    "game.close()\n",
    "plt.title('Pre Processed image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {
    "colab_type": "text",
    "id": "mJMc3HA8Gade"
   },
   "outputs": [],
   "source": [
    "## Step 5: Stacking Frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stack_frames(frames, state, is_new=False):\n",
    "    frame = preprocess_frame(state, (100, -12, -80, 4), 84)\n",
    "    frames = stack_frame(frames, frame, is_new)\n",
    "\n",
    "    return frames\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Step 6: Creating our Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_SHAPE = (4, 84, 84)\n",
    "ACTION_SIZE = len(possible_actions)\n",
    "SEED = 0\n",
    "GAMMA = 0.99           # discount factor\n",
    "BUFFER_SIZE = 100000   # replay buffer size\n",
    "BATCH_SIZE = 32        # Update batch size\n",
    "LR = 0.0001            # learning rate \n",
    "TAU = .1               # for soft update of target parameters\n",
    "UPDATE_EVERY = 100     # how often to update the network\n",
    "UPDATE_TARGET = 10000  # After which thershold replay to be started \n",
    "EPS_START = 0.99       # starting value of epsilon\n",
    "EPS_END = 0.01         # Ending value of epsilon\n",
    "EPS_DECAY = 100        # Rate by which epsilon to be decayed\n",
    "\n",
    "agent = DQNAgent(INPUT_SHAPE, ACTION_SIZE, SEED, device, BUFFER_SIZE, BATCH_SIZE, GAMMA, LR, TAU, UPDATE_EVERY, UPDATE_TARGET, DQNCnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Step 7: Watching untrained agent play"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# watch an untrained agent\n",
    "game.init()\n",
    "score = 0\n",
    "state = stack_frames(None, game.get_state().screen_buffer.transpose(1, 2, 0), True) \n",
    "while True:\n",
    "    action = agent.act(state, 0.01)\n",
    "    score += game.make_action(possible_actions[action])\n",
    "    done = game.is_episode_finished()\n",
    "    if done:\n",
    "        print(\"Your total score is: \", score)\n",
    "        break\n",
    "    else:\n",
    "        state = stack_frames(state, game.get_state().screen_buffer.transpose(1, 2, 0), False)\n",
    "        \n",
    "game.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Step 8: Loading Agent\n",
    "Uncomment line to load a pretrained agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_epoch = 0\n",
    "scores = []\n",
    "scores_window = deque(maxlen=20)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Step 9: Train the Agent with DQN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon_by_epsiode = lambda frame_idx: EPS_END + (EPS_START - EPS_END) * math.exp(-1. * frame_idx /EPS_DECAY)\n",
    "\n",
    "plt.plot([epsilon_by_epsiode(i) for i in range(1000)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(n_episodes=1000):\n",
    "    \"\"\"\n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "    \"\"\"\n",
    "    game.init()\n",
    "    for i_episode in range(start_epoch + 1, n_episodes+1):\n",
    "        game.new_episode()\n",
    "        state = stack_frames(None, game.get_state().screen_buffer.transpose(1, 2, 0), True) \n",
    "        score = 0\n",
    "        eps = epsilon_by_epsiode(i_episode)\n",
    "        while True:\n",
    "            action = agent.act(state, eps)\n",
    "            reward = game.make_action(possible_actions[action])\n",
    "            done = game.is_episode_finished()\n",
    "            score += reward\n",
    "            if done:\n",
    "                agent.step(state, action, reward, state, done)\n",
    "                break\n",
    "            else:\n",
    "                next_state = stack_frames(state, game.get_state().screen_buffer.transpose(1, 2, 0), False)\n",
    "                agent.step(state, action, reward, next_state, done)\n",
    "                state = next_state\n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        \n",
    "        clear_output(True)\n",
    "        fig = plt.figure()\n",
    "        ax = fig.add_subplot(111)\n",
    "        plt.plot(np.arange(len(scores)), scores)\n",
    "        plt.ylabel('Score')\n",
    "        plt.xlabel('Episode #')\n",
    "        plt.show()\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}\\tEpsilon: {:.2f}'.format(i_episode, np.mean(scores_window), eps), end=\"\")\n",
    "    game.close()\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = train(5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Step 10: Watch a Smart Agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "game.init()\n",
    "score = 0\n",
    "state = stack_frames(None, game.get_state().screen_buffer.transpose(1, 2, 0), True) \n",
    "while True:\n",
    "    action = agent.act(state, 0.01)\n",
    "    score += game.make_action(possible_actions[action])\n",
    "    done = game.is_episode_finished()\n",
    "    if done:\n",
    "        print(\"Your total score is: \", score)\n",
    "        break\n",
    "    else:\n",
    "        state = stack_frames(state, game.get_state().screen_buffer.transpose(1, 2, 0), False)\n",
    "        \n",
    "game.close()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "space_invader.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3.7.6 64-bit ('ai': conda)",
   "language": "python",
   "name": "python37664bitaicondac0e8880bb07b42c396c674eb5a2d8b93"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}