Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, I will train a DDPG model using Unity ML environment.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "Run the next code cell to install a few packages(if you haven't already). This line will take a few minutes to run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip -q install ./python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please select one of the two options below for loading the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import torch\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from ddpg_agent import Agent\n",
    "\n",
    "# select this option to load version 1 (with a single agent) of the environment\n",
    "# env = UnityEnvironment(file_name='/data/Reacher_One_Linux_NoVis/Reacher_One_Linux_NoVis.x86_64')\n",
    "\n",
    "# select this option to load version 2 (with 20 agents) of the environment\n",
    "#env = UnityEnvironment(file_name='/data/Reacher_Linux_NoVis/Reacher.x86_64')\n",
    "\n",
    "# select this if you are running the reacher app on your local machine(windows)\n",
    "env = UnityEnvironment(file_name='Reacher_Windows_x86_64/Reacher.exe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this environment, a double-jointed arm can move to target locations. A reward of +0.1 is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of 33 variables corresponding to position, rotation, velocity, and angular velocities of the arm. Each action is a vector with four numbers, corresponding to torque applicable to two joints. Every entry in the action vector should be a number between -1 and 1.\n",
    "\n",
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent\n",
    "\n",
    "**Important Note**:\n",
    "When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = Agent(state_size=state_size, action_size=action_size, num_agents=num_agents, random_seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1\tScore: 0.84\tAverage Score: 0.84\n",
      "Episode 2\tScore: 0.49\tAverage Score: 0.67\n",
      "Episode 3\tScore: 0.74\tAverage Score: 0.69\n",
      "Episode 4\tScore: 0.61\tAverage Score: 0.67\n",
      "Episode 5\tScore: 0.81\tAverage Score: 0.70\n",
      "Episode 6\tScore: 1.13\tAverage Score: 0.77\n",
      "Episode 7\tScore: 1.16\tAverage Score: 0.83\n",
      "Episode 8\tScore: 1.61\tAverage Score: 0.92\n",
      "Episode 9\tScore: 1.48\tAverage Score: 0.99\n",
      "Episode 10\tScore: 1.60\tAverage Score: 1.05\n",
      "Episode 11\tScore: 1.44\tAverage Score: 1.08\n",
      "Episode 12\tScore: 1.49\tAverage Score: 1.12\n",
      "Episode 13\tScore: 1.63\tAverage Score: 1.16\n",
      "Episode 14\tScore: 2.21\tAverage Score: 1.23\n",
      "Episode 15\tScore: 2.11\tAverage Score: 1.29\n",
      "Episode 16\tScore: 2.16\tAverage Score: 1.34\n",
      "Episode 17\tScore: 2.71\tAverage Score: 1.43\n",
      "Episode 18\tScore: 2.99\tAverage Score: 1.51\n",
      "Episode 19\tScore: 4.81\tAverage Score: 1.69\n",
      "Episode 20\tScore: 4.50\tAverage Score: 1.83\n",
      "Episode 21\tScore: 6.82\tAverage Score: 2.06\n",
      "Episode 22\tScore: 7.19\tAverage Score: 2.30\n",
      "Episode 23\tScore: 9.29\tAverage Score: 2.60\n",
      "Episode 24\tScore: 9.19\tAverage Score: 2.88\n",
      "Episode 25\tScore: 8.58\tAverage Score: 3.10\n",
      "Episode 26\tScore: 10.39\tAverage Score: 3.38\n",
      "Episode 27\tScore: 12.37\tAverage Score: 3.72\n",
      "Episode 28\tScore: 13.22\tAverage Score: 4.06\n",
      "Episode 29\tScore: 13.62\tAverage Score: 4.39\n",
      "Episode 30\tScore: 13.03\tAverage Score: 4.67\n",
      "Episode 31\tScore: 15.75\tAverage Score: 5.03\n",
      "Episode 32\tScore: 14.93\tAverage Score: 5.34\n",
      "Episode 33\tScore: 19.31\tAverage Score: 5.76\n",
      "Episode 34\tScore: 19.78\tAverage Score: 6.18\n",
      "Episode 35\tScore: 21.66\tAverage Score: 6.62\n",
      "Episode 36\tScore: 21.57\tAverage Score: 7.03\n",
      "Episode 37\tScore: 23.24\tAverage Score: 7.47\n",
      "Episode 38\tScore: 25.33\tAverage Score: 7.94\n",
      "Episode 39\tScore: 27.71\tAverage Score: 8.45\n",
      "Episode 40\tScore: 27.33\tAverage Score: 8.92\n",
      "Episode 41\tScore: 28.66\tAverage Score: 9.40\n",
      "Episode 42\tScore: 31.05\tAverage Score: 9.92\n",
      "Episode 43\tScore: 33.68\tAverage Score: 10.47\n",
      "Episode 44\tScore: 32.40\tAverage Score: 10.97\n",
      "Episode 45\tScore: 33.44\tAverage Score: 11.47\n",
      "Episode 46\tScore: 33.83\tAverage Score: 11.95\n",
      "Episode 47\tScore: 34.51\tAverage Score: 12.43\n",
      "Episode 48\tScore: 35.54\tAverage Score: 12.92\n",
      "Episode 49\tScore: 35.35\tAverage Score: 13.37\n",
      "Episode 50\tScore: 35.72\tAverage Score: 13.82\n",
      "Episode 51\tScore: 35.91\tAverage Score: 14.25\n",
      "Episode 52\tScore: 35.74\tAverage Score: 14.67\n",
      "Episode 53\tScore: 36.60\tAverage Score: 15.08\n",
      "Episode 54\tScore: 37.26\tAverage Score: 15.49\n",
      "Episode 55\tScore: 37.16\tAverage Score: 15.89\n",
      "Episode 56\tScore: 37.29\tAverage Score: 16.27\n",
      "Episode 57\tScore: 37.91\tAverage Score: 16.65\n",
      "Episode 58\tScore: 37.70\tAverage Score: 17.01\n",
      "Episode 59\tScore: 38.32\tAverage Score: 17.37\n",
      "Episode 60\tScore: 37.72\tAverage Score: 17.71\n",
      "Episode 61\tScore: 37.94\tAverage Score: 18.04\n",
      "Episode 62\tScore: 37.90\tAverage Score: 18.36\n",
      "Episode 63\tScore: 37.60\tAverage Score: 18.67\n",
      "Episode 64\tScore: 36.78\tAverage Score: 18.95\n",
      "Episode 65\tScore: 37.37\tAverage Score: 19.23\n",
      "Episode 66\tScore: 36.81\tAverage Score: 19.50\n",
      "Episode 67\tScore: 36.54\tAverage Score: 19.76\n",
      "Episode 68\tScore: 35.95\tAverage Score: 19.99\n",
      "Episode 69\tScore: 36.91\tAverage Score: 20.24\n",
      "Episode 70\tScore: 37.43\tAverage Score: 20.48\n",
      "Episode 71\tScore: 36.88\tAverage Score: 20.72\n",
      "Episode 72\tScore: 37.01\tAverage Score: 20.94\n",
      "Episode 73\tScore: 36.64\tAverage Score: 21.16\n",
      "Episode 74\tScore: 35.33\tAverage Score: 21.35\n",
      "Episode 75\tScore: 36.13\tAverage Score: 21.55\n",
      "Episode 76\tScore: 36.82\tAverage Score: 21.75\n",
      "Episode 77\tScore: 36.68\tAverage Score: 21.94\n",
      "Episode 78\tScore: 36.81\tAverage Score: 22.13\n",
      "Episode 79\tScore: 35.96\tAverage Score: 22.31\n",
      "Episode 80\tScore: 34.66\tAverage Score: 22.46\n",
      "Episode 81\tScore: 31.38\tAverage Score: 22.57\n",
      "Episode 82\tScore: 36.28\tAverage Score: 22.74\n",
      "Episode 83\tScore: 36.09\tAverage Score: 22.90\n",
      "Episode 84\tScore: 36.12\tAverage Score: 23.06\n",
      "Episode 85\tScore: 35.73\tAverage Score: 23.20\n",
      "Episode 86\tScore: 35.29\tAverage Score: 23.35\n",
      "Episode 87\tScore: 36.67\tAverage Score: 23.50\n",
      "Episode 88\tScore: 37.20\tAverage Score: 23.65\n",
      "Episode 89\tScore: 36.03\tAverage Score: 23.79\n",
      "Episode 90\tScore: 36.30\tAverage Score: 23.93\n",
      "Episode 91\tScore: 36.61\tAverage Score: 24.07\n",
      "Episode 92\tScore: 37.03\tAverage Score: 24.21\n",
      "Episode 93\tScore: 36.86\tAverage Score: 24.35\n",
      "Episode 94\tScore: 36.72\tAverage Score: 24.48\n",
      "Episode 95\tScore: 35.17\tAverage Score: 24.59\n",
      "Episode 96\tScore: 35.12\tAverage Score: 24.70\n",
      "Episode 97\tScore: 36.03\tAverage Score: 24.82\n",
      "Episode 98\tScore: 37.41\tAverage Score: 24.95\n",
      "Episode 99\tScore: 36.94\tAverage Score: 25.07\n",
      "Episode 100\tScore: 36.53\tAverage Score: 25.18\n",
      "Episode 101\tScore: 37.81\tAverage Score: 25.55\n",
      "Episode 102\tScore: 36.34\tAverage Score: 25.91\n",
      "Episode 103\tScore: 37.35\tAverage Score: 26.28\n",
      "Episode 104\tScore: 35.66\tAverage Score: 26.63\n",
      "Episode 105\tScore: 35.64\tAverage Score: 26.98\n",
      "Episode 106\tScore: 36.31\tAverage Score: 27.33\n",
      "Episode 107\tScore: 35.37\tAverage Score: 27.67\n",
      "Episode 108\tScore: 33.60\tAverage Score: 27.99\n",
      "Episode 109\tScore: 35.24\tAverage Score: 28.33\n",
      "Episode 110\tScore: 36.44\tAverage Score: 28.68\n",
      "Episode 111\tScore: 36.34\tAverage Score: 29.02\n",
      "Episode 112\tScore: 36.22\tAverage Score: 29.37\n",
      "Episode 113\tScore: 37.67\tAverage Score: 29.73\n",
      "Episode 114\tScore: 36.58\tAverage Score: 30.08\n",
      "\n",
      "Environment solved in 14 episodes!\tAverage Score: 30.08\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8VfX9+PHXO3snZAIhYSQM2WAYAioIWKpVsdq62mqrpWrrqH5tte2v2mGrdtja2lqtA0cRxb1wg7IiCYS9AgSyIAnZkHXv/fz+ODcxQAIBc/f7+XjkkXvPPeee98mB+76fLcYYlFJKBa4gTweglFLKszQRKKVUgNNEoJRSAU4TgVJKBThNBEopFeA0ESilVIDTRKCUUgHO5YlARIJFZL2IvO18PlhEckVkl4gsFpEwV8eglFKqe+4oEdwGbOv0/EHgYWPMUKAGuN4NMSillOqGuHJksYgMABYC9wN3ABcBlUBfY4xNRM4C7jPGfO1E75OcnGwGDRrksjiVUsof5efnVxljUk62X4iL4/gb8DMg1vk8Cag1xticz0uA9K4OFJEFwAKAzMxM8vLyXByqUkr5FxHZ15P9XFY1JCLfACqMMfmdN3exa5dFEmPM48aYHGNMTkrKSROaUkqp0+TKEsF04GIRuQCIAOKwSggJIhLiLBUMAMpcGINSSqmTcFmJwBhzjzFmgDFmEHAl8Ikx5hrgU+By527XAm+4KgallFIn54lxBD8H7hCRQqw2gyc9EINSSiknVzcWA2CMWQYscz7eA0x2x3mVUkqdnI4sVkqpAKeJQCmlApwmAqV6SfXhVhav3U+b3eHpUJQ6JW5pI1DK37XY7NywcC3r9tdSUFzHHy4djUhXw2aU8j6aCJQ6DcXVRzjcamNE3ziMMfzytc2s21/LucNSWPTFfgYnR7HgnCzqmtp4d1M5R1rtxEeGkhobzvTsZIKDNEko76GJQKlTVFx9hEseXUn14VbGZyQwom8sS/JLuHX2UG6fPZRbFq3nj+9tp6C4lk+3V9LUZj/q+FH94/jtJaM5c2AfD12B6k55XRN3vrSBBy8bS0ZilEdisDsMAgS58cuCthEodQoamtu4YWEeNruDu742nIbmNl5cW8zXRqVx++yhBAUJf/n2OCZm9uGT7RVcPK4/b/1kBgW/nstnd83i71eO51BjK5f9exX3vrEZV076qE7dwlX7WLX7EIvXFnvk/HaHYe5fl3P/u9tOvnMv0hKBUj1kdxhue7GAwspGnv3BZKZnJ3PzzCy2lTeQnRrT8Q0uIjSYRT+cSpvdQXT4l//FEqLCyEyKYs4ZafzxvW0sXL2PyYOTuHBsPwCOtNp48Yti5k9IJzG6Z8t02B2G37+zFbvDcO9Fo7TK6SQO1jfzj092cdPMbNITIo96rdXmYEm+lQDe3FDGnecP62jnWb+/hsToMAYmRbs0vtw9h9hTdZiSNfv40blDSI2NcOn52mkiUOoYrTYHNzybR3H1EUb2jyMrJYadBxrI3XuImiNt/H7+aKZnJwMgIozsH3fce4SFBBEW0nWBOzo8hPsuGkVBcS33vbWFGUOTiQkP4bYXC/hw60HeKCjlfz+celQS6YrdYfjZko28sq6kI+4/fnOMzzVSV9Q389yaffzwnCHERYS69FxPrdzL82v289HWCp69fjLD0mI7Xvtw60GqGlu5cGw/3tlYTkFxLRMy+3CosYVr/pvL+IwE/vfDqS6N7/WCUiJDg2mx2XlyxV7u+foZLj1fO60aUuoYv39nK5/trCQjMYoNxbU88vEuNpXWMfuMNB77zkS+M3XgVz5HSHAQf7x0LIcaW3hw6Xb+8O42Ptx6kMsmDmBzWT03Pp9Pq637bqgOh+HuV6wk8NM5w/jJrGxeXFvMH97dxsaSWh7/bDe/eWsLZbVNXzlWV6psaOGqJ9bwj08KeWZl0Un3f29TObe9uJ6X84qpa2o7pXPZ7A5eW1fKuIwE7MbwrcdWk7+vuuP1RV/sJz0hkj/MH0NYSBBvbrDmw3z8sz0cabWTu7eauiOnds5T0dxm571NB7hgTD8uHNuf51fvc+n5OtMSgVKdvLquhGdX7+OHZw/mlxeOBKz/oBGhwb1+rjED4rlu2mCeWrkXgO9PH8S9F41iyuBEfvbKRm5fvJ7fXTKapJjw447974o9vJxfwu1zhnLbnKEYY2hobuOJz/fyxOfW+4UECa+uK+XBy8Yyb3TfbuNYVVjFB1sPMi0riWnZyRxptbGysIotpfUscGH1hPVNew1ltc2M6BvLs6ut6pDwkOP/1naH4U/v7+Cx5buJCgvmjYIyfvHaJqYOSWJ6djIzspMZnR5/wvN9XlhFRUMLv71kNKP6x/G9p77gqidyuX/+aCYNSmRFYRV3zB1GfFQo5w1P5e2N5dx4bhYLVxcxom8s2w80sGxnBZeM73IJla/sk+0VNLTYmD+hP0nR4by1oYxnVxdxy+yhLjlfZ5oIlHLaVl7PL17bxJTBifx83oiO7a5IAu3uPH8Yy3ZWMKJvLL9yJp5vT8qgtqmVB97bzrIdlXx/+iAWnJNFfKRVbVJ3pI1/flLIrOEp3D5nGGBVUd170SjGDEggLCSIqYMTOdxq59ZF67nx+Xx+PCuLu7424rjzF1cf4cbn86lvtvHMqiKCgwS748sG7PL6Zh69emKvX7cxhh8szGPfoSM8fd0kbA7D9576grc3lHPZmQOO2reivpm7lmxk+c5Krp6Syb0XjWR7eQNvbyxj+c5KHnhvOwAPXzGOSycM6Op0ACzJL6FPVCjnjUglLCSIV26axi2L1nHXko0MTo4mOEj4dk4GABeP78/SLQf40XP5tNkNj14zkSv+s5qPtrkuEby+vpSU2HCmZVndi88bkcrTq4q44ewhRIa57t8gaCJQCrC+cf78lY3EhIfyj6snEBLsnlrT6PAQPrj9nOPOt+CcLM4bkcbfPtrJo5/u5qOtFbx801nERYTy7+W7aWix8bN5R3+wBwUJlx/zIfrKTdP4v5c38NjyPVx71iBS4778dt9qc3DLovUYAx/feS4V9S2sLKwiOjyEs4cm89G2g/zto11cNamKGUOTe/W61+2vYUNxLX+4dAzTspMxxjA0NYYnV+zlmxPTEbES0gu5+/jT0h202Bzcf+lorpliVcuNy0hgXEYCv7zQShRXPr6GxWuLu00EdUfa+HDrQa6enNnRdpMYHcbC70/mwaXbeeLzvcwdmUbfeOvvc96IVGLCQygoruXyMweQlRLDeSNSeW/zAVptji7bf4wxJ22fMcbw4NIdtNkdjEmPZ3R6PEOSo2lotrFsRyXfPWtgR4P/zTOzuPyx1SzbUcHXx/Q77b91T2giUApYvLaYjSV1/P3K8W7rqdGuu6STnRrDP6+eyBWTKvn+02u5+fl1PHDZGJ5euZf549M5o9/xjdTHCgsJ4qdzh/HmhjJezi/hx7OyO1778wc7KCiu5V/XTCQrJYaslBjOyko66vyvrS/l129uZult53Tb+L2ppI77393KzOGpXDohnbS4k//9luSXEBUWzCXj+wNWieYHMwZzz6ubWLOnmmabnT+/v4MtZfXMyE7md/NHMzi56x47qXERfGNsP/75aSGVDS2kxB5flfbWxjJabY7jEmVIcBC/vHAk80b3ZVCnHkERocHMG92X19eXcut5VtXMnDPSeCmvhLVF1R2dBQDa7A5ufC6f+MhQ/nrF+BNed+7eah5bvvuokld8ZCjpCZG02h3M71TayBmUyEd3nEN2amx3b9drtLFYBbzqw6089P52pgxO5OJx/T0dznHOHprCH785hhWFVVzyz5U4jOGOucN6fPzg5GimDklk8dpiHM4Pn893VfL4Z3v47tSBXNDNt82I0GDuu2gUeyoPd7RjdGX5zgrW7Knmgfe2c9YfP2b6A58w/rcfMPxX7/HL1zbRfMyAuuY2O29vKOfro/sd1TPq0gnp9IkK5YaFa/n+02upa2rjkasm8Nz1k7tNAu0uHNsfh4GlWw50+fqS/BJG9I1lVBc9vADOHJh4XFvMLy44g1dumkZmkjWwbMbQZMJDgvhw68Gj9vvz+zv4eHsFr64vZdXuqhPGuXBVEQlRoWy493zev/0cHrp8LPNG9aXZZmdaVhKj04+Ozx1JALREoAJMRX0zr64v5a0NZcRHhnLBmH7k76uhodnG7+Z77/xA38rJYH/1Ef7xSSHXTRt0yqNer5yUye2LC1i95xDjMxK459VNDEmJ5pcXnrh74qwRqcwdmcYjH+/iqsmZHe0UnR2obyYhKpRXb5rG6+tLKa5pIjYihMYWGy/k7mdDSS3/vubMjpjf33KAhhYbl515dF17RGgwP56VzbOr93HPOUP4dk5Gt6WQYw1LiyErJZp3N5bz3WN6da3YVUVBcS33XjTylO5vYnTYUeM5osJCmJFtVZm1v9cHWw7wn8/2cEVOBisKq/jju9t548fTuxwVXFrbxPtbDvCjc7OICQ9heN9YhveN7WiX8CRNBCogHGm18bu3t/FSXjF2h2FCZgIH6pv51eubAbhhxuCj+pR7ozvmDmNiZp+jqm96at7ovsS/GcqLa4v5eFsFJTVNvHzjWT1qCL9t9tCO8Q3fO2vQca8fqGuhb1wEQ1JiuOP84Ue9dsHoftzxUgEXPvI5j14zkbOHprAkv4T0hEimDj7+Om44ewg3nD3klK9PRLhwzPHVQ+0D7jISI7l6SuYpv++x5oxM4+PtFfy/NzbTJyqMZ1YVMSY9nt/OH8W7m8r56eINvLWxrMsG5edW7wPole7Hvc1liUBEIoDPgHDneZYYY+4VkWeAc4E6567XGWMKXBWHCjxNrXYu+/cq4iJDuGziAAYmRXP3KxvZe+gw1541iO+dNZAhKTEYY9hxsIG8ohoum9h9bxNvISLMGpF6WsdGhAZz6YR0Xsjdh81h+M7UTCYNSuzRsaPT4xmTHs//cvfz3akDj/tWXdHQ3G27wJyRabx9y9kseC6P655eyy3nZbOysIqfzMru9bl0Lhzbn0c+KWTplgMdpYKX8orZfqCBf10zsctuqafq/JFpPLZ8N6/kl9LUZqdffETHe18yLp0nPtvLQ0t3MG9036PO19xm58W1+/naqL7HjWj2Bq4sEbQA5xljGkUkFFghIu85X7vLGLPEhedWAeyF3H1sLa8nPSGSu5ZsBKBffAT/u2HqUd+mRYQRfeMY0ffkja7+4MrJGTyzqoh+8RFHdY/tiasmZ/KL1zZ1jLbt7ECdNQ6gO5lJUSy5aRq3LlrP3z7aBXBcF9HecGz1UENzG3/5YAeTBvXh6ycYR3EqkmLCWX7XLMDqAWTMl5PDBQUJv7jgDL7zZC6v5JceVQJ5o6CU2iNtXDttUK/E0dtclgiMNZtWo/NpqPNHZ9hSLtXUauex5XuYnp3E89dPcXZTrOObE9NJiOrZ/D3+akTfOH4+bwSTB/ch9hSncrh4fH9+/85WFn2x/6hEYLM7qGq0qoZOJCY8hCe+l8PfP9pJY4vdJXP2tFcPPfJJIRN++wFRYSFUNbby5LWTXNL2IyIc+7bTs5MYmBTFR9sOHpUIXsorYXhaLFMG96wU5m4ubSMQkWAgH8gGHjXG5IrITcD9IvJr4GPgbmNMiyvjUIHjhdx9VDW28K/ZExERzhyYyJkDvfM/nyfcNDPrtI6LCQ/hkvH9eX19Gb/6xsiOOYEqG1twGEiLP3mX0eAgOa4Nobf9YMZgIsKCKa1poryumSsnZTAuI8Gl5+xMRJg1PJUX1+7vGJFe0dDMuv01/HTOMK/tjODS7qPGGLsxZjwwAJgsIqOBe4ARwCQgEfh5V8eKyAIRyRORvMrKSleGqfyEVRrYzfTsJCZ76TcvX3bV5Eya2uy8UVDWse1AXTPASUsE7pIQFcbNM7O5/9IxPHXdJLdMz3CsmcNTaG5zsGbPIQA+2lqBMXD+qDS3x9JTbhlHYIypBZYB84wx5cbSAjwNTO7mmMeNMTnGmJyUlBR3hKl83LOri6hqbO2YdkH1rjHp8QxLi+H9zV/21T9YbyWCngwiCxRThyQRERrEsh3WF9gPth4gMzGK4V7cK81liUBEUkQkwfk4EpgDbBeRfs5tAswHNrsqBhU4th+o568f7mTW8JQe94ZRp0ZEGJ0eT2FFY8e2jhJBD6qGAkVEaDDTs5KtSeSa21hVeIjzR6Z5bbUQuLZE0A/4VEQ2AmuBD40xbwMviMgmYBOQDPzehTGoANDYYuPmF9YRFxnKQ5eP83Q4fi0rJYYD9c00ttgAOFDfQmiwkBjgDfHHmjkilf3VR3hqRRGtdgdf66VeS67iyl5DG4EJXWw/z1XnVIHHWjh+E0VVh/nfD6d2Oc+M6j1ZKTEA7KlsZOyABA7WN5MaG+HW9XV9wazhVnX2v5YVkhQdxsRM716fWucaUl7NZnd01EMfq83u4Hdvb+ONgjLumDuMqUNOfcStOjXZqVa3z92VVvXQwfpm0uI0+R5rQJ8ohqXF0GJzMOeMNK9fQlQTgfJqz63ZxzkPfdpRF92urLaJK/6zmqdW7uW6aYO4eWZ2N++gelNmojVv/+6Kw4A1z5C2D3Rt1nBrFPjXRntvb6F2OteQ8mp5RTW02BwsXlvMbXOsroA1h1u5+J8raGq188+rJ/CNsd43Y6i/CgsJYmBi1Jclgrpmzh2mvfq68p2pA2mxOZiR7f1/Hy0RKK+2ucyakmrx2v0d87f/d8UeDh1u5cUFZ2kS8ICs1Bh2VzbS0NzG4Va714wh8DYZiVHcd/GoHs+g6kneH6EKWHVNbew7dIRxGQmU1TWzbEcFtUdaWbhqHxeM7seYASdeo1a5RlZKDEVVRyir1a6j/kITgfJaW8vqAbj1vGxSYsN5IXc/T63YS2OLjVtma5uAp2SlRNNqd7C2qBrQwWT+QNsIlNfaXGpVC43PSOCKnAweXVZI7p5DzBvVN2BmDPVGWalWF9KVhdZqXFo15Pu0RKC81uayOvrHR5AUE86Vk61VnA632rnVA/PHqC9lJVuJYLVzLh0tEfg+LREor7W5tI5R6VY7wIA+UXzrzAEEiTCym3VnlXvER4WSHBNOVWMLcREhRIZ99QVflGdpIlBe6XCLjT1Vh7l43JdL/un0Ed4jKyXaWodAG4r9glYNKa+0tbweY2B0un7790bt7QRaLeQfNBEor9TeUDw6XbuIeqNs55xD2lDsHzQRKK+0ubSelNhw/cbppdpLBFo15B80ESivtKWsjtHaKOy1hqfFEhwkDHLB2sPK/bSxWHmd5jY7uyoamTvS+yfrClR94yNYetvZDErWROAPNBEor7OlrB67wzCqv7YPeLOhXrz0ojo1WjWkvE5BcS0AEzITPByJUoHBlWsWR4jIFyKyQUS2iMhvnNsHi0iuiOwSkcUiomvcqaMUFNfSPz5CG4qVchNXlghagPOMMeOA8cA8EZkKPAg8bIwZCtQA17swBuWDCoprGK+lAaXcxmWJwFganU9DnT8GOA9Y4ty+EJjvqhiU76lqbKG4uonxGZoIlHIXl7YRiEiwiBQAFcCHwG6g1hhjc+5SAqR3d7wKPAX7rfaB8Rnevdi3Uv7EpYnAGGM3xowHBgCTgTO62q2rY0VkgYjkiUheZWWlK8NUXqSguJbgIGGMjihWym3c0mvIGFMLLAOmAgki0t5tdQBQ1s0xjxtjcowxOSkp3r/mp+odBcW1DE+L1RktlXIjV/YaShGRBOfjSGAOsA34FLjcudu1wBuuikH5FofDsKG4VhuKlXIzVw4o6wcsFJFgrITzkjHmbRHZCrwoIr8H1gNPujAG5UP2VDXS0GLThmKl3MxlicAYsxGY0MX2PVjtBUodZb2zoXiilgiUcisdWay8RkFxLbERIQxxLoWolHIPTQTKa+QV1TBuQAJBQeLpUJQKKJoIlFfYUlbHjoMNzDkj1dOhKBVwNBEor/DS2mLCQoKYP0HHFyrlbpoIlMc1t9l5bX0p80b1JSFK5yBUyt00ESiPMObLAeXvbzlAfbONKydleDAipQKXJgLldnsqGxn3mw/40/vbsdkdvPhFMRmJkUwdkuTp0JQKSLpCmXK7z3dVUd9s49FPd7NiVxUbSur4v/OHaW8hpTxESwTK7dbvryE1Npy/XTGeXRWNBAlcfqZWCynlKVoiUG5XUFzLhMwE5k9IZ2JmH0prm+gbr6uRKeUpWiJQblVzuJWiQ0c61hvITIrirCxtG1DKkzQRKLdqX5heJ5ZTyntoIlButb64liCBsQN04RmlvIUmAuVWBcW1DEuLJTpcm6eU8haaCJTbOByGgv01TNBpppXyKpoIlNvsPXSY+mYbE3RheqW8iiYC5TYFzoVndClKpbyLK9cszhCRT0Vkm4hsEZHbnNvvE5FSESlw/lzgqhiUd1lfXENMeAhZKbrwjFLexJUtdjbgTmPMOhGJBfJF5EPnaw8bY/7swnMrL1RQXMu4jHiCdSoJpbyKy0oExphyY8w65+MGYBugk80HqMYWG9vKG7R9QCkv5JY2AhEZhLWQfa5z009EZKOIPCUi+skQAPKKqrE7jM4wqpQXcnkiEJEY4BXgdmNMPfBvIAsYD5QDf+nmuAUikicieZWVla4OU7lY7t5qQoKEiQO1oVgpb+PSRCAioVhJ4AVjzKsAxpiDxhi7McYBPAFM7upYY8zjxpgcY0xOSkqKK8NUbpC75xBjB8QTFaYDyZTyNq7sNSTAk8A2Y8xfO23v12m3S4HNropBeYcjrTY2ltQxRauFlPJKrvx6Nh34LrBJRAqc234BXCUi4wEDFAE/cmEMygvk76vBpu0DSnktlyUCY8wKoKt+gu+66pzKO+XuqSY4SDhzoPYLUMob6chi5XK5ew8xOj2eGJ1oTimvpIlAuVRTq50NxXVMHZzo6VCUUt3QRKBcav3+GlrtDqYM0USglLfSRKBcas3eaoIEcgZpIlDKW2kiUC61fEcFYwckEBcR6ulQlFLd0ESgXOZgfTMbSuqYOzLN06EopU5AE4FymQ+3HgTgfE0ESnk1TQTKZT7cepBBSVFkp+r6A0p5M00EyiUamttYvfsQc0emYc02opTyVpoIlEss31lJq93B3JF9PR2KUuokNBEol/hw60ESo8N0WgmlfICO+Ve95v0tB2ixORjRN5ZPt1dw/qi+uiylUj6gx4lARGYAQ40xT4tIChBjjNnrutCUL6mob+ZHz+UftU27jSrlG3qUCETkXiAHGA48DYQCz2NNNa0UKwqrAHjkqgm02RzUHGll9ohUD0ellOqJnpYILsVac7h9MfoyEYl1WVTK53y+q4qk6DC+MaYfQVodpJRP6WljcasxxmAtJoOIRLsuJOVrHA7D57uqmDE0WZOAUj6op4ngJRH5D5AgIj8EPsJab1gpth9ooKqxhbOH6trSSvmiHlUNGWP+LCJzgXqsdoJfG2M+PNExIpIBPAv0BRzA48aYv4tIIrAYGIS1VOW3jTE1p30FyuM+31UJwNlDkz0ciVLqdJw0EYhIMPC+MWYOcMIP/2PYgDuNMeuc7Qn5IvIhcB3wsTHmARG5G7gb+Pmph668xee7qhieFktaXISnQ1FKnYaTVg0ZY+zAERGJP5U3NsaUG2PaG5cbgG1AOnAJsNC520Jg/ilFrLxKU6udL4qqmaGlAaV8Vk97DTUDm5zf6A+3bzTG3NqTg0VkEFavo1wgzRhT7jy+XES0j6EP+6KomlabQ6uFlPJhPU0E7zh/TpmIxACvALcbY+p7OgGZiCwAFgBkZmaezqmVG3y87SBhwUFMGZzk6VCUUqepp43FC0UkDBjm3LTDGNN2suNEJBQrCbxgjHnVufmgiPRzlgb6ARXdnPNx4HGAnJwc05M4lfsYY3hs+R6eXb2P+eP7ExkW7OmQlFKnqUfdR0VkJrALeBT4F7BTRM45yTECPAlsM8b8tdNLbwLXOh9fC7xxijErD3M4DL99eysPLt3OxeP689Dl4zwdklLqK+hp1dBfgPONMTsARGQYsAg48wTHTAe+i9W2UODc9gvgAaxxCdcD+4FvnU7gynPe3lTO0yuL+MH0wfzqwjN0EJlSPq6niSC0PQkAGGN2Oqt9umWMWQF09wkxu4fnVV6o8GADQQL3XDBCk4BSfqCniSBPRJ4EnnM+vwbIP8H+yo+V1jaTFhdBaLAuZ6GUP+hpIrgJ+DFwK9a3/M+w2gpUACqrbaJ/QqSnw1BK9ZKeJoIQ4O/tjb7O0cbhLotKebWyuibGDkjwdBhKqV7S07L9x0Dnr4CRWBPPqQDjcBjKa5tJ1xKBUn6jp4kgwhjT2P7E+TjKNSEpb1Z1uIVWu4P0BJ1XSCl/0dNEcFhEJrY/EZEcoMk1ISlvVlbbDKBtBEr5kZ62EdwOvCwiZViL0/QHrnBZVMprldVa+V8TgVL+44QlAhGZJCJ9jTFrgRFY6wjYgKWALlwfgDQRKOV/TlY19B+g1fn4LKyRwY8CNTjnAVKBpbS2iZjwEOIielqYVEp5u5P9bw42xlQ7H1+BtcrYK8ArnaaNUAGktKaJ/gkR9HQWWaWU9ztZiSBYRNqTxWzgk06v6VfCAFRWp4PJlPI3J/swXwQsF5EqrF5CnwOISDZQ5+LYlBcqq23WwWRK+ZkTJgJjzP0i8jHQD/jAGNO+LkAQcIurg1PepanVTvXhVh1MppSfOWn1jjFmTRfbdromHOXNyuraewzpYDKl/IlOH6l6rKPraLyWCJTyJ5oIVI+1J4L0PpoIlPInmgjUCX2y/SDLdljLSpfWNhMkkBanVUNK+ROXJQIReUpEKkRkc6dt94lIqYgUOH8ucNX5Ve/4zVtbufmFdZTVNlFW26QL0ijlh1z5P/oZYF4X2x82xox3/rzrwvOrr6ihuY19h45wpNXOfW9u0QVplPJTLksExpjPgOqT7qi81rbyBgCmZSXxwdaD5BXVaCJQyg95ooz/ExHZ6Kw66uOB86se2lpmjRl86PKxjOgbS6vdoV1HlfJD7k4E/waygPFAOfCX7nYUkQUikicieZWVle6KT3WypayepOgw0hMiuf/SMQQJZKXEeDospVQvc+t8QcaYg+2PReQJ4O0T7Ps4zhlOc3JyTHf7KdfZWl7PyP5xiAhnDuzD8rs9phurAAAQvklEQVRm0S9eSwRK+Ru3lghEpF+np5cCm7vbV3lWq83BzoMNjOwf17EtIzGKEO0xpJTfcVmJQEQWATOBZBEpAe4FZorIeKxVzoqAH7nq/OqrKaxopM1uGNkv7uQ7K6V8mssSgTHmqi42P+mq86netcXZUDyqf7yHI1FKuZqW81WXtpbXExkazODkaE+HopRyMU0ECoAWm53X15fS0NwGWD2GRvSLJThIVyJTyt9pIlCU1zXx7f+s4fbFBfzouXxabHa2ldVr+4BSAUITQYBbs+cQ33hkBYUHG7j2rIGs2n2IHz6bT0OLTdsHlAoQuu5wAKuob+aGhXmkxoWz+LtTyU6NJTE6nIc/stYd6tx1VCnlvzQRBLD7391Gq83Bk9dO6mgUvnV2NmW1Tby7uZwRfWM9HKFSyh20aihArdpdxRsFZdx47pCjegaJCA9cNobV98wmIjTYgxEqpdxFE0EAarM7+PUbWxjQJ5KbZ2Uf97qIEBOuhUWlAoX+bw9A/8vdT2FFI//9Xo5+61dKaYkgEL3nrP+fMzLN06EopbyAJoIA09RqZ92+WmZkJ3s6FKWUl9BEEGDy9lXTancwXROBUspJE0GAWVl4iJAgYfLgRE+HopTyEpoIAsyq3VWMz0ggWnsFKaWcNBEEkLojbWwqrWOaVgsppTrRRBBAVu85hDEwPSvJ06EopbyIJoIAsnp3FZGhwUzI7OPpUJRSXkQTQQBZufsQkwYnEhait10p9SWXfSKIyFMiUiEimzttSxSRD0Vkl/O3fjV1k4P1zRRWNGq1kFLqOK78avgMMO+YbXcDHxtjhgIfO58rN/h8VxWAjh9QSh3HZYnAGPMZUH3M5kuAhc7HC4H5rjq/OtqyHRWkxIYzStcYUEodw92VxWnGmHIA5+/U7nYUkQUikicieZWVlW4L0B/Z7A4+31XFucNSENE1iJVSR/PaVkNjzOPGmBxjTE5KSoqnw/FpG0pqqWtq49xh+ndUSh3P3YngoIj0A3D+rnDz+QPS8h2VBAmcPVTbB5RSx3P3PANvAtcCDzh/v+Hm8/utPZWN3PPqJjISoxjZL45zh6eQlRIDwLKdlUzI7ENCVJiHo1RKeSOXJQIRWQTMBJJFpAS4FysBvCQi1wP7gW+56vyB5u2N5XxRVM3uysMsyS8h6oNgXr7xLNLiIthYUsedc4d5OkSllJdyWSIwxlzVzUuzXXXOQJa79xAj+sbx3m1nU1R1mKueWMP1z+Rx3fRBAMwc3m27vFIqwHltY7HquVabg/x9NUxxTi09KDmaJ6+dRH1zGw+8t53kmDDtNqqU6pYmAj+wqbSO5jZHRyIAGNk/jkeunICIVRoICtJuo0qprumk9H4gd+8hgOMWm5kzMo03fzyDzKQoT4SllPIRmgj8QO6earJTY0iKCT/utTED4j0QkVLKl2jVkI+z2R3kFVUfVS2klFKnQhOBj9taXs/hVjtThuisokqp06OJwMfl7rHm9dMSgVLqdGki8HG5ew8xKCmKtLgIT4eilPJRmgh82K6DDeTuqWbKYK0WUkqdPk0EPsgYw8JVRXzjHysIDQniO1MHejokpZQP0+6jPugP727jic/3MnN4Cg9dPpbUWK0WUkqdPk0EPuidjeXMHpHKf6/N0YVmlFJfmVYN+ZgDdc2U1TUzPTtZk4BSqldoIvAx6/bXADAhM8HDkSil/IUmAh+zbl8NYSFBjOqvU0copXqHJgIfs764ljHp8YSF6K1TSvUO/TTxIa02B5tK65io1UJKqV7kkV5DIlIENAB2wGaMyfFEHL5mS1kdrTYHEzL7eDoUpZQf8WT30VnGmCoPnt/nrNtfC8BETQRKqV6kVUM+ZP3+GvrHR9A3XgeQKaV6j6cSgQE+EJF8EVnQ1Q4iskBE8kQkr7Ky0s3heY/Vuw9xqLEFgPX7a5kwUEsDSqne5amqoenGmDIRSQU+FJHtxpjPOu9gjHkceBwgJyfHeCJIT9t5sIGrnlhDfGQoC84ZQmltE9+fPsjTYSml/IxHSgTGmDLn7wrgNWCyJ+Lwdst2VAAwNDWGP72/A4CJWiJQSvUyt5cIRCQaCDLGNDgfnw/81t1x+IJlOyoZnhbLyzeexVsby8kvqmZMug4kU0r1Lk9UDaUBrznnyQkB/meMWeqBOLza4RYba4uq+cH0wYgIF4/rz8Xj+ns6LKWUH3J7IjDG7AHGufu8vmbV7kO02Q3nDkvxdChKKT+n3Ue91PKdFUSFBZMzSNciVkq5liYCL2SMYdmOSqZlJeucQkopl9NPGS+0u/IwJTVNzByu1UJKKdfTROCFlu+0BtBp+4BSyh00EXiZ9ftreCF3H1kp0WQkRnk6HKVUANA1i71EWW0Tv3lrC+9vOUhyTBgPfHOsp0NSSgUITQReoKK+maueWENVQwt3zB3G9TMGEx2ut0Yp5R76aeNhtUda+e6TX1DZ0MILN0zRtQaUUm6nicBNXl1XwmvrSxmaGsvo9DiCg4Sy2mbe3ljG3kOHeea6SZoElFIeoYnADZZuLufOlzfQLy6CtUXVNLc5Ol5Ljgnj0asnMi072YMRKqUCmSYCF1tbVM2tLxYwISOBF26YSmiwsLfqMCLQLz5S2wKUUh6nn0I9YIxh8dpith9o4MyBfZg4sA8H65tZt6+GkpomJg1K5JxhycRGhALQ3GYnr6iGzwsrWZS7nwF9Inny2klEhgUDMDQt1pOXo5RSR9FEcBI2u4N739zCC7n7CQsO4plVRUe9HhZibQsNFvonRFJ9uJWGZhsAocHCpEGJPHjZWPpEh3kgeqWUOjlNBF0oq23iQH0zDc02nlm5l093VHLTzCzumDuMbeX1FBTXkhobzoTMPiRFh5G/r4ZPtldQVtdMUnQYSdFhjEqPY8rgJK36UUp5Pf2U6qT2SCsPLt3Ooi+KO7YFBwl/uHQMV0/JBGDsgATGDkg46rgpQ5KYMiTJrbEqpVRv8etE8Pr6UlYWVvHri0Z21N8DVDa0kBgdRnCQANDYYuO1dSU8/NEu6prauH7GYGYMTSYuIoT+CZH0i4/01CUopZTL+XUiKKtr4pV1JazafYg/f2scIcHC3z/axYrCKuIjQ5k6JJHo8BCWbj7AkVY7Zw7sw+/nj+aMfnGeDl0ppdxGjDHuP6nIPODvQDDwX2PMAyfaPycnx+Tl5Z3WufL31fB/L29gb9VhwOq3f/WUgRysa2bl7ipqDrdy0bj+XDEpg/EZCTiX0FRKKZ8nIvnGmJyT7eeJxeuDgUeBuUAJsFZE3jTGbHXF+c4c2Id3bp3BUyv2EhMewhWTMju6cYLVNVQ//JVSgcwTVUOTgULn2sWIyIvAJYBLEgFAVFgIPzlvaJevaRJQSgU6T6xHkA4Ud3pe4tx2FBFZICJ5IpJXWVnptuCUUirQeCIRdPUV/LiGCmPM48aYHGNMTkqKrtSllFKu4olEUAJkdHo+ACjzQBxKKaXwTCJYCwwVkcEiEgZcCbzpgTiUUkrhgcZiY4xNRH4CvI/VffQpY8wWd8ehlFLK4pEBZcaYd4F3PXFupZRSR/NE1ZBSSikvoolAKaUCnEemmDhVIlIJ7DuFQ5KBKheF40l6Xb5Fr8u3+ON1DTTGnLT/vU8kglMlInk9mV/D1+h1+Ra9Lt/ir9fVE1o1pJRSAU4TgVJKBTh/TQSPezoAF9Hr8i16Xb7FX6/rpPyyjUAppVTP+WuJQCmlVA/5XSIQkXkiskNECkXkbk/Hc7pEJENEPhWRbSKyRURuc25PFJEPRWSX83cfT8d6qkQkWETWi8jbzueDRSTXeU2LnXNQ+RwRSRCRJSKy3XnfzvL1+yUiP3X++9ssIotEJMJX75eIPCUiFSKyudO2Lu+PWB5xfo5sFJGJnovc9fwqEXRa/ezrwEjgKhEZ6dmoTpsNuNMYcwYwFfix81ruBj42xgwFPnY+9zW3Ads6PX8QeNh5TTXA9R6J6qv7O7DUGDMCGId1jT57v0QkHbgVyDHGjMaaG+xKfPd+PQPMO2Zbd/fn68BQ588C4N9uitEj/CoR0Gn1M2NMK9C++pnPMcaUG2PWOR83YH2opGNdz0LnbguB+Z6J8PSIyADgQuC/zucCnAcsce7ic9cEICJxwDnAkwDGmFZjTC0+fr+w5iOLFJEQIAoox0fvlzHmM6D6mM3d3Z9LgGeNZQ2QICL93BOp+/lbIujR6me+RkQGAROAXCDNGFMOVrIAUj0X2Wn5G/AzwOF8ngTUGmNszue+es+GAJXA085qr/+KSDQ+fL+MMaXAn4H9WAmgDsjHP+5Xu+7uj19+lnTH3xJBj1Y/8yUiEgO8AtxujKn3dDxfhYh8A6gwxuR33tzFrr54z0KAicC/jTETgMP4UDVQV5z15ZcAg4H+QDRWlcmxfPF+nYy//LvsEX9LBH61+pmIhGIlgReMMa86Nx9sL6I6f1d4Kr7TMB24WESKsKrtzsMqISQ4qx7Ad+9ZCVBijMl1Pl+ClRh8+X7NAfYaYyqNMW3Aq8A0/ON+tevu/vjVZ8nJ+Fsi8JvVz5x1508C24wxf+300pvAtc7H1wJvuDu202WMuccYM8AYMwjr3nxijLkG+BS43LmbT11TO2PMAaBYRIY7N80GtuLD9wurSmiqiEQ5/z22X5PP369Ours/bwLfc/YemgrUtVch+SVjjF/9ABcAO4HdwC89Hc9XuI4ZWEXRjUCB8+cCrDr1j4Fdzt+Jno71NK9vJvC28/EQ4AugEHgZCPd0fKd5TeOBPOc9ex3o4+v3C/gNsB3YDDwHhPvq/QIWYbV1tGF947++u/uDVTX0qPNzZBNWzymPX4OrfnRksVJKBTh/qxpSSil1ijQRKKVUgNNEoJRSAU4TgVJKBThNBEopFeA0ESi/JiJ2ESno9HPC0b4icqOIfK8XzlskIsmncdzXROQ+EekjIu9+1TiU6omQk++ilE9rMsaM7+nOxpjHXBlMD5yNNWDrHGClh2NRAUITgQpIzmkuFgOznJuuNsYUish9QKMx5s8icitwI9aU4FuNMVeKSCLwFNagqiPAAmPMRhFJwhqwlII12Eo6nes7WNM5h2FNHHizMcZ+TDxXAPc43/cSIA2oF5EpxpiLXfE3UKqdVg0pfxd5TNXQFZ1eqzfGTAb+iTXn0bHuBiYYY8ZiJQSwRtqud277BfCsc/u9wApjTTj3JpAJICJnAFcA050lEztwzbEnMsYsxpqbaLMxZgzWSN4JmgSUO2iJQPm7E1UNLer0++EuXt8IvCAir2NNGQHW1B+XARhjPhGRJBGJx6rK+aZz+zsiUuPcfzZwJrDWmq6HSLqfeG4o1pQGAFHGWodCKZfTRKACmenmcbsLsT7gLwb+n4iM4sTTE3f1HgIsNMbcc6JARCQPSAZCRGQr0E9ECoBbjDGfn/gylPpqtGpIBbIrOv1e3fkFEQkCMowxn2ItpJMAxACf4azaEZGZQJWx1onovP3rWBPOgTWR2eUikup8LVFEBh4biDEmB3gHq33gIawJE8drElDuoCUC5e8ind+s2y01xrR3IQ0XkVysL0RXHXNcMPC8s9pHsNborXU2Jj8tIhuxGovbpzD+DbBIRNYBy7GmcMYYs1VEfgV84EwubcCPgX1dxDoRq1H5ZuCvXbyulEvo7KMqIDl7DeUYY6o8HYtSnqZVQ0opFeC0RKCUUgFOSwRKKRXgNBEopVSA00SglFIBThOBUkoFOE0ESikV4DQRKKVUgPv/EyH+OF+35DwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f70c3bb66d8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=1000):\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_list = []\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]     # reset the environment    \n",
    "        states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "        agent.reset()\n",
    "        scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "\n",
    "        while True:\n",
    "            actions = agent.act(states)                        # select an action (for each agent)\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rewards = env_info.rewards                         # get reward (for each agent)\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            agent.step(states, actions, rewards, next_states, dones)\n",
    "            scores += rewards                                  # update the score (for each agent)\n",
    "            states = next_states\n",
    "            if np.any(dones):\n",
    "                break \n",
    "        scores_deque.append(np.mean(scores)) # np.mean(scores) because multiple agents hence avg. over all agents\n",
    "        scores_list.append(np.mean(scores))\n",
    "        print('\\rEpisode {}\\tScore: {:.2f}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores), np.mean(scores_deque)))\n",
    "        \n",
    "        if np.mean(scores_deque)>=30.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_deque)))\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            break\n",
    "            \n",
    "    return scores_list\n",
    "\n",
    "scores = ddpg()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch the agent play!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth', map_location={'cuda:0': 'cpu'}))\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth', map_location={'cuda:0': 'cpu'}))\n",
    "\n",
    "env_info = env.reset(train_mode=False)[brain_name]        \n",
    "states = env_info.vector_observations                  \n",
    "scores = np.zeros(num_agents)                          \n",
    "\n",
    "for i in range(200):\n",
    "    actions = agent.act(states, add_noise=False)                    \n",
    "    env_info = env.step(actions)[brain_name]        \n",
    "    next_states = env_info.vector_observations        \n",
    "    rewards = env_info.rewards                        \n",
    "    dones = env_info.local_done                 \n",
    "    scores += rewards                         \n",
    "    states = next_states                              \n",
    "    if np.any(dones):                              \n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}