reinforcement_learning_pong_keras_policy_gradients.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Reinforcement Learning with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this short project we are gonna **train a neural network to play Pong game** using a reinforcement learning algorithm (**Policy Gradients Method - REINFORCE**). \n",
    "\n",
    "Considering **limited time** and for **learning purposes** I am **not aiming for a perfect trained agent**, but i hope this project could help people get familiar with basic process of rl algorithms and keras. The following video took 3 days for agent to learn on a slow computer. to obtain production results, a lot of more training and tuning is required which is not our focus. \n",
    "\n",
    "prerequisites:\n",
    "familiarity with neural networks,supervised learning, tensorflow and keras, openai gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>simple 200 hidden neuron network after 1300 training batches</p>\n",
       "<video width=\"320\" height=\"420\" controls>\n",
       "  <source src=\"trained_simple_network.mp4\" type=\"video/mp4\">\n",
       "</video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<p>simple 200 hidden neuron network after 1300 training batches</p>\n",
    "<video width=\"320\" height=\"420\" controls>\n",
    "  <source src=\"trained_simple_network.mp4\" type=\"video/mp4\">\n",
    "</video>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>convolutional network after 2000 training batches</p>\n",
       "<video width=\"320\" height=\"420\" controls>\n",
       "  <source src=\"trained_convolutional_network.mp4\" type=\"video/mp4\">\n",
       "</video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<p>convolutional network after 2000 training batches</p>\n",
    "<video width=\"320\" height=\"420\" controls>\n",
    "  <source src=\"trained_convolutional_network.mp4\" type=\"video/mp4\">\n",
    "</video>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OverView"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Intro and Theory\n",
    "2. Defining the approach to modeling the problem\n",
    "3. Modeling the Network\n",
    "4. Defining loss\n",
    "5. Reward Engineering and why it is important\n",
    "6. Example of simluation and training\n",
    "7. Training the network\n",
    "8. Playing the Trained Network\n",
    "9. Loading model from file\n",
    "10. Using Convolutional Model\n",
    "11. Ideas & Thoughts\n",
    "12. References and Suggested readings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Intro and Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### what is Reinforcement Learning\n",
    "In many problems where **supervised learning** methods can be applied, **we know what is right or wrong from data, records and expers knowledge..** , so we can use methods and algorithms to **match inputs to correct outputs**. \n",
    "\n",
    "In simple terms of **rl problems, we initially don't know what is right or wrong**, what is good or bad for us. There is **an environment which we may not completely see or even understand**.\n",
    "**Like in real life, we just see what is available to us and act** upon what we observe and also based on our knowledge from the past experience.\n",
    "\n",
    "### what to do?\n",
    "similar to real life experience that **we go out and try things out and see if it works**, in reinforcement learning approach we first try things randomly, then get feedback if it was good or bad. \n",
    "\n",
    "### for example\n",
    "in an imaginary world, think you are a manager to a new department and you don't know your employees and have no access to resumes or records. you don't know which employee is designer, who is programmer, who is customer support , etc. \n",
    "\n",
    "also consider that you can't communicate with employees. you can just assign them tasks from a computer panel and see how fast they complete the task or even fail at it.\n",
    "\n",
    "what would you do in this situation?\n",
    "\n",
    "a simple approach is to try and error a lot and record the experience. Then view your records and guess roles of some employees . after that you act using the guessed roles and see if you guessed right. then repeat this process until you know everyone's role. \n",
    "\n",
    "### in reinforcement learning we define\n",
    "- **agent**: the one who is learning to win or gather benefits for himself (get more rewards)\n",
    "- **environment**: the world which agent can interact with. **agent sees states** and **takes actions** based on what he sees.\n",
    "- **reward**: the agent gets **rewards or punishments** based on his actions. we can think of low reward or negative reward as punishmen. reward can be many things. even the amount of time the agent stays alive could be considered a reward or score. usually the reward is given to agent (agent does not think and decide what is a reward and what is not)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### actually there could be Many Complexities\n",
    "- what may be **good in the short term might be bad in the long run** and vice versa\n",
    "- **how you think now affects your future experience**. if on a two way road, you have never turned to left because you didnt want to, then you dont have any experience of where it leads to. \n",
    "- **how you think affects your perception of reward**. is making money a good thing? is it bad if your money decrease? what is charity then? is smoking bad for health or good for mood?\n",
    "- you can learn from other people experience.\n",
    "- you may have **a mindset that is wrong, but works good** (http://tylervigen.com/old-version.html)\n",
    "- you have limited resources to educate yourself and learn new things. should you work for money or spend time to learn new skills? (problem of **exploration vs exploitation**)\n",
    "- what worked in the past, may not work now or in future.\n",
    "- environment may change through time, or even have some intelligence\n",
    "- we dont know what we dont know.\n",
    "- ...\n",
    "\n",
    "we dont consider these complexities here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Niccolo Machavelli (another example)\n",
    "Machiavelli said: \n",
    "\n",
    "`\n",
    "... fortune being changeful and mankind steadfast in their ways, so long as the two are in agreement men are successful, but unsuccessful when they fall out. For my part I consider that it is better to be adventurous than cautious..\n",
    "`\n",
    "\n",
    "considering the reinforcement learning problem, he says fortune(**environment**) is changeful and mankind(**agent**) is steadfast(**exploitation rather than exploration**).\n",
    "\n",
    "When agent takes proper actions, the agent benefits, but when environment changes, the agent loses(because he is steadfast, sticks to old solutions and does not learn). So Machiavelli suggests to be more adventurous than cautious(**maintaining exploration and learning over time, because the environment is changing endlessly **)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy\n",
    "\n",
    "in simple terms, policy is the mindset of agent.\n",
    "\n",
    "the agent sees the current situation (state) and chooses to choose an action. (or multiple actions)\n",
    "so we define policy as a function of state, that outputs some actions. \n",
    "\n",
    "`action = f(state)`\n",
    "\n",
    "we call the above f function, policy.\n",
    " \n",
    "\n",
    "### Policy gradients method:\n",
    "\n",
    "Policy gradients are methods that train the policy functions by gradients.  basically in each training, we push the policy to be a bit better. that means taking good actions more frequently and taking bad actions less.\n",
    "\n",
    "There are many other methods aside from reinforcement learning (for example evolutionary methods https://blog.openai.com/evolution-strategies/).\n",
    "\n",
    "\n",
    "here we use a neural network as the policy and use a policy gradient algorithm called REINFORCE.\n",
    "\n",
    "the psuedocode for REINFORCE:\n",
    "\n",
    "##### first initialize a  policy $\\pi$ with random params\n",
    "##### loop forever : \n",
    "1. play and generate episode from start state using current policy -> $S_{0},A_{0},R_{1},  S_{1},A_{1},R_{2} ,  ...  ,  S_{T-1},A_{T-1},R_{T}$\n",
    "2. loop for each timestep t=0,1,....\n",
    "    compute return from the timestep G\n",
    "    update the policy parameters: $\\theta = \\theta+\\alpha G\\nabla_{\\theta}log(\\pi)$ \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in the above formula, $\\theta$ is the neuralnet parameters in last layer.\n",
    "G is the reward from the state till end of episode and $\\alpha$ is the learning rate.\n",
    "\n",
    "$S_0$ is the initial state, $A_0$ is the action that agent selected after observing the $S_0$, and $R_1$ is the reward given to the agent by environment after taking action $A_0$ . The agent continues to play until he wins or loses or the game ends at time $T$, and we save the states and actions and rewards for each state.\n",
    "\n",
    "if you rewrite the formula mentioned above ($\\theta = \\theta+\\alpha G\\nabla_{\\theta}log(\\pi)$ ), you can see that it is similar to grad descent with -log(prob) as loss.\n",
    "\n",
    "$\\theta = \\theta-\\alpha G\\nabla_{\\theta}(-log(\\pi))$ \n",
    "\n",
    "also if you compare it with grad descent algorithm, note that the learning rate is multiplied by G or rewards. so if we get a high reward, we move much more in the direction of normal grad descent.\n",
    "\n",
    "and if we are punished and got a minus reward we move in negative direction. \n",
    "\n",
    "so it is kind of similar to a grad descent algorithm with the difference that we multiply learning rate with reward(which might be high or low or negative).\n",
    "\n",
    "**in simple words**, we experience with current mindset(current policy) and see if we get rewards or punishments. then we update the policy according to the formula above. pushing policy to take more good actions(good means resulted in reward) and less bad actions. I explain how we map actions to rewards."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in the learning phase the policy can output the probability of each action and we sample from that probability.\n",
    "in the playing phase we may opt to pick the most probable action. or just sample like the training phase"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Defining the approach "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/soroush/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import gym # i used 0.10.5\n",
    "import numpy as np # i used 1.14.3\n",
    "import datetime\n",
    "import keras # i used 2.1.16 with tensorflow 1.8.0/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### first lets see the game"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(210, 160, 3)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env = gym.make('Pong-v0')\n",
    "observation = env.reset()\n",
    "observation.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the observation is an image of size 210*160 with 3 channels (rgb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x18156496d8>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADlpJREFUeJzt3X+s1fV9x/Hna4DAFONPmFE6wWAzXTZqiSMxGjfXqmQpdYkdZLG0M72aYNImXVLUZDNLmnSu1qT7QYORVBeHulErf9ApIW1Nk2EBiwhFFCjVK+zS0kXt/NFdeO+P7+euh8u53MN5n+P5nrPXI7k553zO93u+728uL74/7vf7PooIzKx9v9HrAsz6nUNkluQQmSU5RGZJDpFZkkNkltS1EEm6SdJeSfskrerWcsx6Td34O5GkKcArwMeAYWArsDwiftzxhZn1WLe2RFcD+yLiQET8CngcWNqlZZn11NQufe7FwOsNr4eBP5hoYkmn3BzOnjOjQ2WZte7IyHs/j4gLJ5uuWyFSk7ETgiJpCBgCmHX2ND5z5+VdKqU9n7v+ytOe56Hv7e5CJf3vvfefOe15Zky/sQuVnJ6/v3/3T1uZrlu7c8PA3IbXlwCHGieIiDURsSgiFs2cOaVLZZh1X7dCtBVYIGmepDOAZcCGLi3LrKe6sjsXEaOS7gKeAaYAayPC+zo2kLp1TEREbAQ2duvzP2jNjnfaOW6y5sc77Rw31YWvWDBLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCypaxegDhpfbNo5/XyxaTPeEpklOURmSQ6RWZKPiSbgpiOdU4emI93U9pZI0lxJ35W0R9JuSZ8v4/dJekPSjvKzpHPlmtVPZks0CnwxIl6QNAvYLmlTee/BiPhqvjyz+ms7RBFxGDhcnr8taQ9V08bT9svRUbaMHG23FLOe6siJBUmXAh8Bni9Dd0naKWmtpHM7sQyzukqHSNJZwHrgCxHxFrAauAxYSLWlemCC+YYkbZO0bfS949kyzHomFSJJ06gC9FhEfAsgIkYi4lhEHAceompuf5LGDqhTZ/hMu/WvzNk5AQ8DeyLiaw3jFzVMdguwq/3yzOovc3buGuA24CVJO8rYPcBySQupGtgfBO5IVWhWc5mzcz+g+bc/DEzXU7NW+GDELMkhMktyiMySanEB6llTp7J4zvm9LsPsBFv5z5am85bILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySahWiLSNH3fXH+k6tQmTWj9JXcUs6CLwNHANGI2KRpPOAJ4BLqW4R/1RE/Fd2WWZ11Kkt0R9GxMKIWFRerwI2R8QCYHN5bTaQunU/0VLg+vL8EeB7wJcmm8n3FFk/6sSWKIBnJW2XNFTG5pQ2w2Pthmd3YDlmtdSJLdE1EXFI0mxgk6SXW5mpBG4IYNbZ0zpQhllvpLdEEXGoPB4BnqLqeDoy1sSxPB5pMt//dUCdOXNKtgyznsm2ET6zfK0Kks4EPk7V8XQDsKJMtgJ4OrMcszrL7s7NAZ6qOgozFfiXiPh3SVuBJyXdDrwG3JpcjlltpUIUEQeA328yfhS4IfPZZv3CVyyYJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJbV9Z6ukD1N1OR0zH/gr4Bzgc8DPyvg9EbGx7QrNaq7tEEXEXmAhgKQpwBtU3X4+CzwYEV/tSIVmNdep3bkbgP0R8dMOfZ5Z3+hUiJYB6xpe3yVpp6S1ks7t0DLMaikdIklnAJ8A/rUMrQYuo9rVOww8MMF8Q5K2Sdr27rvHsmWY9UwntkQ3Ay9ExAhARIxExLGIOA48RNUR9STugGqDohMhWk7DrtxY++DiFqqOqGYDK9W8UdJvAh8D7mgYvl/SQqpvizg47j2zgZPtgPoOcP64sdtSFZn1GV+xYJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWVLqpjyzunjv/WdOeD1j+o0f2LJb2hKV1ldHJO1qGDtP0iZJr5bHc8u4JH1d0r7SNuuqbhVvVget7s59E7hp3NgqYHNELAA2l9dQdf9ZUH6GqFpomQ2slkIUEc8Bvxg3vBR4pDx/BPhkw/ijUdkCnDOuA5DZQMmcWJgTEYcByuPsMn4x8HrDdMNl7ARu3miDohtn59RkLE4acPNGGxCZEI2M7aaVxyNlfBiY2zDdJcChxHLMai0Tog3AivJ8BfB0w/iny1m6xcCbY7t9ZoOopb8TSVoHXA9cIGkY+GvgK8CTkm4HXgNuLZNvBJYA+4B3qL6vyGxgtRSiiFg+wVs3NJk2gJWZosz6iS/7MUtyiMySHCKzJIfILMkhMktyiMySfD+RDYQP8v6h8bwlMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySJg3RBN1P/07Sy6XD6VOSzinjl0p6V9KO8vONbhZvVgetbIm+ycndTzcBvxsRvwe8Atzd8N7+iFhYfu7sTJlm9TVpiJp1P42IZyNitLzcQtUWy+z/pU4cE/0F8J2G1/Mk/UjS9yVdO9FM7oBqgyJ1K4Ske4FR4LEydBj4UEQclfRR4NuSroyIt8bPGxFrgDUAc35r5kkdUs36RdtbIkkrgD8B/ry0ySIi3o+Io+X5dmA/cHknCjWrq7ZCJOkm4EvAJyLinYbxCyVNKc/nU329yoFOFGpWV5Puzk3Q/fRuYDqwSRLAlnIm7jrgbySNAseAOyNi/FeymA2USUM0QffThyeYdj2wPluUWT/xFQtmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSe12QL1P0hsNnU6XNLx3t6R9kvZK6t230Zp9QNrtgArwYEOn040Akq4AlgFXlnn+aaxxidmgaqsD6iksBR4vrbN+AuwDrk7UZ1Z7mWOiu0pD+7WSzi1jFwOvN0wzXMZO4g6oNijaDdFq4DJgIVXX0wfKuJpM27S7aUSsiYhFEbFo5kzv8Vn/aitEETESEcci4jjwEL/eZRsG5jZMeglwKFeiWb212wH1ooaXtwBjZ+42AMskTZc0j6oD6g9zJZrVW7sdUK+XtJBqV+0gcAdAROyW9CTwY6pG9ysjwgc8NtA62gG1TP9l4MuZosz6ia9YMEtyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILKnd5o1PNDRuPChpRxm/VNK7De99o5vFm9XBpHe2UjVv/Afg0bGBiPizseeSHgDebJh+f0Qs7FSBZnXXyu3hz0m6tNl7kgR8CvijzpZl1j+yx0TXAiMR8WrD2DxJP5L0fUnXJj/frPZa2Z07leXAuobXh4EPRcRRSR8Fvi3pyoh4a/yMkoaAIYBZZ09LlmHWO21viSRNBf4UeGJsrPTgPlqebwf2A5c3m98dUG1QZHbn/hh4OSKGxwYkXTj2LRCS5lM1bzyQK9Gs3lo5xb0O+A/gw5KGJd1e3lrGibtyANcBOyW9CPwbcGdEtPqNEmZ9qd3mjUTEZ5qMrQfW58sy6x++YsEsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS8reHt4RvxwdZcvI0V6XYdYWb4nMkhwis6RWbg+fK+m7kvZI2i3p82X8PEmbJL1aHs8t45L0dUn7JO2UdFW3V8Ksl1rZEo0CX4yI3wEWAyslXQGsAjZHxAJgc3kNcDNVg5IFVC2xVne8arMamTREEXE4Il4oz98G9gAXA0uBR8pkjwCfLM+XAo9GZQtwjqSLOl65WU2c1jFRaSf8EeB5YE5EHIYqaMDsMtnFwOsNsw2XMbOB1HKIJJ1F1cnnC806mjZO2mQsmnzekKRtkraNvne81TLMaqelEEmaRhWgxyLiW2V4ZGw3rTweKePDwNyG2S8BDo3/zMYOqFNn+CSh9a9Wzs4JeBjYExFfa3hrA7CiPF8BPN0w/ulylm4x8ObYbp/ZIGrlioVrgNuAl8a+zAu4B/gK8GTpiPoacGt5byOwBNgHvAN8tqMVm9VMKx1Qf0Dz4xyAG5pMH8DKZF1mfcMHI2ZJDpFZkkNkluQQmSU5RGZJqk6m9bgI6WfAfwM/73UtHXQBg7M+g7Qu0Pr6/HZEXDjZRLUIEYCkbRGxqNd1dMogrc8grQt0fn28O2eW5BCZJdUpRGt6XUCHDdL6DNK6QIfXpzbHRGb9qk5bIrO+1PMQSbpJ0t7S2GTV5HPUj6SDkl6StEPStjLWtJFLHUlaK+mIpF0NY33biGaC9blP0hvld7RD0pKG9+4u67NX0o2nvcCI6NkPMAXYD8wHzgBeBK7oZU1trsdB4IJxY/cDq8rzVcDf9rrOU9R/HXAVsGuy+qluc/kO1ZX9i4Hne11/i+tzH/CXTaa9ovy7mw7MK/8ep5zO8nq9Jboa2BcRByLiV8DjVI1OBsFEjVxqJyKeA34xbrhvG9FMsD4TWQo8HhHvR8RPqO6Du/p0ltfrEA1KU5MAnpW0XdJQGZuokUu/GMRGNHeVXdC1DbvX6fXpdYhaamrSB66JiKuoeu6tlHRdrwvqon79na0GLgMWAoeBB8p4en16HaKWmprUXUQcKo9HgKeodgcmauTSL1KNaOomIkYi4lhEHAce4te7bOn16XWItgILJM2TdAawjKrRSd+QdKakWWPPgY8Du5i4kUu/GKhGNOOO226h+h1BtT7LJE2XNI+qc+8PT+vDa3AmZQnwCtVZkXt7XU8b9c+nOrvzIrB7bB2A86naK79aHs/rda2nWId1VLs4/0P1P/PtE9VPtfvzj+X39RKwqNf1t7g+/1zq3VmCc1HD9PeW9dkL3Hy6y/MVC2ZJvd6dM+t7DpFZkkNkluQQmSU5RGZJDpFZkkNkluQQmSX9LySKQlwfIegmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x18145819b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(observation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x18156d6fd0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADnxJREFUeJzt3X2MXPV1xvHvExODitPgF0DUmGIjExVQ65AVJUEgWkoCThVDJFKjmrgp6oJkS7GSSjUgNahSpDTFIEWtiEBYmJgaaB0CUkiCa0VBUYBgiAE7xvgFBy+27GSJgJYo1Ob0j/vbZryexeM5M547w/ORVnPnN/fOPVe7j++L75xRRGBm7ftArwsw63cOkVmSQ2SW5BCZJTlEZkkOkVlS10Ik6QpJWyVtl7S8W+sx6zV14/+JJE0CXgYuB0aAZ4BrI+LnHV+ZWY91a090AbA9InZGxDvAA8CCLq3LrKeO69L7zgR2NzwfAf50opklvefucNbvT+pQWWat2/3mwV9FxMlHmq9bIVKTsUOCImkYGAaYesIH+MqlH+5SKe25/BMfP+pl1v3kyS5U0v82fOnTR73M0O3f7UIlR2fZ93/9i1bm69bh3Agwq+H56cCexhki4q6IGIqIoSmTm2XOrD90K0TPAHMlzZY0GVgIPNqldZn1VFcO5yLigKSlwA+AScDKiNjcjXWZ9Vq3zomIiMeAx7r1/sdas/Odds6brPn5TjvnTXXhOxbMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJL6toNqIPGN5t2Tj/fbNqM90RmSQ6RWZJDZJbkc6IJuOlI59Sh6Ug3tb0nkjRL0g8lbZG0WdIXy/itkl6TtLH8zO9cuWb1k9kTHQC+HBHPSfoQ8KykdeW1OyLitnx5ZvXXdogiYi+wt0y/JWkLVdPGozZt9nksWr2+3VLMumLZjBktzdeRCwuSzgQ+CjxdhpZKekHSSklTO7EOs7pKh0jSFGAtsCwi3gTuBM4C5lHtqVZMsNywpA2SNoyOjmbLMOuZVIgkfZAqQPdHxLcBImJfRByMiHeBu6ma2x+msQPq9OnTM2WY9VTm6pyAe4AtEXF7w/hpDbNdDWxqvzyz+stcnbsIuA54UdLGMnYzcK2keVQN7HcBN6QqNKu5zNW5H9P82x8GpuupWSt8249ZkkNkluQQmSXV4gbU11/ZxOpFc3tdhllbvCcyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS0rfxS1pF/AWcBA4EBFDkqYBDwJnUn1E/HMR8evsuszqqFN7oj+LiHkRMVSeLwfWR8RcYH15bjaQunU4twBYVaZXAVd1aT1mPdeJEAXwuKRnJQ2XsVNLm+GxdsOndGA9ZrXUiU+2XhQReySdAqyT9FIrC5XADQNMPcHXN6x/pf96I2JPedwPPEzV8XTfWBPH8ri/yXL/3wF1yuRmnbfM+kO2jfCJ5WtVkHQi8EmqjqePAovLbIuBRzLrMauz7OHcqcDDVUdhjgP+PSK+L+kZ4CFJ1wOvAtck12NWW6kQRcRO4E+ajI8Cl2Xe26xf+IzeLMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILKntT7ZK+ghVl9Mxc4B/BE4C/g74ZRm/OSIea7tCs5prO0QRsRWYByBpEvAaVbefLwB3RMRtHanQrOY6dTh3GbAjIn7Rofcz6xudCtFCYE3D86WSXpC0UtLUDq3DrJbSIZI0GfgM8B9l6E7gLKpDvb3AigmWG5a0QdKG/34nsmWY9Uwn9kRXAs9FxD6AiNgXEQcj4l3gbqqOqIdxB1QbFJ0I0bU0HMqNtQ8urqbqiGo2sFLNGyX9HnA5cEPD8NclzaP6tohd414zGzjZDqhvA9PHjV2Xqsisz/iOBbMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJJSH8ozq4sNX/r0Ic+Hbv/uMVt3S3ui0vpqv6RNDWPTJK2TtK08Ti3jkvQNSdtL26zzu1W8WR20ejh3L3DFuLHlwPqImAusL8+h6v4zt/wMU7XQMhtYLYUoIp4AXh83vABYVaZXAVc1jN8XlaeAk8Z1ADIbKJkLC6dGxF6A8nhKGZ8J7G6Yb6SMHcLNG21QdOPqXLNOjIelxM0bbVBkQrRv7DCtPO4v4yPArIb5Tgf2JNZjVmuZED0KLC7Ti4FHGsY/X67SXQi8MXbYZzaIWvp/IklrgEuBGZJGgK8AXwMeknQ98CpwTZn9MWA+sB14m+r7iswGVkshiohrJ3jpsibzBrAkU5RZP/FtP2ZJDpFZkkNkluQQmSU5RGZJDpFZkj9PZAPhWH5+aDzvicySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzpCOGaILup/8i6aXS4fRhSSeV8TMl/UbSxvLzzW4Wb1YHreyJ7uXw7qfrgPMi4o+Bl4GbGl7bERHzys+NnSnTrL6OGKJm3U8j4vGIOFCePkXVFsvsfakT50R/C3yv4flsST+T9CNJF0+0kDug2qBIfRRC0i3AAeD+MrQXOCMiRiV9DPiOpHMj4s3xy0bEXcBdAGd8+DinyPpW23siSYuBvwT+urTJIiJ+GxGjZfpZYAdwdicKNaurtkIk6QrgH4DPRMTbDeMnS5pUpudQfb3Kzk4UalZXRzycm6D76U3A8cA6SQBPlStxlwD/JOkAcBC4MSLGfyWL2UA5Yogm6H56zwTzrgXWZosy6ye+Y8EsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrOkgfl+oss/8fFDnq/7yZM9qsTeb7wn6rBFq7exaPW2Xpdhx5BDZJbkEJklOURmSe12QL1V0msNnU7nN7x2k6TtkrZK+lS3Cjeri1auzt0L/Ctw37jxOyLitsYBSecAC4FzgT8A/kvS2RFxsAO19oXVi+b2ugQ7xtrqgPoeFgAPlNZZrwDbgQsS9ZnVXuacaGlpaL9S0tQyNhPY3TDPSBk7jDug2qBoN0R3AmcB86i6nq4o42oyb9OERMRdETEUEUNTJjdbzKw/tBWiiNgXEQcj4l3gbn53yDYCzGqY9XRgT65Es3prtwPqaQ1PrwbGrtw9CiyUdLyk2VQdUH+aK9Gs3trtgHqppHlUh2q7gBsAImKzpIeAn1M1ul/yfroyZ+9PHe2AWub/KvDVTFFm/cR3LJglOURmSQ6RWdLAfCjPH8KzXvGeyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwis6R2mzc+2NC4cZekjWX8TEm/aXjtm90s3qwO2mreGBF/NTYtaQXwRsP8OyJiXqcKNKu7Vj4e/oSkM5u9JknA54A/72xZZv0je050MbAvIhq/kGe2pJ9J+pGki5Pvb1Z72Q/lXQusaXi+FzgjIkYlfQz4jqRzI+LN8QtKGgaGAaae4Osb1r/a/uuVdBzwWeDBsbHSg3u0TD8L7ADObra8O6DaoMjsAv4CeCkiRsYGJJ0saVKZnkPVvHFnrkSzemvlEvca4EngI5JGJF1fXlrIoYdyAJcAL0h6HvhP4MaIaPUbJcz6UrvNG4mIv2kythZYmy/LrH/4jN4sySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS6rFd7ZOm30ei1av73UZZodYNmNGS/N5T2SW5BCZJbXy8fBZkn4oaYukzZK+WManSVonaVt5nFrGJekbkrZLekHS+d3eCLNeamVPdAD4ckT8EXAhsETSOcByYH1EzAXWl+cAV1I1KJlL1RLrzo5XbVYjRwxRROyNiOfK9FvAFmAmsABYVWZbBVxVphcA90XlKeAkSad1vHKzmjiqc6LSTvijwNPAqRGxF6qgAaeU2WYCuxsWGyljZgOp5RBJmkLVyWdZs46mjbM2GYsm7zcsaYOkDaOjo62WYVY7LYVI0gepAnR/RHy7DO8bO0wrj/vL+Agwq2Hx04E949+zsQPq9OnT263frOdauTon4B5gS0Tc3vDSo8DiMr0YeKRh/PPlKt2FwBtjh31mg6iVOxYuAq4DXhz7Mi/gZuBrwEOlI+qrwDXltceA+cB24G3gCx2t2KxmWumA+mOan+cAXNZk/gCWJOsy6xu+Y8EsySEyS3KIzJIcIrMkh8gsSdXFtB4XIf0S+B/gV72upYNmMDjbM0jbAq1vzx9GxMlHmqkWIQKQtCEihnpdR6cM0vYM0rZA57fHh3NmSQ6RWVKdQnRXrwvosEHankHaFujw9tTmnMisX9VpT2TWl3oeIklXSNpaGpssP/IS9SNpl6QXJW2UtKGMNW3kUkeSVkraL2lTw1jfNqKZYHtulfRa+R1tlDS/4bWbyvZslfSpo15hRPTsB5gE7ADmAJOB54FzellTm9uxC5gxbuzrwPIyvRz4517X+R71XwKcD2w6Uv1UH3P5HtWd/RcCT/e6/ha351bg75vMe075uzsemF3+Hicdzfp6vSe6ANgeETsj4h3gAapGJ4NgokYutRMRTwCvjxvu20Y0E2zPRBYAD0TEbyPiFarPwV1wNOvrdYgGpalJAI9LelbScBmbqJFLvxjERjRLyyHoyobD6/T29DpELTU16QMXRcT5VD33lki6pNcFdVG//s7uBM4C5gF7gRVlPL09vQ5RS01N6i4i9pTH/cDDVIcDEzVy6RepRjR1ExH7IuJgRLwL3M3vDtnS29PrED0DzJU0W9JkYCFVo5O+IelESR8amwY+CWxi4kYu/WKgGtGMO2+7mup3BNX2LJR0vKTZVJ17f3pUb16DKynzgZeprorc0ut62qh/DtXVneeBzWPbAEynaq+8rTxO63Wt77ENa6gOcf6X6l/m6yeqn+rw59/K7+tFYKjX9be4Pd8q9b5QgnNaw/y3lO3ZClx5tOvzHQtmSb0+nDPrew6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJkl/R8n1EHbB+lJ2gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1815607eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# i take some steps to see middle game scene\n",
    "# in pong game,\n",
    "# 0 is numeric action to stay at same place \n",
    "# 2 is numeric action to move paddle up in game\n",
    "# 3 is numeric action to move paddle down in game\n",
    "for i in range(30):\n",
    "    observation, reward, done, info = env.step(0)# 0 means stay the same place(or do nothing)\n",
    "    \n",
    "plt.imshow(observation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in the above image, we control the green paddle which is on the right side"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "when the ball passes our paddle and goes to end right, we get reward of -1 for losing.\n",
    "and if the ball crosses the opponent and reaches the left, we get a reward of +1\n",
    "game finishes if one of the players reach 21 score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so the definition of system in reinforcement learning method is simple: \n",
    "    \n",
    "- state is the screen of game.\n",
    "- action is going up or down\n",
    "\n",
    "we could also define 3 actions (go up, go down, stay still) but here we just use the 2 mentioned above\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since we want to use a neural network as policy, we need to convey enough information so it can learn where the ball is moving. for example in the above image, it is not clear wheter the ball is moving left or right.\n",
    "\n",
    "to convey this information we can define the state as the last 10 frames of game. and then feed it to a big giant neural net. or maybe feed the frams to a RNN one by one so it learns the sequence of game. \n",
    "\n",
    "but **for simplicity we use another trick: we just subtract two consecutive frames.** and feed the resulting image to the network as input. this is not a generalizable solution but makes training faster and life easier for this project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x18157466d8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADnxJREFUeJzt3X2MXPV1xvHvExODitPgF0DUmGIjExVQ65AVJUEgWkoCThVDJFKjmrgp6oJkS7GSSjUgNahSpDTFIEWtiEBYmJgaaB0CUkiCa0VBUYBgiAE7xvgFBy+27GSJgJYo1Ob0j/vbZryexeM5M547w/ORVnPnN/fOPVe7j++L75xRRGBm7ftArwsw63cOkVmSQ2SW5BCZJTlEZkkOkVlS10Ik6QpJWyVtl7S8W+sx6zV14/+JJE0CXgYuB0aAZ4BrI+LnHV+ZWY91a090AbA9InZGxDvAA8CCLq3LrKeO69L7zgR2NzwfAf50opklvefucNbvT+pQWWat2/3mwV9FxMlHmq9bIVKTsUOCImkYGAaYesIH+MqlH+5SKe25/BMfP+pl1v3kyS5U0v82fOnTR73M0O3f7UIlR2fZ93/9i1bm69bh3Agwq+H56cCexhki4q6IGIqIoSmTm2XOrD90K0TPAHMlzZY0GVgIPNqldZn1VFcO5yLigKSlwA+AScDKiNjcjXWZ9Vq3zomIiMeAx7r1/sdas/Odds6brPn5TjvnTXXhOxbMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJL6toNqIPGN5t2Tj/fbNqM90RmSQ6RWZJDZJbkc6IJuOlI59Sh6Ug3tb0nkjRL0g8lbZG0WdIXy/itkl6TtLH8zO9cuWb1k9kTHQC+HBHPSfoQ8KykdeW1OyLitnx5ZvXXdogiYi+wt0y/JWkLVdPGozZt9nksWr2+3VLMumLZjBktzdeRCwuSzgQ+CjxdhpZKekHSSklTO7EOs7pKh0jSFGAtsCwi3gTuBM4C5lHtqVZMsNywpA2SNoyOjmbLMOuZVIgkfZAqQPdHxLcBImJfRByMiHeBu6ma2x+msQPq9OnTM2WY9VTm6pyAe4AtEXF7w/hpDbNdDWxqvzyz+stcnbsIuA54UdLGMnYzcK2keVQN7HcBN6QqNKu5zNW5H9P82x8GpuupWSt8249ZkkNkluQQmSXV4gbU11/ZxOpFc3tdhllbvCcyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS0rfxS1pF/AWcBA4EBFDkqYBDwJnUn1E/HMR8evsuszqqFN7oj+LiHkRMVSeLwfWR8RcYH15bjaQunU4twBYVaZXAVd1aT1mPdeJEAXwuKRnJQ2XsVNLm+GxdsOndGA9ZrXUiU+2XhQReySdAqyT9FIrC5XADQNMPcHXN6x/pf96I2JPedwPPEzV8XTfWBPH8ri/yXL/3wF1yuRmnbfM+kO2jfCJ5WtVkHQi8EmqjqePAovLbIuBRzLrMauz7OHcqcDDVUdhjgP+PSK+L+kZ4CFJ1wOvAtck12NWW6kQRcRO4E+ajI8Cl2Xe26xf+IzeLMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILKntT7ZK+ghVl9Mxc4B/BE4C/g74ZRm/OSIea7tCs5prO0QRsRWYByBpEvAaVbefLwB3RMRtHanQrOY6dTh3GbAjIn7Rofcz6xudCtFCYE3D86WSXpC0UtLUDq3DrJbSIZI0GfgM8B9l6E7gLKpDvb3AigmWG5a0QdKG/34nsmWY9Uwn9kRXAs9FxD6AiNgXEQcj4l3gbqqOqIdxB1QbFJ0I0bU0HMqNtQ8urqbqiGo2sFLNGyX9HnA5cEPD8NclzaP6tohd414zGzjZDqhvA9PHjV2Xqsisz/iOBbMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJJSH8ozq4sNX/r0Ic+Hbv/uMVt3S3ui0vpqv6RNDWPTJK2TtK08Ti3jkvQNSdtL26zzu1W8WR20ejh3L3DFuLHlwPqImAusL8+h6v4zt/wMU7XQMhtYLYUoIp4AXh83vABYVaZXAVc1jN8XlaeAk8Z1ADIbKJkLC6dGxF6A8nhKGZ8J7G6Yb6SMHcLNG21QdOPqXLNOjIelxM0bbVBkQrRv7DCtPO4v4yPArIb5Tgf2JNZjVmuZED0KLC7Ti4FHGsY/X67SXQi8MXbYZzaIWvp/IklrgEuBGZJGgK8AXwMeknQ98CpwTZn9MWA+sB14m+r7iswGVkshiohrJ3jpsibzBrAkU5RZP/FtP2ZJDpFZkkNkluQQmSU5RGZJDpFZkj9PZAPhWH5+aDzvicySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzpCOGaILup/8i6aXS4fRhSSeV8TMl/UbSxvLzzW4Wb1YHreyJ7uXw7qfrgPMi4o+Bl4GbGl7bERHzys+NnSnTrL6OGKJm3U8j4vGIOFCePkXVFsvsfakT50R/C3yv4flsST+T9CNJF0+0kDug2qBIfRRC0i3AAeD+MrQXOCMiRiV9DPiOpHMj4s3xy0bEXcBdAGd8+DinyPpW23siSYuBvwT+urTJIiJ+GxGjZfpZYAdwdicKNaurtkIk6QrgH4DPRMTbDeMnS5pUpudQfb3Kzk4UalZXRzycm6D76U3A8cA6SQBPlStxlwD/JOkAcBC4MSLGfyWL2UA5Yogm6H56zwTzrgXWZosy6ye+Y8EsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrOkgfl+oss/8fFDnq/7yZM9qsTeb7wn6rBFq7exaPW2Xpdhx5BDZJbkEJklOURmSe12QL1V0msNnU7nN7x2k6TtkrZK+lS3Cjeri1auzt0L/Ctw37jxOyLitsYBSecAC4FzgT8A/kvS2RFxsAO19oXVi+b2ugQ7xtrqgPoeFgAPlNZZrwDbgQsS9ZnVXuacaGlpaL9S0tQyNhPY3TDPSBk7jDug2qBoN0R3AmcB86i6nq4o42oyb9OERMRdETEUEUNTJjdbzKw/tBWiiNgXEQcj4l3gbn53yDYCzGqY9XRgT65Es3prtwPqaQ1PrwbGrtw9CiyUdLyk2VQdUH+aK9Gs3trtgHqppHlUh2q7gBsAImKzpIeAn1M1ul/yfroyZ+9PHe2AWub/KvDVTFFm/cR3LJglOURmSQ6RWdLAfCjPH8KzXvGeyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwis6R2mzc+2NC4cZekjWX8TEm/aXjtm90s3qwO2mreGBF/NTYtaQXwRsP8OyJiXqcKNKu7Vj4e/oSkM5u9JknA54A/72xZZv0je050MbAvIhq/kGe2pJ9J+pGki5Pvb1Z72Q/lXQusaXi+FzgjIkYlfQz4jqRzI+LN8QtKGgaGAaae4Osb1r/a/uuVdBzwWeDBsbHSg3u0TD8L7ADObra8O6DaoMjsAv4CeCkiRsYGJJ0saVKZnkPVvHFnrkSzemvlEvca4EngI5JGJF1fXlrIoYdyAJcAL0h6HvhP4MaIaPUbJcz6UrvNG4mIv2kythZYmy/LrH/4jN4sySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS6rFd7ZOm30ei1av73UZZodYNmNGS/N5T2SW5BCZJbXy8fBZkn4oaYukzZK+WManSVonaVt5nFrGJekbkrZLekHS+d3eCLNeamVPdAD4ckT8EXAhsETSOcByYH1EzAXWl+cAV1I1KJlL1RLrzo5XbVYjRwxRROyNiOfK9FvAFmAmsABYVWZbBVxVphcA90XlKeAkSad1vHKzmjiqc6LSTvijwNPAqRGxF6qgAaeU2WYCuxsWGyljZgOp5RBJmkLVyWdZs46mjbM2GYsm7zcsaYOkDaOjo62WYVY7LYVI0gepAnR/RHy7DO8bO0wrj/vL+Agwq2Hx04E949+zsQPq9OnT263frOdauTon4B5gS0Tc3vDSo8DiMr0YeKRh/PPlKt2FwBtjh31mg6iVOxYuAq4DXhz7Mi/gZuBrwEOlI+qrwDXltceA+cB24G3gCx2t2KxmWumA+mOan+cAXNZk/gCWJOsy6xu+Y8EsySEyS3KIzJIcIrMkh8gsSdXFtB4XIf0S+B/gV72upYNmMDjbM0jbAq1vzx9GxMlHmqkWIQKQtCEihnpdR6cM0vYM0rZA57fHh3NmSQ6RWVKdQnRXrwvosEHankHaFujw9tTmnMisX9VpT2TWl3oeIklXSNpaGpssP/IS9SNpl6QXJW2UtKGMNW3kUkeSVkraL2lTw1jfNqKZYHtulfRa+R1tlDS/4bWbyvZslfSpo15hRPTsB5gE7ADmAJOB54FzellTm9uxC5gxbuzrwPIyvRz4517X+R71XwKcD2w6Uv1UH3P5HtWd/RcCT/e6/ha351bg75vMe075uzsemF3+Hicdzfp6vSe6ANgeETsj4h3gAapGJ4NgokYutRMRTwCvjxvu20Y0E2zPRBYAD0TEbyPiFarPwV1wNOvrdYgGpalJAI9LelbScBmbqJFLvxjERjRLyyHoyobD6/T29DpELTU16QMXRcT5VD33lki6pNcFdVG//s7uBM4C5gF7gRVlPL09vQ5RS01N6i4i9pTH/cDDVIcDEzVy6RepRjR1ExH7IuJgRLwL3M3vDtnS29PrED0DzJU0W9JkYCFVo5O+IelESR8amwY+CWxi4kYu/WKgGtGMO2+7mup3BNX2LJR0vKTZVJ17f3pUb16DKynzgZeprorc0ut62qh/DtXVneeBzWPbAEynaq+8rTxO63Wt77ENa6gOcf6X6l/m6yeqn+rw59/K7+tFYKjX9be4Pd8q9b5QgnNaw/y3lO3ZClx5tOvzHQtmSb0+nDPrew6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJkl/R8n1EHbB+lJ2gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x18156e9d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "new_observation, reward, done, info = env.step(2)\n",
    "plt.imshow(observation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_frames(new_frame,last_frame):\n",
    "    # inputs are 2 numpy 2d arrays\n",
    "    n_frame = new_frame.astype(np.int32)\n",
    "    n_frame[(n_frame==144)|(n_frame==109)]=0 # remove backgound colors\n",
    "    l_frame = last_frame.astype(np.int32)\n",
    "    l_frame[(l_frame==144)|(l_frame==109)]=0 # remove backgound colors\n",
    "    diff = n_frame - l_frame\n",
    "    # crop top and bot \n",
    "    diff = diff[35:195]\n",
    "    # down sample \n",
    "    diff=diff[::2,::2]\n",
    "    # convert to grayscale\n",
    "    diff = diff[:,:,0] * 299. / 1000 + diff[:,:,1] * 587. / 1000 + diff[:,:,2] * 114. / 1000\n",
    "    # rescale numbers between 0 and 1\n",
    "    max_val =diff.max() if diff.max()> abs(diff.min()) else abs(diff.min())\n",
    "    if max_val != 0:\n",
    "        diff=diff/max_val\n",
    "    return diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x18157ac518>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADDJJREFUeJzt3V+oZeV9xvHv0xmHdJLqURtldGxVEKM3jumQKpbSOj1iUtFexKKkRYKgF2lRmpJqblRowdwk5qIERU29sFFrlIgE42FiaAtlqkbbREersVYPGscmDqY5kDLJrxd72Rz0jLPOOfvPrPN+P3DYe717bda7WDz7fdeaNeuXqkJSW35l1h2QNH0GX2qQwZcaZPClBhl8qUEGX2qQwZcatK7gJ7kwyfNJXkxy3bg6JWmystYbeJJsAv4DmAcWgceBy6vq2fF1T9IkbF7Hdz8GvFhVLwEkuQe4BDho8Ldu3Vpzc3Pr2KSk97N//36WlpZyqPXWE/wTgVeXLS8Cv/1+X5ibm+Pqq69exyYlvZ9bb72113rrOcdf6VflPecNSa5K8kSSJ5aWltaxOUnjsp7gLwInLVveDrz27pWq6raq2llVO7du3bqOzUkal/UE/3HgtCSnJNkCXAY8NJ5uSZqkNZ/jV9WBJH8GfAvYBNxZVc+MrWeSJmY9F/eoqm8C3xxTXyRNiXfuSQ0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDL7UIIMvNcjgSw0y+FKDDhn8JHcm2Zfk+8vajkmykOSF7vXoyXZT0jj1GfH/DrjwXW3XAbur6jRgd7csaSAOGfyq+kfgx+9qvgS4q3t/F/BHY+6XpAla6zn+8VX1OkD3etz4uiRp0iZ+cc8SWtLhZ63BfyPJNoDudd/BVrSElnT4WWvwHwKu6N5fAXxjPN2RNA19/jnva8C/AKcnWUxyJXAzMJ/kBWC+W5Y0EIcsoVVVlx/ko11j7oukKfHOPalBBl9qkMGXGmTwpQYZfKlBBl9qkMGXGmTwpQYZfKlBBl9qkMGXGmTwpQYZfKlBBl9qkMGXGmTwpQYZfKlBBl9qUJ9n7p2U5LEke5M8k+Sart0yWtJA9RnxDwCfraozgHOAzyQ5E8toSYPVp4TW61X13e79T4C9wIlYRksarFWd4yc5GTgb2INltKTB6h38JB8Cvg5cW1Vvr+J7ltCSDjO9gp/kCEahv7uqHuiae5XRsoSWdPjpc1U/wB3A3qr64rKPLKMlDdQhK+kA5wF/CnwvydNd2+cZlc26ryup9Qpw6WS6KGnc+pTQ+mcgB/nYMlrSAHnnntQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoMMvtQggy81qM8z9ySt065d731K3e7du2fQk5E+T9n9QJJ/TfJvXe28m7r2U5Ls6Wrn3Ztky+S7K2kc+kz1fwacX1VnATuAC5OcA3wB+FJXO+8t4MrJdVPSOPWpnVdV9T/d4hHdXwHnA/d37dbOkwakbyWdTd0z9fcBC8APgP1VdaBbZZFRIc2VvmsJLekw0+viXlX9HNiRZA54EDhjpdUO8t3bgNsATjjhhBXXkTa6WV7IW8mq/jmvqvYD3wHOAeaSvPPDsR14bbxdkzQpfa7qf7gb6Unyq8AfAHuBx4BPdqtZO08akD5T/W3AXUk2MfqhuK+qHk7yLHBPkr8GnmJUWFPSAPSpnffvwNkrtL8EfGwSnZI0Wd6yKzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCfsitNwcLCwnva5ufnZ9CTEUd8qUEGX2qQwZcaZPClBnlxT5qCWV7IW0nvEb97tv5TSR7uli2hJQ3Uaqb61zB6uu47LKElDVTfSjrbgT8Ebu+WgyW0pMHqO+LfAnwO+EW3fCyW0JIGq09BjYuAfVX15PLmFVY9aAmtqtpZVTu3bt26xm5KGqc+V/XPAy5O8gngA8CRjGYAc0k2d6O+JbSkAelTJvv6qtpeVScDlwHfrqpPYQktabDWcwPPXwF/keRFRuf8ltCSBmJVN/BU1XcYVcu1hJY0YN6yKzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsNMvhSgwy+1CCDLzXI4EsN6vXorSQvAz8Bfg4cqKqdSY4B7gVOBl4G/riq3ppMNyWN02pG/N+vqh1VtbNbvg7Y3ZXQ2t0tSxqA9Uz1L2FUOgssoSUNSt/gF/BokieTXNW1HV9VrwN0r8dNooOSxq/v47XPq6rXkhwHLCR5ru8Guh+KqwCOOuqoNXRR0rj1GvGr6rXudR/wIKPn6b+RZBtA97rvIN+1dp50mOlTNPODSX7tnffABcD3gYcYlc4CS2hJg9Jnqn888GCSd9b/+6p6JMnjwH1JrgReAS6dXDcljdMhg9+VyjprhfYfAbsm0SlJk+Wde1KDDL7UIIMvNcjgSw3qewPPYe3RRx99T9sFF1wwg55Iw+CILzXI4EsN2hBT/aFO62+88cYV30uT5ogvNcjgSw3aEFP9Ibnhhhv+/73Te82KI77UIEf8GXLE16w44ksNMvhSg5zqT9lNN9006y5IjvhSiwy+1KBewU8yl+T+JM8l2Zvk3CTHJFlI8kL3evSkOytpPPqO+F8GHqmqjzB6/t5eLKElDVafx2sfCfwucAdAVf1vVe3HElrSYPUZ8U8F3gS+muSpJLd3z9e3hJY0UH2Cvxn4KPCVqjob+CmrmNYnuSrJE0meWFpaWmM3JY1Tn+AvAotVtadbvp/RD4EltKSBOmTwq+qHwKtJTu+adgHPYgktabD63rn358DdSbYALwGfZvSjYQktaYB6Bb+qngZ2rvDRYVFCa35+/j1tCwsLM+iJNAzeuSc1yOBLDdoQ/zvPab20Oo74UoMMvtQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoMMvtQggy81yOBLDTL4UoMMvtQggy81qE9BjdOTPL3s7+0k11pCSxquPk/Zfb6qdlTVDuC3gCXgQSyhJQ3Waqf6u4AfVNV/YQktabBWG/zLgK917y2hJQ1U7+B3z9S/GPiH1WzAElrS4Wc1I/7Hge9W1RvdsiW0pIFaTfAv55fTfLCEljRYvYKfZCswDzywrPlmYD7JC91nN4+/e5ImoW8JrSXg2He1/YjDpISWpNXxzj2pQQZfapDBlxpk8KUGGXypQQZfapDBlxpk8KUGGXypQQZfapDBlxpk8KUGGXypQQZfapDBlxpk8KUGGXypQQZfapDBlxpk8KUGGXypQamq6W0seRP4KfDfU9vodP06G3Pf3K/h+M2q+vChVppq8AGSPFFVO6e60SnZqPvmfm08TvWlBhl8qUGzCP5tM9jmtGzUfXO/Npipn+NLmj2n+lKDphr8JBcmeT7Ji0mum+a2xynJSUkeS7I3yTNJrunaj0mykOSF7vXoWfd1LZJsSvJUkoe75VOS7On2694kW2bdx7VIMpfk/iTPdcfu3I1yzFZrasFPsgn4W+DjwJnA5UnOnNb2x+wA8NmqOgM4B/hMty/XAbur6jRgd7c8RNcAe5ctfwH4UrdfbwFXzqRX6/dl4JGq+ghwFqN93CjHbHWqaip/wLnAt5YtXw9cP63tT3jfvgHMA88D27q2bcDzs+7bGvZlO6MAnA88DITRTS6bVzqOQ/kDjgT+k+661rL2wR+ztfxNc6p/IvDqsuXFrm3QkpwMnA3sAY6vqtcButfjZtezNbsF+Bzwi275WGB/VR3olod63E4F3gS+2p3G3J7kg2yMY7Zq0wx+Vmgb9D8pJPkQ8HXg2qp6e9b9Wa8kFwH7qurJ5c0rrDrE47YZ+Cjwlao6m9Gt421M61cwzeAvAictW94OvDbF7Y9VkiMYhf7uqnqga34jybbu823Avln1b43OAy5O8jJwD6Pp/i3AXJLN3TpDPW6LwGJV7emW72f0QzD0Y7Ym0wz+48Bp3RXiLcBlwENT3P7YJAlwB7C3qr647KOHgCu691cwOvcfjKq6vqq2V9XJjI7Pt6vqU8BjwCe71Qa3XwBV9UPg1SSnd027gGcZ+DFbq2n/77xPMBpBNgF3VtXfTG3jY5Tkd4B/Ar7HL8+FP8/oPP8+4DeAV4BLq+rHM+nkOiX5PeAvq+qiJKcymgEcAzwF/ElV/WyW/VuLJDuA24EtwEvApxkNfhvimK2Gd+5JDfLOPalBBl9qkMGXGmTwpQYZfKlBBl9qkMGXGmTwpQb9HxHeFTtXqNO6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1815721208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(preprocess_frames(new_observation,observation),plt.cm.gray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note that we cropped bottom and top of game screen because it did not provide information.\n",
    "also we remove the backgounds and the pixel values are rescaled between -1 and +1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocess_frames(new_observation,observation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(80, 80)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocess_frames(new_observation,observation).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so our actual **state is a 80*80 image** derived by subtracting two consecutive frames, where as a result most values are 0 but where paddles or ball have moved, it has non-zero values "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "next we define a policy which sees the state and chooses an action in keras.\n",
    "\n",
    "we define our solution approach and network model so that network outputs the probablity of choosing action of moving up. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Modeling the Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 80, 80)            0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 6400)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 200)               1280000   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 200       \n",
      "=================================================================\n",
      "Total params: 1,280,200\n",
      "Trainable params: 1,280,200\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# simple 2 layer model \n",
    "# with 200 hidden units in first layer\n",
    "# and 1 sigmoid output\n",
    "inputs = keras.layers.Input(shape=(80,80))\n",
    "flattened_layer = keras.layers.Flatten()(inputs)\n",
    "full_connect_1 = keras.layers.Dense(units=200,activation='relu',use_bias=False,)(flattened_layer)\n",
    "sigmoid_output = keras.layers.Dense(1,activation='sigmoid',use_bias=False)(full_connect_1)\n",
    "policy_network_model = keras.models.Model(inputs=inputs,outputs=sigmoid_output)\n",
    "policy_network_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the above model has **about 1.28 million parameters **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you can create any model as you like. based on architecture and complexity of your model, it may train slow or fast, or even not train at all(too hard to train).\n",
    "\n",
    "here is another model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 80, 80)            0         \n",
      "_________________________________________________________________\n",
      "reshape_1 (Reshape)          (None, 80, 80, 1)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 16, 16, 10)        4000      \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 4, 4, 20)          20000     \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 2, 2, 40)          7200      \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 160)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 160       \n",
      "=================================================================\n",
      "Total params: 31,360\n",
      "Trainable params: 31,360\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# another model\n",
    "# with convolutional layers \n",
    "# it has about 31000 parameters\n",
    "\n",
    "# inputs = keras.layers.Input(shape=(80,80))\n",
    "# channeled_input = keras.layers.Reshape((80,80,1))(inputs) # Conv2D requries (batch, height, width, channels)  so we need to create a dummy channel \n",
    "# conv_1 = keras.layers.Conv2D(filters=10,kernel_size=20,padding='valid',activation='relu',strides=(4,4),use_bias=False)(channeled_input)\n",
    "# conv_2 = keras.layers.Conv2D(filters=20,kernel_size=10,padding='valid',activation='relu',strides=(2,2),use_bias=False)(conv_1)\n",
    "# conv_3 = keras.layers.Conv2D(filters=40,kernel_size=3,padding='valid',activation='relu',use_bias=False)(conv_2)\n",
    "# flattened_layer = keras.layers.Flatten()(conv_3)\n",
    "# sigmoid_output = keras.layers.Dense(1,activation='sigmoid',use_bias=False)(flattened_layer)\n",
    "# policy_network_model = keras.models.Model(inputs=inputs,outputs=sigmoid_output)\n",
    "# policy_network_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so we define a network with an architecture that inputs a 80*80 image and outputs the probability of going up.\n",
    "\n",
    "in the learning phase the policy can output the probability of going up and we sample from that probability.\n",
    "in the playing phase we may opt to pick the most probable action. or just sample like the training phase"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Defining loss "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we write the REINFORCE aglorithm this way:\n",
    "\n",
    "$\\theta = \\theta-\\alpha G\\nabla_{\\theta}(-log(\\pi))$ \n",
    "\n",
    "and see it as a gradient descent, then the loss = $G\\nabla_{\\theta}(-log(\\pi))$\n",
    "\n",
    "G is the reward from the state we are updating, $\\pi$ is the probablity of taking the action that we took when playing. for example when playing the game, probabilty of taking action up was 0.8 according to policy output. but we took down action(because we sample action from probabilty,this is possible) then the loss is G*-log(0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "episode_reward = keras.layers.Input(shape=(1,),name='episode_reward')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def m_loss(episode_reward):\n",
    "    def loss(y_true,y_pred):\n",
    "        # feed in y_true as actual action taken \n",
    "        # if actual action was up, we feed 1 as y_true and otherwise 0\n",
    "        # y_pred is the network output(probablity of taking up action)\n",
    "        # note that we dont feed y_pred to network. keras computes it\n",
    "        \n",
    "        # first we clip y_pred between some values because log(0) and log(1) are undefined\n",
    "        tmp_pred = keras.layers.Lambda(lambda x: keras.backend.clip(x,0.05,0.95))(y_pred)\n",
    "        # we calculate log of probablity. y_pred is the probablity of taking up action\n",
    "        # note that y_true is 1 when we actually chose up, and 0 when we chose down\n",
    "        # this is probably similar to cross enthropy formula in keras, but here we write it manually to multiply it by the reward value\n",
    "        tmp_loss = keras.layers.Lambda(lambda x:-y_true*keras.backend.log(x)-(1-y_true)*(keras.backend.log(1-x)))(tmp_pred)\n",
    "        # multiply log of policy by reward\n",
    "        policy_loss=keras.layers.Multiply()([tmp_loss,episode_reward])\n",
    "        return policy_loss\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "keras loss function has format of `def loss(y_true,y_pred):...`. since we needed to include reward in loss, i created m_loss function above as a tool to input episode_reward as input.\n",
    "\n",
    "next we create the optimizer and network for training, and:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "episode_reward = keras.layers.Input(shape=(1,),name='episode_reward')\n",
    "policy_network_train = keras.models.Model(inputs=[inputs,episode_reward],outputs=sigmoid_output)\n",
    "\n",
    "my_optimizer = keras.optimizers.RMSprop(lr=0.0001)\n",
    "policy_network_train.compile(optimizer=my_optimizer,loss=m_loss(episode_reward),)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**pay attention** to the inputs of policy_network_train defined above. we input both rewards and the inputs(processed 80*80 image).\n",
    "the output of the network is the same.\n",
    "\n",
    "since both `policy_network_train` and  `policy_network_model`(defined previous section) use same layers (from inputs to outputs), **they share their weights and parameters**. \n",
    "so we just train using `policy_network_train` and then use `policy_network_model` when playing and simlating."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Reward Engineering and why it is important"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's play with the defined policy and see what is the reward.\n",
    "we define a function that takes a policy, and plays according to the policy and keeps the rewards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_episode(policy_network):\n",
    "    states_list = [] # shape = (x,80,80)\n",
    "    up_or_down_action_list=[] # 1 if we chose up. 0 if down\n",
    "    rewards_list=[]\n",
    "    network_output_list=[]\n",
    "    env=gym.make(\"Pong-v0\")\n",
    "    observation = env.reset()\n",
    "    new_observation = observation\n",
    "    done = False\n",
    "    policy_output_list = []\n",
    "    \n",
    "    while done == False:\n",
    "    \n",
    "        processed_network_input = preprocess_frames(new_frame=new_observation,last_frame=observation)\n",
    "        states_list.append(processed_network_input)\n",
    "        reshaped_input = np.expand_dims(processed_network_input,axis=0) # x shape is (80,80) so we need similar reshape(x,(1,80,80))\n",
    "\n",
    "        up_probability = policy_network.predict(reshaped_input,batch_size=1)[0][0]\n",
    "        network_output_list.append(up_probability)\n",
    "        policy_output_list.append(up_probability)\n",
    "        actual_action = np.random.choice(a=[2,3],size=1,p=[up_probability,1-up_probability]) # 2 is up. 3 is down \n",
    "        if actual_action==2:\n",
    "            up_or_down_action_list.append(1)\n",
    "        else:\n",
    "            up_or_down_action_list.append(0)\n",
    "        \n",
    "        observation= new_observation\n",
    "        new_observation, reward, done, info = env.step(actual_action)\n",
    "        \n",
    "        rewards_list.append(reward)\n",
    "        \n",
    "        if done:\n",
    "            break\n",
    "            \n",
    "    env.close()\n",
    "    return states_list,up_or_down_action_list,rewards_list,network_output_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the function is plain and simple, nothing to discuss about it.\n",
    "so lets play 1 game and see what happens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "states_list,up_or_down_action_list,rewards_list,network_output_list = generate_episode(policy_network_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of states= 1462\n",
      "shape of each state=(80, 80)\n",
      "length of rewards= 1462\n"
     ]
    }
   ],
   "source": [
    "print(\"length of states= \"+str(len(states_list)))# this is the number of frames\n",
    "print(\"shape of each state=\"+str(states_list[0].shape))\n",
    "print(\"length of rewards= \"+str(len(rewards_list)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.49087763, 0.5173061, 0.47197872, 0.5205401, 0.5208977, 0.4981439, 0.5146148, 0.5047591, 0.50674325, 0.4966033, 0.5057894, 0.47851637, 0.4832886, 0.48846787, 0.5020138, 0.518649, 0.49806774, 0.4940003, 0.49973598, 0.4741504]\n"
     ]
    }
   ],
   "source": [
    "# lets see sample of policy output\n",
    "print(network_output_list[30:50]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since the network is not trained, its output is about 50% all time. meaning . that it does not know which action is better now and outputs a probablity of about 0.5 for all states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#lets see a sample what we actually did: 1 means we went up, 0 means down\n",
    "up_or_down_action_list[30:50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n"
     ]
    }
   ],
   "source": [
    "# lets see sample of rewards\n",
    "print(rewards_list[50:100]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the rewards are mostly zero and sometimes we may see 1 or -1 in list. this is because environment just gives us reward when ball passes through us or through opponent. \n",
    "\n",
    "when ball passes our paddle, we lose and get -1 reward and when the opponent fails to catch the ball, we win and get +1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count win=1\n",
      "count lose=21\n",
      "count zero rewards=1440\n"
     ]
    }
   ],
   "source": [
    "# lets see how many times we won through whole game:\n",
    "print(\"count win=\"+str(len(list(filter(lambda r: r>0,rewards_list)))))\n",
    "print(\"count lose=\"+str(len(list(filter(lambda r: r<0,rewards_list)))))\n",
    "print(\"count zero rewards=\"+str(len(list(filter(lambda r: r==0,rewards_list)))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so we got a lot of zeros. some loses. and maybe some wins if we are lucky and our dumb network did some lucky actions in some situation. lets plot the reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGdZJREFUeJzt3X+QHOV95/H3BwnJF+REEsJrgVRIIqqU8flOsBsM8ZVLAgkE5UL4DhJUjiPnrFIlOXKX+JKzVFThHLHvRJIKrlxIgBCMkhCWgxijU8mlgNg915UtolUsQECEFlkYRWCwtXBZ4+gH+t4f86zpZ7WzOzvdO7tz+ryquqb76eeZ/qjnx3enp0etiMDMzGzIWZMdwMzMphYXBjMzy7gwmJlZxoXBzMwyLgxmZpZxYTAzs4wLg5mZZVwYzMws48JgZmaZ6ZMdoBnz5s2LRYsWNTX2hz/8Ieecc061gSZAO+Rsh4zgnFVqh4zgnPXs2bPn+xFx3pgdI6Ltps7OzmhWT09P02NbqR1ytkPGCOesUjtkjHDOeoC+aOA91oeSzMws48JgZmYZFwYzM8u4MJiZWcaFwczMMpUUBkn3S3pD0r466yXpjyT1S3pW0qWFdeskHUjTuirymJlZ86r6xPAAsHqU9dcCS9O0AfhTAElzgS8AHwUuA74gaU5FmawN7HllgLt6+tnzysBkRzGzpJIfuEXENyQtGqXLGuAv0nm0uyTNljQfWA48ERFHASQ9Qa3APFRFLpva9rwywKfu28Xxk6eYMf0sHlx/OZ0X+u8Cs8nWql8+XwC8Wlg+nNrqtZ9G0gZqnzbo6Oigt7e3qSCDg4NNj22ldshZNuO2l49z7MQpAjh+4hQPPbmbf7poRmX5hrTDvoT2yNkOGcE5y2pVYdAIbTFK++mNEfcC9wJ0dXXF8uXLmwrS29tLs2NbqR1yls34/sUDbDu0ixMnT3H29LNYu/JnJ+QTQzvsS2iPnO2QEZyzrFYVhsPAwsLyAuBIal8+rL23RZlsknVeOIcH11/OroM/4PIl5/owktkU0arCsBW4RVI3tS+a346I1yTtAP5b4Qvnq4FNLcpkU0DnhXNcEMymmEoKg6SHqP3lP0/SYWpnGp0NEBF3A9uB64B+4B3gl9O6o5J+F9id7ur2oS+izcxsclR1VtLaMdYH8B/qrLsfuL+KHGZmVp5/+WxmZhkXBjMzy7gwmJlZxoXBzMwyLgxmZpZxYTAzs4wLg5mZZVwYzMws48JgZmYZFwYzM8u4MJiZWcaFwczMMi4MZmaWcWEwM7OMC4OZmWUqKQySVkvaL6lf0sYR1t8paW+aXpL0VmHdu4V1W6vIY2ZmzSt9oR5J04C7gFXUruG8W9LWiHhhqE9E/Gah/68DlxTu4kcRsaxsDjMzq0YVnxguA/oj4mBEHAe6gTWj9F8LPFTBds3MbAJUURguAF4tLB9ObaeRdCGwGHiq0Pw+SX2Sdkm6oYI8ZmZWgmqXYy5xB9JNwDURsT4tfxq4LCJ+fYS+nwcWFNdJOj8ijkhaQq1gXBURL48wdgOwAaCjo6Ozu7u7qbyDg4PMmjWrqbGt1A452yEjOGeV2iEjOGc9K1as2BMRXWN2jIhSE3AFsKOwvAnYVKfvt4GfG+W+HgBuHGubnZ2d0ayenp6mx7ZSO+Rsh4wRzlmldsgY4Zz1AH3RwPt6FYeSdgNLJS2WNAO4GTjt7CJJPwPMAb5VaJsjaWaanwd8DHhh+FgzM2ud0mclRcRJSbcAO4BpwP0R8byk26lVp6EisRboTlVryIeAeySdovZ9x+YonM1kZmatV7owAETEdmD7sLbbhi3/zgjjvgl8pIoMZmZWDf/y2czMMi4MZmaWcWEwM7OMC4OZmWVcGMzMLOPCYGZmGRcGMzPLuDCYmVnGhcHMzDIuDGZmlnFhMDOzjAuDmZllXBjMzCzjwmBmZhkXBjMzy7gwmJlZppLCIGm1pP2S+iVtHGH9ZyS9KWlvmtYX1q2TdCBN66rIY2ZmzSt9BTdJ04C7gFXAYWC3pK0jXKLz4Yi4ZdjYucAXgC4ggD1p7EDZXGZm1pwqPjFcBvRHxMGIOA50A2saHHsN8EREHE3F4AlgdQWZzMysSVUUhguAVwvLh1PbcP9O0rOSHpW0cJxjzcysRUofSgI0QlsMW/5fwEMRcUzSrwBbgCsbHFvbiLQB2ADQ0dFBb29vU2EHBwebHttK7ZCzHTKCc1apHTKCc5YWEaUm4ApgR2F5E7BplP7TgLfT/FrgnsK6e4C1Y22zs7MzmtXT09P02FZqh5ztkDHCOavUDhkjnLMeoC8aeF+v4lDSbmCppMWSZgA3A1uLHSTNLyxeD7yY5ncAV0uaI2kOcHVqMzOzSVL6UFJEnJR0C7U39GnA/RHxvKTbqVWnrcB/lHQ9cBI4CnwmjT0q6XepFReA2yPiaNlMZmbWvCq+YyAitgPbh7XdVpjfRO0Q00hj7wfuryKHmZmV518+m5lZxoXBzMwyLgxmZpZxYTAzs4wLg5mZZVwYzMws48JgZmYZFwYzM8u4MJiZWcaFwczMMi4MZmaWcWEwM7OMC4OZmWVcGMzMLOPCYGZmGRcGMzPLVFIYJK2WtF9Sv6SNI6z/nKQXJD0raaekCwvr3pW0N01bh481M7PWKn0FN0nTgLuAVcBhYLekrRHxQqHbt4GuiHhH0q8Cvwf8Qlr3o4hYVjaHmZlVo4pPDJcB/RFxMCKOA93AmmKHiOiJiHfS4i5gQQXbNTOzCaCIKHcH0o3A6ohYn5Y/DXw0Im6p0/+Pgdcj4otp+SSwFzgJbI6Ir9UZtwHYANDR0dHZ3d3dVN7BwUFmzZrV1NhWaoec7ZARnLNK7ZARnLOeFStW7ImIrjE7RkSpCbgJuK+w/Gngf9Tp+4vUPjHMLLSdn26XAIeAi8baZmdnZzSrp6en6bGt1A452yFjhHNWqR0yRjhnPUBfNPC+XsWhpMPAwsLyAuDI8E6SVgK3AtdHxLFCYTqSbg8CvcAlFWQyM7MmVVEYdgNLJS2WNAO4GcjOLpJ0CXAPtaLwRqF9jqSZaX4e8DGg+KW1mZm1WOmzkiLipKRbgB3ANOD+iHhe0u3UPrZsBX4fmAU8IgnguxFxPfAh4B5Jp6gVqc2Rn81kZmYtVrowAETEdmD7sLbbCvMr64z7JvCRKjKYmVk1/MtnMzPLuDCYmVnGhcHMzDIuDGZmlnFhMDOzjAuDmZllXBjMzCzjwmBmZhkXBjMzy7gwmJlZxoXBzMwyLgxmZpZxYTAzs4wLg5mZZVwYzMwsU0lhkLRa0n5J/ZI2jrB+pqSH0/qnJS0qrNuU2vdLuqaKPGZm1rzShUHSNOAu4FrgYmCtpIuHdfssMBARPw3cCdyRxl5M7VKgHwZWA3+S7s/MzCaJIqLcHUhXAL8TEdek5U0AEfHfC312pD7fkjQdeB04D9hY7FvsN9o2u7q6oq+vb9xZN29/ka/8n4McOzXuoWanEdDIq6fRfmZjmX6W+MS/ms+Xb76kqfGS9kRE11j9qjiUdAHwamH5cGobsU9EnATeBs5tcGwlNm9/kbu/4aJg1Wn0zd5Fwapy8lTwtb1H+I3ub0/odqq45rNGaBv+WqjXp5GxtTuQNgAbADo6Oujt7R1HRHis751x9Tczm6qefP4Ivb1vT9j9V1EYDgMLC8sLgCN1+hxOh5J+Cjja4FgAIuJe4F6oHUpavnz5uEJ+8p3aJwYzs3a38sPns3x5c4eTGlHFoaTdwFJJiyXNoPZl8tZhfbYC69L8jcBTUftyYytwczpraTGwFPi7CjKdZuN1H+JXPr6EmT5B1yoy0sfdMv3MxjL9LHHDsvOb/o6hYRFRegKuA14CXgZuTW23A9en+fcBjwD91N74lxTG3prG7QeubWR7nZ2d0ayenp6mx7ZSO+Rsh4wRzlmldsgY4Zz1AH3RwHtsFYeSiIjtwPZhbbcV5v8ZuKnO2C8BX6oih5mZlecDK2ZmlnFhMDOzjAuDmZllXBjMzCzjwmBmZhkXBjMzy7gwmJlZxoXBzMwyLgxmZpZxYTAzs4wLg5mZZVwYzMws48JgZmYZFwYzM8u4MJiZWcaFwczMMqUKg6S5kp6QdCDdzhmhzzJJ35L0vKRnJf1CYd0Dkr4jaW+alpXJY2Zm5ZX9xLAR2BkRS4GdaXm4d4BfiogPA6uBL0uaXVj/2xGxLE17S+YxM7OSyhaGNcCWNL8FuGF4h4h4KSIOpPkjwBvAeSW3a2ZmE6RsYeiIiNcA0u0HRuss6TJgBvByoflL6RDTnZJmlsxjZmYlKSJG7yA9CXxwhFW3AlsiYnah70BEnPY9Q1o3H+gF1kXErkLb69SKxb3AyxFxe53xG4ANAB0dHZ3d3d2j/8vqGBwcZNasWU2NbaV2yNkOGcE5q9QOGcE561mxYsWeiOgas2NEND0B+4H5aX4+sL9Ov58E/h64aZT7Wg5sa2S7nZ2d0ayenp6mx7ZSO+Rsh4wRzlmldsgY4Zz1AH3RwHts2UNJW4F1aX4d8PjwDpJmAI8BfxERjwxbNz/ditr3E/tK5jEzs5LKFobNwCpJB4BVaRlJXZLuS31+Hvg48JkRTkt9UNJzwHPAPOCLJfOYmVlJ08sMjogfAFeN0N4HrE/zfwX8VZ3xV5bZvpmZVc+/fDYzs4wLg5mZZVwYzMws48JgZmYZFwYzM8u4MJiZWcaFwczMMi4MZmaWcWEwM7OMC4OZmWVcGMzMLOPCYGZmGRcGMzPLuDCYmVnGhcHMzDIuDGZmlilVGCTNlfSEpAPpdk6dfu8Wrt62tdC+WNLTafzD6TKgZmY2icp+YtgI7IyIpcDOtDySH0XEsjRdX2i/A7gzjR8APlsyj5mZlVS2MKwBtqT5LcANjQ6UJOBK4NFmxpuZ2cRQRDQ/WHorImYXlgci4rTDSZJOAnuBk8DmiPiapHnAroj46dRnIfD1iPiXdba1AdgA0NHR0dnd3d1U5sHBQWbNmtXU2FZqh5ztkBGcs0rtkBGcs54VK1bsiYiuMTtGxKgT8CSwb4RpDfDWsL4Dde7j/HS7BDgEXAScB/QX+iwEnhsrT0TQ2dkZzerp6Wl6bCu1Q852yBjhnFVqh4wRzlkP0BcNvMdOb6BwrKy3TtL3JM2PiNckzQfeqHMfR9LtQUm9wCXA3wCzJU2PiJPAAuDIWHnMzGxilf2OYSuwLs2vAx4f3kHSHEkz0/w84GPAC6l69QA3jjbezMxaq2xh2AysknQAWJWWkdQl6b7U50NAn6RnqBWCzRHxQlr3eeBzkvqBc4E/L5nHzMxKGvNQ0mgi4gfAVSO09wHr0/w3gY/UGX8QuKxMBjMzq5Z/+WxmZhkXBjMzy7gwmJlZxoXBzMwyLgxmZpZxYTAzs4wLg5mZZVwYzMws48JgZmYZFwYzM8u4MJiZWcaFwczMMi4MZmaWcWEwM7OMC4OZmWVcGMzMLFOqMEiaK+kJSQfS7ZwR+qyQtLcw/bOkG9K6ByR9p7BuWZk8ZmZWXtlPDBuBnRGxFNiZljMR0RMRyyJiGXAl8A7wt4Uuvz20PiL2lsxjZmYllS0Ma4AtaX4LcMMY/W8Evh4R75TcrpmZTRBFRPODpbciYnZheSAiTjucVFj/FPCHEbEtLT8AXAEcI33iiIhjdcZuADYAdHR0dHZ3dzeVeXBwkFmzZjU1tpXaIWc7ZATnrFI7ZATnrGfFihV7IqJrzI4RMeoEPAnsG2FaA7w1rO/AKPczH3gTOHtYm4CZ1D5x3DZWnoigs7MzmtXT09P02FZqh5ztkDHCOavUDhkjnLMeoC8aeI+d3kDhWFlvnaTvSZofEa9Jmg+8Mcpd/TzwWEScKNz3a2n2mKSvAL81Vh4zM5tYZb9j2AqsS/PrgMdH6bsWeKjYkIoJkkTt+4l9JfOYmVlJZQvDZmCVpAPAqrSMpC5J9w11krQIWAj872HjH5T0HPAcMA/4Ysk8ZmZW0piHkkYTET8ArhqhvQ9YX1g+BFwwQr8ry2zfzMyq518+m5lZxoXBzMwyLgxmZpZxYTAzs4wLg5mZZVwYzMws48JgZmYZFwYzM8u4MJiZWcaFwczMMi4MZmaWcWEwM7OMC4OZmWVcGMzMLOPCYGZmmVKFQdJNkp6XdEpS3QtMS1otab+kfkkbC+2LJT0t6YCkhyXNKJPHzMzKK/uJYR/wb4Fv1OsgaRpwF3AtcDGwVtLFafUdwJ0RsRQYAD5bMo+ZmZVUqjBExIsRsX+MbpcB/RFxMCKOA93AmnSd5yuBR1O/LdSu+zyl7XllgLt6+tnzysCU2FajeSai37aXj7dkPzSilY/LeLY3nsewf+Ddlm+zlc+tsqp8bo4nc5X7qoxWPsdLXdqzQRcArxaWDwMfBc4F3oqIk4X20y7/OZXseWWAT923i+MnTzFj+lk8uP5yOi+cMyHb6h94lz/YOfq2Gs0zUf2OnTjFtkO7JnQ/NKKVj8t4ttdIv2Kf6YJLLh2Y8Mewqj7j6VdWlc/N8WSucl+V0ern+JiFQdKTwAdHWHVrRDzewDY0QluM0l4vxwZgA0BHRwe9vb0NbPp0g4ODTY/d9vJxjp04RQDHT5zioSd3808XTczXIs+8/iOOndCo22o0z2T1a5Wx8pR5zJvZ3nj6FfucjGjJY1Omz/B92arnQqv//RN1f8M1+txs9WtuzMIQEStLbuMwsLCwvAA4AnwfmC1pevrUMNReL8e9wL0AXV1dsXz58qbC9Pb20uzY9y8eYNuhXZw4eYqzp5/F2pU/O4GfGHbyxJHjo26r0TwT1e/4iVPMOHti90Mjxspd5jFvZnvj6VfsM01qyWNYps/wfdmq10SVz83xZK5qf9bT6HOzle89AERE6QnoBbrqrJsOHAQWAzOAZ4APp3WPADen+buBX2tke52dndGsnp6epsdGRPQdOhp//NSB6Dt0tNT9jKWnp6ehbTWaZyL6/ef7dkz4fmjUaLnLPubj3d54+w31+bOvPtnybY63z0j7slWviSqfm+PJXOXrcLjxPDer2M9AXzTynt5Ip7qD4ZPUPhEcA74H7Ejt5wPbC/2uA14CXqZ2CGqofQnwd0B/KhIzG9nuZBaGVmmHnO2QMcI5q9QOGSOcs55GC0OpL58j4jHgsRHaj6RiMLS8Hdg+Qr+D1M5aMjOzKcK/fDYzs4wLg5mZZVwYzMws48JgZmYZFwYzM8uodgZTe5H0JvBKk8PnUftx3VTXDjnbISM4Z5XaISM4Zz0XRsR5Y3Vqy8JQhqS+iKj7X4RPFe2Qsx0ygnNWqR0ygnOW5UNJZmaWcWEwM7PMmVgY7p3sAA1qh5ztkBGcs0rtkBGcs5Qz7jsGMzMb3Zn4icHMzEZxRhUGSasl7ZfUL2njJOZYKKlH0ouSnpf0n1L7XElPSDqQbuekdkn6o5T7WUmXtjDrNEnflrQtLS+W9HTK+LCkGal9ZlruT+sXtTDjbEmPSvqHtE+vmKL78jfT471P0kOS3jcV9qek+yW9IWlfoW3c+0/SutT/gKR1Lcr5++lxf1bSY5JmF9ZtSjn3S7qm0D5h7wMjZSys+y1JIWleWp60fTmmRv4L1v8fJmAatf/2ewnvXRfi4knKMh+4NM2/n9p/SX4x8HvAxtS+Ebgj3vtvy79O7ap3lwNPtzDr54C/Bral5f9Jfg2NX03zvwbcneZvBh5uYcYtwPo0PwOYPdX2JbXL1n4H+BeF/fiZqbA/gY8DlwL7Cm3j2n/AXGrXXZkLzEnzc1qQ82pgepq/o5Dz4vQan0ntWjAvp/eACX0fGCljal8I7KD2+6t5k70vx/x3tHJjkzkBV5CuF5GWNwGbJjtXyvI4sArYD8xPbfOB/Wn+HmBtof+P+01wrgXATuBKYFt6An+/8EL88T5NT/or0vz01E8tyPiT6Q1Xw9qn2r4cuvb53LR/tgHXTJX9CSwa9oY7rv0HrAXuKbRn/SYq57B1nwQeTPPZ63tof7bifWCkjMCjwL8GDvFeYZjUfTnadCYdShp6YQ45nNomVTpEcAnwNNAREa8BpNsPpG6Tlf3LwH8BTqXlc4G3onYp1uE5fpwxrX879Z9oS4A3ga+kQ173STqHKbYvI+IfgT8Avgu8Rm3/7GHq7c8h491/U+H19e+p/QXOKHlanlPS9cA/RsQzw1ZNmYzDnUmFQSO0TeopWZJmAX8D/EZE/N/Ruo7QNqHZJX0CeCMi9jSYY7L273RqH93/NCIuAX5I7dBHPZOSMx2jX0PtsMb5wDnAtaNkmXLP16RerknNK+lW4CTw4FBTnTwtzSnpJ4BbgdtGWl0ny6Q/9mdSYThM7TjfkAXAkUnKgqSzqRWFByPiq6n5e5Lmp/XzgTdS+2Rk/xhwvaRDQDe1w0lfBmZLGrryXzHHjzOm9T8FHJ3gjEPbPRwRT6flR6kViqm0LwFWAt+JiDcj4gTwVeDnmHr7c8h499+kvb7Sl7OfAD4V6djLFMp5EbU/Bp5Jr6UFwN9L+uAUyniaM6kw7AaWprNAZlD7Qm/rZASRJODPgRcj4g8Lq7YCQ2cgrKP23cNQ+y+lsxguB94e+pg/USJiU0QsiIhF1PbVUxHxKaAHuLFOxqHsN6b+E/5XTkS8Drwq6WdS01XAC0yhfZl8F7hc0k+kx38o55TanwXj3X87gKslzUmfjq5ObRNK0mrg88D1EfHOsPw3p7O7FgNLqV1fvqXvAxHxXER8ICIWpdfSYWonnrzOFNuXw4OfMRO1swBeonZWwq2TmOPfUPto+CywN03XUTuGvBM4kG7npv4C7kq5nwO6Wpx3Oe+dlbSE2gusH3gEmJna35eW+9P6JS3MtwzoS/vza9TO5Jhy+xL4r8A/APuAv6R2xsyk70/gIWrfe5yg9sb12Wb2H7Vj/P1p+uUW5eyndjx+6HV0d6H/rSnnfuDaQvuEvQ+MlHHY+kO89+XzpO3LsSb/8tnMzDJn0qEkMzNrgAuDmZllXBjMzCzjwmBmZhkXBjMzy7gwmJlZxoXBzMwyLgxmZpb5f3+ymoKXGLmmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x181584f978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(rewards_list,'.')\n",
    "ax=plt.gca()\n",
    "ax.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *if we train our network with this reward, it learns nothing*\n",
    "because the reward is most of the time zero. in this case the agent does not get feedback what is good or bad. most gradients become zero because we multiply by reward in the mentioned algo.\n",
    "\n",
    "it looks like the environment just rewards us when the ball passes through our paddel(we score -1) or our opponent's paddle(we score +1).\n",
    "\n",
    "now think about it:\n",
    "\n",
    "our paddle moves up and down, hits the ball and the ball travels to the left side of the screen. the opponent fails to catch it and we win and get a score of 1. \n",
    "\n",
    "in this scenario, which actions were actually good? \n",
    "\n",
    "* the action we took when we hit the ball? \n",
    "* the action we took when we get score of 1?\n",
    "\n",
    "of course the first option is correct. \n",
    "the second option is wrong, because when the ball is reaching the opponent, our action does not matter. only the action that we took to hit the ball were important to winning. and everything after hitting the ball was irrelevant to our win. \n",
    "\n",
    "(a rather similar argument can be discussed about losing, the actions near when we got score of -1 is more important than actions taken many steps earlier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**so here is the trick.**\n",
    "\n",
    "we set the reward of actions taken before each reward, similar to the reward obtained.\n",
    "\n",
    "for example if we got reward +1 at time 200, we say that reward of time 199 is +0.99, reward of time 198 is +0.98 and so on. \n",
    "\n",
    "with this reward definition, we have the rewards for actions that actually resulted in a +1 or -1. and we assume the more recent the action to the reward gained, the more important it is.\n",
    "\n",
    "we define a function below, that does this for us. \n",
    "\n",
    "note that this may not be applicable in other environments, but serves our purposes for this project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_rewards(r_list):\n",
    "    reward_decay=0.99\n",
    "    tmp_r=0\n",
    "    rew=np.zeros_like(r_list,dtype=np.float32)\n",
    "    for i in range(len(r_list)-1,-1,-1):\n",
    "        if r_list[i]==0:\n",
    "            tmp_r=tmp_r*reward_decay\n",
    "            rew[i]=tmp_r\n",
    "        else: \n",
    "            tmp_r = r_list[i]\n",
    "            rew[i]=tmp_r\n",
    "    return rew"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztvXl0XMd1oP/dbuwAsRPgjoU7RYoLwE0rtMuObdmJvE0WObaPJpk4M5OcJLbGM87EE5+x4xnHM/lpbCu2YiV2LMWyLdGyLImSCC0mxVXcV5AECXAnAQLEQmxdvz/ee0B3473uRnej0SDvdw4Out+r997telV1b926VSXGGBRFURTFwTfRAiiKoijphSoGRVEUJQRVDIqiKEoIqhgURVGUEFQxKIqiKCGoYlAURVFCUMWgKIqihKCKQVEURQlBFYOiKIoSQsZECxAP5eXlprq6Oq5ru7u7yc/PT65A48BkkHMyyAgqZzKZDDKCyunFzp07LxtjpkZNaIyZdH91dXUmXjZt2hT3talkMsg5GWQ0RuVMJpNBRmNUTi+AHSaGNlZdSYqiKEoIqhgURVGUEFQxKIqiKCGoYlAURVFCUMWgKIqihJAUxSAiT4vIRRHZ73FeROT/ikiTiOwVkVVB5x4TkWP232PJkEdRFEWJn2T1GH4IPBzh/AeA+fbf48B3AESkFPhrYC2wBvhrESlJkkyKoihKHCRFMRhj3gbaIiR5BPhnO5T2PaBYRKYDDwEbjTFtxph2YCORFYxygzEwFOC57acZHApMtCiKotikaubzTKAl6Hurfczr+ChE5HGs3gaVlZU0NjbGJUhXV1fc16aSySBnMmT82bF+fnl8gFNNR1kzfXyK42TIS5gcck4GGUHlTJRUKQZxOWYiHB990JingKcA6uvrTUNDQ1yCNDY2Eu+1qWQyyJkMGb+1/12gg3V1y7lrQfSZ+vEwGfISJoeck0FGUDkTJVVRSa3A7KDvs4CzEY4rNwl7WzsAyPRrgJyipAupqo0bgD+wo5PWAR3GmHPAq8CDIlJiDzo/aB9TbgJ0XEFR0pOkuJJE5CdAA1AuIq1YkUaZAMaY7wIvAx8EmoAe4A/tc20i8j+A7fatvmqMiTSIrdxAHD5/baJFUBTFhaQoBmPMp6OcN8CfeJx7Gng6GXIok4udp9qHPxv3oSVFUSYAdewqE0awYlAUJX1QxaBMGDtPtVNekDXRYiiKEoYqBmVCONfRy5mrvdRXlVoH1JOkKGmDKgZlQtjRbLmR6qt1BRRFSTdUMSgTws5T7eRm+lkyvRDQDoOipBOqGJQJYeepdlbMLiZDJ7YpStqhtVJJOd19gxw816luJEVJU1QxKCln1+l2hgKGuqoRxWDUl6QoaYMqBiXlbDl+hQyfsLq6FHFbRlFRlAlFFYOScracuMKts4rIzx6ZeK8znxUlfVDFoKSUrr5B9rZ2sH5uGeC+7rqiKBOLKgYlpWxvbmMoYFhfWz7RoiiK4oEqBiWlvHfiCpl+CRl4Bh18VpR0QhWDklLeO36FFbOLyc3yA+jgs6KkIaoYlJTReX2AfWc6WF9bNtGiKIoSgaQoBhF5WESOiEiTiHzJ5fzfi8hu+++oiFwNOjcUdG5DMuRR0pOtJ9oIGFg3d7RiUE+SoqQPCW/UIyJ+4EngAaw9nLeLyAZjzEEnjTHmz4LS/ymwMugWvcaYFYnKoaQ/bx+9RF6Wf2RFVUDjkhQl/UhGj2EN0GSMOWGM6QeeBR6JkP7TwE+S8FxlEmGMofHoRW6bW05WxuhiZ3T0WVHShmQohplAS9D3VvvYKESkCqgB3gw6nCMiO0TkPRH5aBLkUdKQ5is9tLT1cveC0DBVHXxWlPQjGXs+u1VtL/PvU8DzxpihoGNzjDFnRaQWeFNE9hljjo96iMjjwOMAlZWVNDY2xiVsV1dX3Nemkskg51hk3HhqAIDs9hM0NjYPH2+6ahWFvXv3wrmkbEE+ismQlzA55JwMMoLKmTDGmIT+gPXAq0HfnwCe8Ej7PnBbhHv9EHg02jPr6upMvGzatCnua1PJZJBzLDJ+5umtpuGbo9PvPNVmqr74knnz8IXkCRbGZMhLYyaHnJNBRmNUTi+AHSaGdj0ZrqTtwHwRqRGRLKxewajoIhFZCJQAW4KOlYhItv25HLgdOBh+rTK5uT4wxJYTV7h7wdRR59STpCjpR8J9d2PMoIh8AXgV8ANPG2MOiMhXsbSToyQ+DTxray2HxcD3RCSANd7xdRMUzaTcGGxvbuP6QMBVMQyjY8+KkjYkxalrjHkZeDns2FfCvv93l+s2A8uSIYOSvrx+8AI5mT7WuUxsEx19VpS0Q2c+K+OKMYaNBy9w5/ypw8tgKIqS3qhiUMaVA2c7OdtxnQeXVEZMp/sxKEr6oIpBGVdeO3Aen8B9i90VgzqSFCX9UMWgjCuvHbxAfXUppflZEy2KoigxoopBGTda2no4fP5aVDcS6H4MipJOqGJQxo1XD5wH4IEIikGDkhQl/VDFoIwbv9x7jqUzC6kqy4+aVnsMipI+qGJQxoXTV3rY03KVD986I2I60eFnRUk7VDEo48Iv954F4LdunT7BkiiKMlZUMSjjwkt7z7FqTjGzSvJiSq+eJEVJH1QxKEmn6eI1Dp3r5MPLI7uRQAefFSUdUcWgJJ0Nu88iAr+1LHY3ktHRZ0VJG1QxKEklEDD8bNcZ7phXTkVhzkSLoyhKHKhiUJLK5uNXOHO1l0/Uz55oURRFiRNVDEpS+bcdLRTmZESc1OaGOpIUJX1QxaAkjY6eAV45cJ6PrpxJTmZsS2zr4LOipB9JUQwi8rCIHBGRJhH5ksv5z4jIJRHZbf99PujcYyJyzP57LBnyKBPDhr1n6R8M8PE6dSMpymQm4R3cRMQPPAk8ALQC20Vkg8sWnc8ZY74Qdm0p8NdAPZY3Yad9bXuicimpxRjDc9tPs2jaFJbOLIzj+nEQSlGUuEhGj2EN0GSMOWGM6QeeBR6J8dqHgI3GmDZbGWwEHk6CTEqK2XX6KvvPdPK766rGtF2nLomhKOlHMhTDTKAl6HurfSyc3xGRvSLyvIg4voZYr1XSnGc2NzMlJ4PfXhnv69Mug6KkCwm7knDfhCu8lv8S+Ikxpk9E/gh4Brg3xmuth4g8DjwOUFlZSWNjY1zCdnV1xX1tKpkMcjoyXr0e4Fd7e7lvTgbbt7w7pnu0XAsAsH//AXIuHxkPMSdFXsLkkHMyyAgqZ8IYYxL6A9YDrwZ9fwJ4IkJ6P9Bhf/408L2gc98DPh3tmXV1dSZeNm3aFPe1qWQyyOnI+K3XjpjqL71kTl7qGvM9Dp7tMFVffMn8et/ZJEs3wmTIS2Mmh5yTQUZjVE4vgB0mhnY9Ga6k7cB8EakRkSzgU8CG4AQiErw2wkeAQ/bnV4EHRaREREqAB+1jyiShb3CIf912moYFU6kuj77vghc6+Kwo6UPCriRjzKCIfAGrQfcDTxtjDojIV7G00wbgP4rIR4BBoA34jH1tm4j8DyzlAvBVY0xbojIpqeMXu85w6Voff3h7TVzX6zwGRUk/kjHGgDHmZeDlsGNfCfr8BJaLye3ap4GnkyGHkloCxvDdt46zbGYRd84vT+he2mFQlPRBZz4rcbP9/BDNV3r4Dw1zxxSiqihKeqOKQYkLYwwvnRhg7tR8HrplWtz30XkMipJ+qGJQ4mLz8Su0XAvwxw3z8PkSb9x18FlR0gdVDEpcnLnaC8DamtKE7qMeKEVJP1QxKAmhDbui3HioYlDiI8muH6NxSYqSNqhiUBIi0Wgk7XAoSvqhikFJC3TwWVHSB1UMSlwky/WjYxSKkn6oYlASQtt1RbnxUMWgxEWyXT/qSVKU9EEVg5IQibuCtM+hKOmGKgZFURQlBFUMSlwk2/VjNCxJUdIGVQxKQiS6CJ5GJSlK+qGKQYkLNfAV5cYlKYpBRB4WkSMi0iQiX3I5/+ciclBE9orIGyJSFXRuSER2238bwq9V0ptELX7tMChK+pHwDm4i4geeBB4AWoHtIrLBGHMwKNn7QL0xpkdE/hj4O+CT9rleY8yKROVQFEVRkkMyegxrgCZjzAljTD/wLPBIcAJjzCZjTI/99T1gVhKeq0wgyV70Tl1TipI+JGPP55lAS9D3VmBthPSfA34d9D1HRHYAg8DXjTEvuF0kIo8DjwNUVlbS2NgYl7BdXV1xX5tK0l3Oo6cHANiyeTPFOfHbF+e7AwAcPHSI4o5jSZEtnHTPS4fJIOdkkBFUzoQxxiT0B3wc+H7Q998H/sEj7e9h9Riyg47NsP/XAs3A3GjPrKurM/GyadOmuK9NJeku54/eazZVX3zJXOjoTeg+Jy51maovvmR+vqslSZKNJt3z0mEyyDkZZDRG5fQC2GFiaNeT4UpqBWYHfZ8FnA1PJCL3A18GPmKM6QtSTGft/yeARmBlEmRSxplkuX508FlR0o9kKIbtwHwRqRGRLOBTQEh0kYisBL6HpRQuBh0vEZFs+3M5cDsQPGitpDvasivKDUfCYwzGmEER+QLwKuAHnjbGHBCRr2J1WzYA3wQKgJ/aG7ucNsZ8BFgMfE9EAlhK6usmNJpJSVOSP/M5yTdUFCVukjH4jDHmZeDlsGNfCfp8v8d1m4FlyZBBmRh05rOi3HjozGdFURQlBFUMSnwk2fejriRFSR9UMSgJkfiSGOpLUpR0QxWDEhdJH3xO8v0URYkfVQxKQiRq7+vgs6KkH6oYFEVRlBBUMShxkezBYqOjz4qSNqhiUBJC1BekKDccqhiUuEi2ha/9BUVJH1QxKBOKdjgUJf1QxaAkhLbrinLjoYpBiYuku37Ul6QoaYMqBiUhEp75rL4kRUk7VDEoiqIoIahiUOIi6fMY1JekKGlDUhSDiDwsIkdEpElEvuRyPltEnrPPbxWR6qBzT9jHj4jIQ8mQR0kdCe/HkCQ5FEVJHgkrBhHxA08CHwCWAJ8WkSVhyT4HtBtj5gF/D3zDvnYJ1lagtwAPA//Pvp+S5ugObopy45KMHsMaoMkYc8IY0w88CzwSluYR4Bn78/PAfWKNOj4CPGuM6TPGnASa7Pspk4WEB5+TI4aiKMkjGYphJtAS9L3VPuaaxhgzCHQAZTFeqyiKoqSQZOz57GbzhTsGvNLEcq11A5HHgccBKisraWxsHIOII3R1dcV9bSpJdzmbmgcAePfdd8nPjN/sb7seAODwkSM09pxIimzhpHteOkwGOSeDjKByJkoyFEMrMDvo+yzgrEeaVhHJAIqAthivBcAY8xTwFEB9fb1paGiIS9jGxkbivTaVpLucTe+cgMOHuPPOOyjMyYz7Puc7rkPjGyxcuJCGNXOSKOEI6Z6XDpNBzskgI6iciZIMV9J2YL6I1IhIFtZg8oawNBuAx+zPjwJvGmsVtg3Ap+yopRpgPrAtCTIpkwwdfFaU9CHhHoMxZlBEvgC8CviBp40xB0Tkq8AOY8wG4AfAv4hIE1ZP4VP2tQdE5N+Ag8Ag8CfGmKFEZVIURVHiJxmuJIwxLwMvhx37StDn68DHPa79GvC1ZMihpB7d2lNRbjx05rMSFzrzWVFuXFQxKAmR6CJ42mFQlPRDFYOiKIoSgioGJS6S7frRqCRFSR9UMSgJkbArSH1JipJ2qGJQ4iL5g8+KoqQLqhiUhEh4BzftMihK2qGKQVEURQlBFYMSF0l3/ejos6KkDaoYlIRIeAc39SQpStqhikGJCx18VpQbF1UMiqIoSgiqGJSESDwqSVGUdEMVgxIXOvNZUW5cVDEoE0qii/ApipJ8VDEoiqIoISSkGESkVEQ2isgx+3+JS5oVIrJFRA6IyF4R+WTQuR+KyEkR2W3/rUhEHiV1JD0qSX1JipI2JNpj+BLwhjFmPvCG/T2cHuAPjDG3AA8D3xaR4qDzf2mMWWH/7U5QHiXF6OCzotx4JKoYHgGesT8/A3w0PIEx5qgx5pj9+SxwEZia4HOVGwztLyhK+pCoYqg0xpwDsP9XREosImuALOB40OGv2S6mvxeR7ATlUVKMznxWlBsPiebbFZHXgWkup74MPGOMKQ5K226MGTXOYJ+bDjQCjxlj3gs6dh5LWTwFHDfGfNXj+seBxwEqKyvrnn322ci/zIOuri4KCgriujaVpLucvzzez8+ODfD9B/PI8MXfunf1G77wZg+/uyiLB6oz477PUMDw48P93Dc7k5lTQu2ddM9Lh8kg52SQEVROL+65556dxpj6qAmNMXH/AUeA6fbn6cARj3SFwC7g4xHu1QC8FMtz6+rqTLxs2rQp7msd+geHEr5HNJIh53jyD28cNVVffMn0DSSWF+3dfabqiy+ZH7xzIqH7nLrcbaq++JJp+OYm09nbH3Iu3fPSYTLIORlkNEbl9ALYYWJoYxN1JW0AHrM/Pwa8GJ5ARLKAXwD/bIz5adi56fZ/wRqf2J+gPONOa3sPS//6VX655+xEi5IWpMt+DM6Eu5OXu3ni5/s0yklREiBRxfB14AEROQY8YH9HROpF5Pt2mk8AdwGfcQlL/bGI7AP2AeXA3yYoz7hzuaufvsEA/+UX+2ht75locSaMdFtEz5Fn6cxCXtp7jh9tPZ2wTEpsdPUNct//bmTjwQsTLcqE0dbdz56WqxMtRtJISDEYY64YY+4zxsy3/7fZx3cYYz5vf/6RMSbTjISkDoelGmPuNcYsM8YsNcb8njGmK/GfNL44lui164P82XO7GQrc3JZpuo0d/+FtNdy9YCpf+9VBTl7unmhxbgqudPVx/FI3f/X8Hi5euz7R4kwI33/nBL/9nc3sbb0xlIPOfB4jjhp4ZMUMtje38923jkdMr0QhSZrFeS8+H/zdo7eS5ffxFz/dc9Mr7lTg9Nbaewb4LzepG+/6QIChgOEvfrqHvsGhiRYnYVQxjBGnzH9s5Uw+uGwa/+eNY5y4lPYdnaST7KqfaGPiXC8IlYU5/M0jt7DzVDs/ePdEMsRTIuC8ueWzi3n90EWe39k6ofJMBM4Y19ELXfyf149NsDSJo4ohTkSE//7hW8jO8PHEz/cRuEkt00QXwUvWPIbw3P/oipk8uKSS//XaUS72BJLzEMUVRyk/tr6KNdWlfO3lQ7R190+wVKlnSk4GH6+bxXffOs7+Mx0TLU5CqGIYM45lChWFOXz5g4vZerKNf9vRMrFiKcCIohERvvrIUjJ9wo8P9d+U7o1U4/cJf/uxpXRdH+R/vnxoosVJKU7x+q8fWkJpfhb/9YX9k9pYVMUwRsLbl0+uns3amlL+568Pc7Xn5rGS0q2ddZNnWlEOf/bAAvZcGrqpI2bGm+CsX1A5hc/fWctPd7ayvbltwmSaCAQoys3kiQ8sZnfL1UntUlPFMEacShBsmf7NI7dw7foA374BfItjJVFPUPKimtw11WO3VTOrQPibXx6kp38waU9TRghXyv/xvnnMKMrhv72wn8Ghm8ONZ4wZdqv+9qqZ1FeV8PVXDtPRMzDBksWHKoY4CZ6YtWhaIZ9eM4d/ee8UTRevTaBUqSNdd3ALH/PI9Pv4/SXZnLnay9PvnkzOQ5QwbPeqnfd5WRn81w8t4fD5a/xs1+S1msdKuBvzak8/f//60YkVKk5UMYwRpwELHzT98wcWkJfp52u/url8qwnPfE7S6PPwe3E5t7DUz4NLKvnuWye40tWXlOcpownO+w8sncbKOcV8a+PRm6KnFm7XLJlRyCdXz+bHW0/R0jb5JsKqYhgjXoOYZQXZ/Ol989h05BKbmy6nWCol3MUXzl89vIie/kH+4c2mlMl0s+BmLIkIX/7gYi509t0UPTVjRhsl/+m+BfhE+NbGyddrUMUwRoYbIJdzf7C+mmmFOXxr49EbPgom+UtiJDqPIfL5eRUFwxbc6SuTz4JLZ7yyvr66lIduuTl6agYzqvc7rSiHz95Rwwu7z3DwbOcESRYfqhjixUUz5GT6+ZN757HjVDvvHLs5eg0Jz2NIkhwj9/O+43++fwF+n/DtNyafBZfOjLjxRuf9Xz5k9dT+8Z0bv9fgVvL+6O65FOZk8s1XD6dcnkRQxTBGIlUCgE/Wz2ZmcS7/+wbvNSR/5nOC1w8PgHqnqSzM4XfXVvHi7rPaa0gikfJ+XkUBH7p1Bv+8pZn2G3jSm1f5LcrN5N/fXcumI5fY1zp5Jr3dVIrh678+zD/tT6xLG83lkZXh40/vnceelqtsOnIxoWfdDORk+inNz2LbycRi3iMNPgfz+F21+EX4jq5xlXS88v4L986jp3+Ip39z4/YaDN5Gye+vq6IwJ4MnN02e8a2bSjEMDAV458wgZ672xn8Tj6ikYH6nbhYzi3P5TqM2PtHw+4TH1lfzxuGLHDkff6hvrD2OysIcPrF6Fs/vbOFcRwLlQBkmWt4vqJzCB5dN44e/aaajd3LG9UfDygP3RmFKTiafua2aVw6c59iFyRHOflMphs/eUQOQlCiJSJZppt/H5+6oYXtzO7tOtyf8rLQkiW6yP1hfRW6mn+8lwYqPZcjjj+6eizHwvbd0gb1k4BXCHcwX7pnPtb5B/nlzc0pkmggi/f4/vL2GvCw//2+SGIs3lWKYWZzL2ml+nt12Om7LZSQsMnIL9MnVsynKzeSpG7jxSdbAcUl+Fp9eM4cNe87G3ZszEePFQplVksdHV87kue0tk3ZmajoRS94vmVFIw8KpPLPl1A2xLPVoTMSSV5Kfxb+zy/hkmNeQkGIQkVIR2Sgix+z/JR7phoJ2b9sQdLxGRLba1z9nbwM6rjxck0l3/xD/GucOX7EayvnZGfzeujm8evA8zbphTFQ+f6fVm/v+O/Ep0lis1mA+d0cNvQNDPLtdd3pLFtHy/rO313C5q4+X9pxLjUApJJZ2wfFY/Oi9U+MsTeIk2mP4EvCGMWY+8Ib93Y3eoN3bPhJ0/BvA39vXtwOfS1CeqFQV+rljXjn/9JuT9A+OfR2XWKJfHB67rZpMn4/v34B7AiQ7KmlGcS4fXj6Dn+5opatv/GfKLp5eyLraUp7Z3HzTrOczXsRqLN05v5z5FQU8/ZuTN1zEnjHR24QZxbk8fMs0frLtdNrPBk9UMTwCPGN/fgb4aKwXiuWLuRd4Pp7rE+Fzd9Zw8Vofrxw4H/c9YjFMK6bk8MiKGfx81xmuXVeXRTR+f30VXX2D/OL9M3HfYyzurc/eXsPZjuu8piuvJoVoeS8ifPaOGg6c7WRrglFo6UikOTQOn7m9ms7rg7zw/tkUSBQ/GQleX2mMOQdgjDknIhUe6XJEZAcwCHzdGPMCUAZcNcY4qrMVmOn1IBF5HHgcoLKyksbGxrgE7urqIu/sAabmCk++sofC9rFNdtp/2RL3/fffp6vZHzX94qwhfto/xDeea+T+qswxyRnvb0wFzc39gEmqjMYYqgp9fPf1A8zqPTGmyXPNHZbf+sCB/WRdCp1M5JWXGcYwNVf49su7ybtyJCHZk0G6v3NwlzFS3odTNmQoyIS/e2E7/3FVzniJmfK8PHuuj/7+oajPdMr4kxv3M73nON3d3Wn5zqMqBhF5HZjmcurLY3jOHGPMWRGpBd4UkX2A2xxxz/6lMeYp4CmA+vp609DQMIbHj9DY2EhDQwOfleN845XDzFxcx/zKKTFfL0cvwY5trFq1krqq0qjpG4BfnH6X7W0B/scf3BlzY+fIma7s6DuCnGxKuoyXClr4q5/tJbfqVtbVlsV83d7Wq7DlNyxduoyGJZUh5yLl5b/POMHf/uoQFQtWsWRGYSKiJ0y6v3Nwl3FfawdseZdlLnnvxu/1H+apt4+zaOU6phWNj3JIdV7+6tIemroux/TMy1Na+Mvn95I9exnSuj8t33lUV5Ix5n5jzFKXvxeBCyIyHcD+7zqjyxhz1v5/AmgEVgKXgWIRcZTTLCBl/atP1M8iy+/jx2MchB7xjcZuzf7eujkcuXCNHadu0NDVJPLh5TMoys3kX7bEN0A31kipR+tmkZXh0x34EmCs61x9cvVsAgae33nj5Lkh9rLnlPF/3Za+gQ+JjjFsAB6zPz8GvBieQERKRCTb/lwO3A4cNFYLuwl4NNL140VZQTYfWDaNn+1qjWsgaCxLBH14+Qym5GRMimiEWEn2fgwOuVl+Pl43i1cPnOfStdhnqY81KsmhOC+Lh2+Zxi/eP8P1gRsxjHL8GWve15Tns662lOd2tEzq7S+DsQafY8uAnEw/H1s5k9cOXKCrPz1/f6KK4evAAyJyDHjA/o6I1IvI9+00i4EdIrIHSxF83Rhz0D73ReDPRaQJa8zhBwnKMyZ+d20V164P8tLe2MPnxt5fsDYu+Z1Vs/j1vvM31HoxyV4Az+GTq2czGDC8uDv2Qehoy25He15H7wCvJhCMcDMTT95/es0cWtp62Xz8yrjIlO58on42/UMBtpxLz+ikhBSDMeaKMeY+Y8x8+3+bfXyHMebz9ufNxphlxpjl9v8fBF1/whizxhgzzxjzcWNMStfmXV1dQk15Pj8fyy5TcSp4pyC8tDe9oxFiZTyjDedXTmH5rCKe39kac1hjIuGP62vLmF2ay3PbbxzXxkQQS1SOw0O3TKMoN/OGmUcy1h70khmFLJ1ZyDutN6BimOyICL+9cibvnWiLeTbiyDyGsZmmS2YUsmjaFH62K/5QzJuJR+tmcfj8NQ6McR37sTRODj6f8Im62Ww+fkVXXY2DeJRysDvlhuhFxzCPIZxP1s/m9LUA+8+k36qrN7ViAPjoSitC9oUxxs7H40b5nVWz2N1yleOXuuK4+ubiw8tnkOX3xbxn8HDTFKd/69H6WYjA8zfRHsXJIt68f7RuFv1DAX61b/LPhI60uqoXH1kxk0wfaRn4cNMrhtmleaytKeXn75+JyfJJxIXyyIoZ+AR+cQP0GsZ7yKw4L4sHllTy4u6zMc1Qj3XZbS+mF+WyvraMX+45e8PNyh1v4s37W2YUMq+igA27bwz36lh7q0W5mays8PPS3nMMpNns+5teMYC1TPbJy93sOn01atp4o18AKgpzuGvBVH7x/pkbIhpjvAafHX6nbiZt3f00pmhfi48sn8HJy93sPzO5tmFMF8bqXhURPrpiBtua22htn9wuvHiNiXXTM2jr7uc3abZPvCoG4ANLp5Gd4WNDDFEwI1FJ8TVMoQuIAAAgAElEQVSLH1s5kzNXe3VOQwzcOX8qJXmZMboa4hv7CeYDS6eT6ZcxRUPdyDy5qYlvvx7LygDxGzkfWW65cn85yRfWi8eVBLBsqp/CnAw27EmvXpMqBqyNNBoWTuXX+8/HbMnH2/7cv7iS7AwfLyfJr3q+4zrffPVwjO6W5PVSUuFtyfT7eHjpNF4/eCHqHINEXUkARXmZ3L2ggl/uPctQmvboUunm2nT4Ik+9fYLe/vHL+zlleayaU5xyZTwUMHzjlcOc77ielPsZE9/vz/QJDy+dxmsHopfxVKKKweaDy6Zz8VpfVEs+0YqZn51hK6FzSXEnvX3sEk9uOs47xy5FTNfW3U/d377O1hOR48aNMbx+8EJsK8+Oty8J+K1lM+juH6LxSOTfl8g8hmAeWTGDC519CW81OhY6egbYfDy6K+HvXjnMZ/5pewoksjBAT/8Qbx0d37z/6MqZHD5/LaEd/MZKa3sP32k8zj8lcbvReHurH14+g66+wZS5TGNBFYPNfTFa8smw1z64bDoXOvvYmYTd3RxF9cr+yJOzLl67Tlt3Pz/dGTnqpvlKD5//5x38JMp0/fGa+RzOutpSSvOzUha5cv/iSnIz/fxqX/SufSwr5l4fGIqqZH+6s4V/949budgTOd3xS128dfRSyvzxTtmKtXcbr3v14aXTEIlehi90XuePf7STqz2Rw1uf236al09GTuPYZK8cOB/V2OvtH4q6NHsitWF9bRnlBVlp5U5SxWBTYFvyL++LbMknMvjscN/iSrIyfPwqyozrlraeqLNxHXk2HroQMbLBSfd6lHTO7lqxzAJOQYeBDNud9MahCxFdGiPujMSkys3y07BwKq8duBCxHJy+0sPyv3ktqqX/2NPbeOLn+yKm6bMVx64LsblsXj0QeZnwg2c72ZyEwUzn579xKLKbI1HvVsWUHOrmlERdBn9Py1V+vf981JUKXtp7jheaBqLIbAl96koPh85F7ql86B/e4ZuvRV5915jIO7hFwinjmw5fSht3kiqGIGJ1J0FiDVBBdgYNC6K7k3609RR/9KOdXO7ynhDuXH61Z4CtJ7zdHwG7IlztGYjoJgnYOmPryba0mXj0oWXT6ekfitjVdip6oq4ksGblXrzWx+5W7yi1y919BAz8MoqVd6mrj1f2n4upkdp1MfIsWOddvxrFsn5yUxN//ONdEa3cgaEAVyKUK7Cs4NxMP939Q7xzzFvRJCPvH146jUPnOjl1xXu3Q6emxLJ/Rv8QvBtB5uBq98r+yIrm0rU+Xnw/chizgYQspQeXTKN3YCiizKlEFUMQjjvp1xELSnJcKL91q+VOer/FWwkNDhmMgTcPR2gQg3aUe+WAt9wmpCJ4NyzO/YYChjciPDdFniQA1tSUUpKXycYIDUI8a1h5cc+iCjJ8ErHX5DQSGw9ejNrD7O4fYkuENYGcd3OsPRBl4UAr4fZTbRHTDQwF6OgdYHuzd9n6ly2naPhfjXRH2i3PGOqqSijOy4zoTkpG3j90i7Wyfyx5vuX4ZTojuPEcI+i1g5EUaHC9id4rP995nX1RZign8vvX1ZYxJScjisypQxVDEAXZGdw+r5zXD13wtA6S4UqCkcZn48FIVrD1P2KDaKepryrh1QjuDyddfpaf1w56R18F/+xo7qRUuJLA6mrfs6iCN49cTMk2nEW5mayfW8ZrB6KXg8tdfbzf4t2zMMONlPc7dF6FwXL1RUqXn+XHmMjpHIkjlZv2nn6uXR+M2BMIGMj0Cw8sruSNKC5IIKECMbs0j1tmFEZ0kzl5PjBk2BTJWBp2m3qXl+B6c/RCV8TVCGLJz0QNpawMH/cuqogocypRxRDGfYsraGnr5dhF94KSrOiXwpxM1tSU8kbEhsB62jvHLnn6152G5wNLp3PpmncPxOkJ3Lu4kgud3m4Sp8LMKsnl7aOXPJckT3Uw5/2LK7naM8BODzdfHNtkROShW6Zx8nK3ZzkI1quRrLzAcCMVQWnbuVmeG72XUju1gKqyvMi9Pqc3c8h7YNUpW5HKn8EgIty3uJLO64PR8z5BHrplGjtPtXOx0z2ENLjuRVa0lr+/rbvfU2bnVTy8dDoQ2Qga7oFEUlp2XiXCg0umRZQ5lahiCOO+RdYOVJEsMkh8kBMs19Wxi11RF267PhDwnBnpVJZ7FlXg94mn28mpCPcvdnoq7r/PqQQP3zKNvsEAb0cKVUxVlwG4a8FUsvw+z/cy7FJLklAPLqlExNuf7zS4eVn+KC4uQ16Wn0sRxiycd1NX6Wdz0xXPaCcD+MRqQDcfv0yXhxvIaahb2no5esHDwLHTbDri7QoLBKzn3TG/nCy/z7NsJSvvHXfS64e8yrD1nLo5JTQevjgcKDFKHgNzCn1kZfg8FYgj84yiHG6ZURi1B5KX5efIhWuedTXeeQzB3L1wakSZU4kqhjCmFeWwdGYhb3gUzmTOL7p/sbVFtldjFzCGguwMpmRneDfkdqUuys2krqqENw+7N+ROQ1aYm0l9dYnnvADn5621fZ6RxjdSSUF2BuvmlrHxoLd7J5lUFOZw66xi3vQY8Hba0oaFUzlxqZsmr55FAG6bWx5RGTuFamVFBv1D3kZAwFix8vcsrGBgyHhGHgWMYVqhtWXmRo/ejCP/5a5+794jAEJBdgZra0u9jaUkuVcXVBYwszjXM8jADFv50+juH/Lcy8EYyM2AO+aV89pB916TE2QhAvcuqmDnqXbPMNiAMdy9YCoQuXeY6O8vyM6IKHMqUcXgwn2LKtl1ut01aiN4sDdRqsrymVdRwBuHPawa28d798KpvHHY3RXhHPHZBfzQuU7OdfR6phPgnoVWOrdZn45VluEX7po/lbeOXnItpBNRcB9YXEHzlR6OX3KJXElS4xRMw4Kp7G656hqd5ZSDB5dYVu6bHu8QoDgvk7W13m5Dq8GHecU+pmRneCttYxCB+uoSCrIz2BRBuVcWZrN8drGnMjLG4BPw+8RTLud5APctquDEpW7Od3v7vxPNehGhYeFUftN02bU34JTN9XPLyMn08Zbn77dcSfcsstzCJy+PLi/By+ffs6iCgMFzIp/BmqG9sHJKRIMxGb3VSDKnkoQUg4iUishGETlm/y9xSXOPiOwO+rsuIh+1z/1QRE4GnVuRiDzJ4v7FlRiDZ8WD5HlR7ltcwdYTba5RFgFj8Ilw3+IKLnf1s//s6KiI4Pj9exdZPZBNLr2GkZBCqyKA5Ubwup9PLIV0obPPM847hZ4kwHK9gbtfPMlDDIBVSY2xZpePep79wBnFuSysnMLbR72td59Aw4IKjl7o8lDa1nvO8Al3zC+n8YiXMrZ+X6bfx53zy2k8ctHdGrZ7FvctqmDvmQ4PAweyM/ysri6J2Nj57Ay913ax7rk0usFOpolwz8IKuvuH2BEhoio308+62jLPhtxRtA22le+maIOX8Vg+q5jS/CxPd5Kx62HDwqnsONXmGsmVrAmfd8/3ljmVJNpj+BLwhjFmPvCG/T0EY8wmY8wKY8wK4F6gB3gtKMlfOueNMbsTlCcpLJ1ZSGVhtmtBSVZUksP9iysZDBjX+GVjP+dOu7C4+fsdK0p8ML/C6opHbvCD0rlWBMeBMFKxIt0vlcwozmXRtCkRG+pEBwCDuXVmEaX5WZEbFrF8w9tOtrkO1DuW5N0LI73DEYXWsHAq5zuvc+TCaGXsKBCwGtBzHdc57LKMhGPp371gKsbAuy4up0DAUlj3Lark8PlrnLnqrrAcK3hOWR7zKwrY7TLXIpl5f9u8MrL8PteyGQgybhoWTOXk5W7XeQ/OZLPZpXnUTs13VSDBBpDfJ9y9wOodu62R5SjkuxdMZWDIuIYeG5OcNmFOWR615e4yp5JEFcMjwDP252eAj0ZJ/yjwa2NMWq+xK2K5Ud5tujyqoCS7PVw5u5gp2Rmuax1ZFVwoL8hm6cxCT6sUrIJr9Qbcu+KBYQtJInbZA0ENXkWhNTDn1WWfCO6cX872k+1RF3ZLBj6fcNf8ct4+emmUG89ppHx2A9w/FOA9l3WoAsbgs5X29KIcz0bKafAbFnr3+gKBkcbHUTReStsnwtKZRZTkZbo+0+lVOPd510PZBjd29y6q4Gh7YJTFnEz3al6WNZ4Rzbi5284nb0VrCXP3gqm8d+LKqAmG4TLfs6iC9p4BdruEHjs997rqEvKy/O7vMPafGJW7F7rLnEoyEry+0hhzDsAYc05EKqKk/xTwrbBjXxORr2D3OLz2fRaRx4HHASorK2lsbIxL4K6urpiuLRsYpKN3gGd++Sa1Rf7h4wfPWpVi27bttBYkZ4hmfpHhtb2tPFhyZdjq6urq4szZPgb6h2hsbKQ6u5+XTw7w8sZN5GWO1MBjJy0X1LvvvktuhlDeP0hP/xD/+EIjS8tH5D7SZhWyvXv3MHjGT/nAIN39Q/zgxUaWlI1Ot2/vXobO+KnJsZ77q42byA96bktrH2Difg/xUtQ7SP9QgH98cRO3Th0pvvsvW3Lv3v0+Paf8IdfE+s7dqAgMcqW7nx+GlYN9l6xy8P6u96kq8pHlhx+/uRvf+eyQ6/v6+zl39hxvvdXG/CmDbDp0njfe3ITfN5KXp0/3EzABurq6OLTrPWZP8fHC1qMsJnRnr/arvRjD8G+pKvTxwtZjLCF0/asrbb0MBuCdt99iQWGANw6cZdOm9hCLvqW1j6GhQc4c3EFxtvCz3xyksvtEaL5193D5cu/w86b0DDFk4PsbGlkekvdWXux+/326m0PzPh5m+wd451I///bym1TkjdSxg2essr5161Yq8nxU5AnPbz7M7L7mkOs7O3vJFqvelF4fpG8wwFMvhJaXE1ftcr5vH/4Lh/APWD2oH762nWvzs0LuZwycOn2KLe+eY34RvLLnNPcVhxpply9fp7t37PXBrWyWeMicSqI+VUReB6a5nPryWB4kItOBZcCrQYefAM4DWcBTwBeBr7pdb4x5yk5DfX29aWhoGMvjh2lsbCSWa5d19fHdva/TM2UODQ3zh4937D4De3ezdu0a5k4tiEuGcFpyTvHfXthP9bI11JTnD8s5bVoJRzsv09DQQM6cK7z01Hv4pi+mYenI6zgix+HIYe6+607ysjJY0z/Ik3te41r+DBoaFg+nyz5+Bba9x4oVy7ltbjlr+gf5zp6NXM0NTZdzwk63fDm3zSsnv7qNl767BSoX0XDr9OF073QdRFpOxpSXyWTdwBD/sPs1rubMoKFhyfBx39FLsGMbq1aupL66NOSaWN+5G7d29/OP+zbSmT+bhoYFw8fNkYuwczt1datYOaeEO05v5/ilrlHPyXx3IzNnTqOhYRk9Zed4+8e7KKpdHiLj5p5DZLQ2U1CQS0NDAx+6fph/fPsEq9bdTmFO5nC67xzZAkBDw3oAPtx3hO+8dXxUuu8dfY+hgKGhYT2XClr4y+f3UrmwjiUzCkfypPMAmRdaueeee7jv0h7eOHyBO++6O0Rh5e5spKKikIaGVYCV99/e9QodHnm/0iXv46Hqcjf/eriRvtK5NKyrGj5+aUcL7NvL+nXrmF2ax8Md+/npjlbW33En2RkjCulb+9+F69a7WDcwxP+35zXasqfT0HDLcJqi0+3w3maWL791uJf2g2O/oXUAGhpuD5HHvPIraqqraWhYwOnsZr7y4gGql66m2q6rAD86tYPrV3tpaLhzTL/VrWyuGxjiyT2v0Z4TKnMqiWryGmPuN8Ysdfl7EbhgN/hOwx8ptvETwC+MMcOjrMaYc8aiD/gnYE1iPyd5lDnuG4+Zockc5LxrfjnAKHdS8ODfqjkl5Gf5R/nXRwZdrYR5WRmsnFMyKuQxPNY8LyuDVVXFo9OF+YtXzi5mSk4G7zalhzspJ9PPmprS0Xll/09mVBJAaX4Wt84sGj0GFJZPdy+YSvOVHprDokmC3TG3zyvH75NRrgjLJz4i+N0LpjIYMKPWvgp37dw+r5yhgGFbWLpAUFD9XfY40ahyYww+u3DdtaCcqz0DHAgPbghycYGV9wtLfKPKQrLzvrosj5nFuaPCccPH9xoWTqV3YIjtJ9tHp5MRmdfPHT1Q7RascPu8cva0XA0JBBkO2rC/O2Gro91J8S+iF06OM7g+gS7cRH0hG4DH7M+PAS9GSPtp4CfBB4KUimCNT+xPUJ6kcse8qbx/uj1kItF4DLpWleUzuzR31BiC4wcGa8r8+rmWvzs4EmVkQC5Y7nIOnO0MCbMM9s863D63nIPnwtOF3i/D72NtTZlnzPhEcPeCqRy72MVZlwHT8eC2eeXsbrka4lsPHmNwZILRDbDjnwZ7j9/ZxaMalYAJfS8r5xSTk+lzVe7BDfWqqmKyM3z85nh4upH7VRbmWAP2rs+0Et0+zzFMwsvf6MbuljI/Ry90cSFodvJIeUxO0ygi3Da3jC0nroSM7TjGjSP3utoyMv0y6veHy33nfGugOniAPThKz+H2eeUEDCEKOTBcb6x0VWX5VJfluSj35Bold84v58Tl7pSV8XASVQxfBx4QkWPAA/Z3RKReRL7vJBKRamA28FbY9T8WkX3APqAc+NsE5Ukqd80vZ2DIhGxuExz/nEzunG8NOAWvRxMcRw5w94JyWtt7aQ6afekWJXX7vDKMgS3BcrtEjtzmls7+7wupMGWcutITsg/ARM6/caK0gq34ZDdOwdw+t5zBgGFb8+gGw7H0q2wrNzxiJTjiCCwls/9MR5hVGvperDDSUvd7Bd3MSbe5KTRdeA/krgVT2dEcOmAf3HiWF2SzZHqhay8svJjfYo9bheS9/T+ZVeK2eWVc7Rng4LmR/bcDYWU9LyuDFbOLRxkt4bOQb5tbBhCSn26GkptCdls59rZ55Ww72TZqTaOk/v655aNkTiUJKQZjzBVjzH3GmPn2/zb7+A5jzOeD0jUbY2YaYwJh199rjFlmu6Z+zxjjvZLVBFBXXUJOps91obFkNz93zS+nq2+Q90+PREWEV8z1dmF5L6TBD3URAdw6q5iC7IyQAh5u4Trp8rP8rulCKoL93PAKmGy3TawsqCygYko27zSNb+PkUFdVQpbfF9awhOaTiLCutoz3wq1cE7qGzvraMgKGEPdPIMwAACvPj1y4FrKKqjGhPQawGtDR6cAXVLPXzy2jfyjArqCNoQyhyujOBeXsPNUe0isKjpZymD3FR1l+lmsIbDJxylxoI23994Xl577WUPdPeH4urJxCSV5myPsLV+xgKdo1NWVh9cF55sj91teW0dU3GLLaarLtpEXTLJknqqeuM58j4BQUN0sj2Q3Q+tpyRAgrvKENwdyp+Uydkh2mGKz/wQU30+9jXW1paKVitNyZfh9ra0PdRG73W1BZQHlBVmg+pHwZvRGcRnjriSujJniNh67KzfKzck5xyKY84dYrWA1we89AyByEcBfDyjnFZGX4QnppbnIPW7knRjdmwdw+rLRDlXtwg7e6uhS/T0YptnD348CQCdmLxM2V5BPhtnnlvNt0eSTvgyaLJYvKwhzmVRTwm7D6EP6c9XMt98/2sD1GgtP4fML6uWVsOT4is3ExlADumFfGsYsjrjI3D8G62tHvJryXlig+34ihMRGrDKhiiMLamlKOXLhGm+2HH693VJSXyeJphWw9GdpIBxe1YKt0uIAHnQvmtrnlNAe5f9x8qla6Mk4G+TJHft5IOhFh/dxyNgdVrIlmXW0ZF6/1jbjVxlms2+1xm5H1dEL93WApBghzWTB6ALduTsloAyCshVo6s4gpORlsOR6q3MMt+KUziyjMyQhxJ4X3NAuyM1g2syjEoHAWyHOoqyohwyehbtOgQdyQvJhbxqVrfcNLk4yXe/X2uWVsP9k2vDWqW1lfOccaZ9kclp/hoqyfW87ZjuucbusJuVf477stTNG6GYJTp2Qzv6Jg1HtOtrF429wyzlztHZY5lahiiMK6Wiv8bpvdYIdHASWTtbWl7DrdPlwRwnsMjjwXOvuG11Jxs6LAWhETRtw/xsOqcwYenXRuriSwCumFzj73dYomgLX2e3Eau/FqnBxum2uNxzjPc+sxzCzOZU5pXpiVP9rqXj+3jEPnR5RMuAEA1hpG62rL+E3w+IFLg+ek23witDcTng/rasvY03p1eHZ2+EB2XlYGy2YVsTXI8nZzXYG1cRIwvBOgV9lKlNvmldM7MMT7tgvMzd+fk+mnrqpkVK93VJ7bVv6och6Wcsn0QkryMofzPXjJmZD7zS1jR3N7yH7eyf796z1cuKlAFUMUls20BqTeCwsJHI/2Z21NGdcHAuy1V7s0MKq0Od1YRx4v19b8igJK8jKHu9heDefCyimU5Qe5iVz8uDDistgSZklNFLXloW618WqcHJbPLiYvyz9K0Ybn03rbxeXMmLf8/aMbFUvJjLwbN4V229wyTrf10GJbjOED2Q63zyunpa13JDjAXp8pmHW1pQwMmeG1/t3utbamjL2tV4cHqT06DNTYeR/cu4Xk14l1tWWWezXKO75tbhmHgqLr3HoMc6fmUzEle6SRdXGZgvWuVleXst0ONHAbmwPrPfcODI3U1XGoD47MEzEArYohClkZPuqqSoYtqfF0pTiWWPCzwhue2nKrsIw0iO4NvohQX106HEnjLDPsVhHqq0tGVYTwyje7NJeZxbkhCnKCxp6tZw+PM7SFvJPxGhDP9PtYXV0a1GNwz6f1c8vovD7IITuaxq3HsHxWMbmZ/pDeR/h7ce4FQeUBdwt+tT2pbOQdjpbLGWcIVqThZWatrTx2DVvooxUfWNetqSkdzvtxc6/mZrJoWqFLIz1a0UKQAmH077fcodZ4oTEmqMfn3iM6daWHC53XPYMa1taGug2NW6IEcWTefDz14wyqGGJgbU0Zh8930tEzMK6u7NL8LBZWTgmpvOENhtMgbrHHGYJj1kfLbRXwi8EF3KU5X11dyuk2uyJ4WMKWoilhx6m2lBdSL9bWlHK+8zqnrvSkpAezutraBjK4HIQ3LOHjDOHRP2AZG/XVI+MMxq1rCCyomEJhTgY7T40od7e2Z+G0KUzJyWDbSac3MFqB5GdncOusoqBnmpDIJbC2ufQJw+MMbpa3wzo771vaesfVvbqmuoT3T19lYCjg2Tu+1Va0wa4tN0nW1ZZxuctyw47MiXB5ZpCrzEsZleZnsWjalKDeTPImuHnJnEpUMcTA2ppSjMGyvscpKmn4WbWl7DzVzmDAjIoucVhXaw3+nbjc7Wr5OTiW5LbmNs+xg+B026Okq6+2xjda23s906SSEbfalXFtnByc5R52nh5RjuENS2VhDlVlecNWrrPvQThrqq2gho6eAc80Pp9QV1XCdnsJajclA9Y4Q11VCTuagxpGl/utqy1jb2sH3X2DrmVrSk6mNUg93EPxfsdrapzezBXcfP/JYnVNKT39Qxw82+mpjDP9PlbMLh52k4VHXA3fq9raFWDHqfaI0YVLpheSn+Vne3NbxJVj19aUDistr3slSrDMqUQVQwwsn22FGAZHbIzXIOfamjJ6+oc41RnwrOBraqzCsrO53dVV4XDLjELysvxsP9kWsSKEpMM7nVNItze3jT45Acydmk95QXZIlNZ4Kqvls4rJ9Avbm9sjNhh1VSXsPNU+7GZxk6nOzstdp9sjzpqtry6l6WIXbd39Ea3S1dWlHLvYRXt3v23pu7hIqksZDBj2tF717GmurS1jd8tVrg8MRTQ6nDGsbSfHtywEGy2R3nF9dQkHz3XaSs+9x1BbXkBxXiY7ggwgt5QZfh+rqkrYdjLomS73q6supXdgaNhtOB4Ey5xKVDHEQE6mn5Wzi9nW3Dbu8ftON/ZI+5BrdAmMFJadp9pdQxgdMvw+Vs0pYVtzu+tEuOB0K+cU2w2ed7oFFZbLIjjdRCIi1FeVsOu0+9aUySY3y8/SmUUhDYtb41pfVcqV7n5OXelxdesArJhdjN8n7DzV7pkGRhrGnaciKxAnnWNZusm1ao6tjE61e5attTWl9A8G2NNyNaIicgZptwYZE+NBZWEOc0rzQqx3t7yqqyphKGDY3XLVczDf57PKyw673lj3cn/uaqdH1zvgma6+yjbQnHcz5l8XHZ9PqJtToj2GdKWuqoSDZzvpsSM2xsswnTolm6qyPJraA4C3i2HVnBJ2nm4PWTDNjdXVpRw+3zk8MzTcrxyc7tD5Tjp7Bz3TDVesNOkxgPVeTrf1cMlll7LxYHV1KXtaOrg+YLsPXDK/PthlgfvrycvK4JYZhdaYjUcagFtnFZHl97HDNkq8FIiTznEHuslVlJfJ/IqC4d6M260c5bHzdHvU2Pw1NSNjU0RJmwj11SXsaG5nKIL1vqqqBBHY0dxujcV43KuuqpQTl7q50tVvy+ytkI1h2I3nlm5GcS4zinLs9+yujJJBXXWJLXNqyjioYoiZVXNKGAwY9rVa0+DH02Wxak4JTVcDo9bGCaauqoSmi9ZAqJfVA7C6piS0gHtUGaci7LAHOr3S1Tsui57RW5FOBKuqigGG/cvjPe5RX1VC/9BISLHb8+ZNLRgeNI7kjlk1p4TdLZaP2itNTqafZbOK7Abf+/flZPq5dVaR7f7wNgDq7B7WUMBdyZTkZ1Fbns+uU1c95zEE3wusHgiM3/jOmmqrB3bikrVijptMhTmZLKycMlJ+PV1zjpXfZt/LPd3KOZbb0HEfe6Wrqy5lZ/P49RggtNeYKlQxxMjKOXYDdHp8KwHAqjnFdPYbTrf1eFbMYcvuVHtEWVbOtma0OuMCkSqC3yfDPuNoLotUFtJI3DLDspTHu3FycBpDJwzYLZ98PmFV0KBxpEbq+kCAA2c7Iyq0+uoS9p3poLd/KKJVurqmlP1nOujpH/LMh1VVJXT0DtB0scszp1bOKeH90+2evnqHJTMKrbw/7a0kk8HqsAl1kfLTGQz2EmXZTKu8DN/LI2VOpuU2HB5P83ho3ZxizndeH9dVUB2ZVTGkIWUF2VSX5XHqyvhPT2iJD24AAAvnSURBVF9pN/qRKu/y2UX4fcKxi10RK2Rulp/F0wuH5fZK67g2nCUmvBSI47I4c7V3QucxOFgVuDBkxdnxpKwgm7lT8zlhzwD3Utz1do8uUhpHyTRd7Ipoma+usuYXRMvz+iqrV3vmam/EnibAsQjPXFVVzJXufjp6ByIqouwMK+/He8mG2vJ8SvOzht+xp2KoKqWrb5CL1/oi9qyWzSoansEfqe6smlMStT44kWrNV3rGTTE6ZTyV4wyqGMaAY6XD+LosFk2bQrbfeY77g5yGHLwbHgentxPpfmBtyjOCtyV1y8xC13MTRarei4PTuEZ63qrgNB73mV5kTRqMdJ/we0V61ytmR3/PteX5lORlRnzmWPJzZQryXkRCf5unm7PE9XikdJFkDqk3Hs9cNG0KeVn+iGmSwerqUva1dqRsH2hVDGNgZQyVPRlk+H3UFFmvJppFAwwvv+BFaAGPlC64AYqQbrb93IkPTAJia6iTSWg+uT8wuCELXxIjGEf2SGKX5mdRXZZnpYuQsKzAClyw5HJPIyIjz/S42YLKKRRkZ9hyRc7QECUyjrUi2Gjx+m0zi3OpLLT23I7UsNXFKHMs9cGJ6LNvNm6ssse2Dpwdv9DYYBJSDCLycRE5ICIBEamPkO5hETkiIk0i8qWg4zUislVEjonIcyKS5XWPdCC4QI23H2VesWWF9A0EPNM4Fbw3ihXhNOQQpccQY89ihZ2ud9AzSUoJtc7HXzOEWq/u5GVlsKAy+p7gq+y8vNAZOeLEeWa0X7cyhnROfnX09Lue9/uE5bOLgMgGAoSXmSjCJcCKGMpmSM8igiwrYpR5RlEOFVOyo6ZzlON4WvPOe93dkprQ7ER7DPuB3wbe9kogIn7gSeADwBLg0yLi7CT+DeDvjTHzgXbgcwnKM64snDYlZc+aV2y9moMRJs8EN4iRqCrLG3YfRKroc0rzKM3Pipou1OU08VQW5qT0eQsqR8pBJAV666zo+bRiTozK3U53PmhLTdf72e8mksvJacjOdnjfa1WMcs0ozmVaCvI/OC8jlc0VthHUPeDdna2YMiJvpHwKVjQRDSU7zf6gjXuSTUVhDjOKciaHYjDGHDLGHImSbA3QZIw5YYzpB54FHrH3eb4XeN5O9wzWvs9piz+oRI63ZTrX7jFEYkZRbBUypIBHkDvWdLNKcmN6bipxXk0qXEkh5SDC85bbeXkiwlLli6fHZmw472VPS+TGZ9j9EUGuZTOLoj7PUQyxNHZOr2E8s74oN5N5FVYPLJZGurnDu6cdTKxjKJGSOc+M4tFNmBVzitndkpoB6FSMMcwEWoK+t9rHyoCrxpjBsONpjWMdjffM3ylZ0avZWCbUDBfwKJeMWEjJeW6qcGaMB29NOZ5MtV0MkYrBCtvKPd/pHcqYnRHdAABYPN0a8I9mwS+eXkhWhi+iYs+3xw8i4ZSD4C1DvVgVY9lKlBUx9FRvnWUpvc7+yPXTKS/B+2C7Maz0Ivy4soLsqHIlgxWzi2lp603JRDeJ1sCJyOvANJdTXzbGvGinaQT+whizw+X6jwMPOXtAi8jvY/UivgpsMcbMs4/PBl42xizzkONx4HGAysrKumeffTamHxhOV1cXBQXRfb9eXOoJ8PaZQX57Xua4NpBdXV0c6srBJ1BX6V2Rj7UP0XItwL1zMiPer/16gNdPDfI7CzIjdp9jTXesfYimy718YH78eZlMOvsMv24e4NH5mSEWPST+zt241BPgnTODfCxCOTDGsOH4AGunZzAt39sG23tpkK4BuLXwekQ5G1sGmJ7vY2FpZGUSS7r3Lw4yEIA107zL1isnB1hc5qOqcOQ+bnkZKe+TyenOIfZfGeKDNZGHIjeeGmB6Vh9Lp3vnZUef4ZUYZB4MGF5oGuD+ORkU53i/w23nB8kQWBWhrroxlrJ5omOIX58c4BMLspiaF59Nf8899+w0xniOBw9jLfSV2B/QCNR7nFsPvBr0/Qn7T4DLQIZbukh/dXV1Jl42bdoU97WpZDLIORlkNEblTCaTQUZjVE4vgB0mhjY2Fa6k7cB8OwIpC/gUsMEWchPwqJ3uMeDFFMijKIqiRCDRcNWPiUgrlrX/KxF51T4+Q0ReBjDWGMIXgFeBQ8C/GWMO2Lf4IvDnItKENebwg0TkURRFURJnbA6xMIwxvwB+4XL8LPDBoO8vAy+7pDuBNd6gKIqipAk681lRFEUJQRWDoiiKEoIqBkVRFCUEVQyKoihKCKoYFEVRlBCiznxOR0TkEnAqzsvLsSbWpTuTQc7JICOonMlkMsgIKqcXVcaYqdESTUrFkAgissPEMiV8gpkMck4GGUHlTCaTQUZQORNFXUmKoihKCKoYFEVRlBBuRsXw1EQLECOTQc7JICOonMlkMsgIKmdC3HRjDIqiKEpkbsYeg6IoihKBm0oxiMjDInJERJpE5EsTKMdsEdkkIodE5ICI/Cf7eKmIbBSRY/b/Evu4iMj/teXeKyKrUiirX0TeF5GX7O81IrLVlvE5eyl1RCTb/t5kn69OoYzFIvK8iBy283R9mubln9nve7+I/EREctIhP0XkaRG5KCL7g46NOf9E5DE7/TEReSxFcn7Tfu97ReQXIlIcdO4JW84jIvJQ0PFxawfcZAw69xciYkSk3P4+YXkZlVg2bbgR/gA/cByoBbKAPcCSCZJlOrDK/jwFOAosAf4O+JJ9/EvAN+zPHwR+jbW50Tpgawpl/XPgX4GX7O//BnzK/vxd4I/tz/8B+K79+VPAcymU8Rng8/bnLKA43fISa9vak0BuUD5+Jh3yE7gLWAXsDzo2pvwDSoET9v8S+3NJCuR8kJHNvr4RJOcSu45nAzV23fePdzvgJqN9fDbW1gOngPKJzsuovyOVD5vIPzx2kptouWxZXgQeAI4A0+1j04Ej9ufvAZ8OSj+cbpzlmgW8AdwLvESEXffsQr/e/pxhp5MUyFhoN7gSdjzd8tLZ+7zUzp+XgIfSJT+B6rAGd0z5B3wa+F7Q8ZB04yVn2LmPAT+2P4fUbyc/U9EOuMkIPA8sB5oZUQwTmpeR/m4mV5JTMR1a7WMTiu0iWAlsBSqNMecA7P8VdrKJkv3bwF8BAft7GXDVWJsvhcsxLKN9vsNOP97UApeAf7JdXt8XkXzSLC+NMWeA/wWcBs5h5c9O0i8/Hcaaf+lQvz6LZYETQZ6UyykiHwHOGGP2hJ1KGxnDuZkUg9uO3xMakiUiBcDPgP9sjOmMlNTl2LjKLiIfAi4aY3bGKMdE5W8GVtf9O8aYlUA3luvDiwmR0/bRP4Ll1pgB5AMfiCBL2pVXGy+5JlReEfkyMAj82DnkIU9K5RSRPODLwFfcTnvIMuHv/mZSDK1Yfj6HWcDZCZIFEcnEUgo/Nsb83D58QUSm2+enAxft4xMh++3AR0SkGXgWy530baBYRJyd/4LlGJbRPl8EtI2zjM5zW40xW+3vz2MpinTKS4D7gZPGmEvGmAHg58BtpF9+Oow1/yasftmDsx8CftfYvpc0knMuljGwx65Ls4BdIjItjWQcxc2kGLYD8+0okCysAb0NEyGIiAjW/taHjDHfCjq1AXAiEB7DGntwjv+BHcWwDuhwuvnjhTHmCWPMLGNMNVZevWmM+V1gE/Coh4yO7I/a6cfdyjHGnAdaRGShfeg+4CBplJc2p4F1IpJnv39HzrTKzyDGmn+vAg+KSIndO3rQPjauiMjDWHvHf8QY0xMm/6fs6K4aYD6wjRS3A8aYfcaYCmNMtV2XWrECT86TZnkZLvhN84cVBXAUKyrhyxMoxx1YXcO9wG7774NYPuQ3gGP2/1I7vQBP2nLvA+pTLG8DI1FJtVgVrAn4KZBtH8+xvzfZ52tTKN8KYIedny9gRXKkXV4CfwMcBvYD/4IVMTPh+Qn8BGvcYwCr4fpcPPmH5eNvsv/+MEVyNmH545169N2g9F+25TwCfCDo+Li1A24yhp1vZmTwecLyMtqfznxWFEVRQriZXEmKoihKDKhiUBRFUUJQxaAoiqKEoIpBURRFCUEVg6IoihKCKgZFURQlBFUMiqIoSgiqGBRFUZQQ/n/PSTGGbJBzJwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1815858da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# lets see what this gives us:\n",
    "plt.plot(process_rewards(rewards_list),'-')\n",
    "ax=plt.gca()\n",
    "ax.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you can see that in the above chart we got better distribution for rewards. reward is -1 where it was -1 and now steps before it has some value near -1 (for example -0.9) and similar process happens for positive rewards. \n",
    "\n",
    "further more we refine the function to subtract rewards by mean and divide by std, and then discuss the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_rewards(r_list):\n",
    "    reward_decay=0.99\n",
    "    tmp_r=0\n",
    "    rew=np.zeros_like(r_list,dtype=np.float32)\n",
    "    for i in range(len(r_list)-1,-1,-1):\n",
    "        if r_list[i]==0:\n",
    "            tmp_r=tmp_r*reward_decay\n",
    "            rew[i]=tmp_r\n",
    "        else: \n",
    "            tmp_r = r_list[i]\n",
    "            rew[i]=tmp_r\n",
    "    rew -= np.mean(rew) # subtract by average\n",
    "    rew /= np.std(rew) # divide by std\n",
    "    return rew"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note that we normalized the rewards, subtracting by mean and dividing by std. this may not be perfect but works for this project.(more discussion after the plot)\n",
    "\n",
    "lets see our reformatted rewards: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXl0W9d9578XAAHuJAiSECnui6jN1kJqoXZ5t+MmcZImTidp6qT1pJ3OaZJpq7iZ6ZxumaSdk2nauE3SSaZZmsiuYzeOHFu2LGqztVGydmqhKJKiFu77DuDOH+898AF47wEEHoAH4Pc5h4ckcHHfD/fd+72/+7vLY5xzEARBEMmDKd4GEARBEPpCwk4QBJFkkLATBEEkGSTsBEEQSQYJO0EQRJJBwk4QBJFkkLATBEEkGSTsBEEQSQYJO0EQRJJhicdFCwsLeVVVVVifnZycRFZWlr4GRYFEsDMRbATITj1JBBsBslONM2fODHDOi4Im5JzH/KexsZGHS0tLS9ifjSWJYGci2Mg52akniWAj52SnGgBaeQgaS6EYgiCIJIOEnSAIIskgYScIgkgySNgJgiCSDBJ2giCIJIOEnSAIIskgYScIgkgySNiJiOCc49WzPRifmY+3KQRBiJCwExFx8GofvvLyeXznYHu8TSEIQoSEnYiIk7eGAACZ1ricTkEQhAIk7EREnLs9AgAozU+PsyUEQUiQsBMRIQk7QRDGgYSdCJuRqTnMuTzxNoMgCD9I2ImwOdM17P2bx9EOgiB8IWEnwkYu7ARBGAfdhJ0xZmaMfcAY26dXnoSxae0aRmG2Nd5mEAThh54e+x8BaNMxP8LAzLk8OH97BE2VBcILFIshCMOgi7AzxsoAfAjA/9UjP8L4XL47ilmXB01V9nibQhCEH3p57H8P4E8B0BKJFEGKrzdWCsLOyWUnCMPAhMfoRZABY08DeIpz/geMsV0A/phz/rRCuucBPA8ATqezce/evWFdb2JiAtnZ2RFYHBsSwc5IbPzOBzPoGvPgqxvT8d8OT+O51VbsLEvT2UKBRChLIDHsTAQbAbJTjd27d5/hnDcFTRjKg1G1fgD8LwA9ADoB3AcwBeCnWp+hh1kbg3Bt9Hg8vOmv3+Ff2vsBvzM8xSv37ON7T3Xpa5yMRChLzhPDzkSwkXOyUw3E6mHWnPMXOOdlnPMqAM8COMg5/0yk+RLGpXNwCv3js94wDABEOPAjCEJHaB07sWiO3xwEADTXOsBYnI0hCCIAXYWdc36IK8TXieTieMcginNsqCnM8r5GDjtBGAfy2IlFwTnH8ZuDorfOwEAuO0EYDRJ2YlHc7J/AwMQsmmsc8TaFIAgVSNiJRXG8Q3iwRnOtr7DT5ClBGAcSdmJRnLg5iJK8dFQUZAIATZ4ShAEhYSdChnOOEx2DaK4R4usEQRgTEnYiZK7eH8fg5Bw21wbG1+lIAYIwDiTsRMgcud4PANi5rMj7GvntBGE8SNiJkDl8vR/Ll+TAmRv44GqaPCUI40DCToTE5KwLpzuHfLx1AOSyE4QBIWEnQuL4zUHMu3mgsBMEYThI2ImQOHy9H5lWMxpVHqxBkRiCMA4k7ERIHL7ejy21DtgsZp/X6UgBgjAeJOxEUG4NTKJ7aEo7DEOzpwRhGEjYiaAcuNILANjVUBzwHu1TIgjjQcJOBOXtK/exoiQX5eIxAgRBGBsSdkKTgYlZnOkaxmMrnZrpKBBDEMaBhJ3Q5GBbHzwceFRF2CkSQxDGg4Sd0OTtK71Ymp+BVaW58TaFIIgQIWEnVJmac+HojX48utIZ9DRHWhRDEMaBhJ1Q5cj1fsy6PKphGAB0fC9BGBASdkKVX124B0eWFZuqC4Km5eSyE4RhIGEnFJmcdeHdtl489UAJLGb1akL+OkEYDxJ2QpEDbb2Ymffg6QdL4m0KQRCLhISdUGTfhXtw5tqwoSp4GAagdewEYSRI2IkARqfncfhaP55+sBQmk3awheZOCcJ4kLATAey/dB9z7sWFYWjulCCMAwk7EcC/n7mNmqIsrC3PD5qWju0lCONBwk740NE/gdOdw/jNxnJao04QCQoJO+HDK2d6YGLAx9YvXdTnKBJDEMYhYmFnjKUzxk4xxs4zxi4zxv5CD8OI2OP2cPzibA92NRTDmZse2ofIqScIw6GHxz4L4CHO+RoAawE8wRjbrEO+RIw5cqMfvWOz+M3GsnibQhBEBFgizYALe8knxH/TxB8amScgL526DXtmGh5eoX32uhJ0pABBGAemR4NkjJkBnAFQB+BFzvkehTTPA3geAJxOZ+PevXvDutbExASys7MjsDY2JIKdchsHpz34kyPTeKIqDZ9ssIacx9Q8xx+8O4VPL7fi8aq0qNtpZBLBzkSwESA71di9e/cZznlT0IScc91+AOQDaAGwWitdY2MjD5eWlpawPxtLEsFOuY3ffLONV391H789NLmoPEan53jlnn38X47c1Nm6BRKhLDlPDDsTwUbOyU41ALTyELRY11UxnPMRAIcAPKFnvkR0mZl3Y+/p23hkhRNl9sU915TmTgnCeOixKqaIMZYv/p0B4BEAVyPNl4gdvzp/F0OTc/idLVXxNoUgCB2IePIUQAmAH4lxdhOAlznn+3TIl4gBnHP86Hgnljmz0VzriCAf/WwiCCIy9FgVcwHAOh1sIeLAe+2DuHRnDF9/5oGwdprS7lSCMB608zTFebGlHcU5Nny8cXE7Tf3htMKVIAwDCXsK0z7ixvGOQfze9hrYLOZ4m0MQhE6QsKcwb3TMIy8jDb+1qSLsPCgQQxDGg4Q9RekcmMQHfW48t7UKWbbI59Bp8pQgjAMJe4pyd3QaALCpOvyVMAA9QYkgjAgJe4pDwkwQyQcJe6qic+iEIjEEYRxI2FOcSB12ejQeQRgPEnZCF2jylCCMAwl7iqKXDlOMniCMBwl7ikNHAhBE8kHCnqLoHTqhIwUIwjiQsKc45LATRPJBwk4QBJFkkLCnKHqHTmhVDEEYBxL2FCfidewUyiEIw0HCnqKQh00QyQsJe4oTqcdNO08JwniQsBMEQSQZJOwpit6RGE6xHYIwDCTsKU9koRSaPCUI40HCTugCOewEYRxI2FMUvUIn5LAThPEgYU9xKJRCEMkHCXuKovvkqc75EQQRPiTsKU7kO0/J5ScIo0HCThAEkWSQsKcqep/HTrEYgjAMEQs7Y6ycMdbCGGtjjF1mjP2RHoYRsSHSUAoFYgjCeFh0yMMF4L9xzs8yxnIAnGGMvcM5v6JD3kSU0P3YXpo+JQjDELHHzjm/xzk/K/49DqANwNJI8yViAx3bSxDJh64xdsZYFYB1AE7qmS9BEAQROky3HYiMZQM4DOBvOOevKrz/PIDnAcDpdDbu3bs3rOtMTEwgOzs7ElNjgtHtPNfnwt+fncWfN6ejJs8cdj6cczy3fwofqU3DM/VWHS1cwOhlKZEIdiaCjQDZqcbu3bvPcM6bgibknEf8AyANwH4AXwklfWNjIw+XlpaWsD8bS4xu54Er93nlnn38/O3hiPOq3LOPf+vtazpYpYzRy1IiEexMBBs5JzvVANDKQ9BYPVbFMAA/ANDGOf9WpPkRsUHv5Yk0dUoQxkGPGPtWAJ8F8BBj7Jz485QO+RIEQRBhEPFyR875MdBy5oSFHm1HEMkH7TxNUXQPndDWU4IwDCTsKY4e69BpLTtBGAsSdoIgiCSDhD1F4TqHTigQQxDGgYSdiBiKxBCEsSBhT1F0f4ISuewEYRhI2FMcfSZPyWcnCCNBwk4QBJFkkLCnKPofKUCxGIIwCiTsKY4eO08pEEMQxoKEPWXRebkjOewEYRhI2AmCIJIMEvYUh44UIIjkg4Q9RaHz2AkieSFhT3F08dhp+pQgDAUJO0EQRJJBwp6i0JECBJG8kLCnOLqEUSgSQxCGgoQ9RaGdpwSRvJCwpzj6TJ4SBGEkSNgJgiCSDBL2FEX30AlFYgjCMJCwpzh6hFFo5ylBGAsS9hSFdp4SRPJCwk4QBJFkkLCnOHSkAEEkHyTsKYr+O08pGEMQRoGEPeXR4QlK5LAThKHQRdgZYz9kjPUxxi7pkR9BEAQRPnp57P8K4Amd8iJigN6hE4rEEIRx0EXYOedHAAzpkRcRW+hIAYJIPijGTugCOewEYRyYXkNyxlgVgH2c89Uq7z8P4HkAcDqdjXv37g3rOhMTE8jOzg7TythhdDuP33Xhexdm8Y3tGViSFVn//sV3JrGzzIJPr7DpZJ0vRi9LiUSwMxFsBMhONXbv3n2Gc94UNCHnXJcfAFUALoWStrGxkYdLS0tL2J+NJUa38z8+6OGVe/bxm33jEee16s/f4n/5q8sR5/ONN9v4yY7BgNeNXpYSiWBnItjIOdmpBoBWHoLGpmQoZt7tibcJcUf3IwUizM/j4fjnQzfxn3/Sinuj0/oYRRApil7LHX8O4DiABsZYD2PsC3rkGw2m59xY/1fv4PtHbsbbFEPAdJg91WPyVOoXhqfm8V9/9gF1vgQRAXqtivk057yEc57GOS/jnP9Aj3yjweScC+MzLvztW9dw/vZIvM2JG3of2xtpflx0+VcvzUVr1zD+99vX9DCLCAHOOT764nv4yYmueJsSNyZnXTjZMRhvM3Qj5UIxUsjA5eH40kvnMDnriq9BccZoSxUfW7kEv7WpAt8/0oFTt2gFbSzwcODc7RH81a+u4HrveLzNiQu/PHcXn/r+Cey/fD/epuhC6gm76Fk+/WAJOgcn8ddvtMXZoiRAh95B8vcZgK89tQLl9kz8ySvnMTWX2h1vLJBGS3NuD77y8rmUDIPNzLsBAF977RKGJ+fibE3kpJywSwqyucaB39teg5+f6saJJBqChYrRJk+lzzMGZNks+LtPPIjuoSl8482rkRtHaCLdujXl+bh0ZwzfOdgeV3vigVQGg5Oz+J+vX46rLXqQesIuwhjw5UeWobwgA3/26kVvj51qGGXnqX+MflONA89tqcaPj3fhxnBq3ptYIXWqj64oxkfXluKfDrWjvW8ivkbFic9vrcbr5+/i7QQPyaScsC8M+RkyrGZ8/ZkH0DEwmZJeihGRr9L548eXoTQvHT++MgdXCoYHYg1jDP/96ZXISDPjf/zHpZQ6iln6rn+wqxYNzhz8xa+uYHoucR2K1BN2v7q6vb4IH1u/FN89fBMd/anjpRitzSrZk2m14M9/YyVuj3vw4+Opu2Ij2shHS4XZNvzpE8txvGMQr5+/G0er4kOaxYS/+Mgq3BmZxj8fSlxnL/WEXazE8hDEC0+uQHqaGV//derFc/V4+pEea+HVeHzVEjxQaMa33rmOvrGZqF0nlfHvVD+9sQJryvLwV/vaMD4zHx+jYox3jgfC/NtH1pbiu0c60DkwGVe7wiXlhF1CLkVFOTb8l911ONDWi2M3BuJmUywx6hOU/PsIxhg+s8KKOZcH33rnui7XIJSRyt5sYvjLj6zGwMQsvn+kI75GxRjJSfmzp1bAajYl7Kq5lBN2+eoLOc9trUJ5QQb+at+VlIrn6jJ5qsdyR6/HFJiZM8uEz2yuxMutt3EjRddZxwJ52a8pz8dvrCnFvxztQG8KjJT8J++duen4/V21ONDWi9bOxNtPkXrCrvJ6epoZLzy5Atd6x/Hq2TsxtYlQDpHJ+cOH6pBlteCbb9GOVL1Rc3b+5LEGuD0c/ycFRkryUIzEc1urUJRjwzffuppwE8mpJ+ziDVLyDJ9cvQQPluXhHw7ewJwrub12vStqpLkFM6cgy4ovih7U6QT0oIyM2nEQFY5M/HZzFV5uvZ30O1K9q+VkspBpteBLj9TjdOcw3m3ri4td4ZJywu5FwTNkjOHLjy5Dz/A0XjnTE3ubEhQ9p0618vr81mo4c234u/3kteuJkrcq8Ye765BpteAf3r0RU5vihb/D98mmclQXZuFv91+Fx5M4XnvKCbtWJQaAXcuKsL4iH985eAOzrsRdxxoM/SdPI/y8+FsrXp9hNeOLO2tx6tZQUh3YFG+0yt6eZcVvN1fijYv30N6XvF67Wv1NM5vw5UeX4XrvBN6+kjiblhJK2H947Bb+z5noTuQwxvCVRxtwd3QGL52+HdVrGQE9Jj5L8zNw6tZQROEdrRCZnGc3VKAw24rvtCTuGmOjolb2X9hWjXSLGS+2JO9R11pzPB96oATVhVn4Tkt7wsTaE0rYGQPO97txpms47DwWJorUBWRrnQNNlXZ8/0hHSq2QCZfPb63Gtd5xtFwLPw4ZanPJsJrxu9trcPTGAM6l8LHLehJMrBzZNny2uRK/PHcHtxJ0XXcwtIrAbGL4/Z21uHRnDIev98fOqAhIKGH/ZFM5stKgy0MytPxCxhie31GDnuFp/PpS4gy/FoWOjseH15aiNC8d3z0U+ZrnUEYQn9lcifzMNHznYGrEfaNNKGGw391ejTSzCd89lLxeO6BeBh9dtxSleel4MUFGigkl7Fk2Cx4qT8PbV3rD3v4fbFmdxCMrnKgpysL3j9xMmOFXOOixazTNbMLvbq/Bqc6hsEdTiynibJsFv7OlCgfa+lL2sCo9CaXsi3PS8ZtNZXjt3B0MTMxG36g4oRaOslpMeH5HDU53DifEqqyEEnYAeLjSgjSTCT84diusz4cqICYTw+9tr8GlO2M4ThN1QXl2YznyM9Pw3cNhenQhhMjkfGZzJaxmE370fmd41yMCCFb2z22txpzLg3870R0ji2JHKM7bJzeUIy8jDf/6Xmf0DYqQhBP2fJsJH1u/FK+c6cFgGJ5DKMNOiWfWLUVhti0pt1Xr/Wi8TKsFn91ciQNtvbg9NKVr3koUZtvw4bWleOVMD0anUuM8k6gRYlWoLcrG7oYi/OREV9KtGFPbpCUn02rBsxvK8dbl+7g7YuwHriecsAPCLP2sy4OXWsNftRLK4VfpaWZ8dnMlDl3rT9jDgIKh5xr039pUARNj+OnJxZ/E6A2RLeIzz22twvS8Gy+1Jp8HGUsWU/af31aNgYlZ/Or8vegaFSeClcFnNleCc46fGvz5sAkp7PXOHGyqLsDPTnYvetOAd1ldiAry7MZymE0MPzuVXOIRjWmDkrwMPLrCiZdP3170g0tC8Zj8WVWah03VBfjR+120eikCFlP22+oKUV+cjR8eu5VUc08LI3ntQigvyMSjK534+aluQz+cJyGFHRB6zp7haRy+sbjlR4utis7cdDy20ol/b128WCUCep+4+9vNlRiemse+C4vz6MKViOe2VuPOyDQOXk2sLd9GJJSqwBjD57ZU4cq9MZzvGY26TbFiMX3U72ypxvDUPF4/Z9zz6hNW2B9ftQSF2bZFT+SE42R8ZrMgVm9eSs7hp5401zpQW5SFnxzvDOvzi+1nHllRjOIcG16OICyX6iy2SXxkbSky0szYm0Sj2MWEozbXFKC+ONvQo/iEFXarxYRPbSjDwau9uBPGRMZilvltqXWgpjALP02i1QDRGkQzxvDZzZU43zOKS3dC9+gWQmSLk3aL2YRPNJbh4NW+lDheNhostuxz0tPwoQdL8Pr5u5iYdUXTtJixmHAUYwyf2lCOc7dHcGfcmCHAhBV2QNhezoFFbv1f/CQdYwy/takCZ7qGce1+cp2XoccTlPx5Zl0ZrGbTog5SW8xqJX8+2VQODwcd3BYm4ZT9pzeWY2rOjX1J9vi8UDu3Z9YthcXEcOSOMVdkJbSwlxdkYltdIV492xPyJGq48z3SjXz1bHKIRzTnvfIy0/DoSid+ee5OyMcfR2JPVWEWNtcU4OXW2wl1Ap/RWEyfur7CjvribOxNkvOUFltrHNk2PLLCiffvugx5xHdCCzsAfGz9UvQMT4e8Gyxcz9CRbcOuhmK89sEduJNIPKL1uNJPNJZheGp+0ZOa4ZrzqQ3l6Bqcwslbxt8VaDTC6VTl4YikGMWGUQif2lCO8Tng4NXeKBgUGQkv7I+vWoJMq3nRTz0KJwTx8fVL0Tc+i/faU+O5qJGwvb4QRTk2/CLEEQ5HGOsdZTy5ugQ56Rb8+5nk8CBjSbhl/8y6pTCbGF77IPGfOMax+Kq3Y1kR7DaGl1uNN4rXRdgZY08wxq4xxtoZY1/VI89QybRa8OTqErxx8V5IyxEjGfI/tKIYeRlpSRGO0XvnqT8WswkfW7cULVf7QjtbJMg5+cFITzPjydVL8Pbl3qRclhpVwix7R7YN2+sL8avzd5MiBLbY7282MTSXWnDkej+GJueiYlO4RCzsjDEzgBcBPAlgJYBPM8ZWRprvYvh441JMzLqw/3LwkxhDPQRMCZvFjN9YU4K3Lt/H+IwxJ00WS5QiMQCAjzeWweXhMVvv++E1Qj1ooTXtYRFOm/jo2qW4MzKN1giO0jYC4Tp8m0vMcHk4fn3RWEuh9fDYNwJo55x3cM7nAOwF8BEd8g2ZzdUOlOalhyQgwZ6gFIxn1pVhZt6Dd64YL65mNJY5c7CiJBdvhFDpI1kVI9Fc60Bhtg2/NPDGkVjycutt/NlrF4Omi8TXfnSlExlpZvzyXGKHYzh4WCedlueYUFecjdcNtjpID2FfCkAe2OwRX4sZJhPDkw+U4OiNAYyF6EmHKyDrK/JRmpeuWw89NefC13/dFpLdem7hjtVu8KcfLMGZruGghyYtdLjhK7vZxPD0gyU4eK0v5HoQa2K5Df/99gH87GQ3eoa1D2WLpOyzbBY8utKJNy7ei/nqkH861K7bxC3n4Tl7jDF8eE0pTncO4d6ocQ4Gs+iQh1J5BNRextjzAJ4HAKfTiUOHDoV1sYmJCcXPlsy7Mef24B9/cQhbl6apfr57TIi/Xrp0GekD4T0U+YF8F9692odfv9OCzDTl6qBmpz9tg258//QMpvp78Eilut1uD8eeo9P4eL0VzaXat+3ygBs1+SZkWNSr6vVuQfjeP/4+8m3Rm0N3TAmN/R//4xger1L/fkMzQrrr16/h0LTvaZqhliUAlHvcmHN58O1XDmF7mfr19GTOzXFl0I3ajBlNOw92z+Ng9zz+cmsGTNFajiTjfq+wYevFX77nLXulstQq+1CoMbvw+tQ8/unVg1hbrIekBL/nHs7xt/un0HKuHf9lbXrE1+vungPnfNG6NDExgWJ2G5wD3371GJ6ojk2dCwrnPKIfAM0A9sv+fwHAC1qfaWxs5OHS0tKi+Lrb7eHNXz/Av/CvpzQ/f+nOCK/cs4+/efFe2Da0dg7xyj37+Ktnby/aTn+O3ejnlXv28We/d1wz3fjMPK/cs49/6nvva6abnnPx6q/u43+977Jmuh8f7+SVe/bx3rHpkOyMhKe+fYR/9MVjmmnujkzxyj37+M9OdgW8F2pZcs65x+Ph2775Lv/cD08GTTs2Pcc9Ho9mmjmXm0/PuTTTvHnxHq/cs4//0ysHNNP95a8u88o9+/iJmwNBbdOD//qzs7xyzz7+jKzslcpSKvufK5R9KMzOu/nq//kW/+OXz2mmm5p18S/+pJV3DUxqpjtw5T7/z/+8XzPNnMvNK/fs4yv+x5tB78/MvIvPzGun+cabbbzuz97QTKOEVJ5P/8NR/hv/eHTRn18sAFp5CLqsh6t2GkA9Y6yaMWYF8CyA13XId1FI4Zgj17XDMeGcIujPuvJ8lOSl440gB10NTc7htQ96NIff0lsnbw1qni8v5XHq1pBmOpeHw8OB/Zd7Qxr2R2PnqT8ferAEH3SPaIYEIp37kGCM4YlVS/B++6DmBPfkrAsb/+bdoLtVv/qLi/jcD09pppHOJj/Tq729XvqObwWZ5L89NIW3Q1gIEAyPeMGz3SOaYYJIo0NWiwkPLy/GO229mqds3hqYxJuX7gc9Y+VAWx/e6pzXPCpEsnlqzo0jQZ5D+vyPz+ArL5/XTCOEYsKvfR96sAQXekbDOt4kGkQs7JxzF4A/BLAfQBuAlznnlyPNNxyeeqAEc24PDoQwsRmJgJhMDE+uDt6JvHHxHr780nlcujOmmkZqfB4OzQlZaTWZhwMH2rTSCQm7h6Zw1SAbRz70QAkA4M2L6mKlx+SpxOOrlmDO7UHLNfUGPznnwvS8O+ikV//ELE7e0o6fSiJzts+t2ZlK92b/pfua6X56ogtf/OkZzQ7c4+HoG9c+G4cDyEgzAwDe0nh2rx5l/8TqJRiZmscpjQ1i0oq0t68E67SEdO9odG4eWflpfTcA6B+fxTtXejXPteEIM8gu8thKJwBtm2OJLsFVzvmvOefLOOe1nPO/0SPPcFhXLkxsvhmDB1B/6EFRPDSW1kneyzsaQixvVJqenEwHNBupLJ3m8s8YTuJVOrKwsiRXs+OShE6PEcT6CjsKs22a31/6+sdvDmJ0WmuEJyTUchYkwRqa4bgYwsFnd0dnNNPNu4VRl9au3QNtvdj6jYPoHtSYGOXAUnsGli/J0Zzs16PsdywrQnqaSbMOS2Xe0T+p+axaj+j0vx2Kg8aEstCauPVwjjmXJ6hnH0nNqynKRn1xdkg2x4KE33kqx2RieGSlE0dv9KtuUlkIxUQmIOvKBfHQFivhdyiC1lRpx3vtA6oiI3koWVYz3tMIM8g9wf2XQ2sYseCRlU60dg3FZCOHycTw6EonDl3tC1oPXB6OQ9fUBVRKp9VgPTJN0epMPJzDajHBbGLanY7ksWpcc3hqDvNurun9ejgHg7Art7VrOPhGsQjqQqbVgp3LivD25V7VzUqhOh3S9z95awjDKvVFyqup0o6xGVdIzyXWXKKsg5/z2Cqnps2xJKmEHQAeXuHEzLwH799U3vYfziPYlDCZGB5aXoTD1/sxrxJXlMS47d6Y6nNApQr65OoSzLvVRUaqdw+tcGqGGaT8yuwZ2tdVfDV6PLKiGB4O1RGOt9Hr1NE8vsqJyTm3aj2QD+U1RVtMd6JjUDXsJuVUmME0O1POgWybBZtrCkIadR29MaDaMUna+W6bdqfEGPDwimJwDhwKUmci5fFVS3B/bAbne0aUryPbHKhd5kI1cHu46qhFui876ouQZTUH7VAB4N22XtW2Gs6RAv48tnKJps2xJOmEfXNNAbKsZhzQqPCAPp7qwyucGJ9x4XQIB0+9qxKOkSr7uop8OLKsQStyY0W+5khBSvf4qiUAgoRjEN2dp3JWl+bBmWvTnB/Q054ttYXIsVmw/5JauQtkhLPfAAAgAElEQVRkWs04fK1f9eHMnAtphE5XWRi998ZpRnvfBG72K4cZOAQP+vFVS3CzfxIdaunE/Kbn3arnEklifLpzSHOUZ2IMq0pz4cy1BT2sKtKyf3i5E2YT06ibwu/GCjvO3x7B/VHlOQLOAXs6w5LcdNURiXT/0tPM2FZfiJarfarzFtI9HNNoq5zziMOADyzN07Q5liSdsNssZmyvL8LBNuUbrWdoeXt9IawWk2onIjX4krx01Ti7NIw3mxh2NggjAKXTIyW7zWYTdjUU4YhaOvF3lSMTy5zZhvAeAGGE8/AKJw5fVw+T6YnVYsKOhiK0XFOuB1K4YFdDESZmXTjRodzgPZxjVWkuCrOt6kN5Mfv14hputVGJhwshwN0NxQDUPWgPB3JsFmTbLEE7cJeH47BK7Hhh/obhoeVOHLk+AJdG3Yo0PJmXmYbGSrvGyEC40BOrBadDrU1wzmFiQmhDrb5I358x4KHlxbg3OqO6WMDDOTZVF8BmMWmOFCJ19kwmpmlzLEk6YQeEoef9sRlcvhu4GkXP1ReZVgu21Drw7lXlpYXSS4+tdOJkh7JnJX3KxBgeWl6Mkal5fNAdeO6GPIS0u6EYo9PzOHc7MJ1noZVid0MxTncOKa4GiOHcqZdHVzgxNefGCYV4qF7iImfXsiL0jc/iyj31VUlb6wqRkWbWGFEJne7Dy504dK1PcTmfVOaFGQz1xdmaIQ/GhOcI1BVno0U17CbE4nc2FOFAW59izFqqb1aLSd12vlCeDy8vxsSsC9eG1CcZ9Sj53Q3FuHJvTNEbl75GXXE2ygsycFitnERbdi8vxsy8R3Gljby+SB2lmhPDAWTaLNhWV6jZVnX5/ho2x5KkFPbdy4u9s+Vq6LV+++EVTnQNTikOv6WK/PAKJ1werjislley7fVFMJuYYgWVr7/fVl+omk7qKUwM2NlQhHk3x/saxwzrKaTBaK51INNqVowL6zX3IWdnQxEAZc9YEuOMNDO21DpUV0xwMZyxq6EI4zMunLsdGD+WOwu7Gopw6tYQJhU7U+79frsbinCyQzmd5Nk/vLwYAxOzmg7KtrpCHLrWr9jhSJ4vIHRgNosJ5/oV7NJxxmX3cqHMD19XElnhOibGsHNZEd6/OaAYAvNwDsaEM6CsFpNyRym1BwDFuelYvTRXc/5Guoe3h6bRqbCSSK8S0LQ5hiSlsBdm27CuPF/xRnuFVCcFeWSFurcgNZj1lXbkpFsUxcPjFWKGvAxhKKs0McqV0l1VSCf+ZmBoqhTmG5Tzi73Lni6J6A3176dnP1OcIzR4pQlp+fV2NhShc3AKnQOTiukYA7bUCZ2pUtjDGxYAsKuhGHNuD47fVB6VSEcJ7BbTva+SjjGhowegWF6SF//ICidGp+dxtlu5w5HKM8MqlP05hbX2epZ9gzMHJXnpinXTI7vOrmXFmJpz40ynwuhU9J4zrGZsrnEodhJSmUsd1+6GYpztHsbIVOCKFKlD3blMaKuHVeqDHk5OhtWMTdUFKh1b7EhKYQeEm3jhzmjA0iO95awkLwPLnNk4ekPJGxd+p5kZttYW4sj1fgVBXYgVAkK8sO3eWMCGGLl4AOpDXnns0WoxYVt9IQ6rxJnjwfb6InQNTqFrMFBEo8GuZcU42z0SEAZbEAbBewRUBFT02PMy0rC+Il9R2OW+QlOVXexMlcVIus9NVQWq6SRPuyjHhlWluSqdifB7uzh6O6rYWfpOCD60vBj90zzAY9UzPMkYw66GYhxrHwhYWy53TpprHbCaTaodpWTKzmVFuNk/GbC6Sz5/AAijdA+HalmZGFDhyER1YZbyPQTXbbS4q6FY0eZYkrTCvq2+EJwjwCPS2WEXrlVXhJO3hgImTLhMPHYsK8Ld0ZmAjRkeP2/poeXKE2vyWDygPuRdaDzC710Nxbg7OoMbKhtCYheIEdheXwgAOOLXEeopLnJ2NRTB7eE4pnI9QNhAVenIVIz5ytPtqC/ChZ7RgPXg3jRMmLzfIoZHAjxjLNw/qdM9pLCawyMT5B3LinC2azhg34L0idyMNKwpywsoTyGfhXoAANvEEcAxv05Az81hgBBmmph1obXLN84sd06ybBZsqFaeaOVSIgj3DwAO+Ymx3IEBgDVl+SjIsqrkt3Ak7476QhzvGFSe3NSp7kmOgtqkdixIWmFfU5aHnHSLoicD6Btb3r6sEHMuT8BzVz2yTmTHMkHQ/G+23IsBgPribCzJTccxv7i4x+PbI0lDXv/8PH6NVGoY/mGpePnv1YVZKLNn4GhAOegrLhJry/ORm24J8IzlnS4AMeY7GBDz9cjCJ1LMPqCTkPKSlfmdkemAeRf/QdNusdP17+y5TJB31BfB5eEBoR0uE7bt9UW42DMSEIbgUgKRKkcmHOlMcXSpJ1vrCpFmDgxb+U+Q71xWhGu94wGjU/khVjViffHvdP3zMpsYttUV4lj7QGBH6Vkohp0NRZiZ96DVLwSk54C2tki0mYRdfyxmE7bWFuLoDf8brf8k3abqAljNpoAGIxftMnsmaoqyFDxVX3sYY9hS58D77QOKqyHk6bbWFeL4zUGfdP7xUilU5N9RePOLscvOGMP2+iIcvznos1kkWh67xWzC9voiHPOrB/7ltHNZEabn3QENHrLwyerSPBRkWVUFC7K8ACjUB+7z/bbWCZ29/6S6NHkKAI2VQmjHP0wkr1s7lhXCozg65T4eO2MMqwrNOH5z0GeyVe+yz7JZsK7Cjvfb1TsjQBhNAlAWbbZg866GwIlWpcn2rXUO9I/PKo5OJYdhc40UAgoMgelV9Zg0OawQjooVSSvsgBCOuTMyjVuySbFohJozrRY0VtoDJkf9h4s76otw0m8Y6B+KAYSVDsNT82i7PyZL5+thAkJFHp6a91nOp7RscEttIU53Dqluwok1O5cVYnxWeYVJNNhS58D9sRmfeiCftAbkDd7/Hi6kMZkYdtQHzpV47434f5k9E5WOTLznL2zwvX/lBZmoKMjEe/6CjIUOwGoxobnWgSPX/cV/YfJwTVk+cmyBo1OlJXyrHWaMz7pwvmfUJ53ebK0txKW7oz6jCP9wYn1xNpy5toDv7+G+x85ury/C1Jwb528H2hxaR7nQwWVaLdhYXaDQOYf3BCU1ttcXYXLOrboLN9oktbDvqA/0nKLlGW5fVoir98d9Ttxb8FAWhp6zfiEb/zSAcgVVqshbaoV08m3zkidj8qvwM/MefCBbORHPudTm2kKYGHzCMdG0Z6tYTnIB8Z+MzrJZsLYiPyDkIZ/IA4TVMYOTc7jeu+AVKtm+pdaBkx2+nrHHz2OX0p3wSyetipHYsawI3UO+E84LkTkGi3lB/H1GJQqPe1vpMIMx/3BSYB2MlC11DnAOnz0L/o4OYwxbagtx/KbyaEpic7UDjPnVc7+OGZB3qIEjZ592U+fA9d4J9I/7zpXoqQmbawoEm9uDn2ETDZJa2Cscwo1WirPrHcvdUR8Yf5UvNwOADdUFMJuYT2VXmsx15qajrjjbx+NT8tiduemoLcrySyfmJ8twY3UBTCxwqC5cN9bTp0BeRhrWlOfjaHt0xUWi0pGJ0rx0HFcQBvn1mmscuHx31GcFjf8yuOYaBwD45OUvWIDQ6Y7PunzWoMuXO3rT1RVifMaFSz7puE+6LbXCNX3qjd9qqu3LihRHpya/4sy2MqwuzcOx9ujGf9eU5SNTPLBObg/ga1NzjQMDE3M+4ROPX5nnZaZhVWmuT6fr3zFLbK0rxImOoYAO1aRwD33LU1/yM61YVZqrelZRtElqYQeEG32yY8i7/T4a66UBYGVJLvIz0wIqn7xCZdsseGBpns/29QUP29egbXWFOHVryBujUxtpbPVPp9AB5GWk4YGleb7CFub31IvmGgcu9oxias53w0w0uhnGGJprfecjlMS4udYBD4fPrkF/L7u8IBNl9gzF0wTltm8WxeM9vw7A//tJoi33MqVDsCRqi7JRmG3zqVv+9XibOMo77uchK3Xc2+oL8UH3iHdHcjRWilktJmysLgj4/v5Xaq6VOkp5eSqVk2CzFMb0D+tIbKsrxIR/qAm+9/mBpXnItll8ykqvnadymmscPjbHkqQX9k3VBRifdeGK6BFFaz23ycSwsaoAJ2/JwyyBlaW51oHzt0e8gqbW0WypdWB63u09XkDJwxTSFWJ63u2NV6t9uy11QsPwF9I4OOwABOFzeTjOdPl+v2ghzUf4nyciF4Z1FfmwWUwB4Rh/r7e5xoGTt4Y0O4miHBsanDm+YozA+1yYbcPyJTl+4TRfuxhj2FxTgBMdQ976K11bSlflyERxjg0nO3zrn9L93VpbCJeHo1UMCUYrPLm1thAd/ZPevRYLYryQprwgE+UFGT7f38MDbWmudWDO7ZHVF+XeqLlGDNv4hDF9Q1IWswkbquw4cdN3BKT3aHFLbaGPzbEk6YVd8pxO3hJuYqDPoB+bahzoHpryLt+ST7zJ7REa1bA3jWCPX7paB0xsIS7MVYaezTUOMcwy4JsuoAMQrhvvMywkGivtPmGpaImLhP98hNJQ3mYxo7HSHtTrba51YEQ2ua3m8W6pc/hOWvPA+yLZ1to57PXsPAoeweYaYQK4S9xc5O+xMsawqcaBk7cGvXVAKRQDAOsr82ExMW9d4Cp1MFK21PmORtTqZnONAyc6FjpKrjCy2VAlhDEX6rlksy/2LCEE8p5f2E3JweoYmETv2MKcmN5VTwq9xiMck/TC7sxNR5UjM/D0vigIyKbqAgDwek1Kj9tqqrTDIhc0BW8PAHLT07CyNNd7zKia8OVlpmH10jyvZ6gUxxSuKyzJPO7XUcSLLJsFD5YthKWiJS4SS/LSUVOU5Z1n8JaTXwtornGg7d6Yd8cy5wpp/MIHas7CllrfSWulUAwgjCZmXZ6FVUIqDgGwEGpR6pg2VRegd2xWJv7KoZhMqwUPlOX5jC4B/TvVFUtyYc9M89qs2gHWFmJ0emF1l5LHnm2zYE1ZXuD9UzB6Y5UQApHCk/4hUQBorhFDV3756Ylks9LxEtEm6YUdADZVC56Tx8OjOuRfUZKLnHTLwuhAwWNaEDS/yq7QqDZUFeBs9zDmXB7FyVN5unO3RzDrcquOADKsZqytyA84WTHW69jlbK5x4EKPb3gomvY01zhw+pYw3+K/kcubptZ3hKfksZfkZYjOgp/I+tm+sVpYGSG/10r3r6lSSCd14kodQG1RFopybJr1ZnNNgY/tSp2S3LYLPSOYnnPregiYHJOJoamqwLsKzH+JqYRU5l5vHMp+V3OtAxd6RjEx61ItcwDYWG3HrMuDi3cWwpP+6VaW5iI33eLTOUej7jXXOnBetDmWpIaw1xRgdFqIry5sbND/LpqlOLvXCw30FABJ0EYxOetSnTwFBA9MqKCjmhNcG6qEdJfujCoud1xIZ8flu2OBcfY4sam6APNujrNdI1ETFzkbqoT5lmv3x1VHQA+W5SMjzRy0wTfXCnF2t4azkJeRhgZnjjfGqrTcERBGXQ3OHJzqXBB2//ogxNkdOH5TCLVIm53kYQ1hktXqrX9qk6eAsIRw3s3xQfdwVCZPJTZWFaBrcAp9YzOqo1NnbjqqC7Nw6tZC/FzJls01Drg9HGe7hlUnTwHhPgPw5ufxBJan2cSwsdrhN5rQvwTkNseSFBF2mRcWpVUxC9cqQMfAJEZmPAGrGyS8cfauYU2PvUmsoKc7h2QeioLHV2UX0w17H9yhlp/Lw302BsXRYUdT1cLyz2iKy8L1hHJq7VqYhPRv8FaLCesr83G6c2FSV3G4X12A8Rmxk+BanWkBznYNw+X2iJ2E8jdsqrIvpFMIRQCCR943PotbA5OKdYsxhk3VDm+IRcsLbayyw8SE54pGa6UYIMSZAaFuas2jNFXacUa8L2rff12FYLPQbtQ9dke2DbVFWd6Rglo5bKouQPeQ0Omo5RUpcptjSUoI+9L8DJTZM3xWDERLQDZVC53ItWGPqiisrxRu9pnOIdXQCSCsmKgpysLpII3PJ53GevD1FXYwhsBt83Ei22bB6qV5vsIeRWVfmp+Bkrx0QWQ0rtdYWYCr98cwMesKOAZAoqlSECxBjNSv2VRlx+ScWxgtqniigNABSOn813FLbBQ7+tauYXAojwY31RTgzsg0bg9NBazBlyPN4fhOputf+KtKc5GRZsbpzoXOVKmuN1XZMTw1j5v9k6pzEdk2C1aU5KJV1m7U2FhdIKQTR1RK12z0dvTDiNYCYLnNsSQlhB0QBPdUCBUiUlaV5iLLasa1Ybfi6gZg4Waf6R7WDJ0AQmNu7Rr2rsNXFYZKIZ3XY1dII4UGhEa2uO8VLTZU2nHhzqjqQ4b1hDEx5ntrSDXeCwjeo4cD57pHFFc2AcLDwotzbDjTNSzLK/CaG2SjLjVP1D8dwBXzqi3KRl5GGs6K11T2QgXH4tStIc2OBBAmGc92D2MuimWfZjZhXUW+T51Tivs3+nWUWuV07vYIXGJFV7o3UrqxGReu9Y4HnJkjsbo0DzaLCWfEkXO0fIqmSrtgcwzquETKCHtjpR1Dk3PoFLdlR+vJQRazCWsr8tE+rF3xGivtONc9ApdbOwaxoUqYH7jeO66Z34bqENNJoQGPumcfSxor7ZhzeXDprrShJLr2bKiy4/7YDHqGp1Svtq4iXxjZdA2pntMtdBJ2r/esRml+BpbmZ6BVDEWo3Rcp3WnR+VBKZjIxrK/IXxAihUT1xdnISbeIToP2CGijOIdzQTzPJFpVoamqAG33xjA+I8ztKHnPtUVZsGemobVzWNVjB4T6MjXnxuU7Y5o2b/AJYyqns1pMWFOW7w2JRuv7N1YVYGrOjbZ7ys9kjQYpI+zrK/MBwDuJEU09W19hx+1xDyZmXeoxzkppiC5UULUGv1FaQikOmdUrsl1MN6iZTgoNXLuv/hzQWLK+UrD7TAzuC7AQQjnlLc/AC+akL0x6Cke+Khu1vsKOnuFp7wYcNdObquzeeRKt77ehyo5Tt4YVJ08lGivtuNE3gZGpOUUv1GRiWFueL0wwqow2vPb7twl10yJiY1UBPBze89mV7GaMoVEcdWp1SPJ5EiEv5YRldiHsdkoMT6qWZ5Udl++MYnreHbWlthu8c2CxC8ekjLDXF+cg2yZ4MkB0/cL1FXZwAOduj6hWqPUVYgXt1LanzJ6BJbnp3kqhll9FgbDz8FTQDmBhMssIOHPTsTQ/I+riItGwJAc5Not3BYqWgHzQPQK3R12Mm2QxbyEvtYlRYdKze3BKc4S0oboAAxOz6ByYVC0HeUeoJkTrK+y43juO8Zl5zfIszklHmT3D+1i9aI3e1lXkwyzbEKVV5rcGJjEwMatqd0meMLLx5qWSTgq7CSMA9YSNFXa4PNw7aokGks2x3IGaMsJuFj2ZLoUH2erNugrBE2rvm9AU7OIcm/fwI7VGxRjDugqZ3SoZSqEB6bFnavlJQ/47I9Na2cWUxkq74gOGo4HZxLC+0o6OfiEkp9ZRNlUWYGLWhftjM6rzH6tKc5GeZkJ734RqGmDBY+vQEGzpmgDQqdEBrC0XRPKGxjXXi3MEQj4aF4TQCXRH+RFuWTYLVpbkBq2bTWKn1dE/qWl3U5UdN/ulkKp6uvUV+bg/NoM5l0dzBASEVlaR0Fhp91mNFW1SRtgB4UZLRPMm5mdasSSLiddRF2xpWAmoT54CCx2FkE494bryhfy0vp7k9RkF3/sS/a6mUfb9tUJl3jQqpZlmNuHBsnwxH3W7lxULowRA+z7XFWcjW0ynll2mVRBJIS918ffaHqQ85XUrmiUfynUeKMuD1WIKaos0UgK0v9+6iuDtwZ5lRW1RVtBrRsqGKjt6x2bRMzwdPLEOpJSwr/MRtOgKSF2+OWia9RXBxQMIrYIK6ULtAGLTwYWKtCICiM0IwreclNNIIypAffcmsOBlujWWW5lMDGvKg3cAZhPDmvI80S71dFKn41bx/vIy0lBfnC1cT910AH51MIqFH0rdtFnMeGCp8P212kNjiO1hZUmut6PQKs8mWf2LFpIzdbY7NuGYiISdMfabjLHLjDEPY6xJL6Oixfry2FRiAKjLF4rW/8HHcuReoVYNXV2aB4tJGgFopFsaWrq1skZmBJaX5Hj/jkVHI3nZ4hUV0zDGZKKnMQFZEdroR/Kigx3hKo26QhlxTc2p5yXZFaw8V5TkwiZ5yVEs+7Uhtj2pnNQ6LQBY5syW5aWemdViwurS3KDXlCaR747OqCeKkAZnDjLSzDF7alikHvslAB8DcEQHW6JOXmZazK4Vise+qjTP+7fWED3DavaKn5bnkZ5mxoogw3ThurlBbYslaWYTMq3By0sv8jIW6oFWuUsdoP9DopXSBEPyWOUP3lDMTxQ2zdUsIVxTEqtgc0pWi8nrJUeTKkcm8sX2p/XdpO9/Z0J9zbfFbEJGmlnMS/u6Uoei1QHIO51oYTEL5ZwQws45b+OcX9PLmFggDa+jTWl2cPdHGiYCIcRCQ/DkgAUB0UpnsyyIaDyeoKSE1KBjsVEJAAqzrQCCNXjBpguyhzYE5hNafZLHvTXTSaKtcVuW5mcEzUfy2P3Pn1dioc5Ery4wxkIqAynNmHpfCgB4sEzojObd2pORobSHuuJsjXf1Y21FPi7fHYvJA64tUb+CCGPseQDPA4DT6cShQ4fCymdiYiLszwJAVZYLfePAkRNnMNYRva8/NTkJqTpp2WthgIsDx44eRbpFwxufFB7Xdv78ecz1qHu3tonQ0pkZ4ObAkaNHYDPHX9wLIbTkw62XkD7g6ytEes+VqMpyY2ACOHKiFSM3levBjEsQjf6x6ZCuH6qdwdIUZTCMjoxElJdHFsqQp1Gy0ToubBy6ePEC2P3otYl8t3CPT548gZsZyj4lV7HbnyIm5HW09SLSB66qppudFkS0q6sThw7dDWrjYuvZYupm2pgLcy4PfvpGC2ryojxClU6KU/sBcABCyMX/5yOyNIcANAXLS/ppbGzk4dLS0hL2ZznnfGRyjv/1vst8Zt4VUT7BaGlp4e+3D/C9p7o007X3jfNvH7geNL+xacHuydl5zXTjM/MhpWvvG+df+pf9Qa8bK6ZmXfxv3rjCR6fnAt6L9J4rMTw5G1I9+H/HOvjZriHNNOe6h/kPjnYEtfONC3f5W5fuBbUtlHSnbw3ynxzv1Ezz8ulufuxGv89rSjZKZT+mUPZ6cndkin/jzTbudns00712tod/++V3NNOEarPH4+HfOXiDX7s/ppnuvRv9/KVT3ZpplFhM3bw7MsW/8K+n+fnbw4u+jgSAVh6CxoYkxEEzSSBhjxWJYGci2Mg52akniWAj52SnGqEKe0otdyQIgkgFIl3u+AxjrAdAM4A3GGP79TGLIAiCCJeIZko4568BeE0nWwiCIAgdoFAMQRBEkkHCThAEkWSQsBMEQSQZJOwEQRBJBgk7QRBEksF4HJ5qzBjrB9AV5scLAQzoaE60SAQ7E8FGgOzUk0SwESA71ajknBcFSxQXYY8Exlgr59zwRwQngp2JYCNAdupJItgIkJ2RQqEYgiCIJIOEnSAIIslIRGH/frwNCJFEsDMRbATITj1JBBsBsjMiEi7GThAEQWiTiB47QRAEoUFCCTtj7AnG2DXGWDtj7KtxtKOcMdbCGGsTH+b9R+LrBYyxdxhjN8TfdvF1xhj7B9HuC4yx9TG218wY+4Axtk/8v5oxdlK08yXGmFV83Sb+3y6+XxUj+/IZY68wxq6KZdpsxLJkjH1ZvN+XGGM/Z4ylG6EsGWM/ZIz1McYuyV5bdPkxxj4npr/BGPtcjOz8O/G+X2CMvcYYy5e994Jo5zXG2OOy16OmA0o2yt77Y8YYZ4wViv/HrSyDEsqh7Ub4AWAGcBNADQArgPMAVsbJlhIA68W/cwBcB7ASwN8C+Kr4+lcBfFP8+ykAb0J4Vt5mACdjbO9XAPwMwD7x/5cBPCv+/V0Avy/+/QcAviv+/SyAl2Jk348A/K74txVAvtHKEsBSALcAZMjK8HeMUJYAdgBYD+CS7LVFlR+AAgAd4m+7+Lc9BnY+BsAi/v1NmZ0rxTZuA1Attn1ztHVAyUbx9XIA+yHsvymMd1kG/R6xvFiEBd4MYL/s/xcAvBBvu0RbfgngUQDXAJSIr5UAuCb+/T0An5al96aLgW1lAN4F8BCAfWIlHJA1Jm+5ihW3WfzbIqZjUbYvVxRM5ve6ocoSgrDfFhurRSzLx41SlgCq/ARzUeUH4NMAvid73SddtOz0e+8ZAP8m/u3TvqXyjIUOKNkI4BUAawB0YkHY41qWWj+JFIqRGpZEj/haXBGH2OsAnATg5JzfAwDxd7GYLJ62/z2APwUgPRrdAWCEc+5SsMVrp/j+qJg+mtQA6Afw/8Rw0f9ljGXBYGXJOb8D4H8D6AZwD0LZnIGxylLOYsvPCO3r8xA8YGjYE3M7GWMfBnCHc37e7y3D2OhPIgk7U3gtrkt6GGPZAH4B4Euc8zGtpAqvRd12xtjTAPo452dCtCUedlogDH3/mXO+DsAkhNCBGvEqSzuAj0AIC5QCyALwpIYthquvImp2xdVextjXALgA/Jv0koo9MbWTMZYJ4GsA/lzpbRVb4n7vE0nYeyDEuSTKANyNky1gjKVBEPV/45y/Kr7cyxgrEd8vAdAnvh4v27cC+DBjrBPAXgjhmL8HkM8Yk56eJbfFa6f4fh6AoSjb2AOgh3N+Uvz/FQhCb7SyfATALc55P+d8HsCrALbAWGUpZ7HlF7f2JU4uPg3gP3ExdmEgO2shdObnxXZUBuAsY2yJgWwMIJGE/TSAenEVghXChNTr8TCEMcYA/ABAG+f8W7K3XgcgzYB/DkLsXXr9t8VZ9M0ARqVhcjThnL/AOS/jnFdBKK+DnPP/BKAFwCdU7JTs/4SYPqqeBuf8PoDbjLEG8aWHAVyBwcoSQghmM2MsU7z/kp2GKUs/Flt++8o80FcAAAEhSURBVAE8xhizi6OTx8TXogpj7AkAewB8mHM+5Wf/s+LqomoA9QBOIcY6wDm/yDkv5pxXie2oB8LCifswWFn6G54wPxBmoa9DmBX/Whzt2AZhaHUBwDnx5ykIMdR3AdwQfxeI6RmAF0W7LwJoioPNu7CwKqYGQiNpB/DvAGzi6+ni/+3i+zUxsm0tgFaxPP8DwkoCw5UlgL8AcBXAJQA/gbBiI+5lCeDnEOL+8xCE5wvhlB+EGHe7+PNcjOxshxCPltrRd2XpvybaeQ3Ak7LXo6YDSjb6vd+JhcnTuJVlsB/aeUoQBJFkJFIohiAIgggBEnaCIIgkg4SdIAgiySBhJwiCSDJI2AmCIJIMEnaCIIgkg4SdIAgiySBhJwiCSDL+PxO8wMa2EbB7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x181ae425c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(process_rewards(rewards_list),'-',)\n",
    "ax=plt.gca()\n",
    "ax.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so now we have non zero rewards most of the time. \n",
    "\n",
    "in points we got -1 reward, the new score is aout -1.5 as seen above and steps before it also have negative rewards. the negativity of the reward goes down as we go to the left from each -1.5 score. \n",
    "\n",
    "### some discussion about positive rewards in early steps of losing episodes:\n",
    "also you can see that we got some positive scores in our new chart. this is because we subtracted by mean.\n",
    "think about what it can result?\n",
    "\n",
    "the positive reward happens in the long sequences of playing. these are situations where we catched the ball and didnt get a -1 immediately, but we lost later because we failed to catch the ball when it came toward us.\n",
    "\n",
    "setting a negative score is logical because we ended up getting a -1. setting a positive score could be justified by that we approve actions where we catched the ball and did not get -1 reward immediately (maybe we didnt win either)\n",
    "\n",
    "the above scoring method could prove a bit problematic when we are winning. it may discourage actions in early steps of a sequence which ends in winning. but we dont further consider discuss it here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so now that we have defined our rewards, we can go to next step and start training the network. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Example of simluation and training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first generate an episode:\n",
    "states_list,up_or_down_action_list,rewards_list,network_output_list = generate_episode(policy_network_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of states= 1385\n",
      "shape of each state=(80, 80)\n",
      "length of rewards= 1385\n"
     ]
    }
   ],
   "source": [
    "print(\"length of states= \"+str(len(states_list)))# this is the number of frames\n",
    "print(\"shape of each state=\"+str(states_list[0].shape))\n",
    "print(\"length of rewards= \"+str(len(rewards_list)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode_reward.shape = (1385, 1)\n",
      "x.shape = (1385, 80, 80)\n",
      "y_true.shape = (1385, 1)\n"
     ]
    }
   ],
   "source": [
    "#preprocess inputs for training: \n",
    "    \n",
    "x=np.array(states_list)\n",
    "\n",
    "episode_reward=np.expand_dims(process_rewards(rewards_list),1)\n",
    "\n",
    "y_tmp = np.array(up_or_down_action_list) # 1 if we chose up, 0 if down\n",
    "y_true = np.expand_dims(y_tmp,1) # modify shape. this is neccassary for keras\n",
    "\n",
    "\n",
    "print(\"episode_reward.shape =\",episode_reward.shape)\n",
    "print(\"x.shape =\",x.shape)\n",
    "print(\"y_true.shape =\",y_true.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "1385/1385 [==============================] - 1s 545us/step - loss: -2.6306e-04\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x181ada8b00>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit the model with inputs and outputs.\n",
    "policy_network_train.fit(x=[x,episode_reward],y=y_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Training the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for the actual training, we generate some episode, then train on the batch. \n",
    "we can also log the rewards to keep records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we define a helper function to create a batch of simulations\n",
    "# and after the batch simulations, preprocess data and fit the network\n",
    "def generate_episode_batches_and_train_network(n_batches=10):\n",
    "    env = gym.make('Pong-v0')\n",
    "    batch_state_list=[]\n",
    "    batch_up_or_down_action_list=[]\n",
    "    batch_rewards_list=[]\n",
    "    batch_network_output_list=[]\n",
    "    for i in range(n_batches):\n",
    "        states_list,up_or_down_action_list,rewards_list,network_output_list = generate_episode(policy_network_model)   \n",
    "        batch_state_list.extend(states_list[15:])\n",
    "        batch_network_output_list.extend(network_output_list[15:])\n",
    "        batch_up_or_down_action_list.extend(up_or_down_action_list[15:])\n",
    "        batch_rewards_list.extend(rewards_list[15:])\n",
    "    \n",
    "    episode_reward=np.expand_dims(process_rewards(batch_rewards_list),1)\n",
    "    x=np.array(batch_state_list)\n",
    "    y_tmp = np.array(batch_up_or_down_action_list)\n",
    "    y_true = np.expand_dims(y_tmp,1)\n",
    "    policy_network_train.fit(x=[x,episode_reward],y=y_true)\n",
    "\n",
    "    return batch_state_list,batch_up_or_down_action_list,batch_rewards_list,batch_network_output_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we write a script to save the model in a file and log rewards from time to time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "13330/13330 [==============================] - 7s 552us/step - loss: -9.9711e-06\n",
      "i=0\n",
      "count win=5\n",
      "Epoch 1/1\n",
      "12718/12718 [==============================] - 7s 569us/step - loss: -9.7538e-06\n",
      "Epoch 1/1\n",
      "13721/13721 [==============================] - 8s 597us/step - loss: -1.5545e-04\n",
      "Epoch 1/1\n",
      "13840/13840 [==============================] - 8s 571us/step - loss: -2.3620e-06\n",
      "Epoch 1/1\n",
      "12213/12213 [==============================] - 7s 559us/step - loss: -8.1813e-04\n",
      "Epoch 1/1\n",
      "14535/14535 [==============================] - 8s 585us/step - loss: -9.3401e-04\n",
      "Epoch 1/1\n",
      "14555/14555 [==============================] - 8s 580us/step - loss: -2.4612e-04\n",
      "Epoch 1/1\n",
      "13726/13726 [==============================] - 8s 580us/step - loss: -3.9308e-05\n",
      "Epoch 1/1\n",
      "14706/14706 [==============================] - 8s 574us/step - loss: -1.6321e-04\n",
      "Epoch 1/1\n",
      "13594/13594 [==============================] - 8s 581us/step - loss: -2.4924e-05\n",
      "Epoch 1/1\n",
      "14502/14502 [==============================] - 8s 585us/step - loss: -0.0014\n",
      "i=10\n",
      "count win=11\n",
      "Epoch 1/1\n",
      "14704/14704 [==============================] - 9s 589us/step - loss: -0.0012\n",
      "Epoch 1/1\n",
      "15619/15619 [==============================] - 10s 616us/step - loss: -0.0038\n",
      "Epoch 1/1\n",
      "16999/16999 [==============================] - 10s 599us/step - loss: -0.0018\n",
      "Epoch 1/1\n",
      "16597/16597 [==============================] - 9s 539us/step - loss: -0.0040\n",
      "Epoch 1/1\n",
      "16368/16368 [==============================] - 8s 517us/step - loss: -0.0025\n",
      "Epoch 1/1\n",
      "15080/15080 [==============================] - 8s 509us/step - loss: -9.9850e-04\n",
      "Epoch 1/1\n",
      "17943/17943 [==============================] - 9s 503us/step - loss: -0.0053\n",
      "Epoch 1/1\n",
      "16353/16353 [==============================] - 8s 503us/step - loss: -0.0046\n",
      "Epoch 1/1\n",
      "16925/16925 [==============================] - 9s 519us/step - loss: -0.0054\n",
      "Epoch 1/1\n",
      "16625/16625 [==============================] - 8s 504us/step - loss: -0.0056\n",
      "i=20\n",
      "count win=10\n"
     ]
    }
   ],
   "source": [
    "train_n_times = 21 # for actual training, about 5000 may be a good start. \n",
    "for i in range(train_n_times):\n",
    "    states_list,up_or_down_action_list,rewards_list,network_output_list=generate_episode_batches_and_train_network(10)\n",
    "    if i%10==0:\n",
    "        print(\"i=\"+str(i))\n",
    "        rr=np.array(rewards_list)\n",
    "        # i keep how many times we won in batch. you can use log more details more frequently\n",
    "        print('count win='+str(len(rr[rr>0]))) \n",
    "        policy_network_model.save(\"policy_network_model_simple.h5\")\n",
    "        policy_network_model.save(\"policy_network_model_simple\"+str(i)+\".h5\")\n",
    "        with open('rews_model_simple.txt','a') as f_rew:\n",
    "            f_rew.write(\"i=\"+str(i)+'       reward= '+str(len(rr[rr > 0])))\n",
    "            f_rew.write(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i let it train for some batches here for presentation purposes. **in actual training we should let it run for much more episodes**.(about 1000 to 100000)\n",
    "\n",
    "the model we defined earlier had 1.28 million params. so the process of simulation and training is kind of slow. u can try and simplify the model or maybe use convolutional models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Playing the Trained Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "def play_and_show_episode(policy_network):\n",
    "    env = gym.make('Pong-v0')\n",
    "    done=False\n",
    "    observation = env.reset()\n",
    "    new_observation = observation\n",
    "    while done==False:\n",
    "        time.sleep(1/80)\n",
    "        \n",
    "        processed_network_input = preprocess_frames(new_frame=new_observation,last_frame=observation)\n",
    "        reshaped_input = np.expand_dims(processed_network_input,axis=0) # x shape is (80,80) so we need similar reshape(x,(1,80,80))\n",
    "\n",
    "        up_probability = policy_network.predict(reshaped_input,batch_size=1)[0][0]\n",
    "        actual_action = np.random.choice(a=[2,3],size=1,p=[up_probability,1-up_probability])\n",
    "        \n",
    "        env.render()\n",
    "        \n",
    "        observation= new_observation\n",
    "        new_observation, reward, done, info = env.step(actual_action)\n",
    "        if reward!=0:\n",
    "            print(reward)\n",
    "        if done:\n",
    "            break\n",
    "        \n",
    "    env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n"
     ]
    }
   ],
   "source": [
    "play_and_show_episode(policy_network_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if you have trained the network for some amount of time, you can see that the agent has learned some tricks to defend and even win.\n",
    "\n",
    "note that the default model with 1.2 million params we defined earlier may take a huge amount of training to learn to win sometimes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. Loading model from file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you can load model from file and play the game or even continue training it. \n",
    "\n",
    "here i load one of my trained files. it is the same model with 1.28 million paramteres, trained for 1550 on batches of 10 simulations, which means 15500 games in total. it is still not intelligent, but not dumb either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 80, 80)            0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 6400)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 200)               1280000   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 200       \n",
      "=================================================================\n",
      "Total params: 1,280,200\n",
      "Trainable params: 1,280,200\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/soroush/anaconda3/lib/python3.6/site-packages/keras/models.py:282: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
      "  warnings.warn('No training configuration found in save file: '\n"
     ]
    }
   ],
   "source": [
    "policy_network_model=keras.models.load_model(\"trained_simple_model_3300.h5\")\n",
    "policy_network_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since keras does not save the custom loss in the file we should redefine loss function and training model.\n",
    "\n",
    "note that this is not needed if you dont want to continue the training of the saved model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "episode_reward = keras.layers.Input(shape=(1,),name='episode_reward')\n",
    "\n",
    "def m_loss(episode_reward):\n",
    "    def loss(y_true,y_pred):\n",
    "        # feed in y_true as actual action taken\n",
    "        # loss = reward*(-actual*np.log(y_pred)-(1-actual)*np.log(1-y_pred)))\n",
    "        \n",
    "        tmp_pred = keras.layers.Lambda(lambda x: keras.backend.clip(x,0.05,0.95))(y_pred) # we could also do gradient clipping\n",
    "        tmp_loss = keras.layers.Lambda(lambda x:-y_true*keras.backend.log(x)-(1-y_true)*(keras.backend.log(1-x)))(tmp_pred)\n",
    "        # put reward in effect\n",
    "        policy_loss=keras.layers.Multiply()([tmp_loss,episode_reward])\n",
    "        \n",
    "        return policy_loss\n",
    "    return loss\n",
    "\n",
    "policy_network_train = keras.models.Model(inputs=[policy_network_model.input,episode_reward],outputs=policy_network_model.output)\n",
    "my_optimizer = keras.optimizers.RMSprop(lr=0.0001)\n",
    "policy_network_train.compile(optimizer=my_optimizer,loss=m_loss(episode_reward),)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "play_and_show_episode(policy_network_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10. Using Convolutional Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 80, 80)            0         \n",
      "_________________________________________________________________\n",
      "reshape_1 (Reshape)          (None, 80, 80, 1)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 16, 16, 10)        4000      \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 4, 4, 20)          20000     \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 2, 2, 40)          7200      \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 160)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 160       \n",
      "=================================================================\n",
      "Total params: 31,360\n",
      "Trainable params: 31,360\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/soroush/anaconda3/lib/python3.6/site-packages/keras/models.py:282: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
      "  warnings.warn('No training configuration found in save file: '\n"
     ]
    }
   ],
   "source": [
    "policy_network_model=keras.models.load_model(\"trained_conv_model.h5\")\n",
    "policy_network_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n",
      "-1.0\n"
     ]
    }
   ],
   "source": [
    "play_and_show_episode(policy_network_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video width=\"320\" height=\"420\" controls>\n",
       "  <source src=\"trained_convolutional_network.mp4\" type=\"video/mp4\">\n",
       "</video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<video width=\"320\" height=\"420\" controls>\n",
    "  <source src=\"trained_convolutional_network.mp4\" type=\"video/mp4\">\n",
    "</video>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11. Ideas & Thoughts\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as you can see, both simple and convolutional models have learned to play and score some points.\n",
    "\n",
    "our simple network takes more time for each simulation and training. the convolutional model takes less amount of time both for simulation and playing. probably because it has less parameters.\n",
    "\n",
    "I played around a bit with convolutional models. the problem with them was that they get stuck in the bottom or top of the game, and it is hard for them to learn to get out of that situation. \n",
    "\n",
    "I didn't want to make perfect agents, but if you want, you can try and train the networks a lot more so they can play better.\n",
    "\n",
    "I had access to high performance computing system of my university which has about 30 cores of cpu. The training on my own pc was much faster because of the clock of cpu (the cluster just has many weak cpus), but used the university cluster since it could train day and night. It took the simple network took about 3 days to run 1550 batches of 10 game simulations on the cluster.\n",
    "\n",
    "Here are some ideas to improve the training and agents:\n",
    "\n",
    "* **generate simulations of batches in parallel**. I tried using multithreading to generate and play batches of games at the same time to increase the speed of data generation. the problem was that the backend library for the game had issues creating environments in multiple threads. maybe you can overcome this by implementing a multiprocess method. \n",
    "\n",
    "* The convolutional models I used were very fast at generating actions and also training. Their problem was that after some training, they learned some tricks fast but took a lot of time for them to learn something new. The agent just stuck in the up or bottom of game screen and didnt try new things to score and learn. I think **using an epsilon-greedy policy** can improve the exploration of the networks.which means that each action has a minimum probablity of being chosen(for example 20%). Another idea could be **using off-policy method** which means another more explorative policy(for example a policy that just takes random actions) plays the game, and our target policy learns from the experience generated by the explorative policy.\n",
    "\n",
    "* giving a reward to the agent for catching the ball could be good for not losing(for example a +0.5 reward). now we are just giving positive reward to agent for scoring. if it catched the ball and then got a -1 reward, it thinks it was probably bad that it catched the ball.  \n",
    "\n",
    "* tuning the optimizer parameters may be helpful. I just modified learning rate a bit. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**some thoughts about the agents' mind:**\n",
    "\n",
    "We are far from general intelligence. The agent has no sense if there is a ball which moves around and hits and ricochets from walls and paddles. It has no notion of physics of the game. All it sees is that some neurons of the input fires and it take some actions. unlike a human, if he sees the ball moving in a new position and direction that it had not seen before, it cannot generalize from his past experience. It has no notion of the ball, environment, itself, reward and punishment and... the agent is just a regression model.this article might be interesting: https://medium.com/syncedreview/interview-with-dr-richard-sutton-we-might-have-strong-ai-algorithms-by-2030-a1052332d878"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 12. References and Suggested readings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "great blog post by Andrej Karpathy: http://karpathy.github.io/2016/05/31/rl/\n",
    "this project is actually re implementing the above blog post with some explanations on the coding and implementation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Richard Sutton has a great book: Reinforcement Learning: An Introduction. \n",
    "The book is kind of a bible for reinforcement learning. At the time of writing this, the draft of second version is available online. http://incompleteideas.net/book/the-book-2nd.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}