Value_Function_Approximation_parte2_RL_ITA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "ksLFQ4EUTVCN",
        "alcM2JiRT6p4"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#Deep Q Network (DQN)\n",
        "\n",
        "Mario Fiorino\n",
        "\n",
        "$\\;$\n",
        "\n",
        "**Introduzione**\n",
        "\n",
        "Il Deep Q-Network (DQN), è un approccio basato sui metodi Temporal Difference (TD), che utilizza tecniche di approssimazione per stimare la funzione di valore d'azione $Q$.\n",
        "\n",
        "In pratica, l'algoritmo è stato sviluppato combinando il classico algoritmo Q-Learning con reti neurali (le deep neural networks); ed una tecnica chiamata experience replay. DQN sfrutta l'approccio Semi-gradient TD, calcolando i gradienti tramite backpropagation.\n",
        "\n",
        "Risulatati ottenuti : è stato in grado di risolvere un'ampia gamma di giochi Atari, in alcuni casi superando anche le prestazioni umane.\n",
        "\n",
        "Ref.\n",
        "\n",
        "https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf\n",
        "\n",
        "\n",
        "https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf\n",
        "\n",
        "\n",
        "https://www.tensorflow.org/agents/tutorials/0_intro_rl?hl=it\n",
        "\n",
        "https://github.com/amrzv/awesome-colab-notebooks?tab=readme-ov-file\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Implementazione molto accurata ed interessante:\n",
        "\n",
        "https://github.com/fg91/Deep-Q-Learning/blob/master/DQN.ipynb"
      ],
      "metadata": {
        "id": "T_rjNC-gxtzu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Idee di base\n",
        "\n",
        "DQN sfrutta reti neurali per approssimare in modo non lineare la funzione valore-azione.\n",
        "In pratica addestriamo una rete neurale, per stimare i valori della funzione valore-azione:  $Q(s, a; w)$. Tale rete neurale è detta anche *Q network*.\n",
        "\n",
        "L' addestramento avviene minimizzando la seguente Loss:\n",
        "\n",
        "$\\begin{equation}L(w_t) = \\mathbb{E}_{s, a, r, s'\\sim \\rho(.)} \\left[ (y_t - Q(s, a; w_t))^2 \\right]\\end{equation}$\n",
        "\n",
        "dove:\n",
        "\n",
        "\n",
        "Il  vettore pesi, ad un certo step, è indicato con $w_t$\n",
        "\n",
        "Il target : $y_t = r +  \\gamma \\max_{a^\\circ} Q(s', a^\\circ ; w_{t})$\n",
        "\n",
        "$\\rho$ rappresenta la behaviour distribution, ovvero la distribuzione di probabilità sulle transizioni $\\{s, a, r, s'\\}$ raccolte dall'ambiente.\n",
        "\n",
        "$\\max_{a^\\circ} Q(s', a^\\circ ; w_{t})$ rappresenta il massimo valore previsto per la funzione $Q$, nello stato successivo $s'$, considerando tutte le azioni possibili $a^\\circ$ in quello stato.\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Update dei parametri $w$ della rete si ricava sfruttando la differenziazione della funzione di Loss rispetto a questi, pensado $y_t$  come un costante (approccio Semi-gradient ) :\n",
        "\n",
        "${w}_{t+1} \\leftarrow {w}_t + \\alpha [ R_{t+1} + \\gamma \\max_{a}  \\widehat{q}(s_{t+1},a,{w}_t) - \\widehat{q}(s_{t},a_{t},{w}_t)] ∇ \\widehat{q}(s_{t},a_{t},{w}_t)$\n",
        "\n",
        "Nota : il gradiente $∇ \\widehat{q}(s_{t},a_{t},{w}_t)$ può essere calcolato mediante i metodi di backpropagation.\n",
        "\n",
        "$\\;$\n",
        "\n",
        "In concrento:\n",
        "\n",
        "per approssimare la funzione di valore-azione, impieghiamo una rete neurale convoluzionale, comunemente conosciuta come **convolutional neural network** (CNN).\n",
        "\n",
        "Queste sono una tipologia di reti neurali feed-forward che eccellono nell'elaborazione dei dati disposti in una struttura a griglia, tipicamente immagini.\n",
        "La disposizione dei pixel in un'immagine non è casuale, ma riflette relazioni spaziali tra elementi vicini (i pixel vicini, spesso correlati e forniscono indizi importanti per comprendere il contenuto dell'immagine). Le CNN sono in grado di catturare queste relazioni.\n",
        "\n",
        "Come ci riescono? La chiave risiede  nell'utilizzo di un processo chiamato convoluzione: in sostanza si fa scorrere un filtro, chiamato anche kernel, che è essenzialmente una piccola matrici 2D di pesi, sull'immagine di input. Man mano che il filtro scorre sull'immagine, si calcola il prodotto scalare tra i pesi del filtro e i pixel dell'immagine che ricadono sotto il filtro stesso.\n",
        "\n",
        "Una volta che il filtro ha attraversato l'intera immagine, spostandosi passo dopo passo, si ottiene come output una matrice 2D chiamata Feature Map. Questa mappa rappresenta la sintesi delle informazioni estratte dall'immagine mediante il processo di convoluzione.\n",
        "\n",
        "$FeatureMap[i,j] = \\sum_{u=-k} ^k \\sum_{v=-k} ^k Filter[u,v] \\cdot Image[i-u,j-v]  \\;\\;\\; \\forall (i,j) \\in Image$\n",
        "\n",
        "\n",
        "Per approfondimenti e per comprendere meglio il funzionamento delle CNN, consultare i seguenti riferimenti:\n",
        "\n",
        "Introduzione:\n",
        "\n",
        "https://twice22.github.io/cnn-introduction/\n",
        "\n",
        "\n",
        "https://towardsdatascience.com/gentle-dive-into-math-behind-convolutional-neural-networks-79a07dd44cf9\n",
        "\n",
        "https://www.cs.cornell.edu/courses/cs6670/2011sp/lectures/lec02_filter.pdf\n",
        "\n",
        "Convolutions with multiple filters:\n",
        "\n",
        "https://medium.com/advanced-deep-learning/cnn-operation-with-2-kernels-resulting-in-2-feature-mapsunderstanding-the-convolutional-filter-c4aad26cf32\n",
        "\n",
        "https://www.youtube.com/watch?v=06fMrGaLrDw\n",
        "\n",
        "$\\;$\n",
        "\n",
        "**Input della Q network** : consiste in un'immagine pixel dello schermo in un certo istante $t$ (snapshot della scena al momento $t$, detto anche \"pixel image frame\").Tale immagine rappresenta lo stato dell'ambiente in cui opera l'agente. Si noti , la rete in sè svolge autonomamente l'estrazione di feature (\"feature construction\").\n",
        "\n",
        "\n",
        "**Output** : restituisce una stima del valore di $Q$ per tutte le azioni disponibili nello stato di ingresso.\n",
        "\n",
        "$\\;$\n",
        "\n",
        "**Experience Replay**\n",
        "\n",
        "Premessa: ogni esperienza è una singola transizione tra stati, rappresentata da una tupla $(S_{t}, A, R_{t+1}, S_{t+1})$ che descrive ciò che è accaduto durante un'azione.\n",
        "\n",
        "Il DQN ha introdotto una tecnica chiamata Experience Replay per migliorare la stabilità degli aggiornamenti della rete neurale durante l'apprendimento.\n",
        "\n",
        "Motivazioni principali:\n",
        "\n",
        "- Uso limitato dei dati:\n",
        "gli algoritmi RL \"classici\" usano i dati generati dalla policy per aggiornare i parametri, impiegando ogni esperienza un'unica volta. Questo potrebbe essere inefficiente quando si usano metodi di approssimazione di funzione come le reti neurali che sfruttano tecniche di *gradient descent* per apprendere i parametri ottimali. Infatti, per aggiornare i parametri con il metodo *gradient descent*, ogni aggiornamento deve essere piccolo. Questo perché il gradiente fornisce informazioni utili solo in un’area limitata intorno ai valori attuali dei parametri.\n",
        "Tuttavia, alcuni casi potrebbero richiedere aggiornamenti grandi per ricavare i parametri ottimali (in particolare quando sussiste un ampio divario tra il valore $Q$ stimato e quello del target). In questi casi, i parametri di una rete devono essere aggiornati più volte, quindi dovebbre essere necessario usare le stesse esperienze più volte.\n",
        "\n",
        "- Dati correlati:\n",
        "quando uno specifico aggiornamento dei parametri si basa su dati provenienti da un singolo episodio, le esperienze usate non sono indipendenti (visto che gli stati e le ricompense futuri dipendono fortemente dagli stati e dalle azioni precedenti), ma correlate. Questa dipendenza aumenta la probabilità che gli aggiornamenti dei parametri siano molto variabili da un episodio all'altro (*high variance in parameter updates*).\n",
        "\n",
        "Questa tecnica consiste nell'archiviare, e riutilizzare in modo appropriato, in un buffer  (detto chiamato buffer di replay (https://www.researchgate.net/figure/Illustration-of-the-replay-buffer-in-the-RL_fig5_346346921) ) le esperienze raccolte dell'agente.\n",
        "\n",
        "Durante l'allenamento, dopo ogni step una tupla $(S_{t}, A, R_{t+1}, S_{t+1})$ viene aggiunta nel replay buffer. In questo modo, si accumula esperienze provenienti da molteplici episodi. Così, invece di utilizzare solo l'ultima esperienza acquisita per calcolare la Loss e il suo gradiente; si estraggono in modo casuale ( *randomizes over the data*) mini-batch di esperienze dal buffer, e si utilizzano questi batch per aggiornare la rete.\n",
        "\n",
        "Nel paper: *Human-level control through deep reinforcement learning*, Volodymyr Mnih et al.\n",
        "https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf\n",
        "\n",
        "si sostiene che utilizzando più  volte transizioni non correlate dal mini-batch (in pratica utilizzando esperienze non consecutive, si riduce la correlazione temporale tra gli aggiornamenti) ci sono i seguenti vantaggi:\n",
        "- \"*First, each step of experience is potentially used in manyweight updates,which allowsfor greater data efficiency.*\"\n",
        "- \"*Second, learning directly from consecutive samples is inefficient, owing to the strong correlations between the samples; randomizing the samples breaks these correlations and therefore reduces the variance of the updates*\"\n",
        "- \"*Third, when learning on-policy the current parameters determine the next data sample that the parameters are trained on...It is easy to see how unwanted feedback loops may arise and the parameters could get stuck in a poor localminimum, or even diverge catastrophically. By using experience replay the behaviour distribution is averaged over many of its previous states, smoothing out learning and avoiding oscillations or divergence in the parameters.*\"\n",
        "\n",
        "$\\;$\n",
        "\n",
        "**Convergenza**\n",
        "\n",
        "Gli autori, Mnih et al.,  dichiarano : *... despite lacking any theoretical convergence guarantees, our method is able to train large neural networks using a reinforcement learning signal and stochastic gradient descent in a stable manner.*"
      ],
      "metadata": {
        "id": "y49Vh3VF-IxE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "$\\;$\n",
        "\n",
        "Appendice: **Problema della correlazione nell'update dei pesi**: esiste una correlazione tra la funzione valore-azione $Q(s, a; w_t)$ ed il target : $r +  \\gamma \\max_{a^\\circ} Q(s', a^\\circ; w_{t})$\n",
        "\n",
        "Si legge nel testo di Sutton & Barto, 2018 PDF 462:\n",
        "\n",
        "*As in other methods that bootstrap, the target for a Q-learning update depends on the current action-value function estimate. When a parameterized function approximation method is used to represent action values, the target is a function of the same parameters\n",
        "that are being updated ...  Its dependence on $w_t$ complicates the process compared to the simpler supervised-learning situation in which the targets do not depend on the parameters  being updated. As discussed in Chapter 11 this can lead to oscillations and/or divergence.*\n",
        "\n",
        "Soluzione proposta:\n",
        "\n",
        "L'utilizzo di due reti separate per stimare la funzione valore-azione ed il target. Esattamente definendo il target:\n",
        "\n",
        "$y_t = r + \\gamma \\max_{a^\\circ} Q(s', a^\\circ; w_{t-τ})$\n",
        "\n",
        "\n",
        "Cioè si usano i parametri della rete: $w_{t-τ}$ di qualche iterazione precedente a quella attuale $t$ (Nota : $w_{t-τ} \\neq w_{t}$  ). In pratica si fa una specie di snapshot dei parametri di rete usati in precedenza. Questa copia è chiamata **target network**.\n",
        "\n",
        "$\\;$"
      ],
      "metadata": {
        "id": "OHrn89xJ3C7_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## DQN - Ms Pacman game\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "Per maggiori info sul gioco:\n",
        "\n",
        "https://gymnasium.farama.org/environments/atari/pacman/\n"
      ],
      "metadata": {
        "id": "tbHOU79PrRMW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Moduli"
      ],
      "metadata": {
        "id": "dB8RIPq52Rkb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install gym[atari]\n",
        "!pip install gym[accept-rom-license]\n",
        "import gym\n",
        "\n",
        "import numpy as np\n",
        "import random\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "from collections import deque\n",
        "import os\n",
        "\n",
        "import tensorflow as tf\n",
        "print(tf.__version__)\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Activation, Flatten, Conv2D, MaxPooling2D\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "%matplotlib inline"
      ],
      "metadata": {
        "id": "c3cbFb16rQrc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Controllo uso GPU\n",
        "from tensorflow.python.client import device_lib\n",
        "device_lib.list_local_devices()"
      ],
      "metadata": {
        "id": "0q2i-yylIrH2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Rendering"
      ],
      "metadata": {
        "id": "ksLFQ4EUTVCN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#codice ispirato:\n",
        "# https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_12_02_qlearningreinforcement.ipynb\n",
        "\n",
        "try:\n",
        "    from google.colab import drive\n",
        "    %tensorflow_version 2.x\n",
        "    COLAB = True\n",
        "    print(\"Note: using Google CoLab\")\n",
        "except:\n",
        "    print(\"Note: not using Google CoLab\")\n",
        "    COLAB = False\n",
        "\n",
        "if COLAB:\n",
        "  !sudo apt-get install -y xvfb ffmpeg x11-utils\n",
        "  !pip install -q gym\n",
        "  !pip install -q 'imageio==2.5.0' # 'imageio==2.4.0'\n",
        "  !pip install -q PILLOW\n",
        "  !pip install -q 'pyglet==1.3.2'\n",
        "  !pip install -q pyvirtualdisplay\n",
        "  !pip install -q tf-agents\n",
        "  !pip install -q pygame"
      ],
      "metadata": {
        "collapsed": true,
        "id": "vlNsa0xYTbvR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import tf_agents #TF-Agents is just used to render the mountain care environment\n",
        "from tf_agents.environments import suite_gym\n",
        "import PIL.Image\n",
        "import pyvirtualdisplay\n",
        "\n",
        "from gym.wrappers.record_video import RecordVideo\n",
        "import glob\n",
        "import io\n",
        "import base64\n",
        "from IPython.display import HTML\n",
        "from pyvirtualdisplay import Display\n",
        "from IPython import display as ipythondisplay\n",
        "\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()\n",
        "\n",
        "def show_video():\n",
        "    mp4list = glob.glob('video/*.mp4')\n",
        "    if len(mp4list) > 0:\n",
        "        mp4 = mp4list[0]\n",
        "        video = io.open(mp4, 'r+b').read()\n",
        "        encoded = base64.b64encode(video)\n",
        "        ipythondisplay.display(HTML(data=''''''.format(encoded.decode('ascii'))))\n",
        "    else:\n",
        "        print(\"Could not find video\")\n",
        "\n",
        "\n",
        "def wrap_env(env):\n",
        "    env = RecordVideo(env, './video',  episode_trigger = lambda episode_number: True)\n",
        "    return env"
      ],
      "metadata": {
        "id": "xF2KjCBHTxMT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Testing Pacman episode"
      ],
      "metadata": {
        "id": "alcM2JiRT6p4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#env = wrap_env(gym.make(\"MsPacman-v0\",render_mode='human'))\n",
        "env = wrap_env(gym.make(\"ALE/MsPacman-v5\",render_mode='human'))\n",
        "\n",
        "#help(env.unwrapped)\n",
        "print('State space  :', env.observation_space)\n",
        "print('Action space:', env.action_space,': ', env.get_action_meanings())\n",
        "print(\"\")\n",
        "\n",
        "state = env.reset()\n",
        "\n",
        "    # NOTA\n",
        "    #print(f\"State shape ={(state.shape)}\") # (210, 160, 3)\n",
        "    # ovvero:\n",
        "    # 3 elementi nell'axis 2:  [0, 0, 0]\n",
        "    # 160 elemnti nell'axis 1 : [[0, 0, 0],[0, 0, 0],...,[0 0 0]]\n",
        "    # 210 nell'axis 0 : [ [[]], ... , [[]]\n",
        "    # Possiamo anche dire che abbiamo 210 matrici a 160 righe e 3 colenne\n",
        "\n",
        "    # state:\n",
        "    #[[[  0   0   0]\n",
        "    # [  0   0   0]\n",
        "    # [  0   0   0]\n",
        "    # ...\n",
        "    # [  0   0   0]\n",
        "    # [  0   0   0]\n",
        "    # [  0   0   0]]\n",
        "\n",
        "    #[[228 111 111]\n",
        "    # [228 111 111]\n",
        "    # [228 111 111]\n",
        "    # ...\n",
        "    # [228 111 111]\n",
        "    # [228 111 111]\n",
        "    # [228 111 111]]\n",
        "    # ...\n",
        "    # [[  0   0   0]\n",
        "    # [  0   0   0]\n",
        "    # [  0   0   0]\n",
        "    # ...\n",
        "    # [  0   0   0]\n",
        "    # [  0   0   0]\n",
        "    # [  0   0   0]]]\n",
        "\n",
        "done = False\n",
        "\n",
        "i = 0\n",
        "t_r = 0\n",
        "while not done:\n",
        "\n",
        "    action = env.action_space.sample()\n",
        "    state, reward, done, info = env.step(action)\n",
        "    t_r += reward\n",
        "\n",
        "    # Debug\n",
        "    if i % 100 == 0 or reward != 0 :\n",
        "      print(f\"Step {i}: Info{info}, Reward={reward}, Action={action}\")\n",
        "\n",
        "    i += 1\n",
        "\n",
        "    if done == True:\n",
        "      print(f\" \\n - Terminal state occurs in\\n   Step {i}:  Tot Reward={t_r}\")\n",
        "\n",
        "env.close()"
      ],
      "metadata": {
        "id": "z1Z7DRKBbAtD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "show_video()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "-4ABOPiEeU7K",
        "outputId": "5bf61f06-cbe4-4bfa-aae1-898755370de2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Preprocess the game screen\n",
        "\n",
        "Motivazioni:\n",
        "\n",
        "L'impiego diretto dell'immagine grezza (210 x 160 x 3 pixel) della schermata di gioco come input per la rete presenta un certo impegno computazionale, che ostacola e rallenta l'efficienza dell'apprendimento.\n",
        "\n",
        "Soluzione:\n",
        "\n",
        "Per far fronte a tale problema, viene implementato un processo di pre-elaborazione dell'immagine della schermata di gioco prima di fornirla al DQN. Questo processo consta delle fasi:\n",
        "\n",
        "- Crop and resize\n",
        "\n",
        "- Convert to grayscale\n",
        "\n",
        "- Normalize\n",
        "\n",
        "- Reshape (to 88 x 80 x 1)\n",
        "\n",
        "La funzione preprocess_state racchiude il processo di pre-elaborazione descritto. Essa riceve come:\n",
        "\n",
        "**Input preprocess_state**: lo stato del gioco (l'immagine della schermata di gioco grezza) ---> **Output preprocess_state**: restituisce lo stato del gioco proccessata (un immagine normalizzata 88 x 80 x 1)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "eCWZfk1g2qO2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#codice ispirato :\n",
        "#https://github.com/PacktPublishing/Deep-Reinforcement-Learning-with-Python/blob/master/09.%20%20Deep%20Q%20Network%20and%20its%20Variants/9.03.%20Playing%20Atari%20Games%20using%20DQN.ipynb\n",
        "\n",
        "color = np.array([210, 164, 74]).mean() # 149.33\n",
        "\n",
        "def preprocess_state(state):\n",
        "\n",
        "    #crop  - resize the image\n",
        "    image = state[1:176:2, ::2]  # (88, 80, 3)\n",
        "\n",
        "    #convert the image to greyscale\n",
        "    image = image.mean(axis=2)\n",
        "\n",
        "    #improve image contrast\n",
        "    image[image==color] = 0\n",
        "\n",
        "    #normalize the image\n",
        "    image = (image - 128) / 128 - 1\n",
        "\n",
        "    #reshape the image\n",
        "    image = np.expand_dims(image.reshape(88, 80, 1), axis=0)\n",
        "\n",
        "    return image  # (1, 88, 80, 1)"
      ],
      "metadata": {
        "id": "laoK9drj2qjI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Test\n",
        "\n",
        "env = gym.make(\"ALE/MsPacman-v5\")\n",
        "\n",
        "#Originale\n",
        "state = env.reset()\n",
        "print(state.shape)\n",
        "plt.figure(figsize=(3,3))\n",
        "plt.imshow(env.render('rgb_array'))\n",
        "plt.show()\n",
        "print(\"\")\n",
        "\n",
        "#Dopo il processo\n",
        "pr_i = preprocess_state(state)\n",
        "pr_i = pr_i.squeeze(axis=0)\n",
        "print(pr_i.shape)\n",
        "plt.figure(figsize=(2.5,2.5))\n",
        "plt.imshow(pr_i)\n",
        "plt.show()\n",
        "\n",
        "env.close()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 590
        },
        "id": "u4qOQsXi8fh4",
        "outputId": "316eae75-eb5a-4b0b-a7b0-b64adf22d7f0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(210, 160, 3)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 300x300 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOoAAAEXCAYAAACj79g9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5QUlEQVR4nO29eXwUVb73/6nqPVt3OisBwqIBAgJCwBiE0ZEogbmIwn2uMMwzLqiPDviMoKPDvXdE5s7vxoVx3FB0Hkf0znV3UFGEi6ziRJCwDVsgIRBCNrJ0utNJr3V+fwS6+1R10kt6SSXn/Xr1K6nqb9VZqj5dVed86/vlCCEEDAajX8PHuwIMBiMwTKgMhgxgQmUwZAATKoMhA5hQGQwZwITKYMgAJlQGQwYwoTIYMoAJlcGQAUyoDIYMiJtQ169fj5EjR0Kr1aKwsBAHDhyIV1UYjH5PXIT60UcfYdWqVVizZg0OHTqEyZMnY86cOWhqaopHdRiMfg8XD6f8wsJCTJ8+Ha+99hoAQBAEDB8+HI8++ih++9vfBtxeEATU1dUhOTkZHMdFu7oMRkQhhMBisSAnJwc8H9y1UhnlOklwOBwoLy/H6tWrPet4nkdxcTHKysr8bmO322G32z3Lly5dwvjx46NeVwYjmly8eBHDhg0LyjbmQm1ubobb7UZWVha1PisrC6dPn/a7TWlpKdauXStZv/eXv0SSWh2VejIY0aLD4cBP3nsPycnJQW8Tc6GGw+rVq7Fq1SrPstlsxvDhw5GRkIBkjQYA8IdjyfiyVhevKgbkw1ktGJ3s9izvb1bh0QOpcayRvHm9sA3T0pye5UqzAj/flxbHGvXOXbldWH2dBQCgU3bLLpTHtpgLNT09HQqFAo2NjdT6xsZGZGdn+91Go9FAc0WQPdHm4FHXpYhYPSONU6APit3N9ev69nccbro/naR/96fJ0bexlJiP+qrVahQUFGDHjh2edYIgYMeOHSgqKop1dRgMWRCXW99Vq1bhnnvuwbRp03DDDTfgpZdegtVqxX333ReP6jAY/Z64CPXuu+/G5cuX8fTTT6OhoQHXX389tm7dKhlgYjAY3cRtMGnFihVYsWJF1Pav5AimGJ3UuhMmJTrd3rv9JKWAfL2LsjnUqoKbeJ8nsrRu5CZ6B4EIgIMtKgBem9FJLqRpBM+yQwCOtoU+Gj3V6IDC51GmvotHbaf3EHEg1AAKAJy1KGFyeNuk5gkmp9I2x9pUsPs8I+tVAsak0O0+2KICQc/PUf7KPmNWot3pLVvLE0wUlX20TQWHT9mpagHXJvde9rAEF4bovP3pJsCh1tD7c3KqA2qfh7tmO4/qDt9TvrtNvq2+YFWgyeZ91lVwBFNF59GpdiU6XN4dJygETDDQbTrcqoKLRG6OXxajvuGgVwnYdEsLta5kRzqOm7wdPF7vwt9ENvlfZMHi8nbw/GE2PDPZ7Fl2CcDITfSg18p8C+7KtXmW67t4TN8S+t3Bh7NakaD0+p+8VpGIZ4+neJaVHPC3m1vgO1j4QJkBW+u8o91ZWrek3TO3ZuC81Xuop6U58O5NbZTN6E3ZcAjoEQ0PyX7/975U7GrUepaHJkjLvmFLJjXIMyPDjjdvNHmWCQFGbcqGy8ft5p5rOvHIGKtnucPJYdyX/gcae+MvRW3I8hH8pxd0eOyggbL57OYW6sfx3w6n4N1ziZ7lRCWRtGnBrjSU+/xw5KW4JDYTN2ehrY8DSL4MWKESAFYX3VGCyAfLTaQ2Yjctp0DbuP2czHY3R9l0ucI7QFYXR5XvdEv30yla5ybS+kvaLdqHv3YHwt9+xWULfmzE/ekiXMCyHaL+FLc5WDpF+7H7OXZWF0cJ1eXHTy/weSRtU6Td/eLiQthXzGYz9Ho9Tj38sGce9fGDenx0ISHONeuZHcWXMdbnNntvo7pfz/v1dz6e1YIZmQ7P8ql2JW77NiOONeqdpaOseG5q952ZxW5H/oYNaG9vR0pKSoAtu2GvuTEYMoAJlcGQAUyoDIYMYEJlMGTAgB31DYbRSS6szLeEtI1AOPz6oB7oZc4xWig4gpemmWJebrT59UEDhAjOOQYPwcvT2sFzoY2nvngqmZqPzU104Tfj6fPot4f1sLoidx0c1EJN1wjU/GcwuARcEWrs4QHcOdyGgfSuPCHAyoPSKaRYcWduFzU9EwwbqxJR7bOcqpaeR08f1cOKyMFufRkMGcCEymDIACZUBkMGDOpn1Eih4QkUPgMSBECXO/TfwAQF/aTmJJzkhfPBgIonUIkGeDrD6E+dQqCG/FyEo14QkBNMqBFgXYEpIk75R/6pqVen/MHCkxMsEXHK3zfnckCnfLnAbn0ZDBnArqgR4PFyA5485HvrG97t1eSvMqlll0xv0/rK88eT8eLJpD7v56ZtmfB9H0n8tk+k+EebCnmf03dQXWG+8dMTTKgRoPu5p+8HJpzn2oGIk3B+X/ELFZs7MsclEAK4iAtTDDszGAwZwITKYMgAJlQGQwYM6mfUZjuPTy+EFl1fHIYjWLbXa/APk8qzXGEOvesFAJ/V9N9sAOESjp/vniYNFYupriu8a85nF3TgQ3y8bLHH/vo2qIV6rkMZs3m1Z0/0fT7UTTjZzgNGmvUVfR8VBjisKjdEYD/Rh936MhgygAmVwZABTKgMhgxgQmUwZMCAHUwyqt34Ye5lat1du9Nwot078lpgdOCDWa2UzdSvM6l0BfeMtuLfJnrDbLgJkP9lFnw9XtYVmHDHMK9TfqONx6xttDvg17c2I88nlcP3l9W47+9GyubIPzUiQeEdVn7zbCL+eNKb7FbFEZy4g05XufyAAdvrvdHqhyW4sPO2Zsrmtm/TccEnUv4tWTa85ROtHgCu25zV65slGp7gH/Ppsh8oS8XeJm86zFFJLmybTZd98/9koN5ndLYkx4ZXptNlj/8yi0r/8OQECx641uuUb3VxmPI17aL33k2tuDHdG9e3wqzE/F3plM2+OU3I1HrHlD+/qMWThww+FgSn7mikIjysPZaM/672RspPUQk4OK+J2u/d3xlx2CdS/kSDE5/dLMoQ8E0mlWqkrwxYoXIA9SYKAEkIEwXnx0a0HxVP27j8zCVoeELZaBXSORytyEbDS20SFLSNyo+NTkGodig4af0DtclfuwPhb7/isvmgyqbb6C/8u0rUV/6mxMT96a/PdaL+VPvRTYKSUEJV+vmtErdJvBte1CYg8o6LAzZSvpIjmJQqTe7TJUoSJU6WdKRNRQXaytS6MSyBThJ1uJVOEjUy0QWjT5Iop8BRc6YAMF7vpE4mi5PDWQttc32qg5rTa+hSUHOFHKSJr85ZlDD5JGpS8wTXGcTJsaRJoq5JliY1CpQkSlx2pUUJs0/ZGp5ggqjsf5hU1Du1BrWA0Um9lz00wYUsnyuhQIAjoqRbeclOJKu8/dnl5nCqne7PiQYHVD6qarXzVA4eQJoA6qJVgct2OkmUOOlWhVlJBS5LUAgYJ0o2drSNTjbW10j5A1aoDEZ/gqW0YDAGAUyoDIYMYEJlMGTAgBn1NWoEDEtw9WrTaFOEHCwsSSnAoKaHems7FQh1XC9L6/Y7itsbFidPZfQOBgVHMETnDmwYI+q7FCFHVtCrBCSrQnPVdwocGn0yhQcHoQYKAcDk4KnpuWBQ8QRZ2t77PFXdt6GgASPUf5tooeY7/dGdcVzVq42YxSO7AmYcD4b3Z7ZS+VGDIZzgZjk6N8pE88fxRJxxPBhWjOuggpsFQ7j5Ub8vudxrxvFgGK934utbWwIb9gF268tgyAAmVAZDBjChMhgygAmVwZABA2YwKRzG650ondJOrVvynTHk9Amr8i24OcvuWW6x87i/zNjLFuGh5IjE+bv0eDJ+aNb0sEX/Y0aGHU9NoAf9Fu5Ji0rM3XdmtMLoM2K/q0GDl04n97KFlESlgPdn0i9uPHVIj9Pm0AYl+8qgFmqSkqAgjfbjDDVXJtD91ojvfurDjN8TCA7AVKOTcsoXTx31d1LVAtVXhEQv8u4kg5NKaeGbfDhYFBwk50hiiC80RIJBLdTaTgXWnaBj79jDiE7/9SUdqizeruxwRefUcxPgj6II8pWW2P6y95UzZqWkz8MNGBeIDWcSKVGJnfaDwe7mJPUNdbopEgxqodZ1KUK+FfLH1jotttZpAxv2EQFcROobT85aVHjpdGx+XP5c2fcAaHahf/Q5G0xiMGRAxIX6zDPPgOM46jNu3DjP9zabDcuXL0daWhqSkpKwaNEiNDY29rJHBoMRlSvqhAkTUF9f7/ns27fP893KlSuxefNmfPLJJ9izZw/q6uqwcOHCaFSDwRgwROUZValUIjtb6g/b3t6Ot99+G++//z5uvfVWAMA777yD/Px8/PDDD7jxxhtDKkcg0RuIiDQEYdQ1zLbJpU96JIzjGs/wB6Ee23COT1SEevbsWeTk5ECr1aKoqAilpaXIzc1FeXk5nE4niouLPbbjxo1Dbm4uysrKehSq3W6H3e6dpzSbu53kJ2zOAscHP4gTz/O3+Nv0kKchwqnvxU4FRvwt9JcGokU4bfj/jifjP4+HNoATz2N7rE0VUp8TwRbYSETEhVpYWIiNGzdi7NixqK+vx9q1azFr1iwcP34cDQ0NUKvVMBgM1DZZWVloaGjocZ+lpaVYu3atn2+4sJMGxx4uRidT38p5fdRh5Ou8DgknOlOw4vz1fa5VaMSqryJFqPUN/ZyNuFDnzp3r+X/SpEkoLCzEiBEj8PHHH0OnCy/B0erVq7Fq1SrPstlsxvDhw/tcVwbNX0aXI0/bQb03OymhHW+PLseycwVxrBkj6vOoBoMBY8aMQWVlJW677TY4HA6YTCbqqtrY2Oj3mfYqGo0GGo183OTkyBujDiNP14Hn68ag2paABcZ6pCvteLtpJJIULqwfeQTLY35lZVwl6vOoHR0dqKqqwpAhQ1BQUACVSoUdO3Z4vq+oqEBNTQ2KioqiXRVGL0zQmaHiCKpsiRinsyBfZ4HJrYbJrcKdxnpMSDAH3gkjakT8ivrEE09g/vz5GDFiBOrq6rBmzRooFAosWbIEer0ey5Ytw6pVq2A0GpGSkoJHH30URUVFIY/4BkKnEPCrsXSUgP86l4Amn3AdQ3UuLB7VRdm8ejqp14jx/pg3tAv5PtEbOpwc3jxLe8XcM9qKdJ9YtRc6FPi0hg5v+n/HWag4tAea1fjOJxI9D4KV4zuobb64qKXcCFNUAh7Mo9v99tlEKvbv6CQX7sql2+3r49zs0sDs7j417AKPSw4tlBzB4+NpZ/q/1ego/9lUtYD7r6XLfutMIiw+oU3GJDsxfzg9mPKnk0kQfJ7bfpJpx3SfKPgON/BqBT249C8jOjE80Rv+5LKNx3uiyAwP53Ug0Sf270mTEt/Uhfb4peEJVoyj+/yD6gTKjTBL68YvRndSNusrkmBzR278JOJCra2txZIlS9DS0oKMjAzMnDkTP/zwAzIyusNk/OlPfwLP81i0aBHsdjvmzJmD119/PdLVQIKCYGU+3cHb6rS0UBMEic1bZxJDFurcHBvuyvWefPVdvESovxzdSYVi2duolgh1xVgrFXH9tYpESqgKDnhsXAfllH/CpKSEqldJ2/TZBR0l1FFJLolN7RkABJiR3IJLDh0sbhWGqrowLcmEJqcWKh5YOYbe5lCLihKqUS0t+4PqBFh8ItDkpdBlEwK8fCqJmrKYlWWnQrF0ODmJUP85twszMr1iPtWulAj1wTwr5ZT/6QVd6EL1cx7tbqCTKGfr3BKbv1Qm9m+hfvjhh71+r9VqsX79eqxfvz7SRVM4CYcDzbRPaafIWd7iktq4wxhurOpQUvtp9ZOR+rhJhXant3x/GcfLW1TQ+ETTr7XSzt8EwI8tdH3F+U3sgrRNNtEPT7uDl9gMufIWywOZF6j1kxK7b3kFP/0pDrxmc0ttHKKXe9r8lC3u8lqrgrLxd8KfNiuh9Bn0umCV9ueRNhVSrd4KnOsI3ZneTSCpr/ilC6tL2iZ/qU/6gqwj5SPnGSCEedRweOBaaw/BzeQyLRQce8fvgbqXKIk2gcctJ38SwxrFAoILCxv6HNwsZAQbUPcMi5TPYAw0BvVrbgwaoZeXuGV32zXAYFdUhocFFUU42ZWMp2vz8ZfLI7CtPRMPnZuCyy41fnpyVryrN6hhV1QGAOD2UzNhIzweqb4eLsKDR3ceVhfh8L/OFGKgPZPLDVkLdcdtTUhSB++x9MvvjajoY1AqBQfsn9sU2DACvFuViNfPhBalIEfnxqZbmgMbxog7dqWHnGri0bEd+MXo0CLlh0skbinz9U5snNEa2PAKHQ47Zr8bWhmyFmqOTkCyJvhxcFUEjgrHdc+/xoKUMAKXKTgSs/oFQzjB4lLUQr9qQyDUfGh9blGE3jb2jMpgyAAmVAZDBjChMhgyQNbPqH0lSSlQzvQAcKhVFZWo7ZGAA8E0UTDosxalxI1QzLE2ZUT9TntCpyCYmNp7aslUtYBrk2mbgy2qfhsAQMERTDXSfX6qXRlyDtW+MqiFOl7vwt9uoVNE5H+RBUuUAmj3FSUH/O3mFsop/4EyA7YGcDT/1f5UnPfjCxtprklyYc+c3nOzzsiw480bTZ5lQoBRm7Lh6qceFYlKgk2ic2TBrjSUt6pjWo9BLVQ3AawiUfbT88VDp+jK2F+v/j3hIpykz/s74vrGI3jcoBZqeasaY7/oP4HAAuEknKzq649tdVpZtcHs5PtFfdlgEoMhA5hQGQwZwITKYMgAJlQGQwbIejDpiXIDVEr/TvnJSgGlU/seOW9XgwbNdkOf9xMOZ/yEaxkMfF6jwwlTfPK+HmuLTLm/PZTS41yr02X3u743ZH0mfH1J22MoljS1OyJCrepQoiqMTNWM8DnRrsKJMJIO9ye+vqRDW0+OKGEky2a3vgyGDGBCZTBkABMqgyEDBvXDFw8CrYL2B+t20fM+Qyg5IgmjKbbR8AQKzmtDAHS56d9AnYJQOb8Ewkni7SaIXih2Eg5OyoYgQVRfu8DJyo1QwRFoAvSniidQcWIbuj+1PAFP9TmHLpF7pU4hUK7+LsKJgqtL+9MhcHCR3m1sbg5CjF8iGNRCnZbmDOiUf+81nT3E9fWyrsAkiZQ/fUsWZfPVT5slkfJ/vi+NsjnyT02SSPnPHvfGfVVxQMWCxpCd8vsTJTm2gE75T06wSCLlj/uS7vP3bmqVRMq/7dsMymbfnMuSSPmPHTRQNqcWNPYa1zdFRXDyjkZqm3g45bNbXwZDBgzYSPlpajeOzqeDkJXsSMdxn/k5HoRKIQHgyu0Tfeur4nu3UUtufTnJ+59anoDj6FtfuyC9VfPFJXBwim7DdH5u1XxvfYcnuFA2l37VbObWjKBfc1s/cSTGJXuv0CctXXj0H+eD2tbfa243bMmk8rQo/DxKiPtTxREqXUW3DX1N0Yhvff08SmhFjxtuP7e+4v50+rn1FdvYRbe+k1Md+PpW+s5s4uasXqZnQo+UP6hvfQU/zzViXISDK4BN98Hv3cYWhI34ZJQSuL594f9dPxp5iRqoeG89Jqbo8OfJo/Dg0eqIlOEmgdvgJBycAWzswfS5O5BNMP0Z3T4PlkEtVIaX9ZNGIi9Rixcq63C+04752anIUCvxl5rLSFIq8NrEkVgR5JWVEXnYMyoDADAhWQcVz6HKasfYJB3GJ+tgcrphcrqxIDsV45PlM2A1EGFCZUhocTjR7uxOEmwXBFyyOQJswYg2sr71vWNYF1RK/8GMk5SRGSO7NtmJyanOwIZRoMKsoga/YsEMYxIu2RzocLmRo1OjwJCIJnts2z/R4MSYlPj0+ZFWdUR8u+cP6+ox5IzTZceXdaHtT9ZCfW5qO5I1wae0CIdbshzUPGosea0iMeZCXTYik1qelNKdFd3mjl3k+juGd1HzqLHk3w6nRESo/zml53PGYrfjywOh7Y/d+jIYMoAJlcGQAbK+9WVEFoH0HAZbdl4xAwx2RWV4uOvAGZzq6MKailq8c/Eytl1ux/85Wo3LDhdm//1UvKs3qBnUV9QCowMfzKLzWk79OjPm6QqCRcURnBA5iC8/YMD2ev9ulKFQUnYaNoHgV8fOwyUQ8BzAgYOLENx98Gyf9+8pJ8eGV6abqHXjv8wSue31H1JUAg7Oo11R7/7OiMMsUn7sUHCg3lYB+n9ebZ2CUG/PKLjI3JTaroR/d1z52/2n+397BEPDKzhC9bkcPM3F50g8fsYHtVBPtitxxy76VTNrP/Dr7AkXARbsput7zhL4EG64sQ32GLRL/G6vP76/rJH0ubsfi9Xq4iT1rYhD0LlBLdQOF49DMb6F6QsEXFj1vc7Qe4a1WGJyyKvP3SS8Po80/fNhjMFgUDChMhgygAmVwZABIT+j7t27Fy+88ALKy8tRX1+PTZs24c477/R8TwjBmjVr8Oc//xkmkwk33XQT3njjDeTl5XlsWltb8eijj2Lz5s3geR6LFi3Cyy+/jKSkpJDqcqlTgUSXIrDhFZwRcFclBKjtDL7MvtAeIJO4P1yEw0VrbOoXDOEMFLU7+Ji1YViCmxpFDweHEFqfWx2hty1koVqtVkyePBn3338/Fi5cKPn++eefxyuvvIJ3330Xo0aNwu9+9zvMmTMHJ0+ehFbbPd+3dOlS1NfXY/v27XA6nbjvvvvw0EMP4f333w+pLsXfZvQYiiVauAlQtDUD/XUip75LgaKtmYENe0CjcnuCfbmE7ugWaqUAmzN24n+tIgmvVYT2ox0eBBcWNqCvLTvVrgqtzwVbYBsRIQt17ty5mDt3rt/vCCF46aWX8O///u9YsGABAOC9995DVlYWPv/8cyxevBinTp3C1q1b8eOPP2LatGkAgFdffRXz5s3DunXrkJOTE3IjGJGBA8FfHjqGicMtAICvDmfio7IcPP/zUyh59gb07GDIiDYRfUatrq5GQ0MDiouLPev0ej0KCwtRVlYGACgrK4PBYPCIFACKi4vB8zz279/vd792ux1ms5n6MCLP/6zej+uGWTzLP7u+CY//rAq/fGMyfvj993GsGSOiQm1oaAAAZGXRMW2zsrI83zU0NCAzk75NUCqVMBqNHhsxpaWl0Ov1ns/w4cMjWW3GFTgOnue1t3bm4pnP8jBpuAX/78FjuPk/iuJbuUGOLBweVq9ejVWrVnmWzWYzE2sU+PlrU8BzBP+6oBJLii7BLXDgeUCjiu0zKkNKRIWand0dzbyxsRFDhgzxrG9sbMT111/vsWlqop2cXS4XWltbPduL0Wg00IQYySFFJeC/bqId7lceNOCcz9v74/VOlE5pp2yWfGeUpE8IxKp8C27O8ua8bLHzuL/MSNm8Or0NuYluz/KRNhXWHNVTNh/NaqHc8D6r0eE9n6jtSo7gs5vp+LGlx5PxQ7O3bzK1bvz5xjbK5uH9qaj3ia07Lc2B302kHx/+eU8amszd+1n39WgYEl24aUwrHimuAdAdu/iTn9Blrz2WQnntDE1w4fUbTJTNsrJUNNu9Zc/IsOOpCRbKZuGeNCo28b3XWHHX8C7Pcpebw+LvaDe+P1zfjokGb7iW81Ylfv2jgbJ5Z0YrjGrvUP+uBg1eOp2MUEhUCnh/Jn0ePXVIj9Nmb+SNa5Od+GMBfR794nsjLM7I3bBGVKijRo1CdnY2duzY4RGm2WzG/v378cgjjwAAioqKYDKZUF5ejoKCAgDAzp07IQgCCgsLI1YXFUdQkEbH3RE7VycppTaKMMZLRiW5qP3Ud0kP0Hi9i0pp4S+ezhSjk6pjWTPtusYBmGp0UtMJBjU956ThpW0S53rRqwSJDcfB89JpTUsCalqA4WlesfCAZBu9ii5by0tt1KKuSFXTZRMiHT8fmuCmbDqc0r4ak0z3ufjYAsAkg5NKaVEdRogVBSdtU6KorEQ/55EywuNuIde8o6MDlZWVnuXq6mocOXIERqMRubm5eOyxx/CHP/wBeXl5numZnJwcz1xrfn4+SkpK8OCDD2LDhg1wOp1YsWIFFi9eHNER3043h3Un6CH+Jht91tR2KiQ24uj1wfD1JR2qfJzjO/yIcOO5BKT5iKrGz7zby6eTqORIP7bQQnUT4I8n6fpWWuiYSu1OXtImk2g+9lyHUmLjO99528TLyMuyIm+IN26Ri0CyjfjEb3VIy7aIRHbGLC1b/HLO3kYNrD7bOfwck48v6PD3y97+abZLfxw3nEmkRHUqjOTIdj/nkW/kfwBo6JKeR5EO2h1ySovdu3fjpz/9qWT9Pffcg40bN3ocHt566y2YTCbMnDkTr7/+OsaMGeOxbW1txYoVKyiHh1deeSVoh4dgUlpEigeutfaQJGrgTVXcdt1laFQCFhfVYVKu9/a0oV2N20tvjGPNosWVedRekkRFhViktLjlllvQm7Y5jsPvf/97/P73v+/Rxmg0huzcwIg+T91RhcwUOoav1abAiYuhPdcxIg/z9WV4aDBp4PC5be+08yirNGDlXyfEsVYMgAmV4cMvXp+CozUpsNp5WO08vjmaiVVMpP0CWcyj9gydVi+wNRCvZ0t/9QzdJU+6RXht6rnkB96aRC331r+RKjuc/USmPyNFaCWTMGI6ylqoJ+Y3hhQpf97O9JhHnr/K9uJmjEnxTs9816TGUlHG8UCoOKDqLtp766EfUrG1LrQBtdnZdrwzoy2wYQB++X0qdjeGVvbPhtrwRqGJWjdalHE8GD6c1YqiDO/z9Ol2JW7fkdHLFtFjUqoTX/20JbDhFSx2Oya8GVoZshYqz3V/5AAHuq7hVpsDRK9lhRdwKBL9Fu4ufMsON7iZpD/jeB6I6xKIcPqePaMyGDKACZXBkAFMqAyGDJD1M2pfGapzYfGoLmrdq6eT/Lqs9ca8oV3I9/Hj7XByePNs6BEK/u84C1Q+P50HmtX4rin0lxEezKNTFr59NhGmCDqI90SqWsD919Jlv3UmEZYQMw/8JNOO6enegSKHG3i1InSni4fzOpCo8j4EnzQp8U1daJnTNTzBinEd1LoPqhMkboTRZnALNUHAynz6ILx1JjFkoc7NseGuXG94jfouPiyhrhhrpZzLX6tIDFmoepW0TZ9d0MVEqEa1tOwPqhNgCTGs8KwsO5UftcPJhSXUB/OslFP+pxd0oQtVQSRt2t2gYUKNJRYXhwPN9HRNOMG4qjqU1H5a/TiIB0N5iwoan9fcasMI8GUXpG2yhfGiQTjY3NKyHWEElKu1Kqj92MJ0cD/SpkKq1VuBcx2h96ebQNImfy9dRJtBLdRT7Sos3JPe5/28dCoZL53quz/skhDnVf3RZFNEpE3hcKkrMmW/ey4xIo7xy0TvBIeD1cXHrT99YYNJDIYMYEJlMGQAEyqDIQOYUBkMGTBgBpP+cCwZX9b2PvR+2Rb679IH53XYcim20fijzb4mDW7YEn40/au0hDm63Z8o+obuh3Y/8ZkCcbJdFbA/78rtwurrLL3a9MaAEWqbg4/K3JbVxcPaf9KLRgS7wMV8HrB/Epl+cAbRnyZH36Z05P+TyGAMAphQGQwZwITKYMiAAfOMKkbJEUwx0kGRT5iUVBT8JKVAOdMDwKFWFRW1PUvrpiLcEwAHW1TwfW16dJILaRqvq5pDAI620TF5g2Gq0UGFrqzv4lHbGdohUvMEk1Ppdh9rU1HxivUqgYo2AXS3qbeAIhwIpomCTJ8xK9Hu40Os5Qkmiso+2qYK2Xd6WIILQ3x8dN0EVET+YJmc6qACgDfbeVEs4u42+dbuglWBJpv3eVPBEUwVnUen2pXo8HnRIEEhYIKB7s/DrSq4SORcDQesUPUqAZtuocNjlOxIx3GTt4PH6134m8gm/4ssWHx8OecPs/UQ19fLynyLxCl/+hY6UVYwfDirVeKU/+zx4OK+XiVL65a0e+bWDJy3eg/1tDQH3r2JDsUyelN2r365Gh6S/f7vfanY5ROKZWiCtOwbtmSGPGBzzzWdEqf8cV/6T3fSG38papM45T920EDZfHZzS69xfROVRNKmBbvSUO7zw5GX4pLYTNychbY+DiD5MmCFSiBNGyGOyO4mUhuxT75ToG3cfk5mu5ujbLrCdNq2uuiQXc4wnNH9tltk46/d4ezXTaTlBOrPYHCI+rMzTKf8TtF+7H6OndXFUUL1F7sp8HnERaTdvTFghdrqUGDsF73/Cpe3qgPaBOMg/sQhA544FHIVJUz5OvSrsJjaTmXANu1u1Aa0EWMXuIDbVHcELjsYXjiZjBdO9v0lh1nbAs0Vcxgf4EptdvIB23TcpIpIu3uDDSYxGDKACZXBkAFMqAyGDGBCZTBkwIAdTAqG0UkurMwPzVFaIBx+fVCPeKTGcBHgUVFW7XDmayfonXh4TEdgwwC8cSYJJ0PMOXq4VY0VBwzUunDC30QGgpentYPnQqvAi6eSw0qK3BcGtVDTNQI1/xkMLgFXhBp7CDh8fjG04Fz+yNa5Q263P/5WowtZqHVdioi0IVLcmdsVcpb5jVWJqI5OdXqE3foyGDKACZXBkAFMqAyGDBjAz6gECQp6kMDm5iBEYRBIwxMofAYkCIAud+i/gQkK2sfNSTg4KYd2aZvsAke58nEg0IlsutxcTHKHBlO2giPQ8LRNt4ug10bFE6g4sU3o/alT0EfbRbiQXxAIBh4EWkXvbeorA1aoaWoBR+c3Ueu6nfIjnx91XYEpIk75R/6pqVenfBUHVCxopFIMPlBmwFaf6O/DEtwom3uZ2q/YKT9ajE5yY88cumyxU35Jjg1v3mjyLBMCjBLlR31ygiUiTvn75lwO6JQfCSamOvH1rdF1yme3vgyGDBiwV9QWB4+8z+mrWripEQLxeLkBTx7yvfUNr5zJX9FO5C7RbZqTAGO+oNskvpWr7VTErN1iznUELntrnVZiI35j5fnjyXjxZOi5e8TctC0Tvu8jid/2iRT/aFNJ2tQV4T4fsEIFuIh3Vk90i6XvZQV+rg3cJhLDdodTtpsEtnESLqxX/MTYIvyc2BNCDPqc3foyGDKACZXBkAFMqAyGDBjAz6j9myytgEW5nX3eT9nl2CfV7QtDE1y40SebeLhkaMNIvBolUtUCbs2mfac31+oiOmfLhBonxupdeHl6e5/380CZAXVd/cfJPRDXpzoj0u7+RG6i9FjubNDCEc951L1792L+/PnIyckBx3H4/PPPqe/vvfdecBxHfUpKSiib1tZWLF26FCkpKTAYDFi2bBk6Ovr+2hWDMVAJWahWqxWTJ0/G+vXre7QpKSlBfX295/PBBx9Q3y9duhQnTpzA9u3b8dVXX2Hv3r146KGHQq89gzFICPnWd+7cuZg7d26vNhqNBtnZ/l2+Tp06ha1bt+LHH3/EtGnTAACvvvoq5s2bh3Xr1iEnJyfUKjEYA56ojPru3r0bmZmZGDt2LB555BG0tHj9IMvKymAwGDwiBYDi4mLwPI/9+/f73Z/dbofZbKY+DMZgIuKDSSUlJVi4cCFGjRqFqqoq/Ou//ivmzp2LsrIyKBQKNDQ0IDOTdpVTKpUwGo1oaGjwu8/S0lKsXbs20lWNGOsKTLhjmHfUr9HGBxFTNjoMS3Bh523N1Lrbvk3HhRg45Y9KcmHbbLrsm/8nA/VxGpXeN6cJmT6jw59f1OLJQ4a41KWvRPzoLV682PP/xIkTMWnSJFxzzTXYvXs3Zs+eHdY+V69ejVWrVnmWzWYzhg8f3ue6RgoNT6i3XsSvPMUSDqDqcnVdLODjWLY/dAr6uKhl7DUQ9Z/Z0aNHIz09HZWVlZg9ezays7PR1ES/fuZyudDa2trjc61Go4FGo4l2VcPmjyeT8U6VN5q+MwrvPAZLo02BO3alUetidUWr7ZSWfTmOWcnv/XsqVD7Ft0apLmfNSkm7zWFkLu+NqAu1trYWLS0tGDJkCACgqKgIJpMJ5eXlKCgoAADs3LkTgiCgsLAw2tWJCuetSpy3BraLBQ6BCyvzWSSwx7Fsf/zDFJu6dLr5qLc7ZKF2dHSgsrLSs1xdXY0jR47AaDTCaDRi7dq1WLRoEbKzs1FVVYUnn3wS1157LebMmQMAyM/PR0lJCR588EFs2LABTqcTK1aswOLFi9mIL4PRAyHfCxw8eBBTpkzBlClTAACrVq3ClClT8PTTT0OhUODYsWO44447MGbMGCxbtgwFBQX47rvvqFvX//7v/8a4ceMwe/ZszJs3DzNnzsRbb70VuVYxGAOMkK+ot9xyCwjpebBk27ZtAfdhNBrx/vvvh1o0gzFoGdS+vnYBuGgNbaAl3KjuDTYeCcrID+qEE0TN5uZCbndP+wmVrgiVLaYhzAGzWqsCfIjNiEaAtEAMaqEebVOjaGts5juX7ksLbBQjvr+siVm7xexs0KJoqzawYUzgcFOc5rtDRcYzSwzG4IEJlcGQAUyoDIYMYEJlMGTAgB1MSlEJ+K+bWql1Kw8acM4nr+V4vROlU+g385d8Z6TSJ8wf1oUHrvW6HbkJsHBPGny9WFflW3Bzlt2z3GLncX+Zkdrvq9PbkJvo9iwfaVNhzVE6feNHs1ooP+HPanR475zXNVHJEXx2Mx2RvfR4Mn5o9s5RZ2rd+PONbZTNw/tTKTfCaWkO/G4i/QbSP+9Jg7OXuLdqnuCTn9Blrz2WQnnkDE1w4fUbTJTNsrJUNNu9Zc/IsOOpCXRO2oV70qiYu/deY8Vdw7s8y11uDou/owfj/nB9OyYanJ7l81Ylfi3KHfvOjFYY1V6n/F0NGrx0OtnHgmDTzS3UqO+bZxOx5ZI3YkaiUsD7M+nz6KlDepw2ezMuXJvsxB8L6PPoF98bYXFG7jo4YIWq4ggK0pzUOrHDeJJSaiPOlZmlFSgbl59QPaOSXJRNfZf0AI3XuzBW7/IsW11SUUwxOqk6ljXTbmkcgKlGJ5XSwqCmK6ThpW0S53rRqwSJDccB6GXqiQck2+hVdNlaXmojdoRPVdNlEyJ13B+a4KZsOvz4zY5JpvtcfGwBYJLBSaW08Jd8eGqakzrmGTV0mxSctE2JorIS/ZxHygjP4AxYoXa6Oaw7QUdbb7LRZ01tp0JiYxfNkZW3qCgbf+fy15d0qLJ4u7LDjwg3nktAmo+oavzMJb58OolKjvRjCy1UNwH+KIogX2mhc+m0O3lJm0wOut3nOpQSm0Dzwy4CyTbiE7/VIS3bIhLZGbO0bEFU9t5GDaw+2/mbt/z4gg5/v+ztn2Y/DvcbziRSojrlJ+nyiyeTqB+KI220jd3PeSQOJtfQJT2PIh2QmyO9uRn1U8xmM/R6PU49/DCSr7gmPn5Qj48uJMS5ZgyGf5aOsuK5qd2PGxa7HfkbNqC9vR0pKSkBtuyGDSYxGDKACZXBkAFMqAyGDBgwg0kcByrFHoPRn+jr0NKAEeoLU9vx/NSBFYGdMXBgQr1C9xWVwRiYyFqoP/3rX8FzTJ4MeSGEMSMqa6E2hJGv5v77h+LiRRvMZhd+/vPuGE0Oh4Df/KYipP2sWzcW//EfVbj//mEYObLb5ay8vB3vvVcXcp0Y0WdeejpuMxpRbbPhlZqaeFcnZGTt8BAq9947FB0dLiQkKFBQoMf48d3eJG43we7drXjuuXMI1BscBzz11GgcPNiOSZOSMWOGAXp9tzdLQ4Md27c3M7H2M+7IyECyQoE6ux1JSiXyExPhIgQvXrgQ13oxh4ceaGpyQKdTYMqUFKjVHL755jIAQKHgcOutxgBbe6mttSEnR4ObbzZi3742nDvXnec0O1uDadNC/wFhRJdWpxNHLBbsamtDZWcnCvV6NNjt+MWQIbIZ1xhUQt2y5TJGjdLhuuuS0dUloLXVia4uN7Zvb8a337YEvJoC3U7kf/1rHf7lX4YgIUFxZR8Cqqo6sX17Mw4cYCPP/Y19JhNOWL1vQFndbnzc2IhstRqzjUYoZTDOIetn1L4wbJgWHAccPNiO55+vDns/M2akoqvLjU2bGrFtW3PgDRhxQ69UIlutxhmrFW5C8E1zM14YMwZl7e1wud2BdxBHBq1QDxwwYd26833ez0svncfp0/0kTD6jRxIVCswwGFCQnIz/rK7GMI0Gfxw7FgCQqVKhxu1Gf5bqoLr19UWh4KDV8tBo+tYFGg0PrZaHMtIvIDIiyj9nZmJSUhKeP38eaSoV/jxhgue7N8aPx5B+nNsIGMRCLS5Ox+bNBXj77eu6nSXC1Nm6deOweXMBfv7zIWHvgxEDOA4/NRrxbF4eOI6j5jLDmdeMNYNWqFfJzFRj27Zp2Lp1Wp+E9otf5GDbtml4+eX8yFWOERFWjxyJxVlZAID8xES8kJeHu44e9Xz/L8eOodZu72nzfsGgE2pp6Tl8/HG9Z5njuCuf0PazaNFhtLQ4+rwfRgy4cmy6/+3+3y4IWHDkCBYcOYKOfj6QBAxCoTqdBO+/X4833/R6p9hsbixdeiyo6Zmr2O0Cli8/icpK70DS3//ehjVrzkayuowIsL6mBl9cycl71mrFyopuLzS7IMAu+AmC1Q8ZVJ5Jvuj1SuTkdA8gCAJQURHeyO3IkTrodPyVerlw6VL/voUarGSoVEhTq9HlduOCzRbv6gAIzTNp0AqVwYg3zIWQwRhgMKEyGDJg0HomMWLH6CEJSNR2n2rtVidqmroCbMEQw4TKiCoZejVWLxmDbKMWCp7D8fNmPPfhWbSYHYE3Znhgt76MqKFV89jw2PUYmZ2AVzZV4b++rcHUPAOef2gCtOJcF4xeYVdURtTY/IcboVUr8KuXj+If1d1R4utb7PiP+/Lx4b9Px51P749zDeUD+1ljMGQAEyqDIQOYwwMjamQY1Hhz5fVQ8Bz+9GkVjCkq3DdnBFrMDjzx5nE0tw/uASXmmcToN4wZlojVS8ZAq1ZApeRQe9mGVzZV4Vx9Z7yrFndCESobTGJElTO1Vrz+ZTUMSd2RGhvb7EykYcCEyog6P1aY4l0F2cMGkxgMGcCEymDIgJCEWlpaiunTpyM5ORmZmZm48847UVFBp4Kw2WxYvnw50tLSkJSUhEWLFqGxsZGyqampwc9+9jMkJCQgMzMTv/nNb+ByufreGgZjoEJCYM6cOeSdd94hx48fJ0eOHCHz5s0jubm5pKOjw2Pz8MMPk+HDh5MdO3aQgwcPkhtvvJHMmDHD873L5SLXXXcdKS4uJocPHyZbtmwh6enpZPXq1UHXo729nQBgH/aR9ae9vT3ocz4koYppamoiAMiePXsIIYSYTCaiUqnIJ5984rE5deoUAUDKysoIIYRs2bKF8DxPGhoaPDZvvPEGSUlJIXa7PahymVDZZyB8QhFqn55R29u70zcYjd15W8rLy+F0OlFcXOyxGTduHHJzc1FWVgYAKCsrw8SJE5F1JSocAMyZMwdmsxknTpzwW47dbofZbKY+DMZgImyhCoKAxx57DDfddBOuu+46AEBDQwPUajUMBgNlm5WVhYaGBo+Nr0ivfn/1O3+UlpZCr9d7PsOHDw+32gyGLAlbqMuXL8fx48fx4YcfRrI+flm9ejXa29s9n4sXL0a9TAajPxGWw8OKFSvw1VdfYe/evRg2bJhnfXZ2NhwOB0wmE3VVbWxsRHZ2tsfmwIED1P6ujgpftRGj0Wig6ecpBxiMqBL00ywhRBAEsnz5cpKTk0POnDkj+f7qYNKnn37qWXf69GkCSAeTGhsbPTZvvvkmSUlJITabLah6sMEk9hkIn6iN+j7yyCNEr9eT3bt3k/r6es+ns7PTY/Pwww+T3NxcsnPnTnLw4EFSVFREioqKPN9fnZ65/fbbyZEjR8jWrVtJRkYGm55hn0H3iZpQeyrwnXfe8dh0dXWRX/3qVyQ1NZUkJCSQu+66i9TX11P7OX/+PJk7dy7R6XQkPT2dPP7448TpdAZdDyZU9hkIn1CEyl5zYzDiBAvAzWAMMJhQGQwZIEuhyvBuncGQEMp5LEuhWiyWeFeBwegzoZzHshxMEgQBFRUVGD9+PC5evBj0A7lcMJvNGD58OGubzAi2bYQQWCwW5OTkgOeDu1bKMhQLz/MYOnQoACAlJWXAHfCrsLbJk2DaFuqshSxvfRmMwQYTKoMhA2QrVI1GgzVr1gxIZ33WNnkSzbbJcjCJwRhsyPaKymAMJphQGQwZwITKYMgAJlQGQwYwoTIYMkCWQl2/fj1GjhwJrVaLwsJCSQwmOfDMM8+A4zjqM27cOM/3wWQc6C/s3bsX8+fPR05ODjiOw+eff059TwjB008/jSFDhkCn06G4uBhnz56lbFpbW7F06VKkpKTAYDBg2bJl6OjoiGEr/BOobffee6/kOJaUlFA2kWib7IT60UcfYdWqVVizZg0OHTqEyZMnY86cOWhqaop31UJmwoQJqK+v93z27dvn+W7lypXYvHkzPvnkE+zZswd1dXVYuHBhHGvbM1arFZMnT8b69ev9fv/888/jlVdewYYNG7B//34kJiZizpw5sNlsHpulS5fixIkT2L59uydw3kMPPRSrJvRIoLYBQElJCXUcP/jgA+r7iLQt6FgQ/YQbbriBLF++3LPsdrtJTk4OKS0tjWOtQmfNmjVk8uTJfr8LJuNAfwUA2bRpk2dZEASSnZ1NXnjhBc86k8lENBoN+eCDDwghhJw8eZIAID/++KPH5ptvviEcx5FLly7FrO6BELeNEELuuecesmDBgh63iVTbZHVFdTgcKC8vpyLx8zyP4uJiTyR+OXH27Fnk5ORg9OjRWLp0KWpqagAEl3FALlRXV6OhoYFqi16vR2FhIZU9wWAwYNq0aR6b4uJi8DyP/fv3x7zOobJ7925kZmZi7NixeOSRR9DS0uL5LlJtk5VQm5ub4Xa7/Uba7ynKfn+lsLAQGzduxNatW/HGG2+guroas2bNgsViCSrjgFy4Wt/ejllDQwMyMzOp75VKJYxGY79vb0lJCd577z3s2LEDzz33HPbs2YO5c+fC7XYDiFzbZPma20Bg7ty5nv8nTZqEwsJCjBgxAh9//DF0Ol0ca8YIhcWLF3v+nzhxIiZNmoRrrrkGu3fvxuzZsyNWjqyuqOnp6VAoFJLRT99I/HLFYDBgzJgxqKyspDIO+CLHdl6tb2/HLDs7WzIY6HK50NraKrv2jh49Gunp6aisrAQQubbJSqhqtRoFBQXYsWOHZ50gCNixYweKioriWLO+09HRgaqqKgwZMgQFBQVQqVRUOysqKlBTUyO7do4aNQrZ2dlUW8xmM/bv3+9pS1FREUwmE8rLyz02O3fuhCAIKCwsjHmd+0JtbS1aWlowZMgQABFsW6gjX/Hmww8/JBqNhmzcuJGcPHmSPPTQQ8RgMFD5VuXA448/Tnbv3k2qq6vJ999/T4qLi0l6ejppamoihATOONCfsFgs5PDhw+Tw4cMEAHnxxRfJ4cOHyYULFwghhDz77LPEYDCQL774ghw7dowsWLCAjBo1inR1dXn2UVJSQqZMmUL2799P9u3bR/Ly8siSJUvi1SQPvbXNYrGQJ554gpSVlZHq6mry7bffkqlTp5K8vDwqPUsk2iY7oRJCyKuvvkpyc3OJWq0mN9xwA/nhhx/iXaWQufvuu8mQIUOIWq0mQ4cOJXfffTeprKz0fB9MxoH+wq5du/xGgr/nnnsIId1TNL/73e9IVlYW0Wg0ZPbs2aSiooLaR0tLC1myZAlJSkoiKSkp5L777iMWiyUOraHprW2dnZ3k9ttvJxkZGUSlUpERI0aQBx98UHLRiETb2PuoDIYMkNUzKoMxWGFCZTBkABMqgyEDmFAZDBnAhMpgyAAmVAZDBjChMhgygAmVwZABTKgMhgxgQmUwZAATKoMhA/5/fJMgU2VB2+wAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "(88, 80, 1)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 250x250 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAN4AAADwCAYAAABvwdrxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABBu0lEQVR4nO2dd3gc1b33PzOzfbXSSrItWa4yNriCjatsSCgGX0pCQkm4l9xgQkICNs3JTWIuJfACJrlvbrgkDgQuGPKGkpDQa4hNCca429jYuGEs27JkySqrtm3mvH/MelYrW9LO7kq7MvN5Hj2P5uyZme+end+c9ju/IwkhBBYWFn2KnG0BFhZfRizDs7DIApbhWVhkAcvwLCyygGV4FhZZwDI8C4ssYBmehUUWsAzPwiILWIZnYZEFLMOzsMgCvWZ4S5cuZeTIkbhcLmbOnMmaNWt661YWFv2OXjG8P//5zyxatIi77rqLDRs2cNpppzFv3jwOHz7cG7ezsOh3SL3hJD1z5kymT5/O7373OwA0TWPYsGHceOON/PznP+/2XE3TqKqqwufzIUlSpqVZWPQaQgiam5spKytDlruv02yZvnk4HGb9+vUsXrzYSJNlmblz57Jq1apj8odCIUKhkHF88OBBxo8fn2lZFhZ9xv79+xk6dGi3eTJueHV1daiqSklJSUJ6SUkJn3322TH5lyxZwt13331M+sa1A/HlyQSFYOGey9izsyzTUjPCoJFHeGTssxTGXnBfRB3c8MlVtFd7syusH+IZ3MLDk55muC0CQIMGP9z+b9TuK8qysuNz8tgD/E/5i7hiLbPmFo0p02vx+Xw9nptxwzPL4sWLWbRokXEcCAQYNmwYvjwZn08GTVDZMoTCvZ4squyaWr8dZ54Nn6IBoIU9hA8PoHCvNWBsloDNjfB68DmaAWhWFeoaSinca8+ysuPzxdAheH0ynk5domS6SBk3vAEDBqAoCjU1NQnpNTU1lJaWHpPf6XTidDozLcPCIqfJ+GvZ4XAwdepUli9fbqRpmsby5cupqKjI9O0sLPolvdLUXLRoEVdffTXTpk1jxowZPPjgg7S2tnLNNddk5Prtg0CMbcFmUwEIhew4t3pwNMXzNJdrOMubjeO2Rje+Tx0osXEczQaBcVE8A1vj1z2YR8FOBWLjvJE8aJ/YjssTBkAIiehOH96D5kZb+0pLMuXSmZAfIhPacDj1flU0qiBvz8NVG8/TOlRgG9OMJOli2ludeLe6sMXkChkCJ6u4y1ri37HWS/52G3JUP1Zd0DwhjKegPX7vvT58JpvkfaUlmXJJh14xvG9/+9vU1tZy5513Ul1dzeTJk3nrrbeOGXBJlVBZhBdnPMYARS+UreFiFh66BkdTvOAKx9bz4qlPGMePN8zgL3vOij/sdjj/9C3cUfqOkef7hd+mevcIJP25JeIT3DX9Vc7xfAFAm5C4MLwADprrb/aVlmTKpTOhARoPz3ia8Y4GAGpUB5c3LsBV26FfNbKNV6c/gjNm4/9oG8WSfZdha9UThAzjJ1XyyKjnjVPuqvoX1u46FWIPe9QFP5j2T77rX2/k+br9WsTe4iRKsAN9pCWpckmDXhtcWbhwIQsXLuyVaytNCv9dfR75Nv3JrWrPR2lLrIXqDufzQM25xvH6uqHGGw9AVmFV1UgekOJ59lQPxNthVlMJSjxTNZM1vpMACGk2tHrz/dG+0pJMuXTG1irx6KGvUurSa+RA1IkSSHwsovUuflVzHnZJH0DaFRiIEoxfVxKw89AgHvDG9a85NBxZ6/Ado/B61QSqQn4jraHWR/woOfpKSzLlkg69MoGeDoFAgIKCAnZvL8Hnk2nWBOevXEjeyvibXSigOoCj5S1ACYHUoXA1u/53FEkFJYzRdANQnfq1jiJHQQ53ECPF8nSoMOQwCUbTOCXMa3N/S0lsVHNzOJ/rXv4B+XviJ/WVlmTKpTNC1q+bcE4Yo6YFvSmsOTpI0fTrdtSvOfR83ep3dPqOEf3vKIExGo9/7VEmxkY1q1SFS/5+I/7N8cLrKy3JlEvrmS38Y/bvjVHN5maN0eNqaGpqIj8/n+7I+nRCKkgq2Nq7z9O5II+HEur+cwQoQVPSsqolmXI55hwtibKMJhr4cfOEOz3cnRFJfMck6CstyZRLOliTTRYWWcAyPAuLLGAZnoVFFuiXfbxkEHJi5zkZJC2x89yXCCVx4KS/09/KUlI7DUJJoCnEB1eI9S0zNBR5whpe04QoU8bvNXXOps+Hk7/e2e0oYG+gOaB9ZgsTyw717Y17ka0Hy3Cv9vY4qJRphAyBqUEml+83dd7GbeX4t8TNoX0glFZUUeTSZ+ZbI072rh2Gd39mlqqdsIY3cFgDj5S/ZKotPV+7nAMbyntNU1doCsw9aSf3lC7vOXM/4T9dc1m1bgpkwfAmjqzi0fKXkj5HAy5ougZtywAjLVKg8YuTXkmc2thzI+zPzAT6CdS4sbDoP1iGZ2GRBU7YpmYmiLqhtVxFuGKjBJqE64AdV52567QNFoRLI0ZHXW62kfeF3Of9n2yiOaB5pIbIi81+C3BU2/EcMtdnCg6E4JAIyPooh9SukLdXyYijQ19iGV43hAsEPz77Dc7y7ASgVdj41w+vw1XnNnUdz6QGXjhtmXH8ZP1s3qipwPElMryoG678ykdcVbjaSPvXTd+DQ4WmrhMZ085fZ/8BV2zIdEXrWJbWX4g7mJlBj77CMrxukKMS6wIj0WJj022aA63NfJE1t7p4p3UcSmwsenugFCmnPGT7AAFbA2WscIwFQEWitcVF9x6Nx6K22VjROg6XpNecawMjkKKZNzolKPFq42S2uI4AUBfNQ24zOT/VDZbhdYOzAVa/OomPbZP0BAG+1u7POR7ujR7+sONC41gOgz2F6/Rn7C2w5+1R7LKP0hMEeFLwhfR9ZueJ/fOMZrsUBWdz9+ekgrta4s0XZxlzwZIGeS3dn2MGy/C6QVLB2Zj+dWxt+t+XGUmj2wW5yaIEM+O43hNyRH/x9tr1e+/SFhYWXWEZnoVFFrAMz8IiC5zQfTxVCDARBl4TKYyOCQkViaPunWoans597CLaq2iplIMgoSxT+j2Onmsir5qFIAwnrOEd2VXM18X3TJ1Tt9+P3+TT76ix8e+brsERi+zVGnTgrDf30MkRePOTiaytGW7u5jlMXXU++T2sFO+Ms17mpk1X4olF9gpFFRw15h5RSYMt24dzUbO5iHaNu4tMT22kwwlrePm7ZKK7BvScsQP+FO7jOSTBocKjAaxIJTSvHAX/BgdRzOnNZfwpnOOsB1YUGGWpAGbjh0sa+LfYiG4xV5Z9aXRg9fEsLLKCZXgWFlmgXzY1I3kQHKgZq4zlqIS7RkqYWA0VQagw3mFTgnqeoxGqhAztgwTRDsErHU1yggO05oC2EoHmiDnkCnDWyTgC/VdLZ6JuCA4SaLbYdTVw1crYO3hphPP1AK9HxzqUcEz/0UheEgQHQDg//h1trRLuw5KxqFizQXuJQHXFv6OzXk6YpO5vWtKhXxpe68go957zNwYq+lO3I1TGQ69fSN6+DoFNJzfx8Gl/MY6fPzKDj146zXhQNQeM+8rn3DBkhZHnrl1fp/3NEiNkQahQ8G8XfMAcr+4k3Sac3Prht3Gsi/fk+puWzrSXCG696DXGOKoBqFXzueMfl1OwLe6X2D42yIOzn8Ml6YMeH7SM5W8vnWmEMxcyFM2q5q7RrxrnPHRgLpWvlBth9KIeOHvuJi4rWmfkuXnjlfBefEur/qYlHfql4SFAFRIR9EKICOWYWBiiw+cA0eMMb2sk5hGdh6+FhCpkI89xpwr6mZbjERFK9+cIiYiwocSqjOMN82udvuPx88idvmM/15IG/TKSdNQLwWIRb1Kp4Krt1LzzQ9gf/2pKENy1klGDCFlvkkQ98Tz2ZomYMzoQi4UyUBhRoCUBznoJewen3P6mpTOqC4IDhR7Yh1iT6ohkbAICevMuXCiM5p0cAfdhKb6eUIJgkb6/g6G/VcJ5JB5ASCh6c1rtMOzrbJAS/Df7m5YvXSRpWyvktXY/uepsBGdj13kkDdyHISGMVCfkMD3uDNTftHRGCdJjAB9HAByBbvIIcB3RH8yukFR6XPTa37SkgzWqaWGRBSzDs7DIApbhWVhkAcvwLCyyQM4PrtglmD5iH6u1kV3m0UIKnt2OhMnknhAKBEarSEXx/Zykaie+L2RTw8bBYgidFERSkj9JC9jJ32XrfiupTkQ90DImguwx6XncC2htNvJ22U2tqtccEDg5iuxLPsKTiMq49jh1H85kkfStr0VJfB8uccRJ/h7ZVEj5cD60jQkjO7o+qWJ4Zco1lynDW7JkCS+88AKfffYZbreb2bNn88tf/pJTTjnFyBMMBvnxj3/Mc889RygUYt68efz+979PeRtmlyTxq2GvEhradZ4t4VJ+0vgdHIHki0F1wNkzPuW2wW8ZaTfsvpKayuGmfqDg0AjPnvkYA5XkA4gsa6jgxQNn4jBheOECwe1nvspZnt3Jn9RLvNN6Cr+p+Tq2Hnab7UjEA9+dvZLv+lf3nDlGterhO8Ef4axP/jHVFBg9ZT9LT/qzkXZ31YVsqpyIYuZ3HajxP2c+wwTH4S7zOCX9+UwFU4b3/vvvs2DBAqZPn040GuW2227j/PPPZ9u2bXi9XgBuvfVWXn/9dZ5//nkKCgpYuHAhl156KStXrkxJIOhfztXN9/PJ7aSydMupRCmQ4yfazfwyR5EFPjmccJ2e8Jip6o4igVcOmbpPb+GRU9th0mOynFpF2IifaQaHoibcx62kEEdRhnw52Gvlbcrw3nrrrYTjJ598kkGDBrF+/Xq+8pWv0NTUxOOPP84zzzzDOeecA8CyZcsYN24cH3/8MbNmzcqccguLfkxafbymJn2qv6ioCID169cTiUSYO3eukWfs2LEMHz6cVatWHdfwQqEQoVD8DRoImOiodUPUCxFv/FgO65OvZnYC0mwQyY/vpy0JsLX0wha9EoR9urfEUWztJHil5DoRn+5YfBQlCI5mMupmBfo9onnEPVeievQyM90DIet9uKN7qUsiFgmuD0Mupmx4mqZxyy23MGfOHCZOnAhAdXU1DocDv9+fkLekpITq6urjXmfJkiXcfffdqcrokuDkNn48+R3kWOTYNw5PYvebJ5l6mMN++MpFG5nq2wfoAW3/Z+V5+DdnZseYo6gOGHh2Ff82bI2R9rsdX0WsKMzaHnNmEArYK+q56eQPjLQ/Vc6k8e3BGdn3vCMtY8MsqngHZ8xHbG2gnA9fPy3Bva4nInkw/l92cv6AbYDuy/l/N52Hb6XZZbepk7LhLViwgK1bt/Lhhx+mJWDx4sUsWrTIOA4EAgwbNiytawJ4PCFmuT83DG9HXim7pJNMXUOzCSZ6q5jl/hyANs3Ob91nA5k1PCHD0LxG4z4Az3qn04C58ObZQkgwMK81Qf+KvLE0SoMzfi/FrTLTs9sI4d6iuvin7VS6c7c7BglOyqsz9GpCwu0OY369e+qkZHgLFy7ktdde44MPPmDo0PhwY2lpKeFwmMbGxoRar6amhtLS0uNey+l04nSmEjChe4Jb/Hzz8ELjWG5RyDPZRHQ2Svxmxb/w387z9QQBroOZNToAJQwfrxzHJQVjjDT7ERvefhL9SFZh39qh+v5xMWyNNrwpjCH1hH2Hm2+33BCPJB2UyWsyNwBia4O/vjeLv3inG2nOalufzq2ZupcQghtvvJEXX3yR9957j/LyxE0cp06dit1uZ/ny5Vx22WUA7Nixg8rKSioqKjKnOgk81RKe6vSMRAlCwWcKdFg+0htIKvi+kMh0TdpniJhDcYY2bewO3QE6vfvIEcjfLZNN/xFThrdgwQKeeeYZXn75ZXw+n9FvKygowO12U1BQwLXXXsuiRYsoKioiPz+fG2+8kYqKCmtE08KiA6YM7+GHHwbgrLPOSkhftmwZ8+fPB+A3v/kNsixz2WWXJUygW1hYxDHd1OwJl8vF0qVLWbp0acqiLCxOdCwnaQuLLJCzTtK/OzIDZyi5TnRle5Epv8FMIgdsLK09G78Jj+GPakeZnp9TghLPVc/gk3zzUy2yJBjnrmKILR5G62C0kO3tZSmFSd8aKEMJmTtPjsJb1eNpiia/m25j1IPS3LsDW12htEk8cfgMhroakz4n1BIBXu0xH+Sw4b3xQgWK09VzRgAB7kx7kyRJ3j6ZDw6fbuocSQW7Sb2ORtjz+knsls3NRYLuOCxPbaJiyBdG2kcHRiI2FCCnMEEvaeAwqd/eCnV/H8IryhBT53mz9Lt6qiU2vDSRDSbeL2ooSL83PHuLPr+V68hRkPvAtUvSUnBpkmJuczK0tTjZ1lCCzxHC72wnFLKT12zO1SotRP/anFNSMR1D04zvu9XHO4GJuiB4WhvemXXQ4KDhw1J2HkhteZZFZsnZGq87hJLoUIzQJ7s7OkBrDoxQeKC/wZQQppx2hYweAq7D60kOYUSAzqQW1YWx3zbok7wd36CpaIm6oTC/jSG+JhoiRTgboD2Y2GcScuycDk0qJZhYE2o20Do6F2m6/o7fUXWC6PA0SVES/TSlWJ6O3zFMPCxfBrX0SIa0pEO/NLyWERpnn7GF/Ngygf3thWx+7+SEkG2BU0NceupG43h17UiaVpSaaq6F/TDq3L2c4qsBIKTZeWPN5IRowpnQojlA+Uo95wzdaeR5deckPB95jQcqFS0qMvVhD4Fw1wMa7aWCKWftYEhsEKEx4uH9Dyfi2xu38MApUS6avtlwTN4eKOWL5SONkOdCgdDMFi4evdU45539p6C9X2i8PCJeKDrnENMH7DO0vbx5Mv71joxqSYZMaUmHfml4ar7Kjwa9S0msE7g5PICb3WPo+HoaMLCZmwd+YFQQj9lC/NV2fH/RLu/jFFxeup65Ht2ZtlmTeatwHB2daTOhRVNgRul+bh0Y9+7/tGkw1VJ8XVMqWuo1G083zCIQdiNk/S3e8Q0uZIh6Bd8r+SfjHfqTW6U6WeEbR8eq1eYPc9OgFXhiDudve0bzK+cI42JCgjEltQn6GyMe1spxJ2/NBucP3s58vx42XQM+GHASgmIjT9QjmF+ykomOIylrSYZMaUmHnI0kPeG6+1Ecxx/VDA6E6MltKLHNICMhG67t7oRIwM0jBcrwVqTYDxRqdOH7zB6Pn++GM7+9gXtKlxvnzN9zOQdeKDeaE1EvtI4P4fDoD7KmSUh7vHiq4j9yJrRoNv1N7hwQH8KLVHl1f0KRuhZF0RhR1ECePcT22hJaAy6Ki1s4qbCOA81+qg77kSSBzaEa2qJRBfsOT8KGKW1lAnFSK3JsNXio1UHedme89SBB0xgNx+B4cyJc6yF/h2KUpeqE5nERnAV6WGshJLR93oR9HcIFEBzfjt0RTVmLZoMRl37OE6P+Zpzzn4fmsurPU4yI2pnS0hk1HOTTR287cSNJu2qB2vib/ngNKd8XEnyR122enrC1QsFaJ91tN5kJLXIU/J/aAF+XeVLREvXCoTkqpww4zIRB1TAonneor5Ghvka215agrSzsdsTRUyVBVTdlKaBgpww7u9avhMC/yU53juCOJnCs6v6X6lFLEmRKSzpYo5oWFlnAMjwLiyzQL5uaFskhRaGxNo9PIl3/zO0Nbnz9ILzEiYZleCcwSgjytzoQsqPLPPla4vyVRd+Qs4bXNlggu5IbcJVUcNVLplx2jkeJq5kdZcLcZGwKKCEJZ4O5yVjNDqEiYUQ8yyZyRMJZnzh53xNCgVChPi3SmwgFBrubes7YA6pT34VXmPDR1oLJf7cc+BmPz+2XPI/Hl9y33hkczFNvnE1emvuZ3ViynHmXbknrGsnwUu3pbH5tnKmQ82E/XHTRamb69vSarmT5qHkMb78yo9uh9c5EvDB13ja+PmBjz5nTZKyjJu1rtJcIrrlgBaNdx4+OdzzamlWuTTJgXs4a3lfdh/C5kxv7KVZaWOY4C1ORpo7DUFuUobaqtK6RDLt9pWyWxpk6R7MLpuft5Rx37+vriaBm503bdMyUt5Bhku9gTuhPBs0pmO3dxURH8h7wzdHkm0rWqKaFRRbI2RovXQwXqRiS6MMlMCkgFBL2f5C0nh1/g0IQ6QO/I3sSm3McU95J6M8mqZR3JjlhDa9pQpRJ4yqN4z11xcgfF+TkmjDNDm0z2hg3JN6f2PL5EAo2OLt8GIJCcNvBC1m1d1Sv6ztj1B7uL3sTexfGJ2QITA0xsfygkbbtYCneNZ6cHDGNekBUNDGqOB5+esu24fi39p05nLCGN3BYA/876m9GW/rRwtN5ev25WdXUFZoNzhvzWaLfqLicAxvLuzwnImDV3lHk9UHY8ZVyOZGyrh2shAwTyw8m+ke657Jq/RTIQcNTHXD1mDVc698E6E7SFwSuQds6oM80nLCGV7e3iG/ZrzKOq44U4MzRFe1yFN7aPp5tDfEVC/u+GEhBTrmvd42kwZadw7g8FC/vykNF+LK/h+ZxUULw+KcVvFk0AdBDuDd8UUhBH2o4YQ2v4DOFps/KjHa8V5DxnWsyhRyBgjUuGqUyI82fw3o7I2ng32yn8ZO4/oIc1m9rB9uHeTRKurO1JOhTo4MT2PCO/uhSjv74xyD6kdbj0d/0Z1mvNZ1gYZEFLMOzsMgCluFZWGSBnO3jfRbx4o0c/72QL4Uos0XTemtIAg4H89jWcb/mPuJAqDBnBx56C0nA/mBRVsq7pr37MAzJoAFVURsB0XUEgNZI8jPwOWt4N7z5PWTX8WOuFI2u5+lJyyhKw/LkMGz5YAzXeEenfpEUUYIS7hycyO9NbK3w9+Wn86ZrSt/fu0XCneZU0hFV4tqt/07jnqIu82jBIHB7cprSk9N7+PbIKI7jW1ZdXj6RiTL6eyg1JC0WvyNNx2qL5JCjsc0r+2l5R5CpP1SAf1fXb3s1nHxNYPXxLCyygGV4FhZZIK2m5gMPPMDixYu5+eabefDBBwEIBoP8+Mc/5rnnnkvYEbakpG9j9gdGa+SPajSOmxq85G1xGrEVNRsEJkbwD4qvt2o6UEDBdsUY+Ij4IHRqG3le/SRVSAQ/88eaTDptpQL7hAB2RV/60B6yI3/iS4hsnAktuYyQoWlcFP+Q+Mrexhof+Z/ajVXqqgtaTg1S4I93bgN7/OTvib/7Q4WgndqM26k7eEZUhejWfNw18fJuGSZwj21Eic1+N7e6cG32GBuMZEpLb5Oy4a1du5Y//OEPnHrqqQnpt956K6+//jrPP/88BQUFLFy4kEsvvZSVK1emLdYMxScf4eVJy1BiHvWPNkzl2Z3nxB92O1wweQu/6OiYnH8FVTtGGsuHIj7BL6a+ynkefZVDsyaY17YQ9scdk8OlUZ4//XFKY4a3KeTn+gPX4myI/4iZ0JLLCBkmTajkfzs6SVedz+odp0HsYY+64PtTVvL9wg0AqEJwsfw9tD1xx+RQscajU59hUmxpflXUxjfrFuKuibtnSyPaeHnKY3hlvSzfah3BfXuuwN4iZVRLb5OS4bW0tHDVVVfx2GOPce+99xrpTU1NPP744zzzzDOcc845gL4/+rhx4/j444+ZNWtWZlQnQe2hAm4rvgA5VmVsqi1LiBEiq/B+5UksVuNFsONgCb4ONYzSJvG/lWewwqcvHwlpNrS6xOFkpdHGvQcuwmfTw0Ifas83HoJMakkHIUPLSA1tQHxoT651kLdPzsgaNEmDbQcGs9g5z0hbdWAk9g4vDSUCL1aexp62gQBoSNRXF+DvcB1bi8Rv9p9PiUuv+ZsiLmyNiY9otM7FHQcvxhkrwD2BASjBeHlnSktvk5LhLViwgIsuuoi5c+cmGN769euJRCLMnTvXSBs7dizDhw9n1apVxzW8UChEKBTfViYQMBGIpBvytzhYv32ScSxpYOswpCyHwfbPAtYq8Ro7L5q4GNLZBPVvDOGIHNtMUUB+J4/7vEqJzw6dEh+sExwzdJ0JLekgbDCr4jMeHPa6kbaw8mtsPzgWKQMrNiQNvGvcrN0Q129XE6OX2VohvHwAa+V4rdK5LN2HJSpfKaeyQ1l6Oy0ryt+l8Mne8UZ5Sxo4O+TJlJbexrThPffcc2zYsIG1a9ce81l1dTUOhwO/35+QXlJSQnX18YPGLFmyhLvvTjJCjAnkKEbToss8EbpfLyZ63mxQ0jptR9VbWlJAs+lBe6JeQVvUzhutI5jsOsCpDpdRQ2eKpL5jBspSUkHpofmdCS29jane5P79+7n55pt5+umncXUxuW2WxYsX09TUZPzt378/I9e10PdO+Nq81Tx28WNsrRrMkme/xZ37Lsm2LAtMGt769es5fPgwp59+OjabDZvNxvvvv89DDz2EzWajpKSEcDhMY2Njwnk1NTWUlh5/iyyn00l+fn7Cn0VmEDKc4qnmdEczakTBVQf1wd5fsW7RM6aamueeey5btiTGnbzmmmsYO3YsP/vZzxg2bBh2u53ly5dz2WWXAbBjxw4qKyupqKjInGoLi36OKcPz+XxMnDgxIc3r9VJcXGykX3vttSxatIiioiLy8/O58cYbqaioMD2iqTrockcqyZaZ4T7NhqlIwZlC6uOw6bIkodg0VBfGfKNdVvWtnUXfadHsiZHI+gpJNRf1ukvsml5mXWBm5ifjvpq/+c1vkGWZyy67LGEC3SynfnMbdu/xY/5PyDtEgaySjt+f5oDInADThlT2nDnDbK0dTOSfxX0W8cwjObhtypusG13OBYWbAZg/4J84rojycc1IWt8bZGqL6lSIesFxZh0TBiQfmTlTrNk/AudHvrReMH45yo+mfcCnYwd3mSfSGuaz3yV3vbQN77333ks4drlcLF26lKVLl6Z13fvL3sHn6+71mJ6zrabAWSN2J0T26ise9Z7O0x/3XcQzu6QwP/8w8/MPG2kznHZmDF7HUncNS+1f63VvedUO3xq50Yjs1Zf8pzKXVavTi3jmkSRdezf6m5s1/pLk9SxfTQuLLGAZnoVFFsjZ9Xjp0jpUEC3rMBvbbMe3R8n6xOnxEAo0j9IQhXFxSo2TvEopLSdpOQJP7J3N+gEHu8yzsXYoSrqDKxI0jxBog+LlLdU78O2Vc9LXVHNA82gV8uJf3HbQifdg360VPGENL29CPc+d9oRx/GR9BS9XnYEjBw1PdcBXZn3K4sFvGWkLdl9J7YFhaT249hZo//sgPpYGdZ1JgC1Nw9MUGD21kodOivdw7jl4EZ8cGN+jl0k2iHjg38/4kKv8a4y0KzZ+Hw4W9pmGE9bwmgIe/tp0OkrM4XF9/fCcfPuCPr2wvb6EF72TjbSDDQU4uqntFAmGDmygcnRmPIi6Y+SAxm77JJKAA41+XgxMNtJ2NgzM2U1LZBVWHxmJJ9b8UYVMc8BtRZLOBN6Nbv60PT5yKKtg7+Uh81RRwtD63iCesp+XkNbdg+uSJH4z+i8cGdX7nigDlVYc3ewWJKkgrSrgqfUd9EfSr0l7C1srHPj7CJ5SRhhpecE+1tC3t+s7lCDGerecR+gPg5kfQya2kSaZWc2RLrY2cnInpuMhaWBPfr/JXsEa1bSwyAKW4VlYZAHL8CwsskDO9vEiQhARyU1iqRnyvNXQ42/0NlqK7m4qctJlcvzzjyUVH3Etxfe1KtLTnyyKJKVfowhQkUzpNZM3Zw3vm5vno3i6cQXvQDBox3kkfeN7JjCBp3bPQhW9O5Ha3ODBZ3Lgxx6QuGvd1/ml1/yIkabJtFXlYW88toyiHoFjWCt2e/Lu+22tLlwt5srIFoT/3XAGz/lPN3WeWWyyxjUnfcy/5m9L6zrOIzI3rLsKpzP5oVm1LQT8Kqm8OWt4yrt+FEdyc1SZisb/j9pxiOVFvT7p60/hHHsL2D9yA27z50YFwz+sR/10xzGf2UYM4/A5Q4l6kjckn2kF+ghzwVonXa71yhCqDZYXjE3f8BqAD/LMnRRO/qWYs4Zn0TvYSkvQSoqQG1uIVh7ItpxeIdWmfF9iDa58yYiMKuXwLD/tp5QgKVlYBWwB9NMaL+KD9lINIeudWTki4T4kY2uP5wkOgFBxvM2otMl4DkmmViKrTmgbLNCcMRcSIeGukXE0ZVaLkPX7RPPiriqORhn34biTdDpa7C0QKfLgKB9Ba4EDTZEQsYCwIs9DYDREfLo+SZNwV8sJE8whPwQHacbexXJIxlslxR3OJWgfJAj74/rtzTLuasnwvtFsun7VE8/jrFNwHYnfJ+qBtsEaIhZhICUtSZApLenQLw2vdWSU35z7DAMV3Wvj09BQ/uvVS8jbF29i2CY38ofTnkGJ7Sj07JEK3n1hKg4Tjh5hv+B781bw1bzt+n01Jz9a+e841sT7npnQojng1DN3ceuQvxvn/Gzn5bS8UWr4l6ajxdYm0TDWhTSmDM1GwhrituH5/PvX3uXsPL1PVB3185N/XIn/0/ijERrbziMVf8Ir66sP3m0Zz59ePAd3bF2tkKF01iHuG/2Ccc6vD8xj98tjDO+hqBcumLuObxetBvQR2us3XgUr4h6S7SUat13wEmOdVSlrSYZMaUmHfml4kipxOJpvTCPURXzH7NgViSjURuMRyxoj5gclEBINUQ/VUT8ArZoDEU1snWdKS0vEadwHIBTt9NOko0UC1aFbmxISOAICW3sUoQkkTVAXyTOuWxv1IamJfSQtKlOr5tOs6SN8DRHP0QrHoC1iT9DfGnEmLmkS+vc+mkdFIhJR6BjcQ9Ik6qI+qpX0tPRIhrSkgyREH0ysmCAQCFBQUMCE6+7vclQz6oFQoTB6qFIUnPVSQjDUcD5ECuJfTQlKOOsxapCoG8789oaE0A/z91zOgRfKjTyaHYLFAnE0dL8AR6NkbJCRKS1IECwC1RPPY2uWcDbGr5ERLQIKd4Zwbz2AaGtHDQRQThnNF5cPIno0Xrymn9OxqRzJ02tcI3pzBFxHpIQYJqFCiOZ1+I5tEq56DOMTCgSLQXPG89ibpIQWiOqEULGIB6BKQYtmgxGXfs4THfdOODSXVX+eYtS+mdLSGTUc5NNHb6OpqanHMJX9ssbTHXK7f/s4AuAIpPeGkiPgqe7+GhnRItD7F0e6zpOWFgGyKpBUsAfCRKtrQFaQnE5QVbyHBNHGrq9tb+GY/SA642wAZ0P3Kxj05mDXeZTQ0c1CuyYZLT2RKS3p0C8Nz8IckiYo3BnCcaAR6htRAdvwIbROKCHiVfRQihZ9imV4XwIkAfbaNtTde2MJElq+h+ahNjRb7s95nYhYhvclQMjQMqYA58ApRlqb356V4LIWOpbhfQkQskRLmUJLmTVhnivkrOG1VbQge5Kb7Y6GbLg/c5mao8sUwQEQObkdOebgmYoWzQbNJ6vYiuNDZuohj741sIkx585aUkFTFew73AmTyT0i6dtNK6XxJejROjf5OxVTcW7CBdA+NojNEU1dS4borCUZtLYgPJpc3pw1vJdn/m8PkaTjbAkP4Oa6+TgCfd92Cg6J8OfZj1IaG7NPRYtmh3OnbeWOwW8baT/YdSU1e80FaOqsJRWqVSffbrkB1xF7z5ljaAqcclolj3SIMnZX1QWs3zvJlMN5cKDG72c/zXjHkZS1ZIrOWpKhuVnj1J6zATlseB5JwtNNgJ2OuKQIvbySp2tkgVeKGlpT1WKX1YTva0+l1uqkJRVckppSdHybrCXc15HKLiGSXn7xskxNS0bopCUZVBN5re61hUUWyNkaL10iPoh08KSQwxLOBnP7ims2CBeCdnRbMKFPhJuNppVLWlIh6oWwL+4tIkclHA3mtr4Siu7grHUIFmpvkUw7HeeSlnQ4YQ0vPLmF2ye/YRy/Vnca2147xVThhgvhvIvXMi1Pn/8KCgdLVl6If6O5Gedc0pIKLaeEWTz7DVySvgRgTfNJLH9tKq665K8R8cKUedv5l+L4xqb3brwI+4fmljHnkpZ0OGENz+WKcLKjBjlWraxzlfOpyf6CpgiGueoZ6zwEQJvmRHKYD4+cS1pSQXaojHVW4ZJ0h8hDkULzc4ASDHY1Gfo1IeNwmu8H5pKWdDhhDS+8uZB/rbreOFZaFbwmm2XORolH3jmP3zv1iNSSkHAdMj8XlktaUsG50838wA8RR9fABWW8TT2c1AlbG7z87gxe8Ewz0lyHbZgdr8wlLelwwhqe+zC4D6f39ZQg5O+WSXcMKpe0pIKzHpz16Rm5HAHf3vT155KWtDSYPeHgwYN85zvfobi4GLfbzaRJk1i3bp3xuRCCO++8k8GDB+N2u5k7dy67du3KqGgLi/6OKcNraGhgzpw52O123nzzTbZt28avf/1rCgvj2xv96le/4qGHHuKRRx5h9erVeL1e5s2bRzDYXzYysLDofUy1f375y18ybNgwli1bZqSVl5cb/wshePDBB7n99tu55JJLAPjjH/9ISUkJL730EldeeWWGZFtY9G9M1XivvPIK06ZN44orrmDQoEFMmTKFxx57zPh87969VFdXM3fuXCOtoKCAmTNnsmrVquNeMxQKEQgEEv4sLE50TNV4n3/+OQ8//DCLFi3itttuY+3atdx00004HA6uvvpqqqurASgpKUk4r6SkxPisM0uWLOHuu+/u8p4RIXi7bRQbWkZ0maeqvQBbq7nxeVmFDw+Wc490lpG2o6oEb04FwjCH0mTj/1afj8+WerO+MeJGaeq/Y26SgM+qSrjHe5aR9tHBcmSTMy+2VonfHzqbUlfXFcHpefu40Ls3pSEaUyWsaRrTpk3j/vvvB2DKlCls3bqVRx55hKuvvjqF28PixYtZtGiRcRwIBBg2bJhxHAF+tfV8lNXdxLAQ4DbpFyyHQf2gkHeV6UaaJ2rOmyTXyNsns656UvrXSd3HOutIKrjXeHl3w/SENDPh/wDc1RLbXzqF7d28z1dUnMw50/6Qkm+sKcMbPHgw48ePT0gbN24cf/ubHlimtLQUgJqaGgYPHmzkqampYfLkyce9ptPpxOnsOqy3KiAStuHsBdeoNJz4cxJJ7T+bQ/Ymcti8oXVG0ug2sBFAazj1aQ1TteScOXPYsSMx/v7OnTsZMUJvBpaXl1NaWsry5fHIXYFAgNWrV1NRUZGySAuLEw1TNd6tt97K7Nmzuf/++/nWt77FmjVrePTRR3n0UX31nyRJ3HLLLdx7772MGTOG8vJy7rjjDsrKyvjGN76RMdGaXQ/PdxRJ6G/6jmvXVKf+dxT5aG1wtA8n6dfQOpSAEk7cvlnIesi8ji5JtqD5t2mPWpIgGS3JlMsx11Vi1+3QWrK1JTodaw6Idoi0KGmx65pplkv6fbQOlYQSMt/qSEaL6iIhgJMc7dQSSEJLMuWSDqYMb/r06bz44ossXryYe+65h/Lych588EGuuuoqI89Pf/pTWltbue6662hsbOSMM87grbfewuVKbuefZGgZoXHRmevJjw0iVLYXsnrFhIRwbC2Tg/zbqWuN45W1o6j9xxAjDqXqANdX65g3dLuR5687J+P6p8/4EcN+GHveLsbn6wNDIc3GX1dPx7/V3OBDT1qSIRktyZRLZ9pKBXPO2coQVyMAjREPb30wBd/ncQsPnBLlWzPXYI9Z8NamMna+c5K+o06SRPKgZO4BKgbsNdKe2TQjtoNQ8vSkRSgQmdXMZWM2Gee8UTke9d1i4yWVjJZkyiUdTA9fXXzxxVx88cVdfi5JEvfccw/33HNPWsK6Q/OqfKf4IwbGwnhv8Q7iI9d4Oq6aLCxs4drCxCmMF5Uhxv9ChlMHVCXkWVs8ghopvgmV6hRcNHAL53h2A9AmFF7Mn4zZYutJSzIkoyWZcjnmui7B5QPWMt6uu/fXqG5e955Gx16ILT/M/MKPcMXeSO84T2a7c1S31+2MpsBXBu7mu/41+jHweuEEzG7b1ZMWIUH5gCMJ5X0w6Ge9XGxKSzLlkg79ctzYdcDOlf/8IbJNL3w1pOCpSyyQwKfFnNe80DhWmxzkd+gsK2F4d+0E3i8eHU885MLXocniaJK4d+XFLInFfhGahG2v+Zq7Jy3JkIyWZMqlM65amYX/vArFqdcgWlTGVZX4WMh73FysLUCKbYaittrJMxks2NYOT646g/+XPyN+3Uq36b0Ne9Iiq7Bj43DOOxQvb63OSX6HqNfJaEmmXNKhfxreEXAd6d4A8vZLsN/T5eeSCgWfKUDXeWxtxNa7pbfmrSctyZCMlmTKpTOOADjWdV/reKolqE5h74kOKCHwb7GR7iPXoxaBHiRqT9flnYyWZMolHazQDxYWWcAyPAuLLGAZnoVFFrAMz8IiC/TLwZVkCBdAuMCcx7O9Rd+3rq+RNNhaP5g/e8caaZWNftNvxagntqdbGrEoJQHOI+ail0kC9jUU8ufmuP5P6webchDIJKGixKhuyeBokhK2te5tTljD06Y08+vJf+s5Ywfu3XkR4bcHmorenAmUMBx5bzC/c3wtnhbEVBRmgJbyKLef/QoDbakvraqN5nPf8kso2J68H6KkQvTjQn63Ma5fDoMjTX/JVBAKFMyuYfHoN5M+R0PmZxsuxfFBXi8qS+SENTyfJ8gsV62pWmOQt4UDDOw1TV0iyMy+Dw6NWe69lCipL7GoUusRDg0w5wBsb6ZP41J2hZBgoKeV2a7apM/RgDxPEI2+Mzyrj2dhkQX6ZY0nZL1JYXgsiZjzaqb7FFLMibrDfSTV/Jo9oRDfSxtAO9bZVrOR8BqU1E7OzRnSkgrJlHdGvmOGtGSEXi7vfml4rUMF42btxWPTOxG17XkcXDk0tq915ggVQOGZ1QzzNQIQVhU2bziJ/F3mGgpNEyLMnLjHON5eW0J0ZaExgKE5IFTRzOQhB408a3aPJH+dy/ihM6UlFQKjNU4/fTe22DLufYFCAv8sMQYjhAzN09uZPmqfcc6G/UNxf5yHHHPVinrBNqeeUwbEf6TVW0bHPEgypyVTtA8SDJtzgGJXKwAtESc7Ph6peyFlgH5peNGiKPeOeInS2OjDppCf6z+5FvfhzD6EqkdwQ/n7nOepBKBVE5x/YCHsMuf+NXBoIw8Nf9U4ftQ3lWfXnGMcawqcU76LX5TG1zHOD19B1fqRGdeSCrZB7fzX8JfwyvpD91brCO5bdwWOpphjsgwThh9K+I7/aTuf1WtP00MIAKodvjNqPdf4N+nHQnBx/ffQtgzIqJZMEfEJ/nPk64x36B3XqqiNb+5cCPszE/a2Xxqeo8bG/E+uxmHTDa8l6MTZkPk3v61F4v9svpDf5ukezaomw37zPot1u4v5uhQPjVFb78PbYe2XHIU3t0xkQ+1QI62msgh/h2ZNprSkgtjv4VtbvocSq2Uamj3YO8S4kTTYsn04X2+Nf8eaQ37yOzQ1lRA89skZvFh4mpHWuKeIbgJ6pKQlUzgaZG785Eq8Tr1VFYrYcNR8yZ2kPYckRHUxR59dey/NFzkCwPs+Qh2WCpmcHgKgYKdMaNcg4zi/0zXkCPjXOwhJ8Tz+TnkypSUV8vZJRCsHcNSOPJ3uK2m603Foa9f6be3g+8hNSIq/LDqXQya0ZArXEeAffkIdbDqT9+qXhgf07eRsJu6VzDUylac3yCX9/e23Pw7WdIKFRRawDM/CIgtYhmdhkQX6bx8vR7FLKlp+lFBReqvWlRDYWjMkqg+IehMjqaWC5otil/p2g8iu0Gx6UCSjahJgb00/XudRLMPLMCNtLdx9xkvUzTI7UJ7IEzsrEO8W9IvI1kIBKhr5/piP07pOib2JYbY2zARR6i3aBwv+Ze46RrqOAFAXyePP782O7VGYPpbhZRiPJHGhd1/PGXtgRdEpHJAKMqCo9xESlBfV892CLT1n7pHsGx3oUcYuLVzPxKMT6KrCs96ZZKp3ZvXxLCyygFXj9UTnF3A2dxPKpharHDKKZXjdEM4HdUoz/g5uWo2fFpNX2ffNoWxqaRkh8I8/kuimtTEvK+vvcklLOliG1w3RPMHdp73GOZ4DQMwxObAQKnvfMTmXtEjDW/nbqU/gkjo4Ju+8Antz37+AcklLOliG1w1Km8T/fH4OL8SW4kQ1Ge1w5vaA6C9aojUebvnim8ZSnMpAIbb27DzofaXFHpD4P3svTlgWZKv/kjtJ9xXOJmh7q4QdUnyHW18fx2PJBS35u2X27B0TTxDgyFI59JUW92GJuteGUtchLS+DU4yW4XWHOHYVddbIohZJy52dcvtMi8BYxNsbWNMJFhZZwDI8C4ssYBmehUUWOGH7eKGIjWrVXGzI1kgKjs2qRLWah0pmPZqzpaU66gPN/Chhe9ROlcnyziUtEVUxGUk0PSQhRNLz/qqq8otf/II//elPVFdXU1ZWxvz587n99tuRYvMqQgjuuusuHnvsMRobG5kzZw4PP/wwY8aM6eHqOoFAgIKCAnZvL8Hnk2nWBOevXEjeSnPzVW2DBeESc6MR9iM2vAckU54QIT+0D4uCkln3iaxpUSU8lTZzUbskPfJbpDjDoz99qMVRbdP33jNB65kt/GP27/HEnv3mZo3R42poamoiP797J3lTNd4vf/lLHn74YZ566ikmTJjAunXruOaaaygoKOCmm24C4Fe/+hUPPfQQTz31FOXl5dxxxx3MmzePbdu2ZXQf9J7wHJLwHMpMRKjucDaCszE3Gg5Z0yLAu1/KWASuE0ZLN5j6lT766CMuueQSLrroIgBGjhzJs88+y5o1+l7SQggefPBBbr/9di655BIA/vjHP1JSUsJLL73ElVdemWH5Fhb9E1ODK7Nnz2b58uXs3LkTgM2bN/Phhx9ywQUXALB3716qq6uZO3eucU5BQQEzZ85k1apVx71mKBQiEAgk/FlYnOiYqvF+/vOfEwgEGDt2LIqioKoq9913H1dddRUA1dXVAJSUlCScV1JSYnzWmSVLlnD33XebEh0cAOHR7cixvowaVnDvcCZs/NEyXMCwduM4GnCQv8NmrCDWbNA8RkUpige41Kpd5H8uG/2qqBdaTg5j8+j9BSEklM9dCX2B/qalM+F8aD8lhBJz/9BUCccuN7H1n4DeX1bLg0iSft1om428HY74Vl4SBEZpyKVB4xz1iJP8XYoRol11QvPJUWz5+pcWgFTpTojM3N+0pIMpw/vLX/7C008/zTPPPMOECRPYtGkTt9xyC2VlZVx99dU9X+A4LF68mEWLFhnHgUCAYcOGdXtOcEiEP5/xKANl/UHdEh7EoiPfxRGIV+D544/w11OfMI4fb6jgxcozja2jNDucPe1Tbh/8lpHnR7uvpOaL4cYPFM4X3D7nNc7x7AagTShcwg1QHY8N2d+0HFOWAzV+e8bTjLfrzlE1qpt/bb0e15H4o6GNaufV2b/HFXMZeaftZH596BJsbfqDqilwypRKlo76i3HOPYcuYP0Xk4ytxqJumF/xId/1690SDbjc+33YX9RvtaSDqav8x3/8Bz//+c+NvtqkSZPYt28fS5Ys4eqrr6a0tFQXWVPD4MGDjfNqamqYPHnyca/pdDpxOs0F65BbFf50ZDb5Nv2tVtleiBJMHJFqaMjj8YYK43hl7aiEDTIkDT6pK+NxVzzPviOFuDoMCCohiddrJ7E3X9+6K6TZUAOJnfb+pqUzSlDir3XTGeJqBKAx4kFpTTTUaMDBkw2zscdEb20qQ+kQ6VUSsLeumMcL4/o3Hy5LcO2SVfigdjSRDjubNDV46bjGvr9pSQdT0wnFxcXce++9XH/99UbakiVLWLZsGTt37kQIQVlZGT/5yU/48Y9/DOg12KBBg3jyySeTGlxJZjpBs4Hqxtj5VNL0SMUdH2bVmRh8R47qeToOz0fdem1zFCWsbwh5FCHru6yKDuVta0/04etvWjojFP3eR68rCVDaE/1CNQdEOwxIy6qep+PDrLpA7TD1KEdi3/Eokq5f6zBZpoT0v/6qpc+mE772ta9x3333MXz4cCZMmMDGjRv57//+b773ve/p4iSJW265hXvvvZcxY8YY0wllZWV84xvfMHOrbpGjIPew8LFzQR4PWzvQ3vXnkgb2lhNLyzHXVZO4bhK7uyrBxBfFMYieo6b1Ny3pYMrwfvvb33LHHXdwww03cPjwYcrKyvjhD3/InXfeaeT56U9/SmtrK9dddx2NjY2cccYZvPXWW306h2dhkeuYamr2BZnyXLGw6G3SaWpaTtIWFlkgN3ydukGRQJK1hEEFC4tcQEoj4kTOG54duGrcOj4ceFK2pVhYJHDGgD2k6hGa+4YnSSwoWsuCorXZlmJhcRxSq/Zy1vDO+scNyO4UR0IlwVkTd3D74DdZ1lDB0xtnQNR8W9Vd2M4Dp73AIKWZm7ZfSW2VPzU9Fr2GzRth3pjteG0hXvt8Am312RuE09qDwF1J5c1Zwxvzk0+xSalV5JIk8eEvplD7rfd4dss0xv3HF2jN5idltGnj+PuDkxjnrcL+ZBGnvLIpJT0WvYd80gjeuH4ywqsyepmKsuaTrGmJiggHksybs4YnQiGE2XBSsoKYOZHWYW7EcH02unRgE/XzxuA9FMHx8Xa0trYeLgK2IWU0TxtK40k2vuJswq+0cWSSjKRNJv+TWtRdn6fylSx6AamphfydA1BdCva6WtRQD54KvYgQyYcly1nDSwXZ6+Gzq50s/uorlNoakSWNO0a/zsHbC3nws3MYvqcYbV/Phtdy+lAm37GRaXl7GemoQ0HjF99+jv2XFvHs785noGV4OUP0UA1lT7WALKG19J8NBXPO8I7O50eJmN6MQiECUjNjInshAq2AkwCjCOCX6omiNwd6IiSHKNcqGRY+iBoGFSihGqfaRJhQUtew6CPUCNGm7vzD+o4o+nORjE9KznmuHDhwoMdlQRYWucz+/fsZOnRot3lyzvA0TWPHjh2MHz+e/fv39+h6kwscXUNo6e0d+oteIQTNzc2UlZUhy92PoudcU1OWZYYMGQJAfn5+Thd0Zyy9vUt/0FtQkNwuvpYjloVFFrAMz8IiC+Sk4TmdTu666y7TISGyhaW3d+lvepMh5wZXLCy+DORkjWdhcaJjGZ6FRRawDM/CIgtYhmdhkQUsw7OwyAI5aXhLly5l5MiRuFwuZs6caexGlE2WLFnC9OnT8fl8DBo0iG984xvs2LEjIU8wGGTBggUUFxeTl5fHZZddRk1NTZYUJ/LAAw8YcU+Pkmt6Dx48yHe+8x2Ki4txu91MmjSJdevWGZ8LIbjzzjsZPHgwbrebuXPnsmvXrqzpTQuRYzz33HPC4XCIJ554Qnz66afiBz/4gfD7/aKmpiaruubNmyeWLVsmtm7dKjZt2iQuvPBCMXz4cNHS0mLk+dGPfiSGDRsmli9fLtatWydmzZolZs+enUXVOmvWrBEjR44Up556qrj55puN9FzSW19fL0aMGCHmz58vVq9eLT7//HPx9ttvi927dxt5HnjgAVFQUCBeeuklsXnzZvH1r39dlJeXi/b29qxoToecM7wZM2aIBQsWGMeqqoqysjKxZMmSLKo6lsOHDwtAvP/++0IIIRobG4XdbhfPP/+8kWf79u0CEKtWrcqWTNHc3CzGjBkj3nnnHfHVr37VMLxc0/uzn/1MnHHGGV1+rmmaKC0tFf/1X/9lpDU2Ngqn0ymeffbZvpCYUXKqqRkOh1m/fn3C/nqyLDN37twu99fLFk1N+v7ARUX6DjPr168nEokkaB87dizDhw/PqvYFCxZw0UUXJeiC3NP7yiuvMG3aNK644goGDRrElClTeOyxx4zPU9l7MZfJKcOrq6tDVVVT++tlA03TuOWWW5gzZw4TJ04E9L0BHQ4Hfr8/IW82tT/33HNs2LCBJUuWHPNZrun9/PPPefjhhxkzZgxvv/02119/PTfddBNPPfWUofeovlzQmy45tyyoP7BgwQK2bt3Khx9+mG0pXbJ//35uvvlm3nnnnX6xb4WmaUybNo37778fgClTprB161YeeeSRlPdezGVyqsYbMGAAiqIcM7JWU1Nj7L2XbRYuXMhrr73Gu+++m7DKuLS0lHA4TGNjY0L+bGlfv349hw8f5vTTT8dms2Gz2Xj//fd56KGHsNlslJSU5JTewYMHM378+IS0cePGUVlZCZCw92JHcunZMENOGZ7D4WDq1KksX77cSNM0jeXLl1NRUdHNmb2PEIKFCxfy4osvsmLFCsrLyxM+nzp1Kna7PUH7jh07qKyszIr2c889ly1btrBp0ybjb9q0aVx11VXG/7mkd86cOcdMz+zcuZMRI0YAUF5eTmlpaYLeQCDA6tWrs/5spES2R3c689xzzwmn0ymefPJJsW3bNnHdddcJv98vqqurs6rr+uuvFwUFBeK9994Thw4dMv7a2tqMPD/60Y/E8OHDxYoVK8S6detERUWFqKioyKLqRDqOagqRW3rXrFkjbDabuO+++8SuXbvE008/LTwej/jTn/5k5HnggQeE3+8XL7/8svjkk0/EJZdcYk0nZJLf/va3Yvjw4cLhcIgZM2aIjz/+ONuSBHrMs2P+li1bZuRpb28XN9xwgygsLBQej0d885vfFIcOHcqe6E50Nrxc0/vqq6+KiRMnCqfTKcaOHSseffTRhM81TRN33HGHKCkpEU6nU5x77rlix44dWVKbHtZ6PAuLLJBTfTwLiy8LluFZWGQBy/AsLLKAZXgWFlnAMjwLiyxgGZ6FRRawDM/CIgtYhmdhkQUsw7OwyAKW4VlYZAHL8CwsssD/B9kQG2+USlpBAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Build the deep Q network\n",
        "\n",
        " **Input della Rete** : Stato del gioco (di fatto l'immagine della schermata di gioco processata) --->  **Output della Rete**: i valori di $Q$ di tutte le azioni nello stato dato in input."
      ],
      "metadata": {
        "id": "6hVV2SAC3n6i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class DQN:\n",
        "    def __init__(self, state_size, action_size):\n",
        "\n",
        "        self.state_size = state_size\n",
        "        self.action_size = action_size\n",
        "\n",
        "        # Definiammo il replay buffer\n",
        "        # Deque (Doubly Ended Queue) tale struttura dati\n",
        "        # è preferibile rispetto a un elenco nei casi in cui\n",
        "        # abbiamo bisogno di operazioni di accodamento e pop più veloci da entrambe le estremità del conteiner\n",
        "        self.replay_buffer = deque(maxlen=batch_size*8) # il replay_buffer ha una lunghezza limitata, che\n",
        "        # un volta  riempita, l'aggiunta di un altro elemento in coda comporta  uno shift di tutti gli\n",
        "        # elementi in avanti, e la cancellazione dell'elemento in testa.\n",
        "\n",
        "        #Discount factor\n",
        "        self.gamma = 0.99\n",
        "\n",
        "        #main network\n",
        "        self.main_network = self.build_network()\n",
        "\n",
        "        #target network\n",
        "        self.target_network = self.build_network()\n",
        "\n",
        "        # Copia weights dalla main q network alla target network\n",
        "        # Nota: model.set_weights() consente di trasferire i parametri da un modello\n",
        "        # a un altro modello.\n",
        "        self.target_network.set_weights(self.main_network.get_weights())\n",
        "\n",
        "\n",
        "    #Definiamo una funzione chiamata build_network che è la nostra CNN.\n",
        "    #\n",
        "    # Note:\n",
        "    # 1. Per quanto riguarda inizilizzazione dei pesi:\n",
        "    # kernel_initializer: If None, the default initializer (\"glorot_uniform\") will be used\n",
        "    # https://keras.io/api/layers/convolution_layers/convolution2d/\n",
        "    # https://keras.io/api/layers/initializers/#glorotuniform-class\n",
        "    # https://github.com/keras-team/keras/blob/62d097c4ff6fa694a4dbc670e9c7eb9e2bc27c74/keras/layers/core.py#L798\n",
        "    #\n",
        "    # 2. Per quanto riguarda applicazione di più filtri all'immagine di input, ovvero :\n",
        "    # model.add(Conv2D(filters,..) , con filters = n > 1\n",
        "    # Si tratta di applicare \"n\" filtri a un'immagine, ottenendo in unscita \"n\" matrici (feature maps).\n",
        "    # Alla fine, su ogniuna di questa sarà applicata una la funzione Relu : model.add(Activation('relu'))\n",
        "    # producendo in concreto \"n\" matrici (feature maps), i cui valori sono tutti maggiori od uguali a 0 (effetto Relu).\n",
        "    # Perchè usare più filtri?\n",
        "    # Aumentando il numero di filtri, ognuno dei quali apprenderà pesi diversi in modo indipendente,\n",
        "    # si aumenta la capacità del modello di apprendere funzioni complesse dai dati.\n",
        "\n",
        "    def build_network(self):\n",
        "        model = Sequential()\n",
        "        model.add(Conv2D(16, (8, 8), strides=4, padding='same', input_shape=self.state_size)) # suggeriti: filters = 32\n",
        "        model.add(Activation('relu'))\n",
        "\n",
        "        model.add(Conv2D(32, (4, 4), strides=2, padding='same')) # 64\n",
        "        model.add(Activation('relu'))\n",
        "\n",
        "        model.add(Conv2D(32, (3, 3), strides=1, padding='same')) # 64\n",
        "        model.add(Activation('relu'))\n",
        "        model.add(Flatten())\n",
        "\n",
        "\n",
        "        model.add(Dense(256, activation='relu'))  #512\n",
        "        model.add(Dense(self.action_size, activation='linear'))\n",
        "\n",
        "        model.compile(loss='mse', optimizer=Adam())\n",
        "\n",
        "        return model\n",
        "\n",
        "    #La funzione store_transition memorizza le informazioni della transizione\n",
        "    #nel replay buffer.\n",
        "    def store_transistion(self, state, action, reward, next_state, done):\n",
        "        self.replay_buffer.append((state, action, reward, next_state, done))\n",
        "\n",
        "    # Exploration-exploitation trade off:  Epsilon-greedy policy\n",
        "    def epsilon_greedy(self, state, epsilon):\n",
        "\n",
        "       Q_values = self.main_network.predict(state, verbose=0)\n",
        "       # Note : The verbosity of model.fit() and model.predict() are controlled separately by their own verbose argument\n",
        "       greedy_act = np.argmax(Q_values[0])\n",
        "\n",
        "       if random.uniform(0,1) > epsilon: # policy greedy\n",
        "            return greedy_act\n",
        "       else:\n",
        "            non_greedy_actions = list(set(range(self.action_size)) - {greedy_act}) # [0, 2, 3, 4, 5, 6, 7, 8]\n",
        "            action = np.random.choice(non_greedy_actions) # policy random non-greedy\n",
        "            return action\n",
        "\n",
        "    #Fase di Training\n",
        "    def train(self, batch_size):\n",
        "\n",
        "        #Campiona random un singolo mini batch dal replay buffer\n",
        "        minibatch = random.sample(self.replay_buffer, batch_size)\n",
        "        # minibatch è una lista; len(minibatch) = batch_size, e\n",
        "        # il numero di \"batch_size\" tuple: minibatch[0], minibatch[1], ...\n",
        "        # ogniuna con questa struttura : (state, action, reward, next_state, done).\n",
        "        # Ad esempio se volessi accedere al reward della tupla nella posizione 4:\n",
        "        # minibatch[3][2] = 10.0\n",
        "        #\n",
        "        #Il loop sottostatne scorre su ogni tupla presente nella lista minibatch:\n",
        "        # partendo dalla prima tupla: minibatch[0], ne prende il valore dei sui elementi:\n",
        "        # state, action, reward, next_state, done\n",
        "        # e calcola la target_Q, ovvero il massimo valore che da la Q.target_network nel 'next_state'\n",
        "        # addizionato al reward ottenuto entrando in tale stato,\n",
        "        # poi la Q_values, e conclude con il fit-training.\n",
        "        # Poi passa alla tupla sucessiva:  minibatch[1], e ripete la sequenza\n",
        "        for state, action, reward, next_state, done in minibatch:\n",
        "            if not done:\n",
        "                target_Q = (reward + self.gamma * np.amax(self.target_network.predict(next_state, verbose=0)))\n",
        "            else:\n",
        "                target_Q = reward\n",
        "\n",
        "            Q_values = self.main_network.predict(state, verbose=0)\n",
        "            # Q_values= [[2.18,  1.10,  120.29,  11.99, 12.21, 119.57, 15.33, 1.8, 8.9 ]]\n",
        "\n",
        "            #Otteniamo la Q_values da usare come target nel training.\n",
        "            #Per avere un target appropriato, imponiamo alla Q.main_network che, nello stato corrente ('state'),\n",
        "            # l'azione estratta ('action') debba produrre il valore target_Q\n",
        "            Q_values[0][action] = target_Q\n",
        "\n",
        "            # Qui avviene il proceso di training della main network.\n",
        "            # Facendo riferimento ai cenni di teoria scritti sopra,\n",
        "            # tale istruzione consente l'update dei pesi minimizzando la loss L(w)\n",
        "            self.main_network.fit(state, Q_values, epochs=1, verbose=0) # .fit(input data, target, epochs, ...)\n",
        "\n",
        "\n",
        "    #Aggiorna i parametri(pesi) della target network copiandoli dalla main q network\n",
        "    def update_target_network(self):\n",
        "        self.target_network.set_weights(self.main_network.get_weights())\n",
        "\n"
      ],
      "metadata": {
        "id": "p8RuB2tX3qmK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Nota:\n",
        "\n",
        "Generalmente le CNN, fanno uso della tecnica detta Pooling, per maggiori info:\n",
        "\n",
        "https://www.researchgate.net/figure/Pooling-layer-operation-oproaches-1-Pooling-layers-For-the-function-of-decreasing-the_fig4_340812216\n",
        "\n",
        "https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks/\n",
        "\n",
        "\n",
        "Un'operazione di Pooling, riduce le dimensioni dei dati ed i costi computazionali.\n",
        "\n",
        "Questa tecnica è spesso usata in attività come il rilevamento di oggetti in immagini: in cui vogliamo sapere se l'oggetto desiderato è presente nell'immagine e non ci interessa esattamente la sua posizione nell'immagine. Il motivo di tale utilizzo risiede sia nella sua efficenza computazionale, ma anche perchè il Pooling rende la risposta della CNN meno sensibile alla posizione delle l'oggetto nell'immagine (“local translation invariance”).\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Nel nostro scenario, l'aggiunta di operazioni di Pooling dopo ogni strato convoluzionale, ha determinato un calo delle prestazioni del modello, rispetto a un modello senza Pooling. Pertanto non è stata usata."
      ],
      "metadata": {
        "id": "v7F6nWk6mqdw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Parameters\n"
      ],
      "metadata": {
        "id": "jghaGK_7PDBG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#A causa di limitazioni computazionali, i parametri utilizzati in questa\n",
        "# sperimentazione sono da considerarsi indicativi.\n",
        "\n",
        "num_episodes =  100 # 500\n",
        "\n",
        "batch_size = 6  # 8\n",
        "\n",
        "eps = 0.2\n",
        "eps_decay = 0.03\n",
        "\n",
        "update_rate_target = 300  # 800\n"
      ],
      "metadata": {
        "id": "h92HntQaPFVw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Train Pacman"
      ],
      "metadata": {
        "id": "dXVz-k2xrbEq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env = gym.make(\"ALE/MsPacman-v5\")\n",
        "\n",
        "action_size = env.action_space.n\n",
        "state_size = (88, 80, 1)\n",
        "\n",
        "dqn = DQN(state_size, action_size)\n",
        "\n",
        "#info sul modello\n",
        "dqn.main_network.summary()"
      ],
      "metadata": {
        "id": "FxM1Ozyoto78"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SIsO9laTxi3A"
      },
      "outputs": [],
      "source": [
        "#info sul modello\n",
        "#dqn.main_network.summary()\n",
        "time_step = 0\n",
        "train_is_start = False\n",
        "episode_rewards = [] # struttura dati che ci servirà per il grafico che valuterà le prestazioni del training\n",
        "\n",
        "#Per ogni episodio...\n",
        "for i in range(1,num_episodes+1):\n",
        "\n",
        "    #Epsilon decay, ovvero -> converge ad una policy greedy\n",
        "    if i % (num_episodes/4)==0:\n",
        "      eps -= eps_decay\n",
        "      #Salva la rete nel formato file HDF5\n",
        "      save_path = \".\"\n",
        "      dqn.main_network.save(os.path.join(save_path,\"myDQN.h5\"))\n",
        "      print(\"Ep\",i,\" Model Saved\")\n",
        "\n",
        "    Return = 0 # reward totale accumulato\n",
        "    done = False\n",
        "\n",
        "\n",
        "    #preprocessa lo stato iniziale\n",
        "    state = preprocess_state(env.reset())\n",
        "\n",
        "    while done == False:\n",
        "\n",
        "        #update il time step\n",
        "        time_step += 1\n",
        "\n",
        "        action = dqn.epsilon_greedy(state,eps)\n",
        "\n",
        "        next_state, reward, done, info = env.step(action)\n",
        "\n",
        "        next_state = preprocess_state(next_state)\n",
        "\n",
        "        #memorizzare le informazioni sulla transizione\n",
        "        dqn.store_transistion(state, action, reward, next_state, done)\n",
        "\n",
        "        state = next_state\n",
        "        Return += reward\n",
        "\n",
        "        #Se l'episodio è finito, fai il debug e salva i dati per il grafico\n",
        "        if done:\n",
        "            print('Episode: ',i)\n",
        "            print(f\" - Terminal state occurs at\\n  Step {time_step} - Tot Reward={Return}\")\n",
        "            episode_rewards.append(Return)\n",
        "\n",
        "        #Verifica se il numero di transizioni nel replay buffer è maggiore\n",
        "        # della dimensione del batch, se così allora addestra la rete\n",
        "\n",
        "        if len(dqn.replay_buffer) > (batch_size + (batch_size/2)) :\n",
        "            dqn.train(batch_size)\n",
        "            train_is_start = True\n",
        "\n",
        "        #Update della target network\n",
        "        #Per motivo vedi sopra: Problema della correlazione nell'update dei pesi\n",
        "        if (train_is_start == True) and (time_step % update_rate_target == 0):\n",
        "        # I parametri vengono copiati ogni \"update_rate_target\" step dalla main_network.\n",
        "            dqn.update_target_network()\n",
        "\n",
        "\n",
        "env.close()\n",
        "print(\"\\n...End of training...\\n\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Plot"
      ],
      "metadata": {
        "id": "Tb_VVY6d3YBt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(3, 3))\n",
        "print(episode_rewards)\n",
        "plt.plot(episode_rewards)\n",
        "plt.ylabel(\"Rewards\")\n",
        "plt.xlabel(\"Episodes\")\n",
        "plt.title(\"DQN\")"
      ],
      "metadata": {
        "id": "ptp6tNuo3Zpy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Test - singolo campione con rendering"
      ],
      "metadata": {
        "id": "l4IYb1R_QEKd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env = wrap_env(gym.make(\"ALE/MsPacman-v5\",render_mode='human'))\n",
        "\n",
        "state = env.reset()\n",
        "\n",
        "done = False\n",
        "\n",
        "i = 0\n",
        "t_r = 0\n",
        "\n",
        "while not done:\n",
        "\n",
        "    state = preprocess_state(state)\n",
        "    action = dqn.epsilon_greedy(state, 0.0) # con epsilon=0.0 agisce greedy\n",
        "\n",
        "    state, reward, done, _ = env.step(action)\n",
        "    t_r += reward\n",
        "    i += 1\n",
        "\n",
        "    if done == True:\n",
        "      print(f\" - Terminal state occurs in\\n   Step {i} -- Tot Reward {t_r}\")\n",
        "\n",
        "env.close()"
      ],
      "metadata": {
        "id": "jkaYO0noQH7H"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Test - su 100 campioni"
      ],
      "metadata": {
        "id": "esLxfj0mQIUN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def test(env):\n",
        "\n",
        "   state = env.reset()\n",
        "   done = False\n",
        "\n",
        "   t_r = 0\n",
        "\n",
        "   while not done:\n",
        "\n",
        "      state = preprocess_state(state)\n",
        "      action = dqn.epsilon_greedy(state, 0.0) # con epsilon=0.0 agisce greedy\n",
        "\n",
        "      state, reward, done, _ = env.step(action)\n",
        "\n",
        "      t_r += reward\n",
        "\n",
        "   return t_r\n",
        "\n",
        "\n",
        "#######################\n",
        "\n",
        "env = gym.make(\"ALE/MsPacman-v5\")\n",
        "r_list = []\n",
        "\n",
        "for j in range(1,101):\n",
        "  s = test(env)\n",
        "  r_list.append(s)\n",
        "\n",
        "env.close()\n",
        "\n",
        "##########################\n",
        "\n",
        "\n",
        "max_value = max(r_list)\n",
        "min_value = min(r_list)\n",
        "mean_value = sum(r_list) / len(r_list)\n",
        "\n",
        "print(\"Maximum value:\", max_value)\n",
        "print(\"Minimum value:\", min_value)\n",
        "print(\"Mean value:\", mean_value)"
      ],
      "metadata": {
        "id": "6P5DhflLQNcK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Riferimenti sperimentali"
      ],
      "metadata": {
        "id": "a70Wa8ehMJzN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Facendo riferimento al paper di Van Hasselt et al.\n",
        "\n",
        "https://arxiv.org/pdf/1509.06461\n",
        "\n",
        "dove viene riportata una tabella con i risultati delle loro sperimentazione nel dominio di Atari.\n",
        "\n",
        "Per Pacman abbiamo: *Raw scores for the no-op evaluation condition (5 minutes emulator time): each evaluation\n",
        "episode starts by executing a special no-op action that does not affect the environment up to 30 times, to provide different starting points for the agent* , con diverse modalità:\n",
        "\n",
        "\\begin{array}{ccc}\n",
        "\\text{Random}&\\text{Human}&\\text{DQN}&\\text{Double DQN}\\\\\n",
        "307.30&15693.40&2311.00&3210.00\n",
        "\\end{array}\n",
        "\n"
      ],
      "metadata": {
        "id": "oA74UmU2rFhi"
      }
    }
  ]
}