Value_Function_Approximation_parte1_RL_ITA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "FzK7LtcPH9Gm",
        "_5Ub0OAgh0z0",
        "KZhKoGr3fYqq"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#Value Function Approximation\n",
        "\n",
        "Mario Fiorino"
      ],
      "metadata": {
        "id": "662MfPR9lR3q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Introduzione\n",
        "\n",
        "Problema:\n",
        "\n",
        "Come possiamo estendere i metodi RL tabulari (look-up table), che si basano su una rappresentazione discreta e completa dello spazio di stato, a problemi con spazi di stato arbitrariamente grandi; ad esempio come gestire giochi come Go, che hanno $10^{170}$ stati?\n",
        "\n",
        "I requisiti di memoria per archiviare una tabelle di tali dimensioni sembrano irrealizzabili; inoltre, visitare adeguatamente ciascuna coppia stato-azione richiede un tempo enorme.\n",
        "\n",
        "\n",
        "Una risposta:\n",
        "\n",
        "Una chiave per superare questo limite è combinare l'RL con tecniche di approssimazione di funzioni (in pratica : si utilizza una funzione che approssima il valore di ogni stato. Questa funzione può essere appresa da dati di esempio, utilizzando tecniche di apprendimento supervisionato).\n",
        "\n",
        "\n",
        "Obiettivo del notebook:\n",
        "\n",
        "Sostituire le rappresentazioni tabulari della funzione valore (value-fuction) utilizzando approssimatori di funzioni, come le reti neurali.\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Note:\n",
        "\n",
        "1.\n",
        "\n",
        "Esistono diversi tipi di approssimatori di funzioni che possono essere utilizzati nel RL, tra cui: la combinazioni lineari di features,le Neural network, i Decision tree, il Nearest neighbour, le Basi di Fourier/wavelet (Per maggiori info vedi Sutton & Barto, 2018, PDF: 222-257 ).\n",
        "\n",
        "\n",
        "I metodi più comuni e promettenti sono quelli che utilizzano l’approssimazione di funzioni parametrizzate, in cui la politica è parametrizzata da un vettore dei pesi $\\vec{w}$.\n",
        "Tra l'altro, questi metodi permettono di rappresentare la funzione valore in modo scalabile. Tuttavia, è importante notare che introducono un certo grado di incertezza (dobbiamo accontentarci di una soluzione approssimativa)\n",
        "\n",
        "\n",
        "\n",
        "2.\n",
        "\n",
        "L'utilizzo di tecniche di approssimazione di funzioni all'interno del RL permette di estenderne l'applicazione a problemi con informazioni incomplete sullo stato del sistema (problemi parzialmente osservabili).\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Ref\n",
        "\n",
        "Reinforcement Learning: An Introduction -\n",
        "Richard S. Sutton and Andrew G. Barto, Second Edition - MIT Press, Cambridge, 2018.\n",
        "\n",
        "Corso di David Silver:\n",
        "https://www.davidsilver.uk/wp-content/uploads/2020/03/FA.pdf"
      ],
      "metadata": {
        "id": "5JIPUjq189Fn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##On-policy Prediction - Stochastic-gradient methods\n",
        "\n",
        "Per individuare i valori ottimali del \"weights vector\" $\\vec{w}$ , ci si affida principalmente a diverse varianti dell' algoritmo: Stochastic gradient descent (SGD).\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Componenti essenziali:\n",
        "\n",
        "\"Weights vector\", indicatco con $\\vec{w}$ = $(w_1, w_2, ... w_d)^{\\intercal} $ ; in pratica è un vettore colonna che contiene il valore dei pesi. Nota, affinché uso di queste metodologie abbiano un senso, le dimensioni $d$ sono considerevolmente inferiori rispetto all'intera dimensione dello spazio degli stati.\n",
        "\n",
        "$v̂(s,\\vec{w})$ è \"approximate value function\", in funzione degli stati e dei pesi, ed è differenziabile $\\vec{w}$ rispetto a in tutti gli stati\n",
        "\n",
        "$v_\\pi(s)$ è il \"true value function\", il nostro target. Di fatto sconosciuto\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Idea di base dei metodi SGD:\n",
        "\n",
        "Aggiornare i pesi, dopo ogni esempio, per ridurrebbe l’errore:\n",
        "\n",
        "$w_{t+1} \\doteq \\vec{w_t} + α [v_\\pi(s_t) - v̂(s_t,\\vec{w_t}) ] \\nabla_\\vec{w} v̂(s_t,\\vec{w_t}) $\n",
        "\n",
        "dove:\n",
        "\n",
        "Learning rate:  $ α > 0 $\n",
        "\n",
        "$ \\nabla_\\vec{w} v̂(s_t,\\vec{w_t}) $ è il vettore gradiente rispetto $\\vec{w}$. In pratica un vettore colonna delle derivate parziali rispetto alle componenti del vettore $\\vec{w}$ : $( \\frac{\\partial v̂(s_t,\\vec{w_t}) }{\\partial w_1},\\frac{\\partial v̂(s_t,\\vec{w_t}) }{\\partial w_2} ,...,\\frac{\\partial v̂(s_t,\\vec{w_t}) }{\\partial w_d})^\\intercal $\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Convergenza:\n",
        "\n",
        "Nel caso in cui $\\alpha$ diminusce nel tempo in modo da rispettare le condizioni di \"standard stochastic approximation\" (viste nei notebbok precedenti e descritte nel testo Sutton & Barto, 2018,  PDF 55 - (2.7) ), allora è garantito che il metodo SGD converga ad un ottimo locale.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Caso concreto:\n",
        "\n",
        "In pratica non possiamo eseguire l'aggiornamento esatto dei pesi visto sopra perché $v_\\pi(s)$ è in genere una idealizzazione (se la conoscenza di tale funzione fosse direttamente accessibile, le tecniche di apprendimento non sarebbero necessarie). Però possiamo usare una qualche approssimazione di questo target, ottenuta dai dati a disposizione: indichiamo tale approssimazione con $U_t$.\n",
        "\n",
        "Update rule:\n",
        "\n",
        "$w_{t+1} \\doteq \\vec{w_t} + α [U_t - v̂(s_t,\\vec{w_t}) ] \\nabla_\\vec{w} v̂(s_t,\\vec{w_t}) $\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Se $U_t$ è uno stimatore **unbiased** di $v_\\pi(s_t)$, cioè $\\mathbb{E}[U_t|s_t] = v_\\pi(s_t) \\; ∀ t $ , allora è garantito che il vettore $\\vec{w}$ converga ad un ottimo locale; nelle consuete condizioni di \"standard stochastic approximation\" per α.\n",
        "\n",
        "Esempio:\n",
        "Monte Carlo target è uno stimare unbiased. $U_t = G_t$.\n",
        "Per il Gradient Monte Carlo Algorithm vedi Sutton & Barto, 2018,  PDF 224.\n",
        "\n",
        "Bootstrapping targets sono stimantori **biased** , in quanto dipendono dal valore del vettore dei pesi $\\vec{w}$.  In generale, la convergenza del vettore $\\vec{w}$ non è garantita; ma in casi importanti in cui si sfrutta la modellizzazione lineare lo è.\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Pro-memoria\n",
        "\n",
        "Stimatori unbiased : non fanno ipotesi o assunzioni sulla distribuzione dei dati o sulla forma funzionale della funzione target (esempio decision trees o random forests).\n",
        "\n",
        "Stimatori biased : fanno ipotesi ed assunzioni. Ad esempio i metodi lineari: presuppongono che la funzione target sia lineare. Se la vera funzione target è non lineare, allora il modello di lineare commetterà errori sistematici nelle sue previsioni e apprenderà in modo parziale. Un altro esempio è l’algoritmo k-nearest neighbors che presuppone che i dati siano distribuiti uniformemente. Se i dati non sono distribuiti uniformemente, l’algoritmo potrebbe commettere errori sistematici nelle sue previsioni.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Ak7nVWw_BeAB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Modellizazzione lineare**\n",
        "\n",
        "Idea è di approssimare la funzione $v̂(s,\\vec{w})$ con un funzione lineare rispetto i pesi $\\vec{w}$.\n",
        "\n",
        "In sostanza, si consideri il vettore:\n",
        "$\\vec{x}(s) = (x_1(s), x_2(s),...,x_d(s) )^\\intercal $\n",
        "detto **feature vector** dello stato $s$.\n",
        "Questo ha la stessa dimensione $d$ del vettore $\\vec{w}$. Ogni componente $x_i(s)$ è una funzione del tipo $ x_i : S → \\mathbb{R}$. In questo modo si vuole rappresentare un certo stato $s$ del sistema con un feature vector. Esempio pratico, consideriamo un robot mobile che si muove in un ambiente dotato di punti di riferimento fissi; lo stato del robot in un dato istante può essere rappresentato da un vettore contenente le distanze del robot da ciascun punto di riferimento.\n",
        "\n",
        " I componenti di un feature vector possono essere definiti in molti modi diversi; vedi Sutton & Barto, 2018, PDF 232.\n",
        "\n",
        "Usando la modellizzazione lineare, abbiamo:\n",
        "\n",
        "$v̂(s,\\vec{w}) \\doteq \\vec{w} ^\\intercal \\vec{x}(s) = \\sum_i ^d w_i x_i(s) $\n",
        "\n",
        "il cui gradiente si semplifica così:\n",
        "\n",
        "  $ \\nabla_\\vec{w} v̂(s,\\vec{w}) = \\vec{x}(s) $\n",
        "\n",
        "\n",
        "e la regola di update dei pesi diventa, ad un certo step $t$:\n",
        "\n",
        "$\\vec{w}_{t+1} = \\vec{w}_{t} + \\alpha [U_t - (\\vec{w}_{t} ^\\intercal \\vec{x}(s_{t})) ]  \\vec{x}(s_{t}) $\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "Convergenza:\n",
        "\n",
        "Nel caso lineare, esiste un unico ottimo (o un insieme di ottimi equivalenti in casi degeneri). Pertanto, qualsiasi metodo che garantisce la convergenza a un ottimo locale, od ad  un punto vicino, converge automaticamente verso l'ottimo globale, od ad un punto vicino.\n",
        "\n",
        "- L'algoritmo Gradient Monte Carlo garantisce la convergenza di $\\vec{w}$  all'ottimo globale, quando l'approssimazione usata riguarda funzioni lineari.\n",
        "\n",
        "- Anche l'algoritmo Semi-gradient TD(n), per ogni $n$,  garantisce la convergenza, ad un punto vicino (che si trova all'interno di un certo limite di errore, che diminuisce con aumentare di $n$ ), all'ottimo globale (per più info Sutton & Barto, 2018, PDF 227).\n",
        "\n",
        "\n",
        "Nota: perchè \"Semi-gradinet\" ?\n",
        "\n",
        "I metodi TD non sono veri e propri \"gradient methods\". In tali metodi di bootstrapping (incluso DP), l'update del target : $U_t = R_{t+1} + γ \\cdot v̂(S_{t+1}, \\vec w_{t+1})$ , ma non è utilizzato esplicitamente nel calcolo del gradiente, viene infatti pensato come una costante. Vedi : Sutton & Barto, 2018, PDF 224. L'algoritmo richiede questa impostazione per garantire una ragionevole convergenza; ed i motivi precisi per cui questo sia necessario non sono ancora del tutto chiari.\n",
        "\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "La **Modellizzazione non lineare**, che utilizza funzioni non lineari approssimare la funzione $v̂(s,\\vec{w})$, in particolare usa reti neurali artificiali addestrate con backpropagation e varianti di SGD, ha guadagnato grande popolarità negli ultimi anni sotto il nome di **deep reinforcement learning**.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "nR4yvg2z9-U_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## On-policy Control\n",
        "\n",
        "\n",
        "Ora ci concentriamo sul problema del controllo, utilizzando un'approssimazione parametrica per la funzione valore-azione.\n",
        "\n",
        "$\\widehat{q}(s,a,\\vec{w}) \\approx q_{\\pi}(s, a) $\n",
        "\n",
        "Innanzitutto limitiamo la nostra attenzione al caso:\n",
        "- on-policy\n",
        "- episodico\n",
        "\n",
        "Nota:\n",
        "\n",
        "Attività episodiche (Episodic tasks): possiedono uno stato terminale, ovvero una condizione che conclude l'episodio. Spesso, la ricompensa viene assegnata alla fine dell'episodio.\n",
        "\n",
        "Attività continue (Continuous task ): non hanno uno stato terminale e si protraggono teoricamente all'infinito. La ricompensa non viene assegnata alla fine, in quanto non esiste fine, ma viene distribuita nel corso dell'attività."
      ],
      "metadata": {
        "id": "cC5CkqMyPjo-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Episodic Semi-Gradient one step Sarsa  - Modellizzazione lineare  - Mountain Car\n",
        "\n",
        "\n",
        "One-step Sarsa update:\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "$\\vec{w}_{t+1} \\leftarrow \\vec{w}_t + \\alpha [ R_{t+1} + \\gamma \\widehat{q}(s_{t+1},a_{t+1},\\vec{w}_t) - \\widehat{q}(s_{t},a_{t},\\vec{w}_t)] ∇ \\widehat{q}(s_{t},a_{t},\\vec{w}_t)  $\n",
        "\n",
        "\n",
        "Algoritmo completo: vedi Sutton & Barto, 2018, PDF 266\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Convergenza:\n",
        "\n",
        "Nel testo di Sutton & Barto si legge : *For a constant policy, this method converges in the same way that TD(0) does*.\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "Env : **Mountain Car**\n",
        "\n",
        "\n",
        " https://www.gymlibrary.dev/environments/classic_control/mountain_car/\n",
        "\n",
        "\n",
        "Sommario:\n",
        "\n",
        "\n",
        "L'obiettivo è raggiungere la bandiera posta in cima alla collina di destra il più velocemente possibile, pertanto l'agente viene penalizzato con una ricompensa di -1 per ogni timestep.\n",
        "\n",
        "\n",
        "L' auto per raggiungere la bandiera deve prima allontanarsi momentaneamente da questa: ovvero risalire il pendio ripido a sinistra; poi accelerare al massimo in avanti per guadagnare l'inerzia necessaria per completare la salita. Questo scenario rappresenta un semplice esempio di compito di controllo continuo, in cui la situazione peggiora temporaneamente (*farther from the goal*) prima di migliorare. Molti algoritmi di controllo tradizionali faticano ad affrontare questo tipo di sfide se non vengono guidati esplicitamente da un intervento umano.\n",
        "\n",
        "\n",
        "State space: un feature ndarray con forma (2,) in cui il primo elemento è la posizione dell'auto lungo l'asse x, il secondo la velocità dell'auto.\n",
        "\n",
        "\n",
        "Stato iniziale: Alla posizione dell'auto viene assegnato un valore casuale uniforme in [-0.6, -0.4]. La velocità iniziale dell'auto è sempre assegnata a 0.0\n",
        "\n",
        "Azioni discrete :\n",
        "\n",
        "0: accelera in retromarcia\n",
        "\n",
        "1: non accelera\n",
        "\n",
        "2: accelera in avanti\n",
        "\n",
        "Funzione di Transizione : Data un'azione, il sistema segue la seguente dinamica di transizione:\n",
        "\n",
        "$v_{t+1} = v_t + (azione - 1) * 0.001 - cos(3 * x_t) * 0.001$\n",
        "\n",
        "$x_{t+1} = x_t + v_{t+1}$\n",
        "\n",
        "La posizione $x$ viene confinata nell'intervallo [-1.2, 0.6] e la velocità $v$ viene confinata nell'intervallo [-0.07, 0.07]."
      ],
      "metadata": {
        "id": "bEZEcy9-IIu6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Moduli"
      ],
      "metadata": {
        "id": "gnviaHbNe1P0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import gym\n",
        "import numpy as np\n",
        "import random\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "import pickle\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "%matplotlib inline"
      ],
      "metadata": {
        "id": "mBmQzUC-hpPY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#####Rendering\n",
        "\n",
        "Lanciare i codici sottostanti"
      ],
      "metadata": {
        "id": "FzK7LtcPH9Gm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#codice ispirato : https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_12_02_qlearningreinforcement.ipynb\n",
        "\n",
        "try:\n",
        "    from google.colab import drive\n",
        "    %tensorflow_version 2.x\n",
        "    COLAB = True\n",
        "    print(\"Note: using Google CoLab\")\n",
        "except:\n",
        "    print(\"Note: not using Google CoLab\")\n",
        "    COLAB = False\n",
        "\n",
        "if COLAB:\n",
        "  !sudo apt-get install -y xvfb ffmpeg x11-utils\n",
        "  !pip install -q gym\n",
        "  !pip install -q 'imageio==2.5.0' # 'imageio==2.4.0'\n",
        "  !pip install -q PILLOW\n",
        "  !pip install -q 'pyglet==1.3.2'\n",
        "  !pip install -q pyvirtualdisplay\n",
        "  !pip install -q tf-agents\n",
        "  !pip install -q pygame"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jEmScDyVegbM",
        "outputId": "f5f894f6-4b4b-4eb2-9b54-7de8bc5c4684"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Colab only includes TensorFlow 2.x; %tensorflow_version has no effect.\n",
            "Note: using Google CoLab\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "x11-utils is already the newest version (7.7+5build2).\n",
            "ffmpeg is already the newest version (7:4.4.2-0ubuntu0.22.04.1).\n",
            "xvfb is already the newest version (2:21.1.4-2ubuntu1.7~22.04.10).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 45 not upgraded.\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tf_agents #TF-Agents is just used to render the mountain care environment\n",
        "from tf_agents.environments import suite_gym\n",
        "import PIL.Image\n",
        "import pyvirtualdisplay\n",
        "\n",
        "from gym.wrappers.record_video import RecordVideo\n",
        "import glob\n",
        "import io\n",
        "import base64\n",
        "from IPython.display import HTML\n",
        "from pyvirtualdisplay import Display\n",
        "from IPython import display as ipythondisplay\n",
        "\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()\n",
        "\n",
        "\n",
        "def show_video():\n",
        "    mp4list = glob.glob('video/*.mp4')\n",
        "    if len(mp4list) > 0:\n",
        "        mp4 = mp4list[0]\n",
        "        video = io.open(mp4, 'r+b').read()\n",
        "        encoded = base64.b64encode(video)\n",
        "        ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay\n",
        "                loop controls style=\"height: 400px;\">\n",
        "                <source src=\"data:video/mp4;base64,{0}\"\n",
        "                type=\"video/mp4\" />\n",
        "             </video>'''.format(encoded.decode('ascii'))))\n",
        "    else:\n",
        "        print(\"Could not find video\")\n",
        "\n",
        "\n",
        "def wrap_env(env):\n",
        "    env = RecordVideo(env, './video',  episode_trigger = lambda episode_number: True)\n",
        "    return env\n",
        "\n",
        "\n",
        "#display = pyvirtualdisplay.Display(visible=0, size=size=(1400, 900)).start()\n",
        "#env_name = 'MountainCar-v0'\n",
        "#env = suite_gym.load(env_name)\n",
        "#env.reset()\n",
        "#PIL.Image.fromarray(env.render())\n"
      ],
      "metadata": {
        "id": "3zI5UkEjlSbh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Test: episodio di Mountain Car"
      ],
      "metadata": {
        "id": "_5Ub0OAgh0z0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env = wrap_env(gym.make(\"MountainCar-v0\"))\n",
        "\n",
        "#help(env.unwrapped)\n",
        "print('State space [position velocity] :', env.observation_space)\n",
        "print('Action space:', env.action_space)\n",
        "print('Max Episode Steps: 200')\n",
        "print(\"\")\n",
        "\n",
        "state = env.reset()\n",
        "done = False\n",
        "\n",
        "i = 0\n",
        "while not done:\n",
        "\n",
        "    # Nel if/else successivo il \"software\" che gestisce il movimento di un'auto.\n",
        "    # Questo applica sempre la forza nella direzione del movimento,\n",
        "    # in avanti o dietro. La frenata non è mai utilizzata.\n",
        "    if state[1] > 0:\n",
        "        action = 2\n",
        "    else:\n",
        "        action = 0\n",
        "\n",
        "    state, reward, done, _ = env.step(action)\n",
        "    env.render()\n",
        "\n",
        "    # Debug\n",
        "    if i % 10 == 0:\n",
        "      print(f\"Step {i}: State:[position velocity]={state}, Reward={reward}, Action={action}\")\n",
        "\n",
        "    i += 1\n",
        "\n",
        "    if done == True:\n",
        "      print(f\" - Terminal state occurs in\\n   Step {i}: State={state}, Reward={reward}\")\n",
        "\n",
        "env.close()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xKbRSNEUh75L",
        "outputId": "434c0a3a-529b-4933-dd3a-487eb3054c47"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "State space [position velocity] : Box([-1.2  -0.07], [0.6  0.07], (2,), float32)\n",
            "Action space: Discrete(3)\n",
            "Max Episode Steps: 200\n",
            "\n",
            "Step 0: State:[position velocity]=[-0.48812014 -0.00127509], Reward=-1.0, Action=0\n",
            "Step 10: State:[position velocity]=[-0.5643934 -0.0120168], Reward=-1.0, Action=0\n",
            "Step 20: State:[position velocity]=[-0.70639503 -0.01446564], Reward=-1.0, Action=0\n",
            "Step 30: State:[position velocity]=[-0.8213755 -0.008058 ], Reward=-1.0, Action=0\n",
            "Step 40: State:[position velocity]=[-0.8400478   0.00629901], Reward=-1.0, Action=2\n",
            "Step 50: State:[position velocity]=[-0.62304217  0.0325388 ], Reward=-1.0, Action=2\n",
            "Step 60: State:[position velocity]=[-0.24632375  0.0378255 ], Reward=-1.0, Action=2\n",
            "Step 70: State:[position velocity]=[0.0655299  0.02472444], Reward=-1.0, Action=2\n",
            "Step 80: State:[position velocity]=[0.2414313  0.01263457], Reward=-1.0, Action=2\n",
            "Step 90: State:[position velocity]=[0.32888922 0.0063215 ], Reward=-1.0, Action=2\n",
            "Step 100: State:[position velocity]=[0.37693948 0.0040396 ], Reward=-1.0, Action=2\n",
            "Step 110: State:[position velocity]=[0.41828203 0.00463403], Reward=-1.0, Action=2\n",
            "Step 120: State:[position velocity]=[0.4834298  0.00873422], Reward=-1.0, Action=2\n",
            " - Terminal state occurs in\n",
            "   Step 123: State=[0.50306696 0.01020349], Reward=-1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#il video dell'episodio\n",
        "show_video()"
      ],
      "metadata": {
        "id": "q4wXaow8iY7v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Tile Coding (bucketization)\n",
        "\n",
        "Il Tile Coding è una tecnica per codificare in modo discreto uno spazio continuo (in pratica facendo leva sulla mappatura dello spazio continuo in un vettore con $n$ binary feature : $[0,1,0,0,...1,0,1]$); questo  semplifica la gestione delle value-action fuction.\n",
        "\n",
        "Tile coding dimostra una notevole efficienza nel risolvere l'ambinete Mountain Car; in modo considerevolmente più veloce rispetto a diverse alternative.\n",
        "\n",
        "\n",
        "Idea di base è spiegata molto bene qui:\n",
        "\n",
        "\n",
        "\n",
        "https://medium.com/criteo-engineering/tile-coding-an-efficient-sparse-coding-method-for-real-valued-data-e787eddf630a\n",
        "\n",
        "\n",
        "https://github.com/MeepMoop/tilecoding?tab=readme-ov-file\n",
        "\n",
        "Aspetti teorici:\n",
        "\n",
        "https://www.researchgate.net/publication/220864728_Tile_Coding_Based_on_Hyperplane_Tiles\n",
        "\n"
      ],
      "metadata": {
        "id": "KZhKoGr3fYqq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Tile Coding Software version 3.0beta by Rich Sutton\n",
        "#http://incompleteideas.net/tiles/tiles3.html\n",
        "\n",
        "\"\"\"\n",
        "Tile Coding Software version 3.0beta\n",
        "by Rich Sutton\n",
        "based on a program created by Steph Schaeffer and others\n",
        "External documentation and recommendations on the use of this code is available in the\n",
        "reinforcement learning textbook by Sutton and Barto, and on the web.\n",
        "These need to be understood before this code is.\n",
        "\n",
        "This software is for Python 3 or more.\n",
        "\n",
        "This is an implementation of grid-style tile codings, based originally on\n",
        "the UNH CMAC code (see http://www.ece.unh.edu/robots/cmac.htm), but by now highly changed.\n",
        "Here we provide a function, \"tiles\", that maps floating and integer\n",
        "variables to a list of tiles, and a second function \"tiles-wrap\" that does the same while\n",
        "wrapping some floats to provided widths (the lower wrap value is always 0).\n",
        "\n",
        "The float variables will be gridded at unit intervals, so generalization\n",
        "will be by approximately 1 in each direction, and any scaling will have\n",
        "to be done externally before calling tiles.\n",
        "\n",
        "Num-tilings should be a power of 2, e.g., 16. To make the offsetting work properly, it should\n",
        "also be greater than or equal to four times the number of floats.\n",
        "\n",
        "The first argument is either an index hash table of a given size (created by (make-iht size)),\n",
        "an integer \"size\" (range of the indices from 0), or nil (for testing, indicating that the tile\n",
        "coordinates are to be returned without being converted to indices).\n",
        "\"\"\"\n",
        "\n",
        "basehash = hash\n",
        "\n",
        "class IHT:\n",
        "    \"Structure to handle collisions\"\n",
        "    def __init__(self, sizeval):\n",
        "        self.size = sizeval\n",
        "        self.overfullCount = 0\n",
        "        self.dictionary = {}\n",
        "\n",
        "    def __str__(self):\n",
        "        \"Prepares a string for printing whenever this object is printed\"\n",
        "        return \"Collision table:\" + \\\n",
        "               \" size:\" + str(self.size) + \\\n",
        "               \" overfullCount:\" + str(self.overfullCount) + \\\n",
        "               \" dictionary:\" + str(len(self.dictionary)) + \" items\"\n",
        "\n",
        "    def count (self):\n",
        "        return len(self.dictionary)\n",
        "\n",
        "    def fullp (self):\n",
        "        return len(self.dictionary) >= self.size\n",
        "\n",
        "    def getindex (self, obj, readonly=False):\n",
        "        d = self.dictionary\n",
        "        if obj in d: return d[obj]\n",
        "        elif readonly: return None\n",
        "        size = self.size\n",
        "        count = self.count()\n",
        "        if count >= size:\n",
        "            if self.overfullCount==0: print('IHT full, starting to allow collisions')\n",
        "            self.overfullCount += 1\n",
        "            return basehash(obj) % self.size\n",
        "        else:\n",
        "            d[obj] = count\n",
        "            return count\n",
        "\n",
        "##### End Class ######\n",
        "\n",
        "\n",
        "def hashcoords(coordinates, m, readonly=False):\n",
        "    if type(m)==IHT: return m.getindex(tuple(coordinates), readonly)\n",
        "    if type(m)==int: return basehash(tuple(coordinates)) % m\n",
        "    if m==None: return coordinates\n",
        "\n",
        "from math import floor, log\n",
        "from itertools import zip_longest\n",
        "\n",
        "def tiles_funct (ihtORsize, numtilings, floats, ints=[], readonly=False):\n",
        "    \"\"\"returns num-tilings tile indices corresponding to the floats and ints\"\"\"\n",
        "    qfloats = [floor(f*numtilings) for f in floats]\n",
        "    Tiles = []\n",
        "    for tiling in range(numtilings):\n",
        "        tilingX2 = tiling*2\n",
        "        coords = [tiling]\n",
        "        b = tiling\n",
        "        for q in qfloats:\n",
        "            coords.append( (q + b) // numtilings )\n",
        "            b += tilingX2\n",
        "        coords.extend(ints)\n",
        "        Tiles.append(hashcoords(coords, ihtORsize, readonly))\n",
        "    return Tiles\n",
        "\n",
        "def tileswrap (ihtORsize, numtilings, floats, wrapwidths, ints=[], readonly=False):\n",
        "    \"\"\"returns num-tilings tile indices corresponding to the floats and ints, wrapping some floats\"\"\"\n",
        "    qfloats = [floor(f*numtilings) for f in floats]\n",
        "    Tiles = []\n",
        "    for tiling in range(numtilings):\n",
        "        tilingX2 = tiling*2\n",
        "        coords = [tiling]\n",
        "        b = tiling\n",
        "        for q, width in zip_longest(qfloats, wrapwidths):\n",
        "            c = (q + b%numtilings) // numtilings\n",
        "            coords.append(c%width if width else c)\n",
        "            b += tilingX2\n",
        "        coords.extend(ints)\n",
        "        Tiles.append(hashcoords(coords, ihtORsize, readonly))\n",
        "    return Tiles\n"
      ],
      "metadata": {
        "id": "WxbycpOrhoNy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Testing Tile Coding Software\n",
        "\n",
        "POSITION_MIN = -1.2\n",
        "POSITION_MAX = 0.5\n",
        "VELOCITY_MIN = -0.07\n",
        "VELOCITY_MAX = 0.07\n",
        "\n",
        "num_tiles = 10 # number of tiles. Here both the width and height of the tile coder are the same\n",
        "num_tilings = 8\n",
        "\n",
        "# Scale position and velocity by multiplying the inputs of each by their scale\n",
        "position_scale = num_tiles / (POSITION_MAX - POSITION_MIN)\n",
        "velocity_scale = num_tiles / (VELOCITY_MAX - VELOCITY_MIN)\n",
        "\n",
        "print(\"position_scale=\",position_scale)\n",
        "print(\"velocity_scale=\",velocity_scale)\n",
        "\n",
        "iht = IHT(1024)  # The size of the index hash table, typically a power of 2\n",
        "\n",
        "s1 = tiles_funct (iht, num_tilings, [-0.6*position_scale, 0.01*velocity_scale])\n",
        "s2 = tiles_funct (iht, num_tilings, [-0.5*position_scale, 0.02*velocity_scale])\n",
        "s3 = tiles_funct (iht, num_tilings, [ 0.1*position_scale, 0.0*velocity_scale])\n",
        "s4 = tiles_funct (iht, num_tilings, [ -1.0*position_scale, 0.05*velocity_scale])\n",
        "s5 = tiles_funct (iht, num_tilings, [ -1.0*position_scale, 0.045*velocity_scale])\n",
        "s6 = tiles_funct (iht, num_tilings, [-0.5*position_scale, 0.01*velocity_scale])\n",
        "s7 = tiles_funct (iht, num_tilings, [ 0.5*position_scale, -0.07*velocity_scale])\n",
        "\n",
        "print(\"Decode output: \\n\",s1,s2,s3)\n",
        "print(\"Decode output: \\n\",s4,s5,s6 )\n",
        "print(\"Decode output: \\n\",s7 )\n",
        "#\n",
        "# As the last example shows, far away points have no tiles in common. It also\n",
        "# shows that the idea of a 10 by 10 space is really just in our minds. As far as\n",
        "# the software is concerned there is a 2D plane stretching to infinity in all\n",
        "# directions. (Of course if you visit too much of the space you will run out of\n",
        "# indices -- you only have as many as you specified with the size of iht.)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bVBGIAY-Yo4R",
        "outputId": "16080f84-1508-4b15-96b8-e2dd992b63ec"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "position_scale= 5.882352941176471\n",
            "velocity_scale= 71.42857142857142\n",
            "Decode output: \n",
            " [0, 1, 2, 3, 4, 5, 6, 7] [8, 9, 10, 11, 12, 13, 14, 15] [16, 17, 18, 19, 20, 21, 22, 23]\n",
            "Decode output: \n",
            " [24, 25, 26, 27, 28, 29, 30, 31] [24, 25, 32, 27, 33, 29, 30, 34] [35, 9, 36, 37, 12, 5, 6, 7]\n",
            "Decode output: \n",
            " [38, 39, 40, 41, 42, 43, 44, 45]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Parametri\n"
      ],
      "metadata": {
        "id": "yGPTU76i4p_b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "size_iht = 32768\n",
        "iht = IHT(size_iht)\n",
        "\n",
        "list_actions = [0,1,2]"
      ],
      "metadata": {
        "id": "H43PyMQ54uPr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Data Processing Functions / Class"
      ],
      "metadata": {
        "id": "YEuUA7Ww3A5P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# codice ispirato :\n",
        "# https://github.com/self-supervisor/SARSA-Mountain-Car-Sutton-and-Barto/blob/main/SARSA%20Mountain%20Car.ipynb\n",
        "\n",
        "#Codifica Tile\n",
        "def get_feature_vector(position, velocity, action):\n",
        "    indices = tiles_funct(iht, 8, [8*position/(0.5+1.2), 8*velocity/(0.07+0.07)], action)\n",
        "    #print(\"indices=\",indices) #indices= [727, 581, 821, 571, 830, 631, 577, 833]\n",
        "    feature_vector = np.zeros((size_iht,))\n",
        "    feature_vector[indices] = 1\n",
        "    # print(feature_vector) # [0. 1. 0. ... 0. 1. 0. 0.]\n",
        "    # print(len(feature_vector)) # 32768\n",
        "    return feature_vector\n",
        "\n",
        "###\n",
        "\n",
        "def save_params(w):\n",
        "        pickle.dump(w, open('weights.pkl', 'wb'))\n",
        "\n",
        "\n",
        "def load_params():\n",
        "        w = pickle.load(open('weights.pkl', 'rb'))\n",
        "        return w"
      ],
      "metadata": {
        "id": "3VQFrAtU3DFN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Agent(object):\n",
        "\n",
        "    def __init__(self,\n",
        "                 param_vector_size,\n",
        "                 learning_rate,\n",
        "                 discount):\n",
        "        self.weights = np.random.uniform(low=-0.1, high=0.1, size=param_vector_size)\n",
        "        self.action_space = list_actions\n",
        "        self.learning_rate = learning_rate\n",
        "        self.discount = discount\n",
        "        self.feature_vector = get_feature_vector(0.0, 0.0, [0])\n",
        "        self.action = [1]\n",
        "\n",
        "\n",
        "    def step(self, reward, new_state, eps, done):\n",
        "        if done:\n",
        "            self.update_weights(reward, done)\n",
        "        else:\n",
        "            new_action = self.epsilon_greedy(new_state, eps)\n",
        "            new_feature_vector = get_feature_vector(new_state[0],\n",
        "                                                    new_state[1],\n",
        "                                                    new_action)\n",
        "            self.update_weights(reward, done, new_feature_vector)\n",
        "            self.feature_vector = new_feature_vector\n",
        "            self.action = new_action\n",
        "        return self.action\n",
        "\n",
        "    def epsilon_greedy(self, new_state, eps):\n",
        "        # ricava l'azione greedy\n",
        "        values = []\n",
        "        for action_candidate in self.action_space:\n",
        "                new_feature_vector = get_feature_vector(new_state[0],\n",
        "                                                        new_state[1],\n",
        "                                                        [action_candidate])\n",
        "                values.append(self.compute_value(new_feature_vector))\n",
        "                # https://stackoverflow.com/questions/16945518/\n",
        "                #finding-the-index-of-the-value-which-is-the-min-or-\n",
        "                #max-in-python\n",
        "        greedy_action = [values.index(max(values))] # Nota, values ha una forma del tipo [-4.191, -3.1879, -2.7512]\n",
        "\n",
        "        if (random.uniform(0,1.0) > eps):\n",
        "            #\n",
        "            # policy greedy\n",
        "            #\n",
        "            action = greedy_action\n",
        "        else:\n",
        "            #\n",
        "            #policy random non-greedy\n",
        "            #\n",
        "            non_greedy_actions = list(set(list_actions) - {greedy_action[0]})\n",
        "            action = [np.random.choice(non_greedy_actions)]\n",
        "\n",
        "        return action\n",
        "\n",
        "    def compute_value(self, feature_vector):\n",
        "        return np.dot(self.weights, feature_vector)\n",
        "\n",
        "    def update_weights(self,\n",
        "                       reward,\n",
        "                       terminal,\n",
        "                       feature_vector_tp1=np.zeros((size_iht,))): # Nota : feature_vector_tp1 = sarebbe q(S',A',w)\n",
        "        if terminal:\n",
        "            error = reward - self.compute_value(self.feature_vector)\n",
        "        else:\n",
        "            error = reward + self.discount * self.compute_value(feature_vector_tp1)- self.compute_value(self.feature_vector)\n",
        "\n",
        "        self.weights += self.learning_rate * error * self.feature_vector # Nota, come da teoria: il gradinte = feature_vector"
      ],
      "metadata": {
        "id": "3D_KP-54LXVY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train(environment, agent, epsilon, num_episodes=1_000):\n",
        "    env = environment\n",
        "    episode_rewards = []\n",
        "    eps = epsilon\n",
        "    print(\"\\n\\tEpsilon =\", eps)\n",
        "\n",
        "    for episode in range(1,num_episodes+1):\n",
        "\n",
        "        ob = env.reset()\n",
        "        action = agent.epsilon_greedy(ob, eps)  # action_agent.epsilon_greedy(ob) = [2]\n",
        "        episode_rewards_sum = 0\n",
        "        done = False\n",
        "        count = 0\n",
        "\n",
        "        while count < 3_000:  # max 3_000 step per episodio\n",
        "            count += 1\n",
        "\n",
        "            if count == 3_000:\n",
        "                done = True\n",
        "            else:\n",
        "                done = False\n",
        "\n",
        "            new_ob, reward, done_env, _ = env.step(action[0])\n",
        "\n",
        "            if new_ob[0] >= 0.5:\n",
        "                #debug\n",
        "                print(f\"Goal episod = {episode}  Total reward={episode_rewards_sum}\")\n",
        "                count = 3_000\n",
        "\n",
        "            episode_rewards_sum += reward\n",
        "            action = agent.step(reward, new_ob, eps, done) # action_agent.step = [1]\n",
        "\n",
        "        # Per Gestire la dinamica di epsilon\n",
        "        if (episode > 0) and (episode % (num_episodes/3) == 0):\n",
        "           eps = round(eps - 0.05, 3)\n",
        "           if episode != num_episodes:\n",
        "              print(\"\\n\\tEpsilon =\", eps)\n",
        "\n",
        "        episode_rewards.append(episode_rewards_sum)\n",
        "\n",
        "    save_params(agent.weights)\n",
        "    print(\"\\n...Params Saved...\\n\")\n",
        "    env.close()\n",
        "    print(\"\\n...End of training...\\n\")\n",
        "\n",
        "    return episode_rewards, agent"
      ],
      "metadata": {
        "id": "QjsmgYYNL3Op"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Training loop"
      ],
      "metadata": {
        "id": "n6N1YD2bRl7X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "agent = Agent(param_vector_size=size_iht,\n",
        "              learning_rate=(0.5/8),\n",
        "              discount=0.9)\n",
        "\n",
        "\n",
        "episode_rewards, agent = train(gym.make(\"MountainCar-v0\"), agent, epsilon=0.2, num_episodes=30)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BoSnFq-TMToX",
        "outputId": "21e4590d-e538-43af-aa1f-b88bfee10cf1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\tEpsilon = 0.2\n",
            "Goal episod = 1  Total reward=-1238.0\n",
            "Goal episod = 2  Total reward=-1176.0\n",
            "Goal episod = 3  Total reward=-657.0\n",
            "Goal episod = 4  Total reward=-706.0\n",
            "Goal episod = 5  Total reward=-411.0\n",
            "Goal episod = 6  Total reward=-395.0\n",
            "Goal episod = 7  Total reward=-471.0\n",
            "Goal episod = 8  Total reward=-419.0\n",
            "Goal episod = 9  Total reward=-396.0\n",
            "Goal episod = 10  Total reward=-387.0\n",
            "\n",
            "\tEpsilon = 0.15\n",
            "Goal episod = 11  Total reward=-360.0\n",
            "Goal episod = 12  Total reward=-292.0\n",
            "Goal episod = 13  Total reward=-304.0\n",
            "Goal episod = 14  Total reward=-237.0\n",
            "Goal episod = 15  Total reward=-246.0\n",
            "Goal episod = 16  Total reward=-387.0\n",
            "Goal episod = 17  Total reward=-160.0\n",
            "Goal episod = 18  Total reward=-236.0\n",
            "Goal episod = 19  Total reward=-293.0\n",
            "Goal episod = 20  Total reward=-226.0\n",
            "\n",
            "\tEpsilon = 0.1\n",
            "Goal episod = 21  Total reward=-187.0\n",
            "Goal episod = 22  Total reward=-201.0\n",
            "Goal episod = 23  Total reward=-201.0\n",
            "Goal episod = 24  Total reward=-492.0\n",
            "Goal episod = 25  Total reward=-228.0\n",
            "Goal episod = 26  Total reward=-294.0\n",
            "Goal episod = 27  Total reward=-250.0\n",
            "Goal episod = 28  Total reward=-801.0\n",
            "Goal episod = 29  Total reward=-193.0\n",
            "Goal episod = 30  Total reward=-149.0\n",
            "\n",
            "...Params Saved...\n",
            "\n",
            "\n",
            "...End of training...\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Plotting"
      ],
      "metadata": {
        "id": "UZiCrlybMrTG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(3, 3))\n",
        "\n",
        "plt.plot(episode_rewards)\n",
        "plt.ylabel(\"Rewards\")\n",
        "plt.xlabel(\"Episodes\")\n",
        "plt.title(\"SARSA one step on Mountain Car \")\n",
        "\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 333
        },
        "id": "hJkDZey6MYJx",
        "outputId": "c9e67342-65b1-4eed-a269-a68f7fe64b33"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 300x300 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAE8CAYAAABuPhIPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABVjUlEQVR4nO3dd1xTV/8H8E8SSAgbZIlscWtRUSyOukFLrbSO/qptcT9abR3U9dg62lrqrNY626fq07prtU+tiypuFBcOxIGiUDS4IGEGSM7vD8yVQIAAIcvv+/XKS3Pvyb0nF/LN4dzvOYfHGGMghBBSr/iGrgAhhLwKKNgSQogeULAlhBA9oGBLCCF6QMGWEEL0gIItIYToAQVbQgjRAwq2hBCiBxRsCSFEDyjYEkL05v79++DxeNi0aZOhq6J3FGw1uHbtGgYPHgxfX19YWVmhUaNG6Nu3L1atWlXpa4YOHQoej4eZM2dq3H/s2DHweDzuIRAI4ObmhsGDByM5OVnja/788090794dbm5usLa2RkBAAIYOHYqDBw9qLK9QKODp6Qkej4cDBw7U/I0bga1bt2LFihWGrka9K/v78Ouvv2os06VLF/B4PLRu3VrPtavcmjVrjDZQ7tmzB/3794eLiwuEQiE8PT0xdOhQHD161NBVAwDwaG4EdWfOnEHPnj3h4+ODqKgoeHh4ID09HWfPnsXdu3eRkpJS4TUymQzu7u7w8PCAQqHAgwcPwOPx1MocO3YMPXv2xKeffoqOHTuiuLgYV69exbp162BjY4Pr16/Dw8ODK7906VJMnz4d3bt3x8CBA2FtbY2UlBT8/fffCAoK0vgLHxsbi7CwMPj5+aFLly6VfoiN2VtvvYXr16/j/v37hq5KvVL9PlhZWaFnz57Yv3+/2v779+/D398fVlZWaNy4Ma5fv26gmqpr3bo1XFxccOzYsVq9njEGuVwOS0tLCAQCndSJMYZRo0Zh06ZNaNeuHQYPHgwPDw88evQIe/bswcWLF3H69Gl07txZJ+erS0VJGW+++SZzdXVlWVlZFfZlZmZqfM3PP//MLC0t2dGjRxkAduzYsQpl4uLiGAC2a9cute1r165lANiiRYu4bcXFxcze3p717dtX4/kqq8dHH33E2rdvz1auXMlsbGxYbm5uZW/TaEVERDBfX19DV6PeqX4f3n33XWZhYcGePHmitn/hwoXM3d2dde3albVq1cpAtayoVatWrHv37oauhpolS5YwAGzKlClMqVRW2P/f//6XnTt3rs7nKSgoYAqFotavp2BbTrNmzViPHj1q9JrevXuzN998kzHGWIsWLdjYsWMrlKks2F6/fp0BYOPGjeO2PXr0iAFg8+fP17oO+fn5zM7Oji1evJg9evSI8fl8tmXLFq1ff/fuXTZ48GDm5OTExGIx69SpE9u3b5/G97Bjxw729ddfs0aNGjGRSMR69erF7ty5U+05ZDIZmzx5MvP19WVCoZC5urqyPn36sIsXLzLGGOvevTsDoPYoG3gLCwvZ3LlzWePGjZlQKGReXl5s+vTprLCwUO08ANjEiRPZr7/+ypo2bcpEIhFr3749O378uFbXIjMzk40aNYq5ubkxkUjEXnvtNbZp0ya1MqmpqQwAW7JkCVu/fj0LCAhgQqGQdejQgSUkJFR7DtW13Lx5M7OxsWFr1qxR29+qVSv2ySefsO7du1cItsXFxezLL7/kzunr68tmz56t8TrMmzevwrl9fX1ZVFQU93zjxo0MADt16hSbOnUqc3FxYdbW1iwyMpI9fvxY7XXlfz6qwPvs2TMWHR3NWrduzWxsbJidnR3r168fS0xM1HjdNm7cyG2LiopiNjY27J9//mEDBw5kNjY2zMXFhUVHR7OSkpIqr2N+fj5zdnZmzZs3r7ZsTeqp+vls27aNzZkzh3l6ejIej6exEaYtCrblhIWFMTs7O3bt2jWtymdkZDA+n89++eUXxhhjX375JXNycmJyuVytXGXBdt++fQwAmzlzJrdNoVAwsVjMgoOD2bNnz7Sqx/bt2xmPx2NpaWmMMcZ69erFfQFURyKRMHd3d2ZnZ8fmzJnDli9fzoKCghifz2e///57hffQrl07FhwczL777js2f/58Zm1tzUJCQqo9z7Bhw5hQKGTTpk1jP/30E1u0aBEbMGAA+/XXXxljjB0+fJi1bduWubi4sF9++YX98ssvbM+ePdw1CQsLY9bW1mzKlCls/fr1bNKkSczCwoINHDhQ7TwAWOvWrZmLiwv78ssv2aJFi5ivry8Ti8XV/lzz8/NZixYtmKWlJZs6dSr7/vvvWbdu3RgAtmLFCq6cKmi0a9eOBQYGskWLFrHFixczFxcX5uXlxYqKiqo8T9nfh2HDhrFu3bpx+xITExkAFh8frzHYRkVFMQBs8ODBbPXq1eyjjz5iAFhkZGSF61CTYNuuXTvWq1cvtmrVKhYdHc0EAgEbOnQoV27Pnj3My8uLNW/enPv5HD58mDHG2Pnz51njxo3ZrFmz2Pr169mXX37JGjVqxBwcHFhGRkaF61Y+2FpZWbFWrVqxUaNGsbVr17JBgwYxABW+hMo7fPgwA8C+/PLLKsupaFtP1c+nZcuWrG3btmz58uUsJiaG5eXlaXUeTSjYlnP48GEmEAiYQCBgoaGhbMaMGezQoUOVfniWLl3KxGIxk8lkjDHGbt++zQBwQUJF9cP7+eef2ZMnT9jDhw/ZwYMHWWBgIOPxeBVaQ3PnzmUAmI2NDevfvz9buHAh1wLU5K233mJdunThnm/YsIFZWFiotUwqM2XKFAaAnTx5ktuWk5PD/P39mZ+fH/enk+o9tGjRQu3LZOXKlQxAtYHMwcGBTZw4scoylXUj/PLLL4zP56vVkTHG1q1bxwCw06dPc9tUra4LFy5w2x48eMCsrKzYO++8U+X5V6xYwQBwXwCMMVZUVMRCQ0OZra0t93NWBY0GDRqw58+fc2X/+OMPBoD9+eefVZ6nbLDdt2+f2hfl9OnTWUBAAGOMVQi2qkA8ZswYteN99tlnDAA7evSo2nWoSbDt06eP2p/hU6dOZQKBgGVnZ3PbKutGKCwsrPAndmpqKhOJRGqBsLJgqylgqr7Uq6L63Sv/eauMtvVU/XwCAgJYfn6+VseuDmUjlNO3b1/Ex8fj7bffxpUrV7B48WKEh4ejUaNG+N///leh/JYtWxAREQE7OzsAQJMmTRAcHIwtW7ZoPP6oUaPg6uoKT09P9OvXD1KpFL/88gs6duyoVm7BggXYunUr2rVrh0OHDmHOnDkIDg5G+/btK2QvPHv2DIcOHcL777/PbRs0aBB4PB527txZ7Xvev38/QkJC0LVrV26bra0txo0bh/v37+PGjRtq5UeOHAmhUMg979atGwDg3r17VZ7H0dER586dw8OHD6utU3m7du1CixYt0Lx5czx9+pR79OrVCwAQFxenVj40NBTBwcHccx8fHwwcOBCHDh2CQqGo9Dz79++Hh4eH2rW0tLTEp59+itzcXBw/flyt/HvvvQcnJyfuubbXoqywsDA4Oztj+/btYIxh+/btaucvXz8AmDZtmtr26OhoAMBff/2l9XnLGzdunNqN3W7dunE3fKsjEonA55eGE4VCgWfPnsHW1hbNmjXDpUuXtDr/+PHj1Z5369at2usok8kAgPv86bqeUVFREIvFWh27OhRsNejYsSN+//13ZGVlISEhAbNnz0ZOTg4GDx6sFniSk5Nx+fJldOnSBSkpKdyjR48e2LdvH/eLUNbcuXMRGxuLPXv24KOPPoJUKuV++OW9//77OHnyJLKysnD48GEMGzYMly9fxoABA1BYWMiV27FjB4qLi9GuXTuuDs+fP0enTp0qDfplPXjwAM2aNauwvUWLFtz+snx8fNSeq4JNVlZWledZvHgxrl+/Dm9vb4SEhGD+/PlaB6U7d+4gKSkJrq6uao+mTZsCAB4/fqxWvkmTJhWO0bRpU+Tn5+PJkyeVnufBgwdo0qRJhZ+Jrq9FWZaWlhgyZAi2bt2KEydOID09HcOGDau0fnw+H4GBgWrbPTw84OjoqFVgrExd3otSqcR3332HJk2aQCQSwcXFBa6urrh69SqkUmm1r7eysoKrq2uF81d3bnt7ewBATk5OteeoTT39/f21Oq42LHR2JDMkFArRsWNHdOzYEU2bNsXIkSOxa9cuzJs3DwC41KqpU6di6tSpFV6/e/dujBw5Um1bmzZt0KdPHwBAZGQk8vPzMXbsWHTt2hXe3t4a62Fvb4++ffuib9++sLS0xObNm3Hu3Dl0794dALiA2qVLF42vv3fvHgICAmpxBTSrLGWHVZNFOHToUHTr1g179uzB4cOHsWTJEixatAi///47+vfvX+VrlUol2rRpg+XLl2vcX9m1q2+1vRblDRs2DOvWrcP8+fMRFBSEli1bVlm+fGphTVTWsq/Le/nmm2/wxRdfYNSoUfjqq6/g7OwMPp+PKVOmQKlUVvv62qaBNW/eHEBpbnxkZKTO66mrVi1AwVZrHTp0AAA8evQIQOkv4NatW9GzZ098/PHHFcp/9dVX2LJlS4VgW963336LPXv2YOHChVi3bp1W9di8eTNXj9TUVJw5cwaTJk3igq+KUqnEhx9+iK1bt+Lzzz+v9Ji+vr64detWhe03b97k9utKw4YN8fHHH+Pjjz/G48eP0b59eyxcuJALtpUFkcaNG+PKlSvo3bu3VoHmzp07Fbbdvn0b1tbWFVpQZfn6+uLq1atQKpVqrdv6uBZlde3aFT4+Pjh27BgWLVpUZf2USiXu3LnDtbYBIDMzE9nZ2Wr1c3JyQnZ2ttrri4qKuN+d2qjs2v/222/o2bMn/vOf/6htz87OhouLS63PV52uXbvCyckJ27Ztw7///e9qg7ah6glQN0IFcXFxGr/JVX1lqj+3T58+jfv372PkyJEYPHhwhcd7772HuLi4avsnGzdujEGDBmHTpk2QSCQAgPz8fMTHx2ssrxoZpqqHqlU7Y8aMCnUYOnQounfvXm1XwptvvomEhAS1c+bl5WHDhg3w8/OrtpWlDYVCUeHPNDc3N3h6ekIul3PbbGxsNP45N3ToUGRkZODHH3+ssK+goAB5eXlq2+Lj49X64NLT0/HHH38gLCysyg/km2++CYlEgh07dnDbSkpKsGrVKtja2lb4QtMVHo+H77//HvPmzcOHH35YZf0AVBhlp2rxR0REcNsaN26MEydOqJXbsGFDlX3W1bGxsakQwIHSlmn5z82uXbuQkZFR63Npw9raGjNnzkRycjJmzpyp8bP766+/IiEhwaD1BKhlW8Enn3yC/Px8vPPOO2jevDmKiopw5swZ7NixA35+flxLdcuWLRAIBGq/3GW9/fbbmDNnDrZv317hZkZ506dPx86dO7FixQp8++23yM/PR+fOnfH666+jX79+8Pb2RnZ2Nvbu3YuTJ08iMjIS7dq14+rRtm3bSv+Mfvvtt/HJJ5/g0qVLaN++vcYys2bNwrZt29C/f398+umncHZ2xubNm5Gamordu3dX2qdcEzk5OfDy8sLgwYMRFBQEW1tb/P333zh//jyWLVvGlQsODsaOHTswbdo0dOzYEba2thgwYAA+/PBD7Ny5E+PHj0dcXBy6dOkChUKBmzdvYufOnTh06BD31wdQOtIpPDwcn376KUQiEdasWQOg9MZjVcaNG4f169djxIgRuHjxIvz8/PDbb7/h9OnTWLFihdY3Ympj4MCBGDhwYJVlgoKCEBUVhQ0bNiA7Oxvdu3dHQkICNm/ejMjISPTs2ZMrO2bMGIwfPx6DBg1C3759ceXKFRw6dKhOLbjg4GCsXbsWX3/9NQIDA+Hm5oZevXrhrbfewpdffomRI0eic+fOuHbtGrZs2aLT7qvKTJ8+HUlJSVi2bBni4uK4EWQSiQR79+5FQkICzpw5AwAGrSelfpVz4MABNmrUKNa8eXNma2vLhEIhCwwMZJ988gk3cquoqIg1aNBALTdSE39/f9auXTvGWOV5tio9evRg9vb2LDs7mxUXF7Mff/yRRUZGMl9fXyYSiZi1tTVr164dW7JkCZd2dfHiRQaAffHFF5XW4f79+wwAmzp1apV1VQ1qcHR0ZFZWViwkJKTSQQ3l34OmdJ7y5HI5mz59OgsKCmJ2dnbMxsaGBQUFVcijzM3NZcOGDWOOjo4VBjUUFRWxRYsWsVatWjGRSMScnJxYcHAwW7BgAZNKpVw5lBnU0KRJEyYSiVi7du1YXFxclddAJTMzk40cOZK5uLgwoVDI2rRpU+G9lR3UUB4qSbkqq7rfB5XKBjUsWLCA+fv7M0tLS+bt7a1xUINCoWAzZ87kBimEh4ezlJSUSlO/zp8/r7GOZa+bRCJhERERzM7OTm1QQ2FhIYuOjmYNGzZkYrGYdenShcsTLpsqVtWghvLmzZvHahKifvvtNxYWFsacnZ2ZhYUFa9iwIXvvvffURnRqW09tfz41QXMjELPD4/EwceJE/PDDD4auCiEc6rMlhBA9oGBLCCF6QMGWEEL0gLIRiNmh2xDEGFHLlhBC9ICCLSGE6AF1I9QDpVKJhw8fws7Ork5j2Akh9YMxhpycHHh6eupk0I42KNjWg4cPHxpsYhRCiPbS09Ph5eWll3NRsK0HqiGd6enp3BRwhBDjIZPJ4O3tXa/Dr8ujYFsPVF0H9vb2FGwJMWL67OajG2SEEKIHFGwJIUQPKNgSQogeULAlhBA9oGBLCCF6QMGWEGIWcgqLoVQa77wYFGwJISavsFiBERvPY8KWiygoqv0aa/WJ8mwJISZNqWSY/ttVXHyQBTsrCzySFiDA1dbQ1aqAWraE6NDRm5mI2Z+MwmLjbF2Zo+/+vo0/rzyEBZ+H9R8EG2WgBahlS4hOzftfEtKfF6CBrRDj3mhs6OqYvV0X0rHqaAoA4Jt326BzYO1XDq5v1LIlREfy5CVIf14AANhw4p7R9h2aizMpTzH792sAgEk9AzG0g3FP/kTBlhAdSXmcy/3/aW4Rtpx7YMDamLeUxzkY/+tFlCgZBgR5YlrfpoauUrUo2BKiI7czcwAAQovSj9W643epdVsPnubKMXLTecgKSxDs64Qlg18Dn2/880abTbC9f/8+Ro8eDX9/f4jFYjRu3Bjz5s1DUVGRWrmrV6+iW7dusLKygre3NxYvXlzhWLt27ULz5s1hZWWFNm3aYP/+/fp6G8SEqVq2g4O94OUk1lvrNv15PjadTkXq07x6P5ehFRYrMPa/F5D+vAC+Dayx4cNgWFkKDF0trZhNsL158yaUSiXWr1+PpKQkfPfdd1i3bh3+/e9/c2VkMhnCwsLg6+uLixcvYsmSJZg/fz42bNjAlTlz5gzef/99jB49GpcvX0ZkZCQiIyNx/fp1Q7wtYkJULdsWDe0xqWcgAGD9iXv1mpmQ+jQPg9aewfw/b6Dn0mN4d81pbDn3ANKC4no7p6EolQzRO6/gclo2HMSW+HlERzSwFRm6WlrjMTNeinTJkiVYu3Yt7t27BwBYu3Yt5syZA4lEAqFQCACYNWsW9u7di5s3bwIA3nvvPeTl5WHfvn3ccV5//XW0bdsW69at0+q8MpkMDg4OkEqlNJ/tK6TLt0eRkV2AHeNeRzsfJ/RcegwZ2QX44q2WGN3VX+fnS3+ej6Hr4/FIWogGNkJk5RdBNYBKaMFH35buGNS+Ed5o4goLgem3q+JuPsbITedhKeDhl9Gd8HpAg1ofyxCfUbNO/ZJKpXB2duaex8fH44033uACLQCEh4dj0aJFyMrKgpOTE+Lj4zFt2jS144SHh2Pv3r2Vnkcul0Mul3PPZTKZ7t4EMQl58hJkZJdmIjR1t4PQgo+JPQPx7z3XsO74XQzv5KPTP3cfZhdg2E9n8UhaiMauNtjxr1AolQx7EzOw+2IGbmXm4K+rj/DX1UdwsRXC01Fc6bH6tHDHp72b6Kxu9SXteT6A0vrWJdAaitkG25SUFKxatQpLly7ltkkkEvj7q7cw3N3duX1OTk6QSCTctrJlJBJJpeeKiYnBggULdFh7YmpU/bUutkI42ZR+mQ8O9sLquBRkZBdgW0IaRnbRTev2sawQw386h/TnBfBrYI2tY1+Hy4s/p8e90RhjuwUg6aEMuy/9gz8SH+JpbhGe5hZVerxrGVKMeyPA6Ps+ZS+6RhytLQ1ck9ox+mA7a9YsLFq0qMoyycnJaN68Ofc8IyMD/fr1w5AhQzB27Nj6riJmz56t1hpWrW9EXh13XgTbJm4v17QSWvDxcc/GmLPnOtYeu4v3Q+reun2aK8ewn84h9WkevJzE2Dr2dbjbW6mV4fF4aN3IAa0bOeDfb7bAxQdZyC8q0Xi8T7ZeRl6RAhnZBWhspCOvVGSFpcHW3oqCbb2Ijo7GiBEjqiwTEBDA/f/hw4fo2bMnOnfurHbjCwA8PDyQmZmptk313MPDo8oyqv2aiEQiiESm01FPdO/Oi5tjTd3VA9aQYG+sPpqCh9JCbE9Iw4g6tG6z84vwwU/nkPI4Fw0drLBt7OtVdg8AgKWAX+Wf3N7O1rgpyUH683zjD7YFpV8Y9mLTDLZG32vu6uqK5s2bV/lQ9cFmZGSgR48eCA4OxsaNGyusBx8aGooTJ06guPjlndrY2Fg0a9YMTk5OXJkjR46ovS42NhahoaH1/E6JKVNlIgS6q6/WWtq6Lc1MWHv8bq0zE2SFxfjwPwm4KcmBq50IW8Z0grezdd0qDcDLqfQY6VkFdT5WfXvZsjX6NqJGRh9staUKtD4+Pli6dCmePHkCiUSi1tc6bNgwCIVCjB49GklJSdixYwdWrlyp1gUwefJkHDx4EMuWLcPNmzcxf/58XLhwAZMmTTLE2yImQtWN0NStYutwSAcvNHSwQqZMjp0X0mt8bKWSYczmC7iWIYWzjRBbx3TS2WQrPi8CdvqLm0/aepYrx6ojd/A4p1An9dAGF2ypZWtYsbGxSElJwZEjR+Dl5YWGDRtyDxUHBwccPnwYqampCA4ORnR0NObOnYtx48ZxZTp37oytW7diw4YNCAoKwm+//Ya9e/eidevWhnhbxATkyUvwT9bLTITyRBYCfNyjdFKaNXF3IS+pWet2z+UMJKQ+h41QgF9Hd0ITDeeoLW/n0m6Imgbbn0+nYlnsbXy67TL0lT3KdSNQn61hjRgxotq+XQB47bXXcPLkySrLDBkyBEOGDNFRzYgx+iMxAz+fSkWLhvbo6OeMEH9neDmJwePVfNinpkyE8oZ29MbquLuQyArx69k0rfNu84tKsOTQLQDApF5N0NJTtzmh3lw3Qs2C7Z3M0vd89t5zHErKRL/Wld/T0JWXLVvTDFumWWtC6uBprhxz9lxHrrwEV/6RYvv50j/tPeyt0MHPCSH+pcG3mbudVsFXUyZCeSILAT7t3QT/3nMNyw7fQlhLd636XDecuAeJrBBeTmKM7OKn3RusAW+uG6Fmfbb3n70cGvzN/mT0bO4KkUX9po6pUr9MtWVrNt0IhGjru9jbyJWXoLmHHf71RgDa+TjCgs+DRFaIfVcfYe4fSei34iSWx97W6niqTIQm7lX3o/5fR2+E+Dsjv0iBGb9drXa9LIm0EOuPl45+nNW/eb3kwaq6EaQFxVoP8VUqGR48K20J24oskPY8HxtP39f6nA+zC3BLklOjejLGkFNI2QiEmIxbkhxsS0gDACx4uxVmv9kCez7ugmvzw7Ft7OuY1rcpOvqVZqbsTczQ6phcy7aavlQ+n4fFg16DlSUf8feeYeuLelRm6eFbKChWINjXCRFtGlZZtrashRZwsS3t+tC231YiK4S8RAkLPg9z32oJAPjhaAqe5MireWXp9Q//7gQG/HAKWXmVD7Qor6BYgZIXX07UsiXEBCzcnwwlA/q18kCnMvmnYqEAoY0b4NPeTbBxZAgs+DykPy9A2rPqA5Aq7UtTJkJ5fi42mBFeOgAnZn8y/qmkr/R6hhS7L/0DAPg8okWt+pK1pUr/qqwu5d1/MbuYj7M1Bgd7IcjLAbnyEiw7fKvK1z3OKcSoTeeRIy9BUYmSG96sDdXNMUsBD1aWphm2TLPWhNRC3K3HOHH7CSwFPMx+s3ml5WxFFmjvU9q6PZnypMpjls1E0DZLYERnP3T0c0JekQKzdl+rcDefMYav9t0AY8DAtp5o96Iu9UXVb5umZcs29UV/rW8Da/D5PMwdUNq63XEhHdczpBpfU1CkwJjNF9QCbE1mJlPdHLOzsqzXL576RMGWvBJKFEos/CsZQGmw821gU2X5rk1K17I6nfK0ynJ3n7zMRHCuJBOhPD6fh8WDgyCy4ONUylPuBp3KoaRMnEt9DpEFHzP6Vf6loCs+XPqXdi1NVX+tn0vpNQz2dcaAIE8whhdfEupfHgolw5Qdl3H1HymcrC3h16A0uNco2BaY9oAGgIIteUVsO5+OlMe5cLK2xKRe1c9w1SVQFWyfQVHFjazbmdVnImji72KD6eHNAAAL/0rmWnxFJUrEHCj9UhjbLQCNqhmOqws1Tf9STVLu7/LyC2tW/+YQWfBxLvU5Dl5Xn7Tp2wPJOJSUCaGAjw0fdUDgi2uVnV/zlq2p3hwDKNiSV4C0oBjfvcgsmNq3KRy0+MAGeTnATmQBaUFxpX8aA9pnImgysos/2vs4Ildeglm7r4Ixhv/G38eDZ/lwtRNhfA/9rM7rXcNRZKo+27J/HTRyFONfb5TOUfLNgZdLuf9y9gF+PJkKAFgy5DV09HPmZu2qWcvWtAc0ABRsyStgTVwKnucVobGrDd4P8dHqNRYCPl5vXHoD7VQVXQnaZiJoIuDzsGRIEIQWfJy88xTrT9zDyiN3AACfhTWFrUg/fzJ7l5kfobp0NKWS4cGLoOxfritmfI/GcLcXIf15AX4+nYpjtx5j/v+SAADRfZtiYNtGAMB92WUXaJ+NYOoDGgAKtsTEpT3Lx/MqUojSnr3MAf08oiUsa7BiQbcX/ban7lQebFWZCE20yETQpLGrLT4LK10Z9tsDN5FTWJr/OzhYf1N0NnS0goDPQ1GJEk9yq07feiQrRFGJEpYCHjwd1ad2tBZaYOaLPubVR1MwaetlKJQMg9p7YVKvQK6c44tgK6tVny21bAnRu/i7z9BjaRw6fB2L/9sQj42nUyukE317MBlFCiW6NXFBj2auNTp+1xf9thcfZGlcJbe6ORG0NbprANp6O3LPv3irJQR6XC3WUsBHQ4fSwFldV4KqC8HbyVrjUjuRbRshyNsReUUK5MpL8HqAM2LebaOWQeDwohuhZn22pj2gAaBgS0yUUlmaHqVkgJKVjtFf8OcNdPn2KAasOoXVcSnYezkD+69JwOcBc2qRq+rvYgNPBysUKZRIuP+8wv7aZCJoIuDzsHRIENztRRgc7MXdnNMnbW+SqYbp+rlozubg83mYN6AlLAU8BLrZYv0HHbil3VW4boSaBFszyEYw3ZqTV9rvlzNw45EMdlYW2Db2dZxLfY5D1yU4/+A5rmVIca3MTa33OvqguUfNJ3Dh8Xjo2sQFOy/8g1N3nqB7U/WWsSoTIbCWXQhlBbrZ4ty/+9T5OLXl7SxG/D0g7VnV6V+qlq1fFalz7X2ccHx6TzhaW8JaWDHEOFqXfjHVJs/WlFu2FGyJySkoUmCpaiasnoHcEjCju/rjSY4cfydn4lCSBKdTnsJBLMS0vk1rfa4ugS+CbcqzCvvuPFatzqC7KQ8NhZvXtpqWbepTVY5t1ZPoVLWChKplW5Ngy82LYMJ9thRsicn5z6nSmbAaOYoR1dlPbZ+rnQjvh/jg/RAf5MlLwONBY+tKW6o/6ZMfyfAkRw5Xu5fLH6mmGdTl/LKGom3614Nn1bdsq+NYi2DLdSNQNgIh+vE4pxBrj90FAMysZiYsG5FFnQItALjYitCyYWkXxJm76lkJdc1EMCYv50eovBtBLe2rkj5bbahatrnyEhQrlFq9RnWDzM6EW7YUbIlJWfH3HeQVKRDk7YgBr9XPTFjlddWQApZfpJtMBGOhmmrxobQARSWaA6Bqn6WAx2Uv1EbZfldt078o9YsQPbqTmYPtL6YlnPNm/c6EVZYqBexUylNu3H/Z1RnqkolgLFxtRbCy5IOx0vlmNVHNieDtrDntS1sCPg92L7IKsrUItowxGtRAiD5982J6xPBW7gjxd9bbeTv6OUMo4OORtBD3XtyN12UmgjHg8XjVpn9xcyLUob9WpSZDdguLlShWmPZctgAFW2IiTt15irhbT2DB52FW/xZ6PbdYKECHFxOKq7oSzCkTQaW6qRY1zYlQW47iF+lfWuTaqlq1Aj4P1sL6XXqnPlGwJUZPoWRYuL90JqwPXvet082Z2uL6bV/Mk8BlIphJyxYAvJ2qnmrx/jPVzbHq106rTk3mRyg7oMFU57IFKNgSE/D7pX+Q/GIAw6e9q58esT6o+m3P3n2GEoWSa9maQ9qXinc1ubb3n+muZasasluTlq0pD2gAKNgSI1dQpMDSF8utfNIr0GA3o1p5OsDR2hI58hKcvfeca/2ZYzfCPxq6ERRKxi0RpIu/LF62bLUItmYwvSJAwZYYsez8Iny6/TIyZXJ4OYnxUaifweoi4PPQ+cWUi5vOlM7P2sDGPDIRVFQ3yDT12T6SFqBIoZrtq+4TmtdkYIM5ZCIAFGyJkTpz9yn6rTiJ2BuZsODzsODtVvWylHdNdA0snRvhyM3HAGo3YbgxU+XaZuUXI1deorbv/tOXaV+6mJGMG7KrTTeCGeTYAjRclxiZohIllsfexvoTd8EYEOBig5X/1w5tvBwMXTVuflvVElvm1IUAlI7OcrS2RHZ+MdKf56NFw5eT96j6a3WR9gXULPVLZgbzIgAUbIkRufckF5O3J3Izdr0f4o0v3mpZ5yG3uuLtbA0fZ2vuz2xzykRQ8XG2Rna+tGKwfVr11Io1VaM+W25lXeP4Pagt6kYgBscYw/aENER8fwrXMqRwEFti3QftEfPua0YTaFVUKWCAeWUiqFTWb8vNY9ug7mlfAODwIs82O1+b1C/TnzgcMNNgK5fL0bZtW/B4PCQmJqrtu3r1Krp16wYrKyt4e3tj8eLFFV6/a9cuNG/eHFZWVmjTpg3279+vp5q/Op7lynHwugRf7buBt1adwqzfr6GgWIHOjRvg4JRu6NdaP/Me1FTXMhN7m1s3AgB4vei3LT8hzf1yy5fX1ctuhJJqSpa5QWbiLVvTrn0lZsyYAU9PT1y5ckVtu0wmQ1hYGPr06YN169bh2rVrGDVqFBwdHTFu3DgAwJkzZ/D+++8jJiYGb731FrZu3YrIyEhcunQJrVu3NsTb0RvGGC6nZ8PKQoCWnjWfbLsqmbJCnE55ivP3nyMh9TnuPslT228p4CE6rBnGdQsAX49LwtRU1yYuaOhgBQ8HK7PKRFDhhuyWadmWTfuqy9SKZb2c07YIjLEqByu8nF7RtFu2ZhdsDxw4gMOHD2P37t04cOCA2r4tW7agqKgIP//8M4RCIVq1aoXExEQsX76cC7YrV65Ev379MH36dADAV199hdjYWPzwww9Yt26dxnPK5XLI5S8XypPJZPX07upHflEJ9lzOwH/PPMCtzBwIBXzETntDJ8nrBUUKrDmWgvXH76Go3HR6Td1t0cHPGSF+zght3ADu9rWfSUpf7K0sEfdZD72uEaZPmiYRf5hdmvYlFPB1kvYFvGzZFisYCooVVXYX0Q0yI5SZmYmxY8di7969sLau2LcUHx+PN954A0LhyxZJeHg4Fi1ahKysLDg5OSE+Ph7Tpk1Te114eDj27t1b6XljYmKwYMECnb0Pfbn/NA+/nH2AnRfSuZnwAaBIocQPR1OwZEhQnY5/JDkT8/6XxP1J2rqRPTo3dkFHP2d08HWCk4m2DA2dglafXk4iXsC1OF/O9iXW2ZeM2FIASwEPxQqG7PziKoNtDrVsjQtjDCNGjMD48ePRoUMH3L9/v0IZiUQCf39/tW3u7u7cPicnJ0gkEm5b2TISiaTSc8+ePVstQMtkMnh7628p6po6decp/nPqHo7dfsKlMfk2sMaHr/uiqbsdPvo5Ab9fzsAnvZrApxY3RP7JyseCP28g9kYmAMDTwQpzB7RCeCt3kx7b/irwdLQCjwcUFCvwNLcIrnYipKrSvnQ4JwWPx4ODWIinuXJIC4qrbDGby6AGo6/9rFmzsGjRoirLJCcn4/Dhw8jJycHs2bP1VLOXRCIRRCJR9QWNwOW0LHzwn3Pc8x7NXBEV6ofuTV25vtLuTV1x/PYT/BB3B4sHa9+6LSpR4seT97Dq6B0UFithwedhTLcAfNo70OiyCohmIgsBPOyt8EhaiPSsfLjaiXQ621dZDmILPM2VV7nKLmPMbIbrGv0nIDo6GiNGjKiyTEBAAI4ePYr4+PgKQa9Dhw4YPnw4Nm/eDA8PD2RmZqrtVz338PDg/tVURrXf1N14VNqf3KKhPdYMb6+xtTK5TxMcv/0Ev18qbd2q/rSsyoNneRi16Tx346uTvzO+jmxtlulR5s7b2bo02D7PR3sfp5frjul4trXSVXbzIK1i5i95iZLr66duhHrm6uoKV1fXast9//33+Prrr7nnDx8+RHh4OHbs2IFOnToBAEJDQzFnzhwUFxfD0rL0BxcbG4tmzZrBycmJK3PkyBFMmTKFO1ZsbCxCQ0N1+K4MRzWBSoifU6V/Frb3ccIbTV1x4vYTrI5LwbeDXqvymPISBT7ecgl3n+TBxVaIOREtENm2EXUZmChvJ2skpD7nMhJ0OWl4WdqssqvKRODzABsTnssWMKM8Wx8fH7Ru3Zp7NG1aunx148aN4eXlBQAYNmwYhEIhRo8ejaSkJOzYsQMrV65U62+dPHkyDh48iGXLluHmzZuYP38+Lly4gEmTJhnkfema6i5zda3VyS+mMvzt4j/Vrri6+OAtJD2UwcnaEvs+6YZ32nlRoDVhqjkS0p8XQKFk3Be0r44GNKioJqOpqhuh7PSKpv47ZTbBVhsODg44fPgwUlNTERwcjOjoaMydO5dL+wKAzp07Y+vWrdiwYQOCgoLw22+/Ye/evWaTY6vKDPByqjqFJ9jXCd2auKBEybDmWEql5eJuPsZ/TpXOgrV0SBA86rAQIDEOZZfHqY+0LxUHLeZHMJe0L8AEuhFqy8/Pj1ucr6zXXnsNJ0+erPK1Q4YMwZAhQ+qragalmqtUtXR1VSb3boKTd55i14V/MLFnYIXXPJYV4rNdpQNHRnT2Q+8W7poOQ0xM2eVxVMN0fRroZravsrSZH0HVjWDq8yIAr1jL9lWXJy/Bs7zSmxHa3PTq4OeMroGq1u1dtX1KJcPUnYl4lleEFg3tMat/83qpM9E/1cCGR9JCbhVhXc2JUJY2c9qaU8uWgu0rRNWFYG9lwbUqqjO5T2nf7a4L6cgos8T1+hP3cDrlGcSWAqx6v51ZJ/q/atzsRBBa8KFQMsTffQZAd8N0y9JmaZyXQ3WpZUtMiOpGlzatWpWOfs7o3LgBihUMa+JK+24vp2Vh2Yulaua/3dJslvMmpfh8Hrxe9M9ywbYeFtnkVtitsmVrHhOHAxRsXyn/ZKn6a2t2o0OVmbDzQjpuZ+bg0+2XUaJkiHitIYZ2MN6RcqT2vF58Iee8WLGhPlq29lqssGsu0ysCFGxfKekvuhG8tbg5VlangAYIDSht3Q5acwbpzwvQyFGMb95pY/LpOEQzH2f1L2Q/HSxfXp5qMhqtUr+oZUtMSW26EVRUfbc58hII+Dx8/35brft9iekp+4UsFPDR0EG3aV/Ay2yEnMISKJQVM4cA6rMlJopr2TrX/IPzekADdAksXV12ap8mCPZ11mndiHEp+4VcH2lfANS+rGWV9NuaUzaC6X9dEK0wxrgc25p2I6isGRaMpEdShAY00GXViBEq+ztSH/21AGAp4MNWZIFceQmkBcUap9w0l4nDAWrZvjJkBSXczY5GNbxBpuJgbYnOjV2on/YV4ONcNtjqvr9WpbqBDeayJA5AwfaVoZoTwcVWSNMdkmo5WFtyo7bqI+2LO081AxsoG4GYnPQaDNMlBABaeJSuQ9dKx+vRlcW1bCtZZTfHTJYxB6jP9pWh7WxfhKis+L+2uJ2Zg3Y+TvV2DscqJqMpLFZAXmIec9kCFGxfGapp8mo6oIG8ujwdxTqf6as8rhtBQ66tal08Hg+wNYOuL+pGeEWoRo/VNhOBkPqgmh9B0w0y1c0xO5GFUS9vry0Ktq+IuuTYElJfqrpBZk5pXwAF21cCY4xatsQoqSaj0TRk15wGNAAUbF8JT3LlKCxWgsdDvffBEVITqhtkmkaQmdNQXYCC7StBdXPMw94KQgv6kRPj4VDFzF/mNAkNQMH2lUBdCMRYVd1naz4DGgAKtq8EbpFHujlGjIxDFSvsUsuWmJz0Ok5AQ0h9UfXZykuUKCxWqO2jPlticmj0GDFWtiILbvrG8l0JqmwEO2rZElNBo8eIseLxeJV2JeSY0YxfgI6CrUwmw969e5GcnKyLwxEdUigZHmarBjRQy5YYn8puktGgBgBDhw7FDz/8AAAoKChAhw4dMHToULz22mvYvXu3TitI6kYiK0SJksFSwIOHvZWhq0NIBZXN/EWDGgCcOHEC3bp1AwDs2bMHjDFkZ2fj+++/x9dff63TCpK6Ud0c83QU18vSJoTUVWUzf9ENMgBSqRTOzqVrUB08eBCDBg2CtbU1IiIicOfOHZ1WkNQNZSIQY1dpNwKlfgHe3t6Ij49HXl4eDh48iLCwMABAVlYWrKwM+6fqX3/9hU6dOkEsFsPJyQmRkZFq+9PS0hAREQFra2u4ublh+vTpKCkpUStz7NgxtG/fHiKRCIGBgdi0aZP+3oCOqSagoZtjxFg5arhBJi9RoLDYfOayBWo5n+2UKVMwfPhw2NrawtfXFz169ABQ2r3Qpk0bXdavRnbv3o2xY8fim2++Qa9evVBSUoLr169z+xUKBSIiIuDh4YEzZ87g0aNH+Oijj2BpaYlvvvkGAJCamoqIiAiMHz8eW7ZswZEjRzBmzBg0bNgQ4eHhhnprtfYPpX0RI6epZVt2Lls7kXl0I9TqXXz88ccICQlBeno6+vbtCz6/tIEcEBBgsD7bkpISTJ48GUuWLMHo0aO57S1btuT+f/jwYdy4cQN///033N3d0bZtW3z11VeYOXMm5s+fD6FQiHXr1sHf3x/Lli0DALRo0QKnTp3Cd999Z5rBltK+iJFzsH4x81eZYKvqr7U1k7lsgTqkfnXo0AHvvPMObG1tuW0RERHo0qWLTipWU5cuXUJGRgb4fD7atWuHhg0bon///mot2/j4eLRp0wbu7u7ctvDwcMhkMiQlJXFl+vTpo3bs8PBwxMfHV3puuVwOmUym9jAWNKCBGDtNLVtzy0QAatCynTZtmtYHXb58ea0qUxf37t0DAMyfPx/Lly+Hn58fli1bhh49euD27dtwdnaGRCJRC7QAuOcSiYT7V1MZmUyGgoICiMUVW4gxMTFYsGBBfbytOpGXKCCRFQKgG2TEeDlyS+O8TP0ytxxboAbB9vLly2rPL126hJKSEjRr1gwAcPv2bQgEAgQHB+u0grNmzcKiRYuqLJOcnAylsrQzfc6cORg0aBAAYOPGjfDy8sKuXbvwr3/9S6f1Kmv27NlqX0YymQze3t71dj5tPcwuBGOAlSUfLrZCQ1eHEI0cNKR+ycxs9BhQg2AbFxfH/X/58uWws7PD5s2b4eRUuvJmVlYWRo4cyeXf6kp0dDRGjBhRZZmAgAA8evQIgHofrUgkQkBAANLS0gAAHh4eSEhIUHttZmYmt0/1r2pb2TL29vYaW7Wq84hEIu3flJ6obo55OVmDxzOPfi9ifrhsBLU+W/OaFwGo5Q2yZcuW4fDhw1ygBQAnJyd8/fXXCAsLQ3R0tM4q6OrqCldX12rLBQcHQyQS4datW+jatSsAoLi4GPfv34evry8AIDQ0FAsXLsTjx4/h5uYGAIiNjYW9vT0XpENDQ7F//361Y8fGxiI0NFRn70lfVHMieNPNMWLEHMqs1qBUMvD5vJfzIpjJgAagljfIZDIZnjx5UmH7kydPkJOTU+dK1Ya9vT3Gjx+PefPm4fDhw7h16xYmTJgAABgyZAgAICwsDC1btsSHH36IK1eu4NChQ/j8888xceJErmU6fvx43Lt3DzNmzMDNmzexZs0a7Ny5E1OnTjXI+6oLujlGTIHqBpmSATny0hatuQ1oAGrZsn3nnXcwcuRILFu2DCEhIQCAc+fOYfr06Xj33Xd1WsGaWLJkCSwsLPDhhx+ioKAAnTp1wtGjR7kWuEAgwL59+zBhwgSEhobCxsYGUVFR+PLLL7lj+Pv746+//sLUqVOxcuVKeHl54aeffjLJtC8aPUZMgchCALGlAAXFCkjzi+EgtjS7VRqAWgbbdevW4bPPPsOwYcNQXFz6DWRhYYHRo0djyZIlOq1gTVhaWmLp0qVYunRppWV8fX0rdBOU16NHjwo3BE0RjR4jpsJBbFkabF/0277SN8hUFAoFLly4gIULF2LJkiW4e/cuAKBx48awsbHReQVJ7WVQNwIxEY7WlpDICrmFH1/p1C8VgUCAsLAwJCcnw9/fH6+99lp91IvUUX5RCZ7mlv7iUjcCMXb25QY2mOOghlrdIGvdujU3iIAYJ9Uij3ZWFtzdXkKMVfnJaMxtekWglsH266+/xmeffYZ9+/bh0aNHRjtU1Zz9dPIeRm86j+d5RRr3q26OeVGrlpiA8kN2KRvhhTfffBMA8Pbbb6slyzPGwOPxoFAoKnsp0ZEfT95DpkyOydsvY9PIkAoTg7/MRKCbY8T4lZ9AXJWN4PAq99kC6qPJiP4xxpCVV/pLefLOU3x/5A6m9m2qVkbVjUA3x4gpcFTN/JVfhKISJQpeLGv+yrdsu3fvrut6kBooKFagSKHknn9/9A7a+TiiRzM3bhs3oIFatsQElL1Bpho9BgC2r3LqV1n5+flIS0tDUZF6vyFlKNSvrBc3EYQCPgZ38MLWc2mYsiMR+z7pyvXRckN1qWVLTEDZG2SqTARbkYVZrZtXq2D75MkTjBw5EgcOHNC4n/ps61fWi5tijtaWmDegJa5nSHH1Hyk+3nIJu8aHQmQh4Fq2dIOMmAIHDS1bcxrQANQyG2HKlCnIzs7GuXPnIBaLcfDgQWzevBlNmjTB//73P13XkZSjuongaG0JkYUAq4e1h6O1Ja7+I8VX+25Aml/MLStCo8eIKSh7g8wch+oCtWzZHj16FH/88Qc6dOgAPp8PX19f9O3bF/b29oiJiUFERISu60nKyMpXtWxLbyp4O1tjxXttMXLTefx6Ng1WFgIAQAMbIWzMZP0mYt7KtmzNMe0LqGXLNi8vj5ui0MnJiZsBrE2bNrh06ZLuakc0UvXZOpb55u/RzA2f9GoCAPjpVCoAwIv6a4mJcBSXNhzyixR4misHYF4DGoBaBttmzZrh1q1bAICgoCCsX78eGRkZWLduHRo2bKjTCpKKVMuHOFmrr74wuXcTdGviwj2nTARiKuysLKBK2VfliFPLFsDkyZO5lRHmzZuHAwcOwMfHB99//z23JDipP1zL1kb9l1HA52Hl/7WDp4MVAMC3AbVsiWng83lccFVl0lCfLYAPPviA+39wcDAePHiAmzdvwsfHBy4uLlW8kugC12crrriumLONEJtHheDXsw/wweu++q4aIbXmaG0JaUEx0riWrXl1I9Tq3dy7dw8BAQHcc2tra7Rv315nlSJVk75o2TpVMsFME3c7LBjYWp9VIqTOVDfJVGmL1LIFEBgYCC8vL3Tv3h09evRA9+7dERgYqOu6kUqUz0YgxByogm2OGU6vCNSyzzY9PR0xMTEQi8VYvHgxmjZtCi8vLwwfPhw//fSTrutIyskuk2dLiLkoP+mMnZl1I9Qq2DZq1AjDhw/Hhg0bcOvWLdy6dQt9+vTBzp078a9//UvXdSTlZHPdCNSyJeajfOOBuhFQOifCqVOncOzYMRw7dgyXL19G8+bNMWnSJPTo0UPHVSRlKZUM2Vzql3n9MpJXW/mWrbl1I9Qq2Do6OsLJyQnDhw/HrFmz0K1bN24FW1K/cuQlULLS/9MKDMSclM+uMbdBDbWePPzUqVPYvn07JBIJJBIJevTogaZNm1b/YlInqlattVAA0YthuYSYg/KNB3Nr2daqz3bv3r14+vQpDh48iNDQUBw+fBjdunXj+nJJ/dE0VJcQc2DuN8jq9G7atGmDkpISFBUVobCwEIcOHcKOHTuwZcsWXdWPlJNNaV/ETJVtQNgIBbAQ1KotaLRq9W6WL1+Ot99+Gw0aNECnTp2wbds2NG3aFLt37+YmpSH1g8tEsKGWLTEvZbsRzC0TAahly3bbtm3o3r07xo0bh27dusHBwUHX9SKVqGqoLiGmrOzvtLn11wK1DLbnz5/XdT2IllQtWxrQQMxN2T5bc8tEAGrZjQAAJ0+exAcffIDQ0FBkZGQAAH755RecOnVKZ5UjFWVXMr0iIabOypIPoUVpSDLHlm2tgu3u3bsRHh4OsViMy5cvQy4vnexXKpUadIrF27dvY+DAgXBxcYG9vT26du1aYdn1tLQ0REREwNraGm5ubpg+fTpKSkrUyhw7dgzt27eHSCRCYGAgNm3apMd3UbUsatkSM8Xj8bjWrTn22dYq2H799ddYt24dfvzxR1havrwoXbp0MehKDW+99RZKSkpw9OhRXLx4EUFBQXjrrbcgkUgAlC5EGRERgaKiIpw5cwabN2/Gpk2bMHfuXO4YqampiIiIQM+ePZGYmIgpU6ZgzJgxOHTokKHelpqX8yJQy5aYH1VGgrmlfQEAWC2IxWKWmprKGGPM1taW3b17lzHG2N27d5lIJKrNIevsyZMnDAA7ceIEt00mkzEALDY2ljHG2P79+xmfz2cSiYQrs3btWmZvb8/kcjljjLEZM2awVq1aqR37vffeY+Hh4VrXRSqVMgBMKpXW5S1pNGDVSeY7cx/7+4ak+sKEmJjBa08z35n72JKDN+v1PPX5Ga1MrVq2Hh4eSElJqbD91KlTavPc6lODBg3QrFkz/Pe//0VeXh5KSkqwfv16uLm5ITg4GAAQHx+PNm3awN3dnXtdeHg4ZDIZkpKSuDJ9+vRRO3Z4eDji4+MrPbdcLodMJlN71Be6QUbM2ctuBPNr2dYq2I4dOxaTJ0/GuXPnwOPx8PDhQ2zZsgXR0dGYMGGCruuoFR6Ph7///huXL1+GnZ0drKyssHz5chw8eJCbt0EikagFWgDcc1VXQ2VlZDIZCgoKNJ47JiYGDg4O3MPb21vXb49Dc9kScxbWygMe9lbo3Nj8VnypVbCdNWsWhg0bht69eyM3NxdvvPEGxowZgwkTJmDMmDE6reCsWbPA4/GqfNy8eROMMUycOBFubm44efIkEhISEBkZiQEDBnDrpdWX2bNnQyqVco/09PR6OU+JQslNrEzZCMQcDe3gjbP/7o3Wjcwvd79WbXUej4c5c+Zg+vTpSElJQW5uLlq2bIn169fD39+fayXqQnR0NEaMGFFlmYCAABw9ehT79u1DVlYW7O3tAQBr1qxBbGwsNm/ejFmzZsHDwwMJCQlqr83MzARQ2jWi+le1rWwZe3t7iMWaV6sViUQQiUS1eXs1In1xcwwwv/WZCDF3NfrEyuVyzJ8/H7GxsRCJRJg+fToiIyOxceNGvPPOOxAIBJg6dapOK+jq6gpXV9dqy+Xnl65bxOerN9b5fD6USiUAIDQ0FAsXLsTjx4/h5uYGAIiNjYW9vT1atmzJldm/f7/aMWJjYxEaGlrn91JXqrQveysLsxs3TojZq8ndtBkzZjAHBwc2aNAg1rBhQ2ZhYcHGjh3L2rRpw7Zt28ZKSkrq6T5e9Z48ecIaNGjA3n33XZaYmMhu3brFPvvsM2ZpackSExMZY4yVlJSw1q1bs7CwMJaYmMgOHjzIXF1d2ezZs7nj3Lt3j1lbW7Pp06ez5ORktnr1aiYQCNjBgwe1rkt93ek8n/qM+c7cx7otOqrT4xLyqjFENkKNgq2/vz/7448/GGOMXbt2jfF4PDZy5EimVCrrpXI1df78eRYWFsacnZ2ZnZ0de/3119n+/fvVyty/f5/179+ficVi5uLiwqKjo1lxcbFambi4ONa2bVsmFApZQEAA27hxY43qUV8/yNgkCfOduY+9veqkTo9LyKvGEMGWxxhj2raChUIhUlNT0ahRIwCAWCxGQkIC2rRpUy+tblMlk8ng4OAAqVTK9R/rwq4L6Zj+21V0b+qKzaNCdHZcQl419fUZrUqNOv4UCgWEwpd3wS0sLGBra6vzShHNKMeWENNVoxtkjDGMGDGCu/NeWFiI8ePHw8bGRq3c77//rrsaEk52AU1CQ4ipqlGwjYqKUnv+wQcf6LQypGo0CQ0hpqtGwXbjxo31VQ+iBW5JHDOcEYkQc0fJmibk5ZI41I1AiKmhYGtCXnYjULAlxNRQsDUhUupGIMRkUbA1IaqWLWUjEGJ6KNiaiMJiBQqKFQAAR1rGnBCTQ8HWRKhm/BLwebAT0YxfhJgaCrYmIqtMfy2PxzNwbQghNUXB1kRk5ZW2bB1oQAMhJomCrYmQ0lBdQkwaBVsT8TITgVq2hJgiCrYmQtVn6yCmli0hpoiCrYmQUsuWEJNGwdZEqFq2NC8CIaaJgq2JUE1C40BDdQkxSRRsTUQ2DdUlxKRRsDURXDcC9dkSYpIo2JqI7AIa1ECIKaNgawIYY9wqDdSNQIhpomBrAvKKFChWlK44T8GWENNEwdYEqFq1Qgs+rCzpR0aIKaJPrgnILjOggWb8IsQ0UbA1AS+nV6QuBEJMFQVbE5DNLfRImQiEmCqTCbYLFy5E586dYW1tDUdHR41l0tLSEBERAWtra7i5uWH69OkoKSlRK3Ps2DG0b98eIpEIgYGB2LRpU4XjrF69Gn5+frCyskKnTp2QkJBQD+9Ie5SJQIjpM5lgW1RUhCFDhmDChAka9ysUCkRERKCoqAhnzpzB5s2bsWnTJsydO5crk5qaioiICPTs2ROJiYmYMmUKxowZg0OHDnFlduzYgWnTpmHevHm4dOkSgoKCEB4ejsePH9f7e6xMFrVsCTF9zMRs3LiROTg4VNi+f/9+xufzmUQi4batXbuW2dvbM7lczhhjbMaMGaxVq1Zqr3vvvfdYeHg49zwkJIRNnDiRe65QKJinpyeLiYnRuo5SqZQBYFKpVOvXVGXB/5KY78x9LGZ/sk6OR8irTtefUW2YTMu2OvHx8WjTpg3c3d25beHh4ZDJZEhKSuLK9OnTR+114eHhiI+PB1Daer548aJaGT6fjz59+nBlNJHL5ZDJZGoPXcqmobqEmDyzCbYSiUQt0ALgnkskkirLyGQyFBQU4OnTp1AoFBrLqI6hSUxMDBwcHLiHt7e3Lt4SRzVUl7oRCDFdBg22s2bNAo/Hq/Jx8+ZNQ1ZRK7Nnz4ZUKuUe6enpOj0+l/pFN8gIMVkWhjx5dHQ0RowYUWWZgIAArY7l4eFRIWsgMzOT26f6V7WtbBl7e3uIxWIIBAIIBAKNZVTH0EQkEkEkEmlVz9qg6RUJMX0GDbaurq5wdXXVybFCQ0OxcOFCPH78GG5ubgCA2NhY2Nvbo2XLllyZ/fv3q70uNjYWoaGhAAChUIjg4GAcOXIEkZGRAAClUokjR45g0qRJOqlnbWRzLVvqRiDEVJlMn21aWhoSExORlpYGhUKBxMREJCYmIjc3FwAQFhaGli1b4sMPP8SVK1dw6NAhfP7555g4cSLX6hw/fjzu3buHGTNm4ObNm1izZg127tyJqVOncueZNm0afvzxR2zevBnJycmYMGEC8vLyMHLkSIO8b6WSQUp9toSYPr3lPdRRVFQUA1DhERcXx5W5f/8+69+/PxOLxczFxYVFR0ez4uJitePExcWxtm3bMqFQyAICAtjGjRsrnGvVqlXMx8eHCYVCFhISws6ePVujuuoyrSQrT858Z+5jvjP3MXmxos7HI4QYJvWLxxhjBoz1Zkkmk8HBwQFSqRT29vZ1Otb9p3nosfQYbIQCJH3ZT0c1JOTVpsvPqLZMphvhVUWZCISYBwq2Ro4moSHEPFCwNXLZBTQJDSHmgIKtkcvKo5YtIeaAgq2RoxxbQswDBVsjp5oXgboRCDFtFGyN3Mu5bCnYEmLKKNgaOa4bQUzdCISYMgq2Ro6bhMaGgi0hpoyCrZGjQQ2EmAcKtkZOquqzpW4EQkwaBVsjVqxQIkdeujowZSMQYtoo2BoxVX8tjwfYU8uWEJNGwdaISV8M1bW3soSAzzNwbQghdUHB1ohlccvhUKuWEFNHwdaIZeWVtmwdqL+WEJNHwdaIvRyqSy1bQkwdBVsjpho9RpkIhJg+CrZGTNVn60CZCISYPAq2RowbqkstW0JMHgVbI8Z1I9C8CISYPAq2RiybuhEIMRsUbI1YFt0gI8RsULA1YtRnS4j5oGBrxFQr69L6Y4SYPgq2RqqwWIHCYiUACraEmAMKtkZK1V9rwefBVmRh4NoQQuqKgq2R2n3xHwCAl5MYPB7N+EWIqTOZYLtw4UJ07twZ1tbWcHR0rLD/ypUreP/99+Ht7Q2xWIwWLVpg5cqVFcodO3YM7du3h0gkQmBgIDZt2lShzOrVq+Hn5wcrKyt06tQJCQkJ9fCOKpf+PB+rjqYAAKb2barXcxNC6ofJBNuioiIMGTIEEyZM0Lj/4sWLcHNzw6+//oqkpCTMmTMHs2fPxg8//MCVSU1NRUREBHr27InExERMmTIFY8aMwaFDh7gyO3bswLRp0zBv3jxcunQJQUFBCA8Px+PHj+v9PQIAYwzz/pcEeYkSnRs3wNtBnno5LyGknjETs3HjRubg4KBV2Y8//pj17NmTez5jxgzWqlUrtTLvvfceCw8P556HhISwiRMncs8VCgXz9PRkMTExWtdRKpUyAEwqlWr9GpVD1x8x35n7WOC//2J3MnNq/HpCSPXq8hmtLZNp2daGVCqFs7Mz9zw+Ph59+vRRKxMeHo74+HgApa3nixcvqpXh8/no06cPV0YTuVwOmUym9qiN/KISLPjzBgBg3BsBCHSzrdVxCCHGx2yD7ZkzZ7Bjxw6MGzeO2yaRSODu7q5Wzt3dHTKZDAUFBXj69CkUCoXGMhKJpNJzxcTEwMHBgXt4e3vXqs6rjqYgI7sAjRzFmNSzSa2OQQgxTgYNtrNmzQKPx6vycfPmzRof9/r16xg4cCDmzZuHsLCweqi5utmzZ0MqlXKP9PT0Gh/jTmYOfjxxDwCw4O1WEAsFuq4mIcSADJrAGR0djREjRlRZJiAgoEbHvHHjBnr37o1x48bh888/V9vn4eGBzMxMtW2ZmZmwt7eHWCyGQCCAQCDQWMbDw6PSc4pEIohEohrVsyzGGL744zpKlAx9WrijT0v36l9ECDEpBg22rq6ucHV11dnxkpKS0KtXL0RFRWHhwoUV9oeGhmL//v1q22JjYxEaGgoAEAqFCA4OxpEjRxAZGQkAUCqVOHLkCCZNmqSzepb3R+JDnL33HFaWfMwb0LLezkMIMRyTGZqUlpaG58+fIy0tDQqFAomJiQCAwMBA2Nra4vr16+jVqxfCw8Mxbdo0ro9VIBBwAX38+PH44YcfMGPGDIwaNQpHjx7Fzp078ddff3HnmTZtGqKiotChQweEhIRgxYoVyMvLw8iRI+vlfUkLivH1X8kAgE96NYG3s3W9nIcQYmB6y3uoo6ioKAagwiMuLo4xxti8efM07vf19VU7TlxcHGvbti0TCoUsICCAbdy4scK5Vq1axXx8fJhQKGQhISHs7NmzNaprTdJK5u69xnxn7mO9lsYxebGiRuchhNSOIVK/eIwxZpgwb75kMhkcHBwglUphb29fablr/0gxcPUpKBmwdUwndA500WMtCXl1afsZ1SWzTf0yBdvPp0HJgMi2nhRoCTFzJtNna46+GtgaQV6O6NFcdzcJCSHGiYKtAfH5PAztWLsBEIQQ00LdCIQQogcUbAkhRA8o2BJCiB5QsCWEED2gYEsIIXpAwZYQQvSAgi0hhOgB5dnWA9UI6Nqu2EAIqV+qz6Y+ZyugYFsPcnJyAKDWKzYQQvQjJycHDg4OejkXTURTD5RKJR4+fAg7OzvweLwqy8pkMnh7eyM9PV1vE2KYEro+1aNrVL3y14gxhpycHHh6eoLP109vKrVs6wGfz4eXl1eNXmNvb08flCrQ9akeXaPqlb1G+mrRqtANMkII0QMKtoQQogcUbA1MJBJh3rx5dVow0pzR9akeXaPqGcM1ohtkhBCiB9SyJYQQPaBgSwghekDBlhBC9ICCLSGE6AEFWwNavXo1/Pz8YGVlhU6dOiEhIcHQVTKYEydOYMCAAfD09ASPx8PevXvV9jPGMHfuXDRs2BBisRh9+vTBnTt3DFNZA4mJiUHHjh1hZ2cHNzc3REZG4tatW2plCgsLMXHiRDRo0AC2trYYNGgQMjMzDVRj/Vu7di1ee+01bvBCaGgoDhw4wO035PWhYGsgO3bswLRp0zBv3jxcunQJQUFBCA8Px+PHjw1dNYPIy8tDUFAQVq9erXH/4sWL8f3332PdunU4d+4cbGxsEB4ejsLCQj3X1HCOHz+OiRMn4uzZs4iNjUVxcTHCwsKQl5fHlZk6dSr+/PNP7Nq1C8ePH8fDhw/x7rvvGrDW+uXl5YVvv/0WFy9exIULF9CrVy8MHDgQSUlJAAx8fRgxiJCQEDZx4kTuuUKhYJ6eniwmJsaAtTIOANiePXu450qlknl4eLAlS5Zw27Kzs5lIJGLbtm0zQA2Nw+PHjxkAdvz4ccZY6TWxtLRku3bt4sokJyczACw+Pt5Q1TQ4Jycn9tNPPxn8+lDL1gCKiopw8eJF9OnTh9vG5/PRp08fxMfHG7Bmxik1NRUSiUTtejk4OKBTp06v9PWSSqUAAGdnZwDAxYsXUVxcrHadmjdvDh8fn1fyOikUCmzfvh15eXkIDQ01+PWhiWgM4OnTp1AoFHB3d1fb7u7ujps3bxqoVsZLIpEAgMbrpdr3qlEqlZgyZQq6dOmC1q1bAyi9TkKhEI6OjmplX7XrdO3aNYSGhqKwsBC2trbYs2cPWrZsicTERINeHwq2hJigiRMn4vr16zh16pShq2J0mjVrhsTEREilUvz222+IiorC8ePHDV0tukFmCC4uLhAIBBXugmZmZsLDw8NAtTJeqmtC16vUpEmTsG/fPsTFxalN5enh4YGioiJkZ2erlX/VrpNQKERgYCCCg4MRExODoKAgrFy50uDXh4KtAQiFQgQHB+PIkSPcNqVSiSNHjiA0NNSANTNO/v7+8PDwULteMpkM586de6WuF2MMkyZNwp49e3D06FH4+/ur7Q8ODoalpaXadbp16xbS0tJeqetUnlKphFwuN/z1qfdbcESj7du3M5FIxDZt2sRu3LjBxo0bxxwdHZlEIjF01QwiJyeHXb58mV2+fJkBYMuXL2eXL19mDx48YIwx9u233zJHR0f2xx9/sKtXr7KBAwcyf39/VlBQYOCa68+ECROYg4MDO3bsGHv06BH3yM/P58qMHz+e+fj4sKNHj7ILFy6w0NBQFhoaasBa69esWbPY8ePHWWpqKrt69SqbNWsW4/F47PDhw4wxw14fCrYGtGrVKubj48OEQiELCQlhZ8+eNXSVDCYuLo4BqPCIiopijJWmf33xxRfM3d2diUQi1rt3b3br1i3DVlrPNF0fAGzjxo1cmYKCAvbxxx8zJycnZm1tzd555x326NEjw1Vaz0aNGsV8fX2ZUChkrq6urHfv3lygZcyw14emWCSEED2gPltCCNEDCraEEKIHFGwJIUQPKNgSQogeULAlhBA9oGBLCCF6QMGWEEL0gIItIYToAQVb8sq5f/8+eDweEhMT6+0cI0aMQGRkZL0dn5geCrbE5IwYMQI8Hq/Co1+/flq93tvbG48ePeLmgSVEH2g+W2KS+vXrh40bN6ptE4lEWr1WIBC8UlMOEuNALVtikkQiETw8PNQeTk5OAAAej4e1a9eif//+EIvFCAgIwG+//ca9tnw3QlZWFoYPHw5XV1eIxWI0adJELZBfu3YNvXr1glgsRoMGDTBu3Djk5uZy+xUKBaZNmwZHR0c0aNAAM2bMQPkpR5RKJWJiYuDv7w+xWIygoCC1OlVXB2L6KNgSs/TFF19g0KBBuHLlCoYPH47/+7//Q3JycqVlb9y4gQMHDiA5ORlr166Fi4sLgNJVf8PDw+Hk5ITz589j165d+PvvvzFp0iTu9cuWLcOmTZvw888/49SpU3j+/Dn27Nmjdo6YmBj897//xbp165CUlISpU6figw8+4FYQqKoOxEzoZW4xQnQoKiqKCQQCZmNjo/ZYuHAhY6x0KsLx48ervaZTp05swoQJjDHGUlNTGQB2+fJlxhhjAwYMYCNHjtR4rg0bNjAnJyeWm5vLbfvrr78Yn8/n5h5u2LAhW7x4Mbe/uLiYeXl5sYEDBzLGGCssLGTW1tbszJkzascePXo0e//996utAzEP1GdLTFLPnj2xdu1atW2qVWYBVJh5PzQ0tNLsgwkTJmDQoEG4dOkSwsLCEBkZic6dOwMAkpOTERQUBBsbG658ly5doFQqcevWLVhZWeHRo0fo1KkTt9/CwgIdOnTguhJSUlKQn5+Pvn37qp23qKgI7dq1q7YOxDxQsCUmycbGBoGBgTo5Vv/+/fHgwQPs378fsbGx6N27NyZOnIilS5fq5Piq/t2//voLjRo1UtunuqlX33Ughkd9tsQsnT17tsLzFi1aVFre1dUVUVFR+PXXX7FixQps2LABANCiRQtcuXIFeXl5XNnTp0+Dz+ejWbNmcHBwQMOGDXHu3Dluf0lJCS5evMg9b9myJUQiEdLS0hAYGKj28Pb2rrYOxDxQy5aYJLlcDolEorbNwsKCu6m0a9cudOjQAV27dsWWLVuQkJCA//znPxqPNXfuXAQHB6NVq1aQy+XYt28fF5iHDx+OefPmISoqCvPnz8eTJ0/wySef4MMPP4S7uzsAYPLkyfj222/RpEkTNG/eHMuXL1dbwdXOzg6fffYZpk6dCqVSia5du0IqleL06dOwt7dHVFRUlXUgZsLQncaE1FRUVJTGtbiaNWvGGCu9QbZ69WrWt29fJhKJmJ+fH9uxYwf3+vI3yL766ivWokULJhaLmbOzMxs4cCC7d+8eV/7q1ausZ8+ezMrKijk7O7OxY8eynJwcbn9xcTGbPHkys7e3Z46OjmzatGnso48+4m6QMVa6htqKFStYs2bNmKWlJXN1dWXh4eHs+PHjWtWBmD5ag4yYHR6Phz179tBwWWJUqM+WEEL0gIItIYToAd0gI2aHesaIMaKWLSGE6AEFW0II0QMKtoQQogcUbAkhRA8o2BJCiB5QsCWEED2gYEsIIXpAwZYQQvTg/wHx9IkTH5udBgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#####Testing del modello ottenuto - esempio singolo"
      ],
      "metadata": {
        "id": "WWCEVS7LWAYI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env = gym.make(\"MountainCar-v0\")\n",
        "\n",
        "state = env.reset()\n",
        "env._max_episode_steps = 800\n",
        "done = False\n",
        "\n",
        "i = 0\n",
        "\n",
        "while not done:\n",
        "\n",
        "    action = agent.epsilon_greedy(state, 0.0) # con epsilon=0.0 agisce greedy\n",
        "\n",
        "    state, reward, done, _ = env.step(action[0])\n",
        "\n",
        "    i += 1\n",
        "\n",
        "    if done == True:\n",
        "      print(f\" - Terminal state occurs in\\n   Step {i} -- State={state}\")\n",
        "\n",
        "env.close()"
      ],
      "metadata": {
        "id": "FrDmGK1lV_Bz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "04ca68fa-3876-4f0f-bb44-4c21279b2ef1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " - Terminal state occurs in\n",
            "   Step 141 -- State=[0.5212139  0.03547214]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#####Testing del modello ottenuto - su mille campioni"
      ],
      "metadata": {
        "id": "jLMZvwBw2oOW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def test(env):\n",
        "\n",
        "   state = env.reset()\n",
        "   env._max_episode_steps = 700\n",
        "   done = False\n",
        "\n",
        "   i = 0\n",
        "\n",
        "   while not done:\n",
        "\n",
        "      action = agent.epsilon_greedy(state, 0.0) # con epsilon=0.0 agisce greedy\n",
        "\n",
        "      state, reward, done, _ = env.step(action[0])\n",
        "\n",
        "      i += 1\n",
        "\n",
        "\n",
        "   return i"
      ],
      "metadata": {
        "id": "eNRWiHzx2oOX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "env = gym.make(\"MountainCar-v0\")\n",
        "step_list = []\n",
        "\n",
        "for j in range(1,1001):\n",
        "  s = test(env)\n",
        "  step_list.append(s)\n",
        "\n",
        "env.close()\n"
      ],
      "metadata": {
        "id": "lW8xLAET2oOY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(8, 4))\n",
        "plt.plot(step_list)\n",
        "plt.ylabel(\"Step for Goal\")\n",
        "plt.xlabel(\"Episodes\")\n",
        "\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "AOAwaKgV2oOZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "count_less_than= 0\n",
        "limit = 170\n",
        "\n",
        "for number in step_list:\n",
        "    if number <= limit:\n",
        "        count_less_than += 1\n",
        "\n",
        "percentage_less_than= (count_less_than / len(step_list)) * 100\n",
        "\n",
        "\n",
        "print(f\"Percentage of Step less than\", limit,\":\\n\", percentage_less_than, \"%\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e62dc847-8162-4aa2-860b-f146d2633678",
        "id": "E6MaMfSY2oOa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Percentage of Step less than 170 :\n",
            " 24.9 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Training with Imitation Learning"
      ],
      "metadata": {
        "id": "G3jIBv4UhaE8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class Agent_imit(Agent):\n",
        "\n",
        "    def imit_step(self, reward, new_state, eps, done):\n",
        "        if done:\n",
        "            self.update_weights(reward, done)\n",
        "        else:\n",
        "            if new_state[1] > 0:\n",
        "                new_action = [2]\n",
        "            else:\n",
        "                new_action = [0]\n",
        "\n",
        "            new_feature_vector = get_feature_vector(new_state[0],\n",
        "                                                    new_state[1],\n",
        "                                                    new_action)\n",
        "            self.update_weights(reward, done, new_feature_vector)\n",
        "            self.feature_vector = new_feature_vector\n",
        "            self.action = new_action\n",
        "        return self.action\n",
        "\n"
      ],
      "metadata": {
        "id": "vS9tUmEylDvP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train_imit(environment, agent, epsilon, num_episodes=1_000):\n",
        "    env = environment\n",
        "    episode_rewards = []\n",
        "    eps = epsilon\n",
        "\n",
        "    print(\"\\n\\tImitation phase =\")\n",
        "\n",
        "\n",
        "    for episode in range(1,num_episodes+1):\n",
        "\n",
        "        ob = env.reset()\n",
        "        action = [np.random.choice(list_actions)]\n",
        "        episode_rewards_sum = 0\n",
        "        done = False\n",
        "        count = 0\n",
        "\n",
        "        while count < 3_000:\n",
        "            count += 1\n",
        "\n",
        "            if count == 3_000:\n",
        "                done = True\n",
        "            else:\n",
        "                done = False\n",
        "\n",
        "            new_ob, reward, done_env, _ = env.step(action[0])\n",
        "\n",
        "            if new_ob[0] >= 0.5:\n",
        "                #debug\n",
        "                print(f\"Goal episod = {episode}  Total reward={episode_rewards_sum}\")\n",
        "                count = 3_000\n",
        "\n",
        "            episode_rewards_sum += reward\n",
        "\n",
        "            if ( 1 <= episode <= num_episodes/3 ) :\n",
        "              action = agent.imit_step(reward, new_ob, eps, done)\n",
        "            else:\n",
        "              action = agent.step(reward, new_ob, eps, done)\n",
        "\n",
        "        # Per Gestire la dinamica di epsilon\n",
        "        if (episode >= num_episodes/3) and (episode % (num_episodes/3) == 0):\n",
        "           eps = round(eps - 0.1, 3)\n",
        "           if episode != num_episodes:\n",
        "              print(\"\\n\\tEpsilon =\", eps)\n",
        "\n",
        "        episode_rewards.append(episode_rewards_sum)\n",
        "\n",
        "    save_params(agent.weights)\n",
        "    print(\"\\n...Params Saved...\\n\")\n",
        "    env.close()\n",
        "    print(\"\\n...End of training...\\n\")\n",
        "\n",
        "    return episode_rewards, agent"
      ],
      "metadata": {
        "id": "hwIMznp-hyQL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "agent2 = Agent_imit(param_vector_size=size_iht,\n",
        "              learning_rate=(0.5/8),\n",
        "              discount=0.9)\n",
        "\n",
        "\n",
        "episode_rewards, agent2 = train_imit(gym.make(\"MountainCar-v0\"), agent2, epsilon=0.3, num_episodes=30)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r_WmDuSShypx",
        "outputId": "d0dadfca-029b-4834-b87f-b9c86d6a085b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\tImitation phase =\n",
            "Goal episod = 1  Total reward=-121.0\n",
            "Goal episod = 2  Total reward=-162.0\n",
            "Goal episod = 3  Total reward=-93.0\n",
            "Goal episod = 4  Total reward=-88.0\n",
            "Goal episod = 5  Total reward=-112.0\n",
            "Goal episod = 6  Total reward=-121.0\n",
            "Goal episod = 7  Total reward=-112.0\n",
            "Goal episod = 8  Total reward=-113.0\n",
            "Goal episod = 9  Total reward=-152.0\n",
            "Goal episod = 10  Total reward=-121.0\n",
            "\n",
            "\tEpsilon = 0.2\n",
            "Goal episod = 12  Total reward=-2210.0\n",
            "Goal episod = 13  Total reward=-149.0\n",
            "Goal episod = 14  Total reward=-195.0\n",
            "Goal episod = 15  Total reward=-219.0\n",
            "Goal episod = 16  Total reward=-375.0\n",
            "Goal episod = 17  Total reward=-1376.0\n",
            "Goal episod = 18  Total reward=-251.0\n",
            "Goal episod = 19  Total reward=-317.0\n",
            "Goal episod = 20  Total reward=-662.0\n",
            "\n",
            "\tEpsilon = 0.1\n",
            "Goal episod = 21  Total reward=-439.0\n",
            "Goal episod = 22  Total reward=-577.0\n",
            "Goal episod = 23  Total reward=-325.0\n",
            "Goal episod = 24  Total reward=-507.0\n",
            "Goal episod = 25  Total reward=-223.0\n",
            "Goal episod = 26  Total reward=-239.0\n",
            "Goal episod = 27  Total reward=-158.0\n",
            "Goal episod = 28  Total reward=-359.0\n",
            "Goal episod = 29  Total reward=-1276.0\n",
            "Goal episod = 30  Total reward=-2498.0\n",
            "\n",
            "...Params Saved...\n",
            "\n",
            "\n",
            "...End of training...\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#####Testing del modello ottenuto - su mille campioni"
      ],
      "metadata": {
        "id": "ex9q7BmtoixQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def test(env):\n",
        "\n",
        "   state = env.reset()\n",
        "   env._max_episode_steps = 700\n",
        "   done = False\n",
        "\n",
        "   i = 0\n",
        "\n",
        "   while not done:\n",
        "\n",
        "      action = agent2.epsilon_greedy(state, 0.0) # con epsilon=0.0 agisce greedy\n",
        "\n",
        "      state, reward, done, _ = env.step(action[0])\n",
        "\n",
        "      i += 1\n",
        "\n",
        "\n",
        "   return i"
      ],
      "metadata": {
        "id": "Q2UaJPeqoixS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "env = gym.make(\"MountainCar-v0\")\n",
        "step_list = []\n",
        "\n",
        "for j in range(1,1001):\n",
        "  s = test(env)\n",
        "  step_list.append(s)\n",
        "\n",
        "env.close()\n"
      ],
      "metadata": {
        "id": "XkVkA3v1yUgO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(8, 4))\n",
        "plt.plot(step_list)\n",
        "plt.ylabel(\"Step for Goal\")\n",
        "plt.xlabel(\"Episodes\")\n",
        "\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "a8I66D8G02om"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "count_less_than= 0\n",
        "limit = 170\n",
        "\n",
        "for number in step_list:\n",
        "    if number <= limit:\n",
        "        count_less_than += 1\n",
        "\n",
        "percentage_less_than= (count_less_than / len(step_list)) * 100\n",
        "\n",
        "\n",
        "print(f\"Percentage of Step less than\", limit,\":\\n\", percentage_less_than, \"%\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Farw46Kj1W8D",
        "outputId": "3061c4f3-6654-4903-c8be-17fcff7d3fe0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Percentage of Step less than 170 :\n",
            " 60.5 %\n"
          ]
        }
      ]
    }
  ]
}