GymCLIFFDoubleQLearningTODO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/EmmanuelADAM/IntelligenceArtificiellePython/blob/master/GymCLIFFDoubleQLearningTODO.ipynb\" target=\"_parent\">Ouvrir sur collab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning\n",
    "## Appliqué à [Gym.OpenAI](https://gym.openai.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Test de ML par Double Q-Learning pour atteindre l'objectif\n",
    "\n",
    "**Utilisation de l'environnement gymnasium**\n",
    " (voir la page d'introduction à [gymnasium](https://www.gymlibrary.dev/))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Si besoin, installer gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install gymnasium "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "----\n",
    "#### L'environnement FrozenLake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gymnasium as gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "- Utiliser l'environnement `CliffWalking` (une grille où il faut trouver le plus court chemin vers l'arrivée)\n",
    "- Chaque action coûte 1 point, l'arrivée n'en rapporte pas.\n",
    "- On peut, \n",
    "    - pour accélérer la résolution en décidant de récompenser l'agent de quelques points s'il parvient au but.\n",
    "    - ou, *de préférence*, changer la fonction qlearn pour qu'elle retourne \n",
    "        - total_r : récompense totale; \n",
    "        - r : récompense du dernier état rencontre\n",
    "        - states_list : liste des états traverses\n",
    "        - actions_list : liste des actions effectuées\n",
    "        - goal : but atteint\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### Etude de l'environnement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "specification de l'environnement :  EnvSpec(id='CliffWalking-v0', entry_point='gymnasium.envs.toy_text.cliffwalking:CliffWalkingEnv', reward_threshold=None, nondeterministic=False, max_episode_steps=None, order_enforce=True, autoreset=False, disable_env_checker=False, apply_api_compatibility=False, kwargs={'render_mode': 'ansi'}, namespace=None, name='CliffWalking', version=0, additional_wrappers=(), vector_entry_point=None)\n",
      "espace d'actions :  Discrete(4)\n",
      "espace d'etats :  Discrete(48)\n",
      "Environnement : (x)=position du robot, (o)=zone accessible, (C)=coline où le robot reste coincé, (T)=but \n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "x  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "grille 4x12, chaque cellule (i,j) a comme n° (12*i+j)\n",
      "X = Start ( (36, {'prob': 1}) ), T = Goal (47) , C = obstacle\n",
      "\n",
      " 4 actions :  0->haut, 1->droite, 2->bas, 3->gauche\n",
      "récompense de -1 par état, de -100 par coline, pas de récompense au but T\n",
      "\n",
      "Prendre une action au hasard.. \n",
      "action choisie= 0 ; l'exécuter...\n",
      "nouvel état=24, recompense sur cet etat=-1, etat final (but)=False, feuille=False, info={'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "x  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CliffWalking-v0', render_mode='ansi')\n",
    "print(\"specification de l'environnement : \", env.spec)\n",
    "print(\"espace d'actions : \", env.action_space ) #ici 4 actions discrétisées\n",
    "print(\"espace d'etats : \", env.observation_space) #ici 48 cellules possibles\n",
    "\n",
    "x = env.reset()\n",
    "print(\"Environnement : (x)=position du robot, (o)=zone accessible, (C)=coline où le robot reste coincé, (T)=but \")\n",
    "print(env.render())\n",
    "print('grille 4x12, chaque cellule (i,j) a comme n° (12*i+j)')\n",
    "print(\"X = Start (\",x,\"), T = Goal (47) , C = obstacle\")\n",
    "print()\n",
    "print(' 4 actions :  0->haut, 1->droite, 2->bas, 3->gauche')\n",
    "print('récompense de -1 par état, de -100 par coline, pas de récompense au but T')\n",
    "print()\n",
    "\n",
    "print(\"Prendre une action au hasard.. \")\n",
    "a = env.action_space.sample()\n",
    "print(\"action choisie=\",a,\"; l'exécuter...\")\n",
    "#on recupère des valeurs en retour\n",
    "#new_s, reward, done, truncated, info = env.step(a)\n",
    "new_s, reward, done, truncated,info = env.step(a)\n",
    "print(f\"nouvel état={new_s}, recompense sur cet etat={reward}, etat final (but)={done}, feuille={truncated}, info={info}\")\n",
    "\n",
    "print(env.render())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### Test des actions\n",
    "\n",
    "Sous Gym, `step` permet d'effectuer une action. \n",
    "En retour la fonction retourne une observation sur l'etat d'arrivee, sa recompense, son type (final ou non), et des informations.\n",
    "  \n",
    "Avec la librairie gym installé sous python 3.10, step retourne : \n",
    "  - observation = position où se trouve l'agent,\n",
    "  - reward = recompense,\n",
    "  - done = vrai si état final atteint,\n",
    "  - goal = vrai si but atteint,\n",
    "  - truncated = vrai si but état feuille,\n",
    "  - info = probabilité de succès de l'action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <font color=\"red\">Résolution</font>\n",
    "Ci-dessous la solution qui semble idéale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "x  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 24 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  x  o  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 25 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  x  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 26 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  x  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 27 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  x  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 28 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  x  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 29 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  x  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 30 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  x  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 31 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  x  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 32 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  x  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 33 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  x  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 34 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  x\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "\n",
      "pos° actuelle: 35 ,gain: -1 ,fini: False , {'prob': 1.0}\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  x\n",
      "\n",
      "\n",
      "pos° actuelle: 47 ,gain: -1 ,fini: True , {'prob': 1.0}\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CliffWalking-v0', render_mode='ansi')\n",
    "#solution = politique = suite d'actions menant à un but\n",
    "actions = [0,1,1,1,1,1,1,1,1,1,1,1,2]\n",
    "env.reset()\n",
    "for a in actions:\n",
    "    observation, reward, done, _, info = env.step(a)\n",
    "    print(env.render())\n",
    "    print(\"pos° actuelle:\", observation,\",gain:\", reward,\",fini:\", done,\",\", info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Exemple d'algorithme"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random as rnd\n",
    "\n",
    "env = gym.make('CliffWalking-v0', render_mode='ansi')\n",
    "actions = {0:'Haut', 1:'Droite', 2:'Bas', 3:'Gauche'}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialiser la Q-Table\n",
    "# autant de cases que l'environnement en possède, \n",
    "# contenant autant de valeurs que d'actions possibles\n",
    "# donc ici une matrice 64 x 4\n",
    "QA = np.zeros([env.observation_space.n,env.action_space.n])\n",
    "QB = np.zeros([env.observation_space.n,env.action_space.n])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mise en place des paramètres\n",
    "Pour rappel avec l'algo de Double Q Learning, on réalise, plus ou moins alternativement :\n",
    "  - $a^* \\gets  argmax_{a} QA(s', a)$,\n",
    "  - $Q_A(s,a) \\gets Q_A(s,a) + \\lambda \\times (r + \\gamma \\times Q_B(s', a^*))-  Q_A(s,a))$ <br>\n",
    "ou<br>\n",
    "  - $b^* \\gets  argmax_{a} Q_B(s', a)$\n",
    "  - $Q_B(s,a) \\gets Q_B(s,a) + \\lambda \\times (r + \\gamma \\times QA(s', b^*))-  Q_B(s,a))$<br>\n",
    "avec \n",
    "    - $\\lambda$ : coef d'apprentissage\n",
    "    - $\\gamma$ : coef de réduction \n",
    "    - $r$ : récompense\n",
    "\n",
    "Cette équation note la qualité de l'action *a* à partir de l'état *s* alternativement dans les tables $Q_A$ et $Q_B$.\n",
    "\n",
    "Initialement, les actions sont choisies aléatoirement et notées; puis au fil des tests les actions les plus valuées sont choisies. Pour cela, un tirage est effectuée, s'il est inférieur à un $\\epsilon$, le choix est aléatoire. Cet $\\epsilon$ décroit au fil des tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "lambda_learn = .2\n",
    "gamma = 0.99\n",
    "epsilon = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Récupérer la meilleure action\n",
    "`argmax(tab)` retourne l'indice de la plus grande valeur du tableau.\n",
    "\n",
    "`argmax(QA[2])` retourne donc le no de l'action la plus intéressante à partir de l'état 2 selon la table QA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L'algorithme de Double Q-Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(QA[0, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "##algorithme de Q-Learning simple\n",
    "def q_learn(nb_actions=64):\n",
    "    \"\"\"\n",
    "    effectue un cycle d'apprentissage/recherche de solution' via le Q-Learning simple\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    epoch : no de l'etape\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    total_r : recompense totale\n",
    "    r : recompense du dernier etat rencontre\n",
    "    states_list : liste des etats traverses\n",
    "    actions_list : liste des actions effectuees\n",
    "    goal : but atteint\n",
    "\n",
    "    \"\"\"\n",
    "    s = env.reset()\n",
    "    #sous python 3.10, ajouter\n",
    "    s = s[0]\n",
    "    total_r = 0\n",
    "    done = False\n",
    "    step = 0\n",
    "    states_list = []\n",
    "    actions_list = []\n",
    "\n",
    "    # The Q-Table learning algorithm\n",
    "    while not done and step < nb_actions:\n",
    "        step += 1\n",
    "        # Choose an action by greedily (with noise) picking from Q table\n",
    "        actionsA = QA[s, :]\n",
    "        actionsB = QB[s, :]\n",
    "        if rnd.random()<epsilon :\n",
    "            a = rnd.randint(0, env.action_space.n-1)\n",
    "        else:\n",
    "            bestQa = np.argmax(actionsA)\n",
    "            bestQb = np.argmax(actionsB)\n",
    "            a = bestQa if QA[s, bestQa] > QB[s, bestQb] else bestQb\n",
    "\n",
    "        # Get new state and reward from environment\n",
    "        new_state, r, done, _, _ = env.step(a)\n",
    "\n",
    "\n",
    "        # Q-Learning\n",
    "        if (np.random.uniform(0.1) < 0.5):\n",
    "            aeto= np.argmax(QA[new_state, :])\n",
    "            QA[s, a] = QA[s, a] + lambda_learn*(r + gamma * QB[new_state,aeto] - QA[s, a])\n",
    "        else:\n",
    "            beto=np.argmax(QB[new_state,:])\n",
    "            QB[s, a] = QB[s, a] + lambda_learn*(r + gamma * QA[new_state,beto] - QB[s, a])\n",
    "\n",
    "        s = new_state\n",
    "        total_r = total_r + r\n",
    "        states_list.append(s)\n",
    "        actions_list.append(a)\n",
    "\n",
    "    return total_r, r, states_list, actions_list, done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def try_qlearn(nb_episodes = 4000, nb_actions = 64):\n",
    "    \"\"\"\n",
    "    lance nb_episodes fois un cycle de Q-Learning et memorise chaque solution trouvee\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    solutions_list : liste des solutions (no, recompense totale, liste des etats, liste des actions)\n",
    "    \"\"\"\n",
    "    global epsilon\n",
    "\n",
    "    solutions_list = []\n",
    "    epsilon = 1\n",
    "    for i in range(nb_episodes):\n",
    "        # Reset environment and get first new observation\n",
    "        total_r, r, states_list, actions_list, done = q_learn(nb_actions)\n",
    "        epsilon = epsilon * 0.9999\n",
    "        # memorize if a solution has been found\n",
    "        if done: solutions_list.append((i, total_r, states_list, actions_list))\n",
    "\n",
    "    if (len(solutions_list) == 0): print(\"aucune solution trouvee !!\")\n",
    "    return solutions_list\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Affichage de du résultat\n",
    "Affichons maintenant la liste des actions via l'environnement Gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rendu(solutions_list):\n",
    "    \"\"\" affiche la plus courte sequence d'actions permettant d'atteindre l'objectif q partir des solutions fournies\n",
    "    Parameters\n",
    "    ----------\n",
    "    solutions_list : liste des solutions trouvees\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    None.\n",
    "    \"\"\"\n",
    "    mini_sol = solutions_list[0]\n",
    "    for s in  solutions_list:\n",
    "        if len(s[2]) < len(mini_sol[2]): mini_sol = s\n",
    "    print(\"une solution en \", len(mini_sol[2]), \" etapes : \")\n",
    "    env.reset()\n",
    "    env.render()\n",
    "    for i in range(0, len(mini_sol[2])):\n",
    "        env.env.s = mini_sol[2][i]\n",
    "        print(\"action \", actions[mini_sol[3][i]])\n",
    "        env.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "une solution en  13  etapes : \n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "x  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Haut\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "x  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  x  o  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  x  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  x  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  x  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  x  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  x  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  x  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  x  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  x  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  x  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Droite\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  x\n",
      "o  C  C  C  C  C  C  C  C  C  C  T\n",
      "\n",
      "action  Bas\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  o  o  o  o  o  o  o  o  o  o  o\n",
      "o  C  C  C  C  C  C  C  C  C  C  x\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##ON LANCE LA RESOLUTION : \n",
    "solutions = try_qlearn(15000, 50)\n",
    "if(len(solutions)>0):rendu(solutions)\n",
    "#relancer le bloc si pas de solution trouvee"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "def plot_frequence_sol(solutions_list):\n",
    "    \"\"\"\n",
    "    dessine la frequence de solution trouvees\n",
    "    Parameters\n",
    "    ----------\n",
    "    solutions : liste des solutions\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    None.\n",
    "\n",
    "    \"\"\"\n",
    "    xs = [x[0] for x in solutions_list]\n",
    "    ys = [y[1] for y in solutions_list]\n",
    "    plt.figure(figsize=(10, 4))\n",
    "    plt.plot(xs, ys, '.')\n",
    "    plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1MAAAFfCAYAAACr5ZbOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABO9klEQVR4nO3dfXxU5Z3///eZQMIkkiEwSBJISKAiaCJgoBpQEK0ggpbFX8ViKfFbaSlQ6+INhG7V+qsk3dpuV1atWtfabSt8u2B1VSwoVkRYiOHGoCKKiQmQgIEwQRKTkDnfP8iczmRmksmQZHLzej4eacmZ65xznSuDzDuf61zHME3TFAAAAACgTWyR7gAAAAAAdEeEKQAAAAAIA2EKAAAAAMJAmAIAAACAMBCmAAAAACAMhCkAAAAACANhCgAAAADC0CfSHegq3G63jh49qv79+8swjEh3BwAAAECEmKap06dPKzk5WTZb8PoTYarJ0aNHlZKSEuluAAAAAOgiysrKNGzYsKCvE6aa9O/fX9K5AYuPj49wbwAAAABESnV1tVJSUqyMEAxhqolnal98fDxhCgAAAECrt/+wAAUAAAAAhIEwBQAAAABhIEwBAAAAQBgIUwAAAAAQhh4Vpp544gmlp6erX79+ysrK0jvvvBPpLgEAAADooXpMmFq3bp3uvvtu/eQnP9GePXt09dVXa+bMmSotLY101wAAAAD0QIZpmmakO9EerrjiCl1++eV68sknrW1jxozRnDlzlJeX59e+rq5OdXV11veeteRdLhdLowMAAAC9WHV1tRwOR6vZoEdUpurr61VYWKjp06f7bJ8+fbq2b98ecJ+8vDw5HA7rKyUlpTO6CgAAAKCH6BEP7a2srFRjY6OGDBnis33IkCGqqKgIuE9ubq6WL19ufe+pTPVW5a5avVdyUoZhKGt4gpIcdr82+8qqtKvkpL6eNlBjUxJU7qpVceUZpTvjrPbe2ySpuPKM4qKjdKa+UbX1Z/VZ5RkdPVmrXSUn1T8mSrVn3UoZGKuxwwbo3U++0Fdn3Upy9FO56yud/uqsJCm6j02D4qJV29CoL+saZe9jqKr2rBLsfVR71q2+UTZdENNHwxLsKq48o0+Pf6m6BrfOmo2Saai23q1GnXuzD+wfrcT4fjpcVatTNQ2SpKimZ7H162uTW6bOuk0ZhtR4VjJN6azO/dbBpnN/lqQoSY0tjKchqUeUfAEAADpRX5v02wVZum5MYqS7EpIeEaY8mj+h2DTNoE8tjomJUUxMTGd0KyKaBx3P93HRUSo9WSPDMJSSYFfpyRrt+Oyk/rzT996y2ZmJmnrxYJWerNGF/fvprQPHteXjL6zXB8b20cmas9b3KQn91DfKps8qa9rc132Hq/XK+4FD7/n7R6Q5K+n46XodP13v08Ld1KSh3h30KO6mL4+WgpTvWQEAABCqBrf0vecLdXnqAG1YMjnS3WlVjwhTTqdTUVFRflWo48eP+1WreoN1BaXK3VAktynZDOmfxg/Vi3uOWKEhFK8UVeiVouABxztISVJZ1VfhdhcAAADwsbv0lN78qKLLV6h6xD1T0dHRysrK0ubNm322b968WZMmTYpQryKj3FVrBSnpXNVl/e62BSkAAAAg0v7uNSuqq+oRlSlJWr58uRYsWKAJEyYoOztbTz/9tEpLS7V48eJId61DBLpfSTp3jxLBCQAAAN3dNRcPjnQXWtVjwtS8efN04sQJPfzwwyovL1dGRoZee+01DR8+PNJda3fNp/Hlzc3UvImpkqR3P62McO8AAACA83N56oAuP8VP6kHPmTpfoa4l39kCLSQxOX+LT/XJJund3Gv18t6jytt4IGJ9BQAAAM5HV1nNL9Rs0GMqUz2RdwXKkPTtr6fI2T/GbxqfW9KaNz/Vn3eVBjoMOkF7LoVu72Movl9fHfuyPuDrUZIMQ+oTJfXvFy2bIbm+qle0zabUQXEalmDXgfJqNbhNxfWNUkV1neoaG9V4VnLE9tFFF16gslO1io3uo9SEWBUdccmU1L9flOrPunXWbap/vz66ILqPyqpqZRiGBsb1lSS5TVN9bDZVnq5T3Vm3YvpGKXlAP9mjo3RBTB+5ahp09FStEuKiNS51gFw1DTp8qlYNZ93q36+PYvrYVFFdpzGJ/fWNSxLVr69Nm/Yf07Evv9LoxP4alhArSTpcVaMDFac1JL6fpl8yRLHRfRUbbdP7h10qrjyjQRdEq9xVq08qTit5gF2pg+Lkdps6UHFaoxP760xdo87Un1VCXF9VnTm3BH7fPoZq6xt1Y2aS9R/ofWVV2vLRcUX3tclh76uE2Ghd3vRogDc/qtCr75dr5IUXaO7lwyRJhSVVOlV77ueSEButfn1tKqmsUZoz1upjTb1bsdE2lZ2s1ecnz6j+rFuXDXOott4tw5CGJdhVdrJWp2rrfY4xIS1BF8b3s47nPX13X1mVXtpzVDKkyV8bpNjovkpznhur9YWHVVz5pc91Sed+EeN9rHJXrQpLqmQY0uXDE6zr8Xzv/XiDksoan2upqXdb5/Mc0/vPnuN779d8e7DvO0Kwc4Ry7vbq776yKr1XUmW9N9r7ejtjHL119PWcr84eDwDwoDLVpKtVpgJVoNB+5n89RfMmpmhdweGAIfSK9ATtKq7yCUjegcnz5yjD0Oq5GZo3MTXoB3NJfh8yPzzq0p3PF/ocP8owtG3lNL8PtTX1DdaH7bEpCR0wGgAAAPBGZaqb640LScwYM0Q3j09WTf1ZvX/YpdSBsfryq7Oqa2xU/VlT9Y2NOlxVqzN1jSr8vMoan8kjB+rixHiNSeqvl/Yc1buHTsjUuemPd05J1x2T03W8+itt+ei4BsfH6LoxQ6zAMjYlQT+67mt648NjKq48oxGD46zXA/0mP9hv5T3HChZ2mv+mNMlhV/4tmVq1Yb8aTdMKZd7tkhx2fsMKAADQhVGZakJlKrDvXJkiQ4ZOnqnXqy08d6o5Q9K3r0jR0AF2Pfq3gwGnwNkkPTznUp+pVaFqaUpHd5ru0Z36CgAA0FtQmermkhx25c3N1Mr1Re12L05rAt33kzIwTj+YMlKS5Hjxff15Z1mrx5l/RYp+dO1FVjhwXhBjVWA8vKfHhaOlqk13quh0p74CAADAF5WpJl2tMuXxyvtHtezPezr0HIak/FsydaqmwW81QO/7eEKpls3MSNST38ny2x7s5nQAAACgqwk1G9g6sU8IQ9bwBBkdfA6j6QSZwxx+rzWapkoqayT9o1oW1bSDTfLr26YPjqncVet3nCSHXdkjB2lsSoKyRw4iSAEAAKDbI0x1UzmThsvWSsqa//UUvbR0Uqvt3Ka0asN+xUVH+bWNMgxrwQVJmjcxVdtWTtMLi67UY/PH+00L9A5fAAAAQE9GmOriiivPBLxnasalSXp35bX6/tUj/KpD35+Srh2512r13Ms0NiXBr5o0+zL/h6A1mqZq6t0+bQOtMCf9o8qUNTyh1fAFAAAA9FQsQNHFpTvjZDPkc5+Sd2D53bbPfMKWTdIdk9N9AtC8iamaMmqwz7LerxVVBDxm9shBPm1bmo7nmfbX0vLeAAAAQE9FmOriWgos2w9V+i0G4da55x8FqiZ5b2spBLVlhbnmQY0gBQAAgN6CMNUNBAssrVWtwjlmOFjeGwAAAL0R90x1E577lLxDS/PV9do6zS7QMQEAAACEhspUN8c0OwAAACAyCFM9ANPsAAAAgM7HND8AAAAACANhCgAAAADCQJgCAAAAgDAQpgAAAAAgDIQpAAAAAAgDYQoAAAAAwkCYAgAAAIAwEKYAAAAAIAw8tBeWcletiivPKN0ZF9ZDgL33l+RzrHJXrd4rOSnDMJQ1PKHF4wfrR/Pt5a5abf6wQpVf1islwa5TtQ36etpAjU1J8NsnLjpKpSdrdKq2QQmx0UH7EO4YePpSXFmjQXHRcsT29TlPoOO2NF7no7VzhXL8fWVV2lVyUl9PG6gL4/u1W9+6ovN93wMAgN6LMAVJ0rqCUuVuKJLblGyGlDc3U/Mmplqvt/aB03t/STIkmU3/f/VFTm39pNKnfe7M0bp5XLIVdM7UN6q2/qw27D6iV4sqrHZzxiXpG5ck6sipWuVvPCDTlAxDmnlpol7bX6FArr7IqXunj9J/vlusl/eWywxyzUuvGanRSf1lGIZSEuxaV1CmF3aVWe1vvDRRV35toCq/rNd1oy+0QoWnv56xeGrrIeW9diDo2E4aMVDbPztpff/Nscm6ZGi8frHxgNzmuTGS13h9++spmvQ1p1IS7DpT3+gXBO19bfqs8oxGOONkj+5j9WNfWZX+9fUDevfQP841Z1ySvviyXts/PWEdf2XT2AcKt+WuWt3/3+/rnWY/Lynw+8KjtfD15kcV2nLguK4dfaGuG5MYdKw8wgmagfZp/rNqrrX3fUvnaM/g5T1+3r8MAAAAXZthmmawz5q9SnV1tRwOh1wul+Lj4yPdnQ4T6MPgX94r1X3/XeTTzibpsfnjlZJg1yvvl+uZd4qt15ZeM1KTL3L6fIif8/j2oKGlJ/va4Fh9+kVNpLuhJEc/lbu+Cnv/X9ySqVO1DS2GQo9bs4Ypc5hD37hkiCS1Gr7WFZRpd+kpa/vlqQO0Yclk63vv6uGZ+kYVHXEFDZorZ47WD6aO9DuXdyjy3se7LytuGK3MYQ6lO+N0vPorvfHRMf3HW4fk/V9Am6Sfzbk0YPUy1ODVUhU20N+/e/7vXq3ffcRqc8vlQ/WrW8f5Hbe9dWRFjmofzhfvIQCRFmo2IEw16Q1hKtCHweYfdNtqSoCqE9Ca+6aP0tysYXp531Hlv3agTUHcU630nkI5OX+LVRVtT54K6uYPK/TgSx/6BjRJLy6d5FNJWldQqpXri6x2hqT8W86FrkB//0Yn9tc3H9/ud96Xmh23vQXqy5RRg4N+eG3LB9u2VvuA5ngPAegKCFNt1JPDlOc35T9eu7dDPnACkeAJKikDYzX/mZ2R6YMh5Td90Ct31WpS3ha/YGgzpBeXTNI/PbHd5+9flGFo6bUj9dibn/od96ezxuh7V4/okD4HCp9G07UE+vDalg+2gY4dZRjatnIa1QWEhPcQgK4i1GzAan49SLmrVtsPVarcVWtte2rrIU3K36IfvUCQQs9iSsrdUKS46Chral+n98GUVm3Yb1VuAv0Vc5tSQUmV39+/RtPU4AtiAh53QlrHVaWKK8/49cWUrG1ur2sqd9X63Avp/Vqox240TZVURn4qLLoH3kMAuhsWoOghAv32+FRNg/I2tn4PDNBduU2ppt6tRVen62mv+/ra6r4Zo/To3w6Gdd+f54NeujPOWnjFm82QJqYlyGbI77ft37hkiPaWnfK7Z6ojp/ilO+P8+tKc55pMmUE/2AaqEgQ6dpRhKM0Z2069R0/HewhAd0Nlqgcod9VqZbPfHq9cX6R8ghS6IZtxbpETI4Ryk82Q0pyxuuOqdNnCLE/dcvlQLZ12kfJvyQxrf88HvSSHXfm3ZPpUyYymX2yMTUlQ3txMRTVdVJRhaPXcDCU57PrVreP00tJJ+umsMXpp6aQOX3wiyWH36YtN8qvsea7J88E20GuhHNv7OoFQ8B4C0N1wz1ST7nbPlPcN4cFWUwOC8ayMd+LLev1u22fWKnjn8x8DQ9I3xyXr5X1H/e7Hkf6xIp+Mc9PjAp3v+1PSdcfkdCU57FpXUKpVG/ar0TRlk3TVRU6980nlPxZ38LpfSZJP+5auxSbph9eM1MC4aE1IS/B7LtmaNz/V2oLSgGNiGNLc8UP11z1H1Wia1ge95o8RKCypkmFIlwdYza+kssYKX5Hk3ZetB7+wxq75NXmPa6Drbe3Ykb5OdE+8hwBEGgtQtFF3ClPNn+mEjpM7c7RkyFqqu61uzEjU6/sr5Na5D/F5t2TquW3FOnDsyxb3m3KRU/dMH6Wyk7U6VVuvhNhoDUuwq6berfePnNK/bvxYjV5/dT1Lf8dGR+lvH1To3UMnfJb8jjIMLZ46Qn1shgbHx+i6MUN8ni3l+dAiSYUlVdY5Lx+eoK0Hv7BWqPOEsMuGDfDph/eHbM/xYqNtqql3W8f1Pkfz8wUKHs375lm5L9T2kvTGh8dU+WWdrm16TlioH84CjYn3OXviB72WrqknXi8AAC0hTLVRdwlTHbkMdFdw34xR+uTYl/rr3qPtdszmD80Nlffy1J4Pk94BwpC05JqR+k72cOvDdr++Nr1f5vIJLIE+iL75UYX+/vEXuubiwXJeEKP3SqoUb++j6tqzftWSQAIFlkAVkGCvt1WwD9N8yAYAAD0RYaqNukuY2n6oMmLLQHeG//j2eM0em6ynth4K+PyhOeOS9NLecr/tzy7M0veeL/Q7nicQvfL+US378x6/1386a4wmpCXouXdL9NLeozKlVqcyESAAAAB6tlCzAav5dTOhrMTV1XkqOo///ZDf9qymJaF/MGWkbh6b7DflLMlhV/bIUuWuL/KZOnfdmET94pZMv+2eCk/W8MCrqd14WZKSHHb95rYErZg5OqSQlOSwE6IAAABAZcqju1SmpKZ7pppCQ3v7j2+PV1Zaglb89/vaGsaiFkbT4gLBzBmXrBUzR1sLDDQPP6E+5T6caWfh3EgPAACA3odpfm3UlcOU98p9noCwr6xKc57Y3mJwaStD0vbca33O8dKeozpTf1aStO69wy3uH2UYun/mxT73FHlWbrNJWjFztH4wdaTftXXmlDmm6AEAAKA1TPPrIQI9jHfexFSNTUlQ/txMq9LS3spdtTpT36hFU0dYiyj8pfBw0OmF3pWem8cmB1y5LVB46ewpc0zRAwAAQHuhMtWkK1amAq3cF2UY2rZyms+y1i/sLNVjWz5tl3O+sOhKlZ48EzDANX/uz51T0jUrM6ldVosDAAAAugoqUz1AceUZv0pQo2mqpLLGCi5JDru+fUWq1mz5NKQHrho6d19ToApTlGEoNtrm8wwrtymt2rBfU0YN1ryJqZoyajDT5AAAAAARprq0QCv3RRmGNX3OI8lhV77XSnbBeKpMnkD07qdf6Im/H5Lb/Mc0vTP1jS0GOKbJAQAAAOcQprqwJIddeV73RXkCT6Aw4101io226dX3K/S7bZ+dm6qnc1Py7pic7lPRyh45SLdfOdyn0lTuqg0pwAEAAAC9HfdMNemK90x5hLsCXbj7sYQ4AAAAejOWRm+jrhymIoElxAEAANBbsQAFzgv3RgEAAAAts0W6AwAAAADQHRGmAAAAACAMEQtTJSUl+t73vqf09HTZ7XaNHDlSDz74oOrr633alZaW6qabblJcXJycTqfuuusuvzZFRUWaOnWq7Ha7hg4dqocffljcCgYAAACgI0XsnqkDBw7I7Xbrqaee0te+9jXt379fixYt0pkzZ/Too49KkhobGzVr1iwNHjxY27Zt04kTJ7Rw4UKZpqk1a9ZIOndz2PXXX69p06apoKBABw8eVE5OjuLi4nTPPfdE6vIAAAAA9HBdajW/X/7yl3ryySf12WefSZI2btyo2bNnq6ysTMnJyZKktWvXKicnR8ePH1d8fLyefPJJ5ebm6tixY4qJiZEk5efna82aNTp8+LAMwwjp3KzmBwAAAEAKPRt0qXumXC6XBg4caH2/Y8cOZWRkWEFKkmbMmKG6ujoVFhZabaZOnWoFKU+bo0ePqqSkJOi56urqVF1d7fMFAAAAAKHqMmHq0KFDWrNmjRYvXmxtq6io0JAhQ3zaJSQkKDo6WhUVFUHbeL73tAkkLy9PDofD+kpJSWmvSwEAAADQC7R7mHrooYdkGEaLX++9957PPkePHtUNN9ygb33rW7rzzjt9Xgs0Tc80TZ/tzdt4Zi62NMUvNzdXLpfL+iorK2vztaLz7Cur0jPvHNK+sqpIdwUAAACQ1AELUCxbtky33XZbi23S0tKsPx89elTTpk1Tdna2nn76aZ92iYmJ2rlzp8+2qqoqNTQ0WNWnxMREvwrU8ePHJcmvYuUtJibGZ2og/qHcVaviyjNKd8Z1yIN723r8e/7vXq3ffcT6/pbLh+pXt44L+7y19Wf1WeUZfT1toMamJLT5OF1BR/+MAAAA0Lp2D1NOp1NOpzOktkeOHNG0adOUlZWl5557Tjabb6EsOztbjzzyiMrLy5WUlCRJ2rRpk2JiYpSVlWW1WbVqlerr6xUdHW21SU5O9glt8Nf8A3m5q1b/ua1Yz24rltuUbIaUNzdT8yamBt2ntWPHRUfpTH2j1X5dQalyNxTJ3bTsyezMRC2aMsIKNc2P/5f3Sn2ClCSt331EoxP7S4asQFTuqtV7JSdlGIayhif49K35dXmbmZGoGzMTA+7X2niF26atmh/TewwD/YwAAADQOSK2mt/Ro0c1depUpaam6g9/+IOioqKs1xITEyWdWxp93LhxGjJkiH75y1/q5MmTysnJ0Zw5c6yl0V0uly6++GJde+21WrVqlT755BPl5OTogQceaNPS6L1tNT/vD+SGzoWKjfsr1PzNYJP04tJJGpuS4BeE5oxLUlbaQCXERltBpNxVqzVvfqIXdpX5HMtmSD+cOlJP/P2Q3znUdP5xqQOUv/GAzKY+jRpygT4+9mWr13J56gDtKT1lHdeQlH/LuYDx1NZDynvtQMjj8v2r03XHVel+QeiprYd8+rYoQLvmbVbOHK0rRwzUrpKTQatgrYUv72PaDGnFDaP1i9cP+ITCKMPQtpXTwgpv7RH+qJIBAICeJtRsELEw9fvf/1533HFHwNe8u1RaWqolS5Zoy5Ytstvtmj9/vh599FGfKXpFRUVaunSpdu3apYSEBC1evFgPPPBAyMuiS70rTJW7ajU5f4tflSYYw5BWBvgQ79NG5wLRa/uDL/rR2WZcMkR/+/BYWPsuvWakvpM9XMWVZ/Tup5V6/K1Dfm28q0JPvX1IeRtbDm0zMxL15HeyrO+DVZg84STQeW2S3AGO/cKiK5U9clCbrrE9KlxUyQAAQE/U5cNUV9Pdw1Tz6sC+sirtKjmpEc442aP7+FQNXnn/qJb9eU+bjh/sQ3xvZ5P0zMIsfe/5wpDaL5s2UvfOGB0w0EYZhu6febF+sTF4aJXOhVszhMpUSxWjYOdvS4WrPY4BAADQFYWaDdr9nil0vubVgXEpA7S79JRPG0/VQJJWri9q8zkIUoG5pZCDlCQ9/vdDuv3KcxWv5oGp0TRDmpI4MyNRf9t/TI2mqSjD0Oq5GX7hpbWKUbDzl1TWhByE2uMYAAAA3Rlhqpsrd9X63MfkNuUXpDzbc9cXSYYC3rOEzmGaUklljdKdcbIZCnmqpbe/7T+mDUuyVVPvVpozNmDVqfl7YtWG/ZoyarDVNtD5owxDac7YkPvRHscAAADozrrMQ3sRnkDVgWDcCu/DO9qPJ2wkOezKm5sZ1l/ARtNUTb1b2SMHBawAtVQx8vCcP6rpvsJgFa6WtMcxAAAAujMqU91c0WFXyG1tOleVIk91DkPSwknD9fyOz2Wa/mFj3sRUxcX0afX+NUO+P7PWqj+hVozmTUzVlFGDVVJZE7DCFYr2OAYAAEB3RZjqxspdtfrF66Ev+73ixtHaW3pKG7vQinvtbWH2cE1MG6hTtfVKiI1Wv742/f3AF/qvnaVtOk7zRR6CtlPwcLryxtH6wZSR+sHUkUHDRtbwhKDT/WyS8m45d5/bqg37W7xHypunYhTKPkkO+3kHoPY4BgAAQHdEmOqmyl21euX9o22atrf14y+0/dCJjutUhC2dNlL3zRjtt/2SZIf+tKvUb6xybxytm8cm67ltJXrmnc9k6lyAufPqdD39TrHfcX46a4wmpCWopt6t2Gibdc+SJD23rUS/2/bZuQUfJK2YeS5ISS2HjUDB5/4bLtZlwwb4hK+2Vn+oGAEAAHQ8lkZv0p2WRm/+8FxI87+eqtVNqxUGsq6g1AosVtiZOtJ6vdxVawUPSWEt+e19jLaGl/PZFwAAAO2L50y1UXcJUy09cDfKMDRnfLI27D4S0fuivn/1CP3unc/avJx6S1PmWmKT9G7uta2GkLYEFu/w5Zkmx8NoAQAAegeeM9VDBVu976ezxujGy5KU5LDr3hkXa33hYT266WDnd1DSoP7Rejf3Wq1581P9eVfge5UCLarw9Hcv151/KPS7V8lmSCtuGG1NfXt531HrwbZtWUGuLff2ME0OAAAArSFMdTPBVmrzBClJ2nrwC/16c2SClCT968aPdfPYZK2em6nhzljlv3bAJzjlzhytAbF9/So/141JVL7X/UM2SXdOSdcdk9N9wswPpozUzWOTOzzosLACAAAAWsI0vybdZZqf1PIUtJamATZnk/Ti0kn63+KTVqXHw7MQwtABdv1o7Z6QVrbz9sKiK5U9cpDVp8KSKhmGdPnwBCugBJt2x/1DAAAAiCSm+fVgLU1BCzYNcO74ZG3Yc9Rnm1tSTb3bp9LjvUqd57hn6s8GrBY9+rePtX73Eb9zNX+mUZLDrtljQ1+Wm4oQAAAAugPCVDcVLHAEmwa4cFKa/rr3aNAHubYUYIKFt1/dOk7fzR6uZ7YW67X95W2+hwkAAADozpjm16Q7TfNrTbBpgB25Qh1T8wAAANBTsDR6G/WkMCVxPxIAAAAQLu6Z6uW4HwkAAADoWLZIdwAAAAAAuiPCFAAAAACEgTAFAAAAAGEgTAEAAABAGAhTAAAAABAGwhQAAAAAhIEwBQAAAABhIEwBAAAAQBgIU0AHKXfVavuhSpW7aiPdFQAAAHSAPpHuALq3cletiivPKN0ZpySHvV33CefYXcW6glLlbiiS25RshpQ3N1PzJqZGulsAAABoR4QphK15YFhxw2hlDnO0GH689zEkrZw5Wj+YOrLVY3uHka4esspdtVbfJcltSqs27NeUUYO7ZH8BAAAQHsIUQlLuqtV7JSdlGIayhidIkl9gyNt4QNK5kLTkmpGafJHTJ/A0DxmmmvYxpJvHJlsBKdCxPWFk68EvAoas9gxY53us4sozVt89Gk1TJZU1hCkAAIAehDDVyzUPDp7v46KjVHqyRqdqG/RR+Wm9sLNUnnxgSFp0dbpfYPAwJT3+90N6/O+HrLZ3XJUeMGRIUt5rB5T/2gGZOheQ7rzK/9iNpqnCkqqAIetUbYN+sfFAu0ypC2d6XvMxTHfGyWbI5xqiDEOx0TZtP1TZZStqAAAAaBvDNM0gH4l7l+rqajkcDrlcLsXHx0e6O52ieXD4p/FDtWH3EYX6hjCkkNt6pgHmbzzQ6j62poN7hxGbpJ9981L99KUP/PthSGaz4LJt5bQ238MVFx2lf3piu18I2rZymiQFrFYFC1/rCkq1asN+NZqmogxDc8Yn68U9R7iHCgAAoBsINRsQppp0pTDV3vcEBTpeuatWk/O3BK0udYS2hK/vXz1Cz24rVqPX27Mt+7+w6EpljxzUarvm93AFOv73rx6h3237LODUwuZj6B3kyl21KqmsUWy0LWhI64wKVVe/xwwAAKCrCTUbMM2vi2nvVeCCHS/YlLuO1JYq1qzLEjVsYD89+NKH1n7B9g80pS7NGdvqeQLdw+V3bMkKUpLv/Vut3Rvl+dp+qDJi91CxqiAAAEDH4TlTXUigVeBy1xdpX1lVux1vZdPxig672qnX7W/4oDj90xPb9YBXkGrJnVeNsN7IhqT7b7g4pJDSWqCMMgzdGeDeME8Q8twb5c0m+QW5QO1CDXznI9iqgjz3CgAAoH0QprqQQB/u3ZLmPLFd6wpK2+V4pqRvPr5d+U0r73VFbamaGZKi+xg+1au8jQf01NuHWt3Xs3JgID+fc6m2rZymO65KDxqEkhx25c3NlPfLpqStB7/wae9pF2UY1v6r52b4BL6OeMBvS5UzAAAAnD+m+XUhgVaBk84truD9nKJQ74GJi44K+lpPuVHOlPQfb/kHJ8+S6z+Y4v8MK48kh12zMxP1SlGF32t1DW5rbPPmZvosJuEdhKaMGuyzAIapwM+UmjcxVVNGDVZJZY0VxDw6aipesFUFO7oiBgAA0FsQproQTwUjd32R3M1e81QUgj1nqTnPB/Te7BcbD+jmscktBs5FU0YEDFMT0hKsP7cUhNryTCnPPVTeOvIBv573U7AgCAAAgPNDmOpi5k1M1ejE/przxHa/5b5jo20hffBu/gE9EM/UtJ5SoQrEbarVRR7GpiTolsuHav3uI9a2Wy4fqrEpCT7tAgUh6fyrPx39gN+WgiAAAADOD/dMdUFjUxKUH+AemzP1jSHdAxPKPUfjUwfor0snyWi5WZdlhNDxUEPNr24dp5eWTtJPZ43RS0sn6Ve3jgu5H6HcD9WSzlicIslhV/bIQQQpAACAdkZlqosKVFEod9WGVAUJdu+Vt92lp7T5w2PdsjIVZRi6f+bF+teNH1vT1+6/4WJ9fqJGawtK5TbbHmrGpiT4VaNCdT7VH6biAQAAdF88tLdJV3pob0vu+b97/aakBaqkrCsotT6gt+Vht8G0Fs46is2Qbrg0Ua9/UOETkjwPzW0eYAJt6w66a78BAAB6olCzAWGqSXcIU+WuWk3O3+JXmdq2clrAD+CeD+iHq87ovv8OfzEKmyG9uGSSaurd+vT4af30pQ/CPlbQc0i6NDleRUerrW1TLnLqF//fZVZVjrABAACAzhBqNmCaXzfS1sUKPIsmbD8Ufl42mlYM9EyBS3PG6oGXPmix0hWsEmaTdENGov72wTGf6XmXDRtghaR9ZVV6r6RKE9J8p90FWwACAAAAiBTCVDcS7spx6c64gAGnpRX9DEn//5xLdd2YIT4hJslhV/4tmVqxPnCla+m0kUodGGtNMbRJ+uE1I3XVRYN97v0KVmU6n3uXAAAAgM7ENL8m3WGan+R7L5T3/UOh7LdyfZEVnAxDyp+bqSmjBquwpEo7PjuhF3aV+t2XFEy5q1Zr3vzUWvDBJmnFzNH6wdSR1utMywMAAEB3xD1TbdRdwpQUflApd9WqsKRKhiFdPjwh4ANk23pcQhMAAAB6GsJUG3WnMAUAAACg44SaDXhoLwAAAACEgTAFAAAAAGEgTAEAAABAGLpEmKqrq9O4ceNkGIb27t3r81ppaaluuukmxcXFyel06q677lJ9fb1Pm6KiIk2dOlV2u11Dhw7Vww8/LG4FAwAAANCRusRzpu6//34lJydr3759PtsbGxs1a9YsDR48WNu2bdOJEye0cOFCmaapNWvWSDp3c9j111+vadOmqaCgQAcPHlROTo7i4uJ0zz33ROJyAAAAAPQCEQ9TGzdu1KZNm7R+/Xpt3LjR57VNmzbpww8/VFlZmZKTkyVJv/rVr5STk6NHHnlE8fHx+tOf/qSvvvpKv//97xUTE6OMjAwdPHhQv/71r7V8+XIZhhHotKqrq1NdXZ31fXV1dcddJAAAAIAeJ6LT/I4dO6ZFixbpv/7rvxQbG+v3+o4dO5SRkWEFKUmaMWOG6urqVFhYaLWZOnWqYmJifNocPXpUJSUlQc+dl5cnh8NhfaWkpLTfhQEAAADo8SIWpkzTVE5OjhYvXqwJEyYEbFNRUaEhQ4b4bEtISFB0dLQqKiqCtvF872kTSG5urlwul/VVVlZ2PpcDAAAAoJdp9zD10EMPyTCMFr/ee+89rVmzRtXV1crNzW3xeIGm6Zmm6bO9eRvP4hPBpvhJUkxMjOLj432+AAAAACBU7X7P1LJly3Tbbbe12CYtLU0///nP9b//+78+0/MkacKECbr99tv1/PPPKzExUTt37vR5vaqqSg0NDVb1KTEx0a8Cdfz4cUnyq1gBAAAAQHtp9zDldDrldDpbbffYY4/p5z//ufX90aNHNWPGDK1bt05XXHGFJCk7O1uPPPKIysvLlZSUJOncohQxMTHKysqy2qxatUr19fWKjo622iQnJystLa2drw7lrloVV55RujNOSQ57pLsDAAAARIxhdpEHMpWUlCg9PV179uzRuHHjJJ1bGn3cuHEaMmSIfvnLX+rkyZPKycnRnDlzrKXRXS6XLr74Yl177bVatWqVPvnkE+Xk5OiBBx5o09Lo1dXVcjgccrlcTPkLYl1BqXI3FMltSjZDypubqXkTUyPdLQAAAKBdhZoNusRDe4OJiorSq6++qn79+mny5Mm69dZbNWfOHD366KNWG4fDoc2bN+vw4cOaMGGClixZouXLl2v58uUR7Hn3Uu6q1fZDlSp31bbYxhOkJMltSqs27G9xn3DOH0pfQu0zAAAA0JEi/pwpj7S0NAUqkqWmpuqVV15pcd/MzExt3bq1o7rWo4VabSquPGMFKY9G01RhSZVmjw1/up/3+T3LhZhquS9UyAAAANAVdOnKFDpWS9Wm5pWfdGecbAEWR7xr7R6tKyhtl/ObTV/N+xJqn9t67rZWwwAAAABvXaYyhc4XrNr03LYS/W7bZ36Vn7y5mcpdXyS3V3tPmJkyarC1IEWoi1QEOn/zvpRU1vgco6UK2cAL/M8ZqC/hVMMAAACA5ghTvVhcdFTA7c9s+0xms8rPlFGDNTqxv745Plkv7jnq09479IQ6Ba/cVasTX9bJZihooIoyDKU5Y322FR12+bUzdK5C1vycgfoyZdRgv2qYR6BgCAAAAARDmOqlyl212lVyMuBrzW9dazRNrfjv97X1k8qgx4uNtmlfWZVWbigKGMQkWRWirQe/8KsMBXL/zIt9Qs2+sirlv34gYNvm0/5GJ/YPOB3w3789rs3VMAAAACAQwlQPEmhKW2vT3JrznvbmraUgJUmvvl+hZ975zG+/5tMGmx+/pXX5Lxs6wPrzuoJSrVxfFLB9oHMWlFQFnA6opipVsEBlk/yqYQAAAEAghKkuKJwH4waa0naqtkH5Gw/IbGGaWyDeLxlqOfBI5wLI77b5BykP75AV6kPNbMY/Qo1n0YlA+9qaOul9PVGGoYlpCX6hKcowlJWWEPDeLw9T0taDX3DfFAAAAFrFan5dzLqCUk3O36L5z+zU5PwtIa2UF2iFuxXri5T32gGfKXe564v0wq7SNgWpFufh6VzoufPq9JCPGaoVM0dbQTLYQhU2Q8q7JVN5czMVZZzraJRhaPXcDI1NSQi4Pclh17yJqXps/vigfW2v52cBAACgZ6My1YUECkW564sUF9NHWcMTglapnttW3GKY8XBLeuzNT0Puj2n9T2A2SS8umaQL4/vpmXeKQw5NhiSjqWpka+pXczePTbb+7FmW3fsaPecem5IgSZoyarBKKmuU5oy1xmnexNSA2yUpa7h/5cqD+6YAAAAQCipTXUigCoxb0rI/7wlapSp31eqZd4o7pD8tFaWiDEN5t2RqbMq5kLfo6vSQjmkzpPxbMvXuymv1wqIrg1aISiprrD8nOex+VSbPub3bZI8c5BeAWtqeNzcz4F+AQKsIAgAAAM1RmepCAlVgPIIt211ceSasaXStsRnSnVel6+kAQe2ns8boxsuSfPpxx1Xp+l0IFbLHbhuv2U1VpySHXeWu2oD3NjUPMy1VmcLlOab3Ahne0wEBAACAllCZ6kKaV2Ca80w/85bujAtaQWrldqcWvbhkku64Kl22ZgeJMgy/ICX5990IcH7PAhAt7ddSmAlWZTofSQ67Vs0aY1XKtq2cxuITAAAACIlhms2fKtQ7VVdXy+FwyOVyKT4+PqJ9KXfVqrCkynoQrUeUYWjbyml+YeKptw8pb6Pv85eiDENPf/dy3fmHQr/nRrXEpnOLOngCxbqCUq3asF+NpmkFnZbCRrmr1qoebT34Rcj7eu9HVQgAAACRFGo2IEw16UphyqMtQeaprYf0i40HfKaqzZuY6nMM72c8ef85yjB0/w0X67JhAwKGmfMJOoQkAAAAdDeEqTbqimFKalsYCdbWe7ukgH8m6AAAAADnEKbaqKuGKQAAAACdK9RswAIUAAAAABAGwhQAAAAAhIEwBQAAAABhIEwBAAAAQBgIUwAAAAAQBsIUAAAAAISBMAUAAAAAYSBMAQAAAEAYCFMAAAAAEAbCFAAAAACEgTAFAAAAAGEgTAEAAABAGAhTAAAAABAGwhS6lHJXrbYfqlS5qzbSXQEAAABa1CfSHUDHKXfVqrjyjNKdcUpy2CPdnVatKyhV7oYiuU3JZkh5czM1b2JqpLsFAAAABESY6qHaO5h0dDArd9Va/ZUktymt2rBfU0YN9jlfdwuIAAAA6LkIUz1QqMEkVJ1RMSquPGP116PRNFVSWWP1mcoVAAAAuhLumepBPPcbFX5eFTSYhHPMQMEslHua2nL/U7ozTjbDd1uUYSjNGXve/QAAAAA6ApWpHsK7ahOIIVnBpDnvqXOSfKbRBasYvfp+uWZdlhS00uXdH0PSoqvTNeuyJJ2pbww4RS/JYVfe3Eyt2rBfjaapKMPQ6rkZVrvnthW3WrnqapiSCAAA0LMRpnqA5lWbtmgeeiTJ1D+m0U0ZNVg2Q37H/vmrH2n1ax8FnGrXvD+mpKffKdbT7xRLCj5Fb97EVE0ZNVgllTVKc8ZaAaTcVatnmvb1Zkg6caZO5a7aNoeVjg46TEkEAADo+Zjm1wMEqh41Z0ra/XmVz7ZAocdzGLcp5a4vUuHnVVoxc7SazcCz2gSaatdaf1qaopfksCt75CAlOew+0xaDHW7Zn/docv4WrSsoDX7CZtYVlGpy/hbNf2Znm/cNBVMSAQAAegcqU11cKBUUz/1GrQWqdz89oVmXJVvftxp6dC6sBApSHoGm2oXSn9am6DWvmBmSX6DyDn6hLrDR3otzBBLKYhoAAADo/qhMdWGhVlA89xu19sNcW1DqUx0JtOhDIN4Vq+YC3YsVSn+8F5doLlDFTJLV10DHDXWBjZaCTntpbTENAAAA9AyEqS6qrVPF5k1M1bu51+r7V48IGpDcpnxCgyf0RBnndjAkGSGEKx9B2rfUn+aLSzQXKPCYkh67bbxeWHSlXlw6Keyw0hlBp/m4tna9AAAA6J6Y5tdFhTNVLMlh16pZY3THVWkqLKnSj9bukel1jEChofmiD5JUWFKlu9bu8Tm/9+IU3symgBaoT979KamsUWy0TTX1bp/FJQIJNE0wyjCUlZZg7dfSyn8taW3VwPYSbDENAAAA9ByEqS4qWKAIpYKS5LBr9li7ztSfDSk0JDnsPtuD7Ts6sb/mPLG91YDW2vFD6X9rged8wkpnBZ22XjcAAAC6F8M0zVaWLegdqqur5XA45HK5FB8fH+nuSDp3z1TzQNHW5bXLXbVhh4ZA+7ZHn87n/AAAAEBHCzUbEKaadMUwJXXNQNEV+wQAAAC0l1CzAdP8uriuOFWsK/YJAAAA6Gys5gcAAAAAYSBMAQAAAEAYCFMAAAAAEAbCFAAAAACEIeJh6tVXX9UVV1whu90up9OpuXPn+rxeWlqqm266SXFxcXI6nbrrrrtUX1/v06aoqEhTp06V3W7X0KFD9fDDD4tFCgEAAAB0pIiu5rd+/XotWrRIq1ev1rXXXivTNFVUVGS93tjYqFmzZmnw4MHatm2bTpw4oYULF8o0Ta1Zs0bSuWULr7/+ek2bNk0FBQU6ePCgcnJyFBcXp3vuuSdSlwYAAACgh4vYc6bOnj2rtLQ0/exnP9P3vve9gG02btyo2bNnq6ysTMnJyZKktWvXKicnR8ePH1d8fLyefPJJ5ebm6tixY4qJiZEk5efna82aNTp8+LAMwwipP131OVMAAAAAOleo2SBi0/x2796tI0eOyGazafz48UpKStLMmTP1wQcfWG127NihjIwMK0hJ0owZM1RXV6fCwkKrzdSpU60g5Wlz9OhRlZSUBD1/XV2dqqurfb4AAAAAIFQRC1OfffaZJOmhhx7Sv/zLv+iVV15RQkKCpk6dqpMnT0qSKioqNGTIEJ/9EhISFB0drYqKiqBtPN972gSSl5cnh8NhfaWkpLTbtQEAAADo+do9TD300EMyDKPFr/fee09ut1uS9JOf/ES33HKLsrKy9Nxzz8kwDP3lL3+xjhdomp5pmj7bm7fxzFxsaYpfbm6uXC6X9VVWVnZe1w0AAACgd2n3BSiWLVum2267rcU2aWlpOn36tCTpkksusbbHxMRoxIgRKi0tlSQlJiZq586dPvtWVVWpoaHBqj4lJib6VaCOHz8uSX4VK28xMTE+UwMBAAAAoC3aPUw5nU45nc5W22VlZSkmJkYff/yxrrrqKklSQ0ODSkpKNHz4cElSdna2HnnkEZWXlyspKUmStGnTJsXExCgrK8tqs2rVKtXX1ys6Otpqk5ycrLS0tPa+PAAAAACQFMF7puLj47V48WI9+OCD2rRpkz7++GP98Ic/lCR961vfkiRNnz5dl1xyiRYsWKA9e/bozTff1L333qtFixZZq2rMnz9fMTExysnJ0f79+/Xiiy9q9erVWr58ecgr+QEAAABAW0X0OVO//OUv1adPHy1YsEC1tbW64oortGXLFiUkJEiSoqKi9Oqrr2rJkiWaPHmy7Ha75s+fr0cffdQ6hsPh0ObNm7V06VJNmDBBCQkJWr58uZYvXx6pywIAAADQC0TsOVNdTW94zlS5q1bFlWeU7oxTksPe684PAAAAhCLUbBDRyhQ6z7qCUuVuKJLblGyGlDc3U/Mmpvaa8wMAAADtLWL3TKHzlLtqrSAjSW5TWrVhv8pdted1zO2HKkM6RqDz564v0r6yqrDPDwAAAEQaYaoXKK48YwUZj0bTVEllTVjHW1dQqsn5WzT/mZ2anL9F6wpK23x+t6Q5T2xvdV8AAACgqyJM9QLpzjjZmi1sGGUYSnPGtvlY4VS5Ap1fksx2qJABAAAAkUKY6gWSHHblzc1UVNNS8VGGodVzM/wWgQhl6l64Va7vXZWuQAvVn0+FDAAAAIgkFqDoJeZNTNWUUYNVUlmjNGesX5AKdYEIT5XJO1C1VOXyPq4kGZK8s5j3vqz2BwAAgO6EMNWLJDnsAUNKsAUi4mL6KGt4gs8+nirXqg371WiaQatcgY7r4Qlj3vuy2h8AAAC6G8IUgi4QsezPewIGm9aqXC0d15S05rbxGnRBjLVvuatWK9cXWRUrtymt3FCkKaMGh1ShoqIFAACASCBMIeDUPQ/PAhPNg02wKldrx40yDGWl+Va7Cj+vUvNTm6a0+/Mqzbqs5XNQ0QIAAECksAAF/BaoaC7cRSJCXfjCNAOkOJ0LVC3piOdnAQAAAKGiMgVJ/5i6V1hSpbvW7gl5gYlQj9vSlMAJaQP9FqYwJGWlJbR47JZWFmS6HwAAADoalSlYkhx2zR6bHFI1qa3HzR45KOgxkhx25d+Sab0ZbZLyb8kMeRqht/MJfgAAAEBbGGawOVa9THV1tRwOh1wul+Lj4yPdnYgrd9W2usBEVzjnuoJSv5UFuWcKAAAA5yPUbECYakKY6r4iEfwAAADQc4WaDbhnCt1eKCsLAgAAAO2Ne6YAAAAAIAyEKQAAAAAIA2EKAAAAAMJAmAIAAACAMBCmAAAAACAMhCkAAAAACANhCgAAAADCQJgCAAAAgDAQpgAAAAAgDIQpAAAAAAgDYQoAAAAAwkCYAgAAAIAwEKYAAAAAIAyEKQAAAAAIA2EKAAAAAMJAmIKl3FWr7YcqVe6qjXRXAAAAgC6vT6Q7gK5hXUGpcjcUyW1KNkPKm5upeRNTI90tAAAAoMuiMgWVu2qtICVJblNatWE/FSoAAACgBVSmoOLKM1aQ8mg0TZVU1ijJYQ+4T7mrVsWVZxQXHaUz9Y1Kd8YFbQsAAAD0RISpbsoTZtojxKQ742Qz5BOoogxDac7YgO29pwR6MDUQAAAAvQ3T/LqhdQWlmpy/RfOf2anJ+Vu0rqD0vI6X5LArb26m9WawSVo9NyNgSGs+JdCDqYEAAADobQhT3Uyo9zeFtTKf0ez/Awg0JdDDMzUQAAAA6A2Y5tfNhHJ/U1tX5gsW0KaMGuxXnQo0JdCjpamBAAAAQE9DZaqb8YQZb94hJpyV+VoKaM15pgRGGb6diDKMoFMDAQAAgJ6IylQ34wkzqzbsV6Np+oWYcFbma+sCFPMmpmrKqMEqqaxRbLRNNfVupTljCVIAAADoVQhT3ZB3mGkeYtoajKTWA1qwfQhPAAAA6M0IU91UsDATTjCSWg5oAAAAAPwRpnqgcIMR1SYAAAAgdISpHopgBAAAAHQsVvMDAAAAgDAQpgAAAAAgDIQpAAAAAAgDYQoAAAAAwkCYAgAAAIAwRDRMHTx4UN/85jfldDoVHx+vyZMn66233vJpU1paqptuuklxcXFyOp266667VF9f79OmqKhIU6dOld1u19ChQ/Xwww/LNE0BAAAAQEeJ6NLos2bN0qhRo7RlyxbZ7Xb95je/0ezZs3Xo0CElJiaqsbFRs2bN0uDBg7Vt2zadOHFCCxculGmaWrNmjSSpurpa119/vaZNm6aCggIdPHhQOTk5iouL0z333BPJywMAAADQgxlmhEo4lZWVGjx4sLZu3aqrr75aknT69GnFx8frjTfe0HXXXaeNGzdq9uzZKisrU3JysiRp7dq1ysnJ0fHjxxUfH68nn3xSubm5OnbsmGJiYiRJ+fn5WrNmjQ4fPizDMAKev66uTnV1ddb31dXVSklJkcvlUnx8fAdfPQAAAICuqrq6Wg6Ho9VsELFpfoMGDdKYMWP0hz/8QWfOnNHZs2f11FNPaciQIcrKypIk7dixQxkZGVaQkqQZM2aorq5OhYWFVpupU6daQcrT5ujRoyopKQl6/ry8PDkcDusrJSWlYy4UAAAAQI8UsTBlGIY2b96sPXv2qH///urXr5/+7d/+Ta+//roGDBggSaqoqNCQIUN89ktISFB0dLQqKiqCtvF872kTSG5urlwul/VVVlbWjlcHAAAAoKdr9zD10EMPyTCMFr/ee+89maapJUuW6MILL9Q777yjXbt26Zvf/KZmz56t8vJy63iBpumZpumzvXkbz8zFYFP8JCkmJkbx8fE+XwAAAAAQqnZfgGLZsmW67bbbWmyTlpamLVu26JVXXlFVVZUVZJ544glt3rxZzz//vFauXKnExETt3LnTZ9+qqio1NDRY1afExES/CtTx48clya9iBQAAAADtpd3DlNPplNPpbLVdTU2NJMlm8y2O2Ww2ud1uSVJ2drYeeeQRlZeXKykpSZK0adMmxcTEWPdVZWdna9WqVaqvr1d0dLTVJjk5WWlpae11WQAAAADgI2L3TGVnZyshIUELFy7Uvn37dPDgQd13330qLi7WrFmzJEnTp0/XJZdcogULFmjPnj168803de+992rRokVWNWv+/PmKiYlRTk6O9u/frxdffFGrV6/W8uXLW5zmBwAAAADnI2Jhyul06vXXX9eXX36pa6+9VhMmTNC2bdv00ksvaezYsZKkqKgovfrqq+rXr58mT56sW2+9VXPmzNGjjz5qHcfhcGjz5s06fPiwJkyYoCVLlmj58uVavnx5pC4NAAAAQC8QsedMdTWhriUPAAAAoGfr8s+ZAgAAAIDujDAFAAAAAGEgTAEAAABAGAhTAAAAABAGwhQAAAAAhIEwBQAAAABhIEwBAAAAQBgIUwAAAAAQBsIUAAAAAISBMAUAAAAAYSBMAQAAAEAYCFMAAAAAEAbCFAAAAACEgTAFAAAAAGEgTAEAAABAGAhTAAAAABAGwhQAAAAAhIEwBQAAAABhIEwBAAAAQBgIUwAAAAAQBsIUAAAAAISBMAUAAAAAYSBMAQAAAEAYCFMAAAAAEAbCFAAAAACEgTAFAAAAAGEgTAEAAABAGAhTAAAAABAGwhQAAAAAhIEwBQAAAABhIEwBAAAAQBgIUwAAAAAQBsIUAAAAAISBMAUAAAAAYSBMAQAAAEAYCFO9WLmrVtsPVarcVRvprgAAAADdTp9IdwCRsa6gVLkbiuQ2JZsh5c3N1LyJqZHuFgAAANBtUJnqhcpdtVaQkiS3Ka3asJ8KFQAAANAGhKleqLjyjBWkPBpNUyWVNZHpEAAAANANEaZ6oXRnnGyG77Yow1CaMzYyHQIAAAC6IcJUL5TksCtvbqaijHOJKsowtHpuhpIc9gj3DAAAAOg+WICil5o3MVVTRg1WSWWN0pyxBCkAAACgjQhTvViSw06IAgAAAMLEND8AAAAACANhCgAAAADCQJgCAAAAgDAQpgAAAAAgDIQpAAAAAAhDh4apRx55RJMmTVJsbKwGDBgQsE1paaluuukmxcXFyel06q677lJ9fb1Pm6KiIk2dOlV2u11Dhw7Vww8/LNM0fdq8/fbbysrKUr9+/TRixAj99re/7ajLAgAAAICOXRq9vr5e3/rWt5Sdna1nn33W7/XGxkbNmjVLgwcP1rZt23TixAktXLhQpmlqzZo1kqTq6mpdf/31mjZtmgoKCnTw4EHl5OQoLi5O99xzjySpuLhYN954oxYtWqQ//vGPevfdd7VkyRINHjxYt9xyS0deIgAAAIBeyjCbl3g6wO9//3vdfffdOnXqlM/2jRs3avbs2SorK1NycrIkae3atcrJydHx48cVHx+vJ598Urm5uTp27JhiYmIkSfn5+VqzZo0OHz4swzC0YsUKvfzyy/roo4+sYy9evFj79u3Tjh07QupjdXW1HA6HXC6X4uPj2+fCAQAAAHQ7oWaDiD60d8eOHcrIyLCClCTNmDFDdXV1Kiws1LRp07Rjxw5NnTrVClKeNrm5uSopKVF6erp27Nih6dOn+xx7xowZevbZZ9XQ0KC+ffv6nbuurk51dXXW9y6XS9K5gQMAAADQe3kyQWt1p4iGqYqKCg0ZMsRnW0JCgqKjo1VRUWG1SUtL82nj2aeiokLp6ekBjzNkyBCdPXtWlZWVSkpK8jt3Xl6efvazn/ltT0lJOZ9LAgAAANBDnD59Wg6HI+jrbQ5TDz30UMAQ4q2goEATJkwI6XiGYfhtM03TZ3vzNp6E2NY23nJzc7V8+XLre7fbrZMnT2rQoEFB92kv1dXVSklJUVlZGVMKOxljHzmMfeQw9pHD2EcOYx85jH3kMPbtxzRNnT592mcGXSBtDlPLli3Tbbfd1mKb5pWkYBITE7Vz506fbVVVVWpoaLAqTYmJiVaVyuP48eOS1GqbPn36aNCgQQHPHRMT4zN1UFLQFQc7Snx8PG/0CGHsI4exjxzGPnIY+8hh7COHsY8cxr59tFSR8mhzmHI6nXI6nWF1qLns7Gw98sgjKi8vt6bibdq0STExMcrKyrLarFq1SvX19YqOjrbaJCcnW6EtOztb//M//+Nz7E2bNmnChAkB75cCAAAAgPPVoc+ZKi0t1d69e1VaWqrGxkbt3btXe/fu1ZdffilJmj59ui655BItWLBAe/bs0Ztvvql7771XixYtstL0/PnzFRMTo5ycHO3fv18vvviiVq9ereXLl1vT8RYvXqzPP/9cy5cv10cffaT//M//1LPPPqt77723Iy8PAAAAQC/WoQtQPPDAA3r++eet78ePHy9Jeuutt3TNNdcoKipKr776qpYsWaLJkyfLbrdr/vz5evTRR619HA6HNm/erKVLl2rChAlKSEjQ8uXLfe53Sk9P12uvvaZ//ud/1uOPP67k5GQ99thjXfYZUzExMXrwwQf9phmi4zH2kcPYRw5jHzmMfeQw9pHD2EcOY9/5OuU5UwAAAADQ03ToND8AAAAA6KkIUwAAAAAQBsIUAAAAAISBMAUAAAAAYSBMAQAAAEAYCFMR8MQTTyg9PV39+vVTVlaW3nnnnUh3qVvJy8vTxIkT1b9/f1144YWaM2eOPv74Y582pmnqoYceUnJysux2u6655hp98MEHPm3q6ur0ox/9SE6nU3Fxcbr55pt1+PBhnzZVVVVasGCBHA6HHA6HFixYoFOnTnX0JXYLeXl5MgxDd999t7WNce9YR44c0Xe+8x0NGjRIsbGxGjdunAoLC63XGf/2d/bsWf3Lv/yL0tPTZbfbNWLECD388MNyu91WG8a9/WzdulU33XSTkpOTZRiG/vrXv/q83pljXVpaqptuuklxcXFyOp266667VF9f3xGX3SW0NPYNDQ1asWKFMjMzFRcXp+TkZH33u9/V0aNHfY7B2Ientfe9tx/84AcyDEO/+c1vfLYz9hFkolOtXbvW7Nu3r/nMM8+YH374ofnjH//YjIuLMz///PNId63bmDFjhvncc8+Z+/fvN/fu3WvOmjXLTE1NNb/88kurTX5+vtm/f39z/fr1ZlFRkTlv3jwzKSnJrK6uttosXrzYHDp0qLl582Zz9+7d5rRp08yxY8eaZ8+etdrccMMNZkZGhrl9+3Zz+/btZkZGhjl79uxOvd6uaNeuXWZaWpp52WWXmT/+8Y+t7Yx7xzl58qQ5fPhwMycnx9y5c6dZXFxsvvHGG+ann35qtWH829/Pf/5zc9CgQeYrr7xiFhcXm3/5y1/MCy64wPzNb35jtWHc289rr71m/uQnPzHXr19vSjJffPFFn9c7a6zPnj1rZmRkmNOmTTN3795tbt682UxOTjaXLVvW4WMQKS2N/alTp8xvfOMb5rp168wDBw6YO3bsMK+44gozKyvL5xiMfXhae997vPjii+bYsWPN5ORk89/+7d98XmPsI4cw1cm+/vWvm4sXL/bZNnr0aHPlypUR6lH3d/z4cVOS+fbbb5umaZput9tMTEw08/PzrTZfffWV6XA4zN/+9remaZ77h6Fv377m2rVrrTZHjhwxbTab+frrr5umaZoffvihKcn83//9X6vNjh07TEnmgQMHOuPSuqTTp0+bF110kbl582Zz6tSpVphi3DvWihUrzKuuuiro64x/x5g1a5b5f/7P//HZNnfuXPM73/mOaZqMe0dq/qGyM8f6tddeM202m3nkyBGrzQsvvGDGxMSYLperQ663K2npA73Hrl27TEnWL4MZ+/YRbOwPHz5sDh061Ny/f785fPhwnzDF2EcW0/w6UX19vQoLCzV9+nSf7dOnT9f27dsj1Kvuz+VySZIGDhwoSSouLlZFRYXPOMfExGjq1KnWOBcWFqqhocGnTXJysjIyMqw2O3bskMPh0BVXXGG1ufLKK+VwOHr1z2vp0qWaNWuWvvGNb/hsZ9w71ssvv6wJEyboW9/6li688EKNHz9ezzzzjPU6498xrrrqKr355ps6ePCgJGnfvn3atm2bbrzxRkmMe2fqzLHesWOHMjIylJycbLWZMWOG6urqfKbW9mYul0uGYWjAgAGSGPuO5Ha7tWDBAt1333269NJL/V5n7COrT6Q70JtUVlaqsbFRQ4YM8dk+ZMgQVVRURKhX3Ztpmlq+fLmuuuoqZWRkSJI1loHG+fPPP7faREdHKyEhwa+NZ/+KigpdeOGFfue88MILe+3Pa+3atdq9e7cKCgr8XmPcO9Znn32mJ598UsuXL9eqVau0a9cu3XXXXYqJidF3v/tdxr+DrFixQi6XS6NHj1ZUVJQaGxv1yCOP6Nvf/rYk3vedqTPHuqKiwu88CQkJio6O5uch6auvvtLKlSs1f/58xcfHS2LsO9IvfvEL9enTR3fddVfA1xn7yCJMRYBhGD7fm6bptw2hWbZsmd5//31t27bN77Vwxrl5m0Dte+vPq6ysTD/+8Y+1adMm9evXL2g7xr1juN1uTZgwQatXr5YkjR8/Xh988IGefPJJffe737XaMf7ta926dfrjH/+oP//5z7r00ku1d+9e3X333UpOTtbChQutdox75+mssebnEVhDQ4Nuu+02ud1uPfHEE622Z+zPT2Fhof793/9du3fvbvP1M/adg2l+ncjpdCoqKsov3R8/ftzvNwFo3Y9+9CO9/PLLeuuttzRs2DBre2JioiS1OM6JiYmqr69XVVVVi22OHTvmd94vvviiV/68CgsLdfz4cWVlZalPnz7q06eP3n77bT322GPq06ePNSaMe8dISkrSJZdc4rNtzJgxKi0tlcT7vqPcd999WrlypW677TZlZmZqwYIF+ud//mfl5eVJYtw7U2eOdWJiot95qqqq1NDQ0Kt/Hg0NDbr11ltVXFyszZs3W1UpibHvKO+8846OHz+u1NRU69/ezz//XPfcc4/S0tIkMfaRRpjqRNHR0crKytLmzZt9tm/evFmTJk2KUK+6H9M0tWzZMm3YsEFbtmxRenq6z+vp6elKTEz0Gef6+nq9/fbb1jhnZWWpb9++Pm3Ky8u1f/9+q012drZcLpd27dpltdm5c6dcLlev/Hldd911Kioq0t69e62vCRMm6Pbbb9fevXs1YsQIxr0DTZ482e8RAAcPHtTw4cMl8b7vKDU1NbLZfP+pjIqKspZGZ9w7T2eOdXZ2tvbv36/y8nKrzaZNmxQTE6OsrKwOvc6uyhOkPvnkE73xxhsaNGiQz+uMfcdYsGCB3n//fZ9/e5OTk3Xffffpb3/7myTGPuI6bakLmKb5j6XRn332WfPDDz807777bjMuLs4sKSmJdNe6jR/+8Iemw+Ew//73v5vl5eXWV01NjdUmPz/fdDgc5oYNG8yioiLz29/+dsDlc4cNG2a+8cYb5u7du81rr7024DKil112mbljxw5zx44dZmZmZq9bqrgl3qv5mSbj3pF27dpl9unTx3zkkUfMTz75xPzTn/5kxsbGmn/84x+tNox/+1u4cKE5dOhQa2n0DRs2mE6n07z//vutNox7+zl9+rS5Z88ec8+ePaYk89e//rW5Z88ea8W4zhprzxLR1113nbl7927zjTfeMIcNG9ajl4huaewbGhrMm2++2Rw2bJi5d+9en3976+rqrGMw9uFp7X3fXPPV/EyTsY8kwlQEPP744+bw4cPN6Oho8/LLL7eW9EZoJAX8eu6556w2brfbfPDBB83ExEQzJibGnDJlillUVORznNraWnPZsmXmwIEDTbvdbs6ePdssLS31aXPixAnz9ttvN/v372/279/fvP32282qqqpOuMruoXmYYtw71v/8z/+YGRkZZkxMjDl69Gjz6aef9nmd8W9/1dXV5o9//GMzNTXV7NevnzlixAjzJz/5ic8HSMa9/bz11lsB//u+cOFC0zQ7d6w///xzc9asWabdbjcHDhxoLlu2zPzqq6868vIjqqWxLy4uDvpv71tvvWUdg7EPT2vv++YChSnGPnIM0zTNzqiAAQAAAEBPwj1TAAAAABAGwhQAAAAAhIEwBQAAAABhIEwBAAAAQBgIUwAAAAAQBsIUAAAAAISBMAUAAAAAYSBMAQAAAEAYCFMAAAAAEAbCFAAAAACEgTAFAAAAAGH4fy70wLLMX4x+AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_frequence_sol(solutions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plus on relance les tests, plus la chance de trouver une solution peu coûteuse apparaît..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def showBestValues(Q):\n",
    "    #bestQValues contient les meilleures valeurs des actions possibles pour chaque état\n",
    "    #et on \"dilate\" les valeurs pour augmenter leurs différences (utile pour le choix des couleurs)\n",
    "    bestQ = [-(max(ligne))**50 for ligne in Q]\n",
    "    #on transforme bestQValues en matrice 4 x 12\n",
    "    bestQValues = np.reshape(bestQ, (4,12))\n",
    "    #on demande à python de générer une \"heatmap\" \n",
    "    fig, ax = plt.subplots(figsize=(5,5))\n",
    "    ax.imshow(bestQValues,cmap='Blues')\n",
    "    plt.title(\"Meilleur score d'une action selon l'état\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showBestValues(QA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showBestValues(QB)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}