Validacion-modelos/Cuadernos/validacion_evaluacion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e5246a72-3e39-4ec8-a213-ab3ea8c354fb",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<img src=\"../Imagenes/logo-final-ap.png\"  width=\"80\" height=\"80\" align=\"left\"/> \n",
    "</figure>\n",
    "\n",
    "# <span style=\"color:blue\"><left>Aprendizaje Profundo</left></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32557824-798f-4bc9-b7cf-c1f176062815",
   "metadata": {
    "tags": []
   },
   "source": [
    "# <span style=\"color:red\"><center>Diplomado en Ciencia de Datos</center></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e85999c-24e6-4dd7-838a-d1bd6849da38",
   "metadata": {},
   "source": [
    "# <span style=\"color:green\"><center>Validación y Evaluación</center></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27685269-8d24-4cb4-b339-9d4cfd8f2646",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/metricas_ini.jpg\" width=600 height= 400 align=\"center\" />      \n",
    "</center>\n",
    "</figure>\n",
    "<center><a href=\"https://commons.wikimedia.org/wiki/File:Menschliche_Hand_umgrenzt_menschliche_Silhouette_20200911_DSC3711.jpg\">PantheraLeo1359531</a>, <a href=\"https://creativecommons.org/licenses/by/4.0\">CC BY 4.0</a>, via Wikimedia Commons</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7963cdcc-eb93-4645-9638-67c4cfaec7be",
   "metadata": {},
   "source": [
    "## <span style=\"color:#4361EE\">Profesores</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "263d794d-a057-4a90-8d8a-07c9c7e14bfd",
   "metadata": {},
   "source": [
    "* Alvaro  Montenegro, PhD, <ammontenegrod@unal.edu.co>\n",
    "* Campo Elías Pardo, PhD, <cepardot@unal.edu.co>\n",
    "* Daniel  Montenegro, Msc, <dextronomo@gmail.com>\n",
    "* Camilo José Torres Jiménez, Msc, <cjtorresj@unal.edu.co>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "532a312f-4dfa-41b0-baa8-d0dea79c7895",
   "metadata": {},
   "source": [
    "## <span style=\"color:#4361EE\">Asesora Medios y Marketing digital</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fd639e5-dc11-44e9-b3aa-19978856dd42",
   "metadata": {},
   "source": [
    "* Maria del Pilar Montenegro, pmontenegro88@gmail.com\n",
    "* Jessica López Mejía, jelopezme@unal.edu.co"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de7b48d8-d9f9-4bbe-bf1e-69cd1b9400e6",
   "metadata": {},
   "source": [
    "## <span style=\"color:#4361EE\">Jefe Jurídica</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca61fdea-6f93-48fd-9cd4-842915d5e5b7",
   "metadata": {},
   "source": [
    "* Paula Andrea Guzmán, guzmancruz.paula@gmail.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98418584-4085-4b93-8079-5deb9d8929a2",
   "metadata": {},
   "source": [
    "## <span style=\"color:#4361EE\">Coordinador Jurídico</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fae60388-2d4d-43ae-b746-9124eb5e5651",
   "metadata": {},
   "source": [
    "* David Fuentes, fuentesd065@gmail.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35dcddff-48c8-43c5-ac16-3351235942cc",
   "metadata": {},
   "source": [
    "## <span style=\"color:#4361EE\">Desarrolladores Principales</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3607304-3be0-496c-be78-3cad03e82dd8",
   "metadata": {},
   "source": [
    "* Dairo Moreno, damoralesj@unal.edu.co\n",
    "* Joan Castro, jocastroc@unal.edu.co\n",
    "* Bryan Riveros, briveros@unal.edu.co\n",
    "* Rosmer Vargas, rovargasc@unal.edu.co\n",
    "* Venus Puertas, vpuertasg@unal.edu.co"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76246be5-3cd0-436c-8f80-61cdaf8b64a5",
   "metadata": {},
   "source": [
    "## <span style=\"color:#4361EE\">Expertos en Bases de Datos</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50966ae9-3a5e-464e-be00-e7c1d32849d4",
   "metadata": {},
   "source": [
    "* Giovvani Barrera, udgiovanni@gmail.com\n",
    "* Camilo Chitivo, cchitivo@unal.edu.co"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d767c1b-f397-4d35-b5df-4047a01493c2",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Introducción</span>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21314ebe-95e4-4e4e-a099-f755a769a70c",
   "metadata": {},
   "source": [
    "Después de desarrollar un modelo: ¿Qué sigue? ¿Cómo estamos seguros de que es funcional? Si hacemos varios modelos. ¿Cómo decidimos cuál elegir? Y si el modelo no está funcionando bien, ¿Cómo revisamos sus problemas? Aquí entran las diferentes métricas y métodos que conforman los conceptos conocidos como \"Validación\" y \"Evaluación\"\n",
    "\n",
    "¿Qué son exactamente?:\n",
    "\n",
    "- \"Evaluación\" se referirá a la(s) métrica(s) que ayudan a observar el desempeño final del modelo\n",
    "- Vamos a pensar en \"Validación\" en los métodos que nos permitan revisar y mejorar la capacidad de generalización de un modelo\n",
    "\n",
    "en ciertos casos puede parecer que estamos hablando de lo mismo, pero la diferencia principal radica en cómo aplicamos estos. Cuando \"Validamos\" un modelo, aplicamos algún tipo de técnica sobre los datos, y hasta los transformamos. Mientras que, al \"evaluar\" un modelo, sacamos un valor/valores de este.\n",
    "\n",
    "Ya hemos visto/usado ejemplos de estos dos. ¿Cuáles?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb40b3dd-b58d-4558-96ac-e853e48b2f90",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Evaluación</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2f0d00d-363b-4704-941a-39e3873a88fe",
   "metadata": {},
   "source": [
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://imgs.xkcd.com/comics/machine_learning.png\"/>\n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bf7b2b9-d97f-4939-9592-1394dab48c13",
   "metadata": {},
   "source": [
    "Tenemos diferentes metricas de evaluación para casos de clasificación y regresión"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "573c6361-73b1-44a7-8c2d-8ad13b340c83",
   "metadata": {},
   "source": [
    "Las funciones de pérdida pueden ser usada como métricas en algunos casos. Por ejemplo, es bastante común tener como función de pérdida en un problema el error cuadrático medio (ECM) y como métrica el error absoluto medio (EAM)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bdb86e2-9060-405a-bf17-4d86970304c7",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4361EE\">Funciones de pérdida para regresión</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2573a4c4-baa3-47ea-a7fa-596db5f87c1c",
   "metadata": {},
   "source": [
    "En este caso algunas de las funciones de pérdida más utilizadas son:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8105758-be94-412e-b4ca-3fc91f31d368",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Error cuadrático medio (ECM): Mean squared error (MSE)</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc05fce6-2f37-4b99-b98d-9fd0f4182c03",
   "metadata": {},
   "source": [
    "</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e92956e-bd3e-4761-b2e3-e07485e8a27a",
   "metadata": {},
   "source": [
    "En este caso, si se tiene $N$ datos para un bloque de entrenamiento, digamos $(x_i, y_i)$, en donde las $y_i$ son números, entonces la función de pérdida es \n",
    "\n",
    "$$\n",
    "\\text{ECM} = \\frac{1}{N} \\sum_{i=1}^N (y_i-\\hat{y}_i)^2\n",
    "$$ \n",
    "\n",
    "Si la salida y la etiqueta son vectores, digamos de tamaño $J$, cada componente de los vectores escribe $\\hat{y}_{ij}$ y ${y}_{ij}$ respectivamente. En este caso la función ECM toma la forma\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{ECM} = \\frac{1}{N} \\sum_{i=1}^N \\sum_{j=1}^J (y_{ij}-\\hat{y}_{ij})^2\n",
    "$$ \n",
    "\n",
    "La desventaja de esta función de pérdida es que es muy sensible a valores extremos. Un valor extremo puede hacer crecer mucha la función de pérdida debido al cuadrado en la fórmula. Una opción para resolver esta situación es la siguiente función de pérdida."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7077569e-1991-4325-9063-bbeb302bc4ab",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Error absoluto medio (EAM): Mean absolute Value (MAE) </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef5cbb22-e46a-4147-bfb1-9ba81a3badff",
   "metadata": {},
   "source": [
    "</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90679d79-627e-41a6-a0cd-111c33532e96",
   "metadata": {},
   "source": [
    "En este caso, en lugar de calcular las diferencias al cuadrado de entre los $\\hat{y}_i$ y los $y_i$ se toman las diferencias absolutas, es decir\n",
    "\n",
    "$$\n",
    "\\text{EAM} = \\frac{1}{N} \\sum_{i=1}^N |y_i-\\hat{y}_i|\n",
    "$$ \n",
    "\n",
    "o,\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{EAM} = \\frac{1}{N} \\sum_{i=1}^N \\sum_{j=1}^J|y_{ij}-\\hat{y}_{ij}|\n",
    "$$ \n",
    "\n",
    "La función de pérdida EAM no resuelve del todo los problemas, debido a que, si su valor es cercano a cero, los valores absolutos son cercanos a cero y esto impacta el cálculo del gradiente, debido a que la función valor absoluto no es derivable en cero. Una alternativa viable a estas dos funciones de pérdida es la siguiente.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "277b57c1-7d37-4e13-9450-ad3e4963bbd0",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Función de perdida de Hubber: Hubber loss</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6096d446-ebb0-453f-9b46-e8677cab5c98",
   "metadata": {},
   "source": [
    "Para el caso de salida y etiqueta de forma (1,) la función de pérdida de Hubber se define como: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f59a6fa3-b36b-4103-9378-47f50a0c3475",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\text{Hubber loss} &= \\tfrac{1}{n}\\sum_{i=1}^n (Y_i- \\hat{Y}_i)^2 \\hspace{2cm} |\\hat{Y}_i-Y_i|\\le \\delta\\\\\n",
    "&=  \\tfrac{1}{n}\\sum_{i=1}^n |Y_i- \\hat{Y}_i)| \\hspace{2cm} |\\hat{Y}_i-Y_i|> \\delta\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c79d4e-c99b-4f6b-b71b-10d5de33ab77",
   "metadata": {},
   "source": [
    "y en el caso de tensores de salida y etiqueta de forma $(J,)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9065d81f-fa1c-4ac2-855e-26d0513b2f50",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "\\text{Hubber loss} &= \\tfrac{1}{N}\\sum_{i=1}^N \\sum_{j=1}^J(Y_{ij}- \\hat{Y}_{ij})^2 \\hspace{2cm} \\sum_{j=1}^J|\\hat{Y}_{ij}-Y_{ij}|\\le \\delta\\\\\n",
    "&=  \\tfrac{1}{N}\\sum_{i=1}^N \\sum_{j=1}^J |Y_{ij}- \\hat{Y}_{ij})| \\hspace{2cm} \\sum_{j=1}^J|\\hat{Y}_{ij}-Y_{ij}|> \\delta\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5c80ef3-69c6-420b-9261-0e48518fca54",
   "metadata": {},
   "source": [
    "Si la diferencia absoluta entre el valor real y el predicho es menor o igual que un valor de umbral, $\\delta$, entonces se aplica ECM. De lo contrario, si el error es suficientemente grande, se aplica EAM."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef31a713-8719-4dba-90a5-0b024105d470",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4361EE\">Funciones de pérdida para clasificación</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c71eddce-02de-4f43-a36f-92c94a976d40",
   "metadata": {},
   "source": [
    "En los problemas de clasificación, las etiquetas son discretas. Cada ejemplo de entrenamiento tiene una categoría asociada."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e977f44a-048d-4a03-b4f3-b32bdabb1b36",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Funciones sigmoide y softmax</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8014a746-c03b-4656-a967-00a4214b9e28",
   "metadata": {},
   "source": [
    "En los problemas de clasificación, lo usual es que las salida de la máquina de aprendizaje sea un tensor de tipo float de forma $(1,)$ si se tienen dos clases y en general un tensor de tipo float de de forma $(J,)$ si se tiene $J$ clases. Este tensor se conoce como un tensor de pre-probabilidad.\n",
    "\n",
    "En el caso binario (dos clases), es usual para algunas máquinas transformar la salida con la función `sigmoide`, la cual es la función de distribución acumulada estándar de la distribución logística. Esta función es definida por\n",
    "\n",
    "$$\n",
    "\\sigma(x) = \\frac{e^x}{1 + e^x} = \\frac{1}{1 + e^{-x}} = \\frac{1}{2} + \\frac{1}{2} \\tanh\\left(\\frac{x}{2}\\right).\n",
    "$$\n",
    "\n",
    "La función *softmax* está diseñada para convertir un tensor de tipo float de forma $(J,)$ en un tensor de la misma forma que representa una distribución de probabilidad. Si $(Y_1, \\ldots,Y_j, \\ldots, Y_J)$ es un tensor de números reales, entonces $\\text{softmax(Y)}$ es un tensor $p=(p_1, \\ldots, p_j, \\ldots, p_J)$, en donde\n",
    "\n",
    "$$\n",
    "p_j = \\frac{e^{y_j}}{\\sum_{k=1}^J e^{y_k}}.\n",
    "$$\n",
    "\n",
    "Como podemos observar, la función softmax es una generalización natural del sigmoide cuando se tiene más de dos categorías."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e95fcb5-717c-4444-a271-f7b94e9554ec",
   "metadata": {},
   "source": [
    "En la práctica $p_j$ se interpreta como la probabilidad que la entrada $X$ pertenezca a la clase $j$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "242dccf6-4e85-4748-9830-706b4e77fcee",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Entropía cruzada binaria: Binary cross entropy</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26b1e98e-19ea-4977-aa8e-f95010fc0cf2",
   "metadata": {},
   "source": [
    "Esta función de pérdida es la más utilizada en los casos de clasificación en dos clases. En este caso se espera que la máquina de aprendizaje y la etiqueta sean tensores de forma $(1,)$.\n",
    "\n",
    "Técnicamente, la entropía cruzada para una entrada $X$ con salida $\\hat{Y}$ y etiqueta $Y$ se define en este caso de la siguiente manera. En este caso las etiquetas solamente serán 0 o 1. Tenemos que $\\hat{Y}= f(X)$,  $\\hat{p} = \\sigma(\\hat{Y})$, y\n",
    "\n",
    "$$\n",
    "\\mathfrak{L} = -\\left[(\\log (\\hat{p}) Y) + \\log (1-\\hat{p})(1-Y) \\right].\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "795d9d19-c5ec-4e96-b22d-f350b341fec0",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Entropía cruzada: Cross entropy</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d18e0cca-fcb3-445b-908c-29449013db49",
   "metadata": {},
   "source": [
    "Para el caso de $J$ clases, definimos el tensor $Z = (Z_1, \\ldots, Z_j, \\ldots Z_J)$, en donde\n",
    "\n",
    "$$\n",
    "Z_j = \\begin{cases} &1, \\quad \\text{si } Y=j\\\\\n",
    "& 0, \\quad \\text{en otro caso} \n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Es decir, $Z$ es el vector que representa la codificación `one-hot` de la etiqueta $Y$. Observe que $Z$ solamente tiene 1 en la posición $j$, si la categoría de la etiqueta es $j$, y cero en el resto de posiciones.\n",
    "\n",
    "Finalmente, para este caso la entropía cruzada es definida para el caso general de $N$ muestras en un lote de datos como\n",
    "\n",
    "\n",
    "$$\n",
    "\\mathfrak{L} = -\\frac{1}{N} \\sum_{i=1}^N\\left[ \\sum_{j=1}^J  Z_{ij}log (\\hat{p}_{ij}) \\right],\n",
    "$$\n",
    "\n",
    "en donde $\\hat{Y}_i= (\\hat{Y}_{i1}, \\ldots, \\hat{Y}_{iJ})$ es la salida para la entrada $X_i$, $\\hat{p}_i = \\text{softmax}(Y_i)$, y $Z_i$ la codificación *one-hot* de la etiqueta $Y_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4385a414-e660-4572-8db4-daefe4e61c51",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4361EE\">Matriz de confusión</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c6d78d9-93b5-4086-ae7f-0be013281c6d",
   "metadata": {},
   "source": [
    "La matriz de confusión, como sugiere el nombre, nos da una matriz como resultado y describe el rendimiento completo del modelo.\n",
    "\n",
    "Supongamos que tenemos un problema de clasificación binaria. Tenemos algunas muestras pertenecientes a dos clases: SÍ o NO. Además, tenemos nuestro propio clasificador que predice una clase para una muestra de entrada dada. Al probar nuestro modelo en N muestras, obtenemos el siguiente resultado."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d31f069-94ac-4a4a-86d5-de0c8b9e67d9",
   "metadata": {},
   "source": [
    "|                            | Clasificados como Positivos | Clasificados como Negativos |\n",
    "|----------------------------:|:-----------------------------:|:-----------------------------:|\n",
    "| Etiquetados como Positivos | Verdaderos Positivos        | Falsos Negativos            |\n",
    "| Etiquetados como Negativos | Falsos Positivos            | Verdaderos Negativos        |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9380dbde-1414-43d5-8521-986452459814",
   "metadata": {},
   "source": [
    "Aquí pensamos en los diferentes casos que puede tomar la clasificación binaria.\n",
    "\n",
    "- El modelo predice \"positivo\", cuando el valor real es positivo (Caso preferido 1 o Verdadero Positivo)\n",
    "- El modelo predice \"positivo\", cuando el valor real es negativo (Error tipo 1 o \"Falso Positivo\" )\n",
    "- El modelo predice \"negativo\", cuando el valor real es positivo (Error tipo 2 o \"Falso Negativo\")\n",
    "- El modelo predice \"negativo\", cuando el valor real es negativo (Caso preferido 2 o Verdadero negativo)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e127158e-1c86-4571-b016-aacdfa3d7c60",
   "metadata": {},
   "source": [
    "En problemas con varias clases, la matriz de confusión permite tener una idea de primera mano de como está funcionando el modelo. Revise el siguiente ejemplo. La matriz de confusión, como sugiere el nombre, nos da una matriz como resultado y describe el rendimiento completo del modelo.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e82a968-870d-4035-a06d-cccb465029cf",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<img src=\"../Imagenes/Wikidata_P407_analysis_matrix.png\" width=600 height= 600 align=\"center\" />      \n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b47dfd46-b412-477a-9d5e-3b4c87140b51",
   "metadata": {},
   "source": [
    "Fuente [Wiki-commons](https://commons.wikimedia.org/wiki/Main_Page)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67a2e563-fc66-4b2d-9851-a1f45e8b7372",
   "metadata": {},
   "source": [
    "con base en esto podemos obtener diferentes métricas:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa0ec6b-52ae-4d89-946a-d40404296e46",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Exactitud:  Accuracy</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8fc1cb9-94d2-4125-ab80-bc6b89e20f25",
   "metadata": {},
   "source": [
    "La exactitud de la clasificación es lo que generalmente queremos decir cuando usamos el término exactitud. Es la relación entre el número de predicciones correctas y el número total de muestras de entrada. En el caso binario es usual la siguiente nomenclatura.\n",
    "\n",
    "$$\n",
    "Accuracy = \\frac{\\text{verdaderos positivos} + \\text{verdaderos negativos} }{\\text{total de la muestra}}\n",
    "$$\n",
    "\n",
    "El mejor valor es 1 y el peor valor es 0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6594a58-8684-47bc-a53d-0598006ec14d",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Precisión: precision_score</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d6b54f-9f6e-4bd3-b740-08f2bad06ec4",
   "metadata": {},
   "source": [
    "\n",
    "$$\n",
    "Precision = \\frac{\\text{verdaderos positivos}}{\\text{verdaderos positivos}+ \\text{falsos positivos}}\n",
    "$$\n",
    "\n",
    "Esta métrica indica que tan preciso es el número  de positivos detectados por el modelo. Se puede pensar como enfocada en minimizar los falsos positivos.\n",
    "\n",
    "\n",
    "El mejor valor es 1 y el peor valor es 0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32650238-3d2a-4ad0-9dcd-4634f33fe23d",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Recuperación: recall_score</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bba455e-4b38-4468-87ef-40fbf5471c3a",
   "metadata": {},
   "source": [
    "La sensibilidad o recuperación (recall) es intuitivamente la capacidad del clasificador para encontrar todas las muestras positivas. El mejor valor es 1 y el peor valor es 0.\n",
    "\n",
    "$$\n",
    "Recall = \\frac{\\text{verdaderos positivos}  }{\\text{verdaderos positivos}+ \\text{falsos negativos}}\n",
    "$$\n",
    "\n",
    "En algunos problemas con datos altamente desbalanceados como en el problema de detección de fraude en tarjetas de crédito o la detección temprana de cáncer de seno, un valor alto de exactitud, no es realmente lo mejor. En estos problemas, la recuperación, que es la métrica que mide la capacidad del modelo para detectar muestras positivas es la métrica que se privilegia por encima de las demás.  En estos casos es usual el uso de pesos diferenciados de las observaciones. Se puede pensar como enfocada en minimizar los falsos negativos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c6aaee0-61bc-49c1-b1dd-5c57f7b734af",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Puntaje F1: F1 score</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f9ff629-a869-4658-afa9-ab66d0be850d",
   "metadata": {},
   "source": [
    "El problema con la precisión y la recuperación es que solo toma en cuenta la mitad de la información de la matriz de confusión, así que podríamos estarnos perdiendo de algo."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9e2a56-adcd-4cea-94f6-27c65b05d4a8",
   "metadata": {},
   "source": [
    "La puntuación F1 es la media armónica entre la precisión y la recuperación. El rango para F1 Score es $[0, 1]$. F1 nos dice qué tan preciso es el clasificador (cuántas instancias clasifica correctamente), así como qué tan robusto es (no pierde una cantidad significativa de instancias).\n",
    "\n",
    "Alta precisión pero menor recuperación, le brinda una precisión extrema, pero luego pierde una gran cantidad de instancias que son difíciles de clasificar. Cuanto mayor sea el F1 Score, mejor será el rendimiento de nuestro modelo. Matemáticamente, se puede expresar como:\n",
    "$$\n",
    "\\begin{aligned}\n",
    "F1 &= 2 \\times \\frac{1}{\\frac{1}{precision} + \\frac{1}{recall}} = 2 \\times \\frac{precision \\times recall}{precision + recall}\n",
    "\\end{aligned}\n",
    "$$ \n",
    "\n",
    "El puntaje F1 intenta encontrar el equilibrio entre precision y recall.\n",
    "\n",
    "Como usamos la media armónica, tenemos una métrica bastante robusta contra datos atípicos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97e4fac-35c0-47c4-9248-6fbb41522d63",
   "metadata": {},
   "source": [
    "Podemos obtener varias de estas métricas de forma sencilla con el paquete `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ad6a3213-5ce0-4ffd-914e-33e27444fe0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matrix de confusión:\n",
      "[[ 9  0  0]\n",
      " [ 0 11  1]\n",
      " [ 0  1  8]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAEjCAYAAACxTI37AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcxUlEQVR4nO3debgdVZnv8e/vnJyMJCQhEJIYINgxCNIMRkYbA2gzdYvtdQCRK4pGFAVRH68oLahX2uuVVlvx2hEQkAAi4ICiiYC5gK2YAFFJIg5AGJJITkJIINMZ3v6j6sDO4Qx777P3rqqd3+d56smuYa96dz2cl7VWrVqliMDMrMhasg7AzGyonMjMrPCcyMys8JzIzKzwnMjMrPCcyMys8JzImoikMyQtrFFZiyS9txZl1YKkyZLulrRJ0mVDKOdTkq7oY/vRkn4racLQIrUsDMs6gGYn6TFgKjA1ItpLtj8IHAzMiIjHBiljH+BRoC0iOvs7LiLmA/OHHHQ+zQXagXExhMGPEXFp722SpgOXAv8UEc9UH6JlxTWyxngUOL1nRdKBwOhankBSs/9PaW9g+VCSWH8i4omIeF1EPF3rsq0xnMga47vA/yxZfxdwbekBkk6R9KCkjZKekHRJye670383SHpO0pGSzpL0K0lfkbQOuCTddm9a3lGS2tPaBpIOkvSMpP36ClDSGyT9UdKzkr4BqNf+90hakZaxQNLe/f1YSa+V9F+SNqS/5ax0+66SrpW0VtJKSRdJakn3nSXpXklfTs/xqKST0n1Xp9fsE+nvf72kqyX975JzzpH0ZMn6/5L0VNoUfVjS8en2SyRdV3LcGyUtS2NdJOmVJfsek/RxSb9Pr8v3JI3s73dbdpzIGuM3wDhJr5TUCpwGXNfrmOdJkt144BTgA5LelO47Jv13fETsEhG/TtcPBx4BJgNfKC0sIv4L+E/gGkmj0vP9a0T8sXdwkiYBtwIXAZOAvwJHl+w/FfgU8GZgd+Ae4Ia+fmia4H4GfD099mBgabr768CuwL7A69Lf++6Srx8OPJzG8CXgSkmKiLNImsxfSn//HX2duySGWcCHgNdExFjgBOCxPo57Rfo7PpLGejtwm6ThJYe9DTgRmAH8PXDWQOe2bDiRNU5PrewNwArgqdKdEbEoIv4QEd0R8XuSP7DXDVLmqoj4ekR0RsSWPvZfQpI4fpue7/J+yjkZWBYRN0dEB/BVYE3J/nOAf4uIFWkf3aXAwf3Uyt4B3BERN0RER0Ssi4ilJQn8wojYlPYLXgacWfLdlRHx7YjoAq4BppAk6Up1ASOA/SW1RcRjEfHXPo57O/DTiPhF+ru/DIwCjio55j8iYlVErAduI0nMljNOZI3zXZI/8rPo1awEkHS4pF+mza5nSZLHpEHKfGKgnekf59XAq4DLBuhfmlpaVnpcadl7A19Lm18bgPUkTc9pfZQ1naRG19skoA1YWbJtZa8yXkieEbE5/bhLPzH3KyL+QlLLugR4WtKNkqb2cejU0ngiopvkd/cZE7C5mnis/pzIGiQiVpJ0+p9M0ozr7Xrgx8D0iNgV+BYv9lP1l4AG7PiWNA24GPgOcJmkEf0cupokAfV8T6XrJH/c74+I8SXLqLT52tsTwMv72N4OdJAkxR570atmWoHn2fGGyZ6lOyPi+oh4bXq+AP5PH2WsKo2n5HdXG5NlxImssc4GjouI5/vYNxZYHxFbJR1GUnvrsRboJulbKkv6R3k1cGV63tXA5/s5/KfAAZLenN79PI8dE8O3gAslHZCWvaukt/ZT1nzg9ZLeJmmYpN0kHZw2F28CviBpbNos/Sgv7Sss11LgZEkTJe1JUgMjjW+WpOPSxL0V2EJy/Xq7CThF0vGS2oCPAduAvhK05ZgTWQNFxF8jYkk/uz8IfE7SJuAzJH9kPd/bTNKZ/6u0eXdEGac7D9iDpIM/SDrV3y3pH/qIqx14K/BFYB0wE/hVyf4fkNRobpS0EXgIOKmf3/g4Sa3zYyRN0KXAQenuD5PUpB4B7iWphV5Vxm/py3eB35F04i8Evleyb0T6W9pJmoZ7ABf2EevDwDtJbkK0A/8M/HNEbK8yJsuIPLGimRWda2RmVnhOZGZWeE5kZlZ4TmRmVnhOZGZWeE5kZlZ4TmRmVnhOZGZWeE5kZlZ4TmRmVnhOZGZWeE5kZlZ4TmRmVnhOZGZWeE5kZlZ4TmRmVnhOZGZWeLl6O/WYCcNjwtRRWYeRWxuXt2YdghXcVp5ne2zT4Ef274Rjx8S69V1lHXv/77ctiIgTh3K+cuQqkU2YOopzbzp68AN3UncdOCbrEKzg7os7h1xG+/ou7lvwsrKObZvy18FeaVgTuUpkZlYEQVf09VKq7DiRmVlFAuge+JWqDedEZmYV6+7zNaHZcSIzs4oEQYeblmZWZAF0uWlpZkXnPjIzK7QAusKJzMwKLl89ZE5kZlahINxHZmbFFgEd+cpjTmRmVinRxZAe16w5JzIzq0gA3a6RmVnRuUZmZoWWDIh1IjOzAgugI/I1J2u+ojGz3AtEFy1lLYORdJWkpyU9VLJtoqRfSPpz+u+EwcpxIjOzinWHylrKcDXQewbZTwJ3RsRM4M50fUBOZGZWkZ4+snKWQcuKuBtY32vzqcA16edrgDcNVo77yMysQqKr/D6ySZKWlKzPi4h5g3xnckSsTj+vASYPdhInMjOrSDJDbNmJrD0iZld9roiQNOioNScyM6tIhNgedX2j198kTYmI1ZKmAE8P9gX3kZlZxbpRWUuVfgy8K/38LuBHg33BNTIzq0jS2V+bOpCkG4A5JH1pTwIXA18EbpJ0NrASeNtg5TiRmVmFKursH1BEnN7PruMrKceJzMwqUmFnf0M4kZlZxbrKG+zaME5kZlaRQHREvlJHvqIxs9yrZWd/rTiRmVlFArlpaWbF587+AnjiumGsuqUNAqb+jw6mn9mZdUi5M3vORs75/CpaW4Kf3TCRm74x6ONwO5Vmvj4R1Gz4Ra3UNRpJJ0p6WNJfJA06FUcePPdnseqWNmZfv4XX3LyF9v8/jM2P56sanbWWluDcS5/iojNm8L45szj21A3sNXNr1mHlRrNfn6Szv7WspVHqlsgktQKXAycB+wOnS9q/Xuerlc2PtDDuwC5aR0HLMBg/u4u1d7jiWmrWIZtZ9dhw1jw+gs6OFhb9aDxHnvBs1mHlxs5wfWo1sWKt1PNMhwF/iYhHImI7cCPJPEO5NmZmNxseaKVjA3RtgXX3tLJtjWtkpXbbs4O1q4a/sN6+uo1JUzoyjChfmv36BOVNqljmxIo1Uc+qxjTgiZL1J4HD63i+mhizb7D3ezpYOnckLaNg7H7d0LgaslkhePhFL5LmAnMBxk8ZmXE0ialv7mTqm5MO/r9+rY0Rk3P2Er+MrVvTxu5Tt7+wPmlKB+2r2zKMKF+a/fok77XMVyKrZzRPAdNL1l+WbttBRMyLiNkRMXvMhOG9d2di+7rk362rxdo7hjH5ZN+1LPXw0tFMm7GdydO3MaytmzmnbuA3C3fNOqzcaP7rU9401418ZVw9a2SLgZmSZpAksNOAd9TxfDXzh4+OpGODaBkWvOLT22gbl3VE+dLdJS7/9DQuvf4RWlph4Y0TWfmnfNSm86DZr0/yOrh89bfULZFFRKekDwELSHqZroqIZfU6Xy29+prmuVVeL4vvGsfiu5zh+9PM1ydCuWta1rWPLCJuB26v5znMrPHyNiA2885+MyuWZD6yfA1JciIzswrVbobYWnEiM7OKJMMvXCMzswLredYyT5zIzKxinsbHzAotmcbHTUszKzj3kZlZoSWzX7hpaWYFljyi5ERmZoXmGpmZNQGP7DezQvNdSzNrCm5amlmh9czZnydOZGZWkQA6a1Qjk3QB8N602D8A746IiicEzFf90MwKoTtayloGImkacB4wOyJeRTIB62nVxOMamZlVpravehsGjJLUAYwGVlVTiGtkZlaRnokVy1kGLCfiKeDLwOPAauDZiFhYTUxOZGZWsQpe0DtJ0pKSZW5PGZImkLy0ewYwFRgj6Z3VxOOmpZlVpMKJFdsjYnY/+14PPBoRawEk3QocBVxXaUxOZGZWkUB0dtekMfc4cISk0cAW4HhgSTUFOZGZWcVq8YhSRNwn6WbgAaATeBCYV01ZTmRmVpmo3XxkEXExcPFQy3EiM7OK+OUjZtYUnMjMrNAC0VWbzv6acSIzs4p5PjIzK7SoYWd/rTiRmVnFwonMzIrN85GZWRNwjWwAG5e3cteBY7IOI7cWrFqadQi5d8pRb8w6hFzTU8OHXEYEdHU7kZlZwfmupZkVWuCmpZkVnjv7zawJRGQdwY6cyMysYm5amlmhJXct/aylmRWcm5ZmVnhuWppZoQVyIjOz4stZy9KJzMwqFBB+RMnMis5NSzMrvMLctZT0dQZoCkfEeXWJyMxyrWjPWlb1xl8za3IBFCWRRcQ1peuSRkfE5vqHZGZ5l7em5aDPGUg6UtJy4I/p+kGSvln3yMwsp0R0l7c0SjkPTH0VOAFYBxARvwOOqWNMZpZ3UebSIGXdtYyIJ6QdsmtXfcIxs9yLYnX293hC0lFASGoDzgdW1DcsM8u1ovWRAecA5wLTgFXAwem6me20VObSGIPWyCKiHTijAbGYWVF0Zx3Ajsq5a7mvpNskrZX0tKQfSdq3EcGZWQ71jCMrZxmEpPGSbpb0R0krJB1ZTUjlNC2vB24CpgBTge8DN1RzMjNrDhHlLWX4GvDziNgPOIgq+9/LSWSjI+K7EdGZLtcBI6s5mZk1iRoMv5C0K8lQrisBImJ7RGyoJpyBnrWcmH78maRPAjemob0duL2ak5lZkyh/+MUkSaWPO86LiHnp5xnAWuA7kg4C7gfOj4jnKw1noM7++0kSV0/E7y/ZF8CFlZ7MzJqDyh9+0R4Rs/vZNww4FPhwRNwn6WvAJ4F/rTSegZ61nFFpYWa2EwhBbR4/ehJ4MiLuS9dvJklkFStrZL+kVwH7U9I3FhHXVnNCM2sCNRgQGxFrJD0haVZEPAwcDyyvpqxBE5mki4E5JInsduAk4F7AicxsZ1W7kf0fBuZLGg48Ary7mkLKqZG9heS26IMR8W5Jk4HrqjmZmTWJGiWyiFgK9NeHVrZyhl9siYhuoFPSOOBpYPpQT5xns+ds5Ip7/sh3frWCt33ob1mHkwuXXTCdtx14AHOPnfXCtrtv25X3zZnFidMO4k+/G5VhdPlz/qeWMv+nC7j8ukVZh1J7NRwQWyvlJLIlksYD3ya5k/kA8OvBviTpqvRJgIeGFmJjtbQE5176FBedMYP3zZnFsaduYK+ZW7MOK3P/+Pb1fGH+Izts22e/rXzmisc48IiK75Y3vTtun85nLjg86zDqRlHe0iiDJrKI+GBEbIiIbwFvAN4VEeW0Y68GThxifA0365DNrHpsOGseH0FnRwuLfjSeI094NuuwMnfgEc8zdsKOszftNXMb0/9uW0YR5duypbuxaePwrMOon6LMRybp0IH2RcQDAxUcEXdL2mcIsWVitz07WLvqxf8A21e3sd+hnuHbrFQja1vlGKiz/7IB9gVwXC0CkDQXmAswktG1KNLM6q0oEytGxLGNCCB9XGEewDhNzDzPr1vTxu5Tt7+wPmlKB+2r2zKMyCxnGtxsLEc5nf07lYeXjmbajO1Mnr6NYW3dzDl1A79ZuGvWYZnlS1H6yHZW3V3i8k9P49LrH6GlFRbeOJGVf/JkH//2gb35/a934dn1wzjj1ftz5sfWMHZCF9+8aBrPrhvGv565Ly8/YAuX3vDI4IXtBD7x2fs58JB1jBu/nWt++AvmXzGLhT/ZK+uwakY5m1ixbolM0g0kTwRMkvQkcHFEXFmv89XS4rvGsfiucVmHkSsX/r+VfW4/+iTf0e3Lly5+ddYh1FfOmpblPKIkkqmu942Iz0naC9gzIn470Pci4vQaxWhmOdLoMWLlKKeP7JvAkUBPYtoEXF63iMws/3I2sr+cpuXhEXGopAcBIuKZ9AFPM9tZ5axGVk4i65DUShq6pN3J3TtUzKyR8ta0LCeR/QfwA2APSV8gmQ3jorpGZWb5FQW8axkR8yXdTzLpmYA3RYTfNG62MytajSy9S7kZuK10W0Q8Xs/AzCzHipbIgJ/y4ktIRpK8+eRh4IA6xmVmOVa4PrKIOLB0PZ0V44N1i8jMrEIVj+yPiAckNe+McWY2uKLVyCR9tGS1heQ9dKvqFpGZ5VsR71oCY0s+d5L0md1Sn3DMrBCKVCNLB8KOjYiPNygeM8s5UaDOfknDIqJT0tGNDMjMCqAoiQz4LUl/2FJJPwa+D7zwupyIuLXOsZlZHuVw9oty+shGAutI5ujvGU8WgBOZ2c6qQJ39e6R3LB/ixQTWI2f52MwaqUg1slZgF3ZMYD1y9jPMrKFylgEGSmSrI+JzDYvEzIohh29RGiiR5evFdWaWG0VqWh7fsCjMrFhylsj6nbM/ItY3MhAzKw51l7eUVZbUKulBST+pNh6/oNfMKlPuy3nLr7WdDwxpslYnMjOriCpYBi1LehlwCnDFUGJyIjOzytWuRvZV4BMMcYitE5mZVaznJb2DLcAkSUtKlrkvlCH9E/B0RNw/1HgqnljRzKyC/q/2iJjdz76jgTdKOpnkUchxkq6LiHdWGo5rZGZWmajNXcuIuDAiXhYR+wCnAXdVk8TANTIzq0bOxpE5kZlZxWo9sj8iFgGLqv2+E5mZVc41MqvWKUe9MesQcm/UdzdnHUKutZxdm4nEivSspZnZSwWFmljRzOwlCvXyETOzfjmRmVnRKfKVyZzIzKwyBZsh1sysT+4jM7PCK3fSxEZxIjOzyrlGZmaFVtA3jZuZ7ciJzMyKzANizawpqDtfmcyJzMwq43FkZtYMPPzCzIrPNTIzKzp39ptZsQXgh8bNrOjcR2ZmheZxZGZWfBFuWppZ8blGZmbF50RmZkXnGpmZFVsAXfnKZE5kZlYx18jMrPh819LMis41MjMrNk/jY2ZFJ0A56+xvyToAMyseRZS1DFiGNF3SLyUtl7RM0vnVxuMamZlVpnZNy07gYxHxgKSxwP2SfhERyystyImsD7PnbOScz6+itSX42Q0Tuekbk7MOKVfO/9RSDjv6b2x4ZgTnvnNO1uHk0vbvbaHjJ1tB0LJvKyMvHItGKOuwaqQ2z1pGxGpgdfp5k6QVwDSg4kRWt6ZlLauNjdTSEpx76VNcdMYM3jdnFseeuoG9Zm7NOqxcueP26XzmgsOzDiO3utd2sf2WLYy+Yjxjrp0A3dB557asw6opRXkLMEnSkpJlbp/lSfsAhwD3VRNPPWtkNas2NtKsQzaz6rHhrHl8BACLfjSeI094lsf/PDLjyPJj2dLd2GPPzVmHkW9dwLYgWiG2BprUZN3R5dfI2iNi9kAHSNoFuAX4SERsrCacuiWyWlYbG2m3PTtYu2r4C+vtq9vY71D/0Vr5WnZvZfhpo3juLevRcNF62HCGHTZ88C8WRdTurqWkNpIkNj8ibq22nIb8b2Ko1UazIolN3XTeu50x35vImB9OhC1Bx4Im656IMpcBSBJwJbAiIv59KOHUPZENVm2UNLen/dxB9v0I69a0sfvU7S+sT5rSQfvqtgwjsqLpXNJBy5QWWia0oGFi2OuG0/VQZ9Zh1VQthl8ARwNnAsdJWpouJ1cTT13vWpZTbYyIecA8gHGamPkou4eXjmbajO1Mnr6NdWvamHPqBr547t5Zh2UF0rJHC9uXdRJbA0ZA5/0dtM5qsgECtblreS/J+Nohq9vVrWW1sZG6u8Tln57Gpdc/QksrLLxxIiv/5I7+Up/47P0ceMg6xo3fzjU//AXzr5jFwp/slXVYudF6QBvD5gxn89kboBVaZg6j7Y1N9N9QADvRy0d6qo1/kLQ03fapiLi9juesicV3jWPxXeOyDiO3vnTxq7MOIfdGnD2GEWePyTqMuhBlNRsbqp53LWtWbTSznOnOV5WsyRruZlZ3O1nT0sya1E7TtDSzJuZEZmbF5hf0mlnR+S1KZtYM3EdmZsXnRGZmhRZAtxOZmRWaO/vNrBk4kZlZoQXQla+h/U5kZlahgHAiM7Oic9PSzArNdy3NrCm4RmZmhedEZmaFFgFdXVlHsQMnMjOrnGtkZlZ4TmRmVmzhu5ZmVnAB4QGxZlZ4fkTJzAotwq+DM7Mm4M5+Myu6cI3MzIrNEyuaWdHl8KHxlqwDMLNiCSC6uspaBiPpREkPS/qLpE9WG5MTmZlVJtKJFctZBiCpFbgcOAnYHzhd0v7VhOREZmYVi+4oaxnEYcBfIuKRiNgO3AicWk08TmRmVrka1MiAacATJetPptsqlqvO/k08035H3Lwy6zhKTALasw7iBY9mHcBL5Ov6ALw26wBeIm/XaO+hFrCJZxbcETdPKvPwkZKWlKzPi4h5Q42ht1wlsojYPesYSklaEhGzs44jr3x9BteM1ygiTqxRUU8B00vWX5Zuq5iblmaWlcXATEkzJA0HTgN+XE1BuaqRmdnOIyI6JX0IWAC0AldFxLJqynIiG1jN2/JNxtdncL5GA4iI24Hbh1qOImePGpiZVcp9ZGZWeE5kfajVYxPNStJVkp6W9FDWseSRpOmSfilpuaRlks7POqZm56ZlL+ljE38C3kAyQG8xcHpELM80sByRdAzwHHBtRLwq63jyRtIUYEpEPCBpLHA/8Cb/N1Q/rpG9VM0em2hWEXE3sD7rOPIqIlZHxAPp503ACqocsW7lcSJ7qZo9NmEmaR/gEOC+jENpak5kZnUiaRfgFuAjEbEx63iamRPZS9XssQnbeUlqI0li8yPi1qzjaXZOZC9Vs8cmbOckScCVwIqI+Pes49kZOJH1EhGdQM9jEyuAm6p9bKJZSboB+DUwS9KTks7OOqacORo4EzhO0tJ0OTnroJqZh1+YWeG5RmZmhedEZmaF50RmZoXnRGZmhedEZmaF50RWIJK60lv5D0n6vqTRQyjraklvST9fMdD7BCXNkXRUFed4TNJLXlLR3/ZexzxX4bkukfTxSmO05uBEVixbIuLgdMaJ7cA5pTslVTXjb0S8d5CZGeYAFScys0ZxIiuue4C/S2tL90j6MbBcUquk/ytpsaTfS3o/JKPNJX0jnWftDmCPnoIkLZI0O/18oqQHJP1O0p3pQ8/nABektcF/kLS7pFvScyyWdHT63d0kLUzn4LoC0GA/QtIPJd2ffmdur31fSbffKWn3dNvLJf08/c49kvarydW0QvOc/QWU1rxOAn6ebjoUeFVEPJomg2cj4jWSRgC/krSQZAaGWSSvpp8MLAeu6lXu7sC3gWPSsiZGxHpJ3wKei4gvp8ddD3wlIu6VtBfJUxCvBC4G7o2Iz0k6BShnxP970nOMAhZLuiUi1gFjgCURcYGkz6Rlf4hkDvxzIuLPkg4HvgkcV8VltCbiRFYsoyQtTT/fQ/I831HAbyOi5/W9/wj8fU//F7ArMBM4BrghIrqAVZLu6qP8I4C7e8qKiP7mHHs9sH/ySCEA49KZHo4B3px+96eSninjN50n6V/Sz9PTWNcB3cD30u3XAbem5zgK+H7JuUeUcQ5rck5kxbIlIg4u3ZD+QT9fugn4cEQs6HVcLZ/1awGOiIitfcRSNklzSJLikRGxWdIiYGQ/h0d63g29r4GZ+8iazwLgA+k0Mkh6haQxwN3A29M+tCnAsX189zfAMZJmpN+dmG7fBIwtOW4h8OGeFUkHpx/vBt6RbjsJmDBIrLsCz6RJbD+SGmGPFqCnVvkOkibrRuBRSW9NzyFJBw1yDtsJOJE1nytI+r8eUPJykP8kqXn/APhzuu9aktkrdhARa4G5JM243/Fi0+424F96OvuB84DZ6c2E5bx49/SzJIlwGUkT8/FBYv05MEzSCuCLJIm0x/PAYelvOA74XLr9DODsNL5leBpyw7NfmFkTcI3MzArPiczMCs+JzMwKz4nMzArPiczMCs+JzMwKz4nMzArPiczMCu+/AQzHCBgYzWbEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import confusion_matrix, multilabel_confusion_matrix, classification_report, ConfusionMatrixDisplay\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "iris = load_iris()\n",
    "X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.2, train_size=0.8)\n",
    "knn = KNeighborsClassifier(n_neighbors=2).fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)\n",
    "\n",
    "disp = ConfusionMatrixDisplay.from_predictions(y_test, y_pred)\n",
    "disp.figure_.suptitle(\"Matrix de confusión\")\n",
    "print(f\"Matrix de confusión:\\n{disp.confusion_matrix}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4db109ff-7f41-4921-89d2-8759411d9409",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matriz de confusión múltiple: \n",
      "  [[[21  0]\n",
      "  [ 0  9]]\n",
      "\n",
      " [[17  1]\n",
      "  [ 1 11]]\n",
      "\n",
      " [[20  1]\n",
      "  [ 1  8]]]\n"
     ]
    }
   ],
   "source": [
    "print(\"matriz de confusión múltiple: \\n \", multilabel_confusion_matrix(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "53e75ec6-7550-435f-a51d-881fa0b48543",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "métricas: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00         9\n",
      "           1       0.92      0.92      0.92        12\n",
      "           2       0.89      0.89      0.89         9\n",
      "\n",
      "    accuracy                           0.93        30\n",
      "   macro avg       0.94      0.94      0.94        30\n",
      "weighted avg       0.93      0.93      0.93        30\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"métricas: \\n\", classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e0e88bb-8319-4847-b5c2-18069ba0e4b3",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#4CC9F0\">Especificidad: specificity</span>\n",
    "\n",
    "Si nos damos cuenta, en ningún caso anterior trabajamos con los verdaderos negativos. Eso es porque, en el contexto de hipótesis, el caso nulo no es de \"mucho interés\". sin embargo, existe la métrica de la especificidad, la cual es la proporción de los verdadernos negativos sobre todos los casos negativos.\n",
    "\n",
    "$$\n",
    "\\text{Specificity} = \\frac{\\text{verdaderos negativos}}{\\text{verdaderos negativos}+\\text{falsos positivos}}\n",
    "$$\n",
    "\n",
    "Nótese que el recall y la especificidad explican toda la matriz de confusión"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d5bad85-9c0e-4368-83fa-024b01e3c187",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4361EE\">Curva ROC</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "158bf3d8-e4a2-4e20-aa9c-45a8d659723a",
   "metadata": {},
   "source": [
    "La curva ROC (Receiver Operating Characteristics) es otra manera para evaluar modelos de clasificación binarios, aunque con un poco de creatividad se puede extender a clasificación múltiple.\n",
    "\n",
    "Esta grafica se hace a partir de metricas obtenidas con la matriz de confusión. Aquí hacemos una comparación entre el recall, que también es llamada la rata de verdaderos positivos, y la especificidad, también llamada la rata de verdaderos negativos.\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/1/13/Roc_curve.svg/1200px-Roc_curve.svg.png\" height= 500, width=500/>\n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "\n",
    "gráficamente puede ser dificil de entender, razón por la cual calculamos el \"area bajo la curva\" (Area Under Curve - AUC). Esta métrica Va de 0 a 1. Un modelo que tiene 100% de predicciones acertadas tiene un valor de AUC de 1. \n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://miro.medium.com/max/722/1*pk05QGzoWhCgRiiFbz-oKQ.png\" height= 500, width=500/>\n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e50dfdb3-0e38-4e6a-9ae6-68e3d6e51ee6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score, roc_curve, auc\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.preprocessing import label_binarize\n",
    "from sklearn.svm import SVC\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "X, y = iris.data, iris.target\n",
    "y_bin = label_binarize(y, classes=[0, 1, 2])\n",
    "n_classes = y_bin.shape[1]\n",
    "# shuffle and split training and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y_bin, test_size=.75,\n",
    "                                                    random_state=0)\n",
    "binary_classifier = OneVsRestClassifier(SVC(kernel='linear', probability=True,\n",
    "                                 random_state=13))\n",
    "y_score = binary_classifier.fit(X_train, y_train).decision_function(X_test)\n",
    "\n",
    "fpr = dict()\n",
    "tpr = dict()\n",
    "roc_auc = dict()\n",
    "for i in range(3):\n",
    "    fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])\n",
    "    roc_auc[i] = auc(fpr[i], tpr[i])\n",
    "\n",
    "# Compute micro-average ROC curve and ROC area\n",
    "fpr[\"micro\"], tpr[\"micro\"], _ = roc_curve(y_test.ravel(), y_score.ravel())\n",
    "roc_auc[\"micro\"] = auc(fpr[\"micro\"], tpr[\"micro\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9a56becf-f823-4239-a035-ee51b1e5122b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7lElEQVR4nO3deZxN9RvA8c+DYchWlGRpJD+DsU/KUknJklJSUrKklL1fEqVFoZXSQmijVanIUpESSpJl7OtPpbGTbQizPL8/zplxTTN3ruXeM3Pneb9e9zX33LM958y557nf7/ec7xFVxRhjjMlMHq8DMMYYk71ZojDGGOOXJQpjjDF+WaIwxhjjlyUKY4wxflmiMMYY45clijAhIqtFpLHXcXhNRMaIyBMhXud4ERkaynUGi4jcJSKzTnPesD0GRURF5FKv4/CK2H0UZ5+I/AGUApKBBOBboJeqJngZV7gRkc7AvarayOM4xgPxqvq4x3EMBi5V1Q4hWNd4ssE2h4qIKFBJVTd5HYsXrEQRPDeqamGgFlAbeNTbcE6diOTLjev2ku1zky2pqr3O8gv4A7jOZ/hFYIbP8BXAAmA/sBxo7DPuPOA9YBuwD5jiM64VEOfOtwCokX6dwEXAP8B5PuNqA3uACHf4HmCtu/yZwMU+0yrQE9gI/J7J9t0ErHbj+BGoki6OR4E17vLfAyJPYRsGACuAY0A+YCDwP+CQu8xb3GmrAEc5UWrb734+Hhjqvm8MxAP9gF3AdqCLz/pKANOAg8BvwFDgJz//10Y+/7e/gM4+6xwFzHDj/BWo6DPfq+70B4ElwJU+4wYDnwMfuuPvBeoBv7jr2Q68AeT3maca8B3wN7ATeAxoDhwHEt39sdydthjwjrucre425nXHdQZ+Bl4B9rrjOqfuA0Dccbvc2FYCMUA3dz3H3XVNS3/cA3nduFL/d0uAcpns1wy/D0ADnOO2nDtcE+eYinaHMzw2Mti2/cBmd3md3f/FLqCTz/TjgTHufj0EzOXf34tL3fcFgOHAFnf/jwEKen3eCeo5zesAwvGV7gtT1v2CveoOl3G/lC1xSnRN3eHz3fEzgE+Bc4EI4Gr389ruwX25+yXs5K6nQAbr/AG4zyeel4Ax7vvWwCacE20+4HFggc+06n5Zzsvo4Af+Axx2444AHnGXl98njlVAOXcZP3PixB3INsS58xZ0P7sNJ/nlAdq56y7tjutMuhM7/04UScAzbqwtgSPAue74ie6rEFAV5wSSYaIALsY5gbR3l1UCqOWzzr04J/h8wEfARJ95O7jT58NJWjtwkydOokgEbna3sSBQF+fkmQ+IwknqD7rTF8E56fcDIt3hy32W9WG6uCcDY4FzgAuARcD9PvsvCejtrqsgJyeKZjgn+OI4SaOKz75P28+ZHPf9cY77yu68NYESGezXrL4Pw3CO54Lu8nr5zJvVsZEEdME51obinNhH4Zzor3f/n4V9tucQcJU7/lV8jgVOThSvAFNxju8iOD82nvP6vBPUc5rXAYTjy/3CJLgHngLfA8XdcQOAD9JNPxPnpFkaSME9kaWb5k1gSLrP1nMikfh+Se8FfnDfC84J8Cp3+Bugq88y8uCcPC92hxVo4mfbngA+Szf/Vk78CvwDeMBnfEvgf6ewDfdksW/jgNbu+85knSj+AfL5jN+FcxLOi3OCruwzLtMSBU4paXIm48YDb6fb5nV+tmEfUNN9PxiYl8U2P5i6bpxEtSyT6Qbjkyhw2smO4ZPw3fnn+Oy/LemWkbZPgSbABnd/5clsP6c77lOPwfWp/6csti3T74P7PgInWa3EaeuTUzg2NvqMq45zbJfy+WwvJyd73+ReGKe0mlqaUeBSnO/TYU4uMdYnk9J3uLysjSJ4blbVIjgnq2igpPv5xcBtIrI/9YVTpVEa55f036q6L4PlXQz0SzdfOZxfVOl9AdQXkdI4v5BSgPk+y3nVZxl/4xz8ZXzm/8vPdl0E/Jk6oKop7vSZzf+nT4yBbMNJ6xaRjiIS5zN9DCf2ZSD2qmqSz/ARnJPA+Ti/on3X52+7y+FUc2RmRwbrAEBEHhaRtSJywN2GYpy8Dem3+T8iMl1EdojIQeBZn+mzisPXxTgn2u0++28sTskiw3X7UtUfcKq9RgG7RGSciBQNcN2Bxunv+4CqJuKcxGOAEeqemSGgY2Onz/t/3OWl/6ywz3DavlDnwpO/+ff363ycEugSn/V+634etixRBJmqzsU50Ie7H/2F8wuquM/rHFV93h13nogUz2BRfwHD0s1XSFU/yWCd+4BZOMXxO3F+KanPcu5Pt5yCqrrAdxF+NmkbzpcbABERnJPCVp9pyvm8L+/OE+g2+J4ILgbeAnrhVFsUx6nWkgDizMpunKqJspnEnd5fQMVTXYmIXIlTPXc7TkmxOHCAE9sA/96ON4F1OFfZFMWp60+d/i/gkkxWl345f+GUKEr67O+iqlrNzzwnL1D1NVWti1M19x+cKqUs5yPw/eXv+4CIlAGewmnrGiEiBdzPszo2Tkfa/19ECuNULW1LN80enARTzSfeYupcuBK2LFGExkigqYjUxGm0vFFEmolIXhGJFJHGIlJWVbfjVA2NFpFzRSRCRK5yl/EW8ICIXC6Oc0TkBhEpksk6PwY6Am3d96nGAI+KSDUAESkmIredwrZ8BtwgIteKSAROXfkxnMbIVD1FpKyInAcMwmlzOZ1tOAfnhLTbjbULzq/GVDuBsiKS/xTiB0BVk4EvgcEiUkhEonH2V2Y+Aq4TkdtFJJ+IlBCRWgGsqghOQtoN5BORJ4GsfpUXwWk8TnDj6u4zbjpQWkQeFJECIlJERC53x+0EokQkj7uN23F+MIwQkaIikkdEKorI1QHEjYhc5v6vInCqW47ilE5T15VZwgJ4GxgiIpXc/3UNESmRwXSZfh/cHyHjcRrju+K0zQxx58vq2DgdLUWkkXs8DQEWqupJJS63BP0W8IqIXOCuu4yINDvDdWdrlihCQFV3A+8DT7oHXmucX4m7cX5R9efE/+JunLrzdTj16Q+6y1gM3IdTFbAPpwG5s5/VTgUqATtUdblPLJOBF4CJbrXGKqDFKWzLepzG2ddxfl3diHMp8HGfyT7GOUFtxql+GHo626Cqa4AROFcA7cSpZ/7ZZ5IfcK6+2iEiewLdBh+9cKqBdgAfAJ/gJL2MYtmC0/bQD6dKIg6ngTYrM3GqJjbgVMMdxX8VF8DDOCXBQzgnpdREi6oewmnwvdGNeyNwjTt6kvt3r4gsdd93BPJz4iq0z3GrdQJQ1F3/Pjf2vTgXRoBz8q7qVr9MyWDel3F+VMzCSXrv4DRInySL70MfnGqyJ9wScRegi4hcGcCxcTo+xim9/I1zQUFm96MMwDl2F7rfodk4jfZhy264M2eVODcb3quqs72O5VSJyAvAharayetYTGhJLruB8FRZicLkWiIS7VaJiIjUw6nemOx1XMZkN3YnpsnNiuBUN12EU30xAvjK04iMyYas6skYY4xfVvVkjDHGrxxX9VSyZEmNioryOgxjjMlRlixZskdVT+vGwByXKKKioli8eLHXYRhjTI4iIn9mPVXGrOrJGGOMX5YojDHG+GWJwhhjjF+WKIwxxvhlicIYY4xfliiMMcb4FbREISLvisguEVmVyXgRkddEZJOIrBCROsGKxRhjzOkLZoliPM4D3zPTAqcb7Eo4D2t/M4ixGGOMOU1Bu+FOVeeJSJSfSVoD77v9zC8UkeIiUtp92ErAPv51C1/Fbc16QmOMyWVUla1xc9kaN/eMluPlndllOPkBLvHuZ/9KFCLSDafUQfny5U8a91XcVtZsP0jV0oE+ytcYY8Lf4T3bWfrpCLavXECxMpee0bJyRBceqjoOGAcQGxv7r+5uq5Yuyqf31w95XMYYkx2pKrGxsRzcvJ4RI0bQp08fIiIiTnt5XiaKrZz8MPuy7mfGGGNOw4IFC6hevTpFihTh7bffpmTJkpQrVy7rGbPg5eWxU4GO7tVPVwAHTrV9whhjDOzdu5f77ruPhg0bMmLECABq1659VpIEBLFEISKfAI2BkiISj/PQ8ggAVR0DfI3zsPpNwBGcB6cbY4wJkKry/vvv8/DDD7Nv3z769+9P//79z/p6gnnVU/ssxivQM1jrN8aYcDdgwABeeuklGjRowJgxY6hevXpQ1pMjGrONMcY4/vnnHw4fPkzJkiXp2rUrlSpVomvXruTJE7yWBOvCwxhjcohvv/2WmJgY7r//fgAqV67MfffdF9QkAZYojDEm29u2bRu33347LVq0ICIigl69eoV0/Vb1ZIwx2dj333/PLbfcwvHjxxkyZAj9+/enQIECIY3BEoUxxmRDiYmJREREULNmTVq2bMnQoUO59NIzu8P6dFnVkzHGZCMHDx6kb9++XHnllSQnJ1OyZEkmTpzoWZIASxTGGJMtqCqTJk0iOjqa119/ndjYWI4dO+Z1WIBVPRljjOd2795Np06d+Oabb6hduzZfffUVl112mddhpbEShTHGeKxo0aLs2bOHkSNHsmjRomyVJMAShTHGeGLevHk0a9aMhIQEChQowMKFC+nbty/58mW/ih5LFMYYE0J79uyhS5cuXH311WzYsIE//vgDIOg3zZ2J7BuZMcaEEVXl3XffpXLlynz44Yc8+uijrF69mpiYGK9Dy1L2K+MYY0yY+vDDD6latSpjxoyhWrVqXocTMCtRGGNMkBw5coTHH3+c+Ph4RIQvvviCuXPn5qgkAZYojDEmKL7++muqVavGsGHDmDZtGgDnnntutm6LyEzOi9gYY7Kx+Ph42rZtyw033EDBggWZO3cu3bt39zqsM2KJwhhjzqJhw4YxY8YMnn32WeLi4rjqqqu8DumMWWO2McacoUWLFlGwYEGqV6/O0KFD6d+/P5dcconXYZ01VqIwxpjTdODAAXr27MkVV1zBoEGDAChRokRYJQmwRGGMMadMVZk4cSLR0dGMGTOG3r178+GHH3odVtDkuKqnzbsP027sL2nDa7YfpGrpoh5GZIzJbT788EM6duxIbGws06dPp27dul6HFFQ5LlH8k5h80nDV0kVpXauMR9EYY3KLY8eOsXnzZqpUqcLtt99OUlISHTt2JG/evF6HFnSiql7HcErOu7iK/v3nWq/DMMbkInPmzKF79+4cOXKEjRs3hvxRpGeDiCxR1djTmdfaKIwxJhO7du2iY8eONGnShMTERMaNG5cjk8SZynFVT8YYEwqbNm2iXr16JCQkMGjQIAYNGkTBggW9DssTliiMMcbHwYMHKVq0KBUrVqRr167cc889VKlSxeuwPGVVT8YYAxw+fJgBAwYQFRWV1onfSy+9lOuTBFiJwhhjmDZtGr169WLLli107dqVQoUKeR1StmKJwhiTayUlJXH77bczefJkqlWrxvz582nUqJHXYWU7VvVkjMl1Um8LyJcvH6VLl+b5559n6dKlliQyYYnCGJOrLFy4kNjYWJYuXQrAqFGjGDBgAPnz5/c4suzLEoUxJlfYt28f3bt3p0GDBuzcuZN9+/Z5HVKOEdREISLNRWS9iGwSkYEZjC8vInNEZJmIrBCRlsGMxxiTO3366adER0czbtw4HnzwQdauXcu1117rdVg5RtAas0UkLzAKaArEA7+JyFRVXeMz2ePAZ6r6pohUBb4GooIVkzEmd1q3bh1RUVF8++231K5d2+twcpxglijqAZtUdbOqHgcmAq3TTaNAatevxYBtQYzHGJNLHD16lKeffjrtWdWPPfYYCxYssCRxmoKZKMoAf/kMx7uf+RoMdBCReJzSRO+MFiQi3URksYgsTkxMDEasxpgwMXv2bGrUqMHgwYOZO3cuABEREbmil9dg8boxuz0wXlXLAi2BD0TkXzGp6jhVjVXV2IiIiJAHaYzJ/nbu3Mldd91F06ZNUVVmzZrF8OHDvQ4rLAQzUWwFyvkMl3U/89UV+AxAVX8BIoGSQYzJGBOmvvvuOz7//HOefPJJVq5cSdOmTb0OKWwE887s34BKIlIBJ0HcAdyZbpotwLXAeBGpgpModgcxJmNMGFm+fDkbN26kbdu23HXXXTRs2JAKFSp4HVbYCVqJQlWTgF7ATGAtztVNq0XkGRG5yZ2sH3CfiCwHPgE6a057kpIxJuQSEhLo168fdevWZeDAgSQlJSEiliSCxJ5wZ4zJUaZMmULv3r2Jj4+nW7duPPfcc5x33nleh5XtnckT7qxTQGNMjrFy5UpuueUWqlevzqeffkqDBg28DilX8PqqJ2OM8SsxMZEffvgBgOrVqzNjxgyWLFliSSKELFEYY7KtBQsWULduXZo2bcqmTZsAaNmyJXaZfGhZojDGZDt///033bp1o2HDhuzfv58vv/ySSy+91Ouwci1rozDGZCtHjx6lVq1abNu2jX79+jF48GAKFy7sdVi5miUKY0y2EB8fT9myZYmMjGTIkCHUqlWLmjVreh2WwaqejDEe++eff3jyySepWLFiWid+nTp1siSRjViJwhjjmVmzZtGjRw/+97//0aFDB+rVq+d1SCYDAZcoRKRQMAMxxuQuvXv3plmzZuTJk4fZs2fzwQcfUKpUKa/DMhnIskQhIg2At4HCQHkRqQncr6o9gh2cMSa8JCcnA5A3b16uuOIKSpYsyYABA4iMjPQ4MuNPICWKV4BmwF4AVV0OXBXMoIwx4Wfp0qXUr1+f0aNHA3DXXXfx1FNPWZLIAQKqelLVv9J9lByEWIwxYejQoUP897//5bLLLmPLli2ULl3a65DMKQqkMfsvt/pJRSQC6IvTG6wxxvg1a9Ys7rnnHrZt28YDDzzAs88+S/Hixb0Oy5yiQBLFA8CrOI8x3QrMAqx9whiTpfz583PBBRfwxRdfcPnll3sdjjlNWXYzLiINVfXnrD4LFetm3JjsKzExkZdffpmDBw8ybNgwAFJSUsiTx27Z8tqZdDMeyH/v9QA/M8bkYj/99BO1a9dm4MCBbNy4kZSUFABLEmEg06onEakPNADOF5GHfEYVBfIGOzBjTM6wd+9eBgwYwDvvvEP58uWZNm0arVq18joscxb5S/X5ce6dyAcU8XkdBNoGPzRjTE6wd+9eJk6cyCOPPMKaNWssSYShQNooLlbVP0MUT5asjcIY761du5bPPvuMp556CnC6BbfHkWZvwW6jOCIiL4nI1yLyQ+rrdFZmjMnZjhw5wqBBg6hZsyavvvoq8fHxAJYkwlwgieIjYB1QAXga+AP4LYgxGWOyoW+//ZaYmBieffZZ7rzzTtavX0/ZsmW9DsuEQCD3UZRQ1XdEpK+qzgXmioglCmNykYSEBO6++25KlCjBnDlzaNy4sdchmRAKpESR6P7dLiI3iEhtwMqZxoS55ORkPvzwQ5KTkylcuDCzZ89m+fLlliRyoUBKFENFpBjQD+f+iaLAg8EMyhjjrSVLlnD//fezZMkSChYsyK233moPEsrFsixRqOp0VT2gqqtU9RpVrQv8HYLYjDEhduDAAfr06UO9evXYunUrEydOpE2bNl6HZTzm74a7vMDtOH08fauqq0SkFfAYUBCoHZoQjTGhcuutt/LDDz/Qs2dPhg4dSrFixbwOyWQD/qqe3gHKAYuA10RkGxALDFTVKSGIzRgTAps3b+b888+nSJEiDBs2jDx58nDZZZd5HZbJRvxVPcUCTVX1UaAl0ApoaEnCmPBw/Phxnn32WapVq8bQoUMBuPzyyy1JmH/xV6I4rqopAKp6VEQ2q+reEMVljAmiefPm8cADD7B27Vratm1Lnz59vA7JZGP+EkW0iKxw3wtQ0R0WQFW1RtCjM8acda+88goPPfQQUVFRzJgxg5YtW3odksnm/CWKKiGLwhgTVCkpKRw+fJgiRYpwww03sHv3bh5//HEKFSrkdWgmB8iyU8DsxjoFNObUrF69mgceeCDtSXMmdwp2p4CnTUSai8h6EdkkIgMzmeZ2EVkjIqtF5ONgxmNMbnLkyBEeffRRatWqxdq1a2nVqhU57YehyR4CuTP7tLj3YYwCmgLxwG8iMlVV1/hMUwl4FOdqqn0ickGw4jEmN1m2bBlt2rThjz/+oEuXLrz44ouULFnS67BMDhVQohCRgkB5VV1/CsuuB2xS1c3uMiYCrYE1PtPcB4xS1X0AqrrrFJZvjElHVRERypcvT/ny5ZkwYQJXXXWV12GZHC7LqicRuRGIA751h2uJyNQAll0G+MtnON79zNd/gP+IyM8islBEmgcUtTHmJElJSYwcOZJrr72W5ORkSpQowdy5cy1JmLMikDaKwTilg/0AqhqH82yKsyEfUAloDLQH3hKR4uknEpFuIrJYRBYnJiamH21MrrZo0SLq1avHf//7XyIjIzl48KDXIZkwE1A346p6IN1ngbSIbcXpAiRVWfczX/HAVFVNVNXfgQ04iePklamOU9VYVY2NiIgIYNXGhL+EhAR69uzJFVdcwc6dO5k0aRIzZszg3HPP9To0E2YCSRSrReROIK+IVBKR14EFAcz3G1BJRCqISH7gDiB9ldUUnNIEIlISpypqc4CxG5OrRURE8OOPP9K7d++0O6xFxOuwTBgKJFH0BqoBx4CPgQME8DwKVU0CegEzgbXAZ6q6WkSeEZGb3MlmAntFZA0wB+hv3YQYk7lNmzbRsWNHDh06RIECBViyZAmvvvoqRYsW9To0E8ayvOFOROqo6tIQxZMlu+HO5EbHjh3jxRdfZNiwYeTPn58ZM2Zw5ZVXeh2WyUGCfcPdCBFZKyJDRCTmdFZijDl9c+bMoWbNmjz55JPcfPPNrFu3zpKECaks76NQ1WtE5EKchxiNFZGiwKeqOjTo0RmTy6kqw4YNIzExkW+//ZZmzZp5HZLJhU6prycRqQ48ArRT1fxBi8oPq3oy4S4lJYV33nmH5s2bU65cObZv307x4sUpWLCg16GZHCyoVU8iUkVEBovISiD1iqeyp7MyY4x/K1asoFGjRnTr1o23334bgNKlS1uSMJ4KpAuPd4FPgWaqui3I8RiTKyUkJPD000/zyiuvcO655zJ+/Hg6duzodVjGAIG1UdQPRSDG5GaDBw9mxIgR3HvvvTz//POUKFHC65CMSZNpG4WIfKaqt7tVTr4TefqEO2ujMOHir7/+4vDhw0RHR7Nnzx7WrVtHo0aNvA7LhKkzaaPwlyhKq+p2Ebk4o/Gq+ufprPBMWaIwOV1SUhKvvfYaTz75JHXr1mXu3Lleh2RygaA0ZqvqdvdtD1X90/cF9DidlRmT2y1cuJDY2Fj69etH48aNmTBhgtchGZOlQG64a5rBZy3OdiDGhLsZM2bQoEED9uzZw5dffsm0adOIioryOixjspRpohCR7m77RGURWeHz+h1YEboQjcm5VJWtW51Ok6+77jqeeeYZ1q5dyy233GId+Jkcw18bRTHgXOA5wPd514dU9e8QxJYha6MwOcWGDRvo0aMHGzZsYM2aNRQuXNjrkEwuFqwb7lRV/wB6Aod8XojIeaezMmNyg6NHjzJ48GCqV6/O4sWLefTRR+2GOZOj+buP4mOgFbAE5/JY33KyApcEMS5jcqQdO3Zw1VVXsXHjRtq3b8/LL7/MhRde6HVYxpyRTBOFqrZy/56tx54aE7YSExOJiIigVKlSXHXVVYwaNYqmTTO6DsSYnCeQvp4aisg57vsOIvKyiJQPfmjGZH8pKSmMGTOGihUrEh8fj4jw9ttvW5IwYSWQy2PfBI6ISE2gH/A/4IOgRmVMDrB8+XIaNGhA9+7dqVSpEomJiV6HZExQBJIoktS5NKo18IaqjgKKBDcsY7IvVeXhhx+mbt26bN68mQ8++IDZs2dToYLV0prwFEiiOCQijwJ3AzNEJA8QEdywjMm+RIR9+/bRtWtX1q9fT4cOHeyeCBPWAkkU7YBjwD2qugPnWRQvBTUqY7KZP//8k5tvvpmlS53Hx7/11luMHTuWc8891+PIjAm+LBOFmxw+AoqJSCvgqKq+H/TIjMkGEhMTefHFF6latSrfffcd69evByBPnkB+YxkTHgK56ul2YBFwG85zs38VkbbBDswYry1YsIA6deowYMAAmjZtytq1a2nfvr3XYRkTcoE84W4QcJmq7gIQkfOB2cDnwQzMGK/Nnj2bAwcOMGXKFFq3bu11OMZ4JtO+ntImEFmpqtV9hvMAy30/CyXr68kEi6rywQcfcP7559OiRQuOHTtGYmKi9dFkwkKw+npK9a2IzBSRziLSGZgBfH06KzMmu1q3bh1NmjShU6dOvPfeewAUKFDAkoQxBNaY3R8YC9RwX+NUdUCwAzMmFP755x+eeOIJatSoQVxcHGPHjmXixIleh2VMtpJpG4WIVAKGAxWBlcDDqro1VIEZEwrTpk1j6NChdOjQgeHDh1OqVCmvQzIm2/HXmP0u8D4wD7gReB1oE4qgjAmmHTt2EBcXR/PmzbntttuIioqiXr16XodlTLblL1EUUdW33PfrRWRpKAIyJliSk5MZO3Ysjz76KPnz52fLli0ULFjQkoQxWfDXRhEpIrVFpI6I1AEKphs2JsdYunQp9evXp2fPntSrV48FCxbYw4SMCZC/EsV24GWf4R0+wwo0CVZQxpxNv//+O/Xq1aNkyZJ8/PHH3HHHHdY3kzGnwN+Di64JZSDGnE2qysqVK6lRowYVKlTgvffe48Ybb6R48eJeh2ZMjmMd1piw8/vvv9OqVStq167NihUrALj77rstSRhzmoKaKESkuYisF5FNIjLQz3S3ioiKyGndNWgMwPHjx3n++eepVq0ac+fOZfjw4VStWtXrsIzJ8QLp6+m0iEheYBTQFIgHfhORqaq6Jt10RYC+wK/BisWEv+TkZBo0aMCSJUto06YNI0eOpFy5cl6HZUxYCKT3WHGflf2kO1xeRAK5nrAesElVN6vqcWAizlPy0hsCvAAcPYW4jQHg4MGDAOTNm5d77rmHadOm8cUXX1iSMOYsCqTqaTRQH0jtX/kQTkkhK2WAv3yG493P0riX2ZZT1Rn+FiQi3URksYgstucSG3Aaq8ePH88ll1zCV199BUCPHj1o1aqVx5EZE34CSRSXq2pP3F/8qroPyH+mK3Z7oX0Z6JfVtKo6TlVjVTU2IsKewprbrVmzhsaNG9OlSxeio6OpWLGi1yEZE9YCSRSJbnuDQtrzKFICmG8r4Fv+L+t+lqoIEAP8KCJ/AFcAU61B2/jz4osvUrNmTVatWsXbb7/NvHnziImJ8TosY8JaIIniNWAycIGIDAN+Ap4NYL7fgEoiUkFE8gN3AFNTR6rqAVUtqapRqhoFLARuUtXFp7oRJvylPjflwgsv5K677mLdunV07drVHklqTAhk+eAiABGJBq4FBPheVQN6cpCItARGAnmBd1V1mIg8AyxW1anppv0Rp4dav4nCHlyUu2zbto2+ffty5ZVX0qdPH6/DMSbHOpMHF2V5eayIlAeOANN8P1PVLVnNq6pfk+4hR6r6ZCbTNs5qeSb3SE5OZvTo0QwaNIjExEQaNGjgdUjG5FqB3EcxA6d9QoBIoAKwHqgWxLhMLhYXF8e9997LkiVLuP766xk9erQ1WBvjoSwTRfpnY7uXtPYIWkQm1ztw4ADbtm3j008/5bbbbrMO/Izx2Cnfma2qS0Xk8mAEY3InVWXSpEls3LiRQYMGcfXVV7N582YiIyO9Ds0YQ2BtFA/5DOYB6gDbghaRyVX+97//0atXL7799lsuu+wyHnnkESIiIixJGJONBHJtYRGfVwGcNouMuuIwJmDHjh1j2LBhxMTE8PPPP/Pqq6+yYMEC7IZKY7IfvyUK90a7Iqr6cIjiMbnEX3/9xZAhQ7jxxhsZOXIkZcqUyXomY4wnMi1RiEg+VU0GGoYwHhPGdu/ezRtvvAHApZdeypo1a5g0aZIlCWOyOX9VT4vcv3EiMlVE7haRNqmvUARnwkNKSgrvvPMO0dHRPPTQQ6xfvx6ASy65xOPIjDGBCKSNIhLYi/OM7FbAje5fY7K0atUqrr76au69916qVatGXFwclStX9josY8wp8NdGcYF7xdMqTtxwlyrrfj9Mrnf8+HGuv/56jh8/zrvvvkvnzp3tnghjciB/iSIvUJiTE0QqSxQmUz/88ANXX301+fPn57PPPiM6OpqSJUt6HZYx5jT5SxTbVfWZkEVicrz4+Hj69u3Ll19+ybvvvkuXLl1o1KiR12EZY86QvzYKqyMwAUlKSmLkyJFUqVKFb775hueee4677rrL67CMMWeJvxLFtSGLwuRod999NxMnTqRFixaMGjWKChUqeB2SMeYsCuh5FNmJPY8ie9i/fz/58uWjcOHC/PTTT+zYsYNbb73VGquNyabO5HkU9ngwc0pUlYkTJ1KlShWeeOIJABo1akTbtm0tSRgTpixRmIBt2rSJZs2a0b59e8qWLUuHDh28DskYEwKWKExAPv74Y2JiYvj111954403WLhwIXXr1vU6LGNMCJzy8yhM7pKYmEhERASxsbG0bduWF198kYsuusjrsIwxIWSN2SZDu3btol+/fhw+fJgvv/zS63CMMWfIGrPNWZOSksK4ceOoXLkyn376KdWqVSM5OdnrsIwxHrKqJ5Nm8+bNdOjQgV9++YXGjRvz5ptvEh0d7XVYxhiPWaIwaYoVK8b+/fuZMGECd999t13uaowBrOop15s6dSpt2rQhOTmZEiVKsGrVKjp27GhJwhiTxhJFLrVlyxZuvvlmWrduzYYNG9i+fTsAefLYIWGMOZmdFXKZpKQkhg8fTpUqVZg1axYvvPACy5Yto2zZsl6HZozJpqyNIpdJTk7m7bffpkmTJrz++utERUV5HZIxJpuzEkUusG/fPgYMGMChQ4coUKAAP//8M1OnTrUkYYwJiCWKMKaqfPTRR0RHRzNixAjmzJkDQIkSJayx2hgTMEsUYWrDhg00bdqUDh06EBUVxeLFi7npppu8DssYkwNZG0WYevDBB1m8eDGjR4+mW7du5M2b1+uQjDE5lCWKMPLdd98RHR1NuXLlePPNNylQoAAXXnih12EZY3K4oFY9iUhzEVkvIptEZGAG4x8SkTUiskJEvheRi4MZT7jasWMHd955J9dffz0vvPACABdffLElCWPMWRG0RCEieYFRQAugKtBeRKqmm2wZEKuqNYDPgReDFU84SklJYcyYMURHR/PFF1/w1FNPMXz4cK/DMsaEmWCWKOoBm1R1s6oeByYCrX0nUNU5qnrEHVwI2F1fp+C5556je/fu1K1blxUrVjB48GAiIyO9DssYE2aC2UZRBvjLZzgeuNzP9F2BbzIaISLdgG4AhUtXPFvx5UiHDh1iz549VKhQgQceeIAKFSrQvn17u9zVGBM02eLyWBHpAMQCL2U0XlXHqWqsqsZGRESENrhsQlWZPHkyVatWpV27dqgqJUqU4M4777QkYYwJqmAmiq1AOZ/hsu5nJxGR64BBwE2qeiyI8eRYf/75JzfddBNt2rThvPPO47XXXrPkYIwJmWBWPf0GVBKRCjgJ4g7gTt8JRKQ2MBZorqq7ghhLjvXLL79w3XXXATB8+HD69u1Lvnx2VbMxJnSCVqJQ1SSgFzATWAt8pqqrReQZEUm9RfgloDAwSUTiRGRqsOLJaQ4ePAhAnTp1uOeee1i7di39+vWzJGGMCTlRVa9jOCXnXVxF//5zrddhBM3evXsZOHAgs2bNYvXq1RQuXNjrkIwxYUBElqhq7OnMmy0as43TWP3+++8THR3Ne++9R7t27awdwhiTLVg9RjZw4MABbr75Zn788Ufq16/PmDFjqFGjhtdhGWMMYInCU6qKiFC0aFFKlizJuHHj6Nq1qz2O1BiTrdgZySMzZ86kTp06xMfHIyJMmjSJ++67z5KEMSbbsbNSiG3fvp077riD5s2bc+TIEXbtsquCjTHZmyWKEBo1ahTR0dFMmTKFp59+mhUrVlCnTh2vwzLGGL+sjSKElixZwuWXX86oUaOoVKmS1+EYY0xArEQRRAcPHuTBBx9kyZIlAIwePZqZM2dakjDG5CiWKIJAVfn888+pUqUKr732GnPnzgUgMjLS7o0wxuQ4lijOst9//51WrVpx2223ccEFF/DLL7/w0EMPeR2WMcacNksUZ9lHH33EvHnzeOWVV/jtt9+4/HJ/j+Awxpjsz/p6Ogvmz5/PsWPHuO666zh27Bi7d++mbFl7WJ8xJvuwvp48smfPHu655x6uuuoqnnnmGQAKFChgScIYE1bs8tjToKqMHz+e/v37c+DAAQYMGMATTzzhdVgmm0lMTCQ+Pp6jR496HYrJRSIjIylbtixn82mglihOw9dff80999xDw4YNGTNmDDExMV6HZLKh+Ph4ihQpQlRUlF3tZkJCVdm7dy/x8fFUqFDhrC3Xqp4CdOTIEX7++WcAWrZsyVdffcW8efMsSZhMHT16lBIlSliSMCEjIpQoUeKsl2ItUQTgm2++ISYmhhYtWrB//35EhJtuusk68DNZsiRhQi0Yx5yd6fzYunUrt912Gy1btqRAgQJMmzaN4sWLex2WMcaElCWKTOzatYuqVasyffp0hg4dyvLly7n66qu9DsuYU5I3b15q1apFTEwMN954I/v3708bt3r1apo0aULlypWpVKkSQ4YMwfdy+W+++YbY2FiqVq1K7dq16devnwdb4N+yZcvo2rWr12Fkat68edSpU4d8+fLx+eefZzrdkiVLqF69Opdeeil9+vRJ+z/8/fffNG3alEqVKtG0aVP27dsHwPTp03nyySdDsg2A0/iRk17nlo/WYIqPj097/+qrr+qmTZuCuj4TvtasWeN1CHrOOeekve/YsaMOHTpUVVWPHDmil1xyic6cOVNVVQ8fPqzNmzfXN954Q1VVV65cqZdccomuXbtWVVWTkpJ09OjRZzW2xMTEM15G27ZtNS4uLqTrPBW///67Ll++XO+++26dNGlSptNddtll+ssvv2hKSoo2b95cv/76a1VV7d+/vz733HOqqvrcc8/pI488oqqqKSkpWqtWLT18+HCGy8vo2AMW62med+2qJ9eBAwd4/PHHGTt2LAsXLqROnTr06dPH67BMmHh62mrWbDt4VpdZ9aKiPHVjtYCnr1+/PitWrADg448/pmHDhlx//fUAFCpUiDfeeIPGjRvTs2dPXnzxRQYNGkR0dDTglEy6d+/+r2UmJCTQu3dvFi9ejIjw1FNPceutt1K4cGESEhIA+Pzzz5k+fTrjx4+nc+fOREZGsmzZMho2bMiXX35JXFxcWpVupUqV+Omnn8iTJw8PPPAAW7ZsAWDkyJE0bNjwpHUfOnSIFStWULNmTQAWLVpE3759OXr0KAULFuS9996jcuXKjB8/ni+//JKEhASSk5P5+uuv6d27N6tWrSIxMZHBgwfTunVr/vjjD+6++24OHz4MwBtvvEGDBg0C3r8ZiYqKAvDbnrl9+3YOHjzIFVdcAUDHjh2ZMmUKLVq04KuvvuLHH38EoFOnTjRu3JgXXngBEaFx48ZMnz6d22+//YxiDESuTxSqyqRJk3jwwQfZsWMHvXr1omLFil6HZcxZlZyczPfff59WTbN69Wrq1q170jQVK1YkISGBgwcPsmrVqoCqmoYMGUKxYsVYuXIlQFrViD/x8fEsWLCAvHnzkpyczOTJk+nSpQu//vorF198MaVKleLOO+/kv//9L40aNWLLli00a9aMtWtP7pFh8eLFJ111GB0dzfz588mXLx+zZ8/mscce44svvgBg6dKlrFixgvPOO4/HHnuMJk2a8O6777J//37q1avHddddxwUXXMB3331HZGQkGzdupH379ixevPhf8V955ZUcOnToX58PHz6c6667LsvtT2/r1q0n3aRbtmxZtm7dCsDOnTspXbo0ABdeeCE7d+5Mmy42Npb58+dbogg2VaVNmzZMmTKFOnXqMHXqVGJjT+sOd2P8OpVf/mfTP//8Q61atdi6dStVqlShadOmZ3X5s2fPZuLEiWnD5557bpbz3HbbbeTNmxeAdu3a8cwzz9ClSxcmTpxIu3bt0pa7Zs2atHkOHjxIQkIChQsXTvts+/btnH/++WnDBw4coFOnTmzcuBERITExMW1c06ZNOe+88wCYNWsWU6dOZfjw4YBzGfOWLVu46KKL6NWrF3FxceTNm5cNGzZkGP/8+fOz3MZgEJGTrmi64IIL2LZtW0jWnSsTRWJiIhEREYgIjRo1okmTJvTo0SPt4DUmXBQsWJC4uDiOHDlCs2bNGDVqFH369KFq1arMmzfvpGk3b95M4cKFKVq0KNWqVWPJkiVp1TqnyveElv6a/nPOOSftff369dm0aRO7d+9mypQpPP744wCkpKSwcOFCIiMj/W6b77KfeOIJrrnmGiZPnswff/xB48aNM1ynqvLFF19QuXLlk5Y3ePBgSpUqxfLly0lJScl03We7RFGmTBni4+PThuPj4ylTpgwApUqVYvv27ZQuXZrt27dzwQUXpE2XWsUWCrnuqqcff/yRGjVq8NVXXwHQr18/evfubUnChLVChQrx2muvMWLECJKSkrjrrrv46aefmD17NuCUPPr06cMjjzwCQP/+/Xn22WfTflWnpKQwZsyYfy23adOmjBo1Km04teqpVKlSrF27lpSUFCZPnpxpXCLCLbfcwkMPPUSVKlUoUaIEANdffz2vv/562nRxcXH/mrdKlSps2rQpbfjAgQNpJ9jx48dnus5mzZrx+uuvp11ZtGzZsrT5S5cuTZ48efjggw9ITk7OcP758+cTFxf3r9fpJAmA0qVLU7RoURYuXIiq8v7779O6dWsAbrrpJiZMmADAhAkT0j4H2LBhQ8hu+M01iWL37t106tSJa665hmPHjlGkSBGvQzImpGrXrk2NGjX45JNPKFiwIF999RVDhw6lcuXKVK9encsuu4xevXoBUKNGDUaOHEn79u2pUqUKMTExbN68+V/LfPzxx9m3bx8xMTHUrFmTOXPmAPD888/TqlUrGjRokFbHnpl27drx4YcfplU7Abz22mssXryYGjVqULVq1QyTVHR0NAcOHEj7df/II4/w6KOPUrt2bZKSkjJd3xNPPEFiYiI1atSgWrVqaf209ejRgwkTJlCzZk3WrVt3UinkdP3222+ULVuWSZMmcf/991Ot2okqyFq1aqW9Hz16NPfeey+XXnopFStWpEWLFgAMHDiQ7777jkqVKjF79mwGDhyYNs+cOXO44YYbzjjGQOSKbsY/+eQTevbsSUJCAv3792fQoEEUKlQoSBEa41i7di1VqlTxOoyw9sorr1CkSBHuvfder0MJqZ07d3LnnXfy/fffZzg+o2PPuhnPQlJSEjExMcTFxTFs2DBLEsaEie7du1OgQAGvwwi5LVu2MGLEiJCtLyxLFIcPH2bIkCGUL1+eHj16pNVFWr87JpSsRGG8YiWKLEyfPp1q1arxwgsvpDXEpb+szJhQyWk/xEzOF4xjLmwSRXx8PG3atOHGG2/knHPOYd68eYwcOdLrsEwuFhkZyd69ey1ZmJBR93kU/i4rPh1hcx/F5s2bmTlzJs899xwPPfQQ+fPn9zokk8uVLVuW+Ph4du/e7XUoJhdJfcLd2ZSj2ygWLVrEL7/8Qt++fQHYu3dv2nXYxhhjTsi2bRQi0lxE1ovIJhEZmMH4AiLyqTv+VxGJCmS5+/fvp0ePHlxxxRW8/PLLaZ14WZIwxpizL2iJQkTyAqOAFkBVoL2IVE03WVdgn6peCrwCvJDVco8fOUB0dDRjx46lT58+rFy58qzcGGOMMSZjwSxR1AM2qepmVT0OTARap5umNTDBff85cK1kcXnS4T07KFeuHL/99hsjR46kaNGiZz1wY4wxJwSzMbsM8JfPcDxweWbTqGqSiBwASgB7fCcSkW5AN3fw2OLFi1el7yI5lypJun2Vi9m+OMH2xQm2L06onPUkGcsRVz2p6jhgHICILD7dBplwY/viBNsXJ9i+OMH2xQki8u+HawQomFVPW4FyPsNl3c8ynEZE8gHFgL1BjMkYY8wpCmai+A2oJCIVRCQ/cAcwNd00U4FO7vu2wA+a067XNcaYMBe0qie3zaEXMBPIC7yrqqtF5Bmch3xPBd4BPhCRTcDfOMkkK+OCFXMOZPviBNsXJ9i+OMH2xQmnvS9y3A13xhhjQits+noyxhgTHJYojDHG+JVtE0Wwuv/IiQLYFw+JyBoRWSEi34vIxV7EGQpZ7Quf6W4VERWRsL00MpB9ISK3u8fGahH5ONQxhkoA35HyIjJHRJa535OWXsQZbCLyrojsEpFVmYwXEXnN3U8rRKROQAtW1Wz3wmn8/h9wCZAfWA5UTTdND2CM+/4O4FOv4/ZwX1wDFHLfd8/N+8KdrggwD1gIxHodt4fHRSVgGXCuO3yB13F7uC/GAd3d91WBP7yOO0j74iqgDrAqk/EtgW8AAa4Afg1kudm1RBGU7j9yqCz3harOUdUj7uBCnHtWwlEgxwXAEJx+w46GMrgQC2Rf3AeMUtV9AKq6K8Qxhkog+0KB1P5+igHbQhhfyKjqPJwrSDPTGnhfHQuB4iJSOqvlZtdEkVH3H2Uym0ZVk4DU7j/CTSD7wldXnF8M4SjLfeEWpcup6oxQBuaBQI6L/wD/EZGfRWShiDQPWXShFci+GAx0EJF44Gugd2hCy3ZO9XwC5JAuPExgRKQDEAtc7XUsXhCRPMDLQGePQ8ku8uFUPzXGKWXOE5Hqqrrfy6A80h4Yr6ojRKQ+zv1bMaqa4nVgOUF2LVFY9x8nBLIvEJHrgEHATap6LESxhVpW+6IIEAP8KCJ/4NTBTg3TBu1Ajot4YKqqJqrq78AGnMQRbgLZF12BzwBU9RcgEqfDwNwmoPNJetk1UVj3HydkuS9EpDYwFidJhGs9NGSxL1T1gKqWVNUoVY3Caa+5SVVPuzO0bCyQ78gUnNIEIlISpypqcwhjDJVA9sUW4FoAEamCkyhy4zNqpwId3aufrgAOqOr2rGbKllVPGrzuP3KcAPfFS0BhYJLbnr9FVW/yLOggCXBf5AoB7ouZwPUisgZIBvqratiVugPcF/2At0TkvzgN253D8YeliHyC8+OgpNse8xQQAaCqY3DaZ1oCm4AjQJeAlhuG+8oYY8xZlF2rnowxxmQTliiMMcb4ZYnCGGOMX5YojDHG+GWJwhhjjF+WKEy2JCLJIhLn84ryM23CWVjfeBH53V3XUvfu3VNdxtsiUtV9/1i6cQvONEZ3Oan7ZZWITBOR4llMXytce0o1oWOXx5psSUQSVLXw2Z7WzzLGA9NV9XMRuR4Yrqo1zmB5ZxxTVssVkQnABlUd5mf6zjg96PY627GY3MNKFCZHEJHC7rM2lorIShH5V6+xIlJaROb5/OK+0v38ehH5xZ13kohkdQKfB1zqzvuQu6xVIvKg+9k5IjJDRJa7n7dzP/9RRGJF5HmgoBvHR+64BPfvRBG5wSfm8SLSVkTyishLIvKb+5yA+wPYLb/gdugmIvXcbVwmIgtEpLJ7l/IzQDs3lnZu7O+KyCJ32ox63zXmZF73n24ve2X0wrmTOM59TcbpRaCoO64kzp2lqSXiBPdvP2CQ+z4vTt9PJXFO/Oe4nw8AnsxgfeOBtu7724BfgbrASuAcnDvfVwO1gVuBt3zmLeb+/RH3+RepMflMkxrjLcAE931+nJ48CwLdgMfdzwsAi4EKGcSZ4LN9k4Dm7nBRIJ/7/jrgC/d9Z+ANn/mfBTq474vj9P90jtf/b3tl71e27MLDGOAfVa2VOiAiEcCzInIVkILzS7oUsMNnnt+Ad91pp6hqnIhcjfOgmp/d7k3y4/wSz8hLIvI4Th9AXXH6BpqsqofdGL4ErgS+BUaIyAs41VXzT2G7vgFeFZECQHNgnqr+41Z31RCRtu50xXA68Ps93fwFRSTO3f61wHc+008QkUo4XVREZLL+64GbRORhdzgSKO8uy5gMWaIwOcVdwPlAXVVNFKd32EjfCVR1nptIbgDGi8jLwD7gO1VtH8A6+qvq56kDInJtRhOp6gZxnnvREhgqIt+r6jOBbISqHhWRH4FmQDuch+yA88Sx3qo6M4tF/KOqtUSkEE7fRj2B13Ae1jRHVW9xG/5/zGR+AW5V1fWBxGsMWBuFyTmKAbvcJHEN8K/ngovzrPCdqvoW8DbOIyEXAg1FJLXN4RwR+U+A65wP3CwihUTkHJxqo/kichFwRFU/xOmQMaPnDie6JZuMfIrTGVtq6QSck3731HlE5D/uOjOkzhMN+wD95EQ3+6ndRXf2mfQQThVcqplAb3GLV+L0PGyMX5YoTE7xERArIiuBjsC6DKZpDCwXkWU4v9ZfVdXdOCfOT0RkBU61U3QgK1TVpThtF4tw2izeVtVlQHVgkVsF9BQwNIPZxwErUhuz05mF83Cp2eo8uhOcxLYGWCoiq3C6jfdb4ndjWYHzUJ4Xgefcbfedbw5QNbUxG6fkEeHGttodNsYvuzzWGGOMX1aiMMYY45clCmOMMX5ZojDGGOOXJQpjjDF+WaIwxhjjlyUKY4wxflmiMMYY49f/AdC5Sx8UzMNLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot of a ROC curve for a specific class\n",
    "plt.figure()\n",
    "plt.plot(fpr[2], tpr[2], label='ROC curve (area = %0.2f)' % roc_auc[2])\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic example')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0214cef5-fb59-4a46-873b-b82caee8bce5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEWCAYAAAAkUJMMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABagElEQVR4nO3dd3gU1dfA8e+hht5BJPQAIY3QmxRBEBVBkSIWELBSFURQUZEmAiIo+FJUEEFBsNARURAFEQIGpMMPkN5C7ynn/WMmyxI2yQay2ZT7eZ482d2ZnTl7d2fO3Jk794qqYhiGYRjeksnbARiGYRgZm0lEhmEYhleZRGQYhmF4lUlEhmEYhleZRGQYhmF4lUlEhmEYhleZRJRCRGSpiHT2wnqHichpETme0ut2RUQaiMgub8eRGojIJREpl8LrVBHxS8l1esqdblPp4TcoIo1F5HAC00vZv6/Md7DsAyLywN1FmDRJTkQicp+IrBWR8yJyRkTWiEhNTwSXElKq0FX1IVX9ytPrcSYipYB+QICq3uNiemMRibF/sBdFZJeIdPFkTKr6h6pW8uQ6UiMRWSUizzu/pqq5VXWft2LypuTY7tzdpuIm3zv9DYrIYBGZmdT3pYS45amqB+3fV7Q343JXkhKRiOQFFgGfAgWBEsD7wPXkD81IBqWACFU9mcA8R1U1N5AXeA2YKiJpLlGISJaMuG5v8XJ5i4iYsznpiaq6/QfUAM4lMD0TMAj4DzgJzADy2dPKAAp0AQ4BZ4GXgZrAFuAcMCHO8roCO+x5fwZKJ7DuOsBaezmbgcb26/WA00BJ+3kVe3n+wNdADHAVuAS8kdCy7GmrgKHAGuAisBwobE/zAWYCEfZ7NwDFnN73fBLKqTNw0I797QQ+dz77/afs5Q2yl/+A/bli7M823cV7GwOH47x2EmjnFOdA4H/2Z/oOKOg0731O5XQIeM5+PTswxo7/BDAJyBF3ncAAYF6c9Y8HPnH6bF8Ax4AjwDAgsz3tOfs7+NiObZiLz5cdGAcctf/GAdmd4wDessv4APB0nPcm+Bns+I9j/Y4KYB2kncL6fS0CfO35hwPRwDX7u5hgv66An/14OjARWIz1u/obKO8UT3NgF3Ae+Az4Hfv35OJzZ7Y/1//sZW3k5u9fsba7Pfb3NhEQe1p54De7PE8Ds4D8Tss9YH/mLVgHn1m4+fu4CGwHHo8TywtY23Ds9Grc+XY33P7OrwJ+3LpN+dllct6OfY79+mr7M1+219WBOL97oCTwg/3dRRBnP2TP0wK4AUTay9lsv34vsAA4A+wFXkhgW51uf3dL7WWsAe7B+l2eBXYCVZ3md/w+nN4/zMV2dFt5cnM/kiWBeG77bpy+5wfsx7WAv+zv5BgwAchmTxOs7e8kcAH4Fwiypz1sL/Mi1rb7eoK5xd0kZC88r/1FfQU8BBRwkTj2AuWA3PaX+3WcHewkrB12c6wN8yegKFbt6iTQyJ6/tb2sylg/+EHA2njiKmHH9TDWzrOZ/byI047gNyCHXVg942xcDyRhWauwNryK9vJWASPtaS8BC4GcWDuD6kBeF4nInXKaai+/CtZGXzmezz4DmA/ksd+7G+gWX6KJLxHZn7UV1g+6qv1aH2Ad4Iu1Y54MfGtPK431I+sIZAUKAaH2tI+xNs6CdlwLgQ9crLM0cAXI47QDPQbUsZ//aK8zF9ZvZD3wklMiigJ6Yf0+crj4fEPs+IsCRbB2ckOd4ogCxtqfrRHWzqqSm58hCvjQfm8O+/M/YX/3eYC5wE9xdqTPx4kvbiKKwNrws2Algdn2tMJYG3obe1ofrB1ifImoP9bvvBLWzqIKUMhpnYuA/Fg15lNAC6edeTP7MxXB2omPi7OthGPtuGOTcjusnXEmrJ38ZaC407QjWAebYi+/9F1sdweBQLsMsnLrNvUt8Lb9Xh/gvgR26I25+RvMjJX0Psb6nd3y3jjlOhiYGee11VjJxQcItcuzSQKJ6DTWfsEHa5+0H+hkxzEMWJnURBRPeZYhgUTk7ndjx1rHLvMyWInrVXvag1gHOfntZVR2+u6PAQ3sxwWwk1y8+yJ3ElCcD1DZLpDDWBvjAm4e9f8KdHeatxLWBhP7IRQo4TQ9Aujg9Px7pw+5FHuH6rSjvIKLWhHWUdrXcV77GehsP85qF9i/wDLsI8B4vsDElrUKGOQ0rTuwzH7cFWtnF+IixlXc3GjcKSdfp+nrgSddLDMz1lFagNNrLwGrXP1YXby/MVbiOYeV7KJjy9+evgNo6vS8uFOcbwI/ulimYO2MnI/m6wL749mA/gQ62Y+bAf+zHxezY8rhNG9H7A0VKxEdTOS3+j/gYafnDwIHnOKIAnI5Tf8OeMfNz3AD8Elg3aHAWVffv9NrcRPR507THgZ22o87AX/FKeNDcZfnNH0X0DqeacqtO+nvgIHxzPsY8E+cbaVrImUeHrturO2mTzzzHSDp292QBLapGcAUnLYbV+Uc9zdof6+nSKDm4PS+wTglIqyEHI19IGW/9gEuzj44fcdTnZ73AnY4PQ/G6YyTi7ink3yJyO3vJs60V7G3e6AJ1oFvHSBTnPkOYu2L8iZWrqqa9MYKqrpDVZ9TVV8gCOtoaJw9+V6s00Ox/sPaaRVzeu2E0+OrLp7nth+XBsaLyDkROYdV9RWsI6e4SgPtYue1578Pa8eJqkZifYlBwEdql1Q8ElyWzbkF2hWnmL/G+oJni8hRERklIlldrMOdcopvHc4KYyXZuMtyVUbxOaqq+bFqu59g/bhilQZ+dCqHHVgbXjGsjfB/LpZXBKtWsNHpfcvs1135BivBADxlP49dd1bgmNNyJmPVbmIdSuSzuSrne52en1XVyy6mu/MZTqnqtdgnIpJTRCaLyH8icgHrSDl/Elstxfed34vTZ7V/v/G2mCL+7ybB9YhIMRGZLSJH7M8wE+s35uyWMheRTiIS7lROQU7vSSwOZ+5sdwl9329g7R/Wi8g2Eenq5npLAv+papSb8zu7FzijqhedXkts+3N3/5ds7FaCl+y/bfbLbn03IlJRRBaJyHH7NzEC+/tV1d+wTtVNBE6KyBS7HQFYZwceBv4Tkd9FpG5C67mrC36qupObO3iwzsOXdpqlFNZR5wmS7hDWaZj8Tn85VHVtPPN+HWfeXKo6EkBESgDvAdOAj0Qku/PHSMqyEqKqkar6vqoGYF2baol1NBtXcpXTaawaStxlHUniclDV61hHpcEi8pj98iHgoThl4aOqR+xp5eOJ6SoQ6PSefGo1iHBlLtBYRHyBx7mZiA5h1YgKOy0nr6oGOoedyMdyVc5HnZ4XEJFcLqa78xnirrsfVs22tqrmBRrar4ubsSbkGNbpUWuBIuL83IX4vpvEjMCKM9j+DM9wM/5Yjs8hIqWxTiH3xDr1lx/Y6vSehOK4k+0u3jJU1eOq+oKq3ot1JP6Zm83UDwGl3Gx8EXf9R4GCIpLH6bU72v7icQXrgCjWbS1fE4jt5gSrlWBu+y92+3H3N/J/WNeuKti/ibdw+k2o6ieqWh0IwLpc0d9+fYOqtsY6cPwJq+Ydr6S2mvMXkX72TgMRKYl1NLvOnuVb4DURKSsiubF+2HPu8GhjEvCmiATa68onIu3imXcm8KiIPCgimUXEx26a7GtvtNOxLnp3w9qohzq99wTWtZpEl5VYwCJyv4gE20fBF7CSRIyLWZOlnNRqmvkdMFxE8tg7hr72Z0gyVb0BfAS8a780yV52aQARKSIire1ps4AHRKS9iGQRkUIiEqqqMVg7p49FpKj9vhIi8mA86zyFdYplGtaprx3268ewGoJ8JCJ5RSSTiJQXkUZJ+EjfAoPsuAvbnytu2bwvItlEpAHWgcPcpH4GWx6s5HVORApiHfg4i/s7S4rF2AcI9g6zBwnvlD4HhopIBauBmYSISCE31pMH62L3efvgrX8i8+fC2gGeAhCr6X+Q0/TPgddFpLodh1/sb4lk3O7sdbdzmvesHVfstpdQ2a/H2ieMFJFc9nrrxzPvCaCM2C32VPUQ1qn4D+z3hWDtY5KriXc48JRdHi2wrmPGJ6m/r4S+G2d5sPZll0TEH3gldoKI1BSR2mKd9bmMdc0/xt6enhaRfPbZqAu43g86JLVGdBGoDfwtIpexEtBWrKNBgC+xTk+txroIdw3rPGiSqeqPWBeDZ9tVwq1YDSRczXsIq3HDW1gbxSGsjSgT0BsrK79jn9LoAnSxdzxgndMdJNbpgNcTWVZi7gHmYRX8DqxWPF+7mC/Zysl+32VgH9b1lm/s5d+pL7GOEB/FasG2AFguIhexvu/aYN2ngFX17od12jQc66I4WDWrvcA6+7tbgVVbiM83WK38vonzeicgG1brm7NYZVsc9w0DwrBaef0LbLJfi3XcXu5RrMT6sl3Lv5PPMA6r0cJprHJaFmf6eKCtiJwVkU+S8BlQ1dNYF5dHYV1XDbA/V3y3TYzFOkBZjvVb/MKOLTHvY7VqO4+V/H5IJK7tWAcuf2HtCIOxWoLFTp+L1VDoG6x9x09YjT8gebc7sC66/y0il7B+s3305j1ag4Gv7HW1j/MZooFHsS7WH8Q65dkhnnXMtf9HiMgm+3FHrOsxR7Ea17ynqivcjDkxfezYzgFPY5VffG4pz8QWnMh34+x1rFPmF7EOzuY4Tctrv3YW65RkBDDanvYscMDedl62449XbLNNw8hQRKQx1oVnt464UxP7iPwwVnPzld6OxzDulrkpzDDSAPuUVX6xrm/Gnqdfl8jbDCNNMInIMNKGulitnE5jna55TFWvejckw0ge5tScYRiG4VWmRmQYhmF4VZrrrLFw4cJapkwZb4dhGIaRpmzcuPG0qsZ3Y7lXpblEVKZMGcLCwrwdhmEYRpoiIv8lPpd3mFNzhmEYhleZRGQYhmF4lUlEhmEYhleZRGQYhmF4lUlEhmEYhleZRGQYhmF4lccSkYh8KSInRWRrPNNFRD4Rkb0iskVEqnkqFsMwDCP18mSNaDrQIoHpDwEV7L8XsQZgMgzDMDIYj93QqqqrRaRMArO0BmbYYwSts3sWLm4PiBa/03tg2iOOp2fDL3Bh+6XkCNlIxW5ExxAZneDYWkYacTaTcj6z+S5Tgqqy9tRF/jp1MfGZvcibPSuU4NYx6A/br92WiETkRaxaE9WK3xryhe2XuHbyBj5Fs3kuUsPrIqNjiIlRMmWKO3K1kdaczxzDdVGyq/kuPenE1Rv8364TbIi4RNnc2b0dToLSRBc/qjoFmAJQo0w+pcvimxN/64TPPVD66xleis6Izzd/H2R++JFkWdb2YxcIKJ6XOS/VTZblGd4zelkXAKa1mOblSNIvVaVGjRrsunacjz76iN69e5M1a1ZvhxUvbyaiI0BJp+e+9mtGKudugvl7/xkAapd1NQJx0gQUz0vr0BJ3vRzDSM/Wrl1LcHAwefLk4fPPP6dw4cKULFky8Td6mTcT0QKgp4jMBmoD5xO9PmSkCvPDjzhqKAmpXbYgrUNL8FTtUikUmWFkTBEREQwcOJDPP/+c9957j8GDB1O1alVvh+U2jyUiEfkWaAwUFpHDwHtAVgBVnQQsAR4G9gJXgC6eisW4c65qP+Y0Weo3d/dcluxb4u0wXNp1ZheVClbydhjpgqoyY8YMXn/9dc6ePUv//v3p37+/t8NKMk+2muuYyHQFenhq/UbycFX7MafJUr8l+5ak2h1+pYKVeLjcw94OI10YMGAAo0ePpl69ekyaNIng4GBvh3RH0kRjBSP5uXudx9R+0q5KBSuZBgHp0NWrV7l8+TKFCxemW7duVKhQgW7dupEpU9rtKCftRm7cldiaTmJM7ccwUo9ly5YRFBTESy+9BEClSpV44YUX0nQSAlMjytBMTccw0oajR4/y6quvMnfuXCpVqkTPnj29HVKyMonIMAwjFfv11195/PHHuXHjBkOHDqV///5kz566b1BNKpOI0oDkvDE0ljvNr43U4U5awKXWhgqG+yIjI8maNStVqlTh4YcfZtiwYfj5+Xk7LI9I2ycWMwh3r+ckhbn2k3bEtoBLCtMyLe26cOECffr0oUGDBkRHR1O4cGFmz56dbpMQmBqRV5mWa4a7TAu49E9VmTdvHn369OH48eN0796d69evkzNnTm+H5nEmESWTOzl95m4XOKb2Yhjp26lTp+jcuTNLly6latWqzJ8/n5o1a3o7rBRjElEycbfbG2emCxzDMADy5s3L6dOnGTduHD169CBLloy1a85YnzaZOdeCzOkzwzCSYvXq1QwfPpzvv/+e3Llzs27dujR/P9CdMonoLjjXgszpMyM5uGohZ1rApS+nT5+mf//+TJ8+nTJlynDgwAGCgoIybBICk4juSGxNyNSCjOTmqo840wIufVBVpk2bRv/+/blw4QJvvvkmgwYNyhCNERJjEtEdcE5CphZkJDfTQi79mjlzJgEBAUyaNInAwEBvh5NqmETkJnM9yDCMpLpy5QojRozg5ZdfxtfXl++//558+fJl6NNwrpjScJPzTaWmJmQYRmKWLFlCYGAgw4cPZ+HChQAUKFDAJCEXTI3IDd/8fZC/95+hdtmCphZkuJRcA9GZhglp3+HDh3n11Vf5/vvvqVy5Mr///jsNGzb0dlipmknNbog9JWdqQUZ87qQbHldMw4S0b/jw4SxevJgRI0YQHh5ukpAbTI3ITbXLFjQ3nhoJMo0MMq7169eTI0cOgoODGTZsGP3796dcuXLeDivNMDWiBHzz90E6TP4r2TscNQwjfTh//jw9evSgTp06vP322wAUKlTIJKEkMokoAaaZtmEYrqgqs2fPxt/fn0mTJtGrVy9mzpzp7bDSrDR3au7GmUj+e7aT4/m1nTvx8ff32PpMM23DMOKaOXMmnTp1okaNGixatIjq1at7O6Q0Lc0lopgoveW5j78/eVu29FI0hmFkFNevX2ffvn1UrlyZ9u3bExUVRadOncicObO3Q0vz0lwiypRFKP31jGRdZnxDOJhRTA3DAFi5ciWvvPIKV65cYc+ePWTPnp0uXbp4O6x0w1wjIv4RUM21IcPI2E6ePEmnTp1o0qQJkZGRTJkyhezZs3s7rHQnzdWIPMVcCzIMw9nevXupVasWly5d4u233+btt98mR44c3g4rXTKJyDAMw8mFCxfImzcv5cuXp1u3bnTt2pXKlSt7O6x0LcMnIufue4z0L7m64onLdM2T9l2+fJkhQ4YwdepUtmzZgq+vL6NHj/Z2WBlChr9GZLrvyViSqyueuEzXPGnbwoULCQgIYNSoUbRp08aMEZTCMnyNCEz3PRmN6YrHiBUVFUX79u358ccfCQwM5I8//uC+++7zdlgZToavERmGkfGoWvcjZsmSheLFizNy5Eg2bdpkkpCXmERkGEaGsm7dOmrUqMGmTZsAmDhxIgMGDCBbtmxejizjypCn5lyNtmoYRvp29uxZ3nrrLSZPnsy9997L2bNnvR2SYfNoIhKRFsB4IDPwuaqOjDO9FPAVkN+eZ6CqJn+TpjicOzM1N62mHcnR4s20bsuY5syZQ+/evTl9+jSvvvoq77//Pnny5PF2WIbNY4lIRDIDE4FmwGFgg4gsUNXtTrMNAr5T1f8TkQBgCVDGUzHF1oRik5C5gTVtiW3xdjeJxLRuy5h27txJmTJlWLZsGVWrVvV2OEYcnqwR1QL2quo+ABGZDbQGnBORArHnxfIBRz0YjxnWIR0wLd4Md1y7do0PP/yQatWq8eijj/LWW28xaNAg00FpKuXJRFQCOOT0/DBQO848g4HlItILyAU84GpBIvIi8CJAYG6fJAXh6nqQqQkZRvq1YsUKunfvzp49e+jXrx+PPvooWbNm9XZYRgK83WquIzBdVX2Bh4GvReS2mFR1iqrWUNUamUSStALnDk1NTcgw0q8TJ07w9NNP06xZM1SV5cuXM2bMGG+HZbjBkzWiI0BJp+e+9mvOugEtAFT1LxHxAQoDJ5MzEFMLSj6e6iLHHaahgZGQX375hXnz5vHuu+/y5ptv4uOTtLMnhvd4ska0AaggImVFJBvwJLAgzjwHgaYAIlIZ8AFOeTAm4y55qoscd5iGBkZcmzdvZt68eQA8/fTT7Ny5k/fff98koTTGYzUiVY0SkZ7Az1hNs79U1W0iMgQIU9UFQD9gqoi8htVw4TmNveXZSLVMgwHD2y5dusR7773H+PHjKVOmDI899hhZsmShbNmy3g7NuAMevY/IvidoSZzX3nV6vB2o78kYDMNIX3766Sd69erF4cOHefHFF/nggw/IkiVD3pufbphvzzCMNOPff//l8ccfJzg4mDlz5lCvXj1vh2QkA2+3mvOo2LGGDMNIuyIjI/ntt98ACA4OZvHixWzcuNEkoXQkXdeIzFhDdy9uKznTcs1ISWvXruXll19m27Zt7Nq1Cz8/Px5+2DRYSW/SZY3om78P0mHyX2w/dsGMNXSX4raSMy3XjJRw5swZXnzxRerXr8+5c+f44Ycf8PPz83ZYhoekyxqR6coneZlWckZKunbtGqGhoRw9epR+/foxePBgcufO7e2wDA9Kl4kIzE2shpHWHD58GF9fX3x8fBg6dCihoaFUqVLF22EZKSBdnpozDCPtuHr1Ku+++y7ly5dn4cKFAHTu3NkkoQwk3daIDMNI/ZYvX0737t353//+xzPPPEOtWrW8HZLhBW4nIhHJqapXPBmM4V2u+pEzreQMT+nVqxcTJkygQoUKrFixgqZNm3o7JMNLEk1EIlIP+BzIDZQSkSrAS6ra3dPBGSnL1cBzppWckZyio6MByJw5M3Xq1KFw4cIMGDDA9A2XwblTI/oYeBC7w1JV3SwiDT0a1V2IvYm1dtmC3g4lTTIt5AxP2bRpEy+//DLPPvssvXr14umnn/Z2SEYq4VZjBVU9FOelaA/EkizMTayGkbpcvHiR1157jZo1a3Lw4EGKFy/u7ZCMVMadGtEh+/ScikhWoA+ww7Nh3R1zE6thpA7Lly+na9euHD16lJdffpkRI0aQP39+b4dlpDLuJKKXgfFYQ38fAZYD5vpQOmC67zE8LVu2bBQtWpTvv/+e2rVrezscI5VyJxFVUtVbTuaKSH1gjWdCMlJK3MYJpmGCcbciIyMZO3YsFy5cYPjw4TRu3JiwsDAyZTK3LBrxcycRfQpUc+M1Iw0yjROM5PLnn386Oiht164dMTExZMqUySQhI1HxJiIRqQvUA4qISF+nSXmxRlw1DMMgIiKCAQMG8MUXX1CqVCkWLlxIy5YtvR2WkYYkdKiSDeveoSxAHqe/C0Bbz4dmGEZaEBERwezZs3njjTfYvn27SUJGksVbI1LV34HfRWS6qv6XgjEZhpHK7dixg++++4733nuPihUrcvDgQQoWNPfuGXfGnWtEV0RkNBAIOG5/VtUmHovKSHam+x4jOVy5coXhw4czevRocufOTbdu3fD19TVJyLgr7lxFnAXsBMoC7wMHgA0ejOmOOA+GZ9wu7gB3YFrJGUmzbNkygoKCGDFiBE899RS7du3C19fX22EZ6YA7NaJCqvqFiPRxOl2X6hKRGQwvcaaFnHGnLl26xLPPPkuhQoVYuXIljRs39nZIRjriTiKKtP8fE5FHgKNAqqyHm8HwDCP5REdH8+2339KxY0dy587NihUr8Pf3J3v27N4OzUhn3Dk1N0xE8gH9gNexeuJ+1ZNBJVVsR6eGYSSPjRs3Urt2bZ599ll++uknAKpUqWKSkOERiSYiVV2kqudVdauq3q+q1YFUtdc3HZ0aRvI4f/48vXv3platWhw5coTZs2fTpk0bb4dlpHMJ3dCaGWiP1cfcMlXdKiItgbeAHEDVlAnRPaajU8O4e0888QS//fYbPXr0YNiwYeTLl8/bIRkZQELXiL4ASgLrgU9E5ChQAxioqj+lQGyGYaSAffv2UaRIEfLkycPw4cPJlCkTNWvW9HZYRgaS0Km5GkAzVX0TeBhoCdQ3Scgw0ocbN24wYsQIAgMDGTZsGAC1a9c2SchIcQnViG6oagyAql4TkX2qGpFCcRmG4UGrV6/m5ZdfZseOHbRt25bevXt7OyQjA0soEfmLyBb7sQDl7ecCqKqGeDw6wzCS3ccff0zfvn0pU6YMixcv5uGHzU3NhncllIgqp1gURrIzg94ZzmJiYrh8+TJ58uThkUce4dSpUwwaNIicOXN6OzTDSLDTU9PRaRpmBr0zYm3bto2XX37ZMVJqxYoVGTFihLfDMgwHd3pWuGMi0gJrmPHMwOeqOtLFPO2BwYACm1X1KU/GlJGYLn0ytitXrjB06FDGjBlDvnz56Nq1K6qKiHg7NMO4hccSkX0f0kSgGXAY2CAiC1R1u9M8FYA3sVrjnRWRop6KxzAykn/++Yc2bdpw4MABunTpwqhRoyhcuLC3wzIMl9xKRCKSAyilqrsSnfmmWsBeVd1nL2M20BrY7jTPC8BEVT0LoKonk7B8wzDiiK3xlCpVilKlSvHVV1/RsGFDb4dlGAlKtIsfEXkUCAeW2c9DRWSBG8suARxyen7Yfs1ZRaCiiKwRkXX2qTzDMJIoKiqKcePG0bRpU6KjoylUqBC///67SUJGmuBOp6eDsWo35wBUNRxrbKLkkAWoADQGOgJTRSR/3JlE5EURCRORsBjVZFq1YaQP69evp1atWrz22mv4+Phw4YIZk8tIW9xJRJGqej7Oa+5kgyNYXQTF8rVfc3YYWKCqkaq6H9iNlZhuXZnqFFWtoao1MpkLrYYBWGME9ejRgzp16nDixAnmzp3L4sWLKVCggLdDM4wkcScRbRORp4DMIlJBRD4F1rrxvg1ABREpKyLZgCeBuKf0fsKqDSEihbFO1e1zM3bDyNCyZs3KqlWr6NWrl6OHBNMizkiL3ElEvYBA4DrwDXAeN8YjUtUooCfwM7AD+E5Vt4nIEBFpZc/2MxAhItuBlUB/042QYcRv7969dOrUiYsXL5I9e3Y2btzI+PHjyZs3r7dDM4w75k6rOX9VfRt4O6kLV9UlwJI4r73r9FiBvvafYRjxuH79OqNGjWL48OFky5aNF154gQYNGuDj4+Pt0AzjrrmTiD4SkXuAecAcVd3q4ZiMOOJ21+MO06VP+rFy5UpeeeUVdu3aRYcOHRg7diz33nuvt8MyjGTjzgit9wP3A6eAySLyr4gM8nhkhkNsdz1JYbr0SR9UleHDhxMZGcmyZcuYPXu2SUJGuuPWDa2qehxrcLyVwBvAu8AwTwbmrm/+Psjf+89Qu2xBb4fiUaa7nowjJiaGL774ghYtWlCyZEm+/vpr8ufPT44cObwdmmF4hDs3tFYWkcEi8i8Q22LO1+ORuWl+uNUivHVo3HtlDSPt2bJlC/fddx8vvvgin3/+OQDFixc3SchI19ypEX0JzAEeVNWjHo7Hbd/8fZD54UfYfuwCtcsW5KnapbwdkmHcsUuXLvH+++/z8ccfU6BAAaZPn06nTp28HZZhpIhEE5Gq1k2JQJIqNgkFFM9rakNGmjd48GA++ugjnn/+eUaOHEmhQoW8HZJhpJh4E5GIfKeq7e1Tcs49KaSaEVoDiudlzkupMk8aRqIOHTrE5cuX8ff3Z+DAgTz22GPcd9993g7LMFJcQjWiPvb/likRiGFkFFFRUXzyySe8++67VK9end9//53ChQubJGRkWPE2VlDVY/bD7qr6n/Mf0D1lwjOM9GXdunXUqFGDfv360bhxY7766itvh2QYXudOFz/NXLz2UHIHYhjp3eLFi6lXrx6nT5/mhx9+YOHChZQpU8bbYRmG18WbiETkFfv6UCUR2eL0tx/YknIhGkbapaocOWLdYvDAAw8wZMgQduzYweOPP246KDUMW0LXiL4BlgIfAAOdXr+oqmc8GlUG5qo7H9NdT9q0e/duunfvzu7du9m+fTu5c+dm0CDTKYlhxJXQqTlV1QNAD+Ci0x8ikr67MfAiV935mO560pZr164xePBggoODCQsL48033zQ3pBpGAhKrEbUENmI133Y+j6BAOQ/GlaGZ7nzSruPHj9OwYUP27NlDx44dGTt2LPfcc4+3wzKMVC3eRKSqLe3/yTUsuGGkW5GRkWTNmpVixYrRsGFDJk6cSLNmrtr5GIYRlzt9zdUXkVz242dEZKyImP50DAOrg9JJkyZRvnx5Dh8+jIjw+eefmyRkGEngTvPt/wOuiEgVoB/wP+Brj0ZlGGnA5s2bqVevHq+88goVKlQgMjLS2yEZRprkTiKKskdSbQ1MUNWJQB7PhmUYqZeq8vrrr1O9enX27dvH119/zYoVKyhb1pzFNow74U4iuigibwLPAotFJBOQ1bNhGUbqJSKcPXuWbt26sWvXLp555hlzT5Bh3AV3ElEH4DrQ1R4gzxcY7dGoDCOV+e+//3jsscfYtGkTAFOnTmXy5MkUKFDAy5EZRtrnzlDhx4FZQD4RaQlcU9UZHo/MMFKByMhIRo0aRUBAAL/88gu7dln3eGXK5M4xnGEY7nCn1Vx7YD3QDmgP/C0ibT0dmGF429q1a6lWrRoDBgygWbNm7Nixg44dO3o7LMNId9wZofVtoKaqngQQkSLACmCeJwMzDG9bsWIF58+f56effqJ169beDscw0i13zi9kik1Ctgg332cYaYqqMmPGDJYuXQrAgAED2L59u0lChuFh7iSUZSLys4g8JyLPAYuBJYm8xzDSlJ07d9KkSRM6d+7MtGlW90rZs2cnd+7cXo7MMNI/dxor9AcmAyH23xRVHeDpwAwjJVy9epV33nmHkJAQwsPDmTx5MrNnz/Z2WIaRocR7jUhEKgBjgPLAv8DrqnokpQJLyDd/H+Tv/WeoXdZ0Am7cnYULFzJs2DCeeeYZxowZQ7FixbwdkmFkOAk1VvgSmAGsBh4FPgXapERQiZkfbuXD1qElvByJkRYdP36c8PBwWrRoQbt27ShTpgy1atXydliGkWEllIjyqOpU+/EuEdmUEgG5q3bZgjxV23t9r7oawC45mEHwPCc6OprJkyfz5ptvki1bNg4ePEiOHDlMEjIML0voGpGPiFQVkWoiUg3IEed5huZqALvkYAbB84xNmzZRt25devToQa1atVi7dq0ZrM4wUomEakTHgLFOz487PVegiaeCSivMAHZpw/79+6lVqxaFCxfmm2++4cknnzR9wxlGKpLQwHj3p2QghpGcVJV///2XkJAQypYty7Rp03j00UfJnz+/t0MzDCMOc2Oqke7s37+fli1bUrVqVbZs2QLAs88+a5KQYaRSHk1EItJCRHaJyF4RGZjAfE+IiIpIDU/GY6RvN27cYOTIkQQGBvL7778zZswYAgICvB2WYRiJcKevuTsiIpmBiUAz4DCwQUQWqOr2OPPlAfoAf3sqFiP9i46Opl69emzcuJE2bdowbtw4SpYs6e2wDMNwgzu9b4uIPCMi79rPS4mIO+1dawF7VXWfqt4AZmON8hrXUOBD4FoS4jYMAC5cuABA5syZ6dq1KwsXLuT77783Scgw0hB3Ts19BtQFYvu/v4hV00lMCeCQ0/PD9msOdjPwkqq6OKEFiciLIhImImExqm6s2kjvVJXp06dTrlw55s+fD0D37t1p2bKllyMzDCOp3ElEtVW1B3aNRVXPAtnudsX2kONjgX6JzauqU1S1hqrWyGSa3WZ427dvp3HjxnTp0gV/f3/Kly/v7ZAMw7gL7iSiSPt6j4JjPKIYN953BHA+P+JrvxYrDxAErBKRA0AdYIFpsGAkZNSoUVSpUoWtW7fy+eefs3r1aoKCgrwdlmEYd8GdxgqfAD8CRUVkONAWGOTG+zYAFUSkLFYCehJ4Knaiqp4HCsc+F5FVWB2rhrkdfQqK26WP6YonZakqIsI999zD008/zejRoylSpIi3wzIMIxm4MwzELOAN4AOs3hYeU9W5brwvCugJ/AzsAL5T1W0iMkREWt1d2Ckvbpc+piuelHH06FHatWvHp59+CkCnTp2YPn26SUKGkY4kWiMSkVLAFWCh82uqejCx96rqEuIMoqeq78Yzb+PEludtpkuflBMdHc1nn33G22+/TWRkJPXq1fN2SIZheIg7p+YWY10fEsAHKAvsAgI9GJeRgYWHh/P888+zceNGmjdvzmeffWYaJBhGOpZoIlLVYOfndpPr7h6LyMjwzp8/z9GjR5kzZw7t2rUzHZQaRjqX5J4VVHWTiNT2RDBGxqSqzJ07lz179vD222/TqFEj9u3bh4+Pj7dDMwwjBbhzjaiv09NMQDXgqMciMjKU//3vf/Ts2ZNly5ZRs2ZN3njjDbJmzWqSkGFkIO7cR5TH6S871jUjV131GIbbrl+/zvDhwwkKCmLNmjWMHz+etWvXkjVrVm+HZhhGCkuwRmTfyJpHVV9PoXiMDOLQoUMMHTqURx99lHHjxlGiRInE32QYRroUb41IRLKoajRQPwXjMdKxU6dOMWHCBAD8/PzYvn07c+fONUnIMDK4hE7Nrbf/h4vIAhF5VkTaxP6lRHBG+hATE8MXX3yBv78/ffv2Zdcu68bgcuXKeTkywzBSA3euEfkAEUAToCXwqP3fMBK1detWGjVqxPPPP09gYCDh4eFUqmS6RjIM46aErhEVtVvMbeXmDa2xzFgMGVRkZCSHDx/m2rXEh49SVY4dO8Y777xDgQIFyJ07NwA7duzwdJiGkWH5+Pjg6+ubphr+JJSIMgO5uTUBxTKJKIM6fPgwefLkoUyZMvHeaHrhwgXy5MmDiFCyZEl8fHzS1EZhGGmVqhIREcHhw4cpW7ast8NxW0KJ6JiqDkmxSIw04dq1a/EmoRs3bnDw4EHOnTtHmTJlKFy4MHny5PFClIaRMYkIhQoV4tSpU94OJUkSSkSmXxXDpbhJSFU5efIkR45Yw02VKFGCggULeiM0w8jw0mKXWAkloqYpFoWRpu3fv58zZ86QL18+SpUqRfbs2b0dkmEYaUi8reZU9UxKBpLazN09ly7Lujj+nMciMiAqKoro6GgAihQpQrly5fDz80s1SWjBggWMHDnS22F4XezYTaGhofj7+/Pxxx/fMn3KlCn4+/vj7+9PrVq1+PPPPx3TIiMjGThwIBUqVKBatWrUrVuXpUuXpvRHSNSrr77K6tWrvR1GvDZu3EhwcDB+fn707t0b1dsvsZ89e5bHH3+ckJAQatWqxdatWx3Tzp07R9u2bfH396dy5cr89ddfALz++uv89ttvKfY5PMmd5tsZkhkIL35nzpxh27ZtHD1qdTmYJ08eChYsmKpOCbRq1YqBAwe6Na+qEhMT4+GI4hcVFeXR5Xfo0IHw8HDWrFnD8OHDOXToEACLFi1i8uTJ/Pnnn+zcuZNJkybx1FNPcfz4cQDeeecdjh07xtatW9m0aRM//fQTFy9eTNbYYg9m7lRERATr1q2jYcOGbr/H0+Ud1yuvvMLUqVPZs2cPe/bsYdmyZbfNM2LECEJDQ9myZQszZsygT58+jml9+vShRYsW7Ny5k82bN1O5cmUAevXqlW4OtpLc+3ZGYgbCu9XevXs5ceIEly9fJmfOnEzeEMGeJYeTdR0B9+blvUfjH+rqwIEDtGjRgjp16rB27Vpq1qxJly5deO+99zh58iSzZs2iVq1aTJ8+nbCwMCZMmMCJEyd4+eWX2bdvHwD/93//x7333suDDz5I7dq12bhxI0uWLGHChAksXboUEWHQoEF06NDhtvWvX7+ePn36cO3aNXLkyMG0adOoVKkSderU4YsvviAw0Iq9cePGjBkzhsqVK9OrVy+2bt1KZGQkgwcPpnXr1kyfPp0ffviBS5cuER0dzeLFi2ndujVnz54lMjKSYcOG0bq11aXj0KFDmTlzJkWKFKFkyZJUr16d119/nf/973/06NGDU6dOkTNnTqZOnYq/v3+8ZVeoUCH8/Pw4duwYJUuW5MMPP2T06NEULlwYgGrVqtG5c2cmTpzIm2++ydSpU9m/f7+jllusWDHat29/23I3bNhAnz59uHz5MtmzZ+fXX3/l+++/d5Q/QMuWLXn99ddp3LgxuXPn5qWXXmLFihW0a9eOzZs3M3euNejzqlWrGDNmDIsWLWL58uW89957XL9+nfLlyzNt2jTHLQCxvv/+e1q0aOF4PmTIEBYuXMjVq1epV68ekydPRkRo3LgxoaGh/Pnnn3Ts2JHGjRvTt29fLl26ROHChZk+fTrFixdn6tSpTJkyhRs3buDn58fXX39Nzpw54y3TxBw7dowLFy5Qp04dwBph+KeffuKhhx66Zb7t27c7Dpz8/f05cOAAJ06cwMfHh9WrVzN9+nQAsmXLRrZs2QAoXbo0ERERHD9+nHvuueeOY0wNTI3IcMs333xDUFAQ169fp1SpUlSuXNlrTbL37t1Lv3792LlzJzt37uSbb77hzz//ZMyYMYwYMeK2+Xv37k2jRo3YvHkzmzZtciSLPXv20L17d7Zt20ZYWBjh4eFs3ryZFStW0L9/f44dO3bbsvz9/fnjjz/4559/GDJkCG+99RZg1Tq+++47wNr5HDt2jBo1ajB8+HCaNGnC+vXrWblyJf379+fy5csAbNq0iXnz5vH777/j4+PDjz/+yKZNm1i5ciX9+vVDVdmwYQPff/89mzdvZunSpYSFhTliefHFF/n000/ZuHEjY8aMoXv3hIcJO3jwINeuXSMkJASAbdu2Ub169VvmqVGjBtu2bWPv3r2UKlWKvHnzJrjMGzdu0KFDB8aPH+8ouxw5ciT4nsuXL1O7dm02b97MwIED+fvvvx1lMmfOHJ588klOnz7NsGHDWLFiBZs2baJGjRqMHTv2tmWtWbPmls/Qs2dPNmzYwNatW7l69SqLFi26JdawsDB69+5Nr169mDdvHhs3bqRr1668/fbbALRp04YNGzY4ah5ffPHFbetcuXIloaGht/25GkX4yJEj+Pr6Op77+vo6GvU4q1KlCj/88ANgHez8999/HD58mP3791OkSBG6dOlC1apVef755x1lBdbBw5o1axIs77TA1IiMBEVGRpI1a1Zq1KhB27ZtKVGiBEWLFgVIsObiSWXLliU42BqvMTAwkKZNmyIiBAcHc+DAgdvm/+2335gxYwYAmTNnJl++fJw9e5bSpUs7jlRjj5QzZ85MsWLFaNSoERs2bKBVq1a3LOv8+fN07tyZPXv2ICJERkYC0L59e5o3b87777/Pd999R9u2bQFYvnw5CxYsYMyYMYDV/P3gwYMANGvWzNG6UFV56623WL16NZkyZeLIkSOcOHGCNWvW0Lp1a3x8fPDx8eHRRx8F4NKlS6xdu5Z27do5Yrt+/brL8pozZw6rV69m586dTJgwIVmH2Ni1axfFixenZs2aAIkmLrC+gyeeeAKALFmy0KJFCxYuXEjbtm1ZvHgxo0aN4vfff2f79u3Ur291dXnjxg3q1q1727KOHTtGkSJFHM9XrlzJqFGjuHLlCmfOnCEwMNBRZrE13F27drF161aaNWsGWKcHixcvDlg9gQwaNIhz585x6dIlHnzwwdvWef/99xMeHu5uEbll4MCB9OnTh9DQUIKDg6latSqZM2cmKiqKTZs28emnn1K7dm369OnDyJEjGTp0KABFixZ1nCJPy0wiMlw6efIk/fr14/Lly/zwww9UrFiRmTNnpopeEZwbRGTKlMnxPFOmTEk6/58rV65E55k4cSJTp04FYMmSJbzzzjvcf//9/Pjjjxw4cIDGjRsDVpP1QoUKsWXLFubMmcOkSZMAK8F8//33t3Vr9Pfff9+y/lmzZnHq1Ck2btxI1qxZKVOmTIK9V8TExJA/f363dogdOnRgwoQJhIWF0bx5c1q1asU999xDQEAAGzdupEmTJo55N27cSGBgIH5+fhw8eJALFy64lVziypIlyy3X3Zw/i4+PD5kzZ3Y8f/LJJ5kwYQIFCxakRo0a5MmTB1WlWbNmfPvttwmuJ0eOHI5lX7t2je7duxMWFkbJkiUZPHjwLeuNLW9VJTAw0HHR39lzzz3HTz/9RJUqVZg+fTqrVq26bZ6VK1fy2muv3fZ6zpw5Wbt27S2vlShRgsOHb56+Pnz4sMtOfvPmzcu0adMc8ZUtW5Zy5cpx5coVfH19qV3bGou0bdu2t1wXij1FnNaZU3Pc3kIuI7eSi4mJYcqUKVSqVIk5c+YQGBh41xeUva1p06b83//9H2Ad/Z4/f/62eRo0aMCcOXOIjo7m1KlTrF69mlq1atGjRw/Cw8MJDw/n3nvv5fz5844dSex5+1gdOnRg1KhRnD9/3nH668EHH+TTTz91tJT6559/XMZ4/vx5ihYtStasWVm5ciX//fcfAPXr12fhwoVcu3aNS5cuOU415c2bl7JlyzquragqmzdvTrAcatSowbPPPsv48eMBeOONNxgwYAAREREAhIeHM336dLp3707OnDnp1q0bffr04caNG4DVe3rs+mJVqlSJY8eOsWHDBgAuXrxIVFQUZcqUITw8nJiYGA4dOsT69euJT6NGjdi0aRNTp07lySefBKBOnTqsWbOGvXv3AtbpvN27d9/23sqVKzvmiU06hQsX5tKlS8ybN8/l+ipVqsSpU6cciSgyMpJt27Y54i9evDiRkZHMmjXL5ftja0Rx/+ImIYDixYuTN29e1q1bh6oyY8YMx7U/Z+fOnXOU8+eff07Dhg3Jmzcv99xzDyVLlnR0FPzrr78SEBDgeN/u3bsJCgpyGWdaYhIRt7eQg4zZSm7fvn3cd999vPTSS44WPEOHDr3l6DUtGj9+PCtXriQ4OJjq1auzffv22+aJbTpbpUoVmjRpwqhRo1xeAH7jjTd48803qVq16m21r7Zt2zJ79uxbLui/8847REZGEhISQmBgIO+8847LGJ9++mnCwsIIDg5mxowZjkYHNWvWpFWrVoSEhPDQQw8RHBxMvnz5AKsW9cUXX1ClShUCAwOZP39+omUxYMAApk2bxsWLF2nVqhVdu3alXr16+Pv788ILLzBz5kzHaaphw4ZRpEgRAgICCAoKomXLlrfVjrJly8acOXPo1asXVapUoVmzZly7do369etTtmxZAgIC6N27N9WqVYs3psyZM9OyZUuWLl1Ky5ZWf8pFihRh+vTpdOzYkZCQEOrWrcvOnTtve+8jjzziqLXkz5+fF154gaCgIB588EHH6cK4smXLxrx58xgwYABVqlQhNDTUkUSGDh1K7dq1qV+/foINP5Lis88+4/nnn8fPz4/y5cs7GipMmjTJUXPesWMHQUFBVKpUiaVLlzoOFgA+/fRTnn76aUJCQggPD3dcl4yMjGTv3r3UqFEjWeL0JnHVpj01C8mbUyuP/hWAOS/dfs74TnRZ1gUgw7eQi4iIoEGDBgwcOJBnn33WZXPsHTt2OJqPGinj0qVL5M6dmytXrtCwYUOmTJmS4I49o7nvvvtYtGgR+fPn93YoKSq2cUvs9SJnrrZTEdmoqqkya5lrRBncggULmD59OnPnzqVQoUJs3bqVTJlMRTk1efHFF9m+fTvXrl2jc+fOJgnF8dFHH3Hw4MEMl4iioqLo16+ft8NIFiYRZVAHDx6kd+/ezJ8/n8DAQI4dO4avr69JQqnQN9984+0QUrXYC/kZjXOLybQuQyaiubvnsmTfEsfzXWd2UalgxhisLSoqinHjxvHee++hqnz44Ye89tprZpgGwzC8JkMe/mbk7nuio6P5/PPPadKkCdu3b+eNN94wScgwDK/KkDUiyFjd95w9e5aRI0cyaNAg8uTJw5o1a1Jd33CGYWRcGbJGlFGoKrNmzcLf35+PPvqIlStXAlafYyYJGYaRWphElE7t3r2bZs2a8cwzz1CmTBnCwsJu664mrcqcOTOhoaEEBQXx6KOPcu7cOce0bdu20aRJEypVqkSFChUYOnToLd3uL126lBo1ahAQEEDVqlXTVKuj2Htq4g7lEJ+4HYQmF1Wld+/e+Pn5ERISwqZNm1zOd/XqVRo1apRqb4iOiIjg/vvvJ3fu3PTs2TPe+c6cOUOzZs2oUKECzZo14+zZs0D85XDq1KlbOmI1EmcSUTr16quvEhYWxmeffcbatWsJDQ31dkjJJkeOHISHh7N161YKFizIxIkTAWvHFzv8w65du9i8eTNr167ls88+A6x+xHr27MnMmTPZvn07YWFh+Pn5JWtsnhpi4Pjx42zYsIEtW7a47F4mJS1dutQxpMGUKVN45ZVXXM735Zdf0qZNG7dviE7p4Th8fHwYOnSoox/A+IwcOZKmTZuyZ88emjZt6uhiJ75yKFKkCMWLF08XnZGmlAx7jSg9+uWXX/D396dkyZL83//9H9mzZ/ds9/BLB8Lxf5N3mfcEw0Puj7FSt25dtmzZAljNnOvXr0/z5s0Bq++vCRMm0LhxY3r06MGoUaN4++23HXfMZ86c2eVO9NKlS/Tq1YuwsDBEhPfee48nnniC3Llzc+nSJQDmzZvHokWLmD59Os899xw+Pj78888/1K9fnx9++IHw8HDHfS0VKlTgzz//JFOmTLz88suOTk/HjRvn6NQz1rVr13jllVcICwsjS5YsjB07lvvvv5/mzZtz5MgRQkND+fTTT2nQoIHjPa6GuXDuCfrSpUsuh5i4fPky7du35/Dhw0RHR/POO+/QoUMHBg4cyIIFC8iSJQvNmze/bUc9f/58OnXqhIhQp04dzp07x7Fjxxw9MsSaNWuWo+l5fDEcOHDgtuE4vvvuO7777juuX7/O448/zvvvvw/AY489xqFDh7h27Rp9+vThxRdfdPdn4lKuXLm47777HF0ExWf+/PmO3hs6d+5M48aN+fDDDxMsh8cee4xZs2bd9v0arnk0EYlIC2A8kBn4XFVHxpneF3geiAJOAV1V9T9PxpQeHT9+nL59+/Ltt9/So0cPJkyYQOnSpb0dlsdFR0fz66+/0q1bN8D1sAbly5fn0qVLXLhwga1bt7p1Km7o0KHky5ePf/+1kmzsqZiEHD58mLVr15I5c2aio6P58ccf6dKlC3///TelS5emWLFiPPXUU7z22mvcd999HDx4kAcffPC2TmQnTpyIiPDvv/+yc+dOmjdvzu7du1mwYAEtW7Z02clp7DAXP/74I9HR0Y5kGSt2iIm8efNy+vRp6tSpQ6tWrVi2bBn33nsvixcvBqz+7iIiIvjxxx/ZuXMnInLLac9YR44coWTJko7nsUMbOCeiGzdusG/fPsqUKZNgDGANx/HVV19Rp04dli9fzp49e1i/fj2qSqtWrVi9ejUNGzbkyy+/pGDBgly9epWaNWvyxBNPUKhQoVtie+211xzXQp09+eSTbg+UGNeJEyccn+2ee+7hxIkTiZZDjRo1GDRo0B2tLyPyWCISkczARKAZcBjYICILVNW5o69/gBqqekVEXgFGAbePRma4FNtB6cCBA7l69SrvvffeHW9sdyQJNZfkdPXqVUJDQzly5AiVK1d2dOefXFasWMHs2bMdzwsUKJDoe9q1a+c4BdWhQweGDBlCly5dmD17tmP4gRUrVtzSz92FCxcc3ffE+vPPP+nVqxdgjX1UunRpdu/enWAP2K6GuXAW3xATwcHB9OvXjwEDBtCyZUsaNGhAVFQUPj4+dOvWjZYtWzr6fkuq06dP39LTQXwxALcMx7F8+XKWL19O1apVAasmtWfPHho2bMgnn3zCjz/+CMChQ4fYs2fPbYnI3etnd0pE3Grok16GZ0gpnrxGVAvYq6r7VPUGMBu4pdtZVV2pqlfsp+sAXwy3ffDBB7zyyitUr16dLVu2MHjw4GQdaya1ir1G9N9//6GqjmtEscMaONu3bx+5c+cmb968BAYG3jY9KZx3QHGHaHAe0qFu3brs3buXU6dO8dNPP9GmTRvAOnBYt26do7fmI0eOeKxBgTPnISbCw8MpVqwY165do2LFimzatIng4GAGDRrEkCFDyJIlC+vXr6dt27YsWrTI5UX3EiVKOIYbB9dDGzgPz5BQDHBr2akqb775pqOM9u7dS7du3Vi1ahUrVqzgr7/+YvPmzVStWtXlMBmvvfaay0Hr7mZI7WLFijkGSTx27JhjPK6EyiG9DM+QUjyZiEoAh5yeH7Zfi083YKmrCSLyooiEiUhYTBrrpDW5Xbx4kf379wPw8ssvM2vWLFasWHHbeDcZQc6cOfnkk0/46KOPiIqK4umnn+bPP/9kxYoVgFVz6t27N2+88QYA/fv3Z8SIEY7hBGJiYhy9Hztr1qyZI7nBzVNzxYoVY8eOHcTExDiOzF0RER5//HH69u1L5cqVHUftzZs359NPP3XM5+o0W4MGDRzDD+zevZuDBw8m+t0mNsxFfENMHD16lJw5c/LMM8/Qv39/Nm3axKVLlzh//jwPP/wwH3/8scuhJVq1asWMGTNQVdatW0e+fPluuz5UoEABoqOjHckivhjievDBB/nyyy8dpxePHDnCyZMnOX/+PAUKFCBnzpzs3LmTdevWuXz/xx9/7HKIhrs5U9CqVSu++uorAL766ivHMA4JlUN6GZ4hxaiqR/6AtljXhWKfPwtMiGfeZ7BqRNkTW25wnhzaftJabT9prd6p55Y+p88tfe6O3+8NMTEx+sMPP6ivr6/WrFlTY2JivBLH9u3bvbJeZ7ly5brlecuWLXXGjBmqqrplyxZt1KiRVqxYUcuXL6+DBw++pawWLlyo1apVU39/f61cubL279//tuVfvHhRO3XqpIGBgRoSEqLff/+9qqrOnTtXy5Urp7Vr19YePXpo586dVVW1c+fOOnfu3FuWsWHDBgV0+vTpjtdOnTql7du31+DgYK1cubK+9NJLt6376tWr+txzz2lQUJCGhobqb7/9pqqq+/fv18DAQJflcfz4cW3VqpUGBQVplSpVdO3atbeU06lTp7ROnToaFBSkzz33nPr7++v+/ft12bJlGhwcrFWqVNEaNWrohg0b9OjRo1qzZk0NDg7WoKCgW+KPFRMTo927d9dy5cppUFCQbtiwwWVcXbt21V9++SXBGFx9rnHjxmlQUJAGBQVpnTp1dO/evXrt2jVt0aKF+vv7a+vWrbVRo0a6cuVKl+tNitKlS2uBAgU0V65cWqJECd22bZuqqnbr1s3xuU6fPq1NmjRRPz8/bdq0qUZERCRaDqNHj9ZPPvnkruO7U662UyBMPbS/v9s/TyaiusDPTs/fBN50Md8DwA6gqDvLzYiJ6MCBA9qyZUsFNCQkRP/66y+vxZIaEpGRNmzcuFGfeeYZb4fhFQ0aNNAzZ854bf1pLRF5stXcBqCCiJQFjgBPAk85zyAiVYHJQAtVPenBWNKsv/76iwceeACAMWPG0KdPH7JkMa3ujdSvWrVq3H///URHR6f5wRWT4tSpU/Tt29etRi6GxWPXiFQ1CugJ/IxV4/lOVbeJyBARib3FfzSQG5grIuEissBT8aQ1Fy5cAKyNuWvXruzYsYN+/fqZJGSkKV27ds1QSQisG1ofe+wxb4eRpnh0r6aqS4AlcV571+nxA55cf1oUERHBwIEDWb58Odu2bSN37ty3XOA2DMNIb0wXP6mEqjJjxgz8/f2ZNm0aHTp0MB2TGoaRIaS78zxxB71zJbUNhHf+/Hkee+wxVq1aRd26dZk0aRIhISHeDsswDCNFpLkaUYwq249diHd63EHvXEktA+FZDVkgb968FC5cmClTpvDnn3+aJGQYRsbi7WZ7Sf0LyO2j7Set1Vnr/nPZbDGtNM1etmyZhoaG6qFDh7wdSpKkhubbmTJl0ipVqmhgYKC2bNlSz54965i2detWvf/++7VixYrq5+enQ4YMueU+oiVLlmj16tW1cuXKGhoaqn379vXCJ7gzTz75pAYHB+vYsWPdmj/u/VbJZceOHVqnTh3Nli2bjh49Ot75YmJi9P7779fz5897JI7kMH36dPXz81M/Pz+X90ypqrZv316rVKmiVapU0dKlS2uVKlVUVfXGjRvaqVMnDQoKUn9/fx0xYoSqql6/fl0bNGigkZGRKfUxbpPWmm97PYCk/gXm9knwC0jtiejo0aPaoUMHBbRixYq6ceNGb4eUJKkhETnvYDt16qTDhg1TVdUrV65ouXLl9Oeff1ZV1cuXL2uLFi10woQJqqr677//arly5XTHjh2qqhoVFaWfffZZssbmqZ3PsWPHtHz58kl6j6cS0YkTJ3T9+vX61ltvJZiIFi1apK+++mqSlh0VFXW34bktIiJCy5YtqxEREXrmzBktW7Zsovf+9O3bV99//31VVZ01a5Z26NBBVa3fWunSpXX//v2qqjp48GCdOXOmR+NPSFpLROnuGlFqNnHiRN566y2uX7/O+++/z4ABA8iePbu3w7pjH67/kJ1ndibrMv0L+jOg1gC35zfDQKT8MBBFixalaNGijl674zNr1qxbhmqIbxiH3Llz89JLL7FixQomTpzIgQMH+OSTT7hx4wa1a9fms88+c3xXGzZs4OrVq7Rt29YxPMSd+vnnn2nWrBkFCxYErK6dli1bRseOHV3Or6p89913/Pbbb4DVldPly5eJiori6tWrZMuWzdE57WOPPcabb77J008/fVcxZhQmEaWgjRs3Urt2bSZOnEiFChW8HU6aZ4aBsKT0MBDuWrNmDZMnT3Y8j28Yh8uXL1O7dm0++ugjduzYwYcffsiaNWvImjUr3bt3Z9asWXTq1Inhw4dTsGBBoqOjadq0KVu2bLnteuro0aMdffU5i+2921l8wzjE548//qBYsWKObbdt27bMnz+f4sWLc+XKFT7++GNHUgsKCmLDhg1JL7QMKk0nIlct5FJTi7gLFy7w7rvv8uyzz1K9enU+++wzsmfPnm6aZSel5pKczDAQt0qNw0CANcR2njx5HM/jG8Yhc+bMPPHEEwD8+uuvbNy4kZo1awLWdx3b2/V3333HlClTiIqK4tixY2zfvv22RNS/f3/69+9/xzEn5Ntvv72ltrR+/XoyZ87M0aNHOXv2LA0aNOCBBx6gXLlyZM6cmWzZsnHx4sVbysBwLc21mnPmqoVcamgRp6rMmzePypUr88knn/D7778D1pFpeklC3mSGgUia5B4Gwl1ZsmRxDP2d0DAOPj4+jiSuqnTu3NlRRrt27WLw4MHs37+fMWPG8Ouvv7JlyxYeeeQRl8NAjB492uUwEL17975tXneGs4gVFRXFDz/84DioAOtUcIsWLciaNStFixalfv36hIWFOaZfv349QwzLkhzSdCICK/FMazHtlr92Fdt5LZ79+/fTsmVL2rVrR9GiRfnrr7/o27ev1+JJz8wwEJaUHgbCXZUqVXJct3J3GIemTZsyb948Tp60up48c+YM//33HxcuXCBXrlzky5ePEydOsHSpyxFj6N+/v8thIOKelgNryInly5dz9uxZzp49y/Lly3nwwQddLnfFihX4+/vj63tzyLRSpUo5rhddvnyZdevWOa4/RkREULhwYbJmzepmaWVsaT4RpTazZs1i9erVfPzxx2zYsIHatWt7O6R0rWrVqoSEhPDtt9+SI0cO5s+fz7Bhw6hUqRLBwcHUrFmTnj17AhASEsK4cePo2LEjlStXJigoyLGjdDZo0CDOnj1LUFAQVapUcQw9PXLkSFq2bEm9evVuG38nrg4dOjBz5sxbjqA/+eQTwsLCCAkJISAgwGUS7N69OzExMQQHB9OhQwemT5+eaIOW8ePHs3LlSoKDg6levfotp/8Ann76acLCwggODnb03gHw77//UqtWLUJDQ3n//fcZNGgQFy9epGXLloSEhHDfffcxduzY29Z3/PhxfH19GTt2LMOGDcPX19fRN6KzRx55hFWrVgHQokULoqKiqFy5MgMHDnSMyBpXQEAAw4YNo3nz5oSEhNCsWTOOHTtGlSpVqFq1Kv7+/jz11FO3NfK4EwULFuSdd96hZs2a1KxZk3fffddxjef555+/pXYze/bs2xox9OjRg0uXLhEYGEjNmjXp0qWL41ThypUreeSRR+46xoxCrFZ9aUdQnhy69eJVALos6wLAtBbTvBkSf/zxB9evX+eBBx7g+vXrnDp16pYjp/Rkx44dVK5c2dthGGnAsWPH6NSpE7/88ou3Q0lxbdq0YeTIkVSsWNEr63e1nYrIRlWt4ZWAEmFqRHfh9OnTdO3alYYNGzJkyBAAsmfPnm6TkGEkRfHixXnhhRdc1pbSsxs3bvDYY495LQmlRSYR3QFVZdq0afj7+/P1118zYMCAeM9ZG0ZG1r59+wRb/KVH2bJlo1OnTt4OI01J0823vWXJkiV07dqV+vXrM2nSJDM2vWEYxl0wNSI3XblyhTVr1gDw8MMPM3/+fFavXm2SkGEYxl0yicgNS5cuJSgoiIceeohz584hIrRq1YpMmUzxGYZh3C2zJ03AkSNHaNeuHQ8//DDZs2dn4cKFjv7DDMMwjORhElE8Tp48SUBAAIsWLWLYsGFs3ryZRo0aeTssA6sbm9DQUIKCgnj00Udv6Q9t27ZtNGnShEqVKlGhQgWGDh2K8y0KS5cupUaNGgQEBFC1alW3+p5LLTp27EhISAgff/yxW/N7qteGWbNmERISQnBwMPXq1Yv3pldVpUmTJqm61VyLFi3Inz9/gl0ZXb9+nQ4dOuDn50ft2rU5cOCAY9oHH3yAn58flSpV4ueffwasVnMNGzYkKirK0+GnH97u/jupf87DQHhiyIfDhw87Ho8fP1737t2brMtP68wwEAnLCMNArFmzxjFcwpIlS7RWrVou50vtw0Coqq5YsUIXLFigjzzySLzzTJw4UV966SVVVf3222+1ffv2qqq6bds2DQkJ0WvXrum+ffu0XLlyjvjNMBBmGIg7cv78eQYNGsTkyZNZt24d1apVc9k/lXHT8REjuL4jeYeByF7Zn3veesvt+c0wECk/DITzsuvUqcPhw4ddfjepfRgIsLoUiu39IT7z589n8ODBgNXjds+ePVFV5s+fz5NPPkn27NkpW7Ysfn5+rF+/nrp165phIJIowyciVWXu3Lm8+uqrHD9+nJ49e1K+fHlvh2W4wQwDYfHmMBBffPEFDz30kMtpqX0YCHc5DxeRJUsW8uXLR0REBEeOHLmlqyLnYSTMMBBJk6ETkarSpk0bfvrpJ6pVq8aCBQuoUSNV9oCRKiWl5pKczDAQt/LWMBArV67kiy++4M8//3Q5Pb0NA5EUZhiIpMmQiSgyMpKsWbMiItx33300adKE7t27O3YkRuoWOwzElStXePDBB5k4cSK9e/cmICCA1atX3zKvq2EgqlSpckfrvdNhIAYNGgTcHAYipYcGcB4GImvWrJQpU+aWYSCWLFnCoEGDaNq0Ke+++y7r16/n119/Zd68eUyYMMHRw7SzLVu28Pzzz7N06VJH7+JxxQ4DkSlTpluGgciZMyeNGzdOcBiIDz744JZlxQ4DsWHDBgoUKMBzzz0X7zAQyV0jih0uwtfXl6ioKM6fP0+hQoUSHUbCDAPhvgzXam7VqlWEhIQwf/58APr160evXr1MEkqDzDAQlpQeBuLgwYO0adOGr7/+OsH+1FL7MBDuatWqFV999RVgXRts0qSJ417C2bNnc/36dfbv38+ePXuoVasWYIaBSKoMk4hOnTpF586duf/++7l+/bqpLqcTZhiIlB8GYsiQIURERNC9e3dCQ0PjPZ2d2oeBACvxt2vXjl9//RVfX19HE+x3332XBQsWANCtWzciIiLw8/Nj7NixjBw5EoDAwEDat29PQEAALVq0YOLEiY4DWjMMRBJ5u9leUv/upPn2N998owUKFNCsWbPqW2+9pZcvX070PYZrqaH5tpE2HD16VB944AFvh+EVjz/+uO7atctr6zfNt1OhqKgogoKCmDRpEgEBAd4OxzAyBOdhIDJSD9xmGIikS5en5i5fvszAgQP57LPPAHjmmWf4/fffTRIyjBRmhoEw3JHmakTX5ebIrLvO7KJSwVsv5C5atIiePXvy33//0adPH+DW1k7G3VNVU6aGkUpZZ+HSljRXI3Iu4koFK/FwuYcBq+lkmzZtePTRR8mVKxerV69m3LhxXokxPfPx8SEiIiJN/tgNI71TVSIiItJcs3FJazuUCnlz6J4LV297ffXq1Tz00EO888479O3bl2zZsnkhuvQvMjKSw4cPu7yHwzAM7/Px8cHX1/e2puMislFVU+Ud+2k6Ea1fv56//vrLcQouIiIi3pvrDMMwMrLUnIg8empORFqIyC4R2SsiA11Mzy4ic+zpf4tIGXeWe+7cObp3706dOnUYO3Ysly9fBjBJyDAMIw3yWCISkczAROAhIADoKCJxm611A86qqh/wMfBhYsu9GBmNv78/kydPpnfv3vz777+3dK9iGIZhpC2ebDVXC9irqvsARGQ20Bpwvu27NTDYfjwPmCAiogmcLzx5LZLqQSVZsmQJ1apV80zkhmEYRorxZCIqARxyen4YqB3fPKoaJSLngULAaeeZRORFIHZgk+thYWFb43b3n0EVJk5ZZWCmLG4yZXGTKYubEu600IvSxH1EqjoFmAIgImGp9YJbSjNlcZMpi5tMWdxkyuImEQnzdgzx8WRjhSNASafnvvZrLucRkSxAPiDCgzEZhmEYqYwnE9EGoIKIlBWRbMCTwII48ywAOtuP2wK/JXR9yDAMw0h/PHZqzr7m0xP4GcgMfKmq20RkCFYvsAuAL4CvRWQvcAYrWSVmiqdiToNMWdxkyuImUxY3mbK4KdWWRZq7odUwDMNIX9JcX3OGYRhG+mISkWEYhuFVqTYReap7oLTIjbLoKyLbRWSLiPwqIqW9EWdKSKwsnOZ7QkRURNJt0113ykJE2tu/jW0i8k1Kx5hS3NhGSonIShH5x95OHvZGnJ4mIl+KyEkR2RrPdBGRT+xy2iIiqaNXAG8PEevqD6txw/+AckA2YDMQEGee7sAk+/GTwBxvx+3FsrgfyGk/fiUjl4U9Xx5gNbAOqOHtuL34u6gA/AMUsJ8X9XbcXiyLKcAr9uMA4IC34/ZQWTQEqgFb45n+MLAUEKAO8Le3Y1bVVFsjcnQPpKo3gNjugZy1Br6yH88Dmkr6HK0t0bJQ1ZWqesV+ug7rnq30yJ3fBcBQrH4L0/NYFe6UxQvARFU9C6CqJ1M4xpTiTlkoEDtUbD7gaArGl2JUdTVWC+T4tAZmqGUdkF9EiqdMdPFLrYnIVfdAJeKbR1WjgNjugdIbd8rCWTesI570KNGysE81lFTVxSkZmBe487uoCFQUkTUisk5EWqRYdCnLnbIYDDwjIoeBJUCvlAkt1Unq/iRFpIkufgz3iMgzQA2gkbdj8QYRyQSMBZ7zciipRRas03ONsWrJq0UkWFXPeTMoL+kITFfVj0SkLtb9i0GqGuPtwIzUWyMy3QPd5E5ZICIPAG8DrVT1egrFltISK4s8QBCwSkQOYJ0DX5BOGyy487s4DCxQ1UhV3Q/sxkpM6Y07ZdEN+A5AVf8CfLA6RM1o3NqfpLTUmohM90A3JVoWIlIVmIyVhNLrdQBIpCxU9byqFlbVMqpaBut6WStVTbWdPd4Fd7aRn7BqQ4hIYaxTdftSMMaU4k5ZHASaAohIZaxEdCpFo0wdFgCd7NZzdYDzqnrM20GlylNz6rnugdIcN8tiNJAbmGu31zioqq28FrSHuFkWGYKbZfEz0FxEtgPRQH9VTXdnDdwsi37AVBF5DavhwnPp8cBVRL7FOvgobF8Pew/ICqCqk7Cujz0M7AWuAF28E+mtTBc/hmEYhlel1lNzhmEYRgZhEpFhGIbhVSYRGYZhGF5lEpFhGIbhVSYRGYZhGF5lEpGRKolItIiEO/2VSWDeS8mwvukist9e1yb77vukLuNzEQmwH78VZ9rau43RXk5suWwVkYUikj+R+UPTa0/TRvphmm8bqZKIXFLV3Mk9bwLLmA4sUtV5ItIcGKOqIXexvLuOKbHlishXwG5VHZ7A/M9h9UDeM7ljMYzkYmpERpogIrntsZY2ici/InJbr9siUlxEVjvVGBrYrzcXkb/s984VkcQSxGrAz35vX3tZW0XkVfu1XCKyWEQ22693sF9fJSI1RGQkkMOOY5Y97ZL9f7aIPOIU83QRaSsimUVktIhssMeJecmNYvkLu8NKEallf8Z/RGStiFSyexkYAnSwY+lgx/6liKy353XVe7lhpCxvj0Nh/syfqz+sngDC7b8fsXoByWtPK4x1Z3hsjf6S/b8f8Lb9ODNW33OFsRJLLvv1AcC7LtY3HWhrP24H/A1UB/4FcmH1XLENqAo8AUx1em8++/8q7PGPYmNymic2xseBr+zH2bB6Qs4BvAgMsl/PDoQBZV3Eecnp880FWtjP8wJZ7McPAN/bj58DJji9fwTwjP04P1b/c7m8/X2bv4z9lyq7+DEM4KqqhsY+EZGswAgRaQjEYNUEigHHnd6zAfjSnvcnVQ0XkUZYA6Gtsbs/yoZVk3BltIgMwuqDrBtW32Q/quplO4YfgAbAMuAjEfkQ63TeH0n4XEuB8SKSHWgBrFbVq/bpwBARaWvPlw+rg9L9cd6fQ0TC7c+/A/jFaf6vRKQCVhc2WeNZf3OglYi8bj/3AUrZyzIMrzCJyEgrngaKANVVNVKs3rV9nGdQ1dV2onoEmC4iY4GzwC+q2tGNdfRX1XmxT0SkqauZVHW3WOMePQwME5FfVXWIOx9CVa+JyCrgQaAD1iBuYI2Y2UtVf05kEVdVNVREcmL1rdYD+ARrMMCVqvq43bBjVTzvF+AJVd3lTryGkRLMNSIjrcgHnLST0P1A6bgziEhp4ISqTgU+xxoyeR1QX0Rir/nkEpGKbq7zD+AxEckpIrmwTqv9ISL3AldUdSZWh7PVXLw30q6ZuTIHq7PJ2NoVWEnlldj3iEhFe50uqTUib2+gn9wcBiW2O//nnGa9iHWKMtbPQC+xq4di9dxuGF5lEpGRVswCaojIv0AnYKeLeRoDm0XkH6zaxnhVPYW1Y/5WRLZgnZbzd2eFqroJ69rReqxrRp+r6j9AMLDePkX2HjDMxdunAFtiGyvEsRxr8MIVag1tDVbi3A5sEpGtWMN6JHjGwo5lC9agb6OAD+zP7vy+lUBAbGMFrJpTVju2bfZzw/Aq03zbMAzD8CpTIzIMwzC8yiQiwzAMw6tMIjIMwzC8yiQiwzAMw6tMIjIMwzC8yiQiwzAMw6tMIjIMwzC86v8BWdziy9Jl2bMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot ROC curve\n",
    "plt.figure()\n",
    "plt.plot(fpr[\"micro\"], tpr[\"micro\"],\n",
    "         label='micro-average ROC curve (area = {0:0.2f})'\n",
    "               ''.format(roc_auc[\"micro\"]))\n",
    "for i in range(n_classes):\n",
    "    plt.plot(fpr[i], tpr[i], label='ROC curve of class {0} (area = {1:0.2f})'\n",
    "                                   ''.format(i, roc_auc[i]))\n",
    "\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Some extension of Receiver operating characteristic to multi-class')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7185048b-2c46-4483-ae7d-54f32f8c0474",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Validación</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "028aa545-cb35-4cbb-95c0-43f6e046cc4f",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://imgs.xkcd.com/comics/ai_methodology.png\"/>\n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "568f407a-9fed-4656-bb9b-7e3173e5afcc",
   "metadata": {},
   "source": [
    "###   <span style=\"color:blue\">Separación de datos</span>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fc82a47-e884-4f7c-b52e-c411a3ff0ea9",
   "metadata": {},
   "source": [
    "el clásico caso del \"train test split\". Separamos los datos en dos \"subconjuntos\": Uno para hacer el ajuste del modelo, y otro en el que se harán comparaciónes a nivel de predicciones del modelo ya entrenado.\n",
    "\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://www.mltut.com/wp-content/uploads/2020/05/k-fold-cross-validation.png\"/>\n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "La idea de esto es poder trabajar el modelo en un \"ambiente desconocido pero controlado\". Logramos observar el modelo con datos que no había visto antes, pero no son completamente desconocidos para el científico.\n",
    "\n",
    "Esta es la forma más sencilla de validación. Ahora, tiene debilidades.\n",
    "\n",
    "La más grande de esta es el sesgo de muestra. Si casualmente nuestro subconjunto de prueba no tiene datos lo suficientemente variados, corremos el riesgo de que las métricas que usemos para evaluar el modelo tenga una \"precisión no merecida\". \n",
    "\n",
    "Podemos intentar arreglar esto separando un poco más el dataset. Dejando tres subconjuntos: Uno de entrenamiento, uno de validación y uno de prueba\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://miro.medium.com/max/1400/1*wkaZ-efb_e2ff_lXSWOBfw.png\"/>\n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "Esto puede ciertamente mitigar el sesgo de muestra, pero no es la mejor alternativa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5c0aa0ae-ef45-4698-b9e7-f05cd1f354f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='0'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEOCAYAAACQMUyOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAScElEQVR4nO3df6xkdXnH8fdH0ChIFcK6QWC7hFATai2YLVppGxVo8UeEpo2FqF2rCf2jttqYtKv9gyZNE5r6q0lbKwplmxJso1iJmsqGaomJpS64lQVUqKJlu7BrrILVRClP/7gHuXt3Zu9l58yc+c68X8nNzJxz7pxn5rn3c889c873pKqQJLXnKUMXIEk6Oga4JDXKAJekRhngktQoA1ySGmWAS1Kjjp3lyk4++eTaunXrLFepEW6//fZvVdWmvp7Pvs6PPntrX+fHuL7ONMC3bt3K7t27Z7lKjZDkG30+n32dH3321r7Oj3F9dReKJDXKAJekRhngktQoA1ySGmWAS1KjDPAlluTaJAeS7F017aQku5Lc292eOGSNevLG9PXPk3w5yZeSfCzJswcsUT0xwJfbdcDFa6btAG6pqrOAW7rHast1HN7XXcDzq+oFwFeBd8y6KPXPAF9iVXUr8O01ky8Bdnb3dwKXzrImTW5UX6vq5qp6tHv4b8BpMy9MvZvpiTzr2brjkxM/x/1XvaqHSpba5qra391/ENg8aqEkVwBXAGzZsuWIT2hf586bgH8YNcO+tsUtcI1VK5drGnnJpqq6uqq2VdW2TZt6OytfU5bkj4BHgetHzbevbTHAtdZDSU4B6G4PDFyPepLkjcCrgdeV11JcCAa41roJ2N7d3w58fMBa1JMkFwN/ALymqr4/dD3qhwG+xJLcAHweeF6SB5K8GbgKuCjJvcCF3WM1ZExf/xI4AdiVZE+Svxm0SPVirj7E1GxV1eVjZl0w00LUqzF9vWbmhWjq3AKXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElq1LoBnuT0JJ9JcneSu5K8tZv+x0n2dUNT7knyyumXK0l63EaGk30UeHtV3ZHkBOD2JLu6ee+tqndNrzxJ0jjrBnh3gdv93f1HktwDnDrtwiRJR/ak9oEn2QqcC9zWTXpLki8luTbJiX0XJ0kab8MBnuSZwEeBt1XVw8D7gTOBc1jZQn/3mO+7IsnuJLsPHjw4ecWSJGCDAZ7kqayE9/VVdSNAVT1UVf9XVY8BHwTOG/W9VXV1VW2rqm2bNm3qq25JWnobOQolrFxP756qes+q6aesWuxXgb39lydJGmcjR6GcD7wBuDPJnm7aO4HLk5wDFHA/8NtTqE+SNMZGjkL5HJARsz7VfzmSpI3yTExJapQBLkmNMsAlqVEGuLRguhPrDiTZu2raSUl2Jbm3u/XEuwVggEuL5zrg4jXTdgC3VNVZwC3dYzXOAJcWTFXdCnx7zeRLgJ3d/Z3ApbOsSdNhgEvLYXM3MB3Ag8DmIYtRPwxwaclUVbFyAt5hHLuoLQa4tBweenz4i+72wKiFHLuoLQa4tBxuArZ397cDHx+wFvXEANdhkvx+d/m8vUluSPL0oWvSxiW5Afg88LwkDyR5M3AVcFGSe4ELu8dq3EYGs9ISSXIq8HvA2VX1gyT/CFzGyqFpakBVXT5m1gUzLURT5xa4RjkWeEaSY4HjgP8euB5JIxjgOkRV7QPeBXyTlSstfbeqbh62KkmjGOA6RHeK9SXAGcBzgeOTvH7Ech5uJg3MANdaFwJfr6qDVfUj4EbgJWsX8nAzaXgGuNb6JvDiJMd1l9O7ALhn4JokjWCA6xBVdRvwEeAO4E5WfkauHrQoSSN5GKEOU1VXAlcOXYekI3MLXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGrVugCc5PclnktzdXaXlrd30k5LsSnJvd3vi9MuVJD1uI1vgjwJvr6qzgRcDv5PkbGAHcEtVnQXc0j2WJM3IugFeVfur6o7u/iOsjEx3KitjRu/sFtsJXDqlGiVJIzypfeBJtgLnArcBm6tqfzfrQWBzv6VJko5kwwGe5JnAR4G3VdXDq+dVVQE15vu8coskTcGGAjzJU1kJ7+ur6sZu8kNJTunmnwIcGPW9XrlFkqZjI0ehBLgGuKeq3rNq1k3A9u7+duDj/ZcnSRpnIxd0OB94A3Bnkj3dtHcCVwH/mOTNwDeA106lQknSSOsGeFV9DsiY2Rf0W44kaaM8E1OSGmWAS0skye93Z1TvTXJDkqcPXZOOngEuLYkkpwK/B2yrqucDxwCXDVuVJmGAS8vlWOAZSY4FjgP+e+B6NAEDXFoSVbUPeBfwTWA/8N2qunn1Mp541xYDXFoS3YihlwBnAM8Fjk/y+tXLeOJdWwxwaXlcCHy9qg5W1Y+AG4GXDFyTJmCAS8vjm8CLkxzXnWF9ASuji6pRBri0JKrqNuAjwB3Anaz8/l89aFGayEZOpZe0IKrqSuDKoetQP9wCl6RGGeCS1CgDXJIaZYBLUqMMcB0mybOTfCTJl5Pck+Tnh65J0uE8CkWj/AXwz1X160mexsqYGZLmjAGuQyR5FvBLwBsBquqHwA+HrEnSaO5C0VpnAAeBv03yxSQfSnL80EVJOpwBrrWOBV4IvL+qzgX+F9ixdiFHrZOGZ4BrrQeAB7rTrmHl1OsXrl3IUeuk4RngOkRVPQj8V5LndZMuAO4esCRJY/ghpkb5XeD67giUrwG/NXA9kkYwwHWYqtoDbBu6DklH5i4USWqUAS5JjTLAJalRBrgkNcoAl6RGrRvgSa5NciDJ3lXT/jjJviR7uq9XTrdMSdJaG9kCvw64eMT091bVOd3Xp/otS5K0nnUDvKpuBb49g1okSU/CJPvA35LkS90ulhN7q0iStCFHG+DvB84EzgH2A+8et6Cj1knSdBxVgFfVQ1X1f1X1GPBB4LwjLOuodZI0BUcV4ElOWfXwV4G945aVJE3HuoNZJbkBeClwcpIHgCuBlyY5ByjgfuC3p1eipL4keTbwIeD5rPz+vqmqPj9oUTpq6wZ4VV0+YvI1U6hF0vR5weoF4nCy0pLwgtWLx1PppeXhBasXjAEuLY91L1jtYb9tMcCl5bHuBas97LctBri0JLxg9eLxQ0xpuXjB6gVigEtLxAtWLxZ3oUhSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4RkpyTHfVlk8MXYuk0QxwjfNW4J6hi5A0ngGuwyQ5DXgV8KGha5E0ngGuUd4H/AHw2MB1SDoCA1yHSPJq4EBV3b7Ocl78VhqYAa61zgdek+R+4MPAy5P8/dqFvPitNDwDXIeoqndU1WlVtRW4DPiXqnr9wGVJGsEAl6RGeVFjjVVVnwU+O3AZksZYdws8ybVJDiTZu2raSUl2Jbm3uz1xumVKktbayC6U64CL10zbAdxSVWcBt3SPJUkztG6AV9WtwLfXTL4E2Nnd3wlc2m9ZkqT1HO2HmJuran93/0Fgc0/1SJI2aOKjUKqqgBo33xM+JGk6jjbAH0pyCkB3e2Dcgp7wIc0PR5lcLEcb4DcB27v724GP91OOpClzlMkFspHDCG8APg88L8kDSd4MXAVclORe4MLusaQ55iiTi2fdE3mq6vIxsy7ouRZJ0/U+VkaZPGHgOtQTT6WXloCjTC4mA1xaDo4yuYAMcGkJOMrkYjLAJalRjkYoLRlHmVwcboFLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEOZrXG1h2fnOj777/qVQtRg6T55xa4JDXKAJekRhngktQoA1ySGmWAS1KjDHAdIsnpST6T5O4kdyV569A1SRrNwwi11qPA26vqjiQnALcn2VVVdw9dmKRDuQWuQ1TV/qq6o7v/CHAPcOqwVUkaxQDXWEm2AucCtw1ciqQR3IWikZI8E/go8LaqenjE/CuAKwC2bNky4+qevEnPboXJz3D1DFv1baIt8CT3J7kzyZ4ku/sqSsNK8lRWwvv6qrpx1DJVdXVVbauqbZs2bZptgZKAfrbAX1ZV3+rheTQHkgS4Brinqt4zdD2SxnMfuNY6H3gD8PLuP6s9SV45dFGanIeILp5Jt8ALuDlJAR+oqqt7qEkDqqrPARm6Dk2Fh4gumEkD/Beqal+S5wC7kny5qm5dvUBrH3ZJi6qq9gP7u/uPJHn8EFEDvFET7UKpqn3d7QHgY8B5I5bxwy5pzniI6GI46gBPcnz3bxhJjgd+GdjbV2GSpuNIh4gmuSLJ7iS7Dx48OEyB2rBJtsA3A59L8h/AvwOfrKp/7qcsSdOw3iGi/sfclqPeB15VXwN+tsdaJE2Rh4guHg8jlJaHh4guGE+ll5aEh4guHrfAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoB7OS1KytOz458XPcf9Wrmq3BLXBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGTRTgSS5O8pUk9yXZ0VdRGpZ9XVz2drEcdYAnOQb4K+AVwNnA5UnO7qswDcO+Li57u3gm2QI/D7ivqr5WVT8EPgxc0k9ZGpB9XVz2dsFMEuCnAv+16vED3TS1zb4uLnu7YKZ+RZ4kVwBXdA+/l+QrEzzdycC3jri+P5vg2XuoYQbr76OGn5y0gJ77CsO/r4vyszVRbxewr/NQQx8/WyP7OkmA7wNOX/X4tG7aIarqauDqCdbzY0l2V9W2Pp7LGsaaeV9h+Pd16PXPqIZ1e7tofZ2HGqa5/kl2oXwBOCvJGUmeBlwG3NRPWRqQfV1c9nbBHPUWeFU9muQtwKeBY4Brq+qu3irTIOzr4rK3i2eifeBV9SngUz3VshG9/Ws3gYWvYYC+wvDv69DrhxnU4O/sYq0/VTWt55YkTZGn0ktSowxwSWqUAS5JjZr6iTx9SPIc4HzgucAPgL3A7qp6bIY1PAX42dU1VNWBGa5/8Pegb/Pwmobua1fD4O9Dn+bh9SxLX+f6Q8wkLwN2ACcBXwQOAE8Hfgo4E/gI8O6qeniKNZwJ/CFwIXAvcHBVDd8HPgDsnNYP5zy8B32bh9c0dF+7GgZ/H/o0D69n6fpaVXP7Bfw5sGXMvGOBS4Ffm3INNwC/RPfHbs285wBvA7Yv8nuwiK9p6L7Oy/uwaK9n2fo611vgkqTxmvwQM8klSV40cA3bkjx3wPUP/h70bR5e09B97WoY/H3o0zy8nkXtaxMfYo7wIuBnkhxbVa8YqIbfBV6Q5KtV9RsDrH8e3oO+zcNrGrqvMB/vQ5/m4fUsZF/dhTKhJCdU1SND16F+2dfFtGh9nfsAT/ITwKaq+s81019QVV+aUQ3PAi7micHv9wGfrqrvzGL94yS5qKp2DVnD0bKvR9Zqb+3rkfXd17neB57ktcCXgY8muSvJz62afd2MavhN4A7gpcBx3dfLgNu7eUO6ZuD1HxX7uiHN9da+bkivfZ3rLfAke4BXVNX+JOcBfwe8o6o+luSLVXXuDGr4CvCitX+9k5wI3FZVPzXl9Y8brznAy6vq+Gmufxrs64/XtVC9ta8/XtfM+jrvH2IeU1X7Aarq37sD5D+R5HRgVn95MmZdj3Xzpu0XgdcD31szPaxcpLZF9nXFovXWvq6YWV/nPcAfSXLm4/vTur/sLwX+CfjpGdXwp8AdSW7miQvCbgEuAv5kBuv/N+D7VfWva2f0cL3CodjXFYvWW/u6YmZ9nfddKC8EHq6q+9ZMfyrw2qq6Pklqii8iSYBnA7/C4R+K/M/jy0yrho0897Tfg77Z1ydqWKTe2tcnaphZX/s4nXNaX8BnWTl+c8ua6U8DLgB2Am8csIaXT7uGoddvXxe7Bvvadg3zvgX+dOBNwOuAM4DvsDIozDHAzcBfV9UXF7mGodc/DfPwmqyhf/PwepathrkO8NW6f8NOBn5QAx3POXQNQ69/GubhNVlD/+bh9SxDDc0EuCTpUHN9Io8kaTwDXJIaZYA/CUkuTvKVJPcl2TF0PeqHfV1My9BX94FvUJJjgK+yckLAA8AXgMur6u5BC9NE7OtiWpa+ugW+cecB91XV16rqh8CHgUsGrkmTs6+LaSn6aoBv3Kk8cWourPxVP3XMsmqHfV1MS9FXA1ySGmWAb9w+4PRVj0/rpqlt9nUxLUVfDfCN+wJwVpIzkjwNuAwYN+6v2mFfF9NS9HXeh5OdG1X1aJK3AJ9mZUyDa6vqroHL0oTs62Jalr56GKEkNcpdKJLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RG/T8CYT62hxfHFwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from imblearn.datasets import make_imbalance\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "iris = load_iris()\n",
    "X, y = make_imbalance(\n",
    "    iris.data,\n",
    "    iris.target,\n",
    "    sampling_strategy={0: 10, 1: 10, 2: 50},\n",
    "    random_state=16,\n",
    ")\n",
    "X, X_test, y, y_test = train_test_split(X,y,test_size=0.2,train_size=0.8)\n",
    "X_train, X_val, y_train, y_val = train_test_split(X,y,test_size = 0.25,train_size =0.75)\n",
    "\n",
    "fig, axes = plt.subplots(1,3)\n",
    "pd.DataFrame(y_train).value_counts().plot(kind='bar', ax=axes[0])\n",
    "pd.DataFrame(y_val).value_counts().plot(kind='bar', ax=axes[1])\n",
    "pd.DataFrame(y_test).value_counts().plot(kind='bar', ax=axes[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "849adb4e-53d5-4529-bbb5-af044cae6067",
   "metadata": {},
   "source": [
    "###   <span style=\"color:blue\">Leave One Out</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df7598cf-e0b7-46a5-9136-eaf6a91af051",
   "metadata": {},
   "source": [
    "De ahora en adelante no hacemos una sola separación de datos, sino múltiples.\n",
    "\n",
    "En este caso, tomando en cuenta su nombre, el subconjunto de prueba solo consta de un elemento. Cada observación eventualmente es utilizada como elemento de prueba. Esto significa que se hacen $N$ separaciones de datos y $N$ ajustes de modelo. Evaluamos cada uno de estos modelos y luego podemos, por ejemplo, obtener el promedio de precisión de estos para saber qué tan bien está generalizando el modelo.\n",
    "\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"../Imagenes/Leave-One-Out-Cross-Validation.png\"/>\n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9a6dbc22-2b04-4f6a-a751-dc1926c5a688",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.960\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import LeaveOneOut\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "loo = LeaveOneOut()\n",
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "X, y = load_iris(return_X_y=True)\n",
    "y_true, y_pred = list(), list()\n",
    "for train_ix, test_ix in loo.split(X):\n",
    "    X_train, X_test = X[train_ix, :], X[test_ix, :]\n",
    "    y_train, y_test = y[train_ix], y[test_ix]\n",
    "    # fit model\n",
    "    knn.fit(X_train, y_train)\n",
    "    # evaluate model\n",
    "    yhat = knn.predict(X_test)\n",
    "    # store\n",
    "    y_true.append(y_test[0])\n",
    "    y_pred.append(yhat[0])\n",
    "acc = accuracy_score(y_true, y_pred)\n",
    "print('Accuracy: %.3f' % acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fddec18-a9e6-489f-8f09-a9247506bfd1",
   "metadata": {},
   "source": [
    "Este método asegura la menor cantidad de sesgo posible, ya que revisamos la consistencia del modelo sobre TODOS Y CADA UNO DE LOS DATOS, pero tiene dos problemas:\n",
    "\n",
    "- Sufre con datos con una alta varianza.\n",
    "- Es computacionalmente costoso, así que con datos de gran tamaño no es recomendable."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "570363c7-44c4-496e-aa29-452fec8510b7",
   "metadata": {},
   "source": [
    "###   <span style=\"color:blue\">K Folds</span>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "774a6e87-31aa-4f92-bbca-3a47affae55f",
   "metadata": {},
   "source": [
    "La versión generalizada de Leave One out. Separamos el dataset de en en múltiples subconjuntos de \"casi\" tamaños iguales vamos iterando sobre estos. En la iteración 1, la muestra 1 será el dataset de prueba, mientras que el resto de muestras se usarán como datos de entrenamiento. Este proceso se repetirá $K$ veces, donde $K$ es el número de muestras.\n",
    "\n",
    "K folds es igual a Leave One Out cuando $K=N$\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://www.mltut.com/wp-content/uploads/2020/05/cross-validation.png\"/>\n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "05a05e99-1ac2-45a3-b9ee-162c4f1b2f2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold \n",
    "from sklearn.svm import SVC\n",
    " \n",
    "#Implementing cross validation\n",
    "k = 5\n",
    "kf = KFold(n_splits=k, shuffle=True, random_state=42)\n",
    "rbf = SVC(kernel='rbf', gamma=0.5, C=0.1)\n",
    "poly = SVC(kernel='poly', degree=3, C=1)\n",
    " \n",
    "acc_score_rbf = []\n",
    "acc_score_poly = []\n",
    "\n",
    "X, y = load_iris(return_X_y=True)\n",
    "\n",
    "for train_ix , test_ix in kf.split(X):\n",
    "    X_train, X_test = X[train_ix, :], X[test_ix, :]\n",
    "    y_train, y_test = y[train_ix], y[test_ix]\n",
    "     \n",
    "    rbf.fit(X_train,y_train)\n",
    "    poly.fit(X_train, y_train)\n",
    "    \n",
    "    pred_rbf = rbf.predict(X_test)\n",
    "    pred_poly = poly.predict(X_test)\n",
    "     \n",
    "    acc_rbf = accuracy_score(pred_rbf, y_test)\n",
    "    acc_score_rbf.append(acc_rbf)\n",
    "    \n",
    "    acc_poly = accuracy_score(pred_poly, y_test)\n",
    "    acc_score_poly.append(acc_poly)\n",
    "     \n",
    "avg_acc_score_rbf = sum(acc_score_rbf)/k\n",
    "avg_acc_score_poly = sum(acc_score_poly)/k\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "493f1c56-b315-4db9-85de-620f079c4c1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy of each fold in rbf - [1.0, 1.0, 0.9, 0.9, 0.9333333333333333]\n",
      "Avg rbf accuracy : 0.9466666666666667\n"
     ]
    }
   ],
   "source": [
    "print('accuracy of each fold in rbf - {}'.format(acc_score_rbf))\n",
    "print('Avg rbf accuracy : {}'.format(avg_acc_score_rbf))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8f27b20b-7086-4f78-a47d-3bd7bb5df3db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy of each fold in poly - [1.0, 1.0, 0.9666666666666667, 0.9333333333333333, 0.9666666666666667]\n",
      "Avg poly accuracy : 0.9733333333333334\n"
     ]
    }
   ],
   "source": [
    "print('accuracy of each fold in poly - {}'.format(acc_score_poly))\n",
    "print('Avg poly accuracy : {}'.format(avg_acc_score_poly))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bd9e638-6ac4-4bae-9107-0c5c3458ac1a",
   "metadata": {},
   "source": [
    "K folds sacrifica algo de sesgo a favor de resistir a los datasets con alta varianza"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c7d6171-af2b-4673-b63a-835b0176edd0",
   "metadata": {},
   "source": [
    "####   <span style=\"color:blue\">K folds estratificados</span>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eaf23f3-8541-4c99-9099-68308b42978b",
   "metadata": {},
   "source": [
    "Una variación de K folds en la cual se usa muestreo estratificado en lugar de muestreo aleatorio. Se generan las mismas muestras pero buscando representar adecuadamente las proporciones de cada clase\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://uedufy.com/wp-content/uploads/2021/12/stratified-sampling-2.png\"/>\n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c768286d-8d15-4f47-bbc9-90a02ffccd6a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy of each fold in poly - [0.9666666666666667, 0.9666666666666667, 0.9666666666666667, 0.9666666666666667, 0.9666666666666667]\n",
      "Avg poly accuracy : 0.9666666666666666\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import StratifiedKFold \n",
    "\n",
    "X, y = make_imbalance(\n",
    "    iris.data,\n",
    "    iris.target,\n",
    "    sampling_strategy={0: 15, 1: 15, 2: 50},\n",
    "    random_state=16,\n",
    ")\n",
    "\n",
    "k = 5\n",
    "kf = StratifiedKFold(n_splits=k, random_state=None,shuffle=False)\n",
    "poly = SVC(kernel='poly', degree=3, C=1)\n",
    " \n",
    "acc_score_poly = []\n",
    "\n",
    "for train_ix , test_ix in kf.split(X,y):\n",
    "    X_train, X_test = X[train_ix, :], X[test_ix, :]\n",
    "    y_train, y_test = y[train_ix], y[test_ix]\n",
    "     \n",
    "    poly.fit(X_train, y_train)\n",
    "    \n",
    "    pred_poly = poly.predict(X_test)\n",
    "    \n",
    "    acc_score_poly.append(acc_poly)\n",
    "     \n",
    "avg_acc_score_poly = sum(acc_score_poly)/k\n",
    "\n",
    "print('accuracy of each fold in poly - {}'.format(acc_score_poly))\n",
    "print('Avg poly accuracy : {}'.format(avg_acc_score_poly))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4177e1b7-8f3d-495a-b83a-cc3a6a75a6f0",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Recomendaciones</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b42d2902-e7de-45f3-a258-4c63f1731049",
   "metadata": {},
   "source": [
    "Aquí vimos implementaciones \"manuales\" de como funcionan las metodologías de estos casos de validación cruzada, pero esto en realidad podemos sacarlo en unas cuantas lineas de código"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "68054d57-427e-4b2f-af25-d4020407878f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9875\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "print(cross_val_score(poly, X, y, cv=kf).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b02189-57e4-47af-9524-7136a422e17c",
   "metadata": {},
   "source": [
    "Además, podemos mezclar los metodos de validación cruzada con la separación de datos tradicional\n",
    "\n",
    "<figure> \n",
    "<center>\n",
    "<img src=\"https://scikit-learn.org/stable/_images/grid_search_cross_validation.png\"/>\n",
    "</center>\n",
    "</figure>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}