05_performance_assessment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "866dc4ef",
   "metadata": {},
   "source": [
    "# Assessment\n",
    "\n",
    "Click to open in:  \\[[GitHub](https://github.com/DIDSR/wsi_processing_toolbox/05_performance_assessment.ipynb)\\]\n",
    "\n",
    "Developer: Seyed.kahaki@fda.hhs.gov"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9ac3e0e",
   "metadata": {},
   "source": [
    "## About this notebook\n",
    "This jupyter notebook can be run on any computer with a standard browser and no prior installation of any programming language is required."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74acc53",
   "metadata": {},
   "source": [
    "### About\n",
    "\n",
    "This demonstration guides you through the process of using WSIToolbox to for performance evaluation of ML models. This is enabled using the get_report method of the UncertaintyAnalysis class in the WSIToolbox. The input to this method are the probability results of the ML model as well as the truth values."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c67cb96",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f84d5598",
   "metadata": {},
   "source": [
    "\n",
    "1. At the first you should import some packages that you need ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "53c9347e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from assessment.uncertainty import Uncertainty_Analysis\n",
    "UncertaintyAnalysis = Uncertainty_Analysis()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f78c9c41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "please enter xls file path, for example ./data/ResultsExample.xlsx: \n",
      "Do you want to perform Bootsrap? (1 = Yes ,  0 = No)\n",
      "Do you want to get AUC based on Delong method, for example 100:\n",
      "Do you want to plot ROC curve (1 = Yes ,  0 = No)\n",
      "Do you want to get AUC based on Delong method? (1 = Yes ,  0 = No)\n",
      "1\n",
      "Please enter a name for this results: \n",
      "####################\n",
      "Results for Test\n",
      "          pred:yes  pred:no\n",
      "true:yes       285       14\n",
      "true:no         79       32\n",
      "Precision:  0.782967032967033\n",
      "Precision_CI:  (0.7406191220140836, 0.8253149439199823)\n",
      "Recall:  0.9531772575250836\n",
      "Recall_CI:  (0.9292315454409011, 0.9771229696092661)\n",
      "Delong Method\n",
      "AUC: 0.8234354756093887\n",
      "AUC COV: 0.0005619434162873773\n",
      "95% AUC CI: [0.77697385 0.8698971 ]\n",
      "Original ROC area: 0.823\n",
      "Bootstrap #1 ROC area: 0.763\n",
      "Bootstrap #2 ROC area: 0.816\n",
      "Bootstrap #3 ROC area: 0.790\n",
      "Bootstrap #4 ROC area: 0.829\n",
      "Bootstrap #5 ROC area: 0.855\n",
      "Bootstrap #6 ROC area: 0.785\n",
      "Bootstrap #7 ROC area: 0.825\n",
      "Bootstrap #8 ROC area: 0.832\n",
      "Bootstrap #9 ROC area: 0.831\n",
      "Bootstrap #10 ROC area: 0.823\n",
      "Bootstrap #11 ROC area: 0.836\n",
      "Bootstrap #12 ROC area: 0.835\n",
      "Bootstrap #13 ROC area: 0.828\n",
      "Bootstrap #14 ROC area: 0.822\n",
      "Bootstrap #15 ROC area: 0.827\n",
      "Bootstrap #16 ROC area: 0.780\n",
      "Bootstrap #17 ROC area: 0.812\n",
      "Bootstrap #18 ROC area: 0.778\n",
      "Bootstrap #19 ROC area: 0.842\n",
      "Bootstrap #20 ROC area: 0.773\n",
      "Bootstrap #21 ROC area: 0.834\n",
      "Bootstrap #22 ROC area: 0.830\n",
      "Bootstrap #23 ROC area: 0.831\n",
      "Bootstrap #24 ROC area: 0.856\n",
      "Bootstrap #25 ROC area: 0.836\n",
      "Bootstrap #26 ROC area: 0.798\n",
      "Bootstrap #27 ROC area: 0.813\n",
      "Bootstrap #28 ROC area: 0.825\n",
      "Bootstrap #29 ROC area: 0.847\n",
      "Bootstrap #30 ROC area: 0.823\n",
      "Bootstrap #31 ROC area: 0.811\n",
      "Bootstrap #32 ROC area: 0.869\n",
      "Bootstrap #33 ROC area: 0.841\n",
      "Bootstrap #34 ROC area: 0.820\n",
      "Bootstrap #35 ROC area: 0.810\n",
      "Bootstrap #36 ROC area: 0.806\n",
      "Bootstrap #37 ROC area: 0.822\n",
      "Bootstrap #38 ROC area: 0.867\n",
      "Bootstrap #39 ROC area: 0.830\n",
      "Bootstrap #40 ROC area: 0.803\n",
      "Bootstrap #41 ROC area: 0.783\n",
      "Bootstrap #42 ROC area: 0.863\n",
      "Bootstrap #43 ROC area: 0.822\n",
      "Bootstrap #44 ROC area: 0.822\n",
      "Bootstrap #45 ROC area: 0.802\n",
      "Bootstrap #46 ROC area: 0.849\n",
      "Bootstrap #47 ROC area: 0.823\n",
      "Bootstrap #48 ROC area: 0.861\n",
      "Bootstrap #49 ROC area: 0.813\n",
      "Bootstrap #50 ROC area: 0.829\n",
      "Bootstrap #51 ROC area: 0.818\n",
      "Bootstrap #52 ROC area: 0.785\n",
      "Bootstrap #53 ROC area: 0.758\n",
      "Bootstrap #54 ROC area: 0.805\n",
      "Bootstrap #55 ROC area: 0.849\n",
      "Bootstrap #56 ROC area: 0.803\n",
      "Bootstrap #57 ROC area: 0.817\n",
      "Bootstrap #58 ROC area: 0.825\n",
      "Bootstrap #59 ROC area: 0.830\n",
      "Bootstrap #60 ROC area: 0.805\n",
      "Bootstrap #61 ROC area: 0.813\n",
      "Bootstrap #62 ROC area: 0.846\n",
      "Bootstrap #63 ROC area: 0.836\n",
      "Bootstrap #64 ROC area: 0.793\n",
      "Bootstrap #65 ROC area: 0.808\n",
      "Bootstrap #66 ROC area: 0.857\n",
      "Bootstrap #67 ROC area: 0.760\n",
      "Bootstrap #68 ROC area: 0.853\n",
      "Bootstrap #69 ROC area: 0.840\n",
      "Bootstrap #70 ROC area: 0.787\n",
      "Bootstrap #71 ROC area: 0.823\n",
      "Bootstrap #72 ROC area: 0.856\n",
      "Bootstrap #73 ROC area: 0.847\n",
      "Bootstrap #74 ROC area: 0.819\n",
      "Bootstrap #75 ROC area: 0.816\n",
      "Bootstrap #76 ROC area: 0.818\n",
      "Bootstrap #77 ROC area: 0.836\n",
      "Bootstrap #78 ROC area: 0.803\n",
      "Bootstrap #79 ROC area: 0.846\n",
      "Bootstrap #80 ROC area: 0.834\n",
      "Bootstrap #81 ROC area: 0.838\n",
      "Bootstrap #82 ROC area: 0.827\n",
      "Bootstrap #83 ROC area: 0.806\n",
      "Bootstrap #84 ROC area: 0.831\n",
      "Bootstrap #85 ROC area: 0.823\n",
      "Bootstrap #86 ROC area: 0.845\n",
      "Bootstrap #87 ROC area: 0.837\n",
      "Bootstrap #88 ROC area: 0.789\n",
      "Bootstrap #89 ROC area: 0.742\n",
      "Bootstrap #90 ROC area: 0.841\n",
      "Bootstrap #91 ROC area: 0.845\n",
      "Bootstrap #92 ROC area: 0.823\n",
      "Bootstrap #93 ROC area: 0.815\n",
      "Bootstrap #94 ROC area: 0.810\n",
      "Bootstrap #95 ROC area: 0.822\n",
      "Bootstrap #96 ROC area: 0.796\n",
      "Bootstrap #97 ROC area: 0.832\n",
      "Bootstrap #98 ROC area: 0.830\n",
      "Bootstrap #99 ROC area: 0.816\n",
      "Bootstrap #100 ROC area: 0.813\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEICAYAAAB25L6yAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXJUlEQVR4nO3de7hddX3n8fdXQoAAcpHoY4BwUEQFKkUjCkpLuSglKji1ilNso7UZiwr66EisTkXq1NRR0D52nKFWmYpClWLrCBVEpVQHbbkpl4ByiRBuijVCwKrod/74/Q6sbPY5Z5/LPvsX8n49z36y1/271+Wz1vqtvXMiM5Ektetxoy5AkjQ5g1qSGmdQS1LjDGpJapxBLUmNM6glqXGP2aCOiOsi4tBR1zFKEfHyiLg9IjZExAEDjH9oRKybo2WviIivz8W8NicRkRGx16jrUFs2yaCOiLURcURPv42CITP3zcxLppjPWD0wFgyp1FH7IPCmzNwuM6/qHbgphsJ0TiaPte0bEZdExH/UE++9EXFeRDy5Z5x9IuILEfGTiLg/Ir4WEQf3jLMwIk6JiO9FxAP1ePpERIwNsPwfR8RWffq/vqffRtspihMj4tq6zHUR8bmI+LUZr5DNyCYZ1JuKBgJiD+C6EdfQtAa20XS9KTO3A/YCtqOcjAGIiKcC3wCuAfYElgCfBy6KiIM68zgXeBnwn4EdgP2BK4DDJ1poDfFDgKzTTtdHgJOAE4Gdgb2BfwCWz2BesxYRW4xiuTOWmZvcC1gLHNHTbwXw9X7jAAcClwP3AfcAp9X+t1F2vA31dRDl5PVu4PvAD4C/BXbozPf367AfAf+tZzmnUA6Cs+qyXl+XfRmwHrgL+CiwsDO/BE4AvgfcD/wZ8FTg/9V5fLY7fs9n7lsrsFX9PAk8ANzcZ9pLO8M3AK8CDgXWAW+r87sLeG1nmq0owXBbXY//C9hmgtpWUELjo8BPgBuAwzvDlwBfAP4duAn4o57lfBi4s74+XPttC/wU+FVnmy2Z5vYdr+v0ug3fV9f3V2v3vcCngR179qV3AtcDPwY+CWxdh42vsz+p064Ffm/QdQb817qe7wReV+vda4J1egnw+k73CcB1ne5PARf0me5jwKX1/RF1He4+zWPuT+t6Ow344mR1dddLff804JfAgdNY3grgFsoxcWvPOv0jYE0ddj3w7Nr/mbWW9ZQLlJd1pjmzrocLKPv8EXXf+Xvgh3UZJ3bG77tPjSzzRrnwGRc9/aC+DHhNfb8d8Pz6fqweGAs6072OEhxPqeOeB3yqDtuHcsC/EFhYD8BfsHFQ/wI4lhKi2wDPAZ4PLKjLWwO8pbO8BP4ReDywL/Az4Ct1+TvUHfEPJlgPE9bamXffg77f8HpwPQScCmwJHA08COxUh59OCdedge2B/wu8f5ID7SHgrXVer6IE9s51+KXA/wS2Bn69HiyH1WGnAt8Enggsppy0/qxT47qeZU1n+47X9ea6TbahXJ0eSQnVxbW2D/fsS9cCu9fP/g3gfT3r7LQ6/W9SguDpU60z4ChKCOxHOQl9ZrJtRicQgScAFwP/2Bl+N50Ta6f/b1GCchtgNfDPMzjmbqKcGJ5D2cef1K+unn1pPKjfAHx/GsvalhKQ4+vwycC+9f3vAncAzwWibrs96j52E+WEuRA4jBLk4/M4k7L/vYBybC6i3EX8aR3/KZQTw4sn26dGlnmjXPiMiy4HzgbKmXP89SATB/WlwHuBXXrmM8ajD+SvACd0up9ed8wFdaOe3Rm2CPg5Gwf1pVPU/hbg853uBF7Q6b4COLnT/SE6odEzrwlr7cx7ukH905718QPKiSYoAfTUzrCDgFsnmPcKylVidPr9K/AaSuD9Eti+M+z9wJn1/c3A0Z1hLwbWdmrsDerpbN8VwG1TbKNjgat69qU3dLqPpt6l8EhQb9sZ/lnK3dak6wz4BLC6M2zvybYZJRAfpAROAlcDSzvDHwKO6jPdM+r4uwJ/DZwzzePthXW/2qV23wC8taeuyYL6XcA3p7G8bSnH9O/Qc8cGXAic1GeaQygnqsd1+p0NnFLfnwn8bWfY83r3A8pd0ycn26dG9dqU26iPzcwdx1+Us/1E/pByENwQEf8WES+ZZNwllKaEcd+nhPST6rDbxwdk5oOU2+Wu27sdEbF3RHwxIu6OiPuAPwd26Znmns77n/bp3m4Gtc7UjzLzoU73g3X5i6lXIRGxPiLWA1+q/SdyR9a9vlPfkvr698y8v2fYrvV9v8+1ZJLlTGf7wqO30ZMi4pyIuKNuo7N49DbqTtNbz48z84E+w6daZxvtT2z8mSdyYmbuADwL2AnYrTPsXsrVZ68nU5qLfkzZX/uNM5k/AC7KzHtr92dqv3EPUa5ou7akhDvTXWZdl6+iXInfFRHnR8Qz6uDdKSfyXkuA2zPzV51+3X0KNl7XewBLxrdL3TZ/wiPHznT3qaHalIN6YJn5vcx8NeVW+i+AcyNiW8pVRq87KRtx3FLKjngPpS3x4QMjIrah3IJutLie7o9RrkCelpmPp+wMMfNPM3Ctc+1eyklj384JcocsD7YmsmtEdD/rUh5pd945IrbvGXZHfd/vc91Z3z9qm01z+/abx5/Xfr9Wt9HxPHob7T5BPQA71eX1Dp9qnd3VZ74DycxrKO3rf9VZxxdTmgZ6vRK4rF5YXAwcGBG79RnvUeo+/krgN+vFxt2U5qz9I2L/OtptlLuXrj155MTzFWC3iFg20IcDMvPCzDySEvA3UO4EoITtU/tMciewe0R0M627T8HG2/12yp3Njp3X9pl5dF3+RPvUSGwWQR0Rx0fE4nq2XV97/4rSLvorSvvUuLOBt0bEnhGxHeUg/rt6lXku8NKIODgiFlKaOqYK3e0p7W0b6lXBH8/Rx5qq1kHcw8affUJ13f01cHpEPBEgInaNiBdPMtkTgRMjYsuI+F3Kw54LMvN2Srvz+yNi64h4FuUK5qzO53p3RCyOiF0oTU7jw+4BnhARO4wvZJrbt5/tKU1pP4mIXSkP+Hq9MSJ2i4idKbfyf9cz/L31a2+HAC8BPjfAOvsssKJ+pW4R8J4p6uz1fyhXgOPfwngvcHBE/PeI2Dkito+IN1MegJ8MkJkXA18GPh8Rz4mIBXW8N0TE6/os41hKM9U+lGcJv07Zjv9S50tdF6+NiAPr1/D2poT5OXWZ36M8jzi7fm1vYd3ux0XEqt4F1jucY2ow/oyybcavlD8OvL3WHhGxV0TsAXyLcvf3jrq/HQq8dLyGPv4VuD8iTo6IbSJii4jYLyKeW2uYaJ8ajVG3vczkxfQfJp5FaWvdQHkafGxnvFMpB/R6Slvs4yjBcHvtfxb1YVpnObfxyLc+7gAOqcNOAc7qqes3KFcEGyg796k9dfa2E38dWNHpfh/w8QnWw1S1TtVG/QbKVd16ylXToTy6/be7HremnAxuoZx81tB5Ut5ne3S/9fFd4EWd4bsBX6R86+NmNm4D3hr4y1rbXfX91p3hn6jrfz3llnc623ej/aSOsy/l2cAGSrvv27rrgY2/9bGeEpCL6rBDKd/6eBflCvo26kOoQdYZsIrStjrtb33UficDl3e696vr9b76eS4BXtgzzUJKqN9EaUP/PiUAl/ZZ5peAD/Xp/8pa9/jzkNfVdX9fne8qNm4vDsrX866jBOodlIDft8+8nwz8c91v1tfPsE/Pfntj/XzXAgd0tuP4dNcDL+9Mcyb1AXCn3xLKRcHdlGahbzJAZoziFbUozUC9il1Pada4dcTlaEgiYi0lIC/uM+xQysl5oKYEaSY2i6aPuRQRL42IRfW27IOUHxesHW1Vkh7LDOrpO4ZHHog9DTguvS2RNEQ2fUhS47yilqTGDeU/pNlll11ybGxsGLOWpMekK6644t7M7PsDsqEE9djYGJdffvkwZi1Jj0kRMeEvU236kKTGGdSS1DiDWpIaZ1BLUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxg3ll4mSBjO26vwJh61dvXweK1HLvKKWpMYZ1JLUOINakhpnUEtS4wxqSWqcQS1JjTOoJalxBrUkNc6glqTGGdSS1DiDWpIaZ1BLUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGjdQUEfEWyPiuoi4NiLOjoith12YJKmYMqgjYlfgRGBZZu4HbAEcN+zCJEnFoE0fC4BtImIBsAi4c3glSZK6Fkw1QmbeEREfBG4DfgpclJkX9Y4XESuBlQBLly6d6zolTWFs1fl9+69dvXyeK9FcG6TpYyfgGGBPYAmwbUQc3zteZp6Rmcsyc9nixYvnvlJJ2kwN0vRxBHBrZv4wM38BnAccPNyyJEnjBgnq24DnR8SiiAjgcGDNcMuSJI2bMqgz81vAucCVwDV1mjOGXJckqZryYSJAZr4HeM+Qa5Ek9eEvEyWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMYZ1JLUOINakhpnUEtS4wxqSWqcQS1JjTOoJalxBrUkNc6glqTGGdSS1DiDWpIaZ1BLUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMYZ1JLUOINakho3UFBHxI4RcW5E3BARayLioGEXJkkqFgw43keAL2XmKyJiIbBoiDVJkjqmDOqI2AH4DWAFQGb+HPj5cMuSJI0b5Ip6T+CHwCcjYn/gCuCkzHygO1JErARWAixdunSu65Q2CWOrzu/bf+3q5fNciR5LBmmjXgA8G/hYZh4APACs6h0pM8/IzGWZuWzx4sVzXKYkbb4GCep1wLrM/FbtPpcS3JKkeTBlUGfm3cDtEfH02utw4PqhViVJetig3/p4M/Dp+o2PW4DXDq8kSVLXQEGdmVcDy4ZbiiSpH3+ZKEmNM6glqXEGtSQ1zqCWpMYZ1JLUOINakhpnUEtS4wxqSWqcQS1JjTOoJalxBrUkNc6glqTGGdSS1DiDWpIaZ1BLUuMMaklqnEEtSY0b9E9xSZpnY6vO79t/7erlj8nlamJeUUtS4wxqSWqcQS1JjTOoJalxBrUkNc6glqTGGdSS1DiDWpIaZ1BLUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNGzioI2KLiLgqIr44zIIkSRubzhX1ScCaYRUiSepvoKCOiN2A5cDHh1uOJKnXggHH+zDwDmD7iUaIiJXASoClS5fOujBpNsZWnd+3/9rVy0cyn1Ga6DOMarmb0rprxZRX1BHxEuAHmXnFZONl5hmZuSwzly1evHjOCpSkzd0gTR8vAF4WEWuBc4DDIuKsoVYlSXrYlEGdme/MzN0ycww4DvhqZh4/9MokSYDfo5ak5g36MBGAzLwEuGQolUiS+vKKWpIaZ1BLUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMYZ1JLUOINakho3rb/wos3D2Krz+/Zfu3r5PFcyfyb6zHM1/mPB5rhftMIraklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMYZ1JLUOINakhpnUEtS4wxqSWqcQS1JjTOoJalxUwZ1ROweEV+LiOsj4rqIOGk+CpMkFYP8cduHgLdl5pURsT1wRUR8OTOvH3JtkiQGuKLOzLsy88r6/n5gDbDrsAuTJBWDXFE/LCLGgAOAb/UZthJYCbB06dK5qK0JY6vO79t/7erl81xJu0a5jiZa9lyNP1fmcrmj+gxzZSb1T7QvTXdec7VPzvc+P/DDxIjYDvh74C2ZeV/v8Mw8IzOXZeayxYsXz2WNkrRZGyioI2JLSkh/OjPPG25JkqSuQb71EcDfAGsy87ThlyRJ6hrkivoFwGuAwyLi6vo6esh1SZKqKR8mZubXgZiHWiRJffjLRElqnEEtSY0zqCWpcQa1JDXOoJakxhnUktQ4g1qSGmdQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMYZ1JLUOINakhpnUEtS4wxqSWrclH/hZb7N959hn2uT/fn6Uf2p+slqmovlzmT8Ydeq+TMf22xU+3ArueMVtSQ1zqCWpMYZ1JLUOINakhpnUEtS4wxqSWqcQS1JjTOoJalxBrUkNc6glqTGGdSS1DiDWpIaZ1BLUuMMaklqnEEtSY0zqCWpcQa1JDXOoJakxhnUktS4gYI6Io6KiBsj4qaIWDXsoiRJj5gyqCNiC+CvgN8G9gFeHRH7DLswSVIxyBX1gcBNmXlLZv4cOAc4ZrhlSZLGRWZOPkLEK4CjMvP1tfs1wPMy8009460EVtbOpwM3DrD8XYB7p1t0Q6x/tKx/tKx/bu2RmYv7DVgwV0vIzDOAM6YzTURcnpnL5qqG+Wb9o2X9o2X982eQpo87gN073bvVfpKkeTBIUP8b8LSI2DMiFgLHAV8YblmSpHFTNn1k5kMR8SbgQmAL4BOZed0cLX9aTSUNsv7Rsv7Rsv55MuXDREnSaPnLRElqnEEtSY0bWlBP9bPziDg9Iq6ur+9GxPqe4Y+PiHUR8dFh1TiR2dQeEUsj4qKIWBMR10fE2HzWXmuYTf0fiIjrav1/GRExr8UzUP1LI+JrEXFVRHwnIo7uDHtnne7GiHjx/Fb+cA0zqj8ijoyIKyLimvrvYfNf/ezWf2f4hoh4+/xVvdHyZ7P/PCsiLqvHwDURsfX8Vj+BzJzzF+Wh483AU4CFwLeBfSYZ/82Uh5Tdfh8BPgN8dBg1Dqt24BLgyPp+O2DRplI/cDDwjTqPLYDLgENbq5/yEOiP6/t9gLWd998GtgL2rPPZYhOq/wBgSX2/H3DHfNY+2/o7w88FPge8fVOqn/Lliu8A+9fuJ8z3/jPRa1hX1NP92fmrgbPHOyLiOcCTgIuGVN9kZlx7/T9QFmTmlwEyc0NmPjjsgnvMZt0nsDVlB98K2BK4Z4i19jNI/Qk8vr7fAbizvj8GOCczf5aZtwI31fnNpxnXn5lXZeb4Z7kO2CYitpqHmrtms/6JiGOBWyn1j8Js6n8R8J3M/DZAZv4oM385DzVPaVhBvStwe6d7Xe33KBGxB+Xq56u1+3HAh4CR3DYxi9qBvYH1EXFeva36H/U/tZpPM64/My8DvgbcVV8XZuaaoVb7aIPUfwpwfESsAy6g3BUMOu2wzab+rt8BrszMnw2jyEnMuP6I2A44GXjv8Muc0GzW/95ARsSFEXFlRLxj2MUOqoWHiccB53bOXCcAF2TmuhHWNKje2hcAh1BOMs+l3H6tGE1pA9mo/ojYC3gm5denuwKHRcQhI6xvIq8GzszM3YCjgU/VE/ymYtL6I2Jf4C+A/zKi+qYyUf2nAKdn5oZRFjeAiepfALwQ+L3678sj4vDRlfmIOfu/PnpM52fnxwFv7HQfBBwSESdQ2ngXRsSGzJyv/wd7NrWvA67OzFsAIuIfgOcDfzP3ZU5oNvW/HPjm+IEWEf9E2R7/MoQ6JzJI/X8IHAXlLqA+8NllwGmHbTb1/yAidgM+D/x+Zt48D/X2mk39zwNeEREfAHYEfhUR/5GZ8/mFgNnUvw64NDPvBYiIC4BnA18ZdtFTGlKD/gLgFspt9XiD/r59xnsGsJb6w5s+w1cw/w8TZ1w75UHGt4HFtfuTwBs3ofpfBVxc57ElZQd9aWv1A/8ErKjvn0lpYwxgXzZ+mHgL8/8wcTb171jH/0/zWfNc1d8zzimM5mHibNb/TsCVwKI6n4uB5aPaFhvVPMQVdjTwXcoT2HfVfqcCL+vZmKsnmccK5jmoZ1s7cCTlyfE1wJnAwk2lfsqJ5n8Da4DrgdPmu/ZB6qc8qf9GPQivBl7UmfZddbobgd/elOoH3g08UPuNv564qdTfM49TGEFQz8H+czzlQei1wAdGUX+/lz8hl6TGbUoPYCRps2RQS1LjDGpJapxBLUmNM6glqXEGtSQ1zqCWpMb9f+5MioewOV+6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrap\n",
      "Confidence interval for the score: [0.778 - 0.857]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2ZklEQVR4nO3dd3gU5fbA8e8xoQsIAZHeW0BE5BJ6V5qAXkUjiOAvAoJ6saCiKCICF6SoIEgTUaSrCF65cm1YEFCaCCgSQwtFQhFEpSQ5vz92EpeQkE3I7mZ3z+d59mFmdnb3TBLm7Pu+M+cVVcUYY0zousLfARhjjPEvSwTGGBPiLBEYY0yIs0RgjDEhzhKBMcaEOEsExhgT4iwRGGNMiLNEYIKOiOwRkb9E5LSIHBaRuSJyZZp9morIZyLyu4icFJEPRCQyzT5FRORlEdnnvNcvznoJ3x6RMd5licAEq66qeiVQH7geeCrlCRFpAvwPWA6UASoD3wNrRKSKs09e4FOgDtARKAI0AY4BjbwVtIiEe+u9jcmIJQIT1FT1MLAKV0JI8SLwlqq+oqq/q+pxVX0GWAeMcPa5B6gA3KqqO1Q1WVWPqOoLqroyvc8SkToi8rGIHBeRX0XkaWf7XBEZ5bZfaxGJd1vfIyJPishW4A9n+Z007/2KiEx2louKyOsickhEDojIKBEJu7yflAlllghMUBORckAnINZZLwg0BZams/sS4EZnuT3wkaqe9vBzCgOfAB/hamVUw9Wi8NRdQBfgKmAR0Nl5T5yT/B3AAmffuUCi8xnXAzcB92Xhs4y5gCUCE6zeF5Hfgf3AEeA5Z3txXH/3h9J5zSEgpf8/IoN9MnIzcFhVJ6rqGaelsT4Lr5+sqvtV9S9V3QtsAm51nmsL/Kmq60SkFNAZeFhV/1DVI8BLQHQWPsuYC1giMMHqFlUtDLQGavH3Cf4EkAyUTuc1pYGjzvKxDPbJSHngl2xF6rI/zfoCXK0EgJ783RqoCOQBDonIbyLyGzADuPoyPtuEOEsEJqip6he4ulImOOt/AGuBHunsfgd/d+d8AnQQkUIeftR+oEoGz/0BFHRbvya9UNOsLwVaO11bt/J3ItgPnAVKqOpVzqOIqtbxME5jLmKJwISCl4EbReQ6Z30o0EdE/iUihUWkmDOY2wR43tlnHq6T7rsiUktErhCRCBF5WkQ6p/MZ/wFKi8jDIpLPed8o57ktuPr8i4vINcDDmQWsqgnAauANYLeq/uhsP4TriqeJzuWtV4hIVRFpldUfijEpLBGYoOecVN8ChjvrXwMdgH/iGgfYi2vQtbmq7nL2OYtrwPgn4GPgFPAtri6mi/r+VfV3XAPNXYHDwC6gjfP0PFyXp+7BdRJf7GHoC5wYFqTZfg+QF9iBq6vrHbLWjWXMBcQmpjHGmNBmLQJjjAlxlgiMMSbEWSIwxpgQZ4nAGGNCXMAVuCpRooRWqlTJ32EYY0xA2bhx41FVLZnecwGXCCpVqsSGDRv8HYYxxgQUEdmb0XPWNWSMMSHOEoExxoQ4SwTGGBPiAm6MID3nz58nPj6eM2fO+DsUk0vlz5+fcuXKkSdPHn+HYkyuExSJID4+nsKFC1OpUiVExN/hmFxGVTl27Bjx8fFUrlzZ3+EYk+t4rWtIROaIyBER2ZbB8yIik0UkVkS2ikiD7H7WmTNniIiIsCRg0iUiREREWIvRmAx4c4xgLq5JvzPSCajuPPoDr13Oh1kSMJdifx/GZMxrXUOq+qWIVLrELt1xTSCuwDoRuUpESjv11o0xJuQtWL+P5VsOkJycxLlz52lQ5Wqe65rzcxD5c4ygLBdOzxfvbLsoEYhIf1ytBipUqOCT4IwxxldSTvhprd993LVwZBfh4eFcXzndG4MvW0BcPqqqM1W1oao2LFnSOz+Iy7Fnzx7q1q3rl89WVdq2bcupU6dSt73//vuICD/99FPqttWrV3PzzTdf8Nq+ffvyzjvvAK4rr4YOHUr16tVp0KABTZo04b///e9lx/fvf/+batWqUbNmTVatWpXuPp9++ikNGjSgfv36NG/enNjYWAAmTZpEZGQk9erVo127duzd67oxcu/evan716lTh+nTp6e+V/v27Tlx4sRlx22MtyxYv487Z6y94PH0sh/+Puk7EhMTKXb+KMc+mkKer6byeq9rGdHNO+cZf7YIDuCa8DtFOWdbSElMTCQ8PPu/hpUrV3LddddRpEiR1G0LFy6kefPmLFy4kOeff/4Sr/7bs88+y6FDh9i2bRv58uXj119/5Ysvvsh2XAA7duxg0aJFbN++nYMHD9K+fXt+/vlnwsLCLthv4MCBLF++nNq1azNt2jRGjRrF3Llzuf7669mwYQMFCxbktdde44knnmDx4sWULl2atWvXki9fPk6fPk3dunXp1q0bZcqUoXfv3kybNo1hw4ZdVuzGZEVG3+jTk3LCj6pcPHVbVOXidK9flp5Rrh6PpKQkrr32Wnbu3MmQIUMYMWIZBQoUyPnAHf5MBCuAB0VkERAFnMyJ8YHnP9jOjoOnMt8xCyLLFPG4Xy4uLo7bbruNmTNnUrx4cR544AESEhIoWLAgs2bNolatWvTt25f8+fOzefNmmjVrRnR0NIMHD+bMmTMUKFCAN954g5o1a7J9+3buvfdezp07R3JyMu+++y7Vq1e/4PPmz59P//79U9dPnz7N119/zeeff07Xrl09SgR//vkns2bNYvfu3eTLlw+AUqVKcccdd2Thp3Sx5cuXEx0dTb58+ahcuTLVqlXj22+/pUmTJhfsJyKpLZqTJ09SpkwZANq0aZO6T+PGjXn77bcByJs3b+r2s2fPkpycnLrerVs3WrRoYYnAZElWTuTpSe/knpG0J313x44do3jx4oSFhTF69GjKly9Pw4YNsx2Xp7yWCERkIdAaKCEi8cBzQB4AVZ0OrAQ6A7HAn8C93orFV3bu3El0dDRz587luuuuo127dkyfPp3q1auzfv16Bg0axGeffQa47n345ptvCAsL49SpU3z11VeEh4fzySef8PTTT/Puu+8yffp0Bg8eTK9evTh37hxJSUkXfeaaNWuYMWNG6vry5cvp2LEjNWrUICIigo0bN3LDDTdcMu7Y2FgqVKhwQasiI4888giff/75Rdujo6MZOnToBdsOHDhA48aNU9fLlSvHgQMX/2ebPXs2nTt3pkCBAhQpUoR169ZdtM/rr79Op06dUtf3799Ply5diI2NZfz48anJo1ixYpw9e5Zjx44RERGR6fGY4JLdE3pWTuTpudTJ3ROqyvz58xk8eDBjx46lX79+3Hrrrdl6r+zw5lVDd2XyvAIP5PTnemNE3RMJCQl0796d9957j8jISE6fPs0333xDjx49Uvc5e/Zs6nKPHj1Su0hOnjxJnz592LVrFyLC+fPnAWjSpAmjR48mPj6ef/7znxe1BgCOHz9O4cKFU9cXLlzI4MGDAdfJeeHChdxwww0ZXj6Z1csqX3rppSzt7+l7rly5kqioKMaPH8+jjz7K7NmzU59/++232bBhwwVdVeXLl2fr1q0cPHiQW265hdtvv51SpUoBcPXVV3Pw4EFLBAHicr+Nu8vuCf1yT+SXY//+/dx///2sXLmSxo0b06xZM5/HEBR3FucGRYsWpUKFCnz99ddERkaSnJzMVVddxZYtW9Ldv1ChQqnLzz77LG3atGHZsmXs2bOH1q1bA9CzZ0+ioqL48MMP6dy5MzNmzKBt27YXvE94eDjJyclcccUVHD9+nM8++4wffvgBESEpKQkRYfz48URERFw0iHr8+HFKlChBtWrV2LdvH6dOncq0VZCVFkHZsmXZv//vC8Pi4+MpW7bsBfskJCTw/fffExUVBcCdd95Jx45/337yySefMHr0aL744ovUbit3ZcqUoW7dunz11VfcfvvtAKldbCYwLN9ygB2HThFZOvMWaWb8eULPjoULFzJgwACSkpJ4+eWXefDBBy8aQ/MFSwQ5JG/evCxbtowOHTpw5ZVX0rNnTypXrszSpUvp0aMHqsrWrVu57rrrLnrtyZMnU0+Qc+fOTd0eFxdHlSpV+Ne//sW+ffvYunXrRYmgZs2axMXFUa1aNd555x169+59QVdRq1at+Oqrr4iKiuLgwYP8+OOP1K5dm7179/L9999Tv359ChYsSExMDIMHD2bGjBnkzZuXhIQEVq9efUGLBrLWIujWrRs9e/bk0Ucf5eDBg+zatYtGjRpdsE+xYsU4efIkP//8MzVq1ODjjz+mdu3aAGzevJkBAwbw0UcfcfXVV6e+Jj4+noiICAoUKMCJEyf4+uuveeSRRwBXE/vw4cPY5EX+lZVv+SlJYPGAJpnvHGSKFStGVFQUM2fO9Gv5E0sEOahQoUL85z//4cYbb+TKK69k/vz5DBw4kFGjRnH+/Hmio6PTTQRPPPEEffr0YdSoUXTp0iV1+5IlS5g3bx558uThmmuu4emnn77otV26dGH16tVUq1aNhQsX8uSTT17w/G233cbChQtp2bIlb7/9Nvfeey9nzpwhT548zJ49m6JFiwIwatQonnnmGSIjI8mfPz+FChVi5MiRl/XzqFOnDnfccQeRkZGEh4czderU1G87nTt3Zvbs2ZQpU4ZZs2Zx2223ccUVV1CsWDHmzJkDwOOPP87p06dTk1GFChVYsWIFP/74I4899hgigqoyZMgQrr32WgA2btxI48aNL+tKLHNpnpzks9JFE1m6CN3rl810v2CQmJjISy+9xLlz5xg2bBgdO3akQ4cOfr/zXVxd9YGjYcOGmnaGspRvuaHo0KFD3HPPPXz88cf+DiVXGDx4MN26daNdu3YXPRfKfyc5ISUBeHqSD6QuGl/4/vvviYmJYePGjdxxxx0sWrTIpwlARDaqarqXINnXpgBXunRp+vXr51H/fiioW7duukkgmOTk4GpWuCcAO8l77uzZs4waNYqxY8dSvHhxli5dym233eb3VoC7oEkEqpqrfrC+dLnX+weTfv36pbs90Fq+kHnZgexe6phdlgCyZ9euXYwbN46ePXsyadKkXHk1W1Akgvz586deNx6qycBkLGU+gvz58/s7lEy5n/wzOuHbCTn3O336NMuXL6dXr17UrVuXn376iSpVqvg7rAwFRSIoV64c8fHxJCQk+DsUk0ulzFCWm6T3jd/95G8n/MD08ccf079//9SaWLVr187VSQCCJBHkyZPHZp4yuVZWunjs5B+4Tpw4wZAhQ5gzZw41atTgiy++CJiLE4IiERiTW2T2Ld+dnfSDR1JSEs2aNePnn3/mqaeeYvjw4QHRFZnCEoExlymzfn074Qevo0ePphaJGzNmDBUqVKBBg2zPuus3lgiMyQLr1zfgugBh3rx5PPzww4wdO5b+/ftzyy23+DusbLNEYEwWpFcXx07+oWXv3r0MGDCAVatW0bRpU1q2bOnvkC6bJQJjsihU6+IYVyXcgQMHoqpMmTKFQYMGccUVATHR4yVZIjDG4ckduzlVJdMEppIlS9KsWTNmzJhBxYoV/R1OjrFEYIzDk3LIoVQgzbjm8p44cSLnz5/n2WefpUOHDtx0001Bd+OqJQIT8lJaAqFcDtlcbPPmzcTExLB582aio6NTy9gEWxIACPzOLWMuk3sSsG/75syZMzz99NP84x//4ODBg7z77rssXLgwKBNACmsRmKBkE6OY7IqNjWXChAncc889TJw4kWLFivk7JK+zFoEJSinf8j1hLQFz+vRp5s2bB7hKme/cuZM5c+aERBIAaxGYAJfRN3/7lm88tWrVKvr378/+/ftp2LAhtWvXDrnaZdYiMAEto2/+9i3fZObYsWP06dOHjh07UrBgQb766quAKRKX06xFYAKeffM3WZVSJC42NpZhw4bxzDPPBFSRuJxmicAEjPS6gewGL5MVCQkJREREEBYWxrhx46hYsSL169f3d1h+Z11DJmCk1w1kXUDGE6rKG2+8QY0aNZg1axYA3bt3tyTgsBaBCSjWDWSyas+ePfTv35+PP/6YFi1a0KZNG3+HlOtYi8AYE7TmzZtH3bp1Wbt2LdOmTWP16tXUqFHD32HlOtYiMMYErVKlStGyZUumT59OhQpWJjwjlghMrpPZvQHGZOT8+fO8+OKLJCUlMXz4cG666SZuuukmf4eV61kiMH6VlTl+bWDYXMqmTZv4v//7P77//nt69uyZWiTOZM4SgfE5m+PX5KS//vqL559/ngkTJlCyZEmWLVsW0NNG+oNXE4GIdAReAcKA2ao6Ns3zFYA3gaucfYaq6kpvxmRyTlYKu7mzOX5NToqLi2PSpEn07duX8ePHh0x9oJzktUQgImHAVOBGIB74TkRWqOoOt92eAZao6msiEgmsBCp5KyaTM1ISQEZdOJmxk7+5XKdOneK9996jb9++1KlTh127dgXVjGG+5s0WQSMgVlXjAERkEdAdcE8ECqSM/hUFDnoxHpNDUm7sshO68YeVK1dy//33c+DAAaKioqhdu7YlgcvkzfsIygL73dbjnW3uRgB3i0g8rtbAQ+m9kYj0F5ENIrIhISHBG7GaLEq5scuSgPGVo0eP0rt3b7p06ULhwoVZs2ZNyBaJy2n+vqHsLmCuqpYDOgPzROSimFR1pqo2VNWGJUuW9HmQxhj/SikSt2jRIoYPH86mTZto3Lixv8MKGt7sGjoAlHdbL+dscxcDdARQ1bUikh8oARzxYlzmEjwZALbr+Y2v/Prrr5QsWZKwsDAmTJhAxYoVqVevnr/DCjrebBF8B1QXkcoikheIBlak2Wcf0A5ARGoD+QHr+/EjT2b2suv5jbepKq+//jo1a9Zk5syZAHTt2tWSgJd4rUWgqoki8iCwCteloXNUdbuIjAQ2qOoK4DFglog8gmvguK+qqrdiMp6xwm7Gn+Li4ujXrx+fffYZrVq1on379v4OKeh59T4C556AlWm2DXdb3gE082YMJmNW39/kNm+++SaDBg0iLCyM6dOn069fP664wt9DmcHPfsIhzOr7m9ymTJkytG3blh07djBgwABLAj5iJSZCnHUDGX86d+4cY8eOJTk5mREjRnDjjTdy4403+juskGPp1hjjF9999x033HADzz33HHFxcdjwoP9YIjDG+NSff/7JkCFDaNy4MSdOnGDFihW89dZbVinUjywRGGN8avfu3UyZMoV+/fqxfft2unbt6u+QQp6NEYQY9yuF7Aoh4ysnT57kvffe495776VOnTrExsZSvnz5zF9ofMISQQjIqP6/XSFkfOHDDz9kwIABHDp0iCZNmlCrVi1LArmMJYIgll65aKsYanwlISGBhx9+mAULFlC3bl3ee+89atWq5e+wTDosEQSZjL7928nf+FJSUhLNmzdn9+7dPP/88wwdOpS8efP6OyyTAUsEQSblJrHI0kUsARifO3z4MFdffTVhYWFMnDiRSpUqUbduXX+HZTJhiSAIpDcAbDeJGV9KTk5m1qxZPP7444wbN46BAwdy8803+zss4yGPEoGIFAAqqOpOL8djMpFefSAbADb+FBsbS79+/Vi9ejVt27alQ4cO/g7JZFGmiUBEugITgLxAZRGpD4xU1W5ejs2kw73rJ4V1ARl/eeONNxg0aBB58+Zl1qxZxMTE2I1hAciTFsEIXPMPrwZQ1S0iUtmLMZk0rOvH5FYVKlSgQ4cOTJ06lbJlrSUaqDxJBOdV9WSaLG9FQbzMrv03udHZs2f597//TXJyMiNHjqRdu3a0a9fO32GZy+RJItguIj2BMBGpDvwL+Ma7YRm7+sfkNuvXrycmJobt27fTp08fVNW6gYKEJ4ngIWAYcBZYgGvGsRe8GZRxsS4gkxv88ccfPPvss7z88suULVuW//znP3Tp0sXfYZkc5EnRuS6qOkxV/+E8ngFsoNiYELF3716mTZvG/fffz/bt2y0JBCFPEsFTHm4zxgSJ3377jdmzZwMQGRlJbGws06ZNo0gRK1IYjDLsGhKRTkBnoKyITHZ7qgiQ6O3AjDH+sXz5cgYOHMiRI0do3rw5tWrVoly5cv4Oy3jRpVoEB4ENwBlgo9tjBWB3jBgTZI4cOUJ0dDS33HILJUuWZN26dVYkLkRk2CJQ1e+B70Vkgaqe92FMxhgfS0pKolmzZuzbt49Ro0bxxBNPkCdPHn+HZXzEk6uGKonIv4FIIH/KRlWt4rWojDE+cfDgQa655hrCwsJ45ZVXqFSpEpGRkf4Oy/iYJ4PFbwCv4RoXaAO8BbztzaBC1YL1+7hzxlrunLGWHYdO+TscE8SSk5N57bXXqFWrFtOnTwegc+fOlgRClCeJoICqfgqIqu5V1RGAXT/mBSk3kQF2B7Hxmp9//pk2bdowaNAgoqKi6NSpk79DMn7mSdfQWRG5AtglIg8CB4ArvRtW6LKbyIw3vf766zz44IPkz5+fOXPm0LdvX7s72HjUIhgMFMRVWuIG4G6gjzeDMsZ4R6VKlejUqRM7duzg3nvvtSRggExaBCISBtypqkOA08C9PonKGJMjzp49ywsvuCrCjBo1yorEmXRdskWgqklAcx/FYozJQd988w3169dn9OjRHDp0CFUrGmzS58kYwWYRWQEsBf5I2aiq73ktKmNMtp0+fZphw4YxZcoUypcvz0cffWSzhplL8mSMID9wDGgLdHUeHk1GKiIdRWSniMSKyNAM9rlDRHaIyHYRWeBp4MaY9O3bt48ZM2bwwAMPsG3bNksCJlOZtghUNVvjAs74wlTgRiAe+E5EVqjqDrd9quMqYNdMVU+IyNXZ+SxjQt2JEydYunQp/fv3JzIykri4OMqUKePvsEyA8KRFkF2NgFhVjVPVc8AioHuaffoBU1X1BICqHvFiPMYEpWXLlhEZGcmgQYPYuXMngCUBkyXeTARlgf1u6/HONnc1gBoiskZE1olIx/TeSET6i8gGEdmQkJDgpXCNCSyHDx+mR48e/POf/+Saa67h22+/pWbNmv4OywQgTwaLvf351YHWQDngSxG5VlV/c99JVWcCMwEaNmxolz6YkJeUlESLFi3Yv38/Y8aMYciQIVYkzmRbpolAREoBY4AyqtpJRCKBJqr6eiYvPQCUd1sv52xzFw+sd6qb7haRn3Elhu88PQBjQkl8fDxlypQhLCyMyZMnU7lyZSsVbS6bJ11Dc3HNU5zS6fgz8LAHr/sOqC4ilUUkLxCNay4Dd+/jag0gIiVwdRXFefDexoSU5ORkpkyZQq1atXjttdcA6NSpkyUBkyM8SQQlVHUJkAygqolAUmYvcvZ7EFcS+RFYoqrbRWSkiKTMebwKOCYiO4DPgcdV9Vg2jsOYoPXTTz/RsmVL/vWvf9G8eXNuvtmjq7eN8ZgnYwR/iEgEoAAi0hg46cmbq+pKYGWabcPdlhV41HkYY9KYPXs2Dz74IAULFuTNN9+kd+/eVh/I5DhPEsFjuLp0qorIGqAkcLtXowoxC9bvSy1BHVnaJgc3f6tatSpdu3bl1VdfpVSpUv4OxwQpT24o2ygirYCagAA7berKnOWeBGwOgtB25swZRo4cCcCYMWNo06YNbdq08XNUJth5ctXQVlw3gy1W1V+8H1LwSvnmn1ZKErB5CELbmjVriImJYefOndx3332oqnUDGZ/wZLC4K65pKpeIyHciMkREKng5rqDkPgOZO2sJhLbff/+dhx56iBYtWnD27FlWrVrFrFmzLAkYn/Gka2gv8CLwolMb6FlgHBDm5diCkn3zN2nFx8cze/ZsHnroIUaPHs2VV9oEgMa3PLqzWEQqAnc6jyTgCW8GZUywO3bsGEuWLGHgwIHUrl2buLg4Spcu7e+wTIjyZIxgPZAH13wEPVTVbvgyJptUlXfffZcHHniA48eP07ZtW2rWrGlJwPiVJy2Ce1R1p9cjCVLuA8R2eWhoO3ToEA888ADLli3jhhtu4H//+58ViTO5QoaJQETuVtW3gS4i0iXt86o6yauRBQn3S0NtUDh0pRSJO3DgAC+++CKPPPII4eH+rvlojMul/hILOf8WTuc5qwCaibQ3idkAcWjav38/ZcuWJSwsjKlTp1K5cmVq1Kjh77CMuUCGl4+q6gxn8RNVfd79AXzqm/ACl90kFtqSkpKYPHnyBUXiOnToYEnA5EqetE2nAA082GbSsJZAaPrxxx+JiYlh7dq1dOrUia5du/o7JGMu6VJjBE2ApkBJEXEvClcEu4fAmHTNnDmThx56iMKFCzNv3jx69eplN4aZXO9SLYK8wJXOPu7jBKewonPGpKt69erceuutTJ48mauvvtrf4RjjkQwTgap+AXwhInOdu4tNJuxS0dDz119/MWLECESEsWPHWpE4E5AyHCwWkZedxVdFZEXah2/CCyzutYRskDj4ffnll1x33XW8+OKLnDx5Etf0GsYEnkt1Dc1z/p3gi0CChQ0QB79Tp04xdOhQXnvtNapUqcKnn35K27Zt/R2WMdl2qa6hjc6/X6RsE5FiQHlV3eqD2IzJlQ4ePMjcuXN59NFHGTlyJIUKFcr8RcbkYp7UGloNdHP23QgcEZE1qmrTS5qQcfToUZYsWcKgQYOoVasWu3fvthnDTNDwZD6Coqp6Cvgn8JaqRgHtvRuWMbmDqrJ48WIiIyN5+OGH+fnnnwEsCZig4kkiCBeR0sAdwH+8HI8xucbBgwe55ZZbiI6OpmLFimzcuNHuDDZByZM7i0cCq4A1qvqdiFQBdnk3rMBhl4wGp6SkJFq2bMmBAweYMGECgwcPtiJxJmh5MkPZUlxzEaSsxwG3eTOoQGLVRYPL3r17KVeuHGFhYUybNo0qVapQrVo1f4dljFdl2jUkIuVEZJmIHHEe74pIOV8EFyhSLhldPKAJPaNsOudAlJSUxKRJk6hdu3ZqkbibbrrJkoAJCZ6MEbwBrADKOI8PnG3GBIVt27bRtGlTHnvsMdq1a8ctt9zi75CM8SlPEkFJVX1DVROdx1ygpJfjMsYnpk+fToMGDYiLi2PBggWsWLGCcuWswWtCiyeJ4JiI3C0iYc7jbuCYtwMzxptSykHUrl2bHj16sGPHDu666y6rFGpCkieXQfwfrvkHXnLW1wD3ei0iY7zozz//ZPjw4YSFhTFu3DhatWpFq1at/B2WMX6VaYtAVfeqajdVLek8blHVfb4IzpictHr1aurVq8fEiRM5ffq0FYkzxuHJVUNVROQDEUlwrhpa7txLYExAOHnyJAMGDEgtD/3ZZ58xdepU6wYyxuHJGMECYAlQGtdVQ0uBhd4MypicdOjQId5++22GDBnC1q1bbb4AY9LwJBEUVNV5blcNvQ3k9+TNRaSjiOwUkVgRGXqJ/W4TERWRhp4GbsylJCQkMGXKFABq1arFnj17GD9+PAULFvRzZMbkPp4kgv+KyFARqSQiFUXkCWCliBQXkeIZvUhEwoCpQCcgErhLRCLT2a8wMBhYn71DMOZvqsqCBQuoXbs2jz32WGqRuJIl7YpnYzLiSSK4AxgAfA6sBgYC0bhKUm+4xOsaAbGqGqeq54BFQPd09nsBGAec8TxsYy62f/9+unbtSq9evahWrRqbN2+2InHGeMCTWkOVs/neZYH9buvxQJT7DiLSANdENx+KyOMZvZGI9Af6A1So4P8SDlZoLvdJTEykdevWHD58mJdeeomHHnqIsLAwf4dlTEDwWzlFEbkCmAT0zWxfVZ0JzARo2LCh36/5s0JzuceePXsoX7484eHhzJgxgypVqlClil3UZkxWeNI1lF0HgPJu6+WcbSkKA3WB1SKyB2gMrAiUAWMrNOdfiYmJTJgwgdq1azNt2jQA2rdvb0nAmGzwZovgO6C6iFTGlQCigZ4pT6rqSaBEyrozJeYQVb3UuIMxbN26lZiYGDZs2ED37t257Tarim7M5fDkhjJxag0Nd9YriEijzF6nqonAg7gmtfkRWKKq20VkpIh0u9zATWiaNm0aN9xwA3v37mXx4sUsW7aMMmXK+DssYwKaJy2CaUAy0BbXbGW/A+8C/8jshaq6EliZZtvwDPZt7UEsJkSpKiJC3bp1iY6O5qWXXqJEiRKZv9AYkylPEkGUqjYQkc0AqnpCRPJ6OS5jAPjjjz945plnCA8PZ/z48bRs2ZKWLVv6Oyxjgoong8XnnZvDFEBESuJqIRjjVZ9++inXXnstL7/8MmfPnrUiccZ4iSeJYDKwDLhaREYDXwNjvBqVCWm//fYb9913H+3btyc8PJwvv/ySyZMnW5E4Y7zEkxvK5ovIRqAdIMAtqvqj1yMzIevXX39l0aJFPPnkkzz33HMUKFDA3yEZE9QyTQQiUgH4E9dcxanbbE4Ck5NSTv6DBw+mZs2a7NmzxwaDjfERTwaLP8Q1PiC4qo5WBnYCdbwYlwkRqsr8+fMZPHgwp0+fpnPnzlSvXt2SgDE+5MkMZdeqaj3n3+q4ismt9X5oJtjt27ePLl260Lt3b2rWrMmWLVuoXr26v8MyJuRk+c5iVd0kIlGZ72lMxlKKxB05coTJkyczaNAgKxJnjJ94MkbwqNvqFUAD4KDXIjJBLS4ujooVKxIeHs6sWbOoWrUqlSpV8ndYxoQ0Ty4fLez2yIdrzCC9eQWC3oL1+7hzxlp2HDrl71ACTmJiIuPGjSMyMpKpU6cC0K5dO0sCxuQCl2wRODeSFVbVIT6KJ1dzLz9tpac9t2XLFmJiYti0aRO33norPXr08HdIxhg3GSYCEQlX1UQRaebLgHK7lPLTxjOvvvoqjzzyCBEREbzzzjtWKdSYXOhSLYJvcY0HbBGRFcBS4I+UJ1X1PS/HZgJYSpG4evXq0atXLyZNmkTx4hlOcW2M8SNPrhrKDxzDVX005X4CBSwRmIucPn2aYcOGkSdPHiZMmGBF4owJAJcaLL7auWJoG/CD8+92599tPojNBJj//e9/1K1blylTpnD+/HkrEmdMgLhUiyAMuBJXCyAt+x9uUp04cYJHH32UuXPnUrNmTb788kuaN2/u77CMMR66VCI4pKojfRaJCVhHjhzhnXfe4amnnmL48OHkz5/f3yEZY7LgUonAav6aDB0+fJiFCxfyyCOPpBaJi4iI8HdYxphsuNQYQTufRWEChqry5ptvEhkZyVNPPcWuXbsALAkYE8AyTASqetyXgZjcb8+ePXTs2JG+ffsSGRlpReKMCRJZLjpnQlNiYiJt2rTh6NGjTJ06lfvvv58rrvCkQokxJrezRGAuKTY2lsqVKxMeHs6cOXOoUqUKFStW9HdYxpgcZF/pTLrOnz/PmDFjqFOnTmqRuDZt2lgSMCYIWYvAXGTTpk3ExMSwZcsWevTowZ133unvkIwxXmQtAnOByZMn06hRIw4fPsx7773HkiVLKFWqlL/DMsZ4kbUIMrFg/T6WbzkAkFqCOhilFIm7/vrrueeee5g4cSLFihXzd1jGGB+wRJAJ9zkIgnEegt9//52nnnqKfPnyMXHiRFq0aEGLFi38HZYxxocsEXggWOcg+OijjxgwYAD79+/n4YcfTm0VGGNCi40RhKBjx47Rp08fOnXqRKFChVizZg2TJk2yJGBMiLJEEIKOHTvGsmXLePbZZ9m8eTNNmgRfa8cY4zmvdg2JSEfgFVwlrWer6tg0zz8K3AckAgnA/6nqXm/G5IlgHCA+dOgQ8+fP57HHHqNGjRrs3bvXBoONMYAXWwTOxPdTgU5AJHCXiESm2W0z0FBV6wHvAC96K56sSBkgBgJ+gFhVmTNnDrVr1+bZZ58lNjYWwJKAMSaVN1sEjYBYVY0DEJFFQHdgR8oOqvq52/7rgLu9GE+WBMMA8e7du+nfvz+ffPIJLVu2ZNasWVYkzhhzEW8mgrLAfrf1eCDqEvvHAP9N7wkR6Q/0B6hQoUJOxRfUEhMTadu2LceOHeO1116jf//+ViTOGJOuXHH5qIjcDTQEWqX3vKrOBGYCNGzY0GvTZKaMDQTyuMCuXbuoUqUK4eHhvPHGG1StWpXy5cv7OyxjTC7mzURwAHA/A5Vztl1ARNoDw4BWqnrWi/Gky31geP1u1xQMUZWLB9y4wPnz5xk3bhwvvPACL774IoMHD6Z169b+DssYEwC8mQi+A6qLSGVcCSAa6Om+g4hcD8wAOqrqES/GkiH3FkBKAugZFVjdTxs2bCAmJoatW7cSHR3NXXfd5e+QjDEBxGuJQFUTReRBYBWuy0fnqOp2ERkJbFDVFcB44EpgqXMz0z5V7eatmDISyAPDr7zyCo8++ijXXHMNy5cvp1s3n//4jDEBzqtjBKq6EliZZttwt+X23vz8YJZSDqJhw4bExMTw4osvctVVV/k7LGNMAMoVg8XGc6dOneLJJ58kf/78vPTSSzRr1oxmzZr5OyxjTACz6wkDyMqVK6lTpw4zZ84kPDwcVa9dQGWMCSGWCALA0aNHufvuu+nSpQtFixblm2++Yfz48VYkzhiTIywRBIATJ07wwQcf8Nxzz7Fp0yaioi51X54xxmSNjRHkUgcOHGD+/Pk8/vjjVK9enb1799pgsDHGK6xFkMuoKrNmzSIyMpIRI0bwyy+/AFgSMMZ4jSWCXOSXX36hXbt29O/fnwYNGrB161aqVavm77CMMUHOuoZyicTERNq1a8fx48eZMWMG9913nxWJM8b4hCUCP9u5cydVq1YlPDycN998k6pVq1KuXDl/h2WMCSH2ldNPzp07x/PPP8+1117L1KlTAWjVqpUlAWOMz1mLwA++/fZbYmJi2LZtGz179qRXr17+DskYE8KsReBjL7/8Mk2aNEm9N2D+/PmUKFHC32EZY0KYJQIfSSkH0ahRI/r168f27du5+eab/RyVMcZY15DXnTx5kieeeIICBQrw8ssv07RpU5o2bervsIwxJpW1CLzogw8+IDIyktmzZ5MvXz4rEmeMyZUsEXhBQkICPXv2pFu3bkRERLBu3TrGjRtnReKMMbmSJQIvOHnyJCtXruT5559nw4YN/OMf//B3SMYYkyEbI8gh+/fv5+2332bo0KFUq1aNvXv3UrRoUX+HZYwxmQrJRLBg/T6WbzkAkDpxfXYlJyczc+ZMnnjiCZKSkujRowfVqlWzJGCMCRgh2TW0fMsBdhw6Bbgmru9ev2y23mfXrl20bduWgQMH0qhRI3744QcrEmeMCTgh2SIAVwJYPKBJtl+fmJjIjTfeyG+//cbrr7/Ovffea4PBxpiAFLKJILt+/PFHqlevTnh4OPPmzaNq1aqUKVPG32EZY0y2hWTXUHacPXuW5557jnr16vHqq68C0KJFC0sCxpiAZy0CD6xbt46YmBh27NhB79696d27t79DMsaYHGMtgkxMnDiRpk2b8vvvv7Ny5UreeustIiIi/B2WMcbkGEsEGUhOTgagSZMm3H///Wzbto1OnTr5OSpjjMl51jWUxm+//cZjjz1GwYIFmTJlihWJM8YEvZBpESxYv487Z6zlzhlrU+8hSOv9998nMjKSN998k8KFC1uROGNMSAiZRHCpm8iOHDnCHXfcwa233kqpUqX49ttvGTNmjN0XYIwJCSHVNZTRTWSnTp3i448/ZvTo0Tz++OPkyZPHD9EZY4x/hFQicLdv3z7mzZvH008/TbVq1di3bx+FCxf2d1jGGONzXu0aEpGOIrJTRGJFZGg6z+cTkcXO8+tFpJI34wHX1UDTpk2jTp06jBkzhl9++QXAkoAxJmR5LRGISBgwFegERAJ3iUhkmt1igBOqWg14CRjnrXgA/vrrT1q3bs0DDzxAkyZN2L59uxWJM8aEPG+2CBoBsaoap6rngEVA9zT7dAfedJbfAdqJl0ZoVZWtW7fyww8/8MYbb7Bq1SoqVarkjY8yxpiA4s0xgrLAfrf1eCAqo31UNVFETgIRwFH3nUSkP9AfoEKFCtkKpk7ZohSLqsuI0TsoXbp0tt7DGGOCUUAMFqvqTGAmQMOGDbN1cf9zXesAdXIyLGOMCQre7Bo6AJR3Wy/nbEt3HxEJB4oCx7wYkzHGmDS8mQi+A6qLSGURyQtEAyvS7LMC6OMs3w58pnY7rzHG+JTXuoacPv8HgVVAGDBHVbeLyEhgg6quAF4H5olILHAcV7IwxhjjQ14dI1DVlcDKNNuGuy2fAXp4MwZjjDGXFjK1howxxqTPEoExxoQ4SwTGGBPiLBEYY0yIk0C7WlNEEoC92Xx5CdLctRwC7JhDgx1zaLicY66oqiXTeyLgEsHlEJENqtrQ33H4kh1zaLBjDg3eOmbrGjLGmBBnicAYY0JcqCWCmf4OwA/smEODHXNo8Moxh9QYgTHGmIuFWovAGGNMGpYIjDEmxAVlIhCRjiKyU0RiRWRoOs/nE5HFzvPrRaSSH8LMUR4c86MiskNEtorIpyJS0R9x5qTMjtltv9tEREUk4C819OSYReQO53e9XUQW+DrGnObB33YFEflcRDY7f9+d/RFnThGROSJyRES2ZfC8iMhk5+exVUQaXPaHqmpQPXCVvP4FqALkBb4HItPsMwiY7ixHA4v9HbcPjrkNUNBZHhgKx+zsVxj4ElgHNPR33D74PVcHNgPFnPWr/R23D455JjDQWY4E9vg77ss85pZAA2BbBs93Bv4LCNAYWH+5nxmMLYJGQKyqxqnqOWAR0D3NPt2BN53ld4B2IiI+jDGnZXrMqvq5qv7prK7DNWNcIPPk9wzwAjAOOOPL4LzEk2PuB0xV1RMAqnrExzHmNE+OWYEiznJR4KAP48txqvolrvlZMtIdeEtd1gFXichlTcQejImgLLDfbT3e2ZbuPqqaCJwEInwSnXd4cszuYnB9owhkmR6z02Qur6of+jIwL/Lk91wDqCEia0RknYh09Fl03uHJMY8A7haReFzznzzkm9D8Jqv/3zMVEJPXm5wjIncDDYFW/o7Fm0TkCmAS0NfPofhaOK7uoda4Wn1fisi1qvqbP4PysruAuao6UUSa4Jr1sK6qJvs7sEARjC2CA0B5t/VyzrZ09xGRcFzNyWM+ic47PDlmRKQ9MAzopqpnfRSbt2R2zIWBusBqEdmDqy91RYAPGHvye44HVqjqeVXdDfyMKzEEKk+OOQZYAqCqa4H8uIqzBSuP/r9nRTAmgu+A6iJSWUTy4hoMXpFmnxVAH2f5duAzdUZhAlSmxywi1wMzcCWBQO83hkyOWVVPqmoJVa2kqpVwjYt0U9UN/gk3R3jyt/0+rtYAIlICV1dRnA9jzGmeHPM+oB2AiNTGlQgSfBqlb60A7nGuHmoMnFTVQ5fzhkHXNaSqiSLyILAK1xUHc1R1u4iMBDao6grgdVzNx1hcgzLR/ov48nl4zOOBK4Glzrj4PlXt5regL5OHxxxUPDzmVcBNIrIDSAIeV9WAbe16eMyPAbNE5BFcA8d9A/mLnYgsxJXMSzjjHs8BeQBUdTqucZDOQCzwJ3DvZX9mAP+8jDHG5IBg7BoyxhiTBZYIjDEmxFkiMMaYEGeJwBhjQpwlAmOMCXGWCEyuJSJJIrLF7VHpEvue9mFoGRKRMiLyjrNc370Spoh0u1SVVC/EUklEevrq80zgsstHTa4lIqdV9cqc3tdXRKQvroqnD3rxM8KdelnpPdcaGKKqN3vr801wsBaBCRgicqUzl8ImEflBRC6qNioipUXkS6cFsU1EWjjbbxKRtc5rl4rIRUlDRFaLyCtur23kbC8uIu87td/XiUg9Z3srt9bKZhEp7HwL3+bcBTsSuNN5/k4R6Ssir4pIURHZ69RDQkQKich+EckjIlVF5CMR2SgiX4lIrXTiHCEi80RkDa4bIys5+25yHk2dXccCLZzPf0REwkRkvIh85xzLgBz61ZhA5+/a2/awR0YPXHfGbnEey3DdCV/Eea4ErjsrU1q1p51/HwOGOcthuGoOlcA1J0EhZ/uTwPB0Pm81MMtZbolTDx6YAjznLLcFtjjLHwDNnOUrnfgqub2uL/Cq2/unrgPLgTbO8p3AbGf5U6C6sxyFq/xJ2jhHABuBAs56QSC/s1wd1x234Lo79T9ur+sPPOMs5wM2AJX9/Xu2h/8fQVdiwgSVv1S1fsqKiOQBxohISyAZV+ndUsBht9d8B8xx9n1fVbeISCtcE5asccpr5AXWZvCZC8FVE15EiojIVUBz4DZn+2ciEiEiRYA1wCQRmQ+8p6rx4vm0FotxJYDPcZU4mea0UprydxkQcJ2w07NCVf9ylvMAr4pIfVzJs0YGr7kJqCcitzvrRXEljt2eBm2CkyUCE0h6ASWBG1T1vLiqiuZ338E5gbcEugBzRWQScAL4WFXv8uAz0g6aZTiIpqpjReRDXHVf1ohIBzyfAGcFrqRWHLgB+AwoBPzmnvwu4Q+35UeAX4HrcHX3ZhSDAA+p6ioPYzQhwsYITCApChxxkkAb4KJ5l8U1F/OvqjoLmI1ryr91QDMRqebsU0hEMvrWfKezT3NcVR1PAl/hSkIpA7BHVfWUiFRV1R9UdRyulkja/vzfcXVNXURVTzuveQVX902Sqp4CdotID+ezRESu8/Dnckhd9fd74+oSS+/zVwEDndYSIlJDRAp58P4myFmLwASS+cAHIvIDrv7tn9LZpzXwuIicB04D96hqgnMFz0IRSelqeQZXrf60zojIZlzdLf/nbBuBq7tpK65qjyklzB92ElIysB3XrG/uUwZ+DgwVkS3Av9P5rMXAUifmFL2A10TkGSeGRbjm6b2UacC7InIP8BF/txa2Akki8j0wF1fSqQRsElffUwJwSybvbUKAXT5qjENEVuO63DKQ5ywwJsusa8gYY0KctQiMMSbEWYvAGGNCnCUCY4wJcZYIjDEmxFkiMMaYEGeJwBhjQtz/A9uP8cQ1zb2eAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xls_file =input(\"please enter xls file path, for example ./data/ResultsExample.xlsx: \") or \"./data/ResultsExample.xlsx\"\n",
    "\n",
    "perform_Bootstrap = int(input(r\"Do you want to perform Bootsrap? (1 = Yes ,  0 = No)\") or \"1\") \n",
    "if perform_Bootstrap==1 :\n",
    "    UncertaintyAnalysis.perform_Bootstrap= True\n",
    "elif perform_Bootstrap==0 :\n",
    "    UncertaintyAnalysis.perform_Bootstrap= False\n",
    "else:\n",
    "    print('Input sets incorrectly!')\n",
    "    \n",
    "if perform_Bootstrap==1 :\n",
    "    n_bootstraps = int(input(r\"Do you want to get AUC based on Delong method, for example 100:\") or \"100\")\n",
    "    UncertaintyAnalysis.n_bootstraps= n_bootstraps\n",
    "\n",
    "plot_roc = int(input(r\"Do you want to plot ROC curve (1 = Yes ,  0 = No)\") or \"1\")\n",
    "if plot_roc==1 :\n",
    "    UncertaintyAnalysis.plot_roc= True\n",
    "elif plot_roc==0 :\n",
    "    UncertaintyAnalysis.plot_roc= False\n",
    "else:\n",
    "    print('Input sets incorrectly!')\n",
    "    \n",
    "    \n",
    "perform_Delong = int(input(r\"Do you want to get AUC based on Delong method? (1 = Yes ,  0 = No)\") or \"1\")\n",
    "print(perform_Delong)\n",
    "if perform_Delong==1 :\n",
    "    UncertaintyAnalysis.perform_Delong= True\n",
    "elif perform_Delong==0 :\n",
    "    UncertaintyAnalysis.perform_Delong= False\n",
    "else:\n",
    "    print('Input sets incorrectly!')\n",
    "    \n",
    "\n",
    "UncertaintyAnalysis.tag = str(input(r\"Please enter a name for this results: \") or \"Test\")\n",
    "#loading file tnto dataframe\n",
    "df = pd.read_excel(xls_file)\n",
    "# print(\"\\n \",df)\n",
    "\n",
    "#define two list to save values\n",
    "y_pred=[]\n",
    "y_truth= []\n",
    "\n",
    "#extraction columns values \n",
    "for i in df.values:\n",
    "    y_truth.append(i[0])\n",
    "    y_pred.append(i[1])\n",
    "\n",
    "\n",
    "#convert list to ndarray\n",
    "y_pred = np.array(y_pred)\n",
    "y_truth = np.array(y_truth)\n",
    "\n",
    "Results = UncertaintyAnalysis.get_report(y_pred , y_truth)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae90b8a",
   "metadata": {},
   "source": [
    "### Setting parameters manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b22d2ad7",
   "metadata": {},
   "outputs": [],
   "source": [
    "UncertaintyAnalysis.perform_Delong = True\n",
    "UncertaintyAnalysis.perform_Bootstrap = True\n",
    "UncertaintyAnalysis.plot_roc = True\n",
    "UncertaintyAnalysis.n_bootstraps = 100\n",
    "UncertaintyAnalysis.tag = 'My Results'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1673fe74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "####################\n",
      "Results for My Results\n",
      "          pred:yes  pred:no\n",
      "true:yes         3        1\n",
      "true:no          2        3\n",
      "Precision:  0.6\n",
      "Precision_CI:  (0.17059340550788216, 1.0294065944921178)\n",
      "Recall:  0.75\n",
      "Recall_CI:  (0.32565534972143556, 1.1743446502785644)\n",
      "Delong Method\n",
      "AUC: 0.8\n",
      "AUC COV: 0.028749999999999998\n",
      "95% AUC CI: [0.46767194 1.        ]\n",
      "Original ROC area: 0.800\n",
      "Bootstrap #1 ROC area: 1.000\n",
      "Bootstrap #2 ROC area: 0.857\n",
      "Bootstrap #3 ROC area: 0.778\n",
      "Bootstrap #4 ROC area: 0.778\n",
      "Bootstrap #5 ROC area: 0.700\n",
      "Bootstrap #6 ROC area: 0.700\n",
      "Bootstrap #7 ROC area: 0.650\n",
      "Bootstrap #8 ROC area: 0.950\n",
      "Bootstrap #9 ROC area: 1.000\n",
      "Bootstrap #10 ROC area: 0.833\n",
      "Bootstrap #11 ROC area: 0.900\n",
      "Bootstrap #12 ROC area: 1.000\n",
      "Bootstrap #13 ROC area: 1.000\n",
      "Bootstrap #14 ROC area: 0.600\n",
      "Bootstrap #16 ROC area: 0.900\n",
      "Bootstrap #17 ROC area: 0.714\n",
      "Bootstrap #18 ROC area: 1.000\n",
      "Bootstrap #19 ROC area: 0.750\n",
      "Bootstrap #20 ROC area: 0.800\n",
      "Bootstrap #21 ROC area: 1.000\n",
      "Bootstrap #22 ROC area: 1.000\n",
      "Bootstrap #23 ROC area: 0.667\n",
      "Bootstrap #24 ROC area: 0.889\n",
      "Bootstrap #25 ROC area: 1.000\n",
      "Bootstrap #26 ROC area: 1.000\n",
      "Bootstrap #27 ROC area: 0.700\n",
      "Bootstrap #28 ROC area: 1.000\n",
      "Bootstrap #29 ROC area: 0.800\n",
      "Bootstrap #30 ROC area: 0.900\n",
      "Bootstrap #31 ROC area: 1.000\n",
      "Bootstrap #32 ROC area: 1.000\n",
      "Bootstrap #33 ROC area: 0.778\n",
      "Bootstrap #34 ROC area: 0.700\n",
      "Bootstrap #35 ROC area: 0.643\n",
      "Bootstrap #36 ROC area: 0.500\n",
      "Bootstrap #37 ROC area: 1.000\n",
      "Bootstrap #38 ROC area: 0.900\n",
      "Bootstrap #39 ROC area: 0.800\n",
      "Bootstrap #40 ROC area: 0.900\n",
      "Bootstrap #41 ROC area: 0.900\n",
      "Bootstrap #42 ROC area: 0.950\n",
      "Bootstrap #43 ROC area: 0.750\n",
      "Bootstrap #44 ROC area: 0.850\n",
      "Bootstrap #45 ROC area: 0.778\n",
      "Bootstrap #46 ROC area: 0.667\n",
      "Bootstrap #47 ROC area: 0.700\n",
      "Bootstrap #48 ROC area: 1.000\n",
      "Bootstrap #49 ROC area: 0.778\n",
      "Bootstrap #51 ROC area: 0.875\n",
      "Bootstrap #52 ROC area: 1.000\n",
      "Bootstrap #53 ROC area: 1.000\n",
      "Bootstrap #54 ROC area: 0.800\n",
      "Bootstrap #55 ROC area: 1.000\n",
      "Bootstrap #56 ROC area: 0.714\n",
      "Bootstrap #57 ROC area: 0.643\n",
      "Bootstrap #58 ROC area: 1.000\n",
      "Bootstrap #59 ROC area: 0.800\n",
      "Bootstrap #60 ROC area: 0.889\n",
      "Bootstrap #61 ROC area: 0.875\n",
      "Bootstrap #62 ROC area: 1.000\n",
      "Bootstrap #63 ROC area: 0.786\n",
      "Bootstrap #64 ROC area: 0.900\n",
      "Bootstrap #65 ROC area: 0.833\n",
      "Bootstrap #66 ROC area: 0.889\n",
      "Bootstrap #67 ROC area: 1.000\n",
      "Bootstrap #68 ROC area: 0.778\n",
      "Bootstrap #69 ROC area: 0.850\n",
      "Bootstrap #70 ROC area: 0.286\n",
      "Bootstrap #71 ROC area: 0.650\n",
      "Bootstrap #72 ROC area: 1.000\n",
      "Bootstrap #73 ROC area: 0.600\n",
      "Bootstrap #74 ROC area: 1.000\n",
      "Bootstrap #75 ROC area: 0.714\n",
      "Bootstrap #76 ROC area: 1.000\n",
      "Bootstrap #77 ROC area: 1.000\n",
      "Bootstrap #78 ROC area: 0.600\n",
      "Bootstrap #79 ROC area: 0.667\n",
      "Bootstrap #80 ROC area: 0.875\n",
      "Bootstrap #81 ROC area: 0.800\n",
      "Bootstrap #82 ROC area: 0.944\n",
      "Bootstrap #83 ROC area: 0.833\n",
      "Bootstrap #84 ROC area: 1.000\n",
      "Bootstrap #85 ROC area: 0.889\n",
      "Bootstrap #86 ROC area: 0.850\n",
      "Bootstrap #87 ROC area: 1.000\n",
      "Bootstrap #88 ROC area: 0.722\n",
      "Bootstrap #89 ROC area: 0.950\n",
      "Bootstrap #90 ROC area: 0.800\n",
      "Bootstrap #91 ROC area: 1.000\n",
      "Bootstrap #92 ROC area: 0.900\n",
      "Bootstrap #93 ROC area: 0.889\n",
      "Bootstrap #94 ROC area: 0.850\n",
      "Bootstrap #95 ROC area: 0.889\n",
      "Bootstrap #96 ROC area: 0.857\n",
      "Bootstrap #97 ROC area: 0.889\n",
      "Bootstrap #98 ROC area: 0.750\n",
      "Bootstrap #99 ROC area: 0.889\n",
      "Bootstrap #100 ROC area: 0.900\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXyklEQVR4nO3dfbRcdX3v8fcXA/IUeTBHVoBAEEEbrIKm1GdTRaVYBWqLcK+UVJBSRdRF74Wr3hbRVtqlYLuk9galoCgUUa5UqQoUmuIFbSgoAVQeDA8hQBAiINYKfO8f+3fKzmTOmTnnzDlzfsn7tdas7KfZ+zu/vecze357z0lkJpKk+mw27AIkSZNjgEtSpQxwSaqUAS5JlTLAJalSBrgkVWqTC/CIuCkilgy7jmGKiEMj4u6IeCwi9utj+SURcc+Atr00Iq4exLo2JRGREfG8Ydeh2WWjCvCIWBURB3RMWy8wMnOfzLyqx3oWljfMnGkqddg+ARyfmdtm5vWdM2sMi4l8yGxs+zciroqI/ygfyA9GxFcjYn7HMosi4pKI+FlEPBoRV0bEKzqW2SIiTomIWyPi5+X9dHZELOxj+w9HxDO7TD+mY9p6+ykaJ0TEyrLNeyLiyxHx65NukE3IRhXgtZgFwbE7cNOQa5jVZsE+mqjjM3Nb4HnAtjQf0gBExJ7Ad4AbgT2AnYGLgW9HxMtb67gIeCvw34DtgBcD1wGvH2ujJdxfDWR57kT9NfA+4ARgR2Bv4P8Cb57EuqYsIp4xjO1OWmZuNA9gFXBAx7SlwNXdlgH2B1YAjwD3A6eX6XfRHJCPlcfLaT7sPgzcCTwAfB7YrrXePyjzfgr8747tnELz5jivbOuYsu1rgHXAGuDTwBat9SXwbuBW4FHgo8CewP8r67iwvXzHa+5aK/DM8noS+Dlwe5fnLm/Nfwx4O7AEuAc4saxvDfCHrec8kyYw7irt+HfAVmPUtpQmTD4N/Az4IfD61vydgUuAh4DbgHd1bOdTwL3l8akybRvgF8BTrX228wT372hdZ5R9+LHS3v9cxh8Evghs33Es/S/gZuBh4O+BLcu80Tb7YHnuKuC/99tmwP8o7Xwv8M5S7/PGaNOrgGNa4+8GbmqNfwG4tMvzPgMsL8MHlDZcMMH33J+Wdjsd+Pp4dbXbpQzvBTwJ7D+B7S0F7qB5T/yko03fBdxS5t0MvKRM/7VSyzqaE5e3tp5zTmmHS2mO+QPKsfMVYG3Zxgmt5bseU0PLvGFufOAvZuIBfg1wZBneFnhZGV5Y3jBzWs97J02gPLcs+1XgC2XeIpogeBWwRXlj/or1A/xXwCE04boV8FLgZcCcsr1bgPe3tpfA14BnAfsAvwSuKNvfrhygR43RDmPW2lp31zDoNr+86Z4ATgU2Bw4CHgd2KPPPoAndHYG5wD8CHx/nDfgE8IGyrrfTBPmOZf5y4G+BLYF9y5vodWXeqcC1wHOAEZoPs4+2arynY1sT2b+jdb237JOtaM5m30ATtiOltk91HEsrgQXltX8H+FhHm51env9amoB4fq82Aw6kCYcX0nw4fWm8fUYrKIFnA5cDX2vNv4/WB25r+m/RBOhWwGnAv0ziPXcbzQfGS2mO8Z261dVxLI0G+HHAnRPY1jY0wTnahvOBfcrw7wOrgd8Aouy73csxdhvNB+kWwOtoAn50HefQHH+vpHlvbk3zreNPy/LPpfnAeNN4x9TQMm+YGx/4i2neUI/RfNKOPh5n7ABfDnwEmNexnoVs+Aa/Anh3a/z55YCdU3b2+a15WwP/yfoBvrxH7e8HLm6NJ/DK1vh1wEmt8U/SCpOOdY1Za2vdEw3wX3S0xwM0H0BBE0x7tua9HPjJGOteSnNWGa1p3wOOpAnCJ4G5rXkfB84pw7cDB7XmvQlY1aqxM8Ansn+XAnf12EeHANd3HEvHtcYPonyr4ekA36Y1/0Kab2fjthlwNnBaa97e4+0zmqB8nCaIErgB2K01/wngwC7Pe0FZfhfgLOCCCb7fXlWOq3ll/IfABzrqGi/APwRcO4HtbUPznn4bHd/wgG8B7+vynFfTfIBt1pp2PnBKGT4H+Hxr3m92Hgc037L+frxjaliPjbEP/JDM3H70QXN2MJajad4cP4yIf4uI3xln2Z1puiRG3UkT3juVeXePzsjMx2m+drfd3R6JiL0j4usRcV9EPAL8BTCv4zn3t4Z/0WV820nUOlk/zcwnWuOPl+2PUM5aImJdRKwDvlmmj2V1lndDq76dy+OhzHy0Y94uZbjb69p5nO1MZP/Chvtop4i4ICJWl310Hhvuo/ZzOut5ODN/3mV+rzZb73hi/dc8lhMyczvgRcAOwK6teQ/SnK12mk/T7fQwzfHabZnxHAV8OzMfLONfKtNGPUFzBty2OU3oM9FtlrZ8O82Z+5qI+EZEvKDMXkDzAd9pZ+DuzHyqNa19TMH6bb07sPPofin75oM8/d6Z6DE1rTbGAO9bZt6amUfQfCX/S+CiiNiG5qyk0700O3fUbjQH6P00fZX/9YaJiK1ovsqut7mO8c/QnLHslZnPojlIYvKvpu9aB+1Bmg+TfVofnNtlc0FtLLtERPu17sbT/do7RsTcjnmry3C313VvGd5gn01w/3Zbx1+Uab9e9tE72HAfLRijHoAdyvY65/dqszVd1tuXzLyRpv/+zFYbX07TxdDpMOCacsJxObB/ROzaZbkNlGP8MOC15STkPppusRdHxIvLYnfRfNtp24OnP5CuAHaNiMV9vTggM7+VmW+gCf4f0nxzgCaE9+zylHuBBRHRzrr2MQXr7/e7ab4Jbd96zM3Mg8r2xzqmhmKTDvCIeEdEjJRP53Vl8lM0/a5P0fR/jTof+EBE7BER29K8uf+hnJVeBLwlIl4REVvQdJn0CuO5NP15j5WziD8e0MvqVWs/7mf91z6m0nZnAWdExHMAImKXiHjTOE97DnBCRGweEb9Pc5Hp0sy8m6Zf++MRsWVEvIjmjOe81uv6cESMRMQ8mq6r0Xn3A8+OiO1GNzLB/dvNXJouuZ9FxC40FxY7vScido2IHWm6BP6hY/5Hyu15rwZ+B/hyH212IbC03Pq3NfBnPersdC7NGePoXSEfAV4REX8eETtGxNyIeC/NhfeTADLzcuAy4OKIeGlEzCnLHRcR7+yyjUNoursW0Vyr2JdmP/5rWS+lLf4wIvYvtwvuTRPyF5Rt3kpzveP8cnvhFmW/Hx4RJ3dusHwjOrgE5i9p9s3omfVngT8ptUdEPC8idge+S/Nt8X+W420J8JbRGrr4HvBoRJwUEVtFxDMi4oUR8RulhrGOqeEYdh/OIB9M/CLmeTR9uY/RXJ0+pLXcqTRv9HU0fb2b0QTG3WX6eZSLeK3t3MXTd6GsBl5d5p0CnNdR12toziAeoznoT+2os7Mf+mpgaWv8Y8Bnx2iHXrX26gM/juYscB3NWdYSNuxfbrfjljQfEnfQfCjdQuvKfZf90b4L5cfAG1vzdwW+TnMXyu2s38e8JfA3pbY1ZXjL1vyzS/uvo/nqPJH9u95xUpbZh+baw2M0/conttuB9e9CWUcTnFuXeUto7kL5EM0Z912Ui1/9tBlwMk3f7YTvQinTTgJWtMZfWNr1kfJ6rgJe1fGcLWjC/jaaPvo7aYJxty7b/CbwyS7TDyt1j15veWdp+0fKek9m/f7ooLmN8CaaoF1NE/z7dFn3fOBfynGzrryGRR3H7Y/K61sJ7Nfaj6PPuxk4tPWccygXnlvTdqY5WbiPpnvpWvrIjGE8ohSlASpnvetoukd+MuRyNE0iYhVNcF7eZd4Smg/tvrokpMnYpLtQBiki3hIRW5evd5+g+dHEquFWJWljZoAPzsE8fSFuL+Dw9OuNpGlkF4okVcozcEmq1Iz+wZ558+blwoULZ3KTklS966677sHM3ODHcTMa4AsXLmTFihUzuUlJql5EdP01rl0oklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEoZ4JJUqRn9JaYkbcwWnvyNMeetOu3NA9+eZ+CSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlegZ4RCyIiCsj4uaIuCki3lemnxIRqyPihvI4aPrLlSSN6uePWT0BnJiZ/x4Rc4HrIuKyMu+MzPzE9JUnSRpLzwDPzDXAmjL8aETcAuwy3YVJksY3oT7wiFgI7Ad8t0w6PiJ+EBFnR8QOYzzn2IhYEREr1q5dO7VqJUn/pe8Aj4htga8A78/MR4DPAHsC+9KcoX+y2/Myc1lmLs7MxSMjI1OvWJIE9BngEbE5TXh/MTO/CpCZ92fmk5n5FHAWsP/0lSlJ6tTPXSgBfA64JTNPb02f31rsUGDl4MuTJI2ln7tQXgkcCdwYETeUaR8EjoiIfYEEVgF/NA31SZLG0M9dKFcD0WXWpYMvR5LUL3+JKUmVMsAlqVIGuCRVygCXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcoAl6RKGeCSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEoZ4JJUqZ4BHhELIuLKiLg5Im6KiPeV6TtGxGURcWv5d4fpL1eSNKqfM/AngBMzcxHwMuA9EbEIOBm4IjP3Aq4o45KkGdIzwDNzTWb+exl+FLgF2AU4GDi3LHYucMg01ShJ6mJCfeARsRDYD/gusFNmrimz7gN2GuM5x0bEiohYsXbt2qnUKklq6TvAI2Jb4CvA+zPzkfa8zEwguz0vM5dl5uLMXDwyMjKlYiVJT+srwCNic5rw/mJmfrVMvj8i5pf584EHpqdESVI3/dyFEsDngFsy8/TWrEuAo8rwUcDXBl+eJGksc/pY5pXAkcCNEXFDmfZB4DTgwog4GrgTOGxaKpQkddUzwDPzaiDGmP36wZYjSeqXv8SUpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcoAl6RKGeCSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JleoZ4BFxdkQ8EBErW9NOiYjVEXFDeRw0vWVKkjr1cwZ+DnBgl+lnZOa+5XHpYMuSJPXSM8Azcznw0AzUIkmagKn0gR8fET8oXSw7jLVQRBwbESsiYsXatWunsDlJUttkA/wzwJ7AvsAa4JNjLZiZyzJzcWYuHhkZmeTmJEmdJhXgmXl/Zj6ZmU8BZwH7D7YsSVIvkwrwiJjfGj0UWDnWspKk6TGn1wIRcT6wBJgXEfcAfwYsiYh9gQRWAX80fSVKkrrpGeCZeUSXyZ+bhlokSRPgLzElqVIGuCRVygCXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcoAl6RKGeCSVCkDXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXKAJekShngklQpA1ySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEoZ4JJUKQNckirVM8Aj4uyIeCAiVram7RgRl0XEreXfHaa3TElSp37OwM8BDuyYdjJwRWbuBVxRxiVJM6hngGfmcuChjskHA+eW4XOBQwZbliSpl8n2ge+UmWvK8H3ATmMtGBHHRsSKiFixdu3aSW5OktRpyhcxMzOBHGf+ssxcnJmLR0ZGpro5SVIx2QC/PyLmA5R/HxhcSZKkfkw2wC8BjirDRwFfG0w5kqR+9XMb4fnANcDzI+KeiDgaOA14Q0TcChxQxiVJM2hOrwUy84gxZr1+wLVIkibAX2JKUqUMcEmqVM8uFEmarRae/I2u01ed9uYZrmQ4PAOXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlfI2Qkmz3li3C27qPAOXpEoZ4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalS3gcuaT2b+p9orYln4JJUKQNckiplgEtSpQxwSaqUAS5JlTLAJalSBrgkVcr7wKWNnPd1b7w8A5ekShngklQpA1ySKjWlPvCIWAU8CjwJPJGZiwdRlCSpt0FcxPytzHxwAOuRJE2AXSiSVKmpnoEn8O2ISOD/ZOayzgUi4ljgWIDddtttipuT6uXtfBq0qZ6BvyozXwL8NvCeiHhN5wKZuSwzF2fm4pGRkSluTpI0akoBnpmry78PABcD+w+iKElSb5MO8IjYJiLmjg4DbwRWDqowSdL4ptIHvhNwcUSMrudLmfnNgVQlSepp0gGemXcALx5gLZKkCfA2QkmqlAEuSZXyz8lKDPce7Y35/vBaXlstdXbyDFySKmWAS1KlDHBJqpQBLkmVMsAlqVIGuCRVygCXpEp5H7iqNt337461/kFuY6Jm2z3L47XRMNe1KfAMXJIqZYBLUqUMcEmqlAEuSZUywCWpUga4JFXK2wglzThvFxwMz8AlqVIGuCRVygCXpEoZ4JJUKQNckiplgEtSpQxwSapUNfeBz7Y/obkxm41/QlWD573Y9fMMXJIqZYBLUqUMcEmqlAEuSZWaUoBHxIER8aOIuC0iTh5UUZKk3iYd4BHxDOBM4LeBRcAREbFoUIVJksY3lTPw/YHbMvOOzPxP4ALg4MGUJUnqJTJzck+M+D3gwMw8powfCfxmZh7fsdyxwLFl9PnAjyZfblfzgAcHvM7pUkuttdQJ1jodaqkTNp1ad8/Mkc6J0/5DnsxcBiybrvVHxIrMXDxd6x+kWmqtpU6w1ulQS51grVPpQlkNLGiN71qmSZJmwFQC/N+AvSJij4jYAjgcuGQwZUmSepl0F0pmPhERxwPfAp4BnJ2ZNw2ssv5NW/fMNKil1lrqBGudDrXUCZt4rZO+iClJGi5/iSlJlTLAJalS1QR4r5/tR8RxEXFjRNwQEVcP81eh/f6JgYh4W0RkRAzlNqg+2nRpRKwtbXpDRBwzjDpLLT3bNCIOi4ibI+KmiPjSTNdYaujVpme02vPHEbFuCGWO1tKr1t0i4sqIuD4ifhARBw2jzlJLr1p3j4grSp1XRcSuQ6rz7Ih4ICJWjjE/IuJvyuv4QUS8ZEobzMxZ/6C5SHo78FxgC+D7wKKOZZ7VGn4r8M3ZWmtZbi6wHLgWWDwb6wSWAp+uZP/vBVwP7FDGnzMb6+xY/r00F/9na5suA/64DC8CVs3iWr8MHFWGXwd8YUi1vgZ4CbByjPkHAf8EBPAy4LtT2V4tZ+A9f7afmY+0RrcBhnV1tt8/MfBR4C+B/5jJ4lpq+lMI/dT6LuDMzHwYIDMfmOEaYeJtegRw/oxUtqF+ak3gWWV4O+DeGayvrZ9aFwH/XIav7DJ/RmTmcuChcRY5GPh8Nq4Fto+I+ZPdXi0Bvgtwd2v8njJtPRHxnoi4Hfgr4IQZqq1Tz1rL16YFmTnM/9OqrzYF3la+6l0UEQu6zJ8J/dS6N7B3RHwnIq6NiANnrLqn9dumRMTuwB48HTozrZ9aTwHeERH3AJfSfGMYhn5q/T7wu2X4UGBuRDx7BmqbqL6PkX7UEuB9ycwzM3NP4CTgw8Oup5uI2Aw4HThx2LX04R+BhZn5IuAy4Nwh1zOeOTTdKEtozmzPiojth1lQD4cDF2Xmk8MuZBxHAOdk5q40X/2/UI7f2ehPgNdGxPXAa2l+FT6b23YgZuvO6DTRn+1fABwynQWNo1etc4EXAldFxCqafrBLhnAhs2ebZuZPM/OXZfSzwEtnqLZO/ez/e4BLMvNXmfkT4Mc0gT6TJnKcHs7wuk+gv1qPBi4EyMxrgC1p/iDTTOvnWL03M383M/cDPlSmrZuxCvs32D9BMoyO/klcGJgD3EHzlXP0IsY+Hcvs1Rp+C7BittbasfxVDOciZj9tOr81fChw7WxtU+BA4NwyPI/ma+qzZ1udZbkXAKsoP6SbxW36T8DSMvxrNH3gM15zn7XOAzYrw38OnDrEtl3I2Bcx38z6FzG/N6VtDetFTqJRDqI5q7od+FCZdirw1jL818BNwA00FzHGDM1h19qx7FACvM82/Xhp0++XNn3BbG3T8oY4HbgZuBE4fDbWWcZPAU4bVltOoE0XAd8p+/8G4I2zuNbfA24ty3wWeOaQ6jwfWAP8iuZb4dHAccBxreP0zPI6bpzqe9+f0ktSpWrpA5ckdTDAJalSBrgkVcoAl6RKGeCSVCkDXJIqZYBLUqX+P7zAsYPcudtwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrap\n",
      "Confidence interval for the score: [0.600 - 1.0]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAsTAAALEwEAmpwYAAAziElEQVR4nO3de5xN9f748de7GZcICRW5My4zKiFyv+aaUIdEwhkp0nG6nKKL5EgpUURuuSSXqETll28XTqUUITGSOcMwLhmDQYW5vH9/7GXOGHPZmD1r9t7v5+OxH9bls9Z6r5kx7/l8Pmt9PqKqGGOMCV5XuB2AMcYYd1kiMMaYIGeJwBhjgpwlAmOMCXKWCIwxJshZIjDGmCBnicAYY4KcJQITcERkj4j8JSKnROSQiMwTkasylGkiIl+JyEkRSRSRj0UkPEOZ4iLyuojsdc71X2e9dN7ekTG+ZYnABKquqnoVUBe4BRh5boeINAb+D1gBlAOqAD8D60SkqlOmIPAlEAF0BIoDjYEEoKGvghaRUF+d25isWCIwAU1VDwGr8SSEc14B3lHVN1T1pKoeVdVngfXAaKfM/UBFoIeqRqlqqqoeVtV/q+qqzK4lIhEi8rmIHBWR30XkaWf7PBEZm65cKxGJS7e+R0SeEpGtwB/O8vsZzv2GiEx2lkuIyNsiclBE9ovIWBEJubyvlAlmlghMQBOR8kAnINpZLwI0AZZlUnwpcLuz3A74TFVPeXmdYsAXwGd4ahnV8dQovHUv0AW4GlgCdHbOifNLvhewyCk7D0h2rnEL0B4YdBHXMuY8lghMoPpIRE4C+4DDwPPO9mvw/NwfzOSYg8C59v9SWZTJyh3AIVV9TVVPOzWNHy7i+Mmquk9V/1LVWGAT0MPZ1wb4U1XXi8h1QGfgn6r6h6oeBiYBvS/iWsacxxKBCVTdVbUY0Aqoxf9+wR8DUoGymRxTFjjiLCdkUSYrFYD/XlKkHvsyrC/CU0sA6MP/agOVgALAQRE5LiLHgRnAtZdxbRPkLBGYgKaq/8HTlDLBWf8D+B7omUnxXvyvOecLoIOIFPXyUvuAqlns+wMokm79+sxCzbC+DGjlNG314H+JYB9wBiitqlc7n+KqGuFlnMZcwBKBCQavA7eLyM3O+gigv4j8Q0SKiUhJpzO3MfCCU2YBnl+6H4hILRG5QkRKicjTItI5k2t8ApQVkX+KSCHnvI2cfVvwtPlfIyLXA//MKWBVjQfWAnOB3aq6w9l+EM8TT685j7deISLVRKTlxX5RjDnHEoEJeM4v1XeAUc76t0AH4C48/QCxeDpdm6nqLqfMGTwdxr8CnwMngB/xNDFd0PavqifxdDR3BQ4Bu4DWzu4FeB5P3YPnl/h7Xoa+yIlhUYbt9wMFgSg8TV3vc3HNWMacR2xiGmOMCW5WIzDGmCBnicAYY4KcJQJjjAlylgiMMSbI+d0AV6VLl9bKlSu7HYYxxviVn3766Yiqlslsn98lgsqVK7Nx40a3wzDGGL8iIrFZ7bOmIWOMCXKWCIwxJshZIjDGmCDnd30EmUlKSiIuLo7Tp0+7HYrJpwoXLkz58uUpUKCA26EYk+8ERCKIi4ujWLFiVK5cGRFxOxyTz6gqCQkJxMXFUaVKFbfDMSbf8VnTkIjMEZHDIrIti/0iIpNFJFpEtopIvUu91unTpylVqpQlAZMpEaFUqVJWYzQmC77sI5iHZ9LvrHQCwpzPYOCty7mYJQGTHfv5MCZrPmsaUtWvRaRyNkW64ZlAXIH1InK1iJR1xls3xnhp0Q97WbFlv9thGB9KTU3h7Nkk6lW9lue75v4cRG4+NXQD50/PF+dsu4CIDBaRjSKyMT4+Pk+CM8ZfrNiyn6iDJ9wOw/jI8ePH2bBhI9u3b8dn0waoqs8+QGVgWxb7PsEzEci59S+BBjmds379+ppRVFTUBdvy0u7duzUiIsKVa6empmrr1q01MTExbdvy5csV0B07dqRtW7NmjXbp0uW8Y/v376/Lli1TVdWzZ8/qU089pdWrV9dbbrlFb7vtNl21atVlxzdu3DitVq2a1qhRQz/77LNMy3zxxRd6yy236M0336xNmzbVXbt2qarq6dOntVevXlqtWjVt2LCh7t69O9vznjlzRps3b65JSUmZXsftnxNf6TX9O+01/Tu3wzC57NixYzpo0CAFtHr16rp27drLOh+wUbP4vepmjWA/ngm/zynvbAsqycnJl3X8qlWruPnmmylevHjatsWLF9OsWTMWL17s9Xmee+45Dh48yLZt29i0aRMfffQRJ0+evKzYoqKiWLJkCdu3b+ezzz5j6NChpKSkXFBuyJAhLFy4kC1bttCnTx/Gjh0LwNtvv03JkiWJjo7m0Ucf5amnnsr2vAULFqRt27a89563E4AZkz+lpKTQpEkT5syZw5NPPsnWrVtp2dJ3s5G6+fjoSmCYiCwBGgGJmgv9Ay98vJ2oA7lbTQ4vV9zrdrmYmBjuvvtuZs6cyTXXXMPDDz9MfHw8RYoUYdasWdSqVYsBAwZQuHBhNm/eTNOmTenduzfDhw/n9OnTXHnllcydO5eaNWuyfft2Bg4cyNmzZ0lNTeWDDz4gLCzsvOstXLiQwYMHp62fOnWKb7/9ljVr1tC1a1deeOGFjCFe4M8//2TWrFns3r2bQoUKAXDdddfRq1evi/gqXWjFihX07t2bQoUKUaVKFapXr86PP/5I48aNzysnIpw44fmeJSYmUq5cubTjR48eDcDf/vY3hg0bhqpme97u3bszcuRI+vbte1mxG+OGhIQErrnmGkJCQnjxxRepUKECDRo08Pl1fZYIRGQx0AooLSJxwPNAAQBVnQ6sAjoD0cCfwEBfxZJXdu7cSe/evZk3bx4333wzbdu2Zfr06YSFhfHDDz8wdOhQvvrqK8Dz7sN3331HSEgIJ06c4JtvviE0NJQvvviCp59+mg8++IDp06czfPhw+vbty9mzZzP9a3rdunXMmDEjbX3FihV07NiRGjVqUKpUKX766Sfq16+fbdzR0dFUrFjxvFpFVh599FHWrFlzwfbevXszYsSI87bt37+f2267LW29fPny7N9/YaVv9uzZdO7cmSuvvJLixYuzfv36tOMrVPBUGkNDQylRogQJCQnZnrdOnTps2LAhx/swJj9RVRYuXMjw4cN5+eWXeeCBB+jRo0eeXd+XTw3dm8N+BR7O7ev6okfdG/Hx8XTr1o0PP/yQ8PBwTp06xXfffUfPnj3Typw5cyZtuWfPnoSEhACev4L79+/Prl27EBGSkpIAaNy4MS+++CJxcXHcddddF9QGAI4ePUqxYsXS1hcvXszw4cMBzy/nxYsXU79+/Swfn7zYxyonTZp0UeW9PeeqVato1KgRr776Ko899hizZ8++pHOFhIRQsGBBTp48ed7XxZj8at++fTz00EOsWrWK2267jaZNm+Z5DAHxZnF+UKJECSpWrMi3335LeHg4qampXH311WzZsiXT8kWLFk1bfu6552jdujXLly9nz549tGrVCoA+ffrQqFEjPv30Uzp37syMGTNo06bNeecJDQ0lNTWVK664gqNHj/LVV1/xyy+/ICKkpKQgIrz66quUKlWKY8eOnXfs0aNHKV26NNWrV2fv3r2cOHEix1rBxdQIbrjhBvbt+9+DYXFxcdxww/kPhsXHx/Pzzz/TqFEjAO655x46dux43vHly5cnOTmZxMRESpUqleN5z5w5Q+HChbO9D2Pyg8WLF/Pggw+SkpLC66+/zrBhw9L+QMxLNuhcLilYsCDLly/nnXfeYdGiRRQvXpwqVaqwbNkywFP1+/nnnzM9NjExMe0X2bx589K2x8TEULVqVf7xj3/QrVs3tm7desGxNWvWJCYmBoD333+ffv36ERsby549e9i3bx9VqlThm2++ISwsjAMHDrBjxw4AYmNj+fnnn6lbty5FihQhMjKS4cOHc/bsWcDzC/pc7OlNmjSJLVu2XPDJmAQA7rzzTpYsWcKZM2fYvXs3u3btomHDhueVKVmyJImJifz2228AfP7559SuXTvt+Pnz56fdW5s2bRCRbM+bkJBA6dKlbUwh4xdKlixJo0aN2LZtG8OHD3clCYDVCHJV0aJF+eSTT7j99tu56qqrWLhwIUOGDGHs2LEkJSXRu3dvbr755guOe/LJJ+nfvz9jx46lS5cuaduXLl3KggULKFCgANdffz1PP/30Bcd26dKFtWvXUr16dRYvXpz2ZM05d999N4sXL6ZFixa8++67DBw4kNOnT1OgQAFmz55NiRIlABg7dizPPvss4eHhFC5cmKJFizJmzJjL+npERETQq1cvwsPDCQ0NZerUqWk/6J07d2b27NmUK1eOWbNmcffdd3PFFVdQsmRJ5syZA0BkZCT9+vWjevXqXHPNNSxZsiTH865Zs+a8r6Ex+UlycjKTJk3i7NmzPPPMM3Ts2JEOHTq4/+Z7Vs+V5tdPfnyPwE0HDhzQdu3auR1GvtGjRw/duXNnpvsC9efE3iPwD1u2bNH69esroL169dLU1NQ8vT759D0CkwvKli3LAw88kPb4ZTA7e/Ys3bt3p0aNGm6HYkyaM2fO8Nxzz9GgQQP27dvHsmXLWLJkifu1gHQCpmlIVfPVFzYvXe7z/oGiYMGC3H///ZnuU1+9mm9MDnbt2sX48ePp06cPEydOpFSpUm6HdIGAqBEULlyYhIQE+89uMqXOfAT2JJHJK6dOnWLhwoWA592WX3/9lfnz5+fLJAABUiMoX748cXFx2IB0JivnZigzxtc+//xzBg8eTGxsLPXq1aN27dpUrVrV7bCyFRCJoECBAjbzlDHGVceOHeOJJ55gzpw51KhRg//85z9pj0LndwGRCIwxxk0pKSk0bdqU3377jZEjRzJq1Ci/aoq0RGCMMZfoyJEjaYPEjRs3jooVK1Kv3iXPuuuagOgsNsaYvKSqvPPOO9SoUSNtXKzu3bv7ZRIASwTGGHNRYmNj6dSpE/3796d27dq0aNHC7ZAumyUCY4zx0rvvvkudOnX49ttvmTJlCt988w21atVyO6zLZn0ExhjjpTJlytC0aVNmzJhBpUqV3A4n11giMMaYLCQlJfHaa6+RlJTEc889R4cOHWjfvn3AjWJgTUPGGJOJzZs306hRI0aOHElUVFTayAWBlgTAEoExxpzn9OnTPP3009x6660cOHCADz74gMWLFwdkAjjHEoExxqQTHR3NhAkTuP/++9mxYwd33XWX2yH5nPURGGOC3qlTp1i+fDn9+vWjTp067Ny5M6iGrbEagTEmqK1evZqIiAj69++fNpVrMCUBsERgjAlSCQkJ9O/fn44dO1KkSBG++eYbvxkkLrdZ05AxJuicGyQuOjqaZ555hmeffdavBonLbZYIjDFBIz4+nlKlShESEsL48eOpVKkSdevWdTss11nTkDEm4Kkqc+fOpUaNGsyaNQuAbt26WRJwWCIwxgS0PXv20KFDB/7+979z44030rp1a7dDyncsERhjAtaCBQuoU6cO33//PdOmTWPt2rXUqFHD7bDyHesjMMYErOuuu44WLVowffp0Klas6HY4+ZYlAmNMwEhKSuKVV14hJSWFUaNG0b59e9q3b+92WPmeNQ0ZYwLCpk2buPXWW3n22WfZuXNn2iBxJmeWCIwxfu2vv/5ixIgRNGzYkN9//53ly5ezcOHCgB4kLrf5NBGISEcR2Ski0SIyIpP9FUVkjYhsFpGtItLZl/EYYwJPTEwMEydOZMCAAURFRdG9e3e3Q/I7PksEIhICTAU6AeHAvSISnqHYs8BSVb0F6A1M81U8xpjAceLECebNmwdAREQEu3btYvbs2ZQsWdLdwPyUL2sEDYFoVY1R1bPAEqBbhjIKFHeWSwAHfBiPMSYArFq1ijp16hAZGZk2SFwgTRvpBl8mghuAfenW45xt6Y0G7hOROGAV8EhmJxKRwSKyUUQ2xsfH+yJWY0w+d+TIEfr160eXLl0oVqwY69atC9pB4nKb253F9wLzVLU80BlYICIXxKSqM1W1gao2KFOmTJ4HaYxx17lB4pYsWcKoUaPYtGkTt912m9thBQxfvkewH6iQbr28sy29SKAjgKp+LyKFgdLAYR/GZYzxE7///jtlypQhJCSECRMmUKlSJW666Sa3wwo4vqwRbADCRKSKiBTE0xm8MkOZvUBbABGpDRQGrO3HmCCnqrz99tvUrFmTmTNnAtC1a1dLAj7is0SgqsnAMGA1sAPP00HbRWSMiNzpFHsceEBEfgYWAwPU3gIxJqjFxMTQrl07Bg0aRN26dWnXrp3bIQU8nw4xoaqr8HQCp982Kt1yFNDUlzEYY/zH/PnzGTp0KCEhIUyfPp0HHniAK65wuysz8NlYQ8aYfKNcuXK0adOGt956i/Lly7sdTtCwRGCMcc3Zs2d5+eWXSU1NZfTo0dx+++3cfvvtbocVdKzOZYxxxYYNG6hfvz7PP/88MTExNkici6xGEMAW/bCXFVsyPrFrAk3UwROEly2ec8F84s8//2TUqFFMmjSJsmXLsnLlSrp27ep2WEHNagQBbMWW/UQdPOF2GMbHwssWp1vdjC/t51+7d+9mypQpPPDAA2zfvt2SQD5gNYIAF162OO892NjtMEyQS0xM5MMPP2TgwIFEREQQHR1NhQoVcj7Q5AmrERhjfOrTTz8lIiKCQYMG8euvvwJYEshnLBEYY3wiPj6evn37cscdd1CyZEm+//57atWq5XZYJhPWNGSMyXUpKSk0a9aM3bt388ILLzBixAgKFizodlgmC5YIjDG55tChQ1x77bWEhITw2muvUblyZerUqeN2WCYH1jRkjLlsqampzJgxgxo1ajBjxgwA7rjjDksCfsKrRCAiV4pITV8HY4zxP9HR0bRt25aHHnqIW2+9lQ4dOrgdkrlIOSYCEekKbAE+c9brikjG4aSNMUFo7ty53HjjjWzatIlZs2bxxRdfULVqVbfDMhfJmxrBaDzzDx8HUNUtQBWfRWSM8RsVK1akQ4cOREVFMWjQIETE7ZDMJfCmszhJVRMzfINtUBBjgtCZM2d46aWXSE1NZcyYMbRt25a2bdu6HZa5TN7UCLaLSB8gRETCRGQK8J2P4zLG5DM//PAD9evX54UXXmDv3r02SFwA8SYRPAJEAGeARUAiMNyXQRlj8o8//viDxx57jMaNG5OYmMgnn3zCvHnzrBkogHiTCLqo6jOqeqvzeRa4M8ejjDEBITY2lmnTpvHQQw+xfft2unTp4nZIJpd5kwhGernNGBMgjh8/zuzZswEIDw8nOjqaadOmUby4/wx3bbyXZWexiHQCOgM3iMjkdLuKA8m+DswY444VK1YwZMgQDh8+TLNmzahVq5ZNGxngsqsRHAA2AqeBn9J9VgL2xogxAebw4cP07t2b7t27U6ZMGdavX2+DxAWJLGsEqvoz8LOILFLVpDyMyRiTx1JSUmjatCl79+5l7NixPPnkkxQoUMDtsEwe8eY9gsoi8hIQDhQ+t1FV7fVBY/zcgQMHuP766wkJCeGNN96gcuXKhIeHux2WyWPedBbPBd7C0y/QGngHeNeXQRljfCs1NZW33nqLWrVqMX36dAA6d+5sSSBIeZMIrlTVLwFR1VhVHQ3Y82PG+KnffvuN1q1bM3ToUBo1akSnTp3cDsm4zJumoTMicgWwS0SGAfuBq3wbljHGF95++22GDRtG4cKFmTNnDgMGDLAXw4xXNYLhQBHgH0B94D6gvy+DMsb4RuXKlenUqRNRUVEMHDjQkoABcqgRiEgIcI+qPgGcAgbmSVTGmFxx5swZ/v3vfwMwduxYGyTOZCrbGoGqpgDN8igWY0wu+u6776hbty4vvvgiBw8etEHiTJa86SPY7ExEswz449xGVf3QZ1EZYy7ZqVOneOaZZ5gyZQoVKlTgs88+s1nDTLa86SMoDCQAbYCuzucOb04uIh1FZKeIRIvIiCzK9BKRKBHZLiKLvA3cGJO5vXv3MmPGDB5++GG2bdtmScDkKMcagapeUr+A078wFbgdiAM2iMhKVY1KVyYMzwB2TVX1mIhceynXMibYHTt2jGXLljF48GDCw8OJiYmhXLlybodl/IRXk9dfooZAtKrGqOpZYAnQLUOZB4CpqnoMQFUP+zAeYwLS8uXLCQ8PZ+jQoezcuRPAkoC5KL5MBDcA+9Ktxznb0qsB1BCRdSKyXkQ6ZnYiERksIhtFZGN8fLyPwjXGvxw6dIiePXty1113cf311/Pjjz9Ss2ZNt8MyfsibzmJfXz8MaAWUB74WkRtV9Xj6Qqo6E5gJ0KBBA3v0wQS9lJQUmjdvzr59+xg3bhxPPPGEDRJnLlmOiUBErgPGAeVUtZOIhAONVfXtHA7dD1RIt17e2ZZeHPCDM7rpbhH5DU9i2ODtDRgTTOLi4ihXrhwhISFMnjyZKlWq2FDR5rJ50zQ0D1gNnGt0/A34pxfHbQDCRKSKiBQEeuOZyyC9j/DUBhCR0niaimK8OLcxQSU1NZUpU6ZQq1Yt3nrrLQA6depkScDkCm8SQWlVXQqkAqhqMpCS00FOuWF4ksgOYKmqbheRMSJybs7j1UCCiEQBa4B/qWrCJdyHMQHr119/pUWLFvzjH/+gWbNm3HGHV09vG+M1b/oI/hCRUoACiMhtQKI3J1fVVcCqDNtGpVtW4DHnY4zJYPbs2QwbNowiRYowf/58+vXrZ+MDmVznTSJ4HE+TTjURWQeUAf7m06iMMQBUq1aNrl278uabb3Lddde5HY4JUN68UPaTiLQEagIC7LSpK43xjdOnTzNmzBgAxo0bR+vWrWndurXLUZlAl2MfgYhsBZ4ETqvqNksCxvjGunXrqFu3Li+99BLx8fE2SJzJM950FnfFM03lUhHZICJPiEhFH8dlTNA4efIkjzzyCM2bN+fMmTOsXr2aWbNmWV+AyTM5JgJnespXVLU+0Ae4Cdjt88iMCRJxcXHMnj2bRx55hF9++YX27du7HZIJMl69WSwilYB7nE8KnqYiY8wlSkhIYOnSpQwZMoTatWsTExND2bJl3Q7LBClv3iz+ASiAZz6CnqpqL3wZc4lUlQ8++ICHH36Yo0eP0qZNG2rWrGlJwLjKmz6C+1W1nqq+ZEnAmEt38OBB7r77bnr27EmFChXYuHGjDRJn8oUsawQicp+qvgt0EZEuGfer6kSfRmZMADk3SNz+/ft55ZVXePTRRwkNdXvMR2M8svtJLOr8WyyTffZcmzFe2LdvHzfccAMhISFMnTqVKlWqUKNGDbfDMuY8WTYNqeoMZ/ELVX0h/Qf4Mm/CM8Y/paSkMHny5PMGievQoYMlAZMvedNHMMXLbcYYYMeOHTRv3pzhw4fTsmVLunbt6nZIxmQruz6CxkAToIyIpB8UrjgQ4uvAjPFHM2fO5JFHHqFYsWIsWLCAvn372othJt/Lro+gIHCVUyZ9P8EJ/HDQuUU/7GXFlozz4gS2qIMnCC9b3O0wgkpYWBg9evRg8uTJXHvttW6HY4xXskwEqvof4D8iMk9VY/MwJp9YsWV/0P1iDC9bnG51M04TbXLTX3/9xejRoxERXn75ZRskzvil7JqGXlfVfwJvisgFTwmp6p0XHpW/hZctznsPNnY7DBMgvv76awYNGsSuXbt46KGHUFVrBjJ+KbumoQXOvxPyIhBj/MWJEycYMWIEb731FlWrVuXLL7+kTZs2bodlzCXLrmnoJ+ff/5zbJiIlgQqqujUPYjMmXzpw4ADz5s3jscceY8yYMRQtWjTng4zJx7wZa2gtcKdT9ifgsIisU1WbXtIEjSNHjrB06VKGDh1KrVq12L17t80YZgKGN+8RlFDVE8BdwDuq2gho59uwjMkfVJX33nuP8PBw/vnPf/Lbb78BWBIwAcWbRBAqImWBXsAnPo7HmHzjwIEDdO/end69e1OpUiV++uknezPYBCRvRr0aA6wG1qnqBhGpCuzybVjGuCslJYUWLVqwf/9+JkyYwPDhw22QOBOwvJm8fhmeuQjOrccAd/syKGPcEhsbS/ny5QkJCWHatGlUrVqV6tWrux2WMT7lzeT15UVkuYgcdj4fiEj5vAjOmLySkpLCxIkTqV27dtogce3bt7ckYIKCN30Ec4GVQDnn87GzzZiAsG3bNpo0acLjjz9O27Zt6d69u9shGZOnvEkEZVR1rqomO595QBkfx2VMnpg+fTr16tUjJiaGRYsWsXLlSsqXtwqvCS7eJIIEEblPREKcz31Agq8DM8aXVD2jptSuXZuePXsSFRXFvffea0NEmKDkzWMQf8cz/8AkZ30dMNBnERnjQ3/++SejRo0iJCSE8ePH07JlS1q2bOl2WMa4KscagarGquqdqlrG+XRX1b15EZwxuWnt2rXcdNNNvPbaa5w6dSqtVmBMsPPmqaGqIvKxiMQ7Tw2tcN4lMMYvJCYm8uCDD6YND/3VV18xdepUawYyxuFNH8EiYClQFs9TQ8uAxb4MypjcdPDgQd59912eeOIJtm7davMFGJOBN4mgiKouSPfU0LtAYW9OLiIdRWSniESLyIhsyt0tIioiDbwN3JjsxMfHM2WKZ2rtWrVqsWfPHl599VWKFCnicmTG5D/eJIL/JyIjRKSyiFQSkSeBVSJyjYhck9VBIhICTAU6AeHAvSISnkm5YsBw4IdLuwVj/kdVWbRoEbVr1+bxxx9PGySuTBl74tmYrHiTCHoBDwJrgLXAEKA3niGpN2ZzXEMgWlVjVPUssATolkm5fwPjgdPeh23Mhfbt20fXrl3p27cv1atXZ/PmzTZInDFe8GasoSqXeO4bgH3p1uOARukLiEg9PBPdfCoi/8rqRCIyGBgMULFixUsMxwSy5ORkWrVqxaFDh5g0aRKPPPIIISEhbodljF9wbThFEbkCmAgMyKmsqs4EZgI0aNDAnvkzafbs2UOFChUIDQ1lxowZVK1alapV7aE2Yy6GN01Dl2o/UCHdenln2znFgDrAWhHZA9wGrLQOY+ON5ORkJkyYQO3atZk2bRoA7dq1syRgzCXwZY1gAxAmIlXwJIDeQJ9zO1U1ESh9bt2ZEvMJVc2u38EYtm7dSmRkJBs3bqRbt27cfbeNim7M5fDmhTJxxhoa5axXFJGGOR2nqsnAMDyT2uwAlqrqdhEZIyJ3Xm7gJjhNmzaN+vXrExsby3vvvcfy5cspV66c22EZ49e8qRFMA1KBNnhmKzsJfADcmtOBqroKWJVh26gsyrbyIhYTpFQVEaFOnTr07t2bSZMmUbp06ZwPNMbkyJtE0EhV64nIZgBVPSYiBX0clzEA/PHHHzz77LOEhoby6quv0qJFC1q0aOF2WMYEFG86i5Ocl8MUQETK4KkhGONTX375JTfeeCOvv/46Z86csUHijPERbxLBZGA5cK2IvAh8C4zzaVQmqB0/fpxBgwbRrl07QkND+frrr5k8ebINEmeMj3jzQtlCEfkJaAsI0F1Vd/g8MhO0fv/9d5YsWcJTTz3F888/z5VXXul2SMYEtBwTgYhUBP7EM1dx2jabk8DkpnO//IcPH07NmjXZs2ePdQYbk0e86Sz+FE//gOAZdbQKsBOI8GFcJkioKgsXLmT48OGcOnWKzp07ExYWZknAmDzkzQxlN6rqTc6/YXgGk/ve96GZQLd37166dOlCv379qFmzJlu2bCEsLMztsIwJOhf9ZrGqbhKRRjmXNCZr5waJO3z4MJMnT2bo0KE2SJwxLvGmj+CxdKtXAPWAAz6LyAS0mJgYKlWqRGhoKLNmzaJatWpUrlzZ7bCMCWrePD5aLN2nEJ4+g8zmFTAmS8nJyYwfP57w8HCmTp0KQNu2bS0JGJMPZFsjcF4kK6aqT+RRPCYAbdmyhcjISDZt2kSPHj3o2bOn2yEZY9LJskYgIqGqmgI0zcN4TIB58803ufXWW9m/fz/vv/8+H374IWXLlnU7LGNMOtnVCH7E0x+wRURWAsuAP87tVNUPfRyb8WPnBom76aab6Nu3LxMnTuSaa7Kc4toY4yJvnhoqDCTgGX303PsEClgiMBc4deoUzzzzDAUKFGDChAk2SJwxfiC7zuJrnSeGtgG/OP9ud/7dlgexGT/zf//3f9SpU4cpU6aQlJRkg8QZ4yeyqxGEAFfhqQFkZP/DTZpjx47x2GOPMW/ePGrWrMnXX39Ns2bN3A7LGOOl7BLBQVUdk2eRGL91+PBh3n//fUaOHMmoUaMoXLiw2yEZYy5CdonAxvw1WTp06BCLFy/m0UcfTRskrlSpUm6HZYy5BNn1EbTNsyiM31BV5s+fT3h4OCNHjmTXrl0AlgSM8WNZJgJVPZqXgZj8b8+ePXTs2JEBAwYQHh5ug8QZEyAuetA5E5ySk5Np3bo1R44cYerUqTz00ENccYU3I5QYY/I7SwQmW9HR0VSpUoXQ0FDmzJlD1apVqVSpktthGWNykf1JZzKVlJTEuHHjiIiISBskrnXr1pYEjAlAViMwF9i0aRORkZFs2bKFnj17cs8997gdkjHGh6xGYM4zefJkGjZsyKFDh/jwww9ZunQp1113ndthGWN8yBKBAUgbDuKWW27h/vvvJyoqih49ergclTEmL1jTUJA7efIkI0eOpFChQrz22ms0b96c5s2bux2WMSYPWY0giH322WfUqVOHadOmoao2SJwxQcoSQRBKSEigf//+dOrUiaJFi7Ju3TomTpyIiI0qYkwwskQQhBISEli+fDnPPfccmzdvpnHjxm6HZIxxkU8TgYh0FJGdIhItIiMy2f+YiESJyFYR+VJE7CF1Hzl48CATJkxAValRowaxsbGMGTOGQoUKuR2aMcZlPksEzsT3U4FOQDhwr4iEZyi2GWigqjcB7wOv+CqeYKWqzJkzh9q1a/Pcc88RHR0NQMmSJV2OzBiTX/iyRtAQiFbVGFU9CywBuqUvoKprVPVPZ3U9UN6H8QSd3bt30759eyIjI7n55pv5+eefbZA4Y8wFfPn46A3AvnTrcUCjbMpHAv8vsx0iMhgYDFCxYsXcii+gJScn06ZNGxISEnjrrbcYPHiwDRJnjMlUvniPQETuAxoALTPbr6ozgZkADRo0sGccs7Fr1y6qVq1KaGgoc+fOpVq1alSoUMHtsIwx+Zgv/0TcD6T/DVTe2XYeEWkHPAPcqapnfBhPQEtKSmLs2LHUqVOHN998E4BWrVpZEjDG5MiXNYINQJiIVMGTAHoDfdIXEJFbgBlAR1U97MNYAtrGjRuJjIxk69at9O7dm3vvvdftkIwxfsRnNQJVTQaGAauBHcBSVd0uImNE5E6n2KvAVcAyEdkiIit9FU+geuONN2jUqBFHjhxhxYoVLF68mGuvvdbtsIwxfsSnfQSqugpYlWHbqHTL7Xx5/UCmqogIDRo0IDIykldeeYWrr77a7bCMMX4oX3QWG++dOHGCp556isKFCzNp0iSaNm1K06ZN3Q7LGOPH7HlCP7Jq1SoiIiKYOXMmoaGhNkicMSZXWCLwA0eOHOG+++6jS5culChRgu+++45XX33VBokzxuQKSwR+4NixY3z88cc8//zzbNq0iUaNsnsvzxhjLo71EeRT+/fvZ+HChfzrX/8iLCyM2NhY6ww2xviE1QjyGVVl1qxZhIeHM3r0aP773/8CWBIwxviMJYJ85L///S9t27Zl8ODB1KtXj61bt1K9enW3wzLGBDhrGsonkpOTadu2LUePHmXGjBkMGjTIBokzxuQJSwQu27lzJ9WqVSM0NJT58+dTrVo1ype30biNMXnH/uR0ydmzZ3nhhRe48cYbmTp1KgAtW7a0JGCMyXNWI3DBjz/+SGRkJNu2baNPnz707dvX7ZCMMUHMagR57PXXX6dx48Zp7wYsXLiQ0qVLux2WMSaIWSLII+eGg2jYsCEPPPAA27dv54477nA5KmOMsaYhn0tMTOTJJ5/kyiuv5PXXX6dJkyY0adLE7bCMMSaN1Qh86OOPPyY8PJzZs2dTqFAhGyTOGJMvWSLwgfj4ePr06cOdd95JqVKlWL9+PePHj7dB4owx+ZIlAh9ITExk1apVvPDCC2zcuJFbb73V7ZCMMSZL1keQS/bt28e7777LiBEjqF69OrGxsZQoUcLtsIwxJkdWI7hMqampTJ8+nYiICMaOHZs2SJwlAWOMv7BEcBl27dpFmzZtGDJkCA0bNuSXX36xQeKMMX7HmoYuUXJyMrfffjvHjx/n7bffZuDAgdYZbIzxS5YILtKOHTsICwsjNDSUBQsWUK1aNcqVK+d2WMYYc8msachLZ86c4fnnn+emm27izTffBKB58+aWBIwxfs9qBF5Yv349kZGRREVF0a9fP/r16+d2SMYYk2usRpCD1157jSZNmnDy5ElWrVrFO++8Q6lSpdwOyxhjco0lgiykpqYC0LhxYx566CG2bdtGp06dXI7KGGNynzUNZXD8+HEef/xxihQpwpQpU2yQOGNMwLMaQTofffQR4eHhzJ8/n2LFitkgccaYoGCJADh8+DC9evWiR48eXHfddfz444+MGzfO3gswxgQFSwTAiRMn+Pzzz3nxxRf58ccfqVevntshGWNMngnaPoK9e/eyYMECnn76aapXr87evXspVqyY22EZY0ye82mNQEQ6ishOEYkWkRGZ7C8kIu85+38Qkcq+jAc8TwNNmzaNiIgIxo0blzZInCUBY0yw8lkiEJEQYCrQCQgH7hWR8AzFIoFjqlodmASM91U8AH/99SetWrXi4YcfpnHjxmzfvt0GiTPGBD1f1ggaAtGqGqOqZ4ElQLcMZboB853l94G24qMeWlVl69at/PLLL8ydO5fVq1dTuXJlX1zKGGP8ii/7CG4A9qVbjwMaZVVGVZNFJBEoBRxJX0hEBgODASpWrHhJwUTcUIKSjeow+sUoypYte0nnMMaYQOQXncWqOhOYCdCgQYNLerj/+a4RQERuhmWMMQHBl01D+4EK6dbLO9syLSMioUAJIMGHMRljjMnAl4lgAxAmIlVEpCDQG1iZocxKoL+z/DfgK7XXeY0xJk/5rGnIafMfBqwGQoA5qrpdRMYAG1V1JfA2sEBEooGjeJKFMcaYPOTTPgJVXQWsyrBtVLrl00BPX8ZgjDEmezbEhDHGBDlLBMYYE+QsERhjTJCzRGCMMUFO/O1pTRGJB2Iv8fDSZHhrOQjYPQcHu+fgcDn3XElVy2S2w+8SweUQkY2q2sDtOPKS3XNwsHsODr66Z2saMsaYIGeJwBhjglywJYKZbgfgArvn4GD3HBx8cs9B1UdgjDHmQsFWIzDGGJOBJQJjjAlyAZkIRKSjiOwUkWgRGZHJ/kIi8p6z/wcRqexCmLnKi3t+TESiRGSriHwpIpXciDM35XTP6crdLSIqIn7/qKE39ywivZzv9XYRWZTXMeY2L362K4rIGhHZ7Px8d3YjztwiInNE5LCIbMtiv4jIZOfrsVVE6l32RVU1oD54hrz+L1AVKAj8DIRnKDMUmO4s9wbeczvuPLjn1kARZ3lIMNyzU64Y8DWwHmjgdtx58H0OAzYDJZ31a92OOw/ueSYwxFkOB/a4Hfdl3nMLoB6wLYv9nYH/BwhwG/DD5V4zEGsEDYFoVY1R1bPAEqBbhjLdgPnO8vtAWxGRPIwxt+V4z6q6RlX/dFbX45kxzp95830G+DcwHjidl8H5iDf3/AAwVVWPAajq4TyOMbd5c88KFHeWSwAH8jC+XKeqX+OZnyUr3YB31GM9cLWIXNZE7IGYCG4A9qVbj3O2ZVpGVZOBRKBUnkTnG97cc3qReP6i8Gc53rNTZa6gqp/mZWA+5M33uQZQQ0TWich6EemYZ9H5hjf3PBq4T0Ti8Mx/8kjehOaai/3/niO/mLze5B4RuQ9oALR0OxZfEpErgInAAJdDyWuheJqHWuGp9X0tIjeq6nE3g/Kxe4F5qvqaiDTGM+thHVVNdTswfxGINYL9QIV06+WdbZmWEZFQPNXJhDyJzje8uWdEpB3wDHCnqp7Jo9h8Jad7LgbUAdaKyB48bakr/bzD2JvvcxywUlWTVHU38BuexOCvvLnnSGApgKp+DxTGMzhboPLq//vFCMREsAEIE5EqIlIQT2fwygxlVgL9neW/AV+p0wvjp3K8ZxG5BZiBJwn4e7sx5HDPqpqoqqVVtbKqVsbTL3Knqm50J9xc4c3P9kd4agOISGk8TUUxeRhjbvPmnvcCbQFEpDaeRBCfp1HmrZXA/c7TQ7cBiap68HJOGHBNQ6qaLCLDgNV4njiYo6rbRWQMsFFVVwJv46k+RuPplOntXsSXz8t7fhW4Cljm9IvvVdU7XQv6Mnl5zwHFy3teDbQXkSggBfiXqvptbdfLe34cmCUij+LpOB7gz3/YichiPMm8tNPv8TxQAEBVp+PpB+kMRAN/AgMv+5p+/PUyxhiTCwKxacgYY8xFsERgjDFBzhKBMcYEOUsExhgT5CwRGGNMkLNEYPItEUkRkS3pPpWzKXsqD0PLkoiUE5H3neW66UfCFJE7sxsl1QexVBaRPnl1PeO/7PFRk2+JyClVvSq3y+YVERmAZ8TTYT68RqgzXlZm+1oBT6jqHb66vgkMViMwfkNErnLmUtgkIr+IyAWjjYpIWRH52qlBbBOR5s729iLyvXPsMhG5IGmIyFoReSPdsQ2d7deIyEfO2O/rReQmZ3vLdLWVzSJSzPkrfJvzFuwY4B5n/z0iMkBE3hSREiIS64yHhIgUFZF9IlJARKqJyGci8pOIfCMitTKJc7SILBCRdXhejKzslN3kfJo4RV8GmjvXf1REQkTkVRHZ4NzLg7n0rTH+zu2xt+1jn6w+eN6M3eJ8luN5E764s680njcrz9VqTzn/Pg484yyH4BlzqDSeOQmKOtufAkZlcr21wCxnuQXOePDAFOB5Z7kNsMVZ/hho6ixf5cRXOd1xA4A3050/bR1YAbR2lu8BZjvLXwJhznIjPMOfZIxzNPATcKWzXgQo7CyH4XnjFjxvp36S7rjBwLPOciFgI1DF7e+zfdz/BNwQEyag/KWqdc+tiEgBYJyItABS8Qy9ex1wKN0xG4A5TtmPVHWLiLTEM2HJOmd4jYLA91lcczF4xoQXkeIicjXQDLjb2f6ViJQSkeLAOmCiiCwEPlTVOPF+Wov38CSANXiGOJnm1FKa8L9hQMDzCzszK1X1L2e5APCmiNTFkzxrZHFMe+AmEfmbs14CT+LY7W3QJjBZIjD+pC9QBqivqkniGVW0cPoCzi/wFkAXYJ6ITASOAZ+r6r1eXCNjp1mWnWiq+rKIfIpn3Jd1ItIB7yfAWYknqV0D1Ae+AooCx9Mnv2z8kW75UeB34GY8zb1ZxSDAI6q62ssYTZCwPgLjT0oAh50k0Bq4YN5l8czF/LuqzgJm45nybz3QVESqO2WKikhWfzXf45RphmdUx0TgGzxJ6FwH7BFVPSEi1VT1F1Udj6cmkrE9/ySepqkLqOop55g38DTfpKjqCWC3iPR0riUicrOXX5eD6hl/vx+eJrHMrr8aGOLUlhCRGiJS1IvzmwBnNQLjTxYCH4vIL3jat3/NpEwr4F8ikgScAu5X1XjnCZ7FInKuqeVZPGP1Z3RaRDbjaW75u7NtNJ7mpq14Rns8N4T5P52ElApsxzPrW/opA9cAI0RkC/BSJtd6D1jmxHxOX+AtEXnWiWEJnnl6szMN+EBE7gc+43+1ha1Aioj8DMzDk3QqA5vE0/YUD3TP4dwmCNjjo8Y4RGQtnsct/XnOAmMumjUNGWNMkLMagTHGBDmrERhjTJCzRGCMMUHOEoExxgQ5SwTGGBPkLBEYY0yQ+/+KaGN8RPPE9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#For quick testing\n",
    "y_pred = np.array([0.21, 0.32, 0.63, 0.35, 0.92, 0.79, 0.82, 0.99, 0.04])\n",
    "y_truth= np.array([0,    1,    0,    0,    1,    1,    0,    1,   0  ])\n",
    "\n",
    "\n",
    "Results = UncertaintyAnalysis.get_report(y_pred , y_truth)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8839fd9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#precision,Precision_CI,recall,Recall_CI, auc_delong ,auc_delong_cov ,ci_delong , fpr_keras, tpr_keras ,auc_keras ,cmtx = Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d5395162",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done!\n"
     ]
    }
   ],
   "source": [
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "413bffc6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}