notebooks/cross_validation_ex_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af54391b",
   "metadata": {},
   "source": [
    "# 📝 Exercise M7.01\n",
    "\n",
    "This notebook aims at building baseline classifiers, which we'll use to\n",
    "compare our predictive model. Besides, we will check the differences with\n",
    "the baselines that we saw in regression.\n",
    "\n",
    "We will use the adult census dataset, using only the numerical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8732f225",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:26:15.026597Z",
     "start_time": "2021-06-24T06:26:14.838907Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census-numeric-all.csv\")\n",
    "data, target = adult_census.drop(columns=\"class\"), adult_census[\"class\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df2dff6",
   "metadata": {},
   "source": [
    "First, define a `ShuffleSplit` cross-validation strategy taking half of the\n",
    "sample as a testing at each round."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "225351c8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:26:42.008393Z",
     "start_time": "2021-06-24T06:26:41.796720Z"
    }
   },
   "outputs": [],
   "source": [
    "# Write your code here.\n",
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "cv = ShuffleSplit(n_splits=30, test_size=0.5, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fa17f7c",
   "metadata": {},
   "source": [
    "Next, create a machine learning pipeline composed of a transformer to\n",
    "standardize the data followed by a logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0d2737ca",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:30:30.583390Z",
     "start_time": "2021-06-24T06:30:30.578247Z"
    }
   },
   "outputs": [],
   "source": [
    "# Write your code here.\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "pipeline = make_pipeline(StandardScaler(), LogisticRegression())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b96f29c",
   "metadata": {},
   "source": [
    "Get the test score by using the model, the data, and the cross-validation\n",
    "strategy that you defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "abc8706d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:35:24.325869Z",
     "start_time": "2021-06-24T06:35:20.142048Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score 0.81\n"
     ]
    }
   ],
   "source": [
    "# Write your code here.\n",
    "from sklearn.model_selection import cross_validate\n",
    "import pandas as pd\n",
    "\n",
    "result_classifier = cross_validate(pipeline, data, target, cv=cv)\n",
    "errors_classifier = pd.Series(result_classifier['test_score'], name='classifier error')\n",
    "print(f'Test score {errors_classifier.mean():0.02f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2de13e93",
   "metadata": {},
   "source": [
    "Using the `sklearn.model_selection.permutation_test_score` function,\n",
    "check the chance level of the previous model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bb3f65e7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:37:02.393840Z",
     "start_time": "2021-06-24T06:36:44.531957Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Permutation score 0.76\n"
     ]
    }
   ],
   "source": [
    "# Write your code here.\n",
    "from sklearn.model_selection import permutation_test_score\n",
    "\n",
    "score, permutation_score, pvalue = permutation_test_score(\n",
    "    pipeline, data, target, cv=cv, \n",
    "    n_jobs=-1, n_permutations=30)\n",
    "errors_permutation = pd.Series(permutation_score, name=\"Permuted error\")\n",
    "\n",
    "print(f'Permutation score {errors_permutation.mean():0.02f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9f62606",
   "metadata": {},
   "source": [
    "Finally, compute the test score of a dummy classifier which would predict\n",
    "the most frequent class from the training set. You can look at the\n",
    "`sklearn.dummy.DummyClassifier` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6e353095",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:38:09.893403Z",
     "start_time": "2021-06-24T06:38:09.443953Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dummy score 0.76\n"
     ]
    }
   ],
   "source": [
    "# Write your code here.\n",
    "from sklearn.dummy import DummyClassifier\n",
    "\n",
    "dummy = DummyClassifier()\n",
    "result_dummy = cross_validate(dummy, data, target,\n",
    "                              cv=cv, n_jobs=-1)\n",
    "errors_dummy = pd.Series(result_dummy[\"test_score\"], name=\"Dummy error\")\n",
    "print(f'Dummy score {errors_dummy.mean():0.02f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ccd91e4",
   "metadata": {},
   "source": [
    "Now that we collected the results from the baselines and the model, plot\n",
    "the distributions of the different test scores."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2733a06",
   "metadata": {},
   "source": [
    "We concatenate the different test score in the same pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "acbd4bd4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:38:28.673018Z",
     "start_time": "2021-06-24T06:38:28.659867Z"
    }
   },
   "outputs": [],
   "source": [
    "# Write your code here.\n",
    "final_errors = pd.concat([errors_classifier, errors_dummy, errors_permutation],\n",
    "                         axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68410efe",
   "metadata": {},
   "source": [
    "Next, plot the distributions of the test scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8225e015",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:38:50.744215Z",
     "start_time": "2021-06-24T06:38:50.340407Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnAAAAEtCAYAAACI610XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABNuElEQVR4nO3deZxN9f8H8Nd79hkzZl+MbTDMEMKgiJI9+/IVRSp9K9osbYoKFRKFvqUsRX1TQiNEUiIU4dv8SpgwY1+G2Zl95vP745yrO3fuzNy5s9y54/V8PM7jms/5nM95n3Mu8/Y55/M5opQCEREREdkPB1sHQERERERlwwSOiIiIyM4wgSMiIiKyM0zgiIiIiOwMEzgiIiIiO8MEjoiIiMjOMIGzUyLykIgoEelmy33aIg5b7rc8RCRARD4VkQt67DutbKebvv1DFRth9WeP152IqDIwgbMxo1/GhiVfRJJF5LCIrBKRviIiFbzPGSIypCLbrAz6uZkhIj62jqWCLAAwEsCHAB4A8GZxFUUkTD/2NlUUW4mq8jtTA687EVGFYwJXfXwB7Zf6QwCmAfgRQDcAWwF8b+aX2WcA3AH8bMW+XgMwxIrtyrNPa3SDFqtPNYilIvQCsE0pNUsp9V+l1PYS6oZBO/Y2VRGYBaz9zlijG2rWdSciqnBOtg6AbvifUuq/xgUiMgXAPABToCV49xjWKaXyAeRXRWAi4qWUSq/KfZamOsVSBiEAkmwdhD2z0+tehOHvVDHr3AHkKqXyyrkPZwCOSqms8rRDRNUTe+CqMaVUvlLqWQB7APQVkS6GdcU8j+am33qKFZEMEUkRkT9F5G19fZiIGN6d9qDxrVujNpSIrBSRHiKyR0SuAdhU3D6NOOn7Pi0i2SLyh4iMMq1kaN9MeaG29Tqv6avjjWKdUVIs+nNm74vIWRHJ0T/fFxH/YvbXXUSeE5GTetx/i8iDZo7PLBGpJSJzjLa/pD/n1tCozgz9HAsKn/eHimnzIQA/6T9+YlR/p5m6D4vIX/q+T4vIC8W02V5EokXkql43VkSmiUiJ/4mz5Duj1+spIt/r37ks/fqPN9NeZxHZqp+nLBE5LyJbROR2ff1KlPG6l/VaioijiLyiny9DrCMN10lEwko6J2U9pyKyU0ROiUhjEVknIkkA0gzHq+8zUEQ+FpHLAK4DqGd0/j8Tkcv6Pk6KyGwR8TDZhyH2W0TkHRE5ByALgOG89heRXXqsmSJyRkS+FpFmlhwrEVU/7IGzDysAdAHQH1oyV5z3AYwD8CmAdwE4AmgKoLu+/gq027SfAdgNYGkx7bQHMBzAMgCrLIzxLQC1ACwBoAA8DOALEXFTSq20sA1jHwGoDWAogMkArurlfxS3gYh4A/gFQDiAjwH8D0BbABMAdBeRjmZ6PWZDuyX3EYBsve5KETmhlNpbUoD6L+ptAO4AsA7aM25N9TZ6i0h7pdQ5AF8DOIGi5/2XYpr+WY/rZb3ubr38skm98QCCoX0/UgCMAfCWiJxTSq02irMfgGg9hgXQegE7AZgF7RbtiBIOs9TvjIg8Bu25vn3Qnuu7Du128RIRaaKUel6vFwFgO4BLABbpxxMC7fzdqm9f5utuxNJr+R9o5+4nAPMBBAL4AEC8BfswHHNZz6kngF0A9kJ7RCLIZL3hvLwO7e/RNf0/Ab8B8Ib29+pvaLeXXwJwh4j0MNNL9zmATD0mBeCiiNwFYCOAPwHMgfZdCQXQE9rflb8tPW4iqkaUUlxsuED7B1kBeK6EOu30OuuNyh7Sy7oZlSUB2GLBPhWAlSWsUwB6mllnbp+GstMAvI3KvfWyJADupe27mLZn6GVhFtZ/Uy97wqTuk3r562a2/x2Ai1F5XWi//L+w4Dw+qrcxz6S8v17+maXnvYTvxUMlrLsAwMeo3ANawvWrUZkbtMTgZwBOJu1MNj2HZf3OAKgDradntZl1i6Dd7myi//yM3k7HUvZV1utu8bUEcIte9zsADkblrfRYze7XJIYynVMAO/WyN8y0tVJf918z6z7X1/UzKX9bL3/EzDnbaSamd/R1QZZ897hw4WIfC2+h2oc0/bN2KfVSAdwiIi3Lub//U0r9UMZtliilUg0/6H/+EIAvtISjKgyFlsCY9hJ9BK0nZ6iZbT5QSuUYflBKnYfWI9HUwv0VQOvVuEEp9S2AGACDRaQy/459opRKMdpvBrReLOPYe0HrpfsEgI9ot5gDRCQAwBa9Tu9yxPAvAK4AVhi3rbe/CdpjGj30uobvx2ARcSvHPotjybUcoH8uUkoVGNX9E1pvqiWsPafzS2iz0Dr9ezMIwO9KqS0mdedA+96Z+z4vVEV75QznfXhpt8yJyH4wgbMPhsQtrcRawCRoCdOf+rMyy0XEmiTCmlsqR82UHdE/G1vRnjUaAYg1/QWm/xxbTBxxZsoSAfibKTe3vwtKqWQz6/4C4AUgwIJ2rGVJ7M31z4+hJbfGyzF9XXA5YjC0/4OZ9g2jbA3tf6nXexlAkojsEJEXxeh5wXKy5Hw00j9jzdQ1V2aONef0inGybYbp37lAaLdd/zKtqJRKAnAR5r/P5v7u/gda7+QH0M77FhF5RkQCS4iHiKo5/m/MPrTWP0v8BaOU+kZ/ALsfgLugPePyCIDdItLTuHeiFBlWxKjMlJVl/jpbfReLG9FoSewVOj+fFSwZjWmI8XlovYLmXChHDIb2x0JLKsyJAwClVDaAXiLSEUAfAHdCe2Zshojcr5SKLkccgGXXsiKumTXntMS/U3rvqbl9lFWR/SilEkWkA4Cu0HoP74T2jOxMEemnlPrVyn0RkQ0xgbMPj+if35ZWUf/f+X8B/FdEBMBcAC8AGAxgbaVFCLSA9qC0MUNPhXHPSBIAPzPbm+tNMJcUliQOQISIOBn3wum3jZrBfA9NeZyENjrYx0zvSgtoPaZXi2xlmbIee3GO65/XrbgtXpb2r1ravlLqN2gP50NE6kPrHXoD2qAAoOKO3RzDQIUIFP0+RFjYRmWfUwBIAJAO7Zm9QkTEF9qzhzGWNqa06Vd26gtEpDWAQwCmQ3tmk4jsDG+hVmP6dAfzoY1A3aJKGBWp1/UxLlNKGR7sBgonTddgPokqjwn6KFBDPN7QRvqlQBt9Z/A3gE7G0yDov5AeNtPmNf3T0lg3QLv19G+T8kf18vL28JjbnwOAqcaFInIPtNGvG42fsyqjsh57cbZBSwamikiRtkTEXUS8LIzHXCxfQRsoMFO0+ctM2/cWEVf9z+ZuJ5+DduvR9PuJYvZXXpv0z4nGjxaISCtovYKWqKhzWiz9e7MJQFsR6Wuyeiq0751F3+dizvsxaKNVK+McE1EVYA9c9dFORMbof/aC1hswBEBDAN8DuL+U7b2gTRmwEVrSlgDteZ8JAJLxzy8uQHvQvaeIvAjgDLRc78tyxn8VwH4R+Rja7Z+HATQA8G+T20P/gdZDuENEPoM22/6j0Eashpi0uU//fEtEPoc22vGwUupwMTHMgzZ9w/si0g7aeWgLrQczVl9fkVYCeBDAi/qt65+hTcvwBLQpMl4uR9tHoPXAPCEiGdAS4QSl1I6yNKKUui4iY6Elm7H69TkB7bxHAhgG7WH4naU0ZfY7o5Q6JyITACwHcFS/pqehJcytoH2HWwA4BWC6iPQGsBlaT5gAGKjHMc9kX4Dl191iSqm/RGQpgMcA/CAi0XqsT0L7vkShlB7ACjynpXkZ2i3PDSLygb6PO6G9ju1nWD7FzzIRqQft35HT0KZaGQnt34xPyxkjEdmKrYfB3uwL/pkSwrDkQxs19he0f6D7FrPdQzCargCAC7TRab9Be3A7G9ovzY8BNDXZtim0f8zTDPs1WlfSFCOF9mlS1hPATGi/3LMBHAZwfzHtPA/tF0k2tMEP48y1rdd9Adqtrlx9/YziYtHLDXN6ndO3OQdtfryA0o7FaN1OAKcsvH619PMeByAHWuL8GYCGZupaPI2IXr8ftLnssvRtd5p8Zx4ys81K4+tpVN4SWuJ8Xo/zMrR56F4B4GdBLMV+Z/T1d0DrEUrQ278AbZ61ZwG4GcW9Rv9eZkK7nb4fWo+pWHvdy3otoc2P+JrRd/UPAPdCGwlq8XQblp7Tkr5PxV0vo/WN9O+T4bzGQZvvzsOk3gwUP/XKMGiPN5zTj/cKtF7x4ZZ+F7lw4VL9FlGqMh83ISKyDyKyCdqk17WV9swYEVG1xWfgiOimUsyzeq2hvWt4B5M3IrIH7IEjopuKaO9oHQttVPcVaM+tPQbtP7R3KKV+L2FzIqJqgQkcEd1U9HnoXof2zlI/aINF9gCYqZQ6ZMPQiIgsxgSOiIiIyM5wGpFSiEgetFsrpb3GioiI/lEbQIFSir9niCoBe+BKISIFAMTb27vUukREpElNTQW0KVI4WI6oEvB/RqVL8/b29k5JSbF1HEREdsPHxwepqam8c0FUSfg/IyIiIiI7wwSOiIiIyM4wgSMiIiKyM0zgiIiIiOwMEzgiIiIiO8MEjoiIiMjOMIEjIiIisjOcB87OTZo0CTExMWbXtWnTBgsXLqzSeIiIiKjyMYGzczExMdj92264NXArVJ51JstGEREREVFlYwJXA7g1cEPjlxoXKoubE2ejaIiIiKiy8Rk4IiIiIjvDBI6IiIjIzjCBIyIiIrIzTOCIiIiI7AwTOCIiIiI7wwSOiIiIyM4wgSMiIiKyM0zgiIiIiOwMEzgiIiIiO8MEjoiIiMjOMIEjIiIisjNM4IiIiIjsDBM4IiIiIjvDBI6IiIjIzjCBIyIiIrIzTOCIiIiI7AwTOCIiIiI7wwSOiIiIyM4wgSMiIiKyM0zgiIiIiOwMEzgiIiIiO2PzBE5EuonI9yKSIiIZInJERB4zqdNLRPaJSKaIJIjIRyLiY6YtTxFZLCIX9boHRWRQlR0MERERURWwaQInIg8C+AHASQCjAAwE8D4AF6M63QBsAXBWX/8cgEEAvhUR0/ijAYwGMB1AfwBHAESLSL/KPA4iIiKiquRkqx2LSH0ASwC8rJSaZ7TqR5Oq8wAcBjBSKVWgb3sRwPcARgBYo5f1A9ATwDClVLRe9hOAxgAWQEsCiYiIiOyeLXvgHtE/3yuugojUBdABwGeG5A0AlFLbAZwHMNyo+lAAqQC+MaqnAKwCECkiLSoudCIiIiLbsWUCdyeAowCGiUisiOSLyDkRmSsihluoLfXPw2a2/9NovaHuEeNET/eHSVuF6M/eFbsA8Lbm4IiIiIgqiy0TuFAATaH1wC0G0APAxwCeBfCJXsdf/0wys32S0XpD3eLqwaQuERERkd2y2TNw0JJHLwD3KaW+1Mt2iog7gOdE5DWjuqqYNkzLi6tX7DqllE9JQbIXjoiIiKobW/bAJeqf20zKt+qf7YzqmOs980PhHrfEEuoB5nvniIiIiOyOLRO4P4spF/2zAMBf+p/NPb/WCoWfjfsLQHMzU4u00j/NPUdHREREZHdsmcB9rX+aztHWD9rtzgNKqXMADgIYbZyYiUgPAHWN2gC0OeB8oM0VZ2wsgFil1JGKC52IiIjIdmz2DJxS6jsR2QrgfREJgNaD1h3ARAAfKqVO61VfhDbn2xcishTa4Ie3AOwHsNaoyS0AfgKwQkT8AcQDeBBAFwCDq+CQiIiIiKqELQcxANpEvDMBvAAgEMAZaG9RuDGxr1Jqh4gM0Ot9CyAdwAYALyil8o3qKREZAmC2vvhAexPDMKXUpio4FiIiIqIqYdMETil1HdqrsZ4rpd53AL6zoL00AE/pCxEREVGNZPOX2RMRERFR2TCBIyIiIrIzTOCIiIiI7AwTOCIiIiI7wwSOiIiIyM7YehoRIiIiqx06dMgJwDgAEdDer01k7zIAJADYFhUVdai4SkzgiIjILunJ22xnZ+f73dzcnJ2cnHhXiexefn6+ys3NRU5OzsOHDh1aD2BGVFRUlmk9JnBERGSvxjk7O99fp06dnMDAwEu2DoaoohQUFEhCQkJAQkLC6Nzc3N9Q+NWhAPgMHBER2a8INzc358DAwCRbB0JUkRwcHFRwcPAVNzc3NwB3ma1TxTERERFVFC/eNqWaSkTg5OQEAO7m1vOLT0RERGRnmMARERER2RkOYiAiohpn3Lhx9Q8fPuxh6zhatmyZ8fHHH5+1dRxU8zCBIyKiGufw4cMeB/844ukS1MhmMeQkxFdq+1OmTAl999136yilip0rrDINHz48bN++fV7nz5//01B25MgRl8cff7zh//3f/9VKT093nDx58sXu3bunDxw4sNmmTZv+HjBgQLotYq2JmMAREVGN5BLUCCH3z7XZ/i+tnmqzfVeFWbNmXUxJSblsXPbMM8/UP3r0qMeSJUviQ0JC8sLCwnJ8fX3zf/jhh2Nt27bNtFWsNRETOCIiIiqzW265Jdu07Pjx4+6tW7e+Pnr06FTj8h49elyvqP1mZmaKu7u7qqj2LGm3vPusjJg5iIGIiKgaOnTokFv//v0b+/v73+ri4tKubt26rUaOHNmwpG3mzJkTGBUVFeHr63urh4dH28jIyBZvvfVWYH5+fqF6Gzdu9OrYsWOEt7d3Gzc3t3Z169ZtNWDAgMaG9Xl5eXjhhRfqhIWFtXRzc2tXu3btNs2bN2+xaNEif0Od4cOHh9WtW7cVAGzevNlLRKLOnDnj+tNPP3mLSJSIRMXGxroY1m3evLnQq87WrFnj3aFDh4hatWq1dXd3b9upU6dmu3btKvTc4vDhw8O8vLza7Nmzx6Njx44R7u7ubceMGVPiOfjf//7nds899zT29fW91cXFpV2zZs1afPjhh37GdRYvXuwvIlHR0dG1Bw8e3Mjb27tNeHh4SwDo2LFjRGRkZIsNGzZ4tWzZsrmrq2u7adOm1QG0W8T9+/dv7OPj08bFxaVd06ZNb5k/f36AcduG412yZInf2LFjG/j7+9/q4eHRrqSYrcEeOCIiompm79697r169YoMDAzMnTZt2vmmTZtmnz592mXDhg0+JW0XFxfnet999yU2btw4x9HRUe3bt6/WjBkz6l24cMF50aJFFwDg2LFjLiNGjGjaq1evlGefffaSh4dHwenTp122bt3qbWjnlVdeCXnvvfdCXnjhhQsdOnTIuH79usNff/3llpSUZDZv6Ny58/Uffvjh2KhRo5qEhYVlz5079xwANGjQIPf48eOupvUXLlzoP2XKlLD+/fsnTZ48+VJ+fr4sWrQouG/fvhG7du061rFjxxu3W3NychxGjhzZ5NFHH02YPn36BRcXl2J7svbv3+/evXv3yEaNGmXNnTv3bGBgYN5XX33lO2HChEaZmZkOkydPvmpcf/z48WH9+vVLXrlyZVxGRoYYyi9fvuz8xBNPhE2ZMuViREREtre3d/6ZM2ec7rrrrkgRwYwZM86Ghobmrlmzxu/5559vmJCQ4Dxv3ryLxm2/9tpr9Tp37py+ZMmS+LS0NMeSrps1mMARERFVM1OmTKnv6upa8Ntvvx0NDg6+0X321FNPJZa03bJly84Z/pyfn49+/fql5+fny0cffRT87rvvXnBwcMCvv/5aKycnR1atWnXa39/fbNv79u3z7NKlS9rMmTONn3ErdFvUmJ+fX0GPHj2uu7i4KG9v77ySbpmmpaU5vPrqq/X79u2bvGnTphsjPYYMGZIWHh7e8rXXXquzdevWOEN5Tk6OTJ8+/fyECRNKfePGlClT6vn4+OT98ssvsbVr1y4AgGHDhqUlJiY6vfnmm3WfeeaZq46O/+RSd999d+onn3xSZJRwSkqK05o1a4717t37xnFMmDChbmJiovOvv/565LbbbssEgHvvvTetV69ejnqymxAQEHDjfIaHh2du2LCh0kaylPkWqojcURmBEBEREZCenu5w6NAhrwEDBiQbJ2+W2LNnj0fv3r2bBAUFtXZxcYlycXGJevvtt0PT0tIcz58/7wQAHTt2zHB2dlYDBw5svHz5ct9Tp045m7bTvn3767t27fIeP358va1bt3qmp6dX2CNXO3bsqJWenu44evToxNzcXBgWNze3gttvvz19//79XqbbjBkzJqW0djMyMmT//v1e/fr1S3Z3dy8wbrtv376piYmJTjExMW7G2wwdOtRsu76+vnnGyRsA7Nmzp3azZs0yDcmbwQMPPJCYlZXlsHPnzlrG5YMGDSo15vKwpgdut4gcA7ACwKdKqSsVHBMREdFN6+rVq475+fmoV69eTlm2i42NdenVq1dEkyZNsmbNmnWuSZMm2a6urmrdunU+7733Xp2MjAwHQBt8sHHjxr/feuutkKeffjosKyvLITw8PGvixImXDL1ws2fPvuju7l7w1Vdf+S9dujTY2dlZdenSJXX+/Pnn27Ztm1We47t48aIzAIwZMyZ8zJgxRdaLSKGfPT098728vApKa/fy5ctO+fn5snz58uDly5cHm6uTkJBQKO8p7hwHBgbmmpalpqY6NmzYsMjAjbp16+YCwJUrVwq1HRoaWqSNimRNAjcVwEMA3gYwW0Q2AVgOYJtSqsJHhRAREd1MAgMD8xwdHXHu3DmXsmz31Vdf+WRlZTlER0efbNq06Y3EZP369T6mdfv27Xutb9++J/Ly8vDzzz/XmjdvXvDTTz8dVr9+/ZzBgwenOzs7Y9asWZdnzZp1+erVq47R0dG1Z8yYUa9///5Nz50796eDg/UdckFBQXkAMHfu3DOdOnWqsNGpAQEB+Q4ODvjXv/519cknnzTbudSqVatCyadpslhSuY+PT96VK1eK9FaeP3/eGdCumyVtV5QyJ3BKqXkA5um3Uh8BMALAUADnReQTAJ8opU5VaJREREQ3CU9PT9W+ffv0zZs3+165cuV8YGCgRbdRDQmDq6vrjd6qjIwMWbdunX9x2zg5OaF79+7X69Wrdy4iIsI3JibGffDgwYUm2w0ICMh/9NFHk48dO+b2zjvvhCYmJjpaGpM5PXv2vObp6Zl/7NgxtxdffLHC7uJ5eXkVdOjQIf2vv/7y6NSpU4azc5Fcq1y6du2avmTJkpD9+/e7G99G/fzzz/3d3NwKunXrVmHJqCWsHsSglNoLYK+IPA3gPmjJ3CsAponIDmi9ctFKqUrtQiQiIjInJyHeppPp5iTEAyEtrNp2wYIFZ3v16hXZvn375pMmTbrUtGnT7HPnzjlv2LDB57vvvoszt02/fv3SZs6cqe69997Gzz333KW0tDTHRYsWBTs5ORW6OzZv3rzAXbt2efXt2zc1LCws59q1aw4fffRRoLOzs+rdu3c6AHTv3j28RYsWmR06dLgeFBSUFxsb6/rZZ58Ftm7d+np5kjcA8Pb2LnjjjTfOTpkyJSw5Odlp6NChyQEBAXkXL150PnjwYC1XV9cCw4jZslq0aNHZnj17Rtx+++0R48aNuxIWFpaTnJzsdPToUbeDBw/W2rZt20lr43755Zcvr1271n/QoEFNX3rppfP16tXLXbNmjd/333/vM3Xq1PPGAxiqQrlHoSqlrgNYLiLfAngLwBgAPfXliojMB/COUqpKD4yIiG5eLVu2zLB1DAhpYXUcd9xxR+auXbuOTp8+ve7MmTPrZWRkOAQGBuZ27do1rbht2rVrl/XJJ5/EzZo1K3TMmDHh/v7+uaNHj75ap06d3ClTptyYOy0qKipj+/bttWfPnh2alJTk7OHhkd+8efOM9evXH+/UqVMmoPU2bdy40ffzzz8PuH79umNgYGBu7969U9566y2rEitTEydOTAwLC8uZP39+yNNPPx2WnZ3t4O/vn9uqVauM8ePHW90rd9ttt2X++uuvR1955ZU6M2fOrJecnOzk7e2d36RJk8xhw4YllyfmunXr5u3evfvYs88+W2/GjBn1MzIyHMLCwrIWLFhwesqUKVdLb6FiSXkeWxMRBwADoPW+3QMtIdwDYCmAbABPAegCYIlS6qlyR2sDIpLi7e3tnZKSYutQzOrWrRsOXDqAxi81LlQeNycOHUI6YOfOnbYJjIhuaj4+PkhNTU1VSvlU1j4OHTq01NfXd2jjxo1PV9Y+iGwpLi6uYXJycnRUVNRjpuus6oETkWYAxgEYCyAYQBKA9wAsU0odM6q6VkQ+gHaL1S4TOCIiIqLqpswJnIj8DOAOAAJgF4BnAaxXShU33Hk3gPFWR0hEREREhVjTAxcJ4B0AS5VSxy2o/wOAu63YDxERERGZYU0CV7csI0v1iX53WbEfIiIiIjLDmpn46onIwOJWishAEQmzPiQiIiIiKok1PXBvAqgPYFMx658FcBbAA9YGRURERETFs6YHrguAbSWs/x5AV+vCISIiIqLSWJPABQG4VML6BGhTixARERFRJbAmgUsB0KSE9eEA0ktYT0RERETlYM0zcLsBPCoii5RShXriRCQEwL8B/FwRwREREVlj3Lhx9Q8fPuxh6zhatmyZ8fHHH5+1dRxU81g7iGEggN9FZAGAGAAKQFtoAxg8AcyuqACJiIjK6vDhwx5//3HAs02Io81iiLlk/SvAFy9e7D9x4sQww8+urq6qdu3aeeHh4Zl9+vRJfeqppxL9/f35jvGbWJkTOKVUjIj8C8AnAOZBS94A7c0MVwGMUEodrLgQiYiIyq5NiCN2PlTLZvvvtvI6rHqTvZH//Oc/8ZGRkdm5ubly7tw5559++slr7ty5dT/44IOQr7/++vgdd9yRWSHBkt2x6l2oSqnNItIAQB8ATaElb7EAvldK8ctERERUAdq2bZvZuXPnG79X//3vfyf/9ttvV3r27BkxfPjwpnFxcX96eHioktqgf2RmZoq7u3uR81VceUW0XVmsGcQAAFBKZSqlNiil3lZKzVNKfcPkjYiIqHJ17Ngxc/LkyRcvX77svGLFCj9Ded26dVsNHz48zEz9iI4dO0YYft68ebOXiEQtXbrU9+GHH67v5+d3q4eHR9vevXs3uXz5suOFCxecBg0a1Kh27dptAgMDWz/++OP1cnP/eQFTbGysi4hEzZgxI/jZZ5+tExwc3Nrd3b1tp06dmsXGxrqkp6c7jB07toGfn9+tPj4+bUaMGBGWlpbmAAAFBQVo1KjRLV27dm1qGufFixedXF1d202aNCm0pOPPz8/HzJkzg5o1a9bC1dW1na+v761Dhw4NO3PmTKFOqbp167bq2bNnk48++sivadOmtzg7O7dbtmyZn+H4lyxZ4jd27NgG/v7+t3p4eLQzbDdv3rzApk2b3uLi4tLO19f31oEDBzaKjY11MT2nkZGRLTZs2ODVsmXL5q6uru2mTZtWp6S4K5rVCRwRERHZxvDhw1MAYPfu3Z7WtjFjxox62dnZDsuXL4+fNm3a+V27dnnff//9YX379g0PDw/P/vTTT0/ee++9iUuXLg2eO3dukOn2S5cuDTp8+LDH4sWLT8+bN+/MkSNHPIYPH95k0KBBjZ2cnNSKFSvin3766YvR0dH+kydPrgsADg4OePTRRxP27t1b+/Dhw67G7b333nsB+fn5mDhx4pWS4h4xYkTY7Nmz6/Xp0yf1yy+/PDFz5sxzv/76a+277rorwpAoGvzxxx+15syZEzp58uSL69atO96lS5frhnWvvfZavbS0NMclS5bEL1u2LA4ApkyZEvriiy82aNOmzfXVq1efePXVV8/v37/fq2vXrpHnz58vlCBevnzZ+Yknngh76KGHrqxbt+74kCFDUsp8EcrBqluoIjIKwNPQbp/6m6milFJWtU1EREQlCw8PzwG0JMLaNlq1apWxevXq0/qPab///rvH+vXr/Z9//vkL8+bNuwgAQ4YMSd+xY4f32rVr/V555ZUE4+39/Pzytm7detLBQcuZTp065fLOO++E3nvvvVcNI2+HDh2atm/fPs8NGzb4rVix4iwAPPnkk4lz5sypu3jx4sClS5eeA7RetVWrVgX27NkzpVGjRsW+b3379u21oqOj/efMmXNm6tSpNxK99u3bZ3Tp0qXF+++/7//SSy/dKE9OTnbavXv34YiIiBxD2YULF5z1c5i5YcOGeEP5lStXHD/44IOQPn36JK9du/aUofy222673qVLlxazZ88Ofv/9988bylNSUpzWrFlzrHfv3jeSwqpU5h44EXkewOfQkrd9AD41s3xWgTESERGRkYKCgnK3cc8996QY/xwZGZkJAIMGDUo1Lg8PD8+8cOFCod4yAOjZs2eqIXkDgObNm2cBQL9+/QptHxERkZWUlOR07do1AQBvb++Cf/3rX4lfffVVQEZGhgDAmjVrvC9cuODy5JNPltj7tmnTJm8HBwc8/PDDSbm5uTAsHTp0yAwKCsrdvXu3l+kxGSdvxgYNGlTo+Hfs2OGZnZ0to0ePTjIu79y5c2bTpk0z9+zZU6htX1/fPFslb4B1PXBPAtgPoAefeSMiIqp6J0+edAGAkJCQYnurSmM6DYmLi4sCgMDAwDzT8uzsbDHd3s/Pr1A9V1dXVVK7WVlZDp6envkAMGXKlIRVq1YFLV++3O+ZZ55JXLJkSVDjxo2zBg4cWOKLABISEpwLCgoQEhLSxtz6pKSkQnlNUFCQ2eQNAEJDQwudu8TEREdz5Xo7uWfOnCmUxAYGBlp97iuCNQlcCIB5TN6IiIhsY/369T4AcOedd95IeFxcXApycnKKJFrJyclOvr6+eablttSqVavsrl27pi5btiyoR48e1/bu3Vv7zTffPFPadv7+/nkODg7Ytm3bMUNiaMzb27tQ8ihS5HQUuy4gICAP+OcWq7GEhARnHx+fQuewpLargjWDGE4A8KngOIiIiMgCBw4ccFu4cGGdOnXq5IwbNy7ZUF6vXr2co0ePFnr7xB9//OEaHx9f5PZndfD0008nHD582OPRRx9t6O7uXjB+/PjE0rYZMGBAakFBAS5cuOB85513Zpgut956a7a18XTv3v26q6ur+vzzz/2My/fv3+9+/Phx965du1ar14Ra0wO3AMB0EXlPKVWhByMiMwC8BuD/lFJtTNb1AvA6gFuhvWs1GsCLSqkUk3qGN0GMgJZo/gVgllJqY0XGSkRE1VvMpXx0W2mzR5QQcykfzYqM3Syb33//3T0zM9MhLy9Pzp8/7/zjjz96ff311/7e3t7569evP2E879h9992X+OSTTzYaM2ZMgxEjRiTHx8e7LFq0KMT0Vmd1MWzYsLSGDRtm//rrr15jxoy54uvrW+qDfffcc8+14cOHJ06YMKHRb7/9ltC1a9dr7u7uBWfPnnXeuXNn7X79+qU+/PDDyaW1Y05AQED+M888c/Htt98OHTFiRNjIkSOTzp075zx79uy6gYGBuS+//PJla9qtLNYkcPkAEgAcFZGPAcTrZYUopT4tS6MicguAFwEUOUEi0g3AFgAbAEwHEArgLQAtRaSrUsr4okcDaAfgBT22hwBEi8hApdSWssRERET2qWXLlhkAyv0mhPJoFvRPHNZ66qmnGgGAs7Oz8vb2zmvatGnm1KlTzz/99NNX/fz8CiU848ePT7p48aLzypUrA9etWxcQHh6euXjx4jOvv/56ifOq2YqDgwMGDx6ctHjx4joTJ05MKH0LzVdffXVqwYIF11atWhW4YsWKIBFBUFBQbqdOndKjoqLKdb7nzZt3MTAwMHfp0qXB33zzjZ+Hh0dB165dU995553zoaGh1SoRFqXKNmmwiFgy9EUppSx+AZ2IOAD4BcABAK0A+Bj3wInIbwCcAUQZkjW9R+57AKOUUmv0sn4AvgUwTCkVrZcJgN0A/JVSzS2NyWjfKd7e3t4pKSll3bRKdOvWDQcuHUDjlxoXKo+bE4cOIR2wc+dO2wRGRDc1Hx8fpKampiqlfCprH4cOHVrq6+s7tHHjxqdLr03VUURERAsfH5+8/fv3/23rWKqjuLi4hsnJydFRUVGPma6zpgfu7gqIydRkAPUA9AZQ6FaniNQF0AHAs8Y9bUqp7SJyHsBwAGv04qEAUgF8Y1RPicgqAEtFpIVS6kglxE9EREQWSE5OdoiJiXFbt26d799//+2+bt2647aOyR5Z8zL7XRUZgIg0BjALwGilVJqZUR0t9c/DZjb/02i9oe4Rk1uqAPCH8fryRUxERETW2rt3b62BAwc28/HxyXv++ecvDB8+PM3WMdmjcr0tQURcAQQAuKKUKnaulRK2FwDLAGxTSm0opprhTQ9JZtYlQXvezbiuuW7YJKP1pjGklBKmdynriYiIyEIDBgxIV0odsnUc9s6qd6GKSDsR2QFtNOgZAF308iAR+VFEelrY1KMA2kN7LVdpintYz7S8pIf6yvbAHxEREVE1ZM2rtNpAGxTQBNprs25QSiUAcAfwoAXtBACYB2AOgOsi4iMiPtB6BR31n90AGOaFMffOVT8U7plLLKEeYKYXTynlU9IC7Zk6IiIiomrDmh64WQAuALgFwFQApg+t/QigowXt1IN2e3IOgGSj5Q5oz6olA5gBbR43oPCzbgatUPjZuL8ANNdHtZrWA8w/R0dERERkV6xJ4LoCWKaUugbztyTPQJunrTQnoI1oNV3+D8BJ/c9LlVLnABwEMNo4MRORHgDqAvjaqM1oaJP3DjTZ11gAsRyBSkRERDWBNYMY3FDybcXaljSiJ4A7TcsNgwqUUsbrXoQ259sXIrIU/0zkux/AWqN6WwD8BGCFiPhDm8j3QWjP6A22JC4iIiKi6s6aHriTAKJKWN8dFTxVh1JqB4ABAMKgTdT7jv55j1Iq36ieAjAEwJfQXqe1FUBraBP7bqrImIiIiIhsxZoeuNUAXhGRrwD8rpcpABCRZwH0BTDR2oCUUt2KKf8OwHcWbJ8G4Cl9ISKim9C4cePqHz582KP0mpWrZcuWGR9//PFZW8dBNY81Cdx8AL0AbANwDFry9q6IBAIIAbAdwAcVFiEREVEZHT582OPQ4UOebg3cbBZD1pksq7ddvHix/8SJE8MMP7u6uqrQ0NDsvn37prz++uuX/P39i7yDvLpbunSp76VLl5xfffVVi997aonY2FiXyMjIVosWLTr1zDPPJJa+Rc1gzZsYcvT3kD4NYDSALADNAByHdmtzkZk3IRAREVUptwZuRd4TXZXi5sSVu43//Oc/8ZGRkdnXrl1z+Pbbb70//PDDkD179nj9/vvvxxwdLX7leLXw1Vdf+R09etSjohO4m5VVb2JQSuUBeFdfiIiIqBK0bds2s3PnzpkAMHjw4PSrV686RUdH+//444+1evfufd3advPz85GXlyeurq6c4N4CmZmZ4u7ubvZclbTOErm5uRARODmVLSWz6k0MREREVPU6dux4HQDi4uJcASAjI0MmTZoU2rBhw5YuLi7tAgMDW48dO7ZBcnJyod/vIhI1bty4+q+//npQgwYNWrq6ukZt377dc/Hixf4iErVx40avoUOHhtWuXbuNl5dXm5EjRzbMyMiQI0eOuNx9993htWrVahsaGtrqlVdeCTZu17B9bGysi3H55s2bvUQkavPmzV563BE//vijz4ULF1xEJEpEourWrWuYoxVXrlxxHDduXP3Q0NBWzs7O7UJDQ1tNnDgxNDs7u9Bcs/Hx8c733HNP41q1arX18vJq07dv38Znz551tvT8nTx50nnEiBFhgYGBrZ2dnduFhYW1nDVrVpC52JcsWeI3duzYBv7+/rd6eHi0A4Dhw4eHeXl5tdmzZ49Hx44dI9zd3duOGTOmIQBcuHDBadSoUQ0DAgJudXZ2btewYcOWU6dODcnLy7vRdmxsrIuIRL366qvBkyZNCg0JCWnt6uoadfLkyULnzxJl7oETkbGW1FNKfVp6LSIiIrKUIXELDg7Oy8vLQ48ePZoePnzY45lnnrnUsWPH60eOHHGbO3du3aNHj7rv27cv1vg26+bNm32Dg4NzZ86ced7Lyys/MjIyKy4uzgUAnnjiibDBgwcnffrppyf37t3rOX/+/FClFP73v/95PvDAA1cmTpx4efXq1X5vvPFGvYiIiOwxY8aklCXuJUuWnB4/fnzDU6dOuX755ZcnAcDNza0AAFJTUx06d+4cmZqa6jh58uSLLVq0yNq3b1+td999N/T06dOuGzZsiAeA9PR0h+7duzdLTEx0nj59+rlmzZplb9y40Xvs2LEW3Sc/deqUc6dOnZp7eHgUTJ8+/XzDhg1ztm3bVnvWrFn1ExMTnRYtWnTBuP5rr71Wr3PnzulLliyJT0tLu3Eic3JyHEaOHNnk0UcfTZg+ffoFFxcXlZ6e7nDnnXdGXL582Xnq1KkXmjdvnrV169ba8+bNqxsfH++6Zs2a08Ztf/jhh8GRkZGZCxYsOA0AAQEBZX6m0ZpbqCuhDVwwfQODafchEzgiIqJyyM/Pl9zcXKSlpTl+8803tf/73/8GBgUF5fbp0yd9+fLlfvv27fP68ssvT4wcOTIV0G6z1q9fP/eBBx5osnbtWu9Ro0bdmLc1JyfHYdeuXbF+fn5FnlO/5557kt9///3zADBkyJD0n3/+2Wvt2rUB77//fvwTTzyRBAD9+/dP/+GHH3y++OILv7ImcFFRUVne3t55Li4uLj169Ch063fOnDlB8fHxbnv37j1y22233bhd7OHhUfDqq6/WP3jw4MX27dtnffDBB/6nTp1yMz7e4cOHp917770Oa9euDSgthpdeeik0MzPT4cCBA0cbNmyYazjWnJwchw8//DDk5ZdfvhwcHHwjkQoPD880JI/GcnJyZPr06ecnTJhw4/Wcb731VuDJkyfdVq9efeK+++5LBYBhw4alFRQUyMcffxw0derUy23btr0xqqVWrVr5O3bsOF7W26bGrLmFeje0ud6M357QC8AEAH9Cm1zX0pfZExERUTHuvPPO5i4uLlEBAQFtHnnkkcZNmzbN3Lx5898eHh5q69at3j4+PnnDhg1Lzc3NhWEZPHhwmqOjI3bu3Oll3Fbnzp3TzCVvADBw4MBCE/Q3bdo0CwCGDx9+o9zZ2RkNGzbMPnfuXJlv95Vk+/bt3pGRkRnt2rXLNDmOVAD44YcfvADg559/9vLy8so3JG8GDzzwQJH3nJuzY8cO786dO6eHhobmGu9nwIABqTk5ObJz505P4/qDBg1KKa4t0wR2165dXp6envmG5M1g3LhxV/VjLHQt+vTpk1qe5A2wbhTqrmJW/SgiqwD8BqAdtDciEBERkZWWLFkS36JFiywnJycVFhaWGxoaeuOBqitXrjilpKQ4ubi4mJ1cPzExsdDv+ODg4Nzi9hMQEJBn/LOLi4vStyl0a8/Z2VkZ306sCFevXnU+c+aMa3HHcfXqVScASE5OdgoICChyDPXq1Sv2uIwlJiY6ff/99z7F7efKlSuFzldoaKjZdj09PfO9vLwKJcIpKSlmY6tfv36uvu9C56xOnTo5lsRckvKlfyaUUtki8l8ATwBYUJFtExER3Wxat259YxSqKX9//zx/f/+8r7/++ri59cHBwYWSMhHTJ5/Kz/AcW1ZWVqHGTZOhkvj6+uZ5eHgUfPTRR6fMrW/QoEGuoZ65yZnPnTtn0SAGX1/fvFtuuSVj1qxZF8ytb9asWbbxz2U5X76+vnl//vlnkdgMAyxM5+2riGtRoQmcLhvaS+aJiIiokvTp0yft22+/9XNwcECXLl0ybBFDkyZNcgDg0KFDHrfeeuuNBGjjxo3epnVdXV1VVlZWkUe3evbsmbpkyZKQ0NDQ3PDw8GJ70+666670LVu2+K5Zs8bb+DbqZ5995mdJrN26dUvbs2ePV8uWLbOKu5VsrW7duqVt2bLF94svvvA2vo26cuVKfxFBnz590ipyf0AFJ3AiUgfAeGgvkSciIqJK8vjjjyeuXr3af9CgQU0ff/zxyx06dMgQEZw+fdp5+/bt3pMmTbrcq1cvq+eKs8Rdd911PSwsLGv69On1cnNzRe8R9D1w4ICXad0WLVpkbtmyxXf+/PkBHTp0yHB3d1cdO3bMnDZt2uWNGzf63nXXXZHjx4+/3Lp168zs7GyJj4933bZtm/eHH354plmzZjkTJkxI/OCDD4Ife+yxRvHx8ecNo1D37NlT25JY582bd/72229vfvvtt0c+/vjjCREREdnp6ekOx48fd926davP7t27/7b2ubQJEyYkLlu2LOixxx5rFBcXd6FFixZZ3333Xe0VK1YEjxo16qpxcltRrJlGZEcxq/wARAJwAfBgeYIiIiIqr6wzWRXyNoTy7B8tK699Z2dn7Nix4/isWbOC161b5/fOO++Euri4FNSpUyenS5cu6c2aNSv3c1alcXJywsaNG09MmDChwXPPPdfA2dlZDRo0KGnBggVnRo0aFW5c98UXX0z4448/PGbOnFnv2rVrjqGhoTnnz5//09fXt2D//v3Hpk2bVmf58uVBFy9edHF3dy+oV69e9t13351muBXs5eVV8NNPP8VOmDCh/uuvv15PRNClS5fUTz/9NK5Xr16RpcXaqFGj3AMHDhx5+eWXQxcsWFDnypUrzp6envlhYWHZPXv2THVwsH5qXE9PT7Vr166/J0+eXPfdd9+tk5qa6hgaGpozderU86+//volqxsugShVtsmDReQUik4ZogAkAfgbwH+UUr9USHTVgIikeHt7e6ekpNg6FLO6deuGA5cOFHldTNycOHQI6YCdO3faJjAiuqn5+PggNTU1VSnlU1n7OHTo0FJfX9+hjRs3Pm26ji+zp5ogLi6uYXJycnRUVNRjpuusGYUaViFRERERVRImTVTT8VVaRERERHaGCRwRERGRnbFmEEMBij4DVxqllKqMKUuIiIiIbjrWJFWfQnvTQksAsQCOQnsvaiSACGiv0/pfRQVIRERERIVZk8B9DmA4gCFKqY3GK0RkCIDPADynlNpe/vCIiIiKlZ6Xl1ehE7ISVRdKKeTl5QGA2TdxWPMM3OsAPjJN3vSdbQCwFMAbVrRLRERUFrFZWVm5V65csWgmfiJ7UVBQIJcvXw7MysrKAmD2HfTW9MC1BrCqhPUnoL2NgYiIqDJ9nJubG37x4sX7k5OT6zs5OXFgHtm9/Px8lZOTg9zc3NT8/Px1ALaYq2dNApcMoDeAJcWs7wsgtZh1REREFSIqKirv0KFDL+fm5p7Izc2NAFDk9U1EdigDQAKA76KiooodU2BNArcawLMisgLAfGhvXwCAZgCeBzAAwDtWtEtERFQmUVFRedAe3SG6qViTwE0HEA7gYQAPATA8QOoAbTTqJr0OEREREVUCa16llQ1gqIj0BjAEQCNoidtJAN8opb6v0AiJiIiIqBCrJ9fVEzUma0RERERVrFwjdkQkXETuEBHvigqIiIiIiEpmVQInIgNE5CS0NzH8DCBKLw8SkRMi8q8KjJGIiIiIjJQ5gRORbgCiASQBmAnt+TcAgFIqAdqzcKMqJjwiIiIiMmVND9yrAP4PwG0A3jez/ldo70olIiIiokpgTQLXHsDnSqni3j93DkCI9SERERERUUmsSeAcAWSXsD4AQI514RARERFRaaxJ4I4C6FrC+gHQbrESERERUSWwJoFbAeBfIvKI0fZKRDxEZDGATuBrTYiIiIgqjTVvYlgiIncAWAZgAQAF4AsA/tBur36ilPq8QqMkIiIiohusehODUmqMiKwHMAZAJLSpRPYD+FQptb4C4yMiIiIiE2VK4ETEHcAIALFKqWho88ERERERURUq6zNw2dBunbathFiIiIiIyAJlSuD0ud/OAqhdOeEQERERUWmsGYW6CsADIuJa0cEQERERUemsGcTwC4BhAGJE5AMAxwFkmFZSSv1cztiIiG56kyZNQkxMjNl1bdq0wcKFC6s0HiKqHqxJ4LYb/XkRtGlEjIle5mhtUEREpImJicHP+w7CJahRofKchHgbRURE1YFFCZyIdARwQimVBODhyg2JiIiMuQQ1Qsj9cwuVXVo91UbREFF1YGkP3K8AHgCwWim1SkQ8ob1t4Q2l1JFKi46IiIiIirB0EIOY/OwKYCSAkIoNh4iIiIhKY80oVAPTpI6IiIiIqkB5EjgiIiIisgGbJXAi0kNEVopIrIhkiMg5EflaRFqZqdtLRPaJSKaIJIjIRyLiY6aep4gsFpGLet2DIjKoSg6IiIiIqIqUZRqRfiJieObNA9pUISNEpI2Zukop9W4p7Y0H4A/gXQBHAQQDeAHAARHpppTaBwAi0g3AFgAbAEwHEArgLQAtRaSr/nYIg2gA7fR24gE8BCBaRAYqpbaU4ViJiIiIqq2yJHD364uxx4upq6AlZiV5UimVYFwgIt9DS7yeBzBcL54H4DCAkYZkTUQuAvgewAgAa/SyfgB6AhimlIrWy34C0BjAAmhJIBEREZHdszSBu7uid2yavOllKSJyHEA9ABCRugA6AHjWuKdNKbVdRM5DS/LW6MVDAaQC+MaonhKRVQCWikgLTnlCRERENYFFCZxSaldlBwIAIhIIoCWAL/SilvrnYTPV/zRab6h7xOSWKgD8Yby+gkIlIiIishlrXqVVKUREoE0O7ABgvl7sr38mmdkkCdrzbjCq+3cx9YzbMt1vSimheZeynoiIiKhKVZsEDsDbAIYAeFgpddRknen7VosrL65eaeuIiIiI7Ea1SOBE5E0AzwKYqJRaabQqUf8013vmh8I9c4kl1APM9+JBKeVTSmwpYC8cERERVSM2n8hXRGYBeBnAC0qpxSar/9I/W6KoVij8bNxfAJqLiOkxGeaVM/ccHREREZHdsWkPnIi8BuAVAK8opd42Xa+UOiciBwGMFpGFRtOI9ABQF8DXRtWjATwCYCCMRqICGAsgtqaOQD1x4gTgWcI6IiLdpEmTEBMTY3ZdmzZtsHDhwiqNh4isZ7METkSeBTADwGYAP4jI7Uars5VSv+t/fhHanG9fiMhS/DOR734Aa4222QLgJwArRMQf2nxyDwLoAmBwJR6KTV27dq3YBO7atWtVGwwRVWsxMTH4ed9BuAQ1KlSekxBvo4iIyFq27IEbqH8O0BdjpwGEAYBSaoeIDAAwE8C3ANKhvZXhBaVUvmEDfc63IQBm64sPtGlDhimlNlXWQRAR2ROXoEYIuX9uobJLq6faKBoispbNEjilVLcy1P0OwHcW1EsD8JS+EBEREdVINh/EQERERERlwwSOiIiIyM4wgSMiIiKyM0zgiIiIiOwMEzgiIiIiO8MEjoiIiMjOMIEjIiIisjNM4IiIiIjsDBM4IiIiIjvDBI6IiIjIzjCBIyIiIrIztnyZPZkxadIkxMTEFCk/ceIEACA8PLxQ+bVr1+AK16oIjYiIiKoJJnDVTExMDGL27UKbEMdC5Zcu5MPTBYDTpULl+fn5VRgdERERVQdM4KqhNiGO2PlQrUJlPnPTzJY7zUqrytCIiIioGuAzcERERER2hgkcERERkZ1hAkdERERkZ5jAEREREdkZJnBEREREdoajUG2kuPneYmJi0ManaP3MPCDmUj66rbxeqDxfVU58RFS95SVfREzMWXTr1q3IuuLmjYyJiQFq16+C6IiosjGBs5Hi5nu7lpYP+DgWqZ9fAKQ7OOCAm8mkvQ4ZlRkmEVVTBbmZSMsV7ItLLLIu+8JFiIs7rjoUXpedlg7X2lUVIRFVJiZwNlTcfG/FcWvghsYvNS5UdmTCkUqJjYiqP5egRgi5f26R8jML7zW77szCe6sqNCKqZHwGjoiIiMjOMIEjIiIisjNM4IiIiIjsDBM4IiIiIjvDBI6IiIjIznAUag127do1s3NEtWnTBgsXLqzyeIio+DkgAf7dJCLLMYGryQrygVN7ChXFXMq3UTBEBGhzQP687yBcghoVKs9JiLdRRERkj5jA1WCeLigyz5zpmxyIqOqZm6Pt0uqpNoqGiOwRn4EjIiIisjNM4IiIiIjsDBM4IiIiIjvDBI6IiIjIzjCBIyIiIrIzHIVaiUqa7ykmJgZtfKo0HADAiaQCXEuJKTI/3IkTJwAA4eHhRbbh3FRElS8v+SJiYs4W+bsZExMD1K5vk5iIqPpiAleJYmJiELNvF9qEOBZZdy0tH/ApWl7ZruUoAKlF5oe7dCEfni4AnC4VKue8cURVoyA3E2m5gn1xiYXKs9PS4VrbRkERUbXFBK6StQlxLDIXGwD4zE2zQTQaczH5zE0zW85544iqjrn54c4svNdG0RBRdcZn4IiIiIjsDBM4IiIiIjvDBI6IiIjIzjCBIyIiIrIzTOCIiIiI7AxHoRIR3eSKm4MO4DyQRNUVEzgioptccXPQ5STE2ygiIioNEzgiIjI7B92l1VNtFA0RlYbPwBERERHZmRqXwImIp4gsFpGLIpIpIgdFZJCt4yIiIiKqKDUugQMQDWA0gOkA+gM4AiBaRPrZNCoiIiKiClKjnoHTk7SeAIYppaL1sp8ANAawAMAWG4ZHREREVCFqWg/cUACpAL4xFCilFIBVACJFpIWtAiMiIiKqKKLlNzWDiPwKLWfrbFJ+G4B9AEYqpb4yWZdSSrPeAODt7V3meK5du4b8/Pwyb1ci05S7oJRyIqJycHR0hKenZ5m3S01NBbR/j2taRwFRtVCjbqEC8Afwt5nyJKP11lCpqalpVm5bsbTEzJBNppqUU9Upeg3IFngdKll+fr4hGStOcdegNvgvE1GlqWkJHACU1KVYZJ1SyqfyQqkchl5De4y9puA1qB54HWyP14DINmpa13YizPey+emfSWbWEREREdmVmpbA/QWguYiYHlcr/fNwFcdDREREVOFqWgIXDcAHwECT8rEAYpVSR6o8IiIiIqIKVtOegdsC4CcAK0TEH0A8gAcBdAEw2JaBEREREVWUGpXAKaWUiAwBMFtffKC9iWGYUmqTDUMjIiIiqjA1ah64mwVHfdker0H1wOtge7wGRLZR056BIyIiIqrx2ANHREREZGfYA0dERERkZ5jAEREREdkZJnBEREREdoYJnI2IiKeILBaRiyKSKSIHRWSQBdudEhFVzHLMTP3GIvKZiFwSkWwROS0iH1TOUdmXqrgGIhIiIu+LSJy+j3gR+VBEQivvyOyLtddB33a4iPwiIsn68quI3FtM3WdE5G/978FJEXnBzFtbbkqVfQ1EpJmIvCMiv4tIqogkishuS/dBREXVqHng7Ew0gHYAXoA24fBDAKJFZKBSaksJ2w0F4GpS1grAUgAbjAtFpDWAnQAOAngKwBUADQC0LXf0NUOlXgMRcQGwC9q7eF8FcBRAcwCzAPQQkZZKqewKORL7ZtV1EJEHAawEsB7A63rxIwDWiIinUupjo7rTAcwE8CaAHQA663/2AzC1go/HHlX2NegN4B4AnwE4AMAZwAMAvhGRyUqphRV9QEQ1nlKKSxUvAPoBUACGGpUJgD0AjlrR3mK9vWYm7f0BYDP00cZcqvwadNPLHjGp+4he3s3W58HWS3muA7T/nJwC4GBU5qCX7TQq8weQCWCRyfZvAsgFUM/W5+EmuAYB5v4dgvbmnKu2PgdcuNjjwtsHtjEUQCqAbwwFSikFYBWASBFpYWlDei/P/QD2KKX+Nlp1F7Reobf1tqmwqrgGufpnqskmhp/Z+1a+65AL4JpSqsBo2wIA11D43PYF4Ka3aWwltLsQN/ttvEq/Bkqpq8X8O3QAgL+IuJfvEIhuPkzgbKMlgCPG/+jp/jBab6kh0HoYPjYpv1P/dBCRPSKSoz+f8gWfvwJQNddgH4DfAMwQkfb6c0btAcwA8DOA/WUNugYqz3X4D4DmIjJNRAJEJFBEpgGIAPCuyT4UgL+MN1ZKHYfWM1eWa10TVcU1KEJEBMDdAOKUUplWxk5002ICZxv+AJLMlCcZrbfUOGj/2/3KpNyQpH0N4BcAfaA939ITwC4R8SjDPmqiSr8GSql8AD0AHIfW05Cuf54F0N/ML8ybkdXXQSn1DbTes+egPd+ZAOAlACOUUt+Z7CNDmX/eMLmkfdwkquIamDMRQHsAb5Q1YCJiAmdLJd3WtOiWp4jUA9ALwBql1HWT1YZru0Yp9YJS6iel1DJoDyeHQ7vld7Or1GsgIs4AVgO4FVqSdyeAxwG0hvbwtrM1QddAVl0HEekF7fyuwT8PyUcD+EJE+lfEPm4iVXENjLcbAmA+gJVKqU+sCZjoZsdRqLaRCPP/q/XTP839b9ich6Alaqa37gz7AIBtJuXfA8iHNuLsZlYV12AcgIEA2iil/k8v2y0isdAe/r4PwKcW7qemsuo66LffVgHYoZQab7TqOz2pfg/At0b7qCUirmZ64XyL28dNpCqugfF2/aElfF8D+Hc54ia6qbEHzjb+gvbciOn5b6V/Hi6tAf0fz4cAHFNK/WKmyp+lNHGz376rimvQFkCuUfJmcFD/tHigRA1m7XUIBlAH/5xLYwcBNBIRN6N9CIBbjCuJSDgA9xL2cbOoimsAABCRe6AlblsBjNYfMyAiKzCBs41oAD7QemeMjQUQq5Q6YkEbdwFoAvM9P4D2D2QmtCkCjPUF4Ag+QF8V1+ACAGcRMZ13r5P+ed6yUGs0a69DMoAsAB3NrLsdQKJSKkv/eSu0EZEPmNR7EEAegE1lD7tGqYprABHpo+/rBwD3KqVyzWxHRJay9TwmN+MCrTdgB4Cr0G6z3Q1tSoMCAAON6u2EPqLfTBufQhvCH1zCfl6Edrt0AbTBCxP0fcYAcLH1eajp1wDapMkp0ObE+re+jycBXNIXf1ufB1sv5bkO0EY5KgDLof3HxHBrTgGYZlL3NWjJ2gxoifdUADnQptmx+Xmo6dcAQBcAGQBO6uf/dpPF1dbngQsXe1tsHsDNugCoDW0I/iVo/4v9H4AhJnXMJg8AvABcB/CNBfsZD+AItB6Iy9DeFuBn6+OvDktVXAMAzQB8ridxmQDiAKwA0MDWx19dFmuvA7Se5McBHIKWKCdBm7plDEwmjdWTlEnQRgRnQ3vbwEswmoD2Zl4q+xpAS5xVCUuYrc8BFy72tohSHIBFREREZE/4DBwRERGRnWECR0RERGRnmMARERER2RkmcERERER2hgkcERERkZ1hAkdERERkZ5jAEREREdkZJnBEZSAiviKSJSJKRMbYOh4iIro5MYEjKpvRAFygzeT/iI1jISKimxTfxEBUBiLyO7TXBX0DYCGApkqpkzYNqgQiIgBqKaWu2ToWIiKqOOyBI7KQiLQD0AbAKmjvN80F8LCZei4i8oKIxIhIhoikishBEXnKpF5tEXlTRI7qt2UTRWSPiIwyqrNTRE6Z2UeYfht3hlFZN73sIRF5UkSOQHuv5XP6+o4islJE/tbjSheRvSIytJjjDRGRxSISJyLZIpIgIttFpJe+fqOIXBeR2ma27ajH8krpZ5aIiMrKydYBENmRR6C9wH69Uuq6iHwL4EEReVUpVQBoyRuAbQC6AfgewH+hJVGtAAyD9sJwiIgPgD0AbgGwDsASaC8GbwtgAIAvyxHnJAD+AJZBezn5Wb18KIBIAF8BOK3XeRDA1yIyWim12tCAiIQB2AsgGMCnAA4CqAXgdgA9AWwHsBTAQAD3AfjIJIZxAAoArCzHcRARUTGYwBFZQETcoCUq65RS1/XiVdCSoj4Atuplk6Alb3OUUi+btGHc4z0bWvL2uFJqaQn1rNEAQKRSKsGk/A2l1Esm+1oM4HcA0wGsNlr1AYBQAH2VUtuKiW8rtOTwERglcCLiAe1cbVNKnQUREVU43kIlsswwAL7QkjaDbwEkQOttMhgNIBnALNMGjHrpHACMAnAUWi+Z2Xrl8KmZ5A1GiSdExENE/AF4ANgBoLnhVqiI+AHoC+A70+TNOD6lVD6AjwF0EJFWRlX+BaA2gBXlPA4iIioGEzgiyzwC4AqAcyISLiLhAMKg3UocJCIBer2mAI4ppbJKaCsAWjIYoypnFNHf5gpFJEhElorIZWi3gq9CO6bxehUf/TMcgEDrmSvNCgD5KDwi9xFoie3GMkdOREQW4S1UolKISCMAd0NLaswmRwDGQBuVCgClJWViYb2S6pT0dzejyA610ajfA2gOYDGAAwBSoSVfDwO4H//8h87i+JRSZ0XkOwBjROQFaLdv7wQwXymVW9r2RERkHSZwRKV7GFpS8yiAFDPr34DW67QQWoLXXERclVLZxbR3Bdpt1jYW7DsJQJSZ8sYWbGusNYBbAcxSSr1mvEJE/m1S9zi05K2thW0vBdAfwBCjbXj7lIioEvEWKlEJ9OfVHgLwp1JquVJqnekC4AsALUWkA7TpRXyhDQowbUuAG8+QfQGghYgUmQzYUE/3NwAvEeloEtPkMh5KvmFzk321hDYQ4walVBK0AQr3iEjPUuIDtGcBzwN4HNqo1r1KqWNljI+IiMqAPXBEJesNoD5K7lFaD2AGtF64Z6BNrTFdT+i+hzaNyC0AIqBNwQFoCV53AMtFpDe0KUUEWg+WE4AH9HpLATwLIFpEFgHIgTZIoKx/d48C+AvAC/oo0VgAzaAlXYcBtDOp/xSAXwBsFZFVAA4BcAdwG4BTAF40VFRK5YvIJ/gnaX0ZRERUqdgDR1QyQw/Z18VVUEodhtZTNgraXG69oSUz9aFNFzIbQEfjNpRSyQA6AXgb2i3SBQBegzaAYJNRvXhotyavAHgdwAvQ5md7sCwHoY8Y7a+3/SCARQDu0v+8yUz9eADtoSWuvfX6L0LrXfzezC6WQ5v3LR3A2rLERkREZcdXaRFRuYlIHWhzwq1QSj1u63iIiGo69sARUUWYAK33cWlpFYmIqPz4DBwRWU1/b2sDAM9De/PCIRuHRER0U+AtVCKymogoaIM0dgN4WCl13sYhERHdFJjAEREREdkZPgNHREREZGeYwBERERHZGSZwRERERHaGCRwRERGRnWECR0RERGRnmMARERER2Zn/B18/8GkaXyCJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Write your code here.\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "final_errors.plot.hist(bins=50, density=True, edgecolor=\"black\")\n",
    "plt.legend(bbox_to_anchor=(1.05, 0.8), loc=\"upper left\")\n",
    "plt.xlabel(\"Accuracy\")\n",
    "_ = plt.title(\"Distribution of the testing errors\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed00fd3e",
   "metadata": {},
   "source": [
    "Change the strategy of the dummy classifier to `stratified`, compute the\n",
    "results and plot the distribution together with the other results. Explain\n",
    "why the results get worse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "53188dff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-24T06:39:37.914947Z",
     "start_time": "2021-06-24T06:39:37.494286Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dummy score 0.64\n"
     ]
    }
   ],
   "source": [
    "# Write your code here.\n",
    "from sklearn.dummy import DummyClassifier\n",
    "\n",
    "dummy = DummyClassifier(strategy='stratified')\n",
    "result_dummy = cross_validate(dummy, data, target,\n",
    "                              cv=cv, n_jobs=-1)\n",
    "errors_dummy = pd.Series(result_dummy[\"test_score\"], name=\"Dummy error\")\n",
    "print(f'Dummy score {errors_dummy.mean():0.02f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17c5cbc2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}