notebooks/cross_validation_learning_curve.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ae010d4",
   "metadata": {},
   "source": [
    "# Effect of the sample size in cross-validation\n",
    "\n",
    "In the previous notebook, we presented the general cross-validation framework\n",
    "and how to assess if a predictive model is underfiting, overfitting, or\n",
    "generalizing. Besides these aspects, it is also important to understand how\n",
    "the different errors are influenced by the number of samples available.\n",
    "\n",
    "In this notebook, we will show this aspect by looking a the variability of\n",
    "the different errors.\n",
    "\n",
    "Let's first load the data and create the same model as in the previous\n",
    "notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "88fe8c7b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-27T07:40:14.759517Z",
     "start_time": "2021-05-27T07:40:14.342069Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "housing = fetch_california_housing(as_frame=True)\n",
    "data, target = housing.data, housing.target\n",
    "target *= 100  # rescale the target in k$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9db44e3",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">If you want a deeper overview regarding this dataset, you can refer to the\n",
    "Appendix - Datasets description section at the end of this MOOC.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d3dd9303",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-27T07:40:17.722283Z",
     "start_time": "2021-05-27T07:40:17.678285Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "regressor = DecisionTreeRegressor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3faff19f",
   "metadata": {},
   "source": [
    "## Learning curve\n",
    "\n",
    "To understand the impact of the number of samples available for training on\n",
    "the statistical performance of a predictive model, it is possible to\n",
    "synthetically reduce the number of samples used to train the predictive model\n",
    "and check the training and testing errors.\n",
    "\n",
    "Therefore, we can vary the number of samples in the training set and repeat\n",
    "the experiment. The training and testing scores can be plotted similarly to\n",
    "the validation curve, but instead of varying a hyperparameter, we vary the\n",
    "number of training samples. This curve is called the **learning curve**.\n",
    "\n",
    "It gives information regarding the benefit of adding new training samples\n",
    "to improve a model's statistical performance.\n",
    "\n",
    "Let's compute the learning curve for a decision tree and vary the\n",
    "proportion of the training set from 10% to 100%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "163bd34b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-27T07:40:41.298709Z",
     "start_time": "2021-05-27T07:40:41.292201Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.1  , 0.325, 0.55 , 0.775, 1.   ])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "train_sizes = np.linspace(0.1, 1.0, num=5, endpoint=True)\n",
    "train_sizes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12f17942",
   "metadata": {},
   "source": [
    "We will use a `ShuffleSplit` cross-validation to assess our predictive model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "61211d52",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-27T07:40:44.756868Z",
     "start_time": "2021-05-27T07:40:44.751920Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "cv = ShuffleSplit(n_splits=30, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73f949b5",
   "metadata": {},
   "source": [
    "Now, we are all set to carry out the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f3d37927",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-27T07:40:58.801395Z",
     "start_time": "2021-05-27T07:40:52.690724Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "results = learning_curve(\n",
    "    regressor, data, target, train_sizes=train_sizes, cv=cv,\n",
    "    scoring=\"neg_mean_absolute_error\", n_jobs=2)\n",
    "train_size, train_scores, test_scores = results[:3]\n",
    "# Convert the scores into errors\n",
    "train_errors, test_errors = -train_scores, -test_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcff9a0c",
   "metadata": {},
   "source": [
    "Now, we can plot the curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7f6edb14",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-27T07:41:10.659792Z",
     "start_time": "2021-05-27T07:41:10.290880Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEyCAYAAADHvMbdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABHoklEQVR4nO3deXhU1fnA8e8bIAEJSYCwBBTQiiCiUtw3DCK2ogguxY0fglpRa1t3ahEFi6JVqVr3VkVcELTiijvgAlhcqqggroiyyZawSIAk7++PcyeZTO5k5k5msr6f55lnknPvPffMneXcs4uqYowxxiRbWm0nwBhjTMNkGYwxxpiUsAzGGGNMSlgGY4wxJiUsgzHGGJMSlsEYY4xJCctgTFxEZJmIzK3tdDR0IrKLiNwlIstFpEREltVyerqJiIrI+Lp6HhGZIiI23qIOsgwmhUQk3/vSXFnbaTH1xhjgj8B0YCRwaW0mxsTH+66PF5Gc2k5LXdK0thNg6o0egN0lpt5A4DNVvaq2E1LDfgBaAMUJHPt74MLkJiewfOB6YApQUJsJqUusBNPIiEgzEWke9DhV3a6qO1KRprpIRFrV0qk7AhuSHWktvp64qFOkqoEzGFXdqapFqUhXqiT6PaxvLIOpI0Sku4g8JiKrRGSH1+Zxq4i0jNivp4jcKyJfiMhmEflFRD4Skd/7xDneq6LbR0Qmi8hPQBFwqIiM9LYdIyJXisi3IrJdRL4SkXN84qrUBhMK89L0speeQhF5RkQ6+sSxn4i8LiJbRWS9iDwqIrleOqbEeZ3SReRqEfnEe+2FIvKhiFwStk/UOvnIc4XX/YvI6d613Ab8U0Ru8bbt5xNPtohsE5HnIsKP9V5jgYgUicgiEYl5dx16P4DdgaO981ZokxCRoSIyT0S2eI95IjLEJ67Q+/JrEXlNRAqBRXGk4Ugvzm0iskZE7gYyo+wrInKRd71+8d77OSLSP8r+p3rbC7z9l4pra0r3tvu2wYjICBFZ6B23VUS+E5EnRKRd2D6+77f3eZvpfdaKRGSx99lpErHfFO/c2SJyn4j87O0/T0QOieO6TcGVXgC+j3zvpIrvobc9Q0T+Ku47XeS91hdF5NfVve61zarI6gAROQCYjStaPwCsAPYH/gQcISJHq+pOb/d8oB/wEvA90BL4HfCgiOSq6iSfUzwBbANux1VzrQK6edtuwlVNPABsBy4CpojIN6o6L47kdwbmAjOBq7x0jwaygOPCXmN34F3cTc1d3mscBLwSxzlCcaQDr+GuwevA47gv6r7AKcDd8cblYyjuet8H3A9sAj4DrgZGAJHtaMOA5sCjYem7wDv2feBGYCuuyus+EflVjGqvd4D/A/4BrPOOBy9jEJGLgXuAL4GJuPdxJPCciIxW1Qcj4uuC+0w9DfyHKBlFWNoPAd4ENgO34D6LZwBToxzyGHAm8AzwCJABnA28ISKnqOoLYXHfCPwVWOy9vlXAr4BTgesA35KxiAzHXd93vf22ea/reKA9sLaK13Mg8DawE3fdVgODvde2v5fWSK95cd4AtAUuB2aJSDdV3RztXLjvThZwMnAZ7v2Dypl6pe+hiDQDXgUOx13Tu4FsXLXfPBHpp6ofhsUR93WvE1TVHil64H4IFbgyxn6f4n44WkWEn+wdPzIsrKXP8Wm4H/lCoFlY+Hjv+LlA04hjRnrb/gekh4V3xmU00yL2XwbM9QlTYFhE+D1eeM+wsBle2BER+073wqfEcT2v9va9ye8ahP09xX20feOocC5cRqu4H6K9ffb/AFgJNIkIfxf3Q5Lu/Z+Hy+ye9InjTqAE+FUcr9HvOrcGtgDfAFlh4VnAt7hMIcfnfTk/wGd1Pu6Hfq+wsHRgoRfXeJ/P5QURcTQFPsTd+IgXdrC372ygecT+ErZfN5/zPIvL6JvGSHul9xuYh2vP2S/ifKHP4YDI44F7I+L4nRc+Oo7rN97bt1sV2+ZGvhZchqTAbyLCs4Dl4Z+FINe9rjysiqyWici+wH7Ak0CGuCqjXBHJBd7D3QWXlQRUdWvYsc1FpC3QBndHnwX09DnNHRq9bvteDWtbUdUVwFdA9zhfwkpVnRERNtt73tNLZxNcaWWhVi4V3R7necDdqW3E3WFWoKqlAeLx87KqLvEJfxSXeQwMBYjI7sARuEw4dO1Ow91NPhT+Hnrv44u4m4ABCaZtIK6kepeqbgoFen//E1c6OTbimA24O9yYRKQ9cBjwvKp+FRb/DlyJI9JwXKb2XMTrzMG91m6Uf35CJYVrNKKdRD1VJK0Q2AU4QUQkntcS9noOB15Q1bJShHeum7x/T/Y5NPK1hj7H8X4XYvH7Hg7H3Vx+FHEt04E3gCNFpEXYvvFe9zrBqshq397e8wTv4adD6A8RycTdEQ0DdvPZt7VP2Fc+YSHf+YStB7pWcUw8x4OrZgBoh/uBXOqzr19YNN2BTyJ/qJIk2jWaBkzGVZO96oWNwN0NPxq2X+h9fLOKc3SoYltVdveev/DZ9rn3vEdE+LeqWhJn/KFjv/TZttgnbG+gFbCmijg7UH6jorhSelA34aqDnwPWi8jbuCrV6Vp1lVVV12sxUErl6wURn2VVXe/la2199k2E32dsb1wVddTqPiAX+JFg171OsAym9oXuzG6n/Acs0sawv58ETgQexNXbb8BVBQzCFbf9SqW/VHH+aD9C8d4xVvUjJhHPfoJ2fY5n/2gN/FV93n2vkfcj8zIwVERaeT9sw4ElWrFuPPQaR+DaGPz4ZcbxiPvuPUxV73m0+P2um9+5BfeDeFYVcX4etm9C3dtV9WsR6YUr+Q0Ajgb+BUzw2ia+jXJoIteLKjLkhOLz4feeCK6t7/Iqjlsbtm+8171OsAym9n3tPZeoalV3v4gbxHUi8JiqXhixLbKKpC75GVfV18Nnm1+VXjRfAXuLSIaqbq9ivw0AItJGVcO7/PrdtcbjUVwngN+JyFJc1d9fIvYJvY/rYr2PCQj9kO4DvBWxrZf3nGjmFR7/3j7b/MK+BvYC3lfVLTHiXgr8FlcNvDBowrz3eZb3QEQGAS/jfpD/EOWw0LXYx2dbT9xNWHWul59Ex4h9jSvhz46jmjfIda8TrA2m9v0Pd9dxoYhU+gEUkaYi0sb7N3SHJRH75AHnpzSV1eDdGb4CHCwiR0RsviJAVE/gqgCvjdwQUUcfqiKIzHSDnCvcy7gG/RHeoxTXgy3cDFzniAlhdebh6csWkYwEz/8GLoP+o4SNZ/H+/iOuA8AbCcaNqv6M6/k2RET2Cos/HVcqjjQV99vh12MREQmvCnzSe77J7/VX1bbitS9E+th7buOzDSh7PfOBwSLSO+Jc13j/zox2fIJCP/hR0xXFVNzYJ98STMS1DHLd6wQrwdSMAeI/qGqdqt4vIv+Ha1BcJCIP4+qOd8HdKZ+C+1JMUdXNIvI6MFzcWI0PcG0lo3E9SJJVV5wK1wK/AV4VN77iJ+AE3N0bxHcHeCeuq+m1InIQrmNDEe5OtQflGco0XP39gyLSE9cmdDyuLjswVd0pItOAS4ADgDe9zhDh+/wkIhcB/waWiMhjuNHp7XDdqIfiShvLEjh/gYhcjeud918pH8czEvcZGa2qhcFfWQWX43o5zROReyjvplzpN0JVnxGRR4BLRKQvrsv8OmBXXGeBPfFKi6q6UERuwU2B85GITMd1Gd4d1zHiYKKPfH9d3Bied3BtEDmU9358LMbr+TOum/K73utZjSv9/wbX0y+yJFhd73vPt4jIE7jP5eeqGqvK6k5cJ45bReQY3O/AJlx37AFePP0h2HWvM2q7G1tDflDeTTna48uwfbvixlAsw3UXXQ98hLtb2S1sv1zcj9hK3IfvM1yf+ZFenPlh+44netfJSvuHbZsLLIsIW4Z/N+W5VbzukRHhfXCN4L/gqrGm4n5oKnURreKaNgfG4jLhItyP0wfAxRH7HYLrqlqE+xI+iPuBitZNeXyM8x4Q9r6dXcV+R+Dujn/23seVwBxc6al5HK/P95p6207G3Zlv9R7zgaFB4ohx7n5enEVe+u8Beke7PrhxO+/ifhCLvPM+C5zus++Z3vux2Uv7l8AdlHfzrvQ+eJ/rN3CZww5c29YsoH9E3FPw6ZaOG+/ynPdZ2w4swXV1j+xy7nu8ty2uLvTevlfjqt52hr8Wqvgeetub4sZgfRD23n6NK7EfV53rXtuPUB90Y2qFN8j0Q1w31ptrOz3GmOSxNhhTYyLbJrw68au9fxNuQzDG1E3WBmNq0iciMhtXrdcS155yFG5cw0e1mjJjTNJZFZmpMSLyd1ymshvu5uZ7XD3zLVo+15oxpoGwDMYYY0xKJFRFJiKtcbOZKrBWVTfGOKTOEZFiXBvUplj7GmOMKZMFlKpqzPwjrhKMiKQBQ3Czix6NGxgUbjWua+vTuAnm4p54UETycVN5H4yb4G0ZblK4B8P2GQj8DdftcDOuG+gYVS2I9zw+5y0FJDs7O9EojDGm0SksLATXrTtmJ7EqMxhvFtyLcAP98nD9sz/ATS2xHjeivA1ugM+BuIbbVbhBbvdrjMn2xC1s9RBufqHncf3He3rputvbJx/Xw+g53LoLnXBrOnwHHBUkM4s4d0F2dnZ2QUFBIocbY0yjlJOTQ2FhYaGq5sTaN1YG8yWuQfYp3MjZd6L9oHulnP64iQCHActV1W8eo9D+u+HmKRqvqn+vYr+FQDPggNC5vRLN68AZqjq9ylcYPV7LYIwxJqAgGUysIs4rwB6qep6qzq2qtKCqpar6lqqOwq1WF21m4JDzvOd/RttBRDoDB+Emdyw7t6q+gVsR8dQY5zDGGFNLqsxgVPUyVa1q7YFox61WVb9J8sL1w03dcIq49blLROQnEbnZm2QP3DQV4D8F9Wdh2ysRt6511AduWVJjjDEpUpsj+TvhFiP6J26N9gHAw7g5m0Ir8YUmb9xQ6WgXVpcndzTGmEat2iP5vek+egBFqroswKFpuNXZzlTVp7ywud50IleKyPVh+0ZrKIragBSrftBKMcYYk1pxl2BEZKCIRC5y1QHXq+wL4FsReUtE4v3RDi2r+1pE+Cvec18qL70brg3+JRtjjDF1QJAqsptw07CHuxXXDvI3XNfhI/FZDCqKz6KEhxYgKqV8TW2/tpZ9qWPLgxpjjCkXJIPZF7euBVC22t2pwGRVHa+qf8W1pQyNM75nvedBEeGDcFVfH6jqT7ip3M/2ukGHzj0A6BwWhzHGmDqmyjYYERnh/dkcN8q+Z1jYrkALoElYWDqwW9j/n6jqIr+4VfVVEXkFuMdbGvUL4BjcSnT3q+oP3q5jcGNeponIg5QPtPwvbuaAmvXICfDDe+7vrkfCqJdrPAnGGFMfxGrk3917buY954aFHYZbI/6XsLBMoAludToh9vKwvwMm4NYEaQcsx1WxlQ28VNXZInKit9/LuKlingOujjVTgDHGmNpTZQajqhNCf4vIn4DvQmEi8iRuzenwfQYDv1XVG+I5uapuBa70HlXt9yqxB24aY4ypQ4J0U34V1314B66kchowLmKfQ3FrSRtjjGnkgmQwVwJ743qMgctw7ghtFJGmuHnI7kpW4uokWz/HGGPiEncGo6rLRWR/3KDKHar6bcQuzYERwCfJS14dtG5pxb/f+hvkdIGc3SCnK2TvCk0zai99xhhTRwQaye9NOLkkyrYtwNteSabhKi4q/3vrWnj3tsr7ZHb0MpwukO0953R1Ydm7QfouNZdeY4ypJYEyAxG5UlV9flHLtjfFdR0+uboJq7PCM5hotqx2j58+8N++S25YqacLZHfx/vfCMlolN83GGFMLgpY2bhGRFao6LXKDNxDyKWBwUlJWF5WWutLJpp/c/03SoWQnVUyJ5u+Xde6x8mP/7S1ah5V8wh6hsBY51XkVxhhTI4JmMP8CHhGRn1X1rVCgN+Hlk7iSywVJTF/dkpYGrbuVZzC7Hgz/N9P9X7AcCn70npdDoff3phUQdNHNbRvdY7XvGFXIyIqofosoDe3SBkT8jzXGmBoSNIO5GOgIPCsi/VT1Uy9zeQy3iuXFqvpQshNZpzVNhzZ7uIefkp2waWV5hlOWCf3gwgp/gtLiYOfcvgnWfO4efpq19GkDCnu0bGcZkDEm5QI38ovIGcBbwCwROQq4DjgLuFRV709BGuu3Js2gdVf38FNaAptXh5V6fqiYERX+BCXbg51z51ZY+6V7+Gna3Mt4wjKf7LCSUGZHV1ozxphqCNzjS1WLvKlb5gOLcPORXaWqDXv8S6qkNYHszu7BYZW3l5bC1p+9DOeHsJJQWCZUvC3YOYuLYP3X7uGnSTpkda7Y/Tq8NNQqD5o07M6CxpjqizXZZZcqNo8GXgQeAJ4O31dVlycneYa0NGjV0T12O6jydlX4Zb1X8vFpAypYDju2BDtnyQ7Y+L17+JFQptjFpw1oNzcWqEkz/2ONMY2GaBUj00WklKq7SInfdlVtUv2kpZaIFGRnZ2cXFBTUdlJSS9V1GKjUBrQcCr3nosLknlPSXCknsvebDUY1pt7LycmhsLCwMNaqwRC7iuwGAvfBNXWKiOtVtksbyNvff5+iQpfxVKh+CysJ/bLe/7hotNT1ntu0ApYv8N8nfDBqWSZkg1GNaUiqLME0ZI2mBJMM27e4zgbhpZ7wktDWn5N/zsjBqDldK5aEbDCqMbUiSAnGMhjLYKpv57byDCiy/afgR9i8iqQXhCsMRu1auVu2DUY1JiWSVkUmInup6leJJEJEeqjq0th7mnqvWQvI7e4efop3VB6MGp4JpXIwaqU2IBuMakxNidXIvxM3iHKyqkYZ1VfpmF8DlwNnqGqd7UpkJZg6JNpg1FB1XCKDUWMJH4zqNyDVBqMa4yuZjfyDgduBT0VkEW7J4g+Ab4ENuF5kbYDuuMXGjgd6AYuBExN9AaaRSWgwanhvuFQORvWblNQGoxoTj5htMCLSBG8aGOAI/CvTQ7d6c4H7gP94U/vXWVaCaUCiDkYNq47b+Utyz9kk3XW3jiz5JHsw6iMnwA/vub+7HgmjXq5+nMZUQzJLMKhqCTANmCYiHYCjcaWUdrjMZi3wOfC2qq6rTsKNSUgig1EjM6Edm4Ods2QHbPjOPfyEBqNG9n6zwaimEQk6F9kaYEaK0mJMaohAy1z36HxA5e3RBqOGV8cVFQQ7p5aUV+H5pskGo5qGzyaUMqY6g1FD/6dqMOrOreX/b1oJi19w88RldYLM9m4uO2PqKBsHY20wJhl2bK3c+y28JLRlTfLPKU1cKSirE2TllWc8WZ3CMqGObkkJY5IkqW0wxpg4pLeE9j3dw0/UwajecyKDUbXEjS8KLYAXTcv2FTOdsr+9TKlVnk3NY1LCSjBWgjF1QbTBqF++7BaYS7UWrcszoFZRSkPNs1KfDlPnWQnGmPom2sqo4d2Udz0Yjr/FtcVsWgmbV5b/vWmFey4uSuz8oZkRoq2SCpCeWTnTyeoErcLCbIYEE8YyGGPqiybp0Lmve/gJ9YbbtAI2rSrPdMIzoE0rg3fJDtmxBdZ95R5R05gRkQH5lIZatrPOCY1E3BmMiGQChcB4Vf1b6pJkjElIeG+4jvtG369ok2vz8c2AvPBtGxJLQ8n2qherA0hr6jof+JWGQm1DrfKijxOywaf1RtwZjKpuEZECIAVzsxtjfKXix7N5lnu06xF9n53bwjKfKNVxW34moVmyS4vj6Jwgrht2eAYUahsKOibJ1JpAjfwi8gywQ1XPSl2SaoY18htTTcU7YMvqqqvjNq9yvd1SJa2ZK62FFqorG7Tq/W/LNiRdytaDEZHdgbeBR4DbVbUGurekhmUwxtSA0hLYurbq6rhNK4NPVhqvjKyKGU5kRpTZ3jolBJTKDOY7IBNo6wWtBSJnEVRV/VXckdYSy2CMqSNU4ZcN5ZmNX3Vc4YqKsxokS5MMNy1PtBJQVufkTFragKSym/Jykr40oTGmUROBlm3dI28//31U3XigTavgmXPh5y9ceMv2ri2p8EeXCZXuDHbuku2w4Vv38E1bmuuGXakEtJu3fMNubsG9eDTCzglBJ7vMT1E6jDEmOhFonu0eLVqXh+fuBSNfcn+XlrgpecLnjCv8Mez/H4OXgrQ0rENClDnjWrbzr36zdiAbB2OMaSDSmpR3d+aQyttD44QqZTxh/yfSPXvrWvdY+bH/9lA70OaVYcesgxUfu5mzG/Dg1ISmihGRXwFDgNCw4++A51U1Sjmz7rE2GGNMJdu3uDnjopWAEpkzLpZmLV2pp3XXiovX5XRxGVCL1nUqA0pZIz+AiPwN+AsQORS3FLhJVa8LFGEtsQzGGBNY8Q7X8SAy4yn05pAr/Cl4O1As6Zn+GU/o7xrOgFLZi+xc4N/AfOBW3EqWAPsAVwGHA+er6iNBE13TLIMxxiRdaalrB/IrAS17L/lLdwOkt/LJgLr4Z0BJ6GiQygzmI2AHcJSqFkdsawq8C6Srqs+ygXWLZTDGmBoV/uPesQ/0u7zi4nUbf3DPye6OnZFVntms/J9XzQfk9YHzXg+8cmoquynvDVwTmbkAqGqxiDwFTAoYpzHGNC4ZmdBrSOXw0Jiggh8qZj5ljx+Cl4K2b3KzZEfOlL3qE5eptdsr4ZcRS9AMZgduoGU0rbx9jDHGBBU+Jshv1uwKGVCUTChIBpSzW/LS7iNoFdkbQA/gIFVdE7GtPfAhsERVf5PUVKaAVZEZYxocVfhlvX/ms/EHWP+1G9sDbh6369YFPkUqq8j+BrwFLBGRh4DFXvg+wChcCebsgHEaY4xJBhFomesenX2awh8eBMvnub9zU1c1FhJ0JP87InIKcDdwRcTm5cA5qvpushJnjDEmicK7M4fPiJCq0yU40DINOADYHRDgW+Bj1VDZq+6zKjJjjAkuJVVk3oqWnwL/VNU7gA+8hzHGGFNJWrw7quoW3DT9W1KXHGOMMQ1F3BmM533gwFQkxBhjTMMSNIP5CzBMREaJ1KHZ14wxxtQ5QbspTwY24uYj+7uIfIv/ipYDkpE4Y4wx9VfQDGYP3FzVy73/OyQ3OcYYYxqKoONguqUoHcYYYxqYuNtgRCRTRGaLyHmpTJAxxpiGIWg35YNSmBZjjDENSNBeZJ/gpuxPCREZLyIqIp/4bBsoIu+LyDYR+VlEHhCRnFSlxRhjTPUEzWCuB34vIv2TnRAR2QcYA6zx2ZYPzAJ+BAYDVwInAS9709YYY4ypY4L2IhuO60H2poh8CnyFfzflQO00XibxEK77875ATsQuf8ctz3x6aL4zEVkFvA78Dpge7GUYY4xJtaAZzMiwv/t4j0gKBO0IcBmwK3Ac8EL4BhHpjGv7uSJ8Mk1VfUNEVgCnYhmMMcbUOUG7KSe9OkpE9gBuAM5W1U0+EwT09p4/j9wAfBa23RhjTB0StASTVN50M/8CXlPV56Ls1tZ73uCzbQPgs66om44/xumz40iiMcaYBCWUwYhIS+Aw3Ej+NyOXTw7g97jJM3vFsW+0hWuCL2hjjDEm5QJnMCJyETAJyML9uA8E1ohIO1wvrz+p6oNxxJOLa7yfBGwN63LcFGji/V8ErPfC20bGAbTBv2RDrMVwvBKOlWKMMSZFArWpiMipwD3AHOB83GqWAKjqWuBVYEic0e2K+4GfhJtAM/Q4AteushEYD3zh7e/X1rIv/m0zxhhjalnQEsxVwBxVPVlE2uK6FYf7EFftFY9vAL/xNHcAmbgMbLmq/iQiHwJni8gdYd2UBwCdgWcDvgZjjDE1IGgGsy9uMGQ0q4D28UTkTT0zNzI81DivquHbxuDGvEwTkQeBTsAtwH+Bp+M5nzHGmJoVtNtxSYxjOgFbE0+OP1WdDZwIdANexq1L8zJwvKqWJPt8xhhjqi9oCeZT4DfAXZEbvNH4vwM+qE6CVDU/SviruDYeY4wx9UDQEszdwPEi8jdcDy6ANBHpgauq2gefzMcYY0zjE3Qk/3QR2RcYC1zjBb+K600mwPWq+kpyk2iMMaY+CjwORlWvFZFngbOBnriM5WvgMVX9MMnpM6ZB2bp1K5s2baK4uJjS0tLYBxhTA9LS0mjevDm5ubn4TNeVsIRG8qvqx8DHSUuFMQ1caWkpK1euZPPmzaSlpdGsWTOaNGlS28kyBoCdO3eyZcsWtm/fTufOnZOWydTqXGTGNBaFhYVs3ryZ3Nxc2rZtS1qaLWNk6pYNGzawZs0a1q1bR7t27ZISp33KjakBW7ZsIT09ndzcXMtcTJ3Upk0bMjIyKCoqSlqc9kk3pgaUlpbStGnTpNZvG5NsTZo0SWrboGUwxhhjUsLaYIypx05/YAH//d5NKH7I7m2YPvqwWk6RMeWsBGOMMSYlEspgRGR3ETlfRMaKSDcvLF1EuohIelJTaIyps0QkrseyZcuqdZ78/Hzy8/Nr/FhTPYksOHYLcDnQBLfg2AJgGdAcWAxci5ty3xjTwC1YsKDC/2PGjOGrr75i5syZFcLz8vKqdZ577723Vo411RMogxGR0bg1Ye4CXsJNoQ+Aqm4SkReAwVgGY0yjcOihh1b4v3Xr1mRkZFQKj7R9+3YyMjLiPk+vXvGsqp78Y2tTtGsU9NoFiTvZglaRXQzMVNVLgf/5bF8E9KhuoowxDUd+fj59+vThzTff5KCDDqJ58+ZMmjQJgKeeeoqBAwfSsWNHdtllF/bZZx8mTpzI9u3bK8URXs01d+5cRIQZM2bwl7/8hby8PLKysjj22GNZunRp0o5VVW666Sa6du1K8+bN6du3L6+88krc1W6lpaX84x//YN999y2bimXEiBGsXr26wn7dunVj6NChTJs2jd69e5Oens60adPK0vrEE09wySWX0KFDB5o3b1523P3330/v3r3JyMggNzeXM888s1J1ZFXXP9WCVpHtBdxXxfa1QG7iyTGmcZnw4hcsXrkp4eMXr9pU4e/TH1hQxd6x9eqUxfWD96lWHH5++uknzjvvPMaOHUv37t1p1aoVAN9++y2DBw/m8ssvp0WLFixatIiJEyeydOlSHnvssZjxjhkzhqOOOoqHHnqIwsJCxowZw+DBg1myZEnMqXjiOXbs2LFMmjSJiy66iCFDhvDjjz8yevRoiouL2WuvvWKmb9SoUUyfPp0rrriC/Px8VqxYwbhx48jPz+ejjz6iZcuWZfsuXLiQJUuWMG7cOPLy8ujUqRNr1qwB4Oqrr+aYY45hypQpbN68GYDx48czYcIERo0axW233VYW9+GHH84nn3xC+/blaz9Gu/6pFjSDKQJaVrG9K1CQcGqMaWQWr9xU1s24ujYXFSctrmRbv349zz//PEcccUSF8LFjx5b9raoceeSRtG7dmpEjR3LnnXfSpk2byKgq2G+//Zg6dWrZ/02bNmXYsGEsXLiQww6rust2rGM3bNjA5MmTOfvssyu04/Tu3ZvDDjssZgYzf/58pk6dyj333MPFF19cFt6nTx/69u3LlClT+MMf/lAWvm7dOubPn0+3bt3KwkIZTK9evSpkuBs3buSWW27h1FNP5eGHHy4LP+CAA+jbty//+Mc/KpRSol3/VAtaRbYQONlvg4g0B/4PmFfdRBljGpZ27dr5/rh98803nHPOOXTp0oX09HSaNWvGiBEjKC0t5euvv44Z70knnVTh//322w+AH374odrHvv/++2zfvp1hw4ZV2O/QQw+tkAlEM2vWLNLS0jjjjDMoLi4ue/Tu3ZvOnTvz9ttvV9i/T58+UeM9+eSKP7sLFiygqKiI4cOHV4pj3333Zc6cORXCo13/VAtagrkVeE1EHgNC2WZHEfkNMAHYFTgriekzpkHr1SmrWscvXrWJzUXFALRq3pReedWLr7rpicavF9mmTZs46qijyMzMZPz48XTv3p0WLVqwcOFC/vCHP7Bt27aY8bZt27bC/6GG63jm04p17IYNrjTYoUOHSsf6hUVas2YNpaWllc4Tsm7dugr/V9XTLnJbKG0dO3b03ffbb7+NO+5UCrrg2JsichFwJ+UZSajctgP4vapWrxLYmEakuu0d4SP5e+Vl1dmR/H5zsM2ZM4fVq1fz9ttv069fv7LwTz/9tCaTFlUoYwhVU4Vbs2YNXbt2rfL40MSm7733Hs2aNau0PbIdpKp56iK3hdIW2VkAYNWqVZUytdqaAy/wQEtVfRDYHbgU1+D/AHAlsKeqTklm4owxDVfoRy89vXxstqry0EMP1VaSKjjkkEPIyMhgxowZFcLff//9uAaOHn/88ZSWlrJ69WoOPPDASo8ePRLvcHvYYYfRvHlzHn/88QrhixYt4rPPPuOYY45JOO5kSnTBsdXAP5OcFmNMI3L44YeTk5PDhRdeyIQJExAR7r//ftauXVvbSQPc9PWXX345kyZNIisri6FDh/Ljjz8yYcIE8vLyYi670K9fP0aOHMmIESP44x//yJFHHknz5s1ZsWIFc+bMYdCgQZx22mkJpS0nJ4exY8cybtw4zj33XIYNG8bKlSsZN24cnTp14rLLLkso3mQLVIIRkRIRidrGIiKni0hJ9ZNljGnocnNzefHFF2nevDlnnnkmo0ePpmfPntx11121nbQyN954IxMnTuTFF19k8ODB3HXXXdx33320b9+e7OzsmMc//PDD3Hbbbbz++uucdtppnHTSSUycOJGmTZuWdSpI1LXXXsv999/PwoULGTJkCFdddRX9+vVj/vz5SVswrLpEVePfWaQUGK6qT0bZfgbwhKrW+bVgRaQgOzs7u6CgoLaTYhqBUM+kWPX2QdlsyjXv+++/p2fPnlx33XUVulk3BPF8TnNycigsLCxU1ZxY8SV7uv4uwOYkx2mMMbXi448/5j//+Q+HHXYYmZmZLF26lL///e9kZWVx/vnn13by6ryYGYyIDAGGhAVdICLH+uzaBjgWeC9JaTPGxGAlltTKzMzk/fff54EHHqCwsJDs7Gzy8/O58cYb4+qq3NjFU4LpA4z0/lagn/eItAWYD1ySjIQZY0xt22uvvXjrrbdqOxn1VsxGflWdoKppqpoGCK4NJs3nkaWqx6nqN6lPtjHGmLouaBvM7rgJLY0xxpgqBR3JH3uCH2OMMYbgC47NjmM3VdUBCabHGGNMAxG0imwPXEN/ZBx5uPacdcDWJKTLGGNMPRe0iqybX7iIZACXA6OAo6ufLGNMXB45AX7wRgZ0PRJGvVy76TEmTODJLv2o6nZVnQT8F5icjDiNMcbUb0nJYMK8B/wmyXEaY+ooEYnrEc/sw7F8+eWXjB8/3jeubt26MXLkyGqfwyRXsqeK2R1Ij7mXMaZBWLCg4vJPY8aM4auvvmLmzJkVwpOx4NWXX37JhAkTyM/Pr7Ty48yZM8nKSs1iaSZxQXuRdYmyKTRNzJ+AudVMkzGmnjj00EMr/N+6dWsyMjIqhafar3/96xo9XzKUlpZSUlLiuxjZ9u3by1bYTFQy4qiuoFVky4DvfR4fAX/3tv8peckzxtR3RUVFXHfddXTv3p2MjAzy8vK45JJL2Ly54ry4Tz/9NAcffDBZWVlkZmbSvXv3snVNpkyZUrYuff/+/cuq3ubOnQtUriKbMmUKIsI777zD6NGjyc3NpW3btpxyyimsXLmywnm3b9/OFVdcQceOHdlll1048sgjWbhwYdzVbvG+PhHh0ksv5c4772TPPfckPT2defPmlaX1jTfeYPjw4bRp04aePXsCUFxczMSJE8v2z8vL44ILLmD9+vUV4u7WrRtDhw5l2rRp9O7dm/T0dKZNmxYz7akWtIrsBip3U1ZgA/AV8KaqliYjYcY0Cq/8BVZ/lvjxqxdV/PuRE6qXno77wvE3Vy+OMCUlJZxwwgl89NFHXHPNNRx44IF8+eWXjBs3js8++4w5c+aQlpbGvHnzOP3007n44ouZOHEiTZo04fvvv+fDDz8E4IQTTuCWW25hzJgx3HPPPfTt2xeAXr16VXn+UaNGMXjwYJ588kmWL1/O1VdfzfDhw5k9e3aFfWbMmMGYMWPIz89n8eLFnHbaaWzatClpry/k6aefpnPnztx8881kZmbyq1/9qqxNadSoUZx66qlMnz6dbdu2AXD++eczdepUrrjiCgYOHMiXX37Jtddey4IFC1i4cCEtWrQoi3vhwoUsWbKEcePGkZeXR6dOneJ7k1IoaDfl8SlKhzGN0+rPyrsZV9f2TcmLK0mmT5/O7Nmzefnllxk0aBAAAwYMYNddd2Xo0KG88sornHDCCSxYsIDs7GzuvvvuCseHpsRv164de+21F+AylXir4E488UTuuOOOsv83btzI1VdfzapVq8jLy+OLL75g2rRpjB07lokTJwIwcOBAOnTowJlnnpm01xdSVFTEm2++6dtedPzxx3PnnXeW/b948WIeffRRrrjiCm699VYAjjvuOLp27crQoUOZOnUqo0ePLtt/3bp1zJ8/v1L7VG1Kdi8yY4wpM2vWLNq2bctxxx1HcXFx2WPgwIE0adKEt99+G4BDDjmEgoIChg0bxgsvvMC6deuScv6TTjqpwv+hVSRDC2uFzj9s2LAK+5122mk0bRr7/jve1xcyYMCAqJ0RQlWAIaHqv+HDh1cIHzJkCNnZ2cyZM6dCeJ8+fepU5gIxSjAi4jctf0yq+k5iyTGmkem4b/WOX73IlVwAMrKgY/WW4a12eiKsWbOG9evX+zZkA2UZyVFHHcXMmTO56667+N3vfsfOnTs58MADmTBhAscff3zC52/btm2F/0ON3kVFRQBs2OBWA41c26Vp06aVjvUT7+sLqao3XeS2UNo6duxYad+OHTtWaodJRk+9ZIuVRc+lcptLVcTbv84vmWxMnVDd9o7wkfwd96tzI/lzc3Pp0KEDL730UtTtIUOHDmXo0KHs2LGDefPmccMNN3DSSSfx+eef06NHj5SkL5SJrFmzpkImU1xcXOkH3E+Q1weuoT+ayG2htK1evbpSJrN69Wr69OkTd9y1JVYGM6pGUmGMaZCOP/54pk+fTlpaWlnDfCzp6en079+ftLS0skb3Hj16lJU+Qg3gydCvn6ukmTFjRln1GcAzzzxDcXFxzOMTeX3x6t+/PwCPP/54hczkxRdfpLCwkGOOOSap50uFKjMYVX20phJijGl4zjrrLKZOncpvf/tbLrvsMg444ABEhB9//JHXXnuNP//5zxx++OFcd911rFixggEDBtC5c2fWr1/PrbfeSk5ODocd5paF7tWrFyLCv//9b7KyssjIyKBHjx60atUq4fTts88+nHnmmdx8882oKv379+eLL77g9ttvJzs7u0IPsOq8vkT07NmT8847j8mTJyMiHHfccWU91Pbff39GjBiRULw1Kdkj+Y0xpkzTpk2ZNWsWkydP5oknnmDChAlkZGTQpUsXBgwYwO677w64Rv67776bq666inXr1tG6dWsOPfRQ7r333rLqoa5duzJ58mTuuOMOjj76aEpKSpgzZw75+fnVSuMjjzxCXl4e//73v5k8eTJ9+vRhxowZ/Pa3vyU7Ozspry9RDzzwAHvssQcPP/wwd955J23atOH0009n0qRJNG/evFpx1wRRDdLEAiKSBpwDnIybvh/gO+BZYGp9GQcjIgXZ2dnZBQUFtZ0U0wiEei117do1uRHbbMopMX/+fI444ggef/xxzj777NpOTo2J53Oak5NDYWFhoarmxIov6FQxLYBZQD9cY/4qb9Mg4ARghIgMUtWiIPEaY0xtee211/jwww/p27cvGRkZfPrpp9x88810796dU045pbaTV68FrSK7Frfey23AJFXdCCAiOcA1wFXAWGBcEtNojInGSizV1qpVK2bNmsXtt9/O5s2byc3NZdCgQdx0000VRsqb4AJVkYnIN8CHqnpGlO1PAQeq6p5JSl/KWBWZqUkpqyIzJomSXUUWdCT/rlQ9W/Lb3j7GGGMauaAZTAHQvYrte3r7GGOMaeSCZjBvABeLSKVVK0XkOOAi4LVkJMyYhiQtLY3i4mKC9to0piaVlJTEHPsTRNCYrgU2A7NE5EMRedR7fAi84m27LmmpM6aByMzMZMeOHaxdu5bS0nrRk980Mhs2bGD79u1JHV8TdLr+H0TkQGASMBgIzY2wGZgG/FVVlyctdcY0ENnZ2fzyyy+sX7+ejRs30qxZM5o0sSn7TN1QUlLC9u3badWqVaX506ojcFlIVZer6tlANtARyANyVHV4kMxFRAaIyBQRWSoiv4jITyLyrIhUms5VRAaKyPsisk1EfhaRB7yu0cbUC2lpaXTu3JkuXbqQlZUVdfZdY2pDs2bNyM3NpXPnzkmdNDPhqWLUVSb/XI1zXwi0Bf4BLAE6AFcDH4hIvqq+DyAi+bjBnc/hqug6AbcAvUXkqPoyc4AxAC1btqRly5a1nQxjakTQcTAHA/ur6r/CwoYAE4E2wKOq+tc442qvqj9HhOUA3wOzVfVUL2wh0Aw4IJSZiMhA4HXgDFWdHvcLqHguGwdjjDEBpXIczPVA2RJxItIF1/bSESgExohIXFP8R2YuXlgB8DXeWBoR6QwcBDwWXlJR1TeAFcCpAdNvjDGmhgTNYPYH5oX9fwZukbE+qtoLV6q4INHEiEg7oDfwuRfU23v+3Gf3z8K2G2OMqWOCtsG0BVaH/f8b4B1VXeH9/wLwt0QSIq5l6UFcpndb2PkANvgcsoHyXmx+8RXEOGXV83AbY4yplkRG8ncAEJEM4FDgnbDtCiQ6O9ytwFDgQlVdErEtWkORjVozxpg6KmgJ5hPgfBF5E7ceTHMqjtzfHVgTNBEiciNwBfBnVZ0Stim0KHbbSge5TgV+JRsAYjVAeSUcK8UYY0yKBC3B/A037mUh8FfgTVX9MGz7icB/g0QoIjd4cV2tqndFbP7Ce/Zra9kX/7YZY4wxdUDQkfzzRaQvru2lEHgqtE1E2uIa+WfGG5+IXI9bO2acqt7qc76fvGlozhaRO8K6KQ8AOuNW0TTGGFMHBV4yOWknFrkC15j/EnBjxObtqvo/b79jcBnXf3CdAEIDLZcDR6hqSYLnt3EwxhgTUMqWTA7xGvjzgT28oO+AtwMulTzYez7Re4T7AegGoKqzReREYALwMm7es+dwVWoJZS7GGGNSL3AGIyIjgMlAa9wYGHC9uQpE5IqIRvqoVDU/3nOq6qvAq8FSaowxpjYFymBE5HRgCq566jZgMS6T6YWbW+whEdmW6PQtxhhjGo6gc5F9ipsX7FBV3RSxLRvXg2yHqu6X1FSmgLXBGGNMcKmci6wH8Ehk5gKgqoXAI1S9pLIxxphGImgGs5rydhc/pSQw0NIYY0zDEzSDmQKMFJHMyA0ikgWciyvFGGOMaeSqbOQXkX4RQe/guhR/JiL3Al/iepD1Ai4C1gHvpiCdxhhj6pkqG/lFpJTKE0qGV5GpX5iq1vnFxq2R3xhjgkvmQMu4Fg8zxhhjIlWZwajqozWVEGOMMQ1L0EZ+Y4wxJi6JzkXWATgQN11MpUxKVadWM13GGGPquaBTxaQB9wDnU3XpxzIYY4xp5IJWkV0JjAamAefgeo/9BfgD8DXwITAwmQk0xhhTPwXNYM4BXlPVEcArXthHqno/cACQ6z0bY4xp5IJmMHtQnrGUes/NAFR1K24U//nJSZoxxpj6LGgGsw3Y6f29BTfQsn3Y9tXAbklIlzHGmHouaAbzA/ArAFXdCXwD/DZs+7HYZJfGGGMInsHMBk4O+/8x4EwRmSMic4HfATOSlDZjjDH1WNBxMLcBr4tIhqpuBybhqsiGAyXAg8D1yU2iMcaY+ijQipYNiU12aYwxwaVyRUtjjDEmLpbBGGOMSQnLYIwxxqSEZTDGGGNSwjIYY4wxKWEZjDHGmJSwDMYYY0xKBF5wTERaAmcB3YG2uCn7w6mqnpeEtBljjKnHgi44djDwMi5jiUYBy2CMMaaRC1pFNhk3Pf8wIFdV03weTZKfTGOMMfVN0CqyA4CbVPWZVCTGGGNMwxG0BLMJWJ+KhBhjjGlYgmYwzwK/SUVCjDHGNCxBM5gxQHsR+aeI/EpEInuQGWOMMUDwNpgCXC+xg4GLAXzyGFXVwN2fjTHGNCxBM4KpuAzGGGOMqVKgDEZVR6YoHcYYYxoYmyrGGGNMSiTcViIimUAOPpmUqi6vRpqMMcY0AInMRXYGcC2wdxW72Wh+Y4xp5AJVkYnIUOBJXMb0AG6iy2nA08BO4GPghuQm0RhjTH0UtARzJbAEN2VMJnAh8LCqzhaR3sA84JOkptAYY0y9FLSRfz/gUVUtAkq9sCYAqvo58CBwTfKSZ4wxpr4KmsE0oXwusm3ec3bY9qVA7+omyhhjTP0XNIP5CegKoKrbgJ+BA8O29wC2Jidpxhhj6rOgbTDzgWOB67z/XwD+LCK/4DKrPwAvJi95xhhj6qugGcy9wMki0sIrwYzFzUs23tv+Ba4jgDHGmEYu6FQxHwAfhP2/FugjIvsBJcASVS2NdrwxxpjGIymzHqvqomTEY4wxpuFIaC4yEeknIhNF5F8i0tMLy/TCc5KaQmOMMfVS0JH8TURkOjAH+CtwLtDJ21wMPIe3TowxxpjGLZEVLU8FLsfNRVa22pg3+HImMChpqTPGGFNvBc1gRgBTVfVOYJ3P9iXAr6qdKmOMMfVe0AymG7Cgiu0FQOtEE2OMMabhCJrBbAbaVLF9T2Bt4skxxhjTUATNYN4DhouIRG4Qkda4Rv85yUiYT/yZInKXiKwSkW0i8qGInJSKcxljjKm+oBnMjUB3YDZwohe2v4iMxq0F0xK4OXnJq2AmcDZusbMTgMXATBGxTgXGGFMHiaoGO8D9oD8EdPCCFNeb7GdghKq+ntQUlp/zZeAUVZ3phQnwLtBWVataXTNanAXZ2dnZBQUFSU2rMcY0ZDk5ORQWFhaqak6sfQOP5FfVWSLSDRhIeVflr4HXVPWXoPHF6WSgEHg+LB0qIo8CD4pIL1VdnKJzG2OMSUBCU8Wo6nbgJe9RE3oDi33mOVsUvr0mEjLhxS9YvHJTTZzKGGNSplenLK4fvE9Kz5GUuchqQFvgK5/wDWHbKxCRghhxZsfY7mvxyk389/sNsXc0xphGLmYGIyKzA8apqjogwfRUGW+C24wxxtSCeEow+cBOYEeccabix349PqUUysfkVCpSxGqA8ko4gUsxvTplBT3EGGPqnJr4LYsngynGNeS/CTwCvFQLa758AZwqImkR597Xe/68phKS6jpLY4xpKOIZB9MZuAY3Sn8msEJEbhGRHilNWUUzgRxgcET4CGCp9SAzxpi6J2YGo6prVfV2Vd0XOAzXVfgCYLGILBCR80WkVYrTOQs3Q8BDInKuiPQXkSnAkcBVKT63McaYBAQaya+qC1X1QiAPV3rYCjwArBSR4SlIX+i8CgwFngJuAl4B9sMNvHwxVec1xhiTuETHwRQBT4jIMqAUOBbYI4np8jvnJuAS72GMMaaOC5zBiEgnXOllJG5espXAJFwHAGOMMQaIcy4yEWkGDAFGAccBJcALuEzltVroVVZtIlIKSHZ2QuMtjTGmUSosLATXchGziSVmBiMidwFn4RYSW4TLVB5X1Xo9nF1EinFtUHV13pdM73lLraYi+erq66qNdKX6nKmIP1lxVieeRI8N3U0WJnBOUy4LKFXV2AP148hgSoFtuK7CH8dxclXVf8STShOdiMwFUNX82k1JctXV11Ub6Ur1OVMRf7LirE48iR4bmj4qnlmATXLE2wbTAleKOSuOfRWwDMYYYxq5eDKY/ilPhTHGmAYnZgajqm/XREKMMcY0LEGXTDbGGGPiEnjJZGOMqY+skb/mWQnGGGNMSlgGY4wxJiWsiswYY0xKWAnGGGNMSlgGY4wxJiUsgzHGGJMSlsEYY4xJCctgjDHGpIRlMMYYE0FEDhORUhH5S22npT6zDMYYY8KISBpwB7CwlpNS7wVeMtkYYxq40cA8oE1tJ6S+sxKMMaZeEpFdReROEXlPRLaIiIpIfpR9M0XkLhFZJSLbRORDETnJZ7+2wJ+BCalNfeNgGYwxpr7aEzgTt3TyWzH2nQmcDVwLnAAsBmaKyKCI/SYBk1XVllVOAqsiM8bUV++oansAERkKVCqReNsGAccCp6jqTC9sDrAHcDswyws7AOgLXJjylDcSlsEYY+olVS2Nc9eTgULg+bBjVUQeBR4UkV6quhg4CugF/CwiAJlAiYjsparnJjf1jYNlMMaYhq43sNgnQ1oUvh34N/BM2PbJwDfAbSlPYQNlGYwxpqFrC3zlE74hbDuqugXXngOAiPwCbFLVDT7HmjhYBmOMaQyqWpfEd5uqjkxNUhoP60VmjGno1uOVUiKExrlYCSVFLIMxxjR0XwB7eyP0w+3rPX9ew+lpNCyDMcY0dDOBHGBwRPgIYKnXg8ykgLXBGGPqLRE5zfvzIO/5aBHJBbaq6ite2CxgDvCQN1L/e+Ac4EhgSE2mt7ER1aravowxpu4SkWg/YD+oarew/bKAm4DTcKWZxcANqvpcipPYqFkGY4wxJiWsDcYYY0xKWAZjjDEmJSyDMcYYkxKWwRhjjEkJy2CMMcakhGUwxhhjUsIymDrMWwJ2Sm2nIxEisou3RO1yESkRkWW1naZUE5Hx3nvWrRbO3c079/gaPm+9/YxWpbqvqzY/C3VJo8tgRCTfe+NVRM6Pso+KyEs1nbYGZgzwR2A6MBK4tDYTYxLn/VgOre10AIjISBG5tLbT0Vh5v5/jRSQnnv0bXQYTYYKItKjtRDRQA4HPVPUqVX3MRkyn3A9AC2BiCuK+HhiagngTMZKauVlpAfy+GsdP9OL4ITnJqTPycZ+HnHh2bswZzIdAJ+zOGgARaSIiuyQxyo7YNOg1Rp0iVS2u7bTUJSLSKpHjvGu5M9HzqmqxF0fjnipFVRvVA5cDK3AVLpMpANpG7KPASz5hU3ziG+ltyw8LG++F9QLuAFYBW4G3gB7ePqcAHwPbgGXABT5xKzAFOBZ4H/gFWA3cCbT02T8buAW3zOt2YC0wDdgjSpqPBcYB3wI7gZExrl1TXNXXYqAIt87GTGBfn7gjH+NjxD0CWOi9H1uB74AngHZh+xzsXY+vvGuxGZgHnOwT3xTvvG29v9d5+z8HdPT2uQBY4r2WL4EhEXF0C6UdOBO3xG4RsNwLaxqxf+h975bg+9Lci2Op9/oKgM+AW+P4XJelNUr6TwQ+8NK/Crg1Mv1VxFnp4fMZPQx423vv1uGWH870iTMPuM+7hjuAlcCDQPs4XuOyKOnJ97bP9fbZA7f08YZQWnE302OBd3DfoR1eGu4j4vsf7fse5LX6fRbCwnrg5kX7yfs8fAoM8knDLrhlm1d5n4f3gQFeGjTW9fLiOMFL6zrcb81y4Flgr6DvC+Xfqbi/2415NmXF/Vi+ifvgXZ6CczyKW4L1JqAdcAXwmoiMA/6Oe0MfBs4DHhCRxar6XkQcfXET9P0LmAr0B/4E9BaRgeqtMy4i2cB8oIsX5xe4D83FwH9F5EBVjSyu3wY08+LehPthq8oTwDDgDS/tHYE/AAtE5ChV/R/uC/x/wD9wH+obvWMXVY7OEZHh3rV6F7gO90XoAhwPtMf9IAOcDPQEZuCqHtriZsV9VkTOVtUnfaJ/FfdFvg7YE3ftZorIs7gM5iHcj+6fgGdEZC9V/T4ijsG4ku49uB+nk3DVBF2BUVGvFoHfl3uAc3Hv8z+AJkB34JiqzhGHQd757vfSMAS4EtiI+2xGsxb3Xj6Ge28ejLJfH+Al4BHgSdxN3HlAKe4aAyAiXYAFQDruun+Le08uAvp716KwivRcCkwCcoHLwsKXhP2diftBnYf7Xrf3wtNxN5X/AZ7HZQ4Heek8UkQOUNUdVZw70GuN4VHcDd1tXrouBZ7zPnvLwvZ7GvfePYf7ndodd0MX+fn0JSJHAy/gblIm4W5YOuFuLPfEW0Y6wPvyAJCF+x5ehvt+QxXf7ZSVFOrqg/ISzJXe/6/jfmC6RtypJKME8yLehKJe+J+88M1Al7Dwdl4apvmcU4GhEeF3euFnRIRtA/aP2LcrLvOY4pPmpcAucV63gd4x0yNe035AMfBuxP7LgLlxxv2sl8ZYd9R+pbZdvNexOCJ8ipfeeyLCJ3vhy4GsiNehwKSwsG5eWAnQNyxccF90BQ71ed+7Jfi+bABmJfi5DqV1vE/Y1og0CW6RrVVxxu372Q/bVhp+Hbzwl3E/oplhYc8DPwO7Rux7oPcZGh9HWuYCy6rYpsBEn20CtPAJP887Zlis1xzwtfp9FkJhL1HxO3SQz2dvkBf2r4hzhcI1jmsV+qxXWToM8r74va6qHo25DSZkDC7n/lsK4r5LvXfF8673/LyqLg8Fqupa3I9kd584lmrlBvKbveeTAUREgLNxpYcVIpIbeuB+XN4HjvOJ+z5V/SXO13Ky93xj+GtS1UW4L8yRItIuzrgiFeIyihO81+JLVbeG/va6Qbf1jpuNW7Ewy+ewOyL+D70HU1V1U1jci3A/+H7vwRuq+nHYvoorgUL5dakkgfelENhHRHpHizNBz2nYnbGX/jlARxHJTEL8C1T1/Yiw2bgq1W5QVpI7EXdHXRRxLZbhqg/9PqOJuC0yQJ1tXlqaiEiOd+7Z3i6HxBl3zNcahzsjvkMf4G46wz97ocXRJocfqKqzqFhiq0qoNHiqiPjWVqX6fWn0GYy6ap1pwNkisl+So/8u4v+N3rNfEXcj/uuGV/owqeoqXHF3Dy+onXfscbhqjcjHQKCDT9xfVZn6inbH3b35fbg/D9snETfhqryeA9aKyH9E5PzIBloRaS8iD4rIGsrrv9cCF3q75PjEnZL3ANcOBeXvgZ+g78ulQGvgMxH5VkT+LSJDfJb6DSryGoBrPwP/15uK+Hvgfm/Ow/9a9MD/MxrUWlUt8NsgIsNE5L+4EuVG77yhtLeOM/5kXEu/ODZEHB/6vn3js2+squyQu4H/AfcCG0Rkloj8KeJGMKXvS2Nugwl3La6d4xZcvX8QVV3DkoDhfnfv6hMWuW/o7zdxryFe8ZZeIs+XVKr6tYj0wjVgDgCOxrULTRCRfqr6rVcaeB3YG7gL12BdiLuWo4Cz8LlhUtVUvgexBHpfVPV5b2DeINw1OBb3xX9XRI7V+NoI/ER7reFprI544g89P45rg/CzLQlp8f1Mi8gpuOrdhcCfgR9x1dJNcO108WbiybiW8Xz2Qn8n+tlDVdeLyEHAUbibmX64tr0JIjJIVReQ4vfFMhhAVb8XkfuAP4tI/yi7bQDa+IRXdQebDL0iA0QkD9czKXQntBZXoslS1TdTlI5vgd/gfuAjG/VCaYyr8dGPqm7HLW07C0BEBuHqti/HdSTYD9gftwrh9eHHRhswm0SV3oOwML+70ZDA74uqbsB92R/3MtWbgatxDfNPx5vgOugb3I9lejU/o4n+4P4fLkPpH14tLCI9q5GWVPoel+l1p3IJuke8kXg3WHO9B14tzUe4m+oTCP6+BLr+jb6KLMxEXB18tDvNr4DDwseKiEhrYvQiSoIePqOox3jPzwGo60n2BHCwlK9RXoGItPcLD+A57/ma8HYSr73gJOA9ry0pMK++N1KozSOUqYfu+ircJXrnj9oOkiQDRaRv2DkF96MP5delkiDvS6hdIOJ4xVVxgP/NTU3ZUt3zq+p63M3DKSJyaOR2ceJpw9sCtK6qrS6KEtyPY9lvnhfHtQHjqSkves/hveVCN157xxNBlO/Vl7gSSRtI6H3Z4j3H9XmwEoxHVdeJyK1Eb+y/G3dnOVtEHsPV9/8e13bQMYVJ+wx3N/sv4GtcN+XTcF0xp4ftNxY4ApghIjNwDcg7cL2VBuHuWkYmmghVfcOL9wzcF/wlyrsph7r5Jup1ESnENYb/iLu2I3E/CI95+yzBdfG92svklwJ7AaNxbUB9SZ1Pce/7PbgxCUNw1VePedUMVYn3fWkFrBKRF3CZys+4eviLcO0FL1aKuea8DxwrImNwve9UVZ9KIJ6LgPeAd0RkKu51puFqAYbgumePjyMtJwJ3i8h8XMYxW1V/jnHcM8CpuPdxKq57/lBcJ5G6aBbwGvB7L6MIdVO+AFeDEE978b9EZFdc1XJopofTcZ+1qWH7BXlfQh0cbhGRJ3Df/c9V9XN8WAZT0WTceIG8yA2q+oSIdAIu8fb7DrgB1xAXbw+URHyMqya6EdeYvQmX2f3Vu0MOpa9QRI7AjbUZhvtgFOPGgLyHGwxWXWd76RkJ3I5raH8bGKeqn1Uj3vtwaR6NuzNaj/uQ/1FV54Ar6ovICbgeQucALXEZyzm4qrNUZjAv4DK0a3DVEz/jbkRi9jwM8L78guvxNgCXeWXiMrMXcN1XVybv5QR2MW6MzljcjxNA4AxGVX8UkQNwJfAhwHDcD9SPuAx0RhzR3IH74TsN931Iw910VZnBqOpTXqeRy3CfoVCm/RfKG+nrDFVVETkV970/E9c2vAhXWr8Y/96OkR7DfVfPwXU42YTrnHKaqv4n7Fxxvy+qOs+70bgQ107aFJhAeUefCqRiL1pjTIjX4P49MEFVx9duaoxxROQzoJmq1tX2ozLWBmOMMXWQ+EzE65Xie+Nm06jzrIrMGGPqputE5Ne4QbGFuGlqzsVV6QUZjlBrLIMxxpi66V1cB5GrcMMSNuDmUhunqj/VZsLiZW0wxhhjUsLaYIwxxqSEZTDGGGNSwjIYY4wxKWEZjDHGmJSwDMYYY0xKWAZjjDEmJf4flfi+3H1jchUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.errorbar(train_size, train_errors.mean(axis=1),\n",
    "             yerr=train_errors.std(axis=1), label=\"Training error\")\n",
    "plt.errorbar(train_size, test_errors.mean(axis=1),\n",
    "             yerr=test_errors.std(axis=1), label=\"Testing error\")\n",
    "plt.legend()\n",
    "\n",
    "plt.xscale(\"log\")\n",
    "plt.xlabel(\"Number of samples in the training set\")\n",
    "plt.ylabel(\"Mean absolute error (k$)\")\n",
    "_ = plt.title(\"Learning curve for decision tree\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7c26511",
   "metadata": {},
   "source": [
    "Looking at the training error alone, we see that we get an error of 0 k$. It\n",
    "means that the trained model (i.e. decision tree) is clearly overfitting the\n",
    "training data.\n",
    "\n",
    "Looking at the testing error alone, we observe that the more samples are\n",
    "added into the training set, the lower the testing error becomes. Also, we\n",
    "are searching for the plateau of the testing error for which there is no\n",
    "benefit to adding samples anymore or assessing the potential gain of adding\n",
    "more samples into the training set.\n",
    "\n",
    "If we achieve a plateau and adding new samples in the training set does not\n",
    "reduce the testing error, we might have reach the Bayes error rate using the\n",
    "available model. Using a more complex model might be the only possibility to\n",
    "reduce the testing error further.\n",
    "\n",
    "## Summary\n",
    "\n",
    "In the notebook, we learnt:\n",
    "\n",
    "* the influence of the number of samples in a dataset, especially on the\n",
    "  variability of the errors reported when running the cross-validation;\n",
    "* about the learning curve that is a visual representation of the capacity\n",
    "  of a model to improve by adding new samples."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}