9. Model Selection & Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Selection & Evaluation\n",
    "\n",
    "<hr>\n",
    "\n",
    "### Agenda\n",
    "1. Cross Validation \n",
    "2. Hyperparameter Tuning  \n",
    "3. Model Evaluation \n",
    "4. Model Persistance \n",
    "5. Validation Curves\n",
    "6. Learning Curves\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Cross Validation\n",
    "* Simple models underfit.\n",
    "* Accuracy for training data & validation data is not much different.\n",
    "* But, accurcy ain't that great.\n",
    "* This situation is of low variance & high bias\n",
    "* On moving towards complex models, accuracy improves.\n",
    "* But, gap between accuracy on training data & validation data increases\n",
    "* This situation is of high variance & low bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.researchgate.net/profile/Ljubomir_Jacic2/post/How_does_model_complexity_impact_the_bias-variance_tradeoff/attachment/59d6233579197b807798188f/AS%3A306150770184192%401450003439733/image/biasvariance.png\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We need to compare across models to find the best model.\n",
    "* We need to compare across all hyper-parameters for a particular model.\n",
    "* The data that is used for training should not be used for validation. \n",
    "* The validation accuracy is the one that we claims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x2ccea217da0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPgAAAD8CAYAAABaQGkdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAACstJREFUeJzt3V+IXOUZx/Hfr6vS+g9Da4vshsYVCUihxoSABITGtMQq2osaElCoFNYbRWlBY+9655XYiyKEqBVMlW5UELHaBBUrtNbdJG2NG0u6WLKJNoqRqIWGxKcXO4E0XTtnM+e858zj9wPB/TPs+0zWb87Z2ZnzOiIEIKcvtT0AgOYQOJAYgQOJETiQGIEDiRE4kBiBA4kROJAYgQOJndXEF7Wd8ulxS5YsKbre6OhosbWOHj1abK2DBw8WW+vEiRPF1iotItzvNo0EntW6deuKrnf//fcXW2vnzp3F1tq8eXOxtY4cOVJsrS7iFB1IjMCBxAgcSIzAgcQIHEiMwIHECBxIjMCBxCoFbnu97bdt77dd7lkKAAbSN3DbI5J+Kek6SVdI2mT7iqYHAzC4Kkfw1ZL2R8RsRByT9KSkm5odC0AdqgQ+KunAKe/P9T4GoOOqvNhkoVes/M+rxWxPSJoYeCIAtakS+Jykpae8Pybp0Ok3iogtkrZIeV8uCgybKqfob0i63Palts+RtFHSs82OBaAOfY/gEXHc9h2SXpQ0IumRiNjb+GQABlbpgg8R8byk5xueBUDNeCYbkBiBA4kROJAYgQOJETiQGIEDiRE4kBiBA4mxs8kilNxpRJLGx8eLrVVyW6YPP/yw2FobNmwotpYkTU5OFl2vH47gQGIEDiRG4EBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiVXY2ecT2YdtvlhgIQH2qHMF/JWl9w3MAaEDfwCPiVUnlnjwMoDb8DA4kVturydi6COie2gJn6yKgezhFBxKr8muyJyT9QdJy23O2f9z8WADqUGVvsk0lBgFQP07RgcQIHEiMwIHECBxIjMCBxAgcSIzAgcQIHEhs6LcuWrlyZbG1Sm4lJEmXXXZZsbVmZ2eLrbVjx45ia5X8/0Ni6yIABRE4kBiBA4kROJAYgQOJETiQGIEDiRE4kBiBA4kROJBYlYsuLrX9su0Z23tt31ViMACDq/Jc9OOSfhoRu2xfIGna9o6IeKvh2QAMqMreZO9GxK7e2x9LmpE02vRgAAa3qFeT2V4maYWk1xf4HFsXAR1TOXDb50t6StLdEXH09M+zdRHQPZUeRbd9tubj3hYRTzc7EoC6VHkU3ZIeljQTEQ80PxKAulQ5gq+RdKuktbb39P58v+G5ANSgyt5kr0lygVkA1IxnsgGJETiQGIEDiRE4kBiBA4kROJAYgQOJETiQ2NDvTbZkyZJia01PTxdbSyq7X1hJpf8ev8g4ggOJETiQGIEDiRE4kBiBA4kROJAYgQOJETiQGIEDiVW56OKXbf/J9p97Wxf9vMRgAAZX5amq/5a0NiI+6V0++TXbv42IPzY8G4ABVbnoYkj6pPfu2b0/bGwADIGqGx+M2N4j6bCkHRGx4NZFtqdsT9U9JIAzUynwiDgREVdKGpO02va3FrjNlohYFRGr6h4SwJlZ1KPoEfGRpFckrW9kGgC1qvIo+sW2L+q9/RVJ6yTta3owAIOr8ij6JZIesz2i+X8QfhMRzzU7FoA6VHkU/S+a3xMcwJDhmWxAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJMbWRYuwc+fOYmtlVvJ7duTIkWJrdRFHcCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxAgcSI3AgscqB966Nvts212MDhsRijuB3SZppahAA9au6s8mYpOslbW12HAB1qnoEf1DSPZI+a3AWADWrsvHBDZIOR8R0n9uxNxnQMVWO4Gsk3Wj7HUlPSlpr+/HTb8TeZED39A08Iu6LiLGIWCZpo6SXIuKWxicDMDB+Dw4ktqgrukTEK5rfXRTAEOAIDiRG4EBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiQ791UcmtaVauXFlsrdJKbidU8u9xcnKy2FpdxBEcSIzAgcQIHEiMwIHECBxIjMCBxAgcSIzAgcQIHEis0jPZeldU/VjSCUnHuXIqMBwW81TV70TEB41NAqB2nKIDiVUNPCT9zva07YkmBwJQn6qn6Gsi4pDtr0vaYXtfRLx66g164RM/0CGVjuARcaj338OSnpG0eoHbsHUR0DFVNh88z/YFJ9+W9D1JbzY9GIDBVTlF/4akZ2yfvP2vI+KFRqcCUIu+gUfErKRvF5gFQM34NRmQGIEDiRE4kBiBA4kROJAYgQOJETiQGIEDiTki6v+idv1f9HOMj4+XWkpTU1PF1pKk22+/vdhaN998c7G1Sn7PVq3K+9KIiHC/23AEBxIjcCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxAgcSqxS47Ytsb7e9z/aM7aubHgzA4KpeF/0Xkl6IiB/aPkfSuQ3OBKAmfQO3faGkayT9SJIi4pikY82OBaAOVU7RxyW9L+lR27ttb+1dHx1Ax1UJ/CxJV0l6KCJWSPpU0ubTb2R7wvaU7bIvuQLwuaoEPidpLiJe772/XfPB/xe2LgK6p2/gEfGepAO2l/c+dK2ktxqdCkAtqj6Kfqekbb1H0Gcl3dbcSADqUinwiNgjiVNvYMjwTDYgMQIHEiNwIDECBxIjcCAxAgcSI3AgMQIHEiNwILGh35uspImJiaLr3XvvvcXWmp6eLrbWhg0biq2VGXuTAV9wBA4kRuBAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYn0Dt73c9p5T/hy1fXeJ4QAMpu9FFyPibUlXSpLtEUkHJT3T8FwAarDYU/RrJf09Iv7RxDAA6lX1uugnbZT0xEKfsD0hqeyrMQD8X5WP4L1ND26UNLnQ59m6COiexZyiXydpV0T8s6lhANRrMYFv0uecngPopkqB2z5X0nclPd3sOADqVHVvsn9J+mrDswCoGc9kAxIjcCAxAgcSI3AgMQIHEiNwIDECBxIjcCCxprYuel/SYl9S+jVJH9Q+TDdkvW/cr/Z8MyIu7nejRgI/E7ansr4SLet94351H6foQGIEDiTWpcC3tD1Ag7LeN+5Xx3XmZ3AA9evSERxAzToRuO31tt+2vd/25rbnqYPtpbZftj1je6/tu9qeqU62R2zvtv1c27PUyfZFtrfb3tf73l3d9kyDaP0UvXet9b9p/ooxc5LekLQpIt5qdbAB2b5E0iURscv2BZKmJf1g2O/XSbZ/ImmVpAsj4oa256mL7cck/T4itvYuNHpuRHzU9lxnqgtH8NWS9kfEbEQck/SkpJtanmlgEfFuROzqvf2xpBlJo+1OVQ/bY5Kul7S17VnqZPtCSddIeliSIuLYMMctdSPwUUkHTnl/TklCOMn2MkkrJL3e7iS1eVDSPZI+a3uQmo1Lel/So70fP7baPq/toQbRhcC9wMfSPLRv+3xJT0m6OyKOtj3PoGzfIOlwREy3PUsDzpJ0laSHImKFpE8lDfVjQl0IfE7S0lPeH5N0qKVZamX7bM3HvS0islyRdo2kG22/o/kfp9bafrzdkWozJ2kuIk6eaW3XfPBDqwuBvyHpctuX9h7U2Cjp2ZZnGphta/5nuZmIeKDteeoSEfdFxFhELNP89+qliLil5bFqERHvSTpge3nvQ9dKGuoHRRe7N1ntIuK47TskvShpRNIjEbG35bHqsEbSrZL+antP72M/i4jnW5wJ/d0paVvvYDMr6baW5xlI678mA9CcLpyiA2gIgQOJETiQGIEDiRE4kBiBA4kROJAYgQOJ/Qcpuo92pLZ1pQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2cce9f5c8d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(digits.images[0],cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = DecisionTreeClassifier(max_depth=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainX, testX, trainY, testY = train_test_split(digits.data, digits.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=10,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.fit(trainX,trainY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8488888888888889"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.score(testX,testY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9784706755753526"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.score(trainX,trainY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Decreasing the complexity of model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = DecisionTreeClassifier(max_depth=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=7,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.fit(trainX,trainY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8155555555555556"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.score(testX,testY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8923533778767632"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.score(trainX,trainY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Observation : With decrease in complexity the gap in training & validation accuracy also decreased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross Validation API\n",
    "* Splits data into k parts.\n",
    "* Use k - 1 parts for training the model\n",
    "* Use kth part for validation\n",
    "* Repeat the above steps multiple times to get a genralized behaviour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\awant\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_split.py:1943: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "scores = cross_val_score(dt, digits.data, digits.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7424326888371718"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cross-validate Function : Scores for multiple matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring = ['precision_macro', 'recall_macro', 'accuracy']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\awant\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('train_accuracy'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\awant\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('train_precision_macro'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\awant\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:125: FutureWarning: You are accessing a training score ('train_recall_macro'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'fit_time': array([0.0130055 , 0.01299024, 0.01097131, 0.01200843, 0.01197624]),\n",
       " 'score_time': array([0.00200057, 0.00199866, 0.00092721, 0.00100017, 0.00099969]),\n",
       " 'test_accuracy': array([0.75824176, 0.66574586, 0.77437326, 0.767507  , 0.76056338]),\n",
       " 'test_precision_macro': array([0.76914915, 0.7108411 , 0.77995528, 0.79070953, 0.77760345]),\n",
       " 'test_recall_macro': array([0.75738524, 0.66585586, 0.77432218, 0.76650794, 0.76228291]),\n",
       " 'train_accuracy': array([0.90369853, 0.89407666, 0.90125174, 0.92777778, 0.88834951]),\n",
       " 'train_precision_macro': array([0.90788642, 0.89748337, 0.90470943, 0.931221  , 0.89615475]),\n",
       " 'train_recall_macro': array([0.90373616, 0.89354486, 0.90111448, 0.92752685, 0.8888177 ])}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_validate(dt, digits.data, digits.target, scoring=scoring, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stratification for dealing with imbalanced Classes\n",
    "* StratifiedKFold \n",
    "  - Class frequencies are preserved in data splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = np.append(np.ones(12),np.zeros(6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.ones((18,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "skf = StratifiedKFold(n_splits=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(array([ 4,  5,  6,  7,  8,  9, 10, 11, 14, 15, 16, 17]),\n",
       "  array([ 0,  1,  2,  3, 12, 13])),\n",
       " (array([ 0,  1,  2,  3,  8,  9, 10, 11, 12, 13, 16, 17]),\n",
       "  array([ 4,  5,  6,  7, 14, 15])),\n",
       " (array([ 0,  1,  2,  3,  4,  5,  6,  7, 12, 13, 14, 15]),\n",
       "  array([ 8,  9, 10, 11, 16, 17]))]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(skf.split(X,Y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y[[ 4,  5,  6,  7,  8,  9, 10, 11, 14, 15, 16, 17]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Hyperparameter Tuning\n",
    "* Model parameters are learnt by learning algorithms based on data\n",
    "* Hyper-parameters needs to be configured\n",
    "* Hyper-parameters are data dependent & many times need experiments to find the best\n",
    "* sklearn provides GridSerach for finding the best hyper-parameters\n",
    "\n",
    "##### Exhaustive GridSearch\n",
    "* Searches sequentially for all the configued params\n",
    "* For all possible combinations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainX, testX, trainY, testY = train_test_split(digits.data, digits.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = DecisionTreeClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_search = GridSearchCV(dt, param_grid={'max_depth':range(5,30,5)}, cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "       estimator=DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best'),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'max_depth': range(5, 30, 5)}, pre_dispatch='2*n_jobs',\n",
       "       refit=True, return_train_score='warn', scoring=None, verbose=0)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.fit(digits.data,digits.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 15}"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7840845854201447"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=15,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RandomizedSearch\n",
    "* Unlike GridSearch, not all parameters are tried & tested\n",
    "* But rather a fixed number of parameter settings is sampled from the specified distributions.\n",
    "\n",
    "##### Comparing GridSearch and RandomSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import time\n",
    "\n",
    "#randint is an intertor for generating numbers between range specified\n",
    "from scipy.stats import randint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = digits.data\n",
    "Y = digits.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import RandomizedSearchCV, GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify parameters and distributions to sample from\n",
    "param_dist = {\"max_depth\": [3, None],\n",
    "              \"max_features\": randint(1,11),\n",
    "              \"min_samples_split\": randint(2, 11),\n",
    "              \"bootstrap\": [True, False],\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': [True, False],\n",
       " 'criterion': ['gini', 'entropy'],\n",
       " 'max_depth': [3, None],\n",
       " 'max_features': 1,\n",
       " 'min_samples_split': 3}"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param_dist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(n_estimators=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomizedSearchCV took 7.48 seconds for 20 candidates parameter settings.\n"
     ]
    }
   ],
   "source": [
    "n_iter_search = 20\n",
    "random_search = RandomizedSearchCV(rf, param_distributions=param_dist,\n",
    "                                   n_iter=n_iter_search, cv=5)\n",
    "\n",
    "start = time()\n",
    "random_search.fit(X, Y)\n",
    "print(\"RandomizedSearchCV took %.2f seconds for %d candidates\"\n",
    "      \" parameter settings.\" % ((time() - start), n_iter_search))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9282136894824707"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GridSearchCV took 30.00 seconds for 72 candidate parameter settings.\n"
     ]
    }
   ],
   "source": [
    "param_grid = {\"max_depth\": [3, None],\n",
    "              \"max_features\": [1, 3, 10],\n",
    "              \"min_samples_split\": [2, 3, 10],\n",
    "              \"bootstrap\": [True, False],\n",
    "              \"criterion\": [\"gini\", \"entropy\"]}\n",
    "\n",
    "# run grid search\n",
    "grid_search = GridSearchCV(rf, param_grid=param_grid, cv=5)\n",
    "start = time()\n",
    "grid_search.fit(X, Y)\n",
    "\n",
    "print(\"GridSearchCV took %.2f seconds for %d candidate parameter settings.\"\n",
    "      % (time() - start, len(grid_search.cv_results_['params'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9348914858096828"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* GridSearch & RandomizedSearch can fine tune hyper-parameters of transformers as well when part of pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Model Evaluation\n",
    "* Three different ways to evaluate quality of model prediction\n",
    "  - score method of estimators, a default method is configured .i.e r2_score for regression, accuracy for classification\n",
    "  - Model evalutaion tools like cross_validate or cross_val_score also returns accuracy\n",
    "  - Metrices module is rich with various prediction error calculation techniques"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainX, testX, trainY, testY = train_test_split(X,Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=20, n_jobs=None,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False)"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.fit(trainX, trainY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Technique 1 - Using score function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9688888888888889"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.score(testX,testY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Technique 2 - Using cross_val_score as discussed above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.91758242, 0.8839779 , 0.95264624, 0.95518207, 0.88732394])"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(rf,X,Y,cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cancer prediction sample for understanding metrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = DecisionTreeClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer_data = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainX, testX, trainY, testY = train_test_split(cancer_data.data, cancer_data.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt.fit(trainX,trainY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = dt.predict(testX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Technique 3 - Using metrices\n",
    "##### Classfication metrices\n",
    "* Accuracy Score - Correct classification vs ( Correct classification + Incorrect Classification )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9090909090909091"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.accuracy_score(y_pred=pred, y_true=testY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Confusion Matrix - Shows details of classification inclusing TP,FP,TN,FN\n",
    "  - True Positive (TP), Actual class is 1 & prediction is also 1\n",
    "  - True Negative (TN), Actual class is 0 & prediction is also 0\n",
    "  - False Positive (FP), Acutal class is 0 & prediction is 1\n",
    "  - False Negative (FN), Actual class is 1 & prediction is 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[43,  2],\n",
       "       [11, 87]], dtype=int64)"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.confusion_matrix(y_pred=pred, y_true=testY, labels=[0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/awantik/machine-learning-slides/blob/master/confusion_matrix.png?raw=true\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Precision Score\n",
    "  - Ability of a classifier not to label positive if the sample is negative\n",
    "  - Claculated as TP/(TP+FP)\n",
    "  - We don't want a non-spam mail to be marked as spam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9775280898876404"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.precision_score(y_pred=pred, y_true=testY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Recall Score\n",
    "  - Ability of classifier to find all positive samples\n",
    "  - It's ok to predict patient tumor to be cancer so that it undergoes more test\n",
    "  - But it is not ok to miss a cancer patient without further analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8877551020408163"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.recall_score(y_pred=pred, y_true=testY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* F1 score\n",
    "  - Weighted average of precision & recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9304812834224598"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.f1_score(y_pred=pred, y_true=testY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ROC & AUC "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### House Price Prediction - Understanding matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import california_housing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "house_data = california_housing.fetch_california_housing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,\n",
       "         normalize=False)"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.fit(house_data.data, house_data.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = lr.predict(house_data.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrices for Regression\n",
    "* mean squared error\n",
    "  - Sum of squares of difference between expected value & actual value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5243209861846071"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.mean_squared_error(y_pred=pred, y_true=house_data.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* mean absolute error\n",
    "  - Sum of abs of difference between expected value & actual value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5311643817546461"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.mean_absolute_error(y_pred=pred, y_true=house_data.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* r2 score\n",
    "  - Returns accuracy of model in the scale of 0 & 1\n",
    "  - It measures goodness of fit for regression models\n",
    "  - Calculated as =  (variance explained by the model)/(Total variance)\n",
    "  - High r2 means target is close to prediction\n",
    "  \n",
    "  \n",
    "  <img src=\"https://github.com/awantik/machine-learning-slides/blob/master/Capture.PNG?raw=true\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6062326851998051"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.r2_score(y_pred=pred, y_true=house_data.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metrices for Clustering \n",
    "* Two forms of evaluation \n",
    "* supervised, which uses a ground truth class values for each sample.\n",
    "  - completeness_score\n",
    "  - homogeneity_score\n",
    "* unsupervised, which measures the quality of model itself\n",
    "  - silhoutte_score\n",
    "  - calinski_harabaz_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### completeness_score\n",
    "- A clustering result satisfies completeness if all the data points that are members of a given class are elements of the same cluster.\n",
    "- Accuracy is 1.0 if data belonging to same class belongs to same cluster, even if multiple classes belongs to same cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.cluster import completeness_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "completeness_score( labels_true=[10,10,11,11],labels_pred=[1,1,0,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The acuracy is 1.0 because all the data belonging to same class belongs to same cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3836885465963443"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "completeness_score( labels_true=[11,22,22,11],labels_pred=[1,0,1,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The accuracy is .3 because class 1 - [11,22,11], class 2 - [22]  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(completeness_score([10, 10, 11, 11], [0, 0, 0, 0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### homogeneity_score\n",
    "- A clustering result satisfies homogeneity if all of its clusters contain only data points which are members of a single class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.cluster import homogeneity_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "homogeneity_score([0, 0, 1, 1], [1, 1, 0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9999999999999999"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "homogeneity_score([0, 0, 1, 1], [0, 1, 2, 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "homogeneity_score([0, 0, 0, 0], [1, 1, 0, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Same class data is broken into two clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### silhoutte_score\n",
    "* The Silhouette Coefficient is calculated using the mean intra-cluster distance (a) and the mean nearest-cluster distance (b) for each sample.\n",
    "* The Silhouette Coefficient for a sample is (b - a) / max(a, b). To clarify, b is the distance between a sample and the nearest cluster that the sample is not a part of."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Selecting the number of clusters with silhouette analysis on KMeans clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_blobs\n",
    "X, Y = make_blobs(n_samples=500,\n",
    "                  n_features=2,\n",
    "                  centers=4,\n",
    "                  cluster_std=1,\n",
    "                  center_box=(-10.0, 10.0),\n",
    "                  shuffle=True,\n",
    "                  random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x2ccf17db748>"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnX10HPV577/PriXbYGFhLOQXZPFiI6qYYohrh2OMo2BHhJNbp23SC84NKfTiOqWlTYtvk3KPT67vyWlT5ySFhsY1iTmXFOOU2xBzWxpHpgYMxXbMS4pRkCXsCNnCQsYY5NhG0u5z/9id0W9mZ2ZnX2ZfpO/nHB/vzszOPDOrfZ7f73n7iaqCEEIIsYiVWwBCCCGVBQ0DIYQQBzQMhBBCHNAwEEIIcUDDQAghxAENAyGEEAc0DIQQQhzQMBBCCHFAw0AIIcTBpChPLiItAH5obLocwAZV/VvjmI8D2AHgSHrTj1R1Y9B5Z86cqZdeemlxhSWEkHHOSy+9dEJVG7IdF6lhUNUuAIsAQETiAI4BeMLj0D2q+umw57300ktx4MCB4ghJCCETBBHpDXNcKV1JNwF4U1VDCUYIIaQ8lNIw3ArgMZ9914vIz0Xk30TkI14HiMhaETkgIgcGBwejk5IQQiY4JTEMIlIL4DcBPO6x+2UAzap6DYC/A/Bjr3Oo6hZVXayqixsasrrICCGE5EmpZgyfAvCyqg64d6jqB6p6Ov36KQA1IjKzRHIRQghxUSrDcBt83EgiMktEJP16SVqmd0skFyGEEBeRZiUBgIicB2AVgD8wtq0DAFXdDOCzAL4kIqMAzgK4Vbl6ECGElI3IDYOqngFwkWvbZuP1dwB8J2o5CCGlo6NzAHu6B7F8QQNWtTaWWxySI6x8JoQUlY7OAdzz2Ct45MVe3PPYK+jozAgtkgqHhoEQUlT2dA/i7EgCAHB2JIE93UwvrzZoGAghRWX5ggZMrYkDAKbWxLF8QXWnl3d0DmDDjoMTauYTeYyBEDKxWNXaiAduu7asMYZixTgst9jZkQQeP3AUD9x27YSImdAwEEKKzqrWxrIp0GIqcy+32EQwDHQlETIOqFR3Rz5yFXovxYxxBLnFKvWZFwOpxpKBxYsXK7urEpLCHCFPrYlXjLsjH7mKcS/Ffh5ebqlKfebZEJGXVHVxtuM4YyCkyqnULKB85CrGvVgxjtuvbw5tjIJG/qtaG7Fx9ULHeSr1mRcLGgZCqpxKzQIKK5epmIt1L17K3O/a+dRcVOozLxZ0JREyDqjUSuNscnm5ZACU7F427DiIR14cWyLm9uubsXH1wlCfrdRnHkRYVxKzkggZB5QzCyiIbHJ5uWTCjPSLxfIFDXj8wFHbMAWN/N2GoFKfeTGgYSCElI1cFHMUhK25mGj1DDQMhJCyUY5iuHxG/hOtnoHBZ0KIJ5WQp19sGRhsDgdnDISQDIrpOgkK0gZdJx8ZsgWE8x35F3NmUw1Ba84YCCEZFCtPP9sIPeg6ucoQZjaQbeQfNEMJmwJbqIyVAA0DISSDYrlOsil38zq18Rj6Tp6xlWWuMoQxJEHFb6bSXveDA9i0syuvey5UxkqAriRCSAZ+rpOwbpBNO7uwq/M4rri4DlNr4r5ZR9Z1tu3rxQs972J31yD2Hj5pK21rHwC82ncq8NpmhlM8JqibUuMpr1+w2VTaCQU2P/smFjXVF9XdU+4srLBEXuAmIr8EMAQgAWDUXVwhIgLgfgC3ADgD4PdU9eWgc7LAjZDSE7Y/0KadXXhwd4/9/parZ2PmtNpAY+JXaGZe0yLbtTc/04OEApNigmRSkczyGfP+1v3gABKGSsyl4M3rfF6GrJwxhkrrldSmqot8BPoUgAXpf2sBfLdEMhFCciCsG2RX53HH+zffGcrwzbt9+abbKC5A3ZSajGtaBF176NyIrdhH00Yh22csVrU2on3hbMc2S45cCYolFCNWETWVEGNYDeARTbEXQL2IzM72IUJIaQnr81/ZOivwvZfSXNXaiDtvuAzxmCChwNbnj2T0TrJwxyIsNu3swr/8Zz/EQ6YYUko+W+rrzGm1jvdD50Z8jw2iWmIJfpTCMCiAn4rISyKy1mP/XAB9xvuj6W2EkAoibNfSRU31mBRLqedJMcGipnrHfj+lOXRuBImkOrab17y7bT4Wzp2O0WQSu7sGHSNxy3118lcj8HKON82Yiq3PH8maDeSYuaTjFPlQN6UG8fQzqORYgh+lMAzLVPU6pFxGd4vIja79XgY+47sVkbUickBEDgwOVpf1JWS8EMYNsqd7EKNpBT+a1MBMJEtpdnQOoO/kGdTGY47t5jUXNdXjF/3vI31qh1Fxu69MauMxXN4wLcMYeaWm2jMXARJJtWcuQGqmc8fD+3HHw/sDZx0dnQPY+vwRJJKKuAB33nBZRbuNvIg8K0lV+9P/vyMiTwBYAuA545CjAJqM95cA6Pc4zxYAW4BU8DkygQkhOWMGVOum1KQUq3qPlt0ZTwDsAHNtPIa2lgasWdqcoUz3dA86AsPxmNifX9k6C10DPXATjwnuuvFyLGqqx97DJ+3Aed2UGt/iOTNOYRqfux99GcOJVNTihZ538eDnrwuV3dTZ/36uj7PsRDpjEJHzRaTOeg3gkwAOug57EsDtkuJjAN5X1bejlIsQUjzMmMHdj76Mh547jISmlHLbVRfbo3MTc+ZhKtLhRBJNM87zTUc1A9TrVlxhH7e+vQV3t83H3PopmHH+mPsnkVR09r+f4Qbr7H/fNwbgVVuxbV+vbRQsOa00Wi85rZkPkDIilVrI5kfUrqRGAM+LyM8B7Afwr6r6ExFZJyLr0sc8BeAwgB4ADwH4w4hlIoQUEbditxRoIqnYefDtVMHYP77kWzAWNqhtKvfNX1iM9e0tjv3r21vwwlduwjd+5xo7xgGMKWbLGKVkPmHvr43HHNe0rtPWktq2u2sQL/S86zineV4gM8uqoW4siD2cSFZd8DlSV5KqHgZwjcf2zcZrBXB3lHIQQqLDLNqyRsrDiWQqwygdEEgkFZuf6fEsGAvTh8hyVYUNBieTYz4nUzHv6R7EK2+9Z8dAAKBl1jTPAr6mGedhODFon6OtpQGHBoZw7NS5jPNabqnt+/vsfRbVGHxm5TMhpCC8YgaWEt/87JtjxkGRV7tqryK3oKZ6e7oHkXRtO3F62D6HuNJdZk6bnHGdxw8cxZ03XOao2l6ztBkAHEV+yxc0ZMyYTFoap+He9qsYfCaETDzcbSbM11Ylst/IOVsXVb8it/uf7vacZZgzGIudB9+2A8pms4dJMbEVvjuNdujciOdMxmub14xpak28Ko0CwDWfCSERk60FhLsdRltLyo1jHd/ROeDICHLjtVY0AHxz5xvoGjhtHyfiNAoAcPXcC/D//ni5LWeYlh/Z7tGUo5hGoRitNLjmMyGk7IRRZu64wZ7uExhNKh7d24t1H5+fUSDn5uxIAtv29drpqNasY2XrLBwa6BkrivIYA8+cNtkhY65rLpifNXsqFXuWUOqlRWkYCCGREFaZudtOjBoxic3PvokbF8z0nS0AsDOaTDeQ1a3VtAWKVDWtue2tk2dw1yMp74MlY9imeaVU1qVeWrQSeiURQqoYv8VtwvYLcs8YzNhwIqkYPD3s2B9PHzApJmhrSY3y1yxtdqS8ApmB4HhM8JG50x3b3hz8lf06155GpeyHVOqlRWkYCCF5E9RFNKwyc88YFs69wNFnqMHV2O7GKxvQ1tKAX5tdZ29zF7CtWdrsKDKLIVUQ9yc3LchoymcS1GgvqCNs1Mo6bJ+qYkFXEiEkb7K5OD52+QwA8GxxYeFevOaem660z718QQNe7TuF5w4N2plNrXOm46HnDme0pzBZ1dqIBz9/nV2dbF7fiiOcOD2Mp14ba7Lw0eYLsfX5I3Y9wrL5F9mf83MbFWsd6DD4LTAUBTQMhJC88VuRzJ3hY6WEeuGnYC2FvPX5I3aLjTtvuAxD50Yy2lN87cmDGBwaxnAi6VDcXorU3G6tNLeydRaGzo3gpd737HOaq8n5GcAwyrqcC/PkC11JhJDQuN0pfi6OYvnfHQ3pkoqhcyOom1KTUaR27NQ521jkcr317S3Y+eUVWN/e4rn2g3WufN1GQa62SoaGgRASCj8l59WKOxdFmkucom5KDR567nBGPYJJIf7++Ref7wh+W32U8vXxV+uCPXQlEUJCkUvKZC7+dz/l6VVbsKd70Dd1tTYec8QFcsGr7QYALJt/kcO1let5/VxtlQ4NAyEkFLkqubCK1H1er7USzNqC7fv7MozD3PopuLKxLrRRcPv9vdpuuGMj+cQKSh2gLhZsiUEICU1UgVTzvHu6Bx0tMm6/vtlhGDo6B/DA04dw8NgHUKTqGWIidn+ibK4er9YXgHOxIPfMo5B2GZUEW2IQQopOVCmT7vO6ZyZeBsnqn5RUtaul/VxcbsPjdl1tXL0wcGRfjMrjaspOomEghFQUQUt/Wq4lM9aQ1FQWTRKZi+4A2dtpm2tLh621yDVWEKZ9RiUZDhoGQkjFYSrpDTsOZozW3YsDJVUdi/OYhG2nnU2eQmIF2WYcpW6Slw2mqxJCKhqv1FczfbRl1jTbleS1jKbf590pttnI5zNBMphUWlorZwyEkIomqDIagL2cJuDtSso22i+FCyebDJWW1hpZVpKINAF4BMAspNx/W1T1ftcxHwewA8CR9KYfqerGbOdmVhIhlUc5fORei/w8fMeS0J+vpGyjUjy/SshKGgXw56r6sojUAXhJRDpUtdN13B5V/XSEchBCIqZcPnL3SDuoJ5MXpV7nIIhSNsnLRmQxBlV9W1VfTr8eAvALAHOjuh4hpHyUy0deaDvqUq9zEITfuhbloCQxBhG5FMC1APZ57L5eRH4OoB/Avar6eilkIoQUj3L6yLONtINcNMWsTC7EFVRpWUmRVz6LyDQAzwL4uqr+yLXvAgBJVT0tIrcAuF9VF/icZy2AtQAwb968j/b29nodRggpE5WUh29RqhhCoddxx0rc1d7FImyMIdJ0VRGpAfDPAB51GwUAUNUPVPV0+vVTAGpEZKbXuVR1i6ouVtXFDQ3V0YiKkIlEIemcUVEqF1eh16kklxYQoWEQEQHwfQC/UNVv+RwzK30cRGRJWp53o5KJEDKxyEfh5uPrL1Sxl3rpzmxEma56A4A9AF5DKl0VAP4SwDwAUNXNIvJHAL6EVAbTWQB/pqr/ke3cTFclhIQlFxdXIS6hSnSluSl7uqqqPg9AshzzHQDfiUoGQsj4IV/Fm0saaCHpq5WUbloobIlBCKl4SrVEZqX5+ssFW2IQQiqeUhWiVevCOsWGhoEQUvGUsk5iPLmE8oWGgRBS8XAkX1poGAghVQFH8qWDwWdCCCEOaBgIIYQ4oGEghBDigIaBEEKIAxoGQgghDmgYCCGEOKBhIIQQ4oCGgRBCiAMaBkIIIQ5oGAghhDigYSCEEOKAhoEQQogDGgZCCCEOaBgIIYQ4iNwwiMjNItIlIj0i8hWP/ZNF5Ifp/ftE5NKoZSKEEOJPpIZBROIAHgTwKQCtAG4TkVbXYb8P4D1VnQ/g2wC+EaVMhBBCgol6xrAEQI+qHlbVYQDbAax2HbMawP9Jv/6/AG4SEYlYLkIIIT5EbRjmAugz3h9Nb/M8RlVHAbwP4CL3iURkrYgcEJEDg4ODEYlLCCEkasPgNfLXPI6Bqm5R1cWqurihIbqFwAkhZKITtWE4CqDJeH8JgH6/Y0RkEoDpAE5GLBchhBAfojYMPwOwQEQuE5FaALcCeNJ1zJMAvph+/VkA/66qGTMGQgghpWFSlCdX1VER+SMAOwHEAWxV1ddFZCOAA6r6JIDvA/iBiPQgNVO4NUqZCCGEBBOpYQAAVX0KwFOubRuM1+cAfC5qOQghhIQjcsNAyESmo3MAe7oHsXxBA1a1NpZbHEJCQcNASAR0dA5g275evNDzLoYTSTx+4CgeuO1aGgdSFbBXEiFFpqNzAPc89gp2dw1iOJEEAJwdSWBPN+tvSHVAw0BIkdnTPYizIwnHtqk1cSxfwPobUh3QMBBSZJYvaMDUmjgAoDYeQ1tLA91IpKpgjIGQIrOqtREP3HYtg86kaqFhIOOacmUFrWptpEEgVQsNAxm3WEHgsyOJissKYhorqWQYYyBVTUfnADbsOIiOzoGMfWYQ2J0VFPS5qLEM1iMv9uKex14piwyEBEHDQKqWbArWDAKbWUGFKuZCjUqQwSKkEqBhIFVLNgVrBYFvv77Z4UYqRDEXY7TvZ7AIqRRoGEjVEkbBrmptxMbVCx1+/EIUczFG+34GKxfK6Qoj4x+pxg7Xixcv1gMHDpRbDFIB5BvELeRzVkB7ak28LAHtSpCBVCci8pKqLs52HLOSqhQvxTYRM13yTQst5HPlrlHwmrVMlO+blAYahirEKw0TQMWmZo43/IxKqQzz8gUNePzAUXvGwBgFKTY0DFWIn5+bo8jCcSv3bMre2l83pQZbnz9SkGEOa1gqYdZCxjc0DFWIOWKsjcfQd/IMWudMx9SaeNFGkRPRLeWeid15w2Weyt7LGMRjgkQyFa8LY5i9DFAuMz5WVpMooWEoMqVQqNaI0er3v7trEHsPn8SdN1yGoXMjoa4dJGdYJVVtxiObvO6Z2I9fOeo5M7OejWkMEkmFAFCkGucFGWav55tr3KDanj2pLiJJVxWRTSLyhoj8p4g8ISL1Psf9UkReE5FXRaTq04xKXdF6aGDI0e9/6NxIRmpmPnKGScmsturdMPIuX9CA2vjYT2Lggw/t99YsbNu+XvvZJJLq+AG58/vcKaXWe/Mc1vPNJYW22p49qT6imjF0APiqqo6KyDcAfBXAX/gc26aqJyKSo6SUKluko3MAdz/6sm0UgOyj1FzkNF1VcQHqptTkfI5KI4y8q1obsWz+RdjdlTKEo0nF1XPrMDyaxBUX12Hbvl7s6R77U62Nx9AyaxpeO/aB4zzDiSS27evF3sMn7VlB21UXY+frx5FIKmrjMdTGYxhOJG0jECZuYM0S+k6eqapnT6qPSAyDqv7UeLsXwGejuE6lEXW2iKkYTKMAAMvmXxRaOWSTc1VrI+684TJsfvZNJJKKrc8fwaKmel/jUex7jcJNEiSveb01S5tthV4bj6Hr+GkMJ5LoGjidcc5l8y9C65zpGYbBmmWYyvup19629w8nkmhraUDTjPMc9+gVN/CKZ3gZFkKKSSliDHcC+KHPPgXwUxFRAP+gqltKIE9kBI36ClV2pl+6Nh5DTIC0exu18RjWLG0uWE5TxqFzI4HB1KgyY6LqiBp0z9bsa/v+Pjz4+evs4/pOnrFnD160zpmOoXMjGduXzb/IYWDiAiQMP1M8JliztDm0y88dz/AzLIQUi7wNg4jsAjDLY9d9qrojfcx9AEYBPOpzmmWq2i8iFwPoEJE3VPU5n+utBbAWAObNm5ev2JHjN+orVNmZrpDhRBIxSW2PCXDXjZfnfD63nF4ZOdmynKLIjInSReUl77Z9vfbsy3IBPXzHEjtTyFLuXliB/u37++xzmEb6Y5fPAJAyIHb2kgDrVlwRqijRfBaJpNoGZmpNPJRhISRf8jYMqroyaL+IfBHApwHcpD59N1S1P/3/OyLyBIAlADwNQ3o2sQVItcTIV+4o8fuR56Ls/M7h9vNbs4WkAp3974c+V1gZh86NFG1GkMtsKR8XVTFdTydOf4gNOw5m+P1ffusUDh4be85xgX3Mg5+/Dtv29QKAbRTMlhVrljaHnq34ueuseMbMaZNpFEjkRNIrSURuBvAtACtU1XMuLiLnA4ip6lD6dQeAjar6k2znr8ReSUH9a8L0tunoHLDTTy3fsVXR7OXWcLuS3ErF65rAmMKaFBM0XjAZn7n2Eqxvb8m7/06YArBcz5uLoi+kb5D1zPd0n8BoUjEpPQ0bTQeIzWfqduusW3EF1re3eJ7zmzvfcMQkbr++GRtXL8w49o6H9zu+07aWBjx8xxJPGd1/F8UwDEx5nXiUu1fSdwBMRso9BAB7VXWdiMwB8D1VvQVAI4An0vsnAdgWxihUKkGzgmz++E07u+xAr8XZkYQjs8UdcJx/8fl20HM4kcyYhWSrjh5NKo6dOocHd/fgyIlf4e8N33pYRRHGRZaPaygXF5X7/Nv29YaW3RqtT4oJ2loacOL0h45nas0CrGcSJmvIeh4WhQaHV7U2Yk/3IIYTY99fMdxrlby6HSk/UWUlzffZ3g/glvTrwwCuieL6xSbMyCpMpo+fMtn8TI8jOAmkFMqJ0x864gpmwBFwuivc16ubUmMHLM39lowmT732Njbt7LJ95gAc7hS/5xBG6Rcze8nre3D7+F/oedfO6w/6zszYwmjaIM+cNtlxzODp4Yy4S5BsX9tx0PFsWxqn4d72q3y/dwCYFBN7huKXQBBFBli1pRuT0sK221nIxVURdmpuHrenexCPvNhr74sJsOLKBrTOmY6HnjvsCGp6uYtM37an20OAdR+fb7s9OjoH8MDThzJSLC0jYqVaut0W2VxT7mNz6TcUhqDvwe2SuXruBeh8e8g2il4V4e7PzK2fgs9ce4n9zGvjMUdNg/XdJDWlzJcvmGk/c7+6Evf35XUv1nWyxQ2K7fZh6+6JSbldSeMGL1eF3w80jAvEK/vHchHFAHwprcQ37DjoUDRedQqv9p3Cc4cGkVBg7+GT9o/bkc2icKRUWjL+znf/Ay/1vgcgFUg1UyEtzJGk1whz4+qFGe4VPxdFvim6pgH1GuFaI2/rGU6KicPonR1J2DMyU541S5ttvz0AHDt1DlufP4K7brzcMXOyXHkxjMV0RpNqtyGx7j+XuhJ3hln/qbOB9249v2Iq7qjSjcn4gCu4ZcFsVVAbj+GFnncLakXgVnBmRlEsJujsfx8dnQMZLRLcboaOzoFUXCKtrM6OJPDNnW94ftbteujoHEBnf0p5xgVoXzjbcY/uNhDu52BuX9XqXCGtWOsZu9s+1E2pybi+dYw1ql84dzqSSecMWADHM7LksbKJWhqn2cda2ViWIXq17xTmX3w+5tZPQdOMqRkyWudzZ4xlqysxnyUAdA2cxrofHMAfPvoyNuw4iE07u0rS8sL93RFiwRlDFsyRlZkZlK9f1t1uYvD0sMPXbY5Eg0Z0e7oHHcFqIKVg7nnslVCfNWcUM6fVOo63jnFX5VqN+7Ldn+XzD2rTkS2VNkz67AbDpz+cSGJkNIGk6zqfuno2dr/xjqd/3rqu6VKpm1KTEUAGUu4jq0meRUyAV956D6/3O91ys6c7YxVeo/8HbrvWkb2UUNjV0WZBHP3/pBzQMITA+jGbBU/5BgFXtTrbTRw6PmS7QSxMV4113Q07DqJuSo09ojUNjKmw3J/1wiuY6XZV+H3Wcr+80POurw/d5NW+U54GINtCQ14Fdm4Z3fdxxcV16HnnNBI6NhOaOa02I8awaWcXdnUex8rWWVjf3uIwOKZBMkn1TRqLXcQAxEQyYjUA0HvyLO557BXcecNl6Ox/335mXq61dT84kJF4kFB4Jg4QUipoGLLgHu0Vwy9rtpuwso0AOHLVLWXgFdh8dN9bWLfiClsWs49OGEWS7314VQl7zUbMYywDaCrFbKm0YQvszPuwnoGlVNs/MssxU7CuvWlnFx7c3QMA6BpI/b++vcVxfq/Mrdp4DPfcdKV9j9naZZwdSeC7z/TAnNS5YyN7ugfRvnC23VzPwi9gHgRrEkgxoWEIoJiBVBP3SNfMbnH/uE1lbJFIKjY/04PNX1hsF04taqrPqhi8XBpe28Ny6PgQOjoHfEfxZlDbVIp+6ZfZZjFeWMeYbqVEUvHmO0Oewepdnccdn9/VedxRqOY2Ns8degcnfzWMz1x7iSPY3TpnuqPGZPb0yXj3V8M4/eGYQUl6pCCbsRHrGVkzG3NGmMv3YJ5v+/6+UFlOhARBwxBAVLnefiP2XIxOQpFRRGcprjse3g/AP4XVvRpZ2EKnjEye98/ZMQ13LCJoJuN3/4XMxtyxjZWts/DWycxrr2ydZc8UrPcmZjfTzv737e6qW58/AgCO+3GP6jfsOOhIPbZwp6Q6jJgCO18/jnUrrsjLKACZWU5mnIrGgeQDDUMAhRQWhelTFOZHu2ZpM57tGswIqvplG5luJzMO4GfkcjF+ViaPGTT1+ox5b34zGa/7L2ZK5qKmek9DY80OzBiDhVflssXZkQR2dR7PcHeZrS7cxXZAKivqrhsvd1xn+YIGPLq3144tWDNAd0ptWMy/U1NeBq1JvjBdNQBrZHv79c0ZvY/MlbnceK2wZSntR17sxd2Pvhw6BXFVayNuvnq2Y9vVcy/IUB5Wjx5TKVmtMgD/dNOg1Fav+1zV2oh7268KvdpYLimR2Z5rEO7YhqUUva69vr0FO7+8IsMofHPnG55GAYA9Cwm6b8twzjh/LH1VkdnkcFVrI9Z9fD7i6d5M8Zh4ptSGxfo7bWtp8Ew1JiRXOGPIgnsUm29/IHNxneFEEl978qB9/mzMnFbreH/tvAsdRspsBGdiposGua9M14+plPw6f1qZVdaoO1d/uNcMotDePe5OpH0nz2TEP4Jk8pspmHjNQrxmgdv21QcGpoGUcbJmU7kmD3hhuhIZhCaFQsOQI/n2B3Ln/x87lemf98PPpeWVsQQAM86rwTVN9XaRldn3yC+759W+Uw53RkPdZN8MpI7OAVuRvXUyc3U3P4KUf6HxHMvAWZ1Id3cN4rlDg452IJYMbsXpTlFtaZyGla2zsKvzuO0ys2YhZgrxHQ/v90xFbZ0z3WEYWudM95U5m8stV4rpjiMTF7qSciRbVTHg7YJas7TZsdA84HQbBLlR/FxaXhlLAHBNU73dvjlMBa1XFXXfyTO+z8CrTUgYF1BQVXSY55qNVa2NaJpxnv1MEgpsfvZNWy4vF5/Xte9tvwrr21t8XWZmxbV1LfN+3Ku67eo8nvXZWG4vAFndlPm62wgJC2cMORK2BsA9crP8z+7e+u70RT83StiRoMjYYjFhR+FeVdSOCl/A0eLB7bZJ3c9gVhdQUDA/7HPNhldgNyjQbt2/V92An0xeRXDuuI0ZDDYr0oNSic2/Da9nyVbZpFRwxpAH7qBm2FHcqtZGPHzKnsivAAAXc0lEQVTHEjz4+esco/98+wutWdpsLy5jIQDuf7obdzy837O/kBfmiDkeE3y0+ULH/mubL8wwctYMZtn8izxHzX737zXzMfcX2rvHHdgNCrRb7S8eebEXW58/4tsY0S2Tu39WW0uDZ8quuw+T37PJNgOxKFYfKkKywbbbBVLoCmJ7ugdx4vSw3ScHSGUd3XPTlRnn8UuB/dqTB3Hs1LmM89fGY45uoUFyBbUCB4C72+b7rlhWie2bg4LcfvcZtH6C+/N+xWjm+QHvtuRuvOofvI6v1GdNqoewbbdpGArE/aP2W8bRjXupSLcrx10UFaQUgrJq/OQJWjMBANb940sOmeICbP7CYk9FFMX6C/mQ63X9VlzzU7jZFLN7VbjlC2aidc70rIY5l/UZmHVECoGGoUTkO4pzGxSzo6aJdc5t+3odmS5uhW/5qK31GQDnYjFuxW8psBiAm40OpNbCPgDsnkJ+1/TCXKa0lKPafL8Hq37BXKPZb+aQbRDgXvwHCDY0bjmo8EnUhDUMjDEY5Jvx8bHLZ2T4mbPh9nev+/h8R4GShZX180LPu/a2mMCxBoClVFrnTEc8lvp8DKmKW8somMV19z/dbfuykwD+7bW3HS0aNj/7JhY11ePutvmIp0MYYTKFOjrTy5S6+iPlSj7fQ77+d3fBHjAWLHZfP5/MqbCyFCO+QkixiCwrSUS+BuAuANav4i9V9SmP424GcD+AOIDvqepfRyVTEPlkfLhHqUGLs7gx8+5PnP4Qnf3vY83SZrTOmY4fv3IUAx98iNH0qBtwrqyW1FTPnkVN9QDg6ZJKYixt0t0Vtf89Zyqqe6JiZfJsXL0wI7/empkAyHB37OkedMx64jHJOe0038ybQtqXWN9FmFYfQZlT7l5SQHiDytkCqSSiTlf9tqp+02+niMQBPAhgFYCjAH4mIk+qamfEcmWQT4FVMZrsmYpkT/cJxETsRnBtLTNtY2N18rTwaledSKrtkgpSSHMuPA8fnPvArpROtXtotNs/uxvembEMv15MQGZn1XUrrsj5eeT7TAtNd7WONw29X41KUIryg5+/LjA47YYpqKQSKXcdwxIAPap6GABEZDuA1QBKbhjyGXEWMkoFnP19AKQV9dg6DU0zzrOVxAO3XYsHnj7kWBimbkoNFjXVO2Twysc3R7KTYoKGabVYseIKu4dPUNvvIHnNnkRAcWoRCh35F5rqWqj8Zq2D2WLE71xexYKcPZByE1nwOe1K+j0AHwA4AODPVfU91zGfBXCzqv739PsvAFiqqn8UdO6ogs/5TOkLcQO4R+CTYmLPGLyCln7BzzAyuAuoLCPiNg65yGsGt4tJNbtWCsl0suJLft8/IYVSkqwkEdkFYJbHrvsA7AVwAqkh8P8GMFtV73R9/nMA2l2GYYmq/rHHtdYCWAsA8+bN+2hvb/Daw9WC22cPZK63bB5bSB6727DEZGwxmbBKPijGQLxrEoDgjC7LELpXhQub+kxIWMIahoJcSaq6MqQwDwH4F49dRwE0Ge8vAdDvc60tALYAqRlDbpJWLl7ujyAfdqGL2ThWVzOeotstlIu8ZAyvtRHCtCa3XHmFrilOSDGI0pU0W1XfTr/+MlIuoltdx0wCcAjATQCOAfgZgDWq+nrQuUtZx1DNbg0vzMrdh547HLlbqFyU83vLVh0d5rPj5e+NVBZlL3ATkR8AWISUK+mXAP5AVd8WkTlIpaXekj7uFgB/i1S66lZV/Xq2c5fKMFRzC4Jc4g5AZbiFiqUUq/l7IyRKSuJKCkJVv+CzvR/ALcb7pwBk1DdUAlGt+Rw1YVMgK8ktVMy0zUK/N47ayUSHlc8BFGONgHIQpgq4XH39/a5bzM6hhXxvfms2EDKRmJCGIZc22UFtoiuVbIqxXMov6LrFNMKFfG9sbU1I+QvcSk6uLotKcreEJVv2UrlcZEHXLUZxmUm+31uhRYuEjAcmnGEIqxSr3c8cpBjLpfyyXbcSjHCxDRQh1ciEa7sdJmNlImS1VMuaCYSQ4lH2rKRKxFJKXv2ETKo1GykbbqVcjnuqhFkBISSYCWMYcpkFjEc/s19shSN4QoibCWMYcpkFjEc/s1+2zURu+UyjSIg3EyZdNdd0yPG2opbX/U+k1Ex3ijLrFQjxZ8LMGMbjLCAX/O5/vLnMvPByo3EdBEL8mXBZScTJRHCneK1jsXxBA9dBIBMOZiWRUEyELCF3MoG1spqVnWaugzCestAIyRcahglO1DOGYp/f73zmdgAZry0jUDelBlufP+LITgPgWAehbkoNNuw4OK5nUYQEQVfSBCbqQj5zKdBirPfgltdL2ZtuIS8X0Z7uwcDlUb0MB40DGS+EdSVNmKwkkknUWUnb9vXaCwENJ5L22g/ZCNuBdfMzPXjkxV78/e4ee/twIum4pvXauj+/7DQrC23o3EioZ5JPd9pydbQlJFdoGCYwldhWPGwH1nhM7KVJzTlvbTxmzxTM19b9Zeu8GuaZ5JPqyvRYUk0wxjCBiTqFd83SZrzQ867t1lmztDnrZ8J2YK2bUoPNz76JRHLMLMw4rwYQQd3kOC5vmGZfz4w3WLGDjasXel4/zDPZtq8355Yp47XNChmf0DBMcLJlJeUSPPbqxfTg56/LyfDk2oF18zM9SCgwKSY4eWYEAHDyV8CxU+cchujVvlN27ODxA0cD+2VZ7y03kjvI/ULPu/b72ngs1ExrPLZZIeMXBp+JL7kEp4sZyM7HGD39iwEcO3XOsa+tpcHONooLbNcTAPt9bTyGZfMvcqx5HXQv7pqItpYGPHzHklD3MRFqRkhlU9Y6BhH5IYCW9Nt6AKdUdZHHcb8EMAQgAWA0jMCkdOTi/sjVVRKkJPOprbim6UIcO/W2/X5STGxZgJQRiMcEiaTa/wOpAPXurkHsPXzSNgBB9+Ie+fu5x/yaFtIgkGogkuCzqv5XVV2UNgb/DOBHAYe3pY+lUagwcglO53Kslcb6yIu9+INHDmDTzq685DMDurvfeAe3XD0bc+un4Oq5F+C7/+2jWLO02SHTuhVX4Pbrm7FuxRX2dgszAynoXla1NqLtqotRP3US2q66OCdDSUi1EGmMQUQEwO8C+ESU1yHRkEtwOpdjzTTWJIDvPtODRU31oUbT5kzDrXxnTqvFC1+5yXG8n0yLmuqxbV+vHRyPC1A3pSbrvWza2YWnXkvNTJ567W1s2tmF9e0tcMOYAqlmIo0xiMiNAL7lNxsQkSMA3kMq4/AfVHVLmPMyxlDd3PHwfrsFhYW70MxUyJt2dmFX53FccXEddr/xjqPAzSxGCwoo+7muNu3ssrObwsRG2r/9LLoGTtvvWxqnYeeXV2Qc19E5YNdtmPELQspJ5DEGEdkFYJbHrvtUdUf69W0AHgs4zTJV7ReRiwF0iMgbqvqcz/XWAlgLAPPmzctXbFIBtM6Z7jAMk2KC5QsaPP3yr/adwoO7ewDAoZDPjiQwdG7Ekb5qZh2ZCt6swN6+v89RgT10bsSON4SJjaxsnYWugR7Hezfu4HWYNF1CKom8YwyqulJVF3r82wEAIjIJwG8D+GHAOfrT/78D4AkA3ukdqWO2qOpiVV3c0MBpebkppIp36NyI4/2vza4DAHxtx8EMv/yuzuOe5zAL1pYvaMCuzuO+Pv2gCuxci/zWt7fg7rb5aGmchrvb5nu6kRhfINVOlDGGlQDeUNWjXjtF5HwAMVUdSr/+JICNEcpDioRfxk22z1iunOULGrB9f5+trH/x9hC+9I8vYdQoVrPqA+qm1DhG6LdcPRszp9U6UkAtWSxy8ennU+S3vr3F0yBYML5Aqp0oDcOtcLmRRGQOgO+p6i0AGgE8kYpPYxKAbar6kwjlIUUin9RUtyFZNv8i251kGgSLZfMvcqR37uo8jpWts7Coqd4xAjdlAVI+/3vbr3LIk60COyiN1K9rayGBeNYzkEonMsOgqr/nsa0fwC3p14cBXBPV9Uk0dHQOoO/kGdTGY3bXUr8RsaUA+06eyTAka5Y228VntfEYkqq2gXAr7/XtLXYW0UPPHcZwImkbGCuTyGJl6yzPYrJcKrC9Oq1u398HAI5rA/6Gws/Y5DPbIqTUsCUGCY2p1GrjMbS1NPhm3LiPdRsS96gagG8Wj5e7yM93P3RuJEP5emUreY3arUwiO4XVVQhnXnvbvl7bsG3b9xYaL5iMz1x7ia+LKchI0jCQSoOGgYTGdNsMJ5JomnFeqAKv4UQSbS0NaJpxXmAzuzDnMqmbUoNFTfV2vMKKS2S0506no5ojfctwPLq3F+s+Ph+LmuozjE8iqXbrDElvU8AOVlvHjiYVx06ds7On3MYhm5EkpNJg220SmkIqodcsbbaNgFf76aAsp+ULGhCXjM0Z2U3m8Va77RiQkY5qGo6EpgrszI6pFlNr4mhfOBsxSRkEBRAT4M4bLnNUVZt4ZVG5jeSy+Rf5tv0mpBLgjIGEphiV0H6pnNn87nMvPA9vnTxjv7dqH9ypqKZrCgBiMcEkEccI/dW+U45zJxUYPD2MqTVxe1RvNdbb0z0IMzae1JRBsu7vgacP4bVjH9j7veoavPor0SCQSoaGgeRELo3gvI71SuUMynLyii+kzjMTALCn+4S9zXQlWcZiNKloa5lpu7GsJnluGqbV4k9ucsY8rAC05foxr2Hen1WZvbJ1lmeMIep1LwgpNjQMpOR87PIZAMaCzO4RvJlp5BVfsEbd2/b1OlJdr5xVZyvdoBG6u45iUkzsY6yMJnOt6rtuvByd/e87ZDYx6xr8UlHZWZVUEzQMpOj4KUe/VhGW0rUwYwfmDMO9doJ7DemGabUAso/QrfRVvywot3uqs/993zUX3PftdokB4WofCKkkaBhI0XCne7rjBX7xhaAV0YKUfFDhWrYRurnfCnz7Ke+f/fIklv3V0/jMdf7pqF73Z6a0VnrNAovuiAkNAykKQbUGfovcuOMBwFjFs4lbyZtKzF24FlbBeRWxWcrbNDgAcPrDBE5/mPBNR7VwF9udOP1hYOykUhQxi+6IGxoGUhT8YgFhRv9hVkSz8FJiVhpsWAVnHmcWsVnKe+PqhbjrxsttQ2Cyq/O4r2Fwp8/OnDbZznSynkW2WVU5yLXFCRn/0DCQUGQb4QbFAkzco/9cM3aClFhYBeeoYzCK2ExD5lcj4ZWO6vUMLCNnpbxa5802qyoHbPpH3NAwkKyEGYkXkpKZS8ZO3ZQax9rNpvsmrIJzG7GWWdMwc9pkhyEzjxEAF55fg9uWNAfGGPyegfX/BqOtuEUlKGKm0xI3ka7gFhVcwa20bNhxEI+8OJYB5NXBtBT4tdh2L8oTNsZgunS8Vm/LNQ6Q7Xh3awy/WRUhURH5Cm6keig00GmOnoHUSmr3PPZKyX3jXnEMtysmbAttq9BtODHoeZ5s5/I6t6X0t+/v81T6HJmTaoG9ksY5lsJy9ybKBUuhtTROs7eVY2Uys/+SRVhXjNdzyHX1tiDc/ZB2dw16Pu9VrY3YuHohjQKpaDhjGOcUK+PE+oxZoFZq37g54q6bUhOqlbaF13PYuHph0Ubw7lmVeZ1iuKEIKSWcMYxzijkqthRzOTuDWiPuRU31ju3ZZkZ+zyHfEby7G6z1bNpaxjq7FjKbIaScMPg8ARhvo1F3aw1r1G8GyG+/vtmx1oP1uVyfg9+CPu7rFxK0dgf3vWQnpBgw+ExsxlsDNy+3kLvq2P0eyP05+KXpZnPP5Xod1hGQSqMgV5KIfE5EXheRpIgsdu37qoj0iEiXiLT7fP4yEdknIt0i8kMRqS1EHlI4QQvmVApebiF3QZpfgVou+PV2KqZ7DqgMFx0hJoXOGA4C+G0A/2BuFJFWALcC+AiAOQB2iciVqupen/EbAL6tqttFZDOA3wfw3QJlInlSLT1zwrTWCKusg9w+fiP5KNJOx9usjlQ3RYkxiMgzAO5V1QPp918FAFX9q/T7nQC+pqovGp8RAIMAZqnqqIhcnz7Gc3ZhwhhDNFS7rzufgrSgWEE+5ySkkil3jGEugL3G+6PpbSYXATilqqMBx9iIyFoAawFg3rx5xZOU2FS7rzvXUXeYVF6O5MlEJKthEJFdALw6h92nqjv8PuaxzT01CXPM2A7VLQC2AKkZg99xJH8mWmVutRtCQqIiq2FQ1ZV5nPcogCbj/SUA+l3HnABQLyKT0rMGr2NIiZlII+SJZggJCUtUrqQnAWwTkW8hFXxeAGC/eYCqqojsBvBZANsBfBGA3wyETGCi9PNPJENISFgKTVf9LRE5CuB6AP+aDjJDVV8H8E8AOgH8BMDdVkaSiDwlInPSp/gLAH8mIj1IxRy+X4g8ZPzBqmBCSk9BMwZVfQLAEz77vg7g6x7bbzFeHwaQfZV1MmHh6mKElB72SiIVTbGKyaqhcI+QSoEtMUhFU4wAcbUU7hFSKdAwkIqn0AAx3VGE5AZdSWRc4eUyKnZvI0LGO5wxkHGDn8uI9QqE5AYNAxk3BLmMWK9ASHjoSiLjBrqMCCkOnDGQcQNdRoQUBxoGMq6gy4iQwqEriRBCiAMaBkIIIQ5oGAghhDigYSCEEOKAhoEQQogDGgZCCCEORLX6lk8WkUEAvSW41EykliCtNqpR7mqUGahOuatRZqA65a40mZtVNWvlZ1UahlIhIgdUdXG55ciVapS7GmUGqlPuapQZqE65q1FmgK4kQgghLmgYCCGEOKBhCGZLuQXIk2qUuxplBqpT7mqUGahOuatRZsYYCCGEOOGMgRBCiAMaBg9E5HMi8rqIJEVksbF9lYi8JCKvpf//RDnldOMnd3rfV0WkR0S6RKS9XDIGISKLRGSviLwqIgdEZEm5ZQqLiPxx+tm+LiJ/U255wiIi94qIisjMcssSBhHZJCJviMh/isgTIlJfbpn8EJGb038TPSLylXLLkws0DN4cBPDbAJ5zbT8B4L+o6tUAvgjgB6UWLAuecotIK4BbAXwEwM0A/l5E4qUXLyt/A+B/qeoiABvS7yseEWkDsBrAr6vqRwB8s8wihUJEmgCsAvBWuWXJgQ4AC1X11wEcAvDVMsvjSfr39SCATwFoBXBb+ndYFdAweKCqv1DVLo/tr6hqf/rt6wCmiMjk0krnj5/cSCmt7ar6oaoeAdADoBJH4wrggvTr6QD6A46tJL4E4K9V9UMAUNV3yixPWL4N4H8g9dyrAlX9qaqOpt/uBXBJOeUJYAmAHlU9rKrDALYj9TusCmgY8ud3ALxiKYMKZy6APuP90fS2SuNPAWwSkT6kRt0VORr04EoAy0Vkn4g8KyK/UW6BsiEivwngmKr+vNyyFMCdAP6t3EL4UC2/OU8m7ApuIrILwCyPXfep6o4sn/0IgG8A+GQUsmW5dj5yi8e2sowSg+QHcBOAL6vqP4vI7wL4PoCVpZTPjyxyTwJwIYCPAfgNAP8kIpdrmVP+ssj8lyjD328YwvyNi8h9AEYBPFpK2XKgYn5z+TBhDYOq5qVwROQSAE8AuF1V3yyuVNnJU+6jAJqM95egTG6aIPlF5BEAf5J++ziA75VEqBBkkftLAH6UNgT7RSSJVI+cwVLJ54WfzCJyNYDLAPxcRIDU38PLIrJEVY+XUERPsv2Ni8gXAXwawE3lNr4BVMxvLh/oSsqBdAbEvwL4qqq+UG55cuBJALeKyGQRuQzAAgD7yyyTF/0AVqRffwJAdxllyYUfIyUvRORKALWorMZpDlT1NVW9WFUvVdVLkVJi11WCUciGiNwM4C8A/Kaqnim3PAH8DMACEblMRGqRSv54sswyhYYFbh6IyG8B+DsADQBOAXhVVdtF5H8i5fc2FdYnKyXY6Cd3et99SPlkRwH8qapWnG9WRG4AcD9SM9lzAP5QVV8qr1TZSf/wtwJYBGAYwL2q+u/llSo8IvJLAItVtWKNmYWI9ACYDODd9Ka9qrqujCL5IiK3APhbAHEAW1X162UWKTQ0DIQQQhzQlUQIIcQBDQMhhBAHNAyEEEIc0DAQQghxQMNACCHEAQ0DIYQQBzQMhBBCHNAwEEIIcfD/AVvbbaLJ45/rAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf1d15d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X[:,0],X[:,1],s=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [],
   "source": [
    "range_n_clusters = [2, 3, 4, 5, 6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "from sklearn.metrics import silhouette_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 0.7049787496083262\n",
      "3 0.5882004012129721\n",
      "4 0.6505186632729437\n",
      "5 0.5650860173118194\n",
      "6 0.4504281968733957\n"
     ]
    }
   ],
   "source": [
    "for n_cluster in range_n_clusters:\n",
    "    kmeans = KMeans(n_clusters=n_cluster)\n",
    "    kmeans.fit(X)\n",
    "    labels = kmeans.predict(X)\n",
    "    print (n_cluster, silhouette_score(X,labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The best number of clusters is 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### calinski_harabaz_score\n",
    "* The score is defined as ratio between the within-cluster dispersion and the between-cluster dispersion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 1604.112286409658\n",
      "3 1809.991966958033\n",
      "4 2704.4858735121097\n",
      "5 2282.092875614491\n",
      "6 2043.3289234246856\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import calinski_harabaz_score\n",
    "\n",
    "for n_cluster in range_n_clusters:\n",
    "    kmeans = KMeans(n_clusters=n_cluster)\n",
    "    kmeans.fit(X)\n",
    "    labels = kmeans.predict(X)\n",
    "    print (n_cluster, calinski_harabaz_score(X,labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Model Persistance\n",
    "* Model training is an expensive process\n",
    "* It is desireable to save the model for future reuse\n",
    "* using pickle & joblib this can be achieved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = pickle.dumps(dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickle.loads(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytes"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* joblib is better extension of pickle\n",
    "* Doesn't convert into string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.externals import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['dt.joblib']"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(dt, 'dt.joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Loading the file back into model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = joblib.load('dt.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Validation Curves\n",
    "* To validate a model, we need a scoring function.\n",
    "* Create a grid of possible hyper-prameter configuration.\n",
    "* Select the hyper-parameter which gives the best score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import validation_curve\n",
    "\n",
    "param_range = np.arange(1, 50, 2)\n",
    "\n",
    "train_scores, test_scores = validation_curve(RandomForestClassifier(), \n",
    "                                             digits.data, \n",
    "                                             digits.target, \n",
    "                                             param_name=\"n_estimators\", \n",
    "                                             param_range=param_range,\n",
    "                                             cv=3, \n",
    "                                             scoring=\"accuracy\", \n",
    "                                             n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xl8FfXV+PHPyQJhiRAIIhDCDgEEUSIgoiAisqmIWnABtVXrU0Vpq9U+tVZpterjr7XuK25VkKpYqmhAZVPZggQQyAYEiOwgIQnZc35/zCS92S9Jbm6W83697it31ntmktwz32XmK6qKMcYYU98E+DsAY4wxpjyWoIwxxtRLlqCMMcbUS5agjDHG1EuWoIwxxtRLlqCMMcbUS5agTIVEpLuIqIgEudOfi8jN3qxbjc/6XxF5vSbxNnYiEikiGSISWMk6KiK96zIub4lIioiM83ccpuGwBNWIiUiMiMwtZ/5VInLwdJOJqk5U1bdrIa4xIpJaat+Pq+ptNd13BZ/XSUTeEJEDIpIuIvEi8qiItPLF551GXK+IyIse08EiklnBvBGquldVW6tqgbtshYhU+5yJyCMikucmvRMi8p2IXFCzo/I/EXlLRHLd4yp6Ta/jGCwZ1wJLUI3bW8BMEZFS82cC76lqft2HVLdEpB2wBmgBXKCqocBlQFugVzX2V60SYgVWAaM9pqOBvcDFpeYBbKzFz/X0gaq2BsKB5cC/fPQ5de0pN5kXvT443R1UVlI1dcMSVOP2CdAOuKhohoiEAVOAd9zpySKySUROisg+EXmkop15XrGLSKCIPC0iR0VkFzC51Lq3isgOt8SyS0R+6c5vBXwOdPa4uu3sXs3/02P7K0Vkm3tlv0JE+nssSxGR+0Rki4ikicgHIhJSQdi/AdKBm1Q1BUBV96nqvaq6pbyqyVLHeYuIfCsifxeR48Cf3ZjO9li/g4hkiciZ7vQUEYnzKJUMriC2lUB/EQl3py8CFgCtSs1bo6p5nrGKyGPusufdc/i8x37HiUiSiPwkIi+Uc4FShnux8h7QRUQ6uMcRJiKfisgRd1+fikhEqfP0Z/f8pIvIUo+4EZGZIrJHRI6JyB88P09EmovIMyKy3309IyLN3WVjRCRVRH4nIofFKflOFZFJIpIoIsdF5H+rOqbyiEh/N+4T7t/XlR7L3hKRl0RkiYhkApe4cT4tIntF5JCIvCwiLdz1w91zcsKNabWIBIjIu0Ak8B/3d/O76sRqLEE1aqqaBSwEZnnM/hkQr6qb3elMd3lbnCTzPyIy1Yvd346T6M7Fucq/ttTyw+7yM4Bbgb+LyHmqmglMBPZ7XN3u99xQRPoC84E5QAdgCc4/e7NSxzEB6AEMBm6pIM5xwMeqWujFMVVkOLALOBOYC3wMXF8qlpWqelhEzgPmAb8E2gOvAIuLvnw9qWoqsIf/XkBcDKwGvis1b1U52/7BXfdu9xze7bF4CnA+cI4b2+VVHaB7bmcBx4Cf3NkBwJtAN5wv3Czg+VKb3oDz+z0TaAbc5+5vAPASTmm9s3suIjy2+wMwAhjixjkMeMhj+VlACNAFeBh4DbgJGIpzbh4WkZ5VHVepYwwG/gMsdeOdDbwnIv1KHc9jQCjwDfAk0NeNs7dHPAC/BVJx/kY7Av8LqKrOxCkJX+H+bp46nTjNf1mCavzeBq4ruurD+RIqbkdS1RWqulVVC1V1C05iGF3Ofkr7GfCMWxo5DvzVc6GqfqaqO9WxEudL4aLydlSO6cBnqrpMVfOAp3Gq6EZ6rPOsqu53P/s/OF8g5WkPHPDycyuyX1WfU9V8N+m/T8kEdYM7D5zE/YqqrlPVArfNLgfny7g8K4GLRSQA50t6LU7iKZp3obvO6XhCVU+o6l6caruKzg3Az0TkBE7yuR24tqjqV1WPqepHqnpKVdNxvrhL/228qaqJHhdDRZ91LfCpqq5S1Rzgj4DnRcKNwFxVPayqR4BHcZJZkTzgMff3vwCnCvIfqpquqtuAbTgXJhW5zy3ZnBCRo+68EUBr9/zkqurXwKeU/F3+W1W/dS9octxz8mtVPe6eg8eBGR4xdgK6qWqeqq5We7hprbIE1cip6jfAEeAq94rzfP77ZYqIDBeR5W41ThpwJ86XQVU6A/s8pvd4LhSRiSKy1q36OAFM8nK/Rfsu3p/7ZbEP5+q1yEGP96dwvnjKcwznS6Qm9pWa/hpo4Z67bjhfyovcZd2A33p8OZ4AuuIcU3lW4ZSSBgG7VPUUzpV70bwWwLrTjNfbcwOwUFXb4pQAfsApoQAgIi3F6cixR0ROurG2lZJtMxV9Vom/D7fkfMxj3RK/Y/e95zk6VtQZBCd5AhzyWJ5VxXE9rapt3VfR311nYF+p0vQeSv5def6uOwAtgY0ev8sv3PkA/wckA0vFqcZ+sJJ4TDVYgmoa3sEpOc0Elqqq5z/6+8BioKuqtgFeBqpss8AplXT1mI4seuNWZ32EU/Lp6H4BLvHYb1VXmftxvuiL9ifuZ/3oRVylfQlc7ZZGypPp/mzpMe+sUuuUiNf9gluIc+V9A05JId1dvA/nyr+tx6ulqs6v4PNX4VRxTcYpOYFTOujqztugqtkVbFtrV+uqehSnWvIRESlK6L8F+gHDVfUM/tt547T/PkSkJU5ptkiJ3zHO30+Jql4f2A90LfW3EEnJvyvPc3oUJxEO9PhdtnE7leCW5n6rqj2BK4DfiMil5ezHVJMlqKbhHZy2mNvxqN5zhQLHVTVbRIbhfOF6YyFwj4hEiNPxwvPqsRnQHKfkli8iE4HxHssPAe1FpE0l+54sIpe67Qa/xalu+c7L2Dz9Dacd7G23tIOIdBGRv4nIYLd66UfgJnE6fvwc73r3vY9TFXkjHiVSnLaSO93SlYhIK3E6ooSWtxNVTcY5H/fiJii3mmidO69M+5OHQ8BptcNURlXjgRigqFE/FOcL+oQ4vSH/dBq7+xCYIiKj3PatuZT8vpkPPCROB5NwnHadf5azn9q0DueC5HfidN8fg5NYFpS3snsh8hpO+2lRB5guInK5+36KiPR2L6BOAgXuC2r5d9NUWYJqAtzea98BrXBKS55+BcwVkXScL4mFXu72NZwvs83A9zgdB4o+Lx24x93XTzhJb7HH8nicL6hdbtVJieovVU3AaRB/Ducq9gqcBudcL2Pz3NdxnLarPGCde5xfAWk41TPgJO77caqgBuJFIlTVoi+7zji9Eovmx7r7e9499mQq7sBRZBVOtdG3HvNW4zTkV5ag/gFcK04Pu2eritlL/wfc4X4hP4NTxXgUp23sC2934rYT3YWTvA/gnAvPe9/+AsQCW4CtOH9Df6mF+CuLKRe4EqeTzlHgRWCW+/dYkQdwfodr3WrOL3FKlQB93OkMnFsZXlTVFe6yv+Ik4BMicl9tH0tTIdamZ4wxpj6yEpQxxph6yRKUMcaYeskSlDHGmHrJEpQxxph6qTYffOlX4eHh2r17d3+HYYwxpgobN248qqodqlqv0SSo7t27Exsb6+8wjDHGVEFE9lS9llXxGWOMqacsQRljjKmXLEEZY4yplyxBGWOMqZcsQRljjKmXfJagRGSeOMM1/1DBchGRZ0UkWZyhu8/zWHazOENWJ4nIzb6K0RhjTP3lyxLUWzhDcldkIs7TgPsAd+AMD43HY/2H44ww+id3OAdjjDFNiM/ug1LVVSLSvZJVrgLecce+WSsibd2B0sYAy9xhEhCRZTiJrqIB30wtycvL49SpU2RmZnLq1ClOnTpFYWEhQUFBBAYGlvlZ0bzAwEDy8vLIyckp88rNzS13fk5ODnl5ef4+BcYYL1xyySXUxYMR/HmjbhdKDq+c6s6raH4ZInIHTumLyMjI8lZpNFSVlJQUkpKSyM7OrvBLvqJEkJ2dXSb5lJ7Oz8/392EaYxqAjz76qNEnqPKGjdZK5pedqfoq8CpAdHR0oxnYSlXZs2cPGzduJDY2lo0bN7Jx40aOHz/u1fZBQUE0a9aM5s2bF79CQkJo1aoVLVu2pE2bNnTq1Kl4uuhVerply5YEBARQUFBAfn5+iZ/lzfNcFhwcXOLzi16l4/J8BQcH4wxOaoypzzp0qPIpRbXCnwkqFejqMR0B7Hfnjyk1f0WdRVXHVJV9+/YVJ6Kin8eOHQOcZDNo0CCmTZtGdHQ0AwYMoFWrVhV+2Tdr1ozAwEA/H5UxxtScPxPUYuBuEVmA0yEiTVUPiEgM8LhHx4jxwO/9FaQvpKWl8cwzz7B27VpiY2M5evQo4CSjs88+m6lTpxIdHc3QoUMZNGgQISEhfo7YGGPqns8SlIjMxykJhYtIKk7PvGAAVX0ZWAJMApKBU8Ct7rLjIvJnYIO7q7lFHSYag7i4OK699lp2797NoEGDuPLKK4uT0eDBgy0ZGWOMy5e9+K6vYrkCd1WwbB4wzxdx+Yuq8sYbb3D33XcTHh7OqlWruPDCC/0dljHG1Fv2JIk6cOrUKW699VZuv/12Lr74YjZt2mTJyRhjqmAJyscSEhIYPnw477zzDo888giff/55nfWAMcaYhqzRDFhYH33wwQfcdttthISE8MUXXzB+/Hh/h2SMMQ2GlaB8ICcnh9mzZzNjxgwGDx7Mpk2bLDkZY8xpsgRVy/bs2cPFF1/M888/z29+8xtWrFhBRESEv8MyxpgGx6r4atGSJUu46aabKCgo4KOPPmLatGn+DskYYxosK0HVgvz8fP7whz8wefJkIiMj2bhxoyUnY4ypIStB1dDBgwe54YYbWL58ObfddhvPPvssLVq08HdYxhjT4FmCqoH8/HxGjhzJwYMHeeutt7j5Zhtb0RhjaoslqBpYu3Ytu3fv5r333uOGG27wdzjGGNOoWBtUDSxdupSAgAAmTpzo71CMMabRsQRVAzExMQwfPpywMBuR3hhjapslqGo6duwYGzZs4PLLL/d3KMYY0yhZgqqmL7/8ElW1J0QYY4yPWIKqppiYGNq2bcv555/v71CMMaZRsgRVDarK0qVLGTduHEFB1hHSGGN8wRJUNWzfvp0ff/zR2p+MMcaHLEFVQ0xMDIC1PxljjA9ZgqqGmJgYoqKiiIyM9HcoxhjTaFmCOk1ZWVmsWrXKqveMMcbHLEGdptWrV5OdnW0JyhhjfMwS1GmKiYmhefPmjB492t+hGGNMo2YJ6jTFxMRw0UUX0bJlS3+HYowxjZolqNOQmprKtm3brPeeMcbUAZ8mKBGZICIJIpIsIg+Ws7ybiHwlIltEZIWIRHgsKxCROPe12Jdxemvp0qUA1v5kjDF1wGePQRCRQOAF4DIgFdggIotVdbvHak8D76jq2yIyFvgrMNNdlqWqQ3wVX3UsXbqUTp06MWjQIH+HYowxjZ4vS1DDgGRV3aWqucAC4KpS6wwAvnLfLy9neb1RUFDAsmXLGD9+PCLi73CMMabR82WC6gLs85hOded52gxc476/GggVkfbudIiIxIrIWhGZ6sM4vbJx40aOHz9u1XvGGFNHfJmgyitmaKnp+4DRIrIJGA38COS7yyJVNRq4AXhGRHqV+QCRO9wkFnvkyJFaDL2smJgYRIRx48b59HOMMcY4fJmgUoGuHtMRwH7PFVR1v6pOU9VzgT+489KKlrk/dwErgHNLf4Cqvqqq0aoa3aFDB58cRJGYmBjOO+88fP05xhhjHL5MUBuAPiLSQ0SaATOAEr3xRCRcRIpi+D0wz50fJiLNi9YBLgQ8O1fUqbS0NNauXWvVe8YYU4d8lqBUNR+4G4gBdgALVXWbiMwVkSvd1cYACSKSCHQEHnPn9wdiRWQzTueJJ0r1/qtTX3/9NQUFBZagjDGmDvl0tD1VXQIsKTXvYY/3HwIflrPdd0C96csdExND69atueCCC/wdijGmDqgqO3fuZNOmTQQHBxMeHl78CgsLIzAw0N8hNgk2HGwVVJWYmBjGjh1LcHCwv8MxxvjQ0aNHWbduHevWrePo0aMEBwejquTn5xevExAQQFhYWHHCat++fYkE1qpVK7sVpZZYgqpCUlISKSkp3H///f4OxRjjA9nZ2WzatIm1a9eSlJSEiNC3b18mT57MkCFDCA4O5sSJExw9epRjx45x9OjR4tfWrVs5efJkif2FhITQoUMHevfuTVRUFH369CEkJMRPR9ewWYKqgj3eyJjGp7CwkMTERNauXUtcXBy5ubmceeaZXHHFFQwfPpx27dqVWL9du3Zl5hXJyckpk7gOHDjAN998w/LlywkMDKRHjx7079+fqKgounXrRkBAw34MqqrWSSlRVEvfmtQwRUdHa2xsbK3v94orrmDHjh0kJyfX+r6NMXXr0KFDrF27lnXr1nHixAlatGjB0KFDGTFiBD169KjVL93c3Fx27drFjh07iI+PZ98+57kFLVq0oF+/fkRFRdG/f/9q37pSUFBARkYG6enp5OXlVbheZceUm5tLdnY2WVlZZGdnl3lf0fRtt93G4MGDqxW3G9NG9z7XSlkJqhK5ubksX76cm2++2d+hGFMvHT58mB07dlBQUEBQUBBBQUEEBgYSHBxMYGBg8byi+Z7TLVu2rJNha1SV2NhYVqxYwe7duxERBgwYwLRp0zjnnHN81rbcrFkzoqKiiIqKAiA9PZ2EhITihBUXFwdA+/bti0tXERERZGVlFSeejIyMEi/PeVlZWT6JOyAggBYtWhASElL8s02bNpx11lmEhIQQEhJC+/btq95RLbAEVYlvv/2WzMxMq94zxqWqpKamEhcXR1xcHAcOHKj2vgICApgwYQITJkwgKMg3X0Xp6em8//77bN68mU6dOnH11VczbNgw2rRp45PPq0xoaCjR0dFER0ejqhw+fJj4+Hh27NhBbGws33zzTbnbBQYG0rp16+JXt27daNWqFaGhocXzmjVrVm5JqbIaMlWlWbNmZZJRUFBQvenkYQmqEjExMQQFBTFmzBh/h2JMpdLT04mPjy9+5ebm0q1bN7p37178at26dbX2XVhYyK5du9i8eTNxcXEcO3YMEaF3795cd911DBo0iJYtW5KXl0dBQQH5+fnFr4KCgnLn5+fnEx8fz5IlS9iyZQszZ86ka9euVQdzGrZu3cp7773HqVOnmDZtGmPHjq03bT8iQseOHenYsSOjR4+moKCAlJQUDh8+XJx8in6GhITUm4RR16wNqhLnnXceoaGhrFy5slb3a0xN5ebmkpSUVJyQfvzxRwBatmxJv379CAkJYc+ePRw4cKD4Kjo8PLxEwuratWuF1Vv5+fkkJCSwefNmNm/eTHp6OkFBQURFRTFkyBAGDRpEaGhojY9jy5YtvP/++2RkZNRaaSonJ4ePP/6Y1atX07lzZ2699Va6dCn9nGrjT9YGVUOHDh1i06ZNPPbYY1WvbBqtwsJCMjMz+emnnzhx4gTp6en069eP8PDwOo9j7969xVVCu3fvJj8/n6CgIHr16sWVV15J//796dq1a4lSQnZ2Nnv37iUlJYWUlBSSk5MpupALCAggIiKiOGFFRkZy6NAh4uLi+OGHH8jKyqJ58+YMHDiQIUOGMHDgQFq0aFGrxzV48GB69erFwoULa6U0tXv3bt5++22OHDnCuHHjuOKKK+z+xQbMSlAV+Oc//8nMmTOJjY1l6NChtbZfU38UFBSQlpbGiRMnil9FiajolZaWVuImTaC42nfChAk+beTPyMjg+++/Jz4+noSEhOJG8YiIiOLG9969e9OsWbPT2u+JEyeKE9aePXvYs2cP2dnZxctbtWrF4MGDGTJkCFFRUXX2Bb9582bmz59PRkYGEydOZMKECV4/saGgoIDPP/+cL774grZt2zJr1iz69u3r44hNdXlbgrIEVYGZM2fyxRdfcOjQoXpTb21qRlXZt28fcXFxbN68mYMHD5ZpRA4ODqZt27bFr7CwsBLTzZs358svv2Tt2rW0bNmSKVOmMGrUqFp99E1aWhpffvkl33zzDTk5ObRr1644IfXr169WqtY8FRYWcujQIfbu3UtYWBi9evXy26N8MjMzWbhwIRs2bCAiIoJZs2YRERFR6TaHDh3i7bffJiUlhWHDhjF9+vRaL+mZ2mUJqgYKCwvp1KkTl156Ke+//36t7NP4R1EDf1Gvs+PHjyMi9OnTh169epVIQGFhYbRs2dKrBul9+/bx4YcfkpSURMeOHZk2bRpnn312jRqzjx07xtKlS1mzZg2FhYVER0dz2WWX0blz5ybXSB4XF8eCBQvIyMhg0qRJXH755WWSpqqyevVqPv74Y4KCgrjhhhs477zz/BSxOR2WoGogLi6Oc889l7feesvugWqACgoKSExMLC4pnTx5skQD/+DBg6vdo82TqrJ161Y+/vhjDh8+TFRUFNOmTavyir+0Q4cOERMTw/r16wkICGDEiBGMHz++ztu56puMjAwWLlxIbGwsXbt2ZebMmcXnNi0tjX/+859s27aN/v37M3PmTNq2bevniI23LEHVwJNPPsmDDz7I/v376dSpU63s0/hWbm4uO3bsIC4uji1bttRJA3+R/Px8Vq9ezWeffUZWVhYjR45kypQpVd5rk5qayhdffMGmTZsICgrioosuYty4cfZFW0pcXBzz58/n1KlTTJw4kU6dOjF//nxycnKYOnUqo0ePtmr4BsYSVA2MHTuWY8eOsXnz5lrZn/GN/Px8tm7dSmxsLNu3bycnJ4cWLVowePBgzj33XKKiok67A0FNZGZm8vnnn7Ny5UqCgoIYP348l156aZkYdu/ezRdffMHWrVsJCQlh9OjRjB07ttbblhoTz9IUQNeuXbnlllvsArKBsgRVTRkZGbRr1445c+bw1FNP1UJkpjYVdXRYu3YtGzZsIDMzkzPOOINzzjmHIUOG0LdvX7+P1XP48GEWLVrE5s2bCQsL46qrriI6Oprk5GS++OIL4uPjadWqFZdccgljxoypk8f9NBabN2/myJEjjBkzxmdPnzC+Zwmqmj799FOuuOIKli1bxrhx42ohMlMbTp48yYYNG1izZg379+8nKCiIc845hxEjRhAVFeX3pFSexMREPvroI/bt28cZZ5zByZMnOeOMMxg3bhyjRo2yIRhMk2U36lbT0qVLadGiBaNGjfJ3KE1eURXe2rVr2bZtG4WFhXTv3p0ZM2YQHR1d70seffv25YEHHmDDhg2sX7+eiRMnMnLkSLtx1BgvWYIqJSYmhjFjxtjVrZ+UV4XXpk0bLr30UkaMGNHg2hwCAgIYPnw4w4cP93coxjQ4lqA8pKSkkJiYyK9+9St/h9KkqCoHDx5k+/btJarwBg8ezAUXXFBvq/CMMb5lCcpDTEwMYKPn+lpBQQE//vgjycnJJCUlsXPnTjIyMgCKq/CGDh1Kq1at/BypMcafLEF5WLp0KV27dqVfv37+DqVRycvLY8+ePSQnJ5OcnMyuXbuKn/3Wvn17Bg4cSO/evenTpw9nnnmmn6M1xtQXlqBc+fn5fPXVV1x33XVN7rEytS07O5tdu3YVJ6SUlJTiB6526tSJ888/n969e9O7d2/CwsL8HK0xpr6yBOVat24daWlpVr1XCVXl1KlT/PTTT6SlpRU/+bv0+8zMTMDpINC1a1dGjx5N79696dWrV608YsgY0zR4laBEZBTQR1XfFJEOQGtV3e3FdhOAfwCBwOuq+kSp5d2AeUAH4Dhwk6qmustuBh5yV/2Lqr7t5TFVS0xMDAEBAVx66aW+/JgGIzs7m9WrV5Oamlpi+Im8vLwS64kIoaGhtGnThvDw8OIHsEZGRtKzZ0/rDWmMqbYqb9QVkT8B0UA/Ve0rIp2Bf6nqhVVsFwgkApcBqcAG4HpV3e6xzr+AT1X1bREZC9yqqjNFpB0Q636uAhuBoar6U0WfV9MbdYcPH05gYCDfffddtffRGOTk5LBy5UqWLVtGZmYm7du3L37Sd5s2bcq8b9Omjd3Rb4w5LbV5o+7VwLnA9wCqul9EvHlo2DAgWVV3uQEtAK4CtnusMwD4tft+OfCJ+/5yYJmqHne3XQZMAOZ78bnV8uKLLxZXTTVFubm5rF69mqVLl5Kens7AgQOZPHky3bt393doxpgmypsElauqKiIKICLe9v3tAuzzmE4FSt+tuBm4Bqca8GogVETaV7BtFy8/t1qa6qi5eXl5fPvtt8TExJCWlkZUVBSTJ0+mV69e/g7NGNPEeZOgForIK0BbEbkd+DnwmhfbldcVrnR94n3A8yJyC7AK+BHI93JbROQO4A6AyMhIL0IyRfLz81mzZg2ff/45J06coHfv3tx66602TLYxpt6oMkGp6tMichlwEugHPKyqy7zYdyrQ1WM6Athfat/7gWkAItIauEZV00QkFRhTatsV5cT2KvAqOG1QXsTU5BUUFLBu3To+//xzjh07Ro8ePZg1axb9+vWz7vXGmHql0gTldnSIUdVxgDdJydMGoI+I9MApGc0Abii1/3DguKoWAr/H6dEHEAM8LiJFN8mMd5ebaiosLGTDhg0sWbKEI0eOEBkZyYwZMxgwYIAlJmNMvVRpglLVAhE5JSJtVDXtdHasqvkicjdOsgkE5qnqNhGZC8Sq6mKcUtJf3fatVcBd7rbHReTPOEkOYG5Rhwlz+uLj4/nggw84dOgQERER3HnnnQwaNMgSkzGmXvOmm/lCYAROCaq4m5uq3uPb0E5PbY6o25hs3bqVV199lfDwcK688krOOeccGx7bGONXtdnN/DP3ZRqYbdu28dprr9GlSxfuueeeej9+kjHGePKmk8TbItIMKOrelaCqeZVtY/xvx44dvPLKK3Tq1InZs2dbcjLGNDhVJigRGQO8DaTgdP/uKiI3q+oq34ZmqishIYGXX36Zjh07Mnv2bBu2whjTIHlTxff/gPGqmgAgIn1xnujQNO9srecSExN58cUX6dChA/fcc489nNUY02B501oeXJScAFQ1EQj2XUimupKTk3nxxRcJDw/n3nvvJTTUmydSGWNM/eRNCSpWRN4A3nWnb8R5eKupR3bu3MkLL7xAWFgY99xzjyUnY0yD502C+h+c+5PuwWmDWgW86MugzOnZvXs3L7zwAm3atOHee++lTZs2/g7JGGNqzJsEFQT8Q1X/BsVPl2ju06iM11JSUnjuuecIDQ1lzpw5tG3b1t8hGWNMrfCWR0ZpAAAgAElEQVSmDeoroIXHdAvgS9+EY07H3r17ee6552jVqhX33nuvJSdjTKPiTYIKUdWMogn3vd1U42f79u3j2WefpUWLFsyZM4d27dr5OyRjjKlV3iSoTBE5r2hCRIYCWb4LyVQlNTWVZ599lubNmzNnzhzat2/v75CMMabWedMGNQf4l4gUDZXRCZjuu5BMZfbv38+zzz5LcHAwc+bMITw83N8hGWOMT3jzqKMNIhKFMxaUAPH2qCP/SElJ4aWXXiIwMJA5c+bQoUMHf4dkjDE+U2GCEpHzgX2qelBV89xqvmuAPSLyiA1/UXeysrJYvHgxq1atKu5KfuaZZ/o7LGOM8anKSlCvAOMARORi4AlgNjAEZxTba30eXROnqmzcuJEPP/yQ9PR0Ro8ezRVXXEGLFi2q3tgYYxq4yhJUoEcpaTrwqqp+BHwkInG+D61pO3z4MAsWLCA+Pp7IyEh+9atfERkZ6e+wjDGmzlSaoEQkSFXzgUuBO7zcztRAXl4eS5cuJSYmhqCgIKZPn85FF11kgwwaY5qcyhLNfGCliBzF6Va+GkBEegOnNfy78U58fDwLFizg8OHDREdHc80119hji4wxTVaFCUpVHxORr3C6lS/V/44NH4DTFmVqSVpaGh9//DEbNmygQ4cOzJ49m/79+/s7LGOM8atKq+pUdW058xJ9F07TUlhYyOrVq1m8eDF5eXlMmjSJyy+/nOBgG83EGGOsLclP9u7dy/z589mzZw9RUVFMnz6djh07+jssY4ypNyxB+cGBAwd46qmnaN26NbfeeivR0dGIiL/DMsaYeqXKBCUidwPvqepPdRBPk/DDDz9QWFjI7373O3vIqzHGVMCbvstnARtEZKGITBC71K+xxMREOnbsaMnJGGMqUWWCUtWHgD7AG8AtQJKIPC4ivXwcW6NUUFDAzp076du3r79DMcaYes2ruz/dLuYH3Vc+EAZ8KCJPVbadW+JKEJFkEXmwnOWRIrJcRDaJyBYRmeTO7y4iWSIS575ePu0jq6f27dtHdna2JShjjKmCN21Q9wA3A0eB14H73YfHBgBJwO8q2C4QeAG4DEjFqSZcrKrbPVZ7CFioqi+JyABgCdDdXbZTVYdU77Dqr8REp5d+nz59/ByJMcbUb9704gsHpqnqHs+ZqlooIlMq2W4YkKyquwBEZAFwFeCZoBQ4w33fBthPI5eYmEinTp0444wzql7ZGGOaMG+q+JYAxUNriEioiAwHUNUdlWzXBdjnMZ3qzvP0CHCTiKS6n+P5hIoebtXfShG5qLwPEJE7RCRWRGKPHDnixaH4l7U/GWOM97xJUC8BGR7Tme68qpTX209LTV8PvKWqEcAk4F236vAAEKmq5wK/Ad4XkTJFDlV9VVWjVTW6IQzet2fPHnJycqx6zxhjvOBNghKP5/ChqoV4VzWYCnT1mI6gbBXeL4CF7n7XACFAuKrmqOoxd/5GYCfQ4IsdRe1PVoIyxpiqeZOgdonIPSIS7L7uBXZ5sd0GoI+I9BCRZsAMYHGpdfbiDOWBiPTHSVBHRKSD28kCEemJ083dm8+s1xITE+ncuTOtW7f2dyjGGFPveZOg7gRGAj/ilIqGU3JsqHK540jdDcQAO3B6620TkbkicqW72m+B20VkM87wHre4pbWLgS3u/A+BOxv6EPP5+fns2rXLSk/GGOOlKqvqVPUwTunntKnqEpzOD57zHvZ4vx24sJztPgI+qs5n1ld79uwhNzfXEpQxxnjJm/ugQnDaigbiVMEBoKo/92FcjU5iYiIiYh0kjDHGS95U8b2L8zy+y4GVOJ0d0n0ZVGOUmJhIly5daNWqlb9DMcaYBsGbBNVbVf8IZKrq28BkYJBvw2pc8vLyrP3JGGNOkzcJKs/9eUJEzsZ54kN3n0XUCKWkpJCXl2fVe8YYcxq8uZ/pVREJw3lu3mKgNfBHn0bVyFj7kzHGnL5KE5T7VIeT7mCFq4CedRJVI5OYmEhERAQtW7b0dyjGGNNgVFrF5z414u46iqVRysvLY/fu3db+ZIwxp8mbNqhlInKfiHQVkXZFL59H1kjs2rWL/Px8S1DGGHOavGmDKrrf6S6PeYpV93klKSkJEaF3797+DsUYYxoUb54k0aMuAmmsEhMT6dq1Ky1atPB3KMYY06B48ySJWeXNV9V3aj+cxiU3N5fdu3dzySWX+DsUY4xpcLyp4jvf430IztPHvwcsQVVh165dFBQU0K9fP3+HYowxDY43VXyeo9wiIm1wHn9kqpCYmEhAQAC9evXydyjGGNPgeNOLr7RTOOMzmSokJSURGRlJSEhI1SsbY4wpwZs2qP/w36HaA4ABuKPgmorl5OSQkpLCpZde6u9QjDGmQfKmDeppj/f5wB5VTfVRPI3Gzp07KSgosPufjDGmmrxJUHuBA6qaDSAiLUSku6qm+DSyBi4pKYmAgAB69rTbxYwxpjq8aYP6F1DoMV3gzjOVSExMpFu3btb+ZIwx1eRNggpS1dyiCfd9M9+F1PBlZ2ezZ88eq94zxpga8CZBHRGRK4smROQq4KjvQmr4du7cSWFhod3/ZIwxNeBNG9SdwHsi8rw7nQqU+3QJ40hMTCQwMNDan4wxpga8uVF3JzBCRFoDoqrpvg+rYUtMTKR79+40a2Y1ocYYU11VVvGJyOMi0lZVM1Q1XUTCROQvdRFcQ5SVlcXevXut/ckYY2rImzaoiap6omjCHV13ku9CatiSk5NRVUtQxhhTQ94kqEARaV40ISItgOaVrF9MRCaISIKIJIvIg+UsjxSR5SKySUS2iMgkj2W/d7dLEJHLvfm8+iAxMZGgoCB69LBRSowxpia86STxT+ArEXkT55FHP8eLJ5mLSCDwAnAZTseKDSKyWFW3e6z2ELBQVV8SkQHAEqC7+34GMBDoDHwpIn1VteA0js0vkpKSrP3JGGNqQZUlKFV9CvgL0B8nYfxZVZ/0Yt/DgGRV3eXeO7UAuKr07oEz3PdtgP3u+6uABaqao6q7gWR3f/XaqVOn2Ldvn1XvGWNMLfDqaeaq+oWq3qeqvwUyROQFLzbrAuzzmE5153l6BLhJRFJxSk9FQ3t4sy0icoeIxIpI7JEjR7w5FJ+y9idjjKk9XiUoERkiIk+KSApOaSrem83Kmaelpq8H3lLVCJyOF++KSICX26Kqr6pqtKpGd+jQwYuQfMvan4wxpvZU2AYlIn1x2oGuB44BH+DcB+Xt+OWpQFeP6Qj+W4VX5BfABABVXSMiIUC4l9vWO4mJifTs2ZPg4GB/h2KMMQ1eZSWoeJzh3a9Q1VGq+hzOg2K9tQHoIyI9RKQZTrJbXGqdve5nICL9cYaUP+KuN0NEmotID5wBEtefxmfXuczMTH788Uer3jPGmFpSWS++a3CSynIR+QKnk0N5VW/lUtV8EbkbiAECgXmquk1E5gKxqroY+C3wmoj8GqcK7xZVVWCbiCwEtuOMQXVXfe/Bl5SUZO1PxhhTiypMUKq6CFgkIq2AqcCvgY4i8hKwSFWXVrVzVV2C0/nBc97DHu+3AxdWsO1jwGPeHER9kJiYSHBwMN26dfN3KMYY0yh40808U1XfU9UpOG1BcUCZm26busTERHr16mXtT8YYU0u86sVXRFWPq+orqjrWVwE1RBkZGezfv58+ffr4OxRjjGk0TitBmfIlJSUBWPuTMcbUIktQtSAxMZFmzZpZ+5MxxtQiS1C1oKj9KSjIm0cbGmOM8YYlqBpKT0/nwIEDVr1njDG1zBJUDVn7kzHG+IYlqBpKSEigefPmREZG+jsUY4xpVCxB1VBiYiK9e/cmMDDQ36EYY0yjYgmqBk6cOMGhQ4fs/idjjPEBS1A1kJCQAED//v39HIkxxjQ+lqBqYMeOHbRu3ZouXcqMpWiMMaaGLEFVk6qSkJBAv379CAiw02iMMbXNvlmr6eDBg6SlpREVFeXvUIwxplGyBFVNO3bsALAEZYwxPmIJqpri4+Pp0KED7du393coxhjTKFmCqoaCggKSkpKs9GSMMT5kCaoadu/eTU5OjiUoY4zxIUtQ1RAfH4+I0K9fP3+HYowxjZYlqGpISEggMjKSli1b+jsUY4xptCxBnaasrCx2795t1XvGGONjlqBOU1JSEoWFhZagjDHGxyxBnab4+HiCg4Pp2bOnv0MxxphGzRLUaYqPj6d3794EBwf7OxRjjGnUfJqgRGSCiCSISLKIPFjO8r+LSJz7ShSREx7LCjyWLfZlnN46ceIEBw8etOo9Y4ypA0G+2rGIBAIvAJcBqcAGEVmsqtuL1lHVX3usPxs412MXWao6xFfxVUd8fDxgjzcyxpi64MsS1DAgWVV3qWousAC4qpL1rwfm+zCeGouPj7fhNYwxpo74MkF1AfZ5TKe688oQkW5AD+Brj9khIhIrImtFZGoF293hrhN75MiR2oq7XKpKfHw8UVFRNryGMcbUAV9+00o587SCdWcAH6pqgce8SFWNBm4AnhGRXmV2pvqqqkaranSHDh1qHnElDhw4wMmTJ+3pEcYYU0d8maBSga4e0xHA/grWnUGp6j1V3e/+3AWsoGT7VJ0ran+y4d2NMaZu+DJBbQD6iEgPEWmGk4TK9MYTkX5AGLDGY16YiDR334cDFwLbS29bl+Lj4znzzDNp166dP8Mwxpgmw2cJSlXzgbuBGGAHsFBVt4nIXBG50mPV64EFqupZ/dcfiBWRzcBy4AnP3n91rWh4DaveM8aYuuOzbuYAqroEWFJq3sOlph8pZ7vvgEG+jO10FA2vYdV7xhhTd3yaoBqLouE1+vbt6+9QjKkTeXl5pKamkp2d7e9QTAMWEhJCREREtZ+8YwnKC/Hx8XTr1s2G1zBNRmpqKqGhoXTv3h2R8jrkGlM5VeXYsWOkpqbSo0ePau3DbuipQlZWFikpKfb0CNOkZGdn0759e0tOptpEhPbt29eoFG4Jqgo2vIZpqiw5mZqq6d+QJagqxMfH06xZs2oXUY0xxlSPJagq7Nixw4bXMKaOHTt2jCFDhjBkyBDOOussunTpUjydm5vr1T5uvfVWEhISKl3nhRde4L333quNkI0PWCeJSvz0008cOnSICy+80N+hGNOktG/fnri4OAAeeeQRWrduzX333VdiHVVFVSt8Nuabb75Z5efcddddNQ/WB6o6tqbCElQliq6+rP3JNGVz5swpTha1ZciQITzzzDOnvV1ycjJTp05l1KhRrFu3jk8//ZRHH32U77//nqysLKZPn87DDzu3Wo4aNYrnn3+es88+m/DwcO68804+//xzWrZsyb///W/OPPNMHnroIcLDw5kzZw6jRo1i1KhRfP3116SlpfHmm28ycuRIMjMzmTVrFsnJyQwYMICkpCRef/11hgwpORrQ/fffz2effUZQUBATJ07kySef5ODBg/zyl79k9+7diAivvvoqw4cP56mnnuKdd94B4Je//CWzZ88u99i2bNnC3LlzycnJoU+fPsybN49WrVrV/BfQQDTt9FyFHTt2EBoaSufOnf0dijHGtX37dn7xi1+wadMmunTpwhNPPEFsbCybN29m2bJlbN9e9qEzaWlpjB49ms2bN3PBBRcwb968cvetqqxfv57/+7//Y+7cuQA899xznHXWWWzevJkHH3yQTZs2ldnu0KFDLFmyhG3btrFlyxZ+//vfA04J7bLLLmPLli1s3LiR/v37s379et577z3Wr1/PmjVrePHFF9myZUuZYwsODuaJJ57gq6++4vvvv2fw4MH84x//qK3T2CBYCaoCqkpCQgL9+vVr8sVs07RVp6TjS7169eL8888vnp4/fz5vvPEG+fn57N+/n+3btzNgwIAS27Ro0YKJEycCMHToUFavXl3uvqdNm1a8TkpKCgDffPMNDzzwAADnnHMOAwcOLLNdu3btCAgI4Pbbb2fy5MlMmTIFgBUrVrBgwQIAgoKCOOOMM1i9ejXXXHNN8X2VU6dO5ZtvvmH8+PElju27775j+/btjBw5EoDc3FxGjRp1+iesAbMEVYGi4TWses+Y+sWziispKYl//OMfrF+/nrZt23LTTTeVe99Ns2bNit8HBgaSn59f7r6bN29eZp2SjwktX3BwMLGxsSxbtowFCxbw0ksvsXTpUqBsV+vK9ud5bKrKhAkTePfdd6v8/MbKigYV2LFjB2DtT8bUZydPniQ0NJQzzjiDAwcOEBMTU+ufMWrUKBYuXAjA1q1by61CTE9P5+TJk0yZMoW///3vxdWAl1xyCS+//DLgPHT65MmTXHzxxSxatIisrCwyMjL497//zUUXXVRmnyNHjmTlypXs2rULgMzMTJKSkmr9+OozK0FVwIbXMKb+O++88xgwYABnn302PXv29EmP29mzZzNr1iwGDx7Meeedx9lnn02bNm1KrJOWlsa0adPIycmhsLCQv/3tbwA8//zz3H777bzyyisEBQXxyiuvMGzYMK6//vriqrz/+Z//YdCgQSQnJ5fYZ8eOHXnjjTeYPn16cdf6xx9/nD59+tT6MdZX4k3xtSGIjo7W2NjYWtlXfn4+999/P8OHD2fGjBm1sk9jGpIdO3bY0/td+fn55OfnExISQlJSEuPHjycpKYmgILu+90Z5f0sistEdMb1SdobLUTS8hlXvGWMyMjK49NJLyc/PR1WLS0PG9+wsl8OG1zDGFGnbti0bN270dxhNknWSKIcNr2GMMf5nCaqUrKws9uzZY/XvxhjjZ5agSklMTKSwsJB+/fr5OxRjjGnSLEGVYsNrGGNM/WAJqpT4+Hj69Oljw2sY42cHDx5kxowZ9OrViwEDBjBp0iQSExP9HVa5unfvztGjRwGKH01U2i233MKHH35Y6X7eeust9u/fXzx92223lXtjcFNhCcrD8ePHOXTokHUvN8bPVJWrr76aMWPGsHPnTrZv387jjz/OoUOHSqxXUFDgpwgr9t1331V729IJ6vXXXy/zXMH6oKJHRdU262buoWh4DWt/Mua//vWvf5Gamlqr+4yIiOC6666rcPny5csJDg7mzjvvLJ5XNLzFihUrePTRR+nUqRNxcXFs376dv/3tb8VPKL/tttuYM2cOmZmZ/OxnPyM1NZWCggL++Mc/Mn36dB588EEWL15MUFAQ48eP5+mnny7x2S+99BK7d+/mqaeeApyksXHjRp577jmmTp3Kvn37yM7O5t577+WOO+4oE3vr1q3JyMhAVZk9ezZff/01PXr0KPEMvrlz5/Kf//yHrKwsRo4cySuvvMJHH31EbGwsN954Iy1atGDNmjVMnDiRp59+mujoaObPn8/jjz+OqjJ58mSefPLJ4s+79957+fTTT2nRogX//ve/6dixY4mYVq5cyb333gs4zwZctWoVoaGhPPXUU7z77rsEBAQwceJEnnjiCeLi4rjzzjs5deoUvXr1Yt68eYSFhTFmzBhGjhzJt99+y5VXXsmsWbO488472bt3L+A8VLi2n+RhCcpDfHw8oaGhdOnSxd+hGNOk/fDDDwwdOrTC5evXr+eHH36gR48ebNy4kTfffJN169ahqgwfPpzRo0eza9cuOnfuzGeffQY4jyM6fvw4ixYtKr7X8cSJE2X2fe2113LBBRcUJ6gPPviAP/zhDwDMmzePdu3akZWVxfnnn88111xD+/bty41x0aJFJCQksHXrVg4dOsSAAQP4+c9/DsDdd99dPG7VzJkz+fTTT7n22mt5/vnnixOSp/379/PAAw+wceNGwsLCGD9+PJ988glTp04lMzOTESNG8Nhjj/G73/2O1157jYceeqjE9k8//TQvvPACF154IRkZGYSEhPD555/zySefsG7dOlq2bMnx48cBmDVrFs899xyjR4/m4Ycf5tFHHy1+ov2JEydYuXIlADfccAO//vWvGTVqFHv37uXyyy8vfoZpbfFpghKRCcA/gEDgdVV9otTyvwOXuJMtgTNVta277Gag6Cz/RVXf9mWsqkp8fDxRUVFlnj5sTFNWWUnHX4YNG1bckembb77h6quvLn4S+LRp01i9ejUTJkzgvvvu44EHHmDKlClcdNFFxY8suu2220oMi+GpQ4cO9OzZk7Vr19KnTx8SEhKKSwbPPvssixYtAmDfvn0kJSVVmKBWrVrF9ddfT2BgIJ07d2bs2LHFy5YvX85TTz3FqVOnOH78OAMHDuSKK66o8Hg3bNjAmDFj6NChAwA33ngjq1atYurUqTRr1qz4OIYOHcqyZcvKbH/hhRfym9/8hhtvvJFp06YRERHBl19+ya233lp8v2e7du1IS0vjxIkTjB49GoCbb765xO9/+vTpxe+//PLLEu1jJ0+eJD09ndDQ0AqP43T5rA1KRAKBF4CJwADgehEpUZmqqr9W1SGqOgR4DvjY3bYd8CdgODAM+JOIhPkqVnCuUNLT0639yZh6YODAgZU+vaH0sBTl6du3Lxs3bmTQoEH8/ve/Z+7cuQQFBbF+/XquueYaPvnkEyZMmEBBQQFDhgxhyJAhxaWa6dOns3DhQj766COuvvpqRIQVK1bw5ZdfsmbNGjZv3sy5555b7tAensq72M3OzuZXv/oVH374IVu3buX222+vcj+VPTM1ODi4+HMqGkrkwQcf5PXXXycrK4sRI0YQHx+Pqp72xbjneS8sLGTNmjXExcURFxfHjz/+WKvJCXzbSWIYkKyqu1Q1F1gAXFXJ+tcD8933lwPLVPW4qv4ELAMm+DBW4uPjAWt/MqY+GDt2LDk5Obz22mvF8zZs2FBcveTp4osv5pNPPuHUqVNkZmayaNEiLrroIvbv30/Lli256aabuO+++/j+++/JyMggLS2NSZMm8cwzzxAXF0dgYGDxl2zRKLrTpk3jk08+Yf78+cWlhrS0NMLCwmjZsiXx8fGsXbu20mO4+OKLWbBgAQUFBRw4cIDly5cDFCej8PBwMjIySvTsCw0NJT09vcy+hg8fzsqVKzl69CgFBQXMnz+/uJTjjZ07dzJo0CAeeOABoqOjiY+PZ/z48cybN49Tp04BTiexNm3aEBYWVjyg47vvvlvh54wfP57nn3++eDouLs7reLzlyyq+LsA+j+lUnBJRGSLSDegBfF3JtmUahkTkDuAOgMjIyBoFGx8fT8eOHW14DWPqARFh0aJFzJkzhyeeeIKQkBC6d+/OM888w48//lhi3fPOO49bbrmFYcOGAU4niXPPPZeYmBjuv/9+AgICCA4O5qWXXiI9PZ2rrrqK7OxsVJW///3v5X5+WFgYAwYMYPv27cX7nTBhAi+//DKDBw+mX79+jBgxotJjuPrqq/n6668ZNGgQffv2Lf6ib9u2LbfffjuDBg2ie/fuJUYHvuWWW7jzzjuLO0kU6dSpE3/961+55JJLUFUmTZrEVVdVdr1f0jPPPMPy5csJDAxkwIABTJw4kebNmxMXF0d0dDTNmjVj0qRJPP7447z99tvFnSR69uzJm2++We4+n332We666y4GDx5Mfn4+F198cfHYV7XFZ8NtiMh1wOWqeps7PRMYpqqzy1n3ASCiaJmI3A80V9W/uNN/BE6p6v+r6PNqOtzG119/TUBAAGPGjKn2PoxpLGy4DVNb6utwG6lAV4/pCGB/BevOAO4qte2YUtuuqMXYyvBswDTGGON/vmyD2gD0EZEeItIMJwktLr2SiPQDwoA1HrNjgPEiEuZ2jhjvzjPGGNNE+KwEpar5InI3TmIJBOap6jYRmQvEqmpRsroeWKAedY2qelxE/oyT5ADmqupxX8VqjCmrOr28jPFU0yYkG/LdGFPG7t27CQ0NpX379pakTLWoKseOHSM9Pb3Mw7frQxuUMaaBioiIIDU1lSNHjvg7FNOAhYSEEBERUe3tLUEZY8oIDg62IWeM39nTzI0xxtRLlqCMMcbUS5agjDHG1EuNphefiBwB9nixajhw1MfhNDR2Tsqyc1KWnZOy7JyU5O356KaqHapaqdEkKG+JSKw33RubEjsnZdk5KcvOSVl2Tkqq7fNhVXzGGGPqJUtQxhhj6qWmmKBe9XcA9ZCdk7LsnJRl56QsOycl1er5aHJtUMYYYxqGpliCMsYY0wBYgjLGGFMvNakEJSITRCRBRJJF5EF/x+MPIjJPRA6LyA8e89qJyDIRSXJ/hvkzxrokIl1FZLmI7BCRbSJyrzu/KZ+TEBFZLyKb3XPyqDu/h4isc8/JB+44b02KiASKyCYR+dSdbtLnRERSRGSriMSJSKw7r9b+d5pMghKRQOAFYCIwALheRAb4Nyq/eAuYUGreg8BXqtoH+Mqdbirygd+qan9gBHCX+3fRlM9JDjBWVc8BhgATRGQE8CTwd/ec/AT8wo8x+su9wA6PaTsncImqDvG4/6nW/neaTIIChgHJqrpLVXOBBcBVfo6pzqnqKqD04I9XAW+7798GptZpUH6kqgdU9Xv3fTrOl08XmvY5UVXNcCeD3ZcCY4EP3flN6pwAiEgEMBl43Z0Wmvg5qUCt/e80pQTVBdjnMZ3qzjPQUVUPgPOFDZzp53j8QkS6A+cC62ji58StyooDDgPLgJ3ACVXNd1dpiv8/zwC/Awrd6fbYOVFgqYhsFJE73Hm19r/TlMaDKm9YUOtjbwAQkdbAR8AcVT3Z1EeRVdUCYIiItAUWAf3LW61uo/IfEZkCHFbVjSIypmh2Oas2mXPiulBV94vImcAyEYmvzZ03pRJUKtDVYzoC2O+nWOqbQyLSCcD9edjP8dQpEQnGSU7vqerH7uwmfU6KqOoJYAVO+1xbESm6qG1q/z8XAleKSApO88BYnBJVUz4nqOp+9+dhnAuZYdTi/05TSlAbgD5ur5tmwAxgsZ9jqi8WAze7728G/u3HWOqU247wBrBDVf/msagpn5MObskJEWkBjMNpm1sOXOuu1qTOiar+XlUjVLU7znfH16p6I034nIhIKxEJLXoPjAd+oBb/d5rUkyREZBLOVU8gME9VH/NzSHVOROYDY3Aeiz4H+lQAAAQrSURBVH8I+BPwCbAQiAT2AtepaumOFI2SiIwCVgNb+W/bwv/itEM11XMyGKdxOxDnInahqs4VkZ44pYd2wCbgJlXN8V+k/uFW8d2nqlOa8jlxj32ROxkEvK+qj4lIe2rpf6dJJShjjDENR1Oq4jPGGNOAWIIyxhhTL1mCMsYYUy9ZgjLGGFMvWYIyxhjz/9u7v9CqyziO4+/PKrqo2I3QRV1EK7OSSrMFXfQHQijCIIqiIAgDbWgYFHQhESUxGnoTXVjKYkEXu7PCWCEiIZXCGrMIS6fXBUE0yUT27eL5nnh22vSsnegH5/OCsZ3n9+d8dzG+e37P9nkayQ3KepakkLSzev2ypNe7dO8PJD1+8TM7vl+/pDFJJ/NjTFJ/dXwkk8dHqrHnMmV6StK5KnV6uFt1mf2X3KCsl/0JPCZpxf9dSC2T99vtBWYiYiAiBoBTZGhp2gSsjYhXWgMRMZop03dQEg5aqdPz0qWrJASzRnGDsl52HngPeKn9QPsMSNJsfr5f0iFJ45J+lDQs6ZncP+mYpIHqNg9K+jLPeySvvyRnO0clTUvaVN33oKSPKP80XNdyA3An8GY1/AawTtKApI+BK4BvJD3ZyTcuaYek3ZK+AEYlXSppV34f05Ker859tRp/LceukvSZyp5R33VztmjW4t+crNe9C0xLensJ19xOCU/9FZgB9kTEoMpmh1uBbXnedcB9wABwMBvNs8BvEXGXpMuBw5I+z/MHgdURcart/W4BpjLAFShhrpk2fmtEbJA0mzOlpVgD3BsRZyUNUcJQB7Our7Ou1ZREgLsp4aj7Jd1DybU8HREPQXkEucT3NrsoNyjraZlcPga8CPzR4WVHW9sJSDoJtBrMMeCB6rzxiJgDfpI0A6yi5JXdVs04+oEbgXPAkQWaE5TGsFDky2LjndoXEWfz6/XAzZKeaqtrPWWTz29z/EpgJSUKajjXsz6JiMPLqMNsQW5QZiWfcRIYrcbOk4/AM1C23sq7zlqbq17PMf9nqr15BKWpbI2IifpA5rudWaS+74E1kvqy4SGpjzKT+2GRazpRv5+AoYg40FbXBmBHROxtv1jSOuBhYETSpxHx1jJqMfsHr0FZz8sgy3Hmb9d9mrLuA2WH0Mv+xa2fkNSX61LXA8eBCeCF3OIDSSszCfpC9Z2gzGC2V8Pbgck81g0TwFDrDyYk3ZRJ5hPAxlaNkq6VtELSNcBsRHwI7ALWdqkOs795BmVW7AS2VK/fB/ZJOgIcYPHZzYUcBw4BVwObc61nD2VtajJnZr/Q2ZbYG4F3JJ2gzHa+Yn5DXa7dlLWmqVIWPwOPRsR+Sasoa1IAvwNPU9bFhiXNUR5Pbu5iLWaA08zNzKyh/IjPzMwayQ3KzMwayQ3KzMwayQ3KzMwayQ3KzMwayQ3KzMwayQ3KzMwa6S8OrM5zF99l8wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf20d0e80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_mean = np.mean(train_scores, axis=1)\n",
    "train_std = np.std(train_scores, axis=1)\n",
    "\n",
    "test_mean = np.mean(test_scores, axis=1)\n",
    "test_std = np.std(test_scores, axis=1)\n",
    "\n",
    "plt.plot(param_range, train_mean, label=\"Training score\", color=\"black\")\n",
    "plt.plot(param_range, test_mean, label=\"Cross-validation score\", color=\"dimgrey\")\n",
    "\n",
    "plt.title(\"Validation Curve With Random Forest\")\n",
    "plt.xlabel(\"Number Of Trees\")\n",
    "plt.ylabel(\"Accuracy Score\")\n",
    "plt.tight_layout()\n",
    "plt.legend(loc=\"best\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Learning Curves\n",
    "* Learning curves shows variation in training & validation score on increasing the number of samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import learning_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}