Cross Validation and Hyperparameter Tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Cross Validation and Why should i cross validate my model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally in a machine learning process, data is divided into training and test sets; the training set is then used to train the model and the test set is used to evaluate the performance of a model. However, this approach may lead to variance problems. In simpler words, a variance problem refers to the scenario where our accuracy obtained on one test is very different to accuracy obtained on another test set using the same algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words, we cant be sure that the model will have the desired accuracy and variance in production environment. We need some kind of assurance of the accuracy of the predictions that our model is putting out. For this, we need to validate our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train_Test Split approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many are only familia with the popular train test split, where a certain percentage of the data is kept aside for validation,\n",
    "in this case we randomly split the complete data into training and test sets. Then Perform the model training on the training set and use the test set for validation purpose, ideally split the data into 70:30 or 80:20. With this approach there is a possibility of high bias if we have limited data, because we would miss some information about the data which we have not used for training.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image](images/1-16.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We keep using the same random 20% for testing, what if its just by chance or luck the model is able to predict correctly on this 20%, We really need to be sure that same result will be gotten on an entirely differant data (i.e the model generalizes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K-Folds Cross Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K-Fold is a popular and easy to understand, it generally results in a less biased model compare to other methods. Because it ensures that every observation from the original dataset has the chance of appearing in training and test set. This is one among the best approach if we have a limited input data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*How does it work*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](images/cv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Split the entire data randomly into k folds (value of k shouldn’t be too small or too high, ideally we choose 5 to 10 depending on the data size)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Then fit the model using the K-1 (K minus 1) folds and validate the model using the remaining Kth fold. Note down the scores/errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Repeat this process until every K-fold serve as the test set. Then take the average of your recorded scores. That will be the performance metric for the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cross validation in English, what do we mean...\n",
    "Lets say we have a dataset of 100 rows(samples)..and we pick k(the number of folds) to be 5.\n",
    "meaning each fold will have 20 rows each [20 - 20 - 20 - 20 - 20].\n",
    "\n",
    "As in the picture above..for the first iteration we take k - 1 folds (5 - 1 = 4) i.e 4 folds(80 rows) for training then tests on the fold we left aside (20 row).\n",
    "\n",
    "For the next iteration we take another set of 20 rows for testing, training on the rest..\n",
    "It continues till we have completed all the folds...\n",
    "easy right !!!!\n",
    "by doing this we have succeeded in train on all our data and tested on all the data\n",
    "\n",
    "NB: The score for each iteration is recorded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can decide to code this manually, as per Bad guy  !!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or Use the inbuilt function in scikit learn 'cross_val_score'....No time !!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll be Using these Algorithms \n",
    "- RandomForestClassifier\n",
    "- GradientBoostingClassifier\n",
    "- DecisionTreeClassifier\n",
    "- LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\paul\\Anaconda3\\anaconda\\lib\\importlib\\_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "rfc = RandomForestClassifier()\n",
    "gbc = GradientBoostingClassifier()\n",
    "lr = LogisticRegression()\n",
    "dsc = DecisionTreeClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First Lets Use the Train_test_split approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\paul\\Anaconda3\\anaconda\\lib\\importlib\\_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "data = pd.read_csv('data/clean_train.csv') #importing my already preprocessed data\n",
    "y = data['Claim'] #defining my target variable\n",
    "X = data.drop(['Claim','Customer Id'], axis = 1) #defining my input data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "                       max_features=None, max_leaf_nodes=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, presort=False,\n",
       "                       random_state=None, splitter='best')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfc.fit(X_train, y_train) #training the Randomforestclassifier\n",
    "gbc.fit(X_train, y_train)#training the gradientboostingclassifier\n",
    "lr.fit(X_train, y_train)\n",
    "dsc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#predicting on the test set\n",
    "rfc_pred = rfc.predict_proba(X_test)[:,-1]\n",
    "gbc_pred = gbc.predict_proba(X_test)[:,-1]\n",
    "lr_pred = lr.predict_proba(X_test)[:,-1]\n",
    "dsc_pred = dsc.predict_proba(X_test)[:,-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#evaluation metric for submission\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Algorithm</th>\n",
       "      <th>roc_auc_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.648452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>0.720327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>LinearRegression</td>\n",
       "      <td>0.716450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DecisionTreeClassifier</td>\n",
       "      <td>0.593968</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Algorithm  roc_auc_score\n",
       "0      RandomForestClassifier       0.648452\n",
       "1  GradientBoostingClassifier       0.720327\n",
       "2            LinearRegression       0.716450\n",
       "3      DecisionTreeClassifier       0.593968"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfc_score = roc_auc_score(y_test,rfc_pred)\n",
    "gbc_score = roc_auc_score(y_test,gbc_pred)\n",
    "lr_score = roc_auc_score(y_test,lr_pred)\n",
    "dsc_score = roc_auc_score(y_test,dsc_pred)\n",
    "\n",
    "dict_ = {'Algorithm': ['RandomForestClassifier','GradientBoostingClassifier', \n",
    "                       'LinearRegression','DecisionTreeClassifier'],\n",
    "         'roc_auc_score':[rfc_score, gbc_score, lr_score,dsc_score]}\n",
    "df = pd.DataFrame(dict_, index=[0,1,2,3])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GradientBoostingClassifier and LogisticRegression looks promising, and their Scores are not too far apart`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But Which of the two will i Select as my model: \n",
    "    cross validation will help us out here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### k fold cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbc_cv = cross_val_score(gbc,X,y,scoring='roc_auc', cv = 5)#k=5, meaning 5 fold cross validation\n",
    "lr_cv = cross_val_score(lr,X,y,scoring='roc_auc', cv = 5)#k=5, meaning 5 fold cross validation\n",
    "gbc_cv = [i.round(3) for i in gbc_cv]\n",
    "lr_cv = [i.round(3) for i in lr_cv]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_ = {'algorithm':['GradientBoostingClassifier','LogisticRegression'],\n",
    "         'cv_score':[gbc_cv,lr_cv],\n",
    "          'cv_mean': [np.mean(gbc_cv),np.mean(lr_cv)],\n",
    "         'cv_std': [np.std(gbc_cv),np.std(lr_cv)]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>algorithm</th>\n",
       "      <th>cv_score</th>\n",
       "      <th>cv_mean</th>\n",
       "      <th>cv_std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GradientBoostingClassifier</td>\n",
       "      <td>[0.726, 0.727, 0.714, 0.697, 0.691]</td>\n",
       "      <td>0.7110</td>\n",
       "      <td>0.014738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>LogisticRegression</td>\n",
       "      <td>[0.721, 0.721, 0.713, 0.694, 0.694]</td>\n",
       "      <td>0.7086</td>\n",
       "      <td>0.012274</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    algorithm                             cv_score  cv_mean  \\\n",
       "0  GradientBoostingClassifier  [0.726, 0.727, 0.714, 0.697, 0.691]   0.7110   \n",
       "1          LogisticRegression  [0.721, 0.721, 0.713, 0.694, 0.694]   0.7086   \n",
       "\n",
       "     cv_std  \n",
       "0  0.014738  \n",
       "1  0.012274  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(dict_, index = [0,1])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now We know that GradientBoosting Classifier always perform better\n",
    "than LogisticRegression on Unseen data..Its evident from our cv_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Next ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we still improve on our current score ?.. \n",
    "\n",
    "I think there is still more juice to squeeze out of GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we can find the right combination of hyperparameters, Hopefully our model should perform better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameter/Hyperparameter.. Whats the difference?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A machine learning model has two types of parameters. The first type of parameters are the parameters that are learned through a machine learning model while the second type of parameters are the hyper parameter that we pass to the machine learning model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
      "                           learning_rate=0.1, loss='deviance', max_depth=3,\n",
      "                           max_features=None, max_leaf_nodes=None,\n",
      "                           min_impurity_decrease=0.0, min_impurity_split=None,\n",
      "                           min_samples_leaf=1, min_samples_split=2,\n",
      "                           min_weight_fraction_leaf=0.0, n_estimators=100,\n",
      "                           n_iter_no_change=None, presort='auto',\n",
      "                           random_state=None, subsample=1.0, tol=0.0001,\n",
      "                           validation_fraction=0.1, verbose=0,\n",
      "                           warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "#For GradientBoostingClassifer we used the default hyperparameters\n",
    "print(gbc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally we randomly set the value for these hyper parameters and see what parameters result in best performance. However randomly selecting the parameters for the algorithm can be exhaustive.\n",
    "\n",
    "Also, it is not easy to compare performance of different algorithms by randomly setting the hyper parameters because one algorithm may perform better than the other with different set of parameters. And if the parameters are changed, the algorithm may perform worse than the other algorithms.\n",
    "\n",
    "Therefore, instead of randomly selecting the values of the parameters, a better approach would be to develop an algorithm which automatically finds the best parameters for a particular model. Grid Search is one such algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grid Search with Sklearn "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already have a baseline model without tuning, i'll start with tuning n_estimator, learning_rate, max_depth(most people consider them to be the most important parameters).. \n",
    "\n",
    "\n",
    "- create dictionary of parameters\n",
    "- Define you GridSearch parameters\n",
    "- Fit on training data\n",
    "- get best parameter\n",
    "- Update gridsearch parameter by adding best parameter\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# p_test3 = {'learning_rate':[0.15,0.1,0.05,0.01,0.005,0.001], 'n_estimators':[100,250,500,750,1000,1250,1500,1750]}\n",
    "\n",
    "# tuning = GridSearchCV(estimator =GradientBoostingClassifier(max_depth=4, min_samples_split=2, min_samples_leaf=1, subsample=1,max_features='sqrt', random_state=10), \n",
    "#             param_grid = p_test3, scoring='accuracy',n_jobs=4,iid=False, cv=5)\n",
    "# tuning.fit(X_train,y_train)\n",
    "# tuning.grid_scores_, tuning.best_params_, tuning.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'learning_rate': 0.01, 'n_estimators': 600} 0.7155278708933107\n"
     ]
    }
   ],
   "source": [
    "#Create dictionary of parameters\n",
    "param = {'learning_rate':[0.1,0.02,0.05,0.5,0.01],\n",
    "           'n_estimators':[100,300,500,600,750,900,1000]}\n",
    "#Define GridSearch Parameters\n",
    "tuning = GridSearchCV(estimator =gbc, \n",
    "            param_grid = param, scoring='roc_auc',n_jobs=-1,iid=False, cv=5)\n",
    "#fit on  training data\n",
    "tuning.fit(X_train,y_train)\n",
    "\n",
    "#print to see best parameter and score\n",
    "print( tuning.best_params_, tuning.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Uncomment to see a breakdown of the result\n",
    "# pd.DataFrame(tuning.cv_result_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning Max depth "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_depth': 2} 0.7177232722449656\n"
     ]
    }
   ],
   "source": [
    "param = {'max_depth':[1,2,3,4,5,6,7] }\n",
    "\n",
    "#'learning_rate': 0.01, 'n_estimators': 600} is added to the gridsearch parameters\n",
    "tuning = GridSearchCV(estimator =GradientBoostingClassifier(n_estimators=600, learning_rate= 0.01), \n",
    "            param_grid = param, scoring='roc_auc',n_jobs=-1,iid=False, cv=5)\n",
    "tuning.fit(X_train,y_train)\n",
    "\n",
    "print(tuning.best_params_,tuning.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other factors\n",
    "Tree related parameters: Min sample split and min samples leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'min_samples_leaf': 5, 'min_samples_split': 2} 0.7181150388044063\n"
     ]
    }
   ],
   "source": [
    "param = {'min_samples_split':[2,4,6,8,10],\n",
    "         'min_samples_leaf':[1,3,5,7,9]}\n",
    "\n",
    "#add the new max depth parameter\n",
    "tuning = GridSearchCV(estimator =GradientBoostingClassifier(learning_rate=0.01, n_estimators=600,max_depth=2), \n",
    "            param_grid = param, scoring='roc_auc',n_jobs=-1,iid=False, cv=5)\n",
    "tuning.fit(X_train,y_train)\n",
    "\n",
    "print(tuning.best_params_,tuning.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "# param = {'max_features':[9,10,12,13,17,20,22]}\n",
    "# tuning = GridSearchCV(estimator =GradientBoostingClassifier(learning_rate=0.01, n_estimators=600,max_depth=2, min_samples_split=2,\n",
    "#                                                             min_samples_leaf=5), \n",
    "#                             param_grid = param, scoring='roc_auc',n_jobs=-1,iid=False, cv=5)\n",
    "# tuning.fit(X_train,y_train)\n",
    "\n",
    "# print(tuning.best_score_,tuning.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7181150388044063 {'subsample': 1}\n"
     ]
    }
   ],
   "source": [
    "param= {'subsample':[0.4,0.5,0.55,0.6,0.65,0.7,0.8,1]}\n",
    "\n",
    "tuning = GridSearchCV(estimator =GradientBoostingClassifier(learning_rate=0.01, n_estimators=600,max_depth=2, \n",
    "                                                            min_samples_split=2, min_samples_leaf=5,random_state=10), \n",
    "param_grid = param, scoring='roc_auc',n_jobs=-1,iid=False, cv=5)\n",
    "tuning.fit(X_train,y_train)\n",
    "\n",
    "print(tuning.best_score_,tuning.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                           learning_rate=0.01, loss='deviance', max_depth=2,\n",
       "                           max_features=None, max_leaf_nodes=None,\n",
       "                           min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                           min_samples_leaf=5, min_samples_split=2,\n",
       "                           min_weight_fraction_leaf=0.0, n_estimators=600,\n",
       "                           n_iter_no_change=None, presort='auto',\n",
       "                           random_state=10, subsample=1, tol=0.0001,\n",
       "                           validation_fraction=0.1, verbose=0,\n",
       "                           warm_start=False)"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuning.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now lets see the performance of our tuned model using 5-fold cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuned_gbc_cv = cross_val_score(tuning, X, y, scoring = 'roc_auc', cv =5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_gbc_cv = cross_val_score(gbc, X, y, scoring = 'roc_auc', cv =5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rounding up scores to 4 decimal places\n",
    "tuned_gbc_cv = [i.round(4) for i in tuned_gbc_cv]\n",
    "default_gbc_cv  = [i.round(4) for i in default_gbc_cv]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_ = {'model':['default_model','Tuned_model'],\n",
    "         'cv_score':[default_gbc_cv,tuned_gbc_cv],\n",
    "          'cv_mean': [np.mean(default_gbc_cv),np.mean(tuned_gbc_cv)],\n",
    "         'cv_std': [np.std(default_gbc_cv),np.std(tuned_gbc_cv)]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing both Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>cv_score</th>\n",
       "      <th>cv_mean</th>\n",
       "      <th>cv_std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>default_model</td>\n",
       "      <td>[0.7263, 0.7268, 0.7135, 0.6971, 0.6902]</td>\n",
       "      <td>0.71078</td>\n",
       "      <td>0.014937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tuned_model</td>\n",
       "      <td>[0.7255, 0.7339, 0.7192, 0.704, 0.6927]</td>\n",
       "      <td>0.71506</td>\n",
       "      <td>0.014855</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           model                                  cv_score  cv_mean    cv_std\n",
       "0  default_model  [0.7263, 0.7268, 0.7135, 0.6971, 0.6902]  0.71078  0.014937\n",
       "1    Tuned_model   [0.7255, 0.7339, 0.7192, 0.704, 0.6927]  0.71506  0.014855"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(dict_, index = [0,1])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion:\n",
    "Though the improvement looks insignificant (in some ML competitions 0.0001 improvement can mean a lot)\n",
    "you can always increase the range of values of parameters,but note that the more values you have the longer it \n",
    "takes to run gridsearch, "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another cool option is RandomSearch..You could Check that up yourself"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}