Module-5/day-5-pipelines-boosting/boosting/Boosting_and_pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boosting and Pipelines\n",
    "\n",
    "### Objectives\n",
    "\n",
    "- Understanding how we can create a streamline of procedures (Pipelines)\n",
    "\n",
    "- What is the use of such practices.\n",
    "\n",
    "- Boosting methods - Specifically Gradient and Adaboost\n",
    "\n",
    "- Implementation of GradientBoostClassifier with fine-tuning with gridsearch.\n",
    "\n",
    "\n",
    "### Pipelines\n",
    "\n",
    "__Q:__ What is a pipeline?\n",
    "\n",
    "[sklearn - documentation](https://scikit-learn.org/stable/modules/compose.html#pipeline)\n",
    "\n",
    "> Transformers (scaling, preprocessing, feature selection etc.) are usually combined with classifiers, regressors or other estimators to build a composite estimator. The most common tool is a Pipeline.\n",
    "\n",
    "__Q__: Why should we use pipelines?\n",
    "\n",
    "    - Convenience: You only have to call fit and predict once on your data to fit a whole sequence of estimators.\n",
    "    - Joint parameter selection: You can grid search over parameters of all estimators in the pipeline at once.\n",
    "    - Safety: Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors.\n",
    "    \n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(0)\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier\n",
    "from sklearn.metrics import accuracy_score, f1_score, confusion_matrix, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## load the dataset \n",
    "## Source: https://www.kaggle.com/uciml/pima-indians-diabetes-database/download\n",
    "df = pd.read_csv('data/diabetes.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Let's use describe method to see if there is anything suspicious\n",
    "df.describe().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Now let's use info method\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## separate target variable from features\n",
    "target = df.Outcome\n",
    "df.drop('Outcome', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Let's see the distribution of 1's and 0's\n",
    "np.unique(target, return_counts= True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Split data into test train\n",
    "X_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in this problem it makes sense to focus on recall score as we don't want to  misclassify patients with diabetes.\n",
    "\n",
    "Recall score = $\\frac{tp}{(tp + fn)}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## First let's fit a logistic regression model to see the baseline\n",
    "## we will also use pipelines\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "## we will apply standard scaling to Logistic regression\n",
    "## because we might want to use regularization\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## ll estimators in a pipeline, except the last one,\n",
    "## must be transformers (i.e. must have a transform method). \n",
    "## The last estimator may be any type (transformer, classifier, etc.)\n",
    "\n",
    "pipe = Pipeline([('ss', StandardScaler()),\n",
    "                 ('log_reg', LogisticRegression(random_state=123,\n",
    "                                                max_iter = 500, \n",
    "                                                solver = 'saga'))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## we can access to a particular step in the pipeline\n",
    "\n",
    "print(pipe.steps[0])\n",
    "\n",
    "print(pipe['log_reg'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## we can call fit method with pipeline \n",
    "## we can call them with a gridsearch\n",
    "\n",
    "## let's use fit method from pipeline\n",
    "pipe.fit(X_train, y_train)\n",
    "\n",
    "## We can access the trained estimator from pipe\n",
    "pipe['log_reg'].predict(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## to find a best value for the C\n",
    "## let's use GridSearchCV\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "grid = grid = [{'log_reg__C': np.logspace(-2,2,10, base = 10.0), \n",
    "                'log_reg__penalty': ['l1', 'l2']}]\n",
    "\n",
    "gridsearch = GridSearchCV(estimator=pipe,\n",
    "                  param_grid=grid,\n",
    "                  scoring='recall',\n",
    "                  cv=5, verbose=1, n_jobs=-1)\n",
    "\n",
    "gridsearch.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Best accuracy\n",
    "print('Best accuracy: %.3f' % gridsearch.best_score_)\n",
    "\n",
    "# Best params\n",
    "print('\\nBest params:\\n', gridsearch.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boosting Algorithms\n",
    "\n",
    "__Q:__ What is boosting?\n",
    "\n",
    " - Recall that random forest algorithm uses boosting aggregation (bagging) to decrease the variance of individual trees.\n",
    " - Boosting ~ Bagging \n",
    "      - Bagging: Trees grow parallel\n",
    "      - Boosting: Trees grow sequentially\n",
    " - Idea is to create a slow learner.\n",
    "\n",
    "Recall that in bagging we did bootstrapping in boosting we don't do bootstrapping instead we modify the dataset at each step.\n",
    "\n",
    "__important parameters__(with sklearn notation)\n",
    "\n",
    "__n_estimators:__ # of trees to use in the procedure\n",
    "\n",
    "\n",
    "__learning_rate:__ (Shrinkage parameter)\n",
    "\n",
    "> The shrinkage parameter $\\lambda$, a small positivenumber.This controls the rate at which boosting learns. Typical values are 0.01 or 0.001, and the right choice can depend on the problem. Very small  $\\lambda$ can require using a very large value of B in order to achieve good performance\n",
    "\n",
    "\n",
    "<img src=\"img/boosting_algorithm.png\" width=450, height=450> \n",
    "\n",
    "__max_depth, max_leaf_nodes etc,__ (The number of splits in each trees)\n",
    "\n",
    "> Often d = 1 works well, in which case each tree is called a _stump_, consisting of a single split. In this case, the boosted ensemble is fitting an additive model, since each term involves only a single variable. More generally d is the interaction depth, and controls the interaction order of the boosted model, since d splits can involve at most d variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Now let's investigate the performance of Adaboost and GradientBoost\n",
    "from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier\n",
    "## let's see some of the parameters of the Gradient Boosting\n",
    "?GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbc = GradientBoostingClassifier(random_state= 103019,\n",
    "                                 validation_fraction=0.1, \n",
    "                                 n_iter_no_change= 5, \n",
    "                                 tol = 0.005)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## Let's use a gridsearch to find best parameters for GradientBoost\n",
    "\n",
    "params = {'n_estimators' : [100, 200, 300],\n",
    "         'learning_rate' : np.logspace(-3, -1, 5),\n",
    "         'max_leaf_nodes': [3,5,7,9],\n",
    "         'subsample': [0.2, 0.5, 0.7, 0.9], \n",
    "         'max_features':[0.5,1]}\n",
    "\n",
    "gs = GridSearchCV(estimator = gbc, \n",
    "                  param_grid = params,\n",
    "                  cv = 5, \n",
    "                  scoring= 'recall',\n",
    "                  verbose = 1,\n",
    "                  n_jobs= -1)\n",
    "\n",
    "gs.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some practical tips for Gradient Boost\n",
    "\n",
    "- Apparently max_leaf_nodes = k gives similar results to max_depth = k-1 but according to sklearn documentation max_leaf_nodes works faster. So you might want to use max_leaf_nodes for bigger projects.\n",
    "\n",
    "- Again according to sklearn documentation, smaller learning rate gives better test_scores but you might want to put more estimators if you set the learning rate small.\n",
    "\n",
    "- As it is mentioned above, when small learning rate is used we might increase the number of estimators. To prevent unneccesarry computing then we can put some early stopping criteria by the parameters: n_iter_change, min_impurity_decrease or tol.\n",
    "\n",
    "- It looks like subsampling with shrinkage method (learning rate) might give better results. In this case, out of bag test scoring is also become available. Note that you can access these by oob_improvement method.\n",
    "\n",
    "- Using a small max_features value can significantly decrease the runtime.\n",
    "\n",
    "For more: \n",
    "[sklearn documentation - gradientboost](https://scikit-learn.org/stable/modules/ensemble.html#gradient-boosting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(gs.best_score_)\n",
    "print(gs.best_estimator_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## let's see the best_estimator's test performance\n",
    "best_estimator = gs.best_estimator_\n",
    "y_pred = best_estimator.predict(X_test)\n",
    "\n",
    "\n",
    "## import recall_score from sklearn\n",
    "from sklearn.metrics import recall_score\n",
    "\n",
    "print(recall_score(y_test, y_pred))\n",
    "\n",
    "## similarly log_reg predictor would give\n",
    "\n",
    "log_reg_best = gridsearch.best_estimator_\n",
    "y_pred_log = log_reg_best.predict(X_test)\n",
    "\n",
    "print(recall_score(y_test, y_pred_log))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = best_estimator.predict(X_train)\n",
    "\n",
    "print(recall_score(y_train, y_train_pred))\n",
    "\n",
    "## try the same thing with log_reg_best: Do you expect better score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Try Adaboost algorithm and XGboost here\n",
    "## Use gridsearch or RandomSearchCV to fine-tune parameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}