notebooks/ensemble_hist_gradient_boosting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "23759342",
   "metadata": {},
   "source": [
    "# Speeding-up gradient-boosting\n",
    "In this notebook, we present a modified version of gradient boosting which\n",
    "uses a reduced number of splits when building the different trees. This\n",
    "algorithm is called \"histogram gradient boosting\" in scikit-learn.\n",
    "\n",
    "We previously mentioned that random-forest is an efficient algorithm since\n",
    "each tree of the ensemble can be fitted at the same time independently.\n",
    "Therefore, the algorithm scales efficiently with both the number of cores and\n",
    "the number of samples.\n",
    "\n",
    "In gradient-boosting, the algorithm is a sequential algorithm. It requires\n",
    "the `N-1` trees to have been fit to be able to fit the tree at stage `N`.\n",
    "Therefore, the algorithm is quite computationally expensive. The most\n",
    "expensive part in this algorithm is the search for the best split in the\n",
    "tree which is a brute-force approach: all possible split are evaluated and\n",
    "the best one is picked. We explained this process in the notebook \"tree in\n",
    "depth\", which you can refer to.\n",
    "\n",
    "To accelerate the gradient-boosting algorithm, one could reduce the number of\n",
    "splits to be evaluated. As a consequence, the statistical performance of such\n",
    "a tree would be reduced. However, since we are combining several trees in a\n",
    "gradient-boosting, we can add more estimators to overcome this issue.\n",
    "\n",
    "We will make a naive implementation of such algorithm using building blocks\n",
    "from scikit-learn. First, we will load the California housing dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "65340329",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:55:33.940450Z",
     "start_time": "2021-06-21T11:55:33.535608Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "data, target = fetch_california_housing(return_X_y=True, as_frame=True)\n",
    "target *= 100  # rescale the target in k$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b346ac4",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">If you want a deeper overview regarding this dataset, you can refer to the\n",
    "Appendix - Datasets description section at the end of this MOOC.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2058869e",
   "metadata": {},
   "source": [
    "We will make a quick benchmark of the original gradient boosting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "60bcdeaf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:55:43.849047Z",
     "start_time": "2021-06-21T11:55:37.590229Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "gradient_boosting = GradientBoostingRegressor(n_estimators=200)\n",
    "cv_results_gbdt = cross_validate(\n",
    "    gradient_boosting, data, target, scoring=\"neg_mean_absolute_error\",\n",
    "    n_jobs=-1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f0b66c1a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:55:43.865518Z",
     "start_time": "2021-06-21T11:55:43.854590Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradient Boosting Decision Tree\n",
      "Mean absolute error via cross-validation: 46.444 +/- 2.919 k$\n",
      "Average fit time: 5.448 seconds\n",
      "Average score time: 0.007 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"Gradient Boosting Decision Tree\")\n",
    "print(f\"Mean absolute error via cross-validation: \"\n",
    "      f\"{-cv_results_gbdt['test_score'].mean():.3f} +/- \"\n",
    "      f\"{cv_results_gbdt['test_score'].std():.3f} k$\")\n",
    "print(f\"Average fit time: \"\n",
    "      f\"{cv_results_gbdt['fit_time'].mean():.3f} seconds\")\n",
    "print(f\"Average score time: \"\n",
    "      f\"{cv_results_gbdt['score_time'].mean():.3f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0985a7db",
   "metadata": {},
   "source": [
    "We recall that a way of accelerating the gradient boosting is to reduce the\n",
    "number of split considered within the tree building. One way is to bin the\n",
    "data before to give them into the gradient boosting. A transformer called\n",
    "`KBinsDiscretizer` is doing such transformation. Thus, we can pipeline\n",
    "this preprocessing with the gradient boosting.\n",
    "\n",
    "We can first demonstrate the transformation done by the `KBinsDiscretizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "62e20de5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:56:00.139343Z",
     "start_time": "2021-06-21T11:56:00.107889Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/explore/miniconda3/envs/scikit-learn-course/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:220: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 1 are removed. Consider decreasing the number of bins.\n",
      "  warnings.warn('Bins whose width are too small (i.e., <= '\n",
      "/home/explore/miniconda3/envs/scikit-learn-course/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:220: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 3 are removed. Consider decreasing the number of bins.\n",
      "  warnings.warn('Bins whose width are too small (i.e., <= '\n",
      "/home/explore/miniconda3/envs/scikit-learn-course/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:220: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 6 are removed. Consider decreasing the number of bins.\n",
      "  warnings.warn('Bins whose width are too small (i.e., <= '\n",
      "/home/explore/miniconda3/envs/scikit-learn-course/lib/python3.9/site-packages/sklearn/preprocessing/_discretization.py:220: UserWarning: Bins whose width are too small (i.e., <= 1e-8) in feature 7 are removed. Consider decreasing the number of bins.\n",
      "  warnings.warn('Bins whose width are too small (i.e., <= '\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[249.,  39., 231., ...,  83., 162.,  30.],\n",
       "       [248.,  19., 203., ...,  28., 161.,  30.],\n",
       "       [242.,  49., 249., ..., 125., 160.,  29.],\n",
       "       ...,\n",
       "       [ 17.,  15., 126., ...,  49., 200.,  82.],\n",
       "       [ 23.,  16., 136., ...,  29., 200.,  77.],\n",
       "       [ 53.,  14., 130., ...,  93., 199.,  81.]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.preprocessing import KBinsDiscretizer\n",
    "\n",
    "discretizer = KBinsDiscretizer(\n",
    "    n_bins=256, encode=\"ordinal\", strategy=\"quantile\")\n",
    "data_trans = discretizer.fit_transform(data)\n",
    "data_trans"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a8de3cc",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">The code cell above will generate a couple of warnings. Indeed, for some of\n",
    "the features, we requested too much bins in regard of the data dispersion\n",
    "for those features. The smallest bins will be removed.</p>\n",
    "</div>\n",
    "We see that the discretizer transforms the original data into an integer.\n",
    "This integer represents the bin index when the distribution by quantile is\n",
    "performed. We can check the number of bins per feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dc2054ad",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:56:05.067180Z",
     "start_time": "2021-06-21T11:56:05.058287Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[256, 50, 256, 253, 256, 256, 207, 235]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[len(np.unique(col)) for col in data_trans.T]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "550b3ef5",
   "metadata": {},
   "source": [
    "After this transformation, we see that we have at most 256 unique values per\n",
    "features. Now, we will use this transformer to discretize data before\n",
    "training the gradient boosting regressor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "32bf0fbc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:56:16.425365Z",
     "start_time": "2021-06-21T11:56:12.546892Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "gradient_boosting = make_pipeline(\n",
    "    discretizer, GradientBoostingRegressor(n_estimators=200))\n",
    "cv_results_gbdt = cross_validate(\n",
    "    gradient_boosting, data, target, scoring=\"neg_mean_absolute_error\",\n",
    "    n_jobs=-1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a8ff4348",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:56:16.431299Z",
     "start_time": "2021-06-21T11:56:16.427090Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradient Boosting Decision Tree with KBinsDiscretizer\n",
      "Mean absolute error via cross-validation: 45.813 +/- 2.196 k$\n",
      "Average fit time: 3.360 seconds\n",
      "Average score time: 0.010 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"Gradient Boosting Decision Tree with KBinsDiscretizer\")\n",
    "print(f\"Mean absolute error via cross-validation: \"\n",
    "      f\"{-cv_results_gbdt['test_score'].mean():.3f} +/- \"\n",
    "      f\"{cv_results_gbdt['test_score'].std():.3f} k$\")\n",
    "print(f\"Average fit time: \"\n",
    "      f\"{cv_results_gbdt['fit_time'].mean():.3f} seconds\")\n",
    "print(f\"Average score time: \"\n",
    "      f\"{cv_results_gbdt['score_time'].mean():.3f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1981c104",
   "metadata": {},
   "source": [
    "Here, we see that the fit time has been drastically reduced but that the\n",
    "statistical performance of the model is identical. Scikit-learn provides a\n",
    "specific classes which are even more optimized for large dataset, called\n",
    "`HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`. Each\n",
    "feature in the dataset `data` is first binned by computing histograms, which\n",
    "are later used to evaluate the potential splits. The number of splits to\n",
    "evaluate is then much smaller. This algorithm becomes much more efficient\n",
    "than gradient boosting when the dataset has over 10,000 samples.\n",
    "\n",
    "Below we will give an example for a large dataset and we will compare\n",
    "computation times with the experiment of the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b5f1617a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:56:47.794446Z",
     "start_time": "2021-06-21T11:56:43.902006Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.experimental import enable_hist_gradient_boosting\n",
    "from sklearn.ensemble import HistGradientBoostingRegressor\n",
    "\n",
    "histogram_gradient_boosting = HistGradientBoostingRegressor(\n",
    "    max_iter=200, random_state=0)\n",
    "cv_results_hgbdt = cross_validate(\n",
    "    gradient_boosting, data, target, scoring=\"neg_mean_absolute_error\",\n",
    "    n_jobs=-1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "527e763e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-21T11:56:49.616805Z",
     "start_time": "2021-06-21T11:56:49.613917Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Histogram Gradient Boosting Decision Tree\n",
      "Mean absolute error via cross-validation: 46.124 +/- 2.227 k$\n",
      "Average fit time: 3.369 seconds\n",
      "Average score time: 0.009 seconds\n"
     ]
    }
   ],
   "source": [
    "print(\"Histogram Gradient Boosting Decision Tree\")\n",
    "print(f\"Mean absolute error via cross-validation: \"\n",
    "      f\"{-cv_results_hgbdt['test_score'].mean():.3f} +/- \"\n",
    "      f\"{cv_results_hgbdt['test_score'].std():.3f} k$\")\n",
    "print(f\"Average fit time: \"\n",
    "      f\"{cv_results_hgbdt['fit_time'].mean():.3f} seconds\")\n",
    "print(f\"Average score time: \"\n",
    "      f\"{cv_results_hgbdt['score_time'].mean():.3f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e779dde",
   "metadata": {},
   "source": [
    "The histogram gradient-boosting is the best algorithm in terms of score.\n",
    "It will also scale when the number of samples increases, while the normal\n",
    "gradient-boosting will not."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}