part_1/1.5 Debugging and Improving ML Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Debugging and Improving ML Models\n",
    "\n",
    "Based on [Andrew Ng's Advice for applying Machine Learning](http://cs229.stanford.edu/materials/ML-advice.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quiz\n",
    "\n",
    "What would you do to improve this model's performance?\n",
    "\n",
    "<img src=\"img/ng-high-bias.png\" width=600>\n",
    "<div style=\"text-align: right\">Source: A. Ng. \"Advice for applying Machine Learning\"</div>\n",
    "\n",
    " 1. Choose a model class with higher complexity\n",
    " 1. Get more data\n",
    " 1. Remove features\n",
    " 1. Create new features\n",
    " 1. Choose a model class with less complexity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More on Bias vs Variance\n",
    "\n",
    "Typical learning curve for high variance\n",
    "\n",
    "\n",
    "<img src=\"img/ng-high-variance.png\" width=600>\n",
    "<div style=\"text-align: right\">Source: A. Ng. \"Advice for applying Machine Learning\"</div>\n",
    "\n",
    "  * Test error still decreasing as m increases. Model has enough capacity, suggests larger training set will help.\n",
    "  * Large gap between training and test error. \n",
    "\n",
    "Scikit-learn provides a [Learning Curves](https://scikit-learn.org/stable/modules/learning_curve.html#learning-curve)\n",
    "```python\n",
    "from sklearn.model_selection import learning_curve\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More on Bias vs Variance\n",
    "\n",
    "Typical learning curve for high bias\n",
    "\n",
    "\n",
    "<img src=\"img/ng-high-bias.png\" width=600>\n",
    "<div style=\"text-align: right\">Source: A. Ng. \"Advice for applying Machine Learning\"</div>\n",
    "\n",
    "  * Even training error is unacceptably high; more data wont help.\n",
    "    - Create new features\n",
    "    - Buy a newer GPU to train deeper neural nets\n",
    "  * Small gap between training and test error. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "FIGSIZE = (11, 7)\n",
    "\n",
    "def ground_truth(x):\n",
    "    \"\"\"Ground truth -- function to approximate\"\"\"\n",
    "    return x * np.sin(x) + np.sin(2 * x)\n",
    "\n",
    "def gen_data(n_samples=200):\n",
    "    \"\"\"generate training and testing data\"\"\"\n",
    "    np.random.seed(15)\n",
    "    X = np.random.uniform(0, 10, size=n_samples)[:, np.newaxis]\n",
    "    y = ground_truth(X.ravel()) + np.random.normal(scale=2, size=n_samples)\n",
    "    train_mask = np.random.randint(0, 2, size=n_samples).astype(np.bool)\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=3)\n",
    "    return X_train, X_test, y_train, y_test\n",
    "\n",
    "X_train, X_test, y_train, y_test = gen_data(100)\n",
    "\n",
    "# plot ground truth\n",
    "x_plot = np.linspace(0, 10, 500)\n",
    "\n",
    "def plot_data(alpha=0.4, s=20):\n",
    "    fig = plt.figure(figsize=FIGSIZE)\n",
    "    gt = plt.plot(x_plot, ground_truth(x_plot), alpha=alpha, label='ground truth')\n",
    "\n",
    "    # plot training and testing data\n",
    "    plt.scatter(X_train, y_train, s=s, alpha=alpha)\n",
    "    plt.scatter(X_test, y_test, s=s, alpha=alpha, color='red')\n",
    "    plt.xlim((0, 10))\n",
    "    plt.ylabel('y')\n",
    "    plt.xlabel('x')\n",
    "    \n",
    "annotation_kw = {'xycoords': 'data', 'textcoords': 'data',\n",
    "                 'arrowprops': {'arrowstyle': '->', 'connectionstyle': 'arc'}}\n",
    "    \n",
    "plot_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression Tree\n",
    "\n",
    "  * `max_depth` argument controlls the depth of the tree\n",
    "  * The deeper the tree the more variance can be explained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "plot_data()\n",
    "est = DecisionTreeRegressor(max_depth=1).fit(X_train, y_train)\n",
    "plt.plot(x_plot, est.predict(x_plot[:, np.newaxis]),\n",
    "         label='RT max_depth=1', color='g', alpha=0.9, linewidth=2)\n",
    "\n",
    "est = DecisionTreeRegressor(max_depth=3).fit(X_train, y_train)\n",
    "plt.plot(x_plot, est.predict(x_plot[:, np.newaxis]),\n",
    "         label='RT max_depth=3', color='g', alpha=0.7, linewidth=1)\n",
    "\n",
    "plt.legend(loc='upper left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Function approximation with Gradient Boosting\n",
    "\n",
    "  * `n_estimators` argument controls the number of trees\n",
    "  * `staged_predict` method allows us to step through predictions as we add more trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import islice\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "plot_data()\n",
    "est = GradientBoostingRegressor(n_estimators=1000, max_depth=1, learning_rate=1.0)\n",
    "est.fit(X_train, y_train)\n",
    "\n",
    "ax = plt.gca()\n",
    "first = True\n",
    "\n",
    "# step through prediction as we add 10 more trees.\n",
    "for pred in islice(est.staged_predict(x_plot[:, np.newaxis]), 0, est.n_estimators, 10):\n",
    "    plt.plot(x_plot, pred, color='r', alpha=0.2)\n",
    "    if first:\n",
    "        ax.annotate('High bias - low variance', xy=(x_plot[x_plot.shape[0] // 2],\n",
    "                                                    pred[x_plot.shape[0] // 2]),\n",
    "                                                    xytext=(4, 4), **annotation_kw)\n",
    "        first = False\n",
    "\n",
    "pred = est.predict(x_plot[:, np.newaxis])\n",
    "plt.plot(x_plot, pred, color='r', label='GBRT max_depth=1')\n",
    "ax.annotate('Low bias - high variance', xy=(x_plot[x_plot.shape[0] // 2],\n",
    "                                            pred[x_plot.shape[0] // 2]),\n",
    "                                            xytext=(6.25, -6), **annotation_kw)\n",
    "plt.legend(loc='upper left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model complexity\n",
    "  * The number of trees and the depth of the individual trees control model complexity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deviance plot\n",
    "  * Diagnostic to determine if model is overfitting\n",
    "  * Plots the training/testing error (deviance) as a function of the number of trees (=model complexity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deviance_plot(est, X_test, y_test, ax=None, label='', train_color='#2c7bb6', \n",
    "                  test_color='#d7191c', alpha=1.0, ylim=(0, 10)):\n",
    "    \"\"\"Deviance plot for ``est``, use ``X_test`` and ``y_test`` for test error. \"\"\"\n",
    "    n_estimators = len(est.estimators_)\n",
    "    test_dev = np.empty(n_estimators)\n",
    "\n",
    "    for i, pred in enumerate(est.staged_predict(X_test)):\n",
    "       test_dev[i] = est.loss_(y_test, pred)\n",
    "\n",
    "    if ax is None:\n",
    "        fig = plt.figure(figsize=FIGSIZE)\n",
    "        ax = plt.gca()\n",
    "        \n",
    "    ax.plot(np.arange(n_estimators) + 1, test_dev, color=test_color, label='Test %s' % label, \n",
    "             linewidth=2, alpha=alpha)\n",
    "    ax.plot(np.arange(n_estimators) + 1, est.train_score_, color=train_color, \n",
    "             label='Train %s' % label, linewidth=2, alpha=alpha)\n",
    "    ax.set_ylabel('Error')\n",
    "    ax.set_xlabel('n_estimators')\n",
    "    ax.set_ylim(ylim)\n",
    "    return test_dev, ax\n",
    "\n",
    "test_dev, ax = deviance_plot(est, X_test, y_test)\n",
    "ax.legend(loc='upper right')\n",
    "\n",
    "# add some annotations\n",
    "ax.annotate('Lowest test error', xy=(test_dev.argmin() + 1, test_dev.min() + 0.02),\n",
    "            xytext=(150, 3.5), **annotation_kw)\n",
    "\n",
    "ann = ax.annotate('', xy=(800, test_dev[799]),  xycoords='data',\n",
    "                  xytext=(800, est.train_score_[799]), textcoords='data',\n",
    "                  arrowprops={'arrowstyle': '<->'})\n",
    "ax.text(810, 3.5, 'train-test gap')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfitting\n",
    "\n",
    "  * Large gap means **overfitting** we have to **regularize**!\n",
    "  \n",
    "### GBM Regularization\n",
    "\n",
    "  * Tree depth `max_depth` (variance++)\n",
    "  * Shrinkage `learning_rate` (variance--)\n",
    "  * Randomization (`subsample`, `max_features`) (variance--)\n",
    "  \n",
    "Lets try shallower trees (`min_samples_leaf=3` instead of 1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fmt_params(params):\n",
    "    return \", \".join(\"{0}={1}\".format(key, val) for key, val in params.items())\n",
    "\n",
    "fig = plt.figure(figsize=FIGSIZE)\n",
    "ax = plt.gca()\n",
    "for params, (test_color, train_color) in [({}, ('#d7191c', '#2c7bb6')),\n",
    "                                          ({'min_samples_leaf': 3}, ('#fdae61', '#abd9e9'))]:\n",
    "    est = GradientBoostingRegressor(n_estimators=1000, max_depth=1, \n",
    "                                    learning_rate=1.0)\n",
    "    est.set_params(**params)\n",
    "    est.fit(X_train, y_train)\n",
    "    test_dev, ax = deviance_plot(est, X_test, y_test, ax=ax, label=fmt_params(params),\n",
    "                                 train_color=train_color, test_color=test_color)\n",
    "    \n",
    "ax.annotate('Higher bias', xy=(900, est.train_score_[899]), xytext=(600, 3), **annotation_kw)\n",
    "ax.annotate('Lower variance', xy=(900, test_dev[899]), xytext=(600, 3.5), **annotation_kw)\n",
    "plt.legend(loc='upper right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}