6-Regularization-Identification-Data-Leakage-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regularization\n",
    "\n",
    "\"No Free Lunch\" implies that ML algorithms will need to be \"tweaked\" in order to perform well on specific applications.  _Regularization_ methods are used to do this.\n",
    "\n",
    "According to Goodfellow et al. (2016), \"_Regularization is any modification we make to a learning algorithm that is intended to reduce it's generalization error but not it's training error._\" (p. 117.)\n",
    "\n",
    "May involve accepting additional bias in return to variance reduction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regulation Methods\n",
    "\n",
    "There are many.  Common examples:\n",
    "\n",
    "* Increasing or decreasing the number or dimensionality of the data.\n",
    "* Penalizing the _cost function_ minimized.\n",
    "* Using \"shrinkage estimators\"\n",
    "* Employing ensembles.\n",
    "* Avoiding \"over-training\"\n",
    "* Data augmentation.\n",
    "* Early stopping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Identification\n",
    "\n",
    "### Identification\n",
    "\n",
    "In many statistical applications, parameters need to be adequately _identified_ parameters in order to get \"good\" (consistent) estimates of them.\n",
    "\n",
    "An example that may be familiar is the contrast coding of categorical predictor variables used in regression models. You can have up to k-1 regression coefficient estimates for a k category discrete predictor variables.\n",
    "\n",
    "When you use regression for ML predictive applications, your not primarily interested in the estimates of coefficients, but you _are_ interested in your algorithm learning patterns in a way that generalizes well to new data.\n",
    "\n",
    "Try running the patient satisfaction k-fold CV regression examples using all three patient categories as 0/1 coded predictor variables to see what happens.\n",
    "\n",
    "In ML circles, dummy coding may be referred to as _one hot coding_.   Using `scikit-learn`: [OneHotEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data Reduction and Standardizing Transformations\n",
    "\n",
    "The space of the data can sometimes be reduced so as to improve algorithm performance on\n",
    "new data.  _Principal Components Analysis_ (PCA) is frequently used for this.  PCA is a SVD-based method that can be used to project data into smaller dimensional spaces, albeit with some loss of information.\n",
    "\n",
    "In `scikit-learn`, PCA methods can be found at\n",
    "\n",
    "[sklearn.decomposition.PCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)\n",
    "\n",
    "A variety of transformation methods like rescaling to mean = 0, SD =1 can be used to improve ML algorithm performance.  See [Preprocessing](https://scikit-learn.org/stable/modules/preprocessing.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## \"Information From the Future!\" Data Leakage\n",
    "\n",
    "When creating validation data sets, and when using cross-validation, it's important to avoid using for training your algorithms any information that couldn't be known when applying it to newly available data.  \"Test\" data used for validation or cross validation proxy for data that \"haven't yet been seen\" by an algorithm.\n",
    "\n",
    "Here are two simple examples in which data leakage would be an issue:\n",
    "\n",
    "* Predicting whether hospital inpatients have had a myocardial infarction using whether they have been admitted to a cardiac unit;\n",
    "* Predicting whether insured parties are late on making payments on their policy using the number of follow-up reminders that their payments are late.\n",
    "\n",
    "One thing to note about both of these examples is that \"time's arrow\" is reversed.\n",
    "\n",
    "\n",
    "[Data Leakage](https://www.cs.umb.edu/~ding/history/470_670_fall_2011/papers/cs670_Tran_PreferredPaper_LeakingInDataMining.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##  A Common Cause of Leakage\n",
    "\n",
    "Rescaling or otherwise transforming training and test data together in a way that uses estimates based on the combined data.\n",
    "\n",
    "Avoid any transformations or rescaling of data that involve estimates of  _random variables_ that are computed using both training and test data."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "679px",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}