Lectures/lecture_13_summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture #13: Stochastic Gradient Descent and Simulated Annealing\n",
    "## AM 207: Advanced Scientific Computing\n",
    "### Stochastic Methods for Data Analysis, Inference and Optimization\n",
    "### Fall, 2021"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##### <img src=\"fig/logos.jpg\" style=\"height:150px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lecture #13 Summary\n",
    "\n",
    "For a set of observations $y$, we can collect additional data $x$ and model the ***conditional distribution*** $p(Y|X)$. This is exactly regression and classification models do, they use ***covariates*** $x$ to predict the ***outcome*** $y$.\n",
    "\n",
    "1. **(But Sometimes There are Too Many Covariates)** The more complicated your model is (e.g. lots of explanatory variables), the more it'll overfit. \n",
    "\n",
    "  - Variable selection by cross-validation of all possible subsets of covariates is not practical. Why?<br><br>\n",
    "  - We can do variable selection by ***information criteria***:\n",
    "    $$\\text{*IC} = \\underbrace{D(y|\\mathbf{w}_{\\text{MLE}})}_{\\text{(neg) model fit}} + \\underbrace{f(\\mathrm{dim}(\\mathbf{w}_{\\text{MLE}}))}_{\\text{model complexity}}$$<br><br>\n",
    "  - For Bayesian models, we can do variable selection by:\n",
    "    - the ***Bayes factor***, i.e. the ratio of the ***evidence*** of the data under two different models:\n",
    "  $$ \\frac{p(Y^{\\mathrm{Obs}} | \\text{model 1})}{p(Y^{\\mathrm{Obs}} | \\text{model 2})}$$<br><br>\n",
    "    - WAIC\n",
    "     $$\\text{WAIC} = \\text{\"neg log-likelihood\"} + \\text{\"posterior-variance\"}$$<br><br>\n",
    "2. **(But the Negative Log Likelihood May Be Highly Non-convex)** The more complicated the relationship between $y$ and $x$ the more non-convex the negative log-likelihood.\n",
    "\n",
    "  - Gradient descent will get trapped often in bad local min\n",
    "  - Why does ***stochastic gradient descent*** (adding noise to the gradient during descent) help avoid local min? How in practice do we \"add noise\" to the gradient?\n",
    "  - Turn optimization into sampling:\n",
    "    - Turn the objective function $f(x)$ into a pdf $p(x)$, but with higher mass where $f$ has lower values. This is the ***Gibbs distribution***:\n",
    "$$\n",
    "p(x) \\propto \\exp\\left\\{ -\\frac{f(x)}{T}\\right\\},\\;\\; T \\text{ is a constant}\n",
    "$$\n",
    "    - Sample from $p(x)$. Wait why is this a good idea?\n",
    "     <img src=\"fig/gibbs.jpg\" style=\"height:450px;\">\n",
    "    - To control exploration vs exploitation, we start with large $T$ and decrease slowly. Why?\n",
    "     <img src=\"fig/T.jpg\" style=\"height:450px;\"><br><br>\n",
    "\n",
    "3. **(With Covariates Comes Greater Explanatory Power)** By making the variations in $y$ explicitly depend on measurable factors $\\mathbf{X}$, we can ***explain*** the data in addition to modeling it. <br><br>\n",
    "\n",
    "4. **(With Greater Powers Come Greater Problems and Greater Responsibilities)** Conditional models have so much potential to generate scientific understanding of our data but they are also the most frequently misused! \n",
    "\n",
    "  - **(Data Problems)** Bias in your conclusions is exponentially exacerbated by bias in the data collection process.<br><br>\n",
    "  \n",
    "    - **Asking the wrong questions!** Wanting information 'A' (e.g. sex) but asking a question about 'B' (e.g. gender), because we didn't think carefully about or simply were not aware of the implications of the phrasing of our questions. \n",
    "    \n",
    "      This happens because language and semantics about scientific facts change very little or slowly over time, but language and semantics about human experiences are evolving.<br><br>\n",
    "    - **Biasing the answers!** Offering bad or misrepresentative options as answers, for example, offering an answer option that collapses true variation (e.g. false binaries, overly broad categories).<br><br>\n",
    "    - **Violation of user privacy and trust with poor custodianship of sensitive data!** For example, not sufficintly protection against data breach, selling sensitive information to 3rd-parties, performing inappropriate profiling and targeting based on sensitive attributes, etc.<br><br>\n",
    "    \n",
    "    - These problems can be mitigated by being conscientious and vigilant about the data collection process, and communicating with domain experts as well as affected communities during the entire process!<br><br>\n",
    "   \n",
    "  - **(Interpretation Problems)** It's tempting (and often easy) to interpret conditional models, but most interpretations are not entirely accurate and can be down right wrong!<br><br>\n",
    "  \n",
    "    - **Confusing correlation with causation!** This can be mitigated by designing follow-up experiments to test for causation!\n",
    "$$\n",
    "\\underbrace{p(y=1 | x_1, x_2, x_3)}_{\\text{$Y$ presence of lung cancer}} = \\mathrm{sigm}(-1 + 3 \\underbrace{x_1}_{\\text{nicotine stains on fingers}} + 1.5 \\underbrace{x_2}_{\\text{cholestrol}} - 1.75 \\underbrace{x_3}_{\\text{amount of exercise}}).\n",
    "$$<br><br>\n",
    "    - **Mis-attribution of importance when covariates are correlated** This can be mitigated by performing variable selection!\n",
    "    \\begin{align}\n",
    "(\\text{True model})\\quad \\mathrm{Prob}[\\text{Lung Cancer} = 1| \\mathbf{X}, \\mathbf{w}] &= \\mathrm{sigm}(-5 * \\text{age} - 0.2 * \\text{smoker} - 0.5)\\\\\n",
    "(\\text{Model where video games (hr) = 0.5 * age})\\quad \\mathrm{Prob}[\\text{Lung Cancer} = 1| \\mathbf{X}, \\mathbf{w}] &= \\mathrm{sigm}(0.5 * \\text{age} - 11 * \\text{games} - 0.2 * \\text{smoker} - 0.5)\n",
    "\\end{align}<br><br>\n",
    "    - **Mis-attribution of importance when the covariates are not equally scaled** This can be mitigated by standardizing your data!\n",
    "        \\begin{align}\n",
    "    \\underbrace{p(y=1 | x_1, x_2, x_3)}_{\\text{$Y$ presence of kidney cancer}} = \\mathrm{sigm}(-1 + 3 \\underbrace{x_1}_{\\text{Blood Lead Level g/dL}} + 0.5 \\underbrace{x_2}_{\\text{HCG in mIU/mL}} - 1.75 \\underbrace{x_3}_{\\text{Hemo. A1C}})\n",
    "    \\end{align}<br><br>\n",
    "  - **(Broader Impact Problems)** Because data scientists and machine learning engineers often interact with so many different real-life applications domains (rather than specializing in one domain like medicine or law), we may be the least prepared in terms of domain-specific knowledge but have the greatest impact!\n",
    "  \n",
    "    - (Not good) Influence on public policy\n",
    "    - (Not good) Influence on social and political systems (e.g. Facebook's internal report on the negative effects of the platform on democratic systems and vunerable populations)\n",
    "    - (Not good) Influence on vunerable populations\n",
    "    - Violation of regulatory statues governing application domains (e.g. fair lending in finance, Title VI in health-care etc)<br><br>\n",
    "    \n",
    "5. **(We Can Also Accomplish Social Good)** It's not all doom and gloom: while our mistakes (which can happen even when we are vigilant) can be impactful, so can the benefit we might bring. Conditional models can be really useful!\n",
    "\n",
    "  - **(Personalization)** Having explanatory factors for outcomes at the individual level can help personalize treatment and interventions for patients/users.<br><br>\n",
    "  - **(Recourse)** Having explanatory factors for outcomes at the individual level can help generate recourse (i.e. \"what can I do differently\") and empower users to challenge decision making systems.<br><br>\n",
    "  - **(Generate scientific hypothesis)** Correlation isn't always causation, but sometimes it is and you discover a new scientific insight!<br><br>\n",
    "  - **(Audit black-box decision making systems)** Correlation isn't always causation, but sometimes it is and you discover a bias in the human decision making system that generated the data (e.g. gender and racial biases in everyday language usage uncovered by bias in word embeddings)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}