Lecture_29.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 29: Law of Large Numbers, Central Limit\n",
    "\n",
    "\n",
    "## Stat 110, Prof. Joe Blitzstein, Harvard University\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Law of Large Numbers\n",
    "\n",
    "Let $X_1, X_2, \\dots $ be i.i.d. random variables all with with mean $\\mu$ and variance $\\sigma^2$.\n",
    "\n",
    "Let the *sample mean* be denoted by $\\bar{X_n} = \\frac{1}{n}\\sum_{i=1}^{n}X_j$, where $n$ is the sample size.\n",
    "\n",
    "*What can we say about $\\bar{X_n}$ when $n$ gets large?*\n",
    "\n",
    "\n",
    "### (Strong) Law of Large Numbers: $\\bar{X_n} \\rightarrow \\mu$ as $n \\rightarrow \\infty$ with probability 1.0\n",
    "\n",
    "* this means that the sample mean $\\bar{X_n}$ converges (point-wise) to the true mean $\\mu$ when $n$ gets large\n",
    "* hence it allows us to make an approximation of the true mean without having to actually carry out an infinite amount of experiments\n",
    "\n",
    "#### Example\n",
    "\n",
    "\\begin{align}\n",
    "  X_j &\\sim \\operatorname{Bern}(p) \\text{, then } \\frac{X_1 + \\dots + X_n}{n} \\rightarrow p \\text{ with probability 1.0}\n",
    "\\end{align}\n",
    "\n",
    "\n",
    "### (Weak) Law of Large Numbers: $\\text{for any } c > 0 \\text{, } P( \\lvert \\bar{X_n} - \\mu > c \\rvert ) \\rightarrow 0 \\text{ as } n \\rightarrow \\infty$\n",
    "\n",
    "* $c$ could be some small number, say, 0.001\n",
    "* if $n$ is large, it is extremely likely that the sample mean is close to true mean\n",
    "\n",
    "#### Proof\n",
    "\n",
    "\\begin{align}\n",
    "  P( \\lvert \\bar{X_n} - \\mu \\rvert > c) &\\leq \\frac{\\operatorname{Var}(\\bar{X_n})}{c^2} \\quad \\text{Chebyshev's Inequality} \\\\\n",
    "  \\\\\n",
    "  \\text{but } \\frac{\\operatorname{Var}(\\bar{X_n})}{c^2} &= \\frac{ \\frac{1}{n^2} n \\, \\sigma^2 }{c^2} \\\\\n",
    "  &= \\frac{\\sigma^2}{n \\, c^2} \\\\\n",
    "  &= 0 \\text{ as } n \\rightarrow \\infty \\quad \\blacksquare\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Central Limit Theorem\n",
    "\n",
    "Another way to think about the Law of Large Numbers is to see that \n",
    "\n",
    "\\begin{align}\n",
    "  \\displaystyle{\\lim_{n \\to \\infty}} \\, \\bar{X_n} - \\mu &= 0 \\\\\n",
    "\\end{align}\n",
    "\n",
    "However, this is only thinking point-wise.\n",
    "\n",
    "* *But what is the distribution of $\\bar{X_n}$?*\n",
    "* _What does that distribution look like?_\n",
    "* _That might be going to 0, but how fast is it doing so?_\n",
    "\n",
    "One way to study the distribution of $\\bar{X_n}$ is to multiply $(\\bar{X_n} - \\mu)$ by some variable that itself goes to $\\infty$.\n",
    "\n",
    "* if that product goes to $\\infty$, then we know that the variable we have chosen is dominating over $(\\bar{X_n} - \\mu)$\n",
    "* and if that product instead still approaches 0, then that also tells us something significant about $(\\bar{X_n} - \\mu)$\n",
    "\n",
    "Consider:\n",
    "\n",
    "\\begin{align}\n",
    "  n^{?} \\, (\\bar{X_n} - \\mu)\n",
    "\\end{align}\n",
    "\n",
    "We could learn more about the distribution of $\\bar{X_n}$ by selecting some power of $n > 0$, and thinking about what happens.\n",
    "\n",
    "### The Central Limit Theorem\n",
    "\n",
    "\\begin{align}\n",
    "  n^{1/2} \\, \\frac{(\\bar{X_n} - \\mu)}{\\sigma} \\rightarrow \\mathcal{N}(0,1) \\quad \\text{in distribution} \\\\\n",
    "\\end{align}\n",
    "\n",
    "* $X$ may be discrete or it may be continuous\n",
    "* we do need to assume that $X$ has a finite variance\n",
    "* the CDF of $X$ will converge to CDF $\\Phi$\n",
    "* CLT is the justification for using normal approximations\n",
    "* $n^{1/2}$ is _just right_; that is what allows for a non-degenerate distribution that actually converges into something beautiful and useful (it doesn't just go to 0 or blow up to $\\infty$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Proof: in terms of sum\n",
    "\n",
    "\n",
    "\n",
    "* we claim that the convolution $\\sum_{j=1}^{n} X_j$ is approximately normal with mean $n \\mu$\n",
    "* we can standardize this by centering $\\sum_{j=1}^{n} X_j - n \\mu$; this makes mean $\\mu=0$\n",
    "* the variance of the convolution is $n \\sigma^2$ (c.f. property [7] of covariance)\n",
    "* we complete standardization by dividing by the standard deviation $\\sqrt{n} \\sigma$\n",
    "\n",
    "Here is the standardized version:\n",
    "\\begin{align}\n",
    "  &\\quad \\frac{\\sum_{j=1}^{n} X_j - n\\mu}{\\sqrt{n} \\, \\sigma} \\rightarrow \\mathcal{N}(0,1) \n",
    "\\end{align}\n",
    "\n",
    "* Assume that MGF $M(t)$ of $X_j$ exists\n",
    "* Assume wlog that $\\mu=0$ and $\\sigma=1$, since we could have alternatively standardized the r.v. separately and declared $\\frac{1}{\\sqrt{n}} \\sum_{j=1}^{n} \\frac{(X_j - \\mu)}{\\sigma}$\n",
    " \n",
    "Let $S_n = \\sum_{j=1}^{n} X_j$, show $M \\left[ \\frac{S_n}{\\sqrt{n}} \\right] \\rightarrow M \\left[ \\mathcal{N}(0,1) \\right]$.\n",
    "\n",
    "Here are some quick facts about Moment Generating Functions to keep in mind as we go along our proof:\n",
    "\n",
    "1. $M(t) = \\mathbb{E}(e^{tX_1})$\n",
    "2. $M(0) = 1$\n",
    "3. $M\\prime(0) = \\mu = 0$\n",
    "4. $M\\prime(0) = 1$\n",
    "\n",
    "\\begin{align}\n",
    "  M\\left[ S_n \\right] &= \\mathbb{E}( e^{t \\, S_n / \\sqrt{n} } ) \\\\\n",
    "  &= \\mathbb{E}(e^{t \\, X_1/\\sqrt{n}}) \\dots \\mathbb{E}(e^{t \\, X_n/\\sqrt{n}})  &\\quad \\text{independence, not correlated} \\\\\n",
    "  &= \\left( M\\left[ \\frac{t}{\\sqrt{n}} \\right] \\right)^n &\\quad \\text{but these are just the MGF} \\\\\n",
    "  &= \\displaystyle{\\lim_{n \\to \\infty}} n \\, log  M\\left( \\frac{t}{\\sqrt{n}} \\right) \\\\\n",
    "  &= \\displaystyle{\\lim_{n \\to \\infty}} \\frac{log  M\\left( \\frac{t}{\\sqrt{n}} \\right)}{\\frac{1}{n}} &\\quad \\text{let } y = \\frac{1}{\\sqrt{n}} \\text{ and } y \\text{ is real} \\\\\n",
    "  &= \\displaystyle{\\lim_{y \\to 0}} \\frac{log M(y\\,t)}{y^2} &\\quad \\text{and using l'Hopital's Rule} \\\\\n",
    "  &= \\displaystyle{\\lim_{y \\to 0}} \\frac{t \\, M\\prime(yt)}{2y \\, M(yt)} &\\quad \\text{and applying l'Hopital's Rule again} \\\\\n",
    "  &= \\frac{t}{2} \\, \\displaystyle{\\lim_{y \\to 0}} \\frac{M\\prime(yt)}{y} &\\quad \\text{and applying l'Hopital's Rule yet again} \\\\\n",
    "  &= \\frac{t^2}{2} \\, \\displaystyle{\\lim_{y \\to 0}} \\frac{M\\prime\\prime(yt)}{1} \\\\\n",
    "  &= \\frac{t^2}{2} \\\\\n",
    "\\end{align} \n",
    "\n",
    "But $\\frac{t^2}{2}$ is the log of $e^{t^2 / 2}$, and that is the MGF of $\\mathcal{N}(0,1)$. QED."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Basic Normal Approximation of a Binomial\n",
    "\n",
    "Let $X \\sim \\operatorname{Bin}(n,p)$, and think of $X = \\sum_{j=1}^{n} X_j, \\, X \\sim \\operatorname{Bern}(p)$ i.i.d.\n",
    "\n",
    "By the Central Limit Theorem, we can approximate $X$ with a Normal distribution if $n$ is large enough, and if we standardize $X$ first.\n",
    "\n",
    "\\begin{align}\n",
    "  P(a \\leq X \\leq b) &= P\\left( \\frac{a - np}{\\sqrt{npq}} \\leq \\frac{X - np}{\\sqrt{npq}} \\leq \\frac{b - np}{\\sqrt{npq}} \\right) \\\\\n",
    "  &\\approx \\Phi\\left( \\frac{b - np}{\\sqrt{npq}} \\right) - \\Phi\\left( \\frac{a - np}{\\sqrt{npq}} \\right)\n",
    "\\end{align}\n",
    "\n",
    "Contrast the above **Normal** approximation of a binomial with a **Poisson** approximation. With a Poisson approximation of a binomial, we assumed:\n",
    "\n",
    "* $n$ is very large, i.e., $n \\rightarrow \\infty$\n",
    "* $p$ is very small\n",
    "* let $\\lambda = np$, i.e. $p$ is fixed\n",
    "\n",
    "But in the case of a Normal approximation, while we do wish $n$ to be large, it is best if $p$ is close to $\\frac{1}{2}$. _Why?_\n",
    "\n",
    "Remember that the Normal distribution in a CLR is symmetric about $\\mu = 0$. If $p$ is too far away from the mean, then the distribution will get very skewed. That is bad. If $n$ is really, really, really large, then the CLR would still work no matter what $p$ might be, but you will need to be careful when $n$ is not that large."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuity Correction\n",
    "\n",
    "Now in the above example, we are approximating a discrete distribution using a continuous one. \n",
    "\n",
    "What would we do if we instead started with something like $P(X=a)$, where $a$ is some integer?\n",
    "\n",
    "\\begin{align}\n",
    "  P(X=a) &= P(a - \\epsilon \\leq X \\leq a + \\epsilon)\n",
    "\\end{align}\n",
    "\n",
    "... where $\\epsilon$ is a very small value that allows us to look at a _range centered at $a$_ instead of a single value $a$. Now we can continue using the Normal approximation.\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View [Lecture 29: Law of Large Numbers and Central Limit Theorem | Statistics 110](http://bit.ly/2NnWEaZ) on YouTube."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}