ch06_operations_on_distributions.ipynb

{
  "metadata": {
    "name": "Operations on Distributions"
  },
  "nbformat": 3,
  "nbformat_minor": 0,
  "worksheets": [
    {
      "cells": [
        {
          "cell_type": "heading",
          "level": 1,
          "metadata": {
          },
          "source": "Operations on Distributions"
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Skewness"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "_Skewness_ is a statistic that measures the asymmetry of a distribution. Given a sequence of values, _x_ _i_, the sample skewness is:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "You might recognize _m2 as the mean squared deviation (also known as variance); m3_ is the mean cubed deviation."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Negative skewness indicates that a distribution “skews left”; that is, it extends farther to the left than the right. Positive skewness indicates that a distribution skews right."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "In practice, computing the skewness of a sample is usually not a good idea. If there are any outliers, they have a disproportionate effect on _g\\_\\_1_."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Another way to evaluate the asymmetry of a distribution is to look at the relationship between the mean and median. Extreme values have more effect on the mean than the median, so in a distribution that skews left, the mean is less than the median."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "_Pearson’s median skewness coefficient_ is an alternative measure of skewness that explicitly captures the relationship between the mean, _μ_, and the median, _μ\\_\\_1/2_:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "This statistic is _robust_, which means that it is less vulnerable to the effect of outliers."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "PDFs"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The derivative of a CDF is called a _probability density function_, or _PDF_. For example, the PDF of an exponential distribution is"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The PDF of a normal distribution is"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Evaluating a PDF for a particular value of _x_ is usually not useful. The result is not a probability; it is a probability _density_."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "In physics, density is mass per unit of volume; in order to get a mass, you have to multiply by volume or, if the density is not constant, you have to integrate over volume."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Similarly, probability density measures probability per unit of _x_. In order to get a probability mass, [14](ch06.html#idp841392) you have to integrate over _x_. For example, if _x_ is a random variable whose PDF is PDF\\_X\\_, we can compute the probability that a value from _X_ falls between −0.5 and 0.5:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Or, since the CDF is the integral of the PDF, we can write"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "For some distributions, we can evaluate the CDF explicitly, so we would use the second option. Otherwise, we usually have to integrate the PDF numerically."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Convolution"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Suppose we have two random variables, _X_ and _Y_, with distributions CDF\\_X\\_ and CDF\\_Y\\_. What is the distribution of the sum _Z_ = _X_ + _Y_?"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "One option is to write a RandomVariable object that generates the sum:"
        },
        {
          "cell_type": "code",
          "collapsed": false,
          "input": "class Sum(RandomVariable):\n    def __init__(X, Y):\n        self.X = X\n        self.Y = Y\n\n    def generate():\n        return X.generate() + Y.generate()",
          "language": "python",
          "metadata": {
          },
          "outputs": [

          ]
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Given any RandomVariables, _X_ and _Y_, we can create a Sum object that represents _Z_. Then we can use a sample from _Z_ to approximate CDF\\_Z\\_."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "This approach is simple and versatile, but not very efficient; we have to generate a large sample to estimate CDF\\_Z\\_ accurately, and even then it is not exact."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "If CDF\\_X\\_ and CDF\\_Y\\_ are expressed as functions, sometimes we can find CDF\\_Z\\_ exactly. Here’s how:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ol>\n<li>\n<p>To start, assume that the particular value of <em>X</em> is <em>x</em>. Then CDF_Z_(<em>z</em>) is</p>\n</li>\n\n</ol>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Let’s read that back. The left side is “the probability that the sum is less than _z_, given that the first term is _x_.” Well, if the first term is _x_ and the sum has to be less than _z_, then the second term has to be less than _z_ − _x_."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ol>\n<li>\n<p>To get the probability that <em>Y</em> is less than <em>z</em> − <em>x</em>, we evaluate CDF_Y_.</p>\n</li>\n\n</ol>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "This follows from the definition of the CDF."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ol>\n<li>\n<p>Good so far? Let’s go on. Since we don’t actually know the value of <em>x</em>, we have to consider all values it could have and integrate over them:</p>\n</li>\n\n</ol>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The integrand is “the probability that _Z_ is less than or equal to _z_, given that _X_ = _x_, times the probability that _X_ = _x_.”"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Substituting from the previous steps we get"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The left side is the definition of CDF\\_Z\\_, so we conclude:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ol>\n<li>\n<p>To get PDF_Z_, take the derivative of both sides with respect to <em>z</em>. The result is</p>\n</li>\n\n</ol>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "If you have studied signals and systems, you might recognize that integral. It is the **convolution** of PDF\\_Y\\_ and PDF\\_X\\_, denoted with the operator ∗."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "PDF\\_Z\\_= PDF\\_Y\\_∗ PDF\\_X\\_"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "So the distribution of the sum is the convolution of the distributions. See [http://wiktionary.org/wiki/booyah](http://wiktionary.org/wiki/booyah)!"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "As an example, suppose _X_ and _Y_ are random variables with an exponential distribution with parameter _λ_. The distribution of _Z_ = _X_ + _Y_ is:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Now we have to remember that PDFexpo is 0 for all negative values, but we can handle that by adjusting the limits of integration:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Now we can combine terms and move constants outside the integral:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "This, it turns out, is the PDF of an Erlang distribution with parameter _k_ = 2 (see [http://wikipedia.org/wiki/Erlang\\_distribution](http://wikipedia.org/wiki/Erlang_distribution)). So the convolution of two exponential distributions (with the same parameter) is an Erlang distribution."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Why Normal?"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "I said earlier that normal distributions are amenable to analysis, but I didn’t say why. One reason is that they are closed under linear transformation and convolution. To explain what that means, it will help to introduce some notation."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "If the distribution of a random variable, _X_, is normal with parameters _μ_ and _σ_, you can write"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "_X_ ∼  ![img-0040](figs/web/equations/img-0040.png) (_μ_, _σ_)"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "where the symbol ∼ means “is distributed” and the script letter ![img-0040](figs/web/equations/img-0040.png)  stands for “normal.”"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "A linear transformation of _X_ is something like _X_’ = _a_ _X_ + _b_, where _a_ and _b_ are real numbers. A family of distributions is closed under linear transformation if _X_’ is in the same family as _X_. The normal distribution has this property; if _X_ ∼  ![img-0040](figs/web/equations/img-0040.png) (_μ_, _σ\\_\\_2_),"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "_X_’ ∼  ![img-0040](figs/web/equations/img-0040.png) (_aμ + b, a2σ2_)"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Normal distributions are also closed under convolution. If _Z_ = _X_ + _Y_ and _X_ ∼  ![img-0040](figs/web/equations/img-0040.png) (_μ_ _X_, _σ_ _X2) and Y ∼  ![img-0040](figs/web/equations/img-0040.png) (μ Y, σ Y2_) then"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The other distributions we have looked at do not have these properties."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Central Limit Theorem"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "So far we have seen:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ul>\n<li>\n<p>If we add values drawn from normal distributions, the distribution of the sum is normal.</p>\n</li>\n<li>\n<p>If we add values drawn from other distributions, the sum does not generally have one of the continuous distributions we have seen.</p>\n</li>\n</ul>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "But it turns out that if we add up a large number of values from almost any distribution, the distribution of the sum converges to normal."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "More specifically, if the distribution of the values has mean and standard deviation _μ_ and _σ_, the distribution of the sum is approximately ![img-0040](figs/web/equations/img-0040.png) (_nμ, nσ\\_\\_2_)."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "This is called the _Central Limit Theorem_. It is one of the most useful tools for statistical analysis, but it comes with caveats:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ul>\n<li>\n<p>The values have to be drawn independently.</p>\n</li>\n<li>\n<p>The values have to come from the same distribution (although this requirement can be relaxed).</p>\n</li>\n<li>\n<p>The values have to be drawn from a distribution with finite mean and variance, so most Pareto distributions are out.</p>\n</li>\n<li>\n<p>The number of values you need before you see convergence depends on the skewness of the distribution. Sums from an exponential distribution converge for small sample sizes. Sums from a lognormal distribution do not.</p>\n</li>\n</ul>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The Central Limit Theorem explains, at least in part, the prevalence of normal distributions in the natural world. Most characteristics of animals and other life forms are affected by a large number of genetic and environmental factors whose effect is additive. The characteristics we measure are the sum of a large number of small effects, so their distribution tends to be normal."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Glossary"
        }
      ],
      "metadata": {
      }
    }
  ]
}