ch03_cumulative_distribution_functions.ipynb

{
  "metadata": {
    "name": "Cumulative Distribution Functions"
  },
  "nbformat": 3,
  "nbformat_minor": 0,
  "worksheets": [
    {
      "cells": [
        {
          "cell_type": "heading",
          "level": 1,
          "metadata": {
          },
          "source": "Cumulative Distribution Functions"
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "The Class Size Paradox"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "At many American colleges and universities, the student-to-faculty ratio is about 10:1. But students are often surprised to discover that their average class size is bigger than 10. There are two reasons for the discrepancy:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ul>\n<li>\n<p>Students typically take 4–5 classes per semester, but professors often teach 1 or 2.</p>\n</li>\n<li>\n<p>The number of students who enjoy a small class is small, but the number of students in a large class is (ahem!) large.</p>\n</li>\n</ul>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The first effect is obvious (at least once it is pointed out); the second is more subtle. So let’s look at an example. Suppose that a college offers 65 classes in a given semester, with the following distribution of sizes:"
        },
        {
          "cell_type": "code",
          "collapsed": false,
          "input": " size      count\n 5–9           8\n10–14          8\n15–19         14\n20–24          4\n25–29          6\n30–34         12\n35–39          8\n40–44          3\n45–49          2",
          "language": "python",
          "metadata": {
          },
          "outputs": [

          ]
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "If you ask the Dean for the average class size, he would construct a PMF, compute the mean, and report that the average class size is 24."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "But if you survey a group of students, ask them how many students are in their classes, and compute the mean, you would think that the average class size was higher."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "The Limits of PMFs"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "PMFs work well if the number of values is small. But as the number of values increases, the probability associated with each value gets smaller and the effect of random noise increases."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "For example, we might be interested in the distribution of birth weights. In the NSFG data, the variable `totalwgt_oz` records weight at birth in ounces. Figure 3-1 shows the PMF of these values for first babies and others; it also illustrates a limitation of PMFs: they are hard to compare."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<figure id=\"nsfg_birthwgt_pmf\"><img src=\"files/figs/web/thst_0301.png\" alt=\"thst 0301\"><figcaption>PMF of birth weights</figcaption></figure>"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Overall, these distributions resemble the familiar “bell curve,” with many values near the mean and a few values much higher and lower."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "But parts of this figure are hard to interpret. There are many spikes and valleys, and some apparent differences between the distributions. It is hard to tell which of these features are significant. Also, it is hard to see overall patterns; for example, which distribution do you think has the higher mean?"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "These problems can be mitigated by binning the data; that is, dividing the domain into non-overlapping intervals and counting the number of values in each bin. Binning can be useful, but it is tricky to get the size of the bins right. If they are big enough to smooth out noise, they might also smooth out useful information."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "An alternative that avoids these problems is the _cumulative distribution function_, or _CDF_. But before we can get to that, we have to talk about percentiles."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Percentiles"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "If you have taken a standardized test, you probably got your results in the form of a raw score and a _percentile rank_. In this context, the percentile rank is the fraction of people who scored lower than you (or the same). So if you are “in the 90th percentile,” you did as well as or better than 90% of the people who took the exam."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Here’s how you could compute the percentile rank of a value, `your_score` , relative to the scores in the sequence `scores` :"
        },
        {
          "cell_type": "code",
          "collapsed": false,
          "input": "def PercentileRank(scores, your_score):\n    count = 0\n    for score in scores:\n        if score &lt;= your_score:\n            count += 1\n\n    percentile_rank = 100.0 * count / len(scores)\n    return percentile_rank",
          "language": "python",
          "metadata": {
          },
          "outputs": [

          ]
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "For example, if the scores in the sequence were 55, 66, 77, 88, and 99, and you got the 88, then your percentile rank would be `100 \\* 4 / 5` which is 80."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "If you are given a value, it is easy to find its percentile rank; going the other way is slightly harder. If you are given a percentile rank and you want to find the corresponding value, one option is to sort the values and search for the one you want:"
        },
        {
          "cell_type": "code",
          "collapsed": false,
          "input": "def Percentile(scores, percentile_rank):\n    scores.sort()\n    for score in scores:\n        if PercentileRank(scores, score) &gt;= percentile_rank:\n            return score",
          "language": "python",
          "metadata": {
          },
          "outputs": [

          ]
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The result of this calculation is a _percentile_. For example, the 50th percentile is the value with percentile rank 50. In the distribution of exam scores, the 50th percentile is 77."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Cumulative Distribution Functions"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Now that we understand percentiles, we are ready to tackle the cumulative distribution function (CDF). The CDF is the function that maps values to their percentile rank in a distribution."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The CDF is a function of _x_, where _x_ is any value that might appear in the distribution. To evaluate CDF(_x_) for a particular value of _x_, we compute the fraction of the values in the sample less than (or equal to) _x_."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Here’s what that looks like as a function that takes a sample, `t` , and a value, `x` :"
        },
        {
          "cell_type": "code",
          "collapsed": false,
          "input": "def Cdf(t, x):\n    count = 0.0\n    for value in t:\n        if value &lt;= x:\n            count += 1.0\n\n    prob = count / len(t)\n    return prob",
          "language": "python",
          "metadata": {
          },
          "outputs": [

          ]
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "This function should look familiar; it is almost identical to `PercentileRank` , except that the result is in a probability in the range 0–1 rather than a percentile rank in the range 0–100."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "As an example, suppose a sample has the values {1, 2, 2, 3, 5}. Here are some values from its CDF:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "CDF(0) = 0  \nCDF(1) = 0.2  \nCDF(2) = 0.6  \nCDF(3) = 0.8  \nCDF(4) = 0.8  \nCDF(5) = 1"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "We can evaluate the CDF for any value of _x_, not just values that appear in the sample. If _x_ is less than the smallest value in the sample, CDF(_x_) is 0. If _x_ is greater than the largest value, CDF(_x_) is 1."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Figure 3-2 is a graphical representation of this CDF. The CDF of a sample is a step function. In the next chapter, we will see distributions whose CDFs are continuous functions."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<figure id=\"example_cdf\"><img src=\"files/figs/web/thst_0302.png\" alt=\"thst 0302\"><figcaption>Example of a CDF</figcaption></figure>"
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Representing CDFs"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "I have written a module called `Cdf` that provides a class named `Cdf` that represents CDFs. You can read the documentation of this module at [http://thinkstats.com/Cdf.html](http://thinkstats.com/Cdf.html) and you can download it from [http://thinkstats.com/Cdf.py](http://thinkstats.com/Cdf.py)."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Cdfs are implemented with two sorted lists: `xs` , which contains the values, and `ps` , which contains the probabilities. The most important methods Cdfs provide are:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Because `xs` and `ps` are sorted, these operations can use the bisection algorithm, which is efficient. The run time is proportional to the logarithm of the number of values; see [http://wikipedia.org/wiki/Time\\_complexity](http://wikipedia.org/wiki/Time_complexity)."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Cdfs also provide `Render` , which returns two lists, `xs` and `ps` , suitable for plotting the CDF. Because the CDF is a step function, these lists have two elements for each unique value in the distribution."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "The Cdf module provides several functions for making Cdfs, including `MakeCdfFromList` , which takes a sequence of values and returns their Cdf."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Finally, `myplot.py` provides functions named `Cdf` and `Cdfs` that plot Cdfs as lines."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Random Numbers"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "CDFs are useful for generating random numbers with a given distribution. Here’s how:"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "<ul>\n<li>\n<p>Choose a random probability in the range 0–1.</p>\n</li>\n<li>\n<p>Use <code>Cdf.Value</code> to find the value in the distribution that corresponds to the probability you chose.</p>\n</li>\n</ul>\n"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "It might not be obvious why this works, but since it is easier to implement than to explain, let’s try it out."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        },
        {
          "cell_type": "heading",
          "level": 2,
          "metadata": {
          },
          "source": "Summary Statistics Revisited"
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "Once you have computed a CDF, it is easy to compute other summary statistics. The median is just the 50th percentile. [4](ch03.html#idp431248)The 25th and 75th percentiles are often used to check whether a distribution is symmetric, and their difference, which is called the _interquartile range_, measures the spread."
        },
        {
          "cell_type": "markdown",
          "metadata": {
          },
          "source": "."
        }
      ],
      "metadata": {
      }
    }
  ]
}