assignments/lab08/lab08.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Lab 8: Correlation, Variance of Sample Means\n",
                "\n",
                "Welcome to Lab 8!\n",
                "\n",
                "In today's lab, we will cover two relatively orthogonal concepts. First, we will investigate the variance of sample means, found in [Section 12.5](https:\/\/www.cs.cornell.edu\/courses\/cs1380\/2018sp\/textbook\/chapters\/12\/5\/variability-of-the-sample-mean.html) of our textbook. We will also get some hands-on practice with understanding the association between two variables, which you can read more about in [Section 13.1](https:\/\/www.cs.cornell.edu\/courses\/cs1380\/2018sp\/textbook\/chapters\/13\/1\/correlation.html)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Run this cell, but please don't change it.\n",
                "\n",
                "# These lines import the Numpy and Datascience modules.\n",
                "import numpy as np\n",
                "from datascience import *\n",
                "\n",
                "# These lines do some fancy plotting magic.\n",
                "import matplotlib\n",
                "%matplotlib inline\n",
                "import matplotlib.pyplot as plots\n",
                "plots.style.use('fivethirtyeight')\n",
                "import warnings\n",
                "warnings.simplefilter('ignore', FutureWarning)\n",
                "\n",
                "# This line loads the tests.\n",
                "from test import *"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 1. How Faithful is Old Faithful? \n",
                "\n",
                "(Note: clever title comes from [here](http:\/\/web.pdx.edu\/~jfreder\/M212\/oldfaithful.pdf).)\n",
                "\n",
                "Old Faithful is a geyser in Yellowstone National Park in the central United States.  It's famous for erupting on a fairly regular schedule.  You can see a video of it [here](https:\/\/www.youtube.com\/watch?v=wE8NDuzt8eg).\n",
                "\n",
                "Some of Old Faithful's eruptions last longer than others.  When it has a long eruption, there's generally a longer wait until the next eruption.\n",
                "\n",
                "If you visit Yellowstone, you might want to predict when the next eruption will happen, so you can see the rest of the park and come to see the geyser when it happens.  Today, we will use a dataset on eruption durations and waiting times to see if we can make such predictions accurately with linear regression.\n",
                "\n",
                "The dataset has one row for each observed eruption.  It includes the following columns:\n",
                "- **duration**: Eruption duration, in minutes\n",
                "- **wait**: Time between this eruption and the next, also in minutes\n",
                "\n",
                "Run the next cell to load the dataset."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "faithful = Table.read_table(\"faithful.csv\")\n",
                "faithful"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We would like to use linear regression to make predictions, but that won't work well if the data aren't roughly linearly related.  To check that, we should look at the data."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 1\n",
                "Make a scatter plot of the data.  It's conventional to put the column we will try to predict on the vertical axis and the other column on the horizontal axis."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "...\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 2\n",
                "Look at the scatter plot. Are eruption duration and waiting time roughly linearly related?  Is the relationship positive, as we claimed earlier?  You may want to consult [the textbook chapter 13](https:\/\/www.cs.cornell.edu\/courses\/cs1380\/2018sp\/textbook\/chapters\/13\/prediction.html) for the definition of \"linearly related.\""
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "*Write your answer here, replacing this text.*"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# DO NOT DELETE THIS CELL\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Check your answer with a neighbor or a TA.\n",
                "\n",
                "We're going to continue with the provisional assumption that they are linearly related, so it's reasonable to use linear regression to analyze this data.\n",
                "\n",
                "We'd next like to plot the data in standard units.  Recall that, if `nums` is an array of numbers, then\n",
                "\n",
                "    (nums - np.mean(nums)) \/ np.std(nums)\n",
                "\n",
                "...is an array of those numbers in standard units."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 3\n",
                "Compute the mean and standard deviation of the eruption durations and waiting times.  **Then** create a table called `faithful_standard` containing the eruption durations and waiting times in standard units.  (The columns should be named `\"duration (standard units)\"` and `\"wait (standard units)\"`."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "for_assignment_type": "student"
            },
            "outputs": [],
            "source": [
                "duration_mean = ...\n",
                "duration_std = ...\n",
                "wait_mean = ...\n",
                "wait_std = ...\n",
                "\n",
                "faithful_standard = Table().with_columns(\n",
                "    \"duration (standard units)\", ...,\n",
                "    \"wait (standard units)\", ...)\n",
                "\n",
                "\n",
                "faithful_standard"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "check1_3(duration_mean, duration_std, wait_mean, wait_std, faithful_standard)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "nbgrader": {
                    "grade": true,
                    "grade_id": "test1_3",
                    "locked": true,
                    "points": 1,
                    "schema_version": 1,
                    "solution": false
                }
            },
            "outputs": [],
            "source": [
                "#\n",
                "# AUTOGRADER TEST - DO NOT REMOVE\n",
                "#\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 4\n",
                "Plot the data again, but this time in standard units."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "...\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "You'll notice that this plot looks exactly the same as the last one!  The data really are different, but the axes are scaled differently.  (The method `scatter` scales the axes so the data fill up the available space.)  So it's important to read the ticks on the axes."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 5\n",
                "Among the following numbers, which would you guess is closest to the correlation between eruption duration and waiting time in this dataset?\n",
                "\n",
                "* -1\n",
                "* 0\n",
                "* 1"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "correlation_guess = ...\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "check1_5(correlation_guess)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "nbgrader": {
                    "grade": true,
                    "grade_id": "test1_5",
                    "locked": true,
                    "points": 1,
                    "schema_version": 1,
                    "solution": false
                }
            },
            "outputs": [],
            "source": [
                "#\n",
                "# AUTOGRADER TEST - DO NOT REMOVE\n",
                "#\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 6\n",
                "Compute the correlation `r`.  *Hint:* Use `faithful_standard`.  Section [13.1](https:\/\/www.cs.cornell.edu\/courses\/cs1380\/2018sp\/textbook\/chapters\/13\/1\/correlation.html) explains how to do this."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "r = ...\n",
                "r"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "check1_6(r)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "nbgrader": {
                    "grade": true,
                    "grade_id": "test1_6",
                    "locked": true,
                    "points": 1,
                    "schema_version": 1,
                    "solution": false
                }
            },
            "outputs": [],
            "source": [
                "#\n",
                "# AUTOGRADER TEST - DO NOT REMOVE\n",
                "#\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# 2. Variability of the Sample Mean\n",
                "\n",
                "By the Central Limit Theorem, the probability distribution of the mean of a large random sample is roughly normal. The bell curve is centered at the population mean. Some of the sample means are higher, and some lower, but the deviations from the population mean are roughly symmetric on either side, as we have seen repeatedly. Formally, probability theory shows that the sample mean is an unbiased estimate of the population mean.\n",
                "\n",
                "In our simulations, we also noticed that the means of larger samples tend to be more tightly clustered around the population mean than means of smaller samples. In this section, we will quantify the variability of the sample mean and develop a relation between the variability and the sample size.\n",
                "\n",
                "Let's take a look at the salaries of employees of the City of San Francisco in 2014. The mean salary reported by the city government was about $75463.92."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "salaries = Table.read_table('sf_salaries_2014.csv').select(\"salary\")\n",
                "salaries"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "salary_mean = np.mean(salaries.column('salary'))\n",
                "salary_mean"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "salaries.hist('salary', bins=np.arange(0, 300000+10000*2, 10000))\n",
                "plots.scatter(salary_mean, 0, marker='^', color='red', s=100);"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 1\n",
                "\n",
                "Clearly, the population does not follow a normal distribution. Keep that in mind as we progress through these exercises.\n",
                "\n",
                "Let's take random samples and look at the probability distribution of the sample mean. As usual, we will use simulation to get an empirical approximation to this distribution.\n",
                "\n",
                "We will define a function `simulate_sample_mean` to do this, because we are going to vary the sample size later. The arguments are the name of the table, the label of the column containing the variable, the sample size, and the number of simulations."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\"\"\"Empirical distribution of random sample means\"\"\"\n",
                "\n",
                "def simulate_sample_mean(table, label, sample_size, repetitions):\n",
                "    \n",
                "    means = make_array()\n",
                "\n",
                "    for i in np.arange(repetitions):\n",
                "        new_sample = ...\n",
                "        new_sample_mean = ...\n",
                "        means = ...\n",
                "\n",
                "    sample_means = Table().with_column('Sample Means', means)\n",
                "    \n",
                "    # Display empirical histogram and print all relevant quantities \u2013 don't change this!\n",
                "    sample_means.hist(bins=20)\n",
                "    plots.xlabel('Sample Means')\n",
                "    plots.title('Sample Size ' + str(sample_size))\n",
                "    print(\"Sample size: \", sample_size)\n",
                "    print(\"Population mean:\", np.mean(table.column(label)))\n",
                "    print(\"Average of sample means: \", np.mean(means))\n",
                "    print(\"Population SD:\", np.std(table.column(label)))\n",
                "    print(\"SD of sample means:\", np.std(means))"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Verify with your neighbor or TA that you've implemented the above function correctly. If you haven't implemented it correctly, the rest of the lab won't work properly, so this step is crucial."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 2\n",
                "\n",
                "In the following cell, we will create a sample of size 100 from the salaries table and graph it using our new `simulate_sample_mean` function."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Uncomment the following line to see the simulation:\n",
                "#simulate_sample_mean(salaries, 'salary', 100, 10000) \n",
                "plots.xlim(50000, 100000)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "In the following two cells, simulate the mean of a random sample of 400 salaries and 625 salaries, respectively. In each case, perform 10,000 repetitions of each of these processes. Don't worry about the `plots.xlim` line \u2013 it just makes sure that all of the plots have the same x-axis. "
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "for_assignment_type": "student"
            },
            "outputs": [],
            "source": [
                "# Uncomment the following line, and replace the ellipses with the appropriate arguments:\n",
                "#simulate_sample_mean(..., ..., ..., ...)\n",
                "plots.xlim(50000, 100000)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "for_assignment_type": "student"
            },
            "outputs": [],
            "source": [
                "# Uncomment the following line, and replace the ellipses with the appropriate arguments:\n",
                "#simulate_sample_mean(..., ..., ..., ...)\n",
                "plots.xlim(50000, 100000)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Write your conclusions about what you just saw in the below cell."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Conclusions:** ..."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# DO NOT DELETE THIS CELL\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Check your answer with a neighbor or a TA."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 3"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Assign the variable `bootstrap_sampled_SD` to the integer corresponding to your answer to the following question:\n",
                "\n",
                "When I increase the number of bootstrap samples that I take, for a fixed sample size, the SD of my sample mean will...\n",
                "\n",
                "1. Increase\n",
                "2. Decrease\n",
                "3. Stay about the same\n",
                "4. Vary widly"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "bootstrap_sampled_SD = ...\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "check2_3(bootstrap_sampled_SD)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "nbgrader": {
                    "grade": true,
                    "grade_id": "test2_3",
                    "locked": true,
                    "points": 1,
                    "schema_version": 1,
                    "solution": false
                }
            },
            "outputs": [],
            "source": [
                "#\n",
                "# AUTOGRADER TEST - DO NOT REMOVE\n",
                "#\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Below, we'll look at what happens when we take a fixed sample, then bootstrap from it with different numbers of resamples. How does the distribution of the resampled means change?"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Uncomment the following line to simulate with 500 repetitions:\n",
                "#simulate_sample_mean(salaries, 'salary', 100, 500)\n",
                "plots.xlim(50000, 100000)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Uncomment the following line to simulate with 1000 repetitions:\n",
                "#simulate_sample_mean(salaries, 'salary', 100, 1000)\n",
                "plots.xlim(50000, 100000)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Uncomment the following line to simulate with 5000 repetitions:\n",
                "#simulate_sample_mean(salaries, 'salary', 100, 5000)\n",
                "plots.xlim(50000, 100000)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Uncomment the following line to simulate with 10000 repetitions:\n",
                "#simulate_sample_mean(salaries, 'salary', 100, 10000)\n",
                "plots.xlim(50000, 100000)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "What did you notice about the sample means of the four bootstrapped samples above? Discuss with your neighbors. If you're unsure of your conclusion, ask your TA."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### Question 4\n",
                "Let's think about how the relationships between population SD, sample SD, and SD of sample means change with varying sample size. Which of the following is true? Again, assign the variable `pop_vs_sample` to the integer corresponding to your answer.\n",
                "\n",
                "1. Sample SD gets smaller with increasing sample size, SD of sample means gets smaller with increasing sample size\n",
                "2. Sample SD gets larger with increasing sample size, SD of sample means stays the same with increasing sample size\n",
                "3. Sample SD becomes more consistent with population SD with increasing sample size, SD of sample means gets smaller with increasing sample size\n",
                "4. Sample SD becomes more consistent with populatoin SD with increasing sample size, SD of smaple means stays the same with increasing sample size"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "pop_vs_sample = ...\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "check2_4(pop_vs_sample)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {
                "nbgrader": {
                    "grade": true,
                    "grade_id": "test2_4",
                    "locked": true,
                    "points": 1,
                    "schema_version": 1,
                    "solution": false
                }
            },
            "outputs": [],
            "source": [
                "#\n",
                "# AUTOGRADER TEST - DO NOT REMOVE\n",
                "#\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's see what happens: First, we calculate the population SD so that we can compare the SD of each sample to the SD of the population."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "pop_sd = np.std(salaries.column(\"salary\"))\n",
                "pop_sd"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's then how a small sample behaves. Run the following cells multiple times to see how the SD of the sample changes from sample to sample. Adjust the bins as necessary."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "sample_10 = salaries.sample(10)\n",
                "sample_10.hist(\"salary\")\n",
                "print(\"Sample SD: \", np.std(sample_10.column(\"salary\")))\n",
                "\n",
                "# Uncomment the following line to see the simulation:\n",
                "#simulate_sample_mean(sample_10, 'salary', 10, 1000)\n",
                "plots.xlim(5,120000)\n",
                "plots.ylim(0, .0001);"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "sample_200 = salaries.sample(200)\n",
                "sample_200.hist(\"salary\")\n",
                "print(\"Sample SD: \", np.std(sample_200.column(\"salary\")))\n",
                "\n",
                "# Uncomment the following line to see the simulation:\n",
                "#simulate_sample_mean(sample_200, 'salary', 200, 1000)\n",
                "plots.xlim(5,100000)\n",
                "plots.ylim(0, .00015);"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "sample_1000 = salaries.sample(1000)\n",
                "sample_1000.hist(\"salary\")\n",
                "print(\"Sample SD: \", np.std(sample_1000.column(\"salary\")))\n",
                "\n",
                "# Uncomment the following line to see the simulation:\n",
                "#simulate_sample_mean(sample_1000, 'salary', 1000, 1000)\n",
                "plots.xlim(5,100000)\n",
                "plots.ylim(0, .00025);"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's illustrate this trend. Below, you will see how the average absolute error of SD from the population changes with sample size (N)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Don't change this cell, just run it!\n",
                "sample_n_errors = make_array()\n",
                "for i in np.arange(10, 200, 10):\n",
                "    sample_n_errors = np.append(sample_n_errors, np.average([abs(np.std(salaries.sample(i).column(\"salary\"))-pop_sd)\n",
                "                                                      for d in np.arange(100)]))\n",
                "Table().with_column(\"Average absolute error in SD\", sample_n_errors, \"N\", np.arange(10, 200, 10)).plot(\"N\", \"Average absolute error in SD\")"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "You should notice that the distribution of means gets spiker, and that the distribution of the sample increasingly looks like the distribution of the population as we get to larger sample sizes. \n",
                "\n",
                "Is there a relationship between the sample size and absolute error in standard deviation? Identify this relationship \u2013 if you're having trouble, take a look at [Section 12.5](https:\/www.cs.cornell.edu\/courses\/cs1380\/2018sp\/textbook\/chapters\/12\/5\/variability-of-the-sample-mean.html) in our textbook."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 3. Submit\n",
                "\n",
                "Great job; you're done with this lab! There weren't many tests, but there were a lot of points at which you should've stopped and understood what exactly was going on. Consult the textbook if you have any outstanding questions.\n",
                "\n",
                "Before submitting, we recommend that you use the menu item Kernel -> Restart & Run All. That will re-run all your cells from scratch, just to make sure they all work as you are expecting.  Take a close look to make sure all your cells are still passing the checks.  Then, if they are, click the red Submit button."
            ]
        }
    ],
    "metadata": {
        "anaconda-cloud": [],
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text\/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.6.4"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}