kp_inferential_statistics_2-Q.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inferential statistics II - Bootstrapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous frequentist mini-projects, you did frequentist calculations to perform inference from a sample of data. Such inference relies on theory largely developed from the 19th-Century onwards that is subject to certain assumptions or theoretical limits. These are fine if those assumptions hold for the particular case you're working on, and what you want to do has a known theoretical distribution (for example the mean of a sampling distribution that we looked at in the previous mini-project.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this mini-project, you'll use the same medical charge data you used in the frequentist inference mini-project, but this time you'll make inferences about the population using bootstrapping (ie. simulating repeated re-runs of an experiment.) If frequentism is about using assumptions and theoretical results to calculate what we expect to happen were an experiment to be run again and again and again, then bootstrapping is about using computing power to essentially re-run the sample draw again and again and again to see what actually happens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While these exercises do not strictly depend on these concepts, we encourage you to complete the previous mini-projects before starting this one so that you can approach this assignment with a good understanding of frequentist concepts like:\n",
    "* the _z_-statistic\n",
    "* the _t_-statistic\n",
    "* the difference and relationship between the two\n",
    "* the Central Limit Theorem, its assumptions and consequences\n",
    "* how to estimate the population mean and standard deviation from a sample\n",
    "* the concept of a sampling distribution of a test statistic, particularly for the mean\n",
    "* how to combine these concepts to calculate confidence intervals and p-values\n",
    "* how those confidence intervals and p-values allow you to perform hypothesis (or A/B) tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete mini-project, it's important that you first complete the bootstrap resources listed in this subunit, as they contain valuable information about how to calculate bootstrap replicates of summary statistics. Having an basic understanding of what confidence intervals and p-values are will also be helpful (we touch on them in this mini-project, but please speak to your mentor or conduct individual research if you'd like to learn more.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import t\n",
    "import scipy.stats as stats\n",
    "import numpy as np\n",
    "from numpy.random import seed\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Medical charge data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "med_charges = pd.read_csv('data/insurance2.csv')\n",
    "med_charges_1d = med_charges['charges']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>bmi</th>\n",
       "      <th>children</th>\n",
       "      <th>smoker</th>\n",
       "      <th>region</th>\n",
       "      <th>charges</th>\n",
       "      <th>insuranceclaim</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>27.900</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>16884.92400</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>33.770</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1725.55230</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>33.000</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>4449.46200</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>22.705</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>21984.47061</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "      <td>28.880</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3866.85520</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex     bmi  children  smoker  region      charges  insuranceclaim\n",
       "0   19    0  27.900         0       1       3  16884.92400               1\n",
       "1   18    1  33.770         1       0       2   1725.55230               1\n",
       "2   28    1  33.000         3       0       2   4449.46200               0\n",
       "3   33    1  22.705         0       0       1  21984.47061               0\n",
       "4   32    1  28.880         0       0       1   3866.85520               1"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "med_charges.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous assignment, you used the frequentist approach to estimate the lower limit for the 95% confidence interval on the mean hospital charge. This approach relies on statistical theory that has been developed over the years and is also limited to statistics for which theoretical results on the sampling distribution exist. These results are remarkably useful and applicable much of the time and under a surprisingly wide range of conditions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having calculated the 95% lower confidence interval using frequentist theory in the previous exercise, you'll now use bootstrap inference to verify your calculations and check that you get consistent results without making the assumptions required before. After all, the distribution of charges really was very non-normal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Q:__ Use bootstrap sampling to estimate the same 95% confidence interval lower limit as before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__A:__ The 95% confidence interval lower limit is 12724.4679217"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([12724.4679217])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(47)\n",
    "N_rep = 10000\n",
    "\n",
    "#draw bs replicates of 1d data array\n",
    "def bootstrap_replicates_1d(data,func):\n",
    "    bs_sample = np.random.choice(data,len(data))\n",
    "    return func(bs_sample)\n",
    "#draw bs replicates\n",
    "def bs_reps(data,func,size=1):\n",
    "    bs_replicate = np.empty(N_rep)\n",
    "    \n",
    "    #Generate replicates\n",
    "    for i in range(size):\n",
    "        bs_replicate[i] = bootstrap_replicates_1d(data,func)\n",
    "        \n",
    "    return bs_replicate\n",
    "\n",
    "#calculate mean\n",
    "bs_reps_charges = bs_reps(med_charges_1d,np.mean,size=10000)\n",
    "np.mean(bs_reps_charges)\n",
    "\n",
    "#calculate 95% confidence interval\n",
    "ci_bs_reps_charges = np.percentile(bs_reps_charges,[5])\n",
    "ci_bs_reps_charges\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you performed 10000 replicates immediately after setting the random seed to 47, you should get the value 12724 here, which compares very well with the value 12725 obtained using the _t_-distribution confidence interval previously. It is a most pleasant result to see the predictions of classical frequentist theory match with results that are now possible through the number-crunching ability of computers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, in the previous mini-projects, we saw that there are two ways of performing a _t_-test from a sample, depending on whether we can assume the groups have equal variance or not. We can actually easily test this using the bootstrap approach!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Q:__ Calculate the 95% confidence interval for the difference between the standard deviations of insurance and non-insurance claim charges (insured - non-insured). Calculate the differences over 10000 replicates. Plot the histogram of values and mark the locations of the percentiles. State the null and alternative hypothesis and comment on whether you would retain or reject the null hypothesis in this case and why."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__A:__ <b>Null Hypothesis</b>: There is no distinguishable difference between the standard deviation of the medical charges of patients with insurance, than the medical charges of patients without insurance.\n",
    "<b>Alternative Hypothesis</b>: There is a distinguishable difference between the standard deviation of the medical charges of patients with insurance and medical charges of patients without insurance.\n",
    "\n",
    "I chose to test the null hypothesis with a t test, because a t test is able to tell how significant the differences between two groups are. Completing this t test will give us the p value, which will be an indication as to whether or not there is variance in insured and uninsured charges.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#separate insured from uninsured\n",
    "not_insured = med_charges[med_charges['insuranceclaim']==0]['charges']\n",
    "insured = med_charges[med_charges['insuranceclaim']==1]['charges']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(47)\n",
    "N_rep = 10000\n",
    "#initilize empty array for replicates\n",
    "b_reps = np.empty(N_rep)\n",
    "for i in range(N_rep):\n",
    "    bs0 = np.random.choice(not_insured, size = len(not_insured))\n",
    "    bs1 = np.random.choice(insured, size = len(insured))\n",
    "    b_reps[i] = np.std(bs1) - np.std(bs0)\n",
    "#calculate 95% confidence interval\n",
    "lval = np.percentile(b_reps, 2.5)\n",
    "uval = np.percentile(b_reps, 97.5)\n",
    "\n",
    "diff = np.mean(bs0) - np.mean(bs1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJzt3Xm4HGWZ/vHvbQJhJ4EcMGQhIJF1EDFsrhlBNhmCM6DBQQIuGRREXIHxN7LMOMKMDg7qgEGQMMomLmQQhQgERGUJCGGXI0tyssDBQABZE57fH+/bpNLp7nPq5PTpDuf+XFdfXf3WW1VPLV1P7aWIwMzMrLfe1OoAzMxszeLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHEMAEmTJHX1Y//GSwpJQ/urnyWGPU7S85KG9LH7f5b0g/6Oq5fDXil2SZtLuknSc5K+peSHkp6WdFsrYrRV5WV9m1bH0VuS/lHSta2OoyeSZkv6ZF+6HTSJQ9K7Jf1e0lJJSyT9TtJuud1Rkm5udYz9RdJjkl7MK8Rn8ngfI2m153dEzIuIDSJieS/iWCVhRsS/R0SfFtYehnWUpOU5MTwv6dGcBN7aIPZpwFPARhHxReDdwAeAMRGxe3/H2G5WZ8XxRlZrupTZ+IuIH0fEvs2Jrj0MisQhaSPgKuA7wCbAaOA04OVWxtUbq7FX8XcRsSGwJXAGcCJwfr8F1p7+EBEbABsD+wAvAndI2qlO/S2B+2PFXbBbAo9FxF/LDrgVe39ror7uqVpjA778RcQb/gNMBJ6p02574CVgOfB8pR7wQeCPwLPAfODUQjfjgQCmAvNIW61fLbRfF7gQeBq4H/gy0FVofxLwZ+C53P5DhXZHAb8DzgKWAP8GDAG+mYfzCHBsHv7QOuP0GLBPVdnuwGvATvn3sNzPecATwLnAurndA8BBhW6H5mHvWhj3obnd0bn+czm2f8rl65NW3K/l6fo8sAVwKvCjQr8PBu4DngFmA9tXjceXgLnAUuAyYJ0643wUcHON8quAK6rm29A8f14FXsmx/VPVcnBa7uYg4K4c3++BnaviOzHH93Lu7xbAT4Fu4FHg+EL9U4HLgYvy9LoPmFhoPxb4We72L8B3C+0+nqfz08A1wJa5XKRl5ck8jeZW5nEv/hezgU/m5klAF/DF3K9FwNGFugeSltXngAXAl+pN9zyNt8nNFwLnAFcDfyUl9LrLXu7my3n4C/N4v96/quFMAeZUlX0emNko5jLTpVA2iZX/wwEcAzyc58n3ANWaJj3U3Qa4Mc+7p4DLqpfVOvPrKKrWE42Wk9zuA8CDeVjfzcP9ZG+mySrTqC8drWkfYCPSH3EGcAAwoqp9rYV/EvA3pL2ynfMCfkjVTD2PlCTeRlpxbJ/bnwH8lrR3Mxa4t2qhO4y0gnkT8JH8hxpViGUZ8FnSimjdvNA9mPu1CXBD9UJVFftjVCWOXD4P+HRu/jYwM/dvQ+D/gG/kdl8Dflzo7oPAg7UW6NzuLaQV2PuAF4Bda/3Zctmp5MQBvDWP+weAtYCvAJ3A2oXxuC1Pq03yH+KYOuO8yjws/JGeqBP7heQ/XJ0//K6klegepOQ9Ncc0rBDfXXm+rJvn5x15+q0NbE1KpvsVxv0l0gptCPAN4JbcbghwN2lFsD6wDvDu3O6QPF22z8vE/wN+n9vtl4c5PM+D7VmxLH0UmNvgfzGblRPHMuD0PC8OzPNyRG6/CHhPbh5RmMerTHdWTRxLgXfl6bMOjZe9/Un/tZ3ydLiY+oljPVJSmFAoux2Y0ijmXqwvXp8uVeuD6sRxVZ7u40jJfv86y1GjupcAXy1Mm8o8H0/PiaN6PdFoORlJ2gg+NM/fz+fu+5Q4BsWhqoh4lnT8urKy75Y0U9LmDbqZHRH3RMRrETGXNIPfV1XttIh4MSLuJv3p35bLPwx8PSKWRMR84Oyqfv8kIhbmfl9G2hIpHlNfGBHfiYhlEfFi7t+3I2J+RCwhrXD6YiGwiSQBnwI+n2N8Dvh30hYcpD/rwZLWy78/mstWERG/jIg/R3IjcC3wnl7G8xHglxExKyJeJW2Frgu8s1Dn7DytlpBWMLv0emyThaQVVF98Cvh+RNwaEcsjYgZpA2HPqvjm5/m0G9AREadHxCsR8QhpeZtSqH9zRFwd6TzL/7JimdmdlCC/HBF/jYiXIqJy3u2fSCvWByJiGWle7SJpS9Je04bAdqSt2AciYhFARFwcETuXGN9XgdMj4tWIuJq057Vtod0OkjaKiKcj4s4S/b0yIn4XEa+Rpl+jZe/DwA8j4t5IhwxPrdfTiHgBuBI4HEDSBNJ0mNkPMffGGRHxTETMI23MNVo269V9lXSIdIuqed4b1euJRsvJgaTDslfk/9q3gcWlxrZgUCQOgDwxj4qIMaStmS1IE68mSXtIukFSt6SlpK3+kVXVihP+BWCD3LwF6fBWxeNV/T5S0l35xPUzOZ5iv4vd9ti/EkaTdms7SFtrdxRi+HUuJyI6SVv3f5eTx8HUSRySDpB0S77g4BnSAlo9nerZojguecUyP8dZUW8a91ZlnPtiS+CLlWmUx28sKe6K+VX1t6iq/89AcQOlenzWycenxwKP5z98rTj+u9DPJaS9i9ERcT3psMP3gCckTc/n9PriL1XDL07vfyDN28cl3ShprxL9LU6jhsse5Zf1i8mJg7SB84ucUFYn5mWkrfKitUgr+aIyy2a9ul8hzcvbJN0n6eO9jBFWXU/UXU6omq6RdkOqu++1QZM4iiLiQdIudOWkadSodjFpy2VsRGxMOg6rXg5iEWlFUDGu0pCz/3nAccCmETGcdCir2O/qeOr2r7fyFWSjgZtJx1JfBHaMiOH5s3GkE8sVl5D+kJNJWyqdNfo5jHQ8/5vA5nlcri6MS63pWrSQtLBX+ifSeC4oO34NfIh02LAv5pP2HIcXPutFxCWFOlFV/9Gq+htGxIG9HNa4Oic555POHRX7u25E/B4gIs6OiHcAO5IO/325T2PbQETcHhGTgc2AX5DO1UA61FjZM0XSm2t1Xmjuadkru6xfC4yUtAtpeX19A6dBzD2ZRzpUVLQVfd9gqysiFkfEpyJiC9Iew//kS48rF2isV6hePW2r/1+NlpOVpmvhv9YngyJxSNpO0hcljcm/x5IWsltylSeAMZLWLnS2IbAkIl6StDtpa6a3LgdOljQiD/OzhXbrk2Z4d47laFYksEb9O17SGEkjSCfXe0XSRpIOAi4lnVu4J2/ZnwecJWmzXG+0pP0KnV4K7At8mjp7G6Tj+MPyuCyTdEDupuIJYFNJGzcYrw9K2lvSWqQTsy+TTkL3maQhkraS9B3SsenT+tir84Bj8t6nJK0v6YOSNqxT/zbgWUknSlo3x7FT5bLvHtxG+nOfkYezjqR35XbnkpanHfP4bSzpsNy8W45vLdLKpnKCv99IWlvp3oSN82GOZwvDuBvYUdIuktahwaEleH2vstGydzlwlKQd8t7uKT30bxlwBfCfpEOSs3oRc08uA46WtHue728lnRO4tJfd95qkwyrrJdIJ7QCWR0Q3aQPqiLwcfZx0LrGRussJ8EvSfPr7vHFyPKsmol4bFImDdAJtD+BWSX8lJYx7SSsqgOtJV7gslvRULvsMcLqk50gnO3u7tQJpRfU46aqaa0nHsgGIiPuBbwF/IK1Y/4Z0dUQj55GukLgbuJN05U1P/i/HPp908u2/SFdAVZxIOpF2i6Rngd+w4ng2+Tj5H0jnGy6rNYB8fPp40rR5mpRcZxbaP0jac3kk7z5vUdX9Q8ARpMuknwL+jnQZ8Su9GL9a9pL0PGklMZt0UcRuEXFPX3oWEXNIx+O/Sxq/TtJJyXr1l5PGYRfSvH8K+AHp8uCehlXpdhvSFm8X6RwQEfFz4Ezg0jyv7iVd5AFpHM/L8T1Ougjkm/D6jWj3lRjlRj4GPJaHfwxpvhERfyKdUP8N6Vxdb47R1132IuJXpEPI1+c61/eifxeTrtb6SdWhtpoxa8WNoDX3ZiLiGtLG2Q9JJ/avJl1YM70XsZS1G2m99Dzpv/O5iHg0t/sUae/xL6S9yYYbVI2Wk4h4inRRzhm5fxPoeb1TV+WSMDMzs14ZLHscZmbWT5w4zMysFCcOMzMrxYnDzMxKeUM+mG3kyJExfvz4Voex5nvoofS97baN69nrHsrTbFtPs/q8XLWtO+6446mI6Oip3hsycYwfP545c+a0Oow136RJ6Xv27FZGsUaZlKfZbE+z+rxctS1JvbrJ0YeqzMysFCcOMzMrxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KaljgkXSDpSUn3VpV/VtJD+TWJ/1EoP1lSZ263X6F8/1zWKanXLzAyM7PmaOad4xeSXoBzUaVA0t+SXkW6c0S8XHgD2A6kl9XvSHo37m/yW7cgvUv5A6QX29wuaWZ+GZJZy4waM47FC+q/sjm9mbO2N48ey6Kuec0Iy2xANC1xRMRNksZXFX8aOCMiXs51nszlk4FLc/mjkjqB3XO7zoh4BEDSpbmuE4e11OIF89nyxKtWLb847RS/+aNn1O328TMPalpcZgNhoM9xvBV4j6RbJd1YeBfzaNIrTiu6clm98lVImiZpjqQ53d3dTQjdzMxg4BPHUGAEsCfpXbqXK+3T19qvjwblqxZGTI+IiRExsaOjx4c7mplZHw3003G7gJ9FetH5bZJeA0bm8rGFemOAhbm5XrmZmbXAQO9x/AJ4P0A++b028BQwE5giaZikrYAJwG3A7cAESVtJWpt0An3mAMdsZmYFTdvjkHQJMAkYKakLOAW4ALggX6L7CjA1733cJ+ly0knvZcCxEbE89+c44BpgCHBBRNzXrJjNzKxnzbyq6vA6rY6oU//rwNdrlF8NXN2PoZmZ2WrwneNmZlaKE4fZQBuyFpL69Bk1Zlyrozd7Y75z3KytLX+15s2DveGbB60deI/DzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrxYnDBqVRY8b1+Qm1klodvllL+em4NigtXjC/z0+oBT+l1ga3pu1xSLpA0pP5NbHV7b4kKSSNzL8l6WxJnZLmStq1UHeqpIfzZ2qz4jUzs95p5qGqC4H9qwsljQU+AMwrFB8ATMifacA5ue4mpHeV7wHsDpwiaUQTYzYzsx40LXFExE3AkhqtzgK+AkShbDJwUSS3AMMljQL2A2ZFxJKIeBqYRY1kZGZmA2dAT45LOhhYEBF3V7UaDcwv/O7KZfXKa/V7mqQ5kuZ0d3f3Y9RmZlY0YIlD0nrAV4Gv1WpdoywalK9aGDE9IiZGxMSOjo6+B2pmZg0N5B7HW4CtgLslPQaMAe6U9GbSnsTYQt0xwMIG5WZm1iIDljgi4p6I2CwixkfEeFJS2DUiFgMzgSPz1VV7AksjYhFwDbCvpBH5pPi+uczMzFqkmZfjXgL8AdhWUpekTzSofjXwCNAJnAd8BiAilgD/CtyeP6fnMjMza5Gm3QAYEYf30H58oTmAY+vUuwC4oF+DMzOzPvMjR8zMrBQnDjMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpr5BsALJD0p6d5C2X9KelDSXEk/lzS80O5kSZ2SHpK0X6F8/1zWKemkZsVrZma908w9jguB/avKZgE7RcTOwJ+AkwEk7QBMAXbM3fyPpCGShgDfAw4AdgAOz3XNzKxFmpY4IuImYElV2bURsSz/vAUYk5snA5dGxMsR8Sjp3eO7509nRDwSEa8Al+a6ZmbWIq08x/Fx4Fe5eTQwv9CuK5fVK1+FpGmS5kia093d3YRwzcwMWpQ4JH0VWAb8uFJUo1o0KF+1MGJ6REyMiIkdHR39E6iZma1i6EAPUNJU4CBg74ioJIEuYGyh2hhgYW6uV25mZi0woHsckvYHTgQOjogXCq1mAlMkDZO0FTABuA24HZggaStJa5NOoM8cyJjN2sqQtZDUp8+oMeNaHb29QTRtj0PSJcAkYKSkLuAU0lVUw4BZkgBuiYhjIuI+SZcD95MOYR0bEctzf44DrgGGABdExH3Nitms7S1/lS1PvKpPnT5+5kH9HIwNVk1LHBFxeI3i8xvU/zrw9RrlVwNX92NoZma2GnznuJmZleLEYWZmpThx2Bpr1JhxfT5RbGZ9N+CX45r1l8UL5vtEsVkLeI/DzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrxYnDzMxKaVrikHSBpCcl3Vso20TSLEkP5+8RuVySzpbUKWmupF0L3UzN9R+WNLVZ8ZqZWe80c4/jQmD/qrKTgOsiYgJwXf4NcAAwIX+mAedASjSkd5XvAewOnFJJNmZm1hpNSxwRcROwpKp4MjAjN88ADimUXxTJLcBwSaOA/YBZEbEkIp4GZrFqMjIzswE00Oc4No+IRQD5e7NcPhqYX6jXlcvqla9C0jRJcyTN6e7u7vfAzcwsaZeT47Xe5RkNylctjJgeERMjYmJHR0e/BmdmZisMdOJ4Ih+CIn8/mcu7gLGFemOAhQ3KzcysRQY6ccwEKldGTQWuLJQfma+u2hNYmg9lXQPsK2lEPim+by4zM7MWGdqsHku6BJgEjJTURbo66gzgckmfAOYBh+XqVwMHAp3AC8DRABGxRNK/ArfneqdHRPUJdzMzG0BNSxwRcXidVnvXqBvAsXX6cwFwQT+GZmZmq6FdTo6bmdkawonDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1IaJg5J1xaaT25+OGZm1u562uMoPi3wsLq1zMxs0OgpcdR8Eq2ZmQ1ePT1yZGtJM0mPN680vy4iDm5aZGZm1pZ6ShyTC83fbGYgZma2ZmiYOCLixkqzpI5c5tfrmZkNYj1dVSVJp0h6CngQ+JOkbklfG5jwzMys3fR0cvwE4N3AbhGxaUSMAPYA3iXp802PzszM2k5PieNI4PCIeLRSEBGPAEfkdmZmNsj0lDjWioinqgvzeY61+jpQSZ+XdJ+keyVdImkdSVtJulXSw5Iuk7R2rjss/+7M7cf3dbhmZrb6ekocr/SxXV2SRgPHAxMjYidgCDAFOBM4KyImAE8Dn8idfAJ4OiK2Ac7K9czMrEV6Shxvk/SspOfy59nKb+BvVmO4Q4F1JQ0F1gMWAe8HrsjtZwCH5ObJ+Te5/d6StBrDNjOz1dDT5bhD+nuAEbFA0jeBecCLwLXAHcAzEbEsV+sCRufm0cD83O0ySUuBTYFVDqGZmVnz9XQ57jqSTpD0XUnT8h7CapE0grQXsRWwBbA+cECNqpXHndTau1jlUSg5vjmS5nR3+1YTM7Nm6elQ1QxgInAPcCDwrX4Y5j7AoxHRHRGvAj8D3gkMLySmMcDC3NwFjAXI7TcGllT3NCKmR8TEiJjY0dFR3drMzPpJT4ljh4g4IiK+DxwKvKcfhjkP2FPSevlcxd7A/cANeRgAU4Erc/PM/Jvc/vqI8MMXzcxapKfE8WqloXD+YbVExK2kk9x3kvZk3gRMB04EviCpk3QO4/zcyfnAprn8C8BJ/RGHmZn1TU/nLN4m6dncLNKVUM/m5oiIjfoy0Ig4BTilqvgRYPcadV/C7wIxM2sbA35VlVnRqDHjWLxgfqvDMLMSVvsqKbPVsXjBfLY88ao+dfv4mQf1czRvcEPWYnVugXrz6LEs6prXjwHZmsqJw2ywWP5qn5M0OFHbCj2dHDczM1uJE4eZmZXixGFmZqU4cZiZWSlOHGZmVooTh5mZleLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXSksQhabikKyQ9KOkBSXtJ2kTSLEkP5+8Rua4knS2pU9JcSbu2ImYzM0tatcfx38CvI2I74G3AA6R3iV8XEROA61jxbvEDgAn5Mw04Z+DDNTOzigFPHJI2At4LnA8QEa9ExDPAZGBGrjYDOCQ3TwYuiuQWYLikUQMctpmZZa3Y49ga6AZ+KOmPkn4gaX1g84hYBJC/N8v1RwPFl1J35bKVSJomaY6kOd3d3c0dAzOzQawViWMosCtwTkS8HfgrKw5L1VLrJcmxSkHE9IiYGBETOzo6+idSMzNbRSsSRxfQFRG35t9XkBLJE5VDUPn7yUL9sYXuxwALByhWMzOrMuCJIyIWA/MlbZuL9gbuB2YCU3PZVODK3DwTODJfXbUnsLRySMvMzAbe0BYN97PAjyWtDTwCHE1KYpdL+gQwDzgs170aOBDoBF7Idc3MrEVakjgi4i5gYo1We9eoG8CxTQ/KzMx6xXeOm5lZKU4cZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXixGFmZqU4cZiZWSlOHGZmVooTh5mZleLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHHYahs1ZhyS+vQxszVPq14di6QhwBxgQUQcJGkr4FJgE+BO4GMR8YqkYcBFwDuAvwAfiYjHWhS21bB4wXy2PPGqPnX7+JkH9XM01jRD1upzsn/z6LEs6prXzwFZq7QscQCfAx4ANsq/zwTOiohLJZ0LfAI4J38/HRHbSJqS632kFQGbDWrLX/UGggEtOlQlaQzwQeAH+beA9wNX5CozgENy8+T8m9x+b/kYh5lZy7TqHMe3ga8Ar+XfmwLPRMSy/LsLGJ2bRwPzAXL7pbn+SiRNkzRH0pzu7u5mxm5mNqgNeOKQdBDwZETcUSyuUTV60W5FQcT0iJgYERM7Ojr6IVIzM6ulFec43gUcLOlAYB3SOY5vA8MlDc17FWOAhbl+FzAW6JI0FNgYWDLwYZuZGbRgjyMiTo6IMRExHpgCXB8R/wjcAByaq00FrszNM/NvcvvrI2KVPQ4zMxsY7XQfx4nAFyR1ks5hnJ/Lzwc2zeVfAE5qUXxmZkZrL8clImYDs3PzI8DuNeq8BBw2oIGZmVld7bTHYWZmawAnDjMzK8WJw8zMSnHiMDOzUpw4zMysFCcOMzMrxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4jAzs1KcOMzMrBQnDjMzK6Wlj1W39jFqzDgWL5i/UtkN+ftvVevtvWY2WDlxGACLF8xnyxOvWqlsnYvTO7O2/OgZDbt9/MyDmhaXmbWfAT9UJWmspBskPSDpPkmfy+WbSJol6eH8PSKXS9LZkjolzZW060DHbGZmK7TiHMcy4IsRsT2wJ3CspB1Ir4S9LiImANex4hWxBwAT8mcacM7Ah2xmZhUDnjgiYlFE3JmbnwMeAEYDk4EZudoM4JDcPBm4KJJbgOGSRg1w2GZmlrX0qipJ44G3A7cCm0fEIkjJBdgsVxsNFM/aduWy6n5NkzRH0pzu7u5mhm1mNqi1LHFI2gD4KXBCRDzbqGqNslilIGJ6REyMiIkdHR39FaaZmVVpSeKQtBYpafw4In6Wi5+oHILK30/m8i5gbKHzMcDCgYrVzMxW1oqrqgScDzwQEf9VaDUTmJqbpwJXFsqPzFdX7QksrRzSMrM1xJC1kIQkZt94I7NvvPH13z19Ro0Z1+rorUor7uN4F/Ax4B5Jd+WyfwbOAC6X9AlgHnBYbnc1cCDQCbwAHD2w4ZrZalv+6uv3CfX2/qAK3yfUfgY8cUTEzdQ+bwGwd436ARzb1KDMzKzX/KwqMzMrxYnDzMxKceIwM7NSnDjMzKwUJw4zMyvFicPMzEpx4niDGDVmXK9vqKr1MTPrLb/I6Q2i1ouYyvBNVta28l3nffHm0WNZ1DWvnwMyJw4za2+Fu87L8gZRc/hQlZmZleLEYWZmpThxmJlZKU4cZmZWihOHmZmV4sRhZmalOHG0kdW5ic/Maii8ebDsx28erM/3cbSR1bmJz9erm9Xge0CaYo3Z45C0v6SHJHVKOqnV8ZiZDVZrROKQNAT4HnAAsANwuKQdWhuVmdngtEYkDmB3oDMiHomIV4BLgcnNGtjqnGsYOmxdn6cweyNYjfMjq7suaPfzK4qIVsfQI0mHAvtHxCfz748Be0TEcYU604Bp+ee2wEMDHOZI4KkBHmZZ7R5ju8cHjrG/OMb+0d8xbhkRHT1VWlNOjtfaHF8p40XEdGD6wISzKklzImJiq4bfG+0eY7vHB46xvzjG/tGqGNeUQ1VdwNjC7zHAwhbFYmY2qK0pieN2YIKkrSStDUwBZrY4JjOzQWmNOFQVEcskHQdcAwwBLoiI+1ocVrWWHSYrod1jbPf4wDH2F8fYP1oS4xpxctzMzNrHmnKoyszM2oQTh5mZleLE0YCk4ZKukPSgpAck7SVpE0mzJD2cv0fkupJ0dn4kylxJuxb6MzXXf1jS1AGI8VRJCyTdlT8HFuqfnGN8SNJ+hfKmPNJF0raFOO6S9KykE9ppOjaIsZ2m4+cl3SfpXkmXSFonXyxya54el+ULR5A0LP/uzO3H9xR3E2O8UNKjhWm4S67bqv/L53J890k6IZe1zbLYIMa2WRYBiAh/6nyAGcAnc/PawHDgP4CTctlJwJm5+UDgV6R7TvYEbs3lmwCP5O8RuXlEk2M8FfhSjbo7AHcDw4CtgD+TLjYYkpu3zv24G9ihCdNzCLAY2LLdpmOdGNtiOgKjgUeBdfPvy4Gj8veUXHYu8Onc/Bng3Nw8BbisUdz9NN3qxXghcGiN+gM+n4GdgHuB9UgXBv0GmNBOy2KDGNtiWax8vMdRh6SNgPcC5wNExCsR8QzpUSczcrUZwCG5eTJwUSS3AMMljQL2A2ZFxJKIeBqYBezf5BjrmQxcGhEvR8SjQCfpcS4D9UiXvYE/R8TjtNF0bBBjPa2YjkOBdSUNJa1UFgHvB67I7aunYWXaXgHsLUkN4u4v1TE2uteqFfN5e+CWiHghIpYBNwIfor2WxXox1tOS/7QTR31bA93ADyX9UdIPJK0PbB4RiwDy92a5/mgRubJQAAAIgklEQVRgfqH7rlxWr7yZMQIcl3evL6jsercoxqIpwCW5uZ2mY70YoQ2mY0QsAL4JzCMljKXAHcAzeeVSPazX48jtlwKbNiu+ejFGxLW59dfzNDxL0rDqGKtiaeZ8vhd4r6RNJa1H2qMYS3sti/VihDZYFiucOOobCuwKnBMRbwf+StqNrafeY1F6fFzKaqgX4znAW4BdSH/ib7UwxjTgdPz9YOAnPVWtE0srYmyL6ZhXEpNJhyK2ANYnPSm63rAGfBrWilHSEcDJwHbAbqRDOye2KsaIeAA4k7SH8GvS4ZtlDTpppxjbYlmscOKorwvoiohb8+8rSCvpJ/LuKvn7yUL9Wo9FaebjUmrGGBFPRMTyiHgNOI8VhyNaEWPFAcCdEfFE/t1O07FmjG00HfcBHo2I7oh4FfgZ8E7SoZPKTbzFYb0eR26/MbCkifHVjTEiFuVDPS8DP6TFy2JEnB8Ru0bEe0nT5GHabFmsFWMbLYuAE0ddEbEYmC9p21y0N3A/6VEnlasopgJX5uaZwJH5Sow9Sbvqi0h3u+8raUTeKts3lzUtxsqfIPsQafe3EuOUfNXNVqSTbrcxMI90OZyVDwG1zXSsF2MbTcd5wJ6S1svnKirL4g3AoblO9TSsTNtDgesjnUmtF3d/qBXjA4UVskjnDorTcMDns6TN8vc44O9J87utlsVaMbbRspj011n2N+KHtFs4B5gL/IJ0BcWmwHWkLZXrgE1yXZFeNvVn4B5gYqE/HyedtOoEjh6AGP83xzA3LyyjCvW/mmN8CDigUH4g8Kfc7qv9HON6wF+AjQtl7TYda8XYNtMROA14kLTC+F/SVTRbk1YSnaTDa8Ny3XXy787cfuue4m5ijNfnaXgv8CNggxbP59+Sku7dwN5tuizWirFtlsWI8CNHzMysHB+qMjOzUpw4zMysFCcOMzMrxYnDzMxKceIwM7NSnDiszyR9NT/Bc25+YuceufyE/LiE/hrOY5JGrkb3kyRdVad8aX5cy0OSbpJ00GoM5xhJR/ZQ5xBJOxR+ny5pn74Os9CfhuNSjE3Sdnl+/VHSWyQdr/Rk5R+vbhw2OKwRr4619iNpL+Ag0p3qL+cV+9q59Qmka/ZfaFFsQyJieS+r/zYiDsrd7QL8QtKLEXFd2eFGxLm9qHYIcBXpOn0i4mtlh9NA3XGpiu0Q4MqIOCXX/Qzp+v9HezMQSUNjxTOybBDyHof11SjgqUiPkiAinoqIhZKOJz2r6AZJNwBIOkfSnLx3clqlB3lP4jRJd0q6R9J2uXxTSdfmLeLvU3jujqRfSLoj92taofz5vPV+K7CX0rsIHpR0M+nu2x5FxF3A6cBxuZ8dkn4q6fb8eZekN+W4hxeG3Slpc6V3Jnwpl30qd3N37sd6kt5JehbWf+Yt/rcova/i0NzN3nmc71F6kN2wRtOp5LicKulLSu9xOAH4pKQbJJ1LupFwptL7NNbPw749xzI5d3+UpJ9I+j/g2lz25VxvbmW+Shqf917Oy/PoWknr5nbbSPpNniZ3SnpLg/6sL+mXue69kj7Sm3loA6Q/7yb0Z/B8gA2Au0h3pv4P8L5Cu8eAkYXflTtxhwCzgZ0L9T6bmz8D/CA3nw18LTd/kPRwtpFV/VqXdDfypvl3AB/OzeuQngw6gZR0LgeuqjEOk6rLSXfiP5CbLwbenZvHFcr/m3y3MLAH8JvcfCr5nQmVuHLzvxXG80IK76eo/C7E/NZcfhFwQqPpVHJcirG93lw9v4B/B47IzcPz/F2f9G6NrsL03xeYnqfvm0h7Ue8FxpMeyrdLrnd5oX+3Ah8qzKP1GvTnH4DzCjFuXD3O/rTu4z0O65OIeB54BzCN9Gj3yyQdVaf6hyXdCfwR2JH08pmKn+XvO0grHUgrjh/l4fwSeLpQ/3hJdwO3kB7iNiGXLwd+mpu3Iz1w7+FIa50flRi14lNF9wG+K+ku0mMeNpK0IXAZUNkCnpJ/V9tJ0m8l3QP8I2m8G9k2x/yn/HsGaTpU1JpOZcalt/YFTsrjPJu0gh+X282KiCWFevuS5umdpGlemRePRtrjeT3ePN1GR8TPASLipYh4oUF/7gH2kXSmpPdExNI+jIs1ic9xWJ9FOo8wG5idV5BTSVvQr1N68NqXgN0i4mlJF5JWRhUv5+/lrLw8rvIsHEmTSCvzvSLiBUmzC/16KVY+r9HXZ+m8HXggN78pD+vFqjj+AGwjqYN0vuDfavTnQuCQiLg7J9RJPQy3p5V8venUSHFcekvAP0TEQysVpgsf/lpV7xsR8f2qeuMLsVbiXZf641ezP7lf7yA9b+kbkq6NiNPLjYo1i/c4rE+U3tM9oVC0C1B5a95zwIa5eSPSCmeppM2p/R6JajeRttKRdADpwY2QHg/+dE4a25Fe51nLg8BWlWPopKfe9kjSzsC/kB5sB+lY/nGF9rsA5L2YnwP/RToU9JcavdsQWCRprcq4ZMVpUx3zeEnb5N8fI739rU9qjEtvXQN8VpJyf97eoN7HJW2Q641WfqprLRHxLNAl6ZBcf5jSlXc1+yNpC+CFiPgR6QVRu9brtw0873FYX20AfCefJF5Gekpo5WT1dOBXkhZFxN9K+iNwH+ndzL/rRb9PIz1K+k7SynNeLv81cIykuaQngd5Sq+OIeEnpxPkvJT0F3Ex6l3Mt78nxrUd6D8PxseKKquOB7+XhDSUltGNyu8tIj64+qk5//4V0TP9x0mGXSrK4FDhP6SKCyiPRKzEfDfxE6R0at5PeI15Go3HprX8Fvg3MzcnjMdLVcyuJiGslbQ/8IeeY54EjSHsY9XwM+L6k04FXgcMa9Gcb0kUEr+W6ny45HtZEfjqumZmV4kNVZmZWihOHmZmV4sRhZmalOHGYmVkpThxmZlaKE4eZmZXixGFmZqX8f9ZqSR7DU2iyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot histogram of values with percentiles marked\n",
    "plt.title('Standard Deviation Differences: Insured vs. Uninsured')\n",
    "plt.hist(b_reps,bins = 20, ec ='black')\n",
    "plt.xlabel('Standard Deviation Differences')\n",
    "plt.ylabel('PDF')\n",
    "plt.axvline(np.mean(b_reps), color = 'black')\n",
    "plt.axvline(lval, color ='red')\n",
    "plt.axvline(uval,color = 'red')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ###### We reject the null hypothesis because there is a difference in the standard deviations of medical charges of patients with insurance and patients without insurance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confidence interval and p-value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confidence interval above is often a useful quantity to estimate. If we wish to limit our expected probability of making a Type I error (where we wrongly reject the null hypothesis, and it is, instead, true) to $\\alpha$, the associated confidence interval is our estimate of the interval within which we expect the true population value to be found $100\\times(1 - \\alpha)$% of the time we do this test. In the above we performed bootstrap replicates to estimate the interval and reject the null hypothesis if this interval did not contain zero. You will sometimes see such an interval reported in the output of statistical functions.\n",
    "\n",
    "The partner of the confidence interval is the p-value. The p-value and the confidence interval are linked through our choice of $\\alpha$. The p-value tells us how likely it is, under the null hypothesis, to get an outcome at least as extreme as what was observed. If this fails to reach the level of our _pre-specified_ $\\alpha$, we decide the null hypothesis is sufficiently unlikely to be true and thus reject it. To calculate this p-value via the bootstrap, we have to put ourselves in a position where we are simulating the null hypothesis being true and then calculate the fraction of times we observe a result at least as extreme as that actually observed.\n",
    "\n",
    "Remember how, previously, you used the _t_-test to calculate the p-value for the observed difference between the means of insured and non-insured medical cases. We're now going to repeat this, this time using the bootstrap approach.\n",
    "\n",
    "__Q:__ Perform a bootstrapped hypothesis test at the 5% significance level ($\\alpha = 0.05$) to calculate the p-value of the observed difference between insurance and non-insurance charges, state your null and alternative hypotheses and whether you retain or reject the null hypothesis for the given significance level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__A:__ <b>Null Hypothesis</b>: The mean of medical charges for patients with insurance is the same as patients without insurance.<b>Alternative Hypothesis</b>: The mean of medical charges for patients with insurance is different than the mean of patients without insurance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n"
     ]
    }
   ],
   "source": [
    "# compute differences in the means\n",
    "mean_diff = np.mean(not_insured) - np.mean(insured)\n",
    "mean_diff\n",
    "# create dataframes for charges with insurance and charges without insurance\n",
    "mcharges_ins = med_charges[med_charges.insuranceclaim == 1]\n",
    "mcharges_nins = med_charges[med_charges.insuranceclaim == 0]\n",
    "#mean of charges\n",
    "mean_charge = np.mean(np.concatenate((mcharges_ins,mcharges_nins)))\n",
    "#mean diff between samples\n",
    "diff_means = np.mean(mcharges_ins.charges) - np.mean(mcharges_nins.charges)\n",
    "# Generate shifted arrays\n",
    "ins_shift = mcharges_ins.charges - np.mean(mcharges_ins.charges) + mean_charge\n",
    "nins_shift = mcharges_nins.charges - np.mean(mcharges_nins.charges) + mean_charge \n",
    "# Compute bs replicates from shifted arrays\n",
    "bs_rep1 = bs_reps(ins_shift, np.mean, size = 10000)\n",
    "bs_rep2 = bs_reps(nins_shift, np.mean, size = 10000)\n",
    "# Get replicates of mean diffs\n",
    "bs_replicates = bs_rep1 - bs_rep2\n",
    "# calculate p value\n",
    "pval = np.sum((bs_replicates) >= diff_means) / len(bs_replicates)\n",
    "# Print the result\n",
    "print(pval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### The p value is < 0.05, therefore the null hypothesis is rejected. There is a difference between medical charges of patients who have insurance and medical charges of patients who do not have insurance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Q:__ To put the above result in perspective, plot the histogram of your bootstrapped differences along with lines marking the locations of the observed difference. (Why would we plot more than one line, given that we only have one observed difference?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__A:__ We plot more than one line because this is a two sided t test, where we are looking at both of the sides of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-1104.4550169967074 1154.1453540352304\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBodHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAIABJREFUeJzt3XmcHVWd9/HP1yQQVpOQBpp0QkAiY9BRMEJUHCM7yBB04CE4QlSciKKIy0xAnkdwwSGjI8rLEUSJgAIRESUiChEJqGOAgBCWsDRrOgtpCPuahN/zxzkdKp17u7s6ffv28n2/XvXqqlOn6p469/b93XOq6pQiAjMzs656Q70LYGZm/YsDh5mZleLAYWZmpThwmJlZKQ4cZmZWigOHmZmV4sBhPULS85J2zvObSfqtpGck/TKnfVPSE5JW1Lek1kbSI5L2q3c5ukrS+yTdV+9ydEbSBZK+We9y1JIDRx+T/5lflTS6XfrtkkLS+F4uzxRJr+XA8LykFkmXSXpXMV9EbBkRD+XFI4DtgG0i4khJY4EvARMjYvveLH89DIYvju6oVC+SxufP9dDOto+IP0fErrUroXWVA0ff9DBwdNuCpLcBm9WvOCyLiC2BrYDJwL3AnyXtWyX/jsD9EbGmsPxkRKws+8JK/DntRFe+eK0812tl/ofsm34GHFtYng5cVMwgaVNJ35H0mKTHJZ0rabO8bqSkqyS1SnoqzzcVtp0v6RuS/irpOUnXtm/hVBJJS0R8FfgJMKuwz5C0i6SvAV8FjsotlE8B84Ad8vIFOf9kSf8r6WlJd0ia0q58Z0j6K/AisLOkN0o6X9JySUtz19eQnP9jkv6S6+MpSQ9LOriwv1GSfippWV7/m8K6Q3Nr7ulcnn8srJuZX+s5Sfd1ECirKvyinp7fqycknVpYv6ekhZKeze/jd3P6FEkt7fa1rmtJ0umSLpf0c0nPAh+T9AZJJ0t6UNKTuWU4qrD9MZIezetOpYr83qxoq9+c9iFJizoqc0/Ix/hlSYuUujp/IWl4pTrpJO/o/Ll/WtIqSX9u+wHS9lkt7GddS6jtNfJ7vwL4aU7v6HOyu6Tb8ufkF8DwnqqPPisiPPWhCXgE2A+4D3gLMARYQvrVHsD4nO97wFxgFKkl8FvgP/O6bYB/ATbP634J/KbwGvOBB4E3k1oy84Ezq5RnCtBSIX0f4DVgi7wcwC55/nTg59X2AYwBngQOIf142T8vNxTK9xiwGzAUGAb8BvgRsAWwLXAz8Kmc/2PAauDfcn19GlgGKK//HfALYGTe1/tz+h7ASmCvvN30XP+bArvmet8h5x0PvCnP7w083cF7eAHwzcJ2Afw41/XbgVeAt+T1fwOOyfNbApOr1Xsu236FOl4NHJ7rcDPgJGAB0JSP4UfApTn/ROB54J/yuu8Ca9r2V+EYHgT2Lyz/Eji5ozJ34bO9rl4KaW31M7RwjDcDO5A+24uB46t8jjrK+5/Aufn9Hga8r/B5WPdZrfB+Tcn1MivX02adfE42AR4FvpBf54j8vnyzK3XSXye3OPqutlbH/qSuoaVtKySJ9CX5hYhYFRHPAd8CpgFExJMR8auIeDGvOwN4f7v9/zQi7o+Il4DLgHeULN8yQMCI8ofGR4GrI+LqiHgtIuYBC0mBpM0FEXF3pO6uUcDBwEkR8UKkLq+zyMebPRoRP46ItcCFQCOwnaTGvO3xEfFURKyOiBvyNv8G/CgiboqItRFxIelLfTKwlvTFMFHSsIh4JCIeBIiIv0RE2eP+WkS8FBF3AHeQAgikL5ldJI2OiOcjYkGJff4tIn6T6/Al4FPAqZFaha+QgssRSt0tRwBXRcSNed3/IwX+ai4ld5dK2or03lzaA2XuirMjYllErCL9IOros1kt72rSZ2DH/J7/OXJk6ILXgNMi4pVcrx19TiaTAsb38utcDtxS9oD7GweOvutnwEdIv6YvareugdSauDU3nZ8G/pDTkbS5pB/lbolngRuBEcWuB6B4ddOLpF+OZYwh/XJ7uuR2kFpPR7aVPZd/b9I/epsl7fIPA5YX8v+I1PJos+54IuLFPLslMBZYFRFPVSnHl9qVYyypldFM+gV/OrBS0hxJO3TjWDcoH+vX93Gklt+9km6RdGiJfS5pt7wj8OvCsSwmBcDtSL/K1+WPiBdIrbxqLgE+LGlT4MPAbRHx6EaWeQ3pfSwaRvqiLgaxMp/Nanm/DTQD10p6SNLJXSwjQGtEvFxYrvo5ydPSdkHpUQY4B44+Kv+TPkz6pXdFu9VPAC8Bu0XEiDy9MdIJbEhXMO0K7BURW5O6JyC1EHrKh0hfJi90Y9slwM8KZR8REVtExJmFPNEu/yvA6EL+rSNity6+1ihJlVoIS4Az2pVj84i4FCAiLomIvXm9m3BWhX1slIh4ICKOJgXBWcDlkrYAXiD9OAAgB/2G9ptXOJ6D2x3P8IhYCiwnfdm17W9zUpdmtXLdQ/oCPJj0A+aSLpS5M4+RuqaKdgKWRERHrZ/SIuK5iPhSROwM/DPwxcI5qhcp1C3Q/kq/SvVa7XOyHBiTewHajOvBQ+mTHDj6tuOAfdp/Oed/sh8DZ0naFkDSGEkH5ixbkQLL0/nk6Gk9URglYySdBnwS+Eo3d/Vz4J8lHShpiKTh+aRkU6XMEbEcuBb4b0lb55PAb5LUvvut2ra/B36odNHAMEltgfTHwPGS9srHtoWkD0raStKukvbJv7hfJtXn2m4eb1WSPiqpIb+nba23tcD9wPBcnmHA/yV1nXXkXOAMSTvmfTdImprXXQ4cKmlvSZsAX6fz//9LgBNJPzx+2YUyd+ZXwAclHZDf9x3ycc3pwral5JPZu+Qv9Gdz+drKeDvwkVyGg9iwG7e9qp8T0vmeNcCJkoZK+jCwZ08fT1/jwNGHRcSDEbGwyuqZpKb4gtwd9UdSKwPSifPNSC2TBaRurI2xg6TnSSdXbwHeBkyJiGu7s7OIWAJMJQWeVtIvun+n48/jsaQTkfcAT5G+CBs7yF90DKnP+17SSc6TcjkWkvqvf5D32UzqGoT0JX0mqQ5XkH5dfwXW3Yj2fBdfuzMHAXfn/X0fmBYRL0fEM8BnSFevLSW1QFqq7wby9nNJ3TPPkd77vQAi4m7gBFIwWE463s72dynpZPGfIuKJzsoM624EfV+lneUyHE06cb2K9KV7E/C1TsrRHRNI/xPP59f5YUTMz+s+T2qFPA38K+nCi6o6+pxExKukrryP5XVHsWEPwYDTdpWBmZlZl7jFYWZmpThwmJlZKQ4cZmZWSs0Ch6TZklZKuqtd+ueUhm+4W9J/FdJPkdSc1x1YSD8opzWXvBbbzMxqoGYnx/Mlj88DF0XEW3PaB4BTgQ9GxCuSto2IlZImkq7g2JN0Q80fSTcYQboscX/SFSC3AEfna8yrGj16dIwfP74GR2Vddl8e/XpXD2bamftyXe3quirPn7Medeuttz4REe3vF9pAzUZ+jIgbteEQ4J8mjYn0Ss7TNlrqVGBOTn9YUjOvXwvdHHm4bklzct4OA8f48eNZuLDaVazWK6ZMSX/nz69nKfqFKbmu5ruuyvPnrEdJ6tJd7719juPNwPsk3STpBr3+TIcxrD98QktOq5ZuZmZ10ttjzQ8ljVA6GXgXcJnSU+MqDYURVA5sFfvWJM0AZgCMGzfg7/g3M6ub3m5xtABXRHIzaWCz0Tl9bCFfE2n01WrpG4iI8yJiUkRMamjotIvOzMy6qbcDx29Iz3FA0ptJQ0g8QRomYZrSw4l2Ig0XcDPpZPgESTvl8XWm5bxmZlYnNeuqktQ2zs1opad2nQbMBmbnS3RfBabn4YjvlnQZ6aT3GuCESM9VQNJngWtID1CZnce7MTOzOqnlVVVHV1n10Sr5zyA9cKh9+tXA1T1YNDMz2wi+c9zMzEpx4DAzs1IcOMzMrJTevo/DbNBrbBrHiqXtHxcO6z99tLLtx4xlectjtSiWWZc5cJj1shVLl7DjzKteX74kjd25/UfOrLbJOo/OOrRm5TLrKndVmZlZKQ4cZmZWigOHmZmV4sBhZmalOHCYmVkpDhxmZlaKA4eZmZXiwGHWDY1N45DUrcmsv/MNgGbd0P4mvjJ8E5/1d25xmJlZKQ4cZmZWigOHmZmV4sBhZmal1CxwSJotaWV+vnj7dV+WFJJG52VJOltSs6RFkvYo5J0u6YE8Ta9Vec3MrGtq2eK4ADiofaKkscD+QPGhAgcDE/I0Azgn5x0FnAbsBewJnCZpZA3LbGZmnahZ4IiIG4FVFVadBfwHEIW0qcBFkSwARkhqBA4E5kXEqoh4CphHhWBkZma9p1fPcUg6DFgaEXe0WzUGKD4SrSWnVUuvtO8ZkhZKWtja2tqDpTYzs6JeCxySNgdOBb5aaXWFtOggfcPEiPMiYlJETGpoaOh+Qc36siHDun3HemPTuHqX3gaI3rxz/E3ATsAdediFJuA2SXuSWhJjC3mbgGU5fUq79Pm9UFazvmntat+xbnXXay2OiLgzIraNiPERMZ4UFPaIiBXAXODYfHXVZOCZiFgOXAMcIGlkPil+QE4zM7M6qeXluJcCfwN2ldQi6bgOsl8NPAQ0Az8GPgMQEauAbwC35OnrOc3MzOqkZl1VEXF0J+vHF+YDOKFKvtnA7B4tnJmZdZvvHDczs1IcOMzMrBQHDjMzK8WBw8zMSnHgMDOzUhw4zMysFAcOMzMrxYHDzMxKceAwM7NSHDjMzKwUBw4zMyvFgcPMzEpx4DAzs1IcOMzMrBQHDjMzK8WBw8zMSnHgMDOzUhw4zMyslFo+c3y2pJWS7iqkfVvSvZIWSfq1pBGFdadIapZ0n6QDC+kH5bRmSSfXqrw2uDQ2jUNStyezwaxmzxwHLgB+AFxUSJsHnBIRayTNAk4BZkqaCEwDdgN2AP4o6c15m/8B9gdagFskzY2Ie2pYbhsEVixdwo4zr+r29o/OOrQHS2PWv9SsxRERNwKr2qVdGxFr8uICoCnPTwXmRMQrEfEw0AzsmafmiHgoIl4F5uS8ZmZWJ/U8x/EJ4Pd5fgywpLCuJadVS9+ApBmSFkpa2NraWoPimpkZ1ClwSDoVWANc3JZUIVt0kL5hYsR5ETEpIiY1NDT0TEHNzGwDtTzHUZGk6cChwL4R0RYEWoCxhWxNwLI8Xy3dzMzqoFdbHJIOAmYCh0XEi4VVc4FpkjaVtBMwAbgZuAWYIGknSZuQTqDP7c0ym5nZ+mrW4pB0KTAFGC2pBTiNdBXVpsC8fEnjgog4PiLulnQZcA+pC+uEiFib9/NZ4BpgCDA7Iu6uVZnNzKxzNQscEXF0heTzO8h/BnBGhfSrgat7sGhmZrYRfOe4mZmV4sBhZmalOHCYmVkpDhxmZlaKA4eZmZXiwGFmZqU4cJiZWSkOHGZmVooDh5mZleLAYWZmpThwmJlZKQ4cZmZWigOHmZmV4sBhZmalOHCYmVkpDhxmZlaKA4eZmZXiwGFmZqXULHBImi1ppaS7CmmjJM2T9ED+OzKnS9LZkpolLZK0R2Gb6Tn/A5Km16q8ZmbWNbVscVwAHNQu7WTguoiYAFyXlwEOBibkaQZwDqRAA5wG7AXsCZzWFmzMzKw+ahY4IuJGYFW75KnAhXn+QuDwQvpFkSwARkhqBA4E5kXEqoh4CpjHhsHIzMx6UW+f49guIpYD5L/b5vQxwJJCvpacVi19A5JmSFooaWFra2uPF9ys3xsyDEndnhqbxtX7CKyPGFrvAmSqkBYdpG+YGHEecB7ApEmTKuYxG9TWrmbHmVd1e/NHZx3ag4Wx/qy3WxyP5y4o8t+VOb0FGFvI1wQs6yDdzMzqpLcDx1yg7cqo6cCVhfRj89VVk4FnclfWNcABkkbmk+IH5DQzM6uTmnVVSboUmAKMltRCujrqTOAySccBjwFH5uxXA4cAzcCLwMcBImKVpG8At+R8X4+I9ifczcysF9UscETE0VVW7VshbwAnVNnPbGB2DxbNzMw2gu8cNzOzUhw4zMysFAcOMzMrxYHDzMxKceAwM7NSHDjMzKwUBw4zMyvFgcPMzEpx4DAzs1IcOMzMrBQHDjMzK8WBw8zMSnHgsH6rsWlct59mZ2bd11eeAGhW2oqlS7r9RDs/zc6s+zpscUi6tjB/Su2LY2ZmfV1nXVUNhfkjq+YyM7NBo7PAEb1SCjMz6zc6O8exs6S5gArz60TEYTUrmZmZ9UmdBY6phfnv9NSLSvoC8ElSi+ZO0jPGG4E5wCjgNuCYiHhV0qbARcA7gSeBoyLikZ4qi5mZldNh4IiIG9rmJTXktNaNeUFJY4ATgYkR8ZKky4BpwCHAWRExR9K5wHHAOfnvUxGxi6RpwCzgqI0pg5mZdV9nV1VJ0mmSngDuBe6X1Crpqxv5ukOBzSQNBTYHlgP7AJfn9RcCh+f5qXmZvH5f+UJ8M7O66ezk+EnA3sC7ImKbiBgJ7AW8N3c3lRYRS0ndXo+RAsYzwK3A0xGxJmdrAcbk+THAkrztmpx/m/b7lTRD0kJJC1tbN6pRZGZmHegscBwLHB0RD7clRMRDwEfzutIkjSS1InYCdgC2AA6ukLXtiq5KrYsNrvaKiPMiYlJETGpoaKiwiZmZ9YTOAsewiHiifWI+zzGsm6+5H/BwRLRGxGrgCuA9wIjcdQXQBCzL8y3AWIC8/o3Aqm6+tpmZbaTOAser3VzXkceAyZI2z+cq9gXuAa4Hjsh5pgNX5vm5eZm8/k8R4ftLzMzqpLPLcd8u6Vle7y4qdh8N784LRsRNki4nXXK7Bvg7cB7wO2COpG/mtPPzJucDP5PUTGppTOvO65qZWc/o7HLcIbV40Yg4DTitXfJDwJ4V8r6MhzsxM+szOgwckoYDxwO7AIuA2YUrn8zMbBDq7BzHhcAk0t3dhwD/XfMSmZlZn9bZOY6JEfE2AEnnAzfXvkhmZtaXddbiWN024y4qMzODrl9VBelKqs0KV1lFRGxd09KZmVmfU5erqszMrP/qrKvKzCwZMgxJ3Zoam8bVu/TWgzrrqjIzS9auZseZV3Vr00dnHdrDhbF6covDzMxKceAwM7NSHDjMzKwUBw4zMyvFgcPMzEpx4DAzs1IcOMzMrBQHDjMzK8WBw8zMSnHgMDOzUuoSOCSNkHS5pHslLZb0bkmjJM2T9ED+OzLnlaSzJTVLWiRpj3qU2czMknq1OL4P/CEi/gF4O7AYOBm4LiImANflZYCDgQl5mgGc0/vFNTOzNr0eOCRtDfwTcD5ARLwaEU8DU0mPqiX/PTzPTwUuimQBMEJSYy8X28zMsnq0OHYGWoGfSvq7pJ9I2gLYLiKWA+S/2+b8Y4Alhe1bctp6JM2QtFDSwtbW1toegZnZIFaPwDEU2AM4JyJ2B17g9W6pSlQhLTZIiDgvIiZFxKSGhoaeKamZmW2gHoGjBWiJiJvy8uWkQPJ4WxdU/ruykH9sYfsmYFkvldXMzNrp9cARESuAJZJ2zUn7AvcAc4HpOW06cGWenwscm6+umgw809alZWZmva9eTwD8HHCxpE2Ah4CPk4LYZZKOAx4Djsx5rwYOAZqBF3NeMzOrk7oEjoi4HZhUYdW+FfIGcELNC2VmZl3iO8fNzKwUBw4zMyvFgcPMzEpx4DAzs1IcOMzMrBQHDjMzK8WBw+qqsWkckro1mVl91OsGQDMAVixdwo4zr+rWto/OOrSHS2NmXeEWh5mZleLAYWZmpThwmJlZKQ4cZmZWigOHmZmV4sBhZmalOHCYmVkpDhxmZlaKA4eZmZXiwGFmtTdkWLeHlmlsGlfv0ls7dRtyRNIQYCGwNCIOlbQTMAcYBdwGHBMRr0raFLgIeCfwJHBURDxSp2KbWXesXe2hZQaQerY4Pg8sLizPAs6KiAnAU8BxOf044KmI2AU4K+czM7M6qUvgkNQEfBD4SV4WsA9wec5yIXB4np+al8nr95WHRjUzq5t6tTi+B/wH8Fpe3gZ4OiLW5OUWYEyeHwMsAcjrn8n51yNphqSFkha2trbWsuxmZoNarwcOSYcCKyPi1mJyhazRhXWvJ0ScFxGTImJSQ0NDD5TUzMwqqcfJ8fcCh0k6BBgObE1qgYyQNDS3KpqAZTl/CzAWaJE0FHgjsKr3i21mZlCHFkdEnBIRTRExHpgG/Cki/hW4HjgiZ5sOXJnn5+Zl8vo/RcQGLQ4zM+sdfek+jpnAFyU1k85hnJ/Tzwe2yelfBE6uU/nMzIw6Pzo2IuYD8/P8Q8CeFfK8DBzZqwUzM7Oq+lKLw8zM+gEHDjMzK8WBw8zMSnHgMDOzUhw4zMysFAcOMzMrxYHDzMxKceAwM7NSHDjMzKwUBw4zMyulrkOO2MDQ2DSOFUuXrJd2ff77AT9zy2zAceCwjbZi6ZINnic9/JI0FuWOHzmzw239PGmz/sddVWZmVooDh5mZleLAYWZmpThwmJlZKT45bmZ925BhqMrVeZ1dvbf9mLEsb3msRgUbvBw4zKxvW7t6g6v22nR29Z6v2quNXu+qkjRW0vWSFku6W9Lnc/ooSfMkPZD/jszpknS2pGZJiyTt0dtlNjOz19XjHMca4EsR8RZgMnCCpInAycB1ETEBuC4vAxwMTMjTDOCc3i+ymZm16fXAERHLI+K2PP8csBgYA0wFLszZLgQOz/NTgYsiWQCMkNTYy8U2M7OsrldVSRoP7A7cBGwXEcshBRdg25xtDFAcz6Ilp7Xf1wxJCyUtbG1trWWxzcwGtboFDklbAr8CToqIZzvKWiEtNkiIOC8iJkXEpIaGhp4qppmZtVOXwCFpGCloXBwRV+Tkx9u6oPLflTm9BRhb2LwJWNZbZTUzs/XV46oqAecDiyPiu4VVc4HpeX46cGUh/dh8ddVk4Jm2Li0zM+t99biP473AMcCdkm7PaV8BzgQuk3Qc8BhwZF53NXAI0Ay8CHy8d4trZmZFvR44IuIvVD5vAbBvhfwBnFDTQpmZWZd5rCozMyvFgcPMzEpx4DAzs1IcOMzMrBQHDjMzK8WBw8zMSnHgMAAam8YhqVuTmQ0ufpCTAbBi6ZKqD8vpjB+WYza4uMVhZmalOHCY2cCVn1fenamxaVy9S99nuavKzAauDp5X3hl3wVbnFoeZmZXiwGFmZqU4cJiZWSkOHGZmlWzEifWBfnLdJ8cHiMamcaxYuqTexTAbODbixDoM7JPrDhwDxMbcwAcD+0NuZj3LXVVmZlZKvwkckg6SdJ+kZkkn17s8ZmaDVb8IHJKGAP8DHAxMBI6WNLG+pep5HmjQbAAZwHet95dzHHsCzRHxEICkOcBU4J5avNjGnGgesslw1r76crdf23e5mg0QG3PX+nc+1O0fhNuPGcvylse6tW1XKSJq+gI9QdIRwEER8cm8fAywV0R8tpBnBjAjL+4K3NfrBe3YaOCJeheiD3F9rM/1sT7Xx/p6qz52jIiGzjL1lxZHpdC7XsSLiPOA83qnOOVJWhgRk+pdjr7C9bE+18f6XB/r62v10S/OcQAtwNjCchOwrE5lMTMb1PpL4LgFmCBpJ0mbANOAuXUuk5nZoNQvuqoiYo2kzwLXAEOA2RFxd52LVVaf7UarE9fH+lwf63N9rK9P1Ue/ODluZmZ9R3/pqjIzsz7CgcPMzEpx4OgBkr4t6V5JiyT9WtKIwrpT8jAp90k6sJBecQiVfAHATZIekPSLfDFAvyLpSEl3S3pN0qR26wZdfXRkMA2lI2m2pJWS7iqkjZI0L7+/8ySNzOmSdHaul0WS9ihsMz3nf0DS9HocS0+QNFbS9ZIW5/+Xz+f0vl8nEeFpIyfgAGBonp8FzMrzE4E7gE2BnYAHSSf3h+T5nYFNcp6JeZvLgGl5/lzg0/U+vm7Ux1tIN2HOByYV0gdlfXRQT1WPeyBOwD8BewB3FdL+Czg5z59c+N85BPg96R6uycBNOX0U8FD+OzLPj6z3sXWzPhqBPfL8VsD9+X+kz9eJWxw9ICKujYg1eXEB6T4TSMOizImIVyLiYaCZNHzKuiFUIuJVYA4wVWmMgX2Ay/P2FwKH99Zx9JSIWBwRle7cH5T10YGKx13nMtVMRNwIrGqXPJX0vsL67+9U4KJIFgAjJDUCBwLzImJVRDwFzAMOqn3pe15ELI+I2/L8c8BiYAz9oE4cOHreJ0i/CiB9CIqDXrXktGrp2wBPF4JQW/pA4fpYX7XjHky2i4jlkL5IgW1zetnPSr8maTywO3AT/aBO+sV9HH2BpD8C21dYdWpEXJnznAqsAS5u26xC/qBywI4O8vc5XamPSptVSBsQ9dFNA/34Nka1uhlwdSZpS+BXwEkR8WwHgxv2mTpx4OiiiNivo/X5hNShwL6ROx7peKiUSulPkJqfQ/Ov7D47tEpn9VHFgK2PbvJQOvC4pMaIWJ67XVbm9Gp10wJMaZc+vxfKWROShpGCxsURcUVO7vN14q6qHiDpIGAmcFhEvFhYNReYJmlTSTsBE4CbqTKESg441wNH5O2nA9V+vfdHro/1eSiddLxtVwEV39+5wLH5SqLJwDO52+Ya4ABJI/PVRgfktH4nn8M7H1gcEd8trOr7dVLvKwsGwkQ6ybsEuD1P5xbWnUq6cuY+4OBC+iGkqygeJHXvtKXvTPoybQZ+CWxa7+PrRn18iPQr6BXgceCawVwfndRVxeMeiBNwKbAcWJ0/H8eRzmNdBzyQ/47KeUV6eNuDwJ2sf3XeJ/LnoRn4eL2PayPqY29Sl9KiwnfHIf2hTjzkiJmZleKuKjMzK8WBw8zMSnHgMDOzUhw4zMysFAcOMzMrxYHDBgRJIelnheWhklolXVXj171A0sOS7pB0v6SLJI0prL9aebRkSSfmkVAvzvey/FHS7ZKOqmUZzXqa7xy3geIF4K2SNouIl4D9gaW99Nr/HhGX5xu6TgKul/TWiHg1Ig4p5PsM6d6Vh/MNXMMi4h1dfZHCHfRmdeUWhw0kvwc+mOePJt1wBoCkLfLzIG6R9HdJU3P6eEl/lnRbnt6T06dImi/pcqVnrVysDgYRAojkLGAFcHDezyOSRks6l3Qz41xJM4GfA+/ILY43SXqnpBsk3SrpmjzUBLkM35J0A/B5SQ2SfpWP4xZJ7835Ts/HN1/SQ5JOLBz7sfn5DXe0tco62M/7c5n1Tk+uAAACaklEQVRuz/W01ca+KTYA1fvuSU+eemICngf+kTQE+3DSXbhTgKvy+m8BH83zI0h3a28BbA4Mz+kTgIV5fgrwDGncnzcAfwP2rvC6FwBHtEv7HjAzzz8CjK4wXyzbMOB/gYa8fBQwO8/PB35Y2PclbeUAxpGGqwA4Pe9jU2A08GTe726ku/TbXndUJ/v5LfDePL8l+TkznjwVJ3dV2YAREYvy8NRHA1e3W30AcJikL+fl4aQvzGXADyS9A1gLvLmwzc0R0QIg6XZgPPCXLhSlw5ZJBbsCbwXm5UbNENLQHG1+UZjfD5hYaPxsXWgV/C4iXgFekbQS2I78PJOIeAIgIlZ1sp+/At+VdDFwRdvxmxU5cNhAMxf4DukX/TaFdAH/Eu0eMCXpdNJ4Wm8ntSxeLqx+pTC/lq7/v+xOGmOoqwTcHRHvrrL+hcL8G4B3RzqP8/oOUgCoVF5ReYjtivsBzpT0O9KYSQsk7RcR93b5SGxQ8DkOG2hmA1+PiDvbpV8DfK7tPIWk3XP6G4HlEfEacAzp13635FFLTyQ9EvQPJTa9D2iQ9O68n2GSdquS91rgs4XX7Ozk+nXA/5G0Tc4/qqP9SHpTRNwZEbOAhcA/lDgOGyQcOGxAiYiWiPh+hVXfIPX5L5J0V14G+CEwXdICUjfVCxW27cy3Jd1BOm/yLuADkR4F29Uyv0oaOn5W3s/twHuqZD8RmJRPdt8DHN/Jvu8GzgBuyPtuG7672n5OknRXzvsSrz/N0mwdj45rZmaluMVhZmalOHCYmVkpDhxmZlaKA4eZmZXiwGFmZqU4cJiZWSkOHGZmVsr/B4ArcPtfkwD0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lval = np.percentile(bs_replicates,2.5)\n",
    "uval = np.percentile(bs_replicates,97.5)\n",
    "print(lval, uval)\n",
    "\n",
    "# Plotting histogram of bootstrapped differences\n",
    "plt.title('Mean Differences: Insured vs. Uninsured')\n",
    "plt.hist(bs_replicates, bins = 20,ec = 'black')\n",
    "plt.xlabel(\"Mean Differences\")\n",
    "plt.ylabel(\"PDF\")\n",
    "plt.axvline(np.mean(bs_replicates), color = 'black')\n",
    "plt.axvline(lval,color = 'red')\n",
    "plt.axvline(uval,color = 'red')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Q:__ Compare your p-value above with that obtained using the _t_-test function in the previous assignment. Do you think you would want to try to perform enough bootstrap replicates to observe a random difference as large as that we did observe?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__A:__ Because the p value was very small, we can say fairly consistently that the null hypothesis is rejected. I don't believe that performing additional bootstrap replicates to find random differences would change this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Q:__ Consider the two variants of the _t_-test we performed in the previous assignment. Which one would you use now?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__A:__ We would use the independent sample t test because we are comparing means for two groups (uninsured and insured) and account for unequal variance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Q:__ If, instead of being asked whether the means of two groups were different, you were working with an ad-recommender team who wanted to release a new recommendation algorithm, and you were looking at click-through rate both for the current algorithm (call it A) and from trials of their new algorithm (call it B), would you perform a two-sided test as above? What would be your null and alternative hypotheses and what would be the real-world consequence of rejecting the null hypothesis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__A:__ In this scenario, an A/B test would be more appropriate. The null hypothesis that there are no changes in click throughs between algorithm A and algorithm B. The alternative hypothesis would be that there are changes in click throughs between algorithm A and algorithm B. If the null hypothesis is rejected when it should not be, it could lead to unnecessary changes that could potentially cost the company more money in the long run, because it would possibly not bring in more traffic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning outcomes\n",
    "\n",
    "You've previously applied frequentist methods to calculate confidence intervals, p-values, and perform hypothesis tests. Frequentist methods use theoretical results to calculate what we expect would happen if experiments were to be run again and again and again. Now you've seen how you can do the same things using the bootstrap approach, which does not rely on such theory, and attendant assumptions, but instead literally does run experiments again and again and again.\n",
    "\n",
    "In these exercises, you have:\n",
    "* calculated the same confidence interval lower limit as you did previously\n",
    "* tested the assumption that the variances of the two groups (insured vs. non-insured) were equal - something a bit harder to do using the frequentist method because of the nature of the sampling distribution for variance\n",
    "* calculated the p-value for the difference between the means of the two groups and compared with the result obtained using the previous frequentist approach\n",
    "\n",
    "You are now well equipped to apply the bootstrap approach to a wide variety of problems. Just think about what conditions you wish to recreate in your simulated reruns."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}