Homework/STA130_HW_6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aa1e99a5",
   "metadata": {
    "deletable": false
   },
   "source": [
    "# Weekly Homework 6: [Your Name]\n",
    "\n",
    "Go through this notebook, following the instructions! \n",
    "\n",
    "TAs will mark this assignment by first checking ***MarkUs*** autotests for completion and general correctness, and then manually reviewing your written response to `Q15` and `Q20` and quickly double checking for the presense of plotted figures for `Q2`, `Q17` and `Q18`.\n",
    "- The  `Q0, Q2, Q4, Q15` and `Q16` questions \"automatically fail\" during automated testing so that ***MarkUs*** exposes example answers for student review and consideration for these problems: these failed ***MarkUs*** \"autotests\" are not counted against the student.\n",
    "\n",
    "> You can add new cells if you need (with the \"+\" button above); but, deleting or reordering cells could very likely cause your notebook to fail ***MarkUs*** autotesting (and you'd have to start over and re-enter your answers into a completely fresh version of the notebook to get things working again...).\n",
    ">\n",
    "> - ***MarkUs*** autotesting works by running your notebook from top to bottom, and if there's an error when doing this (such as a variable is getting called that hasn't yet been defined, etc.) it can cause automated tests to fail; so, make sure your notebook runs from top to bottom without error once you're done (which you can do by restarting the \"Restart & Run All\" from the \"Kernel\" menu).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fbb0423",
   "metadata": {
    "deletable": false
   },
   "source": [
    "## Social Media and Anxiety\n",
    "\n",
    "There have been many questions regarding whether or not social media usage increases anxiety levels. For example, do  TikTok and Facebook posts create an unattainable sense of life success and satisfaction?  Does procrastinating by watching YouTube videos or reading Twitter posts contribute unnecessary stress from deadline pressure? A study was conducted to examine the relationship between social media usage and student anxiety. Students were asked to categorize their social media usage as \"High\" if it exceeded more than 2 hours per day, and then student anxiety levels where scored through as series of questions, with higher scores suggesting higher student anxiety."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68cc11e2",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import plotly.express as px\n",
    "from scipy import stats\n",
    "\n",
    "# The `numpy.repeat` function replicates elements as demonstrated here\n",
    "social_media_usage = np.repeat([\"Low\", \"High\"], [30, 16])\n",
    "anxiety_scores = [24.64, 39.29, 16.32, 32.83, 28.02, \n",
    "                   33.31, 20.60, 21.13, 26.69, 28.90,\n",
    "                   26.43, 24.23, 7.10,  32.86, 21.06,\n",
    "                   28.89, 28.71, 31.73, 30.02, 21.96,\n",
    "                   25.49, 38.81, 27.85, 30.29, 30.72,\n",
    "                   21.43, 22.24, 11.12, 30.86, 19.92,\n",
    "                   33.57, 34.09, 27.63, 31.26,\n",
    "                   35.91, 26.68, 29.49, 35.32,\n",
    "                   26.24, 32.34, 31.34, 33.53,\n",
    "                   27.62, 42.91, 30.20, 32.54]\n",
    "anxiety_data = pd.DataFrame({'anxiety_scores': anxiety_scores, 'social_media_usage': social_media_usage})\n",
    "anxiety_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c7e5b9",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q0: In simple terms, what is the claim of the *null hypothesis* for the experiment above that we are naturally trying to provide evidence against? How could this be formally stated with respect to the parameters $Median_{High}$ and $Median_{Low}$ of the two populations in question? And what is $H_1$ in terms of $H_0$?\n",
    "\n",
    "#### Provide your written answer in the markdown cell below.\n",
    "\n",
    "> Hint: In two-sample contexts the *null hypotheses* tends to be the most natural form of a \"no effect\" statement that \"nothing interesting or notable is happening\"; and, it's generally a \"straw man\" that we're trying to provide enough evidence against to reject. For example, a formal *null hypotheses* that there is no difference in the means of two groups would be $H_0: \\mu_{High} = \\mu_{Low}$.\n",
    "\n",
    "- You can compare your answer with the example solution provided by MarkUs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5af87b2b",
   "metadata": {
    "deletable": false
   },
   "source": [
    "> Answer here...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eaf8950",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q1: Revisit your statements regarding the *null hypotheses* above with \"confounding\" in mind; namely, since social media usage is a self-selecting process, perhaps social media users are already more anxious people on average regardless of their social media usage.  If we provide evidence about the *null hypotheses* are we actually addressing the question of \"whether or not usage of social media increases anxiety levels\"? Or are we just using a hypothesis test to examine if there is strong evidence of difference between the two groups (regardless of its causes)?  \n",
    "\n",
    "A. Neither, we are checking if confounding impacts our determination of \"whether or not usage of social media increases anxiety levels\"  \n",
    "\n",
    "B. We actually addressing the question of \"whether or not usage of social media increases anxiety levels\"   \n",
    "\n",
    "C. We are just using a hypothesis test to examine how strong the observable evidence of a difference between the two groups is (regardless of its causes)  \n",
    "\n",
    "D. None of the above: we cannot determine any of the above from a two-sample hypothesis test  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ef4a6a6",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Q1: your answer will be tested!\n",
    "Q1 = None # Assign either 'A' or 'B' or 'C' or 'D' to `Q1` instead of `None`\n",
    "# E.g., Q1 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90b6a5b2",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q2: Construct boxplots of `anxiety_data.anxiety_scores` for the two levels of social media usage, and write 2-3 sentences describing and comparing the distributions of anxiety scores across the social media usage groups.  \n",
    "#### Provide your `plotly` boxplot figure and written answer in the code and markdown cells below, respectively.\n",
    "\n",
    "> Hint: this is very easy to do with the `x` and `y` parameters of the the `px.box` function...\n",
    "\n",
    "- You can compare your answer with the example solution provided by MarkUs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e8211ff",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Code your solution here\n",
    "import plotly.express as px\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56ae8370",
   "metadata": {
    "deletable": false
   },
   "source": [
    "> Answer here..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4d71493",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q3: What do these data visually suggest regarding the claim that the ***median*** anxiety level is different for the population of people with high frequency social media use compared to the population with low frequency use? \n",
    "\n",
    "A. The median anxiety level is likely higher for low frequency social media users compared to higher frequency social media users.  \n",
    "\n",
    "B. We cannot say, since the groups are shuffled and we do not know the proportion of high frequency or lower frequency social media users in each group.    \n",
    "\n",
    "C. The median anxiety level is likely the same for low frequency social media users compared to higher frequency social media users.  \n",
    "\n",
    "D. The median anxiety level is likely higher for high frequency social media users compared to lower frequency social media users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21826f8b",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Q3: your answer will be tested!\n",
    "Q3 = None # Assign either 'A' or 'B' or 'C' or 'D' to `Q3` instead of `None`\n",
    "# E.g., Q3 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88c551c2",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q4: Write a few sentences explaining what the code inside the `for` loop below does and why it's doing it.  \n",
    "#### Provide your written answer in the markdown cell below.\n",
    "- You can compare your answer with the example solution provided by MarkUs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08324cb1",
   "metadata": {
    "deletable": false
   },
   "source": [
    "> Answer here..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ce3fa41",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "observed_test_statistic = np.diff(anxiety_data.groupby('social_media_usage').median().values.flatten())\n",
    "\n",
    "np.random.seed(1) # make the simulation reproducible...\n",
    "repetitions = 5000 \n",
    "irrelevant_labels_null_hypothesis_simulated_values = []\n",
    "\n",
    "shuffled_anxiety_data = anxiety_data.copy() # you should essentially always use `.copy()` for data frames; otherwise,\n",
    "# changes to the new data frame will also appear in the original version of the data frame as well(!)\n",
    "\n",
    "for i in range(repetitions):\n",
    "    shuffled_anxiety_data['social_media_usage'] = anxiety_data['social_media_usage'].sample(frac=1).values\n",
    "    permutation_statistic = np.diff(shuffled_anxiety_data.groupby('social_media_usage').median().values.flatten())[0]\n",
    "    irrelevant_labels_null_hypothesis_simulated_values += [permutation_statistic]\n",
    "\n",
    "fig = px.histogram(pd.DataFrame({'Difference in Medians': irrelevant_labels_null_hypothesis_simulated_values}), \n",
    "                   x='Difference in Medians', color_discrete_sequence=['grey'], nbins=20)\n",
    "fig.update_traces(marker_line_width=1, marker_line_color=\"black\")\n",
    "fig.add_vline(x=observed_test_statistic[0], line_width=3, line_dash=\"dash\", line_color=\"red\")\n",
    "fig.add_vline(x=abs(observed_test_statistic[0]), line_width=3, line_dash=\"dash\", line_color=\"red\")\n",
    "fig.show()\n",
    "\n",
    "num_as_or_more_extreme = (abs(np.array(irrelevant_labels_null_hypothesis_simulated_values)) >= \n",
    "                          abs(observed_test_statistic)).sum()\n",
    "p_value = num_as_or_more_extreme / repetitions\n",
    "p_value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3002c359",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q5: Which statement below best states the evidence you have against the *null hypothesis* based on the simulated p-value above?\n",
    "\n",
    "A. `0.10 < p-value`: No evidence against the null hypothesis  \n",
    "\n",
    "B. `0.05 < p-value <= 0.10`: Weak evidence against the null hypothesis  \n",
    "\n",
    "C. `0.01 < p-value <= 0.05`: Moderate evidence against the null hypothesis  \n",
    "\n",
    "D. `0.001 < p-value <= 0.01`: Strong evidence against the null hypothesis  \n",
    "\n",
    "E. `p-value < 0.001`: Very strong evidence against the null hypothesis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "678adfa5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q5: your answer will be tested!\n",
    "Q5 = None # Assign either 'A' or 'B' or 'C' or 'D' or 'E' to `Q5` instead of `None`\n",
    "# E.g., Q5 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6abcc58f",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q6: Does this data support the claim that the ***median*** anxiety level is different for those who use social media in high frequency compared to those who use social media in lower frequency?  How about the claim that \"usage of social media increases anxiety levels\"?\n",
    "\n",
    "A. Yes to the first, and yes to the second... there is a difference in median anxiety levels for those who use social media in high frequency compared to those who use social media in lower frequency, and it is because social media increases anxiety levels\n",
    "\n",
    "B. Yes to the first; but, no to the second... it seems possible -- not saying it's true, just plausible -- that different predispositions to anxiety could also have different predispositions to social media usage.  \n",
    "\n",
    "C. No to the first, and no to the second... this data does not adequately support the claim that the median anxiety level is different for those who use social media in high frequency compared to those who use social media in lower frequency  \n",
    "\n",
    "D. No to the first; but, yes to the second... the claim that the median anxiety level is different for those who use social media in high frequency compared to those who use social media in lower frequency is not supported, but we know usage of social media increases anxiety levels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5363f3f8",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Q6: your answer will be tested!\n",
    "Q6 = None # Assign either 'A' or 'B' or 'C' or 'D' to `Q6` instead of `None`\n",
    "# E.g., Q6 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e39a7066",
   "metadata": {},
   "source": [
    "### Q7: Use `scipy.stats.median_test` to calculate a p-value which could be used to provide evidence of a difference between the two social media usage groups.\n",
    "\n",
    "#### Provide your answer rounded to 3 decimal places."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25dabc6d",
   "metadata": {},
   "source": [
    "> - Hint 1: The \"median test\" produces a nonparametric p-value under the null hypothesis assumption that the median of two populations (here median anxiety levels of \"High\" and \"Low\" social media usage populations) are the same.\n",
    "> \n",
    ">   ```python\n",
    "> from scipy import stats \n",
    "> stats.median_test(series_A, series_B)\n",
    "> # where `series_A` and `series_B` are your two samples\n",
    "> ```\n",
    ">    - Boolean selection `anxiety_data.anxiety_scores[anxiety_data.social_media_usage=='Low']` could be useful...\n",
    ">\n",
    ">\n",
    "> - Hint 2: A test is nonparametric when it does not make specific distributional assumptions (such as normality) regarding the populations the samples come from; however, making assumptions about specific parameters of a population alone, such assuming something about the medians of two populations, does not make a test parametric. The terminology used here is somewhat unfortunate as it might make the difference between \"parametric\" and \"nonparametric\" a little more confusing than it needs to be...\n",
    ">\n",
    ">\n",
    "> - Hint 3: Use the `numpy.round` function to round your calculation.\n",
    ">\n",
    ">\n",
    "> - Hint 4: The \"median test\" is nonparametric just like the \"permutation shuffling test\" p-value simulated above; however, they are not quite the same test because they are based on slightly different assumptions: the null hypothesis assumptions the two (\"permutation shuffling test\" and \"median test\") testing methodologies are slighly different. \n",
    ">   - Shuffling the labels assumes that the labels don't matter, which implies that there's no difference between the populations the two samples come from; which, is a very strong statement null hypothesis (and we generally might expect stronger assumptions to be helpful in producing slightly small p-values...).\n",
    ">   - The \"median test\" instead is just based on assuming that if the medians of the two populations are the the same, then the null hypothesis is that half of the observations in the first group will be larger than half of the observations in the second group, and vice-versa; which, can be used to contruct another hypothesis test based on the Binomial distribution in a manner that is very much analagously to how such a Binomial test was created for a one-sample context...\n",
    ">     - Since the assumption of the \"median test\" is a little simpler and weaker than assuming two populations are identical and their samples interchangable, we'd expect p-values from a \"median test\" to be a little larger than p-values from a test based on a null hypothesis with slightly stronger assumptions...\n",
    ">     - Nonparametric p-values tend be a little bit larger than their parametric counterparts exactly because nonparametric null hypotheses tend to make fewer assumptions regarding the populations they're testing than their parametric counterparts which are derived on the basis of specific distributional assumptions about the populations that the samples being analyzed are drawn from.\n",
    ">\n",
    ">\n",
    "> - Hint 5: Another difference between the \"median test\" and the \"permutation shuffling test\" is that the former is theoretically derived, whereas the latter is based on a simulation.\n",
    ">    - Both parametric and nonparametric tests give p-values that are theoretically derived, as opposed to simulation based tests like the \"permutation shuffling test\" which use simulation to just simulate how things actually behave in order to give approximate p-values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b49c9ae3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "# space for your work\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90dd38fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q7: your answer will be tested!\n",
    "Q7 = None # Assign p-value rounded to 3 decimal places to Q7 instead of 'None'\n",
    "# E.g., Q7 = 0.987"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c08a57f4",
   "metadata": {},
   "source": [
    "### Q8: Are \"permutation shuffling test\" and \"median test\" methodologies parametric or nonparametric and why?\n",
    "\n",
    "A. Both are parametric since they both imply the assumption that the population medians are equal which subsequently implies that each population is normally distributed  \n",
    "\n",
    "B. Both are parametric since the boxplots of the data and histogram of the sampling distribution appear to be normally distributed  \n",
    "\n",
    "C. Both are nonparametric since if the two populations have different distributions then they couldn't both be normally distributed  \n",
    "\n",
    "D. Both are nonparametric since neither makes specific distributional assumptions (such as normality) about either of the populations  \n",
    "\n",
    "E. The former is parametric since it assumes the samples come from the same population while the latter is nonparametric since it only assumes the population medians are the same but not necessarily otherwise the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b198aace",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q8: your answer will be tested!\n",
    "Q8 = None # Assign either 'A' or 'B' or 'C' or 'D' to `Q8` instead of `None`\n",
    "# E.g., Q8 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ede7d590",
   "metadata": {},
   "source": [
    "### Q9: Use `scipy.stats.mannwhitneyu` to calculate a p-value which could be used to provide evidence of a difference between the two social media usage groups.\n",
    "\n",
    "#### Provide your answer rounded to 3 decimal places.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3406d4e1",
   "metadata": {},
   "source": [
    "> - Hint 1: the \"Mann-Whitney U test\" is based on a null hypothesis that assumes there is \"no actual distributional difference between two populations\" (here \"High\" and \"Low\" social media usage populations). \n",
    "> ```python\n",
    "> stats.mannwhitneyu(series_A, series_B)\n",
    "> # where `series_A` and `series_B` are your two samples\n",
    "> # Do not use `scipy.stats.wilcoxon`... that's for \"paired\" samples \n",
    "> # with one-to-one pairing of observations...\n",
    "> ```\n",
    "> - Hint 2: The \"Mann-Whitney U test\" is based on a null hypothesis that is identical to the assumption of the \"permutation shuffling test\" used to produce the initially simulated p-value (and is then of course again a stronger assumption than the \"median test\"); however, there is a slight difference in the observed test statistic used by the \"Mann-Whitney U test\" compared to the observed difference of medians used in the \"permutation shuffling test\"...\n",
    ">   - Using the assumption of \"no actual distributional difference between two populations\" implies that if all observations are \"ranked\" smallest to largest, the sum of the ranks should end up the same on average. This, however, means the \"Mann-Whitney U test\" will produce its p-value based on all the data (ranks totalling equivalently) as opposed to just considering the difference between the medians in two samples.  Making a p-value based on all the relative ranks of the samples will of course be more informative than just the medians of samples; so, even though the base assumption of the \"Mann-Whitney U test\" null hypothesis is the same as the permutation test, the data is used in a more powerful manner in the \"Mann-Whitney U test\"...\n",
    ">     - It's worth noting and keeping in mind that the different uses of data employed by different hypothesis tests can end up being more or less powerful based on how efficiently and holistically they make use of the information available in the data...\n",
    ">\n",
    ">\n",
    "> - Hint 3: The \"Mann-Whitney U test\" is a two sample test that is (of coures) distinct from the \"Wilcoxon Rank Sum test\" (`scipy.stats.wilcoxon`). The former is for two samples where the observations don't come in pairs; whereas, the latter is for when there are two samples but the observations have a natural pairing with each other.\n",
    ">    - Our samples don't involve having measurements on twins or something like that, so the our samples don't naturally come in pairs...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee764049",
   "metadata": {},
   "outputs": [],
   "source": [
    "# space for your work\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a301d53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q9: your answer will be tested!\n",
    "Q9 = None # Assign p-value rounded to 3 decimal places to Q9 instead of 'None'\n",
    "# E.g., Q9 = 0.987"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e8cbd1a",
   "metadata": {},
   "source": [
    "### Q10: Use `scipy.stats.ttest_ind` to calculate a p-value which could be used to provide evidence of a difference between the two social media usage groups.\n",
    "\n",
    "#### Provide your answer rounded to 3 decimal places.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f48f6fee",
   "metadata": {},
   "source": [
    "> - Hint 1: The \"two-sample t-test\" has a null hypothesis that assumes the means of two populations (here mean anxiety levels of \"High\" and \"Low\" social media usage populations) are equal and that the samples are drawn from normally distributed populations.\n",
    "> ```python\n",
    "> stats.ttest_ind(series_A, series_B, equal_var=False)\n",
    "> # where `series_A` and `series_B` are your two samples\n",
    "> # Do not use `stats.ttest_rel`... that's for \"paired\" samples \n",
    "> # with one-to-one pairing of observations...\n",
    "> ```\n",
    "> - Hint 2: A mean is of course distinct from a median, generally speaking; however, the mean and median are the same for a population that is symmetric such as a normally distributed population; thus, the assumption of the null hypothesis that the distributions are normally disributed and have the same means here implies that they have the same medians...\n",
    ">\n",
    ">\n",
    "> - Hint 3: When doing a two sample t-test it is possible to make additional assumptions about the two normal populations under examination, such as if the variances (or equivalently the standard deviations) of the two populations are the same or not. By default `stats.ttest_ind` uses `equal_var=True`; but, given the observed boxplots of the two samples in question, it seems like `equal_var=False` is a better assumption for this test; so, that's what you should use here as in the example code above...\n",
    ">\n",
    ">\n",
    "> - Hint 4: There is another function `stats.ttest_rel` which is (of course) distinct from `stats.ttest_ind`. The difference between these is analagous to the difference between the \"Wilcoxon Rank Sum test\" (`scipy.stats.wilcoxon`) and the \"Mann-Whitney U test\" (`scipy.stats.mannwhitneyu`); namely, the former is for when the observations have a natural pairing with each other, while the latter is when the observations don't come in pairs.\n",
    ">    - Our samples don't involve having measurements on twins or something like that, so the our samples don't naturally come in pairs...\n",
    ">\n",
    ">\n",
    "> - Hint 5: The p-value from a t-test is theoretically derived on the basis of the normality assumptions it makes about the population(s) in question; but, the p-value from the \"median test\" and \"Mann-Whitney U test\" are also theoretically derived; so, the use of a theoretical derivation is not what distinguishes between these tests; however, there is definitely a distinction between theoretically derived tests and the \"permutation shuffling test\" since that p-value is based on simulation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fcec329",
   "metadata": {},
   "outputs": [],
   "source": [
    "# space for your work\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "140593d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q10: your answer will be tested!\n",
    "Q10 = None # Assign p-value rounded to 3 decimal places to Q10 instead of 'None'\n",
    "# E.g., Q10 = 0.987\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d699173e",
   "metadata": {},
   "source": [
    "### Q11: Are \"Mann-Whitney U test\" and \"two-sample t-test\" methodologies parametric or nonparametric and why?\n",
    "\n",
    "A. Both are parametric since both imply the assumption that the population medians are equal which subsequently implies that we are assuming the samples are drawn from normally distributed populations  \n",
    "\n",
    "B. Both are parametric since the boxplots of the data and histogram of the sampling distribution appear to be normally distributed and the assumption of \"no actual difference between groups\" can only occur if each population is normally distributed   \n",
    "\n",
    "C. Both are nonparametric since neither makes specific distributional assumptions (such as normality) regarding the populations the two samples come from since a sample can never be normally distributed  \n",
    "\n",
    "D. The former is nonparametric since the assumption that the means for each group being equal does not imply that the populations are normally distributed while the latter is parametric since if the two populations have different distributions then only one can be normally distributed   \n",
    "\n",
    "E. The former is nonparametric since it only assumes the samples come from the same population while the latter is parametric since it instead makes specific distributional assumptions (such as normality) regarding the populations the two samples come from\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dad64e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q11: your answer will be tested!\n",
    "Q11 = None # Assign either 'A' or 'B' or 'C' or 'D' to `Q11` instead of `None`\n",
    "# E.g., Q11 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898ac2ac",
   "metadata": {},
   "source": [
    "### Q12: Give the strength of evidence against the null hypothesis for the three tests considered above.\n",
    "\n",
    "1. \"Median test\" which specifies a null hypothesis that assumes the medians of the two populations are equal\n",
    "2. \"Mann-Whitney U test\" which specifies a null hypothesis that assumes the two populations are identical\n",
    "3. \"Two-sample t-test\" which specifies a null hypothesis that assumes the two populations are normally distributed with the same means<br>(but different standard deviations, for the specification above)\n",
    "\n",
    "A. `0.10 < p-value`: No evidence against the null hypothesis  \n",
    "\n",
    "B. `0.05 < p-value <= 0.10`: Weak evidence against the null hypothesis  \n",
    "\n",
    "C. `0.01 < p-value <= 0.05`: Moderate evidence against the null hypothesis  \n",
    "\n",
    "D. `0.001 <= p-value <= 0.01`: Strong evidence against the null hypothesis  \n",
    "\n",
    "E. `p-value < 0.001`: Very strong evidence against the null hypothesis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "161f5275",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q12: your answer will be tested!\n",
    "median_test_evidence = None # Assign either 'A' or 'B' or 'C', 'D', 'E' based on the options above\n",
    "mann_whitney_U_test_evidence = None # Assign either 'A' or 'B' or 'C', 'D', 'E' based on the options above\n",
    "two_sample_t_test = None # Assign either 'A' or 'B' or 'C', 'D', 'E' based on the options above\n",
    "Q12 = (median_test_evidence, mann_whitney_U_test_evidence, two_sample_t_test) \n",
    "# Assign a triple tuple comprised of 'A's, 'B's, 'C's, 'D's and/or 'E' to `Q12` \n",
    "# in the order of Median test, Mann-Whitney U test, and Two-sample t-test\n",
    "# instead of the `None` triple tuple `(None,None,None)` \n",
    "# E.g., Q12 = ('A','B','C')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86716ca6",
   "metadata": {},
   "source": [
    "### Q13: Relative to the p-value calculations based on `scipy.stats.ttest_1samp` and `scipy.stats.binom` from the homework exercise and tutorial assignment of the previous week, which of the following is correct?\n",
    "\n",
    "A. The `stats.ttest_1samp` and `stats.binom` methods **theoretically** derive p-value under their null hypothesis, while `stats.median_test`, `stats.mannwhitneyu`, and `stats.ttest_ind` do not  \n",
    "\n",
    "B. The `stats.ttest_1samp` and `stats.ttest_ind` methods use continuous approximations of the theoretical binomial sampling distribution of the `stats.binom` and `stats.median_test` and `stats.mannwhitneyu` methods, respectively, which is why these t-tests produce nonparametric p-values  \n",
    "\n",
    "C. The `stats.ttest_1samp`, `stats.ttest_ind`, and `stats.ttest_rel` methods are parametric due to their assumption regarding the normally distributed nature of the populations they consider; whereas, the `stats.binom`, `stats.median_test`, `stats.mannwhitneyu`, and `stats.wilcoxon` methods are nonparametric since they rely on no specific population assumptions such as distributional normality. \n",
    "\n",
    "D. With respect to the null hypothesis specification, \n",
    "`stats.ttest_1samp` is most similar to `stats.ttest_ind` and `stats.ttest_rel`, \n",
    "while `stats.binom` is most closely related to `stats.median_test`, `stats.mannwhitneyu`, and `stats.wilcoxon`.\n",
    "\n",
    "E. `stats.ttest_1samp` and `stats.binom` are only good for 50/50 chance problems; so, `stats.median_test`, `stats.mannwhitneyu`, and `stats.ttest_ind` exist in order to consider different comparision populations.  \n",
    " \n",
    "> - Hint 1: The following \"fun fact\" is not at all obvious at this stage; but, it would be possible to reformulate the p-value computed from the `stats.median_test` using `stats.binom`, and it would be similarly possible to reformulate the p-value computed from `stats.ttest_rel` using `stats.ttest_1samp`; however, the null hypotheses for `stats.median_test` and `stats.ttest_rel` always involve two populations; whereas, the null hypotheses for `stats.binom` and `stats.ttest_1samp` can support hypotheses involving only a single population...  `stats.ttest_1samp`, `stats.ttest_ind`, and `stats.ttest_rel` are analagous with respect to the distributional assumptions they make on the populations they consider, but not necessarily with respect to their specific problem contexts, which address different kinds of comparison and hence involve qualitatively different forms of null hypotheses...\n",
    "> - Hint 2: \n",
    "> ```python\n",
    "> print((1-stats.binom(n=100, p=0.5).cdf(60-1))*2)\n",
    "> # Probability of getting a combination more as or extreme than 60 heads in 100 coin flips for a 50-50 coin\n",
    "> coin_flips = [1] * 60 + [0] * 40  # 1 for head, 0 for tails\n",
    "> stats.ttest_1samp(coin_flips, 0.5) \n",
    "> # Probability of getting a combination as or more extreme than 60 heads in 100 coin flips for a 50-50 coin\n",
    "> ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2c22e50",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell for some scratch work if desired...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "250b3cd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q13: your answer will be tested!\n",
    "Q13 = None # Assign either 'A', 'B', 'C', 'D', or 'E' to `Q13` instead of `None`\n",
    "# E.g., Q13 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5e7a873",
   "metadata": {},
   "source": [
    "### Q14: Create a 90% bootstrap confidence interval estimating the difference in median anxiety scores between the social media usage groups. \n",
    "\n",
    "#### Provide your interval endpoint answers rounded to 3 decimal places.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72513439",
   "metadata": {},
   "source": [
    "> - Hint 1: The process within the `for` loop in `Q4` is not quite right since it's based on the assumption of a null hypothesis that the labels don't matter; however, when making a confidence interval we do think the labels matter! So, the `for` loop for a confidence interval should instead provide a simulation producing repeated pairs of bootstrap samples which can then be used to create simulated median difference test statistics that are samples from the bootrapped sampling distribution of the difference in medians...  \n",
    ">\n",
    ">\n",
    "> - Hint 2: Don't forget about the `np.quantile()` function...\n",
    ">\n",
    ">\n",
    "> - Hint 3: Be careful that you haven't accidentally overwritten the values in `anxiety_data.anxiety_scores`...\n",
    ">   - Working instead with something like `bootstrapped_data = anxiety_data.copy()` and `bootstrapped_anxiety_scores = anxiety_data.anxiety_scores.values.copy()` would help protect against this... and you can tell pretty quickly that you might be making this mistake if your simulated median difference statistics all end up having the same value!\n",
    ">\n",
    ">\n",
    "> - Hint 4: code like the following\n",
    ">   ```python\n",
    ">   bootstrapped_data['anxiety_scores'][anxiety_data.social_media_usage==\"Low\"] = \\\n",
    ">      anxiety_data['anxiety_scores'][anxiety_data.social_media_usage==\"Low\"].\\\n",
    ">        sample(frac=1, replace=True).values\n",
    ">   ```\n",
    ">   will produce a\n",
    ">\n",
    ">   `SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame`  \n",
    ">\n",
    ">   because `bootstrapped_data['anxiety_scores'][anxiety_data.social_media_usage==\"Low\"]` is itself a \"slice\" of a `pd.DataFrame` even if `bootstrapped_data = anxiety_data.copy()`; so, instead, sequentially assign into something like \n",
    ">   - `bootstrapped_anxiety_scores = anxiety_data.anxiety_scores.values.copy()` \n",
    ">     - `bootstrapped_anxiety_scores[anxiety_data.social_media_usage==\"Low\"] = ...`\n",
    ">     - `bootstrapped_anxiety_scores[anxiety_data.social_media_usage==\"High\"] = ...`\n",
    ">   - and finally `bootstrapped_data['anxiety_scores'] = bootstrapped_anxiety_scores` without any \"slice\" usage in order to use \n",
    ">   - `np.diff(bootstrapped_data.groupby('social_media_usage').median().values.flatten())[0]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4057fc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# space for scratch work if needed\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "198c5700",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep the following line at the top! And, replace `None` below with the three digit integer \n",
    "np.random.seed(None) # made from the 1st, 3rd, and 5th digits or your student number. \n",
    "\n",
    "# Code your bootstrap sampling distribution simulation here... \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f10fa7c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "three_digit_random_nubmer_seed = None \n",
    "# Match the integer created from the 1st, 3rd, and 5th digits of your student number used above...\n",
    "# your results must be based on this random number seed so that they can be confirmed during autotesting\n",
    "number_of_bootstrap_sample_repetitions = None # number of simulated median difference test statistics\n",
    "# sampled from the bootstrapped sampling distribution of the difference in medians\n",
    "the_90percent_confidence_interval = (None, None) # Assign your 90% confidence interval lower and upper bounds\n",
    "# Round to 3 digits of accuracy\n",
    "Q14 = (three_digit_random_nubmer_seed, number_of_bootstrap_sample_repetitions, the_90percent_confidence_interval)\n",
    "# E.g. `Q14 = (123, 10000, (0.123, 0.456))`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a870f5",
   "metadata": {
    "deletable": false
   },
   "source": [
    "## Airbags\n",
    "The table below is adapted from a \"Biostatistics for the Biological and Health Sciences\" textbook example and presents data from a random sample of passengers sitting in the front seat of cars involved in car crashes. Based on this data we'd like to make a determination as to whether or not death rates differ for passengers in cars with airbags and passengers in cars without airbags.\n",
    "\n",
    "|                           | Airbag available | No airbag available |\n",
    "|---------------------------|------------------|---------------------|                           \n",
    "| Passenger Fatalities      |  45              | 62                  |\n",
    "| Total number of Passengers|  10,541          | 9,867               |\n",
    "\n",
    "- The code below creates a pandas data frame for this problem with the help of the `numpy.repeat` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7c2b718",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "crash_data = pd.DataFrame({'group': np.repeat([\"airbag\", \"no_airbag\"], [10541, 9867]), \n",
    "                           'outcome': np.repeat([\"dead\", \"alive\", \"dead\", \"alive\"], [45, 10541-45, 62, 9867-62])})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f50ade0",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q15: In simple terms, state the claim of the *null hypothesis* for the context above that we are naturally trying to provide evidence against.  Then state this formally in terms of the two population parameters -- probabilities of death $p_{airbag}$ and $p_{no-airbag}$ -- in question. Finally, formally state the *alternative hypothesis* $H_1$ that makes this a *one-sided hypothesis test* such that (during the p-value calculation process) \"as or more extreme\" only considers evidence indicating that survival rates are strictly better in cars with airbags. \n",
    "\n",
    "#### Provide your written answer in the markdown cell below.\n",
    "\n",
    "> - Hint 1: Formulate the null and alternative hypotheses in the requested 'one-sided' manner (as opposed to as a 'two-sided' specification) by using using a \"less than\" \"$<$\" sign as opposed to the typical \"$H_1: H_0\\text{ is false}$\".  Doing so indicates that there is no evidence against the null hypothesis if the survival rates are worse for cars with airbags; but, on the other hand, this actually tends to reduce p-values by a factor of two since it means only considering \"as or more extreme\" to be on one side (\"better than\" direction only) of the sampling distribution implied by the null hypothesis.\n",
    ">   - You might have come across the idea that airbags can actually be dangerous in some situations; but, for now let's just consider airbags as being safety devices that might offer improvements in that regard... as we shall shortly see, this is generally suggested by the data at hand, anyway. \n",
    ">\n",
    ">\n",
    "> - Hint 2: In the previous \"Social Media and Anxiety\" example, it was suggested that it might be plausible for there to be a pre-existing association between anxiety and the choice to use social media more frequently. In this case, does it feel similarly possible that people who are more likely to die in a car crash (for whatever reason) would also be people who would choose to drive a car without an airbag? It feels like an argument of some kind of \"confounding\" affecting things here might be less likely than anxious people using more social media; but, what kind of story could you tell which might in fact produce \"confounding\" in the current car crash surival context? \n",
    "\n",
    "- The TA will manually review and confirm the correctness of your submitted answer for this question."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e3418eb",
   "metadata": {
    "deletable": false
   },
   "source": [
    "> Answer here...\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d7fdabf",
   "metadata": {},
   "source": [
    "### Q16: The code below shows that we observe only $0.2\\%$ less deaths in crashes for cars that have airbags compared to those that don't; but, that the proporitional increase of deaths in crashes for cars that have no airbags is $1.47 \\approx \\frac{0.006284}{0.004269}$.  \n",
    "\n",
    "#### Write 2-3 sentences in the markdown cell below exploring your sensibilities regarding this difference in absolute versus relative risk. \n",
    "\n",
    "```python\n",
    "crash_data['outcome_numeric'] = (crash_data['outcome']=='dead') # Boolean True/False \n",
    "crash_data.groupby('group')['outcome_numeric'].mean() # will coerce to 0/1 numeric\n",
    "# group\n",
    "# airbag       0.004269\n",
    "# no_airbag    0.006284\n",
    "np.diff(crash_data.groupby('group')['outcome_numeric'].mean().values.flatten())[0]\n",
    "# 0.002014526818295126\n",
    "np.divide(*np.flip(crash_data.groupby('group')['outcome_numeric'].mean().values.flatten()))\n",
    "# 1.471891715369976\n",
    "```\n",
    "\n",
    "- You can compare your answer with the example solution provided by MarkUs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45689e85",
   "metadata": {},
   "source": [
    "> Answer here...\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad0741b",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q17: Use label \"permutation shuffling\" to simulate and visualize the sampling distribution of the test statistic of the difference between $\\hat p_{no-airbag}$ and $\\hat p_{airbag}$ under the assumption that the *null hypothesis* stated in the previous question is true; then, provide a (one sided) \"permutation permutation test\" p-value of the observed test statistic relative to this simulated sampling distribution and indicate the strength of evidence provided against the null hypothesis above based on this simulated p-value.\n",
    "\n",
    "A. `0.10 < p-value`: No evidence against the null hypothesis  \n",
    "\n",
    "B. `0.05 < p-value <= 0.10`: Weak evidence against the null hypothesis  \n",
    "\n",
    "C. `0.01 < p-value <= 0.05`: Moderate evidence against the null hypothesis  \n",
    "\n",
    "D. `0.001 < p-value <= 0.01`: Strong evidence against the null hypothesis  \n",
    "\n",
    "E. `p-value <= 0.001`: Very strong evidence against the null hypothesis\n",
    "\n",
    "#### Provide your p-value with 3 digits of accuracy using `np.round(...,3)`.\n",
    "\n",
    "> - Hint 1: This should be a one-sided p-value for the one-sided null hypothesis specified above: how is \"as or more extreme\" calculated in this case?\n",
    ">   - Have we helped strengthen our evidence against the null hypothesis by using a one-sided hypothesis test as opposed to a two-sided hypothesis test?  If you try it out you'll likely see that indeed we have... \n",
    ">\n",
    ">\n",
    "> - Hint 2: The assumption that the *null hypothesis* stated in the previous question is true implies that the two samples must come from identical populations (which have identical survivial proportions); thus, a permutation test p-value similar to the one computed in Q4 is an appropriate nonparametric p-value for this context. \n",
    ">   - Of course we could also choose to use the alternative methods form computing p-values that we've encountered above (although we'd have to take care to use them so that they're based on a one-sided null hypothesis as opposed to a two sided null hypothesis...); but, for this question please create a \"permutation shuffling test\" p-value in order to practice this kind of simulation test. \n",
    ">\n",
    ">\n",
    "> - Hint 3: This problem considers *statistical significance* as opposed to *practical significance*: the question of *practical significance* was addressed in the previous problem where the 0.2% absolute risk increse and the 1.47 relative proprtional risk increase were considered. Always remember that even we're able to use this data to provide statistical evidence against the (one-sided) null hypothesis based that there is no safety difference, there's always still the real question of *practical significance*: is the difference that we estimate sufficiently large enough to be practically meaningful? \n",
    ">\n",
    ">\n",
    "> - Hint 4: Rather than coercing boolean values to compute proportions as is done in the previous problem, you could consider using something like `data.replace({'dead': 1, 'alive': 0})` to replace 'dead' and 'alive' outcomes with 1's and 0's as an initial first step that might simplify working with proportions slightly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f21b34d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# space for scratch work if needed\n",
    "# - Don't use variables here that haven't been defined in previuos cells up above since that would\n",
    "#   cause a \"variable not defined error\" when the notebook is run sequentially by MarkUs autotesting\n",
    "# - Instead use newly created variables by using them in new cells that you add below where they're defined\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74111033",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Keep the following line at the top! And, replace `None` below with the three digit integer \n",
    "np.random.seed(None) # made from the 1st, 3rd, and 5th digits or your student number. \n",
    "\n",
    "# Code your \"permutation shuffling test\" simulation here... \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9dff4e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "three_digit_random_nubmer_seed = None\n",
    "# Match the integer created from the 1st, 3rd, and 5th digits of your student number used above...\n",
    "# your results must be based on this random number seed so that they can be confirmed during autotesting\n",
    "number_of_bootstrap_sample_repetitions = None # integer number of simulated difference test statistics\n",
    "# drawn from the sampling distribution of the null hypothesis used to calculate your p-value\n",
    "permutation_p_value = None # Assign to p_value your calculated p-value rounded to 3 decimal places\n",
    "strength_of_evidence_against_H0 = None # Assign either 'A' or 'B' or 'C' or 'D' or 'E' instead of `None`\n",
    "# E.g., strength_of_evidence_against_H0 = 'A'\n",
    "\n",
    "Q17 = (three_digit_random_nubmer_seed, number_of_bootstrap_sample_repetitions, \n",
    "       permutation_p_value, strength_of_evidence_against_H0)\n",
    "# E.g. `Q17 = (123, 10000, 0.123, 'A')`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6da9829",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q18: Create a 95% bootstrap confidence interval estimating the relative (ratio) risk in probability of death ($p_{no-airbag}$ divided by $p_{airbag}$) between passengers in cars with airbags and passengers in cars without airbags. Then visualize the distribution of the bootstrapped sample proportion ratios.\n",
    "\n",
    "#### Provide your bootstrap confidence interval endpoint answers rounded to 2 decimal places.\n",
    "\n",
    "> - Hint 1: You could first make this analysis for the difference between these two proportions, which would be more analagous to the \"permutation shuffling test\" used in the previous question, and then change the bootstrapped test statistic from a difference to the desired ratio...\n",
    ">   - The difference between the difference and ratio test statistic being used is a question of absolute risk versus relative risk, as was discussed in Q16... making a confidence interval for the relative risk means we've lost the context of the absolute risk, but of course this allows us to emphasize the relative risk in our bootstrapped confidence interval range estimation.\n",
    ">\n",
    ">\n",
    "> - Hint 2: If the test statistic under consideration was the difference rather than the ratio, we'd need to round our answers to 5 decimal places, since the actual absolute probabilities under consideration are quite small (at less than 1%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56c836d7",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# space for scratch work if needed\n",
    "# - Don't use variables here that haven't been defined in previuos cells up above since that would\n",
    "#   cause a \"variable not defined error\" when the notebook is run sequentially by MarkUs autotesting\n",
    "# - Instead use newly created variables by using them in new cells that you add below where they're defined\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a4c5404",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep the following line at the top! And, replace `None` below with the three digit integer \n",
    "np.random.seed(None) # made from the 1st, 3rd, and 5th digits or your student number. \n",
    "\n",
    "# Code your bootstrap sampling distribution simulation here... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "912f9a29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize the distribution of the bootstrapped sample proportion differences here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3222d040",
   "metadata": {},
   "outputs": [],
   "source": [
    "three_digit_random_nubmer_seed = None\n",
    "# Match the integer created from the 1st, 3rd, and 5th digits of your student number used above...\n",
    "# your results must be based on this random number seed so that they can be confirmed during autotesting\n",
    "number_of_bootstrap_sample_repetitions = None # number of simulated proportion difference test statistics\n",
    "# sampled from the bootstrapped sampling distribution of the difference in death proportions\n",
    "the_95percent_confidence_interval = (None, None) # Assign your 95% confidence interval lower and upper bounds\n",
    "# Round to 2 digits of accuracy\n",
    "Q18 = (three_digit_random_nubmer_seed, number_of_bootstrap_sample_repetitions, the_95percent_confidence_interval)\n",
    "# E.g. `Q18 = (123, 10000, (0.12, 0.34))`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1df85ca3",
   "metadata": {},
   "source": [
    "## Type I and II Errors\n",
    "\n",
    "In the analyses of the previous week and this week, you've used a quantiative p-value to provide a qualitative statement of the evidence against the *null hypothesis*. This doesn't necessarily entail a formal decision about a *null hypothesis* (and there are many good reasons to proceed in this manner without doing so). However, it would also be possible to make a formal statement about the *null hypothesis* which should ideally take the form of \"I choose to reject the *null hypothesis* $H_0$ on the basis of the evidence at hand\", or contrarily \"I fail to reject the *null hypothesis*...\" \n",
    " \n",
    "In *formal statstical hypothesis testing*, wrongly rejecting a *null hypothesis* which is actually true is called a *type I error*; whereas, wrongly failing to reject a *null hypothesis* which is actually false is called a *type II error*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c137c4f1",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q19: Make a statement about the null hypothesis based on your analysis above in Q17. What kind of error could you have made in your conclusion?\n",
    "A. I choose to reject the *null hypothesis* $H_0$ on the basis of the evidence at hand, so if the probability of death in crashes for cars that have an airbag and those without an airbag is actually the same, then we have made a Type I Error.  \n",
    "\n",
    "B. I choose to reject the *null hypothesis* $H_0$ on the basis of the evidence at hand, so if the probability of death in crashes for cars that have an airbag and those without an airbag is actually the same, then we have made a Type II Error.   \n",
    "\n",
    "C. I fail to reject the *null hypothesis* $H_0$ on the basis of the evidence at hand, so if the probability of death in crashes for cars without an airbag is actually greater than cars with an airbag, then we have made a Type I Error.  \n",
    "\n",
    "D. I fail to reject the *null hypothesis* $H_0$ on the basis of the evidence at hand, so if the probability of death in crashes for cars without an airbag is actually greater than cars with an airbag, then we have made a Type II Error.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cde3b390",
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "# Q19: your answer will be tested!\n",
    "Q19 = None # Assign either 'A' or 'B' or 'C' or 'D' to `Q19` instead of `None`\n",
    "# E.g., Q19 = 'A'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f41ba201",
   "metadata": {
    "deletable": false
   },
   "source": [
    "### Q20: Assuming your analysis has provided some evidence against the null hypothesis of no survival differences in car crashes between cars with and without airbags, are you ready to suggest and support the claim that this data supports the claim that \"airbags save lives\"? Explain how this data could be confounded by either (a) \"death wish\" drivers and (b) age in a manner that could produce the observed survival rate differences and therefore shed doubt on the claim that \"airbags save lives\" and discuss wheather or not you find entertaining each of these two possibilites compelling.\n",
    "\n",
    "> - Hint: In the \"Social Media and Anxiety\" problem we imagined that people who were already prone to anxiety might also be more likely to use social media with a higher frequency. What would be a story about (a) \"death wish drivers\" and (b) \"age\" that could cause people to both tend to drive cars without airbags and be more likely die in a car crash?\n",
    "\n",
    "#### Provide your written answer about four to six sentences in the markdown cell below.  \n",
    "- This will be manually reviewed by your TA.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c5b5e8d",
   "metadata": {
    "deletable": false
   },
   "source": [
    "> Answer here..."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 6
}