08-sampling.Rmd

# (PART) Statistical Inference via infer {-} 

# Sampling {#sampling}

```{r setup_infer, include=FALSE, purl=FALSE}
chap <- 8
lc <- 0
rq <- 0
# **`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`**
# **`r paste0("(RQ", chap, ".", (rq <- rq + 1), ")")`**

knitr::opts_chunk$set(
  tidy = FALSE, 
  out.width = '\\textwidth', 
  fig.height = 4,
  fig.align='center',
  warning = FALSE
  )

options(scipen = 99, digits = 3)

# Set random number generator see value for replicable pseudorandomness
set.seed(76)
```

In this chapter, we kick off the third segment of this book, statistical inference, by learning about **sampling**. The concepts behind sampling form the basis of confidence intervals and hypothesis testing, which we'll cover in Chapters \@ref(confidence-intervals) and \@ref(hypothesis-testing) respectively. We will see that the tools that you learned in the data science segment of this book, in particular, data visualization and data wrangling, will also play an important role here in the development of your understanding.  As mentioned before, the concepts throughout this text all build into a culmination allowing you to "think with data." 


### Needed packages {-}

Let's load all the packages needed for this chapter (this assumes you've already installed them). Recall from our discussion in Section \@ref(tidyverse-package) that loading the `tidyverse` package by running `library(tidyverse)` loads the following commonly used data science packages all at once:

* `ggplot2` for data visualization
* `dplyr` for data wrangling
* `tidyr` for converting data to "tidy" format
* `readr` for importing spreadsheet data into R
* As well as the more advanced `purrr`, `tibble`, `stringr`, and `forcats` packages

If needed, read Section \@ref(packages) for information on how to install and load R packages. 

```{r message=FALSE, warning=FALSE}
library(tidyverse)
library(moderndive)
```

```{r message=FALSE, warning=FALSE, echo=FALSE}
# Packages needed internally, but not in text.
library(knitr)
library(kableExtra)
library(patchwork)
```







## Sampling bowl activity {#sampling-activity}

Let's start with a hands-on activity.

### What proportion of this bowl's balls are red?

Take a look at the bowl in Figure \@ref(fig:sampling-exercise-1). It has a certain number of red and a certain number of white balls all of equal size. Furthermore, it appears the bowl has been mixed beforehand as there does not seem to be any particular pattern to the spatial distribution of red and white balls. 

Let's now ask ourselves, what proportion of this bowl's balls are red?

```{r sampling-exercise-1, echo=FALSE, fig.cap="A bowl with red and white balls.", purl=FALSE, out.width = "80%"}
knitr::include_graphics("images/sampling/balls/sampling_bowl_1.jpg")
```

One way to answer this question would be to perform an exhaustive count: remove each ball individually, count the number of red balls and the number of white balls, and divide the number of red balls by the total number of balls. However, this would be a long and tedious process. 

### Using the shovel once 

Instead of performing an exhaustive count, let's insert a shovel into the bowl as seen in Figure \@ref(fig:sampling-exercise-2).

```{r sampling-exercise-2, echo=FALSE, fig.cap="Inserting a shovel into the bowl.", purl=FALSE, out.width = "80%"}
knitr::include_graphics("images/sampling/balls/sampling_bowl_2.jpg")
```

Using the shovel, we remove a number of balls as seen in Figure \@ref(fig:sampling-exercise-3).

```{r sampling-exercise-3, echo=FALSE, fig.cap="Fifty balls from the bowl.", purl=FALSE, out.width = "80%"}
knitr::include_graphics("images/sampling/balls/sampling_bowl_3_cropped.jpg")
```

Observe that 17 of the balls are red and there are a total of 5 x 10 = 50 balls and thus 0.34 = 34% of the shovel's balls are red. We can view the proportion of balls that are red *in this shovel* as a guess of the proportion of balls that are red *in the entire bowl*. While not as exact as doing an exhaustive count, our guess of 34% took much less time and energy to obtain. 

However, say, we started this activity over from the beginning. In other words, we replace the 50 balls back into the bowl and start over. Would we remove exactly 17 red balls again? In other words, would our guess at the proportion of the bowl's balls that are red be exactly 34% again? Maybe? 

What if we repeated this exercise several times? Would we obtain exactly 17 red balls each time? In other words, would our guess at the proportion of the bowl's balls that are red be exactly 34% every time? Surely not. Let's actually do and observe the results with the help of 33 of our friends.

### Using the shovel 33 times {#student-shovels}

Each of our 33 friends will do the following: 

- use the shovel to remove 50 balls each, 
- count the number of red balls, 
- use this number to compute the proportion of the 50 balls they removed that are red, 
- return the balls into the bowl, and 
- mix the contents of the bowl a little to not let a previous group's results influence the next group's set of results. 

```{r sampling-exercise-3b, echo=FALSE, fig.show='hold', fig.cap="Repeating sampling activity 33 times.", purl=FALSE, out.width = "20%"}
# Need new picture
knitr::include_graphics(c("images/sampling/balls/tactile_2_a.jpg", "images/sampling/balls/tactile_2_b.jpg", "images/sampling/balls/tactile_2_c.jpg"))
```

However, before returning the balls into the bowl, they are going to mark the proportion of the 50 balls they removed that are red in a histogram as seen in Figure \@ref(fig:sampling-exercise-4).

```{r sampling-exercise-4, echo=FALSE, fig.cap="Constructing a histogram of proportions.", purl=FALSE, out.width = "80%"}
knitr::include_graphics("images/sampling/balls/tactile_3_a.jpg")
```

Recall from Section \@ref(histograms) that histograms allow us to visualize the *distribution* \index{distribution} of a numerical variable: where the values center and in particular how they vary. The resulting hand-drawn histogram can be seen in Figure \@ref(fig:sampling-exercise-5).

```{r sampling-exercise-5, echo=FALSE, fig.cap="Hand-drawn histogram of 33 proportions.", purl=FALSE, out.width = "80%"}
knitr::include_graphics("images/sampling/balls/tactile_3_c.jpg")
```

Observe the following about the histogram in Figure \@ref(fig:sampling-exercise-5):

* At the low end, one group removed 50 balls from the bowl with proportion between 0.20 = 20% and 0.25 = 25%.
* At the high end, another group removed 50 balls from the bowl with proportion between 0.45 = 45% and 0.5 = 50% red.
* However the most frequently occurring proportions were between 0.30 = 30% and 0.35 = 35% red, right in the middle of the distribution.
* The shape of this distribution is somewhat bell-shaped. 

Let's construct this same hand-drawn histogram in R using your data visualization skills that you honed in Chapter \@ref(viz). We saved our 33 group of friends' proportion red in a data frame `tactile_prop_red` which is included in the `moderndive` package you loaded earlier. 

```{r, eval=FALSE}
tactile_prop_red
View(tactile_prop_red)
```

Let's display only the first 10 out of 33 rows of `tactile_prop_red`'s contents in Table \@ref(tab:tactile-red1).

```{r tactile-red1, echo=FALSE}
tactile_prop_red %>% 
  slice(1:10) %>% 
  kable(
    digits = 3,
    caption = "\\label{tab:tactilered}First 10 out of 33 groups' proportion of 50 balls that are red.", 
    booktabs = TRUE,
    longtable = TRUE,
    align = rep('r', 4)
  ) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position", "repeat_header"))
```

Observe for each `group` we have their names, the number of `red_balls` they obtained, and the corresponding proportion out of 50 balls that were red named `prop_red`. Observe, we also have a variable `replicate` enumerating each of the 33 groups; we chose this name because each row can be viewed as one instance of a replicated activity: using the shovel to remove 50 balls and computing the proportion of those balls that are red. 

We visualize the distribution of these 33 proportions using a `geom_histogram()` with `binwidth = 0.05` in Figure \@ref(fig:samplingdistribution-tactile), which is appropriate since the variable `prop_red` is numerical. This computer-generated histogram matches our hand-drawn histogram from the earlier Figure \@ref(fig:sampling-exercise-5). 

```{r eval=FALSE}
ggplot(tactile_prop_red, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = "Proportion of 50 balls that were red", 
       title = "Distribution of 33 proportions red") 
```
```{r samplingdistribution-tactile, echo=FALSE, fig.cap="Distribution of 33 proportions based on 33 samples of size 50."}
tactile_histogram <- ggplot(tactile_prop_red, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white")
tactile_histogram + 
  labs(x = "Proportion of 50 balls that were red", 
       title = "Distribution of 33 proportions red")
```

### What did we just do?

What we just demonstrated in this activity is the statistical concept of \index{sampling} *sampling*. We would like to know the proportion of the bowl's balls that are red, but because the bowl has a very large number of balls performing an exhaustive count of the number of red and white balls in the bowl would be very costly in terms of both time and energy. We, therefore, extract a sample of 50 balls using the shovel. Using this sample of 50 balls, we estimate the proportion of the bowl's balls that are red using the proportion of the shovel's balls that are red. This estimate in our earlier example was 17 red balls out of 50 balls = 34%. Moreover, because we mixed the balls before each use of the shovel, the samples were randomly drawn. Because each sample was drawn at random, the samples were different from each other. Because the samples were different from each other, we obtained the different proportions red observed in Table \@ref(tab:tactile-red1). This is known as the concept of *sampling variation*. \index{sampling!variation}

In Section \@ref(sampling-simulation) we'll mimic the hands-on sampling activity we just performed in a *computer simulation*; \index{computer simulation} using a computer will allow us to repeat the above sampling activity much more than 33 times. Using a computer, not only will be able to repeat the hands-on activity a very large number of times, but we will also be able to repeat it using different sized shovels. 

The purpose of these simulations is to develop an understanding of two key concepts relating to sampling: understanding the concept of sampling variation and the role that sample size plays in this variation. To this end, we'll present you with definitions, terminology, and notation related to sampling in Section \@ref(sampling-framework). As with many disciplines, there are definitions, terminology, and notation that seem very inaccessible and even confusing at first. However, as with many difficult topics, if you truly understand the underlying concepts and practice, practice, practice, you'll be able to master these topics.

To tie the contents of this chapter to the real-word, we'll present an example of one of the most recognizable uses of sampling: polls. In Section \@ref(sampling-case-study) we'll look at a particular case study: a 2013 poll on then President Obama's popularity among young Americans, conducted by the Harvard Kennedy School's Institute of Politics.

We'll close this chapter by generalizing the above sampling from the bowl activity to other scenarios, distinguishing between *random sampling* and *random assignment*, presenting the theoretical result underpinning all our results, and presenting a few mathematical formulas that relate to the concepts and ideas explored in this chapter.

```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Why was it important to mix the bowl before we sampled the balls?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Why is it that our 33 groups of friends did not all have the same numbers of balls that were red out of 50, and hence different proportions red?

```{block, type='learncheck', purl=FALSE}
```




## Computer simulation of sampling {#sampling-simulation}

What we performed in Section \@ref(sampling-activity) is a *simulation* of sampling. In other words, we were not in a real-life sampling scenario in order to answer a real-life question, but rather we were mimicking such a scenario with our bowl and shovel. The crowd-sourced Wikipedia definition of simulation states: "A simulation is an approximate imitation of the operation of a process or system." One example of simulations in practice is flight simulators: before pilots in training are allowed to fly an actual plane, they first practice on a computer that attempts to mimic the reality of flying an actual plane as best as possible. 

Now you might be thinking that simulations must necessarily take place on a computer. However, this is not necessarily true. Take for example crash test dummies: before cars are made available to the market, automobile engineers test their safety by mimicking the reality for passengers of being in an automobile crash. To distinguish between these two simulation types, we'll term a simulation performed in real-life as a "tactile" simulation done with your hands and to the touch as opposed to a "virtual" simulation performed on a computer.

So while in Section \@ref(sampling-activity) we performed a "tactile" simulation of sampling using an actual bowl and an actual shovel with our hands, in this section we'll perform a "virtual" simulation using a "virtual" bowl and a "virtual" shovel with our computers.


### Using the virtual shovel once

Let's start by performing the virtual analogue of the tactile sampling simulation we performed in \@ref(sampling-activity). We first need a virtual analogue of the bowl seen in Figure \@ref(fig:sampling-exercise-1). To this end, we included a data frame `bowl` in the `moderndive` package whose rows correspond exactly with the contents of the actual bowl. 

```{r}
bowl
```

Observe in the output that `bowl` has 2400 rows, telling us that the bowl contains 2400 equally-sized balls. The first variable `ball_ID` is used merely as an "identification variable" for this data frame as discussed in Subsection \@ref(identification-vs-measurement-variables); none of the balls in the actual bowl are marked with numbers. The second variable `color` indicates whether a particular virtual ball is red or white. View the contents of the bowl in RStudio's data viewer and scroll through the contents to convince yourselves that `bowl` is indeed a virtual version of the actual bowl in Figure \@ref(fig:sampling-exercise-1).

Now that we have a virtual analogue of our bowl, we now need a virtual analogue for the shovel seen in Figure \@ref(fig:sampling-exercise-2); we'll use this virtual shovel to generate our virtual random samples of 50 balls. We're going to use the `rep_sample_n()` function included in the `moderndive` package. This function allows us to take `rep`eated, or `rep`licated, `samples` of size `n`. Run the following and explore `virtual_shovel`'s contents in the RStudio viewer.

```{r, eval=FALSE}
virtual_shovel <- bowl %>% 
  rep_sample_n(size = 50)
View(virtual_shovel)
```

Let's display only the first 10 out of 50 rows of `virtual_shovel`'s contents in Table \@ref(tab:virtual-shovel).

```{r virtual-shovel, echo=FALSE}
virtual_shovel <- bowl %>% 
  rep_sample_n(size = 50)
virtual_shovel %>% 
  slice(1:10) %>%
  knitr::kable(
    align = c("r", "r"),
    digits = 3,
    caption = "First 10 sampled balls of 50 in virtual sample",
    booktabs = TRUE
  ) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position"))
```

The `ball_ID` variable identifies which of the balls from `bowl` are included in our sample of 50 balls and `color` denotes its color. However what does the `replicate` variable indicate? In `virtual_shovel`'s case, `replicate` is equal to 1 for all 50 rows. This is telling us that these 50 rows correspond to a first repeated/replicated use of the shovel, in our case our first sample. We'll see below when we "virtually" take 33 samples, `replicate` will take values between 1 and 33. Before we do this, let's compute the proportion of balls in our virtual sample of size 50 that are red using the `dplyr` data wrangling verbs you learned in Chapter \@ref(wrangling). Let's breakdown the steps individually:

First, for each of our 50 sampled balls, identify if it is red using a test for equality using `==`. For every row where `color == "red"`, the Boolean `TRUE` is returned and for every row where `color` is not equal to `"red"`, the Boolean `FALSE` is returned. Let's create a new Boolean variable `is_red` using the `mutate()` function from Section \@ref(mutate):

```{r}
virtual_shovel %>% 
  mutate(is_red = (color == "red"))
```

Second, we compute the number of balls out of 50 that are red using the `summarize()` function. Recall from Section \@ref(summarize) that `summarize()` takes a data frame with many rows and returns a data frame with a single row containing summary statistics that you specify, like `mean()` and `median()`. In this case we use the `sum()`:

```{r}
virtual_shovel %>% 
  mutate(is_red = (color == "red")) %>% 
  summarize(num_red = sum(is_red))  
```

Why does this work? Because R treats `TRUE` like the number `1` and `FALSE` like the number `0`. So summing the number of `TRUE`'s and `FALSE`'s is equivalent to summing `1`'s and `0`'s, which in the end counts the number of balls where `color` is `red`. In our case, 17 of the 50 balls were red. 

Third and last, we compute the proportion of the 50 sampled balls that are red by dividing `num_red` by 50:

```{r}
virtual_shovel %>% 
  mutate(is_red = color == "red") %>% 
  summarize(num_red = sum(is_red)) %>% 
  mutate(prop_red = num_red / 50)
```

In other words, this "virtual" sample's balls were 34% red. Let's make the above code a little more compact and succinct by combining the first `mutate()` and the `summarize()` as follows:

```{r}
virtual_shovel %>% 
  summarize(num_red = sum(color == "red")) %>% 
  mutate(prop_red = num_red / 50)
```

Great! 34% of `virtual_shovel`'s 50 balls were red! So based on this particular sample, our guess at the proportion of the `bowl`'s balls that are red is 34%. But remember from our earlier tactile sampling activity that if we repeated this sampling, we would not necessarily obtain a sample of 50 balls with 34% of them being red again; there will likely be some variation. In fact in Table \@ref(tab:virtual-shovel) we displayed 33 such proportions based on 33 tactile samples and then in Figure \@ref(fig:sampling-exercise-5) we visualized the distribution of the 33 proportions in a histogram. Let's now perform the virtual analogue of having 33 groups of students use the sampling shovel!

### Using the virtual shovel 33 times

Recall that in our tactile sampling exercise in Section \@ref(sampling-activity) we had 33 groups of students each use the shovel, yielding 33 samples of size 50 balls, which we then used to compute 33 proportions. In other words we repeated/replicated using the shovel 33 times. We can perform this repeated/replicated sampling virtually by once again using our virtual shovel function `rep_sample_n()`, but by adding the `reps = 33` argument, indicating we want to repeat the sampling 33 times. Be sure to scroll through the contents of `virtual_samples` in RStudio's viewer. 

```{r, eval=FALSE}
virtual_samples <- bowl %>% 
  rep_sample_n(size = 50, reps = 33)
View(virtual_samples)
```
```{r, echo=FALSE}
virtual_samples <- bowl %>% 
  rep_sample_n(size = 50, reps = 33)
```

Observe that while the first 50 rows of `replicate` are equal to `1`, the next 50 rows of `replicate` are equal to `2`. This is telling us that the first 50 rows correspond to the first sample of 50 balls while the next 50 correspond to the second sample of 50 balls. This pattern continues for all `reps = 33` replicates and thus `virtual_samples` has 33 $\times$ 50 = 1650 rows. 

Let's now take the data frame `virtual_samples` with 33 $\times$ 50 = 1650 rows corresponding to 33 samples of size 50 balls and compute the resulting 33 proportions red. We'll use the same `dplyr` verbs as we did in the previous section, but this time with an additional `group_by()` of the `replicate` variable. Recall from Section \@ref(groupby) that by assigning the grouping variable "meta-data" before `summarizing()`, we'll obtain 33 different proportions red:

```{r, eval=FALSE}
virtual_prop_red <- virtual_samples %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 50)
View(virtual_prop_red)
```

Let's display only the first 10 out of 33 rows of `virtual_prop_red`'s contents in Table \@ref(tab:virtualred). As one would expect, there is variation in the resulting `prop_red` proportions red for the first 10 out 33 repeated/replicated samples.

<!-- Albert will remove `boundary` here on updates to chalkboard image if needed. Or will
explain why `boundary` is needed. -->

```{r virtualred, echo=FALSE}
virtual_prop_red <- virtual_samples %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 50)

virtual_prop_red %>% 
  slice(1:10) %>% 
  kable(
    digits = 3,
    caption = "First 10 out of 33 virtual proportion of 50 balls that are red.", 
    booktabs = TRUE,
    longtable = TRUE
  ) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position", "repeat_header"))
```

Let's visualize the distribution of these 33 proportions red based on 33 virtual samples using a histogram with `binwidth = 0.05` in Figure \@ref(fig:samplingdistribution-virtual). 

```{r eval=FALSE}
ggplot(virtual_prop_red, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = "Proportion of 50 balls that were red", 
       title = "Distribution of 33 proportions red") 
```
```{r samplingdistribution-virtual, echo=FALSE, fig.cap="Distribution of 33 proportions based on 33 samples of size 50."}
virtual_histogram <- ggplot(virtual_prop_red, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white")
virtual_histogram + 
  labs(x = "Proportion of 50 balls that were red", 
       title = "Distribution of 33 proportions red")
``` 

Observe that occasionally we obtained proportions red that is less than 0.3 = 30%, while on the other hand, occasionally we obtained proportions that are greater than 0.45 = 45%. However, the most frequently occurring proportions red out of 50 balls were between 35% and 40% (for 11 out of 33 samples). Why do we have these differences in proportions red? Because of sampling variation. 

Let's now compare our virtual results with our tactile results from the previous section in Figure \@ref(fig:tactile-vs-virtual). We see that both histograms, in other words, the distribution of the 33 proportions red, are *somewhat* similar in their center and spread although not identical. These slight differences are again due to random variation. Furthermore, both distributions are *somewhat* bell-shaped.

```{r tactile-vs-virtual, echo=FALSE, fig.cap="Comparing 33 virtual and 33 tactile proportions red."}
bind_rows(
  virtual_prop_red %>% 
    mutate(type = "Virtual sampling"), 
  tactile_prop_red %>% 
    select(replicate, red = red_balls, prop_red) %>% 
    mutate(type = "Tactile sampling")
) %>% 
  mutate(type = factor(type, levels = c("Virtual sampling", "Tactile sampling"))) %>% 
  ggplot(aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  facet_wrap(~type) +
  labs(x = "Proportion of 50 balls that were red", 
         title = "Comparing distributions")
```

```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Why couldn't we study the effects of sampling variation when we used the virtual shovel only once? Why did we need to take more than one virtual sample (in our case 33 virtual samples)?

```{block, type='learncheck', purl=FALSE}
```


### Using the virtual shovel 1000 times {#shovel-1000-times}

Now say we want to study the variation in proportions red not based on 33 repeated/replicated samples, but rather a very large number of samples say 1000 samples. We have two choices at this point. We could have our students manually take 1000 samples of 50 balls and compute the corresponding 1000 proportion red out 50 balls. This would be cruel and unusual, however, as this would be very tedious and time-consuming. This is where computers excel: automating long and repetitive tasks while performing them very quickly. Therefore, at this point, we will abandon tactile sampling in favor of only virtual sampling. Let's once again use the `rep_sample_n()` function with sample `size` set to 50 once again, but this time with the number of replicates `reps = 1000`. Be sure to scroll through the contents of `virtual_samples` in RStudio's viewer. 

```{r, eval=FALSE}
virtual_samples <- bowl %>% 
  rep_sample_n(size = 50, reps = 1000)
View(virtual_samples)
```
```{r, echo=FALSE}
virtual_samples <- bowl %>% 
  rep_sample_n(size = 50, reps = 1000)
```


Observe that now `virtual_samples` has 1000 $\times$ 50 = 50,000 rows, instead of the 33 $\times$ 50 = 1650 rows from earlier. Using the same code as earlier, let's take the data frame `virtual_samples` with 1000 $\times$ 50 = 50,000 and compute the resulting 1000 proportions red. 

```{r, eval=FALSE}
virtual_prop_red <- virtual_samples %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 50)
View(virtual_prop_red)
```

Observe that we now have 1000 replicates of `prop_red`, the proportion of 50 balls that are red. Using the same code as earlier, let's now visualize the distribution of these 1000 replicates of `prop_red` in a histogram in Figure \@ref(fig:samplingdistribution-virtual-1000).

<!-- Albert will remove `boundary` here on updates to chalkboard image if needed. -->

```{r eval=FALSE}
ggplot(virtual_prop_red, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = "Proportion of 50 balls that were red", 
       title = "Distribution of 1000 proportions red") 
```
```{r samplingdistribution-virtual-1000, echo=FALSE, fig.cap="Distribution of 1000 proportions based on 33 samples of size 50."}
virtual_prop_red <- virtual_samples %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 50)
virtual_histogram <- ggplot(virtual_prop_red, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white")
virtual_histogram + 
  labs(x = "Proportion of 50 balls that were red", 
       title = "Distribution of 1000 proportions red")
``` 

Once again, the most frequently occurring proportions red occur between 35% and 40%. Every now and then, we obtain proportions as low as between 20% and 25%, and others as high as between 55% and 60%. These are rare, however. Furthermore, observe that we now have a much more symmetric and smoother bell-shaped distribution. This distribution is, in fact, a Normal distribution; see the upcoming Section \@ref(normal-distribution) for a brief discussion on properties of the Normal distribution.


```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Why did we not take 1000 "tactile" samples of 50 balls by hand?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Looking at Figure \@ref(fig:samplingdistribution-virtual-1000) above, would you say that sampling 50 balls where 30% of them were red is likely or not? What about sampling 50 balls where 10% of them were red?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** 

```{block, type='learncheck', purl=FALSE}
```


### Using different shovels {#different-shovels}

Now say instead of just one shovel, you had three choices of shovels to extract a sample of balls with.

<!--
A shovel with 25 slots          |  A shovel with 50 slots  | A shovel with 100 slots
:-------------------------:|:-------------------------:|:-------------------------:
![](images/sampling/balls/shovel_025.jpg){ width=1.6in }  |  ![](images/sampling/balls/shovel_050.jpg){ width=1.6in } | ![](images/sampling/balls/shovel_100.jpg){ width=1.6in } 
-->

```{r three-shovels, echo=FALSE, fig.align='center', fig.cap="Three shovels to extract three different sample sizes.", out.width='100%', purl=FALSE}
knitr::include_graphics("images/sampling/balls/three_shovels.png")
```

If your goal was still to estimate the proportion of the bowl's balls that were red, which shovel would you choose? In our experience, most people would choose the shovel with 100 slots since it has the biggest sample size and hence would yield the "best" guess of the proportion of the bowl's 2400 balls that are red. Using our newly developed tools for virtual sampling simulations, let's unpack the effect of having different sample sizes! In other words, let's use `rep_sample_n()` with `size = 25`, `size = 50`, and `size = 100`, while keeping the number of repeated/replicated samples at 1000:

1. Virtually use the appropriate shovel to generate 1000 samples with `size` balls.
1. Compute the resulting 1000 replicated of the proportion of the shovel's balls that are red.
1. Visualize the distribution of these 1000 proportion red using a histogram.

Run each of the following code segments individually and then compare the three resulting histograms.

```{r, eval=FALSE}
# Segment 1: sample size = 25 ------------------------------
# 1.a) Virtually use shovel 1000 times
virtual_samples_25 <- bowl %>% 
  rep_sample_n(size = 25, reps = 1000)

# 1.b) Compute resulting 1000 replicates of proportion red
virtual_prop_red_25 <- virtual_samples_25 %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 25)

# 1.c) Plot distribution via a histogram
ggplot(virtual_prop_red_25, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = "Proportion of 25 balls that were red", title = "25") 
```

```{r, eval=FALSE}
# Segment 2: sample size = 50 ------------------------------
# 2.a) Virtually use shovel 1000 times
virtual_samples_50 <- bowl %>% 
  rep_sample_n(size = 50, reps = 1000)

# 2.b) Compute resulting 1000 replicates of proportion red
virtual_prop_red_50 <- virtual_samples_50 %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 50)

# 2.c) Plot distribution via a histogram
ggplot(virtual_prop_red_50, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = "Proportion of 50 balls that were red", title = "50")  
```

```{r, eval=FALSE}
# Segment 3: sample size = 100 ------------------------------
# 3.a) Virtually using shovel with 100 slots 1000 times
virtual_samples_100 <- bowl %>% 
  rep_sample_n(size = 100, reps = 1000)

# 3.b) Compute resulting 1000 replicates of proportion red
virtual_prop_red_100 <- virtual_samples_100 %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 100)

# 3.c) Plot distribution via a histogram
ggplot(virtual_prop_red_100, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = "Proportion of 100 balls that were red", title = "100") 
```

For easy comparison, we present the three resulting histograms in a single row with matching x and y axes in Figure \@ref(fig:comparing-sampling-distributions). What do you observe?

```{r comparing-sampling-distributions, echo=FALSE, fig.cap="Comparing the distributions of proportion red for different sample sizes."}
# n = 25
if(!file.exists("rds/virtual_samples_25.rds")){
  virtual_samples_25 <- bowl %>% 
    rep_sample_n(size = 25, reps = 1000)
  write_rds(virtual_samples_25, "rds/virtual_samples_25.rds")
} else {
  virtual_samples_25 <- read_rds("rds/virtual_samples_25.rds")
}
virtual_prop_red_25 <- virtual_samples_25 %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 25) %>% 
  mutate(n = 25)

# n = 50
if(!file.exists("rds/virtual_samples_50.rds")){
  virtual_samples_50 <- bowl %>% 
    rep_sample_n(size = 50, reps = 1000)
  write_rds(virtual_samples_50, "rds/virtual_samples_50.rds")
} else {
  virtual_samples_50 <- read_rds("rds/virtual_samples_50.rds")
}

virtual_prop_red_50 <- virtual_samples_50 %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 50) %>% 
  mutate(n = 50)

# n = 100
if(!file.exists("rds/virtual_samples_100.rds")){
  virtual_samples_100 <- bowl %>% 
    rep_sample_n(size = 100, reps = 1000)
  write_rds(virtual_samples_100, "rds/virtual_samples_100.rds")
} else {
  virtual_samples_100 <- read_rds("rds/virtual_samples_100.rds")
}
virtual_prop_red_100 <- virtual_samples_100 %>% 
  group_by(replicate) %>% 
  summarize(red = sum(color == "red")) %>% 
  mutate(prop_red = red / 100) %>% 
  mutate(n = 100)

virtual_prop <- bind_rows(virtual_prop_red_25, virtual_prop_red_50, virtual_prop_red_100)

comparing_sampling_distributions <- ggplot(virtual_prop, aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = "Proportion of shovel's balls that are red", title = "Comparing distributions of proportions red for 3 different shovels.") +
  facet_wrap(~n)
comparing_sampling_distributions
```

Observe that as the sample size increases, the spread of the 1000 replicates of the proportion of red decreases. In other words, as the sample size increases, there are fewer differences due to sampling variation and the distribution centers more tightly around the same value. Eyeballing Figure \@ref(fig:comparing-sampling-distributions), things appear to center tightly around roughly 40%.

We can be numerically explicit about the amount of spread in our 3 sets of 1000 values of `prop_red` using the *standard deviation* \index{standard deviation}: a summary statistic that measures the amount of spread and variation within a numerical variable; see Appendix \@ref(appendixA) for a brief discussion on properties of the standard deviation. For all three sample sizes, let's compute the standard deviation of the 1000 proportions red by running the following data wrangling code that uses the `sd()` summary function.

```{r, eval=FALSE}
# n = 25
virtual_prop_red_25 %>% 
  summarize(sd = sd(prop_red))

# n = 50
virtual_prop_red_50 %>% 
  summarize(sd = sd(prop_red))

# n = 100
virtual_prop_red_100 %>% 
  summarize(sd = sd(prop_red))
```

Let's compare these three measures of spread of the distributions in Table \@ref(tab:comparing-n).

```{r comparing-n, eval=TRUE, echo=FALSE}
comparing_n_table <- virtual_prop %>% 
  group_by(n) %>% 
  summarize(sd = sd(prop_red)) %>% 
  rename(`Number of slots in shovel` = n, `Standard deviation of proportions red` = sd) 

comparing_n_table  %>% 
  kable(
    digits = 3,
      caption = "Comparing standard deviations of proportions red for 3 different shovels.", 
      booktabs = TRUE
) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position"))
```

As we observed visually in Figure \@ref(fig:comparing-sampling-distributions), as the sample size increases our numerical measure of spread decreases; there is less variation in our proportions red. In other words, as the sample size increases, our guesses at the true proportion of the bowl's balls that are red get more consistent and precise. 

```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** In Figure \@ref(fig:comparing-sampling-distributions), we used shovels to take 1000 samples each, computed the resulting 1000 proportions of the shovel's balls that were red, and then visualized the distribution of these 1000 proportions in a histogram. We did this for shovels with 25, 50, and 100 slots in them. As the size of the shovels increased, the histograms got narrower. In other words, as the size of the shovels increased from 25 to 50 to 100, did the 1000 proportions

1. Vary less
1. Vary by the same amount
1. Vary more

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What summary statistic did we use to quantify how much the 1000 proportions red varied?

1. The inter-quartile range
1. The standard deviation
1. The range: the largest value minus the smallest

```{block, type='learncheck', purl=FALSE}
```



## Sampling framework {#sampling-framework}

In both our "hands-on" tactile simulations and our "virtual" simulations using a computer, we used sampling for the purpose of estimation: we extract samples in order to estimate the proportion of the bowl's balls that are red. We used sampling as a cheaper and less time consuming approach than to do a full census of all the balls. Our virtual simulations all built up to the results shown in Figure \@ref(fig:comparing-sampling-distributions) and Table \@ref(tab:comparing-n), comparing 1000 proportions red based on samples of size 25, 50, and 100. This was our first attempt at understanding two key concepts relating to sampling for estimation:

1. The effect of sampling variation on our estimates.
1. The effect of sample size on sampling variation.

Let's now introduce some terminologies and notation as well as statistical definitions related to sampling. Given the number of new words to learn, you will likely have to read these next three subsections multiple times. Keep in mind, however, that none of the concepts underlying these terminology, notation, and definitions are any different than the concepts underlying our simulations in Sections \@ref(sampling-activity) and \@ref(sampling-simulation); it will simply take time and practice to master them. 


### Terminology & notation {#terminology-and-notation}

Here is a list of terminology and mathematical notation relating to sampling. For each item, we'll be sure to tie them to our simulations in Sections \@ref(sampling-activity) and \@ref(sampling-simulation). 

1. **(Study) Population**: A (study) population is a collection of individuals or observations about which we are interested in. We mathematically denote the population's size using upper case $N$.  In our simulations the (study) population was the collection of $N$ = 2400 identically sized red and white balls contained in the bowl.
1. **Population parameter**: A population parameter is a numerical summary quantity about the population that is unknown, but you wish you knew. For example, when this quantity is a mean, the population parameter of interest is the *population mean* which is mathematically denoted with the Greek letter $\mu$ (pronounced "mu"). In our simulations however since we were interested in the proportion of the bowl's balls that were red, the population parameter is the *population proportion* which is mathematically denoted with the letter $p$. 
1. **Census**: An exhaustive enumeration or counting of all $N$ individuals or observations in the population in order to compute the population parameter's value *exactly*. In our simulations, this would correspond to manually going over all $N$ = 2400 balls in the bowl and counting the number that is red and computing the population proportion $p$ of the balls that are red *exactly*. When the number $N$ of individuals or observations in our population is large, as was the case with our bowl, a census can be very expensive in terms of time, energy, and money. 
1. **Sampling**: Sampling is the act of collecting a sample from the population when we don't have the means to perform a census. We mathematically denote the sample's size using lower case $n$, as opposed to upper case $N$ which denotes the population's size. Typically the sample size $n$ is much smaller than the population size $N$, thereby making sampling a much cheaper procedure than a census. In our simulations, we used shovels with 25, 50, and 100 slots to extract a sample of size $n$ = 25, $n$ = 50, and $n$ = 100 balls. 
1. **Point estimate (AKA sample statistic)**: A summary statistic computed from the sample that *estimates* the unknown population parameter. In our simulations, recall that the unknown population parameter was the population proportion and that this is mathematically denoted with $p$. Our point estimate is the *sample proportion*: the proportion of the shovel's balls that are red. In other words, it is our guess of the proportion of the bowl's balls balls that are red. We mathematically denote the sample proportion using $\widehat{p}$; the "hat" on top of the $p$ indicates that it is an estimate of the unknown population proportion $p$. 
1. **Representative sampling**: A sample is said to be a *representative sample* if it is representative of the population. In other words, are the sample's characteristics a good representation of the population's characteristics? In our simulations, are the samples of $n$ balls extracted using our shovels representative of the bowl's $N$=2400 balls?
1. **Generalizability**: We say a sample is *generalizable* if any results based on the sample can generalize to the population. In other words, can the value of the point estimate be generalized to estimate the value of the population parameter well? In our simulations, can we generalize the values of the sample proportions red of our shovels to the population proportion red of the bowl? Using mathematical notation, is $\widehat{p}$ a "good guess" of $p$?
1. **Bias**: In a statistical sense, we say *bias* occurs if certain individuals or observations in a population have a higher chance of being included in a sample than others. We say a sampling procedure is *unbiased* if every observation in a population had an equal chance of being sampled. In our simulations, since each ball had the same size and hence an equal chance of being a sample in our shovels, our samples were unbiased.
1. **Random sampling**: We say a sampling procedure is *random* if we sample randomly from the population in an unbiased fashion. In our simulations, this would correspond to sufficiently mixing the bowl before each use of the shovel. 

Phew, that's a lot of new terminology and notation to learn! Let's put them all together to describe the paradigm of sampling:

> * If the sampling of a sample of size $n$ is done at **random**, then
> * the sample is **unbiased** and **representative** of the population of size $N$, thus
> * any result based on the sample can **generalize** to the population, thus
> * the point estimate is a **"good guess"** of the unknown population parameter, thus
> * instead of performing a census, we can **infer** about the population using sampling.

Restricting consideration to a shovel with 50 slots from our simulations,

> * If we extract a sample of $n=50$ balls at **random**, in other words, we mix the equally-sized balls before using the shovel, then
> * the contents of the shovel are an **unbiased representation** of the contents of the bowl's 2400 balls, thus
> * any result based on the sample of balls can **generalize** to the bowl, thus
> * the sample proportion $\widehat{p}$ of the $n=50$ balls in the shovel that are red is a **"good guess"** of the population proportion $p$ of the $N$=2400 balls that are red, thus
> * instead of manually going over all the balls in the bowl, we can **infer** about the bowl using the shovel. 

Note that last word we wrote in bold: **infer**. The act of "inferring" is to deduce or conclude (information) from evidence and reasoning. In our simulations, we wanted to infer about the proportion of the bowl's balls that are red. *Statistical inference* is the theory, methods, and practice of forming judgments about the parameters of a population and the reliability of statistical relationships, typically on the basis of random sampling (Wikipedia). In other words, statistical inference is the act of inference via sampling. In the upcoming Chapter \@ref(confidence-intervals) on confidence intervals, we'll introduce the `infer` package, which makes statistical inference "tidy" and transparent. It is why this third portion of the book is called "Statistical inference via infer".

```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** In the case of our bowl activity, what is the *population parameter*? Do we know its value?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What would performing a census in our bowl activity correspond to? Why did we not perform a census?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What purpose do *point estimates* serve in general? What is the name of the point estimate specific to our bowl activity? What is its mathematical notation?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** How did we ensure that our tactile samples using the shovel were random?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Why is it important that sampling be done *at random*?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What are we *inferring* about the bowl based the samples using the shovel?


```{block, type='learncheck', purl=FALSE}
```


### Statistical definitions {#sampling-definitions}

Now for some important statistical definitions related to sampling. As a refresher of our 1000 repeated/replicated virtual samples of size $n$ = 25, $n$ = 50, and $n$ = 100 in Section \@ref(sampling-simulation), let's display Figure \@ref(fig:comparing-sampling-distributions) again below.  

```{r comparing-sampling-distributions-1b, echo=FALSE, fig.cap="Previously seen three sampling distributions of the sample proportion $\\widehat{p}$."}
comparing_sampling_distributions
```

These types of distributions have a special name: **sampling distributions**; \index{sampling distributions} their visualization displays the effect of sampling variation on the distribution of any point estimate, in this case, the sample proportion $\widehat{p}$. Using these sampling distributions, for a given sample size $n$, we can make statements about what values we can typically expect. For example, observe the centers of all three sampling distributions: they are all roughly centered around 0.4 = 40%. Furthermore, observe that while we are somewhat likely to observe sample proportions red of 0.2 = 20% when using the shovel with 25 slots, we will almost never observe this sample proportion when using the shovel with 100 slots. Observe also the effect of sample size on the sampling variation. As the sample size $n$ increases from 25 to 50 to 100, \index{sampling distribution!relationship to sample size} the spread/variation of the sampling distribution decreases and thus the values cluster more and more tightly around the same center of around 40%. We quantified this spread/variation using the standard deviation of our proportions in Table \@ref(tab:comparing-n), which we display again below:

```{r comparing-n-repeat, eval=TRUE, echo=FALSE}
comparing_n_table <- virtual_prop %>% 
  group_by(n) %>% 
  summarize(sd = sd(prop_red)) %>% 
  rename(`Number of slots in shovel` = n, `Standard deviation of proportions red` = sd) 

comparing_n_table  %>% 
  kable(
    digits = 3,
      caption = "Previously seen comparing standard deviations of proportions red for 3 different shovels.", 
      booktabs = TRUE
) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position"))
```

So as the number of slots in the shovel increased, this standard deviation decreased. These types of standard deviations have another special name: \index{standard error} **standard errors**; they quantify the effect of sampling variation induced on our estimates. In other words, they are quantifying how much we can expect different proportions of a shovel's balls that are red to vary from a random sample to a random sample. 

Unfortunately, many new statistics practitioners get confused by these names. For example, it's common for people new to statistical inference to call the "sampling distribution" the "sample distribution". Another additional source of confusion is the name "standard deviation" and "standard error". Remember that a standard error is merely a *kind* of standard deviation: the standard deviation of any point estimate from a sampling scenario. In other words, all standard errors are standard deviations, but not all standard deviations are a standard error. 

To help reinforce these concepts, let's re-display Figure \@ref(fig:comparing-sampling-distributions) but using our new terminology, notation, and definitions relating to sampling in Figure \@ref(fig:comparing-sampling-distributions-2). 

```{r comparing-sampling-distributions-2, echo=FALSE, fig.cap="Three sampling distributions of the sample proportion $\\widehat{p}$."}
virtual_prop %>% 
  mutate(
    n = str_c("n = ", n),
    n = factor(n, levels = c("n = 25", "n = 50", "n = 100"))
    ) %>% 
  ggplot( aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = expression(paste("Sample proportion ", hat(p))), 
       title = expression(paste("Sampling distributions of the sample proportion ", hat(p), " based on n = 25, 50, 100.")) ) +
  facet_wrap(~n)
```

Furthermore, let's re-display Table \@ref(tab:comparing-n) but using our new terminology, notation, and definitions relating to sampling in Table \@ref(tab:comparing-n-2).

```{r comparing-n-2, eval=TRUE, echo=FALSE}
comparing_n_table <- virtual_prop %>% 
  group_by(n) %>% 
  summarize(sd = sd(prop_red)) %>% 
  mutate(
    n = str_c("n = ", n),
    n = factor(n, levels = c("n = 25", "n = 50", "n = 100"))
    ) %>% 
  rename(`Sample size (n)` = n, `Standard error of $\\widehat{p}$` = sd) 

comparing_n_table  %>% 
  kable(
    digits = 3,
    caption = "Three standard errors of the sample proportion based on n = 25, 50, 100.", 
    booktabs = TRUE#,
#    escape = TRUE
) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position"))
```

Remember the key message of this last table: that as the sample size $n$ goes up, the "typical" error of your point estimate as quantified by the standard error will go down.

<!-- Potential Learning Check: Have readers fill in portions of the table instead. Have them write "Standard error of p-hat" and "sample size", and match the values of the standard errors with the different values of n. We could also use different sizes here instead of just re-displaying Figure 8.11 to ensure they aren't just looking back to match. -->

```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What purpose did the *sampling distributions* serve?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What does the *standard error* of the sample proportion $\widehat{p}$ quantify? 

```{block, type='learncheck', purl=FALSE}
```

### The moral of the story {#moral-of-the-story}

Let's recap this section so far. We've seen that if a sample is generated at random, then the resulting point estimate is a "good guess" of the true unknown population parameter. In our simulations, since we made sure to mix the balls first before extracting a sample with the shovel, the resulting sample proportion $\widehat{p}$ of the shovel's balls that were red was a "good guess" of the population proportion $p$ of the bowl's balls that were red. 

However, what do we mean by our point estimate being a "good guess"? While sometimes we'll obtain a point estimate less than the true value of the unknown population parameter, other times we'll obtain a point estimate greater than the true value of the unknown population parameter, this is because of sampling variation. However, despite this sampling variation, our point estimates will "on average" be correct. In our simulations, sometimes our sample proportion $\widehat{p}$ was less than the true population proportion $p$, other times the sample proportion $\widehat{p}$ was greater than the true population proportion $p$. This was due to the sampling variability induced by the mixing. However despite this sampling variation, our sample proportions $\widehat{p}$ were always centered around the true population proportion. This is also known as having an **accurate** estimate.

What was the value of the population proportion $p$ of the $N$ = 2400 balls in the actual bowl? There were 900 red balls, for a proportion red of 900/2400 = 0.375 = 37.5%! How do we know this? Did the authors do an exhaustive count of all the balls? No! They were listed in the contents of the box that the bowl came in. Hence we made the contents of the virtual `bowl` match the tactile bowl:

```{r}
bowl %>% 
  summarize(sum_red = sum(color == "red"), 
            sum_not_red = sum(color != "red"))
```

Let's re-display our sampling distributions from Figures \@ref(fig:comparing-sampling-distributions) and \@ref(fig:comparing-sampling-distributions-2), but now with a vertical red line marking the true population proportion $p$ of balls that are red = 37.5% in Figure \@ref(fig:comparing-sampling-distributions-3). We see that while there is a certain amount of error in the sample proportions $\widehat{p}$ for all three sampling distributions, on average the $\widehat{p}$ are centered at the true population proportion red $p$.

```{r comparing-sampling-distributions-3, echo=FALSE, fig.cap="Three sampling distributions with population proportion $p$ marked in red."}
p <- bowl %>% 
  summarize(p = mean(color == "red")) %>% 
  pull(p)
virtual_prop %>% 
  mutate(
    n = str_c("n = ", n),
    n = factor(n, levels = c("n = 25", "n = 50", "n = 100"))
    ) %>% 
  ggplot( aes(x = prop_red)) +
  geom_histogram(binwidth = 0.05, boundary = 0.4, color = "white") +
  labs(x = expression(paste("Sample proportion ", hat(p))), 
       title = expression(paste("Sampling distributions of ", hat(p), " based on n = 25, 50, 100.")) ) +
  facet_wrap(~n) +
  geom_vline(xintercept = p, col = "red", size = 1)
```

We also saw in this section that as your sample size $n$ increases, your point estimates will vary less and less and be more and more concentrated around the true population parameter; this is quantified by the decreasing standard error. In other words, the typical error of your point estimates will decrease. In our simulations, as the sample size increases, the spread/variation of our sample proportions $\widehat{p}$ around the true population proportion $p$ decreases. You can observe this behavior as well in Figure \@ref(fig:comparing-sampling-distributions-3).  This is also known as having a more **precise** estimate. 

So random sampling ensures our point estimates are accurate while having a large sample size ensures our point estimates are precise. While accuracy and precision may sound like the same concept, they are actually not. Accuracy relates to how "on target" our estimates are whereas precision relates to how "consistent" our estimates are. Figure \@ref(fig:accuracy-vs-precision) illustrates the difference.

<!-- Albert will check to see if this image exists on shutterstock. If not, 
it will need to be recreated. -->

```{r accuracy-vs-precision, echo=FALSE, fig.cap="Comparing accuracy and precision.", purl=FALSE, out.width="50%"}
knitr::include_graphics("images/accuracy_vs_precision.jpg")
```

As this point, you might be asking yourself: "If you already knew the true proportion of the bowl's balls that are red was 37.5%, then why did do any sampling?" You might also be asking: "Why did we take 1000 repeated samples of size n = 25, 50, and 100? Shouldn't we be taking only *one* sample that's as large as possible?" 

Recall our definition of a simulation from Section \@ref(sampling-simulation): an approximate imitation of the operation of a process or system. We performed these simulations to study:

1. The effect of sampling variation on our estimates.
1. The effect of sample size on sampling variation.

In a real-life scenario, we won't know what the true value of the population parameter is and furthermore we won't take repeated/replicated samples, but rather a single sample that's as large as we can afford. The ideas developed in our repeated sampling activity only help inform pollsters about the quality of their polls. In particular, how accurate and how precise they are as we saw in Figure \@ref(fig:accuracy-vs-precision) with the targets. 

In the next section, we present a case study of a real-life sampling scenario: polling. 

```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** The table below is a version of Table \@ref(tab:comparing-n-2) matching sample sizes $n$ to different *standard errors* of the sample proportion $\widehat{p}$, but with the rows randomly re-ordered and the sample sizes removed. Fill in the table below by matching the correct sample sizes to the correct standard errors.

```{r comparing-n-3, eval=TRUE, echo=FALSE}
set.seed(76)
comparing_n_table <- virtual_prop %>% 
  group_by(n) %>% 
  summarize(sd = sd(prop_red)) %>% 
  mutate(
    n = str_c("n = ")
    ) %>% 
  rename(`Sample size` = n, `Standard error of p-hat` = sd) %>% 
 sample_frac(1)
  
comparing_n_table  %>% 
  kable(
    digits = 3,
      caption = "Three standard errors of the sample proportion based on n = 25, 50, 100. ", 
      booktabs = TRUE
) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position"))
```

For the following learning checks, let the *estimate* be the sample proportion $\widehat{p}$: the proportion of a shovel's balls that were red. It estimated the population proportion $p$: the proportion of the bowl's balls that were red.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What is the difference between an *accurate* estimate and a *precise* estimate? 

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** How do we ensure that an estimate is *accurate*? How do we ensure that an estimate is *precise*?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** In a real-life situation, we would not take 1000 different samples to infer about a population, but rather only one. Then what was the purpose of our exercises about where we took 1000 different samples?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Figure \@ref(fig:accuracy-vs-precision) with the targets shows four combinations of "accurate versus precise" estimates. Draw four corresponding *sampling distributions* of the sample proportion $\widehat{p}$, like the one in the left-most plot in Figure \@ref(fig:comparing-sampling-distributions-3).

```{block, type='learncheck', purl=FALSE}
```





## Case study: Polls {#sampling-case-study}

Let's now switch gears to a more realistic sampling scenario than our bowl activity: a poll. In practice, pollsters do not take 1000 repeated samples as we did in our sampling activities from above, but rather take only a *single sample* that's as large as possible.

On December 4, 2013, National Public Radio in the US reported on a poll of President Obama's approval rating among young Americans aged 18-29 in an article ["Poll: Support For Obama Among Young Americans Eroding"](https://www.npr.org/sections/itsallpolitics/2013/12/04/248793753/poll-support-for-obama-among-young-americans-eroding). The poll was conducted by the Harvard University Institute of Politics. A quote from the article:

> After voting for him in large numbers in 2008 and 2012, young Americans are souring on President Obama.
> 
> According to a new Harvard University Institute of Politics poll, just 41 percent of millennials — adults ages 18-29 — approve of Obama's job performance, his lowest-ever standing among the group and an 11-point drop from April.

Let's tie elements of the real-life poll in this new article with our "tactile" and "virtual" simulations from Sections \@ref(sampling-activity) and \@ref(sampling-simulation) using the terminology, notations, and definitions we learned in Section \@ref(sampling-framework). You see that our sampling activity with the bowl is an idealized version of what pollsters are trying to do in real-life. 

1. **(Study) Population**: Who is the population of $N$ individuals or observations of interest? \index{sampling!population}
    + Simulation: $N$ = 2400 identically-sized red and white balls
    + Obama poll: $N$ = ? young Americans aged 18-29
1. **Population parameter**: What is the population parameter? \index{sampling!population parameter}
    + Simulation: The population proportion $p$ of ALL the balls in the bowl that are red.
    + Obama poll: The population proportion $p$ of ALL young Americans who approve of Obama's job performance.
1. **Census**: What would a census look like? \index{sampling!census}
    + Simulation: Manually going over all $N$ = 2400 balls and exactly computing the population proportion $p$ of the balls that are red, a time-consuming task.
    + Obama poll: Locating all $N$ = ? young Americans and asking them all if they approve of Obama's job performance, an expensive task.
1. **Sampling**: How do you collect the sample of size $n$ individuals or observations? \index{sampling}
    + Simulation: Using a shovel with $n$ slots. 
    + Obama poll: One method is to get a list of phone numbers of all young Americans and pick out $n$ phone numbers. In this poll's case, the sample size of this poll was $n$ = 2089 young Americans.
1. **Point estimate (AKA sample statistic)**: What is your estimate of the unknown population parameter?
    + Simulation: The sample proportion $\widehat{p}$ of the balls in the shovel that were red. 
    + Obama poll: The sample proportion $\widehat{p}$ of young Americans in the sample that approve of Obama's job performance. In this poll's case, $\widehat{p}$ = 0.41 = 41%, the quoted percentage in the second paragraph of the article. \index{point estimate} \index{sample statistic}
1. **Representative sampling**: Is the sampling procedure *representative*? \index{sampling!representative}
    + Simulation: Are the contents of the shovel representative of the contents of the bowl? Because we mixed the bowl before sampling, we can feel confident that they are. 
    + Obama poll: Is the sample of $n$ = 2089 young Americans representative of all young Americans aged 18-29? This depends on whether the sampling was random.
1. **Generalizability**: Are the samples *generalizable* to the greater population? \index{generalizability}
    + Simulation: Is the sample proportion $\widehat{p}$ of the shovel's balls that are red a "good guess" of the population proportion $p$ of the bowl's balls that are red? Given that the sample was representative, the answer is yes.
    + Obama poll: Is the sample proportion $\widehat{p}$ = 0.41 of the sample of young Americans who support Obama a "good guess" of the population proportion $p$ of all young Americans who support Obama? In other words, can we confidently say that roughly 41% of *all* young Americans approve of Obama? Again, this depends on whether the sampling was random.
1. **Bias**: Is the sampling procedure unbiased? In other words, do all observations have an equal chance of being included in the sample? \index{bias}
    + Simulation: Since each ball was equally sized, each ball had an equal chance of being included in a shovel's sample, and hence the sampling was unbiased. 
    + Obama poll: Did all young Americans have an equal chance at being represented in this poll? Again, this depends on whether the sampling was random.
1. **Random sampling**: Was the sampling random? \index{sampling!random}
    + Simulation: As long as you mixed the bowl sufficiently before sampling, your samples would be random.
    + Obama poll: Was the sample conducted at random? We can't answer this question without knowing about the *sampling methodology* used by the Harvard University Institute of Politics. We'll discuss this more at the end of this section.

Once again, let's revisit the sampling paradigm from Section \@ref(terminology-and-notation):

> * If the sampling of a sample of size $n$ is done at **random**, then
> * the sample is **unbiased** and **representative** of the population of size $N$, thus
> * any result based on the sample can **generalize** to the population, thus
> * the point estimate is a **"good guess"** of the unknown population parameter, thus
> * instead of performing a census, we can **infer** about the population using sampling.

In our simulations using the shovel with 50 slots:

> * If we extract a sample of $n$ = 50 balls at **random**, in other words, we mix the equally-sized balls before using the shovel, then
> * the contents of the shovel are an **unbiased representation** of the contents of the bowl's 2400 balls, thus
> * any result based on the sample of balls can **generalize** to the bowl, thus
> * the sample proportion $\widehat{p}$ of the $n$ = 50 balls in the shovel that are red is a **"good guess"** of the population proportion $p$ of the $N$ = 2400 balls that are red, thus
> * instead of manually going over all the balls in the bowl, we can **infer** about the bowl using the shovel. 

In the in-real-life Obama poll:

> * If we had a way of contacting a **randomly** chosen sample of 2089 young Americans and poll their approval of Obama, then
> * these 2089 young Americans would be an **unbiased** and **representative** sample of *all* young Americans, thus 
> * any results based on this sample of 2089 young Americans can **generalize** to the entire population of all young Americans, thus
> * the reported sample approval rating of 41% of these 2089 young Americans is a **good guess** of the true approval rating among all young Americans, thus
> * instead of performing a highly costly census of all young Americans, we can **infer** about all young Americans using polling.

So as you can see, it was critical for the Harvard University Institute of Politics sample to be truly random in order to infer about *all* young Americans' opinions about Obama. Was their sample truly random? It's hard to answer such questions without knowing about the *sampling methodology* used. For example, if this poll was conducted using only mobile phone numbers, people without mobile phones would be left out and therefore not represented in the sample. What about if the Harvard University Institute of Politics conducted this poll on an internet news site? Then people who don't read this internet news site would be left out. Ensuring that our samples were random was easy to do in our sampling bowl exercises, however in a real-life situation like the Obama poll, this is much harder to do.


```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

Comment on the representativeness of the following *sampling methodologies*:

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** The Royal Air Force wants to study how resistant all their airplanes are to bullets. They study the bullet holes on all the airplanes on the tarmac after an air battle against the Luftwaffe (German Air Force).

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** Imagine it is 1993, a time when almost all households had landlines. You want to know the average number of people in each household in your city. You randomly pick out 500 phone numbers from the phone book and conduct a phone survey.

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** You want to know the prevalence of illegal downloading of TV shows among students at a local college.  You get the emails of 100 randomly chosen students and ask them "How many times did you download a pirated TV show last week?"

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** A local college administrator wants to know the average income of all graduates in the last 10 years. So they get the records of 5 randomly chosen graduates, contact them, and obtain their answers. 

```{block, type='learncheck', purl=FALSE}
```



## Conclusion {#sampling-conclusion}

<!--
### Random sampling vs random assignment {#sampling-conclusion-sampling-vs-assignment}

As big point of confusion is the difference between random sampling and random assignment.
-->


### Sampling scenarios {#sampling-conclusion-table}

In this chapter, we performed both tactile and virtual simulations of sampling to infer about an unknown proportion. We also presented a case study of sampling in real life situation: polls. In both cases, we used the sample proportion $\widehat{p}$ to estimate the population proportion $p$. However, we are not just limited to scenarios related to statistical inference for proportions. In other words, we can consider other population parameter and point estimate scenarios than just the population proportion $p$ and sample proportion $\widehat{p}$ scenarios we studied in this chapter. We present 5 more such scenarios in Table \@ref(tab:table-ch8). 

```{r table-ch8, echo=FALSE, message=FALSE}
# The following Google Doc is published to CSV and loaded below using read_csv() below:
# https://docs.google.com/spreadsheets/d/1QkOpnBGqOXGyJjwqx1T2O5G5D72wWGfWlPyufOgtkk4/edit#gid=0

if(!file.exists("rds/sampling_scenarios.rds")){
  sampling_scenarios <- "https://docs.google.com/spreadsheets/d/e/2PACX-1vRd6bBgNwM3z-AJ7o4gZOiPAdPfbTp_V15HVHRmOH5Fc9w62yaG-fEKtjNUD2wOSa5IJkrDMaEBjRnA/pub?gid=0&single=true&output=csv" %>% 
    read_csv(na = "")
    write_rds(sampling_scenarios, "rds/sampling_scenarios.rds")
} else {
  sampling_scenarios <- read_rds("rds/sampling_scenarios.rds")
}

sampling_scenarios %>% 
  kable(
    caption = "\\label{tab:summarytable-ch8}Scenarios of sampling for inference", 
    booktabs = TRUE,
    escape = FALSE
  ) %>% 
  kable_styling(font_size = ifelse(knitr:::is_latex_output(), 10, 16),
                latex_options = c("HOLD_position")) %>%
  column_spec(1, width = "0.5in") %>% 
  column_spec(2, width = "0.7in") %>%
  column_spec(3, width = "1in") %>%
  column_spec(4, width = "1.1in") %>% 
  column_spec(5, width = "1in")
```

We'll cover all the remaining scenarios as follows, using the terminology, notation, and definitions related to sampling you saw in Section \@ref(sampling-framework):

* In Chapter \@ref(confidence-intervals), we'll cover examples of statistical inference for
    + Scenario 2: The mean age $\mu$ of all pennies in circulation in the US.
    + Scenario 3: The difference $p_1 - p_2$ in the proportion of people who yawn when seeing someone else yawn and the proportion of people who yawn without seeing someone else yawn. This is an example of *two-sample* inference.
* In Chapter \@ref(hypothesis-testing), we'll cover an example of statistical inference for
    + Scenario 4: The difference $\mu_1 - \mu_2$ in average IMDB ratings for action and romance movies. This is another example of *two-sample* inference.
* In Chapter \@ref(inference-for-regression), we'll cover an example of statistical inference for the relationship between teaching score and various instructor demographic variables you saw in Chapter \@ref(regression) on basic regression and Chapter \@ref(multiple-regression) on multiple regression. Specifically
    + Scenario 5: The intercept $\beta_0$ of some population regression line.
    + Scenario 6: The slope $\beta_1$ of some population regression line.


### Central Limit Theorem {#sampling-conclusion-central-limit-theorem}

What you visualized in Figure \@ref(fig:comparing-sampling-distributions) and summarized in Table \@ref(tab:comparing-n)) was a demonstration of a very famous theorem, or mathematically proven truth, called the \index{Central Limit Theorem} *Central Limit Theorem*. It loosely states that when sample means are based on larger and larger sample sizes, the sampling distribution of these sample means both

1. Becomes more and more normally shaped, or in other words 
2. Becomes more and more narrow, or in other words the standard error decreases.

In other words, their sampling distribution increasingly follows a *normal distribution* \index{distribution!normal} and the spread/variation of these sampling distributions, as quantified by their standard errors,  gets smaller. 

Shuyi Chiou, Casey Dunn, and Pathikrit Bhattacharyya created a 3 minute and 38 second video at <https://youtu.be/jvoxEYmQHNM> explaining this crucial statistical theorem using the average weight of wild bunny rabbits and the average wing span of dragons as examples. Figure \@ref(fig:CLT-video-preview) shows a preview of this video.

```{r CLT-video-preview, echo=FALSE, fig.cap="Preview of Central Limit Theorem video.", purl=FALSE, out.width = "80%"}
knitr::include_graphics("images/CLT_video_preview.png")
```


### Normal distributions {#normal-distribution}

In the previous Subsection on the Central Limit Theorem, we introduced the notion of a normal distribution. Such distributions are defined by two values: 1) the mean $\mu$, which locates the center of the distribution, and 2) the standard deviation $\sigma$, which determines the spread of the distribution. In Figure \@ref(fig:normal-curves), we plot three normal distributions where:

1. The solid line normal curve has mean $\mu$ = 5 and standard deviation $\sigma$ = 2.
1. The dashed line normal curve has mean $\mu$ = 5 and standard deviation $\sigma$ = 5.
1. The dotted line normal curve has mean $\mu$ = 14 and standard deviation $\sigma$ = 2.

```{r normal-curves, echo=FALSE, fig.cap="Three normal distributions", purl=FALSE, out.width = "80%"}
ggplot(data = data.frame(x = c(-10, 25)), aes(x)) +
  stat_function(fun = dnorm, args = list(mean = 5, sd = 2), n = 500) +
  stat_function(fun = dnorm, args = list(mean = 5, sd = 5), n = 500, linetype = "dashed") +
  stat_function(fun = dnorm, args = list(mean = 14, sd = 2), n = 500, linetype = "dotted") +
  coord_cartesian(xlim = c(-10, 25)) +
  labs(y = "") +
  theme(axis.title.y = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank())
```

Notice how the solid and dashed line normal curves have the same center owing to their common mean $\mu$ = 5. However the dashed line normal curve is wider owing to its larger standard deviation of $\sigma$ = 5. On the other hand, the solid and dotted line normal curves have the same spread owing to their common standard deviation $\sigma$ = 2. However they are centered at different locations. 

When the mean $\mu$ = 0 and the standard deviation $\sigma$ = 1, the normal distribution has a special name: the standard normal distribution or the $z$-curve\index{distribution!standard normal}.

Furthermore, if a variable follows a normal curve, there are three rules of thumb we can use:

1. 68% of values will lie within $\pm$ 1 standard deviations of the mean.
1. 95% of values will lie within $\pm$ 1.96 $\approx$ 2 standard deviations of the mean.
1. 99.7% of values will lie within $\pm$ 3 standard deviations of the mean.

Let's illustrate this on a standard normal curve in Figure \@ref(fig:normal-rule-of-thumb). The dashed lines are at -3, -1.96, -1, 0, 1, 1.96, and 3 and cut up the x-axis into 8 segments. The areas under the normal curve for each of the 8 segments are marked and add up to 100%.

1. The middle two segments represent the interval -1 to 1. The shaded area above this interval represents 34% + 34% = 68% of the area under the curve. In other words 68% of values. 
1. The middle four segments represent the interval -1.96 to 1.96. The shaded area above this interval represents 13.5% + 34% + 34% + 13.5%= 95% of the area under the curve. In other words 95% of values. 
1. The middle six segments represent the interval -3 to 3. The shaded area above this interval represents 2.35% + 13.5% + 34% + 34% + 13.5% + 2.35% = 99.7% of the area under the curve. In other words 99.7% of values. 

```{r normal-rule-of-thumb, echo=FALSE, fig.cap="Rules of thumb about areas under normal curves", purl=FALSE, out.width = "80%"}
shade_3_sd <- function(x) {
    y <- dnorm(x, mean = 0, sd = 1)
    y[x <= -3 | x >= 3] <- NA
    return(y)
}
shade_2_sd <- function(x) {
    y <- dnorm(x, mean = 0, sd = 1)
    y[x <= -1.96 | x >= 1.96] <- NA
    return(y)
}
shade_1_sd <- function(x) {
    y <- dnorm(x, mean = 0, sd = 1)
    y[x <= -1 | x >= 1] <- NA
    return(y)
}

labels <- tibble(
  x = c(-3.5, -2.5, -1.5, -0.5, 0.5, 1.5, 2.5, 3.5),
  label = c("0.15%", "2.35%", "13.5%", "34%", "34%", "13.5%", "2.35%", "0.15%")
) %>% 
  mutate(y = rep(0.3, times = n()))

ggplot(data = tibble(x = c(-4, 4)), aes(x)) +
  geom_text(data = labels, aes(y=y, label = label)) + 
  # Trace normal curve
  stat_function(fun = dnorm, args = list(mean = 0, sd = 1), n = 1000) + 
  # Shade and delineate +/- 3 SD
  stat_function(fun = shade_3_sd, geom = "area", fill = "black", alpha = 0.25, n = 1000) +
  # annotate(geom = "segment", x = c(3, -3), xend = c(3, -3), y = 0, yend = dnorm(3, mean = 0, sd = 1)) +
  # Shade and delineate +/- 2 SD
  stat_function(fun = shade_2_sd, geom = "area", fill = "black", alpha = 0.25, n = 1000) +
  # annotate(geom = "segment", x = c(1.96, -1.96), xend = c(1.96, -1.96), y = 0, yend = dnorm(1.96, mean = 0, sd = 1)) +
  # Shade and delineate +/- 1 SD
  stat_function(fun = shade_1_sd, geom = "area", fill = "black", alpha = 0.25, n = 1000) +
  # annotate(geom = "segment", x = c(1, -1), xend = c(1, -1), y = 0, yend = dnorm(1, mean = 0, sd = 1)) + 
  geom_vline(xintercept = c(-3, -1.96, -1, 0, 1, 1.96, 3), linetype = "dashed", alpha = 0.5) +
  # Axes
  scale_x_continuous(breaks = seq(from = -3, to = 3, by = 1)) +
  labs(x = "z", y = "") +
  theme(axis.title.y = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank())
```

```{block, type='learncheck', purl=FALSE}
**_Learning check_**
```

Say you have a normal distribution with mean $\mu$ = 6 and standard deviation $\sigma$ = 3. 

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What proportion of the area under the normal curve is less than 3? Greater than 12? Between 0 and 12?

**`r paste0("(LC", chap, ".", (lc <- lc + 1), ")")`** What is the 2.5th percentile of the area under the normal curve? The 95th percentile? The 100th percentile.


```{block, type='learncheck', purl=FALSE}
```

<!--

<https://gallery.shinyapps.io/dist_calc/>


Any guesses to how much area is under the black curve and to the right 1.96? The correct answer is very close to 2.5%. Since the normal distribution is symmetric there is also 2.5% of the area to the left of -1.96. Therefore, if we wanted to encapsulate the middle 95% of the values on the standard normal distribution we'd be pretty close to between -1.96 and 1.96. That's the reason why we choose 1.96 as our multiplier in the formula above.

What if we wanted to get the multiplier for say a 90% theory-based confidence interval? R has a built-in function to help us with that:

```{r}
qnorm(p = 0.95)
```

Here `q` stands for "quantile" and `norm` stands for normal. So the 95^th^ percentile of the standard normal distribution falls at around 1.65. Let's check to see where the 2.5^th^ percentile falls:

```{r}
qnorm(p = 0.025)
```

This is what we expected above. Close to -1.96 corresponds to the spot that is 2.5% of the way into the values of the standard normal distribution. We'll elaborate more on these theory-based methods in Chapter \@ref(hypothesis-testing), but this should give you a good start!

-->


### Additional resources

An R script file of all R code used in this chapter is available [here](scripts/08-sampling.R).


### What's to come?

Recall in our Obama poll case study in Section \@ref(sampling-case-study) that based on this particular sample, the Harvard University Institute of Politics' best guess of Obama's approval rating among all young Americans was 41%. However, this isn't the end of the story. If you read further in the article, it states:

> The online survey of 2,089 adults was conducted from Oct. 30 to Nov. 11, just weeks after the federal government shutdown ended and the problems surrounding the implementation of the Affordable Care Act began to take center stage. The poll's margin of error was plus or minus 2.1 percentage points.

Note the term *margin of error*, which here is plus or minus 2.1 percentage points. What this is saying is that most polls won't get it perfectly right; there will always be a certain amount of error caused by *sampling variation*. The margin of error of plus or minus 2.1 percentage points is saying that a typical range of errors for polls of this type is about $\pm$ 2.1%, in words from about 2.1% too small to about 2.1% too big for an interval of [41% - 2.1%, 41% + 2.1%] = [37.9%, 43.1%].  Remember that this notation corresponds to 37.9% and 43.1% is included as well as all numbers between the two of them. We'll see in the next chapter that such intervals are known as *confidence intervals*.