A Shiny Application for teaching about random assignment
This is how I think will use it in my research methods class. It will be a short exercise (30 minutes).
First I will have normally already done some of the other exercises that as "did the intervention work?" and the class will have come up with a list of threats to internal validity (selection bias, history, placebo effect, testing effect, un measured third variables etc).
Then I will talk about experimental design as one way of eliminating these.
But then I say that the claim about randomization solving these problems is a pretty audacious claim and that they should not really just believe it on my say so.
This is when we will use the application.
First we will all use the same sample size and notice that thre samples are not the same although it will turn out that we'll have a lot of duplicates on the first 1-4 items (there are only 16 of size 4, 8 possible combinations of size 3, 4 combinations of size 2, 2 combinations of size 1 (notice we are squaring).
But will any 2 people have the same exact 10? Yes it is possible, but it's unlikely. there are 2^10 possible combinations (1024). But how many people will have exactly 5 treatment and 5 control? How about 4/5 and 5/4? This overall is a good review of some sampling concepts.
Then I will have all of the students write down selected results for that size (e.g. percent female, mean age, mean education, percent Black, percent picses) and either put them on the board or in a shared google spreadsheet displayed on the screen.
Then we'll talk about the fact that yes overall the randomization did very well. The groups are generally not identical but are pretty close. There may be a few not so great ones however so it's important to keep that in mind. Even though randomization is great, your experiment could be the one where it gave you an assignment that is in one of the tails.
Then we'll take a look at the impact of the sample size on this by using the different sized pools.