Skip to content

Latest commit

 

History

History
9 lines (5 loc) · 1.25 KB

statinf-optimalSampleSize.md

File metadata and controls

9 lines (5 loc) · 1.25 KB

Power Calculations: what is an "optimal" sample size?

Generally speaking, "optimal" sample size is the minimum sample size needed to distinguish an effect of a specific size between test and control groups at a specific alpha level. That is, if a researcher designs an experiment with a test and control group and can distinguish a substantively meaningful effect with a sample size of 60, why should s/he spend money on a sample size of 100?

Remember, effect size for a difference of means test is calculated as (mu_a - mu_0) / (sigma), where mu_a represents the mean from the test group, and mu_0 represents the control group, assuming a pooled variance.

What does "substantively meaningful" mean? Well, if one is testing the difference in a test advertisement and a control advertisement on the probability of a consumer purchasing a product, "substantively meaningful" is when the effect is large enough that it makes sense to deploy the test advertisement based on the amount of additional revenue that will be generated by using the test ad versus the control ad.

With a large enough N, any comparison of means or proportions can be statistically significant, so it's important to ensure that a statistically significant result is also substantively meaningful.