A t-test is used to test whether two sets of data are statistically significantly different from each other. The result of a t-test is the p-value. The p-value should be interpreted as a % (e.g. 0.05 is 5%), and it gives us the probability that the means of the datasets are NOT different. If a p-value is small, then the difference is statistically significant, so you want the p-value to be as small as possible.
For example, a p-value of 0.8 means that there it is very likely that the difference you see in your two groups of observations is just random noise and therefore there is no statistically significant difference. However, a p-value of 0.01 would tell us that there is only a 1% chance the difference is random, and therefore a 99% chance that the means of the two groups are indeed different.
We can understand this further using a simple example which compares made up observations of two characters: Justin Beaver and Harry Otter.
In the first instance, you will see that we have 20 measurements of Justin Beaver's body temperature and 20 measurements of Harry Otter's body temperature. Click on the button that says "Show individual observations" to see how the data points are spread around the mean The initial p-value is not significant.
Now use the sliders to answer these questions:
-
How does moving the three slides change the p-value? Remember that smaller the better (smaller p-value means you have a statistically significant result).
-
How does the sample size change the p-value? i.e. is larger sample size better for yielding a statistically significant result?
-
How does the variability of the observations change the widths of the blue boxes? What do you think the blue boxes on the top picture represent? What about the whiskers (straight lines coming out of the boxes?)
-
What does effect size represent?
- Is there a difference between statistical significance and clinical significance?
-
How many observations do you need if the standard deviations of the samples are 0.4 and the effect size is 0.2 to obtain a statistically significant result (e.g. p-value less than 0.05).
-
If you have 30 observations and a standard deviation of 0.5, what's the smallest difference you can detect (again for a p-value of 0.05 or less)?
-
Summarize how the 3 variables (sample size, standard deviation, and effect size) influence the p-value: what makes the p-value smaller (so better), what makes it larger?
-
As a researcher, you want to see small p-values, what can you do to ensure your study results in small p-values?