Manuscript, code, and data for "Meaningful Inferences: The Importance of Explicit Statistical Arguments for Substantive Significance." [Paper]

Research in political science is gradually moving away from an exclusive focus on statistical significance testing and toward an emphasis on effect magnitude. We argue that the current practice of "magnitude-and-significance," in which researchers interpret only the magnitude of a statistically significant estimate is only a small improvement over the much maligned "sign-and-significance" approach, in which researcher focus only on the statistical significance of an estimate. We argue that instead of interpreting the magnitude of a statistically significant effect, researchers should explicitly account for uncertainty when making judgments about substantive importance of statistical results. This requires the researcher to precisely define the effects that are and are not substantively meaningful and show that those effects are unlikely to generate the observed data. Using the effect of U.N. troops on civilian casualties during civil war and the effect of yard signs on candidate support, we show that our approach might strengthen or weaken claims of substantive significance.

If you have any comments or suggestion, please open an issue or just e-mail me.