Omnibus Test for ANOVA

[EJWagenmakers]

For the classical ANOVA's, can we have an omnibus test please? For why we need it see

Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L. J., & Wagenmakers, E.-J. (2016). Hidden multiplicity in multiway ANOVA: Prevalence, consequences, and remedies. Psychonomic Bulletin & Review, 23, 640-647. (link on my webpage)

E.J.

[tomtomme]

Interesting read that paper.
I would love for multifactorial ANOVA to include

• an omnibus F-Test and
• an optional correction for multiplicity as discussed in Add corrections for multiplicity to multiway ANOVA #11 (http://arxiv.org/abs/1412.3416 & Added anova.afex_aov() and nice() option to adjust F-tests for multiple comparisons singmann/afex#3

While teaching, it is so strange for the students to always see that omnibus ANOVA within (multiple) linear regression, but with ANOVA, they only see it, when it is one-way. That makes no sense at all.

Also, in an exploratory setting, testing main and interaction effects, could be considered post hoc. We could then provide a checkbox to apply the same corrections of "normal" post hoc tests for multiplicity of the family wise alpha error rate to the main and interaction effects.

[Joao-O-Santos]

@EJWagenmakers such a fascinating topic (rant below as post-scriptum).

I'm everything but a fan of SPSS, and I might be misrembering but I think it might be doing a cool thing with its general linear model interface.
With ANOVAs, t-tests, etc... all being specific cases of a regression you can use a single menu to compute them all.
SPSS then outputs the omnibus F test, a traditional ANOVA table (main effects and interactions), and I believe a table with the model parameters (intercepts, slopes, their t-tests, p-values, 95%CIs, etc...).
Maybe the way to address the strangeness @tomtomme alluded to is simply to use a unified interface for all linear models.
This doesn't have to mean getting rid of the standalone modules for each test/model (as they're familiar to users and useful for some teaching approaches), but maybe the functionality could be added in a new more generalized module for generalized linear models.
What do you think?

P.S:
I didn't know about your paper (thanks for sharing), but I've been thinking about ANOVA tables as model comparisons for a while now.
I reckon the model comparison approach (Judd et al., 2017) to teaching stats makes some of your points easy to understand and integrate in class.
It may also be worth noting that ANOVA historically comes from a more experimental research tradition (if I'm not mistaken).
Within that tradition the focus is not so much on exploratory analysis, but on severely testing hypothesis.
In a sense testing if each factor has an effect when we account for the main effect of other factors and their interactions (type III sums of squares) poses a more severe test to the hypothesis.
Still, looking at the omnibus F can still be interesting, and I reckon discussing the different model comparisons in class can be interesting, particularly when using a model comparison approach.

[tomtomme]

@Joao-O-Santos
We already have the linear regression module which allows categorial vars (interpreted as nominal like in anova module) and then also gives the "ANOVA" omnibus F-Test. In that way it is very similar to SPSSs General Linear Modeling module. And we have also a generalized version of that one (for categorial DVs). And both are even implemented a second time in the visual modeling module with emphasis on other parts of the analysis. So no general shortage of omnibus tests compared to SPSS I think.

This discussion is only about adding that omnibus test ALSO to the ANOVA module, to make tables / outputs more consistent across analyses. If I remember correctly SPSS avoided this problem, by forcing the user to do a multifactorial ANOVA in a General Linear Modeling module. The ANOVA was then limited to one factor. Do I remember correctly?
What I do not know is, if SPSS had also the option to do family wise error correction across the main and interaction effects before doing pairwise post hocs. And that would be cool to implement as an option in all modules, also regression modules. There I always have to resort to Excel to correct my p values atm. Even SPSS does not allow that - at least in the normal regression module.