Added anova.afex_aov() and nice() option to adjust F-tests for multiple comparisons

[crsh]

Hi Henrik,

I have added an option p.adjust.method to anova.afex_aov() and nice() that allows the user to adjust the results of ANOVA F-tests for multiple comparisons as per Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., … Wagenmakers, E.-J. (2015). Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. Psychonomic Bulletin & Review, 1–8. http://doi.org/10.3758/s13423-015-0913-5 (Open access)

The option can be passed in the anova_table-list when calling aov_car and is added as an attribute named p.adjust.method to anova_table-objects produced by anova.afex_aov() (and thus also to afex_aov$anova_table). All model objects such as anova, Anova.mlm, aov, lm and output when return = "univariate" are not affected.

anova.afex_aov() and nice() default to the p.adjust.method specified in the call to aov_car() (no adjustment if unspecified) but it can be changed by passing a new value.

When calling summary.afex_aov(), displayed p-values are (un-)adjusted as specified in the call to aov_car() if it is a purely between subject ANOVA (of class anova). This is not easily achieved for within or mixed designs (Should we correct for the intercept F-test? Which sphericity correction should we use?). Thus, the method returns the untouched univariate object (as it previously did) but if the user asked for an adjustment when calling aov_car() a message notifies the user that the displayed results are not adjusted:

Note, results are NOT adjusted for multiple comparisons as requested (p.adjust.method = 'bonferroni') because the desired method of sphericity correction is unknown. For adjusted p-values print the object (to see object$anova_table), or call one of anova.afex_aov() or nice().

I have also updated the documentation and added a unit test to test-aov_car-basic.car-basic.R and tested the new functionality more extensively by hand. I think it should give you no trouble. Let me know what you think.

Best regards,
Frederik

[singmann]

Hi Frederik,
This looks really good, thanks a lot. Before merging it, I would nevertheless like to be more sure whether or not it is a good idea to treat within-subjects design differently from between-designs. I can not immediately see why this should be the case.

1. The issue of sphericity correction seems to be orthogonal to the question at hand. They are simply implemented to provide reasonable tests in the first place. But the corrected tests should still be prone to the same multiplicity problem.
2. The question of the intercept seems also rather non-problematic to me. Very often the test is trivial (and hence not reported per default). If it is interesting and requested the same reasoning should apply and it those tests also be corrected.

Perhaps @EJWagenmakers can chip in as well, but as far as I see it, I would suggest to apply the correction indiscriminately to all reported p-values independently of the actual design (which would also be somewhat more in line with the afex spirit).

Cheers,
Henrik

[EJWagenmakers]

Yes, I think the issue holds regardless of the design
E.J.

---

Eric-Jan Wagenmakers
Department of Psychological Methods, room 2.09
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands

Web: ejwagenmakers.com
Book: bayesmodels.com
Stats: jasp-stats.org
Email: EJ.Wagenmakers@gmail.com
Phone: (+31) 20 525 6420

“Man follows only phantoms.”
Pierre-Simon Laplace, last words

---

On Wed, Sep 30, 2015 at 9:44 AM, singmann notifications@github.com wrote:

Hi Frederik,
This looks really good, thanks a lot. Before merging it, I would
nevertheless like to be more sure whether or not it is a good idea to treat
within-subjects design differently from between-designs. I can not
immediately see why this should be the case.

The issue of sphericity correction seems to be orthogonal to the
question at hand. They are simply implemented to provide reasonable tests
in the first place. But the corrected tests should still be prone to the
same multiplicity problem.
2.

The question of the intercept seems also rather non-problematic to me.
Very often the test is trivial (and hence not reported per default). If it
is interesting and requested the same reasoning should apply and it those
tests also be corrected.

Perhaps @EJWagenmakers https://github.com/EJWagenmakers can chip in as
well, but as far as I see it, I would suggest to apply the correction
indiscriminately to all reported p-values independently of the actual
design (which would also be somewhat more in line with the afex spirit).

Cheers,
Henrik

—
Reply to this email directly or view it on GitHub
#3 (comment).

[crsh]

You are, of course, correct that the adjustment does not depend on the presence of within-subject factors and I'm not arguing for this. This is a technical issue with the way summary.afex_aov() works. I'll try to clarify:

In the current implementation, regardless of the research design

• calling aov_car() with a p.adjust.method in the anova_table argument will adjust all F-tests of interest, i.e., all F-tests included in the afex_aov$anova_table-object. If, as is the default, intercept = FALSE the tests will not be adjusted for the additional intercept test.
• passing an afex_aov-object to anova.afex_aov() or to nice() will adjust F-tests according to the specification when calling aov_car() or according to the p.adjust.method-argument.

In both cases, possible sphericity corrections are applied before p-values are adjusted according to the correction argument.

The technical difficulty arises with the current implementation of summary.afex_aov(). I take it, it's a design decision that the function produces the afex_aov$anova_table for between subject designs (anova) but the univariate output including sphericity tests as well as GG and HF corrections for within- and mixed-designs (Anova.mlm). This is indeed helpful when you want to decide if a correction is necessary (otherwise you'd have to call aov_car() again and set return = "univariate").

But this is precisely where the difficulty with implementing adjusted p-values for this output arises. As is detailed nicely in the paper, for some adjustment methods the resulting p-value is contingent on the other p-values in the family. It is, thus, not possible to provide adjusted p-values if we don't know what p-values are included in the set (uncorrected, GG, or HF). Also, with the univariate-output it is not clear if the intercept is of interest or not.

As I see it, this leaves two of options:

1. Acknowledges that adjustments are not possible and notify the user with instructions on how to obtain the correct values.
2. Change the design of summary.afex_aov().

I chose the first option because I did not want to add more elaborate changes to the design of afex in the same pull request, not because it's the best option. I've put a little thought into how the design of the summary-method could be changed to circumvent this issue but I'd like to tackle this in a separate pull request and discuss possible changes in an issue first.

[singmann]

Hi Frederik,
Thanks for the clarification and I apologize for misunderstanding what you did the first time. This makes a lot of sense. I will accept the changes as is and make some small changes later on (i.e., break the message at 80 characters).
Thanks again for the nice addition.
Cheers,
Henrik