Incorrect CI bars for repeated measures ANOVA (standard and Bayesian)

[Angalanse]

• JASP version: 0.14.0.0 (also reproduced in 0.14.1)

• OS name and version: Win 10 64 bit
• Analysis: I'm plotting skin conductance levels over the course of time (nine repeated variables) in women and men (sex is the between subjects factor). This produces outcomes that are reasonable (and in the standard ANOVA case identical with SPSS output). However, the confidence intervals / credible intervals seem much too small, and are not at all aligned with SPSS output. Could it be an issue with realtively small numbers with many decimals? The plots seem correct for other RM ANOVAs.
• Bug description: When producing descriptive plots for repeated measures ANOVAs, the CI bars seem much smaller than they should be.
• Expected behaviour: See attached SPSS plot [the first two plots are standard RM ANOVA and Bayesian RM ANOVA, respectively]
• Also: fantastic work with JASP! I'm so grateful for having this tool.

[JohnnyDoorn]

Hi @Angalanse,

Thanks for your report. What options did you use for the plots? For the frequentist RM ANOVA, you can either plot by CI or standard error, which can lead to different sized intervals. Also, have you averaged across the unused RM factors, by ticking the corresponding box? This can also have a big impact on the result. It's a bit puzzling though why also the Bayesian RM ANOVA is giving different results.
Would it be possible for you to share the jasp file or data set, so that I can look into this issue? You can also send it by email to j dot b dot vandoorn at uva dot nl

Kind regards,
Johnny

[JohnnyDoorn]

Hi @Angalanse ,

Thanks for the data set. The discrepancy of the results comes from the fact that JASP applies a correction to the intervals (see also the help file for the RM ANOVA):

In order to get accurate confidence intervals and standard errors, the data are normalized by subtracting the appropriate participantʹs mean performance from each observation, and then adding the grand mean score to every observation. The variances of the resulting normalized values in each condition, and thus the size of the bars, no longer depend on the participant effects and are therefore a more accurate representation of the experimental manipulation. See Morey (2008) for a thorough discussion of this procedure.
The paper in question can be found here.

below is R code that reproduces both versions of the results (SPSS first for the uncorrected intervals, and then the JASP results with the corrected results).

Kind regards,
Johnny

dat <- read.csv2("~/Downloads/ForVanDoorn.csv", dec = ",")

# convert wide data to long data
longDat <- reshape2::melt(dat, measure.vars = 3:11)



# compute anova and marginal means with afex and emmeans UNCORRECTED
res <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")
mMeans <- emmeans::emmeans(res, specs = c("variable", "Sex"))

# compute pp means to apply Morey correction
idMeans <- tapply(longDat$value, longDat$FPID, mean)
longDat$value <- longDat$value - idMeans[longDat$FPID]

# compute anova and marginal means with afex and emmeans CORRECTED
resCor <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable")
mMeansCor <- emmeans::emmeans(resCor, specs = c("variable", "Sex"))

# plot marginal means
plot(mMeans)
plot(mMeansCor)

[Angalanse]

Thanks Johnny! That’s interesting! L Från: JohnnyDoorn <notifications@github.com> Skickat: den 2 mars 2021 10:19 Till: jasp-stats/jasp-issues <jasp-issues@noreply.github.com> Kopia: Linus Andersson <linus.andersson@umu.se>; Mention <mention@noreply.github.com> Ämne: Re: [jasp-stats/jasp-issues] Incorrect CI bars for repeated measures ANOVA (standard and Bayesian) (#1187) Hi @Angalanse<https://github.com/Angalanse> , Thanks for the data set. The discrepancy of the results comes from the fact that JASP applies a correction to the intervals (see also the help file for the RM ANOVA): In order to get accurate confidence intervals and standard errors, the data are normalized by subtracting the appropriate participantʹs mean performance from each observation, and then adding the grand mean score to every observation. The variances of the resulting normalized values in each condition, and thus the size of the bars, no longer depend on the participant effects and are therefore a more accurate representation of the experimental manipulation. See Morey (2008) for a thorough discussion of this procedure. The paper in question can be found here<http://pcl.missouri.edu/sites/default/files/morey.2008.pdf>. below is R code that reproduces both versions of the results (SPSS first for the uncorrected intervals, and then the JASP results with the corrected results). Kind regards, Johnny dat <- read.csv2("~/Downloads/ForVanDoorn.csv", dec = ",") # convert wide data to long data longDat <- reshape2::melt(dat, measure.vars = 3:11) # compute anova and marginal means with afex and emmeans UNCORRECTED res <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable") mMeans <- emmeans::emmeans(res, specs = c("variable", "Sex")) # compute pp means to apply Morey correction idMeans <- tapply(longDat$value, longDat$FPID, mean) longDat$value <- longDat$value - idMeans[longDat$FPID] # compute anova and marginal means with afex and emmeans CORRECTED resCor <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable") mMeansCor <- emmeans::emmeans(resCor, specs = c("variable", "Sex")) # plot marginal means plot(mMeans) plot(mMeansCor) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1187 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATBIGIVJ5WETBOJGQM7YZITTBSUP5ANCNFSM4YMXACFA>.

[Angalanse]

Just a follow-up Johnny, One of the arguments in the paper you linked to was that the new confidence intervals are more aligned with the results of the (significant) t-test in the paper. The pedagogical problem I see with the RM ANOVA example I provided is that readers who are used to glancing at the figure to get an overview of the results would probably expect an immense difference between the two factors (men / women). However, there is no supported / significant difference. L Från: JohnnyDoorn <notifications@github.com> Skickat: den 2 mars 2021 10:19 Till: jasp-stats/jasp-issues <jasp-issues@noreply.github.com> Kopia: Linus Andersson <linus.andersson@umu.se>; Mention <mention@noreply.github.com> Ämne: Re: [jasp-stats/jasp-issues] Incorrect CI bars for repeated measures ANOVA (standard and Bayesian) (#1187) Hi @Angalanse<https://github.com/Angalanse> , Thanks for the data set. The discrepancy of the results comes from the fact that JASP applies a correction to the intervals (see also the help file for the RM ANOVA): In order to get accurate confidence intervals and standard errors, the data are normalized by subtracting the appropriate participantʹs mean performance from each observation, and then adding the grand mean score to every observation. The variances of the resulting normalized values in each condition, and thus the size of the bars, no longer depend on the participant effects and are therefore a more accurate representation of the experimental manipulation. See Morey (2008) for a thorough discussion of this procedure. The paper in question can be found here<http://pcl.missouri.edu/sites/default/files/morey.2008.pdf>. below is R code that reproduces both versions of the results (SPSS first for the uncorrected intervals, and then the JASP results with the corrected results). Kind regards, Johnny dat <- read.csv2("~/Downloads/ForVanDoorn.csv", dec = ",") # convert wide data to long data longDat <- reshape2::melt(dat, measure.vars = 3:11) # compute anova and marginal means with afex and emmeans UNCORRECTED res <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable") mMeans <- emmeans::emmeans(res, specs = c("variable", "Sex")) # compute pp means to apply Morey correction idMeans <- tapply(longDat$value, longDat$FPID, mean) longDat$value <- longDat$value - idMeans[longDat$FPID] # compute anova and marginal means with afex and emmeans CORRECTED resCor <- afex::aov_ez(id = "FPID", dv = "value", data = longDat, between = "Sex", within = "variable") mMeansCor <- emmeans::emmeans(resCor, specs = c("variable", "Sex")) # plot marginal means plot(mMeans) plot(mMeansCor) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1187 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATBIGIVJ5WETBOJGQM7YZITTBSUP5ANCNFSM4YMXACFA>.

[AugustOlsson]

@JohnnyDoorn I have a related question. I have noticed in later editions of JASP that 95% Credible-intervals of RM-ANOVAS are much smaller than the 95% credible interval given in the Descriptives tables. As I understand the above discussion this is intended for frequentist confidence intervals, but not for bayesian credible intervals. Is it then a bug? I am currently using old intervals for my paper that I plotted with an older version of JASP where there was no discrepancy between the 95% credible intervlas from the descriptives table and the 95% credible intervals from the descriptive plots. But I would like to know if the smaller intervals that are currently plotted are actually "more" correct.

PS. Note if I should start a new issue on this, but it seems highly related.

EDIT: Basically the Credible intervals look identical to the Confidence intervals (which I think is by design given that a uniform prior is used for the decriptive plots) - thus I guess the same correction is used for the Bayesian Credible intervals. Is that resonable?

Best,
August

[tomtomme]

@AugustOlsson
Can you test this with the current JASP version 0.18.3 and share your .jasp file? Maybe then someone (@JohnnyDoorn ?) can answer this, if still relevant.
Thx

[AugustOlsson]

@tomtomme

I'll test it during next week and get back to you!

[tomtomme]

@AugustOlsson
I have tested this now with 0.19 beta.
Both RM ANOVAs show identical CIs in tables and in plots. So what you witnessed with the discrepancy between table and plot regarding the credible interval must have been a bug - thats now corrected.
Both seem to apply the Morey correction - but the Bayesian help file does not mention this. I will add this to the needed list of improvements for the help files here: #2529

@Angalanse
Closing for now, since #1227 tracks if the correction could be optional to avoid the pedagocical problem you mentioned.

[AugustOlsson]

Thanks for the update @tomtomme. I would really really like the option to get CIs without Morey correction though. If it was possible to include a choice to add/not add Morey correction that would be greatly apprechiated.

Best,
August

[tomtomme]

Yep. This is still on the todo list at #1227

[tomtomme]

@AugustOlsson
This is likely getting done for upcoming 0.19 version of jasp :)