New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug in p-value calculations in adonis #120

Closed
fra-gh opened this Issue Jun 12, 2015 · 3 comments

Comments

Projects
None yet
2 participants
@fra-gh

fra-gh commented Jun 12, 2015

The p-value calculated by adonis is sometimes wrong.
For example, the following code

library("vegan")
  options(digits = 12)
  g = as.factor(c("Control","Control","Group1", "Group2", "Group2"))
  d = as.dist(matrix(c(0,3,7,2,1, 3,0,5,4,1, 7,5,0,2,6, 4,2,4,0,2, 1,1,6,2,0), ncol=5))
  pp = adonis(d ~ g)
  pp

returns


Call:
adonis(formula = d ~ g) 

Permutation: free
Number of permutations: 120

Terms added sequentially (first to last)

          Df SumsOfSqs MeanSqs     F.Model           R2  Pr(>F)
g          2      23.3   11.65 3.584615385 0.7818791946 0.20833
Residuals  2       6.5    3.25             0.2181208054        
Total      4      29.8                     1.0000000000  

The p-value is wrong (only 24 permutations with high f instead of 39). The correct p-value can be computed from the definition to be

(rowSums(t(pp$f.perms >= pp$aov.tab$F.Model[1]))+1) / (length(pp$f.perms) + 1)

which correctly returns

[1] 0.333333333333

It looks like the bug is due to rounding. When comparing f.perms and F.mod, the value of f.perms is rounded but the value of F.mod is not (although it is rounded afterwards). This makes the >= comparison fail in this example because the value of f.perms is rounded down.

jarioksa added a commit that referenced this issue Jun 12, 2015

round-off errors with tied observed and permutation stats in adonis
adonis rounded permuted statistics, but did not round observed
statistic before it was used, and tied values were not correctly
recognized in assessing P-values.

Reported as issue #120 in github by user fra-gh
@jarioksa

This comment has been minimized.

Show comment
Hide comment
@jarioksa

jarioksa Jun 12, 2015

Contributor

Confirmed: this is a bug. Fixed in commit f727b58.

Thanks for reporting this.

Contributor

jarioksa commented Jun 12, 2015

Confirmed: this is a bug. Fixed in commit f727b58.

Thanks for reporting this.

@jarioksa jarioksa closed this Jun 12, 2015

@fra-gh

This comment has been minimized.

Show comment
Hide comment
@fra-gh

fra-gh Jun 12, 2015

thanks for the very prompt reply. keep up the good work!

fra-gh commented Jun 12, 2015

thanks for the very prompt reply. keep up the good work!

jarioksa added a commit that referenced this issue Jun 12, 2015

round-off errors with tied observed and permutation stats in adonis
adonis rounded permuted statistics, but did not round observed
statistic before it was used, and tied values were not correctly
recognized in assessing P-values.

Reported as issue #120 in github by user fra-gh
(cherry picked from commit f727b58)
@jarioksa

This comment has been minimized.

Show comment
Hide comment
@jarioksa

jarioksa Aug 28, 2015

Contributor

PR #136 has a better fix for this issue.

Contributor

jarioksa commented Aug 28, 2015

PR #136 has a better fix for this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment