Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug in p-value calculations in adonis #120

Closed
fra-gh opened this issue Jun 12, 2015 · 4 comments
Closed

Bug in p-value calculations in adonis #120

fra-gh opened this issue Jun 12, 2015 · 4 comments

Comments

@fra-gh
Copy link

fra-gh commented Jun 12, 2015

The p-value calculated by adonis is sometimes wrong.
For example, the following code

library("vegan")
  options(digits = 12)
  g = as.factor(c("Control","Control","Group1", "Group2", "Group2"))
  d = as.dist(matrix(c(0,3,7,2,1, 3,0,5,4,1, 7,5,0,2,6, 4,2,4,0,2, 1,1,6,2,0), ncol=5))
  pp = adonis(d ~ g)
  pp

returns


Call:
adonis(formula = d ~ g) 

Permutation: free
Number of permutations: 120

Terms added sequentially (first to last)

          Df SumsOfSqs MeanSqs     F.Model           R2  Pr(>F)
g          2      23.3   11.65 3.584615385 0.7818791946 0.20833
Residuals  2       6.5    3.25             0.2181208054        
Total      4      29.8                     1.0000000000  

The p-value is wrong (only 24 permutations with high f instead of 39). The correct p-value can be computed from the definition to be

(rowSums(t(pp$f.perms >= pp$aov.tab$F.Model[1]))+1) / (length(pp$f.perms) + 1)

which correctly returns

[1] 0.333333333333

It looks like the bug is due to rounding. When comparing f.perms and F.mod, the value of f.perms is rounded but the value of F.mod is not (although it is rounded afterwards). This makes the >= comparison fail in this example because the value of f.perms is rounded down.

jarioksa added a commit that referenced this issue Jun 12, 2015
adonis rounded permuted statistics, but did not round observed
statistic before it was used, and tied values were not correctly
recognized in assessing P-values.

Reported as issue #120 in github by user fra-gh
@jarioksa
Copy link
Contributor

jarioksa commented Jun 12, 2015

Confirmed: this is a bug. Fixed in commit f727b58.

Thanks for reporting this.

@fra-gh
Copy link
Author

fra-gh commented Jun 12, 2015

thanks for the very prompt reply. keep up the good work!

jarioksa added a commit that referenced this issue Jun 12, 2015
adonis rounded permuted statistics, but did not round observed
statistic before it was used, and tied values were not correctly
recognized in assessing P-values.

Reported as issue #120 in github by user fra-gh
(cherry picked from commit f727b58)
@jarioksa
Copy link
Contributor

jarioksa commented Aug 28, 2015

PR #136 has a better fix for this issue.

@chitrabaniya
Copy link

chitrabaniya commented Feb 19, 2019

Do we always need to calculate p-value separately or it goes automatically fixed?
I got surprised with different p-values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants