Skip to content

solomonR 0.8.2

Latest
Latest

Choose a tag to compare

View all tags
@JUhalt JUhalt released this 09 Oct 13:49
v0.8.2
17b1632

A patch release with one fix. If you have used perm_solomon() with an outcome that has tied scores (a binary outcome, counts, ratings, rounded scores), or with very small groups, rerun those analyses.

The fix: ties in the permutation test (#131)

At the participant level, perm_solomon() let rounding error decide whether a permutation whose statistic equals the observed one was counted as at least as extreme. The validity of the p-value is shown for the count of statistics at least as extreme as the observed one (Phipson & Smyth, 2010), so such ties must be counted. When they were missed, the p-value was too small.

  • Tied scores, at any sample size. With a binary outcome and 20 participants per group, one test gave p = .017 where the correct value is .056.
  • Very small groups. With groups of 4, 4, 3, and 3 and continuous scores, one test gave p = .004 where the correct value is .117.
  • Continuous scores with 10 or more per group. A tie is rare. For a contrast within one pretest condition, about 1 analysis in 100 with 999 permutations meets one, and its p-value rises by at most .001.
  • An observed statistic of exactly zero now gives a p-value of 1, at the participant and the cluster level.
  • Cluster-level tests otherwise give the same results as before.

Ties are now counted at the participant and the cluster level and in plot_perm() by one rule. The fix was reviewed independently three times against exact enumeration and closed-form p-values, and this release was checked against the same references.

What remains

These are tracked in #134.

  • Models fitted by iteration (for example logistic or Poisson), where the contrast's cells include a pretest covariate: a tie can still be missed. In the checks made, no p-value was affected by this with the logit or the Poisson log link.
  • A pretest or covariate far from zero. This release line does not center them. When the mean is about a thousand times the standard deviation or more, as with a calendar year, ties can still be lost. Results were right in every check with a pretest mean up to 500 times its standard deviation.
  • Degenerate cases: an observed labeling that empties an arm, and an outcome that does not vary within a pretest condition.

Everything else

No other function changes from 0.8.1. Within perm_solomon(), the refitted models may take up to 200 iterations where they took 25, which removes convergence warnings that some models gave. The development version, which is working toward v1.0.0, has the same fix.

Install

install.packages(
  "solomonR",
  repos = c(
    JUhalt = "https://juhalt.r-universe.dev",
    CRAN = "https://cloud.r-project.org"
  )
)

Reference

Phipson, B., & Smyth, G. K. (2010). Permutation p-values should never be zero: Calculating exact p-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology, 9(1), Article 39. https://doi.org/10.2202/1544-6115.1585

Assets 2
Loading