Numeric p-value "always" 1 or 0?

[wleoncio]

Having a hard time finding an example where the analytic p-value is NA and the numeric one is neither 0 or 1.

I wonder if the numeric p-value (calculated below) should be computed based on >= or >.

permChacko/R/permChacko.R

Line 55 in 3c263d4

perm_p_value <- sum(perm_chisq_bar >= chisq_bar) / n_perm

If chisq_bar = 0, then all perm_p_values are going to be equal to or larger than that. Assuming the equation below (eq. 5 from Chacko 1966) is always non-negative, which seems to be the case:

The code below offers a template for some benchmarking to be done between the >= and > solutions. Play around with different values for runs, reps, and vec. Bias, error, etc, are some simple ways to compare the estimates to the tabular value:

runs <- 1000L
reps <- 1000L
vec <- c(6, 8, 4, 10, 7, 3, 2)
sapply(seq_len(runs), function(x) permChacko(vec, n_perm = reps)[["p_values"]])

[wleoncio]

As a first attempt at studying the issue through simulations, please see this script, which outputs this the last time I tried:

Which may be an indication that > is the preferrable estimator, especially if the analytic solution is absent.

To reproduce (results vary due to randomization), we need the package version on the issue-13 branch. I used version 0.2.0.9000-1709818294 for this post

remotes::install_github("ocbe-uio/permChacko@issue-13")

[wleoncio]

Solution: use mid-P values (Lancaster, 1961).