inconsistent results from LORDstar and SAFFRONstar

[aaronjfisher]

I was trying the different versions of LORDstar & SAFFRONstar and it looks like they give different results when they should be equivalent. I'm not sure if I'm misunderstanding something in the documentation or not.

For example, the code example in ?LORDstar appears to show two equivalent representations of having each p-value be "in conflict" with the previous p-value. In other words, p_i needs to be specified at time i-2. In the first case, this is done by saying that the decision (reject or not) is not seen until the end of the next stage, and so the threshold for p_i can only be based on observed rejections from tests 1 through i-2. As far as I understand, having a lag of 1 is should produce the same conflict set, but the two settings give different results.

Is this a bug? Or am I misunderstanding the documentation?

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
decision.times = seq_len(15) + 1)

out1 <- LORDstar(sample.df, version='async')

sample.df2 <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
lags = rep(1,15))

out2 <- LORDstar(sample.df2, version='dep')

all(sample.df$pval == sample.df2$pval) 
## TRUE
range(out1$alphai - out2$alphai) / mean(abs(out1$alphai))
## [1] 0.000000 1.294928

SAFFRONstar produces a similar apparent issue

out3 <- SAFFRONstar(sample.df, version='async')
out4 <- SAFFRONstar(sample.df2, version='dep')
range(out3$alphai - out4$alphai) / mean(abs(out3$alphai))
## [1] 0.000000 1.663927

[dsrobertson]

This is a bug - thanks for pointing this out. I've now fixed LORDstar and SAFFRONstar with the latest commit (#54). There was the same problem with LONDstar which has now also been fixed.