New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Group Sequential - Percentile Issue #176
Comments
hi @louisryan by "incorrect" do you mean |
@shansfolder I'm wondering that this problem appeared after the version upgrade |
@louisryan this might be actually a bug fix instead of a problem :) |
@shansfolder, in my case, estimated sample size is 50000. After lowering it to 10000, the |
hi @louisryan , yes this is a fix. |
@louisryan let me try to explain intuitively: when the current sample size is small(compared to estimated sample size), to be conservative, we make the confidence interval very large. A very large interval infers that it should cover 0 --- so it's almost impossible to conclude significance when sample size is still small. |
@shansfolder that makes sense, but do you not think that the confidence interval should be displayed regardless? With The reason why I say this is down to experience running experiments with one core success metric, but other metrics of interest that we want to monitor that might only achieve a fraction of the sample size. Using the Would love to know your thoughts |
Hi, @louisryan my understanding is that if the group sequential method gives you But I admit the way of our result format is confusing. Let me know if you have any suggestions. :) |
Good point! :) |
Thanks for the clarification |
Hi there,
I have upgraded from Expan 0.6.2 -> 0.6.5 and upon re-running the group sequential method, the percentiles are showing incorrect values:
Upon downgrading, the issue has been resolved.
The text was updated successfully, but these errors were encountered: