Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: add option for null_var or null_prop to samplesize_proportions_2indep_onetail #8675

Open
josef-pkt opened this issue Feb 15, 2023 · 1 comment · May be fixed by #8676
Open

ENH: add option for null_var or null_prop to samplesize_proportions_2indep_onetail #8675

josef-pkt opened this issue Feb 15, 2023 · 1 comment · May be fixed by #8676

Comments

@josef-pkt
Copy link
Member

https://stats.stackexchange.com/questions/605466/how-to-get-python-statsmodels-to-match-evan-millers-famous-ab-test-sample-size/605540#605540

I'm not sure what the name of the options should be.

@josef-pkt
Copy link
Member Author

"pooled" is actually the estimated prob under the null of equal proportion if the true DGP is the one specified by the alternative.

using prop2 or prop1 for var_null might not correspond to any hypothesis test method, unless the hypothesis specified a fixed level/value of prop2 under the null, e.g. H0: prob1 = prob2 = 0.05.

The generalization of "pooled" if there is a nonzero null diff (null margin as in noninferiority testing) would be the estimate under the null hypothesis restriction, eg. p2 + p1 - value = 0. We would need a more generic name for the option.
This might differ under wald versus score test.

We need some MonteCarlo studies if we want to match up the power and sample size computation directly with the hypothesis test method.

Aside:
"pooled" uses the props specified by the alternative. Those are expected values under the alternative.
In the actual hypothesis test we use the sample estimates so var_null is a random variable. I don't think using the expected variance is "exact" (or equal to the one we would get with simulations) because of the nonlinearities.
PASS has some "averaged power" but, AFAIR, that refers to a distribution of props under the alternative (and not sampling variation of variance estimate, although using random props also changes the implied var.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant