Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A/A Tests show false positive rate exceeds alpha. #7

Open
sjoelee opened this issue Apr 1, 2021 · 0 comments
Open

A/A Tests show false positive rate exceeds alpha. #7

sjoelee opened this issue Apr 1, 2021 · 0 comments

Comments

@sjoelee
Copy link

sjoelee commented Apr 1, 2021

I'm interested in using this library so I tested to see if the False Positive Rate is controlled within alpha that I set. However, the results are showing that the number of tests that the portion of tests rejected far exceeds alpha. For 100 trials, where each trial is comparing two Bernoulli proportions (of 0.01 conversion rate) and alpha=0.05, beta=0.10, and the total number of visitors is set to 100K, I get the following percent of reject and accept:

% rejected:  43
% accepted:  56.99999999999999

Here's the link to my code: https://gist.github.com/sjoelee/d4fed8b80e1af1d2e0cf7aac37d09a90. Once each visitor arrives and has been bucketed to treatment/control, I simulate a biased coin flip (based on conversion rate for variation) for their conversion. I add each individual data through addData and then look at the results to see if they finished, determining whether the result from addData returned true (accept null) or false (reject null). Could you provide more documentation on how the thresholds are calculated? And have any tests been done to see if A/A tests are still controlled under alpha? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant