Adding a new way to calculate winner probabilities

[caser]

I'd like to add another column to the experimental results which uses a different statistical test to determine the likelihood that an experiment variant is the best.

For A/B tests, a beta distribution can be more accurate than a normal distribution. A beta distribution goes from 0 to 1, whereas a normal distribution goes from -infinty to +infinity.

With conversion rates somewhere in the middle between 0 and 1, a normal distribution may give accurate data, but when the conversion rate nears 0 or 100%, the normal distribution becomes inaccurate. This is because a large portion of its tails extend beyond 0 or 1, which are the realistic limits for a conversion rate.

Also, a normal distribution needs a large sample size to give any actionable results, whereas a beta distribution can give information even with few data points.

Google Content Experiments uses a beta-distribution, and it is also the first step in adding a multi-armed bandit feature (higher performing variants get more traffic in the experiment over time).

Here is Google Content Experiments explanation from their appendix:
https://support.google.com/analytics/answer/2846882?hl=en

Anyways, questions or feedback? Otherwise I'll start coding it.

Cheers,
Casey

[andrew]

Sounds like a great addition, looking forward to seeing it in action.

[andrew]

Closing as this issue is over 2 years old, please reopen if it's still affecting you.