Analytics Testr
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
R
man
tests
.Rbuildignore
DESCRIPTION
NAMESPACE
README.md

README.md

testr

testr is an R package for Bayesian design and analysis of of randomized experiments (A/B tests). It uses an adaptive design (i.e. optional stopping rules), allowing tests to end as early as possible [5].

Two probability models are currently supported:

  1. Beta binomial model for testing retention/conversion [1], [7]
  2. Zero-inflated log-normal model for testing revenue. This model has two stages: modelling spenders vs. non-spenders with a beta-binomial model, and modelling revenue amongst spenders with a lognormal model [3].

testr provides a convenient API to these models, and extends them in several ways:

  • Generalized the early stopping rule [5] to the Zero-inflated log-normal model
  • Uses Monte Carlo integration of the posterior to evaluate stopping rules
  • Implements methods for visualizing the prior/posterior distributions

Example Usage

Install the package using devtools:

#install testr
library(devtools)
install_github('uken/testr')
library(testr)
?testr

To run a simple test for conversion with a Beta-Binomial model:

beta_binomial_ab_test(y=c(100,115), n=c(1000, 1000)) #flat prior
plot_conversion_prior(expected_conversion_rate=.1, alpha0=2) #try tweaking alpha0 to decrase variance
beta_binomial_ab_test(y=c(100,115), n=c(1000, 1000), expected_conversion_rate=.1, alpha0=2) #informative prior

To run a test for revenue with a zero-inflated lognormal model:

##### simulate some data from an A/B/C test
n <- 10000
A_data <- rbinom(n,1,conversion) * rlnorm(n, meanlog=0, sdlog=.2)
B_data <- rbinom(n,1,conversion) * rlnorm(n, meanlog=0.08, sdlog=.2)
C_data <- rbinom(n,1,conversion) * rlnorm(n, meanlog=0.15, sdlog=.2)
data <- data.frame(ab_group=rep(c('A','B','C'), each=n), ltv=c(A_data, B_data, C_data))

##### plot the priors and run the test
plot_revenue_prior(expected_conversion_rate=0.12, alpha0=15, expected_revenue_converted_users=1.5, v0=73, k0=100, s_sq0=1.2) #specify prior
l = lognormal_ab_test(data, expected_conversion_rate=0.65, alpha0=15, expected_revenue_converted_users=1.5, v0=73, k0=100, s_sq0=1.2)
plot(l)

To do

  • A/B/C...N tests - Handle tests with more than two groups using Hierarchical Models. [4], [10], [11] are good references on the topic. testr currently allows for more than two groups in a test, but does not take multiple comparisons into account.

  • Model checking - Add methods for verifying that the probability model good fit for the data. Gelman describes some methods for doing this in [3].

  • Priors - Add more ways to specify priors.

References

  1. http://engineering.richrelevance.com/bayesian-ab-tests

  2. http://engineering.richrelevance.com/bayesian-analysis-of-normal-distributions-with-python/

  3. http://engineering.richrelevance.com/bayesian-ab-testing-with-a-log-normal-model/

  4. Gelman, Andrew, et al. Bayesian data analysis. Vol. 2. London: Chapman & Hall/CRC, 2014.

  5. https://web.archive.org/web/20150419163005/http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html

  6. https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers

  7. http://developers.lyst.com/data/2014/05/10/bayesian-ab-testing/

  8. http://ewulczyn.github.io/How_Naive_AB_Testing_Goes_Wrong/

  9. https://www.chrisstucchio.com/blog/2013/metrics_manifesto.html

  10. http://blog.dominodatalab.com/ab-testing-with-hierarchical-models-in-python/

  11. http://sl8r000.github.io/ab_testing_statistics/use_a_hierarchical_model/

  12. http://varianceexplained.org/r/bayesian-ab-testing/