Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

A/B test analysis library for Ruby - performs Chi-Square tests and G-tests on A/B results

branch: master
README.rdoc

ABAnalyzer

ABAnalyzer is a Ruby library that will perform testing to determine if there is a statistical difference in categorical data (typically called an A/B Test). By default, it uses a G-Test for independence, but a Chi-Square test for independence can also be used. For more information, see:

Installation

Simply run:

gem install abanalyzer

Basic Usage

The simplest test (which uses a gtest):

require 'rubygems'
require 'abanalyzer'

values = {}
values[:agroup] = { :opened => 100, :unopened => 300 }
values[:bgroup] = { :opened => 50, :unopened => 350 }

tester = ABAnalyzer::ABTest.new values
# Are the two different?  Returns true or false (at 0.05 level of significance)
puts tester.different?

Multiple Categories

You can use the ABAnalyzer module to test for differences in more than two categories. For instance, to test accross three:

require 'rubygems'
require 'abanalyzer'

values = {}
values[:agroup] = { :male => 200, :female => 250 }
values[:bgroup] = { :male => 150, :female => 300}
values[:cgroup] = { :male => 50, :female => 50 }

tester = ABAnalyzer::ABTest.new values
# Are the two different?  Returns true or false (at 0.05 level of significance)
puts tester.different?

Tests Available

You can get the actual p-value for either a Chi-Square test for independence or a G-Test for independence.

...
tester = ABAnalyzer::ABTest.new values
puts tester.chisquare_p
puts tester.gtest_p

You can additionally get the actual score for either a Chi-Square test for independence or a G-Test for independence.

...
tester = ABAnalyzer::ABTest.new values
puts tester.chisquare_score
puts tester.gtest_score

Sample Size Calculations

Let's say you want to determine how large your sample size needs to be for an A/B test. Let's say your baseline is 10%, and you want to be able to determine if there's at least a 10% relative lift (1% absolute) to 11%. Let's assume you want a power of 0.8 and a significance level of 0.05 (that is, an 80% chance of that you'll fail to recognize a difference when there is one, and a 5% chance of a false negative).

...
ABAnalyzer.calculate_size(0.1, 0.11, 0.05, 0.8)
 => 14751

This means that you will need at least 14,751 people in each group sample. You can see this same example with R at on the 37 signals blog.

Confidence Intervals

You can also get a confidence interval. Let's say you have the results of a test where there were 711 successes out of 4000 trials. To get a 95% confidence interval of the “true” value of the conversion rate, use:

...
ABAnalyzer.confidence_interval(711, 4000, 0.95)
 => [0.1659025512617185, 0.1895974487382815]

This means (roughly) that if you ran this experiment over and over, 95% of the time the resulting proportion would be between 17% and 19%.

You can also determine what the relative confidence intervals would be. Let's say that your old conversion rate was 13%, and you wanted to know what sort of relative lift you could get.

...
ABAnalyzer.relative_confidence_interval(711, 4000, 0.13, 0.95)
 => [0.27617347124398833, 0.45844191337139606]

This means (roughly) that if you ran this experiment over and over, 95% of the time the resulting proportion would be a relative lift of between 28% and 46%. Go buy yourself a beer!

Running Tests

Testing can be run by using:

rake test
Something went wrong with that request. Please try again.