Confidence.js: Make sense of your A/B test results
JavaScript
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
docs
test
.gitignore
LICENSE
NOTICE
README.md
component.json
confidence.js
package.json

README.md

Confidence.js

Confidence.js is a light-weight JavaScript library to help you make sense of your A/B Test results. Given an A/B test (or "Split Test") result set, Confidence.js will tell you if a statistical "winner" can be determined.

Documentation

Getting Started

Download confidence.js here

Include confidence.js in your HTML.

<script src="path/to/confidence.js"></script>

You're all ready! Start testing... with confidence.

Example

We have many examples in the API Reference. Here's a simple one to get you started:

// Create a new test result
var myConfidence = new Confidence();

// Add variant A results
myConfidence.addVariant({
  id: 'A',
  name: 'Variant A',
  conversionCount: 2500,
  eventCount: 3000
});

// Add variant B results
myConfidence.addVariant({
  id: 'B',
  name: 'Variant B',
  conversionCount: 1500,
  eventCount: 3000
});

// Get results, using Z-Test method
zTestResult = myConfidence.getResult();
/*
{
  hasWinner: true,
  hasEnoughData: true,
  winnerID: 'A',
  winnerName: 'Variant A',
  confidencePercent: 95.00,
  confidenceInterval: { min: 82, max: 84.67 },
  readable: 'With 95% confidence, the true population
    parameter of the "Variant A" variant will
    fall between 82% and 84.67%.'
}
*/

How It Works

Confidence.js provides two methods for evaluating an A/B test:

  • The Z-Test Method
  • Chi Square Test and Marascuillo's Procedure

Selecting an appropriate method depends on your data set and use case - both have advantages.

Learn how each method works here.

Running the Tests

npm install
npm test

Feature Requests and TODO

  • implement "Confidence to beat baseline" comparison (in progress)
  • add removeVariant function

Issues and Questions

Found a bug? Create an issue here on GitHub!

For general questions, tweet me @jessicaraygun.

Authors

Developed and maintained by Jessica Thomas, Data Scientist @ sendwithus.com, with guidance and tutelage from Statistics Mastermind Emily Malcolm.