Bayesian A/B testing for Rails
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
app Added option to use events Dec 18, 2016
config Added ability to update variants from dashboard Dec 18, 2016
lib Version bump to 0.2.4 [skip ci] Jan 4, 2019
test Fixed test numbers Dec 18, 2016
.gitignore First commit Dec 14, 2016
CHANGELOG.md Version bump to 0.2.4 [skip ci] Jan 4, 2019
Gemfile First commit Dec 14, 2016
LICENSE.txt First commit Dec 14, 2016
README.md http -> https Mar 26, 2018
Rakefile First commit Dec 14, 2016
field_test.gemspec Added tests for calculations Dec 18, 2016

README.md

Field Test

🍁 A/B testing for Rails

  • Designed for web and email
  • Comes with a handy dashboard
  • Seamlessly handles the transition from anonymous visitor to logged in user

Uses Bayesian statistics to evaluate results so you don’t need to choose a sample size ahead of time.

Installation

Add this line to your application’s Gemfile:

gem "field_test"

Run:

rails g field_test:install

And mount the dashboard in your config/routes.rb:

mount FieldTest::Engine, at: "field_test"

Be sure to secure the dashboard in production.

Screenshot

Getting Started

Add an experiment to config/field_test.yml.

experiments:
  button_color:
    variants:
      - red
      - green
      - blue

Refer to it in views, controllers, and mailers.

button_color = field_test(:button_color)

When someone converts, record it with:

field_test_converted(:button_color)

When an experiment is over, specify a winner:

experiments:
  button_color:
    winner: green

All calls to field_test will now return the winner, and metrics will stop being recorded.

Features

You can specify a variant with query parameters to make testing easier

?field_test[button_color]=green

Assign a specific variant to a user with:

experiment = FieldTest::Experiment.find(:button_color)
experiment.variant(participant, variant: "green")

You can also change a user’s variant from the dashboard.

Config

By default, bots are returned the first variant and excluded from metrics. Change this with:

exclude:
  bots: false

Keep track of when experiments started and ended. Use any format Time.parse accepts. Variants assigned outside this window are not included in metrics.

experiments:
  button_color:
    started_at: Dec 1, 2016 8 am PST
    ended_at: Dec 8, 2016 2 pm PST

Add a friendlier name and description with:

experiments:
  button_color:
    name: Buttons!
    description: >
      Different button colors
      for the landing page.

By default, variants are given the same probability of being selected. Change this with:

experiments:
  button_color:
    variants:
      - red
      - blue
    weights:
      - 85
      - 15

If the dashboard gets slow, you can make it faster with:

cache: true

This will use the Rails cache to speed up winning probability calculations.

Funnels

You can set multiple goals for an experiment to track conversions at different parts of the funnel. First, run:

rails g field_test:events

And add to your config:

experiments:
  button_color:
    goals:
      - signed_up
      - ordered

Specify a goal during conversion with:

field_test_converted(:button_color, goal: "ordered")

The results for all goals will appear on the dashboard.

Analytics Platforms

You can also send experiment data to analytics platforms like Google Analytics, Mixpanel, and Ahoy. Use:

field_test_experiments

to get all experiments and variants for a participant and pass them as properties.

Security

Devise

authenticate :user, -> (user) { user.admin? } do
  mount FieldTest::Engine, at: "field_test"
end

Basic Authentication

Set the following variables in your environment or an initializer.

ENV["FIELD_TEST_USERNAME"] = "moonrise"
ENV["FIELD_TEST_PASSWORD"] = "kingdom"

Credits

A huge thanks to Evan Miller for deriving the Bayesian formulas.

TODO

  • Code samples for analytics platforms

History

View the changelog

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help: