Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Experiment Driven Development for Ruby
Ruby CSS JavaScript

Turn off Ruby2.2 testing for Rails 3.2.

Although support on 3-2-stable branch, support is not official released as of Feb. '15.
latest commit 9d65cc8497
Phill Baker phillbaker authored




Vanity is an A/B testing framework for Rails that is datastore agnostic.


A/B Testing With Rails

Step 1: Start using Vanity in your Rails application

Step 1.1

Rails 3 & Rails 4 installation

Add to your Gemfile:

gem "vanity"

(For support for older versions of Rails and Ruby 1.8, please see the 1.9.x branch.)

Step 1.2

Choose a datastore that best fits your needs and preferences for storing experiment results. Choose one of: Redis, MongoDB or an SQL database. While Redis is usually faster, it may add additional complexity to your stack. Datastores should be configured using a config/vanity.yml.

Redis Setup

Add to your Gemfile:

gem "redis", ">= 2.1"
gem "redis-namespace", ">= 1.1.0"

By default Vanity is configured to use Redis on localhost port 6379 with database 0.

MongoDB Setup

Add to your Gemfile:

gem "bson_ext"
gem "mongo"

A sample config/vanity.yml might look like:

  adapter: mongodb
  database: analytics
  collecting: false
  adapter: mongodb
  database: analytics
SQL Database Setup

Vanity supports multiple SQL stores (like MySQL, MariaDB, Postgres, Sqlite, etc.) using ActiveRecord, which is built into Rails. If you're using DataMapper, Sequel or another persistence framework, add to your Gemfile:

gem "active_record"

A sample config/vanity.yml might look like:

  adapter: active_record
  active_record_adapter: sqlite3
  database: db/development.sqlite3
  collecting: false
  adapter: active_record
  active_record_adapter: mysql
  database: vanity_test
  pool: 5
  timeout: 5000

If you're going to store data in the database, run the generator and migrations to create the database schema:

$ rails generate vanity
$ rake db:migrate

Step 1.3

Turn Vanity on, and pass a reference to a method that identifies a user. For example:

class ApplicationController < ActionController::Base
  use_vanity :current_user

For more information, please see the identity documentation.

Step 2: Define your first A/B test

This experiment goes in the file experiments/price_options.rb:

ab_test "Price options" do
  description "Mirror, mirror on the wall, who's the better price of all?"
  alternatives 19, 25, 29
  metrics :signups

If the experiment uses a metric as above (“signups”), there needs to be a corresponding ruby file for that metric, experiments/metrics/signups.rb.

metric "Signup (Activation)" do
  description "Measures how many people signed up for our awesome service."

Step 3: Present the different options to your users

<h2>Get started for only $<%= ab_test :price_options %> a month!</h2>

Step 4: Measure conversion

Conversions are created via the track! method. For example:

class SignupController < ApplicationController
  def signup
    @account =[:account])
      track! :signups
      redirect_to @acccount
      render action: :offer

Step 5: Check the report:

vanity report --output vanity.html

To view metrics and experiment results with the dashboard in Rails 3 & Rails 4:

rails generate controller Vanity --helper=false

In config/routes.rb, add:

get '/vanity' =>'vanity#index'
get '/vanity/participant/:id' => 'vanity#participant'
post '/vanity/complete'
post '/vanity/chooses'
post '/vanity/reset'
post '/vanity/add_participant'
get '/vanity/image'

The controller should look like:

class VanityController < ApplicationController
  include Vanity::Rails::Dashboard
  layout false  # exclude this if you want to use your application layout

Registering participants with Javascript

If robots or spiders make up a significant portion of your sites traffic they can affect your conversion rate. Vanity can optionally add participants to the experiments using asynchronous javascript callbacks, which will keep many robots out. For those robots that do execute Javascript and are well-behaved (like Googlebot), Vanity filters out requests based on their user-agent string.

To set this up simply do the following:

  • Add Vanity.playground.use_js!

  • Set Vanity.playground.add_participant_path = '/path/to/vanity/action', this should point to the add_participant path that is added with Vanity::Rails::Dashboard, make sure that this action is accessible by all users (ie does not require authentication).

  • Add <%= vanity_js %> to any page that XX an ab_test. vanity_js needs to be included after your call to ab_test so that it knows which version of the experiment the participant is a member of. The helper will render nothing if the there are no ab_tests running on the current page, so adding vanity_js to the bottom of your layouts is a good option. Keep in mind that if you call use_js! and don't include vanity_js in your view no participants will be recorded.


Here's what's tested and known to work:

Ruby 1.9.3
  Persistence: Redis, Mongo, ActiveRecord
  Rails: 3.2, 4.x
Ruby 2.0
  Persistence: Redis, Mongo, ActiveRecord
  Rails: 3.2, 4.x
Ruby 2.1
  Persistence: Redis, Mongo, ActiveRecord
  Rails: 3.2, 4.x
Ruby 2.2
  Persistence: Redis, Mongo, ActiveRecord
  Rails: 4.0


For view tests/specs or integration testing, it's handy to set the outcome of an experiment. This may be done using the chooses method. For example:



  • Fork the project

  • Please use a feature branch to make your changes, it's easier to test them that way

  • To set up the test suite run `bundle`, then run `appraisal install` to prepare the test suite to run against multiple versions of Rails

  • Fix, patch, enhance, document, improve, sprinkle pixie dust

  • Tests. Please. Run `appraisal rake test`, of if you can, `rake test:all`. (This project uses Travis CI where the test suite is run against multiple versions of ruby, rails and backends.)

  • Send a pull request on GitHub


Original code, copyright of Assaf Arkin, released under the MIT license.

Documentation available under the Creative Commons Attribution license.

For full list of credits and licenses:

Something went wrong with that request. Please try again.