Rails AB Testing - Simple Abs

I recently turned on paid subscriptions to Draft, the writing software I've created. And I wanted a really simple way to test a few alternatives of the payment page in Rails without needing to use a separate service.

But the solutions out there get too complicated. Even the "simplest" ones require things like Redis. They do that because somewhere the AB testing library needs to remember what variation of a test a user has already seen, so it knows what to show them on subsequent visits.

But I don't want to install Redis just to have my AB tests be performant. That's still an extra network call to Redis for this simple operation, not to mention the added complexity of adding Redis to my software stack when I don't need it right now.

Why can't the AB testing library just store what variation a user has already seen in the user's cookies?

That's what SimpleAbs does.

Installation

Add this line to your application's Gemfile:

gem 'simple_abs'

And then execute:

$ bundle

Or install it yourself as:

$ gem install simple_abs

Create the migration to install the Alternatives table:

rails g simple_abs

Run the migrations:

rake db:migrate

Usage

Basic Usage

Use simple_abs to figure out an "alternative" to show your users. You can get ahold of an alternative from either a Rails View or a Controller.

ab_test(experiment name, [variation name 1, variation name 2, etc.])

Here's an example where I might have three different versions of a buy page. simple_abs will randomly pick which version this user should see.

def buy
  @buy_page = ab_test("buy_page", ["short", "medium", "long"])

  render action: 'buy'
end

Then, in your template, you can use an if statement to show them that version of the @buy_page:

<% if @buy_page == "long" %>
  Lots of extra information
<% end %>

If they've already seen one of the alternatives, simple_abs figures that out from the permanent cookies of the user. In other words, if on the first visit, this method:

ab_test("buy_page", ["short", "long"])

Returns "short". On subsequent visits to the page from this same user, you will also get the value "short" from the ab_test method.

Once your user converts you can call "converted!(experiment name)" from a View or Controller:

converted!("buy_page")

You can also force someone to a specific alternative of your page by using the query paramater "test_value" in your url:

http://draftin.com/buy_things?test_value=long

Abort operation

Sometimes it makes sense to abort an experiment. So we can just mark an experiment as done, take out the impression. So it would be as if nothing has ever happened. To do that

ab_test_aborted!("buy_page")

Multiple impression mode

Sometimes it's important to do multiple impression tests. For example, if the test is about whether to show the short or long title for an item among 10 items on the buy page. Then the regular ab test doesn't work that well as you get 10 impressions a page. And the statistics should base on how many item the user actually buys. Say if version A shows 10 products a page and user browsed 2 pages and bought 3 items. Then what matters is 3/20 instead of a 0/1 conversion. In this case, try this version instead.

def buy
  @products = Product.last(10) # A query to search for product
  @buy_page = ab_test_with_impression("buy_page", ["short", "medium", "long"], @products.size)

  render action: 'buy' # Will render the 10 products here
end

Then in your template, you can do this

<% @products.each do |product| %>
  <% if @buy_page == "long" %>
    Lots of extra information
  <% end %>
<% end %>

If they've already seen one of the alternatives, simple_abs figures that out from the permanent cookies of the user. In other words, if on the first visit, this method:

ab_test("buy_page", ["short", "long"], 10)

Returns "short". On subsequent visits to the page from this same user, you will also get the value "short" from the ab_test method. However it will increse the participants by 10 every time you hit it.

Once your user converts you can call "converted!(experiment name)" from a View or Controller:

@purchased_products = ["some items here"]
converted_with_frequency!("buy_page", @purchased_products.size)

### Statistics

When you get some data you can look at it from a Rails console. 

```ruby
irb(main):011:0> pp SimpleAbs::Alternative.where("experiment = 'buy_page'").all
  SimpleAbs::Alternative Load (55.1ms)  SELECT "alternatives".* FROM "alternatives" WHERE (experiment = 'buy_page')
[#<SimpleAbs::Alternative id: 2, which: "short", participants: 16, conversions: 1, experiment: "buy_page", created_at: "2013-04-16 05:14:13", updated_at: "2013-04-16 13:39:14">,
 #<SimpleAbs::Alternative id: 1, which: "long", participants: 20, conversions: 1, experiment: "buy_page", created_at: "2013-04-16 05:11:12", updated_at: "2013-04-16 14:30:01">]

And then you can plug the participants and conversions into a calculator like that found here:

http://tools.seobook.com/ppc-tools/calculators/split-test.html

Trials        = simple_abs' participants
Successes     = simple_abs' conversions    

Feedback

Source code available on Github. Feedback and pull requests are greatly appreciated.

Credit

This library is very much inspired by the other AB testing libraries out there. Not the least of which is A/Bingo from Patrick McKenzie.

---

P.S. It would be awesome to meet you on Twitter.