Permalink
Browse files

Removing work required to add syntax sugar to controllers and views. …

…Also, now can optionally configure params[test_name] to override whatever A/Bingo picks for an alternative. To do so, set Abingo.options[:enable_specification] in your environment. This may be unsafe in production, think it through carefully before enabling it there.
  • Loading branch information...
1 parent 509a0cf commit 96853abc9a2a9231b92f7646a94bc09d9ba329e0 @patio11 patio11 committed Jan 11, 2010
View
@@ -0,0 +1,20 @@
+Copyright (c) 2009 Patrick McKenzie
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
View
@@ -0,0 +1,109 @@
+A/Bingo
+======
+
+Rails A/B testing. One minute to install. One line to set up a new A/B test.
+One line to track conversion.
+
+For usage notes, see: http://www.bingocardcreator.com/abingo
+
+Installation instructions are below usage examples.
+
+Key default features:
+ -- Conversions only tracked once per individual.
+ -- Conversions only tracked if individual saw test.
+ -- Same individual ALWAYS sees same alternative for same test.
+ -- Syntax sugar. Specify alternatives as a range, array,
+ hash of alternative to weighting, or just let it default to true or false.
+ -- A simple z-test of statistical significance, with output so clear anyone in your organization
+ can understand it.
+
+Example: View
+
+<% ab_test("login_button", ["/images/button1.jpg", "/images/button2.jpg"]) do |button_file| %>
+ <%= img_tag(button_file, :alt => "Login!") %>
+<% end %>
+
+Example: Controller
+
+def register_new_user
+ #See what level of free points maximizes users' decision to buy replacement points.
+ @starter_points = ab_test("new_user_free_points", [100, 200, 300])
+end
+
+Example: Controller
+
+def registration
+ if (ab_test("send_welcome_email"))
+ #send the email, track to see if it later increases conversion to full version
+ end
+end
+
+Example: Conversion tracking (in a controller!)
+
+def buy_new_points
+ #some business logic
+ bingo!("new_user_free_points") # could have been just "bingo!" if that is your only test -- I like syntax sugar
+end
+
+Example: Conversion tracking (in a view)
+
+Thanks for signing up, dude! <% bingo!("signup_page_redesign") >
+
+Example: Statistical Significance Testing
+
+Abingo::Experiment.last.describe_result_in_words
+=> "The best alternative you have is: [0], which had 130 conversions from 5000 participants (2.60%).
+ The other alternative was [1], which had 1800 conversions from 100000 participants (1.80%).
+ This difference is 99.9% likely to be statistically significant, which means you can be extremely
+ confident that it is the result of your alternatives actually mattering, rather than being due to
+ random chance. However, this doesn't say anything about how much the first alternative is really
+ likely to be better by."
+
+Installation
+=======
+
+1) REQUIRED: You'll need to generate a DB migration to prepare two tables,
+then migrate your database. (Note: slight edits required if you use the table names
+"experiments" or "alternatives" at present.)
+
+ruby script/generate abingo_migration
+rake db:migrate
+
+2) REQUIRED: You need to tell A/Bingo a user's identity so that it knows who is
+who if they come back to a test. (The same identity will ALWAYS see the same
+alternative for the same test.) How you do this is up to you -- I suggest integrating
+with your login/account infrastructure. The simplest thing that can possibly work
+
+#Somewhere in application.rb
+before_filter :set_abingo_identity
+
+def set_abingo_identity
+ if (session[:abingo_identity])
+ Abingo.identity = session[:abingo_identity]
+ else
+ session[:abingo_identity] = Abingo.identity = rand(10 ** 10).to_i
+ end
+end
+
+3) RECOMMENDED: A/Bingo makes HEAVY use of the cache to reduce load on the
+database and share potentially long-lived "temporary" data, such as what alternative
+a given visitor should be shown for a particular test. You SHOULD use a cache
+which is shared across all Rails processes -- that probably means MemcachedStore,
+although you can get away with MemStore if you only have one Rails process.
+
+You PROBABLY SHOULD use a persistent cache in case you need to restart your
+machine. This is an amazingly good use case for MemcacheDB, so if you want to
+try playing with that, Google it. (Sets up VERY easily on the newer Ubuntu distros.)
+
+If you can't use a persistent cache, you're probably still OK if Memcached very
+rarely needs to be restarted. If the cache gets flushed, you will double-count
+entrants to a particular experiment and possibly double-count conversions, but
+that may not be the worse thing in the world.
+
+A/Bingo defaults to using the same cache store as Rails. If you want to change it
+
+#production.rb
+Abingo.cache = ActiveSupport::Cache::MemCacheStore.new("cache.example.com:12345") #best if really memcacheDB
+
+
+Copyright (c) 2009 Patrick McKenzie, released under the MIT license
View
@@ -0,0 +1,23 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test the abingo plugin.'
+Rake::TestTask.new(:test) do |t|
+ t.libs << 'lib'
+ t.libs << 'test'
+ t.pattern = 'test/**/*_test.rb'
+ t.verbose = true
+end
+
+desc 'Generate documentation for the abingo plugin.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+ rdoc.rdoc_dir = 'rdoc'
+ rdoc.title = 'Abingo'
+ rdoc.options << '--line-numbers' << '--inline-source'
+ rdoc.rdoc_files.include('README')
+ rdoc.rdoc_files.include('lib/**/*.rb')
+end
@@ -0,0 +1,11 @@
+class AbingoMigrationGenerator < Rails::Generator::Base
+ def manifest
+ record do |m|
+ m.migration_template 'abingo_migration.rb', 'db/migrate'
+ end
+ end
+
+ def file_name
+ "abingo_migration"
+ end
+end
@@ -0,0 +1,30 @@
+#Creates the two database tables, plus indexes, you'll need to use A/Bingo.
+
+class AbingoMigration < ActiveRecord::Migration
+ def self.up
+ create_table "experiments", :force => true do |t|
+ t.string "test_name"
+ t.timestamps
+ end
+
+ add_index "experiments", "test_name"
+ #add_index "experiments", "created_on"
+
+ create_table "alternatives", :force => true do |t|
+ t.integer :experiment_id
+ t.string :content
+ t.string :lookup, :limit => 32
+ t.integer :weight, :default => 1
+ t.integer :participants, :default => 0
+ t.integer :conversions, :default => 0
+ end
+
+ add_index "alternatives", "experiment_id"
+ add_index "alternatives", "lookup" #Critical for speed, since we'll primarily be updating by that.
+ end
+
+ def self.down
+ drop_table :experiments
+ drop_table :alternatives
+ end
+end
View
@@ -0,0 +1,5 @@
+require File.dirname(__FILE__) + '/lib/abingo'
+
+ActionController::Base.send :include, AbingoSugar
+
+ActionView::Base.send :include, AbingoViewHelper
View
@@ -0,0 +1 @@
+# Install hook code here
View
@@ -0,0 +1,167 @@
+#This class is outside code's main interface into the ABingo A/B testing framework.
+#Unless you're fiddling with implementation details, it is the only one you need worry about.
+
+#Usage of ABingo, including practical hints, is covered at http://www.bingocardcreator.com/abingo
+
+class Abingo
+
+ #Not strictly necessary, but eh, as long as I'm here.
+ cattr_accessor :salt
+ @@salt = "Not really necessary."
+
+ @@options ||= {}
+ cattr_accessor :options
+
+ #Defined options:
+ # :enable_specification => if true, allow params[test_name] to override the calculated value for a test.
+
+ #ABingo stores whether a particular user has participated in a particular
+ #experiment yet, and if so whether they converted, in the cache.
+ #
+ #It is STRONGLY recommended that you use a MemcacheStore for this.
+ #If you'd like to persist this through a system restart or the like, you can
+ #look into memcachedb, which speaks the memcached protocol. From the perspective
+ #of Rails it is just another MemcachedStore.
+ #
+ #You can overwrite Abingo's cache instance, if you would like it to not share
+ #your generic Rails cache.
+ cattr_writer :cache
+
+ def self.cache
+ @@cache || Rails.cache
+ end
+
+ #This method gives a unique identity to a user. It can be absolutely anything
+ #you want, as long as it is consistent.
+ #
+ #We use the identity to determine, deterministically, which alternative a user sees.
+ #This means that if you use Abingo.identify_user on someone at login, they will
+ #always see the same alternative for a particular test which is past the login
+ #screen. For details and usage notes, see the docs.
+ def self.identity=(new_identity)
+ @@identity = new_identity.to_s
+ end
+
+ def self.identity
+ @@identity ||= rand(10 ** 10).to_i.to_s
+ end
+
+ #A simple convenience method for doing an A/B test. Returns true or false.
+ #If you pass it a block, it will bind the choice to the variable given to the block.
+ def self.flip(test_name)
+ if block_given?
+ yield(self.test(test_name, [true, false]))
+ else
+ self.test(test_name, [true, false])
+ end
+ end
+
+ #This is the meat of A/Bingo.
+ def self.test(test_name, alternatives, options = {})
+ unless Abingo::Experiment.exists?(test_name)
+ Abingo::Experiment.start_experiment!(test_name, self.parse_alternatives(alternatives))
+ end
+
+ choice = self.find_alternative_for_user(test_name, alternatives)
+ participating_tests = Abingo.cache.read("Abingo::participating_tests::#{Abingo.identity}") || []
+
+ #Set this user to participate in this experiment, and increment participants count.
+ if options[:multiple_participation] || !(participating_tests.include?(test_name))
+ unless participating_tests.include?(test_name)
+ participating_tests << test_name
+ Abingo.cache.write("Abingo::participating_tests::#{Abingo.identity}", participating_tests)
+ end
+ Abingo::Alternative.score_participation(test_name)
+ end
+
+ if block_given?
+ yield(choice)
+ else
+ choice
+ end
+ end
+
+
+ def Abingo.bingo!(test_name_or_array = nil, options = {})
+ if test_name_or_array.kind_of? Array
+ test_name_or_array.map do |single_test|
+ self.bingo!(single_test, options)
+ end
+ else
+ participating_tests = Abingo.cache.read("Abingo::participating_tests::#{Abingo.identity}") || []
+ if test_name_or_array.nil?
+ participating_tests.each do |participating_test|
+ self.bingo!(participating_test, options)
+ end
+ else #Individual, non-nil test is named
+ test_name_str = test_name_or_array.to_s
+ if options[:assume_participation] || participating_tests.include?(test_name_str)
+ cache_key = "Abingo::conversions(#{Abingo.identity},#{test_name_str}"
+ if options[:multiple_conversions] || !Abingo.cache.read(cache_key)
+ Abingo::Alternative.score_conversion(test_name_str)
+ if Abingo.cache.exist?(cache_key)
+ Abingo.cache.increment(cache_key)
+ else
+ Abingo.cache.write(cache_key, 1)
+ end
+ end
+ end
+ end
+ end
+ end
+
+ #For programmer convenience, we allow you to specify what the alternatives for
+ #an experiment are in a few ways. Thus, we need to actually be able to handle
+ #all of them. We fire this parser very infrequently (once per test, typically)
+ #so it can be as complicated as we want.
+ # Integer => a number 1 through N
+ # Range => a number within the range
+ # Array => an element of the array.
+ # Hash => assumes a hash of something to int. We pick one of the
+ # somethings, weighted accorded to the ints provided. e.g.
+ # {:a => 2, :b => 3} produces :a 40% of the time, :b 60%.
+ #
+ #Alternatives are always represented internally as an array.
+ def self.parse_alternatives(alternatives)
+ if alternatives.kind_of? Array
+ return alternatives
+ elsif alternatives.kind_of? Integer
+ return (1..alternatives).to_a
+ elsif alternatives.kind_of? Range
+ return alternatives.to_a
+ elsif alternatives.kind_of? Hash
+ alternatives_array = []
+ alternatives.each do |key, value|
+ if value.kind_of? Integer
+ alternatives_array += [key] * value
+ else
+ raise "You gave a hash with #{key} => #{value} as an element. The value must be an integral weight."
+ end
+ end
+ return alternatives_array
+ else
+ raise "I don't know how to turn [#{alternatives}] into an array of alternatives."
+ end
+ end
+
+ def self.retrieve_alternatives(test_name, alternatives)
+ cache_key = "Abingo::#{test_name}::alternatives".gsub(" ","")
+ alternative_array = self.cache.fetch(cache_key) do
+ self.parse_alternatives(alternatives)
+ end
+ alternative_array
+ end
+
+ def self.find_alternative_for_user(test_name, alternatives)
+ alternatives_array = retrieve_alternatives(test_name, alternatives)
+ alternatives_array[self.modulo_choice(test_name, alternatives_array.size)]
+ end
+
+ #Quickly determines what alternative to show a given user. Given a test name
+ #and their identity, we hash them together (which, for MD5, provably introduces
+ #enough entropy that we don't care) otherwise
+ def self.modulo_choice(test_name, choices_count)
+ Digest::MD5.hexdigest(Abingo.salt.to_s + test_name + self.identity.to_s).to_i(16) % choices_count
+ end
+
+end
Oops, something went wrong.

0 comments on commit 96853ab

Please sign in to comment.