Skip to content

Commit

Permalink
Added upgrade command. When upgrading to 1.4 (or even head, from now …
Browse files Browse the repository at this point in the history
…on), please run:

  vanity upgrade

Using Redis namespace and all Vanity objects are now in the vanity: namespace.
  • Loading branch information
assaf committed Jul 1, 2010
1 parent d54b33e commit 5ed70eb
Show file tree
Hide file tree
Showing 14 changed files with 103 additions and 59 deletions.
8 changes: 8 additions & 0 deletions CHANGELOG
@@ -1,5 +1,10 @@
== 1.4.0 (Pending)

Note: Run this command to upgrade your database to 1.4, or you will not have
access to collected metrics and experiment data:

vanity upgrade

Connection adapters! We have a new way for managing connections which extends
to multiple adapters (not just Redis). The easiest is to use the configuration
file config/vanity.yml. For example:
Expand All @@ -24,6 +29,9 @@ To test specific version of Ruby:
To switch around:
rvm 1.9.2@vanity

* Added: Adapter API, see Vanity::Adapters::AbstractAdapter and
Vanity::Adapters::RedisAdapter.
* Added: Upgrade command.
* Change: Vanity.playground.redis and redis= methods are deprecated, use
connection and establish_connection instead.

Expand Down
2 changes: 1 addition & 1 deletion Gemfile
@@ -1,7 +1,7 @@
source "http://rubygems.org/"
gem "garb"
gem "rack", "1.0.1"
gem "redis"
gem "redis-namespace"

group :development do
gem "jekyll"
Expand Down
1 change: 0 additions & 1 deletion README.rdoc
Expand Up @@ -2,7 +2,6 @@ Vanity is an Experiment Driven Development framework for Rails.

* All about Vanity: http://vanity.labnotes.org
* On github: http://github.com/assaf/vanity
* Vanity requires Redis 1.0 or later.

http://farm3.static.flickr.com/2540/4099665871_497f274f68_o.jpg

Expand Down
4 changes: 2 additions & 2 deletions Rakefile
Expand Up @@ -27,7 +27,7 @@ end
# Run the test suit.

task :default=>:test
desc "Run all tests using Redis mock (also default task)"
desc "Run all tests"
Rake::TestTask.new do |task|
task.test_files = FileList['test/*_test.rb']
if Rake.application.options.trace
Expand Down Expand Up @@ -109,7 +109,7 @@ task(:jekyll) { sh "jekyll", "doc", "html" }
desc "Create documentation in docs directory (including API)"
task :docs=>[:jekyll, :yardoc]
desc "Remove temporary files and directories"
task(:clobber) { rm_rf "html" }
task(:clobber) { rm_rf "html" ; rm_rf ".yardoc" }

desc "Publish documentation to vanity.labnotes.org"
task :publish=>[:clobber, :docs] do
Expand Down
20 changes: 16 additions & 4 deletions bin/vanity
Expand Up @@ -17,6 +17,7 @@ opts = OptionParser.new("", 24, " ") do |opts|
opts.banner << "Commands:\n"
opts.banner << " list List all experiments and metrics\n"
opts.banner << " report Report on all running experiments/metrics\n"
opts.banner << " upgrade Upgrade your database when deploying new release\n"

opts.separator ""
opts.separator "Reporting options:"
Expand All @@ -29,8 +30,12 @@ opts = OptionParser.new("", 24, " ") do |opts|
opts.on "--load_path PATH", "Path to experiments directory (default: #{playground.load_path})" do |path|
playground.load_path = path
end
opts.on "--redis HOST:PORT:DB", "Redis server host (default: localhost:6379)" do |redis|
playground.redis = redis
opts.on "-d", "--database url", "Database connection URL (e.g. redis:/localhost:6379)" do |conn|
playground.establish_connection conn
end
opts.on "--redis HOST:PORT:DB", "DEPRECATED: Redis server host (default: localhost:6379)" do |redis|
host, port, db = redis.split(":")
playground.establish_connection "redis:/#{host}:#{port}/#{db}"
end
opts.on_tail "-h", "--help", "Show this message" do
puts opts.to_s.gsub(/^.*DEPRECATED.*$/s, '')
Expand All @@ -50,8 +55,15 @@ end

ARGV.each do |cmd|
case cmd
when "report"; Vanity::Commands.report options.output
when "list"; Vanity::Commands.list
when "report"
require "vanity/commands/report"
Vanity::Commands.report options.output
when "list"
require "vanity/commands/list"
Vanity::Commands.list
when "upgrade"
require "vanity/commands/upgrade"
Vanity::Commands.upgrade
else fail "No such command: #{cmd}"
end
end
1 change: 0 additions & 1 deletion doc/index.textile
Expand Up @@ -7,7 +7,6 @@ title: Welcome to Vanity

!http://farm3.static.flickr.com/2540/4099665871_497f274f68_o.jpg!

Vanity requires Redis 1.0 or later.

h3. Reading Order

Expand Down
4 changes: 2 additions & 2 deletions doc/metrics.textile
Expand Up @@ -220,12 +220,12 @@ You can always populate the hash with your own metrics.

When Vanity loads a metric, it evaluates the metric definition in a context that has two methods: @metric@ and @playground@. The @metric@ method creates a new @Vanity::Metric@ object, and evaluates the block in the context of that object, so when you see the metric definition using methods like @description@ or @model@, these are all invoked on the metric object itself.

A @Vanity::Metric@ object responds to @track!@ and increments a record in the Redis database (an _O(1)_ operation). It creates one record for each day, accumulating that day's count. When generating reports, the @values@ method fetches the values of all these keys (also _O(1)_).
A @Vanity::Metric@ object responds to @track!@ and increments a record in the database (an _O(1)_ operation). It creates one record for each day, accumulating that day's count. When generating reports, the @values@ method fetches the values of all these keys (also _O(1)_).

You can call @track!@ with a value higher than one, and it will increment the day's count by that value.

Any time you track a metric, the metric passes its identifier, timestamp and count (if more than zero) to all its hooks. "A/B tests":ab_testing.html use hooks to manage their own book keeping. When you define an experiment and tell it which metric(s) to use, the experiment registers itself by calling the @hook@ method.

When you call @model@ on a metric, this method changes the metric definition by rewriting the @values@ method to perform a query, rewriting the @track!@ method to update hooks but not Redis, and register an @after_create@ callback that updates the hooks.
When you call @model@ on a metric, this method changes the metric definition by rewriting the @values@ method to perform a query, rewriting the @track!@ method to update hooks but not the database, and register an @after_create@ callback that updates the hooks.

How about some tips & tricks for getting the most out of metrics (you might call them "best practices")? Got any to share?
1 change: 0 additions & 1 deletion lib/vanity.rb
Expand Up @@ -35,6 +35,5 @@ module Version
# Playground.
require "vanity/playground"
require "vanity/helpers"
Vanity.autoload :Commands, "vanity/commands"
# Integration with various frameworks.
require "vanity/frameworks/rails" if defined?(Rails)
75 changes: 35 additions & 40 deletions lib/vanity/adapters/redis_adapter.rb
Expand Up @@ -2,17 +2,17 @@ module Vanity
module Adapters
class << self
def redis_connection(spec)
require "redis"
require "redis/namespace"
RedisAdapter.new(spec)
end
end

class RedisAdapter < AbstractAdapter
def initialize(options)
@options = options.clone
@options[:db] = options[:database] || (options[:path] && options[:path].split("/")[1].to_i)
@options[:db] = @options[:database] || (@options[:path] && @options[:path].split("/")[1].to_i)
@options[:thread_safe] = true
@redis = options[:redis] || ::Redis.new(@options)
connect!
end

def active?
Expand All @@ -26,7 +26,13 @@ def disconnect!

def reconnect!
disconnect!
@redis = ::Redis.new(@options)
connect!
end

def connect!
@redis = @options[:redis] || Redis.new(@options)
@metrics = Redis::Namespace.new("vanity:metrics", :redis=>@redis)
@experiments = Redis::Namespace.new("vanity:experiments", :redis=>@redis)
end

def to_s
Expand All @@ -44,106 +50,95 @@ def flushdb
# -- Metrics --

def set_metric_created_at(metric, time)
@redis.setnx metric_key(metric, :created_at), time.to_i
@metrics.setnx "#{metric}:created_at", time.to_i
end

def get_metric_created_at(metric)
created_at = @redis[metric_key(metric, :created_at)]
created_at = @metrics["#{metric}:created_at"]
created_at && Time.at(created_at.to_i)
end

def metric_track(metric, time, count = 1)
@redis.incrby metric_key(metric, time.to_date, "count"), count
@metrics.incrby "#{metric}:#{time.to_date}:count", count
end

def metric_values(metric, from, to)
@redis.mget(*(from.to_date..to.to_date).map { |date| metric_key(metric, date, "count") }) || []
@metrics.mget(*(from.to_date..to.to_date).map { |date| "#{metric}:#{date}:count" }) || []
end

def destroy_metric(metric)
@redis.del *@redis.keys(metric_key(metric, "*"))
@metrics.del *@metrics.keys("#{metric}:*")
end

# -- Experiments --

def set_experiment_created_at(experiment, time)
@redis.setnx ab_key(experiment, :created_at), time.to_i
@experiments.setnx "#{experiment}:created_at", time.to_i
end

def get_experiment_created_at(experiment)
created_at = @redis[ab_key(experiment, :created_at)]
created_at = @experiments["#{experiment}:created_at"]
created_at && Time.at(created_at.to_i)
end

def set_experiment_completed_at(experiment, time)
@redis.setnx ab_key(experiment, :completed_at), time.to_i
@experiments.setnx "#{experiment}:completed_at", time.to_i
end

def get_experiment_completed_at(experiment)
completed_at = @redis[ab_key(experiment, :completed_at)]
completed_at = @experiments["#{experiment}:completed_at"]
completed_at && Time.at(completed_at.to_i)
end

def is_experiment_completed?(experiment)
@redis.exists(ab_key(experiment, :completed_at))
@experiments.exists("#{experiment}:completed_at")
end

def ab_counts(experiment, alternative)
{ :participants => @redis.scard(ab_key(experiment, "alts", alternative, "participants")).to_i,
:converted => @redis.scard(ab_key(experiment, "alts", alternative, "converted")).to_i,
:conversions => @redis[ab_key(experiment, "alts", alternative, "conversions")].to_i }
{ :participants => @experiments.scard("#{experiment}:alts:#{alternative}:participants").to_i,
:converted => @experiments.scard("#{experiment}:alts:#{alternative}:converted").to_i,
:conversions => @experiments["#{experiment}:alts:#{alternative}:conversions"].to_i }
end

def ab_show(experiment, identity, alternative)
@redis[ab_key(experiment, "participant", identity, "show")] = alternative
@experiments["#{experiment}:participant:#{identity}:show"] = alternative
end

def ab_showing(experiment, identity)
alternative = @redis[ab_key(experiment, "participant", identity, "show")]
alternative = @experiments["#{experiment}:participant:#{identity}:show"]
alternative && alternative.to_i
end

def ab_not_showing(experiment, identity)
@redis.del ab_key(experiment, "participant", identity, "show")
@experiments.del "#{experiment}:participant:#{identity}:show"
end

def ab_add_participant(experiment, alternative, identity)
@redis.sadd ab_key(experiment, "alts", alternative, "participants"), identity
@experiments.sadd "#{experiment}:alts:#{alternative}:participants", identity
end

def ab_add_conversion(experiment, alternative, identity, count = 1, implicit = false)
if implicit
@redis.sadd ab_key(experiment, "alts", alternative, "participants"), identity
@experiments.sadd "#{experiment}:alts:#{alternative}:participants", identity
else
participating = @redis.sismember(ab_key(experiment, "alts", alternative, "participants"), identity)
participating = @experiments.sismember("#{experiment}:alts:#{alternative}:participants", identity)
end
@redis.sadd ab_key(experiment, "alts", alternative, "converted"), identity if implicit || participating
@redis.incrby ab_key(experiment, "alts", alternative, "conversions"), count
@experiments.sadd "#{experiment}:alts:#{alternative}:converted", identity if implicit || participating
@experiments.incrby "#{experiment}:alts:#{alternative}:conversions", count
end

def ab_get_outcome(experiment)
alternative = @redis[ab_key(experiment, :outcome)]
alternative = @experiments["#{experiment}:outcome"]
alternative && alternative.to_i
end

def ab_set_outcome(experiment, alternative = 0)
@redis.setnx ab_key(experiment, :outcome), alternative
@experiments.setnx "#{experiment}:outcome", alternative
end

def destroy_experiment(experiment)
@redis.del ab_key(experiment, :outcome), ab_key(experiment, :created_at), ab_key(experiment, :completed_at)
@redis.del *@redis.keys(ab_key(experiment, "alts:*"))
end

protected

def metric_key(metric, *args)
"metrics:#{metric}:#{args.join(':')}"
end

def ab_key(experiment, *args)
base = "vanity:#{Vanity::Version::MAJOR}:#{experiment}"
args.empty? ? base : "#{base}:#{args.join(":")}"
@experiments.del "#{experiment}:outcome", "#{experiment}:created_at", "#{experiment}:completed_at"
@experiments.del *@experiments.keys("#{experiment}:alts:*")
end

end
Expand Down
2 changes: 0 additions & 2 deletions lib/vanity/commands.rb

This file was deleted.

33 changes: 33 additions & 0 deletions lib/vanity/commands/upgrade.rb
@@ -0,0 +1,33 @@
module Vanity
module Commands
class << self
# Upgrade to newer version of Vanity (this usually means doing magic in
# the database)
def upgrade
if Vanity.playground.connection.respond_to?(:redis)
redis = Vanity.playground.connection.redis
# Upgrade metrics from 1.3 to 1.4
keys = redis.keys("metrics:*")
if keys.empty?
puts "No metrics to upgrade"
else
puts "Updating #{keys.map { |name| name.split(":")[1] }.uniq.length} metrics"
keys.each do |key|
redis.renamenx key, "vanity:#{key}"
end
end
# Upgrade experiments from 1.3 to 1.4
keys = redis.keys("vanity:1:*")
if keys.empty?
puts "No experiments to upgrade"
else
puts "Updating #{keys.map { |name| name.split(":")[2] }.uniq.length} experiments"
keys.each do |key|
redis.renamenx key, key.gsub(":1:", ":experiments:")
end
end
end
end
end
end
end
6 changes: 3 additions & 3 deletions lib/vanity/metric/base.rb
Expand Up @@ -4,9 +4,9 @@ module Vanity
# can also respond to addition methods (+track!+, +bounds+, etc), these are
# optional.
#
# This class implements a basic metric that tracks data and stores it in
# Redis. You can use this as the basis for your metric, or as reference for
# the methods your metric must and can implement.
# This class implements a basic metric that tracks data and stores it in the
# database. You can use this as the basis for your metric, or as reference
# for the methods your metric must and can implement.
#
# @since 1.1.0
class Metric
Expand Down
4 changes: 2 additions & 2 deletions test/test_helper.rb
Expand Up @@ -27,7 +27,7 @@ def setup
end

# Call this on teardown. It wipes put the playground and any state held in it
# (mostly experiments), resets vanity ID, and clears Redis of all experiments.
# (mostly experiments), resets vanity ID, and clears database of all experiments.
def nuke_playground
new_playground
Vanity.playground.connection.flushdb
Expand Down Expand Up @@ -71,7 +71,7 @@ def experiment(name)
def teardown
Vanity.context = nil
FileUtils.rm_rf "tmp"
Vanity.playground.connection.flushdb if Vanity.playground.connection
Vanity.playground.connection.flushdb if Vanity.playground.connected?
end

end
Expand Down
1 change: 1 addition & 0 deletions vanity.gemspec
Expand Up @@ -17,4 +17,5 @@ Gem::Specification.new do |spec|
"--webcvs", "http://github.com/assaf/#{spec.name}"

spec.add_dependency "redis", "~>2.0"
spec.add_dependency "redis-namespace", "~>0.7"
end

0 comments on commit 5ed70eb

Please sign in to comment.