Added: A/B test report now showing number of participants.

CHANGELOG

@@ -11,6 +11,8 @@ The experiment method is no longer overloaded: it looks up an experiment
 (loading its definition if necessary), but does not define an experiment.
 The ab_test method is overloaded, though this may change in the future.
 
+* Added: A/B test report now showing number of participants.
+* Added: AbTest.score method accepts minimum probability (default 90), and
 * Changed: Playground.define now requires an experiment type, ab_test is not
 the default any more.
 * Changed: When you run Vanity in development mode
@@ -20,7 +22,6 @@ request. You can also Vanity.playground.reload!.
 * Changed: You can break long experiment descriptions into multiple paragraphs
 using two consecutive newlines.
 * Changed: AbTest confidence becomes probability.
-* Changed: AbTest score method accepts minimum probability (default 90), and
 only returns choice alternative with probability equal or higher than that.
 
 0.3.1 (2009-11-13)

lib/vanity/experiment/ab_test.rb

@@ -274,6 +274,12 @@ def score(probability = 90)
       # array of claims.
       def conclusion(score = score)
         claims = []
+        participants = score.alts.inject(0) { |t,alt| t + alt.participants }
+        claims << case participants
+          when 0 ; "There are no participants in this experiment yet."
+          when 1 ; "There is one participant in this experiment."
+          else ; "There are #{participants} participants in this experiment."
+        end
         # only interested in sorted alternatives with conversion
         sorted = score.alts.select { |alt| alt.conversion_rate > 0.0 }.sort_by(&:conversion_rate).reverse
         if sorted.size > 1

lib/vanity/templates/_ab_test.erb

@@ -1,6 +1,7 @@
 <% score = experiment.score %>
 <table>
-  <caption><%= experiment.conclusion(score).join(" ") %></caption>
+  <caption>
+    <%= experiment.conclusion(score).join(" ") %></caption>
   <% score.alts.each do |alt| %>
     <tr class="<%= "choice" if score.choice == alt %>">
       <td class="option"><%= alt.name.gsub(/^o/, "O") %>:</td>

test/ab_test_test.rb

@@ -321,6 +321,7 @@ def test_conclusion
     61.times  { |i| experiment(:abcd).count i, :d, :conversion }
 
     assert_equal <<-TEXT, experiment(:abcd).conclusion.join("\n") << "\n"
+There are 739 participants in this experiment.
 The best choice is option D: it converted at 32.4% (30% better than option B).
 With 90% probability this result is statistically significant.
 Option B converted at 25.0%.
@@ -340,6 +341,7 @@ def test_conclusion_with_some_performers
     61.times  { |i| experiment(:abcd).count i, :d, :conversion }
 
     assert_equal <<-TEXT, experiment(:abcd).conclusion.join("\n") << "\n"
+There are 368 participants in this experiment.
 The best choice is option D: it converted at 32.4% (30% better than option B).
 With 90% probability this result is statistically significant.
 Option B converted at 25.0%.
@@ -359,6 +361,7 @@ def test_conclusion_without_clear_winner
     61.times  { |i| experiment(:abcd).count i, :d, :conversion }
 
     assert_equal <<-TEXT, experiment(:abcd).conclusion.join("\n") << "\n"
+There are 368 participants in this experiment.
 The best choice is option D: it converted at 32.4% (1% better than option B).
 This result is not statistically significant, suggest you continue this experiment.
 Option B converted at 32.2%.
@@ -377,6 +380,7 @@ def test_conclusion_without_close_performers
     61.times  { |i| experiment(:abcd).count i, :d, :conversion }
 
     assert_equal <<-TEXT, experiment(:abcd).conclusion.join("\n") << "\n"
+There are 374 participants in this experiment.
 The best choice is option D: it converted at 32.4%.
 This result is not statistically significant, suggest you continue this experiment.
 Option B converted at 32.3%.
@@ -395,6 +399,7 @@ def test_conclusion_without_equal_performers
     61.times  { |i| experiment(:abcd).count i, :d, :conversion }
 
     assert_equal <<-TEXT, experiment(:abcd).conclusion.join("\n") << "\n"
+There are 376 participants in this experiment.
 Option D converted at 32.4%.
 Option B converted at 32.4%.
 Option A did not convert.
@@ -408,12 +413,18 @@ def test_conclusion_with_one_performers
     180.times { |i| experiment(:abcd).count i, :b, :participant }
     45.times  { |i| experiment(:abcd).count i, :b, :conversion }
 
-    assert_equal "This experiment did not run long enough to find a clear winner.", experiment(:abcd).conclusion.join("\n")
+    assert_equal <<-TEXT, experiment(:abcd).conclusion.join("\n") << "\n"
+There are 180 participants in this experiment.
+This experiment did not run long enough to find a clear winner.
+    TEXT
   end
 
   def test_conclusion_with_no_performers
     ab_test(:abcd) { alternatives :a, :b, :c, :d }
-    assert_equal "This experiment did not run long enough to find a clear winner.", experiment(:abcd).conclusion.join("\n")
+    assert_equal <<-TEXT, experiment(:abcd).conclusion.join("\n") << "\n"
+There are no participants in this experiment yet.
+This experiment did not run long enough to find a clear winner.
+    TEXT
   end