Skip to content

Commit

Permalink
Added ToC on various pages
Browse files Browse the repository at this point in the history
  • Loading branch information
assaf committed Mar 2, 2010
1 parent 6f0ad3c commit 48ef436
Show file tree
Hide file tree
Showing 6 changed files with 58 additions and 26 deletions.
19 changes: 14 additions & 5 deletions doc/ab_testing.textile
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,23 @@ layout: page
title: A/B Testing
---

<div id="toc">
# "True or False":#tf
# "Interpreting the Results":#interpret
# "Multiple Alternatives":#multiple
# "A/B Testing and Code Testing":#test
# "Let the Experiment Decide":#decide
</div>


"A/B testing":http://en.wikipedia.org/wiki/A/B_testing (or "split testing") are experiments you can run to compare the performance of different alternatives. A classical example is using an A/B test to compare two versions of a landing page, to find out which alternative leads to more registrations.

You can use A/B tests to gauge interest in a new feature, response to a feature change, improve the site's design and copy, and so forth. In spite of the name, you can use A/B tests to check out more than two alternatives.

bq. "If you are not embarrassed by the first version of your product, you’ve launched too late" -- Reid Hoffman, founder of LinkedIn


h3. True or False
h3(#tf). True or False

Let's start with a simple experiment. We have this idea that a bigger sign-up link will increase the number of people who sign up for our service. Let's see how well our hypothesis holds.

Expand Down Expand Up @@ -56,7 +65,7 @@ end
</pre>


h3. Interpreting the Results
h3(#interpret). Interpreting the Results

We're going to let the experiment run for a while and track the results using "the dashboard":dashboard.html, or by running the command @vanity report@.

Expand Down Expand Up @@ -85,7 +94,7 @@ You might want to instead decide on some target probability (which could change
You'll want to read more about "A/B testing and statistical significance":http://www.cennydd.co.uk/2009/statistical-significance-other-ab-test-pitfalls/


h3. Multiple Alternatives
h3(#multiple). Multiple Alternatives

Your A/B tests can have as many alternatives as you care, with two caveats. The more alternatives you have the larger the sample size you need, and so the longer it will take to find out the outcome of your experiment. You want alternatives that are significantly different from each other, testing two pricing options at $5 and $25 is fast, testing all the prices between $5 and $25 at $1 increments will take a long time to reach any conclusive result.

Expand Down Expand Up @@ -129,7 +138,7 @@ end
</pre>


h3. A/B Testing and Code Testing
h3(#test). A/B Testing and Code Testing

If you're presenting more than one alternative to visitors of your site, you'll want to test more than one alternative. Don't let A/B testing become A/broken.

Expand Down Expand Up @@ -162,7 +171,7 @@ You'll also want to test each alternative visually, from your Web browser. For
Once the experiment is over, simply remove its definition from the experiments directory and run the test suite again. You'll see errors in all the places that touch the experiment (from failing to load it), pointing you to what parts of the code you need to remove/change.


h3. Let the Test Decide
h3(#decide). Let the Experiment Decide

Sample size and probability help you interpret the results, you can also use them to configure an experiment to automatically complete itself.

Expand Down
18 changes: 13 additions & 5 deletions doc/contributing.textile
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,26 @@ layout: page
title: Contributing
---

<div id="toc">
# "Relevant Links":#link
# "Building From Source":#building
# "Documentation":#doc
# "Open Issues":#open
</div>

By all means.

h3. Relevant Links

h3(#link). Relevant Links

* "Google Group":http://groups.google.com/group/vanity-talk
* "Source code":http://github.com/assaf/vanity
* "Continuous integration":http://runcoderun.com/assaf/vanity
* "Code metrics":http://devver.net/caliper/project?repo=git%3A%2F%2Fgithub.com%2Fassaf%2Fvanity.git
* "Open Issues":http://github.com/assaf/vanity/issues
* "Issues":http://github.com/assaf/vanity/issues


h3. Building From Source
h3(#building). Building From Source

To make life easier, you can use Bundler with the included Gemfile to load all the necessary development dependencies.

Expand All @@ -26,7 +34,7 @@ $ bundle exec rake test
</pre>


h3. Documentation
h3(#doc). Documentation

Documentation is written in "Textile":http://redcloth.org/textile/writing-paragraph-text/, and converted to HTML using "Jekyll":http://jekyllrb.com/.

Expand All @@ -40,7 +48,7 @@ $ open html/index.html
</pre>


h3. Open Issues
h3(#open). Open Issues

<notextile>
<table id="issues">
Expand Down
4 changes: 4 additions & 0 deletions doc/css/page.css
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,7 @@ table#issues .votes { color: #ccc; float: right }
#sidebar ul { list-style: none; text-align: right; margin: 0 0 3em 0; padding: 0 }
#content { padding: 1.35em 0 }
#footer { clear: both; font-size: .9em; padding: 0; margin-top: 3em; border-top: 1px solid #ccc; text-align: right }

#toc ol { margin: 1em 0 2em 0.6em; list-style-type: disc; padding-left: 1em; font-size: 0.9em }
#toc li { margin: .3em 0 }

2 changes: 1 addition & 1 deletion doc/index.textile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ h3. Reading Order
* "A/B Testing(Everything you need to know)":ab_testing.html
* "The Dashboard(Setting up and using the Rails dashboard)":dashboard.html
* "Managing Identity":identity.html
* "Configuring Vanity":configuring.html
* "Using with Rails":rails.html

Also:

Expand Down
22 changes: 14 additions & 8 deletions doc/metrics.textile
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ layout: page
title: Metrics
---

<div id="toc">
# "Defining a Metric":#define
# "Metrics From Your Database":#ar
# "Google Analytics":#ga
# "Creating Your Own Metric":#custom
# "Digging Deeper":#deeper
</div>


A good starting point for improving -- on anything -- is measuring. Vanity allows you to measure multiple metrics, best way to tell how well your experiments are doing.

<blockquote>
Expand All @@ -16,7 +25,7 @@ A good starting point for improving -- on anything -- is measuring. Vanity allo



h3. Defining a Metric
h3(#define). Defining a Metric

Vanity always loads metrics defined in the @experiments/metrics@ directory. A metric definition is a Ruby file that looks like this:

Expand Down Expand Up @@ -63,7 +72,7 @@ Calling @track!@ with no value is the same as calling with one, and for convenie
Define, track, and you're ready to roll.


h3. Metrics From Your Database
h3(#ar). Metrics From Your Database

If you already have the data, why not use it?

Expand Down Expand Up @@ -125,7 +134,7 @@ end
</pre>


h3. Google Analytics
h3(#ga). Google Analytics

You can easily include Google Analytics metrics in your Vanity dashboard. You'll need, in addition to Vanity, to use "Garb":http://github.com/vigetlabs/garb, a Ruby wrapper for the Google Analytics API.

Expand Down Expand Up @@ -170,7 +179,7 @@ end
See "the Garb documentation":http://rdoc.info/projects/vigetlabs/garb and "Google Analytics API":http://code.google.com/apis/analytics/docs/gdata/gdataReferenceDimensionsMetrics.html#bounceRate for more details.


h3. Creating Your Own Metric
h3(#custom). Creating Your Own Metric

Got other ideas for metrics? Writing your own metric is fairly simple.

Expand Down Expand Up @@ -201,7 +210,7 @@ A metric may also implement these methods:
If you wrote your own metric implementation, please consider "contributing it to Vanity":contributing.html so we can all benefit from it. Thanks.


h3. Digging Deeper
h3(#deeper). Digging Deeper

All metrics are listed in @Vanity.playground.metrics@, a hash that maps metric identifier to metric object. Methods like @track!@ and @metrics@ (see "A/B tests":ab_testing.html) reference metrics using their identifier.

Expand All @@ -219,7 +228,4 @@ Any time you track a metric, the metric passes its identifier, timestamp and cou

When you call @model@ on a metric, this method changes the metric definition by rewriting the @values@ method to perform a query, rewriting the @track!@ method to update hooks but not Redis, and register an @after_create@ callback that updates the hooks.


h3. Metrics That Work

How about some tips & tricks for getting the most out of metrics (you might call them "best practices")? Got any to share?
19 changes: 12 additions & 7 deletions doc/rails.textile
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,16 @@ layout: page
title: Using with Rails
---

This guide was written for Rails 2.3.5. If you have any tips for Rails 3.0, please share.
<div id="toc">
# "Configuring Vanity":#config
# "Test Environment":#test
# "Dashboard":#dashboard
# "Unicorn and Forking Servers":#fork
</div>

h3. Configuring Vanity
This guide is written for Rails 2.3.5. If you have any tips for Rails 3.0, please share.

h3(#config). Configuring Vanity

Start by telling Rails to use the Vanity gem, either using @config.gem "vanity"@ or by adding @gem "vanity"@ to your Gemfile.

Expand Down Expand Up @@ -38,8 +45,7 @@ end
</pre>



h3. Test Environment
h3(#test). Test Environment

As your tests run, any metric and experiment data is stored in Redis. You might not appreciate tests filling up Redis with useless data, and there are two things you can do about it.

Expand All @@ -59,8 +65,7 @@ Vanity.playground.redis.flushdb # Do this before running tests
</pre>


h3. Dashboard

h3(#dashboard). Dashboard

Start by adding a new resource in @config/routes.rb@:

Expand All @@ -81,7 +86,7 @@ Now open your browser to "http://localhost:3000/vanity":http://localhost:3000/va
The Dashboard renders complete HTML pages with CSS and all necessary JavaScript libraries. Thankfully, HTML is forgiving enough that it will render correctly even with your existing application layout. You can decide to keep your layout, or tell the controller to set @layout false@.


h3. Unicorn and Forking Servers
h3(#fork). Unicorn and Forking Servers

Unicorn forks the master process to create worker processes efficiently. Since the master processes opens a connection to Redis, all workers end up sharing that connection, resulting in ugly contention issues.

Expand Down

0 comments on commit 48ef436

Please sign in to comment.