Bloody simple A/B testing for Python WSGI applications:
- Present display or behavioral differences in one line of code.
- Measure and compare conversions amongst multiple variants, including statistical significance.
- Guarantee the same experience for returning users.
- Integrates easily with existing authentication and storage layers.
Starting an Experiment
Starting a new A/B test is easy. Use this code anywhere within the context of an HTTP request (like a controller or a template) to start automatically segmenting visitors:
cleaver = request.environ['cleaver'] # Start a new A/B experiment, returning True or False show_promo = cleaver('show_promo') # ...later, when the user completes the experiment, score the conversion... cleaver.score('show_promo')
Cleaver can also be used to specify an arbitrary number of variants:
cleaver = request.environ['cleaver'] # Start a new A/B experiment, returning one of several options background_color = cleaver( 'background_color', ('Red', '#F00'), ('Green', '#0F0'), ('Blue', '#00F') )
Maybe you only want to present an experimental change to a small portion of your user base. Variant weights make this simple - just add a third integer argument to each variant.
cleaver = request.environ['cleaver'] background_color = cleaver( 'show_new_experimental_feature', ('True', True, 1), ('False', False, 9) )
The default weight for variants, when left unspecified, is 1.
Adding Cleaver to Your WSGI Application
Cleaver works out of the box with most WSGI frameworks. To get started, wrap
any WSGI application with
cleaver.SplitMiddleware. For example:
from cleaver import SplitMiddleware from cleaver.backend.db import SQLAlchemyBackend def simple_app(environ, start_response): # Get the session object from the environ cleaver = environ['cleaver'] button_size = cleaver( 'Button Size', ('Small', 12), ('Medium', 18), ('Large', 24) ) start_response('200 OK', [('Content-type', 'text/html')]) return ['<button style="font-size:%spx;">Sign Up Now!</button>' % button_size] wsgi_app = SplitMiddleware( simple_app, lambda environ: environ['REMOTE_ADDR'], # Track by IP for examples' sake SQLAlchemyBackend('sqlite:///experiment.data') )
cleaver.SplitMiddleware requires an identity and backend adaptor (for
recognizing returning visitors and storing statistical data). Luckily, Cleaver
comes with a few out of the box, such as support for Beaker
sessions, and storage via
SQLAlchemy. Implementing your own is easy too;
just have a look at the full documentation .
For QA and testing purposes, you may need to force your application to always return a certain variant.
wsgi_app = SplitMiddleware( simple_app, ... allow_override=True )
If your application has an experiment called
button_size with variants
large, a url in the format:
..will always display small buttons. This data won't, however, count towards reporting.
By default, Cleaver does not differentiate between robots (such as Googlebot) and living, breathing, browser-driving humans.
This can cause participants well in excess of the number of actual humans who come to your site (if your A/B tests are publicly visible), and since bots don't generally convert, can result in skewed conversion metrics.
To combat this, Cleaver can be configured to defer counting of visitors as participants until after they've proven they're probably not a bot:
wsgi_app = SplitMiddleware( simple_app, ... count_humans_only=True )
...and then, on every page that presents an A/B decision, add the following
call directly prior to the closing
</body> tag (adapted to your templating
language of choice):
Cleaver comes with a lightweight WSGI front end which can be used to see how your experiments are going.
from cleaver.reports.web import CleaverWebUI from cleaver.backend.db import SQLAlchemyBackend wsgi_app = CleaverWebUI( SQLAlchemyBackend('sqlite:///experiment.data') ) from wsgiref import simple_server simple_server.make_server('', 8000, wsgi_app).serve_forever()
Tests can be run with
python setup.py test or
All contributions must:
- Include accompanying tests.
- Include narrative and API documentation if new features are added.
- Be (generally) compliant with PEP8.
- Not break the tests or build. Before issuing a pull request,
$ pip install tox && toxfrom your source to ensure that all tests still pass across multiple versions of Python.
- Add your name to the (bottom of the) AUTHORS file.