This wiki page is meant to collect some tips on using or extending AlephBet, as well as caveats or things to be aware of or avoid.

Whilst you can create and run AlephBet experiment anywhere, it's probably best to wrap your experiments and goals inside $(document).ready(function() {...}); or similar, to make sure that the DOM was loaded before running experiments. For experiments, this would avoid trying to modify some elements on the page before it's there. For goals that can run without waiting (e.g. goals that check visits to a particular page), it's important to ensure the experiment started running. Otherwise, the goal completion will be lost.

An example:

import {Experiment, Goal} from "alephbet"

const visited_homepage = new Goal('visited home page');

// defining the experiment after DOM was loaded
$(function() {
  const experiment = new Experiment({ /* ... */ });
  experiment.add_goal(visited_homepage);
});

// BAD: this might complete before the experiment was linked to the goal
if (window.location.href.match(/\/$/)) visited_homepage.complete();

Wrapping both goals and experiments inside the ready state function should fit most use-cases and be safe. So the example above should look like this:

import {Experiment, Goal} from "alephbet"

// defining the experiment after DOM was loaded
$(function() {
  const visited_homepage = new Goal('visited home page');
  const experiment = new Experiment({ /* ... */ });
  experiment.add_goal(visited_homepage);
  if (window.location.href.match(/\/$/)) visited_homepage.complete();
});

(see Persistent Queue Adapters below for a potentially more reliable solution)

In some cases, you'd like to redirect users assigned to one variant to a different page. Say, one variant shows the regular registration page, and another variant shows your new one. In this case, you'd want to do the following:

1. Redirect as fast as possible. i.e. In this case, don't wait for DOM load to complete (see above).
2. Whilst you do want to redirect fast, you don't want to lose the tracking event. Otherwise, your experiment will show less participants than you actually had. When you redirect, your tracking adapter might not have a chance to send the event.
3. To make sure users won't see a flash of the page you're redirecting from, you should try to hide the content. However fast the redirect happens, the browser keeps loading the page.

Here's an example snippet explaining how to deal with this

import {Experiment} from "alephbet"

const hide_page = function() {
  // note: we're using plain-javascript here to avoid waiting for jQuery to load
  var style = document.createElement('style');
  style.setAttribute('type', 'text/css');
  style.innerHTML = 'body{ display:none; visibility:hidden; }';
  document.head.appendChild(style);
};

const redirect_experiment = new Experiment({
  name: 'my redirect experiment',
  variants: {
    original: {
      activate: function() { /* do nothing */ }
    },
    redirect: {
      activate: function(experiment) {
        let timeout = 0;
        // check if this is the first time we activate this user
        if (typeof experiment.get_stored_variant() === 'undefined') {        
          // we allow 1 second delay to make sure the tracking event gets sent
          timeout = 1000;
        }
        setTimeout(function () { window.location = '/new/page'; }, timeout);
        hide_page();
      }
    }
  }
});

Note that these adapters are still experimental, and hence not the default for AlephBet. They do however give much stronger eventual data consistency and should increase accuracy.

Take a look at AlephBet's sister project: Gimel - an open-source, self-hosted A/B testing backend using AWS Lambda + Redis HyperLogLog.

Version 0.8.0 introduced a new (optional) tracking adapter: PersistentQueueGoogleAnalyticsAdapter which can help reduce the number of events that might get "lost", primarily in cases like button clicks or redirects. Note that this adapter works with Google Analytics, but a similar adapter can be developed for other trackers.

Before an event gets sent to the tracker, it is stored in a queue with a dynamic unique id (uuid). When events are flushed from the queue (either immediately or when the adapter is loaded), the queue is checked, and events are only cleared once a callback is triggered. Callbacks are registered for each uuid independently.

This ensures that events won't get lost and are guaranteed to arrive to the tracker.

To use this adapter, simply use something like this for your experiment:

import {Experiment, PersistentQueueGoogleAnalyticsAdapter} from "alephbet"

const myExperiment = new Experiment({
  name: 'my experiment',
  tracking_adapter: new PersistentQueueGoogleAnalyticsAdapter,
  // ...
});

The downside for using this adapter is that whilst events won't get lost, there's a chance that events will be recorded more than once. This might happen if the callback fails to trigger before a redirect, for example. To ensure you see accurate event counts on Google Analytics, you can use the Unique Events count though, so any duplicates will be consolidated by Google Analytics for you.

Version 0.9.0 introduced a new (optional) tracking adapter: PersistentQueueKeenAdapter, similar to the persistent queue adapter for Google.

To use this adapter, use something like this for your experiment:

import {Experiment, PersistentQueueKeenAdapter} from "alephbet"

var keen_client = new Keen({ /* your keen.io projectId and writeKey */});

var myExperiment = new Experiment({
  name: 'my experiment',
  tracking_adapter: new PersistentQueueKeenAdapter(keen_client),
  // ...
});

The adapter will create an event collection for each experiment (using the experiment name). The event data will include:

{"variant": "...",  // name of the variant picked
 "event": "...",  // goal name OR 'participate'
 "uuid": "..."  // unique id for each event
}

The uuid field should be used to report data from keen.io using count_unique. Note that it should work reliably both with unique goals and non-unique goals. Alephbet will generate a new event for goals which are set as unique: false for you.