Skip to content
πŸ§ͺ Source-controlled split testing stack for building, launching and analysing A/B tests.
Branch: master
Clone or download
kingo55 Merge pull request #2 from mint-metrics/adding-docs-links
Add links to new documentation site & blog post
Latest commit ecc01ba Sep 30, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LICENSE Modifying license Jul 4, 2019
README.md Add links to new documentation site & blog post Sep 27, 2019
logo.png Init Jul 11, 2019
modules.png Init Jul 11, 2019
reports.png Init Jul 11, 2019

README.md

Mojito experimentation framework

Mojito

A modular, source-controlled split testing framework that lets you build, launch and analyse experiments via Git/CI.

View documentation | Read intro blog post

It's comprised of 3 core modules e.g.:

  1. Mojito JS Delivery: Front-end library for running experiments on your site.
  2. Mojito Snowplow Storage: Data models & events for tracking experiments.
  3. Mojito R Analytics: Templatable RMarkdown experiment reports.

Mojito's 3 components

Features

  • Under 5kb minified & gzipped
  • Define experiments with simple JS or YAML
  • Self-hosted & git-controlled for familiar code review / merging
  • Expressive trigger system & utilities
  • Variant code (JS/CSS) minification & linting
  • Track and handle JS errors caused by your variant code

Mojito vs. [vendor]

Differentiating features between popular vendors' tools and Mojito out of the box:

Feature Optimizely X Google Optimize Mojito
Open-source license ❌ ❌ βœ… BSD3
Light front-end codebase * ❌~80kb ❗~25kb βœ…<5kb
Git source control & CI ❌ ❌ βœ…
Variant error-tracking/handling ❌ ❌ βœ…
Auto CSS/JS minification ❗(not custom code) ❓ βœ…
Self-hosted ❗ (for a fee) ❗(via API) βœ…
Data ownership ❗(via S3 export) ❗(via 360/BigQuery) βœ…
Retroactively add new metrics βœ… ❗(360 only) βœ…
Server-side/App testing βœ… ❗(via API) ❗(via Storage)
WYSIWYG test editor βœ… βœ… ❌

* Tested 2019-07-05

Getting started

Mojito consists of three components, which are often switched out in the course of Mint Metrics' client services:

  1. Delivery: Front-end libraries to reliably control which treatments users are exposed to. e.g. Mojito JS Delivery
  2. Storage: Data collection modules and data modelling steps to power your reports. e.g. Mojito Snowplow Storage
  3. Analytics: Tools to measure & report on the effects caused by your treatments. e.g. Mojito R Analytics

Get up and running quickly with the README files inside each section.

Example experiment

Using Mojito's CI tools, you can set up experiments in YAML & JS:

id: ex1
name: Example test 1
state: live
sampleRate: 0.75
trigger: trigger.js
recipes:
  0:
    name: Original
  1:
    name: Variant
    js: variant.js
    css: variant.css

Where trigger.js activates the experiment when a condition is met and a callback to activate is fired:

function trigger(test) {
    if (document.location.pathname === '/') test.activate();
}

Upon activation, the will include 75% of traffic (sampleRate: 0.75) and split it 50-50 between "Original" and "Variant" groups.

For users assigned to the "Variant" group, we execute a) variant.js and b) variant.css files to transform the page through a a) JS function and b) CSS stylesheet respectively.

After you've defined an experiment YAML...

Run the Gulp pipeline to lint/test/publish your container.

  1. Install the necessary NPM packages: npm install
  2. Build & publish your testing container: gulp scripts-local && gulp publish

Example analytics reports

If you use our Snowplow/Redshift & R Analytics component for reporting, all your metrics can be reported on with a simple array of metrics.

wave_params <- list(
  client_id = "mintmetrics",
  wave_id = "ex1",
  start_date = "2019-05-15 09:19:45",
  stop_date = "2019-06-05 14:29:00",
  time_grain = "hours",
  subject = "usercookie",
  recipes = c("Original", "Variant")
)

goalList <- list(
  list(
    title = "Transactions",
    goal = "purchase",
    operand = "="
  ),
  list(
    title = "Thankyou page views",
    goal = "page_view /contact/thank-you%",
    operand = "like"
  )
)
goalList <- mojitoFullKnit(wave_params, goal_list = goalList)

For this experiment, we'll report on transactions and page views:

Measuring the performance of a treatment relative to the control group in Mojito.

Support for other analytics back-ends

You don't exactly need Snowplow Analytics to use Mojito. You can also track experiments to wherever you like, via a custom storage adapter. E.g. To Google Tag Manager, Adobe etc.

You can even hook Mojito Delivery up to Google Optimize's reports for free.

Credits

Our Delivery JS library is a heavily modified fork of the excellent jamesyu/cohorts lib. Meanwhile we employ heavy use of the Snowplow Analytics event pipeline for our Storage component and RStudio/Knitr for our Analytics reports.

Getting involved

We would love to see PRs! We're able to assist if you hit any snags getting set up.

Reach out to us via:

Learn more

Read the documentation and get Mojito set up. We recommend starting with Mojito JS Delivery.

You can’t perform that action at this time.