ab testing

A/B Testing Guide

This document describes how to use and test the A/B testing framework in Open Library.

1. Configuration

Active experiments are defined in openlibrary/core/experiments.py under the ACTIVE_EXPERIMENTS dictionary.

Every active experiment must declare its variants nested under a "variants" key, along with a targeting rule under the "audience" key:

from openlibrary.core.experiments import Audience

ACTIVE_EXPERIMENTS = {
    "AB_Testing": {
        "variants": {
            "control": 30,
            "a": 35,
            "b": 35,
        },
        "audience": Audience.ALL,  # Target everyone
    },
    "Another_Experiment": {
        "variants": {
            "control": 50,
            "treatment": 50,
        },
        "audience": Audience.LOGGED_IN,  # Logged-out users automatically fall back to control
    }
}

Audience Targeting Options

The framework supports three audience categories imported from openlibrary.core.experiments.Audience:

• Audience.ALL: All requests are split according to the variant weights.
• Audience.LOGGED_IN: Only logged-in users are distributed into the variants; guest users automatically fall back to the control variant.
• Audience.LOGGED_OUT: Only guest (anonymous) users are distributed into the variants; logged-in users automatically fall back to the control variant.

Note

The legacy flat configuration format (defining weights directly under the experiment name key without a nested "variants" key) is no longer supported. The engine requires "variants" to resolve targeting weights.

---

2. Usage Examples

Inside HTML Templates / Templetor Macros

To conditionally render markup based on a user's variant bucket:

$code: variant = ctx.get('experiments', {}).get('AB_Testing') $if variant ==
'a':
<div class="layout-a">
  <!-- Variant A layout -->
</div>
$elif variant == 'b':
<div class="layout-b">
  <!-- Variant B layout -->
</div>
$else:
<div class="layout-control">
  <!-- Control layout -->
</div>

Inside Python Controllers / Handlers

Open Library runs both legacy web.py and async FastAPI routes. The active experiments are made available in their respective request contexts:

In web.py Controllers

Access context-scoped experiments via web.ctx:

import web

variant = web.ctx.get("experiments", {}).get("AB_Testing")
if variant == "a":
    # Execute Variant A logic
    pass
elif variant == "b":
    # Execute Variant B logic
    pass
else:
    # Execute Control logic
    pass

In FastAPI Routes

Access experiments using the request state context (request.state):

from fastapi import Request

@app.get("/example")
async def read_example(request: Request):
    variant = request.state.experiments.get("AB_Testing")
    if variant == "a":
        # Execute Variant A logic
        pass
    elif variant == "b":
        # Execute Variant B logic
        pass
    else:
        # Execute Control logic
        pass

Inside Frontend JS (jQuery, Vue, Lit)

To check variants client-side, query the global window.getExperiment helper or import it:

// Anywhere in your scripts:
const variant = window.getExperiment("AB_Testing");
if (variant === "a") {
  // Enable variant A layout
} else if (variant === "b") {
  // Enable variant B layout
} else {
  // Enable control layout
}

// Or import as an ES module in modern Webpack / Vite bundles:
import { getExperiment } from "./experiments";
const variant = getExperiment("AB_Testing");

---

3. Testing & URL Overrides

For development, QA, and debugging, you can force a specific experiment variant using URL query parameters.

The format is: ?experiment_[experiment_name]=[variant]

Examples

• To force Variant b of the AB_Testing experiment: http://localhost:8080/?experiment_AB_Testing=b
• To force the control group: http://localhost:8080/?experiment_AB_Testing=control
• Overrides also work across multiple parameters: http://localhost:8080/?experiment_AB_Testing=a&experiment_AnotherTest=control

Note

URL overrides will only apply if the specified variant name exists in the configuration of the active experiment. Invalid override variants or incorrectly formatted parameters are safely ignored, falling back to the user's determinisnically assigned bucket.