ext-ab-testing

A/B tests for Chrome extensions with Manifest V3: assign, persist and kill experiments with remote config, without a store re-release.

Used in production by PerfectPixel, a Chrome extension with 350,000+ users.

• Typed experiments: variants carry their behaviour as payload; call sites read variant.color, never compare ids
• Stable assignment: the first resolve picks uniformly at random and persists; every later resolve (any tab, any session) returns the same variant
• Remote kill switch: turn an experiment off remotely and every user falls back to the default behaviour
• Ship the winner: pin every user to the winning arm from the same config, while the code change waits for a store release
• Zero dependencies, environment-free core: you inject storage; works in any extension context

Install

npm i ext-ab-testing

Can be used with TypeScript >= 5.0 and JavaScript codebases.

Quick start

import { defineExperiment, createAbTestingClient } from 'ext-ab-testing';
import { chromeLocalStorage } from 'ext-ab-testing/chrome';

const BUTTON_COLOR = defineExperiment({
    key: 'BUTTON_COLOR_1',
    variants: [
        { id: 'GREEN', color: '#27ae60' }, // 50% of users will be assigned to this variant
        { id: 'BLUE', color: '#2980b9' } // 50% of users will be assigned to this variant
    ]
});

const ab = createAbTestingClient({ storage: chromeLocalStorage });

const { variant } = await ab.resolve(BUTTON_COLOR);
button.style.background = variant.color; // typed: '#27ae60' | '#2980b9'

No config option passed to createAbTestingClient means the client is ungated: every experiment is simply on (see Remote config (kill switch)). Assignments persist in chrome.storage.local under AB_TEST.BUTTON_COLOR_1.

React

ext-ab-testing/react ships a useExperiment hook:

import { useExperiment } from 'ext-ab-testing/react';

function UpgradeButton() {
    const { loading, active, variant } = useExperiment(ab, BUTTON_COLOR);
    if (loading) return null; // resolve in flight, no flash of a wrong arm
    if (!active || !variant) return <DefaultButton />; // kill switch off: default behaviour
    return <Button color={variant.color} />;
}

loading is true until the resolve settles, active: false means the kill switch is off, and variant is typed from the experiment definition. A rejecting resolve (broken storage/config source) fails safe to inactive. The state also carries pinned (see Ship the winner) for the analytics you report from the component.

Define the client and the experiment outside the component (module level), as above. The hook re-resolves when either reference changes, so objects created inline in the render would re-trigger the effect on every render.

Remote config (kill switch)

Host a JSON config anywhere:

{ "experiments": { "BUTTON_COLOR_1": { "active": true } } }

and give the client a config source. Fetch the config in the background service worker: content scripts make requests with the page's origin, so a cross-origin config fetch there is blocked by CORS, while the worker fetches it without any host_permissions. The worker fetches + caches and serves the other contexts by message:

// background service worker
import { normalizeAbTestConfig } from 'ext-ab-testing';

async function getConfig() {
    try {
        const res = await fetch(CONFIG_URL); // cache with a TTL in real code
        return normalizeAbTestConfig(await res.json());
    } catch {
        return undefined; // config unreachable -> every experiment off
    }
}

chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
    if (message?.type === 'GET_AB_CONFIG') {
        getConfig().then(sendResponse);
        return true; // async response
    }
});

// content script / popup
const ab = createAbTestingClient({
    storage: chromeLocalStorage,
    config: () => chrome.runtime.sendMessage({ type: 'GET_AB_CONFIG' })
});

const { active, variant } = await ab.resolve(BUTTON_COLOR);
if (!active) {
    // kill switch is off: run your default pre-experiment behaviour
}

A function config source is re-read on every resolve, so a config change lands within your cache TTL without recreating the client.

Flip active to false (or delete the entry) to abort the experiment for all users, no store re-release. Only a literal active: true enables; everything else (missing entry, malformed JSON, failed fetch, a config source that throws) fails safe to inactive.

A killed experiment never reads or writes storage (gate-then-assign): already-assigned users keep their arm on re-enable, while users who first arrived while it was off get a fresh pick.

Gating is presence-based: omit the config property to run ungated. A provided source that yields undefined is different: it means "config unavailable" and turns every experiment off. A failed fetch must never turn experiments on.

See examples/chrome-extension for the full MV3 wiring (background fetch + cache, content-side resolve, React popup).

Ship the winner (pin)

The kill switch only takes you back to control. When an arm wins, pin it in the same config and every user resolves to it right away, no store re-release:

{ "experiments": { "BUTTON_COLOR_1": { "active": true, "pin": "BLUE" } } }

Bake the winner into the code whenever the next release ships, then delete the experiment. The pin is what covers the gap in between — for a Chrome extension that is days of rollout across your user base.

Like the kill switch, a pin is resolution-time only: storage is never read or written. So assignments survive underneath it: remove the pin and every user goes back to their own arm, and users who first arrived during the ramp still get a fresh uniform pick (a pin burns no assignments).

resolve reports it as pinned: true:

const { active, variant, pinned } = await ab.resolve(BUTTON_COLOR);

Caveats

• A pinned resolve is not experiment data. The user is no longer in a random split, so reporting it as an assignment mixes ramped users into the winning arm and inflates it. Gate your analytics on pinned (see Google Analytics).
• The kill switch wins. active: false with a pin set still resolves inactive: an aborted experiment is aborted.
• A QA force wins over the pin, so you can still inspect the other arm while a ramp is live.
• A pin is untrusted data, not a typed id. A pin naming no variant (typo, renamed arm) resolves like any unmatched id: the fallbackVariantId if declared, otherwise variant: null — with active: true. Callers that render the default on !variant degrade safely, but everyone gets the default instead of the winner, silently. Check the id against your variants before uploading; normalizeAbTestConfig can only drop a non-string pin, it cannot know your arms.
• All-or-nothing. A pin moves 100% of users at once; it is not a percentage ramp. Weights would fight sticky assignment (already-assigned users cannot be re-rolled), so they are deliberately out of scope.

API

defineExperiment(experiment)

Identity function that captures literal types: variant.id becomes a union of the declared ids, payload stays typed through resolve, and a typo'd fallbackVariantId is a compile error.

Property	Type	Default	Description
key	string	(required)	Experiment identity: gates on config[key].active, persists under <keyPrefix><key>.
variants	V[]	(required)	{ id: string } plus any payload your arms need. Two variants → uniform 50/50; N → 1/N.
fallbackVariantId	V['id']	none	Variant returned when a persisted id matches no current variant (e.g. renamed); without it such resolves yield variant: null. Never biases the first pick or rewrites storage.

createAbTestingClient(options)

Option	Type	Default	Description
storage	AbTestStorage	(required)	{ get(key): Promise<string | undefined>, set(key, value): Promise<void> }. get must resolve undefined for "not assigned yet". A chrome.storage.local adapter ships as chromeLocalStorage in ext-ab-testing/chrome.
config	map | () => map | undefined | async	none (ungated)	The remote config: { [key]: { active, pin? } }. Omitting the property runs ungated: every experiment active. A function is re-read on every call; yielding undefined (or throwing) = all off.
keyPrefix	string	'AB_TEST.'	Storage-key namespace.

client.resolve(experiment, options?) → Promise<{ active, variant, pinned }>

Option	Type	Default	Description
force	V['id']	none	QA override: persist this id verbatim and resolve to it. Wins over a config pin.
chooseIndex	(count) => number	uniform random	Seam for the first-time pick; must return an integer in [0, count).

Returns { active: false, variant: null, pinned: false } when the kill switch is off (storage untouched). When active, variant is the assigned (or pinned) variant, or the fallback, or null if neither matches. pinned is true when the config pinned the arm for everyone — keep those out of your experiment analytics.

client.isActive(experiment) → Promise<boolean>

Peek at the kill switch without assigning; resolve would pick and persist a variant for a first-time user. A pinned experiment is still active.

normalizeAbTestConfig(body)

Coerces an untrusted remote config body ({ experiments: { KEY: { active, pin? } } }) into a safe config map: only literal true enables, a non-string pin is dropped, any malformed body yields {}.

Testing your experiments

import { createInMemoryAbTestStorage } from 'ext-ab-testing/testing';

// Pin the arm your test is about: force persists that id and resolves to it.
const ab = createAbTestingClient({ storage: createInMemoryAbTestStorage() });
await ab.resolve(BUTTON_COLOR, { force: 'BLUE' });

// Or simulate a user who was already assigned in an earlier session by
// seeding the storage:
const seeded = createAbTestingClient({
    storage: createInMemoryAbTestStorage({ 'AB_TEST.BUTTON_COLOR_1': 'BLUE' })
});
await seeded.resolve(BUTTON_COLOR); // BLUE: no re-roll, no write

The in-memory storage also exposes .store (the backing Map) and .setCalls() (write counter) for asserting what your code persisted.

URL attribution

For funnels that leave the extension, tag outbound URLs with ?exp=<slug>_<arm> and carry the finished string as variant payload. formatExpMarker / parseExpMarker compose and split markers; the AttributedVariant type standardises a marker field for it.

import { formatExpMarker, parseExpMarker } from 'ext-ab-testing';

const UPGRADE_PAGE = defineExperiment({
    key: 'UPGRADE_PAGE_1',
    variants: [
        {
            id: 'A',
            url: 'https://example.com/upgrade',
            marker: formatExpMarker('upgrade_page', 'a') // 'upgrade_page_a'
        },
        {
            id: 'B',
            url: 'https://example.com/upgrade-new',
            marker: formatExpMarker('upgrade_page', 'b') // 'upgrade_page_b'
        }
    ]
});

const { variant } = await ab.resolve(UPGRADE_PAGE);
if (variant) {
    // e.g. https://example.com/upgrade-new?exp=upgrade_page_b
    chrome.tabs.create({ url: `${variant.url}?exp=${variant.marker}` });
}

// On the receiving site (or in analytics), split the marker back:
parseExpMarker('upgrade_page_b'); // { experimentSlug: 'upgrade_page', arm: 'b' }

Google Analytics

To segment your funnel by experiment arm, report the assignment as event parameters. MV3 extension pages cannot load remote gtag.js, so send it from the service worker via the GA4 Measurement Protocol:

// background service worker
const { active, variant, pinned } = await ab.resolve(BUTTON_COLOR);

// Skip pinned resolves: once you ramp everyone to the winner (`pin` in the
// remote config) the user is no longer in a random split, and reporting it
// as an assignment would pile ramped users onto the winning arm and inflate
// it. Drop the event, or keep it and segment on `pinned` — but never let it
// count as experiment data.
if (active && variant && !pinned) {
    await fetch(
        `https://www.google-analytics.com/mp/collect?measurement_id=${GA_MEASUREMENT_ID}&api_secret=${GA_API_SECRET}`,
        {
            method: 'POST',
            body: JSON.stringify({
                client_id: clientId, // your persisted GA client id
                events: [
                    {
                        name: 'experiment_assignment',
                        params: {
                            experiment: BUTTON_COLOR.key, // 'BUTTON_COLOR_1'
                            variant: variant.id // 'GREEN' or 'BLUE'
                        }
                    }
                ]
            })
        }
    );
}

The same caveat applies to the conversion side of the funnel: an outbound URL marker (URL attribution) sent during a ramp reports the winning arm for users who were never assigned to it. Gate the marker on !pinned too, or carry the ramp in the marker so you can filter site-side.

For funnels that convert on your website, pass the arm along in the outbound URL instead (see URL attribution) and read it site-side.

Known limitations

Concurrent first resolve. AbTestStorage has no atomic get-or-set, so two contexts resolving the same fresh experiment at the same time (say, a content script and the popup racing on first run) can each pick a variant: last write wins, and every context converges on it from its next resolve. Acceptable by design, but worth knowing if you A/B the very first paint in two places at once.

If that matters to you, pre-assign at service-worker startup: call resolve for each experiment when the worker starts. The assignment lands once, from one context, before a popup or content script normally asks.

License

MIT

Author

Alexey Belozerov alex@welldonecode.com