Experiment Guards

[knutties]

RFC: experimentation_guard — continuous experiment monitoring, auto-ramp, and guardrail enforcement

• Status: Draft
• Author: Natarajan
• Target: juspay/superposition, new crate crates/experimentation_guard
• Created: 2026-09-05

Summary

A standalone control-loop service that continuously monitors running experiments, automatically ramps healthy experiments through a configured traffic ladder, and pauses or shuts down experiments that breach guardrail metrics. The first supported backend is Superposition's own experimentation platform, but the core is generic: experiment sources, metric providers, actuators, and notifiers are all trait-abstracted, so the same engine can later drive other experimentation systems or read from other metrics stores.

The service lives in this repository as crates/experimentation_guard, ships as its own binary (and optionally embeds into the main superposition binary behind a feature flag), and interacts with Superposition exclusively through its public API — never through internal crates — so that it remains a reference consumer of the platform.

Positioning: the guard's data contract (guard config on the experiment, guard status on the experiment read model, lifecycle events) is promoted into the platform as a first-class concept, following the same layering precedent as experimentation_platform over CAC; the guard engine (metrics, evaluation, actuation) remains an optional, swappable consumer. See "Positioning" below.

Motivation

Superposition manages experiment definition, deterministic bucketing, and traffic ramping, but ramp and conclude decisions are fully manual today. In practice this means:

1. Slow ramps. Experiments sit at low traffic for days because ramping requires a human to check dashboards and click a button.
2. Slow rollbacks. A regressing variant keeps receiving traffic until someone notices. For payment-critical flows, minutes matter.
3. Inconsistent rigor. Whether anyone checks guardrail metrics before ramping depends on the team and the day.

An automated monitor closes the loop: metrics in, lifecycle actions out, with a full audit trail. This is also a natural piece of the OSS story — most open-source experimentation platforms stop at assignment and leave monitoring to the user.

Goals

• Continuously evaluate configurable guardrail metrics for every in-progress experiment, per variant.
• Automatically advance experiments through a traffic ramp ladder when healthy.
• Automatically pause ramping, or halt/roll back an experiment, when guardrails breach — subject to a per-workspace enforcement mode.
• Alert humans (Slack, generic webhook) on every state transition and action.
• Be safe to run: idempotent actions, append-only audit log, dry-run onboarding path, global kill switch, self-observability.
• Be generic: no compile-time dependency on Superposition server internals; all integration points behind traits.

Non-goals (v1)

• Winner detection / statistical significance for concluding experiments. v1 concludes nothing on its own; it only ramps, pauses, halts, and alerts. (See "Future work" for the sequential-testing path.)
• Metric collection. Applications must already export per-variant metrics to a supported store (Prometheus in v1). The guard reads; it never ingests raw events.
• A UI. v1 surfaces state via its API, structured logs, and notifications. Frontend integration into the Superposition console is future work.
• Multi-experiment interaction analysis (overlapping experiments contaminating each other's metrics).

Background: what Superposition already provides

The experimentation_platform crate exposes an HTTP API with, per experiment:

• Lifecycle: create → in-progress (with traffic_percentage) → concluded / discarded.
• Endpoints for ramp (set traffic percentage), conclude (pick a winning variant), and discard.
• Variants of type CONTROL and EXPERIMENTAL, each mapping to context overrides.
• Org/workspace scoping and token-based auth.

Crucially, Superposition knows which variant a client was assigned (via experimentation_client / SDKs) but has no knowledge of outcome metrics. The guard bridges that gap by requiring applications to label their telemetry with variant IDs (see "The attribution contract").

Positioning: guard as a first-class platform concept

The guard is first-class as a service (own crate, own Smithy service, own API and persistence) but must also be first-class as a domain concept inside the platform — otherwise users' natural mental model ("my experiment has guardrails") is unanswerable by the platform, config attached "next to" experiments drifts, and enforcement lives entirely in a component the platform doesn't know exists (so nothing even informs a human who manually ramps a breaching experiment).

The resolution follows the repository's existing layering precedent: experimentation_platform is a first-class concept layered on CAC, yet CAC runs fine without it. Guard is the next layer in the same pattern — config → experimentation → automation/safety. Concretely, the guard's data contract is promoted into the platform, while the engine remains a separable, optional consumer:

1. guard_config on the experiment model. An optional, platform-validated field on create/update (Smithy: use io.superposition.guard#GuardConfig + one optional member). Stored atomically with the experiment; eliminates the sidecar config table.
2. guard_status on the experiment read model. A compact enum — unguarded | healthy | breaching | paused | halted — plus last_evaluated_at, written back by the guard through one narrow platform endpoint (UpdateGuardStatus, callable only by the guard's service identity). List/get experiment responses and the console then show guard state with no cross-service stitching.
3. Experiment lifecycle events (platform → subscribers). Created / ramped / concluded / discarded webhooks or bus events. Guard-agnostic and useful to any automation; demotes the guard's poll loop to a reconciliation fallback.
4. Guard-aware ramp (later). A workspace policy under which a manual ramp on a guarded experiment whose guard_status is breaching requires an explicit override=true — always available and always audited, so a guard outage can never lock humans out. Closes the ramp-around gap without making the guard a hard dependency.

What deliberately stays out of the platform: metrics providers, guardrail evaluation, the state machine, and actuation. The platform never learns what Prometheus is. A deployment without metrics infrastructure simply has dormant guard_config fields and guard_status = unguarded.

Architecture

┌───────────────────┐         ┌───────────────────────┐
│   Superposition    │         │     Metrics store      │
│ experiments + ramp │         │ Prometheus / (later    │
│ / conclude / discard│        │ ClickHouse, SQL, ...)  │
└─────────┬─────────┘         └───────────┬───────────┘
          │ ExperimentSource               │ MetricsProvider
          ▼                                ▼
┌──────────────────────────────────────────────────────┐
│                 Guard control loop                    │
│  tick → snapshot experiments → fetch per-variant      │
│  metrics → evaluate guardrails → run state machine    │
│  → emit decisions                                     │
└──────────┬──────────────────────────┬────────────────┘
           │ Actuator                  │ Notifier
           ▼                           ▼
   ramp / pause / halt          Slack / webhook / log
   (writes back to
    Superposition API)

The loop is a single logical writer (advisory-lock leased; see "Deployment"), ticking on a fixed interval (default 60s). Each tick is stateless in computation but stateful in bookkeeping: breach counters, dwell timers, and last actions are persisted so restarts are safe.

Crate layout

crates/experimentation_guard/
├── src/
│   ├── lib.rs              # engine, re-exports
│   ├── engine/             # control loop, state machine, evaluator
│   ├── model/              # ExperimentHandle, Guardrail, RampPolicy, GuardEvent
│   ├── source/             # ExperimentSource trait + superposition impl
│   ├── metrics/            # MetricsProvider trait + prometheus impl
│   ├── actuator/           # Actuator trait + superposition impl
│   ├── notify/             # Notifier trait + slack / webhook / log impls
│   ├── store/              # persistence (diesel, Postgres)
│   └── bin/guard.rs        # standalone binary
└── Cargo.toml

Dependencies: superposition_types (shared models only), tokio, reqwest, serde, diesel/diesel_async (matching workspace conventions), tracing. Explicitly not a dependency: experimentation_platform or context_aware_config internals.

Core abstractions

#[async_trait]
pub trait ExperimentSource: Send + Sync {
    /// All experiments the guard should evaluate this tick.
    async fn active_experiments(&self) -> Result<Vec<ExperimentHandle>>;
}

pub struct ExperimentHandle {
    pub id: ExperimentId,
    pub workspace: WorkspaceId,
    pub name: String,
    pub variants: Vec<VariantInfo>,      // id + kind (control / experimental)
    pub traffic_percentage: u8,
    pub started_at: DateTime<Utc>,
    pub guard_config: Option<GuardConfig>, // per-experiment overrides, if attached
}

#[async_trait]
pub trait MetricsProvider: Send + Sync {
    /// Evaluate one metric query over a window, returning one sample per variant.
    async fn query(
        &self,
        q: &MetricQuery,
        window: Window,
        variants: &[VariantId],
    ) -> Result<HashMap<VariantId, MetricSample>>;
}

pub struct MetricSample {
    pub value: f64,
    pub sample_count: u64,   // denominator; gates min_samples
}

#[async_trait]
pub trait Actuator: Send + Sync {
    async fn ramp(&self, exp: &ExperimentHandle, pct: u8) -> Result<ActionReceipt>;
    async fn discard(&self, exp: &ExperimentHandle, reason: &str) -> Result<ActionReceipt>;
    async fn conclude(&self, exp: &ExperimentHandle, winner: VariantId) -> Result<ActionReceipt>;
}

#[async_trait]
pub trait Notifier: Send + Sync {
    async fn notify(&self, event: &GuardEvent) -> Result<()>;
}

Notes:

• MetricsProvider::query takes the variant list so providers can batch (one PromQL call with a variant label matcher rather than N calls).
• ActionReceipt carries the upstream response (e.g. new experiment revision) for the audit log.
• conclude exists on the trait from day one so the v2 winner-detection work needs no interface change, but the v1 engine never calls it.

v1 implementations

• SuperpositionSource / SuperpositionActuator — thin clients over the experimentation HTTP API using a scoped service token per workspace. Halting an experiment maps to discard if it never left ramp, or conclude(control) if rollback-to-control is the configured halt behavior (configurable per workspace; default conclude(control) for in-progress experiments since it restores baseline for all users deterministically).
• PrometheusProvider — executes PromQL with $variant substituted into a label matcher, e.g. variant=~"$variants"; parses instant-vector responses; requires the query to also expose a countable denominator series for sample_count (convention: a paired *_count query, or a count field in the query spec).
• Notifiers — Slack incoming webhook, generic JSON webhook, structured log (always on).

Guardrail specification

Guardrails are declarative and versioned. Example (TOML form; the canonical form is JSON so it can live inside Superposition config, see "Where configuration lives"):

[[guardrail]]
name = "success_rate_floor"
provider = "prometheus"
query = '''
  sum(rate(txn_success{variant=~"$variants"}[5m])) by (variant)
/ sum(rate(txn_total{variant=~"$variants"}[5m])) by (variant)
'''
count_query = 'sum(increase(txn_total{variant=~"$variants"}[5m])) by (variant)'
comparison = "relative_drop_vs_control"   # relative_drop_vs_control | absolute_below | absolute_above
threshold = 0.02                           # 2% relative drop vs control
window = "5m"
min_samples = 2000                         # per variant, within window
consecutive_breaches = 3                   # ticks
severity = "halt"                          # alert | pause_ramp | halt

Evaluation semantics

For each in-progress experiment, each tick:

1. Fetch every guardrail's metric for all variants over window.
2. A guardrail is evaluable for a variant only if sample_count >= min_samples for both that variant and (for vs-control comparisons) the control. Otherwise the guardrail is insufficient_data — neither healthy nor breached, and it blocks ramping (you cannot ramp on ignorance) but never triggers a halt.
3. Comparisons:
   • relative_drop_vs_control: breach if (control - variant) / control > threshold (for "higher is better" metrics; a direction field flips this).
   • absolute_below / absolute_above: breach if the variant's value crosses a fixed bound — for metrics with no meaningful control comparison (error-rate ceilings, latency SLOs).
4. A breached tick increments the guardrail's per-variant breach counter; a healthy tick resets it. Action fires only when the counter reaches consecutive_breaches.
5. The highest severity among triggered guardrails wins: halt > pause_ramp > alert.

min_samples + consecutive_breaches are deliberate, boring hysteresis. They prevent the classic failure of killing an experiment at 1% traffic off ten noisy data points, without requiring statistical machinery in v1.

Built-in guardrail: sample-ratio mismatch (SRM)

Shipped by default for every experiment: compare each variant's observed share of sample_count against the expected share implied by traffic_percentage and variant count, via a chi-squared test at α = 0.001. SRM breaches indicate broken bucketing or broken telemetry — both of which invalidate every other metric — so SRM severity is always at least pause_ramp, recommended halt. This one guardrail catches an outsized share of real-world experiment corruption and requires no per-team configuration.

Ramp policy

[ramp]
steps = [1, 5, 10, 25, 50, 100]
dwell = "30m"                 # minimum healthy time at a step before advancing
max_step_jump = 1             # advance one rung at a time
require_all_guardrails = true # every guardrail evaluable AND healthy through the dwell

Ramping advances only when: current dwell has elapsed, every guardrail has been continuously healthy (not insufficient_data) for the entire dwell, and enforcement mode permits auto-ramp. Reaching 100% does not conclude the experiment; it notifies that the experiment is fully ramped and awaiting human conclusion (v1) or statistical conclusion (v2).

If an experiment's traffic_percentage was changed out-of-band (a human ramped it manually), the guard adopts the observed value as the current rung and restarts the dwell — humans always win, and the guard must never fight a manual action.

Per-experiment state machine

              ┌────────────────────────────────────────────┐
              ▼                                            │
 Observing ──(dwell elapsed, all healthy)──▶ RampEligible ─┤ (ramp applied,
     │                                                     │  next rung)
     │ (guardrail breach tick)                             │
     ▼                                                     │
 Breaching(n) ──(healthy tick)──▶ Observing ◀──────────────┘
     │
     │ (n == consecutive_breaches)
     ▼
 ┌─ severity = alert      → notify, remain Observing
 ├─ severity = pause_ramp → Paused (no auto-ramp until human resume or sustained health)
 └─ severity = halt       → Halted (discard / conclude-to-control) — terminal for the guard

All state (current rung, dwell started, breach counters per guardrail per variant, enforcement decisions) is persisted per experiment. A restarted guard resumes exactly where it left off.

Where configuration lives

Two layers, resolved most-specific-wins:

1. Workspace defaults, stored in Superposition itself (dogfooding): a reserved config namespace (e.g. keys under experimentation_guard.* in the workspace's CAC) holding the default guardrail set, ramp policy, and enforcement mode. The guard reads these through the standard config API, so edits take effect on the next tick with no deploys and full CAC audit history.
2. Per-experiment overrides, carried as the first-class guard_config field on the experiment model (see "Positioning"), attached at creation and validated by the platform. Until that platform change lands, a guard-side table keyed by experiment ID (guard_config_overrides) serves as a transitional fallback and is deleted once the field exists.

For non-Superposition deployments of the engine, a local TOML file provides layer 1.

Safety model

The guard has authority to kill experiments, so it needs rails of its own:

• Enforcement modes, per workspace, forming an onboarding ladder:
  observe (evaluate + log only) → alert (+ notifications) → pause_ramp (may block/pause ramps, never halts) → full_auto (may halt). New workspaces start in observe.
• Idempotent, receipted actions. Every actuator call records intent before execution and the receipt after, in an append-only guard_events table including the exact metric values and thresholds that justified the decision. Replaying a tick never double-acts.
• Single logical writer. Replicas coordinate via a Postgres advisory lock lease; non-leaders evaluate nothing (v1) to keep semantics simple.
• Global kill switch. GUARD_ENFORCEMENT=off env var (checked every tick) drops all workspaces to observe without a deploy.
• Fail-static. If the metrics provider or Superposition API is unreachable, the guard takes no action, does not advance dwells, and raises its own alert. Unreachable metrics are never treated as healthy or as breached.
• Rate limits. At most one lifecycle action per experiment per tick; configurable per-workspace cap on halts per hour to bound blast radius of a bad guardrail config.
• Self-observability. The guard exports its own metrics: tick duration and lag, provider errors, evaluations, breaches, actions taken, lease status. A silent monitor is worse than no monitor; dead-man's-switch alerting on tick lag is part of the deployment recipe.

The attribution contract

The guard is only as good as variant labeling in telemetry. The hard prerequisite, to be documented prominently:

Applications under experiment MUST label the relevant metrics with the applicable variant ID(s) returned by experimentation_client / SDKs (e.g. a variant label on Prometheus series). Unlabeled traffic is invisible to the guard.

Notes:

• Cardinality: variant IDs are bounded (experiments × variants), so label cardinality is manageable; the docs should still warn against putting variant labels on unbounded-cardinality series.
• Overlapping experiments: a request may carry multiple applicable variants. v1 guidance is one label per experiment-scoped metric or exemplar-style separate series; cross-experiment interference is explicitly out of scope.
• The SRM guardrail doubles as a validator of this contract: broken labeling shows up as SRM breach.

Persistence

Reuses the workspace Postgres, own schema experimentation_guard:

• guard_state(experiment_id PK, workspace, machine_state, current_rung, dwell_started_at, breach_counters JSONB, updated_at, ...)
• guard_events(id, experiment_id, ts, kind, severity, guardrail, variant, observed JSONB, action, receipt JSONB) — append-only audit log.
• guard_config_overrides(experiment_id PK, config JSONB) — transitional only; dropped once guard_config is first-class on the experiment model.

Service surface

Minimal HTTP API on the guard binary:

• GET /healthz, GET /metrics (Prometheus format)
• GET /experiments/{id}/state — current machine state, guardrail statuses, recent events
• POST /experiments/{id}/resume — human override to move Paused → Observing
• POST /experiments/{id}/exempt — temporarily exclude an experiment from enforcement (with TTL and audit)

All mutating endpoints require the same auth scheme as the platform.

Deployment

Two modes:

1. Standalone (recommended): experimentation-guard container next to the Superposition deployment; needs network access to the Superposition API, the metrics store, Postgres, and notifier endpoints. Horizontal replicas are safe (leased single writer).
2. Embedded: superposition binary with --features guard spawns the loop in-process — for single-node/self-hosted setups. Same code path, same persistence.

Statistical posture (v1 vs future)

v1 intentionally uses fixed thresholds with sample floors and consecutive-breach hysteresis rather than hypothesis testing. Rationale: guardrail shutdown is an asymmetric-cost decision (missing a real regression is far worse than an occasional false pause), thresholds are auditable and explainable to non-statisticians, and continuous evaluation with naive p-values would invite peeking pathologies anyway.

Future work, in order:

1. Sequential testing for guardrails — mSPRT or group-sequential bounds for vs-control comparisons, allowing tighter thresholds at equal false-positive rates under continuous monitoring.
2. Winner detection and auto-conclude — sequential significance on success metrics, activating the already-present Actuator::conclude path behind a new enforcement mode.
3. Variance reduction (CUPED) where pre-experiment covariates are available.
4. ClickHouse / SQL MetricsProvider.
5. Console integration — surface guard state and events in the Superposition frontend.

Alternatives considered

• Build monitoring into experimentation_platform directly. Rejected: couples the platform to specific metrics stores, complicates its deployment story, and loses the generic-engine property. A separate consumer of the public API keeps the platform honest.
• Push-based (webhook/alertmanager-triggered) instead of a polling loop. Rejected for v1: dwell-based ramping needs a clock anyway, pull keeps the provider interface trivial, and Alertmanager routes can still be layered on later as a MetricsProvider variant.
• Per-team cron jobs / scripts. The status quo, minus consistency, audit, and safety rails.

Open questions

1. Halt semantics default: conclude(control) vs discard for in-progress experiments — does conclude(control) have side effects (e.g. promoting control overrides into default config) that make discard + alert the safer default in some workspaces?
2. UpdateGuardStatus write-back semantics: how stale may guard_status be before the platform annotates it (e.g. surface last_evaluated_at prominently, or auto-degrade to unguarded after a TTL so a dead guard cannot leave a permanently green badge)?
3. Lifecycle events transport: webhooks vs a lightweight bus — and does anything else in the roadmap want the same mechanism?
4. Minimum platform API additions needed, if any (e.g. a filtered "list in-progress experiments across workspaces" endpoint for efficient polling at org scale)?
5. Token/RBAC scope for the guard's service identity — narrowest scope that still permits ramp/discard/conclude plus UpdateGuardStatus, and nothing else?

Build plan

Guard-side phases, with the platform-side promotions (P1–P4 from "Positioning") slotted where they pay off:

1. Phase 0 — skeleton: crate, traits, models, Postgres schema, config loading, observe mode end-to-end with SuperpositionSource + PrometheusProvider, structured-log notifier. Ship and run against a real workspace in observe mode.
2. Phase 1 — alerting: Slack/webhook notifiers, SRM guardrail, service API, self-metrics. Graduate a workspace to alert.
   • Platform: P1 guard_config field on the experiment model (Smithy + storage + validation); drop the transitional overrides table.
3. Phase 2 — pause_ramp + auto-ramp: ramp ladder, dwell gating, manual-override adoption, resume/exempt endpoints.
   • Platform: P2 guard_status + UpdateGuardStatus write-back, surfaced in list/get experiment responses and the console's experiment list.
4. Phase 3 — full_auto: halt path, rate limits, kill switch, runbook.
   • Platform: P3 lifecycle events, demoting the poll loop to reconciliation.
5. Phase 4+: sequential testing, auto-conclude, ClickHouse provider, console UI.
   • Platform: P4 guard-aware ramp (override=true policy for breaching experiments), once guard_status has earned operator trust.