Flaky Scoring

Flaky Scoring

Flakemetry's flaky detection is explainable by design. It is not a black-box classifier — it's a transparent statistical model where every score comes with human-readable reason codes. See ADR-0003.

The strongest signal

Same commit, different result.

If a test passes and fails on the identical commit_sha, the code didn't change — the test did (a race, timing, shared state, order dependence). This same_sha_variance is the highest-confidence flakiness signal and anchors the model.

Signals (per test identity, sliding window + time-decay)

Signal	Definition	Reads as
flip_rate	pass↔fail transitions / total transitions	how often it changes verdict
pass_on_rerun_rate	P(pass | previous attempt failed, same commit)	"green on retry" behaviour
same_sha_variance	different results on identical commit_sha	strongest flake signal
entropy	Shannon entropy of the result distribution	overall instability
fail_isolation	fails alone vs. alongside other tests	infra flake vs. real regression

fail_isolation disambiguates the test is flaky from the whole run was broken (infra/outage). A failure shared across many tests is likely environmental, not a per-test flake — so it should not inflate one test's flaky score.

The score

A Beta-Binomial posterior over pass/fail with exponential time-decay:

• Each execution updates the posterior incrementally (no full recompute).
• Recent results weigh more; a test that stabilizes heals over time as old flakes decay out.
• Output score ∈ [0, 1]; crossing a configurable threshold marks a quarantine candidate.

w(t)      = exp(-λ · age(t))                 # exponential decay, λ from half-life config
α, β      = prior + Σ w(t)·[pass], Σ w(t)·[fail]
stability = mean of Beta(α, β)               # ~ trust the test
score     = f(1 - stability, same_sha_variance, flip_rate, entropy)
            with same_sha_variance dominating

Every input is deterministic given the same history, so scores are reproducible and stamped with a model_version.

Explainability: reason codes

The score is never presented bare. Each carries the why:

["passed on rerun 4/5 times",
 "flipped 3× on the same commit abc123",
 "fails only when run in parallel shards",
 "flaky score rising over the last 7 days"]

This is what earns SDET trust — an engineer can see why a test is flagged and decide whether to quarantine, fix, or investigate.

Auto-quarantine

• score > threshold (default ~0.8) + minimum sample size → quarantine candidate.
• Policy is config-as-code (flakemetry.yml): threshold, min samples, cooldown before un-quarantine.
• Quarantined tests are surfaced back to the reporter so CI can run them non-blocking, with a full audit trail of state transitions.
• Tests auto-un-quarantine when they stabilize (decay + clean streak).

Delivery: quarantine automation lands in M3; the scoring engine + reason codes ship in M1.

Why not ML?

A gradient-boosted classifier could squeeze marginal accuracy, but:

• SDETs won't trust (or act on) an unexplained score.
• Labelled flaky data is scarce and noisy.
• The statistical model is debuggable, reproducible, and tunable per project.

We keep a seam for ML-derived signals to feed the same transparent aggregation later — but the surface stays explainable.

Related: Test Identity Engine provides the stable history this model runs on; Data Model stores flaky_score.