What's new
Everything in the documentation now exists in the binary. v0.1.1 shipped six
commands; this ships ten, plus the analysis that makes the verdicts worth
reading.
Explain any verdict
flakestat explain test_checkout_flow
Prints the full evidence behind one test: outcome history as a strip, how many
disagreements happened on identical code, which commits and branches it was seen
on, and where its failures concentrate. --json emits the same evidence for
tooling.
Confidence on every verdict
Verdicts now carry low, medium or high, derived from how much evidence
exists. A score of 0.62 from three runs and the same score from three hundred
are different claims, and a small sample can never reach high. Classification
uses a lower bound on the score rather than the score itself, so a verdict needs
evidence rather than a lucky flip.
Record where tests ran, and see where failures cluster
flakestat ingest 'reports/**/*.xml' --dimension os=windows --dimension runtime.version=3.13
Platform, runtime and CI context are recorded per observation, and flakestat
reports where failures concentrate:
Where the failures concentrate
os=windows
failures here: 12 / 12 (100.0%)
elsewhere: 0 / 24 (0.0%)
difference: +100.0 pp
Findings must clear an evidence floor, an effect-size floor and a
Benjamini–Hochberg correction, because testing several dimensions will otherwise
turn up something significant by chance. Associations never influence the score,
and the wording is correlational: flakestat says failures cluster on Windows, not
that Windows causes them. When dimensions vary together it says the observations
cannot tell which one matters.
Only a whitelist of known CI variables is read, and only JUnit properties
flakestat recognizes — the process environment is never scraped, so secrets
cannot end up in your history.
Re-ingesting the same results is now safe
Every observation carries the identity of the execution it describes, so an
artifact uploaded twice, a re-run aggregation step or a shard collected by two
jobs counts once — while a genuine retry, which really did run the tests again,
still counts.
This mattered more than it sounds. A duplicate carries its original's timestamp,
sorts beside it and always agrees with itself, so uncounted duplicates make a
flaky test look stable. Measured on a test failing 4 of 12 runs, ingesting
the same reports twice moved the score from 0.64 to 0.30 — the error ran towards
false negatives. flakestat compact removes copies from the file as well.
CI reporting
flakestat ci-report renders job summaries and pull request comments. The
GitHub Action gained comment, annotations and max-rows inputs; a re-run
updates the same comment rather than adding another.
Fixes
Outcomes are only compared when they are comparable. A test that always
passes on Linux and always fails on Windows is deterministic, but read as one
chronological series it looked like constant disagreement — it scored 0.45 with
an explanation asserting direct evidence of nondeterminism. Transitions are now
counted within one execution context: branch, os, arch and runtime held
constant.
The history strip agrees with the verdict. It grouped by branch while
scoring grouped by execution context, so it printed "flip on identical code" at
every platform boundary — at precisely the points the verdict had ruled out as
incomparable.
Execution identity uses the whole recorded context. GitHub reports the same
GITHUB_JOB for every leg of a matrix, so three platforms can ingest under one
provider, run, job and attempt. Keying on the CI fields alone would have merged
them.
Validation
This release was validated against three external projects under a contract
committed before any experiment ran, and scored with a single frozen build. The
full report is in VALIDATION.md.
The load-bearing result: at the commit before ConduitIO's own deflaking fix,
three of the four tests their issue named scored flaky at 0.67 with high
confidence; at the fix itself, all four were stable at 0.00, under an identical
protocol. Their contributors documented the defect, named the tests and wrote
the repair — flakestat was handed observations from both sides of a commit it had
no part in and separated them.
Also recorded there: the protocol arm that found nothing, and a contaminated run
that was discarded rather than reported.
Install
brew install rowhitswami/tap/flakestat
npm install --save-dev flakestat
pip install flakestat
go install github.com/rowhitswami/flakestat/cmd/flakestat@latestDocumentation: https://flakestat.com