v0.2.1

Changelog

Features

• 289f12d: feat: make the validation result reproducible in two minutes (@rowhitswami)

Fixes

• 201b596: fix(ci): restore HOMEBREW_TAP_TOKEN to the goreleaser step (@rowhitswami)
• ce37c54: fix(ci): the release-workflow guard needs pyyaml on the runner (@rowhitswami)
• 1283c55: fix: ** in an ingest pattern silently dropped most of the reports (@rowhitswami)
• 8051bfa: fix: go install produced a binary that reported 'dev' (@rowhitswami)
• 8f81981: fix: match the post to what the script prints, and stop gating CI on a (@rowhitswami)

Other

• 07d5a03: security: make the installers fail closed, and sign what we release (@rowhitswami)

Install

# Homebrew
brew install rowhitswami/tap/flakestat

# Go
go install github.com/rowhitswami/flakestat/cmd/flakestat@v0.2.1

# Script
curl -sSfL https://raw.githubusercontent.com/rowhitswami/flakestat/main/scripts/install.sh | sh

# npm / PyPI
npx flakestat --help
pip install flakestat

v0.2.0

What's new

Everything in the documentation now exists in the binary. v0.1.1 shipped six
commands; this ships ten, plus the analysis that makes the verdicts worth
reading.

Explain any verdict

flakestat explain test_checkout_flow

Prints the full evidence behind one test: outcome history as a strip, how many
disagreements happened on identical code, which commits and branches it was seen
on, and where its failures concentrate. --json emits the same evidence for
tooling.

Confidence on every verdict

Verdicts now carry low, medium or high, derived from how much evidence
exists. A score of 0.62 from three runs and the same score from three hundred
are different claims, and a small sample can never reach high. Classification
uses a lower bound on the score rather than the score itself, so a verdict needs
evidence rather than a lucky flip.

Record where tests ran, and see where failures cluster

flakestat ingest 'reports/**/*.xml' --dimension os=windows --dimension runtime.version=3.13

Platform, runtime and CI context are recorded per observation, and flakestat
reports where failures concentrate:

Where the failures concentrate
  os=windows
    failures here:  12 / 12 (100.0%)
    elsewhere:      0 / 24 (0.0%)
    difference:     +100.0 pp

Findings must clear an evidence floor, an effect-size floor and a
Benjamini–Hochberg correction, because testing several dimensions will otherwise
turn up something significant by chance. Associations never influence the score,
and the wording is correlational: flakestat says failures cluster on Windows, not
that Windows causes them. When dimensions vary together it says the observations
cannot tell which one matters.

Only a whitelist of known CI variables is read, and only JUnit properties
flakestat recognizes — the process environment is never scraped, so secrets
cannot end up in your history.

Re-ingesting the same results is now safe

Every observation carries the identity of the execution it describes, so an
artifact uploaded twice, a re-run aggregation step or a shard collected by two
jobs counts once — while a genuine retry, which really did run the tests again,
still counts.

This mattered more than it sounds. A duplicate carries its original's timestamp,
sorts beside it and always agrees with itself, so uncounted duplicates make a
flaky test look stable. Measured on a test failing 4 of 12 runs, ingesting
the same reports twice moved the score from 0.64 to 0.30 — the error ran towards
false negatives. flakestat compact removes copies from the file as well.

CI reporting

flakestat ci-report renders job summaries and pull request comments. The
GitHub Action gained comment, annotations and max-rows inputs; a re-run
updates the same comment rather than adding another.

Fixes

Outcomes are only compared when they are comparable. A test that always
passes on Linux and always fails on Windows is deterministic, but read as one
chronological series it looked like constant disagreement — it scored 0.45 with
an explanation asserting direct evidence of nondeterminism. Transitions are now
counted within one execution context: branch, os, arch and runtime held
constant.

The history strip agrees with the verdict. It grouped by branch while
scoring grouped by execution context, so it printed "flip on identical code" at
every platform boundary — at precisely the points the verdict had ruled out as
incomparable.

Execution identity uses the whole recorded context. GitHub reports the same
GITHUB_JOB for every leg of a matrix, so three platforms can ingest under one
provider, run, job and attempt. Keying on the CI fields alone would have merged
them.

Validation

This release was validated against three external projects under a contract
committed before any experiment ran, and scored with a single frozen build. The
full report is in VALIDATION.md.

The load-bearing result: at the commit before ConduitIO's own deflaking fix,
three of the four tests their issue named scored flaky at 0.67 with high
confidence; at the fix itself, all four were stable at 0.00, under an identical
protocol. Their contributors documented the defect, named the tests and wrote
the repair — flakestat was handed observations from both sides of a commit it had
no part in and separated them.

Also recorded there: the protocol arm that found nothing, and a contaminated run
that was discarded rather than reported.

Install

brew install rowhitswami/tap/flakestat
npm install --save-dev flakestat
pip install flakestat
go install github.com/rowhitswami/flakestat/cmd/flakestat@latest

Documentation: https://flakestat.com

v0.1.1

Changelog

Fixes

• 6bedf73: fix(pypi): derive the package version from installed metadata (@rowhitswami)

Install

# Homebrew
brew install rowhitswami/tap/flakestat

# Go
go install github.com/rowhitswami/flakestat/cmd/flakestat@v0.1.1

# Script
curl -sSfL https://raw.githubusercontent.com/rowhitswami/flakestat/main/scripts/install.sh | sh

# npm / PyPI
npx flakestat --help
pip install flakestat

v0.1.0

Changelog

Features

• 71da76d: feat(action): add GitHub Action (Rohit Swami rowhitswami1@gmail.com)
• a59e5c3: feat(baseline): record the flakiness a repository already accepts (Rohit Swami rowhitswami1@gmail.com)
• c13e3ed: feat(cli): add init, hunt, ingest, report, check and quarantine (Rohit Swami rowhitswami1@gmail.com)
• 8858a63: feat(config): detect the project and support .flakestat.json (Rohit Swami rowhitswami1@gmail.com)
• 40fe323: feat(junit): add tolerant JUnit XML parser (Rohit Swami rowhitswami1@gmail.com)
• 3c99bc6: feat(npm): add npm wrapper package (Rohit Swami rowhitswami1@gmail.com)
• f75d733: feat(pypi): add PyPI wrapper package (Rohit Swami rowhitswami1@gmail.com)
• ac19e8a: feat(quarantine): emit skip lists test runners actually accept (Rohit Swami rowhitswami1@gmail.com)
• 192c9f9: feat(report): render results as table, JSON or markdown (Rohit Swami rowhitswami1@gmail.com)
• e3630e5: feat(runner): run a test command repeatedly and collect reports (Rohit Swami rowhitswami1@gmail.com)
• f4c06a5: feat(score): score flakiness as inconsistency, not failure (Rohit Swami rowhitswami1@gmail.com)
• 7af6e9e: feat(store): add append-only NDJSON observation log (Rohit Swami rowhitswami1@gmail.com)

Fixes

• 3a39fe2: fix(docker): copy the binary from the platform-scoped context path (Rohit Swami rowhitswami1@gmail.com)

Other

• e66ec61: build: add GoReleaser, container image and install script (Rohit Swami rowhitswami1@gmail.com)
• 18e2e02: test(runner): skip shell-driven fixtures on Windows (Rohit Swami rowhitswami1@gmail.com)

Install

# Homebrew
brew install rowhitswami/tap/flakestat

# Go
go install github.com/rowhitswami/flakestat/cmd/flakestat@v0.1.0

# Script
curl -sSfL https://raw.githubusercontent.com/rowhitswami/flakestat/main/scripts/install.sh | sh

# npm / PyPI
npx flakestat --help
pip install flakestat