FlakePacket

FlakePacket turns two or more JUnit XML reports into a deterministic, portable evidence packet. It helps an engineer answer one narrow question: which supplied test results changed across attempts, and what evidence supports that observation?

It runs offline, has no runtime dependencies, and writes both Markdown for handoff/review and JSON for local tooling.

Three-step workflow

1. Collect at least two JUnit XML reports from attempts you want to compare.
2. Run one flakepacket analyze command.
3. Open flakepacket.md or process flakepacket.json.

flakepacket analyze attempt-1.xml attempt-2.xml --output-dir evidence

Try it without bringing your own reports:

flakepacket demo --output-dir evidence

Those are the only two commands. The primary outcome is a local evidence packet; FlakePacket does not execute tests, connect to CI, or make release decisions.

Install

From a local clone with uv:

uv sync --all-groups --frozen
uv run flakepacket demo --output-dir docs/proof/scratch

Or build and install the wheel:

uv build
python -m pip install dist/flakepacket-0.1.0-py3-none-any.whl

FlakePacket requires Python 3.11 or later.

Classifications

Classification	What the supplied observations show
observed_inconsistent	At least one pass and at least one failure/error.
persistent_failure	At least two observations, all failure/error.
stable_pass	At least two observations, all pass.
persistent_skip	At least two observations, all skipped.
needs_review	A mixed non-pass pattern, such as skipped plus failure.
insufficient_evidence	Fewer than two observations for that test key.

Test identity is the exact tuple (suite, classname, name). Failure signatures normalize common volatile values such as timestamps, UUIDs, memory addresses, temporary paths, line numbers, and durations before hashing. The normalized detail preview remains in JSON so an engineer can detect an over-broad match.

Nested suites retain their full ancestry, such as service-a / unit, so identical leaf-suite names do not collapse together. When input paths share a basename, FlakePacket adds deterministic input-order suffixes such as junit.xml[1] and junit.xml[2]; unique basenames remain unchanged.

Input and safety limits

• At most 50 reports per run.
• At most 20 MiB per report.
• At most 200,000 parsed observations across the run.
• DOCTYPE and ENTITY declarations are rejected.
• XML parsing performs no network calls and resolves no external resources.
• Output paths are explicitly supplied by the operator.

JUnit XML is not perfectly standardized. FlakePacket supports <testsuites>, <testsuite>, standard failure, error, and skipped elements, plus common Maven-style flakyFailure, flakyError, rerunFailure, and rerunError retry evidence. Unknown dialect-specific semantics are not inferred.

Proof boundary

FlakePacket classifies only the reports supplied in one local invocation. It does not prove a root cause, establish that a test is inherently flaky, authorize quarantine, decide whether a release is safe, or validate a hosted/production system.

The committed docs/proof/ packet is fixture-backed local workflow evidence. Public-source publication is proven only by this repository at github.com/ownasquare/flakepacket. There is no hosted, production, provider-dashboard, package-registry, payment, usage, buyer, or demand proof.

Development

uv sync --all-groups --frozen
uv run ruff format --check .
uv run ruff check .
uv run mypy src tests scripts
uv run python -m unittest discover -s tests -v
uv run python scripts/audit.py
uv build

See CONTRIBUTING.md, SECURITY.md, and the completion record.

Monetization hypothesis

A future commercial hypothesis—not validated demand—is paid organization policy packs, trend history, or review workflows layered around the portable packet. This repository contains only the free local experiment and no payment or hosted-service code.

License

MIT. See LICENSE.