Skip to content

feat: native pytest plugin (#175) - #196

Merged
AKogut merged 1 commit into
mainfrom
feat/175-pytest-plugin
Aug 3, 2026
Merged

feat: native pytest plugin (#175)#196
AKogut merged 1 commit into
mainfrom
feat/175-pytest-plugin

Conversation

@AKogut

@AKogut AKogut commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Second half of #175closes the issue.

Python could already report through JUnit XML, but that round-trip loses what matters most:

pytest-flakemetry reads pytest's own reports instead.

What it does

  • Parameters travel as structured params (test_login[chrome] → title test_login, params: {browser: "chrome"}), so variants bucket under one base test.
  • Retries are recognised via pytest-rerunfailures: fail-then-pass is reported as flaky.
  • Suite from the class, file path relative to the rootdir, error message and stack from the report.
  • Run context from the same environment variables the JavaScript reporters read (GITHUB_SHA, GITHUB_REF_NAME, GITHUB_RUN_ID, FLAKEMETRY_* overrides), so a Python run lands looking like any other — including the same idempotency-key scheme, so shards and reruns dedupe identically.
  • Delivery is best-effort throughout — an unreachable endpoint or rejected token goes to stderr and is swallowed. An outage must never turn a green suite red. Stdlib urllib only; the sole runtime dependency is pytest itself.

Guarding against cross-language drift

The real risk in adding a second language is that the two implementations quietly diverge. So CI does more than run the Python tests: it runs a sample suite through the plugin and validates the emitted batch against the TypeScript ingestRunBatchSchema. The Python reporter cannot start emitting something the server would reject without the build going red.

I ran that check locally end to end before wiring it: a 5-execution batch with a failure and two parameterized variants validates clean.

Verification

  • 11 Python tests, all passing locally on 3.9. pytester-based end-to-end coverage: pass/fail/skip mapping, parameterized variants carrying params, pass-on-rerun reported as flaky, class-based suites, no file written when nothing ran, and a dead endpoint leaving the run green (ret == 0). Plus unit tests for context resolution and idempotency keys (GitHub, local fallbacks, explicit overrides, shard distinctness, the ≥8-character contract minimum).
  • New python CI lane on 3.9 and 3.12, path-filtered.
  • pnpm turbo run build lint typecheck test — 61/61 green; pnpm format:check clean; docs build clean with dead-link checking on.

Notes

packages/pytest-flakemetry/package.json exists only so the contract validator can resolve @flakemetry/contracts as a workspace script; it is private and defines no build/test task, so the TS pipeline ignores it.

Python could already report through JUnit XML, but that round-trip loses what
matters most: parameter values get baked into the test name, so every variant
looks like a different test, and a retry is indistinguishable from a fresh
failure, so the strongest flaky signal never arrives.

Add pytest-flakemetry, which reads pytest's own reports instead. Parameters
travel as structured params so variants bucket under one base test, and a test
that passes on rerun is reported as flaky. Run context is resolved from the same
environment variables the JavaScript reporters read, so a Python run lands
looking like any other.

Delivery is best-effort throughout: an unreachable endpoint or a rejected token
is written to stderr and swallowed, because an outage must never turn a green
suite red.

The risk in a second language is that the two implementations drift, so CI
generates a real batch from a sample suite and validates it against the
TypeScript ingestion contract — the Python reporter cannot silently start
emitting something the server would reject.
@AKogut
AKogut merged commit 001c2b7 into main Aug 3, 2026
5 checks passed
@AKogut
AKogut deleted the feat/175-pytest-plugin branch August 3, 2026 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant