feat: native pytest plugin (#175) - #196
Merged
Merged
Conversation
Python could already report through JUnit XML, but that round-trip loses what matters most: parameter values get baked into the test name, so every variant looks like a different test, and a retry is indistinguishable from a fresh failure, so the strongest flaky signal never arrives. Add pytest-flakemetry, which reads pytest's own reports instead. Parameters travel as structured params so variants bucket under one base test, and a test that passes on rerun is reported as flaky. Run context is resolved from the same environment variables the JavaScript reporters read, so a Python run lands looking like any other. Delivery is best-effort throughout: an unreachable endpoint or a rejected token is written to stderr and swallowed, because an outage must never turn a green suite red. The risk in a second language is that the two implementations drift, so CI generates a real batch from a sample suite and validates it against the TypeScript ingestion contract — the Python reporter cannot silently start emitting something the server would reject.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second half of #175 — closes the issue.
Python could already report through JUnit XML, but that round-trip loses what matters most:
pytest-flakemetryreads pytest's own reports instead.What it does
params(test_login[chrome]→ titletest_login,params: {browser: "chrome"}), so variants bucket under one base test.pytest-rerunfailures: fail-then-pass is reported asflaky.GITHUB_SHA,GITHUB_REF_NAME,GITHUB_RUN_ID,FLAKEMETRY_*overrides), so a Python run lands looking like any other — including the same idempotency-key scheme, so shards and reruns dedupe identically.urllibonly; the sole runtime dependency is pytest itself.Guarding against cross-language drift
The real risk in adding a second language is that the two implementations quietly diverge. So CI does more than run the Python tests: it runs a sample suite through the plugin and validates the emitted batch against the TypeScript
ingestRunBatchSchema. The Python reporter cannot start emitting something the server would reject without the build going red.I ran that check locally end to end before wiring it: a 5-execution batch with a failure and two parameterized variants validates clean.
Verification
pytester-based end-to-end coverage: pass/fail/skip mapping, parameterized variants carrying params, pass-on-rerun reported as flaky, class-based suites, no file written when nothing ran, and a dead endpoint leaving the run green (ret == 0). Plus unit tests for context resolution and idempotency keys (GitHub, local fallbacks, explicit overrides, shard distinctness, the ≥8-character contract minimum).pythonCI lane on 3.9 and 3.12, path-filtered.pnpm turbo run build lint typecheck test— 61/61 green;pnpm format:checkclean; docs build clean with dead-link checking on.Notes
packages/pytest-flakemetry/package.jsonexists only so the contract validator can resolve@flakemetry/contractsas a workspace script; it isprivateand defines no build/test task, so the TS pipeline ignores it.