Skip to content

ci(#300): flaky-count honesty gate (annotate-first) + de-flake a perf unit test#316

Merged
TortoiseWolfe merged 2 commits into
mainfrom
fix/300-flaky-count-gate
Jul 18, 2026
Merged

ci(#300): flaky-count honesty gate (annotate-first) + de-flake a perf unit test#316
TortoiseWolfe merged 2 commits into
mainfrom
fix/300-flaky-count-gate

Conversation

@TortoiseWolfe

Copy link
Copy Markdown
Owner

Phase 3 of the #300 flaky cleanup: make a flake visible instead of laundered behind a green tick — plus fix the fifth flake this surfaced.

The gate (annotate-first)

retries: 2 reports a flake as a PASS — the job conclusion stays green, the rollup says CLEAN, and the only trace is a per-test github-reporter annotation nobody reads. That is how four flakes rode into main behind green checkmarks (#300). Nothing enforced the honesty rule "green conclusion ≠ 0 flaky".

Adds scripts/check-flaky-count.mjs + a Flaky-count honesty gate step in the report job. It does its own json merge of the blob reports, walks the merged report for tests Playwright marked flaky (failed an attempt, passed on retry), and surfaces them as one aggregated job-summary table + a GitHub annotation per flake.

  • annotate (default): surfaces flakes loudly (::warning::) but exits 0 — goes green from day one while E2E flaky watch: game-3d touch-drag camera (new on #297's run; check main's next run) #300's four known flakes are still open, instead of redding main immediately.
  • block: flip FLAKY_GATE_MODE: block (a one-word edit in e2e.yml) once those four are cleared → non-zero flaky becomes ::error:: + exit 1, and no NEW flake can ride in behind a green tick again.

Validated against a real Playwright report (a deliberately flaky browser-less test): annotate→warning+exit0, block→error+exit1, clean→"0 flaky"+exit0, missing→notice+exit0 (an infra gap never masquerades as a flake).

De-flake useColorblindMode perf tests

The pre-push gate caught a fifth flake while landing this: two "Performance" unit tests asserted a jsdom act() setState completes in <10ms (read 14.8ms under the loaded gate — and this runs in CI's unit suite too, so it can red main). Same wall-clock antipattern as the T148 backoff-ceiling fix (#315). They now assert the functional outcome (mode applied / patterns toggled) + a generous 1000ms pathological-slowness tripwire.

Verification

🤖 Generated with Claude Code

TurtleWolfe and others added 2 commits July 18, 2026 01:22
retries: 2 reports a flake as a PASS, so the job conclusion stays green and the rollup says CLEAN — which is how four distinct flakes rode into main behind green ticks (#300). Nothing enforced the honesty rule 'green conclusion != 0 flaky'.

Add scripts/check-flaky-count.mjs + a 'Flaky-count honesty gate' step in the report job. It runs its own json merge of the blob reports, walks the merged report for tests Playwright marked 'flaky' (failed an attempt, passed on retry), and surfaces them as one aggregated job-summary table + a GitHub annotation per flake.

annotate-first (FLAKY_GATE_MODE=annotate): surfaces flakes loudly without redding main while #300's four known flakes are still open. Flip to 'block' (::error:: + exit 1) once they are cleared so no NEW flake can ride in behind a green tick again.

Validated against a real Playwright report: annotate->warning+exit0, block->error+exit1, clean->0-flaky+exit0, missing->notice+exit0 (an infra gap never masquerades as a flake).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two 'Performance' tests asserted a jsdom act() setState completes in <10ms. That measures GC/scheduling under load, not real perf, and flaked the pre-push gate + CI unit suite (14.8ms > 10ms). Assert the functional outcome (mode applied / patterns toggled) and keep only a generous 1000ms pathological-slowness tripwire. Same class as the T148 backoff-ceiling flake.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@TortoiseWolfe
TortoiseWolfe merged commit c83e9b6 into main Jul 18, 2026
18 checks passed
@TortoiseWolfe
TortoiseWolfe deleted the fix/300-flaky-count-gate branch July 18, 2026 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants