ci(#300): flaky-count honesty gate (annotate-first) + de-flake a perf unit test#316
Merged
Conversation
retries: 2 reports a flake as a PASS, so the job conclusion stays green and the rollup says CLEAN — which is how four distinct flakes rode into main behind green ticks (#300). Nothing enforced the honesty rule 'green conclusion != 0 flaky'. Add scripts/check-flaky-count.mjs + a 'Flaky-count honesty gate' step in the report job. It runs its own json merge of the blob reports, walks the merged report for tests Playwright marked 'flaky' (failed an attempt, passed on retry), and surfaces them as one aggregated job-summary table + a GitHub annotation per flake. annotate-first (FLAKY_GATE_MODE=annotate): surfaces flakes loudly without redding main while #300's four known flakes are still open. Flip to 'block' (::error:: + exit 1) once they are cleared so no NEW flake can ride in behind a green tick again. Validated against a real Playwright report: annotate->warning+exit0, block->error+exit1, clean->0-flaky+exit0, missing->notice+exit0 (an infra gap never masquerades as a flake). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two 'Performance' tests asserted a jsdom act() setState completes in <10ms. That measures GC/scheduling under load, not real perf, and flaked the pre-push gate + CI unit suite (14.8ms > 10ms). Assert the functional outcome (mode applied / patterns toggled) and keep only a generous 1000ms pathological-slowness tripwire. Same class as the T148 backoff-ceiling flake. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 of the #300 flaky cleanup: make a flake visible instead of laundered behind a green tick — plus fix the fifth flake this surfaced.
The gate (annotate-first)
retries: 2reports a flake as a PASS — the job conclusion stays green, the rollup says CLEAN, and the only trace is a per-testgithub-reporter annotation nobody reads. That is how four flakes rode into main behind green checkmarks (#300). Nothing enforced the honesty rule "green conclusion ≠ 0 flaky".Adds
scripts/check-flaky-count.mjs+ a Flaky-count honesty gate step in thereportjob. It does its own json merge of the blob reports, walks the merged report for tests Playwright markedflaky(failed an attempt, passed on retry), and surfaces them as one aggregated job-summary table + a GitHub annotation per flake.annotate(default): surfaces flakes loudly (::warning::) but exits 0 — goes green from day one while E2E flaky watch: game-3d touch-drag camera (new on #297's run; check main's next run) #300's four known flakes are still open, instead of redding main immediately.block: flipFLAKY_GATE_MODE: block(a one-word edit ine2e.yml) once those four are cleared → non-zero flaky becomes::error::+ exit 1, and no NEW flake can ride in behind a green tick again.Validated against a real Playwright report (a deliberately flaky browser-less test): annotate→warning+exit0, block→error+exit1, clean→"0 flaky"+exit0, missing→notice+exit0 (an infra gap never masquerades as a flake).
De-flake
useColorblindModeperf testsThe pre-push gate caught a fifth flake while landing this: two "Performance" unit tests asserted a jsdom
act()setState completes in<10ms(read 14.8ms under the loaded gate — and this runs in CI's unit suite too, so it can red main). Same wall-clock antipattern as the T148 backoff-ceiling fix (#315). They now assert the functional outcome (mode applied / patterns toggled) + a generous 1000ms pathological-slowness tripwire.Verification
🤖 Generated with Claude Code