ci(tests): run the full suite on every main push - #1911
Open
seonghobae wants to merge 4 commits into
Open
Conversation
No workflow ran `tests/` on an unfiltered `main` push. Verified directly: - `agent-review-runtime-quality-ci.yml` has only a `pull_request` trigger (no `push:` key at all) with a narrow paths list. - `opencode-review-dispatch.yml` triggers solely on `repository_dispatch: types: [opencode-review]`. - `trusted-uv-materializer-quality-ci.yml` does run on `push: branches: [main]` but filters to the materialize/uv surface. - Seven workflows do push to main unfiltered (security scanners, SBOM, scorecard, secret-scan, the merge scheduler) and none of them run `tests/`. So merging a change to, say, `pr_review_merge_scheduler_core.py` or `opencode-review.yml` triggered no full-suite run, a suite-breaking merge landed silently on `main`, and the breakage first appeared as a red check on the next unrelated pull request. That is the failure mode behind #1823, #1826, #1828, #1892, and #1895, and behind the repair PRs #1829, #1874, and #1883. The new workflow deliberately carries no `paths` filter, since the point is to catch merges no path list anticipated. It is not in the organization required-workflow ruleset and is not injected into sibling repositories, so it costs one runner slot per `main` push in this repository only; successive pushes coalesce through its concurrency group instead of stacking. Root cause found by a peer session; this is the prescription half, kept separate from that session's documentation of the gap. actionlint: clean. Full suite with this file present: 2883 passed, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
A read-only Codex audit (a different model family, run per AGENTS.md's verification discipline) caught two claims that were broader than the evidence: - "Every other workflow that runs the full suite is either PR-only or carries a narrow paths filter" missed `repository-metadata-reconcile.yml`, which runs an unrestricted `pytest -q` on an hourly schedule. That does not contradict this workflow's reason to exist — a schedule is not a push, so a broken merge still sits undetected until the schedule fires — but the sweeping phrasing was wrong. - The materializer workflow does not watch "only the materialize/uv surface": its push paths also cover `tests/conftest.py`, `pyproject.toml`, the tooling requirements lock, and the repository branch-coverage tests. The same audit confirmed the two things that would have made this gate inert if wrong: `coverage report` enforces `fail_under = 100` from pyproject.toml without the flag (coverage 7.15.4 exits 2 below threshold), and no-argument `interrogate` reads `fail-under = 100` and the `tests` exclusion from the same file (interrogate 1.7.0). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
No workflow ran
tests/on an unfilteredmainpush. Verified directly against the workflow files:tests/?agent-review-runtime-quality-ci.ymlpull_requestonly, narrowpathsopencode-review-dispatch.ymlrepository_dispatchonlytrusted-uv-materializer-quality-ci.ymlSo merging a change to something like
pr_review_merge_scheduler_core.pyoropencode-review.ymltriggered no full-suite run at all. A suite-breaking merge landed silently onmain, and the breakage first surfaced as a red check on the next, entirely unrelated PR — whose author then had to prove the failure wasn't theirs.That is the failure mode behind #1823, #1826, #1828, #1892 and #1895, and behind the repair PRs #1829, #1874 and #1883.
The fix
A post-merge safety net running the repo's own documented triad —
coverage run -m pytest tests, the 100% coverage gate, and the 100% docstring gate.Deliberately no
pathsfilter: the whole point is catching merges that no path list anticipated.Cost
One runner slot per
mainpush, in this repository only — it is not in the org required-workflow ruleset, so it is not injected into the ~76 sibling repos. Successive pushes coalesce viaconcurrency+cancel-in-progressrather than stacking, which matters while the queue is saturated.Verification
actionlint .github/workflows/main-full-suite-gate.yml— cleanRoot cause was found by a peer session; this is the prescription half, deliberately kept separate from that session's documentation of the gap.
🤖 Generated with Claude Code