fix: disable Claude + CodeRabbit auto-trigger check suites to unblock auto-merge#195
Conversation
… auto-merge GitHub auto-creates "queued" check suites for every GitHub App that has ever run in a repo, on every push. Claude (app_id 1236702) and CodeRabbit (app_id 347564) create these suites proactively but only complete them when they have real work to do. When they have nothing to do (no @claude mention, no CodeRabbit trigger), the suites stay queued forever. GitHub auto-merge waits for ALL check suites — not just required ones — to reach a terminal state before merging. Result: mergeStateStatus: BLOCKED even with reviewDecision: APPROVED and all required checks passing. Fix: PATCH /repos/{owner}/{repo}/check-suites/preferences with auto_trigger_checks: [{app_id: N, setting: false}] for both app IDs. GitHub stops auto-creating the suites; the apps still create them explicitly when they have real work to report. - apply-repo-settings.sh: apply_check_suite_prefs() applies the fix per repo - compliance-audit.sh: check_check_suite_prefs() detects drift and files issues After merge: run apply-repo-settings.sh --all once to apply across all repos. The weekly compliance-audit-and-improvement workflow enforces it going forward. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Warning Rate limit exceeded
To continue reviewing without waiting, purchase usage credits in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds org- and repo-level check-suite auto-trigger handling: an enforcement step in ChangesCheck-Suite Preference Management
Sequence Diagram(s)sequenceDiagram
participant ApplyScript as Apply Script
participant AuditScript as Audit Script
participant GitHubAPI as GitHub API
participant Operator as Operator
Operator->>ApplyScript: run script (--all or single-repo)
ApplyScript->>GitHubAPI: GET repos/:org/:repo/check-suites/preferences
GitHubAPI-->>ApplyScript: preferences (list of apps + auto_trigger_checks)
ApplyScript->>GitHubAPI: PATCH repos/:org/:repo/check-suites/preferences (set auto_trigger_checks:false for targeted apps)
GitHubAPI-->>ApplyScript: 200/response
Operator->>AuditScript: run compliance audit
AuditScript->>GitHubAPI: GET repos/:org/:repo/check-suites/preferences
GitHubAPI-->>AuditScript: preferences
AuditScript-->>Operator: emit finding if any target app has auto_trigger_checks:true
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/apply-repo-settings.sh`:
- Around line 248-249: The script calls warn() on failure paths but never
defines it, causing "command not found" errors; add a warn() function near the
top (alongside any existing log/info/error helpers) that accepts a message,
writes it to stderr (e.g., printf "%s\n" "$*" >&2) and returns a non-zero status
or otherwise matches the behavior of other logger helpers so calls like warn
"Could not read check-suite preferences for $repo — skipping" and the other
usage at the similar branch work correctly; update or reuse existing logger
helpers if present (e.g., info, error) so warn() is consistent with them.
- Around line 247-250: The branch that checks if "$prefs" is empty currently
calls warn "Could not read check-suite preferences..." and then uses return 0
which incorrectly signals success; update the control flow in the function that
contains the prefs check so that when prefs is empty it returns a non-zero
status (e.g., return 1) or otherwise exits with a failure code instead of return
0, ensuring callers detect the error; keep the warn call and only change the
return value from 0 to a failure code to surface enforcement failures.
In `@scripts/compliance-audit.sh`:
- Around line 868-870: The script currently silences failures when querying
check-suite preferences (prefs=$(gh_api ...); [ -z "$prefs" ] && return), which
hides coverage gaps; replace the silent return with logic that emits a finding
for the repo indicating the control was not evaluated (use your existing
findings mechanism — e.g., call record_finding or append a message to the
findings array — with repo name $repo, control "check-suite-preferences", and a
reason like "API unavailable/empty response"), then continue or skip further
checks for that repo; keep the gh_api call and prefs variable but ensure the
error/finding includes the repo and reason so the audit records the missing
evaluation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: b87eec94-c6a1-4a36-b562-3418d5c56082
📒 Files selected for processing (2)
scripts/apply-repo-settings.shscripts/compliance-audit.sh
There was a problem hiding this comment.
Pull request overview
This PR aims to unblock GitHub auto-merge by disabling GitHub’s automatic creation of check suites for the Claude and CodeRabbit GitHub Apps, which can otherwise remain permanently in a queued state and block merging.
Changes:
- Add a compliance-audit check to detect repos where Claude/CodeRabbit check-suite auto-triggering is still enabled.
- Add an apply-repo-settings step that PATCHes
check-suites/preferencesto disable auto-triggered check suites for those app IDs.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| scripts/compliance-audit.sh | Adds check_check_suite_prefs() to report an error finding when Claude/CodeRabbit auto-trigger checks are enabled. |
| scripts/apply-repo-settings.sh | Adds apply_check_suite_prefs() to PATCH repo check-suite preferences and disable auto-trigger for the two app IDs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add warn() to apply-repo-settings.sh (was undefined, would crash on error path) - Return 1 (not 0) when check-suite preferences cannot be fetched (masking enforcement failures) - Treat "missing" setting as compliant in apply_check_suite_prefs, matching audit behavior (app has never run in repo — no orphaned suite possible) - Document check-suite auto-trigger preferences requirement in standards/github-settings.md, including the classic-PAT requirement for the PATCH endpoint Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Test results — behavior confirmed on this PRTested by monitoring check suites on both this PR (#195) and PR #193 (which was rebased on main by the auto-rebase workflow during this session). FindingsBug confirmed on every push: On every commit to a PR branch, GitHub auto-creates check suites for the Claude (app_id Key observation from this PR's own head commit (
The action and the app suite are completely independent. The The fix
PAT requirement discoveredThe
Classic PAT ( After mergeRun with a classic PAT to apply across all org repos: GH_TOKEN=<classic-pat> bash scripts/apply-repo-settings.sh --all |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/apply-repo-settings.sh`:
- Around line 246-250: The gh API calls that read and PATCH check-suite
preferences currently discard stderr (the prefs=$(gh api
"repos/$ORG/$repo/check-suites/preferences" 2>/dev/null || true) and the
subsequent PATCH call), which hides useful GitHub error messages; change these
to capture the command output and stderr into variables (e.g., prefs and
prefs_err or a combined var), check command exit status, and call warn with a
descriptive message including the captured error details and the repo name when
the read or PATCH fails so token/permission errors are preserved for debugging;
update the code around prefs and the PATCH invocation accordingly.
In `@standards/github-settings.md`:
- Around line 258-265: Update the paragraph under "Required configuration" to
explicitly state that a missing `auto_trigger_checks` entry for the listed app
IDs is treated as compliant (until that app has ever created a check run), and
clarify that both `scripts/apply-repo-settings.sh` and
`scripts/compliance-audit.sh` treat a missing entry as acceptable; mention the
specific keys `auto_trigger_checks` and the app IDs `1236702` and `347564` so
readers can find the table rows for Claude (`anthropics/claude-code-action`) and
CodeRabbit and avoid chasing non-issues.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 6097d04c-1a39-4c28-80b7-72a093efa4fd
📒 Files selected for processing (2)
scripts/apply-repo-settings.shstandards/github-settings.md
Instead of silently skipping when the check-suites/preferences API call fails (which hides audit coverage gaps), emit a warning-level finding so operators know the control was not evaluated. Addresses CodeRabbit review comment on PR #195. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- apply-repo-settings.sh: capture API error details on prefs read/PATCH failure instead of swallowing stderr, so token-type and permission issues are visible in output - standards/github-settings.md: clarify that a missing auto_trigger_checks entry (app never ran in repo) is treated as compliant by both apply and audit scripts — operators should not chase non-issues Addresses CodeRabbit nitpick and minor comments on PR #195. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
All review comments have been addressed:
@petry-projects/org-leads — all CI checks passing and review items resolved. Ready for a final look and merge. |
|
All review comments addressed:
@petry-projects/org-leads — ready for review and merge. |
Two paragraphs added in the check-suite auto-trigger section exceeded the 200-character line limit. Wrap at sentence/clause boundaries to pass the markdownlint MD013 check. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
@coderabbitai review All prior review comments have been addressed and all threads resolved. Summary of fixes since the last approval:
|
|
@coderabbitai review All prior review comments are addressed and all 8 threads are resolved. The only change since your last approval (00:40 UTC) is commit 705abb2 — a line-wrap fix in standards/github-settings.md to satisfy MD013. No logic changes. |
|
Tip For best results, initiate chat on the files or code changes.
[review] |
|
Tip For best results, initiate chat on the files or code changes.
[review] |
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: LOW
Reviewed commit: 705abb2e5660d5c8a69fcb1f3a50b16e5bf463c2
Review mode: triage-approved (single reviewer)
Summary
Adds check-suite auto-trigger preference management to the existing apply-repo-settings.sh and compliance-audit.sh scripts. Disables auto-trigger for Claude (app_id 1236702) and CodeRabbit (app_id 347564) GitHub Apps to prevent orphaned queued check suites from permanently blocking GitHub auto-merge. All changes are additive (+130, -0) across 3 files with corresponding documentation in standards/github-settings.md.
Linked issue analysis
No linked issues. The PR body provides thorough context explaining the root cause (GitHub auto-creates check suites that stay queued forever, blocking auto-merge) and the fix mechanism (PATCH /repos/{owner}/{repo}/check-suites/preferences).
Findings
No blocking issues found.
- Security: No credentials, secrets, auth changes, or injection vectors. The
gh apicalls use hardcoded app IDs processed throughjq— no user-controlled input flows into shell commands or API payloads. - Correctness: Both scripts consistently handle the
missingstate (app never seen in repo) by skipping it. Error paths properly return non-zero and emit diagnostic messages with captured API errors. DRY_RUN mode is respected. - Consistency: The new functions follow existing patterns in both scripts (logging helpers, error handling,
add_findingusage in audit). TheCHECK_SUITE_APP_IDSconstant is duplicated across both scripts, which mirrors how other constants (e.g.,REQUIRED_LABEL_SPECS) are already handled. - Prior review feedback: All 8 review threads from CodeRabbit and Copilot were addressed across commits
aefaba3,b131b12,ad005d7, and705abb2. CodeRabbit confirmed each fix and gave final approval. - Documentation: Standards doc updated with clear explanation, required configuration table, manual remediation steps, and PAT requirement note.
CI status
| Check | Status |
|---|---|
| ShellCheck | ✅ SUCCESS |
| Lint | ✅ SUCCESS |
| CodeQL | ✅ SUCCESS |
| SonarCloud | ✅ SUCCESS (0 issues, 0 hotspots) |
| Agent Security Scan | ✅ SUCCESS |
| AgentShield | ✅ SUCCESS |
| claude-code / claude | ✅ SUCCESS |
All required checks pass. Skipped checks (dependabot, npm/pnpm/pip/cargo audit, govulncheck) are not applicable to shell scripts.
Reviewed automatically by the don-petry PR-review agent (single-reviewer mode: opus 4.6). Reply with @don-petry if you need a human.
Summary
1236702) and CodeRabbit (app_id347564) on every push to any branch with an open PR. When those apps have nothing to do, the suites stay inqueuedstate forever.mergeStateStatus: BLOCKEDeven withreviewDecision: APPROVEDand all required checks green.PATCH /repos/{owner}/{repo}/check-suites/preferenceswithauto_trigger_checks: [{app_id: N, setting: false}]stops GitHub from auto-creating these suites. The apps still create them explicitly when they have real work to report.Changes
scripts/apply-repo-settings.sh—apply_check_suite_prefs()applies the preference to a reposcripts/compliance-audit.sh—check_check_suite_prefs()detects drift and creates error-severity findingsTest plan
bash scripts/apply-repo-settings.sh .githubgh api repos/petry-projects/.github/check-suites/preferences --jq '.preferences.auto_trigger_checks'→ should showsetting: falsefor both app IDsqueuedsuites from Claude or CodeRabbitbash scripts/apply-repo-settings.sh --allto apply across all org repos🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation