feat(workflows): unify bot reply format and harden research/resolve guards - #91
Merged
Merged
Conversation
- All bot reply surfaces (review/resolve/fix-thread/explain-thread) now emit the same CodeRabbit-style 3-block layout: status line, bold one- line title, prose reasoning. New formatReply() helper in fix-thread.ts keeps handler-emitted output consistent with what the LLM agent prompts demand. - Resolve handler now requires the agent to mark threads resolved via the resolve-review-thread MCP tool after replying to Valid / Partially Valid threads, and to poll gh pr checks until all checks reach a terminal non-failure state (FIX_ATTEMPTS_CAP=3 per iteration). Survivors are recorded under ## Outstanding in RESOLVE.md. - Research workflow pre-installs @mermaid-js/mermaid-cli and adds a hard validation gate (Step 4.5) so every Mermaid block must validate via mmdc before gh issue create runs (closes #86). Forbidden-issue rules expanded with explicit blocklist of placeholder titles and an empty- body floor (closes #73). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughThe PR standardizes output formatting across multiple bot workflows by enforcing a consistent three-block Markdown structure (status line, bold title, reasoning) for replies and inline comments, updating handler prompts to specify exact formatting requirements, adding Mermaid diagram validation to the research workflow, and extending test assertions to verify the new formats are present. ChangesStandardized Bot Output Formatting
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 46 minutes and 14 seconds.Comment |
There was a problem hiding this comment.
Pull request overview
Unifies the bot’s reply formatting across review/resolve and scoped thread commands, and tightens resolve + research workflow guards (CI wait-for-green, Mermaid validation, and placeholder-issue prevention).
Changes:
- Standardize bot replies to a CodeRabbit-style 3-block format (status line → bold title → reasoning) across prompts and
fix-thread. - Extend
resolveagent instructions to resolve Valid/Partially Valid threads via MCP and to pollgh pr checksuntil CI is green (with a per-iteration attempts cap). - Harden the scheduled research workflow by installing
@mermaid-js/mermaid-cli, adding a mandatory Mermaid validation gate, and forbidding placeholder issues.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/workflows/ship/scoped/fix-thread.ts |
Adds formatReply() helper and updates reply bodies; introduces optional reasoning in results. |
test/workflows/ship/scoped/fix-thread.test.ts |
Updates assertions for the new reply format and adds fallback-reasoning coverage. |
src/workflows/ship/scoped/explain-thread.ts |
Updates system prompt to require the unified 3-block reply layout. |
src/workflows/handlers/review.ts |
Updates the review agent prompt to require the unified inline-comment layout and status lines. |
src/workflows/handlers/resolve.ts |
Updates resolve agent prompt: explicit per-class behavior, mandatory reply template, and CI wait-for-green step. |
docs/use/workflows/review.md |
Documents the new inline status lines alongside existing REVIEW.md severity tags. |
docs/use/workflows/resolve.md |
Documents unified reply format, thread resolution behavior, and CI polling requirement. |
.github/workflows/research.yml |
Installs mermaid-cli, adds Mermaid validation gate instructions, and expands placeholder-issue forbiddance + allowed tools. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/research.yml:
- Around line 349-354: The blacklist currently forbids any title containing the
literal word "test", which conflicts with the allowed test(...) type and
examples like "test(webhook): ..."; update the banned-title rule so it only
blocks standalone or single-word titles containing test (e.g., "test",
"test-can-delete") but explicitly permits conventional-commit style type
prefixes such as "test(...):" (e.g., "test(webhook): ...") and adjust the
example or wording to mention the allowed "test(...)" format to remove the
contradiction.
- Around line 64-67: Update the GitHub Actions step that installs the Mermaid
CLI to pin the package to a stable release instead of grabbing latest; change
the npm install invocation for the package symbol "@mermaid-js/mermaid-cli" to
include a specific version (e.g., `@mermaid-js/mermaid-cli`@11.14.0) so subsequent
runs use a reproducible CLI, and keep the verification step that calls "mmdc
--version" to confirm the pinned version is installed.
In `@src/workflows/handlers/resolve.ts`:
- Around line 272-288: The handler currently treats any successful runPipeline()
as an overall success even when post-fix CI still shows failing checks; change
the finalization logic in this file to re-query GitHub checks after the agent
finishes fixes (use the same gh/octokit check routine used in the CI polling
loop), evaluate terminal conclusions against the "all-green" definition, and
only set/return status: "succeeded" when checks are all-green, all comments
resolved, and reviewDecision is APPROVED; if FIX_ATTEMPTS_CAP is reached with
remaining failures, ensure RESOLVE.md has an "Outstanding" section and
set/return a non-success status (e.g., "failed" or "incomplete") and post that
state via update_claude_comment so downstream workflows see the true post-fix CI
result (update references to runPipeline, update_claude_comment, and RESOLVE.md
in the finalization block).
In `@src/workflows/handlers/review.ts`:
- Around line 258-277: The guidance incorrectly states the emoji STATUS_LINE
strings drive REVIEW.md severity counts but the parser function countFindings()
actually looks for bracketed tags ([blocker]/[major]/[minor]/[nit]), so update
the text around the STATUS_LINE example to explicitly say that REVIEW.md must
include bracketed severity tags (and show the bracketed example) and remove or
correct the claim that the emoji lines drive counting; ensure references to
findings.total and countFindings() are consistent with this change so authors
can't rely on emojis alone.
In `@src/workflows/ship/scoped/explain-thread.ts`:
- Around line 19-33: The template example in explain-thread.ts contains a
two-space indentation before the sample layout which causes the header to become
" _💡 Explanation_" instead of the exact required "_💡 Explanation_"; locate the
prompt/template string (the constant or variable that holds the three-block
layout sample used by the explain thread generator) and remove the leading
spaces so the first line is exactly "_💡 Explanation_" with no extra characters,
ensure the blank second and fourth lines remain, and keep the bold one-sentence
third line format and the bulleted/body formatting unchanged.
In `@src/workflows/ship/scoped/fix-thread.ts`:
- Around line 181-199: Extract the file-local formatReply function into a shared
exported helper (keep the same signature: status, meta?, title, reasoning) and
import it from that single module wherever formatting is needed; replace the
duplicated formatter logic in the explain-thread, review handler, and resolve
handler code paths with calls to the shared formatReply function so all reply
surfaces (including buildResolvePrompt/buildReviewPrompt consumers) use the same
implementation and stay in sync.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: e41dae56-336a-47ad-8260-0630ea9ee78c
📒 Files selected for processing (8)
.github/workflows/research.ymldocs/use/workflows/resolve.mddocs/use/workflows/review.mdsrc/workflows/handlers/resolve.tssrc/workflows/handlers/review.tssrc/workflows/ship/scoped/explain-thread.tssrc/workflows/ship/scoped/fix-thread.tstest/workflows/ship/scoped/fix-thread.test.ts
- Pin @mermaid-js/mermaid-cli to 11.14.0 in research.yml so the Mermaid validation gate behaves reproducibly across runs. - Tighten the placeholder-issue blocklist so legitimate conventional- commit test(<scope>): ... titles are no longer banned alongside actual placeholders like "smoke test" / "test-can-delete". - Clarify in review.ts that STATUS_LINE strings drive inline-comment presentation only; REVIEW.md severity counts come from the bracketed [blocker]/[major]/[minor]/[nit] tags (one source of truth, no contradictory guidance). - Align resolve.ts step 3 wording with the FIX_ATTEMPTS_CAP=3 cap in step 6 — total fix attempts across both steps share the same cap; remove the contradictory "do NOT retry more than once". - Remove leading 2-space indent from the explain-thread template example so the model doesn't mirror it into a contract-violating " _💡 Explanation_" header. - Extract formatReply() to src/workflows/format-reply.ts so future reply surfaces import the helper instead of duplicating it. - Hoist the redundant trim() chain in fix-thread.ts into a single trimmedReasoning local. - Fix British "summarising" -> American "summarizing" in resolve.md to match the resolve.ts handler prompt. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4 tasks
chrisleekr
pushed a commit
that referenced
this pull request
May 2, 2026
# [1.8.0](v1.7.0...v1.8.0) (2026-05-02) ### Bug Fixes * **logger:** redact paths and scrub err.* before pino emits (closes [#52](#52)) ([#89](#89)) ([641f138](641f138)) * **security:** redact raw error messages from public PR comments ([#90](#90)) ([cc70949](cc70949)) ### Features * **workflows:** unify bot reply format and harden research/resolve guards ([#91](#91)) ([7d39fb4](7d39fb4))
Owner
Author
|
🎉 This PR is included in version 1.8.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This was referenced May 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related improvements bundled across the bot's review/comment surfaces:
review(inline findings),resolve(thread replies),bot:fix-thread,bot:explain-thread— now emit the same CodeRabbit-style three-block layout: a status line, a bold one-line title, then 1–3 sentences of reasoning. Severity/classification is encoded in the status line (_⚠️ Potential issue_ | _🔴 Blocker_,_✅ Addressed_,_💬 Design discussion_, etc.), and a smallformatReply()helper infix-thread.tskeeps handler-emitted replies consistent with what the LLM agent prompts demand.resolve-review-threadMCP tool after replying to Valid / Partially Valid threads, and (b) pollgh pr checksafter its final commit until all checks reach a terminal non-failure state, capped atFIX_ATTEMPTS_CAP=3per iteration. Surviving failures are recorded under## OutstandinginRESOLVE.md.@mermaid-js/mermaid-cliand adds a hard validation gate (Step 4.5) so every Mermaid block in the issue body must validate viammdcbeforegh issue createruns — closes the failure mode from issue test(coverage): per-file Bun isolation overwrites coverage/lcov.info on every invocation, eliminating aggregate coverage signal #86. Also explicitly forbids test/smoke/placeholder issues (titles containingtest,smoke,wip,todo, etc., or empty bodies) — closes the failure mode from issue test-can-delete #73.Diagram
flowchart TD classDef before fill:#fde68a;color:#1f2937;stroke:#92400e;stroke-width:2px classDef after fill:#bbf7d0;color:#064e3b;stroke:#065f46;stroke-width:2px Reviewer[Reviewer comment] --> BeforeReply["BEFORE<br/>severity tag in body<br/>e.g. major description<br/>thread NOT auto-resolved<br/>handlers and agents diverge"]:::before Reviewer --> AfterReply["AFTER<br/>3-block CR-style reply<br/>status line + bold title + reasoning<br/>thread resolved via MCP<br/>handlers and agents share format"]:::after ResolveStart[Resolve run starts] --> BeforeCI["BEFORE<br/>fix then push then exit<br/>CI may still be red on exit"]:::before ResolveStart --> AfterCI["AFTER<br/>fix then push then poll checks<br/>exit only when green or cap=3 hit<br/>survivors logged in RESOLVE.md"]:::after ResearchStart[Research workflow starts] --> BeforeMermaid["BEFORE<br/>prompt-only Mermaid rules<br/>invalid blocks ship as parse errors<br/>see issue #86"]:::before ResearchStart --> AfterMermaid["AFTER<br/>mermaid-cli pre-installed<br/>each block validated via mmdc<br/>discard issue if 3 attempts fail"]:::afterChanges
src/workflows/ship/scoped/fix-thread.ts— addformatReply()helper; rewrite all four reply paths (applied, design-discussion refusal, two skip paths) to use the unified 3-block layout; add optionalreasoningfield onApplyMechanicalFixResultso callers can surface the why without the reviewer opening the commit.src/workflows/ship/scoped/explain-thread.ts— rewriteEXPLAIN_THREAD_SYSTEM_PROMPTto require the same 3-block layout (_💡 Explanation_status line, bold summary, prose body).src/workflows/handlers/review.ts— replace the[blocker]/[major]/[minor]/[nit]body convention with the CR-style status line + suggested-fix block; keep the bracketed tags inREVIEW.mdso thecountFindingsparser still works.src/workflows/handlers/resolve.ts— split the per-classification reply guidance into four explicit cases; add the mandatory reply-body template; add the new "wait for CI to be green before exiting" step withFIX_ATTEMPTS_CAP=3semantics; update RESOLVE.md template to include final post-fix CI state and per-thread resolution status.test/workflows/ship/scoped/fix-thread.test.ts— assert the new status lines and bold titles in each reply path, and add a new test covering the generic-reasoning fallback when the callback omitsreasoning.docs/use/workflows/resolve.md— document the unified reply-body format, the per-classification status-line table, the thread-resolution behaviour, and the new wait-for-green step.docs/use/workflows/review.md— replace the severity-prefix table with the new dual-column table that covers both inline-comment status lines and REVIEW.md tags; explain why both are required..github/workflows/research.yml— install@mermaid-js/mermaid-cli; add Step 4.5 (Mermaid validation gate) with retry-up-to-3 semantics and discard-on-final-failure; expand the forbidden-issues rules with the explicit blocklist of placeholder titles and the empty-body floor; widen--allowedToolsto includemmdc,awk,grepfor the validation step.Related Issues
Test plan
bun test test/workflows/ship/scoped/fix-thread.test.tsresolverun on a stale-CI PR and confirm it polls until green and resolves Valid threadsworkflow_dispatchthe research workflow with a focus that historically produced a Mermaid diagram and confirm the validation gate runsSummary by CodeRabbit
Release Notes
New Features
Documentation
Tests