Document reliable Codex review-fix loops#46
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
✅ Files skipped from review due to trivial changes (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a new ChangesReview-Fix-Signoff Loop Skill Introduction
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
skills/review-fix-signoff-loop/SKILL.md (4)
75-103: ⚡ Quick winAlign the example code with the implementation guidance.
The example at line 82 calls
clearStartFromAfterResumedIteration()but this function is never defined. The actual implementation is described later at lines 102-103 as deletingprocess.env.START_FROMandprocess.env.PREVIOUS_RUN_ID.Consider either:
- Implementing the function in the example, or
- Replacing the function call with the actual deletion code
🔧 Suggested fix - Option 2 (inline the logic)
for (let iteration = 1; ; iteration += 1) { await runIteration(iteration, runStamp); // new workflow name, channel, and agent names - clearStartFromAfterResumedIteration(); + // Clear START_FROM after resumed iteration to prevent skipping review/validation in next iteration + delete process.env.START_FROM; + delete process.env.PREVIOUS_RUN_ID; if (hasDualSignoff(iteration)) { writeAndPostSignoffReport(iteration); break; } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/review-fix-signoff-loop/SKILL.md` around lines 75 - 103, The example references a non-existent helper clearStartFromAfterResumedIteration(); fix by either implementing that helper or inlining the described cleanup: after a resumed iteration remove process.env.START_FROM and process.env.PREVIOUS_RUN_ID (or equivalent env keys) before continuing the outer loop; if you add the helper, name it clearStartFromAfterResumedIteration and have it delete those two env vars and be invoked in the example loop where currently called so the example matches the implementation guidance.
18-18: 💤 Low valueClarify the model placeholder in the Codex preflight example.
The preflight probe command uses
-m <supported-model>as a placeholder, but doesn't specify which model to actually use. Consider providing a concrete example using a model constant (e.g.,CodexModels.GPT_5_4) or explaining that any valid model will work for the probe.📝 Suggested clarification
- - Probe the CLIs used by later agent steps. For Codex, `codex login status` is not enough; run a tiny `codex exec --ephemeral --json --sandbox read-only -m <supported-model>` prompt and fail early with a clear re-login instruction if it cannot return the expected token. + - Probe the CLIs used by later agent steps. For Codex, `codex login status` is not enough; run a tiny `codex exec --ephemeral --json --sandbox read-only -m gpt-5.4` prompt (or any supported model) and fail early with a clear re-login instruction if it cannot return the expected token.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/review-fix-signoff-loop/SKILL.md` at line 18, Update the preflight probe text to replace the ambiguous placeholder "-m <supported-model>" with a concrete example and note that any valid model works; specifically mention using a known constant like "CodexModels.GPT_5_4" (or show the concrete model name) in the example `codex exec --ephemeral --json --sandbox read-only -m ...` command and add a short parenthetical that other valid model names can be used for the probe so readers know both a recommended default and that substitution is allowed.
151-160: ⚡ Quick winConsider clarifying how BLOCKED_NO_COMMIT.md should be detected.
The section explains what to write in
BLOCKED_NO_COMMIT.mdwhen blocked, but doesn't explain how the workflow should detect and react to this file. Consider adding guidance on checking for this artifact before commit/push steps.For example:
// Before commit step: .step('check-not-blocked', { type: 'deterministic', command: `test ! -f .workflow-artifacts/<workflow>/iteration-N/BLOCKED_NO_COMMIT.md || (echo "Workflow is blocked"; exit 1)`, failOnError: true, })🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/review-fix-signoff-loop/SKILL.md` around lines 151 - 160, Add a deterministic pre-commit check that detects the presence of .workflow-artifacts/<workflow>/iteration-N/BLOCKED_NO_COMMIT.md and aborts the run if found: implement a step (e.g., step named "check-not-blocked") that runs a filesystem check for BLOCKED_NO_COMMIT.md and fails/throws before any commit/push/post-success actions; reference the step symbol step('check-not-blocked') and ensure it runs prior to commit/push/post-success steps so the workflow will not continue when a BLOCKED_NO_COMMIT.md artifact exists.
49-66: ⚡ Quick winConsider adding a complete verdict example.
The Verdict Contract section defines the field structure but doesn't show what a complete verdict looks like in practice. Adding example outputs for both
COMPREHENSIVELY_SATISFIEDandFINDINGSverdicts would help implementers understand the exact format.For example:
Example satisfied verdict: VERDICT: COMPREHENSIVELY_SATISFIED why_passed: All spec requirements implemented and validated end_to_end_wiring_verified: yes, confirmed via integration tests deterministic_evidence: test-output.log, validation-report.md remaining_risks: none Example findings verdict: VERDICT: FINDINGS finding_id: auth-error-handling-001 severity: high file: src/auth/provider.ts issue: Missing null check on token refresh fix_required: Add guard clause before token.refresh() call test_required: Add unit test for null token case evidence: Line 45 in src/auth/provider.ts🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/review-fix-signoff-loop/SKILL.md` around lines 49 - 66, Add two concrete example verdicts to the "Verdict Contract" section to show exact output formats for both success and findings cases: include a full COMPREHENSIVELY_SATISFIED example with VERDICT, why_passed, end_to_end_wiring_verified, deterministic_evidence, remaining_risks fields and a FINDINGS example with VERDICT, finding_id, severity, file, issue, fix_required, test_required, evidence fields; place these examples immediately after the existing contract block in SKILL.md and ensure they strictly follow the text contract field names and casing so deterministic gates (the VERDICT token and field keys) can parse them reliably.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@skills/review-fix-signoff-loop/SKILL.md`:
- Around line 75-103: The example references a non-existent helper
clearStartFromAfterResumedIteration(); fix by either implementing that helper or
inlining the described cleanup: after a resumed iteration remove
process.env.START_FROM and process.env.PREVIOUS_RUN_ID (or equivalent env keys)
before continuing the outer loop; if you add the helper, name it
clearStartFromAfterResumedIteration and have it delete those two env vars and be
invoked in the example loop where currently called so the example matches the
implementation guidance.
- Line 18: Update the preflight probe text to replace the ambiguous placeholder
"-m <supported-model>" with a concrete example and note that any valid model
works; specifically mention using a known constant like "CodexModels.GPT_5_4"
(or show the concrete model name) in the example `codex exec --ephemeral --json
--sandbox read-only -m ...` command and add a short parenthetical that other
valid model names can be used for the probe so readers know both a recommended
default and that substitution is allowed.
- Around line 151-160: Add a deterministic pre-commit check that detects the
presence of .workflow-artifacts/<workflow>/iteration-N/BLOCKED_NO_COMMIT.md and
aborts the run if found: implement a step (e.g., step named "check-not-blocked")
that runs a filesystem check for BLOCKED_NO_COMMIT.md and fails/throws before
any commit/push/post-success actions; reference the step symbol
step('check-not-blocked') and ensure it runs prior to commit/push/post-success
steps so the workflow will not continue when a BLOCKED_NO_COMMIT.md artifact
exists.
- Around line 49-66: Add two concrete example verdicts to the "Verdict Contract"
section to show exact output formats for both success and findings cases:
include a full COMPREHENSIVELY_SATISFIED example with VERDICT, why_passed,
end_to_end_wiring_verified, deterministic_evidence, remaining_risks fields and a
FINDINGS example with VERDICT, finding_id, severity, file, issue, fix_required,
test_required, evidence fields; place these examples immediately after the
existing contract block in SKILL.md and ensure they strictly follow the text
contract field names and casing so deterministic gates (the VERDICT token and
field keys) can parse them reliably.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 167f4e19-e1cb-444a-8542-dda87532b9d8
📒 Files selected for processing (4)
README.mdprpm.jsonskills/review-fix-signoff-loop/SKILL.mdskills/writing-agent-relay-workflows/SKILL.md
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
prpm.json (1)
134-134: ⚡ Quick winConsider bumping the collection version.
The
agent-relay-startercollection remains at version1.0.3despite adding a new package. Standard practice is to increment the collection version (at minimum a patch bump to1.0.4) when modifying its package list, even for optional packages. This helps consumers track collection changes.📦 Proposed version bump
- "version": "1.0.3", + "version": "1.0.4",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@prpm.json` at line 134, The collection version in prpm.json (the "version" field) was not updated after adding a package to the agent-relay-starter collection; update the "version" value from "1.0.3" to a new patch (e.g., "1.0.4") so consumers can detect the change—edit the "version" property in prpm.json for the agent-relay-starter collection and commit the bump.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@prpm.json`:
- Line 112: Update the ambiguous "Ricky" reference inside the prpm.json
"description" field: either replace "Ricky workflows" with a clear, documented
term such as "Agent Relay" or "iterative review/fix/validation workflow" and
update the description string accordingly, or add a short doc entry that defines
what "Ricky workflows" are and reference that doc from the description; modify
the "description" value to use the clarified term and, if you choose docs, add a
link/identifier to the new documentation so other developers can find the
definition.
---
Nitpick comments:
In `@prpm.json`:
- Line 134: The collection version in prpm.json (the "version" field) was not
updated after adding a package to the agent-relay-starter collection; update the
"version" value from "1.0.3" to a new patch (e.g., "1.0.4") so consumers can
detect the change—edit the "version" property in prpm.json for the
agent-relay-starter collection and commit the bump.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
…off-loop-skill # Conflicts: # README.md # prpm.json # skills/review-fix-signoff-loop/SKILL.md # skills/writing-agent-relay-workflows/SKILL.md
This was referenced May 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing