You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-on from #1056 (the local-strict-review skill proposal). Filed separately because it's a change to existing PR-flow
skills (pr-review-conduct, drive-pr), not the new skill itself.
Pieter's own description of the failure mode this addresses: a local coding agent finishes a unit
of work, judges it ready, opens the PR; CodeRabbit/Qodo find real issues the local agent didn't
catch; the agent fixes, judges it ready again; another round finds more. Review-round volume this
pass measured at mean roundtrips of 3-6 and mean reworks of 1.5-4.6 per substantive PR across
ProjectTemplate, Financial-Modeling, and HomeAutomation-Config (personal PR-reviewer evaluation,
kept outside this repo).
Proposal
pr-review-conduct and/or drive-pr should chain a fixed sequence of local checks before a
PR is opened, escalating from cheap/mechanical to structural to reviewer-parity:
Local lint — already separately in progress (a pre-commit hook running the same
prose/comment-style/lint scripts CI runs).
CodeRabbit's headless CLI (coderabbit review --uncommitted --agent) — not yet, gated on
CodeRabbit actually graduating from evaluation to standard fleet tooling (see the personal
evaluation doc's First-Class Support Criteria). Slot reserved for it, not built yet.
Each step should be advisory (same posture as local lint today), not a hard gate blocking git commit/gh pr create, at least initially — consistent with how Copilot review currently
works as advisory-not-required.
Why this specific order
Cheapest/most mechanical first (lint catches nothing a human or model needs to reason about),
then structural (strict-review needs a model call but no PR-hosted infrastructure), then
reviewer-parity last (the CodeRabbit CLI shares the same account-wide rate-limit budget as
PR-triggered reviews, so it should only be reached for if the cheaper steps didn't already catch
the issue).
Open questions for whoever implements
Should any step become a hard gate over time, once its false-positive rate on this fleet is
established? (Local lint plausibly could; strict-review's model-based findings probably
shouldn't block a commit outright.)
Where does the "review before opening" step live mechanically — a git hook, a pr-review-conduct
skill instruction, or both (a hook for enforcement, the skill for the agent's own judgment)?
Motivation
Follow-on from #1056 (the
local-strict-reviewskill proposal). Filed separately because it's a change to existing PR-flowskills (
pr-review-conduct,drive-pr), not the new skill itself.Pieter's own description of the failure mode this addresses: a local coding agent finishes a unit
of work, judges it ready, opens the PR; CodeRabbit/Qodo find real issues the local agent didn't
catch; the agent fixes, judges it ready again; another round finds more. Review-round volume this
pass measured at mean roundtrips of 3-6 and mean reworks of 1.5-4.6 per substantive PR across
ProjectTemplate, Financial-Modeling, and HomeAutomation-Config (personal PR-reviewer evaluation,
kept outside this repo).
Proposal
pr-review-conductand/ordrive-prshould chain a fixed sequence of local checks before aPR is opened, escalating from cheap/mechanical to structural to reviewer-parity:
prose/comment-style/lint scripts CI runs).
local-strict-review— the adversarial stronger-model pass against staged changes,proposed in New fleet skill proposal: local-strict-review — adversarial pre-PR self-review on a stronger model #1056.
coderabbit review --uncommitted --agent) — not yet, gated onCodeRabbit actually graduating from evaluation to standard fleet tooling (see the personal
evaluation doc's First-Class Support Criteria). Slot reserved for it, not built yet.
Each step should be advisory (same posture as local lint today), not a hard gate blocking
git commit/gh pr create, at least initially — consistent with how Copilot review currentlyworks as advisory-not-required.
Why this specific order
Cheapest/most mechanical first (lint catches nothing a human or model needs to reason about),
then structural (strict-review needs a model call but no PR-hosted infrastructure), then
reviewer-parity last (the CodeRabbit CLI shares the same account-wide rate-limit budget as
PR-triggered reviews, so it should only be reached for if the cheaper steps didn't already catch
the issue).
Open questions for whoever implements
established? (Local lint plausibly could; strict-review's model-based findings probably
shouldn't block a commit outright.)
pr-review-conductskill instruction, or both (a hook for enforcement, the skill for the agent's own judgment)?