Skip to content

pr-review-conduct/drive-pr: chain local checks (lint -> strict-review -> future CodeRabbit CLI) before opening a PR #1057

Description

@ptr727

Motivation

Follow-on from #1056 (the
local-strict-review skill proposal). Filed separately because it's a change to existing PR-flow
skills (pr-review-conduct, drive-pr), not the new skill itself.

Pieter's own description of the failure mode this addresses: a local coding agent finishes a unit
of work, judges it ready, opens the PR; CodeRabbit/Qodo find real issues the local agent didn't
catch; the agent fixes, judges it ready again; another round finds more. Review-round volume this
pass measured at mean roundtrips of 3-6 and mean reworks of 1.5-4.6 per substantive PR across
ProjectTemplate, Financial-Modeling, and HomeAutomation-Config (personal PR-reviewer evaluation,
kept outside this repo).

Proposal

pr-review-conduct and/or drive-pr should chain a fixed sequence of local checks before a
PR is opened, escalating from cheap/mechanical to structural to reviewer-parity:

  1. Local lint — already separately in progress (a pre-commit hook running the same
    prose/comment-style/lint scripts CI runs).
  2. local-strict-review — the adversarial stronger-model pass against staged changes,
    proposed in New fleet skill proposal: local-strict-review — adversarial pre-PR self-review on a stronger model #1056.
  3. CodeRabbit's headless CLI (coderabbit review --uncommitted --agent) — not yet, gated on
    CodeRabbit actually graduating from evaluation to standard fleet tooling (see the personal
    evaluation doc's First-Class Support Criteria). Slot reserved for it, not built yet.

Each step should be advisory (same posture as local lint today), not a hard gate blocking
git commit/gh pr create, at least initially — consistent with how Copilot review currently
works as advisory-not-required.

Why this specific order

Cheapest/most mechanical first (lint catches nothing a human or model needs to reason about),
then structural (strict-review needs a model call but no PR-hosted infrastructure), then
reviewer-parity last (the CodeRabbit CLI shares the same account-wide rate-limit budget as
PR-triggered reviews, so it should only be reached for if the cheaper steps didn't already catch
the issue).

Open questions for whoever implements

  • Should any step become a hard gate over time, once its false-positive rate on this fleet is
    established? (Local lint plausibly could; strict-review's model-based findings probably
    shouldn't block a commit outright.)
  • Where does the "review before opening" step live mechanically — a git hook, a pr-review-conduct
    skill instruction, or both (a hook for enforcement, the skill for the agent's own judgment)?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions