Skip to content

feat(bin): fm-coderabbit-poll.sh — hourly host-cron CodeRabbit review babysitter#281

Open
depohmel wants to merge 6 commits into
kunchenguid:mainfrom
depohmel:feat/coderabbit-poll
Open

feat(bin): fm-coderabbit-poll.sh — hourly host-cron CodeRabbit review babysitter#281
depohmel wants to merge 6 commits into
kunchenguid:mainfrom
depohmel:feat/coderabbit-poll

Conversation

@depohmel

@depohmel depohmel commented Jul 6, 2026

Copy link
Copy Markdown

Adds a host-cron poller that watches a captain-maintained queue of GitHub PRs, detects new CodeRabbit reviews, and dispatches a scoped one-shot crewmate per new actionable review.

Motivation

CodeRabbit free-tier's rate limit is 1 review per hour for the whole org. Existing supervision has no hook for "new external bot review posted on my PR"; a captain either polls manually or leaves reviews unaddressed. Session-based bin/fm-spawn.sh babysitters (tried on this fleet earlier this week) proved brittle for long-idle polling — the model wandered, hallucinated findings, or falsely declared clean without waiting for CodeRabbit to actually review.

Host-cron sidesteps both problems: strict hourly cadence, no live firstmate needed, and each fire spawns a one-shot crewmate with a scoped brief containing the exact review body.

What's here

  • bin/fm-coderabbit-poll.sh — the poll. flock-guarded, per-PR watermarks in state/coderabbit-watermarks.tsv, spawns via existing bin/fm-spawn.sh.
  • docs/coderabbit-poll.md — setup, cadence, files, watermark semantics, crontab example.
  • docs/examples/coderabbit-queue.txt — commented example queue.

Design decisions worth review

  • Host cron over CronCreate: doesn't depend on firstmate being live. Simpler failure mode (cron log tells you).
  • One-shot crewmate per new review, not a persistent babysitter: previous per-session babysitter attempts wedged on idle polling. A crewmate spawned with the exact review body already in hand can't hallucinate findings; it either acts on the concrete body or declines with a comment.
  • Watermark = review submitted_at timestamp, not id: naturally matches CodeRabbit's re-review flow; a fresh review with a newer timestamp triggers dispatch.
  • Watermark advances on spawn success, not on crewmate done: if a crewmate wedges the operator handles it manually; the poll doesn't re-spawn a duplicate on the next fire. If spawn itself fails, watermark stays; next fire retries.
  • Explicit crewmate scope hard rules in the brief: no merges, no force-push, docs and minor fixes only, decline destructive/scope-creep in a PR comment rather than escalating. Escalation reserved for hard blockers only.
  • Actionable-count parsing from CodeRabbit body: Actionable comments posted: N marker; N=0 advances watermark without spawn. If CodeRabbit changes this format, the poll would need an update — this is the load-bearing external contract.

Tests

Smoke-tested locally against:

  • Missing queue file → silent exit 0 (resting state).
  • Empty queue → poll cycle complete, exit 0.
  • Comment-only queue → poll cycle complete, exit 0.
  • Malformed URL → warning + poll cycle complete, exit 0.

Not exercised end-to-end against a live CodeRabbit review here (would double-dispatch on the fleet's own in-flight PRs).

Not in this PR

  • No AGENTS.md updates. If the feature ships, a follow-up should add the three new files (data/coderabbit-queue.txt, state/coderabbit-watermarks.tsv, state/coderabbit-poll.lock) to section 2's layout list.
  • No unit tests under a tests/ dir. The script is mostly shell glue over gh api + fm-spawn.sh; a full test harness would need to mock the GitHub API.
  • No CronCreate variant. Deferred until there's a concrete need for firstmate-driven scheduling.

🤖 Written by Claude Code

Pipeline

Updates from git push no-mistakes

Note: the no-mistakes pipeline was driven manually on this PR — axi run completed intent/rebase/review/test/document/lint successfully; the auto-stamp step failed only because the scratch worktree had HTTPS origin without push credentials. The review pass emitted one actionable finding (the hardcoded LiteLLM tier name in bin/fm-coderabbit-poll.sh), which is fixed by commit 2bb32a9 on this branch (env-var-driven, no hardcoded operator-private tier names).

depohmel added 5 commits July 6, 2026 01:02
…abbit reviews

Periodic host-cron poll that watches a captain-maintained queue of GitHub
PRs, detects new CodeRabbit reviews via the GitHub reviews API, and
dispatches a scoped one-shot crewmate to address each new actionable
review.

Designed for CodeRabbit free-tier's org-wide 1-review-per-hour rate
limit: run hourly via host cron, at most one crewmate per new review.

- bin/fm-coderabbit-poll.sh — the poll script. flock-guarded, watermarked
  per PR, spawns via existing fm-spawn.sh on opencode+litellm/coder.
- docs/coderabbit-poll.md — setup, cadence, file layout, watermark
  semantics, crontab example.
- docs/examples/coderabbit-queue.txt — commented example queue.

Fixes are constrained by an explicit crewmate brief: docs and minor fixes
only, declines destructive/scope-creep findings via PR comment rather than
waking firstmate, never merges or force-pushes. Escalation reserved for
hard blockers (auth, missing tools), not for judgment calls.

Smoke-tested against empty/absent/malformed queue files locally.
…ing fixes

If the crewmate actually pushed commits, it also posts a second PR comment
"@coderabbitai review" to trigger a fresh review of the fixed diff. Skipped
when the crewmate declined every finding — a re-review with no code change
wastes the org-wide 1-per-hour quota.

Also documents this in docs/coderabbit-poll.md.
…tier names

Addresses the pipeline review finding that `bin/fm-coderabbit-poll.sh` baked in
a specific LiteLLM tier alias, which is operator-specific infrastructure that
should not appear in shared/public firstmate code.

- FM_CODERABBIT_POLL_HARNESS - defaults to `opencode` (public, always
  available on a stock firstmate install)
- FM_CODERABBIT_POLL_MODEL - when unset we do NOT pass --model at all; fm-spawn
  then resolves the model through config/crew-dispatch.json and the harness
  default, keeping any private tier names out of this script

Docs updated to reflect the env-var-driven, no-hardcoded-default behavior.
…, log every decision

Before this commit, the poll only ACTED on CodeRabbit reviews that had already
been posted. If CodeRabbit had never been triggered on a queued PR, the poll
would fire hourly and correctly report "nothing new" - but the actual work of
kicking off the first review still had to be done by hand.

Also: PRs that had been merged or closed stayed in the queue and their reviews
kept being consumed against the org quota on every fire.

Changes:
- process_pr now checks pr.state first. Non-open PRs (merged, closed) are
  skipped with a log line, so the queue can safely keep them without wasting
  the CodeRabbit quota.
- When there is NO existing CodeRabbit review on an open PR, the poll posts a
  single "@coderabbitai review" comment - at most one per fire cycle
  (REVIEW_REQUESTED_THIS_CYCLE), so the free-tier one-per-hour quota is
  respected. Remaining unreviewed PRs will get their request on subsequent
  hourly fires.
- Every decision now logs a one-line reason (state=closed skip, no review yet,
  watermark match, N actionable dispatched, spawn failed), so the cron log
  becomes an audit trail: wake up, read one screen, understand what happened.

Docs updated to reflect the new per-PR decision flow.
…uring cooldown

Two fixes so the poll serves both a captain's own repos (paid
coderabbitai[bot]) and OSS repos they contribute to (coderabbit-oss[bot]):

- Match either bot login in the review query and everywhere else via a
  shared BOT_JQ_FILTER. Detect which variant is active on a given PR
  from the newest bot activity (review or issue comment) and mirror the
  @mention accordingly (@coderabbitai vs @coderabbit-oss). Fall back to
  @coderabbitai on PRs with no prior bot activity.
- Before posting @... review, scan the newest bot issue comment for a
  "Review limit reached" cooldown notice. If present, log it (with any
  parseable countdown) and skip - otherwise every hourly fire burned a
  queued-request slot while CodeRabbit was in cooldown.

The existing REVIEW_REQUESTED_THIS_CYCLE gate is preserved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant