pr_watch cannot see reviewer unavailability reported as a status-check description

[topij]

What happened

On #22, CodeRabbit was rate-limited. It surfaced only as a status-check description:

CodeRabbit	pass	0		Review rate limited
toolkit	pass	12s	✓

pr_watch reported the PR as converged, clean, with no unavailability signal. I only caught it by running gh pr checks by hand after a re-review I'd been waiting on never arrived.

Why it slipped through

review.unavailable_markers is matched against comment bodies (collect_comments → review_unavailable_reason). This outage never produced a comment. It produced a check whose name is CodeRabbit — which review.informational_checks deliberately excludes from the blocking tally — and whose description carries the reason.

So the two mechanisms miss each other exactly:

• the unavailability detector only reads comments, and there was no comment
• the check that carried the reason is classified informational, and its conclusion is pass

Net effect: a rate-limited bot is indistinguishable from a bot that reviewed and found nothing. Both render as a clean converged report.

Why it matters

The doctrine's load-bearing rule is that a blocked bot is an action signal, never a review waiver — it's supposed to trigger the configured fallback pass. That rule can only fire if the outage is detected. Here it wasn't, so the mechanism silently degraded to "no review happened, and nothing said so."

This is the mirror of #19. There, a queued bot could have a receipt recorded against it (too eager). Here, an unavailable bot produces no signal at all (too quiet). #19 is the more visible failure; this one is the more dangerous, because #19 at least leaves a receipt to audit.

Related: the informational-check exclusion is load-bearing for a different reason — it's what stops the watch loop wedging on a bot that never reports. So the fix cannot simply be "let informational checks block." That tension is the same one #19 runs into, which is why these two probably want solving together.

Sketch, not a decision

Read a configured review bot's check description through the same unavailable_markers matching already applied to comment bodies, and surface a hit as a review_unavailable_reason — visible and blocking mergeable, without making the check itself block converged. That keeps the anti-wedge property while restoring the action signal.

Needs care in the same place #19 does; worth designing the two together rather than patching this one alone.

Repro

#22, head 32f3e4f. gh pr checks 22 shows CodeRabbit pass "Review rate limited" while uv run scripts/pr_watch.py 22 reports converged with no unavailability notice.

[topij]

Refinement from PR #24, same day: the detection is unreliable rather than absent, which is worse than filed.

On #24 CodeRabbit hit the same rate limit, but that time it also posted a comment containing review limit reached — so unavailable_markers matched and pr_watch surfaced it correctly:

• [review unavailable] @coderabbitai: review limit reached — run the configured fallback review

Same bot, same failure mode, same repo, ~1h apart: on #22 it set only a check description (invisible), on #24 it also commented (detected). So the signal depends on which surface the bot happens to use for that particular outage.

That makes the practical risk higher than 'this is never detected'. A check that works most of the time gets trusted, and the silent case then reads as a clean review rather than as a known gap. The fix sketch is unchanged — also match unavailable_markers against a configured review bot's check description — but the motivation should be 'close the unreliable path', not 'add a missing one'.

[coderabbitai]

@topij, agreed—the issue is unreliable outage detection, not missing detection outright. That is more hazardous because successful comment-based detection creates confidence, while check-description-only failures remain silent and can be mistaken for a clean review.

The issue rationale should therefore focus on making the fallback trigger independent of which GitHub surface the configured review bot uses: evaluate unavailable_markers against matching bot check descriptions as well as comments.

🐇🔎

---

✏️ Learnings added

Learnt from: topij
URL: https://github.com/topij/agentic-dev-kit/issues/23

Timestamp: 2026-07-25T10:52:51.380Z
Learning: In the `pr_watch` review-availability detection flow, CodeRabbit rate-limit failures may be surfaced either in a bot comment or only in the configured review bot's GitHub check description. Comment-only matching of `unavailable_markers` is therefore unreliable and can falsely present an unavailable review as clean; detection should also evaluate matching configured review-bot check descriptions.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.